Translate

четвер, 28 лютого 2013 р.

HighLoad FileDistributor: Part 1 - FileCacheAdapter


On previous meetup we discussed theoretical issue: How to develop application which allows to download few dozens small files from server, write some logs about it and execute http call if download was successful?
Actually task is quite easy but... we have few restrictions
1) We have only one Tomcat (means no cluster)
2) Expected number of downloads about 10 000 000  per day

So together we decided
1) Put files in cache
2) Execute async call to remote server
Pretty simple.

So I started implementing solution. First question was how to put file in cache. I didn't find any implemented  solution and only one article.
Below my version of caching files
 
Lets analyze step by step how it works. Fetching file from cache
public InputStream getFile(String fileName) throws IOException {
    StringBuilder lookupFile = new StringBuilder(filePath);
    lookupFile.append("\\");
    lookupFile.append(fileName);
    Path filePath = Paths.get(lookupFile.toString());
    boolean isExist = Files.exists(filePath);
    if (!isExist) {
      log.debug("File with fileName: {} was removed!", fileName);
      remove(fileName);
      return null;
    }
    long lastModified = Files.getLastModifiedTime(filePath).toMillis();
    FileStamp fileStamp = get(fileName);
    if (fileStamp != null && fileStamp.getLastModified() == lastModified) {
      return new ByteArrayInputStream(fileStamp.getContent());
    }
    updateFile(fileName);
    return getFile(fileName);
  }

I use new NIO API just to avoid file lock. As you see I check if file is present on filesystem and return null if there is no such file.
Then I check if file was modified and updated cache if it is needed.

Next method, actually three methods are responsible for inserting file in cache

private void updateFile(String fileName) throws IOException{
    remove(fileName);
    putFile(fileName);
  }

  public void putFile(String fileName) throws IOException {
    tryWriteLockOnKey(fileName, 2000L);
    try {
      if (get(fileName) != null) {
        return;
      }
      StringBuilder lookupFile = new StringBuilder(filePath);
      lookupFile.append(fileName);
      Path filePath = Paths.get(lookupFile.toString());
      byte[] fileContent = getFileContent(filePath);
      long lastModified = Files.getLastModifiedTime(filePath).toMillis();
      FileStamp fileStamp = new FileStamp(fileName, lastModified, fileContent);
      put(fileName, fileStamp);
    } finally {
      releaseWriteLockOnKey(fileName);
    }
  }

  private byte[] getFileContent(Path path) throws IOException {
    byte[] fileContent = Files.readAllBytes(path);
    return fileContent;
  }

Just to avoid parallel cache updating I lock cache record
tryWriteLockOnKey(fileName, 2000L);
In the next part will be described async call to remote server