On previous meetup we discussed theoretical issue: How to develop application which allows to download few dozens small files from server, write some logs about it and execute http call if download was successful?
Actually task is quite easy but... we have few restrictions
1) We have only one Tomcat (means no cluster)
2) Expected number of downloads about 10 000 000 per day
So together we decided
1) Put files in cache 2) Execute async call to remote server Pretty simple. So I started implementing solution. First question was how to put file in cache. I didn't find any implemented solution and only one article.
Below my version of caching files
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Value;
public class FileCacheAdapter extends CacheAdapter {
private static final Logger log = LoggerFactory.getLogger(FileCacheAdapter.class);
@Value(value = "#{'${file.path}'}")
private String filePath;
public InputStream getFile(String fileName) throws IOException {
StringBuilder lookupFile = new StringBuilder(filePath);
lookupFile.append("\\");
lookupFile.append(fileName);
Path filePath = Paths.get(lookupFile.toString());
boolean isExist = Files.exists(filePath);
if (!isExist) {
log.debug("File with fileName: {} was removed!", fileName);
remove(fileName);
return null;
}
long lastModified = Files.getLastModifiedTime(filePath).toMillis();
FileStamp fileStamp = get(fileName);
if (fileStamp != null && fileStamp.getLastModified() == lastModified) {
return new ByteArrayInputStream(fileStamp.getContent());
}
updateFile(fileName);
return getFile(fileName);
}
private void updateFile(String fileName) throws IOException{
remove(fileName);
putFile(fileName);
}
public void putFile(String fileName) throws IOException {
tryWriteLockOnKey(fileName, 2000L);
try {
if (get(fileName) != null) {
return;
}
StringBuilder lookupFile = new StringBuilder(filePath);
lookupFile.append(fileName);
Path filePath = Paths.get(lookupFile.toString());
byte[] fileContent = getFileContent(filePath);
long lastModified = Files.getLastModifiedTime(filePath).toMillis();
FileStamp fileStamp = new FileStamp(fileName, lastModified, fileContent);
put(fileName, fileStamp);
} finally {
releaseWriteLockOnKey(fileName);
}
}
private byte[] getFileContent(Path path) throws IOException {
byte[] fileContent = Files.readAllBytes(path);
return fileContent;
}
}
public class FileStamp implements Serializable {
/**
*
*/
private static final long serialVersionUID = -4056582522653888648L;
/**
* A name of the file
*/
private String fileName;
/**
* A time when the file was modified last time.
*/
private long lastModified;
/**
* A content of the file.
*/
private byte[] content;
public FileStamp() {
super();
}
public FileStamp(String fileName, long lastModified, byte[] content) {
this.fileName = fileName;
this.lastModified = lastModified;
this.content = content;
}
public String getFileName() {
return fileName;
}
public void setFileName(String fileName) {
this.fileName = fileName;
}
public long getLastModified() {
return lastModified;
}
public void setLastModified(long lastModified) {
this.lastModified = lastModified;
}
public byte[] getContent() {
return content;
}
public void setContent(byte[] content) {
this.content = content;
}
}
Lets analyze step by step how it works. Fetching file from cache
public InputStream getFile(String fileName) throws IOException {
StringBuilder lookupFile = new StringBuilder(filePath);
lookupFile.append("\\");
lookupFile.append(fileName);
Path filePath = Paths.get(lookupFile.toString());
boolean isExist = Files.exists(filePath);
if (!isExist) {
log.debug("File with fileName: {} was removed!", fileName);
remove(fileName);
return null;
}
long lastModified = Files.getLastModifiedTime(filePath).toMillis();
FileStamp fileStamp = get(fileName);
if (fileStamp != null && fileStamp.getLastModified() == lastModified) {
return new ByteArrayInputStream(fileStamp.getContent());
}
updateFile(fileName);
return getFile(fileName);
}
I use new NIO API just to avoid file lock. As you see I check if file is present on filesystem and return null if there is no such file.
Then I check if file was modified and updated cache if it is needed.
Next method, actually three methods are responsible for inserting file in cache
As you know we take part in O'Reilly review program.
Several weeks ago I received for review "MapReduce Design Patterns" by Donald Miner, Adam Shook. Actually it was my first experience with mapreduce and especially with hadoop. Hence it force me to read some articles
conclusion 1) This book is not for dummies but...
I've spent some time (not much really) to pick up some basic knowledge about mapreduce and hadoop afterwards it was much easier to understand. Book is well written and easy to read, I would even say that examples are too extensive.
conclusion 2) Good choice if you need to implement something and don't have enough of knowledge.
All patterns in the book are grouped by such types: Summarization, Filtering, Data Organization, Join, Metapatterns and Input/Output patterns. The materials are well-structured, so it speeds up reading. I really like it 'cause from time to time I need to recall something without parsing the whole book conclusion 3) book is well organized.
Last chapter is called "Final thoughts and the Future of Design patterns" and it was most interesting chapter for me, but I will not reveal it so you'll enjoy reading it too))
За остані 2а роки хадупівський кластер виріс від 30 до 4000 машин. Хочете дізнатися the best practice, що використовуються в Twitter для обробки мільйонів твітів щодня? тоді вам сюди: http://www.infoq.com/presentations/Twitter-Analytics
В червні JUG планує провести Java конференцію. Будемо раді будь-яким думкам, ідеям та зауваженням по темі
Сторінка конференції JDay Lviv
Будемо вдячні всім хто розмістить інформацію про конференцію та посилання на своїх блогах, FB сторінках, Google+ чи Twitter (хештег #JDayLviv). Чим більшу аудиторію ми залучимо тим якісніші будуть спонсори і відповідно доповідачі.
Дорогі Друзі дякую всім за відвідини нашої зустрічі.
Цього разу нас зібралось близько 60 осіб і це була одна з найчисельніших зустрічей за всю історію.
Також це був експеремент. І як кожен експеремент - прийшов час отримати результати.
Напишіть ваші враження, думки, побажання.
Опитування є анонімнм, але якщо ви хочите отримати відповідь на ваше запитання чи побажання - заповніть поле зворотнього звязку.
What is Java Collections Framework? List out some benefits of Collections framework?
Collections are used in every programming language and initial java release contained few classes for collections:Vector,Stack,Hashtable,Array. But looking at the larger scope and usage, Java 1.2 came up with Collections Framework that group all the collections interfaces, implementations and algorithms. Java Collections have come through a long way with usage of Generics and Concurrent Collection classes for thread-safe operations. It also includes blocking interfaces and their implementations in java concurrent package. Some of the benefits of collections framework are:
Reduced development effort by using core collection classes rather than implementing our own collection classes.
Code quality is enhanced with the use of well tested collections framework classes.
Reduced effort for code maintenance by using collection classes shipped with JDK.
