Question

I’m trying to change my current application to scale.

It can currently handle at most a few million events per hour but the volume is expected to grow 10 to 100 times fold as I switch to a SaaS model, so it is important to be able to execute the processing in a distributed fashion.

The app is a web application receiving at peek currently 1.2 million events/hour. It is using 2 Tomcat servers, each listening on 500 threads and a workManager to queue the events and will then spawn a couple of hundred worker threads to postprocess the events.

What I’m trying to do is decoupling the writing from processing and move the processing to a distributed environment.

1. Fast write to the disk the events.

   Here the solution can be as simple as writing to a LinkedBlockingQueue and dump batches of hundred, thousand of entries to a file or use a good library that does this already or tune the database to support this type of queueing in a reasonable fashion.

   Failing to capture the last events if the system becomes unavailable is not paramount, the focus is performance when the server is working.

2. Move the event processing to a distributed system.

   I need to move the data to a distributed system (e.g. HDFS). What other options are there?

   The processing is of medium complexity (e.g. some of the complexity is in a self-join generating a frequent itemset and further filtering down this set, other parts involve aggregating data across multiple hierarchies). I currently use a database (MySql & DB2) and thinking about Hadoop. Any other options?

3. Store the results in a readonly fast read system.

I am currently using SOLR, any better options?

I know the question spawns multiple topics, any input is appreciated. Let me know if there’s a better tag I could use.

Thanks!

Sebi

Answer 1

The best system today, capable of both insertions and queries – indeed is RDBMS. But it is not scalable. NoSQL systems are not scalable because they built better, but because they gave up something..
Let see what can be built from them.
Both HBase a Cassandra are built specially to translate random events insertion to sequential disk IO. In other words – they are write optimized system and you can consider them as perfect distribtuted database index. So you can get to any insertion rate you need by adding more nodes

Regarding joins and aggregations is a problematic point there.
If you will succeed to design your key in the way that data to be aggregated is collocated – data can be pulled and aggregated efficiently.
Joins is also problematic but there is an option to write data already prejoined. You should do it in application level.
For more complex processing you will need to resort to MapReduce but it probabbly will affect insertion rates.
DataStax’s Brisk sounds like good for your case since it has Cassandra preintegrated with the MapReduce with the capability to run MapReduce right over Cassandra Data. It also has capability to reduce MapReduce affect on the OLTP part of the story.