Question

I am building a Java EE web application, which also has a JAX-RS API so that plugins at other web sites can send data to it, think of it like an analytics server which receives data from various client sites and has its own admin interface where one can view the stats.

The data that the clients send is currently saved in a temporary table in the database; and is supposed to be further processed afterwards, and saved into the real table holding the statistic data.

How should I implement the background processing of that data afterwards? Should I create threads myself, or should I use a message queue system (I have looked at RabbitMQ and it seems promising, however I’m not sure if it is appropriate for embedding in a Java EE application)

Also, should I be saving the temporary data (one chunk of information from the user, as a JSON/XML string) in my current database, or would you think that saving in another temporary database is more appropriate, for example I have again heard of nosql datas like Redis, mongodb etc., however cannot decide whether they are suitable for this.

Finally, if I should use a message queue, should the requests be handled by the message queue, or again JAX-RS? My concern with JAX-RS would be that it may not be as performant as queues and as I said, I don’t really need an API, I just want to get a single base-64 encoded string (that contains various statistical info) from the client web sites.

Answer 1

Lots of questions here 🙂 I’ll try to answer in order:

• You won’t be able to embed RabbitMQ in your JavaEE application because it runs on Erlang. If embedding the message broker in the JavaEE application is a requirement for you, you’d rather look at ActiveMQ or HornetQ.
• If you use the durable storage feature of a messaging system, you won’t need to store the incoming messages in a temporary database. This approach would work fine for you if your incoming messages are small (KBs not MBs).
• Using a messaging system as the main keeper of incoming messages will also provide an answer to your threads question: use 1-n concurrent consumers (ie concurrent consumers for the particular messaging system you’ve selected) on the message queue to process the messages and write the results in the statistics database.
• Finally, think twice about exposing your messaging system directly instead of behind an HTTP facade (JAX-RS in your case), as not all your clients may have the capacity to speak the right protocol (if you go for RabbitMQ, AMQP has excellent cross-platform support though). Also, it might be a security concern to expose it directly.