I am looking for a simple persistent buffer as temporary storage for JSON messages in a Java application. Memory usage should be relatively constant and not depend on the number of messages in the buffer. It would nice to be able to replay messages from a point in the past. Deletion of old messages should be efficient. Needs to be able to handle 1m messages/h.
Currently my application uses a local RabbitMQ broker which shovels messages to a remote RabbitMQ broker. When the remote broker is down or not accepting messages the local RabbitMQ broker’s memory usage rises with the queue length and eventually it stops accepting messages. I want to swap this out for a local disk based buffer and a thread copying messages to the remote RabbitMQ broker.
Anyone have any ideas? I have looked at Kafka but it seems like overkill for my use-case. MongoDB is a possibility but I am worried about its memory usage.
Memory usage is always an issue in any system.I am using MongoDB for production and when I compare with similar solutions (CouchDB,CouchBase,redis.io), MongoDB is really good in memory management and easiness of implementation. But I should admit , I never had a chance to test Riak more in detail.
I am storing 5.000.000 user records with 4 index fields and all user session behind a rest/web service api which uses a messaging service behind.
My messaging service uses another db instance on the same server.
My user records have at least 20 fields and session records have just 5 fields.
My ubuntu servers never used more than 10 GB rams even with heavy loading processes.
Hope this helps to figure out.
ps: all depend on data model and how you implement your infrastructure.
Regards,
EDIT:
I think this is a good slideshow about using MongoDB for messaging.
and a nice article about MongoDB and messaging.
You can use the test code and see the results are ok for your solution.
Please don’t forget to share your results if you test.