Question

I am writing a db logging ruby gem which will simply take out a job from a Beanstalk queue and write it in the DB.

That is one process on Server A puts a job (that it wants to log) in the Beanstalk queue on Server B, and my logging process on Server B takes it out and writes it to the mysql DB on Server B.

I want to know if this is worth it?
Is putting a job in the Beanstalk queue faster than writing to the DB. Or can my process that wants to log to DB directly write it to DB instead of using the logging process.

Note that both the beanstalk server and DB are on another server.

Beanstalk internally makes a socket call from Server A to Server B.
I believe mysql would need to do the same as well?

So therefore is mysql to another server going to be slower than putting in the beanstalk queue.

Answer 1

It’ll be much faster, primarily because Beanstalkd jobs, by default, are stored in-memory and are lost if, for example, you lose power on your server, whereas MySQL is a strongly ACID-compliant relational database, and hence will go to a lot of effort and flush each of your logs to disk.

I think you’ll find that, after your do some benchmarking with a lot of logs being made by your system, that disk I/O will be your limiting factor, rather than the speed of TCP/IP sockets. Your current system’s advantage is that when server A files a log on Server B’s beanstalkd instance it takes up very little of Server A’s time, and Server B can periodically flush our many logs at once from beanstalkd to MySQL, making the process more efficient. The disadvantage is that, the more you batch up the logs, the more logs you will lose in the event of a software / power failure, unless you use beanstalkd’s “-b” parameter which makes jobs durable by writing them to disk (and hence making the process slower).

Of course, the only way to truly settle this question is to benchmark!