Question

I am considering building a site using php, but there are several aspects of it that would perform far, far better if made in node.js. At the same time, large portions of of the site need to remain in PHP. This is because a lot of functionality is already developed in PHP, and redeveloping, testing, and so forth would be too large of an undertaking, and quite frankly, those parts of the site run perfectly fine in PHP.

I am considering rebuilding the sections in node.js that would benefit from running most in node.js, then having PHP pass the request to node.js using Gearman. This way, I scan scale out by launching more workers and have gearman handle the load distribution.

Our site gets a lot of traffic, and I am concerned if gearman can handle this load. I wan’t to keep this question productive, so let’s focus largely on the following addressable points:

• Can gearman handle all of our expected load assuming we have the memory (potentially around 3000+ queued jobs at at time, with several thousand being processed per second)?
• Would this run better if I just passed the requests to node.js using CURL, and if so, does node.js provide any way to distribute the load over multiple instances of a given script?
• Can gearman be configured in a way that there is no single point of failure?
• What are some issues that you guys can see arising both in terms of development and scaling?

I am addressing these wide range of points so anyone viewing this post can collect a wide range of information in one place regarding matters that strongly affect each other.

Of course I will test all of this, but I want to collect as much information as possible before potentially undertaking something like this.

Edit: A large reason I am using gearman is not because of it’s non-blocking structure, but because of it’s sheer speed.

Answer 1

I can only speak to your questions on Gearman:

Can gearman handle all of our expected load assuming we have the memory (potentially around 3000+ queued jobs at at time, with several thousand being processed per second)?

Short: Yes

Long: Everything has its limit. If your job payloads are inordinately large you may run into issues. Gearman stores its queue in memory.. so if your payloads exceed the amount of memory available to Gearman you’ll run into problems.

Can gearman be configured in a way that there is no single point of failure?

Gearman has a plugin/extension/component available to use MySQL as a persistence store. That way, if Gearman or the machine itself goes down you can bring it right back up where it left off. Multiple worker-servers can help keep things going if other workers go down.