There’s a beanstalkd queue, which gets filled with a lot of tasks say every 10 minutes and it is top priority that each task is processed ASAP. Task can take more that a few milliseconds to complete for there are calls to third-party services, which tend to timeout every now and then.
So since PHP doesn’t have multithreading, one option would be to create a lot of idle workers, which would try to reserve a task, but it is likely to take too much RAM, which may not be available on those boxes.
Is it a good idea to use PHP-FPM to adjust the number of workers and save some RAM? Is it production-ready? Are there better solutions?
Thanks
I’m running a queue system that is dealing with millions of messages per day. Mostly via Amazon SQS, but I’m also running a new Beanstalkd system with over 600,000 msgs in there right now.
As is described in a blogpost on the subject, I have shell scripts running in a loop processing messages (a loop within the PHP script to run multiple jobs before returning is also somewhat useful, at least for smaller jobs).
Those shell scripts are started with Supervisord. There’s another blog post on the use of that as well. I’m currently running over 800 worker scripts (for a few different types of jobs) across nine machines, all pulling from various queues and putting data back into other queues, writing to the DB or files. Increasing the number of workers per machine is a matter of increasing the “numprocs” (or having it large enough already), and then starting more as required. You could also have say 5 auto-start, and then another block of 50 that are ready to start as required.
I find each worker only takes around 20mb of non-shared memory (the rest is common between processes). This does depend on the tasks that the workers do of course. Resizing images can take a lot of effort. It’s partly for this reason I have setup to be able to frequently restart the PHP script.