Question

I’m looking to run a persistent instance of a python interpreter that can be passed input. The entire system designed to do this will be written in PHP. I’ve noticed the Pheanstalk package for use over the beanstalkd work queue. Has anyone used Pheanstalk to spawn (and maintain) a persistent process? I understand that the common use for Pheanstalk is to run scripts or other tasks that take extended periods of time in an asynchronous matter in order to avoid the user waiting. I am looking to, asynchronously, have a Pheanstalk job/process that manages and maintains a python interpreter process that I can pass input to. Is this possible/has it been done, and if so, how?

Answer 1

I’ve certainly run persistent queue workers using Pheanstalk. It works something like this:

A parent PHP process forks the specified number of worker processes. Each worker uses Pheanstalk to connect to beanstalkd and wait for a job. When a worker gets a job, it performs the task and then exits, freeing its memory. The parent PHP process then spawns a new worker to replace it.

Your requirement sounds more like this:

Parent PHP process uses proc_open() to fork a python interpreter process, storing references to the STDIN and STDOUT pipes created. Then the parent PHP process enters a loop, using Pheanstalk to connect to beanstalk and reserve the next job. When the job is reserved, PHP uses fwrite() to send the job content to the python STDIN pipe, and then uses fread() to read the response from pythons STDOUT pipe.

This is assuming you want to maintain a running python process, rather than fire up a new python process each time a job is reserved.

Crazy either way, but it might work.

The saner option would be using a Python beanstalkd client, but hey, great to see more people using Pheanstalk 🙂

Cheers,
Paul