Question

First things first, I’m aware of this question:

• Gearman: Sending data from a background worker to the client

What I want to know, is it still the case with Gearman? I’m planning on sending a batch of image URLs from a PHP web application to the gearman worker (also written in PHP; let’s call it “The Main Worker”) for processing asynchronously. This worker will then submit a separate task for each image to lower-tier workers (via addTask()), call runTasks() and wait for the tasks to finish, while listening to exceptions, accumulating error messages and updating the overall job status.

While I’m perfectly ok with retrieving the overall status from the Main Worker using jobStatus() calls, then just say that all of the images were processed when [false, false, 0, 0] is returned, I definitely need to be able to inform the users that some of the images couldn’t be retrieved from their respective URLs or stored on the server.

I suppose I could always just store the custom data in memcache, then retrieve it from the web app, but it just seems “dirtier” to me…

I’m not trying to get any result, because from what I’ve seen in the manual on php.net, even the exception handling can only be done when the task is submitted synchronously, not mentioning the custom data retrieval. I just hoped that there could be something I’m missing.
I’m I remember correctly, we’re using Ubuntu Server 12.04 with libgearman6 (v 0.27) and PHP 5.3.10. The version of the gearman extension is 1.0.2. I think the database is irrelevant here, as I will not be using it in either of the workers. And I think we’re not using persistent queues right now.

Answer 1

Since gearman won’t keep any task information in memory after a task has finished (just report it back for a synchronous task), you won’t be able to retrieve it in your web application without storing it in a 3rd party location. We usually use a simple web service in the application for this, letting the worker call back to the application when a task has completed or an error has occured. This allows us to keep the business logic about what we’d like to do when such an error happens in the application where it belongs, and let our workers be more general (we might need image resizing in many apps, but some apps might want to start several sub tasks that depend on the image resizing being done first).

As you write, you may also let the worker write directly to the database with the state of the task or to memcached, but I’ve found that letting the application itself handle the logic instead of having to change and special case the workers work better. It’s also well suited for a worker framework letting you keep the same standardized way of handling callback across actual worker code.