Question

I’m working on a web application that submits tasks to a master/worker system that farms out the tasks to any of a series of worker instances. The work queue master runs as a separate process (on a separate machine altogether) and tasks are submitted to the master via HTTP/REST requests. Once tasks are submitted to the work queue, client applications can submit another HTTP request to get status information about tasks.

For my web application, I’d like it to provide some sort of progress bar view that gives the user some indication of how far along task processing has come. The obvious way to implement this would be an AJAX progress meter widget that periodically polls the work queue for status on the tasks that have been submitted. My question is, is there a better way to accomplish this without the frequent polling?

I’ve considered having the client web application open up a server socket on which it could listen for notifications from the work master. Another similar thought I’ve had is to use XMPP or a similar protocol for the status notifications. (Of course, the master/worker system would need to be updated to provide notifications either way but I own the code for that so can make any necessary updates myself.)

Any thoughts on the best way to set up a notification system like this? Is the extra effort involved worth it, or is the simple polling solution the way to go?

Answer 1

I guess it depends on a few factors

• How accurate the feedback can be (1 percent, 5 percent, 50 percent)
  Accurate feedback makes it worth pursuing some kind of progress bar and comet style push. If you can only say “Busy… hold on… almost there… done” then a simple ajax “are we there yet” poll is certainly easier to code.
• How timely the Done message has to be seen by the client
• How long each task takes (1 second, 10 seconds, 10 minutes)
  1 second makes it a bit moot. 10 seconds makes it worth it. 10 minutes means you’re better off suggesting the user goes for a coffee break 🙂
• How many concurrent requests there will be
  Unless you’ve got a “special” server, live push style systems tend to eat connections and you’ll be maxed out pretty quickly. Having to throw more webservers in for a fancy progress bar might hurt the budget.

I’ve got some sample code on 871184 that shows a hand rolled “forever frame” which seems to work out well. The project I developed that for isn’t hammered all that hard though, the operations take a few seconds and we can give pretty accurate percent. The code uses asp.net and jquery, but the general techniques will work with any server and javascript framework.

edit As John points out, status reporting probably isn’t the job of the RESTful service. But there’s nothing that says you can’t open an iframe on the client that hooks to a page on the server that polls the service. Theory says the server and the service will at least be closer to one another 🙂