Users on a website I’m building can request the availability of user names on various social networks by typing in a box and pressing return (see this website for an example). When the user submits a name for checking, I have to request availabilities from many different third-party services simultaneously. Each availability check requires a HTTP request. That means that one request from the user can trigger many HTTP requests on the backend.
Now I would like to get the results back to the user as fast as possible. Thus, I want to perform each of these backend availability checks separately and return the results as fast as I get them from the third-party. I would also like to use background worker processes to keep the load of making all these HTTP requests off my server.
Is this a viable usage of background workers or should they be only used for cases where the user is not waiting for results immediately (such as sending emails)?
Is this the best way to architect this application?
To me this looks like a perfect use case for combination of WebSockets and a reactor framework like EventMachine or node.js.
Why WebSockets?
For the request part, it does not matter that much. However the various external services will respond with various latencies, which means that in order to present the user with results as soon as possible you would probably have to launch a long-polling request for every one of those services (which typically would block the processes that handle those requests), or use a sequence of long-polling requests to get the responses as they come in. Every HTTP request has some amount of overhead required to establish the connection and you wold be transmitting a lot more data in HTTP headers than in the response itself.
On the other hand, WebSocket connection is established once, and from then on it acts as a sort of a bidirectional pipe that can be used to transmit messages. Those messages could be the responses of various services that will be streamed to the client as soon as they arrive. This saves you a lot of overhead and gets the responses to the user as fast as possible.
Why a reactor framework?
If you use a background job to handle the responses, chances are, that you will have less worker processes than the number of requests to be made. This means that some of the requests will have to wait until the workers are ready, so the user will get the response later than he could get it if all the requests were done in parallel. Asynchronous I/O makes it possible to issue all those requests in parallel and return the results to the user as they come in.
If you used a background job queue, it would also require that you save the results in some sort of data store, so that your web server can poll it to know when a particular request has finished. That server-side polling also increases the latency with wich user receives the data.
Summing up: using a reactor framework + WebSockets will not only make the user experience better, it will also be simpler to implement. Check out the socket.io library for node.js: it should make it possible to implement your use case in a dozen of lines of code or so.