I have a high-throughput WSGI app that receives many POSTs per second from a remote server and writes documents to a couchdb server. Currently, the writes to couchdb can only happen between request and response. That means
- if writes to couchdb are slow, then the client must sit waiting for a response while we write to the database, and
- if there are problems writing to the couchdb server, there is no way to wait a few minutes and retry
Are there any existing solutions to queue up the writes to couch in the background (I’m looking at something like celery, for example), or will I need to roll my own solution?
celerycould do that, yes, or any other task queue of a similar nature.(Alternatively you could go one step lower and use a any message queue server, plus your own independent worker process that consumes from the message queue.)