Question

Is it possible to use the same redis database for multiple projects using celery? Like using the same database for multiple projects as a cache using a key prefix. Or do i have to use a seperate database for every installation?

Answer 1

I’ve used a redis backend for celery while also using the same redis db with prefixed cache data. I was doing this during development, I only used redis for the result backend not to queue tasks, and the production deployment ended up being all AMQP (redis only for caching). I didn’t have any problems and don’t see why one would (other than performance issues).

For running multiple celery projects with different task definitions, I think the issue would be if you have two different types of workers that each can only handle a subset of job types. Without separate databases, I’m not sure how the workers would be able to tell which jobs they could process.

I’d probably either want to make sure all workers had all task types defined and could process anything, or would want to keep the separate projects in separate databases. This wouldn’t require installing anything extra, you’d just specify a REDIS_DB=1 in one of your celery projects. There might be another way to do this. I don’t know for sure that multiple DBs are required, but it kinda makes sense.

If you’re only using redis for a result backend, maybe that would work for having multiple celery projects on one redis db… I’m not really sure.