Currently, we have a custom cms product for schools that is built in django and deployed via heroku. I do a new deployment for each school system we add. (I’d like to switch my system over to a multi-tenant one, but that will be a big project.) I recently added some functionality that works best asynchronously (an emailer & asynchronous cache-refreshes for db-intensive actions). I’ve written my background tasks as celery tasks and they work well. The problem is that to have a celery worker running 24-7 will cost me 34.50/month per app when in reality these tasks will only have a few minutes of non-idle time per day. This cost will make using heroku nearly unfeasible given the way I’m currently using it with a different instance for each client. Has someone dealt with this scenario and can recommend how to be able to have background processing on a regular basis (preferably with celery) without having to run a worker process 24-7. The cache-refresher needs to run every few minutes but the emailer will rarely run (only when they send out an email to subscribers).
Share
If you do not need your workers to engage immediately, you can use the scheduler add-on to regularly clear your task queue.
If you need a faster response time, then look at the Python wrapper for the Heroku API. With this you can use a variety of methods to detect when you need to run a worker, and dynamically spin one up, and spin it back down when it is unneeded. This is a non-trivial engineering task.