Question

I have some spider that download pages and store data in database. I have created flask application with admin panel (by Flask-Admin extension) that show database.
Now I want append function to my flask app for control spider state: switch on/off.

I thing it posible by threads or multiprocessing. Celery is not good decision because total program must use minimum memory.

Which method to choose for implementation this function?

Answer 1

Discounting Celery based on memory usage would probably be a mistake, as Celery has low overhead in both time and space. In fact, using Celery+Flask does not use much more memory than using Flask alone.

In addition Celery comes with several choices you can make that can have an impact
on the amount of memory used. For example, there are 5 different pool implementations that all have different strengths and trade-offs, the pool choices are:

• multiprocessing

By default Celery uses multiprocessing, which means that it will spawn child processes
to offload work to. This is the most memory expensive option – simply because
every child process will duplicate the amount of base memory needed.

But Celery also comes with an autoscale feature that will kill off worker
processes when there’s little work to do, and spawn new processes when there’s more work:

$ celeryd --autoscale=0,10

where 0 is the mininum number of processes, and 10 is the maximum. Here celeryd will
start off with no child processes, and grow based on load up to a maximum of 10 processes. When load decreases, so will the number of worker processes.

• eventlet/gevent

When using the eventlet/gevent pools only a single process will be used, and thus it will
use a lot less memory, but with the downside that tasks calling blocking code will
block other tasks from executing. If your tasks are mostly I/O bound you should be ok,
and you can also combine different pools and send problem tasks to a multiprocessing pool instead.

• threads

Celery also comes with a pool using threads.

The development version that will become version 2.6 includes a lot of optimizations,
and there is no longer any need for the Flask-Celery extension module. If you are not going
into production in the next days then I would encourage you to try the development version
which must be installed like this:

$ pip install https://github.com/ask/kombu/zipball/master
$ pip install https://github.com/ask/celery/zipball/master

The new API is now also Flask inspired, so you should read the new getting started guide:

http://ask.github.com/celery/getting-started/first-steps-with-celery.html

With all this said, most optimization work has been focused on execution speed so far,
and there is probably many more memory optimizations that can be made. It has not been a request so far, but in the unlikely event that Celery does not match your memory constraints, you can open up an issue at our bug tracker and I’m sure it will get focus, or you can even help us to do so.