Question

I have a Python/Flask webapp with MongoDB database on DotCloud hosting.
I would like to set up Celery on the Python service using the MongoDB service as the Celery queue.

Celery starts and it finds my celeryconfig.py file, which contains:

BROKER_BACKEND = "mongodb"
BROKER_HOST = "data.myapp.dotcloud.com"
BROKER_PORT = 8567
BROKER_USER = 'myuser'
BROKER_PASSWORD = 'mypass'
CELERY_IMPORTS = ('myapp.worker', )
CELERY_IGNORE_RESULT = True

Now I have a problem with DB authentification.

Here what I did to create the user:

~# mongo
MongoDB shell version: 1.8.1
connecting to: test
-> use celery
switched to db celery
-> db.getSisterDB("admin").auth("root", "root_password_for_mongodb_service");
1
-> db.addUser("myuser", "mypass");

Here are the Celery logs (on the python service):

-------------- celery@myapp-www v2.3.1
---- **** -----
--- * ***  * -- [Configuration]
-- * - **** ---   . broker:      mongodb://[e-mail]:8567/
- ** ----------   . loader:      celery.loaders.default.Loader
- ** ----------   . logfile:     [stderr]@WARNING
- ** ----------   . concurrency: 4
- ** ----------   . events:      OFF
- *** --- * ---   . beat:        OFF
-- ******* ----
--- ***** ----- [Queues]
 --------------   . celery:      exchange:celery (direct) binding:celery
[2011-08-26 14:18:10,453: WARNING/MainProcess] celery@myapp-www has started.
[2011-08-26 14:18:12,873: WARNING/MainProcess] Traceback (most recent call last):
[2011-08-26 14:18:12,874: WARNING/MainProcess] File "/home/dotcloud/env/bin/celeryd", line 8, in <module>
[2011-08-26 14:18:12,874: WARNING/MainProcess] load_entry_point('celery==2.3.1', 'console_scripts', 'celeryd')()
[2011-08-26 14:18:12,875: WARNING/MainProcess] File "/home/dotcloud/env/lib/python2.6/site-packages/celery/bin/celeryd.py", line 187, in main
[2011-08-26 14:18:12,875: WARNING/MainProcess] worker.execute_from_commandline()
[2011-08-26 14:18:12,876: WARNING/MainProcess] File "/home/dotcloud/env/lib/python2.6/site-packages/celery/bin/base.py", line 72, in execute_from_commandline
[2011-08-26 14:18:12,877: WARNING/MainProcess] return self.handle_argv(prog_name, argv[1:])
[2011-08-26 14:18:12,877: WARNING/MainProcess] File "/home/dotcloud/env/lib/python2.6/site-packages/celery/bin/base.py", line 100, in handle_argv
[2011-08-26 14:18:12,878: WARNING/MainProcess] return self.run(*args, **vars(options))
[2011-08-26 14:18:12,878: WARNING/MainProcess] File "/home/dotcloud/env/lib/python2.6/site-packages/celery/bin/celeryd.py", line 96, in run
[2011-08-26 14:18:12,878: WARNING/MainProcess] return self.app.Worker(**kwargs).run()
[2011-08-26 14:18:12,879: WARNING/MainProcess] File "/home/dotcloud/env/lib/python2.6/site-packages/celery/apps/worker.py", line 140, in run
[2011-08-26 14:18:12,879: WARNING/MainProcess] self.run_worker()
[2011-08-26 14:18:12,880: WARNING/MainProcess] File "/home/dotcloud/env/lib/python2.6/site-packages/celery/apps/worker.py", line 240, in run_worker
[2011-08-26 14:18:12,880: WARNING/MainProcess] worker.start()
[2011-08-26 14:18:12,880: WARNING/MainProcess] File "/home/dotcloud/env/lib/python2.6/site-packages/celery/worker/__init__.py", line 252, in start
[2011-08-26 14:18:12,881: WARNING/MainProcess] blocking(component.start)
[2011-08-26 14:18:12,881: WARNING/MainProcess] File "/home/dotcloud/env/lib/python2.6/site-packages/kombu/syn.py", line 14, in blocking
[2011-08-26 14:18:12,882: WARNING/MainProcess] return __sync_current(fun, *args, **kwargs)
[2011-08-26 14:18:12,882: WARNING/MainProcess] File "/home/dotcloud/env/lib/python2.6/site-packages/kombu/syn.py", line 30, in __blocking__
[2011-08-26 14:18:12,882: WARNING/MainProcess] return fun(*args, **kwargs)
[2011-08-26 14:18:12,883: WARNING/MainProcess] File "/home/dotcloud/env/lib/python2.6/site-packages/celery/worker/consumer.py", line 302, in start
[2011-08-26 14:18:12,883: WARNING/MainProcess] self.reset_connection()
[2011-08-26 14:18:12,883: WARNING/MainProcess] File "/home/dotcloud/env/lib/python2.6/site-packages/celery/worker/consumer.py", line 555, in reset_connection
[2011-08-26 14:18:12,884: WARNING/MainProcess] on_decode_error=self.on_decode_error)
[2011-08-26 14:18:12,884: WARNING/MainProcess] File "/home/dotcloud/env/lib/python2.6/site-packages/celery/app/amqp.py", line 331, in get_task_consumer
[2011-08-26 14:18:12,885: WARNING/MainProcess] **kwargs)
[2011-08-26 14:18:12,885: WARNING/MainProcess] File "/home/dotcloud/env/lib/python2.6/site-packages/kombu/compat.py", line 228, in __init__
[2011-08-26 14:18:12,885: WARNING/MainProcess] super(ConsumerSet, self).__init__(self.backend, queues, **kwargs)
[2011-08-26 14:18:12,886: WARNING/MainProcess] File "/home/dotcloud/env/lib/python2.6/site-packages/kombu/messaging.py", line 242, in __init__
[2011-08-26 14:18:12,886: WARNING/MainProcess] self.declare()
[2011-08-26 14:18:12,887: WARNING/MainProcess] File "/home/dotcloud/env/lib/python2.6/site-packages/kombu/messaging.py", line 252, in declare
[2011-08-26 14:18:12,887: WARNING/MainProcess] queue.declare()
[2011-08-26 14:18:12,888: WARNING/MainProcess] File "/home/dotcloud/env/lib/python2.6/site-packages/kombu/entity.py", line 364, in declare
[2011-08-26 14:18:12,888: WARNING/MainProcess] self.name and self.queue_declare(nowait, passive=False),
[2011-08-26 14:18:12,888: WARNING/MainProcess] File "/home/dotcloud/env/lib/python2.6/site-packages/kombu/entity.py", line 382, in queue_declare
[2011-08-26 14:18:12,889: WARNING/MainProcess] nowait=nowait)
[2011-08-26 14:18:12,889: WARNING/MainProcess] File "/home/dotcloud/env/lib/python2.6/site-packages/kombu/syn.py", line 14, in blocking
[2011-08-26 14:18:12,889: WARNING/MainProcess] return __sync_current(fun, *args, **kwargs)
[2011-08-26 14:18:12,890: WARNING/MainProcess] File "/home/dotcloud/env/lib/python2.6/site-packages/kombu/syn.py", line 30, in __blocking__
[2011-08-26 14:18:12,890: WARNING/MainProcess] return fun(*args, **kwargs)
[2011-08-26 14:18:12,890: WARNING/MainProcess] File "/home/dotcloud/env/lib/python2.6/site-packages/kombu/transport/virtual/__init__.py", line 357, in queue_declare
[2011-08-26 14:18:12,891: WARNING/MainProcess] return queue, self._size(queue), 0
[2011-08-26 14:18:12,891: WARNING/MainProcess] File "/home/dotcloud/env/lib/python2.6/site-packages/kombu/transport/mongodb.py", line 45, in _size
[2011-08-26 14:18:12,892: WARNING/MainProcess] return self.client.count()
[2011-08-26 14:18:12,892: WARNING/MainProcess] File "/home/dotcloud/env/lib/python2.6/site-packages/kombu/transport/mongodb.py", line 81, in client
[2011-08-26 14:18:12,893: WARNING/MainProcess] self._client = self._open()
[2011-08-26 14:18:12,893: WARNING/MainProcess] File "/home/dotcloud/env/lib/python2.6/site-packages/kombu/transport/mongodb.py", line 75, in _open
[2011-08-26 14:18:12,893: WARNING/MainProcess] col.ensure_index([("queue", 1)])
[2011-08-26 14:18:12,894: WARNING/MainProcess] File "/home/dotcloud/env/lib/python2.6/site-packages/pymongo/collection.py", line 724, in ensure_index
[2011-08-26 14:18:12,894: WARNING/MainProcess] ttl, **kwargs)
[2011-08-26 14:18:12,895: WARNING/MainProcess] File "/home/dotcloud/env/lib/python2.6/site-packages/pymongo/collection.py", line 646, in create_index
[2011-08-26 14:18:12,895: WARNING/MainProcess] safe=True)
[2011-08-26 14:18:12,895: WARNING/MainProcess] File "/home/dotcloud/env/lib/python2.6/site-packages/pymongo/collection.py", line 275, in insert
[2011-08-26 14:18:12,896: WARNING/MainProcess] check_keys, safe, kwargs), safe)
[2011-08-26 14:18:12,897: WARNING/MainProcess] File "/home/dotcloud/env/lib/python2.6/site-packages/pymongo/connection.py", line 770, in _send_message
[2011-08-26 14:18:12,897: WARNING/MainProcess] return self.__check_response_to_last_error(response)
[2011-08-26 14:18:12,898: WARNING/MainProcess] File "/home/dotcloud/env/lib/python2.6/site-packages/pymongo/connection.py", line 722, in __check_response_to_last_error
[2011-08-26 14:18:12,898: WARNING/MainProcess] raise OperationFailure(error["err"])
[2011-08-26 14:18:12,899: WARNING/MainProcess] pymongo.errors
[2011-08-26 14:18:12,899: WARNING/MainProcess] .
[2011-08-26 14:18:12,899: WARNING/MainProcess] OperationFailure
[2011-08-26 14:18:12,900: WARNING/MainProcess] :
[2011-08-26 14:18:12,900: WARNING/MainProcess] unauthorized

Here are the MongoDB logs (dotcloud logs myapp.data):

Fri Aug 26 14:19:43 [initandlisten] connection accepted from 10.68.47.216:43499 #6053
Fri Aug 26 14:19:43 [initandlisten] connection accepted from 10.68.47.216:43500 #6054
Fri Aug 26 14:19:43 [conn6054] auth: couldn't find user myuser, kombu_default.system.users
Fri Aug 26 14:19:43 [conn6054] query kombu_default.$cmd ntoreturn:1 command: { authenticate: 1, nonce: "d7feebcf0bdbb839", user: "myuser", key: "ad0b6da89f5bae34c70b34e2dac5b678" } reslen:76 256ms
Fri Aug 26 14:19:45 [conn6053] end connection 10.68.47.216:43499
Fri Aug 26 14:19:45 [conn6054] end connection 10.68.47.216:43500
Fri Aug 26 14:19:47 [initandlisten] connection accepted from 10.68.47.216:43527 #6055
Fri Aug 26 14:19:47 [initandlisten] connection accepted from 10.68.47.216:43530 #6056
Fri Aug 26 14:19:47 [conn6056] auth: couldn't find user myuser, kombu_default.system.users
Fri Aug 26 14:19:49 [conn6055] end connection 10.68.47.216:43527
Fri Aug 26 14:19:49 [conn6056] end connection 10.68.47.216:43530

According to you, where am I wrong?

Regards,

MV

Answer 1

Finally I learnt that “kombu_default” (in the MongoDB logs) was the database used for message queue by Celery.

So, to get Celery + MongoDB working, I only add the parameter BROKER_VHOST = "celery" to the celeryconfig.py file.