Question

Working on getting Celery setup (following the basic tutorial) with a mongodb broker as backend. Following the configuration guidelines set out in the official docs, my celeryconfig.py is setup as follows:

CELERY_RESULT_BACKEND = "mongodb"

BROKER_BACKEND = "mongodb"
BROKER_URL = "mongodb://user:[e-mail]:123456/testdb"

CELERY_MONGODB_BACKEND_SETTINGS = {
    "host":"subdomain.mongolab.com",
    "port":123456,
    "database":"testdb",
    "taskmeta_collection":"taskmeta",
    "user":"user",
    "pass":"pass",
}
CELERY_IMPORTS = ("tasks",)

Running the celeryd with --loglevel=INFO returns the following exception, originating in pymongo but bubbling through both kombu and celery.

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/celery/worker/__init__.py", line 230, in start
    component.start()
  File "/usr/local/lib/python2.7/dist-packages/celery/worker/consumer.py", line 338, in start
    self.reset_connection()
  File "/usr/local/lib/python2.7/dist-packages/celery/worker/consumer.py", line 596, in reset_connection
    on_decode_error=self.on_decode_error)
  File "/usr/local/lib/python2.7/dist-packages/celery/app/amqp.py", line 335, in get_task_consumer
    **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/kombu/compat.py", line 187, in __init__
    super(ConsumerSet, self).__init__(self.backend, queues, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/kombu/messaging.py", line 285, in __init__
    self.declare()
  File "/usr/local/lib/python2.7/dist-packages/kombu/messaging.py", line 295, in declare
    queue.declare()
  File "/usr/local/lib/python2.7/dist-packages/kombu/entity.py", line 388, in declare
    self.queue_declare(nowait, passive=False)
  File "/usr/local/lib/python2.7/dist-packages/kombu/entity.py", line 408, in queue_declare
    nowait=nowait)
  File "/usr/local/lib/python2.7/dist-packages/kombu/transport/virtual/__init__.py", line 380, in queue_declare
    return queue, self._size(queue), 0
  File "/usr/local/lib/python2.7/dist-packages/kombu/transport/mongodb.py", line 74, in _size
    return self.client.messages.find({"queue": queue}).count()
  File "/usr/local/lib/python2.7/dist-packages/kombu/transport/mongodb.py", line 171, in client
    self._client = self._open()
  File "/usr/local/lib/python2.7/dist-packages/kombu/transport/mongodb.py", line 97, in _open
    mongoconn = Connection(host=conninfo.hostname, port=conninfo.port)
  File "/usr/local/lib/python2.7/dist-packages/pymongo/connection.py", line 325, in __init__
    nodes.update(uri_parser.split_hosts(entity, port))
  File "/usr/local/lib/python2.7/dist-packages/pymongo/uri_parser.py", line 198, in split_hosts
    nodes.append(parse_host(entity, default_port))
  File "/usr/local/lib/python2.7/dist-packages/pymongo/uri_parser.py", line 127, in parse_host
    raise ConfigurationError("Reserved characters such as ':' must be "
ConfigurationError: Reserved characters such as ':' must be escaped according RFC 2396. An IPv6 address literal must be enclosed in '[' and ']' according to RFC 2732.

Something about the way Celery is handling the mongouri is not encoding correctly, since it is the uri parser within pymongo that is throwing this error. I have tried escaping the : characters in the uri string, but all this achieves is resetting the transport back to the default AMQP with a mangled connection string.

amqp://guest@localhost:5672/mongodb\http://user\:[e-mail]\:29217/testdb

Which clearly isn’t right.

I’ve tried entering the uri in the config as a raw string using r and nothing changes.

I know this kind of connection configuration has been supported in Celery since 2.4 (I’m using 2.5.1, pymongo 2.1.1) and the official docs all cite it as the preferred method to connect to a mongodb broker.

Could this be a bug, perhaps an incompatibility with the latest pymongo build? If this approach doesn’t work, how would one attach the task queue to a replica set, since I assume these have to be passed in the mongouri using the ?replicaSet parameter.

I should note that I’d rather not switch to using a RabbitMQ broker, since Mongo is already in the stack for the app in question and it just seems more intuitive to use what’s already there. If there is a concrete reason why Mongo would be less effective for this purpose (the amount of tasks per day would be relatively small) I’d love to know! Thanks in advance.

Answer 1

I think it’s a bug. Celery passed hostname instead of server_uri to kombu, thus cause this problem. After tracing the code, I found the following conf to bypass the bug before they fixed it.

CELERY_RESULT_BACKEND = 'mongodb'
BROKER_HOST = "subdomain.mongolab.com"
BROKER_PORT = 123456
BROKER_TRANSPORT = 'mongodb'
BROKER_VHOST = 'testdb'
CELERY_IMPORTS = ('tasks',)
CELERY_MONGODB_BACKEND_SETTINGS = { 
       'host': 'subdomain.mongolab.com',
       'port': 123456,
       'database': 'testdb',
       'user': user,
       'password': password,
       'taskmeta_collection': 'teskmeta'
       }   

just repeating the configuration.