The tech stack looks like this:
- Django frontend and backend
- Celery worker queue for asynchronously processing time consuming tasks
Within the past day or two I’ve noticed a lot of these kinds of stack traces both locally and in production:
[2012-07-05 20:31:01,583: CRITICAL/MainProcess] Task site_endpoint.tasks.async_inbound_message[a950736c-ff93-420c-9fbf-6deb2b88ff4d] INTERNAL ERROR: TypeError('__init__() takes at least 3 arguments (1 given)',)
Traceback (most recent call last):
File "/projects/site/venv/lib/python2.7/site-packages/celery/execute/trace.py", line 192, in trace_task
R = I.handle_error_state(task, eager=eager)
File "/projects/site/venv/lib/python2.7/site-packages/celery/execute/trace.py", line 91, in handle_error_state
}[self.state](task, store_errors=store_errors)
File "/projects/site/venv/lib/python2.7/site-packages/celery/execute/trace.py", line 114, in handle_failure
task.backend.mark_as_failure(req.id, exc, self.strtb)
File "/projects/site/venv/lib/python2.7/site-packages/celery/backends/base.py", line 96, in mark_as_failure
traceback=traceback)
File "/projects/site/venv/lib/python2.7/site-packages/celery/backends/base.py", line 229, in store_result
return self._store_result(task_id, result, status, traceback, **kwargs)
File "/projects/site/venv/lib/python2.7/site-packages/djcelery/backends/database.py", line 26, in _store_result
traceback=traceback)
File "/projects/site/venv/lib/python2.7/site-packages/djcelery/managers.py", line 40, in _inner
return fun(*args, **kwargs)
File "/projects/site/venv/lib/python2.7/site-packages/djcelery/managers.py", line 164, in store_result
"traceback": traceback})
File "/projects/site/venv/lib/python2.7/site-packages/djcelery/managers.py", line 82, in update_or_create
return self.get_query_set().update_or_create(**kwargs)
File "/projects/site/venv/lib/python2.7/site-packages/djcelery/managers.py", line 66, in update_or_create
obj, created = self.get_or_create(**kwargs)
File "/projects/site/venv/lib/python2.7/site-packages/django/db/models/query.py", line 385, in get_or_create
obj.save(force_insert=True, using=self.db)
File "/projects/site/venv/lib/python2.7/site-packages/django/db/models/base.py", line 460, in save
self.save_base(using=using, force_insert=force_insert, force_update=force_update)
File "/projects/site/venv/lib/python2.7/site-packages/django/db/models/base.py", line 543, in save_base
for f in meta.local_fields if not isinstance(f, AutoField)]
File "/projects/site/venv/lib/python2.7/site-packages/django/db/models/fields/subclassing.py", line 28, in inner
return func(*args, **kwargs)
File "/projects/site/venv/lib/python2.7/site-packages/django/db/models/fields/subclassing.py", line 28, in inner
return func(*args, **kwargs)
File "/projects/site/venv/lib/python2.7/site-packages/django/db/models/fields/__init__.py", line 276, in get_db_prep_save
return self.get_db_prep_value(value, connection=connection, prepared=False)
File "/projects/site/venv/lib/python2.7/site-packages/django/db/models/fields/subclassing.py", line 53, in inner
return func(*args, **kwargs)
File "/projects/site/venv/lib/python2.7/site-packages/picklefield/fields.py", line 154, in get_db_prep_value
value = force_unicode(dbsafe_encode(value, self.compress, self.protocol))
File "/projects/site/venv/lib/python2.7/site-packages/picklefield/fields.py", line 57, in dbsafe_encode
value = b64encode(dumps(deepcopy(value), pickle_protocol))
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 190, in deepcopy
y = _reconstruct(x, rv, 1, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 328, in _reconstruct
args = deepcopy(args, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 163, in deepcopy
y = copier(x, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 237, in _deepcopy_tuple
y.append(deepcopy(a, memo))
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 190, in deepcopy
y = _reconstruct(x, rv, 1, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 329, in _reconstruct
y = callable(*args)
TypeError: __init__() takes at least 3 arguments (1 given)
Looking at the database calls when this happens, I can see the following queries being executed:
LOG: statement: SELECT "celery_taskmeta"."id", "celery_taskmeta"."task_id", "celery_taskmeta"."status", "celery_taskmeta"."result", "celery_taskmeta"."date_done", "celery_taskmeta"."traceback" FROM "celery_taskmeta" WHERE "celery_taskmeta"."task_id" = 'a950736c-ff93-420c-9fbf-6deb2b88ff4d'
LOG: statement: SAVEPOINT s140735259576672_x4
LOG: statement: ROLLBACK
LOG: statement: BEGIN
LOG: statement: SELECT "celery_taskmeta"."id", "celery_taskmeta"."task_id", "celery_taskmeta"."status", "celery_taskmeta"."result", "celery_taskmeta"."date_done", "celery_taskmeta"."traceback" FROM "celery_taskmeta" WHERE "celery_taskmeta"."task_id" = 'a950736c-ff93-420c-9fbf-6deb2b88ff4d'
LOG: statement: SAVEPOINT s140735259576672_x5
LOG: statement: ROLLBACK
LOG: statement: BEGIN
LOG: statement: SELECT "celery_taskmeta"."id", "celery_taskmeta"."task_id", "celery_taskmeta"."status", "celery_taskmeta"."result", "celery_taskmeta"."date_done", "celery_taskmeta"."traceback" FROM "celery_taskmeta" WHERE "celery_taskmeta"."task_id" = 'a950736c-ff93-420c-9fbf-6deb2b88ff4d'
LOG: statement: SAVEPOINT s140735259576672_x6
I am having a tough time understanding what the source of this invalid object initialization is. Anyone have ideas?
It looks like you are using new-relic remote performance monitoring/forensic analysis. I have seen this kind of problem before when old versions of the library are being used. I recommend checking to make sure you are using the latest version of their client library.