I am seeing a “release unlocked lock” error appear occasionally in the logs for my app on Google App Engine (gqueues) and cannot figure out why (complete stack trace below). Does anyone know why this error would occur?
Interestingly the app is on python 2.7 with threadsafe=NO, so it seems weird that there would be any locking going on at all, or locking errors.
The backstory: GQueues was on python2.5, and at the beginning of April I migrated it to 2.7 and set threadsafe=True. Everything worked fine for a week. Then on the morning of April 9 the entire app was inaccessible, spitting out 500 errors. It kept spawning new server instances that would have latencies of over 30 seconds. I noticed in my logs I saw the error “Cannot unlock an unlocked lock”. I did not add any locking mechanisms to my code during the migration, so I assumed this was app engine’s locking that the error was referring to. Anyway, I rolled back my app to the previous version that was still on Python 2.5 and everything started working again.
Later in April AppEngine Downtime Notify revealed that there was an issue on their end on April 9 that was causing problems with python 2.7 apps.
https://groups.google.com/d/topic/google-appengine-downtime-notify/QL8TmRn6Ay4/discussion
I followed up with Christina Ilvento from Google and she thought it should be fine to move back to Python 2.7 now and turn threadsafe=True. I migrated back to 2.7 but left threadsafe=False, thinking that I would give it a couple weeks to make sure everything worked well on 2.7 before turning on threading again (which I assumed was causing the problem). Everything worked fine for a few weeks, but now I’m seeing this “release unlocked lock” error even with threading disabled. Does anyone know what’s going on here?
Christina suggested I post here and the App Engine team would follow up.
release unlocked lock
Traceback (most recent call last):
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 545, in dispatch
return method(*args, **kwargs)
File "/base/data/home/apps/s~gqueues-hrd/gq3-7-5.358902352474055782/controllers/util.py", line 69, in check_login
handler_method(self, *args)
File "/base/data/home/apps/s~gqueues-hrd/gq3-7-5.358902352474055782/controllers/main.py", line 297, in get
self.generate('main.html', template_values)
File "/base/data/home/apps/s~gqueues-hrd/gq3-7-5.358902352474055782/controllers/baserequest.py", line 257, in generate
self.response.out.write(template.render(path, values))
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/ext/webapp/template.py", line 92, in render
return t.render(Context(template_dict))
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/ext/webapp/template.py", line 172, in wrap_render
return orig_render(context)
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/_internal/django/template/__init__.py", line 173, in render
return self._render(context)
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/_internal/django/template/__init__.py", line 167, in _render
return self.nodelist.render(context)
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/_internal/django/template/__init__.py", line 794, in render
bits.append(self.render_node(node, context))
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/_internal/django/template/__init__.py", line 807, in render_node
return node.render(context)
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/_internal/django/template/loader_tags.py", line 139, in render
return self.template.render(context)
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/_internal/django/template/__init__.py", line 173, in render
return self._render(context)
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/_internal/django/template/__init__.py", line 167, in _render
return self.nodelist.render(context)
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/_internal/django/template/__init__.py", line 794, in render
bits.append(self.render_node(node, context))
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/_internal/django/template/__init__.py", line 807, in render_node
return node.render(context)
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/_internal/django/template/defaulttags.py", line 173, in render
nodelist.append(node.render(context))
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/_internal/django/template/defaulttags.py", line 257, in render
return self.nodelist_true.render(context)
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/_internal/django/template/__init__.py", line 794, in render
bits.append(self.render_node(node, context))
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/_internal/django/template/__init__.py", line 807, in render_node
return node.render(context)
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/_internal/django/template/loader_tags.py", line 139, in render
return self.template.render(context)
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/_internal/django/template/__init__.py", line 173, in render
return self._render(context)
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/_internal/django/template/__init__.py", line 167, in _render
return self.nodelist.render(context)
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/_internal/django/template/__init__.py", line 794, in render
bits.append(self.render_node(node, context))
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/_internal/django/template/__init__.py", line 807, in render_node
return node.render(context)
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/_internal/django/template/loader_tags.py", line 139, in render
return self.template.render(context)
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/_internal/django/template/__init__.py", line 173, in render
return self._render(context)
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/_internal/django/template/__init__.py", line 167, in _render
return self.nodelist.render(context)
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/_internal/django/template/__init__.py", line 794, in render
bits.append(self.render_node(node, context))
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/_internal/django/template/__init__.py", line 807, in render_node
return node.render(context)
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/_internal/django/template/defaulttags.py", line 173, in render
nodelist.append(node.render(context))
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/_internal/django/template/loader_tags.py", line 139, in render
return self.template.render(context)
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/_internal/django/template/__init__.py", line 173, in render
return self._render(context)
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/_internal/django/template/__init__.py", line 167, in _render
return self.nodelist.render(context)
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/_internal/django/template/__init__.py", line 794, in render
bits.append(self.render_node(node, context))
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/_internal/django/template/__init__.py", line 807, in render_node
return node.render(context)
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/_internal/django/template/loader_tags.py", line 139, in render
return self.template.render(context)
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/_internal/django/template/__init__.py", line 173, in render
return self._render(context)
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/_internal/django/template/__init__.py", line 167, in _render
return self.nodelist.render(context)
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/_internal/django/template/__init__.py", line 794, in render
bits.append(self.render_node(node, context))
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/_internal/django/template/__init__.py", line 807, in render_node
return node.render(context)
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/_internal/django/template/defaulttags.py", line 136, in render
values = list(values)
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/ext/db/__init__.py", line 2312, in next
return self.__model_class.from_entity(self.__iterator.next())
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/datastore/datastore_query.py", line 2809, in next
next_batch = self.__batcher.next()
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/datastore/datastore_query.py", line 2671, in next
return self.next_batch(self.AT_LEAST_ONE)
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/datastore/datastore_query.py", line 2708, in next_batch
batch = self.__next_batch.get_result()
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 592, in get_result
return self.__get_result_hook(self)
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/datastore/datastore_query.py", line 2450, in __query_result_hook
self._batch_shared.conn.check_rpc_success(rpc)
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/datastore/datastore_rpc.py",
UPDATE:
Here is a complete backtrace for one of the errors
46.231.181.199 - sandra.martin [15/May/2012:03:28:44 -0700] "POST /items/ HTTP/1.1" 500 1306 "http://www.gqueues.com/main?q=ag1zfmdxdWV1ZXMtaHJkciILEgdBY2NvdW50GL-psgYMCxILU2hhcmVkUXVldWUYkU4M";;;; "Mozilla/5.0 (Windows NT 6.1; rv:12.0) Gecko/20100101 Firefox/12.0" "www.gqueues.com" ms=59858 cpu_ms=154948 api_cpu_ms=112433 cpm_usd=4.305925 instance=00c61b117cfbf1818fa4b0f779eacc7f4a9030
D 2012-05-15 05:27:45.121
postAction: reorder
E 2012-05-15 05:28:44.714
release unlocked lock
Traceback (most recent call last):
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 545, in dispatch
return method(*args, **kwargs)
File "/base/data/home/apps/s~gqueues-hrd/gq3-7-5.358902352474055782/controllers/util.py", line 139, in check_login
handler_method(self, *args)
File "/base/data/home/apps/s~gqueues-hrd/gq3-7-5.358902352474055782/controllers/items.py", line 371, in post
self.reorder()
File "/base/data/home/apps/s~gqueues-hrd/gq3-7-5.358902352474055782/controllers/items.py", line 856, in reorder
QueueSyncHandler.syncQueueViewers(clientId, itemQueue)
File "/base/data/home/apps/s~gqueues-hrd/gq3-7-5.358902352474055782/controllers/queuesync.py", line 197, in syncQueueViewers
jsonObj['items'] = util.getJSONItems(queueItems, filter, queueViewer.type, accountObj)
File "/base/data/home/apps/s~gqueues-hrd/gq3-7-5.358902352474055782/controllers/util.py", line 1536, in getJSONItems
jsonItem['subitems'] = getJSONItems(subItems, filter, type, accountObj, assignmentCompleted=assignmentCompleted, accountEmail=accountEmail)
File "/base/data/home/apps/s~gqueues-hrd/gq3-7-5.358902352474055782/controllers/util.py", line 1536, in getJSONItems
jsonItem['subitems'] = getJSONItems(subItems, filter, type, accountObj, assignmentCompleted=assignmentCompleted, accountEmail=accountEmail)
File "/base/data/home/apps/s~gqueues-hrd/gq3-7-5.358902352474055782/controllers/util.py", line 1536, in getJSONItems
jsonItem['subitems'] = getJSONItems(subItems, filter, type, accountObj, assignmentCompleted=assignmentCompleted, accountEmail=accountEmail)
File "/base/data/home/apps/s~gqueues-hrd/gq3-7-5.358902352474055782/controllers/util.py", line 1481, in getJSONItems
if item.hasAssignments():
File "/base/data/home/apps/s~gqueues-hrd/gq3-7-5.358902352474055782/models.py", line 1512, in hasAssignments
itemAssignments = Assignment.all(keys_only=True).filter('item =', self).fetch(1)
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/ext/db/__init__.py", line 2143, in fetch
return list(self.run(limit=limit, offset=offset, **kwargs))
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/datastore/datastore_query.py", line 2809, in next
next_batch = self.__batcher.next()
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/datastore/datastore_query.py", line 2671, in next
return self.next_batch(self.AT_LEAST_ONE)
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/datastore/datastore_query.py", line 2708, in next_batch
batch = self.__next_batch.get_result()
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 592, in get_result
return self.__get_result_hook(self)
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/datastore/datastore_query.py", line 2450, in __query_result_hook
self._batch_shared.conn.check_rpc_success(rpc)
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/datastore/datastore_rpc.py", line 1206, in check_rpc_success
rpc.wait()
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 533, in wait
self.__rpc.Wait()
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/api/apiproxy_rpc.py", line 119, in Wait
rpc_completed = self._WaitImpl()
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/runtime/apiproxy.py", line 134, in _WaitImpl
self.__done_event.wait()
File "/base/python27_runtime/python27_dist/lib/python2.7/threading.py", line 407, in wait
self.__cond.release()
error: release unlocked lock
Thanks for the report. We (the GAE python team) are aware of this bug. What’s probably happening is that your handler is probably hitting the request deadline and running into a problem we warn about at https://developers.google.com/appengine/docs/python/runtime#The_Request_Timer.
Can you check a couple of things for me:
1. The exception trace appears to be incomplete. If possible, can you post a full backtrace.
2. Can you check the request times for your requests that are hitting this (in the logs) and post them here.
EDIT: The warning on the page I linked to was posted recently.