Question

A client’s Django application is intermittently (about twice a day) throwing RuntimeError("Unable to create a new session key."):

    Traceback (most recent call last):
  File "/usr/local/lib/python2.6/dist-packages/django/core/handlers/base.py", line 111, in get_response
    response = callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python2.6/dist-packages/django/contrib/admin/views/decorators.py", line 17, in _checklogin
    if request.user.is_active and request.user.is_staff:
  File "/usr/local/lib/python2.6/dist-packages/django/contrib/auth/middleware.py", line 9, in __get__
    request._cached_user = get_user(request)
  File "/usr/local/lib/python2.6/dist-packages/django/contrib/auth/__init__.py", line 107, in get_user
    user_id = request.session[SESSION_KEY]
  File "/usr/local/lib/python2.6/dist-packages/django/contrib/sessions/backends/base.py", line 47, in __getitem__
    return self._session[key]
  File "/usr/local/lib/python2.6/dist-packages/django/contrib/sessions/backends/base.py", line 195, in _get_session
    self._session_cache = self.load()
  File "/usr/local/lib/python2.6/dist-packages/django/contrib/sessions/backends/cache.py", line 16, in load
    self.create()
  File "/usr/local/lib/python2.6/dist-packages/django/contrib/sessions/backends/cache.py", line 33, in create
    raise RuntimeError("Unable to create a new session key.")
RuntimeError: Unable to create a new session key.

As you can see from the traceback, this happens deep in the bowels of django.contrib.sessions when using the cache session backend with the memcached cache backend.

A Django trac ticket (https://code.djangoproject.com/ticket/14093) suggests changing the session key hash from MD5 to UUID4, but that’s no help — the problem is the network. I’ve observed (with tcpdump) that this exception can occur when the TCP connection from app server to memcache server times out due to packet loss.

We have two app servers and one memcached (1.4.2) server, all running in Amazon EC2. During periods of high demand, I’ve observed one app server exchanging 75,000 packets/second with the memcache server. During this period of high demand, I observed one SYN packet for a new memcache connection get lost, resulting in a python-memcache connection timeout (before the kernel even had a chance to retransmit) and a RuntimeError.

I’m at a loss for how to solve this. I’d like to tune Linux’s TCP retransmit timer lower than three seconds, but it’s not tunable. Failing that, I’d like to have python-memcache retry a connection a couple times before giving up, but it won’t. I see that pylibmc has configurable connect and retry behavior, but I haven’t been able to find a combination of options that works around the packet loss.

Ideas?

---

UPDATE:

For people who see "Unable to create a new session key" all the time, that just means your memcache is not set up correctly. Some of the answers below discuss things to check (is the package installed? is the port correct?).

The problem we were having was intermittent — only one in several thousand requests would fail. I used tcpdump to show that this happens when one of the three packets in the TCP three-way handshake is lost (due to network congestion), resulting in python-memcache timing out and raising an exception, resulting in the "Unable to create a new session key" RuntimeError.

Packet loss in cloud provider networks may be unavoidable. Ideally, the Linux kernel would make the TCP initial retransmit timer configurable, but that did not appear to be the case at the time I was investigating this. That means the python-memcache library itself would need to have some kind of timeout-and-retry logic, which it did not have (at the time).

It looks like later versions of Django’s cache backend have added a retry loop, which should avoid these kinds of intermittent failures, at the expense of requests occasionally taking a few seconds longer.

Answer 1

Looking at the python-memcached code on launchpad, you should be able to tweak dead_retry and retry_timeout. Another option may be to run a low memory, low connections instance of memcached on one or both of the app servers as a fallback when the primary memcached server is unreachable.