I’ve been using php 5.2 and Apache 2.2 and memcached 1.2.6 for a long time, with client-side sharding across a number of hosts. This has worked fine AFAICT. Recently, I’ve started upgrading memcached clients to php 5.3. This comes from ubuntu server 10.04 LTS.
However, I’m starting to see a weird bug, where the value for some other key comes back, say once every 1,000,000,000 requests or so (that I know of). I have not been able to yet decide whether this is corrupting on store or load (debugging happens after data has expired).
The corrupted data returned from a get has sometimes been a value from a totally different key, and has also sometimes been a single element from within the array value that the key is supposed to have.
Searching the web finds no obvious mention of these symptoms, but that’s a hard search to get hits for, because most discussions are related to application-level race condition bugs. I have proven to myself that this is not one of those.
So, is this a known bug somewhere in this stack? Any others with similar experiences? Thanks in advance!
To answer some questions:
-
Yes, it’s an old version. It’s been working well for a very long time. Thus, I don’t think it’s the server screwing up (but I guess it could be). We tried upgrading to 1.4.5 but it failed tests, because we depend on some behaviors of older memcache that were incompatibly changed when upgrading. Fixing this will happen in the future, but you know the saying: if it ain’t broke….
-
Each sharded server machine (as well as the PHP clients) has 8 gigs of ECC RAM, so we’d know if there was a memory failure.
-
What I mean by the value of a different key is that, if I store, say, an array of email addresses into a key named “email_addresses_$id”, very infrequently, some later read-back of that key returns, say, a Python-pickled dict of product ids, which are only ever stored into a key named “product_ids_$serial” by totally different code. Also, very infrequently, we’ll get back a single email address, instead of a full array of email addresses (or an array of a single email address, which would be the expected case).
Also: I estimate that we push well over a terabyte of memcached traffic per day, so logging all of it to be able to go back and debug what happened on a failure we see one to three times a month is somewhat unlikely to be viable.
We have been able to reproduce this problem.
It turns out that PHP memcached client library version 3 fails if the connection to memcached fails. It returns an error, but queues the data of the previous request. The next request that comes into the library will cause a new connection to open, and then use the key from the new request, but the data from the old request!
It sounds crazy, I know, but it’s totally reproducible.