Question

This is from just a single memcached server with around 20M keys (no expiry) and around 2G of data.

What’s the easiest way to get a dump of all the key/value pairs into a flat file? I first looked at the java net.spy.memcached.MemcachedClient, but this client does not support getting all keys (I think). If I had a list of all keys (which I don’t), I could easily use this client to get all of the values.

I know I can get all keys using some telnet commands (e.g., telnet localhost 11211; stats items; stats cachedump ), but it’s not clear to me how to automate this robustly.

EDIT: Here’s what I did to get this working on a toy memcached server on my machine. It seems to work but I only put two keys in memcached, so hopefully this method will scale ok:

shell commands:

sudo yum install memcached
sudo /etc/init.d/memcached restart # maybe unnecessary
sudo yum install php
sudo yum install php-pecl-memcache
sudo service httpd reload

php script, based on this:

<?php
$memcache = new Memcache();
$memcache->connect('127.0.0.1', 11211) or die ("Could not connect");
$list = array();
$allSlabs = $memcache->getExtendedStats('slabs');
$items = $memcache->getExtendedStats('items');
foreach($allSlabs as $server => $slabs) {
    foreach($slabs AS $slabId => $slabMeta) {
        if (!is_int($slabId)) {
            continue;
        }
        $cdump = $memcache->getExtendedStats('cachedump', (int) $slabId, 100000000);
        foreach($cdump AS $server => $entries) {
            if ($entries) {
                foreach($entries AS $eName => $eData) {
                    print_r($eName);
                    print_r(":");
                    $val = $memcache->get($eName);
                    print_r($val);
                    print_r("\n");
                }
            }
        }
    }
}
?>

EDIT2: The above script does not seem to return all of the mappings. If I insert the line count($entries), it only returns a little over 50k, even with the limit parameter set to 100M, but executing stats items from telnet shows over 5M entries. Does anyone know why this might be the case?

EDIT3: This link suggests that cachedump doesn’t get all keys from memcached. I’ve hit a limit of around 50k keys that are returned either by cachedump, this PHP script, or a perl script similar to one in the link provided by Zach Bonham. Is there any way around this?

Answer 1

disclaimer: I don’t know what I”m doing, just sounded like an interesting problem.

Did you see this article? “How to Dump Keys from Memcache” by Lars Windolf.

From the article:

Memcache itself provide the means to peak into the data. The protocol
provides commands to peak into the data that is organized by slabs
(categories of data of a given size range. There are some significant
limitations though:

• You can only dump keys per slab class (keys with roughly the same content size)
• You can only dump one page per slab class (1MB of data)
• This is an unofficial feature that might be removed anytime.

Effectively, it requires some knowledge of how memcache stores data in memory (which I don’t). You need to find each ‘slab’, then you can dump the keys for that slab, and then ultimately, the values for those keys.

There is a tools section in the article which uses various languages to dump at least the keys, but only the perl script dumps both keys and values.