Question

My question is related to the “weight” parameter in the addServer function of the PHP Memcache extension.
For last couple of months, I have been using “weight = 1” for all servers. I am now trying to apply the configurations below to eventually remove “10.10.10.3” from the rotation and minimize data loss.
With the new weight values in place, my PHP client fails to retrieve value for keys that it used to be able to fetch. If I revert to “weight = 1”, all keys can be fetched without any problem.

Is there a configuration or anything I am missing in order to use the “weight” option correctly with “Memcache::addServer”?

Thank you for your help.

 $hosts = array(
      array('ip' => '10.10.10.1', 'port' => 11211, 'weight' => 100),
      array('ip' => '10.10.10.2', 'port' => 11211, 'weight' => 100),
      array('ip' => '10.10.10.3', 'port' => 11211, 'weight' => 1),
      array('ip' => '10.10.10.4', 'port' => 11211, 'weight' => 100),
 );

 $memcache = new Memcache();
 foreach($hosts as $host) {
    $host['port'] = isset($host['port']) ? (int) $host['port'] : 11211;
    $host['weight'] = isset($host['weight']) ? (int) $host['weight'] : 1;
    $memcache->addserver($host['ip'], $host['port'], false, $host['weight'], 1, 15);
 }

---

PHP version = 5.3.10

Memcache PHP variables
– memcache support => enabled
– memcache.allow_failover => 1 => 1
– memcache.chunk_size => 32768 => 32768
– memcache.compress_threshold => 20000 => 20000
– memcache.default_port => 11211 => 11211
– memcache.hash_function => crc32 => crc32
– memcache.hash_strategy => consistent => consistent
– memcache.lock_timeout => 15 => 15
– memcache.max_failover_attempts => 20 => 20
– memcache.protocol => ascii => ascii
– memcache.redundancy => 1 => 1
– memcache.session_redundancy => 2 => 2

Memcached versions
10.10.10.1 / 10.10.10.2 / 10.10.10.3 are running 1.4.5
10.10.10.4 is running 1.4.4

Answer 1

The weight parameter effects the consistent hashing used to determine which server to read/write keys from. Changing the weight of any 1 server in the pool will result in some cache misses. The number of servers in the pool and how much you change the weight will factor in on how many misses you may experience.

What you need to understand is that memcached is distributed which means that since you have 4 servers, keys are distributed amongst each of those servers (as close to evenly as possible [weight will effect even distribution]). If one server goes down, the data stored on that server is no longer accessible and will have to be fetched from the database as it is not available on any other servers. *Note that the PHP extensions memcache and memcached are just clients that access a memcached cluster (memcached was the newer of the two and supported more features, but both talk to servers in a memcached cluster).

When you want to store or retrieve a value from the cache, a hash is computed which determines where in the cluster the data should be put or read from. A common way to illustrate this is with a 360 degree circle as seen below. You calculate a hash, and use the node that it lands “closest” to in the circle. Adding or removing servers, or changing the weight of any one server will effect the outcome of the hashing and result in a miss.

Source: http://alpha.mixi.co.jp/blog/?p=158

If you want to phase a server out of the cluster slowly, I would recommend gradually decreasing its weight until it is 0 and you can then remove the server from the list completely. Keep in mind, any small change in weight can/will result in cache misses, but how much the weight changes (and how many servers you have) influences how many misses you will experience.

Here is a snippet from the Memcached tutorial story that may help explain some of this as well.

The preface is that two sysadmins set up a memcached cluster with 3 servers, and told memcached to use 1GB on each server…

So again, he takes keys that the Programmer uses and looks for them on
his memcached servers. ‘get this_key’ ‘get that_key’ But each time he
does this, he only finds each key on one memcached! Now WHY would you
do this, he thinks? And he puzzles all night. That’s silly! Don’t you
want the keys to be on all memcacheds?

“But wait”, he thinks “I gave each memcached 1 gigabyte of memory, and
that means, in total, I can cache three gigabytes of my database,
instead of just ONE! Oh man, this is great,” he thinks. “This’ll save
me a ton of cash. Brad Fitzpatrick, I love your ass!”

“But hmm, the next problem, and this one’s a puzzler, this webserver
right here, this one runing memcached it’s old, it’s sick and needs to
be upgraded. But in order to do that I have to take it offline! What
will happen to my poor memcache cluster? Eh, let’s find out,” he says,
and he shuts down the box. Now he looks at his graphs. “Oh noes, the
DB load, it’s gone up in stride! The load isn’t one, it’s now two.
Hmm, but still tolerable. All of the other memcacheds are still
getting traffic. This ain’t so bad. Just a few cache misses, and I’m
almost done with my work. So he turns the machine back on, and puts
memcached back to work. After a few minutes, the DB load drops again
back down to 1, where it should always be.

“The cache restored itself! I get it now. If it’s not available it
just means a few of my requests get missed. But it’s not enough to
kill me. That’s pretty sweet.”

The moral is that values stored in memcached are only stored on one server (that is the keys are distributed amongst all servers) and if any one of those servers becomes unavailable, the data is not available from cache and has to be fetched from the source.

In order to properly store and retrieve data from the correct nodes in the cluster, keys must be consistently hashed so the client knows which server is the correct one in the pool for that particular key.

As you may guess, different clients use different hashing techniques so using a Perl client and a PHP client to store/retrieve values will not work as intended since they hash differently. The exception to this is any client that uses libmemcached since then they will all be using the same hashing algorithm. PHP’s memcached extension uses libmemcached, where the memcache does not and uses its own hashing algorithm (as far as I know).

Further Reading:
– memcached wiki and the adventure in learning memcached
– Consistent Hashing
– Distributed Hash Tables