Question

My setup:

• 4 webservers
• Static content server (NFS mount)
• 2 db servers
• 2 “do magic” servers
• An additional 8 machines designated multi-purpose.

I’m writing a wrapper for three caching mechanisms so that they can be used in a somewhat normalized manner: Filesystem, Memcached and APC. I’m trying to come up with examples for use (and what to actually put in each cache).

File System

Handles content that we generate and then statically serve. RSS feeds, old report data, user specific pages, etc… This is all cached to the static server.

Memcached

PHP session data, MySQL query results, generally things that need to be available across our systems. We have 8 machines that can be included in the server pool.

APC

I have no idea. The two “do magic” servers are not part of any distributed system, so it seems likely that they could cache query results in APC and work from there. Past that, I can’t think of anything.

Query Caching

Given the nature of our SQL use, query caching reduces performance. I’ve disabled this.

In general, what types of data should be stored where? Does this setup even make sense?

Is there any use for an APC data cache in a distributed system (I can’t think of one)?

Is there something that I’m missing that would make things easier or more efficient?

Edit: I’ve figured out what Pascal was saying, finally. I had it stuck in my head that I would only be moving a portion of my config / whatever to APC, and still loading the rest of the file from disk. Any other suggestions?

Answer 1

I’m using the same kind of caching mecanism for some projects ; and we are using APC + memcached as caching systems.

There are two/three main differences between APC and memcached, when it comes to caching data :

• APC access is a bit faster (something like 5 times faster than memcached, if I remember correctly), as it’s only local — i.e. no network involved.
• Using APC, your cache is duplicated on each server ; using memcached, there is no duplication accross servers
  • whic means that memcached ensures that all servers have the same version of the data ; while data stored in APC can be different on each server

We generally use :

• APC for data that has to be accessed very often, is quick to generate, and :
  • either is not modified often
  • or it doesn’t matter if it’s not identical on all servers
• memcached for data that takes more time to generate, and/or is less used.
  • Or for data for which modifications must be visible immediatly (i.e. when there is a write to the DB, the cached entry is regenerated too)

For instance, we could :

• Use APC to store configuration variables :
  • Don’t change often
  • Are accessed very often
  • Are small
• Use memcached for content of articles (for a CMS application, for example) :
  • Not that small, and there are a lot of them, which means it might need more memory than we have on one server alone
  • Pretty hard/heavy to generate

A couple of sidenotes :

• If several servers try to write to the same file that’s shared via NFS, there can be problems, as there is no locking mecanism on NFS (as far as I remember)
• APC can be used to cache data, yes — but the most important reason to use it is it’s opcode caching feature (can save a large amount of CPU on the PHP servers)
• Entries in memcached are limited in size : you cannot store an entry that’s bigger than 1M (I’ve sometimes run into that problem — rarely, but it’s not good when it happens ^^ )