Question

I wrote the following code to be executed in parallel by several processes:

// spawn 10 times with id=0..9 by a master process.
void slave_processing(int id)
{
  SHARED_TYPE last=id;
   for(;;) {
     /* each slave operates on a specific byte of the shared array. */
     if (shared[id]!=last) fprintf(stderr,"S%i expected %i\n",id,last);
     shared[id]+=id;
     last+=id;
     if (last>10*id) {last=0; shared[id]=0;}
   }
}

They’re all using the same (IPC/linux) shared memory, but each accessing a separate entry of the array. It works flawlessly on my dual-core machine both when SHARED_TYPE is int and char. I compiled with aggressive optimisation (-O3) and checked the assembled binary to confirm memory references are performed for shared[id] accesses and a register is used for last.

Yet, I’m puzzled. I’d have expected that at some point, affecting bytes from one core may not have reflected on the cached content from the other core. E.g. one core might have [xxyyzzuu] in cache and write back [xxyyZZuu] to memory while in the same time another core had upgraded the memory word to [XXyyzzuu] (assuming each pair of char is a byte in my 32-bit word).

Is linux doing some magic setup so that memory obtained with shmget cannot be cached ? or is there some low-level cache syncrhonisation going on that ensure core #1 can read the latest modification of core #2 to update the selected byte without nasty side-effects ?

Is there any other (multi-core) architecture you know where the above code would have failed (entered fprintf) ?

Answer 1

The code assumes the compiler does not cache(e.g. read the content from memory once into a register and continues to use the register instead of memory) any of the values it meddles with in your loop, but does a memory fetch for each read from the shared segment. To ensure all parts see data written to memory, accessing the shared array should be done with memory barriers.

Though compiler dependent , it will normally do a memory fetch if your shared is volatile.

Since you call fprintf(), the compiler must assume fprintf() could access or update something in the shared array – after all, the compiler doesn’t know what fprintf() does, so theoretically it could access a global array, the compiler is required to re-fetch that memory from shared(unless it can otherwise prove it’s impossible for fprintf update it)

Your code might behave differently if shared is not volatile ,and you e.g. do

SHARED_TYPE last=id;
int wrong = 0,i;
for(i = 0; i < 10000000;i++) {
  if (shared[id]!=last) wrong++;
  shared[id]+=id;
  last+=id;
  if (last>10*id) {last=0; shared[id]=0;}
}
return wrong;

The compiler is not required to do a memory fetch to get the latest value of shared[id] on each iteration here. Then again, if the compiler does cache a value, it might just spin the loop, checking the same right thing again and again without getting updated values from memory, leaving wrong = 0 as expected. Whatever you do here is not safe, and depends what code the compiler would generate for you – it might generate code with substantial different results for small code changes.

Intertwined with this is what data types can be atomically read from memory, which will depend on the processor, the data type and whether the access is memory aligned.

This is in addition to cache-coherence. The common desktop and server platforms we use are cache-coherent, which means the hardware takes care of the magic of updating memory so changes are seen by all processors/cores and not e.g. residing in a L1 cache on only 1 core when another core needs to access the same memory location. (Some memory regions e.g. regions used for DMA might not give you such cache coherence guarantees)

Desktop/server platforms today are looking quite NUMA like, but many bigger and more special built NUMA systems not have a cache-coherence guarantee.