Question

can you please help me to find out if it takes longer for a cache write to finish when there are more cores/caches holding a copy of that line.
I also want to measure/quantify how much longer it actually takes.

I couldn’t find anything useful on google and I have trouble measuring it myself plus interpret what I measure because of the many things that can happen on a modern processor.
(reordering, prefetching, buffering and god knows what)

Details:

My basic process of measuring it is roughly as follows:

write soemthing to the cacheline on processor 0
read it on processors 1 to n.

rdtsc
write it on process 0
rdtsc

I am not even sure which instructions to actually use for read/write on process 0 in order to make sure the write/invalidate is finished before the final time measurement.

At the moment I fiddle with an atomic exchange (__sync_fetch_and_add()), but it seems that the number of threads is itself important for the length of this operation (not the number of threads to invalidate) — which is probably not what I want to measure?!.

I also tried a read, then write, then memory barrier (__sync_synchronize()). This looks more like what I expect to see,
but here I am also not sure if the write is finished when the final rdtsc takes place.

As you can guess my knowledge of CPU internals is somewhat limited.

Any help is very much appreciated!

ps:
* I use linux, gcc and pthreads for the measurements.
* I want know this for modeling a parallel algorithm of mine.

Edit:

In a week or so (going on vacation tomorrow) I’ll do some more research and post my code and notes and link it here (In case someone is interested), because the time I can spend on this is limited.

Answer 1

I started writing a very long answer, describing exactly how this works, then realized, I probably don’t know enough about the exact details. So I’ll do a shorter answer….

So, when you write something on one processor, if it’s not already in that processors cache, it will have to be fetched in, and after the processor has read the data, it will perform the actual write. In doing so, it will send a cache-invalidate message to ALL other processors in the system. These will then throw away any content. If another processor has “dirty” content, it will in itself write out the data, and ask for an invalidation – in which case the first processor will have to RELOAD the data before finishing its write (otherwise, some other element in the same cacheline may get destroyed).

Reading it back into the cache will be required on every other processor that is interested in that cache-line.

The __sync_fetch_and_add() wilol use a “lock” prefix [on x86, other processors may vary, but the general idea on processors that support “per instruction” locks is roughtly the same] – this will issue a “I want this cacheline EXCLUSIVELY, everyone else please give it up and invalidate it”. Just like the first case, the processor may well have to re-read anything that another processor may have made dirty.

A memory barrier will not ensure that data is updated “safely” – it will just make sure that “whatever happened (to memory) before now is visible to all processors by the time this instructon finishes”.

The best way to optimize the use of processors is to share as little as possible, and in particular, avoid “false sharing”. In a benchmark many years ago, there was a structure like [simplifed] this:

struct stuff {
    int x[2];
    ... other data ... total data a few cachelines. 
} data;

void thread1()
{
    for( ... big number ...)
        data.x[0]++;
}

void thread2()
{
    for( ... big number ...)
        data.x[1]++;
}

int main()
{
    start = timenow();

    create(thread1);
    create(thread2);

    end = timenow() - start;   
}

Since EVERY time thread1 wrote to the x[0], thread2’s processor had to get rid of it’s copy of x[1], and vice versa, the result is was that the SMP test [vs just running thread1] was running about 15 times slower. By altering the struct like this:

struct stuff {
    int x;
    ... other data ... 
} data[2];

and

void thread1()
{
    for( ... big number ...)
        data[0].x++;
}

we got 200% of the 1 thread variant [give or take a few percent]

Right, so the processor has queues of buffers where write operations are stored when the processor is writing to memory. A memory barrier (mfence, sfence or lfence) instruction is there to ensure that any outstanding read/write, write or read type operation has completely been finished before the processor proceeds to the next instruction. Normally, the processor would just continue on it’s jolly way through any following instructions, and eventualy the memory operation becomes fulfilled some way or another. Since modern processors have a lot of parallel operations and buffers all over the place, it can take quite some time before something ACTUALLY trickles through to where it eventually will end up. So, when it’s CRITICAL to make sure that something has ACTUALLY been done before proceeding (for example, if we have written a bunch of instructions to the video memory, and we now want to kick off the run of those instructions, we need to make sure that the ‘instruction’ writing has actually finished, and some other part of the processor isn’t still working on finishing that. So use an sfence to make sure that the write has really happened – that may not be a very realistic example, but I think you get the idea.)