Question

I don’t understand why this is occurring. Take the following piece of psuedo code:

volatile unsigned long count = 0;
volatile unsigned long sum = 0;

void ThreadFunction() {
    InterlockedIncrement(&count);
    InterlockedExchangeAdd(&sum, rand());
}

int main() {
    for (int i = 0; i < 10; ++i) {
        // This is the problematic instruction
        InterlockedExchange(&count, 0);
        InterlockedExchange(&sum, 0);

        std::vector<boost::thread> threads(i);
        for (int j = 0; j < i; ++j)
            threads[j] = boost::thread(ThreadFunction);

        while (count != i)
            Sleep(0);
    }
}

On the very first run of the program, when i = 0, the atomic exchange on sum from the main thread usually occurs after the spawned thread has completed. The operations on count always occur in the correct order.

This only happens once; it does the operations in the correct order for the remainder of the loop. It doesn’t always happen, but it usually does. If I break into the debugger or sleep before the atomic add, the instructions will be carried out in the correct order.

Either way, the result is that the value written by the thread is replaced by the 0 that should have occurred before the thread was even launched.

Why is this happening? Why can’t I rely on the processor to complete the atomic instructions I gave it in the order I gave them in? Doesn’t an atomic operation imply a memory barrier to prevent reordering?

Side note, this is occurring with Visual Studio 2010. I don’t have other versions to test with.

Answer 1

An atomic operation is atomic in the sense that it’s non-interruptable. It will either not execute or fully complete without another thread getting a look-in.

It does not mean that twenty atomic operations will always occur in the same order, that’s still subject to the vagaries of multi-threading.

Depending on how fast your threads start up, they may be sequenced as main, thrd1, main, thrd2, main, thrd3, ... or main, main, main, main, thrd1, main, main, thrd2, ... or any other combination (though I’m pretty certain main would always be first simply because its atomic operation is before starting any threads).