Question

I have a c program running in Linux kernel 2.6.18-194, server has 1 cpu socket with 6 cores with hyper-threading , thread1 receive data and then thread2 and thread3 pass the data thread1 received to another process , after both thread2 and thread3 successfully finish passing data , thread1 will receive data again !!

The floowing is thread1 source :

        DoGetDataFromSocket() ;
        iGlbBOOKReadDone = 0 ;
        iGlbPOSIReadDone = 0 ;
        sem_post(sembook) ;
        sem_post(semposi) ;
        sem_wait(semfinished) ;

The following is thread2 and thread3 source :

    if(bThisThreadIsBook==1)
        sem_wait(sembook) ;
    else
        sem_wait(semposi) ;

    DoPassDatatoAnotherProcess() ;

    if(bThisThreadIsBook==1)
    {
        __sync_add_and_fetch(&iGlbBOOKReadDone,1) ;
    }
    else
    {
        __sync_add_and_fetch(&iGlbPOSIReadDone,1) ;
    }

    Pthread_mutex_lock(&DoneMutex) ;
    if(  (iGlbBOOKReadDone == 1) && (iGlbPOSIReadDone == 1) )
        sem_post(semfinished) ;
    Pthread_mutex_unlock(&DoneMutex) ;

It works fine to me , I try to remove mutex_lock DoneMutex in thread2 and thread3 , it still works fine , what I am curious is , if thread2 is doing __sync_add_and_fetch(&iGlbBOOKReadDone,1) , and thread3 is doing __sync_add_and_fetch(&iGlbPOSIReadDone,1)
at exact the same time , then both thread will take if( (iGlbBOOKReadDone == 1) && (iGlbPOSIReadDone == 1) ) to false , and sem_post(semfinished) would never be called ,
But I do a lot of pressure tests ,this never happen!! Is it related with __sync_add_and_fetch function ?

Answer 1

As mentioned in my comment, regardless of whether you use the mutex or not you have a race condition where the semfinished semphore can be posted twice, which means that thread1 might not allow the threads to finish in a future round.

It’s easy to see that both thread2 and thread3 can simultaneously ‘reach’ the blank line just before the pthread_mutex_lock() call. If that happens, then both threads will call sem_post(semfinished).

To avoid this situation and to make the code easier to reason about, making it so you can be certain that exactly one of thread2 and thread3 will callsem_post(semfinished)`, you might want to consider doing something like the following:

• thread1:

  DoGetDataFromSocket() ;
  threadDoneCount = 0;
  sem_post(sembook) ;
  sem_post(semposi) ;
  sem_wait(semfinished) ;
  

• thread2 and thread3:

  if(bThisThreadIsBook==1)
      sem_wait(sembook) ;
  else
      sem_wait(semposi) ;
  
  DoPassDatatoAnotherProcess() ;
  
  int doneCount = __sync_add_and_fetch(&threadDoneCount,1) ;
  
  if (doneCount == 2)
      sem_post(semfinished) ;