Question

I’m trying to implement a basic worker pool using pthreads.
The scenario is that I want a fix number of workers, that live throughout the duration of my program.

I’ll never need to signal single threads, but all threads at once, thats why I want to do a single broadcast.

I’ll need to wait for all threads to finish before the main program continues, so I’ve decided to use barrier_wait in each worker thread.

The thing is, that the broadcast doesn’t work if my thread calls barrier_wait.

Full example and compileable code is seen below. This is just for single trigger of broadcast, in my full version, I’ll ofcause loop over something like

while(conditionMet){
  1.prepare data
  2.signal threads using data
  3.post processing of thread results (because of barrier all threads finished)
  4.modify conditionMet if needed
}

Thanks

#include <pthread.h>
#include <stdio.h>
#include <unistd.h>
void checkResults(char *str,int i){
  fprintf(stdout,"%s:%d\n",str,i);
}
void checkResults(char *str,size_t n,int i){
  fprintf(stdout,"%s[%lu]:%d\n",str,n,i);
}

/* For safe condition variable usage, must use a boolean predicate and  */
/* a mutex with the condition.                                          */
int                 conditionMet = 0;
pthread_cond_t      cond  = PTHREAD_COND_INITIALIZER;
pthread_mutex_t     mutex = PTHREAD_MUTEX_INITIALIZER;
pthread_barrier_t barr;

#define NTHREADS    3

void *threadfunc(void *parm)
{
  size_t i = (size_t) parm;
  int           rc;

  rc = pthread_mutex_lock(&mutex);
  checkResults("\tpthread_mutex_lock()",i, rc);

  while (0==conditionMet) {
    printf("\tThread blocked[%d]\n",(int)i);
    rc = pthread_cond_wait(&cond, &mutex);
    checkResults("\tpthread_cond_wait()",i, rc);
    checkResults("\tbefore barrier",i);
   rc = pthread_barrier_wait(&barr);//broadcast works if this is commented out
    if(rc)
      fprintf(stdout,"problems waiting for baarr\n");
    checkResults("\tafter  barrier",i);
  }

  rc = pthread_mutex_unlock(&mutex);
  checkResults("\tpthread_mutex_lock()",i, rc);
  return NULL;
}

int main(int argc, char **argv)
{
  int                   rc=0;
  int                   i;
  pthread_t             threadid[NTHREADS];

  if(pthread_barrier_init(&barr, NULL,NTHREADS))
    {
      printf("Could not create a barrier\n");
    }



  printf("Enter Testcase - %s\n", argv[0]);

  printf("Create %d threads\n", NTHREADS);
  for(i=0; i<NTHREADS; ++i) {
    rc = pthread_create(&threadid[i], NULL, threadfunc,(void *) i);
    if(rc)
      checkResults("pthread_create()", rc);
  }

  sleep(5);  /* Sleep isn't a very robust way to serialize threads */
  rc = pthread_mutex_lock(&mutex);
  checkResults("pthread_mutex_lock()", rc);

  /* The condition has occured. Set the flag and wake up any waiters */
  conditionMet = 1;
  printf("\nWake up all waiters...\n");
  rc = pthread_cond_broadcast(&cond);
  checkResults("pthread_cond_broadcast()", rc);

  rc = pthread_mutex_unlock(&mutex);
  checkResults("pthread_mutex_unlock()", rc);

  printf("Wait for threads and cleanup\n");
  for (i=0; i<NTHREADS; ++i) {
    rc = pthread_join(threadid[i], NULL);
    checkResults("pthread_join()", rc);
  }
  pthread_cond_destroy(&cond);
  pthread_mutex_destroy(&mutex);

  printf("Main completed\n");
  return 0;
}

Answer 1

The thread function will lock the mutex right after receiving the signal. Thus only one thread function will wait on the barrier (with mutex still locked) and the barrier criterion will never be met.

You should redesign the logic of your application in order to use the barrier. The mutex must be unlocked right before waiting for barrier. Also, given the pthread_cond_wait() usage in your code, only one thread will ever be active in your application which eliminates the need of multithreading at all.

Edit:

I’d like to elaborate the last sentence a bit. Lets assume that we modify the thread function like this:

while (0==conditionMet) {     
    printf("\tThread blocked[%d]\n",(int)i);     
    rc = pthread_cond_wait(&cond, &mutex);     
    checkResults("\tpthread_cond_wait()",i, rc);     
    checkResults("\tbefore barrier",i);

    pthread_mutex_unlock(&mutex); //added    

    rc = pthread_barrier_wait(&barr);//broadcast works if this is commented out     
    if(rc)
        fprintf(stdout,"problems waiting for baarr\n");     
    checkResults("\tafter  barrier",i);   
}

This way we can eliminate the deadlock when only one thread is able to reach barrier cause of mutex locked. But still only one thread in the time moment given will operate in critical section: when it’s pthread_cond_wait() returns the mutex is locked and it will stay locked until the thread function reaches _unlock(); _wait(); pair. Only after that next single thread will be able to run and reach its barrier. Wash, rinse, repeat…

What OP prolly wants is to have thread functions to operate simultaneously (why else will anyone want to have a thread pool?). In that case the function may look like this:

void *threadfunc(void *parm)
{
/*...*/
struct ThreadRuntimeData {
} rtd;
while (0==conditionMet) {     
    printf("\tThread blocked[%d]\n",(int)i);     
    rc = pthread_cond_wait(&cond, &mutex);     
    checkResults("\tpthread_cond_wait()",i, rc);

    GetWorkData(&rtd); //Gets some data from critical section and places it in rtd
    pthread_mutex_unlock(&mutex);

    ProcessingOfData(&rtd); //here we do the thread's job 
    //without the modification of global data; this may take a while

    pthread_mutex_lock(&mutex);
    PublishProcessedData(&rtd); //Here we modify some global data 
    //with the results of thread's work. 
    //Other threads may do the same, so we had to enter critical section again
    pthread_mutex_unlock(&mutex);   
    checkResults("\tbefore barrier",i);
    rc = pthread_barrier_wait(&barr);//broadcast works if this is commented out     
    if(rc)
        fprintf(stdout,"problems waiting for baarr\n");     
    checkResults("\tafter  barrier",i);   
}
/*...*/
}

This is just a sketch ofcourse. The optimal design of thread function depends on what OP wants the thread to do.

As a side-note, the code for checking the pthread_barrier_wait() return result must take in account the PTHREAD_BARRIER_SERIAL_THREAD return. Also it would be safer to declare conditionMet as volatile.