Question

I have a multithreaded c program written with pthreads. I want to know if what I am doing is right. Currently the program seems to run when there are no optimizations. But when I use optimizations, it fails. I am trying to figure out why.

I want to know if the basic multi threading synchronization that I am using is correct. Here is a short version which contains only the relevant parts. Please let me know if this is the correct way to multithread.

The way it works:
Each worker has a number of jobs to do. The worker threads are initiated first. Then they wait until a job has been assigned to them by the main thread. They complete the job, intimate the main thread and go back to waiting again.

The main thread assigns a job to N workers and signals them to start. Then it waits for their completion and resumes normal work.

Each worker thread has a context variable named workerCtx. This stores all the relevant info needed by the worker and also the mutexs and the cond variables.

The main thread has a mutex and a cond variable too.

Also the parent has variable called the threadCompletionCount. Initially this variable is 0 before assigning the work. Once each worker thread is done, it will increment this variable by one under the protection of a mutex and signal the parent thread. The parent thread will know that all the workers have completed when this variable is equal to the number of workers.

Here is a short version of the workerCtx

typedef enum {INITIALIZE = 0 , WAIT = 1, WORK1 = 2, WORK2 =3, DIE = 4} worktype_t;

typedef struct workerCtx_t
{
    int id;

    pthread_mutex_t mutex;
    pthread_cond_t cond;

    pthread_mutex_t * parentMutex;
    pthread_cond_t * parentCond;
    int * parentThreadCompletionCount;

    worktype_t workType;

    //Other application specific variables follow.
}workerCtx_t;

Here mutex and cond refer to the local mutex and cond variables specific to this thread. The parentMutex and parentCond are varibles referring to the parent’s varibles. All the threads point to the same parentMutex and parentCond.

Here is how the main thread works:

void waitForWorkerCompletion()
{
    int status;

    status = pthread_mutex_lock(&mutex);
    while (threadCompletionCount < NUM_WORKERS) 
    {
        status = pthread_cond_wait(&cond, &mutex);
    }
    status = pthread_mutex_unlock(&mutex);

    status = pthread_mutex_lock(&mutex);
        threadCompletionCount = 0;
    status = pthread_mutex_unlock(&mutex);
}

void assignWorkToWorker(worktype_t workType)
{
    for(int i=0; i<NUM_WORKERS; i++)
    {
        pthread_mutex_lock(&(workerCtxs[i]->mutex) );
            workerCtxs[i]->workType = workType;
            pthread_cond_signal(&(workerCtxs[i]->cond) );
        pthread_mutex_unlock(&(workerCtxs[i]->mutex));      
    }
    waitForWorkerCompletion();
}

void setup
{
    for(int i=0; i<NUM_WORKERS; i++)
    {       
        workerCtxs[i]->id = i;
        workerCtxs[i]->workType = WAIT;

        assert( pthread_mutex_init(&(workerCtxs[i]->mutex), NULL) == 0) ;
        assert( pthread_cond_init(&(workerCtxs[i]->cond), NULL) == 0);

        workerCtxs[i]->parentMutex = &mutex;
        workerCtxs[i]->parentCond = &cond;
        workerCtxs[i]->parentThreadCompletionCount = &threadCompletionCount;

        pthread_create (&(workers[i]), NULL, workerMain, (workerCtxs[i]) );

    }
}

int main()
{
    setup()

    For each work (workType_t workType) that comes up
        assignWorkToWorker(workType);
}

Here is what each worker does:

void signalCompletion(workerCtx_t * ctx)
{
    pthread_mutex_lock(&ctx->mutex);
        ctx->workType = WAIT;
    pthread_mutex_unlock(&ctx->mutex);

    int rc = pthread_mutex_lock(ctx->parentMutex);
        *(ctx->parentThreadCompletionCount) = *(ctx->parentThreadCompletionCount) + 1;
        pthread_cond_signal(ctx->parentCond);
    rc = pthread_mutex_unlock(ctx->parentMutex);
}

void * workerMain(void * arg)
{
    workerCtx_t * ctx = (workerCtx_t *) arg;

    while(1)
    {
        pthread_mutex_lock(&ctx->mutex);
        while(ctx->workType == WAIT);
        {
            int status = pthread_cond_wait(&ctx->cond, &ctx->mutex);
        }
        pthread_mutex_unlock(&ctx->mutex);

        if(ctx->workType == INITIALIZE)
        {
            init(ctx);
            signalCompletion(ctx);
        }
        else if(ctx->workType == WORK1)
        {
            doWork1(ctx);
            signalCompletion(ctx);
        }
        else if(ctx->workType == WORK2)
        {
            doWork2(ctx);
            signalCompletion(ctx);
        }
        else if(ctx->workType == DIE)
        {
            signalCompletion(ctx);
            break;
        }
    }

    return NULL;
}

I hope that I have made this easy enough to understand and taken out all application oriented details out from the code posted. This code just handles the multithreading part. Now the program hangs. When I use gdb, it tells me that it is waiting in waitForWorkerCompletion() method.

Is this multithreading model correct??

Answer 1

What the heck, I will make this an answer.

The following line looks very suspicious:

while(ctx->workType == WAIT);

I do not think you want the semi-colon there. With it, you are subject to a variety of strange race conditions… Which could explain why the behavior differs between non-optimized and optimized builds.

Otherwise, I see nothing wrong with this code. It is a bit of a strange way to use a worker pool, but it looks to me like it should work.