Question

I have a singly-linked list containing 1 value in each element.

struct listElement
{
    char guid[20];
    struct listElement *next;
    struct listElement *last;
    int numElements;
};

The code launches 10 pthreads in a pool that are all pthread_cond_wait() for an element to get added to the list.

My main() is reading strings from a file, one line at a time and populating the linked list by calling listPush(val)

listPush(val) acquires lock, creates new element, adds to end of list (or creates head if empty) unlocks then calls pthread_cond_signal() to let one of the 10 threads know there is now an element with work to be done.

In the case where numElements > numThreads I call pthread_cond_broadcast() because there should be enough work for each thread to pop and go.

Each thread listPop(rVal)‘s off a value (locks, removes, fixes pointers, unlocks) and processes it, then goes back to pthread_cond_wait() state.

My file has around 200 million lines. (1.2GB) I don’t want my linked list to get this large so I am trying to “throttle” the size of linked list.

Inside listPush(), before I lock the mutex, I have

if(head && head->numElements >= maxNumElements)
{
    while(head && head->numElements >= maxNumElements)
    {
        sleep(1);
    }
}

The idea is that if I “fill up” my list, I wait for the threads to process a chunk of it before adding more. I’ve reached this point where the app starts “pulsing”; basically I can see it waiting 1 second. I would expect this to never happen or happen very infrequently.

Is there a better way to limit the size of my list other than using sleep()?

Answer 1

This sounds like a classic producer-consumer problem. The solution is to have listPush() wait on a condition variable if the list is too big. Then one of the consumer threads can signal this condition variable once a list element has been consumed.

A completely different approach is to just use a pipe() to handle both the communication and the synchronization. That will obviate the need for both the linked list and the mutex.