Question

I have a web interface where the user submits some data and it gets written to a database. In the background there is a C++ program which periodically checks the database for new entries. It then takes these entries, processes them and writes their result to a directory. It then proceeds to sleep and keep checking for new entries to process.

My question is in regards to adding multithreading to the C++ program. I have read that it’s generally a bad idea just to create a new thread every time you need a another job done, but rather add the jobs to a queue and disperse them out to a fixed number of threads that have already been created (say, 5 or so). Is this the proper design route to take for my situation? Also, if I understand pthread_join correctly, I don’t actually need to call it because I don’t want to wait for all of the jobs to finish before continuing to check for new updates to the database.

I just wanted to make sure I’m headed in the right direction, any affirmations/criticisms/resources?

Answer 1

You should first decide whether you even need more than one thread – it sounds like checking the database and writing files at some given interval can be accomplished using only one thread. Multiple threads would become useful when you start having to write different data to multiple files simultaneously at non-regular intervals. You are correct that using a queue of sorts would be the best way to distribute these ‘jobs’ to your threads, and that using a thread pool will give you a little more control over how many ‘jobs’ you want running simultaneously at any given time. The pthread_join method is used when you want to make sure one thread doesn’t exit before another – I’ve used this mostly to make sure that the program’s initial thread doesn’t exit after creating the thread pool, as when the parent thread exits the program’s execution stops. Some psuedo code based on my comments below.

main thread:

spawn child threads
while(some exit condition){
   check database for new jobs
   if(new jobs){
      acquire job queue mutex //mutexes ensures only one thread accesses shared 
      add job to queue        //data at a time
      signal on shared condition variable
      release job queue mutex 
   }
   sleep(some regular duration)
}

child thread:

while(some exit condition){
   acquire job queue mutex
   if(job queue's size == 0){
      wait on the shared condition variable
   }
   grab job from queue
   release job queue mutex
   handle job
}

See here for pthread/mutex/CV usage notes.