Question

I’m refactoring a project that I did not design. It is written in C/C++ for linux. The project has a major client component that looks like this:

Client -> Output Queuing Library (OQL) -> Controller

Client

• Messy semi-complex code, poorly designed (hodgepodge of OOP approximations using singletons/namespaces, just weird in many places – but it works)
• Custom protocol implementation (not my protocol, cannot modify)
• Shared Library
• Multi-threaded
  • Multiple threads call the OQL api, ie multiple threads output
• Accepts commands from controller via API
• Produces massive unsequenced output which is affected but not necessarily directly (and definitely not 1:1) by the controller input)

Output Queuing Library (OQL)

• Simple clean code, not really designed for it’s current workload (was never meant to queue, was actually originally just writing to stdout and then a message queue was shoe-horned in)
• Shared Library
• Single-threaded
• Exposes API which accepts many types of data from the client and builds textual representations of this data
• Inserts data into a sys V message queue

Controller

• Executable
• Single-threaded
• Elegant, fault tolerant C++ which makes extensive use of boost
• Written from scratch by me, the only part of the project I’ve been allowed to completely “fix” so to speak
• Interacts with client library via API to initiate connection to server
  • Saves data produced by Client and read from OQL into database in another layer

So the problem essentially boils down to this, the controller is single threaded and calls many API functions in the client library. Scenarios resulting from Controller calling Client API.

1. Normal (98%+)
   1. Controller calls client API function
   2. Client API function does magic internally
   3. API function returns true
   4. Client receives data as a result of magic in step 2, in another thread of execution and calls OQL put function from a secondary thread
   5. OQL writes data to message queue, queue either blocks or does not block but neither matter since the controller’s main thread of execution is running and processing data.
   6. Success!
2. Problem Scenario
   1. Controller calls client API function
   2. Client API function immediately produces result and BEFORE returning calls OQL put function from the main thread of execution in the Controller
   3. OQL writes data to the message queue and one of the following happens:
      1. Message queue is not full, does not block, everything returns and the controller processes the new data in the message queue and life moves on happily
      2. Problem Scenario Message queue IS full and DOES block

Now what I’m sure you can see is in the problem scenario, which is rare, the main thread of execution is blocking on a full message queue and also no data is being processed off of the other end of the queue since the controller is single threaded…

So yes it’s a bit of a mess, and no I’m not happy with the design but I’ve gotta figure out the best way to solve this without rewriting all of these interactions.

I’m trying to decide between:

1. Digging into the client, synchronizing all of the threads to a single I/O thread that interacts with OQL

   • I have basically zero faith that someone after me will not come in and break this down the road, introduce massive bugs and not be able to understand why

2. Introducing a writer thread into OQL

   • Takes a simple class and complicates it significantly
   • Introduces funny problems
     • Doesn’t the queue need a queue at that point? Since data has to get transported to the writer thread

Really just typing up this question was probably the best thing I could have done, since my thoughts are much more organized now…but does anyone have any input to offer here? Is there some design pattern I’m not seeing which would not require massive refactoring, and are my concerns overblown on either of these? I’m not even sure if it’s possible for anyone to provide meaningful advice here without knowing all angles of the problem, but I appreciate any insight you guys can offer.

Answer 1

You could change the Controller to use a second thread to do the reading from the message queue (and just post the data to a much larger buffer, internal to the Controller, to be read by the main Controller thread).