Question

say i have a process which spawns 2 threads

the first thread listens in a tight loop for packet events on a UDP port.
the second thread is to receive the bytes contained in this UDP packet and parse it/do stuff.

can anyone recommend a faster method than using linux message queues?
i think they are slow because they are copying the bytes upon writing to the queue and copying them again upon reading from the queue

i am aware of the zeromq library but is there a slick way to do this without that overhead?
i realize i could just use a tcp/ip socket between the two threads for simple queued communication but is there a faster way?

i am thinking maybe a ring buffer in memory that is shared between the threads and a mutex used to control a pointer to the most recently updated element?

anyones thoughts here?

Answer 1

The most efficient method that I can think of would use one linked list, one mutex, and one condition variable:

• Thread A:

  1. Allocate a udp-packet-buffer object (the object should contain previous and next pointers, and an array of bytes to store UDP data in)
  2. recv() UDP data into the object’s byte-array
  3. lock the mutex
  4. append the udp-packet-buffer object to tail of the linked-list
  5. unlock the mutex
  6. signal the condition variable
  7. goto 1

• Thread B:

  1. wait on the condition variable (until the condition variable is signaled)
  2. lock the mutex
  3. pop the next udp-packet-buffer off the head of the linked-list
  4. unlock the mutex
  5. parse/handle the UDP data in the udp-packet-buffer object
  6. delete the udp-packet-buffer object
  7. goto 1

That gives you a zero-copy communications mechanism. The time spent holding the mutex is minimal, so contention should be low also.

If you wanted to optimize a bit further, you could have thread B grab all of the items out of the linked-list at once instead of just popping off one at a time — something that is possible to do in O(1) time with a linked list. And then have thread A signal the condition variable only if the linked list was empty just before it appended its latest udp-packet-buffer object. That would cut down on the number of times the threads have to lock/unlock/signal when under heavy load.