We have implemented a message queue by using C# Queue. We know we only have ONE consumer to take available message out from the queue for processing with a while loop. We also know there is only ONE producer to put message onto the queue.
We have a lock on above message queue to make sure the consumer and producer cannot access the queue simultaneously.
My question is is that lock necessary? If the Queue increase its Count property AFTER an item is actually added and if the consumer check the Count before retrieve, the consumer should get a complete message item even we don’t have that lock. Right? So we will not face a partial message item issue. Then we can get rid of that lock?
That lock will slow down the system and occasionally we can see the retrieve thread is blocked for a while because we have a very heavy producer.
EDIT:
Unfortunately we are using .Net 3.5.
The real problem is that internal data structures of the queue may be being mutated whilst an enqueue or dequeue is happening, and during this period the data structure is in an indeterminate state from the point-of-view of another thread.
For example, an enqueue may require internal data structures to be expanded, where a new structure is created, and old items are copied form an old structure to a new structure. There will be many steps involved in this process where at any time it would be dangerous for another thread to access the queue as the operations are not complete.
Therefore, during enqueue\dequeue you have to lock to make these operation appear to be logically atomic.
You may try the new ConcurrentQueue class in .Net 4.0 as this possibly has better performance characteristics as it uses a non-locking algorithm.