Question

An application that I’m building reads logs from some source and displays it on a grid. The logs can be a few MB to a few GB in size. To prevent any issues with memory, I’m using a grid and paging through the logs 500 lines at a time. This is what I hope to do:

I want to create a thread which will read logs and write them to a file about 500 lines each time, then signal another thread that logs have been written. The other thread will then read the file and display the lines on the grid and signal the first thread that it has finished reading. this goes on till there are no more logs to be written to the file.

Is it possible to switch between threads like this?

Answer 1

Yes, of course, it’s a variation of the producer-consumer model.

You can use some basic building blocks here like Thread and AutoResetEvent. The “producer” reads lines from the logs and posts them to a file (maybe you can use an in-memory buffer instead?) and then signals the other thread to read them:

AutoResetEvent consumerEvent = new AutoResetEvent(false);
AutoResetEvent producerEvent = new AutoResetEvent(false);

// producer code
while(/* lines still available */)
{
    // read 500 lines
    // write to shared file
    consumerEvent.Set(); // signal consumer thread
    producerEvent.WaitOne(); // wait to be signaled to continue
}

And the consumer code:

while(/* termination not received */)
{
    consumerEvent.WaitOne(); // wait for the producer to finish     
    // read lines from file and put them in the grid
    producerEvent.Set(); // allow producer to read more logs
}

This will allow some degree of parallelism between the consumer reading the file and the producer reading more logs and preparing the next batch.

When the producer has finished with the logs it can put a special termination message in the file to signal the consumer to exit gracefully.

This is one strategy and it’s pretty low-level and error-prone. You can skip the shared file entirely and use an in-memory buffer in the form of a BlockingCollection.

Define a ProducerTask class to hold some lines of text:

class ProducerTask 
{
    public String[] Lines { get; set; }
}

This task will hold 500 lines at a time.

Then use Task and BlockingCollection (.NET 4.0+) as follows:

BlockingCollection<ProducerTask> buffer = new BlockingCollection<ProducerTask>(1);

// producer task
while(/* lines still available */)
{
    // read 500 lines
    ProducerTask p = new ProducerTask();
    buffer.Add(p); // this will block until the consumer takes the previous task
}

// consumer task
while(/* termination not received */)
{
    ProducerTask p = buffer.Take(); // blocks if no task is available
    // put the lines in the grid
}

Much more simple and elegant.