Question

I have the variable

    StreamReader DebugInfo = GetDebugInfo();
    var text = DebugInfo.ReadToEnd();  // takes 10 seconds!!! because there are a lot of students

text equals:

<student>
    <firstName>Antonio</firstName>
    <lastName>Namnum</lastName>
</student>
<student>
    <firstName>Alicia</firstName>
    <lastName>Garcia</lastName>
</student>
<student>
    <firstName>Christina</firstName>
    <lastName>SomeLattName</lastName>
</student>
... etc
.... many more students

---

what am I doing now is:

  StreamReader DebugInfo = GetDebugInfo();
  var text = DebugInfo.ReadToEnd(); // takes 10 seconds!!!

  var mtch = Regex.Match(text , @"(?s)<student>.+?</student>");
  // keep parsing the file while there are more students
  while (mtch.Success)
  {
     AddStudent(mtch.Value); // parse text node into object and add it to corresponding node
     mtch = mtch.NextMatch();
  } 

the whole process takes about 25 seconds. to convert the streamReader to text (var text = DebugInfo.ReadToEnd();) that takes 10 seconds. the other part takes about 15 seconds. I was hoping I could do the two parts at the same time…

---

EDIT

I will like to have something like:

    const int bufferSize = 1024;

    var sb = new StringBuilder();

    Task.Factory.StartNew(() =>
    {
         Char[] buffer = new Char[bufferSize];
         int count = bufferSize;

         using (StreamReader sr = GetUnparsedDebugInfo())
         {

             while (count > 0)
             {
                 count = sr.Read(buffer, 0, bufferSize);
                 sb.Append(buffer, 0, count);
             }
         }

         var m = sb.ToString();
     });

     Thread.Sleep(100);

     // meanwhile string is being build start adding items

     var mtch = Regex.Match(sb.ToString(), @"(?s)<student>.+?</student>"); 

     // keep parsing the file while there are more nodes
     while (mtch.Success)
     {
         AddStudent(mtch.Value);
         mtch = mtch.NextMatch();
     }  

Edit 2

Summary

I forgot to mention sorry the text is very similar to xml but it is not. That’s why I have to use regular expressions… In short I think I could save time because what am I doing is converting the stream to a string then parsing the string. why not just parse the stream with a regex. Or if that is not possible why not get a chunk of the stream and parse that chunk in a separate thread.

Answer 1

@kakridge was right. I could be dealing with a race condition where one task is writing listToProces[30] for example and another thread could be parsing listToProces[30]. To fix that problem and also to remove the Thread.Sleep methods that are ineficient I ended up using semaphores. Here is my new code:

        StreamReader unparsedDebugInfo = GetUnparsedDebugInfo(); // get streamReader 
        listToProcess = new char[200000][];
        lastPart = null;
        matchLength = 0;

        // Used to signal events between thread that is reading text 
        // from readelf.exe and the thread that is parsing chunks
        Semaphore semaphore = new Semaphore(0, 1);

        // If task1 run out of chunks to process it will be waiting for semaphore to post a message
        bool task1IsWaiting = false;

        // Used to note that there are no more chunks to add to listToProcess.
        bool mainTaskIsDone = false;

        int counter = 0; // keep trak of which chunk we have added to the list

        // This task will be executed on a separate thread. Meanwhile the other thread adds nodes to  
        // "listToProcess" array this task will add those chunks to the dictionary. 
        var task1 = Task.Factory.StartNew(() =>
        {
            semaphore.WaitOne(); // wait until there are at least 1024 nodes to be processed

            int indexOnList = 0; // counter to identify the index of chunk[] we are adding to dictionary

            while (true)
            {
                if (indexOnList>=counter)   // if equal it might be dangerous! 
                {                           // chunk could be being written to and at the same time being parsed.
                    if (mainTaskIsDone)// if the main task is done executing stop
                        break;

                    task1IsWaiting = true; // otherwise wait until there are more chunks to be processed
                    semaphore.WaitOne();
                }

                ProcessChunk(listToProcess[indexOnList]); // add chunk to dictionary
                indexOnList++;
            }
        });


        // this block being executed on main thread  is responsible for placing the streamreader 
        // into chunks of char[] so that task1 can start processing those chunks
        {                
            int waitCounter = 1024; // every time task1 is waiting we use this counter to place at least 256 new chunks before continue to parse them

            while (true) // more chunks on listToProcess before task1 continues executing
            {
                char[] buffer = new char[2048]; // buffer where we will place data read from stream

                var charsRead = unparsedDebugInfo.Read(buffer, 0, buffer.Length);

                if (charsRead < 1){
                    listToProcess[counter] = pattern;
                    break;
                }

                listToProcess[counter] = buffer;
                counter++; // add chunk to list to be proceesed by task1.

                if (task1IsWaiting)
                {               // if task1 is waiting for more nodes process 256
                    waitCounter = counter + 256;    // more nodes then continue execution of task2
                    task1IsWaiting = false;
                }
                else if (counter == waitCounter)                    
                    semaphore.Release();                    
            }
        }

        mainTaskIsDone = true; // let other thread know that this task is done

        semaphore.Release(); // release all threads that might be waiting on this thread

        task1.Wait(); // wait for all nodes to finish processing