Question

I’m writing an app that parses a very large logfile, so that the user can see the contents in a treeview format. I’ve used a BackGroundWorker to read the file, and as it parses each message, I use a BeginInvoke to get the GUI thread to add a node to my treeview. Unfortunately, there’s two issues:

• The treeview is unresponsive to clicks or scrolls while the file is being parsed. I would like users to be able to examine (ie expand) nodes while the file is parsing, so that they don’t have to wait for the whole file to finish parsing.
• The treeview flickers each time a new node is added.

Here’s the code inside the form:

private void btnChangeDir_Click(object sender, EventArgs e)
{
    OpenFileDialog browser = new OpenFileDialog();

    if (browser.ShowDialog() == DialogResult.OK)
    {
        tbSearchDir.Text = browser.FileName;
        BackgroundWorker bgw = new BackgroundWorker();
        bgw.DoWork += (ob, evArgs) => ParseFile(tbSearchDir.Text);
        bgw.RunWorkerAsync();
    }
}

private void ParseFile(string inputfile)
{
    FileStream logFileStream = new FileStream(inputfile, FileMode.Open, FileAccess.Read, FileShare.ReadWrite);
    StreamReader LogsFile = new StreamReader(logFileStream);

    while (!LogsFile.EndOfStream)
    {
        string Msgtxt = LogsFile.ReadLine();
        Message msg = new Message(Msgtxt.Substring(26)); //Reads the text into a class with appropriate members
        AddTreeViewNode(msg);
    }
}

private void AddTreeViewNode(Message msg)
{
    TreeNode newNode = new TreeNode(msg.SeqNum);

    BeginInvoke(new Action(() =>
                               {
                                   treeView1.BeginUpdate();
                                   treeView1.Nodes.Add(newNode);
                                   treeView1.EndUpdate();
                                   Refresh();
                               }
                    )); 
}

What needs to be changed?

EDIT

New code, to replace the last function above:
        List<TreeNode> nodeQueue = new List<TreeNode>(1000);

        private void AddTreeViewNode(Message msg)
        {
            TreeNode newNode = new TreeNode(msg.SeqNum);

            nodeQueue.Add(newNode);

            if (nodeQueue.Count == 1000)
            {
                var buffer = nodeQueue.ToArray();
                nodeQueue.Clear();
                BeginInvoke(new Action(() =>
                                           {
                                               treeView1.BeginUpdate();
                                               treeView1.Nodes.AddRange(buffer);
                                               treeView1.EndUpdate();
                                               Refresh();
                                               Application.DoEvents();
                                           }
                                ));
            }
        }

Not sure why I left the Refresh and the DoEvents in there. A bit of testing other comments…

Answer 1

Invoke/BeginInvoke uses PostMessage internally to marshal a request from an arbitrary thread to the UI thread. BeginInvoke will be flooding your windows message queue with messages which need to be processed by the UI thread, that accompanied by the fact that you are disabling the tree redraw with every node you add is probably impacting your ability to interact with the tree while it is loading.

One option would be to batch a number of updates together and then send them update the tree in batches. So parse the file and update the tree with every 100 or some number of nodes rather than 1 at a time.

Update: After your edit to add nodes in batches I would suggest the following.

1- Rather use Invoke than BeginInvoke, otherwise the queue fills up while the tree is being updated and then once the tree is updated the next thousand nodes are ready to be inserted which puts you right back where you where.

2- Sleep a few 100 milliseconds after inserting each batch so that there is a period that UI can respond. You can play with this, this will balance performance vs. user experience. Longer sleep will feel more responsive for the user, but will ultimately take longer to get all the data loaded.

3- Note that your current batching solution will miss the last few nodes if the total count is not a multiple of 1000

    private void AddTreeViewNode(Message msg) 
    { 
        TreeNode newNode = new TreeNode(msg.SeqNum); 

        nodeQueue.Add(newNode); 

        if (nodeQueue.Count == 1000) 
        { 
            var buffer = nodeQueue.ToArray(); 
            nodeQueue.Clear(); 
            Invoke(new Action(() => 
                { 
                    treeView1.BeginUpdate(); 
                    treeView1.Nodes.AddRange(buffer); 
                    treeView1.EndUpdate(); 
                }));
            System.Threading.Thread.Sleep(500); 
        } 
    }