Question

I’m a beginner in programming in C#.

I want to read large (>1GB) text file (containing values of a voltage readings in one single column) and display it on the MS control chart.

It works okay with small files (~50mb) but it gets stuck on files >300mb and even gives an out-of-memory exception for larger files.

I have ~30e6 values each like this one: “0.189312433308071”.

Here’s how im reading the data right now:

System.IO.StreamReader sr = new
System.IO.StreamReader(openFileDialog1.FileName);
string line;

int pointIndex = 0;


while ((line = sr.ReadLine()) != null)  

{
    dataVoltage.Add(line);
    chart1.Series["Default"].Points.AddXY(pointIndex, Convert.ToDouble(line));
    pointIndex=pointIndex+1;

    }

sr.Close();

How can I successfully do this without waiting for minutes for file to load or not load at all?

Thanks.

Answer 1

I think your performance problems are entirely in adding values to your chart. I created a medium-sided file (~560mb) using the first line of main, then read it and summed its values. Took less than a minute.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.IO;

namespace LargeReadWrite
{
    class Program
    {
        const string fileName = "D:\\PersonalDev\\stackoverflow\\largefile.txt";
        static void Main(string[] args)
        {
            CreateLargeFile();
            ReadLargeFile();
        }

        static void CreateLargeFile()
        {
            Random r = new Random();
            using (StreamWriter sw = new StreamWriter(fileName))
            {
                for (uint i = 0; i < 30000000; i++)
                {
                    sw.WriteLine(r.NextDouble());
                }
            }
        }

        static void ReadLargeFile()
        {
            using (StreamReader sr = new StreamReader(fileName))
            {
                double total = 0.0;
                while (sr.Peek() >= 0)
                {
                    total += Double.Parse(sr.ReadLine());
                }
                Console.WriteLine(total);
            }
        }
    }
}

Run some performance profiling tools and decide if the chart is the right place to hold this, as holding that many values will eat as much memory as you need to hold. You may do better bucketizing the average of your results for every n readings.