Question

Morning,

I’m trying to split a large text file (15,000,000 rows) using StreamReader/StreamWriter. Is there a quicker way?

I tested it with 130,000 rows and it took 2min 40sec which implies 15,000,000 rows will take approx 5hrs which seems a bit excessive.

//Perform split.
public void SplitFiles(int[] newFiles, string filePath, int processorCount)
{
    using (StreamReader Reader = new StreamReader(filePath))
    {
        for (int i = 0; i < newFiles.Length; i++)
        {
            string extension = System.IO.Path.GetExtension(filePath);
            string temp = filePath.Substring(0, filePath.Length - extension.Length)
                              + i.ToString();
            string FilePath = temp + extension;

            if (!File.Exists(FilePath))
            {
                for (int x = 0; x < newFiles[i]; x++)
                {
                    DataWriter(Reader.ReadLine(), FilePath);
                }
            }
            else
            {
                return;
            }
        }
    }
}

public void DataWriter(string rowData, string filePath)
{
    bool appendData = true;
    using (StreamWriter sr = new StreamWriter(filePath, appendData))
    {
        {
            sr.WriteLine(rowData);
        }
    }
}

Thanks for your help.

Answer 1

You haven’t made it very clear, but I’m assuming that the value of each element of the newFiles array is the number of lines to copy from the original into that file. Note that currently you don’t detect the situation where there’s either extra data at the end of the input file, or it’s shorter than expected. I suspect you want something like this:

public void SplitFiles(int[] newFiles, string inputFile)
{
    string baseName = Path.GetFileNameWithoutExtension(inputFile);
    string extension = Path.GetExtension(inputFile);
    using (TextReader reader = File.OpenText(inputFile))
    {
        for (int i = 0; i < newFiles.Length; i++)
        {
            string outputFile = baseName + i + extension;
            if (File.Exists(outputFile))
            {
                // Better than silently returning, I'd suggest...
                throw new IOException("File already exists: " + outputFile);
            }

            int linesToCopy = newFiles[i];
            using (TextWriter writer = File.CreateText(outputFile))
            {
                for (int j = 0; i < linesToCopy; j++)
                {
                    string line = reader.ReadLine();
                    if (line == null)
                    {
                        return; // Premature end of input
                    }
                    writer.WriteLine(line);
                }
            }
        }
    }
}

Note that this still won’t detect if there’s any unconsumed input… it’s not clear what you want to do in that situation.

One option for code clarity is to extract the middle of this into a separate method:

public void SplitFiles(int[] newFiles, string inputFile)
{
    string baseName = Path.GetFileNameWithoutExtension(inputFile);
    string extension = Path.GetExtension(inputFile);
    using (TextReader reader = File.OpenText(inputFile))
    {
        for (int i = 0; i < newFiles.Length; i++)
        {
            string outputFile = baseName + i + extension;
            // Could put this into the CopyLines method if you wanted
            if (File.Exists(outputFile))
            {
                // Better than silently returning, I'd suggest...
                throw new IOException("File already exists: " + outputFile);
            }

            CopyLines(reader, outputFile, newFiles[i]);
        }
    }
}

private static void CopyLines(TextReader reader, string outputFile, int count)
{
    using (TextWriter writer = File.CreateText(outputFile))
    {
        for (int i = 0; i < count; i++)
        {
            string line = reader.ReadLine();
            if (line == null)
            {
                return; // Premature end of input
            }
            writer.WriteLine(line);
        }
    }
}