Question

What I have:

A large XML file @ nearly 1 million lines worth of content. Example of content:

<etc35yh3 etc="numbers" etc234="a" etc345="date"><something><some more something></some more something></something></etc123>
<etc123 etc="numbers" etc234="a" etc345="date"><something><some more something></some more something></something></etc123>
<etc15y etc="numbers" etc234="a" etc345="date"><something><some more something></some more something></something></etc123>

^ repeat that by 900k or so lines (content changing of course)

What I need:

Search the XML file for "<etc123". Once found move (write) that line along with all lines below it to a separate XML file.

Would it be advisable to use a method such as File.ReadAllLines for the search portion? What would you all recommend for the writing portion. Line by line is not an option as far as I can tell as it would take much too long.

Answer 1

To quite literaly discard the content above your search string, I would not use File.ReadAllLines, as it would load the entire file into memory. Try File.Open and wrap it in a StreamReader. Loop on StreamReader.ReadLine, then start writing to a new StreamWriter, or do a byte copy on the underlying filestream.

An example of how to do so with StreamWriter/StreamReader alone is listed below.

//load the input file
//open with read and sharing
using (FileStream fsInput = new FileStream("input.txt", 
    FileMode.Open, FileAccess.Read, FileShare.Read)) 
{
    //use streamreader to search for start
    var srInput = new StreamReader(fsInput);
    string searchString = "two";
    string cSearch = null;
    bool found = false;
    while ((cSearch = srInput.ReadLine()) != null)
    {
        if (cSearch.StartsWith(searchString, StringComparison.CurrentCultureIgnoreCase)
        {
            found = true;
            break;
        }
    }
    if (!found)
        throw new Exception("Searched string not found.");

    //we have the data, write to a new file
    using (StreamWriter sw = new StreamWriter(
        new FileStream("out.txt", FileMode.OpenOrCreate, //create or overwrite
            FileAccess.Write, FileShare.None))) // write only, no sharing
    {
        //write the line that we found in the search
        sw.WriteLine(cSearch);

        string cline = null;
        while ((cline = srInput.ReadLine()) != null)
            sw.WriteLine(cline);
    }
}

//both files are closed and complete