Question

Im having an interesting problem with reading a large file (~400 mb) that’s on a network drive. Orginally, I fed the full network address into a FileInfo and used the CopyTo function to transfer it to a local temp drive and then read it. This seems to work okay, its not slow but its not fast – just meh. The CopyTo function would get the computer running the program’s network utilization consistantly up above 50%, which is pretty good.

In order to speed up the process I tried to read the network file directly into a Memory Stream to cut out the middle man so to speak. When I tried this (using the asynchronous copy pattern described here), it is hilariously slow. My network utilization never even tops 2% – its almost like something is throttling me. FYI, I watched my network utilization when directly copying the same file via windows explorer and it hit like 80-90%… not sure what’s happening here. Below is the asynchronous copy code I used:

string line;
List<string> results = new List<string>();

Parser parser = new Parser(QuerySettings.SelectedFilters, QuerySettings.SearchTerms,
QuerySettings.ExcludedTerms, QuerySettings.HighlightedTerms);

byte[] ActiveBuffer = new byte[60 * 1024];
byte[] BackBuffer = new byte[60 * 1024];
byte[] WriteBuffer = new byte[60 * 1024];

MemoryStream memStream = new MemoryStream();
FileStream fileStream = new FileStream(fullPath, FileMode.Open, FileSystemRights.Read, FileShare.None, 60 * 1024, FileOptions.SequentialScan);

int Readed = 0;
IAsyncResult ReadResult;
IAsyncResult WriteResult;

ReadResult = fileStream.BeginRead(ActiveBuffer, 0, ActiveBuffer.Length, null, null);
do
{
    Readed = fileStream.EndRead(ReadResult);

    WriteResult = memStream.BeginWrite(ActiveBuffer, 0, Readed, null, null);
    WriteBuffer = ActiveBuffer;

    if (Readed > 0)
    {
        ReadResult = fileStream.BeginRead(BackBuffer, 0, BackBuffer.Length, null, null);
        BackBuffer = Interlocked.Exchange(ref ActiveBuffer, BackBuffer);
    }

    memStream.EndWrite(WriteResult);
}
while (Readed > 0);

StreamReader streamReader = new StreamReader(memStream);
while ((line = streamReader.ReadLine()) != null)
{
    if (parser.ParseResults(line))
    results.Add(line);
}

fileStream.Flush();
fileStream.Close();

memStream.Flush();
memStream.Close();

return results;

UPDATE
As per the comments I just tried the following. It only had my network utilization at about 10-15%… why so low?

MemoryStream memStream = new MemoryStream();
FileStream fileStream = File.OpenRead(fullPath);

fileStream.CopyTo(memStream);

memStream.Seek(0, 0);
StreamReader streamReader = new StreamReader(memStream);

Parser parser = new Parser(QuerySettings.SelectedFilters, QuerySettings.SearchTerms,
QuerySettings.ExcludedTerms, QuerySettings.HighlightedTerms);

while ((line = streamReader.ReadLine()) != null)
{
if (parser.ParseResults(line))
results.Add(line);
}

Answer 1

There is no point copying the whole file over and then parsing it. Simply open the file from the network drive and let the .Net Framework do it’s best to deliver the data for you. You can be more clever than MS developers and you may create a copy method faster than they do, but it’s really a challenge.