Question

I have a binary file specification that describes a packetized data structure. Each data packet has a two-byte sync pattern, so scanning for the beginning of a packet is possible, using a BinaryReader and FileStream combination:

while(!reader.EndOfFile)
{
    // Check for sync pattern.
    if (reader.ReadUInt16() != 0xEB25)
    {
        // Move to next byte.
        reader.BaseStream.Seek(-1, SeekOrigin.Current);
        continue;
    }

    // If we got here, a sync pattern was found.
}

This process works perfectly fine in the forward direction, but similar code scanning in the reverse direction is at least two orders of magnitude slower:

while(!reader.BeginningOfFile)
{
    // Check for sync pattern.
    if (reader.ReadUInt16() != 0xEB25)
    {
        // Move to previous byte.
        reader.BaseStream.Seek(-3, SeekOrigin.Current);
        continue;
    }

    // If we got here, a sync pattern was found.
}

I’ve tried a few workarounds, like moving back by an arbitrary amount (currently 1 megabyte) and scanning forward, but it’s becoming clear that what I really need is a BinaryReader or FileStream that is modified to have adequate performance characteristics when reading in both forward and reverse directions.

I already have a FastFileStream which improves forward read performance by subclassing an ordinary FileStream and caching the Position and Length properties (it also provides the BeginningOfFile and EndOfFile properties). That’s what drives the reader variable in the code above.

Is there something similar I could do to improve reverse reading performance, perhaps by incorporating a MemoryStream as a buffer?

Answer 1

L.B mentioned in a comment to use a Memory Mapped file, you may be impressed with the performance.

Please try something like this:

var memoryMapName = Path.GetFileName(fileToRead);

using (var mapStream = new FileStream(fileToRead, FileMode.Open))
{
    using (var myMap = MemoryMappedFile.CreateFromFile(
                            mapStream, 
                            memoryMapName, mapStream.Length,
                            MemoryMappedFileAccess.Read, null, 
                            HandleInheritability.None, false))
    {                    
        long leftToRead = mapStream.Length;
        long mapSize = Math.Min(1024 * 1024, mapStream.Length);
        long bytesRead = 0;
        long mapOffset = Math.Max(mapStream.Length - mapSize, 0);

        while (leftToRead > 1)
        {
            using (var FileMap = myMap.CreateViewAccessor(mapOffset, 
                                 mapSize, MemoryMappedFileAccess.Read))
            {
                long readAt = mapSize - 2;
                while (readAt > -1)
                {
                    var int16Read = FileMap.ReadUInt16(readAt);
                    //0xEB25  <--check int16Read here                            
                    bytesRead += 1;
                    readAt -= 1;
                }
            }

            leftToRead = mapStream.Length- bytesRead;
            mapOffset = Math.Max(mapOffset - mapSize, 0);
            mapSize = Math.Min(mapSize, leftToRead);
        }
    }
}