Question

I have large log files which contains timestamps every one second.what I need is to cut a user defined part from this huge file and save it in another text file..i am confused since the fstream class can deal with a max file size of 2GB and reading all lines is time and memory disaster.

timestamp pattern : !<< dd.mm.yyyy hh:min:sec> every second and one per line .
one prof. guy suggested using LINQ and readline().

a sample of the file :

!<<14.12.2012 16:20:03>
some text some text some 
some text some text some 
some text some text some 
!<<14.12.2012 16:20:04>
some text some text some 
some text some text some 
some text some text some 
some text some text some 
some text some text some 
!<<14.12.2012 16:20:05>
some text some text some
!<<14.12.2012 16:20:06>
some text some text some 
some text some text some 

and so on till EOF.

Answer 1

ReadLine is not at all what you want to do… open a file reader… seek to the position you want, read the data out you want (into another file stream).

“ReadLine” has to actually read the data… whereas seeking (myStream.Position = whereIWantToGo) is basically instant.

You would handle this the same way you would a sorted database. A DB with 1,000,000 records only takes 20 “seek” operations to find… start halfway, too high? just saved 500,000 seeks… come back halfway… too high? just shaved off 250,000 more seeks… rinse, repeat.

If you find funny characters (bad encoding)

Per your email (btw – you should really continue to use S.O., not email – that way other people can benefit)… The answer is that you need to try different encoding types. Your file may not be encoded UTF8 (which is what my code below is expecting). So, use new StreamReader("MyLogFile.txt", Encoding.ASCII), or some other encoding until it works for you.

C# console app that should get you started

Disclaimer… this code is nasty, and might have bugs where there is an infinite loop :)… but, here is a console app that should work for you.

using System;
using System.Collections.Generic;
using System.Globalization;
using System.IO;

namespace ConsoleApplication1
{
    class Program
    {
        static void Main(string[] args)
        {
            // example dates
            var lookFor = new DateTime(2012, 12, 14, 16, 20, 02);
            var readUntilDate = new DateTime(2012, 12, 14, 16, 20, 05);

            using (var stream = File.OpenText("MyLogFile.txt"))
            {
                if (SeekToEntry(stream, lookFor) == false)
                {
                    Console.WriteLine("Could not find entry for date {0}", lookFor);
                    return;
                }

                foreach (var line in ReadEntriesUntil(stream, readUntilDate))
                {
                    Console.WriteLine("Line: {0}", line);
                }
            }
        }

        // This method simply spits out one line at a time until it hits
        // the target cut-off.
        static IEnumerable<string> ReadEntriesUntil(StreamReader stream, DateTime target)
        {
            while (true)
            {
                string line = stream.ReadLine();

                if (line == null)
                {
                    break;
                }

                if (line.StartsWith("!<<"))
                {
                    DateTime entryDate;

                    if (DateTime.TryParseExact(line.Substring(3, 19).Replace(".", ""), @"ddMMyyyy HH:mm:ss",
                        CultureInfo.InvariantCulture, DateTimeStyles.None, out entryDate))
                    {
                        if (entryDate >= target)
                        {
                            break;
                        }
                    }
                }

                yield return line;
            }
        }

        // This method will bounce around the stream till it finds your
        // target entry date.
        static bool SeekToEntry(StreamReader stream, DateTime target)
        {
            long from = 0;
            long to = stream.BaseStream.Length;

            while (true)
            {
                long testIndex = (to - from) / 2;

                stream.BaseStream.Seek(testIndex, SeekOrigin.Begin);

                var entryDate = GetNextEntryDate(stream, out testIndex);

                if (entryDate == null || (from == to))
                {
                    return false;
                }

                switch (entryDate.Value.CompareTo(target))
                {
                    case -1:
                        // Found too low...
                        from = testIndex;
                        break;

                    case 1:
                        // Fount too high...
                        to = testIndex;
                        break;

                    default: return true;
                }
            }
        }

        // This is a function that is meant to keep seeking forward until
        // it hits an entry date.
        static DateTime? GetNextEntryDate(StreamReader stream, out long actualIndex)
        {
            actualIndex = stream.BaseStream.Position;
            DateTime? result = null;
            string line = null;

            // Find the next entry.
            while ((line = stream.ReadLine()) != null && line.StartsWith("!<<") == false) ;

            if (line != null)
            {
                actualIndex = stream.BaseStream.Position - line.Length;

                DateTime timeStamp;

                if (DateTime.TryParseExact(line.Substring(3, 19).Replace(".", ""), @"ddMMyyyy HH:mm:ss",
                    CultureInfo.InvariantCulture, DateTimeStyles.None, out timeStamp))
                {
                    result = timeStamp;
                }
            }

            return result;
        }
    }
}