Home/ Questions/Q 113719
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-11T02:44:48+00:00

There are some text files(Records) which i need to access using C#.Net. But the

  • 0

There are some text files(Records) which i need to access using C#.Net. But the matter is those files are larger than 1GB. (minimum size is 1 GB)

what should I need to do? What are the factors which I need to be concentrate on?

Can some one give me an idea to over come from this situation.

EDIT:

Thanks for the fast responses. yes they are fixed length records. These text files coming from a local company. (There last month transaction records)

Is it possible to access these files like normal text files (using normal file stream).

and

How about the memory management????

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Expanding on CasperOne’s answer

    Simply put there is no way to reliably put a 100GB file into memory at one time. On a 32 bit machine there is simply not enough addressing space. In a 64 bit machine there is enough addressing space but during the time in which it would take to actually get the file in memory, your user will have killed your process out of frustration.

    The trick is to process the file incrementally. The base System.IO.Stream() class is designed to process a variable (and possibly infinite) stream in distinct quantities. It has several Read methods that will only progress down a stream a specific number of bytes. You will need to use these methods in order to divide up the stream.

    I can’t give more information because your scenario is not specific enough. Can you give us more details or your record delimeters or some sample lines from the file?

    Update

    If they are fixed length records then System.IO.Stream will work just fine. You can even use File.Open() to get access to the underlying Stream object. Stream.Read has an overload that requests the number of bytes to be read from the file. Since they are fixed length records this should work well for your scenario.

    As long as you don’t call ReadAllText() and instead use the Stream.Read() methods which take explicit byte arrays, memory won’t be an issue. The underlying Stream class will take care not to put the entire file into memory (that is of course, unless you ask it to 🙂 ).

    • 0