Question

i want to process a text file line by line. In the olden days i loaded the file into a StringList:

slFile := TStringList.Create();
slFile.LoadFromFile(filename);

for i := 0 to slFile.Count-1 do
begin
   oneLine := slFile.Strings[i];
   //process the line
end;

Problem with that is once the file gets to be a few hundred megabytes, i have to allocate a huge chunk of memory; when really i only need enough memory to hold one line at a time. (Plus, you can’t really indicate progress when you the system is locked up loading the file in step 1).

The i tried using the native, and recommended, file I/O routines provided by Delphi:

var
   f: TextFile;
begin
   Reset(f, filename);
   while ReadLn(f, oneLine) do
   begin
       //process the line
   end;

Problem withAssign is that there is no option to read the file without locking (i.e. fmShareDenyNone). The former stringlist example doesn’t support no-lock either, unless you change it to LoadFromStream:

slFile := TStringList.Create;
stream := TFileStream.Create(filename, fmOpenRead or fmShareDenyNone);
   slFile.LoadFromStream(stream);
stream.Free;

for i := 0 to slFile.Count-1 do
begin
   oneLine := slFile.Strings[i];
   //process the line
end;

So now even though i’ve gained no locks being held, i’m back to loading the entire file into memory.

Is there some alternative to Assign/ReadLn, where i can read a file line-by-line, without taking a sharing lock?

i’d rather not get directly into Win32 CreateFile/ReadFile, and having to deal with allocating buffers and detecting CR, LF, CRLF‘s.

i thought about memory mapped files, but there’s the difficulty if the entire file doesn’t fit (map) into virtual memory, and having to maps views (pieces) of the file at a time. Starts to get ugly.

i just want Reset with fmShareDenyNone!

Answer 1

With recent Delphi versions, you can use TStreamReader. Construct it with your file stream, and then call its ReadLine method (inherited from TTextReader).

An option for all Delphi versions is to use Peter Below’s StreamIO unit, which gives you AssignStream. It works just like AssignFile, but for streams instead of file names. Once you’ve used that function to associate a stream with a TextFile variable, you can call ReadLn and the other I/O functions on it just like any other file.