Question

I am parsing very large files (Unicode – Delphi 2009), and I have a very efficient routine for doing so using PChar variables as outlined in the Stackoverflow question: What is the fastest way to Parse a line in Delphi?

Everything was working great until I ran into a file that had some embedded hex:00 characters in it. This character signals the end of a PChar string and my parsing stops at that point.

However, when you load the file, as in:

FileStream := TFileStream.Create(Filename, fmOpenRead or fmShareDenyWrite);
Size := FileStream.Size;

then you find that the size of the file is much larger. If you open the file with Notepad, it loads to the end of the file, not stopping at the first hex:00 as the PChar does.

How can I read to the end of the file while still using PChar parsing without slowing down my reading/parsing too much?

Answer 1

The accepted code in your other question is breaking out when it reaches a #0 character. To fix that you just need to save the length of the input and check that instead. The updated code would look something like this:

type
  TLexer = class
  private
    FData: string;
    FTokenStart: PChar;
    FCurrPos: PChar;
    FEndPos: PChar;                                         // << New
    function GetCurrentToken: string;
  public
    constructor Create(const AData: string);
    function GetNextToken: Boolean;
    property CurrentToken: string read GetCurrentToken;
  end;

{ TLexer }

constructor TLexer.Create(const AData: string);
begin
  FData := AData;
  FCurrPos := PChar(FData);
  FEndPos := FCurrPos + Length(AData);                      // << New
end;

function TLexer.GetCurrentToken: string;
begin
  SetString(Result, FTokenStart, FCurrPos - FTokenStart);
end;

function TLexer.GetNextToken: Boolean;
var
  cp: PChar;
begin
  cp := FCurrPos; // copy to local to permit register allocation

  // skip whitespace
  while (cp <> FEndPos) and (cp^ <= #32) do                 // << Changed
    Inc(cp);

  // terminate at end of input
  Result := cp <> FEndPos;                                  // << Changed

  if Result then
  begin
    FTokenStart := cp;
    Inc(cp);
    while (cp <> FEndPos) and (cp^ > #32) do                // << Changed
      Inc(cp);
  end;

  FCurrPos := cp;
end;