Question

I need regex help to create a delphi function to replace the HyperString ParseWord function in Rad Studio XE2. HyperString was a very useful string library that never made the jump to Unicode. I’ve got it mostly working but it doesn’t honor quote delimiters at all. I need it to be an exact match for the function described below:

function ParseWord(const Source,Table:String;var Index:Integer):String;

Sequential, left to right token parsing using a table of single
character delimiters. Delimiters within quoted strings are ignored.
Quote delimiters are not allowed in Table.

Index is a pointer (initialize to ‘1’ for first word) updated by the
function to point to next word. To retrieve the next word, simply
call the function again using the prior returned Index value.

Note: If Length(Resultant) = 0, no additional words are available.
Delimiters within quoted strings are ignored. (my emphasis)

This is what I have so far:

function ParseWord( const Source, Table: String; var Index: Integer):string;
var
  RE : TRegEx;
  match : TMatch;
  Table2,
  chars : string;
begin
  if index = length(Source) then
  begin
    result:= '';
    exit;
  end;

  // escape the special characters and wrap in a Group
  Table2 :='['+TRegEx.Escape(Table, false)+']';
  RE := TRegEx.create(Table2);
  match := RE.Match(Source,Index);
  if match.success then
  begin
    result := copy( Source, Index, match.Index - Index);
    Index := match.Index+match.Length;
  end
  else
  begin
    result := copy(Source, Index, length(Source)-Index+1);
    Index := length(Source);
  end;
end;

  while ( Length(result)= 0) and (Index<length(Source)) do
  begin
    Inc(Index);
    result := ParseWord(Source,Table, Index);
  end;

cheers and thanks.

Answer 1

In my original code I was looking for the delimiter and taking everything up to that as my next match, but that concept didn’t carry over when looking for something within quotes. @Stephan’s suggestion of negating the search eventually lead me to something that works. An additional complication that I never mentioned earlier is that HyperStr can use anything as a quoting character. The default is double quote but you can change it with a function call.

In my solution I’ve explicitly hardcoded the QuoteChar as double quote, which suits my own purposes, but it would be trivial to make QuoteChar a global and set it within another function. I’ve also successfully tested it with single quote (ascii 39), which would be the tricky one in Delphi.

function ParseWord( const Source, Table: String; var Index: Integer):string;
var
  RE : TRegEx;
  match : TMatch;
  Table2: string;
  Source2 : string;
  QuoteChar : string;
begin
  if index = length(Source) then
  begin
    result:= '';
    exit;
  end;

  // escape the special characters and wrap in a Group
  QuoteChar := #39;
  Table2 :='[^'+TRegEx.Escape(Table, false)+QuoteChar+']*|'+QuoteChar+'.*?'+QuoteChar ;
  Source2 := copy(Source, Index, length(Source)-index+1);
  match := TRegEx.Match(Source2,Table2);
  if match.success then
  begin
    result := copy( Source2, match.index, match.length);
    Index := Index + match.Index + match.Length-1;
  end
  else
  begin
    result := copy(Source, Index, length(Source)-Index+1);
    Index := length(Source);
  end;
  while ( Length(result)= 0) and (Index<length(Source)) do
  begin
    Inc(Index);
    result := ParseWord(Source,Table, Index);
  end;

end;

This solution doesn’t strip the quote chars from around quoted strings, but I can’t tell from my own existing code if it should or not, and I can’t test using Hyperstr. Maybe someone else knows?