Question

In php and java there are explode and tokenizer function to convert a string into array without punctuations. Are are functions or some way in delphi to do the work. Suppose there is a large file ” This is, a large file with punctuations,, and spaces and numbers 123…” How can we get
array “This is a large file with punctuations and spaces and numbers 123”

Thank you very much in advance.

Yes, we want only [0..9],[a..z],[A..Z], like \w in regex. Can we use regex in Tperlregex to extract \w and put them in Tstringlist as if tstringlist is a array, but it may not be so efficient? Thank you.

Answer 1

If you need a function that takes a string and returns an array of strings, these strings being the substrings of the original separated by punctuation, as Eugene suggested in my previous answer, then you can do

type
  StringArray = array of string;
  IntegerArray = array of integer;
  TCharSet = set of char;

function split(const str: string; const delims: TCharSet): StringArray;
var
  SepPos: IntegerArray;
  i: Integer;
begin
  SetLength(SepPos, 1);
  SepPos[0] := 0;
  for i := 1 to length(str) do
    if str[i] in delims then
    begin
      SetLength(SepPos, length(SepPos) + 1);
      SepPos[high(SepPos)] := i;
    end;
  SetLength(SepPos, length(SepPos) + 1);
  SepPos[high(SepPos)] := length(str) + 1;
  SetLength(result, high(SepPos));
  for i := 0 to high(SepPos) -  1 do
    result[i] := Trim(Copy(str, SepPos[i] + 1, SepPos[i+1] - SepPos[i] - 1));
end;

Example:

const
  PUNCT = ['.', ',', ':', ';', '-', '!', '?'];

procedure TForm4.FormCreate(Sender: TObject);
var
  str: string;
begin
  for str in split('this, is, a! test!', PUNCT) do
    ListBox1.Items.Add(str)
end;