Question

I’m looking for a very fast XML parser for Delphi, for very simple data.

Consider the following kind of data:

<node>
    <datatype1>randomdata</datatype1>
    <datatype2>randomdata</datatype2>
    <datatype3>randomdata</datatype3>
    <datatype4>randomdata</datatype4>
    <datatype5>randomdata</datatype5>
    <datatype6>randomdata</datatype6>
    <datatype7>randomdata</datatype7>
    <datatype8>randomdata</datatype8>
    <datatype9>randomdata</datatype9>
    <datatype10>randomdata</datatype10>
    <datatype11>randomdata</datatype11>
    <datatype12>randomdata</datatype12>
    <datatype13>randomdata</datatype13>
    <datatype14>randomdata</datatype14>
    <datatype15>randomdata</datatype15>
    <datatype16>randomdata</datatype16>
    <datatype17>randomdata</datatype17>
    <datatype18>randomdata</datatype18>
    <datatype19>randomdata</datatype19>
    <datatype20>randomdata</datatype20>
</node>

Copy this 10000 times (datatypes and the data being obviously different in a real scenario). Consider also the data contains Unicode.

This will be parsed and loaded into an Array of records like

Type MyData = record
  d1,d2,d3,d4,d5,
  d6,d7,d8,d9,d10,
  d11,d12,d13,d14,d15,
  d16,d17,d18,d19,d20: string;
end;

I wrote a custom parser for this, which in my computer takes approx. 115ms for the entire process, from loading the file to having the 10,000 records filled.

So I am looking for something that can accomplish this faster.

Related questions:

Pos() within utf8 string boundaries

Delphi – Pos() with boundaries

Answer 1

First let me tell you that you’re optimizing the wrong thing here: unless you’re doing this for recreational purposes, then your approach is wrong. XML is not a difficult format but it does have it quirks and it takes it’s liberties. It’s a format designed for data exchange between foreign applications, so the emphasis needs to be put on COMPATIBILITY, not on SPEED! What good is a non-standard ultra-fast parser that gives the wrong result when confronted with a slightly altered XML file?

If you can find a XML parsing LIBRARY that’s guaranteed to be compatible with anything out there that can parse your data at HALF the speed your HDD can read it, then simply implement a producer-consumer multi-threaded application where one thread constantly reads the data from disk while the other two simply do the parsing. In the end you’ll only be limited by the speed of the HDD while maintaining compatibility. If you’re only looking for speed you’re liable to make mistakes, skip XML features, depend on certain particularities of the sample XML file you’re dealing with. Your application is likely to break for numerous reasons.

Remember that the most costly cycle for an application is MAINTENANCE, not production. What you might gain today by making a 50% faster thingy that’s 200% percent more difficult to maintain will be lost in a year or so, when computers get 50% faster (nulling your edge over the competition). Besides, there’s no point in exceeding natural limits for such processes, like the speed of the HDD. It’s irrelevant that you’re testing with a file from a RAM-drive – when the application goes into production it will be used with files from a HDD, and your application’s performance will be limited by the speed of your HDD.

---

Anyhow, I do like a challenge once in a while and I really like parsers. What follows is a very simple parser implementation that only looks at each character in the input string once and only copies stuff where needed: copies the name of the nodes in order to decide what to do next and copies the node’s “Payload” when appropriate, in order to push it into the array. On my “modest” i7 @ 3.4 Ghz parsing a string built by copying your sample data 10,000 times takes 63 ms. It clearly beats your time, BUT a word of warning, this code is fragile: it depends on having a XML file that’s a certain form. No way around that.

program Project28;

{$APPTYPE CONSOLE}

uses SysUtils, DateUtils, Windows;

const SampleData =
    '<node>'#13#10+
    '  <datatype1>randomdata</datatype1>'#13#10+
    '  <datatype2>randomdata</datatype2>'#13#10+
    '  <datatype3>randomdata</datatype3>'#13#10+
    '  <datatype4>randomdata</datatype4>'#13#10+
    '  <datatype5>randomdata</datatype5>'#13#10+
    '  <datatype6>randomdata</datatype6>'#13#10+
    '  <datatype7>randomdata</datatype7>'#13#10+
    '  <datatype8>randomdata</datatype8>'#13#10+
    '  <datatype9>randomdata</datatype9>'#13#10+
    '  <datatype10>randomdata</datatype10>'#13#10+
    '  <datatype11>randomdata</datatype11>'#13#10+
    '  <datatype12>randomdata</datatype12>'#13#10+
    '  <datatype13>randomdata</datatype13>'#13#10+
    '  <datatype14>randomdata</datatype14>'#13#10+
    '  <datatype15>randomdata</datatype15>'#13#10+
    '  <datatype16>randomdata</datatype16>'#13#10+
    '  <datatype17>randomdata</datatype17>'#13#10+
    '  <datatype18>randomdata</datatype18>'#13#10+
    '  <datatype19>randomdata</datatype19>'#13#10+
    '  <datatype20>randomdata</datatype20>'#13#10+
    '</node>'#13#10;
const NodeIterations = 10000;

type
  TDummyRecord = record
    D1, D2, D3, D4, D5, D6, D7, D8, D9, D10, D11, D12, D13,
      D14, D15, D16, D17, D18, D19, D20: string;
  end;
  TDummyRecordArray = array[1..NodeIterations] of TDummyRecord;

procedure ParseDummyXMLToRecordArray(const InputText:string; var A: TDummyRecordArray);
var PInputText: PChar;
    cPos, TextLen: Integer;
    C: Char;
    State: Integer;

    tag_starts_at: Integer;
    last_payload_starts_at: Integer;
    FlagEndTag: Boolean;

    NodeName, Payload: string;

    cNode: Integer;

const st_not_in_node = 1;
      st_in_node = 2;
begin
  cPos := 1;
  TextLen := Length(InputText);
  PInputText := @InputText[1];
  State := st_not_in_node;
  last_payload_starts_at := 1;
  cNode := 0;

  // This is the lexer/parser loop. It's a finite-state machine with only
  // two states: st_not_in_node and st_in_node
  while cPos < TextLen do
  begin
    C := PInputText[cPos-1];
    case State of

      // What happens when we're NOT currently inside a node?
      // Not much. We only jump to st_in_node if we see a "<"
      st_not_in_node:
        case C of
          '<':
            begin
              // A node starts here. Switch state and set up some simple
              // flags.
              state := st_in_node;
              tag_starts_at := cPos + 1;
              FlagEndTag := False;
            end;
        end;

      // What happens while inside a node? Again, not much. We only care about
      // the "/" - as it signals an closing tag, and we only care about the
      // ">" because that means the end of the ndoe.
      st_in_node:
        case C of
          '/': FlagEndTag := True;
          '>':
            begin
              // This is where the magic haepens. We're in one of possibly two states:
              // We're ither seeing the first <name> of a pair, or the second </name>
              //
              if FlagEndTag then
                begin
                  // This is the closing pair of a tag pair, ie, it's the </NodeName> What we'll do
                  // depends on what node is closing, so we retreive the NodeName:
                  NodeName := System.Copy(InputText, tag_starts_at+1, cPos - tag_starts_at-1);
                  if NodeName <> 'node' then // SAMPLE-DATA-SPECIFIC: I know I don't care about "node" tags.
                  begin
                    // SAMPLE-DATA-SPECIFIC: I know there are only two kinds of nodes:
                    // "node" and "datatypeN". I retreive the PAYLOAD for the node because
                    // I know it's not "ndoe" and I know I'll need it.
                    Payload := System.Copy(InputText,last_payload_starts_at, tag_starts_at - last_payload_starts_at -1);
                    // Make sure we're dealing with a valid node
                    if (cNode > 0) and (cNode <= High(A)) then
                      begin
                        // Based on NodeName, copy the Payload into the appropriate field.
                        if NodeName = 'datatype1' then A[cNode].D1 := Payload
                        else if NodeName = 'datatype2' then A[cNode].D2 := Payload
                        else if NodeName = 'datatype3' then A[cNode].D3 := Payload
                        else if NodeName = 'datatype4' then A[cNode].D4 := Payload
                        else if NodeName = 'datatype5' then A[cNode].D5 := Payload
                        else if NodeName = 'datatype6' then A[cNode].D6 := Payload
                        else if NodeName = 'datatype7' then A[cNode].D7 := Payload
                        else if NodeName = 'datatype8' then A[cNode].D8 := Payload
                        else if NodeName = 'datatype9' then A[cNode].D9 := Payload
                        else if NodeName = 'datatype10' then A[cNode].D10 := Payload
                        else if NodeName = 'datatype11' then A[cNode].D11 := Payload
                        else if NodeName = 'datatype12' then A[cNode].D12 := Payload
                        else if NodeName = 'datatype13' then A[cNode].D13 := Payload
                        else if NodeName = 'datatype14' then A[cNode].D14 := Payload
                        else if NodeName = 'datatype15' then A[cNode].D15 := Payload
                        else if NodeName = 'datatype16' then A[cNode].D16 := Payload
                        else if NodeName = 'datatype17' then A[cNode].D17 := Payload
                        else if NodeName = 'datatype18' then A[cNode].D18 := Payload
                        else if NodeName = 'datatype19' then A[cNode].D19 := Payload
                        else if NodeName = 'datatype20' then A[cNode].D20 := Payload
                        else
                          raise Exception.Create('Unknown node: ' + NodeName);
                      end
                    else
                      raise Exception.Create('cNode out of bounds.');
                  end;
                  // Repeat :-)
                  state := st_not_in_node;
                end
              else
                begin
                  // Node start. Retreive node name. I only care about the start of the "NODE" - if I see that
                  // I'll increment the current node counter so I'll go on filling the next position in the array
                  // with whatever I need.
                  NodeName := System.Copy(InputText, tag_starts_at, cPos - tag_starts_at);
                  last_payload_starts_at := cPos+1;
                  if NodeName = 'node' then Inc(cNode);
                  state := st_not_in_node;
                end;
            end;
        end;
    end;
    Inc(cPos);
  end;
end;

var DataString: string;
    SB: TStringBuilder;
    i: Integer;
    DummyArray: TDummyRecordArray;
    T1, T2, F: Int64;

begin
  try
    try
      // Prepare the sample string; 10.000 iterations of the sample data.
      SB := TStringBuilder.Create;
      try
        for i:=1 to NodeIterations do
          SB.Append(SampleData);
        DataString := SB.ToString;
      finally SB.Free;
      end;

      // Invoke the simple parser using the string constant.
      QueryPerformanceCounter(T1);

      ParseDummyXMLToRecordArray(DataString, DummyArray);

      QueryPerformanceCounter(T2);
      QueryPerformanceFrequency(F);
      WriteLn(((T2-T1) * 1000) div F);

      // Test parse validity.
      for i:=1 to NodeIterations do
      begin
        if DummyArray[i].D1 <> 'randomdata' then raise Exception.Create('Bug. D1 doesn''t have the proper value, i=' + IntToStr(i));
        if DummyArray[i].D2 <> 'randomdata' then raise Exception.Create('Bug. D2 doesn''t have the proper value, i=' + IntToStr(i));
        if DummyArray[i].D3 <> 'randomdata' then raise Exception.Create('Bug. D3 doesn''t have the proper value, i=' + IntToStr(i));
        if DummyArray[i].D4 <> 'randomdata' then raise Exception.Create('Bug. D4 doesn''t have the proper value, i=' + IntToStr(i));
        if DummyArray[i].D5 <> 'randomdata' then raise Exception.Create('Bug. D5 doesn''t have the proper value, i=' + IntToStr(i));
        if DummyArray[i].D6 <> 'randomdata' then raise Exception.Create('Bug. D6 doesn''t have the proper value, i=' + IntToStr(i));
        if DummyArray[i].D7 <> 'randomdata' then raise Exception.Create('Bug. D7 doesn''t have the proper value, i=' + IntToStr(i));
        if DummyArray[i].D8 <> 'randomdata' then raise Exception.Create('Bug. D8 doesn''t have the proper value, i=' + IntToStr(i));
        if DummyArray[i].D9 <> 'randomdata' then raise Exception.Create('Bug. D9 doesn''t have the proper value, i=' + IntToStr(i));
        if DummyArray[i].D10 <> 'randomdata' then raise Exception.Create('Bug. D10 doesn''t have the proper value, i=' + IntToStr(i));
        if DummyArray[i].D11 <> 'randomdata' then raise Exception.Create('Bug. D11 doesn''t have the proper value, i=' + IntToStr(i));
        if DummyArray[i].D12 <> 'randomdata' then raise Exception.Create('Bug. D12 doesn''t have the proper value, i=' + IntToStr(i));
        if DummyArray[i].D13 <> 'randomdata' then raise Exception.Create('Bug. D13 doesn''t have the proper value, i=' + IntToStr(i));
        if DummyArray[i].D14 <> 'randomdata' then raise Exception.Create('Bug. D14 doesn''t have the proper value, i=' + IntToStr(i));
        if DummyArray[i].D15 <> 'randomdata' then raise Exception.Create('Bug. D15 doesn''t have the proper value, i=' + IntToStr(i));
        if DummyArray[i].D16 <> 'randomdata' then raise Exception.Create('Bug. D16 doesn''t have the proper value, i=' + IntToStr(i));
        if DummyArray[i].D17 <> 'randomdata' then raise Exception.Create('Bug. D17 doesn''t have the proper value, i=' + IntToStr(i));
        if DummyArray[i].D18 <> 'randomdata' then raise Exception.Create('Bug. D18 doesn''t have the proper value, i=' + IntToStr(i));
        if DummyArray[i].D19 <> 'randomdata' then raise Exception.Create('Bug. D19 doesn''t have the proper value, i=' + IntToStr(i));
        if DummyArray[i].D20 <> 'randomdata' then raise Exception.Create('Bug. D20 doesn''t have the proper value, i=' + IntToStr(i));
      end;

    except on E: Exception do Writeln(E.ClassName, ': ', E.Message);
    end;
  finally
    WriteLn('ENTER to Exit');
    ReadLn;
  end;
end.