Question

I’m using php to take xml files and convert them into single line tab delimited plain text with set columns (i.e. ignores certain tags if database does not need it and certain tags will be empty). The problem I ran into is that it took 13 minutes to go through 56k (+ change) files, which I think is ridiculously slow. (average folder has upwards of a million xml files) I’ll probably cronjob it overnight anyways, but it is completely untestable at a reasonable pace while I’m at work for things like missing files and corrupt files and such.

Here’s hoping someone can help me make the thing faster, the xml files themselves are not too big (<1k lines) and I don’t need every single data tag, just some, here’s my data node method:

function dataNode ($entries) {
    $out = "";

    foreach ($entries as $e) {
        $out .= $e->nodeValue."[ATTRIBS]";
        foreach ($e->attributes as $name => $node)
            $out .= $name."=".$node->nodeValue;
    }

    return $out;
}

where $entries is a DOMNodeList generated from XPath queries for the nodes I need. So the question is, what is the fastest way to go to a target data node or nodes (if I have 10 keyword nodes from my XPath query then I need all of them to be printed from that function) and output the nodevalue and all it’s attributes?

I read here that iterating through a DOMNodeList isn’t constant time but I can’t really use the solution given because a sibling to the node I want might be one that I don’t need or need to call a different format function before I write it to file and I really don’t want to run the node through a gigantic switch statement for every iteration trying to format out the data.

Edit: I’m an idiot, I had my write function inside my processing loop so every iteration it had to reopen the file I was writing to, thanks for both of your help, I’m trying to learn XSLT right now as it seems very useful.

Answer 1

A comment would be a little short, so I write it as an answer:

It’s hard to say where actually your setup can benefit from optimizing. Perhaps it’s possible to join multiple of your many XML files together before loading.

From the information you give in your question I would assume that it’s more the disk operations that are taking the time than the XML parsing. I found DomDocument and Xpath quite fast even on large files. An XML file with up to 60 MB takes about 4-6 secs to load, a file of 2MB only a fraction.

Having many small files (< 1k) would mean a lot of work on the disk, opening / closing files. Additionally, I have no clue how you iterate over directories/files, sometimes this can be speed up dramatically as well. Especially as you say that you have millions of file nodes.

So perhaps concatenating/merging files is an option for you which can be run quite safe so to reduce the time to test your converter.

If you encounter missing or corrupt files, you should create a log and catch these errors. So you can let the job run through and check for errors later.

Additionally, if possible, you can try to make your workflow resumeable. E.g. if an error occurs, the current state is saved and next time you can continue at this state.

The suggestion above in a comment to run an XSLT on the files is a good idea as well to transform them first. Having a new layer in the middle to transpose data can help to reduce the overall problem dramatically as it can reduce complexity.

This workflow on XML files has helped me so far:

1. Preprocess the file (plain text filters, optional)
2. Parse the XML. That’s loading into DomDocument, XPath iterating etc.
3. My Parser sends out events with the parsed data if found.
4. The Parser throws a specific exception if data is encountered that is not in the expected format. That allows to realize errors in the own parser.
5. Every other errors are converted to Exceptions as well.
6. Exceptions can be caught and operations finished. E.g. go to next file etc.
7. Logger, Resumer and Exporter (file-export) can hook onto the events. Sort of the visitor pattern.

I’ve build such a system to process larger XML files which formats change. It’s flexible enough to deal with changes (e.g. replace the parser with a new version while keeping logging and exporting). The event system really pushed it for me.

Instead of a gigantic switch statement I normally use a $state variable for the parsers state while iterating over a domnodelist. $state can be handy to resume operations later. Restore the state and go to the last known position, then continue.