Home/ Questions/Q 6360843
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-24T23:42:00+00:00

I have a stack of data in 200,000+ XML files, which are updated/created every

  • 0

I have a stack of data in 200,000+ XML files, which are updated/created every week. So, each week, I have to parse each XML file, check whether there are new XML files being created, then update my database with all updated data. In between, if there is new XML file, then my database will not have that record, so I have to create a new row.

Here’s my workflow plan:

  1. Put all table X row ID into array A. Let’s say 200,000 numerical values.
  2. Parse each XML files and gather each XML’s ID (ID will be the same as my table X ID), and store in array B. Let’s say now I have 200,010 numerical values, with 10 new records compared to my current table X.
  3. Compare array A and array B to see which values do not exist.
  4. Put the 10 new values into array C.
  5. Create new record in table X with that 10 new IDs from array C.
  6. Parse each XML file again, and store the desired values to my table X row-column.
  7. So now my table X will have 200,010 records, and each are updated, and the new 10 records will also now in the table X.

The problem I have to do this is because I can’t get any information of any new XML file created by the vendor. They just give me a stack of files.

Any better way to do it? I’m worried that my system will crash when they compare two arrays with 200,000+ values. Thanks.

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    I had to do something similar. In the end I did it like this:

    1. Baseline the whole setup by iterating over each record in all XML files, normalizing the item (removing newlines, cleaning up whitespace, substituting certain characters) and then doing a per record MD5 sum. Also import the record.

    2. When I get new data, I iterate through the records (SAX would be a good idea), if the record is not yet in the DB (based on UID) or has changed (based on MD5 sum) it gets imported.

    This works pretty well for what we usually need it for (around 350k records spread through around 100 files), but also worked ok-ish with (much) more data. It’s a wild mix of several tools, including Bash, AWK, sed, grep, the wonderful XMLStarlet and Ruby, and would be in dire need of a proper rewrite.

    • 0