Question

I have IMDB collection from INEX, which consist few million XML files in few thousand directories with structure like this:

- actors
-- 000
--- person_1000.xml
--- ...
-- 001
--- person_1001.xml
--- ...
...
- movies
-- 000
--- 10000.xml
--- ...
...

I need to convert these files to TRECTEXT format, which is

<DOC>
 <DOCNO> document_number </DOCNO>  
 <TEXT>  XML file goes here. </TEXT>  
</DOC>

Where document_number should be file name without extension, e.x. person_1000.xml -> person_1000 and contents of XML file should be wrapped in tags.

I assume I need some script which wraps every XML file in collection in , and tags as shown above and overwrites original file. Could you help me please?

Answer 1

I’m not familiar with TRECTEXT format, but here’s a one-liner using Perl that should do what you want:

find . -type f -name "*.xml" -exec perl -MFile::Basename -0777pi.bak -e 's#(.*)\n#"<DOC>\n <DOCNO> ".basename($ARGV,".xml")." </DOCNO>\n <TEXT> ".$1." </TEXT>\n</DOC>\n"#es' {} \;

Obviously remove the .bak extension if you don’t wish to keep any backup files. Please let me know if you have any problems. Cheers.

---

Update, as per comments:

find . -type f -name "*.xml" -exec perl -MFile::Basename -0777pi.bak -e 's#.*#"<DOC>\n <DOCNO> ".basename($ARGV,".xml")." </DOCNO>\n <TEXT> ".$&." </TEXT>\n</DOC>\n"#es' {} \;