Question

My application creates pieces of data that, in xml, would look like this:

<resource url="someurl">
   <term>
      <name>somename</name>
      <frequency>somenumber</frequency>
   </term>    
   ...
   ...
   ...
</resource>

This is how I’m storing these “resources” now. A resource per XML file. As many “term” per “resource” as needed.
The problem is, I’ll need to generate about 2 million of these resources.
I’ve generated almost 500.000 and my mac isn’t very happy about it.
So my question is: how should I store this data?

• A database? that would be hard, because the structure of the data isn’t fixed…
• Maybe merge some resources into larger XML files?
• …?

I don’t need to change the data once it’s created.
Right now I’m accessing a specific resource by the name of that resource’s file.

Any suggestions are greatly appreciated!

Answer 1

Not all databases are relational. Have a look at for example mongodb. It stores your data as json-like objects, similar to your resources.

An example using the shell:

$ mongo
> db.resources.save({url: "someurl", 
                     terms: [{name: "name1", frequency: 17.0},
                             {name: "name2", frequency: 42.0}]})
> db.resources.find()
{"_id" :  ObjectId( "4b00884b3a77b8b2fa3a8f77"), 
 "url" : "someurl" , 
 "terms" : [{"name" : "name1" , "frequency" : 17},
            {"name" : "name2" , "frequency" : 42}]}