Question

I receive more than 1 million rows per hour, I need a way to save them for later processing.

Write performance is the most important factor here but I don’t know which is the best way to handle this much data.

Should I save to XML, or will a SQL Server DB perform that fast?

Should I split the XML over multiple files so that I can process them in batches or should I have just one very large file?

I would ideally like to use a SQL Server DB because it will make the processing easier, however I do not know if it will be able to handle the amount of data I will be dealing with.

The data I am trying to save is:

// TODO: Data Structure here

Answer 1

I usually like databases over writing files, espcially if you are getting that many rows in an hour. I assume it would be 1 million separate files an hour?

Databases are pretty flexible and are a lot easier to get data from vs a bajillion xml files.

If the database ever gets too big, where you are afraid of performance, you can archive some of the older data and remove it from the main database.