Question

We are designing an update to a current system (C++\CLI and C#).
The system will gather small (~1Mb) amounts of data from ~10K devices (in the near future). Currently, they are used to save device data in a CSV (a table) and store all these in a wide folder structure.

Data is only inserted (create / append to a file, create folder) never updated / removed.
Data processing is done by reading many CSV’s to an external program (like Matlab). Mainly be used for statistical analysis.

There is an option to start saving this data to an MS-SQL database.
Process time (reading the CSV’s to external program) could be up to a few minutes.

• How should we choose which method to use?
• Does one of the methods take significantly more storage than the other?
• Roughly, when does reading the raw data from a database becomes quicker than reading the CSV’s? (10 files, 100 files? …)

I’d appreciate your answers, Pros and Cons are welcome.

Thank you for your time.

Answer 1

Well if you are using data in one CSV to get data in another CSV I would guess that SQL Server is going to be faster than whatever you have come up with. I suspect SQL Server would be faster in most cases, but I can’t say for sure. Microsoft has put a lot of resources into make a DBMS that does exactly what you are trying to do.

Based on your description it sounds like you have almost created your own DBMS based on table data and folder structure. I suspect that if you switched to using SQL Server you would probably find a number of areas where things are faster and easier.

Possible Pros:

• Faster access
• Easier to manage
• Easier to expand should you need to
• Easier to enforce data integrity
• Easier to design more complex relationships

Possible Cons:

• You would have to rewrite your existing code to use SQL Server instead of your current system
• You may have to pay for SQL Server, you would have to check to see if you can use Express

Good luck!