We receive multiple thousands of flat files per week currently, and I have a system that runs reports on these and exports them to PDF for our people to process and reference.
I currently bulk load these into a database, make sure all fields/formatting is valid, export them, and truncate the tables on the next run.
What I’m wondering is what everyone thinks would be the most space efficient way to store possibly 6 months of this bulk load plain text data?
Either in the form of daily SQL backups, or zipped archives, or whatever, so I always had the ability to reload old data for trouble shooting.
Any ideas are welcome, I’m open to any suggestions.
So, you bulk-load flat files of raw data, you use SQL Server 2005 to process them and get a separate bunch of processed flat files, and then dump the data?
Well, if this is correct, SQL backups won’t help since you seem to be saying the data doesn’t stay in the DB. Your only option is efficient compression of the input and/or output files coupled with good organization of the batches in directories.
I would recommend an aggressive compression program, that has scheduled batch functionality, but be careful to not get to esoteric with the program you use for the sake of avoiding being locked in to one program…