Question

I have a process that’s going to initially generate 3-4 million PDF files, and continue at the rate of 80K/day. They’ll be pretty small (50K) each, but what I’m worried about is how to manage the total mass of files I’m generating for easy lookup. Some details:

1. I’ll have some other steps to run once a file have been generated, and there will be a few servers participating, so I’ll need to watch for files as they’re generated.
2. Once generated, the files will be available though a lookup process I’ve written. Essentially, I’ll need to pull them based on an order number, which is unique per file.
3. At any time, an existing order number may be resubmitted, and the generated file will need to overwrite the original copy.

Originally, I had planned to write these files all to a single directory on a NAS, but I realize this might not be a good idea, since there are millions of them and Windows might not handle a million-file-lookup very gracefully. I’m looking for some advice:

1. Is a single folder okay? The files will never be listed – they’ll only be retrieved using a System.IO.File with a filename I’ve already determined.
2. If I do a folder, can I watch for new files with a System.IO.DirectoryWatcher, even with that many files, or will it start to become sluggish with that many files?
3. Should they be stored as BLOBs in a SQL Server database instead? Since I’ll need to retrieve them by a reference value, maybe this makes more sense.

Thank you for your thoughts!

Answer 1

I’d group the files in specific subfolders, and try to organize them (the subfolders) in some business-logic way. Perhaps all files made during a given day? During a six-hour period of each day? Or every # of files, I’d say a few 1000 max. (There’s probably an ideal number out there, hopefully someone will post it.)

Do the files ever age out and get deleted? If so, sort and file be deletable chunk. If not, can I be your hardware vendor?

There’s arguments on both sides of storing files in a database.

• On the one hand you get enhanced security, ’cause it’s more awkward to pull the files from the DB; on the other hand, you get potentially poorer performance, ’cause it’s more awkward to pull the files from the DB.
• In the DB, you don’t have to worry about how many files per folder, sector, NAS cluster, whatever–that’s the DB’s problem, and probably they’ve got a good implementation for this. On the flip side, it’ll be harder to manage/review the data, as it’d be a bazillion blobs in a single table, and, well, yuck. (You could partition the table based on the afore-mentioned business-logic, which would make deletion or archiving infinitely easier to perform. That, or maybe partitioned views, since table partitioning has a limit of 1000 partitions.)
• SQL Server 2008 has the FileStream data type; I don’t know much about it, might be worth looking into.

A last point to worry about is keeping the data “aligned”. If the DB stores the info on the file along with the path/name to the file, and the file gets moved, you could get totally hosed.