we have a project that constitutes a large archive of image files…
We try to split them into sub-folders within the main archive folder.
Each sub-folder contains up to 2500 files in it.
For example:
C:\Archive
C:\Archive\Animals\
C:\Archive\Animals\001 - 2500 files...
C:\Archive\Animals\002 - 2300 files..
C:\Archive\Politics\
C:\Archive\Politics\001 - 2000 files...
C:\Archive\Politics\002 - 2100 files...
Etc… What would be the best way of storing files in such way under Windows ? and why exactly, please … ?
- Later on, the files have their EXIF metadata extracted and indexed for keywords, to be added into a Lucene index… (this is done by a Windows service that lives on the server)
We have an application where we try to make sure we don’t store more than around 1000 files in a directory. Under Windows at least, we noticed extreme degradation in performance over this number. The folder can theoretically store up to 4,294,967,295 in Windows 7. Note that because the OS does a scan of the folder, doing lookups and lists very quickly degrades as you add many more files. Once we got to 100,000 files in a folder it was almost completely unusable.
I’d recommend breaking down the animals even further, perhaps by first letter of name. Same with the other files. This will let you separate things out more so you won’t have to worry about the directory performance. Best advice I can give is to perform some stress tests on your system to see where the performance starts to tail off once you have enough files in a directory. Just be aware you’ll need several thousand files to test this out.