I’m looking to build a server with lots of tiny files delivered by an XML API. It won’t be doing a whole lot of iterating over directories or blocks of sequential files–we’re talking lots and lots of seeks for discontinuous data.
Will seek time on BSD UFS degrade over time for requests for individual files? I understand that the filesystem’s inode limit is based on the size of the partition/slice, but the hard drive has to step through the inode table for every file request before it can discover the location of the data. What filesystem yields the best performance for seek time?
The alternative is to setup 2-4GB ‘blob’ files and have a separate system of seeking a file contained in them from within the software. The software’s ‘inode table’ could be optimized for delivery based on currently logged in user, etc… These ‘inode tables’ would likely be cached in RAM and would only relate to the users currently logged in so that there are fewer wasted resources.
Where do these two solutions rate on a scalability and maintenance standpoint? What sort of performance gains, if any, could I expect by using the second solution?
The most obvious and time-proven mitigation technique is to use a good hierarchical design for directories (and pathname search strategies), and have more directories with fewer files in each.