Question

I have an application which accesses a flat file filesystem whose structure I have to decide. My filesystem stores data for users for each day of the year for every year and each day’s data has information of the form HH:mm, value. My data access is usually limited to about 1 month at a time. My directory structure options are,

1. FileSystem > UserA > 2010_01_01, 2010_01_02, i.e., for each user, store each day’s data as a file. and,

2. FileSystem > UserA > 2010_01, 2010_02, i.e., for each user, store a months data as a file.

using 1 would mean when I fetching user data I have more frequent file open/close and fewer file reads(to get to the line I need) while using 2 implies less frequent file open/close and more file reads(to get to the line I need).
What would I be better off using?

Note: I realize with 1 that it is possible for me to access different files in different threads and probably speeding things up.

Answer 1

Consider that each file open is actually several reads, and that large reads are not much more expensive than small reads. Also, hard drives are physically limited to accessing a single part of the disk at a time, so multiple threads will likely not help you unless each thread reads from a different disk.