Question

Is there a better [pre-existing optional Java 1.6] solution than creating a streaming file reader class that will meet the following criteria?

• Given an ASCII file of arbitrary large size where each line is terminated by a \n
• For each invocation of some method readLine() read a random line from the file
• And for the life of the file handle no call to readLine() should return the same line twice

Update:

• All lines must eventually be read

Context: the file’s contents are created from Unix shell commands to get a directory listing of all paths contained within a given directory; there are between millions to a billion files (which yields millions to a billion lines in the target file). If there is some way to randomly distribute the paths into a file during creation time that is an acceptable solution as well.

Answer 1

If the number of files is truly arbitrary it seems like there could be an associated issue with tracking processed files in terms of memory usage (or IO time if tracking in files instead of a list or set). Solutions that keep a growing list of selected lines also run in to timing-related issues.

I’d consider something along the lines of the following:

1. Create n “bucket” files. n could be determined based on something that takes in to account the number of files and system memory. (If n is large, you could generate a subset of n to keep open file handles down.)
2. Each file’s name is hashed, and goes into an appropriate bucket file, “sharding” the directory based on arbitrary criteria.
3. Read in the bucket file contents (just filenames) and process as-is (randomness provided by hashing mechanism), or pick rnd(n) and remove as you go, providing a bit more randomosity.
4. Alternatively, you could pad and use the random access idea, removing indices/offsets from a list as they’re picked.