Question

I need to generate a large number of files indexed by an integer. For example, assuming int i ranges from 0 to 10000, the generated files would be: f0.xml, f1.xml, … f10000.xml

However the file system performance starts to degrade for more than 1000 files per folder.

I wanted to store the files in directories using a ‘hash’ or ‘spread’ function over the value of i. This function would store no more than 1000 files (or folders) in a single directory but would only create directories as needed.

Any ideas or open-source solutions for my problem?
Thanks.

Answer 1

This is how I implemented using some of the provided tips.
First a function to decompose the index in an array of 3 digit sets:

private static void decompose(final long l, final short[] array) {
    long q = l;
    long r = 0;
    for (int j=array.length-1; j >= 0; j--) {
        // compute remainder
        r = q % 1000;
        // compute quotient
        // converts to int and fractional part is dropped without rounding
        q = q / 1000;

        array[j] = (short) r;
    }
}

Then, using the decomposed array (currentA), create the subdirectories and file objects.

    File dir = parent;
    for (int j=0; j < depth-1; j++) {
        String dirName = String.format("%03d", currentA[j]);
        dir = new File(dir, dirName);
    }
    String fileName = prefix + String.format("%03d", currentA[depth]) + suffix;
    File file = new File(dir, fileName);