Question

What I would like to do is shuffle the rows (read from CSV), then print out the first randomized 10,000 rows to one csv and the remainder to a separate csv. With a smaller file I can do something like

java.util.Collections.shuffle(...)
for (int i=0; i < 10000; i++) printcsv(...)
for (int i=10000; i < data.length; i++) printcsv(...)

However with very large files I now get OutOfMemoryError

Answer 1

Here’s one possible algorithm:

1. Let MAX_LINES be the maximum number of lines in a manageable file;
2. Read MAX_LINES from the input file, randomize these with your original algorithm and write them to a temporary file;
3. Repeat 2. until there are no lines left in your input file;
4. Let N be a random number between 0 and the number of temporary files you wrote; read the next line from the N-th temporary file;
5. Repeat 4. until you read all the lines from all the files; the first 10000 times write each line to the first output file, write all the other lines to the other file.