Question

I am working with large number of integer permutations. The number of elements in each permutation is K. The element size is 1 byte. I need to generate N unique random permutations.
Constraints: K <= 144, N <= 1,000,000.

I came up with the following straightforward algorithm:

1. Generate list of N random permutations. Store all permutations in RAM.
2. Sort the list and delete all duplicates (if any). The number of duplicates will be relatively small.
3. If there were any duplicates, add random permutations to the list until there are N permutations and return to step 2.

Is there a better way to do this? Especially, is there a way to not store all permutations in RAM (write them on disk while generating)?

Edit: In the end, the generated permutations need to be accessed sequentially (one-by-one, no need for random access). The RAM is more crucial factor (I would prefer to not store all permutations at once in RAM).

Answer 1

One possible solution is using bloom filters.

Store your permutations on disk (write them sequentially) and maintain a bloom filter in RAM.

Once you generate a permutation – check if it exists in the bloom filter, if the bloom filter says it is not written to disk yet- write it, bloom filters don’t have false negatives.

If the bloom filter however says it is on the disk – it might be wrong..

if the bloom filter said “the permutation already exists”, you can decide if you want to quit this candidate and go to the next one without checking if it is really already in the set, or you can search the disk to see if it is really there.

If you chose the later, you should consider maintaining a smart DS for the permutations such as a hash table or a B+ tree.

Bloom Filters are perfect match in here – they are designed to represent a set that is expansive to read, while giving 0 false negatives, which is the most important thing here.