Question

I have a very big sequence of strings. Length of each string is 50. Each string includes only chars from english ABC. What is the best(the fastest) way to sort this sequence?

Answer 1

If I had to code that, I’d probably make one pass that split the input into many output files depending on the first couple of characters or so; the goal being to make each output file small enough to fit in main memory. Then I would open each file in order, sort it in memory, and append it to the output. First pass is O(n), second is more or less O(n log n), and you have to do disk I/O four times per record. It might be possible to do better with some arcane algorithm, but probably not by much, and this is easy to understand and code.

If the system limits how many files you can have open at once, you might have to split up the first pass. If the strings aren’t well-distributed, some intermediate files might be too large.

In pseudocode:

open input file (r)
for i in ['aa', 'ab', 'ac', ..., 'zz']:
    open output file[i] (w)
for record in input file:
    write record to output file[record[0:2]]
close all files
open main output file (w)
for i in ['aa', 'ab', 'ac', ..., 'zz']:
    open input file[i] (r)
    slurp whole file into memory
    close input file
    sort data
    append whole sorted file to main output file

EDIT: Wait, do you mean the records only contain the characters A, B, and C? No other letters? In that case you would probably have to split on an initial substring longer than 2. Splitting on the first 3 characters would divide it into 27 files, each of size 370 MB on average.