Question

I am using gcc 4.7.2 on a 64-bit Linux box.

I have 20 large sorted binary POD files that I need to read as a part of the final merge in an external merge-sort.

Normally, I would mmap all the files for reading and use a multiset<T,LessThan> to manage the merge sort from small to large, before doing a mmap write out to disk.

However, I realised that if I keep a std::mutex on each of these files, I can create a second thread which reads the file backwards, and sort from large to small at the same time. If I decide, beforehand, that the first thread will take exactly n/2 elements and the second thread will take the rest, I will have no need for a mutex on the output end of things.

Reading lock contentions, can be expected to occur, on average, maybe 1 in 20 in this particular case, so that’s acceptable.

Now, here’s my question. In the first case, it is obvious that I should call madvise with MADV_SEQUENTIAL, but I have no idea what I should do for the second case, where I’m reading the file backwards.

I see no MADV_REVERSE in the man pages. Should I use MADV_NORMAL or maybe don’t call madvise at all?

Recall that an external sort is needed when the volume of data is so large that it will not fit into memory. So we are left with a more complex algorithm to use disk as a temporary store. Divide-and-conquer algorithms will usually involve breaking up the data, doing partial sorts, and then merging the partial sorts.

My steps for an external merge-sort

1. Take n=1 billion random numbers and break them into 20 shards of equal sizes.
2. Sort each shard individually from small to large, and write each out into its own file.
3. Open 40 mmap’s, 2 for each file, one for going forward, one for going backwards, associate a mutex with each file.
4. Instantiate a std::multiset<T,LessThan> buff_fwd; for the forward thread and a std::multiset<T,GreaterThan> buff_rev for the reverse thread. Some people prefer to use priority queues here, but essentially, and sort-on-insert container will work here.
5. I like to call the two buffers surface and rockbottom, representing the smallest and largest numbers not yet added to the final sort.
6. Add items from the shards until n/2 is used up, and flush the shards to one output file using mmap from beginning towards the middle, and from the end towards to middle in the other thread. You can basically flush at will, but at least one should do it before either buffer uses up too much memory.

Answer 1

I would suggest:

MADV_RANDOM

To prevent useless read-ahead (which is in the wrong direction).