Question

Just wondering if there is a faster way to split a file into N chunks other than unix “split”.

Basically I have large files which I would like to split into smaller chunks and operate on each one in parallel.

Answer 1

I assume you’re using split -b which will be more CPU-efficient than splitting by lines, but still reads the whole input file and writes it out to each file. If the serial nature of the execution of this portion of split is your bottleneck, you can use dd to extract the chunks of the file in parallel. You will need a distinct dd command for each parallel process. Here’s one example command line (assuming the_input_file is a large file this extracts a bit from the middle):

dd skip=400 count=1 if=the_input_file bs=512 of=_output

To make this work you will need to choose appropriate values of count and bs (those above are very small). Each worker will also need to choose a different value of skip so that the chunks don’t overlap. But this is efficient; dd implements skip with a seek operation.

Of course, this is still not as efficient as implementing your data consumer process in such a way that it can read a specified chunk of the input file directly, in parallel with other similar consumer processes. But I assume if you could do that you would not have asked this question.