Question

I need to create an output text file by deleting the first two lines of the input file.

At the moment I’m using
sed “1,2d” input.txt > output.txt

I need to do this for thousands of files, so am using python:

import os
for filename in somelist:
  os.system('sed "1,2d" %s-in.txt > %s-out.txt'%(filename,filename))

but this is quite slow.

I need to keep the original file, so I can’t sed in place.

Is there a way to do this faster? Using something other than sed?Perhaps using some other scripting language than python? Is it worth writing a short C program, or is the file-writing disk-access likely to be the bottleneck?

Answer 1

Use tail. Doubt anything could be significantly faster:

tail -n +3 input.txt > output.txt

Wrap it in your loop of choice. But I really doubt sed is a whole ton slower – as you say, disk i/o is usually the ultimate bottleneck.