Question

The difference between this question and Remove Top Line of Text File with Powershell is that my file is huge (over 300M).

The technique I am using now is based on one of the answers to that question, however it seems inefficient for huge files. Is there a faster (and less pretty) way to do it?

Answer 1

Try using a streamreader and streamwriter. This should perform quickly.

$reader = [IO.File]::OpenText("C:\Users\Andy\Documents\input.txt")
$writer = New-Object System.IO.StreamWriter("C:\Users\Andy\Documents\output.txt")

$reader.ReadLine() > $null # Skip first line.
while ($reader.Peek() -ge 0) {
    $writer.writeline($reader.ReadLine())
}

$reader.Close()
$writer.Close()

I tested by creating a 300MB text file and used this on it. output.txt was created in 3.5 seconds :-).

Update I optimized it slightly so it doesn’t doesn’t evaluate an expression every iteration of the loop. Now it completes in 2.9 seconds 🙂

If you don’t mind memory bloat you can also do this. This is faster still but takes more memory, but less than Get-Content.

$reader = [IO.File]::OpenText("C:\Users\Andy\Documents\input.txt")
$writer = New-Object System.IO.StreamWriter("C:\Users\Andy\Documents\output.txt")

$reader.ReadLine() > $null # Skip first line.
$writer.write($reader.ReadToEnd())

$reader.Close()
$writer.Close()