Question

I hope the subject is clear enough, I haven’t found anything specifically about this in the previously asked bin. I’ve tried implementing this in Perl or Python, but I think I may be trying too hard.

Is there a simple shell command / pipeline that will split my 4mb .txt file into seperate .txt files, based on a beginning and ending regex?

I provide a short sample of the file below.. so you can see that every "story" starts with the phrase "X of XXX DOCUMENTS", which could be used to split the file.

I think this should be easy and I’d be surprised if bash can’t do it – faster than Perl/Py.

Here it is:

                           1 of 999 DOCUMENTS


              Copyright 2011 Virginian-Pilot Companies LLC
                          All Rights Reserved
                   The Virginian-Pilot(Norfolk, VA.)

...



                           3 of 999 DOCUMENTS


                  Copyright 2011 Canwest News Service
                          All Rights Reserved
                          Canwest News Service

...

Thanks in advance for all your help.

Ross

Answer 1

awk '/[0-9]+ of [0-9]+ DOCUMENTS/{g++} { print $0 > g".txt"}' file

OSX users will need gawk, as the builtin awk will produce an error like awk: illegal statement at source line 1

Ruby(1.9+)

#!/usr/bin/env ruby
g=1
f=File.open(g.to_s + ".txt","w")
open("file").each do |line|
  if line[/\d+ of \d+ DOCUMENTS/]
    f.close
    g+=1
    f=File.open(g.to_s + ".txt","w")
  end
  f.print line
end