Question

Regards, SO

I am new to python and Perl. I have been trying to solve a simple problem and getting tied in knots with syntax. I hope someone has the time and patience to help.
I have a 25mb file in “.txt” format which contains news-wire articles going back to 1970. Each news story is concatenated to the next, with only the “Copyright” statement to delimit. Each news story starts with “Item XX of XXX DOCUMENTS”. There are certain metadata that are repeated throughout, I will use these for tagging later on.

I wish to split this 25mb file into separate .txt files, each containing one news story (i.e. the text between “DOCUMENTS” and “Copyright”, saving each with a different name (obviously).

I am trying to 1 ) open the file… 2) iterate over lines in the file checking for the eof delimiter, and if it is not present writing the line to a list 3)write that list to a seperate small file.

I’m having big problems with changing filenames using the counter, and how do I make Python start from where I left off, is the “seek” function appropriate?

so far I have been trying this approach, completely unsuccessfully:

myfile = open ("myfile.txt", 'r')
filenumber = 0
for line in myfile.readline():  
    filenumber += 1    
    w=0  
    while myfile.readline() != '\s+DOCUMENTS\s*\n'  
    ### read my line into a list  
    mysmallfile()['w'] = [myfile.readline()]  
    w += 1  
    output = open('C:\\Users\\dunner7\\Documents\###how do I change the filename      each     iteration???', 'w')  
    output.writelines(mysmallfile)   
    ###go back to start.   

Thank you for your time and patience.

RD

Here is a sample of the text file:

                           1 of 575 DOCUMENTS

                         The Washington Daybook

                            January 28, 2011

Health and Human Services Department (HHS); Food and Drug Administration (FDA)
(F.R. Page 72832) holds a meeting of the Neurological Devices Panel of the
Medical Devices Advisory Committee to discuss and make recommendations regarding
the possible reclassification of devices indicated for use in electroconvulsive
therapy, January 27-28.

TIME: 8 a.m.

LOCATION: Hilton Washington DC North/Gaithersburg, Ballroom, 620 Perry Parkway,
Gaithersburg, Md.

CONTACT: James Engles, 800-741-8138 [Note: Use the code: 3014512513, when
calling for information.]

LOAD-DATE: November 28, 2010

LANGUAGE: ENGLISH

TYPE: Meeting

       Copyright 2011 Federal Information and News Dispatch, Inc.


                           2 of 575 DOCUMENTS

                         The Washington Daybook

                            January 27, 2011

Health and Human Services Department (HHS); Food and Drug Administration (FDA)
(F.R. Page 72832) holds a meeting of the Neurological Devices Panel of the
Medical Devices Advisory Committee to discuss and make recommendations regarding
the possible reclassification of devices indicated for use in electroconvulsive
therapy, January 27-28.

TIME: 8 a.m.

LOCATION: Hilton Washington DC North/Gaithersburg, Ballroom, 620 Perry Parkway,
Gaithersburg, Md.

CONTACT: James Engles, 800-741-8138 [Note: Use the code: 3014512513, when
calling for information.]

LOAD-DATE: November 28, 2010

LANGUAGE: ENGLISH

TYPE: Meeting

       Copyright 2011 Federal Information and News Dispatch, Inc.


                           3 of 575 DOCUMENTS


                              FNS DAYBOOK

                       January 12, 2011 Wednesday
                              FUTURE EVENTS

EVENT: MEETING – HEALTH AND HUMAN SERVICES DEPARTMENT (HHS); FOOD AND DRUG
ADMINISTRATION (FDA) (F.R. PAGE 72832);
LOCATION: Hilton Washington DC North/Gaithersburg, Ballroom, 620 Perry Parkway,
Gaithersburg, Md. — January 27, 2011 8:00 am

SECTION: FEDERAL AGENCIES AND DEPARTMENTS – FUTURES

LENGTH: 72 words

SUBJECT: Health and Human Services Department (HHS); Food and Drug
Administration (FDA) (F.R. Page 72832) holds a meeting of the Neurological
Devices Panel of the Medical Devices Advisory Committee to discuss and make
recommendations regarding the possible reclassification of devices indicated for
use in electroconvulsive therapy, January 27-28.

CONTACT: James Engles, 800-741-8138 [Note: Use the code: 3014512513, when
calling for information.]

LOAD-DATE: January 10, 2011

LANGUAGE: ENGLISH

PUBLICATION-TYPE: Event Schedules

                  Copyright 2011 Federal News Service
                          All Rights Reserved

Answer 1

Something like that:

filenumber = 0
outfile = None
with open('source_file.txt') as f:
    for line in f:
        if line.strip() == 'DOCUMENTS':
            filenumber += 1
            outfile = open('result%03d.txt' % filenumber, 'w')
        elif line.strip().startswith('Copyright') and outfile:
            outfile.close()
            outfile = None
        elif outfile:
            outfile.write(line)
if outfile:
    outfile.close()

I had to guess lots of things because I don’t know exacly how the file looks like. Post the file if you have further problems.