Question

After reading this forum I am not sure which method is best to extract sections of data into a CSV file I.e. Python/Beautiful Soup/html2text. Because of the large number of files, I want to try and write a script I can run within the Terminal.

Output: One CSV file, with lines of text and five columns of data. e.g. first and last line

100 2010-12-20 145 ABC 04110000

1 2010-11-10 133 DDD 041123847

Thanks!

Answer 1

I would recommend using BeautifulSoup. Something like this will do (completely untested). Read the documentation for more.

csvfile = open('dump.csv', 'w')
for file in glob.glob('*.html'):
    print 'Processing', file
    soup = BeautifulSoup(open(file).read())
    for tr in soup.findAll('tr'):
        print >>csvfile, ' '.join(tr.findAll('td'))