Question

I have been given a file that I would like to extract the useful data from. The format of the file goes something like this:

LINE: 1
TOKENKIND: somedata
TOKENKIND: somedata
LINE: 2
TOKENKIND: somedata
LINE: 3

etc…

What I would like to do is remove LINE: and the line number as well as TOKENKIND: so I am just left with a string that consists of ‘somedata somedate somedata…’

I’m using Python to do this, using regular expressions (that I’m not sure are correct) to match the bits of the file I’d like removing.

My question is, how can I get Python to match multiple regex groups and ignore them, adding anything that isn’t matched by my regex to my output string? My current code looks like this:

import re
import sys

ignoredTokens = re.compile('''
    (?P<WHITESPACE>      \s+             ) |
    (?P<LINE>            LINE:\s[0-9]+   ) |
    (?P<TOKEN>           [A-Z]+:         )
''', re.VERBOSE)

tokenList = open(sys.argv[1], 'r').read()
cleanedList = ''

scanner = ignoredTokens.scanner(tokenList)

for line in tokenList:
    match = scanner.match()

    if match.lastgroup not in ('WHITESPACE', 'LINE', 'TOKEN'):
        cleanedList = cleanedList + match.group(match.lastindex) + ' '

print cleanedList

Answer 1

import re

x = '''LINE: 1
TOKENKIND: somedata
TOKENKIND: somedata
LINE: 2
TOKENKIND: somedata
LINE: 3'''

junkre = re.compile(r'(\s*LINE:\s*\d*\s*)|(\s*TOKENKIND:)', re.DOTALL)

print junkre.sub('', x)