Question

I’m trying to write a regular expression in python, and one of the characters involved in it is the \001 character. putting \001 in a string doesn’t seem to work. I also tried ‘string’ + str(chr(1)), but the regex doesn’t seem to catch it. Please for the love of god somebody help me, I’ve been struggling with this all day.

import sys
import postgresql
import re

if len(sys.argv) != 2:
    print("usage: FixToDb <fix log file>")
else:
    f = open(sys.argv[1], 'r')
    timeExp = re.compile(r'(\d{2}):(\d{2}):(\d{2})\.(\d{6}) (\S)')
    tagExp = re.compile('(\\d+)=(\\S*)\001')
    for line in f:
        #parse the time
        m = timeExp.match(line)
        print(m.group(1) + ':' + m.group(2) + ':' + m.group(3) + '.' + m.group(4) + ' ' + m.group(5));
        tagPairs = re.findall('\\d+=\\S*\001', line)
        for t in tagPairs:
            tagPairMatch = tagExp.match(t)
            print ("tag = " + tagPairMatch.group(1) + ", value = " + tagPairMatch.group(2))

Here’s is an example line of for the input. I replaced the ‘\001’ character with a ‘~’ for readability

15:32:36.357227 R 1 0 0 0 8=FIX.4.2~9=0067~35=A~52=20120713-19:32:36~34=1~49=PD~56=P~98=0~108=30~10=134

output:

15:32:36.357227 R
tag = 8, value = FIX.4.29=006735=A52=20120713-19:32:3634=149=PD56=P98=0108=3010=134

So it doesn’t stop at the ‘\001’ character.

Answer 1

chr(1) should work, as will "\x01", as will "\001". (Note that chr(1) already returns a string, so you don’t need to do str(chr(1)).) In your example it looks like you have both "\001" and chr(1), so that won’t work unless you have two of the characters in a row in your data.

You say the regex “doesn’t seem to catch it”, but you don’t give an example of your input data, so it’s impossible to say why.

Edit; Okay, it looks like the problem has nothing to do with the \001. It is the classic greediness problem. The \S* in your tagExp expression will match a \001 character (since that character is not whitespace. So the \S* is gobbling the entire line. Use \S*? to make it non-greedy.

Edit: As others have noted, it also looks like your backslashes are awry. In regular expressions you face a backslash-doubling problem: Python uses the backslash for its own string escapes (like \t for tab, \n for newline), but regular expressions also use the backslash for their own purposes (e.g., \s for whitespace). The usual solution is to use raw strings, but you can’t do that if you want to use the "\001" escape. However, you could use raw strings for your timeExp regex. Then in your other regexes, double the backslashes (except on \001, because you want that one to be interpreted as a character-code escape).