Question

I use a regular expression to extract chords from input text files. While it works most of the time it fails on a certain file.

This is my regexp code:

def getChordMatches(line):
    import re

    notes = "[ABCDEFG]";
    accidentals = "(?:#|##|b|bb)?";
    chords = "(?:maj|min|m|sus|aug|dim)?"
    additions = "[0-9]?"
    chordFormPattern = notes + accidentals + chords + additions
    fullPattern = chordFormPattern + "(?:/%s)?\s" % (notes + accidentals)
    matches = [removeWhitespaces(x) for x in re.findall(fullPattern, line)]
    positions = [x.start() for x in re.finditer(fullPattern, line)]

    return matches, positions

This is the result when it works:

    line:      Em             C  C/B
 matches: [u'Em', u'C', u'C/B']
position: [5, 20, 23]

This line is from a file that doesn’t produce the correct result:

    line:   Am           Am/G       D7/F#                 Fmaj7
 matches: [u'Fmaj7']
position: [48]

Where should I start digging? Encoding, special characters, tabs, … ?

edit

This is where above output is from:

line = unicode(l, encoding='utf-8') 
matches, positions = getChordMatches(line) 
print '    line:', line 
print ' matches:', matches 
print 'position:', positions

edit

The full regex pattern is:

[ABCDEFG](?:#|##|b|bb)?(?:maj|min|m|sus|aug|dim)?[0-9]?(?:/[ABCDEFG](?:#|##|b|bb)?)?\s

edit

A hexdump of the failing line (I think):

hexdump -s 45 -n 99 input.txt 
000002d 20 41 6d 20 20 20 20 20 20 20 20 20 20 41 6d 2f 
000003d 47 20 c2 a0 20 20 20 20 20 20 44 37 2f 46 23 20 
000004d 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 
000005d 46 6d 61 6a 37 0a 49 20 6c 6f 6f 6b 20 61 74 20 
000006d 79 6f 75 20 61 6c 6c 20 73 65 65 20 74 68 65 20 
000007d 6c 6f 76 65 20 74 68 65 72 65 20 74 68 61 74 27 
000008d 73 20 73 
0000090

edit

As mentioned in the accepted answer it was caused by a non breaking space. Using line = unicode(l, encoding='utf-8').replace(u"\u00A0", " ") solves the problem.

Answer 1

I suspect the problem has to do with the following two bytes:

000003d 47 20 c2 a0 20 20 …

This seems to be a UTF-8 encoded non-breaking space (U+00A0). It wouldn’t surprise me if this is what’s tripping up your regex.