Question

I have a list of strings from which I need to remove all elements that match a substring from another list. I am trying to do this with lists, nested loops, and regex.

The output from the following snippet produces [“We don’t”, “need no”, “education”] instead of the desired [“education”]. I’m new to Python and this is my first experiment with regex, and I’m stuck on the sytax.

import re

testfile = ["We don't", "need no", "education"]
stopwords = ["We", "no"]
dellist = []

for x in range(len(testfile)):
    for y in range(len(stopwords)):
        if re.match(r'\b' + stopwords[y] + '\b', testfile[x], re.I):
            dellist.append(testfile[x])

for x in range(len(dellist)):
    if dellist[x] in testfile:
        del testfile[testfile.index(dellist[x])]

print testfile

The line

if re.match(r'\b' + stopwords[y] + '\b', testfile[x], re.I):

returns “None” for all iterations through the loop, so I’m guessing this is where my problem lies…

Answer 1

It’s because re.match tests for a match from the start of the string.

Try re.search instead. Also, you’re missing the r on your second '\b':

if re.search(r'\b' + stopwords[y] + r'\b', testfile[x], re.I):

---

Also, you could just use list comprehension to build up dellist (you could probably use list comprehension to build up the new testfile entirely, but it escapes me at the moment):

dellist = [w for w in testfile for test in stopwords if re.search(test,w,re.I)]

---

Another thought – since you’re using re module anyway, why don’t you combine your stopwords into \b(We|no)\b and then you can just test testfile against the one regex?

regex = r'\b(' + '|'.join(stopwords) + r')\b'  # r'\b(We|no)\b'

Now you just have to look for words that don’t match that regex:

newtestfile = [w for w in testfile if re.search(regex,w,re.I) is None]
# newtestfile is ['education']