Question

I am having some trouble with lists. So, basically, I have a list:

a=["Britney spears", "red dog", "\xa2xe3"]

and I have another list, looking like:

b = ["cat","dog","red dog is stupid", "good stuff \xa2xe3", "awesome Britney spears"]

what I would like to do is check if elements in a are part of some element in b – and if so, remove them from b‘s element. So, I would like b to look like:

b = ["cat","dog","is stupid","good stuff","awesome"]

what is the most pythonic (in 2.7.x) way to achieve this?

I am assuming I can loop around to check each element, but I am not sure this is very efficient – I have a list (b) of around size 50k.

Answer 1

I think I’d use a regex here:

import re

a=["Britney spears", "red dog", "\xa2xe3"]

regex = re.compile('|'.join(re.escape(x) for x in a))

b=["cat","dog","red dog is stupid", "good stuff \xa2xe3", "awesome Britney spears"]

b = [regex.sub("",x) for x in b ]
print (b)  #['cat', 'dog', ' is stupid', 'good stuff ', 'awesome ']

This way, the regular expression engine can optimize the test for the list of alternatives.

Here are a bunch of alternatives to show how different regexs behave.

import re

a = ["Britney spears", "red dog", "\xa2xe3"]
b = ["cat","dog",
     "red dog is stupid", 
     "good stuff \xa2xe3", 
     "awesome Britney spears",
     "transferred dogcatcher"]

#This version leaves whitespace and will match between words.
regex = re.compile('|'.join(re.escape(x) for x in a))
c = [regex.sub("",x) for x in b ]
print (c) #['cat', 'dog', ' is stupid', 'good stuff ', 'awesome ', 'transfercatcher']

#This version strips whitespace from either end
# of the returned string
regex = re.compile('|'.join(r'\s*{}\s*'.format(re.escape(x)) for x in a))
c = [regex.sub("",x) for x in b ]
print (c) #['cat', 'dog', 'is stupid', 'good stuff', 'awesome', 'transfercatcher']

#This version will only match at word boundaries,
# but you lose the match with \xa2xe3 since it isn't a word
regex = re.compile('|'.join(r'\s*\b{}\b\s*'.format(re.escape(x)) for x in a))
c = [regex.sub("",x) for x in b ]
print (c) #['cat', 'dog', 'is stupid', 'good stuff \xa2xe3', 'awesome', 'transferred dogcatcher']


#This version finally seems to get it right.  It matches whitespace (or the start
# of the string) and then the "word" and then more whitespace (or the end of the 
# string).  It then replaces that match with nothing -- i.e. it removes the match 
# from the string.
regex = re.compile('|'.join(r'(?:\s+|^)'+re.escape(x)+r'(?:\s+|$)' for x in a))
c = [regex.sub("",x) for x in b ]
print (c) #['cat', 'dog', 'is stupid', 'good stuff', 'awesome', 'transferred dogcatcher']