Question

Here’s the source:

<span class="new"> <a class="blog" href="http://whatever1.com" rel="nofollow">whatever1</a> do something at <a class="others" href="http://example1.com" rel="nofollow">example1</a></span>

<span class="new"> <a class="blog" href="http://whatever2.com" rel="nofollow">whatever2</a> do other things at <a class="others" href="http://example2.com" rel="nofollow">example2</a></span>

<span class="new"> <a class="blog" href="http://whatever3.com" rel="nofollow">whatever3</a> do something at <a class="others" href="http://example3.com" rel="nofollow">example3</a></span> 

I want to find all <span class="new"> with do something at in it, and here’s my code, I just don’t know why it’s not working:

soup = bs4.BeautifulSoup(html, "lxml")
all_tags = soup.findAll(name = "span", attrs = {"class": "new"}, text = re.compile('do something.*'))

Nothing found. If I remove text = re.compile('.*do something.*') all above tags can be found, I know there should be something wrong with my regex pattern, so what is the correct form?

Answer 1

You could always try a hybrid approach:

soup = bs4.BeautifulSoup(html, "lxml")
spans = soup.findAll("span", attrs = {"class": "new"})
regex = re.compile('.*do something at.*')
desired_tags = [span for span in spans if regex.match(span.text)]