Question

With jQuery selectors you can select a div that contains the innerText “John” with $("div:contains('John')"), so you could match the second <div> in:

<div>Bill</div>
<div>John</div>
<div>Joe</div>

How can I do this in Python’s Beautiful Soup, or some other Python Module?

I just watched a lecture on scraping form PyCon 2010 where he mentions you can use CSS selectors in lxml.. Do I have to use that, or is there a way just with the Soup?

Background: Asking for the purpose of parsing a scraped web page.

Answer 1

A more concise way using BeautifulSoup:

>>> soup('div', text='John')
[u'John']
>>> import re
>>> soup('div', text=re.compile('Jo'))
[u'John', u'Joe']

soup() is equivalent to soup.findAll(). You could use string, regular expression, arbitrary function to select what you need.

stdlib’s ElementTree is enough in your case:

from xml.etree import cElementTree as etree

xml = """
    <div>Bill</div>
    <div>John</div>
    <div>Joe</div>
"""
root = etree.fromstring("<root>%s</root>" % xml)
for div in root.getiterator('div'):
    if "John" in div.text:
       print(etree.tostring(div))