Question

I’m feeling dumb. Python & xpath newbie here. I’m trying to extract the complete text 'Open Box Price: $1079.99' using xpath from

<div class="prod-price">
<p class="opbox-price">
    <strong> Open Box Price:<br>$1079.99</strong>
    </p>
<p class="orig-price">
    Regular Price: <strong>$1499.98</strong>
    </p>
</div>

But I can’t. text stops at <br>. Here’s my code

doc = lxml.html.fromstring(r.content)
elements = doc.xpath(item_xpath)
print elements[1].find('div[3]/p[1]/text()[normalize-space()]')

Answer 1

A basis for the XPath you want is using descendant-or-self – tweak the result how you want:

>>> doc.xpath('//p[1]/descendant-or-self::text()')
['\n    ', ' Open Box Price:', '$1079.99', '\n    ']
>>> doc.xpath('//p[2]/descendant-or-self::text()')
['\n    Regular Price: ', '$1499.98', '\n    ']

Or as you’re doing with lxml.html, you could use text_content()

paras = doc.xpath('//p'): # or findall etc...
for para in paras:
    print para.text_content()