Question

I’m looking for the right xpath expression to search all text() in html page that contains the string: @domain

on a match extract till the first space on the left and till the first space on the right –

just to fetch the email address.

Thanks

Answer 1

This Xpath query will fetch text of all nodes containing ‘@domain’

//*[contains(text(), '@domain')]/text()

You can then parse the text to extract the email using Python

>>> import re
>>> re.findall(r'[\w\.]+@domain\.[\w\.]+', 'this is our info: [e-mail]')
['[e-mail]']

UPDATE:

Looks like XPath selectors in scrapy have re method, which i wasn’t aware of:

>>> hxs.select('//a[contains(@href, "image")]/text()').re(r'Name:\s*(.*)')
[u'My image 1',
 u'My image 2',
 u'My image 3',
 u'My image 4',
 u'My image 5']