Question

I am using lxml and python. I want to fetch the href for the link that reads More reviews (‎40) on this page. I am basically scrapping this site and want to get the reviews.

Would appreciate the help. Thanx

Answer 1

The link is added using client-side javascript. So you cannot get the href using normal HTML parsing. You can however look at the javascript code and get the link from there:

>>> import re
>>> import urllib2
>>> import lxml.html
>>> page = urllib2.urlopen("http://maps.google.com/maps/place?cid=2860002122405830765").read()

# have to search the page source since the link is added in javascript
>>> mo = re.search(r'<div class="pp-more-reviews">.*?</div>', page)
>>> div = lxml.html.fromstring(mo.group(0))
>>> href = div.find("a").attrib["href"]

Other options are:

1. Use selenium to control a real browser.
2. Use the phantomJS headless browser