Question

I’m relatively new to python and scrapy and am in need of some assistance regarding a issue that I’m trying to tackle.
I’m trying to crawl amazon and extract the user comments for a specific product using scrapy and XPath.
I wanted to ask if there is a more elegant solution than the current one that I have.

Let’s say that I want to get the comments from this address.
The structure of the comments page does not look very friendly to extraction (using firebug you can see that there are no specific tags surrounding only the comment area).
Currently I’m using the following selector: hxs.select('//div/text()').extract() but as you can imagine it creates a lot of junk data "\n\n\n\n\n" etc. Is there a more elegant way to write my selector which can improve my results.

Answer 1

One option is to pivot off the “This review is from…” text. Like this:

//table[@id='productReviews']//div[@class='tiny']/following-sibling::text()