Question

i am using scrapy 0.14

bellow is my code

start_urls = ['http://lakebagger.com/ur/']

def parse(self, response):
    hxs = HtmlXPathSelector(response)
    stores=hxs.select('//table[@id]//tr')
    print len(stores)

i am unable to access rows (tr) in tbody , rather then that i can only access thead row. how can i access the rows ?

thanks in advance . looks like HtmlXpathSelector is unable to rander xpath accurately

Answer 1

At this point, I would say that it’s environmental, albeit not an OS issue, since I managed to get a similar script to work on Windows 7. What are the results of typing "pip freeze"?

Mine:

Scrapy==0.16.2
Twisted==12.2.0
distribute==0.6.32
egenix-pyopenssl==0.13.0-1.0.0g-1
lxml==3.0.1
pyOpenSSL==0.9
pywin32==216
w3lib==1.2
zope.interface==3.8.0

Also, I really do think you should post your full spider code for examination since there may be something in the missing lines that is causing the issue.

EDIT: Based on your comments, this could be related to versions since you’re running 0.14. Have you considered upgrading?

Note: I followed these steps to install scrapy.