Question

I’m having a problem iterating a crawl using scrapy. I am extracting a title field and a content field. The problem is that I get a JSON file with all of the titles listed and then all of the content. I’d like to get {title}, {content}, {title}, {content}, meaning I probably have to iterate through the parse function. The problem is that I cannot figure out what element I am looping through (i.e., for x in [???]) Here is the code:

from scrapy.spider import BaseSpider
from scrapy.selector import HtmlXPathSelector
from scrapy.contrib.spiders import SitemapSpider

from Foo.items import FooItem


class FooSpider(SitemapSpider):
    name = "foo"
    sitemap_urls = ['http://www.foo.com/sitemap.xml']
    #sitemap_rules = [


    def parse(self, response):
        hxs = HtmlXPathSelector(response)
        items = [
        item = FooItem()
        item['title'] = hxs.select('//span[@class="headline"]/text()').extract()
        item['content'] = hxs.select('//div[@class="articletext"]/text()').extract()
        items.append(item)
        return items

Answer 1

Your xpath queries returns all titles and all contents on the page. I suppose you can do:

titles = hxs.select('//span[@class="headline"]/text()').extract()
contents = hxs.select('//div[@class="articletext"]/text()').extract()

for title, context in zip(titles, contents):
    item = FooItem()
    item['title'] = title
    item['content'] = context
    yield item

But it is not reliable. Try to perform xpath query that return block with title and content inside. If you showed me xml source I’d help you.

blocks = hxs.select('//div[@class="some_filter"]')
for block in blocks:
    item = FooItem()
    item['title'] = block.select('span[@class="headline"]/text()').extract()
    item['content'] = block.select('div[@class="articletext"]/text()').extract()
    yield item

I’m not sure about xpath queries but I think idea is clear.