Question

I am scraping a soccer site and the spider (a single spider) gets several kinds of items from the site’s pages: Team, Match, Club etc.
I am trying to use the CSVItemExporter to store these items in separate csv files,
teams.csv, matches.csv, clubs.csv etc.

I am not sure what is the right way to do this.
The only way I have thought so far is to create my own custom pipeline like in the example
http://doc.scrapy.org/en/0.14/topics/exporters.html and there open all needed csv files in the spider_opened method, ie create a csv exporter for each csv file and in the process_item put code to figure out what kind of item is the “item” parameter and then send it to the corresponding exporter object.

Anyway I haven’t found any examples of handling multiple csv files (per item type) in scrapy so I am worrying that I am using it in a way that is not meant to be used. (this is my first experience with Scrapy).

diomedes

Answer 1

You approach seems fine to me. Piplines are a great feature of Scrapy and are IMO build for something like your approach.

You could create multiple items (e.g. SoccerItem, MatchItem) and in your MultiCSVItemPipeline just delegate each item to its own CSV class by checking the item class.