Question

I am working with the web scraping framework Scrapy and I am wondering how do I iterate over all of the scraped items which seem to be in a dictionary and strip the white space from each one.

Here is the code I have been playing with in my item pipeline:

for info in item:
   info[info].lstrip()

But this code does not work, because I cannot select items individually. So I tried to do this:

for key, value item.items():
   value[1].lstrip()

This second method works to a degree, but the problem is that I have no idea how then to loop over all of the values.

I know this is probably such an easy fix, but I cannot seem to find it.

Answer 1

Not a direct answer to the question, but I would suggest you look at Item Loaders and input/output processors. A lot of your cleanup can be take care of here.

An example which strips each entry would be:

class ItemLoader(ItemLoader):

    default_output_processor = MapCompose(unicode.strip)