Question

I need to scrape some websites and some images from these sites. When the image is a *.jpg i don’t have any problem, but these sites have *.svg images too, and i need these.

has anyone did this before?

here is the shell output with the error:

2013-01-18 14:44:10-0600 [crawler] DEBUG: Image (downloaded): Downloaded image from <GET http://page/image.svg> referred in <None>
2013-01-18 14:44:10-0600 [crawler] Unhandled Error

Traceback (most recent call last):
File "/virtualenvs/asd/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 576, in _runCallbacks
        current.result = callback(current.result, *args, **kw)
File "/virtualenvs/asd/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 381, in callback
        self._startRunCallbacks(result)
File "/virtualenvs/asd/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 489, in _startRunCallbacks
        self._runCallbacks()
File "/virtualenvs/asd/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 576, in _runCallbacks
        current.result = callback(current.result, *args, **kw)
    --- <exception caught here> ---
File "/virtualenvs/asd/local/lib/python2.7/site-packages/Scrapy-0.16.3-py2.7.egg/scrapy/contrib/pipeline/images.py", line 199, in media_downloaded
        checksum = self.image_downloaded(response, request, info)
File "/virtualenvs/asd/local/lib/python2.7/site-packages/Scrapy-0.16.3-py2.7.egg/scrapy/contrib/pipeline/images.py", line 252, in image_downloaded
        for key, image, buf in self.get_images(response, request, info):
File "/virtualenvs/asd/local/lib/python2.7/site-packages/Scrapy-0.16.3-py2.7.egg/scrapy/contrib/pipeline/images.py", line 261, in get_images
        orig_image = Image.open(StringIO(response.body))
File "/virtualenvs/asd/local/lib/python2.7/site-packages/PIL/Image.py", line 1980, in open
        raise IOError("cannot identify image file")
    exceptions.IOError: cannot identify image file

Thanks !
(sorry for my english)

Answer 1

If it helps someone else, I could solve this as follow

It item.py, add these attributes to the object:

      body = Field()
      url = Field()

In the spider (inside def parse()), add this code:

import urllib2 

(...)

    #select each img url
    relative_urls = info.select('tr/td/a[@class="image"]/img/@src').extract()

    for relative_url in relative_urls:
        #static url
        relative_url = relative_url.split("svg")[0][2:-1]+".svg"
        relative_url = ''.join(relative_url.split("/thumb")).strip()

        relative_url = "http://"+relative_url

        asd = urllib2.urlopen(relative_url)
        data = asd.read()
        with open("%s/%s" % ('/home/user/virtualenvs', img.svg), "wb") as code:
            code.write(data)

it works for me

(obviously can separate the code between the spider and the pipeline)