Question

Hi I am working on scrapy for fetching some html pages,

I had written my spider and i had fetched the required data from the pages in spider.py file, and in my pipeline.py file i want to write all the data in to a csv file created dynamically with the name of the spider and below is my pipeline.py code

pipeline.py:

from scrapy import log
from datetime import datetime


class examplepipeline(object):

    def __init__(self):
        dispatcher.connect(self.spider_opened, signal=signals.spider_opened)
        dispatcher.connect(self.spider_closed, signal=signals.spider_closed)

    def spider_opened(self, spider):
        log.msg("opened spider  %s at time %s" % (spider.name,datetime.now().strftime('%H-%M-%S')))
        self.exampleCsv = csv.writer(open("%s(%s).csv"% (spider.name,datetime.now().strftime("%d/%m/%Y,%H-%M-%S")), "wb"),
                   delimiter=',', quoting=csv.QUOTE_MINIMAL)
        self.exampleCsv.writerow(['Listing Name', 'Address','Pincode','Phone','Website'])           

    def process_item(self, item, spider):
        log.msg("Processsing item " + item['title'], level=log.DEBUG)
        self.exampleCsv.writerow([item['listing_name'].encode('utf-8'),
                                    item['address_1'].encode('utf-8'),
                                    [i.encode('utf-8') for i in item['pincode']],
                                    item['phone'].encode('utf-8'),
                                    [i.encode('utf-8') for i in item['web_site']]
                                    ])
        return item 


    def spider_closed(self, spider):
        log.msg("closed spider %s at %s" % (spider.name,datetime.now().strftime('%H-%M-%S')))

Result:

--- <exception caught here> ---
  File "/usr/lib64/python2.7/site-packages/twisted/internet/defer.py", line 133, in maybeDeferred
    result = f(*args, **kw)
  File "/usr/lib/python2.7/site-packages/Scrapy-0.14.3-py2.7.egg/scrapy/xlib/pydispatch/robustapply.py", line 47, in robustApply
    return receiver(*arguments, **named)
  File "/home/local/user/example/example/pipelines.py", line 19, in spider_opened
    self.examplecsv = csv.writer(open("%s(%s).csv"% (spider.name,datetime.now().strftime("%d/%m/%Y,%H-%M-%S")), "wb"),
exceptions.IOError: [Errno 2] No such file or directory: 'example(27/07/2012,10-30-40).csv'

Here actually spider name is example

I don’t understand whats wrong in the above code, it should create csv file dynamically with spider name, but showing the above mentioned error, can anyone please let me know whats happening there………

Answer 1

The problem is with forward slash(directory separator) in your filename. It is not allowed. Try using some other character in the date.

More info here
http://www.linuxquestions.org/questions/linux-software-2/forward-slash-in-filenames-665010/

This link is helpful for getting the format you want
How to print date in a regular format in Python?

>>> import datetime
>>> datetime.date.today()
datetime.date(2012, 7, 27)
>>> str(datetime.date.today())
'2012-07-27'

Use this in your code

open("%s(%s).csv"% (spider.name,datetime.now().strftime("%d-%m-%Y:%H-%M-%S"))