Question

I want to write to the database (start_time) once the spider start and want to store all statistics that being sent by email to the database too. So how could I do that ?

Should I write middlerware, If so, how could I trigger spider start and end ?

Answer 1

Here is what I figured out, Creating my own extension as described below :
http://doc.scrapy.org/en/latest/topics/extensions.html#writing-your-own-extension

BTW, I couldn’t know where to save the file, so I tried to name it myextension.py with class MyExtension and save in my project folder. Then in settings file (settings.py) call it as follows :

EXTENSIONS = {
    'project_name.myextension.MyExtension': 1000,
}