Question

I am attempting to set up a custom downloader middleware class in Scrapy. I suspect that I’ve missed something obvious, but I’ve read over the docs a few times and have found no solutions. I’m getting a bit frustrated with what should be an extremely simple task, so hopefully someone will be able to provide me with some insight.

I’ve added the following line to my settings.py file.

DOWNLOADER_MIDDLEWARES = { 'myproject.middlewares.TestDownloader': 400 }

After adding that line and running the project, I got an error about the module middlewares not existing. After some research, I discovered that you needed to add a __init__.py file to the middlewares folder for Python to recognize it. I did this, and am now getting the following error:

NameError: Module 'myproject.middlewares' doesn't define any object named 'TestDownloader'

The TestDownloader.py file is not being compiled, while all other *.py files in the project are. If I understand my Python reading correctly, that means it isn’t being imported anywhere, but I can’t find any additional Scrapy settings to change to make this work.

Answer 1

DOWNLOADER_MIDDLEWARES = { 'myproject.middlewares.TestDownloader': 400 }

For this to work, create file middlewares.py inside myproject folder, and in that file put your downloader middleware class called TestDownloader.

Or having middlewares folder with __init__.py inside it, you can put put your downloader middleware class called TestDownloader inside __init__.py — this should work too.