Question

Is it possible to get the request referrer from the response object in parse function?

10x

Answer 1

HTTP Referer field is set up by HTTP client in request headers, not in response headers, as this header tells server where did client come from to current page.

It would be rather weird to receive http Referer header in response.

But when talking about scrapy, there’s a reference to Request object on which the Response was generated, in response’s request field, so the next call result:

response.request.headers.get('Referer', None)

can contain Referer header if it was set when making request.