Question

How do you pass a csrftoken with the python module Requests? This is what I have but it’s not working, and I’m not sure which parameter to pass it into (data, headers, auth…)

import requests
from bs4 import BeautifulSoup

URL = 'https://portal.bitcasa.com/login'

client = requests.session(config={'verbose': sys.stderr})

# Retrieve the CSRF token first
soup = BeautifulSoup(client.get('https://portal.bitcasa.com/login').content)
csrftoken = soup.find('input', dict(name='csrfmiddlewaretoken'))['value']

login_data = dict(username=EMAIL, password=PASSWORD, csrfmiddlewaretoken=csrftoken)
r = client.post(URL, data=login_data, headers={"Referer": "foo"})

Same error message every time.

<h1>Forbidden <span>(403)</span></h1>
<p>CSRF verification failed. Request aborted.</p>

Answer 1

If you are going to set the referrer header, then for that specific site you need to set the referrer to the same URL as the login page:

import sys
import requests

URL = 'https://portal.bitcasa.com/login'

client = requests.session()

# Retrieve the CSRF token first
client.get(URL)  # sets cookie
if 'csrftoken' in client.cookies:
    # Django 1.6 and up
    csrftoken = client.cookies['csrftoken']
else:
    # older versions
    csrftoken = client.cookies['csrf']

login_data = dict(username=EMAIL, password=PASSWORD, csrfmiddlewaretoken=csrftoken, next='/')
r = client.post(URL, data=login_data, headers=dict(Referer=URL))

When using unsecured http, the Referer header is often filtered out and otherwise easily spoofable anyway, so most sites no longer require the header to be set. However, when using an SSL connection and if it is set, it does make sense for the site to validate that it at least references something that could logically have initiated the request. Django does this when the connection is encrypted (uses https://), and actively requires it then.