Question

I am trying to download some documents using requests, but the page is redirecting me to a userlog in screen and therefor downloading the HTML page.

I’ve tried doing:

c=requests.get(url,auth=HTTPBasicAuth('user','pass'))

But I’m not getting authenticated.

I’ve also tried vanilla and Digest.

The form itself looks like this:

<input id="username" name="username" class="required" tabindex="1" type="text" value="" size="25" autocomplete="false"/>
<br/>

<label for="password">Password</label>
<input id="password" name="password" class="required" tabindex="2" type="password" value="" size="25" autocomplete="off"/>

Do I need to pass in the username and password as a part of the payload? If so, how do I do that? I’ve tried a few different ways so far.

Answer 1

Basically, it had to do with grabbing the authentication ID off the page and passing in cookies.

This is basically what I did:

from bs4 import BeautifulSoup as bs
import requests
s = requests.session()
url = r'url_i_care_about'

def authenticate(s, url):
    headers = {'username': 'myuser', 'password': 'mypasss', '_Id': 'submit'}
    page=s.get(url)
    soup=bs(page.content)
    value=soup.form.find_all('input')[2]['value']
    headers.update({'value_name':value})
    auth = s.post(url, params=headers, cookies=page.cookies)