Home/ Questions/Q 3230142
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-17T16:53:32+00:00

I am trying to automate files download via a webserver. I plan on using

  • 0

I am trying to automate files download via a webserver. I plan on using wget or curl or python urllib / urllib2.

Most solutions use wget and urllib and urllib2. They all talk of HHTP based authentication and cookie based authentication. My problem is I dont know which one is used in the website that stores my data.
Here is the interaction with the site:

  1. Normally I login to site http://www.anysite.com/index.cgi?
  2. I get a form with a login and password. I type in both and hit return.
  3. The url stays as http://www.anysite.com/index.cgi? during the entire interaction. But now I have a list of folders and files
  4. If I click on a folder or file the URL changes to http://shamrockstructures.com/cgi-bin/index.cgi?page=download&file=%2Fhome%2Fjanysite%2Fpublic_html%2Fuser_data%2Fuserareas%2Ffile.tar.bz2

And the browser offers me a chance to save the file

I want to know how to figure out whether the site is using HTTP or cookie based authentication. After which I am assuming I can use cookielib or urllib2 in python to connect to it, get the list of files and folders and recursively download everything while staying connected.

p.S: I have tried the cookie cutter ways to connect via wget and wget –http-user “uname” –http-password “passwd” http://www.anysite.com/index.cgi? , but they only return the web form back to me.

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    If you log in using a Web page, the site is probably using cookie-based authentication. (It could technically use HTTP basic auth, by embedding your credentials in the URI, but this would be a dumb thing to do in most cases.) If you get a separate, smallish dialog with a user name and password field (like this one), it is using HTTP basic authentication.

    If you try to log in using HTTP basic auth, and get back the login page, as is happening to you, this is a certain indication that the site is not using HTTP basic auth.

    Most sites use cookie-based authentication these days. To do this with an HTTP cilent such as urllib2, you will need to do an HTTP POST of the fields in the login form. (You may need to actually request the login form first, as a site could include a cookie that you need to even log in, but usually this is not necessary.) This should return a “successfully logged in” page that you can test for. Save the cookies you get back from this request. When making the next request, include these cookies. Each request you make may respond with cookies, and you need to save those and send them again with the next request.

    urllib2 has a function called a “cookie jar” which will automatically handle the cookies for you as you send requests and receive Web pages. That’s what you want.

    • 0