I need to scrape from a website that requires authentication, that is a user must login by supplying there username and password in an html form. The connection is not secure. I plan on using the Pear framework to format the request. I’m stuck at the first part which is posting the login information. How can I submit form data? Thanks in advance for any help.
I’ve got fiddler2 installed but am not sure what to look for when I do the login manually.
I’m using curl and getting some success. I deleted all cookies in my web browser and went to the site to login. In fiddler it gave me the following request headers
GET http://example.com/niceday/dirlogin.php HTTP/1.1
Accept: text/html, application/xhtml+xml, */*
Accept-Language: en-CA
User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)
Accept-Encoding: gzip, deflate
Host: example.com
DNT: 1
Connection: Keep-Alive
Cookie: chatRoomUserID=119972; ASPSESSIONIDSSRQCCBR=26jqrt3f8dus2l2s42g4r9npp5
I’m confused about the last one, am I requesting a cookie?
In my script I faked the first request with
curl_setopt($ch,CURLOPT_HTTPHEADER, array('GET http://example.com/niceday /dirlogin.php HTTP/1.1',
'Accept: text/html, application/xhtml+xml, */*',
'Referer: http://r2sports.bkoehler.j43.ca/tourney/tourneyTop.php?TID=3206',
'Accept-Language: en-CA',
'User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)',
'Accept-Encoding: gzip, deflate',
'Connection: Keep-Alive',
'DNT: 1',
'Host: r2sports.bkoehler.j43.ca',
'Cookie: chatRoomUserID=995222; ASPSESSIONIDSSRQCCBR=26jqrt3f8dus2l2s42g4r9npp5'));
You might consider using this: http://www.php.net/manual/en/book.curl.php#90821 and setting a cookie file location, so your script can remain authenticated. Next time it will contact that website it will use the cookies it previously saved in a file in the cookie file location.