Question

Ok, I’ve been racking my brain on this one solo for too long. I’ve been unable to crack it even with hours spent on this and many other sites.

Essentially, I’m trying to strip some data from a webpage behind a LogIn page using WebRequest/Response. (I have gotten this to work using a WebBrowser control with some layered events which navigate to the different web pages but it’s causing some problems when trying to refactor – not to mention it’s been stated that using a hidden form to do the work is ‘bad practice’.)

This is what I have:

        string formParams = string.Format("j_username={0}&j_password={1}", userName, userPass);
        string cookieHeader;

        WebRequest request = WebRequest.Create(_signInPage);
        request.ContentType = "text/plain";
        request.Method = "POST";
        byte[] bytes = Encoding.ASCII.GetBytes(formParams);
        request.ContentLength = bytes.Length;
        using (Stream os = request.GetRequestStream())
        {
            os.Write(bytes, 0, bytes.Length);
        }
        WebResponse response = request.GetResponse();
        cookieHeader = response.Headers["Set-Cookie"];

        WebRequest getRequest = WebRequest.Create(sessionHistoryPage);
        getRequest.Method = "GET";
        getRequest.Headers.Add("Cookie", cookieHeader);
        WebResponse getResponse = getRequest.GetResponse();
        try
        {
            using (StreamReader sr = new StreamReader(getResponse.GetResponseStream()))
            {
                textBox1.AppendText(sr.ReadToEnd());
            }
        }
        catch (Exception ex)
        {
            MessageBox.Show(ex.Message);
            throw;
        }

So far, I’m able to get to the proper page from the first link but when I go to the second, it sends me back to the login page as if I didn’t log in.

The problem may lie in cookies not getting captured correctly but I’m a novice so maybe I’m just doing it wrong. It captures the cookies sent back from the POST: JSESSIONID and S2V however, when we go to the “GET”, using FireFox WebConsole, the browser shows that it sends JSESSIONID, S2V and a SPRING_SECURITY_REMEMBER_ME_COOKIE, which I believe is the cookie used when I click the “Remember Me” box on the login form.

I’ve tried many different ways of doing this using the resources of SO but I have yet to get to the page I need. So, for the sake of the hair I have left, I’ve decided to ask for help on good ole SO. (This is one of those things I don’t want to let up on – stubborn like that sometimes)

If someone wants the actual address of the site I’m trying to log into, I’d be more than happy to send it to a couple people in a private message.

Code that I have to reflect a suggested answer by Wal:

var request = (HttpWebRequest)WebRequest.Create(sessionHistoryPage);
request.Credentials = new NetworkCredential(userName, userPass);
request.CookieContainer = new CookieContainer();
request.PreAuthenticate = true;


WebResponse getResponse = request.GetResponse();
try
{
     using (StreamReader sr = new StreamReader(getResponse.GetResponseStream()))
     {
          textBox1.AppendText(sr.ReadToEnd());
     }
}
catch (Exception ex)
{
     MessageBox.Show(ex.Message);
     throw;
}

This suggestion, at least the way I implemented it, didn’t work.

As Krizz suggested, I changed the code to use CookieContainer and transferring the cookies from one request to the other however, the response just gives me back the original login page as if I didn’t login.

Are there certain sites that just WILL NOT allow this type of behavior?

Final Solution

The final solution was proposed by Adrian Iftode where he stated that the website I’m trying to log in might not allow to have an authentication without a valid session so adding a GET to the beginning of the process allowed me to get that cookie.

Thanks for all your help guys!

Answer 1

I was doing some sort of cookie transfer for a website written with PHP.
Clearly you are passing the cookie, but maybe is like in that situation

var phpsessid = response.Headers["Set-Cookie"].Replace("; path=/", String.Empty);

The Set-Cookie header contains other related info about the cookie and possible other instructions for other cookies. I had one cookie with its info (Path), the session id which I needed to sent back to the server so the server would know that I am the same client which did the GET request.

The new request had to include this cookie

request.Headers["Cookie"] = phpsessid;

You already do this, but make sure the cookies you receive, you sent back to the server.

Considering session and authentication, there are two cookies, one for session, one for authentication and some servers/application might not allow to have an authentication without a valid session. What I want to say is that you might need to pass the session cookie too. So the steps would be:

• Do first a GET request to obtain the session cookie.
• Next do the POST request to authenticate and get the auth cookie.
• Use both cookies to navigate to the protected pages.

Also check this question, it doesn’t show the entire class, but the idea is to keep the CookieContainer in the same class, add the new cookies from POST/GET requests and assign them to the each new request, like @Krizz answered.