Question

So there is this website that has a login form. I want to log in and then download a file. When submitting the form, not only do username and password get transmitted in the http POST but also a token that is in a hidden <input> tag.
Now, my Problem is that whenever I open the URL in java and get the token to make a POST the token is invalid when I use a HttpClient.
I somehow need to use the same client for calling the website to get the token and making the post. Unfortunately I get a 403 FORBIDDEN return code when trying to access the file.
This is what I have so far:

public static void main(String[] args){



    try {
        String token = getTokenFromPage("http://my.url");   

        HttpContext context = new BasicHttpContext();   
        DefaultHttpClient client = new DefaultHttpClient();         

        List <NameValuePair> parameters = new ArrayList <NameValuePair>();
        HttpPost post = new HttpPost("http://my.url");
        parameters.add(new BasicNameValuePair("username", "MYNAME"));
        parameters.add(new BasicNameValuePair("password", "MYPW"));
        parameters.add(new BasicNameValuePair("token", token));

        UrlEncodedFormEntity entity = new UrlEncodedFormEntity(parameters, HTTP.UTF_8);
        post.setEntity(entity); 

        System.out.println("URL: " + post.getURI());
        HttpResponse postResponse = client.execute(post, context);
        System.out.println(postResponse.getStatusLine());
        EntityUtils.consume(postResponse.getEntity());
//Now download the file

        HttpGet httpget = new HttpGet("http://url.to.file");

        HttpResponse getResponse = client.execute(httpget, context);

        System.out.println(getResponse.toString());

        System.out.print((postResponse.getEntity().getContent()));
        client.getConnectionManager().shutdown();


    } catch (Exception e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    }
}

Answer 1

You are going to have to make an HTTP request for the login page, parse the resulting HTML in the HTTP response stream, and get the token value to use from there. Using a library like jsoup to parse the HTML would be advisable.