Question

I’m in trouble with some url’s from a web-store called Kabum.

The url is http://www.kabum.com.br/cgi-local/kabum3/produtos/descricao.cgi?id=01:02:23:55:159

If I enter the site in the address bar, or click the link, I got a page with the product, but If I use Jsoup, I get a page with only a meta refresh to the same address.

Tried setting the user agent, the referrer and follow the link in meta, but I got the same page.

My code is here:

Document doc;
String url = "http://www.kabum.com.br/cgi-local/kabum3/produtos/descricao.cgi?id=01:02:23:55:159";
try {
    String ua = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:12.0) Gecko/20100101 Firefox/12.0";
    String referrer = "http://www.google.com";
    doc = Jsoup.connect(url).timeout(20000).userAgent(ua).referrer(referrer).get();
    Elements meta = doc.select("html head meta");
    for (Iterator<Element> it = meta.iterator(); it.hasNext();) {
        Element element = it.next();
        if (element.attr("http-equiv").matches("refresh")) {
            String novaUrl = element.attr("content").replaceFirst("\\d?;url=", "");
            System.out.printf("redirecting to %s%n", novaUrl);
            doc = Jsoup.connect(novaUrl).userAgent(ua).referrer(referrer).get();
            break;
        }
    }
} catch (IOException ex) {
    Logger.getLogger(Teste1.class.getName()).log(Level.SEVERE, null, ex);
    return;
}
System.out.println(doc);

Answer 1

You need to re-send a request with the cookies. The site is returning one session cookie which it expects to see in the next request.

String url = "http://www.kabum.com.br/cgi-local/kabum3/produtos/descricao.cgi?id=01:02:23:55:159";
Map<String, String> cookies = Jsoup.connect(url).execute().cookies();
Document document = Jsoup.connect(url).cookies(cookies).get();
System.out.println(document.html());

Note that you should use the same cookies on every subsequent request you’d like to fire in the same session.