Question

I need to download a CSV file from Google insights programatically. Since it requires authentication, I used the clientLogin to get the session id.

How do I download the file by passing the session id as a cookie?

I tried using a new URLConnection object and set the cookie in setRequestParameter method hoping it would authenticate my login then, however it doesn’t seem to be working. I have a feeling I shouldn’t use two separate connections, is that true?

If so then how do I pass session id as parameter when i download the file? I also tried using the same connection this didn’t work either. Please help.

try {
  URL url1 = new URL("https://www.google.com/accounts/ClientLogin?accountType=GOOGLE&Email=*******.com&Passwd=*****&service=trendspro&source=test-test-v1");
  URL url2 = new URL("http://www.google.com/insights/search/overviewReport?cat=0-7&geo=BR&cmpt=geo&content=1&export=1");
  URLConnection conn = url1.openConnection();

  // fake request coming from browser
  conn.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11");
  BufferedReader in = new BufferedReader(new InputStreamReader(conn.getInputStream(), "UTF-8"));
  String f = in.readLine();
  // obtaining the sid.
  String sid=f.substring(4);
  System.out.println(sid);

  URLConnection conn2 = url2.openConnection();
  conn2.setRequestProperty("Cookie", sid);
  BufferedInputStream i= new BufferedInputStream(conn2.getInputStream());
  FileOutputStream fos = new FileOutputStream("f:/testplans.csv");
  BufferedOutputStream bout = new BufferedOutputStream(fos,1024);
  byte data[] = new byte[1024];

  while(i.read(data,0,1024)>=0) {
    bout.write(data);
  }
  bout.close();
  in.close();
}

Answer 1

Try the following: link. Check the top answer: they don’t use the SID, but the Auth.

If it’s working for Google Reader, it will probably work for Google Insights as well.