Question

I want to read a source code (HTML tags) of a given URL from my servlet.

For example, URL is http://www.google.com and my servlet needs to read the HTML source code. Why I need this is, my web application is going to read other web pages and get useful content and do something with it.

Lets say, my application shows a shop list of one category in a city. How that list is generated is, my web application (servlet) goes through a given web page which is displaying various shops and read content. With the source code my servlet filters that source and get useful details. Finally creates the list (because my servlet has no access to the given URL’s web applications database).

Any know any solution? (specially I need this to do in servlets) If do you think that there is another best way to get details from another site, please let me know.

Thank you

Answer 1

What you are trying to do is called web scraping. Kayak and similar websites do it. Do search for it on web 😉 Well in java you can do this.

URL url = new URL(<your URL>);

BufferedReader in = new BufferedReader(new InputStreamReader(url.openStream()));
String inputLine;
StringBuffer response = new StringBuffer();

while ((inputLine = in.readLine()) != null) {
  response.append(inputLine + "\n");
}

in.close();

response will give you complete HTML content returned by that URL.