Question

I’ve followed a tutorial and came up with the following method to read the webpage content into a CharSequence

 public static CharSequence getURLContent(URL url) throws IOException {
       URLConnection conn = url.openConnection();
       String encoding = conn.getContentEncoding();
       if (encoding == null) {
         encoding = "ISO-8859-1";
       }
       BufferedReader br = new BufferedReader(new
           InputStreamReader(conn.getInputStream(),encoding));
       StringBuilder sb = new StringBuilder(16384);
       try {
         String line;
         while ((line = br.readLine()) != null) {
           sb.append(line);
           sb.append('\n');
         }
       } finally {
         br.close();
       }
       return sb;
     }

It will return a representation of the webpage specified by the url.
However,this representation is hugely different from what I use “view page source” in my Firefox,and since I need to scrape data from the original webpage(some data segement in the original “view page source” file),it will always fail to find required text on this Java representation.
Did I go wrong somewhere?I need your advice guys,thanks a lot for helping!

Answer 1

You need to use an HTML-parsing library to build a data structure representing the HTML text on this webpage. My recommendation is to use this library: http://htmlparser.sourceforge.net.