Home/ Questions/Q 6537741
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-25T10:39:11+00:00

I have a problem when trying to parse a webpage using jSoup. If I

  • 0

I have a problem when trying to parse a webpage using jSoup. If I view the page source in chrome, there is all the content you’d expect – the full page. However, when I connect to the url with JSoup, the Jsoup Document only contains a small amount of the page’s html/xml.

Any idea why this is?

Below is an example of the differences:

Page source:

<title>Bath Rugby Official Website | First team</title> 
<base href="http://www.bathrugby.com/"></base> 
<meta name="description" content="The Official Website of Bath Rugby. The only place     for in-depth player, match and club news." /><meta property="og:title" content="Bath Rugby     Official Website | First team" /><meta property="og:url"     content="http://www.bathrugby.com/fixtures-and-results/first-team/" /> 
<meta property="og:type" content="sports_team" /> 
<!-- <meta property="og:image"     content="http://www.bathrugby.com/_assets/img/common/bath_rugby_logo.jpg" /> --> 
<meta property="og:site_name" content="Bath Rugby" /> 
<meta property="fb:admins" content="100001941260923" /> 


<link rel="alternate" type="application/rss+xml" title="Bath Rugby - Latest News - RSS feed" href="/rss/rss-all-news.rss" /> 
<link rel="alternate" type="application/rss+xml" title="Bath Rugby - Match News - RSS   feed" href="/rss/rss-match-news.rss" /> 
<link rel="alternate" type="application/rss+xml" title="Bath Rugby - Team News - RSS feed" href="/rss/rss-team-news.rss" /> 
<link rel="alternate" type="application/rss+xml" title="Bath Rugby - Club News - RSS feed" href="/rss/rss-club-news.rss" /> 
<link rel="alternate" type="application/rss+xml" title="Bath Rugby - Academy News - RSS feed" href="/rss/rss-academy-news.rss" /> 
<link rel="alternate" type="application/rss+xml" title="Bath Rugby - Ladies News - RSS feed" href="/rss/rss-ladies-news.rss" /> 
<link rel="alternate" type="application/rss+xml" title="Bath Rugby - Community News - RSS feed" href="/rss/rss-community-news.rss" /> 
<link rel="alternate" type="application/rss+xml" title="Bath Rugby - Programmes and Projects - Inclusion - RSS feed" href="/rss/rss-programmes-and-projects-inclusion.rss" /> 
<link rel="alternate" type="application/rss+xml" title="Bath Rugby - Programmes and Projects - Education - RSS feed" href="/rss/rss-programmes-and-projects-education.rss" /> 
<link rel="alternate" type="application/rss+xml" title="Bath Rugby - Programmes and Projects - Sports - RSS feed" href="/rss/rss-programmes-and-projects-sport.rss" /> 
<link rel="alternate" type="application/rss+xml" title="Bath Rugby - Corporate News - RSS feed" href="/rss/rss-corporate-news.rss" /> 
<link rel="alternate" type="application/rss+xml" title="Bath Rugby - Redevelopment News - RSS feed" href="/rss/rss-redevelopment-news.rss" />

JSoup Document:

<head> 
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> 
  <title>Bath Rugby - Fixtures and results</title> 
  <meta name="Author" content="Positive -&gt; http://www.positivestudio.co.uk" /> 
  <meta name="HandheldFriendly" content="True" /> 
  <meta name="viewport" content="width=device-width; initial-scale=1.0; minimum-scale=1.0; maximum-scale=1.0; user-scalable=0;" /> 
   <meta name="google-site-verification" content="7dRcSqMI5R3Rag9iNiHeFei6ycyBTd3dmhlZvF8QdhA" /> 
   <link rel="Shortcut Icon" type="image/ico" href="/_assets/favicon.ico" /> 
  <link rel="apple-touch-icon-precomposed" href="/_assets/img/icons/apple-touch-icon.png" /> 
   <link rel="stylesheet" type="text/css" media="all" href="/_assets/css/mobile.css" /> 
   <link rel="stylesheet" type="text/css" href="/_assets/css/retina.css" media="only screen and (-webkit-min-device-pixel-ratio: 2)" /> 
  <script type="text/javascript" src="/_assets/js/jquery-1.4.2.min.js"></script> 
   <script src="/_assets/js/css_browser_selector.js" type="text/javascript"></script> 
  <script type="text/javascript" src="/_assets/js/scripts.js"></script> 
  <script type="text/javascript">

    var _gaq = _gaq || [];
    _gaq.push(['_setAccount', 'UA-24788688-3']);
    _gaq.push(['_trackPageview']);

  (function() {
Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    I answered this by retrieving the page’s source html as a string, then passing this to jSoup.

    HttpClient client = new DefaultHttpClient();
            HttpGet request = new HttpGet(url);
            HttpResponse response = client.execute(request);

            String html = "";
            InputStream in = response.getEntity().getContent();
            BufferedReader reader = new BufferedReader(new     InputStreamReader(in));
            StringBuilder str = new StringBuilder();
            String line = null;
            while((line = reader.readLine()) != null)
            {
                str.append(line);
            }
            in.close();
            html = str.toString();              

            doc = Jsoup.parse(html);
    • 0