Question

I’m trying to scrape an RSS feed located here.

At the moment I’m just trying to wrap my head around JSoup, so the following code is merely proof of concept (or an attempt at it, at least).

    public static void grabShakers(String url) throws IOException {

    doc = Jsoup.connect(url).get();


    desc = doc.select("title");
    links = doc.select("link");
    price = doc.select("span.price");

}

It grabs the title of each item perfectly. The output of each link is simply ten repeated closing link tags and it never finds any prices. I thought perhaps the CDATA was the issue, so I converted doc to html, stripped out the comments using .replace, and then converted it back to a Document for parsing to no avail. Any insight would be greatly appreciated.

The following code is what I’m using to print out each element:

for (Element src : price) {
        System.out.println(src);
    }

Answer 1

There are two Problems with that feed:

1. The document contains only <link />..actual link.. instead of full link tag
2. The description (containing the price tag) is escaped Html, which wont get parsed

Solution:

    final String url = "http://www.amazon.com/gp/rss/movers-and-shakers/appliances/ref=zg_bsms_appliances_rsslink";
    Document doc = Jsoup.connect(url).get();


    for( Element item : doc.select("item") ) // Select all items
    {
        final String title = item.select("title").first().text(); // select the 'title' of the item
        final String link = item.select("link").first().nextSibling().toString().trim(); // select 'link' (-1-)

        final Document descr = Jsoup.parse(StringEscapeUtils.unescapeHtml4(item.select("description").first().toString()));
        final String price = descr.select("span.price").first().text(); // select 'price' (-2-)

        // Output - Example
        System.out.println(title);
        System.out.println(link);
        System.out.println(price);
        System.out.println();
    }

Note 1: Workaround for the link; select the (empty) link tag and get the text of next Node (= TextNode with the actual link).

Note 2: Workaround for price; select the description tag, unescape the html, parse it and select the price. For unescaping i used StringEscapeUtils.unescapeHtml4() from Apache Commons Lang.

Output:
_{(using link from above)}

#1: Epicurean Gourmet Series 20-Inch-by-15-Inch Cutting Board with Cascade Effect, Nutmeg with Natural Core

$72.95

#2: GE 45600 Z-Wave Basic Handheld Remote

$3.00

#3: First Alert RD1 Radon Gas Test Kit

$10.60

#4: Presto 04820 PopLite Hot Air Popper, White

$9.99

#5: New 20 oz Espresso Coffee Milk Frothing Pitcher, Stainless Steel, 18/8 gauge

$8.19

#6: PUR 18 Cup Dispenser with One Pitcher Filter DS-1800Z
http://www.amazon.com/PUR-Dispenser-Pitcher-Filter-DS-1800Z/dp/B0006MQCA4/ref=pd_zg_rss_ms_la_appliances_6
$22.17

#7: Hamilton Beach 70610 500-Watt Food Processor, White

$21.95

#8: West Bend 77203 Electric Can Opener, Metallic
http://www.amazon.com/West-Bend-77203-Electric-Metallic/dp/B00030J1U2/ref=pd_zg_rss_ms_la_appliances_8
$35.79

#9: Custom Leathercraft 2077L Black Ski Glove, Large

$8.83

#10: Cuisinart CPC-600 1000-Watt 6-Quart Electric Pressure Cooker, Brushed Stainless and Matte Black

$64.95