Question

I am trying to retrieve the whole overview section for this url

What would be the elements i look for in the three different articles?

http://xbox360.gamespy.com/xbox-360/project-dark/

Is there anyway to create a default selector to retrieve the overview for this page also?

http://wii.gamespy.com/wii/ben-10-galactic-racing/

EDIT
http://wwww.gamespy.com/pc/6-great-games

i would like to make a selector for the different tags. So if one a url is selected if it has the selector then it will load the data, if it doesnt it will try another selectory.

How could i go about doing this?

Is it possible to create different selectors looking for different tags for the different articles?

Answer 1

This should get the overview text on all three webpages

// Get the overview div
Element overview = doc.select("div#object-overview").last();

// Get the paragraph element
Element paragraph = overview.select("p").last();
System.out.println(paragraph.text());

As for different selectors for different webpages, you could do something like a HashMap.

// Create new HashMap
HashMap<String, String> selectorMap = new HashMap<String, String>();

// Put the Key-Value pair in the Hashmap
selectorMap.put("http://wii.gamespy.com/wii/ben-10-galactic-racing/", "div#object-overview");

// Get the value by supplying the key (the webpage's url)
String selector = selectorMap.get("http://wii.gamespy.com/wii/ben-10-galactic-racing/");

Let me know if this is what you were looking for.

To get the Feature List:

// Get the overview div element
Element featureList = doc.select("div.callout-box").last();

Elements features = featureList.select("li");

ListIterator<Element> featList = features.listIterator();
while (featList.hasNext()) {
    System.out.println(featList.next().text() + "\n");

}

To get the Edition List:

// Get the div.columns element - this is the base of each edition
Elements editions = doc.select("div.columns");

ListIterator<Element> editionsList = editions.listIterator();
while (editionsList.hasNext()) {
    // Get that edition
    Element edition = editionsList.next();

    // Get the edition name element
    Element editionName = edition.select("h3").first();
    System.out.println(editionName.text());

    // Get the edition info element
    Element editionInfo = edition.select("p").last();
    System.out.println(editionInfo.text() + "\n");

}