Question

I use the code below to fetch the first two pages of google search results
but i can only fetch the first page(when search page 2, it is the same with page 1)

import com.gargoylesoftware.htmlunit.WebClient;
import com.gargoylesoftware.htmlunit.html.HtmlElement;
import com.gargoylesoftware.htmlunit.html.HtmlPage;
import com.gargoylesoftware.htmlunit.html.HtmlTextInput;


/**
 * A simple Google search test using HtmlUnit.
 *
 * @author Rahul Poonekar
 * @since Apr 18, 2010
 */
public class Author_search {
    static final WebClient browser;

    static {
        browser = new WebClient();
        browser.setJavaScriptEnabled(false);
    }

    public static void main(String[] arguments) {
            searchTest();
    }

    private static void searchTest() {
        HtmlPage currentPage = null;

        try {
            currentPage = (HtmlPage) browser.getPage("http://www.google.com");
        } catch (Exception e) {
            System.out.println("Could not open browser window");
            e.printStackTrace();
        }
        System.out.println("Simulated browser opened.");

        try {
            ((HtmlTextInput) currentPage.getElementByName("q")).setValueAttribute("xxoo");
            currentPage = currentPage.getElementByName("btnG").click();
            System.out.println("contents: " + currentPage.asText());
            HtmlElement next = (HtmlElement)currentPage.getByXPath("//span[contains(text(), 'Next')]").get(0);
            currentPage = next.click();
            System.out.println("contents: " + currentPage.asText());
        } catch (Exception e) {
            System.out.println("Could not search");
            e.printStackTrace();
        }
    } 
}

can anybody tell me how to fix this?

by the way:

1. How to change the language settings in google using htmlunit? any
   convenient ways?
2. Does htmlunit treat the html like “firebug” in
   firefox, or just treat it like the texts in “file->save”.In my
   opinion, I believe it treat it like it was a explorer, am i right?

Answer 1

I replaced:

HtmlElement next = (HtmlElement)currentPage.getByXPath("//span[contains(text(),'Next')]").get(0);
currentPage = next.click();

with:

HtmlAnchor nextAnchor =currentPage.getAnchorByText("Next");
currentPage = nextAnchor.click();