Question

i have to crawl a website which has some secure pages,and ask for username and password for login ,once crawler is started it and it needs to crawl the secure pages so it should ask username and password from user to login so that crawler can crawl the secure pages.when the crawler is loggedin with the user account who is using the crawler it should crawl the pages

Answer 1

ok i did some digging and replying to my own question ,this will help others,using Htmlunit it can be done easliy

    HtmlPage page = (HtmlPage) webClient.getPage("https:website name.com/loginpage");
    HtmlForm form = page.getFormBy("login_form");
        HtmlForm formm= page.getForms().get(0);
    formm.getInputByName("SignInEmailAddress").setValueAttribute("your emailid");
    formm.getInputByName("SignInPassword").setValueAttribute("password");
    page = (HtmlPage) formm.getInputByValue("Login").click();
        System.out.println(page);
        System.out.println(page.asText());