Question

I am trying to use a headless browser in .Net and I need it to execute JavaScript because the information I need to read with it is loaded via AJAX. After some research I believe my best option is HtmlUnit. I have followed the instructions in Steven Sanderson’s blog http://blog.stevensanderson.com/2010/03/30/using-htmlunit-on-net-for-headless-browser-automation/ in the main this works great. However, as hard as I try I cannot get it to wait for the JavaScript to load. Essentially after selecting something from a selectbox an AJAX request fires and populated another selectbox. HtmlUnit never sees the new options.

I have tried combinations of all of the examples in the FAQ http://htmlunit.sourceforge.net/faq.html#AJAXDoesNotWork so I can only assume that it is a problem with the fact it has been converted to run in .Net. I am hoping someone has some experience in this who can shed some light on what is wrong. I feel like I am going the right way with HtmlUnit and it has been recommended on here before Is there a headless browser for .NET (for automated Web-UI-Tests)? and headless internet browser? but I have reached a dead end.

So if anyone has either a better choice for a headles browser that supports JavaScript, and ideas how to get HtmlUnit to wait for the JavaScript or even just some advice on how to debug this further.

Thanks.

Simon

Answer 1

I think you need to use webClient.waitForBackgroundJavaScript() or webClient.waitForBackgroundJavaScriptStartingBefore().

You can also see these links that might help you little.

1. HTMLUnit doesn't wait for Javascript
2. HtmlUnit doesn't load externally referenced javascript
3. htmlunit javascript question