I am currently using HtmlUnit and Selenium to drive it (WebDriver) within my production code.
I am scaping and interacting with various websites programmatically with these libraries and am having some success and not experiencing memory issues (ensuring sessions are always cleaned up).
I am wondering if these libraries are okay for a production environment or recommended against. This is difficult to find via Google due to the enormous amount of information about automated testing rather than how I am using them.
I realise this is a fairly generic question, but I am seeking advice on these libraries and potentially better alternatives.
WebDriver and Selenium are perfectly suited for production environnement. I use them quite extensively for 2 years now on a multi-machines/multi-datacenters distributed grid and had absolutely no performance nor stability problems we couldn’t have coped with.
Our preferred driver is the Firefox one (heavier than HTMLUnit, and harder to configure), and we had to tweak the grid to understand how many instances we can run. Our maximum for stability was 1 per core
Our selenium/webdriver instances have run 24/7 for 2 years now (1 year with selenium 1, and the other migrating selenium 2/ WebDriver incrementally) and with an appropriate monitoring (you should monitor Memory Usage/CPU Usage) and a bunch of load testing, we had reached the good level where we have experienced several monthes without restarting a process
We’ve used HTMLUnit extensively too, and are equally satisfied with this library
The essential point of my post is : YES, these library are production-ready. But, as all production software, you’ll have to benchmark their use to find the appropriate configuration for the optimal stability. I recommend you to use the Selenium Grid in production, which is a great way to parallelize process