We already have a continuous integration process going where we build, run unit tests, do static code analysis and generate documentation. However, we would like expand this to include automatic performance testing. In this case, we are working on a .NET web application.
We have done some performance testing with JMeter (outside the CI process), but I don’t know if this is the best tool to include in a CI process? Is Selenium an option? WAPT Pro?
On which levels should we test the performance? Should we have a set of “performance unit tests”? Should we run JMeter (or something similar) on a production-like environment and fail if any requests takes > 1 second? Wouldn’t something like this have too high variance?
So, do you guys include automatic performance testing as part of your CI? What do you test, and which tools do you use? What has your experience been like?
First off, JMeter is a good choice for inclusion in CI because it can be run from the command line and you can pass in variables when you do this. I would recommend it for this task.
In general though, integrating Perf. testing into CI is difficult. You’ve already listed many of the reasons why this is so you’re already half way there because you understand the limitations. And that’s the rub: it IS possible to have Perf. tests in CI but only to a limited extent.
I think that a good approach follows some of these principles:
You can’t run full load (or soak or capacity) tests in CI, it’s not practical.
The results are subjective and need human interpretation and it takes time to run the tests. But you can run a simpler, cut down set of tests that measure response times for requests and then you can evaluate these response times either:
You can also run automated load / perf. tests – at full volume – outside of the build process. ‘Semi CI’.
So maybe you could automate a test to run overnight and then check the results in the morning?
Iterate.
Just start doing it and getting results and fine tune the tests and how you interpret them over time. Keep it simple and focus on the areas that appear to be useful. Don’t launch with a fanfare, keep it quiet until you have confidence with the process and then start failing builds and telling people about it – initially, you’re likely to get lots of false negatives.
Instrument your results
Do this. A lot. CI is all about failing early so if you take that as you end objective then the best way to achieve it is to run tests early and often but the problem with that is you risk getting buried in data. So an effective method to crunch the data and present the relevant information helps considerably.
You can’t automate the whole process down to Red Flag Green Flag – but you should try to get as far down that path as possible.
Finally, there was a very good talk given by the lead Perf. tester at Google that covers this subject. It’s a bit old now but the principles still stand. Plus, in a few weeks I’m going to a meetup where Channel4, a British media company, will be talking about how they approached this – maybe you can ask for some slides.