Question

I am thinking of automating performance tests, in the same way than what we currently have with unit tests.

I know how to run performance tests with tools like jMeter or by writing my own code to trigger specific parts of the application. I know how to use time, jvisualvm, nmon or others to gather information about resources being used.

I would like to go further and write a performance test, that would fail if it crosses certain lines (execution time, memory or CPU consumed…). I would then have my CI server (Jenkins) run the tests on a regular basis to ensure the performance remains good.

This is complicated because performance depends on the hardware, and in the current way I do it, it requires human interpretation of results to decide whether this is satisfying or not.

Do you know any tools or frameworks (if possible Java based) that help automate performance tests in that way? If not, do you have some good practice to advice?

Thanks.

Answer 1

In the past, I have used JUnit to do some performance testing. However, it did not need human interpretation – the algorithm either took way too long or it was quick enough. In a way, it was a pass/fail test, based on a time threshold.

If you need subjective performance testing done automatically, I am afraid it will be difficult to build.