I am developing software in C#/.NET but I guess the question can be asked for other programming languages as well. How do you test performance of your software between release versions? Let me elaborate some more.
Before the production release of the software, I would like to compare the performance of the software for a set of functions which were available in previous release of the software. Let’s assume that you are talking about a software library project (no GUI) which results in the release of one or more dll’s. How would one achieve this? What are some of the best practices? I cannot possibly swap the current dll with the previous release dll and run the same test.
One way I can think of is to add the same performance test in the main branch (which is used for current release) and the earlier release branch and then compare the performance. I think that there is some pain involved in doing this, but is possible.
Another way I can think of is start with the lest release branch, stub out the new codes and features which has been put in after the last release, and then run the test. I do not think this would produce correct result, not to mention that this approach is even more painful than the previous approach.
Thanks for other ideas. Would prefer C#/.NET specific answers.
We have a suite of performance tests. These are just NUnit tests. Each test sets up some objects, starts a timer (Stopwatch works well), does the operation we’re interested in (e.g., loading the data for a certain screen), and then writes the elapsed time to a CSV file. (NUnit logs how long each test takes, but we want to exclude the setup logic, which in some cases will vary from test to test — so doing our own timers and logging makes more sense.)
We run these tests from time to time, always on the same hardware and network environment. We import the results into a database. Then it’s easy to build graphs that show trends, or that call out large percentage changes.