What are opinions on the design of a ‘performance assertion checking’ system?
The idea is that a developer makes some assertions about his/her code and use these to test the evolution of the performance of the code. What is the experience with such a system?
My current block is ‘What’s a better way to translate these assertions, written in a specified language (that are to be checked against specified logs or runtime instrumentation) into, say, CLR, or assembly or bytecode that could be executed?’
Currently I have written a parser that parses the specification and holds it in a data structure.
Do we embed performance checks in our application? No. The reason is that the performance checks themselves take time and our application is very sensitive to performance.
Instead, we make our performance checks a test. And for that we use NUnit. For our nightly builds, we run the test, we generate a log with detailed timing data as well as a pass/fail indication given our requirements. Since we keep our logs around for some time — forever for beta and production releases — we can track performance over time as well.