I am working on a C++ application that uses computer vision techniques to identify various types of objects in a sequence of images. The (1000+) images have been hand-classified, so we have an XML file for each image containing a description of where the objects are actually located in the images.
I would like to know if there is a testing framework that can understand/graph results from tests that are numeric, in this case some measure of the error in the program’s classification of the images, rather than just pass/fail style unit tests.
We would like to use something like CDash/CTest for running these automated tests, and viewing over time how improvements to the vision algorithms are causing the images to be more correctly classified.
Does anyone know of a tool/framework that can do this?
I think you should distinguish between Unit testing and algorithm performance (=accuracy and/or speed) evaluation. You should apply both, but separately.
Unit testing should tell you whether your code does what it’s supposed to be. Not sure if/how you can unit test the whole chain from a raw image to extracted objects, but you should be able to test the “units” (modules/methods/classes) individually that are combined to do the job. Unit tests should give you “fail” or “pass”. If a speed optimization changes the code’s behavior, the unit test should tell you this. For unit testing there are plenty of frameworks available (I like Google Test, but there are many others.)
Your question seems to aim more at the second part: evaluate the quality of your algorithm. I personally love TeamCity which is mainly intended as Java/.net Continuous Integration Server, but you can easily use it with C++ too. I wrote a few lines of code in our shop to output Google Test results in a TeamCity format making use of their service API. Each time someone checks in a new revision, TeamCity executes the build (which can be a Visual Studio solution, Ant, command line script or others.) The results are visible to all team mates through a nice web ui. Furthermore, you can report custom build statistics. This can be used for anything like perfomance testing of your algorithms. You simply output a line like
on the console from your application (which must be configured to run in each build) and TeamCity will store these values and provide a nice graph (values over time) on the web user interface.
Don’t forget to setup your custom chart as described here.
I think TeamCity is really simple to setup, so just give it a try! I even like it if I work on a project just by myself!