I’ve got a legacy system that processes extremely complex data that’s changing every second. The modularity of the system is quite poor so I can’t split the business logic into smaller modules to ease functional testing.
The actual test system is: “close your eyes click and pray“, which is not acceptable at all. I want to be confident on the changes we commit on the code.
What are the test good practices, the bibles to read, the changes to operate, to increase confidence in such a system.
The question is not about unit testing, the system wasn’t designed for that and it takes too much time to decouple, mock and stub all the dependencies and most of all, we sadly don’t have the time and budget for that. I don’t want to a philosophic debate about functional testing: I want facts that work in real life.
It sounds like you have yourself a black box as regards testing.
http://en.wikipedia.org/wiki/Black-box_testing
To put it simply, it’s horrible, but may be all you can do if you can’t isolate the system in any way.
You need to insert known data into your system and compare the result with the known output.
You really need known data & output for
normal values – normal data – you’ll find out that it can at least seem to do the right thing
erroneous values – spelling errors, invalid values – so you know that it will tell you if the input is rubbish
out of range – -1 on signed integers, values greater than 2.7 billion (assuming 32bit), and so on – so you know it won’t crash out on seriously mis-inputted or corrupted data
dangerous – input that would break the SQL, simulate SQL injection
Lastly make sure that all errors are being carefully handled rather than getting logged and the bad/corrupt/null value getting passed on through the system.
Any processes you can isolate and test that way will make debugging easier, as black box testing can’t tell you where the error occurred. This means then you need to diagnose the errors based on what happened, more in the style of House MD than a normal debugging session.
Once you have the different data types listed above, you can test all changes in isolation with them, and then in the system as a whole. Over time as you eventually touch most aspect of the system, you’ll have test cases for all areas, and be able to say where failure was most likely to have occurred more easily.
Also: make sure you put tracers in your known data so you don’t accidentally indicate a stockmarket crash when you’re testing the range limits on a module, so you can take it out of the result flow before it ends up on a CEO’s desk.
I hope that’s some help