Question

Hello another question concerning debugging : Automatically generating test cases when i know the parameterset. And doing it all at once, instead during development (could kick myself)

i have a set of parameters for my software that i wish to test. (~ 12 parameters only). However of course these parameters are often integers, so for every parameter i can have 4 values that make sense(0, insanely huge, normally big, normally small).

is there a way i can generate my testcases automatically? would save me a lot of time. I already have to inspect every test case by hand, do i not? Alot of my program produces output to the console so normal assertions probably wont work, also i work on home made datastructures most of the time, so i could not use a simple assertion.

My dream option would be kind of a reverse regular expression, where i set the rules and get myself some file generated that i can use as an input (my software has a crude scripting language). that way i can assemble all input files and test them one by one.

Looking forward to listening to your kind suggestions.

cheers

Answer 1

There are lots of ways to generate test cases in your scenario — though you’re a bit vague on what form the inputs for your programs and units need to take. For one of my Fortran programs I use a template input parameter file, a bash script and a make file. The make file, when called on the test phony target:

a) compiles the program;
b) runs the bash script, which uses sed to replace placeholders in the template parameter file, to create 128 (or whatever) test input files;
c) submits all the test jobs to the job management system on our cluster.

Once they jobs have finished I have some other scripts to compare outputs with benchmarks, collect statistics, that sort of thing.

If you need more specific advice, post more specific questions.

EDIT: Using sed inside a bash script:

Suppose that the parameter input template file contains 3 codes to be replaced: $FREQ$, $NUM$ and $TOL$. Then I write a bash script with a 3-deep loop nest something like this:

for frq in 0.01 0.0 1 10
do
    for np in 1 2 4 8 16
    do
        for tol in 0.001 0.0001 0.00001
            sed ....
        done
    done
done

It’s not pretty but it works, and it saves me wrestling with much more sophisticated solutions such as xUnit testing or Python programming.