Question

Say, I have the following test:

    [Test]
    public void MyTest( [RandomNumbers( Count=100, Minimum=0, Maximum=1000 )] int number )
    {
        ...
    }

And at some point, during my regular build process, it has failed. I got an e-mail notification and set off to investigate.

Now, when I open the test in Visual Studio and click “Run Tests”, it passes. I do it again, and it passes again. And again. And again. Obviously, the failure was related to that particular sequence of random numbers.

So the question is: How do I re-run this test with that exact sequence?
(provided I have full Gallio report)

 
 
UPDATE:
(following a comment about it being a bad idea)

First, I’m not actually asking whether it’s a good idea. The question is different.

Second, when the system being tested is complex enough, and the input data space is of multiple independent dimensions, properly breaking that space into equivalency regions presents a significant challenge in both mental effort and time, which is just not worth it, provided smaller components of the system have already been tested on their own. At the same time, if I can just poke the system here and there, why not do so?

Third, I am actually not a newbie in this area. I always used this technique with other test frameworks (such as csUnit and NUnit), and it proved very successful in catching subtle bugs. At the time, there was no such concepts as generated data, so we used our own custom crutches, in the form of System.Random with a predetermined seed. That seed was being generated as part of fixture initialization (usually based on current time) and carefully written to log. This way, when the test failed, I could take the seed from log, plug it into the test fixture, and get exactly same set of test data, and thus, exactly same failure to debug.

And fourth, if it is such a bad idea, why does the RandomNumbers factory exist in the first place?

Answer 1

There is currently no built-in way in Gallio/MbUnit to generate again the same sequence of random numbers. But I think this could be a useful feature and I did open an issue for that request. I’m going to update the subject answer when it’s ready.

What I propose is the following:

• Display the actual seed of the inner random generator as an annotation into the test report.
• Expose a Seed property to the [RandomNumbers] and [RandomStrings] attributes, and to the fluent data generators as well.

Thus you could easily re-generate the exact same sequence of values by feeding the generator with the same seed number.

UPDATE: This feature is now available in Gallio v3.3.8 and later.

---

Now we all agree with what Péter said. Using random numbers as input for unit tests is rarely a good idea. The corollary is that it is sometimes convenient and mostly appropriate. And that is exactly the reason why we decided to implement that feature in MbUnit. IMHO, a common scenario that could fit well with random test input is stochastic analysis on hash code computations.