Question

I’m coding a package for experimenting with evolutionary algorithms, and needless to say, it includes a lot of stochastic methods. Now, I’d like to write some (doc)tests for this package, so I can verify that everything works, but I run into situations where the test should simply be true “most of the time”. It feels like I’m probably approaching this the wrong way, but I’d still like to hear some of your thoughts on this.

For example, I have something like this in my doctests:

>>> a = Genome()
>>> b = Genome()
>>> a.mutate()
>>> a != b
True # Well, most of the time.

Implementing tests like that would mean that the test will sometimes fail while everything is working.

I read the suggestion to fix the RNG seed before doing tests, but then I would have to make sure that everything works before I can write the test, since the test should include the expected result.

Answer 1

You could make the probability that it fails negligible, e.g.

a = Genome()
genomes = []
for i in range(100):
    b = Genome()
    b.mutate()
    genomes.append(b)
assert any(a != b for b in genomes)

If your original test succeeded most of the time, this one will succeed always for all practical purposes.

This test could also impose reasonable limits on the number of Genomes that are allowed to match.

Arguably the test reads less nice than the original one. Maybe using doctests is the wrong approach here, and you should write separate unit tests.