I have a completely non-interactive python program that takes some command-line options and input files and produces output files. It can be fairly easily tested by choosing simple cases and writing the input and expected output files by hand, then running the program on the input files and comparing output files to the expected ones.
1) What’s the name for this type of testing?
2) Is there a python package to do this type of testing?
It’s not difficult to set up by hand in the most basic form, and I did that already. But then I ran into cases like output files containing the date and other information that can legitimately change between the runs – I considered writing something that would let me specify which sections of the reference files should be allowed to be different and still have the test pass, and realized I might be getting into “reinventing the wheel” territory.
(I rewrote a good part of unittest functionality before I caught myself last time this happened…)
I guess you’re referring to a form of system testing.
No package would know which parts can legitimately change. My suggestion is to mock out the sections of code that result in the changes so that you can be sure that the output is always the same – you can use tools like Mock for that. Comparing two files is pretty straightforward, just dump each to a string and compare strings.