Question

I’m (re)writing a program that does generic unit test scheduling. The current program is a mono-threaded Perl program, but I’m willing to modularize it and parallelize the tests. I’m also considering rewriting it in Python.

Here is what I need to do:

• I have a list of tests, with the following attributes:
  • uri: a URI to test (could be HTTP/HTTPS/SSH/local) ;
  • depends: an associative array of tests/values that this test depends on ;
  • join: a list of DB joints to be added when selecting items to process in this test ;
  • depends_db: additional conditions to add to the DB request when selecting items to process in this test.
• The program builds a dependency tree, beginning with the tests that have no dependencies ;
• for each test:
  • a list of items is selected from the database using the conditions (results of depending tests, joints and depends_db) ;
  • the list of items is sent to the URI (using POST or stdin) ;
  • the result is retrived as a YAML file listing the state and comments for the test for each tested item ;
  • the results are stored in the DB ;
  • the test returns, allowing depending tests to be performed.
• the program generates reports (CSV, DB, graphviz) of the performed tests.

The primary use of this program currently is to test a fleet of machines against services such as backup, DNS, etc. The tests can then be:
– backup: hosted on the backup machine(s), called through HTTP, checks if the machines’ backup went well ;
– DNS: hosted on the local machine, called via stdin, checks if the machines’ fqdn have a valid DNS entry.

Does such a tool/module already exist? What would be the best implementation to achieve this (using Perl or Python)?

Answer 1

This sort of testing is not unit testing.

If you decide not to follow rjh’s advice in order to keep most of the code you’ve already written, then convert your tests to use the standard test toolchain.

• Use Test::Class and friends to emit TAP.
• Use prove‘s -j option or Test::Aggregate to run tests in parallel.
• Use prove’s -a option or App::Prove::History to record results.
• Use TAP::Formatter::HTML for graphical reporting.