Question

I am looking for a test framework to introduce automated tests for a language without much test support. As far as I can understand, I need a framework that’s capable of running the VDF tests using some form of protocol. I would much rather spend my time writing tests than writing VDF code to interface with the test framework, so a lightweight protocol is much preferred.

Slim with Fitnesse seems to be an nice candidate, but I’m interested in all recommendations.

Being able to use the same test framework across programming languages would be an added bonus.

Answer 1

This assumes you are working at the API level. If I read it wrong, and you are working at the GUI level, you’re more apt to consider something like selenium or watir.

Have you considered writing your own simple test framework that outputs TAP results (the test anything protocol) – then parsing that with grind or TAP2HTML?

Seriously, TAP output looks like this:

1..7
ok 1 - hello world with null value returns 'hello world' string
ok 2 - hello world with bob returns 'hello, bob'
ok 3 - hello world with 123 return 'hello, 123'
ok 4 - hello world with 5K string return hello plus string
ok 5 - special characters
ok 6 - internationalization, fr
ok 7 - internationalization, ja
Looks like you failed 0 tests of 7.

(If it died after step 5, the 1..7 would tell you something’s wrong)

The output is straight ASCII. You’ll basically have two globals, numTestsTotal and numTestExecuted, and write functions like this:

sub ok (boolean bExpected, string comment) {
  if (bExpected) {
    print "ok " . numTestsExecuted . " "  . comment . "\n";    
  }else {
    print "not ok" . numTeststotal . " " . comment . "\n";
  }
  numTestsExecuted++;
}


sub eq(int iExpected, int iResult, string comment) {
  if (iExpected==iResult) {
     print "ok " . numTestsExecuted . " " . comment . "\n";
  } else {
     print "not ok" . numTestsExecuted . " " . comment . \n";
  }
  numTestsExecuted++;
}

You write the regular code in a library, then your test app include the library and the test module.

You can overload eq for every type of value, and write is to compare arrays, etc.

See documentation on TAP:
http://testanything.org/wiki/index.php/Main_Page

And Test::More

Yes, you could argue that eq() should “just” call ok(). Or you could insert expected and actual results in the output. Up to you.

Anyway, there are lots of TAP parsers and interpreters for more imperative languages.