Question

I am looking for a test framework to cover our black box integration tests. We need something that is scriptable by non developers (aka not C# unit test type stuff).

The initial scenarios I have in mind are:

1. Restore known DB
2. Run sql agent job (ETL)
3. Execute validation sql scripts against output DB

and

1. Run msi install
2. Check existance of Folders/Files/RegKeys/Services/etc
3. run msi uninstall

So far I have not found anything that seems fitting. Mostly UI testing (Project White/etc) which we will be using but does not cover these cases. Or unit test framework based integration testing which we are not ready to push our QA team towards just yet.

I’m currently experimenting on rolling our own internal tool for this part of testing if I can’t find anything else.

Answer 1

It looks like you want to run a bunch of command-line parameters, right?

Well, I see two ways to do it:

1) you could invent your own domain specific language. That’s a fancy way of saying that you write an interpreter with some very very high-level functions. The non-technical people write something like batch files, and you write some C# that reads the file, executes a switch statement, then runs the commands. FIT is probably the most common way to do this – it’s the framework for integrated tests. ( The way to do it is to separate things by commas: command,param1,param2. Pretend it’s an incredibly simple assembler program. Then your switch statement takes param1..paramx and sticks them in an array of strings and passes that to the function. The function processes the array.)

The problem with this is that your customers will want variables. They’ll want looping. And pretty soon, you’ve implemented a turin-complete programming interpreter that reads in data in columner format. That stinks.

So you could …

2) Teach your customers a scripting language. I’d look into perl and test::more – or possibly some of the ruby test stuff.

And if that doesn’t work, perhaps you could …

3) Give up on having the customers create all the tests. Instead, have a toolsmith that pairs with the customers to create the outline, then goes back and converts that into code.

If you were driving a browser, I’d recommend selenium or watIR, but it looks like you’re command-line-y.

Drop me an email ([e-mail]) or read about test frameworks on my blog (xndev.blogspot.com) for more info. My blog is the #2 search result for asking google what’s a test framework, so I’m comfortable recommending it. 🙂

regards,

–heusser