Question

my first question on stackoverflow and I’m hoping it’s not a stupid one. 🙂

Basically all my datadriven tests look like this one:

[TestMethod]
[DataSource(TestsDataSource)]
public void Test_With_Some_Fancy_Name()
{
   if (myIsThisTestIgnored) return;
...

DataSource attribute + app.config connectionstring + excel odbc driver = excel datadriven tests.

myIsThisTestIgnored is set in the [TestInitialize] method (for every datarow there is a separate test with separate [TestInitialize]), via TestContext.DataRow["Ignore"], so I can “ignore” tests via some true/false in an excel sheet.

Now my problem is: just returning from a test method lets the test pass. How to not let the ignored tests “pass” in the MSTEST testrunner (and in the CI via msbuild), or even better, do not show them at all?

On our Ci Build, it sums up every ignored (a.k.a passed) test, as the count of test is datarows x number of testmethods, giving a faulty impression of the number of tests that ran. But, until now, I found no way to just set the test result to “ignored” (or any other state than “passes”/”failed”) programmatically.

Besides, getting rid of this annoying ignore line in every test and enhancing the assert messages without wrapping every assert in some function would be nice, too.

So the best/only approach to this, I thought, would be AOP style programming and using custom attributes. I found this old post about enhancing your mstest code with custom attributes, good read: http://callumhibbert.blogspot.com/2008/01/extending-mstest.html

Specifically the last comment in the linked article got me thinking, as the guy not just made my day by showing how to access all test state data to the attribute (the MarshalObject stuff), but somehow he got it working, quoting him “In this manner, I can determine whether or not to run a test based on a matrix of data driven information from the text context (using the DataSource attribute), and the new custom attributes I created. “. Yeah.. but how to do that?

I took some of the code and implemeted it, looking something like that:

public class MyCustomAspect : TestAspect<MyCustomAttribute>, IMessageSink, ITestAspect
{ ....

    [SecurityPermission(SecurityAction.LinkDemand, Flags = SecurityPermissionFlag.Infrastructure)]
    public IMessage SyncProcessMessage(IMessage msg)
    { 
      ....
      // see last comment in linked article to how to get this MarshalObject
      MyTestsClass testClass = (MyTestsClass)MarshalObject;

      if(testClass.myTestInitializeRan && testClass.myIsThisTestIgnored)
      {
        // what to do here? I would like to get rid of the SyncProcessMessage invocation, but...
        // return msg or null -> Exception "The method was called with a Message of an unexpected type."
        // constructing a IMessage in an easy way (specifically CallMessage) seems not possible, and I have no clue how to construct a valid CallMessage that says "ignore test!"
      }
      else
      {
        IMessage message = _nextSink.SyncProcessMessage(msg);
        return message;
      }

Again: I want to suppress the testrun for this datarow, or set the testresult somehow to “ignore”/”unknown”.

Doing anything with the returned message or msg seems pointless, as the IMessage Interface just exposes the Properties Dictionary which has only some MethodName and stuff in it, nothing interesting it seems.

Casting IMessage to concrete implementations doesn’t work: they are all internal to System.Runtime. Though one can cast to ResultMessage and it returns something != null when an exception was raised (so you can take the exception message and adjust error message via new throw).

Any Ideas? And many thanks for reading so far. 😉

Answer 1

How to not let the ignored tests “pass” in the MSTEST testrunner (and in the CI via msbuild), or even better, do not show them at all?

You could use Assert.Fail or Assert.Inconclusive instead of simply returning from the test case if you do not want skipped tests to be accounted for as passed.

You may also find Data driven tests with test attributes in MSTest useful. The author showcases how to use postsharp to create parameterised Row tests in order to skip the need of using excel for data driven tests and instead post the parameters as an attribute above the test method. Therefore when you would like to remove or ignore a test case you would simply comment out that data row and it would not be shown at all.