Question

I have an integration testing solution. I have my tests described in XML files. In order to capitalize on Visual Studio 2010 testing infrastructure, I have a C# class where every XML test file has an associated method that loads the XML file and executes its content. It looks like this:

[TestClass]
public class SampleTests
{

    [TestMethod]
    public void Test1()
    {
        XamlTestManager.ConductTest();
    }

    [TestMethod]
    public void Test2()
    {
        XamlTestManager.ConductTest();
    }

    ...

    [TestMethod]
    public void TestN()
    {
        XamlTestManager.ConductTest();
    }
}

Each method name corresponds to an XML file name. Hence, I have to have the following files in my test directory:

• Test1.xml
• Test2.xml
• …
• TestN.xml

XamlTestManager.ConductTest() uses the StackTrace class to get the name of the calling method and this way it can find the correct XML test file to load.

I would like to get rid of the extra administration of adding/removing/renaming test methods whenever I change my tests, adding/removing/renaming an XML test file. How can I automagically generate this class or its methods during the compilation process based on the actual XML files in my test directory?

Option 1:
I have considered PostSharp, but it does not allow me to look up the XML files and generate methods on the fly (or was I superficial?).
Option 2:
The other idea was to build a Visual Studio custom tool that generates my code whenever it is executed. The downside here is the deployment. The custom tool needs to be registered to VS. I want a solution that can be committed into a repository, check it out to another computer and use it right away.
(I believe in simplicity. “Check out and run” just simplifies the life of new developers soooooo much, if they do not need to go through a list of thing to install before they can compile run the application.)

Do you have any recommendation, how to get rid of the unnecessary maintenance issue?

EDIT:
For the request of Justin, I add more details. We use Bizunit (fantastic!!!) as the basis of our framework with a truckload of custom made high level test steps. From these steps we can build our test like from lego blocks in a declarative manner. Our steps include like FileDrop, WebService invokation or even polling, firing up a full blown web server to simulate a partner web application, random data generator, data comparing steps etc. Here is an example test xml (in fact XAML):

<TestCase BizUnitVersion="4.0.154.0" Name="StackOverflowSample" xmlns="clr-namespace:BizUnit.Xaml;assembly=BizUnit" xmlns:nib="clr-namespace:MyCompany.IntegrationTest;assembly=BizUnit.MyCustomSteps" xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml">
  <TestCase.SetupSteps>
    <nib:ClearStep FailOnError="True" RunConcurrently="False" />
    <nib:LaunchSimulatedApp AppKernelCacheKey="provider" FailOnError="True" FireWakeUpCall="False" PortNumber="4000" RepresentedSystem="MyProviderService" RunConcurrently="False" />
    <nib:HttpGetStep FailOnError="True" RunConcurrently="False" Url="http://localhost:10000/Home/StartSvgPolling">
      <nib:HttpGetStep.Parameters>
        <x:String x:Key="PolledAddress">http://localhost:4000/SvgOutputPort.asmx</x:String>
        <x:String x:Key="PollingInterval">10</x:String>
        <x:String x:Key="FilterFile"></x:String>
      </nib:HttpGetStep.Parameters>
    </nib:HttpGetStep>
  </TestCase.SetupSteps>

  <TestCase.ExecutionSteps>
    <nib:DocumentMergeStep FailOnError="True" OutputCacheKey="inputDocument" RunConcurrently="False">
      <nib:DocumentMergeStep.InputDocuments>
        <nib:RandomLoader BoundingBox="Europe" LinkbackUrlPattern="http://MyProviderService/id={0}" MaxAmount="10" MaxID="100" MinAmount="10" MinID="0" NamePattern="EuropeanObject_{0}" NativeFormat="Svg" RepeatableRandomness="False" UriPrefix="European" />
        <nib:RandomLoader BoundingBox="PacificIslands" LinkbackUrlPattern="http://MyProviderService/id={0}" MaxAmount="10" MaxID="100" MinAmount="10" MinID="0" NamePattern="PacificObject_{0}" NativeFormat="Svg" RepeatableRandomness="False" UriPrefix="Pacific" />
      </nib:DocumentMergeStep.InputDocuments>
    </nib:DocumentMergeStep>
    <nib:PushToSimulatedApp AppKernelCacheKey="provider" ContentFormat="Svg" FailOnError="True" RunConcurrently="False">
      <nib:PushToSimulatedApp.InputDocument>
        <nib:CacheLoader SourceCacheKey="inputDocument" />
      </nib:PushToSimulatedApp.InputDocument>
    </nib:PushToSimulatedApp>
    <nib:GeoFilterStep FailOnError="True" OutputCacheKey="filteredDocument" RunConcurrently="False" SelectionBox="Europe">
      <nib:GeoFilterStep.InputDocument>
        <nib:CacheLoader SourceCacheKey="inputDocument" />
      </nib:GeoFilterStep.InputDocument>
    </nib:GeoFilterStep>
    <nib:DeepCompareStep DepthOfComparision="ID, Geo_2MeterAccuracy, PropertyBag, LinkbackUrl" FailOnError="True" RunConcurrently="False" Timeout="30000" TolerateAdditionalItems="False">
      <nib:DeepCompareStep.ReferenceSource>
        <nib:CacheLoader SourceCacheKey="filteredDocument" />
      </nib:DeepCompareStep.ReferenceSource>
      <nib:DeepCompareStep.InvestigatedSource>
        <nib:SvgWebServiceLoader GeoFilter="Europe" NvgServiceUrl="http://localhost:10000/SvgOutputPort.asmx"/>
      </nib:DeepCompareStep.InvestigatedSource>
    </nib:DeepCompareStep>
  </TestCase.ExecutionSteps>

  <TestCase.CleanupSteps>
    <nib:HttpGetStep FailOnError="True" RunConcurrently="False" Url="http://localhost:10000/Home/StopSvgPolling">
      <nib:HttpGetStep.Parameters>
        <x:String x:Key="PolledAddress">http://localhost:4000/SvgOutputPort.asmx</x:String>
      </nib:HttpGetStep.Parameters>
    </nib:HttpGetStep>
    <nib:KillSimulatedApp AppKernelCacheKey="provider" FailOnError="True" PortNumber="4000" RunConcurrently="False" />
  </TestCase.CleanupSteps>
</TestCase>

This is what it does:

1. Invokes a Clear operation on the test subject
2. Launches a webserver on port 4000 as a simulated partner app under the name MyProviderService
3. Invokes the test subject via HTTP Get to poll the simulated partner
4. Creates a new document containing geo info from two random generated content
5. Pushes the document to the simulated partner – hence the test subject will pick it up via polling
6. The test applies a geo filter on the document
7. The deep compare step loads the filtered document as base of comparision, and loads the content of the test subject via a web service
8. As clean-up, it stops the polling via an HTTP GET step and kills the simulated partner’s web server.

The power of Bizunit is that merges the ease of creating tests in C# with intellisense and ease of maintaining/duplicating it in XAML files. For a quick easy read on how it works: http://kevinsmi.wordpress.com/2011/03/22/bizunit-4-0-overview/

Answer 1

As @GeorgeDuckett said, T4 templates are probably the way to go. In the application I am working on, we use them for a lot, including generating Repositories, Services, ViewModels, Enums and recently unit tests.

They are basically code generating scripts written in either VB or C#, looking at a directory for XML files would be no problem for these kinds of templates.

If you do choose to go the T4 route, the Tangible T4 Editor is definitely a must have, it is a free download.

Here is a quick example of a T4 script which should do or be pretty close to what you want:

<#@ template language="C#" debug="true" hostspecific="true"#>
<#@ output extension="g.cs"#>
[TestClass]
public class SampleTests
{
<#
string[] files = Directory.GetFiles(@"C:\TestFiles", "*.xml");
foreach(string filePath in files)
{
    string fileName = Path.GetFileNameWithoutExtension(filePath);
#>
    [TestMethod]
    public void <#=fileName#>()
    {
        XamlTestManager.ConductTest();
    }
<#
}
#>
}

Make sure this is placed in a file with the .tt extension, then on the property windows for this file, ensure the Build Action is None, Custom Tool is TextTemplatingFileGenerator.

Edit: Accessing output directory from T4 template

Add the following two lines to the top of your T4 template, under the <#@ template … #> line:

<#@ assembly name="EnvDTE" #>
<#@ import namespace="EnvDTE" #>

Then inside your template, you can access and use the visual studio API like so:

IServiceProvider serviceProvider = this.Host as IServiceProvider;
DTE dte = serviceProvider.GetService(typeof(DTE)) as DTE;
object[] activeSolutionProjects = dte.ActiveSolutionProjects as object[];

if(activeSolutionProjects != null)
{
    Project project = activeSolutionProjects[0] as Project;
    if(project != null)
    {
        Properties projectProperties = project.Properties;
        Properties configurationProperties = project.ConfigurationManager.ActiveConfiguration.Properties;
        string projectDirectory = Path.GetDirectoryName(project.FullName);  
        string outputPath = configurationProperties.Item("OutputPath").Value.ToString();
        string outputFile = projectProperties.Item("OutputFileName").Value.ToString();

        string outDir = Path.Combine(projectDirectory, outputPath);
        string targetPath = Path.Combine(outDir, outputFile);
    }
}

outDir and targetPath contain the output directory and the full path to the output file.