Data-Driven Code Generation of Unit Tests Part 5: Closing Thoughts

In the previous posts in this series, I walked through the idea of performing data-driven code generation for unit tests, as well as how I implemented it in three different programming languages and build systems.  This post contains some final thoughts about the effort.

Was it worth it?
Almost certainly.  Although it required substantial up-front effort to set up the unit test generators, this approach found numerous, previously-undetected bugs both within my implementation of the calculation library as well as with legacy implementations. It is straightforward to write code generators that test all possible combinations of parameters to the calculations, ensuring that the resulting code coverage is excellent. Adding tests for a new calculation is as straightforward as adding a line to a single file.

Which build system was easiest for integrating code generation?

  1. Visual Studio/MSBuild (it is basically out of the box)
  2. Maven
  3. CMake

Which templating language was the best?

  1. Jinja2/T4 (tied)
  2. StringTemplate (a distant 3rd; I would strongly consider evaluating alternative templating languages for generating Java code)

What’s next?
Code generation opens up a vast number of possibilities for future enhancements. The existing code generators could be improved to only generate code when something changes in order to improve compilation times. More unit tests could be defined within the code generator templates to test invalid parameters, NaNs, etc. Binding libraries (e.g. wrapping the Java calculation library in a set of Spark SQL user-defined aggregates, or the C++ library into a set of PostgreSQL user-defined aggregates) can all be code generated from the same metadata.csv (more on this later).

Advertisements

Data-Driven Code Generation of Unit Tests Part 4: C#, MSBuild, T4, MS Unit Test

This blog post explains how I used C#, MSBuild, T4 Text Templates, and the Microsoft Unit Test Framework for Managed Code to perform data-driven code generation of unit tests for a financial performance analytics library. If you haven’t read it already, I recommend starting with Part 1: Background.

As mentioned in Part 2: C++, CMake, Jinja2, Boost, all performance analytics metadata is stored in a single file called metadata.csv. This file drives all code generation and is what helps ensure inter-platform consistency.

I must admit, I was pleasantly surprised when I discovered that Microsoft provides a template-based code generation engine (T4) out-of-the-box with Visual Studio. Because of this, supporting code generation within a Visual Studio project is as easy as creating a file within your project with the extension .tt. The key part to making it work is that the file must be marked as using the TextTemplatingFileGenerator Custom Tool, which Visual Studio does for you automatically.

I decided the easiest thing for me to do was to create a single .tt file that parses metadata.csv and generates a single C# file with all unit tests for all calculations. I also found it rather convenient to include utility functions within the template itself using the stanza.

The template file I created looked something like:

<#@ import namespace="System.Collections.Generic" #>
<#@ import namespace="System.IO" #>
<#@ import namespace="System.Linq" #>
using ...
<#
    string fileName = this.Host.ResolvePath("..\\..\\..\\metadata.csv");
    var lines = File.ReadLines(fileName);
    var header = lines.First().Split(',');
    // Notice how this for loop will run once per calculation in metdata.csv
    foreach (var line in lines.Skip(1)) {
        // Create a dictionary with the calculation's attributes for
        // use by the code generator
        var arr = line.Split(',');
        Dictionary<string, string> dict = new Dictionary<string, string>();
        for (int i = 0; i < header.Length; ++i) {
            dict[header[i]] = arr[i];
        }
#>

namespace PerformanceAnalyticsUnitTest
{
    [ExcludeFromCodeCoverage]
    [TestClass]
    public class <#= UnderscoreToPascalCase(dict["function_name"]) #>Test {
        [TestMethod]
        public void Test<#= UnderscoreToPascalCase(dict["function_name"]) #>ArrayUnannualized() {
            ...
        }

        ...
    }
}

<#
    }
#>

<#+
    public string UnderscoreToPascalCase(string str) {
        ...
    }
#>

I also made sure that the generated files were excluded from source control by adding them to the .gitignore file — as a reminder, generated source is output, not source code, and should not be checked in to source control.

I ran into a few minor annoyances, such as the source code sometimes not being generated at the proper time in the build cycle, but that was about it. Integrating code generation into a Visual Studio project is about as easy as it gets!

Data-Driven Code Generation of Unit Tests Part 3: Java, Maven, StringTemplate, JUnit

This blog post explains how I used Java, Apache Maven, StringTemplate, and JUnit to perform data-driven code generation of unit tests for a financial performance analytics library. If you haven’t read it already, I recommend starting with Part 1: Background.

As mentioned in Part 2: C++, CMake, Jinja2, Boost, all performance analytics metadata is stored in a single file called metadata.csv. This file drives all code generation and is what helps ensure inter-platform consistency.

In order to integrate code generation into the Maven build process, I was forced to create three separate Java projects:

  1. A Maven project which implements the unit test generation using metadata.csv and StringTemplate (java-gentest-srcgen)
  2. A Maven project which builds a Maven plugin which calls the unit test generator at the right point in the Maven build lifecycle (java-gentest-maven-plugin)
  3. A Maven project which implements the calculations and uses the Maven plugin to generate the source code (java-lib)

All three projects are tied together using a single parent POM which looks like this:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  ...
  <packaging>pom</packaging>
  ...

  <modules>
    <module>java-gentest-srcgen</module>
    <module>java-gentest-maven-plugin</module>
    <module>java-lib</module>
  </modules>
</project>

Unit test source code generator

The Java unit test generator project (java-gentest-srcgen) creates a JAR with a single class, JavaUnitTestGenerator, with a single public method generate().  This method parses metadata.csv and calls StringTemplate to generate a unit test class for each found calculation.

The code looks something like:

public class JavaUnitTestGenerator {
    // A simple logging interface defined in this project so we can integrate
    // into any logging system, including Maven's
    private SimpleLogger logger;
    private File targetDirectory;

    public JavaUnitTestGenerator setLogger(SimpleLogger logger) {
        this.logger = logger;
        return this;
    }

    public JavaUnitTestGenerator setTargetDirectory(File targetDirectory) {
        this.targetDirectory = targetDirectory;
        return this;
    }

    public void generate() throws Exception {
        // The Maven build process adds metadata.csv as a resource to
        // the JAR so that the JAR becomes self-contained       
        try (InputStream is = getClass().getResourceAsStream("metadata.csv")) {
            try (InputStreamReader sr = new InputStreamReader(is)) {
                try (BufferedReader br = new BufferedReader(sr)) {
                    String[] headers = br.readLine().split(",");

                    String line;
                    while ((line = br.readLine()) != null) {
                        String[] arr = line.split(",");
                        Map<String, String> kvp = new HashMap<String, String>();

                        for (int i = 0; i < arr.length; ++i) {
                            kvp.put(headers[i], arr[i]);
                        }
                        generateUnitTest(kvp);
                    }
                }
            }
        }
    }

    private void generateUnitTest(Map<String, String> kvp) throws Exception {
        // Instantiate the template
        URL url = Resources.getResource("unit-test.stg");
        STGroup g = new STGroupFile(url, "US-ASCII", '<', '>');
        ST template = g.getInstanceOf("unit_test");
        for (Map.Entry<String, String> entry : kvp.entrySet()) {
            if (template.getAttributes().containsKey(entry.getKey())) {
                // Don't pass in false values so that if(variable) evaluates to false
                if (!entry.getValue().equals("false")) {
                    template.add(entry.getKey(), entry.getValue());
                }
            }
        }
        // Add various transformations and manipulations to the attributes
        // found in metadata.csv that cannot easily be done in StringTemplate's
        // templating language       
        template.add("algorithm_type_pascal_case", toPascalCase(kvp.get("algorithm_type")));
        template.add("function_name_pascal_case", toPascalCase(kvp.get("function_name")));
        template.add("function_name_camel_case", toCamelCase(kvp.get("function_name")));
        ...

        // Generate the source code file
        File tgtFile = new File(targetDirectory, toPascalCase(kvp.get("function_name")) + "UnitTest.java");
        logger.info("Generating " + tgtFile + "...");
        try (FileOutputStream os = new FileOutputStream(tgtFile)) {
            try (OutputStreamWriter osw = new OutputStreamWriter(os)) {
                STWriter stWriter = new AutoIndentWriter(osw);
                template.write(stWriter);
            }
        }
    }
}

The template itself (unit-test.stg) looks something like:

group javagentest;

unit_test(algorithm_type,
          algorithm_type_pascal_case,
          function_name,
          ...) ::= <<

package com.morningstar.perfanalytics.tests;

...

public class <function_name_pascal_case>UnitTest {
    // BEGIN ARRAY TESTS
    @Test
    public void testArrayUnannualized() {
        ....
        double expected = <expected_value_unannualized>;
        double actual = ...;
        assertEquals(expected, actual, 0.00001);
    }

    ....
>>

Unfortunately, because StringTemplate’s templating language is so weak, we’re stuck with the following limitations:

  1. The JavaUnitTestGenerator class must perform a number of presentation-oriented transformations on attributes found in metadata.csv, such as case conversions or creating small Java snippets. This means that the JavaUnitTestGenerator and the template are extremely tightly coupled in non-obvious ways.
  2. The lack of a for-loop construct in the template language means we cannot perform tricks like we did in Part 2, where we generate all possible combinations of annualization, frequency, etc. for a given calculation.

Maven plugin

The Maven plugin project creates a maven-plugin that wraps the unit test source code generator. The source code looks like:

@Mojo(name = "generate", requiresProject = true, threadSafe = false, requiresDependencyResolution = ResolutionScope.COMPILE_PLUS_RUNTIME, defaultPhase = LifecyclePhase.GENERATE_SOURCES)
public class JavaUnitTestCodegenMojo extends AbstractMojo {
    @Parameter(defaultValue = "${project}")
    private MavenProject project;

    @Parameter(property = "outputDirectory", defaultValue = "${project.build.directory}/generated-test-sources/unit-tests")
    private File outputDirectory;

    @Override
    public void execute() throws MojoExecutionException {
        getLog().info("Generating unit tests for Java calculation library");

        // MojoLogger adapts Maven's logging class to the SimpleLogger
        // interface in the java-gentest-srcgen project
        JavaUnitTestGenerator g = new JavaUnitTestGenerator()
            .setLogger(new MojoLogger(getLog()))
            .setTargetDirectory(this.outputDirectory);
        try {
            g.generate();
        } catch (Exception ex) {
            throw new MojoExecutionException("Error generating source code", ex);
        }

        project.addTestCompileSourceRoot(outputDirectory.getPath());
    }
}

Note how the plugin instructs the calling project to add the generated source code directory to the test compile source root. This way, the calling project doesn’t need to remember to configure this in its pom.xml.

Calculation library

The calculation library contains the implementation of all calculations. It references the Maven plugin project in its pom.xml in order to have the Maven plugin create its unit tests as part of the build process. The relevant stanza looks like this:

  ...
  <build>
    <plugins>
      <!-- Generate unit test code -->
      <plugin>
        <groupId>com.morningstar.perfanalytics</groupId>
        <artifactId>perfanalytics-java-gentest-maven-plugin</artifactId>
        <version>...</version>
        <executions>
          <execution>
            <goals>
              <goal>generate</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>

The Java library also has a few other, hand-written unit tests, in the normal directory (src/test/java/...). The build system knows how to compile and run both the hand-written and the machine-generated unit tests at build time.

I found this approach reliable, and mostly straightforward, but with a few drawbacks. First, the unit test generator, as coded, will generate the source code every time, even if nothing has changed. This means we will perform a lot of unnecessary re-compilation, slowing down builds. Second, weaknesses in the templating language make it frustrating to use; I have to frequently switch back-and-forth between the template file and the Java code to achieve my desired result. Third, it’s unfortunate that we have to write a custom Maven plugin just to perform source code generation at build time; it’d be nice if there was a simpler way. Fourth, the versions of all the Maven projects must be kept in sync at all times; fortunately, the Maven release plugin does this for us automatically.

Next time I’ll talk about doing this in C#, MSBuild, T4 Text Templates, and the Microsoft Unit Test Framework for managed code!