Data-Driven Code Generation of Unit Tests Part 1: Background

At Morningstar, I created a multi-language, cross-platform performance analytics library which implements both online and offline implementations of a number of common financial analytics such as Alpha, Beta, R-Squared, Sharpe Ratio, Sortino Ratio, and Treynor Ratio (more on this library later).  The library relies almost exclusively on a comprehensive suite of automated unit tests to validate its correctness.  I quickly found that maintaining a nearly-identical battery of unit tests in three different programming languages was a chore, and I had a hunch that I could use a common technique to deal with this problem: code generation.

The basic ideas behind the approach are quite straightforward. The first idea is one of language independence — a given calculation, given a known set of inputs, must produce the same output (allowing for rounding error), regardless of programming language. Therefore, a unit test for the implementation of Alpha in C# should be nearly identical in function (and remarkably similar in form) to a unit test for the implementation of Alpha in Java. Perhaps this means that we don’t need to write the unit test twice; we can have the computer perform the translation for us.

The second idea is one of calculation similarity.  Financial performance analytics tend to follow a common pattern: they all take in one to three streams of returns (security, benchmark, risk-free rate); they are almost all aggregate functions; most (but not all) can be implemented in both online and offline forms; and many support annualization.  The code for the unit test for Beta looks remarkably like the code for the unit test for Alpha; the only significant difference is the expected result. Therefore, if we can encode only the differences among the calculations (e.g. their expected results) in some sort of data file, perhaps we can use code generation for the vast majority of the unit tests for the calculation library.

My hunch paid off. In the end, I had a single CSV file which contained all the important differences among the calculations (e.g. their expected values). The build process uses this CSV file to code generate the entire unit test framework in C++ (using CMake, Jinja2, and the Boost Unit Test Framework), Java (using Apache Maven, StringTemplate, and JUnit), and C# (using MSBuild, T4 Text Templates, and the Microsoft Unit Test Framework for Managed Code). I was guaranteed that every single calculation in every language produces the same result given the same input. I found language-specific bugs (typically typos) in the performance analytics library. I found language-specific bugs in previously-existing libraries at Morningstar (fortunately these were niche languages that weren’t actively used in products). I learned a lot about differences in templating systems for code generation (Jinja2 and T4 were pleasant; StringTemplate was much less so) and using code generation in build systems (Maven is a real pain; SBT is probably a lot nicer).  Furthermore, I was able to use the same metadata file and code generation tools to power binding and wrapper libraries around the code performance analytics library (more on this later).

Future posts in this series will explain how I implemented data-driven code generation of unit tests in each of the above programming languages.

I’d love to hear feedback from you if you found this useful, or other places where you’ve applied similar techniques!

Advertisements

Future (circa 2010) Parallel Programming Models

Joe Duffy regularly posts amazing material which is well ahead of our time, such as his current blog post series about Midori.

I’d like to call out this particular assertion made by him way back in 2010:

[D]evelopers must move towards single-threaded programming models connected through message passing, optionally with provably race-free fine-grained parallelism inside of those single-threaded worlds.

Add “async/await everywhere” and you can sign me up!

Threading is a Poor Concurrency Programming Model

Here’s why I try to avoid thread-based programming models for expressing concurrency:

The opponents of thread-based systems line up several drawbacks. For Ousterhout, who probably published the most well-known rant against threads [Ous96], the extreme difficulty of developing correct concurrent code–even for programming experts–is the most harmful trait of threads. As soon as a multi-threaded system shares a single state between multiple threads, coordination and synchronization becomes an imperative. Coordination and synchronization requires locking primitives, which in turn brings along additional issues. Erroneous locking introduces deadlocks or livelocks, and threatens the liveness of the application. Choosing the right locking granularity is also source of trouble. Too coarse locks slow down concurrent code and lead to degraded sequential execution. By contrast, too fine locks increase the danger of deadlocks/livelocks and increase locking overhead. Concurrent components based on threads and locks are not composable. Given two different components that are thread-safe, a composition of them is not thread-safe per se. For instance, placing circular dependencies between multi-threaded components unknowingly can introduce severe deadlocks.

Lee [Lee06] focuses on the lack of understandability and predictability of multi-threaded code, due to nondeterminism and preemptive scheduling. Multithreading appears to be error-prone, and very difficult to debug. The state explosion as a result from all possible interleavings of multiple threads renders a reasonable execution analysis of concurrent code virtually impossible. This is primarily caused by the unpredictability of preemptive scheduling. So, contrary to von Behren [vB03a], Lee argues that threads are precisely not a good abstraction for concurrent flows of execution. Quite the opposite, the oversimplifying abstraction of threads appears to be misleading, as it pretends a continuous execution of code that may not match any real runtime behavior.

Read the complete web page — it’s quite good.

Exploring the .NET CoreFX Part 16: Platform-Specific Builds Using Compile-Time Polymorphism

This is part 16 of my Exploring the .NET CoreFX Series.

While .NET has historically been limited to Windows machines, Mono notwithstanding, the introduction of the cross-platform .NET Core runtime has introduced the possibility of running .NET Core applications on Unix machines. With this possibility, developers may have the need of writing platform-specific code.

One way to write platform-specific code is:

  1. Define a conceptual base class which will have an identical name and methods across all platforms. This does not need to be a C# interface, as we will be using compile-time rather than run-time polymorphism.
  2. Provide an implementation of this class for each target platform.
  3. Use build-time conditions to include the platform-specific class based on target compilation platform.

An example from the .NET Core is the System.Console.ConsolePal class from the System.Console library. The library includes two implementations of this class:

  1. ConsolePal.Unix.cs, which provides a Unix-compatible implementation of the ConsolePal class
  2. ConsolePal.Windows.cs, which provides a Windows-compatible implementation of the ConsolePal class

The project file for this library, System.Console.csproj, then selects the appropriate file to build based on the target platform at compile time:

<!-- System.Console.csproj -->
...
<ItemGroup Condition=" '$(OS)' == 'Windows_NT' ">
  <Compile Include="System\ConsolePal.Windows.cs" />
  ...
</ItemGroup>
<ItemGroup Condition=" '$(OS)' == 'Unix' ">
  <Compile Include="System\ConsolePal.Unix.cs" />
  ...
</ItemGroup>

The advantage of this approach is that it has no run-time polymorphism overhead and thus provides maximum performance. The disadvantage of this approach is that the resulting binaries are platform-specific.

Recommendations

  • Avoid writing platform-specific .NET code unless it is completely unavoidable.
  • Consider using compile-time polymorphism to implement platform-specific code.

Exploring the .NET CoreFX Part 15: Using Non-Generic Factory Classes to Enable Type Inference

This is part 15 of my Exploring the .NET CoreFX Series.

While C# supports type inference for generic methods, it does not support type inference for constructors. In other words, while this code works:

public class FooFactory
{
   public static Foo<T> Create<T>(T value)
   {
      return new Foo<T>(value);
   }
}
var myObj = FooFactory.Create(212);

This code does not:

public class Foo<T>
{
   private readonly T field;
   public Foo(T value) { field = value; }
}

var obj = new Foo(212); // DOES NOT WORK

For more background on why this is, see this StackOverflow post.

Because of this, every generic type in System.Collections.Immutable includes a factory class of the same name which makes construction more convenient. For example:

// Using the generic class which requires explicitly-specified types
var l = ImmutableList<int>.Empty.Add(1);
// Using the factory class to use inferred types
var l = ImmutableList.Create(1);

The same trick is also used with System.Tuple.

Recommendations

  • If you author a generic class, consider also providing a factory class with generic methods to enable type inference.

Exploring the .NET CoreFX Part 14: Inside Immutable Collections

This is part 14 of my Exploring the .NET CoreFX Series.

Back in 2013, Immo Landwerth and Andrew Arnott recorded a Going Deep video called Inside Immutable Collections which describes how and why System.Collections.Immutable is built the way it is. It’s great background material to understand System.Collections.Immutable.