PlayNicely.Executor 1.0.1-prerelease-20240225-085709

Play Nicely Executor

The Play Nicely Executor project supports the execution of MSBuild test case projects within a pre-defined test case environment. There you can define required commands, or exclude commands from the path (so you can force things to fail). This package is published because having a solution that supports the creation of a reliable and clean environment, to run tool testing, seems like a use case for many scenarios.

The main artefacts are the ITestEnvironment, which specifies the target project and ensures a clean environment in which any testing can run. You create an ITestEnvironment using the fluent interface of TestEnvironmentBuilder. With this class you define the desired context for your test environment. After building the ITestEnvironment you can pass it to a concrete implementation of the ITestEnvironmentRunner which executes the test and returns the outcome.

Tests are run via a ITestEnvironmentRunner, current concrete implementations are a basic ProcessRunner which excepts an executable and arguments, much like a command line, and returns a basic IExecutionResult. And the DotNetRunner which executes a dotnet subcommand, with arguments, and returns a IExecutionResult<DotNetExecutionContext> which includes the dotnet command results context, targets ran, projects built, lists of errors, etc.

You can also define your own runners derived from ITestEnvironmentRunner.

Getting Started

This getting started uses the generic ProcessRunner to execute a dotnet build process on a pre-defined test case project, within a clean test environment. The purpose of this getting started is to demonstrate how to set up a test environment.

ℹ️ I'm using ProcessRunner here to illustrate getting started. The PlayNicely.Executor.DotNet package provides a specific implementation for dotnet that includes important context information after execution. If you are running dotnet tests, we recommend using that runner instead.

This getting started is code-first, in a typical configuration you would likely use the IDE to define test case projects and SpecFlow (or some other testing framework) to define the environment.

Define the test case project

Let's build a scenario to test for a failing build, because the .NET target framework is invalid. First, define the test case project and file system.

var testCaseProject = new TestCaseProject("my-failing-project");
var projectFile = testCaseProject.AddFile("proj.csproj");

testCaseProject.ProjectFile = projectFile;

using(var writer = new StreamWriter(projectFile.OpenWriteStream())
{
    writer.WriteLine("<Project Sdk=\"Microsoft.NET.Sdk\">");
    writer.WriteLine("  <PropertyGroup>");
    // Note invalid target framework...
    writer.WriteLine("    <TargetFramework>my-net9.0</TargetFramework>");
    writer.WriteLine("  </PropertyGroup>");
    writer.WriteLine("</Project>");
}

Define the environment

Use the TestEnvironmentBuilder to specify:

• RequiredCommands - any command that must be available in the test environment. Using this method ensures commands can always be executed, by finding them in the current $PATH, and creating a temporary bin directory with symbolic links. That bin directory is prepended to the $PATH in the subsequent environment. On Windows this is less likely to be required, but on Linux, most commands are in a shared directory like /usr/bin or /usr/local/bin, if something is excluded (see next bullet point) it is likely to exclude a lot of other commands, RequiredCommands ensures the essential commands can still be executed.
• ExcludeCommandFromPath - it is often neccesary to negative negative test cases where a command or tool does not exist. This method locates a command on the current $PATH and removes any directories where it is found. The result of this, when running in the ITestEnvironment is that any attempt to run an excluded command will fail (because it isn't found).
• AddPackageSource - often, the usage of this package is to test another NuGet package project. So that the test environment can run release tests using the under development version of a package, this method allows overriding NuGet.Config in the test environment. When attempting to locate packages, dotnet will only look in these locations.
• SetProject - the target project for this test environment, the project defines the file system of the test environment.

Example for clarity

Continuing context from here. We need to ensure dotnet is always available, and specify the test case target project.

var builder = new TestEnvironmentBuilder();
var testEnv = await builder.RequiredCommands("dotnet")
                           .SetProject(testCaseProject)
                           .BuildAsync();

Run the test

With the environment built, simply construct the runner and assert the result (which we expect to fail).

var runner = new ProcessRunner("dotnet", "build", testEnv.ProjectFilename);
var testResult = await runner.ExecuteAsync(testEnv);

Assert.That(testResult.Succeeded, Is.False);

Why?

This project came about to support the use of NodeJS packages within .NET projects in a .NET first way. You can achieve the integration of NodeJS tools using plugins or other tooling. The problem with this (using plugins) is, these can often become out of date or stale. Most of the NodeJS packages are developed by an actively community, so accessing the latest npm packages directly makes the most sense (we get latest features and security updates).

Below here is not done.

If you are going to use this package, you probably want to define test cases for a NodeJS tool package. Before you start, it's a good idea to consider the best layout for your project so that you can use a familar development experience for test cases, i.e. Visual Studio, while also making the maintenance of the test case packages as simple as possible.

Our recommended project structure is (if you're using SpecFlow for testing):

solution-dir
|-- my-project	               # This is the project you are developing
|   |-- *
|-- my-project.Specs           # BDD Specifications for your project
|   |-- Features
|   |   |-- *.features
|   |-- TestCases.Projects     # Define your test case 'packages' in resx files
|       |-- Project1.resx
|       |-- Project2.resx
|       |-- TestCases.cs       # Helper class to load local test case projects from embedded resources
|-- TestCase.Projects          # Define your test case projects in here using familiar tools.
|   |-- Project1
|   |   |-- Project1.csproj
|   |-- Project2
|       |-- Project2.csproj
|-- my-solution.sln

We'll refer to this structure in the rest of this document.

Create your test case projects

Use visual studio to create you test case projects, as you would any other C# project. Add them to the solution-dir/TestCase.Projects directory. Add code files, define properties, etc. Typically, you do not want to build your test cases as part of the normal Visual Studio build command. To disable build use the solution Configuration Manager... to disable build on your test cases.

Define test case packages as embedded (resx) resources

For consistency, when running test cases, a clean temporary directory is created for each test run. This means build artefacts like obj or cached NuGet packages are not available, and ensures consistent test behaviour.

So a clean version of the project can be created, they need to be explicitly defined in embedded resource (resx) files.

1. In your BDD project, my-project.Specs, under the TestCase.Projects directory, add a new resx file. It is not required, but is better for consistency, to name your resx the same as your actual project. In the example, above, Project1.resx matches Project1.csproj
2. Disable code generation in your resx file.
3. Add the relevant files from your test case project to your resx.
   1. In the resource editor, Add existing file...
   2. The default naming for added files, mangles the names and removes the extension. As we are trying to define files in a file system like format, you need to re-add the extension and also any directories, using / as a separator.
4. If you have multiple projects in your test case, like you would in a solution, you can specify the target build project by adding a resource string to the resx called ProjectFile with the value being the path (within the context of the resx) to the project file. In the example above, for Project1.resx, the ProjectFile value would be Project1.csproj.

Add a test case helper

This might be simpler than you think, once we bring over the TestEnvironment stuff.

Use steps in your BDD project

This should probably be centralized also.