Scand.StormPetrel.Generator 2.0.0

Scand Storm Petrel Generator

• Overview
• Primary Use Cases
  • Enabler: Auto-Generate CalculatorTestStormPetrel.AddTestStormPetrel
  • Overwrite CalculatorTest.AddTest expected baseline
  • Overwrite CalculatorTestTheory.AddTestGetExpected expected baseline method
  • Use Case Variations
    • Expected variable expression is static property assignment
    • Expected expression is test method argument what comes from
      • Test case attribute
      • Test data source attribute
    • Snapshots of Snapshot Testing
• Getting Started
• Configuration
• Supported Software
  • Test Frameworks
  • .NET Versions
• CHANGELOG
• FAQs
• References

Overview

.NET Incremental Generator that creates modified copies of unit and/or integration tests to update expected baselines in original tests, automating baseline creation and accelerating test development.

Primary Use Cases

Enabler: Auto-Generate CalculatorTestStormPetrel.AddTestStormPetrel

Given

Calculator class with a bug introduced by the buggyDelta variable:

public class Calculator
{
    public static AddResult Add(int a, int b)
    {
        var buggyDelta = 1;
        var result = a + b + buggyDelta;
        return new AddResult
        {
            Value = result,
            ValueAsHexString = "0x" + result.ToString("x"),
        };
    }
}

public class AddResult
{
    public int Value { get; set; }
    public string ValueAsHexString { get; set; } = string.Empty;
}

And its corresponding test with an expected baseline that matches the Calculator.Add buggy behavior:

public class CalculatorTest
{
    [Xunit.Fact]
    public void AddTest()
    {
        //Arrange
        // incorrect `expected` baseline value will be overwritten with correct `actual` value
        // after manual execution of auto-generated AddTestStormPetrel test.
        var expected = new AddResult
        {
            Value = 5, //incorrect value example
            ValueAsHexString = "0x5"
        };

        //Act
        var actual = Calculator.Add(2, 2);

        //Assert
        actual.Should().BeEquivalentTo(expected);
    }
}

When

The developer configures the test project with StormPetrel.Generator as per the Getting Started.

Then

A new test method, CalculatorTestStormPetrel.AddTestStormPetrel, is generated. This method is a specially modified copy of the original CalculatorTest.AddTest to overwrite its expected baseline.

Overwrite CalculatorTest.AddTest expected baseline

Given

CalculatorTestStormPetrel.AddTestStormPetrel is auto-generated after enabling Storm Petrel functionality.

When

The developer fixes buggyDelta to 0 and executes CalculatorTestStormPetrel.AddTestStormPetrel test.

Then

CalculatorTest.AddTest code is populated with correct expected baseline value, i.e. its code becomes

public class CalculatorTest
{
    [Xunit.Fact]
    public void AddTest()
    {
        //Arrange
        // incorrect `expected` baseline value will be overwritten with correct `actual` value
        // after manual execution of auto-generated AddTestStormPetrel test.
        var expected = new AddResult
        {
            Value = 4,
            ValueAsHexString = "0x4"
        };

        //Act
        var actual = Calculator.Add(2, 2);

        //Assert
        actual.Should().BeEquivalentTo(expected);
    }
}

So that

The developer should only review expected baseline changes (no manual modification) what typically saves development time.

Overwrite CalculatorTestTheory.AddTestGetExpected expected baseline method

Given

CalculatorTestTheory.AddTest test below with

• multiple use cases;
• AddTestGetExpected static method call returning incorrect expected baselines based on arguments;
• AddTestGetExpected method possible variations per its comments.

public class CalculatorTestTheory
{
    [Xunit.Theory]
    [Xunit.InlineData(1, 5)]
    [Xunit.InlineData(2, 2)]
    [Xunit.InlineData(2, 3)]
    public void AddTest(int a, int b)
    {
        //Arrange
        var expected = AddTestGetExpected(a, b);

        //Act
        var actual = Calculator.Add(a, b);

        //Assert
        actual.Should().BeEquivalentTo(expected);
    }

    /// <summary>
    /// Possible variations of AddTestGetExpected static method are:
    /// - Method body may have pattern matches within pattern matches.
    /// - Method body may have `switch` and/or `if` expressions with return statements returning expected baselines.
    /// - The method may be placed in another class and/or file.
    /// </summary>
    private static AddResult AddTestGetExpected(int a, int b) => (a, b) switch
    {
        (1, 5) => new AddResult(), // should be overwritten with correct expected baseline after
                                   // CalculatorTestTheoryStormPetrel.AddTestStormPetrel execution
        (2, 2) => new AddResult(),
        (2, 3) => new AddResult(),
        _ => throw new InvalidOperationException(),
    };
}

When

The developer executes CalculatorTestTheoryStormPetrel.AddTestStormPetrel test.

Then

CalculatorTestTheory.AddTestGetExpected code is populated with correct expected baseline values, i.e. its code becomes

private static AddResult AddTestGetExpected(int a, int b) => (a, b) switch
{
    (1, 5) => new AddResult
    {
        Value = 6,
        ValueAsHexString = "0x6"
    }, //should be overwritten with correct expected baseline
    (2, 2) => new AddResult
    {
        Value = 4,
        ValueAsHexString = "0x4"
    },
    (2, 3) => new AddResult
    {
        Value = 5,
        ValueAsHexString = "0x5"
    },
    _ => throw new InvalidOperationException(),
};

So that

The developer should only review expected baseline changes (no manual modification) what typically saves development time.

Use Case Variations

Expected variable expression is static property assignment

See PropertyTest for more details.

Expected expression is test method argument what comes from

Test case attribute

Supported attributes are xUnit InlineData, NUnit TestCase, MSTest DataRow. See AttributesTest for more details.

Test data source attribute

Supported attributes are xUnit MemberData or ClassData, NUnit TestCaseSource, MSTest DynamicData. See TestCaseSourceMemberDataTest, TestCaseSourceClassDataTest, NUnit TestCaseSourceTest, MSTest TestCaseSourceTest for more details. Known limitations:

• The attributes should be configured against data source methods or properties, not fields.

Snapshots of Snapshot Testing

HTML, JSON, binary or whatever expected snapshots can be hardcoded in tests code. See couple examples in SnapshotTest.

Getting Started

To utilize the StormPetrel tests, add the following NuGet Package references to your test project:

• Scand.StormPetrel.Generator.
• Object to C# code dumper. Represents actual test instance as C# code. See DumperExpression in Configuration for more details.
  • Option A. VarDump. Must be referenced in the case of StormPetrel.Generator default configuration (no appsettings.StormPetrel.json file in the test project). May be additionally configured.
  • Option B. ObjectDumper.NET. May be referenced and configured.
  • Option C. Custom implementation of IGeneratorDumper interface. May be developed and configured.

Configuration

The StormPetrel Generator introduces several interfaces and classes to the Scand.StormPetrel.Generator.TargetProject namespace of the test project. These can be utilized alongside an optional JSON file to customize the rewriting of expected baselines. Key interfaces and classes include:

• IGenerator, Generator;
• IGeneratorBackuper, GeneratorBackuper;
• IGeneratorDumper, GeneratorDumper;
• IGeneratorRewriter, GeneratorRewriter.

Optionally appsettings.StormPetrel.json file (its Build Action should be C# analyzer additional file) can be added to a test project to configure Storm Petrel functionality. The file changes are applied on the fly and can have the following settings:

{
  "TargetProjectGeneratorExpression": "...", // [optional] string, configures the default `Generator`. An expression for the `IGenerator` instance.
  "GeneratorConfig":             // [optional] object to configure `Generator` behavior.
  {
    "BackuperExpression": "...", // [optional] string, instantiates `GeneratorBackuper` by default. An expression for the `IGeneratorBackuper` instance. Set to 'null' to skip creating backup files.
    "DumperExpression": "...",   // [optional] string, instantiates `GeneratorDumper` by default. An expression for the `IGeneratorDumper` instance. `GeneratorDumper` references [VarDump](https://www.nuget.org/packages/VarDump) stuff. Use
                                 // - "new Scand.StormPetrel.Generator.TargetProject.GeneratorDumper(CustomCSharpDumperProvider.GetCSharpDumper())" to have `VarDump` with custom options. Need to implement `CustomCSharpDumperProvider.GetCSharpDumper()` method in this case.
                                 // - "new Scand.StormPetrel.Generator.TargetProject.GeneratorObjectDumper()" expression for `GeneratorObjectDumper` instance which references [ObjectDumper.NET](https://github.com/thomasgalliker/ObjectDumper) stuff.
                                 // - "new Scand.StormPetrel.Generator.TargetProject.GeneratorObjectDumper(CustomOptionsProvider.GetDumpOptions())" to have `ObjectDumper.NET` with custom options. Need to implement `CustomOptionsProvider.GetDumpOptions()` method in this case.
                                 // - "new CustomClassImplementingIGeneratorDumper()" or similar expression to have totally custom implementation of dumping of an instance to C# code.
    "RewriterExpression": "..."  // [optional] string, instantiates `GeneratorRewriter` by default. An expression for the `IGeneratorRewriter` instance.
  },
  "IsDisabled": false,           // [optional] boolean, false is by default. Indicates whether the generator should create 'StormPetrel' classes.
                                 // Even if set to 'false', the generator still adds classes like 'IGeneratorDumper', 'GeneratorDumper' to avoid test project compilation failures
                                 // in the case when custom classes uses them.
  "IgnoreFilePathRegex": "...",  // [optional] string, empty by default. Regular Expression to exclude certain paths from 'StormPetrel' class generation.
  "IgnoreInvocationExpressionRegex": "...",  // [optional] string, empty by default. Regular Expression to detect invocation expressions to not execute StormPetrel rewriting for.
                                 // The property can be utilized in the case of custom IGeneratorRewriter implementations (e.g. when expected baseline is not stored in C# code but binary file as in File Snapshot Testing approach).
  "Serilog": "...",              // [optional] Logging configuration using Serilog (https://github.com/serilog/serilog-settings-configuration?tab=readme-ov-file#serilogsettingsconfiguration--).
                                 // Defaults to logging warnings to the test project's Logs folder. Set to 'null' to disable logging.
                                 // Use the '{StormPetrelRootPath}' token to indicate the target test project root path.
  "TestVariablePairConfigs": [   // [optional] array of objects. Configures naming pairs for actual/expected variables to generate correct expected baselines.
    {
      "ActualVarNameTokenRegex": "[Aa]{1}ctual",     // Default configuration object. Assumes variable pair names like (expected, actual), (myExpected, myActual), (expectedOne, actualOne), (ExpectedTwo, ActualTwo), etc.
      "ExpectedVarNameTokenRegex": "[Ee]{1}xpected", // Corresponds to the `ActualVarNameTokenRegex` for pairing.
    }
  ]
}

Supported Software

Test Frameworks

• xUnit
• NUnit
• MSTest

.NET Versions

• .NET Standard 2.0+
• .NET 8.0+
• .NET Framework 4.6.2+

CHANGELOG

FAQs

Does Scand.StormPetrel help developers track incorrect expected baselines in tests?

Developers can already track incorrect expected baselines without Scand.StormPetrel. They should manually review the changes made by Scand.StormPetrel to the expected baselines and decide if they are correct. This is the same approach used when Scand.StormPetrel is not involved.

Does Scand.StormPetrel violate Test Driven Development (TDD) practices?

We believe it does not. Here are the corresponding TDD steps with comments explaining why it does not violate the practices:

• Write a test for the next bit of functionality you want to add.
  Scand.StormPetrel does not assist in creating test case input parameters. It cannot help create the expected baseline at this step because there is no actual result yet. Therefore, the expected baseline is either:
  • A default instance
  • A manually created instance
• Write the functional code until the test passes.
  Scand.StormPetrel can automatically update the default or manually created instance with the actual value here. The developer should then review the changes to decide if the test truly passes.

What is the suggested configuration of Scand.StormPetrel for typical CI/CD and development environments?

The suggested configuration in the appsettings.StormPetrel.json file is:

{
    "GeneratorConfig": {
        "BackuperExpression": null, //No need to backup because developers typically keep backups under Git or other Version Control Systems.
        "DumperExpression": ...,    //According to your requirements.
        "RewriterExpression": ...   //According to your requirements.
    },
    "IsDisabled": false,            //Keep `true` under Git control for CI/CD to speed up test compilation and execution time.
                                    //Keep `false` on the developer's machine, which should not be tracked by Git. StormPetrel tests are always available.
                                    //Keep `true` on the developer's machine. Can be changed ad hoc to `false` to compile StormPetrel tests.
    "Serilog": null                 //Avoid logging.
}

Can test project compilation fail after enabling Scand.StormPetrel? How can I avoid the failure?

Scand.StormPetrel relies on code syntax, not semantics. Therefore, it cannot properly generate StormPetrel test methods in all cases, and test project compilation might fail. You can detect the original test file causing the failure and add it to the "IgnoreFilePathRegex" property in the configuration to avoid the compilation error while still using Scand.StormPetrel for other tests.

Scand.StormPetrel constantly changes my expected baseline because the actual result has new property values (e.g., CreatedOn with the current time, auto-incremented Id, random Guid, etc.) every StormPetrel test call. How can I avoid these constant changes?

It is likely that you also ignore the property in the test assertion; otherwise, the test would fail. An option is to always have a default value for the property while dumping it to C# code using Scand.StormPetrel. This can be implemented via custom configuration or the implementation of IGeneratorDumper. See an example of how this is implemented via the GetDumpOptions method in Test.Integration.XUnit/Utils and configured in Test.Integration.XUnit/appsettings.StormPetrel.json.

References

At SCAND, we specialize in building advanced .NET solutions to help businesses develop new or modernize their legacy applications. If you need help getting started with Storm Petrel or support with implementation, we're ready to assist. Whether you're refactoring or rewriting, our team can help solve any challenges you might face. Visit our page to learn more, or reach out for hands-on guidance.