Xunit.Combinatorial 1.6.12-alpha

The ID prefix of this package has been reserved for one of the owners of this package by NuGet.org. Prefix Reserved
.NET Standard 2.0 .NET Framework 4.5.2
This is a prerelease version of Xunit.Combinatorial.
dotnet add package Xunit.Combinatorial --version 1.6.12-alpha
NuGet\Install-Package Xunit.Combinatorial -Version 1.6.12-alpha
This command is intended to be used within the Package Manager Console in Visual Studio, as it uses the NuGet module's version of Install-Package.
<PackageReference Include="Xunit.Combinatorial" Version="1.6.12-alpha" />
For projects that support PackageReference, copy this XML node into the project file to reference the package.
paket add Xunit.Combinatorial --version 1.6.12-alpha
#r "nuget: Xunit.Combinatorial, 1.6.12-alpha"
#r directive can be used in F# Interactive and Polyglot Notebooks. Copy this into the interactive tool or source code of the script to reference the package.
// Install Xunit.Combinatorial as a Cake Addin
#addin nuget:?package=Xunit.Combinatorial&version=1.6.12-alpha&prerelease

// Install Xunit.Combinatorial as a Cake Tool
#tool nuget:?package=Xunit.Combinatorial&version=1.6.12-alpha&prerelease

Xunit.Combinatorial

This project allows for parameterizing your Xunit test methods such that they run multiple times, once for each combination of possible arguments for your test method. You can also limit the number of test cases by using a pairwise strategy, which generally provides good coverage for testing but significantly reduces the test case explosion you might have when you have more than two parameters.

NuGet package Build Status

Installation

This project is available as a NuGet package.

Examples

Auto-generated values

Suppose you have this test method:

[Fact]
public void CheckFileSystem(bool recursive)
{
    // verifications here
}

To arrange for your test method to be invoked twice, once for each value of its bool parameter, change the attributes to:

[Theory, CombinatorialData]
public void CheckFileSystem(bool recursive)
{
    // verifications here
}

The CombinatorialDataAttribute will supply Xunit with both true and false arguments to run the test method with, resulting in two invocations of your test method with individual results reported for each invocation.

Custom-supplied values

To supply your own values to pass in for each parameter, use the CombinatorialValuesAttribute:

[Theory, CombinatorialData]
public void CheckValidAge([CombinatorialValues(5, 18, 21, 25)] int age)
{
    // verifications here
}

This will run your test method four times with each of the prescribed values.

Combinatorial effects

Of course it wouldn't be combinatorial without multiple parameters:

[Theory, CombinatorialData]
public void CheckValidAge(
    [CombinatorialValues(5, 18, 21, 25)] int age,
    bool friendlyOfficer)
{
    // This will run with all combinations:
    // 5  true
    // 18 true
    // 21 true
    // 25 true
    // 5  false
    // 18 false
    // 21 false
    // 25 false
}

Once you have more that two parameters, the number of test cases can grow dramatically in order to cover every possible combination. Consider this test with 3 parameters, each taking just two values:

[Theory, CombinatorialData]
public void CheckValidAge(bool p1, bool p2, bool p3)
{
    // Combinatorial generates these 8 test cases:
    // false false false
    // false false true
    // false true  false
    // false true  true
    // true  false false
    // true  false true
    // true  true  false
    // true  true  true
}

We already have 8 test cases. With more parameters or more values per parameter the test cases can quickly grow to a very large number. This can cause your test runs to take too long. This level of exhaustive testing is often not necessary as many bugs will show up whenever two parameters are specific values. This is called "pairwise testing" and it generates far fewer test cases than combinatorial testing because it only ensures there is a test case covering every combination of two parameters, and thus can "compress" the test cases by making each test case significantly test more than one pair.

To use pairwise testing, use the PairwiseDataAttribute instead of the CombinatorialDataAttribute:

[Theory, PairwiseData]
public void CheckValidAge(bool p1, bool p2, bool p3)
{
    // Pairwise generates these 4 test cases:
    // false false false
    // false true  true
    // true  false true
    // true  true  false
}

We have cut the number of test cases in half by using pairwise instead of combinatorial. In many cases the test reduction can be much greater. Notice that although the test cases are fewer, you can still find a test case that covers any two parameter values (thus pairwise).

Values over a range

To run a test with a parameter over a range of values, we have CombinatorialRangeAttribute to generate tests over intervals of integers.

[Theory, CombinatorialData]
public void CombinatorialCustomRange(
    [CombinatorialRange(0, 5)] int p1,
    [CombinatorialRange(0, 3, 2)] int p2)
{
    // Combinatorial generates these test cases:
    // 0 0
    // 1 0
    // 2 0
    // 3 0
    // 4 0
    // 0 2
    // 1 2
    // 2 2
    // 3 2
    // 4 2
}

CombinatorialRangeAttribute has two distinct constructors. When supplied with two integers from and count, Xunit will create a test case where the parameter equals from, and it will increment the parameter by 1 for count number of cases.

In the second constructor, CombinatorialRangeAttribute accepts three integer parameters. In the generated cases, the parameter value will step up from the first integer to the second integer, and the third integer specifies the interval of which to increment.

Value generated by a member

The CombinatorialMemberDataAttribute may be used to generate values for an individual Theory parameter using a static member on the test class. The static member may be a field, property or method.

A value-generating method is used here:

public static IEnumerable<int> GetRange(int start, int count)
{
    return Enumerable.Range(start, count);
}

[Theory, CombinatorialData]
public void CombinatorialMemberDataFromParameterizedMethods(
    [CombinatorialMemberData(nameof(GetRange), 0, 5)] int p1)
{
    Assert.True(true);
}

A value-generating property is used here:

public static IEnumerable<int> IntPropertyValues => GetIntMethodValues();

public static IEnumerable<int> GetIntMethodValues()
{
    for (int i = 0; i < 5; i++)
    {
        yield return Random.Next();
    }
}

[Theory, CombinatorialData]
public void CombinatorialMemberDataFromProperties(
    [CombinatorialMemberData(nameof(IntPropertyValues))] int p1)
{
    Assert.True(true);
}

A value-generating field also works:

public static readonly IEnumerable<int> IntFieldValues = Enumerable.Range(0, 5).Select(_ => Random.Next());

[Theory, CombinatorialData]
public void CombinatorialMemberDataFromFields(
    [CombinatorialMemberData(nameof(IntFieldValues))] int p2)
{
    Assert.True(true);
}

Randomly generated values

The CombinatorialRandomDataAttribute can be applied to theory parameters to generate random integer values. The min, max, and number of values can all be set via named parameters.

[Theory, CombinatorialData]
public void CombinatorialRandomValuesCount(
    [CombinatorialRandomData(Count = 10)] int p1)
{
    Assert.InRange(p1, 0, int.MaxValue);
}

[Theory, CombinatorialData]
public void CombinatorialRandomValuesCountMinMaxValues(
    [CombinatorialRandomData(Count = 10, Minimum = -20, Maximum = -5)] int p1)
{
    Assert.InRange(p1, -20, -5);
}
Product Versions
.NET net5.0 net5.0-windows net6.0 net6.0-android net6.0-ios net6.0-maccatalyst net6.0-macos net6.0-tvos net6.0-windows net7.0 net7.0-android net7.0-ios net7.0-maccatalyst net7.0-macos net7.0-tvos net7.0-windows
.NET Core netcoreapp2.0 netcoreapp2.1 netcoreapp2.2 netcoreapp3.0 netcoreapp3.1
.NET Standard netstandard2.0 netstandard2.1
.NET Framework net452 net46 net461 net462 net463 net47 net471 net472 net48 net481
MonoAndroid monoandroid
MonoMac monomac
MonoTouch monotouch
Tizen tizen40 tizen60
Xamarin.iOS xamarinios
Xamarin.Mac xamarinmac
Xamarin.TVOS xamarintvos
Xamarin.WatchOS xamarinwatchos
Compatible target framework(s)
Additional computed target framework(s)
Learn more about Target Frameworks and .NET Standard.

NuGet packages

This package is not used by any NuGet packages.

GitHub repositories (23)

Showing the top 5 popular GitHub repositories that depend on Xunit.Combinatorial:

Repository Stars
dotnet/roslyn
The Roslyn .NET compiler provides C# and Visual Basic languages with rich code analysis APIs.
dotnet/machinelearning
ML.NET is an open source and cross-platform machine learning framework for .NET.
DotNetAnalyzers/StyleCopAnalyzers
An implementation of StyleCop rules using the .NET Compiler Platform
YarnSpinnerTool/YarnSpinner
Yarn Spinner is a tool for building interactive dialogue in games!
dotnet/roslyn-analyzers
Version Downloads Last updated
1.6.12-alpha 362 11/9/2022
1.5.25 76,621 10/11/2022
1.5.7-beta 13,183 10/26/2021
1.5.2-beta 4,965 6/22/2021
1.4.1 576,564 8/3/2020
1.3.2 158,284 4/9/2020
1.2.7 591,174 7/16/2017
1.2.1 20,942 3/24/2017
1.1.20 11,244 5/15/2016
1.1.12 3,320 12/6/2015
1.1.12-g4ed6e8f5d0 766 12/6/2015
1.1.3 12,942 11/9/2015
1.0.15200-beta 1,600 7/19/2015
1.0.15199-beta2 1,178 7/19/2015
1.0.15199-beta 844 7/19/2015
1.0.0-beta-gee827b2da1 851 7/19/2015