Xunit.Combinatorial 1.6.12-alpha

Prefix Reserved
This is a prerelease version of Xunit.Combinatorial.
There is a newer version of this package available.
See the version list below for details.
dotnet add package Xunit.Combinatorial --version 1.6.12-alpha                
NuGet\Install-Package Xunit.Combinatorial -Version 1.6.12-alpha                
This command is intended to be used within the Package Manager Console in Visual Studio, as it uses the NuGet module's version of Install-Package.
<PackageReference Include="Xunit.Combinatorial" Version="1.6.12-alpha" />                
For projects that support PackageReference, copy this XML node into the project file to reference the package.
paket add Xunit.Combinatorial --version 1.6.12-alpha                
#r "nuget: Xunit.Combinatorial, 1.6.12-alpha"                
#r directive can be used in F# Interactive and Polyglot Notebooks. Copy this into the interactive tool or source code of the script to reference the package.
// Install Xunit.Combinatorial as a Cake Addin
#addin nuget:?package=Xunit.Combinatorial&version=1.6.12-alpha&prerelease

// Install Xunit.Combinatorial as a Cake Tool
#tool nuget:?package=Xunit.Combinatorial&version=1.6.12-alpha&prerelease                

Xunit.Combinatorial

This project allows for parameterizing your Xunit test methods such that they run multiple times, once for each combination of possible arguments for your test method. You can also limit the number of test cases by using a pairwise strategy, which generally provides good coverage for testing but significantly reduces the test case explosion you might have when you have more than two parameters.

NuGet package Build Status

Installation

This project is available as a NuGet package.

Examples

Auto-generated values

Suppose you have this test method:

[Fact]
public void CheckFileSystem(bool recursive)
{
    // verifications here
}

To arrange for your test method to be invoked twice, once for each value of its bool parameter, change the attributes to:

[Theory, CombinatorialData]
public void CheckFileSystem(bool recursive)
{
    // verifications here
}

The CombinatorialDataAttribute will supply Xunit with both true and false arguments to run the test method with, resulting in two invocations of your test method with individual results reported for each invocation.

Custom-supplied values

To supply your own values to pass in for each parameter, use the CombinatorialValuesAttribute:

[Theory, CombinatorialData]
public void CheckValidAge([CombinatorialValues(5, 18, 21, 25)] int age)
{
    // verifications here
}

This will run your test method four times with each of the prescribed values.

Combinatorial effects

Of course it wouldn't be combinatorial without multiple parameters:

[Theory, CombinatorialData]
public void CheckValidAge(
    [CombinatorialValues(5, 18, 21, 25)] int age,
    bool friendlyOfficer)
{
    // This will run with all combinations:
    // 5  true
    // 18 true
    // 21 true
    // 25 true
    // 5  false
    // 18 false
    // 21 false
    // 25 false
}

Once you have more that two parameters, the number of test cases can grow dramatically in order to cover every possible combination. Consider this test with 3 parameters, each taking just two values:

[Theory, CombinatorialData]
public void CheckValidAge(bool p1, bool p2, bool p3)
{
    // Combinatorial generates these 8 test cases:
    // false false false
    // false false true
    // false true  false
    // false true  true
    // true  false false
    // true  false true
    // true  true  false
    // true  true  true
}

We already have 8 test cases. With more parameters or more values per parameter the test cases can quickly grow to a very large number. This can cause your test runs to take too long. This level of exhaustive testing is often not necessary as many bugs will show up whenever two parameters are specific values. This is called "pairwise testing" and it generates far fewer test cases than combinatorial testing because it only ensures there is a test case covering every combination of two parameters, and thus can "compress" the test cases by making each test case significantly test more than one pair.

To use pairwise testing, use the PairwiseDataAttribute instead of the CombinatorialDataAttribute:

[Theory, PairwiseData]
public void CheckValidAge(bool p1, bool p2, bool p3)
{
    // Pairwise generates these 4 test cases:
    // false false false
    // false true  true
    // true  false true
    // true  true  false
}

We have cut the number of test cases in half by using pairwise instead of combinatorial. In many cases the test reduction can be much greater. Notice that although the test cases are fewer, you can still find a test case that covers any two parameter values (thus pairwise).

Values over a range

To run a test with a parameter over a range of values, we have CombinatorialRangeAttribute to generate tests over intervals of integers.

[Theory, CombinatorialData]
public void CombinatorialCustomRange(
    [CombinatorialRange(0, 5)] int p1,
    [CombinatorialRange(0, 3, 2)] int p2)
{
    // Combinatorial generates these test cases:
    // 0 0
    // 1 0
    // 2 0
    // 3 0
    // 4 0
    // 0 2
    // 1 2
    // 2 2
    // 3 2
    // 4 2
}

CombinatorialRangeAttribute has two distinct constructors. When supplied with two integers from and count, Xunit will create a test case where the parameter equals from, and it will increment the parameter by 1 for count number of cases.

In the second constructor, CombinatorialRangeAttribute accepts three integer parameters. In the generated cases, the parameter value will step up from the first integer to the second integer, and the third integer specifies the interval of which to increment.

Value generated by a member

The CombinatorialMemberDataAttribute may be used to generate values for an individual Theory parameter using a static member on the test class. The static member may be a field, property or method.

A value-generating method is used here:

public static IEnumerable<int> GetRange(int start, int count)
{
    return Enumerable.Range(start, count);
}

[Theory, CombinatorialData]
public void CombinatorialMemberDataFromParameterizedMethods(
    [CombinatorialMemberData(nameof(GetRange), 0, 5)] int p1)
{
    Assert.True(true);
}

A value-generating property is used here:

public static IEnumerable<int> IntPropertyValues => GetIntMethodValues();

public static IEnumerable<int> GetIntMethodValues()
{
    for (int i = 0; i < 5; i++)
    {
        yield return Random.Next();
    }
}

[Theory, CombinatorialData]
public void CombinatorialMemberDataFromProperties(
    [CombinatorialMemberData(nameof(IntPropertyValues))] int p1)
{
    Assert.True(true);
}

A value-generating field also works:

public static readonly IEnumerable<int> IntFieldValues = Enumerable.Range(0, 5).Select(_ => Random.Next());

[Theory, CombinatorialData]
public void CombinatorialMemberDataFromFields(
    [CombinatorialMemberData(nameof(IntFieldValues))] int p2)
{
    Assert.True(true);
}

Randomly generated values

The CombinatorialRandomDataAttribute can be applied to theory parameters to generate random integer values. The min, max, and number of values can all be set via named parameters.

[Theory, CombinatorialData]
public void CombinatorialRandomValuesCount(
    [CombinatorialRandomData(Count = 10)] int p1)
{
    Assert.InRange(p1, 0, int.MaxValue);
}

[Theory, CombinatorialData]
public void CombinatorialRandomValuesCountMinMaxValues(
    [CombinatorialRandomData(Count = 10, Minimum = -20, Maximum = -5)] int p1)
{
    Assert.InRange(p1, -20, -5);
}
Product Compatible and additional computed target framework versions.
.NET net5.0 was computed.  net5.0-windows was computed.  net6.0 was computed.  net6.0-android was computed.  net6.0-ios was computed.  net6.0-maccatalyst was computed.  net6.0-macos was computed.  net6.0-tvos was computed.  net6.0-windows was computed.  net7.0 was computed.  net7.0-android was computed.  net7.0-ios was computed.  net7.0-maccatalyst was computed.  net7.0-macos was computed.  net7.0-tvos was computed.  net7.0-windows was computed.  net8.0 was computed.  net8.0-android was computed.  net8.0-browser was computed.  net8.0-ios was computed.  net8.0-maccatalyst was computed.  net8.0-macos was computed.  net8.0-tvos was computed.  net8.0-windows was computed. 
.NET Core netcoreapp2.0 was computed.  netcoreapp2.1 was computed.  netcoreapp2.2 was computed.  netcoreapp3.0 was computed.  netcoreapp3.1 was computed. 
.NET Standard netstandard2.0 is compatible.  netstandard2.1 was computed. 
.NET Framework net452 is compatible.  net46 was computed.  net461 was computed.  net462 was computed.  net463 was computed.  net47 was computed.  net471 was computed.  net472 was computed.  net48 was computed.  net481 was computed. 
MonoAndroid monoandroid was computed. 
MonoMac monomac was computed. 
MonoTouch monotouch was computed. 
Tizen tizen40 was computed.  tizen60 was computed. 
Xamarin.iOS xamarinios was computed. 
Xamarin.Mac xamarinmac was computed. 
Xamarin.TVOS xamarintvos was computed. 
Xamarin.WatchOS xamarinwatchos was computed. 
Compatible target framework(s)
Included target framework(s) (in package)
Learn more about Target Frameworks and .NET Standard.

NuGet packages

This package is not used by any NuGet packages.

GitHub repositories (39)

Showing the top 5 popular GitHub repositories that depend on Xunit.Combinatorial:

Repository Stars
dotnet/roslyn
The Roslyn .NET compiler provides C# and Visual Basic languages with rich code analysis APIs.
gui-cs/Terminal.Gui
Cross Platform Terminal UI toolkit for .NET
dotnet/machinelearning
ML.NET is an open source and cross-platform machine learning framework for .NET.
dotnet/sdk
Core functionality needed to create .NET Core projects, that is shared between Visual Studio and CLI
DotNetAnalyzers/StyleCopAnalyzers
An implementation of StyleCop rules using the .NET Compiler Platform
Version Downloads Last updated
1.6.24 670,528 8/29/2023
1.6.23-alpha 477 7/18/2023
1.6.12-alpha 4,564 11/9/2022
1.5.25 544,304 10/11/2022
1.5.7-beta 21,553 10/26/2021
1.5.2-beta 6,629 6/22/2021
1.4.1 1,004,033 8/3/2020
1.3.2 165,262 4/9/2020
1.2.7 641,339 7/16/2017
1.2.1 21,220 3/24/2017
1.1.20 12,496 5/15/2016
1.1.12 3,474 12/6/2015
1.1.12-g4ed6e8f5d0 851 12/6/2015
1.1.3 25,032 11/9/2015
1.0.15200-beta 1,717 7/19/2015
1.0.15199-beta2 1,262 7/19/2015
1.0.15199-beta 944 7/19/2015
1.0.0-beta-gee827b2da1 968 7/19/2015