Digitals.Testing 0.5.0

Digitals.Testing

One package that brings the Digitals test stack, the shared test severity policy and a coverage gate to a .NET test project. Reference it instead of wiring up six test packages, copying an .editorconfig between repos and re-deciding what "enough coverage" means in every pipeline.

It is the companion to Digitals.Analyzers: that package covers production code, this one covers tests. Everything test-related lives here.

Install

<PackageReference Include="Digitals.Testing" />

In the test project, not in Directory.Build.props — the package configures the project it is referenced from, and that scoping is how the test policy stays off your production code.

No PrivateAssets here. This package ships real runtime assemblies (xunit, NSubstitute, the coverage gate), so a shared test-utilities library that packs itself needs them to flow on.

You also need the Microsoft.Testing.Platform runner selected in global.json at the repo root:

{
	"sdk": {
		"version": "10.0.100",
		"rollForward": "latestMinor"
	},
	"test": {
		"runner": "Microsoft.Testing.Platform"
	}
}

This is enforced: the build fails with DIGT001 when it is missing, because without it .NET 10's dotnet test routes through the VSTest target, no coverage is collected, and the coverage gate never runs — a green build that enforces nothing. Staging a migration? Opt out explicitly, with a comment saying why:

<DigitalsAllowVSTestRunner>true</DigitalsAllowVSTestRunner>

Then delete the packages this one replaces — Microsoft.NET.Test.Sdk, xunit.runner.visualstudio, coverlet.collector, xunit.v3, NSubstitute. Leaving them produces version conflicts, and the first four fail the build with DIGT002 anyway.

Migrating an existing repo: docs/ADOPTION.md.

What you get

Test data generators are deliberately not in that list. Bogus, AutoFixture and the rest are a per-repo choice, and a repo that wants one adds it in a line.

Plus configuration that is injected automatically — nothing to import:

• Project defaults: OutputType=Exe (xunit.v3 test projects are executables), IsTestProject=true, IsPackable=false, and the MTP entry point.
• global using Xunit;: shipped as a Using item, so no test project declares it. Works with or without ImplicitUsings, and the SDK deduplicates, so declaring it yourself as well is harmless. Opt out with <Using Remove="Xunit" />.
• Test severity policy at global_level 300, so it beats both of Digitals.Analyzers' configs and can relax rules that are right for production code and wrong for tests.
• Coverage collection, always on. dotnet test needs no flags and CI needs no bespoke invocation.
• A coverage gate that fails the run below the org floor, plus a per-method CRAP limit and a per-method cyclomatic complexity cap.
• A TRX report in TestResults/, for the same reason: every results-publishing CI step reads one, and without it there is nothing to read. Turn it off per project with <DigitalsTrxReport>false</DigitalsTrxReport>.
• Custom rules (DT#####) — see below.
• Banned test packages (DIGT002, a build error) — see below.

Scoping, and why there is no .Tests convention here

Digitals.Analyzers applies to every project, so its test relaxations have to guess which projects are tests by matching the project name against .Tests. This package is referenced only by test projects, so the set of projects it applies to is the set of test projects. Name your test project whatever you like.

Coverage

Every dotnet test collects coverage into a Cobertura report and the gate judges it. The defaults are 100% line and 100% branch:

<DigitalsMinLineCoverage>100</DigitalsMinLineCoverage>
<DigitalsMinBranchCoverage>100</DigitalsMinBranchCoverage>

Below the floor, the run fails:

Line coverage 82.4% is below the 100.0% floor (416 of 505 covered). Add tests; do not lower the floor.

Both floors are a ratchet. An existing repo lands under them, so lower them once when you adopt the package, next to a comment naming what raises them back, and move them up as coverage climbs. Never down.

What is measured is set by Digitals.Testing.coverage.xml: test assemblies, auto-properties, [ExcludeFromCodeCoverage], generated code and EF migrations are all out. Point at your own file to override the policy:

<DigitalsCoverageSettingsFile>$(MSBuildThisFileDirectory)coverage.xml</DigitalsCoverageSettingsFile>

To turn the gate off while keeping the report:

<DigitalsSkipCoverageGate>true</DigitalsSkipCoverageGate>

Two things the gate does on purpose. It stays quiet when tests fail — a failing run has already said why, and coverage measured from a partial run is not a number worth judging. And it errors when it finds no report at all rather than passing: a gate that goes silent when it has no data is worse than no gate, because the build stays green while nothing is enforced.

Several reports are aggregated by summing covered and valid counts, then dividing once. Averaging the per-report rates would let a tiny uncovered assembly outweigh a large well-covered one.

The CRAP limit

The two floors are run-wide, so they cannot see the one method every bug lands in. CRAP (Change Risk Anti-Patterns) is judged per method instead:

complexity² × (1 − line coverage)³ + complexity

A fully covered method scores exactly its complexity; the same method with no tests scores complexity² + complexity. The cube is what makes the two interact, so covering a hairy method drops its score fast. The default limit is 30, which on its own would let a fully covered method reach a complexity of 30; the separate cap below brings that down to 20.

Over the limit, the run fails and names the five worst:

1 method(s) score above the CRAP limit of 30.0: Lib.Calculator.Classify(int) scores 210.0 (complexity 14, 0 of 11 lines covered). Cover the branches or split the method; raising the limit is the last resort.

Same ratchet as the floors. Raise it once on adoption with a comment naming what brings it back down, then lower it:

<DigitalsMaxCrapScore>220</DigitalsMaxCrapScore>

Or take the check off and keep the floors:

<DigitalsSkipCrapGate>true</DigitalsSkipCrapGate>

The complexity limit

At 100% coverage the CRAP score is the complexity, so a CRAP limit of 30 caps complexity at 30 and nothing more. Cyclomatic complexity is therefore capped on its own, read from the same reports and unmoved by coverage. The default is 20:

<DigitalsMaxCyclomaticComplexity>20</DigitalsMaxCyclomaticComplexity>

That is McCabe's 10 on the scale these reports use. Microsoft's coverage engine counts roughly two per decision point, so five sequential if statements measure 10 and ten measure 20. Read the numbers off your own Cobertura report rather than counting branches by eye, and note that a contiguous switch is the exception at one per arm: a 14-arm status mapper measures 15, while eight sparse cases measure 18.

Over the limit, the run fails and names the five worst:

2 method(s) are above the cyclomatic complexity limit of 20: Lib.Router.Dispatch(Request) has complexity 31; Lib.Calculator.Classify(int) has complexity 24. Split the method or flatten the branching; covering it does not help here.

Whole numbers only, since complexity counts decision points. Same ratchet as everything else, and the same escape hatch:

<DigitalsSkipComplexityGate>true</DigitalsSkipComplexityGate>

Two limits of measuring it this way. Only instrumented code is judged, so anything the coverage settings exclude — [ExcludeFromCodeCoverage], generated code, migrations — is outside the cap as well. And the engine misses a contiguous switch that sits behind an early-return guard: a nine-arm dispatcher after if (!verbose) { return "status"; } reports a complexity of 2.

Custom rules

Rule IDs here come from DT1####; Digitals.Analyzers keeps DT0####, so the two packages never collide in a #pragma or an .editorconfig.

DT10003 takes the trait's presence as policy and leaves the values to you, because a repo using Unit, Integration and Manual is as valid as one using two. Both halves are configurable:

# The trait name to require. Defaults to Category.
digitals_testing.required_trait = Category

# Optional. Unset means any non-empty value passes, which is the default.
digitals_testing.allowed_trait_values = Unit, Integration, Manual

DT10005 follows assertions through any method declared in the test project, however deep, so a test that calls its own AssertValid(result) passes. Code from another assembly is only recognised by name. Out of the box that is any Assert.* in any namespace, Verify's Verify*, NSubstitute's Received, DidNotReceive and Received.InOrder, and Roslyn's AnalyzerTest.RunAsync. A harness from a shared package gets added to the list, which extends the defaults and never replaces them:

# Comma-separated. Method, Type.Method or Namespace.Type.Method; a trailing * matches by prefix.
digitals_testing.assertion_methods = Company.Testing.ApiHarness.Expect*, Snapshot.Match

It replaces Sonar's S2699, which this package turns off because it cannot see into a helper. DT10005 reports at the end of the compilation, so it shows on build and in CI, and in the IDE only with full solution analysis on.

DT10004 applies to everything in the project, fixtures and helpers included, since only test projects reference this package. An integration fixture whose retry loop genuinely needs wall-clock delay is the case to suppress, with a comment saying why.

Relaxed in tests

These are enforced by Digitals.Analyzers on production code and switched off here, because they fight the way tests are written: DT00003, CA1822, CA1852, CA2201, CA1869, CA2012, CA1001, CA1859, S2344, S4144, S3881, S1172, S1186, S2701, IDE0060, VSTHRD200, VSTHRD002, VSTHRD003, CS8600, CS8604, CS8625.

One vendor rule goes the other way. xUnit1004 (a skipped test) ships as Info and is raised to warning here, so a Skip = "..." that got a build green does not stay unnoticed. A test meant to run only on demand is [Fact(Explicit = true)], which it does not flag.

The reason for each one is in Digitals.Testing.globalconfig.

Banned test packages

A direct PackageReference to any of these fails the build with DIGT002. Transitive dependencies are not checked.

The first four are new: they are the VSTest stack. Kept alongside this package they produce MTP0001 warnings, a coverage engine that never runs and a gate that silently enforces nothing.

If you genuinely need one, allow it explicitly with a comment saying why:

<DigitalsAllowBannedPackages>Moq</DigitalsAllowBannedPackages>

Turning something off

A whole rule, repo-wide. A repo-local .editorconfig always beats the packaged policy:

[*.cs]
dotnet_diagnostic.DT10001.severity = none

One line.

#pragma warning disable DT10001 // Named after the RFC section it covers
[Fact]
public void Rfc7231_Section_6_5_1() { }
#pragma warning restore DT10001

Suppressions are greppable and show up in review.

Notes

Build errors, not warnings, for the two structural checks: DIGT001 (MTP runner not selected) and DIGT002 (banned package). The DT##### rules ship as warning; how hard that bites is your repo's TreatWarningsAsErrors decision.

One version to review. Bumping xunit, NSubstitute or the coverage engine means a new version of this package, so an upgrade is a single version bump and a single changelog to read across every repo.

Versions come from git tags via MinVer and follow SemVer. Release notes: Releases.

License

MIT — see LICENSE. This covers the code in this repository: the config files, the banned-package check, the coverage gate and the custom DT##### analyzers. The bundled third-party packages keep their own licences and reach you as ordinary NuGet dependencies.

Working on the package itself: docs/MAINTAINING.md.