dotnet-diagnostics-benchmarkdotnet 0.32.1

dotnet-diagnostics-benchmarkdotnet

A BenchmarkDotNet IDiagnoser that attaches the dotnet-diagnostics engine in-process to a benchmark's child process while it runs and captures EventPipe perf indicators (GC, contention, thread pool, exceptions, JIT, counters, …) as JSON artifacts plus a per-run "biggest offenders" markdown report.

Diagnose, don't measure. EventPipe collectors are observe-only (no ptrace, no code injection) but add modest overhead. Run this on a dedicated diagnostic job (e.g. a RunStrategy.Monitoring job) and treat its timing as non-publication-grade. Keep MemoryDiagnoser/ThreadingDiagnoser for clean measurement; this adds the drill-down the MCP server / CLI provide.

Install

dotnet add package dotnet-diagnostics-benchmarkdotnet

<ItemGroup>
  <PackageReference Include="dotnet-diagnostics-benchmarkdotnet" Version="x.y.z" />
</ItemGroup>

Compatibility

This package targets net8.0, net9.0, and net10.0, so the consuming benchmark project can reference it from any of those TFMs.

Usage

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;
using DotnetDiagnostics.BenchmarkDotNet;

[DotnetDiagnosticsDiagnoser]   // attach the diagnoser + offenders report (like [MemoryDiagnoser])
public class Workload
{
    [Benchmark]
    // Prefer the type-safe, IntelliSense-discoverable enum overload:
    [DiagnosticKind(BenchmarkDiagnosticKind.Gc, DurationSeconds = 5)]
    public void AllocateLots() { /* ... */ }

    [Benchmark]
    [DiagnosticKind(BenchmarkDiagnosticKind.Contention, BenchmarkDiagnosticKind.ThreadPool)]  // run sequentially
    public void LockStorm() { /* ... */ }

    [Benchmark]
    [DiagnosticKind("gc,threadpool")]   // the free-text string overload still works (validated at run time)
    public void Legacy() { /* ... */ }
}

BenchmarkRunner.Run<Workload>();

Discoverability. Pass BenchmarkDiagnosticKind values for compile-time-checked, IntelliSense- completed kinds. The string overload remains for back-compat, but a typo there is only caught at BenchmarkDotNet validation time.

For outbound HTTP spans, explicitly opt in with [DiagnosticKind(BenchmarkDiagnosticKind.Activities, IncludeHttpDestination = true)]. This defaults to false and requires no target instrumentation. The JSON artifact carries separate redacted scheme/host/port destination evidence and availability/ correlation provenance; native tags are unchanged. Reports contain only counts and limitations, never authority values. Missing starts, ambiguous IDs, loss, or caps leave attribution unavailable. Existing native tags can still contain URLs; this option is not global URL scrubbing. See HTTP Activity destination evidence for exact subscription, privacy, and boundedness semantics.

Minimal vs Recommended setup

Minimal opt-in is enough when you just want diagnostics on an existing benchmark class with the least possible setup:

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;
using DotnetDiagnostics.BenchmarkDotNet;

[DotnetDiagnosticsDiagnoser]
public class Workload
{
    [Benchmark]
    [DiagnosticKind(BenchmarkDiagnosticKind.Gc, DurationSeconds = 5)]
    public void AllocateLots() { /* ... */ }
}

BenchmarkRunner.Run<Workload>();

Recommended setup is a dedicated IConfig with RunStrategy.Monitoring when you want a repeatable diagnostic job whose timing stays clearly separate from your publication-grade runs:

using BenchmarkDotNet.Columns;
using BenchmarkDotNet.Configs;
using BenchmarkDotNet.Exporters;
using BenchmarkDotNet.Engines;
using BenchmarkDotNet.Jobs;
using BenchmarkDotNet.Loggers;
using BenchmarkDotNet.Running;
using DotnetDiagnostics.BenchmarkDotNet;

public sealed class DiagnosedConfig : ManualConfig
{
    public DiagnosedConfig()
    {
        AddJob(Job.Default.WithStrategy(RunStrategy.Monitoring).WithId("Diagnose"));
        AddDiagnoser(new DotnetDiagnosticsDiagnoser());
        AddLogger(ConsoleLogger.Default);
        AddColumnProvider(DefaultColumnProviders.Instance);
        AddExporter(MarkdownExporter.GitHub);
    }
}

BenchmarkRunner.Run<Workload>(new DiagnosedConfig());

Use the minimal form for quick local exploration; switch to the dedicated config when you want a cleanly labeled monitoring job in CI or alongside native BenchmarkDotNet diagnosers. See benchmarks/DiagnosedBenchmarks/DiagnosedConfig.cs for the full reference implementation used in this repository. A ManualConfig starts empty, so you must add at least a logger and column provider yourself or BenchmarkDotNet will not show its normal console progress or summary table output.

The diagnoser runs in the BenchmarkDotNet orchestrator process (not the measured child), so the heavy ClrMD/TraceEvent dependencies it pulls in never contaminate the benchmark's timing or allocations. Each [DiagnosticKind] method gets one EventPipe collection per kind against the child PID; results land in <artifacts>/diagnostics/*.json and a consolidated *-dotnet-diagnostics-report.md.

Output

This diagnoser does not add columns to the BenchmarkDotNet summary table; ProcessResults returns Array.Empty<Metric>(), so look for console logs, <artifacts>/diagnostics/*.json, and the separate *-dotnet-diagnostics-report.md report instead.

Real report excerpt:

# dotnet-diagnostics — biggest offenders

## WorkloadBenchmarks.GcChurn: Diagnose(InvocationCount=1, IterationCount=1, RunStrategy=Monitoring, UnrollFactor=1, WarmupCount=1)

| kind | status | headline | artifact |
| --- | --- | --- | --- |
| gc | ok | 5407 collection(s), max pause 0.6ms, total pause 331.4ms. Omitted 200 retained event row(s) from inline; the handle retains them. Raw detail reached maxEvents=200; 5207 later event row(s) were omitted, but totals and generation counts remain exact. 5207 later heap-stat sample(s) were also omitted after maxEvents=200. | `DiagnosedBenchmarks.WorkloadBenchmarks_GcChurn_Diagnose-Diagnose.1.gc.json` |

Read the report top-down: each benchmark gets one table row per diagnostic kind, the headline column is the one-line takeaway, and the artifact file lets you drill into the full JSON payload.

For networking, latency population v2 includes failed completions in existing HTTP/DNS/TLS percentiles and HTTP operation groups. JSON Correlation.ByKind.http|dns|tls.Counts carries latencyPopulationVersion, paired (all samples), pairedFailed and pairedWithoutFailure. HTTP 503 is a status-error response, not automatically a transport failure; cancellation and timeout causes are not reliably classified by the provider. Zero pairs means unavailable, not measured zero. Missing version/metadata means legacy/unknown. LatencyAvailability derives HTTP/queue/DNS/TLS labels from these counts and CaptureQuality, including not-observed, uncorrelatable, incomplete and unknown evidence. Only measured permits interpreting a scalar zero as a duration. Counts also include percentileSamples (retained, not all accepted samples); HTTP adds queueSamples, queuePercentileSamples and queueRejectedSamples. Groups retain their own nullable PercentileSamples and availability. Reservoir approximation is independent of pairing/capture gaps. JSON and report headlines preserve the counts/qualifications and label unavailable p95 rather than 0.0ms. The report headline preserves population and independent capture-quality qualifications. See networking evidence semantics before comparing versions or interpreting a partial capture as complete.

CI regression comparisons

See docs/research/ci-performance-regression-spike.md for the feasibility spike behind this section (issue #647).

Performance gates need two physically separate runs:

1. Clean measurement: ordinary BenchmarkDotNet jobs plus native diagnosers produce timing and allocation observations. Persist at least three independent run documents.
2. Diagnostic attribution: a monitoring job with DotnetDiagnosticsDiagnoser produces EventPipe artifacts that explain where cost moved. Its timing is never benchmark-quality and never enters the regression calculation.

DotnetDiagnostics.BenchmarkDotNet.Regression provides the versioned PerfMeasurementRun, PerfDiagnosticRun, and PerfRegressionReport contracts plus an analyzer. It checks runtime, OS/RID, architecture, GC mode, runner class/image, workload version, parameters, and build identity before comparing runs. Duplicate capture IDs/timestamps, fewer than three compatible repetitions, missing or unstable unchanged controls, excessive run-level coefficient of variation, or an environment mismatch produce inconclusive or environment_changed, not a gate-shaped result. Allocation movement from a zero baseline additionally requires an absolute 32 B/op effect floor.

The issue #647 pilot under benchmarks/DiagnosedBenchmarks demonstrates the full measure → diagnose → report flow. Its GitHub Actions workflow is advisory-only and uploads the immutable input documents, raw BenchmarkDotNet output, EventPipe artifacts, and regenerable JSON/Markdown report.

The issue #651 follow-up adds PerfPairedMeasurement, PerfPairedExperimentManifest, and PerfPairedRegressionReport. It pairs clean captures from two refs on the same VM, requires at least three alternating pairs, and compares only exact workload contracts. Workload-set changes are explicit: new_unbaselined, removed, and contract_changed entries retain evidence but do not receive metric verdicts. The paired report is policy-versioned and always advisory for a single cohort; its feasibility section records checkout, restore/build, clean-pair, diagnostic, report, and bulk-upload duration and bytes separately.

Storage is intentionally tiered. Keep the compact normalized signals (metric name, stable method/type/site identity, value, unit, direction, sampling metadata, and provenance) for baseline history. Keep raw EventPipe captures only for a bounded investigation window; the compact document references each raw artifact by relative path, byte size, SHA-256, and retention period. The final verdict is derived output and can be regenerated rather than becoming the only retained evidence.

Supported collect kinds

counters, exceptions, gc, cpu, allocation, datas, catalog, activities, logs, jit, threadpool, contention, db, kestrel, networking, requests, gcdump. (event_source is intentionally excluded — it needs an explicit provider name.)

The kestrel, networking and requests kinds collect the ASP.NET Core / HTTP pipeline views — Kestrel server request timings, HttpClient/socket/DNS/TLS activity, and in-flight ASP.NET requests respectively — so a web/API benchmark can attribute its own server-side and outbound-I/O cost, not just CPU and allocations.

The gcdump kind captures a managed-heap retention snapshot (per-type instance counts and byte totals) via the EventPipe GCHeapSnapshot keyword — the same mechanism dotnet-gcdump uses, with no ptrace, no ClrMD attach and no dump file. It is the what survives complement to allocation's what churns: allocation (and MemoryDiagnoser) tell you how many bytes a method allocated; gcdump tells you which types are still retained on the heap. The full type table is retained behind the envelope's heap-snapshot handle for query_snapshot drill-down. gcdump is CoreCLR-only — requesting the GCHeapSnapshot keyword crashes .NET 10 NativeAOT targets, so on a NativeAOT child the capability is withheld and surfaces as a NotSupported entry rather than a crash.

The cpu kind runs the EventPipe CPU sampler (CoreCLR only) against the benchmark child and attributes cost per stack frame: each hotspot carries its exclusive (self) and inclusive (self + callees) sample counts, and the consolidated *-dotnet-diagnostics-report.md headlines the hottest self-cost frame (e.g. Hottest self-cost: MyApp.Serialize (59.7% exclusive)). The full caller→callee call tree is retained behind the envelope's drill-down handle. Source-line and generic-instantiation resolution are disabled in-process (no PDB/SourceLink I/O, no ClrMD attach).

The allocation kind runs the EventPipe allocation sampler (GCAllocationTick) and attributes allocated bytes per managed type (top-N by bytes and by event count, SOH vs LOH) and per call site (TopBySite — the leaf frame that allocated, byte-weighted, with a MethodIdentity for the assembly-mcp handoff), with the full merged allocation call-site tree behind the handle. It is the per-type and per-origin complement to MemoryDiagnoser's Allocated column — which types and where they came from, not just how many bytes. The headline reports the top type and the top site (e.g. Top site: System.Private.CoreLib!System.String.Ctor(wchar,int32) (… bytes)). On NativeAOT the runtime emits the event without a TypeName, so types roll up under <unknown> (flagged in the summary); drill the call-tree handle for native allocation sites.

Does the diagnostic count as allocation? No. The sampler is observe-only — it enables a native EventPipe keyword and reads events the target runtime already emits; it performs no managed allocation in the measured process, so it cannot inflate the benchmark's Allocated counter. With the default (out-of-process) toolchain the sampler additionally runs in the orchestrator process, never the measured child — MemoryDiagnoser reads the child's own GC counters across a different run. The one exception is the in-process toolchain ([InProcess]): there the benchmark shares this process, GC.GetTotalAllocatedBytes() is process-wide, and a co-located capture is not isolated — the allocation summary flags this explicitly. Prefer the default toolchain (or a dedicated diagnostic job) for clean allocation numbers.

EventPipe collectors must not run concurrently against one PID, so multiple kinds on a single method are collected sequentially within the measurement window. Keep the count small (1–2) and the per-kind duration short relative to the job's actual-run length.

Cross-collector investigation digest (cpu + allocation)

When a benchmark method carries both [DiagnosticKind("cpu")] and [DiagnosticKind("allocation")] (e.g. via [DiagnosticKind("cpu,allocation")] or two separate attributes), the exported *-dotnet-diagnostics-report.md adds a ### Cross-collector investigation digest (cpu + allocation) section for that benchmark: the top CPU self-time hotspots, top CPU wait/noise categories, the dominant hot-path leaf, and the top allocation types/call sites — the same evidence you'd otherwise gather from two separate drill-down round trips, bundled into one. This reuses the exact same InvestigationDigestBuilder correlation the MCP server's collect_batch tool computes (issue #825) and the standalone CLI's session REPL prints (issue #827) — the ranking/gating logic lives once in DotnetDiagnostics.Core and is not reimplemented per surface. A benchmark tagged with only one of the two kinds gets no digest section, exactly as before.

Not captured (intentionally out of scope)

These diagnostic kinds exist in the engine but are deliberately not exposed by the diagnoser, each for a concrete reason — they are not gaps:

• event_source — needs an explicit provider name, so it can't be selected by a bare kind token.
• startup — the diagnoser attaches after the benchmark host is already running, so it cannot observe cold-start loader/DI events.
• crash-guard — a crash/first-chance guard for a failing process, not a benchmark perf view.
• sweep — a combined multi-kind sweep; redundant here since a method can already list multiple kinds on its [DiagnosticKind].
• method-params — captures sensitive values through an explicit dynamic profiler attach and ReJIT instrumentation; it remains MCP-server-only and is never enabled by the diagnoser.
• off-CPU sampling (off_cpu) — attributes blocked/wait time, but needs host perf + kernel tracepoints + elevated privilege, so it is not portable in-process for a benchmark. Feasibility via the child-launch/container angle is tracked separately (issue #501).
• native-alloc (native-alloc) — native (unmanaged) allocation sampling is Windows/ETW-only and niche for managed benchmarks.

Stability

While this package is 0.x its public API carries no SemVer stability guarantee.

License

MIT — see the repository.