Bromine.Xray.Reporter 1.1.0

Bromine.Xray.Reporter

Framework-agnostic .NET library for reporting BDD/test results to Xray Cloud. Collects results during your test run and batch-uploads them as a Test Execution at the end. Works with Reqnroll, SpecFlow, NUnit, xUnit, MSTest, or anything else that can produce (testKey, status) pairs.

Features

• Zero coupling to a specific test framework
• Find or create long-lived Test Plans and Test Executions by label
• Per-test timing, retries, and ScenarioOutline example tracking
• Attach screenshots/files as evidence from anywhere in your test
• Rich-text (ADF) descriptions — bold/italic/links via a small builder, or raw ADF passthrough
• Set any Jira field on the Test Execution (e.g. a project-specific "Environment" select list)
• ILogger integration (or silent by default)
• CancellationToken support throughout
• Automatic retry with exponential backoff on 429 and 5xx (honors Retry-After)
• Thread-safe — safe for parallel test execution

Install

dotnet add package Bromine.Xray.Reporter

Quick start

using Bromine.Xray.Reporter;
using Bromine.Xray.Reporter.Models;

var reporter = new XrayTestReporter(new XrayOptions
{
    ClientId = Environment.GetEnvironmentVariable("XRAY_CLIENT_ID")!,
    ClientSecret = Environment.GetEnvironmentVariable("XRAY_CLIENT_SECRET")!,
    ProjectKey = "PROJ",
    Jira = new JiraOptions
    {
        BaseUrl = "https://yoursite.atlassian.net",
        Email = Environment.GetEnvironmentVariable("JIRA_EMAIL"),
        ApiToken = Environment.GetEnvironmentVariable("JIRA_API_TOKEN")
    }
});

// Find-or-create Test Plan + Test Execution, both keyed by label
await reporter.StartExecutionAsync(
    summary: "Nightly_Smoke_Tests",
    environment: "Staging",
    plan: new TestPlanRef(
        summary: "Smoke Test Coverage",
        label: "smoke-coverage"),
    execution: new TestExecutionRef(
        label: "nightly-smoke-staging"));

// Record results during your test run
reporter.StartTracking("PROJ-101");
// ... run the test ...
reporter.RecordResult(new XrayTestResult
{
    TestKey = "PROJ-101",
    Status = XrayTestStatus.Passed
});

// Submit at the end
var response = await reporter.SubmitResultsAsync();
Console.WriteLine($"Test Execution: {response?.Key}");

Reqnroll / SpecFlow integration

using Bromine.Xray.Reporter;
using Bromine.Xray.Reporter.Extensions;
using Bromine.Xray.Reporter.Models;

[Binding]
public class XrayHooks(ScenarioContext scenarioContext)
{
    private static XrayTestReporter _reporter = null!;

    [BeforeTestRun]
    public static async Task BeforeTestRun()
    {
        _reporter = new XrayTestReporter(new XrayOptions
        {
            ClientId = Environment.GetEnvironmentVariable("XRAY_CLIENT_ID")!,
            ClientSecret = Environment.GetEnvironmentVariable("XRAY_CLIENT_SECRET")!,
            ProjectKey = "PROJ",
            Jira = new JiraOptions
            {
                BaseUrl = "https://yoursite.atlassian.net",
                Email = Environment.GetEnvironmentVariable("JIRA_EMAIL"),
                ApiToken = Environment.GetEnvironmentVariable("JIRA_API_TOKEN")
            }
        });

        await _reporter.StartExecutionAsync(
            summary: $"Nightly Run {DateTime.UtcNow:yyyy-MM-dd}",
            environment: "Staging",
            plan: new TestPlanRef("Smoke Test Coverage", "smoke-coverage"),
            execution: new TestExecutionRef("nightly-smoke-staging"));
    }

    [BeforeScenario]
    public void BeforeScenario()
    {
        var key = scenarioContext.ScenarioInfo.Tags.GetXrayKey();
        if (key is not null)
            _reporter.StartTracking(key);
    }

    [AfterScenario]
    public void AfterScenario()
    {
        var key = scenarioContext.ScenarioInfo.Tags.GetXrayKey();
        if (key is null) return;

        var error = scenarioContext.TestError;
        _reporter.RecordResult(new XrayTestResult
        {
            TestKey = key,
            Status = error is not null ? XrayTestStatus.Failed : XrayTestStatus.Passed,
            Comment = error?.Message
        }, label: GetScenarioOutlineLabel());
    }

    [AfterTestRun]
    public static async Task AfterTestRun()
    {
        var result = await _reporter.SubmitResultsAsync();
        _reporter.Dispose();
    }

    private string? GetScenarioOutlineLabel()
    {
        var args = scenarioContext.ScenarioInfo.Arguments;
        if (args is null || args.Count == 0) return null;
        return string.Join(", ", args.Keys.Cast<string>().Select(k => $"{k}: {args[k]}"));
    }
}

Tag your scenarios with @Key:PROJ-42 (or use a custom prefix — see Tag format).

Tag format

Default: @Key:PROJ-42. The prefix is configurable:

tags.GetXrayKey()            // matches @Key:PROJ-42
tags.GetXrayKey("TestId")    // matches @TestId:PROJ-42
tags.GetXrayKeys("Xray")     // matches @Xray:PROJ-312, @Xray:PROJ-313 (multi-key)

Appending text to a test's comment

Use AppendComment(testKey, text) to incrementally build up the comment for a test from anywhere during the run — step-by-step progress, log lines, anything that should end up in the Xray Test Run's comment field. Buffered per test key, drained into the result when RecordResult is called.

private DateTime _stepStart;

[BeforeStep]
public void TrackStep() => _stepStart = DateTime.UtcNow;

[AfterStep]
public void RecordStepResult()
{
    var key = scenarioContext.ScenarioInfo.Tags.GetXrayKey();
    if (key is null) return;

    var info = scenarioContext.StepContext.StepInfo;
    var duration = DateTime.UtcNow - _stepStart;
    var err = scenarioContext.TestError;
    var status = err is null ? XrayTestStatus.Passed : XrayTestStatus.Failed;

    var line = $"- {info.StepDefinitionType} {info.Text}: {status} ({duration:mm\\:ss})";
    if (err is not null) line += $" — {err.Message}";

    _reporter.AppendComment(key, line);
}

The comment for the test will look like:

FAILED (00:45)
- Given I am logged in: PASSED (00:02)
- When I click the button: PASSED (00:01)
- Then I see the dialog: FAILED (00:42) — Element not found: #dialog

This works with any framework — the library is just a string buffer with a flush-on-RecordResult lifecycle. Reqnroll, SpecFlow, raw NUnit, anything with before/after hooks works.

Attaching evidence

Attach screenshots or files from anywhere during a scenario — they're merged into the result on RecordResult:

var bytes = screenshot.AsByteArray;
_reporter.AttachEvidence(key, Convert.ToBase64String(bytes), "screenshot.png");
_reporter.AttachEvidence(key, Convert.ToBase64String(logBytes), "test.log", XrayContentType.PlainText);

ScenarioOutline / parameterized tests

Pass a label to RecordResult to distinguish example rows:

_reporter.RecordResult(new XrayTestResult
{
    TestKey = "PROJ-42",
    Status = XrayTestStatus.Passed
}, label: "user_role: admin");

When the same testKey is recorded multiple times with labels:

• All attempts are preserved with timing and evidence
• Overall status is PASSED only if every example row's last attempt passed (retries on failed rows are respected)
• The comment includes a structured breakdown of all rows

Test Plan / Test Execution reuse

StartExecutionAsync with TestPlanRef and TestExecutionRef finds existing issues by label and reuses them, or creates new ones if not found. This lets you maintain long-lived Test Plans/Executions that accumulate history across runs.

await reporter.StartExecutionAsync(
    summary: "Nightly Smoke - 2026-06-04",
    plan: new TestPlanRef("Smoke Coverage", "smoke-coverage"),       // find or create by label
    execution: new TestExecutionRef("nightly-smoke-staging"));        // find or create by label

On reuse:

• The execution's summary and description are updated each run
• Test Plan and Test Environments are set once at creation (Xray Cloud limitation — these custom fields aren't on Jira's edit screen and can't be updated after the fact)

Rich-text descriptions (ADF)

Jira Cloud (REST v3) stores descriptions in Atlassian Document Format and renders text nodes verbatim — wiki markup like *bold* and Markdown are not parsed. Every description parameter in the library accepts an AdfDocument for real formatting:

await reporter.StartExecutionAsync(
    summary: "Nightly_Smoke_Tests",
    description: new AdfDocument()
        .Paragraph(p => p.Text("Nightly run on ").Bold("Staging").Text(" — ").Italic("develop"))
        .Paragraph(p => p.Link("Pipeline", "https://ci.example.com/run/123")),
    execution: new TestExecutionRef("nightly-smoke-staging"));

Plain strings still work everywhere (implicit conversion) — each line becomes its own paragraph, so multi-line descriptions render with real line breaks. The builder supports Text, Bold, Italic, Code, Link, and LineBreak (a line break within the paragraph); for anything else (tables, panels, ...) pass raw ADF:

description: AdfDocument.FromJson("""{"type":"doc","version":1,"content":[ ... ]}""")

The library appends a Last run: ... | Duration: ... paragraph on every submit.

Setting Jira fields on the Test Execution

The environment parameter of StartExecution(Async) maps to Xray's own "Test Environments" field, which Xray only sets at execution creation. If your project uses its own Jira fields — say an "Environment" select list — set them with SetField, or map any number at once with SetFields:

reporter.SetField("Environment", "Staging");        // resolve by display name
reporter.SetField("customfield_10055", "Staging");  // or by field ID

// bulk — however many fields, it's one field-list lookup and one Jira update:
reporter.SetFields(new Dictionary<string, object?>
{
    ["Environment"]       = "Staging",
    ["Operator"]          = "GR",
    ["Release Notes"]     = new AdfDocument().Paragraph(p => p.Bold("v1.4.0")),
    ["customfield_10200"] = new { value = "Nightly" }   // pre-shaped, sent as-is
});

• Names are resolved case-insensitively via Jira's field list. If a name matches both a system field and a single custom field (the classic case: Jira's built-in Environment vs. your own), the custom field wins — pass the customfield_XXXXX ID to target the system field or disambiguate.
• String values are shaped automatically from the field's schema: select lists become {"value": ...}, multi-selects [{"value": ...}], and rich-text fields (including Jira's system Environment) are wrapped as ADF.
• For full control pass an AdfDocument or a pre-shaped object: reporter.SetField("customfield_10200", new { value = "GR" }).
• Fields are applied via Jira REST after every import, so unlike Test Environments this works for reused executions too. Failures are logged and never fail the submission — and if Jira rejects a specific field (e.g. it's not on the project's edit screen), that field is dropped with a warning and the remaining fields are retried.
• To discover what you can map, call reporter.GetJiraFieldsAsync() (every field with its schema), or hit GET /rest/api/3/issue/{key}/editmeta for the definitive list of fields settable on a given issue.

Logging

Pass an ILogger<XrayTestReporter> to see what's happening:

var loggerFactory = LoggerFactory.Create(b => b.AddConsole());
var reporter = new XrayTestReporter(options, loggerFactory.CreateLogger<XrayTestReporter>());

Without a logger, the library is silent.

Cancellation

All async methods accept a CancellationToken:

using var cts = new CancellationTokenSource(TimeSpan.FromMinutes(5));
await reporter.SubmitResultsAsync(cts.Token);

Cancellation propagates cleanly through retries and HTTP calls.

API reference

XrayTestReporter

XrayTestStatus (string constants)

These constants are the Xray Cloud defaults. Your Xray admin can rename them (e.g. "PASSED" → "PASS") or add custom ones (e.g. "PARTIALLY_PASSED") per project. The library accepts any string — Xray itself validates the value on import.

// Use a built-in constant
reporter.RecordResult(new XrayTestResult { TestKey = "PROJ-1", Status = XrayTestStatus.Passed });

// Or any custom string your project supports
reporter.RecordResult(new XrayTestResult { TestKey = "PROJ-1", Status = "PARTIALLY_PASSED" });

If your project renames the success/failure statuses, configure them on XrayOptions so the ScenarioOutline overall-status resolver works correctly:

var options = new XrayOptions
{
    // ... credentials ...
    PassedStatus = "PASS",   // default: "PASSED"
    FailedStatus = "FAIL"    // default: "FAILED"
};

XrayContentType (enum)

Strict enum for evidence MIME types — consumers can't pass arbitrary strings.

XrayOptions

JiraOptions

Permissions

The Xray client ID/secret needs:

• Read/write access to test executions in the configured project

The Jira API token needs:

• Read access to issues (JQL search by label)
• Create issues (for Test Plans / first-time Test Executions)
• Edit issues (summary, description, labels, parent)
• Create issue links (Test Plan ↔ Test Execution)

Xray Cloud constraints

• Status values must match your project's Xray status configuration
• Test Plan and Test Environments custom fields can't be updated on existing executions through the import endpoint — they're set at creation only
• The v2 import/execution endpoint does not support iterations or labels in the info block — this library handles both via Jira REST and structured comments instead

Documentation

• Xray Cloud REST API v2 — Import Execution Results
• Xray Cloud Authentication
• Jira Cloud REST API v3

Contributing

PRs welcome. To build and test locally:

dotnet build
dotnet test

License

MIT — see LICENSE.