Davasorus.Utility.DotNet.Telemetry 2026.2.1.1

Tyler Utility - OpenTelemetry Integration

OpenTelemetry distributed tracing and telemetry integration for Tyler Utility packages. Provides optional activity sources, spans, and tracing capabilities with Fluent API configuration.

Features

• ✅ Optional Integration - Works without telemetry configured; no breaking changes
• ✅ Fluent API Configuration - Chainable, intuitive setup
• ✅ Safe by Default - All extension methods are null-safe and no-op when telemetry is disabled
• ✅ Multiple Exporters - Console, OTLP (gRPC/HTTP), and custom exporters
• ✅ Activity Source Management - Standardized naming and creation
• ✅ Logging Integration - Correlate logs with traces via TraceId/SpanId
• ✅ Comprehensive Instrumentation - HTTP, SQL, ASP.NET Core, Entity Framework, gRPC, AWS SDK, Redis
• ✅ Metrics Collection - Collect and export application metrics with configurable intervals
• ✅ Flexible Sampling - AlwaysOn, AlwaysOff, TraceIdRatio, ParentBased strategies
• ✅ Batch Export Optimization - Configurable queue size, batch size, and delays
• ✅ Compression Support - Gzip compression for bandwidth reduction (70-90% typical)
• ✅ Environment Auto-Detection - Automatic detection from ASPNETCORE_ENVIRONMENT/DOTNET_ENVIRONMENT
• ✅ Resource Detectors - Container and host resource detection
• ✅ Multiple Propagation Formats - W3C, B3, Jaeger, AWS X-Ray
• ✅ Exception Tracking - Record exceptions with full context
• ✅ Diagnostic Logging - Troubleshooting support for telemetry initialization
• ✅ URL Validation - Fail-fast validation at startup
• ✅ Performance Monitoring - Automatic span timing and metrics

Installation

dotnet add package Davasorus.Utility.DotNet.Telemetry

Integration Guide

This section shows how to integrate the Telemetry package into your existing applications and libraries.

Step 1: Add Package Reference

Add the telemetry package reference to your .csproj file:

<PackageReference Include="Davasorus.Utility.DotNet.Telemetry" Version="2025.4.3.1" />

Step 2: Create Activity Source

In your service or client class, create a static ActivitySource:

using System.Diagnostics;
using Tyler.Utility.DotNet.Telemetry;

public class CacheService
{
    // Create activity source for this component
    private static readonly ActivitySource ActivitySource =
        ActivitySourceHelper.Create("Cache", "Memory");

    private readonly ILogger<CacheService> _logger;

    // ... rest of class
}

Step 3: Update Methods to Use Telemetry

Before (Logging Only):

public async Task<T?> GetAsync<T>(string key, CancellationToken cancellationToken = default)
{
    using (_logger.BeginScope(new Dictionary<string, object>
    {
        ["CacheKey"] = key,
        ["Operation"] = "Get"
    }))
    {
        try
        {
            _logger.LogDebug("Retrieving value from cache for key: {Key}", key);
            var result = await _client.GetAsync<T>(key, cancellationToken);

            if (result != null)
            {
                _logger.LogDebug("Cache hit for key: {Key}", key);
            }
            else
            {
                _logger.LogDebug("Cache miss for key: {Key}", key);
            }

            return result;
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, "Error retrieving value from cache for key: {Key}", key);
            throw;
        }
    }
}

After (With Telemetry):

public async Task<T?> GetAsync<T>(string key, CancellationToken cancellationToken = default)
{
    // Start activity (returns null if telemetry not configured - safe!)
    using var activity = ActivitySource.StartActivitySafe("Cache.Get");
    activity.AddTagSafe("cache.key", key);
    activity.AddTagSafe("cache.operation", "get");

    // Enrich logging scope with trace context
    using (_logger.BeginScope(activity.ToLoggingScope(new Dictionary<string, object>
    {
        ["CacheKey"] = key,
        ["Operation"] = "Get"
    })))
    {
        try
        {
            _logger.LogDebug("Retrieving value from cache for key: {Key}", key);
            var result = await _client.GetAsync<T>(key, cancellationToken);

            // Add telemetry tags
            activity.AddTagSafe("cache.hit", result != null);

            if (result != null)
            {
                _logger.LogDebug("Cache hit for key: {Key}", key);
            }
            else
            {
                _logger.LogDebug("Cache miss for key: {Key}", key);
            }

            // Mark success
            activity.SetStatusSafe(ActivityStatusCode.Ok);

            return result;
        }
        catch (Exception ex)
        {
            // Record exception in telemetry
            activity.RecordExceptionSafe(ex);
            _logger.LogError(ex, "Error retrieving value from cache for key: {Key}", key);
            throw;
        }
    }
}

Alternative: Simplified Combined Approach

public async Task<T?> GetAsync<T>(string key, CancellationToken cancellationToken = default)
{
    // Combines activity start and logging scope in one call
    var (activity, logScope) = ActivitySource.StartActivityWithLogging(
        _logger,
        "Cache.Get",
        new Dictionary<string, object>
        {
            ["CacheKey"] = key,
            ["Operation"] = "Get",
            ["cache.operation"] = "get"
        }
    );

    using (activity)
    using (logScope)
    {
        activity.AddTagSafe("cache.key", key);

        try
        {
            _logger.LogDebug("Retrieving value from cache");
            var result = await _client.GetAsync<T>(key, cancellationToken);

            activity.AddTagSafe("cache.hit", result != null);
            activity.SetStatusSafe(ActivityStatusCode.Ok);

            return result;
        }
        catch (Exception ex)
        {
            activity.RecordExceptionSafe(ex);
            _logger.LogError(ex, "Cache retrieval failed");
            throw;
        }
    }
}

Complete Example: Cache Service Integration

using System.Diagnostics;
using Microsoft.Extensions.Logging;
using Tyler.Utility.DotNet.Telemetry;

namespace MyApp.Services;

public class CacheService : ICacheService
{
    private static readonly ActivitySource ActivitySource =
        ActivitySourceHelper.Create("Cache", "Memory");

    private readonly ICacheClient _client;
    private readonly ILogger<CacheService> _logger;

    public CacheService(ICacheClient client, ILogger<CacheService> logger)
    {
        _client = client;
        _logger = logger;
    }

    public async Task<T?> GetAsync<T>(string key, CancellationToken cancellationToken = default)
    {
        using var activity = ActivitySource.StartActivitySafe("Cache.Get");
        activity.AddTagSafe("cache.key", key);
        activity.AddTagSafe("cache.operation", "get");
        activity.AddTagSafe("cache.backend", "memory");

        using (_logger.BeginScope(activity.ToLoggingScope(new Dictionary<string, object>
        {
            ["CacheKey"] = key,
            ["Operation"] = "Get"
        })))
        {
            try
            {
                _logger.LogDebug("Retrieving from cache");
                var result = await _client.GetAsync<T>(key, cancellationToken);

                activity.AddTagSafe("cache.hit", result != null);
                activity.SetStatusSafe(ActivityStatusCode.Ok);

                return result;
            }
            catch (Exception ex)
            {
                activity.RecordExceptionSafe(ex);
                _logger.LogError(ex, "Cache retrieval failed");
                throw;
            }
        }
    }

    public async Task SetAsync<T>(
        string key,
        T value,
        CacheOptions? options = null,
        CancellationToken cancellationToken = default)
    {
        using var activity = ActivitySource.StartActivitySafe("Cache.Set");
        activity.AddTagSafe("cache.key", key);
        activity.AddTagSafe("cache.operation", "set");
        activity.AddTagSafe("cache.backend", "memory");

        using (_logger.BeginScope(activity.ToLoggingScope(new Dictionary<string, object>
        {
            ["CacheKey"] = key,
            ["Operation"] = "Set"
        })))
        {
            try
            {
                _logger.LogDebug("Setting value in cache");
                await _client.SetAsync(key, value, options, cancellationToken);

                activity.SetStatusSafe(ActivityStatusCode.Ok);
                _logger.LogDebug("Successfully set value");
            }
            catch (Exception ex)
            {
                activity.RecordExceptionSafe(ex);
                _logger.LogError(ex, "Cache set failed");
                throw;
            }
        }
    }
}

Testing Without Telemetry

Your existing tests will continue to work without any changes. The telemetry methods are all no-ops when telemetry is not configured:

[Fact]
public async Task GetAsync_ReturnsValue()
{
    // Arrange
    var logger = new Mock<ILogger<CacheService>>();
    var client = new Mock<ICacheClient>();
    var service = new CacheService(client.Object, logger.Object);

    // Act
    var result = await service.GetAsync<string>("key");

    // Assert
    Assert.NotNull(result);
    // Telemetry code runs but doesn't affect the test
}

Migration Checklist

• Add Telemetry package reference to .csproj
• Add using System.Diagnostics; and using Tyler.Utility.DotNet.Telemetry;
• Create static ActivitySource for the component
• Update methods to start activities with StartActivitySafe
• Add relevant tags with AddTagSafe
• Enrich logging scopes with ToLoggingScope
• Record exceptions with RecordExceptionSafe
• Set status codes with SetStatusSafe
• Build and test
• Update README with telemetry information

Quick Start

1. Basic Configuration (Optional)

Telemetry is completely optional. Your application works without it.

Note on Protocols: For standard OTLP collectors (Jaeger, Tempo), use gRPC with port 4317 and set useHttpProtobuf: false. For HTTP/Protobuf collectors (Seq, etc.), use port 4318 or provider-specific endpoint and set useHttpProtobuf: true.

using Tyler.Utility.DotNet.Telemetry.Configuration;

// Option A: Fluent API with OTLP exporter
builder.Services.AddTylerTelemetry(telemetry => telemetry
    .WithServiceName("MyApplication")
    .WithServiceVersion("1.0.0")
    .WithOtlpExporter("http://localhost:4317", useHttpProtobuf: false)  // gRPC for Jaeger/Tempo
);

// Option B: Console for local development
builder.Services.AddTylerTelemetry(telemetry => telemetry
    .WithServiceName("MyApplication")
    .WithConsoleExporter()
);

// Option C: Options-based configuration
builder.Services.AddTylerTelemetry(options =>
{
    options.ServiceName = "MyApplication";
    options.UseOtlpExporter = true;
    options.OtlpEndpoint = "http://localhost:4317";
    options.OtlpUseHttpProtobuf = false;
});

// Option D: Don't configure at all - everything still works!
// No telemetry will be collected, but no errors either

Configuration via appsettings.json (Recommended)

For production deployments, configure telemetry via appsettings.json instead of hardcoding values:

appsettings.json:

{
  "Telemetry": {
    "Enabled": true,
    "ServiceName": "MyApplication",
    "ServiceVersion": "1.0.0",
    "ServiceNamespace": "Production",
    "UseOtlpExporter": true,
    "OtlpUseHttpProtobuf": true,
    "OtlpHttpEndpoint": "http://your-seq-server/ingest/otlp/v1/traces",
    "OtlpHeaders": {
      "X-Seq-ApiKey": "your-api-key-here"
    },
    "EnableHttpInstrumentation": true,
    "EnableSqlInstrumentation": true,
    "EnrichWithSqlCommandText": false,
    "MaxAttributesPerSpan": 128,
    "MaxEventsPerSpan": 128,
    "MaxLinksPerSpan": 128
  }
}

Program.cs:

using Tyler.Utility.DotNet.Telemetry.Configuration;

var builder = WebApplication.CreateBuilder(args);

// Bind configuration to TelemetryOptions
builder.Services.AddTylerTelemetry(options =>
{
    builder.Configuration.GetSection("Telemetry").Bind(options);
});

Example for Jaeger (gRPC):

{
  "Telemetry": {
    "ServiceName": "my-service",
    "UseOtlpExporter": true,
    "OtlpUseHttpProtobuf": false,
    "OtlpEndpoint": "http://jaeger:4317"
  }
}

Example for standard HTTP collector:

{
  "Telemetry": {
    "ServiceName": "my-service",
    "UseOtlpExporter": true,
    "OtlpUseHttpProtobuf": true,
    "OtlpHttpEndpoint": "http://collector:4318/v1/traces"
  }
}

2. Using Activity Sources in Your Code

using System.Diagnostics;
using Tyler.Utility.DotNet.Telemetry;

public class CacheService
{
    // Create ActivitySource for your component
    private static readonly ActivitySource ActivitySource =
        ActivitySourceHelper.Create("Cache", "Memory");

    private readonly ILogger<CacheService> _logger;

    public async Task<T?> GetAsync<T>(string key)
    {
        // Start activity - returns null if telemetry not configured (safe!)
        using var activity = ActivitySource.StartActivitySafe("Cache.Get");

        // Add tags (safe even if activity is null)
        activity.AddTagSafe("cache.key", key);
        activity.AddTagSafe("cache.operation", "get");

        // Create logging scope with trace context
        using var logScope = _logger.BeginScope(
            activity.ToLoggingScope(new Dictionary<string, object>
            {
                ["CacheKey"] = key,
                ["Operation"] = "Get"
            })
        );

        try
        {
            _logger.LogDebug("Retrieving from cache");
            var result = await RetrieveFromCache<T>(key);

            // Record success (safe no-op if activity is null)
            activity.AddTagSafe("cache.hit", result != null);
            activity.SetStatusSafe(ActivityStatusCode.Ok);

            return result;
        }
        catch (Exception ex)
        {
            // Record exception (safe no-op if activity is null)
            activity.RecordExceptionSafe(ex);
            _logger.LogError(ex, "Cache retrieval failed");
            throw;
        }
    }
}

3. Combined Activity + Logging Scope (Simplified)

public async Task<T?> GetAsync<T>(string key)
{
    // Combines activity start and logging scope in one call
    var (activity, logScope) = ActivitySource.StartActivityWithLogging(
        _logger,
        "Cache.Get",
        new Dictionary<string, object>
        {
            ["cache.key"] = key,
            ["cache.operation"] = "get"
        }
    );

    using (activity)
    using (logScope)
    {
        try
        {
            var result = await RetrieveFromCache<T>(key);
            activity.SetStatusSafe(ActivityStatusCode.Ok);
            return result;
        }
        catch (Exception ex)
        {
            activity.RecordExceptionSafe(ex);
            throw;
        }
    }
}

Production Configuration

For production deployments, consider these recommended settings to balance observability with performance and cost:

Recommended Production Setup

services.AddTylerTelemetry(telemetry => telemetry
    .WithServiceName("ProductionService")
    .WithServiceVersion("2.1.0")
    .WithServiceNamespace("Production") // Or auto-detect from env vars

    // Sampling: Use ParentBased with 10% ratio for production
    // This reduces volume by 90% while maintaining trace continuity
    .WithSampling("ParentBased", ratio: 0.1)

    // Batch Export: Optimize for production load
    .WithBatchExport(
        maxQueueSize: 4096,        // Increase for high throughput
        maxBatchSize: 1024,        // Larger batches = fewer exports
        scheduledDelayMs: 10000,   // Export every 10 seconds
        exportTimeoutMs: 30000     // 30 second timeout
    )

    // Compression: Enable to save 70-90% bandwidth
    .WithCompression(enabled: true)

    // OTLP Exporter: Configure for your collector
    .WithOtlpExporter("https://collector.production.com:4317", useHttpProtobuf: false)

    // Instrumentation: Enable what you need
    .WithAspNetCoreInstrumentation(enabled: true)
    .WithHttpInstrumentation(enabled: true)
    .WithSqlInstrumentation(enabled: true, recordCommandText: false) // Never log SQL in production
    .WithEntityFrameworkInstrumentation(enabled: true)
    .WithAwsInstrumentation(enabled: true)

    // Resource Detection: Auto-discover infrastructure details
    .WithEnvironmentDetection(enabled: true) // Reads ASPNETCORE_ENVIRONMENT
    .WithContainerResourceDetection(enabled: true) // Docker/K8s metadata
    .WithHostResourceDetection(enabled: true) // Hostname, OS, CPU

    // Metrics: Enable with 1-minute export interval
    .WithMetrics(enabled: true, exportIntervalMs: 60000)
    .AddMeter("MyApp.*")

    // Propagation: Use W3C for interoperability
    .WithPropagation("W3C")
);

Production Configuration via appsettings.json

appsettings.Production.json:

{
  "Telemetry": {
    "Enabled": true,
    "ServiceName": "my-service",
    "ServiceVersion": "2025.4.3.1",

    "SamplingStrategy": "ParentBased",
    "SamplingRatio": 0.1,

    "MaxQueueSize": 4096,
    "MaxExportBatchSize": 1024,
    "ScheduledDelayMilliseconds": 10000,
    "ExporterTimeoutMilliseconds": 30000,

    "EnableCompression": true,

    "UseOtlpExporter": true,
    "OtlpUseHttpProtobuf": false,
    "OtlpEndpoint": "https://collector.production.com:4317",
    "OtlpHeaders": {
      "Authorization": "Bearer ${TELEMETRY_API_KEY}"
    },

    "EnableAspNetCoreInstrumentation": true,
    "EnableHttpInstrumentation": true,
    "EnableSqlInstrumentation": true,
    "EnrichWithSqlCommandText": false,
    "EnableEntityFrameworkInstrumentation": true,
    "EnableAwsInstrumentation": true,

    "AutoDetectEnvironment": true,
    "EnableContainerResourceDetection": true,
    "EnableHostResourceDetection": true,

    "EnableMetrics": true,
    "MetricExportIntervalMilliseconds": 60000,
    "MeterPatterns": ["MyApp.*", "Tyler.Utility.*"],

    "PropagationFormat": "W3C",
    "ValidateOnStartup": true
  }
}

Sampling Strategies Explained

AlwaysOn - Records every trace (100%)

.WithSampling("AlwaysOn")

• Use when: Development, troubleshooting, low traffic
• Pros: Complete visibility
• Cons: Expensive at scale, overwhelming data volume

AlwaysOff - Records no traces (0%)

.WithSampling("AlwaysOff")

• Use when: Temporarily disabling telemetry, testing
• Pros: Zero overhead
• Cons: No observability

TraceIdRatio - Random sampling based on trace ID

.WithSampling("TraceIdRatio", ratio: 0.1) // 10% of traces

• Use when: Independent service, no upstream/downstream coordination
• Pros: Predictable sampling rate, unbiased
• Cons: Breaks distributed traces (parent/child might sample differently)

ParentBased (Recommended) - Respects parent span decisions

.WithSampling("ParentBased", ratio: 0.1) // Root traces sampled at 10%

• Use when: Production, distributed systems
• Pros: Maintains complete distributed traces, balances cost/visibility
• Cons: Slightly more complex
• How it works:
  • If no parent span: Uses TraceIdRatio with specified ratio
  • If parent sampled: Always samples (maintains trace continuity)
  • If parent not sampled: Never samples

Production Recommendation:

// Start with 10% sampling, adjust based on traffic volume
.WithSampling("ParentBased", ratio: 0.1)

// High-volume services (>1000 req/sec): Use 1-5%
.WithSampling("ParentBased", ratio: 0.01)

// Critical paths or debugging: Temporarily increase to 100%
.WithSampling("ParentBased", ratio: 1.0)

Batch Export Configuration

Fine-tune export behavior for your workload:

// High-throughput service (>1000 req/sec)
.WithBatchExport(
    maxQueueSize: 8192,      // Large queue to handle bursts
    maxBatchSize: 2048,      // Large batches for efficiency
    scheduledDelayMs: 5000,  // Export every 5 seconds
    exportTimeoutMs: 60000   // 60 second timeout
)

// Low-latency service (near real-time)
.WithBatchExport(
    maxQueueSize: 1024,      // Smaller queue
    maxBatchSize: 256,       // Smaller batches = faster exports
    scheduledDelayMs: 2000,  // Export every 2 seconds
    exportTimeoutMs: 15000   // 15 second timeout
)

// Default balanced configuration
.WithBatchExport() // Uses defaults: 2048/512/5000/30000

Compression

Enable gzip compression to reduce bandwidth by 70-90%:

// Always enable compression for production
.WithCompression(enabled: true)

Performance Impact:

• CPU overhead: ~1-2% (negligible)
• Bandwidth savings: 70-90% (significant)
• Latency impact: <5ms per batch (minimal)

When to disable:

• Local development (easier to inspect raw data)
• Collector doesn't support compression
• CPU is severely constrained

Configuration Examples

Console Debugging

services.AddTylerTelemetry(telemetry => telemetry
    .WithServiceName("MyService")
    .WithConsoleExporter()
);

Production with OTLP (Jaeger, Tempo, etc.)

services.AddTylerTelemetry(telemetry => telemetry
    .WithServiceName("ProductionService")
    .WithServiceVersion("2.1.0")
    .WithServiceNamespace("Production")
    .WithOtlpExporter("http://jaeger:4317", useHttpProtobuf: false)
    .WithHttpInstrumentation()
    .WithSqlInstrumentation()
);

Multiple Exporters

services.AddTylerTelemetry(telemetry => telemetry
    .WithServiceName("MyService")
    .WithConsoleExporter()                                     // For local debugging
    .WithOtlpExporter("http://collector:4317", useHttpProtobuf: false) // For production tracing
    .AddActivitySource("MyApp.*")                              // Custom sources
);

HTTP/Protobuf instead of gRPC

services.AddTylerTelemetry(telemetry => telemetry
    .WithServiceName("MyService")
    .WithOtlpExporter("http://collector:4318", useHttpProtobuf: true)
);

With Authentication Headers

services.AddTylerTelemetry(telemetry => telemetry
    .WithServiceName("MyService")
    .WithOtlpExporter("https://api.honeycomb.io:443")
    .AddOtlpHeader("x-honeycomb-team", "your-api-key")
);

Custom Resource Attributes

services.AddTylerTelemetry(telemetry => telemetry
    .WithServiceName("MyService")
    .AddResourceAttribute("deployment.environment", "production")
    .AddResourceAttribute("host.name", Environment.MachineName)
    .AddResourceAttribute("k8s.pod.name", podName)
);

Disable SQL Command Text (Security)

services.AddTylerTelemetry(telemetry => telemetry
    .WithServiceName("MyService")
    .WithSqlInstrumentation(enabled: true, recordCommandText: false) // Safe default
);

Enable Metrics Collection

services.AddTylerTelemetry(telemetry => telemetry
    .WithServiceName("MyService")
    .WithMetrics(enabled: true, exportIntervalMs: 60000) // Export every 60 seconds
    .AddMeter("MyApp.*")               // Listen to MyApp meters
    .AddMeter("Tyler.Utility.*")       // Listen to Tyler Utility meters
    .WithOtlpExporter()                // Metrics use same exporter as traces
);

// In your code, create meters and instruments:
using System.Diagnostics.Metrics;

var meter = new Meter("MyApp.Cache");
var cacheHitCounter = meter.CreateCounter<long>("cache.hits", "hits", "Number of cache hits");
var cacheMissCounter = meter.CreateCounter<long>("cache.misses", "misses", "Number of cache misses");
var cacheLatencyHistogram = meter.CreateHistogram<double>("cache.latency", "ms", "Cache operation latency");

// Record metrics:
cacheHitCounter.Add(1);
cacheMissCounter.Add(1);
cacheLatencyHistogram.Record(15.5);

Enable ASP.NET Core Instrumentation

services.AddTylerTelemetry(telemetry => telemetry
    .WithServiceName("MyWebApi")
    .WithAspNetCoreInstrumentation(enabled: true) // Enabled by default
    .WithOtlpExporter()
);

// Automatically captures:
// - HTTP request/response (method, path, status code)
// - Routing information
// - Middleware execution
// - Exception details

Enable Entity Framework Core Instrumentation

services.AddTylerTelemetry(telemetry => telemetry
    .WithServiceName("MyService")
    .WithEntityFrameworkInstrumentation(enabled: true)
    .WithOtlpExporter()
);

// Automatically captures:
// - Database queries
// - Query execution time
// - Connection information
// - Query parameters (if configured)

Enable gRPC Client Instrumentation

services.AddTylerTelemetry(telemetry => telemetry
    .WithServiceName("MyGrpcClient")
    .WithGrpcClientInstrumentation(enabled: true)
    .WithOtlpExporter()
);

// Automatically captures:
// - gRPC method calls
// - Request/response metadata
// - Status codes
// - Streaming operations

Enable AWS SDK Instrumentation

services.AddTylerTelemetry(telemetry => telemetry
    .WithServiceName("MyAwsService")
    .WithAwsInstrumentation(enabled: true)
    .WithPropagation("AWS") // Use AWS X-Ray propagation
    .WithOtlpExporter()
);

// Automatically captures:
// - S3 operations (GetObject, PutObject, etc.)
// - DynamoDB operations (Query, Scan, PutItem, etc.)
// - SQS operations (SendMessage, ReceiveMessage, etc.)
// - SNS operations (Publish, Subscribe, etc.)
// - Lambda invocations
// - And all other AWS SDK calls

Enable Redis Instrumentation

services.AddTylerTelemetry(telemetry => telemetry
    .WithServiceName("MyService")
    .WithRedisInstrumentation(enabled: true)
    .WithOtlpExporter()
);

// Automatically captures:
// - Redis commands (GET, SET, HGET, etc.)
// - Command execution time
// - Connection information
// - Pipeline operations

Enable All Instrumentation

services.AddTylerTelemetry(telemetry => telemetry
    .WithServiceName("FullyInstrumentedService")
    .WithServiceVersion("1.0.0")

    // Enable all instrumentation types
    .WithAspNetCoreInstrumentation(enabled: true)
    .WithHttpInstrumentation(enabled: true)
    .WithSqlInstrumentation(enabled: true, recordCommandText: false)
    .WithEntityFrameworkInstrumentation(enabled: true)
    .WithGrpcClientInstrumentation(enabled: true)
    .WithAwsInstrumentation(enabled: true)
    .WithRedisInstrumentation(enabled: true)

    .WithOtlpExporter()
);

Environment Auto-Detection

services.AddTylerTelemetry(telemetry => telemetry
    .WithServiceName("MyService")
    .WithEnvironmentDetection(enabled: true) // Enabled by default
    .WithOtlpExporter()
);

// Automatically reads ServiceNamespace from:
// 1. ASPNETCORE_ENVIRONMENT environment variable (ASP.NET Core apps)
// 2. DOTNET_ENVIRONMENT environment variable (.NET apps)
// 3. Falls back to ServiceNamespace property if neither is set

// Example environment variables:
// ASPNETCORE_ENVIRONMENT=Production    -> service.namespace: "Production"
// DOTNET_ENVIRONMENT=Staging          -> service.namespace: "Staging"

Resource Detection (Container & Host)

services.AddTylerTelemetry(telemetry => telemetry
    .WithServiceName("MyService")
    .WithContainerResourceDetection(enabled: true) // Auto-detect Docker/K8s
    .WithHostResourceDetection(enabled: true)      // Auto-detect host info
    .WithOtlpExporter()
);

// Container Resource Detector automatically adds:
// - container.id: Docker container ID
// - container.name: Container name
// - container.image.name: Image name
// - container.image.tag: Image tag

// Host Resource Detector automatically adds:
// - host.name: Machine hostname
// - host.id: Host identifier
// - os.type: Operating system (e.g., "linux", "windows")
// - os.description: OS version details
// - process.pid: Process ID
// - process.executable.name: Executable name
// - process.executable.path: Executable path
// - process.runtime.name: Runtime name (e.g., ".NET")
// - process.runtime.version: Runtime version

Propagation Format Configuration

// W3C Trace Context (Default - Recommended)
services.AddTylerTelemetry(telemetry => telemetry
    .WithServiceName("MyService")
    .WithPropagation("W3C") // Default, most interoperable
    .WithOtlpExporter()
);
// Uses W3C Trace Context headers: traceparent, tracestate

// B3 Propagation (Zipkin)
services.AddTylerTelemetry(telemetry => telemetry
    .WithServiceName("MyService")
    .WithPropagation("B3")
    .WithOtlpExporter()
);
// Uses B3 headers: X-B3-TraceId, X-B3-SpanId, X-B3-Sampled

// Jaeger Propagation
services.AddTylerTelemetry(telemetry => telemetry
    .WithServiceName("MyService")
    .WithPropagation("Jaeger")
    .WithOtlpExporter()
);
// Uses Jaeger headers: uber-trace-id

// AWS X-Ray Propagation
services.AddTylerTelemetry(telemetry => telemetry
    .WithServiceName("MyService")
    .WithPropagation("AWS")
    .WithAwsInstrumentation(enabled: true) // Typically used with AWS services
    .WithOtlpExporter()
);
// Uses AWS X-Ray headers: X-Amzn-Trace-Id

Diagnostic Logging for Troubleshooting

services.AddTylerTelemetry(telemetry => telemetry
    .WithServiceName("MyService")
    .WithDiagnosticLogging(enabled: true) // Enable diagnostic output
    .WithValidation(enabled: true)        // Fail-fast on invalid config
    .WithOtlpExporter("http://localhost:4317")
);

// Diagnostic logging writes to console:
// - Configuration validation results
// - Exporter initialization status
// - Activity source registration
// - Instrumentation setup details
// - Resource attributes detected
// - Any configuration warnings or errors

// Example output:
// [Telemetry] Service Name: MyService
// [Telemetry] Service Version: 1.0.0
// [Telemetry] OTLP Endpoint: http://localhost:4317
// [Telemetry] Sampling Strategy: ParentBased (Ratio: 0.1)
// [Telemetry] Compression: Enabled
// [Telemetry] Instrumentation: HTTP, SQL, ASP.NET Core
// [Telemetry] Resource Attributes: 5 custom attributes
// [Telemetry] Validation: Passed

OTLP Provider Configuration

This section provides provider-specific configuration examples for popular OTLP-compatible telemetry backends.

Seq Configuration

Seq is a centralized log and trace aggregation server that supports OpenTelemetry.

services.AddTylerTelemetry(options =>
{
    options.ServiceName = "MyApplication";
    options.OtlpEndpoint = "http://localhost:5341/ingest/otlp/v1/traces";
    options.OtlpUseHttpProtobuf = true; // Seq requires HTTP/Protobuf
    options.UseOtlpExporter = true;
    // Optional: Add API key
    options.OtlpHeaders = new Dictionary<string, string>
    {
        ["X-Seq-ApiKey"] = "your-api-key"
    };
});

Seq Notes:

• Default Port: 5341 (HTTP)
• Endpoint Path: /ingest/otlp/v1/traces (required)
• Protocol: HTTP/Protobuf (not gRPC)
• API Keys: Optional but recommended for production
• Documentation: https://docs.datalust.co/docs/using-opentelemetry

Jaeger Configuration

Jaeger is an open-source distributed tracing platform.

services.AddTylerTelemetry(options =>
{
    options.ServiceName = "MyApplication";
    options.OtlpEndpoint = "http://localhost:4317"; // gRPC
    options.OtlpUseHttpProtobuf = false; // Use gRPC for Jaeger
    options.UseOtlpExporter = true;
});

// Alternative: HTTP/Protobuf endpoint
services.AddTylerTelemetry(options =>
{
    options.ServiceName = "MyApplication";
    options.OtlpHttpEndpoint = "http://localhost:4318/v1/traces"; // HTTP
    options.OtlpUseHttpProtobuf = true;
    options.UseOtlpExporter = true;
});

Jaeger Notes:

• gRPC Port: 4317 (default)
• HTTP Port: 4318 (default)
• Authentication: Not required for local instances
• UI Port: 16686 (view traces)
• Documentation: https://www.jaegertracing.io/docs/latest/deployment/#otlp

Datadog Configuration

Datadog provides full-stack observability with OTLP support via the Datadog Agent.

services.AddTylerTelemetry(options =>
{
    options.ServiceName = "MyApplication";
    options.OtlpEndpoint = "http://localhost:4317"; // Datadog Agent
    options.UseOtlpExporter = true;
    // API key configured in Datadog Agent or environment
});

Datadog Notes:

• Agent Version: Requires Datadog Agent 7.35+ with OTLP receiver enabled
• Configuration: Set DD_API_KEY environment variable or configure in agent
• Endpoint: OTLP receiver listens on localhost:4317 by default
• Protocol: Supports both gRPC (4317) and HTTP (4318)
• Documentation: https://docs.datadoghq.com/opentelemetry/

Enable OTLP in Datadog Agent (datadog.yaml):

otlp_config:
  receiver:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

Grafana Tempo Configuration

Grafana Tempo is a high-volume distributed tracing backend.

services.AddTylerTelemetry(options =>
{
    options.ServiceName = "MyApplication";
    options.OtlpEndpoint = "http://tempo:4317"; // gRPC
    options.OtlpUseHttpProtobuf = false;
    options.UseOtlpExporter = true;
});

// Alternative: HTTP endpoint
services.AddTylerTelemetry(options =>
{
    options.ServiceName = "MyApplication";
    options.OtlpHttpEndpoint = "http://tempo:4318/v1/traces"; // HTTP
    options.OtlpUseHttpProtobuf = true;
    options.UseOtlpExporter = true;
});

Tempo Notes:

• gRPC Port: 4317 (default)
• HTTP Port: 4318 (default)
• Authentication: Depends on deployment (Grafana Cloud requires API key)
• Visualization: Requires Grafana for trace visualization
• Documentation: https://grafana.com/docs/tempo/latest/configuration/

Honeycomb Configuration

Honeycomb is a full-featured observability platform.

services.AddTylerTelemetry(options =>
{
    options.ServiceName = "MyApplication";
    options.OtlpEndpoint = "https://api.honeycomb.io:443";
    options.UseOtlpExporter = true;
    options.OtlpHeaders = new Dictionary<string, string>
    {
        ["x-honeycomb-team"] = "your-api-key",
        ["x-honeycomb-dataset"] = "your-dataset-name"
    };
});

Honeycomb Notes:

• Endpoint: https://api.honeycomb.io:443
• Authentication: Required via x-honeycomb-team header
• Dataset: Specify dataset with x-honeycomb-dataset header
• Protocol: HTTPS with gRPC
• Documentation: https://docs.honeycomb.io/getting-data-in/opentelemetry/

Generic OTLP Collector

Configuration for a standard OpenTelemetry Collector deployment.

services.AddTylerTelemetry(options =>
{
    options.ServiceName = "MyApplication";
    options.OtlpEndpoint = "http://otel-collector:4317"; // gRPC
    options.UseOtlpExporter = true;

    // Optional: Add custom headers for authentication
    options.OtlpHeaders = new Dictionary<string, string>
    {
        ["Authorization"] = "Bearer your-token"
    };
});

Generic Collector Notes:

• Standard Ports: 4317 (gRPC), 4318 (HTTP)
• Configuration: Depends on collector backend configuration
• Documentation: https://opentelemetry.io/docs/collector/

Azure Monitor / Application Insights

Azure Monitor supports OTLP ingestion (preview feature).

services.AddTylerTelemetry(options =>
{
    options.ServiceName = "MyApplication";
    options.OtlpHttpEndpoint = "https://<region>.in.applicationinsights.azure.com/v1/traces";
    options.OtlpUseHttpProtobuf = true;
    options.UseOtlpExporter = true;
    options.OtlpHeaders = new Dictionary<string, string>
    {
        ["Authorization"] = $"Bearer {instrumentationKey}"
    };
});

Azure Monitor Notes:

• Protocol: HTTP/Protobuf only (gRPC not supported)
• Authentication: Uses Instrumentation Key or connection string
• Endpoint: Region-specific (e.g., eastus, westus2)
• Status: OTLP support is in preview
• Documentation: https://learn.microsoft.com/azure/azure-monitor/app/opentelemetry-enable

AWS X-Ray via OTEL Collector

AWS X-Ray requires the OpenTelemetry Collector with X-Ray exporter.

services.AddTylerTelemetry(options =>
{
    options.ServiceName = "MyApplication";
    options.OtlpEndpoint = "http://localhost:4317"; // OTEL Collector
    options.UseOtlpExporter = true;

    // Enable AWS propagation for X-Ray compatibility
    options.PropagationFormat = "AWS";
});

AWS X-Ray Notes:

• Collector Required: X-Ray does not natively support OTLP; use OTEL Collector with X-Ray exporter
• Propagation: Use AWS propagation format for trace context
• Configuration: Configure OTEL Collector with AWS credentials
• Documentation: https://aws-otel.github.io/docs/components/x-ray-exporter

Example OTEL Collector Config for X-Ray:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

exporters:
  awsxray:
    region: us-east-1

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [awsxray]

API Reference

ActivitySourceHelper

// Create standardized activity source
var source = ActivitySourceHelper.Create("Cache", "Memory");
// Creates: "Tyler.Utility.Cache.Memory"

// Check if enabled
if (ActivitySourceHelper.IsEnabled(source))
{
    // Telemetry is configured
}

TelemetryExtensions

All extension methods are null-safe and work even when telemetry is not configured:

// Start activity (returns null if not configured)
var activity = source.StartActivitySafe("Operation");

// Add tags (no-op if activity is null)
activity.AddTagSafe("key", "value");
activity.AddTagsSafe(new Dictionary<string, object> { ["key"] = "value" });

// Record exception (no-op if activity is null)
activity.RecordExceptionSafe(exception);

// Set status (no-op if activity is null)
activity.SetStatusSafe(ActivityStatusCode.Ok);

// Add event (no-op if activity is null)
activity.AddEventSafe("CacheHit", new Dictionary<string, object> { ["key"] = key });

// Create logging scope with trace context
var scope = activity.ToLoggingScope(additionalProperties);

// Enrich existing scope
activity.EnrichLoggingScope(existingScope);

// Combined activity + logging
var (activity, logScope) = source.StartActivityWithLogging(logger, "Operation", properties);

Fluent Configuration

builder
    // Core Configuration
    .WithEnabled(bool)                              // Enable/disable telemetry
    .WithServiceName(string)                        // Set service name
    .WithServiceVersion(string)                     // Set version
    .WithServiceNamespace(string)                   // Set namespace (env)
    .WithServiceInstanceId(string)                  // Set instance ID

    // Exporters
    .WithConsoleExporter()                          // Add console output
    .WithOtlpExporter(endpoint, useHttp)            // Add OTLP exporter
    .WithOtlpHeaders(headers)                       // Set OTLP headers
    .AddOtlpHeader(key, value)                      // Add single header

    // Sampling
    .WithSampling(strategy, ratio)                  // Configure sampling (AlwaysOn/AlwaysOff/TraceIdRatio/ParentBased)

    // Batch Export
    .WithBatchExport(queueSize, batchSize,          // Configure batch processor
                     scheduledDelayMs, timeoutMs)

    // Compression
    .WithCompression(enabled)                       // Enable gzip compression

    // Metrics
    .WithMetrics(enabled, exportIntervalMs)         // Enable metrics collection
    .AddMeter(pattern)                              // Add meter pattern

    // Trace Instrumentation
    .WithHttpInstrumentation(enabled)               // HTTP client tracing
    .WithSqlInstrumentation(enabled, recordCmd)     // SQL client tracing
    .WithAspNetCoreInstrumentation(enabled)         // ASP.NET Core tracing
    .WithEntityFrameworkInstrumentation(enabled)    // Entity Framework tracing
    .WithGrpcClientInstrumentation(enabled)         // gRPC client tracing
    .WithAwsInstrumentation(enabled)                // AWS SDK tracing
    .WithRedisInstrumentation(enabled)              // Redis tracing

    // Activity Sources
    .AddActivitySource(pattern)                     // Add activity source pattern
    .AddActivitySources(patterns)                   // Add multiple patterns

    // Resource Attributes
    .AddResourceAttribute(key, value)               // Add resource attribute

    // Resource Detection
    .WithEnvironmentDetection(enabled)              // Auto-detect environment
    .WithContainerResourceDetection(enabled)        // Auto-detect container
    .WithHostResourceDetection(enabled)             // Auto-detect host

    // Propagation
    .WithPropagation(format)                        // Configure propagation (W3C/B3/Jaeger/AWS)

    // Span Configuration
    .WithSpanLimits(attrs, events, links)           // Set span limits
    .WithExceptionRecording(enabled)                // Record exceptions

    // Diagnostics
    .WithDiagnosticLogging(enabled)                 // Enable diagnostic logging
    .WithValidation(enabled)                        // Validate configuration at startup

    .Build()                                        // Build options

Semantic Conventions

The package now includes a SemanticConventions class with standard OpenTelemetry attribute names for consistent, interoperable telemetry:

Using Semantic Conventions Constants

using Tyler.Utility.DotNet.Telemetry;

// Service attributes
activity.AddTagSafe(SemanticConventions.Service.Name, "my-service");
activity.AddTagSafe(SemanticConventions.Service.Version, "1.0.0");

// HTTP attributes
activity.AddTagSafe(SemanticConventions.Http.Method, "GET");
activity.AddTagSafe(SemanticConventions.Http.StatusCode, 200);
activity.AddTagSafe(SemanticConventions.Http.Url, url);

// Database attributes
activity.AddTagSafe(SemanticConventions.Database.System, "sqlserver");
activity.AddTagSafe(SemanticConventions.Database.Name, "MyDB");
activity.AddTagSafe(SemanticConventions.Database.Operation, "SELECT");

// Messaging attributes (SQS, SNS, etc.)
activity.AddTagSafe(SemanticConventions.Messaging.System, "sqs");
activity.AddTagSafe(SemanticConventions.Messaging.Destination, "my-queue");
activity.AddTagSafe(SemanticConventions.Messaging.MessageId, messageId);

// Exception attributes
activity.AddTagSafe(SemanticConventions.Exception.Type, ex.GetType().FullName);
activity.AddTagSafe(SemanticConventions.Exception.Message, ex.Message);

Using Helper Methods

The package provides helper methods for common scenarios:

// Service attributes (all at once)
activity.SetServiceAttributes(
    serviceName: "my-application",
    serviceVersion: "2025.4.3.1",
    serviceNamespace: "production",
    serviceInstanceId: Environment.MachineName
);

// HTTP request attributes
activity.SetHttpAttributes(
    method: "POST",
    url: "https://api.example.com/users",
    statusCode: 201
);

// Database operation attributes
activity.SetDatabaseAttributes(
    system: "sqlserver",
    name: "EPSPartitionToolDB",
    operation: "INSERT"
);

Available Semantic Convention Categories

• Service - Service identification (name, version, namespace, instance ID)
• Http - HTTP operations (method, URL, status code, user agent)
• Database - Database operations (system, name, statement, operation)
• Messaging - Message queue operations (system, destination, message ID)
• Exception - Exception details (type, message, stacktrace, escaped)
• Network - Network operations (peer service, hostname, port, transport)
• General - General purpose (user ID, thread ID, code location)

Custom Conventions for Tyler Applications

// Cache Operations
activity.AddTagSafe("cache.operation", "get");  // get, set, remove, clear
activity.AddTagSafe("cache.key", key);
activity.AddTagSafe("cache.hit", true);
activity.AddTagSafe("cache.backend", "memory"); // memory, sqlserver, sqlite

// SQL Operations (in addition to semantic conventions)
activity.AddTagSafe("db.system", "mssql");           // mssql, sqlite
activity.AddTagSafe("db.operation", "query");        // query, execute, stored_procedure
activity.AddTagSafe("db.name", "MyDatabase");
activity.AddTagSafe("db.statement", sanitizedQuery); // Be careful with PII!

// Service Operations
activity.AddTagSafe("service.name", serviceName);
activity.AddTagSafe("service.operation", "start"); // start, stop, restart
activity.AddTagSafe("host.name", hostName);

Baggage (Cross-Service Context Propagation)

New in 2025.4.2.3: Baggage allows you to propagate key-value pairs across service boundaries without explicitly passing them through method parameters or HTTP headers.

What is Baggage?

Baggage is part of the W3C Trace Context specification and automatically propagates across:

• HTTP requests (via baggage header)
• Message queues (SQS, SNS, Kafka)
• gRPC calls
• Any OpenTelemetry-instrumented communication

When to Use Baggage

✅ Good use cases:

• Tenant/Organization ID
• User ID (anonymized/hashed)
• Correlation/Request ID
• Feature flags
• A/B test variants
• Session identifiers

❌ Bad use cases:

• Large payloads (> 1KB total)
• Sensitive data (passwords, tokens, PII)
• Frequently changing data
• Data that doesn't need to propagate

Basic Usage

// Service A: Add baggage to activity
using var activity = ActivitySource.StartActivitySafe("ProcessRequest");
activity.AddBaggageSafe("tenant.id", "acme-corp");
activity.AddBaggageSafe("user.id", "user-12345");
activity.AddBaggageSafe("correlation.id", Guid.NewGuid().ToString());

// Call downstream service...
await CallServiceB();

// Service B: Read baggage (no direct reference to Service A needed!)
var tenantId = Activity.Current?.GetBaggageSafe("tenant.id");
var userId = Activity.Current?.GetBaggageSafe("user.id");

_logger.LogInformation("Processing for tenant {TenantId}, user {UserId}", tenantId, userId);

Multiple Baggage Items

// Add multiple items at once
var baggageItems = new[]
{
    new KeyValuePair<string, string?>("tenant.id", "12345"),
    new KeyValuePair<string, string?>("user.role", "admin"),
    new KeyValuePair<string, string?>("feature.flags", "new-ui,dark-mode")
};

activity.AddBaggageSafe(baggageItems);

// Retrieve all baggage
var allBaggage = activity.GetAllBaggageSafe();
foreach (var item in allBaggage)
{
    Console.WriteLine($"{item.Key}: {item.Value}");
}

Baggage in Background Jobs

public class ProcessOrderJob : IJob
{
    public async Task Execute(IJobExecutionContext context)
    {
        // Background jobs should use useAmbientContext: false
        using var activity = ActivitySource.StartActivitySafe(
            "Job.ProcessOrder",
            useAmbientContext: false
        );

        // Add baggage specific to this job execution
        activity.AddBaggageSafe("job.type", "order-processing");
        activity.AddBaggageSafe("job.priority", context.JobDetail.JobDataMap.GetString("priority"));

        // Baggage will propagate to any downstream calls
        await _orderService.ProcessAsync(orderId);
    }
}

⚠️ Baggage Performance Considerations

• Network Overhead: Baggage adds to every network request (HTTP headers, message attributes)
• Size Limits: Keep total baggage < 1KB (W3C recommendation)
• Count Limits: Use < 10 baggage items per trace
• Sampling: Baggage propagates even for unsampled traces

Baggage vs Tags

Feature	Baggage	Tags
Propagates across services	✅ Yes	❌ No
Network overhead	⚠️ Yes	✅ No
Use for filtering	⚠️ Limited	✅ Yes
Size limit	1KB total	128 per span
Best for	Cross-cutting IDs	Span-specific data

Background Jobs & Independent Traces

New in 2025.4.2.2: The useAmbientContext parameter allows background jobs to create independent traces per execution.

The Problem

By default, background jobs (Quartz.NET, Hangfire, etc.) inherit Activity.Current from the application startup, causing all job executions to share the same TraceId:

// ❌ PROBLEM: All jobs share one trace
public class MyJob : IJob
{
    public async Task Execute(IJobExecutionContext context)
    {
        using var activity = ActivitySource.StartActivitySafe("Job.Execute");
        // TraceId: abc123... (same for every execution!)
    }
}

This makes per-execution debugging impossible in Seq, Jaeger, or other tracing tools.

The Solution

Use useAmbientContext: false to force a new root trace per execution:

// ✅ SOLUTION: Each job gets unique trace
public class MyJob : IJob
{
    public async Task Execute(IJobExecutionContext context)
    {
        using var activity = ActivitySource.StartActivitySafe(
            "Job.Execute",
            useAmbientContext: false  // ← Forces new root trace
        );
        // TraceId: abc123... (unique per execution!)
    }
}

Complete Example

using Tyler.Utility.DotNet.Telemetry;

public class RealmMovementProcessJob : IJob
{
    private static readonly ActivitySource ActivitySource =
        ActivitySourceHelper.Create("Job", "RealmMovement");

    private readonly ILogger<RealmMovementProcessJob> _logger;

    public async Task Execute(IJobExecutionContext context)
    {
        // Create independent trace for this job execution
        using var activity = ActivitySource.StartActivitySafe(
            "Job.RealmMovement.Execute",
            useAmbientContext: false
        );

        // Add job-specific context
        activity
            .AddTagSafe("job.key", context.JobDetail.Key.ToString())
            .AddTagSafe("job.type", nameof(RealmMovementProcessJob))
            .AddTagSafe("job.fire_time", context.FireTimeUtc.ToString("o"));

        // Create logging scope with trace context
        using var logScope = _logger.BeginScope(
            activity.ToLoggingScope(new Dictionary<string, object>
            {
                ["JobKey"] = context.JobDetail.Key.ToString(),
                ["JobType"] = nameof(RealmMovementProcessJob)
            })
        );

        try
        {
            _logger.LogInformation("Job execution started");

            // Your job logic here
            await ProcessRealmMovementAsync();

            activity?.SetStatusSafe(ActivityStatusCode.Ok);
            _logger.LogInformation("Job execution completed successfully");
        }
        catch (Exception ex)
        {
            activity?.RecordExceptionSafe(ex, escaped: true);
            _logger.LogError(ex, "Job execution failed");
            throw;
        }
    }
}

Integration with Existing Logging

The telemetry package works seamlessly with your existing logging scopes:

Before (with logging only):

using var scope = logger.BeginScope(new Dictionary<string, object>
{
    ["Operation"] = "GetCache",
    ["CacheKey"] = key
});

logger.LogInformation("Retrieving from cache");

After (with telemetry + logging):

using var activity = source.StartActivitySafe("Cache.Get");
activity.AddTagSafe("cache.key", key);

using var scope = logger.BeginScope(activity.ToLoggingScope(new Dictionary<string, object>
{
    ["Operation"] = "GetCache",
    ["CacheKey"] = key
}));

logger.LogInformation("Retrieving from cache");
// Log now includes TraceId and SpanId for correlation!

Log Output:

[2025-11-29 10:15:23] [INFO] Retrieving from cache
  Operation: GetCache
  CacheKey: user:123
  TraceId: 4bf92f3577b34da6a3ce929d0e0e4736
  SpanId: 00f067aa0ba902b7

Viewing Traces

Local Development with Jaeger

1. Run Jaeger with Docker:

docker run -d --name jaeger \
  -p 4317:4317 \
  -p 16686:16686 \
  jaegertracing/all-in-one:latest

2. Configure your app:

services.AddTylerTelemetry(t => t
    .WithServiceName("MyApp")
    .WithOtlpExporter("http://localhost:4317", useHttpProtobuf: false)
);

3. View traces at: http://localhost:16686

Production Options

• Jaeger - Open-source distributed tracing
• Grafana Tempo - High-scale distributed tracing
• Azure Monitor - Azure Application Insights
• AWS X-Ray - AWS distributed tracing
• Honeycomb - Observability platform
• Datadog - Full-stack monitoring

Performance Impact

• When telemetry is NOT configured: Zero overhead - all methods are no-ops
• When telemetry IS configured: Minimal overhead (~1-5ms per span)
• Sampling: Production should use ParentBased sampling with 0.01-0.1 ratio
• Batch Export: Larger batches reduce export frequency and network overhead
• Compression: Adds ~1-2% CPU overhead but saves 70-90% bandwidth

Best Practices

Sampling Strategy Recommendations

Development/Staging:

.WithSampling("AlwaysOn") // 100% sampling for complete visibility

Production - Low Traffic (<100 req/sec):

.WithSampling("ParentBased", ratio: 0.1) // 10% sampling

Production - Medium Traffic (100-1000 req/sec):

.WithSampling("ParentBased", ratio: 0.05) // 5% sampling

Production - High Traffic (>1000 req/sec):

.WithSampling("ParentBased", ratio: 0.01) // 1% sampling

Key Principles:

• Always use ParentBased in production to maintain trace continuity
• Start with higher sampling (10%) and reduce if data volume is too high
• Monitor your sampling ratio vs. cost/storage metrics
• Temporarily increase to 100% when debugging specific issues

Instrumentation Selection

Enable selectively based on your application needs:

// Web API Service
.WithAspNetCoreInstrumentation(enabled: true)  // ✅ Required
.WithHttpInstrumentation(enabled: true)        // ✅ For outbound HTTP calls
.WithSqlInstrumentation(enabled: true)         // ✅ If using SQL
.WithEntityFrameworkInstrumentation(false)     // ⚠️ Redundant with SQL instrumentation
.WithGrpcClientInstrumentation(false)          // ❌ Only if using gRPC
.WithAwsInstrumentation(false)                 // ❌ Only if using AWS SDK
.WithRedisInstrumentation(false)               // ❌ Only if using Redis

// Background Job Service
.WithAspNetCoreInstrumentation(enabled: false) // ❌ Not a web service
.WithHttpInstrumentation(enabled: true)        // ✅ For API calls
.WithSqlInstrumentation(enabled: true)         // ✅ For database access
.WithAwsInstrumentation(enabled: true)         // ✅ If using SQS/SNS/S3

Performance Consideration:

• Each instrumentation adds ~1-2ms latency per operation
• Only enable what you actually use
• Entity Framework instrumentation can be expensive; use SQL instrumentation instead if you don't need EF-specific details

Security Best Practices

Never log sensitive data in production:

// ❌ DANGEROUS - Logs SQL queries with parameters
.WithSqlInstrumentation(enabled: true, recordCommandText: true)

// ✅ SAFE - Omits SQL query text
.WithSqlInstrumentation(enabled: true, recordCommandText: false)

// ❌ DANGEROUS - Logs HTTP request/response bodies
.AddResourceAttribute("http.request.body", requestBody)

// ✅ SAFE - Only logs non-sensitive metadata
activity.AddTagSafe("http.request.method", "POST");
activity.AddTagSafe("http.response.status_code", 200);

Review your resource attributes:

// ✅ Safe metadata
.AddResourceAttribute("deployment.environment", "production")
.AddResourceAttribute("service.namespace", "backend")
.AddResourceAttribute("host.name", Environment.MachineName)

// ❌ Never include secrets
// .AddResourceAttribute("database.password", password)
// .AddResourceAttribute("api.key", apiKey)

Sanitize URLs and headers:

// Remove query parameters with sensitive data
var sanitizedUrl = new Uri(url).GetLeftPart(UriPartial.Path);
activity.AddTagSafe("http.url", sanitizedUrl);

// Never log Authorization headers
// activity.AddTagSafe("http.request.header.authorization", authHeader); // ❌ NEVER

Resource Detection Best Practices

Enable in containerized environments:

services.AddTylerTelemetry(telemetry => telemetry
    .WithServiceName("MyService")
    .WithContainerResourceDetection(enabled: true)  // ✅ Docker/Kubernetes
    .WithHostResourceDetection(enabled: true)       // ✅ Adds host metadata
    .WithEnvironmentDetection(enabled: true)        // ✅ Reads env vars
);

Performance considerations:

• Container detection: ~1-5ms startup cost (one-time)
• Host detection: ~10-20ms startup cost (one-time)
• Environment detection: <1ms startup cost (one-time)
• Recommendation: Enable all in production for complete resource context

Propagation Format Selection

W3C (Recommended for most scenarios):

.WithPropagation("W3C") // Default, most interoperable

• ✅ Industry standard
• ✅ Supported by all modern observability platforms
• ✅ Best for polyglot (multi-language) environments

B3 (For Zipkin compatibility):

.WithPropagation("B3")

• Use only if your infrastructure uses Zipkin
• Less common in modern systems

Jaeger (For Jaeger-only environments):

.WithPropagation("Jaeger")

• Use only if exclusively using Jaeger
• W3C is generally better even for Jaeger

AWS (For AWS X-Ray integration):

.WithPropagation("AWS")
.WithAwsInstrumentation(enabled: true)

• Required for AWS X-Ray tracing
• Use when running on AWS and using X-Ray

Metrics Best Practices

Choose appropriate metric types:

var meter = new Meter("MyApp.Service");

// ✅ Counter: For incrementing values (requests, errors, items processed)
var requestCounter = meter.CreateCounter<long>("requests.total");
requestCounter.Add(1);

// ✅ Histogram: For distributions (latency, request size, response time)
var latencyHistogram = meter.CreateHistogram<double>("request.duration", "ms");
latencyHistogram.Record(125.5);

// ✅ UpDownCounter: For values that go up and down (queue depth, active connections)
var activeConnections = meter.CreateUpDownCounter<long>("connections.active");
activeConnections.Add(1);  // Connection opened
activeConnections.Add(-1); // Connection closed

// ✅ ObservableGauge: For current state snapshots (memory usage, CPU %)
var memoryGauge = meter.CreateObservableGauge("memory.usage", () => GC.GetTotalMemory(false));

Metric naming conventions:

// ✅ Good metric names (use dots, lowercase, descriptive)
"http.server.requests.total"
"cache.hits.total"
"db.query.duration"
"queue.messages.pending"

// ❌ Bad metric names (avoid spaces, uppercase, vague names)
"RequestCount"
"Cache Hits"
"time"
"value"

Export interval recommendations:

// Development: Fast feedback
.WithMetrics(enabled: true, exportIntervalMs: 10000) // 10 seconds

// Production: Balance freshness vs. overhead
.WithMetrics(enabled: true, exportIntervalMs: 60000) // 60 seconds (recommended)

// High-frequency metrics: For critical services
.WithMetrics(enabled: true, exportIntervalMs: 30000) // 30 seconds

Diagnostic Logging

Enable during initial setup and troubleshooting:

// Development: Always enable
services.AddTylerTelemetry(telemetry => telemetry
    .WithServiceName("MyService")
    .WithDiagnosticLogging(enabled: true)
    .WithValidation(enabled: true)
);

// Production: Disable after telemetry is working
services.AddTylerTelemetry(telemetry => telemetry
    .WithServiceName("MyService")
    .WithDiagnosticLogging(enabled: false) // Reduce console noise
    .WithValidation(enabled: true)         // Keep validation enabled
);

When to enable diagnostic logging:

• ✅ First-time telemetry setup
• ✅ Debugging missing spans or traces
• ✅ Investigating export failures
• ✅ Validating configuration changes
• ❌ Long-term in production (creates excessive console output)

Activity Naming Best Practices

Use descriptive, hierarchical names:

// ✅ Good activity names (action.object or component.operation)
ActivitySource.StartActivitySafe("Cache.Get");
ActivitySource.StartActivitySafe("Database.Query");
ActivitySource.StartActivitySafe("Queue.Send");
ActivitySource.StartActivitySafe("Service.ProcessOrder");

// ❌ Bad activity names (too vague or too specific)
ActivitySource.StartActivitySafe("Operation");
ActivitySource.StartActivitySafe("DoWork");
ActivitySource.StartActivitySafe("GetUserFromDatabaseById123"); // Too specific

Follow semantic conventions:

// HTTP operations
activity.SetHttpAttributes(method: "POST", url: "/api/users", statusCode: 201);

// Database operations
activity.SetDatabaseAttributes(system: "sqlserver", name: "MyDB", operation: "SELECT");

// Cache operations (custom convention)
activity.AddTagSafe("cache.operation", "get");
activity.AddTagSafe("cache.key", cacheKey);
activity.AddTagSafe("cache.hit", wasHit);

General Best Practices

1. Always use *Safe extension methods - They're null-safe and won't fail
2. Create ActivitySource per component - Use ActivitySourceHelper.Create()
3. Include trace context in logs - Use ToLoggingScope() or EnrichLoggingScope()
4. Follow semantic conventions - Use SemanticConventions constants for standard attributes
5. Never log sensitive data - Be careful with SQL commands, API keys, PII, passwords
6. Use appropriate ActivityKind - Client for outgoing, Server for incoming, Internal for in-process
7. Record exceptions - Use RecordExceptionSafe() for error tracking
8. Set status codes - Use SetStatusSafe() to mark success/failure
9. Enable sampling in production - Use ParentBased with 1-10% ratio to control costs
10. Enable compression - Saves 70-90% bandwidth with minimal CPU overhead
11. Validate configuration at startup - Use WithValidation(true) to fail fast on misconfiguration
12. Use resource detectors - Enable container and host detection for rich context

Troubleshooting

No traces appearing?

1. Check telemetry is enabled:

   .WithEnabled(true)
   

2. Verify activity source patterns match:

   .AddActivitySource("Tyler.Utility.*")
   .AddActivitySource("MyApp.*") // Add your patterns
   

3. Confirm exporter is configured:

   .WithConsoleExporter() // For debugging
   // OR
   .WithOtlpExporter("http://localhost:4317")
   

4. Check sampling isn't excluding traces:

   .WithSampling("AlwaysOn") // Temporarily use 100% sampling
   

5. Enable diagnostic logging:

   .WithDiagnosticLogging(enabled: true)
   

   Review console output for configuration issues.

6. Validate OTLP endpoint is reachable:

   curl http://localhost:4317  # gRPC endpoint
   curl http://localhost:4318  # HTTP endpoint
   

Activity is always null?

Possible causes:

• Telemetry is not configured (this is fine! - null-safe design)
• No exporters are configured
• Activity source pattern doesn't match your source name
• Sampling excluded the trace (check SamplingStrategy and SamplingRatio)

Solutions:

// 1. Ensure exporter is configured
.WithConsoleExporter()

// 2. Verify pattern matches
var source = ActivitySourceHelper.Create("Cache", "Memory");
// Creates: "Tyler.Utility.Cache.Memory"
// Pattern must match: .AddActivitySource("Tyler.Utility.*")

// 3. Check if enabled
if (ActivitySourceHelper.IsEnabled(source))
{
    // Telemetry is working
}

Spans not linked (broken distributed traces)?

Possible causes:

• Not using using statements to properly dispose activities
• Parent activity context not propagated across boundaries
• Propagation format mismatch between services

Solutions:

// 1. Always use 'using' for activity disposal
using var activity = source.StartActivitySafe("Operation");

// 2. Ensure propagation format matches across services
.WithPropagation("W3C") // Use same format everywhere

// 3. For HTTP calls, verify traceparent header exists
var request = new HttpRequestMessage();
// OpenTelemetry automatically adds traceparent header

// 4. For background jobs, use useAmbientContext: false
using var activity = source.StartActivitySafe("Job.Execute", useAmbientContext: false);

Metrics not appearing?

1. Check metrics are enabled:

   .WithMetrics(enabled: true)
   

2. Verify meter patterns match:

   .AddMeter("MyApp.*")
   
   // In your code:
   var meter = new Meter("MyApp.Service"); // Must match pattern
   

3. Confirm metric export interval:

   .WithMetrics(enabled: true, exportIntervalMs: 60000) // 60 seconds
   

   Metrics are exported on interval, not immediately.

4. Check OTLP endpoint supports metrics: Some collectors require separate endpoints for traces vs. metrics.

High memory usage?

Possible causes:

• Queue size too large
• Exporter timeout too long
• No sampling (100% of traces retained)

Solutions:

// 1. Reduce queue size
.WithBatchExport(
    maxQueueSize: 1024,  // Reduce from default 2048
    maxBatchSize: 256    // Reduce from default 512
)

// 2. Enable sampling
.WithSampling("ParentBased", ratio: 0.1) // Only keep 10%

// 3. Reduce export delay
.WithBatchExport(scheduledDelayMs: 2000) // Export more frequently

Export failures / timeouts?

1. Check exporter timeout:

   .WithBatchExport(exportTimeoutMs: 30000) // 30 seconds
   

2. Verify endpoint URL format:

   // ✅ Correct formats
   "http://localhost:4317"           // gRPC
   "https://collector:4317"          // gRPC with TLS
   "http://localhost:4318/v1/traces" // HTTP/Protobuf
   
   // ❌ Incorrect formats
   "localhost:4317"                  // Missing http://
   "http://localhost:4317/v1/traces" // Path not needed for gRPC
   

3. Enable diagnostic logging:

   .WithDiagnosticLogging(enabled: true)
   

4. Check compression compatibility:

   .WithCompression(enabled: false) // Disable if collector doesn't support it
   

Environment not detected?

Check environment variables:

# Linux/Mac
export ASPNETCORE_ENVIRONMENT=Production
export DOTNET_ENVIRONMENT=Production

# Windows
set ASPNETCORE_ENVIRONMENT=Production
set DOTNET_ENVIRONMENT=Production

# Verify
echo $ASPNETCORE_ENVIRONMENT  # Linux/Mac
echo %ASPNETCORE_ENVIRONMENT% # Windows

Or explicitly set namespace:

.WithServiceNamespace("Production")
.WithEnvironmentDetection(enabled: false) // Disable auto-detection

Container metadata missing?

1. Verify running in container:

   # Check if container ID file exists
   cat /proc/self/cgroup  # Linux containers
   

2. Enable container detection:

   .WithContainerResourceDetection(enabled: true)
   

3. Check resource attributes in exported spans: Look for container.id, container.name, container.image.name

Validation errors at startup?

Common validation failures:

// ❌ Invalid sampling ratio
.WithSampling("TraceIdRatio", ratio: 1.5) // Must be 0.0-1.0

// ❌ Invalid URL
.WithOtlpExporter("localhost:4317") // Missing http://

// ❌ Batch size exceeds queue size
.WithBatchExport(maxQueueSize: 512, maxBatchSize: 1024) // Batch > Queue

// ❌ Invalid export interval
.WithMetrics(enabled: true, exportIntervalMs: -1000) // Must be positive

Solution: Enable validation to catch issues early:

.WithValidation(enabled: true) // Enabled by default

Package Dependencies

Core Dependencies

• Microsoft.Extensions.DependencyInjection.Abstractions 10.0.5
• Microsoft.Extensions.Logging.Abstractions 10.0.5
• Microsoft.Extensions.Options 10.0.5

OpenTelemetry Core (Stable)

• OpenTelemetry 1.15.2
• OpenTelemetry.Api 1.15.2
• OpenTelemetry.Extensions.Hosting 1.15.2
• OpenTelemetry.Extensions.Propagators 1.15.2

Exporters (Stable)

• OpenTelemetry.Exporter.Console 1.15.2
• OpenTelemetry.Exporter.OpenTelemetryProtocol 1.15.2

Instrumentation - Stable

• OpenTelemetry.Instrumentation.Http 1.15.0
• OpenTelemetry.Instrumentation.AspNetCore 1.15.1
• OpenTelemetry.Instrumentation.SqlClient 1.15.1
• OpenTelemetry.Instrumentation.AWS 1.15.0

Instrumentation - Beta

• OpenTelemetry.Instrumentation.EntityFrameworkCore 1.15.0-beta.1
• OpenTelemetry.Instrumentation.GrpcNetClient 1.15.0-beta.1
• OpenTelemetry.Instrumentation.StackExchangeRedis 1.15.0-beta.1

Resource Detectors (Beta)

• OpenTelemetry.Resources.Container 1.15.0-beta.1
• OpenTelemetry.Resources.Host 1.15.0-beta.2

Other

• System.Drawing.Common 10.0.5

Note: Some packages are in beta stage. They are production-ready but may have API changes in future releases. The core tracing functionality uses stable OpenTelemetry 1.15.2 packages.

Related Packages

• Davasorus.Utility.DotNet.Cache - Caching utilities
• Davasorus.Utility.DotNet.SQL - SQL utilities
• Davasorus.Utility.DotNet.Api - API utilities
• Davasorus.Utility.DotNet.Services - Windows Services utilities

License

Copyright © Tyler Technologies. All rights reserved.

Support

For issues, questions, or contributions, please open an issue in the repository or contact your development team.