Andy.Llm 2026.10.17-rc.91

Andy.Llm

⚠️ ALPHA SOFTWARE: This library is in active development and APIs may change. Not recommended for production use without thorough testing.

A flexible, provider-agnostic .NET library for integrating with Large Language Models (LLMs) through OpenAI-compatible APIs.

Design Goals

1. Provider Agnostic: Support multiple LLM providers through a unified interface
2. OpenAI Compatibility: First-class support for OpenAI-compatible APIs
3. Extensibility: Easy to add new providers without modifying core logic
4. Type Safety: Strongly-typed models and interfaces
5. Modern .NET: Built on .NET 10.0 with latest C# features
6. Production Ready: Comprehensive error handling, logging, and testing
7. Resilient: Built-in retry policies and circuit breakers with Polly
8. Observable: Telemetry, metrics, and distributed tracing support
9. Secure: API key protection and sensitive data sanitization

Quick Start

Installation

dotnet add package Andy.Llm

Basic Usage

using Andy.Model.Llm;
using Andy.Model.Model;
using Andy.Llm.Extensions;
using Andy.Llm.Providers;
using Microsoft.Extensions.DependencyInjection;

// Register the LLM services, then configure providers from environment variables
// (for example, OPENAI_API_KEY and OPENAI_MODEL).
var services = new ServiceCollection();
services.AddLlmServices(_ => { });
services.ConfigureLlmFromEnvironment();
var serviceProvider = services.BuildServiceProvider();

// Resolve the provider factory and pick the first available provider
var factory = serviceProvider.GetRequiredService<ILlmProviderFactory>();
var provider = await factory.CreateAvailableProviderAsync();

// Send a message and get a response
var request = new LlmRequest
{
    Messages = new List<Message>
    {
        new Message { Role = Role.User, Content = "Hello, how are you?" }
    },
    Config = new LlmClientConfig { Model = "gpt-4o-mini" }
};

var response = await provider.CompleteAsync(request);
Console.WriteLine(response.Content);

Structured Output with JSON Schema

Note: First-class schema enforcement on the request (dedicated ResponseFormat, JsonSchema, and StrictMode fields) is not yet implemented — it is tracked as a TODO in examples/StructuredOutput. Today you request JSON by instructing the model through a system message and then parse the response with the parsing utilities (see Hybrid Parsing below).

using Andy.Model.Llm;
using Andy.Model.Model;

// Describe the desired JSON shape in a system instruction
var schema = @"{
    ""type"": ""object"",
    ""properties"": {
        ""name"": { ""type"": ""string"" },
        ""age"": { ""type"": ""integer"" },
        ""skills"": { ""type"": ""array"", ""items"": { ""type"": ""string"" } }
    },
    ""required"": [""name"", ""age""]
}";

var request = new LlmRequest
{
    Messages = new List<Message>
    {
        new Message
        {
            Role = Role.System,
            Content = "Respond with valid JSON matching this schema:\n" + schema
        },
        new Message { Role = Role.User, Content = "Generate a person profile" }
    },
    Config = new LlmClientConfig { Model = "gpt-4o-mini" }
};

var response = await provider.CompleteAsync(request);
// response.Content contains the JSON produced by the model

Hybrid Parsing for Any Response Format

using Andy.Llm.Parsing;
using Andy.Llm.Parsing.Ast;

// Create a hybrid parser that handles both structured and text responses
var structuredFactory = new StructuredResponseFactory(logger);
var hybridParser = new HybridLlmParser(textParser, structuredFactory, logger);

// Parse any LLM response - OpenAI, Anthropic, or plain text
var ast = hybridParser.Parse(llmResponse);

// Work with the semantic AST
foreach (var node in ast.Children)
{
    switch (node)
    {
        case ToolCallNode toolCall:
            // Handle tool/function calls with structured arguments
            await ExecuteTool(toolCall.ToolName, toolCall.Arguments);
            break;
        case TextNode text:
            // Handle text content
            Console.WriteLine(text.Content);
            break;
        case ErrorNode error:
            // Handle parsing errors gracefully
            logger.LogWarning("Parse error: {Message}", error.Message);
            break;
    }
}

Advanced Usage with Dependency Injection

using Andy.Llm.Extensions;
using Andy.Llm.Providers;
using Microsoft.Extensions.DependencyInjection;

var services = new ServiceCollection();

// Register services and configure providers with human-readable names.
services.AddLlmServices(options =>
{
    options.DefaultProvider = "openai/latest-small";
    options.Providers["openai/latest-small"] = new ProviderConfig
    {
        Provider = "openai",
        ApiKey = "your-api-key",
        Model = "gpt-4o-mini",
        ApiBase = "https://api.openai.com/v1"
    };
    options.Providers["openai/latest-large"] = new ProviderConfig
    {
        Provider = "openai",
        ApiKey = "your-api-key",
        Model = "gpt-5",
        ApiBase = "https://api.openai.com/v1",
        Enabled = false
    };
});

// Fill missing or placeholder values from environment variables.
services.ConfigureLlmFromEnvironment();

var serviceProvider = services.BuildServiceProvider();
var factory = serviceProvider.GetRequiredService<ILlmProviderFactory>();
var provider = factory.CreateProvider("openai/latest-small");

Configuration

Configuration Names

The library supports human-readable hierarchical configuration names for better organization:

{
  "Llm": {
    "DefaultProvider": "openai/latest-small",
    "Providers": {
      "openai/latest-large": {
        "Provider": "openai",
        "ApiBase": "https://api.openai.com/v1",
        "ApiKey": "${OPENAI_API_KEY}",
        "Model": "gpt-5",
        "Enabled": false
      },
      "openai/latest-small": {
        "Provider": "openai",
        "ApiBase": "https://api.openai.com/v1",
        "ApiKey": "${OPENAI_API_KEY}",
        "Model": "gpt-4o-mini",
        "Enabled": true
      },
      "openai/large-code": {
        "Provider": "openai",
        "ApiBase": "https://api.openai.com/v1",
        "ApiKey": "${OPENAI_API_KEY}",
        "Model": "gpt-5-codex",
        "Enabled": false
      },
      "cerebras/large-code": {
        "Provider": "cerebras",
        "ApiBase": "https://api.cerebras.ai/v1",
        "ApiKey": "${CEREBRAS_API_KEY}",
        "Model": "qwen-3-coder-480b",
        "Enabled": false
      }
    }
  }
}

Configuration names can use any format. The Provider property specifies the underlying provider type (openai, cerebras, azure, ollama, anthropic, openrouter). If omitted, the provider type is inferred from the configuration name prefix (e.g., "openai/latest-large" infers "openai").

OpenRouter note: OpenRouter model ids contain a slash (e.g. anthropic/claude-sonnet-4.6). Always put that id in the Model field and keep Provider set to "openrouter" — never encode the model id in the config key, since keys are split on / for routing. See docs/openrouter-provider.md.

Provider Priority

The library supports priority-based provider selection using the Priority field. When using CreateAvailableProviderAsync(), providers are selected in the following order:

1. Default Provider (if fully configured and enabled)
2. Providers with Priority (ordered from highest to lowest)
3. Remaining Enabled Providers (in dictionary order)

A provider is considered "fully configured" when all required fields are present and non-empty:

• OpenAI/Cerebras: ApiKey, ApiBase, and Model must be set
• Azure: ApiKey, ApiBase, and DeploymentName must be set
• Ollama: ApiBase and Model must be set (no API key required)

Example with priorities:

{
  "Llm": {
    "DefaultProvider": "openai/latest-large",
    "Providers": {
      "openai/latest-large": {
        "Provider": "openai",
        "ApiBase": "https://api.openai.com/v1",
        "ApiKey": "${OPENAI_API_KEY}",
        "Model": "gpt-4o",
        "Enabled": true,
        "Priority": 100
      },
      "cerebras/fast-large": {
        "Provider": "cerebras",
        "ApiBase": "https://api.cerebras.ai/v1",
        "ApiKey": "${CEREBRAS_API_KEY}",
        "Model": "llama-3.3-70b",
        "Enabled": true,
        "Priority": 90
      },
      "openai/latest-small": {
        "Provider": "openai",
        "ApiBase": "https://api.openai.com/v1",
        "ApiKey": "${OPENAI_API_KEY}",
        "Model": "gpt-4o-mini",
        "Enabled": true,
        "Priority": 80
      }
    }
  }
}

In this example:

• If OPENAI_API_KEY is set, openai/latest-large will be used (default provider, Priority 100)
• If OPENAI_API_KEY is not set but CEREBRAS_API_KEY is set, cerebras/fast-large will be used (Priority 90)
• If only openai/latest-small is configured, it will be used (Priority 80)

Providers without the Priority field have no explicit priority and are selected after all prioritized providers.

Backward Compatibility

Simple configuration names without hierarchical structure remain supported:

services.AddLlmServices(options =>
{
    options.DefaultProvider = "openai";
    options.Providers["openai"] = new ProviderConfig
    {
        ApiKey = "your-api-key",
        Model = "gpt-4o-mini"
    };
});

Environment Variables

The library supports configuration through environment variables for all major providers:

OpenAI

• OPENAI_API_KEY - Your OpenAI API key
• OPENAI_API_BASE - API base URL (required)
• OPENAI_MODEL - Model to use (required)
• OPENAI_ORGANIZATION - Organization ID (optional)

Cerebras

• CEREBRAS_API_KEY - Your Cerebras API key
• CEREBRAS_API_BASE - API base URL (required)
• CEREBRAS_MODEL - Model to use (required)

Azure OpenAI

• AZURE_OPENAI_ENDPOINT - Your Azure OpenAI endpoint
• AZURE_OPENAI_KEY - Your Azure OpenAI key
• AZURE_OPENAI_DEPLOYMENT - Your deployment name
• AZURE_OPENAI_API_VERSION - API version (default: 2024-02-15-preview)

OpenRouter

• OPENROUTER_API_KEY - Your OpenRouter API key (from https://openrouter.ai/keys)
• OPENROUTER_API_BASE - API base URL (optional; defaults to https://openrouter.ai/api/v1)
• OPENROUTER_MODEL - Default provider/model slug (e.g. openai/gpt-oss-20b:free, anthropic/claude-sonnet-4.6)
• OPENROUTER_HTTP_REFERER - Optional HTTP-Referer header for OpenRouter's app leaderboard
• OPENROUTER_X_TITLE - Optional X-Title header for OpenRouter's app leaderboard

OpenRouter exposes free, tool-capable models (e.g. openai/gpt-oss-20b:free) — great for trying the library at zero cost. See docs/openrouter-provider.md.

Non-streaming completions tolerate leading OpenRouter keep-alive comments, including when using an OpenAI-compatible configuration. Empty or comment-only responses throw LlmProviderException with a synthetic Error.StatusCode of 504, so callers can apply their retry policy. The library does not automatically resubmit these requests.

Local/Ollama

• OLLAMA_API_BASE - Your local endpoint (required, e.g., http://localhost:11434)
• OLLAMA_MODEL - Model to use (required, e.g., llama2)

Features

Core Capabilities

• Multi-Provider Support: OpenAI, Cerebras, Azure OpenAI, Ollama, Anthropic, OpenRouter (unified gateway to 100s of models)
• Streaming Responses: Real-time token streaming
• Function/Tool Calling: OpenAI-compatible function calling with structured outputs
• Structured Outputs: JSON Schema validation and type-safe responses
• Hybrid Parsing: Automatic detection and parsing of structured vs text responses
• AST Generation: Semantic Abstract Syntax Tree for all LLM responses
• Conversation Management: Context and token limit management
• Dependency Injection: Full DI container integration

Enterprise Features

• Security: Secure API key storage with SecureString, sensitive data sanitization
• Resilience: Retry policies, circuit breakers, timeout handling via Polly
• Observability: Metrics collection, distributed tracing, structured logging (see Telemetry Guide)
• Progress Reporting: Real-time progress updates for long operations
• Cancellation Support: Comprehensive cancellation token support

Streaming Responses

var request = new LlmRequest
{
    Messages = new List<Message>
    {
        new Message { Role = Role.User, Content = "Write a story" }
    },
    Config = new LlmClientConfig { Model = "gpt-4o-mini" }
};

// Streaming is exposed through the provider's StreamCompleteAsync method
await foreach (var chunk in provider.StreamCompleteAsync(request))
{
    if (!string.IsNullOrEmpty(chunk.TextDelta))
    {
        Console.Write(chunk.TextDelta);
    }

    if (chunk.IsComplete)
    {
        Console.WriteLine($"\n[Finished: {chunk.FinishReason}]");
    }
}

Tool Calling with Structured Outputs

using Andy.Model.Llm;
using Andy.Model.Model;
using Andy.Model.Tooling;

// Define tools with JSON schemas (parameters use a JSON-schema dictionary)
var tools = new List<ToolDeclaration>
{
    new ToolDeclaration
    {
        Name = "get_weather",
        Description = "Get weather for a location",
        Parameters = new Dictionary<string, object>
        {
            ["type"] = "object",
            ["properties"] = new Dictionary<string, object>
            {
                ["location"] = new Dictionary<string, object> { ["type"] = "string" },
                ["units"] = new Dictionary<string, object>
                {
                    ["type"] = "string",
                    ["enum"] = new[] { "celsius", "fahrenheit" }
                }
            },
            ["required"] = new[] { "location" }
        }
    }
};

// Request with tool calling
var request = new LlmRequest
{
    Messages = messages,
    Tools = tools,
    Config = new LlmClientConfig { Model = "gpt-4o-mini" }
};

// The model decides when to call a tool; inspect the returned tool calls
var response = await provider.CompleteAsync(request);

// Execute any tool calls the model requested
foreach (var toolCall in response.ToolCalls)
{
    var result = await ExecuteTool(toolCall.Name, toolCall.ArgumentsJson);
    // Send result back to the model for a final response
}

Function/Tool Calling

using Andy.Model.Llm;
using Andy.Model.Model;
using Andy.Model.Tooling;

// Define available tools
var weatherTool = new ToolDeclaration
{
    Name = "get_weather",
    Description = "Get current weather for a location",
    Parameters = new Dictionary<string, object>
    {
        ["type"] = "object",
        ["properties"] = new Dictionary<string, object>
        {
            ["location"] = new Dictionary<string, object>
            {
                ["type"] = "string",
                ["description"] = "City and state"
            }
        },
        ["required"] = new[] { "location" }
    }
};

// Build the conversation and request
var messages = new List<Message>
{
    new Message { Role = Role.User, Content = "What's the weather in New York?" }
};

var request = new LlmRequest
{
    Messages = messages,
    Tools = new List<ToolDeclaration> { weatherTool },
    Config = new LlmClientConfig { Model = "gpt-4o-mini" }
};

var response = await provider.CompleteAsync(request);

// Handle tool calls
if (response.ToolCalls != null && response.ToolCalls.Any())
{
    // Record the assistant turn that requested the tools
    messages.Add(new Message
    {
        Role = Role.Assistant,
        Content = response.Content ?? string.Empty,
        ToolCalls = response.ToolCalls
    });

    foreach (var call in response.ToolCalls)
    {
        // Execute the tool and append the result as a tool message
        var result = ExecuteTool(call.Name, call.ArgumentsJson);
        messages.Add(new Message
        {
            Role = Role.Tool,
            Content = result,
            ToolResults = new List<ToolResult>
            {
                new ToolResult
                {
                    CallId = call.Id,
                    Name = call.Name,
                    ResultJson = result,
                    IsError = false
                }
            }
        });
    }
}

Recent API Additions

• LlmResponse.ToolCalls / LlmResponse.FunctionCalls: tool and function calls requested by the model.
• FunctionCall.ArgumentsJson: preserves raw JSON arguments from providers.
• LlmStreamResponse.FinishReason: reason for stream completion on final chunk.

Streaming now emits partial function-call deltas when providers stream arguments, and marks the final chunk with IsComplete = true and FinishReason.

See docs/implementation.md for implementation details and current status.

Conversation Management

// Track the conversation as a list of messages, starting with a system instruction
var messages = new List<Message>
{
    new Message { Role = Role.System, Content = "You are a helpful assistant." }
};

// Build conversation
messages.Add(new Message { Role = Role.User, Content = "Hello!" });
messages.Add(new Message { Role = Role.Assistant, Content = "Hi! How can I help you?" });
messages.Add(new Message { Role = Role.User, Content = "Tell me about AI" });

// Trim older turns yourself to stay within the model's context window
var request = new LlmRequest
{
    Messages = messages,
    Config = new LlmClientConfig { Model = "gpt-4o-mini" }
};

Multi-Provider Support

var factory = serviceProvider.GetRequiredService<ILlmProviderFactory>();

// Use specific configuration by name
var smallModel = factory.CreateProvider("openai/latest-small");
var largeModel = factory.CreateProvider("openai/latest-large");
var codeModel = factory.CreateProvider("cerebras/large-code");

// Backward compatible: simple provider names still work
var openAiProvider = factory.CreateProvider("openai");
var cerebrasProvider = factory.CreateProvider("cerebras");

// Get first available provider based on priority and completeness
var provider = await factory.CreateAvailableProviderAsync();

Architecture

See ARCHITECTURE.md for detailed architecture documentation.

Telemetry and Monitoring

Andy.Llm provides comprehensive telemetry through OpenTelemetry-compatible APIs. See the Telemetry Guide for:

• Metrics collection and export
• Distributed tracing setup
• Integration with Prometheus, Jaeger, Application Insights
• Terminal application monitoring

Testing

The library includes comprehensive unit and integration tests:

# Run all tests
dotnet test

# Run with coverage
dotnet test --collect:"XPlat Code Coverage"

# Run only unit tests
dotnet test --filter "Category!=Integration"

📖 Documentation

• Getting Started Guide - Quick setup and basic usage
• Architecture Documentation - System design and components
• API Reference - Complete API documentation
• Telemetry Guide - Monitoring and observability setup
• Examples - Code examples and patterns

Examples

The examples directory contains complete, runnable projects demonstrating various features:

Core Examples

• SimpleCompletion - Basic text completion with multiple providers
• ConversationChat - Interactive chat with conversation context management
• FunctionCalling - OpenAI-compatible tool/function calling with weather and calculator examples

Provider-Specific Examples

• AzureOpenAI - Enterprise deployment with Azure OpenAI Service
• Ollama - Local LLM execution with complete privacy

Advanced Examples

• Streaming - Real-time streaming responses with cancellation and progress tracking
• MultiProvider - Comparing responses from multiple LLM providers simultaneously
• Telemetry - Metrics collection, distributed tracing, and progress reporting
• StructuredOutput - JSON Schema validation and structured responses
• HybridParsing - Automatic detection and parsing of different response formats
• ToolCallingStructured - Advanced tool calling with schema validation

Run any example with:

dotnet run --project examples/SimpleCompletion
# Set provider with environment variable
LLM_PROVIDER=cerebras dotnet run --project examples/ConversationChat

Response Parsing and AST

Andy.Llm includes a sophisticated parsing system that creates a semantic Abstract Syntax Tree (AST) from any LLM response:

AST Node Types

• ResponseNode: Root node with metadata (provider, model, tokens)
• TextNode: Plain or formatted text content
• ToolCallNode: Function/tool invocations with parsed arguments
• ToolResultNode: Results from tool execution
• CodeNode: Code blocks with language detection
• ErrorNode: Parsing or validation errors
• FileReferenceNode: File paths and references
• MarkdownNode: Structured markdown elements

Visitor Pattern Support

public class CustomVisitor : IAstVisitor<string>
{
    public string VisitToolCall(ToolCallNode node)
    {
        // Process tool calls
        return $"Tool: {node.ToolName}";
    }
    
    public string VisitText(TextNode node)
    {
        // Process text content
        return node.Content;
    }
    // ... other visit methods
}

// Apply visitor to AST
var ast = parser.Parse(response);
var result = ast.Accept(new CustomVisitor());

Provider Detection

The hybrid parser automatically detects response formats:

• OpenAI format (choices, tool_calls)
• Anthropic format (content blocks, tool_use)
• Server-Sent Events (SSE)
• JSON Lines (JSONL)
• Plain text with embedded JSON

Error Handling

All parsing errors are captured in the AST:

foreach (var node in ast.Children.OfType<ToolCallNode>())
{
    if (node.ParseError != null)
    {
        // Handle malformed arguments
        logger.LogWarning("Tool arguments invalid: {Error}", 
            node.ParseError.Message);
        // Access raw JSON for manual processing
        var raw = node.Metadata["RawArgumentsJson"];
    }
}

Contributing

Contributions are welcome! Please read our Contributing Guide for details.

License

Copyright 2025 Rivoli AI

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

See the LICENSE file for the full license text.

Third-Party Licenses

This project uses the following open-source libraries:

• OpenAI SDK - MIT License
• Cerebras.Cloud.SDK - Apache-2.0 License
• Microsoft.Extensions.Http.Polly - MIT License
• Microsoft.Extensions.Logging - MIT License
• System.Diagnostics.DiagnosticSource - MIT License
• Andy.Configuration - Apache-2.0 License

All dependencies are compatible with the Apache-2.0 license.

Security

• API keys are stored securely using SecureString
• Sensitive data is automatically sanitized in logs
• Support for Azure Key Vault and other secret managers
• No hardcoded credentials or secrets

For security concerns, please email security@rivoli-ai.com

Related Projects

• Andy.Configuration - Configuration management
• OpenAI-DotNet - Official OpenAI SDK
• Cerebras.Cloud.SDK - Cerebras SDK

Changelog

See CHANGELOG.md for version history and release notes.

2026-09-07: Structured provider failures

LlmProviderException.Error preserves provider, HTTP status, retry-after seconds and a bounded human message. OpenRouter captures HTTP metadata before disposing the response. ErrorReportingLlmProvider normalizes complete and streaming failures for hosts while preserving cancellation and unsupported-stream fallback (#42).

Optional OpenRouter serving-provider routing uses LlmRequest.ExtraBody; defaults remain unchanged when omitted.