Ai.Tlbx.VoiceAssistant 8.5.0

AI Voice Assistant Toolkit

Real-time voice conversations with AI in .NET — OpenAI, Google Gemini, and xAI Grok in one unified API.

Quick Start (Blazor Server)

1. Install packages:

dotnet add package Ai.Tlbx.VoiceAssistant
dotnet add package Ai.Tlbx.VoiceAssistant.Provider.OpenAi   # and/or .Google, .XAi
dotnet add package Ai.Tlbx.VoiceAssistant.Hardware.Web

2. Add API keys (appsettings.json):

{
  "VoiceProviders": {
    "OpenAI": "sk-...",
    "Google": "AIza...",
    "xAI": "xai-..."
  }
}

3. Configure services (Program.cs):

// Register provider factory and audio hardware
builder.Services.AddSingleton<IVoiceProviderFactory, VoiceProviderFactory>();
builder.Services.AddScoped<IAudioHardwareAccess, WebAudioAccess>();

4. Create a voice page (Voice.razor):

@page "/voice"
@inject IVoiceProviderFactory ProviderFactory
@inject IAudioHardwareAccess AudioHardware
@inject IConfiguration Config

<select @bind="_selectedProvider">
    <option value="openai">OpenAI</option>
    <option value="google">Google Gemini</option>
    <option value="xai">xAI Grok</option>
</select>

<button @onclick="Toggle">@(_assistant?.IsRecording == true ? "Stop" : "Talk")</button>

@foreach (var msg in _messages)
{
    <p><b>@msg.Role:</b> @msg.Content</p>
}

@code {
    private VoiceAssistant? _assistant;
    private List<ChatMessage> _messages = new();
    private string _selectedProvider = "openai";

    private async Task Toggle()
    {
        if (_assistant?.IsRecording == true)
        {
            await _assistant.StopAsync();
            return;
        }

        // Create provider based on selection
        var (provider, settings) = _selectedProvider switch
        {
            "openai" => (
                ProviderFactory.CreateOpenAi(Config["VoiceProviders:OpenAI"]!),
                (IVoiceSettings)new OpenAiVoiceSettings { Instructions = "You are helpful." }
            ),
            "google" => (
                ProviderFactory.CreateGoogle(Config["VoiceProviders:Google"]!),
                (IVoiceSettings)new GoogleVoiceSettings { Instructions = "You are helpful." }
            ),
            "xai" => (
                ProviderFactory.CreateXai(Config["VoiceProviders:xAI"]!),
                (IVoiceSettings)new XaiVoiceSettings { Instructions = "You are helpful." }
            ),
            _ => throw new InvalidOperationException()
        };

        _assistant = new VoiceAssistant(provider, AudioHardware);
        _assistant.OnMessageReceived = msg => InvokeAsync(() => { _messages.Add(msg); StateHasChanged(); });

        await _assistant.StartAsync(settings);
    }
}

That's it. Select a provider, talk to the AI, get voice responses back.

All Packages

Package	Purpose	NuGet
Ai.Tlbx.VoiceAssistant	Core orchestrator
...Provider.OpenAi	OpenAI Realtime API
...Provider.Google	Google Gemini Live API
...Provider.XAi	xAI Grok Voice Agent API
...Hardware.Web	Browser audio (Blazor)
...Hardware.Windows	Native Windows audio
...Hardware.Linux	Native Linux audio (ALSA)
...WebUi	Pre-built Blazor components

Switch Providers in One Line

// OpenAI
var provider = factory.CreateOpenAi(apiKey);
var settings = new OpenAiVoiceSettings { Voice = AssistantVoice.Alloy };

// Google Gemini
var provider = factory.CreateGoogle(apiKey);
var settings = new GoogleVoiceSettings { Voice = GeminiVoice.Puck };

// xAI Grok
var provider = factory.CreateXai(apiKey);
var settings = new XaiVoiceSettings { Voice = XaiVoice.Ara };

Same VoiceAssistant API, same tool definitions — just swap the provider.

Tools: Just Write C#

Define tools with plain C# records. Schema is auto-inferred — no JSON, no manual mapping:

[Description("Get weather for a location")]
public class WeatherTool : VoiceToolBase<WeatherTool.Args>
{
    public record Args(
        [property: Description("City name")] string Location,
        [property: Description("Temperature unit")] TemperatureUnit Unit = TemperatureUnit.Celsius
    );

    public override string Name => "get_weather";

    public override Task<string> ExecuteAsync(Args args)
    {
        return Task.FromResult(CreateSuccessResult(new { temp = 22, location = args.Location }));
    }
}

public enum TemperatureUnit { Celsius, Fahrenheit }

Universal translation: The same tool works on OpenAI, Google, and xAI. Required/optional parameters, enums, nested objects — all inferred from C# types.

Register in DI:

builder.Services.AddTransient<IVoiceTool, WeatherTool>();

Writing Custom Tools

Basic Pattern

1. Create a record for arguments — use [Description] attributes for AI guidance
2. Extend VoiceToolBase<TArgs> — add [Description] to the class itself
3. Implement ExecuteAsync — return results via ToolSuccessResult<T>

[Description("Search for products in the catalog")]
public class ProductSearchTool : VoiceToolBase<ProductSearchTool.Args>
{
    public record Args(
        [property: Description("Search query keywords")] string Query,
        [property: Description("Maximum results to return")] int MaxResults = 10,
        [property: Description("Filter by category")] string? Category = null
    );

    public override string Name => "search_products";

    public override async Task<string> ExecuteAsync(Args args)
    {
        var products = await _catalogService.SearchAsync(args.Query, args.MaxResults, args.Category);

        var result = new ToolSuccessResult<ProductSearchResult>(new ProductSearchResult
        {
            Products = products,
            TotalFound = products.Count
        });

        return JsonSerializer.Serialize(result, YourJsonContext.Default.ToolSuccessResultProductSearchResult);
    }
}

Required vs Optional Parameters

• Required: No default value, non-nullable → AI must provide or ask user
• Optional: Has default value OR nullable (string?) → AI can omit

public record Args(
    string RequiredParam,                    // Required - AI must provide
    string OptionalWithDefault = "default",  // Optional - has default
    string? OptionalNullable = null          // Optional - nullable
);

Returning Results

Always use the provided result types for consistent AI interpretation:

// Success with data
var result = new ToolSuccessResult<YourDataType>(data);
return JsonSerializer.Serialize(result, YourJsonContext.Default.ToolSuccessResultYourDataType);

// Error
return CreateErrorResult("Something went wrong");

AOT Compatibility

For Native AOT or Blazor WebAssembly with trimming, reflection-based JSON serialization won't work. You need source-generated JSON contexts.

Step 1: Create a JSON context for your tool types

[JsonSerializable(typeof(ToolSuccessResult<ProductSearchResult>))]
[JsonSerializable(typeof(ProductSearchTool.Args), TypeInfoPropertyName = "ProductSearchToolArgs")]
[JsonSourceGenerationOptions(
    PropertyNamingPolicy = JsonKnownNamingPolicy.SnakeCaseLower,
    DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull)]
public partial class MyToolsJsonContext : JsonSerializerContext
{
}

Step 2: Register the type info with your tool

// In your DI setup or tool initialization
var tool = new ProductSearchTool();
tool.SetJsonTypeInfo(MyToolsJsonContext.Default.ProductSearchToolArgs);

Step 3: Use source-generated serialization in ExecuteAsync

return JsonSerializer.Serialize(result, MyToolsJsonContext.Default.ToolSuccessResultProductSearchResult);

AOT Checklist

Dependency Injection with Tools

Tools can use constructor injection:

public class DatabaseTool : VoiceToolBase<DatabaseTool.Args>
{
    private readonly IDbConnection _db;

    public DatabaseTool(IDbConnection db) => _db = db;

    // ...
}

// Registration
builder.Services.AddTransient<IVoiceTool, DatabaseTool>();

Key Features

Studio-Quality Audio Processing

The Hardware.Web package includes an AudioWorklet-based audio chain:

• 48kHz capture with browser echo cancellation, noise suppression, and auto gain
• De-esser (high-shelf EQ) to tame sibilance before amplification
• Compressor (8:1 ratio) for consistent loudness across whispers and shouts
• Anti-aliasing filter (Butterworth LPF) before downsampling
• Provider-specific sample rates: 16kHz for Google, 24kHz for OpenAI/xAI

Provider-Agnostic Architecture

Write once, run on any provider. The orchestrator handles:

• Audio format conversion (PCM 16-bit, provider-specific sample rates)
• Tool schema translation per provider
• Streaming audio playback with interruption support
• Chat history management

Built-in Tools

• TimeTool — Current time in any timezone
• WeatherTool — Mock weather (demo)
• CalculatorTool — Basic math operations

Native Apps (Windows/Linux)

Note: Native desktop support works but is less polished than the web implementation. Good for experiments and prototypes.

// Windows (requires Windows 10+)
dotnet add package Ai.Tlbx.VoiceAssistant.Hardware.Windows

// Linux (requires libasound2-dev)
dotnet add package Ai.Tlbx.VoiceAssistant.Hardware.Linux

var provider = new OpenAiVoiceProvider(apiKey, logger);
var hardware = new WindowsAudioHardware(); // or LinuxAudioDevice

var assistant = new VoiceAssistant(provider, hardware);
await assistant.StartAsync(settings);

Console.ReadKey(); // Talk now
await assistant.StopAsync();

Requirements

• .NET 9.0 or .NET 10.0
• API Key: OpenAI, Google AI Studio, or xAI
• Web: Modern browser with microphone permission (HTTPS or localhost)
• Windows: Windows 10+
• Linux: sudo apt-get install libasound2-dev

License

MIT — do whatever you want.

<p align="center"> <a href="https://www.nuget.org/packages/Ai.Tlbx.VoiceAssistant/">NuGet</a> • <a href="https://github.com/AiTlbx/Ai.Tlbx.VoiceAssistant/issues">Issues</a> • <a href="https://github.com/AiTlbx/Ai.Tlbx.VoiceAssistant">GitHub</a> </p>