AICentral 0.16.1-pullrequest0088-0003

This is a prerelease version of AICentral.
There is a newer version of this package available.
See the version list below for details.
dotnet add package AICentral --version 0.16.1-pullrequest0088-0003                
NuGet\Install-Package AICentral -Version 0.16.1-pullrequest0088-0003                
This command is intended to be used within the Package Manager Console in Visual Studio, as it uses the NuGet module's version of Install-Package.
<PackageReference Include="AICentral" Version="0.16.1-pullrequest0088-0003" />                
For projects that support PackageReference, copy this XML node into the project file to reference the package.
paket add AICentral --version 0.16.1-pullrequest0088-0003                
#r "nuget: AICentral, 0.16.1-pullrequest0088-0003"                
#r directive can be used in F# Interactive and Polyglot Notebooks. Copy this into the interactive tool or source code of the script to reference the package.
// Install AICentral as a Cake Addin
#addin nuget:?package=AICentral&version=0.16.1-pullrequest0088-0003&prerelease

// Install AICentral as a Cake Tool
#tool nuget:?package=AICentral&version=0.16.1-pullrequest0088-0003&prerelease                

AI Central

AI Central gives you control over your AI services.

  • Intelligent Routing
  • Custom consumer OAuth2 authorisation
  • Fallback AI service
  • Round Robin AI services
  • Lowest Latency AI service
  • Circuit breakers, and backoff-retry over downstream AI services
  • Request based and Token based rate limiting
  • Prompt and usage logging
    • Works for streaming endpoints as-well as non streaming
  • Additional Open Telemetry Metrics and Traces

Configuration

See docs on Github for more details.

Minimal

This sample produces a AI-Central proxy that

  • Listens on a hostname of your choosing
  • Proxies directly through to a back-end Open AI server
  • Can be accessed using standard SDKs
  • Outputs open-telemetry metrics to capture usage information
{
  "AICentral": {
    "Endpoints": [
      {
        "Type": "AzureOpenAIEndpoint",
        "Name": "openai-1",
        "Properties": {
          "LanguageEndpoint": "https://<my-ai>.openai.azure.com",
          "AuthenticationType": "ApiKey",
          "ApiKey": "<key>"
        }
      }
    ],
    "EndpointSelectors": [
      {
        "Type": "SingleEndpoint",
        "Name": "default",
        "Properties": {
          "Endpoint": "openai-1"
        }
      }
    ],
    "AuthProviders": [
      {
        "Type": "ApiKey",
        "Name": "apikey",
        "Properties": {
          "Clients": [
            {
              "ClientName": "Consumer-1",
              "Key1": "<random-key>",
              "Key2": "<random-key>"
            }
          ]
        }
      }
    ],
    "Pipelines": [
      {
        "Name": "OpenAIPipeline",
        "Host": "mypipeline.mydomain.com",
        "EndpointSelector": "default",
        "AuthProvider": "apikey"
        
      }
    ]
  }
}

Full example

This pipeline will:

  • Present an Azure Open AI, and an Open AI downstream as a single upstream endpoint
    • maps incoming Azure Open AI deployments to Open AI models
  • Present it as an Azure Open AI style endpoint
  • Protect the front-end by requiring an AAD token issued for your own AAD application
  • Put a local Asp.Net core rate-limiting policy over the endpoint
  • Add logging to Azure monitor
    • Logs quota, client caller information (IP and identity name), and in this case the Prompt but not the response.
  • Publish the client-name as a tag in Open Telemetry
{
  "AICentral": {
    "Endpoints": [
      {
        "Type": "AzureOpenAIEndpoint",
        "Name": "openai-priority",
        "Properties": {
          "LanguageEndpoint": "https://<my-ai>.openai.azure.com",
          "AuthenticationType": "Entra|EntraPassThrough|ApiKey",
          "MaxConcurrency": 10
        }
      },
      {
        "Type": "OpenAIEndpoint",
        "Name": "openai-fallback",
        "Properties": {
          "LanguageEndpoint": "https://api.openai.com",
          "ModelMappings": {
            "Gpt35Turbo0613": "gpt-3.5-turbo",
            "Ada002Embedding": "text-embedding-ada-002"
          },
          "ApiKey": "<my-api-key>",
          "Organization": "<optional-organisation>"
        }
      }
    ],
    "AuthProviders": [
      {
        "Type": "Entra",
        "Name": "simple-aad",
        "Properties": {
          "Entra": {
            "ClientId": "<my-client-id>",
            "TenantId": "<my-tenant-id>",
            "Instance": "https://login.microsoftonline.com/",
            "Audience": "<custom-audience>"
          }
        }
      }
    ],
    "EndpointSelectors": [
      {
        "Type": "Prioritised",
        "Name": "my-endpoint-selector",
        "Properties": {
          "PriorityEndpoints": ["openai-1"],
          "FallbackEndpoints": ["openai-fallback"]
        }
      }
    ],
    "GenericSteps": [
      {
        "Type": "AspNetCoreFixedWindowRateLimiting",
        "Name": "token-rate-limiter",
        "Properties": {
          "LimitType": "PerConsumer|PerAICentralEndpoint",
          "MetricType": "Tokens",
          "Options": {
            "Window": "00:01:00",
            "PermitLimit": 1000
          }
        }
      },
      {
        "Type": "AspNetCoreFixedWindowRateLimiting",
        "Name": "window-rate-limiter",
        "Properties": {
          "LimitType": "PerConsumer|PerAICentralEndpoint",
          "MetricType": "Requests",
          "Options": {
            "Window": "00:00:10",
            "PermitLimit": 100
          }
        }
      },
      {
        "Type": "AzureMonitorLogger",
        "Name": "azure-monitor-logger",
        "Properties": {
          "WorkspaceId": "<workspace-id>",
          "Key": "<key>",
          "LogPrompt": true,
          "LogResponse": false,
          "LogClient": true
        }
      },
      {
        "Type": "BulkHead",
        "Name": "bulk-head",
        "Properties": {
          "MaxConcurrency": 20
        }
      }
    ],
    "Pipelines": [
      {
        "Name": "MyPipeline",
        "Host": "prioritypipeline.mydomain.com",
        "EndpointSelector": "my-endpoint-selector",
        "AuthProvider": "simple-aad",
        "Steps": [
          "window-rate-limiter",
          "bulk-head",
          "azure-monitor-logger"
        ],
        "OpenTelemetryConfig": {
          "AddClientNameTag": true
        }
      }
    ]
  }
}

Product Compatible and additional computed target framework versions.
.NET net8.0 is compatible.  net8.0-android was computed.  net8.0-browser was computed.  net8.0-ios was computed.  net8.0-maccatalyst was computed.  net8.0-macos was computed.  net8.0-tvos was computed.  net8.0-windows was computed.  net9.0 was computed.  net9.0-android was computed.  net9.0-browser was computed.  net9.0-ios was computed.  net9.0-maccatalyst was computed.  net9.0-macos was computed.  net9.0-tvos was computed.  net9.0-windows was computed. 
Compatible target framework(s)
Included target framework(s) (in package)
Learn more about Target Frameworks and .NET Standard.

NuGet packages

This package is not used by any NuGet packages.

GitHub repositories

This package is not used by any popular GitHub repositories.

Version Downloads Last updated
0.21.0 117 a month ago
0.21.0-pullrequest0152-0005 47 a month ago
0.21.0-pullrequest0152-0004 45 a month ago
0.21.0-pullrequest0152-0003 45 a month ago
0.21.0-pullrequest0152-0002 47 a month ago
0.20.7 212 2 months ago
0.20.7-pullrequest0151-0002 54 2 months ago
0.20.6 348 2 months ago
0.20.6-pullrequest0150-0003 48 2 months ago
0.20.5 147 3 months ago
0.20.5-pullrequest0148-0004 63 3 months ago
0.20.5-pullrequest0148-0003 56 3 months ago
0.20.5-pullrequest0148-0002 68 3 months ago
0.20.3 104 4 months ago
0.20.3-pullrequest0146-0005 54 4 months ago
0.20.3-pullrequest0146-0004 56 4 months ago
0.20.2 68 4 months ago
0.20.2-pullrequest0145-0004 48 4 months ago
0.20.0 92 5 months ago
0.20.0-pullrequest0143-0013 71 5 months ago
0.20.0-pullrequest0143-0012 71 5 months ago
0.20.0-pullrequest0143-0011 73 5 months ago
0.20.0-pullrequest0143-0010 69 5 months ago
0.20.0-pullrequest0143-0009 69 5 months ago
0.19.6-pullrequest0143-0008 78 5 months ago
0.19.5 63 5 months ago
0.19.5-pullrequest0141-0002 52 5 months ago
0.19.4 65 5 months ago
0.19.4-pullrequest0140-0003 52 5 months ago
0.19.4-pullrequest0139-0001 53 5 months ago
0.19.3 62 5 months ago
0.19.2 65 5 months ago
0.19.2-pullrequest0138-0004 52 5 months ago
0.19.1 59 5 months ago
0.19.1-pullrequest0137-0003 53 5 months ago
0.19.0 53 5 months ago
0.19.0-pullrequest0136-0014 48 5 months ago
0.19.0-pullrequest0136-0013 49 5 months ago
0.19.0-pullrequest0136-0012 140 5 months ago
0.19.0-pullrequest0136-0011 50 5 months ago
0.19.0-pullrequest0136-0010 55 5 months ago
0.19.0-pullrequest0136-0009 66 5 months ago
0.19.0-pullrequest0136-0008 57 5 months ago
0.19.0-pullrequest0136-0007 58 5 months ago
0.18.6 73 6 months ago
0.18.6-pullrequest0135-0002 53 6 months ago
0.18.5 59 6 months ago
0.18.5-pullrequest0134-0003 49 6 months ago
0.18.4 64 6 months ago
0.18.4-pullrequest0133-0009 54 6 months ago
0.18.4-pullrequest0133-0008 50 6 months ago
0.18.4-pullrequest0133-0007 53 6 months ago
0.18.3 68 6 months ago
0.18.3-pullrequest0132-0005 54 6 months ago
0.18.2 61 6 months ago
0.18.2-pullrequest0131-0003 58 6 months ago
0.18.2-pullrequest0130-0003 55 6 months ago
0.18.1 64 6 months ago
0.18.1-pullrequest0129-0007 51 6 months ago
0.18.0 98 6 months ago
0.18.0-pullrequest0125-0014 51 6 months ago
0.18.0-pullrequest0125-0013 59 6 months ago
0.18.0-pullrequest0125-0012 49 6 months ago
0.18.0-pullrequest0125-0011 49 6 months ago
0.18.0-pullrequest0125-0010 50 6 months ago
0.18.0-pullrequest0125-0009 60 6 months ago
0.17.2-pullrequest0125-0008 52 6 months ago
0.17.2-pullrequest0125-0007 58 6 months ago
0.17.2-pullrequest0125-0006 52 6 months ago
0.17.2-pullrequest0125-0005 54 6 months ago
0.17.2-pullrequest0125-0004 58 6 months ago
0.17.2-pullrequest0125-0003 54 6 months ago
0.17.1 63 6 months ago
0.17.1-pullrequest0124-0005 56 6 months ago
0.17.0 65 6 months ago
0.17.0-pullrequest0123-0024 46 6 months ago
0.17.0-pullrequest0123-0023 47 6 months ago
0.17.0-pullrequest0123-0021 60 6 months ago
0.17.0-pullrequest0123-0019 51 6 months ago
0.17.0-pullrequest0123-0018 43 6 months ago
0.17.0-pullrequest0123-0017 54 6 months ago
0.17.0-pullrequest0123-0015 52 6 months ago
0.17.0-pullrequest0123-0014 48 6 months ago
0.17.0-pullrequest0123-0013 53 6 months ago
0.17.0-pullrequest0123-0012 48 6 months ago
0.17.0-pullrequest0123-0011 55 6 months ago
0.17.0-pullrequest0123-0009 57 6 months ago
0.17.0-pullrequest0123-0008 52 6 months ago
0.17.0-pullrequest0123-0007 90 6 months ago
0.16.23 70 7 months ago
0.16.23-pullrequest0118-0005 62 7 months ago
0.16.23-pullrequest0118-0004 55 7 months ago
0.16.23-pullrequest0118-0003 54 7 months ago
0.16.23-pullrequest0118-0002 53 7 months ago
0.16.21 123 7 months ago
0.16.21-pullrequest0114-0002 60 7 months ago
0.16.20 69 7 months ago
0.16.20-pullrequest0112-0002 54 7 months ago
0.16.19 73 8 months ago
0.16.19-pullrequest0110-0004 67 8 months ago
0.16.19-pullrequest0110-0003 57 8 months ago
0.16.17 92 9 months ago
0.16.17-pullrequest0108-0005 70 9 months ago
0.16.17-pullrequest0108-0004 61 9 months ago
0.16.17-pullrequest0108-0003 61 9 months ago
0.16.16 82 9 months ago
0.16.16-pullrequest0107-0003 76 9 months ago
0.16.15 79 9 months ago
0.16.15-pullrequest0106-0004 62 9 months ago
0.16.15-pullrequest0106-0003 60 9 months ago
0.16.15-pullrequest0106-0002 71 9 months ago
0.16.14 140 9 months ago
0.16.13 91 10 months ago
0.16.13-pullrequest0103-0003 79 10 months ago
0.16.12 78 10 months ago
0.16.12-pullrequest0102-0006 68 10 months ago
0.16.11 75 10 months ago
0.16.11-pullrequest0101-0004 60 10 months ago
0.16.11-pullrequest0101-0003 66 10 months ago
0.16.11-pullrequest0101-0002 60 10 months ago
0.16.9 82 10 months ago
0.16.9-pullrequest0098-0008 64 10 months ago
0.16.9-pullrequest0098-0007 66 10 months ago
0.16.8 101 10 months ago
0.16.8-pullrequest0097-0002 64 10 months ago
0.16.7 79 3/10/2024
0.16.7-pullrequest0095-0002 74 3/10/2024
0.16.6 162 3/9/2024
0.16.6-pullrequest0094-0003 65 3/9/2024
0.16.6-pullrequest0094-0002 62 3/9/2024
0.16.2 90 3/5/2024
0.16.2-pullrequest0089-0009 71 3/5/2024
0.16.1 85 3/3/2024
0.16.1-pullrequest0088-0003 58 3/2/2024
0.16.1-pullrequest0088-0002 56 3/2/2024
0.16.0 130 2/28/2024
0.16.0-pullrequest0087-0010 59 2/28/2024
0.16.0-pullrequest0087-0009 63 2/28/2024
0.16.0-pullrequest0087-0007 72 2/28/2024
0.16.0-pullrequest0087-0006 61 2/28/2024
0.16.0-pullrequest0087-0005 58 2/28/2024
0.16.0-pullrequest0087-0004 68 2/28/2024
0.16.0-pullrequest0087-0003 61 2/28/2024
0.15.2 73 2/26/2024
0.15.2-pullrequest0086-0007 57 2/26/2024
0.15.2-pullrequest0086-0006 51 2/26/2024
0.15.1 78 2/23/2024
0.15.1-pullrequest0085-0006 58 2/23/2024
0.15.0 76 2/22/2024
0.15.0-pullrequest0084-0004 60 2/22/2024
0.15.0-pullrequest0084-0003 88 2/22/2024
0.14.1-pullrequest0084-0002 71 2/22/2024
0.14.0 92 2/21/2024
0.14.0-pullrequest0083-0011 83 2/8/2024
0.14.0-pullrequest0083-0010 67 2/8/2024
0.14.0-pullrequest0083-0009 65 2/8/2024
0.13.7 100 2/8/2024
0.13.7-pullrequest0082-0002 77 2/8/2024
0.13.6-pullrequest0081-0006 65 2/8/2024
0.13.5 96 2/7/2024
0.13.5-pullrequest0080-0002 70 2/7/2024
0.13.4 75 2/7/2024
0.13.4-pullrequest0079-0003 61 2/7/2024
0.13.1 87 2/1/2024
0.13.1-pullrequest0076-0003 70 2/1/2024
0.13.1-pullrequest0076-0002 70 2/1/2024
0.13.0 101 2/1/2024
0.13.0-pullrequest0075-0005 68 2/1/2024
0.13.0-pullrequest0075-0004 62 2/1/2024
0.12.2-pullrequest0075-0003 63 2/1/2024
0.12.1 79 2/1/2024
0.12.1-pullrequest0074-0002 65 2/1/2024
0.12.0 75 2/1/2024
0.12.0-pullrequest0073-0002 63 2/1/2024
0.11.0 77 1/31/2024
0.11.0-pullrequest0072-0011 67 1/31/2024
0.11.0-pullrequest0072-0010 64 1/31/2024
0.11.0-pullrequest0072-0008 61 1/31/2024
0.10.0 78 1/31/2024
0.10.0-pullrequest0070-0005 70 1/29/2024
0.10.0-pullrequest0070-0004 66 1/29/2024
0.9.1-pullrequest0069-0001 65 1/29/2024
0.9.0 76 1/29/2024
0.9.0-pullrequest0069-0002 62 1/29/2024
0.8.7-pullrequest0068-0001 66 1/29/2024
0.8.6 79 1/29/2024
0.8.4 96 1/19/2024
0.8.4-pullrequest0066-0009 65 1/19/2024
0.8.4-pullrequest0066-0008 58 1/19/2024
0.8.4-pullrequest0066-0007 65 1/19/2024
0.8.4-pullrequest0066-0005 61 1/19/2024
0.8.4-pullrequest0066-0004 67 1/19/2024
0.8.4-pullrequest0066-0003 61 1/19/2024
0.8.3 75 1/19/2024
0.8.3-pullrequest0065-0003 63 1/19/2024
0.8.3-pullrequest0065-0002 63 1/19/2024
0.8.1 75 1/19/2024
0.8.1-pullrequest0063-0013 63 1/19/2024
0.8.1-pullrequest0063-0011 66 1/19/2024
0.7.16-pullrequest0063-0010 65 1/19/2024
0.7.15 84 1/18/2024
0.7.15-pullrequest0062-0002 68 1/18/2024
0.7.12 91 1/16/2024
0.7.12-pullrequest0059-0006 69 1/16/2024
0.7.10 83 1/15/2024
0.7.10-pullrequest0059-0007 66 1/16/2024
0.7.10-pullrequest0058-0004 58 1/15/2024
0.7.10-pullrequest0057-0002 65 1/15/2024
0.7.9 84 1/15/2024
0.7.9-pullrequest0057-0003 62 1/15/2024
0.7.7 81 1/15/2024
0.7.7-pullrequest0056-0003 77 1/15/2024
0.7.4 84 1/15/2024
0.7.4-pullrequest0055-0012 70 1/15/2024
0.7.4-pullrequest0055-0011 69 1/15/2024
0.7.4-pullrequest0055-0010 75 1/15/2024
0.7.4-pullrequest0055-0009 67 1/15/2024
0.7.4-pullrequest0055-0008 68 1/15/2024
0.7.4-pullrequest0055-0007 70 1/15/2024
0.7.4-pullrequest0054-0005 70 1/15/2024
0.7.4-pullrequest0053-0004 81 1/15/2024
0.7.4-pullrequest0052-0002 60 1/15/2024
0.7.2 79 1/15/2024
0.7.2-pullrequest0051-0013 71 1/15/2024
0.7.2-pullrequest0050-0010 62 1/15/2024
0.7.1 90 1/13/2024
0.7.0 86 1/13/2024
0.7.0-pullrequest0046-0079 81 1/13/2024
0.6.12 102 1/3/2024
0.6.12-pullrequest0045-0018 67 1/13/2024
0.6.12-pullrequest0042-0010 69 1/12/2024
0.6.12-pullrequest0041-0007 69 1/12/2024
0.6.12-pullrequest0039-0002 72 1/3/2024
0.6.10 102 12/22/2023
0.6.10-pullrequest0038-0007 84 1/2/2024
0.6.10-pullrequest0038-0006 74 1/2/2024
0.6.10-pullrequest0038-0005 73 1/2/2024
0.6.8 96 12/21/2023
0.6.6 91 12/18/2023
0.6.6-pullrequest0035-0007 80 12/21/2023
0.6.6-pullrequest0035-0006 79 12/21/2023
0.6.6-pullrequest0035-0005 74 12/21/2023
0.6.3 90 12/18/2023
0.6.3-pullrequest0034-0018 80 12/18/2023
0.6.1 82 12/15/2023
0.6.1-pullrequest0029-0008 78 12/15/2023
0.6.1-pullrequest0028-0006 84 12/15/2023
0.6.1-pullrequest0027-0004 63 12/15/2023
0.5.3 92 12/11/2023
0.5.3-pullrequest0025-0049 84 12/14/2023
0.5.3-pullrequest0025-0047 81 12/14/2023
0.5.3-pullrequest0025-0043 75 12/13/2023
0.5.3-pullrequest0022-0038 76 12/13/2023
0.5.1 88 12/7/2023
0.5.0 96 12/6/2023
0.4.3 101 12/4/2023
0.4.2 91 12/4/2023
0.4.1 100 12/4/2023
0.3.10 109 11/22/2023
0.3.10-pullrequest0022-0008 78 11/27/2023
0.3.10-pullrequest0021-0005 81 11/22/2023
0.3.9 87 11/20/2023
0.3.8 98 11/9/2023
0.3.8-pullrequest0020-0005 75 11/9/2023
0.3.6 88 11/9/2023
0.3.5 96 11/9/2023
0.3.4 89 11/8/2023
0.3.3 90 11/8/2023
0.3.1 93 11/8/2023
0.3.1-pullrequest0018-0004 82 11/8/2023
0.2.1 128 11/2/2023
0.2.1-pullrequest0017-0050 90 11/8/2023
0.2.1-pullrequest0016-0048 86 11/8/2023
0.2.1-pullrequest0015-0045 82 11/8/2023
0.2.1-pullrequest0014-0035 91 11/7/2023
0.2.1-pullrequest0013-0033 83 11/7/2023
0.2.1-pullrequest0012-0031 87 11/7/2023
0.2.1-pullrequest0011-0024 86 11/6/2023
0.2.1-pullrequest0010-0015 97 11/2/2023
0.2.1-pullrequest0009-0011 88 11/2/2023
0.2.1-pullrequest0008-0009 89 11/2/2023
0.2.1-pullrequest0007-0007 96 11/2/2023
0.2.1-pullrequest0006-0005 90 11/2/2023
0.2.1-pullrequest0005-0001 83 11/2/2023
0.1.0 116 10/27/2023
0.1.0-pullrequest0004-0024 90 11/1/2023
0.1.0-pullrequest0003-0022 91 11/1/2023
0.1.0-pullrequest0002-0019 89 11/1/2023
0.1.0-pullrequest0001-0017 92 11/1/2023
0.1.0-pullrequest0001-0010 94 10/30/2023

releasenotes.md