HillPhelmuth.SemanticKernel.LlmAsJudgeEvals 1.0.0-preview

This is a prerelease version of HillPhelmuth.SemanticKernel.LlmAsJudgeEvals.
dotnet add package HillPhelmuth.SemanticKernel.LlmAsJudgeEvals --version 1.0.0-preview                
NuGet\Install-Package HillPhelmuth.SemanticKernel.LlmAsJudgeEvals -Version 1.0.0-preview                
This command is intended to be used within the Package Manager Console in Visual Studio, as it uses the NuGet module's version of Install-Package.
<PackageReference Include="HillPhelmuth.SemanticKernel.LlmAsJudgeEvals" Version="1.0.0-preview" />                
For projects that support PackageReference, copy this XML node into the project file to reference the package.
paket add HillPhelmuth.SemanticKernel.LlmAsJudgeEvals --version 1.0.0-preview                
#r "nuget: HillPhelmuth.SemanticKernel.LlmAsJudgeEvals, 1.0.0-preview"                
#r directive can be used in F# Interactive and Polyglot Notebooks. Copy this into the interactive tool or source code of the script to reference the package.
// Install HillPhelmuth.SemanticKernel.LlmAsJudgeEvals as a Cake Addin
#addin nuget:?package=HillPhelmuth.SemanticKernel.LlmAsJudgeEvals&version=1.0.0-preview&prerelease

// Install HillPhelmuth.SemanticKernel.LlmAsJudgeEvals as a Cake Tool
#tool nuget:?package=HillPhelmuth.SemanticKernel.LlmAsJudgeEvals&version=1.0.0-preview&prerelease                

LlmAsJudgeEvals

This library provides a service for evaluating responses from Large Language Models (LLMs) using the LLM itself as a judge. It leverages Semantic Kernel to define and execute evaluation functions based on prompt templates.

Installation

Install the package via NuGet:

Install-Package HillPhelmuth.SemanticKernel.LlmAsJudgeEvals

Usage

Built-in Evaluation Functions


// Initialize the Semantic Kernel
var kernel = Kernel.CreateBuilder().AddOpenAIChatCompletion("openai-model-name", "openai-apiKey").Build();

// Create an instance of the EvalService
var evalService = new EvalService(kernel);

// Create an input model for the built-in evaluation function
var coherenceInput = InputModel.CoherenceModel("This is the answer to evaluate.", "This is the question or prompt that generated the answer");

// Execute the evaluation
var result = await evalService.ExecuteEval(inputModel);


Console.WriteLine($"Evaluation score: {result.Score}");

Custom Evaluation Functions


// Initialize the Semantic Kernel
var kernel = Kernel.CreateBuilder().AddOpenAIChatCompletion("openai-model-name", "openai-apiKey").Build();

// Create an instance of the EvalService
var evalService = new EvalService(kernel);

// Add an evaluation function (optional)
evalService.AddEvalFunction("MyEvalFunction", "This is the prompt for my evaluation function.", new PromptExecutionSettings());

// Create an input model for the evaluation function
var inputModel = new InputModel
{
    FunctionName = "MyEvalFunction", // Replace with the name of your evaluation function
    RequiredInputs = new Dictionary<string, string>
    {
        { "input", "This is the text to evaluate." }
    }
};

// Execute the evaluation
var result = await evalService.ExecuteEval(inputModel);


Console.WriteLine($"Evaluation score: {result.Score}");

Features

  • Define evaluation functions using prompt templates: You can define evaluation functions using prompt templates written in YAML.
  • Execute evaluations: The EvalService provides methods for executing evaluations on input data.
  • Aggregate results: The EvalService can aggregate evaluation scores across multiple inputs.
  • Built-in evaluation functions: The package includes a set of pre-defined evaluation functions based on common evaluation metrics.
Product Compatible and additional computed target framework versions.
.NET net8.0 is compatible.  net8.0-android was computed.  net8.0-browser was computed.  net8.0-ios was computed.  net8.0-maccatalyst was computed.  net8.0-macos was computed.  net8.0-tvos was computed.  net8.0-windows was computed. 
Compatible target framework(s)
Included target framework(s) (in package)
Learn more about Target Frameworks and .NET Standard.

NuGet packages

This package is not used by any NuGet packages.

GitHub repositories

This package is not used by any popular GitHub repositories.

Version Downloads Last updated
1.0.0-preview 54 10/22/2024
0.1.0-beta 46 10/22/2024
0.0.3-beta 51 10/21/2024
0.0.2-beta 53 9/7/2024
0.0.1-beta 47 9/7/2024