HillPhelmuth.SemanticKernel.LlmAsJudgeEvals
1.0.0-preview
This is a prerelease version of HillPhelmuth.SemanticKernel.LlmAsJudgeEvals.
dotnet add package HillPhelmuth.SemanticKernel.LlmAsJudgeEvals --version 1.0.0-preview
NuGet\Install-Package HillPhelmuth.SemanticKernel.LlmAsJudgeEvals -Version 1.0.0-preview
This command is intended to be used within the Package Manager Console in Visual Studio, as it uses the NuGet module's version of Install-Package.
<PackageReference Include="HillPhelmuth.SemanticKernel.LlmAsJudgeEvals" Version="1.0.0-preview" />
For projects that support PackageReference, copy this XML node into the project file to reference the package.
paket add HillPhelmuth.SemanticKernel.LlmAsJudgeEvals --version 1.0.0-preview
The NuGet Team does not provide support for this client. Please contact its maintainers for support.
#r "nuget: HillPhelmuth.SemanticKernel.LlmAsJudgeEvals, 1.0.0-preview"
#r directive can be used in F# Interactive and Polyglot Notebooks. Copy this into the interactive tool or source code of the script to reference the package.
// Install HillPhelmuth.SemanticKernel.LlmAsJudgeEvals as a Cake Addin #addin nuget:?package=HillPhelmuth.SemanticKernel.LlmAsJudgeEvals&version=1.0.0-preview&prerelease // Install HillPhelmuth.SemanticKernel.LlmAsJudgeEvals as a Cake Tool #tool nuget:?package=HillPhelmuth.SemanticKernel.LlmAsJudgeEvals&version=1.0.0-preview&prerelease
The NuGet Team does not provide support for this client. Please contact its maintainers for support.
LlmAsJudgeEvals
This library provides a service for evaluating responses from Large Language Models (LLMs) using the LLM itself as a judge. It leverages Semantic Kernel to define and execute evaluation functions based on prompt templates.
Installation
Install the package via NuGet:
Install-Package HillPhelmuth.SemanticKernel.LlmAsJudgeEvals
Usage
Built-in Evaluation Functions
// Initialize the Semantic Kernel
var kernel = Kernel.CreateBuilder().AddOpenAIChatCompletion("openai-model-name", "openai-apiKey").Build();
// Create an instance of the EvalService
var evalService = new EvalService(kernel);
// Create an input model for the built-in evaluation function
var coherenceInput = InputModel.CoherenceModel("This is the answer to evaluate.", "This is the question or prompt that generated the answer");
// Execute the evaluation
var result = await evalService.ExecuteEval(inputModel);
Console.WriteLine($"Evaluation score: {result.Score}");
Custom Evaluation Functions
// Initialize the Semantic Kernel
var kernel = Kernel.CreateBuilder().AddOpenAIChatCompletion("openai-model-name", "openai-apiKey").Build();
// Create an instance of the EvalService
var evalService = new EvalService(kernel);
// Add an evaluation function (optional)
evalService.AddEvalFunction("MyEvalFunction", "This is the prompt for my evaluation function.", new PromptExecutionSettings());
// Create an input model for the evaluation function
var inputModel = new InputModel
{
FunctionName = "MyEvalFunction", // Replace with the name of your evaluation function
RequiredInputs = new Dictionary<string, string>
{
{ "input", "This is the text to evaluate." }
}
};
// Execute the evaluation
var result = await evalService.ExecuteEval(inputModel);
Console.WriteLine($"Evaluation score: {result.Score}");
Features
- Define evaluation functions using prompt templates: You can define evaluation functions using prompt templates written in YAML.
- Execute evaluations: The
EvalService
provides methods for executing evaluations on input data. - Aggregate results: The
EvalService
can aggregate evaluation scores across multiple inputs. - Built-in evaluation functions: The package includes a set of pre-defined evaluation functions based on common evaluation metrics.
Product | Versions Compatible and additional computed target framework versions. |
---|---|
.NET | net8.0 is compatible. net8.0-android was computed. net8.0-browser was computed. net8.0-ios was computed. net8.0-maccatalyst was computed. net8.0-macos was computed. net8.0-tvos was computed. net8.0-windows was computed. |
Compatible target framework(s)
Included target framework(s) (in package)
Learn more about Target Frameworks and .NET Standard.
-
net8.0
- Microsoft.SemanticKernel (>= 1.24.1)
- Microsoft.SemanticKernel.Plugins.Memory (>= 1.24.1-alpha)
- Microsoft.SemanticKernel.Yaml (>= 1.24.1)
NuGet packages
This package is not used by any NuGet packages.
GitHub repositories
This package is not used by any popular GitHub repositories.
Version | Downloads | Last updated |
---|---|---|
1.0.0-preview | 54 | 10/22/2024 |
0.1.0-beta | 46 | 10/22/2024 |
0.0.3-beta | 51 | 10/21/2024 |
0.0.2-beta | 53 | 9/7/2024 |
0.0.1-beta | 47 | 9/7/2024 |