TokenFlow.Core 0.4.0

<p align="center"> <img src="https://github.com/AndrewClements84/TokenFlow.AI/blob/master/assets/logo.png?raw=true" alt="TokenFlow.AI" width="500"/> </p>

TokenFlow.AI

💡 Overview

TokenFlow.AI is a lightweight .NET library for tokenization, chunking, and cost estimation across modern large language models (LLMs) such as OpenAI GPT‑4o, Anthropic Claude, and Azure OpenAI.

It provides accurate token counting, intelligent text splitting, cumulative usage tracking, and real‑time cost estimation for any AI‑driven application.
Now includes CLI utilities, developer documentation, and performance benchmarking.

🧩 Key Features

• 🔢 GPT‑style token counting for .NET
• 🧱 Smart text chunking with configurable token limits and overlap
• 💰 Real‑time cost estimation for prompt and completion usage
• 🧮 TokenUsageTracker — track cumulative token and cost usage across analyses
• 🧩 Unified TokenFlowClient — analyze, chunk, and cost in one API
• ⚙️ CLI utilities (TokenFlow.Tools) — structured automation with --format, --input, and --output options
• 📘 Developer documentation site — API reference + usage guides via GitHub Pages
• 🧾 Benchmark suite powered by BenchmarkDotNet
• 🔌 Pluggable tokenizer providers (OpenAI, Anthropic, Azure AI)
• 📦 Zero external dependencies — small, fast, portable
• 🧠 Dual targeting for .NET Standard 2.0 and .NET 8.0

🚀 Installation

dotnet add package TokenFlow.AI

Or install the shared core contracts:

dotnet add package TokenFlow.Core

🧠 Quick Examples

Token analysis and cost estimation:

using TokenFlow.AI.Client;

var client = new TokenFlowClient("gpt-4o");
var result = client.AnalyzeText("TokenFlow.AI brings structure to prompt engineering.");

Console.WriteLine($"Model: {result.ModelId}");
Console.WriteLine($"Tokens: {result.TokenCount}");
Console.WriteLine($"Estimated cost: £{result.EstimatedCost:F4}");

Chunking long text:

var chunks = client.ChunkText("This is a long body of text that exceeds a given token limit...", maxTokens: 50, overlapTokens: 5);

foreach (var chunk in chunks)
    Console.WriteLine($"Chunk: {chunk.Substring(0, Math.Min(40, chunk.Length))}...");

Tracking cumulative usage:

using TokenFlow.AI.Tracking;

var tracker = new TokenUsageTracker(client.GetModel());

tracker.Record(client.AnalyzeText("Hello TokenFlow.AI!"));
tracker.Record(client.AnalyzeText("Let's track token usage across sessions."));

var summary = tracker.GetSummary();

Console.WriteLine($"Analyses: {summary.AnalysisCount}");
Console.WriteLine($"Total Tokens: {summary.TotalTokens}");
Console.WriteLine($"Total Cost: £{summary.TotalCost:F4}");

Model Registry JSON Loading:

using TokenFlow.AI.Registry;

var registry = new ModelRegistry();
registry.LoadFromJsonString("[{ \"Id\": \"custom-model\", \"Family\": \"openai\", \"TokenizerName\": \"tiktoken\", \"MaxInputTokens\": 10000, \"MaxOutputTokens\": 2000, \"InputPricePer1K\": 0.01, \"OutputPricePer1K\": 0.02 }]");

var model = registry.TryGet("custom-model");
Console.WriteLine($"{model.Id}: {model.Family} — {model.MaxInputTokens} tokens");

Running via CLI (v2.1):

# Human-readable table output
dotnet run --project src/TokenFlow.Tools -- analyze "Hello TokenFlow!"

# JSON output
dotnet run --project src/TokenFlow.Tools -- analyze "Hello" --format json

# CSV output to file
dotnet run --project src/TokenFlow.Tools -- analyze "Hello" --format csv --output result.csv

# Quiet mode for CI/CD
dotnet run --project src/TokenFlow.Tools -- analyze "Quiet test" --format quiet

🧪 Running Tests

dotnet test --no-build --verbosity normal

All unit tests are written in xUnit and run automatically through GitHub Actions.
Code coverage is tracked with Codecov, and the project maintains 100% line and branch coverage.

🛠️ Roadmap

✅ Completed

• Core interfaces and models (ITokenizer, ICostEstimator, ModelSpec, TokenCountResult)
• Implemented ApproxTokenizer, CostEstimator, and ModelRegistry
• Added TokenChunker with overlap support
• Added TokenFlowClient — unified entry point for developers
• Added TokenUsageTracker — cumulative cost and token tracking
• Implemented ITokenizerFactory for dynamic tokenizer resolution
• Added TokenFlow.Tools — developer CLI utilities
• Added TokenFlow.Tools.Benchmarks — BenchmarkDotNet performance suite
• Full xUnit test suite with 100% code coverage
• CI/CD pipeline with Codecov and automated NuGet publishing
• Dual targeting for .NET Standard 2.0 and .NET 8.0
• Extended ModelRegistry to support JSON configuration loading ✅
• Developer documentation site (API + usage guides) ✅
• Expanded CLI commands and options (--format, --input, --output, --quiet, CSV + table formatting) ✅

🌟 Future Goals

• Advanced tokenizers (OpenAI tiktoken, Claude tokenizer)
• Integration with other Flow.AI components once public
• Performance regression tracking in CI
• Public release under Flow.AI ecosystem

💬 Contributing

Pull requests are welcome!
If you’d like to contribute to TokenFlow.AI, please read the upcoming CONTRIBUTING.md once published.

🪪 License

Distributed under the MIT License.
See LICENSE for details.

⭐ If you find TokenFlow.AI useful, please give the repository a star on GitHub!
It helps others discover the project and supports ongoing development.