Themis.Inference 1.5.1

Themis.Inference

Themis.Inference validates prepared ONNX and RLCD GGUF bundles, runs typed inference, and provides keyed model packers. Themis.Inference.Ollama provides separate optional chat and System One decision integrations without depending on ONNX Runtime or tokenizers.

The package targets net10.0 and provides the default ONNX Runtime path. Windows applications can add the optional Themis.Inference.WinML package and call AddThemisWinML() from Themis.Inference.Extensions to enable Windows ML execution-provider discovery.

For compatibility, the Themis composition root continues to expose AddThemisOllama(). Chat-only applications can reference Themis.Inference.Ollama directly and call AddThemisOllamaChat() to register an IInferenceProvider and IOllamaModelCatalog without registering decision inference. Configure the endpoint and optional default model through OllamaOptions; this does not replace ONNX decision inference.

Hugging Face downloads

All Hugging Face artifact downloads are disabled by default. Runtime acquisition requires both ThemisOptions.AllowModelDownloads = true and an explicit IModelDownloader registration through Themis.Inference.Artifacts.AddThemisHuggingFaceModelDownloader(). A cache hit remains usable without network access. If a remote cache miss is not permitted or the downloader is not registered, inference fails with an actionable message explaining the required opt-in or local model setup.

Model pack requests also default to downloads disabled and require ModelPackRequest.AllowDownload = true for remote sources. Local model files can be packed without registering a downloader when the selected packer supports local inputs; the built-in GGUF packer accepts a local artifact directory.

RLCD GGUF decisions

AddThemisRlcdGgufDecisionInference() registers a typed decision provider for the separate ModernBERT GGUF backbone and RLCD safetensors head. It sends token IDs to an externally managed llama.cpp /embedding endpoint and runs the two-layer decision head in the .NET process. It neither starts llama.cpp nor serves chat completions. For an Ollama-served System One model, register the separate AddThemisOllamaSystemOneDecisionInference() provider; for Ollama chat, use AddThemisOllama(). The THEMIS_RLCD_GGUF_MODEL_PATH environment variable identifies the packed bundle directory. The previous THEMIS_LAYA_GGUF_MODEL_PATH variable remains accepted for compatibility. When both are unset, the default is %LOCALAPPDATA%\Rldc\models\laya-gguf.

Ollama System One decisions

AddThemisOllamaSystemOneDecisionInference() adapts Ollama's /v1/systemone endpoint to IDecisionInferenceProvider. It is separate from both the Ollama chat provider and the llama.cpp-backed RLCD GGUF provider. Ollama must be version 0.35.0 or later, and the selected local model must be trained for System One. Ollama-compatible GGUF support does not imply that an arbitrary GGUF model implements System One.

The provider maps choice, score, and noul results to Themis' typed answers. It uses the shared decision request's state, questions, and namespaced extras. Themis-owned model context is placed under extras.rldc, including the central question and any attribute-driven currentResults; application extras remain in their own namespaces. The Ollama adapter sends this context in a versioned System One state envelope. Task and Language are routing hints rather than model inputs. System One does not support tools, streaming, or generation controls.

[IncludePreviousResults] enriches DecisionRequest.Extras while keeping the original State unchanged. ONNX and RLCD GGUF providers project those extras into their existing model-input shape; requests without extras retain their existing input.

Conversational inference and tool calling

OllamaInferenceProvider is an optional conversational integration for chat models served by Ollama, including use through the Themis.Maf adapter. It is not needed for Themis' typed decision inference, and registering it does not make the decision models conversational.

The built-in ONNX contracts produce typed decision logits, and the RLCD GGUF provider requests embeddings before applying its typed decision head. Neither path performs autoregressive text generation, so neither can produce assistant tool calls. The shared tool-call response types and MAF adapter only transport tool calls produced by a compatible chat model; they do not add that ability to a decision model.

To support tool calling with a Themis model, Themis would first need a chat-capable model contract and inference runtime that can generate response tokens and represent tool-call outputs, plus a conversational provider that implements IInferenceProvider and advertises only the capabilities it actually supports. Adding SupportsTools to the current decision providers would not be sufficient.

Inference contracts

OnnxDecisionInferenceProvider dispatches by the contract declared in the bundle manifest. Transformers sequence classification is an ONNX contract, not a separate runtime: it shares the provider's session, BPE input preparation, and answer mapping, while using the contract's single logits output. Keeping that route in a private method separates contract-specific work without duplicating service registration or native-session ownership.

Set THEMIS_MODEL_PATH to an absolute model-data directory when resolving a bundle by local name or caching downloads. Direct local ModelBundleDirectory configuration does not need that variable. Hugging Face credentials are read only from HF_TOKEN; downloaded bundles are cached beneath THEMIS_MODEL_PATH.

Runtime memory

OnnxDecisionInferenceProvider owns its ONNX session and artifact bundle and loads only the tokenizer required by the bundle contract. Dispose the provider when it is no longer needed. ONNX Runtime allocates native memory, so the managed heap alone does not describe inference memory use; compare process private bytes and working set with managed heap measurements.

Implementing custom model support

Themis' built-in ONNX provider is not a universal ONNX graph runner. ONNX manifest schema version 1 accepts these contracts:

• typed-heads: BPE tokenizer, token IDs and attention mask (optional token type IDs), fixed sequence length, and separate choice, score, and noul logits.
• transformers-sequence-classification-v1: BPE tokenizer, token IDs and attention mask (optional token type IDs), fixed sequence length, label count, and one logits output. Themis maps the declared labels to typed answers.
• rlcd-v1: token IDs, attention mask, marker positions/mask, question type, decision/action logits, sequence length, and maximum option count.

The loader accepts legacy contractType: "laya-v1" manifests and the layaModelContract property. New bundles declare rlcdContractType: "rlcd-v1" and rlcdModelContract; the built-in packer also writes the legacy fields so older consumers can still read the bundle.

All ONNX manifests identify the model and architecture, artifact paths, input and output shapes, tokenizer format, and files to validate. Each declared file may include a byte length and SHA-256 checksum. The loader currently requires the Hugging Face tokenizer JSON format with a BPE model and rejects unknown schema versions, runtimes, or contract types.

The built-in answer mapping is part of these contracts: choice logits become softmax probabilities; score logits become a probability distribution and an expected ordinal score; scalar noul logits use a sigmoid, while RLCD's two-class noul logits use softmax. RLCD bundles may provide temperature calibration by question type or question type and option-count bucket. Themis does not infer these semantics from arbitrary graph outputs. The built-in packers package/copy model artifacts and write manifests; they do not train or convert arbitrary models.

The separate RLCD GGUF bundle is also model-specific: it combines a GGUF backbone, a Hugging Face tokenizer/config, and a float32 safetensors head. The manifest declares backbone/head/tokenizer/config paths, encoder and head sequence lengths, hidden width, head layer count, and optional calibration temperatures. The .NET head implementation expects its defined tensor names and shapes, three question-type embeddings, attention heads of width 64, and a four-times-expanded feed-forward layer. These values and tensors are not a generic GGUF extension contract.

For a model that fits one of the supported contracts, an author can implement an IModelPacker and register it by format key with the Core packer registry. For a different tokenizer, graph contract, preprocessing rule, or answer mathematics, implement an IDecisionInferenceProvider and its own bundle reader/validator as needed. Register custom services through the AddThemis configureServices callback; set ThemisOptions.DefaultModel to the model ID your provider expects. Existing abstractions are extension points, not a promise that the built-in manifest loader will accept arbitrary fields.

For an ONNX model using an additional ONNX Runtime execution provider, register an IInferenceRuntimeInitializer that makes that provider available and an IOnnxExecutionProviderPolicy for its selection priority, provider options, session settings, and any worker-thread or serialized-run requirements. Add the policy through AddThemisOnnxExecutionProviderPolicy<TPolicy>(); the composition root passes registered policies to local and remote ONNX providers. Use PreferAccelerator, RequireAccelerator, or RequireGpu when the policy must explicitly select a discovered device; RuntimeDefault deliberately leaves selection and session creation to ONNX Runtime. The built-in DirectML policy preserves DirectML's current session and serialized-run requirements. Unrecognized execution providers use the generic ONNX session defaults; they are not automatically validated or supported by Themis.

For Intel GPU/NPU execution, add the optional Themis.Inference.OpenVINO package and call AddThemisOpenVINO(). It registers the Intel OpenVINO plugin and applies its session policy while reusing the existing ONNX bundle and answer contracts. The default configuration requires an accelerator; use PreferAccelerator only when CPU fallback is acceptable. The upstream plugin package provides Windows x64 and Linux x64 native assets. Validate model compatibility and actual device execution on the target hardware; package assets or provider registration alone are not evidence of accelerator support.

This boundary allows optional provider packages for technologies such as NVIDIA CUDA/TensorRT or Qualcomm QNN without putting their native dependencies in the base package. A vendor package should own its runtime registration, provider policy, platform/RID constraints, and hardware-specific tests. For a non-ONNX runtime, implement IDecisionInferenceProvider instead of forcing it through the ONNX policy interface.

services.AddThemis(
    options =>
    {
        options.DefaultModel = "vendor-model-v1";
        options.RouteUnspecifiedRequestsToDefaultModel = true;
    },
    configureServices: registrations =>
        registrations.AddSingleton<IDecisionInferenceProvider, VendorDecisionProvider>());

services.AddThemisModelPackers();
services.AddKeyedSingleton<IModelPacker, VendorModelPacker>("vendor-format-v1");

ModelPackRequest provides common source/output, model/tokenizer/config file, external-data, sequence-length, option-count, revision, and download controls. For additional author-specific settings, give the custom packer its own options/configuration and use a model-owned builder to create that request and the associated bundle.

An author may use a Builder pattern in their own package or tool to compose source discovery, model-specific packing, calibration, manifest writing, and provider registration. Themis does not require a generic Builder: its current extension seams are IModelPacker, IDecisionInferenceProvider, and IServiceCollection. A custom packer can receive additional author-specific options from DI rather than expanding the shared ModelPackRequest for unrelated model families.

License and model notices

The Themis inference code is licensed under MIT, and the NuGet package includes the Themis license text. Model weights and tokenizers are separate assets with their own terms; they are not relicensed by Themis or included in this NuGet package. The built-in packers copy root-level LICENSE, LICENCE, NOTICE, COPYING, and COPYRIGHT files from a source repository when present (including common extensions and suffixes). Check the upstream terms yourself and supply any required notices that are not available as root-level files before redistributing a bundle.