Mostlylucid.StyloFlow.Retrieval.Documents 2.4.0

Mostlylucid.StyloFlow

Declarative orchestration infrastructure with YAML-driven configuration, blackboard pattern, and signal-based coordination. Built on mostlylucid.ephemeral.

Installation

dotnet add package mostlylucid.styloflow

Quick Start

1. Create a YAML Manifest

# manifests/detectors/mydetector.detector.yaml
name: MyDetector
priority: 50
enabled: true
description: Detects specific patterns

taxonomy:
  kind: sensor
  determinism: deterministic
  persistence: ephemeral

triggers:
  requires:
    - signal: request.headers.present
    - signal: request.ip.present
  skip_when:
    - verified.human

emits:
  on_start:
    - detector.mydetector.started
  on_complete:
    - key: detector.mydetector.confidence
      type: double
      confidence_range: [0.0, 1.0]
  on_failure:
    - detector.mydetector.failed

defaults:
  weights:
    base: 1.0
    bot_signal: 1.2
    human_signal: 0.9
  confidence:
    neutral: 0.0
    bot_detected: 0.3
    high_threshold: 0.7
  timing:
    timeout_ms: 100
  features:
    detailed_logging: false
    enable_cache: true
  parameters:
    custom_threshold: 0.5
    patterns:
      - "pattern1"
      - "pattern2"

2. Register Services

// File system manifests
services.AddStyloFlow(
    manifestDirectories: new[] { "manifests" },
    configSectionPath: "Components");

// Or embedded resource manifests
services.AddStyloFlowFromAssemblies(
    sourceAssemblies: new[] { typeof(MyDetector).Assembly },
    manifestPattern: ".yaml",
    configSectionPath: "Components");

3. Create a Configured Component

public class MyDetector : ConfiguredComponentBase
{
    public MyDetector(IConfigProvider configProvider)
        : base(configProvider) { }

    public async Task<double> DetectAsync(RequestContext context)
    {
        // Use configured values - no magic numbers!
        var threshold = GetParam("custom_threshold", 0.5);
        var patterns = GetStringListParam("patterns");

        // Use weight shortcuts
        var weight = WeightBase;

        // Use confidence shortcuts
        if (someCondition)
            return ConfidenceBotDetected * weight;

        return ConfidenceNeutral;
    }
}

Configuration Hierarchy

StyloFlow resolves configuration with a three-tier hierarchy:

1. appsettings.json (highest priority) - Runtime overrides
2. YAML manifest - Default values per component
3. Code defaults - Fallback values

// appsettings.json - overrides YAML defaults
{
  "Components": {
    "MyDetector": {
      "Weights": {
        "Base": 1.5
      },
      "Parameters": {
        "custom_threshold": 0.7
      }
    }
  }
}

Core Concepts

ComponentManifest

The YAML manifest defines everything about a component:

ConfiguredComponentBase

Base class providing shortcuts to manifest configuration:

// Weight shortcuts
protected double WeightBase;
protected double WeightBotSignal;
protected double WeightHumanSignal;
protected double WeightVerified;
protected double WeightEarlyExit;

// Confidence shortcuts
protected double ConfidenceNeutral;
protected double ConfidenceBotDetected;
protected double ConfidenceHumanIndicated;
protected double ConfidenceStrongSignal;
protected double ConfidenceHighThreshold;
protected double ConfidenceLowThreshold;

// Timing shortcuts
protected int TimeoutMs;
protected int CacheRefreshSec;

// Feature shortcuts
protected bool DetailedLogging;
protected bool CacheEnabled;
protected bool CanEarlyExit;
protected bool CanEscalate;

// Parameter access
protected T GetParam<T>(string name, T defaultValue);
protected IReadOnlyList<string> GetStringListParam(string name);
protected bool IsFeatureEnabled(string featureName);

IManifestLoader

Load manifests from different sources:

• FileSystemManifestLoader - Watch directories for YAML files
• EmbeddedManifestLoader - Load from assembly embedded resources

IConfigProvider

Resolve configuration with hierarchy:

public interface IConfigProvider
{
    ComponentManifest? GetManifest(string componentName);
    ComponentDefaults GetDefaults(string componentName);
    T GetParameter<T>(string componentName, string parameterName, T defaultValue);
    IReadOnlyDictionary<string, ComponentManifest> GetAllManifests();
}

Entity Types

Entity types define the input/output contracts for atoms - what data they consume and produce. This enables type-safe orchestration, validation, and documentation.

Built-in Entity Types

StyloFlow includes built-in entity types for common scenarios:

Defining Entity Types in YAML

Create domain-specific entity types in *.entity.yaml files:

# manifests/entities/myapp.entity.yaml
entity_types:
  - type: myapp.document
    category: myapp
    description: Application document with metadata
    persistence: database
    storage_hint: documents_table
    signal_patterns:
      - document.id
      - document.content
      - document.metadata.*
    schema:
      format: inline
      inline:
        type: object
        required: [id, content]
        properties:
          id:
            type: string
          content:
            type: string
          metadata:
            type: object

  - type: myapp.embedding
    category: myapp
    description: Document embedding for search
    persistence: embedded
    vector_dimension: 1536
    signal_patterns:
      - embedding.vector
      - embedding.document_id

Entity Persistence Hints

Entity types can specify how they're typically stored:

Multi-Vector Embeddings

For ColBERT-style late interaction or multi-aspect embeddings:

- type: myapp.multivector_embedding
  category: myapp
  description: Multi-vector embedding with named components
  persistence: embedded
  signal_patterns:
    - embedding.vectors
    - embedding.fingerprint
  schema:
    format: inline
    inline:
      type: object
      required: [vectors, fingerprint]
      properties:
        vectors:
          type: array
          items:
            type: object
            required: [name, vector]
            properties:
              name:
                type: string
                description: Vector identifier (e.g., "title", "content")
              vector:
                type: array
                items:
                  type: number
              description:
                type: string
              weight:
                type: number
                default: 1.0
        fingerprint:
          type: string
        aggregation:
          type: string
          enum: [maxsim, avgpool, concat]
          default: maxsim

Input/Output Contracts in Manifests

Reference entity types in component manifests:

# Input contract - what this component consumes
input:
  accepts:
    - type: http.request
      required: true
      description: HTTP request for analysis
      signal_pattern: request.headers.*
    - type: behavioral.signature
      required: false
      description: Optional behavioral data
      signal_pattern: behavioral.*

  required_signals:
    - request.headers.user-agent
    - request.ip

  optional_signals:
    - request.headers.accept-language

# Output contract - what this component produces
output:
  produces:
    - type: detection.contribution
      description: Analysis contribution
    - type: myapp.embedding
      description: Generated embedding

  signals:
    - key: analysis.confidence
      entity_type: number
      salience: 0.8
      description: Analysis confidence score

EntityTypeRegistry

Register and query entity types programmatically:

var registry = new EntityTypeRegistry();

// Register custom types
registry.Register(new EntityTypeDefinition
{
    Type = "myapp.custom",
    Category = "myapp",
    Description = "Custom entity type",
    Persistence = EntityPersistence.Json,
    SignalPatterns = ["custom.*"]
});

// Query types
var httpTypes = registry.GetByPattern("http.*");
var isValid = registry.Validate("myapp.custom", myObject);

EntityTypeLoader

Load entity types from YAML files or embedded resources:

var loader = new EntityTypeLoader(logger);

// From directory
var types = loader.LoadFromDirectory("manifests/entities", "*.entity.yaml");

// From embedded resources
var types = loader.LoadFromAssembly(typeof(MyApp).Assembly, ".entity.yaml");

foreach (var type in types)
{
    registry.Register(type);
}

YAML Schemas and Validation

StyloFlow provides JSON Schema definitions for validating YAML manifest files. These schemas enable IDE autocompletion, validation, and documentation.

Available Schemas

Using Schemas with VS Code

Add a .vscode/settings.json to your project:

{
  "yaml.schemas": {
    "./node_modules/styloflow/schemas/component-manifest.schema.json": [
      "**/*.detector.yaml",
      "**/*.sensor.yaml",
      "**/*.contributor.yaml"
    ],
    "./node_modules/styloflow/schemas/entity-types.schema.json": [
      "**/*.entity.yaml"
    ]
  }
}

Or reference directly in the YAML file:

# yaml-language-server: $schema=https://styloflow.dev/schemas/component-manifest.schema.json
name: MyDetector
priority: 50
# ... rest of manifest

Manifest File Naming Conventions

Schema Structure Overview

Component Manifest Schema

# Core identification
name: string               # Required: Component identifier
priority: integer          # Execution order (lower = earlier)
enabled: boolean          # Whether component is active
description: string       # Human-readable description

# Classification
scope:
  sink: string            # Signal namespace (e.g., 'botdetection')
  coordinator: string     # Coordinator name
  atom: string           # Atom identifier

taxonomy:
  kind: enum             # sensor | analyzer | proposer | gatekeeper | aggregator | emitter
  determinism: enum      # deterministic | probabilistic
  persistence: enum      # ephemeral | direct_read | direct_write | cached

# Input/Output contracts
input:
  accepts: []            # Entity types consumed
  required_signals: []   # Signals that must be present
  optional_signals: []   # Signals that may enhance processing

output:
  produces: []           # Entity types produced
  signals: []            # Signals emitted (key, entity_type, salience, description)

# Execution control
triggers:
  requires: []           # Conditions to run
  skip_when: []         # Conditions to skip
  when: []              # Additional expressions

emits:
  on_start: []          # Signals on start
  on_complete: []       # Signals on completion
  on_failure: []        # Signals on failure
  conditional: []       # Conditional signals

listens:
  required: []          # Required upstream signals
  optional: []          # Optional upstream signals

# Resource management
lane:
  name: enum            # fast | normal | slow | llm | io
  max_concurrency: int  # Maximum parallel executions
  priority: int         # Priority within lane

budget:
  max_duration: string  # TimeSpan format (00:00:30)
  max_tokens: int       # For LLM components
  max_cost: number      # Cost limit in dollars

# Configuration
config:
  bindings: []          # Configuration key bindings

defaults:
  weights: {}           # Weight values
  confidence: {}        # Confidence thresholds
  timing: {}            # Timing parameters
  features: {}          # Feature flags
  parameters: {}        # Custom parameters

tags: []                # Categorization tags

Entity Types Schema

namespace: string        # Prefix for all types (e.g., 'botdetection')
version: string         # Schema version
description: string     # Collection description

entity_types:
  type_name:
    base_type: enum     # string | number | boolean | object | array | embedded_vector | multivector
    description: string
    persistence: enum   # ephemeral | json | database | embedded | file | cached
    nullable: boolean
    default_value: any

    # For validation
    validation:
      min: number
      max: number
      min_length: int
      max_length: int
      pattern: string   # Regex
      enum: []          # Allowed values
      format: string    # email | uri | ipv4 | etc.

    # For objects
    properties: {}

    # For embeddings
    embedding:
      dimensions: int
      model: string
      distance_metric: enum  # cosine | euclidean | dot_product
      normalize: boolean

    # For multi-vector
    vectors:
      - name: string
        dimensions: int
        description: string
        weight: number
        model: string
        source_field: string

    signal_pattern: string   # For composing from signals
    storage_hint: string     # sqlite | redis | qdrant | etc.
    tags: []

ManifestEntityValidator

Validate entity type references across manifests:

var validator = new ManifestEntityValidator(registry, loader, logger);

// Validate a single manifest
var result = validator.Validate(manifest);
if (!result.IsValid)
{
    foreach (var error in result.Errors)
    {
        Console.WriteLine($"Error: {error}");
    }
}

// Validate all manifests
var results = validator.ValidateAll(manifestLoader);
foreach (var r in results.Where(r => !r.IsValid))
{
    Console.WriteLine($"{r.ManifestName}: {r.Errors.Count} errors");
}

Signal-Based Orchestration

Trigger Conditions

triggers:
  requires:
    - signal: request.headers.present
      condition: ">"
      value: 0
    - signal: ip.analyzed
  skip_when:
    - verified.bot
    - verified.human
  when:
    - confidence < 0.5

Emitted Signals

emits:
  on_start:
    - detector.mydetector.started
  on_complete:
    - key: detector.mydetector.result
      type: enum
      description: Detection result
  conditional:
    - key: detector.mydetector.bot_detected
      type: bool
      when: confidence > 0.7
  on_failure:
    - detector.mydetector.failed

Escalation

escalation:
  targets:
    llm_analysis:
      when:
        - signal: confidence
          condition: ">"
          value: 0.4
        - signal: confidence
          condition: "<"
          value: 0.7
      skip_when:
        - signal: budget.exhausted
      description: Forward ambiguous cases to LLM

Taxonomy Classification

taxonomy:
  kind: sensor          # sensor, aggregator, escalator, enricher
  determinism: deterministic  # deterministic, probabilistic, learning
  persistence: ephemeral      # ephemeral, cached, persisted

Execution Lanes

lane:
  name: fast           # fast, standard, expensive
  max_concurrency: 4
  priority: 50

Budget Constraints

budget:
  max_duration: "00:00:30"
  max_tokens: 1000
  max_cost: 0.01

Architecture

                    ┌─────────────────┐
                    │  appsettings    │
                    │   overrides     │
                    └────────┬────────┘
                             │
                             ▼
┌──────────────┐    ┌─────────────────┐    ┌──────────────┐
│    YAML      │───▶│  ConfigProvider │◀───│     Code     │
│  Manifests   │    │   (resolves)    │    │   Defaults   │
└──────────────┘    └────────┬────────┘    └──────────────┘
                             │
                             ▼
                    ┌─────────────────┐
                    │  Configured     │
                    │  ComponentBase  │
                    └────────┬────────┘
                             │
              ┌──────────────┼──────────────┐
              │              │              │
              ▼              ▼              ▼
        ┌──────────┐  ┌──────────┐  ┌──────────┐
        │Detector 1│  │Detector 2│  │Detector N│
        └──────────┘  └──────────┘  └──────────┘

Licensing Module

StyloFlow includes a comprehensive licensing system for commercial deployments with tiered features, work unit metering, and mesh coordination.

Quick Start

// Full licensing with System Coordinator
services.AddStyloFlow(options =>
{
    options.LicenseToken = myLicenseJson;
    options.HeartbeatInterval = TimeSpan.FromSeconds(30);
});

// Free tier (development/testing)
services.AddStyloFlowFree();

// Mesh mode for distributed deployments
services.AddStyloFlowMesh(new[] { "node1:5200", "node2:5200" });

License Tiers

Licensed Components

Create license-aware components that automatically validate tier requirements:

public class DocumentProcessor : LicensedComponentBase
{
    public DocumentProcessor(
        ILicenseManager licenseManager,
        IWorkUnitMeter workUnitMeter,
        SignalSink signalSink)
        : base(licenseManager, workUnitMeter, signalSink, new LicenseRequirements
        {
            MinimumTier = "starter",
            WorkUnits = 2.0,
            WorkUnitsPerKb = 0.5,
            RequiredFeatures = new[] { "documents.*" },
            AllowFreeTierDegradation = true
        })
    { }

    public override string ComponentId => "styloflow.documents.processor";

    public async Task<Result> ProcessAsync(byte[] document)
    {
        // Check if we can perform the operation
        if (!CanPerformOperation(document.Length))
            return Result.Throttled();

        // Do the work...
        var result = await DoProcessingAsync(document);

        // Record work units
        RecordWorkUnits(document.Length);

        // Emit completion signal
        EmitSignal("processed", document.Length.ToString());

        return result;
    }
}

Work Unit Metering

Track and throttle resource consumption with sliding window metering:

var meter = provider.GetRequiredService<IWorkUnitMeter>();

// Record work
meter.Record(10.0, "documents");

// Check capacity
if (meter.CanConsume(50.0))
{
    // Proceed with operation
}

// Get throttle factor (1.0 = full speed, 0.0 = throttled)
var factor = meter.ThrottleFactor;

// Subscribe to threshold events
meter.ThresholdCrossed += (_, evt) =>
{
    Console.WriteLine($"Threshold {evt.ThresholdPercent}% crossed");
};

System Signals

The SystemCoordinator emits these signals for orchestration:

YAML License Configuration

Configure licensing in component manifests:

license:
  tier: starter
  features:
    - documents.*
    - images.*
  work_units:
    base: 2.0
    per_kb: 0.5
  requires_system: true
  allow_degradation: true

signals:
  defer_on:
    - styloflow.system.ready
  resume_on:
    - styloflow.system.ready
  free_tier:
    - component.degraded
  licensed:
    - component.ready

For detailed licensing documentation, see src/StyloFlow.Licensing/README.md.

Package Signing & Supply Chain Security

StyloFlow includes sfsign, a package signing tool that provides Ed25519-based cryptographic signatures for supply chain integrity.

Installation

dotnet tool install -g Mostlylucid.StyloFlow.Sign

Quick Start

# Generate a signing key
sfsign key generate --id "myvendor" --name "My Vendor Key" --keyring keyring.json

# Sign a package
sfsign sign --package mypackage.sfpkg --key "myvendor" --keyring keyring.json

# Verify a signed package
sfsign verify --package mypackage.sfpkg

Key Management

# Generate a new Ed25519 key pair
sfsign key generate --id "vendor.example" --name "Example Vendor" --keyring keyring.json

# List keys in a keyring
sfsign key list --keyring keyring.json

# Export public key for distribution (safe to share)
sfsign key export-public --id "vendor.example" --keyring keyring.json --output public.json

Signing Packages

# Sign with author type (package creator)
sfsign sign --package mypackage.sfpkg --key "myvendor" --type author --keyring keyring.json

# Add vendor signature (distribution)
sfsign sign --package mypackage.sfpkg --key "distributor" --type vendor --keyring keyring.json

# Add audit signature (third-party verification)
sfsign sign --package mypackage.sfpkg --key "auditor" --type audit --keyring keyring.json

Verification

# Basic verification (checks hash and signatures)
sfsign verify --package mypackage.sfpkg

# Verify with trust configuration
sfsign verify --package mypackage.sfpkg --trust trust.json

# Require specific signers
sfsign verify --package mypackage.sfpkg --require "myvendor" --require "auditor"

Trust Configuration

# Add a key to trusted roots
sfsign trust add --key "myvendor" --keyring keyring.json --config trust.json

# List trusted keys and cross-signings
sfsign trust list --config trust.json

Cross-Signing for Trust Chains

Cross-signing allows a trusted vendor to vouch for another vendor's key, creating trust chains without centralized authorities.

# Create cross-signing certificate (myvendor vouches for partner)
sfsign cross-sign \
  --issuer "myvendor" \
  --subject "partner.vendor" \
  --keyring keyring.json \
  --output trust.json

# Cross-sign with expiration
sfsign cross-sign \
  --issuer "myvendor" \
  --subject "partner.vendor" \
  --valid-to "2027-01-01" \
  --keyring keyring.json

Signature Types

Manifest Format

Signatures are stored in .sig.json files alongside packages:

{
  "version": "1.0",
  "packageId": "mypackage",
  "packageHash": "GSP4PaEWyjVhQ+071f+l...",
  "packageSize": 12345,
  "timestamp": "2026-01-08T18:30:00Z",
  "signatures": [
    {
      "signerId": "myvendor",
      "signerName": "My Vendor",
      "publicKey": "wE47MgQxtBez5t0z+Jgzs...",
      "signature": "jkvkKxSYGT9gNLdLIxUp...",
      "signedAt": "2026-01-08T18:30:00Z",
      "signatureType": "author"
    }
  ]
}

Security Model

• Ed25519: Fast, secure, deterministic signatures with 128-bit security
• SHA-256: Package integrity verification
• Multi-signature: Support for multiple independent signatures
• Cross-signing: Decentralized trust without certificate authorities
• No key escrow: Private keys never leave your control

Package Structure

License

This is free and unencumbered software released into the public domain - see UNLICENSE for details.

Documentation is actively being expanded. Check individual project README files for detailed API documentation.