Fuuga.Image.cpu 1.0.15

Fuuga.Image

This is image generation MCP client for Fuuga (or other agents) to make the agent capable of producing images via other (Stable Diffusion) models from for example HuggingFace.

Standalone CLI tool for image generation and captioning, built on stable-diffusion.cpp and Phi-3.5-vision.

CPU vs CUDA

The compute backend is decided by which package you install, not by a runtime flag:

• Fuuga.Image (NuGet) — CUDA natives (CUDA 12.8, win-x64; needs an NVIDIA GPU).
• Fuuga.Image.cpu (NuGet) — CPU natives, runs anywhere.

Building from source, the TorchBackend MSBuild property makes the same choice: dotnet build -p:TorchBackend=cuda for CUDA, plain dotnet build for the CPU default. The main library follows the identical split (Fuuga / Fuuga.cpu).

This applies to every subcommand: diffusion generation (stable-diffusion.cpp natives), captioning (ONNX Runtime GenAI natives), and the from-scratch pipeline (libtorch natives) all use CUDA in the CUDA package when a GPU is present, and CPU otherwise — no flags involved.

Commands

fuuga-image txt2img     Generate an image from a text prompt
fuuga-image img2img     Transform an image guided by a text prompt
fuuga-image txt2video   Generate a video from a text prompt (video models only)
fuuga-image img2video   Animate a still image into a video (video models only)
fuuga-image vid2vid     Edit/continue a video guided by text (VACE; video models only)
fuuga-image caption     Generate a text description of an image
fuuga-image scratch     Generate with the from-scratch SD 1.5 implementation (see below)
fuuga-image info        Display model metadata

Quick Start

dotnet build

# Text-to-image
dotnet run -- txt2img --model path/to/model.ckpt --prompt "a cat sitting on a windowsill" --steps 20

# Image-to-image
dotnet run -- img2img --model path/to/model.ckpt --source input.png --prompt "in watercolor style" --strength 0.75

# Text-to-video (requires a video model, e.g. a Wan2.1/2.2 T2V GGUF)
dotnet run -- txt2video --model path/to/wan-t2v.gguf --prompt "a wave crashing on rocks" --frames 16 --fps 8 --format mp4

# Image-to-video (animate a still)
dotnet run -- img2video --model path/to/wan-i2v.gguf --source photo.png --prompt "gentle zoom" --frames 24 --format gif

# Video-to-video (edit/continue a video guided by text; --source is a video file or a frames dir)
dotnet run -- vid2vid --model path/to/wan-vace.gguf --source clip.mp4 --prompt "make it snowy" --format mp4

# Attach an audio track to a generated mp4 (models don't make audio; this muxes a provided file)
dotnet run -- txt2video --model path/to/wan-t2v.gguf --prompt "a marching band" --format mp4 --audio track.mp3

# Caption an image
dotnet run -- caption --model path/to/vision-model/ --image photo.jpg --detail standard

# Caption a video (samples frames and describes them; --video is a file or a frames dir)
dotnet run -- caption --model path/to/vision-model/ --video clip.mp4 --frame-samples 8 --detail detailed

# Transcribe/describe audio (needs an audio-capable model, e.g. Phi-4-multimodal — same runtime)
dotnet run -- caption --model path/to/phi-4-multimodal/ --audio speech.wav

# Model info
dotnet run -- info --model path/to/model.ckpt

Common Options

txt2img Options

img2img Options

Same as txt2img, plus:

caption Options

Captioning uses whatever multimodal model directory you pass to --model. Provide one of --image (a still), --video (a video file decoded via ffmpeg, or a directory of frames, sampled to --frame-samples), or --audio (transcribe/describe an audio clip). Audio needs an audio-capable model such as Phi-4-multimodal — it runs on the same ONNX-Runtime-GenAI runtime already bundled (no extra dependency), you just point --model at it.

txt2video / img2video / vid2vid Options

Video is generated only when the loaded model supports it (e.g. Wan2.1/2.2 text-to-video, image-to-video, or VACE GGUF models); the tool checks the model's capability and errors clearly otherwise. img2video additionally takes --source <path> (the starting image); vid2vid takes --source <path> (a video file or a directory of frame images) plus --vace-strength <f>.

Video models produce no audio. --audio <path> muxes a user-provided audio file into an mp4/webp output via ffmpeg (ignored for gif / image sequences).

Output formats. png writes an image sequence (frame_0000.png …) into a per-run subdirectory — this is dependency-free and always available. mp4, gif, and webp are encoded from those frames with ffmpeg, which is an optional external tool (see below). If a single-file format is requested but ffmpeg is not found, the tool falls back to an image sequence and prints a note.

Optional: ffmpeg for mp4 / gif / webp

ffmpeg is not a .NET dependency and is not bundled. To enable single-file video output, install it and make sure it is on PATH (or point FUUGA_FFMPEG at the executable):

• Windows: winget install Gyan.FFmpeg (or choco install ffmpeg), or download from https://ffmpeg.org/download.html and add the bin folder to PATH.
• macOS: brew install ffmpeg
• Linux: sudo apt install ffmpeg (Debian/Ubuntu) or your distro's package manager.
• Explicit path: set FUUGA_FFMPEG=/full/path/to/ffmpeg to use a specific build.

Without ffmpeg, --format png (image sequence) still works and can be turned into a video later with any tool.

From-Scratch Stable Diffusion (scratch)

The scratch subcommand runs a complete Stable Diffusion 1.5 implementation written from first principles in F# on TorchSharp — no stable-diffusion.cpp, no Python. It implements the CLIP text encoder, the VAE encoder/decoder, the latent-diffusion UNet (with self/cross attention), CLIP's BPE tokenizer, and DDIM/DDPM samplers with classifier-free guidance, and loads the standard SD v1.x single-file safetensors checkpoint directly. It exists to be read, hacked, and extended (in the spirit of "stable diffusion from scratch" tutorials) — use txt2img/img2img for fast production generation.

You need two downloads:

1. The checkpoint — an SD v1.x single-file .safetensors, e.g. v1-5-pruned-emaonly-fp16.safetensors (2.1 GB; the fp32 v1-5-pruned-emaonly.safetensors also works).
2. The CLIP tokenizer tables — vocab.json + merges.txt from openai/clip-vit-large-patch14 into one directory.

# Text-to-image
dotnet run -- scratch --model models/v1-5-pruned-emaonly-fp16.safetensors \
    --clip-tokenizer models/clip-tokenizer \
    --prompt "a photograph of an astronaut riding a horse" \
    --steps 30 --cfg 7.5 --width 512 --height 512 --seed 42

# Image-to-image
dotnet run -- scratch --model models/v1-5-pruned-emaonly-fp16.safetensors \
    --clip-tokenizer models/clip-tokenizer \
    --prompt "in watercolor style" --image input.png --strength 0.7

Notes:

• The device follows the installed backend automatically: a CUDA build/package with a GPU present runs on CUDA, otherwise CPU. CPU generation is slow (the UNet runs in float32); a CUDA build is strongly recommended for full-size images. Small sizes are fine for CPU experiments — 64×64 runs in ~10 s and is ideal for pipeline smoke tests; 256×256 takes ~1–2 min.
• Any width/height that is a multiple of 8 works, including non-square aspect ratios (e.g. --width 128 --height 192). Quality is best at the model's native 512×512; far below it images become abstract, far above it content repeats.
• Progressive rendering: pass --preview-every N to write intermediate preview PNGs during denoising, or consume ScratchProgress.PreviewPng from the generate callback in library use.
• Also exposed as the fuuga_image_scratch MCP tool (via --mcp-server) and from the main Fuuga CLI as fuuga image --scratch <prompt> --model <safetensors> --clip-tokenizer <dir>.
• Library usage from F# scripts: see examples/scratch-sd-demo.fsx.
• Fine-tuning: ScratchSdPipeline.diffusionTrainingLoss implements the standard epsilon-prediction objective — noise VAE-encoded images to a random timestep and MSE-score the UNet's noise prediction. Pair it with torch.optim.AdamW over unet.parameters() to retrain, then export — train slow here, serve fast there. RL-style preference tuning (DDPO / reward-weighted) would build on the same loss and is not included.
• Export: ScratchSdWeights.exportSd15 writes a single-file F16 safetensors; exportSd15Gguf writes a stable-diffusion.cpp-compatible GGUF (F16 or Q8_0, ~1 byte/weight). Both are loadable by fuuga-image txt2img, ComfyUI, A1111, and back into this module (verified: sd.cpp generates from a scratch-exported Q8_0 GGUF). Formats deliberately NOT supported: .ckpt (Python pickle — unsafe to parse outside Python), diffusers multi-folder layout (different key naming; convert to single-file with any standard tool), and ONNX (TorchSharp has no graph tracer; ONNX SD serving is a different backend, not an export target).
• GroupNorm is implemented from primitive ops (ScratchGroupNorm): the fused TorchSharp/libtorch kernel on win-x64 CPU can return garbage in forward and silently zero input-gradients in backward, which would break training entirely.
• Module names deliberately mirror the checkpoint key layout (model.diffusion_model.*, first_stage_model.*, cond_stage_model.transformer.*), so weight loading is a prefix strip instead of a mapping table — and the key names are unit-tested against the real SD 1.5 layout.
• Source layout: ScratchSdTokenizer.fs (CLIP BPE) → ScratchSdModels.fs (CLIP/VAE/UNet) → ScratchSdWeights.fs (safetensors reader + loader) → ScratchSdPipeline.fs (DDPM sampler + txt2img/img2img pipeline).

MCP Server Mode

Fuuga.Image can run as an MCP tool server over stdio, enabling integration with Fuuga LLM via the MCP protocol:

dotnet run -- --mcp-server

This exposes these tools via JSON-RPC 2.0 over stdin/stdout:

• fuuga_image_txt2img -- text-to-image generation
• fuuga_image_img2img -- image-to-image transformation
• fuuga_image_txt2video -- text-to-video generation (video models only)
• fuuga_image_img2video -- image-to-video animation (video models only)
• fuuga_image_caption -- image captioning

See architecture.md Section 18 for the full integration design.

F# Script Demo

An interactive demo script shows how to use Fuuga.Image as an F# library — calling the modules directly with record-based configuration, validation, and I/O:

dotnet build Fuuga.Image
dotnet fsi examples/image-demo.fsx

The script runs without models (dry-run mode), exercising types, validation, image I/O, and prompt templates. Uncomment Section 5 and set your model paths for actual generation. See examples/image-demo.fsx for the full source.

Project Structure

Fuuga.Image/
  Types.fs                Layer 0: Domain types, error handling (ImageError DU)
  Config.fs               Layer 1: CLI argument parsing
  ImageIO.fs              Layer 2: Image load/save, format conversion, validation
  VideoIO.fs              Layer 2b: Video frame I/O (ffmpeg optional)
  ScratchSdTokenizer.fs   Layer 2c: CLIP BPE tokenizer (from-scratch SD)
  ScratchSdModels.fs      Layer 2c: CLIP/VAE/UNet architecture in TorchSharp
  ScratchSdWeights.fs     Layer 2c: safetensors reader + checkpoint loading
  ScratchSdPipeline.fs    Layer 2c: DDPM sampler + txt2img/img2img pipeline
  Diffusion.fs            Layer 3: Stable Diffusion model wrapper (txt2img, img2img)
  Caption.fs              Layer 4: Phi-3.5-vision captioning (ONNX Runtime GenAI)
  McpServer.fs            Layer 5: MCP JSON-RPC 2.0 server over stdio
  Program.fs              Layer 6: Entry point, subcommand routing, error handling

Architecture

• Image Extension Architecture -- standalone design
• Main Architecture Section 18 -- Fuuga LLM integration