DeepSharp.Pipelines 0.3.0

<img src="https://raw.githubusercontent.com/xkqg/DeepSharp/main/assets/icon.png" width="96" align="right" alt="" />

DeepSharp — deep learning in C#

The best of three worlds: TensorFlow's way of describing a network, PyTorch's way of running it, and ML.NET's way of learning from a table.

DeepSharp is the C# layer over the engines that already exist: you describe, train and use a network in C#, and the arithmetic runs on .NET's own vector maths out of the box or on a heavier engine later — swapping between them does not change a line of your model. What it adds is everything around the engine: getting your data in, the layers, the training loop, the checkpoints and the pictures. And a table that a tree learns better than a network does not have to become a network: the same prepared data is meant for ML.NET's trainers too.

0.3.0 is the tensors, the data half, and a notebook to see the data in and choose its columns — in Verso, in an application of your own, or in your browser from DeepSharp's own server. What learns from them is next; the roadmap says in which order, and the changelog records what each release added.

dotnet add package DeepSharp
dotnet add package DeepSharp.Pipelines

using DeepSharp.Pipelines;

var prepared = Pdd.Create()
    .ReadCsv("btceur-1d.csv")                                // declared, not opened
    .Declare(schema => schema
        .Timestamp("timestamp")
        .Number("close")
        .Optional("trades", ColumnKind.Number))              // a column that may have gaps
    .SplitByTime("timestamp", train: 0.70, validation: 0.15) // test is the rest
    .FillMissing("trades", With.Mean)                        // only offered after the split
    .Normalise("close")
    .Build()
    .Run();

The course from raw data to a validated model is declared once as an artefact and replayed, and anything that learns from the data is fitted on the training rows alone. That is the whole idea, and PDD is where it is explained.

The steps and what they learned are one file: prepared.ToJson() writes it, and PreparedData.FromJson(text, StepCatalog.BuiltIn()) reads it back in a program that has never seen the data. The catalog is the list of verbs the reader knows — add .WithIndicators() for a file that holds indicators, which read the rows in their order and so need that order said first, with .OrderBy("timestamp").

A notebook to see it in

DeepSharp.Verso.Notebooks writes the same pipeline as a Verso notebook, one block per step, each block the step's own JSON — edited as text, or field by field in Verso's properties panel. "Show the data here" on a block runs the pipeline down to it and shows the rows there, each column coloured over the training rows and every row marked with the part it lands in. A box on the grid leaves a column out — it turns black — or makes it a category, and the notebook writes the step that does it. "Choose the columns" lists every column of the source with its first values: tick it in or out, pick its kind, make it the answer and set the answer's own values — or tick a range, and seventy bands of a flock are taken in, or made the answer, with two ticks. What the blocks decide about their columns is saved beside the notebook, and the toolbar takes a saved file over again, listing every change before it makes one and every saved decision it cannot make. It also runs the whole pipeline and exports it as the same file the chain writes.

Install it from Verso's Extensions panel. The same package runs in Verso's VS Code extension, in the browser editor verso serve opens, and inside an application of your own, through DeepSharp.Verso.Api — the Notebook page says what each needs. Or let DeepSharp's own server show it in your browser, with nothing else to install:

dotnet tool install --global DeepSharp.Verso.Serve
deepsharp-serve titanic.verso

It listens on this computer alone and answers only the address it prints, token and all.

A C# cell in the same notebook reads what the blocks declare, as text — it is there after "Show the data here" or the toolbar's run, and taken back whenever the blocks may no longer make it:

#r "nuget: DeepSharp.Pipelines.Indicators"
using DeepSharp.Pipelines;

if (Variables.TryGet<string>("deepsharp.pipeline", out var text))
{
    var folder = Variables.TryGet<string>("deepsharp.folder", out var saved) ? SourceFolder.Of(saved) : SourceFolder.WorkingDirectory;
    var declaration = PipelineDeclaration.FromJson(text, StepCatalog.BuiltIn().WithIndicators());
    var prepared = new Pipeline(declaration, rows: null, folder).Run();
}

Read on

The packages

Runs on .NET 8 and .NET 10. MIT — see LICENSE. The page deepsharp-serve serves carries Mermaid and KaTeX, each under its own MIT licence, named in the tool's third-party notices.