Radiate 1.4.3

.NET port of Rust crate radiate.

Algorithms

1. Random Forest
2. Support Vector Machine
3. MultiLayer Perceptron
   • Dense Layer
   • Dropout Layer
   • Flatten Layer
   • LSTM Layer
   • RNN Layer
   • Convolutional Layer
   • MaxPooling Layer
4. KMeans Clustering
5. KNearestNeighbors
6. Evolution Engine
   • Evolve any object which implements the IGenome interface.
     • Implementation of NEAT for evolving NeuralNetworks is included.
     • Implementation of evolutionarily generated RandomForest/DecisionTree.

Callbacks

Similar to Keras Callbacks. Hook into the training loop with custom code. Create an object that implements any or all of the following:

1. IEpochStartedCallback
2. IBatchCompletedCallback
3. IEpochCompletedCallback
4. ITrainingCompletedCallback
5. IGenerationEvolvedCallback
6. IEarlyStopCallback
7. ILossCalculatedCallback
8. ITrainingFailedCallback
9. ITrainingStartedCallback

See this for example callbacks or this for a VerboseTrainingCallback to print training progress to the Console.

Feature Engineering

Functional feature engineering with TensorDataSet. Transform input data (featurs, targets) with specific options below.

1. Batch - Set a batch size to train on.
2. Layer - Layer data by n rows.
3. Split - Split the data into a training set and testing set. Default is 75% split training, 25% testing.
4. Reshape - Reshape the row vector to a shape of (height, width, depth), useful for images.
5. Pad - Pad an image Tensor with n zeros.
6. Shuffle - Shuffle the rows of the dataset randomly.
7. Kernel - Add kernel transform for the features, possible options are RBF, Polynomial, and Linear (None).
8. TransformFeatures - Transform the feature data. Options are Normalize, Standardize, OHE (One Hot Encode), and Image (divide data point by 255).
9. TransformTargets - Transform the target data. Options same as above.

Model saving and loading

The Optimizer is not Json serializable, but the OptimizerWrap is, so the Optimizer must be converted to a concrete object before serializing.

OptimizerWrap contains three items:

1. TensorDataSet options, the options used to transform the input features/targets. During predicion the Optimizer uses these options to transform the input vector so it matches the trained features in order to get accurate predictions.
2. LossFunction, the loss function used during training. If you save a model mid training, the loss function is needed when loading back in the model to continue training.
3. ModelWrap, the machine learning model being trained/used for prediction.

// Create an Optimizer
var forest = new RandomForest(numTrees, new ForestInfo(minSampleSplit, maxDepth));
var optimizer = new Optimizer(forest, tensorDataSet);

// Generate a json serializable object.
var wrappedModel = optimizer.Save();

// The same model can be loaded back in for prediction/additional training.
var newOptimizer = new Optimizer(wrappedModel);

The Optimizer can also be converted to a json string or a memory stream like so:

var optimizer = new Optimizer(forest, TensorDataSet);
var jsonString = ModelWriter.ToJson(optimizer);
var stream = ModelWriter.ToStream(optimizer);

// Loading in an Opimizer from the above options
var optimizer = ModelReader.FromJson(jsonString);
var optimizer = ModelReader.FromStream(stream);

Make predictions

Because the Optimizer has TensorDataSet, it can transform a given float[] to acceptable input to the model even after saving/loading a model. This makes making predictions as easy as

var vectorToPredict = new float[] { 1f, 2f, 3f, 4f, 5f };
var prediction = trainedOptimizer.Predict(vectorToPredict);

// Prediction(Tensor Result, int Classification = 0, float Confidence = 0f);

The prediction created returns a Prediction record which holds:

1. Result, the Model's predicted output vector
2. Classification value.
3. Confidence value which can be used as the regression value.

Random

All random numbers are generated by calling RandomGenerator.Next. This allows for random seeds to be set for model building.

RandomGenerator.Seed = 3;

Will set a random seed for the Optimizer.

Execution Strategies

Choose to execute model training & predicting synchronously or asyncronously. This is extremely helpful in models with large weight Tensors or large training dataset. However, choosing to execute asynchronously in certain situations can result in a slower training time than synchronous execution. It is best to experiment with this and choose the best option for your problem. The default option for every model is ExecutionStrategy.Sync.

// MultiLayerPerceptron
var lstm = new MultiLayerPerceptron(new GradientInfo(Gradient.Adam, 1e-4f), ExecutionStrategy.Async)
    .Add(new LSTMInfo(128)) // Because no ExecutionStrategy is defined, the layer inherits the MultiLayerPerceptron's ExecutionStrategy
    .Add(new DenseInfo(1, Activation.Linear)
        .WithExecutionStrategy(ExecutionStrategy.Sync)); // override the network's ExecutionStrategy in favor of executing just this layer synchronously

// RandomForest
var randomForest = new RandomForest(new ForestInfo(25, 2, 10), ExecutionStrategy.Async);

// SupportVectorMachine
var svm = new SupportVectorMachine(new SVMInfo(new Shape(28), 10, 1e-3f), ExecutionStrategy.Async);

// KNearestNeighbors
var knn = new KNN(new KNNInfo(3, ExecutionStrategy.Sync, DistanceMetric.Euclidean));

// Population - The Population takes PopulationSettings as an argument, so ExecutionStrategy is available for any IGenome implementation.
var info = new PopulationInfo<Neat, NeatEnvironment>()
    .AddSettings(settings => settings.ExecutionStrategy = ExecutionStrategy.Async);
var population = new Population<Neat, NeatEnvironment>(info);

Examples

Datasets coming from Radiate.Data which provides easily accessable common machine learning datasets.

Convolutional Neural Network on MNist handwritten digets dataset

<img src="https://camo.githubusercontent.com/01c057a753e92a9bc70b8c45d62b295431851c09cffadf53106fc0aea7e2843f/687474703a2f2f692e7974696d672e636f6d2f76692f3051493378675875422d512f687164656661756c742e6a7067" width="300px">

const int FeatureLimit = 5000;
const int BatchSize = 128;
const int MaxEpochs = 10;

var (rawInputs, rawLabels) = await new Mnist(FeatureLimit).GetDataSet();

var data = new TensorDataSet(rawInputs, rawLabels)
   .Reshape(new Shape(28, 28, 1))
   .Transform(Norm.Image, Norm.OHE)
   .Batch(BatchSize)
   .Split();

var neuralNet = new MultilayerPerceptron(ExecutionStrategy.Async)
   .Layer(new ConvInfo(32, 3))
   .Layer(new MaxPoolInfo(2))
   .Layer(new FlattenInfo())
   .Layer(new DenseInfo(64, Activation.Sigmoid))
   .Layer(new DenseInfo(data.OutputCategories, Activation.SoftMax));

var epoch = neuralNet.Fit(data).ToEpoch(MaxEpochs);

Console.WriteLine($"{epoch.ToValidation()}");

Random Forest on Iris Flowers dataset

<img src="https://upload.wikimedia.org/wikipedia/commons/5/56/Iris_dataset_scatterplot.svg" width="300px">

const int NumTrees = 10;
const int MaxDepth = 10;
const int MinSampleSplit = 2;
 
var (rawFeatures, rawLabels) = dataSet;

var pair = new TensorDataSet(rawFeatures, rawLabels).Shuffle().Split();

var validation = new RandomForest(NumTrees, MinSampleSplit, MaxDepth).Fit(pair).ToValidation();

Console.WriteLine($"{validation}");

More

See the examples for how to use the API.