Determinet

📦 Be sure to check out the NuGet package: https://www.nuget.org/packages/NTDLS.Determinet

Determinet is a versatile multilayer perceptron neural network for .NET, designed for ease of use and extendibility. In addition to the library, you'll find a test harness which includes a character recognition trainer, validator and visual testing tool. These provide working examples of training a model as well as generating predictions.

Features

• Fully-connected feed-forward networks with any number of hidden layers.
• 18 built-in activation functions (see below), each with a gradient-checked derivative.
• Cross-entropy loss for SoftMax outputs (classification), squared-error loss for everything else (regression).
• SGD or Adam (AdamW) optimization, single-sample or mini-batch training.
• Optional layer normalization, weight decay and global gradient-norm clipping.
• Activation-aware weight initialization (He, LeCun or Glorot).
• Named input/output labels, so you can work with "temperature" and "rain" instead of array indexes.
• Compact save/load of the whole model, including optimizer state, so training can be resumed exactly.
• Thread-safe inference: Forward() never modifies the network.
• Targets .NET 8, 9 and 10.

Installation

dotnet add package NTDLS.Determinet

Quick start

A network that learns XOR, a classic problem a single layer can't solve:

using NTDLS.Determinet;
using NTDLS.Determinet.Types;
using static NTDLS.Determinet.DniParameters;

// 2 inputs -> 8 hidden (Tanh) -> 2 outputs (SoftMax), with a label for each output.
var configuration = new DniConfiguration { LearningRate = 0.01 };
configuration.AddInputLayer(2);
configuration.AddIntermediateLayer(8, DniActivationType.Tanh);
configuration.AddOutputLayer(2, DniActivationType.SoftMax, ["false", "true"]);

var dni = new DniNeuralNetwork(configuration);
dni.Parameters.Set(Network.UseAdamOptimization, true);

// Each sample is (inputs, expected outputs). For classification the expected output is one-hot.
(double[] inputs, double[] expected)[] samples =
[
    ([0, 0], [1, 0]),
    ([0, 1], [0, 1]),
    ([1, 0], [0, 1]),
    ([1, 1], [1, 0]),
];

for (int epoch = 0; epoch < 500; epoch++)
{
    foreach (var (inputs, expected) in samples)
    {
        double loss = dni.Train(inputs, expected);
    }
}

Making predictions

// Raw outputs: for SoftMax these are probabilities that sum to 1.
double[] outputs = dni.Forward([1, 0]);
int index = outputs.IndexOfMaxValue(out double confidence);   // 1, ~0.99

// Or use the output labels.
dni.Forward([1, 0], out var labels);
var best = labels.Max();
Console.WriteLine($"{best.Key} ({best.Value:n2})");         // true (0.99)

Saving and loading

dni.SaveToFile("xor.dni");

var loaded = DniNeuralNetwork.LoadFromFile("xor.dni");

The file includes the weights, the layer configuration, all parameters and the optimizer state, so a loaded model can be used for predictions or trained further right where it left off.

Mini-batch training

Train() updates the weights after every sample. TrainBatch() averages the gradient over several samples and applies a single update, which gives smoother, less noisy training:

foreach (var batch in samples.Chunk(32))
{
    double averageLoss = dni.TrainBatch(batch);
}

Regression

For predicting real numbers, use an Identity output layer. The network is then trained with squared-error loss:

var configuration = new DniConfiguration { LearningRate = 0.01 };
configuration.AddInputLayer(1);
configuration.AddIntermediateLayer(16, DniActivationType.Tanh);
configuration.AddOutputLayer(1, DniActivationType.Identity);

var dni = new DniNeuralNetwork(configuration);
dni.Parameters.Set(Network.UseAdamOptimization, true);

for (int epoch = 0; epoch < 2000; epoch++)
{
    for (double x = -3; x <= 3; x += 0.25)
    {
        dni.Train([x], [Math.Sin(x)]);
    }
}

double y = dni.Forward([1.0])[0];   // approximately sin(1)

Named inputs

Give the input layer labels and you can supply values by name. Missing names are treated as 0.

var configuration = new DniConfiguration();
configuration.AddInputLayer(2, ["temperature", "humidity"]);
configuration.AddIntermediateLayer(8, DniActivationType.ReLU);
configuration.AddOutputLayer(2, DniActivationType.SoftMax, ["dry", "rain"]);

var dni = new DniNeuralNetwork(configuration);

var input = new DniNamedLabelValues();
input.Set("temperature", 0.7);
input.Set("humidity", 0.9);

dni.Forward(input, out var prediction);
Console.WriteLine(prediction.Max().Key);

Parameters

Network and layer behavior is controlled through named parameters.

// Network-wide settings.
dni.Parameters.Set(Network.LearningRate, 0.001);
dni.Parameters.Set(Network.UseAdamOptimization, true);
dni.Parameters.Set(Network.WeightDecay, 0.01);
dni.Parameters.Set(Network.GradientClip, 5.0);

// Per-layer settings are passed when the layer is added.
var hiddenParams = new DniNamedParameterCollection();
hiddenParams.Set(LeakyReLU.Alpha, 0.05);
hiddenParams.Set(Layer.UseLayerNorm, true);
configuration.AddIntermediateLayer(32, DniActivationType.LeakyReLU, hiddenParams);

// You can also store your own values in the model; they are saved with it.
dni.Parameters.Set("Epochs", 12);
int epochs = dni.Parameters.Get<int>("Epochs");

Built-in Activation Functions

Loss is chosen from the output layer: SoftMax and SimpleSoftMax outputs are trained with cross-entropy loss; every other output activation is trained with squared-error loss. SoftMax is only allowed on the output layer.

Test harness

The repository includes a complete handwritten character recognition example (digits plus upper and lower case letters):

• TestHarness.Train trains a model from a folder of labeled images, with augmentation (rotation, shift and blur), a learning-rate scheduler, early stopping and checkpointing.
• TestHarness.Validate measures a trained model's accuracy against a separate validation set.
• TestHarness.Draw lets you draw characters with the mouse and see the model's predictions live (pictured above).