feat(nisps/ml): MLP library with fixed-architecture template + spread-aware RL (meml-wmh)
Stream 2 of the clean-slate rewrite: nisps/ml/ replaces src/memlp/ with a
header-only, heap-free MLP that satisfies nisps::core::MLEngine.
Files (nisps/ml/):
- activations.hpp — ReLU (leaky 0.01 for parity), sigmoid, tanh
- loss.hpp — MSE per-sample (fixes meml-ues double-scaling: returns the
sample's MSE without an extra 1/N multiplication; the training loop
averages explicitly)
- init.hpp — uniform/Xavier/spread-aware weight init
- training.hpp — gradient clip helper (±10.0 matches legacy)
- rl.hpp — move_weights with per-layer Xavier scaling, weight decay
(10% * spread), gaussian noise via the deterministic Rng (matches the
legacy JS sum-of-three-uniforms shape); draw_weights also spread-aware
- stats.hpp — per-layer mean/max/dead/saturating diagnostics
- mlp.hpp — 4-layer (3 hidden + sigmoid output) MLP class with
std::array-backed weights, biases, gradient accumulators, dataset
ring buffer (default 128 examples), loss history (default 4096 iters).
Bias is a separate per-layer parameter — no input-vector mutation.
Flat get_weights/set_weights layout: weights all layers (row-major,
layer order), then biases all layers.
Tests (tests/cpp/, all 50 passing under -Wall -Wextra -Werror -Wpedantic):
- test_mlp_init.cpp — deterministic seeding, spread regimes,
static_assert MLEngine concept satisfied
- test_mlp_inference.cpp — golden hand-computed forward pass match,
sigmoid output range, set_input bounds
- test_mlp_training.cpp — XOR convergence (loss < 0.01 in <2k iters),
ring-buffer eviction
- test_mlp_loss.cpp — meml-ues regression test: reported loss equals
hand-computed average MSE without extra 1/N scaling; sample weights
honoured
- test_mlp_rl.cpp — move_weights respects output_pin_mask (final-layer
rows + biases preserved); spread regimes; grad clear after draw_weights
- test_mlp_serialize.cpp — get_weights/set_weights round-trip preserves
inference exactly; eval_loss is non-mutating; infer_batch matches
individual inference
Verification:
- Clean build, no warnings
- 50 tests pass (22 prior + 28 new)
- No std::vector / new / malloc in nisps/ml/
- All float literals .f-suffixed in code (comments excepted)
2026-04-29 14:55:43 +02:00
|
|
|
// nisps/ml/training.hpp — gradient-clipping helper and per-layer SGD update.
|
|
|
|
|
//
|
|
|
|
|
// The MLP class owns the training loop because it knows the dataset layout
|
|
|
|
|
// and weight buffers. This header provides:
|
|
|
|
|
// - kGradClip: ±10.0, matches the legacy firmware/Layer.h clamp.
|
|
|
|
|
// - clip_gradient(): scalar clipper, applied to each accumulated gradient
|
|
|
|
|
// before the weight update.
|
|
|
|
|
//
|
|
|
|
|
// The full training loop (forward, loss, backprop, weight update) is
|
|
|
|
|
// implemented inline in mlp.hpp because everything it touches is either a
|
|
|
|
|
// member array or layer-templated. Splitting it across translation units
|
|
|
|
|
// would require type-erasing the layers, which we don't want.
|
|
|
|
|
//
|
|
|
|
|
// Optimizer choice: this MVP ships SGD only. RMSProp is planned (the legacy
|
|
|
|
|
// firmware uses it for `TrainBatch`) but the playground was using plain SGD
|
|
|
|
|
// until very recently and the XOR-convergence benchmark in the test suite
|
2026-07-11 23:19:01 +02:00
|
|
|
// is the clearer target. RMSProp can land as a follow-up — see the Ergo
|
|
|
|
|
// task notes. For now `train()` is SGD with optional sample-weight
|
feat(nisps/ml): MLP library with fixed-architecture template + spread-aware RL (meml-wmh)
Stream 2 of the clean-slate rewrite: nisps/ml/ replaces src/memlp/ with a
header-only, heap-free MLP that satisfies nisps::core::MLEngine.
Files (nisps/ml/):
- activations.hpp — ReLU (leaky 0.01 for parity), sigmoid, tanh
- loss.hpp — MSE per-sample (fixes meml-ues double-scaling: returns the
sample's MSE without an extra 1/N multiplication; the training loop
averages explicitly)
- init.hpp — uniform/Xavier/spread-aware weight init
- training.hpp — gradient clip helper (±10.0 matches legacy)
- rl.hpp — move_weights with per-layer Xavier scaling, weight decay
(10% * spread), gaussian noise via the deterministic Rng (matches the
legacy JS sum-of-three-uniforms shape); draw_weights also spread-aware
- stats.hpp — per-layer mean/max/dead/saturating diagnostics
- mlp.hpp — 4-layer (3 hidden + sigmoid output) MLP class with
std::array-backed weights, biases, gradient accumulators, dataset
ring buffer (default 128 examples), loss history (default 4096 iters).
Bias is a separate per-layer parameter — no input-vector mutation.
Flat get_weights/set_weights layout: weights all layers (row-major,
layer order), then biases all layers.
Tests (tests/cpp/, all 50 passing under -Wall -Wextra -Werror -Wpedantic):
- test_mlp_init.cpp — deterministic seeding, spread regimes,
static_assert MLEngine concept satisfied
- test_mlp_inference.cpp — golden hand-computed forward pass match,
sigmoid output range, set_input bounds
- test_mlp_training.cpp — XOR convergence (loss < 0.01 in <2k iters),
ring-buffer eviction
- test_mlp_loss.cpp — meml-ues regression test: reported loss equals
hand-computed average MSE without extra 1/N scaling; sample weights
honoured
- test_mlp_rl.cpp — move_weights respects output_pin_mask (final-layer
rows + biases preserved); spread regimes; grad clear after draw_weights
- test_mlp_serialize.cpp — get_weights/set_weights round-trip preserves
inference exactly; eval_loss is non-mutating; infer_batch matches
individual inference
Verification:
- Clean build, no warnings
- 50 tests pass (22 prior + 28 new)
- No std::vector / new / malloc in nisps/ml/
- All float literals .f-suffixed in code (comments excepted)
2026-04-29 14:55:43 +02:00
|
|
|
// scaling and gradient clipping.
|
|
|
|
|
|
|
|
|
|
#pragma once
|
|
|
|
|
|
|
|
|
|
#include "../core/perf.hpp"
|
|
|
|
|
|
|
|
|
|
namespace nisps::ml {
|
|
|
|
|
|
|
|
|
|
// Per-element gradient clip threshold. Matches the legacy firmware's
|
|
|
|
|
// gradientClipValue in Layer.h::ApplyAccumulatedGradients.
|
|
|
|
|
inline constexpr float kGradClip = 10.f;
|
|
|
|
|
|
|
|
|
|
NISPS_FORCE_INLINE float clip_gradient(float g) noexcept {
|
|
|
|
|
if (g > kGradClip) return kGradClip;
|
|
|
|
|
if (g < -kGradClip) return -kGradClip;
|
|
|
|
|
return g;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
} // namespace nisps::ml
|