f57cddc278
Upstream memlp (github.com/MusicallyEmbodiedML/memlp @ ea777502, the commit
upstream/main pins) applies gradients with RMSProp everywhere: Layer.h:239
ApplyAccumulatedGradients, the m_sq_grad_avg running average at Layer.h:601,
StaticMLP.h:268. nisps/ml/training.hpp shipped SGD only and filed the
difference as an optimiser-choice research question. It was not one.
RMSProp divides each step by the running gradient magnitude, so an upstream
lr is a NORMALISED step; under SGD the same number multiplies the raw
gradient. Every learning rate ported from upstream therefore landed in an
optimiser that reads it differently — most visibly feedback.hpp's
`geo_lr_ = 0.001f // upstream InterfaceRL.hpp:312`, an RMSProp LR pasted
into a single SGD step.
rmsprop_step() ports Layer.h:239 exactly: clip at +/-10, sq = min(0.9*sq +
0.1*g^2, 1e6), adj = min(lr/(sqrt(sq)+1e-6), 1.0), w -= adj*g. The
adjusted-LR clamp stays one-sided as upstream's std::min is, so the negative
lr used by train_targets' "train away from this target" path behaves as it
does upstream. The per-weight squared-gradient average is new persistent
state and lives in the storage policies (FixedStorage arrays /
DynamicStorage arena) so nisps/ stays allocation-free and the firmware's
zero-heap contract holds. It is optimiser state, not model state: excluded
from weight_count()/get_weights()/set_weights(), matching upstream, and
cleared by MLPCore::reset_optimizer_state() (upstream ResetOptimizerState).
draw_weights() deliberately does NOT clear it — upstream's DrawWeights
doesn't either.
Measured with tests/cpp/ml_bench.cpp:
D1 one geometric dislike moves the mapping 1.6e-2, up from 5.3e-5 (~295x),
and repeated presses now CONVERGE on the intended 0.5 push (0.12 at 10,
0.56 at 100) instead of creeping linearly forever.
A4 geometric-vs-Diffuse gap narrows from ~4100x to ~14x in one press.
U4 the upstream-LR positive path actually trains now (range_util 0.71 at
100 ticks/gesture, was 0.016 — it was inert under SGD).
Not fixed by this, and now tracked as ALIGNMENT defect 6d: the dose
asymmetry. lurch_max is still ~1.08 against a [0,1] output range.
Golden vector stages 2 and 3 re-captured; stages 0 and 1 are pre-training
and did not move, which is the cross-check that only the update rule
changed. manifold/public/nisps.wasm rebuilt so parity-check compares like
with like — it FAILED at up to 5e-2 against the stale artifact and PASSES at
2.4e-7 against a fresh one. parity-check.sh only builds the WASM when it is
missing, never when it is stale; noted in MAP.md and filed separately.
ALIGNMENT defect 6 resolved (moved to Recently resolved); 6b's optimiser
cross-reference updated; new defect 6d for the positive-training dose.
Gates: build-cpp-tests 138 tests / ctest 4/4, parity-check PASS, lint-cpp
clean, manifold typecheck clean.
644 lines
29 KiB
C++
644 lines
29 KiB
C++
// nisps/ml/mlp.hpp — four-layer MLP (three hidden + output), written ONCE
|
||
// against a storage policy (docs/specs/plans/one-core-engine-refactor.md P2).
|
||
//
|
||
// ARCHITECTURE
|
||
// MLPCore<Storage>
|
||
// ┌──────┐ Linear+Bias ┌────────┐ ReLU ┌────────┐ ReLU ┌────────┐ Sigmoid
|
||
// │ NIn │ ─────────────▶ │ NH1 │ ──────▶ │ NH2 │ ──────▶ │ NH3 │ ──────▶ NOut
|
||
// └──────┘ └────────┘ └────────┘ └────────┘
|
||
// Layer 0 (NIn → NH1) ReLU
|
||
// Layer 1 (NH1 → NH2) ReLU
|
||
// Layer 2 (NH2 → NH3) ReLU
|
||
// Layer 3 (NH3 → NOut) Sigmoid
|
||
//
|
||
// The topology (4 layers, ReLU×3 + Sigmoid) is fixed; the DIMENSIONS come
|
||
// from the storage policy:
|
||
//
|
||
// * `MLP<NIn, NH1, NH2, NH3, NOut, NMaxExamples, NMaxIterTrain>` — alias
|
||
// over `MLPCore<FixedStorage<...>>`. All buffers template-sized
|
||
// std::array, zero heap. This is the firmware model and preserves the
|
||
// pre-P2 class's exact compile-time surface (`kInput`, `kHidden1..3`,
|
||
// `kOutput`, `kNumLayers`, `weight_count()` — all constexpr).
|
||
// * `MLPCore<DynamicStorage>` — runtime-shaped (WASM/native-test/VCV
|
||
// only; heap at construction time, never per-call). Compile-time
|
||
// excluded from RP2350 builds.
|
||
//
|
||
// BIT-PARITY CONTRACT: for identical shapes and seeds the two storage models
|
||
// produce bit-identical results — the algorithm code below is shared and
|
||
// float op order is storage-independent. Enforced by
|
||
// tests/cpp/test_mlp_storage_parity.cpp.
|
||
//
|
||
// FLAT WEIGHT LAYOUT (`get_weights` / `set_weights`)
|
||
// [layer0_weights ...] [layer1_weights ...] [layer2_weights ...] [layer3_weights ...]
|
||
// [layer0_biases ...] [layer1_biases ...] [layer2_biases ...] [layer3_biases ...]
|
||
//
|
||
// CONCEPT SATISFACTION
|
||
// The class satisfies `nisps::MLEngine`: set_input, process, outputs,
|
||
// add_example, train (no-arg overload returning float), move_weights,
|
||
// draw_weights, reset, seed. Plus diagnostics: eval_loss, layer_stats,
|
||
// get/set_weights, weight_count, infer_batch, loss_history.
|
||
|
||
#pragma once
|
||
|
||
#include <cmath>
|
||
#include <cstddef>
|
||
#include <cstdint>
|
||
#include <span>
|
||
|
||
#include "../core/concepts.hpp"
|
||
#include "../core/perf.hpp"
|
||
#include "../core/rng.hpp"
|
||
#include "generated/ml_defaults.hpp"
|
||
#include "activations.hpp"
|
||
#include "init.hpp"
|
||
#include "loss.hpp"
|
||
#include "rl.hpp"
|
||
#include "stats.hpp"
|
||
#include "storage.hpp"
|
||
#include "training.hpp"
|
||
|
||
namespace nisps::ml {
|
||
|
||
// Per-instance training-hyperparameter config (S26, docs/specs/recon/
|
||
// simplification-audit-2026-07.md): the ONE learning_rate/max_iterations/
|
||
// min_error default used to be duplicated identically across all nine
|
||
// schemas/modes/*.json (unread at runtime), hardcoded again in this file's
|
||
// no-arg train() overload, again in manifold's wasm-iml.ts TS default
|
||
// parameters, and a FOURTH time (diverging: 0.1/200/0.00001) in vcv/src/
|
||
// iml.hpp. Default member initialisers below pull the single generated
|
||
// constant (schemas/ml_defaults.json -> nisps/ml/generated/ml_defaults.hpp)
|
||
// so every MLPCore instance — firmware, WASM handle, VCV adapter — starts
|
||
// pre-configured identically; `set_train_config` makes it runtime-overridable,
|
||
// same as the codebase-wide decision requires.
|
||
//
|
||
// Note on layering: ml_defaults.hpp is generated into nisps/ml/generated/, NOT
|
||
// alongside schema_types.hpp in nisps/modes/generated/ where the rest of the
|
||
// codegen output lives. Training hyperparameters are an ML fact, not a mode
|
||
// fact, and nisps/ml sits below nisps/modes — emitting them there would make
|
||
// this file include upward. The TS side has no equivalent layering to respect
|
||
// and keeps all generated output in one directory.
|
||
struct TrainConfig {
|
||
float learning_rate = ::nisps::ml::generated::kMlTrainDefaults.learning_rate;
|
||
std::size_t max_iterations = ::nisps::ml::generated::kMlTrainDefaults.max_iterations;
|
||
float min_error = ::nisps::ml::generated::kMlTrainDefaults.min_error;
|
||
};
|
||
|
||
// Activation of layer L in the fixed 4-layer topology.
|
||
template <std::size_t L>
|
||
inline constexpr Activation kLayerActivation =
|
||
(L == 3u) ? Activation::Sigmoid : Activation::ReLU;
|
||
|
||
template <typename Storage>
|
||
class MLPCore : public Storage {
|
||
public:
|
||
static constexpr std::size_t kNumLayers = kMlpNumLayers;
|
||
|
||
// ---------------------------------------------------------------
|
||
// Lifecycle. Extra arguments are forwarded to the storage policy —
|
||
// FixedStorage takes none (`MLP m(seed)`), DynamicStorage takes its
|
||
// runtime dimensions (`MLPCore<DynamicStorage> m(seed, n_in, hidden,
|
||
// n_out, ...)`).
|
||
// ---------------------------------------------------------------
|
||
template <typename... StorageArgs>
|
||
explicit MLPCore(std::uint64_t seed, StorageArgs&&... storage_args) noexcept
|
||
: Storage(static_cast<StorageArgs&&>(storage_args)...), rng_(seed) {
|
||
// Default-init weights with spread=1 (Xavier-like). The IML
|
||
// interface caller is expected to draw_weights() with its own
|
||
// spread before the first inference; this default simply gives a
|
||
// non-degenerate starting state for tests that skip an explicit
|
||
// draw.
|
||
if (!storage_ok_()) return;
|
||
draw_weights(1.f);
|
||
clear_dataset_();
|
||
loss_history_count_ = 0u;
|
||
}
|
||
|
||
// ---------------------------------------------------------------
|
||
// Inference API (concept: set_input / process / outputs)
|
||
// ---------------------------------------------------------------
|
||
NISPS_FORCE_INLINE void set_input(std::size_t i, float v) noexcept {
|
||
if (!storage_ok_()) return;
|
||
if (i < this->n_in()) this->input_buf()[i] = v;
|
||
}
|
||
|
||
NISPS_HOT void process() noexcept {
|
||
if (!storage_ok_()) return;
|
||
forward_(this->input_buf());
|
||
// Mirror final activation into the output buffer so callers can
|
||
// read a stable span.
|
||
const auto a = this->template act_l<3u>();
|
||
auto out = this->output_buf();
|
||
const std::size_t n_out = this->n_out();
|
||
for (std::size_t i = 0; i < n_out; ++i) out[i] = a[i];
|
||
}
|
||
|
||
NISPS_FORCE_INLINE std::span<const float> outputs() const noexcept {
|
||
return this->output_buf();
|
||
}
|
||
|
||
// ---------------------------------------------------------------
|
||
// Dataset / Training (concept: add_example / train)
|
||
// ---------------------------------------------------------------
|
||
// FIFO ring buffer; oldest example evicted when full. No allocation.
|
||
void add_example(std::span<const float> features,
|
||
std::span<const float> labels) noexcept {
|
||
if (!storage_ok_()) return;
|
||
const std::size_t n_in = this->n_in();
|
||
const std::size_t n_out = this->n_out();
|
||
if (features.size() < n_in || labels.size() < n_out) return;
|
||
|
||
std::size_t slot;
|
||
if (dataset_count_ < this->max_examples()) {
|
||
slot = dataset_count_++;
|
||
} else {
|
||
// Buffer full: overwrite the slot pointed at by head_ (oldest)
|
||
// and advance head_ to the next-oldest.
|
||
slot = dataset_head_;
|
||
dataset_head_ = (dataset_head_ + 1u) % this->max_examples();
|
||
}
|
||
auto dsf = this->ds_features();
|
||
auto dsl = this->ds_labels();
|
||
const std::size_t f_off = slot * n_in;
|
||
const std::size_t l_off = slot * n_out;
|
||
for (std::size_t i = 0; i < n_in; ++i) dsf[f_off + i] = features[i];
|
||
for (std::size_t i = 0; i < n_out; ++i) dsl[l_off + i] = labels[i];
|
||
}
|
||
|
||
// Concept-required no-arg overload. Reads the runtime-configurable
|
||
// `train_config_` (default-initialised from the single generated default;
|
||
// see `TrainConfig` above) rather than hardcoding numbers here.
|
||
float train() noexcept {
|
||
return train(train_config_.learning_rate, train_config_.max_iterations,
|
||
train_config_.min_error, std::span<const float>{});
|
||
}
|
||
|
||
// Runtime knob for the no-arg train() overload (S26). Does not affect the
|
||
// explicit-argument train() below, which stays the always-available
|
||
// explicit path.
|
||
void set_train_config(float lr, std::size_t max_iter, float min_err) noexcept {
|
||
train_config_.learning_rate = lr;
|
||
train_config_.max_iterations = max_iter;
|
||
train_config_.min_error = min_err;
|
||
}
|
||
const TrainConfig& train_config() const noexcept { return train_config_; }
|
||
|
||
// Full per-sample training (RMSProp — see training.hpp). `sample_weights`,
|
||
// if non-empty, must size to the
|
||
// current example count and sum to 1.0 (caller's responsibility — we
|
||
// do NOT renormalize).
|
||
//
|
||
// Returns final epoch loss. Records per-iteration loss in the loss
|
||
// history (bounded by max_iter_train()).
|
||
float train(float lr,
|
||
std::size_t max_iter,
|
||
float min_err,
|
||
std::span<const float> sample_weights = {}) noexcept {
|
||
loss_history_count_ = 0u;
|
||
if (!storage_ok_()) return 0.f;
|
||
if (dataset_count_ == 0u) return 0.f;
|
||
|
||
const bool weighted = !sample_weights.empty();
|
||
const float uniform_w = 1.f / static_cast<float>(dataset_count_);
|
||
|
||
auto loss_hist = this->loss_hist_buf();
|
||
|
||
float epoch_loss = 0.f;
|
||
for (std::size_t iter = 0; iter < max_iter; ++iter) {
|
||
epoch_loss = 0.f;
|
||
// Per-sample forward → loss → backprop+update. The order
|
||
// is the dataset insertion order; we do not shuffle (matches
|
||
// the legacy `Train()` exactly — `TrainBatch` shuffles, but
|
||
// we're not implementing batch yet).
|
||
for (std::size_t s = 0; s < dataset_count_; ++s) {
|
||
const float w = weighted ? sample_weights[s] : uniform_w;
|
||
|
||
// Forward pass on sample s.
|
||
std::span<const float> x = sample_features_(s);
|
||
forward_(x);
|
||
|
||
// Per-sample loss (NOT scaled by 1/N — the meml-ues fix).
|
||
// The eval scratch of the final layer doubles as the loss-
|
||
// derivative buffer (mse_per_sample fully overwrites it;
|
||
// eval_loss never runs concurrently).
|
||
auto deriv = this->template eval_act_l<3u>();
|
||
const float sample_loss = mse_per_sample(
|
||
sample_labels_(s),
|
||
std::span<const float>(this->template act_l<3u>()),
|
||
deriv);
|
||
|
||
// Aggregate weighted loss.
|
||
epoch_loss += w * sample_loss;
|
||
|
||
// Backprop with the same w as the gradient scaler.
|
||
backprop_(x, deriv, w);
|
||
|
||
// Apply gradient (per-sample, RMSProp).
|
||
apply_grad_<3u>(lr);
|
||
apply_grad_<2u>(lr);
|
||
apply_grad_<1u>(lr);
|
||
apply_grad_<0u>(lr);
|
||
}
|
||
|
||
if (loss_history_count_ < this->max_iter_train()) {
|
||
loss_hist[loss_history_count_++] = epoch_loss;
|
||
}
|
||
|
||
if (epoch_loss < min_err) break;
|
||
}
|
||
return epoch_loss;
|
||
}
|
||
|
||
// Train ONE step toward a COMPUTED target vector (not a stored label) —
|
||
// the geometric-dislike hook (docs/adr/rl-feedback-design.md §4). The
|
||
// dataset is untouched. A negative `lr` trains AWAY from the target (the
|
||
// upstream cold-start fallback). `out_mask` (1 = active) zeroes the
|
||
// loss-derivative of inactive output dims before backprop — the solo/
|
||
// focus gate; empty ⇒ all active. Returns the sample loss.
|
||
float train_targets(std::span<const float> input,
|
||
std::span<const float> target,
|
||
float lr,
|
||
std::span<const std::uint8_t> out_mask = {}) noexcept {
|
||
if (!storage_ok_()) return 0.f;
|
||
const std::size_t n_out = this->n_out();
|
||
if (input.size() < this->n_in() || target.size() < n_out) return 0.f;
|
||
|
||
forward_(input);
|
||
|
||
auto deriv = this->template eval_act_l<3u>();
|
||
const float loss = mse_per_sample(
|
||
target,
|
||
std::span<const float>(this->template act_l<3u>()),
|
||
deriv);
|
||
if (!out_mask.empty()) {
|
||
for (std::size_t j = 0; j < n_out; ++j) {
|
||
const bool active = (j < out_mask.size() && out_mask[j] != 0u);
|
||
if (!active) deriv[j] = 0.f;
|
||
}
|
||
}
|
||
|
||
backprop_(input, deriv, 1.f);
|
||
apply_grad_<3u>(lr);
|
||
apply_grad_<2u>(lr);
|
||
apply_grad_<1u>(lr);
|
||
apply_grad_<0u>(lr);
|
||
return loss;
|
||
}
|
||
|
||
// ---------------------------------------------------------------
|
||
// RL ops (concept: move_weights / draw_weights)
|
||
// ---------------------------------------------------------------
|
||
void move_weights(float speed, float spread,
|
||
std::span<const std::uint8_t> output_pin_mask = {}) noexcept {
|
||
if (!storage_ok_()) return;
|
||
move_weights_layer(this->template weights_l<0u>(), this->template biases_l<0u>(),
|
||
this->template fan_in_l<0u>(), speed, spread, /*final=*/false, {}, rng_);
|
||
move_weights_layer(this->template weights_l<1u>(), this->template biases_l<1u>(),
|
||
this->template fan_in_l<1u>(), speed, spread, /*final=*/false, {}, rng_);
|
||
move_weights_layer(this->template weights_l<2u>(), this->template biases_l<2u>(),
|
||
this->template fan_in_l<2u>(), speed, spread, /*final=*/false, {}, rng_);
|
||
move_weights_layer(this->template weights_l<3u>(), this->template biases_l<3u>(),
|
||
this->template fan_in_l<3u>(), speed, spread, /*final=*/true,
|
||
output_pin_mask, rng_);
|
||
}
|
||
|
||
void draw_weights(float spread) noexcept {
|
||
if (!storage_ok_()) return;
|
||
draw_weights_layer(this->template weights_l<0u>(), this->template biases_l<0u>(),
|
||
this->template fan_in_l<0u>(), spread, rng_);
|
||
draw_weights_layer(this->template weights_l<1u>(), this->template biases_l<1u>(),
|
||
this->template fan_in_l<1u>(), spread, rng_);
|
||
draw_weights_layer(this->template weights_l<2u>(), this->template biases_l<2u>(),
|
||
this->template fan_in_l<2u>(), spread, rng_);
|
||
draw_weights_layer(this->template weights_l<3u>(), this->template biases_l<3u>(),
|
||
this->template fan_in_l<3u>(), spread, rng_);
|
||
clear_grad_<0u>();
|
||
clear_grad_<1u>();
|
||
clear_grad_<2u>();
|
||
clear_grad_<3u>();
|
||
}
|
||
|
||
// Zero the RMSProp running squared-gradient averages (upstream
|
||
// `MLP<T>::ResetOptimizerState`, MLP.h:205). Note `draw_weights()`
|
||
// deliberately does NOT call this: upstream's `DrawWeights` leaves the
|
||
// optimiser state alone, so a randomise gesture keeps the step-size
|
||
// statistics it had. Only a full `reset()` clears them.
|
||
void reset_optimizer_state() noexcept {
|
||
if (!storage_ok_()) return;
|
||
clear_sq_grad_<0u>();
|
||
clear_sq_grad_<1u>();
|
||
clear_sq_grad_<2u>();
|
||
clear_sq_grad_<3u>();
|
||
}
|
||
|
||
// Concept reset: clear weights, dataset, loss history and optimiser
|
||
// state. Seed is intentionally NOT reset (use `seed()` for that).
|
||
void reset() noexcept {
|
||
if (!storage_ok_()) return;
|
||
clear_dataset_();
|
||
reset_optimizer_state();
|
||
loss_history_count_ = 0u;
|
||
// Re-init weights from current rng state with default spread.
|
||
draw_weights(1.f);
|
||
auto in = this->input_buf();
|
||
auto out = this->output_buf();
|
||
for (std::size_t i = 0; i < in.size(); ++i) in[i] = 0.f;
|
||
for (std::size_t i = 0; i < out.size(); ++i) out[i] = 0.f;
|
||
}
|
||
|
||
void seed(std::uint64_t s) noexcept { rng_.seed(s); }
|
||
|
||
// ---------------------------------------------------------------
|
||
// Diagnostics
|
||
// ---------------------------------------------------------------
|
||
// Average MSE across the training set without updating weights or the
|
||
// cached activations (runs through the mutable eval scratch).
|
||
float eval_loss() const noexcept {
|
||
if (!storage_ok_()) return 0.f;
|
||
if (dataset_count_ == 0u) return 0.f;
|
||
const float inv_n = 1.f / static_cast<float>(dataset_count_);
|
||
const std::size_t n_out = this->n_out();
|
||
auto dsl = this->ds_labels();
|
||
|
||
float total = 0.f;
|
||
for (std::size_t s = 0; s < dataset_count_; ++s) {
|
||
forward_eval_layer_<0u>(sample_features_(s));
|
||
forward_eval_layer_<1u>(this->template eval_act_l<0u>());
|
||
forward_eval_layer_<2u>(this->template eval_act_l<1u>());
|
||
forward_eval_layer_<3u>(this->template eval_act_l<2u>());
|
||
|
||
const auto ao = this->template eval_act_l<3u>();
|
||
float sse = 0.f;
|
||
const float inv_o = 1.f / static_cast<float>(n_out);
|
||
const std::size_t l_off = s * n_out;
|
||
for (std::size_t j = 0; j < n_out; ++j) {
|
||
const float d = dsl[l_off + j] - ao[j];
|
||
sse += d * d * inv_o;
|
||
}
|
||
total += sse * inv_n;
|
||
}
|
||
return total;
|
||
}
|
||
|
||
LayerStats layer_stats(std::size_t layer_idx) const noexcept {
|
||
if (!storage_ok_()) return {};
|
||
switch (layer_idx) {
|
||
case 0: return compute_layer_stats(this->template weights_l<0u>(),
|
||
this->template biases_l<0u>());
|
||
case 1: return compute_layer_stats(this->template weights_l<1u>(),
|
||
this->template biases_l<1u>());
|
||
case 2: return compute_layer_stats(this->template weights_l<2u>(),
|
||
this->template biases_l<2u>());
|
||
case 3: return compute_layer_stats(this->template weights_l<3u>(),
|
||
this->template biases_l<3u>());
|
||
default: return {};
|
||
}
|
||
}
|
||
|
||
// Returns a span into a storage-owned scratch buffer that holds a copy
|
||
// of the flat weights+biases. The buffer is regenerated on each call,
|
||
// so don't hold onto the span across mutations.
|
||
std::span<const float> get_weights() noexcept {
|
||
if (!storage_ok_()) return {};
|
||
auto flat = this->flat_buf();
|
||
std::size_t k = 0u;
|
||
// Weights, layer-major.
|
||
for (float v : this->template weights_l<0u>()) flat[k++] = v;
|
||
for (float v : this->template weights_l<1u>()) flat[k++] = v;
|
||
for (float v : this->template weights_l<2u>()) flat[k++] = v;
|
||
for (float v : this->template weights_l<3u>()) flat[k++] = v;
|
||
// Biases.
|
||
for (float v : this->template biases_l<0u>()) flat[k++] = v;
|
||
for (float v : this->template biases_l<1u>()) flat[k++] = v;
|
||
for (float v : this->template biases_l<2u>()) flat[k++] = v;
|
||
for (float v : this->template biases_l<3u>()) flat[k++] = v;
|
||
return std::span<const float>(flat.data(), k);
|
||
}
|
||
|
||
void set_weights(std::span<const float> w) noexcept {
|
||
if (!storage_ok_()) return;
|
||
if (w.size() < this->weight_count()) return;
|
||
std::size_t k = 0u;
|
||
for (float& v : this->template weights_l<0u>()) v = w[k++];
|
||
for (float& v : this->template weights_l<1u>()) v = w[k++];
|
||
for (float& v : this->template weights_l<2u>()) v = w[k++];
|
||
for (float& v : this->template weights_l<3u>()) v = w[k++];
|
||
for (float& v : this->template biases_l<0u>()) v = w[k++];
|
||
for (float& v : this->template biases_l<1u>()) v = w[k++];
|
||
for (float& v : this->template biases_l<2u>()) v = w[k++];
|
||
for (float& v : this->template biases_l<3u>()) v = w[k++];
|
||
}
|
||
|
||
// Run inference on N points (each n_in-sized) and write N output vectors
|
||
// (each n_out-sized) into `outs`. NO heap. Modifies the internal cached
|
||
// activations as a side effect.
|
||
void infer_batch(std::span<const float> points,
|
||
std::span<float> outs) noexcept {
|
||
if (!storage_ok_()) return;
|
||
const std::size_t n_in = this->n_in();
|
||
const std::size_t n_out = this->n_out();
|
||
const std::size_t n = points.size() / n_in;
|
||
if (outs.size() < n * n_out) return;
|
||
auto in = this->input_buf();
|
||
auto out = this->output_buf();
|
||
for (std::size_t i = 0; i < n; ++i) {
|
||
const std::size_t in_off = i * n_in;
|
||
for (std::size_t j = 0; j < n_in; ++j) in[j] = points[in_off + j];
|
||
process();
|
||
const std::size_t out_off = i * n_out;
|
||
for (std::size_t j = 0; j < n_out; ++j) outs[out_off + j] = out[j];
|
||
}
|
||
}
|
||
|
||
std::span<const float> loss_history() const noexcept {
|
||
return std::span<const float>(this->loss_hist_buf().data(), loss_history_count_);
|
||
}
|
||
|
||
std::size_t example_count() const noexcept { return dataset_count_; }
|
||
|
||
void clear_examples() noexcept { clear_dataset_(); }
|
||
|
||
private:
|
||
// ---------------------------------------------------------------
|
||
// Internal helpers
|
||
// ---------------------------------------------------------------
|
||
// DynamicStorage construction can fail (arena allocation); FixedStorage
|
||
// cannot. The check is compile-time `true` for storages without a
|
||
// `valid()` member, so the fixed/firmware path carries no branch.
|
||
NISPS_FORCE_INLINE bool storage_ok_() const noexcept {
|
||
if constexpr (requires(const Storage& s) { { s.valid() } -> std::convertible_to<bool>; }) {
|
||
return this->valid();
|
||
} else {
|
||
return true;
|
||
}
|
||
}
|
||
|
||
template <std::size_t L>
|
||
NISPS_HOT NISPS_FORCE_INLINE void forward_layer_(std::span<const float> in) noexcept {
|
||
const std::size_t fan_in = this->template fan_in_l<L>();
|
||
const std::size_t fan_out = this->template fan_out_l<L>();
|
||
auto w = this->template weights_l<L>();
|
||
auto b = this->template biases_l<L>();
|
||
auto pa = this->template pre_act_l<L>();
|
||
auto a = this->template act_l<L>();
|
||
for (std::size_t node = 0; node < fan_out; ++node) {
|
||
const std::size_t row = node * fan_in;
|
||
float sum = b[node];
|
||
for (std::size_t j = 0; j < fan_in; ++j) {
|
||
sum += w[row + j] * in[j];
|
||
}
|
||
pa[node] = sum;
|
||
a[node] = activate<kLayerActivation<L>>(sum);
|
||
}
|
||
}
|
||
|
||
NISPS_HOT NISPS_FORCE_INLINE
|
||
void forward_(std::span<const float> in) noexcept {
|
||
forward_layer_<0u>(in);
|
||
forward_layer_<1u>(this->template act_l<0u>());
|
||
forward_layer_<2u>(this->template act_l<1u>());
|
||
forward_layer_<3u>(this->template act_l<2u>());
|
||
}
|
||
|
||
// Backprop one layer: compute the incoming-error vector for the previous
|
||
// layer into delta_l<L>() and accumulate per-weight/per-bias gradients.
|
||
template <std::size_t L>
|
||
NISPS_HOT NISPS_FORCE_INLINE
|
||
void backprop_layer_(std::span<const float> input,
|
||
std::span<const float> upstream_err,
|
||
float sample_weight) noexcept {
|
||
const std::size_t fan_in = this->template fan_in_l<L>();
|
||
const std::size_t fan_out = this->template fan_out_l<L>();
|
||
auto w = this->template weights_l<L>();
|
||
auto pa = this->template pre_act_l<L>();
|
||
auto gw = this->template grad_w_l<L>();
|
||
auto gb = this->template grad_b_l<L>();
|
||
auto delta_in = this->template delta_l<L>();
|
||
|
||
for (std::size_t j = 0; j < fan_in; ++j) delta_in[j] = 0.f;
|
||
|
||
for (std::size_t node = 0; node < fan_out; ++node) {
|
||
const float err_signal =
|
||
upstream_err[node] *
|
||
activate_deriv_pre<kLayerActivation<L>>(pa[node]) * sample_weight;
|
||
const std::size_t row = node * fan_in;
|
||
for (std::size_t j = 0; j < fan_in; ++j) {
|
||
gw[row + j] += err_signal * input[j];
|
||
delta_in[j] += err_signal * w[row + j];
|
||
}
|
||
gb[node] += err_signal;
|
||
}
|
||
}
|
||
|
||
// Backprop with sample_weight applied to every error signal (so the
|
||
// accumulated gradient is already weighted). No weight update happens
|
||
// here — caller does it after each sample.
|
||
NISPS_HOT NISPS_FORCE_INLINE
|
||
void backprop_(std::span<const float> input,
|
||
std::span<const float> output_deriv,
|
||
float sample_weight) noexcept {
|
||
backprop_layer_<3u>(this->template act_l<2u>(), output_deriv, sample_weight);
|
||
backprop_layer_<2u>(this->template act_l<1u>(), this->template delta_l<3u>(), 1.f);
|
||
backprop_layer_<1u>(this->template act_l<0u>(), this->template delta_l<2u>(), 1.f);
|
||
backprop_layer_<0u>(input, this->template delta_l<1u>(), 1.f);
|
||
}
|
||
|
||
// Apply accumulated gradient to weights+biases via RMSProp (clip, advance
|
||
// the running squared-gradient average, normalise the step — see
|
||
// training.hpp for the ported formula and why it is not SGD). Resets the
|
||
// accumulators to zero for the next sample/iteration; the squared-gradient
|
||
// averages PERSIST, which is the whole point of the optimiser.
|
||
template <std::size_t L>
|
||
NISPS_FORCE_INLINE void apply_grad_(float lr) noexcept {
|
||
auto w = this->template weights_l<L>();
|
||
auto b = this->template biases_l<L>();
|
||
auto gw = this->template grad_w_l<L>();
|
||
auto gb = this->template grad_b_l<L>();
|
||
auto sw = this->template sq_grad_w_l<L>();
|
||
auto sb = this->template sq_grad_b_l<L>();
|
||
const std::size_t nw = gw.size();
|
||
const std::size_t nb = gb.size();
|
||
for (std::size_t i = 0; i < nw; ++i) {
|
||
w[i] -= rmsprop_step(gw[i], sw[i], lr);
|
||
gw[i] = 0.f;
|
||
}
|
||
for (std::size_t i = 0; i < nb; ++i) {
|
||
b[i] -= rmsprop_step(gb[i], sb[i], lr);
|
||
gb[i] = 0.f;
|
||
}
|
||
}
|
||
|
||
template <std::size_t L>
|
||
NISPS_FORCE_INLINE void clear_grad_() noexcept {
|
||
auto gw = this->template grad_w_l<L>();
|
||
auto gb = this->template grad_b_l<L>();
|
||
for (std::size_t i = 0; i < gw.size(); ++i) gw[i] = 0.f;
|
||
for (std::size_t i = 0; i < gb.size(); ++i) gb[i] = 0.f;
|
||
}
|
||
|
||
template <std::size_t L>
|
||
NISPS_FORCE_INLINE void clear_sq_grad_() noexcept {
|
||
auto sw = this->template sq_grad_w_l<L>();
|
||
auto sb = this->template sq_grad_b_l<L>();
|
||
for (std::size_t i = 0; i < sw.size(); ++i) sw[i] = 0.f;
|
||
for (std::size_t i = 0; i < sb.size(); ++i) sb[i] = 0.f;
|
||
}
|
||
|
||
// const forward pass for diagnostics — writes into the mutable eval
|
||
// scratch, never the real caches.
|
||
template <std::size_t L>
|
||
NISPS_FORCE_INLINE void forward_eval_layer_(std::span<const float> in) const noexcept {
|
||
const std::size_t fan_in = this->template fan_in_l<L>();
|
||
const std::size_t fan_out = this->template fan_out_l<L>();
|
||
auto w = this->template weights_l<L>();
|
||
auto b = this->template biases_l<L>();
|
||
auto out = this->template eval_act_l<L>();
|
||
for (std::size_t node = 0; node < fan_out; ++node) {
|
||
const std::size_t row = node * fan_in;
|
||
float sum = b[node];
|
||
for (std::size_t j = 0; j < fan_in; ++j) sum += w[row + j] * in[j];
|
||
out[node] = activate<kLayerActivation<L>>(sum);
|
||
}
|
||
}
|
||
|
||
NISPS_FORCE_INLINE std::span<const float> sample_features_(std::size_t s) const noexcept {
|
||
const std::size_t n_in = this->n_in();
|
||
return this->ds_features().subspan(s * n_in, n_in);
|
||
}
|
||
NISPS_FORCE_INLINE std::span<const float> sample_labels_(std::size_t s) const noexcept {
|
||
const std::size_t n_out = this->n_out();
|
||
return this->ds_labels().subspan(s * n_out, n_out);
|
||
}
|
||
|
||
void clear_dataset_() noexcept {
|
||
dataset_count_ = 0u;
|
||
dataset_head_ = 0u;
|
||
}
|
||
|
||
// ---------------------------------------------------------------
|
||
// Members (shape-independent; everything sized lives in Storage)
|
||
// ---------------------------------------------------------------
|
||
std::size_t dataset_count_ = 0u;
|
||
std::size_t dataset_head_ = 0u;
|
||
std::size_t loss_history_count_ = 0u;
|
||
TrainConfig train_config_{};
|
||
|
||
Rng rng_;
|
||
};
|
||
|
||
// The classic fixed-architecture MLP — the firmware model and the default
|
||
// everywhere a compile-time shape is known. `MLP<NIn, 10, 10, 14, NOut>`
|
||
// matches the legacy firmware default [10, 10, 14].
|
||
template <std::size_t NIn,
|
||
std::size_t NHidden1,
|
||
std::size_t NHidden2,
|
||
std::size_t NHidden3,
|
||
std::size_t NOut,
|
||
std::size_t NMaxExamples = kDefaultMaxExamples,
|
||
std::size_t NMaxIterTrain = 4096u>
|
||
using MLP = MLPCore<
|
||
FixedStorage<NIn, NHidden1, NHidden2, NHidden3, NOut, NMaxExamples, NMaxIterTrain>>;
|
||
|
||
// MLP satisfies the MLEngine concept. We keep a static_assert in the test
|
||
// suite (test_mlp_concept_satisfied) — see test_mlp_init.cpp.
|
||
|
||
} // namespace nisps::ml
|