825ed6ad33
Stream 2 of the clean-slate rewrite: nisps/ml/ replaces src/memlp/ with a header-only, heap-free MLP that satisfies nisps::core::MLEngine. Files (nisps/ml/): - activations.hpp — ReLU (leaky 0.01 for parity), sigmoid, tanh - loss.hpp — MSE per-sample (fixes meml-ues double-scaling: returns the sample's MSE without an extra 1/N multiplication; the training loop averages explicitly) - init.hpp — uniform/Xavier/spread-aware weight init - training.hpp — gradient clip helper (±10.0 matches legacy) - rl.hpp — move_weights with per-layer Xavier scaling, weight decay (10% * spread), gaussian noise via the deterministic Rng (matches the legacy JS sum-of-three-uniforms shape); draw_weights also spread-aware - stats.hpp — per-layer mean/max/dead/saturating diagnostics - mlp.hpp — 4-layer (3 hidden + sigmoid output) MLP class with std::array-backed weights, biases, gradient accumulators, dataset ring buffer (default 128 examples), loss history (default 4096 iters). Bias is a separate per-layer parameter — no input-vector mutation. Flat get_weights/set_weights layout: weights all layers (row-major, layer order), then biases all layers. Tests (tests/cpp/, all 50 passing under -Wall -Wextra -Werror -Wpedantic): - test_mlp_init.cpp — deterministic seeding, spread regimes, static_assert MLEngine concept satisfied - test_mlp_inference.cpp — golden hand-computed forward pass match, sigmoid output range, set_input bounds - test_mlp_training.cpp — XOR convergence (loss < 0.01 in <2k iters), ring-buffer eviction - test_mlp_loss.cpp — meml-ues regression test: reported loss equals hand-computed average MSE without extra 1/N scaling; sample weights honoured - test_mlp_rl.cpp — move_weights respects output_pin_mask (final-layer rows + biases preserved); spread regimes; grad clear after draw_weights - test_mlp_serialize.cpp — get_weights/set_weights round-trip preserves inference exactly; eval_loss is non-mutating; infer_batch matches individual inference Verification: - Clean build, no warnings - 50 tests pass (22 prior + 28 new) - No std::vector / new / malloc in nisps/ml/ - All float literals .f-suffixed in code (comments excepted)
62 lines
2.5 KiB
C++
62 lines
2.5 KiB
C++
// nisps/ml/loss.hpp — MSE loss with a deliberate fix for the meml-ues
|
|
// double-scaling bug.
|
|
//
|
|
// THE BUG (in `src/memlp/Loss.h::MSE` and `src/memlp/MLP.cpp::Train`):
|
|
// The legacy MSE computed loss as
|
|
// sum((diff^2) / NOut) * sampleSizeReciprocal in Loss.h
|
|
// and then `Train()` did
|
|
// current_iteration_cost_function *= sampleSizeReciprocal; // ← second
|
|
// So the unweighted average loss was scaled by 1/N twice — once in MSE()
|
|
// and once in the outer training loop. Loss values for the same training
|
|
// set get reported 1/N too small, which is misleading for diagnostics.
|
|
//
|
|
// THE FIX:
|
|
// Loss is computed PER-SAMPLE here. The training loop is responsible for
|
|
// averaging across samples once. There is no implicit per-sample weight in
|
|
// the MSE function — that is the caller's policy decision.
|
|
//
|
|
// For a single sample: mse = (1/NOut) * sum_j (label_j - pred_j)^2
|
|
// Derivative wrt pred_j: -(2/NOut) * (label_j - pred_j)
|
|
//
|
|
// When sample weights are supplied to `train()` they are applied at the
|
|
// sample level (loss_total = sum_i weight_i * mse_i), and the gradient
|
|
// per-sample is scaled by weight_i. Sample weights must sum to 1.0 — the
|
|
// caller normalizes (matches legacy contract).
|
|
|
|
#pragma once
|
|
|
|
#include <cstddef>
|
|
#include <span>
|
|
|
|
#include "../core/perf.hpp"
|
|
|
|
namespace nisps::ml {
|
|
|
|
// Compute MSE loss for one sample and write the per-output derivative into
|
|
// `loss_deriv`. Returns the scalar loss for this sample.
|
|
//
|
|
// loss = (1/NOut) * sum_j (label_j - pred_j)^2
|
|
// d(loss)/d(pred_j) = -(2/NOut) * (label_j - pred_j)
|
|
//
|
|
// `loss_deriv` and `pred` must both have size NOut. `label` likewise.
|
|
//
|
|
// NB: We deliberately do NOT multiply by any sampleSizeReciprocal here. The
|
|
// caller is responsible for averaging across samples (and scaling by sample
|
|
// weights, if any) — see training.hpp.
|
|
NISPS_FORCE_INLINE float mse_per_sample(std::span<const float> label,
|
|
std::span<const float> pred,
|
|
std::span<float> loss_deriv) noexcept {
|
|
const std::size_t n = pred.size();
|
|
if (n == 0u) return 0.f;
|
|
const float inv_n = 1.f / static_cast<float>(n);
|
|
|
|
float accum = 0.f;
|
|
for (std::size_t j = 0; j < n; ++j) {
|
|
const float diff = label[j] - pred[j]; // sign chosen so deriv is wrt pred
|
|
accum += diff * diff * inv_n;
|
|
loss_deriv[j] = -2.f * inv_n * diff;
|
|
}
|
|
return accum;
|
|
}
|
|
|
|
} // namespace nisps::ml
|