825ed6ad33
Stream 2 of the clean-slate rewrite: nisps/ml/ replaces src/memlp/ with a header-only, heap-free MLP that satisfies nisps::core::MLEngine. Files (nisps/ml/): - activations.hpp — ReLU (leaky 0.01 for parity), sigmoid, tanh - loss.hpp — MSE per-sample (fixes meml-ues double-scaling: returns the sample's MSE without an extra 1/N multiplication; the training loop averages explicitly) - init.hpp — uniform/Xavier/spread-aware weight init - training.hpp — gradient clip helper (±10.0 matches legacy) - rl.hpp — move_weights with per-layer Xavier scaling, weight decay (10% * spread), gaussian noise via the deterministic Rng (matches the legacy JS sum-of-three-uniforms shape); draw_weights also spread-aware - stats.hpp — per-layer mean/max/dead/saturating diagnostics - mlp.hpp — 4-layer (3 hidden + sigmoid output) MLP class with std::array-backed weights, biases, gradient accumulators, dataset ring buffer (default 128 examples), loss history (default 4096 iters). Bias is a separate per-layer parameter — no input-vector mutation. Flat get_weights/set_weights layout: weights all layers (row-major, layer order), then biases all layers. Tests (tests/cpp/, all 50 passing under -Wall -Wextra -Werror -Wpedantic): - test_mlp_init.cpp — deterministic seeding, spread regimes, static_assert MLEngine concept satisfied - test_mlp_inference.cpp — golden hand-computed forward pass match, sigmoid output range, set_input bounds - test_mlp_training.cpp — XOR convergence (loss < 0.01 in <2k iters), ring-buffer eviction - test_mlp_loss.cpp — meml-ues regression test: reported loss equals hand-computed average MSE without extra 1/N scaling; sample weights honoured - test_mlp_rl.cpp — move_weights respects output_pin_mask (final-layer rows + biases preserved); spread regimes; grad clear after draw_weights - test_mlp_serialize.cpp — get_weights/set_weights round-trip preserves inference exactly; eval_loss is non-mutating; infer_batch matches individual inference Verification: - Clean build, no warnings - 50 tests pass (22 prior + 28 new) - No std::vector / new / malloc in nisps/ml/ - All float literals .f-suffixed in code (comments excepted)
76 lines
2.5 KiB
C++
76 lines
2.5 KiB
C++
// nisps/ml/stats.hpp — per-layer weight-health diagnostics.
|
|
//
|
|
// Mirrors the `nisps_mlp_get_layer_stats` WASM binding semantics: for each
|
|
// layer, return mean(|w|), max(|w|), dead fraction, and saturating fraction.
|
|
// These are used by the playground "weight health" panel.
|
|
//
|
|
// Definitions:
|
|
// - mean_abs: average of |w| across all weights+biases of the layer
|
|
// - max_abs: max of |w| across all weights+biases
|
|
// - dead_frac: fraction of weights with |w| < kDeadThresh
|
|
// - saturating_frac: fraction with |w| > kSaturatingThresh
|
|
//
|
|
// Thresholds are chosen to match the JS engine's heuristic targets:
|
|
// dead: 0.001 (a weight smaller than this contributes negligibly)
|
|
// sat: 3.0 (weights larger than this typically push sigmoid to its
|
|
// rails — a smell during normal training but normal during
|
|
// RL exploration)
|
|
//
|
|
// We intentionally include biases in mean/max but not in dead/sat fractions
|
|
// — biases are scarce relative to weights and including them would skew the
|
|
// fractions, especially at startup when biases are zero.
|
|
|
|
#pragma once
|
|
|
|
#include <cmath>
|
|
#include <cstddef>
|
|
#include <span>
|
|
|
|
namespace nisps::ml {
|
|
|
|
inline constexpr float kDeadThresh = 0.001f;
|
|
inline constexpr float kSaturatingThresh = 3.0f;
|
|
|
|
struct LayerStats {
|
|
float mean_abs = 0.f;
|
|
float max_abs = 0.f;
|
|
float dead_frac = 0.f;
|
|
float saturating_frac = 0.f;
|
|
};
|
|
|
|
inline LayerStats compute_layer_stats(std::span<const float> weights,
|
|
std::span<const float> biases) noexcept {
|
|
LayerStats out;
|
|
const std::size_t total = weights.size() + biases.size();
|
|
if (total == 0u) return out;
|
|
|
|
float sum = 0.f;
|
|
float max_abs = 0.f;
|
|
std::size_t dead = 0u;
|
|
std::size_t sat = 0u;
|
|
|
|
for (float w : weights) {
|
|
const float a = std::fabs(w);
|
|
sum += a;
|
|
if (a > max_abs) max_abs = a;
|
|
if (a < kDeadThresh) ++dead;
|
|
if (a > kSaturatingThresh) ++sat;
|
|
}
|
|
for (float b : biases) {
|
|
const float a = std::fabs(b);
|
|
sum += a;
|
|
if (a > max_abs) max_abs = a;
|
|
}
|
|
|
|
out.mean_abs = sum / static_cast<float>(total);
|
|
out.max_abs = max_abs;
|
|
|
|
const float w_count = static_cast<float>(weights.size());
|
|
if (w_count > 0.f) {
|
|
out.dead_frac = static_cast<float>(dead) / w_count;
|
|
out.saturating_frac = static_cast<float>(sat) / w_count;
|
|
}
|
|
return out;
|
|
}
|
|
|
|
} // namespace nisps::ml
|