825ed6ad33
Stream 2 of the clean-slate rewrite: nisps/ml/ replaces src/memlp/ with a header-only, heap-free MLP that satisfies nisps::core::MLEngine. Files (nisps/ml/): - activations.hpp — ReLU (leaky 0.01 for parity), sigmoid, tanh - loss.hpp — MSE per-sample (fixes meml-ues double-scaling: returns the sample's MSE without an extra 1/N multiplication; the training loop averages explicitly) - init.hpp — uniform/Xavier/spread-aware weight init - training.hpp — gradient clip helper (±10.0 matches legacy) - rl.hpp — move_weights with per-layer Xavier scaling, weight decay (10% * spread), gaussian noise via the deterministic Rng (matches the legacy JS sum-of-three-uniforms shape); draw_weights also spread-aware - stats.hpp — per-layer mean/max/dead/saturating diagnostics - mlp.hpp — 4-layer (3 hidden + sigmoid output) MLP class with std::array-backed weights, biases, gradient accumulators, dataset ring buffer (default 128 examples), loss history (default 4096 iters). Bias is a separate per-layer parameter — no input-vector mutation. Flat get_weights/set_weights layout: weights all layers (row-major, layer order), then biases all layers. Tests (tests/cpp/, all 50 passing under -Wall -Wextra -Werror -Wpedantic): - test_mlp_init.cpp — deterministic seeding, spread regimes, static_assert MLEngine concept satisfied - test_mlp_inference.cpp — golden hand-computed forward pass match, sigmoid output range, set_input bounds - test_mlp_training.cpp — XOR convergence (loss < 0.01 in <2k iters), ring-buffer eviction - test_mlp_loss.cpp — meml-ues regression test: reported loss equals hand-computed average MSE without extra 1/N scaling; sample weights honoured - test_mlp_rl.cpp — move_weights respects output_pin_mask (final-layer rows + biases preserved); spread regimes; grad clear after draw_weights - test_mlp_serialize.cpp — get_weights/set_weights round-trip preserves inference exactly; eval_loss is non-mutating; infer_batch matches individual inference Verification: - Clean build, no warnings - 50 tests pass (22 prior + 28 new) - No std::vector / new / malloc in nisps/ml/ - All float literals .f-suffixed in code (comments excepted)
72 lines
2.7 KiB
C++
72 lines
2.7 KiB
C++
// nisps/ml/init.hpp — weight initialization strategies.
|
|
//
|
|
// Three strategies are exposed:
|
|
// - `uniform_init`: w ~ U[-1, 1]. Matches the legacy
|
|
// `utils::gen_rand<T>()` baseline; produces high-magnitude pre-activations
|
|
// that drive sigmoid output saturation.
|
|
// - `xavier_init`: w ~ U[-1, 1] / sqrt(fan_in). Centered pre-activations,
|
|
// better for sigmoid output layers.
|
|
// - `spread_init`: linearly interpolates per-layer scale between the two.
|
|
// scale = (1 - spread) + spread / sqrt(fan_in). spread=0 ⇒ uniform,
|
|
// spread=1 ⇒ Xavier. This is the playground knob.
|
|
//
|
|
// All three operate on a flat row-major weight buffer of size fan_in*fan_out.
|
|
// Biases are initialized separately and ALWAYS to zero — the legacy code
|
|
// initialized biases to zero, and the playground spread parameter never
|
|
// touches biases. We keep that.
|
|
|
|
#pragma once
|
|
|
|
#include <cmath>
|
|
#include <cstddef>
|
|
#include <span>
|
|
|
|
#include "../core/rng.hpp"
|
|
|
|
namespace nisps::ml {
|
|
|
|
// Compute the spread-aware weight scale for one layer.
|
|
// spread=0 → 1.0 (uniform [-1, 1])
|
|
// spread=1 → 1/sqrt(fan_in) (Xavier)
|
|
inline float spread_scale(float spread, std::size_t fan_in) noexcept {
|
|
if (fan_in == 0u) return 1.f;
|
|
const float inv_sqrt = 1.f / std::sqrt(static_cast<float>(fan_in));
|
|
return (1.f - spread) + spread * inv_sqrt;
|
|
}
|
|
|
|
// Initialize one layer's weights (flat row-major) and biases.
|
|
// weights: span of size fan_in * fan_out
|
|
// biases: span of size fan_out (zeroed)
|
|
// spread: see spread_scale
|
|
// rng: state advanced; caller owns it
|
|
inline void spread_init(std::span<float> weights,
|
|
std::span<float> biases,
|
|
std::size_t fan_in,
|
|
float spread,
|
|
Rng& rng) noexcept {
|
|
const float scale = spread_scale(spread, fan_in);
|
|
for (std::size_t i = 0; i < weights.size(); ++i) {
|
|
weights[i] = rng.next_float_signed() * scale;
|
|
}
|
|
for (std::size_t i = 0; i < biases.size(); ++i) {
|
|
biases[i] = 0.f;
|
|
}
|
|
}
|
|
|
|
// Convenience aliases for the endpoints. Cheap to build on top of
|
|
// spread_init; useful for tests that want to assert a specific regime.
|
|
inline void uniform_init(std::span<float> weights,
|
|
std::span<float> biases,
|
|
std::size_t fan_in,
|
|
Rng& rng) noexcept {
|
|
spread_init(weights, biases, fan_in, 0.f, rng);
|
|
}
|
|
|
|
inline void xavier_init(std::span<float> weights,
|
|
std::span<float> biases,
|
|
std::size_t fan_in,
|
|
Rng& rng) noexcept {
|
|
spread_init(weights, biases, fan_in, 1.f, rng);
|
|
}
|
|
|
|
} // namespace nisps::ml
|