w1n5t0n/memlnaut-nisps
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

test(ml): behavioural benchmark + 20 invariants for the control mapping

Browse source 
NISPS is a controller, not a synth: the object of study is the mapping
f: control-space -> parameter-space and how a musician's gestures deform
it. Loss measures fit to points the user dictated, which is the one thing
they never experience. So this measures geometry and gesture-response.

  tests/cpp/ml_bench.cpp          61 scenarios, REPORTS never asserts (same
                                  discipline as engine_bench.cpp). Shape-
                                  agnostic via MLPCore<DynamicStorage>
                                  (--shape, default 2,16,16,16,8), seeded
                                  RNG throughout, branch points replayed
                                  from scratch rather than snapshotted.
  tests/cpp/test_ml_behaviour.cpp 20 asserting invariants, wired into
                                  nisps_core_tests.
  scripts/bench-ml.sh             native + WASM from one source; --compare,
                                  --sweep-shape, --smoke, --scenario, --seed.

Documents two contracts that fail SILENTLY (both now pinned by tests):
a thumbs-up must call BOTH mlp.add_example() and fb.store_positive(),
since dislike_geometric k-NNs the replay buffer and not the MLP dataset;
and placed_output() is valid only while state == Placing, after which an
empty span whose l2() is 0 scores a broken lifecycle as a perfect place.

ALIGNMENT defect 6 re-ranked (SGD-vs-RMSProp is not a research axis - it
silently invalidated every ported hyperparameter) and split into 6b (the
geometric dislike was ported from a superseded upstream design) and 6c
(InterfaceRL, the reference impl, is not in the tree).

Gates: build-cpp-tests (138 tests, ctest 4/4), parity-check PASS,
lint-cpp clean, bench-ml.sh --smoke runs end to end.
This commit is contained in:
monkey-w1n5t0n 2026-07-25 11:02:24 +02:00
parent 60584606a8
commit 1603ea798e
6 changed files with 3409 additions and 3 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

51
ALIGNMENT.md
View file

@ -63,11 +63,56 @@ command surface to report through.
**Rough cost.** Host half is done. On-device: ~a day, and it wants defect 3's serial protocol
to have somewhere to send the number.
### 6. RMSProp still deferred from `nisps/ml/` (2026-04-29; reaffirmed 2026-07-21)
### 6. SGD-vs-RMSProp is not a research axis — it silently invalidated every ported hyperparameter (2026-04-29; **re-ranked 2026-07-25**)
**What.** `training.hpp` ships SGD only; the legacy firmware used RMSProp for `TrainBatch`. Optimizer choice is a research axis. Not blocking current fits; will matter for harder loss landscapes. Port target: upstream MusicallyEmbodiedML `memlp` (the in-repo `src/memlp` copy is deleted; use the GitHub remote or archive branch).
**What.** `training.hpp` ships SGD only. Upstream `memlp` (pinned `ea777502` by
`upstream/main`) applies gradients with **RMSProp everywhere** — `Layer.h:239`, the
`m_sq_grad_avg` running squared-gradient average at `Layer.h:601`, `StaticMLP.h:268`
"Mini-batch RMSProp training". This was filed as an optimiser-choice research question
and ranked last. That was wrong, and the 2026-07-25 benchmark work shows why.
**Rough cost.** A day, plus batch-convergence tests.
**Why it blocks the mission.** Every learning rate ported from upstream landed in a
different optimiser than the one it was tuned for. RMSProp normalises each step by the
running gradient magnitude, so `lr=1e-3` there is a normalised step; under SGD it is
literally `1e-3 x raw gradient`. The two numbers are unrelated. Concretely:
`feedback.hpp:789` carries `geo_lr_ = 0.001f // upstream InterfaceRL.hpp:312` — an
RMSProp LR pasted into a single SGD step. `tests/cpp/ml_bench.cpp` D1 measures the
result: the geometric dislike aims at a target 0.5 output-units away and moves the
mapping by **5.1e-5**, linearly, so ~10,000 presses would be needed for one press's
intended effect. Meanwhile likes train at `lr 1.0 x 1000 iterations`, making a like
~2e6x stronger than a dislike and heaving the whole mapping on every press
(`ml_bench` U1/U4: `lurch_max` ~1.1 against a [0,1] output range).
**Rough cost.** A day for the port plus batch-convergence tests — but it must come
BEFORE any retuning of `geo_lr`, `kGeometricPushScale` or the neg-LR base, or those
constants get tuned twice.
### 6b. The geometric dislike was ported from a superseded upstream design (2026-07-25)
**What.** `geo_push.hpp`/`replay.hpp` cite `memllib @ 0a541cc`. `upstream/main` now pins
`e291192`, where the same code has been deliberately redesigned. Upstream:
`kGeometricPushScale` 1.0 (ours 0.5); neg-LR base 1.5 (ours 0.5, `geo_push.hpp:92`);
the `/(1+len)` taper **deleted**, with the comment "a 'no' should clearly move the
mapping away even from a sound already far from the liked region (the taper used to
kill exactly that case)" — ours still applies it at `geo_push.hpp:66`; negatives trained
as a **batch over ALL of them every tick** rather than one item one step; and a fixed
`kDislikeLifetimeMs = 2500` full-strength lifetime replacing the proportional decay we
ported. On the shared constants (dedup radius 0.05, `kCentroidK` 4) we match.
**Why it blocks the mission.** We are carrying a design upstream diagnosed and fixed,
and the fix is documented in their source comments. Compounded with defect 6 the ported
dislike is ~9.4x weaker on constants alone before the optimiser mismatch.
**Rough cost.** Small once defect 6 lands — mostly deleting the taper and re-basing
three constants, then re-running `scripts/bench-ml.sh` D1/A4/A7 to confirm.
### 6c. `InterfaceRL` — the reference implementation — is not in the tree (2026-07-25)
**What.** It lives in `memllib/examples/`, and the vendoring dropped `examples/`
(`VENDORED.md`). So the source of truth for our most contested subsystem is absent, and
the divergences in 6b went unnoticed for months. Either vendor
`examples/InterfaceRL.{hpp,cpp,tpp}` read-only alongside the rest, or record its pinned
commit and a fetch recipe in `VENDORED.md`.
## Open mission questions

4
MAP.md
View file

@ -125,6 +125,8 @@ includes; no `nisps-core`.
### `tests/cpp/` — host C++ tests
- Per-component tests: `test_dsp_*.cpp`, `test_engine_*.cpp`, `test_mlp_*.cpp`, `test_mode_*.cpp`, `test_ring_buffer.cpp`, `test_rng.cpp`, `test_math.cpp`. Helpers in `test_helpers.hpp`.
- Verification: `ml_golden_vectors.cpp`, `engine_impulse.cpp` (+ `engine_impulse_baseline.bin`), `parity_check.cpp` + `parity_wasm.mjs` + `parity_diff.mjs` — native-vs-WASM bit-equivalence within 1e-5.
- Behaviour: `test_ml_behaviour.cpp` — 20 INVARIANTS of the interaction model (not tuning). Asserts what must hold however the knobs are set: a like is reachable and a re-like overwrites; dislike never yields non-finite weights/outputs in either `AvoidStyle`, and is safe at cold start; repeat dislikes inside `kReplayDedupRadius` deepen ONE negative while distant ones store separately; explore→reroll→undo→exit restores weights *bit-exactly* and over-undoing is safe; `RandomiseOutputs` leaves weights untouched (the outputs-vs-weights randomisation distinction); same seed + same gesture sequence ⇒ bit-identical weights (what makes any behavioural benchmark comparable); example-ring overflow keeps the newest; contradictory examples stay finite; the ExploreAndPlace accessor contract (`placed_output()` while Placing, `committed_output()` after commit — for both place and reposition); a FULLY-masked focus gate freezes every weight; switching mode mid-exploration never strands the net in a randomised scratchpad; and all of it holds at 1×1, 2×8, 1×33, 8×2 and 32×8 shapes.
- Measurement (asserts nothing): `ml_bench.cpp` — the BEHAVIOURAL benchmark. NISPS is a controller, so this measures the shape of the control→parameter mapping and how interaction journeys deform it, never loss alone. Shape-agnostic (`--shape N_IN,H1,H2,H3,N_OUT`, default `2,16,16,16,8`) via `MLPCore<DynamicStorage>`, so "does a wider/deeper net change the UX?" becomes a number; sample points come from a deterministic Kronecker low-discrepancy sequence rather than a raster, which is what makes it work at any input arity. Field metrics: local gain p50/p95, cliff index, dead fraction, range utilisation, rail occupancy, effective dimensionality (participation ratio — trace²/‖C‖²_F, no eigendecomposition). Displacement metrics: at-point, rings, global, **blast ratio**, and collateral damage at the protected positives. **61 scenarios.** Diagnostic (D1 — the geometric-dislike dose decomposed: intended push vs effective LR vs measured movement at 1/10/100/1000 presses). Atomic probes A1–A14: at/around/far-from an example; one dislike under BOTH candidate designs; twice at one point; adjacent-then-return across the dedup radius; near a protected positive; roll-a-patch-and-place; the full explore→audition→place→commit lifecycle; explore-then-cancel; reposition; like-then-dislike in place; dislike-then-repair; focus/solo mask leakage. Journeys J1–J11: positive-only retention curve, randomise-place-only, mixed, branch (one shared prefix, three divergent gestures, replayed from scratch per branch because that is exact under a deterministic RNG), explore-place-only, 120-gesture long session with drift + weight-norm checkpoints, dislike storm, revisit-after-wandering, two-region interference, sweep-and-teach along a continuous path, undo-heavy. Edge cases E1–E13: cold start, single example, contradictory, collinear, corners, capacity overflow, undo exhaustion, minimal shape, identical targets, rail targets, rapid like/dislike alternation, mode-switch mid-exploration, fully-masked dislike. Upstream comparison U1–U3: the older memllib `interfaceRL` (a DDPG actor-critic, recoverable from this repo's own git history at blob `755ff8b`) differs structurally — the user HEARS `actorTarget`, a soft copy updated `target += alpha*(online-target)` at alpha=0.005, and it trains a batch of 4 from replay only every `optimiseDivisor=40` gestures. The critic half is not reproducible here (MLPCore has `train_targets` but not the per-layer gradient extraction the policy-gradient step needs), but both OUTPUT-PATH ideas are: U1 sweeps the soft-target alpha (alpha=1 IS NISPS today, a free control), U2 sweeps the train-every-Nth divisor, U3 compares actor shapes, U4 sweeps the positive-path training dose. **U4 carries a load-bearing caveat**: upstream LRs are RMSProp LRs (memlp `Layer.h:239`, `StaticMLP.h:268`) and `nisps/ml/training.hpp` is SGD-only, so the numbers are not comparable — see ALIGNMENT.md defect 6. Their shared metric is **lurch** — how far the mapping the musician is playing moves per single gesture, averaged over the whole field. Knobs: `--spread` (1 = Xavier, 0 = uniform with NO fan_in coupling — i.e. the post-removal behaviour, measurable before paying for the refactor), `--geo-lr`, `--geo-iters`. Driven by `scripts/bench-ml.sh`. **Two contracts that fail SILENTLY and are pinned by tests:** (1) a thumbs-up must go through BOTH `mlp.add_example` AND `fb.store_positive` — `dislike_geometric` k-NNs the replay buffer, not the MLP dataset, so a harness that only calls `add_example` measures the cold-start branch instead; (2) `placed_output()` is valid ONLY while state is `Placing` — after `commit_place()`/`commit_reposition()` the vector moves to `committed_output()`, and reading the wrong one yields an empty span whose `l2()` is 0, i.e. a broken lifecycle scored as a perfect placement.
- Measurement (asserts nothing): `engine_bench.cpp` + `bench_report.mjs` — per-engine throughput (ns/sample, blocks/s, realtime factor) for the `process()` hot path. ONE source compiled twice (CMake `nisps_engine_bench` natively, emcc for WASM) so the two targets are comparable without adding a single export to `nisps/wasm/bindings.cpp`. Engines are driven into a working state (transport running + event drain for the sequencers, periodic `note_on` for paf_synth, a noise+sine input bed for the fx/analysis engines) and every row prints its own working-state evidence, so a number produced by an idle engine is visible rather than plausible. Driven by `scripts/bench-engines.sh`.
### `scripts/` — build + verify entry points
@ -132,6 +134,7 @@ includes; no `nisps-core`.
- `build-wasm.sh` — Emscripten compile producing `manifold/public/nisps.{wasm,js}`.
- `build-cpp-tests.sh` — CMake configure + build + ctest (Ninja).
- `parity-check.sh` — runs native + WASM and diffs binary outputs.
- `bench-ml.sh` — the ML BEHAVIOUR benchmark on native + WASM from one source (same trick as `bench-engines.sh`). `--shape`, `--scenario`, `--smoke`, `--seed`, `--compare`, and `--sweep-shape` (runs the corpus across a ladder of architectures and arities — the knob-sensitivity instrument). **Reports, never asserts**: a cliff index is a description, not a pass/fail. Invariants live in `tests/cpp/test_ml_behaviour.cpp` instead. Reports land in `nisps/build/bench-ml/`.
- `bench-engines.sh` — engine throughput on native + WASM; `--compare <report.json>` prints per-engine Δ%. **Reports, never asserts** (a wall-clock threshold on shared hardware is meaningless or flaky — same call as the firmware size job). Reports land in `nisps/build/bench/`.
- `lint-cpp.sh` — `.f` literal warn + heap/`Arduino.h` violation fail.
- `run-all-tests.sh` — master verification script.
@ -158,6 +161,7 @@ includes; no `nisps-core`.
- **Host C++ tests**: `bash scripts/build-cpp-tests.sh`.
- **Parity check**: `bash scripts/parity-check.sh`.
- **Engine benchmark**: `bash scripts/bench-engines.sh` (add `--compare nisps/build/bench/latest.json` to diff against the previous run).
- **ML behaviour benchmark**: `bash scripts/bench-ml.sh` (`--smoke` for a fast run, `--shape 2,16,16,16,8`, `--sweep-shape` for the architecture/arity sweep, `--compare` to diff).
- **All tests**: `bash scripts/run-all-tests.sh` (stage 6 is a bench smoke report; it does not gate).
- **Playwright**: `cd manifold && node node_modules/.bin/playwright test` (non-snap node runner on the VPS — BUILD-PLAN gotcha; `bunx playwright test` works elsewhere).
- **Codegen**: `cd codegen && bun run generate.ts` (regenerates `nisps/modes/generated/` + `nisps/ml/generated/` C++ and `manifold/src/modes/generated/` TS).

24
nisps/CMakeLists.txt
View file

@ -61,6 +61,7 @@ if(NOT EMSCRIPTEN)
${NISPS_TEST_DIR}/test_mlp_ou_noise.cpp
${NISPS_TEST_DIR}/test_mlp_feedback.cpp
${NISPS_TEST_DIR}/test_mlp_geo_dislike.cpp
${NISPS_TEST_DIR}/test_ml_behaviour.cpp
${NISPS_TEST_DIR}/test_mlp_serialize.cpp
${NISPS_TEST_DIR}/test_pipeline.cpp
${NISPS_TEST_DIR}/test_vcv_iml_parity.cpp
@ -189,6 +190,29 @@ if(NOT EMSCRIPTEN)
target_compile_options(nisps_engine_bench PRIVATE /W4 /WX)
endif()
# ---------------------------------------------------------------------
# Standalone ML BEHAVIOUR benchmark. Same discipline as nisps_engine_bench:
# it reports numbers and asserts nothing, so it is not a ctest gate. It
# measures the shape of the control mapping and how interaction journeys
# deform it — see MAP.md § `tests/cpp/`. Driven by
# scripts/bench-ml.sh, which compiles the SAME source to WASM via emcc.
#
# Uses MLPCore<DynamicStorage>, so it is host-only by construction
# (dynamic_storage.hpp #errors on RP2350). That is correct: the point is
# arbitrary runtime shapes, which firmware deliberately cannot have.
# ---------------------------------------------------------------------
add_executable(nisps_ml_bench
${NISPS_TEST_DIR}/ml_bench.cpp
)
target_link_libraries(nisps_ml_bench PRIVATE nisps_core)
if(CMAKE_CXX_COMPILER_ID MATCHES "GNU|Clang|AppleClang")
target_compile_options(nisps_ml_bench PRIVATE
-Wall -Wextra -Werror -Wpedantic -O3
)
elseif(MSVC)
target_compile_options(nisps_ml_bench PRIVATE /W4 /WX)
endif()
# Standalone parity-check runner. NOT registered with ctest — it's
# invoked from scripts/parity-check.sh which orchestrates native+WASM
# together.

180
scripts/bench-ml.sh Executable file
View file

@ -0,0 +1,180 @@
#!/usr/bin/env bash
# scripts/bench-ml.sh — behavioural benchmark for the NISPS control mapping.
#
# Sibling to bench-engines.sh, and deliberately the same shape: it builds
# tests/cpp/ml_bench.cpp twice from ONE source (native via CMake, WASM via
# emcc with the flags build-wasm.sh uses), runs both, and reports.
#
# NOTHING HERE FAILS. Behaviour is not a threshold — a "cliff index of 4.9" is
# neither pass nor fail, it is a description. Regressions are noticed by
# running with --compare against a previous report, which prints per-metric Δ.
# Invariants that genuinely must hold are ctest assertions in
# tests/cpp/test_ml_behaviour.cpp, not here.
#
# WHY BOTH TARGETS
# ----------------
# Firmware and browser must feel the same. Native-vs-WASM here is a BEHAVIOURAL
# comparison, which parity-check.sh's 1e-5 bit-equivalence does not give you:
# parity proves two builds of the SAME commit agree, and says nothing about
# whether a mapping is playable or whether a gesture does anything.
#
# Usage:
# scripts/bench-ml.sh # native + wasm
# scripts/bench-ml.sh --native-only
# scripts/bench-ml.sh --smoke # fast, proves it runs
# scripts/bench-ml.sh --shape 2,16,16,16,8 # any net shape
# scripts/bench-ml.sh --sweep-shape # the architecture sweep
# scripts/bench-ml.sh --scenario A4_negative_once
# scripts/bench-ml.sh --compare old.json
# scripts/bench-ml.sh --out bench-ml-2026-07-25.json
#
# Env:
# NISPS_BUILD_DIR default nisps/build
# EMCC emcc path (same convention as build-wasm.sh)
#
# Exit codes: 0 on a completed run, 2 on bad args/missing artifacts, 3 on build
# failure.
set -euo pipefail
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
BUILD_DIR="${NISPS_BUILD_DIR:-$ROOT/nisps/build}"
BENCH_DIR="$BUILD_DIR/bench-ml"
NATIVE_BIN="$BUILD_DIR/nisps_ml_bench"
SRC="$ROOT/tests/cpp/ml_bench.cpp"
REPORT="$ROOT/tests/cpp/ml_bench_report.mjs"
EMCC="${EMCC:-emcc}"
NATIVE_ONLY=0
SMOKE=""
SHAPE=""
SCENARIO=""
COMPARE=""
OUT=""
SWEEP_SHAPE=0
SEED=""
while [[ $# -gt 0 ]]; do
case "$1" in
--native-only) NATIVE_ONLY=1; shift ;;
--smoke) SMOKE="--smoke"; shift ;;
--shape) SHAPE="$2"; shift 2 ;;
--seed) SEED="$2"; shift 2 ;;
--scenario) SCENARIO="$2"; shift 2 ;;
--compare) COMPARE="$2"; shift 2 ;;
--out) OUT="$2"; shift 2 ;;
--sweep-shape) SWEEP_SHAPE=1; shift ;;
-h|--help) sed -n '2,40p' "$0"; exit 0 ;;
*) echo "bench-ml: unknown arg '$1'" >&2; exit 2 ;;
esac
done
mkdir -p "$BENCH_DIR"
bench_args() {
local a=()
[[ -n "$SMOKE" ]] && a+=("$SMOKE")
[[ -n "$SHAPE" ]] && a+=(--shape "$SHAPE")
[[ -n "$SEED" ]] && a+=(--seed "$SEED")
[[ -n "$SCENARIO" ]] && a+=(--scenario "$SCENARIO")
printf '%s\n' "${a[@]:-}"
}
# ---------------------------------------------------------------------------
# native
# ---------------------------------------------------------------------------
echo "==> building native ml_bench" >&2
if ! cmake -S "$ROOT/nisps" -B "$BUILD_DIR" -G Ninja -DCMAKE_BUILD_TYPE=Release >/dev/null 2>&1; then
echo "bench-ml: cmake configure failed" >&2; exit 3
fi
if ! cmake --build "$BUILD_DIR" --target nisps_ml_bench >/dev/null 2>&1; then
echo "bench-ml: native build failed" >&2; exit 3
fi
mapfile -t ARGS < <(bench_args)
NATIVE_JSON="$BENCH_DIR/native.json"
if [[ ${#ARGS[@]} -gt 0 && -n "${ARGS[0]}" ]]; then
"$NATIVE_BIN" "${ARGS[@]}" > "$NATIVE_JSON"
else
"$NATIVE_BIN" > "$NATIVE_JSON"
fi
echo "==> native report: $NATIVE_JSON" >&2
# ---------------------------------------------------------------------------
# wasm — one source, second compiler. Same flags build-wasm.sh uses for the
# shipped module, so the comparison is against what actually ships.
# ---------------------------------------------------------------------------
WASM_JSON=""
if [[ $NATIVE_ONLY -eq 0 ]]; then
if ! command -v "$EMCC" >/dev/null 2>&1; then
echo "bench-ml: emcc not found; skipping wasm (use --native-only to silence)" >&2
else
echo "==> building wasm ml_bench" >&2
WASM_JS="$BENCH_DIR/ml_bench.mjs"
if ! "$EMCC" "$SRC" -o "$WASM_JS" \
-std=gnu++20 -O3 \
-s ENVIRONMENT=node -s EXIT_RUNTIME=1 -s ALLOW_MEMORY_GROWTH=1 \
-s SINGLE_FILE=1 >/dev/null 2>&1; then
echo "bench-ml: wasm build failed" >&2; exit 3
fi
WASM_JSON="$BENCH_DIR/wasm.json"
if [[ ${#ARGS[@]} -gt 0 && -n "${ARGS[0]}" ]]; then
node "$WASM_JS" "${ARGS[@]}" > "$WASM_JSON"
else
node "$WASM_JS" > "$WASM_JSON"
fi
echo "==> wasm report: $WASM_JSON" >&2
fi
fi
# ---------------------------------------------------------------------------
# architecture sweep — the "how does net shape change the UX" instrument.
# Runs the corpus at a ladder of shapes and emits one combined report.
# ---------------------------------------------------------------------------
if [[ $SWEEP_SHAPE -eq 1 ]]; then
SWEEP_JSON="$BENCH_DIR/sweep-shape.json"
echo "==> shape sweep" >&2
{
echo "["
first=1
for s in \
"2,4,4,4,8" \
"2,8,8,8,8" \
"2,16,16,16,8" \
"2,32,32,32,8" \
"2,64,64,64,8" \
"1,16,16,16,8" \
"4,16,16,16,8" \
"8,16,16,16,8" \
"32,16,16,16,8" \
"2,16,16,16,1" \
"2,16,16,16,4" \
"2,16,16,16,16" \
"2,16,16,16,33"
do
[[ $first -eq 0 ]] && echo ","
first=0
"$NATIVE_BIN" --shape "$s" ${SMOKE:+$SMOKE}
done
echo "]"
} > "$SWEEP_JSON"
echo "==> sweep report: $SWEEP_JSON" >&2
fi
# ---------------------------------------------------------------------------
# format + diff
# ---------------------------------------------------------------------------
if [[ -f "$REPORT" ]] && command -v node >/dev/null 2>&1; then
if [[ -n "$COMPARE" ]]; then
node "$REPORT" "$NATIVE_JSON" ${WASM_JSON:+"$WASM_JSON"} --compare "$COMPARE"
else
node "$REPORT" "$NATIVE_JSON" ${WASM_JSON:+"$WASM_JSON"}
fi
else
echo "(no formatter; raw JSON is in $BENCH_DIR)" >&2
fi
if [[ -n "$OUT" ]]; then
command cp -f "$NATIVE_JSON" "$OUT"
echo "==> saved $OUT" >&2
fi

2602
tests/cpp/ml_bench.cpp Normal file
View file

File diff suppressed because it is too large Load diff

551
tests/cpp/test_ml_behaviour.cpp Normal file
View file

@ -0,0 +1,551 @@
// tests/cpp/test_ml_behaviour.cpp — INVARIANTS of the interaction model.
//
// Split of responsibility with tests/cpp/ml_bench.cpp:
// ml_bench.cpp REPORTS behaviour (how local is a dislike? how playable is
// the mapping?). Those are descriptions, not pass/fail, so it
// asserts nothing and is not a ctest gate.
// this file ASSERTS the things that must be TRUE regardless of tuning.
// If one of these breaks, the interaction model is broken, not
// merely different — a musician would call it a bug.
//
// Every test here is shape-parameterised where it can be, and runs at the
// operator's default 2-in/8-out as well as at degenerate shapes, because the
// edge cases are exactly where an interaction model quietly stops holding.
//
// These use MLPCore<DynamicStorage>, which is host-only by construction
// (dynamic_storage.hpp #errors on RP2350). That is fine: these are behavioural
// invariants of the shared algorithms, and the fixed-storage firmware path is
// bit-pinned to the dynamic one by test_mlp_storage_parity.cpp.
#include <cmath>
#include <span>
#include <vector>
#include "../../nisps/ml/dynamic_storage.hpp"
#include "../../nisps/ml/feedback.hpp"
#include "../../nisps/ml/mlp.hpp"
#include "test_helpers.hpp"
namespace {
using nisps::ml::AvoidStyle;
using nisps::ml::DynamicFeedbackStorage;
using nisps::ml::DynamicStorage;
using nisps::ml::FeedbackControllerCore;
using nisps::ml::FeedbackMode;
using nisps::ml::MLPCore;
using Mlp = MLPCore<DynamicStorage>;
using Feedback = FeedbackControllerCore<DynamicFeedbackStorage>;
constexpr std::uint64_t kSeed = 0xC0FFEEu;
struct Rig {
std::size_t n_in, n_out;
Mlp mlp;
Feedback fb;
Rig(std::size_t nin, std::size_t nout, std::size_t h = 16u,
std::size_t max_examples = 128u)
: n_in(nin), n_out(nout),
mlp(kSeed, nin, std::span<const std::size_t>(hidden_(h), 3u), nout,
max_examples, 4096u),
fb(kSeed ^ 0xF33Dull, nout, mlp.weight_count(), 4u, nin, 64u) {}
// Static storage for the hidden dims so the span outlives the ctor call.
static const std::size_t* hidden_(std::size_t h) {
static std::size_t buf[3];
buf[0] = buf[1] = buf[2] = h;
return buf;
}
void at(std::span<const float> x, std::vector<float>& out) {
out.assign(n_out, 0.f);
for (std::size_t i = 0; i < n_in; ++i) mlp.set_input(i, x[i]);
mlp.process();
auto o = mlp.outputs();
for (std::size_t j = 0; j < n_out; ++j) out[j] = o[j];
}
// The product path for a thumbs-up: dataset example AND replay positive.
void like(std::span<const float> x, std::span<const float> y) {
for (std::size_t i = 0; i < n_in; ++i) mlp.set_input(i, x[i]);
mlp.process();
mlp.add_example(x, y);
fb.store_positive(mlp, y);
}
void down(std::span<const float> x, float speed = 0.1f, float spread = 1.f) {
std::vector<float> heard;
at(x, heard);
for (std::size_t i = 0; i < n_in; ++i) mlp.set_input(i, x[i]);
mlp.process();
fb.on_down(mlp, heard, speed, spread, {});
}
// get_weights() returns a span into a storage-owned scratch buffer, so it
// must be COPIED before the next call reuses that buffer.
void weights(std::vector<float>& w) {
auto s = mlp.get_weights();
w.assign(s.begin(), s.end());
}
};
float l2(std::span<const float> a, std::span<const float> b) {
float acc = 0.f;
const std::size_t n = a.size() < b.size() ? a.size() : b.size();
for (std::size_t i = 0; i < n; ++i) { const float d = a[i] - b[i]; acc += d * d; }
return std::sqrt(acc);
}
bool all_finite(std::span<const float> v) {
for (float x : v) if (!std::isfinite(x)) return false;
return true;
}
} // namespace
// ===========================================================================
// PLACEMENT — a thumbs-up must actually take.
// ===========================================================================
// The single most basic promise the instrument makes: if you stand somewhere,
// like a set of outputs, and then stand in the SAME place again, you get
// approximately what you liked. Without this nothing else means anything.
NISPS_TEST(behaviour_single_like_is_reachable) {
Rig rig(2u, 8u);
const float x[2] = {0.25f, -0.4f};
std::vector<float> y(8u);
for (std::size_t j = 0; j < 8u; ++j) y[j] = 0.2f + 0.07f * static_cast<float>(j);
rig.like(x, y);
rig.mlp.train();
std::vector<float> got;
rig.at(x, got);
NISPS_EXPECT(all_finite(got));
// Loose bound on purpose: this asserts "the placement took", not "the
// optimiser is good". Tightening it would make it a tuning test.
NISPS_EXPECT(l2(got, y) < 0.25f);
}
// Placing at the same position twice with a NEW target must move toward the
// new one — the musician overwrote their earlier choice and expects that to win.
NISPS_TEST(behaviour_relike_same_spot_overwrites) {
Rig rig(2u, 8u);
const float x[2] = {0.1f, 0.1f};
std::vector<float> y1(8u, 0.2f), y2(8u, 0.8f), got;
rig.like(x, y1);
rig.mlp.train();
rig.at(x, got);
const float err_to_y2_before = l2(got, y2);
rig.like(x, y2);
rig.mlp.train();
rig.at(x, got);
const float err_to_y2_after = l2(got, y2);
NISPS_EXPECT(err_to_y2_after < err_to_y2_before);
}
// ===========================================================================
// FEEDBACK — the negative path must be well-behaved even when it is gentle.
// ===========================================================================
// A dislike must never produce non-finite outputs. This is the one thing that
// turns "the instrument feels wrong" into "the instrument is dead", and a
// perturbation path plus a sigmoid is exactly where NaN would come from.
NISPS_TEST(behaviour_dislike_never_produces_nonfinite) {
for (AvoidStyle style : {AvoidStyle::Geometric, AvoidStyle::Diffuse}) {
Rig rig(2u, 8u);
std::vector<float> y(8u, 0.5f), got;
const float a[2] = {0.3f, 0.3f};
const float b[2] = {-0.3f, 0.2f};
rig.like(a, y);
rig.like(b, y);
rig.mlp.train();
rig.fb.set_mode(FeedbackMode::Avoid, rig.mlp);
rig.fb.set_avoid_style(style);
const float press[2] = {0.f, 0.25f};
for (int i = 0; i < 32; ++i) rig.down(press);
rig.at(press, got);
NISPS_EXPECT(all_finite(got));
std::vector<float> w;
rig.weights(w);
NISPS_EXPECT(all_finite(w));
}
}
// Dislike at cold start (no positives at all) must not crash or corrupt. This
// is literally the first thing a new user might press.
NISPS_TEST(behaviour_dislike_cold_start_is_safe) {
Rig rig(2u, 8u);
rig.fb.set_mode(FeedbackMode::Avoid, rig.mlp);
rig.fb.set_avoid_style(AvoidStyle::Geometric);
const float press[2] = {0.f, 0.f};
NISPS_ASSERT(rig.fb.positive_count() == 0u);
for (int i = 0; i < 8; ++i) rig.down(press);
std::vector<float> got;
rig.at(press, got);
NISPS_EXPECT(all_finite(got));
NISPS_EXPECT(rig.fb.negative_count() > 0u);
}
// Repeated dislikes at the same point must deepen ONE replay entry rather than
// filling the buffer — replay.hpp's dedup radius (0.05) exists for exactly
// this, and if it regresses a user holding the button evicts their own history.
NISPS_TEST(behaviour_repeat_dislike_dedups_not_accumulates) {
Rig rig(2u, 8u);
std::vector<float> y(8u, 0.5f);
const float a[2] = {0.4f, 0.4f};
rig.like(a, y);
rig.mlp.train();
rig.fb.set_mode(FeedbackMode::Avoid, rig.mlp);
rig.fb.set_avoid_style(AvoidStyle::Geometric);
const float press[2] = {-0.2f, 0.1f};
for (int i = 0; i < 20; ++i) rig.down(press);
// 20 presses inside the dedup radius: one negative, deepened.
NISPS_EXPECT(rig.fb.negative_count() == 1u);
}
// A dislike far outside the dedup radius must create a SEPARATE negative.
// Together with the test above this pins the "move a little to the left"
// behaviour the operator asked about: inside 0.05 deepens, outside stores.
NISPS_TEST(behaviour_distant_dislike_stores_separately) {
Rig rig(2u, 8u);
std::vector<float> y(8u, 0.5f);
const float a[2] = {0.4f, 0.4f};
rig.like(a, y);
rig.mlp.train();
rig.fb.set_mode(FeedbackMode::Avoid, rig.mlp);
rig.fb.set_avoid_style(AvoidStyle::Geometric);
const float p1[2] = {-0.5f, 0.f};
const float p2[2] = {0.5f, 0.f}; // well beyond kReplayDedupRadius
rig.down(p1);
rig.down(p2);
NISPS_EXPECT(rig.fb.negative_count() == 2u);
}
// ===========================================================================
// EXPLORE / UNDO — the scratchpad must be exactly reversible.
// ===========================================================================
// Enter explore, roll, then undo everything and exit: the net must return to
// EXACTLY where it started. Not approximately — the whole point of the
// scratchpad model is that auditioning costs nothing.
NISPS_TEST(behaviour_explore_undo_restores_exactly) {
Rig rig(2u, 8u);
std::vector<float> y(8u, 0.5f);
const float a[2] = {0.2f, -0.2f};
rig.like(a, y);
rig.mlp.train();
std::vector<float> before, after;
rig.weights(before);
rig.fb.set_mode(FeedbackMode::ExploreAndPlace, rig.mlp);
rig.fb.enter_explore(rig.mlp, 1.f);
for (int i = 0; i < 3; ++i) rig.fb.reroll(rig.mlp, 1.f);
for (int i = 0; i < 3; ++i) rig.fb.undo(rig.mlp);
rig.fb.exit_explore(rig.mlp);
rig.weights(after);
bool identical = true;
for (std::size_t i = 0; i < before.size(); ++i) {
if (before[i] != after[i]) { identical = false; break; }
}
NISPS_EXPECT(identical);
}
// Undoing more times than the ring holds must not corrupt anything. The user
// mashing undo is not a bug report.
NISPS_TEST(behaviour_undo_past_ring_depth_is_safe) {
Rig rig(2u, 8u);
std::vector<float> y(8u, 0.5f);
const float a[2] = {0.f, 0.f};
rig.like(a, y);
rig.mlp.train();
rig.fb.set_mode(FeedbackMode::ExploreAndPlace, rig.mlp);
rig.fb.enter_explore(rig.mlp, 1.f);
for (int i = 0; i < 3; ++i) rig.fb.reroll(rig.mlp, 1.f);
for (int i = 0; i < 40; ++i) rig.fb.undo(rig.mlp); // way past depth 4
rig.fb.exit_explore(rig.mlp);
std::vector<float> w, got;
rig.weights(w);
NISPS_EXPECT(all_finite(w));
rig.at(a, got);
NISPS_EXPECT(all_finite(got));
}
// Output randomisation must NOT touch weights. This is the distinction the
// operator drew explicitly: RandomiseOutputs rolls a patch, RandomiseMlp
// scrambles the net. If the former ever moved weights, auditioning a patch
// would silently damage the instrument.
NISPS_TEST(behaviour_randomise_outputs_leaves_weights_untouched) {
Rig rig(2u, 8u);
std::vector<float> y(8u, 0.5f);
const float a[2] = {0.15f, 0.15f};
rig.like(a, y);
rig.mlp.train();
std::vector<float> before, after;
rig.weights(before);
rig.fb.set_mode(FeedbackMode::RandomiseOutputs, rig.mlp);
const float press[2] = {-0.1f, 0.3f};
for (int i = 0; i < 5; ++i) rig.down(press); // enter + 4 re-rolls
rig.weights(after);
bool identical = true;
for (std::size_t i = 0; i < before.size(); ++i) {
if (before[i] != after[i]) { identical = false; break; }
}
NISPS_EXPECT(identical);
// ...and it must actually be holding a patch to audition.
std::vector<float> patch(8u, 0.f);
NISPS_EXPECT(rig.fb.static_output(patch));
NISPS_EXPECT(all_finite(patch));
}
// ===========================================================================
// DETERMINISM — the benchmark is worthless if the engine is not reproducible.
// ===========================================================================
// Two rigs built with the same seed, driven through the same gesture sequence,
// must end bit-identical. This is what makes a behavioural benchmark
// comparable across commits at all.
NISPS_TEST(behaviour_same_seed_same_journey_is_bit_identical) {
auto journey = [](Rig& rig) {
std::vector<float> y(8u);
for (std::size_t j = 0; j < 8u; ++j) y[j] = 0.3f + 0.05f * static_cast<float>(j);
const float a[2] = {0.2f, 0.1f};
const float b[2] = {-0.3f, 0.4f};
rig.like(a, y);
rig.like(b, y);
rig.mlp.train();
rig.fb.set_mode(FeedbackMode::Avoid, rig.mlp);
rig.fb.set_avoid_style(AvoidStyle::Diffuse);
const float press[2] = {0.f, 0.f};
for (int i = 0; i < 5; ++i) rig.down(press);
};
Rig r1(2u, 8u), r2(2u, 8u);
journey(r1);
journey(r2);
std::vector<float> w1, w2;
r1.weights(w1);
r2.weights(w2);
NISPS_ASSERT(w1.size() == w2.size());
bool identical = true;
for (std::size_t i = 0; i < w1.size(); ++i) {
if (w1[i] != w2[i]) { identical = false; break; }
}
NISPS_EXPECT(identical);
}
// ===========================================================================
// CAPACITY + DEGENERATE SHAPES
// ===========================================================================
// Overflowing the example ring must not corrupt state — it evicts, silently,
// and the newest examples must still be the ones honoured.
NISPS_TEST(behaviour_example_overflow_keeps_newest) {
const std::size_t kCap = 8u;
Rig rig(2u, 8u, 16u, kCap);
std::vector<float> got;
// 3x the cap. Each example has a distinguishable target.
const std::size_t N = kCap * 3u;
std::vector<float> xs(N * 2u), ys(N * 8u);
for (std::size_t i = 0; i < N; ++i) {
const float t = static_cast<float>(i) / static_cast<float>(N);
xs[i * 2u] = 2.f * t - 1.f;
xs[i * 2u + 1u] = 0.f;
for (std::size_t j = 0; j < 8u; ++j) ys[i * 8u + j] = t;
rig.like(std::span<const float>(&xs[i * 2u], 2u),
std::span<const float>(&ys[i * 8u], 8u));
}
rig.mlp.train();
// The LAST example must be honoured; it cannot have been evicted.
rig.at(std::span<const float>(&xs[(N - 1u) * 2u], 2u), got);
NISPS_EXPECT(all_finite(got));
NISPS_EXPECT(l2(got, std::span<const float>(&ys[(N - 1u) * 8u], 8u)) < 0.5f);
}
// Contradictory examples (same input, different targets) must converge to
// something finite rather than diverging. A musician WILL do this by accident.
NISPS_TEST(behaviour_contradictory_examples_stay_finite) {
Rig rig(2u, 8u);
const float x[2] = {0.f, 0.f};
for (int k = 0; k < 6; ++k) {
std::vector<float> y(8u, 0.1f + 0.15f * static_cast<float>(k));
rig.like(x, y);
}
rig.mlp.train();
std::vector<float> got, w;
rig.at(x, got);
rig.weights(w);
NISPS_EXPECT(all_finite(got));
NISPS_EXPECT(all_finite(w));
}
// The interaction model must hold at degenerate shapes, not just the default.
// 1-in/1-out is the narrowest legal net; a wide-output net is the realistic
// upper end once a mode drives 33 synth parameters.
NISPS_TEST(behaviour_holds_at_degenerate_shapes) {
struct Shape { std::size_t n_in, n_out, hidden; };
const Shape shapes[] = {
{1u, 1u, 4u}, // narrowest legal
{2u, 8u, 16u}, // operator default
{1u, 33u, 8u}, // one control, many parameters
{8u, 2u, 8u}, // many controls, few parameters
{32u, 8u, 16u}, // the over-provisioned browser head
};
for (const Shape& s : shapes) {
Rig rig(s.n_in, s.n_out, s.hidden);
std::vector<float> x(s.n_in, 0.2f), y(s.n_out, 0.6f), got, w;
rig.like(x, y);
rig.mlp.train();
rig.at(x, got);
NISPS_EXPECT(all_finite(got));
rig.fb.set_mode(FeedbackMode::Avoid, rig.mlp);
rig.fb.set_avoid_style(AvoidStyle::Geometric);
rig.down(x);
rig.weights(w);
NISPS_EXPECT(all_finite(w));
NISPS_EXPECT(rig.mlp.n_in() == s.n_in);
NISPS_EXPECT(rig.mlp.n_out() == s.n_out);
}
}
// ===========================================================================
// EXPLORE-AND-PLACE ACCESSOR CONTRACT
//
// This exists because getting it wrong fails SILENTLY, and it did: reading
// placed_output() after commit_place() returns an EMPTY span, l2() over an
// empty span is 0, and 0 reads as a perfect placement. A benchmark scored a
// broken lifecycle as flawless until ASAN found the out-of-bounds downstream.
// ===========================================================================
NISPS_TEST(behaviour_placed_vs_committed_output_contract) {
Rig rig(2u, 8u);
std::vector<float> y(8u, 0.4f);
const float a[2] = {0.1f, -0.1f};
rig.like(a, y);
rig.mlp.train();
rig.fb.set_mode(FeedbackMode::ExploreAndPlace, rig.mlp);
rig.fb.enter_explore(rig.mlp, 1.f);
rig.fb.reroll(rig.mlp, 1.f);
// While Exploring: nothing is placed yet.
NISPS_EXPECT(rig.fb.placed_output().empty());
rig.mlp.process();
std::vector<float> audition(8u);
{ auto o = rig.mlp.outputs();
for (std::size_t j = 0; j < 8u; ++j) audition[j] = o[j]; }
rig.fb.begin_place(rig.mlp, audition);
// While Placing: placed_output() is the live accessor.
NISPS_ASSERT(rig.fb.placed_output().size() == 8u);
NISPS_EXPECT(l2(rig.fb.placed_output(), audition) == 0.f);
rig.fb.commit_place(rig.mlp);
// After commit: state is Idle, so placed_output() goes EMPTY and the
// caller must read committed_output() instead.
NISPS_EXPECT(rig.fb.placed_output().empty());
NISPS_ASSERT(rig.fb.committed_output().size() == 8u);
NISPS_EXPECT(l2(rig.fb.committed_output(), audition) == 0.f);
}
// Reposition hands its carried vector over the same way.
NISPS_TEST(behaviour_reposition_carries_via_committed_output) {
Rig rig(2u, 8u);
std::vector<float> y(8u, 0.35f);
const float src[2] = {0.3f, 0.3f};
rig.like(src, y);
rig.mlp.train();
rig.fb.set_mode(FeedbackMode::ExploreAndPlace, rig.mlp);
for (std::size_t i = 0; i < 2u; ++i) rig.mlp.set_input(i, src[i]);
rig.mlp.process();
rig.fb.begin_reposition(rig.mlp);
NISPS_ASSERT(rig.fb.placed_output().size() == 8u);
std::vector<float> grabbed(rig.fb.placed_output().begin(),
rig.fb.placed_output().end());
rig.fb.commit_reposition();
NISPS_ASSERT(rig.fb.committed_output().size() == 8u);
NISPS_EXPECT(l2(rig.fb.committed_output(), grabbed) == 0.f);
NISPS_EXPECT(!rig.fb.repositioning());
}
// A fully-masked focus gate must freeze EVERY output: zero active dims means
// zero gradient, so a dislike is a no-op on the weights.
NISPS_TEST(behaviour_full_focus_mask_freezes_everything) {
Rig rig(2u, 8u);
std::vector<float> y(8u, 0.5f);
const float a[2] = {0.2f, 0.2f};
rig.like(a, y);
rig.mlp.train();
std::vector<std::uint8_t> mask(8u, 0u); // nothing active
rig.fb.set_focus_mask(mask);
rig.fb.set_mode(FeedbackMode::Avoid, rig.mlp);
rig.fb.set_avoid_style(AvoidStyle::Geometric);
std::vector<float> before, after;
rig.weights(before);
const float press[2] = {-0.1f, 0.1f};
for (int i = 0; i < 16; ++i) rig.down(press);
rig.weights(after);
bool identical = true;
for (std::size_t i = 0; i < before.size(); ++i) {
if (before[i] != after[i]) { identical = false; break; }
}
NISPS_EXPECT(identical);
}
// Switching feedback mode mid-exploration must tear down cleanly and never
// strand the net in a randomised scratchpad state. Live, that would be fatal.
NISPS_TEST(behaviour_mode_switch_midflight_never_strands) {
Rig rig(2u, 8u);
std::vector<float> y(8u, 0.45f);
const float a[2] = {0.f, 0.2f};
rig.like(a, y);
rig.mlp.train();
std::vector<float> before, after;
rig.weights(before);
rig.fb.set_mode(FeedbackMode::ExploreAndPlace, rig.mlp);
rig.fb.enter_explore(rig.mlp, 1.f);
for (int i = 0; i < 3; ++i) rig.fb.reroll(rig.mlp, 1.f);
rig.fb.set_mode(FeedbackMode::Avoid, rig.mlp); // yank it out
rig.weights(after);
bool identical = true;
for (std::size_t i = 0; i < before.size(); ++i) {
if (before[i] != after[i]) { identical = false; break; }
}
NISPS_EXPECT(identical);
NISPS_EXPECT(!rig.fb.exploring());
}