From 1603ea798e0bf00c43257eca968ef23296663ab1 Mon Sep 17 00:00:00 2001 From: monkey-w1n5t0n Date: Sat, 25 Jul 2026 11:02:24 +0200 Subject: [PATCH] test(ml): behavioural benchmark + 20 invariants for the control mapping NISPS is a controller, not a synth: the object of study is the mapping f: control-space -> parameter-space and how a musician's gestures deform it. Loss measures fit to points the user dictated, which is the one thing they never experience. So this measures geometry and gesture-response. tests/cpp/ml_bench.cpp 61 scenarios, REPORTS never asserts (same discipline as engine_bench.cpp). Shape- agnostic via MLPCore (--shape, default 2,16,16,16,8), seeded RNG throughout, branch points replayed from scratch rather than snapshotted. tests/cpp/test_ml_behaviour.cpp 20 asserting invariants, wired into nisps_core_tests. scripts/bench-ml.sh native + WASM from one source; --compare, --sweep-shape, --smoke, --scenario, --seed. Documents two contracts that fail SILENTLY (both now pinned by tests): a thumbs-up must call BOTH mlp.add_example() and fb.store_positive(), since dislike_geometric k-NNs the replay buffer and not the MLP dataset; and placed_output() is valid only while state == Placing, after which an empty span whose l2() is 0 scores a broken lifecycle as a perfect place. ALIGNMENT defect 6 re-ranked (SGD-vs-RMSProp is not a research axis - it silently invalidated every ported hyperparameter) and split into 6b (the geometric dislike was ported from a superseded upstream design) and 6c (InterfaceRL, the reference impl, is not in the tree). Gates: build-cpp-tests (138 tests, ctest 4/4), parity-check PASS, lint-cpp clean, bench-ml.sh --smoke runs end to end. --- ALIGNMENT.md | 51 +- MAP.md | 4 + nisps/CMakeLists.txt | 24 + scripts/bench-ml.sh | 180 +++ tests/cpp/ml_bench.cpp | 2602 +++++++++++++++++++++++++++++++ tests/cpp/test_ml_behaviour.cpp | 551 +++++++ 6 files changed, 3409 insertions(+), 3 deletions(-) create mode 100755 scripts/bench-ml.sh create mode 100644 tests/cpp/ml_bench.cpp create mode 100644 tests/cpp/test_ml_behaviour.cpp diff --git a/ALIGNMENT.md b/ALIGNMENT.md index 676f300..ce15fc0 100644 --- a/ALIGNMENT.md +++ b/ALIGNMENT.md @@ -63,11 +63,56 @@ command surface to report through. **Rough cost.** Host half is done. On-device: ~a day, and it wants defect 3's serial protocol to have somewhere to send the number. -### 6. RMSProp still deferred from `nisps/ml/` (2026-04-29; reaffirmed 2026-07-21) +### 6. SGD-vs-RMSProp is not a research axis — it silently invalidated every ported hyperparameter (2026-04-29; **re-ranked 2026-07-25**) -**What.** `training.hpp` ships SGD only; the legacy firmware used RMSProp for `TrainBatch`. Optimizer choice is a research axis. Not blocking current fits; will matter for harder loss landscapes. Port target: upstream MusicallyEmbodiedML `memlp` (the in-repo `src/memlp` copy is deleted; use the GitHub remote or archive branch). +**What.** `training.hpp` ships SGD only. Upstream `memlp` (pinned `ea777502` by +`upstream/main`) applies gradients with **RMSProp everywhere** — `Layer.h:239`, the +`m_sq_grad_avg` running squared-gradient average at `Layer.h:601`, `StaticMLP.h:268` +"Mini-batch RMSProp training". This was filed as an optimiser-choice research question +and ranked last. That was wrong, and the 2026-07-25 benchmark work shows why. -**Rough cost.** A day, plus batch-convergence tests. +**Why it blocks the mission.** Every learning rate ported from upstream landed in a +different optimiser than the one it was tuned for. RMSProp normalises each step by the +running gradient magnitude, so `lr=1e-3` there is a normalised step; under SGD it is +literally `1e-3 x raw gradient`. The two numbers are unrelated. Concretely: +`feedback.hpp:789` carries `geo_lr_ = 0.001f // upstream InterfaceRL.hpp:312` — an +RMSProp LR pasted into a single SGD step. `tests/cpp/ml_bench.cpp` D1 measures the +result: the geometric dislike aims at a target 0.5 output-units away and moves the +mapping by **5.1e-5**, linearly, so ~10,000 presses would be needed for one press's +intended effect. Meanwhile likes train at `lr 1.0 x 1000 iterations`, making a like +~2e6x stronger than a dislike and heaving the whole mapping on every press +(`ml_bench` U1/U4: `lurch_max` ~1.1 against a [0,1] output range). + +**Rough cost.** A day for the port plus batch-convergence tests — but it must come +BEFORE any retuning of `geo_lr`, `kGeometricPushScale` or the neg-LR base, or those +constants get tuned twice. + +### 6b. The geometric dislike was ported from a superseded upstream design (2026-07-25) + +**What.** `geo_push.hpp`/`replay.hpp` cite `memllib @ 0a541cc`. `upstream/main` now pins +`e291192`, where the same code has been deliberately redesigned. Upstream: +`kGeometricPushScale` 1.0 (ours 0.5); neg-LR base 1.5 (ours 0.5, `geo_push.hpp:92`); +the `/(1+len)` taper **deleted**, with the comment "a 'no' should clearly move the +mapping away even from a sound already far from the liked region (the taper used to +kill exactly that case)" — ours still applies it at `geo_push.hpp:66`; negatives trained +as a **batch over ALL of them every tick** rather than one item one step; and a fixed +`kDislikeLifetimeMs = 2500` full-strength lifetime replacing the proportional decay we +ported. On the shared constants (dedup radius 0.05, `kCentroidK` 4) we match. + +**Why it blocks the mission.** We are carrying a design upstream diagnosed and fixed, +and the fix is documented in their source comments. Compounded with defect 6 the ported +dislike is ~9.4x weaker on constants alone before the optimiser mismatch. + +**Rough cost.** Small once defect 6 lands — mostly deleting the taper and re-basing +three constants, then re-running `scripts/bench-ml.sh` D1/A4/A7 to confirm. + +### 6c. `InterfaceRL` — the reference implementation — is not in the tree (2026-07-25) + +**What.** It lives in `memllib/examples/`, and the vendoring dropped `examples/` +(`VENDORED.md`). So the source of truth for our most contested subsystem is absent, and +the divergences in 6b went unnoticed for months. Either vendor +`examples/InterfaceRL.{hpp,cpp,tpp}` read-only alongside the rest, or record its pinned +commit and a fetch recipe in `VENDORED.md`. ## Open mission questions diff --git a/MAP.md b/MAP.md index 0dbe7f1..0a031f6 100644 --- a/MAP.md +++ b/MAP.md @@ -125,6 +125,8 @@ includes; no `nisps-core`. ### `tests/cpp/` — host C++ tests - Per-component tests: `test_dsp_*.cpp`, `test_engine_*.cpp`, `test_mlp_*.cpp`, `test_mode_*.cpp`, `test_ring_buffer.cpp`, `test_rng.cpp`, `test_math.cpp`. Helpers in `test_helpers.hpp`. - Verification: `ml_golden_vectors.cpp`, `engine_impulse.cpp` (+ `engine_impulse_baseline.bin`), `parity_check.cpp` + `parity_wasm.mjs` + `parity_diff.mjs` — native-vs-WASM bit-equivalence within 1e-5. +- Behaviour: `test_ml_behaviour.cpp` — 20 INVARIANTS of the interaction model (not tuning). Asserts what must hold however the knobs are set: a like is reachable and a re-like overwrites; dislike never yields non-finite weights/outputs in either `AvoidStyle`, and is safe at cold start; repeat dislikes inside `kReplayDedupRadius` deepen ONE negative while distant ones store separately; explore→reroll→undo→exit restores weights *bit-exactly* and over-undoing is safe; `RandomiseOutputs` leaves weights untouched (the outputs-vs-weights randomisation distinction); same seed + same gesture sequence ⇒ bit-identical weights (what makes any behavioural benchmark comparable); example-ring overflow keeps the newest; contradictory examples stay finite; the ExploreAndPlace accessor contract (`placed_output()` while Placing, `committed_output()` after commit — for both place and reposition); a FULLY-masked focus gate freezes every weight; switching mode mid-exploration never strands the net in a randomised scratchpad; and all of it holds at 1×1, 2×8, 1×33, 8×2 and 32×8 shapes. +- Measurement (asserts nothing): `ml_bench.cpp` — the BEHAVIOURAL benchmark. NISPS is a controller, so this measures the shape of the control→parameter mapping and how interaction journeys deform it, never loss alone. Shape-agnostic (`--shape N_IN,H1,H2,H3,N_OUT`, default `2,16,16,16,8`) via `MLPCore`, so "does a wider/deeper net change the UX?" becomes a number; sample points come from a deterministic Kronecker low-discrepancy sequence rather than a raster, which is what makes it work at any input arity. Field metrics: local gain p50/p95, cliff index, dead fraction, range utilisation, rail occupancy, effective dimensionality (participation ratio — trace²/‖C‖²_F, no eigendecomposition). Displacement metrics: at-point, rings, global, **blast ratio**, and collateral damage at the protected positives. **61 scenarios.** Diagnostic (D1 — the geometric-dislike dose decomposed: intended push vs effective LR vs measured movement at 1/10/100/1000 presses). Atomic probes A1–A14: at/around/far-from an example; one dislike under BOTH candidate designs; twice at one point; adjacent-then-return across the dedup radius; near a protected positive; roll-a-patch-and-place; the full explore→audition→place→commit lifecycle; explore-then-cancel; reposition; like-then-dislike in place; dislike-then-repair; focus/solo mask leakage. Journeys J1–J11: positive-only retention curve, randomise-place-only, mixed, branch (one shared prefix, three divergent gestures, replayed from scratch per branch because that is exact under a deterministic RNG), explore-place-only, 120-gesture long session with drift + weight-norm checkpoints, dislike storm, revisit-after-wandering, two-region interference, sweep-and-teach along a continuous path, undo-heavy. Edge cases E1–E13: cold start, single example, contradictory, collinear, corners, capacity overflow, undo exhaustion, minimal shape, identical targets, rail targets, rapid like/dislike alternation, mode-switch mid-exploration, fully-masked dislike. Upstream comparison U1–U3: the older memllib `interfaceRL` (a DDPG actor-critic, recoverable from this repo's own git history at blob `755ff8b`) differs structurally — the user HEARS `actorTarget`, a soft copy updated `target += alpha*(online-target)` at alpha=0.005, and it trains a batch of 4 from replay only every `optimiseDivisor=40` gestures. The critic half is not reproducible here (MLPCore has `train_targets` but not the per-layer gradient extraction the policy-gradient step needs), but both OUTPUT-PATH ideas are: U1 sweeps the soft-target alpha (alpha=1 IS NISPS today, a free control), U2 sweeps the train-every-Nth divisor, U3 compares actor shapes, U4 sweeps the positive-path training dose. **U4 carries a load-bearing caveat**: upstream LRs are RMSProp LRs (memlp `Layer.h:239`, `StaticMLP.h:268`) and `nisps/ml/training.hpp` is SGD-only, so the numbers are not comparable — see ALIGNMENT.md defect 6. Their shared metric is **lurch** — how far the mapping the musician is playing moves per single gesture, averaged over the whole field. Knobs: `--spread` (1 = Xavier, 0 = uniform with NO fan_in coupling — i.e. the post-removal behaviour, measurable before paying for the refactor), `--geo-lr`, `--geo-iters`. Driven by `scripts/bench-ml.sh`. **Two contracts that fail SILENTLY and are pinned by tests:** (1) a thumbs-up must go through BOTH `mlp.add_example` AND `fb.store_positive` — `dislike_geometric` k-NNs the replay buffer, not the MLP dataset, so a harness that only calls `add_example` measures the cold-start branch instead; (2) `placed_output()` is valid ONLY while state is `Placing` — after `commit_place()`/`commit_reposition()` the vector moves to `committed_output()`, and reading the wrong one yields an empty span whose `l2()` is 0, i.e. a broken lifecycle scored as a perfect placement. - Measurement (asserts nothing): `engine_bench.cpp` + `bench_report.mjs` — per-engine throughput (ns/sample, blocks/s, realtime factor) for the `process()` hot path. ONE source compiled twice (CMake `nisps_engine_bench` natively, emcc for WASM) so the two targets are comparable without adding a single export to `nisps/wasm/bindings.cpp`. Engines are driven into a working state (transport running + event drain for the sequencers, periodic `note_on` for paf_synth, a noise+sine input bed for the fx/analysis engines) and every row prints its own working-state evidence, so a number produced by an idle engine is visible rather than plausible. Driven by `scripts/bench-engines.sh`. ### `scripts/` — build + verify entry points @@ -132,6 +134,7 @@ includes; no `nisps-core`. - `build-wasm.sh` — Emscripten compile producing `manifold/public/nisps.{wasm,js}`. - `build-cpp-tests.sh` — CMake configure + build + ctest (Ninja). - `parity-check.sh` — runs native + WASM and diffs binary outputs. +- `bench-ml.sh` — the ML BEHAVIOUR benchmark on native + WASM from one source (same trick as `bench-engines.sh`). `--shape`, `--scenario`, `--smoke`, `--seed`, `--compare`, and `--sweep-shape` (runs the corpus across a ladder of architectures and arities — the knob-sensitivity instrument). **Reports, never asserts**: a cliff index is a description, not a pass/fail. Invariants live in `tests/cpp/test_ml_behaviour.cpp` instead. Reports land in `nisps/build/bench-ml/`. - `bench-engines.sh` — engine throughput on native + WASM; `--compare ` prints per-engine Δ%. **Reports, never asserts** (a wall-clock threshold on shared hardware is meaningless or flaky — same call as the firmware size job). Reports land in `nisps/build/bench/`. - `lint-cpp.sh` — `.f` literal warn + heap/`Arduino.h` violation fail. - `run-all-tests.sh` — master verification script. @@ -158,6 +161,7 @@ includes; no `nisps-core`. - **Host C++ tests**: `bash scripts/build-cpp-tests.sh`. - **Parity check**: `bash scripts/parity-check.sh`. - **Engine benchmark**: `bash scripts/bench-engines.sh` (add `--compare nisps/build/bench/latest.json` to diff against the previous run). +- **ML behaviour benchmark**: `bash scripts/bench-ml.sh` (`--smoke` for a fast run, `--shape 2,16,16,16,8`, `--sweep-shape` for the architecture/arity sweep, `--compare` to diff). - **All tests**: `bash scripts/run-all-tests.sh` (stage 6 is a bench smoke report; it does not gate). - **Playwright**: `cd manifold && node node_modules/.bin/playwright test` (non-snap node runner on the VPS — BUILD-PLAN gotcha; `bunx playwright test` works elsewhere). - **Codegen**: `cd codegen && bun run generate.ts` (regenerates `nisps/modes/generated/` + `nisps/ml/generated/` C++ and `manifold/src/modes/generated/` TS). diff --git a/nisps/CMakeLists.txt b/nisps/CMakeLists.txt index 7d34d41..42b6853 100644 --- a/nisps/CMakeLists.txt +++ b/nisps/CMakeLists.txt @@ -61,6 +61,7 @@ if(NOT EMSCRIPTEN) ${NISPS_TEST_DIR}/test_mlp_ou_noise.cpp ${NISPS_TEST_DIR}/test_mlp_feedback.cpp ${NISPS_TEST_DIR}/test_mlp_geo_dislike.cpp + ${NISPS_TEST_DIR}/test_ml_behaviour.cpp ${NISPS_TEST_DIR}/test_mlp_serialize.cpp ${NISPS_TEST_DIR}/test_pipeline.cpp ${NISPS_TEST_DIR}/test_vcv_iml_parity.cpp @@ -189,6 +190,29 @@ if(NOT EMSCRIPTEN) target_compile_options(nisps_engine_bench PRIVATE /W4 /WX) endif() + # --------------------------------------------------------------------- + # Standalone ML BEHAVIOUR benchmark. Same discipline as nisps_engine_bench: + # it reports numbers and asserts nothing, so it is not a ctest gate. It + # measures the shape of the control mapping and how interaction journeys + # deform it — see MAP.md § `tests/cpp/`. Driven by + # scripts/bench-ml.sh, which compiles the SAME source to WASM via emcc. + # + # Uses MLPCore, so it is host-only by construction + # (dynamic_storage.hpp #errors on RP2350). That is correct: the point is + # arbitrary runtime shapes, which firmware deliberately cannot have. + # --------------------------------------------------------------------- + add_executable(nisps_ml_bench + ${NISPS_TEST_DIR}/ml_bench.cpp + ) + target_link_libraries(nisps_ml_bench PRIVATE nisps_core) + if(CMAKE_CXX_COMPILER_ID MATCHES "GNU|Clang|AppleClang") + target_compile_options(nisps_ml_bench PRIVATE + -Wall -Wextra -Werror -Wpedantic -O3 + ) + elseif(MSVC) + target_compile_options(nisps_ml_bench PRIVATE /W4 /WX) + endif() + # Standalone parity-check runner. NOT registered with ctest — it's # invoked from scripts/parity-check.sh which orchestrates native+WASM # together. diff --git a/scripts/bench-ml.sh b/scripts/bench-ml.sh new file mode 100755 index 0000000..8d8c7cf --- /dev/null +++ b/scripts/bench-ml.sh @@ -0,0 +1,180 @@ +#!/usr/bin/env bash +# scripts/bench-ml.sh — behavioural benchmark for the NISPS control mapping. +# +# Sibling to bench-engines.sh, and deliberately the same shape: it builds +# tests/cpp/ml_bench.cpp twice from ONE source (native via CMake, WASM via +# emcc with the flags build-wasm.sh uses), runs both, and reports. +# +# NOTHING HERE FAILS. Behaviour is not a threshold — a "cliff index of 4.9" is +# neither pass nor fail, it is a description. Regressions are noticed by +# running with --compare against a previous report, which prints per-metric Δ. +# Invariants that genuinely must hold are ctest assertions in +# tests/cpp/test_ml_behaviour.cpp, not here. +# +# WHY BOTH TARGETS +# ---------------- +# Firmware and browser must feel the same. Native-vs-WASM here is a BEHAVIOURAL +# comparison, which parity-check.sh's 1e-5 bit-equivalence does not give you: +# parity proves two builds of the SAME commit agree, and says nothing about +# whether a mapping is playable or whether a gesture does anything. +# +# Usage: +# scripts/bench-ml.sh # native + wasm +# scripts/bench-ml.sh --native-only +# scripts/bench-ml.sh --smoke # fast, proves it runs +# scripts/bench-ml.sh --shape 2,16,16,16,8 # any net shape +# scripts/bench-ml.sh --sweep-shape # the architecture sweep +# scripts/bench-ml.sh --scenario A4_negative_once +# scripts/bench-ml.sh --compare old.json +# scripts/bench-ml.sh --out bench-ml-2026-07-25.json +# +# Env: +# NISPS_BUILD_DIR default nisps/build +# EMCC emcc path (same convention as build-wasm.sh) +# +# Exit codes: 0 on a completed run, 2 on bad args/missing artifacts, 3 on build +# failure. + +set -euo pipefail + +ROOT="$(cd "$(dirname "$0")/.." && pwd)" +BUILD_DIR="${NISPS_BUILD_DIR:-$ROOT/nisps/build}" +BENCH_DIR="$BUILD_DIR/bench-ml" +NATIVE_BIN="$BUILD_DIR/nisps_ml_bench" +SRC="$ROOT/tests/cpp/ml_bench.cpp" +REPORT="$ROOT/tests/cpp/ml_bench_report.mjs" +EMCC="${EMCC:-emcc}" + +NATIVE_ONLY=0 +SMOKE="" +SHAPE="" +SCENARIO="" +COMPARE="" +OUT="" +SWEEP_SHAPE=0 +SEED="" + +while [[ $# -gt 0 ]]; do + case "$1" in + --native-only) NATIVE_ONLY=1; shift ;; + --smoke) SMOKE="--smoke"; shift ;; + --shape) SHAPE="$2"; shift 2 ;; + --seed) SEED="$2"; shift 2 ;; + --scenario) SCENARIO="$2"; shift 2 ;; + --compare) COMPARE="$2"; shift 2 ;; + --out) OUT="$2"; shift 2 ;; + --sweep-shape) SWEEP_SHAPE=1; shift ;; + -h|--help) sed -n '2,40p' "$0"; exit 0 ;; + *) echo "bench-ml: unknown arg '$1'" >&2; exit 2 ;; + esac +done + +mkdir -p "$BENCH_DIR" + +bench_args() { + local a=() + [[ -n "$SMOKE" ]] && a+=("$SMOKE") + [[ -n "$SHAPE" ]] && a+=(--shape "$SHAPE") + [[ -n "$SEED" ]] && a+=(--seed "$SEED") + [[ -n "$SCENARIO" ]] && a+=(--scenario "$SCENARIO") + printf '%s\n' "${a[@]:-}" +} + +# --------------------------------------------------------------------------- +# native +# --------------------------------------------------------------------------- +echo "==> building native ml_bench" >&2 +if ! cmake -S "$ROOT/nisps" -B "$BUILD_DIR" -G Ninja -DCMAKE_BUILD_TYPE=Release >/dev/null 2>&1; then + echo "bench-ml: cmake configure failed" >&2; exit 3 +fi +if ! cmake --build "$BUILD_DIR" --target nisps_ml_bench >/dev/null 2>&1; then + echo "bench-ml: native build failed" >&2; exit 3 +fi + +mapfile -t ARGS < <(bench_args) +NATIVE_JSON="$BENCH_DIR/native.json" +if [[ ${#ARGS[@]} -gt 0 && -n "${ARGS[0]}" ]]; then + "$NATIVE_BIN" "${ARGS[@]}" > "$NATIVE_JSON" +else + "$NATIVE_BIN" > "$NATIVE_JSON" +fi +echo "==> native report: $NATIVE_JSON" >&2 + +# --------------------------------------------------------------------------- +# wasm — one source, second compiler. Same flags build-wasm.sh uses for the +# shipped module, so the comparison is against what actually ships. +# --------------------------------------------------------------------------- +WASM_JSON="" +if [[ $NATIVE_ONLY -eq 0 ]]; then + if ! command -v "$EMCC" >/dev/null 2>&1; then + echo "bench-ml: emcc not found; skipping wasm (use --native-only to silence)" >&2 + else + echo "==> building wasm ml_bench" >&2 + WASM_JS="$BENCH_DIR/ml_bench.mjs" + if ! "$EMCC" "$SRC" -o "$WASM_JS" \ + -std=gnu++20 -O3 \ + -s ENVIRONMENT=node -s EXIT_RUNTIME=1 -s ALLOW_MEMORY_GROWTH=1 \ + -s SINGLE_FILE=1 >/dev/null 2>&1; then + echo "bench-ml: wasm build failed" >&2; exit 3 + fi + WASM_JSON="$BENCH_DIR/wasm.json" + if [[ ${#ARGS[@]} -gt 0 && -n "${ARGS[0]}" ]]; then + node "$WASM_JS" "${ARGS[@]}" > "$WASM_JSON" + else + node "$WASM_JS" > "$WASM_JSON" + fi + echo "==> wasm report: $WASM_JSON" >&2 + fi +fi + +# --------------------------------------------------------------------------- +# architecture sweep — the "how does net shape change the UX" instrument. +# Runs the corpus at a ladder of shapes and emits one combined report. +# --------------------------------------------------------------------------- +if [[ $SWEEP_SHAPE -eq 1 ]]; then + SWEEP_JSON="$BENCH_DIR/sweep-shape.json" + echo "==> shape sweep" >&2 + { + echo "[" + first=1 + for s in \ + "2,4,4,4,8" \ + "2,8,8,8,8" \ + "2,16,16,16,8" \ + "2,32,32,32,8" \ + "2,64,64,64,8" \ + "1,16,16,16,8" \ + "4,16,16,16,8" \ + "8,16,16,16,8" \ + "32,16,16,16,8" \ + "2,16,16,16,1" \ + "2,16,16,16,4" \ + "2,16,16,16,16" \ + "2,16,16,16,33" + do + [[ $first -eq 0 ]] && echo "," + first=0 + "$NATIVE_BIN" --shape "$s" ${SMOKE:+$SMOKE} + done + echo "]" + } > "$SWEEP_JSON" + echo "==> sweep report: $SWEEP_JSON" >&2 +fi + +# --------------------------------------------------------------------------- +# format + diff +# --------------------------------------------------------------------------- +if [[ -f "$REPORT" ]] && command -v node >/dev/null 2>&1; then + if [[ -n "$COMPARE" ]]; then + node "$REPORT" "$NATIVE_JSON" ${WASM_JSON:+"$WASM_JSON"} --compare "$COMPARE" + else + node "$REPORT" "$NATIVE_JSON" ${WASM_JSON:+"$WASM_JSON"} + fi +else + echo "(no formatter; raw JSON is in $BENCH_DIR)" >&2 +fi + +if [[ -n "$OUT" ]]; then + command cp -f "$NATIVE_JSON" "$OUT" + echo "==> saved $OUT" >&2 +fi diff --git a/tests/cpp/ml_bench.cpp b/tests/cpp/ml_bench.cpp new file mode 100644 index 0000000..4c7c697 --- /dev/null +++ b/tests/cpp/ml_bench.cpp @@ -0,0 +1,2602 @@ +// tests/cpp/ml_bench.cpp — behavioural benchmark for the NISPS ML control +// mapping. Compiles TWICE from this one source, exactly like engine_bench.cpp: +// +// native : CMake target `nisps_ml_bench` (Release/-O3, see nisps/CMakeLists.txt) +// wasm : emcc, driven by scripts/bench-ml.sh with the same flags +// scripts/build-wasm.sh uses for the shipped module +// +// WHAT THIS IS FOR +// ---------------- +// NISPS is a CONTROLLER, not a synth. It maps a small control vector (joystick, +// pad, gamepad, MIDI CC) onto an arbitrary-range output vector that downstream +// systems bind to synth or visual parameters. Nothing here makes sound, and +// nothing here should reason about sound. The object of study is the MAPPING — +// the shape of f: control-space -> parameter-space — and the way a musician's +// interaction journey deforms it. +// +// The musician's only teaching channel is positive feedback ("I like these +// outputs HERE"), occasional negative feedback ("not this, here"), and +// exploration gestures. There is no ground truth and no test set, so loss is +// nearly worthless as a description of behaviour: it measures fit to points the +// user already dictated, which is the one thing they never experience. What +// they experience is the mapping's geometry and how their gestures move it. +// +// So this benchmark REPORTS geometry and gesture-response, never loss alone. +// +// NOTHING HERE ASSERTS. Same call as engine_bench.cpp and the firmware size +// job: a behavioural threshold is either slack enough to be meaningless or +// tight enough to fail on an unrelated change. Regressions are noticed by +// running with --compare against a previous report. Invariants that genuinely +// MUST hold (undo restores exactly, capacity never corrupts, placement takes) +// live in tests/cpp/test_ml_behaviour.cpp as real ctest assertions. +// +// DETERMINISM +// ----------- +// Every random draw in this file comes from a seeded nisps::Rng, and the net's +// own RNG is seeded per rig construction. Two runs of the same binary with the +// same --seed produce bit-identical reports. Branch points are implemented by +// REPLAYING THE PREFIX FROM SCRATCH rather than by snapshotting controller +// state — replay is exact under a deterministic RNG, needs no serialisation +// surface, and cannot drift from what the real code path does. +// +// SHAPE-AGNOSTIC BY CONSTRUCTION +// ------------------------------ +// Everything takes its dimensions from the rig, so the same corpus runs at any +// (n_in, hidden[3], n_out). That is the point: it is how "does a wider/deeper +// net change the UX?" becomes a number instead of an opinion. Because a raster +// grid is exponential in n_in, the sample set is a deterministic Kronecker +// (golden-ratio additive-recurrence) low-discrepancy sequence, which is +// well-distributed at any dimension and identical across runs and targets. +// +// A NOTE ON `spread` +// ------------------ +// `spread` is still a parameter of the core API (draw_weights/move_weights/ +// enter_explore) and this harness therefore still passes it. It is pinned to +// kSpread below so it is a constant of the experiment rather than a free knob. +// When the spread knob is removed from the core, delete kSpread and the call +// sites follow. Note that removing the KNOB does not remove the arity coupling +// it exposes: spread=1 is Xavier, whose scale is 1/sqrt(fan_in), so +// perturbation size still tracks input arity. See MAP.md § `tests/cpp/`. + +#include +#include +#include +#include +#include +#include +#include + +#include "../../nisps/core/rng.hpp" +#include "../../nisps/ml/dynamic_storage.hpp" +#include "../../nisps/ml/feedback.hpp" +#include "../../nisps/ml/geo_push.hpp" +#include "../../nisps/ml/mlp.hpp" + +namespace { + +using nisps::Rng; +using nisps::ml::AvoidStyle; +using nisps::ml::DynamicFeedbackStorage; +using nisps::ml::DynamicStorage; +using nisps::ml::FeedbackControllerCore; +using nisps::ml::FeedbackMode; +using nisps::ml::MLPCore; + +using Mlp = MLPCore; +using Feedback = FeedbackControllerCore; + +// Pinned constants of the experiment. These are deliberately NOT knobs: a +// benchmark whose every parameter floats cannot be compared across runs. +constexpr float kMoveSpeed = 0.1f; // nominal perturbation "speed" + +// EXCEPT these two, which are the live design questions and therefore have to +// be sweepable: +// +// spread — 1.0 is Xavier (scale 1/sqrt(fan_in)), 0.0 is plain uniform +// [-1,1] with NO fan_in coupling. `--spread 0` is exactly the +// behaviour the core would have once the spread knob and Xavier +// are removed, so this flag measures that change BEFORE paying for +// the refactor (which shifts every golden vector). +// geo_lr — feedback.hpp's default is 0.001, ported from upstream +// InterfaceRL.hpp:312. Upstream applied it inside a multi-pass +// optimise() loop; NISPS applies it ONCE per press. `--geo-lr` +// makes the consequence measurable. +constexpr std::size_t kUndoDepth = 4u; +constexpr std::size_t kReplayCap = 64u; +constexpr std::size_t kProbePoints = 2048u; // sample set size for field metrics +constexpr float kNearRadius[] = {0.01f, 0.05f, 0.10f, 0.25f}; +constexpr std::size_t kNearRings = sizeof(kNearRadius) / sizeof(float); + +// --------------------------------------------------------------------------- +// Config +// --------------------------------------------------------------------------- +struct Config { + std::size_t n_in = 2u; // operator default (2026-07-25) + std::size_t hidden[3] = {16u, 16u, 16u}; + std::size_t n_out = 8u; // operator default (2026-07-25) + std::uint64_t seed = 0x5EEDu; + std::size_t max_examples = 128u; + std::string only; // run one scenario by id, empty = all + bool smoke = false; // reduced point counts, proves it still runs + float spread = 1.0f; // 1 = Xavier, 0 = uniform (post-removal) + float geo_lr = 0.001f; // feedback.hpp default + std::size_t geo_iters = 1u; // presses applied per dislike (1 = today) +}; + +std::size_t probe_points(const Config& c) { return c.smoke ? 128u : kProbePoints; } + +// --------------------------------------------------------------------------- +// Deterministic low-discrepancy sampler over [0,1]^n, mapped to [-1,1]^n. +// +// Additive recurrence with the generalised golden ratio: x_k = frac(k * a_i), +// a_i = phi_d^-(i+1) where phi_d solves x^(d+1) = x + 1. Well-distributed at +// any dimension, needs no state beyond k, identical on every target. +// --------------------------------------------------------------------------- +class Kronecker { + public: + explicit Kronecker(std::size_t dim) : dim_(dim), alpha_(dim) { + // Solve x^(d+1) = x + 1 by fixed-point iteration; converges fast. + double phi = 2.0; + for (int it = 0; it < 64; ++it) { + phi = std::pow(1.0 + phi, 1.0 / static_cast(dim + 1u)); + } + double a = 1.0; + for (std::size_t i = 0; i < dim; ++i) { + a /= phi; + alpha_[i] = a; + } + } + + // Write sample k into `out` (dim floats), mapped to [-1, 1]. + void point(std::size_t k, std::span out) const { + const double kk = static_cast(k) + 0.5; + for (std::size_t i = 0; i < dim_ && i < out.size(); ++i) { + double v = kk * alpha_[i]; + v -= std::floor(v); + out[i] = static_cast(v * 2.0 - 1.0); + } + } + + private: + std::size_t dim_; + std::vector alpha_; +}; + +// --------------------------------------------------------------------------- +// Rig — an MLP + feedback controller at an arbitrary shape, plus the sample +// set the field metrics are computed over. +// --------------------------------------------------------------------------- +struct Rig { + Config cfg; + Mlp mlp; + Feedback fb; + Rng rng; // scenario-side RNG (never the net's) + Kronecker sampler; + std::vector probe_pts; // n_pts * n_in + std::vector scratch_out; // n_pts * n_out + + explicit Rig(const Config& c) + : cfg(c), + mlp(c.seed, + c.n_in, + std::span(c.hidden, 3u), + c.n_out, + c.max_examples, + 4096u), + fb(c.seed ^ 0xF33DBACCull, + c.n_out, + mlp.weight_count(), + kUndoDepth, + c.n_in, + kReplayCap), + rng(c.seed ^ 0xA5A5A5A5ull), + sampler(c.n_in) { + const std::size_t n = probe_points(c); + probe_pts.resize(n * c.n_in); + scratch_out.resize(n * c.n_out); + for (std::size_t k = 0; k < n; ++k) { + sampler.point(k, std::span(&probe_pts[k * c.n_in], c.n_in)); + } + fb.set_geo_lr(c.geo_lr); + // MLPCore's ctor draws at spread=1; re-draw when the experiment asks + // for a different init regime. Same RNG stream either way. + if (c.spread != 1.0f) mlp.draw_weights(c.spread); + } + + // Every call site reads the regime off the rig rather than a constant, so + // --spread reaches the RL perturbation path too, not just init. + float spread() const { return cfg.spread; } + + std::size_t n_pts() const { return probe_pts.size() / cfg.n_in; } + + // Infer over the whole sample set into `dst` (n_pts * n_out). + void field(std::vector& dst) { + dst.resize(n_pts() * cfg.n_out); + mlp.infer_batch(probe_pts, dst); + } + + // Infer at one point. + void at(std::span x, std::vector& out) { + out.resize(cfg.n_out); + for (std::size_t i = 0; i < cfg.n_in; ++i) mlp.set_input(i, x[i]); + mlp.process(); + auto o = mlp.outputs(); + for (std::size_t j = 0; j < cfg.n_out; ++j) out[j] = o[j]; + } +}; + +// --------------------------------------------------------------------------- +// Small numeric helpers +// --------------------------------------------------------------------------- +float l2(std::span a, std::span b) { + float acc = 0.f; + const std::size_t n = a.size() < b.size() ? a.size() : b.size(); + for (std::size_t i = 0; i < n; ++i) { + const float d = a[i] - b[i]; + acc += d * d; + } + return std::sqrt(acc); +} + +float percentile(std::vector v, float p) { + if (v.empty()) return 0.f; + // Deterministic partial selection: full sort. n is a few thousand; the + // cost is irrelevant next to the inference it summarises, and a full sort + // has no tie-break ambiguity across targets. + for (std::size_t i = 1; i < v.size(); ++i) { + float key = v[i]; + std::size_t j = i; + while (j > 0 && v[j - 1] > key) { v[j] = v[j - 1]; --j; } + v[j] = key; + } + const float idx = p * static_cast(v.size() - 1); + const std::size_t lo = static_cast(idx); + const std::size_t hi = (lo + 1u < v.size()) ? lo + 1u : lo; + const float frac = idx - static_cast(lo); + return v[lo] * (1.f - frac) + v[hi] * frac; +} + +float mean(const std::vector& v) { + if (v.empty()) return 0.f; + float s = 0.f; + for (float x : v) s += x; + return s / static_cast(v.size()); +} + +// --------------------------------------------------------------------------- +// FIELD METRICS — the static geometry of the mapping. +// --------------------------------------------------------------------------- +struct FieldMetrics { + float gain_p50 = 0.f; // median ||J|| over the sample set + float gain_p95 = 0.f; + float cliff_index = 0.f; // p95 / p50 — coexisting cliffs and dead zones + float dead_frac = 0.f; // share of points with gain < 10% of median + float range_util = 0.f; // mean over outputs of (p99-p1), outputs in [0,1] + float rail_frac = 0.f; // share of (point,out) within 1% of a rail + float eff_dim = 0.f; // participation ratio of the output covariance + float eff_dim_norm = 0.f; // eff_dim / n_out +}; + +// Frobenius norm of the Jacobian at x, by central differences. +float jacobian_norm(Rig& rig, std::span x, float h) { + std::vector xp(rig.cfg.n_in), xm(rig.cfg.n_in); + std::vector op, om; + float acc = 0.f; + for (std::size_t i = 0; i < rig.cfg.n_in; ++i) { + for (std::size_t k = 0; k < rig.cfg.n_in; ++k) { xp[k] = x[k]; xm[k] = x[k]; } + xp[i] += h; + xm[i] -= h; + rig.at(xp, op); + rig.at(xm, om); + for (std::size_t j = 0; j < rig.cfg.n_out; ++j) { + const float d = (op[j] - om[j]) / (2.f * h); + acc += d * d; + } + } + return std::sqrt(acc); +} + +FieldMetrics measure_field(Rig& rig) { + FieldMetrics m; + const std::size_t n = rig.n_pts(); + const std::size_t nin = rig.cfg.n_in; + const std::size_t nout = rig.cfg.n_out; + + std::vector outs; + rig.field(outs); + + // --- gain ------------------------------------------------------------- + // Sub-sample the Jacobian: it costs 2*n_in inferences per point, which at + // 2048 points x 32 inputs would dominate the whole run for no extra + // resolution. 256 points is plenty for a p50/p95 and is deterministic. + const std::size_t n_jac = (n < 256u) ? n : 256u; + const std::size_t stride = n / n_jac; + std::vector gains; + gains.reserve(n_jac); + for (std::size_t k = 0; k < n_jac; ++k) { + const std::size_t idx = k * stride; + gains.push_back(jacobian_norm( + rig, std::span(&rig.probe_pts[idx * nin], nin), 1e-3f)); + } + m.gain_p50 = percentile(gains, 0.50f); + m.gain_p95 = percentile(gains, 0.95f); + m.cliff_index = (m.gain_p50 > 1e-9f) ? (m.gain_p95 / m.gain_p50) : 0.f; + + const float dead_thresh = 0.1f * m.gain_p50; + std::size_t dead = 0u; + for (float g : gains) if (g < dead_thresh) ++dead; + m.dead_frac = gains.empty() ? 0.f : static_cast(dead) / static_cast(gains.size()); + + // --- range utilisation + rails ---------------------------------------- + // Network outputs are sigmoid, so [0,1] is the full nominal range. Range + // mapping (min/max/curve) happens DOWNSTREAM of the net and is deliberately + // not modelled here — this measures what the net itself offers. + float util_acc = 0.f; + std::size_t rails = 0u; + for (std::size_t j = 0; j < nout; ++j) { + std::vector col; + col.reserve(n); + for (std::size_t k = 0; k < n; ++k) { + const float v = outs[k * nout + j]; + col.push_back(v); + if (v < 0.01f || v > 0.99f) ++rails; + } + util_acc += percentile(col, 0.99f) - percentile(col, 0.01f); + } + m.range_util = (nout > 0u) ? util_acc / static_cast(nout) : 0.f; + m.rail_frac = (n * nout > 0u) + ? static_cast(rails) / static_cast(n * nout) : 0.f; + + // --- effective dimensionality ---------------------------------------- + // Participation ratio PR = (sum lambda)^2 / sum lambda^2. Because + // sum lambda = trace(C) and sum lambda^2 = ||C||_F^2, this needs no + // eigendecomposition — exact, cheap, and target-stable. + std::vector mu(nout, 0.f); + for (std::size_t k = 0; k < n; ++k) + for (std::size_t j = 0; j < nout; ++j) mu[j] += outs[k * nout + j]; + for (std::size_t j = 0; j < nout; ++j) mu[j] /= static_cast(n); + + std::vector cov(nout * nout, 0.f); + for (std::size_t k = 0; k < n; ++k) { + for (std::size_t a = 0; a < nout; ++a) { + const float da = outs[k * nout + a] - mu[a]; + for (std::size_t b = 0; b < nout; ++b) { + cov[a * nout + b] += da * (outs[k * nout + b] - mu[b]); + } + } + } + const float inv = 1.f / static_cast(n); + float trace = 0.f, fro2 = 0.f; + for (std::size_t a = 0; a < nout; ++a) { + for (std::size_t b = 0; b < nout; ++b) { + cov[a * nout + b] *= inv; + fro2 += cov[a * nout + b] * cov[a * nout + b]; + } + trace += cov[a * nout + a]; + } + m.eff_dim = (fro2 > 1e-20f) ? (trace * trace) / fro2 : 0.f; + m.eff_dim_norm = (nout > 0u) ? m.eff_dim / static_cast(nout) : 0.f; + return m; +} + +// --------------------------------------------------------------------------- +// DISPLACEMENT — how much a gesture moved the mapping, and WHERE. +// +// This is the core measurement for negative feedback. `before`/`after` are +// full fields over the sample set. We report displacement at the pressed +// point, in rings around it, at the stored positive examples, and globally. +// The ratio local/global is the "blast radius" — a good dislike is local. +// --------------------------------------------------------------------------- +struct Displacement { + float at_point = 0.f; + float ring[kNearRings] = {0.f, 0.f, 0.f, 0.f}; + float global_p50 = 0.f; + float global_p95 = 0.f; + float blast_ratio = 0.f; // at_point / global_p50; high = local, low = smeared + float positives_mean = 0.f; // mean displacement at stored positive positions + float positives_max = 0.f; +}; + +Displacement measure_displacement(Rig& rig, + const std::vector& before, + const std::vector& after, + std::span pressed, + const std::vector& positive_xs) { + Displacement d; + const std::size_t n = rig.n_pts(); + const std::size_t nin = rig.cfg.n_in; + const std::size_t nout = rig.cfg.n_out; + + std::vector per_pt; + per_pt.reserve(n); + for (std::size_t k = 0; k < n; ++k) { + per_pt.push_back(l2(std::span(&before[k * nout], nout), + std::span(&after[k * nout], nout))); + } + d.global_p50 = percentile(per_pt, 0.50f); + d.global_p95 = percentile(per_pt, 0.95f); + + // Nearest sample point to the pressed position stands in for "at_point"; + // the sample set is dense enough that this is a fair proxy and it keeps + // before/after strictly comparable (same evaluation points). + std::size_t best = 0u; + float best_d = 1e30f; + for (std::size_t k = 0; k < n; ++k) { + const float dd = l2(std::span(&rig.probe_pts[k * nin], nin), pressed); + if (dd < best_d) { best_d = dd; best = k; } + } + d.at_point = per_pt[best]; + d.blast_ratio = (d.global_p50 > 1e-9f) ? d.at_point / d.global_p50 : 0.f; + + // Rings: mean displacement among points whose distance to `pressed` falls + // in [r_prev, r]. Empty rings report 0 (visible as such in the report). + for (std::size_t r = 0; r < kNearRings; ++r) { + const float lo = (r == 0u) ? 0.f : kNearRadius[r - 1u]; + const float hi = kNearRadius[r]; + float acc = 0.f; + std::size_t cnt = 0u; + for (std::size_t k = 0; k < n; ++k) { + const float dd = l2(std::span(&rig.probe_pts[k * nin], nin), pressed); + if (dd >= lo && dd < hi) { acc += per_pt[k]; ++cnt; } + } + d.ring[r] = cnt ? acc / static_cast(cnt) : 0.f; + } + + // Collateral damage at the stored positives. + const std::size_t n_pos = positive_xs.size() / (nin ? nin : 1u); + float acc = 0.f, mx = 0.f; + for (std::size_t p = 0; p < n_pos; ++p) { + std::span px(&positive_xs[p * nin], nin); + std::size_t bi = 0u; float bd = 1e30f; + for (std::size_t k = 0; k < n; ++k) { + const float dd = l2(std::span(&rig.probe_pts[k * nin], nin), px); + if (dd < bd) { bd = dd; bi = k; } + } + acc += per_pt[bi]; + if (per_pt[bi] > mx) mx = per_pt[bi]; + } + d.positives_mean = n_pos ? acc / static_cast(n_pos) : 0.f; + d.positives_max = mx; + return d; +} + +// --------------------------------------------------------------------------- +// EXAMPLE LAYOUTS — the "given these training examples" half of the question. +// Deterministic, shape-agnostic, and named so a report row is legible. +// --------------------------------------------------------------------------- +struct Dataset { + std::vector xs; // n * n_in + std::vector ys; // n * n_out + std::size_t n = 0u; +}; + +Dataset make_dataset(const Config& cfg, const char* layout, std::size_t count, Rng& rng) { + Dataset ds; + ds.n = count; + ds.xs.resize(count * cfg.n_in); + ds.ys.resize(count * cfg.n_out); + + Kronecker k_in(cfg.n_in); + Kronecker k_out(cfg.n_out); + + for (std::size_t i = 0; i < count; ++i) { + std::span x(&ds.xs[i * cfg.n_in], cfg.n_in); + std::span y(&ds.ys[i * cfg.n_out], cfg.n_out); + + if (std::strcmp(layout, "scattered") == 0) { + k_in.point(i * 7u + 3u, x); + } else if (std::strcmp(layout, "clustered") == 0) { + // All examples inside a small ball — the "I only played in one + // corner" case, which is what most real sessions look like. + k_in.point(i * 7u + 3u, x); + for (std::size_t j = 0; j < cfg.n_in; ++j) x[j] = 0.3f + 0.15f * x[j]; + } else if (std::strcmp(layout, "corners") == 0) { + for (std::size_t j = 0; j < cfg.n_in; ++j) { + x[j] = ((i >> (j % 8u)) & 1u) ? 1.f : -1.f; + } + } else if (std::strcmp(layout, "collinear") == 0) { + // Degenerate: every example on one line through the space. + const float t = (count > 1u) + ? (2.f * static_cast(i) / static_cast(count - 1u) - 1.f) : 0.f; + for (std::size_t j = 0; j < cfg.n_in; ++j) x[j] = t; + } else if (std::strcmp(layout, "coincident") == 0) { + // Pathological: every example at the SAME input position, with + // different targets. Tests contradictory teaching. + for (std::size_t j = 0; j < cfg.n_in; ++j) x[j] = 0.f; + } else { // "uniform" + k_in.point(i, x); + } + + // Targets: a deterministic spread of output vectors. `randomised` + // draws from the scenario RNG so that the "randomise a patch then + // place it" journey uses genuinely unstructured targets. + if (std::strcmp(layout, "randomised") == 0) { + for (std::size_t j = 0; j < cfg.n_out; ++j) { + y[j] = 0.5f + 0.5f * rng.next_float_signed(); + } + } else { + k_out.point(i * 11u + 5u, y); + for (std::size_t j = 0; j < cfg.n_out; ++j) y[j] = 0.5f + 0.5f * y[j]; + } + } + return ds; +} + +// --------------------------------------------------------------------------- +// Report emission. Plain JSON on stdout — bench-ml.sh captures it and +// tests/cpp/ml_bench_report.mjs formats + diffs it, mirroring bench_report.mjs. +// --------------------------------------------------------------------------- +class Json { + public: + void begin_run(const Config& c) { + printf("{\n"); + printf(" \"schema\": \"nisps-ml-bench/1\",\n"); + printf(" \"shape\": {\"n_in\": %zu, \"hidden\": [%zu, %zu, %zu], \"n_out\": %zu},\n", + c.n_in, c.hidden[0], c.hidden[1], c.hidden[2], c.n_out); + printf(" \"seed\": %llu,\n", static_cast(c.seed)); + printf(" \"max_examples\": %zu,\n", c.max_examples); + printf(" \"spread\": %.6g, \"geo_lr\": %.6g, \"geo_iters\": %zu,\n", + static_cast(c.spread), static_cast(c.geo_lr), c.geo_iters); + printf(" \"target\": \"%s\",\n", target_name()); + printf(" \"scenarios\": [\n"); + } + void end_run() { printf("\n ]\n}\n"); } + + void begin_scenario(const char* id, const char* what) { + if (!first_) printf(",\n"); + first_ = false; + printf(" {\"id\": \"%s\", \"what\": \"%s\", \"metrics\": {", id, what); + first_metric_ = true; + } + void end_scenario() { printf("}}"); } + + void kv(const char* k, float v) { + if (!first_metric_) printf(", "); + first_metric_ = false; + // %.6g keeps the report diffable without pretending to more precision + // than a float carries. + printf("\"%s\": %.6g", k, static_cast(v)); + } + void kv(const char* k, std::size_t v) { + if (!first_metric_) printf(", "); + first_metric_ = false; + printf("\"%s\": %zu", k, v); + } + + void field(const char* prefix, const FieldMetrics& m) { + char b[96]; + auto p = [&](const char* n) { snprintf(b, sizeof b, "%s%s", prefix, n); return b; }; + kv(p("gain_p50"), m.gain_p50); + kv(p("gain_p95"), m.gain_p95); + kv(p("cliff_index"), m.cliff_index); + kv(p("dead_frac"), m.dead_frac); + kv(p("range_util"), m.range_util); + kv(p("rail_frac"), m.rail_frac); + kv(p("eff_dim"), m.eff_dim); + kv(p("eff_dim_norm"), m.eff_dim_norm); + } + + void disp(const char* prefix, const Displacement& d) { + char b[96]; + auto p = [&](const char* n) { snprintf(b, sizeof b, "%s%s", prefix, n); return b; }; + kv(p("at_point"), d.at_point); + kv(p("ring_001"), d.ring[0]); + kv(p("ring_005"), d.ring[1]); + kv(p("ring_010"), d.ring[2]); + kv(p("ring_025"), d.ring[3]); + kv(p("global_p50"), d.global_p50); + kv(p("global_p95"), d.global_p95); + kv(p("blast_ratio"), d.blast_ratio); + kv(p("positives_mean"), d.positives_mean); + kv(p("positives_max"), d.positives_max); + } + + static const char* target_name() { +#if defined(__EMSCRIPTEN__) + return "wasm"; +#else + return "native"; +#endif + } + + private: + bool first_ = true; + bool first_metric_ = true; +}; + +// --------------------------------------------------------------------------- +// Scenario plumbing +// --------------------------------------------------------------------------- +bool selected(const Config& c, const char* id) { + return c.only.empty() || c.only == id; +} + +// Place ONE positive example the way the real product path does. +// +// This is load-bearing and easy to get wrong. A thumbs-up in firmware/browser +// does TWO things, not one: +// 1. mlp.add_example(x, y) — the supervised dataset the net trains on +// 2. fb.store_positive(mlp, y) — the feedback controller's REPLAY memory, +// a separate buffer (feedback.hpp storage, +// nisps/ml/replay.hpp algorithms) +// dislike_geometric reads (2), not (1): it k-NNs the replay positives to build +// a push-away target. A harness that only calls add_example leaves the replay +// buffer empty, so every dislike takes the documented cold-start branch +// (FeedbackAction::GeometricColdStart) and measures a path no real session +// reaches after its first like. store_positive records at the mlp's CURRENT +// input position, so the inputs must be set and processed first. +void place_positive(Rig& rig, std::span x, std::span y) { + for (std::size_t j = 0; j < rig.cfg.n_in; ++j) rig.mlp.set_input(j, x[j]); + rig.mlp.process(); + rig.mlp.add_example(x, y); + rig.fb.store_positive(rig.mlp, y); +} + +// The ExploreAndPlace accessor contract, in one place because getting it wrong +// fails SILENTLY. placed_output() is valid ONLY while state == Placing; +// commit_place()/commit_reposition() move state to Idle and hand the vector to +// committed_output() instead (feedback.hpp:583, :641). Reading placed_output() +// after commit yields an EMPTY span, and an empty span makes l2() return 0 — +// which reads as a perfect placement rather than a broken one. Returns false +// when nothing was committed, so callers can report that instead of scoring it. +bool take_committed(Rig& rig, std::vector& out) { + auto c = rig.fb.committed_output(); + if (c.size() < rig.cfg.n_out) { out.clear(); return false; } + out.assign(c.begin(), c.end()); + return true; +} + +// Train a rig on a dataset, returning the stored positive input positions +// (needed by the collateral-damage metric). +std::vector teach(Rig& rig, const Dataset& ds) { + for (std::size_t i = 0; i < ds.n; ++i) { + place_positive(rig, + std::span(&ds.xs[i * rig.cfg.n_in], rig.cfg.n_in), + std::span(&ds.ys[i * rig.cfg.n_out], rig.cfg.n_out)); + } + rig.mlp.train(); + return ds.xs; +} + +// ========================================================================= +// ATOMIC PROBES — "unit tests" of the mapping. One gesture, one measurement. +// ========================================================================= + +// A1 — at_example: infer exactly where an example was placed. How well does +// the placement hold after training? This is the floor: if this is bad, every +// downstream journey metric is meaningless. +void probe_at_example(const Config& cfg, Json& js) { + if (!selected(cfg, "A1_at_example")) return; + Rig rig(cfg); + Dataset ds = make_dataset(cfg, "scattered", 12u, rig.rng); + teach(rig, ds); + + std::vector got, errs; + for (std::size_t i = 0; i < ds.n; ++i) { + rig.at(std::span(&ds.xs[i * cfg.n_in], cfg.n_in), got); + errs.push_back(l2(got, std::span(&ds.ys[i * cfg.n_out], cfg.n_out))); + } + js.begin_scenario("A1_at_example", "infer exactly at each stored example"); + js.kv("n_examples", ds.n); + js.kv("err_mean", mean(errs)); + js.kv("err_p95", percentile(errs, 0.95f)); + js.kv("err_max", percentile(errs, 1.0f)); + js.kv("final_loss", rig.mlp.eval_loss()); + js.end_scenario(); +} + +// A2 — around_example: infer on rings at increasing radius from an example. +// How fast does the taught value decay into the surrounding field? This is +// "how big is the region my thumbs-up actually controls". +void probe_around_example(const Config& cfg, Json& js) { + if (!selected(cfg, "A2_around_example")) return; + Rig rig(cfg); + Dataset ds = make_dataset(cfg, "scattered", 12u, rig.rng); + teach(rig, ds); + + std::span x0(&ds.xs[0], cfg.n_in); + std::span y0(&ds.ys[0], cfg.n_out); + std::vector probe(cfg.n_in), got; + + js.begin_scenario("A2_around_example", "output drift on rings around one example"); + for (std::size_t r = 0; r < kNearRings; ++r) { + // Deterministic ring: perturb along each axis in turn, average. + float acc = 0.f; + std::size_t cnt = 0u; + for (std::size_t axis = 0; axis < cfg.n_in; ++axis) { + for (int sign = -1; sign <= 1; sign += 2) { + for (std::size_t j = 0; j < cfg.n_in; ++j) probe[j] = x0[j]; + probe[axis] += static_cast(sign) * kNearRadius[r]; + rig.at(probe, got); + acc += l2(got, y0); + ++cnt; + } + } + char key[32]; + snprintf(key, sizeof key, "drift_r%03d", static_cast(kNearRadius[r] * 100.f)); + js.kv(key, cnt ? acc / static_cast(cnt) : 0.f); + } + js.end_scenario(); +} + +// A3 — far_field: what does the mapping do where nothing was ever taught? +// Reported as distance from the nearest taught output, plus the field metrics +// restricted to the far region. An instrument that collapses to one value +// away from its examples is unplayable outside the taught spots. +void probe_far_field(const Config& cfg, Json& js) { + if (!selected(cfg, "A3_far_field")) return; + Rig rig(cfg); + Dataset ds = make_dataset(cfg, "clustered", 12u, rig.rng); + teach(rig, ds); + + const std::size_t n = rig.n_pts(); + std::vector outs; + rig.field(outs); + + std::vector far_dists, novelty; + for (std::size_t k = 0; k < n; ++k) { + std::span x(&rig.probe_pts[k * cfg.n_in], cfg.n_in); + float nearest = 1e30f; + std::size_t nearest_i = 0u; + for (std::size_t i = 0; i < ds.n; ++i) { + const float d = l2(x, std::span(&ds.xs[i * cfg.n_in], cfg.n_in)); + if (d < nearest) { nearest = d; nearest_i = i; } + } + if (nearest > 0.75f) { // "far" = well outside the taught cluster + far_dists.push_back(nearest); + novelty.push_back(l2(std::span(&outs[k * cfg.n_out], cfg.n_out), + std::span(&ds.ys[nearest_i * cfg.n_out], cfg.n_out))); + } + } + js.begin_scenario("A3_far_field", "mapping behaviour far from every example"); + js.kv("far_points", far_dists.size()); + js.kv("novelty_mean", mean(novelty)); + js.kv("novelty_p95", percentile(novelty, 0.95f)); + FieldMetrics fm = measure_field(rig); + js.field("field_", fm); + js.end_scenario(); +} + +// A4 — negative_once: one dislike at a point, under BOTH candidate designs. +// Reports the displacement field for each so they can be compared directly. +// This is the measurement that adjudicates the negative-feedback design. +void probe_negative_once(const Config& cfg, Json& js) { + if (!selected(cfg, "A4_negative_once")) return; + + struct Variant { const char* id; FeedbackMode mode; AvoidStyle style; }; + const Variant variants[] = { + {"A4_negative_once_geometric", FeedbackMode::Avoid, AvoidStyle::Geometric}, + {"A4_negative_once_diffuse", FeedbackMode::Avoid, AvoidStyle::Diffuse}, + }; + + for (const Variant& v : variants) { + Rig rig(cfg); + Dataset ds = make_dataset(cfg, "scattered", 12u, rig.rng); + std::vector pos_xs = teach(rig, ds); + + std::vector before; + rig.field(before); + + // Press at a point deliberately BETWEEN examples — the realistic case. + std::vector press(cfg.n_in); + for (std::size_t j = 0; j < cfg.n_in; ++j) { + press[j] = 0.5f * (ds.xs[j] + ds.xs[cfg.n_in + j]); + } + std::vector heard; + rig.at(press, heard); + + rig.fb.set_mode(v.mode, rig.mlp); + rig.fb.set_avoid_style(v.style); + for (std::size_t j = 0; j < cfg.n_in; ++j) rig.mlp.set_input(j, press[j]); + rig.mlp.process(); + rig.fb.on_down(rig.mlp, heard, kMoveSpeed, rig.spread(), {}); + + std::vector after; + rig.field(after); + Displacement d = measure_displacement(rig, before, after, press, pos_xs); + + js.begin_scenario(v.id, "one dislike between two examples"); + js.disp("", d); + js.kv("positives_after", rig.fb.positive_count()); + js.kv("negatives_after", rig.fb.negative_count()); + js.end_scenario(); + } +} + +// A5 — negative_twice_same: press dislike twice at the SAME point. Does the +// second press compound, saturate, or diverge? Replay memory deepens an +// existing negative within 0.05 rather than storing a new one +// (replay.hpp:106), so the two presses are NOT independent — this measures +// what that actually feels like. +void probe_negative_twice(const Config& cfg, Json& js) { + if (!selected(cfg, "A5_negative_twice_same")) return; + Rig rig(cfg); + Dataset ds = make_dataset(cfg, "scattered", 12u, rig.rng); + std::vector pos_xs = teach(rig, ds); + rig.fb.set_mode(FeedbackMode::Avoid, rig.mlp); + rig.fb.set_avoid_style(AvoidStyle::Geometric); + + std::vector press(cfg.n_in); + for (std::size_t j = 0; j < cfg.n_in; ++j) press[j] = 0.5f * (ds.xs[j] + ds.xs[cfg.n_in + j]); + + std::vector s0, s1, s2, heard; + rig.field(s0); + for (int press_i = 0; press_i < 2; ++press_i) { + rig.at(press, heard); + for (std::size_t j = 0; j < cfg.n_in; ++j) rig.mlp.set_input(j, press[j]); + rig.mlp.process(); + rig.fb.on_down(rig.mlp, heard, kMoveSpeed, rig.spread(), {}); + rig.field(press_i == 0 ? s1 : s2); + } + + Displacement d1 = measure_displacement(rig, s0, s1, press, pos_xs); + Displacement d2 = measure_displacement(rig, s1, s2, press, pos_xs); + + js.begin_scenario("A5_negative_twice_same", "two dislikes at the same point"); + js.disp("p1_", d1); + js.disp("p2_", d2); + // >1 means the second press did MORE than the first (compounding); + // <1 means it did less (saturating). Either is a design fact worth knowing. + js.kv("compounding", (d1.at_point > 1e-9f) ? d2.at_point / d1.at_point : 0.f); + js.kv("negatives_after", rig.fb.negative_count()); + js.end_scenario(); +} + +// A6 — negative_adjacent_then_return: dislike at x, move slightly, dislike +// again, then go BACK to x and measure whether the first dislike survived. +// This is the operator's exact scenario, and it is the one most likely to +// expose replay dedup behaving unlike what a musician expects: the 0.05 +// dedup radius means "a little to the left" may deepen the SAME negative +// rather than create a new one. +void probe_negative_adjacent(const Config& cfg, Json& js) { + if (!selected(cfg, "A6_negative_adjacent")) return; + + // Two offsets: inside the dedup radius, and outside it. + const float offsets[] = {0.02f, 0.20f}; + for (float off : offsets) { + Rig rig(cfg); + Dataset ds = make_dataset(cfg, "scattered", 12u, rig.rng); + std::vector pos_xs = teach(rig, ds); + rig.fb.set_mode(FeedbackMode::Avoid, rig.mlp); + rig.fb.set_avoid_style(AvoidStyle::Geometric); + + std::vector a(cfg.n_in), b(cfg.n_in); + for (std::size_t j = 0; j < cfg.n_in; ++j) { + a[j] = 0.5f * (ds.xs[j] + ds.xs[cfg.n_in + j]); + b[j] = a[j]; + } + b[0] -= off; // "a little to the left" + + std::vector s0, s1, s2, heard; + rig.field(s0); + + auto press_at = [&](std::span p) { + rig.at(p, heard); + for (std::size_t j = 0; j < cfg.n_in; ++j) rig.mlp.set_input(j, p[j]); + rig.mlp.process(); + rig.fb.on_down(rig.mlp, heard, kMoveSpeed, rig.spread(), {}); + }; + + press_at(a); + rig.field(s1); + press_at(b); + rig.field(s2); + + Displacement d_a = measure_displacement(rig, s0, s1, a, pos_xs); + Displacement d_b = measure_displacement(rig, s1, s2, b, pos_xs); + // Did pressing at b UNDO the effect at a? Compare the field at a + // across s0 -> s2 against s0 -> s1. + Displacement d_net = measure_displacement(rig, s0, s2, a, pos_xs); + + char id[64]; + snprintf(id, sizeof id, "A6_negative_adjacent_%03d", static_cast(off * 100.f)); + js.begin_scenario(id, "dislike, shift left, dislike again, look back at the first"); + js.kv("offset", off); + js.kv("dedup_radius", 0.05f); // replay.hpp kReplayDedupRadius + js.disp("first_", d_a); + js.disp("second_", d_b); + js.disp("net_", d_net); + // <1 means the second press partly UNDID the first at a. + js.kv("first_survives", (d_a.at_point > 1e-9f) ? d_net.at_point / d_a.at_point : 0.f); + js.kv("negatives_after", rig.fb.negative_count()); + js.end_scenario(); + } +} + +// A7 — negative_near_positive: the operator's stated design intent, made +// measurable. "I don't like this here, but DON'T disturb the positives I gave +// nearby." Places a positive example at a known distance, then dislikes next +// to it, and reports damage to that positive as a function of separation. +void probe_negative_near_positive(const Config& cfg, Json& js) { + if (!selected(cfg, "A7_negative_near_positive")) return; + const float seps[] = {0.05f, 0.15f, 0.40f}; + for (float sep : seps) { + Rig rig(cfg); + Dataset ds = make_dataset(cfg, "scattered", 8u, rig.rng); + std::vector pos_xs = teach(rig, ds); + rig.fb.set_mode(FeedbackMode::Avoid, rig.mlp); + rig.fb.set_avoid_style(AvoidStyle::Geometric); + + // The protected positive is example 0. Press `sep` away from it. + std::vector press(cfg.n_in); + for (std::size_t j = 0; j < cfg.n_in; ++j) press[j] = ds.xs[j]; + press[0] += sep; + + std::vector before, after, heard, at_pos_before, at_pos_after; + rig.field(before); + rig.at(std::span(&ds.xs[0], cfg.n_in), at_pos_before); + + rig.at(press, heard); + for (std::size_t j = 0; j < cfg.n_in; ++j) rig.mlp.set_input(j, press[j]); + rig.mlp.process(); + rig.fb.on_down(rig.mlp, heard, kMoveSpeed, rig.spread(), {}); + + rig.field(after); + rig.at(std::span(&ds.xs[0], cfg.n_in), at_pos_after); + + Displacement d = measure_displacement(rig, before, after, press, pos_xs); + + char id[64]; + snprintf(id, sizeof id, "A7_negative_near_positive_%03d", static_cast(sep * 100.f)); + js.begin_scenario(id, "dislike near a protected positive example"); + js.kv("separation", sep); + js.disp("", d); + // The headline number: how far the protected positive moved. + js.kv("protected_drift", l2(at_pos_before, at_pos_after)); + // ...relative to what the dislike achieved where it was pressed. + js.kv("damage_ratio", + (d.at_point > 1e-9f) ? l2(at_pos_before, at_pos_after) / d.at_point : 0.f); + js.end_scenario(); + } +} + +// A8 — randomise_and_place: the operator's named journey. RandomiseOutputs +// rolls a whole patch WITHOUT touching weights, the user auditions it, then +// places it as a positive example at the current input position; the net then +// retrains with that example. Distinct from RandomiseMlp, which scrambles +// weights. Measures whether the placed patch takes, and what it costs. +void probe_randomise_and_place(const Config& cfg, Json& js) { + if (!selected(cfg, "A8_randomise_and_place")) return; + Rig rig(cfg); + Dataset ds = make_dataset(cfg, "scattered", 8u, rig.rng); + std::vector pos_xs = teach(rig, ds); + + std::vector before; + rig.field(before); + + // Stand somewhere untaught. + std::vector where(cfg.n_in); + Kronecker k(cfg.n_in); + k.point(999u, where); + + std::vector heard; + rig.at(where, heard); + + rig.fb.set_mode(FeedbackMode::RandomiseOutputs, rig.mlp); + for (std::size_t j = 0; j < cfg.n_in; ++j) rig.mlp.set_input(j, where[j]); + rig.mlp.process(); + + // Down = enter randomise; down again = re-roll. Audition three patches. + rig.fb.on_down(rig.mlp, heard, kMoveSpeed, rig.spread(), {}); + std::size_t rerolls = 0u; + for (int i = 0; i < 2; ++i) { rig.fb.on_down(rig.mlp, heard, kMoveSpeed, rig.spread(), {}); ++rerolls; } + + // Take the held static vector and place it as a positive example. + std::vector patch(cfg.n_out, 0.f); + const bool have_static = rig.fb.static_output(patch); + place_positive(rig, where, patch); + rig.mlp.train(); + + std::vector after, got; + rig.field(after); + rig.at(where, got); + + Displacement d = measure_displacement(rig, before, after, where, pos_xs); + + js.begin_scenario("A8_randomise_and_place", "roll a patch (outputs only), place it as a positive"); + js.kv("rerolls", rerolls); + js.kv("have_static", static_cast(have_static ? 1u : 0u)); + js.kv("placement_err", l2(got, patch)); // did the placed patch take? + js.disp("", d); // what did placing it cost elsewhere? + js.kv("examples", ds.n + 1u); + js.end_scenario(); +} + +// ========================================================================= +// JOURNEYS — composite, multi-gesture sessions. These are the "interaction +// tests". Each reports a trajectory, not a single number. +// ========================================================================= + +// J1 — positive_only: N likes at scattered positions, retraining after each. +// Reports the RETENTION CURVE: after example k, how far have examples 1..k-1 +// drifted from what was taught? This is catastrophic forgetting, measured. +void journey_positive_only(const Config& cfg, Json& js) { + if (!selected(cfg, "J1_positive_only")) return; + Rig rig(cfg); + const std::size_t N = cfg.smoke ? 6u : 20u; + Dataset ds = make_dataset(cfg, "scattered", N, rig.rng); + + js.begin_scenario("J1_positive_only", "N likes, retention of every earlier like"); + std::vector got; + for (std::size_t k = 0; k < N; ++k) { + place_positive(rig, std::span(&ds.xs[k * cfg.n_in], cfg.n_in), + std::span(&ds.ys[k * cfg.n_out], cfg.n_out)); + rig.mlp.train(); + float acc = 0.f; + for (std::size_t i = 0; i <= k; ++i) { + rig.at(std::span(&ds.xs[i * cfg.n_in], cfg.n_in), got); + acc += l2(got, std::span(&ds.ys[i * cfg.n_out], cfg.n_out)); + } + char key[32]; + snprintf(key, sizeof key, "retain_%02zu", k + 1u); + js.kv(key, acc / static_cast(k + 1u)); + } + FieldMetrics fm = measure_field(rig); + js.field("field_", fm); + js.end_scenario(); +} + +// J2 — randomise_place_only: a whole session made ONLY of "roll a patch, +// place it". No dislikes, no weight randomisation. The operator named this +// as a distinct way of working and it deserves its own row. +void journey_randomise_place_only(const Config& cfg, Json& js) { + if (!selected(cfg, "J2_randomise_place_only")) return; + Rig rig(cfg); + const std::size_t N = cfg.smoke ? 5u : 15u; + Kronecker k_where(cfg.n_in); + + rig.fb.set_mode(FeedbackMode::RandomiseOutputs, rig.mlp); + js.begin_scenario("J2_randomise_place_only", "session of roll-a-patch-and-place, no dislikes"); + + std::vector where(cfg.n_in), heard, patch(cfg.n_out), got; + std::vector placed_xs, placed_ys; + for (std::size_t k = 0; k < N; ++k) { + k_where.point(k * 13u + 1u, where); + rig.at(where, heard); + for (std::size_t j = 0; j < cfg.n_in; ++j) rig.mlp.set_input(j, where[j]); + rig.mlp.process(); + rig.fb.on_down(rig.mlp, heard, kMoveSpeed, rig.spread(), {}); // enter/roll + rig.fb.static_output(patch); + place_positive(rig, where, patch); + rig.mlp.train(); + placed_xs.insert(placed_xs.end(), where.begin(), where.end()); + placed_ys.insert(placed_ys.end(), patch.begin(), patch.end()); + } + // Retention across everything placed. + const std::size_t placed_n = placed_xs.size() / cfg.n_in; + float acc = 0.f, mx = 0.f; + for (std::size_t i = 0; i < placed_n; ++i) { + rig.at(std::span(&placed_xs[i * cfg.n_in], cfg.n_in), got); + const float e = l2(got, std::span(&placed_ys[i * cfg.n_out], cfg.n_out)); + acc += e; + if (e > mx) mx = e; + } + js.kv("placed", placed_n); + js.kv("retain_mean", placed_n ? acc / static_cast(placed_n) : -1.f); + js.kv("retain_max", placed_n ? mx : -1.f); + FieldMetrics fm = measure_field(rig); + js.field("field_", fm); + js.end_scenario(); +} + +// J3 — mixed_session: likes and dislikes interleaved, the realistic case. +// Reports the field metrics at checkpoints so drift over a session is visible. +void journey_mixed(const Config& cfg, Json& js) { + if (!selected(cfg, "J3_mixed_session")) return; + Rig rig(cfg); + const std::size_t N = cfg.smoke ? 8u : 24u; + Dataset ds = make_dataset(cfg, "scattered", N, rig.rng); + rig.fb.set_mode(FeedbackMode::Avoid, rig.mlp); + rig.fb.set_avoid_style(AvoidStyle::Geometric); + + js.begin_scenario("J3_mixed_session", "likes and dislikes interleaved"); + std::vector heard, got; + Kronecker k_dis(cfg.n_in); + std::vector dis(cfg.n_in); + + for (std::size_t k = 0; k < N; ++k) { + place_positive(rig, std::span(&ds.xs[k * cfg.n_in], cfg.n_in), + std::span(&ds.ys[k * cfg.n_out], cfg.n_out)); + rig.mlp.train(); + if (k % 3u == 2u) { // every third gesture is a dislike somewhere else + k_dis.point(k * 17u + 7u, dis); + rig.at(dis, heard); + for (std::size_t j = 0; j < cfg.n_in; ++j) rig.mlp.set_input(j, dis[j]); + rig.mlp.process(); + rig.fb.on_down(rig.mlp, heard, kMoveSpeed, rig.spread(), {}); + } + if ((k + 1u) % 8u == 0u) { + FieldMetrics fm = measure_field(rig); + char pre[32]; + snprintf(pre, sizeof pre, "ck%02zu_", k + 1u); + js.field(pre, fm); + } + } + // Retention of every like after the whole session. + float acc = 0.f; + for (std::size_t i = 0; i < N; ++i) { + rig.at(std::span(&ds.xs[i * cfg.n_in], cfg.n_in), got); + acc += l2(got, std::span(&ds.ys[i * cfg.n_out], cfg.n_out)); + } + js.kv("retain_mean", acc / static_cast(N)); + js.kv("positives", rig.fb.positive_count()); + js.kv("negatives", rig.fb.negative_count()); + js.end_scenario(); +} + +// J4 — branch: one shared prefix, three different continuations, compared +// against each other. Branching is implemented by replaying the prefix from +// scratch per branch, which is exact under the deterministic RNG. +// +// The question this answers: standing in the same place with the same history, +// how differently do "dislike", "roll-and-place" and "do nothing" leave the +// instrument? That comparison is what "which UX journey shapes the space how" +// actually means. +void journey_branch(const Config& cfg, Json& js) { + if (!selected(cfg, "J4_branch")) return; + + const std::size_t PREFIX = cfg.smoke ? 5u : 10u; + + auto build_prefix = [&](Rig& rig, Dataset& ds, std::vector& pos_xs) { + ds = make_dataset(cfg, "scattered", PREFIX, rig.rng); + pos_xs = teach(rig, ds); + }; + + // Common branch point: a spot between examples 0 and 1. + auto branch_point = [&](const Dataset& ds, std::vector& p) { + p.resize(cfg.n_in); + for (std::size_t j = 0; j < cfg.n_in; ++j) + p[j] = 0.5f * (ds.xs[j] + ds.xs[cfg.n_in + j]); + }; + + struct Branch { const char* id; int kind; }; // 0=nothing 1=dislike 2=roll+place + const Branch branches[] = { + {"J4_branch_control", 0}, + {"J4_branch_dislike", 1}, + {"J4_branch_roll_place", 2}, + }; + + // Reference field from the control branch, so the other two can be + // reported as displacement FROM the untouched instrument. + std::vector control_field; + + for (const Branch& b : branches) { + Rig rig(cfg); + Dataset ds; + std::vector pos_xs; + build_prefix(rig, ds, pos_xs); + + std::vector before; + rig.field(before); + if (b.kind == 0) control_field = before; + + std::vector p, heard, patch(cfg.n_out); + branch_point(ds, p); + rig.at(p, heard); + for (std::size_t j = 0; j < cfg.n_in; ++j) rig.mlp.set_input(j, p[j]); + rig.mlp.process(); + + if (b.kind == 1) { + rig.fb.set_mode(FeedbackMode::Avoid, rig.mlp); + rig.fb.set_avoid_style(AvoidStyle::Geometric); + rig.fb.on_down(rig.mlp, heard, kMoveSpeed, rig.spread(), {}); + } else if (b.kind == 2) { + rig.fb.set_mode(FeedbackMode::RandomiseOutputs, rig.mlp); + rig.fb.on_down(rig.mlp, heard, kMoveSpeed, rig.spread(), {}); + rig.fb.static_output(patch); + place_positive(rig, p, patch); + rig.mlp.train(); + } + + std::vector after; + rig.field(after); + Displacement d = measure_displacement(rig, before, after, p, pos_xs); + + js.begin_scenario(b.id, "shared prefix, one divergent gesture"); + js.kv("prefix_examples", PREFIX); + js.disp("", d); + FieldMetrics fm = measure_field(rig); + js.field("field_", fm); + js.end_scenario(); + } +} + +// ========================================================================= +// EDGE CASES — the degenerate corners. These REPORT; the ones that are true +// invariants are asserted in tests/cpp/test_ml_behaviour.cpp. +// ========================================================================= +void edge_cases(const Config& cfg, Json& js) { + // E1 — cold start: dislike with NO examples at all. feedback.hpp takes a + // documented degenerate branch here (ALIGNMENT.md:91) with its own RNG + // draw. Worth a row precisely because it is the first thing a new user does. + if (selected(cfg, "E1_cold_start_dislike")) { + Rig rig(cfg); + std::vector before, after, heard, p(cfg.n_in, 0.f); + rig.field(before); + rig.at(p, heard); + rig.fb.set_mode(FeedbackMode::Avoid, rig.mlp); + rig.fb.set_avoid_style(AvoidStyle::Geometric); + rig.mlp.process(); + rig.fb.on_down(rig.mlp, heard, kMoveSpeed, rig.spread(), {}); + rig.field(after); + Displacement d = measure_displacement(rig, before, after, p, {}); + js.begin_scenario("E1_cold_start_dislike", "dislike with zero examples stored"); + js.disp("", d); + js.kv("negatives", rig.fb.negative_count()); + js.end_scenario(); + } + + // E2 — single example: the whole instrument taught by one point. + if (selected(cfg, "E2_single_example")) { + Rig rig(cfg); + Dataset ds = make_dataset(cfg, "scattered", 1u, rig.rng); + teach(rig, ds); + std::vector got; + rig.at(std::span(&ds.xs[0], cfg.n_in), got); + js.begin_scenario("E2_single_example", "one example is the entire dataset"); + js.kv("placement_err", l2(got, std::span(&ds.ys[0], cfg.n_out))); + FieldMetrics fm = measure_field(rig); + js.field("field_", fm); + js.end_scenario(); + } + + // E3 — contradictory examples: same input position, different targets. + // The net cannot satisfy both; what it does instead is a design fact. + if (selected(cfg, "E3_contradictory")) { + Rig rig(cfg); + Dataset ds = make_dataset(cfg, "coincident", 6u, rig.rng); + teach(rig, ds); + std::vector got; + rig.at(std::span(&ds.xs[0], cfg.n_in), got); + float spread_of_targets = 0.f; + for (std::size_t i = 0; i < ds.n; ++i) { + spread_of_targets += l2(got, std::span(&ds.ys[i * cfg.n_out], cfg.n_out)); + } + js.begin_scenario("E3_contradictory", "N examples at ONE position with different targets"); + js.kv("n_examples", ds.n); + js.kv("mean_err_to_all", spread_of_targets / static_cast(ds.n)); + js.kv("final_loss", rig.mlp.eval_loss()); + js.end_scenario(); + } + + // E4 — collinear examples: every example on one line. Degenerate geometry + // that a k-NN centroid handles differently from a scattered cloud. + if (selected(cfg, "E4_collinear")) { + Rig rig(cfg); + Dataset ds = make_dataset(cfg, "collinear", 8u, rig.rng); + teach(rig, ds); + js.begin_scenario("E4_collinear", "all examples on one line through the space"); + FieldMetrics fm = measure_field(rig); + js.field("field_", fm); + js.kv("final_loss", rig.mlp.eval_loss()); + js.end_scenario(); + } + + // E5 — corners: examples pinned at the domain bounds, where the input + // pipeline's circular clamp and the sigmoid rails both bite. + if (selected(cfg, "E5_corners")) { + Rig rig(cfg); + Dataset ds = make_dataset(cfg, "corners", 8u, rig.rng); + teach(rig, ds); + js.begin_scenario("E5_corners", "examples pinned at the domain corners"); + FieldMetrics fm = measure_field(rig); + js.field("field_", fm); + js.end_scenario(); + } + + // E6 — capacity overflow: push past max_examples and see what the FIFO + // silently dropped. The oldest liked sounds go first and nothing tells + // the user (mlp.hpp add_example, ring eviction). + if (selected(cfg, "E6_capacity_overflow")) { + Config c2 = cfg; + c2.max_examples = cfg.smoke ? 8u : 16u; // small cap so the run is quick + Rig rig(c2); + const std::size_t N = c2.max_examples * 2u; + Dataset ds = make_dataset(c2, "scattered", N, rig.rng); + // Deliberately raw add_example: this probe is about the MLP dataset + // ring, not the replay buffer, and mixing the two would confuse which + // capacity is being overflowed. + for (std::size_t i = 0; i < N; ++i) { + rig.mlp.add_example(std::span(&ds.xs[i * c2.n_in], c2.n_in), + std::span(&ds.ys[i * c2.n_out], c2.n_out)); + } + rig.mlp.train(); + std::vector got; + float early = 0.f, late = 0.f; + for (std::size_t i = 0; i < c2.max_examples; ++i) { + rig.at(std::span(&ds.xs[i * c2.n_in], c2.n_in), got); + early += l2(got, std::span(&ds.ys[i * c2.n_out], c2.n_out)); + } + for (std::size_t i = c2.max_examples; i < N; ++i) { + rig.at(std::span(&ds.xs[i * c2.n_in], c2.n_in), got); + late += l2(got, std::span(&ds.ys[i * c2.n_out], c2.n_out)); + } + js.begin_scenario("E6_capacity_overflow", "twice the cap in examples; what survived"); + js.kv("cap", c2.max_examples); + js.kv("submitted", N); + js.kv("err_first_half", early / static_cast(c2.max_examples)); + js.kv("err_second_half", late / static_cast(N - c2.max_examples)); + js.end_scenario(); + } + + // E7 — undo depth exhaustion: more scratchpad ops than the undo ring + // holds (kUndoDepth). Reports how far back the user can actually get. + if (selected(cfg, "E7_undo_exhaustion")) { + Rig rig(cfg); + Dataset ds = make_dataset(cfg, "scattered", 6u, rig.rng); + teach(rig, ds); + rig.fb.set_mode(FeedbackMode::ExploreAndPlace, rig.mlp); + + std::vector origin; + rig.field(origin); + rig.fb.enter_explore(rig.mlp, rig.spread()); + for (std::size_t i = 0; i < kUndoDepth + 3u; ++i) rig.fb.reroll(rig.mlp, rig.spread()); + const std::size_t depth_before = rig.fb.undo_depth(); + for (std::size_t i = 0; i < kUndoDepth + 3u; ++i) rig.fb.undo(rig.mlp); + rig.fb.exit_explore(rig.mlp); + + std::vector restored; + rig.field(restored); + js.begin_scenario("E7_undo_exhaustion", "more rerolls than the undo ring holds"); + js.kv("undo_cap", kUndoDepth); + js.kv("ops", kUndoDepth + 3u); + js.kv("depth_before_undo", depth_before); + js.kv("residual", l2(origin, restored)); // 0 = fully recovered + js.end_scenario(); + } + + // E8 — one output / one input: the narrowest legal shape. + if (selected(cfg, "E8_minimal_shape")) { + Config c2 = cfg; + c2.n_in = 1u; c2.n_out = 1u; + c2.hidden[0] = c2.hidden[1] = c2.hidden[2] = 4u; + Rig rig(c2); + Dataset ds = make_dataset(c2, "scattered", 6u, rig.rng); + teach(rig, ds); + js.begin_scenario("E8_minimal_shape", "1 input, 1 output, tiny hidden layers"); + FieldMetrics fm = measure_field(rig); + js.field("field_", fm); + js.kv("final_loss", rig.mlp.eval_loss()); + js.end_scenario(); + } +} + + +// ========================================================================= +// DIAGNOSTIC — why does the geometric dislike do nothing? +// +// The machinery is not broken; the DOSE is. Per press, dislike_geometric: +// 1. computes push_step = clamp(|avg_neg_reward|, 0.25, 1) * 0.5 (=0.5 fresh) +// 2. builds a target displaced from the heard action by +// push_step / (1 + ||heard - centroid||) (~0.2-0.5) +// 3. takes ONE gradient step toward it at geo_lr * geo_neg_lr_ratio +// (= 0.001 * ~0.47 = ~4.7e-4) (mlp.hpp +// train_targets: one forward, one backprop, one apply_grad per layer) +// +// So it aims at a target ~0.3 away and then moves ~5e-5. Upstream InterfaceRL +// applied the same LR inside a multi-pass shuffled optimise() over the WHOLE +// replay buffer; P3 collapsed that to a single step on a single item and kept +// the LR (ALIGNMENT.md:91). The loop was dropped, the dose was not re-scaled. +// +// This row reports the arithmetic next to the measured displacement, and the +// presses/LR needed to actually reach the target. --geo-lr and --geo-iters +// let the fix be measured before it is committed to. +// ========================================================================= +void diag_geo_anatomy(const Config& cfg, Json& js) { + if (!selected(cfg, "D1_geo_anatomy")) return; + Rig rig(cfg); + Dataset ds = make_dataset(cfg, "scattered", 12u, rig.rng); + std::vector pos_xs = teach(rig, ds); + rig.fb.set_mode(FeedbackMode::Avoid, rig.mlp); + rig.fb.set_avoid_style(AvoidStyle::Geometric); + + std::vector press(cfg.n_in); + for (std::size_t j = 0; j < cfg.n_in; ++j) + press[j] = 0.5f * (ds.xs[j] + ds.xs[cfg.n_in + j]); + + std::vector heard0, got, before; + rig.at(press, heard0); + rig.field(before); + + // The analytic dose, from the same free functions feedback.hpp calls. + const float push_step = nisps::ml::geo_push_step(-1.f); // one fresh negative + const float ratio = nisps::ml::geo_neg_lr_ratio(1u, rig.fb.positive_count()); + const float eff_lr = cfg.geo_lr * ratio; + + auto press_once = [&]() { + std::vector h; + rig.at(press, h); + for (std::size_t j = 0; j < cfg.n_in; ++j) rig.mlp.set_input(j, press[j]); + rig.mlp.process(); + rig.fb.on_down(rig.mlp, h, kMoveSpeed, rig.spread(), {}); + }; + + press_once(); + rig.at(press, got); + const float after_1 = l2(got, heard0); + + for (int i = 1; i < 10; ++i) press_once(); + rig.at(press, got); + const float after_10 = l2(got, heard0); + + for (int i = 10; i < 100; ++i) press_once(); + rig.at(press, got); + const float after_100 = l2(got, heard0); + + for (int i = 100; i < 1000; ++i) press_once(); + rig.at(press, got); + const float after_1000 = l2(got, heard0); + + std::vector after; + rig.field(after); + Displacement d = measure_displacement(rig, before, after, press, pos_xs); + + js.begin_scenario("D1_geo_anatomy", "the geometric dislike dose, decomposed"); + js.kv("push_step", push_step); // ~0.5 — the intended push + js.kv("neg_lr_ratio", ratio); // ~0.47 + js.kv("effective_lr", eff_lr); // ~4.7e-4 — ONE step at this + js.kv("train_lr_for_comparison", 1.0f); // what a LIKE trains at + js.kv("train_iters_for_comparison", std::size_t{1000}); + js.kv("moved_after_1", after_1); + js.kv("moved_after_10", after_10); + js.kv("moved_after_100", after_100); + js.kv("moved_after_1000", after_1000); + js.disp("cumulative_", d); + js.kv("positives", rig.fb.positive_count()); + js.kv("negatives", rig.fb.negative_count()); + js.end_scenario(); +} + +// ========================================================================= +// EXPLORE-AND-PLACE — the DEFAULT product mode. Everything above tested the +// Avoid path; this is the lifecycle a musician actually drives. +// ========================================================================= + +// A9 — the full lifecycle: enter explore, audition, place, commit. The +// controller owns the weight snapshot; the CALLER owns add_example + train +// (feedback.hpp:475 contract). Getting that split wrong is the classic bug: +// commit without add_example silently discards the placement. +void probe_explore_place(const Config& cfg, Json& js) { + if (!selected(cfg, "A9_explore_and_place")) return; + Rig rig(cfg); + Dataset ds = make_dataset(cfg, "scattered", 8u, rig.rng); + std::vector pos_xs = teach(rig, ds); + rig.mlp.train(); + + std::vector before; + rig.field(before); + + std::vector where(cfg.n_in); + Kronecker k(cfg.n_in); + k.point(555u, where); + for (std::size_t j = 0; j < cfg.n_in; ++j) rig.mlp.set_input(j, where[j]); + rig.mlp.process(); + + rig.fb.set_mode(FeedbackMode::ExploreAndPlace, rig.mlp); + rig.fb.enter_explore(rig.mlp, rig.spread()); + for (int i = 0; i < 3; ++i) rig.fb.reroll(rig.mlp, rig.spread()); + rig.fb.nudge(rig.mlp, 0.2f); + + // Audition: what the scratchpad net says here is what gets placed. + rig.mlp.process(); + std::vector auditioned(cfg.n_out); + { auto o = rig.mlp.outputs(); + for (std::size_t j = 0; j < cfg.n_out; ++j) auditioned[j] = o[j]; } + + rig.fb.begin_place(rig.mlp, auditioned); + rig.fb.commit_place(rig.mlp); // restores the REAL net + std::vector placed_v; + const bool committed = take_committed(rig, placed_v); + if (committed) { + place_positive(rig, where, placed_v); // the caller's half of the contract + rig.mlp.train(); + } + + std::vector after, got; + rig.field(after); + rig.at(where, got); + Displacement d = measure_displacement(rig, before, after, where, pos_xs); + + js.begin_scenario("A9_explore_and_place", "default mode: explore, audition, place, commit"); + js.kv("committed", static_cast(committed ? 1u : 0u)); + js.kv("placement_err", committed ? l2(got, placed_v) : -1.f); + js.kv("audition_vs_placed", committed ? l2(auditioned, placed_v) : -1.f); // must be ~0 + js.disp("", d); + js.kv("examples", ds.n + 1u); + js.end_scenario(); +} + +// A10 — explore then CANCEL. Auditioning must be free: the net must come back +// exactly, and the field must be untouched. +void probe_explore_cancel(const Config& cfg, Json& js) { + if (!selected(cfg, "A10_explore_cancel")) return; + Rig rig(cfg); + Dataset ds = make_dataset(cfg, "scattered", 8u, rig.rng); + teach(rig, ds); + rig.mlp.train(); + + std::vector before, after; + rig.field(before); + + rig.fb.set_mode(FeedbackMode::ExploreAndPlace, rig.mlp); + rig.fb.enter_explore(rig.mlp, rig.spread()); + for (int i = 0; i < 5; ++i) rig.fb.reroll(rig.mlp, rig.spread()); + rig.fb.cancel_place(); + rig.fb.exit_explore(rig.mlp); + + rig.field(after); + js.begin_scenario("A10_explore_cancel", "audition then cancel — must cost nothing"); + js.kv("residual", l2(before, after)); // 0 = auditioning was free + js.end_scenario(); +} + +// A11 — reposition: grab an existing example's outputs and re-place them at a +// NEW input position. The musician moving a sound they like to a comfier spot. +void probe_reposition(const Config& cfg, Json& js) { + if (!selected(cfg, "A11_reposition")) return; + Rig rig(cfg); + Dataset ds = make_dataset(cfg, "scattered", 8u, rig.rng); + std::vector pos_xs = teach(rig, ds); + rig.mlp.train(); + + std::vector before; + rig.field(before); + + // Reposition is an ExploreAndPlace-mode gesture; begin_reposition is a + // no-op in any other mode (feedback.hpp:616). + rig.fb.set_mode(FeedbackMode::ExploreAndPlace, rig.mlp); + + // Stand at example 0, grab it. During the hold, state IS Placing, so + // placed_output() is the right accessor here. + std::span src(&ds.xs[0], cfg.n_in); + for (std::size_t j = 0; j < cfg.n_in; ++j) rig.mlp.set_input(j, src[j]); + rig.mlp.process(); + rig.fb.begin_reposition(rig.mlp); + auto grabbed = rig.fb.placed_output(); + const bool grabbed_ok = grabbed.size() >= cfg.n_out; + std::vector grabbed_v(grabbed.begin(), grabbed.end()); + + // Move somewhere else and drop it. + std::vector dst(cfg.n_in); + Kronecker k(cfg.n_in); + k.point(321u, dst); + for (std::size_t j = 0; j < cfg.n_in; ++j) rig.mlp.set_input(j, dst[j]); + rig.mlp.process(); + rig.fb.commit_reposition(); + // After commit the carried vector lives in committed_output(). + std::vector carried; + const bool carried_ok = take_committed(rig, carried); + if (carried_ok) { + place_positive(rig, dst, carried); + rig.mlp.train(); + } + + std::vector after, at_dst, at_src; + rig.field(after); + rig.at(dst, at_dst); + rig.at(src, at_src); + Displacement d = measure_displacement(rig, before, after, dst, pos_xs); + + js.begin_scenario("A11_reposition", "grab an example's outputs, drop them elsewhere"); + js.kv("grabbed", static_cast(grabbed_ok ? 1u : 0u)); + js.kv("carried", static_cast(carried_ok ? 1u : 0u)); + js.kv("arrived", carried_ok ? l2(at_dst, carried) : -1.f); // small = it landed + js.kv("source_still_holds", carried_ok ? l2(at_src, carried) : -1.f); + js.disp("", d); + js.end_scenario(); +} + +// A12 — like then dislike at exactly the same spot. A direct contradiction: +// the user taught something and then rejected it in place. Which wins? +void probe_like_then_dislike(const Config& cfg, Json& js) { + if (!selected(cfg, "A12_like_then_dislike")) return; + Rig rig(cfg); + Dataset ds = make_dataset(cfg, "scattered", 8u, rig.rng); + std::vector pos_xs = teach(rig, ds); + rig.mlp.train(); + + std::span x(&ds.xs[0], cfg.n_in); + std::span y(&ds.ys[0], cfg.n_out); + + std::vector before, after, got; + rig.field(before); + rig.at(x, got); + const float held_before = l2(got, y); + + rig.fb.set_mode(FeedbackMode::Avoid, rig.mlp); + rig.fb.set_avoid_style(AvoidStyle::Geometric); + for (std::size_t i = 0; i < cfg.geo_iters; ++i) { + std::vector h; + rig.at(x, h); + for (std::size_t j = 0; j < cfg.n_in; ++j) rig.mlp.set_input(j, x[j]); + rig.mlp.process(); + rig.fb.on_down(rig.mlp, h, kMoveSpeed, rig.spread(), {}); + } + + rig.field(after); + rig.at(x, got); + const float held_after = l2(got, y); + Displacement d = measure_displacement(rig, before, after, x, pos_xs); + + js.begin_scenario("A12_like_then_dislike", "dislike exactly where a like was placed"); + js.kv("held_before", held_before); + js.kv("held_after", held_after); + js.kv("rejection_moved", held_after - held_before); // >0 = the dislike won ground + js.disp("", d); + js.end_scenario(); +} + +// A13 — repair journey: dislike somewhere, then explore-and-place a +// replacement AT THE SAME SPOT. This is the operator's stated design intent +// for what a dislike should lead to, executed end to end. +void probe_dislike_then_repair(const Config& cfg, Json& js) { + if (!selected(cfg, "A13_dislike_then_repair")) return; + Rig rig(cfg); + Dataset ds = make_dataset(cfg, "scattered", 8u, rig.rng); + std::vector pos_xs = teach(rig, ds); + rig.mlp.train(); + + std::vector before; + rig.field(before); + + std::vector where(cfg.n_in); + for (std::size_t j = 0; j < cfg.n_in; ++j) + where[j] = 0.5f * (ds.xs[j] + ds.xs[cfg.n_in + j]); + + std::vector disliked; + rig.at(where, disliked); + + rig.fb.set_mode(FeedbackMode::Avoid, rig.mlp); + rig.fb.set_avoid_style(AvoidStyle::Geometric); + for (std::size_t j = 0; j < cfg.n_in; ++j) rig.mlp.set_input(j, where[j]); + rig.mlp.process(); + rig.fb.on_down(rig.mlp, disliked, kMoveSpeed, rig.spread(), {}); + + // Now repair: explore for something else and place it here. + rig.fb.set_mode(FeedbackMode::ExploreAndPlace, rig.mlp); + rig.fb.enter_explore(rig.mlp, rig.spread()); + for (int i = 0; i < 2; ++i) rig.fb.reroll(rig.mlp, rig.spread()); + rig.mlp.process(); + std::vector replacement(cfg.n_out); + { auto o = rig.mlp.outputs(); + for (std::size_t j = 0; j < cfg.n_out; ++j) replacement[j] = o[j]; } + rig.fb.begin_place(rig.mlp, replacement); + rig.fb.commit_place(rig.mlp); + place_positive(rig, where, replacement); + rig.mlp.train(); + + std::vector after, got; + rig.field(after); + rig.at(where, got); + Displacement d = measure_displacement(rig, before, after, where, pos_xs); + + js.begin_scenario("A13_dislike_then_repair", "dislike, then explore-and-place a replacement here"); + js.kv("repair_took", l2(got, replacement)); // small = the fix landed + js.kv("moved_off_disliked", l2(got, disliked)); // large = it is genuinely different + js.disp("", d); + js.kv("positives", rig.fb.positive_count()); + js.kv("negatives", rig.fb.negative_count()); + js.end_scenario(); +} + +// A14 — focus/solo mask: dislike with only half the outputs active. The +// masked dims must not move at all; that is what "solo this parameter" means. +void probe_focus_mask(const Config& cfg, Json& js) { + if (!selected(cfg, "A14_focus_mask")) return; + Rig rig(cfg); + Dataset ds = make_dataset(cfg, "scattered", 8u, rig.rng); + teach(rig, ds); + rig.mlp.train(); + + std::vector mask(cfg.n_out, 0u); + for (std::size_t j = 0; j < cfg.n_out; j += 2u) mask[j] = 1u; // evens active + rig.fb.set_focus_mask(mask); + rig.fb.set_mode(FeedbackMode::Avoid, rig.mlp); + rig.fb.set_avoid_style(AvoidStyle::Geometric); + + std::vector where(cfg.n_in); + for (std::size_t j = 0; j < cfg.n_in; ++j) + where[j] = 0.5f * (ds.xs[j] + ds.xs[cfg.n_in + j]); + + std::vector b4, aft; + rig.at(where, b4); + for (std::size_t i = 0; i < (cfg.geo_iters > 50u ? cfg.geo_iters : 50u); ++i) { + std::vector h; + rig.at(where, h); + for (std::size_t j = 0; j < cfg.n_in; ++j) rig.mlp.set_input(j, where[j]); + rig.mlp.process(); + rig.fb.on_down(rig.mlp, h, kMoveSpeed, rig.spread(), {}); + } + rig.at(where, aft); + + float active_moved = 0.f, masked_moved = 0.f; + for (std::size_t j = 0; j < cfg.n_out; ++j) { + const float dj = std::fabs(aft[j] - b4[j]); + if (mask[j]) active_moved += dj; else masked_moved += dj; + } + js.begin_scenario("A14_focus_mask", "dislike with half the outputs soloed out"); + js.kv("active_moved", active_moved); + js.kv("masked_moved", masked_moved); // should be ~0 + js.kv("leak_ratio", (active_moved > 1e-12f) ? masked_moved / active_moved : 0.f); + js.end_scenario(); +} + +// ========================================================================= +// MORE JOURNEYS +// ========================================================================= + +// J5 — a session made ONLY of explore-and-place, the default mode's natural +// way of working. Contrast with J2 (RandomiseOutputs) and J1 (direct likes). +void journey_explore_place_only(const Config& cfg, Json& js) { + if (!selected(cfg, "J5_explore_place_only")) return; + Rig rig(cfg); + const std::size_t N = cfg.smoke ? 5u : 15u; + Kronecker kw(cfg.n_in); + rig.fb.set_mode(FeedbackMode::ExploreAndPlace, rig.mlp); + + std::vector where(cfg.n_in), placed_xs, placed_ys, got; + for (std::size_t k = 0; k < N; ++k) { + kw.point(k * 19u + 2u, where); + for (std::size_t j = 0; j < cfg.n_in; ++j) rig.mlp.set_input(j, where[j]); + rig.mlp.process(); + rig.fb.enter_explore(rig.mlp, rig.spread()); + for (int i = 0; i < 2; ++i) rig.fb.reroll(rig.mlp, rig.spread()); + rig.mlp.process(); + std::vector patch(cfg.n_out); + { auto o = rig.mlp.outputs(); + for (std::size_t j = 0; j < cfg.n_out; ++j) patch[j] = o[j]; } + rig.fb.begin_place(rig.mlp, patch); + rig.fb.commit_place(rig.mlp); + std::vector pv; + if (!take_committed(rig, pv)) continue; // nothing committed: skip, do not fake + place_positive(rig, where, pv); + rig.mlp.train(); + placed_xs.insert(placed_xs.end(), where.begin(), where.end()); + placed_ys.insert(placed_ys.end(), pv.begin(), pv.end()); + } + // Iterate over what was ACTUALLY placed, never over the requested count. + const std::size_t placed_n = placed_xs.size() / cfg.n_in; + float acc = 0.f, mx = 0.f; + for (std::size_t i = 0; i < placed_n; ++i) { + rig.at(std::span(&placed_xs[i * cfg.n_in], cfg.n_in), got); + const float e = l2(got, std::span(&placed_ys[i * cfg.n_out], cfg.n_out)); + acc += e; if (e > mx) mx = e; + } + js.begin_scenario("J5_explore_place_only", "session of explore-audition-place, no dislikes"); + js.kv("attempted", N); + js.kv("placed", placed_n); + js.kv("retain_mean", placed_n ? acc / static_cast(placed_n) : -1.f); + js.kv("retain_max", placed_n ? mx : -1.f); + FieldMetrics fm = measure_field(rig); + js.field("field_", fm); + js.end_scenario(); +} + +// J6 — long session with drift checkpoints. Some failure modes (going numb, +// weight-norm growth, buffer exhaustion) only appear after tens of gestures. +void journey_long_session(const Config& cfg, Json& js) { + if (!selected(cfg, "J6_long_session")) return; + Rig rig(cfg); + const std::size_t N = cfg.smoke ? 20u : 120u; + Dataset ds = make_dataset(cfg, "scattered", N, rig.rng); + rig.fb.set_mode(FeedbackMode::Avoid, rig.mlp); + rig.fb.set_avoid_style(AvoidStyle::Geometric); + Kronecker kd(cfg.n_in); + std::vector dis(cfg.n_in), heard; + + js.begin_scenario("J6_long_session", "120 gestures with periodic drift checkpoints"); + for (std::size_t k = 0; k < N; ++k) { + place_positive(rig, std::span(&ds.xs[k * cfg.n_in], cfg.n_in), + std::span(&ds.ys[k * cfg.n_out], cfg.n_out)); + rig.mlp.train(); + if (k % 4u == 3u) { + kd.point(k * 23u + 11u, dis); + rig.at(dis, heard); + for (std::size_t j = 0; j < cfg.n_in; ++j) rig.mlp.set_input(j, dis[j]); + rig.mlp.process(); + rig.fb.on_down(rig.mlp, heard, kMoveSpeed, rig.spread(), {}); + } + const std::size_t every = cfg.smoke ? 10u : 40u; + if ((k + 1u) % every == 0u) { + FieldMetrics fm = measure_field(rig); + char pre[32]; + snprintf(pre, sizeof pre, "ck%03zu_", k + 1u); + js.field(pre, fm); + // Weight-norm growth: unbounded growth ends in sigmoid saturation. + auto w = rig.mlp.get_weights(); + float wn = 0.f; + for (float v : w) wn += v * v; + char key[32]; + snprintf(key, sizeof key, "ck%03zu_wnorm", k + 1u); + js.kv(key, std::sqrt(wn)); + } + } + js.kv("positives", rig.fb.positive_count()); + js.kv("negatives", rig.fb.negative_count()); + js.kv("replay_size", rig.fb.replay_size()); + js.end_scenario(); +} + +// J7 — dislike storm: the "I hate this whole area" gesture. Many dislikes +// clustered in one region. Does the region become usable, or does the whole +// instrument degrade? +void journey_dislike_storm(const Config& cfg, Json& js) { + if (!selected(cfg, "J7_dislike_storm")) return; + Rig rig(cfg); + Dataset ds = make_dataset(cfg, "scattered", 10u, rig.rng); + std::vector pos_xs = teach(rig, ds); + rig.mlp.train(); + rig.fb.set_mode(FeedbackMode::Avoid, rig.mlp); + rig.fb.set_avoid_style(AvoidStyle::Geometric); + + std::vector before; + rig.field(before); + FieldMetrics fm_before = measure_field(rig); + + // 40 dislikes scattered inside a small ball. + Kronecker kb(cfg.n_in); + std::vector p(cfg.n_in), heard; + const std::size_t M = cfg.smoke ? 10u : 40u; + for (std::size_t i = 0; i < M; ++i) { + kb.point(i * 5u + 1u, p); + for (std::size_t j = 0; j < cfg.n_in; ++j) p[j] = -0.4f + 0.2f * p[j]; + rig.at(p, heard); + for (std::size_t j = 0; j < cfg.n_in; ++j) rig.mlp.set_input(j, p[j]); + rig.mlp.process(); + rig.fb.on_down(rig.mlp, heard, kMoveSpeed, rig.spread(), {}); + } + + std::vector after; + rig.field(after); + FieldMetrics fm_after = measure_field(rig); + std::vector centre(cfg.n_in, -0.4f); + Displacement d = measure_displacement(rig, before, after, centre, pos_xs); + + js.begin_scenario("J7_dislike_storm", "40 dislikes clustered in one region"); + js.kv("dislikes", M); + js.disp("", d); + js.kv("gain_p50_before", fm_before.gain_p50); + js.kv("gain_p50_after", fm_after.gain_p50); + js.kv("cliff_before", fm_before.cliff_index); + js.kv("cliff_after", fm_after.cliff_index); + js.kv("negatives", rig.fb.negative_count()); + js.end_scenario(); +} + +// J8 — revisit: teach a spot, wander far away and keep working, then come +// BACK and check it held. The operator asked this directly. +void journey_revisit(const Config& cfg, Json& js) { + if (!selected(cfg, "J8_revisit")) return; + Rig rig(cfg); + Dataset ds = make_dataset(cfg, "scattered", cfg.smoke ? 8u : 20u, rig.rng); + + // Teach the anchor FIRST, remember what it should be. + std::span anchor_x(&ds.xs[0], cfg.n_in); + std::span anchor_y(&ds.ys[0], cfg.n_out); + place_positive(rig, anchor_x, anchor_y); + rig.mlp.train(); + + std::vector got; + rig.at(anchor_x, got); + const float held_immediately = l2(got, anchor_y); + + js.begin_scenario("J8_revisit", "teach a spot, work elsewhere, come back to it"); + js.kv("held_immediately", held_immediately); + + // Now work everywhere else, checking the anchor periodically. + for (std::size_t k = 1u; k < ds.n; ++k) { + place_positive(rig, std::span(&ds.xs[k * cfg.n_in], cfg.n_in), + std::span(&ds.ys[k * cfg.n_out], cfg.n_out)); + rig.mlp.train(); + if (k % 4u == 0u) { + rig.at(anchor_x, got); + char key[32]; + snprintf(key, sizeof key, "held_after_%02zu", k); + js.kv(key, l2(got, anchor_y)); + } + } + rig.at(anchor_x, got); + js.kv("held_at_end", l2(got, anchor_y)); + js.end_scenario(); +} + +// J9 — two-region interference: likes in one half of the space, dislikes in +// the other. How much does work in region B damage region A? This is the +// cleanest test of "does my editing stay local". +void journey_two_region(const Config& cfg, Json& js) { + if (!selected(cfg, "J9_two_region")) return; + Rig rig(cfg); + const std::size_t N = cfg.smoke ? 6u : 12u; + Kronecker k(cfg.n_in), ko(cfg.n_out); + + std::vector xa(cfg.n_in), ya(cfg.n_out), got; + std::vector region_a_xs, region_a_ys; + for (std::size_t i = 0; i < N; ++i) { + k.point(i * 7u + 1u, xa); + for (std::size_t j = 0; j < cfg.n_in; ++j) xa[j] = 0.5f + 0.4f * xa[j]; // region A + ko.point(i * 11u + 3u, ya); + for (std::size_t j = 0; j < cfg.n_out; ++j) ya[j] = 0.5f + 0.4f * ya[j]; + place_positive(rig, xa, ya); + region_a_xs.insert(region_a_xs.end(), xa.begin(), xa.end()); + region_a_ys.insert(region_a_ys.end(), ya.begin(), ya.end()); + } + rig.mlp.train(); + + float held_before = 0.f; + for (std::size_t i = 0; i < N; ++i) { + rig.at(std::span(®ion_a_xs[i * cfg.n_in], cfg.n_in), got); + held_before += l2(got, std::span(®ion_a_ys[i * cfg.n_out], cfg.n_out)); + } + held_before /= static_cast(N); + + // Now hammer region B with dislikes. + rig.fb.set_mode(FeedbackMode::Avoid, rig.mlp); + rig.fb.set_avoid_style(AvoidStyle::Geometric); + std::vector xb(cfg.n_in), heard; + const std::size_t M = cfg.smoke ? 10u : 30u; + for (std::size_t i = 0; i < M; ++i) { + k.point(i * 13u + 5u, xb); + for (std::size_t j = 0; j < cfg.n_in; ++j) xb[j] = -0.5f + 0.4f * xb[j]; // region B + rig.at(xb, heard); + for (std::size_t j = 0; j < cfg.n_in; ++j) rig.mlp.set_input(j, xb[j]); + rig.mlp.process(); + rig.fb.on_down(rig.mlp, heard, kMoveSpeed, rig.spread(), {}); + } + + float held_after = 0.f; + for (std::size_t i = 0; i < N; ++i) { + rig.at(std::span(®ion_a_xs[i * cfg.n_in], cfg.n_in), got); + held_after += l2(got, std::span(®ion_a_ys[i * cfg.n_out], cfg.n_out)); + } + held_after /= static_cast(N); + + js.begin_scenario("J9_two_region", "likes in region A, dislikes in region B"); + js.kv("a_held_before", held_before); + js.kv("a_held_after", held_after); + js.kv("interference", held_after - held_before); // >0 = B damaged A + js.end_scenario(); +} + +// J10 — sweep-and-teach: move CONTINUOUSLY along a path, placing likes as you +// go. This is what performing actually looks like — not isolated pokes at +// scattered coordinates — and consecutive examples are highly correlated, +// which is a different regime for the optimiser. +void journey_sweep_teach(const Config& cfg, Json& js) { + if (!selected(cfg, "J10_sweep_teach")) return; + Rig rig(cfg); + const std::size_t N = cfg.smoke ? 8u : 24u; + Kronecker ko(cfg.n_out); + std::vector x(cfg.n_in), y(cfg.n_out), got; + std::vector xs, ys; + + for (std::size_t i = 0; i < N; ++i) { + const float t = static_cast(i) / static_cast(N - 1u); + // A smooth path through the space, not a scatter. + for (std::size_t j = 0; j < cfg.n_in; ++j) { + x[j] = std::sin(6.2831853f * t * (1.f + static_cast(j)) * 0.5f) * 0.8f; + } + ko.point(i * 3u + 1u, y); + for (std::size_t j = 0; j < cfg.n_out; ++j) y[j] = 0.5f + 0.4f * y[j]; + place_positive(rig, x, y); + rig.mlp.train(); + xs.insert(xs.end(), x.begin(), x.end()); + ys.insert(ys.end(), y.begin(), y.end()); + } + float acc = 0.f, mx = 0.f; + for (std::size_t i = 0; i < N; ++i) { + rig.at(std::span(&xs[i * cfg.n_in], cfg.n_in), got); + const float e = l2(got, std::span(&ys[i * cfg.n_out], cfg.n_out)); + acc += e; if (e > mx) mx = e; + } + js.begin_scenario("J10_sweep_teach", "likes placed along a continuous path, not a scatter"); + js.kv("placed", N); + js.kv("retain_mean", acc / static_cast(N)); + js.kv("retain_max", mx); + FieldMetrics fm = measure_field(rig); + js.field("field_", fm); + js.end_scenario(); +} + +// J11 — undo-heavy: dislike/undo alternation. The indecisive musician. Must +// not accumulate drift. +void journey_undo_heavy(const Config& cfg, Json& js) { + if (!selected(cfg, "J11_undo_heavy")) return; + Rig rig(cfg); + Dataset ds = make_dataset(cfg, "scattered", 8u, rig.rng); + teach(rig, ds); + rig.mlp.train(); + + std::vector before, after; + rig.field(before); + + rig.fb.set_mode(FeedbackMode::ExploreAndPlace, rig.mlp); + const std::size_t R = cfg.smoke ? 5u : 20u; + for (std::size_t i = 0; i < R; ++i) { + rig.fb.enter_explore(rig.mlp, rig.spread()); + rig.fb.reroll(rig.mlp, rig.spread()); + rig.fb.undo(rig.mlp); + rig.fb.exit_explore(rig.mlp); + } + rig.field(after); + + js.begin_scenario("J11_undo_heavy", "explore/reroll/undo/exit, repeated"); + js.kv("cycles", R); + js.kv("drift", l2(before, after)); // 0 = perfectly reversible + js.end_scenario(); +} + +// ========================================================================= +// MORE EDGE CASES +// ========================================================================= +void edge_cases_2(const Config& cfg, Json& js) { + // E9 — every example has the SAME target. The mapping should collapse to + // a constant; effective dimensionality should approach 1. + if (selected(cfg, "E9_identical_targets")) { + Rig rig(cfg); + Dataset ds = make_dataset(cfg, "scattered", 10u, rig.rng); + for (std::size_t i = 0; i < ds.n; ++i) + for (std::size_t j = 0; j < cfg.n_out; ++j) ds.ys[i * cfg.n_out + j] = 0.42f; + teach(rig, ds); + rig.mlp.train(); + js.begin_scenario("E9_identical_targets", "every example teaches the same output"); + FieldMetrics fm = measure_field(rig); + js.field("field_", fm); + js.end_scenario(); + } + + // E10 — targets exactly on the sigmoid rails (0 and 1). Unreachable in + // finite weights; the optimiser will push weights up forever chasing them. + if (selected(cfg, "E10_rail_targets")) { + Rig rig(cfg); + Dataset ds = make_dataset(cfg, "scattered", 8u, rig.rng); + for (std::size_t i = 0; i < ds.n; ++i) + for (std::size_t j = 0; j < cfg.n_out; ++j) + ds.ys[i * cfg.n_out + j] = ((i + j) % 2u) ? 1.f : 0.f; + teach(rig, ds); + rig.mlp.train(); + auto w = rig.mlp.get_weights(); + float wn = 0.f, wmax = 0.f; + for (float v : w) { wn += v * v; const float a = std::fabs(v); if (a > wmax) wmax = a; } + js.begin_scenario("E10_rail_targets", "targets pinned at 0 and 1 (unreachable)"); + js.kv("weight_norm", std::sqrt(wn)); + js.kv("weight_max", wmax); + js.kv("final_loss", rig.mlp.eval_loss()); + FieldMetrics fm = measure_field(rig); + js.field("field_", fm); + js.end_scenario(); + } + + // E11 — rapid like/dislike alternation at one spot. The user arguing with + // themselves. Must not diverge. + if (selected(cfg, "E11_rapid_alternation")) { + Rig rig(cfg); + Dataset ds = make_dataset(cfg, "scattered", 6u, rig.rng); + teach(rig, ds); + rig.mlp.train(); + rig.fb.set_mode(FeedbackMode::Avoid, rig.mlp); + rig.fb.set_avoid_style(AvoidStyle::Geometric); + + std::span x(&ds.xs[0], cfg.n_in); + std::span y(&ds.ys[0], cfg.n_out); + std::vector heard, got; + const std::size_t R = cfg.smoke ? 5u : 25u; + for (std::size_t i = 0; i < R; ++i) { + place_positive(rig, x, y); + rig.mlp.train(); + rig.at(x, heard); + for (std::size_t j = 0; j < cfg.n_in; ++j) rig.mlp.set_input(j, x[j]); + rig.mlp.process(); + rig.fb.on_down(rig.mlp, heard, kMoveSpeed, rig.spread(), {}); + } + rig.at(x, got); + auto w = rig.mlp.get_weights(); + float wn = 0.f; + for (float v : w) wn += v * v; + js.begin_scenario("E11_rapid_alternation", "like/dislike alternating at one spot"); + js.kv("cycles", R); + js.kv("final_err_to_liked", l2(got, y)); + js.kv("weight_norm", std::sqrt(wn)); + js.kv("final_loss", rig.mlp.eval_loss()); + js.end_scenario(); + } + + // E12 — switch feedback mode MID-EXPLORATION. set_mode is documented to + // tear down cleanly (feedback.hpp:215) so the net is never stranded in a + // randomised scratchpad state. A stranded net would be catastrophic live. + if (selected(cfg, "E12_mode_switch_midflight")) { + Rig rig(cfg); + Dataset ds = make_dataset(cfg, "scattered", 6u, rig.rng); + teach(rig, ds); + rig.mlp.train(); + + std::vector before, after; + rig.field(before); + + rig.fb.set_mode(FeedbackMode::ExploreAndPlace, rig.mlp); + rig.fb.enter_explore(rig.mlp, rig.spread()); + for (int i = 0; i < 3; ++i) rig.fb.reroll(rig.mlp, rig.spread()); + // Yank the mode out from under the exploration. + rig.fb.set_mode(FeedbackMode::Avoid, rig.mlp); + rig.field(after); + + js.begin_scenario("E12_mode_switch_midflight", "change mode while exploring"); + js.kv("stranded_drift", l2(before, after)); // 0 = torn down cleanly + js.kv("exploring", static_cast(rig.fb.exploring() ? 1u : 0u)); + js.end_scenario(); + } + + // E13 — dislike with EVERY output masked out. Nothing may move. + if (selected(cfg, "E13_all_outputs_masked")) { + Rig rig(cfg); + Dataset ds = make_dataset(cfg, "scattered", 6u, rig.rng); + teach(rig, ds); + rig.mlp.train(); + std::vector mask(cfg.n_out, 0u); // nothing active + rig.fb.set_focus_mask(mask); + rig.fb.set_mode(FeedbackMode::Avoid, rig.mlp); + rig.fb.set_avoid_style(AvoidStyle::Geometric); + + std::vector before, after, heard, p(cfg.n_in, 0.1f); + rig.field(before); + for (int i = 0; i < 20; ++i) { + rig.at(p, heard); + for (std::size_t j = 0; j < cfg.n_in; ++j) rig.mlp.set_input(j, p[j]); + rig.mlp.process(); + rig.fb.on_down(rig.mlp, heard, kMoveSpeed, rig.spread(), {}); + } + rig.field(after); + js.begin_scenario("E13_all_outputs_masked", "dislike with every output masked off"); + js.kv("moved", l2(before, after)); // should be ~0 + js.end_scenario(); + } +} + + +// ========================================================================= +// UPSTREAM COMPARISON — how the older memllib `interfaceRL` deals with OUTPUTS, +// measured against what NISPS does today. +// +// Source: `interfaceRL.hpp`, recoverable from this repo's own git history +// (blob 755ff8b). It is a DDPG actor-critic and differs from NISPS structurally: +// +// * FOUR nets: actor, actorTarget, critic, criticTarget. +// * The user HEARS actorTarget, not actor (interfaceRL.hpp:188 +// `actorTarget->GetOutput(...)` inside generateAction). actorTarget is +// soft-updated toward the learner every optimise(): +// actorTarget = (1-alpha)*actorTarget + alpha*actor, alpha = 0.005 +// * Negative feedback is a scalar reward feeding a critic; the actor is moved +// by dQ/da (deterministic policy gradient), not by a geometric push. +// * optimise() runs on a batch of 4 sampled from replay and only every +// optimiseDivisor = 40 gestures (optimiseSometimes(), :200). +// * learningRate = 0.005, discountFactor = 0.95. +// +// The critic half cannot be reproduced here: MLPCore exposes train_targets() +// but not the arbitrary per-layer gradient extraction (`CalcGradients` + +// `GetGrads` + `ApplyLoss`) that the policy-gradient step needs. That is a +// genuine port, not a benchmark. +// +// What IS reproducible with today's API — and is the part that shapes how the +// instrument FEELS — is the two output-path ideas: +// +// U1 SOFT TARGET. Keep a second net; train the first; expose the second, +// blended toward it by alpha each gesture. alpha = 1 is exactly NISPS +// today (target == online), so it is a free control in the same sweep. +// Hypothesis: this trades responsiveness for the mapping not lurching +// under your hands when a like retrains the whole net. +// +// U2 AMORTISED LEARNING. Train every Nth gesture instead of every one +// (upstream's optimiseDivisor). Hypothesis: fewer, larger jumps. +// +// The metric that matters for both is LURCH: how far the mapping the musician +// is playing moves per single gesture, measured over the whole field. Retention +// is reported alongside, because the entire question is what lurch costs you. +// ========================================================================= + +// A second net of the same shape, used as the soft-updated target. +struct TargetNet { + Mlp mlp; + TargetNet(const Config& c) + : mlp(c.seed, c.n_in, std::span(c.hidden, 3u), c.n_out, + c.max_examples, 4096u) {} +}; + +// target <- (1-alpha)*target + alpha*online (interfaceRL SmoothUpdateWeights) +void soft_update(Mlp& target, Mlp& online, float alpha) { + auto ow = online.get_weights(); + std::vector o(ow.begin(), ow.end()); // copy: shared scratch buffer + auto tw = target.get_weights(); + std::vector t(tw.begin(), tw.end()); + if (t.size() != o.size()) return; + for (std::size_t i = 0; i < t.size(); ++i) t[i] = (1.f - alpha) * t[i] + alpha * o[i]; + target.set_weights(t); +} + +void upstream_soft_target(const Config& cfg, Json& js) { + if (!selected(cfg, "U1_soft_target")) return; + + // alpha = 1.0 is NISPS today (no target net at all). 0.005 is upstream. + const float alphas[] = {1.0f, 0.5f, 0.1f, 0.005f}; + + for (float alpha : alphas) { + Rig rig(cfg); + TargetNet tgt(cfg); + // Start the target identical to the online net, or the first + // measurement would report the gap between two random inits. + { auto w = rig.mlp.get_weights(); + std::vector v(w.begin(), w.end()); + tgt.mlp.set_weights(v); } + + const std::size_t N = cfg.smoke ? 8u : 20u; + Dataset ds = make_dataset(cfg, "scattered", N, rig.rng); + + // Field as heard THROUGH the target net. + auto heard_field = [&](std::vector& dst) { + dst.resize(rig.n_pts() * cfg.n_out); + tgt.mlp.infer_batch(rig.probe_pts, dst); + }; + + std::vector prev, cur; + heard_field(prev); + + std::vector lurches; + for (std::size_t k = 0; k < N; ++k) { + place_positive(rig, std::span(&ds.xs[k * cfg.n_in], cfg.n_in), + std::span(&ds.ys[k * cfg.n_out], cfg.n_out)); + rig.mlp.train(); + soft_update(tgt.mlp, rig.mlp, alpha); + heard_field(cur); + // Mean per-point displacement of the HEARD mapping, this gesture. + float acc = 0.f; + for (std::size_t i = 0; i < rig.n_pts(); ++i) { + acc += l2(std::span(&prev[i * cfg.n_out], cfg.n_out), + std::span(&cur[i * cfg.n_out], cfg.n_out)); + } + lurches.push_back(acc / static_cast(rig.n_pts())); + prev = cur; + } + + // Retention, measured through the target — what the user can actually + // reach, which is the only retention that counts. + std::vector got(cfg.n_out); + float retain = 0.f; + for (std::size_t i = 0; i < N; ++i) { + std::span x(&ds.xs[i * cfg.n_in], cfg.n_in); + for (std::size_t j = 0; j < cfg.n_in; ++j) tgt.mlp.set_input(j, x[j]); + tgt.mlp.process(); + auto o = tgt.mlp.outputs(); + for (std::size_t j = 0; j < cfg.n_out; ++j) got[j] = o[j]; + retain += l2(got, std::span(&ds.ys[i * cfg.n_out], cfg.n_out)); + } + + char id[64]; + snprintf(id, sizeof id, "U1_soft_target_a%04d", static_cast(alpha * 1000.f)); + js.begin_scenario(id, "outputs heard through a soft-updated target net"); + js.kv("alpha", alpha); + js.kv("is_nisps_today", static_cast(alpha >= 1.f ? 1u : 0u)); + js.kv("lurch_mean", mean(lurches)); + js.kv("lurch_p95", percentile(lurches, 0.95f)); + js.kv("lurch_max", percentile(lurches, 1.0f)); + js.kv("retain_mean", retain / static_cast(N)); + js.end_scenario(); + } +} + +// U2 — amortised learning: train every Nth gesture (upstream optimiseDivisor). +// Examples still accumulate every gesture; only the training is batched. +void upstream_amortised(const Config& cfg, Json& js) { + if (!selected(cfg, "U2_amortised")) return; + + const std::size_t divisors[] = {1u, 4u, 10u, 40u}; // 1 = NISPS today, 40 = upstream + + for (std::size_t div : divisors) { + Rig rig(cfg); + const std::size_t N = cfg.smoke ? 16u : 80u; + Dataset ds = make_dataset(cfg, "scattered", N, rig.rng); + + std::vector prev, cur; + rig.field(prev); + std::vector lurches; + + std::size_t trainings = 0u; + for (std::size_t k = 0; k < N; ++k) { + place_positive(rig, std::span(&ds.xs[k * cfg.n_in], cfg.n_in), + std::span(&ds.ys[k * cfg.n_out], cfg.n_out)); + if ((k + 1u) % div == 0u) { rig.mlp.train(); ++trainings; } + rig.field(cur); + float acc = 0.f; + for (std::size_t i = 0; i < rig.n_pts(); ++i) { + acc += l2(std::span(&prev[i * cfg.n_out], cfg.n_out), + std::span(&cur[i * cfg.n_out], cfg.n_out)); + } + lurches.push_back(acc / static_cast(rig.n_pts())); + prev = cur; + } + rig.mlp.train(); // settle before measuring retention + + std::vector got; + float retain = 0.f; + for (std::size_t i = 0; i < N; ++i) { + rig.at(std::span(&ds.xs[i * cfg.n_in], cfg.n_in), got); + retain += l2(got, std::span(&ds.ys[i * cfg.n_out], cfg.n_out)); + } + + char id[64]; + snprintf(id, sizeof id, "U2_amortised_div%02zu", div); + js.begin_scenario(id, "train every Nth gesture (upstream optimiseDivisor)"); + js.kv("divisor", div); + js.kv("is_nisps_today", static_cast(div == 1u ? 1u : 0u)); + js.kv("gestures", N); + // A divisor larger than the session length never fires, which would + // otherwise report lurch=0 as if it were a result. Read this column + // before believing the row. + js.kv("trainings", trainings); + js.kv("lurch_mean", mean(lurches)); + js.kv("lurch_p95", percentile(lurches, 0.95f)); + js.kv("lurch_max", percentile(lurches, 1.0f)); + js.kv("retain_mean", retain / static_cast(N)); + FieldMetrics fm = measure_field(rig); + js.field("field_", fm); + js.end_scenario(); + } +} + +// U3 — upstream's ACTOR SHAPE against the NISPS default. Upstream ran +// {n_in+1, 10, 10, n_out}: TWO hidden layers of 10. NISPS's dynamic storage +// fixes the topology at three hidden layers, so the closest reachable +// comparison is {10,10,X} against the default {16,16,16}. Reported so the +// architecture difference is not silently conflated with the output-path one. +void upstream_actor_shape(const Config& cfg, Json& js) { + if (!selected(cfg, "U3_actor_shape")) return; + struct S { const char* id; std::size_t h[3]; }; + const S shapes[] = { + {"U3_actor_shape_nisps", {16u, 16u, 16u}}, + {"U3_actor_shape_upstream", {10u, 10u, 1u}}, // 3rd layer minimal ~ 2 hidden + {"U3_actor_shape_10_10_10", {10u, 10u, 10u}}, + }; + for (const S& sh : shapes) { + Config c2 = cfg; + c2.hidden[0] = sh.h[0]; c2.hidden[1] = sh.h[1]; c2.hidden[2] = sh.h[2]; + Rig rig(c2); + const std::size_t N = c2.smoke ? 8u : 20u; + Dataset ds = make_dataset(c2, "scattered", N, rig.rng); + teach(rig, ds); + rig.mlp.train(); + std::vector got; + float retain = 0.f; + for (std::size_t i = 0; i < N; ++i) { + rig.at(std::span(&ds.xs[i * c2.n_in], c2.n_in), got); + retain += l2(got, std::span(&ds.ys[i * c2.n_out], c2.n_out)); + } + js.begin_scenario(sh.id, "upstream actor shape vs the NISPS default"); + js.kv("h0", sh.h[0]); js.kv("h1", sh.h[1]); js.kv("h2", sh.h[2]); + js.kv("weights", rig.mlp.weight_count()); + js.kv("retain_mean", retain / static_cast(N)); + FieldMetrics fm = measure_field(rig); + js.field("field_", fm); + js.end_scenario(); + } +} + + +// U4 — BALANCED POSITIVES, the upstream way. +// +// Upstream e291192 optimise() trains likes with ONE epoch of TrainBatch over a +// random 8-sample from replay, at effLR * avgRewardPos where learningRate = +// 1e-3 (InterfaceRL.tpp:665, InterfaceRL.hpp:430,436). NISPS instead calls +// train() — the WHOLE dataset, lr 1.0, up to 1000 iterations, every gesture. +// +// That is the same root cause as the inert dislike, seen from the other side: +// upstream keeps likes and dislikes on ONE scale (~1e-3 batch steps), so a 'no' +// is comparable in force to a 'yes'. NISPS trains likes ~2e6x harder than +// dislikes, which simultaneously makes the dislike feel like nothing and makes +// every like heave the whole mapping. +// +// !!! READ BEFORE INTERPRETING THE upstream_* ROWS !!! +// Upstream's LR is an RMSPROP learning rate; ours is an SGD one. Upstream memlp +// (pinned ea777502) applies gradients with RMSProp everywhere — Layer.h:239 +// "Apply gradients (RMSProp)", the m_sq_grad_avg running squared-gradient +// average at Layer.h:601, StaticMLP.h:268 "Mini-batch RMSProp training". +// nisps/ml/training.hpp ships SGD only (its own header says so). RMSProp divides +// each step by the running gradient magnitude, so lr=1e-3 there is a NORMALISED +// step; lr=1e-3 under SGD is literally 1e-3 x the raw gradient. The two numbers +// are not comparable, and the U4 upstream_* rows therefore do NOT show that +// upstream's dose fails to learn — they show that upstream's NUMBER means +// something else in our optimiser. Treat them as an SGD sensitivity sweep, and +// see ALIGNMENT.md defect 6. +// +// Reproduced here exactly: train(lr, iters, min_err) with iters=1 IS one epoch +// over the dataset. The comparison is lurch (how far the played mapping moves +// per gesture) against retention (whether it still learns anything). +void upstream_balanced_positives(const Config& cfg, Json& js) { + if (!selected(cfg, "U4_balanced_positives")) return; + + // ticks = how many times the trainer runs per USER GESTURE. This is the + // dimension I first got wrong, and it dominates the comparison. + // + // NISPS trains once per gesture, synchronously, on the press. + // Upstream calls optimiseSometimes() from the loopCallback EVERY TICK with + // optimiseDivisor=1 (InterfaceRL.tpp:229, InterfaceRL.hpp:380), and the loop + // runs at 200 Hz (kJoltLRRampStep is documented as 1/(5s * 200Hz), + // InterfaceRL.hpp:457). So between two gestures ~3 s apart upstream has run + // ~600 optimise() calls. Giving its per-call dose only once per gesture + // measures 1/600th of the real thing. + struct V { const char* id; float lr; std::size_t iters; std::size_t ticks; }; + const V variants[] = { + {"U4_pos_nisps_today", 1.0f, 1000u, 1u}, // train() defaults, on the press + {"U4_pos_lr1_iter1", 1.0f, 1u, 1u}, // isolate: iterations alone + {"U4_pos_upstream_1tick", 0.001f, 1u, 1u}, // upstream dose, 1/600th of its rate + {"U4_pos_upstream_100tick", 0.001f, 1u, 100u}, + {"U4_pos_upstream_600tick", 0.001f, 1u, 600u}, // ~3 s between gestures @200 Hz + }; + + for (const V& v : variants) { + Rig rig(cfg); + const std::size_t N = cfg.smoke ? 10u : 40u; + Dataset ds = make_dataset(cfg, "scattered", N, rig.rng); + + std::vector prev, cur; + rig.field(prev); + std::vector lurches; + + // Per-TICK lurch is what the musician feels: the mapping is live and they + // hear every update, not a per-gesture summary. Sampling every tick at + // 600 ticks/gesture would cost 600 field evaluations per gesture, so the + // field is sampled on a stride and the stride is reported. + const std::size_t stride = (v.ticks > 20u) ? (v.ticks / 20u) : 1u; + std::size_t samples = 0u; + for (std::size_t k = 0; k < N; ++k) { + place_positive(rig, std::span(&ds.xs[k * cfg.n_in], cfg.n_in), + std::span(&ds.ys[k * cfg.n_out], cfg.n_out)); + for (std::size_t t = 0; t < v.ticks; ++t) { + rig.mlp.train(v.lr, v.iters, 0.f); + if ((t + 1u) % stride != 0u && t + 1u != v.ticks) continue; + rig.field(cur); + float acc = 0.f; + for (std::size_t i = 0; i < rig.n_pts(); ++i) { + acc += l2(std::span(&prev[i * cfg.n_out], cfg.n_out), + std::span(&cur[i * cfg.n_out], cfg.n_out)); + } + // Normalise to per-tick so the rows are comparable. + lurches.push_back(acc / static_cast(rig.n_pts()) + / static_cast(stride)); + prev = cur; + ++samples; + } + } + (void)samples; + + std::vector got; + float retain = 0.f; + for (std::size_t i = 0; i < N; ++i) { + rig.at(std::span(&ds.xs[i * cfg.n_in], cfg.n_in), got); + retain += l2(got, std::span(&ds.ys[i * cfg.n_out], cfg.n_out)); + } + + js.begin_scenario(v.id, "positive-path training dose: lurch vs retention"); + js.kv("lr", v.lr); + js.kv("iters", v.iters); + js.kv("ticks_per_gesture", v.ticks); + js.kv("field_sample_stride", stride); + js.kv("gestures", N); + js.kv("lurch_mean", mean(lurches)); + js.kv("lurch_max", percentile(lurches, 1.0f)); + js.kv("retain_mean", retain / static_cast(N)); + FieldMetrics fm = measure_field(rig); + js.field("field_", fm); + js.end_scenario(); + } +} + +// --------------------------------------------------------------------------- +void run(const Config& cfg) { + Json js; + js.begin_run(cfg); + probe_at_example(cfg, js); + probe_around_example(cfg, js); + probe_far_field(cfg, js); + probe_negative_once(cfg, js); + probe_negative_twice(cfg, js); + probe_negative_adjacent(cfg, js); + probe_negative_near_positive(cfg, js); + probe_randomise_and_place(cfg, js); + journey_positive_only(cfg, js); + journey_randomise_place_only(cfg, js); + journey_mixed(cfg, js); + journey_branch(cfg, js); + diag_geo_anatomy(cfg, js); + probe_explore_place(cfg, js); + probe_explore_cancel(cfg, js); + probe_reposition(cfg, js); + probe_like_then_dislike(cfg, js); + probe_dislike_then_repair(cfg, js); + probe_focus_mask(cfg, js); + journey_explore_place_only(cfg, js); + journey_long_session(cfg, js); + journey_dislike_storm(cfg, js); + journey_revisit(cfg, js); + journey_two_region(cfg, js); + journey_sweep_teach(cfg, js); + journey_undo_heavy(cfg, js); + upstream_soft_target(cfg, js); + upstream_amortised(cfg, js); + upstream_actor_shape(cfg, js); + upstream_balanced_positives(cfg, js); + edge_cases(cfg, js); + edge_cases_2(cfg, js); + js.end_run(); +} + +void usage() { + fprintf(stderr, + "ml_bench — behavioural benchmark for the NISPS control mapping\n" + "\n" + " --shape N_IN,H1,H2,H3,N_OUT default 2,16,16,16,8\n" + " --seed N default 0x5EED\n" + " --max-examples N default 128\n" + " --scenario ID run one scenario\n" + " --spread F 1=Xavier (default), 0=uniform/no fan_in\n" + " --geo-lr F geometric-dislike LR (default 0.001)\n" + " --geo-iters N gradient steps per dislike (default 1)\n" + " --smoke reduced point counts\n" + "\n" + "Emits JSON on stdout. Asserts nothing — see scripts/bench-ml.sh --compare.\n"); +} + +} // namespace + +int main(int argc, char** argv) { + Config cfg; + for (int i = 1; i < argc; ++i) { + const std::string a = argv[i]; + auto next = [&]() -> const char* { return (i + 1 < argc) ? argv[++i] : nullptr; }; + if (a == "--shape") { + const char* v = next(); + if (!v) { usage(); return 2; } + std::size_t d[5] = {0, 0, 0, 0, 0}; + int n = std::sscanf(v, "%zu,%zu,%zu,%zu,%zu", &d[0], &d[1], &d[2], &d[3], &d[4]); + if (n != 5) { fprintf(stderr, "ml_bench: --shape needs 5 comma-separated dims\n"); return 2; } + cfg.n_in = d[0]; cfg.hidden[0] = d[1]; cfg.hidden[1] = d[2]; + cfg.hidden[2] = d[3]; cfg.n_out = d[4]; + } else if (a == "--seed") { + const char* v = next(); + if (!v) { usage(); return 2; } + cfg.seed = std::strtoull(v, nullptr, 0); + } else if (a == "--max-examples") { + const char* v = next(); + if (!v) { usage(); return 2; } + cfg.max_examples = std::strtoull(v, nullptr, 0); + } else if (a == "--scenario") { + const char* v = next(); + if (!v) { usage(); return 2; } + cfg.only = v; + } else if (a == "--spread") { + const char* v = next(); + if (!v) { usage(); return 2; } + cfg.spread = static_cast(std::atof(v)); + } else if (a == "--geo-lr") { + const char* v = next(); + if (!v) { usage(); return 2; } + cfg.geo_lr = static_cast(std::atof(v)); + } else if (a == "--geo-iters") { + const char* v = next(); + if (!v) { usage(); return 2; } + cfg.geo_iters = std::strtoull(v, nullptr, 0); + } else if (a == "--smoke") { + cfg.smoke = true; + } else if (a == "--help" || a == "-h") { + usage(); + return 0; + } else { + fprintf(stderr, "ml_bench: unknown arg '%s'\n", a.c_str()); + usage(); + return 2; + } + } + if (cfg.n_in == 0u || cfg.n_out == 0u || + cfg.hidden[0] == 0u || cfg.hidden[1] == 0u || cfg.hidden[2] == 0u) { + fprintf(stderr, "ml_bench: all dims must be >= 1\n"); + return 2; + } + run(cfg); + return 0; +} diff --git a/tests/cpp/test_ml_behaviour.cpp b/tests/cpp/test_ml_behaviour.cpp new file mode 100644 index 0000000..0058a84 --- /dev/null +++ b/tests/cpp/test_ml_behaviour.cpp @@ -0,0 +1,551 @@ +// tests/cpp/test_ml_behaviour.cpp — INVARIANTS of the interaction model. +// +// Split of responsibility with tests/cpp/ml_bench.cpp: +// ml_bench.cpp REPORTS behaviour (how local is a dislike? how playable is +// the mapping?). Those are descriptions, not pass/fail, so it +// asserts nothing and is not a ctest gate. +// this file ASSERTS the things that must be TRUE regardless of tuning. +// If one of these breaks, the interaction model is broken, not +// merely different — a musician would call it a bug. +// +// Every test here is shape-parameterised where it can be, and runs at the +// operator's default 2-in/8-out as well as at degenerate shapes, because the +// edge cases are exactly where an interaction model quietly stops holding. +// +// These use MLPCore, which is host-only by construction +// (dynamic_storage.hpp #errors on RP2350). That is fine: these are behavioural +// invariants of the shared algorithms, and the fixed-storage firmware path is +// bit-pinned to the dynamic one by test_mlp_storage_parity.cpp. + +#include +#include +#include + +#include "../../nisps/ml/dynamic_storage.hpp" +#include "../../nisps/ml/feedback.hpp" +#include "../../nisps/ml/mlp.hpp" +#include "test_helpers.hpp" + +namespace { + +using nisps::ml::AvoidStyle; +using nisps::ml::DynamicFeedbackStorage; +using nisps::ml::DynamicStorage; +using nisps::ml::FeedbackControllerCore; +using nisps::ml::FeedbackMode; +using nisps::ml::MLPCore; + +using Mlp = MLPCore; +using Feedback = FeedbackControllerCore; + +constexpr std::uint64_t kSeed = 0xC0FFEEu; + +struct Rig { + std::size_t n_in, n_out; + Mlp mlp; + Feedback fb; + + Rig(std::size_t nin, std::size_t nout, std::size_t h = 16u, + std::size_t max_examples = 128u) + : n_in(nin), n_out(nout), + mlp(kSeed, nin, std::span(hidden_(h), 3u), nout, + max_examples, 4096u), + fb(kSeed ^ 0xF33Dull, nout, mlp.weight_count(), 4u, nin, 64u) {} + + // Static storage for the hidden dims so the span outlives the ctor call. + static const std::size_t* hidden_(std::size_t h) { + static std::size_t buf[3]; + buf[0] = buf[1] = buf[2] = h; + return buf; + } + + void at(std::span x, std::vector& out) { + out.assign(n_out, 0.f); + for (std::size_t i = 0; i < n_in; ++i) mlp.set_input(i, x[i]); + mlp.process(); + auto o = mlp.outputs(); + for (std::size_t j = 0; j < n_out; ++j) out[j] = o[j]; + } + + // The product path for a thumbs-up: dataset example AND replay positive. + void like(std::span x, std::span y) { + for (std::size_t i = 0; i < n_in; ++i) mlp.set_input(i, x[i]); + mlp.process(); + mlp.add_example(x, y); + fb.store_positive(mlp, y); + } + + void down(std::span x, float speed = 0.1f, float spread = 1.f) { + std::vector heard; + at(x, heard); + for (std::size_t i = 0; i < n_in; ++i) mlp.set_input(i, x[i]); + mlp.process(); + fb.on_down(mlp, heard, speed, spread, {}); + } + + // get_weights() returns a span into a storage-owned scratch buffer, so it + // must be COPIED before the next call reuses that buffer. + void weights(std::vector& w) { + auto s = mlp.get_weights(); + w.assign(s.begin(), s.end()); + } +}; + +float l2(std::span a, std::span b) { + float acc = 0.f; + const std::size_t n = a.size() < b.size() ? a.size() : b.size(); + for (std::size_t i = 0; i < n; ++i) { const float d = a[i] - b[i]; acc += d * d; } + return std::sqrt(acc); +} + +bool all_finite(std::span v) { + for (float x : v) if (!std::isfinite(x)) return false; + return true; +} + +} // namespace + +// =========================================================================== +// PLACEMENT — a thumbs-up must actually take. +// =========================================================================== + +// The single most basic promise the instrument makes: if you stand somewhere, +// like a set of outputs, and then stand in the SAME place again, you get +// approximately what you liked. Without this nothing else means anything. +NISPS_TEST(behaviour_single_like_is_reachable) { + Rig rig(2u, 8u); + const float x[2] = {0.25f, -0.4f}; + std::vector y(8u); + for (std::size_t j = 0; j < 8u; ++j) y[j] = 0.2f + 0.07f * static_cast(j); + + rig.like(x, y); + rig.mlp.train(); + + std::vector got; + rig.at(x, got); + NISPS_EXPECT(all_finite(got)); + // Loose bound on purpose: this asserts "the placement took", not "the + // optimiser is good". Tightening it would make it a tuning test. + NISPS_EXPECT(l2(got, y) < 0.25f); +} + +// Placing at the same position twice with a NEW target must move toward the +// new one — the musician overwrote their earlier choice and expects that to win. +NISPS_TEST(behaviour_relike_same_spot_overwrites) { + Rig rig(2u, 8u); + const float x[2] = {0.1f, 0.1f}; + std::vector y1(8u, 0.2f), y2(8u, 0.8f), got; + + rig.like(x, y1); + rig.mlp.train(); + rig.at(x, got); + const float err_to_y2_before = l2(got, y2); + + rig.like(x, y2); + rig.mlp.train(); + rig.at(x, got); + const float err_to_y2_after = l2(got, y2); + + NISPS_EXPECT(err_to_y2_after < err_to_y2_before); +} + +// =========================================================================== +// FEEDBACK — the negative path must be well-behaved even when it is gentle. +// =========================================================================== + +// A dislike must never produce non-finite outputs. This is the one thing that +// turns "the instrument feels wrong" into "the instrument is dead", and a +// perturbation path plus a sigmoid is exactly where NaN would come from. +NISPS_TEST(behaviour_dislike_never_produces_nonfinite) { + for (AvoidStyle style : {AvoidStyle::Geometric, AvoidStyle::Diffuse}) { + Rig rig(2u, 8u); + std::vector y(8u, 0.5f), got; + const float a[2] = {0.3f, 0.3f}; + const float b[2] = {-0.3f, 0.2f}; + rig.like(a, y); + rig.like(b, y); + rig.mlp.train(); + + rig.fb.set_mode(FeedbackMode::Avoid, rig.mlp); + rig.fb.set_avoid_style(style); + const float press[2] = {0.f, 0.25f}; + for (int i = 0; i < 32; ++i) rig.down(press); + + rig.at(press, got); + NISPS_EXPECT(all_finite(got)); + std::vector w; + rig.weights(w); + NISPS_EXPECT(all_finite(w)); + } +} + +// Dislike at cold start (no positives at all) must not crash or corrupt. This +// is literally the first thing a new user might press. +NISPS_TEST(behaviour_dislike_cold_start_is_safe) { + Rig rig(2u, 8u); + rig.fb.set_mode(FeedbackMode::Avoid, rig.mlp); + rig.fb.set_avoid_style(AvoidStyle::Geometric); + const float press[2] = {0.f, 0.f}; + + NISPS_ASSERT(rig.fb.positive_count() == 0u); + for (int i = 0; i < 8; ++i) rig.down(press); + + std::vector got; + rig.at(press, got); + NISPS_EXPECT(all_finite(got)); + NISPS_EXPECT(rig.fb.negative_count() > 0u); +} + +// Repeated dislikes at the same point must deepen ONE replay entry rather than +// filling the buffer — replay.hpp's dedup radius (0.05) exists for exactly +// this, and if it regresses a user holding the button evicts their own history. +NISPS_TEST(behaviour_repeat_dislike_dedups_not_accumulates) { + Rig rig(2u, 8u); + std::vector y(8u, 0.5f); + const float a[2] = {0.4f, 0.4f}; + rig.like(a, y); + rig.mlp.train(); + + rig.fb.set_mode(FeedbackMode::Avoid, rig.mlp); + rig.fb.set_avoid_style(AvoidStyle::Geometric); + const float press[2] = {-0.2f, 0.1f}; + for (int i = 0; i < 20; ++i) rig.down(press); + + // 20 presses inside the dedup radius: one negative, deepened. + NISPS_EXPECT(rig.fb.negative_count() == 1u); +} + +// A dislike far outside the dedup radius must create a SEPARATE negative. +// Together with the test above this pins the "move a little to the left" +// behaviour the operator asked about: inside 0.05 deepens, outside stores. +NISPS_TEST(behaviour_distant_dislike_stores_separately) { + Rig rig(2u, 8u); + std::vector y(8u, 0.5f); + const float a[2] = {0.4f, 0.4f}; + rig.like(a, y); + rig.mlp.train(); + + rig.fb.set_mode(FeedbackMode::Avoid, rig.mlp); + rig.fb.set_avoid_style(AvoidStyle::Geometric); + const float p1[2] = {-0.5f, 0.f}; + const float p2[2] = {0.5f, 0.f}; // well beyond kReplayDedupRadius + rig.down(p1); + rig.down(p2); + NISPS_EXPECT(rig.fb.negative_count() == 2u); +} + +// =========================================================================== +// EXPLORE / UNDO — the scratchpad must be exactly reversible. +// =========================================================================== + +// Enter explore, roll, then undo everything and exit: the net must return to +// EXACTLY where it started. Not approximately — the whole point of the +// scratchpad model is that auditioning costs nothing. +NISPS_TEST(behaviour_explore_undo_restores_exactly) { + Rig rig(2u, 8u); + std::vector y(8u, 0.5f); + const float a[2] = {0.2f, -0.2f}; + rig.like(a, y); + rig.mlp.train(); + + std::vector before, after; + rig.weights(before); + + rig.fb.set_mode(FeedbackMode::ExploreAndPlace, rig.mlp); + rig.fb.enter_explore(rig.mlp, 1.f); + for (int i = 0; i < 3; ++i) rig.fb.reroll(rig.mlp, 1.f); + for (int i = 0; i < 3; ++i) rig.fb.undo(rig.mlp); + rig.fb.exit_explore(rig.mlp); + + rig.weights(after); + bool identical = true; + for (std::size_t i = 0; i < before.size(); ++i) { + if (before[i] != after[i]) { identical = false; break; } + } + NISPS_EXPECT(identical); +} + +// Undoing more times than the ring holds must not corrupt anything. The user +// mashing undo is not a bug report. +NISPS_TEST(behaviour_undo_past_ring_depth_is_safe) { + Rig rig(2u, 8u); + std::vector y(8u, 0.5f); + const float a[2] = {0.f, 0.f}; + rig.like(a, y); + rig.mlp.train(); + + rig.fb.set_mode(FeedbackMode::ExploreAndPlace, rig.mlp); + rig.fb.enter_explore(rig.mlp, 1.f); + for (int i = 0; i < 3; ++i) rig.fb.reroll(rig.mlp, 1.f); + for (int i = 0; i < 40; ++i) rig.fb.undo(rig.mlp); // way past depth 4 + rig.fb.exit_explore(rig.mlp); + + std::vector w, got; + rig.weights(w); + NISPS_EXPECT(all_finite(w)); + rig.at(a, got); + NISPS_EXPECT(all_finite(got)); +} + +// Output randomisation must NOT touch weights. This is the distinction the +// operator drew explicitly: RandomiseOutputs rolls a patch, RandomiseMlp +// scrambles the net. If the former ever moved weights, auditioning a patch +// would silently damage the instrument. +NISPS_TEST(behaviour_randomise_outputs_leaves_weights_untouched) { + Rig rig(2u, 8u); + std::vector y(8u, 0.5f); + const float a[2] = {0.15f, 0.15f}; + rig.like(a, y); + rig.mlp.train(); + + std::vector before, after; + rig.weights(before); + + rig.fb.set_mode(FeedbackMode::RandomiseOutputs, rig.mlp); + const float press[2] = {-0.1f, 0.3f}; + for (int i = 0; i < 5; ++i) rig.down(press); // enter + 4 re-rolls + + rig.weights(after); + bool identical = true; + for (std::size_t i = 0; i < before.size(); ++i) { + if (before[i] != after[i]) { identical = false; break; } + } + NISPS_EXPECT(identical); + + // ...and it must actually be holding a patch to audition. + std::vector patch(8u, 0.f); + NISPS_EXPECT(rig.fb.static_output(patch)); + NISPS_EXPECT(all_finite(patch)); +} + +// =========================================================================== +// DETERMINISM — the benchmark is worthless if the engine is not reproducible. +// =========================================================================== + +// Two rigs built with the same seed, driven through the same gesture sequence, +// must end bit-identical. This is what makes a behavioural benchmark +// comparable across commits at all. +NISPS_TEST(behaviour_same_seed_same_journey_is_bit_identical) { + auto journey = [](Rig& rig) { + std::vector y(8u); + for (std::size_t j = 0; j < 8u; ++j) y[j] = 0.3f + 0.05f * static_cast(j); + const float a[2] = {0.2f, 0.1f}; + const float b[2] = {-0.3f, 0.4f}; + rig.like(a, y); + rig.like(b, y); + rig.mlp.train(); + rig.fb.set_mode(FeedbackMode::Avoid, rig.mlp); + rig.fb.set_avoid_style(AvoidStyle::Diffuse); + const float press[2] = {0.f, 0.f}; + for (int i = 0; i < 5; ++i) rig.down(press); + }; + + Rig r1(2u, 8u), r2(2u, 8u); + journey(r1); + journey(r2); + + std::vector w1, w2; + r1.weights(w1); + r2.weights(w2); + NISPS_ASSERT(w1.size() == w2.size()); + bool identical = true; + for (std::size_t i = 0; i < w1.size(); ++i) { + if (w1[i] != w2[i]) { identical = false; break; } + } + NISPS_EXPECT(identical); +} + +// =========================================================================== +// CAPACITY + DEGENERATE SHAPES +// =========================================================================== + +// Overflowing the example ring must not corrupt state — it evicts, silently, +// and the newest examples must still be the ones honoured. +NISPS_TEST(behaviour_example_overflow_keeps_newest) { + const std::size_t kCap = 8u; + Rig rig(2u, 8u, 16u, kCap); + std::vector got; + + // 3x the cap. Each example has a distinguishable target. + const std::size_t N = kCap * 3u; + std::vector xs(N * 2u), ys(N * 8u); + for (std::size_t i = 0; i < N; ++i) { + const float t = static_cast(i) / static_cast(N); + xs[i * 2u] = 2.f * t - 1.f; + xs[i * 2u + 1u] = 0.f; + for (std::size_t j = 0; j < 8u; ++j) ys[i * 8u + j] = t; + rig.like(std::span(&xs[i * 2u], 2u), + std::span(&ys[i * 8u], 8u)); + } + rig.mlp.train(); + + // The LAST example must be honoured; it cannot have been evicted. + rig.at(std::span(&xs[(N - 1u) * 2u], 2u), got); + NISPS_EXPECT(all_finite(got)); + NISPS_EXPECT(l2(got, std::span(&ys[(N - 1u) * 8u], 8u)) < 0.5f); +} + +// Contradictory examples (same input, different targets) must converge to +// something finite rather than diverging. A musician WILL do this by accident. +NISPS_TEST(behaviour_contradictory_examples_stay_finite) { + Rig rig(2u, 8u); + const float x[2] = {0.f, 0.f}; + for (int k = 0; k < 6; ++k) { + std::vector y(8u, 0.1f + 0.15f * static_cast(k)); + rig.like(x, y); + } + rig.mlp.train(); + + std::vector got, w; + rig.at(x, got); + rig.weights(w); + NISPS_EXPECT(all_finite(got)); + NISPS_EXPECT(all_finite(w)); +} + +// The interaction model must hold at degenerate shapes, not just the default. +// 1-in/1-out is the narrowest legal net; a wide-output net is the realistic +// upper end once a mode drives 33 synth parameters. +NISPS_TEST(behaviour_holds_at_degenerate_shapes) { + struct Shape { std::size_t n_in, n_out, hidden; }; + const Shape shapes[] = { + {1u, 1u, 4u}, // narrowest legal + {2u, 8u, 16u}, // operator default + {1u, 33u, 8u}, // one control, many parameters + {8u, 2u, 8u}, // many controls, few parameters + {32u, 8u, 16u}, // the over-provisioned browser head + }; + for (const Shape& s : shapes) { + Rig rig(s.n_in, s.n_out, s.hidden); + std::vector x(s.n_in, 0.2f), y(s.n_out, 0.6f), got, w; + rig.like(x, y); + rig.mlp.train(); + rig.at(x, got); + NISPS_EXPECT(all_finite(got)); + + rig.fb.set_mode(FeedbackMode::Avoid, rig.mlp); + rig.fb.set_avoid_style(AvoidStyle::Geometric); + rig.down(x); + rig.weights(w); + NISPS_EXPECT(all_finite(w)); + NISPS_EXPECT(rig.mlp.n_in() == s.n_in); + NISPS_EXPECT(rig.mlp.n_out() == s.n_out); + } +} + +// =========================================================================== +// EXPLORE-AND-PLACE ACCESSOR CONTRACT +// +// This exists because getting it wrong fails SILENTLY, and it did: reading +// placed_output() after commit_place() returns an EMPTY span, l2() over an +// empty span is 0, and 0 reads as a perfect placement. A benchmark scored a +// broken lifecycle as flawless until ASAN found the out-of-bounds downstream. +// =========================================================================== +NISPS_TEST(behaviour_placed_vs_committed_output_contract) { + Rig rig(2u, 8u); + std::vector y(8u, 0.4f); + const float a[2] = {0.1f, -0.1f}; + rig.like(a, y); + rig.mlp.train(); + + rig.fb.set_mode(FeedbackMode::ExploreAndPlace, rig.mlp); + rig.fb.enter_explore(rig.mlp, 1.f); + rig.fb.reroll(rig.mlp, 1.f); + + // While Exploring: nothing is placed yet. + NISPS_EXPECT(rig.fb.placed_output().empty()); + + rig.mlp.process(); + std::vector audition(8u); + { auto o = rig.mlp.outputs(); + for (std::size_t j = 0; j < 8u; ++j) audition[j] = o[j]; } + rig.fb.begin_place(rig.mlp, audition); + + // While Placing: placed_output() is the live accessor. + NISPS_ASSERT(rig.fb.placed_output().size() == 8u); + NISPS_EXPECT(l2(rig.fb.placed_output(), audition) == 0.f); + + rig.fb.commit_place(rig.mlp); + + // After commit: state is Idle, so placed_output() goes EMPTY and the + // caller must read committed_output() instead. + NISPS_EXPECT(rig.fb.placed_output().empty()); + NISPS_ASSERT(rig.fb.committed_output().size() == 8u); + NISPS_EXPECT(l2(rig.fb.committed_output(), audition) == 0.f); +} + +// Reposition hands its carried vector over the same way. +NISPS_TEST(behaviour_reposition_carries_via_committed_output) { + Rig rig(2u, 8u); + std::vector y(8u, 0.35f); + const float src[2] = {0.3f, 0.3f}; + rig.like(src, y); + rig.mlp.train(); + + rig.fb.set_mode(FeedbackMode::ExploreAndPlace, rig.mlp); + for (std::size_t i = 0; i < 2u; ++i) rig.mlp.set_input(i, src[i]); + rig.mlp.process(); + rig.fb.begin_reposition(rig.mlp); + + NISPS_ASSERT(rig.fb.placed_output().size() == 8u); + std::vector grabbed(rig.fb.placed_output().begin(), + rig.fb.placed_output().end()); + + rig.fb.commit_reposition(); + NISPS_ASSERT(rig.fb.committed_output().size() == 8u); + NISPS_EXPECT(l2(rig.fb.committed_output(), grabbed) == 0.f); + NISPS_EXPECT(!rig.fb.repositioning()); +} + +// A fully-masked focus gate must freeze EVERY output: zero active dims means +// zero gradient, so a dislike is a no-op on the weights. +NISPS_TEST(behaviour_full_focus_mask_freezes_everything) { + Rig rig(2u, 8u); + std::vector y(8u, 0.5f); + const float a[2] = {0.2f, 0.2f}; + rig.like(a, y); + rig.mlp.train(); + + std::vector mask(8u, 0u); // nothing active + rig.fb.set_focus_mask(mask); + rig.fb.set_mode(FeedbackMode::Avoid, rig.mlp); + rig.fb.set_avoid_style(AvoidStyle::Geometric); + + std::vector before, after; + rig.weights(before); + const float press[2] = {-0.1f, 0.1f}; + for (int i = 0; i < 16; ++i) rig.down(press); + rig.weights(after); + + bool identical = true; + for (std::size_t i = 0; i < before.size(); ++i) { + if (before[i] != after[i]) { identical = false; break; } + } + NISPS_EXPECT(identical); +} + +// Switching feedback mode mid-exploration must tear down cleanly and never +// strand the net in a randomised scratchpad state. Live, that would be fatal. +NISPS_TEST(behaviour_mode_switch_midflight_never_strands) { + Rig rig(2u, 8u); + std::vector y(8u, 0.45f); + const float a[2] = {0.f, 0.2f}; + rig.like(a, y); + rig.mlp.train(); + + std::vector before, after; + rig.weights(before); + + rig.fb.set_mode(FeedbackMode::ExploreAndPlace, rig.mlp); + rig.fb.enter_explore(rig.mlp, 1.f); + for (int i = 0; i < 3; ++i) rig.fb.reroll(rig.mlp, 1.f); + rig.fb.set_mode(FeedbackMode::Avoid, rig.mlp); // yank it out + + rig.weights(after); + bool identical = true; + for (std::size_t i = 0; i < before.size(); ++i) { + if (before[i] != after[i]) { identical = false; break; } + } + NISPS_EXPECT(identical); + NISPS_EXPECT(!rig.fb.exploring()); +}