test(ml): behavioural benchmark + 20 invariants for the control mapping
NISPS is a controller, not a synth: the object of study is the mapping
f: control-space -> parameter-space and how a musician's gestures deform
it. Loss measures fit to points the user dictated, which is the one thing
they never experience. So this measures geometry and gesture-response.
tests/cpp/ml_bench.cpp 61 scenarios, REPORTS never asserts (same
discipline as engine_bench.cpp). Shape-
agnostic via MLPCore<DynamicStorage>
(--shape, default 2,16,16,16,8), seeded
RNG throughout, branch points replayed
from scratch rather than snapshotted.
tests/cpp/test_ml_behaviour.cpp 20 asserting invariants, wired into
nisps_core_tests.
scripts/bench-ml.sh native + WASM from one source; --compare,
--sweep-shape, --smoke, --scenario, --seed.
Documents two contracts that fail SILENTLY (both now pinned by tests):
a thumbs-up must call BOTH mlp.add_example() and fb.store_positive(),
since dislike_geometric k-NNs the replay buffer and not the MLP dataset;
and placed_output() is valid only while state == Placing, after which an
empty span whose l2() is 0 scores a broken lifecycle as a perfect place.
ALIGNMENT defect 6 re-ranked (SGD-vs-RMSProp is not a research axis - it
silently invalidated every ported hyperparameter) and split into 6b (the
geometric dislike was ported from a superseded upstream design) and 6c
(InterfaceRL, the reference impl, is not in the tree).
Gates: build-cpp-tests (138 tests, ctest 4/4), parity-check PASS,
lint-cpp clean, bench-ml.sh --smoke runs end to end.
This commit is contained in:
monkey-w1n5t0n
parent
60584606a8
commit
1603ea798e
6 changed files with 3409 additions and 3 deletions
51
ALIGNMENT.md
51
ALIGNMENT.md
|
|
@ -63,11 +63,56 @@ command surface to report through.
|
||||||
**Rough cost.** Host half is done. On-device: ~a day, and it wants defect 3's serial protocol
|
**Rough cost.** Host half is done. On-device: ~a day, and it wants defect 3's serial protocol
|
||||||
to have somewhere to send the number.
|
to have somewhere to send the number.
|
||||||
|
|
||||||
### 6. RMSProp still deferred from `nisps/ml/` (2026-04-29; reaffirmed 2026-07-21)
|
### 6. SGD-vs-RMSProp is not a research axis — it silently invalidated every ported hyperparameter (2026-04-29; **re-ranked 2026-07-25**)
|
||||||
|
|
||||||
**What.** `training.hpp` ships SGD only; the legacy firmware used RMSProp for `TrainBatch`. Optimizer choice is a research axis. Not blocking current fits; will matter for harder loss landscapes. Port target: upstream MusicallyEmbodiedML `memlp` (the in-repo `src/memlp` copy is deleted; use the GitHub remote or archive branch).
|
**What.** `training.hpp` ships SGD only. Upstream `memlp` (pinned `ea777502` by
|
||||||
|
`upstream/main`) applies gradients with **RMSProp everywhere** — `Layer.h:239`, the
|
||||||
|
`m_sq_grad_avg` running squared-gradient average at `Layer.h:601`, `StaticMLP.h:268`
|
||||||
|
"Mini-batch RMSProp training". This was filed as an optimiser-choice research question
|
||||||
|
and ranked last. That was wrong, and the 2026-07-25 benchmark work shows why.
|
||||||
|
|
||||||
**Rough cost.** A day, plus batch-convergence tests.
|
**Why it blocks the mission.** Every learning rate ported from upstream landed in a
|
||||||
|
different optimiser than the one it was tuned for. RMSProp normalises each step by the
|
||||||
|
running gradient magnitude, so `lr=1e-3` there is a normalised step; under SGD it is
|
||||||
|
literally `1e-3 x raw gradient`. The two numbers are unrelated. Concretely:
|
||||||
|
`feedback.hpp:789` carries `geo_lr_ = 0.001f // upstream InterfaceRL.hpp:312` — an
|
||||||
|
RMSProp LR pasted into a single SGD step. `tests/cpp/ml_bench.cpp` D1 measures the
|
||||||
|
result: the geometric dislike aims at a target 0.5 output-units away and moves the
|
||||||
|
mapping by **5.1e-5**, linearly, so ~10,000 presses would be needed for one press's
|
||||||
|
intended effect. Meanwhile likes train at `lr 1.0 x 1000 iterations`, making a like
|
||||||
|
~2e6x stronger than a dislike and heaving the whole mapping on every press
|
||||||
|
(`ml_bench` U1/U4: `lurch_max` ~1.1 against a [0,1] output range).
|
||||||
|
|
||||||
|
**Rough cost.** A day for the port plus batch-convergence tests — but it must come
|
||||||
|
BEFORE any retuning of `geo_lr`, `kGeometricPushScale` or the neg-LR base, or those
|
||||||
|
constants get tuned twice.
|
||||||
|
|
||||||
|
### 6b. The geometric dislike was ported from a superseded upstream design (2026-07-25)
|
||||||
|
|
||||||
|
**What.** `geo_push.hpp`/`replay.hpp` cite `memllib @ 0a541cc`. `upstream/main` now pins
|
||||||
|
`e291192`, where the same code has been deliberately redesigned. Upstream:
|
||||||
|
`kGeometricPushScale` 1.0 (ours 0.5); neg-LR base 1.5 (ours 0.5, `geo_push.hpp:92`);
|
||||||
|
the `/(1+len)` taper **deleted**, with the comment "a 'no' should clearly move the
|
||||||
|
mapping away even from a sound already far from the liked region (the taper used to
|
||||||
|
kill exactly that case)" — ours still applies it at `geo_push.hpp:66`; negatives trained
|
||||||
|
as a **batch over ALL of them every tick** rather than one item one step; and a fixed
|
||||||
|
`kDislikeLifetimeMs = 2500` full-strength lifetime replacing the proportional decay we
|
||||||
|
ported. On the shared constants (dedup radius 0.05, `kCentroidK` 4) we match.
|
||||||
|
|
||||||
|
**Why it blocks the mission.** We are carrying a design upstream diagnosed and fixed,
|
||||||
|
and the fix is documented in their source comments. Compounded with defect 6 the ported
|
||||||
|
dislike is ~9.4x weaker on constants alone before the optimiser mismatch.
|
||||||
|
|
||||||
|
**Rough cost.** Small once defect 6 lands — mostly deleting the taper and re-basing
|
||||||
|
three constants, then re-running `scripts/bench-ml.sh` D1/A4/A7 to confirm.
|
||||||
|
|
||||||
|
### 6c. `InterfaceRL` — the reference implementation — is not in the tree (2026-07-25)
|
||||||
|
|
||||||
|
**What.** It lives in `memllib/examples/`, and the vendoring dropped `examples/`
|
||||||
|
(`VENDORED.md`). So the source of truth for our most contested subsystem is absent, and
|
||||||
|
the divergences in 6b went unnoticed for months. Either vendor
|
||||||
|
`examples/InterfaceRL.{hpp,cpp,tpp}` read-only alongside the rest, or record its pinned
|
||||||
|
commit and a fetch recipe in `VENDORED.md`.
|
||||||
|
|
||||||
## Open mission questions
|
## Open mission questions
|
||||||
|
|
||||||
|
|
|
||||||
4
MAP.md
4
MAP.md
|
|
@ -125,6 +125,8 @@ includes; no `nisps-core`.
|
||||||
### `tests/cpp/` — host C++ tests
|
### `tests/cpp/` — host C++ tests
|
||||||
- Per-component tests: `test_dsp_*.cpp`, `test_engine_*.cpp`, `test_mlp_*.cpp`, `test_mode_*.cpp`, `test_ring_buffer.cpp`, `test_rng.cpp`, `test_math.cpp`. Helpers in `test_helpers.hpp`.
|
- Per-component tests: `test_dsp_*.cpp`, `test_engine_*.cpp`, `test_mlp_*.cpp`, `test_mode_*.cpp`, `test_ring_buffer.cpp`, `test_rng.cpp`, `test_math.cpp`. Helpers in `test_helpers.hpp`.
|
||||||
- Verification: `ml_golden_vectors.cpp`, `engine_impulse.cpp` (+ `engine_impulse_baseline.bin`), `parity_check.cpp` + `parity_wasm.mjs` + `parity_diff.mjs` — native-vs-WASM bit-equivalence within 1e-5.
|
- Verification: `ml_golden_vectors.cpp`, `engine_impulse.cpp` (+ `engine_impulse_baseline.bin`), `parity_check.cpp` + `parity_wasm.mjs` + `parity_diff.mjs` — native-vs-WASM bit-equivalence within 1e-5.
|
||||||
|
- Behaviour: `test_ml_behaviour.cpp` — 20 INVARIANTS of the interaction model (not tuning). Asserts what must hold however the knobs are set: a like is reachable and a re-like overwrites; dislike never yields non-finite weights/outputs in either `AvoidStyle`, and is safe at cold start; repeat dislikes inside `kReplayDedupRadius` deepen ONE negative while distant ones store separately; explore→reroll→undo→exit restores weights *bit-exactly* and over-undoing is safe; `RandomiseOutputs` leaves weights untouched (the outputs-vs-weights randomisation distinction); same seed + same gesture sequence ⇒ bit-identical weights (what makes any behavioural benchmark comparable); example-ring overflow keeps the newest; contradictory examples stay finite; the ExploreAndPlace accessor contract (`placed_output()` while Placing, `committed_output()` after commit — for both place and reposition); a FULLY-masked focus gate freezes every weight; switching mode mid-exploration never strands the net in a randomised scratchpad; and all of it holds at 1×1, 2×8, 1×33, 8×2 and 32×8 shapes.
|
||||||
|
- Measurement (asserts nothing): `ml_bench.cpp` — the BEHAVIOURAL benchmark. NISPS is a controller, so this measures the shape of the control→parameter mapping and how interaction journeys deform it, never loss alone. Shape-agnostic (`--shape N_IN,H1,H2,H3,N_OUT`, default `2,16,16,16,8`) via `MLPCore<DynamicStorage>`, so "does a wider/deeper net change the UX?" becomes a number; sample points come from a deterministic Kronecker low-discrepancy sequence rather than a raster, which is what makes it work at any input arity. Field metrics: local gain p50/p95, cliff index, dead fraction, range utilisation, rail occupancy, effective dimensionality (participation ratio — trace²/‖C‖²_F, no eigendecomposition). Displacement metrics: at-point, rings, global, **blast ratio**, and collateral damage at the protected positives. **61 scenarios.** Diagnostic (D1 — the geometric-dislike dose decomposed: intended push vs effective LR vs measured movement at 1/10/100/1000 presses). Atomic probes A1–A14: at/around/far-from an example; one dislike under BOTH candidate designs; twice at one point; adjacent-then-return across the dedup radius; near a protected positive; roll-a-patch-and-place; the full explore→audition→place→commit lifecycle; explore-then-cancel; reposition; like-then-dislike in place; dislike-then-repair; focus/solo mask leakage. Journeys J1–J11: positive-only retention curve, randomise-place-only, mixed, branch (one shared prefix, three divergent gestures, replayed from scratch per branch because that is exact under a deterministic RNG), explore-place-only, 120-gesture long session with drift + weight-norm checkpoints, dislike storm, revisit-after-wandering, two-region interference, sweep-and-teach along a continuous path, undo-heavy. Edge cases E1–E13: cold start, single example, contradictory, collinear, corners, capacity overflow, undo exhaustion, minimal shape, identical targets, rail targets, rapid like/dislike alternation, mode-switch mid-exploration, fully-masked dislike. Upstream comparison U1–U3: the older memllib `interfaceRL` (a DDPG actor-critic, recoverable from this repo's own git history at blob `755ff8b`) differs structurally — the user HEARS `actorTarget`, a soft copy updated `target += alpha*(online-target)` at alpha=0.005, and it trains a batch of 4 from replay only every `optimiseDivisor=40` gestures. The critic half is not reproducible here (MLPCore has `train_targets` but not the per-layer gradient extraction the policy-gradient step needs), but both OUTPUT-PATH ideas are: U1 sweeps the soft-target alpha (alpha=1 IS NISPS today, a free control), U2 sweeps the train-every-Nth divisor, U3 compares actor shapes, U4 sweeps the positive-path training dose. **U4 carries a load-bearing caveat**: upstream LRs are RMSProp LRs (memlp `Layer.h:239`, `StaticMLP.h:268`) and `nisps/ml/training.hpp` is SGD-only, so the numbers are not comparable — see ALIGNMENT.md defect 6. Their shared metric is **lurch** — how far the mapping the musician is playing moves per single gesture, averaged over the whole field. Knobs: `--spread` (1 = Xavier, 0 = uniform with NO fan_in coupling — i.e. the post-removal behaviour, measurable before paying for the refactor), `--geo-lr`, `--geo-iters`. Driven by `scripts/bench-ml.sh`. **Two contracts that fail SILENTLY and are pinned by tests:** (1) a thumbs-up must go through BOTH `mlp.add_example` AND `fb.store_positive` — `dislike_geometric` k-NNs the replay buffer, not the MLP dataset, so a harness that only calls `add_example` measures the cold-start branch instead; (2) `placed_output()` is valid ONLY while state is `Placing` — after `commit_place()`/`commit_reposition()` the vector moves to `committed_output()`, and reading the wrong one yields an empty span whose `l2()` is 0, i.e. a broken lifecycle scored as a perfect placement.
|
||||||
- Measurement (asserts nothing): `engine_bench.cpp` + `bench_report.mjs` — per-engine throughput (ns/sample, blocks/s, realtime factor) for the `process()` hot path. ONE source compiled twice (CMake `nisps_engine_bench` natively, emcc for WASM) so the two targets are comparable without adding a single export to `nisps/wasm/bindings.cpp`. Engines are driven into a working state (transport running + event drain for the sequencers, periodic `note_on` for paf_synth, a noise+sine input bed for the fx/analysis engines) and every row prints its own working-state evidence, so a number produced by an idle engine is visible rather than plausible. Driven by `scripts/bench-engines.sh`.
|
- Measurement (asserts nothing): `engine_bench.cpp` + `bench_report.mjs` — per-engine throughput (ns/sample, blocks/s, realtime factor) for the `process()` hot path. ONE source compiled twice (CMake `nisps_engine_bench` natively, emcc for WASM) so the two targets are comparable without adding a single export to `nisps/wasm/bindings.cpp`. Engines are driven into a working state (transport running + event drain for the sequencers, periodic `note_on` for paf_synth, a noise+sine input bed for the fx/analysis engines) and every row prints its own working-state evidence, so a number produced by an idle engine is visible rather than plausible. Driven by `scripts/bench-engines.sh`.
|
||||||
|
|
||||||
### `scripts/` — build + verify entry points
|
### `scripts/` — build + verify entry points
|
||||||
|
|
@ -132,6 +134,7 @@ includes; no `nisps-core`.
|
||||||
- `build-wasm.sh` — Emscripten compile producing `manifold/public/nisps.{wasm,js}`.
|
- `build-wasm.sh` — Emscripten compile producing `manifold/public/nisps.{wasm,js}`.
|
||||||
- `build-cpp-tests.sh` — CMake configure + build + ctest (Ninja).
|
- `build-cpp-tests.sh` — CMake configure + build + ctest (Ninja).
|
||||||
- `parity-check.sh` — runs native + WASM and diffs binary outputs.
|
- `parity-check.sh` — runs native + WASM and diffs binary outputs.
|
||||||
|
- `bench-ml.sh` — the ML BEHAVIOUR benchmark on native + WASM from one source (same trick as `bench-engines.sh`). `--shape`, `--scenario`, `--smoke`, `--seed`, `--compare`, and `--sweep-shape` (runs the corpus across a ladder of architectures and arities — the knob-sensitivity instrument). **Reports, never asserts**: a cliff index is a description, not a pass/fail. Invariants live in `tests/cpp/test_ml_behaviour.cpp` instead. Reports land in `nisps/build/bench-ml/`.
|
||||||
- `bench-engines.sh` — engine throughput on native + WASM; `--compare <report.json>` prints per-engine Δ%. **Reports, never asserts** (a wall-clock threshold on shared hardware is meaningless or flaky — same call as the firmware size job). Reports land in `nisps/build/bench/`.
|
- `bench-engines.sh` — engine throughput on native + WASM; `--compare <report.json>` prints per-engine Δ%. **Reports, never asserts** (a wall-clock threshold on shared hardware is meaningless or flaky — same call as the firmware size job). Reports land in `nisps/build/bench/`.
|
||||||
- `lint-cpp.sh` — `.f` literal warn + heap/`Arduino.h` violation fail.
|
- `lint-cpp.sh` — `.f` literal warn + heap/`Arduino.h` violation fail.
|
||||||
- `run-all-tests.sh` — master verification script.
|
- `run-all-tests.sh` — master verification script.
|
||||||
|
|
@ -158,6 +161,7 @@ includes; no `nisps-core`.
|
||||||
- **Host C++ tests**: `bash scripts/build-cpp-tests.sh`.
|
- **Host C++ tests**: `bash scripts/build-cpp-tests.sh`.
|
||||||
- **Parity check**: `bash scripts/parity-check.sh`.
|
- **Parity check**: `bash scripts/parity-check.sh`.
|
||||||
- **Engine benchmark**: `bash scripts/bench-engines.sh` (add `--compare nisps/build/bench/latest.json` to diff against the previous run).
|
- **Engine benchmark**: `bash scripts/bench-engines.sh` (add `--compare nisps/build/bench/latest.json` to diff against the previous run).
|
||||||
|
- **ML behaviour benchmark**: `bash scripts/bench-ml.sh` (`--smoke` for a fast run, `--shape 2,16,16,16,8`, `--sweep-shape` for the architecture/arity sweep, `--compare` to diff).
|
||||||
- **All tests**: `bash scripts/run-all-tests.sh` (stage 6 is a bench smoke report; it does not gate).
|
- **All tests**: `bash scripts/run-all-tests.sh` (stage 6 is a bench smoke report; it does not gate).
|
||||||
- **Playwright**: `cd manifold && node node_modules/.bin/playwright test` (non-snap node runner on the VPS — BUILD-PLAN gotcha; `bunx playwright test` works elsewhere).
|
- **Playwright**: `cd manifold && node node_modules/.bin/playwright test` (non-snap node runner on the VPS — BUILD-PLAN gotcha; `bunx playwright test` works elsewhere).
|
||||||
- **Codegen**: `cd codegen && bun run generate.ts` (regenerates `nisps/modes/generated/` + `nisps/ml/generated/` C++ and `manifold/src/modes/generated/` TS).
|
- **Codegen**: `cd codegen && bun run generate.ts` (regenerates `nisps/modes/generated/` + `nisps/ml/generated/` C++ and `manifold/src/modes/generated/` TS).
|
||||||
|
|
|
||||||
|
|
@ -61,6 +61,7 @@ if(NOT EMSCRIPTEN)
|
||||||
${NISPS_TEST_DIR}/test_mlp_ou_noise.cpp
|
${NISPS_TEST_DIR}/test_mlp_ou_noise.cpp
|
||||||
${NISPS_TEST_DIR}/test_mlp_feedback.cpp
|
${NISPS_TEST_DIR}/test_mlp_feedback.cpp
|
||||||
${NISPS_TEST_DIR}/test_mlp_geo_dislike.cpp
|
${NISPS_TEST_DIR}/test_mlp_geo_dislike.cpp
|
||||||
|
${NISPS_TEST_DIR}/test_ml_behaviour.cpp
|
||||||
${NISPS_TEST_DIR}/test_mlp_serialize.cpp
|
${NISPS_TEST_DIR}/test_mlp_serialize.cpp
|
||||||
${NISPS_TEST_DIR}/test_pipeline.cpp
|
${NISPS_TEST_DIR}/test_pipeline.cpp
|
||||||
${NISPS_TEST_DIR}/test_vcv_iml_parity.cpp
|
${NISPS_TEST_DIR}/test_vcv_iml_parity.cpp
|
||||||
|
|
@ -189,6 +190,29 @@ if(NOT EMSCRIPTEN)
|
||||||
target_compile_options(nisps_engine_bench PRIVATE /W4 /WX)
|
target_compile_options(nisps_engine_bench PRIVATE /W4 /WX)
|
||||||
endif()
|
endif()
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------
|
||||||
|
# Standalone ML BEHAVIOUR benchmark. Same discipline as nisps_engine_bench:
|
||||||
|
# it reports numbers and asserts nothing, so it is not a ctest gate. It
|
||||||
|
# measures the shape of the control mapping and how interaction journeys
|
||||||
|
# deform it — see MAP.md § `tests/cpp/`. Driven by
|
||||||
|
# scripts/bench-ml.sh, which compiles the SAME source to WASM via emcc.
|
||||||
|
#
|
||||||
|
# Uses MLPCore<DynamicStorage>, so it is host-only by construction
|
||||||
|
# (dynamic_storage.hpp #errors on RP2350). That is correct: the point is
|
||||||
|
# arbitrary runtime shapes, which firmware deliberately cannot have.
|
||||||
|
# ---------------------------------------------------------------------
|
||||||
|
add_executable(nisps_ml_bench
|
||||||
|
${NISPS_TEST_DIR}/ml_bench.cpp
|
||||||
|
)
|
||||||
|
target_link_libraries(nisps_ml_bench PRIVATE nisps_core)
|
||||||
|
if(CMAKE_CXX_COMPILER_ID MATCHES "GNU|Clang|AppleClang")
|
||||||
|
target_compile_options(nisps_ml_bench PRIVATE
|
||||||
|
-Wall -Wextra -Werror -Wpedantic -O3
|
||||||
|
)
|
||||||
|
elseif(MSVC)
|
||||||
|
target_compile_options(nisps_ml_bench PRIVATE /W4 /WX)
|
||||||
|
endif()
|
||||||
|
|
||||||
# Standalone parity-check runner. NOT registered with ctest — it's
|
# Standalone parity-check runner. NOT registered with ctest — it's
|
||||||
# invoked from scripts/parity-check.sh which orchestrates native+WASM
|
# invoked from scripts/parity-check.sh which orchestrates native+WASM
|
||||||
# together.
|
# together.
|
||||||
|
|
|
||||||
180
scripts/bench-ml.sh
Executable file
180
scripts/bench-ml.sh
Executable file
|
|
@ -0,0 +1,180 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
# scripts/bench-ml.sh — behavioural benchmark for the NISPS control mapping.
|
||||||
|
#
|
||||||
|
# Sibling to bench-engines.sh, and deliberately the same shape: it builds
|
||||||
|
# tests/cpp/ml_bench.cpp twice from ONE source (native via CMake, WASM via
|
||||||
|
# emcc with the flags build-wasm.sh uses), runs both, and reports.
|
||||||
|
#
|
||||||
|
# NOTHING HERE FAILS. Behaviour is not a threshold — a "cliff index of 4.9" is
|
||||||
|
# neither pass nor fail, it is a description. Regressions are noticed by
|
||||||
|
# running with --compare against a previous report, which prints per-metric Δ.
|
||||||
|
# Invariants that genuinely must hold are ctest assertions in
|
||||||
|
# tests/cpp/test_ml_behaviour.cpp, not here.
|
||||||
|
#
|
||||||
|
# WHY BOTH TARGETS
|
||||||
|
# ----------------
|
||||||
|
# Firmware and browser must feel the same. Native-vs-WASM here is a BEHAVIOURAL
|
||||||
|
# comparison, which parity-check.sh's 1e-5 bit-equivalence does not give you:
|
||||||
|
# parity proves two builds of the SAME commit agree, and says nothing about
|
||||||
|
# whether a mapping is playable or whether a gesture does anything.
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# scripts/bench-ml.sh # native + wasm
|
||||||
|
# scripts/bench-ml.sh --native-only
|
||||||
|
# scripts/bench-ml.sh --smoke # fast, proves it runs
|
||||||
|
# scripts/bench-ml.sh --shape 2,16,16,16,8 # any net shape
|
||||||
|
# scripts/bench-ml.sh --sweep-shape # the architecture sweep
|
||||||
|
# scripts/bench-ml.sh --scenario A4_negative_once
|
||||||
|
# scripts/bench-ml.sh --compare old.json
|
||||||
|
# scripts/bench-ml.sh --out bench-ml-2026-07-25.json
|
||||||
|
#
|
||||||
|
# Env:
|
||||||
|
# NISPS_BUILD_DIR default nisps/build
|
||||||
|
# EMCC emcc path (same convention as build-wasm.sh)
|
||||||
|
#
|
||||||
|
# Exit codes: 0 on a completed run, 2 on bad args/missing artifacts, 3 on build
|
||||||
|
# failure.
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||||
|
BUILD_DIR="${NISPS_BUILD_DIR:-$ROOT/nisps/build}"
|
||||||
|
BENCH_DIR="$BUILD_DIR/bench-ml"
|
||||||
|
NATIVE_BIN="$BUILD_DIR/nisps_ml_bench"
|
||||||
|
SRC="$ROOT/tests/cpp/ml_bench.cpp"
|
||||||
|
REPORT="$ROOT/tests/cpp/ml_bench_report.mjs"
|
||||||
|
EMCC="${EMCC:-emcc}"
|
||||||
|
|
||||||
|
NATIVE_ONLY=0
|
||||||
|
SMOKE=""
|
||||||
|
SHAPE=""
|
||||||
|
SCENARIO=""
|
||||||
|
COMPARE=""
|
||||||
|
OUT=""
|
||||||
|
SWEEP_SHAPE=0
|
||||||
|
SEED=""
|
||||||
|
|
||||||
|
while [[ $# -gt 0 ]]; do
|
||||||
|
case "$1" in
|
||||||
|
--native-only) NATIVE_ONLY=1; shift ;;
|
||||||
|
--smoke) SMOKE="--smoke"; shift ;;
|
||||||
|
--shape) SHAPE="$2"; shift 2 ;;
|
||||||
|
--seed) SEED="$2"; shift 2 ;;
|
||||||
|
--scenario) SCENARIO="$2"; shift 2 ;;
|
||||||
|
--compare) COMPARE="$2"; shift 2 ;;
|
||||||
|
--out) OUT="$2"; shift 2 ;;
|
||||||
|
--sweep-shape) SWEEP_SHAPE=1; shift ;;
|
||||||
|
-h|--help) sed -n '2,40p' "$0"; exit 0 ;;
|
||||||
|
*) echo "bench-ml: unknown arg '$1'" >&2; exit 2 ;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
|
||||||
|
mkdir -p "$BENCH_DIR"
|
||||||
|
|
||||||
|
bench_args() {
|
||||||
|
local a=()
|
||||||
|
[[ -n "$SMOKE" ]] && a+=("$SMOKE")
|
||||||
|
[[ -n "$SHAPE" ]] && a+=(--shape "$SHAPE")
|
||||||
|
[[ -n "$SEED" ]] && a+=(--seed "$SEED")
|
||||||
|
[[ -n "$SCENARIO" ]] && a+=(--scenario "$SCENARIO")
|
||||||
|
printf '%s\n' "${a[@]:-}"
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# native
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
echo "==> building native ml_bench" >&2
|
||||||
|
if ! cmake -S "$ROOT/nisps" -B "$BUILD_DIR" -G Ninja -DCMAKE_BUILD_TYPE=Release >/dev/null 2>&1; then
|
||||||
|
echo "bench-ml: cmake configure failed" >&2; exit 3
|
||||||
|
fi
|
||||||
|
if ! cmake --build "$BUILD_DIR" --target nisps_ml_bench >/dev/null 2>&1; then
|
||||||
|
echo "bench-ml: native build failed" >&2; exit 3
|
||||||
|
fi
|
||||||
|
|
||||||
|
mapfile -t ARGS < <(bench_args)
|
||||||
|
NATIVE_JSON="$BENCH_DIR/native.json"
|
||||||
|
if [[ ${#ARGS[@]} -gt 0 && -n "${ARGS[0]}" ]]; then
|
||||||
|
"$NATIVE_BIN" "${ARGS[@]}" > "$NATIVE_JSON"
|
||||||
|
else
|
||||||
|
"$NATIVE_BIN" > "$NATIVE_JSON"
|
||||||
|
fi
|
||||||
|
echo "==> native report: $NATIVE_JSON" >&2
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# wasm — one source, second compiler. Same flags build-wasm.sh uses for the
|
||||||
|
# shipped module, so the comparison is against what actually ships.
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
WASM_JSON=""
|
||||||
|
if [[ $NATIVE_ONLY -eq 0 ]]; then
|
||||||
|
if ! command -v "$EMCC" >/dev/null 2>&1; then
|
||||||
|
echo "bench-ml: emcc not found; skipping wasm (use --native-only to silence)" >&2
|
||||||
|
else
|
||||||
|
echo "==> building wasm ml_bench" >&2
|
||||||
|
WASM_JS="$BENCH_DIR/ml_bench.mjs"
|
||||||
|
if ! "$EMCC" "$SRC" -o "$WASM_JS" \
|
||||||
|
-std=gnu++20 -O3 \
|
||||||
|
-s ENVIRONMENT=node -s EXIT_RUNTIME=1 -s ALLOW_MEMORY_GROWTH=1 \
|
||||||
|
-s SINGLE_FILE=1 >/dev/null 2>&1; then
|
||||||
|
echo "bench-ml: wasm build failed" >&2; exit 3
|
||||||
|
fi
|
||||||
|
WASM_JSON="$BENCH_DIR/wasm.json"
|
||||||
|
if [[ ${#ARGS[@]} -gt 0 && -n "${ARGS[0]}" ]]; then
|
||||||
|
node "$WASM_JS" "${ARGS[@]}" > "$WASM_JSON"
|
||||||
|
else
|
||||||
|
node "$WASM_JS" > "$WASM_JSON"
|
||||||
|
fi
|
||||||
|
echo "==> wasm report: $WASM_JSON" >&2
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# architecture sweep — the "how does net shape change the UX" instrument.
|
||||||
|
# Runs the corpus at a ladder of shapes and emits one combined report.
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
if [[ $SWEEP_SHAPE -eq 1 ]]; then
|
||||||
|
SWEEP_JSON="$BENCH_DIR/sweep-shape.json"
|
||||||
|
echo "==> shape sweep" >&2
|
||||||
|
{
|
||||||
|
echo "["
|
||||||
|
first=1
|
||||||
|
for s in \
|
||||||
|
"2,4,4,4,8" \
|
||||||
|
"2,8,8,8,8" \
|
||||||
|
"2,16,16,16,8" \
|
||||||
|
"2,32,32,32,8" \
|
||||||
|
"2,64,64,64,8" \
|
||||||
|
"1,16,16,16,8" \
|
||||||
|
"4,16,16,16,8" \
|
||||||
|
"8,16,16,16,8" \
|
||||||
|
"32,16,16,16,8" \
|
||||||
|
"2,16,16,16,1" \
|
||||||
|
"2,16,16,16,4" \
|
||||||
|
"2,16,16,16,16" \
|
||||||
|
"2,16,16,16,33"
|
||||||
|
do
|
||||||
|
[[ $first -eq 0 ]] && echo ","
|
||||||
|
first=0
|
||||||
|
"$NATIVE_BIN" --shape "$s" ${SMOKE:+$SMOKE}
|
||||||
|
done
|
||||||
|
echo "]"
|
||||||
|
} > "$SWEEP_JSON"
|
||||||
|
echo "==> sweep report: $SWEEP_JSON" >&2
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# format + diff
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
if [[ -f "$REPORT" ]] && command -v node >/dev/null 2>&1; then
|
||||||
|
if [[ -n "$COMPARE" ]]; then
|
||||||
|
node "$REPORT" "$NATIVE_JSON" ${WASM_JSON:+"$WASM_JSON"} --compare "$COMPARE"
|
||||||
|
else
|
||||||
|
node "$REPORT" "$NATIVE_JSON" ${WASM_JSON:+"$WASM_JSON"}
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
echo "(no formatter; raw JSON is in $BENCH_DIR)" >&2
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ -n "$OUT" ]]; then
|
||||||
|
command cp -f "$NATIVE_JSON" "$OUT"
|
||||||
|
echo "==> saved $OUT" >&2
|
||||||
|
fi
|
||||||
2602
tests/cpp/ml_bench.cpp
Normal file
2602
tests/cpp/ml_bench.cpp
Normal file
File diff suppressed because it is too large
Load diff
551
tests/cpp/test_ml_behaviour.cpp
Normal file
551
tests/cpp/test_ml_behaviour.cpp
Normal file
|
|
@ -0,0 +1,551 @@
|
||||||
|
// tests/cpp/test_ml_behaviour.cpp — INVARIANTS of the interaction model.
|
||||||
|
//
|
||||||
|
// Split of responsibility with tests/cpp/ml_bench.cpp:
|
||||||
|
// ml_bench.cpp REPORTS behaviour (how local is a dislike? how playable is
|
||||||
|
// the mapping?). Those are descriptions, not pass/fail, so it
|
||||||
|
// asserts nothing and is not a ctest gate.
|
||||||
|
// this file ASSERTS the things that must be TRUE regardless of tuning.
|
||||||
|
// If one of these breaks, the interaction model is broken, not
|
||||||
|
// merely different — a musician would call it a bug.
|
||||||
|
//
|
||||||
|
// Every test here is shape-parameterised where it can be, and runs at the
|
||||||
|
// operator's default 2-in/8-out as well as at degenerate shapes, because the
|
||||||
|
// edge cases are exactly where an interaction model quietly stops holding.
|
||||||
|
//
|
||||||
|
// These use MLPCore<DynamicStorage>, which is host-only by construction
|
||||||
|
// (dynamic_storage.hpp #errors on RP2350). That is fine: these are behavioural
|
||||||
|
// invariants of the shared algorithms, and the fixed-storage firmware path is
|
||||||
|
// bit-pinned to the dynamic one by test_mlp_storage_parity.cpp.
|
||||||
|
|
||||||
|
#include <cmath>
|
||||||
|
#include <span>
|
||||||
|
#include <vector>
|
||||||
|
|
||||||
|
#include "../../nisps/ml/dynamic_storage.hpp"
|
||||||
|
#include "../../nisps/ml/feedback.hpp"
|
||||||
|
#include "../../nisps/ml/mlp.hpp"
|
||||||
|
#include "test_helpers.hpp"
|
||||||
|
|
||||||
|
namespace {
|
||||||
|
|
||||||
|
using nisps::ml::AvoidStyle;
|
||||||
|
using nisps::ml::DynamicFeedbackStorage;
|
||||||
|
using nisps::ml::DynamicStorage;
|
||||||
|
using nisps::ml::FeedbackControllerCore;
|
||||||
|
using nisps::ml::FeedbackMode;
|
||||||
|
using nisps::ml::MLPCore;
|
||||||
|
|
||||||
|
using Mlp = MLPCore<DynamicStorage>;
|
||||||
|
using Feedback = FeedbackControllerCore<DynamicFeedbackStorage>;
|
||||||
|
|
||||||
|
constexpr std::uint64_t kSeed = 0xC0FFEEu;
|
||||||
|
|
||||||
|
struct Rig {
|
||||||
|
std::size_t n_in, n_out;
|
||||||
|
Mlp mlp;
|
||||||
|
Feedback fb;
|
||||||
|
|
||||||
|
Rig(std::size_t nin, std::size_t nout, std::size_t h = 16u,
|
||||||
|
std::size_t max_examples = 128u)
|
||||||
|
: n_in(nin), n_out(nout),
|
||||||
|
mlp(kSeed, nin, std::span<const std::size_t>(hidden_(h), 3u), nout,
|
||||||
|
max_examples, 4096u),
|
||||||
|
fb(kSeed ^ 0xF33Dull, nout, mlp.weight_count(), 4u, nin, 64u) {}
|
||||||
|
|
||||||
|
// Static storage for the hidden dims so the span outlives the ctor call.
|
||||||
|
static const std::size_t* hidden_(std::size_t h) {
|
||||||
|
static std::size_t buf[3];
|
||||||
|
buf[0] = buf[1] = buf[2] = h;
|
||||||
|
return buf;
|
||||||
|
}
|
||||||
|
|
||||||
|
void at(std::span<const float> x, std::vector<float>& out) {
|
||||||
|
out.assign(n_out, 0.f);
|
||||||
|
for (std::size_t i = 0; i < n_in; ++i) mlp.set_input(i, x[i]);
|
||||||
|
mlp.process();
|
||||||
|
auto o = mlp.outputs();
|
||||||
|
for (std::size_t j = 0; j < n_out; ++j) out[j] = o[j];
|
||||||
|
}
|
||||||
|
|
||||||
|
// The product path for a thumbs-up: dataset example AND replay positive.
|
||||||
|
void like(std::span<const float> x, std::span<const float> y) {
|
||||||
|
for (std::size_t i = 0; i < n_in; ++i) mlp.set_input(i, x[i]);
|
||||||
|
mlp.process();
|
||||||
|
mlp.add_example(x, y);
|
||||||
|
fb.store_positive(mlp, y);
|
||||||
|
}
|
||||||
|
|
||||||
|
void down(std::span<const float> x, float speed = 0.1f, float spread = 1.f) {
|
||||||
|
std::vector<float> heard;
|
||||||
|
at(x, heard);
|
||||||
|
for (std::size_t i = 0; i < n_in; ++i) mlp.set_input(i, x[i]);
|
||||||
|
mlp.process();
|
||||||
|
fb.on_down(mlp, heard, speed, spread, {});
|
||||||
|
}
|
||||||
|
|
||||||
|
// get_weights() returns a span into a storage-owned scratch buffer, so it
|
||||||
|
// must be COPIED before the next call reuses that buffer.
|
||||||
|
void weights(std::vector<float>& w) {
|
||||||
|
auto s = mlp.get_weights();
|
||||||
|
w.assign(s.begin(), s.end());
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
float l2(std::span<const float> a, std::span<const float> b) {
|
||||||
|
float acc = 0.f;
|
||||||
|
const std::size_t n = a.size() < b.size() ? a.size() : b.size();
|
||||||
|
for (std::size_t i = 0; i < n; ++i) { const float d = a[i] - b[i]; acc += d * d; }
|
||||||
|
return std::sqrt(acc);
|
||||||
|
}
|
||||||
|
|
||||||
|
bool all_finite(std::span<const float> v) {
|
||||||
|
for (float x : v) if (!std::isfinite(x)) return false;
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
} // namespace
|
||||||
|
|
||||||
|
// ===========================================================================
|
||||||
|
// PLACEMENT — a thumbs-up must actually take.
|
||||||
|
// ===========================================================================
|
||||||
|
|
||||||
|
// The single most basic promise the instrument makes: if you stand somewhere,
|
||||||
|
// like a set of outputs, and then stand in the SAME place again, you get
|
||||||
|
// approximately what you liked. Without this nothing else means anything.
|
||||||
|
NISPS_TEST(behaviour_single_like_is_reachable) {
|
||||||
|
Rig rig(2u, 8u);
|
||||||
|
const float x[2] = {0.25f, -0.4f};
|
||||||
|
std::vector<float> y(8u);
|
||||||
|
for (std::size_t j = 0; j < 8u; ++j) y[j] = 0.2f + 0.07f * static_cast<float>(j);
|
||||||
|
|
||||||
|
rig.like(x, y);
|
||||||
|
rig.mlp.train();
|
||||||
|
|
||||||
|
std::vector<float> got;
|
||||||
|
rig.at(x, got);
|
||||||
|
NISPS_EXPECT(all_finite(got));
|
||||||
|
// Loose bound on purpose: this asserts "the placement took", not "the
|
||||||
|
// optimiser is good". Tightening it would make it a tuning test.
|
||||||
|
NISPS_EXPECT(l2(got, y) < 0.25f);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Placing at the same position twice with a NEW target must move toward the
|
||||||
|
// new one — the musician overwrote their earlier choice and expects that to win.
|
||||||
|
NISPS_TEST(behaviour_relike_same_spot_overwrites) {
|
||||||
|
Rig rig(2u, 8u);
|
||||||
|
const float x[2] = {0.1f, 0.1f};
|
||||||
|
std::vector<float> y1(8u, 0.2f), y2(8u, 0.8f), got;
|
||||||
|
|
||||||
|
rig.like(x, y1);
|
||||||
|
rig.mlp.train();
|
||||||
|
rig.at(x, got);
|
||||||
|
const float err_to_y2_before = l2(got, y2);
|
||||||
|
|
||||||
|
rig.like(x, y2);
|
||||||
|
rig.mlp.train();
|
||||||
|
rig.at(x, got);
|
||||||
|
const float err_to_y2_after = l2(got, y2);
|
||||||
|
|
||||||
|
NISPS_EXPECT(err_to_y2_after < err_to_y2_before);
|
||||||
|
}
|
||||||
|
|
||||||
|
// ===========================================================================
|
||||||
|
// FEEDBACK — the negative path must be well-behaved even when it is gentle.
|
||||||
|
// ===========================================================================
|
||||||
|
|
||||||
|
// A dislike must never produce non-finite outputs. This is the one thing that
|
||||||
|
// turns "the instrument feels wrong" into "the instrument is dead", and a
|
||||||
|
// perturbation path plus a sigmoid is exactly where NaN would come from.
|
||||||
|
NISPS_TEST(behaviour_dislike_never_produces_nonfinite) {
|
||||||
|
for (AvoidStyle style : {AvoidStyle::Geometric, AvoidStyle::Diffuse}) {
|
||||||
|
Rig rig(2u, 8u);
|
||||||
|
std::vector<float> y(8u, 0.5f), got;
|
||||||
|
const float a[2] = {0.3f, 0.3f};
|
||||||
|
const float b[2] = {-0.3f, 0.2f};
|
||||||
|
rig.like(a, y);
|
||||||
|
rig.like(b, y);
|
||||||
|
rig.mlp.train();
|
||||||
|
|
||||||
|
rig.fb.set_mode(FeedbackMode::Avoid, rig.mlp);
|
||||||
|
rig.fb.set_avoid_style(style);
|
||||||
|
const float press[2] = {0.f, 0.25f};
|
||||||
|
for (int i = 0; i < 32; ++i) rig.down(press);
|
||||||
|
|
||||||
|
rig.at(press, got);
|
||||||
|
NISPS_EXPECT(all_finite(got));
|
||||||
|
std::vector<float> w;
|
||||||
|
rig.weights(w);
|
||||||
|
NISPS_EXPECT(all_finite(w));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Dislike at cold start (no positives at all) must not crash or corrupt. This
|
||||||
|
// is literally the first thing a new user might press.
|
||||||
|
NISPS_TEST(behaviour_dislike_cold_start_is_safe) {
|
||||||
|
Rig rig(2u, 8u);
|
||||||
|
rig.fb.set_mode(FeedbackMode::Avoid, rig.mlp);
|
||||||
|
rig.fb.set_avoid_style(AvoidStyle::Geometric);
|
||||||
|
const float press[2] = {0.f, 0.f};
|
||||||
|
|
||||||
|
NISPS_ASSERT(rig.fb.positive_count() == 0u);
|
||||||
|
for (int i = 0; i < 8; ++i) rig.down(press);
|
||||||
|
|
||||||
|
std::vector<float> got;
|
||||||
|
rig.at(press, got);
|
||||||
|
NISPS_EXPECT(all_finite(got));
|
||||||
|
NISPS_EXPECT(rig.fb.negative_count() > 0u);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Repeated dislikes at the same point must deepen ONE replay entry rather than
|
||||||
|
// filling the buffer — replay.hpp's dedup radius (0.05) exists for exactly
|
||||||
|
// this, and if it regresses a user holding the button evicts their own history.
|
||||||
|
NISPS_TEST(behaviour_repeat_dislike_dedups_not_accumulates) {
|
||||||
|
Rig rig(2u, 8u);
|
||||||
|
std::vector<float> y(8u, 0.5f);
|
||||||
|
const float a[2] = {0.4f, 0.4f};
|
||||||
|
rig.like(a, y);
|
||||||
|
rig.mlp.train();
|
||||||
|
|
||||||
|
rig.fb.set_mode(FeedbackMode::Avoid, rig.mlp);
|
||||||
|
rig.fb.set_avoid_style(AvoidStyle::Geometric);
|
||||||
|
const float press[2] = {-0.2f, 0.1f};
|
||||||
|
for (int i = 0; i < 20; ++i) rig.down(press);
|
||||||
|
|
||||||
|
// 20 presses inside the dedup radius: one negative, deepened.
|
||||||
|
NISPS_EXPECT(rig.fb.negative_count() == 1u);
|
||||||
|
}
|
||||||
|
|
||||||
|
// A dislike far outside the dedup radius must create a SEPARATE negative.
|
||||||
|
// Together with the test above this pins the "move a little to the left"
|
||||||
|
// behaviour the operator asked about: inside 0.05 deepens, outside stores.
|
||||||
|
NISPS_TEST(behaviour_distant_dislike_stores_separately) {
|
||||||
|
Rig rig(2u, 8u);
|
||||||
|
std::vector<float> y(8u, 0.5f);
|
||||||
|
const float a[2] = {0.4f, 0.4f};
|
||||||
|
rig.like(a, y);
|
||||||
|
rig.mlp.train();
|
||||||
|
|
||||||
|
rig.fb.set_mode(FeedbackMode::Avoid, rig.mlp);
|
||||||
|
rig.fb.set_avoid_style(AvoidStyle::Geometric);
|
||||||
|
const float p1[2] = {-0.5f, 0.f};
|
||||||
|
const float p2[2] = {0.5f, 0.f}; // well beyond kReplayDedupRadius
|
||||||
|
rig.down(p1);
|
||||||
|
rig.down(p2);
|
||||||
|
NISPS_EXPECT(rig.fb.negative_count() == 2u);
|
||||||
|
}
|
||||||
|
|
||||||
|
// ===========================================================================
|
||||||
|
// EXPLORE / UNDO — the scratchpad must be exactly reversible.
|
||||||
|
// ===========================================================================
|
||||||
|
|
||||||
|
// Enter explore, roll, then undo everything and exit: the net must return to
|
||||||
|
// EXACTLY where it started. Not approximately — the whole point of the
|
||||||
|
// scratchpad model is that auditioning costs nothing.
|
||||||
|
NISPS_TEST(behaviour_explore_undo_restores_exactly) {
|
||||||
|
Rig rig(2u, 8u);
|
||||||
|
std::vector<float> y(8u, 0.5f);
|
||||||
|
const float a[2] = {0.2f, -0.2f};
|
||||||
|
rig.like(a, y);
|
||||||
|
rig.mlp.train();
|
||||||
|
|
||||||
|
std::vector<float> before, after;
|
||||||
|
rig.weights(before);
|
||||||
|
|
||||||
|
rig.fb.set_mode(FeedbackMode::ExploreAndPlace, rig.mlp);
|
||||||
|
rig.fb.enter_explore(rig.mlp, 1.f);
|
||||||
|
for (int i = 0; i < 3; ++i) rig.fb.reroll(rig.mlp, 1.f);
|
||||||
|
for (int i = 0; i < 3; ++i) rig.fb.undo(rig.mlp);
|
||||||
|
rig.fb.exit_explore(rig.mlp);
|
||||||
|
|
||||||
|
rig.weights(after);
|
||||||
|
bool identical = true;
|
||||||
|
for (std::size_t i = 0; i < before.size(); ++i) {
|
||||||
|
if (before[i] != after[i]) { identical = false; break; }
|
||||||
|
}
|
||||||
|
NISPS_EXPECT(identical);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Undoing more times than the ring holds must not corrupt anything. The user
|
||||||
|
// mashing undo is not a bug report.
|
||||||
|
NISPS_TEST(behaviour_undo_past_ring_depth_is_safe) {
|
||||||
|
Rig rig(2u, 8u);
|
||||||
|
std::vector<float> y(8u, 0.5f);
|
||||||
|
const float a[2] = {0.f, 0.f};
|
||||||
|
rig.like(a, y);
|
||||||
|
rig.mlp.train();
|
||||||
|
|
||||||
|
rig.fb.set_mode(FeedbackMode::ExploreAndPlace, rig.mlp);
|
||||||
|
rig.fb.enter_explore(rig.mlp, 1.f);
|
||||||
|
for (int i = 0; i < 3; ++i) rig.fb.reroll(rig.mlp, 1.f);
|
||||||
|
for (int i = 0; i < 40; ++i) rig.fb.undo(rig.mlp); // way past depth 4
|
||||||
|
rig.fb.exit_explore(rig.mlp);
|
||||||
|
|
||||||
|
std::vector<float> w, got;
|
||||||
|
rig.weights(w);
|
||||||
|
NISPS_EXPECT(all_finite(w));
|
||||||
|
rig.at(a, got);
|
||||||
|
NISPS_EXPECT(all_finite(got));
|
||||||
|
}
|
||||||
|
|
||||||
|
// Output randomisation must NOT touch weights. This is the distinction the
|
||||||
|
// operator drew explicitly: RandomiseOutputs rolls a patch, RandomiseMlp
|
||||||
|
// scrambles the net. If the former ever moved weights, auditioning a patch
|
||||||
|
// would silently damage the instrument.
|
||||||
|
NISPS_TEST(behaviour_randomise_outputs_leaves_weights_untouched) {
|
||||||
|
Rig rig(2u, 8u);
|
||||||
|
std::vector<float> y(8u, 0.5f);
|
||||||
|
const float a[2] = {0.15f, 0.15f};
|
||||||
|
rig.like(a, y);
|
||||||
|
rig.mlp.train();
|
||||||
|
|
||||||
|
std::vector<float> before, after;
|
||||||
|
rig.weights(before);
|
||||||
|
|
||||||
|
rig.fb.set_mode(FeedbackMode::RandomiseOutputs, rig.mlp);
|
||||||
|
const float press[2] = {-0.1f, 0.3f};
|
||||||
|
for (int i = 0; i < 5; ++i) rig.down(press); // enter + 4 re-rolls
|
||||||
|
|
||||||
|
rig.weights(after);
|
||||||
|
bool identical = true;
|
||||||
|
for (std::size_t i = 0; i < before.size(); ++i) {
|
||||||
|
if (before[i] != after[i]) { identical = false; break; }
|
||||||
|
}
|
||||||
|
NISPS_EXPECT(identical);
|
||||||
|
|
||||||
|
// ...and it must actually be holding a patch to audition.
|
||||||
|
std::vector<float> patch(8u, 0.f);
|
||||||
|
NISPS_EXPECT(rig.fb.static_output(patch));
|
||||||
|
NISPS_EXPECT(all_finite(patch));
|
||||||
|
}
|
||||||
|
|
||||||
|
// ===========================================================================
|
||||||
|
// DETERMINISM — the benchmark is worthless if the engine is not reproducible.
|
||||||
|
// ===========================================================================
|
||||||
|
|
||||||
|
// Two rigs built with the same seed, driven through the same gesture sequence,
|
||||||
|
// must end bit-identical. This is what makes a behavioural benchmark
|
||||||
|
// comparable across commits at all.
|
||||||
|
NISPS_TEST(behaviour_same_seed_same_journey_is_bit_identical) {
|
||||||
|
auto journey = [](Rig& rig) {
|
||||||
|
std::vector<float> y(8u);
|
||||||
|
for (std::size_t j = 0; j < 8u; ++j) y[j] = 0.3f + 0.05f * static_cast<float>(j);
|
||||||
|
const float a[2] = {0.2f, 0.1f};
|
||||||
|
const float b[2] = {-0.3f, 0.4f};
|
||||||
|
rig.like(a, y);
|
||||||
|
rig.like(b, y);
|
||||||
|
rig.mlp.train();
|
||||||
|
rig.fb.set_mode(FeedbackMode::Avoid, rig.mlp);
|
||||||
|
rig.fb.set_avoid_style(AvoidStyle::Diffuse);
|
||||||
|
const float press[2] = {0.f, 0.f};
|
||||||
|
for (int i = 0; i < 5; ++i) rig.down(press);
|
||||||
|
};
|
||||||
|
|
||||||
|
Rig r1(2u, 8u), r2(2u, 8u);
|
||||||
|
journey(r1);
|
||||||
|
journey(r2);
|
||||||
|
|
||||||
|
std::vector<float> w1, w2;
|
||||||
|
r1.weights(w1);
|
||||||
|
r2.weights(w2);
|
||||||
|
NISPS_ASSERT(w1.size() == w2.size());
|
||||||
|
bool identical = true;
|
||||||
|
for (std::size_t i = 0; i < w1.size(); ++i) {
|
||||||
|
if (w1[i] != w2[i]) { identical = false; break; }
|
||||||
|
}
|
||||||
|
NISPS_EXPECT(identical);
|
||||||
|
}
|
||||||
|
|
||||||
|
// ===========================================================================
|
||||||
|
// CAPACITY + DEGENERATE SHAPES
|
||||||
|
// ===========================================================================
|
||||||
|
|
||||||
|
// Overflowing the example ring must not corrupt state — it evicts, silently,
|
||||||
|
// and the newest examples must still be the ones honoured.
|
||||||
|
NISPS_TEST(behaviour_example_overflow_keeps_newest) {
|
||||||
|
const std::size_t kCap = 8u;
|
||||||
|
Rig rig(2u, 8u, 16u, kCap);
|
||||||
|
std::vector<float> got;
|
||||||
|
|
||||||
|
// 3x the cap. Each example has a distinguishable target.
|
||||||
|
const std::size_t N = kCap * 3u;
|
||||||
|
std::vector<float> xs(N * 2u), ys(N * 8u);
|
||||||
|
for (std::size_t i = 0; i < N; ++i) {
|
||||||
|
const float t = static_cast<float>(i) / static_cast<float>(N);
|
||||||
|
xs[i * 2u] = 2.f * t - 1.f;
|
||||||
|
xs[i * 2u + 1u] = 0.f;
|
||||||
|
for (std::size_t j = 0; j < 8u; ++j) ys[i * 8u + j] = t;
|
||||||
|
rig.like(std::span<const float>(&xs[i * 2u], 2u),
|
||||||
|
std::span<const float>(&ys[i * 8u], 8u));
|
||||||
|
}
|
||||||
|
rig.mlp.train();
|
||||||
|
|
||||||
|
// The LAST example must be honoured; it cannot have been evicted.
|
||||||
|
rig.at(std::span<const float>(&xs[(N - 1u) * 2u], 2u), got);
|
||||||
|
NISPS_EXPECT(all_finite(got));
|
||||||
|
NISPS_EXPECT(l2(got, std::span<const float>(&ys[(N - 1u) * 8u], 8u)) < 0.5f);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Contradictory examples (same input, different targets) must converge to
|
||||||
|
// something finite rather than diverging. A musician WILL do this by accident.
|
||||||
|
NISPS_TEST(behaviour_contradictory_examples_stay_finite) {
|
||||||
|
Rig rig(2u, 8u);
|
||||||
|
const float x[2] = {0.f, 0.f};
|
||||||
|
for (int k = 0; k < 6; ++k) {
|
||||||
|
std::vector<float> y(8u, 0.1f + 0.15f * static_cast<float>(k));
|
||||||
|
rig.like(x, y);
|
||||||
|
}
|
||||||
|
rig.mlp.train();
|
||||||
|
|
||||||
|
std::vector<float> got, w;
|
||||||
|
rig.at(x, got);
|
||||||
|
rig.weights(w);
|
||||||
|
NISPS_EXPECT(all_finite(got));
|
||||||
|
NISPS_EXPECT(all_finite(w));
|
||||||
|
}
|
||||||
|
|
||||||
|
// The interaction model must hold at degenerate shapes, not just the default.
|
||||||
|
// 1-in/1-out is the narrowest legal net; a wide-output net is the realistic
|
||||||
|
// upper end once a mode drives 33 synth parameters.
|
||||||
|
NISPS_TEST(behaviour_holds_at_degenerate_shapes) {
|
||||||
|
struct Shape { std::size_t n_in, n_out, hidden; };
|
||||||
|
const Shape shapes[] = {
|
||||||
|
{1u, 1u, 4u}, // narrowest legal
|
||||||
|
{2u, 8u, 16u}, // operator default
|
||||||
|
{1u, 33u, 8u}, // one control, many parameters
|
||||||
|
{8u, 2u, 8u}, // many controls, few parameters
|
||||||
|
{32u, 8u, 16u}, // the over-provisioned browser head
|
||||||
|
};
|
||||||
|
for (const Shape& s : shapes) {
|
||||||
|
Rig rig(s.n_in, s.n_out, s.hidden);
|
||||||
|
std::vector<float> x(s.n_in, 0.2f), y(s.n_out, 0.6f), got, w;
|
||||||
|
rig.like(x, y);
|
||||||
|
rig.mlp.train();
|
||||||
|
rig.at(x, got);
|
||||||
|
NISPS_EXPECT(all_finite(got));
|
||||||
|
|
||||||
|
rig.fb.set_mode(FeedbackMode::Avoid, rig.mlp);
|
||||||
|
rig.fb.set_avoid_style(AvoidStyle::Geometric);
|
||||||
|
rig.down(x);
|
||||||
|
rig.weights(w);
|
||||||
|
NISPS_EXPECT(all_finite(w));
|
||||||
|
NISPS_EXPECT(rig.mlp.n_in() == s.n_in);
|
||||||
|
NISPS_EXPECT(rig.mlp.n_out() == s.n_out);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ===========================================================================
|
||||||
|
// EXPLORE-AND-PLACE ACCESSOR CONTRACT
|
||||||
|
//
|
||||||
|
// This exists because getting it wrong fails SILENTLY, and it did: reading
|
||||||
|
// placed_output() after commit_place() returns an EMPTY span, l2() over an
|
||||||
|
// empty span is 0, and 0 reads as a perfect placement. A benchmark scored a
|
||||||
|
// broken lifecycle as flawless until ASAN found the out-of-bounds downstream.
|
||||||
|
// ===========================================================================
|
||||||
|
NISPS_TEST(behaviour_placed_vs_committed_output_contract) {
|
||||||
|
Rig rig(2u, 8u);
|
||||||
|
std::vector<float> y(8u, 0.4f);
|
||||||
|
const float a[2] = {0.1f, -0.1f};
|
||||||
|
rig.like(a, y);
|
||||||
|
rig.mlp.train();
|
||||||
|
|
||||||
|
rig.fb.set_mode(FeedbackMode::ExploreAndPlace, rig.mlp);
|
||||||
|
rig.fb.enter_explore(rig.mlp, 1.f);
|
||||||
|
rig.fb.reroll(rig.mlp, 1.f);
|
||||||
|
|
||||||
|
// While Exploring: nothing is placed yet.
|
||||||
|
NISPS_EXPECT(rig.fb.placed_output().empty());
|
||||||
|
|
||||||
|
rig.mlp.process();
|
||||||
|
std::vector<float> audition(8u);
|
||||||
|
{ auto o = rig.mlp.outputs();
|
||||||
|
for (std::size_t j = 0; j < 8u; ++j) audition[j] = o[j]; }
|
||||||
|
rig.fb.begin_place(rig.mlp, audition);
|
||||||
|
|
||||||
|
// While Placing: placed_output() is the live accessor.
|
||||||
|
NISPS_ASSERT(rig.fb.placed_output().size() == 8u);
|
||||||
|
NISPS_EXPECT(l2(rig.fb.placed_output(), audition) == 0.f);
|
||||||
|
|
||||||
|
rig.fb.commit_place(rig.mlp);
|
||||||
|
|
||||||
|
// After commit: state is Idle, so placed_output() goes EMPTY and the
|
||||||
|
// caller must read committed_output() instead.
|
||||||
|
NISPS_EXPECT(rig.fb.placed_output().empty());
|
||||||
|
NISPS_ASSERT(rig.fb.committed_output().size() == 8u);
|
||||||
|
NISPS_EXPECT(l2(rig.fb.committed_output(), audition) == 0.f);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Reposition hands its carried vector over the same way.
|
||||||
|
NISPS_TEST(behaviour_reposition_carries_via_committed_output) {
|
||||||
|
Rig rig(2u, 8u);
|
||||||
|
std::vector<float> y(8u, 0.35f);
|
||||||
|
const float src[2] = {0.3f, 0.3f};
|
||||||
|
rig.like(src, y);
|
||||||
|
rig.mlp.train();
|
||||||
|
|
||||||
|
rig.fb.set_mode(FeedbackMode::ExploreAndPlace, rig.mlp);
|
||||||
|
for (std::size_t i = 0; i < 2u; ++i) rig.mlp.set_input(i, src[i]);
|
||||||
|
rig.mlp.process();
|
||||||
|
rig.fb.begin_reposition(rig.mlp);
|
||||||
|
|
||||||
|
NISPS_ASSERT(rig.fb.placed_output().size() == 8u);
|
||||||
|
std::vector<float> grabbed(rig.fb.placed_output().begin(),
|
||||||
|
rig.fb.placed_output().end());
|
||||||
|
|
||||||
|
rig.fb.commit_reposition();
|
||||||
|
NISPS_ASSERT(rig.fb.committed_output().size() == 8u);
|
||||||
|
NISPS_EXPECT(l2(rig.fb.committed_output(), grabbed) == 0.f);
|
||||||
|
NISPS_EXPECT(!rig.fb.repositioning());
|
||||||
|
}
|
||||||
|
|
||||||
|
// A fully-masked focus gate must freeze EVERY output: zero active dims means
|
||||||
|
// zero gradient, so a dislike is a no-op on the weights.
|
||||||
|
NISPS_TEST(behaviour_full_focus_mask_freezes_everything) {
|
||||||
|
Rig rig(2u, 8u);
|
||||||
|
std::vector<float> y(8u, 0.5f);
|
||||||
|
const float a[2] = {0.2f, 0.2f};
|
||||||
|
rig.like(a, y);
|
||||||
|
rig.mlp.train();
|
||||||
|
|
||||||
|
std::vector<std::uint8_t> mask(8u, 0u); // nothing active
|
||||||
|
rig.fb.set_focus_mask(mask);
|
||||||
|
rig.fb.set_mode(FeedbackMode::Avoid, rig.mlp);
|
||||||
|
rig.fb.set_avoid_style(AvoidStyle::Geometric);
|
||||||
|
|
||||||
|
std::vector<float> before, after;
|
||||||
|
rig.weights(before);
|
||||||
|
const float press[2] = {-0.1f, 0.1f};
|
||||||
|
for (int i = 0; i < 16; ++i) rig.down(press);
|
||||||
|
rig.weights(after);
|
||||||
|
|
||||||
|
bool identical = true;
|
||||||
|
for (std::size_t i = 0; i < before.size(); ++i) {
|
||||||
|
if (before[i] != after[i]) { identical = false; break; }
|
||||||
|
}
|
||||||
|
NISPS_EXPECT(identical);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Switching feedback mode mid-exploration must tear down cleanly and never
|
||||||
|
// strand the net in a randomised scratchpad state. Live, that would be fatal.
|
||||||
|
NISPS_TEST(behaviour_mode_switch_midflight_never_strands) {
|
||||||
|
Rig rig(2u, 8u);
|
||||||
|
std::vector<float> y(8u, 0.45f);
|
||||||
|
const float a[2] = {0.f, 0.2f};
|
||||||
|
rig.like(a, y);
|
||||||
|
rig.mlp.train();
|
||||||
|
|
||||||
|
std::vector<float> before, after;
|
||||||
|
rig.weights(before);
|
||||||
|
|
||||||
|
rig.fb.set_mode(FeedbackMode::ExploreAndPlace, rig.mlp);
|
||||||
|
rig.fb.enter_explore(rig.mlp, 1.f);
|
||||||
|
for (int i = 0; i < 3; ++i) rig.fb.reroll(rig.mlp, 1.f);
|
||||||
|
rig.fb.set_mode(FeedbackMode::Avoid, rig.mlp); // yank it out
|
||||||
|
|
||||||
|
rig.weights(after);
|
||||||
|
bool identical = true;
|
||||||
|
for (std::size_t i = 0; i < before.size(); ++i) {
|
||||||
|
if (before[i] != after[i]) { identical = false; break; }
|
||||||
|
}
|
||||||
|
NISPS_EXPECT(identical);
|
||||||
|
NISPS_EXPECT(!rig.fb.exploring());
|
||||||
|
}
|
||||||
Loading…
Reference in a new issue