ec3118004d
geo_push.hpp and replay.hpp cited memllib @ 0a541cc. upstream/main pins
e291192, where the same code had been deliberately redesigned — and because
InterfaceRL was not in the tree (fixed one commit ago), we carried the
superseded version for months. Three changes, all upstream's:
kGeometricPushScale 0.5 -> 1.0 (InterfaceRL.hpp:409)
kNegLRBase 0.5 -> 1.5 (InterfaceRL.hpp:410)
/(1+len) taper deleted (InterfaceRL.tpp:724)
Upstream's own comment on the taper: "a 'no' should clearly move the mapping
away even from a sound already far from the liked region (the taper used to
kill exactly that case)". The direction is already a unit vector, so the
taper only ever shrank the push for exactly the sounds a user is most likely
to be rejecting.
Cold start is folded into the same path. Upstream's useRandom is
`!havePositives || len <= 1e-4`: with nothing liked yet there is no centroid
to push away from, so every dim goes in a random direction. Ours instead
kept the older 0a541cc fallback — train AWAY from the heard action at a
NEGATIVE lr — which was inert whenever the heard action equalled the net's
own output, i.e. in the common case. One path now, and a "no" moves the
mapping before any likes exist (ml_bench E1: 0 -> 2.3e-3). The
GeometricColdStart action is still reported so callers keep their "like a
few sounds first" prompt; only the training changed.
Measured (ml_bench, one dislike at a point):
A4 0.0157 -> 0.0533 at-point displacement (3.4x), so end to end across
this and the RMSProp fix: 5.3e-5 -> 5.3e-2, ~1000x. The gap to the
legacy Diffuse design closes from ~4100x to ~4.2x.
D1 effective_lr 4.7e-4 -> 1.5e-3; 10 presses now reach 0.34, 100 reach
the full intended push.
A5 compounding 0.87 -> 0.96 (a second press at the same spot is no
longer noticeably weaker than the first).
A7 damage_ratio essentially unchanged (0.87-1.57) — the collateral
damage to protected positives scales with the push and is NOT
addressed here; it is the negative-feedback design question.
NOT adopted, deliberately: upstream's per-tick batch retraining over all
live negatives, and its fixed kDislikeLifetimeMs=2500 in place of our
proportional decay. Both need something the core does not have — a per-tick
call site and a millisecond clock inside nisps/ml — so they change
FeedbackControllerCore's interface rather than its constants. Recorded in
ALIGNMENT's deferred-debt entry alongside the existing one-press-one-step
divergence, and filed as its own task.
test_mlp_geo_dislike.cpp: the taper test now pins its ABSENCE (equal
displacement near and far), the cold-start test pins movement where it used
to pin inertness, and a new test covers the random-direction branch.
ALIGNMENT defect 6b resolved. Gates: build-cpp-tests 139 tests / ctest 4/4,
parity-check PASS (WASM rebuilt), lint-cpp clean, firmware slpworkshop
SUCCESS.
167 lines
16 KiB
Markdown
167 lines
16 KiB
Markdown
# ALIGNMENT
|
||
|
||
> Opinionated diagnosis of how well the codebase serves its mission, ranked by impact. Dated entries; remove when resolved rather than checking off. **Pruned every few weeks** — a stale diagnosis is worse than none.
|
||
|
||
## Mission
|
||
|
||
A research platform for interactive ML control of audio. We're building it to figure out what works and what doesn't — different ergonomics and ergodynamics of parameter sets, modes, ML architectures, audio engines, UI, and UX. Therefore: keep most/all parameters tweakable, ML/engine/UI/UX should each be configurable on their own axis, and the codebase has to enable/assist agentic AI coding patterns (confident changes, verifiable without hardware).
|
||
|
||
**Target vision (operator, 2026-07-20):** (1) one C++20 NISPS core serving RP2350 firmware and the browser, performance-sensitive on the MCU; (2) firmware modes runnable as modes in Manifold; (3) Manifold defaults to curated presets/modes, with the maximalist surface behind an "advanced" dev mode used to author them; (4) PlatformIO for hardware, no more .ino; (5) Manifold doubles as interface/editor for the hardware MEMLNaut (settings, presets, training, examples, visualisation).
|
||
|
||
The clean-slate rewrite (2026-04-29) consolidated everything into one C++20 codebase compiling to firmware AND WASM. Since 2026-07-13 (P1) the sole browser app is the React Manifold. JSON schemas remain the firmware↔browser parameter contract. A full-repo audit (2026-07-21, `docs/specs/recon/simplification-audit-2026-07.md`) grounds the entries below; mitigations are phased in `docs/specs/plans/simplification-plan.md`.
|
||
|
||
## Top defects (ranked by mission impact)
|
||
|
||
### 1. The mode layer is not shared: WASM re-orchestrates modes by hand (2026-07-21)
|
||
|
||
**What.** `nisps/modes/` — the CRTP layer binding ML config, engine, voice-space and I/O — compiles only into firmware. `nisps/wasm/bindings.cpp` includes engines and ML primitives but zero mode headers, and Manifold re-assembles mode behaviour (jolt stepping, OU, routing) in TS. "Firmware and WASM share the same modes" is true only at the engine level; every ModeBase behaviour must be mirrored browser-side by hand.
|
||
|
||
**Why it blocks the mission.** Vision bullet 2 is precisely this. Until the control-tick orchestration exists once in C++, every new mode behaviour is a dual implementation with drift risk.
|
||
|
||
**Rough cost.** Spec first, then ~a week: storage-policy the ModeBase orchestration the way P2 did MLPCore (verified shape in plan §6.5a — *not* binding monolithic mode objects, which would contradict the locked two-instance RT architecture). Related honesty gap: Manifold currently catalogues 4 modes that structurally cannot run in the browser (no mic input, event-only engines) — plan §6.5b (absorbs the old C15/mic-input defect; C15 itself lives on `archive/playground-solidjs`).
|
||
|
||
### 2. No curated/advanced split and no in-UI mode picker — the UI fights vision 3 (2026-07-21)
|
||
|
||
**What.** Manifold is 100% dev-maximalist: five drawers of everything, no preset data model to author against, and mode switching exists only via the debug hook — there is no instrument picker in the UI at all (the plumbing, `ctx.modes`/`setModeId`, already exists unused). A stratum of decorative controls (training-param sliders, master volume, bpm, A/B, snapshots, fabricated gradient health) renders real-looking UI that drives nothing.
|
||
|
||
**Why it blocks the mission.** The default experience is supposed to be curated presets; the advanced surface is the authoring tool. Neither exists, and the decorative stratum actively misleads research use.
|
||
|
||
**Rough cost.** Product-model decision first (plan §7.6), then incremental: picker is days; the curated-preset model seeds from `backends/presets.ts` + schemas; disclosure via per-drawer depth levels — which as of §6.5e (2026-07-21) has its first genuinely advanced-only consumer, the training-health panel, so the mechanism is proven rather than theoretical. The decorative stratum itself went in the Phase-1 sweep.
|
||
|
||
### 3. Manifold-as-hardware-editor is a facade (2026-07-21)
|
||
|
||
**What.** Vision bullet 5 exists as a 237-line Web Serial shell: sound connect lifecycle, zero protocol (`saveModel`/`restoreModel`/`getSettings` are literal stubs), and firmware has no serial command surface or on-device persistence to talk to.
|
||
|
||
**Why it blocks the mission.** The hardware research loop (train on device, inspect/curate in browser) is closed only by this bridge.
|
||
|
||
**Rough cost.** Week+, spec-first (plan §6.5d). The right discipline already exists in-repo: useq-celium's C-header wire truth + TS mirror + parity test; settings payloads should derive from schema codegen.
|
||
|
||
### 4. Dead mass and registry sprawl across every layer (2026-07-21)
|
||
|
||
**What.** *Phase 1 landed 2026-07-21 and removed the bulk of this:* the dead focus/altitude UI system, the decorative control stratum, 12 dead WASM API entries across the 5-file registration chain, the vendored daisysp tree, retired-playground artifacts and root planning relics, 5 unused primitives, the duplicate backend editor and catalogue, `voice_space.hpp`, `fixed_buffer.hpp`, the dead perf-macro regime, and the OSC bridge twin. What remains is the *registry* half: mode identity spread across ~6 hand-maintained registries with demonstrated drift, MLP dims typed twice, per-mode schema blocks hand-written in C++, and assorted stale specs presenting a deleted world as present tense.
|
||
|
||
**Why it blocks the mission.** The registries are dual-truth bugs waiting to fire (one already did: the selftest table). Stale specs are agent-confusing surface area.
|
||
|
||
**Rough cost.** Plan phase 3 (~2–3 days, codegen takes ownership) plus the docs disposition pass (§8). The behaviour bugs found en route (dataset-cap divergence, VCV 2-D input truncation, VCV audio-thread race and JSON) were fixed in phase 2 on 2026-07-21.
|
||
|
||
### 5. Performance is measured on the host but not on the target that constrains it (2026-07-21)
|
||
|
||
**What.** *Mostly closed 2026-07-21.* Size: the Phase 4 firmware CI job reports per-variant
|
||
flash/RAM on every push. Time: `scripts/bench-engines.sh` now reports per-engine ns/sample,
|
||
blocks/s and realtime factor on native AND WASM from one source
|
||
(`tests/cpp/engine_bench.cpp`), engines driven into a working state, with `--compare` for
|
||
per-engine deltas and a report step in CI. An engine getting 3x slower is now visible.
|
||
|
||
**What is left.** The numbers are HOST numbers. The mission's performance constraint is the
|
||
**RP2350 at 150 MHz**, and nothing measures there — a host realtime factor of 100x says
|
||
nothing about whether an engine fits in the MCU's per-block budget, and the two targets have
|
||
different FPU, cache and memory behaviour. The honest next step is an on-device timing report
|
||
(cycle counter around the audio callback, published over the existing display/serial surface),
|
||
which lands naturally with the hardware editor (defect 3) since that is what gives firmware a
|
||
command surface to report through.
|
||
|
||
**Rough cost.** Host half is done. On-device: ~a day, and it wants defect 3's serial protocol
|
||
to have somewhere to send the number.
|
||
|
||
### 6d. One like still heaves the whole mapping (2026-07-25)
|
||
|
||
**What.** A thumbs-up trains at `lr 1.0 x 1000 iterations` on every gesture. `ml_bench`
|
||
U1/U4 measure **lurch** — how far the mapping the musician is playing moves per single
|
||
gesture, averaged over the field: `lurch_max` 1.08 against a [0,1] output range, i.e.
|
||
one thumbs-up can move the mapping somewhere in the space by more than the entire output
|
||
range. Retention (how much of the previous teaching survives) is 0.38; at `iters=1` it
|
||
is 0.80.
|
||
|
||
**Why it blocks the mission.** This is the other half of the feedback asymmetry, and
|
||
RMSProp did NOT fix it — normalising the step size does not change the dose. Upstream
|
||
keeps both directions on small repeated steps; we take one enormous positive step and
|
||
one small negative one, so teaching feels like a lurch and correcting feels like
|
||
nothing. The two numbers now differ by ~70x rather than ~2e6x, which is progress and
|
||
still not a design.
|
||
|
||
**Rough cost.** Cheap to change, expensive to choose: the tuning space is now measurable
|
||
(`ml_bench` U4 sweeps dose; U1 sweeps upstream's soft-target alpha, where alpha=1 is
|
||
NISPS today). It wants a matched-N head-to-head, not a guess.
|
||
|
||
## Open mission questions
|
||
|
||
### Q1: Per-mode MLP architectures or one shared shape? (2026-04-29)
|
||
|
||
Schemas declare per-mode dims and since P5.3 both targets honour them. Is the mission served by maintaining per-mode shapes (research diversity) or collapsing to one (simpler ops)? Note the audit found all 9 mode schemas share copy-pasted ML defaults and 20 params are anonymous placeholders — the per-mode diversity is currently nominal (plan L40).
|
||
|
||
### Q2: Engine event taxonomy (2026-04-29)
|
||
|
||
`ControlEvent` is a flat enum consumed by the two sequencer modes. Revisit when a third event-emitting mode lands.
|
||
|
||
### Q3: Should Manifold stay desktop-first? (2026-04-29)
|
||
|
||
Legacy a-immersive was mobile-first; Manifold is desktop-first. Defer until user data exists.
|
||
|
||
## Deferred / accepted debt
|
||
|
||
- **EOC effects chain, ShapeSeq sequencer, modular engine (Phase E)** — legacy features consciously out of the v1 rewrite; revisit only if a mode wants them.
|
||
- **Inputs multi-source composition** (2026-06-28, reaffirmed 2026-07-21) — mix-and-match pad+gamepad+MIDI is a recorded, unreversed decision; the UI currently enforces exclusive single-source and the composition machinery sits dormant *by design*. Schedule or keep dormant — but the inputs-spec must stop presenting composition as current behaviour (plan §8).
|
||
- **Schema content is partially placeholder** (2026-07-21) — 20 anonymous "Param NN" slots across paf_synth/channel_strip/xiasri and copy-pasted ML defaults across all 9 modes. Name them during the first curated-preset pass per mode (plan §6.5c), or shrink `output_size` where the engine allows.
|
||
- **Geometric-dislike deliberate divergences** (2026-07-14, one-core P3; reaffirmed 2026-07-25): (1) the degenerate-branch RNG draws from the controller's deterministic `nisps::Rng`, not libc `rand()` — native==WASM parity holds; (2) upstream's async shuffled two-LR `optimise()` is collapsed into one synchronous `dislike_geometric()` training only the pressed negative's target — behavioural, not bitwise, parity with firmware upstream, by design; (3) `RandomiseMlp` uses `draw_weights(spread)` rather than the old asymmetric ranges. All intentional. The 2026-07-25 re-base onto `e291192` took upstream's constants and deleted the taper but did NOT adopt (2)'s counterpart — upstream retrains a batch over ALL live negatives on every 200 Hz tick and holds each negative at full strength for a fixed `kDislikeLifetimeMs = 2500` instead of decaying it proportionally. Adopting that needs a per-tick call site and a millisecond clock inside `nisps/ml`, which the core does not have; it is a real interface decision, not a constant, and is tracked as its own task.
|
||
- **Manifold dock splits `state`/`muted`/`armed`** (2026-06-28) — deliberate divergence from the deployed conflated `frozen`↔`muted` model (dock-spec §3.3). `muted`-downstream and the `soloMode` gradient-mask variants remain UI-only; the C API exposes `set_focus` but no per-mode gradient masking yet. (The audit found `soloMode` behaviourally inert in the controller — plan L20 trims it until `train_masked` exists.)
|
||
|
||
## Recently resolved (delete after a few weeks)
|
||
|
||
- 2026-07-25: **The geometric dislike is re-based on upstream `e291192` (defect 6b).**
|
||
`kGeometricPushScale` 0.5 -> 1.0, `kNegLRBase` 0.5 -> 1.5, and the `/(1+len)` taper
|
||
deleted — upstream's own comment is that a "no" should clearly move the mapping away
|
||
even from a sound already far from the liked region, which is exactly the case the
|
||
taper killed. Cold start folded into the same path (random direction when nothing is
|
||
liked yet) instead of the superseded negative-LR branch, so a "no" now moves the
|
||
mapping before any likes exist (`ml_bench` E1: 0 -> 2.3e-3). One dislike moves the
|
||
mapping **5.3e-2**, up from 1.6e-2 after the RMSProp fix and 5.3e-5 before it — a
|
||
~1000x change end to end, and now within ~4x of the legacy Diffuse design instead of
|
||
~4100x (`ml_bench` A4). Not adopted: upstream's per-tick batch retraining and fixed
|
||
2500 ms dislike lifetime — see "Deferred / accepted debt".
|
||
- 2026-07-25: **`InterfaceRL` is back in the tree (defect 6c).** Vendored verbatim from
|
||
memllib `e291192` at `firmware/MEMLNaut-NISPS/lib/memllib/reference/` — outside `src/`,
|
||
so PlatformIO never compiles it. Upstream drift in the feedback subsystem is a `diff`
|
||
again rather than an archaeology session.
|
||
- 2026-07-25: **The optimiser mismatch (defect 6) is fixed.** `nisps/ml/training.hpp` was
|
||
SGD-only while upstream `memlp` (`ea777502`) applies **RMSProp everywhere**, so every
|
||
learning rate we ported landed in an optimiser that reads it differently — an RMSProp
|
||
`lr` is a normalised step, an SGD `lr` multiplies the raw gradient. `rmsprop_step()`
|
||
now ports `Layer.h:239 ApplyAccumulatedGradients` exactly (decay 0.9, eps 1e-6,
|
||
sq-avg clamp 1e6, one-sided adjusted-LR clamp 1.0), with the per-weight running
|
||
squared-gradient average living in the storage policies so the zero-heap contract
|
||
holds. Measured on `ml_bench` D1: one geometric dislike moves the mapping **1.6e-2,
|
||
up from 5.3e-5**, and repeated presses now converge on the intended 0.5 push (0.12 at
|
||
10 presses, 0.56 at 100) instead of creeping linearly. Golden vector stages 2 and 3
|
||
were re-captured; stages 0 and 1 are pre-training and did not move. What this does NOT
|
||
fix: the dose asymmetry, now tracked as defect 6d.
|
||
|
||
- 2026-07-21: **Q4 (who owns memllib) closed.** Vendored at `firmware/MEMLNaut-NISPS/lib/memllib/`
|
||
from upstream `e291192`; the submodule and the fork are both gone. **Q5 (legacy feedback modes)
|
||
closed** — operator kept all four (`RandomiseOutputs`/`RandomiseMlp`/`Diffuse`/`on_drag`) as
|
||
building blocks for comparing how instruments feel under different behaviours, which upholds
|
||
rather than reverses `docs/adr/rl-feedback-design.md`.
|
||
- 2026-07-21: **Training-health telemetry (old defect 6) is gone** — the browser reads the real
|
||
per-iteration loss the core records, both fabrication sites are deleted (`wasm-worker.ts`'s
|
||
1-element array AND `wasm-iml.ts`'s sync-train twin, which the audit missed), and the display
|
||
sits behind the existing `expanded` drawer depth. The firmware buffer stays, per the L25 call.
|
||
|
||
- 2026-07-21: **Training-health telemetry (old defect 6) is real.** `nisps_ml_loss_history` now
|
||
crosses all five WASM registration layers, so the browser reads the SAME per-iteration curve the
|
||
firmware MLP records; `wasm-worker.ts`'s 1-element `new Float32Array([loss])` fake is gone from
|
||
both the sync and the async train paths; and the curve + the already-plumbed `get_layer_stats`
|
||
render in `manifold/src/console/TrainingHealth.tsx` at the Learning drawer's `expanded` depth.
|
||
The firmware buffer stays, per the operator call — it is the record the hardware editor
|
||
(defect 3) will read. One more fabrication went with it: `ConsoleCtx.loss`, a synthetic
|
||
`prev * 0.82` series no drawer read. With no history the panel says "no training run yet"
|
||
rather than drawing a plausible curve.
|
||
|
||
- 2026-07-21: **Arduino-CLI build machinery (old defect 3) is gone.** Phase 4 replaced it with a
|
||
PlatformIO project: one `[env:]` per variant is now the only variant registry, the `.ino`-mutating
|
||
Python/sed machinery and the `NISPS_ST_*` token-paste table and the sketch symlink forest and the
|
||
global TFT_eSPI mutation are all deleted, memllib is vendored (no submodule), and firmware finally
|
||
entered CI — three representative envs per run, which is what would have caught the SelfTest
|
||
variant sitting broken. All 16 envs build; sizes match arduino-cli within ~520 bytes.
|
||
|
||
- 2026-07-21: Full-repo simplification audit landed (recon + plan + this rewrite). Superseded entries removed: "browser-only engines incomplete" (→ defect 2/plan 5b), "loss curve not plumbed" (→ defect 7), "NISPS_AUDIO_FUNC misshapen" (→ plan Phase 1, S21/L13), stale "VCV not currently maintained" note (vcv/ is active and consumes `nisps/` directly post-P6).
|
||
- 2026-07-18: Browser curve maths unified onto the canonical `nisps/core/math.hpp` catalog at P4; four silently-divergent TS curves re-baselined.
|
||
- 2026-07-14: WASM MLP fixed-architecture defect resolved by P2 (`MLPCore<Storage>`; browser runtime-shaped, firmware zero-heap fixed).
|