diff --git a/docs/AGENT-REFERENCE.md b/docs/AGENT-REFERENCE.md index 03c4aad..026a7d4 100644 --- a/docs/AGENT-REFERENCE.md +++ b/docs/AGENT-REFERENCE.md @@ -39,6 +39,8 @@ Build: `cmake -S nisps -B nisps/build -G Ninja && cmake --build nisps/build && c Tests: 4 executables (`nisps_core_tests`, `nisps_dsp_engine_tests`, `nisps_modes_tests`, `nisps_golden_tests`). Run all: `bash scripts/build-cpp-tests.sh`. Parity vs WASM: `bash scripts/parity-check.sh` (asserts native and WASM produce identical outputs within 1e-5). +Throughput: `bash scripts/bench-engines.sh` (per-engine ns/sample, blocks/s, realtime factor on native + WASM; `--compare ` for deltas). It **reports and never asserts** — no threshold, no failure mode. Read it before and after any change to `nisps/dsp/` or `nisps/engines/`; nothing else in the repo will tell you an engine got slower. + ### Performance contract (RP2350) These rules apply to **all** code under `nisps/`. They are inert in WASM but kept globally for consistency. @@ -117,10 +119,10 @@ The browser MLP is runtime-shaped since P2 (`MLPCore`): `nisps_m ### Known limitations -- Loss history not yet plumbed through C API; only the final loss of a training run reaches TS. - Mic input through the worklet for XIASRI / SoundAnalysisMIDI is not wired in manifold. - C15 has no home on main (see `ALIGNMENT.md` defect 1, browser mode coverage). - (P3, 2026-07-14) The browser Jolt/OU gestures and the geometric dislike run the C++ core through WASM: `nisps_ml_jolt_*`, `nisps_ml_explore_*`, `nisps_ml_feedback_dislike_geometric` — no TS gesture math remains. +- (§6.5e, 2026-07-21) The per-iteration loss curve IS plumbed: `nisps_ml_loss_history(ml, out, max)` returns the total entry count and fills `min(count, max)`, so `max=0` is a count probe. Both train paths publish it to the spine; `EngineApi.lossHistory()` reads it and `console/TrainingHealth.tsx` displays it at `expanded` drawer depth. `MLPCore::train()` resets the history per run; `train_targets()` (geometric dislike) does not record. ## URL parameters (manifold) @@ -164,6 +166,9 @@ bash scripts/build-wasm.sh # Cross-platform parity bash scripts/parity-check.sh +# Engine throughput (reports; never fails) +bash scripts/bench-engines.sh + # Lint bash scripts/lint-cpp.sh diff --git a/docs/specs/dock-spec.md b/docs/specs/dock-spec.md index 73f14be..7ba1046 100644 --- a/docs/specs/dock-spec.md +++ b/docs/specs/dock-spec.md @@ -29,6 +29,15 @@ string "C15" MUST NEVER appear in the UI.* > solo/arm + min/max/curve). > - The feedback surface shipped richer than §1 (Explore-and-Place with reroll/nudge/place, plus > geometric dislike) — see `docs/adr/rl-feedback-design.md` and `nisps/ml/feedback.hpp`. +> - **§1.3's diagnostics suite shipped PARTLY, 2026-07-21** (simplification-plan §6.5e). The +> `nisps_ml_loss_history` C API the section waited on now exists, and +> `manifold/src/console/TrainingHealth.tsx` renders the real per-iteration loss curve plus the +> per-layer weight-health table (`nisps_ml_get_layer_stats`) at `expanded` depth — the advanced +> surface, since only two depths shipped. `` is NOT built and is not planned as +> drawn: the core records no per-layer gradient magnitudes, and the fabricated version of it was +> deleted in the Phase-1 sweep. `` as an edge-glow is likewise unbuilt; the same +> numbers are in the table. §1.3's six live sliders were deleted as decorative (S16) — `noise +> cap` and the Xavier switch are the survivors, because they drive real engine setters. --- diff --git a/docs/specs/plans/browser-mode-coverage-spec.md b/docs/specs/plans/browser-mode-coverage-spec.md new file mode 100644 index 0000000..7b18ea4 --- /dev/null +++ b/docs/specs/plans/browser-mode-coverage-spec.md @@ -0,0 +1,524 @@ +--- +kind: plan +status: active +--- + +# Browser Mode Coverage Honesty — audio topology as declared, verified truth + +*Dated 2026-07-21. Spec for `plans/simplification-plan.md` §6.5b (audit finding `A2`), written +spec-first per plan §6 — **proposal only, no phase adopted**. Companion decisions live in §8. +Absorbs the old ALIGNMENT "browser-only engines incomplete / mic input" defect.* + +*Every claim below was checked against the tree at `fd0aee2`. Measurements are reproducible; the +exact commands are given. Where I could not verify something I say so instead of asserting it.* + +--- + +## §0 What is actually true today (measured, not inferred) + +Two independent probes, both run 2026-07-21: + +1. **Native**, engines compiled straight from `nisps/engines/*.hpp` (`g++ -std=gnu++20 -O2 -I nisps`). +2. **The committed WASM artifact** `manifold/public/nisps.{js,wasm}`, driven through the exact C ABI + the AudioWorklet uses (`nisps_engine_create` / `set_params` / `process_block`), loaded with the + same glue dance as `manifold/tests/loss-history.test.ts`. + +Both were run for 2 s at 48 kHz with all params 0.5, first with a silent input, then with a +0.3-amplitude 220 Hz sine. Output RMS (left channel): + +| engine | silent in | sine in | native == wasm | +|---|---|---|---| +| `paf_synth` | **0.000000000** | *(ignores input)* | yes | +| `channel_strip` | 0.000000000 | 0.446555 | yes (≤1e-6) | +| `xiasri` | 0.000000000 | 0.848909 | yes (≤1e-6) | +| `verb_fx` | 0.000346808 | 0.267881 | yes (≤1e-6) | +| `memlcelium` | **0.399765** | *(ignores input)* | yes | +| `breakor` | 0.000000000 | — | yes | +| `elysiamorf` | 0.000000000 | — | yes | +| `analysis` | 0.000000000 | 0.000000000 | yes | +| `thru` | 0.000000000 | — | yes | + +Reading that against what the browser host actually supplies: + +- **Nothing is ever connected to the worklet's audio input.** `EngineHost` constructs the node with + `numberOfInputs: 1` (`manifold/src/engine/engine-host.ts:117`) and connects it to the destination + (`:121`), but there is no `getUserMedia`, `MediaStream`, or `createMediaStreamSource` anywhere in + `manifold/src` (grep: zero hits). The worklet already *reads* `inputs[0]` + (`manifold/src/engine/worklet/nisps-processor.ts:266-271`) and zero-fills when absent — so the + DSP side is done and the gap is one main-thread connection. +- **The engine C ABI has four entry points only**: create / destroy / set_params / process_block + (`nisps/wasm/bindings.cpp:1074-1108`, exported at `scripts/build-wasm.sh:70-71`). There is no + `note_on`, no `set_playing`, no `update_bpm`, no `pop_events`, no feature readback. + +Therefore, **of the nine schema-backed modes, exactly two produce audio in the browser today**: +`memlcelium` and `slp_workshop` (which share the `memlcelium` engine). + +| mode | why it is silent / inert in the browser | +|---|---| +| `paf_synth` | Note-gated. `process()` multiplies by `env_.play()` and the ADSR idles in `WaitToTrig` → exactly 0 (`nisps/engines/paf_synth.hpp:114`, `nisps/dsp/env.hpp`). Only `note_on` starts it (`:132-138`), and no browser path calls it. | +| `channel_strip` | Pure input processor; no audio input connected. | +| `verb_fx` | Pure input processor; RMS 3.5e-4 from a zero input is residual, not a signal. | +| `xiasri` | Pure input processor; no audio input connected. | +| `breakor` | Event-only by design (`nisps/engines/breakor.hpp:81-109` returns `{0,0}`). `playing_` defaults `true` (`:159`) so it *does* tick and enqueue, but `pop_events` is not exported, so the 64-slot `EventQueue` fills and silently drops (`nisps/core/event_queue.hpp:44-49`). | +| `elysiamorf` | Same shape (`nisps/engines/elysiamorf.hpp:100-104`, `playing_ = true` at `:136`). | +| `sound_analysis_midi` | Needs mic *and* a features readback. `AnalysisEngine::process` returns silence and exposes `features()` / `copy_features()` (`nisps/engines/analysis.hpp:129`, `:188-196`) — neither is exported. | +| `visualizer` (manifold-only) | Correctly silent: it is a visual mode on `thru`. | +| `c15` (manifold-only) | Already honest — `placeholder: true`, `badge: 'soon'` (`manifold/src/console/model.ts:301-313`). | + +### How the table was produced + +Both probes were throwaway scripts in `/tmp`, not committed — **§5 step 2 turns them into permanent +tests, which is the point of this document.** Their shape, so they can be rebuilt in ten minutes: + +- **Native.** One `.cpp` including `nisps/engines/*.hpp`; per engine: default-construct, + `setup(48000.f)`, `set_params` with a `param_count()`-sized vector of `0.5f`, then 96 000 calls to + `process({s, s})` accumulating `y.L² + y.R²`; `s` is 0 or `0.3 · sin(2π·220·i/48000)`. Built with + `g++ -std=gnu++20 -O2 -I nisps`. +- **Committed WASM.** A `bun` script using the same glue-loading dance as + `manifold/tests/loss-history.test.ts:34-45` (`new Function(...)` over `public/nisps.js`, then + `factory({ wasmBinary: public/nisps.wasm })`), then `M.cwrap` for `nisps_engine_create`, + `nisps_engine_set_params`, `nisps_engine_process_block`, `nisps_engine_destroy`; engine id written + as NUL-terminated ASCII into a 32-byte `_malloc` buffer exactly as the worklet does + (`nisps-processor.ts:217-230`); 750 blocks × 128 frames. + +The two agreed to ≤1e-6 on every engine — incidental corroboration that WASM parity holds on a path +`scripts/parity-check.sh` does not cover. + +--- + +## §1 Where the brief and the audit are wrong + +Trust the code over these documents, including this one. + +1. **"4 modes that structurally cannot run" is an undercount.** Audit `A2` names `breakor`, + `elysiamorf`, `xiasri`, `sound_analysis_midi`. `channel_strip` and `verb_fx` are in exactly the + same position as `xiasri` — all three are pure input processors with no input connected — and + `paf_synth`, the catalogue's flagship, is silent for a *third* reason the audit's four-way enum + cannot even express. The real number is **7 of 9**. +2. **"The UI presents them as available" is false today.** There is no instrument-mode picker in + Manifold at all. `ConsoleApp.tsx:326-329` says so explicitly; `ctx.modes` / `ctx.setModeId` are + plumbed with no renderer (`manifold/src/console/types.ts:50-57, 64-66`); the Dock's "Mode" + selector is the **output backend** picker, not the instrument (`manifold/src/console/output-mode.ts:1-18`). + The only way to reach another mode is `window.__mf.setMode()` under `?debug=1`. + **Consequence for sequencing:** the dishonesty is *latent*. It fires the moment §6.5c builds the + picker. 5b is therefore a **prerequisite of 5c**, and a "badge" alone would ship a component no + user can see. What 5b must deliver is the *data* plus the *capabilities*, so that 5c's picker has + something true to render and most modes no longer need a warning at all. +3. **The parity harness's PAFSynth stage is vacuous.** `tests/cpp/parity_check.cpp:175-190` runs + PAFSynth for 128 frames with silent input and no `note_on`, then averages — both sides push + exactly `0.0, 0.0`. Stage 3 compares zero to zero. (Stage 4, ChannelStrip, at least feeds a + 0.25 DC step.) This is on top of the known limits the brief states. A parity PASS is not evidence + for anything in this document. +4. **§7.2's DriverConfig work changed the firmware, not the browser.** `Mode::driver_config()` is + now part of the `nisps::Mode` concept (`nisps/core/concepts.hpp:74-80`), defaults to + `engine().driver_config()` in `ModeBase` (`nisps/modes/base.hpp:145-165`), and firmware brings + the codec up from it (`firmware/MEMLNaut-NISPS/glue/audio_driver.hpp`). **Mic vs line is real on + hardware.** Nothing in `manifold/` or `nisps/wasm/bindings.cpp` reads `DriverConfig` — the WASM + bridge never calls it. So yes, the browser side is the remaining gap for *routing*, but see the + next point, which is why 5b cannot simply reuse `DriverConfig`. +5. **`DriverConfig` is not a topology declaration and must not be pressed into service as one.** It + is a codec-negotiation record: `mic_input`, `mic_gain_db`, `line_level`, `output_volume`, + `sample_rate` (`nisps/core/types.hpp:50-71`, struct at `:65`). It *correlates* with topology by accident — the + three input-processing engines all set `line_level = 6u` — but `paf_synth` also deviates + (`output_volume = 0.9f`) while consuming no input, `breakor`/`elysiamorf` return a default + `{}` identical to a mode that expresses no opinion, and `analysis`'s `mic_input = true` is about + *which codec pin*, not about *whether audio input is required*. Inferring topology from + `line_level != 3` would be a guess dressed as a fact. Topology needs its own declaration. + +--- + +## §2 The model + +### §2.1 Why not the flat four-class enum + +The plan proposes `generator | audio-in-fx | event-only | analysis`. That enum cannot express +`paf_synth`, which is a generator that produces nothing without note events — the single biggest +user-visible gap in the catalogue. Three independent facts are being conflated: + +| axis | question | values | +|---|---|---| +| `input` | does the engine need an audio input signal to do its job? | `none` \| `required` | +| `output` | what does the engine emit? | `audio` \| `events` \| `none` | +| `trigger` | does it need note events to sound? | `free` \| `notes` | + +The four familiar class names survive as a **derived label**, not stored data: + +``` +output === 'events' → "Event sequencer" +output === 'none' && input==='required'→ "Analysis" +input === 'required' → "Audio FX" +otherwise → "Generator" +``` + +### §2.2 The declared values + +| mode | `input` | `output` | `trigger` | derived label | +|---|---|---|---|---| +| `paf_synth` | none | audio | **notes** | Generator | +| `channel_strip` | **required** | audio | free | Audio FX | +| `verb_fx` | **required** | audio | free | Audio FX | +| `xiasri` | **required** | audio | free | Audio FX | +| `memlcelium` | none | audio | free | Generator | +| `slp_workshop` | none | audio | free | Generator | +| `breakor` | none | **events** | free | Event sequencer | +| `elysiamorf` | none | **events** | free | Event sequencer | +| `sound_analysis_midi` | **required** | **none** | free | Analysis | + +### §2.3 Host capability, and a derived verdict — not a hard-coded badge + +The catalogue must never carry a hand-maintained "works in browser" flag: it would be wrong the day +a capability lands. Instead, the browser declares what it can currently provide, and support is +**computed**: + +```ts +interface HostAudioCapabilities { + audioInput: boolean; // a MediaStream is connected to the worklet node + noteTrigger: boolean; // nisps_engine_note_on/off exist and are wired + eventDrain: boolean; // worklet → host event channel exists + analysisFeatures: boolean; // analysis feature readback exists +} + +type ModeSupport = + | { ok: true } + | { ok: false; reason: 'needs-audio-input' | 'needs-note-source' | 'hardware-only'; detail: string }; +``` + +`modeSupport(schema.audio, host)` is a pure function; the UI renders its result. As each capability +in §5 lands, its flag flips to `true` and modes stop being labelled — no catalogue edit, no badge +list to maintain. Today the flags are `{ audioInput: , noteTrigger: false, eventDrain: +false, analysisFeatures: false }`. + +--- + +## §3 Where the truth lives + +**In `schemas/modes/.json`, as a new required `audio` object; codegen emits C++ and TS.** + +Rejected alternatives, with reasons: + +- *A hand-written topology map in `manifold/src/console/model.ts`* (the audit's suggested fix) — + this is a seventh hand-maintained mode registry, straight against Phase 3, which just finished + deleting six of them (`simplification-plan.md` §4). It also puts a fact about C++ engines in + TypeScript, where no C++ test can check it. +- *A WASM export `nisps_engine_topology(id)`* — `MF_MODES` is built at module load + (`model.ts:267-316`); making it await WASM would make the whole catalogue async for a static fact. +- *Deriving it from `DriverConfig`* — see §1.5. + +### §3.1 Schema shape + +Added to `schemas/schema.json` (`required` gains `"audio"`, `additionalProperties: false` preserved): + +```json +"audio": { + "type": "object", + "required": ["input", "output", "trigger"], + "additionalProperties": false, + "properties": { + "input": { "enum": ["none", "required"] }, + "output": { "enum": ["audio", "events", "none"] }, + "trigger": { "enum": ["free", "notes"] }, + "dsp_engine_id": { + "type": "string", "pattern": "^[a-z][a-z0-9_]*$", + "description": "The engine that occupies the audio path for this mode. Defaults to the mode's engine_id; declared only when the mode's ModeBase engine slot is a NoOp and a separately-composed engine owns the audio (sound_analysis_midi → analysis)." + } + } +} +``` + +`dsp_engine_id` is optional and is set on exactly one mode, `sound_analysis_midi: "analysis"`. It +exists to **delete** a hard-coded exception — see §6. + +### §3.2 Codegen anchors + +`codegen/generate.ts`, mirroring what `ui` already does — each of these is one line-group: + +| what | anchor | +|---|---| +| input TS type for a schema file | `interface ModeSchema` at `:93-126` (its `ui` block is `:116-120`) | +| C++ enums + `struct AudioConfig` in `schema_types.hpp` | beside `enum class PrimaryInput` `:264-272` / `struct UIConfig` `:273-277` | +| `AudioConfig audio;` member on `ParamSchema` | `struct ParamSchema` at `:288-300` | +| per-mode `inline constexpr AudioConfig kAudio` | beside the `// UI` block, `:503-518` | +| TS `AudioConfig` interface + `ModeSchema.audio` | `export interface UIConfig` at `:374-378` | +| TS per-mode emission | beside `:631-635` | + +Regenerate with `cd codegen && bun run generate.ts`; the golden snapshot +`codegen/tests/golden/paf_synth_schema.{hpp,ts}` must be updated in the same change, and CI's +codegen dirty-diff gate (added in plan §1.3) will fail on anything stale. + +--- + +## §4 Invariants + +Beyond the repo-wide constraints (platform-neutral allocation-free `nisps/`, 1e-5 parity, both +codegen outputs in one change, RT-safe worklet/dual-core comms), this area has its own: + +1. **Declared topology is verified by behaviour, never by comment.** Every value in §2.2 is asserted + by a test that drives the engine (§5 step 2). Adding an engine or a mode without a topology + declaration must fail to build; declaring one falsely must fail a test. +2. **The support verdict is derived, never stored.** No "browser-ok" boolean in a schema, a + catalogue, or an overlay. See §2.3. +3. **`nisps/` gains no browser-only concept.** `audio` is a schema/codegen fact consumed by the + generated `ParamSchema` and by tests. Firmware behaviour must not change: no engine or mode + source file changes in step 1, and `nisps_modes_tests` / `nisps_dsp_engine_tests` must stay green + with no edits to existing cases. +4. **Microphone input is opt-in, per session, and never auto-enabled.** Mic → worklet → destination + is an acoustic feedback loop through the room. Enabling requires an explicit user gesture and a + headphones warning; the mic is disconnected on `EngineHost.stop()` and on mode switch away from + an `input: 'required'` mode. +5. **No new audio-thread allocation.** Anything crossing worklet↔main (events, analysis features) + uses a pre-allocated channel. `postMessage` of a freshly-allocated typed array per render quantum + is not acceptable; COOP/COEP are already set for `manifold` (see `vite.config.ts` / + `ONBOARDING.md` §2), so `SharedArrayBuffer` is available and is the intended mechanism. +6. **The exclusive `InputMode` picker stays exclusive and stays ML-only.** `internal | gamepad | + midi` (`manifold/src/console/Drawers.tsx:284-288`) selects the *ML input source*. Microphone + audio is not an ML input source (except transitively for `sound_analysis_midi`, whose features + would enter through the engine, not through `input-layer.ts`). Do not add a fourth entry. +7. **Silence must be explained, not shipped.** If a mode is selectable and cannot sound, the UI says + which capability is missing. "It just doesn't make a noise" is the bug this document exists to + remove. + +--- + +## §5 Implementation sequence + +Each step is independently landable and independently verifiable. Steps 1–3 are the honest minimum; +4–6 are scoped by the decisions in §8. + +### Step 1 — declare topology (schema + codegen, both languages) + +Edit `schemas/schema.json` (§3.1) and all nine `schemas/modes/*.json` with the §2.2 values, plus +`dsp_engine_id: "analysis"` on `sound_analysis_midi`. Extend `codegen/generate.ts` at the §3.2 +anchors. Regenerate; commit `nisps/modes/generated/`, `manifold/src/modes/generated/`, and the +codegen golden. + +*No behaviour changes.* Note honestly: `ui.primary_input` / `show_voice_space_selector` / +`show_synth_visualizer` are generated into both languages and read by **nothing** outside codegen and +its golden (verified by grep across `manifold/src`, `nisps/`, `firmware/`). `audio` must not repeat +that — it acquires two consumers in steps 2 and 3 of this same plan. + +**Verification:** `bash scripts/build-cpp-tests.sh` (generated headers still compile into +`nisps_modes_tests`); `cd codegen && bun run test` (golden + curve-drift); `cd manifold && bun run +typecheck`; `bun run codegen/generate.ts` twice → no diff (idempotence). + +### Step 2 — make the declaration honest (two behavioural tests) + +**2a. `tests/cpp/test_mode_audio_topology.cpp`**, added to the `nisps_modes_tests` target +(`nisps/CMakeLists.txt:119-144` — that target already has the repo root on its include path and +already links the generated schemas). For every generated mode schema, resolve +`audio.dsp_engine_id ?? engine_id` to its C++ engine type and assert: + +| declared | assertion | +|---|---| +| `input: 'none'` | output is *identical* driven silent vs driven with a noise bed (the engine ignores its input). For `output: 'none'` engines, compare `features()` instead of audio. | +| `input: 'required'` | output RMS with the noise bed exceeds output RMS with silence by > 1e-2. | +| `output: 'audio'` | with the mode's sounding precondition satisfied (a `note_on` when `trigger === 'notes'`, a noise bed when `input === 'required'`), RMS > 1e-3. | +| `output: 'events'` | RMS is exactly 0 under both drives **and** `pop_events()` yields > 0 events within 2 s at the default tempo. | +| `output: 'none'` | RMS is exactly 0 under both drives. | +| `trigger: 'notes'` | RMS is exactly 0 before `note_on`, and > 1e-3 within 1 s after `note_on(60, 100)`. | +| `trigger: 'free'` | covered by the `output: 'audio'` row. | + +Drive params pseudo-randomly in [0.05, 0.95] from a fixed-seed `nisps::Rng`, **not** all-0.5 — that +vector is a degenerate corner and is precisely why `parity_check.cpp` stage 3 tests nothing +(`simplification-plan.md` §6.5f(3) made the same call for the bench). + +The engine_id → C++ type resolution is a `switch` local to the test. Guard it: assert the set of +resolved ids equals the set of ids across all generated schemas, so a new engine cannot be added +without the test noticing. + +**2b. `manifold/tests/engine-topology.test.ts`** (`bun test`, picked up by the existing +`bun test src tests/*.test.ts` glob — do not change that script). The §0 WASM probe, promoted: +drive the **committed** `manifold/public/nisps.{js,wasm}` through `nisps_engine_*` and assert the +same table at the browser's own ABI. This is the layer parity-check does not cover and the layer the +deploy actually ships. + +**Verification:** `bash scripts/build-cpp-tests.sh` (5 ctest targets green); `cd manifold && bun run +test`. Both are hardware-free and run in CI. If a §2.2 value is wrong, this is where it surfaces. + +### Step 3 — surface the truth in Manifold + +- `manifold/src/console/model.ts`: `MFMode` gains `audio: AudioConfig` (derived in + `modeFromSchema`, `:246-259` — one line, straight off the schema). +- New `manifold/src/console/mode-support.ts`: `HostAudioCapabilities`, `modeSupport()`, and + `modeClassLabel()` (§2.1/§2.3). Pure, unit-testable, no React. +- `ConsoleApp` builds the capability record from what the engine actually exposes (today: + `audioInput` from `engine.audio.hasAudioInput`, the other three literal `false` constants that get + deleted as steps 4–6 land) and puts it on `ConsoleCtx`. +- Render: a support chip in the synth `ModeConfig` block (`manifold/src/console/Drawers.tsx:549-570`, + the "Transport" section — it already names the active mode), stating why the active mode cannot + sound. When 5c builds the picker it reads the same `modeSupport()` for its per-entry badge and for + disabling `hardware-only` entries. + +**Verification:** unit test for `modeSupport()` over all nine modes × the current capability record +(`bun test`); `bun run typecheck`; e2e — extend `manifold/tests/e2e/schema-modes.spec.ts`, which +already switches modes through `window.__mf`, to assert the chip's text for one mode per class. + +### Step 4 — wire microphone input (makes the Audio FX class real) + +This is the cheapest real capability in the document: the worklet already consumes `inputs[0]` +(§0), so **no `nisps/` change, no new WASM export, no worklet change**. + +- `EngineHost`: `setAudioInput(stream: MediaStream | null)` — hold a + `MediaStreamAudioSourceNode`, `connect(this.node)` / `disconnect()`; handle "called before + `start()`" by stashing the stream and connecting once the node exists; drop it in `stop()`. +- `EngineApi.audio` gains `setAudioInput()` + `hasAudioInput` (`manifold/src/engine/engine-api.ts:95-102, + 203-207`). +- UI in the synth Transport block beside the play button, rendered only when the active mode + declares `input: 'required'`: an "Audio input" toggle that calls + `navigator.mediaDevices.getUserMedia({ audio: { echoCancellation: false, noiseSuppression: false, + autoGainControl: false } })` — the three defaults must be off or the codec fights the DSP — with + a headphones warning (invariant 4) and a clear denied/unavailable state. Requires a secure + context: production is HTTPS, dev is `http://localhost:5273`, both fine. +- Device picker: **out of scope**, default device only. Note it. + +**Verification, honestly graded:** +- *Strong:* step 2b already proves these three engines transform a driven input correctly at the + committed-WASM layer. +- *Medium:* an e2e with `--use-fake-device-for-media-stream --use-fake-ui-for-media-stream` and + `context.grantPermissions(['microphone'])` can assert `engine.audio.hasAudioInput === true` and + that the mode chip clears. That proves **wiring**, not audibility. +- *Weak / not automatable here:* that sound actually comes out of the speakers with a real mic. The + worklet's output is not observable from the page. This needs an operator listening check, once, + per engine. Say so in the commit rather than implying the gates cover it. + +### Step 5 — `paf_synth` note triggering *(gated on decision §8.1)* + +Two exports, `nisps_engine_note_on(void*, int note, int vel)` and `nisps_engine_note_off(void*, +int note)`, dispatched with `if constexpr (requires { e->note_on(n, v); })` so only engines that +have them respond (today: `paf_synth` alone). The chain is **worklet-side**, which differs from the +main-thread five-layer chain — the main thread never creates engines: + +1. `EMSCRIPTEN_KEEPALIVE` in `nisps/wasm/bindings.cpp` (beside `:1089-1108`) +2. `EXPORTED_FUNCTIONS` in `scripts/build-wasm.sh:70-71` +3. `WasmInstance.exports` interface + a `pickFn(...)` in `init_` and the re-export in the + `this.instance = { exports: { … } }` shim (`nisps-processor.ts:29-42`, `:145-165`, `:176-188`) +4. `HostToWorkletMessage` union + `onMessage_` branch (`engine-host.ts:35-53`, + `nisps-processor.ts:72-90`) +5. `EngineHost.noteOn/noteOff` → `EngineAudioApi` (`engine-api.ts:95-102`, `:203`) + +Miss a layer and it fails at runtime, not compile time. `manifold/src/engine/types.ts:159-167` also +declares the engine ABI for the main-thread module; it is unused by `wasm-iml.ts` — decide whether +to keep it symmetric or leave it alone, but do not assume it is load-bearing. + +RT safety: notes arrive on the worklet's existing `port`, handled between render quanta exactly as +params already are — bounded, no allocation inside `process()`. + +Trigger sources: `manifold/src/inputs/midi-input-source.ts:204-211` **already** emits `note:` +press/release actions with velocity from WebMIDI; `ConsoleApp` binds actions to verdicts today, so +routing note actions to `engine.audio.noteOn/Off` is a small addition there. Add one on-screen +momentary "Note" button in the Transport block as the no-hardware fallback. + +Note for context: on firmware, `paf_synth` is *also* note-driven — `glue/midi_io.hpp:41-50` routes +MIDI note-in to `mode.note_on`. The browser gap is not a divergence from hardware behaviour, it is +the absence of the same route. + +**Verification:** extend `tests/cpp/test_mode_audio_topology.cpp`'s `trigger: 'notes'` row (already +specified in step 2a — it will already be green before this step, since it tests the C++ engine, not +the browser); add a `manifold/tests/engine-topology.test.ts` case asserting the *exported* note +functions exist and produce non-zero output through the committed WASM; e2e asserting the chip +clears after a synthetic note. Audibility again needs an ear. + +### Step 6 — event-only and analysis modes *(gated on decisions §8.2 / §8.3)* + +**Default position: label them `hardware-only` and stop.** That is honest, costs nothing, and the +label disappears by itself when a capability lands (§2.3). + +If instead they are to be made real: + +- **Event-only (`breakor`, `elysiamorf`)** needs (a) `nisps_engine_pop_events`, + `nisps_engine_set_playing`, `nisps_engine_update_bpm` across the step-5 chain; (b) a worklet→main + event channel — a `SharedArrayBuffer` ring written by the worklet and drained on the main thread's + rAF (invariant 5; COOP/COEP are already set); (c) transport UI (play/stop/bpm) and a route from + events to `midi-backend.ts` that bypasses the per-param output path entirely, because these modes + emit notes, not parameter values. **This overlaps 5a**: on firmware the mode layer owns the event + drain (`nisps/modes/breakor.hpp:42-46`, `glue/midi_io.hpp:53-63`), so building a second, + browser-only orchestration here is exactly the duplication 5a exists to remove. **Sequence it + after 5a.** +- **Analysis (`sound_analysis_midi`)** needs step 4 *plus* `nisps_engine_copy_features` *plus* a + worklet→main path for six floats per control tick (same SAB channel) *plus* routing those into ML + input slots 0–5 alongside `joy_*` in slots 6–9 — which is `input-layer.ts`'s composition contract + and touches the exclusive-source picker (invariant 6, and the dormant multi-source decision, + §7.7 of the plan). Not a small item; do not smuggle it into 5b. + +--- + +## §6 Proposed deletion + +One, with its consumer named: + +**`modeEngineId()`'s hard-coded exception** — `manifold/src/console/model.ts:392-395`: + +```ts +export function modeEngineId(modeId: string): string { + if (modeId === 'sound_analysis_midi') return 'analysis'; + return MF_MODES.find((m) => m.id === modeId)?.engineId ?? 'thru'; +} +``` + +Consumer: `ConsoleApp.tsx:242` (`engine.audio.setBackend(...)`). Replaced by +`schema.audio.dsp_engine_id ?? schema.engine_id` (§3.1). This is the same class of hand-exception +Phase 3 spent two days removing, and it is the only place in the app where a mode's audio engine is +decided by an `if`. + +Nothing else in this document deletes anything. In particular, do **not** delete `nisps/engines/` +sequencer or analysis code because the browser cannot use it — firmware runs all of it. + +--- + +## §7 Adjacent, verified, deliberately out of scope + +Two findings from the same files. Recording them so the implementer does not rediscover them +mid-change; neither is part of 5b. + +1. **`SCHEMA_MODE_OVERLAYS.input` duplicates `ui.primary_input` and has already drifted.** + `model.ts:229-244` hand-declares `breakor: { input: 'joystick' }` while + `schemas/modes/breakor.json` declares `"primary_input": "xy_pad"`. The overlay value is live — + `ConsoleApp.tsx:814` feeds it to `resolveInputMap()` (`settings/settings-store.ts:146-153`) to + pick the input-map shape — so the two sources disagree about a rendered behaviour. Fixing it + means deriving `MFMode.input` from the schema and deleting the overlay field. +2. **`ui` is generated into both languages and consumed by nothing.** `primary_input`, + `show_voice_space_selector`, `show_synth_visualizer` have zero readers outside `codegen/` and its + golden. Either wire them (item 1 above wires `primary_input`) or delete them. Doing this in the + same commit as step 1 is cheaper than separately — same files, same regeneration — but it is a + different decision and should be asked for explicitly. + +--- + +## §8 Operator decisions + +These genuinely need a call; everything else in this document I decided and justified above. + +1. **`paf_synth` note triggering (step 5): build it, or label the flagship mode "needs a note + source"?** This is the most user-visible silence in the catalogue and the most likely to be read + as "the app is broken". Building it is two exports plus a UI button, and the MIDI note plumbing + already exists. Labelling it is free but leaves the best-developed engine unusable in the browser. + *Recommendation: build it.* Counter-argument worth weighing: note routing is arguably 5a's + territory (firmware does it in `ModeBase`/`midi_io.hpp`), so building it here creates a small + browser-only path that 5a will later absorb. +2. **Event-only modes (`breakor`, `elysiamorf`): label hardware-only, or build the event channel?** + *Recommendation: label now, build after 5a* — the browser-side orchestration is precisely what 5a + is meant to stop hand-writing. +3. **`sound_analysis_midi`: label hardware-only, or build the analysis path?** It needs mic + + feature readback + ML-input composition, and it collides with the dormant multi-source input + decision (plan §7.7). *Recommendation: label now.* +4. **`audio.dsp_engine_id` — adopt the field (and delete the `modeEngineId` exception), or keep the + exception and give the test a local special case?** Adopting it is the single-source answer and + is what §3.1/§6 assume; the cost is one more optional schema field used by exactly one mode. +5. **Bundle the §7 `ui` clean-up into step 1?** Same files and same regeneration, so bundling is + cheaper — but it widens a change that is otherwise purely additive. +6. **How much microphone UX?** Step 4 ships default-device-only with no picker and no input meter. + A device picker and a level meter are obvious follow-ons; say now whether they are wanted, since + a meter needs a tap on the input node that the current host does not have. + +--- + +## §9 What this plan does not do + +- It does not build the instrument-mode picker. That is §6.5c (`plans/curated-presets-spec.md`); + this plan gives it truthful data, and 5c should not ship a picker without it. +- It does not touch `nisps/modes/` orchestration. That is §6.5a + (`plans/mode-layer-reunification.md`), which is also where the browser event pump belongs — see + §5 step 6 and decision §8.2. +- It does not change firmware behaviour in any step. Every mode remains fully functional on hardware. +- It does not improve the parity harness, despite §1.3. Fixing stage 3 to trigger a note before + averaging is a one-line change with a golden-baseline consequence; it belongs with whoever next + touches `tests/cpp/parity_check.cpp`, and would bump `kVersion` (`:93`). diff --git a/docs/specs/plans/curated-presets-spec.md b/docs/specs/plans/curated-presets-spec.md new file mode 100644 index 0000000..e9c99f2 --- /dev/null +++ b/docs/specs/plans/curated-presets-spec.md @@ -0,0 +1,780 @@ +--- +kind: plan +status: active +--- + +# Curated Presets, the Instrument Library, and Per-Surface Disclosure + +*Dated 2026-07-21. The spec for `simplification-plan.md` §6.5c ("5c Curated/advanced split", +audit findings A3 + A7, `ALIGNMENT.md` defect 2). **Proposal only — no code is authorised until the +operator adopts it.** Every code claim below was read out of the tree on 2026-07-21 and is cited +with `file:line`; where a repo document disagrees with the code, the code is recorded as the truth +and the disagreement is called out in §1.6. British spelling in product copy. The built-in synth is +"Powerful Synth Engine" / "Built-in Synth" — the string "C15" must never reach the UI.* + +**Reading order for an implementer:** §1 (ground truth) → §2 (product model) → §3–§6 (design) → +§9 (sequenced plan). §7 is the invariant list to check work against; §11 is what the operator still +has to decide. + +--- + +## 0. Scope, and what this supersedes + +**In scope** (one item, three limbs that only make sense together): + +1. A **preset data model** — what a curated preset is as data, where it is stored, how it is + validated against schema truth, and how it is applied. +2. The **Instrument Library** — the missing in-UI instrument picker, built as a preset browser + (choosing a preset is how you choose an instrument). +3. **Progressive disclosure** via per-surface depth *levels*, replacing the accidental use of the + drawer geometry toggle as the advanced flag. + +**Out of scope, deliberately:** + +- Browser mode-coverage honesty / audio topology (`simplification-plan.md` §6.5b, spec'd in + `plans/browser-mode-coverage-spec.md`). The Library renders whatever signal 5b produces; it does + not invent one. **That spec's §0 calls itself a prerequisite of this item, and it is right** — the + picker is what makes the latent dishonesty visible. Sequencing recommendation: land 5b before §9 + step 6. If it has not, the Library lists every catalogue entry with no viability claim (§5.4) — + honest by omission rather than wrong, but a worse first impression. +- The mode-layer reunification (§6.5a) and the hardware editor (§6.5d). Presets are browser-local + data in this item; §11.Q4 records the promotion path if they later have to reach firmware. +- Region/param pins and the "control surface" (boldness/memory/precision) of + `manifold-parity-features-spec.md` §2 and §5. Neither is implemented, neither is needed here. + +**Supersedes** `plans/manifold-parity-features-spec.md` §1 ("Session presets (composed layers)"). +That section has **no code consumer** — nothing under `manifold/src/` implements or references it +(verified: no `SessionPresetV1`, no `mf-session` key, no `persist` helper in the tree). Its +composed-layer idea survives here in reduced form; three of its specifics do not, and the +divergences are explicit: + +| §1 of manifold-parity-features-spec | Disposition here | +|---|---| +| `layers.weights?: string` (base64url Float32Array, opt-in) | **Deleted.** Contradicts the operator's §7.6 decision that a curated preset is configuration only, network untrained (§2.1). | +| `PipelineLayer` = params of `engine/input-pipeline.ts` / `output-pipeline.ts` | **Deleted as written** — both files were deleted at one-core-engine P4; the config now lives in `manifold/src/engine/pipeline-types.ts` and has **zero UI writers** (§1.5). Nothing can author it, so a preset cannot meaningfully carry it yet. | +| `ControlLayer` = `{ boldness, memory, precision, presetId, offsets }` | **Deferred** — the control surface (§5 of that spec) is unimplemented and out of scope. | +| URL sharing (`?session=`, `?boldness=`) | **Deferred** to §11.Q5; the model is designed not to preclude it. | + +On adoption, `manifold-parity-features-spec.md`'s status block gains a line pointing §1 here. That +edit belongs to whoever lands step 3 of §9, not to this document. + +--- + +## 1. Ground truth (read out of the tree, 2026-07-21) + +### 1.1 There is no instrument picker, and the active instrument is invisible + +- The instrument mode is a plain `useState` in the console spine: + `manifold/src/console/ConsoleApp.tsx:99` — `const [modeId, setModeId] = useState('paf_synth')`. +- `ConsoleCtx` carries `modes: MFMode[]` and `setModeId` (`manifold/src/console/types.ts:64-66`), + populated at `ConsoleApp.tsx:735-737`. **Neither has a single reader.** `grep -rn "ctx.modes|ctx.setModeId" manifold/src` + returns only the comment at `ConsoleApp.tsx:326` that says so. +- The only place the active instrument's name is rendered anywhere in the app is one sentence of + explanatory prose — `Drawers.tsx:566`, `The active engine follows the selected mode ({ctx.mode.label})` — + which is itself gated behind `depth === 'expanded'` *and* the synth output target. In the default + session (particles target, condensed drawers) the user cannot discover which instrument is loaded. +- `MFMode.glyph`, `MFMode.cls`, `MFMode.placeholder`, `MFMode.badge` + (`manifold/src/console/model.ts:136-160`) have **zero consumers**. They are display fields + maintained for a renderer that does not exist. The Library is their first consumer, or they should + be deleted; see §8.4. +- Mode switching today happens only through the `?debug=1` seam + (`ConsoleApp.tsx:331-345`, `window.__mf.setMode/getModeId/paramCount/modeIds`), which + `manifold/tests/e2e/schema-modes.spec.ts` drives. **That seam must survive** (§7.8). + +### 1.2 Mode switching itself works, and reshapes the net + +Three effects fire on a `modeId` change, in declaration order: + +| Effect | Lines | Does | +|---|---|---| +| audio backend | `ConsoleApp.tsx:241-243` | `engine.audio.setBackend(modeEngineId(modeId))` | +| net reshape | `ConsoleApp.tsx:252-257` | `engine.reshape(mode.ml)` — schema dims, warm-started | +| transient reset | `ConsoleApp.tsx:260-271` | rebuilds `params` from the mode, recentres input, clears examples/pins/markers, closes drawers | + +`engine.reshape` (`engine-api.ts:323` → `wasm-iml.ts:355` → `nisps_ml_reshape`, +`nisps/wasm/bindings.cpp:397`) constructs a fresh `MLPCore` at the new dims and swaps it in. +Because the dataset lives **inside** `MLPCore` (`nisps/ml/mlp.hpp:143`, `:441`, `:443`), a reshape +destroys every stored example; `wasm-iml.ts:431` clears the TS mirror to match. This is the single +most important fact for preset application ordering — see §4.2. + +### 1.3 The only preset machinery is per-backend output routing + +`manifold/src/backends/presets.ts` (162 lines) is real, consumed, and narrow: + +- `OutputPreset` = `{ name, backend, rows: OutputPresetRow[], settings?, savedAt }` (`:16-37`). +- Storage is localStorage, **namespaced per backend**: `manifold-output-presets:` (`:38-41`). +- `OutputPresetRow` (`:16-29`) captures `name, status, muted, armed, min, max, curve, val, midi, osc, vcv`. +- The sole UI is `PresetBar` in `manifold/src/dock/OutputsBackendConfig.tsx:99-208`, rendered from + `:686`, itself gated behind `depth === 'expanded'` in `Drawers.tsx:634`. +- `PresetBar.backendSettings()` (`OutputsBackendConfig.tsx:113-118`) persists MIDI port/ccCount, OSC + url/sendRaw, VCV url/sendRaw — and `{}` for everything else. + +**The `cv` bug is confirmed and is worse than "a missing field".** `MFParam.cv` (`model.ts:133`, +`{ channel, gateThreshold }`) is live state: `useBackendManager.ts:161` reads `p.cv` per output to +build the uSEQ channel map, and `CvConfig` (`OutputsBackendConfig.tsx:590`) edits it. But +`OutputPresetRow` has no `cv` (`presets.ts:16-29`), `rowsFromParams` does not copy it (`:72-86`), +and `applyPreset` does not restore it (`:141-162`). Saving a preset for the `cvgate` backend +therefore silently discards **the entire channel assignment**, which is the only thing a CV preset +is for. Fix = three one-line additions (§9 step 1), exactly as the audit said. + +### 1.4 Two different things are called "Mode", and one backend is unreachable + +- **Instrument mode**: `MF_MODES` / `modeId`, schema-backed (`model.ts:316`). +- **Output mode**: `OutputMode` (`types.ts:26`) / `OUTPUT_MODES` (`console/output-mode.ts:33-73`), + which is what the dock's top button is labelled — `Dock.tsx:126` renders the header + `Mode · output target`. + +Adding an instrument picker beside a control already called "Mode" is a naming collision the design +has to resolve (§2.2). + +Also observed, and not recorded anywhere: **`BackendId` includes `'vcv'` +(`dock/output-state.ts:31`) but no `OutputModeDescriptor` maps to it** (`output-mode.ts:33-73` +lists particles/midi/osc/cv/synth/editor). `outputBackend` is derived solely from the selected +output mode (`ConsoleApp.tsx:137`), so the VCV backend — `vcv-backend.ts`, 240 lines, registered at +`backends/manager.ts:68`, with its own config panel at `OutputsBackendConfig.tsx:691` — **cannot be +selected from the UI at all.** That is out of this item's scope to fix, but the Library work will +put the output-target list in front of the operator, so it should be reported rather than quietly +inherited. + +### 1.5 What state a preset could capture, and what nothing can author + +| Candidate state | Where it lives | Authorable in the UI today? | +|---|---|---| +| instrument mode id | `ConsoleApp.tsx:99` | only via `?debug=1` | +| output target | `ConsoleApp.tsx:136` | yes (dock top button) | +| per-output routing (`MFParam` status/muted/armed/min/max/curve/val + midi/osc/vcv/cv) | `ConsoleApp` `params` store; `OutputControlRow` | yes | +| backend settings (MIDI port + CC count, OSC/VCV url + sendRaw) | `ConsoleApp` state, `ConsoleCtx` | yes | +| input source + gamepad stick mode + MIDI learn bindings | `inputs/useInputLayer.ts:113-120` | yes | +| feedback mode, solo mode, explore intensity, noise cap, Xavier spread | `ConsoleApp.tsx:104-133` | yes | +| **input/output pipeline config** (zoom, deadzone, curve, smoothing, slew, freeze) | `engine/pipeline-types.ts` | **no** — `grep` for `setInputConfig|setOutputConfig|defaultInputConfig` outside `src/engine/` finds only `tests/`. There is no UI writer. | +| **voice space** | `nisps/modes/base.hpp:279-291` | **no** — there is no `nisps_*_voice_space` export in `nisps/wasm/bindings.cpp`, and the worklet reaches the audio engine through raw WASM exports (`worklet/nisps-processor.ts:33-36`), so exposing it means walking the full export chain twice. See §11.Q3. | +| network weights | `wasm-iml.ts` | yes (`randomise`, training) — **deliberately excluded**, §2.1 | + +Schema fields `ui.show_voice_space_selector` and `ui.show_synth_visualizer` exist in every mode +schema (e.g. `schemas/modes/paf_synth.json`) and reach TS as `UIConfig` +(`manifold/src/modes/generated/types.ts:52-56`) but have **no consumer on either target**. + +### 1.6 Where documents disagree with the code + +Recorded here so the next session does not re-derive them. None of these are this item's job to fix +except where §8/§9 says so. + +1. **`ONBOARDING.md` §4** says `OUTPUT_MODES` = "particles (default) / midi / osc / synth / editor" — + five. The code has six; `cv` (uSEQ CV/gate) is missing from the doc. `output-mode.ts:32`'s own + comment ("The five Modes") is wrong the same way, one line above a six-element array. +2. **`model.ts` overlay vs schema — a live dual truth with real drift.** `ModeOverlay.input` + (`model.ts:211-216`) is hand-written per mode (`:229-247`) while every schema already declares + `ui.primary_input` (`manifold/src/modes/generated/types.ts:52-56`). They agree for eight of nine + modes and **disagree for `breakor`**: the schema + says `xy_pad`, the overlay says `joystick` (`model.ts:235`). The overlay wins today, and it is + the input to `resolveInputMap(settings.inputMap, mode.input)` (`ConsoleApp.tsx:814`), so Breakor + renders a circular joystick surface where its schema asks for a rectangular pad. Single consumer, + five-line fix (§9 step 2). +3. **`dock-spec.md` §0** prescribes three depths (`peek | expand | FULL`) as **per-drawer** state. + Two shipped, and they are **one global state** shared by all five drawers + (`ConsoleApp.tsx:110`), reset to `condensed` on every drawer switch (`Dock.tsx:204`) and every + mode switch (`ConsoleApp.tsx:268`). Its own 2026-07-21 grounding note already records the + two-depth divergence; the *global-not-per-drawer* part is not recorded anywhere. +4. **`manifold-parity-features-spec.md` §1.1** describes a `PipelineLayer` over + `engine/input-pipeline.ts` and `output-pipeline.ts`. Both files were deleted at P4. +5. `simplification-plan.md` §7.6 still lists the curated-preset product model as an *open* operator + decision. The operator has since decided it (§2.1). Reconciling the plan document is the + orchestrator's, not this spec's. + +--- + +## 2. The product model + +### 2.1 What a curated preset is (operator decision, §7.6) + +> **A curated preset is configuration only: mode + parameters + routing, with the network +> UNTRAINED — though it may sometimes ship seed training data.** + +Consequences taken as binding: + +- **A preset never carries weights.** Not optionally, not opt-in. The type has no weights field + (§3.1) and the apply routine never calls `setWeights`. This is testable — see §7.1. +- **A preset may carry seed examples**: `(input vector, output vector)` pairs at the mode's schema + arity. Loading them puts them in the dataset; it does **not** train (§4.4). +- Therefore applying a preset is *deterministic and cheap*: it reshapes the net to the mode's schema + dims (which mode switching already does) and writes UI/routing state. The sound you get is the + untrained net through the preset's routing — the preset defines the *instrument and the mapping + space*, and the user teaches it. + +### 2.2 Naming: Instrument vs Output + +To end the collision in §1.4, product copy uses two words and never the word "Mode": + +| Concept | UI label | Code | +|---|---|---| +| which instrument/mode is loaded | **Instrument** | `modeId`, `MF_MODES` (unchanged — `mode_id` is schema truth) | +| where the outputs go | **Output** | `OutputMode` → renamed `OutputTarget` (§8.3) | + +`Dock.tsx:126`'s `Mode · output target` header becomes `Output`. This is copy plus a mechanical +type rename; it changes no behaviour and is typecheck-verified. + +### 2.3 The Library is the picker + +There is no separate "instrument picker" and "preset browser". One surface — the **Library** — +lists curated presets grouped by instrument; choosing one selects its instrument. Below the presets, +at the advanced level, sits the bare instrument list (load an instrument with no preset). This is +one surface instead of two, and it makes the curated path the default path structurally rather than +by exhortation, which is what vision bullet 3 asks for. + +--- + +## 3. Data model + +### 3.1 Types + +New module `manifold/src/presets/types.ts` — types + pure validation only, no React, no storage: + +```ts +import type { ParamStatus } from '../console/model'; +import type { OutputTarget } from '../console/types'; +import type { CvSpec, MidiCcSpec, OscSpec, VcvSpec } from '../dock/output-state'; + +/** One output's deviation from the mode default. `i` indexes the mode's schema params. */ +export interface OutputOverride { + i: number; + /** Display name override (MIDI/OSC outputs get renamed by users). */ + label?: string; + status?: ParamStatus; + muted?: boolean; + armed?: boolean; + min?: number; + max?: number; + curve?: number; + /** Held value; only meaningful when status === 'fixed'. */ + val?: number; + midi?: MidiCcSpec; + osc?: OscSpec; + vcv?: VcvSpec; + cv?: CvSpec; +} + +export interface PresetRouting { + /** Status for every output NOT named in `outputs`. Omit to leave mode defaults alone. */ + rest?: ParamStatus; + outputs: OutputOverride[]; +} + +export interface PresetInputs { + mode: 'internal' | 'gamepad' | 'midi'; + gamepadStickMode?: 'single' | 'double'; +} + +export interface PresetLearning { + feedbackMode?: 'explore-and-place' | 'geometric-dislike'; + exploreIntensity?: number; + noiseCap?: number; + spread?: boolean; +} + +/** Seed examples, at the mode's schema arity. Validated in `validatePreset`. */ +export interface PresetSeed { + inputs: number[][]; // each length === schema.ml.input_size + outputs: number[][]; // each length === schema.ml.output_size +} + +export interface Preset { + v: 1; + id: string; // slug, e.g. 'paf-synth/glass-formants' + name: string; + /** Optional one-line description shown in the Library. */ + blurb?: string; + modeId: string; // MUST be an MF_MODES id + target: OutputTarget; + routing?: PresetRouting; + backendSettings?: Record; // same blob shape PresetBar already writes + inputs?: PresetInputs; + learning?: PresetLearning; + seed?: PresetSeed; + origin: 'builtin' | 'user'; + savedAt?: number; // user presets only +} +``` + +**There is no `weights` field, by design.** A reviewer should treat any future addition of one as a +reversal of §2.1 requiring the operator. + +Rationale for the choices that are not obvious: + +- **`OutputOverride` carries its own index and layers are optional**, so a hand-authored curated + preset for a 56-output mode is a dozen lines, not 56 rows of boilerplate. Index (not name) is + canonical because `MFParam.name` is user-editable for MIDI/OSC (`presets.ts:147-149`) while the schema + param order is stable codegen output. +- **`rest`** lets a curated preset say "these six outputs are live; everything else is off" in one + field — which is what curation mostly *is*. +- **Backend settings stay an untyped blob**, matching `OutputPreset.settings` + (`presets.ts:35`) and the writer/reader pair at `OutputsBackendConfig.tsx:113-131`. Typing it + properly is a separate, larger job with no consumer asking for it. +- **The layer types are `Preset`-prefixed** because `InputLayer` is already taken: it is the input + composition hub class in `manifold/src/inputs/input-layer.ts`. Do not reuse the name. + +### 3.2 Storage + +- **Built-in (curated) presets**: TypeScript data modules, `manifold/src/presets/builtin/.ts`, + re-exported from `builtin/index.ts` as `BUILTIN_PRESETS: readonly Preset[]`. Typed, typechecked, + tree-shaken, no build step, no codegen. + + *Why not `schemas/` + codegen?* Codegen exists to keep firmware and browser honest about a shared + contract. Presets have no firmware consumer today (firmware modes are compile-time + `-DMEMLNAUT_MODE_TYPE` selections with no on-device persistence — `ALIGNMENT.md` defect 3). + Adding a C++ emitter for data nothing in C++ reads would be complexity without a requirement. + **Removal condition:** when §6.5d (hardware editor) gives firmware a command surface and + persistence, promote presets to `schemas/presets/*.json` + codegen with both outputs in the same + change, per the standing schema rule. + +- **User presets**: one localStorage document, key `mf-presets`, shape + `{ v: 1, presets: Preset[], lastAppliedId?: string }`, written through a store modelled on + `settings/settings-store.ts` (framework-neutral class + `useSyncExternalStore` hook + module-level + singleton). No new persistence mechanism. + +- The per-backend namespaces `manifold-output-presets:` are **retired** (§8.1). See §11.Q1 + — deleting them discards any presets the operator saved in the live app. + +### 3.3 Validation against schema truth + +`validatePreset(p, schemas): string[]` in `presets/types.ts`, pure, returns human-readable problems: + +- `modeId` resolves in `MF_MODES`. +- every `OutputOverride.i` is `< schema.params.length`. +- `min <= max`, all of `min/max/curve/val` within `[0,1]` (the routing-knob space — `MFParam`'s + `min/max/curve/val` are 0..1 routing semantics; schema engine units live in the `schema*` fields, + `model.ts:107-121`). +- `seed.inputs[k].length === schema.ml.input_size`, `seed.outputs[k].length === schema.ml.output_size`, + `inputs.length === outputs.length`, and `inputs.length <= 128` — the shared dataset capacity + (`nisps::ml::kDefaultMaxExamples`, `nisps/ml/storage.hpp:57`, mirrored at `wasm-iml.ts:52` and + reported through `nisps_ml_describe`, `bindings.cpp:926`). +- `target` resolves in `OUTPUT_TARGETS`. + +A unit test runs `validatePreset` over every entry of `BUILTIN_PRESETS` (§9 step 4). This is the +mechanism that stops curated data rotting when a schema changes — the failure is a red `bun test`, +not a silent misbehaviour. + +--- + +## 4. Applying a preset + +### 4.1 One imperative entry point, not an effect chain + +`manifold/src/presets/apply.ts` exports: + +```ts +export interface PresetApplier { + engine: EngineApi; + selectInstrument(modeId: string): void; // reshape + rebuild params + reset transients + setParams(next: MFParam[]): void; + setTarget(t: OutputTarget): void; + setBackendSettings(s: Record | undefined): void; + setInputs(l: PresetInputs | undefined): void; + setLearning(l: PresetLearning | undefined): void; +} + +export function applyPreset(a: PresetApplier, p: Preset): void; +``` + +`ConsoleApp` provides the applier; the sequencing logic lives in `apply.ts` where it can be unit +tested against a fake applier. `ConsoleApp.tsx` is already 1082 lines — this must not add to it. + +### 4.2 The ordering invariant (the thing that will silently break) + +`engine.reshape` destroys the dataset (§1.2). Today the reshape happens in an **effect** +(`ConsoleApp.tsx:252-257`), which runs *after* the event handler that changed `modeId`. So the naive +implementation — "handler sets modeId, then adds seed examples" — loads the seeds and then has them +wiped by the effect one tick later, silently, with a dataset counter that briefly reads right. + +**Fix, and it is a simplification rather than a workaround:** collapse the three mode-switch effects +(`ConsoleApp.tsx:241-243`, `:252-257`, `:260-271`) into one imperative `selectInstrument(modeId)` +function that does, in order: `engine.audio.setBackend` → `engine.reshape` → rebuild `params` from +the mode → reset transients → `setModeId`. Keep exactly one effect, keyed on `[engine]`, that calls +`selectInstrument(modeId)` once when WASM becomes ready (the boot path the current effects rely on). +`applyPreset` then runs entirely inside one handler, in a guaranteed order, with no implicit +cross-effect contract. + +`window.__mf.setMode` (`ConsoleApp.tsx:336`) repoints to `selectInstrument`. + +**This is a refactor of live behaviour and must be verified as one** — see §9 step 3. + +### 4.3 Apply order + +1. `selectInstrument(p.modeId)` — reshape lands here, dataset is now empty. +2. `setTarget(p.target)`; `setBackendSettings(p.backendSettings)`. +3. Build the params array: start from the mode's schema defaults (what step 1 produced), apply + `routing.rest` to every output, then apply each `OutputOverride` by index. One `setParams`. +4. `setInputs(p.inputs)`, `setLearning(p.learning)`. +5. If `p.seed`: `engine.addExample(inputs[k], outputs[k])` for each pair. **No training.** +6. Record `lastAppliedId`. + +Steps 3–5 must not be reordered: 5 after 1 (else wiped), 3 after 1 (else overwritten by the mode +rebuild). + +### 4.4 Seeds are loaded, not trained + +After apply, the Learning drawer shows `N examples` (its existing chip, `Drawers.tsx:219`, fed +by `ctx.datasetCount`), and the user trains or gives verdicts. Nothing auto-trains, because §2.1 +says the network ships untrained and auto-training would make that false. §11.Q2 records the +alternative if the operator wants a preset to be able to opt into one training pass. + +--- + +## 5. The Library (the picker) + +### 5.1 Placement + +A second pinned control at the top of the dock rail (`Dock.tsx:405`), **above** the existing output +selector, rendering the active instrument's glyph and opening a popover — the same interaction and +the same visual language as `ModeSelector` (`Dock.tsx:64-174`), which becomes `OutputSelector`. Two +labelled controls, top of rail: **Instrument**, **Output**. + +Rejected alternatives, briefly: a sixth drawer (a preset browser is a chooser, not a workbench, and +drawers are mutually exclusive with the surface you are auditioning against); a stage-header title +bar (the stages are full-bleed by design and CompositeStage owns its own chrome). + +### 5.2 Contents + +At **basic** level: + +- Curated presets, grouped by instrument, each row: preset name, instrument label + glyph, `blurb`. +- The active preset marked; applying another switches instrument as a side effect. +- A "seeds" chip on presets that carry `seed`, showing the example count — so the difference between + "config only" and "config + seeds" is visible before you commit to it, not after. + +At **advanced** level, additionally: + +- **Instruments** — the bare `MF_MODES` list (`ctx.modes`, finally consumed), each entry loading the + instrument with no preset. This is the authoring entry point: load a bare instrument, configure, + save as a user preset. +- **Save current as preset** / rename / delete for user presets, reusing `PresetBar`'s interaction + pattern (`OutputsBackendConfig.tsx:99-208`) rather than inventing a second one. + +### 5.3 Applying is not silent + +Applying a preset switches instrument, which resets the dataset and the net. Today that is already +true of a mode switch and is done without confirmation ("switching instrument is already a +deliberate act", `ConsoleApp.tsx:250`). Keep that: **no modal** — but the Library rows carry a +persistent one-line warning ("loading resets examples and the network") and, when +`ctx.datasetCount > 0`, the row click routes through the existing `ReshapeModal` pattern +(`console/ReshapeModal.tsx`) with preset-specific copy. Losing 40 hand-taught examples to a +mis-click is the failure this prevents; losing an untouched default session is not worth a modal. +The condition is deliberately `datasetCount` alone — "unsaved routing edits" would need a +dirty-tracking mechanism the app does not have and this item does not justify building. + +### 5.4 Viability signalling + +The Library renders `MFMode.badge` when present (`model.ts:158`) and disables entries with +`placeholder: true` (`:157`) — both fields exist and neither has a consumer today (§1.1). If §8.5 +lands, `c15` is the only `placeholder: true` entry and the disable path becomes +dormant-but-correct; keep it, it costs one line. + +The Library does **not** invent a viability model. `plans/browser-mode-coverage-spec.md` §2.3 +proposes exactly the right shape — a pure `modeSupport(schema.audio, host)` returning +`{ ok: true } | { ok: false; reason; detail }`, computed from declared per-mode audio topology and +declared host capabilities rather than a hand-maintained flag — and the Library should render that +result verbatim: a chip carrying `detail`, the row disabled when `ok` is false. If 5b has not +landed, entries render unmarked. **Do not add a topology map to `model.ts` in the meantime** — that +is precisely the seventh hand-maintained mode registry Phase 3 spent two days eliminating, and 5b +rejects it by name. + +### 5.5 Boot + +Boot applies `lastAppliedId` if it resolves, else `DEFAULT_PRESET_ID`. + +**`DEFAULT_PRESET_ID` must be a `paf_synth` preset.** The e2e suite clears localStorage before every +run (`tests/e2e/helpers.ts:31-41`), so boot always takes the default path, and `ONBOARDING.md` §5 +plus several tests pin the boot shape at `4→[10,10,14]→33`, weights 809. Choosing a default on any +other instrument silently changes the boot contract several tests assert. If a different default is +wanted, the boot-shape claims in `ONBOARDING.md` and the affected specs move in the same change. + +--- + +## 6. Progressive disclosure: per-surface levels + +### 6.1 What `depth` actually is today + +`DrawerDepth = 'condensed' | 'expanded'` (`types.ts:38`) is **one global state** +(`ConsoleApp.tsx:110`) that simultaneously controls panel geometry (360px side panel vs 80vw×80vh +centred modal, `Dock.tsx:265-299`) and content disclosure. §6.5e adopted `expanded` as "the +advanced-surface flag" for the training-health panel, and `ONBOARDING.md` now documents that as the +rule. + +It is not a rule; it is an accident. Classifying every gate in `Drawers.tsx`: + +| Site | Content | What the gate is really for | +|---|---|---| +| `:171` | feedback-mode explanation | **prose** | +| `:208` | explore/Jolt explanation | **prose** | +| `:229` | solo/arm scope, solo behaviour, noise-cap slider | **advanced** | +| `:254` | feedback lab, Xavier switch, `` | **advanced** | +| `:396` | active input source's status message | **status — should never have been hidden** | +| `:403` | internal-pad explanation | **prose** | +| `:410` | gamepad stick mode + button legend | **function** — you cannot pick one/two sticks without opening a modal | +| `:441` | MIDI device picker + MIDI-Learn | **function** — same | +| `:523` | dedicated-dimensions explanation | **prose** | +| `:564`, `:583`, `:592`, `:598` | per-target explanations | **prose** | +| `:620-621` | rows sliced to 6 when condensed | **geometry** | +| `:634` | `` (CC table, CV channel map, preset bar) | **function + geometry** | +| `:641` | row list max-height 460 vs 220 | **geometry** | +| `:654` | `showCurve` on each row | **geometry** | +| `:659` | "+N more — expand to edit" | **geometry** | +| `:691` | unfocused icon colour + preview | **advanced (cosmetic detail)** | +| `:720`, `:737` | settings explanations | **prose** | + +Three of twenty-one gates are genuinely about expertise. The rest are geometry, prose, or — in three +cases — real functionality hidden by a size switch. Using this as the advanced flag means the +advanced surface and the big-panel surface can never diverge, and it means beginners' explanatory +prose is hidden from beginners, which is exactly backwards. + +### 6.2 The split + +Two orthogonal axes, one existing mechanism each: + +- **`presentation: 'panel' | 'modal'`** — today's geometry toggle, renamed. Per-drawer, transient, + driven by the ⤢ tab (`Dock.tsx:302-328`) and `\` (`ConsoleApp.tsx:642`). Behaviour unchanged. +- **`level: 'basic' | 'advanced'`** — the disclosure axis. **Per surface, persisted** in the existing + settings document (`settings-store.ts`, key `mf-settings`) as + `levels: Record`. No new store, no new persistence pattern, + no global boolean. The Library is a surface with a level for the same reason the drawers are, so + §5.2's advanced tier needs no second mechanism. + +Two levels, not `dock-spec.md` §0's three: there is exactly one advanced tier of content to put +anywhere, and inventing a third empty level is the kind of speculative structure this repo has been +deleting all week. `dock-spec.md` §0's `peek` maps onto nothing that shipped. + +Control: a small `basic | advanced` segmented toggle in each drawer's header +(`Dock.tsx:355-368`), beside the existing depth label at `:364` — which currently renders the +literal string `condensed`/`expanded` and becomes the presentation state's label. + +### 6.3 Re-gating, item by item + +Applying §6.1's classification: + +- **advanced** (`:229`, `:254`, `:691`): gate on `level === 'advanced'`. `` keeps + its advanced gating — the §6.5e decision survives intact, only its mechanism changes. +- **function** (`:410`, `:441`, `:634`): ungate. Configuring your gamepad or your CV channel map is + not advanced and must not require a modal. They stay in whatever presentation the drawer is in. +- **status** (`:396`): ungate. +- **geometry** (`:620-621`, `:641`, `:654`, `:659`): gate on `presentation === 'modal'`. Unchanged + behaviour, renamed condition. +- **prose** (`:171`, `:208`, `:403`, `:523`, `:564`, `:583`, `:592`, `:598`, `:720`, `:737`): + gate on `level === 'basic'` — *shown* to beginners, *hidden* from the advanced surface, which is + the inversion of today. This is a judgement call, not a derivation; §12.3 records it as mine and + the cheap fallback (always show) if the operator disagrees. + +### 6.4 Level does not reset + +`depth` is reset to `condensed` on every drawer switch (`Dock.tsx:204`) and every mode switch +(`ConsoleApp.tsx:268`). `presentation` keeps that behaviour. `level` **must not** reset — a persisted +stance that resets when you change instrument is not a stance. This is the concrete reason level +lives in `settings-store` and not in `ConsoleApp` state. + +--- + +## 7. Invariants + +The repo-wide hard constraints (platform-neutral allocation-free `nisps/`, ≤1e-5 native↔WASM parity, +schema changes shipping both codegen outputs, bounded RT-safe worklet comms, dual-core ownership) +are the floor. This item's own: + +1. **A preset never changes network weights.** Concretely: from a fixed starting state, + `applyPreset(p)` and `selectInstrument(p.modeId)` must leave `getWeights()` **bit-identical**. + (Weights *do* change across the reshape — `nisps_ml_reshape` redraws and warm-starts, + `bindings.cpp:397-418` — but the preset must contribute nothing beyond it.) This is the machine + check on §2.1 and belongs in `bun test`. +2. **Seeds load after the reshape or not at all** (§4.2). A preset whose examples vanish is worse + than one with no examples: the dataset counter lies. +3. **The engine's net shape always equals `MF_MODES[modeId].ml`** after any apply — the P5.3 + contract that `tests/e2e/schema-modes.spec.ts` already enforces. +4. **`ctx.datasetCount` equals the engine's real example count** after apply. Same bar as §6.5e: + nothing may render a plausible number it did not get from the core. +5. **No C++, no schema, no codegen, no new WASM export in this item.** If a step appears to need one, + stop and re-spec — it means the scope was wrong. (Corollary: `scripts/parity-check.sh` is + *irrelevant* to this work; see §10.) +6. **Preset application is main-thread config only** — no worklet message, no audio-thread work + beyond the existing `setBackend`. `addExample` is a bounded heap copy per example, capped at 128 + by validation (§3.3). +7. **`level` is persisted and never auto-reset** (§6.4); `presentation` is transient. +8. **`window.__mf` survives** with the same four methods (`ConsoleApp.tsx:331-345`), repointed at + `selectInstrument`. Playwright drives mode switches through it and must keep working *even after + a real picker exists* — the seam is faster and less brittle than clicking a popover. +9. **No "C15" in the bundle** (`tests/e2e/smoke.spec.ts:59`). The Library renders `MFMode.label`; + if the `c15` entry survives §8.5, its label is already "Powerful Synth Engine". + +--- + +## 8. Deletions, with their consumers + +| Delete | Consumers today | Replacement | +|---|---|---| +| **8.1** `backends/presets.ts` per-backend keying: `OutputPreset.backend`, `KEY_PREFIX:` namespaces, `listPresets/savePreset/getPreset/deletePreset/renamePreset(backend, …)` signatures, `OutputPresetRow` | `PresetBar` (`OutputsBackendConfig.tsx:99-208`, rendered `:686`) — the only one | `presets/store.ts` + `Preset`/`OutputOverride`. `PresetBar` becomes a view onto the unified store filtered by the active target, keeping its save/restore/rename/delete affordances. **Note the sequencing:** step 1 of §9 fixes the `cv` bug inside the old file; step 5 deletes the file. That is deliberate — the bug fix is independently valuable if 5c is never adopted. | +| **8.2** `DrawerDepth` as a disclosure flag | `Drawers.tsx` (21 sites, §6.1), `Dock.tsx`, `ConsoleApp.tsx:110/268/642` | `presentation` + `level` (§6.2) | +| **8.3** `OutputMode` / `outputMode` / `OUTPUT_MODES` / `output-mode.ts` names | `types.ts:26,110-111`, `output-mode.ts`, `Dock.tsx`, `Drawers.tsx:553,615`, `ConsoleApp.tsx:136-137,280,755-756,882,904` | `OutputTarget` / `target` / `OUTPUT_TARGETS` / `output-target.ts`. Mechanical rename; typecheck is the proof. Optional — §12.2. | +| **8.4** `ModeOverlay.input` + `MFMode.input` + `ModeInput` (`model.ts:41,141,211-216,229-247`) | `resolveInputMap(settings.inputMap, mode.input)` (`ConsoleApp.tsx:814`) — the only one | Derive from `schema.ui.primary_input` in `modeFromSchema` (`model.ts:246`); fixes the Breakor drift (§1.6.2). Manifold-only modes keep a literal. | +| **8.5** the `c15` placeholder mode (`model.ts:301-313`) | `MF_MODES` only; nothing renders it (§1.1) | Delete. It is a 2-fake-param entry on the default net shape that will become *visible* the moment a picker exists — the last survivor of the decorative stratum Phase 1 swept. Keep only if the operator wants roadmap signalling in the Library (§12.4). | + +Not deleted, and why: `MFMode.glyph`/`cls`/`badge`/`placeholder` (§1.1) get their first real +consumer here. `visualizer` stays — it works, and its removal is a product call unrelated to this +item. + +--- + +## 9. Sequenced implementation plan + +Each step is independently landable and independently verifiable. Steps 1–2 are worth doing even if +the operator rejects everything after them. + +**Step 1 — the `cv` preset bug** (§1.3). Three additions to `manifold/src/backends/presets.ts`: +`cv?: MFParam['cv']` in `OutputPresetRow` (`:16-29`), `cv: p.cv` in `rowsFromParams` (`:72-86`), +`cv: r.cv ?? p.cv` in `applyPreset` (`:141-162`). +*Verify:* a new `manifold/tests/presets.test.ts` case — build params with `cv` specs, round-trip +through `rowsFromParams`/`applyPreset`, assert channel + threshold survive. Runs in `bun run test`. +Red before, green after. + +**Step 2 — schema-truth input kind** (§8.4, §1.6.2). Derive `MFMode.input` from +`schema.ui.primary_input` in `modeFromSchema`; drop `input` from `ModeOverlay`. +*Verify:* `bun run typecheck`; a unit test asserting `MF_MODES.find(m => m.id === 'breakor')!.input` +now follows the schema. Note this **changes visible behaviour** — Breakor's input surface flips from +circular to rectangular. That is the bug being fixed, and it belongs in the commit message. + +**Step 3 — collapse mode switching into `selectInstrument`** (§4.2). Replace the three effects at +`ConsoleApp.tsx:241-243`, `:252-257`, `:260-271` with one function plus one boot effect keyed +`[engine]`. Repoint `window.__mf.setMode`. +*Verify:* this is the step with real regression risk and it already has coverage — +`tests/e2e/schema-modes.spec.ts` (four modes: dims, weight count, output arity, UI param count, +training after switch), `tests/e2e/reshape.spec.ts`, `tests/e2e/spine.spec.ts` (probe survives a +mode switch). Run the full Playwright suite, not just the smoke. On the VPS use the non-snap node +runner (`ONBOARDING.md` §2). **No new test proves the boot path** beyond the suite's own boot — if +this step misbehaves it will most likely be a mode switch that no longer reshapes, which +`schema-modes` catches, or a double-apply on boot, which nothing catches; add an explicit +"reshape called exactly once on boot" assertion via a probe counter if that risk is judged real. + +**Step 4 — preset model + store + built-in catalogue** (§3). `presets/types.ts`, `presets/store.ts`, +`presets/builtin/`. Author **two or three** curated presets only — enough to exercise the model +(one config-only, one with `rest: 'off'` plus a handful of overrides, one with seeds). Authoring the +full curated set is a separate, per-mode pass — and note that pass is **not** governed by §7.5: the +20 anonymous "Param NN" slots (`ALIGNMENT.md` deferred debt; `schemas/modes/paf_synth.json` params +`p00`…) can only be named by editing the schemas, which means codegen and both generated outputs in +that change. Keep it out of this item. +*Verify:* `manifold/tests/presets.test.ts` — `validatePreset` over every `BUILTIN_PRESETS` entry +against `ALL_MODE_SCHEMAS`; store round-trip through a localStorage fake. `bun run test`. + +**Step 5 — `applyPreset` + the weights invariant** (§4, §7.1). `presets/apply.ts` + `ConsoleApp` +wiring; retire the per-backend preset store (§8.1) and repoint `PresetBar`. +*Verify:* two parts, and neither alone is sufficient — say so in the commit rather than claiming +§7.1 is fully proven. +(a) **Unit, fake applier**: assert the call ORDER (seeds strictly after `selectInstrument`; routing +after the mode rebuild) and assert that no weight-mutating call — `setWeights`, `randomise`, +`train`, `trainAsync` — is made at all. This is the direct machine check on "configuration only". +(b) **C-ABI, real WASM**: in the style of `manifold/tests/loss-history.test.ts` (which drives the +committed `public/nisps.{js,wasm}` under `bun test` via `cwrap` — note it does NOT go through +`WasmIML`/`EngineApi`, which need a browser-ish environment), assert that +`nisps_ml_reshape` followed by N × `nisps_ml_add_example` leaves `nisps_ml_get_weights` bit-identical +to `nisps_ml_reshape` alone. That is the core-side half of the invariant. +**Residual gap:** nothing asserts the invariant end-to-end through the real `EngineApi` under +`bun test`. Step 6's e2e covers it indirectly (dims + dataset count after an apply). Closing it +properly would mean making `WasmIML` constructible headlessly, which is a bigger job than this +item. + +**Step 6 — the Library** (§5). `console/InstrumentPicker.tsx` + the rail changes in `Dock.tsx`. +*Depends on* §6.5b (`plans/browser-mode-coverage-spec.md`) if its `modeSupport()` is to be rendered +(§5.4); buildable without it, at the cost of listing modes that cannot run. +*Verify:* a new `tests/e2e/library.spec.ts` — open the Library, apply a preset, assert (a) the +engine reshaped to that preset's mode dims via `window.__nisps.describe()`, (b) the rendered param +count matches the schema via `window.__mf.paramCount()`, (c) `datasetCount` equals the preset's seed +count, (d) no console errors. Plus `bun run test:e2e` for the smoke's no-"C15" assertion. + +**Step 7 — presentation/level split** (§6). Rename `DrawerDepth`, add `levels` to `settings-store`, +re-gate all 21 sites per §6.3. +*Verify:* `tests/e2e/training-health.spec.ts` currently asserts the panel appears at `expanded` +depth — it must be updated to `advanced` level in the same change, and it is the direct proof the +mechanism swap preserved the §6.5e behaviour. Add one case: level persists across reload (the e2e +helper clears localStorage per test, so this needs an explicit in-test reload rather than a fresh +`loadProbe`). `bun run typecheck` catches the rename fallout. + +**Step 8 — docs.** `manifold/ONBOARDING.md` (§4 drawer/dock description, the "`depth === 'expanded'` +is the advanced-surface flag" rule, the five-vs-six output modes error of §1.6.1), `MAP.md` +(`manifold/src/presets/`), `ALIGNMENT.md` defect 2, `dock-spec.md` (a grounding note recording the +depth model this replaces), `manifold-parity-features-spec.md` §1 (superseded pointer). Same commit +as the code that makes each true. + +**Step 9 (optional) — `OutputTarget` rename** (§8.3). Pure mechanical; `bun run typecheck` is the +whole verification. Do it first or last, never in the middle of a behavioural step. + +--- + +## 10. What the gates prove, and what they do not + +- `bun run typecheck` / `bun run test` / `bun run test:e2e` are the real gates here. Baseline on + 2026-07-21 before any of this work: typecheck clean, `bun test src tests/*.test.ts` → **23 pass, + 0 fail, 6 files**. +- **`scripts/parity-check.sh` proves nothing about this item, in either direction.** It exercises + PAFSynth and ChannelStrip from an all-params-0.5 baseline; this item touches no C++ and no + schema, so a PASS is not evidence of correctness and could not be evidence of a regression. Do + not cite it. Do not skip it either — CI runs it regardless, and a red one means something else + broke. +- `bash scripts/build-cpp-tests.sh` and `scripts/lint-cpp.sh` likewise have no bearing; running them + is cheap insurance against having touched C++ by accident, which §7.5 forbids. +- **The weakest verification in this plan is step 3** (mode-switch refactor). It is covered by e2e, + which is slow, environment-sensitive on the VPS, and asserts outcomes rather than call counts. If + the operator wants stronger evidence, the cheap addition is a probe-visible reshape counter. +- **Nothing here needs hardware.** The one place hardware would matter — a preset reaching the + MEMLNaut — is explicitly out of scope (§0, §11.Q4). + +--- + +## 11. Open questions for the operator + +**Q1. Do you have saved output presets in the live app that must survive?** +§8.1 retires the `manifold-output-presets:` localStorage namespaces. If you have presets +saved at `meml.lnfinitemonkeys.org/next/` you care about, the answer is a one-off import shim +(~20 lines, deleted after one run); if not, they are simply orphaned and the code is cleaner. I +cannot check your browser storage. + +**Q2. May a curated preset opt into one training pass over its seeds?** +§2.1 says the network ships untrained, so the default is no. But a preset with seeds and no training +sounds like an untrained net until the user presses Train — which may be exactly the intent +(the seeds are a *starting lesson*, not a *sound*), or may be a footgun. A `seed.autotrain?: boolean` +is three lines. Default proposed: absent, i.e. never. + +**Q3. Should voice space be part of a preset?** +Every schema declares `voice_spaces` and `ui.show_voice_space_selector`, `nisps/modes/base.hpp:279` +implements selection, and **the browser cannot reach it** — there is no WASM export, and the audio +engine lives in the worklet behind raw exports (`worklet/nisps-processor.ts:33-36`), so this costs +the full 5-layer export chain *plus* worklet + engine-host message plumbing. It is a real +per-instrument character control that curation would obviously want. Out of scope as specified; say +if it should be pulled in, because it changes this item from "no C++" to "C++ and a new export". + +**Q4. Where does curation ultimately live — browser only, or shared with firmware?** +§3.2 chooses browser-local TS data with a stated promotion condition (the hardware editor, §6.5d). +If you expect to author a preset in Manifold and send it to the MEMLNaut, say so now: the format +should then be JSON under `schemas/` from the start, because retrofitting codegen later means +rewriting the catalogue. + +**Q5. URL sharing?** `manifold-parity-features-spec.md` §1.3 specified `?session=` and a +compact axis form. Deferred here (nothing depends on it, and the axis form's control surface does +not exist). Worth scheduling, or drop it from the corpus? + +--- + +## 12. Decisions I made (not questions) + +Recorded so they can be reversed knowingly rather than rediscovered. + +1. **The picker *is* the preset browser** (§2.3), one surface not two. Rationale: vision 3 wants the + curated path to be the default path; a separate mode list beside a preset list would make the raw + instrument the front door again. +2. **`OutputMode` → `OutputTarget`** (§8.3) is proposed but optional and sequenced last. It is pure + naming hygiene for a collision that only becomes painful once two "Mode" controls sit adjacent on + the rail. +3. **Explanatory prose shows at `basic` and hides at `advanced`** (§6.3), inverting today's + accidental behaviour where beginners' help is behind a modal. If you dislike it, the fallback is + one line: show prose unconditionally. +4. **Delete the `c15` placeholder mode** (§8.5), because a picker makes it visible and it does + nothing. Trivially reversible by keeping the entry and rendering its `badge: 'soon'` disabled. +5. **Two disclosure levels, not three** (§6.2) — `dock-spec.md` §0's `peek` never shipped and there + is exactly one tier of advanced content in the tree. +6. **Index-keyed output overrides** (§3.1), because `MFParam.name` is user-editable while schema + param order is codegen-stable. +7. **Two or three built-in presets in step 4, not a full catalogue** (§9). Authoring curated presets + per instrument is real design work that wants ears and hardware; the model must be proven first. diff --git a/docs/specs/plans/hardware-editor-spec.md b/docs/specs/plans/hardware-editor-spec.md new file mode 100644 index 0000000..4a05c72 --- /dev/null +++ b/docs/specs/plans/hardware-editor-spec.md @@ -0,0 +1,465 @@ +--- +kind: plan +status: active +--- + +# Manifold as Hardware Editor — MEMLNaut USB-Serial Protocol, Firmware Command Surface, On-Device Persistence + +*Dated 2026-07-21. Spec for plan item **§6.5d** (`simplification-plan.md:130`), audit findings **A4** / **S14** / +**L29**, ALIGNMENT **defect 3** (and the live half of **defect 5**). Vision bullet 5: "Manifold doubles as +interface/editor for the hardware MEMLNaut (settings, presets, training, examples, visualisation)."* + +**Nothing here is implemented. No code was written for this document.** Every `file:line` below was read from +the working tree on 2026-07-21 and is marked ✓ (verified by reading the file) or ✗ (could not verify — stated +as unknown, never asserted). The tree was mid-flight with concurrent edits from other sessions +(`git status` showed 20+ modified files including `src/main.cpp`, `codegen/generate.ts`, `ConsoleApp.tsx`); +re-check citations in those files before relying on an exact line number. + +--- + +## 1. Ground truth — what exists today + +### 1.1 The browser end is a connect-lifecycle shell + +| Fact | Evidence | +|---|---| +| `MemlnautSerial` handles `requestPort`/`open`/`close` and a 5-state status store, nothing else | ✓ `manifold/src/serial/memlnaut-serial.ts` (140 lines, not the 237 the audit reports — stale) | +| `saveModel` returns `false`, `restoreModel` returns `null`, `getSettings` returns `{}`, each with a `TODO: real protocol` | ✓ same file, the three methods under `// ---- Protocol stubs ----` | +| Baud 115200 is explicitly a placeholder | ✓ same file, `TODO(memlnaut-serial): negotiate the real baud rate` | +| The only consumer is `EditorPanel.tsx`; the only consumer of *that* is the Settings-drawer `editor` case | ✓ `manifold/src/console/Drawers.tsx:33` (import), `:579` (render). Repo-wide grep for `memlnaut-serial`/`getMemlnautSerial` finds no other importer. | +| `editor` is a top-level dock **output mode**, `audio: false`, description already promises "configure / save / restore" | ✓ `manifold/src/console/output-mode.ts:69-75` | +| The panel already carries an honest "not yet wired" note | ✓ `EditorPanel.tsx`, final `

` | + +`manifold/src/serial/web-serial.d.ts` is an **ambient** (global) declaration file. Its header comment claims it +declares "just the surface memlnaut-serial.ts uses" — false: `manifold/src/backends/cv-backend.ts:56,95,123,133` +type-checks against the same global `SerialPort`/`navigator.serial`. There is one Web Serial *type* surface and +two *transport* implementations, not three (✓ grep for `navigator.serial|SerialPort` over `manifold/src` returns +exactly `web-serial.d.ts`, `cv-backend.ts`, `memlnaut-serial.ts`). + +### 1.2 The firmware end has a USB CDC port, and nothing listening on it + +| Fact | Evidence | +|---|---| +| `Serial` (USB CDC) is opened at boot and used **only** for diagnostics | ✓ `firmware/MEMLNaut-NISPS/src/main.cpp:125` (`Serial.begin(115200)`) | +| Core 0's `loop()` prints `"."` ~10×/s and a perf line ~1×/s onto that same stream | ✓ `src/main.cpp:166`, `:172` (`Serial.printf("ml: %d, aud: %d, q: %f\n", …)`) | +| **The USB CDC port is otherwise free**: MIDI runs on `Serial2` (UART1), not USB | ✓ `lib/memllib/src/interface/MIDIInOut.cpp:15` `MIDI_CREATE_CUSTOM_INSTANCE(HardwareSerial, Serial2, MIDI, …)`, `:81-88` | +| USB-MIDI (TinyUSB composite) is compiled out — `MIDI_USB_CLIENT` is defined nowhere | ✓ `MIDIInOut.cpp:55` is `#ifdef MIDI_USB_CLIENT`; grep across `platformio.ini`, `src/`, `glue/` finds no definition. The device enumerates as a plain CDC port. | +| There is no serial *reader* anywhere in firmware — no `Serial.read()`, no parser, no command dispatch | ✓ grep over `src/` + `glue/` | +| Core 0's control cadence is 5 ms (`ML_INFERENCE_PERIOD_US 5000`), core 1's drain 1 ms | ✓ `src/main.cpp:152-160`, `:208-216` | + +### 1.3 There is no on-device persistence, but two mechanisms are already compiled in + +- **LittleFS** (internal flash). memllib wraps it in `lib/memllib/src/hardware/FlashFS.hpp` ✓ — which has **zero + consumers** in this repo (✓ grep for `FlashFS` hits only vendored TFT_eSPI example sketches under `.pio/`). + Two problems with that wrapper: it defines `void begin()` and `bool exists()` as **non-`inline` free functions + in a header** (✓ read in full — 22 lines), so including it from two translation units is a duplicate-symbol + link error; and using it at all buys nothing over calling `LittleFS` directly. +- **The filesystem is currently sized to zero.** `platformio.ini` does not set `board_build.filesystem_size` + (✓ grep), and the platform's default is `"0MB"` (✓ `~/.platformio/platforms/raspberrypi/builder/main.py:63`). + With `_size == 0`, `LittleFS.begin()` returns `false` immediately (✓ + `~/.platformio/packages/framework-arduinopico/libraries/LittleFS/src/LittleFS.h:173`). **On-device flash + persistence therefore requires a `platformio.ini` change that alters the flash layout of all 16 envs.** +- **LittleFS writes park core 1.** ✓ `…/LittleFS/src/LittleFS.cpp:186-193` and `:203-209`: + `noInterrupts(); rp2040.idleOtherCore(); flash_range_program/flash_range_erase(...); rp2040.resumeOtherCore(); + interrupts();`. Core 1 is the audio core. **Any flash write stalls audio for the duration of the erase + + program.** This is the single hardest constraint in this document. +- **SD card.** memllib ships `interface/SDCard.{hpp,cpp}` (SdFat over hardware SPI) ✓, the MEMLNaut board has + dedicated SD pins ✓ `lib/memllib/src/hardware/memlnaut/Pins.hpp:87-90` (`SD_CS 13`, `SD_SCK 14`, `SD_MISO 12`, + `SD_MOSI 15`), and `SDCard.cpp` **already compiles into every variant** ✓ (`.pio/build/pafsynth/…/interface/ + SDCard.cpp.o` exists). SPI writes do **not** lock out core 1. It needs a physical card fitted; whether the + operator's units have one is ✗ unknown. + +### 1.4 The model *is* transferable — the shapes already agree + +- Firmware instantiates a compile-time `MLP<…>` whose dims come from the mode's generated schema; the browser + reshapes its `MLPCore` to **the same schema dims** on every mode switch ✓ + `manifold/src/console/ConsoleApp.tsx:252-257` (`engine.reshape({ inputSize, outputSize, hidden }, defaultSpread)` + from `mode.ml`). So for a given `mode_id` the two nets have the *same* `weight_count` by construction. +- **But not unconditionally.** The browser also offers an axis-count reshape that changes `inputSize` alone ✓ + `ConsoleApp.tsx:290-322`, and `nisps_ml_create`'s default shape is a 32-input head ✓ + `manifold/src/inputs/input-layer.ts:17,38`. A live browser net can therefore be at `6→[10,10,14]→33` while the + device is at `4→[10,10,14]→33`. **The protocol must carry the shape and refuse mismatches; it must never infer + compatibility from `mode_id` alone.** +- Flat weight layout is already specified and identical on both targets: `[l0_w][l1_w][l2_w][l3_w][l0_b][l1_b] + [l2_b][l3_b]`, little-endian `float32` ✓ `nisps/wasm/bindings.cpp:25-34` (comment), `nisps/ml/mlp.hpp:385-414` + (`get_weights`/`set_weights`). +- Per-mode blob sizes (computed from `schemas/modes/*.json` ✓): + + | mode | dims | weights+biases | bytes | + |---|---|---|---| + | `sound_analysis_midi` | 10→[10,10,14]→8 | 494 | 1 976 | + | `channel_strip`, `xiasri` | 4→[10,10,14]→24 | 674 | 2 696 | + | `paf_synth` | 4→[10,10,14]→33 | 809 | 3 236 | + | `elysiamorf` | 4→[10,14,18]→40 | 1 234 | 4 936 | + | `verb_fx` | 4→[10,14,18]→47 | 1 367 | 5 468 | + | `breakor`, `memlcelium`, `slp_workshop` | 4→[10,14,18]→56 | 1 538 | **6 152** | + + Worst case is ~6 KB. Small — but not small enough to sit in one frame without either a 6 KB RX buffer or + chunking. + +### 1.5 Training examples are **not** readable out of the core today + +`MLPCore` exposes `example_count()` ✓ `nisps/ml/mlp.hpp:441` and `clear_examples()` ✓ `:443`, but the per-example +accessors `sample_features_(s)` / `sample_labels_(s)` are **private** ✓ `:577`, `:581`. Exporting the training +set from the device is therefore a real (small) addition to `nisps/ml/mlp.hpp`, not a wiring job. + +Worse for the editor: the firmware's `FeedbackController` — which owns explore/place state and drives +`add_example` — is a **function-local static inside `bind_peripherals()`** ✓ +`firmware/MEMLNaut-NISPS/glue/peripherals.hpp:152`. It is unreachable from `main.cpp` or any command handler. +Any command that touches feedback state requires hoisting it first. + +### 1.6 The discipline to copy already exists in-repo + +`firmware/useq-celium/shared/protocol.h` ✓ is a C header of `static const` constants + `static inline` helpers, +mirrored constant-for-constant by `manifold/src/backends/useq-protocol.ts` ✓, pinned by +`manifold/src/backends/useq-protocol.test.ts` ✓ (`test('frame sizes match the C header')`), documented as +`kind: spec` in `docs/specs/useq-cv-protocol.md` ✓. The device-side parser is a ~20-line resync state machine ✓ +`firmware/useq-celium/main/src/main.cpp:93-110`. **Verified live**: `bun test src tests/*.test.ts` in `manifold/` +→ `23 pass, 0 fail, 6 files, 65 ms`. + +### 1.7 L29 — `InputChain`/`OutputChain` have no firmware consumer + +✓ Repo-wide grep: the only non-test, non-self references are `nisps/wasm/bindings.cpp:76-77,194-195,968,1011` +and `tests/cpp/{test_pipeline,parity_check}.cpp`. `nisps/pipeline/output_chain.hpp:13` says "firmware would pick +its mode's NOut" — aspirational, present-tense-adjacent, and currently false. `InputChain` already serialises +(`state_size()`, round-trip test ✓ `tests/cpp/test_pipeline.cpp:131-150`), which matters below. + +--- + +## 2. Invariants + +Beyond the repo-wide hard constraints (platform-neutral allocation-free `nisps/`; native↔WASM parity ≤1e-5; +schema changes ship both codegen outputs; RT-safe worklet + SPSC dual-core discipline), this area adds: + +- **I1 — The wire format has exactly one source of truth.** A single C header under `firmware/`, mirrored in TS, + pinned by a test that fails when they diverge. No constant is written twice by hand in the same language. +- **I2 — No hand-maintained per-mode table anywhere in the protocol.** Mode identity, net shape, param count and + ordering reach the wire from codegen output on both sides, or they do not reach it at all. +- **I3 — A model blob is applied only against a proven-identical contract.** Shape *and* schema fingerprint must + match. On mismatch the device NAKs; it never partially applies, never truncates, never zero-pads. +- **I4 — Serial servicing is bounded.** The core-0 pump reads at most a fixed byte budget per `loop()` iteration. + The 5 ms `tick_control()` cadence ✓ (`src/main.cpp:152,160`) is not to be perturbed by a host that streams. +- **I5 — Serial handling never runs on core 1.** Core 1 is audio + MIDI drain. All parsing, buffering, filesystem + and ML mutation happen on core 0. +- **I6 — Flash writes are an explicit, announced, user-initiated act.** Because they park core 1 (§1.3), a + persist is never automatic, never periodic, and never a side effect of another command. +- **I7 — Diagnostics and protocol do not share the stream unframed.** Either the diagnostics are silenced for the + session or they are carried inside a frame. Never raw ASCII interleaved with binary payloads. +- **I8 — The device is authoritative about itself.** The browser asks and believes; it never assumes a firmware + build from a mode name, a port name, or a USB descriptor. +- **I9 — No compat shim.** The stubs are deleted, not deprecated. `EditorPanel` is the only consumer (§1.1). + +--- + +## 3. Design + +### 3.1 Transport and channel + +USB CDC (`Serial`), 8N1; the CDC baud parameter is ignored by the hardware but the host must supply one — +**115200**, matching what the firmware already calls and what `cv-backend.ts:133` uses. No second UART, no +composite USB device, no TinyUSB reconfiguration. + +**The diagnostics collision (I7) is resolved by session state.** On `HELLO` the firmware sets an +`editor_session` flag; while set, `loop()`'s `Serial.println(".")` and perf `printf` (✓ `src/main.cpp:166,172`) +are suppressed and the same information is emitted as a `TELEMETRY` frame instead. `BYE`, a host disconnect +(DTR drop), or a 5 s silence timeout clears the flag and restores the prints. Boot-time prints +(`"Serial initialised."`, `"Bound peripherals to mode."`) happen before any session and are harmless — the +host-side parser drops non-sync bytes anyway. + +*Rejected:* moving diagnostics to a second UART (costs pins and a second cable) and CDC-composite dual ports +(complicates the Web Serial port picker for one debug convenience). + +### 3.2 Framing + +Variable-length, because payloads range from 0 bytes to a 6 KB model (§1.4). Fixed-length-per-type (the useq +choice) does not survive that. + +``` + byte 0 sync 0xA5 host→device, 0x5A device→host + byte 1 type u8 + bytes 2..3 length u16 LE, payload bytes, 0..MEMLED_MAX_PAYLOAD + bytes 4.. payload + last 2 crc u16 LE, CRC-16/CCITT-FALSE over bytes [1 .. 3+length] +``` + +- **CRC-16, not XOR-8.** useq's XOR-8 is right for a 26-byte frame streamed at 100 Hz where a corrupt frame is + discarded and replaced 10 ms later. It is wrong for a 6 KB model transfer, where a 1-in-256 undetected + corruption silently installs wrong weights and the user hears an inexplicable instrument. Bitwise CRC-16 is + ~15 lines, table-free, and identical on both sides. +- **`MEMLED_MAX_PAYLOAD = 512`.** Bounds the device RX buffer at 518 bytes and each frame's service time at + ~45 ms of wire time worst case — hence I4's byte budget. Larger objects chunk (§3.4). +- Both directions use the same frame shape; only the sync byte differs, so one parser implementation serves + both ends. + +Home: **`firmware/shared/memlnaut-editor-protocol.h`** — deliberately *not* under `firmware/MEMLNaut-NISPS/`, +because it is shared with the browser exactly as `firmware/useq-celium/shared/protocol.h` is. Mirror: +**`manifold/src/serial/memlnaut-protocol.ts`**. Parity test: **`manifold/src/serial/memlnaut-protocol.test.ts`** +(picked up by the existing `bun test src tests/*.test.ts` glob — do not touch that script). + +### 3.3 Identity: a codegen-emitted schema fingerprint (discharges I2, I3) + +The one genuinely new idea in this spec, and the thing that makes "derive from schema codegen, not hand-defined +tables" concrete. + +`codegen/generate.ts` gains one emitted constant per mode: a **FNV-1a 64 hash over a canonical serialisation of +the mode's contract-bearing schema fields** — `mode_id`, `engine_id`, `ml` (`input_size`, `hidden_layers`, +`output_size`, `default_spread`), `params[]` as `(name, min, max, default, curve, group)` in order, and +`voice_spaces` (names + curve overrides) in order. Deliberately **excluded**: `label`, `_note`, `ui` — cosmetic, +and a label edit must not invalidate a user's trained model. + +Emitted into both generated worlds in the same change (hard constraint): + +- `nisps/modes/generated/_schema.hpp` → `inline constexpr std::uint64_t kSchemaHash = 0x…ull;` + alongside the existing `kModeId` ✓ (`nisps/modes/generated/paf_synth_schema.hpp:10`) +- `manifold/src/modes/generated/_schema.ts` → `schema_hash: '0x…'` on the existing `ModeSchema` const + +CI already fails on a stale generated tree (`.github/workflows/ci.yml`, step *"Codegen is committed and +idempotent"* ✓ `:129`), and `codegen/tests/golden/paf_synth_schema.{hpp,ts}` ✓ will need regenerating in the +same commit. + +The device reports its hash in `DESCRIBE`; the browser compares against **its own** generated hash for that +mode. Equal ⇒ weights, settings and examples are interchangeable. Unequal ⇒ the UI says *"this MEMLNaut is +running a different build of `paf_synth`"* and disables transfer. No table, no version negotiation matrix, no +per-mode special case — and the check is exact rather than heuristic. + +### 3.4 Command set (protocol v1) + +Types are `static const uint8_t` in the shared header. `ACK`/`NAK` are the universal replies; `NAK` carries a +one-byte reason code from an enum in the same header. + +**Host → device** + +| Type | Payload | Reply | +|---|---|---| +| `HELLO` | `u16 host_proto_ver` | `DESCRIBE` — opens the session, silences diagnostics | +| `BYE` | — | `ACK` — closes the session, restores diagnostics | +| `GET_MODEL` | — | `MODEL_BEGIN`, then `MODEL_CHUNK`×N | +| `PUT_MODEL_BEGIN` | `u64 schema_hash, u16 dims[5], u32 weight_count, u32 crc32` | `ACK`/`NAK` | +| `PUT_MODEL_CHUNK` | `u16 index, bytes` | `ACK` | +| `PUT_MODEL_END` | — | `ACK` (applied) / `NAK` (CRC or count mismatch — **nothing applied**) | +| `GET_SETTINGS` | — | `SETTINGS` | +| `SET_SETTING` | `u16 id, f32 value` | `ACK`/`NAK` | +| `PERSIST` | `u8 slot` | `ACK`/`NAK` — see I6 | +| `LOAD_PERSISTED` | `u8 slot` | `ACK`/`NAK` | +| `CLEAR_EXAMPLES` | — | `ACK` | +| `SUBSCRIBE` | `u16 mask, u8 rate_hz` | `ACK` — telemetry stream on/off | + +**Device → host** + +`DESCRIBE`, `MODEL_BEGIN`, `MODEL_CHUNK`, `SETTINGS`, `TELEMETRY`, `LOG` (UTF-8 text, framed — the session +replacement for the raw prints), `ACK`, `NAK`. + +`DESCRIBE` payload: `u16 proto_ver`, `u64 schema_hash`, `mode_id` (fixed 24-byte NUL-padded ASCII, from +`generated::kModeId` ✓), `u16 dims[5]`, `u32 weight_count`, `u16 max_examples`, `u16 capability_bits`, +`char fw_build[16]`. The capability bits are what let one protocol serve 16 firmware variants without a +per-variant table: `HAS_PERSIST`, `HAS_SD`, `HAS_EXAMPLE_EXPORT`, `HAS_TELEMETRY`, `IS_SEQUENCER`. + +**PUT_MODEL is atomic (I3).** Chunks land in a staging buffer sized `weight_count * 4` — for the largest mode +6 152 bytes of core-0 RAM, allocated as a `static` array sized from `MLPType::weight_count()` at compile time, +so no heap. `set_weights` is called once, from `PUT_MODEL_END`, only after the CRC-32 over the whole blob +matches and `weight_count` and `dims[5]` equal the device's own. + +**Settings (I2, honestly).** Not everything can come from codegen, and pretending otherwise would be the +hand-defined table under a different name. The split: + +- *Derived from codegen, on both sides:* the training triple `learning_rate` / `max_iterations` / `min_error` + (defaults + identity from `schemas/ml_defaults.json` → `nisps::ml::generated::kMlTrainDefaults` ✓ and + `manifold/src/modes/generated/ml_defaults.ts` ✓ — both mirrors already exist), and `voice_space_index` whose + legal range is the generated `voice_spaces` list ✓. +- *Genuinely device-side, enumerated in the shared header:* `pin_value`, `joystick_single` (the Dual/Single + toggle ✓ `glue/settings_view.hpp:54-58`), `explore_intensity` (RVX1 ✓ `glue/peripherals.hpp:124-126`), + `master_volume` (RVGain1 ✓ `:115-117` — note it drives `AudioDriver::SetMasterVolume` directly, not the mode). These are `ModeBase`/peripheral concepts with no schema home; there + are four of them; they live in one enum in the shared header and nowhere else. + +**Telemetry closes ALIGNMENT defect 5's live half.** The `TELEMETRY` payload carries `PERF_GET_MEAN(MLSTATS)` +and `AUDIOLOOP_MEAN` — numbers the firmware **already computes and prints** ✓ `src/main.cpp:172` — plus +`example_count()`, `eval_loss()`, and (paged) the loss history the operator deliberately kept on device +(audit L25). That is the on-device timing report ALIGNMENT defect 5 asks for, at near-zero marginal cost, +displayed in a browser panel that already knows how to draw a loss curve (`console/TrainingHealth.tsx` ✓). + +### 3.5 Persistence + +**Primary: LittleFS on internal flash**, because it needs no accessory and every unit has it. + +Required changes: `board_build.filesystem_size` in `platformio.ini`'s `[env]` block (§1.3 — currently absent, +default `0MB`, `begin()` fails), a `glue/persistence.hpp` that calls `LittleFS` **directly** (not memllib's +`FlashFS.hpp`, whose non-`inline` header functions are a duplicate-symbol trap and which adds nothing — §1.3), +and a fixed on-disk record: + +``` +/memlnaut//.mdl magic, u16 record_ver, u64 schema_hash, u16 dims[5], + u32 weight_count, f32 weights[], settings block, u32 crc32 +``` + +`schema_hash` on disk is checked on load exactly as on the wire (I3), so a firmware reflash that changes the +schema invalidates old saves loudly instead of loading garbage. + +**The audio stall is real and must be surfaced, not hidden.** Sequence for `PERSIST`: mute via +`AudioDriver::SetMasterVolume(0)`, emit `LOG "saving…"`, write, restore volume, `ACK`. The browser shows a +determinate "Saving to device — audio will pause" state. The alternative — pretending a flash erase is free — +would be exactly the kind of plausible-looking lie this repo has been deleting all week. + +**Not chosen, but noted:** the SD path (`interface/SDCard.hpp` ✓, already compiled ✓, pins ✓) does not park +core 1 and is the right home for bulk artefacts (full example sets, session recordings) if the operator's units +have cards fitted. Left behind the `HAS_SD` capability bit rather than built speculatively. + +### 3.6 Browser side + +`manifold/src/serial/memlnaut-serial.ts` is **rewritten, not extended**: the three stub methods and their +`TODO`s are deleted (I9 — sole consumer is `EditorPanel.tsx`, §1.1). The class becomes a framed-transport +driver: a read loop over `port.readable`, a resync parser identical in shape to the TS mirror's encoder, a +promise-keyed request/reply map, and the same `subscribe()` store it already has (which `EditorPanel` already +consumes via `useSyncExternalStore` ✓). + +`EditorPanel.tsx` grows: a device card (mode, build, dims, compatibility verdict), Save-to-device / +Load-from-device with the compatibility gate and a progress state, a settings section, and a live telemetry +strip. **The "not yet wired" note is deleted in the same commit as the code that wires it** — a note that +outlives its truth is the failure mode this repo keeps finding. + +`output-mode.ts:72`'s description ("configure / save / restore") becomes true and needs no edit. Two small +hygiene items ride along: fix `web-serial.d.ts`'s false header comment (§1.1) and move it to a shared location +now that two modules provably depend on it. + +--- + +## 4. Deletions + +| Deleted | Named consumers | Why safe | +|---|---|---| +| `MemlnautSerial.saveModel` / `.restoreModel` / `.getSettings` bodies + `TODO`s | `EditorPanel.tsx` only (✓ grep) | Replaced in the same change; no external caller | +| `EditorPanel`'s "not yet wired" `

` | none | Becomes false the moment step 6 lands | +| The unconditional `Serial.println(".")` / perf `printf` in `src/main.cpp:166-175` | none (human eyeball on a serial monitor) | Not deleted — made session-conditional. Outside a session behaviour is unchanged. | +| `lib/memllib/src/hardware/FlashFS.hpp` — *not* deleted | none (✓ zero consumers) | Vendored upstream code; deleting it diverges the vendor tree for no gain (`VENDORED.md` re-sync). We simply do not use it, and `persistence.hpp` says why. | + +Nothing else. In particular the `cvgate` backend's Web Serial code stays untouched — different device, different +protocol, no shared transport worth extracting for two implementations. + +--- + +## 5. Sequenced implementation plan + +Each step is independently landable and independently verifiable. Steps 1–4 are agent-safe with the existing +gates. Steps 5–8 need hardware and are honestly marked. + +**Step 0 — L29 disposition (do first, it is one line either way).** +`nisps/pipeline/output_chain.hpp:13` and `input_chain.hpp`'s header currently imply a firmware consumer that does +not exist (§1.7). If the operator answers **Q4** (below) with "not now", soften both comments to say the chains +are browser-side today, in this step. If "yes", they stay and step 8 fulfils them. Either way L29 stops being an +open lie after step 0. +*Verification:* `bash scripts/lint-cpp.sh` (comment-only change; the lint strips comments before matching ✓). + +**Step 1 — the shared header + TS mirror + parity test. No behaviour anywhere.** +`firmware/shared/memlnaut-editor-protocol.h`, `manifold/src/serial/memlnaut-protocol.ts`, +`manifold/src/serial/memlnaut-protocol.test.ts`. Encoder/decoder + CRC-16 in both languages; the test asserts +frame offsets, `MEMLED_MAX_PAYLOAD`, every type byte, and CRC-16 against fixed vectors. +*Verification:* `cd manifold && bun run typecheck && bun run test` (the new test is picked up by the existing +glob — **do not change the test script**). Additionally register a host C++ test that includes the header and +asserts the same vectors: add a source to `nisps_modes_tests`, which already puts the repo root on its include +path for exactly this reason ✓ (`nisps/CMakeLists.txt:134`). Then `bash scripts/build-cpp-tests.sh` proves the C +and TS encoders agree on the same vectors on both sides. +*Note:* `scripts/parity-check.sh` says nothing about any of this — it exercises PAFSynth and ChannelStrip at +all-params-0.5. A green parity run is not evidence for this step. + +**Step 2 — codegen emits the schema fingerprint.** +`codegen/generate.ts` + `codegen/lib.ts`; C++ and TS in the same commit; `codegen/tests/golden/*` regenerated. +*Verification:* `cd codegen && bun run generate.ts` twice (idempotence), `bun run test` (golden + curve-drift), +then `bash scripts/build-cpp-tests.sh` (the generated headers are compiled by `nisps_modes_tests` ✓). CI's +"Codegen is committed and idempotent" step ✓ is the backstop. + +**Step 3 — firmware command surface, read-only commands.** +`glue/editor_serial.hpp`: the bounded pump (I4), the parser, `HELLO`/`DESCRIBE`/`GET_SETTINGS`/`GET_MODEL`/ +`SUBSCRIBE`/`BYE`, the session flag gating the diagnostics. Called from `loop()` on core 0 next to +`tick_control()`. Deliberately no mutation yet. +*Verification (weak, be honest):* `pio run -e slpworkshop -e pafsynth -e selftest` compiles and the flash/RAM +delta is reported (CI's firmware job already does this ✓ `.github/workflows/ci.yml:213,223`). Compilation is +**all** that is verified without hardware. The parser itself should be structured so its byte-level core is +Arduino-free and host-testable — the precedent is `glue/codec_config.hpp`, deliberately Arduino-free and covered +by `tests/cpp/test_mode_driver_config.cpp` ✓. Follow it: put the state machine in the shared header or a pure +sibling, and the `Serial` reads in the glue. + +**Step 4 — browser transport + read-only editor UI.** +Rewrite `memlnaut-serial.ts` onto the framed protocol; `EditorPanel` renders `DESCRIBE` + telemetry + the +compatibility verdict. Save/Load still absent. +*Verification:* `bun run typecheck && bun run test && bun run build`. A Playwright e2e can cover the panel's +**disconnected** rendering only — Web Serial cannot be driven headlessly. Do not claim more. The frame codec is +covered by step 1's unit test; the *round trip* is not covered by anything until hardware. + +**Step 5 — firmware mutating commands.** +`PUT_MODEL_*` (staging buffer + atomic apply), `SET_SETTING`, `CLEAR_EXAMPLES`. Requires hoisting the +`FeedbackController` out of `bind_peripherals`'s function-local static (§1.5) if any command touches feedback +state; if v1 avoids feedback entirely, say so and leave it. +*Verification:* compile + size. The atomicity logic (CRC-32 accumulate, count check, single `set_weights`) is +host-testable if written as a pure function over spans — do that, and cover it in `nisps_modes_tests`. + +**Step 6 — Save-to-device / Load-from-device in the UI**, gated on the schema hash + dims match, with progress +and the explicit incompatibility message. Delete the "not yet wired" note here. +*Verification:* typecheck + unit + build; e2e for the **gate** (a fake `DESCRIBE` with a wrong hash must disable +the buttons) is worth writing because it is pure logic. + +**Step 7 — persistence.** +`board_build.filesystem_size` in `platformio.ini` (**changes the flash layout of all 16 envs — rebuild and +re-report sizes**), `glue/persistence.hpp` on `LittleFS` directly, the record format, `PERSIST`/`LOAD_PERSISTED`, +the mute-around-write sequence. +*Verification:* all 16 envs build and the size table is re-baselined (`scripts/build-firmware.sh --all`). The +record encode/decode is pure and host-testable; the flash behaviour, the stall duration, and whether the audio +interruption is acceptable are **operator chokepoints — no automated gate reaches them.** + +**Step 8 — (conditional on Q4) `InputChain`/`OutputChain` firmware wiring.** +`InputChain` in `bind_peripherals` between the joystick callbacks and `mode.set_input`; `OutputChain` in `ModeBase::tick_control` before `engine_.set_params`. Both configs become editor settings, which is +the only reason firmware would want them configurable at all. Note the cost honestly: `OutputChain` adds an +`NMax`-float state array to every mode instance and a per-tick pass at 200 Hz. +*Verification:* host tests already exist for both chains ✓ (`tests/cpp/test_pipeline.cpp`) and parity stage 7 +covers them ✓ (`tests/cpp/parity_check.cpp:341-394`) — but **neither proves the firmware wiring**, and the thing +that actually changes (how the joystick *feels*) can only be judged by ear on hardware. + +--- + +## 6. Open questions — operator only + +1. **Persistence medium and the audio stall.** Internal flash (always present, but every save parks core 1 and + interrupts audio for the erase+program — §1.3) or SD (no stall, but needs a card fitted, and it is ✗ unknown + whether your units have one)? If flash: is a brief, announced audio interruption on an explicit save + acceptable, or does that rule flash out? +2. **What "presets" means for the device.** Vision bullet 5 says *settings, presets, training, examples, + visualisation*. This spec covers settings, model transfer, telemetry, and persistence slots. It does **not** + define a preset object — and §6.5c (curated/advanced split) is supposed to define exactly that, and is + scheduled *before* this item. Does the editor wait for 5c's preset model, or ship slots-of-weights now and + absorb presets later? +3. **Do examples need to leave the device?** Exporting the training set requires a new public accessor on + `MLPCore` (§1.5) and probably hoisting the `FeedbackController` (§1.5). "Train on device, curate in browser" + implies yes; "save/restore a model" implies no. This is the difference between a small v1 and a much larger + one. +4. **L29 / step 8: should firmware gain the browser's input pipeline?** Deadzone, circular clamp, momentum zoom + and EMA would change the feel of every hardware mode, and only your ears can judge it. If the answer is "not + now", step 0 softens the comments and the question closes cleanly. +5. **Whose model wins on connect?** When the browser connects and both ends hold a trained net for the same + mode, does the editor do nothing until told (proposed default), or offer/auto-pull the device's? + +## 7. Decisions taken here (not questions) + +- CRC-16 for frames, CRC-32 for whole-model transfers, not useq's XOR-8 — §3.2. +- Variable-length frames with a 512-byte cap and chunked models, not one large frame — §3.2, I4. +- Diagnostics silenced by session flag rather than moved to another port — §3.1, I7. +- Compatibility by codegen-emitted schema fingerprint + explicit dims, never by `mode_id` alone — §3.3, I3. +- `LittleFS` called directly; memllib's `FlashFS.hpp` not used and not deleted — §3.5, §4. +- The four device-side settings live in an enum in the shared header, and the spec says plainly that they are + not codegen-derived rather than pretending — §3.4. +- Protocol header at `firmware/shared/`, not inside `firmware/MEMLNaut-NISPS/`, mirroring useq-celium. + +## 8. Where verification is weak + +Say this out loud in any PR description that lands these steps: + +- **`scripts/parity-check.sh` proves nothing here.** It exercises PAFSynth and ChannelStrip with all params at + 0.5. It touches no serial code, no framing, no persistence. +- **Nothing in CI executes firmware.** The firmware job compiles three envs and reports sizes ✓ + (`.github/workflows/ci.yml:213,223`). A protocol that compiles is not a protocol that works. +- **Web Serial cannot be exercised headlessly.** Playwright can cover the panel's disconnected and + incompatible-device states, and the frame codec is unit-tested on both sides — the *round trip over a real + cable* is untested until someone plugs a MEMLNaut in. +- **Mitigation worth building early:** a tiny host-side loopback harness — the TS encoder feeding the C parser + compiled natively into `nisps_modes_tests` — turns "the two ends agree about bytes" from a hope into a gate, + and costs about an hour. It still says nothing about USB CDC, flash timing, or feel. +- **Step 7 has no automated gate at all** beyond "16 envs still build". Flash layout, stall duration and + recovery-after-power-loss are hardware chokepoints. diff --git a/docs/specs/plans/mode-layer-reunification.md b/docs/specs/plans/mode-layer-reunification.md new file mode 100644 index 0000000..8247f76 --- /dev/null +++ b/docs/specs/plans/mode-layer-reunification.md @@ -0,0 +1,665 @@ +--- +kind: plan +status: active +--- + +# Mode-Layer Reunification — the shared control tick (P5 item 5a) + +*Dated 2026-07-21. Spec-first, per `simplification-plan.md` §6 ("each item spec-first, own +session"). Closes `ALIGNMENT.md` defect 1; audit findings A1 + A5 +(`../recon/simplification-audit-2026-07.md`). **This document authorises no code.** It fixes +the seam, the invariants, the sequence and the verification so an implementation session can +start cold. §8 lists the decisions that must come from the operator before step 3 lands.* + +**Every code claim below was re-read against the working tree on 2026-07-21** (`fd0aee2` plus +the uncommitted in-flight work of parallel sessions — `nisps/modes/base.hpp`, +`nisps/wasm/bindings.cpp` and several `manifold/src/engine/` files are all dirty relative to +HEAD, so line numbers are working-tree numbers and may shift by a few lines once those land). +Where a cited document (audit, plan, ALIGNMENT) disagreed with the code, the code won and the +disagreement is called out inline. + +Sibling P5 specs written in the same session: `browser-mode-coverage-spec.md` (5b), +`curated-presets-spec.md` (5c), `hardware-editor-spec.md` (5d). This one is 5a. + +--- + +## 0. The gap, stated precisely + +`nisps/modes/` compiles into exactly one binary: the firmware. Its consumers are +`firmware/MEMLNaut-NISPS/glue/{mode_select,midi_io,output_router,peripherals,settings_view}.hpp` +and the host tests in `tests/cpp/test_mode_*.cpp`. `nisps/wasm/bindings.cpp` includes eight +engine headers (lines 54–62) and six `nisps/ml/` headers (65–72) and **zero** mode headers — +verified by `grep -n modes nisps/wasm/bindings.cpp`, whose only hit is the word "modes" inside +a comment about `schemas/modes/*.json` (line 90). + +The consequence is not that the browser lacks modes. It is that the **control tick** — +`ModeBase::tick_control()`, `nisps/modes/base.hpp:198–234` — exists twice: once in C++ for the +firmware, once hand-assembled in TypeScript for the browser out of `Spine.setInputs` +(`manifold/src/engine/spine.ts:254–303`) plus `ExplorationController`'s two timers +(`manifold/src/engine/exploration.ts:40–42, 57, 84–120`). The primitives underneath (MLP, Jolt, +OU, feedback, pipelines, curves) are already shared; the **orchestration** is not. + +That is the whole defect. Everything in this document serves one outcome: *the control tick is +written once, in C++, and both targets call it.* + +--- + +## 1. Verified ground truth + +### 1.1 What `ModeBase` actually does today + +`nisps/modes/base.hpp`, `ModeBase`: + +| Responsibility | Where | Notes | +|---|---|---| +| Own the engine, the MLP, the input channels | members, `:322–341` | `input_channels_`, `input_pinned_`, `pin_value_`, `out_buf_`, `jolt_buf_` | +| `setup(sr)` — engine setup, **all channels to 0.5f**, one inference, route to engine, `on_setup` hook | `:126–142` | | +| `set_input(idx, v)` — bounds-drop + clamp to [0,1] | `:165–170` | | +| Input pinning (single/double-joystick neutralise) | `:179–196` | `set_input_pinned` / `set_pin_value` / `effective_input`; driven by `glue/settings_view.hpp` | +| `tick_control()` — the control tick | `:198–234` | order: `on_pre_inference` → jolt step (if active) → `jolt_.tick_lr_ramp()` → forward pinned channels → `ml_.process()` → OU on a copy (if enabled) → `engine_.set_params()` → `on_post_inference` | +| `apply_jolt_()` — get→glide→set of the flat weights | `:312–320` | out-of-line by design | +| Jolt / OU gesture surface | `:241–255` | `jolt_press/release/active/lr_scale`, `set_explore_intensity`, `jolt()`, `ou_noise()` | +| `driver_config()` | `:157–163` | firmware-only concept (codec / mic vs line) | +| Voice-space selection | `:279–290` | delegates to the engine, re-applies params | +| `ControlEvent` ring (`push`/`pop_control_events`) | `:294–305` | drained by `glue/midi_io.hpp:69–98` | +| `process(stereosample_t)` | `:257–259` | pure engine forward, RT | + +Firmware drives it from two cores: `tick_control()` on core 0 every 5 ms +(`firmware/MEMLNaut-NISPS/src/main.cpp:154–160`, `ML_INFERENCE_PERIOD_US 5000`), `process()` on +core 1 from the audio callback, and `drain_outputs()` on core 1 at ~1 kHz +(`main.cpp:208–212`). + +### 1.2 What the browser does instead + +`Spine.setInputs` (`manifold/src/engine/spine.ts:254–303`) is the browser's control tick: + +1. `iml.processInput(x, y, dt)` — the 2-D `InputChain` (**no firmware counterpart**, see 1.3). +2. `iml.setInput(i, v)` for every axis → `nisps_ml_set_input` → `MLPCore::set_input`. +3. `iml.processInto(mlBuf)` → `nisps_ml_process`. +4. `iml.processOutput(routedBuf, dt)` — the `OutputChain`. +5. `this.outputMorph(routedBuf)` — the OU walk, registered by + `ExplorationController`'s constructor (`exploration.ts:57`) → `nisps_ml_explore_apply`. +6. `backendSend(routedBuf)` → `EngineHost.setParams` → `postMessage` → the **worklet's own + WASM instance** → `nisps_engine_set_params`. + +Jolt is driven by a separate 5 ms `setInterval` (`exploration.ts:40, 84–91`) calling +`nisps_ml_jolt_step` + `engine.process()`; the OU walk gets a second 30 ms interval +(`exploration.ts:42, 107–113`). + +**Three WASM instances exist**, not one: main thread (`WasmIML` — MLP, feedback, pipelines), +AudioWorklet (`worklet/nisps-processor.ts` — engine only, four exports), and an on-demand +training worker (`wasm-worker.ts` — a mirror MLP). `manifold/ONBOARDING.md` §3 says "loaded +TWICE"; the training worker is a third. This is why a monolithic mode object cannot be bound +(see §2). + +### 1.3 The divergences this hand-mirroring has produced + +Each row is a fact about the current tree, verified today. + +| # | Divergence | Evidence | +|---|---|---| +| D1 | **`nisps_ml_jolt_step` is a line-for-line duplicate of `ModeBase::apply_jolt_`** | `bindings.cpp:842–849` vs `base.hpp:312–320` — same get→copy→`jolt.step`→set sequence, two copies of the code | +| D2 | **Jolt/OU RNG salts differ between targets** | `base.hpp:122–123` uses `0x91E10C5E` / `0x0CEA0FF5`; `bindings.cpp:101–102` uses `0xB01DFACE` / `0x0DDBA11`. The bindings comment claims it "mirrors the firmware ModeBase seeding" — it mirrors the *practice*, not the values. Same seed ⇒ different jolt/OU streams. (`kFeedbackSalt` *is* matched across `bindings.cpp:98` and `glue/peripherals.hpp:58`.) | +| D3 | **OU is applied at a different point in the chain.** Firmware: ML → OU → engine (`base.hpp:216–229`). Browser: ML → OutputChain → OU (`spine.ts:292, 296`). | Consequence: in the browser the per-output **freeze mask holds a value that the OU walk then moves again** — `OutputChain::process` writes `prev_[i]` for frozen dims (`output_chain.hpp:116–119`) and `nisps_ml_explore_apply` perturbs every entry up to `n_out` (`bindings.cpp:880–887`). Freeze is defeated whenever Explore > 0. | +| D4 | **Input clamping and pinning do not exist in the browser.** `nisps_ml_set_input` bounds-checks the index only (`bindings.cpp:434–441`); `MLPCore::set_input` does not clamp (`mlp.hpp:119–122`). No browser path calls anything equivalent to `set_input_pinned`. | The single/double-joystick neutralise is a firmware-only feature today (`glue/settings_view.hpp`), although the operator decision recorded in memory covers "firmware + manifold input toggle" | +| D5 | **Initial state differs.** Firmware `setup()` sets every channel to `0.5f` before its first inference (`base.hpp:129`). The browser's boot inference comes from `ConsoleApp`'s mount effect calling `engine.setInput(0.5, 0.5)` (`ConsoleApp.tsx:268`), and `Spine.setInputs` zero-fills axes 2..N (`spine.ts:277–281`). | For any mode with >2 inputs the two targets boot from different points in input space | +| D6 | **Stateful gestures advance per *call*, not per unit of time.** `Jolt::morph_rate` is documented "EMA per tick, ~1s @200Hz" (`jolt.hpp:44`); `Jolt::tick_lr_ramp` adds a fixed `0.001` per call (`jolt.hpp:107–111`); `OUNoise` uses a fixed internal `dt_ = 0.001f`, *not* the caller's dt (`ou_noise.hpp:74, 96`). Firmware calls the tick at exactly 200 Hz. The browser advances the OU walk on **every spine tick** — i.e. faster when the user moves the pad faster — plus a 33 Hz idle driver. | The same Explore setting produces materially different roam speeds on the two targets, and a browser-side rate that depends on user gesture speed | +| D7 | **`jolt_lr_scale()` has zero consumers on either target.** | `grep -rn 'jolt_lr_scale\|lr_scale' nisps/ firmware/ manifold/src tests/ vcv/src` → only `base.hpp:246`, `jolt.hpp:65`, and tests. The firmware's two `train()` call sites (`glue/peripherals.hpp:89, 225`) do not scale the LR. The documented "training eases back in after a jolt" behaviour is **not wired anywhere**. | +| D8 | **`nisps/pipeline/` is WASM-only.** `InputChain`/`OutputChain` are included by `bindings.cpp:76–77` and `tests/cpp/parity_check.cpp:67–68` and by nothing else. | Firmware has no input conditioning and no output smoothing/slew/freeze. (`simplification-plan` §6.5d already assigns this wiring to the hardware-editor item, L29.) | + +D1, D2, D6 are what this document fixes. D3, D4, D5 are fixed as a side effect and are +behaviour changes that need the operator's eyes (§5, §8). D7 and D8 are named here so an +implementer does not "discover" them mid-flight and widen scope; both are out of scope (§9). + +--- + +## 2. What this design deliberately does **not** do + +- **It does not bind mode objects into WASM.** A `ModeBase` instance owns an MLP *and* an + engine. In the browser those live in different threads and different WASM instances + (§1.2): the MLP on the main thread, the engine in the AudioWorklet, joined by a + `postMessage` of a `Float32Array` (`engine-host.ts:162–168`). Binding a mode object would + require either running the engine on the main thread (kills the RT guarantee) or the MLP in + the worklet (kills training and the whole feedback surface). The two-instance split is a + locked decision from the one-core refactor; this design respects it. +- **It does not delete the per-primitive C API.** `nisps_ml_*` stays exactly as it is — + `simplification-plan` §6.5a and audit A5 both say so explicitly, and the parity harness, + the training worker, the feedback controller and the probe all consume it. +- **It does not move the audio engine, the voice-space surface, the `ControlEvent` ring, or + `driver_config()`.** Those are the *other* half of a mode and belong to items 5b (browser + mode coverage / audio topology) and 5d (hardware editor). Mixing them in is how this becomes + a month. +- **It does not touch `nisps/pipeline/`'s ownership.** The input chain stays a caller-side + pre-stage and the output chain stays a caller-side post-stage, both on the existing + `nisps_pipeline_*` handle. Only their *ordering relative to OU* changes (§5). + +--- + +## 3. Invariants + +The repo-wide constraints (`CLAUDE.md`) are the floor. These are the ones specific to this area: + +1. **`nisps/` stays platform-neutral and heap-free.** The new shared code lives under + `nisps/modes/`, and its fixed storage policy must be `std::array`-only. The dynamic policy is + a second deliberate exception to the no-heap rule and must carry the same + `#if defined(NISPS_TARGET_EMBEDDED) → #error` guard as `nisps/ml/dynamic_storage.hpp:22–24`, + and must be added to **both** the allowlist branch (`scripts/lint-cpp.sh:146`) and the + guard-survival check (`:194–198`). Today that script hardcodes exactly one path; generalise + it to a list rather than adding a second special case. +2. **Firmware behaviour is bit-identical at the current cadence.** With the mode ticked at + 5 ms, every mode's audio output must be unchanged. `tests/cpp/test_mode_learning.cpp`'s + inert-parity test and `test_mode_paf_synth.cpp`'s reference-vector tests are the existing + guards; step 1 of §6 adds a stronger one. +3. **The `nisps::Mode` concept keeps its current shape** (`nisps/core/concepts.hpp:70–88`). + `tick_control()` must remain callable with no arguments, or every mode `static_assert` and + all of `firmware/glue/` breaks. +4. **The control tick stays bounded and allocation-free.** It runs on core 0 on firmware and on + the main thread in the browser; neither may block. Any catch-up loop introduced for + rate-independence (§4.3) must be hard-capped. +5. **Dual-core ownership and SPSC discipline are preserved.** `engine_.set_params()` is written + from core 0 while `engine_.process()` runs on core 1 — that is the existing arrangement and + this change must not add a second writer. The `ControlEvent` ring has exactly one producer + per mode (core 0 for the CC-emitting modes via `on_post_inference`, core 1 for the sequencer + modes via `pump_engine_events`); the ring must not move into shared code where a mode could + acquire two producers. +6. **Deterministic per-instance RNG.** Jolt and OU keep their own `nisps::Rng` streams, + independent of the MLP's and the feedback controller's. After this change the *salts must + match across targets* (D2) — that is the point. +7. **Native↔WASM parity ≤1e-5 must cover the new code.** The existing harness does not + (§7). Extending it is part of the work, not a follow-up. +8. **A WASM C-API addition walks all five layers**: `EMSCRIPTEN_KEEPALIVE` → + `EXPORTED_FUNCS` in `scripts/build-wasm.sh:40–72` → `NispsModule` in + `manifold/src/engine/types.ts` → `WasmIML` → `EngineApi`. A missed layer fails at runtime. + +--- + +## 4. The design + +### 4.1 The seam: `ControlCore` + +A new header `nisps/modes/control_core.hpp` defining `nisps::ControlCore` — the +control tick and nothing else. It is storage-policied exactly the way `MLPCore` +(`nisps/ml/mlp.hpp:91–92`) and `FeedbackControllerCore` (`nisps/ml/feedback.hpp:42–51`) +already are, including the `ControlCore : public CtlStorage` + forwarded-`StorageArgs` +constructor idiom (`mlp.hpp:103–106`). + +It **owns**: the input channels, the pin mask and pin value, a `ml::Jolt`, the OU walk and its +state, the produced parameter buffer, and the fixed-rate accumulator. + +It **does not own** the MLP. Like `FeedbackControllerCore`, every method that needs the network +takes it by reference as a method-level template, so a `MLPCore` and a +`MLPCore` both work: + +```cpp +namespace nisps { + +template +class ControlCore : public CtlStorage { + public: + template + explicit ControlCore(std::uint64_t seed, StorageArgs&&... args) noexcept; + + bool valid() const noexcept; // dynamic storage may fail to allocate + + // ---- input channels (was ModeBase :165–196) ---- + void set_input(std::size_t idx, float v) noexcept; // bounds-drop + clamp [0,1] + void set_input_pinned(std::size_t idx, bool p) noexcept; + bool is_input_pinned(std::size_t idx) const noexcept; + void set_pin_value(float v) noexcept; + float pin_value() const noexcept; + float effective_input(std::size_t i) const noexcept; + std::span input_channels() const noexcept; + std::span mutable_input_channels() noexcept; // SoundAnalysisMIDI's hook + + // ---- gestures (was ModeBase :241–255) ---- + void jolt_press(std::size_t weight_count) noexcept; + void jolt_release() noexcept; + bool jolt_active() const noexcept; + float jolt_lr_scale() const noexcept; + void set_explore_intensity(float level) noexcept; + float explore_intensity() const noexcept; + + // ---- the tick ---- + // Runs `n` fixed-rate control steps (n derived from dt_s, see 4.3), one + // inference, and the OU walk. Returns the produced parameter vector — a + // STABLE span into this object's own buffer, valid until the next reshape. + template + std::span tick(MLPT& ml, float dt_s) noexcept; + + std::span params() const noexcept; // same span, without ticking + void reset() noexcept; // clears OU state + accumulator +}; + +} // namespace nisps +``` + +### 4.2 Storage policies + +Surface (documented in the header the way `nisps/ml/storage.hpp:20–28` documents the MLP's): + +``` +dims: n_in(), n_out() +buffers: input_channels(), input_pinned(), out_buf(), ou_state() +validity: valid() +``` + +- `nisps/modes/control_storage.hpp` — `FixedControlStorage`: four `std::array` + members, all dims `constexpr`-foldable, zero heap. Firmware + host tests. +- `nisps/modes/dynamic_control_storage.hpp` — `DynamicControlStorage`: dims at construction, + **one arena allocation in the constructor and none after**, `#error` under + `NISPS_TARGET_EMBEDDED`. Browser only. Mirrors `nisps/ml/dynamic_storage.hpp` file-for-file + in shape so the lint allowlist reads as one pattern rather than two exceptions. + +`FixedControlStorage`'s buffers are ~`2*NIn + 2*NOut` floats — for MEMLCelium (4 in, 56 out) +about 480 bytes. + +**`jolt_buf_` is deleted, not moved.** `ModeBase::jolt_buf_` is a +`std::array` (`base.hpp:341`) — **6,152 bytes of SRAM on every +MEMLCelium-class mode** (`MLP<4,10,14,18,56>` ⇒ 1,538 weights). It exists only to hold a copy of +the flat weights across `jolt_.step()`. But `MLPCore`'s storage already owns exactly such a +buffer: `flat_buf()` (`nisps/ml/storage.hpp:164`, `nisps/ml/dynamic_storage.hpp:162`), which is +what `get_weights()` fills (`mlp.hpp:385–400`). Add a `std::span weights_scratch()` +accessor to `MLPCore` and the jolt step becomes: + +```cpp +const auto w = ml.get_weights(); // fills the storage's flat buffer +jolt_.step(ml.weights_scratch().first(w.size())); +ml.set_weights(ml.weights_scratch().first(w.size())); +``` + +`MLHandle::jolt_scratch` (`bindings.cpp:167`, a `std::vector`) goes the same way. + +**`OUNoise` loses its template parameter and splits `apply()` in two.** +`ml::OUNoise` (`nisps/ml/ou_noise.hpp:44–99`) holds `std::array state_` and its +`apply()` advances the walk and adds it to the output in one loop (`:68–81`). Two changes: +(i) the state span comes from the caller, so `OUNoise` becomes a plain class and `ControlCore`'s +storage owns the state; (ii) split into `advance(std::span state)` and +`add_to(std::span out, std::span state)`, because a catch-up tick must +advance N times and add once (§4.3a). Both keep the existing early-out at intensity 0, so the +inert path stays free and parity-safe. This also deletes the browser's +`OUNoise` — a 16 KB fixed array (`bindings.cpp:166`) sized for a net that is never +4096-wide. Three call sites: `base.hpp`, `bindings.cpp`, `tests/cpp/test_mlp_ou_noise.cpp`. +*Lower-risk fallback if this proves noisy: keep `OUNoise` and have each storage policy expose +an `OU` typedef at its own N. It costs the browser its 16 KB and leaves a template where none is +needed, but it is strictly smaller.* + +### 4.3 The canonical tick + +``` +tick(ml, dt_s): + steps = accumulate_steps(dt_s) # see below; firmware @5ms => exactly 1 + for s in 0..steps-1: + if jolt.active(): jolt.step(ml's flat weight scratch) ; ml.set_weights(...) + jolt.tick_lr_ramp() + for i in 0..n_in-1: ml.set_input(i, effective_input(i)) + ml.process() + copy ml.outputs() -> out_buf # ALWAYS, see below + for s in 0..steps-1: ou.advance(ou_state) # no-op while disabled + ou.add_to(out_buf, ou_state) # no-op while disabled + return out_buf +``` + +Four decisions are baked in here; each is a change from at least one target's current behaviour. + +**(a) Fixed-rate stepping (fixes D6).** `ControlCore` owns a `float accum_` and a compile-time +`kControlPeriodS = 1.f / 200.f`. `accumulate_steps(dt_s)` adds `dt_s`, takes out whole periods, +and **caps the result at `kMaxCatchUpSteps` (8)** — invariant 4. At the firmware's exact 5 ms +cadence this yields exactly one step with zero residual, so firmware is unchanged; in the browser +a 30 ms driver call performs six steps and the walk becomes wall-clock-correct regardless of how +fast the user moves the pad. The rate constant, not wall-clock dt, stays the calibration for +`Jolt::morph_rate`, `Jolt::lr_ramp_step` and `OUNoise::dt_` — those are shape constants tuned at +200 Hz upstream. Re-deriving them from wall-clock dt is *not* equivalent: `OUNoise::dt_` is +`0.001`, not the firmware's 5 ms period, so feeding wall-clock dt would multiply firmware's +mean-reversion term by 5 and its per-step noise by √5. + +**(b) Inference runs once per `tick()`, not once per control step.** Only the stateful walks are +rate-sensitive; a forward pass is idempotent given the same inputs and weights. This keeps a +catch-up tick cheap. + +**(c) The output copy is unconditional.** Today firmware skips the copy when OU is disabled and +hands `ml_.outputs()` straight to the engine (`base.hpp:228`). If `tick()` did that, the returned +span would sometimes point into the MLP's output buffer and sometimes into `out_buf` — a +pointer-stability footgun for the C API, which caches the pointer JS-side. Always copying costs +`n_out` floats per tick (56 at 200 Hz for MEMLCelium: ~11 k float copies/s against a 1,538-weight +forward pass at the same rate) and is bit-identical. + +**(d) OU is applied to the mode's parameter vector, before any caller-side output chain +(changes the browser — D3).** Firmware order is preserved; the browser's `OutputChain` moves from +*before* the OU walk to *after* it. Rationale: OU is part of what the mapping *produces*; the +output chain is how the *destination* conditions it. This is also what makes per-output freeze +mean something again. It is user-audible — see §5 and §8-Q1. + +### 4.4 `ModeBase` afterwards + +`ModeBase` keeps its entire public surface (invariant 3) and becomes a composition: + +```cpp +NISPS_HOT void tick_control(float dt_s = kControlPeriodS) noexcept { + if constexpr (requires(Derived& d) { d.on_pre_inference(); }) + static_cast(*this).on_pre_inference(); + const auto params = control_.tick(ml_, dt_s); + if constexpr (kRouteOutputsToEngine) engine_.set_params(params); + if constexpr (requires(Derived& d) { d.on_post_inference(); }) + static_cast(*this).on_post_inference(); +} +``` + +with `control_` a `ControlCore>`, and +`set_input` / `set_input_pinned` / `pin_value` / `input_channels` / `mutable_input_channels` / +`jolt_*` / `set_explore_intensity` / `explore_intensity` all one-line forwards. The default +argument preserves the concept and every existing call site (`main.cpp:157`, +`test_mode_*.cpp`). + +What stays in `ModeBase`: the engine, the CRTP hooks, `setup()`, `driver_config()`, +`set_voice_space()`, the `ControlEvent` ring, `process()`. What leaves: the input-channel state, +`jolt_`, `ou_`, `out_buf_`, `jolt_buf_`, `apply_jolt_()`. + +Two accessors need a call: `ml::Jolt& jolt()` (`base.hpp:247–248`) has **no consumer outside +`base.hpp`** — delete it. `ml::OUNoise& ou_noise()` (`:254–255`) is used by +`tests/cpp/test_mode_learning.cpp:89–103` — keep it (forwarding into `control_`) or rewrite those +three assertions against `explore_intensity()`; implementer's call. + +The seeding fix (D2) rides along: `ControlCore`'s constructor takes the salts, and both targets +pass the same ones. Name them once, in `control_core.hpp`, as +`kJoltSalt = 0x91E10C5Eull` / `kOUSalt = 0x0CEA0FF5ull` (the firmware's current values, so +firmware weight streams are unchanged; the browser's jolt/OU streams change, which no committed +artifact or golden depends on — verified: no fixture, parity stage or e2e drives jolt/OU). + +### 4.5 The browser afterwards + +`MLHandle` (`bindings.cpp:148–185`) drops `jolt`, `ou` and `jolt_scratch` and gains one +`ControlCore` member, constructed at `(d.n_in, d.n_out)` and rebuilt by +`nisps_ml_reshape` alongside the feedback controller (`bindings.cpp:396–421`). + +The existing `nisps_ml_jolt_*` and `nisps_ml_explore_intensity` entries stay in place and +**forward into the head** — so the per-primitive C API survives (§2) while its *implementation* +stops being a second copy (kills D1). + +`Spine.setInputs` becomes: + +1. `iml.processInput(x, y, dt)` — unchanged. +2. `iml.setControlInput(i, v)` for each axis → `nisps_control_set_input` (clamped + pin-aware). +3. `iml.controlTick(dt)` → `nisps_control_tick` — replaces today's steps 3 *and* 5. +4. read the raw ML outputs (`nisps_ml_outputs`) into `mlBuf` for `liveOutputs`, and the head's + params (`nisps_control_params`) into `routedBuf`. +5. `iml.processOutput(routedBuf, dt)` — unchanged call, now *after* the OU walk. +6. `backendSend(routedBuf)` — unchanged. + +`Spine.setOutputMorph` and the `outputMorph` field are **deleted** (`spine.ts:105–109, 221–229, +294–296`); its only consumer is `ExplorationController`'s constructor and `dispose` +(`exploration.ts:57, 131`). `ExplorationController` collapses from two timers to one ~60 Hz +driver that exists while `joltActive() || exploreIntensity() > 0` and simply calls +`engine.process()` — the fixed-rate accumulator makes the driver's exact period irrelevant. + +### 4.6 The C API additions — the five-layer walk + +| Export | `bindings.cpp` | `build-wasm.sh` `EXPORTED_FUNCS` | `types.ts` `NispsModule` | `WasmIML` | `EngineApi` | +|---|---|---|---|---|---| +| `void nisps_control_set_input(void* ml, int idx, float v)` | new | add | add | `setControlInput` | via `spine` | +| `void nisps_control_set_input_pinned(void* ml, int idx, int pinned)` | new | add | add | `setControlInputPinned` | `EngineApi.control.setInputPinned` | +| `void nisps_control_set_pin_value(void* ml, float v)` | new | add | add | `setControlPinValue` | `EngineApi.control.setPinValue` | +| `int nisps_control_tick(void* ml, float dt_s)` → n params | new | add | add | `controlTick` | via `spine` | +| `const float* nisps_control_params(void* ml)` | new | add | add | (internal) | — | +| `void nisps_ml_explore_apply(void* ml, float*, int)` | **delete** (`:880–887`) | **remove** (`:63`) | **remove** (`:133`) | **remove** `exploreApply` | **remove** `explore.exploreApply` | + +Deleting `nisps_ml_explore_apply` is safe: its consumers are `WasmIML.exploreApply` → +`EngineApi.explore.exploreApply` → `exploration.ts:57`, all of which this change removes. It is +**not** touched by `tests/cpp/parity_wasm.mjs`, `manifold/tests/`, or `src/debug/probe.ts` +(verified by grep). Re-verify before deleting. + +`nisps_ml_set_input` **stays**: `tests/cpp/parity_wasm.mjs` drives parity stage 1 through it, and +it is the raw-MLP write that `infer_batch` and the training worker semantics assume. The two are +distinguished in the header comment: `nisps_ml_set_input` = raw network input; `nisps_control_set_input` += the mode control path (clamped, pinnable). + +### 4.7 Deletions, with named consumers + +| Deleted | Named consumers, and what happens to them | +|---|---| +| `ModeBase::jolt_buf_` (`base.hpp:341`) + `apply_jolt_` (`:312–320`) | `tick_control` only → replaced by `ControlCore::tick` over `MLPCore::weights_scratch()` | +| `MLHandle::jolt_scratch` (`bindings.cpp:167`) | `nisps_ml_jolt_step` only → same | +| `MLHandle::jolt`, `MLHandle::ou` (`:165–166`) | the `nisps_ml_jolt_*` / `nisps_ml_explore_*` bodies → forward into `control` | +| `ModeBase::ou_` / `out_buf_` / `input_channels_` / `input_pinned_` / `pin_value_` | `ModeBase` methods only → forwarded into `control_` | +| `ml::OUNoise`'s template parameter + `state_` | `base.hpp`, `bindings.cpp:166`, `tests/cpp/test_mlp_ou_noise.cpp` → state moves to `ControlCore`'s storage | +| `ModeBase::jolt()` accessor (`:247–248`) | none — verified by grep across `nisps/ firmware/ manifold/ tests/ vcv/` | +| `nisps_ml_explore_apply` + its 4 downstream layers | see 4.6 | +| `Spine.outputMorph` + `setOutputMorph` (`spine.ts:105–109, 221–229, 294–296`) | `exploration.ts:57, 131` only | +| `ExplorationController`'s `joltTimer`/`exploreTimer` pair | internal → one driver | + +--- + +## 5. Behaviour changes this lands + +Stated plainly, because two of them are audible. + +1. **Browser: the output chain now runs after the OU walk** (D3). With Explore > 0, per-output + freeze actually freezes, and the EMA/slew smooth the OU walk instead of the walk overwriting + smoothed values. Audible whenever Explore > 0 *and* smoothing/slew/freeze are non-default. +2. **Browser: the OU roam rate becomes wall-clock-correct and gesture-independent** (D6). + Today the walk advances once per spine tick, so it roams faster while the user is moving. + After: a fixed 200 steps/second. At the current 33 Hz idle driver the browser's walk is + ~6× slower than firmware's; after the change they match. **Existing Explore slider positions + will feel different** — most users will want a lower setting. +3. **Browser: inputs are clamped to [0,1] and honour a pin mask** (D4). No current caller feeds + out-of-range values, so this is latent until the pin mask is wired into the UI (5c). +4. **Browser: jolt/OU RNG streams change** (D2), because the salts become the firmware's. No + committed artifact depends on them. +5. **Firmware: no behaviour change.** The tick order, the constants, the cadence and the + arithmetic are preserved by construction; §6 step 1 makes that a test rather than a claim. + The one *near*-exception is `jolt_lr_scale()`, whose ramp accumulates identically per step — + and which has no consumers on either target (D7). + +--- + +## 6. Sequenced implementation plan + +Each step is separately landable and separately verifiable. Step 1 is pure C++ and changes no +behaviour anywhere. Step 2 adds the browser-side plumbing and, because the jolt/OU salts and the +fixed-rate accumulator take effect as soon as `bindings.cpp` forwards into the head, it already +carries §5.2 and §5.4. Step 3 is the browser swap and carries §5.1 and §5.3. + +**Sequencing hazard:** `nisps_ml_explore_apply` must be deleted in **step 3**, not step 2 — it is +still called by `WasmIML.exploreApply` until the TS swap lands, and removing the export first +leaves a runtime failure that no compiler catches (invariant 8, in reverse). + +### Step 0 — capture a firmware-behaviour baseline (half a day) + +Add `tests/cpp/test_mode_control_tick.cpp` to the `nisps_modes_tests` target +(`nisps/CMakeLists.txt:119–128`) that pins the *current* `tick_control` for at least PAFSynth, +MEMLCelium and SoundAnalysisMIDI: N ticks with a scripted input trace, asserting the exact +engine-param vector and the exact ML output vector, with Jolt and Explore both exercised. +Land it **before** touching `base.hpp` so it is a genuine before/after. + +*Verification:* `bash scripts/build-cpp-tests.sh` (baseline today: 4/4 ctest suites pass in +0.03 s — run and confirmed 2026-07-21). + +### Step 1 — `ControlCore` + fixed storage; `ModeBase` composes it (1–2 days) + +New: `nisps/modes/control_storage.hpp`, `nisps/modes/control_core.hpp`. Modified: +`nisps/modes/base.hpp` (compose + forward), `nisps/ml/mlp.hpp` (`weights_scratch()`), +`nisps/ml/ou_noise.hpp` (externalise state). Concrete mode headers should need **no changes** — +if one does, the forwarding surface is wrong. + +*Verification:* step 0's baseline test must pass **unchanged** — that is the whole point of doing +it first. Plus `nisps_modes_tests`, `nisps_core_tests`, `scripts/lint-cpp.sh`, and a firmware +compile of at least `slpworkshop`, `pafsynth`, `soundanalysismidi` via `scripts/build-firmware.sh` +(needs `nix-shell -p platformio-core`; first build pulls 1–2 GB). Compare flash/RAM against the +pre-change build — the expectation is a **reduction** of roughly `weight_count() * 4` bytes of +RAM (6,152 for MEMLCelium-class modes) from the `jolt_buf_` deletion. A size *increase* means +the composition failed to inline and wants investigating. + +*Not verifiable here:* that the hardware still boots and sounds right. Operator chokepoint, +same as every firmware change in this repo. + +### Step 2 — dynamic storage + the C API + parity coverage (1–2 days) + +New: `nisps/modes/dynamic_control_storage.hpp`. Modified: `nisps/wasm/bindings.cpp` (MLHandle +member, reshape, the five new `nisps_control_*` exports, the `nisps_ml_jolt_*` / +`nisps_ml_explore_intensity` bodies forwarding into the head — but **not** the +`nisps_ml_explore_apply` deletion, which waits for step 3), `scripts/build-wasm.sh`, +`scripts/lint-cpp.sh` (allowlist → list). + +**Add parity stage 8** to `tests/cpp/parity_check.cpp` and `tests/cpp/parity_wasm.mjs`: drive a +`ControlCore` over the harness's `MLP<32,10,14,18,126>` through a scripted trace — inputs set, +some channels pinned, a jolt pressed and released across several ticks, Explore raised, at least +one catch-up tick with `dt_s` > one period — and push the parameter vector plus a weight probe. +Bump `kVersion` 5 → 6 (`parity_check.cpp:93`) in both drivers, and extend the header's +authoritative stage list (`:14–33`) and `parity_diff.mjs`'s section names. + +This is the step that converts "a parity PASS is not evidence for this change" into "it is". +Without stage 8, `scripts/parity-check.sh` exercises PAFSynth and ChannelStrip under an +all-params-0.5 vector and says nothing whatsoever about the control tick. + +*Verification:* `bash scripts/build-wasm.sh && bash scripts/parity-check.sh` (both native and +WASM at 1e-5, now including stage 8); `bash scripts/lint-cpp.sh`; +`bash scripts/build-cpp-tests.sh`. `emcc`, `bun`, `cmake` and `ninja` are all on PATH here, so +this step is fully verifiable locally. + +### Step 3 — the browser swap (1–2 days) + +Modified: `manifold/src/engine/types.ts`, `wasm-iml.ts`, `engine-api.ts`, `spine.ts`, +`exploration.ts`. Deleted: `Spine.setOutputMorph` + the `outputMorph` field, `exploreApply` +through all four TS layers, one of the two exploration timers. + +Rebuild and **commit** `manifold/public/nisps.{js,wasm}` in the same change — the webhook builds +only `manifold/` and ships whatever is committed (`manifold/ONBOARDING.md` §2), and CI's WASM +freshness gate runs the parity harness against the committed artifact before rebuilding it. + +*Verification:* `cd manifold && bun run typecheck && bun run test && bun run build`, then +Playwright. Note that `manifold`'s test script is deliberately `bun test src tests/*.test.ts` — +do not "fix" it to `tests`, that drags the e2e suite into the unit run. On the VPS, e2e needs the +non-snap node runner (`ONBOARDING.md` §2). + +Add e2e coverage for the two changes users can see, in `manifold/tests/e2e/`: +- Explore > 0 with a freeze mask set ⇒ the frozen outputs do **not** move (this fails today). +- Explore > 0 with the pad held still ⇒ the routed vector's drift over 2 s is within a + tolerance band, i.e. wall-clock-rated rather than tick-rated. + +A unit test in the `bun test` set driving `nisps_control_tick` straight at the committed WASM — +the way `manifold/tests/loss-history.test.ts` drives `nisps_ml_loss_history` — is the cheapest +guard on the C ABI and should exist too. + +### Step 4 — documentation sync (same commits) + +`MAP.md` (`nisps/modes/` and `nisps/wasm/` bullets), `docs/specs/MAIN.md` (registry row + +"Loaded TWICE" is wrong — it is three instances), `manifold/ONBOARDING.md` §3 (same), +`ALIGNMENT.md` defect 1 (delete it when step 3 lands, per the repo's doc-sync rule), +`simplification-plan.md` §6.5a (burn down), and this file's `status:` → `executed`. + +--- + +## 7. What the verification actually covers — and what it does not + +| Claim | Covered by | Confidence | +|---|---|---| +| Firmware tick behaviour unchanged | step 0's pinned baseline + `nisps_modes_tests` | **High** — it is a before/after on the exact byte values | +| Fixed↔dynamic control head agree | parity stage 8 (step 2) | **High**, once written | +| Native↔WASM agreement of the new code | `scripts/parity-check.sh` **after** stage 8 exists | High after; **zero** before — the current harness touches neither modes nor jolt nor OU | +| The C API is reachable end-to-end | the `bun test` C-ABI test + Playwright | Medium-high — the 5-layer chain fails at runtime, so a browser-level test is the only real proof | +| Freeze is no longer defeated by Explore | new e2e case | High | +| OU roam rate is wall-clock | new e2e case (tolerance band) | Medium — timing assertions in a browser are inherently soft; keep the band wide | +| Firmware flash/RAM does not regress | `firmware-build` CI job's per-variant report + local `pio run` | High for size | +| **The hardware still boots and sounds correct** | *nothing* | **None.** This is an operator chokepoint, as it was for the PlatformIO cut (`simplification-plan` §5) | +| Engines other than PAFSynth/ChannelStrip behave identically | `nisps_dsp_engine_tests` + `engine_impulse` baseline; unchanged by this work | High, and unaffected — no engine code is touched | + +Honest summary: everything except on-device behaviour can be verified in CI, **provided stage 8 +is written**. If the implementation session runs out of time and skips stage 8, the change should +not land: the parity gate would go green while covering none of it. + +--- + +## 8. Open questions for the operator + +**Q1. The output-chain / OU ordering (§4.3d, §5.1) — confirm.** The recommendation is the +firmware's order: OU is part of the mapping, the output chain conditions what the destination +receives, and per-output freeze starts working. It changes the browser's sound whenever Explore +is up and smoothing/slew/freeze are non-default. The alternative — keeping each target's current +order — means the tick is not actually unified and needs a per-target branch, which is the thing +this whole item exists to remove. **Recommend: adopt the firmware order.** + +**Q2. Fixed 200 Hz control rate as a shared contract (§4.3a) — confirm.** This makes the +browser's Explore feel roughly 6× faster at the same slider position. Options: (a) adopt it and +accept that existing slider positions read differently; (b) adopt it and rescale +`kOUMaxAmplitude` or the slider mapping so the browser's *current* feel is preserved — which +then changes the *firmware's* feel; (c) adopt it and expose the rate as a per-target constant, +which re-forks the behaviour. **Recommend (a)**, with a note in the Learning drawer copy. + +**Q3. `jolt_lr_scale()` (D7) — wire it or delete it?** It is documented on both `Jolt` and +`ModeBase` as gating the learning rate after a jolt, and *nothing on either target multiplies by +it*. Wiring it means touching the firmware's two `train()` call sites and the browser's train +path — a behaviour change to training, not to this seam. Deleting it removes a documented +feature. It is cheap either way once `ControlCore` exists. **Not blocking**; needed before this +item can be called done rather than parked. + +**Q4. Does the browser's input pin mask (D4) get UI in this item or in 5c?** The C API and the +core behaviour come free with `ControlCore`. Rendering a Single/Double control in the Inputs +drawer is UI work that belongs with the curated/advanced split +(`plans/curated-presets-spec.md`). **Recommend: ship the API in this item, the UI in 5c**, and +say so in `inputs-spec.md`. + +**Q5. `DynamicControlStorage`'s home.** Recommended: `nisps/modes/`, with `lint-cpp.sh`'s +allowlist generalised from one hardcoded path to a list. The alternative — defining the dynamic +storage policy inside `nisps/wasm/bindings.cpp`, where heap is already legitimate and the lint +does not look — needs no lint change at all, but puts a storage policy outside `nisps/` where a +future consumer (VCV, a native fixture) cannot reach it. **Recommend `nisps/modes/`**; it is a +~10-line shell change and it keeps the "one auditable exception list" discipline. + +Not asked, because they were decided here and are recorded rather than open: `ControlCore` does +not own the MLP (§4.1, matching `FeedbackControllerCore`); the parameter copy is unconditional +(§4.3c); the jolt scratch buffer is deleted in favour of `MLPCore`'s existing flat scratch +(§4.2); `OUNoise` loses its template parameter (§4.2); the salts become the firmware's (§4.4). + +--- + +## 9. Explicitly out of scope + +- **The engine half of a mode in the browser** — voice-space selection, `note_on`, transport + (`set_playing`/`update_bpm`), and the `ControlEvent` drain have no browser surface at all + (verified: `grep -rn 'voiceSpace\|noteOn\|setPlaying\|popEvents' manifold/src` returns only + generated-schema data). That is item **5b** — see `plans/browser-mode-coverage-spec.md` — and + it is the reason four catalogued modes cannot actually run in the browser. +- **Firmware `InputChain`/`OutputChain` wiring (D8, L29)** — assigned to **5d** by + `simplification-plan` §6.5d; see `plans/hardware-editor-spec.md`. This item only fixes where OU + sits relative to a chain the caller already owns. +- **Feedback-controller ownership.** Firmware keeps it as a function-local static in + `glue/peripherals.hpp:152`; the browser keeps it inside `MLHandle`. Both already call the same + shared core, so it is an asymmetry, not a duplication. The related `FOLLOW-UP` comment at + `glue/peripherals.hpp:33–39` (hold `feedback.static_output()` while Placing instead of running + fresh inference) *would* have a natural home in `ControlCore::tick` once it exists — note it, + do not build it here. +- **The browser's multi-anchor ExploreAndPlace session policy** + (`manifold/src/feedback/controller.ts:200–290`) differs from the firmware's freeze-and-carry + gesture. That is an application-policy divergence above the core, not a core duplication. +- **`jolt_lr_scale` wiring** (Q3) and **the pin-mask UI** (Q4). + +--- + +## 10. Source-file index + +Read in this order. + +**The gap** +- `nisps/modes/base.hpp` — `ModeBase`; the tick is `:198–234`, the jolt copy `:312–320` +- `nisps/wasm/bindings.cpp` — `MLHandle` `:148–185`, jolt/OU C API `:827–886`, reshape `:396–421` +- `manifold/src/engine/spine.ts` — `setInputs` `:254–303`, the morph hook `:105–109, 221–229` +- `manifold/src/engine/exploration.ts` — the two timers and the morph registration + +**The pattern being copied** +- `nisps/ml/mlp.hpp:91–116` + `nisps/ml/storage.hpp` + `nisps/ml/dynamic_storage.hpp` — + `MLPCore`, the storage-surface documentation style, the embedded `#error` guard +- `nisps/ml/feedback.hpp:42–61` — `FeedbackControllerCore`, and the + "does NOT own the MLP" convention this design follows + +**Consumers that must keep compiling unchanged** +- `firmware/MEMLNaut-NISPS/glue/peripherals.hpp`, `settings_view.hpp`, `midi_io.hpp`, + `output_router.hpp`, `src/main.cpp:154–160` +- `tests/cpp/test_mode_{concepts,paf_synth,voice_space,breakor_events,learning,curve_overrides,driver_config}.cpp` +- `nisps/modes/sound_analysis_midi.hpp:86–111` and `external_synth_midi.hpp` — the only modes + with `on_pre_inference` / `on_post_inference` hooks, i.e. the sharpest test of the forwarding + surface + +**Gates** +- `scripts/build-cpp-tests.sh`, `scripts/parity-check.sh`, `scripts/lint-cpp.sh`, + `scripts/build-wasm.sh`, `scripts/build-firmware.sh`, `.github/workflows/ci.yml` +- `tests/cpp/parity_check.cpp:14–33` (the authoritative stage list) + `parity_wasm.mjs` + + `parity_diff.mjs`