w1n5t0n/memlnaut-nisps
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs(specs): P5 architectural specs — 5a, 5b, 5c, 5d

Browse source 
Plan §6 says each P5 item is spec-first with its own session. These are the
specs; no implementation is authorised by them and none was written.

  plans/mode-layer-reunification.md   5a — ALIGNMENT defect 1, the largest
                                      architectural gap. Storage-policies the
                                      ModeBase orchestration the way P2 did
                                      MLPCore, rather than binding monolithic
                                      mode objects into WASM (which would
                                      contradict the locked two-instance RT
                                      architecture).
  plans/browser-mode-coverage-spec.md 5b — an audio-topology notion so Manifold
                                      stops cataloguing 4 modes that
                                      structurally cannot run in a browser.
  plans/curated-presets-spec.md       5c — ALIGNMENT defect 2, built on the
                                      operator's §7.6 definition: a curated
                                      preset is configuration only, network
                                      untrained.
  plans/hardware-editor-spec.md       5d — ALIGNMENT defect 3, applying
                                      useq-celium's existing discipline (C
                                      header as wire truth + TS mirror + parity
                                      test) to a MEMLNaut serial protocol.

2434 lines. I spot-checked their path citations mechanically against
git ls-files: of 170 backticked paths, every unresolved one is either a file
the spec proposes to create or a cross-reference to a sibling spec in this same
commit. None describes deleted code as live — which is the failure mode that
made half the existing corpus untrustworthy, and the reason the §8 pass earlier
today had so much to do.

The preset spec is the most valuable byproduct: en route it found five places
where existing docs still describe a deleted world, including
manifold-parity-features-spec.md §1.1 specifying a PipelineLayer over
engine/input-pipeline.ts and output-pipeline.ts, both deleted at one-core P4.

Also carries the doc sync for the telemetry and benchmark work in a77770f:
AGENT-REFERENCE gains the throughput and loss-history entries, and dock-spec
§1.3 records that its long-deferred diagnostics suite shipped PARTLY — the loss
curve and weight-health table are real, GradientFlow is not built and is not
planned as drawn (the core records no per-layer gradient magnitudes, and the
fabricated version was deleted in Phase 1).
This commit is contained in:
monkey-w1n5t0n 2026-07-21 22:03:39 +02:00
parent a77770f95d
commit 60584606a8
6 changed files with 2449 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

7
docs/AGENT-REFERENCE.md
View file

@ -39,6 +39,8 @@ Build: `cmake -S nisps -B nisps/build -G Ninja && cmake --build nisps/build && c
Tests: 4 executables (`nisps_core_tests`, `nisps_dsp_engine_tests`, `nisps_modes_tests`, `nisps_golden_tests`). Run all: `bash scripts/build-cpp-tests.sh`. Parity vs WASM: `bash scripts/parity-check.sh` (asserts native and WASM produce identical outputs within 1e-5).
Throughput: `bash scripts/bench-engines.sh` (per-engine ns/sample, blocks/s, realtime factor on native + WASM; `--compare <report.json>` for deltas). It **reports and never asserts** — no threshold, no failure mode. Read it before and after any change to `nisps/dsp/` or `nisps/engines/`; nothing else in the repo will tell you an engine got slower.
### Performance contract (RP2350)
These rules apply to **all** code under `nisps/`. They are inert in WASM but kept globally for consistency.
@ -117,10 +119,10 @@ The browser MLP is runtime-shaped since P2 (`MLPCore<DynamicStorage>`): `nisps_m
### Known limitations
- Loss history not yet plumbed through C API; only the final loss of a training run reaches TS.
- Mic input through the worklet for XIASRI / SoundAnalysisMIDI is not wired in manifold.
- C15 has no home on main (see `ALIGNMENT.md` defect 1, browser mode coverage).
- (P3, 2026-07-14) The browser Jolt/OU gestures and the geometric dislike run the C++ core through WASM: `nisps_ml_jolt_*`, `nisps_ml_explore_*`, `nisps_ml_feedback_dislike_geometric` — no TS gesture math remains.
- (§6.5e, 2026-07-21) The per-iteration loss curve IS plumbed: `nisps_ml_loss_history(ml, out, max)` returns the total entry count and fills `min(count, max)`, so `max=0` is a count probe. Both train paths publish it to the spine; `EngineApi.lossHistory()` reads it and `console/TrainingHealth.tsx` displays it at `expanded` drawer depth. `MLPCore::train()` resets the history per run; `train_targets()` (geometric dislike) does not record.
## URL parameters (manifold)
@ -164,6 +166,9 @@ bash scripts/build-wasm.sh
# Cross-platform parity
bash scripts/parity-check.sh
# Engine throughput (reports; never fails)
bash scripts/bench-engines.sh
# Lint
bash scripts/lint-cpp.sh

9
docs/specs/dock-spec.md
View file

@ -29,6 +29,15 @@ string "C15" MUST NEVER appear in the UI.*
> solo/arm + min/max/curve).
> - The feedback surface shipped richer than §1 (Explore-and-Place with reroll/nudge/place, plus
> geometric dislike) — see `docs/adr/rl-feedback-design.md` and `nisps/ml/feedback.hpp`.
> - **§1.3's diagnostics suite shipped PARTLY, 2026-07-21** (simplification-plan §6.5e). The
> `nisps_ml_loss_history` C API the section waited on now exists, and
> `manifold/src/console/TrainingHealth.tsx` renders the real per-iteration loss curve plus the
> per-layer weight-health table (`nisps_ml_get_layer_stats`) at `expanded` depth — the advanced
> surface, since only two depths shipped. `<GradientFlow>` is NOT built and is not planned as
> drawn: the core records no per-layer gradient magnitudes, and the fabricated version of it was
> deleted in the Phase-1 sweep. `<WeightHealth>` as an edge-glow is likewise unbuilt; the same
> numbers are in the table. §1.3's six live sliders were deleted as decorative (S16) — `noise
> cap` and the Xavier switch are the survivors, because they drive real engine setters.
---

524
docs/specs/plans/browser-mode-coverage-spec.md Normal file
View file

@ -0,0 +1,524 @@
---
kind: plan
status: active
---
# Browser Mode Coverage Honesty — audio topology as declared, verified truth
*Dated 2026-07-21. Spec for `plans/simplification-plan.md` §6.5b (audit finding `A2`), written
spec-first per plan §6 — **proposal only, no phase adopted**. Companion decisions live in §8.
Absorbs the old ALIGNMENT "browser-only engines incomplete / mic input" defect.*
*Every claim below was checked against the tree at `fd0aee2`. Measurements are reproducible; the
exact commands are given. Where I could not verify something I say so instead of asserting it.*
---
## §0 What is actually true today (measured, not inferred)
Two independent probes, both run 2026-07-21:
1. **Native**, engines compiled straight from `nisps/engines/*.hpp` (`g++ -std=gnu++20 -O2 -I nisps`).
2. **The committed WASM artifact** `manifold/public/nisps.{js,wasm}`, driven through the exact C ABI
the AudioWorklet uses (`nisps_engine_create` / `set_params` / `process_block`), loaded with the
same glue dance as `manifold/tests/loss-history.test.ts`.
Both were run for 2 s at 48 kHz with all params 0.5, first with a silent input, then with a
0.3-amplitude 220 Hz sine. Output RMS (left channel):
| engine | silent in | sine in | native == wasm |
|---|---|---|---|
| `paf_synth` | **0.000000000** | *(ignores input)* | yes |
| `channel_strip` | 0.000000000 | 0.446555 | yes (≤1e-6) |
| `xiasri` | 0.000000000 | 0.848909 | yes (≤1e-6) |
| `verb_fx` | 0.000346808 | 0.267881 | yes (≤1e-6) |
| `memlcelium` | **0.399765** | *(ignores input)* | yes |
| `breakor` | 0.000000000 | — | yes |
| `elysiamorf` | 0.000000000 | — | yes |
| `analysis` | 0.000000000 | 0.000000000 | yes |
| `thru` | 0.000000000 | — | yes |
Reading that against what the browser host actually supplies:
- **Nothing is ever connected to the worklet's audio input.** `EngineHost` constructs the node with
`numberOfInputs: 1` (`manifold/src/engine/engine-host.ts:117`) and connects it to the destination
(`:121`), but there is no `getUserMedia`, `MediaStream`, or `createMediaStreamSource` anywhere in
`manifold/src` (grep: zero hits). The worklet already *reads* `inputs[0]`
(`manifold/src/engine/worklet/nisps-processor.ts:266-271`) and zero-fills when absent — so the
DSP side is done and the gap is one main-thread connection.
- **The engine C ABI has four entry points only**: create / destroy / set_params / process_block
(`nisps/wasm/bindings.cpp:1074-1108`, exported at `scripts/build-wasm.sh:70-71`). There is no
`note_on`, no `set_playing`, no `update_bpm`, no `pop_events`, no feature readback.
Therefore, **of the nine schema-backed modes, exactly two produce audio in the browser today**:
`memlcelium` and `slp_workshop` (which share the `memlcelium` engine).
| mode | why it is silent / inert in the browser |
|---|---|
| `paf_synth` | Note-gated. `process()` multiplies by `env_.play()` and the ADSR idles in `WaitToTrig` → exactly 0 (`nisps/engines/paf_synth.hpp:114`, `nisps/dsp/env.hpp`). Only `note_on` starts it (`:132-138`), and no browser path calls it. |
| `channel_strip` | Pure input processor; no audio input connected. |
| `verb_fx` | Pure input processor; RMS 3.5e-4 from a zero input is residual, not a signal. |
| `xiasri` | Pure input processor; no audio input connected. |
| `breakor` | Event-only by design (`nisps/engines/breakor.hpp:81-109` returns `{0,0}`). `playing_` defaults `true` (`:159`) so it *does* tick and enqueue, but `pop_events` is not exported, so the 64-slot `EventQueue` fills and silently drops (`nisps/core/event_queue.hpp:44-49`). |
| `elysiamorf` | Same shape (`nisps/engines/elysiamorf.hpp:100-104`, `playing_ = true` at `:136`). |
| `sound_analysis_midi` | Needs mic *and* a features readback. `AnalysisEngine::process` returns silence and exposes `features()` / `copy_features()` (`nisps/engines/analysis.hpp:129`, `:188-196`) — neither is exported. |
| `visualizer` (manifold-only) | Correctly silent: it is a visual mode on `thru`. |
| `c15` (manifold-only) | Already honest — `placeholder: true`, `badge: 'soon'` (`manifold/src/console/model.ts:301-313`). |
### How the table was produced
Both probes were throwaway scripts in `/tmp`, not committed — **§5 step 2 turns them into permanent
tests, which is the point of this document.** Their shape, so they can be rebuilt in ten minutes:
- **Native.** One `.cpp` including `nisps/engines/*.hpp`; per engine: default-construct,
`setup(48000.f)`, `set_params` with a `param_count()`-sized vector of `0.5f`, then 96 000 calls to
`process({s, s})` accumulating `y.L² + y.R²`; `s` is 0 or `0.3 · sin(2π·220·i/48000)`. Built with
`g++ -std=gnu++20 -O2 -I nisps`.
- **Committed WASM.** A `bun` script using the same glue-loading dance as
`manifold/tests/loss-history.test.ts:34-45` (`new Function(...)` over `public/nisps.js`, then
`factory({ wasmBinary: public/nisps.wasm })`), then `M.cwrap` for `nisps_engine_create`,
`nisps_engine_set_params`, `nisps_engine_process_block`, `nisps_engine_destroy`; engine id written
as NUL-terminated ASCII into a 32-byte `_malloc` buffer exactly as the worklet does
(`nisps-processor.ts:217-230`); 750 blocks × 128 frames.
The two agreed to ≤1e-6 on every engine — incidental corroboration that WASM parity holds on a path
`scripts/parity-check.sh` does not cover.
---
## §1 Where the brief and the audit are wrong
Trust the code over these documents, including this one.
1. **"4 modes that structurally cannot run" is an undercount.** Audit `A2` names `breakor`,
`elysiamorf`, `xiasri`, `sound_analysis_midi`. `channel_strip` and `verb_fx` are in exactly the
same position as `xiasri` — all three are pure input processors with no input connected — and
`paf_synth`, the catalogue's flagship, is silent for a *third* reason the audit's four-way enum
cannot even express. The real number is **7 of 9**.
2. **"The UI presents them as available" is false today.** There is no instrument-mode picker in
Manifold at all. `ConsoleApp.tsx:326-329` says so explicitly; `ctx.modes` / `ctx.setModeId` are
plumbed with no renderer (`manifold/src/console/types.ts:50-57, 64-66`); the Dock's "Mode"
selector is the **output backend** picker, not the instrument (`manifold/src/console/output-mode.ts:1-18`).
The only way to reach another mode is `window.__mf.setMode()` under `?debug=1`.
**Consequence for sequencing:** the dishonesty is *latent*. It fires the moment §6.5c builds the
picker. 5b is therefore a **prerequisite of 5c**, and a "badge" alone would ship a component no
user can see. What 5b must deliver is the *data* plus the *capabilities*, so that 5c's picker has
something true to render and most modes no longer need a warning at all.
3. **The parity harness's PAFSynth stage is vacuous.** `tests/cpp/parity_check.cpp:175-190` runs
PAFSynth for 128 frames with silent input and no `note_on`, then averages — both sides push
exactly `0.0, 0.0`. Stage 3 compares zero to zero. (Stage 4, ChannelStrip, at least feeds a
0.25 DC step.) This is on top of the known limits the brief states. A parity PASS is not evidence
for anything in this document.
4. **§7.2's DriverConfig work changed the firmware, not the browser.** `Mode::driver_config()` is
now part of the `nisps::Mode` concept (`nisps/core/concepts.hpp:74-80`), defaults to
`engine().driver_config()` in `ModeBase` (`nisps/modes/base.hpp:145-165`), and firmware brings
the codec up from it (`firmware/MEMLNaut-NISPS/glue/audio_driver.hpp`). **Mic vs line is real on
hardware.** Nothing in `manifold/` or `nisps/wasm/bindings.cpp` reads `DriverConfig` — the WASM
bridge never calls it. So yes, the browser side is the remaining gap for *routing*, but see the
next point, which is why 5b cannot simply reuse `DriverConfig`.
5. **`DriverConfig` is not a topology declaration and must not be pressed into service as one.** It
is a codec-negotiation record: `mic_input`, `mic_gain_db`, `line_level`, `output_volume`,
`sample_rate` (`nisps/core/types.hpp:50-71`, struct at `:65`). It *correlates* with topology by accident — the
three input-processing engines all set `line_level = 6u` — but `paf_synth` also deviates
(`output_volume = 0.9f`) while consuming no input, `breakor`/`elysiamorf` return a default
`{}` identical to a mode that expresses no opinion, and `analysis`'s `mic_input = true` is about
*which codec pin*, not about *whether audio input is required*. Inferring topology from
`line_level != 3` would be a guess dressed as a fact. Topology needs its own declaration.
---
## §2 The model
### §2.1 Why not the flat four-class enum
The plan proposes `generator | audio-in-fx | event-only | analysis`. That enum cannot express
`paf_synth`, which is a generator that produces nothing without note events — the single biggest
user-visible gap in the catalogue. Three independent facts are being conflated:
| axis | question | values |
|---|---|---|
| `input` | does the engine need an audio input signal to do its job? | `none` \| `required` |
| `output` | what does the engine emit? | `audio` \| `events` \| `none` |
| `trigger` | does it need note events to sound? | `free` \| `notes` |
The four familiar class names survive as a **derived label**, not stored data:
```
output === 'events' → "Event sequencer"
output === 'none' && input==='required'→ "Analysis"
input === 'required' → "Audio FX"
otherwise → "Generator"
```
### §2.2 The declared values
| mode | `input` | `output` | `trigger` | derived label |
|---|---|---|---|---|
| `paf_synth` | none | audio | **notes** | Generator |
| `channel_strip` | **required** | audio | free | Audio FX |
| `verb_fx` | **required** | audio | free | Audio FX |
| `xiasri` | **required** | audio | free | Audio FX |
| `memlcelium` | none | audio | free | Generator |
| `slp_workshop` | none | audio | free | Generator |
| `breakor` | none | **events** | free | Event sequencer |
| `elysiamorf` | none | **events** | free | Event sequencer |
| `sound_analysis_midi` | **required** | **none** | free | Analysis |
### §2.3 Host capability, and a derived verdict — not a hard-coded badge
The catalogue must never carry a hand-maintained "works in browser" flag: it would be wrong the day
a capability lands. Instead, the browser declares what it can currently provide, and support is
**computed**:
```ts
interface HostAudioCapabilities {
audioInput: boolean; // a MediaStream is connected to the worklet node
noteTrigger: boolean; // nisps_engine_note_on/off exist and are wired
eventDrain: boolean; // worklet → host event channel exists
analysisFeatures: boolean; // analysis feature readback exists
}
type ModeSupport =
| { ok: true }
| { ok: false; reason: 'needs-audio-input' | 'needs-note-source' | 'hardware-only'; detail: string };
```
`modeSupport(schema.audio, host)` is a pure function; the UI renders its result. As each capability
in §5 lands, its flag flips to `true` and modes stop being labelled — no catalogue edit, no badge
list to maintain. Today the flags are `{ audioInput: <runtime>, noteTrigger: false, eventDrain:
false, analysisFeatures: false }`.
---
## §3 Where the truth lives
**In `schemas/modes/<mode>.json`, as a new required `audio` object; codegen emits C++ and TS.**
Rejected alternatives, with reasons:
- *A hand-written topology map in `manifold/src/console/model.ts`* (the audit's suggested fix) —
this is a seventh hand-maintained mode registry, straight against Phase 3, which just finished
deleting six of them (`simplification-plan.md` §4). It also puts a fact about C++ engines in
TypeScript, where no C++ test can check it.
- *A WASM export `nisps_engine_topology(id)`* — `MF_MODES` is built at module load
(`model.ts:267-316`); making it await WASM would make the whole catalogue async for a static fact.
- *Deriving it from `DriverConfig`* — see §1.5.
### §3.1 Schema shape
Added to `schemas/schema.json` (`required` gains `"audio"`, `additionalProperties: false` preserved):
```json
"audio": {
"type": "object",
"required": ["input", "output", "trigger"],
"additionalProperties": false,
"properties": {
"input": { "enum": ["none", "required"] },
"output": { "enum": ["audio", "events", "none"] },
"trigger": { "enum": ["free", "notes"] },
"dsp_engine_id": {
"type": "string", "pattern": "^[a-z][a-z0-9_]*$",
"description": "The engine that occupies the audio path for this mode. Defaults to the mode's engine_id; declared only when the mode's ModeBase engine slot is a NoOp and a separately-composed engine owns the audio (sound_analysis_midi → analysis)."
}
}
}
```
`dsp_engine_id` is optional and is set on exactly one mode, `sound_analysis_midi: "analysis"`. It
exists to **delete** a hard-coded exception — see §6.
### §3.2 Codegen anchors
`codegen/generate.ts`, mirroring what `ui` already does — each of these is one line-group:
| what | anchor |
|---|---|
| input TS type for a schema file | `interface ModeSchema` at `:93-126` (its `ui` block is `:116-120`) |
| C++ enums + `struct AudioConfig` in `schema_types.hpp` | beside `enum class PrimaryInput` `:264-272` / `struct UIConfig` `:273-277` |
| `AudioConfig audio;` member on `ParamSchema` | `struct ParamSchema` at `:288-300` |
| per-mode `inline constexpr AudioConfig k<Mode>Audio` | beside the `// UI` block, `:503-518` |
| TS `AudioConfig` interface + `ModeSchema.audio` | `export interface UIConfig` at `:374-378` |
| TS per-mode emission | beside `:631-635` |
Regenerate with `cd codegen && bun run generate.ts`; the golden snapshot
`codegen/tests/golden/paf_synth_schema.{hpp,ts}` must be updated in the same change, and CI's
codegen dirty-diff gate (added in plan §1.3) will fail on anything stale.
---
## §4 Invariants
Beyond the repo-wide constraints (platform-neutral allocation-free `nisps/`, 1e-5 parity, both
codegen outputs in one change, RT-safe worklet/dual-core comms), this area has its own:
1. **Declared topology is verified by behaviour, never by comment.** Every value in §2.2 is asserted
by a test that drives the engine (§5 step 2). Adding an engine or a mode without a topology
declaration must fail to build; declaring one falsely must fail a test.
2. **The support verdict is derived, never stored.** No "browser-ok" boolean in a schema, a
catalogue, or an overlay. See §2.3.
3. **`nisps/` gains no browser-only concept.** `audio` is a schema/codegen fact consumed by the
generated `ParamSchema` and by tests. Firmware behaviour must not change: no engine or mode
source file changes in step 1, and `nisps_modes_tests` / `nisps_dsp_engine_tests` must stay green
with no edits to existing cases.
4. **Microphone input is opt-in, per session, and never auto-enabled.** Mic → worklet → destination
is an acoustic feedback loop through the room. Enabling requires an explicit user gesture and a
headphones warning; the mic is disconnected on `EngineHost.stop()` and on mode switch away from
an `input: 'required'` mode.
5. **No new audio-thread allocation.** Anything crossing worklet↔main (events, analysis features)
uses a pre-allocated channel. `postMessage` of a freshly-allocated typed array per render quantum
is not acceptable; COOP/COEP are already set for `manifold` (see `vite.config.ts` /
`ONBOARDING.md` §2), so `SharedArrayBuffer` is available and is the intended mechanism.
6. **The exclusive `InputMode` picker stays exclusive and stays ML-only.** `internal | gamepad |
midi` (`manifold/src/console/Drawers.tsx:284-288`) selects the *ML input source*. Microphone
audio is not an ML input source (except transitively for `sound_analysis_midi`, whose features
would enter through the engine, not through `input-layer.ts`). Do not add a fourth entry.
7. **Silence must be explained, not shipped.** If a mode is selectable and cannot sound, the UI says
which capability is missing. "It just doesn't make a noise" is the bug this document exists to
remove.
---
## §5 Implementation sequence
Each step is independently landable and independently verifiable. Steps 1–3 are the honest minimum;
4–6 are scoped by the decisions in §8.
### Step 1 — declare topology (schema + codegen, both languages)
Edit `schemas/schema.json` (§3.1) and all nine `schemas/modes/*.json` with the §2.2 values, plus
`dsp_engine_id: "analysis"` on `sound_analysis_midi`. Extend `codegen/generate.ts` at the §3.2
anchors. Regenerate; commit `nisps/modes/generated/`, `manifold/src/modes/generated/`, and the
codegen golden.
*No behaviour changes.* Note honestly: `ui.primary_input` / `show_voice_space_selector` /
`show_synth_visualizer` are generated into both languages and read by **nothing** outside codegen and
its golden (verified by grep across `manifold/src`, `nisps/`, `firmware/`). `audio` must not repeat
that — it acquires two consumers in steps 2 and 3 of this same plan.
**Verification:** `bash scripts/build-cpp-tests.sh` (generated headers still compile into
`nisps_modes_tests`); `cd codegen && bun run test` (golden + curve-drift); `cd manifold && bun run
typecheck`; `bun run codegen/generate.ts` twice → no diff (idempotence).
### Step 2 — make the declaration honest (two behavioural tests)
**2a. `tests/cpp/test_mode_audio_topology.cpp`**, added to the `nisps_modes_tests` target
(`nisps/CMakeLists.txt:119-144` — that target already has the repo root on its include path and
already links the generated schemas). For every generated mode schema, resolve
`audio.dsp_engine_id ?? engine_id` to its C++ engine type and assert:
| declared | assertion |
|---|---|
| `input: 'none'` | output is *identical* driven silent vs driven with a noise bed (the engine ignores its input). For `output: 'none'` engines, compare `features()` instead of audio. |
| `input: 'required'` | output RMS with the noise bed exceeds output RMS with silence by > 1e-2. |
| `output: 'audio'` | with the mode's sounding precondition satisfied (a `note_on` when `trigger === 'notes'`, a noise bed when `input === 'required'`), RMS > 1e-3. |
| `output: 'events'` | RMS is exactly 0 under both drives **and** `pop_events()` yields > 0 events within 2 s at the default tempo. |
| `output: 'none'` | RMS is exactly 0 under both drives. |
| `trigger: 'notes'` | RMS is exactly 0 before `note_on`, and > 1e-3 within 1 s after `note_on(60, 100)`. |
| `trigger: 'free'` | covered by the `output: 'audio'` row. |
Drive params pseudo-randomly in [0.05, 0.95] from a fixed-seed `nisps::Rng`, **not** all-0.5 — that
vector is a degenerate corner and is precisely why `parity_check.cpp` stage 3 tests nothing
(`simplification-plan.md` §6.5f(3) made the same call for the bench).
The engine_id → C++ type resolution is a `switch` local to the test. Guard it: assert the set of
resolved ids equals the set of ids across all generated schemas, so a new engine cannot be added
without the test noticing.
**2b. `manifold/tests/engine-topology.test.ts`** (`bun test`, picked up by the existing
`bun test src tests/*.test.ts` glob — do not change that script). The §0 WASM probe, promoted:
drive the **committed** `manifold/public/nisps.{js,wasm}` through `nisps_engine_*` and assert the
same table at the browser's own ABI. This is the layer parity-check does not cover and the layer the
deploy actually ships.
**Verification:** `bash scripts/build-cpp-tests.sh` (5 ctest targets green); `cd manifold && bun run
test`. Both are hardware-free and run in CI. If a §2.2 value is wrong, this is where it surfaces.
### Step 3 — surface the truth in Manifold
- `manifold/src/console/model.ts`: `MFMode` gains `audio: AudioConfig` (derived in
`modeFromSchema`, `:246-259` — one line, straight off the schema).
- New `manifold/src/console/mode-support.ts`: `HostAudioCapabilities`, `modeSupport()`, and
`modeClassLabel()` (§2.1/§2.3). Pure, unit-testable, no React.
- `ConsoleApp` builds the capability record from what the engine actually exposes (today:
`audioInput` from `engine.audio.hasAudioInput`, the other three literal `false` constants that get
deleted as steps 4–6 land) and puts it on `ConsoleCtx`.
- Render: a support chip in the synth `ModeConfig` block (`manifold/src/console/Drawers.tsx:549-570`,
the "Transport" section — it already names the active mode), stating why the active mode cannot
sound. When 5c builds the picker it reads the same `modeSupport()` for its per-entry badge and for
disabling `hardware-only` entries.
**Verification:** unit test for `modeSupport()` over all nine modes × the current capability record
(`bun test`); `bun run typecheck`; e2e — extend `manifold/tests/e2e/schema-modes.spec.ts`, which
already switches modes through `window.__mf`, to assert the chip's text for one mode per class.
### Step 4 — wire microphone input (makes the Audio FX class real)
This is the cheapest real capability in the document: the worklet already consumes `inputs[0]`
(§0), so **no `nisps/` change, no new WASM export, no worklet change**.
- `EngineHost`: `setAudioInput(stream: MediaStream | null)` — hold a
`MediaStreamAudioSourceNode`, `connect(this.node)` / `disconnect()`; handle "called before
`start()`" by stashing the stream and connecting once the node exists; drop it in `stop()`.
- `EngineApi.audio` gains `setAudioInput()` + `hasAudioInput` (`manifold/src/engine/engine-api.ts:95-102,
203-207`).
- UI in the synth Transport block beside the play button, rendered only when the active mode
declares `input: 'required'`: an "Audio input" toggle that calls
`navigator.mediaDevices.getUserMedia({ audio: { echoCancellation: false, noiseSuppression: false,
autoGainControl: false } })` — the three defaults must be off or the codec fights the DSP — with
a headphones warning (invariant 4) and a clear denied/unavailable state. Requires a secure
context: production is HTTPS, dev is `http://localhost:5273`, both fine.
- Device picker: **out of scope**, default device only. Note it.
**Verification, honestly graded:**
- *Strong:* step 2b already proves these three engines transform a driven input correctly at the
committed-WASM layer.
- *Medium:* an e2e with `--use-fake-device-for-media-stream --use-fake-ui-for-media-stream` and
`context.grantPermissions(['microphone'])` can assert `engine.audio.hasAudioInput === true` and
that the mode chip clears. That proves **wiring**, not audibility.
- *Weak / not automatable here:* that sound actually comes out of the speakers with a real mic. The
worklet's output is not observable from the page. This needs an operator listening check, once,
per engine. Say so in the commit rather than implying the gates cover it.
### Step 5 — `paf_synth` note triggering *(gated on decision §8.1)*
Two exports, `nisps_engine_note_on(void*, int note, int vel)` and `nisps_engine_note_off(void*,
int note)`, dispatched with `if constexpr (requires { e->note_on(n, v); })` so only engines that
have them respond (today: `paf_synth` alone). The chain is **worklet-side**, which differs from the
main-thread five-layer chain — the main thread never creates engines:
1. `EMSCRIPTEN_KEEPALIVE` in `nisps/wasm/bindings.cpp` (beside `:1089-1108`)
2. `EXPORTED_FUNCTIONS` in `scripts/build-wasm.sh:70-71`
3. `WasmInstance.exports` interface + a `pickFn(...)` in `init_` and the re-export in the
`this.instance = { exports: { … } }` shim (`nisps-processor.ts:29-42`, `:145-165`, `:176-188`)
4. `HostToWorkletMessage` union + `onMessage_` branch (`engine-host.ts:35-53`,
`nisps-processor.ts:72-90`)
5. `EngineHost.noteOn/noteOff` → `EngineAudioApi` (`engine-api.ts:95-102`, `:203`)
Miss a layer and it fails at runtime, not compile time. `manifold/src/engine/types.ts:159-167` also
declares the engine ABI for the main-thread module; it is unused by `wasm-iml.ts` — decide whether
to keep it symmetric or leave it alone, but do not assume it is load-bearing.
RT safety: notes arrive on the worklet's existing `port`, handled between render quanta exactly as
params already are — bounded, no allocation inside `process()`.
Trigger sources: `manifold/src/inputs/midi-input-source.ts:204-211` **already** emits `note:<n>`
press/release actions with velocity from WebMIDI; `ConsoleApp` binds actions to verdicts today, so
routing note actions to `engine.audio.noteOn/Off` is a small addition there. Add one on-screen
momentary "Note" button in the Transport block as the no-hardware fallback.
Note for context: on firmware, `paf_synth` is *also* note-driven — `glue/midi_io.hpp:41-50` routes
MIDI note-in to `mode.note_on`. The browser gap is not a divergence from hardware behaviour, it is
the absence of the same route.
**Verification:** extend `tests/cpp/test_mode_audio_topology.cpp`'s `trigger: 'notes'` row (already
specified in step 2a — it will already be green before this step, since it tests the C++ engine, not
the browser); add a `manifold/tests/engine-topology.test.ts` case asserting the *exported* note
functions exist and produce non-zero output through the committed WASM; e2e asserting the chip
clears after a synthetic note. Audibility again needs an ear.
### Step 6 — event-only and analysis modes *(gated on decisions §8.2 / §8.3)*
**Default position: label them `hardware-only` and stop.** That is honest, costs nothing, and the
label disappears by itself when a capability lands (§2.3).
If instead they are to be made real:
- **Event-only (`breakor`, `elysiamorf`)** needs (a) `nisps_engine_pop_events`,
`nisps_engine_set_playing`, `nisps_engine_update_bpm` across the step-5 chain; (b) a worklet→main
event channel — a `SharedArrayBuffer` ring written by the worklet and drained on the main thread's
rAF (invariant 5; COOP/COEP are already set); (c) transport UI (play/stop/bpm) and a route from
events to `midi-backend.ts` that bypasses the per-param output path entirely, because these modes
emit notes, not parameter values. **This overlaps 5a**: on firmware the mode layer owns the event
drain (`nisps/modes/breakor.hpp:42-46`, `glue/midi_io.hpp:53-63`), so building a second,
browser-only orchestration here is exactly the duplication 5a exists to remove. **Sequence it
after 5a.**
- **Analysis (`sound_analysis_midi`)** needs step 4 *plus* `nisps_engine_copy_features` *plus* a
worklet→main path for six floats per control tick (same SAB channel) *plus* routing those into ML
input slots 0–5 alongside `joy_*` in slots 6–9 — which is `input-layer.ts`'s composition contract
and touches the exclusive-source picker (invariant 6, and the dormant multi-source decision,
§7.7 of the plan). Not a small item; do not smuggle it into 5b.
---
## §6 Proposed deletion
One, with its consumer named:
**`modeEngineId()`'s hard-coded exception** — `manifold/src/console/model.ts:392-395`:
```ts
export function modeEngineId(modeId: string): string {
if (modeId === 'sound_analysis_midi') return 'analysis';
return MF_MODES.find((m) => m.id === modeId)?.engineId ?? 'thru';
}
```
Consumer: `ConsoleApp.tsx:242` (`engine.audio.setBackend(...)`). Replaced by
`schema.audio.dsp_engine_id ?? schema.engine_id` (§3.1). This is the same class of hand-exception
Phase 3 spent two days removing, and it is the only place in the app where a mode's audio engine is
decided by an `if`.
Nothing else in this document deletes anything. In particular, do **not** delete `nisps/engines/`
sequencer or analysis code because the browser cannot use it — firmware runs all of it.
---
## §7 Adjacent, verified, deliberately out of scope
Two findings from the same files. Recording them so the implementer does not rediscover them
mid-change; neither is part of 5b.
1. **`SCHEMA_MODE_OVERLAYS.input` duplicates `ui.primary_input` and has already drifted.**
`model.ts:229-244` hand-declares `breakor: { input: 'joystick' }` while
`schemas/modes/breakor.json` declares `"primary_input": "xy_pad"`. The overlay value is live —
`ConsoleApp.tsx:814` feeds it to `resolveInputMap()` (`settings/settings-store.ts:146-153`) to
pick the input-map shape — so the two sources disagree about a rendered behaviour. Fixing it
means deriving `MFMode.input` from the schema and deleting the overlay field.
2. **`ui` is generated into both languages and consumed by nothing.** `primary_input`,
`show_voice_space_selector`, `show_synth_visualizer` have zero readers outside `codegen/` and its
golden. Either wire them (item 1 above wires `primary_input`) or delete them. Doing this in the
same commit as step 1 is cheaper than separately — same files, same regeneration — but it is a
different decision and should be asked for explicitly.
---
## §8 Operator decisions
These genuinely need a call; everything else in this document I decided and justified above.
1. **`paf_synth` note triggering (step 5): build it, or label the flagship mode "needs a note
source"?** This is the most user-visible silence in the catalogue and the most likely to be read
as "the app is broken". Building it is two exports plus a UI button, and the MIDI note plumbing
already exists. Labelling it is free but leaves the best-developed engine unusable in the browser.
*Recommendation: build it.* Counter-argument worth weighing: note routing is arguably 5a's
territory (firmware does it in `ModeBase`/`midi_io.hpp`), so building it here creates a small
browser-only path that 5a will later absorb.
2. **Event-only modes (`breakor`, `elysiamorf`): label hardware-only, or build the event channel?**
*Recommendation: label now, build after 5a* — the browser-side orchestration is precisely what 5a
is meant to stop hand-writing.
3. **`sound_analysis_midi`: label hardware-only, or build the analysis path?** It needs mic +
feature readback + ML-input composition, and it collides with the dormant multi-source input
decision (plan §7.7). *Recommendation: label now.*
4. **`audio.dsp_engine_id` — adopt the field (and delete the `modeEngineId` exception), or keep the
exception and give the test a local special case?** Adopting it is the single-source answer and
is what §3.1/§6 assume; the cost is one more optional schema field used by exactly one mode.
5. **Bundle the §7 `ui` clean-up into step 1?** Same files and same regeneration, so bundling is
cheaper — but it widens a change that is otherwise purely additive.
6. **How much microphone UX?** Step 4 ships default-device-only with no picker and no input meter.
A device picker and a level meter are obvious follow-ons; say now whether they are wanted, since
a meter needs a tap on the input node that the current host does not have.
---
## §9 What this plan does not do
- It does not build the instrument-mode picker. That is §6.5c (`plans/curated-presets-spec.md`);
this plan gives it truthful data, and 5c should not ship a picker without it.
- It does not touch `nisps/modes/` orchestration. That is §6.5a
(`plans/mode-layer-reunification.md`), which is also where the browser event pump belongs — see
§5 step 6 and decision §8.2.
- It does not change firmware behaviour in any step. Every mode remains fully functional on hardware.
- It does not improve the parity harness, despite §1.3. Fixing stage 3 to trigger a note before
averaging is a one-line change with a golden-baseline consequence; it belongs with whoever next
touches `tests/cpp/parity_check.cpp`, and would bump `kVersion` (`:93`).

780
docs/specs/plans/curated-presets-spec.md Normal file
View file

@ -0,0 +1,780 @@
---
kind: plan
status: active
---
# Curated Presets, the Instrument Library, and Per-Surface Disclosure
*Dated 2026-07-21. The spec for `simplification-plan.md` §6.5c ("5c Curated/advanced split",
audit findings A3 + A7, `ALIGNMENT.md` defect 2). **Proposal only — no code is authorised until the
operator adopts it.** Every code claim below was read out of the tree on 2026-07-21 and is cited
with `file:line`; where a repo document disagrees with the code, the code is recorded as the truth
and the disagreement is called out in §1.6. British spelling in product copy. The built-in synth is
"Powerful Synth Engine" / "Built-in Synth" — the string "C15" must never reach the UI.*
**Reading order for an implementer:** §1 (ground truth) → §2 (product model) → §3–§6 (design) →
§9 (sequenced plan). §7 is the invariant list to check work against; §11 is what the operator still
has to decide.
---
## 0. Scope, and what this supersedes
**In scope** (one item, three limbs that only make sense together):
1. A **preset data model** — what a curated preset is as data, where it is stored, how it is
validated against schema truth, and how it is applied.
2. The **Instrument Library** — the missing in-UI instrument picker, built as a preset browser
(choosing a preset is how you choose an instrument).
3. **Progressive disclosure** via per-surface depth *levels*, replacing the accidental use of the
drawer geometry toggle as the advanced flag.
**Out of scope, deliberately:**
- Browser mode-coverage honesty / audio topology (`simplification-plan.md` §6.5b, spec'd in
`plans/browser-mode-coverage-spec.md`). The Library renders whatever signal 5b produces; it does
not invent one. **That spec's §0 calls itself a prerequisite of this item, and it is right** — the
picker is what makes the latent dishonesty visible. Sequencing recommendation: land 5b before §9
step 6. If it has not, the Library lists every catalogue entry with no viability claim (§5.4) —
honest by omission rather than wrong, but a worse first impression.
- The mode-layer reunification (§6.5a) and the hardware editor (§6.5d). Presets are browser-local
data in this item; §11.Q4 records the promotion path if they later have to reach firmware.
- Region/param pins and the "control surface" (boldness/memory/precision) of
`manifold-parity-features-spec.md` §2 and §5. Neither is implemented, neither is needed here.
**Supersedes** `plans/manifold-parity-features-spec.md` §1 ("Session presets (composed layers)").
That section has **no code consumer** — nothing under `manifold/src/` implements or references it
(verified: no `SessionPresetV1`, no `mf-session` key, no `persist<T>` helper in the tree). Its
composed-layer idea survives here in reduced form; three of its specifics do not, and the
divergences are explicit:
| §1 of manifold-parity-features-spec | Disposition here |
|---|---|
| `layers.weights?: string` (base64url Float32Array, opt-in) | **Deleted.** Contradicts the operator's §7.6 decision that a curated preset is configuration only, network untrained (§2.1). |
| `PipelineLayer` = params of `engine/input-pipeline.ts` / `output-pipeline.ts` | **Deleted as written** — both files were deleted at one-core-engine P4; the config now lives in `manifold/src/engine/pipeline-types.ts` and has **zero UI writers** (§1.5). Nothing can author it, so a preset cannot meaningfully carry it yet. |
| `ControlLayer` = `{ boldness, memory, precision, presetId, offsets }` | **Deferred** — the control surface (§5 of that spec) is unimplemented and out of scope. |
| URL sharing (`?session=`, `?boldness=`) | **Deferred** to §11.Q5; the model is designed not to preclude it. |
On adoption, `manifold-parity-features-spec.md`'s status block gains a line pointing §1 here. That
edit belongs to whoever lands step 3 of §9, not to this document.
---
## 1. Ground truth (read out of the tree, 2026-07-21)
### 1.1 There is no instrument picker, and the active instrument is invisible
- The instrument mode is a plain `useState` in the console spine:
`manifold/src/console/ConsoleApp.tsx:99` — `const [modeId, setModeId] = useState('paf_synth')`.
- `ConsoleCtx` carries `modes: MFMode[]` and `setModeId` (`manifold/src/console/types.ts:64-66`),
populated at `ConsoleApp.tsx:735-737`. **Neither has a single reader.** `grep -rn "ctx.modes|ctx.setModeId" manifold/src`
returns only the comment at `ConsoleApp.tsx:326` that says so.
- The only place the active instrument's name is rendered anywhere in the app is one sentence of
explanatory prose — `Drawers.tsx:566`, `The active engine follows the selected mode ({ctx.mode.label})` —
which is itself gated behind `depth === 'expanded'` *and* the synth output target. In the default
session (particles target, condensed drawers) the user cannot discover which instrument is loaded.
- `MFMode.glyph`, `MFMode.cls`, `MFMode.placeholder`, `MFMode.badge`
(`manifold/src/console/model.ts:136-160`) have **zero consumers**. They are display fields
maintained for a renderer that does not exist. The Library is their first consumer, or they should
be deleted; see §8.4.
- Mode switching today happens only through the `?debug=1` seam
(`ConsoleApp.tsx:331-345`, `window.__mf.setMode/getModeId/paramCount/modeIds`), which
`manifold/tests/e2e/schema-modes.spec.ts` drives. **That seam must survive** (§7.8).
### 1.2 Mode switching itself works, and reshapes the net
Three effects fire on a `modeId` change, in declaration order:
| Effect | Lines | Does |
|---|---|---|
| audio backend | `ConsoleApp.tsx:241-243` | `engine.audio.setBackend(modeEngineId(modeId))` |
| net reshape | `ConsoleApp.tsx:252-257` | `engine.reshape(mode.ml)` — schema dims, warm-started |
| transient reset | `ConsoleApp.tsx:260-271` | rebuilds `params` from the mode, recentres input, clears examples/pins/markers, closes drawers |
`engine.reshape` (`engine-api.ts:323` → `wasm-iml.ts:355` → `nisps_ml_reshape`,
`nisps/wasm/bindings.cpp:397`) constructs a fresh `MLPCore` at the new dims and swaps it in.
Because the dataset lives **inside** `MLPCore` (`nisps/ml/mlp.hpp:143`, `:441`, `:443`), a reshape
destroys every stored example; `wasm-iml.ts:431` clears the TS mirror to match. This is the single
most important fact for preset application ordering — see §4.2.
### 1.3 The only preset machinery is per-backend output routing
`manifold/src/backends/presets.ts` (162 lines) is real, consumed, and narrow:
- `OutputPreset` = `{ name, backend, rows: OutputPresetRow[], settings?, savedAt }` (`:16-37`).
- Storage is localStorage, **namespaced per backend**: `manifold-output-presets:<backend>` (`:38-41`).
- `OutputPresetRow` (`:16-29`) captures `name, status, muted, armed, min, max, curve, val, midi, osc, vcv`.
- The sole UI is `PresetBar` in `manifold/src/dock/OutputsBackendConfig.tsx:99-208`, rendered from
`:686`, itself gated behind `depth === 'expanded'` in `Drawers.tsx:634`.
- `PresetBar.backendSettings()` (`OutputsBackendConfig.tsx:113-118`) persists MIDI port/ccCount, OSC
url/sendRaw, VCV url/sendRaw — and `{}` for everything else.
**The `cv` bug is confirmed and is worse than "a missing field".** `MFParam.cv` (`model.ts:133`,
`{ channel, gateThreshold }`) is live state: `useBackendManager.ts:161` reads `p.cv` per output to
build the uSEQ channel map, and `CvConfig` (`OutputsBackendConfig.tsx:590`) edits it. But
`OutputPresetRow` has no `cv` (`presets.ts:16-29`), `rowsFromParams` does not copy it (`:72-86`),
and `applyPreset` does not restore it (`:141-162`). Saving a preset for the `cvgate` backend
therefore silently discards **the entire channel assignment**, which is the only thing a CV preset
is for. Fix = three one-line additions (§9 step 1), exactly as the audit said.
### 1.4 Two different things are called "Mode", and one backend is unreachable
- **Instrument mode**: `MF_MODES` / `modeId`, schema-backed (`model.ts:316`).
- **Output mode**: `OutputMode` (`types.ts:26`) / `OUTPUT_MODES` (`console/output-mode.ts:33-73`),
which is what the dock's top button is labelled — `Dock.tsx:126` renders the header
`Mode · output target`.
Adding an instrument picker beside a control already called "Mode" is a naming collision the design
has to resolve (§2.2).
Also observed, and not recorded anywhere: **`BackendId` includes `'vcv'`
(`dock/output-state.ts:31`) but no `OutputModeDescriptor` maps to it** (`output-mode.ts:33-73`
lists particles/midi/osc/cv/synth/editor). `outputBackend` is derived solely from the selected
output mode (`ConsoleApp.tsx:137`), so the VCV backend — `vcv-backend.ts`, 240 lines, registered at
`backends/manager.ts:68`, with its own config panel at `OutputsBackendConfig.tsx:691` — **cannot be
selected from the UI at all.** That is out of this item's scope to fix, but the Library work will
put the output-target list in front of the operator, so it should be reported rather than quietly
inherited.
### 1.5 What state a preset could capture, and what nothing can author
| Candidate state | Where it lives | Authorable in the UI today? |
|---|---|---|
| instrument mode id | `ConsoleApp.tsx:99` | only via `?debug=1` |
| output target | `ConsoleApp.tsx:136` | yes (dock top button) |
| per-output routing (`MFParam` status/muted/armed/min/max/curve/val + midi/osc/vcv/cv) | `ConsoleApp` `params` store; `OutputControlRow` | yes |
| backend settings (MIDI port + CC count, OSC/VCV url + sendRaw) | `ConsoleApp` state, `ConsoleCtx` | yes |
| input source + gamepad stick mode + MIDI learn bindings | `inputs/useInputLayer.ts:113-120` | yes |
| feedback mode, solo mode, explore intensity, noise cap, Xavier spread | `ConsoleApp.tsx:104-133` | yes |
| **input/output pipeline config** (zoom, deadzone, curve, smoothing, slew, freeze) | `engine/pipeline-types.ts` | **no** — `grep` for `setInputConfig|setOutputConfig|defaultInputConfig` outside `src/engine/` finds only `tests/`. There is no UI writer. |
| **voice space** | `nisps/modes/base.hpp:279-291` | **no** — there is no `nisps_*_voice_space` export in `nisps/wasm/bindings.cpp`, and the worklet reaches the audio engine through raw WASM exports (`worklet/nisps-processor.ts:33-36`), so exposing it means walking the full export chain twice. See §11.Q3. |
| network weights | `wasm-iml.ts` | yes (`randomise`, training) — **deliberately excluded**, §2.1 |
Schema fields `ui.show_voice_space_selector` and `ui.show_synth_visualizer` exist in every mode
schema (e.g. `schemas/modes/paf_synth.json`) and reach TS as `UIConfig`
(`manifold/src/modes/generated/types.ts:52-56`) but have **no consumer on either target**.
### 1.6 Where documents disagree with the code
Recorded here so the next session does not re-derive them. None of these are this item's job to fix
except where §8/§9 says so.
1. **`ONBOARDING.md` §4** says `OUTPUT_MODES` = "particles (default) / midi / osc / synth / editor" —
five. The code has six; `cv` (uSEQ CV/gate) is missing from the doc. `output-mode.ts:32`'s own
comment ("The five Modes") is wrong the same way, one line above a six-element array.
2. **`model.ts` overlay vs schema — a live dual truth with real drift.** `ModeOverlay.input`
(`model.ts:211-216`) is hand-written per mode (`:229-247`) while every schema already declares
`ui.primary_input` (`manifold/src/modes/generated/types.ts:52-56`). They agree for eight of nine
modes and **disagree for `breakor`**: the schema
says `xy_pad`, the overlay says `joystick` (`model.ts:235`). The overlay wins today, and it is
the input to `resolveInputMap(settings.inputMap, mode.input)` (`ConsoleApp.tsx:814`), so Breakor
renders a circular joystick surface where its schema asks for a rectangular pad. Single consumer,
five-line fix (§9 step 2).
3. **`dock-spec.md` §0** prescribes three depths (`peek | expand | FULL`) as **per-drawer** state.
Two shipped, and they are **one global state** shared by all five drawers
(`ConsoleApp.tsx:110`), reset to `condensed` on every drawer switch (`Dock.tsx:204`) and every
mode switch (`ConsoleApp.tsx:268`). Its own 2026-07-21 grounding note already records the
two-depth divergence; the *global-not-per-drawer* part is not recorded anywhere.
4. **`manifold-parity-features-spec.md` §1.1** describes a `PipelineLayer` over
`engine/input-pipeline.ts` and `output-pipeline.ts`. Both files were deleted at P4.
5. `simplification-plan.md` §7.6 still lists the curated-preset product model as an *open* operator
decision. The operator has since decided it (§2.1). Reconciling the plan document is the
orchestrator's, not this spec's.
---
## 2. The product model
### 2.1 What a curated preset is (operator decision, §7.6)
> **A curated preset is configuration only: mode + parameters + routing, with the network
> UNTRAINED — though it may sometimes ship seed training data.**
Consequences taken as binding:
- **A preset never carries weights.** Not optionally, not opt-in. The type has no weights field
(§3.1) and the apply routine never calls `setWeights`. This is testable — see §7.1.
- **A preset may carry seed examples**: `(input vector, output vector)` pairs at the mode's schema
arity. Loading them puts them in the dataset; it does **not** train (§4.4).
- Therefore applying a preset is *deterministic and cheap*: it reshapes the net to the mode's schema
dims (which mode switching already does) and writes UI/routing state. The sound you get is the
untrained net through the preset's routing — the preset defines the *instrument and the mapping
space*, and the user teaches it.
### 2.2 Naming: Instrument vs Output
To end the collision in §1.4, product copy uses two words and never the word "Mode":
| Concept | UI label | Code |
|---|---|---|
| which instrument/mode is loaded | **Instrument** | `modeId`, `MF_MODES` (unchanged — `mode_id` is schema truth) |
| where the outputs go | **Output** | `OutputMode` → renamed `OutputTarget` (§8.3) |
`Dock.tsx:126`'s `Mode · output target` header becomes `Output`. This is copy plus a mechanical
type rename; it changes no behaviour and is typecheck-verified.
### 2.3 The Library is the picker
There is no separate "instrument picker" and "preset browser". One surface — the **Library** —
lists curated presets grouped by instrument; choosing one selects its instrument. Below the presets,
at the advanced level, sits the bare instrument list (load an instrument with no preset). This is
one surface instead of two, and it makes the curated path the default path structurally rather than
by exhortation, which is what vision bullet 3 asks for.
---
## 3. Data model
### 3.1 Types
New module `manifold/src/presets/types.ts` — types + pure validation only, no React, no storage:
```ts
import type { ParamStatus } from '../console/model';
import type { OutputTarget } from '../console/types';
import type { CvSpec, MidiCcSpec, OscSpec, VcvSpec } from '../dock/output-state';
/** One output's deviation from the mode default. `i` indexes the mode's schema params. */
export interface OutputOverride {
i: number;
/** Display name override (MIDI/OSC outputs get renamed by users). */
label?: string;
status?: ParamStatus;
muted?: boolean;
armed?: boolean;
min?: number;
max?: number;
curve?: number;
/** Held value; only meaningful when status === 'fixed'. */
val?: number;
midi?: MidiCcSpec;
osc?: OscSpec;
vcv?: VcvSpec;
cv?: CvSpec;
}
export interface PresetRouting {
/** Status for every output NOT named in `outputs`. Omit to leave mode defaults alone. */
rest?: ParamStatus;
outputs: OutputOverride[];
}
export interface PresetInputs {
mode: 'internal' | 'gamepad' | 'midi';
gamepadStickMode?: 'single' | 'double';
}
export interface PresetLearning {
feedbackMode?: 'explore-and-place' | 'geometric-dislike';
exploreIntensity?: number;
noiseCap?: number;
spread?: boolean;
}
/** Seed examples, at the mode's schema arity. Validated in `validatePreset`. */
export interface PresetSeed {
inputs: number[][]; // each length === schema.ml.input_size
outputs: number[][]; // each length === schema.ml.output_size
}
export interface Preset {
v: 1;
id: string; // slug, e.g. 'paf-synth/glass-formants'
name: string;
/** Optional one-line description shown in the Library. */
blurb?: string;
modeId: string; // MUST be an MF_MODES id
target: OutputTarget;
routing?: PresetRouting;
backendSettings?: Record<string, unknown>; // same blob shape PresetBar already writes
inputs?: PresetInputs;
learning?: PresetLearning;
seed?: PresetSeed;
origin: 'builtin' | 'user';
savedAt?: number; // user presets only
}
```
**There is no `weights` field, by design.** A reviewer should treat any future addition of one as a
reversal of §2.1 requiring the operator.
Rationale for the choices that are not obvious:
- **`OutputOverride` carries its own index and layers are optional**, so a hand-authored curated
preset for a 56-output mode is a dozen lines, not 56 rows of boilerplate. Index (not name) is
canonical because `MFParam.name` is user-editable for MIDI/OSC (`presets.ts:147-149`) while the schema
param order is stable codegen output.
- **`rest`** lets a curated preset say "these six outputs are live; everything else is off" in one
field — which is what curation mostly *is*.
- **Backend settings stay an untyped blob**, matching `OutputPreset.settings`
(`presets.ts:35`) and the writer/reader pair at `OutputsBackendConfig.tsx:113-131`. Typing it
properly is a separate, larger job with no consumer asking for it.
- **The layer types are `Preset`-prefixed** because `InputLayer` is already taken: it is the input
composition hub class in `manifold/src/inputs/input-layer.ts`. Do not reuse the name.
### 3.2 Storage
- **Built-in (curated) presets**: TypeScript data modules, `manifold/src/presets/builtin/<mode_id>.ts`,
re-exported from `builtin/index.ts` as `BUILTIN_PRESETS: readonly Preset[]`. Typed, typechecked,
tree-shaken, no build step, no codegen.
*Why not `schemas/` + codegen?* Codegen exists to keep firmware and browser honest about a shared
contract. Presets have no firmware consumer today (firmware modes are compile-time
`-DMEMLNAUT_MODE_TYPE` selections with no on-device persistence — `ALIGNMENT.md` defect 3).
Adding a C++ emitter for data nothing in C++ reads would be complexity without a requirement.
**Removal condition:** when §6.5d (hardware editor) gives firmware a command surface and
persistence, promote presets to `schemas/presets/*.json` + codegen with both outputs in the same
change, per the standing schema rule.
- **User presets**: one localStorage document, key `mf-presets`, shape
`{ v: 1, presets: Preset[], lastAppliedId?: string }`, written through a store modelled on
`settings/settings-store.ts` (framework-neutral class + `useSyncExternalStore` hook + module-level
singleton). No new persistence mechanism.
- The per-backend namespaces `manifold-output-presets:<backend>` are **retired** (§8.1). See §11.Q1
— deleting them discards any presets the operator saved in the live app.
### 3.3 Validation against schema truth
`validatePreset(p, schemas): string[]` in `presets/types.ts`, pure, returns human-readable problems:
- `modeId` resolves in `MF_MODES`.
- every `OutputOverride.i` is `< schema.params.length`.
- `min <= max`, all of `min/max/curve/val` within `[0,1]` (the routing-knob space — `MFParam`'s
`min/max/curve/val` are 0..1 routing semantics; schema engine units live in the `schema*` fields,
`model.ts:107-121`).
- `seed.inputs[k].length === schema.ml.input_size`, `seed.outputs[k].length === schema.ml.output_size`,
`inputs.length === outputs.length`, and `inputs.length <= 128` — the shared dataset capacity
(`nisps::ml::kDefaultMaxExamples`, `nisps/ml/storage.hpp:57`, mirrored at `wasm-iml.ts:52` and
reported through `nisps_ml_describe`, `bindings.cpp:926`).
- `target` resolves in `OUTPUT_TARGETS`.
A unit test runs `validatePreset` over every entry of `BUILTIN_PRESETS` (§9 step 4). This is the
mechanism that stops curated data rotting when a schema changes — the failure is a red `bun test`,
not a silent misbehaviour.
---
## 4. Applying a preset
### 4.1 One imperative entry point, not an effect chain
`manifold/src/presets/apply.ts` exports:
```ts
export interface PresetApplier {
engine: EngineApi;
selectInstrument(modeId: string): void; // reshape + rebuild params + reset transients
setParams(next: MFParam[]): void;
setTarget(t: OutputTarget): void;
setBackendSettings(s: Record<string, unknown> | undefined): void;
setInputs(l: PresetInputs | undefined): void;
setLearning(l: PresetLearning | undefined): void;
}
export function applyPreset(a: PresetApplier, p: Preset): void;
```
`ConsoleApp` provides the applier; the sequencing logic lives in `apply.ts` where it can be unit
tested against a fake applier. `ConsoleApp.tsx` is already 1082 lines — this must not add to it.
### 4.2 The ordering invariant (the thing that will silently break)
`engine.reshape` destroys the dataset (§1.2). Today the reshape happens in an **effect**
(`ConsoleApp.tsx:252-257`), which runs *after* the event handler that changed `modeId`. So the naive
implementation — "handler sets modeId, then adds seed examples" — loads the seeds and then has them
wiped by the effect one tick later, silently, with a dataset counter that briefly reads right.
**Fix, and it is a simplification rather than a workaround:** collapse the three mode-switch effects
(`ConsoleApp.tsx:241-243`, `:252-257`, `:260-271`) into one imperative `selectInstrument(modeId)`
function that does, in order: `engine.audio.setBackend` → `engine.reshape` → rebuild `params` from
the mode → reset transients → `setModeId`. Keep exactly one effect, keyed on `[engine]`, that calls
`selectInstrument(modeId)` once when WASM becomes ready (the boot path the current effects rely on).
`applyPreset` then runs entirely inside one handler, in a guaranteed order, with no implicit
cross-effect contract.
`window.__mf.setMode` (`ConsoleApp.tsx:336`) repoints to `selectInstrument`.
**This is a refactor of live behaviour and must be verified as one** — see §9 step 3.
### 4.3 Apply order
1. `selectInstrument(p.modeId)` — reshape lands here, dataset is now empty.
2. `setTarget(p.target)`; `setBackendSettings(p.backendSettings)`.
3. Build the params array: start from the mode's schema defaults (what step 1 produced), apply
`routing.rest` to every output, then apply each `OutputOverride` by index. One `setParams`.
4. `setInputs(p.inputs)`, `setLearning(p.learning)`.
5. If `p.seed`: `engine.addExample(inputs[k], outputs[k])` for each pair. **No training.**
6. Record `lastAppliedId`.
Steps 3–5 must not be reordered: 5 after 1 (else wiped), 3 after 1 (else overwritten by the mode
rebuild).
### 4.4 Seeds are loaded, not trained
After apply, the Learning drawer shows `N examples` (its existing chip, `Drawers.tsx:219`, fed
by `ctx.datasetCount`), and the user trains or gives verdicts. Nothing auto-trains, because §2.1
says the network ships untrained and auto-training would make that false. §11.Q2 records the
alternative if the operator wants a preset to be able to opt into one training pass.
---
## 5. The Library (the picker)
### 5.1 Placement
A second pinned control at the top of the dock rail (`Dock.tsx:405`), **above** the existing output
selector, rendering the active instrument's glyph and opening a popover — the same interaction and
the same visual language as `ModeSelector` (`Dock.tsx:64-174`), which becomes `OutputSelector`. Two
labelled controls, top of rail: **Instrument**, **Output**.
Rejected alternatives, briefly: a sixth drawer (a preset browser is a chooser, not a workbench, and
drawers are mutually exclusive with the surface you are auditioning against); a stage-header title
bar (the stages are full-bleed by design and CompositeStage owns its own chrome).
### 5.2 Contents
At **basic** level:
- Curated presets, grouped by instrument, each row: preset name, instrument label + glyph, `blurb`.
- The active preset marked; applying another switches instrument as a side effect.
- A "seeds" chip on presets that carry `seed`, showing the example count — so the difference between
"config only" and "config + seeds" is visible before you commit to it, not after.
At **advanced** level, additionally:
- **Instruments** — the bare `MF_MODES` list (`ctx.modes`, finally consumed), each entry loading the
instrument with no preset. This is the authoring entry point: load a bare instrument, configure,
save as a user preset.
- **Save current as preset** / rename / delete for user presets, reusing `PresetBar`'s interaction
pattern (`OutputsBackendConfig.tsx:99-208`) rather than inventing a second one.
### 5.3 Applying is not silent
Applying a preset switches instrument, which resets the dataset and the net. Today that is already
true of a mode switch and is done without confirmation ("switching instrument is already a
deliberate act", `ConsoleApp.tsx:250`). Keep that: **no modal** — but the Library rows carry a
persistent one-line warning ("loading resets examples and the network") and, when
`ctx.datasetCount > 0`, the row click routes through the existing `ReshapeModal` pattern
(`console/ReshapeModal.tsx`) with preset-specific copy. Losing 40 hand-taught examples to a
mis-click is the failure this prevents; losing an untouched default session is not worth a modal.
The condition is deliberately `datasetCount` alone — "unsaved routing edits" would need a
dirty-tracking mechanism the app does not have and this item does not justify building.
### 5.4 Viability signalling
The Library renders `MFMode.badge` when present (`model.ts:158`) and disables entries with
`placeholder: true` (`:157`) — both fields exist and neither has a consumer today (§1.1). If §8.5
lands, `c15` is the only `placeholder: true` entry and the disable path becomes
dormant-but-correct; keep it, it costs one line.
The Library does **not** invent a viability model. `plans/browser-mode-coverage-spec.md` §2.3
proposes exactly the right shape — a pure `modeSupport(schema.audio, host)` returning
`{ ok: true } | { ok: false; reason; detail }`, computed from declared per-mode audio topology and
declared host capabilities rather than a hand-maintained flag — and the Library should render that
result verbatim: a chip carrying `detail`, the row disabled when `ok` is false. If 5b has not
landed, entries render unmarked. **Do not add a topology map to `model.ts` in the meantime** — that
is precisely the seventh hand-maintained mode registry Phase 3 spent two days eliminating, and 5b
rejects it by name.
### 5.5 Boot
Boot applies `lastAppliedId` if it resolves, else `DEFAULT_PRESET_ID`.
**`DEFAULT_PRESET_ID` must be a `paf_synth` preset.** The e2e suite clears localStorage before every
run (`tests/e2e/helpers.ts:31-41`), so boot always takes the default path, and `ONBOARDING.md` §5
plus several tests pin the boot shape at `4→[10,10,14]→33`, weights 809. Choosing a default on any
other instrument silently changes the boot contract several tests assert. If a different default is
wanted, the boot-shape claims in `ONBOARDING.md` and the affected specs move in the same change.
---
## 6. Progressive disclosure: per-surface levels
### 6.1 What `depth` actually is today
`DrawerDepth = 'condensed' | 'expanded'` (`types.ts:38`) is **one global state**
(`ConsoleApp.tsx:110`) that simultaneously controls panel geometry (360px side panel vs 80vw×80vh
centred modal, `Dock.tsx:265-299`) and content disclosure. §6.5e adopted `expanded` as "the
advanced-surface flag" for the training-health panel, and `ONBOARDING.md` now documents that as the
rule.
It is not a rule; it is an accident. Classifying every gate in `Drawers.tsx`:
| Site | Content | What the gate is really for |
|---|---|---|
| `:171` | feedback-mode explanation | **prose** |
| `:208` | explore/Jolt explanation | **prose** |
| `:229` | solo/arm scope, solo behaviour, noise-cap slider | **advanced** |
| `:254` | feedback lab, Xavier switch, `<TrainingHealth/>` | **advanced** |
| `:396` | active input source's status message | **status — should never have been hidden** |
| `:403` | internal-pad explanation | **prose** |
| `:410` | gamepad stick mode + button legend | **function** — you cannot pick one/two sticks without opening a modal |
| `:441` | MIDI device picker + MIDI-Learn | **function** — same |
| `:523` | dedicated-dimensions explanation | **prose** |
| `:564`, `:583`, `:592`, `:598` | per-target explanations | **prose** |
| `:620-621` | rows sliced to 6 when condensed | **geometry** |
| `:634` | `<OutputsBackendConfig/>` (CC table, CV channel map, preset bar) | **function + geometry** |
| `:641` | row list max-height 460 vs 220 | **geometry** |
| `:654` | `showCurve` on each row | **geometry** |
| `:659` | "+N more — expand to edit" | **geometry** |
| `:691` | unfocused icon colour + preview | **advanced (cosmetic detail)** |
| `:720`, `:737` | settings explanations | **prose** |
Three of twenty-one gates are genuinely about expertise. The rest are geometry, prose, or — in three
cases — real functionality hidden by a size switch. Using this as the advanced flag means the
advanced surface and the big-panel surface can never diverge, and it means beginners' explanatory
prose is hidden from beginners, which is exactly backwards.
### 6.2 The split
Two orthogonal axes, one existing mechanism each:
- **`presentation: 'panel' | 'modal'`** — today's geometry toggle, renamed. Per-drawer, transient,
driven by the ⤢ tab (`Dock.tsx:302-328`) and `\` (`ConsoleApp.tsx:642`). Behaviour unchanged.
- **`level: 'basic' | 'advanced'`** — the disclosure axis. **Per surface, persisted** in the existing
settings document (`settings-store.ts`, key `mf-settings`) as
`levels: Record<DrawerKey | 'library', DrawerLevel>`. No new store, no new persistence pattern,
no global boolean. The Library is a surface with a level for the same reason the drawers are, so
§5.2's advanced tier needs no second mechanism.
Two levels, not `dock-spec.md` §0's three: there is exactly one advanced tier of content to put
anywhere, and inventing a third empty level is the kind of speculative structure this repo has been
deleting all week. `dock-spec.md` §0's `peek` maps onto nothing that shipped.
Control: a small `basic | advanced` segmented toggle in each drawer's header
(`Dock.tsx:355-368`), beside the existing depth label at `:364` — which currently renders the
literal string `condensed`/`expanded` and becomes the presentation state's label.
### 6.3 Re-gating, item by item
Applying §6.1's classification:
- **advanced** (`:229`, `:254`, `:691`): gate on `level === 'advanced'`. `<TrainingHealth/>` keeps
its advanced gating — the §6.5e decision survives intact, only its mechanism changes.
- **function** (`:410`, `:441`, `:634`): ungate. Configuring your gamepad or your CV channel map is
not advanced and must not require a modal. They stay in whatever presentation the drawer is in.
- **status** (`:396`): ungate.
- **geometry** (`:620-621`, `:641`, `:654`, `:659`): gate on `presentation === 'modal'`. Unchanged
behaviour, renamed condition.
- **prose** (`:171`, `:208`, `:403`, `:523`, `:564`, `:583`, `:592`, `:598`, `:720`, `:737`):
gate on `level === 'basic'` — *shown* to beginners, *hidden* from the advanced surface, which is
the inversion of today. This is a judgement call, not a derivation; §12.3 records it as mine and
the cheap fallback (always show) if the operator disagrees.
### 6.4 Level does not reset
`depth` is reset to `condensed` on every drawer switch (`Dock.tsx:204`) and every mode switch
(`ConsoleApp.tsx:268`). `presentation` keeps that behaviour. `level` **must not** reset — a persisted
stance that resets when you change instrument is not a stance. This is the concrete reason level
lives in `settings-store` and not in `ConsoleApp` state.
---
## 7. Invariants
The repo-wide hard constraints (platform-neutral allocation-free `nisps/`, ≤1e-5 native↔WASM parity,
schema changes shipping both codegen outputs, bounded RT-safe worklet comms, dual-core ownership)
are the floor. This item's own:
1. **A preset never changes network weights.** Concretely: from a fixed starting state,
`applyPreset(p)` and `selectInstrument(p.modeId)` must leave `getWeights()` **bit-identical**.
(Weights *do* change across the reshape — `nisps_ml_reshape` redraws and warm-starts,
`bindings.cpp:397-418` — but the preset must contribute nothing beyond it.) This is the machine
check on §2.1 and belongs in `bun test`.
2. **Seeds load after the reshape or not at all** (§4.2). A preset whose examples vanish is worse
than one with no examples: the dataset counter lies.
3. **The engine's net shape always equals `MF_MODES[modeId].ml`** after any apply — the P5.3
contract that `tests/e2e/schema-modes.spec.ts` already enforces.
4. **`ctx.datasetCount` equals the engine's real example count** after apply. Same bar as §6.5e:
nothing may render a plausible number it did not get from the core.
5. **No C++, no schema, no codegen, no new WASM export in this item.** If a step appears to need one,
stop and re-spec — it means the scope was wrong. (Corollary: `scripts/parity-check.sh` is
*irrelevant* to this work; see §10.)
6. **Preset application is main-thread config only** — no worklet message, no audio-thread work
beyond the existing `setBackend`. `addExample` is a bounded heap copy per example, capped at 128
by validation (§3.3).
7. **`level` is persisted and never auto-reset** (§6.4); `presentation` is transient.
8. **`window.__mf` survives** with the same four methods (`ConsoleApp.tsx:331-345`), repointed at
`selectInstrument`. Playwright drives mode switches through it and must keep working *even after
a real picker exists* — the seam is faster and less brittle than clicking a popover.
9. **No "C15" in the bundle** (`tests/e2e/smoke.spec.ts:59`). The Library renders `MFMode.label`;
if the `c15` entry survives §8.5, its label is already "Powerful Synth Engine".
---
## 8. Deletions, with their consumers
| Delete | Consumers today | Replacement |
|---|---|---|
| **8.1** `backends/presets.ts` per-backend keying: `OutputPreset.backend`, `KEY_PREFIX:<backend>` namespaces, `listPresets/savePreset/getPreset/deletePreset/renamePreset(backend, …)` signatures, `OutputPresetRow` | `PresetBar` (`OutputsBackendConfig.tsx:99-208`, rendered `:686`) — the only one | `presets/store.ts` + `Preset`/`OutputOverride`. `PresetBar` becomes a view onto the unified store filtered by the active target, keeping its save/restore/rename/delete affordances. **Note the sequencing:** step 1 of §9 fixes the `cv` bug inside the old file; step 5 deletes the file. That is deliberate — the bug fix is independently valuable if 5c is never adopted. |
| **8.2** `DrawerDepth` as a disclosure flag | `Drawers.tsx` (21 sites, §6.1), `Dock.tsx`, `ConsoleApp.tsx:110/268/642` | `presentation` + `level` (§6.2) |
| **8.3** `OutputMode` / `outputMode` / `OUTPUT_MODES` / `output-mode.ts` names | `types.ts:26,110-111`, `output-mode.ts`, `Dock.tsx`, `Drawers.tsx:553,615`, `ConsoleApp.tsx:136-137,280,755-756,882,904` | `OutputTarget` / `target` / `OUTPUT_TARGETS` / `output-target.ts`. Mechanical rename; typecheck is the proof. Optional — §12.2. |
| **8.4** `ModeOverlay.input` + `MFMode.input` + `ModeInput` (`model.ts:41,141,211-216,229-247`) | `resolveInputMap(settings.inputMap, mode.input)` (`ConsoleApp.tsx:814`) — the only one | Derive from `schema.ui.primary_input` in `modeFromSchema` (`model.ts:246`); fixes the Breakor drift (§1.6.2). Manifold-only modes keep a literal. |
| **8.5** the `c15` placeholder mode (`model.ts:301-313`) | `MF_MODES` only; nothing renders it (§1.1) | Delete. It is a 2-fake-param entry on the default net shape that will become *visible* the moment a picker exists — the last survivor of the decorative stratum Phase 1 swept. Keep only if the operator wants roadmap signalling in the Library (§12.4). |
Not deleted, and why: `MFMode.glyph`/`cls`/`badge`/`placeholder` (§1.1) get their first real
consumer here. `visualizer` stays — it works, and its removal is a product call unrelated to this
item.
---
## 9. Sequenced implementation plan
Each step is independently landable and independently verifiable. Steps 1–2 are worth doing even if
the operator rejects everything after them.
**Step 1 — the `cv` preset bug** (§1.3). Three additions to `manifold/src/backends/presets.ts`:
`cv?: MFParam['cv']` in `OutputPresetRow` (`:16-29`), `cv: p.cv` in `rowsFromParams` (`:72-86`),
`cv: r.cv ?? p.cv` in `applyPreset` (`:141-162`).
*Verify:* a new `manifold/tests/presets.test.ts` case — build params with `cv` specs, round-trip
through `rowsFromParams`/`applyPreset`, assert channel + threshold survive. Runs in `bun run test`.
Red before, green after.
**Step 2 — schema-truth input kind** (§8.4, §1.6.2). Derive `MFMode.input` from
`schema.ui.primary_input` in `modeFromSchema`; drop `input` from `ModeOverlay`.
*Verify:* `bun run typecheck`; a unit test asserting `MF_MODES.find(m => m.id === 'breakor')!.input`
now follows the schema. Note this **changes visible behaviour** — Breakor's input surface flips from
circular to rectangular. That is the bug being fixed, and it belongs in the commit message.
**Step 3 — collapse mode switching into `selectInstrument`** (§4.2). Replace the three effects at
`ConsoleApp.tsx:241-243`, `:252-257`, `:260-271` with one function plus one boot effect keyed
`[engine]`. Repoint `window.__mf.setMode`.
*Verify:* this is the step with real regression risk and it already has coverage —
`tests/e2e/schema-modes.spec.ts` (four modes: dims, weight count, output arity, UI param count,
training after switch), `tests/e2e/reshape.spec.ts`, `tests/e2e/spine.spec.ts` (probe survives a
mode switch). Run the full Playwright suite, not just the smoke. On the VPS use the non-snap node
runner (`ONBOARDING.md` §2). **No new test proves the boot path** beyond the suite's own boot — if
this step misbehaves it will most likely be a mode switch that no longer reshapes, which
`schema-modes` catches, or a double-apply on boot, which nothing catches; add an explicit
"reshape called exactly once on boot" assertion via a probe counter if that risk is judged real.
**Step 4 — preset model + store + built-in catalogue** (§3). `presets/types.ts`, `presets/store.ts`,
`presets/builtin/`. Author **two or three** curated presets only — enough to exercise the model
(one config-only, one with `rest: 'off'` plus a handful of overrides, one with seeds). Authoring the
full curated set is a separate, per-mode pass — and note that pass is **not** governed by §7.5: the
20 anonymous "Param NN" slots (`ALIGNMENT.md` deferred debt; `schemas/modes/paf_synth.json` params
`p00`…) can only be named by editing the schemas, which means codegen and both generated outputs in
that change. Keep it out of this item.
*Verify:* `manifold/tests/presets.test.ts` — `validatePreset` over every `BUILTIN_PRESETS` entry
against `ALL_MODE_SCHEMAS`; store round-trip through a localStorage fake. `bun run test`.
**Step 5 — `applyPreset` + the weights invariant** (§4, §7.1). `presets/apply.ts` + `ConsoleApp`
wiring; retire the per-backend preset store (§8.1) and repoint `PresetBar`.
*Verify:* two parts, and neither alone is sufficient — say so in the commit rather than claiming
§7.1 is fully proven.
(a) **Unit, fake applier**: assert the call ORDER (seeds strictly after `selectInstrument`; routing
after the mode rebuild) and assert that no weight-mutating call — `setWeights`, `randomise`,
`train`, `trainAsync` — is made at all. This is the direct machine check on "configuration only".
(b) **C-ABI, real WASM**: in the style of `manifold/tests/loss-history.test.ts` (which drives the
committed `public/nisps.{js,wasm}` under `bun test` via `cwrap` — note it does NOT go through
`WasmIML`/`EngineApi`, which need a browser-ish environment), assert that
`nisps_ml_reshape` followed by N × `nisps_ml_add_example` leaves `nisps_ml_get_weights` bit-identical
to `nisps_ml_reshape` alone. That is the core-side half of the invariant.
**Residual gap:** nothing asserts the invariant end-to-end through the real `EngineApi` under
`bun test`. Step 6's e2e covers it indirectly (dims + dataset count after an apply). Closing it
properly would mean making `WasmIML` constructible headlessly, which is a bigger job than this
item.
**Step 6 — the Library** (§5). `console/InstrumentPicker.tsx` + the rail changes in `Dock.tsx`.
*Depends on* §6.5b (`plans/browser-mode-coverage-spec.md`) if its `modeSupport()` is to be rendered
(§5.4); buildable without it, at the cost of listing modes that cannot run.
*Verify:* a new `tests/e2e/library.spec.ts` — open the Library, apply a preset, assert (a) the
engine reshaped to that preset's mode dims via `window.__nisps.describe()`, (b) the rendered param
count matches the schema via `window.__mf.paramCount()`, (c) `datasetCount` equals the preset's seed
count, (d) no console errors. Plus `bun run test:e2e` for the smoke's no-"C15" assertion.
**Step 7 — presentation/level split** (§6). Rename `DrawerDepth`, add `levels` to `settings-store`,
re-gate all 21 sites per §6.3.
*Verify:* `tests/e2e/training-health.spec.ts` currently asserts the panel appears at `expanded`
depth — it must be updated to `advanced` level in the same change, and it is the direct proof the
mechanism swap preserved the §6.5e behaviour. Add one case: level persists across reload (the e2e
helper clears localStorage per test, so this needs an explicit in-test reload rather than a fresh
`loadProbe`). `bun run typecheck` catches the rename fallout.
**Step 8 — docs.** `manifold/ONBOARDING.md` (§4 drawer/dock description, the "`depth === 'expanded'`
is the advanced-surface flag" rule, the five-vs-six output modes error of §1.6.1), `MAP.md`
(`manifold/src/presets/`), `ALIGNMENT.md` defect 2, `dock-spec.md` (a grounding note recording the
depth model this replaces), `manifold-parity-features-spec.md` §1 (superseded pointer). Same commit
as the code that makes each true.
**Step 9 (optional) — `OutputTarget` rename** (§8.3). Pure mechanical; `bun run typecheck` is the
whole verification. Do it first or last, never in the middle of a behavioural step.
---
## 10. What the gates prove, and what they do not
- `bun run typecheck` / `bun run test` / `bun run test:e2e` are the real gates here. Baseline on
2026-07-21 before any of this work: typecheck clean, `bun test src tests/*.test.ts` → **23 pass,
0 fail, 6 files**.
- **`scripts/parity-check.sh` proves nothing about this item, in either direction.** It exercises
PAFSynth and ChannelStrip from an all-params-0.5 baseline; this item touches no C++ and no
schema, so a PASS is not evidence of correctness and could not be evidence of a regression. Do
not cite it. Do not skip it either — CI runs it regardless, and a red one means something else
broke.
- `bash scripts/build-cpp-tests.sh` and `scripts/lint-cpp.sh` likewise have no bearing; running them
is cheap insurance against having touched C++ by accident, which §7.5 forbids.
- **The weakest verification in this plan is step 3** (mode-switch refactor). It is covered by e2e,
which is slow, environment-sensitive on the VPS, and asserts outcomes rather than call counts. If
the operator wants stronger evidence, the cheap addition is a probe-visible reshape counter.
- **Nothing here needs hardware.** The one place hardware would matter — a preset reaching the
MEMLNaut — is explicitly out of scope (§0, §11.Q4).
---
## 11. Open questions for the operator
**Q1. Do you have saved output presets in the live app that must survive?**
§8.1 retires the `manifold-output-presets:<backend>` localStorage namespaces. If you have presets
saved at `meml.lnfinitemonkeys.org/next/` you care about, the answer is a one-off import shim
(~20 lines, deleted after one run); if not, they are simply orphaned and the code is cleaner. I
cannot check your browser storage.
**Q2. May a curated preset opt into one training pass over its seeds?**
§2.1 says the network ships untrained, so the default is no. But a preset with seeds and no training
sounds like an untrained net until the user presses Train — which may be exactly the intent
(the seeds are a *starting lesson*, not a *sound*), or may be a footgun. A `seed.autotrain?: boolean`
is three lines. Default proposed: absent, i.e. never.
**Q3. Should voice space be part of a preset?**
Every schema declares `voice_spaces` and `ui.show_voice_space_selector`, `nisps/modes/base.hpp:279`
implements selection, and **the browser cannot reach it** — there is no WASM export, and the audio
engine lives in the worklet behind raw exports (`worklet/nisps-processor.ts:33-36`), so this costs
the full 5-layer export chain *plus* worklet + engine-host message plumbing. It is a real
per-instrument character control that curation would obviously want. Out of scope as specified; say
if it should be pulled in, because it changes this item from "no C++" to "C++ and a new export".
**Q4. Where does curation ultimately live — browser only, or shared with firmware?**
§3.2 chooses browser-local TS data with a stated promotion condition (the hardware editor, §6.5d).
If you expect to author a preset in Manifold and send it to the MEMLNaut, say so now: the format
should then be JSON under `schemas/` from the start, because retrofitting codegen later means
rewriting the catalogue.
**Q5. URL sharing?** `manifold-parity-features-spec.md` §1.3 specified `?session=<base64url>` and a
compact axis form. Deferred here (nothing depends on it, and the axis form's control surface does
not exist). Worth scheduling, or drop it from the corpus?
---
## 12. Decisions I made (not questions)
Recorded so they can be reversed knowingly rather than rediscovered.
1. **The picker *is* the preset browser** (§2.3), one surface not two. Rationale: vision 3 wants the
curated path to be the default path; a separate mode list beside a preset list would make the raw
instrument the front door again.
2. **`OutputMode` → `OutputTarget`** (§8.3) is proposed but optional and sequenced last. It is pure
naming hygiene for a collision that only becomes painful once two "Mode" controls sit adjacent on
the rail.
3. **Explanatory prose shows at `basic` and hides at `advanced`** (§6.3), inverting today's
accidental behaviour where beginners' help is behind a modal. If you dislike it, the fallback is
one line: show prose unconditionally.
4. **Delete the `c15` placeholder mode** (§8.5), because a picker makes it visible and it does
nothing. Trivially reversible by keeping the entry and rendering its `badge: 'soon'` disabled.
5. **Two disclosure levels, not three** (§6.2) — `dock-spec.md` §0's `peek` never shipped and there
is exactly one tier of advanced content in the tree.
6. **Index-keyed output overrides** (§3.1), because `MFParam.name` is user-editable while schema
param order is codegen-stable.
7. **Two or three built-in presets in step 4, not a full catalogue** (§9). Authoring curated presets
per instrument is real design work that wants ears and hardware; the model must be proven first.

465
docs/specs/plans/hardware-editor-spec.md Normal file
View file

@ -0,0 +1,465 @@
---
kind: plan
status: active
---
# Manifold as Hardware Editor — MEMLNaut USB-Serial Protocol, Firmware Command Surface, On-Device Persistence
*Dated 2026-07-21. Spec for plan item **§6.5d** (`simplification-plan.md:130`), audit findings **A4** / **S14** /
**L29**, ALIGNMENT **defect 3** (and the live half of **defect 5**). Vision bullet 5: "Manifold doubles as
interface/editor for the hardware MEMLNaut (settings, presets, training, examples, visualisation)."*
**Nothing here is implemented. No code was written for this document.** Every `file:line` below was read from
the working tree on 2026-07-21 and is marked ✓ (verified by reading the file) or ✗ (could not verify — stated
as unknown, never asserted). The tree was mid-flight with concurrent edits from other sessions
(`git status` showed 20+ modified files including `src/main.cpp`, `codegen/generate.ts`, `ConsoleApp.tsx`);
re-check citations in those files before relying on an exact line number.
---
## 1. Ground truth — what exists today
### 1.1 The browser end is a connect-lifecycle shell
| Fact | Evidence |
|---|---|
| `MemlnautSerial` handles `requestPort`/`open`/`close` and a 5-state status store, nothing else | ✓ `manifold/src/serial/memlnaut-serial.ts` (140 lines, not the 237 the audit reports — stale) |
| `saveModel` returns `false`, `restoreModel` returns `null`, `getSettings` returns `{}`, each with a `TODO: real protocol` | ✓ same file, the three methods under `// ---- Protocol stubs ----` |
| Baud 115200 is explicitly a placeholder | ✓ same file, `TODO(memlnaut-serial): negotiate the real baud rate` |
| The only consumer is `EditorPanel.tsx`; the only consumer of *that* is the Settings-drawer `editor` case | ✓ `manifold/src/console/Drawers.tsx:33` (import), `:579` (render). Repo-wide grep for `memlnaut-serial`/`getMemlnautSerial` finds no other importer. |
| `editor` is a top-level dock **output mode**, `audio: false`, description already promises "configure / save / restore" | ✓ `manifold/src/console/output-mode.ts:69-75` |
| The panel already carries an honest "not yet wired" note | ✓ `EditorPanel.tsx`, final `<p>` |
`manifold/src/serial/web-serial.d.ts` is an **ambient** (global) declaration file. Its header comment claims it
declares "just the surface memlnaut-serial.ts uses" — false: `manifold/src/backends/cv-backend.ts:56,95,123,133`
type-checks against the same global `SerialPort`/`navigator.serial`. There is one Web Serial *type* surface and
two *transport* implementations, not three (✓ grep for `navigator.serial|SerialPort` over `manifold/src` returns
exactly `web-serial.d.ts`, `cv-backend.ts`, `memlnaut-serial.ts`).
### 1.2 The firmware end has a USB CDC port, and nothing listening on it
| Fact | Evidence |
|---|---|
| `Serial` (USB CDC) is opened at boot and used **only** for diagnostics | ✓ `firmware/MEMLNaut-NISPS/src/main.cpp:125` (`Serial.begin(115200)`) |
| Core 0's `loop()` prints `"."` ~10×/s and a perf line ~1×/s onto that same stream | ✓ `src/main.cpp:166`, `:172` (`Serial.printf("ml: %d, aud: %d, q: %f\n", …)`) |
| **The USB CDC port is otherwise free**: MIDI runs on `Serial2` (UART1), not USB | ✓ `lib/memllib/src/interface/MIDIInOut.cpp:15` `MIDI_CREATE_CUSTOM_INSTANCE(HardwareSerial, Serial2, MIDI, …)`, `:81-88` |
| USB-MIDI (TinyUSB composite) is compiled out — `MIDI_USB_CLIENT` is defined nowhere | ✓ `MIDIInOut.cpp:55` is `#ifdef MIDI_USB_CLIENT`; grep across `platformio.ini`, `src/`, `glue/` finds no definition. The device enumerates as a plain CDC port. |
| There is no serial *reader* anywhere in firmware — no `Serial.read()`, no parser, no command dispatch | ✓ grep over `src/` + `glue/` |
| Core 0's control cadence is 5 ms (`ML_INFERENCE_PERIOD_US 5000`), core 1's drain 1 ms | ✓ `src/main.cpp:152-160`, `:208-216` |
### 1.3 There is no on-device persistence, but two mechanisms are already compiled in
- **LittleFS** (internal flash). memllib wraps it in `lib/memllib/src/hardware/FlashFS.hpp` ✓ — which has **zero
consumers** in this repo (✓ grep for `FlashFS` hits only vendored TFT_eSPI example sketches under `.pio/`).
Two problems with that wrapper: it defines `void begin()` and `bool exists()` as **non-`inline` free functions
in a header** (✓ read in full — 22 lines), so including it from two translation units is a duplicate-symbol
link error; and using it at all buys nothing over calling `LittleFS` directly.
- **The filesystem is currently sized to zero.** `platformio.ini` does not set `board_build.filesystem_size`
(✓ grep), and the platform's default is `"0MB"` (✓ `~/.platformio/platforms/raspberrypi/builder/main.py:63`).
With `_size == 0`, `LittleFS.begin()` returns `false` immediately (✓
`~/.platformio/packages/framework-arduinopico/libraries/LittleFS/src/LittleFS.h:173`). **On-device flash
persistence therefore requires a `platformio.ini` change that alters the flash layout of all 16 envs.**
- **LittleFS writes park core 1.** ✓ `…/LittleFS/src/LittleFS.cpp:186-193` and `:203-209`:
`noInterrupts(); rp2040.idleOtherCore(); flash_range_program/flash_range_erase(...); rp2040.resumeOtherCore();
interrupts();`. Core 1 is the audio core. **Any flash write stalls audio for the duration of the erase +
program.** This is the single hardest constraint in this document.
- **SD card.** memllib ships `interface/SDCard.{hpp,cpp}` (SdFat over hardware SPI) ✓, the MEMLNaut board has
dedicated SD pins ✓ `lib/memllib/src/hardware/memlnaut/Pins.hpp:87-90` (`SD_CS 13`, `SD_SCK 14`, `SD_MISO 12`,
`SD_MOSI 15`), and `SDCard.cpp` **already compiles into every variant** ✓ (`.pio/build/pafsynth/…/interface/
SDCard.cpp.o` exists). SPI writes do **not** lock out core 1. It needs a physical card fitted; whether the
operator's units have one is ✗ unknown.
### 1.4 The model *is* transferable — the shapes already agree
- Firmware instantiates a compile-time `MLP<…>` whose dims come from the mode's generated schema; the browser
reshapes its `MLPCore<DynamicStorage>` to **the same schema dims** on every mode switch ✓
`manifold/src/console/ConsoleApp.tsx:252-257` (`engine.reshape({ inputSize, outputSize, hidden }, defaultSpread)`
from `mode.ml`). So for a given `mode_id` the two nets have the *same* `weight_count` by construction.
- **But not unconditionally.** The browser also offers an axis-count reshape that changes `inputSize` alone ✓
`ConsoleApp.tsx:290-322`, and `nisps_ml_create`'s default shape is a 32-input head ✓
`manifold/src/inputs/input-layer.ts:17,38`. A live browser net can therefore be at `6→[10,10,14]→33` while the
device is at `4→[10,10,14]→33`. **The protocol must carry the shape and refuse mismatches; it must never infer
compatibility from `mode_id` alone.**
- Flat weight layout is already specified and identical on both targets: `[l0_w][l1_w][l2_w][l3_w][l0_b][l1_b]
[l2_b][l3_b]`, little-endian `float32` ✓ `nisps/wasm/bindings.cpp:25-34` (comment), `nisps/ml/mlp.hpp:385-414`
(`get_weights`/`set_weights`).
- Per-mode blob sizes (computed from `schemas/modes/*.json` ✓):
| mode | dims | weights+biases | bytes |
|---|---|---|---|
| `sound_analysis_midi` | 10→[10,10,14]→8 | 494 | 1 976 |
| `channel_strip`, `xiasri` | 4→[10,10,14]→24 | 674 | 2 696 |
| `paf_synth` | 4→[10,10,14]→33 | 809 | 3 236 |
| `elysiamorf` | 4→[10,14,18]→40 | 1 234 | 4 936 |
| `verb_fx` | 4→[10,14,18]→47 | 1 367 | 5 468 |
| `breakor`, `memlcelium`, `slp_workshop` | 4→[10,14,18]→56 | 1 538 | **6 152** |
Worst case is ~6 KB. Small — but not small enough to sit in one frame without either a 6 KB RX buffer or
chunking.
### 1.5 Training examples are **not** readable out of the core today
`MLPCore` exposes `example_count()` ✓ `nisps/ml/mlp.hpp:441` and `clear_examples()` ✓ `:443`, but the per-example
accessors `sample_features_(s)` / `sample_labels_(s)` are **private** ✓ `:577`, `:581`. Exporting the training
set from the device is therefore a real (small) addition to `nisps/ml/mlp.hpp`, not a wiring job.
Worse for the editor: the firmware's `FeedbackController` — which owns explore/place state and drives
`add_example` — is a **function-local static inside `bind_peripherals()`** ✓
`firmware/MEMLNaut-NISPS/glue/peripherals.hpp:152`. It is unreachable from `main.cpp` or any command handler.
Any command that touches feedback state requires hoisting it first.
### 1.6 The discipline to copy already exists in-repo
`firmware/useq-celium/shared/protocol.h` ✓ is a C header of `static const` constants + `static inline` helpers,
mirrored constant-for-constant by `manifold/src/backends/useq-protocol.ts` ✓, pinned by
`manifold/src/backends/useq-protocol.test.ts` ✓ (`test('frame sizes match the C header')`), documented as
`kind: spec` in `docs/specs/useq-cv-protocol.md` ✓. The device-side parser is a ~20-line resync state machine ✓
`firmware/useq-celium/main/src/main.cpp:93-110`. **Verified live**: `bun test src tests/*.test.ts` in `manifold/`
→ `23 pass, 0 fail, 6 files, 65 ms`.
### 1.7 L29 — `InputChain`/`OutputChain` have no firmware consumer
✓ Repo-wide grep: the only non-test, non-self references are `nisps/wasm/bindings.cpp:76-77,194-195,968,1011`
and `tests/cpp/{test_pipeline,parity_check}.cpp`. `nisps/pipeline/output_chain.hpp:13` says "firmware would pick
its mode's NOut" — aspirational, present-tense-adjacent, and currently false. `InputChain` already serialises
(`state_size()`, round-trip test ✓ `tests/cpp/test_pipeline.cpp:131-150`), which matters below.
---
## 2. Invariants
Beyond the repo-wide hard constraints (platform-neutral allocation-free `nisps/`; native↔WASM parity ≤1e-5;
schema changes ship both codegen outputs; RT-safe worklet + SPSC dual-core discipline), this area adds:
- **I1 — The wire format has exactly one source of truth.** A single C header under `firmware/`, mirrored in TS,
pinned by a test that fails when they diverge. No constant is written twice by hand in the same language.
- **I2 — No hand-maintained per-mode table anywhere in the protocol.** Mode identity, net shape, param count and
ordering reach the wire from codegen output on both sides, or they do not reach it at all.
- **I3 — A model blob is applied only against a proven-identical contract.** Shape *and* schema fingerprint must
match. On mismatch the device NAKs; it never partially applies, never truncates, never zero-pads.
- **I4 — Serial servicing is bounded.** The core-0 pump reads at most a fixed byte budget per `loop()` iteration.
The 5 ms `tick_control()` cadence ✓ (`src/main.cpp:152,160`) is not to be perturbed by a host that streams.
- **I5 — Serial handling never runs on core 1.** Core 1 is audio + MIDI drain. All parsing, buffering, filesystem
and ML mutation happen on core 0.
- **I6 — Flash writes are an explicit, announced, user-initiated act.** Because they park core 1 (§1.3), a
persist is never automatic, never periodic, and never a side effect of another command.
- **I7 — Diagnostics and protocol do not share the stream unframed.** Either the diagnostics are silenced for the
session or they are carried inside a frame. Never raw ASCII interleaved with binary payloads.
- **I8 — The device is authoritative about itself.** The browser asks and believes; it never assumes a firmware
build from a mode name, a port name, or a USB descriptor.
- **I9 — No compat shim.** The stubs are deleted, not deprecated. `EditorPanel` is the only consumer (§1.1).
---
## 3. Design
### 3.1 Transport and channel
USB CDC (`Serial`), 8N1; the CDC baud parameter is ignored by the hardware but the host must supply one —
**115200**, matching what the firmware already calls and what `cv-backend.ts:133` uses. No second UART, no
composite USB device, no TinyUSB reconfiguration.
**The diagnostics collision (I7) is resolved by session state.** On `HELLO` the firmware sets an
`editor_session` flag; while set, `loop()`'s `Serial.println(".")` and perf `printf` (✓ `src/main.cpp:166,172`)
are suppressed and the same information is emitted as a `TELEMETRY` frame instead. `BYE`, a host disconnect
(DTR drop), or a 5 s silence timeout clears the flag and restores the prints. Boot-time prints
(`"Serial initialised."`, `"Bound peripherals to mode."`) happen before any session and are harmless — the
host-side parser drops non-sync bytes anyway.
*Rejected:* moving diagnostics to a second UART (costs pins and a second cable) and CDC-composite dual ports
(complicates the Web Serial port picker for one debug convenience).
### 3.2 Framing
Variable-length, because payloads range from 0 bytes to a 6 KB model (§1.4). Fixed-length-per-type (the useq
choice) does not survive that.
```
byte 0 sync 0xA5 host→device, 0x5A device→host
byte 1 type u8
bytes 2..3 length u16 LE, payload bytes, 0..MEMLED_MAX_PAYLOAD
bytes 4.. payload
last 2 crc u16 LE, CRC-16/CCITT-FALSE over bytes [1 .. 3+length]
```
- **CRC-16, not XOR-8.** useq's XOR-8 is right for a 26-byte frame streamed at 100 Hz where a corrupt frame is
discarded and replaced 10 ms later. It is wrong for a 6 KB model transfer, where a 1-in-256 undetected
corruption silently installs wrong weights and the user hears an inexplicable instrument. Bitwise CRC-16 is
~15 lines, table-free, and identical on both sides.
- **`MEMLED_MAX_PAYLOAD = 512`.** Bounds the device RX buffer at 518 bytes and each frame's service time at
~45 ms of wire time worst case — hence I4's byte budget. Larger objects chunk (§3.4).
- Both directions use the same frame shape; only the sync byte differs, so one parser implementation serves
both ends.
Home: **`firmware/shared/memlnaut-editor-protocol.h`** — deliberately *not* under `firmware/MEMLNaut-NISPS/`,
because it is shared with the browser exactly as `firmware/useq-celium/shared/protocol.h` is. Mirror:
**`manifold/src/serial/memlnaut-protocol.ts`**. Parity test: **`manifold/src/serial/memlnaut-protocol.test.ts`**
(picked up by the existing `bun test src tests/*.test.ts` glob — do not touch that script).
### 3.3 Identity: a codegen-emitted schema fingerprint (discharges I2, I3)
The one genuinely new idea in this spec, and the thing that makes "derive from schema codegen, not hand-defined
tables" concrete.
`codegen/generate.ts` gains one emitted constant per mode: a **FNV-1a 64 hash over a canonical serialisation of
the mode's contract-bearing schema fields** — `mode_id`, `engine_id`, `ml` (`input_size`, `hidden_layers`,
`output_size`, `default_spread`), `params[]` as `(name, min, max, default, curve, group)` in order, and
`voice_spaces` (names + curve overrides) in order. Deliberately **excluded**: `label`, `_note`, `ui` — cosmetic,
and a label edit must not invalidate a user's trained model.
Emitted into both generated worlds in the same change (hard constraint):
- `nisps/modes/generated/<mode>_schema.hpp` → `inline constexpr std::uint64_t k<Mode>SchemaHash = 0x…ull;`
alongside the existing `k<Mode>ModeId` ✓ (`nisps/modes/generated/paf_synth_schema.hpp:10`)
- `manifold/src/modes/generated/<mode>_schema.ts` → `schema_hash: '0x…'` on the existing `ModeSchema` const
CI already fails on a stale generated tree (`.github/workflows/ci.yml`, step *"Codegen is committed and
idempotent"* ✓ `:129`), and `codegen/tests/golden/paf_synth_schema.{hpp,ts}` ✓ will need regenerating in the
same commit.
The device reports its hash in `DESCRIBE`; the browser compares against **its own** generated hash for that
mode. Equal ⇒ weights, settings and examples are interchangeable. Unequal ⇒ the UI says *"this MEMLNaut is
running a different build of `paf_synth`"* and disables transfer. No table, no version negotiation matrix, no
per-mode special case — and the check is exact rather than heuristic.
### 3.4 Command set (protocol v1)
Types are `static const uint8_t` in the shared header. `ACK`/`NAK` are the universal replies; `NAK` carries a
one-byte reason code from an enum in the same header.
**Host → device**
| Type | Payload | Reply |
|---|---|---|
| `HELLO` | `u16 host_proto_ver` | `DESCRIBE` — opens the session, silences diagnostics |
| `BYE` | — | `ACK` — closes the session, restores diagnostics |
| `GET_MODEL` | — | `MODEL_BEGIN`, then `MODEL_CHUNK`×N |
| `PUT_MODEL_BEGIN` | `u64 schema_hash, u16 dims[5], u32 weight_count, u32 crc32` | `ACK`/`NAK` |
| `PUT_MODEL_CHUNK` | `u16 index, bytes` | `ACK` |
| `PUT_MODEL_END` | — | `ACK` (applied) / `NAK` (CRC or count mismatch — **nothing applied**) |
| `GET_SETTINGS` | — | `SETTINGS` |
| `SET_SETTING` | `u16 id, f32 value` | `ACK`/`NAK` |
| `PERSIST` | `u8 slot` | `ACK`/`NAK` — see I6 |
| `LOAD_PERSISTED` | `u8 slot` | `ACK`/`NAK` |
| `CLEAR_EXAMPLES` | — | `ACK` |
| `SUBSCRIBE` | `u16 mask, u8 rate_hz` | `ACK` — telemetry stream on/off |
**Device → host**
`DESCRIBE`, `MODEL_BEGIN`, `MODEL_CHUNK`, `SETTINGS`, `TELEMETRY`, `LOG` (UTF-8 text, framed — the session
replacement for the raw prints), `ACK`, `NAK`.
`DESCRIBE` payload: `u16 proto_ver`, `u64 schema_hash`, `mode_id` (fixed 24-byte NUL-padded ASCII, from
`generated::k<Mode>ModeId` ✓), `u16 dims[5]`, `u32 weight_count`, `u16 max_examples`, `u16 capability_bits`,
`char fw_build[16]`. The capability bits are what let one protocol serve 16 firmware variants without a
per-variant table: `HAS_PERSIST`, `HAS_SD`, `HAS_EXAMPLE_EXPORT`, `HAS_TELEMETRY`, `IS_SEQUENCER`.
**PUT_MODEL is atomic (I3).** Chunks land in a staging buffer sized `weight_count * 4` — for the largest mode
6 152 bytes of core-0 RAM, allocated as a `static` array sized from `MLPType::weight_count()` at compile time,
so no heap. `set_weights` is called once, from `PUT_MODEL_END`, only after the CRC-32 over the whole blob
matches and `weight_count` and `dims[5]` equal the device's own.
**Settings (I2, honestly).** Not everything can come from codegen, and pretending otherwise would be the
hand-defined table under a different name. The split:
- *Derived from codegen, on both sides:* the training triple `learning_rate` / `max_iterations` / `min_error`
(defaults + identity from `schemas/ml_defaults.json` → `nisps::ml::generated::kMlTrainDefaults` ✓ and
`manifold/src/modes/generated/ml_defaults.ts` ✓ — both mirrors already exist), and `voice_space_index` whose
legal range is the generated `voice_spaces` list ✓.
- *Genuinely device-side, enumerated in the shared header:* `pin_value`, `joystick_single` (the Dual/Single
toggle ✓ `glue/settings_view.hpp:54-58`), `explore_intensity` (RVX1 ✓ `glue/peripherals.hpp:124-126`),
`master_volume` (RVGain1 ✓ `:115-117` — note it drives `AudioDriver::SetMasterVolume` directly, not the mode). These are `ModeBase`/peripheral concepts with no schema home; there
are four of them; they live in one enum in the shared header and nowhere else.
**Telemetry closes ALIGNMENT defect 5's live half.** The `TELEMETRY` payload carries `PERF_GET_MEAN(MLSTATS)`
and `AUDIOLOOP_MEAN` — numbers the firmware **already computes and prints** ✓ `src/main.cpp:172` — plus
`example_count()`, `eval_loss()`, and (paged) the loss history the operator deliberately kept on device
(audit L25). That is the on-device timing report ALIGNMENT defect 5 asks for, at near-zero marginal cost,
displayed in a browser panel that already knows how to draw a loss curve (`console/TrainingHealth.tsx` ✓).
### 3.5 Persistence
**Primary: LittleFS on internal flash**, because it needs no accessory and every unit has it.
Required changes: `board_build.filesystem_size` in `platformio.ini`'s `[env]` block (§1.3 — currently absent,
default `0MB`, `begin()` fails), a `glue/persistence.hpp` that calls `LittleFS` **directly** (not memllib's
`FlashFS.hpp`, whose non-`inline` header functions are a duplicate-symbol trap and which adds nothing — §1.3),
and a fixed on-disk record:
```
/memlnaut/<mode_id>/<slot>.mdl magic, u16 record_ver, u64 schema_hash, u16 dims[5],
u32 weight_count, f32 weights[], settings block, u32 crc32
```
`schema_hash` on disk is checked on load exactly as on the wire (I3), so a firmware reflash that changes the
schema invalidates old saves loudly instead of loading garbage.
**The audio stall is real and must be surfaced, not hidden.** Sequence for `PERSIST`: mute via
`AudioDriver::SetMasterVolume(0)`, emit `LOG "saving…"`, write, restore volume, `ACK`. The browser shows a
determinate "Saving to device — audio will pause" state. The alternative — pretending a flash erase is free —
would be exactly the kind of plausible-looking lie this repo has been deleting all week.
**Not chosen, but noted:** the SD path (`interface/SDCard.hpp` ✓, already compiled ✓, pins ✓) does not park
core 1 and is the right home for bulk artefacts (full example sets, session recordings) if the operator's units
have cards fitted. Left behind the `HAS_SD` capability bit rather than built speculatively.
### 3.6 Browser side
`manifold/src/serial/memlnaut-serial.ts` is **rewritten, not extended**: the three stub methods and their
`TODO`s are deleted (I9 — sole consumer is `EditorPanel.tsx`, §1.1). The class becomes a framed-transport
driver: a read loop over `port.readable`, a resync parser identical in shape to the TS mirror's encoder, a
promise-keyed request/reply map, and the same `subscribe()` store it already has (which `EditorPanel` already
consumes via `useSyncExternalStore` ✓).
`EditorPanel.tsx` grows: a device card (mode, build, dims, compatibility verdict), Save-to-device /
Load-from-device with the compatibility gate and a progress state, a settings section, and a live telemetry
strip. **The "not yet wired" note is deleted in the same commit as the code that wires it** — a note that
outlives its truth is the failure mode this repo keeps finding.
`output-mode.ts:72`'s description ("configure / save / restore") becomes true and needs no edit. Two small
hygiene items ride along: fix `web-serial.d.ts`'s false header comment (§1.1) and move it to a shared location
now that two modules provably depend on it.
---
## 4. Deletions
| Deleted | Named consumers | Why safe |
|---|---|---|
| `MemlnautSerial.saveModel` / `.restoreModel` / `.getSettings` bodies + `TODO`s | `EditorPanel.tsx` only (✓ grep) | Replaced in the same change; no external caller |
| `EditorPanel`'s "not yet wired" `<p>` | none | Becomes false the moment step 6 lands |
| The unconditional `Serial.println(".")` / perf `printf` in `src/main.cpp:166-175` | none (human eyeball on a serial monitor) | Not deleted — made session-conditional. Outside a session behaviour is unchanged. |
| `lib/memllib/src/hardware/FlashFS.hpp` — *not* deleted | none (✓ zero consumers) | Vendored upstream code; deleting it diverges the vendor tree for no gain (`VENDORED.md` re-sync). We simply do not use it, and `persistence.hpp` says why. |
Nothing else. In particular the `cvgate` backend's Web Serial code stays untouched — different device, different
protocol, no shared transport worth extracting for two implementations.
---
## 5. Sequenced implementation plan
Each step is independently landable and independently verifiable. Steps 1–4 are agent-safe with the existing
gates. Steps 5–8 need hardware and are honestly marked.
**Step 0 — L29 disposition (do first, it is one line either way).**
`nisps/pipeline/output_chain.hpp:13` and `input_chain.hpp`'s header currently imply a firmware consumer that does
not exist (§1.7). If the operator answers **Q4** (below) with "not now", soften both comments to say the chains
are browser-side today, in this step. If "yes", they stay and step 8 fulfils them. Either way L29 stops being an
open lie after step 0.
*Verification:* `bash scripts/lint-cpp.sh` (comment-only change; the lint strips comments before matching ✓).
**Step 1 — the shared header + TS mirror + parity test. No behaviour anywhere.**
`firmware/shared/memlnaut-editor-protocol.h`, `manifold/src/serial/memlnaut-protocol.ts`,
`manifold/src/serial/memlnaut-protocol.test.ts`. Encoder/decoder + CRC-16 in both languages; the test asserts
frame offsets, `MEMLED_MAX_PAYLOAD`, every type byte, and CRC-16 against fixed vectors.
*Verification:* `cd manifold && bun run typecheck && bun run test` (the new test is picked up by the existing
glob — **do not change the test script**). Additionally register a host C++ test that includes the header and
asserts the same vectors: add a source to `nisps_modes_tests`, which already puts the repo root on its include
path for exactly this reason ✓ (`nisps/CMakeLists.txt:134`). Then `bash scripts/build-cpp-tests.sh` proves the C
and TS encoders agree on the same vectors on both sides.
*Note:* `scripts/parity-check.sh` says nothing about any of this — it exercises PAFSynth and ChannelStrip at
all-params-0.5. A green parity run is not evidence for this step.
**Step 2 — codegen emits the schema fingerprint.**
`codegen/generate.ts` + `codegen/lib.ts`; C++ and TS in the same commit; `codegen/tests/golden/*` regenerated.
*Verification:* `cd codegen && bun run generate.ts` twice (idempotence), `bun run test` (golden + curve-drift),
then `bash scripts/build-cpp-tests.sh` (the generated headers are compiled by `nisps_modes_tests` ✓). CI's
"Codegen is committed and idempotent" step ✓ is the backstop.
**Step 3 — firmware command surface, read-only commands.**
`glue/editor_serial.hpp`: the bounded pump (I4), the parser, `HELLO`/`DESCRIBE`/`GET_SETTINGS`/`GET_MODEL`/
`SUBSCRIBE`/`BYE`, the session flag gating the diagnostics. Called from `loop()` on core 0 next to
`tick_control()`. Deliberately no mutation yet.
*Verification (weak, be honest):* `pio run -e slpworkshop -e pafsynth -e selftest` compiles and the flash/RAM
delta is reported (CI's firmware job already does this ✓ `.github/workflows/ci.yml:213,223`). Compilation is
**all** that is verified without hardware. The parser itself should be structured so its byte-level core is
Arduino-free and host-testable — the precedent is `glue/codec_config.hpp`, deliberately Arduino-free and covered
by `tests/cpp/test_mode_driver_config.cpp` ✓. Follow it: put the state machine in the shared header or a pure
sibling, and the `Serial` reads in the glue.
**Step 4 — browser transport + read-only editor UI.**
Rewrite `memlnaut-serial.ts` onto the framed protocol; `EditorPanel` renders `DESCRIBE` + telemetry + the
compatibility verdict. Save/Load still absent.
*Verification:* `bun run typecheck && bun run test && bun run build`. A Playwright e2e can cover the panel's
**disconnected** rendering only — Web Serial cannot be driven headlessly. Do not claim more. The frame codec is
covered by step 1's unit test; the *round trip* is not covered by anything until hardware.
**Step 5 — firmware mutating commands.**
`PUT_MODEL_*` (staging buffer + atomic apply), `SET_SETTING`, `CLEAR_EXAMPLES`. Requires hoisting the
`FeedbackController` out of `bind_peripherals`'s function-local static (§1.5) if any command touches feedback
state; if v1 avoids feedback entirely, say so and leave it.
*Verification:* compile + size. The atomicity logic (CRC-32 accumulate, count check, single `set_weights`) is
host-testable if written as a pure function over spans — do that, and cover it in `nisps_modes_tests`.
**Step 6 — Save-to-device / Load-from-device in the UI**, gated on the schema hash + dims match, with progress
and the explicit incompatibility message. Delete the "not yet wired" note here.
*Verification:* typecheck + unit + build; e2e for the **gate** (a fake `DESCRIBE` with a wrong hash must disable
the buttons) is worth writing because it is pure logic.
**Step 7 — persistence.**
`board_build.filesystem_size` in `platformio.ini` (**changes the flash layout of all 16 envs — rebuild and
re-report sizes**), `glue/persistence.hpp` on `LittleFS` directly, the record format, `PERSIST`/`LOAD_PERSISTED`,
the mute-around-write sequence.
*Verification:* all 16 envs build and the size table is re-baselined (`scripts/build-firmware.sh --all`). The
record encode/decode is pure and host-testable; the flash behaviour, the stall duration, and whether the audio
interruption is acceptable are **operator chokepoints — no automated gate reaches them.**
**Step 8 — (conditional on Q4) `InputChain`/`OutputChain` firmware wiring.**
`InputChain` in `bind_peripherals` between the joystick callbacks and `mode.set_input`; `OutputChain<MLPType::
kOutput>` in `ModeBase::tick_control` before `engine_.set_params`. Both configs become editor settings, which is
the only reason firmware would want them configurable at all. Note the cost honestly: `OutputChain<NMax>` adds an
`NMax`-float state array to every mode instance and a per-tick pass at 200 Hz.
*Verification:* host tests already exist for both chains ✓ (`tests/cpp/test_pipeline.cpp`) and parity stage 7
covers them ✓ (`tests/cpp/parity_check.cpp:341-394`) — but **neither proves the firmware wiring**, and the thing
that actually changes (how the joystick *feels*) can only be judged by ear on hardware.
---
## 6. Open questions — operator only
1. **Persistence medium and the audio stall.** Internal flash (always present, but every save parks core 1 and
interrupts audio for the erase+program — §1.3) or SD (no stall, but needs a card fitted, and it is ✗ unknown
whether your units have one)? If flash: is a brief, announced audio interruption on an explicit save
acceptable, or does that rule flash out?
2. **What "presets" means for the device.** Vision bullet 5 says *settings, presets, training, examples,
visualisation*. This spec covers settings, model transfer, telemetry, and persistence slots. It does **not**
define a preset object — and §6.5c (curated/advanced split) is supposed to define exactly that, and is
scheduled *before* this item. Does the editor wait for 5c's preset model, or ship slots-of-weights now and
absorb presets later?
3. **Do examples need to leave the device?** Exporting the training set requires a new public accessor on
`MLPCore` (§1.5) and probably hoisting the `FeedbackController` (§1.5). "Train on device, curate in browser"
implies yes; "save/restore a model" implies no. This is the difference between a small v1 and a much larger
one.
4. **L29 / step 8: should firmware gain the browser's input pipeline?** Deadzone, circular clamp, momentum zoom
and EMA would change the feel of every hardware mode, and only your ears can judge it. If the answer is "not
now", step 0 softens the comments and the question closes cleanly.
5. **Whose model wins on connect?** When the browser connects and both ends hold a trained net for the same
mode, does the editor do nothing until told (proposed default), or offer/auto-pull the device's?
## 7. Decisions taken here (not questions)
- CRC-16 for frames, CRC-32 for whole-model transfers, not useq's XOR-8 — §3.2.
- Variable-length frames with a 512-byte cap and chunked models, not one large frame — §3.2, I4.
- Diagnostics silenced by session flag rather than moved to another port — §3.1, I7.
- Compatibility by codegen-emitted schema fingerprint + explicit dims, never by `mode_id` alone — §3.3, I3.
- `LittleFS` called directly; memllib's `FlashFS.hpp` not used and not deleted — §3.5, §4.
- The four device-side settings live in an enum in the shared header, and the spec says plainly that they are
not codegen-derived rather than pretending — §3.4.
- Protocol header at `firmware/shared/`, not inside `firmware/MEMLNaut-NISPS/`, mirroring useq-celium.
## 8. Where verification is weak
Say this out loud in any PR description that lands these steps:
- **`scripts/parity-check.sh` proves nothing here.** It exercises PAFSynth and ChannelStrip with all params at
0.5. It touches no serial code, no framing, no persistence.
- **Nothing in CI executes firmware.** The firmware job compiles three envs and reports sizes ✓
(`.github/workflows/ci.yml:213,223`). A protocol that compiles is not a protocol that works.
- **Web Serial cannot be exercised headlessly.** Playwright can cover the panel's disconnected and
incompatible-device states, and the frame codec is unit-tested on both sides — the *round trip over a real
cable* is untested until someone plugs a MEMLNaut in.
- **Mitigation worth building early:** a tiny host-side loopback harness — the TS encoder feeding the C parser
compiled natively into `nisps_modes_tests` — turns "the two ends agree about bytes" from a hope into a gate,
and costs about an hour. It still says nothing about USB CDC, flash timing, or feel.
- **Step 7 has no automated gate at all** beyond "16 envs still build". Flash layout, stall duration and
recovery-after-power-loss are hardware chokepoints.

665
docs/specs/plans/mode-layer-reunification.md Normal file
View file

@ -0,0 +1,665 @@
---
kind: plan
status: active
---
# Mode-Layer Reunification — the shared control tick (P5 item 5a)
*Dated 2026-07-21. Spec-first, per `simplification-plan.md` §6 ("each item spec-first, own
session"). Closes `ALIGNMENT.md` defect 1; audit findings A1 + A5
(`../recon/simplification-audit-2026-07.md`). **This document authorises no code.** It fixes
the seam, the invariants, the sequence and the verification so an implementation session can
start cold. §8 lists the decisions that must come from the operator before step 3 lands.*
**Every code claim below was re-read against the working tree on 2026-07-21** (`fd0aee2` plus
the uncommitted in-flight work of parallel sessions — `nisps/modes/base.hpp`,
`nisps/wasm/bindings.cpp` and several `manifold/src/engine/` files are all dirty relative to
HEAD, so line numbers are working-tree numbers and may shift by a few lines once those land).
Where a cited document (audit, plan, ALIGNMENT) disagreed with the code, the code won and the
disagreement is called out inline.
Sibling P5 specs written in the same session: `browser-mode-coverage-spec.md` (5b),
`curated-presets-spec.md` (5c), `hardware-editor-spec.md` (5d). This one is 5a.
---
## 0. The gap, stated precisely
`nisps/modes/` compiles into exactly one binary: the firmware. Its consumers are
`firmware/MEMLNaut-NISPS/glue/{mode_select,midi_io,output_router,peripherals,settings_view}.hpp`
and the host tests in `tests/cpp/test_mode_*.cpp`. `nisps/wasm/bindings.cpp` includes eight
engine headers (lines 54–62) and six `nisps/ml/` headers (65–72) and **zero** mode headers —
verified by `grep -n modes nisps/wasm/bindings.cpp`, whose only hit is the word "modes" inside
a comment about `schemas/modes/*.json` (line 90).
The consequence is not that the browser lacks modes. It is that the **control tick** —
`ModeBase::tick_control()`, `nisps/modes/base.hpp:198–234` — exists twice: once in C++ for the
firmware, once hand-assembled in TypeScript for the browser out of `Spine.setInputs`
(`manifold/src/engine/spine.ts:254–303`) plus `ExplorationController`'s two timers
(`manifold/src/engine/exploration.ts:40–42, 57, 84–120`). The primitives underneath (MLP, Jolt,
OU, feedback, pipelines, curves) are already shared; the **orchestration** is not.
That is the whole defect. Everything in this document serves one outcome: *the control tick is
written once, in C++, and both targets call it.*
---
## 1. Verified ground truth
### 1.1 What `ModeBase` actually does today
`nisps/modes/base.hpp`, `ModeBase<Derived, EngineT, MLPType, NInputs>`:
| Responsibility | Where | Notes |
|---|---|---|
| Own the engine, the MLP, the input channels | members, `:322–341` | `input_channels_`, `input_pinned_`, `pin_value_`, `out_buf_`, `jolt_buf_` |
| `setup(sr)` — engine setup, **all channels to 0.5f**, one inference, route to engine, `on_setup` hook | `:126–142` | |
| `set_input(idx, v)` — bounds-drop + clamp to [0,1] | `:165–170` | |
| Input pinning (single/double-joystick neutralise) | `:179–196` | `set_input_pinned` / `set_pin_value` / `effective_input`; driven by `glue/settings_view.hpp` |
| `tick_control()` — the control tick | `:198–234` | order: `on_pre_inference` → jolt step (if active) → `jolt_.tick_lr_ramp()` → forward pinned channels → `ml_.process()` → OU on a copy (if enabled) → `engine_.set_params()` → `on_post_inference` |
| `apply_jolt_()` — get→glide→set of the flat weights | `:312–320` | out-of-line by design |
| Jolt / OU gesture surface | `:241–255` | `jolt_press/release/active/lr_scale`, `set_explore_intensity`, `jolt()`, `ou_noise()` |
| `driver_config()` | `:157–163` | firmware-only concept (codec / mic vs line) |
| Voice-space selection | `:279–290` | delegates to the engine, re-applies params |
| `ControlEvent` ring (`push`/`pop_control_events`) | `:294–305` | drained by `glue/midi_io.hpp:69–98` |
| `process(stereosample_t)` | `:257–259` | pure engine forward, RT |
Firmware drives it from two cores: `tick_control()` on core 0 every 5 ms
(`firmware/MEMLNaut-NISPS/src/main.cpp:154–160`, `ML_INFERENCE_PERIOD_US 5000`), `process()` on
core 1 from the audio callback, and `drain_outputs()` on core 1 at ~1 kHz
(`main.cpp:208–212`).
### 1.2 What the browser does instead
`Spine.setInputs` (`manifold/src/engine/spine.ts:254–303`) is the browser's control tick:
1. `iml.processInput(x, y, dt)` — the 2-D `InputChain` (**no firmware counterpart**, see 1.3).
2. `iml.setInput(i, v)` for every axis → `nisps_ml_set_input` → `MLPCore::set_input`.
3. `iml.processInto(mlBuf)` → `nisps_ml_process`.
4. `iml.processOutput(routedBuf, dt)` — the `OutputChain`.
5. `this.outputMorph(routedBuf)` — the OU walk, registered by
`ExplorationController`'s constructor (`exploration.ts:57`) → `nisps_ml_explore_apply`.
6. `backendSend(routedBuf)` → `EngineHost.setParams` → `postMessage` → the **worklet's own
WASM instance** → `nisps_engine_set_params`.
Jolt is driven by a separate 5 ms `setInterval` (`exploration.ts:40, 84–91`) calling
`nisps_ml_jolt_step` + `engine.process()`; the OU walk gets a second 30 ms interval
(`exploration.ts:42, 107–113`).
**Three WASM instances exist**, not one: main thread (`WasmIML` — MLP, feedback, pipelines),
AudioWorklet (`worklet/nisps-processor.ts` — engine only, four exports), and an on-demand
training worker (`wasm-worker.ts` — a mirror MLP). `manifold/ONBOARDING.md` §3 says "loaded
TWICE"; the training worker is a third. This is why a monolithic mode object cannot be bound
(see §2).
### 1.3 The divergences this hand-mirroring has produced
Each row is a fact about the current tree, verified today.
| # | Divergence | Evidence |
|---|---|---|
| D1 | **`nisps_ml_jolt_step` is a line-for-line duplicate of `ModeBase::apply_jolt_`** | `bindings.cpp:842–849` vs `base.hpp:312–320` — same get→copy→`jolt.step`→set sequence, two copies of the code |
| D2 | **Jolt/OU RNG salts differ between targets** | `base.hpp:122–123` uses `0x91E10C5E` / `0x0CEA0FF5`; `bindings.cpp:101–102` uses `0xB01DFACE` / `0x0DDBA11`. The bindings comment claims it "mirrors the firmware ModeBase seeding" — it mirrors the *practice*, not the values. Same seed ⇒ different jolt/OU streams. (`kFeedbackSalt` *is* matched across `bindings.cpp:98` and `glue/peripherals.hpp:58`.) |
| D3 | **OU is applied at a different point in the chain.** Firmware: ML → OU → engine (`base.hpp:216–229`). Browser: ML → OutputChain → OU (`spine.ts:292, 296`). | Consequence: in the browser the per-output **freeze mask holds a value that the OU walk then moves again** — `OutputChain::process` writes `prev_[i]` for frozen dims (`output_chain.hpp:116–119`) and `nisps_ml_explore_apply` perturbs every entry up to `n_out` (`bindings.cpp:880–887`). Freeze is defeated whenever Explore > 0. |
| D4 | **Input clamping and pinning do not exist in the browser.** `nisps_ml_set_input` bounds-checks the index only (`bindings.cpp:434–441`); `MLPCore::set_input` does not clamp (`mlp.hpp:119–122`). No browser path calls anything equivalent to `set_input_pinned`. | The single/double-joystick neutralise is a firmware-only feature today (`glue/settings_view.hpp`), although the operator decision recorded in memory covers "firmware + manifold input toggle" |
| D5 | **Initial state differs.** Firmware `setup()` sets every channel to `0.5f` before its first inference (`base.hpp:129`). The browser's boot inference comes from `ConsoleApp`'s mount effect calling `engine.setInput(0.5, 0.5)` (`ConsoleApp.tsx:268`), and `Spine.setInputs` zero-fills axes 2..N (`spine.ts:277–281`). | For any mode with >2 inputs the two targets boot from different points in input space |
| D6 | **Stateful gestures advance per *call*, not per unit of time.** `Jolt::morph_rate` is documented "EMA per tick, ~1s @200Hz" (`jolt.hpp:44`); `Jolt::tick_lr_ramp` adds a fixed `0.001` per call (`jolt.hpp:107–111`); `OUNoise` uses a fixed internal `dt_ = 0.001f`, *not* the caller's dt (`ou_noise.hpp:74, 96`). Firmware calls the tick at exactly 200 Hz. The browser advances the OU walk on **every spine tick** — i.e. faster when the user moves the pad faster — plus a 33 Hz idle driver. | The same Explore setting produces materially different roam speeds on the two targets, and a browser-side rate that depends on user gesture speed |
| D7 | **`jolt_lr_scale()` has zero consumers on either target.** | `grep -rn 'jolt_lr_scale\|lr_scale' nisps/ firmware/ manifold/src tests/ vcv/src` → only `base.hpp:246`, `jolt.hpp:65`, and tests. The firmware's two `train()` call sites (`glue/peripherals.hpp:89, 225`) do not scale the LR. The documented "training eases back in after a jolt" behaviour is **not wired anywhere**. |
| D8 | **`nisps/pipeline/` is WASM-only.** `InputChain`/`OutputChain` are included by `bindings.cpp:76–77` and `tests/cpp/parity_check.cpp:67–68` and by nothing else. | Firmware has no input conditioning and no output smoothing/slew/freeze. (`simplification-plan` §6.5d already assigns this wiring to the hardware-editor item, L29.) |
D1, D2, D6 are what this document fixes. D3, D4, D5 are fixed as a side effect and are
behaviour changes that need the operator's eyes (§5, §8). D7 and D8 are named here so an
implementer does not "discover" them mid-flight and widen scope; both are out of scope (§9).
---
## 2. What this design deliberately does **not** do
- **It does not bind mode objects into WASM.** A `ModeBase` instance owns an MLP *and* an
engine. In the browser those live in different threads and different WASM instances
(§1.2): the MLP on the main thread, the engine in the AudioWorklet, joined by a
`postMessage` of a `Float32Array` (`engine-host.ts:162–168`). Binding a mode object would
require either running the engine on the main thread (kills the RT guarantee) or the MLP in
the worklet (kills training and the whole feedback surface). The two-instance split is a
locked decision from the one-core refactor; this design respects it.
- **It does not delete the per-primitive C API.** `nisps_ml_*` stays exactly as it is —
`simplification-plan` §6.5a and audit A5 both say so explicitly, and the parity harness,
the training worker, the feedback controller and the probe all consume it.
- **It does not move the audio engine, the voice-space surface, the `ControlEvent` ring, or
`driver_config()`.** Those are the *other* half of a mode and belong to items 5b (browser
mode coverage / audio topology) and 5d (hardware editor). Mixing them in is how this becomes
a month.
- **It does not touch `nisps/pipeline/`'s ownership.** The input chain stays a caller-side
pre-stage and the output chain stays a caller-side post-stage, both on the existing
`nisps_pipeline_*` handle. Only their *ordering relative to OU* changes (§5).
---
## 3. Invariants
The repo-wide constraints (`CLAUDE.md`) are the floor. These are the ones specific to this area:
1. **`nisps/` stays platform-neutral and heap-free.** The new shared code lives under
`nisps/modes/`, and its fixed storage policy must be `std::array`-only. The dynamic policy is
a second deliberate exception to the no-heap rule and must carry the same
`#if defined(NISPS_TARGET_EMBEDDED) → #error` guard as `nisps/ml/dynamic_storage.hpp:22–24`,
and must be added to **both** the allowlist branch (`scripts/lint-cpp.sh:146`) and the
guard-survival check (`:194–198`). Today that script hardcodes exactly one path; generalise
it to a list rather than adding a second special case.
2. **Firmware behaviour is bit-identical at the current cadence.** With the mode ticked at
5 ms, every mode's audio output must be unchanged. `tests/cpp/test_mode_learning.cpp`'s
inert-parity test and `test_mode_paf_synth.cpp`'s reference-vector tests are the existing
guards; step 1 of §6 adds a stronger one.
3. **The `nisps::Mode` concept keeps its current shape** (`nisps/core/concepts.hpp:70–88`).
`tick_control()` must remain callable with no arguments, or every mode `static_assert` and
all of `firmware/glue/` breaks.
4. **The control tick stays bounded and allocation-free.** It runs on core 0 on firmware and on
the main thread in the browser; neither may block. Any catch-up loop introduced for
rate-independence (§4.3) must be hard-capped.
5. **Dual-core ownership and SPSC discipline are preserved.** `engine_.set_params()` is written
from core 0 while `engine_.process()` runs on core 1 — that is the existing arrangement and
this change must not add a second writer. The `ControlEvent` ring has exactly one producer
per mode (core 0 for the CC-emitting modes via `on_post_inference`, core 1 for the sequencer
modes via `pump_engine_events`); the ring must not move into shared code where a mode could
acquire two producers.
6. **Deterministic per-instance RNG.** Jolt and OU keep their own `nisps::Rng` streams,
independent of the MLP's and the feedback controller's. After this change the *salts must
match across targets* (D2) — that is the point.
7. **Native↔WASM parity ≤1e-5 must cover the new code.** The existing harness does not
(§7). Extending it is part of the work, not a follow-up.
8. **A WASM C-API addition walks all five layers**: `EMSCRIPTEN_KEEPALIVE` →
`EXPORTED_FUNCS` in `scripts/build-wasm.sh:40–72` → `NispsModule` in
`manifold/src/engine/types.ts` → `WasmIML` → `EngineApi`. A missed layer fails at runtime.
---
## 4. The design
### 4.1 The seam: `ControlCore<CtlStorage>`
A new header `nisps/modes/control_core.hpp` defining `nisps::ControlCore<CtlStorage>` — the
control tick and nothing else. It is storage-policied exactly the way `MLPCore<Storage>`
(`nisps/ml/mlp.hpp:91–92`) and `FeedbackControllerCore<FbStorage>` (`nisps/ml/feedback.hpp:42–51`)
already are, including the `ControlCore : public CtlStorage` + forwarded-`StorageArgs`
constructor idiom (`mlp.hpp:103–106`).
It **owns**: the input channels, the pin mask and pin value, a `ml::Jolt`, the OU walk and its
state, the produced parameter buffer, and the fixed-rate accumulator.
It **does not own** the MLP. Like `FeedbackControllerCore`, every method that needs the network
takes it by reference as a method-level template, so a `MLPCore<FixedStorage>` and a
`MLPCore<DynamicStorage>` both work:
```cpp
namespace nisps {
template <typename CtlStorage>
class ControlCore : public CtlStorage {
public:
template <typename... StorageArgs>
explicit ControlCore(std::uint64_t seed, StorageArgs&&... args) noexcept;
bool valid() const noexcept; // dynamic storage may fail to allocate
// ---- input channels (was ModeBase :165–196) ----
void set_input(std::size_t idx, float v) noexcept; // bounds-drop + clamp [0,1]
void set_input_pinned(std::size_t idx, bool p) noexcept;
bool is_input_pinned(std::size_t idx) const noexcept;
void set_pin_value(float v) noexcept;
float pin_value() const noexcept;
float effective_input(std::size_t i) const noexcept;
std::span<const float> input_channels() const noexcept;
std::span<float> mutable_input_channels() noexcept; // SoundAnalysisMIDI's hook
// ---- gestures (was ModeBase :241–255) ----
void jolt_press(std::size_t weight_count) noexcept;
void jolt_release() noexcept;
bool jolt_active() const noexcept;
float jolt_lr_scale() const noexcept;
void set_explore_intensity(float level) noexcept;
float explore_intensity() const noexcept;
// ---- the tick ----
// Runs `n` fixed-rate control steps (n derived from dt_s, see 4.3), one
// inference, and the OU walk. Returns the produced parameter vector — a
// STABLE span into this object's own buffer, valid until the next reshape.
template <typename MLPT>
std::span<const float> tick(MLPT& ml, float dt_s) noexcept;
std::span<const float> params() const noexcept; // same span, without ticking
void reset() noexcept; // clears OU state + accumulator
};
} // namespace nisps
```
### 4.2 Storage policies
Surface (documented in the header the way `nisps/ml/storage.hpp:20–28` documents the MLP's):
```
dims: n_in(), n_out()
buffers: input_channels(), input_pinned(), out_buf(), ou_state()
validity: valid()
```
- `nisps/modes/control_storage.hpp` — `FixedControlStorage<NIn, NOut>`: four `std::array`
members, all dims `constexpr`-foldable, zero heap. Firmware + host tests.
- `nisps/modes/dynamic_control_storage.hpp` — `DynamicControlStorage`: dims at construction,
**one arena allocation in the constructor and none after**, `#error` under
`NISPS_TARGET_EMBEDDED`. Browser only. Mirrors `nisps/ml/dynamic_storage.hpp` file-for-file
in shape so the lint allowlist reads as one pattern rather than two exceptions.
`FixedControlStorage`'s buffers are ~`2*NIn + 2*NOut` floats — for MEMLCelium (4 in, 56 out)
about 480 bytes.
**`jolt_buf_` is deleted, not moved.** `ModeBase::jolt_buf_` is a
`std::array<float, MLPType::weight_count()>` (`base.hpp:341`) — **6,152 bytes of SRAM on every
MEMLCelium-class mode** (`MLP<4,10,14,18,56>` ⇒ 1,538 weights). It exists only to hold a copy of
the flat weights across `jolt_.step()`. But `MLPCore`'s storage already owns exactly such a
buffer: `flat_buf()` (`nisps/ml/storage.hpp:164`, `nisps/ml/dynamic_storage.hpp:162`), which is
what `get_weights()` fills (`mlp.hpp:385–400`). Add a `std::span<float> weights_scratch()`
accessor to `MLPCore` and the jolt step becomes:
```cpp
const auto w = ml.get_weights(); // fills the storage's flat buffer
jolt_.step(ml.weights_scratch().first(w.size()));
ml.set_weights(ml.weights_scratch().first(w.size()));
```
`MLHandle::jolt_scratch` (`bindings.cpp:167`, a `std::vector<float>`) goes the same way.
**`OUNoise` loses its template parameter and splits `apply()` in two.**
`ml::OUNoise<N>` (`nisps/ml/ou_noise.hpp:44–99`) holds `std::array<float, N> state_` and its
`apply()` advances the walk and adds it to the output in one loop (`:68–81`). Two changes:
(i) the state span comes from the caller, so `OUNoise` becomes a plain class and `ControlCore`'s
storage owns the state; (ii) split into `advance(std::span<float> state)` and
`add_to(std::span<float> out, std::span<const float> state)`, because a catch-up tick must
advance N times and add once (§4.3a). Both keep the existing early-out at intensity 0, so the
inert path stays free and parity-safe. This also deletes the browser's
`OUNoise<kMaxDim>` — a 16 KB fixed array (`bindings.cpp:166`) sized for a net that is never
4096-wide. Three call sites: `base.hpp`, `bindings.cpp`, `tests/cpp/test_mlp_ou_noise.cpp`.
*Lower-risk fallback if this proves noisy: keep `OUNoise<N>` and have each storage policy expose
an `OU` typedef at its own N. It costs the browser its 16 KB and leaves a template where none is
needed, but it is strictly smaller.*
### 4.3 The canonical tick
```
tick(ml, dt_s):
steps = accumulate_steps(dt_s) # see below; firmware @5ms => exactly 1
for s in 0..steps-1:
if jolt.active(): jolt.step(ml's flat weight scratch) ; ml.set_weights(...)
jolt.tick_lr_ramp()
for i in 0..n_in-1: ml.set_input(i, effective_input(i))
ml.process()
copy ml.outputs() -> out_buf # ALWAYS, see below
for s in 0..steps-1: ou.advance(ou_state) # no-op while disabled
ou.add_to(out_buf, ou_state) # no-op while disabled
return out_buf
```
Four decisions are baked in here; each is a change from at least one target's current behaviour.
**(a) Fixed-rate stepping (fixes D6).** `ControlCore` owns a `float accum_` and a compile-time
`kControlPeriodS = 1.f / 200.f`. `accumulate_steps(dt_s)` adds `dt_s`, takes out whole periods,
and **caps the result at `kMaxCatchUpSteps` (8)** — invariant 4. At the firmware's exact 5 ms
cadence this yields exactly one step with zero residual, so firmware is unchanged; in the browser
a 30 ms driver call performs six steps and the walk becomes wall-clock-correct regardless of how
fast the user moves the pad. The rate constant, not wall-clock dt, stays the calibration for
`Jolt::morph_rate`, `Jolt::lr_ramp_step` and `OUNoise::dt_` — those are shape constants tuned at
200 Hz upstream. Re-deriving them from wall-clock dt is *not* equivalent: `OUNoise::dt_` is
`0.001`, not the firmware's 5 ms period, so feeding wall-clock dt would multiply firmware's
mean-reversion term by 5 and its per-step noise by √5.
**(b) Inference runs once per `tick()`, not once per control step.** Only the stateful walks are
rate-sensitive; a forward pass is idempotent given the same inputs and weights. This keeps a
catch-up tick cheap.
**(c) The output copy is unconditional.** Today firmware skips the copy when OU is disabled and
hands `ml_.outputs()` straight to the engine (`base.hpp:228`). If `tick()` did that, the returned
span would sometimes point into the MLP's output buffer and sometimes into `out_buf` — a
pointer-stability footgun for the C API, which caches the pointer JS-side. Always copying costs
`n_out` floats per tick (56 at 200 Hz for MEMLCelium: ~11 k float copies/s against a 1,538-weight
forward pass at the same rate) and is bit-identical.
**(d) OU is applied to the mode's parameter vector, before any caller-side output chain
(changes the browser — D3).** Firmware order is preserved; the browser's `OutputChain` moves from
*before* the OU walk to *after* it. Rationale: OU is part of what the mapping *produces*; the
output chain is how the *destination* conditions it. This is also what makes per-output freeze
mean something again. It is user-audible — see §5 and §8-Q1.
### 4.4 `ModeBase` afterwards
`ModeBase` keeps its entire public surface (invariant 3) and becomes a composition:
```cpp
NISPS_HOT void tick_control(float dt_s = kControlPeriodS) noexcept {
if constexpr (requires(Derived& d) { d.on_pre_inference(); })
static_cast<Derived&>(*this).on_pre_inference();
const auto params = control_.tick(ml_, dt_s);
if constexpr (kRouteOutputsToEngine) engine_.set_params(params);
if constexpr (requires(Derived& d) { d.on_post_inference(); })
static_cast<Derived&>(*this).on_post_inference();
}
```
with `control_` a `ControlCore<FixedControlStorage<NInputs, MLPType::kOutput>>`, and
`set_input` / `set_input_pinned` / `pin_value` / `input_channels` / `mutable_input_channels` /
`jolt_*` / `set_explore_intensity` / `explore_intensity` all one-line forwards. The default
argument preserves the concept and every existing call site (`main.cpp:157`,
`test_mode_*.cpp`).
What stays in `ModeBase`: the engine, the CRTP hooks, `setup()`, `driver_config()`,
`set_voice_space()`, the `ControlEvent` ring, `process()`. What leaves: the input-channel state,
`jolt_`, `ou_`, `out_buf_`, `jolt_buf_`, `apply_jolt_()`.
Two accessors need a call: `ml::Jolt& jolt()` (`base.hpp:247–248`) has **no consumer outside
`base.hpp`** — delete it. `ml::OUNoise& ou_noise()` (`:254–255`) is used by
`tests/cpp/test_mode_learning.cpp:89–103` — keep it (forwarding into `control_`) or rewrite those
three assertions against `explore_intensity()`; implementer's call.
The seeding fix (D2) rides along: `ControlCore`'s constructor takes the salts, and both targets
pass the same ones. Name them once, in `control_core.hpp`, as
`kJoltSalt = 0x91E10C5Eull` / `kOUSalt = 0x0CEA0FF5ull` (the firmware's current values, so
firmware weight streams are unchanged; the browser's jolt/OU streams change, which no committed
artifact or golden depends on — verified: no fixture, parity stage or e2e drives jolt/OU).
### 4.5 The browser afterwards
`MLHandle` (`bindings.cpp:148–185`) drops `jolt`, `ou` and `jolt_scratch` and gains one
`ControlCore<DynamicControlStorage>` member, constructed at `(d.n_in, d.n_out)` and rebuilt by
`nisps_ml_reshape` alongside the feedback controller (`bindings.cpp:396–421`).
The existing `nisps_ml_jolt_*` and `nisps_ml_explore_intensity` entries stay in place and
**forward into the head** — so the per-primitive C API survives (§2) while its *implementation*
stops being a second copy (kills D1).
`Spine.setInputs` becomes:
1. `iml.processInput(x, y, dt)` — unchanged.
2. `iml.setControlInput(i, v)` for each axis → `nisps_control_set_input` (clamped + pin-aware).
3. `iml.controlTick(dt)` → `nisps_control_tick` — replaces today's steps 3 *and* 5.
4. read the raw ML outputs (`nisps_ml_outputs`) into `mlBuf` for `liveOutputs`, and the head's
params (`nisps_control_params`) into `routedBuf`.
5. `iml.processOutput(routedBuf, dt)` — unchanged call, now *after* the OU walk.
6. `backendSend(routedBuf)` — unchanged.
`Spine.setOutputMorph` and the `outputMorph` field are **deleted** (`spine.ts:105–109, 221–229,
294–296`); its only consumer is `ExplorationController`'s constructor and `dispose`
(`exploration.ts:57, 131`). `ExplorationController` collapses from two timers to one ~60 Hz
driver that exists while `joltActive() || exploreIntensity() > 0` and simply calls
`engine.process()` — the fixed-rate accumulator makes the driver's exact period irrelevant.
### 4.6 The C API additions — the five-layer walk
| Export | `bindings.cpp` | `build-wasm.sh` `EXPORTED_FUNCS` | `types.ts` `NispsModule` | `WasmIML` | `EngineApi` |
|---|---|---|---|---|---|
| `void nisps_control_set_input(void* ml, int idx, float v)` | new | add | add | `setControlInput` | via `spine` |
| `void nisps_control_set_input_pinned(void* ml, int idx, int pinned)` | new | add | add | `setControlInputPinned` | `EngineApi.control.setInputPinned` |
| `void nisps_control_set_pin_value(void* ml, float v)` | new | add | add | `setControlPinValue` | `EngineApi.control.setPinValue` |
| `int nisps_control_tick(void* ml, float dt_s)` → n params | new | add | add | `controlTick` | via `spine` |
| `const float* nisps_control_params(void* ml)` | new | add | add | (internal) | — |
| `void nisps_ml_explore_apply(void* ml, float*, int)` | **delete** (`:880–887`) | **remove** (`:63`) | **remove** (`:133`) | **remove** `exploreApply` | **remove** `explore.exploreApply` |
Deleting `nisps_ml_explore_apply` is safe: its consumers are `WasmIML.exploreApply` →
`EngineApi.explore.exploreApply` → `exploration.ts:57`, all of which this change removes. It is
**not** touched by `tests/cpp/parity_wasm.mjs`, `manifold/tests/`, or `src/debug/probe.ts`
(verified by grep). Re-verify before deleting.
`nisps_ml_set_input` **stays**: `tests/cpp/parity_wasm.mjs` drives parity stage 1 through it, and
it is the raw-MLP write that `infer_batch` and the training worker semantics assume. The two are
distinguished in the header comment: `nisps_ml_set_input` = raw network input; `nisps_control_set_input`
= the mode control path (clamped, pinnable).
### 4.7 Deletions, with named consumers
| Deleted | Named consumers, and what happens to them |
|---|---|
| `ModeBase::jolt_buf_` (`base.hpp:341`) + `apply_jolt_` (`:312–320`) | `tick_control` only → replaced by `ControlCore::tick` over `MLPCore::weights_scratch()` |
| `MLHandle::jolt_scratch` (`bindings.cpp:167`) | `nisps_ml_jolt_step` only → same |
| `MLHandle::jolt`, `MLHandle::ou` (`:165–166`) | the `nisps_ml_jolt_*` / `nisps_ml_explore_*` bodies → forward into `control` |
| `ModeBase::ou_` / `out_buf_` / `input_channels_` / `input_pinned_` / `pin_value_` | `ModeBase` methods only → forwarded into `control_` |
| `ml::OUNoise<N>`'s template parameter + `state_` | `base.hpp`, `bindings.cpp:166`, `tests/cpp/test_mlp_ou_noise.cpp` → state moves to `ControlCore`'s storage |
| `ModeBase::jolt()` accessor (`:247–248`) | none — verified by grep across `nisps/ firmware/ manifold/ tests/ vcv/` |
| `nisps_ml_explore_apply` + its 4 downstream layers | see 4.6 |
| `Spine.outputMorph` + `setOutputMorph` (`spine.ts:105–109, 221–229, 294–296`) | `exploration.ts:57, 131` only |
| `ExplorationController`'s `joltTimer`/`exploreTimer` pair | internal → one driver |
---
## 5. Behaviour changes this lands
Stated plainly, because two of them are audible.
1. **Browser: the output chain now runs after the OU walk** (D3). With Explore > 0, per-output
freeze actually freezes, and the EMA/slew smooth the OU walk instead of the walk overwriting
smoothed values. Audible whenever Explore > 0 *and* smoothing/slew/freeze are non-default.
2. **Browser: the OU roam rate becomes wall-clock-correct and gesture-independent** (D6).
Today the walk advances once per spine tick, so it roams faster while the user is moving.
After: a fixed 200 steps/second. At the current 33 Hz idle driver the browser's walk is
~6× slower than firmware's; after the change they match. **Existing Explore slider positions
will feel different** — most users will want a lower setting.
3. **Browser: inputs are clamped to [0,1] and honour a pin mask** (D4). No current caller feeds
out-of-range values, so this is latent until the pin mask is wired into the UI (5c).
4. **Browser: jolt/OU RNG streams change** (D2), because the salts become the firmware's. No
committed artifact depends on them.
5. **Firmware: no behaviour change.** The tick order, the constants, the cadence and the
arithmetic are preserved by construction; §6 step 1 makes that a test rather than a claim.
The one *near*-exception is `jolt_lr_scale()`, whose ramp accumulates identically per step —
and which has no consumers on either target (D7).
---
## 6. Sequenced implementation plan
Each step is separately landable and separately verifiable. Step 1 is pure C++ and changes no
behaviour anywhere. Step 2 adds the browser-side plumbing and, because the jolt/OU salts and the
fixed-rate accumulator take effect as soon as `bindings.cpp` forwards into the head, it already
carries §5.2 and §5.4. Step 3 is the browser swap and carries §5.1 and §5.3.
**Sequencing hazard:** `nisps_ml_explore_apply` must be deleted in **step 3**, not step 2 — it is
still called by `WasmIML.exploreApply` until the TS swap lands, and removing the export first
leaves a runtime failure that no compiler catches (invariant 8, in reverse).
### Step 0 — capture a firmware-behaviour baseline (half a day)
Add `tests/cpp/test_mode_control_tick.cpp` to the `nisps_modes_tests` target
(`nisps/CMakeLists.txt:119–128`) that pins the *current* `tick_control` for at least PAFSynth,
MEMLCelium and SoundAnalysisMIDI: N ticks with a scripted input trace, asserting the exact
engine-param vector and the exact ML output vector, with Jolt and Explore both exercised.
Land it **before** touching `base.hpp` so it is a genuine before/after.
*Verification:* `bash scripts/build-cpp-tests.sh` (baseline today: 4/4 ctest suites pass in
0.03 s — run and confirmed 2026-07-21).
### Step 1 — `ControlCore` + fixed storage; `ModeBase` composes it (1–2 days)
New: `nisps/modes/control_storage.hpp`, `nisps/modes/control_core.hpp`. Modified:
`nisps/modes/base.hpp` (compose + forward), `nisps/ml/mlp.hpp` (`weights_scratch()`),
`nisps/ml/ou_noise.hpp` (externalise state). Concrete mode headers should need **no changes** —
if one does, the forwarding surface is wrong.
*Verification:* step 0's baseline test must pass **unchanged** — that is the whole point of doing
it first. Plus `nisps_modes_tests`, `nisps_core_tests`, `scripts/lint-cpp.sh`, and a firmware
compile of at least `slpworkshop`, `pafsynth`, `soundanalysismidi` via `scripts/build-firmware.sh`
(needs `nix-shell -p platformio-core`; first build pulls 1–2 GB). Compare flash/RAM against the
pre-change build — the expectation is a **reduction** of roughly `weight_count() * 4` bytes of
RAM (6,152 for MEMLCelium-class modes) from the `jolt_buf_` deletion. A size *increase* means
the composition failed to inline and wants investigating.
*Not verifiable here:* that the hardware still boots and sounds right. Operator chokepoint,
same as every firmware change in this repo.
### Step 2 — dynamic storage + the C API + parity coverage (1–2 days)
New: `nisps/modes/dynamic_control_storage.hpp`. Modified: `nisps/wasm/bindings.cpp` (MLHandle
member, reshape, the five new `nisps_control_*` exports, the `nisps_ml_jolt_*` /
`nisps_ml_explore_intensity` bodies forwarding into the head — but **not** the
`nisps_ml_explore_apply` deletion, which waits for step 3), `scripts/build-wasm.sh`,
`scripts/lint-cpp.sh` (allowlist → list).
**Add parity stage 8** to `tests/cpp/parity_check.cpp` and `tests/cpp/parity_wasm.mjs`: drive a
`ControlCore` over the harness's `MLP<32,10,14,18,126>` through a scripted trace — inputs set,
some channels pinned, a jolt pressed and released across several ticks, Explore raised, at least
one catch-up tick with `dt_s` > one period — and push the parameter vector plus a weight probe.
Bump `kVersion` 5 → 6 (`parity_check.cpp:93`) in both drivers, and extend the header's
authoritative stage list (`:14–33`) and `parity_diff.mjs`'s section names.
This is the step that converts "a parity PASS is not evidence for this change" into "it is".
Without stage 8, `scripts/parity-check.sh` exercises PAFSynth and ChannelStrip under an
all-params-0.5 vector and says nothing whatsoever about the control tick.
*Verification:* `bash scripts/build-wasm.sh && bash scripts/parity-check.sh` (both native and
WASM at 1e-5, now including stage 8); `bash scripts/lint-cpp.sh`;
`bash scripts/build-cpp-tests.sh`. `emcc`, `bun`, `cmake` and `ninja` are all on PATH here, so
this step is fully verifiable locally.
### Step 3 — the browser swap (1–2 days)
Modified: `manifold/src/engine/types.ts`, `wasm-iml.ts`, `engine-api.ts`, `spine.ts`,
`exploration.ts`. Deleted: `Spine.setOutputMorph` + the `outputMorph` field, `exploreApply`
through all four TS layers, one of the two exploration timers.
Rebuild and **commit** `manifold/public/nisps.{js,wasm}` in the same change — the webhook builds
only `manifold/` and ships whatever is committed (`manifold/ONBOARDING.md` §2), and CI's WASM
freshness gate runs the parity harness against the committed artifact before rebuilding it.
*Verification:* `cd manifold && bun run typecheck && bun run test && bun run build`, then
Playwright. Note that `manifold`'s test script is deliberately `bun test src tests/*.test.ts` —
do not "fix" it to `tests`, that drags the e2e suite into the unit run. On the VPS, e2e needs the
non-snap node runner (`ONBOARDING.md` §2).
Add e2e coverage for the two changes users can see, in `manifold/tests/e2e/`:
- Explore > 0 with a freeze mask set ⇒ the frozen outputs do **not** move (this fails today).
- Explore > 0 with the pad held still ⇒ the routed vector's drift over 2 s is within a
tolerance band, i.e. wall-clock-rated rather than tick-rated.
A unit test in the `bun test` set driving `nisps_control_tick` straight at the committed WASM —
the way `manifold/tests/loss-history.test.ts` drives `nisps_ml_loss_history` — is the cheapest
guard on the C ABI and should exist too.
### Step 4 — documentation sync (same commits)
`MAP.md` (`nisps/modes/` and `nisps/wasm/` bullets), `docs/specs/MAIN.md` (registry row +
"Loaded TWICE" is wrong — it is three instances), `manifold/ONBOARDING.md` §3 (same),
`ALIGNMENT.md` defect 1 (delete it when step 3 lands, per the repo's doc-sync rule),
`simplification-plan.md` §6.5a (burn down), and this file's `status:` → `executed`.
---
## 7. What the verification actually covers — and what it does not
| Claim | Covered by | Confidence |
|---|---|---|
| Firmware tick behaviour unchanged | step 0's pinned baseline + `nisps_modes_tests` | **High** — it is a before/after on the exact byte values |
| Fixed↔dynamic control head agree | parity stage 8 (step 2) | **High**, once written |
| Native↔WASM agreement of the new code | `scripts/parity-check.sh` **after** stage 8 exists | High after; **zero** before — the current harness touches neither modes nor jolt nor OU |
| The C API is reachable end-to-end | the `bun test` C-ABI test + Playwright | Medium-high — the 5-layer chain fails at runtime, so a browser-level test is the only real proof |
| Freeze is no longer defeated by Explore | new e2e case | High |
| OU roam rate is wall-clock | new e2e case (tolerance band) | Medium — timing assertions in a browser are inherently soft; keep the band wide |
| Firmware flash/RAM does not regress | `firmware-build` CI job's per-variant report + local `pio run` | High for size |
| **The hardware still boots and sounds correct** | *nothing* | **None.** This is an operator chokepoint, as it was for the PlatformIO cut (`simplification-plan` §5) |
| Engines other than PAFSynth/ChannelStrip behave identically | `nisps_dsp_engine_tests` + `engine_impulse` baseline; unchanged by this work | High, and unaffected — no engine code is touched |
Honest summary: everything except on-device behaviour can be verified in CI, **provided stage 8
is written**. If the implementation session runs out of time and skips stage 8, the change should
not land: the parity gate would go green while covering none of it.
---
## 8. Open questions for the operator
**Q1. The output-chain / OU ordering (§4.3d, §5.1) — confirm.** The recommendation is the
firmware's order: OU is part of the mapping, the output chain conditions what the destination
receives, and per-output freeze starts working. It changes the browser's sound whenever Explore
is up and smoothing/slew/freeze are non-default. The alternative — keeping each target's current
order — means the tick is not actually unified and needs a per-target branch, which is the thing
this whole item exists to remove. **Recommend: adopt the firmware order.**
**Q2. Fixed 200 Hz control rate as a shared contract (§4.3a) — confirm.** This makes the
browser's Explore feel roughly 6× faster at the same slider position. Options: (a) adopt it and
accept that existing slider positions read differently; (b) adopt it and rescale
`kOUMaxAmplitude` or the slider mapping so the browser's *current* feel is preserved — which
then changes the *firmware's* feel; (c) adopt it and expose the rate as a per-target constant,
which re-forks the behaviour. **Recommend (a)**, with a note in the Learning drawer copy.
**Q3. `jolt_lr_scale()` (D7) — wire it or delete it?** It is documented on both `Jolt` and
`ModeBase` as gating the learning rate after a jolt, and *nothing on either target multiplies by
it*. Wiring it means touching the firmware's two `train()` call sites and the browser's train
path — a behaviour change to training, not to this seam. Deleting it removes a documented
feature. It is cheap either way once `ControlCore` exists. **Not blocking**; needed before this
item can be called done rather than parked.
**Q4. Does the browser's input pin mask (D4) get UI in this item or in 5c?** The C API and the
core behaviour come free with `ControlCore`. Rendering a Single/Double control in the Inputs
drawer is UI work that belongs with the curated/advanced split
(`plans/curated-presets-spec.md`). **Recommend: ship the API in this item, the UI in 5c**, and
say so in `inputs-spec.md`.
**Q5. `DynamicControlStorage`'s home.** Recommended: `nisps/modes/`, with `lint-cpp.sh`'s
allowlist generalised from one hardcoded path to a list. The alternative — defining the dynamic
storage policy inside `nisps/wasm/bindings.cpp`, where heap is already legitimate and the lint
does not look — needs no lint change at all, but puts a storage policy outside `nisps/` where a
future consumer (VCV, a native fixture) cannot reach it. **Recommend `nisps/modes/`**; it is a
~10-line shell change and it keeps the "one auditable exception list" discipline.
Not asked, because they were decided here and are recorded rather than open: `ControlCore` does
not own the MLP (§4.1, matching `FeedbackControllerCore`); the parameter copy is unconditional
(§4.3c); the jolt scratch buffer is deleted in favour of `MLPCore`'s existing flat scratch
(§4.2); `OUNoise` loses its template parameter (§4.2); the salts become the firmware's (§4.4).
---
## 9. Explicitly out of scope
- **The engine half of a mode in the browser** — voice-space selection, `note_on`, transport
(`set_playing`/`update_bpm`), and the `ControlEvent` drain have no browser surface at all
(verified: `grep -rn 'voiceSpace\|noteOn\|setPlaying\|popEvents' manifold/src` returns only
generated-schema data). That is item **5b** — see `plans/browser-mode-coverage-spec.md` — and
it is the reason four catalogued modes cannot actually run in the browser.
- **Firmware `InputChain`/`OutputChain` wiring (D8, L29)** — assigned to **5d** by
`simplification-plan` §6.5d; see `plans/hardware-editor-spec.md`. This item only fixes where OU
sits relative to a chain the caller already owns.
- **Feedback-controller ownership.** Firmware keeps it as a function-local static in
`glue/peripherals.hpp:152`; the browser keeps it inside `MLHandle`. Both already call the same
shared core, so it is an asymmetry, not a duplication. The related `FOLLOW-UP` comment at
`glue/peripherals.hpp:33–39` (hold `feedback.static_output()` while Placing instead of running
fresh inference) *would* have a natural home in `ControlCore::tick` once it exists — note it,
do not build it here.
- **The browser's multi-anchor ExploreAndPlace session policy**
(`manifold/src/feedback/controller.ts:200–290`) differs from the firmware's freeze-and-carry
gesture. That is an application-policy divergence above the core, not a core duplication.
- **`jolt_lr_scale` wiring** (Q3) and **the pin-mask UI** (Q4).
---
## 10. Source-file index
Read in this order.
**The gap**
- `nisps/modes/base.hpp` — `ModeBase`; the tick is `:198–234`, the jolt copy `:312–320`
- `nisps/wasm/bindings.cpp` — `MLHandle` `:148–185`, jolt/OU C API `:827–886`, reshape `:396–421`
- `manifold/src/engine/spine.ts` — `setInputs` `:254–303`, the morph hook `:105–109, 221–229`
- `manifold/src/engine/exploration.ts` — the two timers and the morph registration
**The pattern being copied**
- `nisps/ml/mlp.hpp:91–116` + `nisps/ml/storage.hpp` + `nisps/ml/dynamic_storage.hpp` —
`MLPCore<Storage>`, the storage-surface documentation style, the embedded `#error` guard
- `nisps/ml/feedback.hpp:42–61` — `FeedbackControllerCore<FbStorage>`, and the
"does NOT own the MLP" convention this design follows
**Consumers that must keep compiling unchanged**
- `firmware/MEMLNaut-NISPS/glue/peripherals.hpp`, `settings_view.hpp`, `midi_io.hpp`,
`output_router.hpp`, `src/main.cpp:154–160`
- `tests/cpp/test_mode_{concepts,paf_synth,voice_space,breakor_events,learning,curve_overrides,driver_config}.cpp`
- `nisps/modes/sound_analysis_midi.hpp:86–111` and `external_synth_midi.hpp` — the only modes
with `on_pre_inference` / `on_post_inference` hooks, i.e. the sharpest test of the forwarding
surface
**Gates**
- `scripts/build-cpp-tests.sh`, `scripts/parity-check.sh`, `scripts/lint-cpp.sh`,
`scripts/build-wasm.sh`, `scripts/build-firmware.sh`, `.github/workflows/ci.yml`
- `tests/cpp/parity_check.cpp:14–33` (the authoritative stage list) + `parity_wasm.mjs` +
`parity_diff.mjs`