memlnaut-nisps/ALIGNMENT.md

ALIGNMENT

Opinionated diagnosis of how well the codebase serves its mission, ranked by impact. Dated entries; remove when resolved rather than checking off. Pruned every few weeks — a stale diagnosis is worse than none.

Mission

A research platform for interactive ML control of audio. We're building it to figure out what works and what doesn't — different ergonomics and ergodynamics of parameter sets, modes, ML architectures, audio engines, UI, and UX. Therefore: keep most/all parameters tweakable, ML/engine/UI/UX should each be configurable on their own axis, and the codebase has to enable/assist agentic AI coding patterns (confident changes, verifiable without hardware).

Target vision (operator, 2026-07-20): (1) one C++20 NISPS core serving RP2350 firmware and the browser, performance-sensitive on the MCU; (2) firmware modes runnable as modes in Manifold; (3) Manifold defaults to curated presets/modes, with the maximalist surface behind an "advanced" dev mode used to author them; (4) PlatformIO for hardware, no more .ino; (5) Manifold doubles as interface/editor for the hardware MEMLNaut (settings, presets, training, examples, visualisation).

The clean-slate rewrite (2026-04-29) consolidated everything into one C++20 codebase compiling to firmware AND WASM. Since 2026-07-13 (P1) the sole browser app is the React Manifold. JSON schemas remain the firmware↔browser parameter contract. A full-repo audit (2026-07-21, docs/specs/recon/simplification-audit-2026-07.md) grounds the entries below; mitigations are phased in docs/specs/plans/simplification-plan.md.

Top defects (ranked by mission impact)

1. The mode layer is not shared: WASM re-orchestrates modes by hand (2026-07-21)

What. nisps/modes/ — the CRTP layer binding ML config, engine, voice-space and I/O — compiles only into firmware. nisps/wasm/bindings.cpp includes engines and ML primitives but zero mode headers, and Manifold re-assembles mode behaviour (jolt stepping, OU, routing) in TS. "Firmware and WASM share the same modes" is true only at the engine level; every ModeBase behaviour must be mirrored browser-side by hand.

Why it blocks the mission. Vision bullet 2 is precisely this. Until the control-tick orchestration exists once in C++, every new mode behaviour is a dual implementation with drift risk.

Rough cost. Spec first, then ~a week: storage-policy the ModeBase orchestration the way P2 did MLPCore (verified shape in plan §6.5a — not binding monolithic mode objects, which would contradict the locked two-instance RT architecture). Related honesty gap: Manifold currently catalogues 4 modes that structurally cannot run in the browser (no mic input, event-only engines) — plan §6.5b (absorbs the old C15/mic-input defect; C15 itself lives on archive/playground-solidjs).

2. No curated/advanced split and no in-UI mode picker — the UI fights vision 3 (2026-07-21)

What. Manifold is 100% dev-maximalist: five drawers of everything, no preset data model to author against, and mode switching exists only via the debug hook — there is no instrument picker in the UI at all (the plumbing, ctx.modes/setModeId, already exists unused). A stratum of decorative controls (training-param sliders, master volume, bpm, A/B, snapshots, fabricated gradient health) renders real-looking UI that drives nothing.

Why it blocks the mission. The default experience is supposed to be curated presets; the advanced surface is the authoring tool. Neither exists, and the decorative stratum actively misleads research use.

Rough cost. Product-model decision first (plan §7.6), then incremental: picker is days; the curated-preset model seeds from backends/presets.ts + schemas; disclosure via per-drawer depth levels. Deleting the decorative stratum is part of the Phase-1 sweep.

3. Arduino-CLI build machinery is actively hostile — vision 4 unstarted (2026-07-21)

What. The build script sed-mutates the committed .ino to select variants (polluting history), the mode list is triple-bookkept (a NISPS_ST_* token-paste table is already silently missing the currently-active SLPWorkshop variant), a symlink forest works around Arduino's include rules, and the toolchain globally mutates the installed TFT_eSPI library. Firmware compilation is in no automated gate anywhere.

Why it blocks the mission. Fragile-by-design builds are the opposite of "confident agentic changes"; the vision names PlatformIO explicitly. firmware/useq-celium/ already proves the PIO pattern in-repo.

Rough cost. 2–3 days, one cut (plan §5): env-per-variant platformio.ini, delete ~400 lines of hackery, then a firmware CI job. Gated on the memllib ownership decision (plan §7.5).

4. Manifold-as-hardware-editor is a facade (2026-07-21)

What. Vision bullet 5 exists as a 237-line Web Serial shell: sound connect lifecycle, zero protocol (saveModel/restoreModel/getSettings are literal stubs), and firmware has no serial command surface or on-device persistence to talk to.

Why it blocks the mission. The hardware research loop (train on device, inspect/curate in browser) is closed only by this bridge.

Rough cost. Week+, spec-first (plan §6.5d). The right discipline already exists in-repo: useq-celium's C-header wire truth + TS mirror + parity test; settings payloads should derive from schema codegen.

5. Dead mass and registry sprawl across every layer (2026-07-21)

What. Phase 1 landed 2026-07-21 and removed the bulk of this: the dead focus/altitude UI system, the decorative control stratum, 12 dead WASM API entries across the 5-file registration chain, the vendored daisysp tree, retired-playground artifacts and root planning relics, 5 unused primitives, the duplicate backend editor and catalogue, voice_space.hpp, fixed_buffer.hpp, the dead perf-macro regime, and the OSC bridge twin. What remains is the registry half: mode identity spread across ~6 hand-maintained registries with demonstrated drift, MLP dims typed twice, per-mode schema blocks hand-written in C++, and assorted stale specs presenting a deleted world as present tense.

Why it blocks the mission. The registries are dual-truth bugs waiting to fire (one already did: the selftest table). Stale specs are agent-confusing surface area.

Rough cost. Plan phase 3 (~2–3 days, codegen takes ownership) plus the docs disposition pass (§8). The behaviour bugs found en route (dataset-cap divergence, VCV 2-D input truncation, VCV audio-thread race and JSON) were fixed in phase 2 on 2026-07-21.

6. No performance measurement despite a performance-defined mission (2026-07-21)

What. The "super performance-sensitive" constraint is enforced only by static discipline (no-heap lint — itself with proven false negatives — and section attrs, 3/5 of which are dead macros). No benchmark, no CPU-load assertion, no flash/RAM size report on either target; the 16 KB dead buffer was found by reading, not by any gate.

Rough cost. ~A day for a host-side blocks-per-second benchmark + a per-variant size report in build-firmware.sh (plan §6.5f).

7. Training-health telemetry: decided, not yet built (2026-07-21)

What. Four fragments of one feature. Fragment 3 (decorative gradient-health UI) was deleted in Phase 1. The other three stand: a 16 KB loss-history buffer in every firmware MLP that nothing reads (nisps/ml/mlp.hpp); a WASM worker faking a 1-element loss history (manifold/src/engine/wasm-worker.ts:310, new Float32Array([loss])); and a real layer_stats / nisps_ml_get_layer_stats API plumbed end-to-end and consumed by nobody.

Decisions are now complete (operator, §7.3 + L25): telemetry becomes real, browser-only, behind a feature flag; the fakes go; and the firmware buffer stays — it is the on-device record the hardware editor (defect 4) will want, and it costs flash we demonstrably have (16% RAM on the largest variant). So the remaining work is one coherent job, not a judgement call: plumb loss_history through the C API (Drawers.tsx:263 already marks the gap), replace the worker's fake with it, and put the display plus get_layer_stats behind the advanced-mode flag.

Why it blocks the mission. "Is the network learning?" is a core research affordance, and today it looks answered while being fabricated — worse than absent.

Rough cost. ~A day, spec-light: plan §6.5e, no longer gated on anything.

8. RMSProp still deferred from nisps/ml/ (2026-04-29; reaffirmed 2026-07-21)

What. training.hpp ships SGD only; the legacy firmware used RMSProp for TrainBatch. Optimizer choice is a research axis. Not blocking current fits; will matter for harder loss landscapes. Port target: upstream MusicallyEmbodiedML memlp (the in-repo src/memlp copy is deleted; use the GitHub remote or archive branch).

Rough cost. A day, plus batch-convergence tests.

Open mission questions

Q1: Per-mode MLP architectures or one shared shape? (2026-04-29)

Schemas declare per-mode dims and since P5.3 both targets honour them. Is the mission served by maintaining per-mode shapes (research diversity) or collapsing to one (simpler ops)? Note the audit found all 9 mode schemas share copy-pasted ML defaults and 20 params are anonymous placeholders — the per-mode diversity is currently nominal (plan L40).

Q2: Engine event taxonomy (2026-04-29)

ControlEvent is a flat enum consumed by the two sequencer modes. Revisit when a third event-emitting mode lands.

Q3: Should Manifold stay desktop-first? (2026-04-29)

Legacy a-immersive was mobile-first; Manifold is desktop-first. Defer until user data exists.

Q4: Who owns memllib? — DECIDED, half-executed (2026-07-21)

Operator decision: vendor, self-contained in this repo. The inventory (docs/specs/recon/memllib-usage-inventory.md) settled the shape: there is no small load-bearing subset — it is all of memllib bar examples/ (~1.8 MB, 24/24 compiled TUs link). The fork is dissolved: its three commits touch only examples/, which the firmware never compiles and whose content already lives in nisps/ml/{jolt,ou_noise,feedback,geo_push}.hpp, so the submodule now points at upstream and is pinned to current main (verified: all three variants build, +316 bytes flash, and it brings the l r input swap hardware fix plus the NavigateToView the SelfTest variant was already written against). Remaining: the vendoring copy itself, which lands with the PlatformIO cut (plan §5). Delete this entry when it does.

Q5: Legacy feedback modes — delete or keep for A/B? (2026-07-21)

RandomiseOutputs/RandomiseMlp/Diffuse/on_drag have no product consumer, but docs/adr/rl-feedback-design.md explicitly kept Diffuse for A/B comparison. Deleting reverses a recorded decision — operator call (plan §7.1).

Deferred / accepted debt

• EOC effects chain, ShapeSeq sequencer, modular engine (Phase E) — legacy features consciously out of the v1 rewrite; revisit only if a mode wants them.
• Inputs multi-source composition (2026-06-28, reaffirmed 2026-07-21) — mix-and-match pad+gamepad+MIDI is a recorded, unreversed decision; the UI currently enforces exclusive single-source and the composition machinery sits dormant by design. Schedule or keep dormant — but the inputs-spec must stop presenting composition as current behaviour (plan §8).
• Schema content is partially placeholder (2026-07-21) — 20 anonymous "Param NN" slots across paf_synth/channel_strip/xiasri and copy-pasted ML defaults across all 9 modes. Name them during the first curated-preset pass per mode (plan §6.5c), or shrink output_size where the engine allows.
• Geometric-dislike deliberate divergences (2026-07-14, one-core P3): (1) the degenerate-branch RNG draws from the controller's deterministic nisps::Rng, not libc rand() — native==WASM parity holds; (2) upstream's async shuffled two-LR optimise() is collapsed into one synchronous dislike_geometric() training only the pressed negative's target — behavioural, not bitwise, parity with firmware upstream, by design; (3) RandomiseMlp uses draw_weights(spread) rather than the old asymmetric ranges. All intentional.
• Manifold dock splits state/muted/armed (2026-06-28) — deliberate divergence from the deployed conflated frozen↔muted model (dock-spec §3.3). muted-downstream and the soloMode gradient-mask variants remain UI-only; the C API exposes set_focus but no per-mode gradient masking yet. (The audit found soloMode behaviourally inert in the controller — plan L20 trims it until train_masked exists.)

Recently resolved (delete after a few weeks)

• 2026-07-21: Full-repo simplification audit landed (recon + plan + this rewrite). Superseded entries removed: "browser-only engines incomplete" (→ defect 2/plan 5b), "loss curve not plumbed" (→ defect 8), "NISPS_AUDIO_FUNC misshapen" (→ plan Phase 1, S21/L13), stale "VCV not currently maintained" note (vcv/ is active and consumes nisps/ directly post-P6).
• 2026-07-18: Browser curve maths unified onto the canonical nisps/core/math.hpp catalog at P4; four silently-divergent TS curves re-baselined.
• 2026-07-14: WASM MLP fixed-architecture defect resolved by P2 (MLPCore<Storage>; browser runtime-shaped, firmware zero-heap fixed).