1f4513802f
- ALIGNMENT: retract the 2026-06-18 'geometric push not ported' accepted divergence (it IS ported); record the two deliberate divergences (useRandom via nisps::Rng; synchronous per-press SGD vs upstream's shuffled TrainBatch — behavioural parity with firmware, 1e-5 native↔WASM) - MAP: nisps/ml gains replay/geo_push/warm_start; feedback is storage- policied with Geometric default + Diffuse legacy - AGENT-REFERENCE: Jolt/OU/dislike TS-math limitation closed - plan: P3 software marked landed; chokepoint A pending hardware + bug 10c3e55c (explore wiring linker-GC'd out of PAFSynth ELF, pre-existing)
90 lines
9.4 KiB
Markdown
90 lines
9.4 KiB
Markdown
# ALIGNMENT
|
||
|
||
> Opinionated diagnosis of how well the codebase serves its mission, ranked by impact. Dated entries; remove when resolved rather than checking off. **Pruned every few weeks** — a stale diagnosis is worse than none.
|
||
|
||
## Mission
|
||
|
||
A research platform for interactive ML control of audio. We're building it to figure out what works and what doesn't — different ergonomics and ergodynamics of parameter sets, modes, ML architectures, audio engines, UI, and UX. Therefore: keep most/all parameters tweakable, ML/engine/UI/UX should each be configurable on their own axis, and the codebase has to enable/assist agentic AI coding patterns (confident changes, verifiable without hardware).
|
||
|
||
The clean-slate rewrite (2026-04-29) consolidated everything into one C++20 codebase compiling to firmware AND WASM. Since 2026-07-13 (P1 of `docs/specs/plans/one-core-engine-refactor.md`) the sole browser app is the React Manifold; the SolidJS playground is archived (`archive/playground-solidjs`). JSON schemas remain the firmware↔browser parameter contract.
|
||
|
||
## Top defects (ranked by mission impact)
|
||
|
||
### 1. Browser-only audio engines incomplete (2026-04-29; updated 2026-07-13)
|
||
|
||
**What.** C15 never got past a stubbed placeholder mode, and with the playground retired at P1 (2026-07-13) it has NO home on main — the stub UI, `c15.wasm`, and `c15-glue.js` live only on branch `archive/playground-solidjs`. Mic input for XIASRI / SoundAnalysisMIDI is likewise not wired in manifold.
|
||
|
||
**Why it blocks the mission.** "Browser engines ⊇ firmware engines" was a non-negotiable. Without C15 + mic input, manifold can't fully demonstrate the modes; users can't audition XIASRI or SoundAnalysisMIDI in the browser.
|
||
|
||
**Rough cost.** ~2 days. Reviving C15 now means porting the archived bridge into manifold's engine host. Mic input requires the engine-host to expose an input stream to the worklet (small worklet refactor).
|
||
|
||
### 2. Per-iteration loss curve not plumbed through WASM (2026-04-29)
|
||
|
||
**What.** Stream 7's WASM C API exposes `_nisps_ml_train` but only returns the final loss; `MLP::loss_history()` exists in C++ but isn't reached. `lossHistory` in `mlStore` is a single-element array per training run.
|
||
|
||
**Why it blocks the mission.** Gradient flow / loss visualization is a core UX affordance for "is the network learning?" — a research-mode debugging tool that's been implemented end-to-end in C++ but stops at the WASM boundary.
|
||
|
||
**Rough cost.** Half a day. Add `nisps_ml_train_with_history` (or extend the existing call) returning a pointer to the loss array; copy on the JS side.
|
||
|
||
### 4. `NISPS_AUDIO_FUNC` host fallback is misshapen (2026-04-29)
|
||
|
||
**What.** `nisps/core/perf.hpp` defines `NISPS_AUDIO_FUNC(decl) decl` for the host but the firmware path `__not_in_flash_func(name)` takes only a function name (it stringifies into a section attribute). The two forms don't match. Stream 6 (firmware glue) avoided the macro to dodge the inconsistency, but it's still a footgun.
|
||
|
||
**Why it blocks the mission.** Future agents touching `nisps/` will hit this. Either decoration form in the codebase is fine; what's wrong is that the same call site shape doesn't work both places.
|
||
|
||
**Rough cost.** Tiny. Pick one form and apply consistently:
|
||
- Option A: `NISPS_AUDIO_FUNC` decorates a function name (e.g. `void NISPS_AUDIO_FUNC(my_callback)(...) { ... }`). Host stub: `#define NISPS_AUDIO_FUNC(name) name`.
|
||
- Option B: separate `NISPS_AUDIO_FUNC_BEGIN` / `_END` markers around the function, or a different macro.
|
||
Pick A. Update perf.hpp + every `nisps/` use site.
|
||
|
||
### 5. RMSProp deferred from `nisps/ml/` (2026-04-29)
|
||
|
||
**What.** The legacy MLP supported both SGD and RMSProp paths. Stream 2 shipped only SGD as MVP. The architecture spec called for both. Documented as a follow-up Ergo task when needed.
|
||
|
||
**Why it blocks the mission.** Optimizer choice is one of the things research wants to vary. Not blocking for the current XOR-style fits, but as soon as we tune for harder loss landscapes, RMSProp will matter.
|
||
|
||
**Rough cost.** A day. Port the firmware's RMSProp from `src/memlp/MLP.cpp:415-543` (decay 0.9, epsilon 1e-6, gradient accumulation, batch size). Add tests for batch training convergence.
|
||
|
||
## Open mission questions
|
||
|
||
### Q1: Per-mode MLP architectures or one shared shape? (2026-04-29)
|
||
|
||
Schemas currently declare per-mode `hidden_layers` (some `[10, 10, 14]`, some `[10, 14, 18]`). Browser is fixed at one shape; firmware compiles per-mode. This works for now. Is the mission served by maintaining per-mode shapes (research diversity) or by collapsing to one (simpler ops)?
|
||
|
||
### Q2: How to express "advanced" features (gradient flow, weight health) without cluttering modes? (2026-04-29)
|
||
|
||
The retired playground reproduced the a-immersive "Advanced" toggle; manifold hides power features in drawers instead. Is this the right model, or should the mode UI itself decide what's exposed (some modes are "expert-only", some are simpler)?
|
||
|
||
### Q3: Engine event taxonomy (2026-04-29)
|
||
|
||
`nisps/modes/base.hpp` exposes a `ControlEvent` ring buffer pop_events interface for sequencer modes (BreakOr, Elysiamorf). Currently events are a flat enum. As we add more event-emitting modes (custom MIDI mappings, lighting, networked control), how should the event vocabulary grow? Open question; revisit when we add the third event-emitting mode.
|
||
|
||
### Q4: Should the browser app (manifold) stay desktop-first? (2026-04-29)
|
||
|
||
The original a-immersive was mobile-first ("designed for touch / foldable phone use"). The SolidJS rewrite is desktop-first by default. If the research story is "the user holds a phone and pinches to zoom while a synth runs in their pocket", we'll need a responsive pass. Defer until we have user data.
|
||
|
||
## Deferred / accepted debt
|
||
|
||
- **EOC effects chain integration** — out of v1 rewrite (recon flagged as legacy complexity).
|
||
- **ShapeSeq sequencer** — gated behind `?shapeseq=1` in legacy; out of v1.
|
||
- **Modular engine (Phase E)** — newer JS-side feature in legacy; out of v1.
|
||
- **Engine configuration panel** (SPEC-controls Part 8) — backlog. Would let users tune network architecture, loss, optimizer at runtime. Currently compile-time only.
|
||
- **VCV Rack module** — used to consume `nisps-core/`. Now gone. If revived, it'd consume `nisps/` directly via CMake; not currently maintained.
|
||
- **Geometric-dislike deliberate divergences** (2026-07-14, one-core-engine P3; supersedes and RETRACTS the 2026-06-18 "Avoid = move_weights, geometric push not ported" note — the k-NN centroid push IS now ported, upstream `InterfaceRL` @ `0a541cc`, into `nisps/ml/{replay,geo_push}.hpp` + `feedback.hpp` `AvoidStyle::Geometric` default). Two divergences are by design:
|
||
1. The upstream `useRandom` degenerate branch (disliked action exactly on the centroid) draws from the controller's deterministic `nisps::Rng`, not libc `rand()` — native==WASM parity holds (parity Stage 6); the value is generated, never compared against upstream.
|
||
2. Upstream trains via async `optimise()` with shuffled `TrainBatch` over positive+geometric batches at two LRs; nisps collapses press+optimise into ONE synchronous `dislike_geometric()` that trains only the pressed negative's target (per-sample SGD, no shuffle). Behavioural — not bitwise — parity with firmware upstream, by design; `native == WASM` is pinned at 1e-5 instead.
|
||
Also: `RandomiseMlp` still uses `draw_weights(spread)` rather than the old asymmetric `RandomiseWeightsAndBiasesLin(-0.9,1.1,-0.9,0.3)` (unchanged accepted divergence).
|
||
|
||
- **Manifold dock splits `state`/`muted`/`armed` into three fields, diverging from the deployed conflated `frozen`↔`muted`** (2026-06-28) — the deployed a-immersive override system maps the heatmap-popup `frozen` and the group-drawer `muted` onto ONE underlying field. The Manifold per-output model (`manifold/src/dock/output-state.ts`, folded onto `MFParam`) deliberately separates them: `status` carries the off/fixed/live tri-state, `muted` is downstream-silence (still computed + visible), `armed` is solo/focus-training. Cleaner semantics; intentional divergence (dock-spec §3.3, open choice 3). Note `muted`-downstream and the `soloMode` gradient-mask variants (mask-gradients / zero-loss / dont-care) are UI+state only so far — the engine C API exposes `set_focus` but not per-mode gradient masking nor a downstream mute gate yet (TODOs in `ConsoleApp.tsx` / `Drawers.tsx` reference rl-feedback-design §3 and dock-spec §3.3).
|
||
|
||
## Recently resolved (delete after a few weeks)
|
||
|
||
- 2026-07-14: Defect "WASM MLP architecture is fixed" resolved by one-core-engine P2: `nisps/ml/` is storage-policied (`MLPCore<Storage>`); the browser MLP is runtime-shaped (`DynamicStorage`), `nisps_ml_create` honours dims, `nisps_ml_reshape` warm-starts. Firmware keeps the zero-heap fixed template (`.text` +0.30%, within contract).
|
||
|
||
- 2026-04-29: Three-implementation ML duplication (firmware `memlp`, `nisps-core`, JS engine) collapsed to single `nisps/` C++ codebase.
|
||
- 2026-04-29: Firmware mode forks (~280–400 lines duplicated across 8 modes) collapsed via `nisps/modes/base.hpp` CRTP scaffold; concrete modes are now ~50–130 lines.
|
||
- 2026-04-29: meml-ues double-scaling MSE bug fixed in `nisps/ml/loss.hpp` + `mlp.hpp`.
|
||
- 2026-04-29: `nisps-core/` retired; firmware is the canonical source of truth for ML.
|
||
- 2026-04-29: `src/memlp/` submodule deleted.
|
||
- 2026-04-29: Legacy playground variants (a-immersive.html, b-workbench, c-journey, designs.html, all `js/`) deleted in favor of SolidJS scaffold.
|
||
- 2026-04-29: Native↔WASM parity verified within 1e-5 (max delta 2.4e-7) for representative ML + engine outputs.
|