memlnaut-nisps/docs/specs/plans/one-core-engine-refactor.md

kind	status
plan	active

One Core Engine — Firmware + Manifold Reunification Plan

Dated 2026-07-13. Operator-confirmed scope (see §1). Principle: one core engine (nisps/) for both the resource-constrained RP2350 hardware and the Manifold browser app. Everything with behaviour — ML, feedback, exploration, RNG, DSP, input/output processing, curves — lives once, in C++. TypeScript keeps only UI, reactivity, and transport. This plan supersedes the browser-side "one engine, two skins" framing of ../engine-architecture.md (that seam is built — Manifold is its realisation); the axis of unification now runs vertically, core↔targets, not horizontally, skin↔skin.

0. Where duplication actually lives today (audited 2026-07-13)

Component	Canonical	Duplicate / divergence	Disposition
MLP, training, init, RL move_weights	nisps/ml/	vcv/src/iml.hpp full runtime reimpl ("delta #5")	Reunify (P2, P6)
Feedback explore-and-place	nisps/ml/feedback.hpp	none — firmware + manifold both call it ✓	Keep
Geometric dislike (Mode 1)	(missing from core)	manifold/src/feedback/controller.ts TS approximation, explicit C++ GAP markers (lines ~369, ~436)	Port to core (P3)
Jolt weight-morph	nisps/ml/jolt.hpp	playground/src/ml/jolt.ts TS port; absent in manifold	Expose via WASM (P3)
OU exploration noise	nisps/ml/ou_noise.hpp	playground/src/output/ou-explore.ts TS port; absent in manifold	Expose via WASM (P3)
Deterministic RNG	nisps/core/rng.hpp	manifold/src/feedback/rng.ts (SeededRng, self-described stand-in)	Replace with core stream (P3)
Input pipeline	(TS-only concept)	playground/src/input/pipeline.ts ≡ manifold/src/engine/input-pipeline.ts	Port to core (P4)
Output pipeline + curves	curves mirror nisps/core/math.hpp	byte-identical TS pairs in playground + manifold	Port to core (P4)
Audio DSP engines (×8)	nisps/engines/	none — both targets run the WASM ✓	Keep
TS engine spine (wasm-iml, worker, host, worklet)	—	manifold forked from playground; several files byte-identical	Single home in manifold (P1)
Compiled nisps.wasm	scripts/build-wasm.sh → playground/public/	manifold/public/ byte-identical manual copy	Retarget build (P0)
Mode param schemas	schemas/modes/*.json → codegen	manifold bypasses codegen; hardcodes MF_MODES in console/model.ts	Codegen for manifold (P5)
WASM MLP arity	bindings.cpp MLP<32,10,14,18,126>; nisps_ml_create ignores requested dims	forces clamping, blocks per-mode arch, motivated the VCV reimpl	Runtime-shaped browser MLP (P2)

1. Locked decisions (operator, 2026-07-13)

Decision	Choice
playground/	Retired. Archived to branch archive/playground-solidjs, deleted from main. Manifold is the sole browser app; WASM build, parity-check, codegen all retarget manifold.
Depth of "one core"	All algorithms in C++. Feedback (incl. geometric dislike), Jolt, OU, RNG, input/output pipelines, curves move into nisps/; TS keeps UI/reactivity/transport only.
MLP arity	Template the hardware, dynamic browser. Firmware keeps compile-time MLP<...> (zero-heap contract intact); WASM/VCV targets get a runtime-shaped MLP sharing the same algorithm code. Heap allowed at construction time on non-RP2350 targets only.
VCV	In scope, final phase (P6); may slip without blocking anything else.

Supersedes: BUILD-PLAN.md locked decision "Modular N×M MLP: multiple WASM modules + warm-start; NOT runtime-shaped" (2026-06-27). The runtime-shaped MLP is now the chosen mechanism; warm-start on reshape is retained as a feature of it.

2. Target architecture

                    ┌───────────────────────── nisps/ (ONE CORE) ─────────────────────────┐
                    │ core/   rng, math+curves, buffers, perf attrs                        │
                    │ ml/     algorithms templated on StoragePolicy:                       │
                    │           Fixed<NIn,H1,H2,H3,NOut>  (arrays, zero-heap → RP2350)     │
                    │           Dynamic                    (sized at construction → WASM,VCV)│
                    │         mlp / training / init / rl / jolt / ou_noise / feedback /    │
                    │         geo_push / replay  (geometric dislike lands here)            │
                    │ pipeline/  input chain (deadzone→zoom→curve→smooth→momentum)         │
                    │            output chain (curve→smooth→slew→freeze)      ← NEW        │
                    │ dsp/ engines/ modes/   (unchanged)                                   │
                    └──────────────────────────────────────────────────────────────────────┘
                       ▲ direct #include                ▲ nisps/wasm/bindings.cpp (C ABI)
              ┌────────┴─────────┐            ┌─────────┴──────────┐        ┌───────────────┐
              │ firmware/ glue   │            │ manifold/ (React)  │        │ vcv/ (P6)     │
              │ RP2350, Fixed    │            │ UI + reactive spine│        │ Dynamic       │
              │ storage, no heap │            │ + transports only  │        │ storage       │
              └──────────────────┘            └────────────────────┘        └───────────────┘

Litmus test per file: if it computes behaviour that affects sound, weights, or parameter values, it is C++ in nisps/; if it schedules, renders, or transports, it may be TS. The existing browser litmus (JSX/DOM = skin) still applies inside manifold; this plan adds the vertical one.

3. Phases

Each phase ends green on its test gate and is independently landable. File phases as ergo tasks (one per phase, sub-tasks per bullet); do not start a phase before its predecessor's gate is green.

P0 — Plumbing hygiene (hours, no behaviour change) — ✅ landed 2026-07-13

• ✅ scripts/build-wasm.sh emits to manifold/public/ (keep playground/public/ copy only until P1 lands).
• ✅ scripts/parity-check.sh (and tests/cpp/parity_wasm.mjs) read the manifold artifact.
• ✅ Fix stale doc: the MLP<2,…> line was in docs/AGENT-REFERENCE.md + nisps/wasm/README.md (MAP.md was already correct); both now say MLP<32u,10u,14u,18u,126u>.
• ✅ ml-debug.log/graphify-out/ were already absent from the tree; .claude/worktrees/ gitignored.
• Gate met: run-all-tests.sh green (Playwright leg via the BUILD-PLAN non-snap-node VPS runner); parity PASS reading manifold/public/ (max delta 2.4e-7); manifold typecheck+build green against the freshly-built artifact.

P1 — Retire playground, single TS home (≈1 day) — ✅ landed 2026-07-13

• ✅ Branch archive/playground-solidjs + tag playground-solidjs-final; playground/ deleted from main. Parity fixtures for P4 captured FIRST: manifold/tests/fixtures/ (288-event gesture trace, curve goldens, input/output pipeline goldens under 14+8 configs) + drift-guard pipeline-golden.test.ts. Note: input pipeline's momentum path reads performance.now() — fixtures pin a clock contract (see fixtures README). C15 (stub + c15.wasm) now lives only on the archive branch (ALIGNMENT defect #1).
• ✅ Jolt press + OU explore UI shells in the Learning drawer; interim TS math via get/set_weights in manifold/src/engine/{exploration,jolt,ou-explore}.ts, marked P3 SWAP POINT.
• ✅ Keeper Playwright specs ported: probe-api.spec.ts (15 tests), spine.spec.ts (4). Playground UI/ persistence specs dropped with the playground.
• ✅ References retargeted: run-all-tests stage 5 → manifold, ci.yml manifold-tests job, osc-bridge.yml (was already broken), codegen TS target removed (returns P5), AGENTS/README/MAP/ALIGNMENT/ AGENT-REFERENCE/specs-MAIN. The VPS deploy webhook already built only manifold — no change needed.
• Gate met: full run-all-tests.sh green with playground gone (ctest 4/4, parity PASS, lint clean, manifold 9 unit + 20 e2e).

P2 — Storage-policy split: templated hardware, dynamic browser (the structural centre, ≈1 wk) — ✅ landed 2026-07-14 (hardware timing spot-check of chokepoint B pending physical MEMLNaut)

• ✅ (landed 2026-07-13) Refactor nisps/ml/ so algorithms (forward, backprop/SGD, init, move_weights) are written once against a storage policy (mlp.hpp MLPCore<Storage>; jolt/OU/feedback already operate on the MLP surface and needed no change). Two models:
  • FixedStorage<NIn,H1,H2,H3,NOut> (nisps/ml/storage.hpp) — std::array, zero heap; MLP<...> is an alias preserving the full compile-time surface. RP2350 contract verified: PAFSynth .text 122324→122692 = +0.30% (±1% budget); ctest + golden vectors + WASM parity bit-stable.
  • DynamicStorage (nisps/ml/dynamic_storage.hpp) — sizes at construction, single arena allocation, nothing per-call. #errors under NISPS_TARGET_EMBEDDED; sole lint-cpp.sh heap-allowlist entry, with a lint check that fails if the guard is ever removed.
• ✅ (landed 2026-07-14, operator-approved) nisps_ml_create(input, output, hidden[]) honours its arguments (non-positive/null → the historical 32→[10,14,18]→126 defaults, keeping pre-P2 callers bit-identical). nisps_ml_reshape = new instance + warm-start copy of overlapping weights (nisps/ml/warm_start.hpp); feedback controller re-created (state resets — front-end modal). FeedbackController got the same storage split (FeedbackControllerCore<FbStorage>, fixed alias for firmware/tests, DynamicFeedbackStorage for the browser). nisps_ml_describe now takes the handle (null → default shape). Verified: reshape ABI smoke (dims honoured, overlap survives, invalid dims rejected), warm-start ctest (grow+shrink), parity PASS unchanged, firmware .text unchanged.
• ✅ (landed 2026-07-14) Manifold reshape wiring: WasmIML.reshape + EngineApi.reshape (buffers/worker/ spine follow the new dims; training-worker protocol carries hidden); reset-on-reshape confirm modal offered on genuine input-layout changes (default 32-in over-provisioned head untouched on load/decline); stale even/odd-blend copy deleted. The mean-blend clamping itself was already gone (dedicated dims); the min(n, inputSize) guard + zero-padding stay as the legitimate declined-reshape behaviour. XIASRI/sound-analysis multi-input modes become browser-viable (schema-real dims arrive at P5).
• Gate met: fixed↔dynamic bit-identical ctest (+ warm-start grow/shrink ctest + reshape ABI smoke); parity-check.sh native↔WASM PASS unchanged (2.4e-7); PAFSynth .text 122324→122692 (+0.30%, within ±1%); manifold 9 unit + 25 e2e. Hardware audio-callback timing check deferred to the chokepoint-B hardware session (operator, physical MEMLNaut).

P3 — Exploration + feedback fully in core (≈3–4 days) — ✅ software landed 2026-07-14; ⏳ chokepoint A hardware spot-check pending (operator, physical MEMLNaut)

• ✅ Geometric dislike: firmware k-NN centroid push (upstream InterfaceRL @ 0a541cc, verbatim constants) ported into nisps/ml/geo_push.hpp + replay.hpp + mlp.train_targets; the press+optimise halves collapsed into one synchronous dislike_geometric() per the ADR. AvoidStyle::Geometric is the Avoid default; Diffuse (legacy move_weights) kept for A/B. Exposed via nisps_ml_feedback_dislike_geometric (+ store_positive/counts/set_avoid_style). TS approximation + both C++ GAP blocks deleted; two call-site bugs fixed en route (raw vs HEARD post-pipeline vector — raw gives a zero-derivative no-op). Cold-start prompt per ADR §7.
• ✅ Jolt / OU: exposed through bindings (nisps_ml_jolt_press/step/release/lr_scale/..., nisps_ml_explore_intensity/apply; browser OUNoise<4096> over-provisioned); manifold's P1 UI shells switched at the marked swap point; interim TS ports (jolt.ts, ou-explore.ts) deleted. OU applies through the spine's output morph hook.
• ✅ RNG: manifold/src/feedback/rng.ts deleted; all feedback/exploration randomness is the core's seeded per-instance Rng streams. (Debug-only fixed seed/dt under ?debug=1 for deterministic e2e.)
• Gate: ✅ parity Stage 6 (v4) — scripted seeded feedback session (2 likes → centroid → 2 dislikes) produces identical weight trajectories native↔WASM (961 floats, 2.4e-7); ctest geo suite; manifold 9 unit + 27 e2e. ⏳ Hardware spot-check (chokepoint A) BLOCKED on the physical MEMLNaut — and on ergo bug 10c3e55c: the explore/place wiring is linker-GC'd out of the PAFSynth ELF (pre-existing, discovered during P3; the gestures are dead code on that build until fixed).

P4 — Input/output pipelines + curves into core (≈3 days) — ✅ landed 2026-07-18

• ✅ New nisps/pipeline/: input_chain.hpp + output_chain.hpp (capacity-templated), perf-contract structs, caller-supplied dt with an internal clock (no wall time — deterministic), serialisable state.
• ✅ Curves: nisps/core/math.hpp is the single catalog (+ parameterised centered_power); bindings expose nisps_curve_apply(id, x, param) + batch. TS curves.ts deleted. NOTE: the TS mirror's exp/log/ sigmoid/cubic maths silently diverged from the canonical catalog — the browser now behaves firmware-exact (re-baselined in curves-golden.json with a provenance note; ALIGNMENT entry).
• ✅ Manifold pipelines are thin WASM calls (one nisps_pipeline_create handle per WasmIML; state C++-side; config stays TS-side and is pushed over the 15-float wire layout).
• ✅ Golden TS-vs-C++ tests retired → the fixture suite drives the WASM directly.
• Gate met: spine e2e unchanged (27 e2e green); recorded-gesture regression against the P1 pre-migration fixtures passes — non-momentum configs at ≤5e-7, momentum configs at 1e-2 with the drift PROVEN inherent f32-vs-f64 (a byte-faithful f32 reference reproduces the WASM to <6e-8; documented in the test + fixtures README). Parity stage 7 (v5): native↔WASM pipeline/curve floats bit-identical.

P5 — Schema/codegen serves manifold (≈2 days) — ✅ landed 2026-07-18

• ✅ codegen/generate.ts emits manifold/src/modes/generated/ alongside the C++ headers. MF_MODES is derived from the generated schemas (real param names/groups/counts + per-mode ml config + engine_id); labels/glyphs/class/ordering stay a manifold overlay; xiasri + slp_workshop join the browser catalog; schema-less visualizer/c15 remain hand-written overlays on the default shape.
• ✅ Per-mode ml.input_size/output_size/hidden is REAL in the browser: mode switch reshapes the runtime MLP to the schema config (boot mode paf_synth runs 4→[10,10,14]→33, not the old fixed 32→126). Codegen gained the firmware-fit check (exactly 3 hidden layers; dims ≤4096).
• Gate met: codegen idempotence golden covers both languages and runs in run-all-tests.sh stage 5; schema-modes.spec.ts asserts dims/weight-count/param-count FROM the imported schemas for 4 modes; the existing shape-assuming specs were re-derived from schema constants. 33 e2e green.

P6 — VCV reunification (later; may slip) — ✅ landed 2026-07-18

• ✅ The vendored vcv/src/iml.hpp (DetRng + 3D-weight runtime MLP + Dataset) is now a THIN Rack-free adapter over MLPCore<DynamicStorage> + nisps::Rng + the core dataset; hidden [16,24,16] kept (maps 1:1 onto the 3-hidden topology). Patch JSON weights went flat (version 2→3). The module's double-buffered audio/worker threading discipline is unchanged (two self-contained adapter instances). Closes vcv-module.md delta #5.
• ✅ OSC server stayed vcv-local.
• Gate met: plugin.so builds/links clean against the Rack SDK; new ctest test_vcv_iml_parity.cpp — the adapter is memcmp-identical to driving the core directly across construction/draw/train/inference/move_weights(pin mask).

6. Execution wrap-up (2026-07-18)

All six phases landed on main with green gates (P0 2026-07-13 → P6 2026-07-18). The plan flips to status: executed once the two OPERATOR items close (physical MEMLNaut required):

• Chokepoint A — geometric-dislike feel spot-check on hardware (ergo 425f4a8f), blocked on ergo bug 10c3e55c (explore/place wiring linker-GC'd out of the PAFSynth ELF — pre-existing, discovered in P3).
• Chokepoint B (timing half) — audio-callback timing on hardware; the compile half (.text ±1%) passed at P2.

4. Risks

Risk	Mitigation
P2 template refactor destabilises firmware perf (chokepoint B)	Fixed-storage path stays std::array + same attrs; compare .text/RAM and audio-callback timing before/after; land behind a ctest parity gate.
Dynamic storage leaks heap use into firmware paths	Compile-time guard (#if NISPS_TARGET_EMBEDDED excludes DynamicStorage); lint-cpp.sh already fails heap under nisps/ — carve an explicit allowlist for the dynamic TU only.
Per-event WASM calls for pipelines add latency	Pointer-rate ≈120 Hz, one call each — negligible; batch API as fallback. rAF/canvas never calls WASM.
Geometric-dislike port changes feel vs firmware	Port from upstream source verbatim, seeded-trajectory parity test, then hardware A/B (chokepoint A) before deleting the TS fallback.
Retiring playground loses a working reference	Archive branch + tag; parity fixtures (recorded gestures, golden outputs) extracted before deletion in P1.
Manifold worklet + main thread now both depend on retargeted wasm build	P0 gate builds manifold from fresh artifact in CI, killing the manual-copy hazard permanently.

5. Doc-sync obligations

• On each phase landing: update MAP.md (it is the shipped-behaviour authority) and burn down this plan's phase table; when all phases land, set status: executed.
• MAIN.md registry: this file added; engine-architecture.md remains authoritative for the browser-side seam only; BUILD-PLAN.md's MLP row annotated as superseded (done 2026-07-13).
• ALIGNMENT.md defect #3 (fixed WASM arity) closes at P2; the accepted-divergence log entries for Jolt/OU TS ports and vcv delta #5 close at P3/P6.