Supplement to [`../AGENTS.md`](../AGENTS.md); deep target, schema, build, and verification
detail lives here.
This file provides guidance to coding agents working with this repository.
## Overview
MEMLNaut-NISPS — Neural Interactive Shaping of Parameter Spaces. A research platform for interactive ML control of audio. **One C++20 codebase** (`nisps/`) compiles to two targets:
1.**RP2350 firmware** for the MEMLNaut hardware platform (`firmware/`).
2.**WASM** in a SolidJS browser playground (`playground/`) — same engines + ML, run through an AudioWorklet.
Browser audio engines are a superset of firmware engines (C15 is browser-only). Parameter contracts are JSON schemas (`schemas/`) with codegen producing both C++ headers and TypeScript types.
For the codebase index, see `MAP.md`. For strategic gaps and open mission questions, see `ALIGNMENT.md`.
**For anything UI-related in the Manifold front-end (`manifold/`), read `manifold/ONBOARDING.md` first** — it's a single-file agent orientation (run/build/deploy/test, the UI/engine-spine/WASM layering, the convertible Stages, the Dock + drawers, and the non-obvious gotchas).
Tests: 4 executables (`nisps_core_tests`, `nisps_dsp_engine_tests`, `nisps_modes_tests`, `nisps_golden_tests`). Run all: `bash scripts/build-cpp-tests.sh`. Parity vs WASM: `bash scripts/parity-check.sh` (asserts native and WASM produce identical outputs within 1e-5).
### Performance contract (RP2350)
These rules apply to **all** code under `nisps/`. They are inert in WASM but kept globally for consistency.
- **No heap.** No `new`, `malloc`, `std::vector` in hot paths. Use `nisps::FixedBuffer<T, N>` or `std::array<T, N>`.
- **Constants discipline.** Float literals >255 used in hot paths must be `static const float val = X.f;` not inline.
- **`.f` suffix on all float literals.** No double promotion in audio/inference paths.
│ └── debug/probe.ts # synchronous window.__nisps for Playwright
├── public/ # nisps.{wasm,js}, c15.{wasm,glue}
└── tests/e2e/ # Playwright specs + helpers
```
Dev: `cd playground && bun install && bun run dev`. Build: `bun run build`. Typecheck: `bun run typecheck`. E2E: `bunx playwright test`.
### Stores + reactivity
All stores use SolidJS `createStore` for objects, `createSignal` for primitives. ML outputs are stored in a separate Float32Array signal (per the migration plan's perf guidance). Persistence (debounced 200ms localStorage round-trip) wired in `playground/src/stores/persistence.ts`. The signal bus (`bus.ts`) handles cross-store events (`ml.*`, `mode.*`, `pin.*`, `ui.*`).
### Control surface
Three compound axes (Boldness / Memory / Precision) interpolate per-axis tables to drive ~6 underlying parameters each, with offset overrides ("trim-pot" model). State is in `control-store`. Six built-in control presets (Default, First Touch, Jazz Hands, Sculptor, Improviser, Microscope) available via the ModeShell control bar.
### Debug probe (Playwright)
`window.__nisps` is exposed synchronously and bypasses Solid reactivity (uses `untrack`/`batch`). API matches the `.local/recon/04-playground.md` spec — `setInputs`, `getOutputs`, `getLoss`, `train`, `thumbsUp`/`thumbsDown`, `randomise`, `clearExamples`, `inferBatch`, `getLayerStats`, `saveState`, etc.
## The `schemas/` + `codegen/` contract
Each mode has a `schemas/modes/<mode>.json` describing its parameters (name, label, range, default, curve, group), ML config (input/output sizes, hidden layers), voice spaces (names — bodies are inline lambdas in the C++ engine), and UI config. The meta-schema at `schemas/schema.json` validates these.
Codegen (`bun run codegen/generate.ts`) emits:
-`nisps/modes/generated/<mode>_schema.hpp` — `constexpr` C++ data, namespace `nisps::modes::generated`, re-exports `nisps::Curve` from `nisps/core/math.hpp`.
Codegen is idempotent. Golden test ensures regenerating produces byte-identical output.
## WASM bridge
Two WASM instances at runtime:
1.**Main thread** (`playground/src/ml/wasm-iml.ts`): ML inference + sync training + RL primitives. Update store after each call. Async training via disposable Web Worker (`wasm-worker.ts`).
2.**AudioWorklet** (`playground/src/audio/worklet/nisps-processor.ts`): runs engine `process_block` per audio block. Loads `nisps.wasm` directly via `WebAssembly.compile` (no Emscripten glue in worklet). Bytes posted from main thread.
C API is in `nisps/wasm/bindings.cpp`. Build: `bash scripts/build-wasm.sh` (~94KB output to `manifold/public/`, transitional copy to `playground/public/` until P1).
The WASM target is fixed at `MLP<32u, 10u, 14u, 18u, 126u>` (`nisps_ml_create` ignores requested dims — see `docs/specs/plans/one-core-engine-refactor.md` P2). Modes with smaller `input_size`/`output_size` use a slice.
- Loss history not yet plumbed through C API; `lossHistory` in the store is a single-element array per training run.
- Engine MLP architecture is fixed at compile time — supporting per-mode hidden-layer shapes would need either multiple WASM modules or runtime variation.
- Mic input through the worklet for XIASRI / SoundAnalysisMIDI is not wired; UI scaffolds render but feature is TODO.
- C15 voice space integration in C15Mode is a placeholder.
- The browser Jolt/OU controls (`playground/src/ml/jolt.ts`, `playground/src/output/ou-explore.ts`) reimplement the gesture math in TS rather than calling the C++ `ml::Jolt`/`ml::OUNoise` through WASM. They drive weights via the existing `nisps_ml_get/set_weights` bindings and use `Math.random()` (not the deterministic `Rng`) — fine for stochastic exploration aids, but firmware↔browser bit-parity of the noise itself is intentionally not guaranteed.
The MLP uses ReLU hidden layers with a sigmoid output. With uniform [-1,1] weights, the sum of many weighted inputs at each layer drives sigmoid pre-activations far from zero (std dev ≈ √fan_in), causing outputs to saturate. The `spread` parameter addresses this:
-`spread=0` (polarised): uniform [-1,1] weights, RL noise cap 0.3, no decay. Outputs cluster at extremes — good for radical exploration.
-`spread=1` (centered): Xavier-scaled weights, RL noise cap 0.05, 10% weight decay per move. Outputs spread across [0,1] — better for fine-grained shaping.
- Intermediate values interpolate.
## Verification chokepoints (user-confirmed)
- **A. Hardware**: each firmware mode flashes and produces correct audio on RP2350.
- **B. RP2350 perf**: no regression vs current main.
- **C. Browser parity**: each firmware mode runs in browser via WASM, sounds equivalent.
