203 lines
13 KiB
Markdown
203 lines
13 KiB
Markdown
# Detailed Agent Reference — MEMLNaut-NISPS
|
|
|
|
Supplement to [`../AGENTS.md`](../AGENTS.md); deep target, schema, build, and verification
|
|
detail lives here.
|
|
|
|
This file provides guidance to coding agents working with this repository.
|
|
|
|
## Overview
|
|
|
|
MEMLNaut-NISPS — Neural Interactive Shaping of Parameter Spaces. A research platform for interactive ML control of audio. **One C++20 codebase** (`nisps/`) compiles to two targets:
|
|
|
|
1. **RP2350 firmware** for the MEMLNaut hardware platform (`firmware/`).
|
|
2. **WASM** in the Manifold React browser app (`manifold/`) — same engines + ML, run through an AudioWorklet.
|
|
|
|
(The former SolidJS playground was retired 2026-07-13 at P1 of `docs/specs/plans/one-core-engine-refactor.md`; archived on branch `archive/playground-solidjs`, tag `playground-solidjs-final`. The browser-only C15 engine currently lives only there.) Parameter contracts are JSON schemas (`schemas/`) with codegen producing the C++ headers AND the TypeScript modules (both live since P5; CI fails if either is stale).
|
|
|
|
Project documentation: https://musicallyembodiedml.github.io/memlnaut/approaches/nisps
|
|
|
|
For the codebase index, see `MAP.md`. For strategic gaps and open mission questions, see `ALIGNMENT.md`.
|
|
|
|
**For anything UI-related in the Manifold front-end (`manifold/`), read `manifold/ONBOARDING.md` first** — it's a single-file agent orientation (run/build/deploy/test, the UI/engine-spine/WASM layering, the convertible Stages, the Dock + drawers, and the non-obvious gotchas).
|
|
|
|
## The `nisps/` core
|
|
|
|
```
|
|
nisps/
|
|
├── core/ types, perf attrs, concepts (AudioEngine, MLEngine, Mode), fixed/ring buffers, deterministic RNG, math
|
|
├── ml/ MLP class template (4-layer, 3 hidden); SGD, gradient clipping, spread-aware Xavier init,
|
|
│ RL move_weights with output pin mask + per-layer scaling + weight decay
|
|
├── dsp/ biquad, delay, reverb, filter, env, osc, pitch_shift, dc_blocker
|
|
├── engines/ 8 audio engines (paf_synth, channel_strip, xiasri, verb_fx, memlcelium, breakor,
|
|
│ elysiamorf, analysis) + NoOpEngine. Each satisfies the AudioEngine concept.
|
|
├── modes/ 8 platform-agnostic modes binding {ML, engine, voice space, abstract I/O channels}.
|
|
│ CRTP base eliminates the duplication that plagued firmware modes.
|
|
└── wasm/ Emscripten C API bindings (compiled only for WASM target)
|
|
```
|
|
|
|
Build: `cmake -S nisps -B nisps/build -G Ninja && cmake --build nisps/build && ctest --test-dir nisps/build`.
|
|
|
|
Tests: 4 executables (`nisps_core_tests`, `nisps_dsp_engine_tests`, `nisps_modes_tests`, `nisps_golden_tests`). Run all: `bash scripts/build-cpp-tests.sh`. Parity vs WASM: `bash scripts/parity-check.sh` (asserts native and WASM produce identical outputs within 1e-5).
|
|
|
|
Throughput: `bash scripts/bench-engines.sh` (per-engine ns/sample, blocks/s, realtime factor on native + WASM; `--compare <report.json>` for deltas). It **reports and never asserts** — no threshold, no failure mode. Read it before and after any change to `nisps/dsp/` or `nisps/engines/`; nothing else in the repo will tell you an engine got slower.
|
|
|
|
### Performance contract (RP2350)
|
|
|
|
These rules apply to **all** code under `nisps/`. They are inert in WASM but kept globally for consistency.
|
|
|
|
- **No heap.** No `new`, `malloc`, `std::vector` in hot paths. Use `std::array<T, N>`. (`nisps::FixedBuffer` is gone — deleted in the Phase 1 sweep; `std::array` was doing the same job.)
|
|
- **Constants discipline.** Float literals >255 used in hot paths must be `static const float val = X.f;` not inline.
|
|
- **`.f` suffix on all float literals.** No double promotion in audio/inference paths.
|
|
- **Hot-path attributes.** Apply `NISPS_HOT` / `NISPS_FORCE_INLINE` (from `nisps/core/perf.hpp`). The SRAM-section macros were deleted in the 2026-07 sweep — they had no real use sites.
|
|
- **No virtual dispatch in audio path.** `AudioEngine` and `Mode` are C++20 concepts, not interfaces.
|
|
- **Deterministic RNG.** All RNG state is per-instance; constructors take a seed; cross-platform parity tests rely on this.
|
|
|
|
Lint: `bash scripts/lint-cpp.sh` warns on missing `.f` and fails on heap/`Arduino.h` use under `nisps/`.
|
|
|
|
## The `firmware/` target
|
|
|
|
```
|
|
firmware/MEMLNaut-NISPS/
|
|
├── MEMLNaut-NISPS.ino # Entry point; mode selected via #define MEMLNAUT_MODE_TYPE
|
|
├── glue/
|
|
│ ├── audio_driver.hpp # memllib AudioDriver block callback → Mode::process per-sample
|
|
│ ├── peripherals.hpp # joystick / pots / buttons → Mode::set_input + ML primitives
|
|
│ ├── midi_io.hpp # MIDI in → mode handlers; drain ControlEvent ring → MIDI UART
|
|
│ ├── mode_select.hpp # type aliases firmware mode name → nisps::modes::*Mode
|
|
│ ├── output_router.hpp # drain_outputs() entry point
|
|
│ └── settings_view.hpp # wire_settings(): TFT/rotary menu (Joystick Dual/Single for 4-in modes)
|
|
└── src/{memllib,nisps} # symlinks (Arduino-CLI sketch tree convention)
|
|
```
|
|
|
|
Build: `scripts/build-firmware.sh [VARIANT]`. Verified compiling for PAFSynth, ChannelStrip, BreakOr on `rp2040:rp2040:solderparty_rp2350_stamp_xl:opt=Optimize3` with `-std=gnu++20`. Flash: `scripts/flash-firmware.sh`. One-shot: `scripts/build-and-flash-firmware.sh`.
|
|
|
|
### Dual-core orchestration (firmware)
|
|
|
|
- **Core 0**: UI loop, ML inference (`Mode::tick_control`), peripheral polling (5ms period).
|
|
- **Core 1**: Real-time audio processing (`Mode::process`), MIDI polling.
|
|
- **Sync**: `nisps::core::ring_buffer` (templated SPSC lock-free, replaces pico/util/queue) + memory barriers (`nisps::core::memory_barrier`, `write_volatile`/`read_volatile`).
|
|
|
|
## The `manifold/` target
|
|
|
|
```
|
|
manifold/ # Vite + React + TypeScript (the sole browser app)
|
|
├── src/
|
|
│ ├── engine/ # framework-neutral TS engine spine: wasm-iml, engine-host + worklet,
|
|
│ │ # thin WASM wrappers over nisps/pipeline + the curve catalog (P4), spine.ts, EngineProvider
|
|
│ ├── primitives/ # 12 design primitives as typed React
|
|
│ ├── console/ # convertible Console (CompositeStage, Dock, Drawers, Manifold canvas, …)
|
|
│ ├── dock/, backends/, inputs/, feedback/, settings/, serial/, midi-devices/
|
|
│ └── debug/probe.ts # window.__nisps behind ?debug=1 for Playwright
|
|
├── public/ # nisps.{wasm,js} — canonical build-wasm.sh output
|
|
└── tests/ # e2e Playwright specs + fixtures/ (P4 golden parity fixtures)
|
|
```
|
|
|
|
Dev: `cd manifold && bun install && bun run dev`. Build: `bun run build`. Typecheck: `bun run typecheck`. Unit: `bun run test`. E2E: `bunx playwright test` (on the VPS run the runner via non-snap node — BUILD-PLAN gotcha).
|
|
|
|
Read `manifold/ONBOARDING.md` before touching manifold UI — layering, Stages, Dock, gotchas.
|
|
|
|
## The `schemas/` + `codegen/` contract
|
|
|
|
Each mode has a `schemas/modes/<mode>.json` describing its parameters (name, label, range, default, curve, group), ML config (input/output sizes, hidden layers), voice spaces (names — bodies are inline lambdas in the C++ engine), and UI config. The meta-schema at `schemas/schema.json` validates these.
|
|
|
|
Codegen (`bun run codegen/generate.ts`) emits:
|
|
- `nisps/modes/generated/<mode>_schema.hpp` — `constexpr` C++ data, namespace `nisps::modes::generated`, re-exports `nisps::Curve` from `nisps/core/math.hpp`.
|
|
- `manifold/src/modes/generated/<mode>_schema.ts` (+ `types.ts`, `index.ts`) — typed const schemas; `MF_MODES` derives params/ml-config from them (P5).
|
|
|
|
Codegen validates the firmware fit (exactly 3 hidden layers; dims ≤4096) and is idempotent. The golden test (both languages) runs in `run-all-tests.sh` stage 5.
|
|
|
|
## WASM bridge
|
|
|
|
Two WASM instances at runtime:
|
|
|
|
1. **Main thread** (`manifold/src/engine/wasm-iml.ts`): ML inference + sync training + RL primitives, reporting into an injected `EngineSink`. Async training via disposable Web Worker (`wasm-worker.ts`).
|
|
2. **AudioWorklet** (`manifold/src/engine/worklet/nisps-processor.ts`): runs engine `process_block` per audio block. Loads `nisps.wasm` directly via `WebAssembly.compile` (no Emscripten glue in worklet). Bytes posted from main thread.
|
|
|
|
C API is in `nisps/wasm/bindings.cpp`. Build: `bash scripts/build-wasm.sh` (~94KB output to `manifold/public/`).
|
|
|
|
The browser MLP is runtime-shaped since P2 (`MLPCore<DynamicStorage>`): `nisps_ml_create` honours `(input, output, hidden[3])`; non-positive/null args default to `32→[10,14,18]→126`. Raw `nisps_ml_reshape` reconstructs and prefix-warm-starts a new shape. Manifold's higher-level `engine/io-reshape.ts` seam adds stable input/output identity: it can permute weights and examples without reconstruction while capacity suffices, or reconstruct with arbitrary surviving-dimension remaps. Persistent settings select capacity-vs-exact arity and adapt-vs-clear examples (neutral new-input/output defaults 0/0.5); feedback/exploration scratch state resets on either identity edit. Per-mode dims have been schema-real since P5.3 on both targets — modes no longer slice a shared 126-wide default.
|
|
|
|
### Known limitations
|
|
|
|
- Mic input through the worklet for XIASRI / SoundAnalysisMIDI is not wired in manifold.
|
|
- C15 has no home on main (see `ALIGNMENT.md` defect 1, browser mode coverage).
|
|
- (P3, 2026-07-14) The browser Jolt/OU gestures and the geometric dislike run the C++ core through WASM: `nisps_ml_jolt_*`, `nisps_ml_explore_*`, `nisps_ml_feedback_dislike_geometric` — no TS gesture math remains.
|
|
- (§6.5e, 2026-07-21) The per-iteration loss curve IS plumbed: `nisps_ml_loss_history(ml, out, max)` returns the total entry count and fills `min(count, max)`, so `max=0` is a count probe. Both train paths publish it to the spine; `EngineApi.lossHistory()` reads it and `console/TrainingHealth.tsx` displays it at `expanded` drawer depth. `MLPCore::train()` resets the history per run; `train_targets()` (geometric dislike) does not record.
|
|
|
|
## URL parameters (manifold)
|
|
|
|
| Param | Range | Default | Effect |
|
|
|-------|-------|---------|--------|
|
|
| `debug` | 1 | _(off)_ | Exposes the `window.__nisps` debug probe. |
|
|
|
|
(The playground-era `tame`/`spread`/`preset` URL params died with the playground; `spread` survives as an opt-in engine concept — see below.)
|
|
|
|
### `spread` — sigmoid saturation control
|
|
|
|
The MLP uses ReLU hidden layers with a sigmoid output. With uniform [-1,1] weights, the sum of many weighted inputs at each layer drives sigmoid pre-activations far from zero (std dev ≈ √fan_in), causing outputs to saturate. The `spread` parameter addresses this:
|
|
|
|
- `spread=0` (polarised): uniform [-1,1] weights, RL noise cap 0.3, no decay. Outputs cluster at extremes — good for radical exploration.
|
|
- `spread=1` (centered): Xavier-scaled weights, RL noise cap 0.05, 10% weight decay per move. Outputs spread across [0,1] — better for fine-grained shaping.
|
|
- Intermediate values interpolate.
|
|
|
|
Manifold deliberately defaults to `spread=0` at every browser boundary: initial construction,
|
|
mode-switch reshape, direct re-roll, explore-and-place scratchpad re-roll, and forwarded VCV
|
|
randomise. The old schema spread and the expanded Learning-drawer centred switch are available only
|
|
after enabling Settings → Experimental features → **Xavier / spread randomisation**. This is a
|
|
Manifold compatibility flag; the shared C++ core and generated mode schemas still expose spread for
|
|
firmware, VCV, benchmarks, and explicit API callers.
|
|
|
|
## Verification chokepoints (user-confirmed)
|
|
|
|
- **A. Hardware**: each firmware mode flashes and produces correct audio on RP2350.
|
|
- **B. RP2350 perf**: no regression vs current main.
|
|
- **C. Browser parity**: each firmware mode runs in browser via WASM, sounds equivalent.
|
|
- **D. a-immersive feature parity**: control surface, snapshots, A/B compare, region/param pins, heatmap, weight health, gradient flow, output pipeline, session presets.
|
|
- **E. CI green**: `bash scripts/run-all-tests.sh` (cmake build + ctest + WASM build + parity + lint + Playwright).
|
|
|
|
## Build system summary
|
|
|
|
```bash
|
|
# Initialize submodules (required for memllib)
|
|
git submodule update --init --recursive
|
|
|
|
# Codegen (run after editing any schemas/modes/*.json)
|
|
cd codegen && bun install && bun run generate.ts
|
|
|
|
# C++ host tests
|
|
bash scripts/build-cpp-tests.sh
|
|
|
|
# WASM
|
|
bash scripts/build-wasm.sh
|
|
|
|
# Cross-platform parity
|
|
bash scripts/parity-check.sh
|
|
|
|
# Engine throughput (reports; never fails)
|
|
bash scripts/bench-engines.sh
|
|
|
|
# Lint
|
|
bash scripts/lint-cpp.sh
|
|
|
|
# Firmware
|
|
scripts/build-firmware.sh PAFSynth # or any other variant
|
|
scripts/flash-firmware.sh
|
|
scripts/build-and-flash-firmware.sh
|
|
|
|
# Manifold
|
|
cd manifold && bun install
|
|
bun run dev # Vite dev (COOP/COEP enabled)
|
|
bun run typecheck
|
|
bun run test # bun unit tests
|
|
bun run build
|
|
bunx playwright test # on the VPS: node node_modules/.bin/playwright test
|
|
|
|
# All tests
|
|
bash scripts/run-all-tests.sh
|
|
```
|
|
|
|
## Issue tracking
|
|
|
|
Use `ergo` exclusively for coding-work tasks over the Holon core; see the `ergo` skill.
|
|
Do not use TodoWrite or markdown task lists. Legacy tracker stores are retired/frozen
|
|
and reference-only.
|