bf3d088ff1
The three top-level orienting docs are each falsified by several of the eight preceding commits, so they land here as one sync rather than being split across commits that would each leave them half-true. Same phase, same push. MAP.md — removed fixed_buffer.hpp, voice_space.hpp, the src/daisysp entry and its symlink from the sketch-tree list, input_router.hpp/wire_inputs (inputs are wired by bind_peripherals now), test_fixed_buffer.cpp, SplitStage/ReadoutStrip/ InputMini, BackendAdvanced.tsx, and feedback/rng.ts; corrected the primitives count 12 -> 7; replaced the "engine LIFTED from playground/src" provenance; rewrote the perf-attribute convention and deleted the NISPS_AUDIO_FUNC gotcha (both macros and gotcha are gone). docs/AGENT-REFERENCE.md — the memory-section-attribute instruction now names only the macros that exist; input_router.hpp dropped from the firmware tree; symlink list and the submodule-init note no longer mention daisysp. ALIGNMENT.md — defect 5 (dead mass and registry sprawl) rewritten: the deletion half is done, so the entry now scopes to what actually remains, which is the registry/dual-truth half (Phase 3) plus the stale specs (docs pass). Per the ALIGNMENT convention this is a rewrite-to-current, not a checkbox.
191 lines
11 KiB
Markdown
191 lines
11 KiB
Markdown
# Detailed Agent Reference — MEMLNaut-NISPS
|
|
|
|
Supplement to [`../AGENTS.md`](../AGENTS.md); deep target, schema, build, and verification
|
|
detail lives here.
|
|
|
|
This file provides guidance to coding agents working with this repository.
|
|
|
|
## Overview
|
|
|
|
MEMLNaut-NISPS — Neural Interactive Shaping of Parameter Spaces. A research platform for interactive ML control of audio. **One C++20 codebase** (`nisps/`) compiles to two targets:
|
|
|
|
1. **RP2350 firmware** for the MEMLNaut hardware platform (`firmware/`).
|
|
2. **WASM** in the Manifold React browser app (`manifold/`) — same engines + ML, run through an AudioWorklet.
|
|
|
|
(The former SolidJS playground was retired 2026-07-13 at P1 of `docs/specs/plans/one-core-engine-refactor.md`; archived on branch `archive/playground-solidjs`, tag `playground-solidjs-final`. The browser-only C15 engine currently lives only there.) Parameter contracts are JSON schemas (`schemas/`) with codegen producing the C++ headers (TS emission returns at P5).
|
|
|
|
Project documentation: https://musicallyembodiedml.github.io/memlnaut/approaches/nisps
|
|
|
|
For the codebase index, see `MAP.md`. For strategic gaps and open mission questions, see `ALIGNMENT.md`.
|
|
|
|
**For anything UI-related in the Manifold front-end (`manifold/`), read `manifold/ONBOARDING.md` first** — it's a single-file agent orientation (run/build/deploy/test, the UI/engine-spine/WASM layering, the convertible Stages, the Dock + drawers, and the non-obvious gotchas).
|
|
|
|
## The `nisps/` core
|
|
|
|
```
|
|
nisps/
|
|
├── core/ types, perf attrs, concepts (AudioEngine, MLEngine, Mode), fixed/ring buffers, deterministic RNG, math
|
|
├── ml/ MLP class template (4-layer, 3 hidden); SGD, gradient clipping, spread-aware Xavier init,
|
|
│ RL move_weights with output pin mask + per-layer scaling + weight decay
|
|
├── dsp/ biquad, delay, reverb, filter, env, osc, pitch_shift, dc_blocker
|
|
├── engines/ 8 audio engines (paf_synth, channel_strip, xiasri, verb_fx, memlcelium, breakor,
|
|
│ elysiamorf, analysis) + NoOpEngine. Each satisfies the AudioEngine concept.
|
|
├── modes/ 8 platform-agnostic modes binding {ML, engine, voice space, abstract I/O channels}.
|
|
│ CRTP base eliminates the duplication that plagued firmware modes.
|
|
└── wasm/ Emscripten C API bindings (compiled only for WASM target)
|
|
```
|
|
|
|
Build: `cmake -S nisps -B nisps/build -G Ninja && cmake --build nisps/build && ctest --test-dir nisps/build`.
|
|
|
|
Tests: 4 executables (`nisps_core_tests`, `nisps_dsp_engine_tests`, `nisps_modes_tests`, `nisps_golden_tests`). Run all: `bash scripts/build-cpp-tests.sh`. Parity vs WASM: `bash scripts/parity-check.sh` (asserts native and WASM produce identical outputs within 1e-5).
|
|
|
|
### Performance contract (RP2350)
|
|
|
|
These rules apply to **all** code under `nisps/`. They are inert in WASM but kept globally for consistency.
|
|
|
|
- **No heap.** No `new`, `malloc`, `std::vector` in hot paths. Use `nisps::FixedBuffer<T, N>` or `std::array<T, N>`.
|
|
- **Constants discipline.** Float literals >255 used in hot paths must be `static const float val = X.f;` not inline.
|
|
- **`.f` suffix on all float literals.** No double promotion in audio/inference paths.
|
|
- **Hot-path attributes.** Apply `NISPS_HOT` / `NISPS_FORCE_INLINE` (from `nisps/core/perf.hpp`). The SRAM-section macros were deleted in the 2026-07 sweep — they had no real use sites.
|
|
- **No virtual dispatch in audio path.** `AudioEngine` and `Mode` are C++20 concepts, not interfaces.
|
|
- **Deterministic RNG.** All RNG state is per-instance; constructors take a seed; cross-platform parity tests rely on this.
|
|
|
|
Lint: `bash scripts/lint-cpp.sh` warns on missing `.f` and fails on heap/`Arduino.h` use under `nisps/`.
|
|
|
|
## The `firmware/` target
|
|
|
|
```
|
|
firmware/MEMLNaut-NISPS/
|
|
├── MEMLNaut-NISPS.ino # Entry point; mode selected via #define MEMLNAUT_MODE_TYPE
|
|
├── glue/
|
|
│ ├── audio_driver.hpp # memllib AudioDriver block callback → Mode::process per-sample
|
|
│ ├── peripherals.hpp # joystick / pots / buttons → Mode::set_input + ML primitives
|
|
│ ├── midi_io.hpp # MIDI in → mode handlers; drain ControlEvent ring → MIDI UART
|
|
│ ├── mode_select.hpp # type aliases firmware mode name → nisps::modes::*Mode
|
|
│ ├── output_router.hpp # drain_outputs() entry point
|
|
│ └── settings_view.hpp # wire_settings(): TFT/rotary menu (Joystick Dual/Single for 4-in modes)
|
|
└── src/{memllib,nisps} # symlinks (Arduino-CLI sketch tree convention)
|
|
```
|
|
|
|
Build: `scripts/build-firmware.sh [VARIANT]`. Verified compiling for PAFSynth, ChannelStrip, BreakOr on `rp2040:rp2040:solderparty_rp2350_stamp_xl:opt=Optimize3` with `-std=gnu++20`. Flash: `scripts/flash-firmware.sh`. One-shot: `scripts/build-and-flash-firmware.sh`.
|
|
|
|
### Dual-core orchestration (firmware)
|
|
|
|
- **Core 0**: UI loop, ML inference (`Mode::tick_control`), peripheral polling (5ms period).
|
|
- **Core 1**: Real-time audio processing (`Mode::process`), MIDI polling.
|
|
- **Sync**: `nisps::core::ring_buffer` (templated SPSC lock-free, replaces pico/util/queue) + memory barriers (`nisps::core::memory_barrier`, `write_volatile`/`read_volatile`).
|
|
|
|
## The `manifold/` target
|
|
|
|
```
|
|
manifold/ # Vite + React + TypeScript (the sole browser app)
|
|
├── src/
|
|
│ ├── engine/ # framework-neutral TS engine spine: wasm-iml, engine-host + worklet,
|
|
│ │ # thin WASM wrappers over nisps/pipeline + the curve catalog (P4), spine.ts, EngineProvider
|
|
│ ├── primitives/ # 12 design primitives as typed React
|
|
│ ├── console/ # convertible Console (CompositeStage, Dock, Drawers, Manifold canvas, …)
|
|
│ ├── dock/, backends/, inputs/, feedback/, settings/, serial/, midi-devices/
|
|
│ └── debug/probe.ts # window.__nisps behind ?debug=1 for Playwright
|
|
├── public/ # nisps.{wasm,js} — canonical build-wasm.sh output
|
|
└── tests/ # e2e Playwright specs + fixtures/ (P4 golden parity fixtures)
|
|
```
|
|
|
|
Dev: `cd manifold && bun install && bun run dev`. Build: `bun run build`. Typecheck: `bun run typecheck`. Unit: `bun run test`. E2E: `bunx playwright test` (on the VPS run the runner via non-snap node — BUILD-PLAN gotcha).
|
|
|
|
Read `manifold/ONBOARDING.md` before touching manifold UI — layering, Stages, Dock, gotchas.
|
|
|
|
## The `schemas/` + `codegen/` contract
|
|
|
|
Each mode has a `schemas/modes/<mode>.json` describing its parameters (name, label, range, default, curve, group), ML config (input/output sizes, hidden layers), voice spaces (names — bodies are inline lambdas in the C++ engine), and UI config. The meta-schema at `schemas/schema.json` validates these.
|
|
|
|
Codegen (`bun run codegen/generate.ts`) emits:
|
|
- `nisps/modes/generated/<mode>_schema.hpp` — `constexpr` C++ data, namespace `nisps::modes::generated`, re-exports `nisps::Curve` from `nisps/core/math.hpp`.
|
|
- `manifold/src/modes/generated/<mode>_schema.ts` (+ `types.ts`, `index.ts`) — typed const schemas; `MF_MODES` derives params/ml-config from them (P5).
|
|
|
|
Codegen validates the firmware fit (exactly 3 hidden layers; dims ≤4096) and is idempotent. The golden test (both languages) runs in `run-all-tests.sh` stage 5.
|
|
|
|
## WASM bridge
|
|
|
|
Two WASM instances at runtime:
|
|
|
|
1. **Main thread** (`manifold/src/engine/wasm-iml.ts`): ML inference + sync training + RL primitives, reporting into an injected `EngineSink`. Async training via disposable Web Worker (`wasm-worker.ts`).
|
|
2. **AudioWorklet** (`manifold/src/engine/worklet/nisps-processor.ts`): runs engine `process_block` per audio block. Loads `nisps.wasm` directly via `WebAssembly.compile` (no Emscripten glue in worklet). Bytes posted from main thread.
|
|
|
|
C API is in `nisps/wasm/bindings.cpp`. Build: `bash scripts/build-wasm.sh` (~94KB output to `manifold/public/`).
|
|
|
|
The browser MLP is runtime-shaped since P2 (`MLPCore<DynamicStorage>`): `nisps_ml_create` honours `(input, output, hidden[3])`; non-positive/null args default to `32→[10,14,18]→126`. `nisps_ml_reshape` swaps in a new shape warm-started from the overlapping weights (examples + feedback state reset). Modes currently still slice the default 126 outputs; per-mode dims become schema-real at P5.
|
|
|
|
### Known limitations
|
|
|
|
- Loss history not yet plumbed through C API; only the final loss of a training run reaches TS.
|
|
- Mic input through the worklet for XIASRI / SoundAnalysisMIDI is not wired in manifold.
|
|
- C15 has no home on main (see `ALIGNMENT.md` defect #1).
|
|
- (P3, 2026-07-14) The browser Jolt/OU gestures and the geometric dislike run the C++ core through WASM: `nisps_ml_jolt_*`, `nisps_ml_explore_*`, `nisps_ml_feedback_dislike_geometric` — no TS gesture math remains.
|
|
|
|
## URL parameters (manifold)
|
|
|
|
| Param | Range | Default | Effect |
|
|
|-------|-------|---------|--------|
|
|
| `debug` | 1 | _(off)_ | Exposes the `window.__nisps` debug probe. |
|
|
|
|
(The playground-era `tame`/`spread`/`preset` URL params died with the playground; `spread` survives as an engine concept — see below.)
|
|
|
|
### `spread` — sigmoid saturation control
|
|
|
|
The MLP uses ReLU hidden layers with a sigmoid output. With uniform [-1,1] weights, the sum of many weighted inputs at each layer drives sigmoid pre-activations far from zero (std dev ≈ √fan_in), causing outputs to saturate. The `spread` parameter addresses this:
|
|
|
|
- `spread=0` (polarised): uniform [-1,1] weights, RL noise cap 0.3, no decay. Outputs cluster at extremes — good for radical exploration.
|
|
- `spread=1` (centered): Xavier-scaled weights, RL noise cap 0.05, 10% weight decay per move. Outputs spread across [0,1] — better for fine-grained shaping.
|
|
- Intermediate values interpolate.
|
|
|
|
## Verification chokepoints (user-confirmed)
|
|
|
|
- **A. Hardware**: each firmware mode flashes and produces correct audio on RP2350.
|
|
- **B. RP2350 perf**: no regression vs current main.
|
|
- **C. Browser parity**: each firmware mode runs in browser via WASM, sounds equivalent.
|
|
- **D. a-immersive feature parity**: control surface, snapshots, A/B compare, region/param pins, heatmap, weight health, gradient flow, output pipeline, session presets.
|
|
- **E. CI green**: `bash scripts/run-all-tests.sh` (cmake build + ctest + WASM build + parity + lint + Playwright).
|
|
|
|
## Build system summary
|
|
|
|
```bash
|
|
# Initialize submodules (required for memllib)
|
|
git submodule update --init --recursive
|
|
|
|
# Codegen (run after editing any schemas/modes/*.json)
|
|
cd codegen && bun install && bun run generate.ts
|
|
|
|
# C++ host tests
|
|
bash scripts/build-cpp-tests.sh
|
|
|
|
# WASM
|
|
bash scripts/build-wasm.sh
|
|
|
|
# Cross-platform parity
|
|
bash scripts/parity-check.sh
|
|
|
|
# Lint
|
|
bash scripts/lint-cpp.sh
|
|
|
|
# Firmware
|
|
scripts/build-firmware.sh PAFSynth # or any other variant
|
|
scripts/flash-firmware.sh
|
|
scripts/build-and-flash-firmware.sh
|
|
|
|
# Manifold
|
|
cd manifold && bun install
|
|
bun run dev # Vite dev (COOP/COEP enabled)
|
|
bun run typecheck
|
|
bun run test # bun unit tests
|
|
bun run build
|
|
bunx playwright test # on the VPS: node node_modules/.bin/playwright test
|
|
|
|
# All tests
|
|
bash scripts/run-all-tests.sh
|
|
```
|
|
|
|
## Issue tracking
|
|
|
|
Use `ergo` exclusively for coding-work tasks over the Holon core; see the `ergo` skill.
|
|
Do not use TodoWrite or markdown task lists. Legacy tracker stores are retired/frozen
|
|
and reference-only.
|