60584606a8
Plan §6 says each P5 item is spec-first with its own session. These are the
specs; no implementation is authorised by them and none was written.
plans/mode-layer-reunification.md 5a — ALIGNMENT defect 1, the largest
architectural gap. Storage-policies the
ModeBase orchestration the way P2 did
MLPCore, rather than binding monolithic
mode objects into WASM (which would
contradict the locked two-instance RT
architecture).
plans/browser-mode-coverage-spec.md 5b — an audio-topology notion so Manifold
stops cataloguing 4 modes that
structurally cannot run in a browser.
plans/curated-presets-spec.md 5c — ALIGNMENT defect 2, built on the
operator's §7.6 definition: a curated
preset is configuration only, network
untrained.
plans/hardware-editor-spec.md 5d — ALIGNMENT defect 3, applying
useq-celium's existing discipline (C
header as wire truth + TS mirror + parity
test) to a MEMLNaut serial protocol.
2434 lines. I spot-checked their path citations mechanically against
git ls-files: of 170 backticked paths, every unresolved one is either a file
the spec proposes to create or a cross-reference to a sibling spec in this same
commit. None describes deleted code as live — which is the failure mode that
made half the existing corpus untrustworthy, and the reason the §8 pass earlier
today had so much to do.
The preset spec is the most valuable byproduct: en route it found five places
where existing docs still describe a deleted world, including
manifold-parity-features-spec.md §1.1 specifying a PipelineLayer over
engine/input-pipeline.ts and output-pipeline.ts, both deleted at one-core P4.
Also carries the doc sync for the telemetry and benchmark work in a77770f:
AGENT-REFERENCE gains the throughput and loss-history entries, and dock-spec
§1.3 records that its long-deferred diagnostics suite shipped PARTLY — the loss
curve and weight-health table are real, GradientFlow is not built and is not
planned as drawn (the core records no per-layer gradient magnitudes, and the
fabricated version was deleted in Phase 1).
196 lines
12 KiB
Markdown
196 lines
12 KiB
Markdown
# Detailed Agent Reference — MEMLNaut-NISPS
|
|
|
|
Supplement to [`../AGENTS.md`](../AGENTS.md); deep target, schema, build, and verification
|
|
detail lives here.
|
|
|
|
This file provides guidance to coding agents working with this repository.
|
|
|
|
## Overview
|
|
|
|
MEMLNaut-NISPS — Neural Interactive Shaping of Parameter Spaces. A research platform for interactive ML control of audio. **One C++20 codebase** (`nisps/`) compiles to two targets:
|
|
|
|
1. **RP2350 firmware** for the MEMLNaut hardware platform (`firmware/`).
|
|
2. **WASM** in the Manifold React browser app (`manifold/`) — same engines + ML, run through an AudioWorklet.
|
|
|
|
(The former SolidJS playground was retired 2026-07-13 at P1 of `docs/specs/plans/one-core-engine-refactor.md`; archived on branch `archive/playground-solidjs`, tag `playground-solidjs-final`. The browser-only C15 engine currently lives only there.) Parameter contracts are JSON schemas (`schemas/`) with codegen producing the C++ headers AND the TypeScript modules (both live since P5; CI fails if either is stale).
|
|
|
|
Project documentation: https://musicallyembodiedml.github.io/memlnaut/approaches/nisps
|
|
|
|
For the codebase index, see `MAP.md`. For strategic gaps and open mission questions, see `ALIGNMENT.md`.
|
|
|
|
**For anything UI-related in the Manifold front-end (`manifold/`), read `manifold/ONBOARDING.md` first** — it's a single-file agent orientation (run/build/deploy/test, the UI/engine-spine/WASM layering, the convertible Stages, the Dock + drawers, and the non-obvious gotchas).
|
|
|
|
## The `nisps/` core
|
|
|
|
```
|
|
nisps/
|
|
├── core/ types, perf attrs, concepts (AudioEngine, MLEngine, Mode), fixed/ring buffers, deterministic RNG, math
|
|
├── ml/ MLP class template (4-layer, 3 hidden); SGD, gradient clipping, spread-aware Xavier init,
|
|
│ RL move_weights with output pin mask + per-layer scaling + weight decay
|
|
├── dsp/ biquad, delay, reverb, filter, env, osc, pitch_shift, dc_blocker
|
|
├── engines/ 8 audio engines (paf_synth, channel_strip, xiasri, verb_fx, memlcelium, breakor,
|
|
│ elysiamorf, analysis) + NoOpEngine. Each satisfies the AudioEngine concept.
|
|
├── modes/ 8 platform-agnostic modes binding {ML, engine, voice space, abstract I/O channels}.
|
|
│ CRTP base eliminates the duplication that plagued firmware modes.
|
|
└── wasm/ Emscripten C API bindings (compiled only for WASM target)
|
|
```
|
|
|
|
Build: `cmake -S nisps -B nisps/build -G Ninja && cmake --build nisps/build && ctest --test-dir nisps/build`.
|
|
|
|
Tests: 4 executables (`nisps_core_tests`, `nisps_dsp_engine_tests`, `nisps_modes_tests`, `nisps_golden_tests`). Run all: `bash scripts/build-cpp-tests.sh`. Parity vs WASM: `bash scripts/parity-check.sh` (asserts native and WASM produce identical outputs within 1e-5).
|
|
|
|
Throughput: `bash scripts/bench-engines.sh` (per-engine ns/sample, blocks/s, realtime factor on native + WASM; `--compare <report.json>` for deltas). It **reports and never asserts** — no threshold, no failure mode. Read it before and after any change to `nisps/dsp/` or `nisps/engines/`; nothing else in the repo will tell you an engine got slower.
|
|
|
|
### Performance contract (RP2350)
|
|
|
|
These rules apply to **all** code under `nisps/`. They are inert in WASM but kept globally for consistency.
|
|
|
|
- **No heap.** No `new`, `malloc`, `std::vector` in hot paths. Use `std::array<T, N>`. (`nisps::FixedBuffer` is gone — deleted in the Phase 1 sweep; `std::array` was doing the same job.)
|
|
- **Constants discipline.** Float literals >255 used in hot paths must be `static const float val = X.f;` not inline.
|
|
- **`.f` suffix on all float literals.** No double promotion in audio/inference paths.
|
|
- **Hot-path attributes.** Apply `NISPS_HOT` / `NISPS_FORCE_INLINE` (from `nisps/core/perf.hpp`). The SRAM-section macros were deleted in the 2026-07 sweep — they had no real use sites.
|
|
- **No virtual dispatch in audio path.** `AudioEngine` and `Mode` are C++20 concepts, not interfaces.
|
|
- **Deterministic RNG.** All RNG state is per-instance; constructors take a seed; cross-platform parity tests rely on this.
|
|
|
|
Lint: `bash scripts/lint-cpp.sh` warns on missing `.f` and fails on heap/`Arduino.h` use under `nisps/`.
|
|
|
|
## The `firmware/` target
|
|
|
|
```
|
|
firmware/MEMLNaut-NISPS/
|
|
├── MEMLNaut-NISPS.ino # Entry point; mode selected via #define MEMLNAUT_MODE_TYPE
|
|
├── glue/
|
|
│ ├── audio_driver.hpp # memllib AudioDriver block callback → Mode::process per-sample
|
|
│ ├── peripherals.hpp # joystick / pots / buttons → Mode::set_input + ML primitives
|
|
│ ├── midi_io.hpp # MIDI in → mode handlers; drain ControlEvent ring → MIDI UART
|
|
│ ├── mode_select.hpp # type aliases firmware mode name → nisps::modes::*Mode
|
|
│ ├── output_router.hpp # drain_outputs() entry point
|
|
│ └── settings_view.hpp # wire_settings(): TFT/rotary menu (Joystick Dual/Single for 4-in modes)
|
|
└── src/{memllib,nisps} # symlinks (Arduino-CLI sketch tree convention)
|
|
```
|
|
|
|
Build: `scripts/build-firmware.sh [VARIANT]`. Verified compiling for PAFSynth, ChannelStrip, BreakOr on `rp2040:rp2040:solderparty_rp2350_stamp_xl:opt=Optimize3` with `-std=gnu++20`. Flash: `scripts/flash-firmware.sh`. One-shot: `scripts/build-and-flash-firmware.sh`.
|
|
|
|
### Dual-core orchestration (firmware)
|
|
|
|
- **Core 0**: UI loop, ML inference (`Mode::tick_control`), peripheral polling (5ms period).
|
|
- **Core 1**: Real-time audio processing (`Mode::process`), MIDI polling.
|
|
- **Sync**: `nisps::core::ring_buffer` (templated SPSC lock-free, replaces pico/util/queue) + memory barriers (`nisps::core::memory_barrier`, `write_volatile`/`read_volatile`).
|
|
|
|
## The `manifold/` target
|
|
|
|
```
|
|
manifold/ # Vite + React + TypeScript (the sole browser app)
|
|
├── src/
|
|
│ ├── engine/ # framework-neutral TS engine spine: wasm-iml, engine-host + worklet,
|
|
│ │ # thin WASM wrappers over nisps/pipeline + the curve catalog (P4), spine.ts, EngineProvider
|
|
│ ├── primitives/ # 12 design primitives as typed React
|
|
│ ├── console/ # convertible Console (CompositeStage, Dock, Drawers, Manifold canvas, …)
|
|
│ ├── dock/, backends/, inputs/, feedback/, settings/, serial/, midi-devices/
|
|
│ └── debug/probe.ts # window.__nisps behind ?debug=1 for Playwright
|
|
├── public/ # nisps.{wasm,js} — canonical build-wasm.sh output
|
|
└── tests/ # e2e Playwright specs + fixtures/ (P4 golden parity fixtures)
|
|
```
|
|
|
|
Dev: `cd manifold && bun install && bun run dev`. Build: `bun run build`. Typecheck: `bun run typecheck`. Unit: `bun run test`. E2E: `bunx playwright test` (on the VPS run the runner via non-snap node — BUILD-PLAN gotcha).
|
|
|
|
Read `manifold/ONBOARDING.md` before touching manifold UI — layering, Stages, Dock, gotchas.
|
|
|
|
## The `schemas/` + `codegen/` contract
|
|
|
|
Each mode has a `schemas/modes/<mode>.json` describing its parameters (name, label, range, default, curve, group), ML config (input/output sizes, hidden layers), voice spaces (names — bodies are inline lambdas in the C++ engine), and UI config. The meta-schema at `schemas/schema.json` validates these.
|
|
|
|
Codegen (`bun run codegen/generate.ts`) emits:
|
|
- `nisps/modes/generated/<mode>_schema.hpp` — `constexpr` C++ data, namespace `nisps::modes::generated`, re-exports `nisps::Curve` from `nisps/core/math.hpp`.
|
|
- `manifold/src/modes/generated/<mode>_schema.ts` (+ `types.ts`, `index.ts`) — typed const schemas; `MF_MODES` derives params/ml-config from them (P5).
|
|
|
|
Codegen validates the firmware fit (exactly 3 hidden layers; dims ≤4096) and is idempotent. The golden test (both languages) runs in `run-all-tests.sh` stage 5.
|
|
|
|
## WASM bridge
|
|
|
|
Two WASM instances at runtime:
|
|
|
|
1. **Main thread** (`manifold/src/engine/wasm-iml.ts`): ML inference + sync training + RL primitives, reporting into an injected `EngineSink`. Async training via disposable Web Worker (`wasm-worker.ts`).
|
|
2. **AudioWorklet** (`manifold/src/engine/worklet/nisps-processor.ts`): runs engine `process_block` per audio block. Loads `nisps.wasm` directly via `WebAssembly.compile` (no Emscripten glue in worklet). Bytes posted from main thread.
|
|
|
|
C API is in `nisps/wasm/bindings.cpp`. Build: `bash scripts/build-wasm.sh` (~94KB output to `manifold/public/`).
|
|
|
|
The browser MLP is runtime-shaped since P2 (`MLPCore<DynamicStorage>`): `nisps_ml_create` honours `(input, output, hidden[3])`; non-positive/null args default to `32→[10,14,18]→126`. `nisps_ml_reshape` swaps in a new shape warm-started from the overlapping weights (examples + feedback state reset). Per-mode dims have been schema-real since P5.3 on both targets — modes no longer slice a shared 126-wide default.
|
|
|
|
### Known limitations
|
|
|
|
- Mic input through the worklet for XIASRI / SoundAnalysisMIDI is not wired in manifold.
|
|
- C15 has no home on main (see `ALIGNMENT.md` defect 1, browser mode coverage).
|
|
- (P3, 2026-07-14) The browser Jolt/OU gestures and the geometric dislike run the C++ core through WASM: `nisps_ml_jolt_*`, `nisps_ml_explore_*`, `nisps_ml_feedback_dislike_geometric` — no TS gesture math remains.
|
|
- (§6.5e, 2026-07-21) The per-iteration loss curve IS plumbed: `nisps_ml_loss_history(ml, out, max)` returns the total entry count and fills `min(count, max)`, so `max=0` is a count probe. Both train paths publish it to the spine; `EngineApi.lossHistory()` reads it and `console/TrainingHealth.tsx` displays it at `expanded` drawer depth. `MLPCore::train()` resets the history per run; `train_targets()` (geometric dislike) does not record.
|
|
|
|
## URL parameters (manifold)
|
|
|
|
| Param | Range | Default | Effect |
|
|
|-------|-------|---------|--------|
|
|
| `debug` | 1 | _(off)_ | Exposes the `window.__nisps` debug probe. |
|
|
|
|
(The playground-era `tame`/`spread`/`preset` URL params died with the playground; `spread` survives as an engine concept — see below.)
|
|
|
|
### `spread` — sigmoid saturation control
|
|
|
|
The MLP uses ReLU hidden layers with a sigmoid output. With uniform [-1,1] weights, the sum of many weighted inputs at each layer drives sigmoid pre-activations far from zero (std dev ≈ √fan_in), causing outputs to saturate. The `spread` parameter addresses this:
|
|
|
|
- `spread=0` (polarised): uniform [-1,1] weights, RL noise cap 0.3, no decay. Outputs cluster at extremes — good for radical exploration.
|
|
- `spread=1` (centered): Xavier-scaled weights, RL noise cap 0.05, 10% weight decay per move. Outputs spread across [0,1] — better for fine-grained shaping.
|
|
- Intermediate values interpolate.
|
|
|
|
## Verification chokepoints (user-confirmed)
|
|
|
|
- **A. Hardware**: each firmware mode flashes and produces correct audio on RP2350.
|
|
- **B. RP2350 perf**: no regression vs current main.
|
|
- **C. Browser parity**: each firmware mode runs in browser via WASM, sounds equivalent.
|
|
- **D. a-immersive feature parity**: control surface, snapshots, A/B compare, region/param pins, heatmap, weight health, gradient flow, output pipeline, session presets.
|
|
- **E. CI green**: `bash scripts/run-all-tests.sh` (cmake build + ctest + WASM build + parity + lint + Playwright).
|
|
|
|
## Build system summary
|
|
|
|
```bash
|
|
# Initialize submodules (required for memllib)
|
|
git submodule update --init --recursive
|
|
|
|
# Codegen (run after editing any schemas/modes/*.json)
|
|
cd codegen && bun install && bun run generate.ts
|
|
|
|
# C++ host tests
|
|
bash scripts/build-cpp-tests.sh
|
|
|
|
# WASM
|
|
bash scripts/build-wasm.sh
|
|
|
|
# Cross-platform parity
|
|
bash scripts/parity-check.sh
|
|
|
|
# Engine throughput (reports; never fails)
|
|
bash scripts/bench-engines.sh
|
|
|
|
# Lint
|
|
bash scripts/lint-cpp.sh
|
|
|
|
# Firmware
|
|
scripts/build-firmware.sh PAFSynth # or any other variant
|
|
scripts/flash-firmware.sh
|
|
scripts/build-and-flash-firmware.sh
|
|
|
|
# Manifold
|
|
cd manifold && bun install
|
|
bun run dev # Vite dev (COOP/COEP enabled)
|
|
bun run typecheck
|
|
bun run test # bun unit tests
|
|
bun run build
|
|
bunx playwright test # on the VPS: node node_modules/.bin/playwright test
|
|
|
|
# All tests
|
|
bash scripts/run-all-tests.sh
|
|
```
|
|
|
|
## Issue tracking
|
|
|
|
Use `ergo` exclusively for coding-work tasks over the Holon core; see the `ergo` skill.
|
|
Do not use TodoWrite or markdown task lists. Legacy tracker stores are retired/frozen
|
|
and reference-only.
|