2026-02-08 15:49:16 +01:00
# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## Overview
2026-04-29 18:57:27 +02:00
MEMLNaut-NISPS — Neural Interactive Shaping of Parameter Spaces. A research platform for interactive ML control of audio. **One C++20 codebase** (`nisps/`) compiles to two targets:
2026-02-08 15:49:16 +01:00
2026-04-29 18:57:27 +02:00
1. **RP2350 firmware** for the MEMLNaut hardware platform (`firmware/`).
2. **WASM** in a SolidJS browser playground (`playground/`) — same engines + ML, run through an AudioWorklet.
2026-02-08 15:49:16 +01:00
2026-04-29 18:57:27 +02:00
Browser audio engines are a superset of firmware engines (C15 is browser-only). Parameter contracts are JSON schemas (`schemas/`) with codegen producing both C++ headers and TypeScript types.
feat: extract nisps-core platform-agnostic ML library
Extract the interactive machine learning engine from MEMLNaut-NISPS
firmware into a standalone, platform-agnostic C++20 header-only library.
What is nisps-core?
-------------------
NISPS (Neural Interactive Shaping of Parameter Spaces) core is a
parameter mapping engine. It takes N input parameters (joystick,
sensors, audio features) and maps them to M output parameters through
an interactively-trained neural network.
Use it to control: synthesizers, effects, lights, robots, game
parameters, or anything that responds to continuous control data.
Key Features
------------
- Header-only: No compilation needed, just include and use
- Platform-agnostic: Pure C++20, works anywhere
- Zero dependencies: Only standard library
- Interactive learning: Train by demonstration
- Lightweight: ~3,500 lines of optimized neural network code
- Flexible: Map 1-100 inputs to 1-100 outputs
Architecture
------------
Core components:
- IML: High-level interactive ML interface
- MLP: Multi-layer perceptron (feedforward neural network)
- Dataset: Training data management with replay memory
- Layer/Node: Neural network building blocks
- Loss: MSE and categorical cross-entropy functions
- Utils: Activation functions (sigmoid, ReLU, tanh, etc.)
Transformations Applied
-----------------------
✅ Removed Arduino/RP2040 dependencies (Serial, SD, Pico SDK)
✅ Removed audio synthesis code (nisps-core is control-only)
✅ Added nisps namespace to all code
✅ Converted to header-only library with _impl.hpp pattern
✅ Updated to C++20 (required for std::span)
✅ Removed platform-specific serialization
✅ Replaced debug macros with no-op stubs
✅ Added comprehensive documentation and examples
Files Added
-----------
- nisps-core/README.md: Complete documentation and API reference
- nisps-core/CHANGELOG.md: Version history and migration guide
- nisps-core/include/nisps/*.hpp: 13 header files (~3,500 lines)
- nisps-core/test/main.cpp: XOR test demonstrating basic usage
- nisps-core/examples/simple_mapping.cpp: Interactive demo
- nisps-core/CMakeLists.txt: Build system for tests
Testing
-------
✅ Compiles with GCC 14.2 (C++20)
✅ All tests passing
✅ Successfully instantiates networks and runs inference
Performance
-----------
- Inference: 1-10 µs for small networks (2-10-10-4)
- Training: 10-100 ms for 100 examples, 1000 iterations
- Memory: ~1 KB per hidden neuron
Migration from Embedded IMLInterface
------------------------------------
Old (embedded):
IMLInterface iml(n_inputs, n_outputs);
New (nisps-core):
nisps::IML<float> iml(n_inputs, n_outputs);
All method names remain the same, just add the namespace.
Related
-------
- Implements: NISPS_CORE_EXTRACTION_PLAN.md
- Task graph: NISPS_CORE_TASKS.md
- Origin: MEMLNaut-NISPS firmware
- Docs: https://musicallyembodiedml.github.io/memlnaut/
Co-authored-by: Claude Code <claude@anthropic.com>
2026-02-08 17:47:23 +01:00
2026-04-29 18:57:27 +02:00
Project documentation: https://musicallyembodiedml.github.io/memlnaut/approaches/nisps
feat: extract nisps-core platform-agnostic ML library
Extract the interactive machine learning engine from MEMLNaut-NISPS
firmware into a standalone, platform-agnostic C++20 header-only library.
What is nisps-core?
-------------------
NISPS (Neural Interactive Shaping of Parameter Spaces) core is a
parameter mapping engine. It takes N input parameters (joystick,
sensors, audio features) and maps them to M output parameters through
an interactively-trained neural network.
Use it to control: synthesizers, effects, lights, robots, game
parameters, or anything that responds to continuous control data.
Key Features
------------
- Header-only: No compilation needed, just include and use
- Platform-agnostic: Pure C++20, works anywhere
- Zero dependencies: Only standard library
- Interactive learning: Train by demonstration
- Lightweight: ~3,500 lines of optimized neural network code
- Flexible: Map 1-100 inputs to 1-100 outputs
Architecture
------------
Core components:
- IML: High-level interactive ML interface
- MLP: Multi-layer perceptron (feedforward neural network)
- Dataset: Training data management with replay memory
- Layer/Node: Neural network building blocks
- Loss: MSE and categorical cross-entropy functions
- Utils: Activation functions (sigmoid, ReLU, tanh, etc.)
Transformations Applied
-----------------------
✅ Removed Arduino/RP2040 dependencies (Serial, SD, Pico SDK)
✅ Removed audio synthesis code (nisps-core is control-only)
✅ Added nisps namespace to all code
✅ Converted to header-only library with _impl.hpp pattern
✅ Updated to C++20 (required for std::span)
✅ Removed platform-specific serialization
✅ Replaced debug macros with no-op stubs
✅ Added comprehensive documentation and examples
Files Added
-----------
- nisps-core/README.md: Complete documentation and API reference
- nisps-core/CHANGELOG.md: Version history and migration guide
- nisps-core/include/nisps/*.hpp: 13 header files (~3,500 lines)
- nisps-core/test/main.cpp: XOR test demonstrating basic usage
- nisps-core/examples/simple_mapping.cpp: Interactive demo
- nisps-core/CMakeLists.txt: Build system for tests
Testing
-------
✅ Compiles with GCC 14.2 (C++20)
✅ All tests passing
✅ Successfully instantiates networks and runs inference
Performance
-----------
- Inference: 1-10 µs for small networks (2-10-10-4)
- Training: 10-100 ms for 100 examples, 1000 iterations
- Memory: ~1 KB per hidden neuron
Migration from Embedded IMLInterface
------------------------------------
Old (embedded):
IMLInterface iml(n_inputs, n_outputs);
New (nisps-core):
nisps::IML<float> iml(n_inputs, n_outputs);
All method names remain the same, just add the namespace.
Related
-------
- Implements: NISPS_CORE_EXTRACTION_PLAN.md
- Task graph: NISPS_CORE_TASKS.md
- Origin: MEMLNaut-NISPS firmware
- Docs: https://musicallyembodiedml.github.io/memlnaut/
Co-authored-by: Claude Code <claude@anthropic.com>
2026-02-08 17:47:23 +01:00
2026-04-29 18:57:27 +02:00
For the codebase index, see `MAP.md` . For strategic gaps and open mission questions, see `ALIGNMENT.md` .
feat: extract nisps-core platform-agnostic ML library
Extract the interactive machine learning engine from MEMLNaut-NISPS
firmware into a standalone, platform-agnostic C++20 header-only library.
What is nisps-core?
-------------------
NISPS (Neural Interactive Shaping of Parameter Spaces) core is a
parameter mapping engine. It takes N input parameters (joystick,
sensors, audio features) and maps them to M output parameters through
an interactively-trained neural network.
Use it to control: synthesizers, effects, lights, robots, game
parameters, or anything that responds to continuous control data.
Key Features
------------
- Header-only: No compilation needed, just include and use
- Platform-agnostic: Pure C++20, works anywhere
- Zero dependencies: Only standard library
- Interactive learning: Train by demonstration
- Lightweight: ~3,500 lines of optimized neural network code
- Flexible: Map 1-100 inputs to 1-100 outputs
Architecture
------------
Core components:
- IML: High-level interactive ML interface
- MLP: Multi-layer perceptron (feedforward neural network)
- Dataset: Training data management with replay memory
- Layer/Node: Neural network building blocks
- Loss: MSE and categorical cross-entropy functions
- Utils: Activation functions (sigmoid, ReLU, tanh, etc.)
Transformations Applied
-----------------------
✅ Removed Arduino/RP2040 dependencies (Serial, SD, Pico SDK)
✅ Removed audio synthesis code (nisps-core is control-only)
✅ Added nisps namespace to all code
✅ Converted to header-only library with _impl.hpp pattern
✅ Updated to C++20 (required for std::span)
✅ Removed platform-specific serialization
✅ Replaced debug macros with no-op stubs
✅ Added comprehensive documentation and examples
Files Added
-----------
- nisps-core/README.md: Complete documentation and API reference
- nisps-core/CHANGELOG.md: Version history and migration guide
- nisps-core/include/nisps/*.hpp: 13 header files (~3,500 lines)
- nisps-core/test/main.cpp: XOR test demonstrating basic usage
- nisps-core/examples/simple_mapping.cpp: Interactive demo
- nisps-core/CMakeLists.txt: Build system for tests
Testing
-------
✅ Compiles with GCC 14.2 (C++20)
✅ All tests passing
✅ Successfully instantiates networks and runs inference
Performance
-----------
- Inference: 1-10 µs for small networks (2-10-10-4)
- Training: 10-100 ms for 100 examples, 1000 iterations
- Memory: ~1 KB per hidden neuron
Migration from Embedded IMLInterface
------------------------------------
Old (embedded):
IMLInterface iml(n_inputs, n_outputs);
New (nisps-core):
nisps::IML<float> iml(n_inputs, n_outputs);
All method names remain the same, just add the namespace.
Related
-------
- Implements: NISPS_CORE_EXTRACTION_PLAN.md
- Task graph: NISPS_CORE_TASKS.md
- Origin: MEMLNaut-NISPS firmware
- Docs: https://musicallyembodiedml.github.io/memlnaut/
Co-authored-by: Claude Code <claude@anthropic.com>
2026-02-08 17:47:23 +01:00
2026-06-28 22:28:44 +02:00
**For anything UI-related in the Manifold front-end (`manifold/`), read `manifold/ONBOARDING.md` first** — it's a single-file agent orientation (run/build/deploy/test, the UI/engine-spine/WASM layering, the convertible Stages, the Dock + drawers, and the non-obvious gotchas).
2026-04-29 18:57:27 +02:00
## The `nisps/` core
feat: extract nisps-core platform-agnostic ML library
Extract the interactive machine learning engine from MEMLNaut-NISPS
firmware into a standalone, platform-agnostic C++20 header-only library.
What is nisps-core?
-------------------
NISPS (Neural Interactive Shaping of Parameter Spaces) core is a
parameter mapping engine. It takes N input parameters (joystick,
sensors, audio features) and maps them to M output parameters through
an interactively-trained neural network.
Use it to control: synthesizers, effects, lights, robots, game
parameters, or anything that responds to continuous control data.
Key Features
------------
- Header-only: No compilation needed, just include and use
- Platform-agnostic: Pure C++20, works anywhere
- Zero dependencies: Only standard library
- Interactive learning: Train by demonstration
- Lightweight: ~3,500 lines of optimized neural network code
- Flexible: Map 1-100 inputs to 1-100 outputs
Architecture
------------
Core components:
- IML: High-level interactive ML interface
- MLP: Multi-layer perceptron (feedforward neural network)
- Dataset: Training data management with replay memory
- Layer/Node: Neural network building blocks
- Loss: MSE and categorical cross-entropy functions
- Utils: Activation functions (sigmoid, ReLU, tanh, etc.)
Transformations Applied
-----------------------
✅ Removed Arduino/RP2040 dependencies (Serial, SD, Pico SDK)
✅ Removed audio synthesis code (nisps-core is control-only)
✅ Added nisps namespace to all code
✅ Converted to header-only library with _impl.hpp pattern
✅ Updated to C++20 (required for std::span)
✅ Removed platform-specific serialization
✅ Replaced debug macros with no-op stubs
✅ Added comprehensive documentation and examples
Files Added
-----------
- nisps-core/README.md: Complete documentation and API reference
- nisps-core/CHANGELOG.md: Version history and migration guide
- nisps-core/include/nisps/*.hpp: 13 header files (~3,500 lines)
- nisps-core/test/main.cpp: XOR test demonstrating basic usage
- nisps-core/examples/simple_mapping.cpp: Interactive demo
- nisps-core/CMakeLists.txt: Build system for tests
Testing
-------
✅ Compiles with GCC 14.2 (C++20)
✅ All tests passing
✅ Successfully instantiates networks and runs inference
Performance
-----------
- Inference: 1-10 µs for small networks (2-10-10-4)
- Training: 10-100 ms for 100 examples, 1000 iterations
- Memory: ~1 KB per hidden neuron
Migration from Embedded IMLInterface
------------------------------------
Old (embedded):
IMLInterface iml(n_inputs, n_outputs);
New (nisps-core):
nisps::IML<float> iml(n_inputs, n_outputs);
All method names remain the same, just add the namespace.
Related
-------
- Implements: NISPS_CORE_EXTRACTION_PLAN.md
- Task graph: NISPS_CORE_TASKS.md
- Origin: MEMLNaut-NISPS firmware
- Docs: https://musicallyembodiedml.github.io/memlnaut/
Co-authored-by: Claude Code <claude@anthropic.com>
2026-02-08 17:47:23 +01:00
2026-04-29 18:57:27 +02:00
```
nisps/
├── core/ types, perf attrs, concepts (AudioEngine, MLEngine, Mode), fixed/ring buffers, deterministic RNG, math
├── ml/ MLP class template (4-layer, 3 hidden); SGD, gradient clipping, spread-aware Xavier init,
│ RL move_weights with output pin mask + per-layer scaling + weight decay
├── dsp/ biquad, delay, reverb, filter, env, osc, pitch_shift, dc_blocker
├── engines/ 8 audio engines (paf_synth, channel_strip, xiasri, verb_fx, memlcelium, breakor,
│ elysiamorf, analysis) + NoOpEngine. Each satisfies the AudioEngine concept.
├── modes/ 8 platform-agnostic modes binding {ML, engine, voice space, abstract I/O channels}.
│ CRTP base eliminates the duplication that plagued firmware modes.
└── wasm/ Emscripten C API bindings (compiled only for WASM target)
```
feat: extract nisps-core platform-agnostic ML library
Extract the interactive machine learning engine from MEMLNaut-NISPS
firmware into a standalone, platform-agnostic C++20 header-only library.
What is nisps-core?
-------------------
NISPS (Neural Interactive Shaping of Parameter Spaces) core is a
parameter mapping engine. It takes N input parameters (joystick,
sensors, audio features) and maps them to M output parameters through
an interactively-trained neural network.
Use it to control: synthesizers, effects, lights, robots, game
parameters, or anything that responds to continuous control data.
Key Features
------------
- Header-only: No compilation needed, just include and use
- Platform-agnostic: Pure C++20, works anywhere
- Zero dependencies: Only standard library
- Interactive learning: Train by demonstration
- Lightweight: ~3,500 lines of optimized neural network code
- Flexible: Map 1-100 inputs to 1-100 outputs
Architecture
------------
Core components:
- IML: High-level interactive ML interface
- MLP: Multi-layer perceptron (feedforward neural network)
- Dataset: Training data management with replay memory
- Layer/Node: Neural network building blocks
- Loss: MSE and categorical cross-entropy functions
- Utils: Activation functions (sigmoid, ReLU, tanh, etc.)
Transformations Applied
-----------------------
✅ Removed Arduino/RP2040 dependencies (Serial, SD, Pico SDK)
✅ Removed audio synthesis code (nisps-core is control-only)
✅ Added nisps namespace to all code
✅ Converted to header-only library with _impl.hpp pattern
✅ Updated to C++20 (required for std::span)
✅ Removed platform-specific serialization
✅ Replaced debug macros with no-op stubs
✅ Added comprehensive documentation and examples
Files Added
-----------
- nisps-core/README.md: Complete documentation and API reference
- nisps-core/CHANGELOG.md: Version history and migration guide
- nisps-core/include/nisps/*.hpp: 13 header files (~3,500 lines)
- nisps-core/test/main.cpp: XOR test demonstrating basic usage
- nisps-core/examples/simple_mapping.cpp: Interactive demo
- nisps-core/CMakeLists.txt: Build system for tests
Testing
-------
✅ Compiles with GCC 14.2 (C++20)
✅ All tests passing
✅ Successfully instantiates networks and runs inference
Performance
-----------
- Inference: 1-10 µs for small networks (2-10-10-4)
- Training: 10-100 ms for 100 examples, 1000 iterations
- Memory: ~1 KB per hidden neuron
Migration from Embedded IMLInterface
------------------------------------
Old (embedded):
IMLInterface iml(n_inputs, n_outputs);
New (nisps-core):
nisps::IML<float> iml(n_inputs, n_outputs);
All method names remain the same, just add the namespace.
Related
-------
- Implements: NISPS_CORE_EXTRACTION_PLAN.md
- Task graph: NISPS_CORE_TASKS.md
- Origin: MEMLNaut-NISPS firmware
- Docs: https://musicallyembodiedml.github.io/memlnaut/
Co-authored-by: Claude Code <claude@anthropic.com>
2026-02-08 17:47:23 +01:00
2026-04-29 18:57:27 +02:00
Build: `cmake -S nisps -B nisps/build -G Ninja && cmake --build nisps/build && ctest --test-dir nisps/build` .
2026-02-11 13:17:17 +01:00
2026-04-29 18:57:27 +02:00
Tests: 4 executables (`nisps_core_tests`, `nisps_dsp_engine_tests` , `nisps_modes_tests` , `nisps_golden_tests` ). Run all: `bash scripts/build-cpp-tests.sh` . Parity vs WASM: `bash scripts/parity-check.sh` (asserts native and WASM produce identical outputs within 1e-5).
2026-02-11 13:17:17 +01:00
2026-04-29 18:57:27 +02:00
### Performance contract (RP2350)
2026-02-11 13:17:17 +01:00
2026-04-29 18:57:27 +02:00
These rules apply to **all** code under `nisps/` . They are inert in WASM but kept globally for consistency.
2026-04-03 18:40:12 +02:00
2026-04-29 18:57:27 +02:00
- **No heap.** No `new` , `malloc` , `std::vector` in hot paths. Use `nisps::FixedBuffer<T, N>` or `std::array<T, N>` .
- **Constants discipline.** Float literals >255 used in hot paths must be `static const float val = X.f;` not inline.
- **`.f` suffix on all float literals.** No double promotion in audio/inference paths.
- **Memory section attributes.** Apply `NISPS_AUDIO_MEM` / `NISPS_AUDIO_FUNC` / `NISPS_APP_SRAM` / `NISPS_HOT` / `NISPS_FORCE_INLINE` (from `nisps/core/perf.hpp` ).
- **No virtual dispatch in audio path.** `AudioEngine` and `Mode` are C++20 concepts, not interfaces.
- **Deterministic RNG.** All RNG state is per-instance; constructors take a seed; cross-platform parity tests rely on this.
2026-04-03 18:40:12 +02:00
2026-04-29 18:57:27 +02:00
Lint: `bash scripts/lint-cpp.sh` warns on missing `.f` and fails on heap/`Arduino.h` use under `nisps/` .
2026-04-03 18:40:12 +02:00
2026-04-29 18:57:27 +02:00
## The `firmware/` target
2026-04-03 18:40:12 +02:00
```
2026-04-29 18:57:27 +02:00
firmware/MEMLNaut-NISPS/
├── MEMLNaut-NISPS.ino # Entry point; mode selected via #define MEMLNAUT_MODE_TYPE
├── glue/
│ ├── audio_driver.hpp # memllib AudioDriver block callback → Mode::process per-sample
│ ├── peripherals.hpp # joystick / pots / buttons → Mode::set_input + ML primitives
│ ├── midi_io.hpp # MIDI in → mode handlers; drain ControlEvent ring → MIDI UART
│ ├── mode_select.hpp # type aliases firmware mode name → nisps::modes::*Mode
│ ├── input_router.hpp # wire_inputs() entry point
2026-06-28 21:02:23 +02:00
│ ├── output_router.hpp # drain_outputs() entry point
│ └── settings_view.hpp # wire_settings(): TFT/rotary menu (Joystick Dual/Single for 4-in modes)
2026-04-29 18:57:27 +02:00
└── src/{memllib,daisysp,nisps} # symlinks (Arduino-CLI sketch tree convention)
2026-04-03 18:40:12 +02:00
```
2026-04-29 18:57:27 +02:00
Build: `scripts/build-firmware.sh [VARIANT]` . Verified compiling for PAFSynth, ChannelStrip, BreakOr on `rp2040:rp2040:solderparty_rp2350_stamp_xl:opt=Optimize3` with `-std=gnu++20` . Flash: `scripts/flash-firmware.sh` . One-shot: `scripts/build-and-flash-firmware.sh` .
2026-04-03 18:40:12 +02:00
2026-04-29 18:57:27 +02:00
### Dual-core orchestration (firmware)
2026-04-03 18:40:12 +02:00
2026-04-29 18:57:27 +02:00
- **Core 0**: UI loop, ML inference (`Mode::tick_control`), peripheral polling (5ms period).
- **Core 1**: Real-time audio processing (`Mode::process`), MIDI polling.
- **Sync**: `nisps::core::ring_buffer` (templated SPSC lock-free, replaces pico/util/queue) + memory barriers (`nisps::core::memory_barrier`, `write_volatile` /`read_volatile`).
2026-04-03 18:40:12 +02:00
2026-04-29 18:57:27 +02:00
## The `playground/` target
2026-03-21 22:11:45 +01:00
2026-04-29 18:57:27 +02:00
```
playground/ # Vite + SolidJS + TypeScript
├── src/
│ ├── primitives/ # 16 UI building blocks (Slider, JoyMap, Heatmap, …) + .demo.tsx for /dev/primitives
│ ├── modes/ # one TSX per firmware mode + C15Mode (browser-only); ModeShell + ModeSwitcher + mode-runtime
│ ├── stores/ # Solid stores (ml, input, output, mode, control, session, exploration, bus + persistence)
│ ├── audio/ # engine-host + AudioWorklet processor (loads nisps.wasm separately on each thread)
│ ├── ml/ # WasmIML class + disposable async-training Worker + dataset
│ ├── input/, output/ # pure-fn pipelines (deadzone→zoom→curve→smoothing→momentum, then global curve→smoothing→slew→freeze)
│ ├── features/ # heatmap, snapshots, A/B compare, region/param pin, trail, weight health, gradient flow
│ └── debug/probe.ts # synchronous window.__nisps for Playwright
├── public/ # nisps.{wasm,js}, c15.{wasm,glue}
└── tests/e2e/ # Playwright specs + helpers
```
2026-03-22 01:10:26 +01:00
2026-04-29 18:57:27 +02:00
Dev: `cd playground && bun install && bun run dev` . Build: `bun run build` . Typecheck: `bun run typecheck` . E2E: `bunx playwright test` .
2026-03-22 01:10:26 +01:00
2026-04-29 18:57:27 +02:00
### Stores + reactivity
2026-03-22 01:10:26 +01:00
2026-04-29 18:57:27 +02:00
All stores use SolidJS `createStore` for objects, `createSignal` for primitives. ML outputs are stored in a separate Float32Array signal (per the migration plan's perf guidance). Persistence (debounced 200ms localStorage round-trip) wired in `playground/src/stores/persistence.ts` . The signal bus (`bus.ts`) handles cross-store events (`ml.*`, `mode.*` , `pin.*` , `ui.*` ).
2026-03-22 01:10:26 +01:00
2026-04-29 18:57:27 +02:00
### Control surface
2026-03-22 01:10:26 +01:00
2026-04-29 18:57:27 +02:00
Three compound axes (Boldness / Memory / Precision) interpolate per-axis tables to drive ~6 underlying parameters each, with offset overrides ("trim-pot" model). State is in `control-store` . Six built-in control presets (Default, First Touch, Jazz Hands, Sculptor, Improviser, Microscope) available via the ModeShell control bar.
2026-03-22 01:10:26 +01:00
2026-04-29 18:57:27 +02:00
### Debug probe (Playwright)
2026-03-21 22:11:45 +01:00
2026-04-29 18:57:27 +02:00
`window.__nisps` is exposed synchronously and bypasses Solid reactivity (uses `untrack` /`batch`). API matches the `.local/recon/04-playground.md` spec — `setInputs` , `getOutputs` , `getLoss` , `train` , `thumbsUp` /`thumbsDown`, `randomise` , `clearExamples` , `inferBatch` , `getLayerStats` , `saveState` , etc.
2026-03-21 22:11:45 +01:00
2026-04-29 18:57:27 +02:00
## The `schemas/` + `codegen/` contract
2026-02-11 13:17:17 +01:00
2026-04-29 18:57:27 +02:00
Each mode has a `schemas/modes/<mode>.json` describing its parameters (name, label, range, default, curve, group), ML config (input/output sizes, hidden layers), voice spaces (names — bodies are inline lambdas in the C++ engine), and UI config. The meta-schema at `schemas/schema.json` validates these.
2026-03-22 23:50:56 +01:00
2026-04-29 18:57:27 +02:00
Codegen (`bun run codegen/generate.ts`) emits:
- `nisps/modes/generated/<mode>_schema.hpp` — `constexpr` C++ data, namespace `nisps::modes::generated` , re-exports `nisps::Curve` from `nisps/core/math.hpp` .
- `playground/src/modes/generated/<mode>_schema.ts` — typed const objects + per-mode params interface.
2026-03-22 23:50:56 +01:00
2026-04-29 18:57:27 +02:00
Codegen is idempotent. Golden test ensures regenerating produces byte-identical output.
2026-03-22 23:50:56 +01:00
2026-04-29 18:57:27 +02:00
## WASM bridge
2026-03-22 23:50:56 +01:00
2026-04-29 18:57:27 +02:00
Two WASM instances at runtime:
2026-03-22 23:50:56 +01:00
2026-04-29 18:57:27 +02:00
1. **Main thread** (`playground/src/ml/wasm-iml.ts`): ML inference + sync training + RL primitives. Update store after each call. Async training via disposable Web Worker (`wasm-worker.ts`).
2. **AudioWorklet** (`playground/src/audio/worklet/nisps-processor.ts`): runs engine `process_block` per audio block. Loads `nisps.wasm` directly via `WebAssembly.compile` (no Emscripten glue in worklet). Bytes posted from main thread.
feat(playground): implement Phase 1 control surface — compound axes, input pipeline, enhanced joy-map
Three new standalone ES modules + integration into a-app.js:
- input-pipeline.js: 5-stage processing (deadzone → zoom → curve → smoothing →
momentum-as-zoom), 3 anchor modes, zoom-at-zero freeze, per-axis overrides
- control-surface.js: compound axes (Boldness/Memory/Precision) with interpolation
tables, offset-based override resolution (trim-pot model), 6 built-in presets
- control-surface-ui.js: floating bar axis sliders, gear icon settings drawer with
Input/Training/Exploration/Output sections, log-scale sliders, override dots
- joy-map-enhanced.js: zoom minimap with adaptive grid (4×4→32×32), vanishing trail
with Catmull-Rom spline + tap-to-return, dual concentric noise rings, frozen overlay
Integration fixes from fresh-eyes review:
- getCurrentInputs()/setCurrentInputs() use cached pipeline coords (not raw)
- CSS noise ring hidden when canvas version active (no doubling)
- Input mode switch re-runs through pipeline
- Control surface state persisted to localStorage
Implements full Phase 1 of SPEC-controls.md plus bonus items from later phases
(zoom-aware feedback, control presets with override resolution, input curve/deadzone/
smoothing/momentum all wired).
2026-03-26 09:24:24 +01:00
2026-04-29 18:57:27 +02:00
C API is in `nisps/wasm/bindings.cpp` . Build: `bash scripts/build-wasm.sh` (~94KB output to `playground/public/` ).
feat(playground): implement Phase 1 control surface — compound axes, input pipeline, enhanced joy-map
Three new standalone ES modules + integration into a-app.js:
- input-pipeline.js: 5-stage processing (deadzone → zoom → curve → smoothing →
momentum-as-zoom), 3 anchor modes, zoom-at-zero freeze, per-axis overrides
- control-surface.js: compound axes (Boldness/Memory/Precision) with interpolation
tables, offset-based override resolution (trim-pot model), 6 built-in presets
- control-surface-ui.js: floating bar axis sliders, gear icon settings drawer with
Input/Training/Exploration/Output sections, log-scale sliders, override dots
- joy-map-enhanced.js: zoom minimap with adaptive grid (4×4→32×32), vanishing trail
with Catmull-Rom spline + tap-to-return, dual concentric noise rings, frozen overlay
Integration fixes from fresh-eyes review:
- getCurrentInputs()/setCurrentInputs() use cached pipeline coords (not raw)
- CSS noise ring hidden when canvas version active (no doubling)
- Input mode switch re-runs through pipeline
- Control surface state persisted to localStorage
Implements full Phase 1 of SPEC-controls.md plus bonus items from later phases
(zoom-aware feedback, control presets with override resolution, input curve/deadzone/
smoothing/momentum all wired).
2026-03-26 09:24:24 +01:00
2026-04-29 18:57:27 +02:00
The WASM target is fixed at `MLP<2, 10, 14, 18, 126>` . Modes with smaller `output_size` use the first N outputs only.
feat(playground): implement Phases 2-4 of control surface spec
Phase 2 — Pinning + History:
- snapshot-stack.js: ring buffer (20 max) with auto-snapshot on
train/randomize/thumbs-down, multi-level undo, tagged entries
- ab-compare.js: A/B weight state comparison with capture/toggle/accept/revert
- region-pin.js: pin rectangular input-space regions (Approach A: example
pinning), pinned examples always included in training
- param-pin.js: per-output pin flags, pin mask skips pinned nodes in moveWeights
- phase2-ui.js: undo button with history popup, A/B toggle, long-press region
pin, double-tap param pin
- Modified mlp.js/iml.js/nisps-wasm.js to accept outputPinMask in moveWeights
Phase 3 — Input Refinement + Exploration:
- pressure-feedback.js: touch force + hold duration → intensity multiplier
- auto-explore.js: automated thumbs-down at configurable interval, zoom-scaled
- input-heatmap.js: 16×16 MLP sampling, 3 color modes (luminance/variance/
divergence), zoom-aware resampling, offscreen canvas rendering
- phase3-ui.js: auto-explore toggle with progress ring, heatmap eye icon,
pressure indicators, settings drawer section
- joy-map-enhanced.js: added setHeatmap() for background layer rendering
Phase 4 — Output Pipeline + Visualization + Polish:
- output-pipeline.js: global curve → smoothing → slew rate → freeze gate
- weight-health.js: weight magnitude histogram, dead/saturating/healthy status
- gradient-flow.js: per-layer weight-delta analysis, vanishing/exploding detection
- session-presets.js: save/load full state, URL sharing via compact params
- phase4-ui.js: freeze button, network health panel, session preset UI
All phases merged into a-app.js with proper integration: auto-snapshots,
pressure-modulated RL, heatmap triggers, output pipeline in routeOutputs,
gradient capture around training, persistence for all new state.
2026-03-26 09:48:12 +01:00
2026-04-29 18:57:27 +02:00
### Known limitations
feat(playground): implement Phase 1 control surface — compound axes, input pipeline, enhanced joy-map
Three new standalone ES modules + integration into a-app.js:
- input-pipeline.js: 5-stage processing (deadzone → zoom → curve → smoothing →
momentum-as-zoom), 3 anchor modes, zoom-at-zero freeze, per-axis overrides
- control-surface.js: compound axes (Boldness/Memory/Precision) with interpolation
tables, offset-based override resolution (trim-pot model), 6 built-in presets
- control-surface-ui.js: floating bar axis sliders, gear icon settings drawer with
Input/Training/Exploration/Output sections, log-scale sliders, override dots
- joy-map-enhanced.js: zoom minimap with adaptive grid (4×4→32×32), vanishing trail
with Catmull-Rom spline + tap-to-return, dual concentric noise rings, frozen overlay
Integration fixes from fresh-eyes review:
- getCurrentInputs()/setCurrentInputs() use cached pipeline coords (not raw)
- CSS noise ring hidden when canvas version active (no doubling)
- Input mode switch re-runs through pipeline
- Control surface state persisted to localStorage
Implements full Phase 1 of SPEC-controls.md plus bonus items from later phases
(zoom-aware feedback, control presets with override resolution, input curve/deadzone/
smoothing/momentum all wired).
2026-03-26 09:24:24 +01:00
2026-04-29 18:57:27 +02:00
- Loss history not yet plumbed through C API; `lossHistory` in the store is a single-element array per training run.
- Engine MLP architecture is fixed at compile time — supporting per-mode hidden-layer shapes would need either multiple WASM modules or runtime variation.
- Mic input through the worklet for XIASRI / SoundAnalysisMIDI is not wired; UI scaffolds render but feature is TODO.
- C15 voice space integration in C15Mode is a placeholder.
feat(playground): implement Phase 1 control surface — compound axes, input pipeline, enhanced joy-map
Three new standalone ES modules + integration into a-app.js:
- input-pipeline.js: 5-stage processing (deadzone → zoom → curve → smoothing →
momentum-as-zoom), 3 anchor modes, zoom-at-zero freeze, per-axis overrides
- control-surface.js: compound axes (Boldness/Memory/Precision) with interpolation
tables, offset-based override resolution (trim-pot model), 6 built-in presets
- control-surface-ui.js: floating bar axis sliders, gear icon settings drawer with
Input/Training/Exploration/Output sections, log-scale sliders, override dots
- joy-map-enhanced.js: zoom minimap with adaptive grid (4×4→32×32), vanishing trail
with Catmull-Rom spline + tap-to-return, dual concentric noise rings, frozen overlay
Integration fixes from fresh-eyes review:
- getCurrentInputs()/setCurrentInputs() use cached pipeline coords (not raw)
- CSS noise ring hidden when canvas version active (no doubling)
- Input mode switch re-runs through pipeline
- Control surface state persisted to localStorage
Implements full Phase 1 of SPEC-controls.md plus bonus items from later phases
(zoom-aware feedback, control presets with override resolution, input curve/deadzone/
smoothing/momentum all wired).
2026-03-26 09:24:24 +01:00
2026-04-29 18:57:27 +02:00
## URL parameters (playground)
feat(playground): implement Phase 1 control surface — compound axes, input pipeline, enhanced joy-map
Three new standalone ES modules + integration into a-app.js:
- input-pipeline.js: 5-stage processing (deadzone → zoom → curve → smoothing →
momentum-as-zoom), 3 anchor modes, zoom-at-zero freeze, per-axis overrides
- control-surface.js: compound axes (Boldness/Memory/Precision) with interpolation
tables, offset-based override resolution (trim-pot model), 6 built-in presets
- control-surface-ui.js: floating bar axis sliders, gear icon settings drawer with
Input/Training/Exploration/Output sections, log-scale sliders, override dots
- joy-map-enhanced.js: zoom minimap with adaptive grid (4×4→32×32), vanishing trail
with Catmull-Rom spline + tap-to-return, dual concentric noise rings, frozen overlay
Integration fixes from fresh-eyes review:
- getCurrentInputs()/setCurrentInputs() use cached pipeline coords (not raw)
- CSS noise ring hidden when canvas version active (no doubling)
- Input mode switch re-runs through pipeline
- Control surface state persisted to localStorage
Implements full Phase 1 of SPEC-controls.md plus bonus items from later phases
(zoom-aware feedback, control presets with override resolution, input curve/deadzone/
smoothing/momentum all wired).
2026-03-26 09:24:24 +01:00
2026-04-29 18:57:27 +02:00
| Param | Range | Default | Effect |
|-------|-------|---------|--------|
| `tame` | 0–
| `spread` | 0–
| `preset` | preset id | _(none)_ | Auto-loads a synth preset on first visit. |
| `debug` | 1 | _(off)_ | Exposes `window.__nisps` debug probe. |
feat(playground): implement Phase 1 control surface — compound axes, input pipeline, enhanced joy-map
Three new standalone ES modules + integration into a-app.js:
- input-pipeline.js: 5-stage processing (deadzone → zoom → curve → smoothing →
momentum-as-zoom), 3 anchor modes, zoom-at-zero freeze, per-axis overrides
- control-surface.js: compound axes (Boldness/Memory/Precision) with interpolation
tables, offset-based override resolution (trim-pot model), 6 built-in presets
- control-surface-ui.js: floating bar axis sliders, gear icon settings drawer with
Input/Training/Exploration/Output sections, log-scale sliders, override dots
- joy-map-enhanced.js: zoom minimap with adaptive grid (4×4→32×32), vanishing trail
with Catmull-Rom spline + tap-to-return, dual concentric noise rings, frozen overlay
Integration fixes from fresh-eyes review:
- getCurrentInputs()/setCurrentInputs() use cached pipeline coords (not raw)
- CSS noise ring hidden when canvas version active (no doubling)
- Input mode switch re-runs through pipeline
- Control surface state persisted to localStorage
Implements full Phase 1 of SPEC-controls.md plus bonus items from later phases
(zoom-aware feedback, control presets with override resolution, input curve/deadzone/
smoothing/momentum all wired).
2026-03-26 09:24:24 +01:00
2026-04-29 18:57:27 +02:00
### `spread` — sigmoid saturation control
feat(playground): implement Phase 1 control surface — compound axes, input pipeline, enhanced joy-map
Three new standalone ES modules + integration into a-app.js:
- input-pipeline.js: 5-stage processing (deadzone → zoom → curve → smoothing →
momentum-as-zoom), 3 anchor modes, zoom-at-zero freeze, per-axis overrides
- control-surface.js: compound axes (Boldness/Memory/Precision) with interpolation
tables, offset-based override resolution (trim-pot model), 6 built-in presets
- control-surface-ui.js: floating bar axis sliders, gear icon settings drawer with
Input/Training/Exploration/Output sections, log-scale sliders, override dots
- joy-map-enhanced.js: zoom minimap with adaptive grid (4×4→32×32), vanishing trail
with Catmull-Rom spline + tap-to-return, dual concentric noise rings, frozen overlay
Integration fixes from fresh-eyes review:
- getCurrentInputs()/setCurrentInputs() use cached pipeline coords (not raw)
- CSS noise ring hidden when canvas version active (no doubling)
- Input mode switch re-runs through pipeline
- Control surface state persisted to localStorage
Implements full Phase 1 of SPEC-controls.md plus bonus items from later phases
(zoom-aware feedback, control presets with override resolution, input curve/deadzone/
smoothing/momentum all wired).
2026-03-26 09:24:24 +01:00
2026-04-29 18:57:27 +02:00
The MLP uses ReLU hidden layers with a sigmoid output. With uniform [-1,1] weights, the sum of many weighted inputs at each layer drives sigmoid pre-activations far from zero (std dev ≈ √fan_in), causing outputs to saturate. The `spread` parameter addresses this:
feat(playground): implement Phase 1 control surface — compound axes, input pipeline, enhanced joy-map
Three new standalone ES modules + integration into a-app.js:
- input-pipeline.js: 5-stage processing (deadzone → zoom → curve → smoothing →
momentum-as-zoom), 3 anchor modes, zoom-at-zero freeze, per-axis overrides
- control-surface.js: compound axes (Boldness/Memory/Precision) with interpolation
tables, offset-based override resolution (trim-pot model), 6 built-in presets
- control-surface-ui.js: floating bar axis sliders, gear icon settings drawer with
Input/Training/Exploration/Output sections, log-scale sliders, override dots
- joy-map-enhanced.js: zoom minimap with adaptive grid (4×4→32×32), vanishing trail
with Catmull-Rom spline + tap-to-return, dual concentric noise rings, frozen overlay
Integration fixes from fresh-eyes review:
- getCurrentInputs()/setCurrentInputs() use cached pipeline coords (not raw)
- CSS noise ring hidden when canvas version active (no doubling)
- Input mode switch re-runs through pipeline
- Control surface state persisted to localStorage
Implements full Phase 1 of SPEC-controls.md plus bonus items from later phases
(zoom-aware feedback, control presets with override resolution, input curve/deadzone/
smoothing/momentum all wired).
2026-03-26 09:24:24 +01:00
2026-04-29 18:57:27 +02:00
- `spread=0` (polarised): uniform [-1,1] weights, RL noise cap 0.3, no decay. Outputs cluster at extremes — good for radical exploration.
- `spread=1` (centered): Xavier-scaled weights, RL noise cap 0.05, 10% weight decay per move. Outputs spread across [0,1] — better for fine-grained shaping.
- Intermediate values interpolate.
feat(playground): implement Phase 1 control surface — compound axes, input pipeline, enhanced joy-map
Three new standalone ES modules + integration into a-app.js:
- input-pipeline.js: 5-stage processing (deadzone → zoom → curve → smoothing →
momentum-as-zoom), 3 anchor modes, zoom-at-zero freeze, per-axis overrides
- control-surface.js: compound axes (Boldness/Memory/Precision) with interpolation
tables, offset-based override resolution (trim-pot model), 6 built-in presets
- control-surface-ui.js: floating bar axis sliders, gear icon settings drawer with
Input/Training/Exploration/Output sections, log-scale sliders, override dots
- joy-map-enhanced.js: zoom minimap with adaptive grid (4×4→32×32), vanishing trail
with Catmull-Rom spline + tap-to-return, dual concentric noise rings, frozen overlay
Integration fixes from fresh-eyes review:
- getCurrentInputs()/setCurrentInputs() use cached pipeline coords (not raw)
- CSS noise ring hidden when canvas version active (no doubling)
- Input mode switch re-runs through pipeline
- Control surface state persisted to localStorage
Implements full Phase 1 of SPEC-controls.md plus bonus items from later phases
(zoom-aware feedback, control presets with override resolution, input curve/deadzone/
smoothing/momentum all wired).
2026-03-26 09:24:24 +01:00
2026-04-29 18:57:27 +02:00
## Verification chokepoints (user-confirmed)
feat(playground): implement Phase 1 control surface — compound axes, input pipeline, enhanced joy-map
Three new standalone ES modules + integration into a-app.js:
- input-pipeline.js: 5-stage processing (deadzone → zoom → curve → smoothing →
momentum-as-zoom), 3 anchor modes, zoom-at-zero freeze, per-axis overrides
- control-surface.js: compound axes (Boldness/Memory/Precision) with interpolation
tables, offset-based override resolution (trim-pot model), 6 built-in presets
- control-surface-ui.js: floating bar axis sliders, gear icon settings drawer with
Input/Training/Exploration/Output sections, log-scale sliders, override dots
- joy-map-enhanced.js: zoom minimap with adaptive grid (4×4→32×32), vanishing trail
with Catmull-Rom spline + tap-to-return, dual concentric noise rings, frozen overlay
Integration fixes from fresh-eyes review:
- getCurrentInputs()/setCurrentInputs() use cached pipeline coords (not raw)
- CSS noise ring hidden when canvas version active (no doubling)
- Input mode switch re-runs through pipeline
- Control surface state persisted to localStorage
Implements full Phase 1 of SPEC-controls.md plus bonus items from later phases
(zoom-aware feedback, control presets with override resolution, input curve/deadzone/
smoothing/momentum all wired).
2026-03-26 09:24:24 +01:00
2026-04-29 18:57:27 +02:00
- **A. Hardware**: each firmware mode flashes and produces correct audio on RP2350.
- **B. RP2350 perf**: no regression vs current main.
- **C. Browser parity**: each firmware mode runs in browser via WASM, sounds equivalent.
- **D. a-immersive feature parity**: control surface, snapshots, A/B compare, region/param pins, heatmap, weight health, gradient flow, output pipeline, session presets.
- **E. CI green**: `bash scripts/run-all-tests.sh` (cmake build + ctest + WASM build + parity + lint + Playwright).
2026-04-03 18:40:12 +02:00
2026-04-29 18:57:27 +02:00
## Build system summary
2026-04-03 18:40:12 +02:00
```bash
2026-04-29 18:57:27 +02:00
# Initialize submodules (required for memllib + daisysp)
git submodule update --init --recursive
2026-04-03 18:40:12 +02:00
2026-04-29 18:57:27 +02:00
# Codegen (run after editing any schemas/modes/*.json)
cd codegen & & bun install & & bun run generate.ts
2026-02-08 15:49:16 +01:00
2026-04-29 18:57:27 +02:00
# C++ host tests
bash scripts/build-cpp-tests.sh
2026-02-08 15:49:16 +01:00
2026-04-29 18:57:27 +02:00
# WASM
bash scripts/build-wasm.sh
2026-02-08 15:49:16 +01:00
2026-04-29 18:57:27 +02:00
# Cross-platform parity
bash scripts/parity-check.sh
2026-04-15 17:32:32 +02:00
2026-04-29 18:57:27 +02:00
# Lint
bash scripts/lint-cpp.sh
2026-04-15 17:37:58 +02:00
2026-04-29 18:57:27 +02:00
# Firmware
scripts/build-firmware.sh PAFSynth # or any other variant
2026-04-15 17:32:32 +02:00
scripts/flash-firmware.sh
scripts/build-and-flash-firmware.sh
2026-02-08 15:49:16 +01:00
2026-04-29 18:57:27 +02:00
# Playground
cd playground & & bun install
bun run dev # Vite dev (COOP/COEP enabled)
bun run typecheck
bun run build
bunx playwright test
2026-02-08 15:49:16 +01:00
2026-04-29 18:57:27 +02:00
# All tests
bash scripts/run-all-tests.sh
```
2026-02-08 15:49:16 +01:00
2026-04-29 18:57:27 +02:00
## Issue tracking
2026-02-08 15:49:16 +01:00
2026-06-25 05:04:39 +02:00
Coding-work tasks go in **ergo** (`ergo ready | show | claim | done | block`), over the Holon EAV core — see the `ergo` skill. **bd (beads) is RETIRED** (migrated 2026-06-15; frozen read-only). Do NOT use `bd` , TodoWrite, or markdown TODO lists.
