This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## Overview
MEMLNaut-NISPS (Neural Interactive Shaping of Parameter Spaces) is firmware for the MEMLNaut hardware platform - a custom embedded audio device built on Raspberry Pi Pico (RP2040). It implements interactive machine learning for real-time audio synthesis and processing, enabling users to shape sound parameters through reinforcement learning.
The `nisps-core/` directory contains a platform-agnostic C++20 extraction of the interactive ML engine. This header-only library can be used in any C++ project for neural network-based parameter mapping.
**Key differences from firmware**:
- ✅ Platform-agnostic (no Arduino/RP2040 dependencies)
- ✅ Header-only (just include and use)
- ✅ C++20 (uses std::span)
- ✅ Namespaced (`nisps::`)
- ❌ No audio synthesis (use it to *control* your synth)
- ❌ No hardware drivers
**Use case**: Control synthesizers, effects, lights, game parameters, or any system that responds to continuous parameters.
See `nisps-core/README.md` for complete documentation and examples.
The immersive app (`a-immersive.html` / `a-app.js`) uses a WASM-compiled MLP for all inference and training. The legacy JS engine (`iml.js`, `mlp.js`, `layer.js`, `node.js`) is still used by the three older playground variants (`app.js`, `b-app.js`, `c-app.js`) but is slated for migration to WASM (see meml-dj9).
**Architecture:**
```
Main thread Worker thread
WasmIML (nisps-wasm.js) nisps-wasm-worker.js
├─ WASM instance A (persistent) └─ WASM instance B (lazy)
│ inference() — every rAF tick trainEx() — off-thread
│ inferBatch() — heatmap sampling returns: weights + loss curve
cd playground/wasm && ./build.sh # requires emcc (Emscripten)
```
**Key difference from JS engine:** WASM uses float32 (not float64). The `spread`-aware `drawWeights`/`moveWeights` functions are implemented in the bindings file, not in nisps-core proper — they're playground-specific.
**Known issue:** Both the C++ `Train()` and WASM `train_ex` double-scale the loss when no sample weights are provided (each sample loss is weighted by 1/n, then the sum is multiplied by 1/n again). This is a backward-compat pattern from the C++ core (meml-ues).
### Debug Probe
The immersive app exposes `window.__nisps` when loaded with `?debug=1`. Used by Playwright e2e tests. Zero footprint in production.
| Method | Returns |
|--------|---------|
| `getOutputs()` | Current 126-element output vector |
The MLP uses ReLU hidden layers with a sigmoid output layer. With uniform [-1,1] weights, the sum of many weighted inputs at each layer drives sigmoid pre-activations far from zero (std dev ≈ √fan_in), causing outputs to saturate near 0 or 1. The `spread` parameter addresses this:
- **`spread=0`** (polarised): Weights drawn from uniform [-1,1]. RL noise cap = 0.3. Noise applied uniformly across layers. Outputs cluster at extremes — good for exploration of radical mappings.
- **`spread=1`** (centered): Weights scaled by 1/√fan_in per layer (Xavier initialization). RL noise cap = 0.05. Noise also scaled per-layer. Weight decay prevents magnitude drift. Outputs spread across the full [0,1] range — better for fine-grained RL shaping.
- **Intermediate values** interpolate linearly between these two regimes.
2.**`moveWeights(speed, spread)`** — RL exploration noise scale per layer
3.**Weight decay in `moveWeights`** — each call decays weights by `10% * spread` before adding noise, preventing unbounded magnitude drift from repeated thumbs-down. At spread=0 there is no decay (original behavior). At spread=1, weights decay ~10% per call, creating a natural equilibrium where exploration noise and decay balance out rather than weights growing until sigmoid permanently saturates.
4.**Noise cap** in thumbs-down handler — `0.3*(1-spread) + 0.05*spread`
Presets (`js/synth/presets.js`) control which parameters the ML engine can modify, with unselected params muted at safe defaults. Each preset defines per-param `{ muted, fixedValue, min, max, curve }` — no training examples or model weights.
| 3 (Advanced) | 3.1–3.3 | ~95 | + Cross-oscillator PM, feedback mixer, dual shapers, comb/gap filters, ring mod |
| 4 (Expert) | 4.1–4.2 | 126 | Full engine |
Presets use `curve` values to bias parameter distributions (<0.5=spendmoretimelow,> 0.5 = bias high) without clamping extremes. Users can tweak any preset via the group drawer after loading.
The immersive app (`a-immersive.html`) has a control surface system for tuning how exploration and learning feel. Full spec: `playground/SPEC-controls.md`.
**Architecture** — modular ES modules organized by phase, wired into `a-app.js`:
| Module | Phase | Purpose |
|--------|-------|---------|
| `js/ui/input-pipeline.js` | 1 | Processes raw joystick input through deadzone → zoom → curve → smoothing → momentum-as-zoom. Pure math, no DOM. |
| `js/ui/control-surface.js` | 1 | Compound axes (Boldness, Memory, Precision) that map single sliders to multiple underlying params. Offset-based override resolution (trim-pot model). 6 built-in control presets. |
| `js/ui/control-surface-ui.js` | 1 | DOM layer: 3 axis sliders on floating bar, gear icon settings drawer with per-param overrides. Injects its own CSS. |
| `js/ui/joy-map-enhanced.js` | 1 | Enhanced joy-map canvas: zoom minimap with adaptive grid, vanishing trail with Catmull-Rom spline and tap-to-return, dual concentric noise rings, frozen state overlay. |
| `js/ui/snapshot-stack.js` | 2 | Ring buffer (20 max) of weight snapshots. Auto-snapshot on train/randomize/thumbs-down. Multi-level undo. |
| `js/ui/ab-compare.js` | 2 | Rapid A/B weight state comparison. Capture, toggle, accept or revert. |
| `js/ui/region-pin.js` | 2 | Pins rectangular input-space regions (Approach A: example pinning). Pinned examples always included in training. |
| `js/ui/param-pin.js` | 2 | Per-output pin flags. Pin mask passed to `moveWeights()` to skip pinned output nodes. |
| `js/ui/phase2-ui.js` | 2 | DOM: undo button with history popup, A/B toggle, region pin via long-press, param pin via double-tap. |
| `js/ui/pressure-feedback.js` | 3 | Touch force + hold duration → intensity multiplier for noise growth/decay. |
When a user manually overrides an individual param, the offset from the axis-derived value persists as the axis moves (like a trim pot on a mixing desk). Double-tap an axis to re-link all params.
**Input Pipeline** — sits between physical joystick and MLP. Key feature: **zoom** narrows the effective input window around an anchor point (`effective = anchor + (raw - 0.5) * zoom_level`). Zoom-at-zero freezes input. Three anchor modes: auto (anchor follows current position when zoom changes), sticky (explicit anchor), center (always 0.5).
**Control Presets**: Default, First Touch, Jazz Hands, Sculptor, Improviser, Microscope. These set compound axis positions — they don't include network weights or synth preset selection.
**Integration** — the control surface dispatches `controlsurface:change` CustomEvents. `a-app.js` listens and updates the input pipeline config, spread level, and RL parameters (noise cap, growth, decay, floor, zoom-aware feedback scaling). Pipeline-processed coordinates are cached (`_lastPipeX/Y`) so `getCurrentInputs()` and `setCurrentInputs()` use the same values the MLP sees. State is persisted to localStorage alongside existing app state.
Playwright e2e tests cover the immersive app's ML engine, UI state machines, input pipeline, and persistence. Tests run headless Chromium against a Python HTTP server.
```bash
# Run all tests (starts server automatically on port 7331)
npx playwright test
# Run with browser visible
npx playwright test --headed
# Run a specific test file
npx playwright test tests/e2e/ml-engine.spec.js
```
**Test files** (`tests/e2e/`):
| File | Coverage |
|------|----------|
| `ml-engine.spec.js` | WASM inference bounds, training loss, thumbs up/down, async training, example capture |
| `wasm-api.spec.js` | Batch inference, evalLoss, getLayerStats, loss history curve, pin mask |
Tests use the `?debug=1` probe (`window.__nisps`) for programmatic access to the ML engine. The `helpers.js` module provides `loadApp(page)` which clears localStorage, sets `nisps-help-seen`, and waits for WASM initialization.
This is an Arduino project targeting the MEMLNaut RP2350 hardware. Build and flash it with the repo-local helper scripts, which wrap the correct board target and compiler settings.
The active mode is selected at compile-time via `#define MEMLNAUT_MODE_TYPE` in `MEMLNaut-NISPS.ino`. Modes implement the `MEMLNautMode` concept (see `modes/MEMLNautMode.hpp`):
| `MEMLNautModeXIASRI` | Audio-reactive mode using machine listening analysis |
| `MEMLNautModeSoundAnalysisMIDI` | Sound analysis with MIDI output |
### Voice Spaces
Voice spaces map ML output parameters to audio engine parameters. They are defined as lambda functions that translate a normalized parameter array into synthesizer/processor settings. See `voicespaces/` for examples:
- PAF synth presets: `VoiceSpace1.hpp`, `VoiceSpaceQuadDetune.hpp`, etc.
The codebase uses RP2040-specific memory placement:
-`AUDIO_MEM` / `AUDIO_FUNC`: Place audio-critical code/data in SRAM
-`APP_SRAM` / `__not_in_flash("app")`: Keep frequently-accessed data out of flash
## Audio Parameters
Sample rate is defined in `AudioDriver::GetSampleRate()`. The audio callback `audio_block_callback` runs on Core 1 and processes stereo audio (`stereosample_t`).
