memlnaut-nisps/CLAUDE.md

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Overview

MEMLNaut-NISPS (Neural Interactive Shaping of Parameter Spaces) is firmware for the MEMLNaut hardware platform - a custom embedded audio device built on Raspberry Pi Pico (RP2040). It implements interactive machine learning for real-time audio synthesis and processing, enabling users to shape sound parameters through reinforcement learning.

Project documentation: https://musicallyembodiedml.github.io/memlnaut/approaches/nisps

## NISPS Core Library

The `nisps-core/` directory contains a platform-agnostic C++20 extraction of the interactive ML engine. This header-only library can be used in any C++ project for neural network-based parameter mapping.

**Key differences from firmware**:
- ✅ Platform-agnostic (no Arduino/RP2040 dependencies)
- ✅ Header-only (just include and use)
- ✅ C++20 (uses std::span)
- ✅ Namespaced (`nisps::`)
- ❌ No audio synthesis (use it to *control* your synth)
- ❌ No hardware drivers

**Use case**: Control synthesizers, effects, lights, game parameters, or any system that responds to continuous parameters.

See `nisps-core/README.md` for complete documentation and examples.

## Web Playground

The `playground/` directory contains a browser-based interactive demo of the NISPS ML engine. It's a faithful JavaScript port of nisps-core's MLP + IML, with no build step or dependencies.

- **2 inputs** (virtual joystick X/Y) mapped through a `[3, 32, 48, 64, 126]` MLP to **126 outputs**
- **Two output modes**:
  - **Visual**: first 20 outputs control a Canvas2D flow-field particle system
  - **Synth (C15)**: all 126 outputs control the C15 WASM synthesizer — every sonically meaningful continuous parameter across envelopes, oscillators, shapers, filters, feedback/output mixers, cabinet, and effects
- **Two learning modes**: Examples (set slider targets, add examples, train) and RL Feedback (thumbs up/down with exploration noise)
- **Serve statically**: `cd playground && python3 -m http.server`
- **Mobile-first**: designed for touch/foldable phone use

Key files: `js/nisps/` (ML core port), `js/ui/` (visualizer, joystick, controls, input pipeline, control surface), `js/synth/` (C15 bridge, param map, arpeggiator), `js/a-app.js` (immersive app wiring).

### URL Parameters

| Param | Range | Default | Effect |
|-------|-------|---------|--------|
| `tame` | 0–1 | 1 | Constrains synth output ranges toward safe limits |
| `spread` | 0–1 | 0.6 | Controls weight initialization, RL noise scaling, and weight decay (see below) |
| `preset` | preset id | _(none)_ | Auto-loads a synth parameter preset on first visit (e.g. `?preset=beginner-1`) |

#### `spread` — sigmoid saturation control

The MLP uses ReLU hidden layers with a sigmoid output layer. With uniform [-1,1] weights, the sum of many weighted inputs at each layer drives sigmoid pre-activations far from zero (std dev ≈ √fan_in), causing outputs to saturate near 0 or 1. The `spread` parameter addresses this:

- **`spread=0`** (polarised): Weights drawn from uniform [-1,1]. RL noise cap = 0.3. Noise applied uniformly across layers. Outputs cluster at extremes — good for exploration of radical mappings.
- **`spread=1`** (centered): Weights scaled by 1/√fan_in per layer (Xavier initialization). RL noise cap = 0.05. Noise also scaled per-layer. Weight decay prevents magnitude drift. Outputs spread across the full [0,1] range — better for fine-grained RL shaping.
- **Intermediate values** interpolate linearly between these two regimes.

Affects four code paths:
1. **`drawWeights(spread)`** — initial randomisation weight scale
2. **`moveWeights(speed, spread)`** — RL exploration noise scale per layer
3. **Weight decay in `moveWeights`** — each call decays weights by `10% * spread` before adding noise, preventing unbounded magnitude drift from repeated thumbs-down. At spread=0 there is no decay (original behavior). At spread=1, weights decay ~10% per call, creating a natural equilibrium where exploration noise and decay balance out rather than weights growing until sigmoid permanently saturates.
4. **Noise cap** in thumbs-down handler — `0.3*(1-spread) + 0.05*spread`

### C15 Parameter Map

The 126 synth parameters in `js/synth/param-map.js` were curated from the C15's 287 total parameters. Excluded categories:

| Excluded | Count | Reason |
|----------|-------|--------|
| Hardware Amount/Source | 56 | No physical MIDI hardware in browser |
| Macro Controls/Times | 12 | Meta-routing layer conflicts with direct ML control |
| Scale offsets | 13 | Microtuning would break pitch unpredictably |
| Key tracking (`*_KT`) | 11 | Pitch-dependent scaling needs calibrated defaults |
| Velocity (`*_Vel`) | 11 | Velocity-dependent, ML can't observe key velocity |
| Envelope mod depths (`*_Env_A/B/C`) | 19 | Multiplicative interaction with envelope shapes makes space too hard to learn |
| Discrete/structural | 15 | Osc Pitch (full sweep), Master Vol/Tune, Voice Mute/Fade, Unison Voices, Mono modes, Split, Osc Reset |
| Secondary config | 7 | Att Curve, Elevate, Chirp, Decay Gate, Retrigger |
| PM shaper blend | 4 | Secondary routing params |
| FB Mix source selects | 4 | Discrete A/B selectors |

### Synth Presets

Presets (`js/synth/presets.js`) control which parameters the ML engine can modify, with unselected params muted at safe defaults. Each preset defines per-param `{ muted, fixedValue, min, max, curve }` — no training examples or model weights.

4 tiers of progressive complexity:

| Tier | Presets | Active params | What's exposed |
|------|---------|---------------|----------------|
| 1 (Beginner) | 1.1–1.4 | 15 | Basic ADSR, SVF cutoff/res, Shaper A drive/fold, output levels, reverb mix |
| 2 (Intermediate) | 2.1–2.4 | 40 | + Env B/C, filter FM, effects (reverb/echo/flanger), cabinet, stereo panning |
| 3 (Advanced) | 3.1–3.3 | ~95 | + Cross-oscillator PM, feedback mixer, dual shapers, comb/gap filters, ring mod |
| 4 (Expert) | 4.1–4.2 | 126 | Full engine |

Presets use `curve` values to bias parameter distributions (< 0.5 = spend more time low, > 0.5 = bias high) without clamping extremes. Users can tweak any preset via the group drawer after loading.

### Control Surface (Phase 1)

The immersive app (`a-immersive.html`) has a control surface system for tuning how exploration and learning feel. Full spec: `playground/SPEC-controls.md`.

**Architecture** — modular ES modules organized by phase, wired into `a-app.js`:

| Module | Phase | Purpose |
|--------|-------|---------|
| `js/ui/input-pipeline.js` | 1 | Processes raw joystick input through deadzone → zoom → curve → smoothing → momentum-as-zoom. Pure math, no DOM. |
| `js/ui/control-surface.js` | 1 | Compound axes (Boldness, Memory, Precision) that map single sliders to multiple underlying params. Offset-based override resolution (trim-pot model). 6 built-in control presets. |
| `js/ui/control-surface-ui.js` | 1 | DOM layer: 3 axis sliders on floating bar, gear icon settings drawer with per-param overrides. Injects its own CSS. |
| `js/ui/joy-map-enhanced.js` | 1 | Enhanced joy-map canvas: zoom minimap with adaptive grid, vanishing trail with Catmull-Rom spline and tap-to-return, dual concentric noise rings, frozen state overlay. |
| `js/ui/snapshot-stack.js` | 2 | Ring buffer (20 max) of weight snapshots. Auto-snapshot on train/randomize/thumbs-down. Multi-level undo. |
| `js/ui/ab-compare.js` | 2 | Rapid A/B weight state comparison. Capture, toggle, accept or revert. |
| `js/ui/region-pin.js` | 2 | Pins rectangular input-space regions (Approach A: example pinning). Pinned examples always included in training. |
| `js/ui/param-pin.js` | 2 | Per-output pin flags. Pin mask passed to `moveWeights()` to skip pinned output nodes. |
| `js/ui/phase2-ui.js` | 2 | DOM: undo button with history popup, A/B toggle, region pin via long-press, param pin via double-tap. |
| `js/ui/pressure-feedback.js` | 3 | Touch force + hold duration → intensity multiplier for noise growth/decay. |
| `js/ui/auto-explore.js` | 3 | Automated thumbs-down at configurable interval. Zoom-scaled intensity. |
| `js/ui/input-heatmap.js` | 3 | 2D color field sampling MLP across input space. 3 color modes, zoom-aware resampling. |
| `js/ui/phase3-ui.js` | 3 | DOM: auto-explore toggle with progress ring, heatmap eye icon, pressure indicators. |
| `js/ui/output-pipeline.js` | 4 | Global curve → smoothing → slew rate → freeze gate on MLP outputs before synth/visual routing. |
| `js/ui/weight-health.js` | 4 | Weight magnitude histogram, dead/saturating/healthy status detection, ambient visualization. |
| `js/ui/gradient-flow.js` | 4 | Per-layer weight-delta analysis after training. Vanishing/exploding/converged detection. |
| `js/ui/session-presets.js` | 4 | Save/load full session state. URL sharing via compact params. |
| `js/ui/phase4-ui.js` | 4 | DOM: freeze button, network health panel, session preset UI, output pipeline slider wiring. |

**Compound Axes** — each controls 4-6 underlying parameters via interpolation tables:

- **Boldness** (Caution ↔ Bold): input zoom, noise cap, noise growth, learning rate, weight decay, noise distribution
- **Memory** (Amnesia ↔ Elephant): max examples, example decay, weight decay, noise decay, convergence threshold
- **Precision** (Raw ↔ Precise): input curve, deadzone, smoothing, slew rate, momentum-zoom mode

When a user manually overrides an individual param, the offset from the axis-derived value persists as the axis moves (like a trim pot on a mixing desk). Double-tap an axis to re-link all params.

**Input Pipeline** — sits between physical joystick and MLP. Key feature: **zoom** narrows the effective input window around an anchor point (`effective = anchor + (raw - 0.5) * zoom_level`). Zoom-at-zero freezes input. Three anchor modes: auto (anchor follows current position when zoom changes), sticky (explicit anchor), center (always 0.5).

**Control Presets**: Default, First Touch, Jazz Hands, Sculptor, Improviser, Microscope. These set compound axis positions — they don't include network weights or synth preset selection.

**Integration** — the control surface dispatches `controlsurface:change` CustomEvents. `a-app.js` listens and updates the input pipeline config, spread level, and RL parameters (noise cap, growth, decay, floor, zoom-aware feedback scaling). Pipeline-processed coordinates are cached (`_lastPipeX/Y`) so `getCurrentInputs()` and `setCurrentInputs()` use the same values the MLP sees. State is persisted to localStorage alongside existing app state.

**Remaining**: Engine configuration panel (Part 8 of spec) — network architecture, loss function, optimizer selection.

## Build System

This is an Arduino project targeting Raspberry Pi Pico. Build and upload using Arduino IDE or arduino-cli with the earlephilhower/pico board package.

```bash
# Initialize submodules (required for memllib and memlp)
git submodule update --init --recursive

# Build (adjust port as needed)
arduino-cli compile --fqbn rp2040:rp2040:rpipico -b 115200 MEMLNaut-NISPS.ino
arduino-cli upload --fqbn rp2040:rp2040:rpipico -p /dev/ttyACM0 MEMLNaut-NISPS.ino
```

## Architecture

### Dual-Core Design

The RP2040's dual cores are used for separation of concerns:
- **Core 0**: UI loop, ML inference, hardware interface polling (5ms period)
- **Core 1**: Real-time audio processing, parameter updates, MIDI polling

Inter-core synchronization uses memory barriers (`MEMORY_BARRIER()`, `WRITE_VOLATILE()`, `READ_VOLATILE()`) and RP2040 queues (`queue_t`).

### Mode System

The active mode is selected at compile-time via `#define MEMLNAUT_MODE_TYPE` in `MEMLNaut-NISPS.ino`. Modes implement the `MEMLNautMode` concept (see `modes/MEMLNautMode.hpp`):

| Mode | Purpose |
|------|---------|
| `MEMLNautModeChannelStrip` | Audio channel strip (EQ, compression, gain staging) |
| `MEMLNautModePAFSynth` | PAF (Phase Aligned Formant) synthesis with MIDI |
| `MEMLNautModeXIASRI` | Audio-reactive mode using machine listening analysis |
| `MEMLNautModeSoundAnalysisMIDI` | Sound analysis with MIDI output |

### Voice Spaces

Voice spaces map ML output parameters to audio engine parameters. They are defined as lambda functions that translate a normalized parameter array into synthesizer/processor settings. See `voicespaces/` for examples:
- PAF synth presets: `VoiceSpace1.hpp`, `VoiceSpaceQuadDetune.hpp`, etc.
- Channel strip presets: `voicespaces/ChannelStrip/basic.hpp` (Neve, SSL emulations)

### Key Components

- **IMLInterface** (`IMLInterface.hpp`): Interactive ML interface using an MLP for inference/training
- **InterfaceRL**: Reinforcement learning interface from memllib that handles joystick input and learning
- **AudioAppBase**: Template base class for audio applications
- **XiasriAnalysis**: Real-time audio feature extraction (pitch, aperiodicity, energy, brightness)

### Submodules (in `src/`)

- **memllib**: Hardware abstraction, audio drivers, synth components, RL interfaces
- **memlp**: MLP (Multi-Layer Perceptron) implementation for embedded ML
- **daisysp**: DSP library (filters, drums, effects, synthesis)

## Memory Sections

The codebase uses RP2040-specific memory placement:
- `AUDIO_MEM` / `AUDIO_FUNC`: Place audio-critical code/data in SRAM
- `APP_SRAM` / `__not_in_flash("app")`: Keep frequently-accessed data out of flash

## Audio Parameters

Sample rate is defined in `AudioDriver::GetSampleRate()`. The audio callback `audio_block_callback` runs on Core 1 and processes stereo audio (`stereosample_t`).
-												add CLAUDE.md for AI assistant context

											
										
										
											2026-02-08 15:49:16 +01:00
+								# CLAUDE.md
 								This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
 								## Overview
 								MEMLNaut-NISPS (Neural Interactive Shaping of Parameter Spaces) is firmware for the MEMLNaut hardware platform - a custom embedded audio device built on Raspberry Pi Pico (RP2040). It implements interactive machine learning for real-time audio synthesis and processing, enabling users to shape sound parameters through reinforcement learning.
 								Project documentation: https://musicallyembodiedml.github.io/memlnaut/approaches/nisps
-												feat: extract nisps-core platform-agnostic ML library

Extract the interactive machine learning engine from MEMLNaut-NISPS
firmware into a standalone, platform-agnostic C++20 header-only library.

What is nisps-core?
-------------------
NISPS (Neural Interactive Shaping of Parameter Spaces) core is a
parameter mapping engine. It takes N input parameters (joystick,
sensors, audio features) and maps them to M output parameters through
an interactively-trained neural network.

Use it to control: synthesizers, effects, lights, robots, game
parameters, or anything that responds to continuous control data.

Key Features
------------
- Header-only: No compilation needed, just include and use
- Platform-agnostic: Pure C++20, works anywhere
- Zero dependencies: Only standard library
- Interactive learning: Train by demonstration
- Lightweight: ~3,500 lines of optimized neural network code
- Flexible: Map 1-100 inputs to 1-100 outputs

Architecture
------------
Core components:
- IML: High-level interactive ML interface
- MLP: Multi-layer perceptron (feedforward neural network)
- Dataset: Training data management with replay memory
- Layer/Node: Neural network building blocks
- Loss: MSE and categorical cross-entropy functions
- Utils: Activation functions (sigmoid, ReLU, tanh, etc.)

Transformations Applied
-----------------------
✅ Removed Arduino/RP2040 dependencies (Serial, SD, Pico SDK)
✅ Removed audio synthesis code (nisps-core is control-only)
✅ Added nisps namespace to all code
✅ Converted to header-only library with _impl.hpp pattern
✅ Updated to C++20 (required for std::span)
✅ Removed platform-specific serialization
✅ Replaced debug macros with no-op stubs
✅ Added comprehensive documentation and examples

Files Added
-----------
- nisps-core/README.md: Complete documentation and API reference
- nisps-core/CHANGELOG.md: Version history and migration guide
- nisps-core/include/nisps/*.hpp: 13 header files (~3,500 lines)
- nisps-core/test/main.cpp: XOR test demonstrating basic usage
- nisps-core/examples/simple_mapping.cpp: Interactive demo
- nisps-core/CMakeLists.txt: Build system for tests

Testing
-------
✅ Compiles with GCC 14.2 (C++20)
✅ All tests passing
✅ Successfully instantiates networks and runs inference

Performance
-----------
- Inference: 1-10 µs for small networks (2-10-10-4)
- Training: 10-100 ms for 100 examples, 1000 iterations
- Memory: ~1 KB per hidden neuron

Migration from Embedded IMLInterface
------------------------------------
Old (embedded):
  IMLInterface iml(n_inputs, n_outputs);

New (nisps-core):
  nisps::IML<float> iml(n_inputs, n_outputs);

All method names remain the same, just add the namespace.

Related
-------
- Implements: NISPS_CORE_EXTRACTION_PLAN.md
- Task graph: NISPS_CORE_TASKS.md
- Origin: MEMLNaut-NISPS firmware
- Docs: https://musicallyembodiedml.github.io/memlnaut/

Co-authored-by: Claude Code <claude@anthropic.com>

											
										
										
											2026-02-08 17:47:23 +01:00
+								## NISPS Core Library
 								The `nisps-core/` directory contains a platform-agnostic C++20 extraction of the interactive ML engine. This header-only library can be used in any C++ project for neural network-based parameter mapping.
 								**Key differences from firmware**:
 								- ✅ Platform-agnostic (no Arduino/RP2040 dependencies)
 								- ✅ Header-only (just include and use)
 								- ✅ C++20 (uses std::span)
 								- ✅ Namespaced (`nisps::`)
 								- ❌ No audio synthesis (use it to *control* your synth)
 								- ❌ No hardware drivers
 								**Use case**: Control synthesizers, effects, lights, game parameters, or any system that responds to continuous parameters.
 								See `nisps-core/README.md` for complete documentation and examples.
-												feat: add web-based interactive playground for NISPS

Faithful JS port of nisps-core MLP + IML engine with a touch-friendly
UI for exploring neural parameter mapping visually. Two learning modes:
example-based (set slider targets) and RL feedback (thumbs up/down with
exploration noise). Flow field particle system controlled by 8 MLP outputs.

											
										
										
											2026-02-11 13:17:17 +01:00
+								## Web Playground
 								The `playground/` directory contains a browser-based interactive demo of the NISPS ML engine. It's a faithful JavaScript port of nisps-core's MLP + IML, with no build step or dependencies.
-												docs: update CLAUDE.md and playground README for 126-param C15 map

Document the expanded parameter map, MLP architecture, exclusion
rationale table, and synth mode in both project-level and playground docs.

											
										
										
											2026-03-21 22:11:45 +01:00
+								- **2 inputs** (virtual joystick X/Y) mapped through a `[3, 32, 48, 64, 126]` MLP to **126 outputs**
 								- **Two output modes**:
 								  - **Visual**: first 20 outputs control a Canvas2D flow-field particle system
 								  - **Synth (C15)**: all 126 outputs control the C15 WASM synthesizer — every sonically meaningful continuous parameter across envelopes, oscillators, shapers, filters, feedback/output mixers, cabinet, and effects
-												feat: add web-based interactive playground for NISPS

Faithful JS port of nisps-core MLP + IML engine with a touch-friendly
UI for exploring neural parameter mapping visually. Two learning modes:
example-based (set slider targets) and RL feedback (thumbs up/down with
exploration noise). Flow field particle system controlled by 8 MLP outputs.

											
										
										
											2026-02-11 13:17:17 +01:00
+								- **Two learning modes**: Examples (set slider targets, add examples, train) and RL Feedback (thumbs up/down with exploration noise)
 								- **Serve statically**: `cd playground && python3 -m http.server`
 								- **Mobile-first**: designed for touch/foldable phone use
-												feat(playground): implement Phase 1 control surface — compound axes, input pipeline, enhanced joy-map

Three new standalone ES modules + integration into a-app.js:

- input-pipeline.js: 5-stage processing (deadzone → zoom → curve → smoothing →
  momentum-as-zoom), 3 anchor modes, zoom-at-zero freeze, per-axis overrides
- control-surface.js: compound axes (Boldness/Memory/Precision) with interpolation
  tables, offset-based override resolution (trim-pot model), 6 built-in presets
- control-surface-ui.js: floating bar axis sliders, gear icon settings drawer with
  Input/Training/Exploration/Output sections, log-scale sliders, override dots
- joy-map-enhanced.js: zoom minimap with adaptive grid (4×4→32×32), vanishing trail
  with Catmull-Rom spline + tap-to-return, dual concentric noise rings, frozen overlay

Integration fixes from fresh-eyes review:
- getCurrentInputs()/setCurrentInputs() use cached pipeline coords (not raw)
- CSS noise ring hidden when canvas version active (no doubling)
- Input mode switch re-runs through pipeline
- Control surface state persisted to localStorage

Implements full Phase 1 of SPEC-controls.md plus bonus items from later phases
(zoom-aware feedback, control presets with override resolution, input curve/deadzone/
smoothing/momentum all wired).

											
										
										
											2026-03-26 09:24:24 +01:00
+								Key files: `js/nisps/` (ML core port), `js/ui/` (visualizer, joystick, controls, input pipeline, control surface), `js/synth/` (C15 bridge, param map, arpeggiator), `js/a-app.js` (immersive app wiring).
-												docs: update CLAUDE.md and playground README for 126-param C15 map

Document the expanded parameter map, MLP architecture, exclusion
rationale table, and synth mode in both project-level and playground docs.

											
										
										
											2026-03-21 22:11:45 +01:00
-												feat(playground): add spread param for sigmoid saturation control and fix randomise sync

Add ?spread=0-1 URL param that controls weight initialization scaling,
RL noise scaling per layer, noise cap, and weight decay to prevent
sigmoid output saturation. At spread=0 (original behavior) weights are
uniform [-1,1] and outputs polarise near 0/1. At spread=1 weights use
Xavier scaling (1/sqrt(fan_in)), noise is proportionally reduced, and
10% weight decay per thumbs-down prevents unbounded magnitude drift.

Also fix randomise to re-inject current joystick position and re-run
inference before routing outputs, eliminating the jump on first
joystick move after randomise.

Defaults: tame=1, spread=0.6 across all app variants.

											
										
										
											2026-03-22 01:10:26 +01:00
+								### URL Parameters
 								| Param | Range | Default | Effect |
 								|-------|-------|---------|--------|
 								| `tame` | 0–1 | 1 | Constrains synth output ranges toward safe limits |
 								| `spread` | 0–1 | 0.6 | Controls weight initialization, RL noise scaling, and weight decay (see below) |
-												feat(playground): add tiered synth parameter preset system

4 tiers of progressive complexity (Beginner 15 params → Expert 126),
13 presets total with per-param min/max/curve overrides that bias
distributions without clamping extremes. Preset dropdown in UI,
persisted to localStorage, supports ?preset= URL param.

											
										
										
											2026-03-22 23:50:56 +01:00
+								| `preset` | preset id | _(none)_ | Auto-loads a synth parameter preset on first visit (e.g. `?preset=beginner-1`) |
-												feat(playground): add spread param for sigmoid saturation control and fix randomise sync

Add ?spread=0-1 URL param that controls weight initialization scaling,
RL noise scaling per layer, noise cap, and weight decay to prevent
sigmoid output saturation. At spread=0 (original behavior) weights are
uniform [-1,1] and outputs polarise near 0/1. At spread=1 weights use
Xavier scaling (1/sqrt(fan_in)), noise is proportionally reduced, and
10% weight decay per thumbs-down prevents unbounded magnitude drift.

Also fix randomise to re-inject current joystick position and re-run
inference before routing outputs, eliminating the jump on first
joystick move after randomise.

Defaults: tame=1, spread=0.6 across all app variants.

											
										
										
											2026-03-22 01:10:26 +01:00
 								#### `spread` — sigmoid saturation control
 								The MLP uses ReLU hidden layers with a sigmoid output layer. With uniform [-1,1] weights, the sum of many weighted inputs at each layer drives sigmoid pre-activations far from zero (std dev ≈ √fan_in), causing outputs to saturate near 0 or 1. The `spread` parameter addresses this:
 								- **`spread=0`** (polarised): Weights drawn from uniform [-1,1]. RL noise cap = 0.3. Noise applied uniformly across layers. Outputs cluster at extremes — good for exploration of radical mappings.
 								- **`spread=1`** (centered): Weights scaled by 1/√fan_in per layer (Xavier initialization). RL noise cap = 0.05. Noise also scaled per-layer. Weight decay prevents magnitude drift. Outputs spread across the full [0,1] range — better for fine-grained RL shaping.
 								- **Intermediate values** interpolate linearly between these two regimes.
 								Affects four code paths:
 . **`drawWeights(spread)`** — initial randomisation weight scale
 . **`moveWeights(speed, spread)`** — RL exploration noise scale per layer
 . **Weight decay in `moveWeights`** — each call decays weights by `10% * spread` before adding noise, preventing unbounded magnitude drift from repeated thumbs-down. At spread=0 there is no decay (original behavior). At spread=1, weights decay ~10% per call, creating a natural equilibrium where exploration noise and decay balance out rather than weights growing until sigmoid permanently saturates.
 . **Noise cap** in thumbs-down handler — `0.3*(1-spread) + 0.05*spread`
-												docs: update CLAUDE.md and playground README for 126-param C15 map

Document the expanded parameter map, MLP architecture, exclusion
rationale table, and synth mode in both project-level and playground docs.

											
										
										
											2026-03-21 22:11:45 +01:00
+								### C15 Parameter Map
 								The 126 synth parameters in `js/synth/param-map.js` were curated from the C15's 287 total parameters. Excluded categories:
 								| Excluded | Count | Reason |
 								|----------|-------|--------|
 								| Hardware Amount/Source | 56 | No physical MIDI hardware in browser |
 								| Macro Controls/Times | 12 | Meta-routing layer conflicts with direct ML control |
 								| Scale offsets | 13 | Microtuning would break pitch unpredictably |
 								| Key tracking (`*_KT`) | 11 | Pitch-dependent scaling needs calibrated defaults |
 								| Velocity (`*_Vel`) | 11 | Velocity-dependent, ML can't observe key velocity |
 								| Envelope mod depths (`*_Env_A/B/C`) | 19 | Multiplicative interaction with envelope shapes makes space too hard to learn |
 								| Discrete/structural | 15 | Osc Pitch (full sweep), Master Vol/Tune, Voice Mute/Fade, Unison Voices, Mono modes, Split, Osc Reset |
 								| Secondary config | 7 | Att Curve, Elevate, Chirp, Decay Gate, Retrigger |
 								| PM shaper blend | 4 | Secondary routing params |
 								| FB Mix source selects | 4 | Discrete A/B selectors |
-												feat: add web-based interactive playground for NISPS

Faithful JS port of nisps-core MLP + IML engine with a touch-friendly
UI for exploring neural parameter mapping visually. Two learning modes:
example-based (set slider targets) and RL feedback (thumbs up/down with
exploration noise). Flow field particle system controlled by 8 MLP outputs.

											
										
										
											2026-02-11 13:17:17 +01:00
-												feat(playground): add tiered synth parameter preset system

4 tiers of progressive complexity (Beginner 15 params → Expert 126),
13 presets total with per-param min/max/curve overrides that bias
distributions without clamping extremes. Preset dropdown in UI,
persisted to localStorage, supports ?preset= URL param.

											
										
										
											2026-03-22 23:50:56 +01:00
+								### Synth Presets
 								Presets (`js/synth/presets.js`) control which parameters the ML engine can modify, with unselected params muted at safe defaults. Each preset defines per-param `{ muted, fixedValue, min, max, curve }` — no training examples or model weights.
 tiers of progressive complexity:
 								| Tier | Presets | Active params | What's exposed |
 								|------|---------|---------------|----------------|
 								| 1 (Beginner) | 1.1–1.4 | 15 | Basic ADSR, SVF cutoff/res, Shaper A drive/fold, output levels, reverb mix |
 								| 2 (Intermediate) | 2.1–2.4 | 40 | + Env B/C, filter FM, effects (reverb/echo/flanger), cabinet, stereo panning |
 								| 3 (Advanced) | 3.1–3.3 | ~95 | + Cross-oscillator PM, feedback mixer, dual shapers, comb/gap filters, ring mod |
 								| 4 (Expert) | 4.1–4.2 | 126 | Full engine |
 								Presets use `curve` values to bias parameter distributions (< 0.5 = spend more time low, > 0.5 = bias high) without clamping extremes. Users can tweak any preset via the group drawer after loading.
-												feat(playground): implement Phase 1 control surface — compound axes, input pipeline, enhanced joy-map

Three new standalone ES modules + integration into a-app.js:

- input-pipeline.js: 5-stage processing (deadzone → zoom → curve → smoothing →
  momentum-as-zoom), 3 anchor modes, zoom-at-zero freeze, per-axis overrides
- control-surface.js: compound axes (Boldness/Memory/Precision) with interpolation
  tables, offset-based override resolution (trim-pot model), 6 built-in presets
- control-surface-ui.js: floating bar axis sliders, gear icon settings drawer with
  Input/Training/Exploration/Output sections, log-scale sliders, override dots
- joy-map-enhanced.js: zoom minimap with adaptive grid (4×4→32×32), vanishing trail
  with Catmull-Rom spline + tap-to-return, dual concentric noise rings, frozen overlay

Integration fixes from fresh-eyes review:
- getCurrentInputs()/setCurrentInputs() use cached pipeline coords (not raw)
- CSS noise ring hidden when canvas version active (no doubling)
- Input mode switch re-runs through pipeline
- Control surface state persisted to localStorage

Implements full Phase 1 of SPEC-controls.md plus bonus items from later phases
(zoom-aware feedback, control presets with override resolution, input curve/deadzone/
smoothing/momentum all wired).

											
										
										
											2026-03-26 09:24:24 +01:00
+								### Control Surface (Phase 1)
 								The immersive app (`a-immersive.html`) has a control surface system for tuning how exploration and learning feel. Full spec: `playground/SPEC-controls.md`.
-												feat(playground): implement Phases 2-4 of control surface spec

Phase 2 — Pinning + History:
- snapshot-stack.js: ring buffer (20 max) with auto-snapshot on
  train/randomize/thumbs-down, multi-level undo, tagged entries
- ab-compare.js: A/B weight state comparison with capture/toggle/accept/revert
- region-pin.js: pin rectangular input-space regions (Approach A: example
  pinning), pinned examples always included in training
- param-pin.js: per-output pin flags, pin mask skips pinned nodes in moveWeights
- phase2-ui.js: undo button with history popup, A/B toggle, long-press region
  pin, double-tap param pin
- Modified mlp.js/iml.js/nisps-wasm.js to accept outputPinMask in moveWeights

Phase 3 — Input Refinement + Exploration:
- pressure-feedback.js: touch force + hold duration → intensity multiplier
- auto-explore.js: automated thumbs-down at configurable interval, zoom-scaled
- input-heatmap.js: 16×16 MLP sampling, 3 color modes (luminance/variance/
  divergence), zoom-aware resampling, offscreen canvas rendering
- phase3-ui.js: auto-explore toggle with progress ring, heatmap eye icon,
  pressure indicators, settings drawer section
- joy-map-enhanced.js: added setHeatmap() for background layer rendering

Phase 4 — Output Pipeline + Visualization + Polish:
- output-pipeline.js: global curve → smoothing → slew rate → freeze gate
- weight-health.js: weight magnitude histogram, dead/saturating/healthy status
- gradient-flow.js: per-layer weight-delta analysis, vanishing/exploding detection
- session-presets.js: save/load full state, URL sharing via compact params
- phase4-ui.js: freeze button, network health panel, session preset UI

All phases merged into a-app.js with proper integration: auto-snapshots,
pressure-modulated RL, heatmap triggers, output pipeline in routeOutputs,
gradient capture around training, persistence for all new state.

											
										
										
											2026-03-26 09:48:12 +01:00
+								**Architecture** — modular ES modules organized by phase, wired into `a-app.js`:
 								| Module | Phase | Purpose |
 								|--------|-------|---------|
 								| `js/ui/input-pipeline.js` | 1 | Processes raw joystick input through deadzone → zoom → curve → smoothing → momentum-as-zoom. Pure math, no DOM. |
 								| `js/ui/control-surface.js` | 1 | Compound axes (Boldness, Memory, Precision) that map single sliders to multiple underlying params. Offset-based override resolution (trim-pot model). 6 built-in control presets. |
 								| `js/ui/control-surface-ui.js` | 1 | DOM layer: 3 axis sliders on floating bar, gear icon settings drawer with per-param overrides. Injects its own CSS. |
 								| `js/ui/joy-map-enhanced.js` | 1 | Enhanced joy-map canvas: zoom minimap with adaptive grid, vanishing trail with Catmull-Rom spline and tap-to-return, dual concentric noise rings, frozen state overlay. |
 								| `js/ui/snapshot-stack.js` | 2 | Ring buffer (20 max) of weight snapshots. Auto-snapshot on train/randomize/thumbs-down. Multi-level undo. |
 								| `js/ui/ab-compare.js` | 2 | Rapid A/B weight state comparison. Capture, toggle, accept or revert. |
 								| `js/ui/region-pin.js` | 2 | Pins rectangular input-space regions (Approach A: example pinning). Pinned examples always included in training. |
 								| `js/ui/param-pin.js` | 2 | Per-output pin flags. Pin mask passed to `moveWeights()` to skip pinned output nodes. |
 								| `js/ui/phase2-ui.js` | 2 | DOM: undo button with history popup, A/B toggle, region pin via long-press, param pin via double-tap. |
 								| `js/ui/pressure-feedback.js` | 3 | Touch force + hold duration → intensity multiplier for noise growth/decay. |
 								| `js/ui/auto-explore.js` | 3 | Automated thumbs-down at configurable interval. Zoom-scaled intensity. |
 								| `js/ui/input-heatmap.js` | 3 | 2D color field sampling MLP across input space. 3 color modes, zoom-aware resampling. |
 								| `js/ui/phase3-ui.js` | 3 | DOM: auto-explore toggle with progress ring, heatmap eye icon, pressure indicators. |
 								| `js/ui/output-pipeline.js` | 4 | Global curve → smoothing → slew rate → freeze gate on MLP outputs before synth/visual routing. |
 								| `js/ui/weight-health.js` | 4 | Weight magnitude histogram, dead/saturating/healthy status detection, ambient visualization. |
 								| `js/ui/gradient-flow.js` | 4 | Per-layer weight-delta analysis after training. Vanishing/exploding/converged detection. |
 								| `js/ui/session-presets.js` | 4 | Save/load full session state. URL sharing via compact params. |
 								| `js/ui/phase4-ui.js` | 4 | DOM: freeze button, network health panel, session preset UI, output pipeline slider wiring. |
-												feat(playground): implement Phase 1 control surface — compound axes, input pipeline, enhanced joy-map

Three new standalone ES modules + integration into a-app.js:

- input-pipeline.js: 5-stage processing (deadzone → zoom → curve → smoothing →
  momentum-as-zoom), 3 anchor modes, zoom-at-zero freeze, per-axis overrides
- control-surface.js: compound axes (Boldness/Memory/Precision) with interpolation
  tables, offset-based override resolution (trim-pot model), 6 built-in presets
- control-surface-ui.js: floating bar axis sliders, gear icon settings drawer with
  Input/Training/Exploration/Output sections, log-scale sliders, override dots
- joy-map-enhanced.js: zoom minimap with adaptive grid (4×4→32×32), vanishing trail
  with Catmull-Rom spline + tap-to-return, dual concentric noise rings, frozen overlay

Integration fixes from fresh-eyes review:
- getCurrentInputs()/setCurrentInputs() use cached pipeline coords (not raw)
- CSS noise ring hidden when canvas version active (no doubling)
- Input mode switch re-runs through pipeline
- Control surface state persisted to localStorage

Implements full Phase 1 of SPEC-controls.md plus bonus items from later phases
(zoom-aware feedback, control presets with override resolution, input curve/deadzone/
smoothing/momentum all wired).

											
										
										
											2026-03-26 09:24:24 +01:00
 								**Compound Axes** — each controls 4-6 underlying parameters via interpolation tables:
 								- **Boldness** (Caution ↔ Bold): input zoom, noise cap, noise growth, learning rate, weight decay, noise distribution
 								- **Memory** (Amnesia ↔ Elephant): max examples, example decay, weight decay, noise decay, convergence threshold
 								- **Precision** (Raw ↔ Precise): input curve, deadzone, smoothing, slew rate, momentum-zoom mode
 								When a user manually overrides an individual param, the offset from the axis-derived value persists as the axis moves (like a trim pot on a mixing desk). Double-tap an axis to re-link all params.
 								**Input Pipeline** — sits between physical joystick and MLP. Key feature: **zoom** narrows the effective input window around an anchor point (`effective = anchor + (raw - 0.5) * zoom_level`). Zoom-at-zero freezes input. Three anchor modes: auto (anchor follows current position when zoom changes), sticky (explicit anchor), center (always 0.5).
 								**Control Presets**: Default, First Touch, Jazz Hands, Sculptor, Improviser, Microscope. These set compound axis positions — they don't include network weights or synth preset selection.
 								**Integration** — the control surface dispatches `controlsurface:change` CustomEvents. `a-app.js` listens and updates the input pipeline config, spread level, and RL parameters (noise cap, growth, decay, floor, zoom-aware feedback scaling). Pipeline-processed coordinates are cached (`_lastPipeX/Y`) so `getCurrentInputs()` and `setCurrentInputs()` use the same values the MLP sees. State is persisted to localStorage alongside existing app state.
-												feat(playground): implement Phases 2-4 of control surface spec

Phase 2 — Pinning + History:
- snapshot-stack.js: ring buffer (20 max) with auto-snapshot on
  train/randomize/thumbs-down, multi-level undo, tagged entries
- ab-compare.js: A/B weight state comparison with capture/toggle/accept/revert
- region-pin.js: pin rectangular input-space regions (Approach A: example
  pinning), pinned examples always included in training
- param-pin.js: per-output pin flags, pin mask skips pinned nodes in moveWeights
- phase2-ui.js: undo button with history popup, A/B toggle, long-press region
  pin, double-tap param pin
- Modified mlp.js/iml.js/nisps-wasm.js to accept outputPinMask in moveWeights

Phase 3 — Input Refinement + Exploration:
- pressure-feedback.js: touch force + hold duration → intensity multiplier
- auto-explore.js: automated thumbs-down at configurable interval, zoom-scaled
- input-heatmap.js: 16×16 MLP sampling, 3 color modes (luminance/variance/
  divergence), zoom-aware resampling, offscreen canvas rendering
- phase3-ui.js: auto-explore toggle with progress ring, heatmap eye icon,
  pressure indicators, settings drawer section
- joy-map-enhanced.js: added setHeatmap() for background layer rendering

Phase 4 — Output Pipeline + Visualization + Polish:
- output-pipeline.js: global curve → smoothing → slew rate → freeze gate
- weight-health.js: weight magnitude histogram, dead/saturating/healthy status
- gradient-flow.js: per-layer weight-delta analysis, vanishing/exploding detection
- session-presets.js: save/load full state, URL sharing via compact params
- phase4-ui.js: freeze button, network health panel, session preset UI

All phases merged into a-app.js with proper integration: auto-snapshots,
pressure-modulated RL, heatmap triggers, output pipeline in routeOutputs,
gradient capture around training, persistence for all new state.

											
										
										
											2026-03-26 09:48:12 +01:00
+								**Remaining**: Engine configuration panel (Part 8 of spec) — network architecture, loss function, optimizer selection.
-												feat(playground): implement Phase 1 control surface — compound axes, input pipeline, enhanced joy-map

Three new standalone ES modules + integration into a-app.js:

- input-pipeline.js: 5-stage processing (deadzone → zoom → curve → smoothing →
  momentum-as-zoom), 3 anchor modes, zoom-at-zero freeze, per-axis overrides
- control-surface.js: compound axes (Boldness/Memory/Precision) with interpolation
  tables, offset-based override resolution (trim-pot model), 6 built-in presets
- control-surface-ui.js: floating bar axis sliders, gear icon settings drawer with
  Input/Training/Exploration/Output sections, log-scale sliders, override dots
- joy-map-enhanced.js: zoom minimap with adaptive grid (4×4→32×32), vanishing trail
  with Catmull-Rom spline + tap-to-return, dual concentric noise rings, frozen overlay

Integration fixes from fresh-eyes review:
- getCurrentInputs()/setCurrentInputs() use cached pipeline coords (not raw)
- CSS noise ring hidden when canvas version active (no doubling)
- Input mode switch re-runs through pipeline
- Control surface state persisted to localStorage

Implements full Phase 1 of SPEC-controls.md plus bonus items from later phases
(zoom-aware feedback, control presets with override resolution, input curve/deadzone/
smoothing/momentum all wired).

											
										
										
											2026-03-26 09:24:24 +01:00
-												add CLAUDE.md for AI assistant context

											
										
										
											2026-02-08 15:49:16 +01:00
+								## Build System
 								This is an Arduino project targeting Raspberry Pi Pico. Build and upload using Arduino IDE or arduino-cli with the earlephilhower/pico board package.
 								```bash
 								# Initialize submodules (required for memllib and memlp)
 								git submodule update --init --recursive
 								# Build (adjust port as needed)
 								arduino-cli compile --fqbn rp2040:rp2040:rpipico -b 115200 MEMLNaut-NISPS.ino
 								arduino-cli upload --fqbn rp2040:rp2040:rpipico -p /dev/ttyACM0 MEMLNaut-NISPS.ino
 								```
 								## Architecture
 								### Dual-Core Design
 								The RP2040's dual cores are used for separation of concerns:
 								- **Core 0**: UI loop, ML inference, hardware interface polling (5ms period)
 								- **Core 1**: Real-time audio processing, parameter updates, MIDI polling
 								Inter-core synchronization uses memory barriers (`MEMORY_BARRIER()`, `WRITE_VOLATILE()`, `READ_VOLATILE()`) and RP2040 queues (`queue_t`).
 								### Mode System
 								The active mode is selected at compile-time via `#define MEMLNAUT_MODE_TYPE` in `MEMLNaut-NISPS.ino`. Modes implement the `MEMLNautMode` concept (see `modes/MEMLNautMode.hpp`):
 								| Mode | Purpose |
 								|------|---------|
 								| `MEMLNautModeChannelStrip` | Audio channel strip (EQ, compression, gain staging) |
 								| `MEMLNautModePAFSynth` | PAF (Phase Aligned Formant) synthesis with MIDI |
 								| `MEMLNautModeXIASRI` | Audio-reactive mode using machine listening analysis |
 								| `MEMLNautModeSoundAnalysisMIDI` | Sound analysis with MIDI output |
 								### Voice Spaces
 								Voice spaces map ML output parameters to audio engine parameters. They are defined as lambda functions that translate a normalized parameter array into synthesizer/processor settings. See `voicespaces/` for examples:
 								- PAF synth presets: `VoiceSpace1.hpp`, `VoiceSpaceQuadDetune.hpp`, etc.
 								- Channel strip presets: `voicespaces/ChannelStrip/basic.hpp` (Neve, SSL emulations)
 								### Key Components
 								- **IMLInterface** (`IMLInterface.hpp`): Interactive ML interface using an MLP for inference/training
 								- **InterfaceRL**: Reinforcement learning interface from memllib that handles joystick input and learning
 								- **AudioAppBase**: Template base class for audio applications
 								- **XiasriAnalysis**: Real-time audio feature extraction (pitch, aperiodicity, energy, brightness)
 								### Submodules (in `src/`)
 								- **memllib**: Hardware abstraction, audio drivers, synth components, RL interfaces
 								- **memlp**: MLP (Multi-Layer Perceptron) implementation for embedded ML
 								- **daisysp**: DSP library (filters, drums, effects, synthesis)
 								## Memory Sections
 								The codebase uses RP2040-specific memory placement:
 								- `AUDIO_MEM` / `AUDIO_FUNC`: Place audio-critical code/data in SRAM
 								- `APP_SRAM` / `__not_in_flash("app")`: Keep frequently-accessed data out of flash
 								## Audio Parameters
 								Sample rate is defined in `AudioDriver::GetSampleRate()`. The audio callback `audio_block_callback` runs on Core 1 and processes stereo audio (`stereosample_t`).