memlnaut-nisps/CLAUDE.md

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Overview

MEMLNaut-NISPS (Neural Interactive Shaping of Parameter Spaces) is firmware for the MEMLNaut hardware platform - a custom embedded audio device built on Raspberry Pi Pico (RP2040). It implements interactive machine learning for real-time audio synthesis and processing, enabling users to shape sound parameters through reinforcement learning.

Project documentation: https://musicallyembodiedml.github.io/memlnaut/approaches/nisps

## NISPS Core Library

The `nisps-core/` directory contains a platform-agnostic C++20 extraction of the interactive ML engine. This header-only library can be used in any C++ project for neural network-based parameter mapping.

**Key differences from firmware**:
- ✅ Platform-agnostic (no Arduino/RP2040 dependencies)
- ✅ Header-only (just include and use)
- ✅ C++20 (uses std::span)
- ✅ Namespaced (`nisps::`)
- ❌ No audio synthesis (use it to *control* your synth)
- ❌ No hardware drivers

**Use case**: Control synthesizers, effects, lights, game parameters, or any system that responds to continuous parameters.

See `nisps-core/README.md` for complete documentation and examples.

## Web Playground

The `playground/` directory contains a browser-based interactive demo of the NISPS ML engine. It's a faithful JavaScript port of nisps-core's MLP + IML, with no build step or dependencies.

- **2 inputs** (virtual joystick X/Y) mapped through a `[3, 32, 48, 64, 126]` MLP to **126 outputs**
- **Two output modes**:
  - **Visual**: first 20 outputs control a Canvas2D flow-field particle system
  - **Synth (C15)**: all 126 outputs control the C15 WASM synthesizer — every sonically meaningful continuous parameter across envelopes, oscillators, shapers, filters, feedback/output mixers, cabinet, and effects
- **Two learning modes**: Examples (set slider targets, add examples, train) and RL Feedback (thumbs up/down with exploration noise)
- **Serve statically**: `cd playground && python3 -m http.server`
- **Mobile-first**: designed for touch/foldable phone use

Key files: `js/nisps/` (ML core port), `js/ui/` (visualizer, joystick, controls), `js/synth/` (C15 bridge, param map, arpeggiator), `js/app.js` (wiring).

### URL Parameters

| Param | Range | Default | Effect |
|-------|-------|---------|--------|
| `tame` | 0–1 | 1 | Constrains synth output ranges toward safe limits |
| `spread` | 0–1 | 0.6 | Controls weight initialization, RL noise scaling, and weight decay (see below) |

#### `spread` — sigmoid saturation control

The MLP uses ReLU hidden layers with a sigmoid output layer. With uniform [-1,1] weights, the sum of many weighted inputs at each layer drives sigmoid pre-activations far from zero (std dev ≈ √fan_in), causing outputs to saturate near 0 or 1. The `spread` parameter addresses this:

- **`spread=0`** (polarised): Weights drawn from uniform [-1,1]. RL noise cap = 0.3. Noise applied uniformly across layers. Outputs cluster at extremes — good for exploration of radical mappings.
- **`spread=1`** (centered): Weights scaled by 1/√fan_in per layer (Xavier initialization). RL noise cap = 0.05. Noise also scaled per-layer. Weight decay prevents magnitude drift. Outputs spread across the full [0,1] range — better for fine-grained RL shaping.
- **Intermediate values** interpolate linearly between these two regimes.

Affects four code paths:
1. **`drawWeights(spread)`** — initial randomisation weight scale
2. **`moveWeights(speed, spread)`** — RL exploration noise scale per layer
3. **Weight decay in `moveWeights`** — each call decays weights by `10% * spread` before adding noise, preventing unbounded magnitude drift from repeated thumbs-down. At spread=0 there is no decay (original behavior). At spread=1, weights decay ~10% per call, creating a natural equilibrium where exploration noise and decay balance out rather than weights growing until sigmoid permanently saturates.
4. **Noise cap** in thumbs-down handler — `0.3*(1-spread) + 0.05*spread`

### C15 Parameter Map

The 126 synth parameters in `js/synth/param-map.js` were curated from the C15's 287 total parameters. Excluded categories:

| Excluded | Count | Reason |
|----------|-------|--------|
| Hardware Amount/Source | 56 | No physical MIDI hardware in browser |
| Macro Controls/Times | 12 | Meta-routing layer conflicts with direct ML control |
| Scale offsets | 13 | Microtuning would break pitch unpredictably |
| Key tracking (`*_KT`) | 11 | Pitch-dependent scaling needs calibrated defaults |
| Velocity (`*_Vel`) | 11 | Velocity-dependent, ML can't observe key velocity |
| Envelope mod depths (`*_Env_A/B/C`) | 19 | Multiplicative interaction with envelope shapes makes space too hard to learn |
| Discrete/structural | 15 | Osc Pitch (full sweep), Master Vol/Tune, Voice Mute/Fade, Unison Voices, Mono modes, Split, Osc Reset |
| Secondary config | 7 | Att Curve, Elevate, Chirp, Decay Gate, Retrigger |
| PM shaper blend | 4 | Secondary routing params |
| FB Mix source selects | 4 | Discrete A/B selectors |

## Build System

This is an Arduino project targeting Raspberry Pi Pico. Build and upload using Arduino IDE or arduino-cli with the earlephilhower/pico board package.

```bash
# Initialize submodules (required for memllib and memlp)
git submodule update --init --recursive

# Build (adjust port as needed)
arduino-cli compile --fqbn rp2040:rp2040:rpipico -b 115200 MEMLNaut-NISPS.ino
arduino-cli upload --fqbn rp2040:rp2040:rpipico -p /dev/ttyACM0 MEMLNaut-NISPS.ino
```

## Architecture

### Dual-Core Design

The RP2040's dual cores are used for separation of concerns:
- **Core 0**: UI loop, ML inference, hardware interface polling (5ms period)
- **Core 1**: Real-time audio processing, parameter updates, MIDI polling

Inter-core synchronization uses memory barriers (`MEMORY_BARRIER()`, `WRITE_VOLATILE()`, `READ_VOLATILE()`) and RP2040 queues (`queue_t`).

### Mode System

The active mode is selected at compile-time via `#define MEMLNAUT_MODE_TYPE` in `MEMLNaut-NISPS.ino`. Modes implement the `MEMLNautMode` concept (see `modes/MEMLNautMode.hpp`):

| Mode | Purpose |
|------|---------|
| `MEMLNautModeChannelStrip` | Audio channel strip (EQ, compression, gain staging) |
| `MEMLNautModePAFSynth` | PAF (Phase Aligned Formant) synthesis with MIDI |
| `MEMLNautModeXIASRI` | Audio-reactive mode using machine listening analysis |
| `MEMLNautModeSoundAnalysisMIDI` | Sound analysis with MIDI output |

### Voice Spaces

Voice spaces map ML output parameters to audio engine parameters. They are defined as lambda functions that translate a normalized parameter array into synthesizer/processor settings. See `voicespaces/` for examples:
- PAF synth presets: `VoiceSpace1.hpp`, `VoiceSpaceQuadDetune.hpp`, etc.
- Channel strip presets: `voicespaces/ChannelStrip/basic.hpp` (Neve, SSL emulations)

### Key Components

- **IMLInterface** (`IMLInterface.hpp`): Interactive ML interface using an MLP for inference/training
- **InterfaceRL**: Reinforcement learning interface from memllib that handles joystick input and learning
- **AudioAppBase**: Template base class for audio applications
- **XiasriAnalysis**: Real-time audio feature extraction (pitch, aperiodicity, energy, brightness)

### Submodules (in `src/`)

- **memllib**: Hardware abstraction, audio drivers, synth components, RL interfaces
- **memlp**: MLP (Multi-Layer Perceptron) implementation for embedded ML
- **daisysp**: DSP library (filters, drums, effects, synthesis)

## Memory Sections

The codebase uses RP2040-specific memory placement:
- `AUDIO_MEM` / `AUDIO_FUNC`: Place audio-critical code/data in SRAM
- `APP_SRAM` / `__not_in_flash("app")`: Keep frequently-accessed data out of flash

## Audio Parameters

Sample rate is defined in `AudioDriver::GetSampleRate()`. The audio callback `audio_block_callback` runs on Core 1 and processes stereo audio (`stereosample_t`).
-												add CLAUDE.md for AI assistant context

											
										
										
											2026-02-08 15:49:16 +01:00
+								# CLAUDE.md
 								This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
 								## Overview
 								MEMLNaut-NISPS (Neural Interactive Shaping of Parameter Spaces) is firmware for the MEMLNaut hardware platform - a custom embedded audio device built on Raspberry Pi Pico (RP2040). It implements interactive machine learning for real-time audio synthesis and processing, enabling users to shape sound parameters through reinforcement learning.
 								Project documentation: https://musicallyembodiedml.github.io/memlnaut/approaches/nisps
-												feat: extract nisps-core platform-agnostic ML library

Extract the interactive machine learning engine from MEMLNaut-NISPS
firmware into a standalone, platform-agnostic C++20 header-only library.

What is nisps-core?
-------------------
NISPS (Neural Interactive Shaping of Parameter Spaces) core is a
parameter mapping engine. It takes N input parameters (joystick,
sensors, audio features) and maps them to M output parameters through
an interactively-trained neural network.

Use it to control: synthesizers, effects, lights, robots, game
parameters, or anything that responds to continuous control data.

Key Features
------------
- Header-only: No compilation needed, just include and use
- Platform-agnostic: Pure C++20, works anywhere
- Zero dependencies: Only standard library
- Interactive learning: Train by demonstration
- Lightweight: ~3,500 lines of optimized neural network code
- Flexible: Map 1-100 inputs to 1-100 outputs

Architecture
------------
Core components:
- IML: High-level interactive ML interface
- MLP: Multi-layer perceptron (feedforward neural network)
- Dataset: Training data management with replay memory
- Layer/Node: Neural network building blocks
- Loss: MSE and categorical cross-entropy functions
- Utils: Activation functions (sigmoid, ReLU, tanh, etc.)

Transformations Applied
-----------------------
✅ Removed Arduino/RP2040 dependencies (Serial, SD, Pico SDK)
✅ Removed audio synthesis code (nisps-core is control-only)
✅ Added nisps namespace to all code
✅ Converted to header-only library with _impl.hpp pattern
✅ Updated to C++20 (required for std::span)
✅ Removed platform-specific serialization
✅ Replaced debug macros with no-op stubs
✅ Added comprehensive documentation and examples

Files Added
-----------
- nisps-core/README.md: Complete documentation and API reference
- nisps-core/CHANGELOG.md: Version history and migration guide
- nisps-core/include/nisps/*.hpp: 13 header files (~3,500 lines)
- nisps-core/test/main.cpp: XOR test demonstrating basic usage
- nisps-core/examples/simple_mapping.cpp: Interactive demo
- nisps-core/CMakeLists.txt: Build system for tests

Testing
-------
✅ Compiles with GCC 14.2 (C++20)
✅ All tests passing
✅ Successfully instantiates networks and runs inference

Performance
-----------
- Inference: 1-10 µs for small networks (2-10-10-4)
- Training: 10-100 ms for 100 examples, 1000 iterations
- Memory: ~1 KB per hidden neuron

Migration from Embedded IMLInterface
------------------------------------
Old (embedded):
  IMLInterface iml(n_inputs, n_outputs);

New (nisps-core):
  nisps::IML<float> iml(n_inputs, n_outputs);

All method names remain the same, just add the namespace.

Related
-------
- Implements: NISPS_CORE_EXTRACTION_PLAN.md
- Task graph: NISPS_CORE_TASKS.md
- Origin: MEMLNaut-NISPS firmware
- Docs: https://musicallyembodiedml.github.io/memlnaut/

Co-authored-by: Claude Code <claude@anthropic.com>

											
										
										
											2026-02-08 17:47:23 +01:00
+								## NISPS Core Library
 								The `nisps-core/` directory contains a platform-agnostic C++20 extraction of the interactive ML engine. This header-only library can be used in any C++ project for neural network-based parameter mapping.
 								**Key differences from firmware**:
 								- ✅ Platform-agnostic (no Arduino/RP2040 dependencies)
 								- ✅ Header-only (just include and use)
 								- ✅ C++20 (uses std::span)
 								- ✅ Namespaced (`nisps::`)
 								- ❌ No audio synthesis (use it to *control* your synth)
 								- ❌ No hardware drivers
 								**Use case**: Control synthesizers, effects, lights, game parameters, or any system that responds to continuous parameters.
 								See `nisps-core/README.md` for complete documentation and examples.
-												feat: add web-based interactive playground for NISPS

Faithful JS port of nisps-core MLP + IML engine with a touch-friendly
UI for exploring neural parameter mapping visually. Two learning modes:
example-based (set slider targets) and RL feedback (thumbs up/down with
exploration noise). Flow field particle system controlled by 8 MLP outputs.

											
										
										
											2026-02-11 13:17:17 +01:00
+								## Web Playground
 								The `playground/` directory contains a browser-based interactive demo of the NISPS ML engine. It's a faithful JavaScript port of nisps-core's MLP + IML, with no build step or dependencies.
-												docs: update CLAUDE.md and playground README for 126-param C15 map

Document the expanded parameter map, MLP architecture, exclusion
rationale table, and synth mode in both project-level and playground docs.

											
										
										
											2026-03-21 22:11:45 +01:00
+								- **2 inputs** (virtual joystick X/Y) mapped through a `[3, 32, 48, 64, 126]` MLP to **126 outputs**
 								- **Two output modes**:
 								  - **Visual**: first 20 outputs control a Canvas2D flow-field particle system
 								  - **Synth (C15)**: all 126 outputs control the C15 WASM synthesizer — every sonically meaningful continuous parameter across envelopes, oscillators, shapers, filters, feedback/output mixers, cabinet, and effects
-												feat: add web-based interactive playground for NISPS

Faithful JS port of nisps-core MLP + IML engine with a touch-friendly
UI for exploring neural parameter mapping visually. Two learning modes:
example-based (set slider targets) and RL feedback (thumbs up/down with
exploration noise). Flow field particle system controlled by 8 MLP outputs.

											
										
										
											2026-02-11 13:17:17 +01:00
+								- **Two learning modes**: Examples (set slider targets, add examples, train) and RL Feedback (thumbs up/down with exploration noise)
 								- **Serve statically**: `cd playground && python3 -m http.server`
 								- **Mobile-first**: designed for touch/foldable phone use
-												docs: update CLAUDE.md and playground README for 126-param C15 map

Document the expanded parameter map, MLP architecture, exclusion
rationale table, and synth mode in both project-level and playground docs.

											
										
										
											2026-03-21 22:11:45 +01:00
+								Key files: `js/nisps/` (ML core port), `js/ui/` (visualizer, joystick, controls), `js/synth/` (C15 bridge, param map, arpeggiator), `js/app.js` (wiring).
-												feat(playground): add spread param for sigmoid saturation control and fix randomise sync

Add ?spread=0-1 URL param that controls weight initialization scaling,
RL noise scaling per layer, noise cap, and weight decay to prevent
sigmoid output saturation. At spread=0 (original behavior) weights are
uniform [-1,1] and outputs polarise near 0/1. At spread=1 weights use
Xavier scaling (1/sqrt(fan_in)), noise is proportionally reduced, and
10% weight decay per thumbs-down prevents unbounded magnitude drift.

Also fix randomise to re-inject current joystick position and re-run
inference before routing outputs, eliminating the jump on first
joystick move after randomise.

Defaults: tame=1, spread=0.6 across all app variants.

											
										
										
											2026-03-22 01:10:26 +01:00
+								### URL Parameters
 								| Param | Range | Default | Effect |
 								|-------|-------|---------|--------|
 								| `tame` | 0–1 | 1 | Constrains synth output ranges toward safe limits |
 								| `spread` | 0–1 | 0.6 | Controls weight initialization, RL noise scaling, and weight decay (see below) |
 								#### `spread` — sigmoid saturation control
 								The MLP uses ReLU hidden layers with a sigmoid output layer. With uniform [-1,1] weights, the sum of many weighted inputs at each layer drives sigmoid pre-activations far from zero (std dev ≈ √fan_in), causing outputs to saturate near 0 or 1. The `spread` parameter addresses this:
 								- **`spread=0`** (polarised): Weights drawn from uniform [-1,1]. RL noise cap = 0.3. Noise applied uniformly across layers. Outputs cluster at extremes — good for exploration of radical mappings.
 								- **`spread=1`** (centered): Weights scaled by 1/√fan_in per layer (Xavier initialization). RL noise cap = 0.05. Noise also scaled per-layer. Weight decay prevents magnitude drift. Outputs spread across the full [0,1] range — better for fine-grained RL shaping.
 								- **Intermediate values** interpolate linearly between these two regimes.
 								Affects four code paths:
 . **`drawWeights(spread)`** — initial randomisation weight scale
 . **`moveWeights(speed, spread)`** — RL exploration noise scale per layer
 . **Weight decay in `moveWeights`** — each call decays weights by `10% * spread` before adding noise, preventing unbounded magnitude drift from repeated thumbs-down. At spread=0 there is no decay (original behavior). At spread=1, weights decay ~10% per call, creating a natural equilibrium where exploration noise and decay balance out rather than weights growing until sigmoid permanently saturates.
 . **Noise cap** in thumbs-down handler — `0.3*(1-spread) + 0.05*spread`
-												docs: update CLAUDE.md and playground README for 126-param C15 map

Document the expanded parameter map, MLP architecture, exclusion
rationale table, and synth mode in both project-level and playground docs.

											
										
										
											2026-03-21 22:11:45 +01:00
+								### C15 Parameter Map
 								The 126 synth parameters in `js/synth/param-map.js` were curated from the C15's 287 total parameters. Excluded categories:
 								| Excluded | Count | Reason |
 								|----------|-------|--------|
 								| Hardware Amount/Source | 56 | No physical MIDI hardware in browser |
 								| Macro Controls/Times | 12 | Meta-routing layer conflicts with direct ML control |
 								| Scale offsets | 13 | Microtuning would break pitch unpredictably |
 								| Key tracking (`*_KT`) | 11 | Pitch-dependent scaling needs calibrated defaults |
 								| Velocity (`*_Vel`) | 11 | Velocity-dependent, ML can't observe key velocity |
 								| Envelope mod depths (`*_Env_A/B/C`) | 19 | Multiplicative interaction with envelope shapes makes space too hard to learn |
 								| Discrete/structural | 15 | Osc Pitch (full sweep), Master Vol/Tune, Voice Mute/Fade, Unison Voices, Mono modes, Split, Osc Reset |
 								| Secondary config | 7 | Att Curve, Elevate, Chirp, Decay Gate, Retrigger |
 								| PM shaper blend | 4 | Secondary routing params |
 								| FB Mix source selects | 4 | Discrete A/B selectors |
-												feat: add web-based interactive playground for NISPS

Faithful JS port of nisps-core MLP + IML engine with a touch-friendly
UI for exploring neural parameter mapping visually. Two learning modes:
example-based (set slider targets) and RL feedback (thumbs up/down with
exploration noise). Flow field particle system controlled by 8 MLP outputs.

											
										
										
											2026-02-11 13:17:17 +01:00
-												add CLAUDE.md for AI assistant context

											
										
										
											2026-02-08 15:49:16 +01:00
+								## Build System
 								This is an Arduino project targeting Raspberry Pi Pico. Build and upload using Arduino IDE or arduino-cli with the earlephilhower/pico board package.
 								```bash
 								# Initialize submodules (required for memllib and memlp)
 								git submodule update --init --recursive
 								# Build (adjust port as needed)
 								arduino-cli compile --fqbn rp2040:rp2040:rpipico -b 115200 MEMLNaut-NISPS.ino
 								arduino-cli upload --fqbn rp2040:rp2040:rpipico -p /dev/ttyACM0 MEMLNaut-NISPS.ino
 								```
 								## Architecture
 								### Dual-Core Design
 								The RP2040's dual cores are used for separation of concerns:
 								- **Core 0**: UI loop, ML inference, hardware interface polling (5ms period)
 								- **Core 1**: Real-time audio processing, parameter updates, MIDI polling
 								Inter-core synchronization uses memory barriers (`MEMORY_BARRIER()`, `WRITE_VOLATILE()`, `READ_VOLATILE()`) and RP2040 queues (`queue_t`).
 								### Mode System
 								The active mode is selected at compile-time via `#define MEMLNAUT_MODE_TYPE` in `MEMLNaut-NISPS.ino`. Modes implement the `MEMLNautMode` concept (see `modes/MEMLNautMode.hpp`):
 								| Mode | Purpose |
 								|------|---------|
 								| `MEMLNautModeChannelStrip` | Audio channel strip (EQ, compression, gain staging) |
 								| `MEMLNautModePAFSynth` | PAF (Phase Aligned Formant) synthesis with MIDI |
 								| `MEMLNautModeXIASRI` | Audio-reactive mode using machine listening analysis |
 								| `MEMLNautModeSoundAnalysisMIDI` | Sound analysis with MIDI output |
 								### Voice Spaces
 								Voice spaces map ML output parameters to audio engine parameters. They are defined as lambda functions that translate a normalized parameter array into synthesizer/processor settings. See `voicespaces/` for examples:
 								- PAF synth presets: `VoiceSpace1.hpp`, `VoiceSpaceQuadDetune.hpp`, etc.
 								- Channel strip presets: `voicespaces/ChannelStrip/basic.hpp` (Neve, SSL emulations)
 								### Key Components
 								- **IMLInterface** (`IMLInterface.hpp`): Interactive ML interface using an MLP for inference/training
 								- **InterfaceRL**: Reinforcement learning interface from memllib that handles joystick input and learning
 								- **AudioAppBase**: Template base class for audio applications
 								- **XiasriAnalysis**: Real-time audio feature extraction (pitch, aperiodicity, energy, brightness)
 								### Submodules (in `src/`)
 								- **memllib**: Hardware abstraction, audio drivers, synth components, RL interfaces
 								- **memlp**: MLP (Multi-Layer Perceptron) implementation for embedded ML
 								- **daisysp**: DSP library (filters, drums, effects, synthesis)
 								## Memory Sections
 								The codebase uses RP2040-specific memory placement:
 								- `AUDIO_MEM` / `AUDIO_FUNC`: Place audio-critical code/data in SRAM
 								- `APP_SRAM` / `__not_in_flash("app")`: Keep frequently-accessed data out of flash
 								## Audio Parameters
 								Sample rate is defined in `AudioDriver::GetSampleRate()`. The audio callback `audio_block_callback` runs on Core 1 and processes stereo audio (`stereosample_t`).