memlnaut-nisps/CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Overview

MEMLNaut-NISPS (Neural Interactive Shaping of Parameter Spaces) is firmware for the MEMLNaut hardware platform - a custom embedded audio device built on Raspberry Pi Pico (RP2040). It implements interactive machine learning for real-time audio synthesis and processing, enabling users to shape sound parameters through reinforcement learning.

Project documentation: https://musicallyembodiedml.github.io/memlnaut/approaches/nisps

NISPS Core Library

The nisps-core/ directory contains a platform-agnostic C++20 extraction of the interactive ML engine. This header-only library can be used in any C++ project for neural network-based parameter mapping.

Key differences from firmware:

• ✅ Platform-agnostic (no Arduino/RP2040 dependencies)
• ✅ Header-only (just include and use)
• ✅ C++20 (uses std::span)
• ✅ Namespaced (nisps::)
• ❌ No audio synthesis (use it to control your synth)
• ❌ No hardware drivers

Use case: Control synthesizers, effects, lights, game parameters, or any system that responds to continuous parameters.

See nisps-core/README.md for complete documentation and examples.

Web Playground

The playground/ directory contains a browser-based interactive demo of the NISPS ML engine. It's a faithful JavaScript port of nisps-core's MLP + IML, with no build step or dependencies.

• 2 inputs (virtual joystick X/Y) mapped through a [3, 32, 48, 64, 126] MLP to 126 outputs
• Two output modes:
  • Visual: first 20 outputs control a Canvas2D flow-field particle system
  • Synth (C15): all 126 outputs control the C15 WASM synthesizer — every sonically meaningful continuous parameter across envelopes, oscillators, shapers, filters, feedback/output mixers, cabinet, and effects
• Two learning modes: Examples (set slider targets, add examples, train) and RL Feedback (thumbs up/down with exploration noise)
• Serve statically: cd playground && python3 -m http.server
• Mobile-first: designed for touch/foldable phone use

Key files: js/nisps/ (ML core port), js/ui/ (visualizer, joystick, controls, input pipeline, control surface), js/synth/ (C15 bridge, param map, arpeggiator), js/a-app.js (immersive app wiring).

URL Parameters

Param	Range	Default	Effect
tame	0–1	1	Constrains synth output ranges toward safe limits
spread	0–1	0.6	Controls weight initialization, RL noise scaling, and weight decay (see below)
preset	preset id	(none)	Auto-loads a synth parameter preset on first visit (e.g. ?preset=beginner-1)

spread — sigmoid saturation control

The MLP uses ReLU hidden layers with a sigmoid output layer. With uniform [-1,1] weights, the sum of many weighted inputs at each layer drives sigmoid pre-activations far from zero (std dev ≈ √fan_in), causing outputs to saturate near 0 or 1. The spread parameter addresses this:

• spread=0 (polarised): Weights drawn from uniform [-1,1]. RL noise cap = 0.3. Noise applied uniformly across layers. Outputs cluster at extremes — good for exploration of radical mappings.
• spread=1 (centered): Weights scaled by 1/√fan_in per layer (Xavier initialization). RL noise cap = 0.05. Noise also scaled per-layer. Weight decay prevents magnitude drift. Outputs spread across the full [0,1] range — better for fine-grained RL shaping.
• Intermediate values interpolate linearly between these two regimes.

Affects four code paths:

1. drawWeights(spread) — initial randomisation weight scale
2. moveWeights(speed, spread) — RL exploration noise scale per layer
3. Weight decay in moveWeights — each call decays weights by 10% * spread before adding noise, preventing unbounded magnitude drift from repeated thumbs-down. At spread=0 there is no decay (original behavior). At spread=1, weights decay ~10% per call, creating a natural equilibrium where exploration noise and decay balance out rather than weights growing until sigmoid permanently saturates.
4. Noise cap in thumbs-down handler — 0.3*(1-spread) + 0.05*spread

C15 Parameter Map

The 126 synth parameters in js/synth/param-map.js were curated from the C15's 287 total parameters. Excluded categories:

Excluded	Count	Reason
Hardware Amount/Source	56	No physical MIDI hardware in browser
Macro Controls/Times	12	Meta-routing layer conflicts with direct ML control
Scale offsets	13	Microtuning would break pitch unpredictably
Key tracking (*_KT)	11	Pitch-dependent scaling needs calibrated defaults
Velocity (*_Vel)	11	Velocity-dependent, ML can't observe key velocity
Envelope mod depths (*_Env_A/B/C)	19	Multiplicative interaction with envelope shapes makes space too hard to learn
Discrete/structural	15	Osc Pitch (full sweep), Master Vol/Tune, Voice Mute/Fade, Unison Voices, Mono modes, Split, Osc Reset
Secondary config	7	Att Curve, Elevate, Chirp, Decay Gate, Retrigger
PM shaper blend	4	Secondary routing params
FB Mix source selects	4	Discrete A/B selectors

Synth Presets

Presets (js/synth/presets.js) control which parameters the ML engine can modify, with unselected params muted at safe defaults. Each preset defines per-param { muted, fixedValue, min, max, curve } — no training examples or model weights.

4 tiers of progressive complexity:

Tier	Presets	Active params	What's exposed
1 (Beginner)	1.1–1.4	15	Basic ADSR, SVF cutoff/res, Shaper A drive/fold, output levels, reverb mix
2 (Intermediate)	2.1–2.4	40	+ Env B/C, filter FM, effects (reverb/echo/flanger), cabinet, stereo panning
3 (Advanced)	3.1–3.3	~95	+ Cross-oscillator PM, feedback mixer, dual shapers, comb/gap filters, ring mod
4 (Expert)	4.1–4.2	126	Full engine

Presets use curve values to bias parameter distributions (< 0.5 = spend more time low, > 0.5 = bias high) without clamping extremes. Users can tweak any preset via the group drawer after loading.

Control Surface (Phase 1)

The immersive app (a-immersive.html) has a control surface system for tuning how exploration and learning feel. Full spec: playground/SPEC-controls.md.

Architecture — modular ES modules organized by phase, wired into a-app.js:

Module	Phase	Purpose
js/ui/input-pipeline.js	1	Processes raw joystick input through deadzone → zoom → curve → smoothing → momentum-as-zoom. Pure math, no DOM.
js/ui/control-surface.js	1	Compound axes (Boldness, Memory, Precision) that map single sliders to multiple underlying params. Offset-based override resolution (trim-pot model). 6 built-in control presets.
js/ui/control-surface-ui.js	1	DOM layer: 3 axis sliders on floating bar, gear icon settings drawer with per-param overrides. Injects its own CSS.
js/ui/joy-map-enhanced.js	1	Enhanced joy-map canvas: zoom minimap with adaptive grid, vanishing trail with Catmull-Rom spline and tap-to-return, dual concentric noise rings, frozen state overlay.
js/ui/snapshot-stack.js	2	Ring buffer (20 max) of weight snapshots. Auto-snapshot on train/randomize/thumbs-down. Multi-level undo.
js/ui/ab-compare.js	2	Rapid A/B weight state comparison. Capture, toggle, accept or revert.
js/ui/region-pin.js	2	Pins rectangular input-space regions (Approach A: example pinning). Pinned examples always included in training.
js/ui/param-pin.js	2	Per-output pin flags. Pin mask passed to moveWeights() to skip pinned output nodes.
js/ui/phase2-ui.js	2	DOM: undo button with history popup, A/B toggle, region pin via long-press, param pin via double-tap.
js/ui/pressure-feedback.js	3	Touch force + hold duration → intensity multiplier for noise growth/decay.
js/ui/auto-explore.js	3	Automated thumbs-down at configurable interval. Zoom-scaled intensity.
js/ui/input-heatmap.js	3	2D color field sampling MLP across input space. 3 color modes, zoom-aware resampling.
js/ui/phase3-ui.js	3	DOM: auto-explore toggle with progress ring, heatmap eye icon, pressure indicators.
js/ui/output-pipeline.js	4	Global curve → smoothing → slew rate → freeze gate on MLP outputs before synth/visual routing.
js/ui/weight-health.js	4	Weight magnitude histogram, dead/saturating/healthy status detection, ambient visualization.
js/ui/gradient-flow.js	4	Per-layer weight-delta analysis after training. Vanishing/exploding/converged detection.
js/ui/session-presets.js	4	Save/load full session state. URL sharing via compact params.
js/ui/phase4-ui.js	4	DOM: freeze button, network health panel, session preset UI, output pipeline slider wiring.

Compound Axes — each controls 4-6 underlying parameters via interpolation tables:

• Boldness (Caution ↔ Bold): input zoom, noise cap, noise growth, learning rate, weight decay, noise distribution
• Memory (Amnesia ↔ Elephant): max examples, example decay, weight decay, noise decay, convergence threshold
• Precision (Raw ↔ Precise): input curve, deadzone, smoothing, slew rate, momentum-zoom mode

When a user manually overrides an individual param, the offset from the axis-derived value persists as the axis moves (like a trim pot on a mixing desk). Double-tap an axis to re-link all params.

Input Pipeline — sits between physical joystick and MLP. Key feature: zoom narrows the effective input window around an anchor point (effective = anchor + (raw - 0.5) * zoom_level). Zoom-at-zero freezes input. Three anchor modes: auto (anchor follows current position when zoom changes), sticky (explicit anchor), center (always 0.5).

Control Presets: Default, First Touch, Jazz Hands, Sculptor, Improviser, Microscope. These set compound axis positions — they don't include network weights or synth preset selection.

Integration — the control surface dispatches controlsurface:change CustomEvents. a-app.js listens and updates the input pipeline config, spread level, and RL parameters (noise cap, growth, decay, floor, zoom-aware feedback scaling). Pipeline-processed coordinates are cached (_lastPipeX/Y) so getCurrentInputs() and setCurrentInputs() use the same values the MLP sees. State is persisted to localStorage alongside existing app state.

Remaining: Engine configuration panel (Part 8 of spec) — network architecture, loss function, optimizer selection.

Build System

This is an Arduino project targeting Raspberry Pi Pico. Build and upload using Arduino IDE or arduino-cli with the earlephilhower/pico board package.

# Initialize submodules (required for memllib and memlp)
git submodule update --init --recursive

# Build (adjust port as needed)
arduino-cli compile --fqbn rp2040:rp2040:rpipico -b 115200 MEMLNaut-NISPS.ino
arduino-cli upload --fqbn rp2040:rp2040:rpipico -p /dev/ttyACM0 MEMLNaut-NISPS.ino

Architecture

Dual-Core Design

The RP2040's dual cores are used for separation of concerns:

• Core 0: UI loop, ML inference, hardware interface polling (5ms period)
• Core 1: Real-time audio processing, parameter updates, MIDI polling

Inter-core synchronization uses memory barriers (MEMORY_BARRIER(), WRITE_VOLATILE(), READ_VOLATILE()) and RP2040 queues (queue_t).

Mode System

The active mode is selected at compile-time via #define MEMLNAUT_MODE_TYPE in MEMLNaut-NISPS.ino. Modes implement the MEMLNautMode concept (see modes/MEMLNautMode.hpp):

Mode	Purpose
MEMLNautModeChannelStrip	Audio channel strip (EQ, compression, gain staging)
MEMLNautModePAFSynth	PAF (Phase Aligned Formant) synthesis with MIDI
MEMLNautModeXIASRI	Audio-reactive mode using machine listening analysis
MEMLNautModeSoundAnalysisMIDI	Sound analysis with MIDI output

Voice Spaces

Voice spaces map ML output parameters to audio engine parameters. They are defined as lambda functions that translate a normalized parameter array into synthesizer/processor settings. See voicespaces/ for examples:

• PAF synth presets: VoiceSpace1.hpp, VoiceSpaceQuadDetune.hpp, etc.
• Channel strip presets: voicespaces/ChannelStrip/basic.hpp (Neve, SSL emulations)

Key Components

• IMLInterface (IMLInterface.hpp): Interactive ML interface using an MLP for inference/training
• InterfaceRL: Reinforcement learning interface from memllib that handles joystick input and learning
• AudioAppBase: Template base class for audio applications
• XiasriAnalysis: Real-time audio feature extraction (pitch, aperiodicity, energy, brightness)

Submodules (in src/)

• memllib: Hardware abstraction, audio drivers, synth components, RL interfaces
• memlp: MLP (Multi-Layer Perceptron) implementation for embedded ML
• daisysp: DSP library (filters, drums, effects, synthesis)

Memory Sections

The codebase uses RP2040-specific memory placement:

• AUDIO_MEM / AUDIO_FUNC: Place audio-critical code/data in SRAM
• APP_SRAM / __not_in_flash("app"): Keep frequently-accessed data out of flash

Audio Parameters

Sample rate is defined in AudioDriver::GetSampleRate(). The audio callback audio_block_callback runs on Core 1 and processes stereo audio (stereosample_t).