memlnaut-nisps/CLAUDE.md

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Overview

MEMLNaut-NISPS (Neural Interactive Shaping of Parameter Spaces) is firmware for the MEMLNaut hardware platform - a custom embedded audio device built on Raspberry Pi Pico (RP2040). It implements interactive machine learning for real-time audio synthesis and processing, enabling users to shape sound parameters through reinforcement learning.

Project documentation: https://musicallyembodiedml.github.io/memlnaut/approaches/nisps

## NISPS Core Library

The `nisps-core/` directory contains a platform-agnostic C++20 extraction of the interactive ML engine. This header-only library can be used in any C++ project for neural network-based parameter mapping.

**Key differences from firmware**:
- ✅ Platform-agnostic (no Arduino/RP2040 dependencies)
- ✅ Header-only (just include and use)
- ✅ C++20 (uses std::span)
- ✅ Namespaced (`nisps::`)
- ❌ No audio synthesis (use it to *control* your synth)
- ❌ No hardware drivers

**Use case**: Control synthesizers, effects, lights, game parameters, or any system that responds to continuous parameters.

See `nisps-core/README.md` for complete documentation and examples.

## Build System

This is an Arduino project targeting Raspberry Pi Pico. Build and upload using Arduino IDE or arduino-cli with the earlephilhower/pico board package.

```bash
# Initialize submodules (required for memllib and memlp)
git submodule update --init --recursive

# Build (adjust port as needed)
arduino-cli compile --fqbn rp2040:rp2040:rpipico -b 115200 MEMLNaut-NISPS.ino
arduino-cli upload --fqbn rp2040:rp2040:rpipico -p /dev/ttyACM0 MEMLNaut-NISPS.ino
```

## Architecture

### Dual-Core Design

The RP2040's dual cores are used for separation of concerns:
- **Core 0**: UI loop, ML inference, hardware interface polling (5ms period)
- **Core 1**: Real-time audio processing, parameter updates, MIDI polling

Inter-core synchronization uses memory barriers (`MEMORY_BARRIER()`, `WRITE_VOLATILE()`, `READ_VOLATILE()`) and RP2040 queues (`queue_t`).

### Mode System

The active mode is selected at compile-time via `#define MEMLNAUT_MODE_TYPE` in `MEMLNaut-NISPS.ino`. Modes implement the `MEMLNautMode` concept (see `modes/MEMLNautMode.hpp`):

| Mode | Purpose |
|------|---------|
| `MEMLNautModeChannelStrip` | Audio channel strip (EQ, compression, gain staging) |
| `MEMLNautModePAFSynth` | PAF (Phase Aligned Formant) synthesis with MIDI |
| `MEMLNautModeXIASRI` | Audio-reactive mode using machine listening analysis |
| `MEMLNautModeSoundAnalysisMIDI` | Sound analysis with MIDI output |

### Voice Spaces

Voice spaces map ML output parameters to audio engine parameters. They are defined as lambda functions that translate a normalized parameter array into synthesizer/processor settings. See `voicespaces/` for examples:
- PAF synth presets: `VoiceSpace1.hpp`, `VoiceSpaceQuadDetune.hpp`, etc.
- Channel strip presets: `voicespaces/ChannelStrip/basic.hpp` (Neve, SSL emulations)

### Key Components

- **IMLInterface** (`IMLInterface.hpp`): Interactive ML interface using an MLP for inference/training
- **InterfaceRL**: Reinforcement learning interface from memllib that handles joystick input and learning
- **AudioAppBase**: Template base class for audio applications
- **XiasriAnalysis**: Real-time audio feature extraction (pitch, aperiodicity, energy, brightness)

### Submodules (in `src/`)

- **memllib**: Hardware abstraction, audio drivers, synth components, RL interfaces
- **memlp**: MLP (Multi-Layer Perceptron) implementation for embedded ML
- **daisysp**: DSP library (filters, drums, effects, synthesis)

## Memory Sections

The codebase uses RP2040-specific memory placement:
- `AUDIO_MEM` / `AUDIO_FUNC`: Place audio-critical code/data in SRAM
- `APP_SRAM` / `__not_in_flash("app")`: Keep frequently-accessed data out of flash

## Audio Parameters

Sample rate is defined in `AudioDriver::GetSampleRate()`. The audio callback `audio_block_callback` runs on Core 1 and processes stereo audio (`stereosample_t`).
add CLAUDE.md for AI assistant context 2026-02-08 15:49:16 +01:00			`# CLAUDE.md`

			`This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.`

			`## Overview`

			`MEMLNaut-NISPS (Neural Interactive Shaping of Parameter Spaces) is firmware for the MEMLNaut hardware platform - a custom embedded audio device built on Raspberry Pi Pico (RP2040). It implements interactive machine learning for real-time audio synthesis and processing, enabling users to shape sound parameters through reinforcement learning.`

			`Project documentation: https://musicallyembodiedml.github.io/memlnaut/approaches/nisps`

feat: extract nisps-core platform-agnostic ML library Extract the interactive machine learning engine from MEMLNaut-NISPS firmware into a standalone, platform-agnostic C++20 header-only library. What is nisps-core? ------------------- NISPS (Neural Interactive Shaping of Parameter Spaces) core is a parameter mapping engine. It takes N input parameters (joystick, sensors, audio features) and maps them to M output parameters through an interactively-trained neural network. Use it to control: synthesizers, effects, lights, robots, game parameters, or anything that responds to continuous control data. Key Features ------------ - Header-only: No compilation needed, just include and use - Platform-agnostic: Pure C++20, works anywhere - Zero dependencies: Only standard library - Interactive learning: Train by demonstration - Lightweight: ~3,500 lines of optimized neural network code - Flexible: Map 1-100 inputs to 1-100 outputs Architecture ------------ Core components: - IML: High-level interactive ML interface - MLP: Multi-layer perceptron (feedforward neural network) - Dataset: Training data management with replay memory - Layer/Node: Neural network building blocks - Loss: MSE and categorical cross-entropy functions - Utils: Activation functions (sigmoid, ReLU, tanh, etc.) Transformations Applied ----------------------- ✅ Removed Arduino/RP2040 dependencies (Serial, SD, Pico SDK) ✅ Removed audio synthesis code (nisps-core is control-only) ✅ Added nisps namespace to all code ✅ Converted to header-only library with _impl.hpp pattern ✅ Updated to C++20 (required for std::span) ✅ Removed platform-specific serialization ✅ Replaced debug macros with no-op stubs ✅ Added comprehensive documentation and examples Files Added ----------- - nisps-core/README.md: Complete documentation and API reference - nisps-core/CHANGELOG.md: Version history and migration guide - nisps-core/include/nisps/*.hpp: 13 header files (~3,500 lines) - nisps-core/test/main.cpp: XOR test demonstrating basic usage - nisps-core/examples/simple_mapping.cpp: Interactive demo - nisps-core/CMakeLists.txt: Build system for tests Testing ------- ✅ Compiles with GCC 14.2 (C++20) ✅ All tests passing ✅ Successfully instantiates networks and runs inference Performance ----------- - Inference: 1-10 µs for small networks (2-10-10-4) - Training: 10-100 ms for 100 examples, 1000 iterations - Memory: ~1 KB per hidden neuron Migration from Embedded IMLInterface ------------------------------------ Old (embedded): IMLInterface iml(n_inputs, n_outputs); New (nisps-core): nisps::IML<float> iml(n_inputs, n_outputs); All method names remain the same, just add the namespace. Related ------- - Implements: NISPS_CORE_EXTRACTION_PLAN.md - Task graph: NISPS_CORE_TASKS.md - Origin: MEMLNaut-NISPS firmware - Docs: https://musicallyembodiedml.github.io/memlnaut/ Co-authored-by: Claude Code <claude@anthropic.com> 2026-02-08 17:47:23 +01:00			`## NISPS Core Library`

			The `nisps-core/` directory contains a platform-agnostic C++20 extraction of the interactive ML engine. This header-only library can be used in any C++ project for neural network-based parameter mapping.

			`Key differences from firmware:`
			`- ✅ Platform-agnostic (no Arduino/RP2040 dependencies)`
			`- ✅ Header-only (just include and use)`
			`- ✅ C++20 (uses std::span)`
			- ✅ Namespaced (`nisps::`)
			`- ❌ No audio synthesis (use it to control your synth)`
			`- ❌ No hardware drivers`

			`Use case: Control synthesizers, effects, lights, game parameters, or any system that responds to continuous parameters.`

			See `nisps-core/README.md` for complete documentation and examples.

add CLAUDE.md for AI assistant context 2026-02-08 15:49:16 +01:00			`## Build System`

			`This is an Arduino project targeting Raspberry Pi Pico. Build and upload using Arduino IDE or arduino-cli with the earlephilhower/pico board package.`

			```bash
			`# Initialize submodules (required for memllib and memlp)`
			`git submodule update --init --recursive`

			`# Build (adjust port as needed)`
			`arduino-cli compile --fqbn rp2040:rp2040:rpipico -b 115200 MEMLNaut-NISPS.ino`
			`arduino-cli upload --fqbn rp2040:rp2040:rpipico -p /dev/ttyACM0 MEMLNaut-NISPS.ino`
			```

			`## Architecture`

			`### Dual-Core Design`

			`The RP2040's dual cores are used for separation of concerns:`
			`- Core 0: UI loop, ML inference, hardware interface polling (5ms period)`
			`- Core 1: Real-time audio processing, parameter updates, MIDI polling`

			Inter-core synchronization uses memory barriers (`MEMORY_BARRIER()`, `WRITE_VOLATILE()`, `READ_VOLATILE()`) and RP2040 queues (`queue_t`).

			`### Mode System`

			The active mode is selected at compile-time via `#define MEMLNAUT_MODE_TYPE` in `MEMLNaut-NISPS.ino`. Modes implement the `MEMLNautMode` concept (see `modes/MEMLNautMode.hpp`):

			`\| Mode \| Purpose \|`
			`\|------\|---------\|`
			\| `MEMLNautModeChannelStrip` \| Audio channel strip (EQ, compression, gain staging) \|
			\| `MEMLNautModePAFSynth` \| PAF (Phase Aligned Formant) synthesis with MIDI \|
			\| `MEMLNautModeXIASRI` \| Audio-reactive mode using machine listening analysis \|
			\| `MEMLNautModeSoundAnalysisMIDI` \| Sound analysis with MIDI output \|

			`### Voice Spaces`

			Voice spaces map ML output parameters to audio engine parameters. They are defined as lambda functions that translate a normalized parameter array into synthesizer/processor settings. See `voicespaces/` for examples:
			- PAF synth presets: `VoiceSpace1.hpp`, `VoiceSpaceQuadDetune.hpp`, etc.
			- Channel strip presets: `voicespaces/ChannelStrip/basic.hpp` (Neve, SSL emulations)

			`### Key Components`

			- IMLInterface (`IMLInterface.hpp`): Interactive ML interface using an MLP for inference/training
			`- InterfaceRL: Reinforcement learning interface from memllib that handles joystick input and learning`
			`- AudioAppBase: Template base class for audio applications`
			`- XiasriAnalysis: Real-time audio feature extraction (pitch, aperiodicity, energy, brightness)`

			### Submodules (in `src/`)

			`- memllib: Hardware abstraction, audio drivers, synth components, RL interfaces`
			`- memlp: MLP (Multi-Layer Perceptron) implementation for embedded ML`
			`- daisysp: DSP library (filters, drums, effects, synthesis)`

			`## Memory Sections`

			`The codebase uses RP2040-specific memory placement:`
			- `AUDIO_MEM` / `AUDIO_FUNC`: Place audio-critical code/data in SRAM
			- `APP_SRAM` / `__not_in_flash("app")`: Keep frequently-accessed data out of flash

			`## Audio Parameters`

			Sample rate is defined in `AudioDriver::GetSampleRate()`. The audio callback `audio_block_callback` runs on Core 1 and processes stereo audio (`stereosample_t`).