w1n5t0n/memlnaut-nisps
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
memlnaut-nisps/CLAUDE.md

4.7 KiB
Raw Blame History

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Overview

MEMLNaut-NISPS (Neural Interactive Shaping of Parameter Spaces) is firmware for the MEMLNaut hardware platform - a custom embedded audio device built on Raspberry Pi Pico (RP2040). It implements interactive machine learning for real-time audio synthesis and processing, enabling users to shape sound parameters through reinforcement learning.

Project documentation: https://musicallyembodiedml.github.io/memlnaut/approaches/nisps

NISPS Core Library

The nisps-core/ directory contains a platform-agnostic C++20 extraction of the interactive ML engine. This header-only library can be used in any C++ project for neural network-based parameter mapping.

Key differences from firmware:

  • ✅ Platform-agnostic (no Arduino/RP2040 dependencies)
  • ✅ Header-only (just include and use)
  • ✅ C++20 (uses std::span)
  • ✅ Namespaced (nisps::)
  • ❌ No audio synthesis (use it to control your synth)
  • ❌ No hardware drivers

Use case: Control synthesizers, effects, lights, game parameters, or any system that responds to continuous parameters.

See nisps-core/README.md for complete documentation and examples.

Web Playground

The playground/ directory contains a browser-based interactive demo of the NISPS ML engine. It's a faithful JavaScript port of nisps-core's MLP + IML, with no build step or dependencies.

  • 2 inputs (virtual joystick X/Y) mapped through a [3, 10, 10, 14, 14] MLP to 14 outputs controlling a Canvas2D flow-field particle system
  • Two learning modes: Examples (set slider targets, add examples, train) and RL Feedback (thumbs up/down with exploration noise)
  • Serve statically: cd playground && python3 -m http.server
  • Mobile-first: designed for touch/foldable phone use

Key files: js/nisps/ (ML core port), js/ui/ (visualizer, joystick, controls), js/app.js (wiring).

Build System

This is an Arduino project targeting Raspberry Pi Pico. Build and upload using Arduino IDE or arduino-cli with the earlephilhower/pico board package.

# Initialize submodules (required for memllib and memlp)
git submodule update --init --recursive

# Build (adjust port as needed)
arduino-cli compile --fqbn rp2040:rp2040:rpipico -b 115200 MEMLNaut-NISPS.ino
arduino-cli upload --fqbn rp2040:rp2040:rpipico -p /dev/ttyACM0 MEMLNaut-NISPS.ino

Architecture

Dual-Core Design

The RP2040's dual cores are used for separation of concerns:

  • Core 0: UI loop, ML inference, hardware interface polling (5ms period)
  • Core 1: Real-time audio processing, parameter updates, MIDI polling

Inter-core synchronization uses memory barriers (MEMORY_BARRIER(), WRITE_VOLATILE(), READ_VOLATILE()) and RP2040 queues (queue_t).

Mode System

The active mode is selected at compile-time via #define MEMLNAUT_MODE_TYPE in MEMLNaut-NISPS.ino. Modes implement the MEMLNautMode concept (see modes/MEMLNautMode.hpp):

Mode Purpose
MEMLNautModeChannelStrip Audio channel strip (EQ, compression, gain staging)
MEMLNautModePAFSynth PAF (Phase Aligned Formant) synthesis with MIDI
MEMLNautModeXIASRI Audio-reactive mode using machine listening analysis
MEMLNautModeSoundAnalysisMIDI Sound analysis with MIDI output

Voice Spaces

Voice spaces map ML output parameters to audio engine parameters. They are defined as lambda functions that translate a normalized parameter array into synthesizer/processor settings. See voicespaces/ for examples:

  • PAF synth presets: VoiceSpace1.hpp, VoiceSpaceQuadDetune.hpp, etc.
  • Channel strip presets: voicespaces/ChannelStrip/basic.hpp (Neve, SSL emulations)

Key Components

  • IMLInterface (IMLInterface.hpp): Interactive ML interface using an MLP for inference/training
  • InterfaceRL: Reinforcement learning interface from memllib that handles joystick input and learning
  • AudioAppBase: Template base class for audio applications
  • XiasriAnalysis: Real-time audio feature extraction (pitch, aperiodicity, energy, brightness)

Submodules (in src/)

  • memllib: Hardware abstraction, audio drivers, synth components, RL interfaces
  • memlp: MLP (Multi-Layer Perceptron) implementation for embedded ML
  • daisysp: DSP library (filters, drums, effects, synthesis)

Memory Sections

The codebase uses RP2040-specific memory placement:

  • AUDIO_MEM / AUDIO_FUNC: Place audio-critical code/data in SRAM
  • APP_SRAM / __not_in_flash("app"): Keep frequently-accessed data out of flash

Audio Parameters

Sample rate is defined in AudioDriver::GetSampleRate(). The audio callback audio_block_callback runs on Core 1 and processes stereo audio (stereosample_t).