w1n5t0n/memlnaut-nisps
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
memlnaut-nisps/MAP.md

9.8 KiB
Raw Blame History

MAP

MEMLNaut-NISPS: Neural Interactive Shaping of Parameter Spaces. Two living artefacts share one ML core: (1) Arduino/RP2040 firmware for the MEMLNaut hardware, and (2) a browser playground that uses a WASM build of the same MLP to drive a C15 synth + other outputs. A header-only nisps-core/ extraction is reused by the playground (via WASM bindings) and a VCV Rack module. See CLAUDE.md for the long-form architecture narrative.

Layout

Firmware (Arduino, RP2040)

  • MEMLNaut-NISPS.ino — top-level sketch; dual-core setup, mode selected at compile-time via #define MEMLNAUT_MODE_TYPE.
  • IMLInterface.hpp — interactive ML controller; wraps MLP + RL with STORE_VALUE / STORE_POSITION modes.
  • modes/MEMLNautMode.hpp — concept/contract satisfied by each mode class.
  • modes/MEMLNautMode*.hpp — mode implementations (PAFSynth, ChannelStrip, XIASRI, SoundAnalysisMIDI, VerbFX, BreakOr, Elysiamorfs).
  • modes/AudioApps/ — audio engines used by the newer modes (VerbFX, BreakOr, Elysiamorf, RatioSeq).
  • PAFSynthAudioApp.hpp, ChannelStripAudioApp.hpp, ThruAudioApp.hpp, XIASRIAudioApp.hpp — older audio engines at the repo root.
  • XiasriAnalysis.{hpp,cpp} — real-time audio feature extraction (pitch, aperiodicity, brightness, etc.).
  • voicespaces/ — lambdas mapping MLP output vectors to synth parameters. Subdir ChannelStrip/ for EQ/comp presets.
  • src/memllib/, src/memlp/ — git submodules (hardware abstraction + MLP). Not auto-initialized — build breaks without git submodule update --init --recursive.
  • src/daisysp/ — vendored DSP library (filters, drums, effects).
  • data/ — preset/asset CSVs.

nisps-core (platform-agnostic C++20 header-only)

  • nisps-core/include/nisps/ — iml.hpp, mlp.hpp, layer.hpp, node.hpp, dataset.hpp, loss.hpp, utils.hpp.
  • nisps-core/test/, nisps-core/examples/, nisps-core/README.md, CHANGELOG.md.

Playground (browser ML demo)

  • playground/index.html — hub linking to the three variants.
  • playground/a-immersive.html + js/a-app.js — primary app. WASM engine, full control surface, modular/engine-switcher, C15 + MIDI + audio-canvas outputs.
  • playground/b-workbench.html + js/b-app.js, c-journey.html + js/c-app.js — older variants on the legacy JS engine. Feature-frozen; drift vs. a-app is intentional (see CLAUDE.md memory on playground/RECONCILIATION.md — note: file does not currently exist).
  • playground/designs.html, js/app.js — oldest experimental app. Kept for reference.
  • playground/wasm/ — Emscripten build: nisps_bindings.cpp (C API, float32), build.sh, compiled nisps.wasm/nisps.js.
  • playground/js/nisps/ — nisps-wasm.js (WasmIML wrapper), nisps-wasm-worker.js (off-thread train), dataset.js (FIFO ring buffer, max 100), legacy pure-JS engine (iml.js, mlp.js, layer.js, node.js) used by b/c apps.
  • playground/js/synth/ — c15-bridge.js, param-map.js (126 curated C15 params), presets.js (4 tiers), arpeggiator.js.
  • playground/js/ui/ — UI modules. Categories:
    • Input: input-pipeline.js, joystick.js, joy-map-enhanced.js, gamepad.js, hand-tracker.js, eoc-*.js.
    • Control surface: control-surface.js, control-surface-ui.js (3 compound axes: Boldness / Memory / Precision).
    • Training/exploration: snapshot-stack.js, ab-compare.js, region-pin.js, param-pin.js, auto-explore.js, pressure-feedback.js, input-heatmap.js.
    • Output/debug: output-pipeline.js, weight-health.js, gradient-flow.js, session-presets.js, visualizer.js, param-display.js, dev-panel.js.
    • Phase wiring: phase2-ui.js, phase3-ui.js, phase4-ui.js.
    • Modular mode: modular-ui.js (~52k, large), engine-switcher.js — newer; not yet documented in CLAUDE.md.
  • playground/c15/, playground/faust/, playground/osc-bridge/ — external synth/bridge assets.
  • playground/SPEC-controls.md, SPEC-shapeseq.md, ARCHITECTURE.md, PLAN-solidjs-migration.md, TODOS.md, README.md, devlog/ — docs.

Other consumers

  • vcv/ — VCV Rack plugin using nisps-core (src/MEMLNaut.cpp, SPEC.md, NISPS-FORMAT.md).

Tests

  • tests/e2e/*.spec.js — Playwright e2e against the immersive app via the ?debug=1 probe (window.__nisps). Covers ml-engine, wasm-api, ui-interactions, input-pipeline, persistence, engine-switching, modular-mode. Shared helpers in tests/e2e/helpers.js.
  • playwright.config.js, package.json — auto-starts a static server on port 7331.

Top-level docs / planning

  • CLAUDE.md — architecture narrative for both firmware and playground.
  • AGENTS.md — beads/bd conventions.
  • NISPS_CORE_EXTRACTION_PLAN.md, NISPS_CORE_TASKS.md — extraction task list; status unclear, likely stale now that nisps-core/ exists.
  • README.md — short quickstart.

Entry points

  • Firmware: scripts/build-firmware.sh, scripts/flash-firmware.sh, or scripts/build-and-flash-firmware.sh (requires submodules initialised). build-firmware.sh can take an explicit variant name like memlcelium or prompt interactively from the parsed MEMLNautMode* list and rewrite the active mode in MEMLNaut-NISPS.ino. The scripts target rp2040:rp2040:solderparty_rp2350_stamp_xl:opt=Optimize3 and force C++20. Execution = setup()/loop() on Core 0, setup1()/loop1() on Core 1, audio ISR on Core 1.
  • Playground: cd playground && python3 -m http.server (or serve.sh / serve-coop.py), open a-immersive.html. Append ?debug=1 to expose window.__nisps.
  • WASM rebuild: cd playground/wasm && ./build.sh (needs emcc).
  • Tests: npx playwright test (auto-spawns server on 7331).
  • VCV module: built inside vcv/ with the VCV Rack SDK.

Conventions

  • Firmware mode selection is compile-time only; only one MEMLNAUT_MODE_TYPE uncommented at a time in MEMLNaut-NISPS.ino.
  • RP2040 memory placement via APP_SRAM, AUDIO_MEM, AUDIO_FUNC, __not_in_flash("app"). Audio hot paths use __force_inline / __hot / __flatten.
  • Cross-core sync: MEMORY_BARRIER(), WRITE_VOLATILE/READ_VOLATILE, RP2040 queue_t.
  • Voice spaces are header-only structs whose mappings are lambdas capturing synth state — implicit coupling to synth members.
  • Playground ML engines (IML, WasmIML) share a duck-typed interface (inference, train, getWeights/setWeights, …); WASM uses float32, JS engine uses float64.
  • Dataset is a FIFO ring buffer, default max 100 examples; recency/spatial sample weighting is computed JS-side.
  • Spread-aware weight init / RL noise (drawWeightsSpread, moveWeightsEx) live in playground/wasm/nisps_bindings.cpp, not in nisps-core proper — they are playground-specific.
  • Playground UI modules dispatch controlsurface:change CustomEvents; a-app.js listens and reconfigures the input pipeline, spread, and RL params.
  • URL params: ?tame, ?spread, ?preset, ?debug=1, ?shapeseq=1.
  • Persistent memory (bd remember) notes:
    • ShapeSeq is gated behind ?shapeseq=1 until solid — arp remains default.
    • ShapeSeq MLP plan is switchable mode (unified single-MLP first, then dual-MLP option).
    • playground/RECONCILIATION.md is supposed to track features landed in a-app but not yet in b/c. File does not currently exist — if you add a-only features, either create it or explicitly accept the drift.

Gotchas

  • memllib and memlp submodules are not auto-checked-out — a fresh clone will fail to compile the firmware silently.
  • Double-scaled loss: C++ Train() and WASM train_ex both divide by n when no sample weights are supplied (known, backward-compat, tracked as meml-ues).
  • VerbFXAudioApp / MEMLNautModeVerbFX have entire analysis blocks commented out — the mode was migrated from analysis-driven to joystick-driven and cleanup is unfinished. Don't assume XiasriAnalysis wiring is live there.
  • XiasriAnalysis output struct is union-cast to a float array inside XIASRI mode — fragile if the struct layout changes.
  • Recent fixes cluster around the modular voice: matrix rebuild, mod_amp positive-only floor, MLP bypass when untrained, worklet blob-url registration — modular mode is under active churn, so expect rough edges.
  • a-app.js is the single source of truth. Do not reflexively mirror changes to b-app.js / c-app.js — they are frozen legacy variants.
  • window.__nisps only exists with ?debug=1; Playwright helpers expect this.

Open questions / smells

  • modular-ui.js is ~52k and undocumented in CLAUDE.md; needs a docs/modular.md stub, especially given recent bug cluster.
  • engine-switcher.js + engine-switching.spec.js + modular-mode.spec.js — newer engine-selection mechanism not described in CLAUDE.md. Verify whether there is now a supported alternative engine besides WASM-MLP.
  • playground/RECONCILIATION.md is referenced by persistent memory but missing on disk. Either the memory is stale or the file needs creating.
  • NISPS_CORE_TASKS.md / NISPS_CORE_EXTRACTION_PLAN.md at the repo root likely describe completed work — candidates for deletion or archiving under docs/history/.
  • PLAN-solidjs-migration.md (34k) describes an unstarted rewrite. Either flag it "aspirational / not started" at the top or move to docs/.
  • No README.md for playground/wasm/ — a 10-line binding table (C API ↔ JS wrapper ↔ nisps-core call) would save future agents a trip through nisps_bindings.cpp.
  • Firmware IMLInterface's STORE_VALUE vs STORE_POSITION modes have no docs — decide which modes use which and document.
  • Global std::shared_ptr<MIDIInOut> in the sketch introduces refcount traffic on the 1 ms MIDI poll — likely benign, worth confirming.
  • Two duplicated CLAUDE.md copies at ~/.claude/CLAUDE.md and ~/.claude-gp/CLAUDE.md (symlinked), and a per-repo one — not a repo smell, just noted so future agents don't try to "reconcile".