From 1f0eecfe78a5c65b74578e2ca1b1f8be5a04e09a Mon Sep 17 00:00:00 2001 From: monkey-w1n5t0n Date: Sat, 25 Jul 2026 11:14:35 +0200 Subject: [PATCH] docs(firmware): vendor InterfaceRL as read-only reference MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit InterfaceRL.{hpp,cpp,tpp} + InterfaceRLFileFormat.hpp are the upstream reference implementation of the entire NISPS feedback subsystem — nisps/ml/{geo_push,replay,feedback,jolt,ou_noise}.hpp are all ports of it, several still carrying `// upstream InterfaceRL.hpp:NNN` line references. The Phase-4 vendoring dropped examples/ because nothing compiled it. That was correct for the build and wrong for the codebase: with the source of truth out of tree, upstream redesigned the geometric dislike (deleted the /(1+len) taper, doubled kGeometricPushScale, tripled the negative-LR base, moved to batch training over all negatives every tick) and we did not notice for months. Copied verbatim from memllib @ e291192 — the same commit the rest of the vendored tree pins — into lib/memllib/reference/, which sits OUTSIDE src/ and is therefore never compiled: PlatformIO's LDF only recursively builds an Arduino-format library's src/ folder. Verified: slpworkshop still builds (RAM 28.8%, flash 2.1%). reference/README.md states the two rules (never compiled, never edited — a divergence from upstream is a recorded decision, not an edit here) and VENDORED.md's "what was dropped" section now tells the truth. Resolves ALIGNMENT defect 6c. --- ALIGNMENT.md | 12 +- MAP.md | 2 +- .../MEMLNaut-NISPS/lib/memllib/VENDORED.md | 12 +- .../lib/memllib/reference/InterfaceRL.cpp | 6 + .../lib/memllib/reference/InterfaceRL.hpp | 492 ++++++++ .../lib/memllib/reference/InterfaceRL.tpp | 1014 +++++++++++++++++ .../reference/InterfaceRLFileFormat.hpp | 19 + .../lib/memllib/reference/README.md | 33 + 8 files changed, 1578 insertions(+), 12 deletions(-) create mode 100644 firmware/MEMLNaut-NISPS/lib/memllib/reference/InterfaceRL.cpp create mode 100644 firmware/MEMLNaut-NISPS/lib/memllib/reference/InterfaceRL.hpp create mode 100644 firmware/MEMLNaut-NISPS/lib/memllib/reference/InterfaceRL.tpp create mode 100644 firmware/MEMLNaut-NISPS/lib/memllib/reference/InterfaceRLFileFormat.hpp create mode 100644 firmware/MEMLNaut-NISPS/lib/memllib/reference/README.md diff --git a/ALIGNMENT.md b/ALIGNMENT.md index 840abbb..f6d05e9 100644 --- a/ALIGNMENT.md +++ b/ALIGNMENT.md @@ -105,14 +105,6 @@ still not a design. (`ml_bench` U4 sweeps dose; U1 sweeps upstream's soft-target alpha, where alpha=1 is NISPS today). It wants a matched-N head-to-head, not a guess. -### 6c. `InterfaceRL` — the reference implementation — is not in the tree (2026-07-25) - -**What.** It lives in `memllib/examples/`, and the vendoring dropped `examples/` -(`VENDORED.md`). So the source of truth for our most contested subsystem is absent, and -the divergences in 6b went unnoticed for months. Either vendor -`examples/InterfaceRL.{hpp,cpp,tpp}` read-only alongside the rest, or record its pinned -commit and a fetch recipe in `VENDORED.md`. - ## Open mission questions ### Q1: Per-mode MLP architectures or one shared shape? (2026-04-29) @@ -137,6 +129,10 @@ Legacy a-immersive was mobile-first; Manifold is desktop-first. Defer until user ## Recently resolved (delete after a few weeks) +- 2026-07-25: **`InterfaceRL` is back in the tree (defect 6c).** Vendored verbatim from + memllib `e291192` at `firmware/MEMLNaut-NISPS/lib/memllib/reference/` — outside `src/`, + so PlatformIO never compiles it. Upstream drift in the feedback subsystem is a `diff` + again rather than an archaeology session. - 2026-07-25: **The optimiser mismatch (defect 6) is fixed.** `nisps/ml/training.hpp` was SGD-only while upstream `memlp` (`ea777502`) applies **RMSProp everywhere**, so every learning rate we ported landed in an optimiser that reads it differently — an RMSProp diff --git a/MAP.md b/MAP.md index 7337f51..26f11a9 100644 --- a/MAP.md +++ b/MAP.md @@ -27,7 +27,7 @@ MEMLNaut-NISPS — Neural Interactive Shaping of Parameter Spaces. One C++20 cod - `selftest.hpp` — standalone guided hardware self-test rig (`SelfTest` variant; no engine/ML). Step-driven state machine on a `SelfTestView`: TFT prompts the operator through every control, auto-advances on detection, encoder-press skips. Ends with optional L/R/BOTH sine-sweep headphone check (core 1 block callback) + MIDI loopback-cable test. Lives firmware-side (touches TFT + raw pins) so it stays out of platform-agnostic `nisps/`. - `output_router.hpp` — top-level `drain_outputs()` entry point. (Inputs are wired directly by `peripherals.hpp`'s `bind_peripherals()`.) - `settings_view.hpp` — `wire_settings(mode)`: adds on-device settings views to the MEMLNaut display carousel (TFT + rotary encoder). Joystick Dual/Single toggle for the 4-input ("two 2-D joystick") modes — "Single" pins ML input channels 2,3 to neutral via `ModeBase::set_input_pinned` (no net rebuild). Registered in the `.ino` after `addSystemInfoView()`. -- `firmware/MEMLNaut-NISPS/lib/memllib/` — **vendored** memllib (was the `src/memllib` submodule): hardware abstraction (audio driver, TFT display, MIDI, peripherals), ~1.9 MB / 100 files, `examples/` dropped. `VENDORED.md` records the upstream commit and the re-sync procedure; `LICENSE` is MPL-2.0, copied verbatim. **Sources must sit under `lib/memllib/src/`** — PlatformIO's library builder falls back to a flat root-only scan without it and silently compiles nothing while still linking (see VENDORED.md). +- `firmware/MEMLNaut-NISPS/lib/memllib/` — **vendored** memllib (was the `src/memllib` submodule): hardware abstraction (audio driver, TFT display, MIDI, peripherals), ~1.9 MB / 100 files, `examples/` dropped **except** `reference/InterfaceRL.{hpp,cpp,tpp}` + `InterfaceRLFileFormat.hpp` — upstream's reference implementation of the feedback subsystem `nisps/ml/{geo_push,replay,feedback,jolt,ou_noise}.hpp` were ported from, kept verbatim and NEVER compiled (`reference/` sits outside `src/`, which is the only thing PlatformIO builds). It is there so upstream drift is a `diff`; losing it is how the e291192 geometric-dislike redesign went unnoticed for months. `VENDORED.md` records the upstream commit and the re-sync procedure; `LICENSE` is MPL-2.0, copied verbatim. **Sources must sit under `lib/memllib/src/`** — PlatformIO's library builder falls back to a flat root-only scan without it and silently compiles nothing while still linking (see VENDORED.md). - `firmware/README.md` — structure + build instructions. - `firmware/useq-celium/` — standalone RP2040 firmware (PlatformIO, Arduino-Pico core) that turns a uSEQ module + CV expander into a USB→CV/gate converter driven by the manifold `cvgate` backend. `shared/protocol.h` is the v2 wire-protocol single source of truth (mirrored by `manifold/src/backends/useq-protocol.ts`); `main/` (USB serial → CV1–3 + GATE1–3, I2C → expander) and `expander/` (I2C slave → CV4–11). Wire spec: `docs/specs/useq-cv-protocol.md`. Restored from the April-2026 "uSEQ-Celium" mode. diff --git a/firmware/MEMLNaut-NISPS/lib/memllib/VENDORED.md b/firmware/MEMLNaut-NISPS/lib/memllib/VENDORED.md index 3232a70..d010583 100644 --- a/firmware/MEMLNaut-NISPS/lib/memllib/VENDORED.md +++ b/firmware/MEMLNaut-NISPS/lib/memllib/VENDORED.md @@ -23,9 +23,15 @@ migration) — only the wrapping `src/` folder and the `library.properties` mani new, both required for PlatformIO to discover and recursively compile this tree (see below). -Dropped: `examples/` (17 files — never compiled; the firmware never referenced it, and -its content that mattered was already ported into `nisps/ml/{jolt,ou_noise,feedback, -geo_push}.hpp` per the pre-Phase-4 submodule-bump decision), `.git` (submodule gitlink), +Dropped: `examples/` (never compiled; the firmware never referenced it, and its content +that mattered was already ported into `nisps/ml/{jolt,ou_noise,feedback,geo_push}.hpp` +per the pre-Phase-4 submodule-bump decision) — **except** `InterfaceRL.{hpp,cpp,tpp}` and +`InterfaceRLFileFormat.hpp`, which were added back on 2026-07-25 under `reference/`. +Dropping them was correct for the build and wrong for the codebase: `InterfaceRL` is the +source of truth for the whole feedback subsystem we ported, and with it out of tree we +missed upstream's redesign of the geometric dislike for months. `reference/` sits outside +`src/`, so PlatformIO does not compile it; see `reference/README.md`. Also dropped: +`.git` (submodule gitlink), `.gitignore` (build-artifact patterns, meaningless once vendored — this repo's own `.gitignore` covers it), `README.md` (described the old Arduino-IDE TFT_eSPI `User_Setup_Select.h` copy-paste workflow, which PlatformIO replaces with diff --git a/firmware/MEMLNaut-NISPS/lib/memllib/reference/InterfaceRL.cpp b/firmware/MEMLNaut-NISPS/lib/memllib/reference/InterfaceRL.cpp new file mode 100644 index 0000000..60204d1 --- /dev/null +++ b/firmware/MEMLNaut-NISPS/lib/memllib/reference/InterfaceRL.cpp @@ -0,0 +1,6 @@ +// InterfaceRL is now a class template (InterfaceRL); its definitions +// live in InterfaceRL.tpp, included from InterfaceRL.hpp so each per-mode +// instantiation is available at its use site. This translation unit is +// intentionally empty (kept so existing build scripts that reference it still +// find a file). +#include "InterfaceRL.hpp" diff --git a/firmware/MEMLNaut-NISPS/lib/memllib/reference/InterfaceRL.hpp b/firmware/MEMLNaut-NISPS/lib/memllib/reference/InterfaceRL.hpp new file mode 100644 index 0000000..03e9255 --- /dev/null +++ b/firmware/MEMLNaut-NISPS/lib/memllib/reference/InterfaceRL.hpp @@ -0,0 +1,492 @@ +#ifndef INTERFACERL_HPP +#define INTERFACERL_HPP + +#include "../interface/InterfaceBase.hpp" + +#include "../../memlp/StaticMLP.h" +#include "../../memlp/ReplayMemory.hpp" +#include "../../memlp/OrnsteinUhlenbeckNoise.h" +#include +#include "../utils/sharedMem.hpp" + +#include "../PicoDefs.hpp" +//#include "../hardware/memlnaut/display.hpp" +#include "../hardware/memlnaut/display/MessageView.hpp" + +#include "../interface/UARTInput.hpp" +#include "../interface/MIDIInOut.hpp" + +#include "../hardware/memlnaut/display/MessageView.hpp" +#include "../hardware/memlnaut/display/BarGraphView.hpp" +#include "../hardware/memlnaut/display/RLView.hpp" +#include "../hardware/memlnaut/display/BlockSelectView.hpp" +#include "../hardware/memlnaut/display/SingleSelectView.hpp" +#include "../hardware/memlnaut/display/RotarySelectView.hpp" +#include "../hardware/memlnaut/display/NameInputView.hpp" +#include "../hardware/memlnaut/display/CCSelectView.hpp" +#include "InterfaceRLFileFormat.hpp" + +#define RL_MEM __not_in_flash("rlmem") + +struct trainStatelessRLItem { + std::vector input ; + std::vector action; + float reward; +}; + +// Non-template base holding the nested types + constants that callers reference +// without a template argument (e.g. InterfaceRLBase::INPUT_MODES in the modes +// and the .ino). Because this (and InterfaceBase) are *non-dependent* bases of +// InterfaceRL, the template sees every inherited member by ordinary +// lookup — no this->/using-declarations required. +class InterfaceRLBase : public InterfaceBase +{ +public: + using OnMIDICtrlCallback = std::function; + + // Training data type (independent of the network's output width). + using training_pair_t = std::pair>, + std::vector>>; + + static constexpr size_t kMaxNNInputs = 10; + + enum class INPUT_MODES { + JOYSTICK, + MACHINE_LISTENING, + JOYSTICK_AND_MACHINE_LISTENING, + SERIAL_INPUT + }; + + enum class INPUT_SOURCE : uint8_t { + JOYSTICK_3D = 0, + JOYSTICK_4D, + MACHINE_LISTENING, + MIDI_1CC, + MIDI_3CC, + MIDI_8CC, + COMBINED, + COUNT + }; + + enum class MEMORY_STORE_MODES { + ADD, + REPLACE_5_PERCENT, + REPLACE_10_PERCENT, + REPLACE_15_PERCENT, + REWARD_DECAY_10_PERCENT, + REWARD_DECAY_20_PERCENT + }; + + // Input-source state is independent of the network's output width, so it + // lives in the base: this lets N-agnostic helpers (e.g. MachineListeningMixin) + // hold an InterfaceRLBase* and still query/configure the input source. + INPUT_SOURCE getInputSource() const { return input_source_; } + void setHasMachineListening(bool v) { hasMachineListening_ = v; } + +protected: + INPUT_SOURCE input_source_ = INPUT_SOURCE::JOYSTICK_3D; + bool hasMachineListening_ = false; +}; + +// The RL interface. N_OUTPUTS (the active mode's parameter count) is fixed at +// compile time, so the synth-mapping network is a static-memory StaticMLP with +// no heap allocation. The mode declares e.g. InterfaceRL. +template +class InterfaceRL : public InterfaceRLBase +{ +public: + + // Compile-time mapping network: kMaxNNInputs -> 16 -> 16 -> N_OUTPUTS. + using SynthMLP = smlp::StaticMLP, + smlp::Activations, + loss::LOSS_FUNCTIONS::LOSS_MSE>; + + InterfaceRL() : InterfaceRLBase() +// , ou_noise(0.02f, 0.0f, 0.2f, 0.001f, 0.0f) +{ + + } + void setup(size_t n_inputs, size_t n_outputs, bool addMessageView = true); + + void optimise(); + + inline void setState(const size_t index, float value) { + controlInput[index] = value; + newInput = true; + } + + // Force the next loop to regenerate + re-send the action, even if no input changed. + // Use when an output-stage parameter (e.g. a fade/home value) changes. + inline void markInputDirty() { newInput = true; } + + void readAnalysisParameters(std::vector params) override; + + void generateAction(bool donthesitate=false); + + inline void optimiseSometimes() { + if (optimiseCounter>=optimiseDivisor) { + optimise(); + optimiseCounter=0; + newInput = true; + }else{ + optimiseCounter++; + } + } + + void storeExperience(float reward, std::vector &experienceState, std::vector &experienceAction ); + + #define randomWeightVariance 1.f + + inline void randomiseTheNetwork() + { + synthMapping.RandomiseWeightsAndBiasesLin(-0.9f,1.1f, -0.9f, 0.3f); + newInput = true; + resetMinMaxFlag = true; + } + + + inline void setOptimiseDivisor(size_t newDiv) { + optimiseDivisor = newDiv; + } + + void setOptimiseDivisorInterf(float value); + + inline void forgetMemory() { + replayMem.clear(); + } + + inline void setRewardScale(float scale) { + rewardScale = scale; + } + + inline void setLRScale(const float scale) { + learningRateScaled = learningRate * scale; // knob at 0 -> LR 0 -> training off (intended) + String msg = "LR scale: " + String(scale); + if (msgView) msgView->post(msg); + } + + void setRewardScaleInterf(float value); + + inline void setNoiseLevel(float level) { + // Knob [0,1] -> roaming amplitude (the OU walk's stationary std) in param space. + // theta/dt (set in setup) fix the smoothness; this only sets how far each param + // drifts from the mapping output. Low = gentle local wander; full = slow sweeps + // across the whole [0,1] range. kMaxAmplitude sets the "depth": higher reaches + // deeper into the param space (and interacts more with the [0,1] rails via the + // gentle reflection in generateAction), lower keeps it shallower/more local. + constexpr float kMaxAmplitude = 0.65f; + float amplitude = level * kMaxAmplitude; + if (amplitude < 0.01f) { + amplitude = 0.f; + if (msgView) msgView->post("Noise off"); + } else { + String msg = "Explore amount: " + String(amplitude, 3); + if (msgView) msgView->post(msg); + } + for(auto& ou_noise: ou_noises) { + ou_noise->setStationaryStd(amplitude); + } + // Learning stays active during exploration on purpose: likes/dislikes given while + // the noise roams are what steer the network toward sounds the player wants. + if (nnOutputsGraphView) nnOutputsGraphView->setNoiseActive(amplitude > 0.f); + } + + // Jolt = permanent weight modulation (B2, held). At press, pick a random subset of + // weights scattered across the net and roll a bounded random target for each. While + // held, EMA-glide each toward its target (stepJolt, called per loop); release just + // freezes them, so the change persists. Bounded by construction (interpolation toward + // targets in the weight-init range can't run away) and smooth (no per-tick jitter). + // Runs on the main loop / under the mlpActive lock, so touching the MLP here is safe. + inline float randomJoltTarget() const { + return kJoltWeightMin + (static_cast(rand()) / RAND_MAX) * (kJoltWeightMax - kJoltWeightMin); + } + + inline void startJolt() { + joltActive_ = true; + joltWeightLoc_.clear(); + joltTarget_.clear(); + // StaticMLP exposes a flat view over all weights (layer 0 first); pick + // random global indices to modulate. + const size_t total = SynthMLP::TotalWeights(); + if (total == 0) return; + for (size_t i = 0; i < kJoltNumWeights; i++) { + joltWeightLoc_.push_back(rand() % total); + joltTarget_.push_back(randomJoltTarget()); + } + if (nnOutputsGraphView) nnOutputsGraphView->setLastAction("jolt"); + if (msgView) msgView->post("Jolt: morphing weights"); + } + + inline void stepJolt() { + const size_t total = SynthMLP::TotalWeights(); + for (size_t i = 0; i < joltWeightLoc_.size(); i++) { + const size_t idx = joltWeightLoc_[i]; + if (idx >= total) continue; // stale after a model load + float* wp = synthMapping.WeightPtrAt(idx); + if (!wp) continue; + float& w = *wp; + w += kJoltMorphRate * (joltTarget_[i] - w); + // Reached this target (EMA only asymptotes, so use a threshold) -> roll a new + // one, keeping the weight in motion for as long as the button is held. + float gap = joltTarget_[i] - w; + if (gap < 0.f) gap = -gap; + if (gap < kJoltTargetEpsilon) joltTarget_[i] = randomJoltTarget(); + } + markInputDirty(); // weights changed -> regenerate + re-send the action + } + + inline void stopJolt() { + joltActive_ = false; // weights stay where they morphed to (permanent) + joltLRRamp_ = 0.f; // resume learning from 0, ramping back to full over ~5s + } + + void bind_RL_interface(INPUT_MODES input_mode = INPUT_MODES::JOYSTICK, bool joystick4D = false); + + void bindInterface(INPUT_MODES input_mode = INPUT_MODES::JOYSTICK,bool joystick4D = false) { + bind_RL_interface(input_mode, joystick4D); + } + + void bindInterface(bool disable_joystick=false, bool joystick4D = false) { + bind_RL_interface(disable_joystick ? INPUT_MODES::MACHINE_LISTENING : INPUT_MODES::JOYSTICK, joystick4D); + } + + + void bindUARTInput(std::shared_ptr uart_input, + const std::vector& kUARTListenInputs) + { + uart_input->SetCallback([this](size_t channel, float value) { + // Serial.println("UART input: " + String(channel) + " value: " + String(value)); + if (channel < controlInput.size()) { + setState(channel, value); + } + }); + } + + void bindMIDI(std::shared_ptr midi_interf, bool enableFootcontroller=false); + + void setModeInfo(const String& modeRoot, const String& modeTag); + + using ExtraSaveDataFn = std::function()>; + using ExtraLoadDataFn = std::function; + void setExtraSaveCallback(ExtraSaveDataFn fn) { _extraSaveFn = fn; } + void setExtraLoadCallback(ExtraLoadDataFn fn) { _extraLoadFn = fn; } + + using RVCallback = std::function; + void setRVX1Override(RVCallback fn) { rvX1Override = std::move(fn); } + void setRVY1Override(RVCallback fn) { rvY1Override = std::move(fn); } + void setRVZ1Override(RVCallback fn) { rvZ1Override = std::move(fn); } + + void setActiveDims(std::vector dims) { activeDims_ = std::move(dims); } + + std::function&)> inputInjectionHook; + + void trigger_like(); + void trigger_dislike(); + + inline void getAction(std::vector &out_action) { + out_action = action; + } + + size_t getActiveInputCount() const { + switch (input_source_) { + case INPUT_SOURCE::JOYSTICK_3D: return 3; + case INPUT_SOURCE::JOYSTICK_4D: return 4; + case INPUT_SOURCE::MACHINE_LISTENING: return 6; + case INPUT_SOURCE::MIDI_1CC: return 1; + case INPUT_SOURCE::MIDI_3CC: return 3; + case INPUT_SOURCE::MIDI_8CC: return 8; + case INPUT_SOURCE::COMBINED: return kMaxNNInputs; + default: return kMaxNNInputs; + } + } + const std::vector& getControlInput() const { return controlInput; } + + // Recompute the unused-input pad value. Done only when the input mode changes (not per + // frame). Scales as 1.1/n_unused so the total constant injected into layer 1 stays + // bounded regardless of how many dims are unused — avoids over-driving the net. + void updateUnusedInputDefault() { + const size_t used = getActiveInputCount(); + const size_t unused = (used < kMaxNNInputs) ? (kMaxNNInputs - used) : 0; + unusedInputDefault_ = (unused > 0) ? (1.1f / static_cast(unused)) : 0.f; + } + + // persist=false applies the change in memory + updates the bar graph without writing + // flash. A flash write stalls XIP execution on the RP2040 (blanking the display), so + // the rotary-driven path applies immediately but debounces the save (see loopCallback). + void setInputSource(INPUT_SOURCE src, bool persist = true) { + input_source_ = src; + updateUnusedInputDefault(); + if (persist) saveInputSource(); + if (nnInputsGraphView) nnInputsGraphView->setNumDisplayBars(getActiveInputCount()); + } + + // ISR-safe entry point (the rotary-encoder dispatch runs in interrupt context). + // setInputSource() does heap allocation (bar-graph resize), SPI (fillRect) and flash + // file IO — all unsafe in an ISR — so only record the request here and let the main + // loop apply it via the pendingInputSourceChange_ handler in bind_RL_interface(). + void requestInputSource(INPUT_SOURCE src) { + pendingInputSource_ = src; + pendingInputSourceChange_ = true; + } + void addInputSourceView(bool includeCCSelect = true); + + void SetMIDI5Callback(OnMIDICtrlCallback _cb_) { + midi5cb = _cb_; + } + void SetMIDI6Callback(OnMIDICtrlCallback _cb_) { + midi6cb = _cb_; + } + + // Display views + std::shared_ptr msgView; + std::shared_ptr fileSaveView; + std::shared_ptr fileLoadView; + std::shared_ptr nameInputView; + std::shared_ptr nnInputsGraphView; + std::shared_ptr nnOutputsGraphView; + std::shared_ptr memoryStoreModeView; + std::shared_ptr ccSelectView; + + const std::vector& getLastAction() const { return action; } + +protected: + // Helper methods for trigger actions + void _perform_like_action(); + void _perform_dislike_action(); + void _perform_randomiseRL_action(); + bool _save_RL_to_SD(String id); + bool _load_RL_from_SD(String id); + void _forget_replay_mem_interf(); + void _saveSlotNames(); + void _loadSlotNames(); + + static constexpr int kNumSlots = 12; + String slotNames[kNumSlots]; + int pendingSaveSlot = -1; + + +private: + + OnMIDICtrlCallback midi5cb = nullptr; + OnMIDICtrlCallback midi6cb = nullptr; + + RVCallback rvX1Override; + RVCallback rvY1Override; + RVCallback rvZ1Override; + + static constexpr size_t bias=1; + + size_t optimiseDivisor = 1; + size_t optimiseCounter = 0; + bool newInput=false; + + bool actionBeingDragged=false; + + std::vector itemsToRemove; + + float raw_joystick_[4] = {}; + float raw_ml_[6] = {}; + float raw_midi_[8] = {}; + // Constant used to pad the unused NN input dims; recomputed only on input-mode change. + float unusedInputDefault_ = 0.5f; + + static constexpr const char* kInputSourceFile = "/input_source.bin"; + void assembleInputs(); + void copyAndZero(const float* src, size_t n); + void saveInputSource(); + void loadInputSource(); + void saveCCNumbers(); + void loadCCNumbers(); + + size_t analysisParamsOffset = 0; + + MEMORY_STORE_MODES memoryStoreMode = MEMORY_STORE_MODES::REPLACE_10_PERCENT; + std::array memOptions = {"Add", "Replace 5%", "Replace 10%", "Replace 15%", "Reward Decay 10%", "Reward Decay 20%"}; + // Dislike repulsion: how far a 'no' moves the disliked action's training target away + // from the liked region (untapered), and the negative-batch LR base. Bigger = a 'no' + // slides the sound clearly further away; >1 tends to push params to the [0,1] rails. + static constexpr float kGeometricPushScale = 1.0f; + static constexpr float kNegLRBase = 1.5f; // negLRRatio = kNegLRBase - 0.4*negFraction + // A 'no' pushes at full strength for this long (wall-clock), then expires — no decay. + // The number of optimise cycles within the window depends on the mode's NN update rate. + static constexpr uint32_t kDislikeLifetimeMs = 2500; + static constexpr size_t kCentroidK = 4; + std::vector activeDims_; + bool removeItemsAtDistance(std::vector &experienceState, const float distThreshold, const float reward); + void decayItemsAtDistance(std::vector &experienceState, const float distThreshold); + + + + + + std::vector layers_nodes; + + const bool use_constant_weight_init = false; + const float constant_weight_init = 0; + + SynthMLP synthMapping; // value member -> lives in the (static) mode object: zero heap + + float learningRate = 1e-3; + float learningRateScaled = learningRate; + std::vector action; + + ReplayMemory replayMem; + static constexpr size_t memoryLimit = 64; + static constexpr size_t batchSize = 8; + + std::vector mappingOutput; + std::vector controlInput; + std::vector savedAction; + + + float rewardScale = 1.0f; + + // OrnsteinUhlenbeckNoise ou_noise; + std::vector> ou_noises; + + // Exploration-noise travel speed. + static constexpr float kNoiseDt = 0.004f; // normal OU travel speed (set in setup) + + // Jolt = permanent weight modulation while B2 is held (see startJolt/stepJolt). + static constexpr size_t kJoltNumWeights = 40; // random weights perturbed per press + static constexpr float kJoltMorphRate = 0.017f; // EMA per tick (~1s to target @200Hz) + static constexpr float kJoltWeightMin = -1.2f; // target range == weight-init range + static constexpr float kJoltWeightMax = 0.9f; + static constexpr float kJoltTargetEpsilon = 0.05f; // re-roll target once within this + static constexpr float kJoltLRRampStep = 0.001f; // LR recovery rate: 1/(5s * 200Hz) + std::vector joltWeightLoc_; // global flat weight indices (StaticMLP::WeightPtrAt) + std::vector joltTarget_; // per-selected-weight target value + bool joltActive_ = false; + // After a jolt releases, learning resumes gently: effective LR *= joltLRRamp_, which + // climbs 0 -> 1 over ~5s so fresh training doesn't immediately drag the net off the + // jolted sound. 1.0 = normal (full LR). + float joltLRRamp_ = 1.0f; + + bool resetMinMaxFlag = false; + + // Deferred actions: set from ISR, consumed in main-loop loopCallback before optimise() + volatile bool pendingLike_{false}; + volatile bool pendingDislike_{false}; + volatile bool pendingDragStore_{false}; // drag-release: store savedAction + volatile bool pendingInputSourceChange_{false}; // input-source change: deferred from rotary ISR + INPUT_SOURCE pendingInputSource_{INPUT_SOURCE::JOYSTICK_3D}; + // Debounced flash persistence: a save is scheduled this many ms after the last change, + // so scrolling through sources doesn't trigger a flash write (XIP stall) per detent. + static constexpr uint32_t kInputSourceSaveDelayMs = 600; + uint32_t inputSourceSaveDueMs_ = 0; // 0 = no save pending + + spin_lock_t *mlpActive; + + String _modeRoot{"mlp_rl"}; + String _modeTag{"Unknown"}; + ExtraSaveDataFn _extraSaveFn; + ExtraLoadDataFn _extraLoadFn; + +}; + +// Template method definitions (header-only so the per-mode instantiation is +// available wherever InterfaceRL is used). +#include "InterfaceRL.tpp" + +#endif // INTERFACERL_HPP \ No newline at end of file diff --git a/firmware/MEMLNaut-NISPS/lib/memllib/reference/InterfaceRL.tpp b/firmware/MEMLNaut-NISPS/lib/memllib/reference/InterfaceRL.tpp new file mode 100644 index 0000000..9ec762b --- /dev/null +++ b/firmware/MEMLNaut-NISPS/lib/memllib/reference/InterfaceRL.tpp @@ -0,0 +1,1014 @@ +#include +#include "../utils/sharedMem.hpp" // Required for READ_VOLATILE, sharedMem constants and PERIODIC_DEBUG +#include // Required for Serial, millis, delay +#include "../hardware/memlnaut/MEMLNaut.hpp" // Required for MEMLNaut::Instance() +// display.hpp is included via InterfaceRL.hpp + +inline float euclideanDistance(const std::vector& a, const std::vector& b) { + float sum = 0.0f; + for (size_t i = 0; i < a.size(); ++i) { + float diff = a[i] - b[i]; + sum += diff * diff; + } + return sqrtf(sum); +} + + + +// Protected helper method implementations +template +void InterfaceRL::_perform_like_action() { + static std::vector likemsgs = { + "Wow, incredible", "Awesome", "That's amazing", "Unbelievable+", + "I love it!!", "More of this", "Yes!!!!", "A-M-A-Z-I-N-G", + "Keep going!", "In flow", "I believe in you", + "Absolutely brilliant!", "This is perfection!", + "Stunning work!", "Pure genius!", + "Keep shining!", "Fantastic!", "Incredible vibes!", "Love this journey!", + "Super cool!" + }; + String msg = likemsgs[rand() % likemsgs.size()]; + this->storeExperience(1.f, controlInput, action); + if (nnOutputsGraphView) nnOutputsGraphView->setLastAction("yes"); + DEBUG_PRINTLN(msg); + if (msgView) msgView->post(msg); +} + +template +void InterfaceRL::_perform_dislike_action() { + static std::vector dislikemsgs = { + "oh no!", "Get rid of this sound", + "Why even bother?", "New sound please!", "No, please no!!!", + "Thumbs down", "I'm so sorry", "I'm trying my hardest...", + "I'm doing the best I can", "Learning...", + "I'll try to do better.", "Still figuring things out.", + "Thanks for the feedback.", "Working on it!", "Oops, my bad.", + "Learning from this.", "I'll adjust, promise.", "Noted", "Rearranging", + "Let's move on!" + }; + String msg = dislikemsgs[rand() % dislikemsgs.size()]; + this->storeExperience(-1.f, controlInput, action); + if (nnOutputsGraphView) nnOutputsGraphView->setLastAction("no"); + DEBUG_PRINTLN(msg); + if (msgView) msgView->post(msg); +} + +template +void InterfaceRL::_perform_randomiseRL_action() { + + this->randomiseTheNetwork(); + this->generateAction(true); + if (nnOutputsGraphView) nnOutputsGraphView->setLastAction("scramble"); + DEBUG_PRINTLN("Randomising networks"); + if (msgView) msgView->post("Scrambling the network"); +} + +// Public trigger methods — called from ISR context, so only set a flag. +// The actual action runs in the main-loop loopCallback before optimise(). +template +void InterfaceRL::trigger_like() { + pendingLike_ = true; +} + +template +void InterfaceRL::trigger_dislike() { + pendingDislike_ = true; +} + +// void InterfaceRL::trigger_randomiseRL() { +// _perform_randomiseRL_action(); +// } + + +template +void InterfaceRL::setOptimiseDivisorInterf(float value) +{ + size_t divisor = 1 + (value * 100); + String msg; + if (divisor > 90) { + divisor = 999999; + msg = "Optimisation paused"; + }else{ + msg = "Optimise every " + String(divisor) + " cycles"; + } + if (msgView) msgView->post(msg); + this->setOptimiseDivisor(divisor); + DEBUG_PRINTLN(msg); +} + + +template +void InterfaceRL::bind_RL_interface(INPUT_MODES input_mode, bool joystick4D) { + + loadInputSource(); + if (nnInputsGraphView) nnInputsGraphView->setNumDisplayBars(getActiveInputCount()); + + // Set up momentary switch callbacks + MEMLNaut::Instance()->setMomA1Callback([this]() { + if (MEMLNaut::Instance()->getMOMA1State()) { + this->trigger_like(); + } + }); + MEMLNaut::Instance()->setMomA2Callback([this]() { + if (MEMLNaut::Instance()->getMOMA2State()) { + this->trigger_dislike(); + } + }); + MEMLNaut::Instance()->setMomB1Callback([this]() { + if (MEMLNaut::Instance()->getMOMB1State()) { + _perform_randomiseRL_action(); + } + }); + // B2 held = momentary fast exploration (was: perturb network weights). The button ISR + // only dispatches a callback on press, so start the jolt here and detect *release* by + // polling getMOMB2State() in the loop callback below. + MEMLNaut::Instance()->setMomB2Callback([this]() { + if (MEMLNaut::Instance()->getMOMB2State()) { + startJolt(); + } + }); + + // Always register joystick callbacks — they write to raw_joystick_ + // (ignored by assembleInputs() when a non-joystick source is active) + MEMLNaut::Instance()->setJoySWCallback([this](bool state) { + if (state) { + savedAction = action; + actionBeingDragged = true; + if (nnOutputsGraphView) nnOutputsGraphView->setLastAction("drag"); + if (msgView) msgView->post("Where do you want it?"); + } else { + if (actionBeingDragged) { + actionBeingDragged = false; + pendingDragStore_ = true; + if (nnOutputsGraphView) nnOutputsGraphView->setLastAction("drop"); + if (msgView) msgView->post("Here!"); + } + } + }); + MEMLNaut::Instance()->setJoyXCallback([this](float value) { raw_joystick_[0] = value; newInput = true; }); + MEMLNaut::Instance()->setJoyYCallback([this](float value) { raw_joystick_[1] = value; newInput = true; }); + MEMLNaut::Instance()->setJoyZCallback([this](float value) { raw_joystick_[2] = value; newInput = true; }); + MEMLNaut::Instance()->setADC3Callback([this](float value) { raw_joystick_[3] = value; newInput = true; }); + + + MEMLNaut::Instance()->setTogB1Callback([this](bool state) { // scr_ref no longer captured directly + if (state) { + this->_forget_replay_mem_interf(); + } + }); + + MEMLNaut::Instance()->setTogA1Callback([this](bool state) { + if (state) { + savedAction = action; + actionBeingDragged = true; + if (nnOutputsGraphView) nnOutputsGraphView->setLastAction("drag"); + if (msgView) msgView->post("Where do you want it?"); + } else { + if (actionBeingDragged) { + actionBeingDragged = false; + pendingDragStore_ = true; // deferred: storeExperience in loopCallback + if (nnOutputsGraphView) nnOutputsGraphView->setLastAction("drop"); + if (msgView) msgView->post("Here!"); + } + } + }); + + + MEMLNaut::Instance()->setRVX1Callback( + rvX1Override ? rvX1Override : RVCallback([this](float value) { this->setRewardScaleInterf(value); })); + + MEMLNaut::Instance()->setRVY1Callback( + rvY1Override ? rvY1Override : RVCallback([this](float value) { this->setLRScale(value); })); + + MEMLNaut::Instance()->setRVZ1Callback( + rvZ1Override ? rvZ1Override : RVCallback([this](float value) { setNoiseLevel(value); })); + // Set up loop callback + MEMLNaut::Instance()->setLoopCallback([this]() { + // Jolt release: the B2 ISR only fires on press, so poll the live pin to end it. + if (joltActive_ && !MEMLNaut::Instance()->getMOMB2State()) { + stopJolt(); + } + // Process deferred actions from ISR before touching replayMem in optimise + if (pendingLike_) { + pendingLike_ = false; + _perform_like_action(); + } + if (pendingDislike_) { + pendingDislike_ = false; + _perform_dislike_action(); + } + if (pendingDragStore_) { + pendingDragStore_ = false; + this->storeExperience(1.f, controlInput, savedAction); + if (nnOutputsGraphView) { + size_t pos = 0; + for (size_t i = 0; i < replayMem.size(); i++) + if (replayMem.getItem(i).reward > 0.f) pos++; + nnOutputsGraphView->setMemoryCounts(pos, replayMem.size() - pos); + } + } + // Apply a deferred input-source change off the rotary ISR (heap/SPI/flash IO). + // Apply in-memory now for a responsive UI, but debounce the flash write: scrolling + // through sources would otherwise stall XIP per detent and blank the display. + if (pendingInputSourceChange_) { + pendingInputSourceChange_ = false; + setInputSource(pendingInputSource_, false); + inputSourceSaveDueMs_ = millis() + kInputSourceSaveDelayMs; + } + if (inputSourceSaveDueMs_ != 0 && millis() >= inputSourceSaveDueMs_) { + inputSourceSaveDueMs_ = 0; + saveInputSource(); // persist once the selection has settled + } + uint32_t save = spin_lock_blocking(mlpActive); + if (joltActive_) { + this->stepJolt(); // B2 held: morph weights, learning paused + } else { + // Ramp learning rate back up after a jolt (0 -> full over ~5s) so training + // doesn't immediately drag the net off the jolted sound. + if (joltLRRamp_ < 1.f) joltLRRamp_ = std::min(1.f, joltLRRamp_ + kJoltLRRampStep); + this->optimiseSometimes(); + } + this->generateAction(); + spin_unlock(mlpActive, save); + }); +} + + +template +void InterfaceRL::setRewardScaleInterf(float value) +{ + this->setRewardScale(value); + String msg = "Reward scale: " + String(value); + if (msgView) msgView->post(msg); +} + + + +template +void InterfaceRL::_forget_replay_mem_interf() +{ + this->forgetMemory(); + if (nnOutputsGraphView) nnOutputsGraphView->setLastAction("forget"); + static std::vector forgetmsgs = { + "Erasing my memory", "Forgetting everything", "Memory wiped","Thank you Susan?", + "Starting afresh", "Why care about the past?","Living in the moment" + }; + String msg = forgetmsgs[rand() % forgetmsgs.size()]; + + if (msgView) msgView->post(msg); +} + + +template +void InterfaceRL::bindMIDI(std::shared_ptr midi_interf, bool enableFootcontroller) +{ + if (midi_interf) { + midi_interf->SetCCCallback([this, enableFootcontroller] (uint8_t cc_number, uint8_t cc_value) { + // Route CC1-CC8 to raw_midi_ when a MIDI input source is active + bool is_midi_source = (input_source_ == INPUT_SOURCE::MIDI_1CC || + input_source_ == INPUT_SOURCE::MIDI_3CC || + input_source_ == INPUT_SOURCE::MIDI_8CC); + if (is_midi_source && cc_number >= 1 && cc_number <= 8) { + raw_midi_[cc_number - 1] = static_cast(cc_value) / 127.f; + newInput = true; + return; + } + if (!enableFootcontroller) return; + Serial.printf("MIDI CC %d: %d\n", cc_number, cc_value); + switch(cc_number) { + case 1: + { + if (cc_value > 0) this->_perform_like_action(); + break; + } + case 2: + { + if (cc_value > 0) this->_perform_dislike_action(); + break; + } + case 3: + { + if (cc_value > 0) this->_perform_randomiseRL_action(); + break; + } + case 4: + { + if (cc_value > 0) this->_forget_replay_mem_interf(); + break; + } + case 5: + { + if (midi5cb) { + midi5cb(cc_value); + + }else{ + static constexpr float cc_scale = 1.f/(127.f-20.f); + // Less than 20 on cc_value is considered 0 + // scale [20..127] to [0.0, 1.0] + if (cc_value < 20) { + cc_value = 0; + } else { + cc_value -= 20; // Shift range to [0, 107] + } + float scale = static_cast(cc_value) * cc_scale; + //this->setRewardScaleInterf(scale); + this->setNoiseLevel(scale); + } + break; + } + case 6: + { + if (midi6cb) { + midi6cb(cc_value); + + }else{ + static constexpr float cc_scale = 1.f/(127.f-20.f); + // Less than 20 on cc_value is considered 0 + // scale [20..127] to [0.0, 1.0] + if (cc_value < 20) { + cc_value = 0; + } else { + cc_value -= 20; // Shift range to [0, 107] + } + float opt = static_cast(cc_value) * cc_scale; + this->setOptimiseDivisorInterf(1.f - opt); + } + break; + } + }; + }); + } + + midi_ = midi_interf; + + if (ccSelectView && !ccSelectView->getSelectedCCs().empty()) { + midi_->SetParamCCNumbers(ccSelectView->getSelectedCCs()); + } +} + +template +void InterfaceRL::setup(size_t n_inputs, size_t n_outputs, bool addMessageView) +{ + + InterfaceBase::setup(n_inputs, n_outputs); + + mlpActive = spin_lock_init(spin_lock_claim_unused(true)); + + // The compile-time network fixes n_outputs == N_OUTPUTS; the runtime arg + // is the active mode's kN_Params, which equals N_OUTPUTS by construction. + (void)n_outputs; + + layers_nodes = { n_inputs, 16, 16, n_outputs }; + + controlInput.resize(layers_nodes[0]); + action.resize(n_outputs, 0.5f); // Initialize action vector with default values + mappingOutput.resize(n_outputs); + + //init networks — StaticMLP is a value member (fixed arch, all weights in + // the static mode object: no heap). Just initialise its weights. + // synthMapping.InitXavier(); + synthMapping.RandomiseWeightsAndBiasesLin(-1.2f,0.9f, 0, 0.5); + + rewardScale = 1.0f; // Default reward scale + + // randomiseTheNetwork(); + + // Memory limit + replayMem.setMemoryLimit(memoryLimit); + + ou_noises.reserve(n_outputs); + for(size_t i=0; i < n_outputs; i++) { + // OU(theta, mu, sigma, dt, x0). theta & dt set the *smoothness*: correlation + // time ~= 1/(theta*dt) calls = 1/(0.02*0.004) = 12500 calls ~= 62 s at the 200 Hz + // control rate, so the walk drifts in long smooth sweeps rather than per-frame + // kicks. sigma (amplitude) starts at 0 and is set by the intensity knob via + // setNoiseLevel()/setStationaryStd() on boot-sync. To make sweeps faster/coarser + // raise dt; slower/smoother, lower it. + ou_noises.push_back(std::make_unique(0.02f, 0.0f, 0.0f, kNoiseDt, 0.0f)); + } + + itemsToRemove.reserve(replayMem.getMemoryLimit()); + + joltWeightLoc_.reserve(kJoltNumWeights); + joltTarget_.reserve(kJoltNumWeights); + + // GUI + if (!nnOutputsGraphView) { + nnOutputsGraphView = std::make_shared("RL", n_outputs, 4, TFT_GREEN, 0.f, 1.f); + } + MEMLNaut::Instance()->disp->AddView(nnOutputsGraphView); + nnInputsGraphView = std::make_shared("NN Inputs", n_inputs, 10, TFT_YELLOW, 0.f, 1.f); + MEMLNaut::Instance()->disp->AddView(nnInputsGraphView); + + // memoryStoreModeView = std::make_shared("Mem Mode"); + // MEMLNaut::Instance()->disp->AddView(memoryStoreModeView); + // memoryStoreModeView->setOptions(memOptions); + // memoryStoreModeView->setNewVoiceCallback([this](size_t idx) { + // memoryStoreMode = static_cast(idx); + // }); + + if (addMessageView) { + msgView = std::make_shared("Messages"); + MEMLNaut::Instance()->disp->AddView(msgView); + } + + // 12 slots laid out 6 columns x 2 rows. Size the buttons to fill the screen: width + // 10 + 6*43 + 5*10 gap = 318px (of 320); height 78 x 2 rows clears the message line. + // Smaller font (2) so slot names fit the narrower buttons. + fileSaveView = std::make_shared("Save Model", TFT_BLUE, kNumSlots, 43, 78, + TFT_WHITE, std::vector{}, TFT_BLUE, 2 /* fontNum */); + fileSaveView->SetOnSelectCallback([this](size_t id) { + pendingSaveSlot = static_cast(id) - 1; + nameInputView->reset(slotNames[pendingSaveSlot]); + MEMLNaut::Instance()->disp->ShowDialog(nameInputView); + }); + MEMLNaut::Instance()->disp->AddView(fileSaveView); + + fileLoadView = std::make_shared("Load Model", TFT_PURPLE, kNumSlots, 43, 78, + TFT_WHITE, std::vector{}, TFT_PURPLE, 2 /* fontNum */); + fileLoadView->SetOnSelectCallback([this](size_t id) { + int slotIdx = static_cast(id) - 1; + String filename = (slotNames[slotIdx].length() > 0) ? slotNames[slotIdx] : String(id); + fileLoadView->SetMessage("Loading " + filename); + uint32_t save = spin_lock_blocking(mlpActive); + if (MEMLNaut::Instance()->startSD()) { + if (this->_load_RL_from_SD(filename)) { + fileLoadView->SetMessage("Loaded " + filename); + } else { + fileLoadView->SetMessage("Failed to load model"); + } + MEMLNaut::Instance()->stopSD(); + } else { + fileLoadView->SetMessage("SD card error - is it inserted and formatted?"); + } + spin_unlock(mlpActive, save); + }); + MEMLNaut::Instance()->disp->AddView(fileLoadView); + + nameInputView = std::make_shared("Name"); + nameInputView->setCallbacks( + [this](const String& name) { + if (pendingSaveSlot >= 0 && pendingSaveSlot < kNumSlots) { + String displayName = (name.length() > 0) ? name : String(pendingSaveSlot + 1); + slotNames[pendingSaveSlot] = name; + fileSaveView->updateButtonName(static_cast(pendingSaveSlot), displayName); + fileLoadView->updateButtonName(static_cast(pendingSaveSlot), displayName); + fileSaveView->SetMessage("Saving as " + displayName); + uint32_t save = spin_lock_blocking(mlpActive); + if (MEMLNaut::Instance()->startSD()) { + _saveSlotNames(); + if (this->_save_RL_to_SD(displayName)) { + fileSaveView->SetMessage("Saved as " + displayName); + } else { + fileSaveView->SetMessage("Failed to save model"); + } + MEMLNaut::Instance()->stopSD(); + } else { + fileSaveView->SetMessage("SD card error - is it inserted and formatted?"); + } + spin_unlock(mlpActive, save); + } + MEMLNaut::Instance()->disp->DismissDialog(); + }, + [this]() { + MEMLNaut::Instance()->disp->DismissDialog(); + } + ); + MEMLNaut::Instance()->disp->RegisterDialog(nameInputView); +} + + +template +void InterfaceRL::setModeInfo(const String& modeRoot, const String& modeTag) { + _modeRoot = modeRoot; + _modeTag = modeTag; + if (MEMLNaut::Instance()->startSD()) { + _loadSlotNames(); + MEMLNaut::Instance()->stopSD(); + } +} + +template +bool InterfaceRL::_save_RL_to_SD(String id) { + String dir = "/" + _modeRoot; + String path = dir + "/" + id + ".bin"; + + if (!SD.exists(dir.c_str())) { + SD.mkdir(dir.c_str()); + } + + auto file = SD.open(path.c_str(), FILE_WRITE); + if (!file) { + Serial.println("Failed to open file for writing: " + path); + return false; + } + file.seek(0); + + MEMLFileHeader header; + memcpy(header.magic, "MEML", 4); + header.format_version = MEML_FILE_FORMAT_VERSION; + memset(header.mode_tag, 0, sizeof(header.mode_tag)); + strncpy(header.mode_tag, _modeTag.c_str(), sizeof(header.mode_tag) - 1); + + std::vector extraData; + if (_extraSaveFn) { + extraData = _extraSaveFn(); + } + header.extra_size = static_cast(extraData.size()); + + if (file.write((const char*)&header, sizeof(header)) != sizeof(header)) { + file.close(); + return false; + } + if (!extraData.empty()) { + if (file.write(extraData.data(), extraData.size()) != extraData.size()) { + file.close(); + return false; + } + } + + bool success = synthMapping.SaveMLPNetworkToFile(file); + file.close(); + return success; +} + +template +bool InterfaceRL::_load_RL_from_SD(String id) { + String path = "/" + _modeRoot + "/" + id + ".bin"; + + auto file = SD.open(path.c_str(), FILE_READ); + if (!file) { + Serial.println("File not found: " + path); + return false; + } + + MEMLFileHeader header; + if (file.read((uint8_t*)&header, sizeof(header)) != sizeof(header)) { + file.close(); + Serial.println("File too small to contain header"); + return false; + } + if (memcmp(header.magic, "MEML", 4) != 0) { + file.close(); + Serial.println("Unrecognised file format (bad magic)"); + return false; + } + if (header.format_version > MEML_FILE_FORMAT_VERSION) { + file.close(); + Serial.println("File saved with newer firmware (version " + String(header.format_version) + ")"); + return false; + } + char expected_tag[17] = {}; + strncpy(expected_tag, _modeTag.c_str(), 16); + if (memcmp(header.mode_tag, expected_tag, 16) != 0) { + char tag_buf[17] = {}; + memcpy(tag_buf, header.mode_tag, 16); + file.close(); + Serial.println(String("Wrong mode: file is for '") + tag_buf + "'"); + return false; + } + + if (header.extra_size > 0) { + std::vector extraData(header.extra_size); + if (file.read(extraData.data(), header.extra_size) != header.extra_size) { + file.close(); + return false; + } + if (_extraLoadFn) { + _extraLoadFn(extraData.data(), header.extra_size, header.format_version); + } + } + + bool success = synthMapping.LoadMLPNetworkFromFile(file); + file.close(); + + // With a StaticMLP the architecture is fixed at compile time and + // LoadMLPNetworkFromFile already rejects (returns false) any on-card model + // whose geometry/activations don't match — so a loaded model is always + // architecture-correct. Keep a defensive rebuild for the mismatch case. + if (success && (synthMapping.get_num_inputs() != (int)controlInput.size() + || synthMapping.get_num_outputs() != (int)n_outputs_)) { + synthMapping.RandomiseWeightsAndBiasesLin(-1.2f, 0.9f, 0, 0.5f); + if (msgView) msgView->post("Model incompatible: wrong architecture"); + return false; + } + return success; +} + +template +void InterfaceRL::_saveSlotNames() { + String dir = "/" + _modeRoot; + if (!SD.exists(dir.c_str())) { + SD.mkdir(dir.c_str()); + } + String path = dir + "/slots.txt"; + auto file = SD.open(path.c_str(), FILE_WRITE); + if (!file) return; + file.seek(0); + for (int i = 0; i < kNumSlots; i++) { + file.println(slotNames[i]); + } + file.close(); +} + +template +void InterfaceRL::_loadSlotNames() { + String path = "/" + _modeRoot + "/slots.txt"; + auto file = SD.open(path.c_str(), FILE_READ); + if (!file) return; + for (int i = 0; i < kNumSlots; i++) { + String line = file.readStringUntil('\n'); + line.trim(); + slotNames[i] = line; + if (line.length() > 0) { + fileSaveView->updateButtonName(static_cast(i), line); + fileLoadView->updateButtonName(static_cast(i), line); + } + } + file.close(); +} + + +template +void InterfaceRL::optimise() { + + float lossPositive{0.f}; + float lossNegative{0.f}; + size_t batchSizeNeg=0; + const float effLR = learningRateScaled * joltLRRamp_; + + //positive batch + std::vector sample = replayMem.sampleIndices(batchSize); + if (sample.size() >1) { + //run sample through network + size_t batchSizePos=0; + float avgRewardPos=0.f; + training_pair_t tsPositive; + + // Pre-allocate to avoid repeated allocations + tsPositive.first.reserve(sample.size()); + tsPositive.second.reserve(sample.size()); + + + // Positive batch: random sample (diversity for generalisation) + for (auto &i : sample) { + if (replayMem.getItem(i).reward > 0) { + tsPositive.first.push_back(replayMem.getItem(i).input); + tsPositive.second.push_back(replayMem.getItem(i).action); + batchSizePos++; + avgRewardPos += replayMem.getItem(i).reward; + } + } + + // Post-jolt recovery: scale the LR by the ramp (0 -> 1 over ~5s after a jolt) so + // training eases back in rather than yanking the net off the jolted sound. + if (batchSizePos > 0){ + avgRewardPos /= static_cast(batchSizePos); + lossPositive = synthMapping.TrainBatch(tsPositive, effLR * avgRewardPos, 1, batchSize, 0.f, false); + // Serial.printf("[DEBUG] Loss after positive TrainBatch: %f (inf=%d, nan=%d)\n", + // lossPositive, std::isinf(lossPositive), std::isnan(lossPositive)); + } + } + + // Negative batch: scan ALL negatives so every dislike is guaranteed to push. + // No decay — a 'no' pushes at full strength until it's lived kDislikeLifetimeMs, + // then it's removed outright. + training_pair_t tsNegative; + tsNegative.first.reserve(sample.size()); + tsNegative.second.reserve(sample.size()); + // Single scan over all memory: tally positives (for display + LR ratio) and collect + // negatives (expiring any that have outlived kDislikeLifetimeMs). + float avgRewardNeg=0.f; + size_t totalPosCount=0; + const uint32_t now = millis(); + for (size_t i = 0; i < replayMem.size(); i++) { + float reward = replayMem.getItem(i).reward; + if (reward > 0.f) { totalPosCount++; continue; } + if ((now - static_cast(replayMem.getTimestamp(i))) >= kDislikeLifetimeMs) { + itemsToRemove.push_back(i); // lived its lifetime -> stop pushing, remove + continue; + } + tsNegative.first.push_back(replayMem.getItem(i).input); + tsNegative.second.push_back(replayMem.getItem(i).action); + batchSizeNeg++; + avgRewardNeg += reward; + } + if (batchSizeNeg > 0){ + + struct PosCandidate { float dist; size_t idx; }; + std::vector candidates; + candidates.reserve(replayMem.size()); + for (size_t i = 0; i < replayMem.size(); i++) { + const auto& item = replayMem.getItem(i); + if (item.reward > 0.f) + candidates.push_back({euclideanDistance(item.input, controlInput), i}); + } + std::sort(candidates.begin(), candidates.end(), + [](const PosCandidate& a, const PosCandidate& b){ return a.dist < b.dist; }); + + std::vector meanPositiveAction(action.size(), 0.f); + size_t posMemCount = 0; + const size_t kUsed = std::min(candidates.size(), kCentroidK); + for (size_t ci = 0; ci < kUsed; ci++) { + const auto& item = replayMem.getItem(candidates[ci].idx); + for (size_t j = 0; j < meanPositiveAction.size(); j++) + meanPositiveAction[j] += item.action[j]; + posMemCount++; + } + if (posMemCount > 0) { + for (auto& v : meanPositiveAction) v /= static_cast(posMemCount); + } + avgRewardNeg /= static_cast(batchSizeNeg); + + // Push each disliked action's training target strongly away from the liked + // centroid — or in a random direction when there are no likes yet. No taper: + // a 'no' should clearly move the mapping away even from a sound already far + // from the liked region (the taper used to kill exactly that case). Bigger + // kGeometricPushScale + higher negLRRatio => the sound slides away faster/further. + const bool havePositives = (posMemCount > 0); + training_pair_t tsGeometric; + tsGeometric.first = tsNegative.first; + tsGeometric.second.reserve(tsNegative.second.size()); + + float pushStep = std::clamp(fabsf(avgRewardNeg), 0.25f, 1.0f) * kGeometricPushScale; + + for (const auto& neg_action : tsNegative.second) { + // Fix 3: guard against size mismatch with old saved actions + const size_t dimCount = std::min(neg_action.size(), meanPositiveAction.size()); + float len = 0.f; + std::vector dir(dimCount); + for (size_t j = 0; j < dimCount; j++) { + dir[j] = neg_action[j] - meanPositiveAction[j]; // meanPositiveAction is 0 when no likes + len += dir[j] * dir[j]; + } + len = sqrtf(len); + const bool useRandom = !havePositives || (len <= 1e-4f); + std::vector target(neg_action); // copy keeps out-of-range dims intact + for (size_t j = 0; j < dimCount; j++) { + bool active = activeDims_.empty() || (j < activeDims_.size() && activeDims_[j]); + if (!active) continue; + float d = useRandom + ? (static_cast(rand() & 0xFF) / 127.5f - 1.f) + : (dir[j] / len); + target[j] = std::clamp(neg_action[j] + d * pushStep, 0.f, 1.f); + } + tsGeometric.second.push_back(std::move(target)); + } + // Dynamic LR ratio: push harder when dislikes are rare, gentler when they flood the buffer + const float negFraction = static_cast(batchSizeNeg) + / static_cast(std::max(batchSizeNeg + totalPosCount, size_t{1})); + const float negLRRatio = kNegLRBase - 0.4f * negFraction; + lossNegative = synthMapping.TrainBatch(tsGeometric, effLR * negLRRatio, 1, batchSizeNeg, 0.f, false); + } + + // Fix 4: always clear — stale indices corrupt subsequent optimise() calls + replayMem.removeItems(itemsToRemove); + itemsToRemove.clear(); + + if (nnOutputsGraphView) { + nnOutputsGraphView->setLoss(lossPositive); + nnOutputsGraphView->setMemoryCounts(totalPosCount, replayMem.size() - totalPosCount); + } + +} + +template +void InterfaceRL::readAnalysisParameters(std::vector params) { + for (size_t i = 0; i < params.size() && i < 6; i++) { + raw_ml_[i] = params[i]; + } + generateAction(true); +} + +template +void InterfaceRL::assembleInputs() { + switch (input_source_) { + case INPUT_SOURCE::JOYSTICK_3D: copyAndZero(raw_joystick_, 3); break; + case INPUT_SOURCE::JOYSTICK_4D: copyAndZero(raw_joystick_, 4); break; + case INPUT_SOURCE::MACHINE_LISTENING: copyAndZero(raw_ml_, 6); break; + case INPUT_SOURCE::MIDI_1CC: copyAndZero(raw_midi_, 1); break; + case INPUT_SOURCE::MIDI_3CC: copyAndZero(raw_midi_, 3); break; + case INPUT_SOURCE::MIDI_8CC: copyAndZero(raw_midi_, 8); break; + case INPUT_SOURCE::COMBINED: + memcpy(&controlInput[0], raw_joystick_, 4 * sizeof(float)); + memcpy(&controlInput[4], raw_ml_, 6 * sizeof(float)); + break; + default: break; + } +} + +template +void InterfaceRL::copyAndZero(const float* src, size_t n) { + // Pad the unused input tail with a non-zero constant instead of 0. A constant input + // only adds a fixed term (Σ_j W1[i,j]·c) to each hidden unit — i.e. a per-unit layer-1 + // bias shift — which spreads effective biases to mixed signs so units switch both on + // and off across a single-input sweep (more non-linear, direction-changing mapping). + // unusedInputDefault_ is recomputed only on input-mode change (see updateUnusedInputDefault). + size_t i = 0; + for (; i < n && i < kMaxNNInputs; ++i) controlInput[i] = src[i]; + for (; i < kMaxNNInputs; ++i) controlInput[i] = unusedInputDefault_; +} + +template +void InterfaceRL::saveInputSource() { + FILE* f = fopen(kInputSourceFile, "wb"); + if (f) { fwrite(&input_source_, sizeof(input_source_), 1, f); fclose(f); } +} + +template +void InterfaceRL::loadInputSource() { + FILE* f = fopen(kInputSourceFile, "rb"); + if (f) { fread(&input_source_, sizeof(input_source_), 1, f); fclose(f); } + updateUnusedInputDefault(); +} + +template +void InterfaceRL::addInputSourceView(bool includeCCSelect) { + static const String srcNames[] = { + "3D Joystick", "4D Joystick", "Machine Listen", + "MIDI Mod Whl", "MIDI 3 CC", "MIDI 8 CC", "Combined" + }; + std::vector available = { + INPUT_SOURCE::JOYSTICK_3D, INPUT_SOURCE::JOYSTICK_4D, + INPUT_SOURCE::MIDI_1CC, INPUT_SOURCE::MIDI_3CC, INPUT_SOURCE::MIDI_8CC + }; + if (hasMachineListening_) { + available.push_back(INPUT_SOURCE::MACHINE_LISTENING); + available.push_back(INPUT_SOURCE::COMBINED); + } + + std::vector opts; + for (auto src : available) opts.push_back(srcNames[static_cast(src)]); + + size_t initialSel = 0; + auto it = std::find(available.begin(), available.end(), input_source_); + if (it != available.end()) initialSel = std::distance(available.begin(), it); + + auto view = std::make_shared("Input Source"); + view->setOptions(std::span(opts.data(), opts.size())); + view->setSelection(initialSel); + view->setNewSelectionCallback([this, available](size_t idx) { + // Runs in the rotary-encoder ISR — defer the actual switch to the main loop. + if (idx < available.size()) requestInputSource(available[idx]); + }); + MEMLNaut::Instance()->disp->AddView(view); + + if (includeCCSelect) { + size_t maxCC = midi_ ? midi_->getParamCount() : n_outputs_; + ccSelectView = std::make_shared(maxCC, "MIDI CC Out"); + loadCCNumbers(); + ccSelectView->setOnChangeCallback([this](const std::vector& ccs) { + if (midi_) midi_->SetParamCCNumbers(ccs); + saveCCNumbers(); + }); + MEMLNaut::Instance()->disp->AddView(ccSelectView); + } +} + +template +void InterfaceRL::generateAction(bool donthesitate) { + if (newInput || donthesitate) { + newInput = false; + + assembleInputs(); + if (inputInjectionHook) inputInjectionHook(controlInput); + + if (!actionBeingDragged) { + synthMapping.GetOutput(controlInput, &mappingOutput); + for(size_t i=0; i < mappingOutput.size(); i++) { + const float noise = ou_noises[i]->sample(); + mappingOutput[i] += noise; + if (mappingOutput[i] < 0.f) { + mappingOutput[i] = fmod(-mappingOutput[i],1.f); // reflect + } else if (mappingOutput[i] > 1.f) { + mappingOutput[i] = 1.f - fmod(mappingOutput[i], 1.f); // reflect at 1.0 + } + } + } + if (paramTransformHook) paramTransformHook(mappingOutput); + SendParamsToQueue(mappingOutput); + action = mappingOutput; + nnOutputsGraphView->UpdateValues(mappingOutput, resetMinMaxFlag); + resetMinMaxFlag = false; + nnInputsGraphView->UpdateValues(controlInput, false); + } +} + +// void InterfaceRL::storeExperience(float reward) { +// std::vector state = controlInput; +// trainStatelessRLItem trainItem = {state, action, reward}; // state is s_t, action is a_t, reward is r_t, nextState is s_t +// replayMem.add(trainItem, millis()); +// } + + +template +bool InterfaceRL::removeItemsAtDistance(std::vector &experienceState, const float distThreshold, const float reward) { + std::vector indicesToRemove; + bool accumulated = false; + for(size_t i=0; i < replayMem.size(); i++) { + trainStatelessRLItem& item = replayMem.getItem(i); + float dist = euclideanDistance(item.input, experienceState); + if (dist < distThreshold) { + if (reward < 0.f && item.reward < 0.f) { + // Strengthen existing dislike rather than replacing it + item.reward = std::max(item.reward + reward, -1.0f); + accumulated = true; + } else if (reward < 0.f && item.reward > 0.f) { + // A dislike near a like: delete the like so it stops pulling the + // model back towards the disliked region. + indicesToRemove.push_back(i); + if (msgView) msgView->post("Removing nearby like"); + } else if (item.reward > 0.f && reward > 0.f) { + indicesToRemove.push_back(i); + if (msgView) msgView->post("Removing similar memory item"); + } + } + } + replayMem.removeItems(indicesToRemove); + return accumulated; +} + +template +void InterfaceRL::decayItemsAtDistance(std::vector &experienceState, const float distThreshold) { + std::vector indicesToRemove; + for(size_t i=0; i < replayMem.size(); i++) { + trainStatelessRLItem& item = replayMem.getItem(i); + float dist = euclideanDistance(item.input, experienceState); + if (dist < distThreshold) { + float decayFactor = (dist/distThreshold); + item.reward *= decayFactor; // Decay reward + if (item.reward < 0.05f) { + indicesToRemove.push_back(i); + } + if (msgView) msgView->post("Decaying memory item"); + Serial.printf("Decayed item %d reward to %f\n", i, item.reward); + } + } + replayMem.removeItems(indicesToRemove); +} + +template +void InterfaceRL::storeExperience(float reward, std::vector &experienceState, std::vector &experienceAction ) { + trainStatelessRLItem trainItem = {experienceState, experienceAction, reward * rewardScale}; // state is s_t, action is a_t, reward is r_t, nextState is s_t + bool skip_add = false; + switch(memoryStoreMode) { + case MEMORY_STORE_MODES::ADD: + break; + case MEMORY_STORE_MODES::REPLACE_5_PERCENT: + skip_add = removeItemsAtDistance(experienceState, 0.05f, trainItem.reward); + break; + case MEMORY_STORE_MODES::REPLACE_10_PERCENT: + skip_add = removeItemsAtDistance(experienceState, 0.10f, trainItem.reward); + break; + case MEMORY_STORE_MODES::REPLACE_15_PERCENT: + skip_add = removeItemsAtDistance(experienceState, 0.15f, trainItem.reward); + break; + case MEMORY_STORE_MODES::REWARD_DECAY_10_PERCENT: + decayItemsAtDistance(experienceState, 0.10f); + break; + case MEMORY_STORE_MODES::REWARD_DECAY_20_PERCENT: + decayItemsAtDistance(experienceState, 0.20f); + break; + } + if (!skip_add) replayMem.add(trainItem, millis()); + if (nnOutputsGraphView) { + size_t pos = 0; + for (size_t i = 0; i < replayMem.size(); i++) + if (replayMem.getItem(i).reward > 0.f) pos++; + nnOutputsGraphView->setMemoryCounts(pos, replayMem.size() - pos); + } +} + +template +void InterfaceRL::saveCCNumbers() { + if (!ccSelectView) return; + String path = "/" + _modeRoot + "_cc_numbers.bin"; + FILE* f = fopen(path.c_str(), "wb"); + if (f) { + const auto& ccs = ccSelectView->getSelectedCCs(); + fwrite(ccs.data(), 1, ccs.size(), f); + fclose(f); + } +} + +template +void InterfaceRL::loadCCNumbers() { + if (!ccSelectView) return; + String path = "/" + _modeRoot + "_cc_numbers.bin"; + FILE* f = fopen(path.c_str(), "rb"); + if (f) { + std::vector ccs; + uint8_t b; + while (fread(&b, 1, 1, f) == 1) ccs.push_back(b); + fclose(f); + if (!ccs.empty()) { + ccSelectView->setSelectedCCs(ccs); + return; + } + } + // Default: CC1..n_outputs + size_t nDefault = std::min(ccSelectView->getMaxActive(), (size_t)32); + std::vector defaults(nDefault); + for (size_t i = 0; i < nDefault; i++) defaults[i] = static_cast(i + 1); + ccSelectView->setSelectedCCs(defaults); +} diff --git a/firmware/MEMLNaut-NISPS/lib/memllib/reference/InterfaceRLFileFormat.hpp b/firmware/MEMLNaut-NISPS/lib/memllib/reference/InterfaceRLFileFormat.hpp new file mode 100644 index 0000000..df78cf4 --- /dev/null +++ b/firmware/MEMLNaut-NISPS/lib/memllib/reference/InterfaceRLFileFormat.hpp @@ -0,0 +1,19 @@ +#ifndef INTERFACE_RL_FILE_FORMAT_HPP +#define INTERFACE_RL_FILE_FORMAT_HPP + +#include + +static constexpr uint16_t MEML_FILE_FORMAT_VERSION = 1; + +// Binary file layout: +// MEMLFileHeader (26 bytes) +// extra_size bytes of mode-specific data (may be 0) +// MLP binary (existing format) +struct MEMLFileHeader { + char magic[4]; // always "MEML" + uint16_t format_version; // MEML_FILE_FORMAT_VERSION + char mode_tag[16]; // null-padded mode identifier, e.g. "VerbFX" + uint16_t extra_size; // bytes of mode-specific data immediately following +}; + +#endif // INTERFACE_RL_FILE_FORMAT_HPP diff --git a/firmware/MEMLNaut-NISPS/lib/memllib/reference/README.md b/firmware/MEMLNaut-NISPS/lib/memllib/reference/README.md new file mode 100644 index 0000000..7156f27 --- /dev/null +++ b/firmware/MEMLNaut-NISPS/lib/memllib/reference/README.md @@ -0,0 +1,33 @@ +# `reference/` — upstream source of truth, never compiled + +These four files are `examples/InterfaceRL.{hpp,cpp,tpp}` + +`examples/InterfaceRLFileFormat.hpp`, copied verbatim from +[`MusicallyEmbodiedML/memllib`](https://github.com/MusicallyEmbodiedML/memllib) at +commit `e291192d8e4f2fca7b79670c4df9c2ec8bdf03cd` — the same commit the rest of this +vendored tree pins (see `../VENDORED.md`). + +## Why they are here + +`InterfaceRL` is the reference implementation of the whole NISPS feedback subsystem. +`nisps/ml/geo_push.hpp`, `nisps/ml/replay.hpp`, `nisps/ml/feedback.hpp`, +`nisps/ml/jolt.hpp` and `nisps/ml/ou_noise.hpp` are all ports of it, and several of them +still carry `// upstream InterfaceRL.hpp:NNN` line references. + +The Phase-4 vendoring dropped `examples/` because nothing compiled it. That was correct +for the build and wrong for the codebase: with the source of truth out of tree, upstream +redesigned the geometric dislike (deleted the `/(1+len)` taper, doubled +`kGeometricPushScale`, tripled the negative-LR base, moved to batch training over all +negatives every tick) and we did not notice for months. Keeping these files in-tree turns +the next upstream drift into a `diff` instead of an archaeology session. + +## Rules + +- **Never compiled.** This directory sits OUTSIDE `../src/`, and PlatformIO's Library + Dependency Finder only recursively compiles an Arduino-format library's `src/` folder + (`../VENDORED.md` explains that mechanism at length). Do not move these under `src/` + and do not add them to any build. +- **Never edited.** They are upstream's bytes. Our behaviour lives in `nisps/ml/`. A + divergence from upstream is a decision recorded in `ALIGNMENT.md` or a task, not an + edit here. +- **Re-sync with the rest of the tree**, in the same step and to the same commit — + `../VENDORED.md` § "Re-syncing with upstream".