9ad1f78ddd
Roughly 20k lines were deleted from this repo in the last week and much of the
corpus still described the pre-deletion world in the present tense. Executes
the §8 table: archive the retired, reclassify the executed, prune the stale.
aimmersive-clone-spec -> _archive/ with a deprecated-by note
feedback-modes-port-spec -> plans/, kind: plan, status: executed
manifold-parity-features -> plans/, kind: plan, status: active
playground-2.0-rewrite -> status: superseded
engine-architecture 434 -> ~120 lines; seam + spine kept, rewritten
present-tense against the shipped engine/
MAIN.md six contradicted claims fixed; registry resynced
vcv-module.md pruned to the current 8->16 contract and made the
single .nisps format spec
vcv/NISPS-FORMAT.md DELETED — documented a v1 format that no longer loads
vcv/README.md, BUILDING.md rewritten to the real contract, menu, OSC table
inputs/backends/dock trio grounding sections marked historical, dead cites fixed
Two rows of the §8 table were themselves wrong, corrected here: the deleted
full-state sync lives in backends-spec.md §6.3, not vcv-module.md (which has no
§6.3), and codegen/README.md was already a MAP pointer with no port-solidjs
trigger left to remove.
Beyond the table — found by sweeping every backticked path in the changed docs
against `git ls-files`, which is how these should have been caught before:
manifold/ONBOARDING.md documented a UI that Phase 1 deleted, as if current:
SplitStage, ReadoutStrip, InputMini, BackendAdvanced, AltitudeNav, and a
shot.spec.ts that does not exist. The whole stage table was keyed on a `focus`
axis that no longer exists — selection is now sandwich > particles >
composite. This matters more than the rest: CLAUDE.md tells every agent to
read ONBOARDING.md first for Manifold work, so it was actively teaching a
fiction. Rewritten against ConsoleApp.tsx.
MAP.md claimed the input layer reduces axes to the engine arity with an
"even/odd blend". input-layer.ts says the opposite in its own header: one
dedicated slot per axis, 1:1, into a 32-input over-provisioned head, and
mean-blending was removed deliberately because it diluted every source.
AGENT-REFERENCE.md still promised TS emission "returns at P5" (landed),
per-mode dims "become schema-real at P5" (landed at P5.3), and pointed at
nisps::FixedBuffer (deleted).
Doc-right/code-suspect, filed rather than fixed: VCV computes derivedMean/Std/
Delta and cachedNovelty behind a live context-menu toggle that nothing reads;
vcv/plugin.json points at the MusicallyEmbodiedML org rather than this repo's
origin; and the module defaults to UDP 7001+id%64 while bridge.ts defaults to
9000, so out of the box they do not meet.
Firmware-build docs are deliberately untouched — the PlatformIO migration
lands next and rewrites all of them.
238 lines
10 KiB
Markdown
238 lines
10 KiB
Markdown
# MEMLNaut for VCV Rack
|
|
|
|
MEMLNaut is a CV-to-CV mapper powered by a neural network that you train in real time using
|
|
reinforcement learning. Patch any CV sources into its 8 inputs, connect its 16 outputs to your
|
|
synth parameters, and shape the mapping by giving thumbs-up/thumbs-down feedback. The module
|
|
learns your preferences, producing complex, nonlinear modulation that evolves with your taste.
|
|
It does not generate sound itself -- it generates control voltages.
|
|
|
|
## Installation
|
|
|
|
MEMLNaut is built from source against the VCV Rack 2 SDK. Pre-built `.vcvplugin` bundles are
|
|
published at https://meml.lnfinitemonkeys.org/next/vcv/ (locally-built bundles land in the
|
|
git-ignored `dist/`; see `DISTRIBUTION.md`).
|
|
|
|
### Requirements
|
|
|
|
- VCV Rack 2 (Community Edition or Pro)
|
|
- VCV Rack SDK (v2.x)
|
|
- C++20 compiler (GCC 10+, Clang 12+, or MSVC 2019+)
|
|
- GNU Make
|
|
|
|
### Build
|
|
|
|
```bash
|
|
git clone --recursive https://github.com/monkey-w1n5t0n/MEMLNaut-NISPS.git
|
|
cd MEMLNaut-NISPS/vcv
|
|
|
|
# Point to your Rack SDK (or set in environment)
|
|
export RACK_DIR=/path/to/Rack-SDK
|
|
|
|
make
|
|
make install # copies plugin to your VCV Rack plugins directory
|
|
```
|
|
|
|
The default `RACK_DIR` is `~/.local/share/Rack2/Rack-SDK`. If your SDK lives there, you can skip
|
|
the export.
|
|
|
|
The ML engine is the repository's shared `nisps/` C++20 core, reached via relative includes from
|
|
`src/iml.hpp` -- the plugin must be built from a full repo checkout. See `BUILDING.md`.
|
|
|
|
## Quick Start
|
|
|
|
1. Add **MEMLNaut** from the module browser.
|
|
2. Patch LFOs (or any CV source) into one or more of the 8 **IN** jacks.
|
|
3. Connect several of the 16 outputs to parameters on your synth voice -- filter cutoff,
|
|
oscillator pitch, waveshape, VCA level, etc.
|
|
4. Press **RAND** to randomize the network. You should hear your synth respond as the LFOs sweep.
|
|
5. Flip the **LEARN** switch on.
|
|
6. When you hear something you like, press **+** (thumbs up). When you hear something you
|
|
dislike, press **-** (thumbs down).
|
|
7. Keep exploring. The mapping will converge toward sounds you prefer.
|
|
|
|
## Panel Controls
|
|
|
|
### Knobs
|
|
|
|
| Control | Description |
|
|
|---------|-------------|
|
|
| **SPREAD** | Controls weight initialization scale, RL noise amplitude, and weight decay. Low values produce extreme, polarized mappings. High values produce balanced, subtle mappings. Default: 60%. |
|
|
| **RATE** | Inference rate. Full CCW = ~170 Hz (block rate, cheapest). Noon = ~2.8 kHz (good for CV). Full CW = 44.1 kHz (audio rate, most expensive). |
|
|
|
|
### Buttons
|
|
|
|
| Control | Description |
|
|
|---------|-------------|
|
|
| **+** | Thumbs up. Captures the current input/output pair as a training example and trains the network. Requires LEARN to be enabled. |
|
|
| **-** | Thumbs down. Increases exploration noise and perturbs the network weights to try something different. Requires LEARN to be enabled. |
|
|
| **LEARN** | Toggle switch. Enables/disables RL feedback. When off, the module still runs inference -- it just ignores +/- presses. |
|
|
| **RAND** | Randomize all network weights (using current SPREAD setting). |
|
|
| **CLEAR** | Long-press (~1 second) to clear all training examples and reset the network. |
|
|
|
|
### Inputs
|
|
|
|
| Port | Description |
|
|
|------|-------------|
|
|
| **IN 1-8** | CV inputs feeding the 8-input network. Default range: 0-10V (unipolar); per-input bipolar toggle in the context menu. |
|
|
| **SPREAD CV** | CV modulation of the SPREAD knob (added to knob value, 0-10V). |
|
|
| **LEARN** | Gate input. High = enable learning. Works alongside the LEARN toggle (either enables it). |
|
|
| **+ TRIG** | Trigger input for thumbs-up. Alternative to pressing the + button. |
|
|
| **- TRIG** | Trigger input for thumbs-down. Alternative to pressing the - button. |
|
|
|
|
### Outputs
|
|
|
|
| Port | Description |
|
|
|------|-------------|
|
|
| **OUT 1-16** | Raw MLP outputs. Default: 0-10V unipolar (per-output bipolar toggle in the context menu). |
|
|
|
|
Each output has a **trimpot attenuverter** (-100% to +100%) for scaling and inverting individual
|
|
outputs without external modules, and is surrounded by an **LED ring** whose arc fill tracks the
|
|
output's current level (color-coded per output, orange-to-cyan ramp).
|
|
|
|
### LEDs & Display
|
|
|
|
| Indicator | Meaning |
|
|
|-----------|---------|
|
|
| **LEARN** (green) | Lit when learning is enabled. |
|
|
| **TRAIN** (yellow) | Flashes during background training. |
|
|
| **Output LED rings** | Arc fill tracks each output's current level. |
|
|
|
|
The built-in display shows all 16 output levels as color-coded bars, the current noise level
|
|
(N:), the example count (e.g. `42/100`), and "TRAIN" during active training.
|
|
|
|
## RL Workflow
|
|
|
|
The reinforcement learning loop works like this:
|
|
|
|
1. **Start exploring.** Patch LFOs or sequencers into the IN jacks. Connect outputs to
|
|
interesting synth parameters. Press RAND a few times to hear different random mappings.
|
|
|
|
2. **Enable learning.** Flip the LEARN switch on (or send a gate to the LEARN input).
|
|
|
|
3. **Thumbs up (+)** when you like what you hear. This:
|
|
- Saves the current input position and output values as a training example
|
|
- Trains the network to reproduce this mapping
|
|
- Slightly reduces exploration noise (the network becomes more "settled")
|
|
|
|
4. **Thumbs down (-)** when you dislike what you hear. This:
|
|
- Increases exploration noise
|
|
- Perturbs the network weights to try a different mapping
|
|
- Does NOT save any training example
|
|
|
|
5. **Repeat.** Over time, the network learns to produce outputs you tend to like across the
|
|
input space. Regions near your thumbs-up examples will be stable; distant regions remain
|
|
exploratory.
|
|
|
|
6. **Disable learning** when you are happy with the mapping. The module continues running
|
|
inference with the trained network. You now have a complex, personalized CV source.
|
|
|
|
### Tips
|
|
|
|
- Give thumbs-up at several different input positions to teach the network about different
|
|
regions of the input space.
|
|
- The network holds up to 100 examples. Oldest examples are dropped when full (FIFO).
|
|
- Use the SPREAD knob to control how wild the exploration is. Low spread = dramatic changes.
|
|
High spread = subtle refinements.
|
|
|
|
## Context Menu
|
|
|
|
Right-click the module to access these settings:
|
|
|
|
- **Output ranges** -- toggle each of the 16 outputs between unipolar (0-10V, default) and
|
|
bipolar (+/-5V).
|
|
- **Input ranges** -- toggle each of the 8 inputs between unipolar (0-10V, default) and bipolar
|
|
(+/-5V). Set to bipolar if your source produces +/-5V signals (e.g., standard LFOs).
|
|
- **Compute derived stats (Mean/Std/Delta)** -- internal statistics toggle (no dedicated output
|
|
jacks in the current module).
|
|
- **Output slew** -- smoothing time applied when network weights change (after training or
|
|
perturbation). Prevents clicks from sudden output jumps. Options: 0, 5, 10, 20, 50, 100 ms.
|
|
Default: 10 ms.
|
|
- **Presets (.nisps)** -- save/load the full module state (weights, training examples, knob
|
|
positions, ranges) to/from a `.nisps` JSON file.
|
|
- **Browser bridge (WS<->OSC)** -- enable the OSC server for the companion browser app; choose
|
|
the listen port (7001, 7002, 7003, 9000, 9001).
|
|
|
|
## Presets (.nisps)
|
|
|
|
MEMLNaut uses `.nisps` files for saving and sharing trained networks. The format (version 3,
|
|
flat core-exact weight vector) is specified in `docs/specs/vcv-module.md` -- the single source of
|
|
truth. Files saved by pre-2026-07 builds (version 1, nested weight arrays) no longer load.
|
|
|
|
Full module state is also saved automatically with your VCV Rack patch file; you do not need
|
|
`.nisps` exports to preserve work between sessions.
|
|
|
|
## Browser Integration (Manifold)
|
|
|
|
MEMLNaut can be driven live from the Manifold browser app (`manifold/`, VCV backend) over a
|
|
WebSocket-to-OSC bridge (`manifold/osc-bridge/bridge.ts`, Deno).
|
|
|
|
1. Right-click the module and enable **OSC server** (default port 7001; each module instance
|
|
offsets its default port automatically).
|
|
2. Run the bridge with `--osc-port` matching the module's port.
|
|
3. In Manifold, select the VCV backend.
|
|
|
|
What flows over the wire (the complete protocol -- see `docs/specs/vcv-module.md`):
|
|
|
|
| OSC Address | Direction | Content |
|
|
|-------------|-----------|---------|
|
|
| `/nisps/input` | Web -> VCV | Input vector (browser drives the model: "bridged mode") |
|
|
| `/nisps/output` | VCV -> Web | Current output values (~10 times/sec) |
|
|
| `/nisps/input` | VCV -> Web | Current input values (~10 times/sec) |
|
|
| `/nisps/feedback` | Web -> VCV | Verdict ops (up / down / rand / clear) |
|
|
|
|
Training is bidirectional: both the browser verdict loop and the module's panel buttons drive
|
|
the same network. There is no weight/state sync channel -- persistence belongs to the Rack patch.
|
|
|
|
## Technical Details
|
|
|
|
### Network architecture
|
|
|
|
```
|
|
Inputs: 8
|
|
Hidden: 16 -> 24 -> 16 (3 hidden layers, ReLU activation)
|
|
Output: 16 (sigmoid activation, producing values in [0, 1])
|
|
```
|
|
|
|
The ML engine is the repository's shared `nisps/` core
|
|
(`nisps::ml::MLPCore<DynamicStorage>` behind the thin adapter `src/iml.hpp`) -- bit-identical
|
|
training/inference semantics to the MEMLNaut firmware and the browser WASM engine.
|
|
|
|
### Inference
|
|
|
|
The MLP runs in the VCV `process()` callback. The RATE knob controls how often inference runs,
|
|
from once per 256 samples (~170 Hz) to every sample (44.1 kHz). Between inference steps, outputs
|
|
are linearly interpolated to avoid staircase artifacts.
|
|
|
|
### Threading
|
|
|
|
- **Audio thread**: Reads CV inputs, runs MLP inference, writes CV outputs. Never blocks.
|
|
- **Worker thread**: Handles training (thumbs-up) and weight perturbation (thumbs-down). When
|
|
complete, signals the audio thread to crossfade to the new outputs.
|
|
- Each module instance has its own independent worker thread and ML engine.
|
|
|
|
## Building
|
|
|
|
See `BUILDING.md` (build steps, troubleshooting) and `DISTRIBUTION.md` (packaging `.vcvplugin`
|
|
bundles, cross-platform builds).
|
|
|
|
### Project structure
|
|
|
|
```
|
|
vcv/
|
|
Makefile # Build configuration (C++20; core via relative includes)
|
|
plugin.json # Plugin manifest (name, version, tags)
|
|
src/
|
|
plugin.cpp # Plugin initialization
|
|
plugin.hpp # Plugin globals
|
|
MEMLNaut.cpp # Module logic, widget/UI, serialization, OSC wiring
|
|
iml.hpp # Thin adapter over the shared nisps/ ML core
|
|
osc_server.hpp # OSC bridge server (transport only)
|
|
LedRing.hpp # Per-output LED ring widget
|
|
palette.hpp # Color palette from the frontend design tokens
|
|
res/
|
|
MEMLNaut-wide.svg # 44HP panel (the one the module loads)
|
|
MEMLNaut.svg # Unused 30HP variant
|
|
MEMLNaut-expander.svg # Unused 8HP expander panel
|
|
dist/ # Built .vcvplugin bundles
|
|
dep/ # Build dependencies
|
|
```
|