memlnaut-nisps/vcv/SPEC.md

# MEMLNaut VCV Rack Module — Specification

## Overview

A VCV Rack module that embeds the NISPS interactive ML engine (nisps-core C++ library) as a CV-to-CV mapper. Users explore high-dimensional parameter spaces via reinforcement learning feedback, producing 12 raw CV outputs and 5 derived meta-signals from configurable CV inputs.

The module does **not** produce sound. It maps input CVs through a trained neural network to output CVs, which the user patches into other modules (VCOs, VCFs, VCAs, etc.). The result: a learned, nonlinear, high-dimensional modulation source shaped by the user's aesthetic preferences.

**Plugin name:** MEMLNaut
**Module name:** MEMLNaut (initially single module, future modules possible)
**License:** Proprietary / undecided (will not be submitted to VCV Library initially)
**Target:** VCV Rack 2 (primary), VCV Rack Free v1 (compatibility where feasible)

---

## Architecture

### Core Stack

```
┌─────────────────────────────────────┐
│          VCV Module (UI + I/O)      │
│  Panel, knobs, ports, display       │
├─────────────────────────────────────┤
│        VCV process() callback       │
│  Reads CV inputs, writes CV outputs │
│  Decimated inference trigger        │
├─────────────────────────────────────┤
│         nisps-core (C++20)          │
│  IML → MLP → forward inference     │
│  Background thread: training        │
├─────────────────────────────────────┤
│          State Manager              │
│  Serialize/deserialize weights,     │
│  examples, config to JSON           │
└─────────────────────────────────────┘
```

### Threading Model

- **Audio thread** (`process()`): Reads input CVs, runs MLP inference (decimated), writes output CVs. Never blocks.
- **Background thread**: Handles training (SGD/RMSProp). On completion, atomically swaps weight buffer into the inference path.
- **Widget thread**: Draws UI, handles user interaction (buttons, knobs). Reads output values for display.

Weight double-buffering: inference reads from buffer A while training writes to buffer B. Atomic pointer swap on training completion.

---

## I/O Specification

### Inputs (Configurable: 2–8, default 2)

| Port | Default Label | Notes |
|------|---------------|-------|
| IN 1 | X | Primary input CV |
| IN 2 | Y | Primary input CV |
| IN 3–8 | IN 3–8 | Hidden by default, shown when enabled |
| LEARN | Learn | Gate input: when high, RL feedback is accepted |
| + TRIG | Positive | Trigger input: register thumbs-up |
| − TRIG | Negative | Trigger input: register thumbs-down |

- All CV inputs normalized to [0, 1] internally (0–10V → [0,1] or ±5V → [0,1] depending on input mode)
- LEARN gate has a corresponding panel toggle button (either/or — gate OR button enables learning)
- +/− triggers work only when LEARN is enabled (gate high OR toggle on)

### Outputs (17 total: 12 raw + 5 derived)

| Port | Type | Description |
|------|------|-------------|
| OUT 1–12 | Raw MLP | Direct MLP output activations, scaled to configured CV range |
| MEAN | Derived | Mean of the 12 raw outputs |
| SPREAD | Derived | Standard deviation of the 12 raw outputs |
| DELTA | Derived | Rate of change (L2 norm of output difference from previous inference) |
| NOVELTY | Derived | Gate: fires when current input is far from all training examples |
| CONFIDENCE | Derived | Inverse of loss on nearest training example (high = near trained region) |

Each output has:
- Per-output range configuration (0–10V unipolar or ±5V bipolar) via context menu
- Small attenuverter knob for fine-tuning range/polarity

### Panel Controls

| Control | Type | Description |
|---------|------|-------------|
| SPREAD | Knob | Controls weight init scale, RL noise scaling, weight decay (see webapp spec) |
| RATE | Knob | Inference rate: from block-rate (~170Hz) to audio-rate (44.1kHz) |
| + | Momentary button | Thumbs up (register positive RL feedback) |
| − | Momentary button | Thumbs down (register negative RL feedback) |
| LEARN | Toggle button + LED | Enable/disable learning (mirrors LEARN gate input) |
| RAND | Momentary button | Randomize network weights |
| CLEAR | Momentary button (long-press) | Clear all examples and reset network |

### Advanced Controls (Right-Click Context Menu)

| Setting | Description |
|---------|-------------|
| Input count | Number of CV inputs (2–8). **Warning: changing rebuilds MLP and clears all state.** |
| Noise level | Manual override for RL exploration noise (default: auto from spread) |
| Decay rate | Weight decay per RL step (default: auto from spread) |
| Learning rate | MLP training learning rate |
| Max iterations | Training iteration cap |
| Per-output range | Unipolar (0–10V) or Bipolar (±5V) for each output |

---

## Visual Feedback

### Primary Display (Custom OpenGL Widget)

The module includes a real-time rendered display area showing:

**Option A — Full Custom Display:**
- 12 vertical bars showing raw output levels (color-coded)
- Neuron activation heatmap (simplified MLP visualization)
- Training state indicator (idle / training / converged)
- Example count
- Current noise level

**Option B — Bars + Input Position:**
- 12 vertical bars showing raw output levels
- Small 2D dot plot showing current input position (XY scope style)
- Training state, example count, noise level as text overlays

Both options to be prototyped; converge based on usability and CPU cost.

### LED Indicators

- Per-output LEDs showing signal level (brightness = voltage)
- LEARN LED (green when active)
- Training activity LED (flashes during training)

---

## MLP Configuration

### Default Network

```
Inputs:  2 (+ bias = 3 input nodes)
Hidden:  [16, 24, 16]  (3 hidden layers, ReLU activation)
Output:  12 (sigmoid activation, maps to [0, 1])
```

Significantly smaller than the webapp's [3, 32, 48, 64, 126] — appropriate for 12 outputs and real-time inference constraints.

### When Input Count Changes

- User selects new input count from context menu
- Confirmation dialog: "This will reset the network and clear all training data. Continue?"
- On confirm: rebuild MLP with new input layer size, clear dataset, randomize weights

### Spread Parameter

Identical behavior to webapp (see CLAUDE.md for full spec):

1. **Weight initialization**: `drawWeights(spread)` — scales from uniform [-1,1] (spread=0) to Xavier 1/√fan_in (spread=1)
2. **RL noise**: `moveWeights(speed, spread)` — noise cap from 0.3 (spread=0) to 0.05 (spread=1), per-layer scaling
3. **Weight decay**: 0% (spread=0) to 10% per step (spread=1)
4. **Noise cap**: `0.3*(1-spread) + 0.05*spread`

---

## Inference Rate

User-configurable via RATE knob:

| Position | Rate | Behavior |
|----------|------|----------|
| Full CCW | ~170 Hz | Once per VCV process block (256 samples). Cheapest. |
| 12 o'clock | ~2 kHz | Every ~22 samples. Good for CV-rate modulation. |
| Full CW | 44.1 kHz | Every sample. Audio-rate CV. Most expensive. |

Between inference steps, output values are linearly interpolated (slew) to avoid staircase artifacts.

---

## RL Feedback Workflow

### Thumbs Up (+)

1. Capture current input vector and output vector
2. Add as training example to dataset
3. Enqueue training on background thread
4. Decay noise: `noiseLevel *= 0.97`

### Thumbs Down (−)

1. Increase noise: `noiseLevel = min(noiseLevel * 1.5, noiseCap)`
2. Perturb weights: `mlp.moveWeights(noiseLevel, spread)`
3. Re-run inference to produce new exploration output

### Learn Enable Gate

- When LEARN is disabled (gate low AND toggle off): +/− buttons and triggers are ignored. Inference still runs normally.
- When LEARN is enabled: +/− feedback is accepted.
- "Learn off = play mode" — the module always runs inference. Learning only controls whether feedback is registered.

---

## State Persistence

### Patch Save/Load

Full state serialized into VCV patch JSON:

```json
{
  "inputCount": 2,
  "spread": 0.6,
  "inferenceRate": 0.5,
  "noiseLevel": 0.1,
  "outputRanges": [{"unipolar": true, "attenuation": 1.0}, ...],
  "weights": [[...], ...],
  "examples": {"features": [[...]], "labels": [[...]]},
  "mlpConfig": {"layers": [3, 16, 24, 16, 12], "activations": ["relu", "relu", "relu", "sigmoid"]}
}
```

### Preset Files (.nisps)

- **Save**: Export current state to a `.nisps` JSON file (same format as patch state)
- **Load**: Import from `.nisps` file via right-click menu → "Load preset..."
- **Location**: User-chosen, no enforced directory
- Enables sharing trained networks between patches and with the companion webapp

---

## Companion Webapp Integration

### Bidirectional State Transfer

**File-based (offline):**
- Webapp: "Export .nisps" button → downloads JSON file
- VCV: Right-click → "Load .nisps preset" → imports weights + examples + config
- VCV: Right-click → "Save .nisps preset" → exports for webapp import
- Webapp: "Import .nisps" → loads and continues training

**OSC-based (live):**
- VCV module runs an OSC server (configurable port, default 9000)
- Webapp connects via WebSocket → OSC bridge
- Messages:
  - `/nisps/weights` — full weight transfer (either direction)
  - `/nisps/examples` — example set transfer
  - `/nisps/state` — full state sync
  - `/nisps/input` — current input values (for webapp visualization)
  - `/nisps/output` — current output values (for webapp visualization)

### Webapp Modifications Required

- Add .nisps file import/export buttons
- Add OSC client mode (connect to VCV module)
- Network size configuration to match VCV (12 outputs vs 126)
- Shared .nisps file format specification

---

## Panel Layout (Prototyping Phase)

Three panel width variants to prototype:

### Compact (20HP)
```
┌──────────────────────┐
│     MEMLNaut         │
│  ┌────────────────┐  │
│  │   DISPLAY      │  │
│  │  (bars + dot)  │  │
│  └────────────────┘  │
│                      │
│  SPREAD    RATE      │
│  [knob]    [knob]    │
│                      │
│  [+] [−] [LEARN]    │
│  [RAND]  [CLEAR]    │
│                      │
│  IN1  IN2  LEARN TRG│
│  (o)  (o)  (o)      │
│  +TRG  −TRG         │
│  (o)   (o)           │
│                      │
│  1  2  3  4  5  6   │
│  (o)(o)(o)(o)(o)(o)  │
│  7  8  9 10 11 12   │
│  (o)(o)(o)(o)(o)(o)  │
│  MN SP DL NV CF     │
│  (o)(o)(o)(o)(o)     │
└──────────────────────┘
```
No individual attenuverters. Outputs tightly packed.

### Standard (30HP)
```
┌──────────────────────────────────┐
│          MEMLNaut                │
│  ┌──────────────────────────┐   │
│  │       DISPLAY             │   │
│  │  (bars + XY + metrics)    │   │
│  └──────────────────────────┘   │
│                                  │
│  SPREAD     RATE                 │
│  [knob]     [knob]               │
│                                  │
│  [+]  [−]  [LEARN]  [RAND]     │
│                                  │
│  IN:  (1) (2)    LEARN (+) (−) │
│                                  │
│  OUT:                            │
│  1[a](o) 2[a](o) 3[a](o) 4[a](o)│
│  5[a](o) 6[a](o) 7[a](o) 8[a](o)│
│  9[a](o) 10[a](o) 11[a](o) 12[a]│
│  MN(o) SP(o) DL(o) NV(o) CF(o) │
└──────────────────────────────────┘
```
[a] = small attenuverter knob per output.

### Wide (44HP)
Full display, all attenuverters, room for 8 input jacks, OpenGL network visualization.

### Expander Module (16HP)
Adds: 6 extra input jacks, per-output attenuverters, secondary display.

---

## Build System

### VCV Rack 2 Plugin Structure

```
vcv/
├── plugin.json          # Plugin manifest
├── Makefile             # VCV SDK Makefile
├── src/
│   ├── plugin.hpp       # Plugin globals
│   ├── plugin.cpp       # Plugin init
│   ├── MEMLNaut.cpp     # Module logic (process, state, threading)
│   └── MEMLNautWidget.cpp  # Panel UI (widgets, display, layout)
├── res/
│   ├── MEMLNaut.svg     # Panel artwork
│   └── components/      # Custom SVG components
└── dep/
    └── nisps-core/      # Symlink or copy of nisps-core headers
```

### Dependencies

- **VCV Rack SDK** (v2.x)
- **nisps-core** (header-only, C++20, already in this repo)
- No other external dependencies

### Build Commands

```bash
cd vcv
export RACK_DIR=/path/to/Rack-SDK
make
make install  # Copies to VCV plugin directory
```

---

## Development Phases

### Phase 1: Skeleton (get it compiling)
- VCV plugin scaffold from template
- Integrate nisps-core headers
- Empty module that appears in VCV module browser
- 2 input ports, 12 output ports, no processing

### Phase 2: Core Engine
- Wire CV inputs → IML → CV outputs
- MLP inference in process() callback (fixed rate)
- Spread knob controlling weight initialization
- Randomize button

### Phase 3: RL Feedback
- +/− buttons on panel
- +/− trigger inputs
- Learn toggle + gate input
- Background thread training with weight double-buffering
- Noise level tracking

### Phase 4: Visual Feedback
- Custom display widget (prototype both bar graph and full visualization)
- LED indicators per output
- Training state display

### Phase 5: Configurability
- Inference rate knob with interpolation
- Per-output range configuration (unipolar/bipolar)
- Attenuverter knobs
- Input count configuration with MLP rebuild

### Phase 6: Persistence
- Full state serialization to patch JSON
- .nisps preset file save/load
- Right-click menu integration

### Phase 7: Derived Outputs
- Mean, spread, delta, novelty, confidence computations
- 5 additional output ports

### Phase 8: Companion Webapp Bridge
- .nisps file format shared between webapp and VCV
- Webapp import/export buttons
- OSC server in VCV module
- OSC client in webapp

### Phase 9: Panel Variants
- Prototype compact, standard, wide, expander layouts
- User testing, converge on final layout

### Phase 10: Polish & Distribution
- Panel artwork / graphic design
- Performance optimization
- VCV Rack v1 compatibility pass
- Documentation
- Distribution packaging

---

## Open Questions (to resolve during implementation)

1. **MLP hidden layer sizing**: [16, 24, 16] is a guess. May need tuning based on real-world training performance with 12 outputs.
2. **Novelty/confidence thresholds**: How to calibrate the novelty gate and confidence output. May need user-adjustable sensitivity.
3. **OSC port conflicts**: What if multiple MEMLNaut instances run in the same patch? Per-instance port assignment?
4. **V1 compatibility**: How much of the v2-specific API (polyphonic ports, new widget system) do we actually use? Determines v1 compat effort.
5. **Attenuverter UX**: Tiny trimpots on a VCV panel can be fiddly. May need to test whether attenuverters per output are actually useful vs. just using VCV's built-in attenuverter modules.
6. **Training convergence with 12 outputs**: The webapp trains 126 outputs — RL feedback on 12 is a different dynamic. May converge faster or feel less "exploratory".