24057e7b5f
Phase-1 audits (feedback-behaviour, engine-surface, design/Manifold), the judge-panelled rl-feedback-design, dock/backends/inputs specs, the upstream-firmware survey, the Manifold design mirror, and BUILD-PLAN.md (resume anchor + locked decisions).
502 lines
42 KiB
Markdown
502 lines
42 KiB
Markdown
# Output Backends — Specification (`docs/redesign/backends-spec.md`)
|
||
|
||
*Workstream E. Design-only, read-only audit 2026-06-27. The new app is Vite + React + TS in `manifold/`, wired to the parity-tested TS engine lifted from `playground/src`. British spelling in product copy. The built-in synth is the **"Powerful Synth Engine"** — the string "C15" must never reach the user.*
|
||
|
||
> **Naming guard (non-negotiable).** The codename `C15` survives only in internal module/file names that the user never sees (`c15-adapter.js`, `c15-bridge.js`). Every label, tooltip, dock entry, menu item, status string, and aria-label says **"Powerful Synth Engine"** (or just "Synth"). A lint allowlist + a Playwright assertion (`expect(page).not.toContainText('C15')`) enforce this across `manifold/` and the VCV panel SVG/strings.
|
||
|
||
---
|
||
|
||
## 0. The one idea: backends are adapters behind one interface
|
||
|
||
Today the deployed app (`js/a-app.js`) fans output out to four ad-hoc sinks inline in `routeOutputs()` (`a-app.js:2425`): synth (`activeEngine.setParam`), MIDI CC (`midiOutput.sendBatch`), audio-canvas, and visual (`visualizer.setParams`). Each has its own throttle, dead-zone, and override handling copy-pasted. That fan-out *is* the debt this workstream removes.
|
||
|
||
**Replace it with one `OutputBackend` interface and a registry. Exactly one backend is "active" at a time, chosen in the Console dock.** The reactive spine (per `engine-architecture.md` §2 and `findings-design-and-manifold.md` §4) ends in a single side-effect that calls `activeBackend.send(routedOutput)`. Swapping backends swaps nothing else — the input pipeline, ML, output pipeline, training loop, and verdict loop are all backend-agnostic.
|
||
|
||
```
|
||
gesture → input pipeline → ML (WasmIML) → output pipeline → [ activeBackend.send(routed) ]
|
||
│
|
||
┌───────────────┬───────────────┬───────────────┬─────────────┼──────────────┐
|
||
WebAudioBackend ParticleBackend WebMidiBackend OscBridgeBackend CvGateBackend VcvBridgeBackend
|
||
(Powerful Synth) (flow field) (advanced CC) (paths+ranges) (1V/oct+gate) (8→model→16, LED rings)
|
||
```
|
||
|
||
The active backend is a property of the **output dock** (the Console's right rail / a-immersive's Mode drawer). Backends self-describe (id, label, capability) so the dock renders a picker without hard-coding the list.
|
||
|
||
---
|
||
|
||
## 1. The `OutputBackend` adapter interface (TS)
|
||
|
||
Lives at `manifold/src/engine/backends/backend.ts`. The engine never imports a concrete backend; it imports the interface + the registry. Concrete backends may import the engine's pure helpers (curves, param-map data) but **never** React.
|
||
|
||
```ts
|
||
// manifold/src/engine/backends/backend.ts
|
||
|
||
/** What a backend needs to know about the active mode to map outputs. */
|
||
export interface BackendContext {
|
||
modeId: string;
|
||
outputCount: number; // model output dims actually in use (≤ 126)
|
||
paramMeta: ReadonlyArray<ParamMeta>; // name/label/min/max/curve/group per output
|
||
sampleRate?: number; // for audio backends
|
||
audioContext?: AudioContext; // lazily provided; only audio backends use it
|
||
}
|
||
|
||
export interface ParamMeta {
|
||
id: string; // stable machine id, e.g. 'Env_A_Att'
|
||
label: string; // user-facing
|
||
min: number; // baseline range floor (normalised 0..1 maps here)
|
||
max: number; // baseline range ceil
|
||
curve: number; // 0..1, 0.5 = linear (see §3 universal mapping)
|
||
group: string; // for colour grouping (LED rings, heatmap)
|
||
}
|
||
|
||
export type BackendId =
|
||
| 'synth' | 'particles' | 'midi' | 'osc' | 'cvgate' | 'vcv';
|
||
|
||
export interface BackendDescriptor {
|
||
id: BackendId;
|
||
label: string; // dock label — NEVER "C15"
|
||
description: string;
|
||
/** crossOriginIsolated / WebMIDI / WebSocket etc. availability probe. */
|
||
isAvailable(): boolean;
|
||
/** true when this backend can ALSO feed inputs back (VCV bridge, OSC return). */
|
||
bidirectional?: boolean;
|
||
}
|
||
|
||
export interface OutputBackend {
|
||
readonly descriptor: BackendDescriptor;
|
||
|
||
/** Called once when this backend becomes active. May be async (audio start,
|
||
* WS connect, MIDI access). Resolve only when ready to receive send(). */
|
||
start(ctx: BackendContext): Promise<void>;
|
||
|
||
/** Hot per-frame path. `routed` is the post-pipeline Float32Array (0..1),
|
||
* length = ctx.outputCount. MUST NOT allocate; MUST NOT mutate `routed`.
|
||
* Throttling/dead-zone live INSIDE each backend (rates differ per sink). */
|
||
send(routed: Float32Array): void;
|
||
|
||
/** Called when switching away or unmounting. Release WS/MIDI/audio/threads. */
|
||
teardown(): Promise<void>;
|
||
|
||
/** Optional input return path for bidirectional backends. The engine
|
||
* subscribes; values drive model inputs (e.g. VCV CV-in, OSC /nisps/input). */
|
||
onInputs?(cb: (values: Float32Array) => void): () => void;
|
||
|
||
/** Optional: backends that own training transport (VCV/OSC bridge) expose
|
||
* the remote verdict/example surface here. See §7. */
|
||
remote?: RemoteTrainingBridge;
|
||
}
|
||
```
|
||
|
||
Registry (`manifold/src/engine/backends/registry.ts`): a `Map<BackendId, () => OutputBackend>` of lazy factories. The dock reads `descriptors` (filtered by `isAvailable()`); selecting one calls `engine.setBackend(id)`, which `teardown()`s the old and `start()`s the new with the current `BackendContext`.
|
||
|
||
**Why `send(Float32Array)` and not per-param events:** matches the spine's single transferable-buffer effect (`engine-architecture.md` §2.1), keeps the hot path allocation-free, and lets each backend decide its own decimation. The legacy code already proves the pattern — every sink takes the full output vector and self-throttles (synth 50 ms/0.002 dead-zone `a-app.js:2425`; MIDI 50 ms/Δ1 `midi-output.js:114`; OSC 50 ms/0.002 `osc-output.js:97`).
|
||
|
||
---
|
||
|
||
## 2. The backends
|
||
|
||
### 2.1 Built-in Synth — the "Powerful Synth Engine" (`web-audio.ts` + `synth.ts`)
|
||
|
||
Two cooperating backends, both labelled as the synth in the UI, but architecturally distinct:
|
||
|
||
- **`WebAudioBackend`** wraps the parity-tested repo engine: `EngineHost` (`playground/src/audio/engine-host.ts`) + the worklet `nisps-processor.ts` running `_nisps_engine_process_block` (`nisps/wasm/bindings.cpp`). This is the firmware-parity audio path — the engine *is* the sound. `send()` → `EngineHost.setParams(routed)` → worklet. This is the default and the one that satisfies browser-parity chokepoint C.
|
||
- **`PowerfulSynthBackend`** (the C15 path) wraps `deployments/meml-aimmersive/js/synth/c15-adapter.js` → `c15-bridge.js` (SharedArrayBuffer ring + its own `c15_engine.wasm` worklet). Param mapping comes from `param-map.js` (`SYNTH_PARAM_MAP`, 126 entries) and `presets.js` (tiered presets). `setParam(index, normalised)` maps index→hardware id (`c15-adapter.js:121`). This is browser-only (firmware has no C15), and its 126-param surface + group/section overrides power the synth visualiser and the group-override drawer.
|
||
|
||
**Reuse, verbatim:** `c15-adapter.js`, `c15-bridge.js`, `param-map.js`, `presets.js` move under `manifold/src/engine/backends/synth/` unchanged (internal names keep "c15"; UI strings do not). The `engine-interface.js` `SynthEngine` base maps cleanly onto `OutputBackend`: `init(ctx)`→`start`, `setParam` loop driven by `send`, `stop`→`teardown`. Curve/override math is `applyCurve`/`applyGroupOverride` (`param-map.js:287`) — fold into the universal mapping (§3).
|
||
|
||
**Throttle (keep — load-bearing):** ≥50 ms send interval + 0.002 dead-zone per param prevents flooding the C15 ring buffer at 126×30 fps (`a-immersive.html` clone-spec §10 flags this).
|
||
|
||
`isAvailable()`: WebAudio + (for the C15 path) `crossOriginIsolated === true` (SAB needs COOP/COEP; already server-scoped per `findings-engine-surface.md`).
|
||
|
||
### 2.2 Particle System — faithful port of `visualizer.js` (`particles.ts`)
|
||
|
||
A `ParticleBackend` whose `send(routed)` calls a ported `FlowFieldVisualizer.setParams(routed)`. The visual canvas runs in its own `requestAnimationFrame` loop (per `findings-design-and-manifold.md` §4.3 — rAF touches drawing only, never inference); `send()` only updates the param struct. The port MUST look and behave **exactly** as the deployed version. The full algorithm is documented in §4 so the React port is byte-faithful.
|
||
|
||
`isAvailable()`: always (Canvas2D). This backend produces no audio — it is the "visual" output mode.
|
||
|
||
### 2.3 MIDI out — advanced CC config (`web-midi.ts`)
|
||
|
||
A `WebMidiBackend` wrapping the salvaged `midi-output.js` (Web MIDI API, `sendBatch`, per-CC dead-zone + 50 ms throttle, device hot-plug handling — `midi-output.js`). The **advanced CC config comes from workstream D's config model**, persisted as a CC map: per output dim → `{ name, cc (0–127), channel (1–16), min, max, curve, muted, fixedValue }`. The map shape and storage are already defined in `midi-cc-map.js` (`createCCParam`, `loadCCMap`/`saveCCMap`, well-known `CC_NAMES`, default 8-CC starter set). Lift that file verbatim into `manifold/src/engine/backends/midi/cc-map.ts`.
|
||
|
||
`send(routed)`: for each non-muted CC param, `value = round(applyGroupOverride(routed[i], curve, min, max) * 127)`; batch the changed ones; `midiOutput.sendBatch(...)`. Storage key stays engine-scoped (`nisps-midi-cc-map:<modeId>`) for migration continuity.
|
||
|
||
`isAvailable()`: `!!navigator.requestMIDIAccess`.
|
||
|
||
### 2.4 OSC out — paths + ranges (`osc-bridge.ts`)
|
||
|
||
An `OscBridgeBackend` that **salvages the existing OSC bridge** rather than reinventing it. Two pieces already exist and are good:
|
||
|
||
- **Browser client:** `deployments/meml-aimmersive/js/synth/osc-output.js` (param-named WS messages, 50 ms/0.002 dead-zone) and the richer `js/nisps/osc-client.js` (`NispsOscClient` — `EventTarget`, `sendState`/`sendWeights`/`sendParams`, `onOutputsReceived`/`onInputsReceived`, auto-reconnect with backoff). Lift `osc-client.js` as the transport (it already speaks the bridge protocol and is bidirectional).
|
||
- **Bridge server:** `deployments/meml-aimmersive/osc-bridge/bridge.ts` — a Deno WebSocket↔UDP-OSC bridge, zero-dependency OSC encode/decode, bidirectional. Keep it as-is; it is the canonical transport between browser and any OSC target (VCV, SuperCollider).
|
||
|
||
**OSC path + range contract (salvaged from `bridge.ts`):**
|
||
|
||
| Direction | Address | Args | Meaning |
|
||
|---|---|---|---|
|
||
| browser→target | `/nisps/<param_name>` | `f` | one param, **post-baseline-mapping value** (see §3) |
|
||
| browser→target | `/nisps/state` | `s` | full JSON state (weights + examples + config) |
|
||
| browser→target | `/nisps/weights` | `s` | weights-only JSON |
|
||
| target→browser | `/nisps/output` | `f…f` | output vector (visualisation / monitoring) |
|
||
| target→browser | `/nisps/input` | `f…f` | input vector → drives model inputs |
|
||
|
||
**Ranges:** OSC floats are sent in the param's mapped range by default (`applyGroupOverride` applied before send, matching `osc-output.js`), with a per-backend toggle to send **raw normalised 0..1** instead (some OSC targets want 0..1 and do their own scaling). Address prefix (`/nisps`), target host/port (default `127.0.0.1:9000`), and listen port (default `9001`) are configurable — `bridge.ts` already exposes `--osc-host/--osc-port/--osc-prefix/--ws-port/--listen-port`.
|
||
|
||
`isAvailable()`: always (attempts WS to `ws://localhost:8765`; surfaces a "bridge not running" status if the connect fails — `osc-client.js` already reconnects with backoff). Bidirectional (`onInputs` wired to `/nisps/input`).
|
||
|
||
### 2.5 CV / gate backend (`cvgate.ts`)
|
||
|
||
For browser-side CV/gate there is no native hardware path, so this backend has **two transports** selectable in config:
|
||
|
||
1. **DC-coupled WebAudio CV** (browser-native): each output dim drives a `ConstantSourceNode` (or a sample-accurate `AudioWorklet` channel) whose `offset` = mapped voltage, summed/routed to the audio interface's output channels. Gate outputs are derived from a configurable threshold on a chosen dim (value > τ → high). Pitch (1V/oct) uses a per-output "voltage role" config: `{ role: 'cv' | 'gate' | 'voct', vmin, vmax, gateThreshold }`. This is the only way to emit real CV from a browser (DC-coupled interface required; surfaced as a caveat in the UI).
|
||
2. **Bridged CV via VCV / OSC** (recommended default): reuse the OSC/VCV transport — the *VCV module's 16 CV outputs* (§5) are the real CV/gate jacks. In this mode `CvGateBackend` is a thin alias that delegates to `VcvBridgeBackend` with a "treat outputs as CV/gate" preset (per-output unipolar 0–10 V / bipolar ±5 V / 1V-oct, matching the VCV per-output range menu in `MEMLNaut.cpp:818`).
|
||
|
||
`isAvailable()`: WebAudio path always; native-CV quality flagged as "requires DC-coupled interface". **Open choice:** whether browser-native DC CV is worth shipping vs. making CV strictly a VCV-bridge concern (recommendation: ship the VCV-bridge alias first, defer DC-coupled WebAudio CV).
|
||
|
||
### 2.6 VCV Rack module — first-class (`vcv-bridge.ts` browser side + `vcv/` C++ side)
|
||
|
||
The headline new backend. A first-class **VCV Rack 2 module** (`MEMLNaut`) with **8 CV inputs → model → 16 CV outputs**, an **LED ring around each of the 16 outputs**, and a **browser↔VCV bridge** so the tool is controllable AND trainable from both inside Rack and entirely from the browser. Full design in §5–§7.
|
||
|
||
The browser-side adapter `VcvBridgeBackend` reuses `NispsOscClient` (§2.4) as transport. When active in **bridged mode**, the browser supplies inputs in real time (`/nisps/input`) and the verdict/example loop is mirrored over the bridge (`/nisps/state`, `/nisps/weights`, new `/nisps/feedback`).
|
||
|
||
---
|
||
|
||
## 3. Universal per-output baseline mapping
|
||
|
||
Every backend shares ONE baseline mapping from a normalised model output `v ∈ [0,1]` to a sink value, so behaviour is identical across sinks and the override UI (heatmap popup, group drawer) is backend-agnostic. This is the existing curve math (`param-map.js:287`), promoted to `manifold/src/engine/backends/mapping.ts`:
|
||
|
||
```ts
|
||
// 0.5 = linear; <0.5 ease-in, >0.5 ease-out. Bit-identical to legacy + nisps/core/math.hpp.
|
||
export function applyCurve(v: number, c: number): number {
|
||
return c === 0.5 ? v : Math.pow(v, Math.pow(2, 4 * (c - 0.5)));
|
||
}
|
||
|
||
/** Per-output baseline: curve, then scale into [min,max]; honour freeze/mute. */
|
||
export function mapOutput(v: number, p: OutputMapping): number {
|
||
if (p.frozen) return p.fixedValue; // pinned, ignores model
|
||
return p.min + applyCurve(v, p.curve) * (p.max - p.min);
|
||
}
|
||
|
||
export interface OutputMapping {
|
||
min: number; max: number; curve: number; // baseline range + curve
|
||
frozen: boolean; fixedValue: number; // pin
|
||
muted?: boolean; // excluded from this sink
|
||
}
|
||
```
|
||
|
||
Backends then apply only their **sink-specific final transform** on top of the baseline:
|
||
|
||
| Backend | Baseline → sink transform |
|
||
|---|---|
|
||
| Powerful Synth | `mapOutput` → `setParam(i, value)` (value already in param range) |
|
||
| Particles | **raw 0..1** (the visualiser owns its own param ranges — see §4); curve/freeze still applied, min/max default to 0..1 |
|
||
| MIDI CC | `mapOutput` → `round(value * 127)` clamped 0–127 |
|
||
| OSC | `mapOutput` (or raw 0..1 if "send raw" toggled) → `/nisps/<name> <f>` |
|
||
| CV/gate | `mapOutput` → voltage by role (`cv`: `value*10` or `(value-0.5)*10`; `voct`: 1V/oct; `gate`: `value>τ ? high : 0`) |
|
||
| VCV | model output `0..1` sent raw over bridge; the **module** applies its own per-output range + attenuverter (`MEMLNaut.cpp:342 outputToVoltage`) |
|
||
|
||
The override store (one per mode, `OutputMapping[]`) is owned by the engine and shared by all backends; freeze/mute/curve/range edits in the UI apply uniformly. Note the legacy split where the heatmap calls it `frozen` and the group drawer calls it `muted` but both map to the same field (`aimmersive-clone-spec.md` §10) — unify to the single `OutputMapping` above.
|
||
|
||
---
|
||
|
||
## 4. Particle system — faithful-port plan with the documented algorithm
|
||
|
||
Source of truth: `deployments/meml-aimmersive/js/ui/visualizer.js` (289 lines, read in full). The React port (`manifold/src/engine/backends/particles/flow-field.ts`) must reproduce this **exactly**. Below is the complete algorithm with line citations so the port is verifiable.
|
||
|
||
### 4.1 Constants & noise source
|
||
- **Particle count: `numParticles = 400`** (`visualizer.js:53`).
|
||
- **Noise: a self-contained 2-D value/Perlin-style noise**, not a library. A `Uint8Array(512)` permutation table `PERM` is built once at module load by Fisher–Yates shuffling `[0..255]` then duplicating (`:5–14`). `fade(t)=t³(t(6t−15)+10)` (`:16`), `lerp` (`:17`), `grad(hash,x,y)` using `hash & 3` (`:19–24`), `noise2D(x,y)` doing the standard 4-corner bilinear-with-fade interpolation (`:26–44`). **The shuffle uses `Math.random()` at module load**, so the field is non-deterministic per page load — the port must keep this (or seed it; flagged as an open choice if determinism is wanted for tests).
|
||
- `TWO_PI = Math.PI*2` (`:46`); `this.time` advances `+= 0.003` per `draw()` (`:185`).
|
||
|
||
### 4.2 Output→param mapping (20 dims, `setParams`, `:159–181`) — reproduce verbatim
|
||
Guard: returns early if `outputs.length < 20` (`:160`). Then:
|
||
```
|
||
angleOffset = out0 * TWO_PI
|
||
scale = 0.001 + out1 * 0.009
|
||
speed = 0.5 + out2 * 4.5
|
||
hueBase = out3 * 360
|
||
hueSpread = out4 * 120
|
||
particleSize = 1 + out5 * 5
|
||
fadeRate = 0.01 + out6 * 0.14
|
||
turbulence = out7 * 2
|
||
attractStrength = 0.1 + out8 * 2.9
|
||
attractRadius = 40 + out9 * 420
|
||
dispersionRate = 0.2 + out10 * 8
|
||
dispersionAmount = out11 * 3
|
||
particleLifetime = 30 + out12 * 470
|
||
respawnStyle = out13 // 0=random,1=edge,2=center-burst
|
||
advectionMode = out14 // flow→orbit→radial blend
|
||
inertia = out15 * 0.98
|
||
drag = out16 * 0.35
|
||
repulsorStrength = out17 * 4.5
|
||
repulsorCount = floor(out18 * 4.999) // 0..4
|
||
repulsorOrbitRate = 0.1 + out19 * 2.9
|
||
```
|
||
The 20 output names (for labels/the heatmap colouring) are `VISUAL_PARAM_NAMES` in `a-app.js:46`: `Flow, Scale, Speed, Hue, Spread, Size, Trail, Turb, Attract, Radius, DispRate, DispAmt, Lifetime, Respawn, Advection, Inertia, Drag, Repulse, RepCnt, RepRate`.
|
||
|
||
### 4.3 Particle lifecycle (`:104–156`)
|
||
- `makeParticle(id)`: random x,y in canvas; `age = floor(rand*particleLifetime)`; `life = computeLifetime()`; `vx=vy=0` (`:104–114`).
|
||
- `computeLifetime()`: `max(10, floor(particleLifetime * (0.65 + rand*0.7)))` (`:116–119`).
|
||
- `respawnParticle(p)`: `mode = min(2, floor(respawnStyle*2.999))` (`:122`):
|
||
- **edge (1):** spawn on a random one of 4 sides; give inward impulse of magnitude 2.0 toward centre (`:125–137`).
|
||
- **center-burst (2):** spawn within radius `min(w,h)*0.08` of centre at random angle; velocity `2.5` outward along that angle (`:138–145`).
|
||
- **random (0):** random position; velocity `(rand*2−1)*0.5` each axis (`:146–152`).
|
||
- reset `age=0`, `life=computeLifetime()`.
|
||
|
||
### 4.4 Per-frame `draw()` (`:183–287`) — the exact integration
|
||
1. `time += 0.003`. **Trail fade:** fill whole canvas with `rgba(13,13,13, fadeRate)` (`:188`) — this is the trail length control, not a clear.
|
||
2. For each particle, with centre `(cx,cy)=(w/2,h/2)`:
|
||
- **Flow field:** `nx=p.x*scale`, `ny=p.y*scale`; `angle = noise2D(nx+time, ny)*TWO_PI + angleOffset`; `curl = noise2D(nx+100, ny+100+time*0.5)*turbulence` (`:196–199`).
|
||
- **Three advection fields** (`:202–214`): flow `(cos(angle+curl), sin(angle+curl))*speed`; orbit = perpendicular to the radial-from-centre unit vector, `*speed`; radial = radial unit vector `*speed`.
|
||
- **Blend by `advectionMode`** (`:216–225`): `modeBlend = advectionMode*2`; if `<1` lerp(flow→orbit, modeBlend) else lerp(orbit→radial, modeBlend−1).
|
||
- **Inertia + drag** (`:226–229`): `vx = vx*inertia + targetVx*(1−inertia)` (same for vy); then `vx *= (1−drag)`.
|
||
- `nextX = x+vx`, `nextY = y+vy`.
|
||
- **Central attractor** (`:233–242`): pull toward centre with `falloff = 1/(1+normDist²)` where `normDist = min(dist/attractRadius, 2)`; add `nCenter * attractStrength * falloff`.
|
||
- **Dispersion pulse** (`:244–248`): `pulse = 0.5+0.5*sin(time*dispersionRate + id*0.07)`; subtract `nCenter * dispersionAmount*pulse*falloff` (pushes outward near centre).
|
||
- **Orbiting repulsors** (`:250–264`): `repulsorRadius = min(w,h)*0.28`; for each of `repulsorCount`: phase from `time*repulsorOrbitRate + (r/4)*TWO_PI`, `wobble = 0.6+0.15*r`, position uses `cos(phase*(1+wobble))`/`sin(phase*(1.3+wobble))`; inverse-square push `force = repulsorStrength*(650/distSq)` with `distSq = d²+160`.
|
||
- **Wrap edges** (`:270–273`); `age += 1`; respawn when `age >= life` (`:275–276`).
|
||
- **Colour** (`:279–282`): `hue = (hueBase + (id/numParticles)*hueSpread) % 360`; `lightness = 50 + sin(id*0.1 + time)*15`; fill `hsl(hue, 75%, lightness%)`; `arc(x,y,particleSize)`.
|
||
|
||
### 4.5 Port plan
|
||
- **Class, not component.** `FlowFieldVisualizer` is a plain TS class taking a `<canvas>` ref — identical to today. The React `<ParticleCanvas>` mounts it in `onMount`, drives `draw()` from one `requestAnimationFrame` loop, and calls `resize()` on the window resize handler (DPR scaling at `:84–92`). React renders the canvas element; the class owns all pixels.
|
||
- **`ParticleBackend.send(routed)`** → `viz.setParams(routed)` (no alloc; just field writes). Because the visualiser reads `outputs[0..19]`, the backend asserts `ctx.outputCount >= 20` and slices/pads to 20.
|
||
- **Faithfulness gate (Playwright):** pixel-diff a fixed seed (seed the `PERM` shuffle behind a `?seed=` for tests) at fixed param vectors against a golden capture from the deployed app; assert SSIM ≥ threshold. Also unit-test `setParams` mapping numerically (the §4.2 table).
|
||
- **Verbatim copy is allowed**: the noise + integration math is pure and has no DOM coupling beyond `ctx`/`canvas`; lift `:1–288` essentially unchanged into TS, add types, keep numeric constants exact.
|
||
|
||
---
|
||
|
||
## 5. VCV Rack module design (`MEMLNaut`, 8 in / 16 out)
|
||
|
||
There is **already a working VCV module** at `vcv/` (`MEMLNaut.cpp`, 959 lines; `plugin.json`; `SPEC.md`; `osc_server.hpp`; `Makefile`; `res/*.svg`) — but it is **2-in / 12-out + 5 derived**. The new requirement is **8-in / 16-out with LED rings**. This is an evolution of the existing module, reusing its threading model, OSC server, and serialization wholesale.
|
||
|
||
### 5.1 What changes vs. the existing module
|
||
- `NUM_ML_INPUTS`: 2 → **8** (the existing `MAX_ML_INPUTS = 8` already anticipated this; `MEMLNaut.cpp:14`). All 8 are first-class jacks (not "reserved").
|
||
- `NUM_ML_OUTPUTS`: 12 → **16**. The 5 derived outputs (MEAN/STD/DELTA/NOVELTY/CONFIDENCE) remain but become **optional context-menu extras** or move to the expander — the 16 raw outputs are the headline.
|
||
- MLP shape: stays within "the modular N×M envelope" — `nisps::IML<float> iml{8, 16, {16, 24, 16}}` (the existing default hidden stack is fine; sized for real-time inference per `SPEC.md`). Inputs feed model input dims; the 16 outputs are the inference outputs.
|
||
- **LED ring per output** replaces the single `SmallLight<WhiteLight>` next to each jack (`MEMLNaut.cpp:804`).
|
||
|
||
### 5.2 Panel layout (Wide, ~32–44 HP)
|
||
```
|
||
┌────────────────────────────────────────────┐
|
||
│ MEMLNaut │ ← brand; "Powerful Synth" wording N/A (this is the CV mapper)
|
||
│ ┌──────────────────────────────────────┐ │
|
||
│ │ DISPLAY: 16 bars + XY dot + metrics │ │ ← NanoVG LedDisplay/drawLayer (existing :699)
|
||
│ └──────────────────────────────────────┘ │
|
||
│ SPREAD RATE LEARN● RAND CLEAR │ ← knobs + buttons (existing controls)
|
||
│ [+] [−] (verdict buttons) │
|
||
│ │
|
||
│ INPUTS (8 jacks, 2 rows × 4) │
|
||
│ IN1 IN2 IN3 IN4 │
|
||
│ IN5 IN6 IN7 IN8 + LEARN_GATE +TRIG −TRIG│
|
||
│ │
|
||
│ OUTPUTS (16, 4 rows × 4), each: │
|
||
│ ◉jack with an LED RING around the jack │
|
||
│ [(◯1)(◯2)(◯3)(◯4)] │
|
||
│ [(◯5)(◯6)(◯7)(◯8)] │
|
||
│ [(◯9)(◯10)(◯11)(◯12)] │
|
||
│ [(◯13)(◯14)(◯15)(◯16)] │
|
||
│ (optional) MEAN STD DELTA NOVELTY CONF │
|
||
└────────────────────────────────────────────┘
|
||
```
|
||
Each output is a `PJ301MPort` jack with a **`LedRingWidget`** drawn concentric around it (no separate attenuverter trimpot in the default skin — attenuverter moves to right-click/expander to make room for the ring; keep `PARAM_ATTEN_*` in the model for range scaling).
|
||
|
||
### 5.3 The LED-ring widget + palette mapping
|
||
A custom widget that draws a ring whose **arc fill is proportional to the output value** and whose **colour comes from the frontend design tokens**. Per the Rack manual, self-illuminating custom widgets override `drawLayer(args, 1)` and draw on layer 1 (so they stay bright when room brightness is lowered) ([VCV custom lights](https://community.vcvrack.com/t/how-to-use-custom-lights/1941), [Migrate2](https://vcvrack.com/manual/Migrate2)).
|
||
|
||
```cpp
|
||
// vcv/src/LedRing.hpp (new)
|
||
struct LedRingWidget : Widget {
|
||
MEMLNaut* module = nullptr;
|
||
int outIdx = 0;
|
||
NVGcolor ringColor = nvgRGB(0xff, 0x6a, 0x00); // default --accent
|
||
float radius = 7.f; // mm-ish, around a PJ301M jack
|
||
void drawLayer(const DrawArgs& args, int layer) override {
|
||
if (layer != 1 || !module) return;
|
||
float v = clamp(module->slewOutputs[outIdx], 0.f, 1.f); // 0..1
|
||
Vec c = box.size.div(2);
|
||
// track (dim full ring)
|
||
nvgBeginPath(args.vg);
|
||
nvgArc(args.vg, c.x, c.y, radius, -M_PI/2, -M_PI/2 + 2*M_PI, NVG_CW);
|
||
nvgStrokeColor(args.vg, nvgRGBA(ringColor.r*255, ringColor.g*255, ringColor.b*255, 40));
|
||
nvgStrokeWidth(args.vg, 1.4f); nvgStroke(args.vg);
|
||
// value arc (proportional)
|
||
nvgBeginPath(args.vg);
|
||
nvgArc(args.vg, c.x, c.y, radius, -M_PI/2, -M_PI/2 + v*2*M_PI, NVG_CW);
|
||
nvgStrokeColor(args.vg, ringColor);
|
||
nvgStrokeWidth(args.vg, 1.8f); nvgStroke(args.vg);
|
||
// glow halo (matches the frontend "glow not shadow" signature)
|
||
nvgGlobalCompositeOperation(args.vg, NVG_LIGHTER); /* … bloom pass … */
|
||
}
|
||
};
|
||
```
|
||
|
||
**Palette mapping — derived from `docs/redesign/manifold-export/tokens/colors.css` (read).** Ring colours come from the design tokens so VCV matches the frontend. A `kRingPalette[16]` table assigns each output a colour by its **parameter group**, cycling through the token accents and group/pin colours:
|
||
|
||
| Source token (colors.css) | Hex | Used for |
|
||
|---|---|---|
|
||
| `--accent` | `#ff6a00` | primary outputs / group 0 (orange — the live colour) |
|
||
| `--accent-2` | `#00ccff` | data outputs / group 1 (cyan) |
|
||
| `--accent-3` | `#ffa860` | group 2 (warm tint) |
|
||
| `--pin-3` base `#b464ff` | `#b464ff` | group 3 (violet) |
|
||
| `--good` | `#6bc26b` | group 4 (green) |
|
||
| `--warn` | `#f5c45e` | group 5 (amber) |
|
||
| `--info` | `#5b9eef` | group 6 (blue) |
|
||
| `--danger` | `#ff4466` | bipolar / perturbed outputs (red) |
|
||
|
||
Mapping rule: `ringColor = kRingPalette[paramMeta[i].group % paletteLen]`, so outputs in the same mode-group glow the same colour, identical to the heatmap/Console grouping. Bipolar outputs (per-output range menu) tint toward `--danger`. The palette is a single header (`vcv/src/palette.hpp`) generated from `colors.css` so a token change propagates to both frontend and module (a small codegen step; **open choice** whether to automate or hand-sync).
|
||
|
||
### 5.4 Reused, unchanged from the existing module
|
||
- **Threading model** (`MEMLNaut.cpp:67–312`): audio-thread `iml` + worker-thread `imlShadow`, job queue (Train/Perturb/Randomize/Clear), atomic weight hand-off, single-writer invariant. Already correct; just resize the I/O.
|
||
- **Inference-rate decimation + slew + post-swap crossfade** (`:447–499`).
|
||
- **Verdict loop** (`:412–445`): `+`/`−` buttons and `+TRIG`/`−TRIG` gated by LEARN. `+` → add example (current inputs→current outputs) + enqueue Train + decay noise ×0.97; `−` → bump noise (cap `0.3(1−s)+0.05s`) + enqueue Perturb. Identical semantics to the browser verdict loop.
|
||
- **Serialization** (`dataToJson`/`dataFromJson`, `:548–696`) + **`.nisps` preset save/load** (`:862–924`) + **OSC server** (`osc_server.hpp`). The `.nisps` format is the shared interchange with the browser (`vcv/NISPS-FORMAT.md`).
|
||
- Per-output / per-input **range menu** (uni/bipolar, attenuverter) (`:818–843`).
|
||
|
||
### 5.5 Plugin scaffold files
|
||
```
|
||
vcv/
|
||
├── plugin.json # slug MEMLNaut (exists) — bump version; tags Controller/Utility/Random
|
||
├── Makefile # VCV SDK Makefile (exists); RACK_DIR-driven
|
||
├── Makefile.dist # dist packaging (exists)
|
||
├── src/
|
||
│ ├── plugin.{hpp,cpp} # plugin init / model registration (exist)
|
||
│ ├── MEMLNaut.cpp # module — resize 8 in / 16 out (edit)
|
||
│ ├── LedRing.hpp # NEW — LED-ring widget (§5.3)
|
||
│ ├── palette.hpp # NEW — ring colour table from colors.css (§5.3)
|
||
│ └── osc_server.hpp # OSC server (exists; extend with /nisps/feedback §7)
|
||
├── res/
|
||
│ ├── MEMLNaut.svg / -wide.svg / -expander.svg # panels (exist; redraw for 8/16 layout)
|
||
├── test/smoke_test.cpp # exists
|
||
├── plugin.json, SPEC.md, BUILDING.md, README.md, NISPS-FORMAT.md (exist)
|
||
```
|
||
|
||
---
|
||
|
||
## 6. Browser↔VCV bridge design
|
||
|
||
The module must be **controllable AND trainable from BOTH inside Rack AND entirely from the browser**. Both ends operate on the same model; the bridge keeps them coherent.
|
||
|
||
### 6.1 Transport — propose options (operator open choice)
|
||
The existing salvage path is **WebSocket↔UDP-OSC** (`osc-bridge/bridge.ts` + `osc-client.js` + `osc_server.hpp`). This is the recommended default and already works. Three options to flag:
|
||
|
||
| Option | How | Pros | Cons |
|
||
|---|---|---|---|
|
||
| **A. WS↔OSC bridge server (recommended)** | Browser ⇄ `bridge.ts` (Deno WS server, localhost:8765) ⇄ UDP OSC ⇄ module's `osc_server.hpp` (port 9000/9001) | Already built + bidirectional; standard OSC; works with SuperCollider/Max too; no browser perms | Needs a helper process running locally (Deno or the compiled `bridge.mjs`) |
|
||
| **B. Direct WebMIDI** | Browser ⇄ Web MIDI ⇄ a virtual MIDI port ⇄ a tiny MIDI-in path in the module | No helper process if a virtual MIDI port exists; browser-native | 7-bit/14-bit only — too coarse for weights/state; really only for live CC; module would need MIDI parsing |
|
||
| **C. Native (module hosts a WS server)** | The VCV module itself opens a WebSocket/HTTP server; browser connects directly | No external bridge process | Adds a WS/TLS stack inside the plugin; COOP/COEP + mixed-content (`https://` page → `ws://localhost`) friction; more attack surface in the audio plugin |
|
||
|
||
**Recommendation:** ship **A** (it exists, it's bidirectional, it already targets this very module — see `osc-client.js:1` "Connects the webapp to VCV Rack MEMLNaut module"). Keep **B** as a live-performance CC convenience only. Treat **C** as a future "no-helper" nicety. **This transport choice is an explicit operator open choice.**
|
||
|
||
### 6.2 Two modes
|
||
- **Standalone:** module runs entirely inside Rack (CV in → model → CV out; verdict via panel buttons/triggers). No browser. Works today.
|
||
- **Bridged:** browser connects via the bridge. In bridged mode **the browser supplies inputs in real time** — the Manifold pointer / joystick / pads stream `/nisps/input <f…f>` to the module, which uses them instead of (or blended with, configurable) the physical CV-in jacks. The module streams `/nisps/output` and `/nisps/input` back at ~100 ms (`MEMLNaut.cpp:536`) for browser visualisation. State/weights sync both ways via `/nisps/state` + `/nisps/weights` (already staged atomically into the audio thread, `:360–384`).
|
||
|
||
### 6.3 Coherence model
|
||
One model, one owner of weights at a time. The bridge sends **whole-model snapshots** (`/nisps/state`, `/nisps/weights`) on any structural change (train completes, randomize, clear, load preset), and **continuous I/O vectors** (`/nisps/input`, `/nisps/output`) for live feel. Last-writer-wins on weights with a short "training in progress" lock (the module already coalesces jobs at queue depth 1). The browser's `WasmIML` and the module's `nisps::IML` use the **same `.nisps` weight layout** (`vcv/NISPS-FORMAT.md`), so a snapshot from either side loads losslessly — *provided the architectures match*. **Caveat / open choice:** the browser engine is fixed `MLP<2,…,126>` while the module is `8→{16,24,16}→16`. For true weight transfer the bridged session must run a **matched architecture** (e.g. a browser mode configured to 8-in/16-out, within the modular envelope) — otherwise the bridge degrades to I/O + example transfer only (no raw-weight sync). Flag this explicitly.
|
||
|
||
---
|
||
|
||
## 7. Training over the bridge — both directions
|
||
|
||
The verdict loop (place examples / thumbs-up / thumbs-down / undo) must work from **either** end and stay coherent. Add one OSC address and a small `RemoteTrainingBridge`:
|
||
|
||
```ts
|
||
// manifold/src/engine/backends/backend.ts
|
||
export interface RemoteTrainingBridge {
|
||
thumbsUp(): void; // add current (input,output) example + train
|
||
thumbsDown(): void; // perturb weights (explore)
|
||
addExample(input: Float32Array, output: Float32Array): void;
|
||
randomise(spread: number): void;
|
||
clear(): void;
|
||
undo(): void;
|
||
onState(cb: (s: NispsState) => void): () => void; // remote → local sync
|
||
}
|
||
```
|
||
|
||
New OSC verb: `/nisps/feedback <s>` carrying `{"op":"up|down|rand|clear|undo","spread":f,"input":[…],"output":[…]}`. The module's `osc_server.hpp` gets an `onFeedback` callback (mirroring the existing `onState`/`onWeights` at `MEMLNaut.cpp:120–132`) that stages the op atomically for the audio thread, which routes it through the **same** `enqueueJob`/`add_example` path the panel buttons use (`:412–445`). So:
|
||
|
||
- **Browser → VCV training:** user clicks thumbs-up in the Manifold → `VcvBridgeBackend.remote.thumbsUp()` → `/nisps/feedback {op:up,input,output}` → module stages → worker trains `imlShadow` → atomic weight swap → module streams `/nisps/state` back → browser `WasmIML.setWeights` updates so the UI/heatmap reflect the new mapping. The browser need not run its own training in bridged mode (or runs it and pushes weights; configurable — see §6.3 caveat).
|
||
- **VCV → browser training:** user presses `+`/`−` on the panel (or sends `+TRIG`) → module trains/perturbs → streams `/nisps/state` → browser applies, so the verdict placed in Rack appears in the browser's example list and weight-health view.
|
||
- **Example placing over the bridge:** either side can `addExample`; examples ride in `/nisps/state` (the module already serializes `examples.features`/`examples.labels`, `:587–604`) so the dataset stays in sync. Undo is local-history on each side, but a remote undo can be sent as `/nisps/feedback {op:undo}` to roll the module's last job (module keeps a one-deep snapshot, matching the browser undo stack semantics).
|
||
|
||
**Result:** the tool is fully controllable and trainable from inside VCV Rack and entirely from the browser, with the verdict loop and example-placing working either way over the same bridge.
|
||
|
||
---
|
||
|
||
## 8. Build / install notes
|
||
|
||
### Browser side
|
||
Backends live under `manifold/src/engine/backends/`; lifted JS (`c15-*`, `param-map`, `presets`, `midi-output`, `midi-cc-map`, `osc-client`, `visualizer`) ported to TS, parity-checked. No new build step — they ride the existing Vite `manifold` build. COOP/COEP stays server-scoped (needed by the Powerful Synth's SAB path).
|
||
|
||
### OSC bridge server
|
||
```bash
|
||
cd osc-bridge
|
||
deno run --allow-net bridge.ts # or the compiled bridge.mjs
|
||
# --osc-host 127.0.0.1 --osc-port 9000 --ws-port 8765 --listen-port 9001
|
||
```
|
||
Ship `bridge.mjs` (already compiled) + `compile.sh` so users without Deno can run it via Node. Surface "bridge not running" in the OSC/VCV backend status (the client auto-reconnects).
|
||
|
||
### VCV module
|
||
```bash
|
||
cd vcv
|
||
export RACK_DIR=/path/to/Rack-SDK # VCV Rack 2 SDK
|
||
make # builds plugin.so/.dylib/.dll
|
||
make install # copies into the VCV user plugins dir
|
||
# distribution: make dist (per Makefile.dist; produces the .vcvplugin)
|
||
```
|
||
Per the search, the standard flow is `export RACK_DIR=…; make clean; make dist` ([Plugin Development Tutorial](https://vcvrack.com/manual/PluginDevelopmentTutorial)). Requires the VCV Rack 2 SDK; nisps-core is header-only C++20 (symlinked under `vcv/dep/` per `SPEC.md`). Ship v2-only (rationale in `SPEC.md`'s v1-compat section). License caveat: VCV SDK is GPLv3, nisps-core is MPL-2.0 — combined binary is effectively GPL; not submitting to the VCV Library initially (`SPEC.md` §License).
|
||
|
||
---
|
||
|
||
## 9. Open choices for the operator
|
||
|
||
1. **Bridge transport (§6.1):** confirm **A — WS↔OSC bridge server** as default (recommended; already built and bidirectional), with WebMIDI as a live-CC-only convenience and a native in-module WS server deferred. This is the biggest call.
|
||
2. **Bridged weight-sync vs I/O-only (§6.3):** the browser engine is fixed `MLP<2,…,126>`; the module is `8→16`. Either (a) run a **matched 8-in/16-out browser mode** for true raw-weight transfer, or (b) accept that bridged sessions sync **I/O + examples only** and each side trains its own weights. Recommendation: (b) for v1, (a) when the modular N×M browser MLP lands (workstream F).
|
||
3. **CV/gate native path (§2.5):** ship only the **VCV-bridge CV alias** for v1, or also build browser-native DC-coupled WebAudio CV? Recommendation: VCV-bridge first; defer DC CV.
|
||
4. **Particle noise determinism (§4.1):** keep the `Math.random()`-seeded permutation (non-deterministic per load, faithful to today) or add a `?seed=` for reproducible visuals/tests? Recommendation: keep default behaviour, add an opt-in test seed.
|
||
5. **Ring palette sync (§5.3):** auto-generate `vcv/src/palette.hpp` from `colors.css` via a codegen step, or hand-sync? Recommendation: small codegen so a token change updates both surfaces.
|
||
6. **Derived outputs on the 16-out module (§5.1):** keep MEAN/STD/DELTA/NOVELTY/CONFIDENCE as menu-toggled extras / move to the expander, or drop them? Recommendation: move to the expander; keep the 16 raw outputs + LED rings as the headline panel.
|
||
7. **Default active backend:** confirm **WebAudioBackend (firmware-parity engine)** is the default, with the Powerful Synth (C15 path), particles, MIDI, OSC, CV, VCV selectable in the dock.
|
||
|
||
---
|
||
|
||
## 10. Cited source files (absolute paths)
|
||
- Particle algorithm (faithful port): `/home/w1n5t0n/deployments/meml-aimmersive/js/ui/visualizer.js` (`:5–288`)
|
||
- Built-in synth: `/home/w1n5t0n/deployments/meml-aimmersive/js/synth/c15-adapter.js`, `…/c15-bridge.js`, `…/param-map.js` (`:287` curve math), `…/presets.js`
|
||
- MIDI: `/home/w1n5t0n/deployments/meml-aimmersive/js/midi/midi-output.js` (`:114` batch throttle), `…/midi/midi-cc-map.js`
|
||
- OSC client: `/home/w1n5t0n/deployments/meml-aimmersive/js/nisps/osc-client.js`; param-named client `…/js/synth/osc-output.js` (`:97`)
|
||
- OSC bridge server: `/home/w1n5t0n/deployments/meml-aimmersive/osc-bridge/bridge.ts` (addresses `:298–305`, encode/decode `:70–182`), `bridge.mjs`, `compile.sh`
|
||
- VCV module: `/home/w1n5t0n/src/MEMLNaut-NISPS/vcv/src/MEMLNaut.cpp` (threading `:67–312`, verdict `:412–445`, ranges `:818–843`, display `:699`, LED `:804`), `…/vcv/src/osc_server.hpp`, `…/vcv/plugin.json`, `…/vcv/SPEC.md`, `…/vcv/Makefile`, `…/vcv/NISPS-FORMAT.md`, `…/vcv/res/*.svg`
|
||
- Design tokens (ring palette): `/home/w1n5t0n/src/MEMLNaut-NISPS/docs/redesign/manifold-export/tokens/colors.css`
|
||
- Spine/engine context: `/home/w1n5t0n/src/MEMLNaut-NISPS/docs/redesign/engine-architecture.md` (§2), `…/findings-design-and-manifold.md` (§4), `…/findings-engine-surface.md`, `…/aimmersive-clone-spec.md` (routeOutputs §6, §7 visual table, §10)
|
||
|
||
Sources (VCV SDK / widgets): [VCV custom lights](https://community.vcvrack.com/t/how-to-use-custom-lights/1941), [Migrate2 (drawLayer/layer 1)](https://vcvrack.com/manual/Migrate2), [Plugin Development Tutorial (RACK_DIR/make dist)](https://vcvrack.com/manual/PluginDevelopmentTutorial), [Plugin API Guide](https://vcvrack.com/manual/PluginGuide).
|
||
---
|
||
|
||
## Verification corrections (adversarial pass, 2026-06-27) — verdict: minor-issues
|
||
|
||
Strongly grounded: every cited file exists, the particle algorithm (§4) is reproduced faithfully, all 8 token
|
||
hex values verify against `colors.css`, the VCV cites + OSC bridge protocol + MIDI/OSC salvage files + C15
|
||
adapter chain are accurate. Apply these fixes:
|
||
|
||
- **`applyCurve` must keep the input clamp.** Real code (`param-map.js:287-291`) clamps to [0,1] BEFORE the
|
||
pow: `Math.pow(clamp01(value), exponent)`. Drop the "bit-identical" wording or add the clamp to the TS port.
|
||
- **`ParamMeta.id` vs `name` conflation** — the example `id: 'Env_A_Att'` is actually the `name` field; keep
|
||
`id` (slug) and `name` (param key) distinct when porting `param-map.js`.
|
||
- VCV module already at `vcv/` (2-in/12-out) — evolve to 8→16, do not rebuild; reuse its threading/OSC/serialisation.
|
||
|
||
---
|
||
|
||
## Implementation status (2026-06-28)
|
||
|
||
MIDI + OSC backends are BUILT in `manifold/src/backends/` (OutputBackend interface + BackendManager consuming the
|
||
spine; WebMIDI out with per-output CC#/ch/name/range; OSC-over-WS to the Deno bridge in `manifold/osc-bridge/`).
|
||
The Outputs dock panel specialises per backend (`manifold/src/dock/OutputsBackendConfig.tsx`) with named presets
|
||
(`manifold/src/backends/presets.ts`). Audio gated via `engine.audio.setMuted` on non-synth modes.
|
||
|
||
**VCV module** — see `vcv/SPEC.md` "⚠️ BUILD DELTAS (2026-06-28)" for the authoritative build target: 8 inputs ×
|
||
16 outputs, an LED ring per output (drawLayer + nvgArc), palette from the frontend tokens, WS↔OSC bridge
|
||
(browser OSC backend → `manifold/osc-bridge/` Deno relay → the module's OSC server), bidirectional training, and
|
||
the `nisps-core`→`nisps/` core-path repoint. The existing `vcv/` module (2→12) is evolved, not rebuilt.
|
||
|
||
**Still TODO** (clear in-code): the VCV↔browser bridge browser-side wiring to the new `/nisps/feedback` verb;
|
||
the particle backend is a no-op passthrough pending the `visualizer.js` faithful port; CV/gate transport.
|