--- kind: spec stability: evolving layer: binding counterpart: aimmersive-clone-spec.md --- # Output Backends — Specification *Workstream E. Design-only, read-only audit 2026-06-27. The new app is Vite + React + TS in `manifold/`, wired to the parity-tested TS engine lifted from `playground/src`. British spelling in product copy. The built-in synth is the **"Powerful Synth Engine"** — the string "C15" must never reach the user.* > **Naming guard (non-negotiable).** The codename `C15` survives only in internal module/file names that the user never sees (`c15-adapter.js`, `c15-bridge.js`). Every label, tooltip, dock entry, menu item, status string, and aria-label says **"Powerful Synth Engine"** (or just "Synth"). A lint allowlist + a Playwright assertion (`expect(page).not.toContainText('C15')`) enforce this across `manifold/` and the VCV panel SVG/strings. --- ## 0. The one idea: backends are adapters behind one interface Today the deployed app (`js/a-app.js`) fans output out to four ad-hoc sinks inline in `routeOutputs()` (`a-app.js:2425`): synth (`activeEngine.setParam`), MIDI CC (`midiOutput.sendBatch`), audio-canvas, and visual (`visualizer.setParams`). Each has its own throttle, dead-zone, and override handling copy-pasted. That fan-out *is* the debt this workstream removes. **Replace it with one `OutputBackend` interface and a registry. Exactly one backend is "active" at a time, chosen in the Console dock.** The reactive spine (per `engine-architecture.md` §2 and `findings-design-and-manifold.md` §4) ends in a single side-effect that calls `activeBackend.send(routedOutput)`. Swapping backends swaps nothing else — the input pipeline, ML, output pipeline, training loop, and verdict loop are all backend-agnostic. ``` gesture → input pipeline → ML (WasmIML) → output pipeline → [ activeBackend.send(routed) ] │ ┌───────────────┬───────────────┬───────────────┬─────────────┼──────────────┐ WebAudioBackend ParticleBackend WebMidiBackend OscBridgeBackend CvGateBackend VcvBridgeBackend (Powerful Synth) (flow field) (advanced CC) (paths+ranges) (1V/oct+gate) (8→model→16, LED rings) ``` The active backend is a property of the **output dock** (the Console's right rail / a-immersive's Mode drawer). Backends self-describe (id, label, capability) so the dock renders a picker without hard-coding the list. --- ## 1. The `OutputBackend` adapter interface (TS) Lives at `manifold/src/engine/backends/backend.ts`. The engine never imports a concrete backend; it imports the interface + the registry. Concrete backends may import the engine's pure helpers (curves, param-map data) but **never** React. ```ts // manifold/src/engine/backends/backend.ts /** What a backend needs to know about the active mode to map outputs. */ export interface BackendContext { modeId: string; outputCount: number; // model output dims actually in use (≤ 126) paramMeta: ReadonlyArray; // name/label/min/max/curve/group per output sampleRate?: number; // for audio backends audioContext?: AudioContext; // lazily provided; only audio backends use it } export interface ParamMeta { id: string; // stable machine id, e.g. 'Env_A_Att' label: string; // user-facing min: number; // baseline range floor (normalised 0..1 maps here) max: number; // baseline range ceil curve: number; // 0..1, 0.5 = linear (see §3 universal mapping) group: string; // for colour grouping (LED rings, heatmap) } export type BackendId = | 'synth' | 'particles' | 'midi' | 'osc' | 'cvgate' | 'vcv'; export interface BackendDescriptor { id: BackendId; label: string; // dock label — NEVER "C15" description: string; /** crossOriginIsolated / WebMIDI / WebSocket etc. availability probe. */ isAvailable(): boolean; /** true when this backend can ALSO feed inputs back (VCV bridge, OSC return). */ bidirectional?: boolean; } export interface OutputBackend { readonly descriptor: BackendDescriptor; /** Called once when this backend becomes active. May be async (audio start, * WS connect, MIDI access). Resolve only when ready to receive send(). */ start(ctx: BackendContext): Promise; /** Hot per-frame path. `routed` is the post-pipeline Float32Array (0..1), * length = ctx.outputCount. MUST NOT allocate; MUST NOT mutate `routed`. * Throttling/dead-zone live INSIDE each backend (rates differ per sink). */ send(routed: Float32Array): void; /** Called when switching away or unmounting. Release WS/MIDI/audio/threads. */ teardown(): Promise; /** Optional input return path for bidirectional backends. The engine * subscribes; values drive model inputs (e.g. VCV CV-in, OSC /nisps/input). */ onInputs?(cb: (values: Float32Array) => void): () => void; /** Optional: backends that own training transport (VCV/OSC bridge) expose * the remote verdict/example surface here. See §7. */ remote?: RemoteTrainingBridge; } ``` Registry (`manifold/src/engine/backends/registry.ts`): a `Map OutputBackend>` of lazy factories. The dock reads `descriptors` (filtered by `isAvailable()`); selecting one calls `engine.setBackend(id)`, which `teardown()`s the old and `start()`s the new with the current `BackendContext`. **Why `send(Float32Array)` and not per-param events:** matches the spine's single transferable-buffer effect (`engine-architecture.md` §2.1), keeps the hot path allocation-free, and lets each backend decide its own decimation. The legacy code already proves the pattern — every sink takes the full output vector and self-throttles (synth 50 ms/0.002 dead-zone `a-app.js:2425`; MIDI 50 ms/Δ1 `midi-output.js:114`; OSC 50 ms/0.002 `osc-output.js:97`). --- ## 2. The backends ### 2.1 Built-in Synth — the "Powerful Synth Engine" (`web-audio.ts` + `synth.ts`) Two cooperating backends, both labelled as the synth in the UI, but architecturally distinct: - **`WebAudioBackend`** wraps the parity-tested repo engine: `EngineHost` (`playground/src/audio/engine-host.ts`) + the worklet `nisps-processor.ts` running `_nisps_engine_process_block` (`nisps/wasm/bindings.cpp`). This is the firmware-parity audio path — the engine *is* the sound. `send()` → `EngineHost.setParams(routed)` → worklet. This is the default and the one that satisfies browser-parity chokepoint C. - **`PowerfulSynthBackend`** (the C15 path) wraps `deployments/meml-aimmersive/js/synth/c15-adapter.js` → `c15-bridge.js` (SharedArrayBuffer ring + its own `c15_engine.wasm` worklet). Param mapping comes from `param-map.js` (`SYNTH_PARAM_MAP`, 126 entries) and `presets.js` (tiered presets). `setParam(index, normalised)` maps index→hardware id (`c15-adapter.js:121`). This is browser-only (firmware has no C15), and its 126-param surface + group/section overrides power the synth visualiser and the group-override drawer. **Reuse, verbatim:** `c15-adapter.js`, `c15-bridge.js`, `param-map.js`, `presets.js` move under `manifold/src/engine/backends/synth/` unchanged (internal names keep "c15"; UI strings do not). The `engine-interface.js` `SynthEngine` base maps cleanly onto `OutputBackend`: `init(ctx)`→`start`, `setParam` loop driven by `send`, `stop`→`teardown`. Curve/override math is `applyCurve`/`applyGroupOverride` (`param-map.js:287`) — fold into the universal mapping (§3). **Throttle (keep — load-bearing):** ≥50 ms send interval + 0.002 dead-zone per param prevents flooding the C15 ring buffer at 126×30 fps (`a-immersive.html` clone-spec §10 flags this). `isAvailable()`: WebAudio + (for the C15 path) `crossOriginIsolated === true` (SAB needs COOP/COEP; already server-scoped per `findings-engine-surface.md`). ### 2.2 Particle System — faithful port of `visualizer.js` (`particles.ts`) A `ParticleBackend` whose `send(routed)` calls a ported `FlowFieldVisualizer.setParams(routed)`. The visual canvas runs in its own `requestAnimationFrame` loop (per `findings-design-and-manifold.md` §4.3 — rAF touches drawing only, never inference); `send()` only updates the param struct. The port MUST look and behave **exactly** as the deployed version. The full algorithm is documented in §4 so the React port is byte-faithful. `isAvailable()`: always (Canvas2D). This backend produces no audio — it is the "visual" output mode. ### 2.3 MIDI out — advanced CC config (`web-midi.ts`) A `WebMidiBackend` wrapping the salvaged `midi-output.js` (Web MIDI API, `sendBatch`, per-CC dead-zone + 50 ms throttle, device hot-plug handling — `midi-output.js`). The **advanced CC config comes from workstream D's config model**, persisted as a CC map: per output dim → `{ name, cc (0–127), channel (1–16), min, max, curve, muted, fixedValue }`. The map shape and storage are already defined in `midi-cc-map.js` (`createCCParam`, `loadCCMap`/`saveCCMap`, well-known `CC_NAMES`, default 8-CC starter set). Lift that file verbatim into `manifold/src/engine/backends/midi/cc-map.ts`. `send(routed)`: for each non-muted CC param, `value = round(applyGroupOverride(routed[i], curve, min, max) * 127)`; batch the changed ones; `midiOutput.sendBatch(...)`. Storage key stays engine-scoped (`nisps-midi-cc-map:`) for migration continuity. `isAvailable()`: `!!navigator.requestMIDIAccess`. ### 2.4 OSC out — paths + ranges (`osc-bridge.ts`) An `OscBridgeBackend` that **salvages the existing OSC bridge** rather than reinventing it. Two pieces already exist and are good: - **Browser client:** `deployments/meml-aimmersive/js/synth/osc-output.js` (param-named WS messages, 50 ms/0.002 dead-zone) and the richer `js/nisps/osc-client.js` (`NispsOscClient` — `EventTarget`, `sendState`/`sendWeights`/`sendParams`, `onOutputsReceived`/`onInputsReceived`, auto-reconnect with backoff). Lift `osc-client.js` as the transport (it already speaks the bridge protocol and is bidirectional). - **Bridge server:** `deployments/meml-aimmersive/osc-bridge/bridge.ts` — a Deno WebSocket↔UDP-OSC bridge, zero-dependency OSC encode/decode, bidirectional. Keep it as-is; it is the canonical transport between browser and any OSC target (VCV, SuperCollider). **OSC path + range contract (salvaged from `bridge.ts`):** | Direction | Address | Args | Meaning | |---|---|---|---| | browser→target | `/nisps/` | `f` | one param, **post-baseline-mapping value** (see §3) | | browser→target | `/nisps/state` | `s` | full JSON state (weights + examples + config) | | browser→target | `/nisps/weights` | `s` | weights-only JSON | | target→browser | `/nisps/output` | `f…f` | output vector (visualisation / monitoring) | | target→browser | `/nisps/input` | `f…f` | input vector → drives model inputs | **Ranges:** OSC floats are sent in the param's mapped range by default (`applyGroupOverride` applied before send, matching `osc-output.js`), with a per-backend toggle to send **raw normalised 0..1** instead (some OSC targets want 0..1 and do their own scaling). Address prefix (`/nisps`), target host/port (default `127.0.0.1:9000`), and listen port (default `9001`) are configurable — `bridge.ts` already exposes `--osc-host/--osc-port/--osc-prefix/--ws-port/--listen-port`. `isAvailable()`: always (attempts WS to `ws://localhost:8765`; surfaces a "bridge not running" status if the connect fails — `osc-client.js` already reconnects with backoff). Bidirectional (`onInputs` wired to `/nisps/input`). ### 2.5 CV / gate backend (`cvgate.ts`) For browser-side CV/gate there is no native hardware path, so this backend has **two transports** selectable in config: 1. **DC-coupled WebAudio CV** (browser-native): each output dim drives a `ConstantSourceNode` (or a sample-accurate `AudioWorklet` channel) whose `offset` = mapped voltage, summed/routed to the audio interface's output channels. Gate outputs are derived from a configurable threshold on a chosen dim (value > τ → high). Pitch (1V/oct) uses a per-output "voltage role" config: `{ role: 'cv' | 'gate' | 'voct', vmin, vmax, gateThreshold }`. This is the only way to emit real CV from a browser (DC-coupled interface required; surfaced as a caveat in the UI). 2. **Bridged CV via VCV / OSC** (recommended default): reuse the OSC/VCV transport — the *VCV module's 16 CV outputs* (§5) are the real CV/gate jacks. In this mode `CvGateBackend` is a thin alias that delegates to `VcvBridgeBackend` with a "treat outputs as CV/gate" preset (per-output unipolar 0–10 V / bipolar ±5 V / 1V-oct, matching the VCV per-output range menu in `MEMLNaut.cpp:818`). `isAvailable()`: WebAudio path always; native-CV quality flagged as "requires DC-coupled interface". **Open choice:** whether browser-native DC CV is worth shipping vs. making CV strictly a VCV-bridge concern (recommendation: ship the VCV-bridge alias first, defer DC-coupled WebAudio CV). ### 2.6 VCV Rack module — first-class (`vcv-bridge.ts` browser side + `vcv/` C++ side) The headline new backend. A first-class **VCV Rack 2 module** (`MEMLNaut`) with **8 CV inputs → model → 16 CV outputs**, an **LED ring around each of the 16 outputs**, and a **browser↔VCV bridge** so the tool is controllable AND trainable from both inside Rack and entirely from the browser. Full design in §5–§7. The browser-side adapter `VcvBridgeBackend` reuses `NispsOscClient` (§2.4) as transport. When active in **bridged mode**, the browser supplies inputs in real time (`/nisps/input`) and the verdict/example loop is mirrored over the bridge (`/nisps/state`, `/nisps/weights`, new `/nisps/feedback`). --- ## 3. Universal per-output baseline mapping Every backend shares ONE baseline mapping from a normalised model output `v ∈ [0,1]` to a sink value, so behaviour is identical across sinks and the override UI (heatmap popup, group drawer) is backend-agnostic. This is the existing curve math (`param-map.js:287`), promoted to `manifold/src/engine/backends/mapping.ts`: ```ts // 0.5 = linear; <0.5 ease-in, >0.5 ease-out. Bit-identical to legacy + nisps/core/math.hpp. export function applyCurve(v: number, c: number): number { return c === 0.5 ? v : Math.pow(v, Math.pow(2, 4 * (c - 0.5))); } /** Per-output baseline: curve, then scale into [min,max]; honour freeze/mute. */ export function mapOutput(v: number, p: OutputMapping): number { if (p.frozen) return p.fixedValue; // pinned, ignores model return p.min + applyCurve(v, p.curve) * (p.max - p.min); } export interface OutputMapping { min: number; max: number; curve: number; // baseline range + curve frozen: boolean; fixedValue: number; // pin muted?: boolean; // excluded from this sink } ``` Backends then apply only their **sink-specific final transform** on top of the baseline: | Backend | Baseline → sink transform | |---|---| | Powerful Synth | `mapOutput` → `setParam(i, value)` (value already in param range) | | Particles | **raw 0..1** (the visualiser owns its own param ranges — see §4); curve/freeze still applied, min/max default to 0..1 | | MIDI CC | `mapOutput` → `round(value * 127)` clamped 0–127 | | OSC | `mapOutput` (or raw 0..1 if "send raw" toggled) → `/nisps/ ` | | CV/gate | `mapOutput` → voltage by role (`cv`: `value*10` or `(value-0.5)*10`; `voct`: 1V/oct; `gate`: `value>τ ? high : 0`) | | VCV | model output `0..1` sent raw over bridge; the **module** applies its own per-output range + attenuverter (`MEMLNaut.cpp:342 outputToVoltage`) | The override store (one per mode, `OutputMapping[]`) is owned by the engine and shared by all backends; freeze/mute/curve/range edits in the UI apply uniformly. Note the legacy split where the heatmap calls it `frozen` and the group drawer calls it `muted` but both map to the same field (`aimmersive-clone-spec.md` §10) — unify to the single `OutputMapping` above. --- ## 4. Particle system — faithful-port plan with the documented algorithm Source of truth: `deployments/meml-aimmersive/js/ui/visualizer.js` (289 lines, read in full). The React port (`manifold/src/engine/backends/particles/flow-field.ts`) must reproduce this **exactly**. Below is the complete algorithm with line citations so the port is verifiable. ### 4.1 Constants & noise source - **Particle count: `numParticles = 400`** (`visualizer.js:53`). - **Noise: a self-contained 2-D value/Perlin-style noise**, not a library. A `Uint8Array(512)` permutation table `PERM` is built once at module load by Fisher–Yates shuffling `[0..255]` then duplicating (`:5–14`). `fade(t)=t³(t(6t−15)+10)` (`:16`), `lerp` (`:17`), `grad(hash,x,y)` using `hash & 3` (`:19–24`), `noise2D(x,y)` doing the standard 4-corner bilinear-with-fade interpolation (`:26–44`). **The shuffle uses `Math.random()` at module load**, so the field is non-deterministic per page load — the port must keep this (or seed it; flagged as an open choice if determinism is wanted for tests). - `TWO_PI = Math.PI*2` (`:46`); `this.time` advances `+= 0.003` per `draw()` (`:185`). ### 4.2 Output→param mapping (20 dims, `setParams`, `:159–181`) — reproduce verbatim Guard: returns early if `outputs.length < 20` (`:160`). Then: ``` angleOffset = out0 * TWO_PI scale = 0.001 + out1 * 0.009 speed = 0.5 + out2 * 4.5 hueBase = out3 * 360 hueSpread = out4 * 120 particleSize = 1 + out5 * 5 fadeRate = 0.01 + out6 * 0.14 turbulence = out7 * 2 attractStrength = 0.1 + out8 * 2.9 attractRadius = 40 + out9 * 420 dispersionRate = 0.2 + out10 * 8 dispersionAmount = out11 * 3 particleLifetime = 30 + out12 * 470 respawnStyle = out13 // 0=random,1=edge,2=center-burst advectionMode = out14 // flow→orbit→radial blend inertia = out15 * 0.98 drag = out16 * 0.35 repulsorStrength = out17 * 4.5 repulsorCount = floor(out18 * 4.999) // 0..4 repulsorOrbitRate = 0.1 + out19 * 2.9 ``` The 20 output names (for labels/the heatmap colouring) are `VISUAL_PARAM_NAMES` in `a-app.js:46`: `Flow, Scale, Speed, Hue, Spread, Size, Trail, Turb, Attract, Radius, DispRate, DispAmt, Lifetime, Respawn, Advection, Inertia, Drag, Repulse, RepCnt, RepRate`. ### 4.3 Particle lifecycle (`:104–156`) - `makeParticle(id)`: random x,y in canvas; `age = floor(rand*particleLifetime)`; `life = computeLifetime()`; `vx=vy=0` (`:104–114`). - `computeLifetime()`: `max(10, floor(particleLifetime * (0.65 + rand*0.7)))` (`:116–119`). - `respawnParticle(p)`: `mode = min(2, floor(respawnStyle*2.999))` (`:122`): - **edge (1):** spawn on a random one of 4 sides; give inward impulse of magnitude 2.0 toward centre (`:125–137`). - **center-burst (2):** spawn within radius `min(w,h)*0.08` of centre at random angle; velocity `2.5` outward along that angle (`:138–145`). - **random (0):** random position; velocity `(rand*2−1)*0.5` each axis (`:146–152`). - reset `age=0`, `life=computeLifetime()`. ### 4.4 Per-frame `draw()` (`:183–287`) — the exact integration 1. `time += 0.003`. **Trail fade:** fill whole canvas with `rgba(13,13,13, fadeRate)` (`:188`) — this is the trail length control, not a clear. 2. For each particle, with centre `(cx,cy)=(w/2,h/2)`: - **Flow field:** `nx=p.x*scale`, `ny=p.y*scale`; `angle = noise2D(nx+time, ny)*TWO_PI + angleOffset`; `curl = noise2D(nx+100, ny+100+time*0.5)*turbulence` (`:196–199`). - **Three advection fields** (`:202–214`): flow `(cos(angle+curl), sin(angle+curl))*speed`; orbit = perpendicular to the radial-from-centre unit vector, `*speed`; radial = radial unit vector `*speed`. - **Blend by `advectionMode`** (`:216–225`): `modeBlend = advectionMode*2`; if `<1` lerp(flow→orbit, modeBlend) else lerp(orbit→radial, modeBlend−1). - **Inertia + drag** (`:226–229`): `vx = vx*inertia + targetVx*(1−inertia)` (same for vy); then `vx *= (1−drag)`. - `nextX = x+vx`, `nextY = y+vy`. - **Central attractor** (`:233–242`): pull toward centre with `falloff = 1/(1+normDist²)` where `normDist = min(dist/attractRadius, 2)`; add `nCenter * attractStrength * falloff`. - **Dispersion pulse** (`:244–248`): `pulse = 0.5+0.5*sin(time*dispersionRate + id*0.07)`; subtract `nCenter * dispersionAmount*pulse*falloff` (pushes outward near centre). - **Orbiting repulsors** (`:250–264`): `repulsorRadius = min(w,h)*0.28`; for each of `repulsorCount`: phase from `time*repulsorOrbitRate + (r/4)*TWO_PI`, `wobble = 0.6+0.15*r`, position uses `cos(phase*(1+wobble))`/`sin(phase*(1.3+wobble))`; inverse-square push `force = repulsorStrength*(650/distSq)` with `distSq = d²+160`. - **Wrap edges** (`:270–273`); `age += 1`; respawn when `age >= life` (`:275–276`). - **Colour** (`:279–282`): `hue = (hueBase + (id/numParticles)*hueSpread) % 360`; `lightness = 50 + sin(id*0.1 + time)*15`; fill `hsl(hue, 75%, lightness%)`; `arc(x,y,particleSize)`. ### 4.5 Port plan - **Class, not component.** `FlowFieldVisualizer` is a plain TS class taking a `` ref — identical to today. The React `` mounts it in `onMount`, drives `draw()` from one `requestAnimationFrame` loop, and calls `resize()` on the window resize handler (DPR scaling at `:84–92`). React renders the canvas element; the class owns all pixels. - **`ParticleBackend.send(routed)`** → `viz.setParams(routed)` (no alloc; just field writes). Because the visualiser reads `outputs[0..19]`, the backend asserts `ctx.outputCount >= 20` and slices/pads to 20. - **Faithfulness gate (Playwright):** pixel-diff a fixed seed (seed the `PERM` shuffle behind a `?seed=` for tests) at fixed param vectors against a golden capture from the deployed app; assert SSIM ≥ threshold. Also unit-test `setParams` mapping numerically (the §4.2 table). - **Verbatim copy is allowed**: the noise + integration math is pure and has no DOM coupling beyond `ctx`/`canvas`; lift `:1–288` essentially unchanged into TS, add types, keep numeric constants exact. --- ## 5. VCV Rack module design (`MEMLNaut`, 8 in / 16 out) There is **already a working VCV module** at `vcv/` (`MEMLNaut.cpp`, 959 lines; `plugin.json`; `SPEC.md`; `osc_server.hpp`; `Makefile`; `res/*.svg`) — but it is **2-in / 12-out + 5 derived**. The new requirement is **8-in / 16-out with LED rings**. This is an evolution of the existing module, reusing its threading model, OSC server, and serialization wholesale. ### 5.1 What changes vs. the existing module - `NUM_ML_INPUTS`: 2 → **8** (the existing `MAX_ML_INPUTS = 8` already anticipated this; `MEMLNaut.cpp:14`). All 8 are first-class jacks (not "reserved"). - `NUM_ML_OUTPUTS`: 12 → **16**. The 5 derived outputs (MEAN/STD/DELTA/NOVELTY/CONFIDENCE) remain but become **optional context-menu extras** or move to the expander — the 16 raw outputs are the headline. - MLP shape: stays within "the modular N×M envelope" — `nisps::IML iml{8, 16, {16, 24, 16}}` (the existing default hidden stack is fine; sized for real-time inference per `SPEC.md`). Inputs feed model input dims; the 16 outputs are the inference outputs. - **LED ring per output** replaces the single `SmallLight` next to each jack (`MEMLNaut.cpp:804`). ### 5.2 Panel layout (Wide, ~32–44 HP) ``` ┌────────────────────────────────────────────┐ │ MEMLNaut │ ← brand; "Powerful Synth" wording N/A (this is the CV mapper) │ ┌──────────────────────────────────────┐ │ │ │ DISPLAY: 16 bars + XY dot + metrics │ │ ← NanoVG LedDisplay/drawLayer (existing :699) │ └──────────────────────────────────────┘ │ │ SPREAD RATE LEARN● RAND CLEAR │ ← knobs + buttons (existing controls) │ [+] [−] (verdict buttons) │ │ │ │ INPUTS (8 jacks, 2 rows × 4) │ │ IN1 IN2 IN3 IN4 │ │ IN5 IN6 IN7 IN8 + LEARN_GATE +TRIG −TRIG│ │ │ │ OUTPUTS (16, 4 rows × 4), each: │ │ ◉jack with an LED RING around the jack │ │ [(◯1)(◯2)(◯3)(◯4)] │ │ [(◯5)(◯6)(◯7)(◯8)] │ │ [(◯9)(◯10)(◯11)(◯12)] │ │ [(◯13)(◯14)(◯15)(◯16)] │ │ (optional) MEAN STD DELTA NOVELTY CONF │ └────────────────────────────────────────────┘ ``` Each output is a `PJ301MPort` jack with a **`LedRingWidget`** drawn concentric around it (no separate attenuverter trimpot in the default skin — attenuverter moves to right-click/expander to make room for the ring; keep `PARAM_ATTEN_*` in the model for range scaling). ### 5.3 The LED-ring widget + palette mapping A custom widget that draws a ring whose **arc fill is proportional to the output value** and whose **colour comes from the frontend design tokens**. Per the Rack manual, self-illuminating custom widgets override `drawLayer(args, 1)` and draw on layer 1 (so they stay bright when room brightness is lowered) ([VCV custom lights](https://community.vcvrack.com/t/how-to-use-custom-lights/1941), [Migrate2](https://vcvrack.com/manual/Migrate2)). ```cpp // vcv/src/LedRing.hpp (new) struct LedRingWidget : Widget { MEMLNaut* module = nullptr; int outIdx = 0; NVGcolor ringColor = nvgRGB(0xff, 0x6a, 0x00); // default --accent float radius = 7.f; // mm-ish, around a PJ301M jack void drawLayer(const DrawArgs& args, int layer) override { if (layer != 1 || !module) return; float v = clamp(module->slewOutputs[outIdx], 0.f, 1.f); // 0..1 Vec c = box.size.div(2); // track (dim full ring) nvgBeginPath(args.vg); nvgArc(args.vg, c.x, c.y, radius, -M_PI/2, -M_PI/2 + 2*M_PI, NVG_CW); nvgStrokeColor(args.vg, nvgRGBA(ringColor.r*255, ringColor.g*255, ringColor.b*255, 40)); nvgStrokeWidth(args.vg, 1.4f); nvgStroke(args.vg); // value arc (proportional) nvgBeginPath(args.vg); nvgArc(args.vg, c.x, c.y, radius, -M_PI/2, -M_PI/2 + v*2*M_PI, NVG_CW); nvgStrokeColor(args.vg, ringColor); nvgStrokeWidth(args.vg, 1.8f); nvgStroke(args.vg); // glow halo (matches the frontend "glow not shadow" signature) nvgGlobalCompositeOperation(args.vg, NVG_LIGHTER); /* … bloom pass … */ } }; ``` **Palette mapping — derived from `docs/redesign/manifold-export/tokens/colors.css` (read).** Ring colours come from the design tokens so VCV matches the frontend. A `kRingPalette[16]` table assigns each output a colour by its **parameter group**, cycling through the token accents and group/pin colours: | Source token (colors.css) | Hex | Used for | |---|---|---| | `--accent` | `#ff6a00` | primary outputs / group 0 (orange — the live colour) | | `--accent-2` | `#00ccff` | data outputs / group 1 (cyan) | | `--accent-3` | `#ffa860` | group 2 (warm tint) | | `--pin-3` base `#b464ff` | `#b464ff` | group 3 (violet) | | `--good` | `#6bc26b` | group 4 (green) | | `--warn` | `#f5c45e` | group 5 (amber) | | `--info` | `#5b9eef` | group 6 (blue) | | `--danger` | `#ff4466` | bipolar / perturbed outputs (red) | Mapping rule: `ringColor = kRingPalette[paramMeta[i].group % paletteLen]`, so outputs in the same mode-group glow the same colour, identical to the heatmap/Console grouping. Bipolar outputs (per-output range menu) tint toward `--danger`. The palette is a single header (`vcv/src/palette.hpp`) generated from `colors.css` so a token change propagates to both frontend and module (a small codegen step; **open choice** whether to automate or hand-sync). ### 5.4 Reused, unchanged from the existing module - **Threading model** (`MEMLNaut.cpp:67–312`): audio-thread `iml` + worker-thread `imlShadow`, job queue (Train/Perturb/Randomize/Clear), atomic weight hand-off, single-writer invariant. Already correct; just resize the I/O. - **Inference-rate decimation + slew + post-swap crossfade** (`:447–499`). - **Verdict loop** (`:412–445`): `+`/`−` buttons and `+TRIG`/`−TRIG` gated by LEARN. `+` → add example (current inputs→current outputs) + enqueue Train + decay noise ×0.97; `−` → bump noise (cap `0.3(1−s)+0.05s`) + enqueue Perturb. Identical semantics to the browser verdict loop. - **Serialization** (`dataToJson`/`dataFromJson`, `:548–696`) + **`.nisps` preset save/load** (`:862–924`) + **OSC server** (`osc_server.hpp`). The `.nisps` format is the shared interchange with the browser (`vcv/NISPS-FORMAT.md`). - Per-output / per-input **range menu** (uni/bipolar, attenuverter) (`:818–843`). ### 5.5 Plugin scaffold files ``` vcv/ ├── plugin.json # slug MEMLNaut (exists) — bump version; tags Controller/Utility/Random ├── Makefile # VCV SDK Makefile (exists); RACK_DIR-driven ├── Makefile.dist # dist packaging (exists) ├── src/ │ ├── plugin.{hpp,cpp} # plugin init / model registration (exist) │ ├── MEMLNaut.cpp # module — resize 8 in / 16 out (edit) │ ├── LedRing.hpp # NEW — LED-ring widget (§5.3) │ ├── palette.hpp # NEW — ring colour table from colors.css (§5.3) │ └── osc_server.hpp # OSC server (exists; extend with /nisps/feedback §7) ├── res/ │ ├── MEMLNaut.svg / -wide.svg / -expander.svg # panels (exist; redraw for 8/16 layout) ├── test/smoke_test.cpp # exists ├── plugin.json, SPEC.md, BUILDING.md, README.md, NISPS-FORMAT.md (exist) ``` --- ## 6. Browser↔VCV bridge design The module must be **controllable AND trainable from BOTH inside Rack AND entirely from the browser**. Both ends operate on the same model; the bridge keeps them coherent. ### 6.1 Transport — propose options (operator open choice) The existing salvage path is **WebSocket↔UDP-OSC** (`osc-bridge/bridge.ts` + `osc-client.js` + `osc_server.hpp`). This is the recommended default and already works. Three options to flag: | Option | How | Pros | Cons | |---|---|---|---| | **A. WS↔OSC bridge server (recommended)** | Browser ⇄ `bridge.ts` (Deno WS server, localhost:8765) ⇄ UDP OSC ⇄ module's `osc_server.hpp` (port 9000/9001) | Already built + bidirectional; standard OSC; works with SuperCollider/Max too; no browser perms | Needs a helper process running locally (Deno or the compiled `bridge.mjs`) | | **B. Direct WebMIDI** | Browser ⇄ Web MIDI ⇄ a virtual MIDI port ⇄ a tiny MIDI-in path in the module | No helper process if a virtual MIDI port exists; browser-native | 7-bit/14-bit only — too coarse for weights/state; really only for live CC; module would need MIDI parsing | | **C. Native (module hosts a WS server)** | The VCV module itself opens a WebSocket/HTTP server; browser connects directly | No external bridge process | Adds a WS/TLS stack inside the plugin; COOP/COEP + mixed-content (`https://` page → `ws://localhost`) friction; more attack surface in the audio plugin | **Recommendation:** ship **A** (it exists, it's bidirectional, it already targets this very module — see `osc-client.js:1` "Connects the webapp to VCV Rack MEMLNaut module"). Keep **B** as a live-performance CC convenience only. Treat **C** as a future "no-helper" nicety. **This transport choice is an explicit operator open choice.** ### 6.2 Two modes - **Standalone:** module runs entirely inside Rack (CV in → model → CV out; verdict via panel buttons/triggers). No browser. Works today. - **Bridged:** browser connects via the bridge. In bridged mode **the browser supplies inputs in real time** — the Manifold pointer / joystick / pads stream `/nisps/input ` to the module, which uses them instead of (or blended with, configurable) the physical CV-in jacks. The module streams `/nisps/output` and `/nisps/input` back at ~100 ms (`MEMLNaut.cpp:536`) for browser visualisation. State/weights sync both ways via `/nisps/state` + `/nisps/weights` (already staged atomically into the audio thread, `:360–384`). ### 6.3 Coherence model One model, one owner of weights at a time. The bridge sends **whole-model snapshots** (`/nisps/state`, `/nisps/weights`) on any structural change (train completes, randomize, clear, load preset), and **continuous I/O vectors** (`/nisps/input`, `/nisps/output`) for live feel. Last-writer-wins on weights with a short "training in progress" lock (the module already coalesces jobs at queue depth 1). The browser's `WasmIML` and the module's `nisps::IML` use the **same `.nisps` weight layout** (`vcv/NISPS-FORMAT.md`), so a snapshot from either side loads losslessly — *provided the architectures match*. **Caveat / open choice:** the browser engine is fixed `MLP<2,…,126>` while the module is `8→{16,24,16}→16`. For true weight transfer the bridged session must run a **matched architecture** (e.g. a browser mode configured to 8-in/16-out, within the modular envelope) — otherwise the bridge degrades to I/O + example transfer only (no raw-weight sync). Flag this explicitly. --- ## 7. Training over the bridge — both directions The verdict loop (place examples / thumbs-up / thumbs-down / undo) must work from **either** end and stay coherent. Add one OSC address and a small `RemoteTrainingBridge`: ```ts // manifold/src/engine/backends/backend.ts export interface RemoteTrainingBridge { thumbsUp(): void; // add current (input,output) example + train thumbsDown(): void; // perturb weights (explore) addExample(input: Float32Array, output: Float32Array): void; randomise(spread: number): void; clear(): void; undo(): void; onState(cb: (s: NispsState) => void): () => void; // remote → local sync } ``` New OSC verb: `/nisps/feedback ` carrying `{"op":"up|down|rand|clear|undo","spread":f,"input":[…],"output":[…]}`. The module's `osc_server.hpp` gets an `onFeedback` callback (mirroring the existing `onState`/`onWeights` at `MEMLNaut.cpp:120–132`) that stages the op atomically for the audio thread, which routes it through the **same** `enqueueJob`/`add_example` path the panel buttons use (`:412–445`). So: - **Browser → VCV training:** user clicks thumbs-up in the Manifold → `VcvBridgeBackend.remote.thumbsUp()` → `/nisps/feedback {op:up,input,output}` → module stages → worker trains `imlShadow` → atomic weight swap → module streams `/nisps/state` back → browser `WasmIML.setWeights` updates so the UI/heatmap reflect the new mapping. The browser need not run its own training in bridged mode (or runs it and pushes weights; configurable — see §6.3 caveat). - **VCV → browser training:** user presses `+`/`−` on the panel (or sends `+TRIG`) → module trains/perturbs → streams `/nisps/state` → browser applies, so the verdict placed in Rack appears in the browser's example list and weight-health view. - **Example placing over the bridge:** either side can `addExample`; examples ride in `/nisps/state` (the module already serializes `examples.features`/`examples.labels`, `:587–604`) so the dataset stays in sync. Undo is local-history on each side, but a remote undo can be sent as `/nisps/feedback {op:undo}` to roll the module's last job (module keeps a one-deep snapshot, matching the browser undo stack semantics). **Result:** the tool is fully controllable and trainable from inside VCV Rack and entirely from the browser, with the verdict loop and example-placing working either way over the same bridge. --- ## 8. Build / install notes ### Browser side Backends live under `manifold/src/engine/backends/`; lifted JS (`c15-*`, `param-map`, `presets`, `midi-output`, `midi-cc-map`, `osc-client`, `visualizer`) ported to TS, parity-checked. No new build step — they ride the existing Vite `manifold` build. COOP/COEP stays server-scoped (needed by the Powerful Synth's SAB path). ### OSC bridge server ```bash cd osc-bridge deno run --allow-net bridge.ts # or the compiled bridge.mjs # --osc-host 127.0.0.1 --osc-port 9000 --ws-port 8765 --listen-port 9001 ``` Ship `bridge.mjs` (already compiled) + `compile.sh` so users without Deno can run it via Node. Surface "bridge not running" in the OSC/VCV backend status (the client auto-reconnects). ### VCV module ```bash cd vcv export RACK_DIR=/path/to/Rack-SDK # VCV Rack 2 SDK make # builds plugin.so/.dylib/.dll make install # copies into the VCV user plugins dir # distribution: make dist (per Makefile.dist; produces the .vcvplugin) ``` Per the search, the standard flow is `export RACK_DIR=…; make clean; make dist` ([Plugin Development Tutorial](https://vcvrack.com/manual/PluginDevelopmentTutorial)). Requires the VCV Rack 2 SDK; nisps-core is header-only C++20 (symlinked under `vcv/dep/` per `SPEC.md`). Ship v2-only (rationale in `SPEC.md`'s v1-compat section). License caveat: VCV SDK is GPLv3, nisps-core is MPL-2.0 — combined binary is effectively GPL; not submitting to the VCV Library initially (`SPEC.md` §License). --- ## 9. Open choices for the operator 1. **Bridge transport (§6.1):** confirm **A — WS↔OSC bridge server** as default (recommended; already built and bidirectional), with WebMIDI as a live-CC-only convenience and a native in-module WS server deferred. This is the biggest call. 2. **Bridged weight-sync vs I/O-only (§6.3):** the browser engine is fixed `MLP<2,…,126>`; the module is `8→16`. Either (a) run a **matched 8-in/16-out browser mode** for true raw-weight transfer, or (b) accept that bridged sessions sync **I/O + examples only** and each side trains its own weights. Recommendation: (b) for v1, (a) when the modular N×M browser MLP lands (workstream F). 3. **CV/gate native path (§2.5):** ship only the **VCV-bridge CV alias** for v1, or also build browser-native DC-coupled WebAudio CV? Recommendation: VCV-bridge first; defer DC CV. 4. **Particle noise determinism (§4.1):** keep the `Math.random()`-seeded permutation (non-deterministic per load, faithful to today) or add a `?seed=` for reproducible visuals/tests? Recommendation: keep default behaviour, add an opt-in test seed. 5. **Ring palette sync (§5.3):** auto-generate `vcv/src/palette.hpp` from `colors.css` via a codegen step, or hand-sync? Recommendation: small codegen so a token change updates both surfaces. 6. **Derived outputs on the 16-out module (§5.1):** keep MEAN/STD/DELTA/NOVELTY/CONFIDENCE as menu-toggled extras / move to the expander, or drop them? Recommendation: move to the expander; keep the 16 raw outputs + LED rings as the headline panel. 7. **Default active backend:** confirm **WebAudioBackend (firmware-parity engine)** is the default, with the Powerful Synth (C15 path), particles, MIDI, OSC, CV, VCV selectable in the dock. --- ## 10. Cited source files (absolute paths) - Particle algorithm (faithful port): `/home/w1n5t0n/deployments/meml-aimmersive/js/ui/visualizer.js` (`:5–288`) - Built-in synth: `/home/w1n5t0n/deployments/meml-aimmersive/js/synth/c15-adapter.js`, `…/c15-bridge.js`, `…/param-map.js` (`:287` curve math), `…/presets.js` - MIDI: `/home/w1n5t0n/deployments/meml-aimmersive/js/midi/midi-output.js` (`:114` batch throttle), `…/midi/midi-cc-map.js` - OSC client: `/home/w1n5t0n/deployments/meml-aimmersive/js/nisps/osc-client.js`; param-named client `…/js/synth/osc-output.js` (`:97`) - OSC bridge server: `/home/w1n5t0n/deployments/meml-aimmersive/osc-bridge/bridge.ts` (addresses `:298–305`, encode/decode `:70–182`), `bridge.mjs`, `compile.sh` - VCV module: `/home/w1n5t0n/src/MEMLNaut-NISPS/vcv/src/MEMLNaut.cpp` (threading `:67–312`, verdict `:412–445`, ranges `:818–843`, display `:699`, LED `:804`), `…/vcv/src/osc_server.hpp`, `…/vcv/plugin.json`, `…/docs/specs/vcv-module.md`, `…/vcv/Makefile`, `…/vcv/NISPS-FORMAT.md`, `…/vcv/res/*.svg` - Design tokens (ring palette): `/home/w1n5t0n/src/MEMLNaut-NISPS/docs/redesign/manifold-export/tokens/colors.css` - Spine/engine context: `/home/w1n5t0n/src/MEMLNaut-NISPS/docs/specs/engine-architecture.md` (§2), `…/recon/findings-design-and-manifold.md` (§4), `…/recon/findings-engine-surface.md`, `…/aimmersive-clone-spec.md` (routeOutputs §6, §7 visual table, §10) Sources (VCV SDK / widgets): [VCV custom lights](https://community.vcvrack.com/t/how-to-use-custom-lights/1941), [Migrate2 (drawLayer/layer 1)](https://vcvrack.com/manual/Migrate2), [Plugin Development Tutorial (RACK_DIR/make dist)](https://vcvrack.com/manual/PluginDevelopmentTutorial), [Plugin API Guide](https://vcvrack.com/manual/PluginGuide). --- ## Verification corrections (adversarial pass, 2026-06-27) — verdict: minor-issues Strongly grounded: every cited file exists, the particle algorithm (§4) is reproduced faithfully, all 8 token hex values verify against `colors.css`, the VCV cites + OSC bridge protocol + MIDI/OSC salvage files + C15 adapter chain are accurate. Apply these fixes: - **`applyCurve` must keep the input clamp.** Real code (`param-map.js:287-291`) clamps to [0,1] BEFORE the pow: `Math.pow(clamp01(value), exponent)`. Drop the "bit-identical" wording or add the clamp to the TS port. - **`ParamMeta.id` vs `name` conflation** — the example `id: 'Env_A_Att'` is actually the `name` field; keep `id` (slug) and `name` (param key) distinct when porting `param-map.js`. - VCV module already at `vcv/` (2-in/12-out) — evolve to 8→16, do not rebuild; reuse its threading/OSC/serialisation. --- ## Implementation status (2026-06-28) MIDI + OSC backends are BUILT in `manifold/src/backends/` (OutputBackend interface + BackendManager consuming the spine; WebMIDI out with per-output CC#/ch/name/range; OSC-over-WS to the Deno bridge in `manifold/osc-bridge/`). The Outputs dock panel specialises per backend (`manifold/src/dock/OutputsBackendConfig.tsx`) with named presets (`manifold/src/backends/presets.ts`). Audio gated via `engine.audio.setMuted` on non-synth modes. **VCV module** — see `docs/specs/vcv-module.md` "⚠️ BUILD DELTAS (2026-06-28)" for the authoritative build target: 8 inputs × 16 outputs, an LED ring per output (drawLayer + nvgArc), palette from the frontend tokens, WS↔OSC bridge (browser OSC backend → `manifold/osc-bridge/` Deno relay → the module's OSC server), bidirectional training, and the `nisps-core`→`nisps/` core-path repoint. The existing `vcv/` module (2→12) is evolved, not rebuilt. **Still TODO** (clear in-code): the VCV↔browser bridge browser-side wiring to the new `/nisps/feedback` verb; the particle backend is a no-op passthrough pending the `visualizer.js` faithful port; CV/gate transport.