docs: the specs disposition pass (plan §8)
Roughly 20k lines were deleted from this repo in the last week and much of the
corpus still described the pre-deletion world in the present tense. Executes
the §8 table: archive the retired, reclassify the executed, prune the stale.
aimmersive-clone-spec -> _archive/ with a deprecated-by note
feedback-modes-port-spec -> plans/, kind: plan, status: executed
manifold-parity-features -> plans/, kind: plan, status: active
playground-2.0-rewrite -> status: superseded
engine-architecture 434 -> ~120 lines; seam + spine kept, rewritten
present-tense against the shipped engine/
MAIN.md six contradicted claims fixed; registry resynced
vcv-module.md pruned to the current 8->16 contract and made the
single .nisps format spec
vcv/NISPS-FORMAT.md DELETED — documented a v1 format that no longer loads
vcv/README.md, BUILDING.md rewritten to the real contract, menu, OSC table
inputs/backends/dock trio grounding sections marked historical, dead cites fixed
Two rows of the §8 table were themselves wrong, corrected here: the deleted
full-state sync lives in backends-spec.md §6.3, not vcv-module.md (which has no
§6.3), and codegen/README.md was already a MAP pointer with no port-solidjs
trigger left to remove.
Beyond the table — found by sweeping every backticked path in the changed docs
against `git ls-files`, which is how these should have been caught before:
manifold/ONBOARDING.md documented a UI that Phase 1 deleted, as if current:
SplitStage, ReadoutStrip, InputMini, BackendAdvanced, AltitudeNav, and a
shot.spec.ts that does not exist. The whole stage table was keyed on a `focus`
axis that no longer exists — selection is now sandwich > particles >
composite. This matters more than the rest: CLAUDE.md tells every agent to
read ONBOARDING.md first for Manifold work, so it was actively teaching a
fiction. Rewritten against ConsoleApp.tsx.
MAP.md claimed the input layer reduces axes to the engine arity with an
"even/odd blend". input-layer.ts says the opposite in its own header: one
dedicated slot per axis, 1:1, into a 32-input over-provisioned head, and
mean-blending was removed deliberately because it diluted every source.
AGENT-REFERENCE.md still promised TS emission "returns at P5" (landed),
per-mode dims "become schema-real at P5" (landed at P5.3), and pointed at
nisps::FixedBuffer (deleted).
Doc-right/code-suspect, filed rather than fixed: VCV computes derivedMean/Std/
Delta and cachedNovelty behind a live context-menu toggle that nothing reads;
vcv/plugin.json points at the MusicallyEmbodiedML org rather than this repo's
origin; and the module defaults to UDP 7001+id%64 while bridge.ts defaults to
9000, so out of the box they do not meet.
Firmware-build docs are deliberately untouched — the PlatformIO migration
lands next and rewrites all of them.
This commit is contained in:
monkey-w1n5t0n
parent
b16f26e6ab
commit
9ad1f78ddd
18 changed files with 642 additions and 1371 deletions
9
MAP.md
9
MAP.md
|
|
@ -65,8 +65,13 @@ anchor + locked decisions) and the `docs/specs/*-spec.md` set.
|
|||
config (`dock/OutputsBackendConfig.tsx`) reads it for the device picker + param-select that fills the CC table.
|
||||
- `manifold/src/inputs/` — modular INPUT layer feeding the ML head. The Inputs dock picks ONE exclusive mode
|
||||
(`InputMode` = `internal` | `gamepad` | `midi`; Internal/XY-pad is default). `input-layer.ts` owns a single rAF
|
||||
loop composing the active source's axes → reduced to the engine arity (fixed 2-in WASM → even/odd blend) → one
|
||||
`setInputs`, plus an `onReducedInput` callback the manifold tracks. Sources: `xy-pad-source` (push-driven),
|
||||
loop composing the active source's axes → **one dedicated engine input slot per axis, 1:1, no blending** → one
|
||||
`setInputs`, plus an `onReducedInput` callback the manifold tracks. The WASM net is over-provisioned to a
|
||||
32-input head (`MAX_AXES`, `nisps/wasm/bindings.cpp`); unused slots are zero-padded and a zero input is inert,
|
||||
so idle sources cannot perturb the net. Mean-blending was removed deliberately — it diluted every source and
|
||||
biased the net toward idle sources' resting values. Changing the ACTIVE axis count offers a reshape
|
||||
(`ConsoleApp` → `ReshapeModal`): new net at the new arity, warm-started from overlapping weights, examples and
|
||||
feedback state reset; declining keeps the over-provisioned head. Sources: `xy-pad-source` (push-driven),
|
||||
`gamepad-source` (sticks→axes single/double; buttons emit press+release actions, bound in `ConsoleApp` to
|
||||
verdicts — LB/RB=down/up, X/Y/B=randomise/nudge/undo, A-hold=reposition), `midi-input-source` (device picker +
|
||||
BATCH "MIDI Learn": every CC swept while armed becomes an axis, shown as read-only meters). `useInputLayer.ts`
|
||||
|
|
|
|||
|
|
@ -12,7 +12,7 @@ MEMLNaut-NISPS — Neural Interactive Shaping of Parameter Spaces. A research pl
|
|||
1. **RP2350 firmware** for the MEMLNaut hardware platform (`firmware/`).
|
||||
2. **WASM** in the Manifold React browser app (`manifold/`) — same engines + ML, run through an AudioWorklet.
|
||||
|
||||
(The former SolidJS playground was retired 2026-07-13 at P1 of `docs/specs/plans/one-core-engine-refactor.md`; archived on branch `archive/playground-solidjs`, tag `playground-solidjs-final`. The browser-only C15 engine currently lives only there.) Parameter contracts are JSON schemas (`schemas/`) with codegen producing the C++ headers (TS emission returns at P5).
|
||||
(The former SolidJS playground was retired 2026-07-13 at P1 of `docs/specs/plans/one-core-engine-refactor.md`; archived on branch `archive/playground-solidjs`, tag `playground-solidjs-final`. The browser-only C15 engine currently lives only there.) Parameter contracts are JSON schemas (`schemas/`) with codegen producing the C++ headers AND the TypeScript modules (both live since P5; CI fails if either is stale).
|
||||
|
||||
Project documentation: https://musicallyembodiedml.github.io/memlnaut/approaches/nisps
|
||||
|
||||
|
|
@ -43,7 +43,7 @@ Tests: 4 executables (`nisps_core_tests`, `nisps_dsp_engine_tests`, `nisps_modes
|
|||
|
||||
These rules apply to **all** code under `nisps/`. They are inert in WASM but kept globally for consistency.
|
||||
|
||||
- **No heap.** No `new`, `malloc`, `std::vector` in hot paths. Use `nisps::FixedBuffer<T, N>` or `std::array<T, N>`.
|
||||
- **No heap.** No `new`, `malloc`, `std::vector` in hot paths. Use `std::array<T, N>`. (`nisps::FixedBuffer` is gone — deleted in the Phase 1 sweep; `std::array` was doing the same job.)
|
||||
- **Constants discipline.** Float literals >255 used in hot paths must be `static const float val = X.f;` not inline.
|
||||
- **`.f` suffix on all float literals.** No double promotion in audio/inference paths.
|
||||
- **Hot-path attributes.** Apply `NISPS_HOT` / `NISPS_FORCE_INLINE` (from `nisps/core/perf.hpp`). The SRAM-section macros were deleted in the 2026-07 sweep — they had no real use sites.
|
||||
|
|
@ -113,13 +113,13 @@ Two WASM instances at runtime:
|
|||
|
||||
C API is in `nisps/wasm/bindings.cpp`. Build: `bash scripts/build-wasm.sh` (~94KB output to `manifold/public/`).
|
||||
|
||||
The browser MLP is runtime-shaped since P2 (`MLPCore<DynamicStorage>`): `nisps_ml_create` honours `(input, output, hidden[3])`; non-positive/null args default to `32→[10,14,18]→126`. `nisps_ml_reshape` swaps in a new shape warm-started from the overlapping weights (examples + feedback state reset). Modes currently still slice the default 126 outputs; per-mode dims become schema-real at P5.
|
||||
The browser MLP is runtime-shaped since P2 (`MLPCore<DynamicStorage>`): `nisps_ml_create` honours `(input, output, hidden[3])`; non-positive/null args default to `32→[10,14,18]→126`. `nisps_ml_reshape` swaps in a new shape warm-started from the overlapping weights (examples + feedback state reset). Per-mode dims have been schema-real since P5.3 on both targets — modes no longer slice a shared 126-wide default.
|
||||
|
||||
### Known limitations
|
||||
|
||||
- Loss history not yet plumbed through C API; only the final loss of a training run reaches TS.
|
||||
- Mic input through the worklet for XIASRI / SoundAnalysisMIDI is not wired in manifold.
|
||||
- C15 has no home on main (see `ALIGNMENT.md` defect #1).
|
||||
- C15 has no home on main (see `ALIGNMENT.md` defect 1, browser mode coverage).
|
||||
- (P3, 2026-07-14) The browser Jolt/OU gestures and the geometric dislike run the C++ core through WASM: `nisps_ml_jolt_*`, `nisps_ml_explore_*`, `nisps_ml_feedback_dislike_geometric` — no TS gesture math remains.
|
||||
|
||||
## URL parameters (manifold)
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
# RL / Feedback Learning-Engine Design — Manifold
|
||||
|
||||
*Design sign-off. Dated 2026-06-27. Supersedes the AVOID-out-of-scope decision in `docs/specs/feedback-modes-port-spec.md` §2.5/§7. Author: lead architect, synthesising a 4-candidate / 4-judge design panel against verified source.*
|
||||
*Design sign-off. Dated 2026-06-27. Supersedes the AVOID-out-of-scope decision in `docs/specs/plans/feedback-modes-port-spec.md` §2.5/§7 (reclassified `kind: plan, status: executed` 2026-07-21). Author: lead architect, synthesising a 4-candidate / 4-judge design panel against verified source.*
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
|
|
@ -19,6 +19,8 @@ This corpus distinguishes four document kinds by temporal stance and mutability:
|
|||
- **`kind: finding`** (in `docs/specs/recon/`) — past-tense, dated, **IMMUTABLE**. Never edited, only superseded; exempt from drift lint. Research artifacts capturing ground truth at a specific moment.
|
||||
- **ADRs** (in `docs/adr/`) — decision rationale ("why we chose X"). Outside the spec root; used as backlinks from specs that encode those decisions.
|
||||
|
||||
Archived documents live in `docs/specs/_archive/` with `deprecated-by`/`superseded-by` front matter.
|
||||
|
||||
---
|
||||
|
||||
## Source Files (Architectural Roots)
|
||||
|
|
@ -27,7 +29,7 @@ The codebase's most architecturally significant areas referenced by this corpus:
|
|||
|
||||
| Layer | Path | Purpose |
|
||||
|-------|------|---------|
|
||||
| **Shared C++20** | `nisps/core/`, `nisps/ml/`, `nisps/dsp/`, `nisps/engines/`, `nisps/modes/` | One audio+ML library compiling to both firmware and WASM |
|
||||
| **Shared C++20** | `nisps/core/`, `nisps/ml/`, `nisps/dsp/`, `nisps/engines/`, `nisps/modes/`, `nisps/pipeline/` | One audio+ML library compiling to both firmware and WASM |
|
||||
| **Schema/Codegen** | `schemas/`, `codegen/` | Parameter contracts + code generation (C++ headers, TS types) |
|
||||
| **Firmware** | `firmware/MEMLNaut-NISPS/glue/` | Hardware bindings (audio, peripherals, MIDI, settings) |
|
||||
| **Browser (Manifold)** | `manifold/src/engine/`, `manifold/src/console/` | Headless TS engine + AudioWorklet + the convertible React Console |
|
||||
|
|
@ -38,18 +40,18 @@ The codebase's most architecturally significant areas referenced by this corpus:
|
|||
|
||||
**What it is.** MEMLNaut-NISPS is a single C++20 codebase that compiles once to two distinct targets:
|
||||
|
||||
1. **Firmware** — runs on RP2350 hardware (MEMLNaut instrument). Real-time audio engines (8 variants) + interactive ML (4-layer MLP, SGD + RL feedback) with deterministic RNG + dual-core orchestration (audio on core 1, control on core 0).
|
||||
2. **Browser (WASM)** — runs in the React Manifold front-end via AudioWorklet. Same C++20 engines + ML, compiled to WASM; audio engines are a superset of firmware (C15 synth browser-only).
|
||||
1. **Firmware** — runs on RP2350 hardware (MEMLNaut instrument). Real-time audio engines + interactive ML (4-layer MLP, SGD + RL feedback) with deterministic RNG + dual-core orchestration (audio on core 1, control on core 0).
|
||||
2. **Browser (WASM)** — runs in the React Manifold front-end via AudioWorklet. The **same eight `nisps/` engines** as firmware — neither target is a superset. (The browser-only C15 synth lives only on the retired-playground archive branch; browser audio-*input* — mic — is not wired, so the analysis-driven modes are hardware-only today.)
|
||||
|
||||
**The unifying constraint**: one source tree, one ML architecture, cross-platform parity (native ↔ WASM within 1e-5 numerical tolerance). Parameter shapes are JSON schemas with codegen producing both C++ headers and TypeScript types.
|
||||
|
||||
**Performance contract** (applies globally to `nisps/`, inert in WASM but kept for consistency):
|
||||
**Performance contract** (applies to `nisps/`, inert in WASM but kept for consistency):
|
||||
|
||||
- No heap in audio/inference paths; use `FixedBuffer` or `std::array`.
|
||||
- Float literals >255 in hot paths must be `static const`; all floats have `.f` suffix.
|
||||
- No virtual dispatch in audio path (C++20 concepts, not interfaces).
|
||||
- No heap in audio/inference paths; use `std::array` (the one deliberate exception: `nisps/ml/dynamic_storage.hpp`, browser/VCV-only, `#error`s on RP2350 builds, sole lint allowlist entry).
|
||||
- Float literals have the `.f` suffix; non-trivial float constants are `static const`.
|
||||
- No virtual dispatch in the audio path (C++20 concepts, not interfaces).
|
||||
- Deterministic per-instance RNG; all random state is local and seeded at construction.
|
||||
- Memory section attributes per hardware (AUDIO_MEM, AUDIO_FUNC, HOT, FORCE_INLINE).
|
||||
- Hot-path attributes from `nisps/core/perf.hpp`: `NISPS_HOT` / `NISPS_FORCE_INLINE` (the former SRAM-section macro regime was deleted 2026-07 as dead).
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -61,35 +63,34 @@ The codebase's most architecturally significant areas referenced by this corpus:
|
|||
|
||||
### Schema/codegen contract
|
||||
|
||||
Each firmware mode has a `schemas/modes/<mode>.json` describing parameters (name, label, range, default, curve, group), ML config (input/output sizes, hidden layers), voice spaces (names; bodies are inline lambdas in the engine), and UI config. The meta-schema at `schemas/schema.json` validates these. Codegen (`bun run codegen/generate.ts`) is idempotent and produces:
|
||||
Each mode has a `schemas/modes/<mode>.json` describing parameters (name, label, range, default, curve, group), ML config (input/output sizes, hidden layers), voice spaces (names; bodies are inline lambdas in the engine), and UI config. The meta-schema at `schemas/schema.json` validates these. Codegen (`bun run codegen/generate.ts`) is idempotent and produces **both language outputs**:
|
||||
|
||||
- `nisps/modes/generated/<mode>_schema.hpp` — `constexpr` C++ data under `nisps::modes::generated`.
|
||||
- TS schema emission is dormant since P1; it returns at P5 targeting `manifold/src/modes/generated/`.
|
||||
- `nisps/modes/generated/<mode>_schema.hpp` — `constexpr` C++ data under `nisps::modes::generated` (+ `nisps/ml/generated/ml_defaults.hpp` from `schemas/ml_defaults.json`).
|
||||
- `manifold/src/modes/generated/<mode>_schema.ts` (+ `types.ts`, `index.ts`) — the source of truth for the Manifold `MF_MODES` catalogue.
|
||||
|
||||
Regenerate after editing any `schemas/modes/*.json`. Golden test ensures output is byte-identical.
|
||||
A second generator, `codegen/generate-midi-devices.ts`, emits `nisps/midi/generated/midi_devices.hpp` + `manifold/src/midi-devices/generated/` from `schemas/midi_devices/`.
|
||||
|
||||
Schema changes ship with the regenerated C++ **and** TypeScript in the same commit; CI re-runs both generators and fails on any diff (plus the golden test).
|
||||
|
||||
### No-heap + deterministic RNG
|
||||
|
||||
These are inert in WASM but kept globally to avoid surprises when porting. Lint catches violations: `bash scripts/lint-cpp.sh` warns on missing `.f` suffix and fails on `new`/`malloc`/`std::vector` under `nisps/`, plus `Arduino.h` use outside firmware glue.
|
||||
These are inert in WASM but kept globally to avoid surprises when porting. Lint catches violations: `bash scripts/lint-cpp.sh` warns on missing `.f` suffix and fails on heap use under `nisps/` (allowlist: `dynamic_storage.hpp`), plus `Arduino.h` use outside firmware glue.
|
||||
|
||||
---
|
||||
|
||||
## Sub-specs (Behavioural + Binding)
|
||||
|
||||
All specs below are stable or evolving (no aspirational ones in shipped features); aspirational specs describe target contracts not yet fully implemented.
|
||||
|
||||
| File | Stability | Layer | Summary |
|
||||
|------|-----------|-------|---------|
|
||||
| `aimmersive-clone-spec.md` | evolving | behavioural | Faithful SolidJS rebuild of the deployed vanilla a-immersive app (446-line DOM, 2538-line CSS, 4521-line JS). Component-by-component spec for replicating look/behaviour. |
|
||||
| `backends-spec.md` | evolving | binding | Output backends (synth, MIDI, OSC, VCV) unified behind one `OutputBackend` interface + registry. One active backend per session; input/ML/output pipelines backend-agnostic. |
|
||||
| `dock-spec.md` | evolving | behavioural | Console right-dock drawers (Shape/Feel/Route/Health/Help) + per-output controls (off/fixed/live tri-state, mute, solo, curve). Three-depth dock model. |
|
||||
| `engine-architecture.md` | evolving | cross-cutting | Foundation: headless `EngineApi` boundary separating the pure engine (input→ML→output reactive spine) from presentation skins (a-immersive, Console). No JSX/DOM in engine; skins are pure consumers of `EngineApi` accessors. |
|
||||
| `feedback-modes-port-spec.md` | evolving | cross-cutting | Implementation spec: "Down Action" negative-feedback feature ported from firmware `InterfaceRL` into `nisps/` core. Three modes (Avoid / RandomiseOutputs / RandomiseMlp), deterministic RNG, per-instance state. |
|
||||
| `inputs-spec.md` | evolving | binding | Modular input layer (sources: XY pad, MIDI input, gamepad single/double-stick) composing into N-dimensional vector. MLP rebuild on input-set change. Scope: `manifold/` React app. |
|
||||
| `manifold-parity-features-spec.md` | aspirational | behavioural | Prescriptive spec for porting five playground features into Manifold: session presets, pins, Jolt, OU-Explore, control surface. Awaiting review; no implementation authorised. |
|
||||
| `backends-spec.md` | evolving | binding | Output backends (synth, particles, MIDI, OSC, CV/gate, VCV) unified behind one `OutputBackend` interface + manager. Largely built (`manifold/src/backends/`); carries a 2026-07 grounding note — the full-state OSC sync legs it designed were deleted. |
|
||||
| `dock-spec.md` | evolving | behavioural | Console dock drawers + the per-output control row (off/fixed/live tri-state, mute, solo/arm, min/max/curve). Built with operator restructuring (top Mode selector + 5 drawers, 2 depths); grounding note maps spec→shipped. |
|
||||
| `engine-architecture.md` | stable | cross-cutting | The surviving browser-engine contract: the headless `EngineApi` seam + the reactive-spine invariant, as realised in `manifold/src/engine/`. Trimmed 2026-07; superseded framing documented in its header. |
|
||||
| `inputs-spec.md` | evolving | binding | Modular input layer (XY pad, MIDI input, gamepad single/double-stick) composing into an N-dim vector. Built as `manifold/src/inputs/` (runtime-reshaped net, exclusive-mode UI; mixing groundwork laid — operator decision §7.7 pending). |
|
||||
| `slp-workshop-firmware.md` | evolving | binding | SLP-Workshop (Synth Library Portland workshop build). Part I (Jolt + OU explore adaptive-learning gestures) shipped & stable. Part II (output-mode evolution, gate sequences, Manifold config) planned/evolving. |
|
||||
| `useq-cv-protocol.md` | stable | binding | uSEQ-CV wire protocol v2 (USB Web Serial ↔ uSEQ main module → CV/gate jacks + I2C expander). Single source of truth: `firmware/useq-celium/shared/protocol.h`, mirrored in `manifold/src/backends/useq-protocol.ts`. |
|
||||
| `vcv-module.md` | evolving | binding | MEMLNaut VCV Rack 2 module: CV-to-CV mapper (8 inputs × 16 outputs) with RL feedback. Embeds the WASM engine; WS↔OSC browser bridge. 2026-06-28 BUILD DELTAS folded in; prior design sections retained for reference. |
|
||||
| `vcv-module.md` | evolving | binding | MEMLNaut VCV Rack 2 module: CV-to-CV mapper (8 in × 16 out, LED rings) on the shared `nisps/` core (P6). Also the single `.nisps`/patch format spec (v3 flat weights). Pruned 2026-07-21 to the current contract. |
|
||||
|
||||
**Archived** (in `_archive/`): `aimmersive-clone-spec.md` — the SolidJS clone spec for the deployed vanilla a-immersive app; the clone was never built and its target framework was retired. Kept as a behavioural reference for the still-deployed vanilla app.
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -97,16 +98,18 @@ All specs below are stable or evolving (no aspirational ones in shipped features
|
|||
|
||||
| File | Status | Summary |
|
||||
|------|--------|---------|
|
||||
| `plans/BUILD-PLAN.md` | active | Manifold build resume anchor (dated 2026-06-27). Locked decisions: React app in `manifold/`, parity-tested TS engine from `playground/src`, staging deploy at `meml.lnfinitemonkeys.org/next`, default feedback mode = Explore-and-Place. |
|
||||
| `plans/playground-2.0-rewrite-plan.md` | executed | SolidJS clean-room rewrite plan (June 2026). Largely implemented in Manifold + playground foundation: one fullscreen instrument, Console interaction model, right-edge dock with three depths, snapshot DAG, control-point tri-state. |
|
||||
| `plans/one-core-engine-refactor.md` | active | Firmware+Manifold core reunification (dated 2026-07-13). Locked: retire `playground/`, all algorithms into `nisps/` C++ (geometric dislike, Jolt, OU, RNG, pipelines, curves), storage-policy MLP (fixed template on RP2350, runtime-shaped in WASM/VCV), codegen serves manifold, VCV last. Supersedes BUILD-PLAN's "multiple WASM modules" MLP decision. |
|
||||
| `plans/simplification-plan.md` | active | Phased burn-down of the 2026-07-21 simplification audit (proposal; phases gated on operator adoption). Phase 0 restores CI/verification (memllib pin unreachable, deploys ungated); then dead-mass deletion, behaviour bugs, registry consolidation, PlatformIO migration, vision-facing builds. §7 lists the operator decisions. |
|
||||
| `plans/BUILD-PLAN.md` | executed | Manifold build resume anchor (2026-06-27). The app shipped to `/next/`; survivors (non-snap-node e2e invocation, naming/copy decisions) noted in its header; deploy is now the CI-gated webhook. |
|
||||
| `plans/feedback-modes-port-spec.md` | executed | The "Down Action" negative-feedback port into `nisps/`. Landed, then evolved past it — `docs/adr/rl-feedback-design.md` + `nisps/ml/feedback.hpp` are the surviving truth (the ADR explicitly supersedes its §2.5/§7 AVOID decision). |
|
||||
| `plans/manifold-parity-features-spec.md` | active | Five playground features for Manifold. Jolt + OU-explore: executed by other means (one-core P3 core bindings). Session presets, pins, control surface: still the live prescription, feeding the curated/advanced split (simplification-plan §6.5c). |
|
||||
| `plans/playground-2.0-rewrite-plan.md` | superseded | SolidJS clean-room rewrite plan (2026-06-17). Its target (the playground) was retired; many of its ideas shipped in Manifold instead. Still cited as design source by the parity-features plan. |
|
||||
| `plans/one-core-engine-refactor.md` | active | Firmware+Manifold core reunification (2026-07-13). **All six phases landed on main by 2026-07-18**; stays `active` only for the two hardware-verification chokepoints (its §6) before flipping to executed. |
|
||||
| `plans/simplification-plan.md` | active | Phased burn-down of the 2026-07-21 simplification audit. Phases 0 (CI/verification), 2 (behaviour bugs) and 3 (truth consolidation, minus S26) burned down 2026-07-21; Phase 1 deletions largely landed; PlatformIO (§5) in flight; §7 lists remaining operator decisions. |
|
||||
|
||||
---
|
||||
|
||||
## Findings (Immutable, Dated Research)
|
||||
|
||||
All findings are dated 2026-06-27 unless otherwise noted; exempt from drift lint.
|
||||
Exempt from drift lint.
|
||||
|
||||
| File | Date | Summary |
|
||||
|------|------|---------|
|
||||
|
|
@ -117,6 +120,7 @@ All findings are dated 2026-06-27 unless otherwise noted; exempt from drift lint
|
|||
| `recon/upstream-firmware-survey.md` | 2026-06-27 | Git archaeology of the MusicallyEmbodiedML ecosystem. origin/main (C++20/SolidJS rewrite) and upstream/main (old `.ino` firmware) forked at `6efbe9c` (2026-04-14); 49 commits upstream not in origin are ports, not merges. |
|
||||
| `recon/playground-2026.md` | 2026-04-12 | Design intent snapshot of an unfinished playground UI redesign. Reference-only for SolidJS rewrite; do NOT merge into vanilla playground (those files moved on independently). |
|
||||
| `recon/simplification-audit-2026-07.md` | 2026-07-21 | Full-repo smell/bloat/spec audit vs the five-bullet one-core vision (66-agent workflow, adversarially verified). 113 findings: CI red since 2026-07-13, memllib pin unpushed, ungated deploys, mode layer unshared, no curated/advanced split, dead-mass inventory. Mitigations in `plans/simplification-plan.md`. |
|
||||
| `recon/memllib-usage-inventory.md` | 2026-07-21 | Per-file inventory of the memllib surface the firmware actually compiles — the prerequisite for the §7.5 ownership decision. Result: all of memllib bar `examples/` is load-bearing (~1.8 MB / 84 files); no small subset to vendor. |
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -131,11 +135,10 @@ All findings are dated 2026-06-27 unless otherwise noted; exempt from drift lint
|
|||
## Cross-References
|
||||
|
||||
- **`MAP.md`** — the neutral inventory of the codebase as it stands (ground truth for what exists).
|
||||
- **`CLAUDE.md`** — long-form architecture narrative (entry point for agent onboarding).
|
||||
- **`CLAUDE.md`** — agent contract (gates, hard constraints); `docs/AGENT-REFERENCE.md` — the long-form reference.
|
||||
- **`ALIGNMENT.md`** — strategic gaps + open mission questions (dated, opinionated gap diagnosis).
|
||||
- **`manifold/ONBOARDING.md`** — agent-oriented breakdown of the Manifold React app (run/build/deploy/test, UI/engine-spine/WASM layering, convertible Stages, Dock+drawers, gotchas).
|
||||
- **`docs/redesign/manifold-export/`** — design-token and UI-kit asset export from Manifold design project (not part of this corpus; static asset reference, not migrated).
|
||||
- **`anima/`** — separate project (excluded from this corpus).
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
|
|
@ -1,12 +1,19 @@
|
|||
---
|
||||
kind: spec
|
||||
stability: evolving
|
||||
layer: behavioural
|
||||
deprecated-by: 2026-07-21
|
||||
superseded-by: MAIN.md
|
||||
---
|
||||
|
||||
I now have a thorough, complete understanding of the entire deployed app — DOM, CSS theme tokens, all interaction handlers, state model, engine call points, persistence, and the help-modal copy. I have everything needed to write the faithful clone spec. Let me write it now.
|
||||
# NISPS Immersive (`a-immersive.html`) — SolidJS Clone Spec (Archived)
|
||||
|
||||
# NISPS Immersive (`a-immersive.html`) — SolidJS Clone Spec
|
||||
**Note:** This specced a faithful SolidJS clone of the deployed vanilla a-immersive app — a
|
||||
product that was never built. The SolidJS playground target was retired 2026-07-13 (branch
|
||||
`archive/playground-solidjs`, tag `playground-solidjs-final`); the current browser app is the
|
||||
Manifold convertible Console (`manifold/`), covered by the live corpus (`../MAIN.md`). The
|
||||
vanilla a-immersive app it describes remains deployed at `meml.lnfinitemonkeys.org/`, so this
|
||||
survives as a behavioural reference for that app, frozen as of 2026-06-27. File paths cited
|
||||
below (e.g. `/home/w1n5t0n/deployments/meml-aimmersive/`) refer to that deployment, not this
|
||||
repo.
|
||||
|
||||
This is a faithful, component-by-component rebuild spec for the deployed vanilla-JS app at `/home/w1n5t0n/deployments/meml-aimmersive/`. Source of truth: `a-immersive.html` (446-line DOM), `css/a-immersive.css` (2538 lines), `js/a-app.js` (4521-line module), and supporting modules under `js/nisps/`, `js/ui/`, `js/synth/`, `js/midi/`, `js/audio/`. Goal: a from-scratch SolidJS reimplementation that matches the look and behaviour 1:1, binding to a shared engine layer (the WASM IML + synth engines) rather than re-porting it.
|
||||
|
||||
|
|
@ -2,12 +2,25 @@
|
|||
kind: spec
|
||||
stability: evolving
|
||||
layer: binding
|
||||
counterpart: aimmersive-clone-spec.md
|
||||
---
|
||||
|
||||
# Output Backends — Specification
|
||||
|
||||
*Workstream E. Design-only, read-only audit 2026-06-27. The new app is Vite + React + TS in `manifold/`, wired to the parity-tested TS engine lifted from `playground/src`. British spelling in product copy. The built-in synth is the **"Powerful Synth Engine"** — the string "C15" must never reach the user.*
|
||||
*Workstream E. Designed 2026-06-27 against the pre-refactor tree; largely built (see the
|
||||
Implementation status section at the end). British spelling in product copy. The built-in synth
|
||||
is the **"Powerful Synth Engine"** — the string "C15" must never reach the user.*
|
||||
|
||||
> **Grounding note (2026-07-21).** Treat every `file:line` cite in the body as historical
|
||||
> grounding, not a live pointer: `playground/*` died with the retired SolidJS playground (branch
|
||||
> `archive/playground-solidjs`); `aimmersive-clone-spec.md` is archived at
|
||||
> `_archive/aimmersive-clone-spec.md`; `/home/w1n5t0n/deployments/meml-aimmersive/*` cites refer
|
||||
> to the still-deployed vanilla app *outside this repo*; VCV line numbers pre-date the P6
|
||||
> reunification. The shipped backends live at `manifold/src/backends/` (not the proposed
|
||||
> `engine/backends/`); the shipped bridge protocol is **`/nisps/input` + `/nisps/output` +
|
||||
> `/nisps/feedback` only** — the `/nisps/state` / `/nisps/weights` / `/nisps/examples`
|
||||
> full-state-sync legs described below were **deleted 2026-07-21** (zero consumers on either
|
||||
> side; Rack patch save/load owns persistence). The current protocol contract is
|
||||
> `vcv-module.md` §Browser Bridge.
|
||||
|
||||
> **Naming guard (non-negotiable).** The codename `C15` survives only in internal module/file names that the user never sees (`c15-adapter.js`, `c15-bridge.js`). Every label, tooltip, dock entry, menu item, status string, and aria-label says **"Powerful Synth Engine"** (or just "Synth"). A lint allowlist + a Playwright assertion (`expect(page).not.toContainText('C15')`) enforce this across `manifold/` and the VCV panel SVG/strings.
|
||||
|
||||
|
|
@ -133,18 +146,20 @@ A `WebMidiBackend` wrapping the salvaged `midi-output.js` (Web MIDI API, `sendBa
|
|||
|
||||
An `OscBridgeBackend` that **salvages the existing OSC bridge** rather than reinventing it. Two pieces already exist and are good:
|
||||
|
||||
- **Browser client:** `deployments/meml-aimmersive/js/synth/osc-output.js` (param-named WS messages, 50 ms/0.002 dead-zone) and the richer `js/nisps/osc-client.js` (`NispsOscClient` — `EventTarget`, `sendState`/`sendWeights`/`sendParams`, `onOutputsReceived`/`onInputsReceived`, auto-reconnect with backoff). Lift `osc-client.js` as the transport (it already speaks the bridge protocol and is bidirectional).
|
||||
- **Bridge server:** `deployments/meml-aimmersive/osc-bridge/bridge.ts` — a Deno WebSocket↔UDP-OSC bridge, zero-dependency OSC encode/decode, bidirectional. Keep it as-is; it is the canonical transport between browser and any OSC target (VCV, SuperCollider).
|
||||
- **Browser client:** `deployments/meml-aimmersive/js/synth/osc-output.js` (param-named WS messages, 50 ms/0.002 dead-zone) and the richer `js/nisps/osc-client.js` (`NispsOscClient` — `EventTarget`, auto-reconnect with backoff). The shipped transport is `manifold/src/backends/osc-client.ts` (`sendParams` / `sendInput` / `sendFeedback` — the legacy `sendState`/`sendWeights` legs were deleted with the full-state sync).
|
||||
- **Bridge server:** the repo's canonical bridge is `manifold/osc-bridge/bridge.ts` — a Deno WebSocket↔UDP-OSC bridge, zero-dependency OSC encode/decode, bidirectional (WS verbs: `params`, `input`, `feedback`).
|
||||
|
||||
**OSC path + range contract (salvaged from `bridge.ts`):**
|
||||
**OSC path + range contract (the live set):**
|
||||
|
||||
| Direction | Address | Args | Meaning |
|
||||
|---|---|---|---|
|
||||
| browser→target | `/nisps/<param_name>` | `f` | one param, **post-baseline-mapping value** (see §3) |
|
||||
| browser→target | `/nisps/state` | `s` | full JSON state (weights + examples + config) |
|
||||
| browser→target | `/nisps/weights` | `s` | weights-only JSON |
|
||||
| browser→target | `/nisps/input` | `f…f` | input vector → drives model inputs (VCV bridged mode) |
|
||||
| browser→target | `/nisps/feedback` | `s` | verdict op JSON (`up` / `down` / `rand` / `clear`) |
|
||||
| target→browser | `/nisps/output` | `f…f` | output vector (visualisation / monitoring) |
|
||||
| target→browser | `/nisps/input` | `f…f` | input vector → drives model inputs |
|
||||
| target→browser | `/nisps/input` | `f…f` | input vector echo |
|
||||
|
||||
(The 2026-06 design also carried `/nisps/state` and `/nisps/weights` full-JSON sync — deleted, see the grounding note.)
|
||||
|
||||
**Ranges:** OSC floats are sent in the param's mapped range by default (`applyGroupOverride` applied before send, matching `osc-output.js`), with a per-backend toggle to send **raw normalised 0..1** instead (some OSC targets want 0..1 and do their own scaling). Address prefix (`/nisps`), target host/port (default `127.0.0.1:9000`), and listen port (default `9001`) are configurable — `bridge.ts` already exposes `--osc-host/--osc-port/--osc-prefix/--ws-port/--listen-port`.
|
||||
|
||||
|
|
@ -163,7 +178,7 @@ For browser-side CV/gate there is no native hardware path, so this backend has *
|
|||
|
||||
The headline new backend. A first-class **VCV Rack 2 module** (`MEMLNaut`) with **8 CV inputs → model → 16 CV outputs**, an **LED ring around each of the 16 outputs**, and a **browser↔VCV bridge** so the tool is controllable AND trainable from both inside Rack and entirely from the browser. Full design in §5–§7.
|
||||
|
||||
The browser-side adapter `VcvBridgeBackend` reuses `NispsOscClient` (§2.4) as transport. When active in **bridged mode**, the browser supplies inputs in real time (`/nisps/input`) and the verdict/example loop is mirrored over the bridge (`/nisps/state`, `/nisps/weights`, new `/nisps/feedback`).
|
||||
The browser-side adapter (shipped as `manifold/src/backends/vcv-backend.ts`) reuses the OSC client (§2.4) as transport. When active in **bridged mode**, the browser supplies inputs in real time (`/nisps/input`) and forwards verdicts to the module's embedded learner (`/nisps/feedback`).
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -355,26 +370,26 @@ Mapping rule: `ringColor = kRingPalette[paramMeta[i].group % paletteLen]`, so ou
|
|||
- **Threading model** (`MEMLNaut.cpp:67–312`): audio-thread `iml` + worker-thread `imlShadow`, job queue (Train/Perturb/Randomize/Clear), atomic weight hand-off, single-writer invariant. Already correct; just resize the I/O.
|
||||
- **Inference-rate decimation + slew + post-swap crossfade** (`:447–499`).
|
||||
- **Verdict loop** (`:412–445`): `+`/`−` buttons and `+TRIG`/`−TRIG` gated by LEARN. `+` → add example (current inputs→current outputs) + enqueue Train + decay noise ×0.97; `−` → bump noise (cap `0.3(1−s)+0.05s`) + enqueue Perturb. Identical semantics to the browser verdict loop.
|
||||
- **Serialization** (`dataToJson`/`dataFromJson`, `:548–696`) + **`.nisps` preset save/load** (`:862–924`) + **OSC server** (`osc_server.hpp`). The `.nisps` format is the shared interchange with the browser (`vcv/NISPS-FORMAT.md`).
|
||||
- **Serialization** (`dataToJson`/`dataFromJson`) + **`.nisps` preset save/load** + **OSC server** (`osc_server.hpp`). The `.nisps` v3 format is specified in `vcv-module.md` §State Persistence (the separate `vcv/NISPS-FORMAT.md` was deleted).
|
||||
- Per-output / per-input **range menu** (uni/bipolar, attenuverter) (`:818–843`).
|
||||
|
||||
### 5.5 Plugin scaffold files
|
||||
### 5.5 Plugin scaffold files (as they exist today)
|
||||
```
|
||||
vcv/
|
||||
├── plugin.json # slug MEMLNaut (exists) — bump version; tags Controller/Utility/Random
|
||||
├── Makefile # VCV SDK Makefile (exists); RACK_DIR-driven
|
||||
├── Makefile.dist # dist packaging (exists)
|
||||
├── plugin.json # slug MEMLNaut; tags Controller/Utility/Random
|
||||
├── Makefile # VCV SDK Makefile; RACK_DIR-driven (make dist via the SDK's plugin.mk)
|
||||
├── src/
|
||||
│ ├── plugin.{hpp,cpp} # plugin init / model registration (exist)
|
||||
│ ├── MEMLNaut.cpp # module — resize 8 in / 16 out (edit)
|
||||
│ ├── LedRing.hpp # NEW — LED-ring widget (§5.3)
|
||||
│ ├── palette.hpp # NEW — ring colour table from colors.css (§5.3)
|
||||
│ └── osc_server.hpp # OSC server (exists; extend with /nisps/feedback §7)
|
||||
├── res/
|
||||
│ ├── MEMLNaut.svg / -wide.svg / -expander.svg # panels (exist; redraw for 8/16 layout)
|
||||
├── test/smoke_test.cpp # exists
|
||||
├── plugin.json, SPEC.md, BUILDING.md, README.md, NISPS-FORMAT.md (exist)
|
||||
│ ├── plugin.{hpp,cpp} # plugin init / model registration
|
||||
│ ├── MEMLNaut.cpp # module — 8 in / 16 out
|
||||
│ ├── iml.hpp # thin adapter over the shared nisps/ core (P6)
|
||||
│ ├── LedRing.hpp # LED-ring widget (§5.3)
|
||||
│ ├── palette.hpp # ring colour table from the design tokens (§5.3)
|
||||
│ └── osc_server.hpp # OSC server (transport-only; /nisps/feedback per §7)
|
||||
├── res/ # MEMLNaut.svg / -wide.svg / -expander.svg
|
||||
├── BUILDING.md, DISTRIBUTION.md, README.md
|
||||
```
|
||||
(The 2026-06 draft also listed `Makefile.dist`, `test/smoke_test.cpp`, `SPEC.md` and
|
||||
`NISPS-FORMAT.md` — all since deleted or absorbed; the module spec is `vcv-module.md`.)
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -387,7 +402,7 @@ The existing salvage path is **WebSocket↔UDP-OSC** (`osc-bridge/bridge.ts` + `
|
|||
|
||||
| Option | How | Pros | Cons |
|
||||
|---|---|---|---|
|
||||
| **A. WS↔OSC bridge server (recommended)** | Browser ⇄ `bridge.ts` (Deno WS server, localhost:8765) ⇄ UDP OSC ⇄ module's `osc_server.hpp` (port 9000/9001) | Already built + bidirectional; standard OSC; works with SuperCollider/Max too; no browser perms | Needs a helper process running locally (Deno or the compiled `bridge.mjs`) |
|
||||
| **A. WS↔OSC bridge server (recommended)** | Browser ⇄ `bridge.ts` (Deno WS server, localhost:8765) ⇄ UDP OSC ⇄ module's `osc_server.hpp` | Already built + bidirectional; standard OSC; works with SuperCollider/Max too; no browser perms | Needs a helper process running locally (Deno, or a compiled `nisps-osc-bridge` binary from `compile.sh`) |
|
||||
| **B. Direct WebMIDI** | Browser ⇄ Web MIDI ⇄ a virtual MIDI port ⇄ a tiny MIDI-in path in the module | No helper process if a virtual MIDI port exists; browser-native | 7-bit/14-bit only — too coarse for weights/state; really only for live CC; module would need MIDI parsing |
|
||||
| **C. Native (module hosts a WS server)** | The VCV module itself opens a WebSocket/HTTP server; browser connects directly | No external bridge process | Adds a WS/TLS stack inside the plugin; COOP/COEP + mixed-content (`https://` page → `ws://localhost`) friction; more attack surface in the audio plugin |
|
||||
|
||||
|
|
@ -395,37 +410,35 @@ The existing salvage path is **WebSocket↔UDP-OSC** (`osc-bridge/bridge.ts` + `
|
|||
|
||||
### 6.2 Two modes
|
||||
- **Standalone:** module runs entirely inside Rack (CV in → model → CV out; verdict via panel buttons/triggers). No browser. Works today.
|
||||
- **Bridged:** browser connects via the bridge. In bridged mode **the browser supplies inputs in real time** — the Manifold pointer / joystick / pads stream `/nisps/input <f…f>` to the module, which uses them instead of (or blended with, configurable) the physical CV-in jacks. The module streams `/nisps/output` and `/nisps/input` back at ~100 ms (`MEMLNaut.cpp:536`) for browser visualisation. State/weights sync both ways via `/nisps/state` + `/nisps/weights` (already staged atomically into the audio thread, `:360–384`).
|
||||
- **Bridged:** browser connects via the bridge. In bridged mode **the browser supplies inputs in real time** — the Manifold pointer / joystick / pads stream `/nisps/input <f…f>` to the module, which uses them instead of the physical CV-in jacks. The module streams `/nisps/output` and `/nisps/input` back at ~100 ms for browser visualisation.
|
||||
|
||||
### 6.3 Coherence model
|
||||
One model, one owner of weights at a time. The bridge sends **whole-model snapshots** (`/nisps/state`, `/nisps/weights`) on any structural change (train completes, randomize, clear, load preset), and **continuous I/O vectors** (`/nisps/input`, `/nisps/output`) for live feel. Last-writer-wins on weights with a short "training in progress" lock (the module already coalesces jobs at queue depth 1). The browser's `WasmIML` and the module's `nisps::IML` use the **same `.nisps` weight layout** (`vcv/NISPS-FORMAT.md`), so a snapshot from either side loads losslessly — *provided the architectures match*. **Caveat / open choice:** the browser engine is fixed `MLP<2,…,126>` while the module is `8→{16,24,16}→16`. For true weight transfer the bridged session must run a **matched architecture** (e.g. a browser mode configured to 8-in/16-out, within the modular envelope) — otherwise the bridge degrades to I/O + example transfer only (no raw-weight sync). Flag this explicitly.
|
||||
**One model — the module's.** In bridged mode the browser acts as a remote controller and
|
||||
trainer for the module's embedded learner: it streams inputs and forwards verdicts
|
||||
(`/nisps/feedback`), and reads the module's live I/O back. There is **no weight or state sync**
|
||||
between the browser's own WASM net and the module — the 2026-06 whole-model-snapshot design
|
||||
(`/nisps/state` / `/nisps/weights`, last-writer-wins, matched-architecture weight transfer) was
|
||||
deleted 2026-07-21 with zero consumers on either side. Persistence belongs to the Rack patch and
|
||||
`.nisps` files (`vcv-module.md`).
|
||||
|
||||
---
|
||||
|
||||
## 7. Training over the bridge — both directions
|
||||
|
||||
The verdict loop (place examples / thumbs-up / thumbs-down / undo) must work from **either** end and stay coherent. Add one OSC address and a small `RemoteTrainingBridge`:
|
||||
The verdict loop works from **either** end, on the module's model. The shipped surface:
|
||||
|
||||
```ts
|
||||
// manifold/src/engine/backends/backend.ts
|
||||
export interface RemoteTrainingBridge {
|
||||
thumbsUp(): void; // add current (input,output) example + train
|
||||
thumbsDown(): void; // perturb weights (explore)
|
||||
addExample(input: Float32Array, output: Float32Array): void;
|
||||
randomise(spread: number): void;
|
||||
clear(): void;
|
||||
undo(): void;
|
||||
onState(cb: (s: NispsState) => void): () => void; // remote → local sync
|
||||
}
|
||||
```
|
||||
- OSC verb `/nisps/feedback <s>` carrying a JSON op — **`up` / `down` / `rand` / `clear`**
|
||||
(optional `spread`, `input`/`output` vectors). The module's `osc_server.hpp` `onFeedback`
|
||||
callback stages the op atomically for the audio thread, which routes it through the **same**
|
||||
job/`add_example` paths the panel buttons use.
|
||||
- **Browser → VCV training:** thumbs-up in the Manifold → `vcv-backend.ts` `sendFeedback('up')`
|
||||
→ the module's worker trains its shadow instance → atomic weight swap. The browser observes
|
||||
the result through the live `/nisps/output` stream (not through weight sync — §6.3).
|
||||
- **VCV panel training** works exactly as in standalone mode; the browser sees the new mapping
|
||||
through the output stream.
|
||||
|
||||
New OSC verb: `/nisps/feedback <s>` carrying `{"op":"up|down|rand|clear|undo","spread":f,"input":[…],"output":[…]}`. The module's `osc_server.hpp` gets an `onFeedback` callback (mirroring the existing `onState`/`onWeights` at `MEMLNaut.cpp:120–132`) that stages the op atomically for the audio thread, which routes it through the **same** `enqueueJob`/`add_example` path the panel buttons use (`:412–445`). So:
|
||||
|
||||
- **Browser → VCV training:** user clicks thumbs-up in the Manifold → `VcvBridgeBackend.remote.thumbsUp()` → `/nisps/feedback {op:up,input,output}` → module stages → worker trains `imlShadow` → atomic weight swap → module streams `/nisps/state` back → browser `WasmIML.setWeights` updates so the UI/heatmap reflect the new mapping. The browser need not run its own training in bridged mode (or runs it and pushes weights; configurable — see §6.3 caveat).
|
||||
- **VCV → browser training:** user presses `+`/`−` on the panel (or sends `+TRIG`) → module trains/perturbs → streams `/nisps/state` → browser applies, so the verdict placed in Rack appears in the browser's example list and weight-health view.
|
||||
- **Example placing over the bridge:** either side can `addExample`; examples ride in `/nisps/state` (the module already serializes `examples.features`/`examples.labels`, `:587–604`) so the dataset stays in sync. Undo is local-history on each side, but a remote undo can be sent as `/nisps/feedback {op:undo}` to roll the module's last job (module keeps a one-deep snapshot, matching the browser undo stack semantics).
|
||||
|
||||
**Result:** the tool is fully controllable and trainable from inside VCV Rack and entirely from the browser, with the verdict loop and example-placing working either way over the same bridge.
|
||||
The 2026-06 design's richer `RemoteTrainingBridge` (remote `addExample`, `undo`, `onState`
|
||||
callbacks) was never built; its state-sync legs are deleted (§6.3).
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -436,11 +449,13 @@ Backends live under `manifold/src/engine/backends/`; lifted JS (`c15-*`, `param-
|
|||
|
||||
### OSC bridge server
|
||||
```bash
|
||||
cd osc-bridge
|
||||
deno run --allow-net bridge.ts # or the compiled bridge.mjs
|
||||
cd manifold/osc-bridge
|
||||
deno run --allow-net --unstable-net bridge.ts
|
||||
# --osc-host 127.0.0.1 --osc-port 9000 --ws-port 8765 --listen-port 9001
|
||||
```
|
||||
Ship `bridge.mjs` (already compiled) + `compile.sh` so users without Deno can run it via Node. Surface "bridge not running" in the OSC/VCV backend status (the client auto-reconnects).
|
||||
For users without Deno, `compile.sh` cross-compiles standalone `nisps-osc-bridge-<platform>`
|
||||
binaries into `dist/` via `deno compile`. Surface "bridge not running" in the OSC/VCV backend
|
||||
status (the client auto-reconnects).
|
||||
|
||||
### VCV module
|
||||
```bash
|
||||
|
|
@ -448,9 +463,12 @@ cd vcv
|
|||
export RACK_DIR=/path/to/Rack-SDK # VCV Rack 2 SDK
|
||||
make # builds plugin.so/.dylib/.dll
|
||||
make install # copies into the VCV user plugins dir
|
||||
# distribution: make dist (per Makefile.dist; produces the .vcvplugin)
|
||||
# distribution: make dist (SDK plugin.mk; produces the .vcvplugin — see vcv/DISTRIBUTION.md)
|
||||
```
|
||||
Per the search, the standard flow is `export RACK_DIR=…; make clean; make dist` ([Plugin Development Tutorial](https://vcvrack.com/manual/PluginDevelopmentTutorial)). Requires the VCV Rack 2 SDK; nisps-core is header-only C++20 (symlinked under `vcv/dep/` per `SPEC.md`). Ship v2-only (rationale in `SPEC.md`'s v1-compat section). License caveat: VCV SDK is GPLv3, nisps-core is MPL-2.0 — combined binary is effectively GPL; not submitting to the VCV Library initially (`SPEC.md` §License).
|
||||
Requires the VCV Rack 2 SDK; the shared `nisps/` core is header-only C++20, reached via relative
|
||||
includes from `vcv/src/` (see `vcv/BUILDING.md`). Ship v2-only (rationale in `vcv-module.md`).
|
||||
License caveat: VCV SDK is GPLv3 — the combined binary is effectively GPL; not submitting to the
|
||||
VCV Library initially.
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -472,9 +490,9 @@ Per the search, the standard flow is `export RACK_DIR=…; make clean; make dist
|
|||
- MIDI: `/home/w1n5t0n/deployments/meml-aimmersive/js/midi/midi-output.js` (`:114` batch throttle), `…/midi/midi-cc-map.js`
|
||||
- OSC client: `/home/w1n5t0n/deployments/meml-aimmersive/js/nisps/osc-client.js`; param-named client `…/js/synth/osc-output.js` (`:97`)
|
||||
- OSC bridge server: `/home/w1n5t0n/deployments/meml-aimmersive/osc-bridge/bridge.ts` (addresses `:298–305`, encode/decode `:70–182`), `bridge.mjs`, `compile.sh`
|
||||
- VCV module: `/home/w1n5t0n/src/MEMLNaut-NISPS/vcv/src/MEMLNaut.cpp` (threading `:67–312`, verdict `:412–445`, ranges `:818–843`, display `:699`, LED `:804`), `…/vcv/src/osc_server.hpp`, `…/vcv/plugin.json`, `…/docs/specs/vcv-module.md`, `…/vcv/Makefile`, `…/vcv/NISPS-FORMAT.md`, `…/vcv/res/*.svg`
|
||||
- VCV module: `/home/w1n5t0n/src/MEMLNaut-NISPS/vcv/src/MEMLNaut.cpp` (line numbers are pre-P6 grounding), `…/vcv/src/osc_server.hpp`, `…/vcv/plugin.json`, `…/docs/specs/vcv-module.md`, `…/vcv/Makefile`, `…/vcv/res/*.svg`
|
||||
- Design tokens (ring palette): `/home/w1n5t0n/src/MEMLNaut-NISPS/docs/redesign/manifold-export/tokens/colors.css`
|
||||
- Spine/engine context: `/home/w1n5t0n/src/MEMLNaut-NISPS/docs/specs/engine-architecture.md` (§2), `…/recon/findings-design-and-manifold.md` (§4), `…/recon/findings-engine-surface.md`, `…/aimmersive-clone-spec.md` (routeOutputs §6, §7 visual table, §10)
|
||||
- Spine/engine context: `/home/w1n5t0n/src/MEMLNaut-NISPS/docs/specs/engine-architecture.md` (trimmed 2026-07), `…/recon/findings-design-and-manifold.md` (§4), `…/recon/findings-engine-surface.md`, `…/_archive/aimmersive-clone-spec.md` (routeOutputs §6, §7 visual table, §10)
|
||||
|
||||
Sources (VCV SDK / widgets): [VCV custom lights](https://community.vcvrack.com/t/how-to-use-custom-lights/1941), [Migrate2 (drawLayer/layer 1)](https://vcvrack.com/manual/Migrate2), [Plugin Development Tutorial (RACK_DIR/make dist)](https://vcvrack.com/manual/PluginDevelopmentTutorial), [Plugin API Guide](https://vcvrack.com/manual/PluginGuide).
|
||||
---
|
||||
|
|
@ -500,10 +518,13 @@ spine; WebMIDI out with per-output CC#/ch/name/range; OSC-over-WS to the Deno br
|
|||
The Outputs dock panel specialises per backend (`manifold/src/dock/OutputsBackendConfig.tsx`) with named presets
|
||||
(`manifold/src/backends/presets.ts`). Audio gated via `engine.audio.setMuted` on non-synth modes.
|
||||
|
||||
**VCV module** — see `docs/specs/vcv-module.md` "⚠️ BUILD DELTAS (2026-06-28)" for the authoritative build target: 8 inputs ×
|
||||
16 outputs, an LED ring per output (drawLayer + nvgArc), palette from the frontend tokens, WS↔OSC bridge
|
||||
(browser OSC backend → `manifold/osc-bridge/` Deno relay → the module's OSC server), bidirectional training, and
|
||||
the `nisps-core`→`nisps/` core-path repoint. The existing `vcv/` module (2→12) is evolved, not rebuilt.
|
||||
**VCV module** — built: 8 inputs × 16 outputs, an LED ring per output (drawLayer + nvgArc), palette from the
|
||||
frontend tokens, WS↔OSC bridge (browser VCV backend → `manifold/osc-bridge/` Deno relay → the module's OSC
|
||||
server), bidirectional training over `/nisps/feedback`, and (P6, 2026-07-18) the core reunification onto
|
||||
`nisps::ml::MLPCore<DynamicStorage>`. Authoritative module contract: `vcv-module.md`.
|
||||
|
||||
**Still TODO** (clear in-code): the VCV↔browser bridge browser-side wiring to the new `/nisps/feedback` verb;
|
||||
the particle backend is a no-op passthrough pending the `visualizer.js` faithful port; CV/gate transport.
|
||||
**Update 2026-07-21:** the earlier TODOs closed — `/nisps/feedback` browser-side wiring shipped
|
||||
(`vcv-backend.ts` `sendFeedback`); the flow-field port shipped (`manifold/src/console/flow-field.ts` +
|
||||
`ParticleStage`; `particle-backend.ts` is a deliberate no-op transport because the visualiser reads the spine
|
||||
directly, and selecting it gates the synth audio); CV/gate shipped as the `cvgate` uSEQ Web Serial backend
|
||||
(`manifold/src/backends/cv-backend.ts`, protocol spec `useq-cv-protocol.md`) rather than the §2.5 designs.
|
||||
|
|
|
|||
|
|
@ -6,9 +6,29 @@ layer: behavioural
|
|||
|
||||
# Dock Spec — Console Right-Dock Drawers + Per-Output Controls
|
||||
|
||||
*Workstream D. Read-only design, 2026-06-27. Target app: `manifold/` (Vite + React + TS), wired to the parity-tested TS engine (`playground/src`) via the headless `EngineApi` boundary (`engine-architecture.md` §2). Citations are `file:line`. British spelling in product copy. The built-in synth is the **"Powerful Synth Engine"** — the string "C15" MUST NEVER appear in the UI.*
|
||||
*Workstream D. Designed 2026-06-27; built 2026-06-28 with operator-directed restructuring.
|
||||
British spelling in product copy. The built-in synth is the **"Powerful Synth Engine"** — the
|
||||
string "C15" MUST NEVER appear in the UI.*
|
||||
|
||||
This spec replaces the placeholder Shape/Feel/Route/Health/Help drawers in `ConsoleApp.jsx` (`recon/findings-design-and-manifold.md` §2 — keyboard `1-5` map drawers, `\` toggles depth) with the real dock contents.
|
||||
> **Grounding note (2026-07-21).** Treat every `file:line` cite in the body as historical
|
||||
> grounding: `playground/*` died with the retired SolidJS playground, `aimmersive-clone-spec.md`
|
||||
> is archived at `_archive/aimmersive-clone-spec.md`, and `engine-architecture.md` was trimmed
|
||||
> 2026-07 (its cited § numbers are gone). What shipped diverges from the prescription in shape
|
||||
> but not in substance:
|
||||
>
|
||||
> - **Drawer roster**: the operator restructured the dock during the build — a top **Mode
|
||||
> selector** plus five drawers (`learn`, `inputs`, `route`, `settings`, `help` —
|
||||
> `manifold/src/console/Dock.tsx`/`Drawers.tsx`). The §0 six-icon roster (SYNTH/VISUAL as
|
||||
> drawers) became per-Mode config inside the Outputs drawer instead.
|
||||
> - **Depths**: two shipped, not three — `condensed` (360px side panel) and `expanded` (centred
|
||||
> modal). The separate FULL "advanced backend modal" (§4, `<BackendAdvanced/>`) was built,
|
||||
> then deleted as a duplicate editor (2026-07 sweep, S18); the sole per-backend editor is
|
||||
> `manifold/src/dock/OutputsBackendConfig.tsx` in the Outputs drawer.
|
||||
> - **The per-output control row** (§3, the heart of this spec) shipped essentially as designed:
|
||||
> `manifold/src/dock/OutputControlRow.tsx` + `output-state.ts` (off/fixed/live + mute +
|
||||
> solo/arm + min/max/curve).
|
||||
> - The feedback surface shipped richer than §1 (Explore-and-Place with reroll/nudge/place, plus
|
||||
> geometric dislike) — see `docs/adr/rl-feedback-design.md` and `nisps/ml/feedback.hpp`.
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -47,18 +67,18 @@ Owns *how the model learns from your gestures*: the feedback-mode selector, the
|
|||
|
||||
### 1.1 FEEDBACK_MODE selector ("Down Action")
|
||||
|
||||
The `+` (up) verdict is always "keep this" (`addExample` + train, `findings-feedback-behaviour.md:102`); the `−` (down) verdict is **selectable** among the ported feedback modes (`feedback-modes-port-spec.md` §1; `FeedbackController<MLP_T>`). The new 2-mode product surface (per the prompt's "Mode 1 / Mode 2"):
|
||||
The `+` (up) verdict is always "keep this" (`addExample` + train, `findings-feedback-behaviour.md:102`); the `−` (down) verdict is **selectable** among the ported feedback modes (`plans/feedback-modes-port-spec.md` §1; `FeedbackController<MLP_T>`). The new 2-mode product surface (per the prompt's "Mode 1 / Mode 2"):
|
||||
|
||||
| Selector label (UI copy) | Engine `FeedbackMode` | Behaviour |
|
||||
|---|---|---|
|
||||
| **Push away** (Mode 1) | `Avoid` (`feedback.hpp` enum, `findings-feedback-behaviour.md:90`) | down → geometric-dislike: perturbs the mapping away from what you disliked. In the deployed core this routes to `move_weights(speed, spread, pinMask)` (`feedback-modes-port-spec.md` §2.5 — the true k-NN centroid push is firmware-only, out of scope). |
|
||||
| **Push away** (Mode 1) | `Avoid` (`feedback.hpp` enum, `findings-feedback-behaviour.md:90`) | down → geometric-dislike: perturbs the mapping away from what you disliked. In the deployed core this routes to `move_weights(speed, spread, pinMask)` (`plans/feedback-modes-port-spec.md` §2.5 — the true k-NN centroid push is firmware-only, out of scope). |
|
||||
| **Explore & place** (Mode 2) | `RandomiseMlp` (`findings-feedback-behaviour.md:122`) | down → snapshot + `draw_weights(spread)` re-rolls the whole net into a scratchpad you audition by moving the joystick; `+`/drag commits a `+1` example at the chosen input and **restores the real net** (`findings-feedback-behaviour.md:135-146`); down-again **cancels** (restore snapshot). |
|
||||
|
||||
- A third engine mode `RandomiseOutputs` (bypass MLP, hold a static random vector, `findings-feedback-behaviour.md:111`) exists in the core but is **not** surfaced as a product mode in v1 — expose it only behind `?debug=1` as "Static roll". **(Open choice 1.)**
|
||||
- **Peek**: a 2-segment pill (`Push away` / `Explore & place`), the current mode highlighted in `--accent`. This pill is *also* mirrored next to the Verdict cluster (`VerdictCluster`, `findings-design-and-manifold.md:50`) so it is reachable during live play without opening the drawer — matching how a-immersive puts the `rl-label` above the RL buttons (`aimmersive-clone-spec.md:283`).
|
||||
- **Expand**: the pill + a one-line plain-English description of the active mode + an **"exploring…" indicator** that lights when `engine.feedback.exploring()` is true (`findings-feedback-behaviour.md:237`), reusing the `<NoiseRing>` colour ramp (off/active/high, `aimmersive-clone-spec.md:98`). While exploring in Mode 2, training is paused (`learning_paused()`, `findings-feedback-behaviour.md:48`) — show a small "learning paused" badge.
|
||||
- **FULL**: the feedback lab — a diagram of the active state machine (idle → exploring → commit/cancel), the raw `FeedbackAction` log of the last presses (`findings-feedback-behaviour.md:90` enum), plus the spread/tame "Health lab" sliders (§1.3) for radical exploration tuning.
|
||||
- **Bind point**: `engine.feedback.setMode(mode)` → `nisps_ml_feedback_set_mode` (`feedback-modes-port-spec.md` §4). Switching mode while exploring auto-aborts and restores the net (`set_mode` calls `abort_explore`, `findings-feedback-behaviour.md:96`). Persist `feedbackMode` in the session blob.
|
||||
- **Bind point**: `engine.feedback.setMode(mode)` → `nisps_ml_feedback_set_mode` (`plans/feedback-modes-port-spec.md` §4). Switching mode while exploring auto-aborts and restores the net (`set_mode` calls `abort_explore`, `findings-feedback-behaviour.md:96`). Persist `feedbackMode` in the session blob.
|
||||
|
||||
### 1.2 SOLO / arm variant chooser
|
||||
|
||||
|
|
@ -323,7 +343,7 @@ Shared leaf primitives live in `shared/primitives/` (`<Slider>`, `<DualRangeSlid
|
|||
|
||||
| UI surface | EngineApi call | Engine route |
|
||||
|---|---|---|
|
||||
| Feedback-mode pill | `engine.feedback.setMode(m)` | `nisps_ml_feedback_set_mode` (`feedback-modes-port-spec.md` §4) |
|
||||
| Feedback-mode pill | `engine.feedback.setMode(m)` | `nisps_ml_feedback_set_mode` (`plans/feedback-modes-port-spec.md` §4) |
|
||||
| `−` verdict | `engine.feedback.thumbsDown()` | dispatch on mode → `on_down` (`findings-feedback-behaviour.md:90`) |
|
||||
| `+` verdict | `engine.feedback.thumbsUp()` | `addExample` + train, or `CommitStore` when exploring |
|
||||
| Arm (S) | `engine.feedback.setFocus(mask)` | `nisps_ml_feedback_set_focus` + `buildPinMask` |
|
||||
|
|
@ -363,7 +383,7 @@ The design reasoning, drawer model, tri-state/mute/arm separation and per-output
|
|||
`mapping.ts`), not imported from the deployed snapshot.
|
||||
- **`--dock-width` and min-thumb `#4488ff` are deployed-a-immersive values, absent from the Manifold tokens** —
|
||||
use Manifold tokens (`manifold-export/tokens/`) or add the missing ones deliberately.
|
||||
- A few `findings-feedback-behaviour.md:NN` citations are out of range / point at `feedback-modes-port-spec.md`
|
||||
- A few `findings-feedback-behaviour.md:NN` citations are out of range / point at `plans/feedback-modes-port-spec.md`
|
||||
instead (`learning_paused()` = `feedback.hpp:85` / port-spec:238; `exploring()` = `feedback.hpp:84`).
|
||||
- The React `<EngineProvider>`/`useEngine` pattern is in `findings-design-and-manifold.md §4`, not
|
||||
`engine-architecture.md §2.2` (that doc is SolidJS).
|
||||
|
|
|
|||
|
|
@ -1,434 +1,88 @@
|
|||
---
|
||||
kind: spec
|
||||
stability: evolving
|
||||
stability: stable
|
||||
layer: cross-cutting
|
||||
---
|
||||
|
||||
# NISPS Foundation Architecture — One Engine, Two Skins
|
||||
# Browser Engine Architecture — the EngineApi Seam and the Reactive Spine
|
||||
|
||||
*Status: implementation-ready spec. Scope: the foundation only — the headless engine layer plus the two-skin shell. The detailed UX of each skin (Console drawers, interactive heatmap craft, etc.) is owned by `plans/playground-2.0-rewrite-plan.md` and not re-litigated here.*
|
||||
|
||||
*Constraint from the operator: "lay solid ground that's simple and doesn't paint us into a corner." Build BOTH skins on ONE engine; don't port the engine twice; don't inherit existing tech debt.*
|
||||
*Supersession note (2026-07): this spec's original "One Engine, Two Skins" framing — a fresh
|
||||
`playground2/` tree hosting a faithful a-immersive skin plus a Console skin — is superseded by
|
||||
`plans/one-core-engine-refactor.md` (executed), which states: "the seam is built — Manifold is its
|
||||
realisation; the axis of unification now runs vertically, core↔targets, not horizontally,
|
||||
skin↔skin." The playground2 directory plan, reuse-vs-rebuild table, two-entry Vite/nginx serving
|
||||
scheme, S0–S7 build sequence, and operator open questions were all consumed by the Manifold build
|
||||
(`plans/BUILD-PLAN.md`, executed) and are deleted here; git history keeps them. What remains below
|
||||
is the surviving contract: the headless `EngineApi` boundary and the reactive-spine invariant, as
|
||||
realised in `manifold/src/engine/`.*
|
||||
|
||||
---
|
||||
|
||||
## 0. The one decision that drives everything
|
||||
## 1. The seam: engine vs presentation
|
||||
|
||||
**Separate the engine from the skin by an explicit, headless boundary — `EngineApi` — and let both skins be pure consumers of it.** Today there is no such boundary: `mode-runtime.ts` fuses WASM lifecycle, the reactive graph, input adapters, audio host, snapshots, heatmap sampling, and per-mode component wiring into one 663-line hook that each cloned `*Mode.tsx` calls. That fusion is *the* debt. The foundation's whole job is to draw the line the god-hook erased, on the correct side:
|
||||
**The engine is separated from presentation by an explicit, headless boundary — `EngineApi` — and
|
||||
the UI is a pure consumer of it.** The litmus test for every file: if it imports React, JSX, or
|
||||
touches the DOM, it is presentation; otherwise it is the engine. The engine compiles and is
|
||||
testable headlessly (the `window.__nisps` probe under `?debug=1`) with zero presentation.
|
||||
|
||||
```
|
||||
┌──────────────────────────── ONE ENGINE ────────────────────────────┐
|
||||
gesture │ input pipeline → ML (WASM) → output pipeline → backends │ sound
|
||||
────────► │ (the reactive spine: one memo chain + one side-effect) │ ────────►
|
||||
│ exposes a headless EngineApi { inputRaw, mlOutput, routedOutput, │
|
||||
│ verdict actions, history, status } — NO JSX, NO DOM │
|
||||
────────► │ (the reactive spine: one derivation path + one side-effect) │ ────────►
|
||||
│ exposes a headless EngineApi — NO JSX, NO DOM │
|
||||
└────────────────────────────────────────────────────────────────────┘
|
||||
▲ ▲
|
||||
│ reads EngineApi │ reads EngineApi
|
||||
┌──────────┴───────────┐ ┌──────────┴───────────┐
|
||||
│ skin-a-immersive │ │ skin-2.0 (Console) │
|
||||
│ (faithful clone) │ │ (right dock+drawers) │
|
||||
│ served at / │ │ served at /2.0 │
|
||||
└──────────────────────┘ └──────────────────────┘
|
||||
▲
|
||||
│ reads EngineApi
|
||||
┌──────────┴───────────┐
|
||||
│ Manifold Console │
|
||||
│ (React, manifold/) │
|
||||
└──────────────────────┘
|
||||
```
|
||||
|
||||
The litmus test for every file: **if it imports `solid-js/web`, JSX, or touches the DOM, it is a skin; otherwise it is the engine.** The engine compiles and is testable headlessly (Node/`__nisps` probe) with zero presentation. Two skins on one engine is then trivially true: a skin is just a tree of components that read `EngineApi` accessors and call `EngineApi` actions. Neither skin can break live-feedback, because neither skin owns the spine.
|
||||
Realisation (see `MAP.md` §manifold for the per-file inventory):
|
||||
|
||||
- `manifold/src/engine/engine-api.ts` — the `EngineApi` class: the entire UI-facing surface
|
||||
(inputs, training, feedback wrappers over the `nisps_ml_feedback_*` C ABI, audio lifecycle).
|
||||
- `manifold/src/engine/spine.ts` — the reactive spine (§2).
|
||||
- `manifold/src/engine/EngineProvider.tsx` + `useEngine.ts` — the React binding via
|
||||
`useSyncExternalStore`. These two files are the only presentation-side entry to the engine.
|
||||
- `manifold/src/debug/probe.ts` — `window.__nisps`, reads `EngineApi` only, gated by `?debug=1`.
|
||||
|
||||
The UI cannot create a second data path because it has no access to `WasmIML`, the backend, or the
|
||||
spine's internals — only to `EngineApi`. That structural property, not vigilance, is what makes
|
||||
the historic "MLP output stops updating when I move the joystick" bug class impossible.
|
||||
|
||||
---
|
||||
|
||||
## 1. Module / directory structure
|
||||
|
||||
**Build in a fresh tree: `playground2/`** (justification in §3). Inside it, the engine and the two skins are siblings; presentation never reaches into engine internals except via `EngineApi`.
|
||||
|
||||
```
|
||||
playground2/
|
||||
├── index.html # skin-a-immersive entry (default, root)
|
||||
├── 2.0.html # skin-2.0 entry (Console variant)
|
||||
├── vite.config.ts # two inputs; base resolved per-build (§4)
|
||||
├── package.json # solid-js, vite, vite-plugin-solid, @playwright/test
|
||||
├── public/
|
||||
│ ├── nisps.wasm nisps.js # copied from scripts/build-wasm.sh output
|
||||
│ └── c15.wasm c15.glue.js # (only if/when C15 lands; deferred)
|
||||
│
|
||||
├── src/
|
||||
│ ├── engine/ # ───────── HEADLESS. No JSX. No DOM. ─────────
|
||||
│ │ │
|
||||
│ │ ├── index.ts # createEngine(schema, opts): EngineApi ← THE boundary
|
||||
│ │ ├── types.ts # EngineApi, EngineOptions, EngineStatus
|
||||
│ │ │
|
||||
│ │ ├── spine/
|
||||
│ │ │ ├── control-graph.ts # the memo chain + the single send-effect (§2). ~1 file.
|
||||
│ │ │ └── coalesce.ts # pointer-rate → display-cadence batch()+microtask gate
|
||||
│ │ │
|
||||
│ │ ├── ml/ # LIFTED from playground/ (parity-tested), de-storified
|
||||
│ │ │ ├── wasm-iml.ts # WasmIML class — adapt: HeapVec, reused out-buffer, no store writes
|
||||
│ │ │ ├── heap-vec.ts # NEW: re-derive HEAPF32 view on every access (replaces rebind())
|
||||
│ │ │ ├── wasm-worker.ts # LIFT as-is: disposable training worker (own WASM instance)
|
||||
│ │ │ └── dataset.ts # LIFT as-is
|
||||
│ │ │
|
||||
│ │ ├── audio/ # LIFTED from playground/ — already correct
|
||||
│ │ │ ├── engine-host.ts # LIFT ~as-is: ?worker&url, fetch+post bytes, transferables
|
||||
│ │ │ └── worklet/
|
||||
│ │ │ ├── nisps-processor.ts # LIFT as-is: 2nd WASM, hand-rolled imports
|
||||
│ │ │ └── audioworklet-globals.d.ts # LIFT as-is
|
||||
│ │ │
|
||||
│ │ ├── pipelines/ # LIFTED — pure fns, golden-tested
|
||||
│ │ │ ├── input.ts # deadzone→zoom→curve→smoothing→momentum (was input/pipeline.ts)
|
||||
│ │ │ ├── output.ts # global curve→smoothing→slew→freeze (was output/pipeline.ts)
|
||||
│ │ │ └── curves.ts # ONE curve catalog, golden-tested vs nisps/core/math.hpp (§3)
|
||||
│ │ │
|
||||
│ │ ├── backends/ # OutputBackend adapters (plan §3.6)
|
||||
│ │ │ ├── backend.ts # interface OutputBackend { send, start, teardown }
|
||||
│ │ │ ├── web-audio.ts # wraps engine-host (v1)
|
||||
│ │ │ ├── web-midi.ts # 7-bit CC out (v1 adapter; stub-OK at foundation)
|
||||
│ │ │ └── osc-bridge.ts # WS bridge (deferred; behind locked contract)
|
||||
│ │ │
|
||||
│ │ ├── stores/ # module singletons, split by update cadence (plan §3.2)
|
||||
│ │ │ ├── ml-store.ts # status/arch/dataset + outputs F32 signal + weightsRevision
|
||||
│ │ │ ├── input-store.ts # config store + raw-axes F32 signal {equals:false}
|
||||
│ │ │ ├── output-store.ts # global gate config + reuse buffer
|
||||
│ │ │ ├── control-store.ts # compound axes + per-param createMemo fanout
|
||||
│ │ │ ├── routing-store.ts # control-point off/fixed/live matrix
|
||||
│ │ │ ├── history-store.ts # snapshot DAG (undo/A-B/trail/snapshots)
|
||||
│ │ │ ├── session-store.ts # presets, persistence, URL params, mode switch
|
||||
│ │ │ └── bus.ts # LIFT as-is: typed sync pub/sub, cross-cutting events only
|
||||
│ │ │
|
||||
│ │ ├── runtime/ # the decomposed god-hook (plan §3.4) — engine-side hooks
|
||||
│ │ │ ├── use-input-adapters.ts # pointer/joystick/gamepad/mic → input-store.setRaw
|
||||
│ │ │ ├── use-audio-lifecycle.ts # backend start/stop/teardown on mount/mode-switch
|
||||
│ │ │ ├── use-snapshots.ts # DAG ops (→ history-store)
|
||||
│ │ │ ├── use-heatmap-sampler.ts # input-space heatmap, throttled to weightsRevision
|
||||
│ │ │ ├── use-trail.ts
|
||||
│ │ │ └── use-auto-explore.ts
|
||||
│ │ │
|
||||
│ │ ├── feedback/
|
||||
│ │ │ └── controller.ts # the 3-mode FeedbackController: thumbsUp/thumbsDown/randomize
|
||||
│ │ │ # + undo, wrapping moveWeights/drawWeights/train + auto-snapshot
|
||||
│ │ │
|
||||
│ │ ├── persist.ts # persist<T>(store, version, migrate): versioned + base64 weights
|
||||
│ │ └── probe.ts # window.__nisps — reads EngineApi only; ?debug=1-gated
|
||||
│ │
|
||||
│ ├── shared/ # ───── presentation shared by BOTH skins ─────
|
||||
│ │ ├── primitives/ # LIFTED: Slider, JoyMap, Heatmap, XYPad, LossPlot,
|
||||
│ │ │ # GradientFlow, WeightHealth, ProgressRing, ... (+ .demo.tsx)
|
||||
│ │ ├── theme/
|
||||
│ │ │ └── tokens.css # orange #ff6a00, danger #ff4466, glass, JetBrains Mono;
|
||||
│ │ │ # lint allowlist enforced
|
||||
│ │ ├── EngineProvider.tsx # createContext(EngineApi); both skins wrap their tree
|
||||
│ │ └── GenericMode.tsx # ONE schema-driven mode component (plan §3.5)
|
||||
│ │
|
||||
│ ├── skin-a-immersive/ # ───── DEFAULT skin (faithful a-immersive) ─────
|
||||
│ │ ├── main.tsx # mounts <EngineProvider><AImmersiveApp/></EngineProvider>
|
||||
│ │ ├── AImmersiveApp.tsx
|
||||
│ │ └── components/ # bottom-sheet-faithful chrome, floating RL buttons, top strip
|
||||
│ │
|
||||
│ ├── skin-2.0/ # ───── Console skin (Playground 2.0) ─────
|
||||
│ │ ├── main.tsx # mounts <EngineProvider><ConsoleApp/></EngineProvider>
|
||||
│ │ ├── ConsoleApp.tsx # the Manifold + dock + drawers + Verdict cluster
|
||||
│ │ └── components/ # Dock, Drawer (3-depth), VerdictCluster, ReadoutStrip, ...
|
||||
│ │
|
||||
│ └── dev/
|
||||
│ └── PrimitivesShowcase.tsx # /dev/primitives (shared, skin-agnostic)
|
||||
│
|
||||
└── tests/e2e/ # Playwright: spine invariant, both skins, parity hooks
|
||||
```
|
||||
|
||||
### Why this shape is "simple and doesn't paint us into a corner"
|
||||
|
||||
- **One axis of variation per directory.** Adding an *engine* touches `nisps/` + `schemas/` + `public/*.wasm`; zero skin files. Adding a *backend* touches `engine/backends/`; zero spine, zero skin. Adding/altering a *skin* touches one `skin-*/` tree; zero engine. This is the orthogonality principle made physical.
|
||||
- **`EngineApi` is the only seam.** Skins import `useEngine()` (the context) and nothing from `engine/` internals. A lint rule (`no-restricted-imports`: skins may not import `engine/**` except `engine/index.ts`/`engine/types.ts`) keeps it honest. You could delete a whole skin and the engine + tests still pass.
|
||||
- **`GenericMode` lives in `shared/`, not in a skin** — both skins render the same mode bodies; they differ only in *chrome* (how drawers/strips/clusters frame the canvas), not in *what a mode is*. This is what makes "two skins, one engine" cheap rather than a fork.
|
||||
|
||||
---
|
||||
|
||||
## 2. The reactive spine, concretely
|
||||
|
||||
This is the load-bearing structural fix and the literal definition of "doesn't paint us into a corner": there is exactly one path from gesture to sound, every consumer reads it, and a desync is a failing test rather than a recurring prod bug.
|
||||
|
||||
### 2.1 The chain (in `engine/spine/control-graph.ts`)
|
||||
|
||||
```ts
|
||||
// One input entry point. {equals:false} so identical-reference writes still notify.
|
||||
const inputRaw: Accessor<Float32Array> // input-store's raw-axes signal
|
||||
|
||||
// ── pure memos ──────────────────────────────────────────────────────────────
|
||||
const processedInput = createMemo(() => // deadzone→zoom→curve→smoothing→momentum
|
||||
runInputPipeline(inputRaw(), inputStore.config, inputState)); // pure, golden-tested
|
||||
|
||||
const mlOutput = createMemo(() => { // WasmIML.infer INTO a reused buffer
|
||||
const inp = processedInput(); // (no per-frame Float32Array alloc)
|
||||
iml.setInputsClamped(inp); // clamps to REAL arch (no phantom channels — §6/D)
|
||||
return iml.inferInto(mlOutBuf); // returns the reused buffer; pure *read* of weights
|
||||
}, mlOutBuf, { equals: false });
|
||||
|
||||
const routedOutput = createMemo(() => // voice-space + global gate
|
||||
runOutputPipeline(mlOutput(), routingStore, outputStore.config, outputState));
|
||||
|
||||
// ── the ONE side-effect ──────────────────────────────────────────────────────
|
||||
createEffect(() => {
|
||||
const out = routedOutput(); // single dependency
|
||||
backend.send(fillSendBuffer(out)); // dedicated re-filled transferable (neutering-safe)
|
||||
}); // ← the ONLY postMessage / engine post in the app
|
||||
```
|
||||
|
||||
Rules made structural, not aspirational:
|
||||
- **Memos are pure.** No `postMessage`, no `backend.send`, no store writes inside a memo. (Lint/review rule; the plan explicitly rejects P3's "postMessage in a memo" and P1's `createComputed`.)
|
||||
- **Weights mutate only through `ml-store` actions** that bump `weightsRevision`. `mlOutput`'s memo reads weights; because RL/train actions bump the revision and the memo's input or revision is a dependency, output re-derives. There is no leaked write path that can update audio without updating the UI.
|
||||
- **`mlOutput` writes into `mlOutBuf` and returns it** with `{equals:false}` — fixes today's per-frame `new Float32Array(...)` alloc in `WasmIML.process()` (line 302) and the god-hook's `recomputeOutputs`.
|
||||
- **Pointer-rate is coalesced to display cadence** in `coalesce.ts` via `batch()` + a microtask, *reactively* (not a rAF poll). rAF touches **canvas drawing only**, never inference.
|
||||
- **The send buffer is dedicated and owned by the effect**, separate from any signal buffer, because transferring neuters the source (today's `EngineHost.setParams` transfers `params.buffer`).
|
||||
|
||||
### 2.2 How BOTH skins consume it identically
|
||||
|
||||
Every consumer **reads accessors**; nobody is *pushed to*. There is no "push outputs to the UI" path that can rot — the bug class is deleted, not patched.
|
||||
|
||||
```ts
|
||||
// shared/EngineProvider.tsx
|
||||
const EngineContext = createContext<EngineApi>();
|
||||
export const useEngine = () => useContext(EngineContext)!;
|
||||
|
||||
// EngineApi (engine/types.ts) — the entire skin-facing surface
|
||||
interface EngineApi {
|
||||
// live reactive reads (the spine)
|
||||
inputRaw: Accessor<Float32Array>;
|
||||
processedInput:Accessor<Float32Array>;
|
||||
mlOutput: Accessor<Float32Array>; // raw model space (heatmap, diagnostics)
|
||||
routedOutput: Accessor<Float32Array>; // post-pipeline (engine, readout strip, visualizer)
|
||||
status: EngineStatus; // ready, audioStarted, training, examples, ... (store proxy)
|
||||
weightsRevision: Accessor<number>;
|
||||
layerStats: () => Float32Array; // diagnostics pull (throttled by caller)
|
||||
lossHistory: Accessor<ReadonlyArray<number>>;
|
||||
// actions
|
||||
setInput: (x: number, y: number) => void; // the ONLY input door
|
||||
feedback: FeedbackController; // thumbsUp/thumbsDown/randomize/undo/canUndo
|
||||
train: () => void;
|
||||
history: HistoryApi; // DAG: A/B pin/swap, jump-to-node
|
||||
audio: { start(): Promise<void>; stop(): Promise<void>; setMuted(b): void };
|
||||
schema: ModeSchema;
|
||||
}
|
||||
```
|
||||
|
||||
- **skin-a-immersive**: a floating RL button reads `engine.feedback.thumbsUp`; the top strip reads `engine.routedOutput()`; the XY pad calls `engine.setInput(x,y)`.
|
||||
- **skin-2.0**: the Verdict cluster's 👍 reads the *same* `engine.feedback.thumbsUp`; the ReadoutStrip reads the *same* `engine.routedOutput()`; the Manifold's pointer-down calls the *same* `engine.setInput(x,y)`.
|
||||
|
||||
Identical bindings, different chrome. A skin cannot create a second reactive path because it has no access to `WasmIML`, `backend`, or the stores' internal setters — only to `EngineApi`.
|
||||
|
||||
### 2.3 Live-feedback guarantee + the e2e assertion
|
||||
|
||||
**Guarantee:** any change to `inputRaw` (gesture) *or* `weightsRevision` (RL/train) propagates — in the same synchronous tick — to `routedOutput`, which the single effect sends to the backend and which every UI consumer reads. There is no code path that updates audio without updating the readout, or vice versa.
|
||||
|
||||
**The e2e invariant, asserted on every mode, in both skins, in CI** (the centerpiece; mirrors plan §3.1):
|
||||
|
||||
```ts
|
||||
// tests/e2e/spine.spec.ts — runs for each {mode} × {skin: '/', '/2.0'}
|
||||
const before = await page.evaluate(() => window.__nisps.getOutputs());
|
||||
await page.evaluate(() => window.__nisps.setInputs([0.9, 0.1])); // synchronous probe
|
||||
const after = await page.evaluate(() => window.__nisps.getOutputs());
|
||||
expect(after).not.toEqual(before); // ML output changed
|
||||
const eng = await page.evaluate(() => window.__nisps.getEngineParams());
|
||||
expect(eng).toChange(); // engine params changed, same input
|
||||
```
|
||||
|
||||
`window.__nisps` (in `engine/probe.ts`) is exposed synchronously, bypasses Solid reactivity with `untrack`/`batch`, and **reads `EngineApi` only** — so the probe exercises the exact path both skins use. Because the probe lives in the engine layer (not a skin), the *same* spec file runs unchanged against both `index.html` and `2.0.html`; if either skin ever fails to read the spine, that skin's row goes red.
|
||||
|
||||
A complementary **no-per-frame-alloc** check (heap-snapshot fuzz around a `setInputs` storm) guards the reused-buffer discipline.
|
||||
|
||||
---
|
||||
|
||||
## 3. Reuse-vs-rebuild
|
||||
|
||||
**Recommendation: build the foundation in a FRESH directory `playground2/`, lifting the parity-tested engine modules wholesale and discarding the skin/runtime layer.** This is strictly simpler than refactoring `playground/` in place and carries the least debt, *because the debt and the value are cleanly separable in the existing tree* — the value is the headless engine modules (already DOM-free), the debt is the god-hook + cloned modes + toy `App.tsx`.
|
||||
|
||||
### Reuse table
|
||||
|
||||
| Existing `playground/` module | Verdict | Action / why |
|
||||
|---|---|---|
|
||||
| `audio/engine-host.ts` | **Lift ~as-is** | Already correct: `?worker&url`, fetch-on-main + post bytes, transferables, lazy AudioContext. Best code in the repo. Minor: wrap behind `WebAudioBackend`. |
|
||||
| `audio/worklet/nisps-processor.ts` | **Lift as-is** | Two-WASM, hand-rolled auto-discovered imports, 128-sample shuttle. Load-bearing and correct. |
|
||||
| `audio/worklet/audioworklet-globals.d.ts` | **Lift as-is** | — |
|
||||
| `ml/wasm-iml.ts` | **Lift + adapt** | Keep the C-API surface and lifecycle. **Adapt:** (1) replace `HeapBuffer.rebind()` (manual, error-prone via `Object.defineProperty`) with `HeapVec` re-derive-on-access; (2) `inferInto(buf)` instead of `process()` allocating; (3) **remove `mlStore.__set*` writes from the class** — the class becomes a pure WASM wrapper, the store observes it. (4) Plumb real `nisps_ml_loss_history` (replaces the `lossHistory=[loss]` fake at line 419). |
|
||||
| `ml/wasm-worker.ts` | **Lift as-is** | Disposable training worker w/ own WASM instance. |
|
||||
| `ml/dataset.ts` | **Lift as-is** | JS mirror of the C++ ring. |
|
||||
| `input/pipeline.ts` | **Lift as-is** → `engine/pipelines/input.ts` | Pure, golden-tested, bit-equivalent to legacy. Becomes the `processedInput` memo body. |
|
||||
| `output/pipeline.ts` | **Lift as-is** → `engine/pipelines/output.ts` | Pure; becomes `routedOutput` body. **Fix:** use reuse buffer. |
|
||||
| `output/curves.ts` | **Lift + fix** | **Unify with the diverging `Curve` enum** in `generated/types.ts` (`exp/log/square/sqrt/sigmoid/cubic`) into ONE catalog, golden-tested against `nisps/core/math.hpp`. (Plan §4 "Pure-fn pipelines" Keep+fix.) |
|
||||
| `stores/bus.ts` | **Lift as-is** | Typed sync pub/sub; cross-cutting events only. |
|
||||
| `stores/ml-store.ts` | **Adapt** | Keep the signal/store split (it's already right: `outputs` F32 `{equals:false}`, store for status). **Add** `weightsRevision`. **Invert ownership:** store observes `WasmIML`, class stops writing the store. |
|
||||
| `stores/input-store.ts`, `output-store.ts` | **Adapt** | Keep config-store + raw F32 signal pattern; align to spine (input-store's raw signal becomes `inputRaw`). |
|
||||
| `stores/control-store.ts` | **Adapt** | Keep axis tables/presets/`interpolateAxis`. **Replace** the routing mechanism with **per-param `createMemo` fanout** (kills the `control-routing.ts` `JSON.stringify`-in-untracked-effect anti-pattern). |
|
||||
| `stores/session-store.ts`, `mode-store.ts`, `exploration-store.ts` | **Salvage values, re-cut along plan §3.2** | Re-partition into `routing-store` (control-point `off/fixed/live`), `history-store` (DAG), `session-store` (persist+presets+URL+mode). Lift constants/preset data; rebuild the store shapes. |
|
||||
| `stores/persistence.ts` | **Rebuild small** → `engine/persist.ts` | Replace bespoke per-store `Partial<>` merges + `Infinity↔null` slew encoding + slow `Array.from()` weight JSON with one `persist<T>(store, version, migrate)`; base64 weights. |
|
||||
| `features/snapshots.ts`, `overrides.ts`, `trail.ts`, `heatmap-sampler.ts`, `weight-health.ts`, `mic-input.ts` | **Salvage logic into `engine/runtime/` hooks** | The *algorithms* are fine; the *wiring* (god-hook calling them imperatively) is the debt. Re-home as `use-snapshots`/`use-trail`/`use-heatmap-sampler` reading stores, not as side-effects of `setInput`. Snapshots fold into the DAG. |
|
||||
| `features/control-routing.ts` | **Discard** | The `JSON.stringify`-inside-`untrack` effect anti-pattern; replaced by control-store memo fanout. |
|
||||
| `features/session-preset.ts` | **Rebuild** | Composed-layers model (control/synth/weights/mode), per plan §3.8. |
|
||||
| `primitives/*` (16 + demos) | **Lift as-is** → `shared/primitives/` | Genuinely good, skin-agnostic, already have `.demo.tsx`. The reusable presentation layer. |
|
||||
| `dev/PrimitivesShowcase.tsx`, `debug/probe.ts` | **Lift + adapt** | Showcase as-is. Probe re-pointed at `EngineApi` (it currently pokes stores directly); `?debug=1`-gate it. |
|
||||
| **`modes/mode-runtime.ts` (663 lines)** | **DISCARD** | The god-hook. Its responsibilities are split across `engine/spine/`, `engine/runtime/`, `engine/feedback/`. **Do not inherit.** |
|
||||
| **`modes/*Mode.tsx` (9 cloned files, ~5.6k lines)** | **DISCARD** | Replaced by one `shared/GenericMode.tsx` driven by schema. The clones *are* the debt. |
|
||||
| `modes/ModeShell.tsx`, `ModeSwitcher.tsx`, `SettingsDrawer.tsx`, `mode-helpers.ts` | **Discard / re-design per skin** | Chrome belongs to skins now; a capability-class switcher is rebuilt minimally. |
|
||||
| `App.tsx`, `main.tsx` | **Discard** | Toy hash-router with a "home page." Replaced by two skin entries + `EngineProvider`. |
|
||||
| `modes/generated/*` | **Regenerate, don't lift** | Codegen output; add `capability_class` + `tier` fields (plan §3.5/§3.9) and emit into `playground2/`. |
|
||||
|
||||
**Verdict in one line:** lift the entire `engine/` + `shared/primitives/` (≈ the `ml/`, `audio/`, `input/`, `output/`, `primitives/`, `bus.ts` value — the parity-tested core) and rebuild only the ≈6.5k lines of god-hook + cloned modes + toy app that *are* the debt.
|
||||
|
||||
### Fresh dir vs refactor-in-place — justification
|
||||
|
||||
| | **Fresh `playground2/` (recommended)** | Refactor `playground/` in place |
|
||||
|---|---|---|
|
||||
| Debt inheritance | Zero — debt files never copied; engine lifted file-by-file on purpose | High — easy to leave a god-hook tendril; "delete later" rarely happens |
|
||||
| "Simple ground" | Clean import graph from day 1; lint seam enforceable immediately | Mixed old/new imports during migration; seam blurry for weeks |
|
||||
| Parity risk | None — engine files lifted verbatim, parity-check runs against the same `nisps.wasm` | None, but harder to prove which path a test exercised |
|
||||
| Live deploy | Untouched. `playground/dist` keeps building; we wire `playground2/dist` only when ready | Risk of breaking the live (if pointed) build mid-refactor |
|
||||
| Rollback | `rm -rf playground2/` | `git revert` across an entangled history |
|
||||
| Cost | One-time copy of ~10 good files | Ongoing vigilance against re-entanglement |
|
||||
|
||||
A fresh dir is the cheaper path to *less debt while reusing the parity-tested core* — exactly the operator's brief. `playground/` stays as the working reference (and keeps deploying) until `playground2/` reaches parity, then `playground/` is deleted and `playground2/`→`playground/` (a rename, by which point the engine seam is proven).
|
||||
|
||||
> **Note for the operator (open question A in the plan):** if 2.0 is actually destined for the laptop `~/src/manifold` tree, this whole `playground2/` lands there instead — the architecture is identity-agnostic and the dir name is the only thing that changes. This blocks namespace/repo identity; see §6.
|
||||
|
||||
---
|
||||
|
||||
## 4. Two-skins mechanism (build + serve)
|
||||
|
||||
**Decision: two Vite entries, one build, one `dist/` — NOT a runtime route.** Two HTML entry points compiled in a single `vite build` produce `dist/index.html` (skin A) and `dist/2.0.html` (skin B), sharing all common chunks (engine, primitives, wasm). Rationale against "simple, no corners":
|
||||
|
||||
- **A runtime route** (`/2.0` as a client route inside one bundle) would force both skins into one app shell, one router, and — crucially — risk a *shared mutable* engine instance and shared chrome assumptions, reintroducing exactly the cross-skin coupling we're eliminating. It also makes "delete a skin" hard.
|
||||
- **Two entries** keep the skins genuinely independent (each owns its `main.tsx` + `EngineProvider` mount) while Vite's `manualChunks`/automatic splitting means the engine + primitives are **one shared chunk loaded by both** — zero engine duplication on disk or over the wire. This is the cleanest expression of "one engine, two skins."
|
||||
|
||||
### Vite config
|
||||
|
||||
```ts
|
||||
// playground2/vite.config.ts
|
||||
import { defineConfig } from 'vite';
|
||||
import solid from 'vite-plugin-solid';
|
||||
import { resolve } from 'node:path';
|
||||
|
||||
const isolation = {
|
||||
'Cross-Origin-Opener-Policy': 'same-origin',
|
||||
'Cross-Origin-Embedder-Policy': 'require-corp',
|
||||
};
|
||||
|
||||
export default defineConfig(({ command }) => ({
|
||||
// Skin A is served at site root; skin B at /2.0/. We build ONCE with base '/2.0/'-aware
|
||||
// asset URLs? No — simpler: build relative, let nginx alias map. See note below.
|
||||
base: './', // relative asset URLs → same dist works at / and at /2.0/
|
||||
plugins: [solid()],
|
||||
server: { port: 5173, headers: isolation }, // dev: COOP/COEP for SharedArrayBuffer + worklet
|
||||
preview: { port: 4173, headers: isolation },
|
||||
build: {
|
||||
target: 'es2022',
|
||||
sourcemap: true,
|
||||
rollupOptions: {
|
||||
input: {
|
||||
main: resolve(__dirname, 'index.html'), // → dist/index.html (skin A, default)
|
||||
v2: resolve(__dirname, '2.0.html'), // → dist/2.0.html (skin B, Console)
|
||||
},
|
||||
},
|
||||
},
|
||||
}));
|
||||
```
|
||||
|
||||
**Why `base: './'` (relative) rather than a fixed `/2.0/` base.** The two skins share `dist/assets/*`. If skin B used absolute `base:'/2.0/'`, skin A (at `/`) would request the wrong asset prefix. Relative `base` makes **the same `dist/` mount correctly at both `/` and `/2.0/`** with no second build. The one caveat is the **AudioWorklet URL** and the **`?worker&url` chunk**: Vite emits these as module URLs resolved relative to the importing chunk, which works under a sub-path *as long as the chunk itself is loaded relatively* — relative `base` satisfies this. The worklet's WASM is fetched by absolute `/nisps.wasm`; under a sub-path we make that **origin-relative and configurable** (engine reads `import.meta.env.BASE_URL` or a runtime `<base>`), so `/2.0/` resolves `…/nisps.wasm` correctly. (Today `wasm-iml.ts`/`engine-host.ts` hardcode `/nisps.wasm` against `window.location.origin` — fine at root, must become base-aware for the sub-path. This is the single concrete code change the two-skins serving imposes.)
|
||||
|
||||
Two `package.json` scripts make dev ergonomic:
|
||||
```jsonc
|
||||
"dev": "vite", // serves both index.html and 2.0.html
|
||||
"dev:2.0": "vite --open /2.0.html",
|
||||
"build": "tsc --noEmit && vite build", // emits dist/index.html + dist/2.0.html
|
||||
```
|
||||
|
||||
### nginx (keeping the legacy a-immersive root untouched)
|
||||
|
||||
The current root is `meml-aimmersive` serving the **legacy vanilla** `a-immersive.html`. The new SolidJS app builds to `…/meml.lnfinitemonkeys.org/playground/dist` (currently unserved). The cleanest, no-corner serving plan: **add location blocks for the new app under explicit sub-paths, leave `/` on the legacy root for now**, and flip `/` to the new skin A only when the operator signs off.
|
||||
|
||||
```nginx
|
||||
# inside the existing 443 server { } for meml.lnfinitemonkeys.org
|
||||
# (COOP/COEP already set at server scope → inherited by all locations below)
|
||||
|
||||
set $pg2 /home/w1n5t0n/deployments/meml.lnfinitemonkeys.org/playground2/dist;
|
||||
|
||||
# ── Skin B: Console at /2.0 ────────────────────────────────────────────────
|
||||
location = /2.0 { return 301 /2.0/; }
|
||||
location /2.0/ {
|
||||
alias $pg2/;
|
||||
index 2.0.html;
|
||||
try_files $uri $uri/ /2.0/2.0.html; # SPA fallback to skin B's entry
|
||||
}
|
||||
|
||||
# ── Skin A: faithful clone, staged at /next while it's proven ──────────────
|
||||
location = /next { return 301 /next/; }
|
||||
location /next/ {
|
||||
alias $pg2/;
|
||||
index index.html;
|
||||
try_files $uri $uri/ /next/index.html; # SPA fallback to skin A's entry
|
||||
}
|
||||
|
||||
# ── Shared assets (one chunk set for both skins) ──────────────────────────
|
||||
location /assets/ { alias $pg2/assets/; } # relative-base requests resolve here
|
||||
location = /nisps.wasm { alias $pg2/nisps.wasm; }
|
||||
location = /nisps.js { alias $pg2/nisps.js; }
|
||||
|
||||
# legacy root stays:
|
||||
# root /home/w1n5t0n/deployments/meml-aimmersive; index a-immersive.html; (UNCHANGED)
|
||||
```
|
||||
|
||||
**Cutover (one-line change, when signed off):** point `/` at skin A by setting the server `root $pg2;` + `index index.html;` and a top-level `try_files $uri /index.html;`. Until then `/` serves the legacy artifact untouched, `/next/` is the new faithful skin, `/2.0/` is the Console. The auto-deploy script gains one line: after `playground/`'s `npm run build`, also `cd playground2 && npm install && npm run build` (or replace once `playground/` is retired). COOP/COEP is **server-scoped already**, so every sub-path inherits cross-origin isolation — no per-location header duplication needed (the one real correctness requirement for SharedArrayBuffer + the worklet under a sub-path is satisfied for free).
|
||||
|
||||
---
|
||||
|
||||
## 5. Build sequence (small, shippable, Playwright-testable)
|
||||
|
||||
Each step ends green and observable. Parity checkpoints (verification chokepoints C/E) called out. This mirrors the plan's Phase 0–4 but with the **two-skin seam established early** so neither skin can diverge.
|
||||
|
||||
| Step | Deliverable | Test gate | Parity |
|
||||
|---|---|---|---|
|
||||
| **S0 — Scaffold + seam** | `playground2/` Vite skeleton; two HTML entries; `tokens.css` + CSS-var lint allowlist; `EngineApi`/`EngineProvider` stubs; codegen re-emitting into `playground2/` with `capability_class`+`tier` and **build-fails-on-arch-mismatch**; `persist<T>` helper; `bus.ts` lifted. Two trivial skins each render "hello from skin A/B" reading a stub `EngineApi`. | `bun run typecheck` green; codegen golden test; Playwright loads `/` and `/2.0.html`, both mount. | Codegen idempotent (byte-identical regen). |
|
||||
| **S1 — Spine against a stub** | `control-graph.ts` memo chain + single send-effect with a **stubbed `mlOutput`** (deterministic fn of input); `input-store` + `output-store` with lifted pure pipelines (curve enums unified, golden-tested); `coalesce.ts`. `EngineApi.setInput`/`routedOutput` real. | **Spine invariant e2e** (`setInputs→getOutputs changes`) passes against the stub, **on both skins**; no-per-frame-alloc heap fuzz. | Pipeline golden vs legacy bit-equivalence. |
|
||||
| **S2 — WASM ML bridge** | Lift `wasm-iml.ts` + `wasm-worker.ts` + `dataset.ts`; add `HeapVec` re-derive; `inferInto(buf)`; `ml-store` observes the class (no class→store writes); `weightsRevision`; real `nisps_ml_loss_history` plumbed. `mlOutput` memo now calls real `WasmIML`. | Probe `infer`/`train`/`getLayerStats`; loss plot draws a *real* curve; spine invariant still green on both skins. | **Checkpoint:** main-thread inference matches native within 1e-5 (`parity-check.sh`). |
|
||||
| **S3 — Audio + first real mode via GenericMode** | Lift `engine-host.ts` + worklet behind `WebAudioBackend`; base-aware WASM URL; `GenericMode.tsx` rendering **PAFSynth** (synth class). Mount `GenericMode` inside *both* skin shells (minimal chrome each). | Per-mode live-feedback e2e (engine params change on input move) on both skins; "Start audio" gesture works. | **Checkpoint:** `parity-check.sh` green; browser PAFSynth audio equivalent to firmware (chokepoint C). |
|
||||
| **S4 — Skin A default, faithful** | Build out `skin-a-immersive/` to faithfully reproduce a-immersive chrome (bottom-sheet, floating RL buttons, top strip) over `GenericMode` + `EngineApi`. Feedback controller (`thumbsUp/Down/randomize/undo`) wired; snapshot DAG (`history-store`) behind undo. Serve at `/next/`. | Playwright drives the faithful loop via probe + via real DOM on `/next/`; visual-faithfulness check (key anchors, orange tokens); a-immersive feature-parity subset (chokepoint D start). | — |
|
||||
| **S5 — Skin B Console at /2.0** | Build out `skin-2.0/`: Manifold + right dock + 3-depth drawers + Verdict cluster + interactive ReadoutStrip, all reading the *same* `EngineApi`. RL-undo + A/B against the DAG. Serve at `/2.0/`. | The **same** spine + feedback e2e specs run against `/2.0/` unchanged and pass; Console-specific drawer/depth e2e. | — |
|
||||
| **S6 — Mode coverage + backends + persistence** | All in-scope engines via `GenericMode` (synth/sequencer/controller/visual classes; `SequencerLane`, `VisualEngine`, controller meters); capability-class switcher in both skins; `WebMidiBackend` + (stubbed) `OscBridgeBackend`; versioned persistence + base64 weights; control surface (axis memo fanout, presets, pinning). | Every mode passes live-feedback e2e in CI on both skins; tactile-constant e2e (3px/300ms) on the Console strip. | **Checkpoint:** parity per mode; full `run-all-tests.sh` green = chokepoint E. |
|
||||
| **S7 — Cutover** | Point nginx `/` at skin A; delete legacy `meml-aimmersive` serving (archive the artifact); retire `playground/`, rename `playground2/`→`playground/`; update deploy script. | Post-cutover smoke e2e against `/` and `/2.0/`. | — |
|
||||
|
||||
The deliberate ordering choice vs the plan: **establish `EngineApi` + both skin mounts at S0** (not after the Console is built), so from the very first stub the "two skins read one engine" property is *tested*, and no skin can quietly grow its own data path. The Console UX detail (drawers, depths) is still de-risked against `/dev/primitives` and user-tested with Dimi before S5, per the plan.
|
||||
|
||||
---
|
||||
|
||||
## 6. Risks & open questions for the operator
|
||||
|
||||
**Open questions that block or shape the build (need a call before S0/S2):**
|
||||
|
||||
1. **Canonical tree / identity (plan §8-A).** Is the foundation `playground2/` *in this VPS `MEMLNaut-NISPS` tree* (`nisps::`), or does it belong in the laptop `~/src/manifold` tree (`manifold::`)? This blocks the dir name, namespace, repo, and codegen identity. *Recommendation:* build here (the live tree the recon ran against; rename is trivial), but confirm.
|
||||
|
||||
2. **Fixed-2-input contract (plan §8-D) — the big one.** The foundation hard-codes the honest fixed `MLP<2,10,14,18,126>` contract: codegen **fails the build** on schema/arch mismatch and `GenericMode` clamps `setInput` to the real arch (killing today's phantom-channel OOB writes in `mode-runtime.ts` lines 224-230 and `wasm-iml` warn-and-ignore at lines 198-203). Consequence: **multi-input mic modes (XIASRI / sound_analysis_midi) are firmware-only in v1** and show a "single-input in browser" badge; the runtime-shaped MLP is deferred *behind a passing parity check, never bundled into this rewrite*. Confirm you accept this v1 contraction — it's a materially smaller browser surface than "all modes work in browser."
|
||||
|
||||
3. **Two entries vs runtime route — confirm.** I've chosen two Vite entries / one `dist` (cleanest "one engine, two skins"). If you'd rather a single bundle with a `/2.0` client route (e.g. to share an app shell), say so now — it changes the skin-mount and engine-instance model.
|
||||
|
||||
4. **Serving cutover policy.** I propose: legacy a-immersive stays at `/` untouched; new skins live at `/next/` (faithful) and `/2.0/` (Console) until you sign off, then `/`→skin A. Confirm you want the new default *staged* rather than replacing `/` immediately. (Also: do you want skin A to eventually *replace* the legacy vanilla a-immersive, or coexist?)
|
||||
|
||||
5. **`capability_class` + `tier` schema fields (plan §8-B/§8-C).** The foundation's `GenericMode` and switcher depend on these new schema fields. Confirm the four classes (Synth/Controller/Sequencer/Visual) and the per-param/diagnostic `tier: 0|1|2` disclosure model are canonical, since codegen and `EngineApi.schema` bake them in from S0.
|
||||
|
||||
**Risks carried into the build (mitigations baked into the architecture):**
|
||||
|
||||
| Risk | Mitigation |
|
||||
|---|---|
|
||||
| A skin quietly grows a second data path → live-feedback rot returns | `EngineApi` is the only seam; `no-restricted-imports` lint forbids skins importing `engine/**` internals; spine e2e runs on **both** skins in CI. |
|
||||
| Memo purity violated (side-effect creeps into a memo) | Single documented send-`createEffect`; lint/review rule; the chain is one tested file (`control-graph.ts`). |
|
||||
| WASM URL breaks under `/2.0/` sub-path | Base-aware asset resolution (`import.meta.env.BASE_URL`); the one concrete change the sub-path imposes; covered by a load-under-sub-path e2e. |
|
||||
| Shared `dist` + relative base mis-resolves worklet/`?worker&url` chunk | `base:'./'` + nginx `alias`; explicit load-both-skins e2e at S0 catches it immediately. |
|
||||
| Lifting `wasm-iml` re-imports its store-coupling debt | Adapt-on-lift: strip `mlStore.__set*` from the class; store observes the class. Enforced by the headless-engine lint (engine files may not import skin/JSX). |
|
||||
| Two builds in the deploy pipeline slow/again-unserved | One `dist`, one `vite build` (two entries); deploy script gains one block; `/2.0/` + `/next/` wired in nginx from the start so it isn't "built but unserved" like today's `playground/dist`. |
|
||||
| Parity drift on any core touch | Every engine-touching step (S2, S3, S6) gated on `parity-check.sh` green before the skin lands. |
|
||||
| Probe ships to prod | `engine/probe.ts` gated behind `?debug=1`; reads `EngineApi` only. |
|
||||
|
||||
---
|
||||
|
||||
### Relevant file paths
|
||||
- Authoritative UX/feature plan (owns the skins' detail): `/home/w1n5t0n/src/MEMLNaut-NISPS/docs/specs/plans/playground-2.0-rewrite-plan.md`
|
||||
- Design-intent reference for skin-a-immersive: `/home/w1n5t0n/src/MEMLNaut-NISPS/docs/specs/recon/playground-2026.md`
|
||||
- The god-hook to discard (the debt): `/home/w1n5t0n/src/MEMLNaut-NISPS/playground/src/modes/mode-runtime.ts`
|
||||
- Engine modules to lift: `/home/w1n5t0n/src/MEMLNaut-NISPS/playground/src/audio/engine-host.ts`, `…/audio/worklet/nisps-processor.ts`, `…/ml/wasm-iml.ts`, `…/ml/wasm-worker.ts`, `…/ml/dataset.ts`, `…/input/pipeline.ts`, `…/output/pipeline.ts`, `…/output/curves.ts`, `…/stores/bus.ts`, `…/stores/ml-store.ts`, `…/primitives/*`
|
||||
- Codegen TS types to extend (`capability_class`+`tier`): `/home/w1n5t0n/src/MEMLNaut-NISPS/playground/src/modes/generated/types.ts`
|
||||
- Serving facts: nginx `/etc/nginx/sites-available/meml.lnfinitemonkeys.org` (server-scope COOP/COEP, root `meml-aimmersive`); deploy script `/home/w1n5t0n/.config/webhooks/meml-deploy.sh` (builds `…/meml.lnfinitemonkeys.org/playground/dist`, currently unserved); webhook def `/home/w1n5t0n/.config/webhooks/hooks.json`
|
||||
- New foundation tree to create: `/home/w1n5t0n/src/MEMLNaut-NISPS/playground2/` (or the laptop `manifold` tree, pending open question 1)
|
||||
## 2. The reactive spine
|
||||
|
||||
There is exactly one path from gesture to sound, every consumer reads it, and a desync is a
|
||||
failing test rather than a recurring prod bug.
|
||||
|
||||
### 2.1 The derivation path (`manifold/src/engine/spine.ts`)
|
||||
|
||||
The spine is an external store living BELOW React (deliberately not on the render scheduler): the
|
||||
`setInput` action derives processed → ml → routed **eagerly and synchronously** (the input
|
||||
pipeline, inference, and output pipeline are all C++/WASM since one-core-engine P4) and fires the
|
||||
single `backend.send` at the action's tail, off-render. Contract rules:
|
||||
|
||||
- **One side-effect.** `backend.send(routed)` at the tail of `setInput` is the only transport
|
||||
call in the derivation path.
|
||||
- **No per-frame allocation.** Buffers are reused (`routedBuf` threaded through the output
|
||||
pipeline and handed to the backend).
|
||||
- **Weights mutate only through engine actions** (train / feedback / reshape), each of which bumps
|
||||
the spine's version counter — there is no write path that can update audio without notifying
|
||||
the UI.
|
||||
- **React subscribes via `useSyncExternalStore(subscribe, version)`** — the version counter, not
|
||||
the arrays; canvases read the live `Float32Array`s imperatively in rAF and never re-render per
|
||||
frame.
|
||||
|
||||
### 2.2 The live-feedback guarantee and its e2e assertion
|
||||
|
||||
**Guarantee:** any change to the input (gesture) or the weights (train / feedback) propagates — in
|
||||
the same synchronous action — to the routed output, which the single effect sends to the backend
|
||||
and which every UI consumer reads.
|
||||
|
||||
The invariant is asserted in CI by `manifold/tests/e2e/spine.spec.ts` via the synchronous
|
||||
`window.__nisps` probe: set inputs, read outputs, expect change (plus probe-survives-mode-switch).
|
||||
Because the probe reads `EngineApi` only, it exercises the exact path the UI uses.
|
||||
|
|
|
|||
|
|
@ -2,12 +2,38 @@
|
|||
kind: spec
|
||||
stability: evolving
|
||||
layer: binding
|
||||
counterpart: aimmersive-clone-spec.md
|
||||
---
|
||||
|
||||
# Inputs Spec — Modular Input Layer (Workstream F)
|
||||
|
||||
*Status: implementation-ready spec for the `manifold/` React app. Scope: the modular input layer — sources (XY pad, MIDI input, gamepad single/double-stick), how they compose into an N-dimensional input vector, the MLP-rebuild-on-input-change mechanism, the binding to the reactive spine, and the dock INPUTS panel. Read alongside `recon/findings-engine-surface.md` (the fixed-2-input gap), `engine-architecture.md` (the `EngineApi` seam — note that doc says SolidJS/`playground2`; this app is React/`manifold/`), and `aimmersive-clone-spec.md` (existing gamepad/MIDI/joystick behaviour). British spelling in product copy. The built-in synth is always shown as the "Powerful Synth Engine" — never "C15".*
|
||||
*Scope: the modular input layer of the `manifold/` React app — sources (XY pad, MIDI input,
|
||||
gamepad single/double-stick), how they compose into an N-dimensional input vector, the
|
||||
MLP-reshape-on-input-change mechanism, the binding to the reactive spine, and the dock INPUTS
|
||||
panel. British spelling in product copy. The built-in synth is always shown as the "Powerful
|
||||
Synth Engine" — never "C15".*
|
||||
|
||||
> **Grounding note (2026-07-21).** This spec was written 2026-06-28 and implemented; the body is
|
||||
> kept for the behavioural intent, but treat it as follows:
|
||||
>
|
||||
> - **Every `file:line` cite below is historical grounding, not a live pointer.** `playground/*`
|
||||
> files died with the retired SolidJS playground (branch `archive/playground-solidjs`);
|
||||
> `aimmersive-clone-spec.md` is archived at `_archive/aimmersive-clone-spec.md`;
|
||||
> `engine-architecture.md` was trimmed 2026-07 so its cited line numbers no longer exist;
|
||||
> `findings-*` recon docs are immutable snapshots of the pre-refactor world.
|
||||
> - **What shipped** lives in `manifold/src/inputs/` (not the proposed `engine/input/`):
|
||||
> `input-layer.ts` (single rAF compose loop), `xy-pad-source` / `gamepad-source` /
|
||||
> `midi-input-source`, `useInputLayer.ts` React binding. See `MAP.md` §manifold.
|
||||
> - **§4's option analysis was overtaken by events.** The recommendation (Option B, multiple
|
||||
> fixed-shape WASM modules) never shipped: one-core-engine P2 made the ONE parity-tested core
|
||||
> runtime-shaped in the browser (`MLPCore<DynamicStorage>`, `nisps_ml_reshape` warm-start) — so
|
||||
> honest N-dim nets landed via Option A's shape *without* the divergent-second-core cost that
|
||||
> §4 rejected it for. The shipped layer feeds axes into dedicated input slots of an
|
||||
> over-provisioned 32-input head and offers a warm-started reshape when the active axis count
|
||||
> changes (reset-on-reshape modal).
|
||||
> - **Source composition**: the engine layer composes multiple sources, but the dock currently
|
||||
> exposes an exclusive one-at-a-time picker (`InputMode` = internal | gamepad | midi).
|
||||
> Mix-and-match UI is an unreversed decision with groundwork deliberately laid — operator
|
||||
> decision pending (`plans/simplification-plan.md` §7.7).
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
|
|
@ -1,13 +1,15 @@
|
|||
---
|
||||
kind: plan
|
||||
status: active
|
||||
status: executed
|
||||
---
|
||||
|
||||
# Manifold — Build Plan & Locked Decisions (resume anchor)
|
||||
|
||||
*Dated 2026-06-27. This is the single resume anchor for the Manifold convertible-app build. Read this + the
|
||||
specs it points to before continuing. Mission: one working browser instrument putting the NEW Manifold
|
||||
"convertible" front-end on top of the REAL parity-tested ML+audio engine.*
|
||||
*Dated 2026-06-27. **Executed** — the Manifold app was built, deployed to `/next/`, and is the sole
|
||||
browser app; current ground truth is `MAP.md` §manifold + `manifold/ONBOARDING.md` (the manual
|
||||
redeploy recipe below is superseded by the CI-gated push-to-main webhook). What survives from here:
|
||||
the VPS non-snap-node Playwright invocation (now in `MAP.md` Entry points) and the locked
|
||||
naming/copy decisions (never "C15", British spelling).*
|
||||
|
||||
## Locked decisions (operator-confirmed 2026-06-27)
|
||||
| Decision | Choice |
|
||||
|
|
@ -32,8 +34,8 @@ specs it points to before continuing. Mission: one working browser instrument pu
|
|||
- `inputs-spec.md` — modular XY/WebMIDI/gamepad; multiple-WASM-module reshape with warm-start.
|
||||
- `recon/findings-{feedback-behaviour,engine-surface,design-and-manifold}.md` — Phase-1 audits.
|
||||
- `recon/upstream-firmware-survey.md` — git topology; latest InterfaceRL = `0a541cc`.
|
||||
- `plans/playground-2.0-rewrite-plan.md`, `engine-architecture.md`, `aimmersive-clone-spec.md`,
|
||||
`feedback-modes-port-spec.md` — the prior planning corpus.
|
||||
- `plans/playground-2.0-rewrite-plan.md`, `engine-architecture.md`, `_archive/aimmersive-clone-spec.md`,
|
||||
`plans/feedback-modes-port-spec.md` — the prior planning corpus.
|
||||
|
||||
## Reactive spine → React (load-bearing, from findings-design-and-manifold.md §4)
|
||||
Spine lives BELOW React in an external store: `setInput` action derives processed→ml→routed eagerly+synchronously
|
||||
|
|
|
|||
|
|
@ -1,13 +1,18 @@
|
|||
---
|
||||
kind: spec
|
||||
stability: evolving
|
||||
layer: cross-cutting
|
||||
kind: plan
|
||||
status: executed
|
||||
---
|
||||
|
||||
I have complete grounding now. I'll write the implementation-ready spec.
|
||||
|
||||
# Implementation Spec — "Down Action" Negative-Feedback Feature in the `nisps/` Core
|
||||
|
||||
**Executed 2026-06/07; not authority for current behaviour.** The port landed, then evolved past
|
||||
this document: the surviving truth is `docs/adr/rl-feedback-design.md` (which explicitly
|
||||
supersedes §2.5/§7's AVOID-out-of-scope decision — the geometric push WAS ported, via
|
||||
`nisps/ml/geo_push.hpp` + `nisps/ml/replay.hpp`) plus the code itself, `nisps/ml/feedback.hpp`
|
||||
(now `FeedbackControllerCore<FbStorage>`, storage-policied, with ExploreAndPlace added). The
|
||||
`playground/` paths below refer to the retired SolidJS app (branch `archive/playground-solidjs`);
|
||||
the browser driver is now `manifold/src/feedback/controller.ts`.
|
||||
|
||||
## 0. Provenance
|
||||
|
||||
Every behavior below is read directly from local branch `fork-feedback` of the memllib submodule at `/home/w1n5t0n/src/MEMLNaut-NISPS-upstream/`:
|
||||
|
|
@ -1,38 +1,45 @@
|
|||
---
|
||||
kind: spec
|
||||
stability: aspirational
|
||||
layer: behavioural
|
||||
kind: plan
|
||||
status: active
|
||||
---
|
||||
|
||||
# Manifold Parity Features — Session Presets · Pins · Jolt · OU-Explore · Control Surface
|
||||
|
||||
*Status: draft — awaiting review. Date: 2026-07-12.*
|
||||
*Scope: prescriptive spec for porting five playground features into `manifold/`. Spec only — no
|
||||
implementation is authorised by this document; break into ergo issues after sign-off.*
|
||||
*Date: 2026-07-12; status updated 2026-07-21. Prescriptive plan for porting five playground
|
||||
features into `manifold/`. Now part-executed:*
|
||||
|
||||
**Traces up to:** `playground-2.0-rewrite-plan.md` §2.3 (Feel drawer), §2.6 (pins), §3.8
|
||||
(composed-layers presets); `engine-architecture.md` §3.2 (fanout, control-point tri-state), §3.8
|
||||
(persist helper); `docs/specs/slp-workshop-firmware.md` §3–4 (Jolt / OU-explore, Part I shipped in
|
||||
firmware); `dock-spec.md` (drawer depths, tri-state semantics); BUILD-PLAN locked decisions
|
||||
- ***§3 Jolt and §4 OU-explore: EXECUTED by other means*** *— one-core-engine P3 landed the core
|
||||
bindings (this document's own preferred route, §0 principle 1). The shipped C ABI is
|
||||
`nisps_ml_jolt_{press,step,release,active}` and `nisps_ml_explore_{intensity,get_intensity,apply}`
|
||||
— not the §3.2/§4.2 names proposed below — driven by `manifold/src/engine/exploration.ts`.
|
||||
§3/§4 are kept as behavioural reference; the code wins.*
|
||||
- ***§1 session presets, §2 pins, §5 control surface: NOT implemented.*** *These remain the live
|
||||
prescription, feeding the curated/advanced-split work (`simplification-plan.md` §6.5c).*
|
||||
|
||||
**Traces up to:** `playground-2.0-rewrite-plan.md` (superseded plan — its target died but these
|
||||
sections are still the cited design source) §2.3 (Feel drawer), §2.6 (pins), §3.2 (fanout,
|
||||
control-point tri-state), §3.8 (composed-layers presets, persist helper);
|
||||
`../slp-workshop-firmware.md` §3–4 (Jolt / OU-explore, Part I shipped in firmware);
|
||||
`../dock-spec.md` (drawer depths, tri-state semantics); BUILD-PLAN locked decisions
|
||||
(parity-tested engine; British spelling; the synth is "Powerful Synth Engine", never the forbidden
|
||||
string).
|
||||
|
||||
**Reference implementations:** the playground versions are the behavioural ground truth for
|
||||
constants and algorithms — `playground/src/features/session-preset.ts`,
|
||||
`stores/session-store.ts` (pins), `ml/jolt.ts`, `output/ou-explore.ts`, `stores/control-store.ts` +
|
||||
`features/control-routing.ts`. Where this spec and the playground disagree, this spec wins (each
|
||||
divergence is called out and justified inline).
|
||||
constants and algorithms — `src/features/session-preset.ts`, `src/stores/session-store.ts` (pins),
|
||||
`src/ml/jolt.ts`, `src/output/ou-explore.ts`, `src/stores/control-store.ts` +
|
||||
`src/features/control-routing.ts`, all on the retired-playground archive (branch
|
||||
`archive/playground-solidjs`, tag `playground-solidjs-final` — these files no longer exist on
|
||||
main). Where this spec and the playground disagree, this spec wins (each divergence is called out
|
||||
and justified inline).
|
||||
|
||||
---
|
||||
|
||||
## 0. Principles applied throughout
|
||||
|
||||
1. **C++ owns gesture math where a C++ class exists.** `nisps::ml::Jolt` and
|
||||
`nisps::ml::OUNoise<N>` already exist (`nisps/ml/jolt.hpp`, `nisps/ml/ou_noise.hpp`) but are
|
||||
not exposed to WASM. Manifold binds them rather than re-porting to TS — this closes two
|
||||
documented `--- C++ GAP ---` items and buys deterministic, firmware-parity noise for free.
|
||||
(The playground's TS reimplementations used `Math.random()`; that shortcut is *not* carried
|
||||
over.)
|
||||
`nisps::ml::OUNoise<N>` (`nisps/ml/jolt.hpp`, `nisps/ml/ou_noise.hpp`) are bound to WASM
|
||||
rather than re-ported to TS — deterministic, firmware-parity noise for free. *(Done — this
|
||||
landed via one-core-engine P3; see the status block above.)*
|
||||
2. **No new mechanisms where an existing one already expresses the idea.** Param pins are the
|
||||
existing `off|fixed|live` tri-state + arm mask, not a parallel pin system (§2.2).
|
||||
3. **Transparent defaults.** Every default constant appears in this spec as a number with its
|
||||
|
|
@ -40,7 +47,7 @@ divergence is called out and justified inline).
|
|||
the user should never wonder what an axis is secretly doing.
|
||||
4. **One persistence pattern.** All new persisted state goes through a single versioned
|
||||
`persist<T>(key, version, migrate)` helper modelled on `settings-store.ts`
|
||||
(localStorage, debounced 200 ms), per `engine-architecture.md` §3.8.
|
||||
(localStorage, debounced 200 ms), per `playground-2.0-rewrite-plan.md` §3.8.
|
||||
5. **Probe parity.** Each feature activates its currently-inert `window.__nisps` methods
|
||||
(`manifold/src/debug/probe.ts`) so Playwright can drive it headlessly. New e2e specs are part
|
||||
of each feature's acceptance criteria.
|
||||
|
|
@ -51,8 +58,8 @@ divergence is called out and justified inline).
|
|||
|
||||
### 1.1 Model
|
||||
|
||||
Adopt the composed-layers model prescribed by `engine-architecture.md` §3.8 rather than the
|
||||
playground's monolithic `SessionPresetPayload`. A preset is a bundle of independently optional
|
||||
Adopt the composed-layers model prescribed by `playground-2.0-rewrite-plan.md` §3.8 rather than
|
||||
the playground's monolithic `SessionPresetPayload`. A preset is a bundle of independently optional
|
||||
layers:
|
||||
|
||||
```ts
|
||||
|
|
@ -222,7 +229,7 @@ The Jolt instance lives beside the MLP handle inside the WASM module (one per ne
|
|||
the session seed so runs are reproducible. Parity CI: extend `tests/cpp/parity_check.cpp` with a
|
||||
golden jolt sequence (seed → press → k steps → weight vector) asserted native-vs-WASM.
|
||||
|
||||
### 3.3 Constants (defaults — from `nisps/ml/jolt.hpp:42–49`, upstream-verbatim)
|
||||
### 3.3 Constants (defaults — from `nisps/ml/jolt.hpp` `JoltParams`, upstream-verbatim)
|
||||
|
||||
| Constant | Value | Meaning |
|
||||
|---|---|---|
|
||||
|
|
@ -402,7 +409,7 @@ non-zero offset.
|
|||
|
||||
### 5.5 Fanout (architecture requirement)
|
||||
|
||||
Per `engine-architecture.md` §3.2: resolution is a memoised derivation per target param — in
|
||||
Per `playground-2.0-rewrite-plan.md` §3.2: resolution is a memoised derivation per target param — in
|
||||
React terms one `useMemo` producing the resolved record + an effect per target store that
|
||||
writes only on change. The playground's `JSON.stringify`-signature-inside-effect pattern
|
||||
(`control-routing.ts:34`) is explicitly **not** ported. Resolution runs off the render cycle for
|
||||
|
|
@ -1,11 +1,11 @@
|
|||
---
|
||||
kind: plan
|
||||
status: executed
|
||||
status: superseded
|
||||
---
|
||||
|
||||
# NISPS Playground 2.0 — Clean-Room SolidJS Rewrite Plan
|
||||
|
||||
*Status: sign-off-ready. Author: lead architect, synthesizing four designer proposals (P1–P4) against three judge panels (mission-fit, feasibility, UX coherence). Date: 2026-06-17. Executed: largely implemented in the Manifold convertible app (June 2026) + playground SolidJS foundation.*
|
||||
*Author: lead architect, synthesizing four designer proposals (P1–P4) against three judge panels (mission-fit, feasibility, UX coherence). Date: 2026-06-17. **Superseded**: its target — the SolidJS playground — was retired 2026-07-13 (branch `archive/playground-solidjs`, tag `playground-solidjs-final`). Many of its ideas (Console interaction model, reactive spine, right-edge dock, off/fixed/live tri-state, schema-driven modes) shipped instead in the Manifold React app; see `BUILD-PLAN.md` (executed) and `MAP.md` §manifold.*
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
|
|
@ -7,20 +7,30 @@ counterpart: backends-spec.md
|
|||
|
||||
# MEMLNaut VCV Rack Module — Specification
|
||||
|
||||
**Revision note:** 2026-06-28 BUILD DELTAS folded into body (2026-07-13). The module now implements 8→16 I/O with per-output LED rings, design-token palette, and WS↔OSC bridge per the Manifold mission and BUILD-PLAN locked decisions. Prior design (2→12) sections retained for threading/persistence/RL workflow reference.
|
||||
**Revision note:** pruned 2026-07-21 to the current 8→16 contract. The 2→12-era design
|
||||
sections (phase plan, per-node cost tables, nisps-core prerequisites, dead OSC verbs) are in git
|
||||
history. This file is also the single `.nisps`/patch format spec — the separate
|
||||
`vcv/NISPS-FORMAT.md` (which documented the retired v1 nested-weights format) was deleted with
|
||||
this prune.
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
A VCV Rack module that embeds the NISPS interactive ML engine as a CV-to-CV mapper. Users explore high-dimensional parameter spaces via reinforcement learning feedback, producing 16 raw CV outputs (each with a custom LED ring indicator) from 8 CV inputs.
|
||||
A VCV Rack module that embeds the NISPS interactive ML engine as a CV-to-CV mapper. Users explore
|
||||
high-dimensional parameter spaces via reinforcement-learning feedback, producing **16 raw CV
|
||||
outputs** (each with a custom LED-ring indicator) from **8 CV inputs**.
|
||||
|
||||
The module does **not** produce sound. It maps input CVs through a trained neural network to output CVs, which the user patches into other modules (VCOs, VCFs, VCAs, etc.). The result: a learned, nonlinear, high-dimensional modulation source shaped by the user's aesthetic preferences. The 8→16 architecture enables the module to serve as a general-purpose modulation transformer for complex polyphonic and sequencer-driven patches.
|
||||
The module does **not** produce sound. It maps input CVs through a trained neural network to
|
||||
output CVs, which the user patches into other modules (VCOs, VCFs, VCAs, etc.). The result: a
|
||||
learned, nonlinear, high-dimensional modulation source shaped by the user's aesthetic preferences.
|
||||
|
||||
**Plugin name:** MEMLNaut
|
||||
**Module name:** MEMLNaut (initially single module, future modules possible)
|
||||
**License:** Undecided. Note: nisps-core is MPL-2.0 (file-level copyleft — MPL files must remain open, but wrapper code can be any license). VCV SDK is GPLv3 — linking against it effectively makes the combined binary GPL. Will not be submitted to VCV Library initially.
|
||||
**Target:** VCV Rack 2 (primary, both Community Edition [free] and Pro), v1 compatibility where feasible
|
||||
**Plugin name:** MEMLNaut (`vcv/plugin.json`, currently v0.2.0, license field "proprietary")
|
||||
**Target:** VCV Rack 2 only (see the v2-only decision at the end)
|
||||
**Distribution:** not submitted to the VCV Library; `.vcvplugin` bundles published at
|
||||
`meml.lnfinitemonkeys.org/next/vcv/` (see `vcv/DISTRIBUTION.md`; local builds land in the
|
||||
git-ignored `vcv/dist/`). Note: VCV SDK is GPLv3 — linking against it effectively makes the
|
||||
combined binary GPL.
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -37,9 +47,9 @@ The module does **not** produce sound. It maps input CVs through a trained neura
|
|||
│ Reads CV inputs, writes CV outputs │
|
||||
│ Decimated inference trigger │
|
||||
├─────────────────────────────────────┤
|
||||
│ nisps-core (C++20) │
|
||||
│ IML → MLP → forward inference │
|
||||
│ Background thread: training │
|
||||
│ src/iml.hpp — thin adapter over │
|
||||
│ nisps::ml::MLPCore<DynamicStorage> │
|
||||
│ (the shared C++20 core in nisps/) │
|
||||
├─────────────────────────────────────┤
|
||||
│ State Manager │
|
||||
│ Serialize/deserialize weights, │
|
||||
|
|
@ -47,525 +57,265 @@ The module does **not** produce sound. It maps input CVs through a trained neura
|
|||
└─────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Threading Model
|
||||
|
||||
- **Audio thread** (`process()`): Reads input CVs, runs MLP inference (decimated), writes output CVs. Never blocks.
|
||||
- **Background thread**: Handles training (SGD/RMSProp). On completion, atomically swaps weight buffer into the inference path. Also computes novelty/confidence maps post-training.
|
||||
- **Widget thread**: Draws UI, handles user interaction (buttons, knobs). Reads output values for display.
|
||||
|
||||
**Weight double-buffering**: Two separate MLP instances are maintained (nisps-core is not thread-safe — no locks, public `m_layers`, `std::mt19937` without synchronization). The audio thread reads from MLP-A while the training thread clones weights into MLP-B, trains MLP-B, then signals completion. An `std::atomic<bool>` flag tells the audio thread to swap. Memory cost is ~2x network weights (~20KB for the default architecture — trivial).
|
||||
|
||||
**Threading invariant**: Only the background thread ever writes to an MLP instance. This applies to both training (thumbs-up) and weight perturbation (thumbs-down). Both operations are enqueued as jobs for the background thread, which clones → mutates → signals swap. Thumbs-down adds ~1-5ms latency vs. direct mutation, but maintains the single-writer invariant and eliminates data races.
|
||||
|
||||
**Rapid feedback queueing**: If the user taps +/− while a training/perturbation job is in progress, incoming examples are buffered into a pending list. When the current job completes, if pending work exists, a new job starts immediately with the full (now-updated) dataset. Maximum queue depth of 1 — latest pending state wins, intermediate states are coalesced.
|
||||
|
||||
**Thread lifecycle**: Background thread checks an `std::atomic<bool> shouldStop` flag each training iteration. On module destruction, set flag and join with a timeout (~100ms). If training doesn't finish in time, the thread is detached (VCV can't hang waiting for a stuck training loop). Each module instance owns its own background thread — no shared thread pool (simplicity over efficiency; revisit if profiling shows thread overhead with many instances).
|
||||
|
||||
**Multiple instances**: Each module instance is fully independent (own IML, own background thread, own state). 4 instances = 4 threads + ~80KB weight memory — negligible. The OSC server (Phase 8) needs per-instance port assignment to avoid conflicts.
|
||||
|
||||
**Post-swap output crossfade**: When weights are swapped, outputs may jump discontinuously. A configurable slew parameter (default ~10ms) linearly interpolates between old and new output vectors over a short crossfade window to prevent audible clicks in downstream audio. Accessible via right-click context menu.
|
||||
|
||||
### Input Signal Handling
|
||||
|
||||
- **Polyphonic inputs**: Channel 0 only (monophonic). Extra channels are ignored. Standard behavior for non-polyphonic module designs.
|
||||
- **Input clipping**: All CV inputs are hard-clamped to their expected range before normalization. For 0–10V mode: clamp to [0, 10V]. For ±5V mode: clamp to [-5V, +5V]. Out-of-range signals are silently clipped.
|
||||
|
||||
---
|
||||
|
||||
## I/O Specification
|
||||
|
||||
### Inputs (Fixed: 8 CV inputs + control ports)
|
||||
|
||||
The module has **8 fixed CV inputs** feeding the MLP's input layer (no runtime reconfiguration of input count).
|
||||
|
||||
| Port | Default Label | Notes |
|
||||
|------|---------------|-------|
|
||||
| IN 1–8 | IN 1–8 | CV inputs feeding the 8-input MLP. Each input jack is visible on the panel. |
|
||||
| SPREAD CV | Spread | CV modulation of SPREAD knob (attenuated, added to knob value) |
|
||||
| LEARN | Learn | Gate input: when high, RL feedback is accepted |
|
||||
| + TRIG | Positive | Trigger input: register thumbs-up |
|
||||
| − TRIG | Negative | Trigger input: register thumbs-down |
|
||||
|
||||
- All CV inputs normalized to [0, 1] internally. Per-input range configuration via context menu:
|
||||
- **Unipolar (0–10V)**: default. Clamp to [0, 10V], divide by 10. Good for envelopes, sequencers.
|
||||
- **Bipolar (±5V)**: Clamp to [-5, +5V], add 5, divide by 10. Good for LFOs, oscillators.
|
||||
- LEARN gate has a corresponding panel toggle button (either/or — gate OR button enables learning)
|
||||
- +/− triggers work only when LEARN is enabled (gate high OR toggle on)
|
||||
|
||||
### Outputs (16 raw + optional derived)
|
||||
|
||||
| Port | Type | Description |
|
||||
|------|------|-------------|
|
||||
| OUT 1–16 | Raw MLP | Direct MLP output activations, scaled to configured CV range. Each jack is surrounded by a custom LED ring (see Visual Feedback). |
|
||||
| MEAN, STD, DELTA, NOVELTY, CONFIDENCE | Derived | Available on the expander module or via context-menu toggle (hidden by default to keep the main panel clean). See prior design section for semantics. |
|
||||
|
||||
Each output has:
|
||||
- Per-output range configuration (0–10V unipolar or ±5V bipolar) via context menu
|
||||
- Small attenuverter knob for fine-tuning range/polarity
|
||||
|
||||
### Panel Controls
|
||||
|
||||
| Control | Type | Description |
|
||||
|---------|------|-------------|
|
||||
| SPREAD | Knob | Controls weight init scale, RL noise scaling, weight decay (see webapp spec) |
|
||||
| RATE | Knob | Inference rate: from block-rate (~170Hz) to audio-rate (44.1kHz) |
|
||||
| + | Momentary button | Thumbs up (register positive RL feedback) |
|
||||
| − | Momentary button | Thumbs down (register negative RL feedback) |
|
||||
| LEARN | Toggle button + LED | Enable/disable learning (mirrors LEARN gate input) |
|
||||
| RAND | Momentary button | Randomize network weights |
|
||||
| CLEAR | Momentary button (long-press) | Clear all examples and reset network |
|
||||
|
||||
### Advanced Controls (Right-Click Context Menu)
|
||||
|
||||
| Setting | Description |
|
||||
|---------|-------------|
|
||||
| Input count | Number of CV inputs (2–8). **Warning: changing rebuilds MLP and clears all state.** |
|
||||
| Per-input range | Unipolar (0–10V, default) or Bipolar (±5V) for each CV input |
|
||||
| Noise level | Manual override for RL exploration noise (default: auto from spread) |
|
||||
| Decay rate | Weight decay per RL step (default: auto from spread) |
|
||||
| Learning rate | MLP training learning rate |
|
||||
| Max iterations | Training iteration cap |
|
||||
| Per-output range | Unipolar (0–10V) or Bipolar (±5V) for each output |
|
||||
| Output slew | Post-training crossfade time in ms (default: 10ms, range: 0–100ms) |
|
||||
|
||||
---
|
||||
|
||||
## Visual Feedback
|
||||
|
||||
### Per-Output LED Rings
|
||||
|
||||
Each of the 16 output jacks is surrounded by a **custom LED ring** (SVG + NanoVG on `drawLayer()` layer 1). The ring arc fills proportionally to the output's current value (0 → 0V, 1 → full arc = 2π). The track ring is dimly visible always; the fill is bright and **color-coded by parameter group** to match the frontend design-token palette:
|
||||
|
||||
- **Formant group** → `--accent` (#ff6a00, orange)
|
||||
- **Pitch group** → `--accent-2` (#00ccff, cyan)
|
||||
- **Amplitude group** → `--good` (#6bc26b, green)
|
||||
- **Filter group** → `--warn` (#f5c45e, yellow)
|
||||
- **Effects group** → `--info` (#5b9eef, blue)
|
||||
- **Modulation group** → `--accent-3` (#ffa860, light orange)
|
||||
|
||||
If the 16 outputs don't have a pre-assigned group mapping, use a smooth 16-step ramp interpolating between orange and cyan across the jacks left-to-right. The palette is defined in a minimal `vcv/src/palette.hpp` (hand-written, no build-time codegen required).
|
||||
|
||||
### Control Panel Display
|
||||
|
||||
A small real-time display area (if space permits on the panel) showing:
|
||||
- Training state indicator (idle / training / converged)
|
||||
- Example count
|
||||
- Current noise level
|
||||
- Current spread value
|
||||
|
||||
### Additional LEDs
|
||||
|
||||
- LEARN LED (green when active)
|
||||
- Training activity LED (flashes during training)
|
||||
|
||||
---
|
||||
|
||||
## MLP Configuration
|
||||
|
||||
### Fixed Network Architecture
|
||||
|
||||
```
|
||||
Inputs: 8 (+ bias = 9 input nodes)
|
||||
Hidden: [24, 32, 16] (3 hidden layers, ReLU activation)
|
||||
Output: 16 (sigmoid activation, maps to [0, 1])
|
||||
```
|
||||
|
||||
The architecture is **fixed at compile time**; no runtime reconfiguration of input/output layer sizes. The hidden layer widths are appropriate for 8 inputs → 16 outputs and real-time inference constraints (well under 1% CPU on any modern machine at typical inference rates).
|
||||
|
||||
### Core Library Integration
|
||||
|
||||
**Delta #5 CLOSED (2026-07-18, one-core-engine-refactor P6).** The module no longer
|
||||
vendors its own MLP. `vcv/src/iml.hpp` is now a THIN, Rack-free adapter over the shared
|
||||
vendors its own MLP. `vcv/src/iml.hpp` is a THIN, Rack-free adapter over the shared
|
||||
core: `nisps::ml::MLPCore<nisps::ml::DynamicStorage>` (the runtime-shaped branch of the one
|
||||
core MLP — fixed 4-layer ReLU×3 + Sigmoid topology, three runtime hidden sizes) with the
|
||||
module's real `[16, 24, 16]` shape, `nisps::Rng` (nisps/core/rng.hpp) replacing the vendored
|
||||
`DetRng`, and the core MLP's own FIFO dataset replacing the vendored `Dataset`. The adapter
|
||||
includes only nisps headers + the standard library (no Rack includes) so the host ctest can
|
||||
compile it. Behaviour therefore **changed** from the vendored *approximation* of firmware/
|
||||
browser training semantics to **core-exact**: weight init, RL `move_weights`, SGD training,
|
||||
activations and RNG are now bit-identical to the firmware/WASM engine — pinned by
|
||||
compile it. Behaviour is **core-exact**: weight init, RL `move_weights`, SGD training,
|
||||
activations and RNG are bit-identical to the firmware/WASM engine — pinned by
|
||||
`tests/cpp/test_vcv_iml_parity.cpp` (adapter == bare `MLPCore<DynamicStorage>`, memcmp-equal).
|
||||
Persisted weights are now the core's FLAT `[weights…][biases…]` vector (patch `version` = 3);
|
||||
old 3D vendored weight blobs will not load.
|
||||
|
||||
### Threading Model
|
||||
|
||||
- **Audio thread** (`process()`): reads input CVs, runs MLP inference (decimated), writes output
|
||||
CVs. Never blocks.
|
||||
- **Worker thread**: handles training (thumbs-up) and weight perturbation (thumbs-down). On
|
||||
completion, signals the audio thread to swap in the new weights and crossfade.
|
||||
- **Widget thread**: draws UI, reads output values for display.
|
||||
|
||||
**Threading invariant** (documented at `vcv/src/MEMLNaut.cpp` `iml`/`imlShadow`): only the audio
|
||||
thread touches `iml`; the worker thread operates exclusively on `imlShadow`. Hand-off is through
|
||||
atomic-flagged staging buffers (`pendingWeights`, staged example copies) — the worker deep-copies
|
||||
example vectors into staging before flagging, and the audio thread never reads `imlShadow`
|
||||
directly. Feedback taps arriving mid-job are coalesced (latest pending state wins).
|
||||
|
||||
**Post-swap output crossfade**: when weights are swapped, outputs may jump discontinuously. The
|
||||
context-menu slew setting (default 10 ms) crossfades old→new output vectors to prevent clicks.
|
||||
|
||||
**Multiple instances**: each module instance is fully independent (own adapter instances, own
|
||||
worker thread, own OSC port = 7001 + instance-id % 64).
|
||||
|
||||
### Input Signal Handling
|
||||
|
||||
- **Polyphonic inputs**: channel 0 only (monophonic).
|
||||
- **Input clipping**: CV inputs are hard-clamped to their configured range before normalisation
|
||||
to [0, 1]. Unipolar (default): clamp [0, 10 V], ÷10. Bipolar: clamp [−5, +5 V], +5, ÷10.
|
||||
|
||||
---
|
||||
|
||||
## I/O Specification
|
||||
|
||||
### Inputs (8 CV inputs + control ports)
|
||||
|
||||
| Port | Description |
|
||||
|------|-------------|
|
||||
| IN 1–8 | CV inputs feeding the 8-input MLP |
|
||||
| SPREAD CV | CV modulation of SPREAD knob (added to knob value, /10 V) |
|
||||
| LEARN | Gate input: high accepts RL feedback (OR'd with the LEARN toggle) |
|
||||
| + TRIG | Trigger input: thumbs-up |
|
||||
| − TRIG | Trigger input: thumbs-down |
|
||||
|
||||
### Outputs
|
||||
|
||||
| Port | Description |
|
||||
|------|-------------|
|
||||
| OUT 1–16 | Raw MLP outputs scaled to the configured CV range; each jack has an LED ring and an attenuverter trimpot (−100%..+100%, default +100%) |
|
||||
|
||||
There are **no derived-output jacks** in the 8→16 module (the 2→12-era MEAN/STD/DELTA/NOVELTY/
|
||||
CONFIDENCE jacks were dropped in the redesign). A context-menu toggle "Compute derived stats
|
||||
(Mean/Std/Delta)" computes the stats on the audio thread, and a novelty distance is cached after
|
||||
training — but as of 2026-07-21 **no jack, display, or OSC verb consumes these values** (see Open
|
||||
Questions).
|
||||
|
||||
### Panel Controls
|
||||
|
||||
| Control | Type | Description |
|
||||
|---------|------|-------------|
|
||||
| SPREAD | Knob (default 60%) | Weight-init scale, RL noise cap, weight decay |
|
||||
| RATE | Knob | Inference rate: block-rate (~170 Hz) … audio-rate (44.1 kHz) |
|
||||
| + | Momentary | Thumbs up (requires LEARN) |
|
||||
| − | Momentary | Thumbs down (requires LEARN) |
|
||||
| LEARN | Toggle + green LED | Enable/disable learning; inference always runs |
|
||||
| RAND | Momentary | Randomise network weights (uses SPREAD) |
|
||||
| CLEAR | Momentary (long-press ~1 s) | Clear all examples and reset network |
|
||||
|
||||
A yellow TRAIN LED flashes during worker-thread training. The display strip draws the 16 output
|
||||
bars (palette-coloured), the noise level (`N:`), the example count (`n/100`), and `TRAIN` while
|
||||
training.
|
||||
|
||||
### Context Menu (the real, current one)
|
||||
|
||||
| Setting | Description |
|
||||
|---------|-------------|
|
||||
| Output ranges | Per-output Unipolar (0–10 V, default) / Bipolar (±5 V) |
|
||||
| Input ranges | Per-input Unipolar (0–10 V, default) / Bipolar (±5 V) |
|
||||
| Compute derived stats (Mean/Std/Delta) | Toggle; see note above — currently unconsumed |
|
||||
| Output slew | 0 / 5 / 10 (default) / 20 / 50 / 100 ms |
|
||||
| Presets (.nisps) | Save / Load `.nisps` preset files (format below) |
|
||||
| Browser bridge (WS↔OSC) | Enable OSC server; listen port ∈ {7001, 7002, 7003, 9000, 9001} |
|
||||
|
||||
---
|
||||
|
||||
## Visual Feedback — Per-Output LED Rings
|
||||
|
||||
Each of the 16 output jacks is surrounded by a custom LED ring (`vcv/src/LedRing.hpp`, NanoVG on
|
||||
`drawLayer()` layer 1). The ring arc fills proportionally to the output's current value; the
|
||||
track ring is dimly visible always. Colours come from `vcv/src/palette.hpp` (hand-written from
|
||||
the frontend design tokens — `--accent` #ff6a00 etc.): the 16 outputs read as a clean
|
||||
orange→cyan-anchored ramp across the jacks.
|
||||
|
||||
---
|
||||
|
||||
## MLP Configuration
|
||||
|
||||
```
|
||||
Inputs: 8
|
||||
Hidden: [16, 24, 16] (3 hidden layers, ReLU)
|
||||
Output: 16 (sigmoid, values in [0, 1])
|
||||
```
|
||||
|
||||
`vcv/src/MEMLNaut.cpp`: `NUM_ML_INPUTS = 8`, `NUM_ML_OUTPUTS = 16`,
|
||||
`nisps::IML<float> iml{NUM_ML_INPUTS, NUM_ML_OUTPUTS, {16, 24, 16}}`. The topology is fixed; the
|
||||
core stores biases explicitly (no phantom "+1 bias node" in the layer counts).
|
||||
|
||||
### Spread Parameter
|
||||
|
||||
Identical behavior to webapp (see CLAUDE.md for full spec):
|
||||
Identical to the browser/firmware engine (the core methods):
|
||||
|
||||
1. **Weight initialization**: `DrawWeights(spread)` — scales from uniform [-1,1] (spread=0) to Xavier 1/√fan_in (spread=1)
|
||||
2. **RL noise**: `MoveWeights(speed, spread)` — noise cap from 0.3 (spread=0) to 0.05 (spread=1), per-layer scaling
|
||||
3. **Weight decay**: 0% (spread=0) to 10% per step (spread=1)
|
||||
4. **Noise cap**: `0.3*(1-spread) + 0.05*spread`
|
||||
|
||||
**Prerequisite**: These spread-aware methods do **not** currently exist in nisps-core C++. The JS webapp implements spread interpolation, per-layer noise scaling, and weight decay in `playground/js/nisps/mlp.js`. The existing C++ `DrawWeights(scale)` is deprecated and `MoveWeights(speed)` has no spread parameter. **Phase 0** (below) ports this logic into nisps-core as proper C++ MLP methods.
|
||||
1. **Weight initialisation**: `draw_weights(spread)` — uniform [−1,1] (spread=0) → Xavier (spread=1)
|
||||
2. **RL perturbation**: `move_weights(speed, spread)` — per-layer scaling + weight decay
|
||||
3. **Noise cap**: `0.3·(1−spread) + 0.05·spread`
|
||||
|
||||
### Dataset Capacity
|
||||
|
||||
The dataset has a maximum of 100 examples (`Dataset::kMax_examples`). When full, FIFO forgetting drops the oldest example. This means extended RL sessions will gradually lose the user's earliest preferences. This is acceptable for RL exploration but should be surfaced in the UI (example count display should show "42/100" style).
|
||||
100 examples (`vcv/src/iml.hpp` `kMaxExamples = 100`, kept equal to the old vendored capacity so
|
||||
patch round-trips preserve counts). FIFO: oldest example dropped when full. Note the nisps core's
|
||||
own default is 128 (`kDefaultMaxExamples`); the VCV adapter pins 100 explicitly.
|
||||
|
||||
---
|
||||
|
||||
## Inference Rate
|
||||
|
||||
User-configurable via RATE knob:
|
||||
RATE knob, exponential mapping `period = 256 · (1/256)^rate` samples:
|
||||
|
||||
| Position | Rate | Behavior |
|
||||
|----------|------|----------|
|
||||
| Full CCW | ~170 Hz | Once per VCV process block (256 samples). Cheapest. |
|
||||
| 12 o'clock | ~2 kHz | Every ~22 samples. Good for CV-rate modulation. |
|
||||
| Full CW | 44.1 kHz | Every sample. Audio-rate CV. Most expensive. |
|
||||
| Full CCW | ~170 Hz | Once per 256 samples. Cheapest. |
|
||||
| 12 o'clock | ~2.8 kHz | Every 16 samples. Good for CV-rate modulation. |
|
||||
| Full CW | 44.1 kHz | Every sample. Audio-rate CV. |
|
||||
|
||||
Between inference steps, output values are linearly interpolated (slew) to avoid staircase artifacts.
|
||||
|
||||
Note: two interpolation systems coexist — decimation interpolation (sub-ms, between inference steps) and post-training crossfade (10ms+, between weight swaps). These compose cleanly: the crossfade produces a smooth target that the decimation interpolation tracks. No special interaction handling needed.
|
||||
Between inference steps, outputs are linearly interpolated; the post-training crossfade (slew)
|
||||
composes on top.
|
||||
|
||||
---
|
||||
|
||||
## RL Feedback Workflow
|
||||
|
||||
### Thumbs Up (+)
|
||||
**Thumbs up (+)**: capture current input/output pair as a training example → enqueue training on
|
||||
the worker → `noiseLevel *= 0.97`.
|
||||
|
||||
1. Capture current input vector and output vector
|
||||
2. Add as training example to dataset
|
||||
3. Enqueue training on background thread
|
||||
4. Decay noise: `noiseLevel *= 0.97`
|
||||
**Thumbs down (−)**: `noiseLevel = min(noiseLevel · 1.5, noiseCap)` → enqueue perturbation
|
||||
(`move_weights(noiseLevel, spread)`) on the worker → audio thread crossfades to the new weights.
|
||||
|
||||
### Thumbs Down (−)
|
||||
|
||||
1. Increase noise: `noiseLevel = min(noiseLevel * 1.5, noiseCap)`
|
||||
2. Enqueue perturbation job on background thread: clone weights → `mlp.moveWeights(noiseLevel, spread)` → signal swap
|
||||
3. Audio thread picks up new weights on next swap, crossfades via output slew
|
||||
|
||||
Note: perturbation goes through the background thread (not direct mutation) to maintain the single-writer threading invariant. The ~1-5ms latency is imperceptible on a button press.
|
||||
|
||||
### Learn Enable Gate
|
||||
|
||||
- When LEARN is disabled (gate low AND toggle off): +/− buttons and triggers are ignored. Inference still runs normally.
|
||||
- When LEARN is enabled: +/− feedback is accepted.
|
||||
- "Learn off = play mode" — the module always runs inference. Learning only controls whether feedback is registered.
|
||||
**Learn gate**: when LEARN is off (gate low AND toggle off), +/− are ignored; inference always
|
||||
runs ("learn off = play mode").
|
||||
|
||||
---
|
||||
|
||||
## State Persistence
|
||||
## State Persistence — the ONE `.nisps`/patch format (version 3)
|
||||
|
||||
### Patch Save/Load
|
||||
|
||||
Full state serialized into VCV patch JSON:
|
||||
Both the VCV patch state and menu-saved `.nisps` files use the same JSON, produced by
|
||||
`MEMLNaut::dataToJson()`:
|
||||
|
||||
```json
|
||||
{
|
||||
"version": 1,
|
||||
"inputCount": 2,
|
||||
"spread": 0.6,
|
||||
"inferenceRate": 0.5,
|
||||
"version": 3,
|
||||
"inputCount": 8,
|
||||
"outputCount": 16,
|
||||
"noiseLevel": 0.1,
|
||||
"slewMs": 10,
|
||||
"outputRanges": [{"unipolar": true, "attenuation": 1.0}, ...],
|
||||
"weights": [[...], ...],
|
||||
"examples": {"features": [[...]], "labels": [[...]]},
|
||||
"mlpConfig": {"layers": [3, 16, 24, 16, 12], "activations": ["relu", "relu", "relu", "sigmoid"]}
|
||||
"slewMs": 10.0,
|
||||
"computeDerived": false,
|
||||
"oscEnabled": false,
|
||||
"oscPort": 7001,
|
||||
"outputRangeUnipolar": [true, "… ×16"],
|
||||
"inputRangeUnipolar": [true, "… ×8"],
|
||||
"weights": ["… flat float array"],
|
||||
"examples": { "features": [["… ×8"]], "labels": [["… ×16"]] },
|
||||
"mlpConfig": { "layers": [8, 16, 24, 16, 16] },
|
||||
"params": ["only in menu-saved .nisps files: all param values incl. attenuverters"]
|
||||
}
|
||||
```
|
||||
|
||||
The `version` field enables forward compatibility. On load, validate that `mlpConfig.layers` matches the current module configuration; if not, warn the user and offer to rebuild the MLP to match the file's architecture.
|
||||
|
||||
### Preset Files (.nisps)
|
||||
|
||||
- **Save**: Export current state to a `.nisps` JSON file (same format as patch state)
|
||||
- **Load**: Import from `.nisps` file via right-click menu → "Load preset..."
|
||||
- **Location**: User-chosen, no enforced directory
|
||||
- Enables sharing trained networks between patches and with the companion webapp
|
||||
- **`weights` is the core's FLAT vector**: `[layer0_w … layer3_w][layer0_b … layer3_b]` — the
|
||||
exact `MLPCore` weight layout, biases included. The old v1 nested 3-D
|
||||
`weights[layer][node][weight]` blobs (bias not serialised) do **not** load: `set_weights`
|
||||
rejects them on a size guard.
|
||||
- **`mlpConfig.layers`** are true node counts `[8, 16, 24, 16, 16]` — no "+1 bias" first entry.
|
||||
- **`examples`**: parallel `features` (×8 floats) / `labels` (×16 floats) arrays, ≤100 entries.
|
||||
- The menu "Load .nisps preset…" rejects files with `version < 1`; unknown extra fields are
|
||||
ignored. `params` is appended only by the menu save (patch save round-trips params natively).
|
||||
|
||||
---
|
||||
|
||||
## Companion Webapp Integration (Manifold)
|
||||
## Browser Bridge (WS ↔ UDP OSC)
|
||||
|
||||
### Locked Transport: WebSocket ↔ UDP OSC Bridge
|
||||
The module runs a UDP OSC server (`vcv/src/osc_server.hpp`, transport-only). The browser
|
||||
(`manifold/src/backends/vcv-backend.ts`) connects via the Deno WS↔UDP bridge
|
||||
(`manifold/osc-bridge/bridge.ts`; WS port default 8765 — pass `--osc-port` to match the module's
|
||||
port, the bridge's own default is 9000).
|
||||
|
||||
The module runs its own **UDP OSC server** (`src/osc_server.hpp` — evolve the existing skeleton). The browser app (`manifold/src/backends/osc-backend.ts`) connects via **WebSocket to a Deno bridge** (`manifold/osc-bridge/`), which relays UDP OSC to the module. This enables **bidirectional training**: both the browser verdict loop (thumbs-up/down + example-placing) and the module's panel (+/− buttons, Learn gate, triggers) can drive the same MLP.
|
||||
### Live OSC verbs (the complete set)
|
||||
|
||||
### OSC Verbs (Module ← → Browser)
|
||||
| Verb | Direction | Payload |
|
||||
|------|-----------|---------|
|
||||
| `/nisps/input` | Browser → VCV | float array — input vector; while streaming, the module runs in **bridged mode** (browser drives the model inputs instead of the jacks) |
|
||||
| `/nisps/output` | VCV → Browser (~100 ms throttle) | 16 floats — live outputs (alive-proof + visualisation) |
|
||||
| `/nisps/input` | VCV → Browser (~100 ms throttle) | 8 floats — live inputs (echo) |
|
||||
| `/nisps/feedback` | Browser → VCV | JSON string op (`up` / `down` / `rand` / `clear`, optional `spread`, `input`/`output` vectors), routed through the same paths as the panel buttons |
|
||||
|
||||
| Verb | Source | Direction | Payload |
|
||||
|------|--------|-----------|---------|
|
||||
| `/nisps/input` | Browser | → VCV | [8 floats] — CV input values |
|
||||
| `/nisps/output` | VCV | → Browser | [16 floats] — raw MLP outputs (for visualization) |
|
||||
| `/nisps/feedback` | Browser | → VCV | (thumbsUp|thumbsDown, optional posLabel [float, float] for placement) |
|
||||
| `/nisps/feedback` | VCV | → Browser | Same (module panel triggers) |
|
||||
| `/nisps/weights` | Either | ↔ | Full weight transfer |
|
||||
| `/nisps/examples` | Either | ↔ | Example dataset transfer |
|
||||
| `/nisps/state` | Either | ↔ | Full module state (weights, examples, spread, noise level, etc.) |
|
||||
The 2→12-era `/nisps/weights`, `/nisps/examples`, and `/nisps/state` full-state sync verbs were
|
||||
**deleted** (both directions had zero consumers; Rack patch save/load owns persistence).
|
||||
Bidirectional *training* remains: both the browser verdict loop and the module panel drive the
|
||||
same MLP.
|
||||
|
||||
**Port assignment:** Fixed default UDP port 7001, with per-instance offset if multiple modules exist in the same patch (e.g. module 2 → 7002). The Deno bridge maps `ws://localhost:8765` ↔ that UDP port.
|
||||
|
||||
### File-Based Preset Export/Import
|
||||
|
||||
File-based transfer remains supported via `.nisps` JSON format (same structure as patch state, see State Persistence section):
|
||||
- VCV: Right-click → "Export .nisps preset" → JSON file
|
||||
- VCV: Right-click → "Import .nisps preset" → restores weights + examples + config
|
||||
- Webapp can read/write the same `.nisps` format for offline interchange with other modules or archival
|
||||
**Port assignment:** default UDP 7001 + per-instance offset (`7001 + id % 64`); the context menu
|
||||
offers {7001, 7002, 7003, 9000, 9001}.
|
||||
|
||||
---
|
||||
|
||||
## Panel Layout
|
||||
|
||||
The module spans **44HP** (wide variant) to accommodate all 8 input jacks, 16 output jacks with LED rings, control knobs, and buttons. An optional **expander module (16HP)** provides additional attenuverter fine-tuning and advanced display.
|
||||
The module is **44HP** (`res/MEMLNaut-wide.svg`, 223.52 mm — the SVG the widget actually loads).
|
||||
`res/MEMLNaut.svg` is an unused 30HP variant and `res/MEMLNaut-expander.svg` an unused 8HP
|
||||
expander panel; no expander module is implemented.
|
||||
|
||||
### Main Module (44HP) Layout
|
||||
|
||||
Typical layout (exact spacing subject to panel artwork):
|
||||
- Top: small DISPLAY showing training state, example count, spread, noise level
|
||||
- Upper: SPREAD knob, RATE knob, LEARN button + LED, RAND button, CLEAR button
|
||||
- Middle: 8 input jacks (IN 1–8) in a column, LEARN gate, +/− trigger inputs
|
||||
- Lower: 16 output jacks arranged in 2 rows of 8, each jack surrounded by an LED ring
|
||||
- Under each output jack: small attenuverter trim pot for per-output scaling
|
||||
|
||||
### Expander Module (16HP) — Optional
|
||||
|
||||
Adds:
|
||||
- Extended attenuverter controls (alternate layout for finer per-output adjustment)
|
||||
- Secondary display (novelty grid, confidence map, or advanced metrics)
|
||||
- Space for future I/O expansion
|
||||
- Top: display strip (16 output bars, noise level, example count, TRAIN)
|
||||
- Upper: SPREAD + RATE knobs, LEARN toggle + LED, RAND, CLEAR, +/− buttons
|
||||
- Middle: 8 input jacks, SPREAD CV, LEARN gate, +/− trigger inputs
|
||||
- Lower: 16 output jacks (2 rows of 8) with LED rings and per-output attenuverter trimpots
|
||||
|
||||
---
|
||||
|
||||
## Build System
|
||||
|
||||
### VCV Rack 2 Plugin Structure
|
||||
|
||||
```
|
||||
vcv/
|
||||
├── plugin.json # Plugin manifest
|
||||
├── Makefile # VCV SDK Makefile + RACK_DIR auto-download
|
||||
├── src/
|
||||
│ ├── plugin.hpp # Plugin globals
|
||||
│ ├── plugin.cpp # Plugin init
|
||||
│ ├── MEMLNaut.cpp # Module logic (process, state, threading, OSC)
|
||||
│ ├── MEMLNautWidget.cpp # Panel UI + LED ring drawing
|
||||
│ ├── palette.hpp # Hand-written color palette (from design tokens)
|
||||
│ └── osc_server.hpp # OSC server impl (evolve existing skeleton)
|
||||
├── res/
|
||||
│ ├── MEMLNaut.svg # Panel artwork (44HP, 16 outputs + LED rings)
|
||||
│ ├── MEMLNaut-wide.svg # Wide variant (if needed)
|
||||
│ ├── MEMLNaut-expander.svg # Expander panel (future)
|
||||
│ └── components/ # Custom SVG components
|
||||
└── dep/
|
||||
└── nisps/ # Symlink to ../nisps (C++20 core library)
|
||||
```
|
||||
|
||||
### Dependencies
|
||||
|
||||
- **VCV Rack SDK 2.x** — must be installed or auto-fetched. The build step should check for `RACK_DIR` env var, and if not set, fetch the Linux SDK zip from vcvrack.com and set it automatically.
|
||||
- **nisps/** (C++20 core library) — use the current repo's `nisps/ml/` and `nisps/core/` directly (not the retired `nisps-core`)
|
||||
- **OSC library** — `oscpack` or `liblo` for UDP OSC server, or a minimal from-scratch UDP impl to avoid the dependency
|
||||
|
||||
### Build Commands
|
||||
## Build
|
||||
|
||||
```bash
|
||||
cd vcv
|
||||
# RACK_DIR will be auto-fetched/set if not already present
|
||||
export RACK_DIR=/path/to/Rack-SDK # default: ~/.local/share/Rack2/Rack-SDK
|
||||
make
|
||||
make install # Copies to VCV plugin directory
|
||||
make install
|
||||
```
|
||||
|
||||
### Panel SVG Updates
|
||||
Files: `src/plugin.{hpp,cpp}`, `src/MEMLNaut.cpp` (module + widget + serialization + OSC wiring),
|
||||
`src/iml.hpp` (core adapter), `src/osc_server.hpp`, `src/LedRing.hpp`, `src/palette.hpp`,
|
||||
`res/*.svg`, `Makefile` (adds `-std=c++20`; the core is reached via relative `../../nisps/…`
|
||||
includes from `src/`, no extra `-I`). Cross-platform bundles are produced by `build-mac.sh` /
|
||||
`build-win.sh` into `dist/` (see `vcv/BUILDING.md`, `vcv/DISTRIBUTION.md`).
|
||||
|
||||
Widen and relayout panel SVGs (`MEMLNaut.svg` / `-wide.svg` / `-expander.svg`) for:
|
||||
- 8 input jacks (vertical column on left)
|
||||
- 16 output jacks with LED ring enclosures (2 rows of 8)
|
||||
- Attenuverter trim pots under each output
|
||||
- Knobs, buttons, display, and gate/trigger inputs in the remaining space
|
||||
**Rack version decision: ship v2-only.** The v1 SDK lacks the menu/tooltip APIs used throughout
|
||||
and may not support the C++20 the core requires; the port (~8 h) can be done later if demand
|
||||
materialises.
|
||||
|
||||
---
|
||||
|
||||
## Development Phases
|
||||
## Open Questions
|
||||
|
||||
### Phase 0: nisps-core Spread API
|
||||
- Port `drawWeights(spread)` from JS `mlp.js:209` to C++ `MLP::DrawWeights(T spread)`
|
||||
- Port `moveWeights(speed, spread)` from JS `mlp.js:231` to C++ `MLP::MoveWeights(T speed, T spread)`
|
||||
- Per-layer Xavier noise scaling
|
||||
- Weight decay proportional to spread
|
||||
- Deprecate old `DrawWeights(float scale)` (already marked `[[deprecated]]`)
|
||||
- Update `IML` to expose spread-aware methods
|
||||
- Unit tests for spread=0, spread=0.5, spread=1 behavior
|
||||
|
||||
### Phase 1: Skeleton (get it compiling)
|
||||
- VCV plugin scaffold from template
|
||||
- Integrate nisps-core headers
|
||||
- Empty module that appears in VCV module browser
|
||||
- 2 input ports, 12 output ports, no processing
|
||||
|
||||
### Phase 2: Core Engine
|
||||
- Wire CV inputs → IML → CV outputs
|
||||
- MLP inference in process() callback (fixed rate)
|
||||
- Spread knob controlling weight initialization
|
||||
- Randomize button
|
||||
|
||||
### Integration Smoke Test (after Phase 2, before Phase 3)
|
||||
- Patch MEMLNaut outputs into a VCV synth voice (VCO → VCF → VCA)
|
||||
- Verify outputs change when inputs change (patch LFO into input)
|
||||
- Verify Randomize produces audibly different mappings
|
||||
- Evaluate if [16, 24, 16] network feels expressive enough for 12 outputs
|
||||
- This is the first "playable moment" — assess if the core concept works before building RL
|
||||
|
||||
### Phase 3: RL Feedback
|
||||
- +/− buttons on panel
|
||||
- +/− trigger inputs
|
||||
- Learn toggle + gate input
|
||||
- Background thread training with weight double-buffering
|
||||
- Noise level tracking
|
||||
|
||||
### Phase 4: Visual Feedback
|
||||
- Custom display widget (prototype both bar graph and full visualization)
|
||||
- LED indicators per output
|
||||
- Training state display
|
||||
|
||||
### Phase 5: Configurability
|
||||
- Inference rate knob with interpolation
|
||||
- Per-output range configuration (unipolar/bipolar)
|
||||
- Attenuverter knobs
|
||||
- Input count configuration with MLP rebuild
|
||||
|
||||
### Phase 6: Persistence
|
||||
- Full state serialization to patch JSON
|
||||
- .nisps preset file save/load
|
||||
- Right-click menu integration
|
||||
|
||||
### Phase 7: Derived Outputs
|
||||
- Mean, spread, delta: computed on audio thread (trivial cost)
|
||||
- Novelty, confidence: computed on training thread after each training run
|
||||
- Pre-compute a novelty/confidence map over a grid of input positions
|
||||
- Audio thread looks up nearest grid point (cheap)
|
||||
- **Future consideration**: alternative approaches (display-rate update, lazy compute on input change) may be more accurate — document in backlog
|
||||
- 5 additional output ports
|
||||
|
||||
### Phase 8: Companion Webapp Bridge
|
||||
- .nisps file format shared between webapp and VCV
|
||||
- Webapp import/export buttons
|
||||
- OSC server in VCV module
|
||||
- OSC client in webapp
|
||||
|
||||
### Phase 9: Panel Variants
|
||||
- Prototype standard (30HP), wide (44HP), and expander (16HP) layouts
|
||||
- User testing, converge on final layout
|
||||
|
||||
### Phase 10: Polish & Distribution
|
||||
- Panel artwork / graphic design
|
||||
- Performance optimization
|
||||
- VCV Rack v1 compatibility pass
|
||||
- Documentation
|
||||
- Distribution packaging
|
||||
|
||||
---
|
||||
|
||||
## Performance Characteristics
|
||||
|
||||
### Inference Cost
|
||||
|
||||
The MLP has architecture `[3, 16, 24, 16, 12]` (3 = 2 inputs + 1 bias node). Each layer's nodes compute a weighted sum of all inputs from the previous layer (including bias weight), then apply an activation function.
|
||||
|
||||
| Layer transition | Nodes | Weights per node | Multiply-adds | Activations |
|
||||
|-----------------|-------|-----------------|---------------|-------------|
|
||||
| Input (3) -> Hidden 1 (16) | 16 | 3 | 48 | 16 (ReLU) |
|
||||
| Hidden 1 (16) -> Hidden 2 (24) | 24 | 17 (16 + bias) | 408 | 24 (ReLU) |
|
||||
| Hidden 2 (24) -> Hidden 3 (16) | 16 | 25 (24 + bias) | 400 | 16 (ReLU) |
|
||||
| Hidden 3 (16) -> Output (12) | 12 | 17 (16 + bias) | 204 | 12 (sigmoid) |
|
||||
| **Total** | | | **1,060** | **68** |
|
||||
|
||||
One forward pass: ~1,060 multiply-adds + 68 activation evaluations (56 ReLU, 12 sigmoid). This is trivially cheap for any modern CPU.
|
||||
|
||||
### Memory
|
||||
|
||||
- ~20KB per MLP instance (weights + node state for [3, 16, 24, 16, 12])
|
||||
- 2 MLP instances per module (double-buffering for thread safety): ~40KB
|
||||
- Dataset: up to 100 examples, each with 2 floats (inputs) + 12 floats (labels) = 5.6KB max
|
||||
- Total per module instance: ~46KB — negligible
|
||||
|
||||
### Threading
|
||||
|
||||
- 1 background thread per module instance for training/perturbation
|
||||
- No shared thread pool (simplicity over efficiency)
|
||||
- 4 module instances = 4 threads + ~184KB total memory
|
||||
|
||||
### Rate Knob Range
|
||||
|
||||
| Knob position | Inference rate | Period (samples at 44.1kHz) | Behavior |
|
||||
|--------------|---------------|----------------------------|----------|
|
||||
| Full CCW (0.0) | ~170 Hz | 256 | Once per process block. Cheapest. |
|
||||
| 12 o'clock (0.5) | ~2,756 Hz | 16 | Good for CV-rate modulation. |
|
||||
| Full CW (1.0) | 44,100 Hz | 1 | Every sample. Audio-rate CV. |
|
||||
|
||||
Mapping is exponential: `period = 256 * (1/256)^rate`, giving perceptually linear response.
|
||||
|
||||
### CPU Estimate
|
||||
|
||||
At default rate (~2kHz with knob at 0.5): ~2,000 forward passes/sec. Each pass is ~1,060 multiply-adds. Total: ~2.1M multiply-adds/sec. For context, a single modern CPU core can sustain billions of multiply-adds per second. Even at audio rate (44.1kHz = ~46M multiply-adds/sec), the MLP inference is a small fraction of available compute.
|
||||
|
||||
The dominant CPU cost at audio rate is not the MLP math but the per-sample overhead in `process()` (derived output computation, slew interpolation, voltage scaling). At default rate this overhead is amortized across ~16 samples.
|
||||
|
||||
---
|
||||
|
||||
## VCV Rack v1 Compatibility
|
||||
|
||||
### v2-Specific APIs Used
|
||||
|
||||
The following VCV Rack v2 APIs are used throughout `src/MEMLNaut.cpp`:
|
||||
|
||||
| API | Usage | v1 Equivalent |
|
||||
|-----|-------|--------------|
|
||||
| `createPanel()` | `setPanel(createPanel(asset::plugin(...)))` — loads SVG panel | `SVGPanel` + `setPanel()` manual setup |
|
||||
| `configButton()` | Configures momentary button params (RAND, +, -, CLEAR) | `configParam()` with min=0, max=1, default=0 |
|
||||
| `configSwitch()` | Configures LEARN toggle with string labels | `configParam()` (no label strings) |
|
||||
| `configParam()` | All knob/attenuverter configuration | Same name, but v2 added display formatting args |
|
||||
| `configInput()` / `configOutput()` | Port labels for tooltips | Not available in v1 (no port tooltips) |
|
||||
| `createCheckMenuItem()` | Context menu toggle items (output ranges, input ranges, OSC, slew) | Manual `MenuItem` subclass with `rightText` checkmark |
|
||||
| `createSubmenuItem()` | Nested submenus (slew time, OSC port) | Manual `MenuItem` subclass overriding `createChildMenu()` |
|
||||
| `createMenuItem()` | Simple menu actions (save/load preset) | Manual `MenuItem` subclass overriding `onAction()` |
|
||||
| `createMenuLabel()` | Section headers in context menu | `MenuLabel` direct construction |
|
||||
| `LedDisplay` | Base class for NanoVG bar graph display widget | `LedDisplayWidget` (similar but slightly different API) |
|
||||
| `drawLayer()` | Layer-based drawing (layer 1 = foreground) | `draw()` only (no layer separation) |
|
||||
| `createModel<M, W>()` | Template model registration | Same syntax (available since late v1) |
|
||||
| `string::f()` | Printf-style string formatting | `string::f()` (available in v1) |
|
||||
| `dsp::BooleanTrigger` | Edge detection on boolean params | Available in v1 |
|
||||
| `dsp::SchmittTrigger` | Edge detection on CV triggers | Available in v1 |
|
||||
| `json_*` (jansson) | State serialization | Same (jansson is used in both v1 and v2) |
|
||||
| `osdialog_*` | Native file dialogs for preset save/load | Same (osdialog bundled in both) |
|
||||
| Standard widgets: `RoundBlackKnob`, `VCVButton`, `CKSS`, `Trimpot`, `PJ301MPort`, `SmallLight` | Panel components | All available in v1 (VCVButton may need renaming to `BefacoButton` or similar) |
|
||||
|
||||
### APIs Without Direct v1 Equivalents
|
||||
|
||||
- **`configInput()` / `configOutput()`**: v1 has no port tooltip system. These calls would simply be removed — ports would work but lack hover labels.
|
||||
- **`configButton()` / `configSwitch()`**: Would revert to `configParam()` calls. Lose the semantic distinction and string labels.
|
||||
- **`createCheckMenuItem()` / `createSubmenuItem()`**: The most labor-intensive change. Each menu item in v1 requires a dedicated `struct` subclass of `MenuItem` with `onAction()` and `rightText` overrides. Our context menu has ~20+ items. A v1 port would need ~20 small structs or a templated helper.
|
||||
|
||||
### Estimated Effort
|
||||
|
||||
- **Mechanical changes** (configButton -> configParam, remove configInput/configOutput labels): ~1 hour
|
||||
- **Context menu rewrite** (createCheckMenuItem/createSubmenuItem -> manual MenuItem subclasses): ~3-4 hours. This is the bulk of the work — approximately 20 menu items each needing a small struct.
|
||||
- **Display widget** (LedDisplay/drawLayer -> LedDisplayWidget/draw): ~30 minutes
|
||||
- **Widget naming** (VCVButton and similar may have different names): ~30 minutes of research + find/replace
|
||||
- **Testing**: ~2 hours (v1 has different SDK build, need separate build environment)
|
||||
- **Total estimate**: ~8 hours of focused work
|
||||
|
||||
### Recommendation
|
||||
|
||||
**Ship v2-only.** Rationale:
|
||||
|
||||
1. VCV Rack v1 userbase is declining — v2 Community Edition is free, removing the cost barrier that kept some users on v1.
|
||||
2. The 8-hour port effort is not large, but maintaining two codepaths adds ongoing cost for every future feature.
|
||||
3. nisps-core requires C++20 (`std::span`, concepts). The v1 SDK toolchain may not support C++20 on all platforms, which could require additional workarounds or feature-gating.
|
||||
4. If v1 demand materializes, the port is straightforward and can be done as a one-time effort.
|
||||
|
||||
---
|
||||
|
||||
## Open Questions (to resolve during implementation)
|
||||
|
||||
1. **MLP hidden layer sizing**: [16, 24, 16] is a guess. May need tuning based on real-world training performance with 12 outputs.
|
||||
2. **Novelty/confidence thresholds**: How to calibrate the novelty gate and confidence output. May need user-adjustable sensitivity.
|
||||
3. **Novelty grid resolution**: The training-thread novelty map approach trades accuracy for speed. What grid resolution is needed for useful novelty output? With 2 inputs a 32x32 grid is 1024 points; with 8 inputs the grid is impractical (32^8 = 1 trillion). Higher-dimensional inputs may need a different strategy (e.g. only compute for current input neighborhood).
|
||||
4. **OSC port conflicts**: What if multiple MEMLNaut instances run in the same patch? Per-instance port assignment?
|
||||
5. **V1 compatibility**: How much of the v2-specific API (polyphonic ports, new widget system) do we actually use? Determines v1 compat effort.
|
||||
6. **Attenuverter UX**: Tiny trimpots on a VCV panel can be fiddly. May need to test whether attenuverters per output are actually useful vs. just using VCV's built-in attenuverter modules.
|
||||
7. **Training convergence with 12 outputs**: The webapp trains 126 outputs — RL feedback on 12 is a different dynamic. May converge faster or feel less "exploratory".
|
||||
8. **Expander module protocol**: VCV uses `ExpanderMessage` structs with left/right adjacency detection. Need to design the data protocol between main module and expander if/when we build it.
|
||||
9. **C++20 compiler support**: VCV SDK Makefile targets specific compiler versions. Verify that C++20 features used by nisps-core (std::span, concepts) are supported by the VCV toolchain on all platforms (Linux, macOS, Windows).
|
||||
1. **Derived stats are computed but unconsumed** — the context-menu toggle computes Mean/Std/
|
||||
Delta (and novelty is cached after training) with no jack, display, or OSC consumer. Either
|
||||
wire a consumer or delete the compute path.
|
||||
2. **Hidden-layer sizing**: [16, 24, 16] carried over from the 2→12 era; untuned for 8→16.
|
||||
3. **Expander**: an 8HP expander SVG exists with no module behind it — build or delete.
|
||||
|
|
|
|||
|
|
@ -39,7 +39,7 @@ bun run test:e2e # Playwright smoke (needs `bun run build` first; runs again
|
|||
- **E2E on the VPS** needs a non-snap node runner (bun is snap-confined and hides libs from
|
||||
Chromium): `PLAYWRIGHT_BROWSERS_PATH=/home/w1n5t0n/snap/bun-js/87/.cache/ms-playwright node node_modules/.bin/playwright test`. Preview via bun is fine. The smoke spec (`tests/e2e/smoke.spec.ts`)
|
||||
asserts: engine WASM loads, spine invariant (setInputs → outputs change), feedback runs, console
|
||||
renders, **no "C15" in the bundle**, no console errors. `shot.spec.ts` takes screenshots.
|
||||
renders, **no "C15" in the bundle**, no console errors.
|
||||
- **Deploy is automatic on push to GitHub `main`, but gated on CI** → webhook → waits for the
|
||||
`CI` workflow to conclude `success` on that exact SHA → builds `manifold/` → rsyncs to the live
|
||||
`/next/` subdir. A red or missing CI run aborts the deploy (fail-closed, 20 min timeout);
|
||||
|
|
@ -99,16 +99,25 @@ decision. Buffers are reused frame-to-frame; never assume a fresh array.
|
|||
(`outputMode`/`midiOutputId`/`oscUrl`/`vcvUrl`), and UI flags (`sandwich`, `split`, `stripPinned`,
|
||||
`snapshots`, `markers`, `health`, `rev`). Builds the flat `ConsoleCtx` passed to the Dock + drawers.
|
||||
|
||||
### The "convertible" Stages (one renders at a time, chosen by `focus` + `outputMode`)
|
||||
### The Stages (one renders at a time)
|
||||
**There is no `focus` axis any more.** The focus/altitude system (`AltitudeNav`, `SplitStage`,
|
||||
`ReadoutStrip`, `InputMini`, `CompactAxis`) was deleted in the 2026-07 simplification audit —
|
||||
Manifold ships a single "composite" altitude. Selection is now a plain three-way in `ConsoleApp`:
|
||||
`sandwich` wins, else `outputMode==='particles'`, else CompositeStage.
|
||||
|
||||
| Stage | File | Renders when | What it is |
|
||||
|---|---|---|---|
|
||||
| Manifold | `Manifold.tsx` | `focus==='in'` (default input view) | Full-bleed 2D input surface; canvas trail + pins + feedback markers; pointer → `onMove`. **Double-click the input mark → follow-mouse mode** (self-contained state; a window `pointermove` listener maps the whole viewport onto this surface's space so the knob tracks the cursor across the entire UI; Esc / second double-click exits). |
|
||||
| OutputStage | `OutputStage.tsx` | `focus==='out'` | Full-bleed output columns; drag bars set value; `InputMini` docked in a corner. |
|
||||
| SplitStage | `SplitStage.tsx` | `focus==='split'` | Manifold left, OutputStage right, equal width. |
|
||||
| CompositeStage | `CompositeStage.tsx` | `focus==='composite'` (**app default / hero**) | Draggable split-ratio; magnet-snaps to 0.14/0.33/0.5/0.66/0.86; collapses a side to a corner minimap at extremes. |
|
||||
| SandwichStage | `SandwichStage.tsx` | `sandwich===true` (overrides) | 3D parameter-landscape view (input → MLP heatmap grid → outputs); drag to orbit. |
|
||||
| CompositeStage | `CompositeStage.tsx` | **default / hero** | Draggable split-ratio; magnet-snaps to 0.14/0.33/0.5/0.66/0.86; collapses a side to a corner minimap at extremes. |
|
||||
| SandwichStage | `SandwichStage.tsx` | `sandwich===true` (wins over the others) | Three-pane layout: `Manifold` input surface left, 3D parameter-landscape centre (input → MLP heatmap grid → outputs, drag to orbit), compact `OutputStage` right. |
|
||||
| ParticleStage | `ParticleStage.tsx` | `outputMode==='particles'` | Flow-field visualiser (`flow-field.ts`, 400-particle Canvas2D port) + macro-axis bar + corner joystick. |
|
||||
|
||||
`Manifold.tsx` and `OutputStage.tsx` are no longer top-level stages — they are panes composed by
|
||||
CompositeStage/SandwichStage. `Manifold.tsx` is the full-bleed 2D input surface (canvas trail + pins
|
||||
+ feedback markers; pointer → `onMove`; **double-click the input mark → follow-mouse mode**, a window
|
||||
`pointermove` listener mapping the whole viewport onto this surface's space, Esc or a second
|
||||
double-click exits). `OutputStage.tsx` is the output columns; drag a bar to set value, and it takes a
|
||||
`compact` prop for the narrow pane.
|
||||
|
||||
- **Output modes** (the TOP dock selector, NOT the same axis as `focus`): `src/console/output-mode.ts`
|
||||
defines `OUTPUT_MODES` = **particles** (default) / midi / osc / synth / editor, each mapping to a
|
||||
`BackendId`. `DEFAULT_OUTPUT_MODE='particles'`.
|
||||
|
|
@ -144,7 +153,6 @@ decision. Buffers are reused frame-to-frame; never assume a fresh array.
|
|||
· curve pad · live value. **Writes eagerly to the shared `MFParam` store via `onChange`.**
|
||||
- `OutputsBackendConfig.tsx` — preset bar (save/restore/rename/delete) + per-backend config (MIDI
|
||||
CC#/channel, OSC path/range, VCV polarity).
|
||||
- `BackendAdvanced.tsx` — full-depth modal version of the same editors.
|
||||
|
||||
### Primitives — `src/primitives/` (barrel: `index.ts`)
|
||||
`Button`, `Slider`, `PillToggle`, `Panel`, `Badge`, `Switch`, `StatusLine`, `XYPad`,
|
||||
|
|
@ -152,12 +160,10 @@ decision. Buffers are reused frame-to-frame; never assume a fresh array.
|
|||
Side-effect import of `styles/primitives.css` styles the range inputs.
|
||||
|
||||
### Other shared UI files
|
||||
- `shared-ui.tsx` — `AltitudeNav` (focus IN/DUAL/OUT/FLEX switcher) + `MiniMeters` (read-only output bars).
|
||||
- `shared-ui.tsx` — `MiniMeters` (read-only output bars). `AltitudeNav`/`CompactAxis` were deleted with the focus system.
|
||||
- `icons.tsx` — monochrome inline-SVG icons (mode icons + drawer icons + `GLYPH_FALLBACK` for when monochrome is off).
|
||||
- `ReadoutStrip.tsx` — thin horizontal heatmap strip (pinned, `focus==='in'`); same per-output control as OutputStage.
|
||||
- `VerdictCluster.tsx` — floating bottom-centre feedback UI (perturb ▽ / undo ↺ / commit △ + A/B); labels adapt to feedback mode.
|
||||
- `CurvePad.tsx` — square canvas curve editor (vertical drag reshapes [0,1]; ~0.43 ≈ linear). Used in OutputEditor + OutputControlRow.
|
||||
- `InputMini.tsx` — compact XYPad/joystick docked in a corner when input is demoted.
|
||||
- `OutputEditor.tsx` — inline min/max/curve popup for a single output (hover/click on a bar).
|
||||
|
||||
### Styling — `src/styles/`
|
||||
|
|
@ -310,8 +316,8 @@ All in `docs/specs/` (at the repo root, not under `manifold/`), with subdirector
|
|||
- `plans/BUILD-PLAN.md` — locked decisions + the 12-step build sequence + spec pointers (the resume anchor).
|
||||
- `engine-architecture.md` — full engine/spine/WASM design.
|
||||
- `dock-spec.md` — dock + drawers spec. `inputs-spec.md` — mixed-input design. `backends-spec.md` — backends.
|
||||
- `docs/adr/rl-feedback-design.md` + `feedback-modes-port-spec.md` + `recon/findings-feedback-behaviour.md` — feedback modes (Mode 1/Mode 2).
|
||||
- `aimmersive-clone-spec.md` / `recon/playground-2026.md` — the a-immersive feature parity target.
|
||||
- `docs/adr/rl-feedback-design.md` + `plans/feedback-modes-port-spec.md` (executed) + `recon/findings-feedback-behaviour.md` — feedback modes (Mode 1/Mode 2).
|
||||
- `_archive/aimmersive-clone-spec.md` / `recon/playground-2026.md` — the a-immersive feature parity target (archived reference).
|
||||
- `src/backends/README.md` — backend wiring notes.
|
||||
|
||||
**Memories** (auto-loaded): `manifold-build` (status + locked decisions), `manifold-mixed-inputs`
|
||||
|
|
|
|||
|
|
@ -8,7 +8,7 @@
|
|||
## Prerequisites
|
||||
|
||||
- **VCV Rack 2 SDK** — download from https://vcvrack.com/manual/PluginDevelopmentTutorial or build from source
|
||||
- **C++20 compiler** — GCC 10+, Clang 11+, or MSVC 19.29+ (required by nisps-core for `std::span` and concepts)
|
||||
- **C++20 compiler** — GCC 10+, Clang 11+, or MSVC 19.29+ (required by the `nisps/` core for `std::span` and concepts)
|
||||
- **GNU Make**
|
||||
|
||||
## Build Steps
|
||||
|
|
@ -24,7 +24,7 @@ make
|
|||
# This produces plugin.so (Linux), plugin.dylib (macOS), or plugin.dll (Windows)
|
||||
```
|
||||
|
||||
The Makefile adds `-std=c++20` and includes nisps-core headers from `../nisps-core/include`. The VCV SDK's default `-std=c++11` flag is filtered out to avoid conflicts.
|
||||
The Makefile adds `-std=c++20`; the shared `nisps/` core is reached via relative `../../nisps/…` includes from `src/` (no extra `-I`). The VCV SDK's default `-std=c++11` flag is filtered out to avoid conflicts.
|
||||
|
||||
## Local Installation
|
||||
|
||||
|
|
@ -65,5 +65,5 @@ The VCV Library submission process handles multi-platform builds automatically v
|
|||
## Troubleshooting
|
||||
|
||||
- **`-std=c++11` conflicts**: The Makefile filters this out, but if you see C++20 errors, verify your `RACK_DIR` points to a v2 SDK and that your compiler supports C++20.
|
||||
- **nisps-core not found**: The include path assumes nisps-core is at `../nisps-core/include` relative to the `vcv/` directory. Verify the path or adjust `-I` in the Makefile.
|
||||
- **nisps headers not found**: `src/iml.hpp` reaches the core via `../../nisps/…` relative includes, so the plugin must be built from a full `MEMLNaut-NISPS` checkout (a standalone copy of `vcv/` will not compile).
|
||||
- **Plugin not appearing**: Check that the built `.so`/`.dylib`/`.dll` is in the correct plugins directory and that `plugin.json` is alongside it.
|
||||
|
|
|
|||
|
|
@ -1,206 +0,0 @@
|
|||
# .nisps File Format Specification
|
||||
|
||||
Version: 1
|
||||
|
||||
## Overview
|
||||
|
||||
The `.nisps` format is a JSON file that captures the complete state of a NISPS interactive ML engine: network weights, training examples, I/O configuration, and MLP architecture metadata. It enables preset sharing between the VCV Rack module and the web playground.
|
||||
|
||||
## Top-Level Structure
|
||||
|
||||
```json
|
||||
{
|
||||
"version": 1,
|
||||
"noiseLevel": 0.1,
|
||||
"slewMs": 10.0,
|
||||
"outputRangeUnipolar": [true, false, ...],
|
||||
"inputRangeUnipolar": [true, true, ...],
|
||||
"weights": [[[...], ...], ...],
|
||||
"examples": {
|
||||
"features": [[...], ...],
|
||||
"labels": [[...], ...]
|
||||
},
|
||||
"mlpConfig": {
|
||||
"layers": [3, 16, 24, 16, 12],
|
||||
"activations": ["relu", "relu", "relu", "sigmoid"]
|
||||
},
|
||||
"params": [0.5, 0.8, ...]
|
||||
}
|
||||
```
|
||||
|
||||
## Field Reference
|
||||
|
||||
### `version` (integer, required)
|
||||
|
||||
Format version number. Currently `1`. Loaders must reject files where `version < 1`. Future versions will increment this value; loaders should accept any version they understand and reject higher versions gracefully.
|
||||
|
||||
### `noiseLevel` (float, optional)
|
||||
|
||||
RL exploration noise amplitude. Range: 0.0-1.0. Default: `0.1`. Controls how much random perturbation is applied to weights during reinforcement learning exploration.
|
||||
|
||||
### `slewMs` (float, optional)
|
||||
|
||||
Output slew rate in milliseconds. Default: `10.0`. Smooths transitions between output values to prevent clicks/artifacts. In VCV Rack this is the time constant for exponential smoothing on output voltages.
|
||||
|
||||
### `outputRangeUnipolar` (boolean[], optional)
|
||||
|
||||
Per-output voltage range flag. Length must equal the number of MLP outputs. When `true`, the output maps to 0-10V (unipolar); when `false`, it maps to +/-5V (bipolar). Default: all `false`.
|
||||
|
||||
This field is VCV Rack-specific. The web playground ignores it but should preserve it on round-trip.
|
||||
|
||||
### `inputRangeUnipolar` (boolean[], optional)
|
||||
|
||||
Per-input voltage range flag. Length must equal the maximum number of inputs (8 in VCV Rack). When `true`, the input expects 0-10V; when `false`, +/-5V. Default: all `false`.
|
||||
|
||||
This field is VCV Rack-specific. The web playground ignores it but should preserve it on round-trip.
|
||||
|
||||
### `weights` (float[][][], required for state restore)
|
||||
|
||||
MLP weights as a 3D array: `weights[layer][node][weight]`.
|
||||
|
||||
**Serialization order:**
|
||||
- Outer dimension: layers, from first hidden layer to output layer. Length = number of layers - 1 (i.e., number of weight matrices).
|
||||
- Middle dimension: nodes within that layer. Length = `mlpConfig.layers[i+1]` (the number of nodes in the destination layer).
|
||||
- Inner dimension: connection weights from source nodes. Length = `mlpConfig.layers[i]` (the number of nodes in the source layer, including bias for the first layer).
|
||||
|
||||
**Bias handling:** In the VCV Rack C++ implementation, bias is stored separately from connection weights and is **not** included in this array. The bias values are not serialized. On load, biases remain at their current values (typically 0).
|
||||
|
||||
In the web playground, bias is also stored separately per node. The webapp's export/import methods handle the structural difference by serializing weights without bias (matching the VCV format) and preserving bias in a separate optional field.
|
||||
|
||||
**Example for a `[3, 4, 2]` network** (3 inputs, 4 hidden, 2 outputs):
|
||||
```json
|
||||
"weights": [
|
||||
[ // layer 0: input -> hidden
|
||||
[0.1, -0.3, 0.5], // hidden node 0: 3 weights from 3 inputs
|
||||
[0.2, 0.4, -0.1], // hidden node 1
|
||||
[-0.6, 0.3, 0.2], // hidden node 2
|
||||
[0.1, -0.5, 0.7] // hidden node 3
|
||||
],
|
||||
[ // layer 1: hidden -> output
|
||||
[0.3, -0.2, 0.4, 0.1], // output node 0: 4 weights from 4 hidden nodes
|
||||
[-0.1, 0.5, -0.3, 0.2] // output node 1
|
||||
]
|
||||
]
|
||||
```
|
||||
|
||||
### `examples` (object, optional)
|
||||
|
||||
Training examples as parallel arrays.
|
||||
|
||||
#### `examples.features` (float[][], required if examples present)
|
||||
|
||||
Input feature vectors. Each inner array has length equal to `mlpConfig.layers[0]` minus the bias term. In the VCV module, features are stored **without** the bias term appended.
|
||||
|
||||
#### `examples.labels` (float[][], required if examples present)
|
||||
|
||||
Output label vectors. Each inner array has length equal to `mlpConfig.layers[last]`. Must have the same outer length as `features`.
|
||||
|
||||
**Example:**
|
||||
```json
|
||||
"examples": {
|
||||
"features": [
|
||||
[0.3, 0.7],
|
||||
[0.8, 0.2]
|
||||
],
|
||||
"labels": [
|
||||
[0.1, 0.9, 0.5, 0.3, 0.7, 0.2, 0.8, 0.4, 0.6, 0.1, 0.5, 0.3],
|
||||
[0.9, 0.1, 0.4, 0.7, 0.3, 0.8, 0.2, 0.6, 0.4, 0.9, 0.5, 0.7]
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### `mlpConfig` (object, required)
|
||||
|
||||
Architecture metadata for validation on load.
|
||||
|
||||
#### `mlpConfig.layers` (integer[], required)
|
||||
|
||||
Layer sizes including input (with bias) and output. For the default VCV configuration with 2 inputs and 12 outputs: `[3, 16, 24, 16, 12]`. The first element includes the +1 bias node.
|
||||
|
||||
#### `mlpConfig.activations` (string[], optional)
|
||||
|
||||
Activation function names per weight layer. Length = `layers.length - 1`. Valid values: `"relu"`, `"sigmoid"`, `"tanh"`, `"linear"`. Default: all hidden layers use `"relu"`, output layer uses `"sigmoid"`.
|
||||
|
||||
The VCV Rack module currently does not serialize this field (the architecture is fixed). The web playground includes it for forward compatibility.
|
||||
|
||||
### `params` (float[], optional)
|
||||
|
||||
VCV Rack module parameter values (knob positions, attenuators, etc.). Array indexed by the module's parameter enum. Present only in files saved from the VCV Rack "Save .nisps preset" menu.
|
||||
|
||||
The web playground may use this field to store synth parameter values for preset portability. Consumers that don't understand the parameter layout should ignore this field.
|
||||
|
||||
## Validation Rules
|
||||
|
||||
Loaders should check the following on import:
|
||||
|
||||
1. **Version gate**: `version` must be present and >= 1. Reject unknown future versions.
|
||||
2. **Weight dimensions**: If `weights` and `mlpConfig.layers` are both present, verify:
|
||||
- `weights.length === mlpConfig.layers.length - 1`
|
||||
- `weights[i].length === mlpConfig.layers[i + 1]`
|
||||
- `weights[i][j].length === mlpConfig.layers[i]`
|
||||
3. **Example dimensions**: If `examples` is present:
|
||||
- `features.length === labels.length`
|
||||
- Each feature vector length should equal `mlpConfig.layers[0]` (or `mlpConfig.layers[0] - 1` if bias is excluded)
|
||||
- Each label vector length should equal `mlpConfig.layers[last]`
|
||||
4. **Architecture compatibility**: If the loader's MLP has a different architecture than `mlpConfig.layers`, the file cannot be loaded directly. The loader should reject or warn.
|
||||
5. **Numeric validity**: All weight and example values must be finite (not NaN or Infinity).
|
||||
|
||||
## Compatibility Notes
|
||||
|
||||
### VCV Rack -> Web Playground
|
||||
|
||||
- The VCV module uses a `[3, 16, 24, 16, 12]` architecture (2 inputs + bias, 12 outputs).
|
||||
- The web playground uses a `[3, 32, 48, 64, 126]` architecture (2 inputs + bias, 126 outputs).
|
||||
- Direct weight transfer between these architectures is not possible. The `mlpConfig.layers` field enables loaders to detect this mismatch and report it.
|
||||
- Training examples (features/labels) are also architecture-dependent due to different output counts.
|
||||
|
||||
### Bias Values
|
||||
|
||||
The VCV C++ serialization does not include node bias values in the `weights` array. Bias is stored separately in `Node::m_bias` but is not written to JSON. This means bias values are reset to their pre-load state when restoring from a `.nisps` file. In practice this has minimal impact because:
|
||||
- The MLP uses leaky ReLU (hidden) and sigmoid (output) activations
|
||||
- Training quickly adjusts bias values
|
||||
- Initial bias is typically 0
|
||||
|
||||
### Future Extensions
|
||||
|
||||
New fields may be added to the top level without incrementing the version number, as long as they are optional and backward-compatible. The version number increments only for breaking changes to existing field semantics.
|
||||
|
||||
## Complete Example
|
||||
|
||||
A minimal but complete `.nisps` file for a `[3, 4, 2]` network (2 inputs, 4 hidden nodes, 2 outputs):
|
||||
|
||||
```json
|
||||
{
|
||||
"version": 1,
|
||||
"noiseLevel": 0.15,
|
||||
"slewMs": 10.0,
|
||||
"outputRangeUnipolar": [true, false],
|
||||
"inputRangeUnipolar": [false, false],
|
||||
"weights": [
|
||||
[
|
||||
[0.123, -0.456, 0.789],
|
||||
[-0.321, 0.654, -0.987],
|
||||
[0.111, -0.222, 0.333],
|
||||
[-0.444, 0.555, -0.666]
|
||||
],
|
||||
[
|
||||
[0.12, -0.34, 0.56, -0.78],
|
||||
[0.91, -0.23, 0.45, -0.67]
|
||||
]
|
||||
],
|
||||
"examples": {
|
||||
"features": [
|
||||
[0.3, 0.7],
|
||||
[0.8, 0.2]
|
||||
],
|
||||
"labels": [
|
||||
[0.9, 0.1],
|
||||
[0.2, 0.8]
|
||||
]
|
||||
},
|
||||
"mlpConfig": {
|
||||
"layers": [3, 4, 2],
|
||||
"activations": ["relu", "sigmoid"]
|
||||
}
|
||||
}
|
||||
```
|
||||
227
vcv/README.md
227
vcv/README.md
|
|
@ -1,10 +1,16 @@
|
|||
# MEMLNaut for VCV Rack
|
||||
|
||||
MEMLNaut is a CV-to-CV mapper powered by a neural network that you train in real time using reinforcement learning. Patch any CV sources into its 2 inputs, connect its 12 outputs to your synth parameters, and shape the mapping by giving thumbs-up/thumbs-down feedback. The module learns your preferences, producing complex, nonlinear modulation that evolves with your taste. It does not generate sound itself -- it generates control voltages.
|
||||
MEMLNaut is a CV-to-CV mapper powered by a neural network that you train in real time using
|
||||
reinforcement learning. Patch any CV sources into its 8 inputs, connect its 16 outputs to your
|
||||
synth parameters, and shape the mapping by giving thumbs-up/thumbs-down feedback. The module
|
||||
learns your preferences, producing complex, nonlinear modulation that evolves with your taste.
|
||||
It does not generate sound itself -- it generates control voltages.
|
||||
|
||||
## Installation
|
||||
|
||||
MEMLNaut is built from source against the VCV Rack 2 SDK.
|
||||
MEMLNaut is built from source against the VCV Rack 2 SDK. Pre-built `.vcvplugin` bundles are
|
||||
published at https://meml.lnfinitemonkeys.org/next/vcv/ (locally-built bundles land in the
|
||||
git-ignored `dist/`; see `DISTRIBUTION.md`).
|
||||
|
||||
### Requirements
|
||||
|
||||
|
|
@ -16,7 +22,7 @@ MEMLNaut is built from source against the VCV Rack 2 SDK.
|
|||
### Build
|
||||
|
||||
```bash
|
||||
git clone --recursive https://github.com/MusicallyEmbodiedML/MEMLNaut-NISPS.git
|
||||
git clone --recursive https://github.com/monkey-w1n5t0n/MEMLNaut-NISPS.git
|
||||
cd MEMLNaut-NISPS/vcv
|
||||
|
||||
# Point to your Rack SDK (or set in environment)
|
||||
|
|
@ -26,18 +32,22 @@ make
|
|||
make install # copies plugin to your VCV Rack plugins directory
|
||||
```
|
||||
|
||||
The default `RACK_DIR` is `~/.local/share/Rack2/Rack-SDK`. If your SDK lives there, you can skip the export.
|
||||
The default `RACK_DIR` is `~/.local/share/Rack2/Rack-SDK`. If your SDK lives there, you can skip
|
||||
the export.
|
||||
|
||||
The `nisps-core` headers (the ML engine) are included in the parent repository and referenced automatically via the Makefile.
|
||||
The ML engine is the repository's shared `nisps/` C++20 core, reached via relative includes from
|
||||
`src/iml.hpp` -- the plugin must be built from a full repo checkout. See `BUILDING.md`.
|
||||
|
||||
## Quick Start
|
||||
|
||||
1. Add **MEMLNaut** from the module browser (under Controller / Utility).
|
||||
2. Patch two LFOs (or any CV source) into the **X** and **Y** inputs.
|
||||
3. Connect several of the 12 outputs to parameters on your synth voice -- filter cutoff, oscillator pitch, waveshape, VCA level, etc.
|
||||
1. Add **MEMLNaut** from the module browser.
|
||||
2. Patch LFOs (or any CV source) into one or more of the 8 **IN** jacks.
|
||||
3. Connect several of the 16 outputs to parameters on your synth voice -- filter cutoff,
|
||||
oscillator pitch, waveshape, VCA level, etc.
|
||||
4. Press **RAND** to randomize the network. You should hear your synth respond as the LFOs sweep.
|
||||
5. Flip the **LEARN** switch on.
|
||||
6. When you hear something you like, press **+** (thumbs up). When you hear something you dislike, press **-** (thumbs down).
|
||||
6. When you hear something you like, press **+** (thumbs up). When you hear something you
|
||||
dislike, press **-** (thumbs down).
|
||||
7. Keep exploring. The mapping will converge toward sounds you prefer.
|
||||
|
||||
## Panel Controls
|
||||
|
|
@ -47,7 +57,7 @@ The `nisps-core` headers (the ML engine) are included in the parent repository a
|
|||
| Control | Description |
|
||||
|---------|-------------|
|
||||
| **SPREAD** | Controls weight initialization scale, RL noise amplitude, and weight decay. Low values produce extreme, polarized mappings. High values produce balanced, subtle mappings. Default: 60%. |
|
||||
| **RATE** | Inference rate. Full CCW = ~170 Hz (block rate, cheapest). Noon = ~2 kHz (good for CV). Full CW = 44.1 kHz (audio rate, most expensive). |
|
||||
| **RATE** | Inference rate. Full CCW = ~170 Hz (block rate, cheapest). Noon = ~2.8 kHz (good for CV). Full CW = 44.1 kHz (audio rate, most expensive). |
|
||||
|
||||
### Buttons
|
||||
|
||||
|
|
@ -63,8 +73,7 @@ The `nisps-core` headers (the ML engine) are included in the parent repository a
|
|||
|
||||
| Port | Description |
|
||||
|------|-------------|
|
||||
| **X** | Primary CV input. Default range: 0-10V (unipolar). |
|
||||
| **Y** | Secondary CV input. Default range: 0-10V (unipolar). |
|
||||
| **IN 1-8** | CV inputs feeding the 8-input network. Default range: 0-10V (unipolar); per-input bipolar toggle in the context menu. |
|
||||
| **SPREAD CV** | CV modulation of the SPREAD knob (added to knob value, 0-10V). |
|
||||
| **LEARN** | Gate input. High = enable learning. Works alongside the LEARN toggle (either enables it). |
|
||||
| **+ TRIG** | Trigger input for thumbs-up. Alternative to pressing the + button. |
|
||||
|
|
@ -74,32 +83,29 @@ The `nisps-core` headers (the ML engine) are included in the parent repository a
|
|||
|
||||
| Port | Description |
|
||||
|------|-------------|
|
||||
| **OUT 1-12** | Raw MLP outputs, each with its own attenuverter trimpot and LED. Default: 0-10V unipolar. |
|
||||
| **MEAN** | Mean of the 12 raw outputs (0-10V). |
|
||||
| **STD** | Standard deviation of the 12 raw outputs (0-10V). |
|
||||
| **DELTA** | Rate of change across all outputs (L2 norm of frame-to-frame difference). |
|
||||
| **NOVELTY** | How far the current input is from any training example. 10V when untrained, drops as you add examples near the current position. |
|
||||
| **CONFIDENCE** | Inverse of novelty. 0V when untrained, rises as examples accumulate near the current input. |
|
||||
| **OUT 1-16** | Raw MLP outputs. Default: 0-10V unipolar (per-output bipolar toggle in the context menu). |
|
||||
|
||||
Each of the 12 raw outputs has a **trimpot attenuverter** (-100% to +100%) for scaling and inverting individual outputs without external modules.
|
||||
Each output has a **trimpot attenuverter** (-100% to +100%) for scaling and inverting individual
|
||||
outputs without external modules, and is surrounded by an **LED ring** whose arc fill tracks the
|
||||
output's current level (color-coded per output, orange-to-cyan ramp).
|
||||
|
||||
### LEDs
|
||||
### LEDs & Display
|
||||
|
||||
| LED | Meaning |
|
||||
|-----|---------|
|
||||
| Indicator | Meaning |
|
||||
|-----------|---------|
|
||||
| **LEARN** (green) | Lit when learning is enabled. |
|
||||
| **TRAIN** (yellow) | Flashes during background training. |
|
||||
| **Output LEDs** (white) | Brightness tracks each output's current level. |
|
||||
| **Output LED rings** | Arc fill tracks each output's current level. |
|
||||
|
||||
### Display
|
||||
|
||||
The built-in bar graph shows all 12 output levels in real time, color-coded by output index. The top-left corner shows the current noise level (N:). "TRAIN" appears in the top-right during active training.
|
||||
The built-in display shows all 16 output levels as color-coded bars, the current noise level
|
||||
(N:), the example count (e.g. `42/100`), and "TRAIN" during active training.
|
||||
|
||||
## RL Workflow
|
||||
|
||||
The reinforcement learning loop works like this:
|
||||
|
||||
1. **Start exploring.** Patch LFOs or sequencers into X and Y. Connect outputs to interesting synth parameters. Press RAND a few times to hear different random mappings.
|
||||
1. **Start exploring.** Patch LFOs or sequencers into the IN jacks. Connect outputs to
|
||||
interesting synth parameters. Press RAND a few times to hear different random mappings.
|
||||
|
||||
2. **Enable learning.** Flip the LEARN switch on (or send a gate to the LEARN input).
|
||||
|
||||
|
|
@ -113,155 +119,120 @@ The reinforcement learning loop works like this:
|
|||
- Perturbs the network weights to try a different mapping
|
||||
- Does NOT save any training example
|
||||
|
||||
5. **Repeat.** Over time, the network learns to produce outputs you tend to like across the input space. Regions near your thumbs-up examples will be stable; distant regions remain exploratory.
|
||||
5. **Repeat.** Over time, the network learns to produce outputs you tend to like across the
|
||||
input space. Regions near your thumbs-up examples will be stable; distant regions remain
|
||||
exploratory.
|
||||
|
||||
6. **Disable learning** when you are happy with the mapping. The module continues running inference with the trained network. You now have a complex, personalized CV source.
|
||||
6. **Disable learning** when you are happy with the mapping. The module continues running
|
||||
inference with the trained network. You now have a complex, personalized CV source.
|
||||
|
||||
### Tips
|
||||
|
||||
- Give thumbs-up at several different input positions to teach the network about different regions of the input space.
|
||||
- Give thumbs-up at several different input positions to teach the network about different
|
||||
regions of the input space.
|
||||
- The network holds up to 100 examples. Oldest examples are dropped when full (FIFO).
|
||||
- Use the SPREAD knob to control how wild the exploration is. Low spread = dramatic changes. High spread = subtle refinements.
|
||||
- The NOVELTY and CONFIDENCE outputs are useful for self-patching: route NOVELTY to control something that signals "unexplored territory."
|
||||
- Use the SPREAD knob to control how wild the exploration is. Low spread = dramatic changes.
|
||||
High spread = subtle refinements.
|
||||
|
||||
## Context Menu
|
||||
|
||||
Right-click the module to access these settings:
|
||||
|
||||
### Output Ranges
|
||||
- **Output ranges** -- toggle each of the 16 outputs between unipolar (0-10V, default) and
|
||||
bipolar (+/-5V).
|
||||
- **Input ranges** -- toggle each of the 8 inputs between unipolar (0-10V, default) and bipolar
|
||||
(+/-5V). Set to bipolar if your source produces +/-5V signals (e.g., standard LFOs).
|
||||
- **Compute derived stats (Mean/Std/Delta)** -- internal statistics toggle (no dedicated output
|
||||
jacks in the current module).
|
||||
- **Output slew** -- smoothing time applied when network weights change (after training or
|
||||
perturbation). Prevents clicks from sudden output jumps. Options: 0, 5, 10, 20, 50, 100 ms.
|
||||
Default: 10 ms.
|
||||
- **Presets (.nisps)** -- save/load the full module state (weights, training examples, knob
|
||||
positions, ranges) to/from a `.nisps` JSON file.
|
||||
- **Browser bridge (WS<->OSC)** -- enable the OSC server for the companion browser app; choose
|
||||
the listen port (7001, 7002, 7003, 9000, 9001).
|
||||
|
||||
Toggle each output between **unipolar (0-10V)** and **bipolar (+/-5V)**. Default is unipolar. Use bipolar for parameters that expect centered modulation (e.g., FM depth, panning).
|
||||
## Presets (.nisps)
|
||||
|
||||
### Input Ranges
|
||||
MEMLNaut uses `.nisps` files for saving and sharing trained networks. The format (version 3,
|
||||
flat core-exact weight vector) is specified in `docs/specs/vcv-module.md` -- the single source of
|
||||
truth. Files saved by pre-2026-07 builds (version 1, nested weight arrays) no longer load.
|
||||
|
||||
Toggle each input between **unipolar (0-10V)** and **bipolar (+/-5V)**. Default is unipolar. Set to bipolar if your input source produces +/-5V signals (e.g., standard LFOs).
|
||||
Full module state is also saved automatically with your VCV Rack patch file; you do not need
|
||||
`.nisps` exports to preserve work between sessions.
|
||||
|
||||
### Output Slew
|
||||
## Browser Integration (Manifold)
|
||||
|
||||
Smoothing time applied when network weights change (after training or perturbation). Prevents audible clicks from sudden output jumps. Options: 0, 5, 10, 20, 50, 100 ms. Default: 10 ms.
|
||||
MEMLNaut can be driven live from the Manifold browser app (`manifold/`, VCV backend) over a
|
||||
WebSocket-to-OSC bridge (`manifold/osc-bridge/bridge.ts`, Deno).
|
||||
|
||||
### Presets (.nisps)
|
||||
1. Right-click the module and enable **OSC server** (default port 7001; each module instance
|
||||
offsets its default port automatically).
|
||||
2. Run the bridge with `--osc-port` matching the module's port.
|
||||
3. In Manifold, select the VCV backend.
|
||||
|
||||
- **Save .nisps preset...** -- Export the full module state (weights, training examples, knob positions, ranges) to a `.nisps` JSON file.
|
||||
- **Load .nisps preset...** -- Import a `.nisps` file, restoring the network and all settings.
|
||||
|
||||
### OSC Bridge
|
||||
|
||||
- **Enable OSC server** -- Start a UDP/WebSocket OSC server for live communication with the companion web app.
|
||||
- **OSC listen port** -- Choose the port (default 9000). Change this if running multiple MEMLNaut instances.
|
||||
|
||||
## Presets
|
||||
|
||||
MEMLNaut uses `.nisps` files for saving and sharing trained networks.
|
||||
|
||||
### What gets saved
|
||||
|
||||
- All network weights (the learned mapping)
|
||||
- All training examples (input/output pairs from thumbs-up)
|
||||
- Knob positions (SPREAD, RATE, attenuverters)
|
||||
- Input/output range settings
|
||||
- Noise level and slew time
|
||||
|
||||
### Saving and loading
|
||||
|
||||
1. Right-click the module.
|
||||
2. Under "Presets (.nisps)", choose **Save** or **Load**.
|
||||
3. Pick a location and filename.
|
||||
|
||||
### Sharing between VCV and the web playground
|
||||
|
||||
The `.nisps` format is shared with the [NISPS web playground](https://musicallyembodiedml.github.io/memlnaut/). However, the VCV module and web app use different network architectures (VCV: 12 outputs, web: 126 outputs), so weights are not directly transferable between them. Training examples and configuration metadata are preserved for reference. The `mlpConfig.layers` field in the file lets each loader detect architecture mismatches.
|
||||
|
||||
### Patch save/load
|
||||
|
||||
Full module state is also saved automatically with your VCV Rack patch file. You do not need to manually export `.nisps` files to preserve your work between sessions.
|
||||
|
||||
## OSC Integration
|
||||
|
||||
MEMLNaut can communicate with the companion web app over OSC for live, bidirectional state sync.
|
||||
|
||||
### Setup
|
||||
|
||||
1. Right-click the module and enable **OSC server** (default port 9000).
|
||||
2. In the web playground, open the OSC connection panel and connect to `localhost:9000`.
|
||||
3. The web app connects via a WebSocket-to-OSC bridge.
|
||||
|
||||
### What syncs
|
||||
What flows over the wire (the complete protocol -- see `docs/specs/vcv-module.md`):
|
||||
|
||||
| OSC Address | Direction | Content |
|
||||
|-------------|-----------|---------|
|
||||
| `/nisps/outputs` | VCV -> Web | Current output values (~10 times/sec) |
|
||||
| `/nisps/inputs` | VCV -> Web | Current input values (~10 times/sec) |
|
||||
| `/nisps/weights` | Both | Full weight transfer |
|
||||
| `/nisps/state` | Both | Complete state sync (weights + examples + config) |
|
||||
| `/nisps/input` | Web -> VCV | Input vector (browser drives the model: "bridged mode") |
|
||||
| `/nisps/output` | VCV -> Web | Current output values (~10 times/sec) |
|
||||
| `/nisps/input` | VCV -> Web | Current input values (~10 times/sec) |
|
||||
| `/nisps/feedback` | Web -> VCV | Verdict ops (up / down / rand / clear) |
|
||||
|
||||
### Multiple instances
|
||||
|
||||
Each MEMLNaut instance needs its own OSC port. Use the port selector in the context menu (9000, 9001, 9002, 8000, 7000) to avoid conflicts.
|
||||
Training is bidirectional: both the browser verdict loop and the module's panel buttons drive
|
||||
the same network. There is no weight/state sync channel -- persistence belongs to the Rack patch.
|
||||
|
||||
## Technical Details
|
||||
|
||||
### Network architecture
|
||||
|
||||
The module uses a multi-layer perceptron (MLP) with the following default architecture:
|
||||
|
||||
```
|
||||
Inputs: 2 (+ 1 bias = 3 input nodes)
|
||||
Inputs: 8
|
||||
Hidden: 16 -> 24 -> 16 (3 hidden layers, ReLU activation)
|
||||
Output: 12 (sigmoid activation, producing values in [0, 1])
|
||||
Output: 16 (sigmoid activation, producing values in [0, 1])
|
||||
```
|
||||
|
||||
The ML engine is the repository's shared `nisps/` core
|
||||
(`nisps::ml::MLPCore<DynamicStorage>` behind the thin adapter `src/iml.hpp`) -- bit-identical
|
||||
training/inference semantics to the MEMLNaut firmware and the browser WASM engine.
|
||||
|
||||
### Inference
|
||||
|
||||
The MLP runs in the VCV `process()` callback. The RATE knob controls how often inference runs, from once per audio block (~170 Hz) to every sample (44.1 kHz). Between inference steps, outputs are linearly interpolated to avoid staircase artifacts.
|
||||
The MLP runs in the VCV `process()` callback. The RATE knob controls how often inference runs,
|
||||
from once per 256 samples (~170 Hz) to every sample (44.1 kHz). Between inference steps, outputs
|
||||
are linearly interpolated to avoid staircase artifacts.
|
||||
|
||||
### Threading
|
||||
|
||||
- **Audio thread**: Reads CV inputs, runs MLP inference, writes CV outputs. Never blocks.
|
||||
- **Background thread**: Handles training (thumbs-up) and weight perturbation (thumbs-down). When complete, signals the audio thread to crossfade to the new outputs.
|
||||
- Each module instance has its own independent background thread and ML engine.
|
||||
|
||||
### Performance
|
||||
|
||||
The default network is small (~20 KB of weights). At block-rate inference, CPU usage is negligible. At audio-rate inference with all 12 outputs patched, expect moderate CPU usage comparable to a complex oscillator module. Multiple instances scale linearly.
|
||||
- **Worker thread**: Handles training (thumbs-up) and weight perturbation (thumbs-down). When
|
||||
complete, signals the audio thread to crossfade to the new outputs.
|
||||
- Each module instance has its own independent worker thread and ML engine.
|
||||
|
||||
## Building
|
||||
|
||||
### Full build commands
|
||||
|
||||
```bash
|
||||
cd vcv
|
||||
export RACK_DIR=/path/to/Rack-SDK
|
||||
make # build the plugin
|
||||
make install # install to VCV plugins directory
|
||||
make clean # remove build artifacts
|
||||
```
|
||||
|
||||
### SDK setup
|
||||
|
||||
1. Download the VCV Rack SDK from https://vcvrack.com/manual/PluginDevelopmentTutorial
|
||||
2. Extract it somewhere (e.g., `~/Rack-SDK`)
|
||||
3. Set `RACK_DIR` to that path, or place it at `~/.local/share/Rack2/Rack-SDK`
|
||||
See `BUILDING.md` (build steps, troubleshooting) and `DISTRIBUTION.md` (packaging `.vcvplugin`
|
||||
bundles, cross-platform builds).
|
||||
|
||||
### Project structure
|
||||
|
||||
```
|
||||
vcv/
|
||||
Makefile # Build configuration
|
||||
Makefile # Build configuration (C++20; core via relative includes)
|
||||
plugin.json # Plugin manifest (name, version, tags)
|
||||
src/
|
||||
plugin.cpp # Plugin initialization
|
||||
plugin.hpp # Plugin globals
|
||||
MEMLNaut.cpp # Module logic, UI, serialization
|
||||
osc_server.hpp # OSC bridge server
|
||||
MEMLNaut.cpp # Module logic, widget/UI, serialization, OSC wiring
|
||||
iml.hpp # Thin adapter over the shared nisps/ ML core
|
||||
osc_server.hpp # OSC bridge server (transport only)
|
||||
LedRing.hpp # Per-output LED ring widget
|
||||
palette.hpp # Color palette from the frontend design tokens
|
||||
res/
|
||||
MEMLNaut.svg # Panel artwork
|
||||
MEMLNaut-wide.svg # 44HP panel (the one the module loads)
|
||||
MEMLNaut.svg # Unused 30HP variant
|
||||
MEMLNaut-expander.svg # Unused 8HP expander panel
|
||||
dist/ # Built .vcvplugin bundles
|
||||
dep/ # Build dependencies
|
||||
```
|
||||
|
||||
### Dependencies
|
||||
|
||||
- **VCV Rack SDK** (v2.x) -- provides the module framework
|
||||
- **nisps-core** -- header-only C++20 ML library (included in the parent repo at `../nisps-core/`)
|
||||
|
||||
No external package manager dependencies are required. The nisps-core headers are referenced directly from the Makefile.
|
||||
|
|
|
|||
Loading…
Reference in a new issue