diff --git a/MAP.md b/MAP.md index 0d181a0..b592645 100644 --- a/MAP.md +++ b/MAP.md @@ -26,7 +26,8 @@ MEMLNaut-NISPS — Neural Interactive Shaping of Parameter Spaces. One C++20 cod - `settings_view.hpp` — `wire_settings(mode)`: adds on-device settings views to the MEMLNaut display carousel (TFT + rotary encoder). Joystick Dual/Single toggle for the 4-input ("two 2-D joystick") modes — "Single" pins ML input channels 2,3 to neutral via `ModeBase::set_input_pinned` (no net rebuild). Registered in the `.ino` after `addSystemInfoView()`. - `firmware/MEMLNaut-NISPS/src/{memllib,daisysp,nisps}` — symlinks (Arduino-CLI requires sketch-tree includes; preprocessor refuses `..` in headers). - `firmware/README.md` — structure + build instructions. -- `firmware/useq-celium/` — standalone RP2040 firmware (PlatformIO, Arduino-Pico core) that turns a uSEQ module + CV expander into a USB→CV/gate converter driven by the manifold `cvgate` backend. `shared/protocol.h` is the v2 wire-protocol single source of truth (mirrored by `manifold/src/backends/useq-protocol.ts`); `main/` (USB serial → CV1–3 + GATE1–3, I2C → expander) and `expander/` (I2C slave → CV4–11). Wire spec: `docs/useq-celium/protocol.md`. Restored from the April-2026 "uSEQ-Celium" mode. +- `firmware/MEMLCelium-upstream/` — **vendored** verbatim snapshot of the upstream `MusicallyEmbodiedML/MEMLNaut-NISPS` @ `main` Arduino sketch (pre-refactor monorepo, does NOT use `nisps/`), preset to `MODE_MEMLCELIUM`. Self-contained: upstream `memllib`@`e291192d` + `memlp`@`ea777502` vendored as plain files; `src/daisysp` in-tree. Built directly with `arduino-cli` (not the repo build scripts) — see its `README.md` for provenance + the compile command. +- `firmware/useq-celium/` — standalone RP2040 firmware (PlatformIO, Arduino-Pico core) that turns a uSEQ module + CV expander into a USB→CV/gate converter driven by the manifold `cvgate` backend. `shared/protocol.h` is the v2 wire-protocol single source of truth (mirrored by `manifold/src/backends/useq-protocol.ts`); `main/` (USB serial → CV1–3 + GATE1–3, I2C → expander) and `expander/` (I2C slave → CV4–11). Wire spec: `docs/specs/useq-cv-protocol.md`. Restored from the April-2026 "uSEQ-Celium" mode. ### `playground/` — SolidJS + Vite + TypeScript app - `playground/index.html`, `vite.config.ts`, `tsconfig.json`, `package.json` — scaffold. COOP/COEP headers configured. @@ -45,8 +46,8 @@ MEMLNaut-NISPS — Neural Interactive Shaping of Parameter Spaces. One C++20 cod ### `manifold/` — Vite + React + TS convertible-mode app (the NEW front-end, WIP) The Manifold "convertible" Console on the real engine, deployed at `meml.lnfinitemonkeys.org/next` (staging, -alongside the live vanilla a-immersive at `/`). Built 2026-06-27/28; see `docs/redesign/BUILD-PLAN.md` (resume -anchor + locked decisions) and the `docs/redesign/*-spec.md` set. +alongside the live vanilla a-immersive at `/`). Built 2026-06-27/28; see `docs/specs/plans/BUILD-PLAN.md` (resume +anchor + locked decisions) and the `docs/specs/*-spec.md` set. - `manifold/src/engine/` — the parity-tested TS engine LIFTED from `playground/src` (same `nisps.wasm`), made framework-neutral: `wasm-iml.ts` (rewired off Solid stores onto an injected `EngineSink`), `engine-host.ts` + `worklet/nisps-processor.ts` (audio), `input-pipeline.ts`/`output-pipeline.ts`/`curves.ts`, `wasm-worker.ts`, @@ -89,7 +90,7 @@ anchor + locked decisions) and the `docs/redesign/*-spec.md` set. ### `vcv/` — VCV Rack 2 plugin (MEMLNaut module, WIP) Native C++ Rack module: ML CV-mapper with RL feedback + a browser bridge. Currently 2→12 (being evolved to **8 inputs × 16 outputs + per-output LED rings**, palette from the frontend tokens, WS↔OSC browser bridge — see -the "BUILD DELTAS" block at the top of `vcv/SPEC.md`). `src/MEMLNaut.cpp` (module), `src/osc_server.hpp` (bridge), +the "BUILD DELTAS" block at the top of `docs/specs/vcv-module.md`). `src/MEMLNaut.cpp` (module), `src/osc_server.hpp` (bridge), `src/plugin.{hpp,cpp}`, `res/*.svg` (panels), `Makefile` (needs `RACK_DIR`). Was built against the retired `nisps-core`; the core include path is being repointed. @@ -170,9 +171,10 @@ See `ALIGNMENT.md`. ## Specs - **Root**: `docs/specs/` -- **Entry**: `README.md` -- **Layout**: `flat` -- **Index**: none +- **Entry**: `MAIN.md` +- **Layout**: flat (with `plans/`, `recon/`, `_archive/` subdirs) +- **Index**: none (intentionally — generate when a consumer exists) - **Skill**: invoke `/specs` to review/maintain/add/navigate. +- **Conventions**: Four-genre ontology — `kind: spec` (timeless contract, wins by intent), `kind: plan` (status: active|executed|superseded, never authority for behaviour), `kind: finding` (dated, immutable, exempt from drift lint), ADRs in `docs/adr/`. -The corpus is a tiered T0–T4 prescriptive plan ("what we are going to build"); only `README.md` plus a handful of files exist so far. Alongside the tiers it also holds **feature specs** — e.g. `slp-workshop-firmware.md`, the single SLP-Workshop spec: Part I (shipped) covers the mode + Jolt / OU-explore learning gestures; Part II (planned) covers the output-mode evolution, gate sequences, and Manifold config UX. Before changing the behaviour a spec covers, find it via `/specs`; the spec wins by intent — if it's wrong, update it in the same commit as the code. +The corpus holds platform-level specs (engine architecture, I/O backends, feedback design), implementation specs (port specs, wire protocols), feature specs (e.g. slp-workshop-firmware.md), historical findings (dated research artifacts), and finite build plans. Before changing behaviour a spec covers, find it via `/specs`; the spec wins by intent — if it's wrong, update it in the same commit as the code. diff --git a/assets/media/README.md b/assets/media/README.md index d51da58..8252553 100644 --- a/assets/media/README.md +++ b/assets/media/README.md @@ -14,7 +14,7 @@ British spelling throughout the on-screen copy (randomise, centre, behaviour). | `01-knob-mapping.mp4` | ~21s | A knob rotates while a paired 0–1 number rises/falls in lockstep. The point: a control is just a readout of one number. | **Strong** — a live knob bound to a number is the simplest possible interactive widget. | Onboarding step 1 / the very first "what is a control" beat. Pairs with the `Slider`/`Knob` primitives. | | `02-xy-pad.mp4` | ~20s | A 2-D pad projects to two independent 0–1 values A and B, each shown as a slider, as the dot moves. | **Strong** — a `JoyMap` driving two live readouts is already close to an in-app primitive demo. | Onboarding step 2. Directly mirrors the `JoyMap` primitive + two `Slider`s. | | `03-fanout.mp4` | ~27s | The same 2-D control driving 3 → 4 → 5 outputs (N sliders one side, control the other), then the control type cycled across knob, fader, touchpad, 2-D joystick, 3-D joystick. | **Medium** — the N-slider fan-out is demoable live; the control-type carousel is better as video. | Onboarding step 3 — the "why a network in the middle" moment, just before the mode UI. Connects to the dimensionality story in the mode schemas. | -| `04-feedback.mp4` | ~53s | Training on the mapping surface (projected to 2-D) and how a verdict reshapes it. **Contrasts the two feedback modes:** geometric push-away (directed repulsion from the liked centroid) vs explore-and-place (randomise → audition → place → interpolate). Ends on a side-by-side and the recommendation. | **Medium/low** — concept-heavy; the explore-and-place loop could become a guided interactive walkthrough, but the contrast slide is best as video. | The learning-behaviour onboarding / the dock-panel `FEEDBACK_MODE` selector help. Coordinates with `docs/redesign/rl-feedback-design.md` (explore-and-place is the recommended default). | +| `04-feedback.mp4` | ~53s | Training on the mapping surface (projected to 2-D) and how a verdict reshapes it. **Contrasts the two feedback modes:** geometric push-away (directed repulsion from the liked centroid) vs explore-and-place (randomise → audition → place → interpolate). Ends on a side-by-side and the recommendation. | **Medium/low** — concept-heavy; the explore-and-place loop could become a guided interactive walkthrough, but the contrast slide is best as video. | The learning-behaviour onboarding / the dock-panel `FEEDBACK_MODE` selector help. Coordinates with `docs/adr/rl-feedback-design.md` (explore-and-place is the recommended default). | `04-feedback.mp4` is `s4_feedback.py`'s three scenes (`GeometricDislike`, `ExploreAndPlace`, `FeedbackContrast`) concatenated. diff --git a/docs/redesign/rl-feedback-design.md b/docs/adr/rl-feedback-design.md similarity index 99% rename from docs/redesign/rl-feedback-design.md rename to docs/adr/rl-feedback-design.md index 5e4f9ac..301aa52 100644 --- a/docs/redesign/rl-feedback-design.md +++ b/docs/adr/rl-feedback-design.md @@ -1,6 +1,6 @@ # RL / Feedback Learning-Engine Design — Manifold -*Design sign-off. Dated 2026-06-27. Supersedes the AVOID-out-of-scope decision in `docs/redesign/feedback-modes-port-spec.md` §2.5/§7. Author: lead architect, synthesising a 4-candidate / 4-judge design panel against verified source.* +*Design sign-off. Dated 2026-06-27. Supersedes the AVOID-out-of-scope decision in `docs/specs/feedback-modes-port-spec.md` §2.5/§7. Author: lead architect, synthesising a 4-candidate / 4-judge design panel against verified source.* --- diff --git a/docs/redesign/README.md b/docs/redesign/README.md new file mode 100644 index 0000000..6575717 --- /dev/null +++ b/docs/redesign/README.md @@ -0,0 +1,3 @@ +# Design Documentation + +Design docs have moved to `docs/specs/` (see `docs/specs/MAIN.md` for the corpus entry point). Only `manifold-export/` (design-token and UI-kit asset export) remains here. diff --git a/docs/specs/MAIN.md b/docs/specs/MAIN.md new file mode 100644 index 0000000..8f7908f --- /dev/null +++ b/docs/specs/MAIN.md @@ -0,0 +1,145 @@ +--- +kind: spec +stability: stable +layer: cross-cutting +--- + +# MEMLNaut-NISPS Spec Corpus + +**Neural Interactive Shaping of Parameter Spaces** — a research platform for interactive ML control of audio. One C++20 codebase compiles to two targets: RP2350 firmware for the MEMLNaut hardware, and WASM in a SolidJS browser playground running the same engines + ML through an AudioWorklet. + +--- + +## 0. Spec Corpus Ontology + +This corpus distinguishes four document kinds by temporal stance and mutability: + +- **`kind: spec`** (flat in `docs/specs/`) — timeless present-tense contracts. **Wins over code by intent.** Drift-lintable; updated when the spec itself changes or code breaks it. +- **`kind: plan`** (in `docs/specs/plans/`) — future-tense, finite prescriptions carrying `status: active | executed | superseded`. Burn down as work lands; NEVER citable as authority for shipped behaviour (executed items migrate to `MAP.md` per the repo's doc-sync rule). +- **`kind: finding`** (in `docs/specs/recon/`) — past-tense, dated, **IMMUTABLE**. Never edited, only superseded; exempt from drift lint. Research artifacts capturing ground truth at a specific moment. +- **ADRs** (in `docs/adr/`) — decision rationale ("why we chose X"). Outside the spec root; used as backlinks from specs that encode those decisions. + +--- + +## Source Files (Architectural Roots) + +The codebase's most architecturally significant areas referenced by this corpus: + +| Layer | Path | Purpose | +|-------|------|---------| +| **Shared C++20** | `nisps/core/`, `nisps/ml/`, `nisps/dsp/`, `nisps/engines/`, `nisps/modes/` | One audio+ML library compiling to both firmware and WASM | +| **Schema/Codegen** | `schemas/`, `codegen/` | Parameter contracts + code generation (C++ headers, TS types) | +| **Firmware** | `firmware/MEMLNaut-NISPS/glue/` | Hardware bindings (audio, peripherals, MIDI, settings) | +| **Browser** | `playground/src/engine/`, `playground/src/audio/` | Headless TS engine + AudioWorklet, wired to Solid stores | +| **Manifold** | `manifold/src/engine/`, `manifold/src/console/` | Same TS engine + React wrapper (the new convertible front-end) | + +--- + +## 1. Frame + +**What it is.** MEMLNaut-NISPS is a single C++20 codebase that compiles once to two distinct targets: + +1. **Firmware** — runs on RP2350 hardware (MEMLNaut instrument). Real-time audio engines (8 variants) + interactive ML (4-layer MLP, SGD + RL feedback) with deterministic RNG + dual-core orchestration (audio on core 1, control on core 0). +2. **Browser (WASM)** — runs in a SolidJS playground (or React Manifold front-end) via AudioWorklet. Same C++20 engines + ML, compiled to WASM; audio engines are a superset of firmware (C15 synth browser-only). + +**The unifying constraint**: one source tree, one ML architecture, cross-platform parity (native ↔ WASM within 1e-5 numerical tolerance). Parameter shapes are JSON schemas with codegen producing both C++ headers and TypeScript types. + +**Performance contract** (applies globally to `nisps/`, inert in WASM but kept for consistency): + +- No heap in audio/inference paths; use `FixedBuffer` or `std::array`. +- Float literals >255 in hot paths must be `static const`; all floats have `.f` suffix. +- No virtual dispatch in audio path (C++20 concepts, not interfaces). +- Deterministic per-instance RNG; all random state is local and seeded at construction. +- Memory section attributes per hardware (AUDIO_MEM, AUDIO_FUNC, HOT, FORCE_INLINE). + +--- + +## 2. Cross-Cutting Contracts + +### Parity discipline + +**Native ↔ WASM bit-equivalence within 1e-5.** The C++ core and its WASM compilation are asserted to produce identical outputs given the same input + same seed. See `tests/cpp/parity_check.cpp` (native test) + `scripts/parity-check.sh` (verify runner). + +### Schema/codegen contract + +Each firmware mode has a `schemas/modes/.json` describing parameters (name, label, range, default, curve, group), ML config (input/output sizes, hidden layers), voice spaces (names; bodies are inline lambdas in the engine), and UI config. The meta-schema at `schemas/schema.json` validates these. Codegen (`bun run codegen/generate.ts`) is idempotent and produces: + +- `nisps/modes/generated/_schema.hpp` — `constexpr` C++ data under `nisps::modes::generated`. +- `playground/src/modes/generated/_schema.ts` — typed const objects + per-mode params interface. + +Regenerate after editing any `schemas/modes/*.json`. Golden test ensures output is byte-identical. + +### No-heap + deterministic RNG + +These are inert in WASM but kept globally to avoid surprises when porting. Lint catches violations: `bash scripts/lint-cpp.sh` warns on missing `.f` suffix and fails on `new`/`malloc`/`std::vector` under `nisps/`, plus `Arduino.h` use outside firmware glue. + +--- + +## Sub-specs (Behavioural + Binding) + +All specs below are stable or evolving (no aspirational ones in shipped features); aspirational specs describe target contracts not yet fully implemented. + +| File | Stability | Layer | Summary | +|------|-----------|-------|---------| +| `aimmersive-clone-spec.md` | evolving | behavioural | Faithful SolidJS rebuild of the deployed vanilla a-immersive app (446-line DOM, 2538-line CSS, 4521-line JS). Component-by-component spec for replicating look/behaviour. | +| `backends-spec.md` | evolving | binding | Output backends (synth, MIDI, OSC, VCV) unified behind one `OutputBackend` interface + registry. One active backend per session; input/ML/output pipelines backend-agnostic. | +| `dock-spec.md` | evolving | behavioural | Console right-dock drawers (Shape/Feel/Route/Health/Help) + per-output controls (off/fixed/live tri-state, mute, solo, curve). Three-depth dock model. | +| `engine-architecture.md` | evolving | cross-cutting | Foundation: headless `EngineApi` boundary separating the pure engine (input→ML→output reactive spine) from presentation skins (a-immersive, Console). No JSX/DOM in engine; skins are pure consumers of `EngineApi` accessors. | +| `feedback-modes-port-spec.md` | evolving | cross-cutting | Implementation spec: "Down Action" negative-feedback feature ported from firmware `InterfaceRL` into `nisps/` core. Three modes (Avoid / RandomiseOutputs / RandomiseMlp), deterministic RNG, per-instance state. | +| `inputs-spec.md` | evolving | binding | Modular input layer (sources: XY pad, MIDI input, gamepad single/double-stick) composing into N-dimensional vector. MLP rebuild on input-set change. Scope: `manifold/` React app. | +| `manifold-parity-features-spec.md` | aspirational | behavioural | Prescriptive spec for porting five playground features into Manifold: session presets, pins, Jolt, OU-Explore, control surface. Awaiting review; no implementation authorised. | +| `slp-workshop-firmware.md` | evolving | binding | SLP-Workshop (Synth Library Portland workshop build). Part I (Jolt + OU explore adaptive-learning gestures) shipped & stable. Part II (output-mode evolution, gate sequences, Manifold config) planned/evolving. | +| `useq-cv-protocol.md` | stable | binding | uSEQ-CV wire protocol v2 (USB Web Serial ↔ uSEQ main module → CV/gate jacks + I2C expander). Single source of truth: `firmware/useq-celium/shared/protocol.h`, mirrored in `manifold/src/backends/useq-protocol.ts`. | +| `vcv-module.md` | evolving | binding | MEMLNaut VCV Rack 2 module: CV-to-CV mapper (8 inputs × 16 outputs) with RL feedback. Embeds the WASM engine; WS↔OSC browser bridge. 2026-06-28 BUILD DELTAS folded in; prior design sections retained for reference. | + +--- + +## Plans (Finite, Burn-Down) + +| File | Status | Summary | +|------|--------|---------| +| `plans/BUILD-PLAN.md` | active | Manifold build resume anchor (dated 2026-06-27). Locked decisions: React app in `manifold/`, parity-tested TS engine from `playground/src`, staging deploy at `meml.lnfinitemonkeys.org/next`, default feedback mode = Explore-and-Place. | +| `plans/playground-2.0-rewrite-plan.md` | executed | SolidJS clean-room rewrite plan (June 2026). Largely implemented in Manifold + playground foundation: one fullscreen instrument, Console interaction model, right-edge dock with three depths, snapshot DAG, control-point tri-state. | + +--- + +## Findings (Immutable, Dated Research) + +All findings are dated 2026-06-27 unless otherwise noted; exempt from drift lint. + +| File | Date | Summary | +|------|------|---------| +| `recon/findings-design-and-manifold.md` | 2026-06-27 | Build-oriented brief synthesizing five redesign docs + Manifold token export + `ConsoleApp.jsx`. VERIFIED from source. | +| `recon/findings-engine-surface.md` | 2026-06-27 | Engine surface audit (deployed JS, WASM, C++ core). Uncovers fixed-2-input WASM gap, loss-history plumbing gap, C15 placeholder. | +| `recon/findings-feedback-behaviour.md` | 2026-06-27 | Current RL feedback behaviour audit. Deployed JS + C++ core use undirected Gaussian noise (not geometric push-away); firmware InterfaceRL implements the true geometric mode. | +| `recon/midi-gamepad-inputs-worklog.md` | 2026-06-27 | Work log of `feat/midi-inputs` branch. Modular input sources (MIDI, gamepad), 32-input WASM foundation, gamepad→verdict wiring. Exclusive picker (not mixing) shipped; mixing engine groundwork done but UI deferred. | +| `recon/upstream-firmware-survey.md` | 2026-06-27 | Git archaeology of the MusicallyEmbodiedML ecosystem. origin/main (C++20/SolidJS rewrite) and upstream/main (old `.ino` firmware) forked at `6efbe9c` (2026-04-14); 49 commits upstream not in origin are ports, not merges. | +| `recon/playground-2026.md` | 2026-04-12 | Design intent snapshot of an unfinished playground UI redesign. Reference-only for SolidJS rewrite; do NOT merge into vanilla playground (those files moved on independently). | + +--- + +## ADRs (Decision Rationale) + +| File | Decision | Audience | +|------|----------|----------| +| `adr/rl-feedback-design.md` | Rationale for the Explore-and-Place + Geometric-Dislike feedback model choices | Implementation team | + +--- + +## Cross-References + +- **`MAP.md`** — the neutral inventory of the codebase as it stands (ground truth for what exists). +- **`CLAUDE.md`** — long-form architecture narrative (entry point for agent onboarding). +- **`ALIGNMENT.md`** — strategic gaps + open mission questions (dated, opinionated gap diagnosis). +- **`manifold/ONBOARDING.md`** — agent-oriented breakdown of the Manifold React app (run/build/deploy/test, UI/engine-spine/WASM layering, convertible Stages, Dock+drawers, gotchas). +- **`docs/redesign/manifold-export/`** — design-token and UI-kit asset export from Manifold design project (not part of this corpus; static asset reference, not migrated). +- **`anima/`** — separate project (excluded from this corpus). + +--- + +## Open / Deferred + +- **Index generation (`.index.json`)**: intentionally not generated yet. No consumer defined; add when crawler/search infra lands. +- **Per-spec §-numbering normalisation**: deferred to a future maintainer pass. Current state mixes flat (§1.1, §1.2…) and three-level nesting (§6.x.y). +- **ADR corpus**: currently minimal (only RL feedback design). Strategic decisions captured in spec Open/Deferred sections; promotion to dedicated ADRs is a future pass. +- **Spec-touched validator**: git-aware lint rule that flags when a code diff touches spec-cited files without also touching the spec. Useful as a pre-commit hook or PR check; infrastructure TBD. diff --git a/docs/specs/README.md b/docs/specs/_archive/tiered-plan-README.md similarity index 88% rename from docs/specs/README.md rename to docs/specs/_archive/tiered-plan-README.md index 29c2392..0444805 100644 --- a/docs/specs/README.md +++ b/docs/specs/_archive/tiered-plan-README.md @@ -1,6 +1,13 @@ -# Tiered Specs — MEMLNaut-NISPS +--- +deprecated-by: 2026-07-13 +superseded-by: MAIN.md +--- -This corpus is the **prescriptive plan** for the project: what we are building and why, organised from +# Tiered Specs — MEMLNaut-NISPS (Deprecated) + +**Note:** This tiered planning approach was not built out; the corpus adopted a flat MAIN.md layout instead (see `docs/specs/MAIN.md`). + +This corpus was the **prescriptive plan** for the project: what we are building and why, organised from high-level intent down to implementation specifics. It is distinct from the two orienting docs at the repo root: diff --git a/docs/redesign/aimmersive-clone-spec.md b/docs/specs/aimmersive-clone-spec.md similarity index 99% rename from docs/redesign/aimmersive-clone-spec.md rename to docs/specs/aimmersive-clone-spec.md index d961a80..cd54fc8 100644 --- a/docs/redesign/aimmersive-clone-spec.md +++ b/docs/specs/aimmersive-clone-spec.md @@ -1,3 +1,9 @@ +--- +kind: spec +stability: evolving +layer: behavioural +--- + I now have a thorough, complete understanding of the entire deployed app — DOM, CSS theme tokens, all interaction handlers, state model, engine call points, persistence, and the help-modal copy. I have everything needed to write the faithful clone spec. Let me write it now. # NISPS Immersive (`a-immersive.html`) — SolidJS Clone Spec diff --git a/docs/redesign/backends-spec.md b/docs/specs/backends-spec.md similarity index 98% rename from docs/redesign/backends-spec.md rename to docs/specs/backends-spec.md index 69e9556..fc58e60 100644 --- a/docs/redesign/backends-spec.md +++ b/docs/specs/backends-spec.md @@ -1,4 +1,11 @@ -# Output Backends — Specification (`docs/redesign/backends-spec.md`) +--- +kind: spec +stability: evolving +layer: binding +counterpart: aimmersive-clone-spec.md +--- + +# Output Backends — Specification *Workstream E. Design-only, read-only audit 2026-06-27. The new app is Vite + React + TS in `manifold/`, wired to the parity-tested TS engine lifted from `playground/src`. British spelling in product copy. The built-in synth is the **"Powerful Synth Engine"** — the string "C15" must never reach the user.* @@ -465,9 +472,9 @@ Per the search, the standard flow is `export RACK_DIR=…; make clean; make dist - MIDI: `/home/w1n5t0n/deployments/meml-aimmersive/js/midi/midi-output.js` (`:114` batch throttle), `…/midi/midi-cc-map.js` - OSC client: `/home/w1n5t0n/deployments/meml-aimmersive/js/nisps/osc-client.js`; param-named client `…/js/synth/osc-output.js` (`:97`) - OSC bridge server: `/home/w1n5t0n/deployments/meml-aimmersive/osc-bridge/bridge.ts` (addresses `:298–305`, encode/decode `:70–182`), `bridge.mjs`, `compile.sh` -- VCV module: `/home/w1n5t0n/src/MEMLNaut-NISPS/vcv/src/MEMLNaut.cpp` (threading `:67–312`, verdict `:412–445`, ranges `:818–843`, display `:699`, LED `:804`), `…/vcv/src/osc_server.hpp`, `…/vcv/plugin.json`, `…/vcv/SPEC.md`, `…/vcv/Makefile`, `…/vcv/NISPS-FORMAT.md`, `…/vcv/res/*.svg` +- VCV module: `/home/w1n5t0n/src/MEMLNaut-NISPS/vcv/src/MEMLNaut.cpp` (threading `:67–312`, verdict `:412–445`, ranges `:818–843`, display `:699`, LED `:804`), `…/vcv/src/osc_server.hpp`, `…/vcv/plugin.json`, `…/docs/specs/vcv-module.md`, `…/vcv/Makefile`, `…/vcv/NISPS-FORMAT.md`, `…/vcv/res/*.svg` - Design tokens (ring palette): `/home/w1n5t0n/src/MEMLNaut-NISPS/docs/redesign/manifold-export/tokens/colors.css` -- Spine/engine context: `/home/w1n5t0n/src/MEMLNaut-NISPS/docs/redesign/engine-architecture.md` (§2), `…/findings-design-and-manifold.md` (§4), `…/findings-engine-surface.md`, `…/aimmersive-clone-spec.md` (routeOutputs §6, §7 visual table, §10) +- Spine/engine context: `/home/w1n5t0n/src/MEMLNaut-NISPS/docs/specs/engine-architecture.md` (§2), `…/recon/findings-design-and-manifold.md` (§4), `…/recon/findings-engine-surface.md`, `…/aimmersive-clone-spec.md` (routeOutputs §6, §7 visual table, §10) Sources (VCV SDK / widgets): [VCV custom lights](https://community.vcvrack.com/t/how-to-use-custom-lights/1941), [Migrate2 (drawLayer/layer 1)](https://vcvrack.com/manual/Migrate2), [Plugin Development Tutorial (RACK_DIR/make dist)](https://vcvrack.com/manual/PluginDevelopmentTutorial), [Plugin API Guide](https://vcvrack.com/manual/PluginGuide). --- @@ -493,7 +500,7 @@ spine; WebMIDI out with per-output CC#/ch/name/range; OSC-over-WS to the Deno br The Outputs dock panel specialises per backend (`manifold/src/dock/OutputsBackendConfig.tsx`) with named presets (`manifold/src/backends/presets.ts`). Audio gated via `engine.audio.setMuted` on non-synth modes. -**VCV module** — see `vcv/SPEC.md` "⚠️ BUILD DELTAS (2026-06-28)" for the authoritative build target: 8 inputs × +**VCV module** — see `docs/specs/vcv-module.md` "⚠️ BUILD DELTAS (2026-06-28)" for the authoritative build target: 8 inputs × 16 outputs, an LED ring per output (drawLayer + nvgArc), palette from the frontend tokens, WS↔OSC bridge (browser OSC backend → `manifold/osc-bridge/` Deno relay → the module's OSC server), bidirectional training, and the `nisps-core`→`nisps/` core-path repoint. The existing `vcv/` module (2→12) is evolved, not rebuilt. diff --git a/docs/redesign/dock-spec.md b/docs/specs/dock-spec.md similarity index 98% rename from docs/redesign/dock-spec.md rename to docs/specs/dock-spec.md index 8a625ee..228861b 100644 --- a/docs/redesign/dock-spec.md +++ b/docs/specs/dock-spec.md @@ -1,8 +1,14 @@ +--- +kind: spec +stability: evolving +layer: behavioural +--- + # Dock Spec — Console Right-Dock Drawers + Per-Output Controls -*Workstream D. Read-only design, 2026-06-27. Target app: `manifold/` (Vite + React + TS), wired to the parity-tested TS engine (`playground/src`) via the headless `EngineApi` boundary (`docs/redesign/engine-architecture.md` §2). Citations are `file:line`. British spelling in product copy. The built-in synth is the **"Powerful Synth Engine"** — the string "C15" MUST NEVER appear in the UI.* +*Workstream D. Read-only design, 2026-06-27. Target app: `manifold/` (Vite + React + TS), wired to the parity-tested TS engine (`playground/src`) via the headless `EngineApi` boundary (`engine-architecture.md` §2). Citations are `file:line`. British spelling in product copy. The built-in synth is the **"Powerful Synth Engine"** — the string "C15" MUST NEVER appear in the UI.* -This spec replaces the placeholder Shape/Feel/Route/Health/Help drawers in `ConsoleApp.jsx` (`docs/redesign/findings-design-and-manifold.md` §2 — keyboard `1-5` map drawers, `\` toggles depth) with the real dock contents. +This spec replaces the placeholder Shape/Feel/Route/Health/Help drawers in `ConsoleApp.jsx` (`recon/findings-design-and-manifold.md` §2 — keyboard `1-5` map drawers, `\` toggles depth) with the real dock contents. --- diff --git a/docs/redesign/engine-architecture.md b/docs/specs/engine-architecture.md similarity index 96% rename from docs/redesign/engine-architecture.md rename to docs/specs/engine-architecture.md index 86c7058..c936ce8 100644 --- a/docs/redesign/engine-architecture.md +++ b/docs/specs/engine-architecture.md @@ -1,17 +1,12 @@ -I now have everything I need. Key findings confirmed: - -- The **engine layer is genuinely good and parity-tested**: `WasmIML`, `EngineHost` (already uses `?worker&url` + hand-rolled imports), `nisps-processor.ts`, the pure pipelines, the stores. These are reusable. -- The **debt is concentrated** in `mode-runtime.ts` (663-line god-hook), the 9 cloned `*Mode.tsx`, `ModeShell`/`ModeSwitcher`, and `App.tsx` (toy nav). The pull-vs-push reactivity bug lives in the god-hook (output flows through imperative `setInput` → `mlStore.process()` → effects, not a clean memo chain). -- **Serving**: nginx root is `meml-aimmersive` (the legacy vanilla a-immersive at `a-immersive.html`, served untouched). The auto-deploy already builds `playground/dist` into `/home/w1n5t0n/deployments/meml.lnfinitemonkeys.org/playground/dist` but it's **currently unserved** (no location block points to it). COOP/COEP is server-scoped. The deploy clone is separate from the dev repo. -- `FeedbackController` = the RL feedback bindings (`thumbsUp`/`thumbsDown`/`moveWeights`/`drawWeights` against the 3-mode loop), not a named class. - -I have enough to write the spec. It's a foundation-architecture document, not a code change, so I'll return it directly. - +--- +kind: spec +stability: evolving +layer: cross-cutting --- # NISPS Foundation Architecture — One Engine, Two Skins -*Status: implementation-ready spec. Scope: the foundation only — the headless engine layer plus the two-skin shell. The detailed UX of each skin (Console drawers, interactive heatmap craft, etc.) is owned by `docs/redesign/playground-2.0-rewrite-plan.md` and not re-litigated here.* +*Status: implementation-ready spec. Scope: the foundation only — the headless engine layer plus the two-skin shell. The detailed UX of each skin (Console drawers, interactive heatmap craft, etc.) is owned by `plans/playground-2.0-rewrite-plan.md` and not re-litigated here.* *Constraint from the operator: "lay solid ground that's simple and doesn't paint us into a corner." Build BOTH skins on ONE engine; don't port the engine twice; don't inherit existing tech debt.* @@ -430,8 +425,8 @@ The deliberate ordering choice vs the plan: **establish `EngineApi` + both skin --- ### Relevant file paths -- Authoritative UX/feature plan (owns the skins' detail): `/home/w1n5t0n/src/MEMLNaut-NISPS/docs/redesign/playground-2.0-rewrite-plan.md` -- Design-intent reference for skin-a-immersive: `/home/w1n5t0n/src/MEMLNaut-NISPS/docs/redesign/playground-2026.md` +- Authoritative UX/feature plan (owns the skins' detail): `/home/w1n5t0n/src/MEMLNaut-NISPS/docs/specs/plans/playground-2.0-rewrite-plan.md` +- Design-intent reference for skin-a-immersive: `/home/w1n5t0n/src/MEMLNaut-NISPS/docs/specs/recon/playground-2026.md` - The god-hook to discard (the debt): `/home/w1n5t0n/src/MEMLNaut-NISPS/playground/src/modes/mode-runtime.ts` - Engine modules to lift: `/home/w1n5t0n/src/MEMLNaut-NISPS/playground/src/audio/engine-host.ts`, `…/audio/worklet/nisps-processor.ts`, `…/ml/wasm-iml.ts`, `…/ml/wasm-worker.ts`, `…/ml/dataset.ts`, `…/input/pipeline.ts`, `…/output/pipeline.ts`, `…/output/curves.ts`, `…/stores/bus.ts`, `…/stores/ml-store.ts`, `…/primitives/*` - Codegen TS types to extend (`capability_class`+`tier`): `/home/w1n5t0n/src/MEMLNaut-NISPS/playground/src/modes/generated/types.ts` diff --git a/docs/redesign/feedback-modes-port-spec.md b/docs/specs/feedback-modes-port-spec.md similarity index 99% rename from docs/redesign/feedback-modes-port-spec.md rename to docs/specs/feedback-modes-port-spec.md index 2013208..70f4268 100644 --- a/docs/redesign/feedback-modes-port-spec.md +++ b/docs/specs/feedback-modes-port-spec.md @@ -1,3 +1,9 @@ +--- +kind: spec +stability: evolving +layer: cross-cutting +--- + I have complete grounding now. I'll write the implementation-ready spec. # Implementation Spec — "Down Action" Negative-Feedback Feature in the `nisps/` Core diff --git a/docs/redesign/inputs-spec.md b/docs/specs/inputs-spec.md similarity index 98% rename from docs/redesign/inputs-spec.md rename to docs/specs/inputs-spec.md index b73e82a..8dbd82e 100644 --- a/docs/redesign/inputs-spec.md +++ b/docs/specs/inputs-spec.md @@ -1,6 +1,13 @@ +--- +kind: spec +stability: evolving +layer: binding +counterpart: aimmersive-clone-spec.md +--- + # Inputs Spec — Modular Input Layer (Workstream F) -*Status: implementation-ready spec for the `manifold/` React app. Scope: the modular input layer — sources (XY pad, MIDI input, gamepad single/double-stick), how they compose into an N-dimensional input vector, the MLP-rebuild-on-input-change mechanism, the binding to the reactive spine, and the dock INPUTS panel. Read alongside `docs/redesign/findings-engine-surface.md` (the fixed-2-input gap), `docs/redesign/engine-architecture.md` (the `EngineApi` seam — note that doc says SolidJS/`playground2`; this app is React/`manifold/`), and `docs/redesign/aimmersive-clone-spec.md` (existing gamepad/MIDI/joystick behaviour). British spelling in product copy. The built-in synth is always shown as the "Powerful Synth Engine" — never "C15".* +*Status: implementation-ready spec for the `manifold/` React app. Scope: the modular input layer — sources (XY pad, MIDI input, gamepad single/double-stick), how they compose into an N-dimensional input vector, the MLP-rebuild-on-input-change mechanism, the binding to the reactive spine, and the dock INPUTS panel. Read alongside `recon/findings-engine-surface.md` (the fixed-2-input gap), `engine-architecture.md` (the `EngineApi` seam — note that doc says SolidJS/`playground2`; this app is React/`manifold/`), and `aimmersive-clone-spec.md` (existing gamepad/MIDI/joystick behaviour). British spelling in product copy. The built-in synth is always shown as the "Powerful Synth Engine" — never "C15".* --- diff --git a/docs/specs/manifold-parity-features-spec.md b/docs/specs/manifold-parity-features-spec.md new file mode 100644 index 0000000..82e7804 --- /dev/null +++ b/docs/specs/manifold-parity-features-spec.md @@ -0,0 +1,479 @@ +--- +kind: spec +stability: aspirational +layer: behavioural +--- + +# Manifold Parity Features — Session Presets · Pins · Jolt · OU-Explore · Control Surface + +*Status: draft — awaiting review. Date: 2026-07-12.* +*Scope: prescriptive spec for porting five playground features into `manifold/`. Spec only — no +implementation is authorised by this document; break into ergo issues after sign-off.* + +**Traces up to:** `playground-2.0-rewrite-plan.md` §2.3 (Feel drawer), §2.6 (pins), §3.8 +(composed-layers presets); `engine-architecture.md` §3.2 (fanout, control-point tri-state), §3.8 +(persist helper); `docs/specs/slp-workshop-firmware.md` §3–4 (Jolt / OU-explore, Part I shipped in +firmware); `dock-spec.md` (drawer depths, tri-state semantics); BUILD-PLAN locked decisions +(parity-tested engine; British spelling; the synth is "Powerful Synth Engine", never the forbidden +string). + +**Reference implementations:** the playground versions are the behavioural ground truth for +constants and algorithms — `playground/src/features/session-preset.ts`, +`stores/session-store.ts` (pins), `ml/jolt.ts`, `output/ou-explore.ts`, `stores/control-store.ts` + +`features/control-routing.ts`. Where this spec and the playground disagree, this spec wins (each +divergence is called out and justified inline). + +--- + +## 0. Principles applied throughout + +1. **C++ owns gesture math where a C++ class exists.** `nisps::ml::Jolt` and + `nisps::ml::OUNoise` already exist (`nisps/ml/jolt.hpp`, `nisps/ml/ou_noise.hpp`) but are + not exposed to WASM. Manifold binds them rather than re-porting to TS — this closes two + documented `--- C++ GAP ---` items and buys deterministic, firmware-parity noise for free. + (The playground's TS reimplementations used `Math.random()`; that shortcut is *not* carried + over.) +2. **No new mechanisms where an existing one already expresses the idea.** Param pins are the + existing `off|fixed|live` tri-state + arm mask, not a parallel pin system (§2.2). +3. **Transparent defaults.** Every default constant appears in this spec as a number with its + source, and the control surface UI must *show* the derived per-param values live (§5.6) — + the user should never wonder what an axis is secretly doing. +4. **One persistence pattern.** All new persisted state goes through a single versioned + `persist(key, version, migrate)` helper modelled on `settings-store.ts` + (localStorage, debounced 200 ms), per `engine-architecture.md` §3.8. +5. **Probe parity.** Each feature activates its currently-inert `window.__nisps` methods + (`manifold/src/debug/probe.ts`) so Playwright can drive it headlessly. New e2e specs are part + of each feature's acceptance criteria. + +--- + +## 1. Session presets (composed layers) + +### 1.1 Model + +Adopt the composed-layers model prescribed by `engine-architecture.md` §3.8 rather than the +playground's monolithic `SessionPresetPayload`. A preset is a bundle of independently optional +layers: + +```ts +interface SessionPresetV1 { + v: 1; + id: string; // `preset-${epoch36}` + name: string; + createdAt: number; + layers: { + control?: ControlLayer; // axes + offsets + presetId (§5) + pipelines?: PipelineLayer; // input/output pipeline params + routing?: RoutingLayer; // per-output MFParam state/min/max/curve/mute + backend specs + mode?: ModeLayer; // modeId + outputMode + weights?: string; // base64url Float32Array — OPT-IN only + }; +} +``` + +- `ControlLayer` = `{ boldness, memory, precision, presetId, offsets }` exactly as §5.4 state. +- `PipelineLayer` = the configurable params of `engine/input-pipeline.ts` and + `output-pipeline.ts` (deadzone, zoom, invert, curve, smoothing, momentum; global curve, + smoothing, slew, freeze). +- `RoutingLayer` = serialised `MFParam[]` control fields + the per-backend specs already handled + by the `OutputsBackendConfig` preset bar. The existing backend preset bar becomes a *view onto + the routing layer* — one storage model, two entry points (see §1.4). +- `weights` uses the playground's base64url `Float32Array` codec verbatim + (`session-preset.ts:63–98`): standard base64 with `+/` → `-_`, padding stripped; byte length + rounded down to a 4-byte boundary on decode; size-mismatch on restore is silently skipped. + +Restore is best-effort and layer-wise: absent layers leave current state untouched; unknown keys +inside a layer are ignored (forward compatibility). + +### 1.2 Persistence + +- Key: `mf-session`, via the shared `persist` helper, version field `v: 1`, 200 ms debounce. +- Stored shape: `{ v: 1, presets: SessionPresetV1[], regionPins: RegionPin[] }` (pins ride in + the same document, §2). +- Weights are **excluded by default** on save; the save UI offers an explicit + "include network weights" toggle (default off). Rationale: weight blobs are ~KB-scale × + N presets and stale weights surprise users; matches playground `withWeights=false` default. + +### 1.3 URL sharing + +Carry over both playground URL forms (`session-preset.ts:249–283`): + +- **Compact axis form** — `?boldness=0.5&memory=0.5&precision=0.3`: applies only the control + layer; each value clamped to [0,1]. +- **Full form** — `?session=`: applied on first load, + after engine ready, before first render of the dock. Never include weights in a generated + share URL (URL length); the share button produces the full form minus `weights`. + +URL params are read once at boot in `App.tsx` (same place the `?debug=1` probe is installed) and +then stripped from the address bar via `history.replaceState` so a reload doesn't re-apply over +user changes. + +### 1.4 UI + +- **Save/restore/rename/delete** lives in a preset bar at the top of the **route drawer's + expanded depth**, reusing the exact interaction pattern of + `dock/OutputsBackendConfig.tsx` (save-as / restore / rename / delete). The existing + backend-scoped preset bar remains; a session preset is the superset bundle. +- **Share** button beside the bar copies the URL to the clipboard and flashes confirmation. +- Restoring a preset that contains a `mode` layer while a different mode is active shows the + same confirm modal used for net-reset (input-layer reshape) — mode switches discard training + state and must not be silent. + +### 1.5 Probe + acceptance + +- Probe: `saveSessionPreset(name, withWeights?)`, `loadSessionPreset(id)`, `listSessionPresets()`, + `buildShareUrl()`, `applySession(json)`. +- e2e: save → mutate axes/routing → restore → assert state round-trips; share URL → fresh page + load with `?session=` → assert layers applied; weights round-trip when opted in + (`getWeights()` equality within exact bits). + +--- + +## 2. Region & param pins + +### 2.1 Region pins + +**Type (unchanged from playground `session-store.ts:34–44`):** + +```ts +interface RegionPin { + id: string; // `pin-${epoch36}-${rand}` + x: number; y: number; // bottom-left corner, [0,1] input space + width: number; height: number; // [0,1] + colourSlot: number; // 0..4 → --pin-1..--pin-5 tokens + createdAt: number; +} +``` + +- **Cap: 5 pins, FIFO** (oldest evicted). Colour slot defaults to `pins.length % 5`. +- **Colour tokens** join `manifold/src/styles/tokens/colors.css` with the playground values: + fills `--pin-1..5` at 0.25 alpha (orange `255,106,0` / cyan `0,204,255` / purple + `180,100,255` / green `80,200,120` / tan `255,200,80`), borders same hues at 0.7 alpha, + 2 px logical border width. +- **Creation gesture:** long-press (600 ms — the constant Manifold.tsx already uses) on the + Manifold stage pins the *currently visible region* (whole surface when unzoomed; the zoom + window once input zoom exists in manifold — until then, `{x:0,y:0,w:1,h:1}` is not useful, so + the long-press instead opens a small radial affordance: "pin region around cursor" + creating a 0.25×0.25 region centred on the press point, clamped to [0,1]²). + *Divergence from playground (which pins the zoom window) because manifold has no input zoom + yet; revisit when the inputs workstream lands zoom.* +- **Removal:** tap a pin border → small popover with colour dot + "remove"; "clear pins" action + in the settings drawer. +- **Rendering:** `Manifold.tsx` already accepts `pins: Pin[]` — extend the `Pin` type in + `console/types.ts` to the full `RegionPin` shape (it currently has only `x/y/colour`), and + render rects behind trail/markers, in front of the grid. Long-press must not fire when the + press initiates a drag > 8 px (existing drag threshold logic) or while `picking` is true + (explore-and-place anchor picking takes priority). +- **Persistence:** in the `mf-session` document (§1.2). +- **Semantics:** purely annotational in v1 (visual memory of "good areas"). They do **not** + constrain training. A future spec may bind snapshot anchors to pins. + +### 2.2 Param pins = the existing tri-state (decision, not new code) + +The playground's separate `ParamPin` list + `paramPinMask()` is **not ported**. Manifold already +has a strictly more expressive mechanism, per `dock-spec.md` §3.3: + +- `off` — excluded from model control *and* pinned out of training; +- `fixed` — held at a static value, pinned out of training; +- `live` — model-driven, trainable. + +"Pinning a param" in manifold = setting it to `fixed` (hold current value) via the existing +`OutputControlRow` segmented control or by clicking the value readout in `OutputStage`. + +**Gap this spec closes:** the tri-state must actually feed the training mask. Requirement: the +arm/focus mask sent through `_nisps_ml_feedback_set_focus` (and any `moveWeights`-style +perturbation mask) is computed as `armed ∧ (state === 'live')` — `off`/`fixed` columns are never +perturbed and never accumulate gradient. Owner: `dock/output-state.ts` `buildArmMask()`. + +Probe: `pinParam(i)` / `unpinParam(i)` map to `setParam(i, {state:'fixed'|'live'})`; the inert +playground-compat methods in `probe.ts` are rewired to these. + +Acceptance: e2e asserts (a) a `fixed` output's value is bit-stable across training and jolt; +(b) region pins survive reload; (c) 6th region pin evicts the 1st. + +--- + +## 3. Jolt + +### 3.1 What it is + +A held-gesture weight morph: while held, N randomly-chosen weights each glide (EMA) toward their +own random target, re-rolling the target on arrival — a continuous, directional "stirring" of +the network. On release the net freezes where it landed, and the learning rate ramps back up +from 0 so training doesn't immediately fight the new position. Ported from upstream +`InterfaceRL`; C++ implementation shipped in firmware (SLP workshop Part I). + +### 3.2 Binding, not porting + +Expose the existing `nisps::ml::Jolt` through the C ABI. New bindings in +`nisps/wasm/bindings.cpp` + `scripts/build-wasm.sh` EXPORTED_FUNCS: + +``` +_nisps_ml_jolt_press(ml) // picks indices+targets from the MLP's weight count +_nisps_ml_jolt_step(ml) // one tick: mutates weights in place, then caller reprocesses +_nisps_ml_jolt_release(ml) +_nisps_ml_jolt_active(ml) -> i32 +_nisps_ml_jolt_lr_scale(ml) -> f32 // 0 while held; ramps 0→1 after release +_nisps_ml_jolt_tick_lr_ramp(ml) +_nisps_ml_jolt_seed(ml, u64) +``` + +The Jolt instance lives beside the MLP handle inside the WASM module (one per net), seeded from +the session seed so runs are reproducible. Parity CI: extend `tests/cpp/parity_check.cpp` with a +golden jolt sequence (seed → press → k steps → weight vector) asserted native-vs-WASM. + +### 3.3 Constants (defaults — from `nisps/ml/jolt.hpp:42–49`, upstream-verbatim) + +| Constant | Value | Meaning | +|---|---|---| +| `num_weights` | 40 | weights morphed simultaneously (hard cap 64) | +| `morph_rate` | 0.017 | EMA per tick: `w += 0.017·(target − w)` | +| `target_min / max` | −1.2 / 0.9 | random target range | +| `target_epsilon` | 0.05 | re-roll target when within this distance | +| `lr_ramp_step` | 0.001 | post-release LR ramp per tick (~5 s at 200 Hz) | +| tick period | 5 ms (200 Hz) | `JOLT_TICK_MS`, matches firmware control rate | + +### 3.4 Manifold integration + +- **Driver:** a `setInterval(5 ms)` in `FeedbackController` while active: `jolt_step` → + `engine.reprocess()` (the spine already re-ticks the last input after weight changes) → bump + version. LR suppression: training calls multiply their LR by `jolt_lr_scale()`; the ramp is + ticked from the same driver until it reaches 1.0. +- **Snapshot before press:** entering jolt pushes an auto-snapshot (tag `before jolt`) so undo + recovers the pre-jolt net — same behaviour as playground `mode-runtime.ts:551`. +- **UI:** a press-and-hold **⚡ Jolt** button in the **learn drawer** (condensed depth, beside + the VerdictCluster-adjacent controls) and keyboard hold `J`. Visual: the Manifold noise ring + pulses while active. Pointer-cancel/blur must release (never a stuck jolt). +- **Interlocks:** disabled while `exploring` (explore-and-place owns the net during a scratchpad + session); `off`/`fixed` outputs are unaffected by definition (§2.2 mask applies only to + training — jolt perturbs *weights*, so v1 explicitly documents that jolt ignores per-output + pins; a per-column jolt mask is out of scope). + +### 3.5 Probe + acceptance + +Probe: `joltPress()`, `joltRelease()`, `joltActive()`. e2e: press → 200 ms → assert outputs +changed and `getWeights()` differs; release → assert weights stable; undo restores pre-jolt +weights; determinism: same seed + same step count ⇒ identical weight vector twice. + +--- + +## 4. OU-explore noise + +### 4.1 What it is + +Ornstein–Uhlenbeck drift added to the *output vector* (not the weights): each output channel +carries a state that takes small random steps while being pulled back toward zero, producing +smooth, temporally-correlated wander around whatever the net currently outputs. Learning stays +live throughout — it is an audition aid, not a training mode. C++ `OUNoise` shipped in +firmware (SLP workshop Part I). + +Discrete update per channel (Euler–Maruyama, μ = 0): + +``` +state += theta · (−state) · dt + noise_scale · N(0,1) +out = clamp(out + state, 0, 1) +noise_scale = stationary_std · √(2·theta·dt), stationary_std = intensity · 0.65 +``` + +### 4.2 Binding + +Instantiate `nisps::ml::OUNoise<126>` (the WASM output width) beside the MLP handle. Bindings: + +``` +_nisps_ml_ou_set_intensity(ml, f32) // [0,1] → stationary_std = v·0.65; 0 disables +_nisps_ml_ou_intensity(ml) -> f32 +_nisps_ml_ou_apply(ml, out_ptr, n) // advance state + add + clamp, in place +_nisps_ml_ou_reset(ml) +_nisps_ml_ou_set_theta(ml, f32) _nisps_ml_ou_set_dt(ml, f32) +_nisps_ml_ou_seed(ml, u64) +``` + +### 4.3 Constants (defaults — `nisps/ml/ou_noise.hpp`, upstream-verbatim) + +| Constant | Value | +|---|---| +| `theta` (mean reversion) | 0.02 | +| `dt` | 0.001 | +| `kOUMaxAmplitude` (full-scale stationary std) | 0.65 | +| default intensity | 0 (**inert** — `apply()` is a no-op, preserving current behaviour/parity) | +| idle tick period | 30 ms (`EXPLORE_TICK_MS`) | + +### 4.4 Manifold integration + +- **Where in the chain:** in the spine, after the TS output-pipeline (curve→smoothing→slew→ + freeze) and before the backend send — i.e. `ou_apply` on the routed buffer. This matches the + playground ordering (post-pipeline, pre-overrides). `fixed`/`off` outputs are re-asserted + *after* `ou_apply` so pinned values do not wander (this is the §2.2 guarantee). +- **Idle ticking:** when intensity > 0 and no input events arrive, a 30 ms interval calls + `reprocess()` so the wander is audible with a static joystick. Interval torn down and + `ou_reset()` called when intensity returns to 0. +- **UI:** an **Explore** slider (0–1) in the **learn drawer** condensed depth, next to the + existing Noise/Spread knobs; the Manifold noise ring radius reflects + `noiseCap + ou_intensity·0.65` so the visual language stays truthful. +- **Interaction with Jolt:** independent and composable (jolt stirs weights, OU wanders + outputs); both default inert. + +### 4.5 Probe + acceptance + +Probe: `setExploreIntensity(v)`, `getExploreIntensity()`. e2e: intensity 0.5 + static input ⇒ +outputs vary over 500 ms with bounded step size; intensity 0 ⇒ outputs bit-stable; `fixed` +output does not wander; determinism under seeded runs. + +--- + +## 5. Boldness / Memory / Precision control surface + +### 5.1 Concept + +Three compound axes, each a `[0,1]` fader that drives ~5–6 underlying parameters through a +piecewise-linear lookup table, with per-param trim-pot offsets on top. This is the "Feel" +surface from `playground-2.0-rewrite-plan.md` §2.3, ported with the playground's tables as the +**transparent defaults** — reproduced in full below so the mapping is reviewable here, not +buried in code. + +### 5.2 Axis tables (defaults — `playground/src/stores/control-store.ts:38–54`, verbatim) + +Numeric params interpolate linearly between breakpoints; discrete params snap to the upper row +at t ≥ 0.75. + +**Boldness** — "how hard do moves hit" + +| axis | zoom | noiseCap | noiseGrowth | learningRate | weightDecay | noiseDistribution | +|---|---|---|---|---|---|---| +| 0.0 | 0.1 | 0.02 | 1.1 | 0.1 | 0.15 | gaussian | +| 0.5 | 0.5 | 0.12 | 1.5 | 1.0 | 0.06 | gaussian | +| 1.0 | 1.0 | 0.30 | 2.5 | 3.0 | 0.00 | cauchy | + +**Memory** — "how much does the net remember" + +| axis | maxExamples | exampleDecay | memoryWeightDecay | noiseDecay | convergenceThreshold | +|---|---|---|---|---|---| +| 0.0 | 5 | 0.3 | 0.20 | 0.85 | 1e-3 | +| 0.5 | 50 | 0.7 | 0.06 | 0.97 | 1e-5 | +| 1.0 | 500 | 1.0 | 0.00 | 0.995 | 1e-8 | + +**Precision** — "how surgical is the gesture" + +| axis | inputCurve | deadzone | smoothing | slewRate | momentumZoom | +|---|---|---|---|---|---| +| 0.0 | 1.0 | 0.00 | 0.00 | 1.0 | off | +| 0.5 | 1.5 | 0.05 | 0.15 | 0.3 | off | +| 1.0 | 3.0 | 0.15 | 0.40 | 0.1 | off | + +Routing targets in manifold: `zoom/deadzone/inputCurve/smoothing/momentumZoom` → +`engine/input-pipeline` config; `slewRate` → `engine/output-pipeline`; `noiseCap/noiseGrowth/ +noiseDecay/learningRate/weightDecay` → `FeedbackController` + training params; +`maxExamples/exampleDecay/convergenceThreshold` → `engine/dataset.ts` + training loop. +`noiseDistribution` and `momentumZoom` are carried in the schema but may no-op until their +consumers exist in manifold (each no-op must be logged once at boot — no silent dead params). +`spread` is deliberately **not** axis-driven (stays an independent lab knob, as in playground). + +### 5.3 Presets (defaults — verbatim) + +| id | label | boldness | memory | precision | +|---|---|---|---|---| +| `default` | Default | 0.5 | 0.5 | 0.3 | +| `first-touch` | First Touch | 0.2 | 0.7 | 0.6 | +| `jazz-hands` | Jazz Hands | 0.8 | 0.2 | 0.0 | +| `sculptor` | Sculptor | 0.3 | 0.9 | 0.8 | +| `improviser` | Improviser | 0.6 | 0.3 | 0.2 | +| `microscope` | Microscope | 0.1 | 1.0 | 1.0 | + +Boot default = the `default` preset (0.5 / 0.5 / 0.3). **This replaces ConsoleApp's current +ad-hoc `{0.55, 0.4, 0.5}` initial state.** Moving any axis manually clears `presetId` (chips +show "custom"). + +### 5.4 State + trim-pot offsets + +```ts +interface ControlState { + boldness: number; memory: number; precision: number; // [0,1] + offsets: Record<'boldness'|'memory'|'precision', Record>; + presetId: string | null; +} +``` + +Resolution: interpolate all three tables, merge, then add each axis's offsets to numeric params +(`resolved[k] = table[k] + offset[k]`), clamp to each param's legal range. Offsets are created +when the user adjusts a *derived* param directly (§5.6) — the delta from the current table value +becomes the offset ("trim-pot"). Double-tap on a derived param's readout clears its offset +("re-link"); a per-axis "re-link all" clears the axis's offset map. A dot marks any param with a +non-zero offset. + +### 5.5 Fanout (architecture requirement) + +Per `engine-architecture.md` §3.2: resolution is a memoised derivation per target param — in +React terms one `useMemo` producing the resolved record + an effect per target store that +writes only on change. The playground's `JSON.stringify`-signature-inside-effect pattern +(`control-routing.ts:34`) is explicitly **not** ported. Resolution runs off the render cycle for +engine targets (direct setter calls on the engine config objects), with React state only for +what the UI displays. + +### 5.6 UI — transparency is the feature + +- **Placement:** the **learn drawer** gains a top "Feel" section (this keeps manifold at five + drawers; if the section crowds the condensed depth, promoting Feel to its own drawer is a + pre-approved fallback — note it in the implementation PR). + - *Condensed:* three vertical `ControlAxis` faders (the existing primitive, finally wired) + + six preset chips. + - *Expanded:* under each fader, the live list of its derived params — name, resolved value + (table + offset), and a mini trim-pot. This satisfies the "reasonable and transparent + defaults" requirement: the mapping is always visible in the UI, not just in this document. +- Axis moves are continuous (no apply button); param writes are rate-limited to one per frame. +- The existing `preset = 'Sculpt'` placeholder state in ConsoleApp is removed in favour of + `presetId`. + +### 5.7 Persistence, probe, acceptance + +- Persisted as the `control` slice of `mf-session` (also the `control` layer of session + presets, §1 — one shape, two containers). +- Probe: `setAxis(name, v)`, `getAxes()`, `resolveControlParams()`, `applyControlPreset(id)`, + `setAxisOffset(axis, param, v)`. +- e2e: boldness 0→1 sweeps noiseCap 0.02→0.30 and zoom 0.1→1.0 (assert via + `resolveControlParams()` + engine config readback); preset chip sets all three axes; manual + axis move clears presetId; offset survives axis movement (offset is additive, not absolute); + double-tap re-links; state round-trips through reload and through a session preset. + +--- + +## 6. Cross-cutting + +### 6.1 Build & sequencing (proposed ergo breakdown, post-sign-off) + +1. **W1 — WASM bindings**: jolt + OU C ABI, EXPORTED_FUNCS, parity golden tests. (Blocks W4, W5.) +2. **W2 — `persist` helper + `mf-session` document** (control slice, pins, presets scaffold). +3. **W3 — Control surface**: state, tables, fanout, Feel section, presets, offsets. (Needs W2.) +4. **W4 — Jolt + OU in FeedbackController + learn drawer UI.** (Needs W1.) +5. **W5 — Region pins** (tokens, gesture, rendering, popover) + tri-state training-mask + guarantee (§2.2). (Needs W2.) +6. **W6 — Session presets**: layers, preset bar, URL sharing, probe. (Needs W2, W3.) + +Each lands with its e2e spec; smoke suite additions run in the same CI gate as the existing +`tests/e2e/smoke.spec.ts`. + +### 6.2 Out of scope (explicit) + +- Snapshot-with-weights / snapshot DAG, A/B compare (separate spec — the biggest remaining + parity gap, but architecturally entangled with the C++ feedback snapshot lifecycle). +- 2D heatmap sampler, weight health, gradient flow (blocked on loss-history C API plumbing). +- Input zoom (inputs workstream) — noted where it interacts with region pins (§2.1). +- Per-column jolt masking (§3.4). + +### 6.3 Open questions for review + +1. §2.1 creation gesture: is the 0.25×0.25 press-centred region an acceptable stand-in until + input zoom lands, or should region pins wait for the zoom feature? +2. §5.6: Feel section inside the learn drawer vs a sixth drawer — reviewer's call if condensed + depth gets crowded. +3. §1.1 RoutingLayer: should backend connection details (OSC URL, MIDI port id) be included in + session presets, or excluded as machine-specific? Proposal: exclude port/URL, include + per-output mappings. +4. Memory-axis params (`maxExamples`, `exampleDecay`, `convergenceThreshold`): confirm the + manifold training loop actually consumes these; if any are dead in the current engine, they + land as logged no-ops (§5.2) with an ergo follow-up. + +--- + +*Doc-sync rule applies: when implementation diverges from this spec, update the spec in the same +commit. When a workstream ships and is verified, migrate its section to ONBOARDING.md ("what +is") and prune it here.* diff --git a/docs/redesign/BUILD-PLAN.md b/docs/specs/plans/BUILD-PLAN.md similarity index 97% rename from docs/redesign/BUILD-PLAN.md rename to docs/specs/plans/BUILD-PLAN.md index 2b5ae16..ea8a2f1 100644 --- a/docs/redesign/BUILD-PLAN.md +++ b/docs/specs/plans/BUILD-PLAN.md @@ -1,3 +1,8 @@ +--- +kind: plan +status: active +--- + # Manifold — Build Plan & Locked Decisions (resume anchor) *Dated 2026-06-27. This is the single resume anchor for the Manifold convertible-app build. Read this + the @@ -17,17 +22,17 @@ specs it points to before continuing. Mission: one working browser instrument pu | Synth UI label | **"Powerful Synth Engine"** — the string "C15" must NEVER appear in the UI | | Product copy | British spelling (randomise, visualise, colour) | -## Spec docs (all in docs/redesign/) -- `rl-feedback-design.md` — the learning engine (both modes + solo); **§1 = verified ground truth: the feedback +## Spec docs (now in docs/specs/ and subdirs) +- `docs/adr/rl-feedback-design.md` — the learning engine (both modes + solo); **§1 = verified ground truth: the feedback C API, `MLHandle.feedback`, `0xFEEDBACC0DE` salt, parity Stage 5, and `CMakeLists.txt:59` registration ALL ALREADY EXIST** — edit/extend, don't re-scaffold. Mode 1 ports to upstream **`0a541cc`** InterfaceRL. - `dock-spec.md` — six drawers + per-output control row + tri-state semantics + advanced modals. - `backends-spec.md` — `OutputBackend` adapter interface; **VCV module already at `vcv/` (2-in/12-out)** → evolve to 8→16 + LED rings; faithful `visualizer.js` particle port (algorithm documented). - `inputs-spec.md` — modular XY/WebMIDI/gamepad; multiple-WASM-module reshape with warm-start. -- `findings-{feedback-behaviour,engine-surface,design-and-manifold}.md` — Phase-1 audits. -- `upstream-firmware-survey.md` — git topology; latest InterfaceRL = `0a541cc`. -- `playground-2.0-rewrite-plan.md`, `engine-architecture.md`, `aimmersive-clone-spec.md`, +- `recon/findings-{feedback-behaviour,engine-surface,design-and-manifold}.md` — Phase-1 audits. +- `recon/upstream-firmware-survey.md` — git topology; latest InterfaceRL = `0a541cc`. +- `plans/playground-2.0-rewrite-plan.md`, `engine-architecture.md`, `aimmersive-clone-spec.md`, `feedback-modes-port-spec.md` — the prior planning corpus. ## Reactive spine → React (load-bearing, from findings-design-and-manifold.md §4) diff --git a/docs/redesign/playground-2.0-rewrite-plan.md b/docs/specs/plans/playground-2.0-rewrite-plan.md similarity index 99% rename from docs/redesign/playground-2.0-rewrite-plan.md rename to docs/specs/plans/playground-2.0-rewrite-plan.md index 3507e61..9ab65a5 100644 --- a/docs/redesign/playground-2.0-rewrite-plan.md +++ b/docs/specs/plans/playground-2.0-rewrite-plan.md @@ -1,6 +1,11 @@ +--- +kind: plan +status: executed +--- + # NISPS Playground 2.0 — Clean-Room SolidJS Rewrite Plan -*Status: sign-off-ready. Author: lead architect, synthesizing four designer proposals (P1–P4) against three judge panels (mission-fit, feasibility, UX coherence). Date: 2026-06-17.* +*Status: sign-off-ready. Author: lead architect, synthesizing four designer proposals (P1–P4) against three judge panels (mission-fit, feasibility, UX coherence). Date: 2026-06-17. Executed: largely implemented in the Manifold convertible app (June 2026) + playground SolidJS foundation.* --- diff --git a/docs/redesign/findings-design-and-manifold.md b/docs/specs/recon/findings-design-and-manifold.md similarity index 96% rename from docs/redesign/findings-design-and-manifold.md rename to docs/specs/recon/findings-design-and-manifold.md index 63c8bd6..a56b059 100644 --- a/docs/redesign/findings-design-and-manifold.md +++ b/docs/specs/recon/findings-design-and-manifold.md @@ -1,3 +1,9 @@ +--- +kind: finding +date: 2026-06-27 +immutable: true +--- + # Findings — Design + Manifold Console: build-oriented brief (Phase-1) *Read-only synthesis, 2026-06-27, of the five redesign docs + the on-disk Manifold token export + the @@ -8,15 +14,15 @@ > claude.ai design project and must be pulled via DesignSync before Phase-3 build. `ConsoleApp.jsx` was read > directly and is summarised in §2. -## Source map (`docs/redesign/`) -- `playground-2.0-rewrite-plan.md` (399ln) — authoritative SolidJS UX+architecture plan: Console IA, drawers, +## Source map (now in docs/specs/ and subdirs) +- `plans/playground-2.0-rewrite-plan.md` (399ln) — authoritative SolidJS UX+architecture plan: Console IA, drawers, reactive spine, feature table, roadmap, open questions. - `engine-architecture.md` (440ln) — one-engine-two-skins: `EngineApi` headless boundary, reuse table, COOP/COEP, build sequence S0–S7. - `aimmersive-clone-spec.md` (351ln) — faithful clone spec of deployed `a-immersive.html`: feature inventory, exact tactile constants, state→store map, engine call points. - `feedback-modes-port-spec.md` (548ln) — 3-mode "Down Action" ported to `nisps/` as `FeedbackController`. -- `playground-2026.md` (341ln) — older design-intent doc (dock+drawer, interactive heatmap); reference-only. +- `recon/playground-2026.md` (341ln) — older design-intent doc (dock+drawer, interactive heatmap); reference-only. > **Framework mismatch:** every planning doc targets **SolidJS**; the mission is a **React** port. The §4 spine > translation is the largest design decision and biggest risk (§7). diff --git a/docs/redesign/findings-engine-surface.md b/docs/specs/recon/findings-engine-surface.md similarity index 99% rename from docs/redesign/findings-engine-surface.md rename to docs/specs/recon/findings-engine-surface.md index 5283e69..c0fdbc4 100644 --- a/docs/redesign/findings-engine-surface.md +++ b/docs/specs/recon/findings-engine-surface.md @@ -1,3 +1,9 @@ +--- +kind: finding +date: 2026-06-27 +immutable: true +--- + # Findings — Engine Surface Audit (Phase-1) *Read-only audit, 2026-06-27. VERIFIED from source unless `[INFER]`. Engine A = deployed vanilla JS live at diff --git a/docs/redesign/findings-feedback-behaviour.md b/docs/specs/recon/findings-feedback-behaviour.md similarity index 97% rename from docs/redesign/findings-feedback-behaviour.md rename to docs/specs/recon/findings-feedback-behaviour.md index 5ed2eaa..dd9f6aa 100644 --- a/docs/redesign/findings-feedback-behaviour.md +++ b/docs/specs/recon/findings-feedback-behaviour.md @@ -1,3 +1,9 @@ +--- +kind: finding +date: 2026-06-27 +immutable: true +--- + # Findings — Current RL / Feedback Behaviour (Phase-1 Audit) *Read-only audit, 2026-06-27. Citations are `file:line`. "VERIFIED" = read in source; "INFER" = deduced. @@ -18,7 +24,7 @@ hook blocks subagent report-file writes. Adversarially spot-checked against sour random Gaussian noise"** — undirected diffusion, not directed avoidance. 4. The true geometric push-away (move output away from k-NN centroid of liked examples) exists only in the firmware submodule (`memllib` InterfaceRL; "geo push" commits) and is documented in - `docs/redesign/feedback-modes-port-spec.md` §1.1. **This is the algorithm Mode 1 must match** (workstream B). + `docs/specs/feedback-modes-port-spec.md` §1.1. **This is the algorithm Mode 1 must match** (workstream B). 5. **Why often inaudible (deployed AVOID):** noise added across all 4 layers of `MLP<2,10,14,18,126>`, per-layer scaled `(1-spread)+spread/√fan_in`, capped at `noiseCap=0.12`, spread default 0.6 → tiny per-step perturbation that the sigmoid output + output smoothing/slew pipeline absorb. One press moves the @@ -36,7 +42,7 @@ hook blocks subagent report-file writes. Adversarially spot-checked against sour `LikeStore` (anchor) + `trainOnCurrent` (train). Missing in deployed build: the randomise-audition-then-anchor loop; today thumbs-down only diffuses noise and never anchors. 10. The full port spec (component, WASM C API, TS FFI, ~22 ctest cases, parity stage) already exists at - `docs/redesign/feedback-modes-port-spec.md` and matches the untracked `feedback.hpp`. + `docs/specs/feedback-modes-port-spec.md` and matches the untracked `feedback.hpp`. 11. **Parity/perf constraints:** no heap / `std::array` only, per-instance deterministic `nisps::Rng` (no libc `rand()`), no virtual dispatch, `.f` literals, fixed WASM arch `MLP<2,10,14,18,126>`, native↔WASM parity within `1e-5`. @@ -153,6 +159,6 @@ prototype but commits at the *current* input position, not a chosen one, and lac - WASM API: `nisps/wasm/bindings.cpp:419-437` - TS: `playground/src/ml/wasm-iml.ts:488-508`; runtime `playground/src/modes/mode-runtime.ts:444-565` - Exploration knobs: `playground/src/stores/exploration-store.ts` -- Port spec: `docs/redesign/feedback-modes-port-spec.md` +- Port spec: `docs/specs/feedback-modes-port-spec.md` - Firmware geo-push: `src/memllib` (commits `2429bcc`, `d301cc7`; upstream `SB2026`; main `e291192`/`abe93ec`); design docs on `origin/feat/feedback-explore-modes`: `docs/dislike_system_{analysis,design_space}.md` diff --git a/docs/redesign/midi-gamepad-inputs-worklog.md b/docs/specs/recon/midi-gamepad-inputs-worklog.md similarity index 97% rename from docs/redesign/midi-gamepad-inputs-worklog.md rename to docs/specs/recon/midi-gamepad-inputs-worklog.md index 34e487f..ca03bd2 100644 --- a/docs/redesign/midi-gamepad-inputs-worklog.md +++ b/docs/specs/recon/midi-gamepad-inputs-worklog.md @@ -1,8 +1,14 @@ +--- +kind: finding +date: 2026-06-27 +immutable: true +--- + # Work log — MIDI + Game Controller inputs, N-D engine foundation *Scope: what was actually built on the `feat/midi-inputs` branch. This is a description of the work, not a spec. The design intent lives in -`docs/redesign/inputs-spec.md`; where this branch diverges from or only partially +`docs/specs/inputs-spec.md`; where this branch diverges from or only partially realises that spec, it is called out below.* ## Summary diff --git a/docs/redesign/playground-2026.md b/docs/specs/recon/playground-2026.md similarity index 98% rename from docs/redesign/playground-2026.md rename to docs/specs/recon/playground-2026.md index 0405e0a..69c7739 100644 --- a/docs/redesign/playground-2026.md +++ b/docs/specs/recon/playground-2026.md @@ -1,10 +1,16 @@ +--- +kind: finding +date: 2026-04-12 +immutable: true +--- + # Playground Redesign — 2026 (Design Intent) > **Purpose of this doc.** Captures design intent of an unfinished playground UI > redesign that lived as uncommitted changes in a `MEMLNaut-NISPS_rewound` > working copy. The implementation will not be merged into the current vanilla > playground — the playground is being rewritten in SolidJS — but the *design* -> shouldn't be lost. Read this as a feature spec for the SolidJS rewrite, not as +> shouldn't be lost. Read this as a reference for the SolidJS rewrite, not as > a description of current code. > > **Source snapshot.** `archive/playground-redesign-2026-snapshot` branch in @@ -12,7 +18,7 @@ > `playground/a-immersive.html`, `playground/css/a-immersive.css`, > `playground/js/a-app.js`. ~1200 lines net of changes. > -> **Status.** Reference-only. Do not merge into the vanilla playground — those +> **Status.** Reference-only, immutable. Do not merge into the vanilla playground — those > files have moved on independently in `main` and a textual merge would be > meaningless. diff --git a/docs/redesign/upstream-firmware-survey.md b/docs/specs/recon/upstream-firmware-survey.md similarity index 98% rename from docs/redesign/upstream-firmware-survey.md rename to docs/specs/recon/upstream-firmware-survey.md index 0bbca5c..bb7d272 100644 --- a/docs/redesign/upstream-firmware-survey.md +++ b/docs/specs/recon/upstream-firmware-survey.md @@ -1,3 +1,9 @@ +--- +kind: finding +date: 2026-06-27 +immutable: true +--- + # Upstream Firmware Survey — git archaeology of the MusicallyEmbodiedML ecosystem _Dated 2026-06-27. Author: research agent (read-only). All SHAs from a fresh `git fetch --all` + memllib submodule-gitdir fetch (incl. operator's memllib fork `monkey-w1n5t0n/memllib`)._ @@ -29,7 +35,7 @@ fast-forward or 3-way merge path; every integration is a re-implementation again | Branch | Remote | Last push | Author | vs its main¹ | Purpose | Judgment | |---|---|---|---|---|---|---| | `main` | origin | 2026-06-16 | monkey-w1n5t0n | — | The C++20/SolidJS rewrite. | canonical | -| `feat/feedback-explore-modes` | origin | 2026-06-08 | w1n5t0n | +42 / −193 | Pins memllib `abe93ec` = the `FEEDBACK_MODE` (avoid / randomise-outputs / randomise-MLP) commit. Source material for `docs/redesign/feedback-modes-port-spec.md`. | **real RL work to port** (already spec'd, not yet in main) | +| `feat/feedback-explore-modes` | origin | 2026-06-08 | w1n5t0n | +42 / −193 | Pins memllib `abe93ec` = the `FEEDBACK_MODE` (avoid / randomise-outputs / randomise-MLP) commit. Source material for `docs/specs/feedback-modes-port-spec.md`. | **real RL work to port** (already spec'd, not yet in main) | | `feat/useq-celium-opus46` | origin | 2026-06-14 | w1n5t0n | +2 / −36 | MEMLCelium useq experiment (Opus-4.6 session). | experiment — superseded by opus47 | | `useq-celium-opus47` | origin | 2026-04-20 | w1n5t0n | +6 / −36 | Later MEMLCelium useq iteration. | experiment | | `feat/unified-preset-system` | origin | 2026-04-16 | w1n5t0n | +46 / −55 | Preset-system spike; heavily diverged, stale. | abandoned/spike | @@ -157,7 +163,7 @@ not `git merge`. Ranked by mission impact (RL/feedback richness and firmware-fea `origin/feat/feedback-explore-modes`; cross-checked against memllib main `80420d1`/`d9bdcd6` (SB2026 dislikes in `InterfaceRL`) which upstream/main pins via `e291192`. - **Why first:** This is the negative-feedback / RL core of the whole project, and the work is *already - written up* as an implementation-ready spec at `docs/redesign/feedback-modes-port-spec.md` (new + written up* as an implementation-ready spec at `docs/specs/feedback-modes-port-spec.md` (new `nisps::ml::FeedbackController`, WASM C API, 11 ctest cases, parity Stage 5). The branch `feat/feedback-explore-modes` and the half-present `nisps/ml/feedback.hpp` + `tests/cpp/test_mlp_feedback.cpp` in the working tree show this is mid-flight. **Finish it and merge to main.** The spec's AVOID reconciliation @@ -213,7 +219,7 @@ The 49 upstream commits are firmware apps on the old tree. Port the ones with mi operator's own memllib fork — an off-graph gitlink that a clean checkout cannot resolve. 3. **The dislikes/RL feedback work the task flagged as "to find" is already (a) merged into upstream via SB2026 → memllib `e291192`, AND (b) re-implemented by the operator on `feat/feedback-explore-modes` (memllib - `abe93ec`), AND (c) fully spec'd for the new core** in `docs/redesign/feedback-modes-port-spec.md` — with + `abe93ec`), AND (c) fully spec'd for the new core** in `docs/specs/feedback-modes-port-spec.md` — with `nisps/ml/feedback.hpp` and `tests/cpp/test_mlp_feedback.cpp` already present (untracked) in the working tree. The feedback integration is not a research question; it's a half-finished implementation to land. @@ -250,4 +256,4 @@ The **latest upstream `InterfaceRL` tip is `0a541cc` "highlighting"** (990 lines which are older. **Mode 1 "Geometric dislike" must be ported to `0a541cc` parity.** Note: the in-tree submodule gitdir resolves none of `4733ca0`/`e291192`/`abe93ec`/`0a541cc` — only the separate `-upstream` memllib checkout has them; the orphaned-pin footgun stands and `0a541cc` is the re-pin/port target. See -`docs/redesign/rl-feedback-design.md` §1 for the verified-ground-truth list. +`docs/adr/rl-feedback-design.md` §1 for the verified-ground-truth list. diff --git a/docs/specs/slp-workshop-firmware.md b/docs/specs/slp-workshop-firmware.md index 91c47bb..a3a4213 100644 --- a/docs/specs/slp-workshop-firmware.md +++ b/docs/specs/slp-workshop-firmware.md @@ -1,4 +1,5 @@ --- +kind: spec stability: evolving layer: binding --- @@ -41,7 +42,7 @@ layer: binding - `playground/src/modes/SLPWorkshopMode.tsx`, `playground/src/modes/mode-runtime.ts` — browser controls. - `playground/src/ml/jolt.ts`, `playground/src/output/ou-explore.ts` — browser-side gesture math. - `tests/cpp/test_mlp_jolt.cpp`, `tests/cpp/test_mlp_ou_noise.cpp`, `tests/cpp/test_mode_learning.cpp`, `tests/cpp/test_mode_concepts.cpp` — anchoring tests. -- **Part II companions (planned work):** `docs/useq-celium/protocol.md` (CV wire protocol), `manifold/src/backends/` (CV/MIDI output backends), `manifold/src/inputs/` (input layer). +- **Part II companions (planned work):** `docs/specs/useq-cv-protocol.md` (CV wire protocol), `manifold/src/backends/` (CV/MIDI output backends), `manifold/src/inputs/` (input layer). --- @@ -226,7 +227,7 @@ A **mode** is just *which streams are active* and *whether they share a network* 12.3 **CV mode** hardware is fixed (11 PWM/CV-capable jacks + 3 digital gate-only jacks), so the two counts are **linked** and gates are optional. The rule **MUST** be `gates ∈ [0, 14]`; `CV = gates ≤ 3 ? 11 : 14 − gates`. **Why:** the first 3 gate sequences land on the dedicated gate-only jacks and cost no CV (0→11 CV, 3→11 CV, 6→8 CV, 14→0 CV); converting CV jacks only begins past 3. Pure CV (0 gates) is valid. -12.4 Wire-protocol impact **MUST** be none: a CV jack acting as a gate carries 0/full (or the 2-level velocity) in its `u16` slot; the 3 dedicated gate bits stay digital pins (`docs/useq-celium/protocol.md`). `CvSpec` extends so PWM jacks can also be gate targets. +12.4 Wire-protocol impact **MUST** be none: a CV jack acting as a gate carries 0/full (or the 2-level velocity) in its `u16` slot; the 3 dedicated gate bits stay digital pins (`docs/specs/useq-cv-protocol.md`). `CvSpec` extends so PWM jacks can also be gate targets. **Why:** reusing the existing protocol slots avoids a protocol revision for a routing feature. 12.5 The **Rhythm network** toggle (shown only when gate sequences > 0) selects **Separate** (default) or **Shared**. Separate gives the Rhythm stream its own MLP and routes each input channel to one engine, automatically by source kind: XY pad → a second on-screen pad (pad 1 → Continuous, pad 2 → Rhythm); gamepad → double-stick (left → Continuous, right → Rhythm); MIDI controller → per-CC `Continuous | Rhythm` toggle. Shared uses one MLP for both streams (all inputs feed it; the hardware-parity hybrid). diff --git a/docs/useq-celium/protocol.md b/docs/specs/useq-cv-protocol.md similarity index 97% rename from docs/useq-celium/protocol.md rename to docs/specs/useq-cv-protocol.md index e49bd36..6ea38fe 100644 --- a/docs/useq-celium/protocol.md +++ b/docs/specs/useq-cv-protocol.md @@ -1,4 +1,10 @@ -# uSEQ-CV wire protocol v2 +--- +kind: spec +stability: stable +layer: binding +--- + +# uSEQ-CV Wire Protocol v2 The protocol the Manifold **CV output backend** (`manifold/src/backends/cv-backend.ts`) speaks over USB Web Serial to the uSEQ main module, which drives CV/gate jacks and diff --git a/vcv/SPEC.md b/docs/specs/vcv-module.md similarity index 71% rename from vcv/SPEC.md rename to docs/specs/vcv-module.md index dac069d..4af5774 100644 --- a/vcv/SPEC.md +++ b/docs/specs/vcv-module.md @@ -1,56 +1,21 @@ -# MEMLNaut VCV Rack Module — Specification - +--- +kind: spec +stability: evolving +layer: binding +counterpart: backends-spec.md --- -## ⚠️ BUILD DELTAS (2026-06-28) — AUTHORITATIVE OVERRIDES +# MEMLNaut VCV Rack Module — Specification -These supersede any conflicting detail below. They reflect the Manifold mission + the locked decisions in -`docs/redesign/BUILD-PLAN.md` and `docs/redesign/backends-spec.md`. Build to THESE. - -1. **I/O = 8 inputs × 16 outputs** (was 2→12). `NUM_ML_INPUTS = 8`, `NUM_ML_OUTPUTS = 16`. The IML is sized - 8→16 (a runtime-shaped native MLP is fine here — the module is C++, not the fixed WASM target). The 8 CV - inputs feed the model's input dims; the 16 CV outputs are the model's inference outputs (the modular N×M - envelope). Keep the control inputs (Spread CV, Learn gate, + / − triggers). -2. **LED RING around EACH of the 16 outputs** — a custom ring widget encircling each output jack whose arc - fills in proportion to that output's value (0..1 → 0..2π). Draw on `drawLayer()` layer 1 with `nvgArc` for - the proportional fill + a dim track ring. Each ring is COLOURED from a palette that MATCHES the frontend - design tokens. -3. **Palette from the frontend tokens** — generate `vcv/src/palette.hpp` from - `docs/redesign/manifold-export/tokens/colors.css`: `--accent #ff6a00` (orange), `--accent-2 #00ccff` (cyan), - and the group colours (formant→accent, pitch→accent-2, amp→`--good #6bc26b`, filter→`--warn #f5c45e`, - fx→`--info #5b9eef`, mod→`--accent-3 #ffa860`). Assign the 16 rings across these group colours (or a clean - 16-step ramp between orange and cyan) so the module reads as the same instrument as the browser. A tiny - hand-written `palette.hpp` is acceptable (no build-time codegen needed). -4. **Browser ↔ VCV bridge = WS↔OSC** (locked transport). The module runs its OSC server (`src/osc_server.hpp` - already exists — evolve it). The browser's OSC backend (`manifold/src/backends/osc-backend.ts`) sends over a - WebSocket to the Deno bridge (`manifold/osc-bridge/`), which relays UDP-OSC to the module. **Bidirectional - training**: drive + train the module FROM the browser (the verdict loop + example-placing over the bridge) - AND from the module's own panel (+/− buttons, Learn gate, triggers). OSC verbs to support both directions: - `/nisps/input` (drive), `/nisps/output` (module→browser viz), `/nisps/feedback` (thumbs up/down + place), - `/nisps/weights`, `/nisps/examples`, `/nisps/state`. Pick a fixed default UDP port (e.g. 7001) + per-instance - offset; the Deno bridge maps `ws://localhost:8765` ↔ that UDP port. -5. **Core include path** — the Makefile's `-I../nisps-core/include` points at the RETIRED `nisps-core`. Repoint - to the current core (`../nisps/`) OR vendor a minimal runtime IML inside `vcv/src/`. Goal: get it COMPILING - with an 8→16 runtime MLP that shares the firmware/browser training semantics (spread-aware draw/move_weights, - deterministic RNG) as closely as the native runtime-shaped form allows. If full nisps/ml reuse is blocked by - the templated fixed-size API, keep a self-contained IML in the module and note the alignment as a follow-up. -6. **Derived outputs** (Mean/Std/Delta/Novelty/Confidence) → move to the optional EXPANDER or a context-menu - toggle; the headline is 16 raw outputs + their LED rings. Do not let them crowd the 16-jack panel. -7. **Build** needs the VCV Rack 2 SDK (`RACK_DIR`, default `$HOME/.local/share/Rack2/Rack-SDK`, NOT installed). - The build step must fetch the Linux Rack-SDK zip from vcvrack.com, set `RACK_DIR`, and `make` to verify the - plugin compiles. Panel SVGs in `vcv/res/` exist (MEMLNaut.svg / -wide / -expander) — widen/relayout for 16 - outputs + rings as needed. - -The rest of this document is the prior (2→12) design — useful for threading, persistence, RL workflow, and -panel/build mechanics, but the I/O counts, LED rings, palette, bridge, and core path above WIN. +**Revision note:** 2026-06-28 BUILD DELTAS folded into body (2026-07-13). The module now implements 8→16 I/O with per-output LED rings, design-token palette, and WS↔OSC bridge per the Manifold mission and BUILD-PLAN locked decisions. Prior design (2→12) sections retained for threading/persistence/RL workflow reference. --- ## Overview -A VCV Rack module that embeds the NISPS interactive ML engine (nisps-core C++ library) as a CV-to-CV mapper. Users explore high-dimensional parameter spaces via reinforcement learning feedback, producing 12 raw CV outputs and 5 derived meta-signals from configurable CV inputs. +A VCV Rack module that embeds the NISPS interactive ML engine as a CV-to-CV mapper. Users explore high-dimensional parameter spaces via reinforcement learning feedback, producing 16 raw CV outputs (each with a custom LED ring indicator) from 8 CV inputs. -The module does **not** produce sound. It maps input CVs through a trained neural network to output CVs, which the user patches into other modules (VCOs, VCFs, VCAs, etc.). The result: a learned, nonlinear, high-dimensional modulation source shaped by the user's aesthetic preferences. +The module does **not** produce sound. It maps input CVs through a trained neural network to output CVs, which the user patches into other modules (VCOs, VCFs, VCAs, etc.). The result: a learned, nonlinear, high-dimensional modulation source shaped by the user's aesthetic preferences. The 8→16 architecture enables the module to serve as a general-purpose modulation transformer for complex polyphonic and sequencer-driven patches. **Plugin name:** MEMLNaut **Module name:** MEMLNaut (initially single module, future modules possible) @@ -109,13 +74,13 @@ The module does **not** produce sound. It maps input CVs through a trained neura ## I/O Specification -### Inputs (Configurable: 2–8, default 2) +### Inputs (Fixed: 8 CV inputs + control ports) + +The module has **8 fixed CV inputs** feeding the MLP's input layer (no runtime reconfiguration of input count). | Port | Default Label | Notes | |------|---------------|-------| -| IN 1 | X | Primary input CV | -| IN 2 | Y | Primary input CV | -| IN 3–8 | IN 3–8 | Hidden by default, shown when enabled | +| IN 1–8 | IN 1–8 | CV inputs feeding the 8-input MLP. Each input jack is visible on the panel. | | SPREAD CV | Spread | CV modulation of SPREAD knob (attenuated, added to knob value) | | LEARN | Learn | Gate input: when high, RL feedback is accepted | | + TRIG | Positive | Trigger input: register thumbs-up | @@ -127,16 +92,12 @@ The module does **not** produce sound. It maps input CVs through a trained neura - LEARN gate has a corresponding panel toggle button (either/or — gate OR button enables learning) - +/− triggers work only when LEARN is enabled (gate high OR toggle on) -### Outputs (17 total: 12 raw + 5 derived) +### Outputs (16 raw + optional derived) | Port | Type | Description | |------|------|-------------| -| OUT 1–12 | Raw MLP | Direct MLP output activations, scaled to configured CV range | -| MEAN | Derived | Mean of the 12 raw outputs | -| STD | Derived | Standard deviation of the 12 raw outputs (named "STD" to avoid collision with the SPREAD knob) | -| DELTA | Derived | Rate of change (L2 norm of output difference from previous inference) | -| NOVELTY | Derived | Gate: fires when current input is far from all training examples (computed on training thread, cached). **Default with 0 examples: 10V** (everything is novel when untrained). | -| CONFIDENCE | Derived | Inverse of loss on nearest training example (computed on training thread, cached). **Default with 0 examples: 0V** (no confidence with no data). | +| OUT 1–16 | Raw MLP | Direct MLP output activations, scaled to configured CV range. Each jack is surrounded by a custom LED ring (see Visual Feedback). | +| MEAN, STD, DELTA, NOVELTY, CONFIDENCE | Derived | Available on the expander module or via context-menu toggle (hidden by default to keep the main panel clean). See prior design section for semantics. | Each output has: - Per-output range configuration (0–10V unipolar or ±5V bipolar) via context menu @@ -171,27 +132,29 @@ Each output has: ## Visual Feedback -### Primary Display (Custom OpenGL Widget) +### Per-Output LED Rings -The module includes a real-time rendered display area showing: +Each of the 16 output jacks is surrounded by a **custom LED ring** (SVG + NanoVG on `drawLayer()` layer 1). The ring arc fills proportionally to the output's current value (0 → 0V, 1 → full arc = 2π). The track ring is dimly visible always; the fill is bright and **color-coded by parameter group** to match the frontend design-token palette: -**Option A — Full Custom Display:** -- 12 vertical bars showing raw output levels (color-coded) -- Neuron activation heatmap (simplified MLP visualization) +- **Formant group** → `--accent` (#ff6a00, orange) +- **Pitch group** → `--accent-2` (#00ccff, cyan) +- **Amplitude group** → `--good` (#6bc26b, green) +- **Filter group** → `--warn` (#f5c45e, yellow) +- **Effects group** → `--info` (#5b9eef, blue) +- **Modulation group** → `--accent-3` (#ffa860, light orange) + +If the 16 outputs don't have a pre-assigned group mapping, use a smooth 16-step ramp interpolating between orange and cyan across the jacks left-to-right. The palette is defined in a minimal `vcv/src/palette.hpp` (hand-written, no build-time codegen required). + +### Control Panel Display + +A small real-time display area (if space permits on the panel) showing: - Training state indicator (idle / training / converged) - Example count - Current noise level +- Current spread value -**Option B — Bars + Input Position:** -- 12 vertical bars showing raw output levels -- Small 2D dot plot showing current input position (XY scope style) -- Training state, example count, noise level as text overlays +### Additional LEDs -Both options to be prototyped; converge based on usability and CPU cost. - -### LED Indicators - -- Per-output LEDs showing signal level (brightness = voltage) - LEARN LED (green when active) - Training activity LED (flashes during training) @@ -199,21 +162,21 @@ Both options to be prototyped; converge based on usability and CPU cost. ## MLP Configuration -### Default Network +### Fixed Network Architecture ``` -Inputs: 2 (+ bias = 3 input nodes) -Hidden: [16, 24, 16] (3 hidden layers, ReLU activation) -Output: 12 (sigmoid activation, maps to [0, 1]) +Inputs: 8 (+ bias = 9 input nodes) +Hidden: [24, 32, 16] (3 hidden layers, ReLU activation) +Output: 16 (sigmoid activation, maps to [0, 1]) ``` -Significantly smaller than the webapp's [3, 32, 48, 64, 126] — appropriate for 12 outputs and real-time inference constraints. +The architecture is **fixed at compile time**; no runtime reconfiguration of input/output layer sizes. The hidden layer widths are appropriate for 8 inputs → 16 outputs and real-time inference constraints (well under 1% CPU on any modern machine at typical inference rates). -### When Input Count Changes +### Core Library Integration -- User selects new input count from context menu -- Confirmation dialog: "This will reset the network and clear all training data. Continue?" -- On confirm: rebuild MLP with new input layer size, clear dataset, randomize weights +The MLP uses a **runtime-shaped IML** (not the fixed-size WASM template). Point the build at the current `../nisps/` (not the retired `nisps-core`) and either: +1. Reuse `nisps/ml/mlp.hpp` and compile with `MLP<8, 24, 32, 16, 16>` type, or +2. Vendor a minimal self-contained 8→16 IML in `vcv/src/`, ensuring it shares the firmware/browser training semantics (spread-aware `DrawWeights`/`MoveWeights`, deterministic RNG). If templated-API constraints block option 1, option 2 is acceptable with a note of alignment as a follow-up. ### Spread Parameter @@ -305,69 +268,54 @@ The `version` field enables forward compatibility. On load, validate that `mlpCo --- -## Companion Webapp Integration +## Companion Webapp Integration (Manifold) -### Bidirectional State Transfer +### Locked Transport: WebSocket ↔ UDP OSC Bridge -**File-based (offline):** -- Webapp: "Export .nisps" button → downloads JSON file -- VCV: Right-click → "Load .nisps preset" → imports weights + examples + config -- VCV: Right-click → "Save .nisps preset" → exports for webapp import -- Webapp: "Import .nisps" → loads and continues training +The module runs its own **UDP OSC server** (`src/osc_server.hpp` — evolve the existing skeleton). The browser app (`manifold/src/backends/osc-backend.ts`) connects via **WebSocket to a Deno bridge** (`manifold/osc-bridge/`), which relays UDP OSC to the module. This enables **bidirectional training**: both the browser verdict loop (thumbs-up/down + example-placing) and the module's panel (+/− buttons, Learn gate, triggers) can drive the same MLP. -**OSC-based (live):** -- VCV module runs an OSC server (configurable port, default 9000) -- Webapp connects via WebSocket → OSC bridge -- Messages: - - `/nisps/weights` — full weight transfer (either direction) - - `/nisps/examples` — example set transfer - - `/nisps/state` — full state sync - - `/nisps/input` — current input values (for webapp visualization) - - `/nisps/output` — current output values (for webapp visualization) +### OSC Verbs (Module ← → Browser) -### Webapp Modifications Required +| Verb | Source | Direction | Payload | +|------|--------|-----------|---------| +| `/nisps/input` | Browser | → VCV | [8 floats] — CV input values | +| `/nisps/output` | VCV | → Browser | [16 floats] — raw MLP outputs (for visualization) | +| `/nisps/feedback` | Browser | → VCV | (thumbsUp|thumbsDown, optional posLabel [float, float] for placement) | +| `/nisps/feedback` | VCV | → Browser | Same (module panel triggers) | +| `/nisps/weights` | Either | ↔ | Full weight transfer | +| `/nisps/examples` | Either | ↔ | Example dataset transfer | +| `/nisps/state` | Either | ↔ | Full module state (weights, examples, spread, noise level, etc.) | -- Add .nisps file import/export buttons -- Add OSC client mode (connect to VCV module) -- Network size configuration to match VCV (12 outputs vs 126) -- Shared .nisps file format specification +**Port assignment:** Fixed default UDP port 7001, with per-instance offset if multiple modules exist in the same patch (e.g. module 2 → 7002). The Deno bridge maps `ws://localhost:8765` ↔ that UDP port. + +### File-Based Preset Export/Import + +File-based transfer remains supported via `.nisps` JSON format (same structure as patch state, see State Persistence section): +- VCV: Right-click → "Export .nisps preset" → JSON file +- VCV: Right-click → "Import .nisps preset" → restores weights + examples + config +- Webapp can read/write the same `.nisps` format for offline interchange with other modules or archival --- -## Panel Layout (Prototyping Phase) +## Panel Layout -Three panel variants to prototype. The compact (20HP) option was cut — 22 jacks + 5 buttons + 2 knobs + display cannot physically fit in 128.5mm of vertical panel space. +The module spans **44HP** (wide variant) to accommodate all 8 input jacks, 16 output jacks with LED rings, control knobs, and buttons. An optional **expander module (16HP)** provides additional attenuverter fine-tuning and advanced display. -### Standard (30HP) — minimum viable panel -``` -┌──────────────────────────────────┐ -│ MEMLNaut │ -│ ┌──────────────────────────┐ │ -│ │ DISPLAY │ │ -│ │ (bars + XY + metrics) │ │ -│ └──────────────────────────┘ │ -│ │ -│ SPREAD RATE │ -│ [knob] [knob] │ -│ │ -│ [+] [−] [LEARN] [RAND] │ -│ │ -│ IN: (1) (2) LEARN (+) (−) │ -│ │ -│ OUT: │ -│ 1[a](o) 2[a](o) 3[a](o) 4[a](o)│ -│ 5[a](o) 6[a](o) 7[a](o) 8[a](o)│ -│ 9[a](o) 10[a](o) 11[a](o) 12[a]│ -│ MN(o) SP(o) DL(o) NV(o) CF(o) │ -└──────────────────────────────────┘ -``` -[a] = small attenuverter knob per output. +### Main Module (44HP) Layout -### Wide (44HP) -Full display, all attenuverters, room for 8 input jacks, OpenGL network visualization. +Typical layout (exact spacing subject to panel artwork): +- Top: small DISPLAY showing training state, example count, spread, noise level +- Upper: SPREAD knob, RATE knob, LEARN button + LED, RAND button, CLEAR button +- Middle: 8 input jacks (IN 1–8) in a column, LEARN gate, +/− trigger inputs +- Lower: 16 output jacks arranged in 2 rows of 8, each jack surrounded by an LED ring +- Under each output jack: small attenuverter trim pot for per-output scaling -### Expander Module (16HP) -Adds: 6 extra input jacks, per-output attenuverters, secondary display. +### Expander Module (16HP) — Optional + +Adds: +- Extended attenuverter controls (alternate layout for finer per-output adjustment) +- Secondary display (novelty grid, confidence map, or advanced metrics) +- Space for future I/O expansion --- @@ -378,34 +326,46 @@ Adds: 6 extra input jacks, per-output attenuverters, secondary display. ``` vcv/ ├── plugin.json # Plugin manifest -├── Makefile # VCV SDK Makefile +├── Makefile # VCV SDK Makefile + RACK_DIR auto-download ├── src/ │ ├── plugin.hpp # Plugin globals │ ├── plugin.cpp # Plugin init -│ ├── MEMLNaut.cpp # Module logic (process, state, threading) -│ └── MEMLNautWidget.cpp # Panel UI (widgets, display, layout) +│ ├── MEMLNaut.cpp # Module logic (process, state, threading, OSC) +│ ├── MEMLNautWidget.cpp # Panel UI + LED ring drawing +│ ├── palette.hpp # Hand-written color palette (from design tokens) +│ └── osc_server.hpp # OSC server impl (evolve existing skeleton) ├── res/ -│ ├── MEMLNaut.svg # Panel artwork +│ ├── MEMLNaut.svg # Panel artwork (44HP, 16 outputs + LED rings) +│ ├── MEMLNaut-wide.svg # Wide variant (if needed) +│ ├── MEMLNaut-expander.svg # Expander panel (future) │ └── components/ # Custom SVG components └── dep/ - └── nisps-core/ # Symlink or copy of nisps-core headers + └── nisps/ # Symlink to ../nisps (C++20 core library) ``` ### Dependencies -- **VCV Rack SDK** (v2.x) -- **nisps-core** (header-only, C++20, already in this repo) -- **OSC library** (Phase 8 only): oscpack or liblo for UDP OSC server. Not needed until Phase 8. Alternative: minimal from-scratch UDP implementation to avoid the dependency. +- **VCV Rack SDK 2.x** — must be installed or auto-fetched. The build step should check for `RACK_DIR` env var, and if not set, fetch the Linux SDK zip from vcvrack.com and set it automatically. +- **nisps/** (C++20 core library) — use the current repo's `nisps/ml/` and `nisps/core/` directly (not the retired `nisps-core`) +- **OSC library** — `oscpack` or `liblo` for UDP OSC server, or a minimal from-scratch UDP impl to avoid the dependency ### Build Commands ```bash cd vcv -export RACK_DIR=/path/to/Rack-SDK +# RACK_DIR will be auto-fetched/set if not already present make make install # Copies to VCV plugin directory ``` +### Panel SVG Updates + +Widen and relayout panel SVGs (`MEMLNaut.svg` / `-wide.svg` / `-expander.svg`) for: +- 8 input jacks (vertical column on left) +- 16 output jacks with LED ring enclosures (2 rows of 8) +- Attenuverter trim pots under each output +- Knobs, buttons, display, and gate/trigger inputs in the remaining space + --- ## Development Phases diff --git a/firmware/useq-celium/README.md b/firmware/useq-celium/README.md index a6b526f..d373088 100644 --- a/firmware/useq-celium/README.md +++ b/firmware/useq-celium/README.md @@ -4,7 +4,7 @@ Turns a [uSEQ](https://www.emutelabinstruments.co.uk/useq/) module (+ its CV expander) into a USB→CV/gate converter driven by the Manifold browser app's **CV output backend** (or, in future, the MEMLNaut RP2350 firmware directly). This is the restored + modernised descendant of the April-2026 "uSEQ-Celium" output mode -(provenance in `docs/useq-celium/protocol.md`). +(provenance in `docs/specs/useq-cv-protocol.md`). Both boards are RP2040 (Raspberry Pi Pico / uSEQ hardware) flashed with the Arduino-Pico (Earle Philhower) core. The wire protocol (v2) is defined once in @@ -43,4 +43,4 @@ add `firmware/useq-celium/shared` to the include path.) Hold the browser's **Identify** button (CV backend config) → both boards run an LED sweep, confirming the USB + I2C links end-to-end. Full frame layout: -`docs/useq-celium/protocol.md`. +`docs/specs/useq-cv-protocol.md`. diff --git a/manifold/ONBOARDING.md b/manifold/ONBOARDING.md index c698181..7958ef3 100644 --- a/manifold/ONBOARDING.md +++ b/manifold/ONBOARDING.md @@ -248,12 +248,12 @@ a setting → `--r-*` tokens. ## 7. Deeper references (only when this doc isn't enough) -All in `docs/redesign/` (at the repo root, not under `manifold/`): -- `BUILD-PLAN.md` — locked decisions + the 12-step build sequence + spec pointers (the resume anchor). +All in `docs/specs/` (at the repo root, not under `manifold/`), with subdirectories: +- `plans/BUILD-PLAN.md` — locked decisions + the 12-step build sequence + spec pointers (the resume anchor). - `engine-architecture.md` — full engine/spine/WASM design. - `dock-spec.md` — dock + drawers spec. `inputs-spec.md` — mixed-input design. `backends-spec.md` — backends. -- `rl-feedback-design.md` + `feedback-modes-port-spec.md` + `findings-feedback-behaviour.md` — feedback modes (Mode 1/Mode 2). -- `aimmersive-clone-spec.md` / `playground-2026.md` — the a-immersive feature parity target. +- `docs/adr/rl-feedback-design.md` + `feedback-modes-port-spec.md` + `recon/findings-feedback-behaviour.md` — feedback modes (Mode 1/Mode 2). +- `aimmersive-clone-spec.md` / `recon/playground-2026.md` — the a-immersive feature parity target. - `src/backends/README.md` — backend wiring notes. **Memories** (auto-loaded): `manifold-build` (status + locked decisions), `manifold-mixed-inputs` diff --git a/manifold/src/backends/README.md b/manifold/src/backends/README.md index 961c9b2..d46dd0f 100644 --- a/manifold/src/backends/README.md +++ b/manifold/src/backends/README.md @@ -5,7 +5,7 @@ time, chosen by the dock **Mode** (Particle / MIDI / OSC / VCV / Built-in Synth Editor → `BackendId`). The `BackendManager` consumes the engine spine and forwards each routed output vector to the active backend's `send()`. -See `docs/redesign/backends-spec.md` for the authoritative design. +See `docs/specs/backends-spec.md` for the authoritative design. ## Files diff --git a/manifold/src/backends/cv-backend.ts b/manifold/src/backends/cv-backend.ts index e9a37a0..43fa39c 100644 --- a/manifold/src/backends/cv-backend.ts +++ b/manifold/src/backends/cv-backend.ts @@ -1,7 +1,7 @@ /** * UseqCvBackend — real CV/gate output over USB Web Serial to a uSEQ module * (+ CV expander). The restored, modernised descendant of the April-2026 - * "uSEQ-Celium" output mode (provenance: docs/useq-celium/protocol.md). + * "uSEQ-Celium" output mode (provenance: docs/specs/useq-cv-protocol.md). * * Per non-silent output: map (0..1) → baseline (min/max/curve) → assign to the * output's configured uSEQ channel — a CV jack (12-bit value) or a gate diff --git a/manifold/src/backends/useq-protocol.ts b/manifold/src/backends/useq-protocol.ts index 306e497..17b451c 100644 --- a/manifold/src/backends/useq-protocol.ts +++ b/manifold/src/backends/useq-protocol.ts @@ -4,7 +4,7 @@ * header; useq-protocol.test.ts asserts the frame sizes match. * * Pure + framework-neutral (no navigator / DOM) so it is unit-testable and safe - * to import from anywhere. See docs/useq-celium/protocol.md. + * to import from anywhere. See docs/specs/useq-cv-protocol.md. */ // ─── Sync bytes ────────────────────────────────────────────────────────────── diff --git a/manifold/src/backends/vcv-backend.ts b/manifold/src/backends/vcv-backend.ts index 32030c8..2a7218a 100644 --- a/manifold/src/backends/vcv-backend.ts +++ b/manifold/src/backends/vcv-backend.ts @@ -1,6 +1,6 @@ /** * VcvBackend — drives + trains the VCV Rack NISPS module over the OSC↔WS bridge - * (backends-spec §2.6; vcv/SPEC.md OSC verbs). In "bridged" mode the BROWSER is + * (backends-spec §2.6; docs/specs/vcv-module.md OSC verbs). In "bridged" mode the BROWSER is * authoritative: it streams the current input vector to the module and forwards * the verdict loop (thumbs up/down, explore-and-place) so the module's embedded * net trains in lock-step with the browser session. diff --git a/manifold/src/console/ConsoleApp.tsx b/manifold/src/console/ConsoleApp.tsx index aca34b7..5c23bd7 100644 --- a/manifold/src/console/ConsoleApp.tsx +++ b/manifold/src/console/ConsoleApp.tsx @@ -13,7 +13,7 @@ * - The default feedback mode is "Explore and place" → the shared C++ core's * FeedbackMode::ExploreAndPlace (set on mount; the controller forwards the * Idle→Exploring→Placing lifecycle to engine.feedback.* — nisps/ml/feedback.hpp, - * per docs/redesign/rl-feedback-design.md). + * per docs/adr/rl-feedback-design.md). * - AltitudeNav switches `focus` via React state (in|split|out|composite), not * by navigating to separate HTML files. * - `c15` is labelled "Powerful Synth Engine" (in model.ts) — "C15" never shows. diff --git a/manifold/src/console/Drawers.tsx b/manifold/src/console/Drawers.tsx index d0993ae..e50bdb1 100644 --- a/manifold/src/console/Drawers.tsx +++ b/manifold/src/console/Drawers.tsx @@ -468,7 +468,7 @@ function InputsDrawer(ctx: ConsoleCtx, depth: DrawerDepth) { {/* ---- Reshape note (only when >2 axes feed the fixed WASM head) ---- */} {reshaping && depth === 'expanded' && (

- {/* TODO(workstream F, docs/redesign/inputs-spec.md — "multiple WASM modules + + {/* TODO(workstream F, docs/specs/inputs-spec.md — "multiple WASM modules + warm-start"): give every axis its own genuine input dimension by (re)loading a WASM module whose MLP arity matches axisCount and warm-starting from the prior net. Deferred — the reduction lives in InputLayer.compose(). */} diff --git a/manifold/src/dock/BackendAdvanced.tsx b/manifold/src/dock/BackendAdvanced.tsx index 269df47..575686c 100644 --- a/manifold/src/dock/BackendAdvanced.tsx +++ b/manifold/src/dock/BackendAdvanced.tsx @@ -270,7 +270,7 @@ function VcvChannelEditor({ ); } -// ---- uSEQ CV / gate (docs/useq-celium) ------------------------------------- +// ---- uSEQ CV / gate (docs/specs/useq-cv-protocol.md) ------------------------------------- function CvChannelEditor({ params, @@ -282,7 +282,7 @@ function CvChannelEditor({ return (

- Live CV/gate over USB serial to a uSEQ module + expander (docs/useq-celium). Assign each model output to a + Live CV/gate over USB serial to a uSEQ module + expander (docs/specs/useq-cv-protocol.md). Assign each model output to a CV jack or gate; gates threshold the mapped 0–1 value. Connect the device in the Outputs panel.

diff --git a/manifold/src/dock/OutputsBackendConfig.tsx b/manifold/src/dock/OutputsBackendConfig.tsx index 63cde6a..62e503b 100644 --- a/manifold/src/dock/OutputsBackendConfig.tsx +++ b/manifold/src/dock/OutputsBackendConfig.tsx @@ -583,7 +583,7 @@ function VcvConfig({ ctx }: { ctx: ConsoleCtx }) { ); } -// ---- uSEQ CV / gate config (backends-spec §2.5; docs/useq-celium) ----------- +// ---- uSEQ CV / gate config (backends-spec §2.5; docs/specs/useq-cv-protocol.md) ----------- function CvConfig({ ctx }: { ctx: ConsoleCtx }) { const s = ctx.backendStatus; diff --git a/manifold/src/dock/output-state.ts b/manifold/src/dock/output-state.ts index 74d66ed..7800519 100644 --- a/manifold/src/dock/output-state.ts +++ b/manifold/src/dock/output-state.ts @@ -1,6 +1,6 @@ /** * The per-output control model for the Outputs / Routing dock (workstream D, - * docs/redesign/dock-spec.md §3.2). + * docs/specs/dock-spec.md §3.2). * * DELIBERATE DIVERGENCE from the deployed a-immersive app (dock-spec §3.3 note, * open choice 3): the deployed override system conflates "frozen" (heatmap @@ -72,7 +72,7 @@ export interface VcvSpec { * uSEQ CV/gate backend per-output extras. Each model output is assigned to one * physical uSEQ channel (or 'none'); gate channels threshold the mapped 0..1 * value. The fixed hardware topology is 11 CV + 3 gate — see - * docs/useq-celium/protocol.md. + * docs/specs/useq-cv-protocol.md. */ export type CvChannelId = | 'none' diff --git a/manifold/src/feedback/controller.ts b/manifold/src/feedback/controller.ts index 6dd6a33..a815c81 100644 --- a/manifold/src/feedback/controller.ts +++ b/manifold/src/feedback/controller.ts @@ -3,9 +3,9 @@ * feedback modes plus solo/arm, prototyped in pure TS on the EXISTING engine * primitives (NO C++/WASM change). * - * Authoritative design: docs/redesign/rl-feedback-design.md (Mode 2 default; + * Authoritative design: docs/adr/rl-feedback-design.md (Mode 2 default; * Mode 1 selectable; SOLO default MaskGradients). Engine primitives audited in - * docs/redesign/findings-feedback-behaviour.md. + * docs/specs/recon/findings-feedback-behaviour.md. * * This class holds NO React. ConsoleApp owns one instance and exposes its * actions + state into the console context; VerdictCluster + Manifold drive it. diff --git a/manifold/src/feedback/index.ts b/manifold/src/feedback/index.ts index d464643..fda4a1b 100644 --- a/manifold/src/feedback/index.ts +++ b/manifold/src/feedback/index.ts @@ -2,7 +2,7 @@ * The learning-engine behaviour module (workstream B) — the two feedback modes * plus solo, prototyped in TS on the existing engine primitives. * - * See docs/redesign/rl-feedback-design.md for the authoritative design and the + * See docs/adr/rl-feedback-design.md for the authoritative design and the * C++ integration plan. Everything here is the TS-prototype-first layer; the * controller comments mark each place that becomes a C++ core primitive. */ diff --git a/manifold/src/inputs/midi-input-source.ts b/manifold/src/inputs/midi-input-source.ts index 353fb31..05984ed 100644 --- a/manifold/src/inputs/midi-input-source.ts +++ b/manifold/src/inputs/midi-input-source.ts @@ -25,7 +25,7 @@ * Pull-based: messages latch the latest per-binding value into `values`; * `sample()` copies them out. Hot path performs no IO/allocation. * - * NOTE: the canonical learn-map spec was `docs/redesign/inputs-spec.md`, which + * NOTE: the canonical learn-map spec was `docs/specs/inputs-spec.md`, which * is absent from this tree. The shape here follows the WebMidiBackend output * pattern (src/backends/midi-backend.ts) for symmetry. */ diff --git a/manifold/src/inputs/types.ts b/manifold/src/inputs/types.ts index 35efbc1..09d197c 100644 --- a/manifold/src/inputs/types.ts +++ b/manifold/src/inputs/types.ts @@ -7,7 +7,7 @@ * the active sources are concatenated into one N-dim input vector at the head of * the reactive spine (engine.setInput / setInputs). * - * Design intent (the canonical spec was `docs/redesign/inputs-spec.md`, which is + * Design intent (the canonical spec was `docs/specs/inputs-spec.md`, which is * NOT present in this tree — see TODO refs below): * * - **Pull-based.** The InputLayer owns a single rAF loop and pulls every diff --git a/vcv/src/iml.hpp b/vcv/src/iml.hpp index 3c631f5..090e76b 100644 --- a/vcv/src/iml.hpp +++ b/vcv/src/iml.hpp @@ -18,7 +18,7 @@ // the threading double-buffer stays race-free (each MLP owns its own RNG). // // It is NOT bit-identical to the firmware core (different optimiser internals), -// and that divergence is an accepted follow-up (see vcv/SPEC.md delta #5). The +// and that divergence is an accepted follow-up (see docs/specs/vcv-module.md delta #5). The // public method names mirror the core so `MEMLNaut.cpp` is unchanged in spirit. // // MPL-2.0 in spirit with the rest of nisps; wrapper code under the VCV module's