diff --git a/docs/redesign/BUILD-PLAN.md b/docs/redesign/BUILD-PLAN.md new file mode 100644 index 0000000..2b5ae16 --- /dev/null +++ b/docs/redesign/BUILD-PLAN.md @@ -0,0 +1,183 @@ +# Manifold — Build Plan & Locked Decisions (resume anchor) + +*Dated 2026-06-27. This is the single resume anchor for the Manifold convertible-app build. Read this + the +specs it points to before continuing. Mission: one working browser instrument putting the NEW Manifold +"convertible" front-end on top of the REAL parity-tested ML+audio engine.* + +## Locked decisions (operator-confirmed 2026-06-27) +| Decision | Choice | +|---|---| +| App directory | `manifold/` (Vite + React + TS), beside `playground/` until parity + sign-off | +| Engine to wire | The parity-tested **TS engine** (`playground/src/ml/wasm-iml.ts` + `audio/engine-host.ts` + worklet), same `nisps/` core the firmware builds from | +| Staging deploy | `meml.lnfinitemonkeys.org/next` (server-scope COOP/COEP already set); live a-immersive stays at `/` | +| Default FEEDBACK_MODE | **Explore & place** (Mode 2, positive-only); Geometric-dislike selectable | +| Default SOLO/arm | **Mask-gradients / column-freeze** (variant a); ZeroLoss + DontCareExample selectable | +| Modular N×M MLP | **Multiple WASM modules + warm-start** ({2,4} then {2,4,8}); NOT runtime-shaped, NOT padded | +| VCV bridge transport | **WS↔OSC bridge server** (reuse existing Deno bridge; bidirectional training) | +| Synth UI label | **"Powerful Synth Engine"** — the string "C15" must NEVER appear in the UI | +| Product copy | British spelling (randomise, visualise, colour) | + +## Spec docs (all in docs/redesign/) +- `rl-feedback-design.md` — the learning engine (both modes + solo); **§1 = verified ground truth: the feedback + C API, `MLHandle.feedback`, `0xFEEDBACC0DE` salt, parity Stage 5, and `CMakeLists.txt:59` registration ALL + ALREADY EXIST** — edit/extend, don't re-scaffold. Mode 1 ports to upstream **`0a541cc`** InterfaceRL. +- `dock-spec.md` — six drawers + per-output control row + tri-state semantics + advanced modals. +- `backends-spec.md` — `OutputBackend` adapter interface; **VCV module already at `vcv/` (2-in/12-out)** → + evolve to 8→16 + LED rings; faithful `visualizer.js` particle port (algorithm documented). +- `inputs-spec.md` — modular XY/WebMIDI/gamepad; multiple-WASM-module reshape with warm-start. +- `findings-{feedback-behaviour,engine-surface,design-and-manifold}.md` — Phase-1 audits. +- `upstream-firmware-survey.md` — git topology; latest InterfaceRL = `0a541cc`. +- `playground-2.0-rewrite-plan.md`, `engine-architecture.md`, `aimmersive-clone-spec.md`, + `feedback-modes-port-spec.md` — the prior planning corpus. + +## Reactive spine → React (load-bearing, from findings-design-and-manifold.md §4) +Spine lives BELOW React in an external store: `setInput` action derives processed→ml→routed eagerly+synchronously +and fires the single `backend.send` at its tail (off-render). React subscribes via `useSyncExternalStore` +(version counter, not the array); canvases (Manifold/particles) read in one rAF loop. `EngineApi` exposed via +React Context. Lint rule: skins may not import engine internals; engine may not import React. + +## Phase-3 build sequence (each step keeps live a-immersive up; gate core touches on parity-check.sh @1e-5) +1. **Pull Manifold JSX mirror** — DesignSync (session-bound) → `docs/redesign/manifold-export/` (only tokens on + disk now; pull `components/**`, `ui_kits/memlnaut/**` incl. `console/**`). +2. **Scaffold** `manifold/` — Vite + `@vitejs/plugin-react` + strict TS, COOP/COEP dev+preview headers, + `base:'./'` + base-aware WASM URLs, worklet via `?worker&url`, copy `public/nisps.{wasm,js}`. +3. **Engine adapter** — lift `WasmIML` + `EngineHost` + worklet + input/output pipelines from `playground/src`; + replace Solid `mlStore`/`coreBus` side-effects with an injected emitter; build the framework-neutral + `EngineApi` + the external-store reactive spine + React `EngineProvider`/`useEngine`. +4. **Primitives** — port the 12 design primitives (Button/Slider/PillToggle/Panel/Badge/Switch/StatusLine/ + XYPad/VirtualJoystick/ControlAxis/CurvePlot/Sparkline) from window-global JSX to ES-module React/TS + tokens. +5. **Console convertible shell** — ConsoleApp/CompositeStage/SplitStage/OutputStage/InputMini/Manifold/Dock/ + Drawers/VerdictCluster/ReadoutStrip/AltitudeNav (+Perform/Zen). Replace `MF_infer` with real engine. The + single draggable divider with width-based representation demotion (full manifold → pad → minimap) is the centre. +6. **Dock real contents** (dock-spec) — six drawers, per-output control rows, tri-state, advanced modals. +7. **Modular inputs** (inputs-spec) — InputSource adapters (XY/WebMIDI/gamepad), N-channel pipeline, multi-module + warm-start rebuild. +8. **Backends** (backends-spec) — OutputBackend adapters: Powerful Synth Engine, particle visualiser (faithful + port), WebMIDI, OSC (reuse Deno bridge), CV/gate. +9. **Feedback engine** (rl-feedback-design) — TS-prototype-first (audible directional-away + interpolate-between- + corners are pass/fail oracles via existing WasmIML primitives) → C++ core (ReplayStore + train-toward-targets + + solo column mask) → C API → TS FFI → dock selector. Every core step gated on parity. +10. **VCV module** — evolve `vcv/` to 8→16 + LED-ring widget (palette from tokens) + WS↔OSC bridge + + bidirectional training. +11. **e2e** — `window.__nisps` synchronous probe + Playwright (spine invariant: setInputs → outputs+params + change in one tick; no-per-frame-alloc fuzz). +12. **Deploy** `/next`, MAP.md + ALIGNMENT.md + CLAUDE.md sync, hand operator URL. Archive playground only after + parity + sign-off. + +## Animations (workstream G, parallel to build) — Manim skill at ~/src/hermes-agent/skills/creative/manim-video +Knob 1:1 · XY→2 numbers · dimensionality fan-out (knob/fader/touchpad/2D/3D joystick) · feedback contrast +(geometric push-away vs explore-and-place). Plus interactive in-app onboarding demos. Feedback piece reflects +the locked default (Explore & place). + +## Manifold mirror status (docs/redesign/manifold-export/) — updated 2026-06-27 +PRESENT on disk: all 6 token CSS + styles.css; 12 primitives (components/{core,control,data}/*.jsx); +console kit — ConsoleApp, CompositeStage, SplitStage, OutputStage, InputMini, Manifold, model, shared-ui, +Dock, Drawers, OutputEditor, VerdictCluster, ReadoutStrip, CurvePad. +STILL TO PULL via DesignSync (non-core; project id 49091559-c3ce-47cb-a14f-08d62c26fe09): console PerformApp.jsx, +ZenApp.jsx; ui_kits/memlnaut top-level (App, ModeView, OutputBars, TopBar, TrainingPanel); output-routing/ +(Editor, model, parts, variants); support.js; the specimen *.html. These are altitude/alt-routing variants and +specimens — not needed for the core convertible Console build (steps 2-9). Pull when building Perform/Zen +altitudes or the routing-matrix Full modal. + +Key port notes from the mirrored JSX: +- CompositeStage IS the convertible: single `split` ratio ∈[0,1], SNAPS [0.14,0.33,0.5,0.66,0.86] w/ magnetism, + SHUT=0.1 collapses a side to a draggable corner minimap; tier demotion by measured width (full→pad / field→list). +- model.jsx `MF_infer` is the sin/cos placeholder to REPLACE with WasmIML; it still lists a `c15` mode labelled + "C15" → relabel to "Powerful Synth Engine" (and the synth backend everywhere). +- Drawers.jsx = the PLACEHOLDER drawers (Shape/Feel/Route/Health/Help) → replace with dock-spec's six real drawers. +- OutputEditor/OutputStage/ReadoutStrip already implement the off/fixed/live tri-state + per-output min/max/curve + + alt-click cycle — the baseline the dock-spec per-output control row formalises (add mute + solo/arm). +- Components use `window.*` globals + `window.ManifoldDesignSystem_490915` — convert to ES-module imports. + +## Build status — 2026-06-27 (Phase-3 in progress) +DONE + verified (typecheck+build green): scaffold `manifold/`; engine layer (framework-neutral EngineApi + +reactive spine + EngineProvider/useEngine + window.__nisps probe; added the missing nisps_ml_feedback_* TS +bindings — no C change); 12 primitives; convertible Console shell (ConsoleApp/CompositeStage/etc) wired to the +real engine (MF_infer replaced; default mode = Explore-and-place; synth = "Powerful Synth Engine", 0 "C15" in +bundle). **DEPLOYED to https://meml.lnfinitemonkeys.org/next/** (docroot subdir of meml-aimmersive, no nginx +change; COOP/COEP inherited; live a-immersive untouched at /). To redeploy: `cd manifold && bun run build` then +`rm -rf /home/w1n5t0n/deployments/meml-aimmersive/next && cp -r manifold/dist /home/w1n5t0n/deployments/meml-aimmersive/next && cp .../next/index.html .../next/a-immersive.html` (the a-immersive.html copy satisfies nginx `index a-immersive.html` so /next/ resolves). + +CAVEAT — e2e not yet run on the VPS: headless Chromium needs `libnspr4`/`libnss3` which aren't installed and +need operator sudo (`sudo bunx playwright install-deps chromium`, or run e2e on the laptop/CI). The smoke spec is +written at `manifold/tests/e2e/smoke.spec.ts` (asserts: engine WASM loads, spine invariant setInputs→outputs +change, feedback runs, console renders, no "C15"). Serving smoke (HTML+COOP/COEP+wasm) passed via curl. Operator +should verify the live runtime in a real browser at /next/. + +REMAINING Phase-3+: real dock contents (dock-spec) · modular inputs (inputs-spec) · output backends + particle +port (backends-spec) · feedback C++ engine (rl-feedback-design, TS-prototype-first) · VCV module · full e2e · +animations (G). Then MAP.md/ALIGNMENT.md sync + commit + operator sign-off → archive playground. + +## Build status update — 2026-06-28 +ADDED + verified (typecheck+build green, redeployed to /next): real dock contents (six drawers + OutputControlRow +with off/fixed/live + mute + solo/arm + dual-range + curve, unified output-state store, per-backend advanced +modals, feedback-mode + solo + audio wired) AND the feedback-engine TS prototype (manifold/src/feedback/: +FeedbackController — Explore-and-place scratchpad loop snapshot→randomise/nudge→place-at-chosen-location→ +warm-start anchor interpolation, undo ring, geometric-dislike mode, example-level solo respect; SeededRng; +mode-aware VerdictCluster + Manifold pick-location state). C++ crystallisation gaps tagged inline `--- C++ GAP ---` +→ rl-feedback-design §4 (replay store, k-NN geo-push, train_masked column-freeze). ALIGNMENT.md got the +output-state divergence note from the dock agent. +STILL REMAINING: backends + particle visualiser port (E), modular MIDI/gamepad inputs (F), VCV module (E), +C++ feedback crystallisation (B phase 2, gated on parity), full browser e2e (needs libnspr4 via operator sudo), +Manim animations (G), MAP.md sync + commit + sign-off → archive playground. + +## RUNTIME VERIFIED — 2026-06-28 ✅ +The /next deploy is browser-verified: the e2e smoke (manifold/tests/e2e/smoke.spec.ts) PASSES — engine WASM +loads, spine invariant holds (setInputs→outputs change), feedback runs, console renders, no "C15", no errors. +Two runtime bugs found+fixed in the lifted engine: +1. assetUrl resolved against location.origin → dropped the /next sub-path. FIXED: resolve against document.baseURI + (main thread) + pass the deploy base to the async-train worker via its init message. +2. nisps.js is Emscripten MODULARIZE glue with NO ES6 exports — import() yields an empty namespace. FIXED: + fetch + indirect-eval the glue to install global createNispsModule (works on main thread AND module worker). + (The playground's identical import()-based loader has the same latent flaw — worth fixing there too.) +E2E INVOCATION on this VPS (bun runs under snap confinement that hides system libs from its Chromium subprocess; +run the test RUNNER via non-snap node, browsers under the bun cache path): + cd manifold && (bun run preview &) + PLAYWRIGHT_BROWSERS_PATH=/home/w1n5t0n/snap/bun-js/87/.cache/ms-playwright node node_modules/.bin/playwright test + +## Build status update — 2026-06-28 (later) +DEPLOYED + e2e-verified to /next, all green: +- Dock RESTRUCTURED (operator): top Mode selector (Particle System[default]/MIDI/OSC/Built-in Synth/MEMLNaut + Editor) + 5 vertically-centred drawers (Learning/Inputs/Outputs/Settings/Help). Old Synth/Visual drawers folded + into per-Mode config in Outputs. +- Settings drawer: monochrome icons (focused orange / unfocused off-white|white|orange), input-map rect↔circular + toggle, corner-radius slider (default 2px, overrides --r-1/--r-2 on :root). Store: manifold/src/settings/. +- Feedback markers on the 2D map: positive (filled dot) + negative (open red ring) at their input location. +- Monochrome SVG icon set (manifold/src/console/icons.tsx, currentColor). +- MEMLNaut Editor mode: Web Serial USB scaffold (manifold/src/serial/, stub). +- MIDI + OSC backends REAL: manifold/src/backends/ (OutputBackend + BackendManager consuming the spine; WebMIDI + out w/ per-output CC#/ch/name/range; OSC-over-WS to the Deno bridge in manifold/osc-bridge/). Outputs panel + SPECIALISES per backend (dock/OutputsBackendConfig.tsx) + named presets (backends/presets.ts, per-backend + localStorage). Audio gated via engine.audio.setMuted on non-synth modes. +Naming: synth shows as "Built-in Synth" (operator's latest wording; still never "C15"). +REMAINING: faithful particle-system port (Particle System is default Mode — currently passthrough/no flow field +yet), modular MIDI/gamepad INPUTS (F), VCV module (E), C++ feedback crystallisation (B ph2, parity-gated), Manim +animations (G), MAP/ALIGNMENT sync + commit + sign-off → archive playground. Nothing committed yet. + +## VCV module — BUILT 2026-06-28 ✅ +vcv/ evolved 2→12 → **8 inputs × 16 outputs**, **compiles + links** against Rack SDK 2.6.4 +(installed at ~/.local/share/Rack2/Rack-SDK; `cd vcv && RACK_DIR=$HOME/.local/share/Rack2/Rack-SDK make` → +plugin.so 2.4MB). Per-output **LED rings** (src/LedRing.hpp, drawLayer+nvgArc, proportional fill + glow), +**palette.hpp** from frontend tokens (16-step orange→cyan ramp). **WS↔OSC bridge** (src/osc_server.hpp) verbs +/nisps/{input,output,feedback,weights,examples,state}, default UDP 7001+id%64, routed through the SAME +doThumbsUp/Down/enqueueJob paths as the panel — trainable from BOTH VCV and the browser. Retired nisps-core +resolved by **vendoring src/iml.hpp** (self-contained runtime IML, firmware-aligned semantics but not bit-identical +— follow-up). Derived outputs moved behind a context-menu toggle. TODOs: browser-side /nisps/feedback emission; +redraw panel SVG art for 8/16 (coords relaid in code); align vendored IML optimiser to nisps/ml bit-exactly. +REMAINING overall: faithful particle-system port, modular MIDI/gamepad inputs (F), C++ feedback crystallisation +(B ph2), Manim animations (G), MAP/ALIGNMENT sync (done this round) + commit + sign-off. + +## Explore-and-place CRYSTALLISED to shared C++ core — 2026-06-28 ✅ (parity-verified independently) +nisps/ml/feedback.hpp gained FeedbackMode::ExploreAndPlace + Idle/Exploring/Placing state machine (no-heap, +nisps::Rng, .f). Granular methods (enter_explore/exit_explore/reroll/nudge/undo/begin_place/commit_place/ +cancel_place) for firmware button-mapping + on_down/on_up browser default policy. Controller owns weights+ +scratchpad+undo-ring+frozen placed_out_; caller owns add_example+train. C API + EXPORTED_FUNCTIONS added. +VERIFIED (orchestrator re-ran, not agent's word): native ctest 4/4; fresh build-wasm + parity-check.sh = PASS +821 floats within 1e-5 (max 2.4e-7), parity Stage 5d exercises enter→reroll→nudge→undo→place→commit. Fixed bug: +set_mode(3) fell back to Avoid. nisps.wasm synced to playground/public + manifold/public (md5-identical). +manifold/src/feedback/controller.ts REWIRED to call the shared core (TS scratchpad deleted) — app explore-and-place +is now firmware-identical; e2e green; redeployed /next. Firmware glue (peripherals.hpp) wired to the hardware flow +(down=enter/exit explore, reroll+nudge buttons, like=begin place, up=commit) — NOT compiled (no arduino-cli); +Placing-audition static_output() hook is a flagged follow-up. Mode 1 (geometric dislike) untouched. +REMAINING: faithful particle-system port, modular MIDI/gamepad inputs (F), Manim animations (G), firmware build ++ flash verification, browser-side /nisps/feedback VCV bridge emission, commit + sign-off. diff --git a/docs/redesign/aimmersive-clone-spec.md b/docs/redesign/aimmersive-clone-spec.md new file mode 100644 index 0000000..d961a80 --- /dev/null +++ b/docs/redesign/aimmersive-clone-spec.md @@ -0,0 +1,350 @@ +I now have a thorough, complete understanding of the entire deployed app — DOM, CSS theme tokens, all interaction handlers, state model, engine call points, persistence, and the help-modal copy. I have everything needed to write the faithful clone spec. Let me write it now. + +# NISPS Immersive (`a-immersive.html`) — SolidJS Clone Spec + +This is a faithful, component-by-component rebuild spec for the deployed vanilla-JS app at `/home/w1n5t0n/deployments/meml-aimmersive/`. Source of truth: `a-immersive.html` (446-line DOM), `css/a-immersive.css` (2538 lines), `js/a-app.js` (4521-line module), and supporting modules under `js/nisps/`, `js/ui/`, `js/synth/`, `js/midi/`, `js/audio/`. Goal: a from-scratch SolidJS reimplementation that matches the look and behaviour 1:1, binding to a shared engine layer (the WASM IML + synth engines) rather than re-porting it. + +--- + +## 0. Critical scoping note before you build + +**The deployed app does NOT use the Boldness/Memory/Precision control surface.** `README.md` and `SPEC-controls.md` describe that system (`js/ui/control-surface.js`, `control-surface-ui.js`, `input-pipeline.js`), but **`js/a-app.js` never imports any of them**. (Verified: `grep "control-surface" js/a-app.js` → no hits.) Those modules belong to a different design and to the SolidJS migration target (`PLAN-solidjs-migration.md`). + +What a-immersive *actually* exposes for ML tuning is a flat **"NISPS" params drawer** with 6 raw sliders: **Spread, Noise, RL Decay, Learn Rate, Max Iters, Convergence** (`buildEngineParams()`, a-app.js:3137). To clone a-immersive 1:1, build **that** drawer. If the new SolidJS app is meant to also gain Boldness/Memory/Precision, treat it as an additive feature on top of this spec, not a replacement. + +Joystick input is fed **directly** (`joyX, joyY` ∈ [0,1]) into `iml.setInput(0/1, …)` — there is **no deadzone/zoom/curve/smoothing pipeline** in a-immersive. The only output-side shaping is the per-param override system (curve/min/max/freeze). + +Many imports in a-app.js are for features visible only behind flags or that exist as dock entries (EOC Effects Chain, Modular engine, ShapeSeq behind `?shapeseq=1`, Faust additive/FM engines, audio-canvas, hand tracking). See §10 for what's core vs deferrable. + +--- + +## 1. Component tree + +Top-level provider tree wraps everything in store contexts (see §5). All components are absolutely/fixed-positioned over a full-viewport canvas — this is a single-screen HUD, not a flow layout. + +``` + // root; owns rAF loop, resize, init, keyboard, gamepad +├── // fixed top-left chevron → designs.html +├── // shown only when outputMode==='synth' +│ ├── // play/pause icon, audio-needs-init pulse +│ ├── // hover flyout: Vol + BPM sliders +│ └── // tiered , preset ` (min blue `#4488ff`, max orange `#ff6a00`) with a fill bar; clamps min≤max; label `"min–max"`. Routes live. + - **Freeze**: button toggles `frozen`; when frozen, captures current output into `fixedValue`, reveals a value slider (blue), and the param is removed from NISPS control (its output is pinned to `fixedValue`). Frozen cells get a hatched overlay + dimmed bar (`.heatmap-cell-frozen`). + - Popup stays open while hovered; closes 300 ms after leaving both popup and cell. + +### 2.5 Synth visualizer + group drawer (`SynthVisualizer`:584, `showGroupDrawer()`:3953) +- **Bars**: only non-muted params drawn, grouped with 2 px gaps between sections; each section gets a centered label at the top (`_drawSectionLabel`). Bar height = display value (lerped at 0.12/frame toward target). Hover any bar → in-canvas tooltip (name / Val / Range / Curve). +- **Drag a bar** (`enableInteraction(true)` in synth mode): sets that param's raw value from Y (`yToValue`), routes + updates heatmap. Sliding across bars reassigns the dragged index. +- **Hover/click a section label** → `` (320 px glass card, portal). Contents: + - **Group master curve**: 48×48 draggable canvas. Vertical drag applies a *relative delta* to the group curve **and to every per-param curve** in the section (preserving relative offsets). `delta = dy/80`. + - **Per-param rows**: name, 28×28 per-param curve canvas (vertical drag), dual-range min/max slider, and an **M (mute)** button. Muted rows hide curve+range, show a grey value slider, and strike-through the name. Mute removes the param from NISPS control. + - For C15 this is backed by `groupOverrides[si]` + `SYNTH_SECTIONS`; for Faust engines by `engineParamOverrides` + `nonC15Sections` (group curves remembered by name across engine swaps). Unify behind a `getSectionView(sectionIndex)` adapter (a-app.js:3916). + +### 2.6 Output-mode tabs (`setOutputMode()`:2786) +In ``, pill toggle: **Visual / Synth / MIDI CC / Audio Canvas** (`#output-toggle-float`). On switch: +- Compute `targetOutputs = outputCountForMode(mode)` (visual 20, synth = engine paramCount [126 for C15], midi-cc = `midiCCMap.length`, audio-canvas = `audioCanvas.getOutputCount()`). +- If output count changes **and** there are training examples: `confirm()` a weight-reset warning; on cancel, revert pill. Then `resizeMLP()` (warm-starts joystick weights, clears examples, recreates hand IML). +- Toggle canvases/strips: synth → hide `#vis-canvas` (`.hidden-canvas`), show `#synth-vis-canvas` (`.active`), hide heatmap strip, show synth quick controls, `synthVisualizer.enableInteraction(true)`. Other modes show heatmap strip, hide synth vis, disable interaction; midi-cc shows MIDI quick controls; audio-canvas shows its wrap. +- `buildHeatmap()` + `routeOutputs()` + `buildEngineParams()` + sync. + +### 2.7 Input toggle + follow (`wireInputToggle()`:2622, `toggleFollowMode()`:2360) +- `` "Input" pill toggle: **Joystick / Hands**. Switching to Hands lazily constructs `HandTracker`, requests camera, swaps `iml=imlHand`, hides joystick, shows PIP; on camera error reverts to joystick. (Hand tracking deferrable — see §10.) +- **Follow pill** (`#follow-pill`) and the joystick double-tap both call `toggleFollowMode()`. `updateFollowUI()` shows the FOLLOW badge, adds `.follow-active` (pulsing border) and marks the pill active. + +### 2.8 Dock icons → drawers (`wireDock()`:2521) +- Click a `.dock-icon[data-drawer=X]`: if `X==='help'` open the modal; else toggle `#drawer-X` `.hidden` and the icon's `.active`. `.drawer-close` buttons close their drawer + deactivate the icon. Drawers are independent (stack scrolls). The 6 icons: **Training, Mode, Synth, NISPS (params), FX (eoc), Help**. + +### 2.9 Presets +Two distinct preset systems — keep them separate: +- **Visual/RL presets** (`` chips: Calm/Chaos, Rainbow, Vortex, Spiral, Embers) — `loadPreset(name)` (3060): clears the **joystick** dataset, adds the preset's hardcoded input→output examples (`PRESETS`, a-app.js:59; each output is 20 floats padded to N), trains synchronously, re-infers. These teach the *network*. +- **Synth presets** (`` dropdown, tiered) — `applyPreset(presetId)` (460): sets which of the engine's params are active vs muted and their min/max/curve/fixedValue, via `groupOverrides` (C15) or `engineParamOverrides` (Faust). Tiers expose progressively more params (Beginner 15 → Expert full). These shape the *output mapping*, not the network. `?preset=` URL param auto-applies on load. + +### 2.10 Gamepad (`wireGamepad()`:3645) +`GamepadInput` with `invertY:true`. Left stick → `joyX/joyY` + `onJoystickMove`. Buttons: LB→down, RB→up, A→train, X→randomize, B→clearExamples. Status text in `#gamepad-status`. + +--- + +## 3. Theme / branding tokens and key copy + +### CSS custom properties (`:root`, css:11) — reproduce verbatim +```css +--glass-bg: rgba(13, 13, 13, 0.65); +--glass-border: rgba(255, 255, 255, 0.08); +--glass-blur: 16px; +--accent: #ff6a00; /* signature orange */ +--accent-dim: rgba(255, 106, 0, 0.25); +--danger: #ff4466; +--text: #e0e0e0; +--text-dim: #888; +--radius: 12px; +--radius-sm: 8px; +--safe-bottom: env(safe-area-inset-bottom, 0px); +--dock-width: 48px; +--dock-gap: 8px; +``` +- **Background**: `#0d0d0d` everywhere; `html,body { overflow:hidden; touch-action:none; user-select:none }`. +- **Font**: `'JetBrains Mono', 'SF Mono', 'Fira Code', 'Cascadia Code', monospace`; base `13px`. Loaded from Google Fonts (`JetBrains+Mono:wght@400;500;700`). +- **Glass mixin**: `background: var(--glass-bg); backdrop-filter: blur(var(--glass-blur)); border: 1px solid var(--glass-border)`. Drawers use a denser `rgba(13,13,13,0.88)` + `blur(20px)`. +- **RL up** = teal `#00c8a0` (border/bg tinted); **RL down** = red-orange `#dc3c14`. EOC RL up = `#4ecdc4`. Noise ring = `rgba(255,100,50,…)` → high `rgba(255,60,40,…)`. +- **Dock**: macOS magnify — `.dock-icon:hover { transform: scale(1.35) }`, neighbours scale 1.12 via `:has()`/sibling selectors; active icon tinted orange. +- **Animations to reproduce**: `drawerSlideIn` (0.2 s translateX), `follow-pulse` (1.5 s box-shadow), `rl-flash` / `btn-flash` (0.25 s scale), `audioInitPulse` (2 s, on play button when audio not started), `helpFadeIn`. +- **Heatmap frozen cell**: 45° hatched `rgba(80,160,255,0.15)` overlay + bar opacity 0.4. + +### Key copy to reproduce verbatim +- **Title** (``): "NISPS Immersive". **Help modal H2**: "Welcome to NISPS"; subtitle: "Neural Interactive Shaping of Parameter Spaces". +- **Status line** default: `"0 examples · untrained"`; live format: `"<N> example(s) · loss <0.00000> · noise <0.000>"` (or `· untrained` when never trained). Middots are U+00B7. +- **RL buttons**: down has `−` + `1`; up has `+` + `2`; titles "Explore more" / "Keep this". Undo title "Undo last action". +- **Help modal** is a long static block — reproduce sections **What is this? / How it works / What to expect / Controls (Touch/Mouse, Keyboard 1/2/3/4/Z, Gamepad) / Hand Tracking / Synth Controls**, plus the "Got it" button. (Full text is in `a-immersive.html` lines 354–441 — copy it wholesale; it's the authoritative onboarding copy.) Auto-opens when `localStorage['nisps-help-seen']` is unset; closing sets it. +- **Dock labels**: Train, Mode, Synth, NISPS, FX, Help. **Drawer headers** (uppercase orange): Training, Mode, Synth, MIDI CC, NISPS, Effects Chain. +- **Training drawer buttons**: Add Example, Train, Clear Ex, Clear All, Randomize; preset chips Calm/Chaos, Rainbow, Vortex, Spiral, Embers. +- **Synth preset `<select>`**: option "Manual" (value `""`), then optgroups Beginner(1.1–1.4), Intermediate(2.1–2.4), Advanced(3.1–3.3), Expert(4.1–4.2). +- **Back button** → `designs.html`. + +--- + +## 4. SVG icons (inline, reproduce as components) +All dock icons, the back chevron, the undo arrow, the play/pause/MIDI-keyboard icons, and the gesture ring are inline SVGs in the HTML. Reproduce as small SolidJS components (paths are in `a-immersive.html`): +- Back: `<path d="M10 2L4 8l6 6"/>` (stroke). +- Play: `<path d="M4 2l10 6-10 6z"/>` (fill); Pause: two rects. +- Dock: Train (mixer faders), Mode (two circles + diagonal), Synth (sine wave `M2 10c2-4 4-4 6 0s4-4 6 0`), NISPS (gear), FX (infinity-ish loop), Help (question mark in circle). +- Undo: `<path d="M3 7h6a4 4 0 0 1 0 8H7"/><path d="M3 7l3-3M3 7l3 3"/>`. + +--- + +## 5. State model → SolidJS stores/signals + +The deployed app keeps state in module-level `let`s. Map to stores/signals. **Performance rule** (from migration plan): ML outputs should be a single `Float32Array` signal updated once per frame, not a reactive array of 126 cells — drive canvases imperatively, drive the heatmap DOM widths via a cheap effect. + +| Deployed state (a-app.js) | SolidJS home | Notes | +|---|---|---| +| `iml`/`imlJoy`/`imlHand`/`imlEoc` | non-reactive refs in an **EngineContext** | WASM instances; never put in a store | +| `inputMode` ('joystick'\|'hands') | `inputStore` signal | drives joystick vs PIP | +| `outputMode` ('visual'\|'synth'\|'midi-cc'\|'audio-canvas') | `modeStore` signal | gates canvases, heatmap, quick controls | +| `joyX,joyY,joyDragging,joyFollowMode,joyTrail` | `inputStore` (x/y signals + follow signal) | x/y change every frame → keep as plain signals | +| current outputs (`iml.getOutputs()`) | `outputStore` — **one `Float32Array` signal** | updated once per `routeOutputs` | +| `rawParamValues` | `outputStore.raw` (Float32Array signal) | the "set sliders" buffer for examples/heatmap drag | +| `noiseLevel,spreadLevel,rlExplorationDecay` | `mlStore` signals | spread also from `?spread`; default 0.6, noise 0.05, decay 0.97 | +| `iml.learningRate / maxIterations / convergenceThreshold` | exposed via `mlStore` getters/setters | the 6 NISPS sliders write these | +| `exampleCount,lastLoss,lossHistory` | `mlStore` (derived from iml after each op) | status line + loss canvas | +| `undoStack` (cap 20) | `mlStore.undo` (non-reactive array + `hasUndo` signal) | snapshot = {weights,noiseLevel,exampleCount} | +| `visualOverrides[20]` | `overrideStore.visual` (createStore) | {min,max,curve,frozen,fixedValue} | +| `groupOverrides[18]` (C15 sections) | `overrideStore.group` (createStore) | {curve, params:[{min,max,curve,muted,fixedValue}]} | +| `engineParamOverrides` (Faust, flat) | `overrideStore.engine` | null for C15 | +| `midiCCMap,midiCCOverrides` | `midiStore` | per-engine storage key | +| `audioCanvasOverrides` | `overrideStore.audioCanvas` | dynamic length | +| `activeEngine,activeSynthPresetId` | `synthStore` signals | engine id + preset id | +| `eocChain,eocTrainingTarget,nispsMode` | `eocStore` | deferrable | +| drawer open flags, active dock icon | `uiStore` (per-drawer booleans) | independent drawers | +| `activePopupParam,activeDrawerSection` | `uiStore` signals | popup/group-drawer routing | + +**Persistence** (`saveState()`:4298, `loadState()`:4346): single `localStorage['nisps-a-immersive']` JSON blob. Saved fields: `features/labels` (joystick dataset), `handFeatures/handLabels`, `noiseLevel`, `outputMode`, `inputMode`, `joyX/joyY`, `groupOverrides`, `visualOverrides`, `midiCCOverrides`, `audioCanvasState`, `synthPresetId`, `engineId`, EOC module/mode state, `modularDspState`. On load: re-add examples and **train synchronously**, restore overrides, restore output mode (`skipConfirm:true`), but **never auto-restore `inputMode='hands'`** (camera permission). Auto-save fires on a 10 s interval and `saveState()` is also called after preset apply etc. In SolidJS, run a debounced effect (the migration plan uses 200 ms) that serializes the same shape; keep the storage key identical for migration continuity. Separate keys: `nisps-help-seen`, `nisps-midi-cc-map:<engineId>`, `nisps-modular-state`. + +--- + +## 6. Engine call points (bind to the shared engine layer) + +The shared engine layer is **`WasmIML`** (`js/nisps/nisps-wasm.js`) for ML, plus a **`SynthEngine`** (default `C15Adapter`) for audio. The new SolidJS app should transplant these (per migration plan they move under `core/` largely unchanged) and call them from store actions. Exact bind points: + +**Construction (`init()` / `resizeMLP()`):** +- `imlJoy = await WasmIML.create(2, N_OUTPUTS, [32,48,64], 1000, 1.0, 0.00001)` then `imlJoy.randomiseWeights(spreadLevel)`. +- `imlHand = await WasmIML.create(14, N_OUTPUTS, [48,48,64], …)`. +- On output-count change: `WasmIML.createWithWarmStart(snapshot, newCount, …)` for the joystick IML (preserves hidden weights, re-randomises new output nodes), fresh `create` for hand. +- WASM module loaded from `wasm/nisps.js` (Emscripten). Note: the **engine MLP that runs audio in the worklet** is a separate WASM instance fixed at `MLP<2,10,14,18,126>` per CLAUDE.md — but in a-immersive the *inference path* is this `WasmIML`, sized to the active mode's output count. + +**Per-frame inference (joystick/hand move, `onJoystickMove`/`onHandInput`):** +- `iml.setInput(0,x); iml.setInput(1,y); iml.process(); const out = iml.getOutputs();` → `routeOutputs(out)`. + +**`routeOutputs(out)` (2425) — the output fan-out** (bind one store action): +- **synth**: apply `applyGroupOverrides(out[i], i)` per index → `synthVisualizer.setParams(overridden)`; throttled (≥50 ms) + dead-zone (>0.002) `activeEngine.setParam(i, v)` for indices < engine paramCount; shared-mode EOC params routed to `eocChain.setParam`. +- **midi-cc**: per CC, `applyGroupOverride(out[i],curve,min,max)` → `midiOutput.sendBatch([{channel,cc,value:round(v*127)}])`. +- **audio-canvas**: `audioCanvas.setOutputs(out)`. +- **visual**: per-index `applyGroupOverride` (or `fixedValue` if frozen) → `visualizer.setParams(vis)`. + +**RL / training engine calls** (the interaction→engine mapping the prompt cares about): +| UI action | Engine call | +|---|---| +| **thumbsUp / + / key 2 / RB / hand-up** | `iml.addExample(inputs, rawOutputs)` **+** `iml.trainAsync(cb)` | +| **thumbsDown / − / key 1 / LB / hand-down** | `iml.moveWeights(noiseLevel, spread)` *(no example)* | +| **Add Example button** | `iml.addExample(getCurrentInputs(), [...rawParamValues])` | +| **Train button / gamepad A** | `iml.trainAsync(cb)` (async; updates loss canvas) | +| **Randomize / gamepad X** | `iml.randomiseWeights(spread)` → re-infer; resets noise to 0.05 | +| **Clear Ex / gamepad B** | `iml.clearDataset()` | +| **Clear All** | `clearDataset()` + reset `lossHistory/bestLoss/iterations` + `clearState()` | +| **Undo / key Z** | `iml._setFlatWeights(snapshot.weights)` → re-infer | +| **Visual preset chip** | clear joystick dataset, `addExample` × preset rows, `iml.train()` (sync) | +| **NISPS sliders** | write `spreadLevel` / `noiseLevel` / `iml.learningRate` / `iml.maxIterations` / `iml.convergenceThreshold` | + +**WasmIML public surface to rely on** (stable contract): `setInput/setInputs/getOutputs/process/inferBatch`, `addExample/clearDataset/exampleCount`, `train()/trainAsync(cb)/isTraining/evalLoss()`, `randomiseWeights(spread)`, `moveWeights(speed,spread,pinMask?)`, `extractWeights()/createWithWarmStart()`, `_getFlatWeights()/_setFlatWeights()`, `getLayerStats()`, `lastLoss/bestLoss/lossHistory/totalTrainingIterations`, `dataset.features/labels`, `destroy()`. Training runs in a disposable Web Worker (`nisps-wasm-worker.js`). + +**SynthEngine (`C15Adapter`) surface**: `id`, `displayName`, `paramMeta` (array of `{id,name,min,max,init,curve,group}`), `paramCount`, `init(ctx?)`, `stop()`, `running`, `setParam(index, normalized)`, `noteOn(note,vel)`, `noteOff(note)`, `getOutputNode()`, `setMasterVolume(v)`, `panic()`. Audio is started lazily on the play button (`activeEngine.init()`), which also wires EOC and (behind flag) ShapeSeq. Arpeggiator (`Arpeggiator`) and `MIDIInput` are attached to the active engine. + +### Where the NEW 3-mode feedback selector ("Down Action") slots in +Today **`−` (thumbsDown) is hardcoded to `moveWeights`** (explore) while **`+` is hardcoded to `addExample`+`train`** (reinforce). The new feature wants the `−` action to be selectable among 3 feedback modes. Slot it as a **`<DownActionSelector>`** control: + +- **Placement (UI)**: a small segmented control. Most faithful spot is inside the **Training drawer** (a new "Down Action" row above the example actions) and/or a compact 3-way toggle adjacent to the RL cluster (mirroring how the existing `rl-label`/EOC label sit above the buttons in linked mode). It must be reachable without opening a drawer if it's meant for live use — consider a tiny pill under `<StatusLine>` or next to the `−` button. +- **State**: add `downAction: 'explore' | <mode2> | <mode3>` to `mlStore` (persist it in the state blob). +- **Bind point**: `onThumbsDown()` is the single chokepoint. Branch on `mlStore.downAction`: + - `explore` (current): `target.moveWeights(noiseLevel, spread)`. + - the two new modes: call the corresponding engine primitive (e.g. a "negative example" path via `addExample` of a contrasting target, or a stronger/weighted `moveWeights`, or `randomiseWeights` — whatever the 3 modes are defined to do). `WasmIML.moveWeights` already accepts an `outputPinMask`, so a "pin-aware explore" mode is a natural third option. + - Keep `pushUndoSnapshot()` + `updateNoiseRing()` + `flash` shared across all branches. +- The existing `_rlTarget()` indirection (which lets EOC linked-mode redirect feedback to `imlEoc`) should compose with the new selector — i.e. `downAction` chooses *what* the feedback does, `_rlTarget()` chooses *which network* receives it. + +--- + +## 7. Reference data tables (must match exactly) + +### Visual mode: 20 named outputs (`VISUAL_PARAM_NAMES`, a-app.js:46 + `visualizer.setParams`:159) +Names: `Flow, Scale, Speed, Hue, Spread, Size, Trail, Turb, Attract, Radius, DispRate, DispAmt, Lifetime, Respawn, Advection, Inertia, Drag, Repulse, RepCnt, RepRate`. Colors in `VISUAL_PARAM_COLORS` (a-app.js:51, e.g. `Flow=#ff6a00, Scale=#00ccff, Speed=#ff6600, Hue=#ff00cc, …`). Output→visual-param ranges (verbatim, for the FlowFieldCanvas): `angleOffset=out0*2π`, `scale=0.001+out1*0.009`, `speed=0.5+out2*4.5`, `hueBase=out3*360`, `hueSpread=out4*120`, `particleSize=1+out5*5`, `fadeRate=0.01+out6*0.14`, `turbulence=out7*2`, `attractStrength=0.1+out8*2.9`, `attractRadius=40+out9*420`, `dispersionRate=0.2+out10*8`, `dispersionAmount=out11*3`, `particleLifetime=30+out12*470`, `respawnStyle=out13`, `advectionMode=out14`, `inertia=out15*0.98`, `drag=out16*0.35`, `repulsorStrength=out17*4.5`, `repulsorCount=floor(out18*4.999)`, `repulsorOrbitRate=0.1+out19*2.9`. 400 particles, value-noise flow field with central attractor, dispersion pulses, and orbiting repulsors. Transplant `FlowFieldVisualizer` as-is. + +### Synth mode: C15 sections (`SYNTH_SECTIONS`, a-app.js:217) — for grouping the 126 bars +`Env A(7), Env B(7), Env C(6), Osc A(5), Osc B(5), Shp A(6), Shp B(6), Comb(8), SVF(9), Gap(6), FB Mix(9), Out Mix(14), Cabinet(8), Flanger(13), Echo(7), Reverb(6), Unison(3), Mono(1)` = 126, each with a section color. Param data (id/name/label/default/safeMin/safeMax/bipolar) is `SYNTH_PARAM_MAP` (`js/synth/param-map.js`, 126 entries) — transplant as data, unchanged. + +### Curve math (`param-map.js`:287) — reproduce exactly +``` +applyCurve(v, c) = c===0.5 ? v : v^(2^(4*(c-0.5))) // 0.5 = linear +applyGroupOverride(v,c,min,max) = min + applyCurve(v,c) * (max-min) +``` + +### `?tame` (default 1): seeds each C15 param's default override range to `[safeMin*tame, 1-(1-safeMax)*tame]` (a-app.js:248). `?spread` (default 0.6), `?preset`, `?debug`, `?shapeseq=1` are the URL params. + +--- + +## 8. Layout / positioning cheat-sheet (fixed coordinates to match) +- Back button: `top:45px; left:8px; 36×36`. Synth quick controls: `top:45px; left:52px` (row). Heatmap strip: `top:0; height:22px`. Joystick: `bottom:100px; left:24px; 160×160`. Hand PIP: same anchor, `180×135`. RL buttons: `bottom:36px; left:50%` centered, 64×64 circles, 12 px grid gap, undo 28×28 spanning. EOC RL: `bottom:36px; left:calc(50%+100px)`, 48×48. Status line: `bottom:8px; center`. Dock: `right:8px; top:50%` vertical, 36×36 icons, 16 px radius glass. Drawer stack: `right:calc(48px+16px); top:28px; bottom:28px; width:260px`, scrolls, `pointer-events:none` on stack / `auto` on each drawer. Drawer max-height 400 px. Group drawer 320 px, param popup 260 px — both portalled to body and positioned relative to the hovered element. +- z-index ladder: canvases 0–1, heatmap strip 20, tooltip 25, joystick/back/quick 30, status 40, drawers/RL 45, dock 50, group-drawer/popup 50, dev panel 90, help overlay 100. + +--- + +## 9. SolidJS implementation notes +- **Canvases**: `<FlowFieldCanvas>` / `<SynthVisualizer>` / `<JoyMapCanvas>` are components with a `ref` and an internal draw method; the **shell's single rAF** calls the active one (don't give each its own loop — matches `animate()` and avoids double-drawing). Resize via a window listener that re-inits particles. +- **Heatmap**: render `<For>` over `count` cells once per mode change; update bar widths via a `createEffect` reading the outputs signal — but throttle DOM writes (the deployed app writes width % directly each frame; with 126 cells prefer writing only changed cells). +- **Drawers/overlays**: build headless primitives (`Drawer`, `Overlay`, `Popover`, `Dock`) per the migration plan; the dock/drawer toggle is just boolean signals in `uiStore`. Independent open state (multiple drawers can coexist). +- **Pointer capture**: the joystick, heatmap cells, synth bars, curve canvases, and dual-range sliders all use `setPointerCapture` + manual drag math — replicate the exact thresholds (3 px click-vs-drag on heatmap; double-tap 350 ms; group-curve `dy/80`; per-param curve drag). +- **Untracked debug probe**: keep `window.__nisps` (synchronous, bypasses reactivity via `untrack`/`batch`) for Playwright — its API is the table in §6 plus `getLayerStats/inferBatch/saveState/evalLoss`. Also `window.__nispsEoc` (training target). These are how E2E tests drive the app. + +--- + +## 10. Gaps / risks for a faithful clone + +**Core (must build to match the page):** +- WASM IML inference + sync/async training + `moveWeights`/`randomiseWeights`/warm-start — the whole ML surface. Non-negotiable; transplant `nisps-wasm.js` + `nisps.wasm`. +- C15 synth engine (`C15Adapter`/`C15Bridge` + `param-map.js` + `presets.js`) — runs via AudioWorklet + SharedArrayBuffer; needs COOP/COEP headers (the deployed app uses `serve-coop.py`). The 126-param map, tiered presets, group/param overrides, and the synth visualizer are all core to "synth mode." +- Flow-field visualizer (20 outputs), heatmap + override popup, joystick + noise ring, RL cluster + keyboard + undo, dock/drawers, status line, help modal, the 6-slider NISPS drawer, visual presets, persistence. All core. +- Arpeggiator + MIDI input (used by synth mode); MIDI CC output mode (`midi-output.js`, `midi-cc-map.js`, `midi-cc-presets.js`). + +**Deferrable / flagged (render scaffolds but feature can lag):** +- **Hand tracking / MediaPipe** (`hand-tracker.js`, PIP, gesture ring, 14-input `imlHand`) — heavy dep, camera permission; the help text and Mode toggle reference it but it can be a later add. Note as optional. +- **EOC Effects Chain** (`eoc/`, `eoc-chain-ui.js`, `eoc-joystick.js`, linked/independent/shared NISPS modes, second `imlEoc`, FX RL buttons, keys 3/4) — large subsystem; the dock has an "FX" icon but it's a self-contained module. Deferrable. +- **Modular engine** (512 params, `modular-engine.js`, `modular-ui.js` — 51 KB UI) and **Faust additive/FM engines** (`faust-engine-base.js`, additive/fm presets) — accessed via the engine switcher in the Synth drawer. The switcher + C15 are core; the alternative engines are deferrable (and Faust needs a running AudioContext + the Faust runtime). CLAUDE.md flags Faust engines as not-yet-wired in the new codebase. +- **Audio Canvas mode** (`audio/audio-canvas.js`, 60 KB) — a fourth output mode with dynamic output count; scaffolded but can defer. +- **ShapeSeq** (`shapeseq/`, behind `?shapeseq=1`) — sequencer; explicitly flag-gated, defer. +- **OSC bridge** (`nisps/osc-client.js`, `synth/osc-output.js`, `osc-bridge/`) — desktop/hardware bridge, not used in the default browser path; defer. +- **Gamepad** — small (`gamepad.js`), low-risk; include if cheap, else defer (it just mirrors joystick + RL). + +**Behavioural risks to watch:** +- Output-count changes trigger `confirm()` dialogs and weight warm-start — replicate the guard so mode switches don't silently wipe training. +- Param throttling (50 ms / 0.002 dead-zone) on synth param sends is load-bearing (prevents ring-buffer flooding at 126×30 fps); keep it. +- The two preset systems (visual examples vs synth override tiers) are easy to conflate — keep them as distinct stores/actions. +- Frozen/muted semantics differ between the heatmap popup (`frozen`) and the group drawer (`muted`) but map to the same underlying field via the `getParamOverride` adapter — preserve that mapping (`muted`↔`frozen`). +- Help/onboarding, `nisps-help-seen`, and the 10 s autosave are small but part of the felt experience. + +**Files that ARE the engine layer to bind to** (absolute paths): `/home/w1n5t0n/deployments/meml-aimmersive/js/nisps/nisps-wasm.js`, `/home/w1n5t0n/deployments/meml-aimmersive/js/nisps/nisps-wasm-worker.js`, `/home/w1n5t0n/deployments/meml-aimmersive/js/nisps/dataset.js`, `/home/w1n5t0n/deployments/meml-aimmersive/js/synth/c15-adapter.js`, `/home/w1n5t0n/deployments/meml-aimmersive/js/synth/c15-bridge.js`, `/home/w1n5t0n/deployments/meml-aimmersive/js/synth/param-map.js`, `/home/w1n5t0n/deployments/meml-aimmersive/js/synth/presets.js`, `/home/w1n5t0n/deployments/meml-aimmersive/js/ui/visualizer.js`. UI behaviour source of truth: `/home/w1n5t0n/deployments/meml-aimmersive/js/a-app.js`. Theme: `/home/w1n5t0n/deployments/meml-aimmersive/css/a-immersive.css`. DOM + copy: `/home/w1n5t0n/deployments/meml-aimmersive/a-immersive.html`. diff --git a/docs/redesign/backends-spec.md b/docs/redesign/backends-spec.md new file mode 100644 index 0000000..69e9556 --- /dev/null +++ b/docs/redesign/backends-spec.md @@ -0,0 +1,502 @@ +# Output Backends — Specification (`docs/redesign/backends-spec.md`) + +*Workstream E. Design-only, read-only audit 2026-06-27. The new app is Vite + React + TS in `manifold/`, wired to the parity-tested TS engine lifted from `playground/src`. British spelling in product copy. The built-in synth is the **"Powerful Synth Engine"** — the string "C15" must never reach the user.* + +> **Naming guard (non-negotiable).** The codename `C15` survives only in internal module/file names that the user never sees (`c15-adapter.js`, `c15-bridge.js`). Every label, tooltip, dock entry, menu item, status string, and aria-label says **"Powerful Synth Engine"** (or just "Synth"). A lint allowlist + a Playwright assertion (`expect(page).not.toContainText('C15')`) enforce this across `manifold/` and the VCV panel SVG/strings. + +--- + +## 0. The one idea: backends are adapters behind one interface + +Today the deployed app (`js/a-app.js`) fans output out to four ad-hoc sinks inline in `routeOutputs()` (`a-app.js:2425`): synth (`activeEngine.setParam`), MIDI CC (`midiOutput.sendBatch`), audio-canvas, and visual (`visualizer.setParams`). Each has its own throttle, dead-zone, and override handling copy-pasted. That fan-out *is* the debt this workstream removes. + +**Replace it with one `OutputBackend` interface and a registry. Exactly one backend is "active" at a time, chosen in the Console dock.** The reactive spine (per `engine-architecture.md` §2 and `findings-design-and-manifold.md` §4) ends in a single side-effect that calls `activeBackend.send(routedOutput)`. Swapping backends swaps nothing else — the input pipeline, ML, output pipeline, training loop, and verdict loop are all backend-agnostic. + +``` + gesture → input pipeline → ML (WasmIML) → output pipeline → [ activeBackend.send(routed) ] + │ + ┌───────────────┬───────────────┬───────────────┬─────────────┼──────────────┐ + WebAudioBackend ParticleBackend WebMidiBackend OscBridgeBackend CvGateBackend VcvBridgeBackend + (Powerful Synth) (flow field) (advanced CC) (paths+ranges) (1V/oct+gate) (8→model→16, LED rings) +``` + +The active backend is a property of the **output dock** (the Console's right rail / a-immersive's Mode drawer). Backends self-describe (id, label, capability) so the dock renders a picker without hard-coding the list. + +--- + +## 1. The `OutputBackend` adapter interface (TS) + +Lives at `manifold/src/engine/backends/backend.ts`. The engine never imports a concrete backend; it imports the interface + the registry. Concrete backends may import the engine's pure helpers (curves, param-map data) but **never** React. + +```ts +// manifold/src/engine/backends/backend.ts + +/** What a backend needs to know about the active mode to map outputs. */ +export interface BackendContext { + modeId: string; + outputCount: number; // model output dims actually in use (≤ 126) + paramMeta: ReadonlyArray<ParamMeta>; // name/label/min/max/curve/group per output + sampleRate?: number; // for audio backends + audioContext?: AudioContext; // lazily provided; only audio backends use it +} + +export interface ParamMeta { + id: string; // stable machine id, e.g. 'Env_A_Att' + label: string; // user-facing + min: number; // baseline range floor (normalised 0..1 maps here) + max: number; // baseline range ceil + curve: number; // 0..1, 0.5 = linear (see §3 universal mapping) + group: string; // for colour grouping (LED rings, heatmap) +} + +export type BackendId = + | 'synth' | 'particles' | 'midi' | 'osc' | 'cvgate' | 'vcv'; + +export interface BackendDescriptor { + id: BackendId; + label: string; // dock label — NEVER "C15" + description: string; + /** crossOriginIsolated / WebMIDI / WebSocket etc. availability probe. */ + isAvailable(): boolean; + /** true when this backend can ALSO feed inputs back (VCV bridge, OSC return). */ + bidirectional?: boolean; +} + +export interface OutputBackend { + readonly descriptor: BackendDescriptor; + + /** Called once when this backend becomes active. May be async (audio start, + * WS connect, MIDI access). Resolve only when ready to receive send(). */ + start(ctx: BackendContext): Promise<void>; + + /** Hot per-frame path. `routed` is the post-pipeline Float32Array (0..1), + * length = ctx.outputCount. MUST NOT allocate; MUST NOT mutate `routed`. + * Throttling/dead-zone live INSIDE each backend (rates differ per sink). */ + send(routed: Float32Array): void; + + /** Called when switching away or unmounting. Release WS/MIDI/audio/threads. */ + teardown(): Promise<void>; + + /** Optional input return path for bidirectional backends. The engine + * subscribes; values drive model inputs (e.g. VCV CV-in, OSC /nisps/input). */ + onInputs?(cb: (values: Float32Array) => void): () => void; + + /** Optional: backends that own training transport (VCV/OSC bridge) expose + * the remote verdict/example surface here. See §7. */ + remote?: RemoteTrainingBridge; +} +``` + +Registry (`manifold/src/engine/backends/registry.ts`): a `Map<BackendId, () => OutputBackend>` of lazy factories. The dock reads `descriptors` (filtered by `isAvailable()`); selecting one calls `engine.setBackend(id)`, which `teardown()`s the old and `start()`s the new with the current `BackendContext`. + +**Why `send(Float32Array)` and not per-param events:** matches the spine's single transferable-buffer effect (`engine-architecture.md` §2.1), keeps the hot path allocation-free, and lets each backend decide its own decimation. The legacy code already proves the pattern — every sink takes the full output vector and self-throttles (synth 50 ms/0.002 dead-zone `a-app.js:2425`; MIDI 50 ms/Δ1 `midi-output.js:114`; OSC 50 ms/0.002 `osc-output.js:97`). + +--- + +## 2. The backends + +### 2.1 Built-in Synth — the "Powerful Synth Engine" (`web-audio.ts` + `synth.ts`) + +Two cooperating backends, both labelled as the synth in the UI, but architecturally distinct: + +- **`WebAudioBackend`** wraps the parity-tested repo engine: `EngineHost` (`playground/src/audio/engine-host.ts`) + the worklet `nisps-processor.ts` running `_nisps_engine_process_block` (`nisps/wasm/bindings.cpp`). This is the firmware-parity audio path — the engine *is* the sound. `send()` → `EngineHost.setParams(routed)` → worklet. This is the default and the one that satisfies browser-parity chokepoint C. +- **`PowerfulSynthBackend`** (the C15 path) wraps `deployments/meml-aimmersive/js/synth/c15-adapter.js` → `c15-bridge.js` (SharedArrayBuffer ring + its own `c15_engine.wasm` worklet). Param mapping comes from `param-map.js` (`SYNTH_PARAM_MAP`, 126 entries) and `presets.js` (tiered presets). `setParam(index, normalised)` maps index→hardware id (`c15-adapter.js:121`). This is browser-only (firmware has no C15), and its 126-param surface + group/section overrides power the synth visualiser and the group-override drawer. + +**Reuse, verbatim:** `c15-adapter.js`, `c15-bridge.js`, `param-map.js`, `presets.js` move under `manifold/src/engine/backends/synth/` unchanged (internal names keep "c15"; UI strings do not). The `engine-interface.js` `SynthEngine` base maps cleanly onto `OutputBackend`: `init(ctx)`→`start`, `setParam` loop driven by `send`, `stop`→`teardown`. Curve/override math is `applyCurve`/`applyGroupOverride` (`param-map.js:287`) — fold into the universal mapping (§3). + +**Throttle (keep — load-bearing):** ≥50 ms send interval + 0.002 dead-zone per param prevents flooding the C15 ring buffer at 126×30 fps (`a-immersive.html` clone-spec §10 flags this). + +`isAvailable()`: WebAudio + (for the C15 path) `crossOriginIsolated === true` (SAB needs COOP/COEP; already server-scoped per `findings-engine-surface.md`). + +### 2.2 Particle System — faithful port of `visualizer.js` (`particles.ts`) + +A `ParticleBackend` whose `send(routed)` calls a ported `FlowFieldVisualizer.setParams(routed)`. The visual canvas runs in its own `requestAnimationFrame` loop (per `findings-design-and-manifold.md` §4.3 — rAF touches drawing only, never inference); `send()` only updates the param struct. The port MUST look and behave **exactly** as the deployed version. The full algorithm is documented in §4 so the React port is byte-faithful. + +`isAvailable()`: always (Canvas2D). This backend produces no audio — it is the "visual" output mode. + +### 2.3 MIDI out — advanced CC config (`web-midi.ts`) + +A `WebMidiBackend` wrapping the salvaged `midi-output.js` (Web MIDI API, `sendBatch`, per-CC dead-zone + 50 ms throttle, device hot-plug handling — `midi-output.js`). The **advanced CC config comes from workstream D's config model**, persisted as a CC map: per output dim → `{ name, cc (0–127), channel (1–16), min, max, curve, muted, fixedValue }`. The map shape and storage are already defined in `midi-cc-map.js` (`createCCParam`, `loadCCMap`/`saveCCMap`, well-known `CC_NAMES`, default 8-CC starter set). Lift that file verbatim into `manifold/src/engine/backends/midi/cc-map.ts`. + +`send(routed)`: for each non-muted CC param, `value = round(applyGroupOverride(routed[i], curve, min, max) * 127)`; batch the changed ones; `midiOutput.sendBatch(...)`. Storage key stays engine-scoped (`nisps-midi-cc-map:<modeId>`) for migration continuity. + +`isAvailable()`: `!!navigator.requestMIDIAccess`. + +### 2.4 OSC out — paths + ranges (`osc-bridge.ts`) + +An `OscBridgeBackend` that **salvages the existing OSC bridge** rather than reinventing it. Two pieces already exist and are good: + +- **Browser client:** `deployments/meml-aimmersive/js/synth/osc-output.js` (param-named WS messages, 50 ms/0.002 dead-zone) and the richer `js/nisps/osc-client.js` (`NispsOscClient` — `EventTarget`, `sendState`/`sendWeights`/`sendParams`, `onOutputsReceived`/`onInputsReceived`, auto-reconnect with backoff). Lift `osc-client.js` as the transport (it already speaks the bridge protocol and is bidirectional). +- **Bridge server:** `deployments/meml-aimmersive/osc-bridge/bridge.ts` — a Deno WebSocket↔UDP-OSC bridge, zero-dependency OSC encode/decode, bidirectional. Keep it as-is; it is the canonical transport between browser and any OSC target (VCV, SuperCollider). + +**OSC path + range contract (salvaged from `bridge.ts`):** + +| Direction | Address | Args | Meaning | +|---|---|---|---| +| browser→target | `/nisps/<param_name>` | `f` | one param, **post-baseline-mapping value** (see §3) | +| browser→target | `/nisps/state` | `s` | full JSON state (weights + examples + config) | +| browser→target | `/nisps/weights` | `s` | weights-only JSON | +| target→browser | `/nisps/output` | `f…f` | output vector (visualisation / monitoring) | +| target→browser | `/nisps/input` | `f…f` | input vector → drives model inputs | + +**Ranges:** OSC floats are sent in the param's mapped range by default (`applyGroupOverride` applied before send, matching `osc-output.js`), with a per-backend toggle to send **raw normalised 0..1** instead (some OSC targets want 0..1 and do their own scaling). Address prefix (`/nisps`), target host/port (default `127.0.0.1:9000`), and listen port (default `9001`) are configurable — `bridge.ts` already exposes `--osc-host/--osc-port/--osc-prefix/--ws-port/--listen-port`. + +`isAvailable()`: always (attempts WS to `ws://localhost:8765`; surfaces a "bridge not running" status if the connect fails — `osc-client.js` already reconnects with backoff). Bidirectional (`onInputs` wired to `/nisps/input`). + +### 2.5 CV / gate backend (`cvgate.ts`) + +For browser-side CV/gate there is no native hardware path, so this backend has **two transports** selectable in config: + +1. **DC-coupled WebAudio CV** (browser-native): each output dim drives a `ConstantSourceNode` (or a sample-accurate `AudioWorklet` channel) whose `offset` = mapped voltage, summed/routed to the audio interface's output channels. Gate outputs are derived from a configurable threshold on a chosen dim (value > τ → high). Pitch (1V/oct) uses a per-output "voltage role" config: `{ role: 'cv' | 'gate' | 'voct', vmin, vmax, gateThreshold }`. This is the only way to emit real CV from a browser (DC-coupled interface required; surfaced as a caveat in the UI). +2. **Bridged CV via VCV / OSC** (recommended default): reuse the OSC/VCV transport — the *VCV module's 16 CV outputs* (§5) are the real CV/gate jacks. In this mode `CvGateBackend` is a thin alias that delegates to `VcvBridgeBackend` with a "treat outputs as CV/gate" preset (per-output unipolar 0–10 V / bipolar ±5 V / 1V-oct, matching the VCV per-output range menu in `MEMLNaut.cpp:818`). + +`isAvailable()`: WebAudio path always; native-CV quality flagged as "requires DC-coupled interface". **Open choice:** whether browser-native DC CV is worth shipping vs. making CV strictly a VCV-bridge concern (recommendation: ship the VCV-bridge alias first, defer DC-coupled WebAudio CV). + +### 2.6 VCV Rack module — first-class (`vcv-bridge.ts` browser side + `vcv/` C++ side) + +The headline new backend. A first-class **VCV Rack 2 module** (`MEMLNaut`) with **8 CV inputs → model → 16 CV outputs**, an **LED ring around each of the 16 outputs**, and a **browser↔VCV bridge** so the tool is controllable AND trainable from both inside Rack and entirely from the browser. Full design in §5–§7. + +The browser-side adapter `VcvBridgeBackend` reuses `NispsOscClient` (§2.4) as transport. When active in **bridged mode**, the browser supplies inputs in real time (`/nisps/input`) and the verdict/example loop is mirrored over the bridge (`/nisps/state`, `/nisps/weights`, new `/nisps/feedback`). + +--- + +## 3. Universal per-output baseline mapping + +Every backend shares ONE baseline mapping from a normalised model output `v ∈ [0,1]` to a sink value, so behaviour is identical across sinks and the override UI (heatmap popup, group drawer) is backend-agnostic. This is the existing curve math (`param-map.js:287`), promoted to `manifold/src/engine/backends/mapping.ts`: + +```ts +// 0.5 = linear; <0.5 ease-in, >0.5 ease-out. Bit-identical to legacy + nisps/core/math.hpp. +export function applyCurve(v: number, c: number): number { + return c === 0.5 ? v : Math.pow(v, Math.pow(2, 4 * (c - 0.5))); +} + +/** Per-output baseline: curve, then scale into [min,max]; honour freeze/mute. */ +export function mapOutput(v: number, p: OutputMapping): number { + if (p.frozen) return p.fixedValue; // pinned, ignores model + return p.min + applyCurve(v, p.curve) * (p.max - p.min); +} + +export interface OutputMapping { + min: number; max: number; curve: number; // baseline range + curve + frozen: boolean; fixedValue: number; // pin + muted?: boolean; // excluded from this sink +} +``` + +Backends then apply only their **sink-specific final transform** on top of the baseline: + +| Backend | Baseline → sink transform | +|---|---| +| Powerful Synth | `mapOutput` → `setParam(i, value)` (value already in param range) | +| Particles | **raw 0..1** (the visualiser owns its own param ranges — see §4); curve/freeze still applied, min/max default to 0..1 | +| MIDI CC | `mapOutput` → `round(value * 127)` clamped 0–127 | +| OSC | `mapOutput` (or raw 0..1 if "send raw" toggled) → `/nisps/<name> <f>` | +| CV/gate | `mapOutput` → voltage by role (`cv`: `value*10` or `(value-0.5)*10`; `voct`: 1V/oct; `gate`: `value>τ ? high : 0`) | +| VCV | model output `0..1` sent raw over bridge; the **module** applies its own per-output range + attenuverter (`MEMLNaut.cpp:342 outputToVoltage`) | + +The override store (one per mode, `OutputMapping[]`) is owned by the engine and shared by all backends; freeze/mute/curve/range edits in the UI apply uniformly. Note the legacy split where the heatmap calls it `frozen` and the group drawer calls it `muted` but both map to the same field (`aimmersive-clone-spec.md` §10) — unify to the single `OutputMapping` above. + +--- + +## 4. Particle system — faithful-port plan with the documented algorithm + +Source of truth: `deployments/meml-aimmersive/js/ui/visualizer.js` (289 lines, read in full). The React port (`manifold/src/engine/backends/particles/flow-field.ts`) must reproduce this **exactly**. Below is the complete algorithm with line citations so the port is verifiable. + +### 4.1 Constants & noise source +- **Particle count: `numParticles = 400`** (`visualizer.js:53`). +- **Noise: a self-contained 2-D value/Perlin-style noise**, not a library. A `Uint8Array(512)` permutation table `PERM` is built once at module load by Fisher–Yates shuffling `[0..255]` then duplicating (`:5–14`). `fade(t)=t³(t(6t−15)+10)` (`:16`), `lerp` (`:17`), `grad(hash,x,y)` using `hash & 3` (`:19–24`), `noise2D(x,y)` doing the standard 4-corner bilinear-with-fade interpolation (`:26–44`). **The shuffle uses `Math.random()` at module load**, so the field is non-deterministic per page load — the port must keep this (or seed it; flagged as an open choice if determinism is wanted for tests). +- `TWO_PI = Math.PI*2` (`:46`); `this.time` advances `+= 0.003` per `draw()` (`:185`). + +### 4.2 Output→param mapping (20 dims, `setParams`, `:159–181`) — reproduce verbatim +Guard: returns early if `outputs.length < 20` (`:160`). Then: +``` +angleOffset = out0 * TWO_PI +scale = 0.001 + out1 * 0.009 +speed = 0.5 + out2 * 4.5 +hueBase = out3 * 360 +hueSpread = out4 * 120 +particleSize = 1 + out5 * 5 +fadeRate = 0.01 + out6 * 0.14 +turbulence = out7 * 2 +attractStrength = 0.1 + out8 * 2.9 +attractRadius = 40 + out9 * 420 +dispersionRate = 0.2 + out10 * 8 +dispersionAmount = out11 * 3 +particleLifetime = 30 + out12 * 470 +respawnStyle = out13 // 0=random,1=edge,2=center-burst +advectionMode = out14 // flow→orbit→radial blend +inertia = out15 * 0.98 +drag = out16 * 0.35 +repulsorStrength = out17 * 4.5 +repulsorCount = floor(out18 * 4.999) // 0..4 +repulsorOrbitRate = 0.1 + out19 * 2.9 +``` +The 20 output names (for labels/the heatmap colouring) are `VISUAL_PARAM_NAMES` in `a-app.js:46`: `Flow, Scale, Speed, Hue, Spread, Size, Trail, Turb, Attract, Radius, DispRate, DispAmt, Lifetime, Respawn, Advection, Inertia, Drag, Repulse, RepCnt, RepRate`. + +### 4.3 Particle lifecycle (`:104–156`) +- `makeParticle(id)`: random x,y in canvas; `age = floor(rand*particleLifetime)`; `life = computeLifetime()`; `vx=vy=0` (`:104–114`). +- `computeLifetime()`: `max(10, floor(particleLifetime * (0.65 + rand*0.7)))` (`:116–119`). +- `respawnParticle(p)`: `mode = min(2, floor(respawnStyle*2.999))` (`:122`): + - **edge (1):** spawn on a random one of 4 sides; give inward impulse of magnitude 2.0 toward centre (`:125–137`). + - **center-burst (2):** spawn within radius `min(w,h)*0.08` of centre at random angle; velocity `2.5` outward along that angle (`:138–145`). + - **random (0):** random position; velocity `(rand*2−1)*0.5` each axis (`:146–152`). + - reset `age=0`, `life=computeLifetime()`. + +### 4.4 Per-frame `draw()` (`:183–287`) — the exact integration +1. `time += 0.003`. **Trail fade:** fill whole canvas with `rgba(13,13,13, fadeRate)` (`:188`) — this is the trail length control, not a clear. +2. For each particle, with centre `(cx,cy)=(w/2,h/2)`: + - **Flow field:** `nx=p.x*scale`, `ny=p.y*scale`; `angle = noise2D(nx+time, ny)*TWO_PI + angleOffset`; `curl = noise2D(nx+100, ny+100+time*0.5)*turbulence` (`:196–199`). + - **Three advection fields** (`:202–214`): flow `(cos(angle+curl), sin(angle+curl))*speed`; orbit = perpendicular to the radial-from-centre unit vector, `*speed`; radial = radial unit vector `*speed`. + - **Blend by `advectionMode`** (`:216–225`): `modeBlend = advectionMode*2`; if `<1` lerp(flow→orbit, modeBlend) else lerp(orbit→radial, modeBlend−1). + - **Inertia + drag** (`:226–229`): `vx = vx*inertia + targetVx*(1−inertia)` (same for vy); then `vx *= (1−drag)`. + - `nextX = x+vx`, `nextY = y+vy`. + - **Central attractor** (`:233–242`): pull toward centre with `falloff = 1/(1+normDist²)` where `normDist = min(dist/attractRadius, 2)`; add `nCenter * attractStrength * falloff`. + - **Dispersion pulse** (`:244–248`): `pulse = 0.5+0.5*sin(time*dispersionRate + id*0.07)`; subtract `nCenter * dispersionAmount*pulse*falloff` (pushes outward near centre). + - **Orbiting repulsors** (`:250–264`): `repulsorRadius = min(w,h)*0.28`; for each of `repulsorCount`: phase from `time*repulsorOrbitRate + (r/4)*TWO_PI`, `wobble = 0.6+0.15*r`, position uses `cos(phase*(1+wobble))`/`sin(phase*(1.3+wobble))`; inverse-square push `force = repulsorStrength*(650/distSq)` with `distSq = d²+160`. + - **Wrap edges** (`:270–273`); `age += 1`; respawn when `age >= life` (`:275–276`). + - **Colour** (`:279–282`): `hue = (hueBase + (id/numParticles)*hueSpread) % 360`; `lightness = 50 + sin(id*0.1 + time)*15`; fill `hsl(hue, 75%, lightness%)`; `arc(x,y,particleSize)`. + +### 4.5 Port plan +- **Class, not component.** `FlowFieldVisualizer` is a plain TS class taking a `<canvas>` ref — identical to today. The React `<ParticleCanvas>` mounts it in `onMount`, drives `draw()` from one `requestAnimationFrame` loop, and calls `resize()` on the window resize handler (DPR scaling at `:84–92`). React renders the canvas element; the class owns all pixels. +- **`ParticleBackend.send(routed)`** → `viz.setParams(routed)` (no alloc; just field writes). Because the visualiser reads `outputs[0..19]`, the backend asserts `ctx.outputCount >= 20` and slices/pads to 20. +- **Faithfulness gate (Playwright):** pixel-diff a fixed seed (seed the `PERM` shuffle behind a `?seed=` for tests) at fixed param vectors against a golden capture from the deployed app; assert SSIM ≥ threshold. Also unit-test `setParams` mapping numerically (the §4.2 table). +- **Verbatim copy is allowed**: the noise + integration math is pure and has no DOM coupling beyond `ctx`/`canvas`; lift `:1–288` essentially unchanged into TS, add types, keep numeric constants exact. + +--- + +## 5. VCV Rack module design (`MEMLNaut`, 8 in / 16 out) + +There is **already a working VCV module** at `vcv/` (`MEMLNaut.cpp`, 959 lines; `plugin.json`; `SPEC.md`; `osc_server.hpp`; `Makefile`; `res/*.svg`) — but it is **2-in / 12-out + 5 derived**. The new requirement is **8-in / 16-out with LED rings**. This is an evolution of the existing module, reusing its threading model, OSC server, and serialization wholesale. + +### 5.1 What changes vs. the existing module +- `NUM_ML_INPUTS`: 2 → **8** (the existing `MAX_ML_INPUTS = 8` already anticipated this; `MEMLNaut.cpp:14`). All 8 are first-class jacks (not "reserved"). +- `NUM_ML_OUTPUTS`: 12 → **16**. The 5 derived outputs (MEAN/STD/DELTA/NOVELTY/CONFIDENCE) remain but become **optional context-menu extras** or move to the expander — the 16 raw outputs are the headline. +- MLP shape: stays within "the modular N×M envelope" — `nisps::IML<float> iml{8, 16, {16, 24, 16}}` (the existing default hidden stack is fine; sized for real-time inference per `SPEC.md`). Inputs feed model input dims; the 16 outputs are the inference outputs. +- **LED ring per output** replaces the single `SmallLight<WhiteLight>` next to each jack (`MEMLNaut.cpp:804`). + +### 5.2 Panel layout (Wide, ~32–44 HP) +``` +┌────────────────────────────────────────────┐ +│ MEMLNaut │ ← brand; "Powerful Synth" wording N/A (this is the CV mapper) +│ ┌──────────────────────────────────────┐ │ +│ │ DISPLAY: 16 bars + XY dot + metrics │ │ ← NanoVG LedDisplay/drawLayer (existing :699) +│ └──────────────────────────────────────┘ │ +│ SPREAD RATE LEARN● RAND CLEAR │ ← knobs + buttons (existing controls) +│ [+] [−] (verdict buttons) │ +│ │ +│ INPUTS (8 jacks, 2 rows × 4) │ +│ IN1 IN2 IN3 IN4 │ +│ IN5 IN6 IN7 IN8 + LEARN_GATE +TRIG −TRIG│ +│ │ +│ OUTPUTS (16, 4 rows × 4), each: │ +│ ◉jack with an LED RING around the jack │ +│ [(◯1)(◯2)(◯3)(◯4)] │ +│ [(◯5)(◯6)(◯7)(◯8)] │ +│ [(◯9)(◯10)(◯11)(◯12)] │ +│ [(◯13)(◯14)(◯15)(◯16)] │ +│ (optional) MEAN STD DELTA NOVELTY CONF │ +└────────────────────────────────────────────┘ +``` +Each output is a `PJ301MPort` jack with a **`LedRingWidget`** drawn concentric around it (no separate attenuverter trimpot in the default skin — attenuverter moves to right-click/expander to make room for the ring; keep `PARAM_ATTEN_*` in the model for range scaling). + +### 5.3 The LED-ring widget + palette mapping +A custom widget that draws a ring whose **arc fill is proportional to the output value** and whose **colour comes from the frontend design tokens**. Per the Rack manual, self-illuminating custom widgets override `drawLayer(args, 1)` and draw on layer 1 (so they stay bright when room brightness is lowered) ([VCV custom lights](https://community.vcvrack.com/t/how-to-use-custom-lights/1941), [Migrate2](https://vcvrack.com/manual/Migrate2)). + +```cpp +// vcv/src/LedRing.hpp (new) +struct LedRingWidget : Widget { + MEMLNaut* module = nullptr; + int outIdx = 0; + NVGcolor ringColor = nvgRGB(0xff, 0x6a, 0x00); // default --accent + float radius = 7.f; // mm-ish, around a PJ301M jack + void drawLayer(const DrawArgs& args, int layer) override { + if (layer != 1 || !module) return; + float v = clamp(module->slewOutputs[outIdx], 0.f, 1.f); // 0..1 + Vec c = box.size.div(2); + // track (dim full ring) + nvgBeginPath(args.vg); + nvgArc(args.vg, c.x, c.y, radius, -M_PI/2, -M_PI/2 + 2*M_PI, NVG_CW); + nvgStrokeColor(args.vg, nvgRGBA(ringColor.r*255, ringColor.g*255, ringColor.b*255, 40)); + nvgStrokeWidth(args.vg, 1.4f); nvgStroke(args.vg); + // value arc (proportional) + nvgBeginPath(args.vg); + nvgArc(args.vg, c.x, c.y, radius, -M_PI/2, -M_PI/2 + v*2*M_PI, NVG_CW); + nvgStrokeColor(args.vg, ringColor); + nvgStrokeWidth(args.vg, 1.8f); nvgStroke(args.vg); + // glow halo (matches the frontend "glow not shadow" signature) + nvgGlobalCompositeOperation(args.vg, NVG_LIGHTER); /* … bloom pass … */ + } +}; +``` + +**Palette mapping — derived from `docs/redesign/manifold-export/tokens/colors.css` (read).** Ring colours come from the design tokens so VCV matches the frontend. A `kRingPalette[16]` table assigns each output a colour by its **parameter group**, cycling through the token accents and group/pin colours: + +| Source token (colors.css) | Hex | Used for | +|---|---|---| +| `--accent` | `#ff6a00` | primary outputs / group 0 (orange — the live colour) | +| `--accent-2` | `#00ccff` | data outputs / group 1 (cyan) | +| `--accent-3` | `#ffa860` | group 2 (warm tint) | +| `--pin-3` base `#b464ff` | `#b464ff` | group 3 (violet) | +| `--good` | `#6bc26b` | group 4 (green) | +| `--warn` | `#f5c45e` | group 5 (amber) | +| `--info` | `#5b9eef` | group 6 (blue) | +| `--danger` | `#ff4466` | bipolar / perturbed outputs (red) | + +Mapping rule: `ringColor = kRingPalette[paramMeta[i].group % paletteLen]`, so outputs in the same mode-group glow the same colour, identical to the heatmap/Console grouping. Bipolar outputs (per-output range menu) tint toward `--danger`. The palette is a single header (`vcv/src/palette.hpp`) generated from `colors.css` so a token change propagates to both frontend and module (a small codegen step; **open choice** whether to automate or hand-sync). + +### 5.4 Reused, unchanged from the existing module +- **Threading model** (`MEMLNaut.cpp:67–312`): audio-thread `iml` + worker-thread `imlShadow`, job queue (Train/Perturb/Randomize/Clear), atomic weight hand-off, single-writer invariant. Already correct; just resize the I/O. +- **Inference-rate decimation + slew + post-swap crossfade** (`:447–499`). +- **Verdict loop** (`:412–445`): `+`/`−` buttons and `+TRIG`/`−TRIG` gated by LEARN. `+` → add example (current inputs→current outputs) + enqueue Train + decay noise ×0.97; `−` → bump noise (cap `0.3(1−s)+0.05s`) + enqueue Perturb. Identical semantics to the browser verdict loop. +- **Serialization** (`dataToJson`/`dataFromJson`, `:548–696`) + **`.nisps` preset save/load** (`:862–924`) + **OSC server** (`osc_server.hpp`). The `.nisps` format is the shared interchange with the browser (`vcv/NISPS-FORMAT.md`). +- Per-output / per-input **range menu** (uni/bipolar, attenuverter) (`:818–843`). + +### 5.5 Plugin scaffold files +``` +vcv/ +├── plugin.json # slug MEMLNaut (exists) — bump version; tags Controller/Utility/Random +├── Makefile # VCV SDK Makefile (exists); RACK_DIR-driven +├── Makefile.dist # dist packaging (exists) +├── src/ +│ ├── plugin.{hpp,cpp} # plugin init / model registration (exist) +│ ├── MEMLNaut.cpp # module — resize 8 in / 16 out (edit) +│ ├── LedRing.hpp # NEW — LED-ring widget (§5.3) +│ ├── palette.hpp # NEW — ring colour table from colors.css (§5.3) +│ └── osc_server.hpp # OSC server (exists; extend with /nisps/feedback §7) +├── res/ +│ ├── MEMLNaut.svg / -wide.svg / -expander.svg # panels (exist; redraw for 8/16 layout) +├── test/smoke_test.cpp # exists +├── plugin.json, SPEC.md, BUILDING.md, README.md, NISPS-FORMAT.md (exist) +``` + +--- + +## 6. Browser↔VCV bridge design + +The module must be **controllable AND trainable from BOTH inside Rack AND entirely from the browser**. Both ends operate on the same model; the bridge keeps them coherent. + +### 6.1 Transport — propose options (operator open choice) +The existing salvage path is **WebSocket↔UDP-OSC** (`osc-bridge/bridge.ts` + `osc-client.js` + `osc_server.hpp`). This is the recommended default and already works. Three options to flag: + +| Option | How | Pros | Cons | +|---|---|---|---| +| **A. WS↔OSC bridge server (recommended)** | Browser ⇄ `bridge.ts` (Deno WS server, localhost:8765) ⇄ UDP OSC ⇄ module's `osc_server.hpp` (port 9000/9001) | Already built + bidirectional; standard OSC; works with SuperCollider/Max too; no browser perms | Needs a helper process running locally (Deno or the compiled `bridge.mjs`) | +| **B. Direct WebMIDI** | Browser ⇄ Web MIDI ⇄ a virtual MIDI port ⇄ a tiny MIDI-in path in the module | No helper process if a virtual MIDI port exists; browser-native | 7-bit/14-bit only — too coarse for weights/state; really only for live CC; module would need MIDI parsing | +| **C. Native (module hosts a WS server)** | The VCV module itself opens a WebSocket/HTTP server; browser connects directly | No external bridge process | Adds a WS/TLS stack inside the plugin; COOP/COEP + mixed-content (`https://` page → `ws://localhost`) friction; more attack surface in the audio plugin | + +**Recommendation:** ship **A** (it exists, it's bidirectional, it already targets this very module — see `osc-client.js:1` "Connects the webapp to VCV Rack MEMLNaut module"). Keep **B** as a live-performance CC convenience only. Treat **C** as a future "no-helper" nicety. **This transport choice is an explicit operator open choice.** + +### 6.2 Two modes +- **Standalone:** module runs entirely inside Rack (CV in → model → CV out; verdict via panel buttons/triggers). No browser. Works today. +- **Bridged:** browser connects via the bridge. In bridged mode **the browser supplies inputs in real time** — the Manifold pointer / joystick / pads stream `/nisps/input <f…f>` to the module, which uses them instead of (or blended with, configurable) the physical CV-in jacks. The module streams `/nisps/output` and `/nisps/input` back at ~100 ms (`MEMLNaut.cpp:536`) for browser visualisation. State/weights sync both ways via `/nisps/state` + `/nisps/weights` (already staged atomically into the audio thread, `:360–384`). + +### 6.3 Coherence model +One model, one owner of weights at a time. The bridge sends **whole-model snapshots** (`/nisps/state`, `/nisps/weights`) on any structural change (train completes, randomize, clear, load preset), and **continuous I/O vectors** (`/nisps/input`, `/nisps/output`) for live feel. Last-writer-wins on weights with a short "training in progress" lock (the module already coalesces jobs at queue depth 1). The browser's `WasmIML` and the module's `nisps::IML` use the **same `.nisps` weight layout** (`vcv/NISPS-FORMAT.md`), so a snapshot from either side loads losslessly — *provided the architectures match*. **Caveat / open choice:** the browser engine is fixed `MLP<2,…,126>` while the module is `8→{16,24,16}→16`. For true weight transfer the bridged session must run a **matched architecture** (e.g. a browser mode configured to 8-in/16-out, within the modular envelope) — otherwise the bridge degrades to I/O + example transfer only (no raw-weight sync). Flag this explicitly. + +--- + +## 7. Training over the bridge — both directions + +The verdict loop (place examples / thumbs-up / thumbs-down / undo) must work from **either** end and stay coherent. Add one OSC address and a small `RemoteTrainingBridge`: + +```ts +// manifold/src/engine/backends/backend.ts +export interface RemoteTrainingBridge { + thumbsUp(): void; // add current (input,output) example + train + thumbsDown(): void; // perturb weights (explore) + addExample(input: Float32Array, output: Float32Array): void; + randomise(spread: number): void; + clear(): void; + undo(): void; + onState(cb: (s: NispsState) => void): () => void; // remote → local sync +} +``` + +New OSC verb: `/nisps/feedback <s>` carrying `{"op":"up|down|rand|clear|undo","spread":f,"input":[…],"output":[…]}`. The module's `osc_server.hpp` gets an `onFeedback` callback (mirroring the existing `onState`/`onWeights` at `MEMLNaut.cpp:120–132`) that stages the op atomically for the audio thread, which routes it through the **same** `enqueueJob`/`add_example` path the panel buttons use (`:412–445`). So: + +- **Browser → VCV training:** user clicks thumbs-up in the Manifold → `VcvBridgeBackend.remote.thumbsUp()` → `/nisps/feedback {op:up,input,output}` → module stages → worker trains `imlShadow` → atomic weight swap → module streams `/nisps/state` back → browser `WasmIML.setWeights` updates so the UI/heatmap reflect the new mapping. The browser need not run its own training in bridged mode (or runs it and pushes weights; configurable — see §6.3 caveat). +- **VCV → browser training:** user presses `+`/`−` on the panel (or sends `+TRIG`) → module trains/perturbs → streams `/nisps/state` → browser applies, so the verdict placed in Rack appears in the browser's example list and weight-health view. +- **Example placing over the bridge:** either side can `addExample`; examples ride in `/nisps/state` (the module already serializes `examples.features`/`examples.labels`, `:587–604`) so the dataset stays in sync. Undo is local-history on each side, but a remote undo can be sent as `/nisps/feedback {op:undo}` to roll the module's last job (module keeps a one-deep snapshot, matching the browser undo stack semantics). + +**Result:** the tool is fully controllable and trainable from inside VCV Rack and entirely from the browser, with the verdict loop and example-placing working either way over the same bridge. + +--- + +## 8. Build / install notes + +### Browser side +Backends live under `manifold/src/engine/backends/`; lifted JS (`c15-*`, `param-map`, `presets`, `midi-output`, `midi-cc-map`, `osc-client`, `visualizer`) ported to TS, parity-checked. No new build step — they ride the existing Vite `manifold` build. COOP/COEP stays server-scoped (needed by the Powerful Synth's SAB path). + +### OSC bridge server +```bash +cd osc-bridge +deno run --allow-net bridge.ts # or the compiled bridge.mjs +# --osc-host 127.0.0.1 --osc-port 9000 --ws-port 8765 --listen-port 9001 +``` +Ship `bridge.mjs` (already compiled) + `compile.sh` so users without Deno can run it via Node. Surface "bridge not running" in the OSC/VCV backend status (the client auto-reconnects). + +### VCV module +```bash +cd vcv +export RACK_DIR=/path/to/Rack-SDK # VCV Rack 2 SDK +make # builds plugin.so/.dylib/.dll +make install # copies into the VCV user plugins dir +# distribution: make dist (per Makefile.dist; produces the .vcvplugin) +``` +Per the search, the standard flow is `export RACK_DIR=…; make clean; make dist` ([Plugin Development Tutorial](https://vcvrack.com/manual/PluginDevelopmentTutorial)). Requires the VCV Rack 2 SDK; nisps-core is header-only C++20 (symlinked under `vcv/dep/` per `SPEC.md`). Ship v2-only (rationale in `SPEC.md`'s v1-compat section). License caveat: VCV SDK is GPLv3, nisps-core is MPL-2.0 — combined binary is effectively GPL; not submitting to the VCV Library initially (`SPEC.md` §License). + +--- + +## 9. Open choices for the operator + +1. **Bridge transport (§6.1):** confirm **A — WS↔OSC bridge server** as default (recommended; already built and bidirectional), with WebMIDI as a live-CC-only convenience and a native in-module WS server deferred. This is the biggest call. +2. **Bridged weight-sync vs I/O-only (§6.3):** the browser engine is fixed `MLP<2,…,126>`; the module is `8→16`. Either (a) run a **matched 8-in/16-out browser mode** for true raw-weight transfer, or (b) accept that bridged sessions sync **I/O + examples only** and each side trains its own weights. Recommendation: (b) for v1, (a) when the modular N×M browser MLP lands (workstream F). +3. **CV/gate native path (§2.5):** ship only the **VCV-bridge CV alias** for v1, or also build browser-native DC-coupled WebAudio CV? Recommendation: VCV-bridge first; defer DC CV. +4. **Particle noise determinism (§4.1):** keep the `Math.random()`-seeded permutation (non-deterministic per load, faithful to today) or add a `?seed=` for reproducible visuals/tests? Recommendation: keep default behaviour, add an opt-in test seed. +5. **Ring palette sync (§5.3):** auto-generate `vcv/src/palette.hpp` from `colors.css` via a codegen step, or hand-sync? Recommendation: small codegen so a token change updates both surfaces. +6. **Derived outputs on the 16-out module (§5.1):** keep MEAN/STD/DELTA/NOVELTY/CONFIDENCE as menu-toggled extras / move to the expander, or drop them? Recommendation: move to the expander; keep the 16 raw outputs + LED rings as the headline panel. +7. **Default active backend:** confirm **WebAudioBackend (firmware-parity engine)** is the default, with the Powerful Synth (C15 path), particles, MIDI, OSC, CV, VCV selectable in the dock. + +--- + +## 10. Cited source files (absolute paths) +- Particle algorithm (faithful port): `/home/w1n5t0n/deployments/meml-aimmersive/js/ui/visualizer.js` (`:5–288`) +- Built-in synth: `/home/w1n5t0n/deployments/meml-aimmersive/js/synth/c15-adapter.js`, `…/c15-bridge.js`, `…/param-map.js` (`:287` curve math), `…/presets.js` +- MIDI: `/home/w1n5t0n/deployments/meml-aimmersive/js/midi/midi-output.js` (`:114` batch throttle), `…/midi/midi-cc-map.js` +- OSC client: `/home/w1n5t0n/deployments/meml-aimmersive/js/nisps/osc-client.js`; param-named client `…/js/synth/osc-output.js` (`:97`) +- OSC bridge server: `/home/w1n5t0n/deployments/meml-aimmersive/osc-bridge/bridge.ts` (addresses `:298–305`, encode/decode `:70–182`), `bridge.mjs`, `compile.sh` +- VCV module: `/home/w1n5t0n/src/MEMLNaut-NISPS/vcv/src/MEMLNaut.cpp` (threading `:67–312`, verdict `:412–445`, ranges `:818–843`, display `:699`, LED `:804`), `…/vcv/src/osc_server.hpp`, `…/vcv/plugin.json`, `…/vcv/SPEC.md`, `…/vcv/Makefile`, `…/vcv/NISPS-FORMAT.md`, `…/vcv/res/*.svg` +- Design tokens (ring palette): `/home/w1n5t0n/src/MEMLNaut-NISPS/docs/redesign/manifold-export/tokens/colors.css` +- Spine/engine context: `/home/w1n5t0n/src/MEMLNaut-NISPS/docs/redesign/engine-architecture.md` (§2), `…/findings-design-and-manifold.md` (§4), `…/findings-engine-surface.md`, `…/aimmersive-clone-spec.md` (routeOutputs §6, §7 visual table, §10) + +Sources (VCV SDK / widgets): [VCV custom lights](https://community.vcvrack.com/t/how-to-use-custom-lights/1941), [Migrate2 (drawLayer/layer 1)](https://vcvrack.com/manual/Migrate2), [Plugin Development Tutorial (RACK_DIR/make dist)](https://vcvrack.com/manual/PluginDevelopmentTutorial), [Plugin API Guide](https://vcvrack.com/manual/PluginGuide). +--- + +## Verification corrections (adversarial pass, 2026-06-27) — verdict: minor-issues + +Strongly grounded: every cited file exists, the particle algorithm (§4) is reproduced faithfully, all 8 token +hex values verify against `colors.css`, the VCV cites + OSC bridge protocol + MIDI/OSC salvage files + C15 +adapter chain are accurate. Apply these fixes: + +- **`applyCurve` must keep the input clamp.** Real code (`param-map.js:287-291`) clamps to [0,1] BEFORE the + pow: `Math.pow(clamp01(value), exponent)`. Drop the "bit-identical" wording or add the clamp to the TS port. +- **`ParamMeta.id` vs `name` conflation** — the example `id: 'Env_A_Att'` is actually the `name` field; keep + `id` (slug) and `name` (param key) distinct when porting `param-map.js`. +- VCV module already at `vcv/` (2-in/12-out) — evolve to 8→16, do not rebuild; reuse its threading/OSC/serialisation. + +--- + +## Implementation status (2026-06-28) + +MIDI + OSC backends are BUILT in `manifold/src/backends/` (OutputBackend interface + BackendManager consuming the +spine; WebMIDI out with per-output CC#/ch/name/range; OSC-over-WS to the Deno bridge in `manifold/osc-bridge/`). +The Outputs dock panel specialises per backend (`manifold/src/dock/OutputsBackendConfig.tsx`) with named presets +(`manifold/src/backends/presets.ts`). Audio gated via `engine.audio.setMuted` on non-synth modes. + +**VCV module** — see `vcv/SPEC.md` "⚠️ BUILD DELTAS (2026-06-28)" for the authoritative build target: 8 inputs × +16 outputs, an LED ring per output (drawLayer + nvgArc), palette from the frontend tokens, WS↔OSC bridge +(browser OSC backend → `manifold/osc-bridge/` Deno relay → the module's OSC server), bidirectional training, and +the `nisps-core`→`nisps/` core-path repoint. The existing `vcv/` module (2→12) is evolved, not rebuilt. + +**Still TODO** (clear in-code): the VCV↔browser bridge browser-side wiring to the new `/nisps/feedback` verb; +the particle backend is a no-op passthrough pending the `visualizer.js` faithful port; CV/gate transport. diff --git a/docs/redesign/dock-spec.md b/docs/redesign/dock-spec.md new file mode 100644 index 0000000..8a625ee --- /dev/null +++ b/docs/redesign/dock-spec.md @@ -0,0 +1,363 @@ +# Dock Spec — Console Right-Dock Drawers + Per-Output Controls + +*Workstream D. Read-only design, 2026-06-27. Target app: `manifold/` (Vite + React + TS), wired to the parity-tested TS engine (`playground/src`) via the headless `EngineApi` boundary (`docs/redesign/engine-architecture.md` §2). Citations are `file:line`. British spelling in product copy. The built-in synth is the **"Powerful Synth Engine"** — the string "C15" MUST NEVER appear in the UI.* + +This spec replaces the placeholder Shape/Feel/Route/Health/Help drawers in `ConsoleApp.jsx` (`docs/redesign/findings-design-and-manifold.md` §2 — keyboard `1-5` map drawers, `\` toggles depth) with the real dock contents. + +--- + +## 0. The 3-depth dock model (frame the design) + +The Console's right dock is a 48px rail (`--dock-width`, `aimmersive-clone-spec.md:181`) of icons; each icon owns one drawer with **three depths**: + +| Depth | Width / placement | Purpose | Source vocabulary | +|---|---|---|---| +| **peek** | narrow flyout (~72px) hugging the rail | glanceable status + 1–2 primary toggles; no scrolling | `playground-2026.md` dock+drawer | +| **expand** | glass panel ~300px (cf. a-immersive drawer-stack `width:260px`, `aimmersive-clone-spec.md:312`) | the working surface: the per-output rows, the live sliders, the mode toggles | a-immersive drawers | +| **FULL** | near-fullscreen **modal** (`--z-modal`, glass over dimmed canvas) | the "advanced config page" for one entry — backend CC tables, full param matrix, weight-health lab | mission "advanced" page | + +- **Depth is per-drawer state** in `uiStore` (`peek | expand | full`), independent across drawers (a-immersive allows multiple open at once, `aimmersive-clone-spec.md:84`). FULL is mutually exclusive — only one modal at a time (it captures the screen). +- `\` toggles the *focused* drawer peek↔expand; a drawer's header **⤢ button** (or double-click a row) opens FULL. +- Esc closes FULL→expand; closing FULL restores prior depth. +- Drawers slide in via `drawerSlideIn` 0.2s translateX (`aimmersive-clone-spec.md:189`). FULL fades via `helpFadeIn`. +- Dock icons get `.active` tint (`--accent #ff6a00`) when their drawer is open; macOS magnify on hover (`scale(1.35)`, `aimmersive-clone-spec.md:188`). + +**Drawer roster** (six icons, top→bottom on the rail): + +| Icon | Drawer | One-line role | +|---|---|---| +| 🧠 LEARN | **Learning-Behaviour** | Feedback-mode selector, SOLO/arm chooser, live training params | +| 🎚 IN | **Inputs** | Input source + per-axis pipeline (workstream F territory — referenced, §6) | +| 🔀 OUT | **Outputs / Routing** | The per-output control matrix + backend selector (workstream E + §3, §4) | +| 🔊 SYNTH | **Powerful Synth Engine** | Engine switch + audio/arp controls + group overrides | +| ✦ VISUAL | **Particle / Visual System** | Flow-field visualiser params + presets | +| ? HELP | **Help** | Onboarding overlay (opens modal directly, not a drawer) | + +--- + +## 1. LEARNING-BEHAVIOUR drawer + +Owns *how the model learns from your gestures*: the feedback-mode selector, the SOLO/arm variant chooser, and the live training knobs. Binds to `EngineApi.feedback` (`engine-architecture.md` §2.2) and the feedback C API (`findings-engine-surface.md:52`, `nisps_ml_feedback_*`). + +### 1.1 FEEDBACK_MODE selector ("Down Action") + +The `+` (up) verdict is always "keep this" (`addExample` + train, `findings-feedback-behaviour.md:102`); the `−` (down) verdict is **selectable** among the ported feedback modes (`feedback-modes-port-spec.md` §1; `FeedbackController<MLP_T>`). The new 2-mode product surface (per the prompt's "Mode 1 / Mode 2"): + +| Selector label (UI copy) | Engine `FeedbackMode` | Behaviour | +|---|---|---| +| **Push away** (Mode 1) | `Avoid` (`feedback.hpp` enum, `findings-feedback-behaviour.md:90`) | down → geometric-dislike: perturbs the mapping away from what you disliked. In the deployed core this routes to `move_weights(speed, spread, pinMask)` (`feedback-modes-port-spec.md` §2.5 — the true k-NN centroid push is firmware-only, out of scope). | +| **Explore & place** (Mode 2) | `RandomiseMlp` (`findings-feedback-behaviour.md:122`) | down → snapshot + `draw_weights(spread)` re-rolls the whole net into a scratchpad you audition by moving the joystick; `+`/drag commits a `+1` example at the chosen input and **restores the real net** (`findings-feedback-behaviour.md:135-146`); down-again **cancels** (restore snapshot). | + +- A third engine mode `RandomiseOutputs` (bypass MLP, hold a static random vector, `findings-feedback-behaviour.md:111`) exists in the core but is **not** surfaced as a product mode in v1 — expose it only behind `?debug=1` as "Static roll". **(Open choice 1.)** +- **Peek**: a 2-segment pill (`Push away` / `Explore & place`), the current mode highlighted in `--accent`. This pill is *also* mirrored next to the Verdict cluster (`VerdictCluster`, `findings-design-and-manifold.md:50`) so it is reachable during live play without opening the drawer — matching how a-immersive puts the `rl-label` above the RL buttons (`aimmersive-clone-spec.md:283`). +- **Expand**: the pill + a one-line plain-English description of the active mode + an **"exploring…" indicator** that lights when `engine.feedback.exploring()` is true (`findings-feedback-behaviour.md:237`), reusing the `<NoiseRing>` colour ramp (off/active/high, `aimmersive-clone-spec.md:98`). While exploring in Mode 2, training is paused (`learning_paused()`, `findings-feedback-behaviour.md:48`) — show a small "learning paused" badge. +- **FULL**: the feedback lab — a diagram of the active state machine (idle → exploring → commit/cancel), the raw `FeedbackAction` log of the last presses (`findings-feedback-behaviour.md:90` enum), plus the spread/tame "Health lab" sliders (§1.3) for radical exploration tuning. +- **Bind point**: `engine.feedback.setMode(mode)` → `nisps_ml_feedback_set_mode` (`feedback-modes-port-spec.md` §4). Switching mode while exploring auto-aborts and restores the net (`set_mode` calls `abort_explore`, `findings-feedback-behaviour.md:96`). Persist `feedbackMode` in the session blob. + +### 1.2 SOLO / arm variant chooser + +"Solo" and "arm" are the SAME concept (the prompt: *solo(=arm)*): **focus training on one output**. It maps to the engine's **focus mask** (`activeDims_` → `nisps_ml_feedback_set_focus`, `findings-feedback-behaviour.md:159-167`) AND to the **output-pin mask** used by `move_weights`/`buildPinMask` (`findings-feedback-behaviour.md:117`, `mode-runtime.ts:548`). + +Two armable scopes (the chooser): + +| Variant | Effect | Engine wiring | +|---|---|---| +| **Arm output** (per-row solo) | only the armed output(s) learn/re-roll; all others are frozen in the model | `set_focus(mask)` where `mask[i]=1` for armed dims; unmuted-but-unarmed dims set `0` so `move_weights`/`roll_static_outputs` leave them put (`findings-feedback-behaviour.md:71,103-107`) | +| **Arm all** (default) | every live output learns (empty mask ⇒ all active, `findings-feedback-behaviour.md:161`) | `clear_focus_mask()` | + +- Arming is a per-output toggle (the **S** button on each control row, §3) — arming any output flips the drawer into "arm-output" scope automatically; clicking the drawer's **"Arm all"** chip clears it. +- Multiple outputs may be armed at once (mask is a vector). The armed set is highlighted on the ReadoutStrip and on the heatmap (border glow, `--glow-focus`). +- **Peek**: shows "Arm: all" or "Arm: 3 outputs"; tapping cycles to "Arm all". +- **Expand**: the scope chips + a compact list of currently-armed output names with quick-unarm. +- **Persisted** as part of routing state (the focus mask is reconstructable from per-row `armed` flags). + +### 1.3 Live training params + +Six live knobs, lifted from a-immersive's "NISPS" drawer (`aimmersive-clone-spec.md:67` — the deployed app's *actual* tuning surface, six raw sliders) plus the spread/tame regime controls: + +| Slider | Range | Default | Binds to | +|---|---|---|---| +| **Noise** | 0–noiseCap | 0.05 | `mlStore.noiseLevel`; the `−` perturbation magnitude (`findings-feedback-behaviour.md:110`) | +| **Learning rate** | 1e-6 – 1e-2 (log) | 1e-5 | `iml.learningRate` (`aimmersive-clone-spec.md:226`) | +| **Decay** | 0.8–1.0 | 0.97 | `rlExplorationDecay` (`findings-feedback-behaviour.md:103`) | +| **Spread** | 0–1 | 0.6 | `spreadLevel` — master noise/Xavier regime (`findings-feedback-behaviour.md:108`; CLAUDE.md `spread`) | +| **Tame** | 0–1 | 1 | output-range safety toward `[safeMin,safeMax]` (`param-map.js:261` `applyTame`) | +| **Max iters / Convergence** | — | — | `iml.maxIterations`/`convergenceThreshold` (`aimmersive-clone-spec.md:274`) — FULL-depth only | + +- **Peek**: Noise + Spread (the two most live). +- **Expand**: Noise, Spread, Tame, Learning rate, Decay as `<Slider>`s. +- **FULL**: all six + Max iters/Convergence + a real `<LossPlot>` (needs `nisps_ml_loss_history`, `findings-design-and-manifold.md:62`) + `<WeightHealth>` edge-glow + `<LayerStats>` + `<GradientFlow>` (the diagnostics suite, `findings-design-and-manifold.md:63`). This is the "advanced learning" page. +- All writes are eager engine setters (no React-render coupling, `engine-architecture.md` §4); persisted in the session blob. + +--- + +## 2. INPUTS drawer (workstream F territory — referenced) + +**Owner: workstream F (modular N×M inputs + input pipeline).** This drawer is specified there; here we only fix its *dock shape* so the dock model is complete and the two specs dovetail. + +- **Source**: input source selector — **Joystick / Hands / MIDI / Mic** (a-immersive Input pill, `aimmersive-clone-spec.md:150`). v1 browser is **fixed-2-input** (`findings-engine-surface.md:67`; `MLP<2,…>`) — modes needing >2 inputs show a "single-input in browser" badge (`engine-architecture.md` §6 open Q2). +- **Per-axis pipeline** (deadzone→zoom→curve→smoothing→momentum, `input/pipeline.ts`, `findings-design-and-manifold.md:84`): one `<ControlAxis>` row per input axis with its pipeline knobs. +- **Depth**: peek = source + zoom; expand = per-axis pipeline rows; FULL = the N×M input-routing matrix (workstream F). +- **Bind**: `engine.setInput(x,y)` is the only input door (`engine-architecture.md` §2.2); pipeline config lives in `input-store`. +- **Per-axis controls reuse the per-output row component** (§3) where sensible (an input axis has min/max/curve too), but mute/solo/freeze semantics differ — defer those to workstream F. **(Open choice 2: confirm input-axis rows share the §3 component or get their own.)** + +--- + +## 3. OUTPUTS / ROUTING drawer + the per-output control row + +The heart of this spec. **Owner of the routing matrix + per-output baseline is shared; backend-specific advanced layouts are workstream E (VCV/OSC/MIDI LED-ring backends, `findings-design-and-manifold.md:30`).** + +### 3.1 The per-output baseline (EVERY output, EVERY backend) + +Every output — synth param, MIDI CC, OSC path, VCV channel, visual param — shares one baseline control set. This faithfully ports the deployed vanilla-JS per-param override system (`aimmersive-clone-spec.md` §2.4 heatmap popup + §2.5 group drawer; `param-map.js` `applyGroupOverride`/`applyCurve`/`applyTame`). The baseline: + +| Control | Glyph / widget | Semantics | +|---|---|---| +| **Mute** | **M** toggle | silenced *downstream* but still computed + visible (distinct from off) | +| **Solo / Arm** | **S** toggle | focus training on this output (§1.2; focus mask + pin mask) | +| **Tri-state** | **off / fixed / live** segmented | model-control state (table §3.3) | +| **Min / Max** | `<DualRangeSlider>` (min blue `#4488ff`, max orange `#ff6a00`, `aimmersive-clone-spec.md:130`) | output range remap; `applyGroupOverride(v,curve,min,max)` (`param-map.js:301`) | +| **Curve** | `<CurvePad>` 36×36 drag canvas, 0.5=linear (`aimmersive-clone-spec.md:128`; `applyCurve`, `param-map.js:287`) | per-output response curve | +| **Value** | inline read-out / drag bar | live model value, or the held value when fixed | + +### 3.2 The control-row component + state model + +```ts +// manifold/src/engine/routing/output-control.ts (engine-side, headless) +type OutputState = 'off' | 'fixed' | 'live'; // the tri-state + +interface OutputControl { + index: number; // model output index (0..125 for synth) + name: string; // schema label (NEVER "C15"; synth params from SYNTH_PARAM_MAP) + group: string; // section (Env A, Osc B, …) for grouping + state: OutputState; // off | fixed | live + muted: boolean; // downstream silence; still computed + armed: boolean; // solo / focus-training (=arm) + min: number; // [0,1] + max: number; // [0,1], min<=max enforced + curve: number; // [0,1], 0.5 linear + fixedValue: number; // held value when state==='fixed' (captured on freeze) + // backend-specific, populated by the active backend adapter: + backend?: MidiCcSpec | OscSpec | VcvSpec; +} +``` + +- Stored in `routing-store` as a `createStore` array indexed by output (`engine-architecture.md` §1; `overrideStore` shape, `aimmersive-clone-spec.md:230`). This unifies a-immersive's split `visualOverrides`/`groupOverrides`/`engineParamOverrides`/`midiCCOverrides` (`aimmersive-clone-spec.md:230-233`) behind one `getSectionView` adapter (`aimmersive-clone-spec.md:140`). +- **Mute ↔ off are distinct fields** (the prompt insists): `muted` is a downstream gate; `state==='off'` excludes from model control. They compose (an output can be off AND muted). +- **The deployed app conflates frozen↔muted via one field** (`aimmersive-clone-spec.md:347` — heatmap popup `frozen` ↔ group drawer `muted` map to the same underlying field). The new model **separates them deliberately**: `state` carries off/fixed/live, `muted` is its own boolean, `armed` its own boolean. This is the one deliberate divergence from the deployed semantics — record in `ALIGNMENT.md`. **(Open choice 3.)** +- **Reactive binding**: the row's controls write the store eagerly; `routedOutput` memo (`engine-architecture.md` §2.1) reads `routing-store` to remap each output via `applyGroupOverride`. The single send-effect pushes to the active backend. A row never holds a second data path. + +### 3.3 Tri-state semantics table (precise) + +This matches the deployed per-param override system: `off`/`fixed` correspond to a-immersive's `frozen` (excluded/pinned, `aimmersive-clone-spec.md:131`), `live` to the default model-driven path. + +| State | Computed by model? | Pin mask (`buildPinMask`) | Sent downstream? | Value emitted | UI | +|---|---|---|---|---|---| +| **off** | **No** — removed from model control; final-layer weights pinned so RL/train never touch it (`findings-feedback-behaviour.md:117`) | pinned (excluded) | yes, at last held value | `fixedValue` (held/excluded) | dimmed row, no bar motion | +| **fixed** (freeze) | computed but result discarded; held at a static value | pinned (protect the held dim) | yes | `fixedValue` (captured on freeze, draggable, `aimmersive-clone-spec.md:131`) | hatched bar overlay (`aimmersive-clone-spec.md:190`), value slider shown | +| **live** | **Yes** — model-driven | not pinned | yes | `applyGroupOverride(modelOut, curve, min, max)` | full bar, animates | + +Orthogonal modifiers (compose with any state): + +| Modifier | Computed? | Visible? | Sent downstream? | Training focus | +|---|---|---|---|---| +| **mute** | yes (still computed) | **yes** (bar visible) | **no** (silenced) | unaffected | +| **off** (state) | no | yes (dimmed) | yes (held) | excluded | +| **solo / arm** | yes | yes (focus glow) | yes | **this output only** (focus + pin mask) | + +> Precise difference the prompt demands: **off** = the model no longer drives this output (excluded from learning, held/pinned). **fixed/freeze** = held at a static value (also pinned, but conceptually "I chose this value", with a draggable `fixedValue`). **mute** = silenced downstream but *still computed and visible* (distinct from off — you still see it move, you just don't hear it). **solo/arm** = focus training on this output. + +### 3.4 Drawer depths + +- **Peek**: backend badge (Synth / MIDI / OSC / VCV) + count of live/fixed/off/muted outputs + "Arm all" status. +- **Expand**: the **routing matrix** — a scrollable `<For>` of `<OutputControlRow>`s grouped by `group` (collapsible section headers carry a **group master curve** + **mute-group** like a-immersive's group drawer, `aimmersive-clone-spec.md:137`). Each row: name · M · S · [off|fixed|live] · dual-range · curve pad · value. 126 rows for synth — virtualise / collapse non-live by default. +- **FULL**: the **Advanced backend modal** (§4) — backend selector at the top, then the backend-specific editor over the full output set. +- **Backend selector** (workstream E): **Web Audio (synth) / Web MIDI / OSC bridge / VCV** (`engine-architecture.md` §1 backends; `aimmersive-clone-spec.md:143` output-mode tabs). Switching backend may change output count → reuse a-immersive's `confirm()` weight-reset guard (`aimmersive-clone-spec.md:144,344`). + +--- + +## 4. ADVANCED modal — backend-specific layouts (FULL depth) + +All backends share the §3.1 baseline (M/S/off-fixed-live/min/max/curve). The FULL modal adds the backend-specific fields. **(Workstream E owns the VCV/OSC/MIDI backend internals; this fixes the modal layout + state contract.)** + +### 4.1 MIDI backend + +Per-output extra fields (ported from a-immersive's MIDI-CC popup rows, `aimmersive-clone-spec.md:128`): + +```ts +interface MidiCcSpec { + cc: number; // 0..127, auto-named from CC_NAMES + channel: number; // 1..16 + value: number; // last sent, round(v*127) + // min/max/curve from baseline define the 0..127 range mapping +} +``` + +- **Top of modal**: **number of CCs** (the output count for the MIDI backend = `midiCCMap.length`, `aimmersive-clone-spec.md:144`), output-device `<select>`, MIDI preset `<select>` (`aimmersive-clone-spec.md:32`). +- **Per-CC row**: editable Name (text), **CC#** (0–127 spinner, auto-renames from `CC_NAMES`), **Ch** (1–16), plus the baseline min/max/curve/state. 7-bit value = `round(applyGroupOverride(v,curve,min,max) * 127)` (`findings-engine-surface.md:36`). Persist via `nisps-midi-cc-map:<engineId>` key (`aimmersive-clone-spec.md:239`). +- **Send**: batched per block via `WebMidiBackend.send` (`engine-architecture.md` §1; `midiOutput.sendBatch`, `aimmersive-clone-spec.md:258`). + +### 4.2 OSC backend + +```ts +interface OscSpec { + path: string; // e.g. "/synth/cutoff" + min: number; // physical range lo (engineering units, not [0,1]) + max: number; // physical range hi + // curve/state from baseline +} +``` + +- **Top of modal**: bridge connection status (WS, deferred contract — `engine-architecture.md` §1 `osc-bridge.ts`), add/remove path rows. +- **Per-output row**: **OSC path** (text), **range min/max** (typed numeric, physical units — distinct from the [0,1] baseline dual-range; the baseline min/max selects the normalised window, the OSC range maps that window to engineering units), plus baseline curve/state. +- v1: OSC bridge is **stubbed behind a locked contract** (`engine-architecture.md` §1) — the modal renders and persists config but emits only when a bridge is connected. + +### 4.3 VCV backend + +```ts +interface VcvSpec { + // per-output min/max/freeze ONLY — the simplest backend + // baseline min/max ARE the VCV range; baseline state 'fixed' IS freeze +} +``` + +- VCV adds **nothing beyond the baseline** — per-output min/max + freeze (= baseline `state==='fixed'`). The LED-ring palette derives from theme tokens (`findings-design-and-manifold.md:30`). +- **Top of modal**: per-channel min/max/freeze grid; that is the whole VCV advanced surface. + +### 4.4 Web Audio (synth) — the default + +The synth backend's advanced modal *is* the **group-override matrix** (§5 / the Powerful Synth Engine drawer's FULL depth): per-group master curve + per-param min/max/curve/mute over the 126 synth params (`aimmersive-clone-spec.md:137`). No extra per-output struct — the synth output *is* the baseline. + +--- + +## 5. POWERFUL SYNTH ENGINE drawer + +Owns the audio engine and its parameter shaping. **The string "C15" must never render** — the built-in engine is labelled **"Powerful Synth Engine"** (`findings-engine-surface.md:74`; CLAUDE.md). Engine display names come from `SynthEngine.displayName` (`aimmersive-clone-spec.md:278`). + +- **Peek**: play/pause + master volume + the active engine name ("Powerful Synth Engine") + BPM. +- **Expand**: + - **Engine switcher** — Powerful Synth Engine (default) + other engines (PAFSynth, ChannelStrip, etc. from `mode_select`); alternative engines flagged where not-yet-wired (`findings-engine-surface.md:74`). + - **Audio controls**: Vol, BPM (a-immersive play drawer, `aimmersive-clone-spec.md:31`). + - **Arpeggiator** controls (`aimmersive-clone-spec.md:278`). + - **Synth preset `<select>`** — tiered Manual/Beginner/Intermediate/Advanced/Expert (`aimmersive-clone-spec.md:199`); these set which params are active/muted + their min/max/curve (the synth-side override tier — *distinct* from the visual/RL example presets, keep separate, `aimmersive-clone-spec.md:157`). +- **FULL**: the synth **group-override matrix** (§4.4) — 18 collapsible sections (Env A…Mono, `aimmersive-clone-spec.md:298`), each with a group master curve (48×48 drag, applies relative delta to all child curves, `aimmersive-clone-spec.md:138`) and per-param baseline rows (§3.2). This is `<GroupOverrideDrawer>` promoted to FULL depth. +- **Bind**: `engine.audio.start/stop`, `SynthEngine.setParam(i, v)` throttled ≥50ms / dead-zone >0.002 (`aimmersive-clone-spec.md:257,345` — load-bearing, prevents ring-buffer flooding). Audio starts lazily on the play gesture (`aimmersive-clone-spec.md:278`). +- COOP/COEP server-scoped (`findings-design-and-manifold.md:100`); SharedArrayBuffer only for the browser-only Powerful Synth Engine path (`findings-engine-surface.md:74`). + +--- + +## 6. PARTICLE / VISUAL SYSTEM drawer + +Owns the flow-field visualiser — a faithful port of `js/ui/visualizer.js` (`findings-design-and-manifold.md:120` — exact look + behaviour; workstream E). 400-particle Canvas2D flow field driven by 20 named outputs (`aimmersive-clone-spec.md:296`). + +- **Peek**: visual on/off + the active visual preset name. +- **Expand**: the 20 visual params as per-output rows (§3 baseline — Flow/Scale/Speed/Hue/Spread/… `aimmersive-clone-spec.md:296`), each with its `VISUAL_PARAM_COLORS` swatch; plus **visual/RL preset chips** (Calm/Chaos, Rainbow, Vortex, Spiral, Embers, `aimmersive-clone-spec.md:158`) which teach the *network* by clearing the dataset and adding hardcoded input→output examples (distinct from synth tiers — keep separate). +- **FULL**: the full visual param matrix + the output→visual-param range table (verbatim ranges, `aimmersive-clone-spec.md:296`) as editable advanced mappings + particle-count / lifetime tuning. +- **Bind**: when the visual backend is active, `routedOutput` fans the 20 outputs into `FlowFieldVisualizer.setParams` (`aimmersive-clone-spec.md:260`); the canvas reads `engine.routedOutput()` in one rAF loop, never inference (`engine-architecture.md` §2.1). + +--- + +## 7. React component tree + +The Console wraps everything in `<EngineProvider>` (`engine-architecture.md` §2.2). The dock + drawers read `EngineApi` only; no drawer imports engine internals (lint seam, `engine-architecture.md` §6). + +``` +<ConsoleApp> // focus stage + dock; owns rAF for canvases +├── <Dock> // 48px right rail +│ └── <DockIcon> ×6 // LEARN, IN, OUT, SYNTH, VISUAL, HELP +│ (data-drawer; click → uiStore.setDrawer(depth)) +│ +├── <DrawerHost> // renders the open drawer at its depth +│ ├── <Drawer depth="peek|expand"> // generic glass panel; ⤢ → FULL +│ │ ├── <LearningBehaviourDrawer> +│ │ │ ├── <FeedbackModeSelector> // pill → engine.feedback.setMode (§1.1) +│ │ │ │ └── <ExploringIndicator> // <NoiseRing> ramp; learning-paused badge +│ │ │ ├── <ArmScopeChooser> // arm-all / arm-output (§1.2) +│ │ │ └── <TrainingParams> // <Slider> ×5/6 (§1.3) +│ │ ├── <InputsDrawer> // workstream F (§6); <ControlAxis> rows +│ │ ├── <RoutingDrawer> // §3 +│ │ │ ├── <BackendSelector> // Web Audio/MIDI/OSC/VCV +│ │ │ └── <For each=group> +│ │ │ └── <OutputGroupSection> // header: group master curve + mute-group +│ │ │ └── <For each=output> +│ │ │ └── <OutputControlRow> // ← the shared baseline component (§3.2) +│ │ │ ├── <MuteToggle/> M +│ │ │ ├── <ArmToggle/> S +│ │ │ ├── <TriStateSegmented/> off|fixed|live +│ │ │ ├── <DualRangeSlider/> min/max +│ │ │ ├── <CurvePad/> 36×36 +│ │ │ └── <ValueBar/> live | fixedValue +│ │ ├── <SynthEngineDrawer> // §5 ("Powerful Synth Engine" — never "C15") +│ │ │ ├── <EngineSwitcher/> +│ │ │ ├── <AudioControls/> Vol/BPM/Arp +│ │ │ └── <SynthPresetSelect/> // tiered +│ │ └── <VisualSystemDrawer> // §6 +│ │ ├── <For each=visualOutput><OutputControlRow/></For> +│ │ └── <VisualPresetChips/> +│ │ +│ └── <AdvancedModal depth="full"> // near-fullscreen, --z-modal, glass over dim +│ ├── <FeedbackLab/> // LEARN FULL (state-machine + diagnostics) +│ ├── <InputMatrix/> // IN FULL (workstream F) +│ ├── <BackendAdvanced/> // OUT FULL — switches on active backend: +│ │ ├── <MidiCcEditor/> // CC#/Ch/Name + count (§4.1) +│ │ ├── <OscPathEditor/> // path + physical range (§4.2) +│ │ ├── <VcvChannelEditor/> // min/max/freeze grid (§4.3) +│ │ └── <SynthGroupMatrix/> // 18 sections × params (§4.4 / §5 FULL) +│ └── <VisualMatrix/> // VISUAL FULL +│ +├── <VerdictCluster> // +/−/undo/reroll; mirrors <FeedbackModeSelector> pill +└── <HelpModal> // HELP icon opens directly +``` + +Shared leaf primitives live in `shared/primitives/` (`<Slider>`, `<DualRangeSlider>`, `<CurvePad>`, `<NoiseRing>`, `<LossPlot>`, `<WeightHealth>`, `<LayerStats>`, `<GradientFlow>`, `<PillToggle>`, `<ControlAxis>` — `findings-design-and-manifold.md:60`). `<OutputControlRow>` is the one new composite component this spec introduces, reused across the Routing, Synth, Visual, and (optionally, §6 open choice 2) Inputs drawers. + +--- + +## 8. Binding to the reactive spine + engine API (summary) + +| UI surface | EngineApi call | Engine route | +|---|---|---| +| Feedback-mode pill | `engine.feedback.setMode(m)` | `nisps_ml_feedback_set_mode` (`feedback-modes-port-spec.md` §4) | +| `−` verdict | `engine.feedback.thumbsDown()` | dispatch on mode → `on_down` (`findings-feedback-behaviour.md:90`) | +| `+` verdict | `engine.feedback.thumbsUp()` | `addExample` + train, or `CommitStore` when exploring | +| Arm (S) | `engine.feedback.setFocus(mask)` | `nisps_ml_feedback_set_focus` + `buildPinMask` | +| Tri-state off/fixed | `routing-store` write → `weightsRevision`/pin recompute | `routedOutput` memo + pin mask (`engine-architecture.md` §2.1) | +| min/max/curve | `routing-store` write | `routedOutput` memo `applyGroupOverride` (`param-map.js:301`) | +| mute | `routing-store.muted` | send-effect zeroes/holds downstream, value still in `mlOutput` | +| training sliders | `mlStore` setters | live engine params | +| backend select | `engine.setBackend(b)` | swaps `OutputBackend` adapter; output-count guard | +| synth setParam | (internal) send-effect throttle | `SynthEngine.setParam` ≥50ms/>0.002 | + +All writes are **eager, synchronous, off-React-render** (`engine-architecture.md` §4) — drawers mutate stores; the single `routedOutput`→send-effect carries it to the backend; canvases read accessors in rAF. No drawer creates a second data path (the live-feedback guarantee, `engine-architecture.md` §2.3). + +--- + +## 9. Open choices for the operator + +1. **Surface `RandomiseOutputs` (third feedback mode)?** Spec hides it behind `?debug=1` ("Static roll"). Confirm v1 ships only the two product modes (Push away / Explore & place) or all three. +2. **Do input-axis rows reuse `<OutputControlRow>`?** They share min/max/curve but not mute/off/live semantics. Spec defers to workstream F; confirm whether to unify the component or fork it. +3. **Separate `state`/`muted`/`armed` fields vs the deployed conflated `frozen`↔`muted` single field.** Spec deliberately splits them (cleaner tri-state); this is a divergence from the deployed override system to record in `ALIGNMENT.md`. Confirm. +4. **Default backend output counts on switch** trigger a `confirm()` weight-reset guard (ported from a-immersive). Confirm you want that friction, or prefer silent warm-start (`createWithWarmStart`, `aimmersive-clone-spec.md:250`). +5. **Group master curve relative-delta behaviour** (drag the group curve nudges all child curves preserving offsets, `aimmersive-clone-spec.md:138`) — keep this a-immersive behaviour, or switch to absolute group curve? Spec keeps relative. +6. **Where does the feedback-mode pill live for live play** — only mirrored next to the Verdict cluster, or also a permanent sub-`<StatusLine>` pill (`aimmersive-clone-spec.md:283`)? Spec puts it on the Verdict cluster. + +--- + +## Verification corrections (adversarial pass, 2026-06-27) — verdict: minor-issues + +The design reasoning, drawer model, tri-state/mute/arm separation and per-output baseline are sound; most +`aimmersive-clone-spec.md` citations verified line-by-line. Apply these fixes in the build: + +- **`engine.feedback.setMode` / `setFocus` / `exploring()` are not yet a JS surface** — only the C ABI + (`nisps_ml_feedback_set_mode` / `set_focus` / `exploring`) exists. The Manifold `EngineApi.feedback` must + ADD these JS wrappers over the C ABI in Phase 3 (this is intended new surface, not a mistake — but it must be + built, not assumed). Likewise `engine.setBackend` and synth `setParam` are internal, not on EngineApi today. +- **`param-map.js` (`applyTame/applyCurve/applyGroupOverride`) lives only in `deployments/meml-aimmersive`**, + not in `playground/src`. The curve/override math must be **ported into `manifold/`** (see backends-spec + `mapping.ts`), not imported from the deployed snapshot. +- **`--dock-width` and min-thumb `#4488ff` are deployed-a-immersive values, absent from the Manifold tokens** — + use Manifold tokens (`manifold-export/tokens/`) or add the missing ones deliberately. +- A few `findings-feedback-behaviour.md:NN` citations are out of range / point at `feedback-modes-port-spec.md` + instead (`learning_paused()` = `feedback.hpp:85` / port-spec:238; `exploring()` = `feedback.hpp:84`). +- The React `<EngineProvider>`/`useEngine` pattern is in `findings-design-and-manifold.md §4`, not + `engine-architecture.md §2.2` (that doc is SolidJS). diff --git a/docs/redesign/engine-architecture.md b/docs/redesign/engine-architecture.md new file mode 100644 index 0000000..86c7058 --- /dev/null +++ b/docs/redesign/engine-architecture.md @@ -0,0 +1,439 @@ +I now have everything I need. Key findings confirmed: + +- The **engine layer is genuinely good and parity-tested**: `WasmIML`, `EngineHost` (already uses `?worker&url` + hand-rolled imports), `nisps-processor.ts`, the pure pipelines, the stores. These are reusable. +- The **debt is concentrated** in `mode-runtime.ts` (663-line god-hook), the 9 cloned `*Mode.tsx`, `ModeShell`/`ModeSwitcher`, and `App.tsx` (toy nav). The pull-vs-push reactivity bug lives in the god-hook (output flows through imperative `setInput` → `mlStore.process()` → effects, not a clean memo chain). +- **Serving**: nginx root is `meml-aimmersive` (the legacy vanilla a-immersive at `a-immersive.html`, served untouched). The auto-deploy already builds `playground/dist` into `/home/w1n5t0n/deployments/meml.lnfinitemonkeys.org/playground/dist` but it's **currently unserved** (no location block points to it). COOP/COEP is server-scoped. The deploy clone is separate from the dev repo. +- `FeedbackController` = the RL feedback bindings (`thumbsUp`/`thumbsDown`/`moveWeights`/`drawWeights` against the 3-mode loop), not a named class. + +I have enough to write the spec. It's a foundation-architecture document, not a code change, so I'll return it directly. + +--- + +# NISPS Foundation Architecture — One Engine, Two Skins + +*Status: implementation-ready spec. Scope: the foundation only — the headless engine layer plus the two-skin shell. The detailed UX of each skin (Console drawers, interactive heatmap craft, etc.) is owned by `docs/redesign/playground-2.0-rewrite-plan.md` and not re-litigated here.* + +*Constraint from the operator: "lay solid ground that's simple and doesn't paint us into a corner." Build BOTH skins on ONE engine; don't port the engine twice; don't inherit existing tech debt.* + +--- + +## 0. The one decision that drives everything + +**Separate the engine from the skin by an explicit, headless boundary — `EngineApi` — and let both skins be pure consumers of it.** Today there is no such boundary: `mode-runtime.ts` fuses WASM lifecycle, the reactive graph, input adapters, audio host, snapshots, heatmap sampling, and per-mode component wiring into one 663-line hook that each cloned `*Mode.tsx` calls. That fusion is *the* debt. The foundation's whole job is to draw the line the god-hook erased, on the correct side: + +``` + ┌──────────────────────────── ONE ENGINE ────────────────────────────┐ + gesture │ input pipeline → ML (WASM) → output pipeline → backends │ sound + ────────► │ (the reactive spine: one memo chain + one side-effect) │ ────────► + │ exposes a headless EngineApi { inputRaw, mlOutput, routedOutput, │ + │ verdict actions, history, status } — NO JSX, NO DOM │ + └────────────────────────────────────────────────────────────────────┘ + ▲ ▲ + │ reads EngineApi │ reads EngineApi + ┌──────────┴───────────┐ ┌──────────┴───────────┐ + │ skin-a-immersive │ │ skin-2.0 (Console) │ + │ (faithful clone) │ │ (right dock+drawers) │ + │ served at / │ │ served at /2.0 │ + └──────────────────────┘ └──────────────────────┘ +``` + +The litmus test for every file: **if it imports `solid-js/web`, JSX, or touches the DOM, it is a skin; otherwise it is the engine.** The engine compiles and is testable headlessly (Node/`__nisps` probe) with zero presentation. Two skins on one engine is then trivially true: a skin is just a tree of components that read `EngineApi` accessors and call `EngineApi` actions. Neither skin can break live-feedback, because neither skin owns the spine. + +--- + +## 1. Module / directory structure + +**Build in a fresh tree: `playground2/`** (justification in §3). Inside it, the engine and the two skins are siblings; presentation never reaches into engine internals except via `EngineApi`. + +``` +playground2/ +├── index.html # skin-a-immersive entry (default, root) +├── 2.0.html # skin-2.0 entry (Console variant) +├── vite.config.ts # two inputs; base resolved per-build (§4) +├── package.json # solid-js, vite, vite-plugin-solid, @playwright/test +├── public/ +│ ├── nisps.wasm nisps.js # copied from scripts/build-wasm.sh output +│ └── c15.wasm c15.glue.js # (only if/when C15 lands; deferred) +│ +├── src/ +│ ├── engine/ # ───────── HEADLESS. No JSX. No DOM. ───────── +│ │ │ +│ │ ├── index.ts # createEngine(schema, opts): EngineApi ← THE boundary +│ │ ├── types.ts # EngineApi, EngineOptions, EngineStatus +│ │ │ +│ │ ├── spine/ +│ │ │ ├── control-graph.ts # the memo chain + the single send-effect (§2). ~1 file. +│ │ │ └── coalesce.ts # pointer-rate → display-cadence batch()+microtask gate +│ │ │ +│ │ ├── ml/ # LIFTED from playground/ (parity-tested), de-storified +│ │ │ ├── wasm-iml.ts # WasmIML class — adapt: HeapVec, reused out-buffer, no store writes +│ │ │ ├── heap-vec.ts # NEW: re-derive HEAPF32 view on every access (replaces rebind()) +│ │ │ ├── wasm-worker.ts # LIFT as-is: disposable training worker (own WASM instance) +│ │ │ └── dataset.ts # LIFT as-is +│ │ │ +│ │ ├── audio/ # LIFTED from playground/ — already correct +│ │ │ ├── engine-host.ts # LIFT ~as-is: ?worker&url, fetch+post bytes, transferables +│ │ │ └── worklet/ +│ │ │ ├── nisps-processor.ts # LIFT as-is: 2nd WASM, hand-rolled imports +│ │ │ └── audioworklet-globals.d.ts # LIFT as-is +│ │ │ +│ │ ├── pipelines/ # LIFTED — pure fns, golden-tested +│ │ │ ├── input.ts # deadzone→zoom→curve→smoothing→momentum (was input/pipeline.ts) +│ │ │ ├── output.ts # global curve→smoothing→slew→freeze (was output/pipeline.ts) +│ │ │ └── curves.ts # ONE curve catalog, golden-tested vs nisps/core/math.hpp (§3) +│ │ │ +│ │ ├── backends/ # OutputBackend adapters (plan §3.6) +│ │ │ ├── backend.ts # interface OutputBackend { send, start, teardown } +│ │ │ ├── web-audio.ts # wraps engine-host (v1) +│ │ │ ├── web-midi.ts # 7-bit CC out (v1 adapter; stub-OK at foundation) +│ │ │ └── osc-bridge.ts # WS bridge (deferred; behind locked contract) +│ │ │ +│ │ ├── stores/ # module singletons, split by update cadence (plan §3.2) +│ │ │ ├── ml-store.ts # status/arch/dataset + outputs F32 signal + weightsRevision +│ │ │ ├── input-store.ts # config store + raw-axes F32 signal {equals:false} +│ │ │ ├── output-store.ts # global gate config + reuse buffer +│ │ │ ├── control-store.ts # compound axes + per-param createMemo fanout +│ │ │ ├── routing-store.ts # control-point off/fixed/live matrix +│ │ │ ├── history-store.ts # snapshot DAG (undo/A-B/trail/snapshots) +│ │ │ ├── session-store.ts # presets, persistence, URL params, mode switch +│ │ │ └── bus.ts # LIFT as-is: typed sync pub/sub, cross-cutting events only +│ │ │ +│ │ ├── runtime/ # the decomposed god-hook (plan §3.4) — engine-side hooks +│ │ │ ├── use-input-adapters.ts # pointer/joystick/gamepad/mic → input-store.setRaw +│ │ │ ├── use-audio-lifecycle.ts # backend start/stop/teardown on mount/mode-switch +│ │ │ ├── use-snapshots.ts # DAG ops (→ history-store) +│ │ │ ├── use-heatmap-sampler.ts # input-space heatmap, throttled to weightsRevision +│ │ │ ├── use-trail.ts +│ │ │ └── use-auto-explore.ts +│ │ │ +│ │ ├── feedback/ +│ │ │ └── controller.ts # the 3-mode FeedbackController: thumbsUp/thumbsDown/randomize +│ │ │ # + undo, wrapping moveWeights/drawWeights/train + auto-snapshot +│ │ │ +│ │ ├── persist.ts # persist<T>(store, version, migrate): versioned + base64 weights +│ │ └── probe.ts # window.__nisps — reads EngineApi only; ?debug=1-gated +│ │ +│ ├── shared/ # ───── presentation shared by BOTH skins ───── +│ │ ├── primitives/ # LIFTED: Slider, JoyMap, Heatmap, XYPad, LossPlot, +│ │ │ # GradientFlow, WeightHealth, ProgressRing, ... (+ .demo.tsx) +│ │ ├── theme/ +│ │ │ └── tokens.css # orange #ff6a00, danger #ff4466, glass, JetBrains Mono; +│ │ │ # lint allowlist enforced +│ │ ├── EngineProvider.tsx # createContext(EngineApi); both skins wrap their tree +│ │ └── GenericMode.tsx # ONE schema-driven mode component (plan §3.5) +│ │ +│ ├── skin-a-immersive/ # ───── DEFAULT skin (faithful a-immersive) ───── +│ │ ├── main.tsx # mounts <EngineProvider><AImmersiveApp/></EngineProvider> +│ │ ├── AImmersiveApp.tsx +│ │ └── components/ # bottom-sheet-faithful chrome, floating RL buttons, top strip +│ │ +│ ├── skin-2.0/ # ───── Console skin (Playground 2.0) ───── +│ │ ├── main.tsx # mounts <EngineProvider><ConsoleApp/></EngineProvider> +│ │ ├── ConsoleApp.tsx # the Manifold + dock + drawers + Verdict cluster +│ │ └── components/ # Dock, Drawer (3-depth), VerdictCluster, ReadoutStrip, ... +│ │ +│ └── dev/ +│ └── PrimitivesShowcase.tsx # /dev/primitives (shared, skin-agnostic) +│ +└── tests/e2e/ # Playwright: spine invariant, both skins, parity hooks +``` + +### Why this shape is "simple and doesn't paint us into a corner" + +- **One axis of variation per directory.** Adding an *engine* touches `nisps/` + `schemas/` + `public/*.wasm`; zero skin files. Adding a *backend* touches `engine/backends/`; zero spine, zero skin. Adding/altering a *skin* touches one `skin-*/` tree; zero engine. This is the orthogonality principle made physical. +- **`EngineApi` is the only seam.** Skins import `useEngine()` (the context) and nothing from `engine/` internals. A lint rule (`no-restricted-imports`: skins may not import `engine/**` except `engine/index.ts`/`engine/types.ts`) keeps it honest. You could delete a whole skin and the engine + tests still pass. +- **`GenericMode` lives in `shared/`, not in a skin** — both skins render the same mode bodies; they differ only in *chrome* (how drawers/strips/clusters frame the canvas), not in *what a mode is*. This is what makes "two skins, one engine" cheap rather than a fork. + +--- + +## 2. The reactive spine, concretely + +This is the load-bearing structural fix and the literal definition of "doesn't paint us into a corner": there is exactly one path from gesture to sound, every consumer reads it, and a desync is a failing test rather than a recurring prod bug. + +### 2.1 The chain (in `engine/spine/control-graph.ts`) + +```ts +// One input entry point. {equals:false} so identical-reference writes still notify. +const inputRaw: Accessor<Float32Array> // input-store's raw-axes signal + +// ── pure memos ────────────────────────────────────────────────────────────── +const processedInput = createMemo(() => // deadzone→zoom→curve→smoothing→momentum + runInputPipeline(inputRaw(), inputStore.config, inputState)); // pure, golden-tested + +const mlOutput = createMemo(() => { // WasmIML.infer INTO a reused buffer + const inp = processedInput(); // (no per-frame Float32Array alloc) + iml.setInputsClamped(inp); // clamps to REAL arch (no phantom channels — §6/D) + return iml.inferInto(mlOutBuf); // returns the reused buffer; pure *read* of weights +}, mlOutBuf, { equals: false }); + +const routedOutput = createMemo(() => // voice-space + global gate + runOutputPipeline(mlOutput(), routingStore, outputStore.config, outputState)); + +// ── the ONE side-effect ────────────────────────────────────────────────────── +createEffect(() => { + const out = routedOutput(); // single dependency + backend.send(fillSendBuffer(out)); // dedicated re-filled transferable (neutering-safe) +}); // ← the ONLY postMessage / engine post in the app +``` + +Rules made structural, not aspirational: +- **Memos are pure.** No `postMessage`, no `backend.send`, no store writes inside a memo. (Lint/review rule; the plan explicitly rejects P3's "postMessage in a memo" and P1's `createComputed`.) +- **Weights mutate only through `ml-store` actions** that bump `weightsRevision`. `mlOutput`'s memo reads weights; because RL/train actions bump the revision and the memo's input or revision is a dependency, output re-derives. There is no leaked write path that can update audio without updating the UI. +- **`mlOutput` writes into `mlOutBuf` and returns it** with `{equals:false}` — fixes today's per-frame `new Float32Array(...)` alloc in `WasmIML.process()` (line 302) and the god-hook's `recomputeOutputs`. +- **Pointer-rate is coalesced to display cadence** in `coalesce.ts` via `batch()` + a microtask, *reactively* (not a rAF poll). rAF touches **canvas drawing only**, never inference. +- **The send buffer is dedicated and owned by the effect**, separate from any signal buffer, because transferring neuters the source (today's `EngineHost.setParams` transfers `params.buffer`). + +### 2.2 How BOTH skins consume it identically + +Every consumer **reads accessors**; nobody is *pushed to*. There is no "push outputs to the UI" path that can rot — the bug class is deleted, not patched. + +```ts +// shared/EngineProvider.tsx +const EngineContext = createContext<EngineApi>(); +export const useEngine = () => useContext(EngineContext)!; + +// EngineApi (engine/types.ts) — the entire skin-facing surface +interface EngineApi { + // live reactive reads (the spine) + inputRaw: Accessor<Float32Array>; + processedInput:Accessor<Float32Array>; + mlOutput: Accessor<Float32Array>; // raw model space (heatmap, diagnostics) + routedOutput: Accessor<Float32Array>; // post-pipeline (engine, readout strip, visualizer) + status: EngineStatus; // ready, audioStarted, training, examples, ... (store proxy) + weightsRevision: Accessor<number>; + layerStats: () => Float32Array; // diagnostics pull (throttled by caller) + lossHistory: Accessor<ReadonlyArray<number>>; + // actions + setInput: (x: number, y: number) => void; // the ONLY input door + feedback: FeedbackController; // thumbsUp/thumbsDown/randomize/undo/canUndo + train: () => void; + history: HistoryApi; // DAG: A/B pin/swap, jump-to-node + audio: { start(): Promise<void>; stop(): Promise<void>; setMuted(b): void }; + schema: ModeSchema; +} +``` + +- **skin-a-immersive**: a floating RL button reads `engine.feedback.thumbsUp`; the top strip reads `engine.routedOutput()`; the XY pad calls `engine.setInput(x,y)`. +- **skin-2.0**: the Verdict cluster's 👍 reads the *same* `engine.feedback.thumbsUp`; the ReadoutStrip reads the *same* `engine.routedOutput()`; the Manifold's pointer-down calls the *same* `engine.setInput(x,y)`. + +Identical bindings, different chrome. A skin cannot create a second reactive path because it has no access to `WasmIML`, `backend`, or the stores' internal setters — only to `EngineApi`. + +### 2.3 Live-feedback guarantee + the e2e assertion + +**Guarantee:** any change to `inputRaw` (gesture) *or* `weightsRevision` (RL/train) propagates — in the same synchronous tick — to `routedOutput`, which the single effect sends to the backend and which every UI consumer reads. There is no code path that updates audio without updating the readout, or vice versa. + +**The e2e invariant, asserted on every mode, in both skins, in CI** (the centerpiece; mirrors plan §3.1): + +```ts +// tests/e2e/spine.spec.ts — runs for each {mode} × {skin: '/', '/2.0'} +const before = await page.evaluate(() => window.__nisps.getOutputs()); +await page.evaluate(() => window.__nisps.setInputs([0.9, 0.1])); // synchronous probe +const after = await page.evaluate(() => window.__nisps.getOutputs()); +expect(after).not.toEqual(before); // ML output changed +const eng = await page.evaluate(() => window.__nisps.getEngineParams()); +expect(eng).toChange(); // engine params changed, same input +``` + +`window.__nisps` (in `engine/probe.ts`) is exposed synchronously, bypasses Solid reactivity with `untrack`/`batch`, and **reads `EngineApi` only** — so the probe exercises the exact path both skins use. Because the probe lives in the engine layer (not a skin), the *same* spec file runs unchanged against both `index.html` and `2.0.html`; if either skin ever fails to read the spine, that skin's row goes red. + +A complementary **no-per-frame-alloc** check (heap-snapshot fuzz around a `setInputs` storm) guards the reused-buffer discipline. + +--- + +## 3. Reuse-vs-rebuild + +**Recommendation: build the foundation in a FRESH directory `playground2/`, lifting the parity-tested engine modules wholesale and discarding the skin/runtime layer.** This is strictly simpler than refactoring `playground/` in place and carries the least debt, *because the debt and the value are cleanly separable in the existing tree* — the value is the headless engine modules (already DOM-free), the debt is the god-hook + cloned modes + toy `App.tsx`. + +### Reuse table + +| Existing `playground/` module | Verdict | Action / why | +|---|---|---| +| `audio/engine-host.ts` | **Lift ~as-is** | Already correct: `?worker&url`, fetch-on-main + post bytes, transferables, lazy AudioContext. Best code in the repo. Minor: wrap behind `WebAudioBackend`. | +| `audio/worklet/nisps-processor.ts` | **Lift as-is** | Two-WASM, hand-rolled auto-discovered imports, 128-sample shuttle. Load-bearing and correct. | +| `audio/worklet/audioworklet-globals.d.ts` | **Lift as-is** | — | +| `ml/wasm-iml.ts` | **Lift + adapt** | Keep the C-API surface and lifecycle. **Adapt:** (1) replace `HeapBuffer.rebind()` (manual, error-prone via `Object.defineProperty`) with `HeapVec` re-derive-on-access; (2) `inferInto(buf)` instead of `process()` allocating; (3) **remove `mlStore.__set*` writes from the class** — the class becomes a pure WASM wrapper, the store observes it. (4) Plumb real `nisps_ml_loss_history` (replaces the `lossHistory=[loss]` fake at line 419). | +| `ml/wasm-worker.ts` | **Lift as-is** | Disposable training worker w/ own WASM instance. | +| `ml/dataset.ts` | **Lift as-is** | JS mirror of the C++ ring. | +| `input/pipeline.ts` | **Lift as-is** → `engine/pipelines/input.ts` | Pure, golden-tested, bit-equivalent to legacy. Becomes the `processedInput` memo body. | +| `output/pipeline.ts` | **Lift as-is** → `engine/pipelines/output.ts` | Pure; becomes `routedOutput` body. **Fix:** use reuse buffer. | +| `output/curves.ts` | **Lift + fix** | **Unify with the diverging `Curve` enum** in `generated/types.ts` (`exp/log/square/sqrt/sigmoid/cubic`) into ONE catalog, golden-tested against `nisps/core/math.hpp`. (Plan §4 "Pure-fn pipelines" Keep+fix.) | +| `stores/bus.ts` | **Lift as-is** | Typed sync pub/sub; cross-cutting events only. | +| `stores/ml-store.ts` | **Adapt** | Keep the signal/store split (it's already right: `outputs` F32 `{equals:false}`, store for status). **Add** `weightsRevision`. **Invert ownership:** store observes `WasmIML`, class stops writing the store. | +| `stores/input-store.ts`, `output-store.ts` | **Adapt** | Keep config-store + raw F32 signal pattern; align to spine (input-store's raw signal becomes `inputRaw`). | +| `stores/control-store.ts` | **Adapt** | Keep axis tables/presets/`interpolateAxis`. **Replace** the routing mechanism with **per-param `createMemo` fanout** (kills the `control-routing.ts` `JSON.stringify`-in-untracked-effect anti-pattern). | +| `stores/session-store.ts`, `mode-store.ts`, `exploration-store.ts` | **Salvage values, re-cut along plan §3.2** | Re-partition into `routing-store` (control-point `off/fixed/live`), `history-store` (DAG), `session-store` (persist+presets+URL+mode). Lift constants/preset data; rebuild the store shapes. | +| `stores/persistence.ts` | **Rebuild small** → `engine/persist.ts` | Replace bespoke per-store `Partial<>` merges + `Infinity↔null` slew encoding + slow `Array.from()` weight JSON with one `persist<T>(store, version, migrate)`; base64 weights. | +| `features/snapshots.ts`, `overrides.ts`, `trail.ts`, `heatmap-sampler.ts`, `weight-health.ts`, `mic-input.ts` | **Salvage logic into `engine/runtime/` hooks** | The *algorithms* are fine; the *wiring* (god-hook calling them imperatively) is the debt. Re-home as `use-snapshots`/`use-trail`/`use-heatmap-sampler` reading stores, not as side-effects of `setInput`. Snapshots fold into the DAG. | +| `features/control-routing.ts` | **Discard** | The `JSON.stringify`-inside-`untrack` effect anti-pattern; replaced by control-store memo fanout. | +| `features/session-preset.ts` | **Rebuild** | Composed-layers model (control/synth/weights/mode), per plan §3.8. | +| `primitives/*` (16 + demos) | **Lift as-is** → `shared/primitives/` | Genuinely good, skin-agnostic, already have `.demo.tsx`. The reusable presentation layer. | +| `dev/PrimitivesShowcase.tsx`, `debug/probe.ts` | **Lift + adapt** | Showcase as-is. Probe re-pointed at `EngineApi` (it currently pokes stores directly); `?debug=1`-gate it. | +| **`modes/mode-runtime.ts` (663 lines)** | **DISCARD** | The god-hook. Its responsibilities are split across `engine/spine/`, `engine/runtime/`, `engine/feedback/`. **Do not inherit.** | +| **`modes/*Mode.tsx` (9 cloned files, ~5.6k lines)** | **DISCARD** | Replaced by one `shared/GenericMode.tsx` driven by schema. The clones *are* the debt. | +| `modes/ModeShell.tsx`, `ModeSwitcher.tsx`, `SettingsDrawer.tsx`, `mode-helpers.ts` | **Discard / re-design per skin** | Chrome belongs to skins now; a capability-class switcher is rebuilt minimally. | +| `App.tsx`, `main.tsx` | **Discard** | Toy hash-router with a "home page." Replaced by two skin entries + `EngineProvider`. | +| `modes/generated/*` | **Regenerate, don't lift** | Codegen output; add `capability_class` + `tier` fields (plan §3.5/§3.9) and emit into `playground2/`. | + +**Verdict in one line:** lift the entire `engine/` + `shared/primitives/` (≈ the `ml/`, `audio/`, `input/`, `output/`, `primitives/`, `bus.ts` value — the parity-tested core) and rebuild only the ≈6.5k lines of god-hook + cloned modes + toy app that *are* the debt. + +### Fresh dir vs refactor-in-place — justification + +| | **Fresh `playground2/` (recommended)** | Refactor `playground/` in place | +|---|---|---| +| Debt inheritance | Zero — debt files never copied; engine lifted file-by-file on purpose | High — easy to leave a god-hook tendril; "delete later" rarely happens | +| "Simple ground" | Clean import graph from day 1; lint seam enforceable immediately | Mixed old/new imports during migration; seam blurry for weeks | +| Parity risk | None — engine files lifted verbatim, parity-check runs against the same `nisps.wasm` | None, but harder to prove which path a test exercised | +| Live deploy | Untouched. `playground/dist` keeps building; we wire `playground2/dist` only when ready | Risk of breaking the live (if pointed) build mid-refactor | +| Rollback | `rm -rf playground2/` | `git revert` across an entangled history | +| Cost | One-time copy of ~10 good files | Ongoing vigilance against re-entanglement | + +A fresh dir is the cheaper path to *less debt while reusing the parity-tested core* — exactly the operator's brief. `playground/` stays as the working reference (and keeps deploying) until `playground2/` reaches parity, then `playground/` is deleted and `playground2/`→`playground/` (a rename, by which point the engine seam is proven). + +> **Note for the operator (open question A in the plan):** if 2.0 is actually destined for the laptop `~/src/manifold` tree, this whole `playground2/` lands there instead — the architecture is identity-agnostic and the dir name is the only thing that changes. This blocks namespace/repo identity; see §6. + +--- + +## 4. Two-skins mechanism (build + serve) + +**Decision: two Vite entries, one build, one `dist/` — NOT a runtime route.** Two HTML entry points compiled in a single `vite build` produce `dist/index.html` (skin A) and `dist/2.0.html` (skin B), sharing all common chunks (engine, primitives, wasm). Rationale against "simple, no corners": + +- **A runtime route** (`/2.0` as a client route inside one bundle) would force both skins into one app shell, one router, and — crucially — risk a *shared mutable* engine instance and shared chrome assumptions, reintroducing exactly the cross-skin coupling we're eliminating. It also makes "delete a skin" hard. +- **Two entries** keep the skins genuinely independent (each owns its `main.tsx` + `EngineProvider` mount) while Vite's `manualChunks`/automatic splitting means the engine + primitives are **one shared chunk loaded by both** — zero engine duplication on disk or over the wire. This is the cleanest expression of "one engine, two skins." + +### Vite config + +```ts +// playground2/vite.config.ts +import { defineConfig } from 'vite'; +import solid from 'vite-plugin-solid'; +import { resolve } from 'node:path'; + +const isolation = { + 'Cross-Origin-Opener-Policy': 'same-origin', + 'Cross-Origin-Embedder-Policy': 'require-corp', +}; + +export default defineConfig(({ command }) => ({ + // Skin A is served at site root; skin B at /2.0/. We build ONCE with base '/2.0/'-aware + // asset URLs? No — simpler: build relative, let nginx alias map. See note below. + base: './', // relative asset URLs → same dist works at / and at /2.0/ + plugins: [solid()], + server: { port: 5173, headers: isolation }, // dev: COOP/COEP for SharedArrayBuffer + worklet + preview: { port: 4173, headers: isolation }, + build: { + target: 'es2022', + sourcemap: true, + rollupOptions: { + input: { + main: resolve(__dirname, 'index.html'), // → dist/index.html (skin A, default) + v2: resolve(__dirname, '2.0.html'), // → dist/2.0.html (skin B, Console) + }, + }, + }, +})); +``` + +**Why `base: './'` (relative) rather than a fixed `/2.0/` base.** The two skins share `dist/assets/*`. If skin B used absolute `base:'/2.0/'`, skin A (at `/`) would request the wrong asset prefix. Relative `base` makes **the same `dist/` mount correctly at both `/` and `/2.0/`** with no second build. The one caveat is the **AudioWorklet URL** and the **`?worker&url` chunk**: Vite emits these as module URLs resolved relative to the importing chunk, which works under a sub-path *as long as the chunk itself is loaded relatively* — relative `base` satisfies this. The worklet's WASM is fetched by absolute `/nisps.wasm`; under a sub-path we make that **origin-relative and configurable** (engine reads `import.meta.env.BASE_URL` or a runtime `<base>`), so `/2.0/` resolves `…/nisps.wasm` correctly. (Today `wasm-iml.ts`/`engine-host.ts` hardcode `/nisps.wasm` against `window.location.origin` — fine at root, must become base-aware for the sub-path. This is the single concrete code change the two-skins serving imposes.) + +Two `package.json` scripts make dev ergonomic: +```jsonc +"dev": "vite", // serves both index.html and 2.0.html +"dev:2.0": "vite --open /2.0.html", +"build": "tsc --noEmit && vite build", // emits dist/index.html + dist/2.0.html +``` + +### nginx (keeping the legacy a-immersive root untouched) + +The current root is `meml-aimmersive` serving the **legacy vanilla** `a-immersive.html`. The new SolidJS app builds to `…/meml.lnfinitemonkeys.org/playground/dist` (currently unserved). The cleanest, no-corner serving plan: **add location blocks for the new app under explicit sub-paths, leave `/` on the legacy root for now**, and flip `/` to the new skin A only when the operator signs off. + +```nginx +# inside the existing 443 server { } for meml.lnfinitemonkeys.org +# (COOP/COEP already set at server scope → inherited by all locations below) + +set $pg2 /home/w1n5t0n/deployments/meml.lnfinitemonkeys.org/playground2/dist; + +# ── Skin B: Console at /2.0 ──────────────────────────────────────────────── +location = /2.0 { return 301 /2.0/; } +location /2.0/ { + alias $pg2/; + index 2.0.html; + try_files $uri $uri/ /2.0/2.0.html; # SPA fallback to skin B's entry +} + +# ── Skin A: faithful clone, staged at /next while it's proven ────────────── +location = /next { return 301 /next/; } +location /next/ { + alias $pg2/; + index index.html; + try_files $uri $uri/ /next/index.html; # SPA fallback to skin A's entry +} + +# ── Shared assets (one chunk set for both skins) ────────────────────────── +location /assets/ { alias $pg2/assets/; } # relative-base requests resolve here +location = /nisps.wasm { alias $pg2/nisps.wasm; } +location = /nisps.js { alias $pg2/nisps.js; } + +# legacy root stays: +# root /home/w1n5t0n/deployments/meml-aimmersive; index a-immersive.html; (UNCHANGED) +``` + +**Cutover (one-line change, when signed off):** point `/` at skin A by setting the server `root $pg2;` + `index index.html;` and a top-level `try_files $uri /index.html;`. Until then `/` serves the legacy artifact untouched, `/next/` is the new faithful skin, `/2.0/` is the Console. The auto-deploy script gains one line: after `playground/`'s `npm run build`, also `cd playground2 && npm install && npm run build` (or replace once `playground/` is retired). COOP/COEP is **server-scoped already**, so every sub-path inherits cross-origin isolation — no per-location header duplication needed (the one real correctness requirement for SharedArrayBuffer + the worklet under a sub-path is satisfied for free). + +--- + +## 5. Build sequence (small, shippable, Playwright-testable) + +Each step ends green and observable. Parity checkpoints (verification chokepoints C/E) called out. This mirrors the plan's Phase 0–4 but with the **two-skin seam established early** so neither skin can diverge. + +| Step | Deliverable | Test gate | Parity | +|---|---|---|---| +| **S0 — Scaffold + seam** | `playground2/` Vite skeleton; two HTML entries; `tokens.css` + CSS-var lint allowlist; `EngineApi`/`EngineProvider` stubs; codegen re-emitting into `playground2/` with `capability_class`+`tier` and **build-fails-on-arch-mismatch**; `persist<T>` helper; `bus.ts` lifted. Two trivial skins each render "hello from skin A/B" reading a stub `EngineApi`. | `bun run typecheck` green; codegen golden test; Playwright loads `/` and `/2.0.html`, both mount. | Codegen idempotent (byte-identical regen). | +| **S1 — Spine against a stub** | `control-graph.ts` memo chain + single send-effect with a **stubbed `mlOutput`** (deterministic fn of input); `input-store` + `output-store` with lifted pure pipelines (curve enums unified, golden-tested); `coalesce.ts`. `EngineApi.setInput`/`routedOutput` real. | **Spine invariant e2e** (`setInputs→getOutputs changes`) passes against the stub, **on both skins**; no-per-frame-alloc heap fuzz. | Pipeline golden vs legacy bit-equivalence. | +| **S2 — WASM ML bridge** | Lift `wasm-iml.ts` + `wasm-worker.ts` + `dataset.ts`; add `HeapVec` re-derive; `inferInto(buf)`; `ml-store` observes the class (no class→store writes); `weightsRevision`; real `nisps_ml_loss_history` plumbed. `mlOutput` memo now calls real `WasmIML`. | Probe `infer`/`train`/`getLayerStats`; loss plot draws a *real* curve; spine invariant still green on both skins. | **Checkpoint:** main-thread inference matches native within 1e-5 (`parity-check.sh`). | +| **S3 — Audio + first real mode via GenericMode** | Lift `engine-host.ts` + worklet behind `WebAudioBackend`; base-aware WASM URL; `GenericMode.tsx` rendering **PAFSynth** (synth class). Mount `GenericMode` inside *both* skin shells (minimal chrome each). | Per-mode live-feedback e2e (engine params change on input move) on both skins; "Start audio" gesture works. | **Checkpoint:** `parity-check.sh` green; browser PAFSynth audio equivalent to firmware (chokepoint C). | +| **S4 — Skin A default, faithful** | Build out `skin-a-immersive/` to faithfully reproduce a-immersive chrome (bottom-sheet, floating RL buttons, top strip) over `GenericMode` + `EngineApi`. Feedback controller (`thumbsUp/Down/randomize/undo`) wired; snapshot DAG (`history-store`) behind undo. Serve at `/next/`. | Playwright drives the faithful loop via probe + via real DOM on `/next/`; visual-faithfulness check (key anchors, orange tokens); a-immersive feature-parity subset (chokepoint D start). | — | +| **S5 — Skin B Console at /2.0** | Build out `skin-2.0/`: Manifold + right dock + 3-depth drawers + Verdict cluster + interactive ReadoutStrip, all reading the *same* `EngineApi`. RL-undo + A/B against the DAG. Serve at `/2.0/`. | The **same** spine + feedback e2e specs run against `/2.0/` unchanged and pass; Console-specific drawer/depth e2e. | — | +| **S6 — Mode coverage + backends + persistence** | All in-scope engines via `GenericMode` (synth/sequencer/controller/visual classes; `SequencerLane`, `VisualEngine`, controller meters); capability-class switcher in both skins; `WebMidiBackend` + (stubbed) `OscBridgeBackend`; versioned persistence + base64 weights; control surface (axis memo fanout, presets, pinning). | Every mode passes live-feedback e2e in CI on both skins; tactile-constant e2e (3px/300ms) on the Console strip. | **Checkpoint:** parity per mode; full `run-all-tests.sh` green = chokepoint E. | +| **S7 — Cutover** | Point nginx `/` at skin A; delete legacy `meml-aimmersive` serving (archive the artifact); retire `playground/`, rename `playground2/`→`playground/`; update deploy script. | Post-cutover smoke e2e against `/` and `/2.0/`. | — | + +The deliberate ordering choice vs the plan: **establish `EngineApi` + both skin mounts at S0** (not after the Console is built), so from the very first stub the "two skins read one engine" property is *tested*, and no skin can quietly grow its own data path. The Console UX detail (drawers, depths) is still de-risked against `/dev/primitives` and user-tested with Dimi before S5, per the plan. + +--- + +## 6. Risks & open questions for the operator + +**Open questions that block or shape the build (need a call before S0/S2):** + +1. **Canonical tree / identity (plan §8-A).** Is the foundation `playground2/` *in this VPS `MEMLNaut-NISPS` tree* (`nisps::`), or does it belong in the laptop `~/src/manifold` tree (`manifold::`)? This blocks the dir name, namespace, repo, and codegen identity. *Recommendation:* build here (the live tree the recon ran against; rename is trivial), but confirm. + +2. **Fixed-2-input contract (plan §8-D) — the big one.** The foundation hard-codes the honest fixed `MLP<2,10,14,18,126>` contract: codegen **fails the build** on schema/arch mismatch and `GenericMode` clamps `setInput` to the real arch (killing today's phantom-channel OOB writes in `mode-runtime.ts` lines 224-230 and `wasm-iml` warn-and-ignore at lines 198-203). Consequence: **multi-input mic modes (XIASRI / sound_analysis_midi) are firmware-only in v1** and show a "single-input in browser" badge; the runtime-shaped MLP is deferred *behind a passing parity check, never bundled into this rewrite*. Confirm you accept this v1 contraction — it's a materially smaller browser surface than "all modes work in browser." + +3. **Two entries vs runtime route — confirm.** I've chosen two Vite entries / one `dist` (cleanest "one engine, two skins"). If you'd rather a single bundle with a `/2.0` client route (e.g. to share an app shell), say so now — it changes the skin-mount and engine-instance model. + +4. **Serving cutover policy.** I propose: legacy a-immersive stays at `/` untouched; new skins live at `/next/` (faithful) and `/2.0/` (Console) until you sign off, then `/`→skin A. Confirm you want the new default *staged* rather than replacing `/` immediately. (Also: do you want skin A to eventually *replace* the legacy vanilla a-immersive, or coexist?) + +5. **`capability_class` + `tier` schema fields (plan §8-B/§8-C).** The foundation's `GenericMode` and switcher depend on these new schema fields. Confirm the four classes (Synth/Controller/Sequencer/Visual) and the per-param/diagnostic `tier: 0|1|2` disclosure model are canonical, since codegen and `EngineApi.schema` bake them in from S0. + +**Risks carried into the build (mitigations baked into the architecture):** + +| Risk | Mitigation | +|---|---| +| A skin quietly grows a second data path → live-feedback rot returns | `EngineApi` is the only seam; `no-restricted-imports` lint forbids skins importing `engine/**` internals; spine e2e runs on **both** skins in CI. | +| Memo purity violated (side-effect creeps into a memo) | Single documented send-`createEffect`; lint/review rule; the chain is one tested file (`control-graph.ts`). | +| WASM URL breaks under `/2.0/` sub-path | Base-aware asset resolution (`import.meta.env.BASE_URL`); the one concrete change the sub-path imposes; covered by a load-under-sub-path e2e. | +| Shared `dist` + relative base mis-resolves worklet/`?worker&url` chunk | `base:'./'` + nginx `alias`; explicit load-both-skins e2e at S0 catches it immediately. | +| Lifting `wasm-iml` re-imports its store-coupling debt | Adapt-on-lift: strip `mlStore.__set*` from the class; store observes the class. Enforced by the headless-engine lint (engine files may not import skin/JSX). | +| Two builds in the deploy pipeline slow/again-unserved | One `dist`, one `vite build` (two entries); deploy script gains one block; `/2.0/` + `/next/` wired in nginx from the start so it isn't "built but unserved" like today's `playground/dist`. | +| Parity drift on any core touch | Every engine-touching step (S2, S3, S6) gated on `parity-check.sh` green before the skin lands. | +| Probe ships to prod | `engine/probe.ts` gated behind `?debug=1`; reads `EngineApi` only. | + +--- + +### Relevant file paths +- Authoritative UX/feature plan (owns the skins' detail): `/home/w1n5t0n/src/MEMLNaut-NISPS/docs/redesign/playground-2.0-rewrite-plan.md` +- Design-intent reference for skin-a-immersive: `/home/w1n5t0n/src/MEMLNaut-NISPS/docs/redesign/playground-2026.md` +- The god-hook to discard (the debt): `/home/w1n5t0n/src/MEMLNaut-NISPS/playground/src/modes/mode-runtime.ts` +- Engine modules to lift: `/home/w1n5t0n/src/MEMLNaut-NISPS/playground/src/audio/engine-host.ts`, `…/audio/worklet/nisps-processor.ts`, `…/ml/wasm-iml.ts`, `…/ml/wasm-worker.ts`, `…/ml/dataset.ts`, `…/input/pipeline.ts`, `…/output/pipeline.ts`, `…/output/curves.ts`, `…/stores/bus.ts`, `…/stores/ml-store.ts`, `…/primitives/*` +- Codegen TS types to extend (`capability_class`+`tier`): `/home/w1n5t0n/src/MEMLNaut-NISPS/playground/src/modes/generated/types.ts` +- Serving facts: nginx `/etc/nginx/sites-available/meml.lnfinitemonkeys.org` (server-scope COOP/COEP, root `meml-aimmersive`); deploy script `/home/w1n5t0n/.config/webhooks/meml-deploy.sh` (builds `…/meml.lnfinitemonkeys.org/playground/dist`, currently unserved); webhook def `/home/w1n5t0n/.config/webhooks/hooks.json` +- New foundation tree to create: `/home/w1n5t0n/src/MEMLNaut-NISPS/playground2/` (or the laptop `manifold` tree, pending open question 1) diff --git a/docs/redesign/feedback-modes-port-spec.md b/docs/redesign/feedback-modes-port-spec.md new file mode 100644 index 0000000..2013208 --- /dev/null +++ b/docs/redesign/feedback-modes-port-spec.md @@ -0,0 +1,547 @@ +I have complete grounding now. I'll write the implementation-ready spec. + +# Implementation Spec — "Down Action" Negative-Feedback Feature in the `nisps/` Core + +## 0. Provenance + +Every behavior below is read directly from local branch `fork-feedback` of the memllib submodule at `/home/w1n5t0n/src/MEMLNaut-NISPS-upstream/`: + +- `examples/InterfaceRL.hpp` / `.cpp` — the FEEDBACK_MODE enum, all new state, all helpers. +- `hardware/memlnaut/display/RLView.hpp` — display only (out of scope for the core; informs JS only). +- `modes/MEMLNautModeTR8S.hpp` — selector + focus-mask wiring (`activeDims_` source). + +Target core read from `/home/w1n5t0n/src/MEMLNaut-NISPS/`: `nisps/ml/{mlp,rl,init,training}.hpp`, `nisps/core/{concepts,perf,rng}.hpp`, `nisps/wasm/bindings.cpp`, plus the JS driver (`playground/src/modes/mode-runtime.ts`, `playground/src/ml/{wasm-iml.ts,types.ts}`, `playground/src/stores/exploration-store.ts`) and the test/parity harness (`tests/cpp/`). + +--- + +## 1. Exact behavior of all 3 modes (from the real source) + +The original feature replaces a single "thumbs-down" semantic with three selectable behaviors. The selector is `FEEDBACK_MODE` (`InterfaceRL.hpp`): + +```cpp +enum class FEEDBACK_MODE : uint8_t { + AVOID = 0, // down = negative reward / geometric push away (current behaviour) + RANDOMISE_OUTPUTS, // down = bypass MLP, hold static random outputs (re-roll on each down); up = keep at current input + RANDOMISE_MLP // down = snapshot+randomise the live net (re-roll via left toggle); up/drag = keep & restore; down again = cancel +}; +``` + +The exploration state, all `private` members in `InterfaceRL`: + +```cpp +FEEDBACK_MODE feedbackMode_ = FEEDBACK_MODE::AVOID; +bool explore_active_ = false; // inside a RANDOMISE_OUTPUTS static hold or RANDOMISE_MLP temp net +bool learningPaused_ = false; // optimise() suspended while exploring +std::vector<float> staticRandomOut_; // held output in RANDOMISE_OUTPUTS +MLP<float>::mlp_weights weightSnapshot_; // original net, saved in RANDOMISE_MLP +std::vector<bool> activeDims_; // focus mask (set by the mode; empty == all active) +``` + +There are exactly three event entry points, dispatched in `loopCallback` (thread context, never ISR): +- **up** = thumbs-up (MomA1) → `_feedback_up()` +- **down** = thumbs-down (MomA2) → `_feedback_down()` +- **drag-store** = joystick-click / Tog-A1 release → `_feedback_drag_store()` + +Plus a fourth implicit transition: **switching the mode** while exploring → `_abort_explore()`. + +A critical cross-cutting fact: `optimiseSometimes()` early-returns when `learningPaused_` is true, so **training is fully suspended during any active exploration** in modes 2 and 3. + +### 1.1 Mode AVOID (`FEEDBACK_MODE::AVOID`) — historical behavior, unchanged + +State machine is trivial — `explore_active_`/`learningPaused_` are never touched. + +| Event | Action | +|-------|--------| +| **up** | `_perform_like_action()` — store `+1` example at `(controlInput, action)`; reset `dislikeMultiplier_ = 1`. | +| **down** | `_perform_dislike_action()` — store/strengthen a negative example (see below). | +| **drag-store** | `storeExperience(1.f, controlInput, savedAction)` — place the frozen output as a `+1` example. | + +`_perform_dislike_action()` (`InterfaceRL.cpp:41`): scan replay memory; if a *negative* item exists within Euclidean distance `0.05` of `controlInput`, **deepen** it (`reward = max(reward - 1, -kMaxDislikeMultiplier)` where `kMaxDislikeMultiplier = 16`) instead of adding a duplicate; otherwise `storeExperience(-1.f, ...)`. Then `dislikeMultiplier_ = min(dislikeMultiplier_ * 2, 16)`. + +The actual "push away" happens later in `optimise()` (`InterfaceRL.cpp:717`), **not** at press time. The geometric-push formula (the semantic that matters for reconciliation, §2.5): + +```cpp +// meanPositiveAction = mean of the kCentroidK (=4) positive memories +// nearest to controlInput (k-NN centroid). +float pushStep = std::clamp(fabsf(avgRewardNeg), 0.25f, 1.0f) * kGeometricPushScale; // kGeometricPushScale = 0.5 +for (const auto& neg_action : tsNegative.second) { + dir[j] = neg_action[j] - meanPositiveAction[j]; // direction AWAY from positive centroid + len = ||dir||; + const float effectivePushStep = pushStep / (1.0f + len); // taper when already far + // useRandom when len <= 1e-4 (degenerate: neg sits on the centroid) + for each dim j with activeDims_[j]: + float d = useRandom ? (rand&0xFF/127.5 - 1) : (dir[j]/len); + target[j] = clamp(neg_action[j] + d * effectivePushStep, 0, 1); + // train MLP toward `target` (push the disliked action away from the liked centroid) +} +``` + +So **AVOID = "move the network's output at the disliked input geometrically away from the centroid of nearby liked outputs."** Focus-aware: dims with `activeDims_[j] == false` are left untouched (`if (!active) continue;`). + +### 1.2 Mode RANDOMISE_OUTPUTS (`FEEDBACK_MODE::RANDOMISE_OUTPUTS`) + +Bypasses the MLP entirely and holds a static, hand-rolled output vector. The MLP and replay memory are not modified until commit. + +| Event | `explore_active_ == false` (idle) | `explore_active_ == true` (exploring) | +|-------|-----------------------------------|----------------------------------------| +| **down** (1st) | `_enter_randomise_outputs()` | — | +| **down** (subsequent) | — | `_roll_static_outputs()` (re-roll) | +| **up** | `_perform_like_action()` (normal like) | `_commit_explore(controlInput, action)` (keep) | +| **drag-store** | normal `+1` store | normal `+1` store, **stays in explore** (keep re-rolling) | + +`_enter_randomise_outputs()` (`InterfaceRL.cpp:150`): +```cpp +explore_active_ = true; +learningPaused_ = true; +staticRandomOut_ = action; // seed inactive dims with current value +_roll_static_outputs(); +``` + +`_roll_static_outputs()` — **the randomisation formula** (`InterfaceRL.cpp:159`): +```cpp +if (staticRandomOut_.size() != action.size()) staticRandomOut_ = action; +for (size_t i = 0; i < staticRandomOut_.size(); i++) { + const bool active = activeDims_.empty() || (i < activeDims_.size() && activeDims_[i]); + if (active) staticRandomOut_[i] = static_cast<float>(rand() & 0xFFFF) / 65535.f; // uniform [0,1] + // inactive dims keep their seeded entry value +} +newInput = true; +``` + +Per-dim: **focused dims → fresh uniform [0,1]; unfocused dims → frozen at the value `action` had on entry.** The held vector is emitted in `generateAction()` with **no inference and no OU noise** (`InterfaceRL.cpp:998`): +```cpp +if (feedbackMode_ == FEEDBACK_MODE::RANDOMISE_OUTPUTS && explore_active_) { + mappingOutput = staticRandomOut_; // MLP bypassed +} else { /* GetOutput + OU noise + reflect-at-bounds */ } +``` + +`_commit_explore(inputPos, outputVal)` (`InterfaceRL.cpp:188`): `storeExperience(1.f, inputPos, outputVal)` then `_restore_after_explore()`. For OUTPUTS, `outputVal == action`, i.e. **the static sound as currently heard becomes a +1 example at the current input**, after which inference resumes. + +`_restore_after_explore()` (`InterfaceRL.cpp:194`): for OUTPUTS it does **not** touch weights (no snapshot exists); it only `learningPaused_ = false; explore_active_ = false; newInput = true;`. + +### 1.3 Mode RANDOMISE_MLP (`FEEDBACK_MODE::RANDOMISE_MLP`) + +Snapshots the live weights, randomises the live net into a "temp net", and lets the user audition. Commit/restore swaps the original net back. + +| Event | `explore_active_ == false` (idle) | `explore_active_ == true` (exploring) | +|-------|-----------------------------------|----------------------------------------| +| **down** (1st) | `_enter_randomise_mlp()` | — | +| **down** (subsequent) | — | `_cancel_explore()` (discard temp net, restore original) | +| **re-roll** | — | (firmware: left-toggle → `randomiseTheNetwork()`; **not a `_feedback_*` hook**) | +| **up** | `_perform_like_action()` | `_commit_explore(controlInput, action)` (keep) | +| **drag-store** | normal `+1` store | `+1` store **and** `_restore_after_explore()` (reposition-commit) | + +`_enter_randomise_mlp()` (`InterfaceRL.cpp:170`): +```cpp +explore_active_ = true; +weightSnapshot_ = synthMapping->GetWeights(); // stash the REAL net +learningPaused_ = true; +randomiseTheNetwork(); // RandomiseWeightsAndBiasesLin(-0.9,1.1, -0.9,0.3); sets newInput +``` + +`_cancel_explore()` (`InterfaceRL.cpp:179`): `SetWeights(weightSnapshot_)` (discard temp), resume learning, clear flags. + +`_commit_explore()` → `storeExperience(1.f, controlInput, action)` then `_restore_after_explore()`. For MLP, `_restore_after_explore()` **restores the original net** (`SetWeights(weightSnapshot_)`) — the just-stored example then trains the *original* net toward the auditioned output. So the temp net is never kept as-is; instead the example it produced is grafted onto the real net via training. + +The drag-store interaction (`_feedback_drag_store()`, `InterfaceRL.cpp:139`) is the only place the drag is more than a plain store: +```cpp +storeExperience(1.f, controlInput, savedAction); +_refresh_mem_counts(); +if (feedbackMode_ == FEEDBACK_MODE::RANDOMISE_MLP && explore_active_) { + _restore_after_explore(); // drag == reposition-commit +} +``` + +### 1.4 Forced teardown — `_abort_explore()` and mode switching + +`setFeedbackMode(m)` (`InterfaceRL.hpp`): `if (explore_active_) _abort_explore(); feedbackMode_ = m;`. `_abort_explore()` is `_restore_after_explore()` minus the display calls: it restores the snapshot **only in MLP mode**, resumes learning, clears flags. This guarantees switching modes mid-exploration leaves the real net intact and learning resumed. + +### 1.5 Focus-awareness summary (`activeDims_`) + +`activeDims_` is a `std::vector<bool>` set by the mode (`setActiveDims`). Empty ⇒ all dims active. It gates three places: +- AVOID geometric push: unfocused dims not pushed. +- RANDOMISE_OUTPUTS roll: unfocused dims frozen at entry value. +- (RANDOMISE_MLP has no per-dim focus — randomising weights affects all outputs; focus is only relevant via the output transform downstream.) + +In TR-8S (`MEMLNautModeTR8S.hpp`) the mask is derived from a `FocusManager<kN_Params, 12>` (one group per drum voice + FX). This is **mode/UI policy**, not core logic. + +--- + +## 2. Clean design for the new core component + +### 2.1 Where the semantics diverge between old and new (must reconcile first) + +| Concept | Old (`fork-feedback`) | New core (`nisps/ml/`) | +|---------|-----------------------|------------------------| +| MLP type | `MLP<float>` runtime-sized, heap (`std::vector`, `std::make_shared`) | `MLP<NIn,H1,H2,H3,NOut,...>` compile-time, all `std::array`, zero heap | +| Weight snapshot | `MLP<float>::mlp_weights` (nested `std::vector`) | flat `std::span<const float>` from `get_weights()` (size `weight_count()`) | +| RNG | global libc `rand()` | per-instance `nisps::Rng` (xoshiro256+), deterministic | +| Randomise net | `RandomiseWeightsAndBiasesLin(-0.9,1.1,-0.9,0.3)` (asymmetric uniform) | `draw_weights(spread)` (spread-aware Xavier↔uniform) | +| "Avoid" | geometric push toward/away centroid, deferred in `optimise()` | `move_weights(speed, spread, pin_mask)` = decay + Gaussian perturbation | +| Output buffer | `std::vector<float> staticRandomOut_` runtime-sized | fixed `std::array<float, NOut>` | +| Training pause | `learningPaused_` short-circuits `optimise()` | core has no training loop driver; pause must be a queryable flag the *caller's* train path honors | + +**Recommendation for "Avoid" in the new core:** Do **not** port the k-NN geometric centroid push. It depends on `ReplayMemory<trainStatelessRLItem>` (rewards, decay, accumulation) which is firmware-only and absent from `nisps/`. The new playground already implements AVOID as **`move_weights(cap, spread, pinMask)` + noise growth** (`mode-runtime.ts::thumbsDown`). Define **`Avoid` in the new core as "delegate to the existing `move_weights` Gaussian perturbation."** This is the documented parity contract in `rl.hpp`. The geometric-centroid behavior is a separate, richer algorithm that should be filed as a follow-up if desired — it is NOT part of this port, and the spec should say so explicitly. The FeedbackController's job for AVOID is purely to **route** to `move_weights`; it owns no AVOID-specific state. + +This keeps the controller small: it only owns state for the two RANDOMISE modes. + +### 2.2 Proposed component: `nisps::ml::FeedbackController<MLP_T>` + +A header-only class template parameterized on the concrete MLP type, living at **`/home/w1n5t0n/src/MEMLNaut-NISPS/nisps/ml/feedback.hpp`**, namespace `nisps::ml`. It does **not** own the MLP; every mutating method takes `MLP_T&`. It owns only the exploration state. No virtual dispatch, no heap, per-instance RNG (its own, seeded independently from the MLP's so the static-output stream is reproducible without perturbing inference noise). + +```cpp +// nisps/ml/feedback.hpp +#pragma once +#include <array> +#include <cstdint> +#include <span> +#include "../core/perf.hpp" +#include "../core/rng.hpp" + +namespace nisps::ml { + +enum class FeedbackMode : std::uint8_t { + Avoid = 0, // down → move_weights (Gaussian perturb). No internal state. + RandomiseOutputs= 1, // down → bypass MLP, hold static random vector; re-roll each down. + RandomiseMlp = 2, // down → snapshot+draw_weights live net; down-again cancels. +}; + +// What a down/up/drag press resolved to — the caller (JS runtime / firmware glue) +// acts on this (store example, run move_weights, refresh UI). Keeps side effects +// that touch replay memory / training OUT of the core. +enum class FeedbackAction : std::uint8_t { + None, + AvoidPerturb, // caller: run mlp.move_weights(speed, spread, pin) + grow noise + LikeStore, // caller: add +1 example at (input, output) + train + EnterExplore, // entered a RANDOMISE_* exploration (UI: show "exploring") + Reroll, // re-rolled within an exploration (UI feedback) + CommitStore, // caller: add +1 example at (input, current output), THEN explore ended + Cancel, // exploration discarded; net restored + Restore, // exploration kept (drag reposition path); net restored +}; + +template <typename MLP_T> +class FeedbackController { + public: + static constexpr std::size_t kNOut = MLP_T::kOutput; + static constexpr std::size_t kWeights = MLP_T::weight_count(); + + explicit FeedbackController(std::uint64_t seed) noexcept : rng_(seed) {} + + void set_mode(FeedbackMode m, MLP_T& mlp) noexcept { + if (explore_active_) abort_explore(mlp); // clean teardown on switch + mode_ = m; + } + FeedbackMode mode() const noexcept { return mode_; } + + bool exploring() const noexcept { return explore_active_; } + bool learning_paused() const noexcept { return learning_paused_; } + + // Focus mask: 1 byte per output; mask[i]==0 means "frozen" (unfocused). + // Empty span ⇒ all active. Copied into a fixed buffer (no heap, no dangling). + void set_focus_mask(std::span<const std::uint8_t> mask) noexcept { + focus_count_ = (mask.size() < kNOut) ? mask.size() : kNOut; + for (std::size_t i = 0; i < focus_count_; ++i) focus_[i] = mask[i]; + } + void clear_focus_mask() noexcept { focus_count_ = 0; } + + // --- press handlers. `current_out` is the live post-pipeline output the + // user is hearing (NOut floats). They return a FeedbackAction telling the + // caller what replay-memory / training side effect to perform. --- + FeedbackAction on_down(MLP_T& mlp, std::span<const float> current_out, + float speed, float spread, + std::span<const std::uint8_t> pin_mask) noexcept; + FeedbackAction on_up (MLP_T& mlp) noexcept; + FeedbackAction on_drag(MLP_T& mlp) noexcept; + + // Inference hook: returns true and fills `out` with the held static vector + // when RandomiseOutputs is bypassing the MLP; returns false otherwise + // (caller should run mlp.process() normally). + bool static_output(std::span<float> out) const noexcept { + if (!(mode_ == FeedbackMode::RandomiseOutputs && explore_active_)) return false; + for (std::size_t i = 0; i < kNOut; ++i) out[i] = static_out_[i]; + return true; + } + + void seed(std::uint64_t s) noexcept { rng_.seed(s); } + + private: + void enter_randomise_outputs(std::span<const float> seed_out) noexcept { + explore_active_ = true; learning_paused_ = true; + for (std::size_t i = 0; i < kNOut; ++i) static_out_[i] = seed_out[i]; + roll_static_outputs(); + } + void roll_static_outputs() noexcept { + for (std::size_t i = 0; i < kNOut; ++i) { + const bool active = (focus_count_ == 0u) || (i < focus_count_ && focus_[i] != 0u); + if (active) static_out_[i] = rng_.next_float_uniform(); // [0,1) — see §2.4 + } + } + void enter_randomise_mlp(MLP_T& mlp, float spread) noexcept { + explore_active_ = true; learning_paused_ = true; + auto w = mlp.get_weights(); // flat snapshot + for (std::size_t i = 0; i < kWeights; ++i) snapshot_[i] = w[i]; + mlp.draw_weights(spread); // randomise live net + } + void restore_after_explore(MLP_T& mlp) noexcept { + if (mode_ == FeedbackMode::RandomiseMlp) + mlp.set_weights(std::span<const float>(snapshot_.data(), kWeights)); + learning_paused_ = false; explore_active_ = false; + } + void cancel_explore(MLP_T& mlp) noexcept { + if (mode_ == FeedbackMode::RandomiseMlp) + mlp.set_weights(std::span<const float>(snapshot_.data(), kWeights)); + learning_paused_ = false; explore_active_ = false; + } + void abort_explore(MLP_T& mlp) noexcept { cancel_explore(mlp); } + + FeedbackMode mode_ = FeedbackMode::Avoid; + bool explore_active_ = false; + bool learning_paused_ = false; + std::array<float, kNOut> static_out_{}; + std::array<float, kWeights> snapshot_{}; // flat weight snapshot, no heap + std::array<std::uint8_t, kNOut> focus_{}; + std::size_t focus_count_ = 0; // 0 ⇒ all active + Rng rng_; +}; +} // namespace nisps::ml +``` + +The press handlers mirror the old `_feedback_up/_down/_drag_store` switch exactly: + +```cpp +template <typename M> +FeedbackAction FeedbackController<M>::on_down(M& mlp, std::span<const float> current_out, + float speed, float spread, std::span<const std::uint8_t> pin_mask) noexcept { + switch (mode_) { + case FeedbackMode::Avoid: + mlp.move_weights(speed, spread, pin_mask); // AVOID == perturb (see §2.5) + return FeedbackAction::AvoidPerturb; + case FeedbackMode::RandomiseOutputs: + if (!explore_active_) { enter_randomise_outputs(current_out); return FeedbackAction::EnterExplore; } + roll_static_outputs(); return FeedbackAction::Reroll; + case FeedbackMode::RandomiseMlp: + if (!explore_active_) { enter_randomise_mlp(mlp, spread); return FeedbackAction::EnterExplore; } + cancel_explore(mlp); return FeedbackAction::Cancel; + } + return FeedbackAction::None; +} + +template <typename M> +FeedbackAction FeedbackController<M>::on_up(M& mlp) noexcept { + if ((mode_ == FeedbackMode::RandomiseOutputs || mode_ == FeedbackMode::RandomiseMlp) && explore_active_) { + restore_after_explore(mlp); // caller stores +1 at (input, current_out) FIRST + return FeedbackAction::CommitStore; + } + return FeedbackAction::LikeStore; // AVOID up, or idle RANDOMISE up +} + +template <typename M> +FeedbackAction FeedbackController<M>::on_drag(M& mlp) noexcept { + if (mode_ == FeedbackMode::RandomiseMlp && explore_active_) { + restore_after_explore(mlp); + return FeedbackAction::Restore; // caller already stored the +1 from savedAction + } + return FeedbackAction::LikeStore; // plain drag-store +} +``` + +> Note the ordering contract: in the old `_commit_explore`, `storeExperience` runs **before** `_restore_after_explore` (so in MLP mode the example is captured from the temp net's output, then the original net is restored and trained). The new core encodes this as: `on_up` returns `CommitStore` and the **caller must add the example using the output it captured before calling `on_up`** (which is exactly the live `current_out`). Document this loudly at the call site. + +### 2.3 Why a "FeedbackController" and not an MLP/engine extension + +- **The MLP must stay a pure numeric kernel.** It already satisfies `MLEngine` (`concepts.hpp`) with no notion of "exploration", "focus", or "modes". Bolting feedback state onto `MLP` would (a) inflate `weight_count()`-sized snapshot storage onto every MLP instance even when unused, and (b) couple the parity/golden vectors to UI policy. Keep it separate. +- **The controller is the natural home for the snapshot** because the snapshot size is `MLP_T::weight_count()` — known at compile time, so a `std::array` works and the no-heap rule is honored. +- It composes: firmware modes and the WASM handle both already hold an MLP; they add one `FeedbackController` next to it. + +### 2.4 RNG reconciliation + +Old `_roll_static_outputs` uses `rand() & 0xFFFF / 65535.f` (libc, non-deterministic, global). New core uses the per-instance `Rng`. Two choices for the static-output stream: +- **Recommended:** the controller owns its **own** `Rng rng_` seeded separately. Rationale: re-rolling outputs must not advance the MLP's inference/move RNG (that would make inference noise depend on how many times the user pressed "down"), and a separate stream makes the OUTPUTS path independently reproducible in tests. `rng_.next_float_uniform()` already returns `[0,1)` — semantically equivalent to the old `[0, 65535]/65535` up to the endpoint and resolution, well within parity tolerance because this value is *generated*, not *compared against the MLP*. +- For RANDOMISE_MLP, `draw_weights(spread)` uses the **MLP's** RNG (correct — it's the net being randomised), matching how `nisps_ml_draw_weights` already advances the MLP stream. + +### 2.5 AVOID reconciliation, restated as a decision + +`move_weights` (Gaussian decay+perturb) and the old geometric centroid push are **different algorithms**. The new playground already shipped AVOID-as-`move_weights`. Therefore: +- **`FeedbackController::Avoid` routes `on_down` → `mlp.move_weights(speed, spread, pin_mask)`** and returns `AvoidPerturb`. It carries no replay memory and no centroid math. +- The richer geometric-push avoidance (with `ReplayMemory`, reward decay, k-NN centroid) is **explicitly out of scope** for this port; if wanted later it belongs in a separate replay-memory component, not the FeedbackController. Flag this in `ALIGNMENT.md` as an accepted divergence so a future session doesn't "rediscover" it as a bug. + +--- + +## 3. The C++/JS boundary — what lives where + +**Lives in C++ (`FeedbackController`, shared with firmware):** +- `mode_` enum, `explore_active_`, `learning_paused_`. +- `static_out_` buffer (`NOut` floats) + the re-roll formula + the bypass-inference hook. +- `snapshot_` flat weights + snapshot/restore/cancel via `get_weights`/`set_weights`. +- focus mask (`NOut` bytes) + the focus-gated roll. +- The on_down/up/drag state-transition logic and the `FeedbackAction` it returns. + +**Stays in the SolidJS runtime (inherently UI / orchestration):** +- *Which input/output vector* to store as the example (depends on the pipeline-processed outputs, overrides, mic features) — `mode-runtime.ts::thumbsUp/thumbsDown` already computes these. +- *When* to train, the LR, snapshot stack for undo (`autoSnapshot`), noise growth/decay (`exploration-store.ts`). +- Building the `pin_mask` (`buildPinMask` from overrides + param pins). +- Display strings, "Down Action" selector widget, toasts. (`RLView` in firmware is the analogous display — out of core scope.) +- The replay-memory / example dataset itself stays caller-side (the JS `Dataset` + the MLP's own ring buffer). + +**Max-sharing principle honored:** all the *fiddly state-machine logic* (the part with off-by-one re-roll/cancel/commit bugs) is shared C++; only data the core cannot know (pipeline outputs, pins, UI) stays in JS. The boundary is "the controller decides *what transition happened*; the caller decides *what to persist*." + +The JS runtime keeps `learning_paused()` honored by gating its `trainOnCurrent()` and auto-explore tick: when paused, skip training (mirrors `optimiseSometimes()`'s early return). + +--- + +## 4. WASM C API additions + +Add to `nisps/wasm/bindings.cpp`. The `MLHandle` gains a `FeedbackController` next to its `DefaultMLP` (`FeedbackController<DefaultMLP>`), seeded off the same JS seed XOR a salt so its stream is independent of the MLP's: + +```cpp +// inside struct MLHandle: +nisps::ml::FeedbackController<DefaultMLP> feedback; +std::array<float, kDefaultOutputs> static_out_scratch{}; // for nisps_ml_feedback_static_output +explicit MLHandle(std::uint64_t seed) noexcept + : mlp(seed), feedback(seed ^ 0xFEEDBACC0DEull) {} +``` + +New `extern "C"` functions (all `EMSCRIPTEN_KEEPALIVE`): + +```cpp +// Mode + state. +void nisps_ml_feedback_set_mode(void* ml, int mode); // 0=Avoid 1=RandOut 2=RandMlp +int nisps_ml_feedback_get_mode(void* ml); +int nisps_ml_feedback_exploring(void* ml); // 0/1 +int nisps_ml_feedback_learning_paused(void* ml); // 0/1 + +// Focus mask (NOut bytes; pass nullptr/0 to clear). +void nisps_ml_feedback_set_focus(void* ml, const uint8_t* mask, int n); + +// Press handlers. `current_out` = NOut floats the user is hearing. +// `pin_mask` may be null. Return value is the FeedbackAction enum (int) so JS +// knows what to persist (store example / perturb / commit / cancel). +int nisps_ml_feedback_down(void* ml, const float* current_out, + float speed, float spread, const uint8_t* pin_mask); +int nisps_ml_feedback_up(void* ml); +int nisps_ml_feedback_drag(void* ml); + +// Inference hook: if returns 1, `out` (NOut floats) holds the static bypass +// vector and the caller should NOT call nisps_ml_process(); if 0, run process(). +int nisps_ml_feedback_static_output(void* ml, float* out); +``` + +Representative implementation (matches the existing guard/cast style in `bindings.cpp`): + +```cpp +EMSCRIPTEN_KEEPALIVE +int nisps_ml_feedback_down(void* ml, const float* current_out, + float speed, float spread, const uint8_t* pin_mask) { + if (!ml) return 0; + auto* h = static_cast<MLHandle*>(ml); + std::span<const float> out = current_out + ? std::span<const float>(current_out, kDefaultOutputs) : std::span<const float>{}; + std::span<const std::uint8_t> mask = pin_mask + ? std::span<const std::uint8_t>(pin_mask, kDefaultOutputs) : std::span<const std::uint8_t>{}; + return static_cast<int>(h->feedback.on_down(h->mlp, out, speed, spread, mask)); +} +``` + +**TS FFI declarations** to add to `playground/src/ml/types.ts` (`NispsModule` interface, mirroring the existing `_nisps_ml_*` block): + +```ts +// Feedback / Down-Action. +_nisps_ml_feedback_set_mode(ml: number, mode: number): void; +_nisps_ml_feedback_get_mode(ml: number): number; +_nisps_ml_feedback_exploring(ml: number): number; +_nisps_ml_feedback_learning_paused(ml: number): number; +_nisps_ml_feedback_set_focus(ml: number, mask_ptr: number, n: number): void; +_nisps_ml_feedback_down(ml: number, current_out_ptr: number, speed: number, spread: number, pin_mask_ptr: number): number; +_nisps_ml_feedback_up(ml: number): number; +_nisps_ml_feedback_drag(ml: number): number; +_nisps_ml_feedback_static_output(ml: number, out_ptr: number): number; +``` + +Plus a `FeedbackAction` / `FeedbackMode` TS enum in `types.ts` mirroring the C++ enums (numeric values must match). `WasmIML` (`wasm-iml.ts`) gains thin wrappers (`feedbackDown(currentOut, speed, spread, pinMask)`, etc.) that reuse the existing `pinMaskBuf` and add a small `static_out` heap buffer; the parity `.mjs` shim adds `cwrap` bindings for the same names. + +--- + +## 5. Test plan + +### 5.1 Host ctest — new file `tests/cpp/test_mlp_feedback.cpp` + +Register it in the `nisps_core_tests` executable in `nisps/CMakeLists.txt` (it links `nisps_core`, builds under `-Wall -Wextra -Werror -Wpedantic`). Use the existing harness (`NISPS_TEST` / `NISPS_EXPECT` / `NISPS_EXPECT_NEAR`) and a small MLP like the existing RL test (`MLP<2,4,4,4,6,8,32>`). Because the controller takes `MLP_T&`, instantiate `FeedbackController<SmallMLP>`. + +Cases (one `NISPS_TEST` each): + +1. **`feedback_avoid_routes_to_move_weights`** — mode Avoid; snapshot weights; `on_down(...)` returns `AvoidPerturb` and weights changed (most distinct), mirroring `mlp_move_weights_changes_unpinned_weights`. `exploring()` stays false; `learning_paused()` stays false. +2. **`feedback_randout_enter_roll_state`** — mode RandomiseOutputs; `on_down` (1st) → `EnterExplore`, `exploring()==true`, `learning_paused()==true`; `static_output(buf)` returns true and buf differs from `current_out` on focused dims. +3. **`feedback_randout_reroll_changes_focused_only`** — set focus mask `{1,0,1,0,1,0}`; enter; capture static; `on_down` again → `Reroll`; assert unfocused dims **unchanged** (frozen at seed value), focused dims **changed**. +4. **`feedback_randout_commit_clears_state`** — while exploring, `on_up` → `CommitStore`, then `exploring()==false`, `learning_paused()==false`, `static_output()` returns false (MLP no longer bypassed). Weights unchanged (OUTPUTS never touches the net). +5. **`feedback_randmlp_snapshot_and_cancel_restores`** — mode RandomiseMlp; snapshot weights `W0` (via `get_weights`); `on_down` (1st) → `EnterExplore`, weights now differ from `W0`; `on_down` again → `Cancel`; `get_weights()` byte-equals `W0` (exact `==`). +6. **`feedback_randmlp_commit_restores_then_caller_trains`** — enter; `on_up` → `CommitStore`; assert net restored to `W0` (the kept example is the caller's responsibility; here we assert the restore semantics: post-commit weights == `W0`). +7. **`feedback_randmlp_drag_repositions`** — enter; `on_drag` → `Restore`, net restored to `W0`, `exploring()==false`. +8. **`feedback_mode_switch_aborts_explore`** — enter RandomiseMlp; `set_mode(Avoid, mlp)` → net restored to `W0`, `exploring()==false`, `learning_paused()==false`. +9. **`feedback_determinism_fixed_seed`** — two `FeedbackController`s with the same seed, same MLP-seed, identical press sequence (enter RandomiseOutputs, two re-rolls) → `static_output` buffers byte-identical. Proves per-instance RNG determinism (no libc `rand()`). +10. **`feedback_focus_empty_means_all_active`** — no focus mask set; enter RandomiseOutputs; all NOut dims changed from a fixed seed-vector of constants. +11. **`feedback_static_output_bypass_only_in_randout`** — `static_output()` returns false in Avoid and in idle RandomiseOutputs, true only when `RandomiseOutputs && exploring`. + +### 5.2 Native↔WASM parity + +Extend the standalone parity runner in lock-step (`parity_check.cpp` + `parity_wasm.mjs`), appending a feedback block to the existing payload **after** the current stages (bump `kVersion` to 2 in both files and `parity_diff.mjs` if it checks version). Deterministic sequence, identical on both sides: + +``` +// Stage 5 (feedback): with the SAME ParityMLP, after stage-2 training: +// feedback.set_mode(RandomiseOutputs) +// feedback.on_down(current_out = stage-2 outputs, speed=0.1, spread=0.5, pin=null) → EnterExplore +// feedback.static_output(buf) → push 126 floats +// feedback.on_down(... ) → Reroll +// feedback.static_output(buf) → push 126 floats +// feedback.set_mode(RandomiseMlp); feedback.on_down(...) (enter, draws weights) +// push 12 probed weights of the now-randomised temp net +// feedback.on_up() → CommitStore (restores) +// push 12 probed weights of the restored net (must equal pre-explore probes) +``` + +The controller must be seeded **identically** in `parity_check.cpp` and `parity_wasm.mjs` (e.g. seed `kSeed ^ 0xFEEDBACC0DE`, matching the `MLHandle` salt). `scripts/parity-check.sh` already float32-diffs at 1e-5 — the new floats are covered automatically. The RandomiseOutputs static vectors are RNG output (not compared against the MLP), so as long as both sides run the *same* `nisps::Rng` from the *same* seed they match exactly; the RandomiseMlp restore-probes prove `get_weights`/`set_weights` round-trips identically across native/WASM. + +### 5.3 Existing suites + +`nisps_core_tests`, `nisps_dsp_engine_tests`, `nisps_modes_tests`, `nisps_golden_tests` are unaffected (controller is additive). `bash scripts/run-all-tests.sh` (chokepoint E) gains the new ctest case and the extended parity blob. **No hardware** is exercised. + +--- + +## 6. Firmware note — feasibility + +The same `FeedbackController<MLP_T>` drops into a firmware mode unchanged, because it honors every rule in the RP2350 performance contract (`CLAUDE.md`): +- **No heap:** `static_out_`, `snapshot_`, `focus_` are all `std::array` sized from `MLP_T::weight_count()` / `kOutput` at compile time. No `std::vector`, no `make_shared` (unlike the old `mlp_weights`). +- **No virtual dispatch:** plain class template; press handlers are direct calls. +- **Deterministic per-instance RNG:** uses `nisps::Rng`, constructed with a seed. +- **`.f` literals / `NISPS_*` attrs:** the controller's hot path is trivial (a loop over `NOut`); mark `roll_static_outputs`/`static_output` `NISPS_FORCE_INLINE` and any audio-touched members `NISPS_AUDIO_MEM` if a firmware mode reads `static_out_` from the audio core (it would not — feedback runs on the control/UI core, like the old `loopCallback`). + +Wiring in a future firmware mode (analogous to the TR-8S `bind_RL_interface` callbacks): +- MomA1 (up) handler → set a `volatile pendingUp_` flag; control loop calls `controller.on_up(mlp)` and, on `CommitStore`/`LikeStore`, adds the example to whatever replay/dataset the firmware mode keeps. +- MomA2 (down) → `pendingDown_` → `controller.on_down(mlp, current_out, speed, spread, pin)`. +- Joystick-click / Tog-A1 release → `pendingDrag_` → `controller.on_drag(mlp)`. +- The mode's per-sample/synth output path checks `controller.static_output(buf)`: if true, emit `buf` (bypass), else run the MLP — exactly the `generateAction()` branch at `InterfaceRL.cpp:998`. +- The mode's training tick checks `controller.learning_paused()` and skips training when true (mirrors `optimiseSometimes()` early-return). +- The "Down Action" `RotarySelectView` calls `controller.set_mode(m, mlp)` (the old `addFeedbackModeView`). + +All handlers run in thread/control context (never ISR), identical to the original which dispatched everything via deferred flags in `loopCallback`. No new locking is needed beyond the mode's existing `mlpActive` spin-lock around `get_weights`/`set_weights`/`draw_weights` — the same lock the old code held around `optimise()`/`generateAction()`. + +--- + +## 7. File-touch summary (implementation-ready) + +| File | Change | +|------|--------| +| `/home/w1n5t0n/src/MEMLNaut-NISPS/nisps/ml/feedback.hpp` | **NEW** — `FeedbackMode`, `FeedbackAction`, `FeedbackController<MLP_T>` (as in §2.2). | +| `/home/w1n5t0n/src/MEMLNaut-NISPS/nisps/wasm/bindings.cpp` | Add `FeedbackController<DefaultMLP>` + `static_out_scratch` to `MLHandle`; add 9 `nisps_ml_feedback_*` C functions (§4). | +| `/home/w1n5t0n/src/MEMLNaut-NISPS/playground/src/ml/types.ts` | Add 9 `_nisps_ml_feedback_*` to `NispsModule`; add `FeedbackMode`/`FeedbackAction` enums. | +| `/home/w1n5t0n/src/MEMLNaut-NISPS/playground/src/ml/wasm-iml.ts` | Thin wrapper methods + a small static-output heap buffer; reuse `pinMaskBuf`. | +| `/home/w1n5t0n/src/MEMLNaut-NISPS/playground/src/modes/mode-runtime.ts` | Route `thumbsDown` through the controller per active mode; gate training on `learning_paused()`. | +| `/home/w1n5t0n/src/MEMLNaut-NISPS/playground/src/stores/exploration-store.ts` | Add `feedbackMode` + `exploring` UI state (selector + indicator). | +| `/home/w1n5t0n/src/MEMLNaut-NISPS/tests/cpp/test_mlp_feedback.cpp` | **NEW** — 11 ctest cases (§5.1). | +| `/home/w1n5t0n/src/MEMLNaut-NISPS/nisps/CMakeLists.txt` | Add `test_mlp_feedback.cpp` to `nisps_core_tests`. | +| `/home/w1n5t0n/src/MEMLNaut-NISPS/tests/cpp/parity_check.cpp` + `parity_wasm.mjs` | Append feedback Stage 5; bump version to 2 (§5.2). | + +### Key reconciliation decisions to record in `ALIGNMENT.md` +1. **AVOID** in the new core = `move_weights` Gaussian perturbation (the new playground's existing behavior), **not** the old k-NN geometric centroid push (which depended on firmware-only `ReplayMemory` and is out of scope). Accepted divergence. +2. RANDOMISE_OUTPUTS static vector uses the controller's **own** `nisps::Rng` (deterministic, per-instance) instead of libc `rand()` — endpoint/resolution differ trivially from `[0,65535]/65535` but the value is generated, not compared. +3. RANDOMISE_MLP randomisation uses `draw_weights(spread)` (spread-aware) instead of the old asymmetric `RandomiseWeightsAndBiasesLin(-0.9,1.1,-0.9,0.3)`. If the asymmetric distribution matters perceptually, that's a follow-up `draw_weights_range(lo,hi)` primitive — not part of this port. diff --git a/docs/redesign/findings-design-and-manifold.md b/docs/redesign/findings-design-and-manifold.md new file mode 100644 index 0000000..63c8bd6 --- /dev/null +++ b/docs/redesign/findings-design-and-manifold.md @@ -0,0 +1,120 @@ +# Findings — Design + Manifold Console: build-oriented brief (Phase-1) + +*Read-only synthesis, 2026-06-27, of the five redesign docs + the on-disk Manifold token export + the +`ConsoleApp.jsx` read directly from the design project. VERIFIED = read from a file. INFER = deduced.* + +> **Note on the mirror:** as of this audit `docs/redesign/manifold-export/` contains ONLY the six token CSS +> files + `styles.css` manifest. The component/console JSX (ConsoleApp, CompositeStage, Dock, …) is in the +> claude.ai design project and must be pulled via DesignSync before Phase-3 build. `ConsoleApp.jsx` was read +> directly and is summarised in §2. + +## Source map (`docs/redesign/`) +- `playground-2.0-rewrite-plan.md` (399ln) — authoritative SolidJS UX+architecture plan: Console IA, drawers, + reactive spine, feature table, roadmap, open questions. +- `engine-architecture.md` (440ln) — one-engine-two-skins: `EngineApi` headless boundary, reuse table, COOP/COEP, + build sequence S0–S7. +- `aimmersive-clone-spec.md` (351ln) — faithful clone spec of deployed `a-immersive.html`: feature inventory, + exact tactile constants, state→store map, engine call points. +- `feedback-modes-port-spec.md` (548ln) — 3-mode "Down Action" ported to `nisps/` as `FeedbackController<MLP_T>`. +- `playground-2026.md` (341ln) — older design-intent doc (dock+drawer, interactive heatmap); reference-only. + +> **Framework mismatch:** every planning doc targets **SolidJS**; the mission is a **React** port. The §4 spine +> translation is the largest design decision and biggest risk (§7). + +## 1. Manifold design language (VERIFIED — token files) +Pure CSS-custom-property layer; React port imports `styles.css` and references vars directly. +- **Colors:** `--bg #0d0d0d` / `--bg-1..3`. `--fg #e8e8e8` / `--fg-mute` / `--fg-dim`. `--line`/`--line-strong`. + Accents `--accent #ff6a00` (warm primary), `--accent-2 #00ccff` (cool data/plots), `--accent-3 #ffa860` + (hover). Semantic `--good #6bc26b`/`--warn #f5c45e`/`--bad #ef5b5b`/`--info #5b9eef`. Console 2.0: + `--danger #ff4466`, `--glass rgba(13,13,13,0.65)`, `--glass-line rgba(255,255,255,0.07)`. Region pins + `--pin-1..5`. Glow alphas `--glow-accent/-2/-focus`. **VCV LED-ring palette derives from these** (workstream E). +- **Spacing:** 4px scale `--sp-0..8`=2/4/8/12/16/24/32/48/64. Radius `--r-1 4`/`--r-2 8`/`--r-3 14`/`--r-pill`. + Z `--z-bg/content/overlay/drawer/modal`. `--control-h 48px`, `--hit-min 44px`. +- **Typography:** mono hero `--font-mono: 'JetBrains Mono',…`; sizes `--fs-xs 11`…`--fs-3xl 48` (body 15); + letter-spacing `--ls-label 0.08em`/`--ls-wide 0.12em`; `--label-transform uppercase`. +- **Effects:** *"glow halos, not drop shadows, on live elements"* (signature). Easings `--ease`,`--ease-out`, + `--ease-console`; durations 120/220/360ms; `--glow-sm/md/lg`, `--focus-ring`. +- **Takeaway:** dark, dense, mono-everywhere, orange-primary/cyan-data, glow-halos, uppercase letter-spaced + labels, glass chrome over a full-bleed canvas. A lint allowlist of these vars is mandated to prevent drift. + +## 2. Console shape (ConsoleApp.jsx VERIFIED; rest INFER) +One fullscreen instrument. `focus = in|out|composite|split` drives the convertible stage: +- `focus='in'` → **Manifold** full-bleed stage (input-first), optional top **ReadoutStrip**. +- `focus='out'` → **OutputStage** + **InputMini** corner PiP (output-first). +- `focus='composite'` → **CompositeStage** with a draggable split persisted to `localStorage['mf-composite-split']`; + `[`/`]` nudge split ±0.04, `=`/`0` reset to 0.5. +- `focus='split'` → **SplitStage**. +- Keyboard: `1-5`→drawers (shape/feel/route/health/help), `\` toggles drawer depth peek↔full, space/↑=commit, + ↓=perturb, z=undo. +- **Dock** (right 48px rail) with peek/full drawers. **VerdictCluster** (perturb/undo/commit/reroll + A/B + hold-preview). **AltitudeNav** console|perform|zen. Health glow at screen edge from a `health` signal. +- **Reactive spine (pseudo today):** `infer(pos, seed, params, axes)` = `MF_infer` placeholder in `model.jsx`; + `values` memo every consumer reads. **Replace `MF_infer` with WasmIML** — that is the core wiring task. + +> The updated mission's CONVERTIBLE spec (§1): one continuous view, a single draggable divider where each panel +> demotes its representation by measured width (full manifold → pad → minimap) and a fully-shut side pops out as +> a draggable corner minimap. This refines `CompositeStage`/`OutputStage`/`InputMini` — build to the mission +> spec, using the JSX as the structural starting point. + +## 3. Component inventory (port to React) +**Primitives** (12 in the design system, VERIFIED present in the cloud project): Button, Slider, PillToggle, +Panel, Badge, Switch, StatusLine, XYPad, VirtualJoystick, ControlAxis, CurvePlot, Sparkline. Plus a-immersive +craft pieces to add: Heatmap/HeatmapCell (3px drag threshold, pointer-capture, 300ms popup grace — non-negotiable), +NoiseRing, LossPlot (real history → needs new `nisps_ml_loss_history` C API), GradientFlow, WeightHealth, +LayerStats, CurvePad, DualRangeSlider. +**Console shell:** ConsoleApp, Manifold, CompositeStage, SplitStage, OutputStage, InputMini, ReadoutStrip, +VerdictCluster, Dock, Drawers, OutputEditor, AltitudeNav, PerformApp, ZenApp, model.jsx, shared-ui.jsx. +**Schema-driven:** one `GenericMode` reading `ui.primary_input`, `params[].{group,curve,tier}`, +`capability_class`, `output_kind` — replaces the 8-9 cloned mode files. + +## 4. Reactive spine → React (load-bearing translation) +SolidJS spine: `inputRaw (signal {equals:false}) → createMemo(processed: deadzone→zoom→curve→smoothing→momentum) +→ createMemo(ml: WasmIML.infer into reused buffer) → createMemo(routed: voice-space + global +curve→smoothing→slew→freeze) → ONE createEffect(backend.send)`. Every consumer reads `ml`/`routed`. + +**React translation (recommended):** do NOT use `useMemo`/`useEffect` for the spine — they recompute on render +and per-frame audio inference must not couple to React's scheduler. +1. **Spine lives below React** in a tiny external store (Zustand / `useSyncExternalStore` / hand-rolled + observable) holding `inputRaw`; derive `processed→ml→routed` **eagerly+synchronously inside the `setInput` + action**; fire the single `backend.send` at the action tail (off-render). +2. **Expose `EngineApi` via React Context** (`EngineProvider`/`useEngine`); components call `engine.setInput`/ + `engine.feedback.thumbsUp()` and **subscribe** via `useSyncExternalStore(subscribe, () => versionCounter)` + (version counter, not the array); read the live `Float32Array` imperatively. +3. **Canvas consumers (Manifold/ReadoutStrip/particle-visualiser) bypass React** — read `engine.routedOutput()` + in one `requestAnimationFrame` loop; rAF touches drawing only, never inference. +4. **Pointer-rate coalescing** stays in the engine (batch to display cadence via microtask). +**E2E invariant (port verbatim):** `__nisps.setInputs([x,y]); expect(getOutputs()).toChange() && +expect(getEngineParams()).toChange()` in one tick, per mode in CI; + a no-per-frame-alloc heap fuzz. + +## 5. a-immersive feature-parity checklist +3 compound axes Boldness/Memory/Precision (per-axis table → ~6 params), trim-pot offsets + re-link, 6 control +presets (Default/First Touch/Jazz Hands/Sculptor/Improviser/Microscope), snapshots→snapshot-DAG (unifies +undo/A-B/trail), A/B compare + Freeze Output, region pins (long-press joymap, `--pin-1..5`), param pins (pin +mask ≠ mute), interactive heatmap, weight health (edge glow), gradient flow, output pipeline (reuse buffer, no +per-frame alloc), session presets, input pipeline (deadzone→zoom→curve→smoothing→momentum), zoom (log/anchor), +joy-map+adaptive grid, vanishing trail+tap-to-return, noise rings, layer-stats, **real** loss plot, +auto-explore+follow, spread/tame Health lab, WebMIDI+OSC-bridge backends, **3-mode→2-mode Down Action** via +`FeedbackController`. **Research-validity fixes:** train on raw model-space outputs not post-pipeline; real +loss history; no phantom input channels; heap-safe WASM. + +## 6. COOP/COEP + Vite + React + TS scaffolding +COOP `same-origin` + COEP `require-corp` (dev `server.headers`, `preview.headers`, prod server-scoped on the +nginx vhost). `@vitejs/plugin-react`; `base:'./'` + **base-aware WASM URLs** (`import.meta.env.BASE_URL`, not +hardcoded `/nisps.wasm`); `build.target es2022`, sourcemap. Worklet via **`?worker&url`** (verified fix, commit +`f256217`); two-WASM architecture kept. Strict TS, `tsc --noEmit && vite build`, `@playwright/test`. Assets +`public/nisps.{wasm,js}` from `scripts/build-wasm.sh`. Serve new app at `/next/` sub-path, legacy a-immersive +stays at `/` until sign-off; SPA fallback `try_files`. + +## 7. Open risks +1. **Framework mismatch (biggest)** — keep the spine off React's render cycle or live-feedback rot returns. + Lint-enforce "skins may not import engine internals; engine may not import React". +2. **Console JSX must be pulled** before committing component contracts; `focus`/`zen` vs the plan's + depth/auto-dissolve vocabulary must be reconciled. +3. **Fixed-2-input contraction** vs the modular N×M mission requirement (F): hardcoded `MLP<2,10,14,18,126>`; + runtime-shaped MLP deferred behind a passing parity check — confirm scope with Dimi. +4. **Tactile-constant fidelity** (3px threshold, 300ms grace, pointer-capture, double-tap 350ms) — port exactly, + e2e-test. +5. **Parity drift on core touches** — new FeedbackController C API + real loss-history C API + heap-safe vec all + touch the parity-tested core; gate every core-touching step on `parity-check.sh` green. +6. **Two preset systems** (visual/RL examples vs synth override tiers) — keep distinct. +7. **Particle visualiser** must be a faithful port of `js/ui/visualizer.js` (workstream E) — exact look+behaviour. diff --git a/docs/redesign/findings-engine-surface.md b/docs/redesign/findings-engine-surface.md new file mode 100644 index 0000000..5283e69 --- /dev/null +++ b/docs/redesign/findings-engine-surface.md @@ -0,0 +1,82 @@ +# Findings — Engine Surface Audit (Phase-1) + +*Read-only audit, 2026-06-27. VERIFIED from source unless `[INFER]`. Engine A = deployed vanilla JS live at +meml.lnfinitemonkeys.org (`/home/w1n5t0n/deployments/meml-aimmersive/js/`, a snapshot — repo source has +diverged). Engine B = parity-tested TS playground in this repo (`playground/src/`).* + +## Recommendation: wire the new React app to Engine B (the TS playground stack) +Confirmed by the operator's directive ("training must use the same core the firmware builds from"): Engine B's +`nisps.wasm` is compiled from the `nisps/` C++20 core and **parity-locked to native within 1e-5** +(`scripts/parity-check.sh`, `tests/cpp/parity_*`). Engine A's deployed WASM exports an **older C ABI** +(`nisps_mlp_*`) the repo no longer builds (`nisps_ml_*` + `nisps_engine_*`). Lift Engine B, don't fork. + +## Executive summary +1. Engine B is the *current* codebase; Engine A's WASM is a stale ABI snapshot. +2. Engine B parity-tested (native==WASM ≤1e-5); A has no such guarantee. +3. Engine B is fully typed TS as ES modules (`WasmIML`, `EngineHost`); A is clean ESM JS but untyped. +4. **Decisive:** in Engine B, `nisps.wasm` produces the *audio itself* (`_nisps_engine_process_block` in the + worklet) — true firmware parity. In Engine A, `nisps.wasm` is MLP-only; audio is separate JS/Faust/C15. +5. So "reuse the MLP" via A still leaves audio to reimplement; via B you reuse MLP **and** the 8 audio engines + from one WASM. +6. Engine B's worklet is clean: no Emscripten glue, raw `WebAssembly.compile`, 128-sample blocks, per-thread + instance, no SharedArrayBuffer for nisps audio. +7. Engine B exposes the richer ML surface (feedback modes C API, layer stats, RL move/pin, batch infer) and a + synchronous Playwright probe `window.__nisps`. +8. Caveats (none blocking): fixed MLP arch (2→126), single-element `lossHistory` in sync train, mic input not + wired, Solid coupling in `WasmIML` must be abstracted for React. + +## Engine A — deployed vanilla JS (LIVE) +WASM at `meml-aimmersive/wasm/nisps.{js,wasm}`. `WasmIML` (`js/nisps/nisps-wasm.js`, 612L): +`create(nInputs,nOutputs,hiddenLayers=[10,10,14],…)`, `createWithWarmStart(snapshot,newOutputCount,…)` (transfers +hidden weights across output-count changes, used live), `process()` sync, `inferBatch`, dataset add/clear, +`train(options)` sync **(captures real per-iter loss history)**, `trainAsync(onComplete)` worker, `evalLoss`, +`randomiseWeights(spread)`, `moveWeights(speed,spread,outputPinMask)` (pin mask supported), weights get/set, +`getLayerStats()`→{meanAbs,maxAbs,deadFrac,satFrac}. **No feedback-mode API.** Emscripten MODULARIZE glue + +`cwrap`. Exported ABI: `nisps_mlp_{create,destroy,draw_weights_spread,eval_loss,get_layer_stats,get_weights, +infer_batch,inference,move_weights_ex,move_weights_spread,set_weights,train,train_ex,weight_count}` — **MLP +only, no audio**. Audio is separate `SynthEngine` impls: C15 (`c15-bridge.js`, own `c15_engine.wasm` + +worklet + SAB ring), Faust, JS `modular-engine.js`. Data path: input → `WasmIML.process()` → `outputs[]` → +`param-map.js` (`applyCurve`,`applyGroupOverride`,`SYNTH_PARAM_MAP`) → `engine.setParam` → audio. Live arch +bigger than expected: hidden `[32,48,64]`/`[48,48,64]`, warm-start across output counts. COOP/COEP needed only +by the C15 path; the nisps MLP path needs no SAB. + +## Engine B — parity-tested TS playground (in repo) +Files: `ml/wasm-iml.ts` (650), `audio/engine-host.ts` (232), `audio/worklet/nisps-processor.ts` (309), +`input/pipeline.ts` (307), `output/pipeline.ts` (153), `output/curves.ts` (128), `debug/probe.ts`. +`WasmIML`: `create(opts)` — arch **resolved from the WASM build** via `_nisps_ml_describe` (caller sizes +warned+ignored); fixed **2→126**, hidden `[10,14,18]`, 4 layers. `setInput`, `process():Float32Array` +(copies out + pushes to `mlStore`), `inferXY`, `inferBatch` (chunked at 4096), dataset add/clear, `train` +sync, `trainAsync` worker, `evalLoss`, `randomiseWeights`, `moveWeights(speed,spread,pinMask)`, weights +get/set, `getLayerStats()`→{meanAbs,maxAbs,deadFrac,saturatingFrac}, `getLayerStatsFlat`, `dispose`, `reset`, +debounced `saveNow()`, `tryLoadFromStorage_`. **Richer ABI** (`nisps/wasm/bindings.cpp`): all of A's MLP ops +PLUS `nisps_ml_feedback_{up,down,drag,exploring,set_focus,set_mode,get_mode,static_output,learning_paused}`, +`nisps_ml_describe`, AND audio `nisps_engine_{create,destroy,set_params,process_block}`. + +**Audio VIA nisps.wasm (key difference):** `EngineHost` lazy-creates `AudioContext`, registers worklet via +Vite `?worker&url`, fetches `nisps.wasm` on main thread, transfers bytes to worklet. `NispsProcessor` holds a +2nd nisps.wasm instance compiled **without Emscripten glue** (raw `WebAssembly.compile` + import-shape +discovery); per 128-sample block calls `_nisps_engine_process_block(handle,inL,inR,outL,outR,128)`. Engine +switch = `port` message → `_nisps_engine_create` with new ASCII id. Data path: input pipeline +(invert→deadzone→circular clamp→zoom→curve→EMA→momentum, pure fn) → `WasmIML.inferXY` → 126-vec → output +pipeline (global curve→EMA→slew→freeze) → `EngineHost.setParams(Float32Array)` → worklet. **Same C++ engines +as firmware.** COOP/COEP set in `vite.config.ts` (`same-origin`/`require-corp`); nisps audio needs no SAB +(separate per-thread instances); SAB only for the browser-only C15 path. Three isolated nisps.wasm instances +(main infer/sync-train, worker async-train, worklet audio). Debug probe `window.__nisps` (`debug/probe.ts`): +synchronous, bypasses Solid via `untrack`/`batch`; never throws. + +## Gaps to budget for (none blocking) +- **Fixed MLP arch** (2→126, hidden [10,14,18]) baked into WASM; the modular N×M requirement (workstream F) + needs either multiple WASM modules or runtime-shaped MLP (deferred behind a passing parity check). Engine A's + configurable layers + warm-start is the reference for how to add this. +- **`lossHistory` single-element on sync `train()`** (`wasm-iml.ts:419`); `trainAsync` returns a real array; + C API lacks `nisps_ml_loss_history`. Port A's per-iter capture if loss curves matter. +- **Mic input not wired** through the worklet (XIASRI / SoundAnalysisMIDI) — UI scaffolds only. +- **C15** is browser-only, own WASM + SAB; voice space a placeholder. (UI label: "Powerful Synth Engine".) +- **Solid coupling** in `WasmIML`/probe (`mlStore.__set*`, `coreBus.emit`) must become an injected + callback/emitter to make it framework-neutral for React. + +## Reuse plan for the React app +Import `WasmIML` for the MLP, `EngineHost` + the worklet for audio, lift `input/pipeline.ts`, +`output/pipeline.ts`, `output/curves.ts` as pure functions. Replace the Solid `mlStore`/`coreBus` side-effects +with an injected emitter so the engine is framework-neutral, then mount it under React via context (see +`findings-design-and-manifold.md` §4 reactive spine). diff --git a/docs/redesign/findings-feedback-behaviour.md b/docs/redesign/findings-feedback-behaviour.md new file mode 100644 index 0000000..5ed2eaa --- /dev/null +++ b/docs/redesign/findings-feedback-behaviour.md @@ -0,0 +1,158 @@ +# Findings — Current RL / Feedback Behaviour (Phase-1 Audit) + +*Read-only audit, 2026-06-27. Citations are `file:line`. "VERIFIED" = read in source; "INFER" = deduced. +Produced by the Phase-1 research fan-out (workflow `wf_db146f21`), persisted by the orchestrator because a +hook blocks subagent report-file writes. Adversarially spot-checked against source.* + +## Executive summary + +1. The operator's mental model ("negative feedback takes the n-D opposite vector and pushes weights the other + way — geometric push-away") describes the **firmware `InterfaceRL`** path, NOT what is deployed in the + browser/aimmersive build. +2. The **deployed JS engine** and the **`nisps/` C++ core** both implement thumbs-down as + `move_weights(speed, spread, pinMask)` = per-weight decay (`w *= 1-0.1·spread`) then additive Gaussian + noise (`stddev = speed · layer_scale`). There is **no opposite vector, no centroid, no geometric direction** + in deployed/core code (VERIFIED: grep for `centroid|geometric|meanPositive|neg_action` returns nothing in + `deployments/meml-aimmersive/js/`). +3. So "push the weights the other way" is, in the deployed system, actually **"jitter the whole network with + random Gaussian noise"** — undirected diffusion, not directed avoidance. +4. The true geometric push-away (move output away from k-NN centroid of liked examples) exists only in the + firmware submodule (`memllib` InterfaceRL; "geo push" commits) and is documented in + `docs/redesign/feedback-modes-port-spec.md` §1.1. **This is the algorithm Mode 1 must match** (workstream B). +5. **Why often inaudible (deployed AVOID):** noise added across all 4 layers of `MLP<2,10,14,18,126>`, + per-layer scaled `(1-spread)+spread/√fan_in`, capped at `noiseCap=0.12`, spread default 0.6 → tiny + per-step perturbation that the sigmoid output + output smoothing/slew pipeline absorb. One press moves the + heard 126-vector very little, in a random direction uncorrelated with what was disliked. +6. **Three FEEDBACK_MODEs already scaffolded** in untracked working files `nisps/ml/feedback.hpp` + + `tests/cpp/test_mlp_feedback.cpp` (`FeedbackController<MLP_T>`): `Avoid`, `RandomiseOutputs`, + `RandomiseMlp`. Not yet wired into bindings/TS/UI; no CMake registration. +7. `RandomiseMlp` = the operator's Mode 2 as an audition: snapshot weights → `draw_weights(spread)` → user + moves joystick to hear → up commits a +1 example, down/switch restores the snapshot. **This is the seed of + Mode 2** but the updated spec is stricter (scratchpad NEVER trained; explicit place-at-location; warm-start + real model to interpolate anchors). +8. `RandomiseOutputs` = bypass the MLP entirely, hold a static random output vector (focus-aware per-dim), + re-roll on each down; up commits the held vector as a +1 example. +9. The operator's "randomise → anchor → train" alternative maps onto `RandomiseMlp` (randomise) + thumbs-up + `LikeStore` (anchor) + `trainOnCurrent` (train). Missing in deployed build: the randomise-audition-then-anchor + loop; today thumbs-down only diffuses noise and never anchors. +10. The full port spec (component, WASM C API, TS FFI, ~22 ctest cases, parity stage) already exists at + `docs/redesign/feedback-modes-port-spec.md` and matches the untracked `feedback.hpp`. +11. **Parity/perf constraints:** no heap / `std::array` only, per-instance deterministic `nisps::Rng` + (no libc `rand()`), no virtual dispatch, `.f` literals, fixed WASM arch `MLP<2,10,14,18,126>`, + native↔WASM parity within `1e-5`. +12. **Bottom line:** deployed negative-feedback is undirected Gaussian noise (not geometric push-away), which is + why it is weak/inaudible; the in-progress `FeedbackController` already implements the seed of the operator's + randomise-anchor-train idea but is unplumbed past the header. + +## 1. Current algorithm, precisely, per layer + +### 1.1 DEPLOYED (browser/aimmersive) — NOT geometric +Thumbs-down = `move_weights`. Pure-JS reference `deployments/meml-aimmersive/js/nisps/mlp.js:238-265` +(VERIFIED): per layer/node/weight `j`: `decay = 1-0.1*spread`; `layerScale = (1-spread)+(1/√fanIn)*spread`; +`node.weights[j] *= decay`; `accum = Σ_{n=0..2}(rand*2-1)`; `node.weights[j] += 3*accum*speed*layerScale`. +Output-layer nodes with `outputPinMask[ni]` skipped (`mlp.js:248`). WASM wrapper `nisps-wasm.js:389-406` → +`cwrap('nisps_mlp_move_weights_ex')`. IML `iml.js:166-170` forwards then re-infers — **no dislike-direction +logic, no example stored on dislike**. + +**Key finding (VERIFIED, exhaustive grep):** `deployments/meml-aimmersive/js/` has zero matches for +`centroid|meanPositive|geometric|pushStep|dislikeMultiplier|neg_action|RANDOMISE|FEEDBACK_MODE|explore_active`. +Deployed negative feedback is purely undirected Gaussian weight diffusion. The disliked 126-vector is never read. + +### 1.2 `nisps/` C++ core — same move_weights, no geometric push +`nisps/ml/rl.hpp:63-100` `move_weights_layer` (VERIFIED): `layer_scale = spread_scale(spread,fan_in) = +(1-spread)+spread/√fan_in` (`init.hpp:31-35`); `noise_stddev = speed*layer_scale`; `decay = 1-0.1*spread` +(`rl.hpp:49-51`). Bias: `biases[node] += rng.next_float_gaussian(noise_stddev)` unless pinned (`rl.hpp:83`). +Each weight: `w = w*decay + rng.next_float_gaussian(noise_stddev)` (`rl.hpp:92-97`); RNG advanced even on skip. +Pin mask only on `is_final_layer` (`rl.hpp:78-79`). `mlp.hpp:312-322` iterates 4 layers, only layer3 +final+masked. `draw_weights` (`mlp.hpp:324-332`→`init.hpp:42-54`): `w = rng.next_float_signed()*spread_scale`, +biases zeroed. C API `nisps_ml_move_weights`/`nisps_ml_draw_weights` (`bindings.cpp:419-437`). TS +`wasm-iml.ts:495-508`. Handler `mode-runtime.ts:548-565`: builds pin mask, `moveWeights(cap,spread,pinMask)`, +`growNoise`, re-`setInput`. **No example stored on thumbs-down** (vs thumbs-up `mode-runtime.ts:523-546`). + +### 1.3 TRUE geometric push-away (firmware only) — THE MODE-1 TARGET +Per `feedback-modes-port-spec.md:61-79` (sourced from firmware `InterfaceRL.cpp`), applied in `optimise()` not +at press time: `pushStep = clamp(|avgRewardNeg|,0.25,1.0)*0.5`; `dir[j] = neg_action[j] - +meanPositiveAction[j]` (away from k-NN(+) centroid); `target[j] = clamp(neg_action[j] + +(dir/||dir||)[j]*pushStep/(1+||dir||), 0, 1)`; then train toward `target`. Depends on firmware-only +`ReplayMemory`. Lives in `src/memllib` submodule (commits `2429bcc` "geo push", `d301cc7` "rlview and dislike"; +upstream `SB2026`; memllib main `e291192`/`abe93ec`). **Workstream A must confirm the newest variant.** + +## 2. FEEDBACK_MODEs implemented / scaffolded, and where + +| Mode | Firmware memllib | nisps/ core (untracked) | Deployed JS | Playground TS | +|---|---|---|---|---| +| AVOID geometric push | VERIFIED (geo-push commits) | NOT ported | — | — | +| AVOID = move_weights | — | `feedback.hpp:103-105` | `mlp.js`/`nisps-wasm.js` | `mode-runtime.ts:548` | +| RANDOMISE_OUTPUTS | VERIFIED (spec §1.2) | `feedback.hpp:106-112,160-181` | — | — | +| RANDOMISE_MLP | VERIFIED (spec §1.3) | `feedback.hpp:113-119,183-189` | — | — | + +**Untracked `nisps/ml/feedback.hpp`** (214 lines, VERIFIED): `nisps::ml::FeedbackController<MLP_T>`. +`enum FeedbackMode {Avoid=0,RandomiseOutputs=1,RandomiseMlp=2}` (`:47-51`); `enum FeedbackAction +{None,AvoidPerturb,LikeStore,EnterExplore,Reroll,CommitStore,Cancel,Restore}` (`:56-65`). State all +`std::array` (no heap), own `Rng` seeded separately (`:203-210`). `on_down`: Avoid→move_weights; +RandOut→enter/roll; RandMlp→enter(snapshot+draw_weights)/cancel. `on_up`: exploring→restore+CommitStore else +LikeStore. `on_drag`: RandMlp+exploring→restore+Restore else LikeStore. `static_output`: bypass hook only when +RandomiseOutputs&&exploring. `roll_static_outputs`: focused→`rng_.next_float_uniform()`, unfocused frozen. +`set_mode` aborts active exploration first. + +**Untracked `tests/cpp/test_mlp_feedback.cpp`** (437 lines, ~22 NISPS_TEST, VERIFIED): +`SmallMLP=MLP<2,4,4,4,6,8,32>`; covers all transitions, RNG determinism, focus edge cases, a golden RNG +stream. **Not registered in `nisps/CMakeLists.txt`** (spec §5.1/§7 lists as TODO). + +**Not yet wired (gaps):** no `nisps_ml_feedback_*` in `bindings.cpp`; no `FeedbackController` on `MLHandle`; +no TS FFI in `types.ts`; no UI selector. Firmware parent branch `origin/feat/feedback-explore-modes` wires a +TR-8S FEEDBACK_MODE selector (`5c2d4d2`) and adds `docs/dislike_system_{analysis,design_space}.md` (4 meanings +of "dislike"; here/anywhere × point/radius scope matrix — VERIFIED). + +## 3. Spread / noise / zoom machinery +- **spread∈[0,1]** (`exploration-store.ts:35,74`, default 0.6): feeds per-layer scale and decay. spread=0: + uniform weights, big noise, no decay (saturated); spread=1: Xavier, small noise, 10% decay (spread out). +- **noiseCap∈[floor,1]** (default 0.12): the `speed`/`cap` magnitude for thumbs-down and auto-explore. +- **growNoise/decayNoise** driven by pointer `pressureForce` (`mode-runtime.ts:491-499,544,562`). +- **Zoom** scales auto-explore intensity: `scaledIntensity = intensity*(0.3+0.7*zoom)` — an input-pipeline + gain, not output magnitude. +- **Auto-explore loop** (`mode-runtime.ts:444-482`): periodic `move_weights`+`growNoise`, default 2000ms, off + by default — same diffusion as thumbs-down. +- **Pin mask** `buildPinMask(...)`: freezes final-layer weights for pinned/overridden params — the only + directional gating today (protects dims, doesn't push them). *Relevant to the SOLO/arm design (B).* + +## 4. Parity/perf constraints any core change must respect +No heap in `nisps/` (`std::array`/`FixedBuffer`; `feedback.hpp` obeys); deterministic per-instance `nisps::Rng`, +no libc `rand()`; no virtual dispatch; `.f` literals, no double promotion, `NISPS_*` mem attrs; fixed WASM arch +`MLP<2,10,14,18,126>` (snapshot = `weight_count()` floats, compile-time fixed); native↔WASM parity within +`1e-5`. RandomiseMlp snapshot/restore must round-trip `get_weights`/`set_weights` byte-identically; +RandomiseOutputs static vectors need same-seed/same-Rng; golden tests pin RNG streams. + +## 5. WHY the effect is inaudible — the mechanism +1. **Undirected, not opposite.** Zero-mean Gaussian noise on every weight in all 4 layers; the disliked output + is never read. Expected output change ≈ 0; only variance moves — wobbles randomly, doesn't move *away*. +2. **Small magnitude, multiply-attenuated.** `speed=0.12, spread=0.6` → output layer `layer_scale≈0.54`, + `stddev≈0.065` per weight, squashed by a near-saturated sigmoid, then damped by smoothing→slew→freeze. +3. **Decay fights accumulation.** `w *= 0.94` each press pulls back toward the basin → equilibrium by design. +4. **Pinned/override dims frozen** on the final layer → never respond. +5. **No anchoring.** Thumbs-down stores no example and triggers no training, so there's no learning signal. + +## 6. Operator's Mode 2 ("explore-and-place") vs what exists +Already half-built as `RandomiseMlp`: +- **Randomise** = `RandomiseMlp::on_down` enter → `snapshot + draw_weights(spread)`: whole net re-rolled → + audibly different; user auditions via joystick. +- **Anchor** = `on_up` → `restore_after_explore` + `CommitStore`: caller stores +1 example at + `(current_input, heard_output)`, original net restored. +- **Train** = existing `trainOnCurrent()` trains restored net toward the anchored example. +- **Discard** = `on_down` again → `Cancel`, or mode-switch `abort_explore` restores snapshot. + +**Updated-spec deltas to implement (workstream B):** the scratchpad must be NEVER trained (only randomise + +undoable nudge); the place step must let the user CHOOSE the input location (press → pick location → release); +the real model is warm-started to interpolate ALL placed anchors. The existing `RandomiseMlp` is the closest +prototype but commits at the *current* input position, not a chosen one, and lacks the explicit nudge op. + +### Key files +- Deployed: `deployments/meml-aimmersive/js/nisps/{mlp.js,nisps-wasm.js,iml.js}` +- Core RL: `nisps/ml/{rl.hpp,init.hpp,mlp.hpp}` +- Core feedback (untracked): `nisps/ml/feedback.hpp`, `tests/cpp/test_mlp_feedback.cpp` +- WASM API: `nisps/wasm/bindings.cpp:419-437` +- TS: `playground/src/ml/wasm-iml.ts:488-508`; runtime `playground/src/modes/mode-runtime.ts:444-565` +- Exploration knobs: `playground/src/stores/exploration-store.ts` +- Port spec: `docs/redesign/feedback-modes-port-spec.md` +- Firmware geo-push: `src/memllib` (commits `2429bcc`, `d301cc7`; upstream `SB2026`; main `e291192`/`abe93ec`); + design docs on `origin/feat/feedback-explore-modes`: `docs/dislike_system_{analysis,design_space}.md` diff --git a/docs/redesign/inputs-spec.md b/docs/redesign/inputs-spec.md new file mode 100644 index 0000000..b73e82a --- /dev/null +++ b/docs/redesign/inputs-spec.md @@ -0,0 +1,270 @@ +# Inputs Spec — Modular Input Layer (Workstream F) + +*Status: implementation-ready spec for the `manifold/` React app. Scope: the modular input layer — sources (XY pad, MIDI input, gamepad single/double-stick), how they compose into an N-dimensional input vector, the MLP-rebuild-on-input-change mechanism, the binding to the reactive spine, and the dock INPUTS panel. Read alongside `docs/redesign/findings-engine-surface.md` (the fixed-2-input gap), `docs/redesign/engine-architecture.md` (the `EngineApi` seam — note that doc says SolidJS/`playground2`; this app is React/`manifold/`), and `docs/redesign/aimmersive-clone-spec.md` (existing gamepad/MIDI/joystick behaviour). British spelling in product copy. The built-in synth is always shown as the "Powerful Synth Engine" — never "C15".* + +--- + +## 0. The problem this layer solves + +Today input is hard-wired to exactly two channels. `WasmIML.inferXY(x, y)` calls `setInput(0,x); setInput(1,y); process()` (`playground/src/ml/wasm-iml.ts:307-312`); the whole input pipeline is a 2-tuple pure function `processInput(raw: readonly [number, number], …)` (`playground/src/input/pipeline.ts:225-230`); the WASM build is fixed at `MLP<2u, 10u, 14u, 18u, 126u>` (`nisps/wasm/bindings.cpp:88`). The mission wants the input set to be **modular**: the operator picks one source *or a combination* (XY pad, MIDI input, single-stick gamepad = 2 dims, double-stick gamepad = 4 dims), and the **MLP is (re)built for N inputs × M outputs** whenever that set changes. The same composed input vector drives all backends (synth / MIDI / OSC / VCV) — it is upstream of the backend choice, which is workstream E/output's concern. + +This collides head-on with the fixed-2-input WASM (`findings-engine-surface.md:68-71`, "Gaps to budget for → Fixed MLP arch"). Section 4 resolves that collision and recommends an approach. + +--- + +## 1. Architecture: where the input layer sits + +The input layer is **engine-side and headless** — it produces an N-dim `Float32Array` that is the head of the reactive spine. It owns: the registry of active sources, each source's adapter lifecycle, the composition of per-source axes into one vector, and the contract that fires an MLP rebuild when the active-channel count changes. It exposes its surface through `EngineApi` (the React app's single seam to the engine). Skins (the dock INPUTS panel, the Manifold stage) are pure consumers. + +``` + physical inputs InputLayer (engine-side, headless) spine + ┌──────────────┐ adapter ┌───────────────────────────────────────────┐ + │ XY pad (DOM) │──emit axes──►│ InputSource registry (ordered, enabled) │ + │ WebMIDI │──emit axes──►│ → compose into channel vector raw[N] │──► inputRaw: Float32Array (N) + │ Gamepad API │──emit axes──►│ → per-channel pipeline (deadzone…) │ │ + └──────────────┘ │ → reshape signal on channelCount change │ ▼ + └───────────────────────────────────────────┘ processedInput → mlOutput → routedOutput → backends + │ channelCount change + ▼ + MLP rebuild (§4): WasmIML.rebuild(N, M) with warm-start +``` + +Litmus test (from `engine-architecture.md:40`): anything that touches the DOM/JSX is a skin; the InputLayer adapters read raw browser APIs (`PointerEvent`, `navigator.requestMIDIAccess()`, `navigator.getGamepads()`) but emit **plain numeric axes** into the engine — they hold no React state. The dock panel is the only React surface and it only calls `EngineApi` actions and reads `EngineApi` accessors. + +### File layout (under `manifold/src/engine/input/`) + +``` +engine/input/ +├── source.ts # InputSource interface + InputAxis types + SourceId +├── registry.ts # active source set (ordered, enabled), channel layout derivation +├── compose.ts # sources' axes → flat raw[N] vector (channel map) +├── pipeline.ts # LIFTED from playground/src/input/pipeline.ts, generalised to N channels (§3.3) +├── sources/ +│ ├── xy-pad.ts # XYPadSource — 2 axes, fed by the Manifold/XYPad pointer +│ ├── midi-input.ts # MidiInputSource — WebMIDI, N learn-mapped CC/note channels +│ └── gamepad.ts # GamepadSource — single (2) or double (4) stick, + buttons→actions +└── reshape.ts # channelCount-change → MLP rebuild orchestration (§4) +``` + +--- + +## 2. The `InputSource` adapter interface + +Every source is an adapter that contributes an ordered list of **axes** (each a normalised scalar in `[0,1]`) and, optionally, **action events** (button-style: thumbs-up/down/train/etc., already an a-immersive convention — gamepad LB/RB/A/X/B, `aimmersive-clone-spec.md:161-162`). The interface is deliberately tiny so a fourth source (hand-tracking, OSC-in, sensor) drops in without touching compose/reshape. + +```ts +// engine/input/source.ts +export type SourceId = 'xy' | 'midi' | 'gamepad'; + +/** A single named input dimension produced by a source. Value in [0,1]. */ +export interface InputAxis { + /** Stable per-source key, e.g. 'x', 'y', 'stickL.x', 'cc:74'. Used for persistence + UI labels. */ + readonly key: string; + /** Human label for the dock panel, British spelling. e.g. 'Stick L — X'. */ + readonly label: string; + /** Current normalised value [0,1]; 0.5 = centre for bipolar axes. */ + value: number; +} + +/** Discrete control event a source can raise (maps to verdict/train actions). */ +export type InputAction = + | 'thumbsUp' | 'thumbsDown' | 'train' | 'randomise' | 'clearExamples' | 'undo'; + +export interface InputSource { + readonly id: SourceId; + /** Stable, ordered axes this source contributes when enabled. Length is fixed + * per source *configuration* (gamepad: 2 or 4 depending on stick mode; + * midi: however many channels the user has learn-mapped). */ + axes(): readonly InputAxis[]; + + /** Begin producing values. `onAxes` is called when any axis changes (coalesced + * to display cadence by the engine, not by the source). `onAction` fires discrete + * events. Returns a teardown. Idempotent. */ + attach(sink: { + onAxes: () => void; // pull model: engine reads axes() after notify + onAction: (a: InputAction) => void; + }): () => void; + + /** Availability + status for the dock panel (e.g. 'no MIDI device', 'gamepad 0 connected'). */ + status(): { available: boolean; connected: boolean; detail: string }; + + /** Serialisable per-source config (mappings, invert, stick mode). */ + toJSON(): unknown; + fromJSON(data: unknown): void; +} +``` + +Design notes: +- **Pull, not push, for values.** `onAxes()` is a *notify* (a dirty flag); the engine then reads `source.axes()` during composition. This matches the spine's "every consumer reads accessors" rule (`engine-architecture.md:188`) and avoids each source independently writing into the spine — there is exactly one write of `inputRaw` per coalesced tick. +- **Actions are separate from axes.** Gamepad buttons and (optionally) MIDI notes raise `InputAction`s routed to `EngineApi.feedback.*`/`train()`, exactly as the legacy gamepad does (`aimmersive-clone-spec.md:161-162`). They do **not** add input channels. +- **Axis count per source is a function of its config**, not fixed: gamepad single-stick = 2 axes, double-stick = 4; MIDI = the number of learn-mapped controls. This variable count is what drives the rebuild. + +--- + +## 3. The three sources + +### 3.1 `XYPadSource` (2 axes) + +The default, always-available source. Axes `x`, `y` in `[0,1]`, centre 0.5. Fed by the Manifold stage / dock XYPad pointer through `EngineApi.setInput(x, y)` — i.e. the XYPad does not poll a device; the skin pushes pointer coordinates into this source. This preserves the exact a-immersive joystick mapping (`aimmersive-clone-spec.md:92-94`: tap = snap, drag = relative, Y inverted) which lives in the **skin** XYPad component, not here; the source just receives `(x,y)`. + +- `attach`: registers the setter the skin calls; no device polling. +- `status`: always `{available:true, connected:true, detail:'XY pad'}`. +- This is the only source where the skin is the producer; MIDI and gamepad poll the browser directly. + +### 3.2 `MidiInputSource` (N learn-mapped axes) — NEW + +WebMIDI input, the new required source. Uses `navigator.requestMIDIAccess({ sysex:false })`. The user **learn-maps** physical controls to input channels: + +- **Continuous channels** from CC messages: each mapped CC# (on a chosen channel 1–16) becomes one axis, value = `cc/127` → `[0,1]`. This mirrors the *output* CC convention already in the codebase (`aimmersive-clone-spec.md:128`, CC 0–127, Ch 1–16) but inverted to *input*. +- **Note channels (optional)**: a mapped note can act as a momentary axis (`velocity/127` while held, 0 on note-off) or be routed to an `InputAction` (e.g. a pad → thumbsUp). Default: pitch-bend → one bipolar axis (centre 0.5) is offered as a convenience mapping. +- **Learn flow**: dock panel "+ Learn" → next incoming CC/note within a timeout binds a new axis; the axis appears in `axes()` and the channel count increases → triggers a rebuild (§4). Removing a mapping decreases the count → rebuild. + +```ts +// engine/input/sources/midi-input.ts (shape) +interface MidiMapping { + key: string; // 'cc:74@1' | 'note:36@10' | 'pb@1' + kind: 'cc' | 'note-hold' | 'pitchbend'; + channel: number; // 1..16 + selector: number; // cc# | note# (ignored for pitchbend) + label: string; // user-editable, British copy default 'MIDI CC 74' + invert: boolean; +} +``` + +- `attach`: subscribes to `MIDIInput.onmidimessage`, decodes status bytes, updates the matching mapping's axis value, calls `onAxes()`. Hot-plug handled via `access.onstatechange`. +- `status`: reports `access` state and connected input device names; `{available:false}` if `requestMIDIAccess` rejected or unsupported (Safari). +- Persistence: `toJSON` serialises the mapping array under the per-app storage key (consistent with the legacy `nisps-midi-cc-map:<engineId>` pattern, `aimmersive-clone-spec.md:239`, but for the *input* side use `nisps:midi-in-map`). + +> **Browser note (open choice):** WebMIDI input is Chromium/Edge/Opera and Firefox-with-flag; **Safari has no WebMIDI**. The dock panel must degrade gracefully (`status.available=false`, greyed source, explanatory copy). See §7. + +### 3.3 `GamepadSource` (2 or 4 axes) — single OR double stick + +Uses the Gamepad API (`navigator.getGamepads()`), polled inside the engine's single coalescing tick (not its own rAF — the engine already runs one display-cadence loop per `engine-architecture.md:183`). Generalises the legacy `GamepadInput` (`aimmersive-clone-spec.md:161-162`, currently left-stick-only with `invertY:true`). + +- **Stick mode `single`** (default): left stick → 2 axes `stick.x`, `stick.y`, mapped `axis*0.5+0.5` → `[0,1]`, Y inverted (preserve legacy `invertY:true`). +- **Stick mode `double`**: left + right stick → 4 axes `stickL.x`, `stickL.y`, `stickR.x`, `stickR.y`. This is the "double joystick = 4 input dims" requirement. Switching single↔double changes the axis count 2↔4 → triggers a rebuild (§4). +- **Buttons → actions** (unchanged from legacy): LB→`thumbsDown`, RB→`thumbsUp`, A→`train`, X→`randomise`, B→`clearExamples`. Raised via `onAction`, never as axes. +- `status`: `{available: 'getGamepads' in navigator, connected: <any pad index present>, detail: pad.id}`. Connection via `gamepadconnected`/`gamepaddisconnected` window events. + +### 3.4 The shared per-channel pipeline (lift + generalise `input/pipeline.ts`) + +`playground/src/input/pipeline.ts` is a parity-relevant, golden-tested pure function but hard-coded to a 2-tuple with **coupled** X/Y stages (circular clamp to a unit disk, momentum computed from 2D velocity — `pipeline.ts:258-268`, `164-198`). For N channels it must be generalised: + +- **Per-channel stages stay**: invert, deadzone, zoom-around-anchor, centred power curve, EMA smoothing, freeze (these are all per-axis already — `pipeline.ts:251-290`). +- **The 2D-coupled stages become opt-in pairing**: circular clamp and 2D-velocity momentum only make sense for a *pair* of axes that represent an XY plane (the XY pad, one gamepad stick). The generalised pipeline groups channels into **pairs declared by the source** (XYPad → one pair; gamepad double → two pairs; MIDI CCs → unpaired/independent). Paired channels run the circular-clamp + 2D-momentum path bit-for-bit identical to today; unpaired channels skip those stages (no behavioural change for the existing 2-input case → parity preserved). +- Config becomes per-channel (`InputConfig[]` indexed by channel, defaults from `defaultInputConfig()` `pipeline.ts:107-125`), and state per-channel (`InputState[]`, `defaultInputState()` `pipeline.ts:127-135`). + +**Critical research-validity rule** (`findings-design-and-manifold.md:97-98`): train on **raw model-space**, and feed the MLP the **processed** input — but the *training examples* must record the same channel vector the MLP infers on, and there must be **no phantom input channels**. With a true N-input MLP (§4) this is automatic; with padding (§4 option C) it is the central hazard. + +--- + +## 4. The MLP-rebuild-on-input-change mechanism — the core collision + +When the composed channel count changes (gamepad single→double, MIDI learn add/remove, source toggled on/off), the MLP's input dimension must change. The WASM build is `MLP<2,…>` fixed at compile time (`bindings.cpp:88`; `nisps_ml_describe` returns the baked dims, `bindings.cpp:578`; `WasmIML` resolves arch from the build and *warns-and-ignores* caller sizes, `wasm-iml.ts:195-203`). Three candidate approaches: + +### Option A — Runtime-shaped MLP (one WASM, dynamic dims) +Make the C++ MLP store layer sizes at runtime (heap-allocated weight buffers sized at `nisps_ml_create(input_size, output_size, hidden, n_hidden)`) instead of as template params. `nisps_ml_create` already *accepts* these args (`bindings.cpp:289`); today they're ignored. +- **Pros**: one WASM module; honest N×M; no padding; cleanest semantics; warm-start across both input- and output-count changes is natural (legacy already has `createWithWarmStart` for output changes, `aimmersive-clone-spec.md:250`). +- **Cons / parity + perf**: **directly contradicts the firmware performance contract** — the firmware path forbids heap in hot paths and relies on compile-time-sized `std::array`/`FixedBuffer` (`CLAUDE.md` "No heap"; `nisps/core`). A runtime-shaped MLP is a *second*, divergent ML implementation → it would **not be the parity-tested core** the operator mandated ("training must use the same core the firmware builds from", `findings-engine-surface.md:7`). Parity check `scripts/parity-check.sh` asserts native==WASM ≤1e-5 against the *fixed* templated MLP; a runtime MLP needs its own parity harness. High cost, high parity risk. + +### Option B — Multiple WASM modules (one per common arch) +Pre-build a small matrix of WASM modules: `MLP<2,…,126>`, `MLP<4,…,126>`, and (for MIDI-heavy) `MLP<8,…,126>`. On channel-count change, tear down and recreate `WasmIML` against the module matching N (rounding up to the nearest built arch). +- **Pros**: every module is the *real templated, parity-tested core* — firmware-identical, parity harness unchanged per module. No heap. Honest input dims (up to the nearest built size). +- **Cons / perf**: N is quantised to the built set; download weight ×K modules (each ~94KB, `bindings.cpp` build is small, so ~3 modules ≈ 280KB — acceptable). Warm-start across modules requires weight transplant between *different* WASM instances (hidden weights transfer; new input/output rows re-randomise — the `createWithWarmStart` logic generalised). Build/CI must produce and parity-check each module (`scripts/build-wasm.sh` gains a loop). Medium cost, **low parity risk**. + +### Option C — Max-N padded input with active-channel masking +Keep one WASM at a generous fixed arch, e.g. `MLP<8,…,126>`. Always run 8 inputs; "inactive" channels are pinned to a constant (0.5/centre) and masked. The composed vector fills the first N; the rest are held neutral. +- **Pros**: one module; no rebuild ever (zero-latency source switching); single parity harness. +- **Cons / research-validity**: this is exactly the **phantom-input-channel** anti-pattern the redesign explicitly calls a research-validity bug (`findings-design-and-manifold.md:98`, `engine-architecture.md:409` "killing today's phantom-channel OOB writes"). Held-constant inputs still consume weights and bias the network; training examples recorded at different active-channel sets become inconsistent; the model "sees" 8 dims always. Also wastes firmware budget if the same arch ships to hardware. Low rebuild cost but **high validity cost** — rejected on principle. + +### Recommendation: **Option B (multiple WASM modules), with a phased rollout** + +Option B is the only approach that keeps the **parity-tested, firmware-identical templated core** (the operator's hard constraint) while honestly representing N input dims. Concretely: + +1. **Phase F0 (ships first):** build **two** modules — `MLP<2,…,126>` (existing) and `MLP<4,…,126>`. This covers XY pad (2), single-stick gamepad (2), and double-stick gamepad (4) — the gamepad requirement in full. MIDI maps onto these by capping learn-mapped continuous channels and/or rounding N up to 4. Both modules go through `scripts/parity-check.sh` in CI. +2. **Phase F1:** add `MLP<8,…,126>` for richer MIDI rigs (up to 8 continuous channels). N rounds up to the nearest built arch; unused channels of the *chosen* module are still real channels fed real values where present, **never phantom-padded across the active set** — i.e. we only ever select a module whose input size ≥ N, and we feed exactly N real channels, leaving at most a few genuinely-unused trailing channels held at centre *with a recorded "module larger than active set" caveat* (the one residual padding, bounded and surfaced, not silent). +3. **Rebuild orchestration** (`engine/input/reshape.ts`): on channel-count change → (a) snapshot current weights via `WasmIML.getWeights()`; (b) pick target module for new N; (c) `WasmIML.rebuild(targetModule, N, M)` which creates a fresh handle and **warm-starts**: hidden + output weights transfer where shapes match, the input→layer0 weight rows are preserved for surviving channels and re-randomised for new ones (generalise `createWithWarmStart`, `aimmersive-clone-spec.md:250`); (d) if training examples exist and the input dimension changed, the dataset's feature vectors are re-shaped (truncate/zero-extend with a **user confirm**, mirroring the legacy output-count-change confirm at `aimmersive-clone-spec.md:145`) — or cleared; (e) bump `weightsRevision` so the spine re-derives. +4. **The fixed-2 contraction is explicitly time-boxed:** Phase F0/F1 deliver 2 and 4 (and later 8) — *not* arbitrary N. Arbitrary-N runtime shaping (Option A) is deferred and only revisited if a use case needs counts outside the built matrix, and only *behind a passing dedicated parity check* (per `findings-engine-surface.md:68-71` / `engine-architecture.md:409`). + +This is a strictly smaller, safer change than Option A and avoids the validity landmine of Option C, while satisfying "the MLP is (re)built for N×M when the input set changes" for the concrete source matrix the operator named. + +### `EngineApi` + `WasmIML` surface changes + +```ts +// EngineApi additions (engine/types.ts) +interface EngineApi { + // ... existing spine accessors/actions ... + input: { + sources: () => readonly { id: SourceId; enabled: boolean; axisCount: number; status: SourceStatus }[]; + setEnabled: (id: SourceId, on: boolean) => void; // toggling → may rebuild + channelCount: Accessor<number>; // current N (drives rebuild) + channelLabels: () => readonly string[]; // for dock + heatmap axes + setInput: (x: number, y: number) => void; // XYPad fast-path (back-compat) + gamepadStickMode: (m: 'single' | 'double') => void; // → rebuild on change + midi: MidiInputApi; // learn / map / clear / list devices + rebuilding: Accessor<boolean>; // UI disables training mid-rebuild + }; +} +``` + +`WasmIML` (lifted from `wasm-iml.ts`) gains `rebuild(moduleUrl, inputSize, outputSize, { warmStart: boolean })`. Today `init_` hard-binds one module and ignores caller dims (`wasm-iml.ts:171-225`); the rebuild path instantiates the module whose `nisps_ml_describe` reports `inputSize===N` (or the smallest ≥ N), re-allocates the heap buffers sized to the new arch (`wasm-iml.ts:218-225`), and re-pushes weights. The per-module factory cache keys on module URL (extend `cachedFactory`, `wasm-iml.ts:45`). + +--- + +## 5. Composition into the N-dim vector + +`engine/input/compose.ts` builds the flat `raw[N]` consumed by the spine: + +1. **Channel layout** = concatenation, in registry order, of each *enabled* source's `axes()`. So `[XY:x, XY:y]` then `[gamepad:stickL.x, …]` then `[midi:cc74, midi:cc71, …]`. The layout (ordered list of `{sourceId, axisKey, label}`) is the single source of truth for: the MLP input index of each channel, the dock panel rows, and the heatmap/diagnostics axis labels. +2. On any source `onAxes()` notify, the engine (in its one coalescing tick, `coalesce.ts` per `engine-architecture.md:183-185`) reads every enabled source's `axes()`, writes their values into `raw[]` at their layout indices, and sets the `inputRaw` signal `{equals:false}`. +3. **Channel-count change detection:** the layout length is compared to the live MLP input size each time the registry changes (source toggled, gamepad stick mode flipped, MIDI mapping added/removed). A mismatch enqueues a rebuild (§4) *before* the next infer — never mid-tick. +4. **Per-channel pipeline** (§3.4) runs on `raw[]` to produce `processedInput[]`, which is what the MLP infers on and what training examples record (no phantom channels — the recorded feature vector length == live MLP input size). + +Multiple simultaneous sources compose naturally: XY pad + a MIDI CC = 3 channels → module `MLP<4,…>` (round up), 3 real channels fed, 1 trailing channel held at centre with the bounded-padding caveat surfaced in the panel. + +--- + +## 6. Binding to the reactive spine (React) + +Per `findings-design-and-manifold.md:70-88`, the spine must live **below React** and not on the render cycle. The input layer is the head of that spine: + +- `inputRaw` is an external-store signal (`Float32Array(N)`, `{equals:false}`), updated only inside the engine's coalesced tick. React components **never** write it directly except via `EngineApi.input.setInput`/source toggles. +- The dock INPUTS panel subscribes via `useSyncExternalStore(subscribe, () => versionCounter)` (version counter, not the array — `findings-design-and-manifold.md:81-82`) to re-render on source-set / status / channel-count changes. It reads live axis values imperatively for meters. +- The Manifold/visualiser reads channel values in its own `requestAnimationFrame` loop (drawing only; `findings-design-and-manifold.md:84`). +- **Rebuild is an engine action**, not a render effect: `EngineApi.input.setEnabled`/`gamepadStickMode`/`midi.learn` run the §4 reshape synchronously off-render, bump `weightsRevision`, then flip `rebuilding()` back to false. The single send-effect (`engine-architecture.md:173-177`) re-fires once after rebuild. +- **E2E invariant extension** (`findings-design-and-manifold.md:86-87`): the existing `setInputs([x,y]) → getOutputs() changes` probe assertion is extended with a per-source-set case — e.g. enable double-stick gamepad, assert `channelCount()===4`, assert `getOutputs()` length and that moving channel 3 changes outputs. `window.__nisps` (engine probe, `engine-architecture.md:237`) gains `setChannel(i, v)`, `getChannelCount()`, `setSources([...])`. + +--- + +## 7. The dock INPUTS panel (skin) + +A dedicated dock entry (its own rail icon + drawer), parallel to the other dock drawers (`findings-design-and-manifold.md:50`). Pure consumer of `EngineApi.input`. Manifold token language: dark glass chrome, orange `--accent #ff6a00` primary, cyan `--accent-2 #00ccff` for live data/meters, uppercase letter-spaced labels, glow halos on live elements (`findings-design-and-manifold.md:24-38`). + +**Layout:** + +- **Source toggles** (top): three `PillToggle`s — **XY Pad** / **MIDI** / **Gamepad** — each enableable independently (combination allowed). Each shows a `StatusLine`: connected device / "No MIDI device" / "No gamepad detected". A disabled/unavailable source (e.g. MIDI on Safari) is greyed with explanatory copy. +- **Channel readout**: a live list of the composed channels in layout order, each row = label + a cyan value meter (Sparkline/bar) reading the live axis. Shows total **N** prominently ("4 INPUT CHANNELS"). A subtle badge when a module larger than the active set is in use ("1 channel held at centre"). +- **MIDI sub-panel** (when MIDI enabled): device picker; a **+ Learn** button (arm → next CC/note binds, with countdown); a list of mappings each with editable **Label**, **CC#/Note**, **Ch**, **Invert**, and a remove (✕). Add/remove warns when training examples exist (rebuild confirm). +- **Gamepad sub-panel** (when gamepad enabled): a `PillToggle` **Single / Double** stick (the 2↔4 dim switch), an invert-Y switch (default on), and a small button-map legend (LB Down / RB Up / A Train / X Randomise / B Clear). +- **XY Pad sub-panel**: per-axis invert switches and the existing input-shaping controls (deadzone / zoom / curve / smoothing / momentum) — these are the per-channel pipeline params (§3.4), reused for any paired axes. +- **Rebuild affordance**: whenever a control would change `channelCount`, the panel shows a confirm if examples exist ("Changing inputs rebuilds the network and may clear training examples"), then disables interaction while `rebuilding()` is true. + +**Keyboard / accessibility:** the dock drawer opens via its number key (consistent with `findings-design-and-manifold.md:48`). All hit targets ≥ `--hit-min 44px`. + +--- + +## 8. Persistence & VCV/bridge note + +- Per-source config persists via `InputSource.toJSON`/`fromJSON` under one versioned blob (use `engine/persist.ts`, `engine-architecture.md:113`); keys: `nisps:input-sources` (enabled set + per-source config), `nisps:midi-in-map` (mappings). The **active channel count is persisted** so the correct WASM module is selected on load *before* weights rehydrate (rebuild-then-load order, mirroring legacy load-then-train, `aimmersive-clone-spec.md:239`). Never auto-request MIDI/gamepad permission on load beyond what the browser grants silently. +- **Bridge / VCV (workstream E):** the composed `inputRaw[N]` is the same vector the VCV module receives over the bridge — the bridge is just another *producer* into a source slot (a future `BridgeInputSource`) or a *mirror* of the composed vector, depending on direction. This spec keeps the input layer backend-agnostic: it produces channels; who consumes them (synth/MIDI/OSC/VCV) is downstream. The four-source interface (XY/MIDI/gamepad + a future bridge/OSC-in) is why `InputSource` is kept minimal. + +--- + +## 9. Open choices for the operator + +1. **MLP-reshape strategy — confirm Option B (multiple WASM modules).** Recommended because it keeps the parity-tested, firmware-identical templated core and avoids phantom channels. Trade-off: input dim is quantised to the built matrix (2/4, later 8), not arbitrary N. Confirm the initial matrix is `{2, 4}` for F0 and `{2, 4, 8}` for F1. If the operator wants *arbitrary* N now, that forces Option A (runtime-shaped MLP) — a divergent, separately-parity-tested core, larger cost, and a departure from "same core as firmware". +2. **Trailing-channel handling when N < module input size.** Recommended: feed N real channels, hold the remaining built-but-unused channels at centre and surface a badge. Confirm this bounded, visible padding is acceptable (it is *not* the rejected silent phantom-channel pattern, but it is a small residual). +3. **Browser support for MIDI/gamepad.** WebMIDI: Chromium/Edge/Opera (+ Firefox flag), **no Safari** → MIDI source degrades to unavailable. Gamepad API: broad including Safari. Confirm the supported-browser target (recommend "Chromium-first, graceful degradation elsewhere", matching the COOP/COEP-heavy build already in `vite.config.ts`). +4. **MIDI note → action vs axis.** Default offered: pitch-bend as a bipolar axis; pads/notes optionally route to `thumbsUp`/`thumbsDown`. Confirm whether note-as-axis (velocity-hold) should be a first-class channel type or stay an advanced toggle. +5. **Does enabling/disabling a source mid-session clear training examples?** Recommended: confirm-and-reshape (truncate/extend feature vectors) rather than hard-clear, to match the legacy warm-start ethos. Confirm acceptable, or prefer the simpler hard-clear-on-input-change. +6. **Dock entry placement/icon** among the existing drawers (`shape/feel/route/health/help`, `findings-design-and-manifold.md:48`) — INPUTS as a new top-level drawer vs a section of an existing one. diff --git a/docs/redesign/manifold-export/components/control/ControlAxis.jsx b/docs/redesign/manifold-export/components/control/ControlAxis.jsx new file mode 100644 index 0000000..e68dfb7 --- /dev/null +++ b/docs/redesign/manifold-export/components/control/ControlAxis.jsx @@ -0,0 +1,54 @@ +import React from 'react'; + +/** + * Manifold ControlAxis — a named macro slider with bipolar endpoint labels + * (e.g. Boldness: Caution ↔ Bold). Shows a live preset tag and value. The + * track accent can be themed per-axis via `accent`. + */ +export function ControlAxis({ label, endpoints = ['', ''], value = 0.5, onChange, preset, accent = 'var(--accent)', disabled = false, style }) { + return ( + <div + style={{ + display: 'flex', + flexDirection: 'column', + gap: 'var(--sp-1)', + background: 'var(--bg-1)', + border: '1px solid var(--line)', + borderRadius: 'var(--r-2)', + padding: 'var(--sp-2) var(--sp-3)', + fontFamily: 'var(--font-mono)', + opacity: disabled ? 0.5 : 1, + pointerEvents: disabled ? 'none' : 'auto', + ...style, + }} + > + <div style={{ display: 'flex', alignItems: 'center', gap: 'var(--sp-2)', fontSize: 'var(--fs-sm)' }}> + <span style={{ fontWeight: 600, textTransform: 'uppercase', letterSpacing: '0.08em', color: 'var(--fg)', flex: 1 }}>{label}</span> + {preset && <span style={{ color: accent, fontSize: 'var(--fs-xs)', textTransform: 'uppercase', letterSpacing: '0.06em' }}>{preset}</span>} + <span style={{ fontVariantNumeric: 'tabular-nums', color: 'var(--fg-mute)', fontSize: 'var(--fs-xs)', minWidth: '4ch', textAlign: 'right' }}>{value.toFixed(2)}</span> + </div> + <input + type="range" + min={0} + max={1} + step={0.01} + value={value} + disabled={disabled} + onChange={(e) => onChange && onChange(parseFloat(e.target.value))} + className="mf-axis-input" + style={{ WebkitAppearance: 'none', appearance: 'none', width: '100%', height: 24, background: 'transparent', margin: 0, cursor: 'pointer', '--mf-axis-accent': accent }} + /> + <div style={{ display: 'flex', justifyContent: 'space-between', fontSize: 10, textTransform: 'uppercase', letterSpacing: '0.08em', color: 'var(--fg-dim)' }}> + <span>{endpoints[0]}</span> + <span>{endpoints[1]}</span> + </div> + <style dangerouslySetInnerHTML={{ __html: ` + .mf-axis-input::-webkit-slider-runnable-track { height: 6px; border-radius: 999px; background: var(--bg-3); } + .mf-axis-input::-moz-range-track { height: 6px; border-radius: 999px; background: var(--bg-3); } + .mf-axis-input::-webkit-slider-thumb { -webkit-appearance: none; appearance: none; width: 18px; height: 18px; border-radius: 50%; background: var(--mf-axis-accent, var(--accent)); margin-top: -6px; box-shadow: 0 0 10px var(--mf-axis-accent, var(--accent)); cursor: pointer; } + .mf-axis-input::-moz-range-thumb { width: 18px; height: 18px; border-radius: 50%; background: var(--mf-axis-accent, var(--accent)); border: none; box-shadow: 0 0 10px var(--mf-axis-accent, var(--accent)); } + .mf-axis-input:focus { outline: none; } + ` }} /> + </div> + ); +} diff --git a/docs/redesign/manifold-export/components/control/VirtualJoystick.jsx b/docs/redesign/manifold-export/components/control/VirtualJoystick.jsx new file mode 100644 index 0000000..51329ad --- /dev/null +++ b/docs/redesign/manifold-export/components/control/VirtualJoystick.jsx @@ -0,0 +1,80 @@ +import React from 'react'; + +/** + * Manifold VirtualJoystick — circular control. Drag the glowing orange knob; + * motion is constrained to the circle. Emits normalized (x, y) in [0,1], y-up. + */ +export function VirtualJoystick({ size = 200, position, onMove, onGrab, onRelease, disabled = false, ariaLabel = 'virtual joystick', style }) { + const [internal, setInternal] = React.useState([0.5, 0.5]); + const [dragging, setDragging] = React.useState(false); + const ref = React.useRef(null); + const pos = position || internal; + + const update = (e) => { + const el = ref.current; + if (!el) return; + const r = el.getBoundingClientRect(); + let x = Math.max(0, Math.min(1, (e.clientX - r.left) / r.width)); + let y = Math.max(0, Math.min(1, 1 - (e.clientY - r.top) / r.height)); + const dx = x - 0.5, dy = y - 0.5; + const dist = Math.sqrt(dx * dx + dy * dy); + if (dist > 0.5 && dist > 1e-12) { x = 0.5 + (dx / dist) * 0.5; y = 0.5 + (dy / dist) * 0.5; } + if (!position) setInternal([x, y]); + onMove && onMove(x, y); + }; + + const down = (e) => { if (disabled) return; e.currentTarget.setPointerCapture?.(e.pointerId); setDragging(true); onGrab && onGrab(); update(e); }; + const move = (e) => { if (dragging) update(e); }; + const up = (e) => { if (!dragging) return; e.currentTarget.releasePointerCapture?.(e.pointerId); setDragging(false); onRelease && onRelease(); }; + + const [x, y] = pos; + return ( + <div + ref={ref} + role="application" + aria-label={ariaLabel} + tabIndex={disabled ? -1 : 0} + onPointerDown={down} + onPointerMove={move} + onPointerUp={up} + onPointerCancel={up} + style={{ + position: 'relative', + width: size, + height: size, + background: 'var(--bg-1)', + border: '1px solid var(--line)', + borderRadius: '50%', + touchAction: 'none', + cursor: dragging ? 'grabbing' : 'grab', + outline: 'none', + userSelect: 'none', + overflow: 'hidden', + opacity: disabled ? 0.4 : 1, + pointerEvents: disabled ? 'none' : 'auto', + ...style, + }} + > + <div style={{ position: 'absolute', inset: '6%', borderRadius: '50%', border: '1px dashed var(--line-strong)', pointerEvents: 'none' }} /> + <div aria-hidden="true" style={{ position: 'absolute', inset: 0, opacity: 0.4, pointerEvents: 'none' }}> + <div style={{ position: 'absolute', left: 0, right: 0, top: '50%', height: 1, background: 'var(--line-strong)' }} /> + <div style={{ position: 'absolute', top: 0, bottom: 0, left: '50%', width: 1, background: 'var(--line-strong)' }} /> + </div> + <div + aria-hidden="true" + style={{ + position: 'absolute', + top: 0, + left: 0, + width: 24, + height: 24, + borderRadius: '50%', + background: 'var(--accent)', + boxShadow: '0 0 12px var(--glow-accent)', + transform: `translate(${x * size}px, ${(1 - y) * size}px) translate(-50%, -50%)`, + pointerEvents: 'none', + }} + /> + </div> + ); +} diff --git a/docs/redesign/manifold-export/components/control/XYPad.jsx b/docs/redesign/manifold-export/components/control/XYPad.jsx new file mode 100644 index 0000000..b59c502 --- /dev/null +++ b/docs/redesign/manifold-export/components/control/XYPad.jsx @@ -0,0 +1,83 @@ +import React from 'react'; + +/** + * Manifold XYPad — square control surface. Drag the glowing cyan dot; emits + * normalized (x, y) in [0,1] with y-up. Uncontrolled by default; pass + * `position` + `onMove` to control it. + */ +export function XYPad({ size = 240, showGrid = true, position, onMove, onGrab, onRelease, disabled = false, ariaLabel = 'XY pad', style }) { + const [internal, setInternal] = React.useState([0.5, 0.5]); + const [dragging, setDragging] = React.useState(false); + const ref = React.useRef(null); + const pos = position || internal; + + const update = (e) => { + const el = ref.current; + if (!el) return; + const r = el.getBoundingClientRect(); + const x = Math.max(0, Math.min(1, (e.clientX - r.left) / r.width)); + const y = Math.max(0, Math.min(1, 1 - (e.clientY - r.top) / r.height)); + if (!position) setInternal([x, y]); + onMove && onMove(x, y); + }; + + const down = (e) => { + if (disabled) return; + e.currentTarget.setPointerCapture?.(e.pointerId); + setDragging(true); onGrab && onGrab(); update(e); + }; + const move = (e) => { if (dragging) update(e); }; + const up = (e) => { if (!dragging) return; e.currentTarget.releasePointerCapture?.(e.pointerId); setDragging(false); onRelease && onRelease(); }; + + const [x, y] = pos; + return ( + <div + ref={ref} + role="application" + aria-label={ariaLabel} + tabIndex={disabled ? -1 : 0} + onPointerDown={down} + onPointerMove={move} + onPointerUp={up} + onPointerCancel={up} + style={{ + position: 'relative', + width: size, + height: size, + background: 'var(--bg-1)', + border: '1px solid var(--line)', + borderRadius: 'var(--r-2)', + touchAction: 'none', + cursor: 'crosshair', + outline: 'none', + userSelect: 'none', + overflow: 'hidden', + opacity: disabled ? 0.4 : 1, + pointerEvents: disabled ? 'none' : 'auto', + ...style, + }} + > + {showGrid && ( + <div aria-hidden="true" style={{ position: 'absolute', inset: 0, opacity: 0.5, pointerEvents: 'none' }}> + <div style={{ position: 'absolute', left: 0, right: 0, top: '50%', height: 1, background: 'var(--line-strong)' }} /> + <div style={{ position: 'absolute', top: 0, bottom: 0, left: '50%', width: 1, background: 'var(--line-strong)' }} /> + </div> + )} + <div + aria-hidden="true" + style={{ + position: 'absolute', + top: 0, + left: 0, + width: 14, + height: 14, + borderRadius: '50%', + background: 'var(--accent-2)', + boxShadow: '0 0 10px var(--glow-accent-2)', + transform: `translate(${x * size}px, ${(1 - y) * size}px) translate(-50%, -50%)`, + pointerEvents: 'none', + }} + /> + </div> + ); +} diff --git a/docs/redesign/manifold-export/components/core/Badge.jsx b/docs/redesign/manifold-export/components/core/Badge.jsx new file mode 100644 index 0000000..997835c --- /dev/null +++ b/docs/redesign/manifold-export/components/core/Badge.jsx @@ -0,0 +1,41 @@ +import React from 'react'; + +const TONES = { + neutral: { fg: 'var(--fg-mute)', bd: 'var(--line)', bg: 'var(--bg-2)' }, + accent: { fg: 'var(--accent)', bd: 'rgba(255,106,0,0.4)', bg: 'rgba(255,106,0,0.12)' }, + good: { fg: 'var(--good)', bd: 'rgba(107,194,107,0.4)', bg: 'rgba(107,194,107,0.14)' }, + warn: { fg: 'var(--warn)', bd: 'rgba(245,196,94,0.4)', bg: 'rgba(245,196,94,0.14)' }, + bad: { fg: 'var(--bad)', bd: 'rgba(239,91,91,0.4)', bg: 'rgba(239,91,91,0.14)' }, + info: { fg: 'var(--info)', bd: 'rgba(91,158,239,0.4)', bg: 'rgba(91,158,239,0.14)' }, +}; + +/** + * Manifold Badge — small status capsule. `dot` prepends a status dot; + * `tone` sets the color. Use for state labels (frozen, training, healthy). + */ +export function Badge({ children, tone = 'neutral', dot = false, style }) { + const t = TONES[tone] || TONES.neutral; + return ( + <span + style={{ + display: 'inline-flex', + alignItems: 'center', + gap: 'var(--sp-1)', + fontFamily: 'var(--font-mono)', + fontSize: 'var(--fs-xs)', + textTransform: 'uppercase', + letterSpacing: '0.06em', + color: t.fg, + background: t.bg, + border: `1px solid ${t.bd}`, + borderRadius: 'var(--r-pill)', + padding: '2px 10px', + lineHeight: 1.6, + ...style, + }} + > + {dot && <span style={{ width: 6, height: 6, borderRadius: '50%', background: t.fg, boxShadow: `0 0 6px ${t.fg}` }} />} + {children} + </span> + ); +} diff --git a/docs/redesign/manifold-export/components/core/Button.jsx b/docs/redesign/manifold-export/components/core/Button.jsx new file mode 100644 index 0000000..e8e2b9b --- /dev/null +++ b/docs/redesign/manifold-export/components/core/Button.jsx @@ -0,0 +1,92 @@ +import React from 'react'; + +/** + * Manifold Button — terminal-styled action. + * Variants: primary (solid orange), secondary (outlined raised), ghost (text). + * Sizes: sm, md, lg. Optional leading glyph. + */ +export function Button({ + children, + variant = 'secondary', + size = 'md', + disabled = false, + glyph, + active = false, + type = 'button', + onClick, + style, + ...rest +}) { + const sizes = { + sm: { padding: '4px 12px', fontSize: 'var(--fs-xs)', height: 28 }, + md: { padding: '8px 12px', fontSize: 'var(--fs-sm)', height: 34 }, + lg: { padding: '10px 18px', fontSize: 'var(--fs-md)', height: 44 }, + }; + const s = sizes[size] || sizes.md; + + const base = { + display: 'inline-flex', + alignItems: 'center', + justifyContent: 'center', + gap: 'var(--sp-2)', + fontFamily: 'var(--font-mono)', + fontSize: s.fontSize, + height: s.height, + padding: s.padding, + borderRadius: 'var(--r-1)', + border: '1px solid var(--line)', + cursor: disabled ? 'not-allowed' : 'pointer', + userSelect: 'none', + transition: 'background var(--dur-fast) var(--ease), border-color var(--dur-fast) var(--ease), color var(--dur-fast) var(--ease)', + whiteSpace: 'nowrap', + }; + + const variants = { + primary: { + background: 'var(--accent)', + borderColor: 'var(--accent)', + color: 'var(--bg)', + fontWeight: 600, + }, + secondary: { + background: active ? 'var(--bg-3)' : 'var(--bg-2)', + borderColor: active ? 'var(--accent)' : 'var(--line)', + color: active ? 'var(--accent)' : 'var(--fg)', + }, + ghost: { + background: 'transparent', + borderColor: 'transparent', + color: active ? 'var(--accent)' : 'var(--fg-mute)', + }, + }; + + const disabledStyle = disabled + ? { opacity: 0.45, color: 'var(--fg-dim)', boxShadow: 'none' } + : null; + + return ( + <button + type={type} + disabled={disabled} + onClick={onClick} + style={{ ...base, ...(variants[variant] || variants.secondary), ...disabledStyle, ...style }} + onMouseEnter={(e) => { + if (disabled) return; + if (variant === 'secondary') { e.currentTarget.style.background = 'var(--bg-3)'; e.currentTarget.style.borderColor = 'var(--line-strong)'; } + if (variant === 'ghost') e.currentTarget.style.color = 'var(--fg)'; + if (variant === 'primary') e.currentTarget.style.background = 'var(--accent-3)'; + }} + onMouseLeave={(e) => { + if (disabled) return; + const v = variants[variant] || variants.secondary; + e.currentTarget.style.background = v.background; + e.currentTarget.style.borderColor = v.borderColor; + e.currentTarget.style.color = v.color; + }} + {...rest} + > + {glyph && <span aria-hidden="true" style={{ fontSize: '1.1em', lineHeight: 1 }}>{glyph}</span>} + {children} + </button> + ); +} diff --git a/docs/redesign/manifold-export/components/core/Panel.jsx b/docs/redesign/manifold-export/components/core/Panel.jsx new file mode 100644 index 0000000..57b15d0 --- /dev/null +++ b/docs/redesign/manifold-export/components/core/Panel.jsx @@ -0,0 +1,30 @@ +import React from 'react'; + +/** + * Manifold Panel — the house surface: bg-1 fill, 1px hairline border, 8px + * radius, no shadow. Optional header row with an uppercase title + actions, + * separated by a hairline. + */ +export function Panel({ title, label, actions, children, padding = 'var(--sp-3)', style }) { + return ( + <section + style={{ + background: 'var(--bg-1)', + border: '1px solid var(--line)', + borderRadius: 'var(--r-2)', + fontFamily: 'var(--font-mono)', + color: 'var(--fg)', + ...style, + }} + > + {(title || label || actions) && ( + <header style={{ display: 'flex', alignItems: 'center', gap: 'var(--sp-2)', padding: 'var(--sp-2) var(--sp-3)', borderBottom: '1px solid var(--line)' }}> + {label && <span style={{ fontSize: 'var(--fs-xs)', color: 'var(--fg-dim)', textTransform: 'uppercase', letterSpacing: '0.08em' }}>{label}</span>} + {title && <h3 style={{ margin: 0, fontSize: 'var(--fs-sm)', fontWeight: 600, color: 'var(--fg)' }}>{title}</h3>} + {actions && <div style={{ marginLeft: 'auto', display: 'flex', gap: 'var(--sp-2)' }}>{actions}</div>} + </header> + )} + <div style={{ padding }}>{children}</div> + </section> + ); +} diff --git a/docs/redesign/manifold-export/components/core/PillToggle.jsx b/docs/redesign/manifold-export/components/core/PillToggle.jsx new file mode 100644 index 0000000..d989df2 --- /dev/null +++ b/docs/redesign/manifold-export/components/core/PillToggle.jsx @@ -0,0 +1,55 @@ +import React from 'react'; + +/** + * Manifold PillToggle — segmented radio control in a pill capsule. + * The selected segment fills solid orange. Options: [{value,label}]. + */ +export function PillToggle({ options = [], value, onChange, ariaLabel = 'segmented control', disabled = false, style }) { + return ( + <div + role="radiogroup" + aria-label={ariaLabel} + style={{ + display: 'inline-flex', + background: 'var(--bg-2)', + border: '1px solid var(--line)', + borderRadius: 'var(--r-pill)', + padding: 2, + gap: 2, + opacity: disabled ? 0.5 : 1, + pointerEvents: disabled ? 'none' : 'auto', + ...style, + }} + > + {options.map((opt) => { + const selected = value === opt.value; + return ( + <button + key={opt.value} + type="button" + role="radio" + aria-checked={selected} + onClick={() => onChange && onChange(opt.value)} + style={{ + background: selected ? 'var(--accent)' : 'transparent', + color: selected ? 'var(--bg)' : 'var(--fg-mute)', + border: 0, + borderRadius: 'var(--r-pill)', + padding: '6px 14px', + fontFamily: 'var(--font-mono)', + fontSize: 'var(--fs-xs)', + textTransform: 'uppercase', + letterSpacing: '0.08em', + cursor: 'pointer', + transition: 'background var(--dur-fast) var(--ease), color var(--dur-fast) var(--ease)', + }} + onMouseEnter={(e) => { if (!selected) e.currentTarget.style.color = 'var(--fg)'; }} + onMouseLeave={(e) => { if (!selected) e.currentTarget.style.color = 'var(--fg-mute)'; }} + > + {opt.label} + </button> + ); + })} + </div> + ); +} diff --git a/docs/redesign/manifold-export/components/core/Slider.jsx b/docs/redesign/manifold-export/components/core/Slider.jsx new file mode 100644 index 0000000..6f5eb79 --- /dev/null +++ b/docs/redesign/manifold-export/components/core/Slider.jsx @@ -0,0 +1,63 @@ +import React from 'react'; + +/** + * Manifold Slider — labeled horizontal range with a glowing orange thumb and + * a tabular value readout. Controlled via value/onChange (0..max). + */ +export function Slider({ + label, + value = 0, + min = 0, + max = 1, + step = 0.01, + unit = '', + onChange, + disabled = false, + format, + style, +}) { + const pct = max > min ? (value - min) / (max - min) : 0; + const display = format ? format(value) : (Number.isInteger(step) ? value : value.toFixed(2)); + + return ( + <div style={{ display: 'flex', flexDirection: 'column', gap: 'var(--sp-1)', fontFamily: 'var(--font-mono)', userSelect: 'none', opacity: disabled ? 0.5 : 1, pointerEvents: disabled ? 'none' : 'auto', ...style }}> + {label && ( + <span style={{ color: 'var(--fg-mute)', fontSize: 'var(--fs-xs)', textTransform: 'uppercase', letterSpacing: '0.06em' }}>{label}</span> + )} + <div style={{ display: 'flex', gap: 'var(--sp-3)', alignItems: 'center' }}> + <input + type="range" + min={min} + max={max} + step={step} + value={value} + disabled={disabled} + onChange={(e) => onChange && onChange(parseFloat(e.target.value))} + style={{ + flex: 1, + WebkitAppearance: 'none', + appearance: 'none', + background: 'transparent', + height: 24, + margin: 0, + cursor: 'pointer', + '--mf-pct': `${pct}`, + }} + className="mf-slider-input" + /> + <span style={{ fontVariantNumeric: 'tabular-nums', fontSize: 'var(--fs-xs)', color: 'var(--fg-mute)', minWidth: '4ch', textAlign: 'right' }}> + {display}{unit && <span style={{ color: 'var(--fg-dim)', marginLeft: 2 }}>{unit}</span>} + </span> + </div> + <style dangerouslySetInnerHTML={{ __html: ` + .mf-slider-input::-webkit-slider-runnable-track { height: 4px; border-radius: 999px; background: linear-gradient(to right, var(--accent) 0%, var(--accent) calc(var(--mf-pct) * 100%), var(--bg-3) 0%); } + .mf-slider-input::-moz-range-track { height: 4px; border-radius: 999px; background: var(--bg-3); } + .mf-slider-input::-webkit-slider-thumb { -webkit-appearance: none; appearance: none; width: 16px; height: 16px; border-radius: 50%; background: var(--accent); margin-top: -6px; box-shadow: 0 0 8px var(--glow-accent); cursor: pointer; transition: transform var(--dur-fast) var(--ease); } + .mf-slider-input::-moz-range-thumb { width: 16px; height: 16px; border-radius: 50%; background: var(--accent); border: none; box-shadow: 0 0 8px var(--glow-accent); } + .mf-slider-input:hover::-webkit-slider-thumb { transform: scale(1.15); } + .mf-slider-input:focus { outline: none; } + .mf-slider-input:focus::-webkit-slider-thumb { box-shadow: 0 0 0 3px var(--glow-focus); } + ` }} /> + </div> + ); +} diff --git a/docs/redesign/manifold-export/components/core/StatusLine.jsx b/docs/redesign/manifold-export/components/core/StatusLine.jsx new file mode 100644 index 0000000..bd4dead --- /dev/null +++ b/docs/redesign/manifold-export/components/core/StatusLine.jsx @@ -0,0 +1,43 @@ +import React from 'react'; + +/** + * Manifold StatusLine — the dim mono readout strip at the bottom of a mode. + * Pass an array of items; strings render plain, {label,value,tone} render a + * labeled readout. Items are joined with the house middle-dot separator. + */ +export function StatusLine({ items = [], style }) { + return ( + <p + style={{ + display: 'flex', + flexWrap: 'wrap', + alignItems: 'center', + gap: 'var(--sp-2)', + margin: 0, + fontFamily: 'var(--font-mono)', + fontSize: 'var(--fs-xs)', + color: 'var(--fg-dim)', + }} + > + {items.map((it, i) => { + const isObj = it && typeof it === 'object'; + const toneColor = isObj && it.tone + ? { accent: 'var(--accent)', cyan: 'var(--accent-2)', good: 'var(--good)', warn: 'var(--warn)', bad: 'var(--bad)' }[it.tone] + : null; + return ( + <React.Fragment key={i}> + {i > 0 && <span aria-hidden="true">·</span>} + {isObj ? ( + <span style={{ color: toneColor || 'var(--fg-dim)' }}> + {it.label && <span style={{ color: 'var(--fg-dim)' }}>{it.label} </span>} + <span style={{ fontVariantNumeric: 'tabular-nums', color: toneColor || 'var(--fg-mute)' }}>{it.value}</span> + </span> + ) : ( + <span>{it}</span> + )} + </React.Fragment> + ); + })} + </p> + ); +} diff --git a/docs/redesign/manifold-export/components/core/Switch.jsx b/docs/redesign/manifold-export/components/core/Switch.jsx new file mode 100644 index 0000000..382d58d --- /dev/null +++ b/docs/redesign/manifold-export/components/core/Switch.jsx @@ -0,0 +1,44 @@ +import React from 'react'; + +/** + * Manifold Switch — compact toggle. On = orange track + glow. Optional label. + */ +export function Switch({ checked = false, onChange, label, disabled = false, style }) { + return ( + <label style={{ display: 'inline-flex', alignItems: 'center', gap: 'var(--sp-2)', fontFamily: 'var(--font-mono)', fontSize: 'var(--fs-sm)', color: 'var(--fg)', cursor: disabled ? 'not-allowed' : 'pointer', opacity: disabled ? 0.5 : 1, userSelect: 'none', ...style }}> + <button + type="button" + role="switch" + aria-checked={checked} + disabled={disabled} + onClick={() => onChange && onChange(!checked)} + style={{ + position: 'relative', + width: 36, + height: 20, + padding: 0, + borderRadius: 'var(--r-pill)', + border: `1px solid ${checked ? 'var(--accent)' : 'var(--line)'}`, + background: checked ? 'var(--accent)' : 'var(--bg-2)', + cursor: disabled ? 'not-allowed' : 'pointer', + transition: 'background var(--dur-fast) var(--ease), border-color var(--dur-fast) var(--ease)', + boxShadow: checked ? '0 0 8px var(--glow-accent)' : 'none', + }} + > + <span + style={{ + position: 'absolute', + top: 2, + left: checked ? 18 : 2, + width: 14, + height: 14, + borderRadius: '50%', + background: checked ? 'var(--bg)' : 'var(--fg-mute)', + transition: 'left var(--dur-fast) var(--ease), background var(--dur-fast) var(--ease)', + }} + /> + </button> + {label && <span>{label}</span>} + </label> + ); +} diff --git a/docs/redesign/manifold-export/components/data/CurvePlot.jsx b/docs/redesign/manifold-export/components/data/CurvePlot.jsx new file mode 100644 index 0000000..4563574 --- /dev/null +++ b/docs/redesign/manifold-export/components/data/CurvePlot.jsx @@ -0,0 +1,66 @@ +import React from 'react'; + +const clamp01 = (v) => (v < 0 ? 0 : v > 1 ? 1 : v); + +const CURVES = { + linear: (x) => x, + exp: (x) => (Math.exp(4 * x) - 1) / (Math.exp(4) - 1), + log: (x) => Math.log(1 + x * (Math.exp(4) - 1)) / 4, + square: (x) => x * x, + sqrt: (x) => Math.sqrt(clamp01(x)), + sigmoid: (x) => { const s = (v) => 1 / (1 + Math.exp(-(v - 0.5) * 8)); const lo = s(0), hi = s(1); return (s(x) - lo) / (hi - lo); }, + cubic: (x) => { const v = clamp01(x); return v * v * (3 - 2 * v); }, + centered_power: (x) => { const o = x - 0.5, sg = o < 0 ? -1 : 1; return clamp01((sg * Math.pow(Math.abs(o) * 2, 0.5)) / 2 + 0.5); }, +}; + +/** + * Manifold CurvePlot — renders one of the named response curves (or a custom + * function f:[0,1]→[0,1]) on the dark grid. The brand's straight-line & + * parabolic/bézier motif. + */ +export function CurvePlot({ curve = 'cubic', fn, width = 200, height = 120, color = 'var(--accent)', showAxes = true, ariaLabel, style }) { + const ref = React.useRef(null); + + React.useEffect(() => { + const cv = ref.current; + if (!cv) return; + const dpr = window.devicePixelRatio || 1; + const w = width * dpr, h = height * dpr; + cv.width = w; cv.height = h; + const ctx = cv.getContext('2d'); + if (!ctx) return; + ctx.clearRect(0, 0, w, h); + const cs = getComputedStyle(cv); + const resolve = (c) => { ctx.fillStyle = c; return ctx.fillStyle; }; + const stroke = color.startsWith('var(') ? cs.getPropertyValue(color.slice(4, -1).trim()).trim() || '#ff6a00' : color; + const pad = 6 * dpr; + + if (showAxes) { + ctx.strokeStyle = 'rgba(255,255,255,0.06)'; + ctx.lineWidth = 1; + ctx.strokeRect(0.5, 0.5, w - 1, h - 1); + ctx.beginPath(); ctx.moveTo(0, h / 2); ctx.lineTo(w, h / 2); ctx.stroke(); + ctx.beginPath(); ctx.moveTo(w / 2, 0); ctx.lineTo(w / 2, h); ctx.stroke(); + } + const f = fn || CURVES[curve] || CURVES.linear; + ctx.strokeStyle = stroke; + ctx.lineWidth = 2 * dpr; + ctx.beginPath(); + for (let p = 0; p <= 120; p++) { + const x = p / 120, y = clamp01(f(x)); + const px = pad + x * (w - 2 * pad); + const py = (h - pad) - y * (h - 2 * pad); + p === 0 ? ctx.moveTo(px, py) : ctx.lineTo(px, py); + } + ctx.stroke(); + }, [curve, fn, width, height, color, showAxes]); + + return ( + <canvas + ref={ref} + role="img" + aria-label={ariaLabel || `${curve} curve`} + style={{ display: 'block', width, height, background: 'var(--bg-1)', border: '1px solid var(--line)', borderRadius: 'var(--r-2)', ...style }} + /> + ); +} diff --git a/docs/redesign/manifold-export/components/data/Sparkline.jsx b/docs/redesign/manifold-export/components/data/Sparkline.jsx new file mode 100644 index 0000000..e747c05 --- /dev/null +++ b/docs/redesign/manifold-export/components/data/Sparkline.jsx @@ -0,0 +1,62 @@ +import React from 'react'; + +/** + * Manifold Sparkline — a compact time-series trace (training loss, a feature + * envelope). Cyan line on a faint grid, with an optional last-value readout. + */ +export function Sparkline({ data = [], width = 320, height = 70, color = 'var(--accent-2)', log = false, showLast = true, format, ariaLabel = 'time series', style }) { + const ref = React.useRef(null); + + React.useEffect(() => { + const cv = ref.current; + if (!cv) return; + const dpr = window.devicePixelRatio || 1; + const w = width * dpr, h = height * dpr; + cv.width = w; cv.height = h; + const ctx = cv.getContext('2d'); + if (!ctx) return; + ctx.clearRect(0, 0, w, h); + if (!data.length) return; + + const cs = getComputedStyle(cv); + const stroke = color.startsWith('var(') ? cs.getPropertyValue(color.slice(4, -1).trim()).trim() || '#00ccff' : color; + + const ys = data.map((v) => (log ? Math.log(Math.max(1e-10, v) + 1) : v)); + let lo = Infinity, hi = -Infinity; + for (const y of ys) { if (y < lo) lo = y; if (y > hi) hi = y; } + if (hi === lo) hi = lo + 1e-6; + + ctx.strokeStyle = 'rgba(255,255,255,0.05)'; + ctx.lineWidth = 1; + for (let i = 1; i < 4; i++) { const y = (i / 4) * h; ctx.beginPath(); ctx.moveTo(0, y); ctx.lineTo(w, y); ctx.stroke(); } + + ctx.strokeStyle = stroke; + ctx.lineWidth = 1.5 * dpr; + ctx.beginPath(); + for (let i = 0; i < ys.length; i++) { + const x = (i / Math.max(1, ys.length - 1)) * w; + const norm = (ys[i] - lo) / (hi - lo); + const y = h - norm * h; + i === 0 ? ctx.moveTo(x, y) : ctx.lineTo(x, y); + } + ctx.stroke(); + + if (showLast) { + const last = data[data.length - 1]; + const txt = format ? format(last) : (typeof last === 'number' ? last.toExponential(2) : String(last)); + ctx.fillStyle = '#9a9a9a'; + ctx.font = `${10 * dpr}px ui-monospace, monospace`; + ctx.textAlign = 'right'; + ctx.fillText(txt, w - 4 * dpr, 12 * dpr); + } + }, [data, width, height, color, log, showLast]); + + return ( + <canvas + ref={ref} + role="img" + aria-label={ariaLabel} + style={{ display: 'block', width, height, background: 'var(--bg-1)', border: '1px solid var(--line)', borderRadius: 'var(--r-2)', ...style }} + /> + ); +} diff --git a/docs/redesign/manifold-export/readme.md b/docs/redesign/manifold-export/readme.md new file mode 100644 index 0000000..c218e23 --- /dev/null +++ b/docs/redesign/manifold-export/readme.md @@ -0,0 +1,101 @@ +# Manifold — Design System + +> Train machine learning, on the fly, to play your instruments. + +Manifold is an **open-source project and a family of commercial products** that train and run on-the-fly machine learning (small MLPs and related techniques) to **program, control, patch, and perform with electronic musical instruments** — synths, samplers, sequencers, Eurorack, and software plugins. + +The shape of the system is always the same three stages: + +``` + INPUTS ENGINE OUTPUTS + joystick ┐ ┌ MIDI + XY pad ├──▶ on-the-fly ML ──▶ ├ CV / gate + trackpad ┘ (MLP / regression) └ OSC → devices + (hw + sw, every hardware input has a software visualisation) +``` + +A performer moves an **input** (a joystick, an XY pad, a trackpad — some hardware, some software, all with live software visualisations). The signal runs through the **engine**, a tiny neural net trained live by the player's own thumbs-up / thumbs-down feedback. The result is sent to the **world** as MIDI, CV, or OSC to control other devices. + +The reference application is the **MEMLNaut playground** — a browser shell hosting nine "modes", each pairing an input surface with a synthesis/effects engine (PAF formant synth, channel strip, granular morph, reverb, drum/breakbeat, sound-analysis→MIDI, and more). + +## Sources given + +- **Codebase** `playground/` — the MEMLNaut playground, a Vite + **SolidJS** + TypeScript app. ML inference runs in-browser via a WASM module (`public/nisps.wasm`); audio synthesis runs in an AudioWorklet. Key paths: + - `src/styles/tokens.css` — the original design tokens (dark canvas, mono, warm+cool accents). **This system is a faithful port + extension of that file.** + - `src/primitives/` — the control + visualisation primitives: `XYPad`, `VirtualJoystick`, `Slider`, `PillToggle`, `ControlAxis`, `TrainingControls`, `LossPlot`, `Heatmap`, `JoyMap`, `OutputDisplay`, `GradientFlow`, `ProgressRing`, `SliderBank`, `WeightHealth`, `Drawer`, `ParamEditor`. + - `src/modes/` — nine modes + `ModeShell` (the shared scaffold) + `ModeSwitcher`. + - `src/output/curves.ts` — the named-curve catalog (`linear, exp, log, square, sqrt, sigmoid, cubic, centered_power`) — a core brand motif (straight lines & parabolic / polynomial / bézier curves). + +There were no Figma links, slide decks, or logo binaries in the source — the brand wordmark in the app is the plain text **`MEMLNaut`** + dim `playground`. See ICONOGRAPHY for how the mark is treated and Caveats at the bottom. + +--- + +## CONTENT FUNDAMENTALS + +How Manifold writes. The voice is a **terminal operator's notebook**: lowercase, terse, technical, unfussy. It trusts the reader is a musician *and* a tinkerer. + +- **Casing.** UI labels and routes are **lowercase** (`home`, `/modes`, `/dev/primitives`). Micro-labels above controls are **UPPERCASE with wide letter-spacing** (`EXAMPLES`, `LOSS`, `BOLDNESS`). Mode names are Title Case (`PAF Synth`, `Channel Strip`). Sentences in prose are sentence case. +- **Person.** Mostly **imperative, addressed to "you"** — *"pick a mode and start playing"*, *"Drag to sculpt formants"*, *"start it from inside any mode"*. Rarely first-person. +- **Tone.** Plain, confident, slightly hacker. States facts and constraints without apology: *"Audio cannot autoplay (browser policy)."*, *"ML inference runs on the main thread via WASM."* Comments in code carry a dry wit (*"Amnesia ↔ Elephant"*, *"Stream 9: nine modes, one shell."*). +- **Density.** Pack real values, not filler. Readouts are always shown with units and tabular numerals: `loss 4.2e-3`, `input (0.62, 0.41)`, `noise 0.018`, `33 in → 1 out`. +- **Verbs of the domain.** train, randomise (British -ise), explore, pin, snapshot, undo, freeze, converge, route. Feedback is **thumbs-up / thumbs-down** (rendered `+` / `−`). +- **Punctuation.** Middle dot `·` separates inline metadata (`engine: paf_synth · 33 in → 1 out`). Arrows `→` `↔` `▶` `⏹` `↶` carry meaning. Ellipsis `…` for in-progress states (`training…`). +- **Emoji.** Used **only as functional glyphs** in dense toolbars where an icon font is absent — `🎤` mic, `⚙` settings, `📚` history, `▶ ⏹` transport. Never decorative, never in prose. Prefer geometric Unicode/SVG over emoji where possible (see ICONOGRAPHY). +- **British spelling** (randomise, visualisation, colour-adjacent terms) — the project's origin. + +Examples (verbatim from the product): + +> *"Interactive ML control of audio. Stream 9: nine modes, one shell."* +> *"Drag to sculpt formants. Voice space: **Bright**"* +> *"pick a mode and start playing"* + +--- + +## VISUAL FOUNDATIONS + +The look is a **vintage-computer / instrument-panel terminal**: near-black canvas, monospace everywhere, hairline borders, and **glow** instead of drop-shadow to signal what's alive. + +- **Palette.** A dark greyscale stack (`#0d0d0d → #141414 → #1c1c1c → #242424`) for canvas → panel → raised → track. Text steps down `#e8e8e8 → #9a9a9a → #5a5a5a`. **Two accents do all the work:** warm **orange `#ff6a00`** (the primary — actions, focus, the live control dot) and cool **cyan `#00ccff`** (data, plots, the secondary dot). Semantic green/amber/red/blue are reserved for training health (loss, gradient flow). Five translucent **pin** colors mark regions on the joy-map. +- **Type.** **JetBrains Mono** is the hero face — used for ~everything, including titles. A system sans is the quiet fallback for long prose only. Fixed px scale (`11 / 13 / 15 / 18 / 24`). Values use `font-variant-numeric: tabular-nums`. Micro-labels are 11px uppercase, `letter-spacing: 0.08em`, in `--fg-mute`. +- **Borders.** Everything is outlined with a **1px hairline** `#2a2a2a`. Stronger `#3a3a3a` for internal grid lines and dashed guides (the joystick boundary is a 1px **dashed** circle). Borders, not shadows, define cards. +- **Backgrounds.** Flat solids only — **no gradients, no images, no texture** on surfaces. The only "imagery" is generated: canvas-drawn plots, heatmaps, and the named math curves. The motif is **line + curve** (a loss trace, a parabola, a bézier), never photography or illustration. +- **Glow, not shadow.** Live, draggable elements carry a colored **box-shadow halo** in their accent (`0 0 8–12px rgba(accent, .45)`): the XY-pad dot (cyan), the joystick knob (orange), slider thumbs. Drop-shadows (`--shadow-1/2`) appear only on floating chrome (drawers, popovers). Depth is communicated by the bg-step ladder, not elevation. +- **Radius.** Tight and consistent: `4px` (buttons, inputs, chips), `8px` (panels, pads, cards), `14px` (drawers), `999px` (pills/segmented controls). XY pad = 8px square; joystick = full circle. +- **Cards / panels.** `--bg-1` fill + 1px `--line` border + 8px radius + ~12–16px padding. No shadow. A header row is separated by a 1px bottom border; uppercase micro-labels sit above their values. +- **Hover.** Surfaces lighten one bg step (`--bg-2 → --bg-3`) and the border goes `--line → --line-strong`. Text-only controls brighten `--fg-mute → --fg`. ~120ms. +- **Press / active.** The selected pill/tab fills **solid orange with `--bg` text**. Active nav gets an orange border + orange text. Slider thumbs scale `1.15` on hover. No bounce. +- **Focus.** Accent border + a **3px orange glow ring** (`0 0 0 3px rgba(255,106,0,.3)`) — visible and on-brand, replacing the browser outline. +- **Motion.** One house easing `cubic-bezier(.25,.8,.35,1)`; durations `120 / 220 / 360ms`. Transitions are **fades and slides** (drawers slide in from the side). No spring, no bounce, no infinite decorative loops — the only continuous motion is *data* (a loss plot advancing, a live readout updating). +- **Transparency / blur.** Used sparingly: translucent pin fills on the joy-map; faint white grid lines on plots (`rgba(255,255,255,0.05)`). No backdrop-blur glass. +- **Layout.** Dense grids. The mode shell is a 4-row grid: header / body (input + map) / controls (axes + training) / status line. Hit targets ≥ 44px; primary training buttons are 48px tall. Everything is left-aligned and grid-gapped, never centered-flowing. +- **Imagery vibe.** Cool and synthetic — black background, neon-on-dark plots, pixelated heatmaps. No warmth, grain, or photography. + +--- + +## ICONOGRAPHY + +Manifold's iconography is **geometric and minimal — drawn from the control surfaces themselves**, not from a packaged icon set. + +- **No bundled icon font or SVG sprite** exists in the codebase. The product leans on three things, in order of preference: + 1. **Functional Unicode glyphs** rendered in the mono face — transport `▶ ⏹`, undo `↶`, arrows `→ ↔`, math `+ − ×`, middle-dot `·`. These match the terminal aesthetic and need no assets. + 2. **A few emoji as utility icons** in dense toolbars only — `🎤` mic, `⚙` settings/drawer, `📚` snapshot history. Treat these as placeholders for a future geometric set; never use emoji decoratively or in prose. + 3. **Canvas-drawn "live" iconography** — the real visual language is the *instrument itself*: the XY-pad crosshair + glowing dot, the joystick's dashed boundary + knob, the loss sparkline, the layer-norm bars, the heatmap. When you need to represent a Manifold concept, draw the control, don't reach for a metaphor icon. +- **Geometric system, if you must add icons.** Use a thin, single-weight, square-cornered line set. The closest CDN match is **[Lucide](https://lucide.dev)** (2px stroke, geometric, open) or **[Phosphor](https://phosphoricons.com)** (`regular` weight). This system **substitutes Lucide via CDN** for kit screens that need wayfinding icons (settings, close, chevrons) — flagged as a substitution; swap for a bespoke set when one exists. See `assets/icons/` for the curated subset and `assets/README.md`. +- **The logo / wordmark.** The only mark in the source is the text lockup **`MEMLNaut`** (bold, `--accent` orange) beside a dim lowercase `playground`, set in JetBrains Mono. There is no symbol/glyph logo. The brand name **Manifold** is set the same way: bold mono wordmark, optionally with the orange dot of a live control as the "o". See `assets/` for the recreated lockups. +- **Emoji policy:** functional-only, toolbar-only. Default to Unicode geometry or Lucide. Never emoji in headings, marketing, or body copy. + +--- + +## Index / manifest + +- `styles.css` — global entry (link this). `@import` manifest only. +- `tokens/` — `colors.css`, `typography.css`, `spacing.css`, `effects.css`, `fonts.css`, `base.css`. +- `assets/` — recreated wordmarks, curated Lucide icon subset, README. (See Caveats.) +- `guidelines/` — foundation specimen cards (Type / Colors / Spacing / Brand) shown on the Design System tab. +- `components/` — reusable React primitives (see list below) + per-group cards. +- `ui_kits/memlnaut/` — full-screen recreation of the MEMLNaut playground mode shell. Plus `ui_kits/memlnaut/console/` — the **Playground 2.0 Console** main view (full-bleed manifold + verdict cluster + right dock with depth drawers). +- `SKILL.md` — Agent-Skill front-matter for downloadable use. + +**Components** (`window.ManifoldDesignSystem_490915.*`): Button, PillToggle, Slider, Panel, Badge, Switch, StatusLine · XYPad, VirtualJoystick, ControlAxis · CurvePlot, Sparkline. + +**Caveats / open questions** — see the chat summary; the wordmark and icon set are recreations/substitutions pending real brand assets. diff --git a/docs/redesign/manifold-export/styles.css b/docs/redesign/manifold-export/styles.css new file mode 100644 index 0000000..2235239 --- /dev/null +++ b/docs/redesign/manifold-export/styles.css @@ -0,0 +1,12 @@ +/** + * Manifold Design System — global entry point. + * Consumers link THIS file. It is an @import manifest only; never put rules + * directly here. Everything reachable from these imports ships to consumers. + */ + +@import url('./tokens/fonts.css'); +@import url('./tokens/colors.css'); +@import url('./tokens/typography.css'); +@import url('./tokens/spacing.css'); +@import url('./tokens/effects.css'); +@import url('./tokens/base.css'); diff --git a/docs/redesign/manifold-export/tokens/base.css b/docs/redesign/manifold-export/tokens/base.css new file mode 100644 index 0000000..2d00f8f --- /dev/null +++ b/docs/redesign/manifold-export/tokens/base.css @@ -0,0 +1,51 @@ +/** + * Manifold — base element styles. + * Mirrors the playground's global resets so specimen cards and UI kits read + * like the real product even before a single component mounts. + */ + +* { + box-sizing: border-box; +} + +html, body { + margin: 0; + padding: 0; + background: var(--bg); + color: var(--fg); + font-family: var(--font-mono); + font-size: var(--fs-md); + line-height: var(--lh-normal); + -webkit-tap-highlight-color: transparent; + -webkit-font-smoothing: antialiased; +} + +a { + color: var(--accent-2); + text-decoration: none; +} +a:hover { + text-decoration: underline; +} + +code, kbd { + font-family: var(--font-mono); +} + +::selection { + background: var(--selection-bg); + color: var(--selection-text); +} + +/* Uppercase micro-label helper used across the system. */ +.mf-label { + font-size: var(--fs-xs); + color: var(--fg-mute); + text-transform: uppercase; + letter-spacing: var(--ls-label); +} + +/* Tabular numerals for any live readout. */ +.mf-num { + font-variant-numeric: tabular-nums; +} diff --git a/docs/redesign/manifold-export/tokens/colors.css b/docs/redesign/manifold-export/tokens/colors.css new file mode 100644 index 0000000..15576c4 --- /dev/null +++ b/docs/redesign/manifold-export/tokens/colors.css @@ -0,0 +1,72 @@ +/** + * Manifold — color tokens + * Dark terminal canvas, warm-orange primary, cool-cyan secondary. + * Ported from the MEMLNaut playground (src/styles/tokens.css) and extended + * with semantic aliases. + */ + +:root { + /* ---- Surfaces (dark, layered) ---- */ + --bg: #0d0d0d; /* app canvas */ + --bg-1: #141414; /* panel / card */ + --bg-2: #1c1c1c; /* raised control */ + --bg-3: #242424; /* hover / track */ + + /* ---- Foreground / text ---- */ + --fg: #e8e8e8; /* primary text */ + --fg-mute: #9a9a9a; /* secondary text / labels */ + --fg-dim: #5a5a5a; /* tertiary / disabled */ + + /* ---- Lines / borders ---- */ + --line: #2a2a2a; /* default 1px hairline */ + --line-strong: #3a3a3a; /* grid lines, dashed guides */ + + /* ---- Accents ---- */ + --accent: #ff6a00; /* warm primary — actions, focus, the live dot */ + --accent-2: #00ccff; /* cool secondary — data, plots, secondary dot */ + --accent-3: #ffa860; /* warm hover / tint */ + + /* ---- Semantic ---- */ + --good: #6bc26b; + --warn: #f5c45e; + --bad: #ef5b5b; + --info: #5b9eef; + + /* ---- Console 2.0 surface language ---- */ + --danger: #ff4466; /* the 2.0 verdict-perturb / destructive red */ + --glass: rgba(13, 13, 13, 0.65); /* frosted chrome over the manifold */ + --glass-line: rgba(255, 255, 255, 0.07); /* hairline on glass */ + + /* ---- Region pins (translucent map markers) ---- */ + --pin-1: rgba(255, 106, 0, 0.25); + --pin-2: rgba(0, 204, 255, 0.25); + --pin-3: rgba(180, 100, 255, 0.25); + --pin-4: rgba(80, 200, 120, 0.25); + --pin-5: rgba(255, 200, 80, 0.25); + + /* ---- Glow alphas (for box-shadow halos on live controls) ---- */ + --glow-accent: rgba(255, 106, 0, 0.45); + --glow-accent-2: rgba(0, 204, 255, 0.45); + --glow-focus: rgba(255, 106, 0, 0.30); + + /* ============ Semantic aliases ============ */ + --surface-app: var(--bg); + --surface-panel: var(--bg-1); + --surface-raised: var(--bg-2); + --surface-track: var(--bg-3); + + --text-primary: var(--fg); + --text-secondary: var(--fg-mute); + --text-tertiary: var(--fg-dim); + --text-accent: var(--accent); + --text-link: var(--accent-2); + + --border-default: var(--line); + --border-strong: var(--line-strong); + --border-focus: var(--accent); + + --action-primary: var(--accent); + --action-primary-text: var(--bg); + --selection-bg: var(--accent); + --selection-text: var(--bg); +} diff --git a/docs/redesign/manifold-export/tokens/effects.css b/docs/redesign/manifold-export/tokens/effects.css new file mode 100644 index 0000000..eba6148 --- /dev/null +++ b/docs/redesign/manifold-export/tokens/effects.css @@ -0,0 +1,31 @@ +/** + * Manifold — motion, shadow & glow tokens + * Manifold rarely uses drop shadows for depth; instead it uses *glow halos* + * on live, interactive elements (the dot on an XY pad, a slider thumb). + */ + +:root { + /* ---- Motion ---- */ + --ease: cubic-bezier(.25, .8, .35, 1); /* @kind other */ + --ease-out: cubic-bezier(.16, 1, .3, 1); /* @kind other */ + --ease-console: cubic-bezier(0.22, 1, 0.36, 1); /* @kind other */ /* the 2.0 Console drawer/chrome easing */ + --dur-fast: 120ms; /* @kind other */ + --dur-med: 220ms; /* @kind other */ + --dur-slow: 360ms; /* @kind other */ + + /* ---- Glow halos (the signature) ---- */ + --glow-sm: 0 0 8px var(--glow-accent); + --glow-md: 0 0 12px var(--glow-accent); + --glow-lg: 0 0 18px var(--glow-accent); + --glow-cyan: 0 0 10px var(--glow-accent-2); + --focus-ring: 0 0 0 3px var(--glow-focus); + + /* ---- Shadows (used sparingly: drawers, popovers) ---- */ + --shadow-1: 0 2px 8px rgba(0, 0, 0, 0.4); + --shadow-2: 0 8px 24px rgba(0, 0, 0, 0.5); + + /* ---- Borders ---- */ + --bw: 1px; /* default hairline */ + --border: var(--bw) solid var(--line); + --border-strong-rule: var(--bw) solid var(--line-strong); +} diff --git a/docs/redesign/manifold-export/tokens/fonts.css b/docs/redesign/manifold-export/tokens/fonts.css new file mode 100644 index 0000000..6a42a35 --- /dev/null +++ b/docs/redesign/manifold-export/tokens/fonts.css @@ -0,0 +1,11 @@ +/** + * Manifold — webfonts + * JetBrains Mono is the brand face. The original codebase referenced it by + * name without bundling binaries, so we load it from Google Fonts here. + * + * SUBSTITUTION NOTE: shipped via Google Fonts CDN (OFL licensed). To self-host, + * drop the .woff2 files in assets/fonts/ and replace this @import with + * local @font-face rules. + */ + +@import url('https://fonts.googleapis.com/css2?family=JetBrains+Mono:ital,wght@0,400;0,500;0,600;0,700;1,400&display=swap'); diff --git a/docs/redesign/manifold-export/tokens/spacing.css b/docs/redesign/manifold-export/tokens/spacing.css new file mode 100644 index 0000000..89754fd --- /dev/null +++ b/docs/redesign/manifold-export/tokens/spacing.css @@ -0,0 +1,34 @@ +/** + * Manifold — spacing, radius, layout, z-index + * Compact 4px-based scale (the playground is dense, instrument-panel UI). + */ + +:root { + /* ---- Spacing (px) ---- */ + --sp-0: 2px; + --sp-1: 4px; + --sp-2: 8px; + --sp-3: 12px; + --sp-4: 16px; + --sp-5: 24px; + --sp-6: 32px; + --sp-7: 48px; + --sp-8: 64px; + + /* ---- Radius ---- */ + --r-1: 4px; /* buttons, inputs, small chips */ + --r-2: 8px; /* panels, pads, cards */ + --r-3: 14px; /* large surfaces, drawers */ + --r-pill: 999px; + + /* ---- Z layers ---- */ + --z-bg: 0; /* @kind other */ + --z-content: 10; /* @kind other */ + --z-overlay: 100; /* @kind other */ + --z-drawer: 200; /* @kind other */ + --z-modal: 1000; /* @kind other */ + + /* ---- Control sizing ---- */ + --control-h: 48px; /* training buttons, large hit targets */ + --hit-min: 44px; /* minimum touch target */ +} diff --git a/docs/redesign/manifold-export/tokens/typography.css b/docs/redesign/manifold-export/tokens/typography.css new file mode 100644 index 0000000..be817e2 --- /dev/null +++ b/docs/redesign/manifold-export/tokens/typography.css @@ -0,0 +1,46 @@ +/** + * Manifold — typography tokens + * Monospace is the hero (terminal vibe). Sans is a quiet system fallback, + * used rarely for long-form prose. + */ + +:root { + /* ---- Families ---- */ + --font-mono: 'JetBrains Mono', 'Fira Code', 'SF Mono', Menlo, Consolas, monospace; + --font-sans: -apple-system, BlinkMacSystemFont, 'Segoe UI', system-ui, sans-serif; + + /* Hero family alias — Manifold reads almost entirely in mono. */ + --font-display: var(--font-mono); + --font-body: var(--font-mono); + --font-prose: var(--font-sans); + + /* ---- Sizes (px, fixed scale from the playground) ---- */ + --fs-xs: 11px; /* labels, captions, status */ + --fs-sm: 13px; /* secondary UI text */ + --fs-md: 15px; /* body / default */ + --fs-lg: 18px; /* mode titles, emphasis */ + --fs-xl: 24px; /* page titles */ + --fs-2xl: 34px; /* hero / display (specimen / marketing) */ + --fs-3xl: 48px; + + /* ---- Weights ---- */ + --fw-regular: 400; /* @kind font */ + --fw-medium: 500; /* @kind font */ + --fw-semibold: 600; /* @kind font */ + --fw-bold: 700; /* @kind font */ + + /* ---- Line heights ---- */ + --lh-tight: 1.1; /* @kind other */ + --lh-snug: 1.3; /* @kind other */ + --lh-normal: 1.5; /* @kind other */ + + /* ---- Letter spacing ---- */ + --ls-tight: -0.01em; /* @kind other */ + --ls-normal: 0; /* @kind other */ + --ls-label: 0.08em; /* @kind other */ + --ls-wide: 0.12em; /* @kind other */ + + /* ---- Semantic label style ---- */ + --label-transform: uppercase; /* @kind other */ + --label-spacing: var(--ls-label); /* @kind other */ +} diff --git a/docs/redesign/manifold-export/ui_kits/memlnaut/console/CompositeStage.jsx b/docs/redesign/manifold-export/ui_kits/memlnaut/console/CompositeStage.jsx new file mode 100644 index 0000000..9dc3b7b --- /dev/null +++ b/docs/redesign/manifold-export/ui_kits/memlnaut/console/CompositeStage.jsx @@ -0,0 +1,254 @@ +/* Console 2.0 — CompositeStage: ONE continuous view that becomes inputs-first, + outputs-first, or 50/50 by dragging a single divider. No discrete modes — the + layout is a single ratio `split` ∈ [0,1] (= the input's share of the width): + split → 1 inputs-first (output demotes to a slim readout list, then a minimap) + split = 0.5 dual / 50-50 + split → 0 outputs-first (input demotes to a pad, then a minimap) + Pull the seam ALL THE WAY to an edge and the small side snaps shut, popping + out as a draggable minimap rectangle that lives in any corner. Each panel + chooses its own representation from its MEASURED width, so it never becomes a + useless sliver — it demotes. Handle snaps to 0.14·0.33·0.5·0.66·0.86 with + light magnetism; presets tween the ratio. */ +function CompositeStage({ split, onSplit, mode, pos, onMove, noiseCap, pins, follow, onLongPress, params, values, onChange }) { + const ref = React.useRef(null); + const [size, setSize] = React.useState({ w: 0, h: 0 }); + const drag = React.useRef(false); + const splitRef = React.useRef(split); splitRef.current = split; + + const SNAPS = [0.14, 0.33, 0.5, 0.66, 0.86]; + const MAGNET = 0.026; + const SHUT = 0.1; // pull past this and the small side collapses to a minimap + + // which corner each minimap rests in (persisted) + const [inCorner, setInCorner] = React.useState(() => localStorage.getItem('mf-mm-incorner') || 'tl'); + const [outCorner, setOutCorner] = React.useState(() => localStorage.getItem('mf-mm-outcorner') || 'tl'); + React.useEffect(() => { localStorage.setItem('mf-mm-incorner', inCorner); }, [inCorner]); + React.useEffect(() => { localStorage.setItem('mf-mm-outcorner', outCorner); }, [outCorner]); + + React.useEffect(() => { + const el = ref.current; if (!el) return; + // Only commit non-zero measurements; some hosts lay out the flex child after + // mount, so the first read can be 0. Keep ResizeObserver AND a setTimeout + // kick loop (RO is withheld in some preview contexts) until width is real. + const read = () => { if (el.clientWidth) setSize({ w: el.clientWidth, h: el.clientHeight }); }; + const ro = new ResizeObserver(read); ro.observe(el); read(); + let t = null, k = 0; + const kick = () => { read(); if (!el.clientWidth && k++ < 80) t = setTimeout(kick, 40); }; + kick(); + return () => { ro.disconnect(); clearTimeout(t); }; + }, []); + + const setFromClientX = (clientX) => { + const el = ref.current; if (!el) return; + const r = el.getBoundingClientRect(); + let f = (clientX - r.left) / r.width; + f = Math.max(0, Math.min(1, f)); + if (f < SHUT) f = 0; // collapse input → minimap + else if (f > 1 - SHUT) f = 1; // collapse output → minimap + else for (const s of SNAPS) if (Math.abs(f - s) < MAGNET) { f = s; break; } + onSplit(f); + }; + const down = (e) => { drag.current = true; e.currentTarget.setPointerCapture?.(e.pointerId); setFromClientX(e.clientX); }; + const move = (e) => { if (drag.current) setFromClientX(e.clientX); }; + const up = (e) => { drag.current = false; e.currentTarget.releasePointerCapture?.(e.pointerId); }; + + const tweenTo = (target) => { + const start = splitRef.current, t0 = performance.now(), dur = 300; + const ease = (p) => 1 - Math.pow(1 - p, 3); + const step = () => { + const p = Math.min(1, (performance.now() - t0) / dur); + onSplit(start + (target - start) * ease(p)); + if (p < 1) requestAnimationFrame(step); + }; + requestAnimationFrame(step); + }; + + const { w, h } = size; + const knownW = w > 0; // until measured, default both sides to their hero tier + const collapsed = split <= 0.001 ? 'in' : split >= 0.999 ? 'out' : null; + const wIn = w * split, wOut = w * (1 - split); + const inTier = (!knownW || wIn >= 300) ? 'full' : 'pad'; + const outTier = (!knownW || wOut >= 230) ? 'field' : 'list'; + + const DS = window.ManifoldDesignSystem_490915; + const Pad = mode && mode.input === 'joystick' ? DS.VirtualJoystick : DS.XYPad; + const tag = (text, side) => ( + <div style={{ position: 'absolute', top: 10, [side]: 12, zIndex: 8, pointerEvents: 'none', + fontSize: 9, letterSpacing: '0.14em', color: 'var(--fg-dim)', fontFamily: 'var(--font-mono)' }}>{text}</div> + ); + + // ---- INPUT panel content by tier (non-collapsed) ---- + const renderInput = () => { + if (inTier === 'full') { + return <window.Manifold pos={pos} onMove={onMove} noiseCap={noiseCap} pins={pins} follow={follow} onLongPress={onLongPress} />; + } + const s = Math.max(88, Math.min(wIn - 28, h - 88)); + return ( + <div style={{ position: 'absolute', inset: 0, display: 'flex', flexDirection: 'column', alignItems: 'center', justifyContent: 'center', gap: 10 }}> + <Pad size={s} position={pos} onMove={(x, y) => onMove(x, y)} showGrid /> + <span style={{ fontSize: 10, color: 'var(--fg-mute)', fontFamily: 'var(--font-mono)', fontVariantNumeric: 'tabular-nums' }}> + {pos[0].toFixed(2)}, {pos[1].toFixed(2)} + </span> + </div> + ); + }; + + // ---- OUTPUT panel content by tier (non-collapsed) ---- + const GC = { formant: '--accent', pitch: '--accent-2', amp: '--good', filter: '--warn', fx: '--info', mod: '--accent-3' }; + const renderOutput = () => { + if (outTier === 'field') { + return <window.OutputStage params={params} values={values} onChange={onChange} compact={wOut < 440} />; + } + return ( + <div style={{ position: 'absolute', inset: 0, padding: '34px 10px 12px', display: 'flex', flexDirection: 'column', gap: 4, overflowY: 'auto' }}> + {params.map((p, i) => { + const eff = values[i] ?? 0, gc = `var(${GC[p.group] || '--accent'})`, dim = p.status === 'off'; + const set = (cx, el) => { const r = el.getBoundingClientRect(); onChange(i, { val: Math.max(0, Math.min(1, (cx - r.left) / r.width)) }); }; + return ( + <div key={i} style={{ display: 'flex', flexDirection: 'column', gap: 2, opacity: dim ? 0.5 : 1 }}> + <div style={{ display: 'flex', justifyContent: 'space-between', gap: 6, fontFamily: 'var(--font-mono)' }}> + <span style={{ fontSize: 9, color: 'var(--fg-mute)', overflow: 'hidden', whiteSpace: 'nowrap', textOverflow: 'ellipsis' }}>{p.name}</span> + <span style={{ fontSize: 9, color: 'var(--fg)', fontVariantNumeric: 'tabular-nums' }}>{eff.toFixed(2)}</span> + </div> + <div onPointerDown={(e) => { e.currentTarget.setPointerCapture?.(e.pointerId); e.currentTarget._d = true; set(e.clientX, e.currentTarget); }} + onPointerMove={(e) => { if (e.currentTarget._d) set(e.clientX, e.currentTarget); }} + onPointerUp={(e) => { e.currentTarget._d = false; e.currentTarget.releasePointerCapture?.(e.pointerId); }} + style={{ position: 'relative', height: 6, background: 'var(--bg-1)', border: '1px solid var(--line)', borderRadius: 999, cursor: 'ew-resize', touchAction: 'none' }}> + <div style={{ position: 'absolute', left: 0, top: 0, bottom: 0, width: `${eff * 100}%`, background: gc, opacity: 0.6, borderRadius: 999 }} /> + <div style={{ position: 'absolute', left: `${eff * 100}%`, top: -2, width: 2, height: 10, marginLeft: -1, background: gc, boxShadow: `0 0 6px ${gc}` }} /> + </div> + </div> + ); + })} + </div> + ); + }; + + // ---- minimap (the collapsed side as a draggable corner rectangle) ---- + const CORNERS = { tl: { top: 62, left: 14 }, tr: { top: 62, right: 14 }, bl: { bottom: 14, left: 14 }, br: { bottom: 14, right: 14 } }; + const [mmPos, setMmPos] = React.useState(null); // {x,y,side} while dragging + const mmRef = React.useRef({ side: null }); + const mmDown = (side) => (e) => { + const card = e.currentTarget.closest('[data-mm]'); if (!card || !ref.current) return; + e.currentTarget.setPointerCapture?.(e.pointerId); + const r = ref.current.getBoundingClientRect(), cr = card.getBoundingClientRect(); + mmRef.current = { side, dx: e.clientX - cr.left, dy: e.clientY - cr.top, cw: cr.width, ch: cr.height }; + setMmPos({ x: cr.left - r.left, y: cr.top - r.top, side }); + }; + const mmMove = (e) => { + const m = mmRef.current; if (!m.side || !ref.current) return; + const r = ref.current.getBoundingClientRect(); + const x = Math.max(8, Math.min(r.width - m.cw - 8, e.clientX - r.left - m.dx)); + const y = Math.max(8, Math.min(r.height - m.ch - 8, e.clientY - r.top - m.dy)); + setMmPos({ x, y, side: m.side }); + }; + const mmUp = (e) => { + const m = mmRef.current; if (!m.side || !ref.current) return; + const r = ref.current.getBoundingClientRect(), p = mmPos || { x: 0, y: 0 }; + const corner = (p.y + m.ch / 2 < r.height / 2 ? 't' : 'b') + (p.x + m.cw / 2 < r.width / 2 ? 'l' : 'r'); + (m.side === 'in' ? setInCorner : setOutCorner)(corner); + mmRef.current = { side: null }; setMmPos(null); + e.currentTarget.releasePointerCapture?.(e.pointerId); + }; + + const miniCard = (side, corner, body) => { + const dragging = mmPos && mmPos.side === side; + const place = dragging ? { left: mmPos.x, top: mmPos.y } : CORNERS[corner]; + const restore = () => tweenTo(0.5); + return ( + <div data-mm={side} style={{ position: 'absolute', zIndex: 65, ...place, + background: 'var(--glass)', backdropFilter: 'blur(12px)', WebkitBackdropFilter: 'blur(12px)', + border: '1px solid var(--glass-line)', borderRadius: 'var(--r-2)', boxShadow: 'var(--shadow-2)', + padding: 8, display: 'flex', flexDirection: 'column', gap: 6 }}> + {/* header = drag handle */} + <div onPointerDown={mmDown(side)} onPointerMove={mmMove} onPointerUp={mmUp} onPointerCancel={mmUp} + style={{ display: 'flex', alignItems: 'center', justifyContent: 'space-between', gap: 12, cursor: 'move', touchAction: 'none', userSelect: 'none' }}> + <span style={{ fontSize: 9, letterSpacing: '0.12em', color: 'var(--fg-mute)', fontFamily: 'var(--font-mono)' }}> + <span style={{ color: 'var(--fg-dim)' }}>⠿</span> {side === 'in' ? 'INPUT' : 'OUTPUT'} + </span> + <button type="button" onClick={restore} title="expand back to dual" + style={{ border: 'none', background: 'transparent', color: 'var(--fg-dim)', cursor: 'pointer', fontSize: 12, lineHeight: 1, padding: 0 }}>⤢</button> + </div> + {body} + </div> + ); + }; + const miniInput = () => miniCard('in', inCorner, ( + <Pad size={118} position={pos} onMove={(x, y) => onMove(x, y)} showGrid /> + )); + const miniOutput = () => miniCard('out', outCorner, ( + <div style={{ width: 168 }}> + <window.MiniMeters params={params} values={values} /> + </div> + )); + + // ---- divider geometry ---- + const pctIn = Math.round(split * 100); + const presetActive = Math.abs(split - 0.86) < 0.04 ? 'in' : Math.abs(split - 0.14) < 0.04 ? 'out' : Math.abs(split - 0.5) < 0.04 ? 'dual' : null; + const presetBtn = (id, label, target) => ( + <button type="button" onClick={() => tweenTo(target)} title={label} + style={{ fontSize: 9, letterSpacing: '0.08em', padding: '3px 9px', borderRadius: 'var(--r-pill)', cursor: 'pointer', fontFamily: 'var(--font-mono)', + border: `1px solid ${presetActive === id ? 'var(--accent)' : 'transparent'}`, + background: presetActive === id ? 'rgba(255,106,0,0.14)' : 'transparent', + color: presetActive === id ? 'var(--accent)' : 'var(--fg-dim)' }}>{label}</button> + ); + const handleLeft = split <= 0 ? '0%' : split >= 1 ? '100%' : `${split * 100}%`; + const handleMargin = split <= 0 ? 0 : split >= 1 ? -18 : -9; + + return ( + <div ref={ref} style={{ position: 'absolute', inset: 0, overflow: 'hidden' }}> + {collapsed === 'in' ? ( + <React.Fragment> + <div style={{ position: 'absolute', inset: 0 }}>{tag('OUTPUT', 'right')}{renderOutput()}</div> + {miniInput()} + </React.Fragment> + ) : collapsed === 'out' ? ( + <React.Fragment> + <div style={{ position: 'absolute', inset: 0 }}>{tag('INPUT', 'left')}{renderInput()}</div> + {miniOutput()} + </React.Fragment> + ) : ( + <React.Fragment> + <div style={{ position: 'absolute', top: 0, bottom: 0, left: 0, width: `${split * 100}%`, overflow: 'hidden' }}> + {tag('INPUT', 'left')}{renderInput()} + </div> + <div style={{ position: 'absolute', top: 0, bottom: 0, right: 0, width: `${(1 - split) * 100}%`, borderLeft: '1px solid var(--line)', overflow: 'hidden' }}> + {tag('OUTPUT', 'right')}{renderOutput()} + </div> + {SNAPS.map((s) => ( + <div key={s} style={{ position: 'absolute', top: 0, left: `${s * 100}%`, width: 1, height: 6, marginLeft: -0.5, + background: 'var(--line-strong)', opacity: Math.abs(split - s) < 0.012 ? 0 : 0.6, pointerEvents: 'none', zIndex: 14 }} /> + ))} + </React.Fragment> + )} + + {/* the divider handle — at the seam, or an edge tab when collapsed */} + <div onPointerDown={down} onPointerMove={move} onPointerUp={up} onPointerCancel={up} + onDoubleClick={() => tweenTo(presetActive === 'dual' ? 0.86 : presetActive === 'in' ? 0.14 : 0.5)} + title={collapsed ? 'pull to reveal' : 'drag to rebalance · double-click to cycle'} + style={{ position: 'absolute', top: 0, bottom: 0, left: handleLeft, width: 18, marginLeft: handleMargin, zIndex: 26, + cursor: 'col-resize', touchAction: 'none', display: 'flex', alignItems: 'center', justifyContent: 'center' }}> + <div style={{ position: 'absolute', top: 0, bottom: 0, left: collapsed === 'out' ? 'auto' : 9, right: collapsed === 'out' ? 9 : 'auto', + width: 1, background: 'var(--accent)', opacity: collapsed ? 0.5 : 0.35 }} /> + <div style={{ width: 6, height: 46, borderRadius: 999, background: 'var(--bg-3)', border: '1px solid var(--line-strong)', + display: 'flex', flexDirection: 'column', alignItems: 'center', justifyContent: 'center', gap: 3, boxShadow: '0 0 0 4px var(--bg)' }}> + {[0, 1, 2].map((k) => <span key={k} style={{ width: 2, height: 2, borderRadius: '50%', background: 'var(--fg-mute)' }} />)} + </div> + </div> + + {/* presets + ratio readout, top-centre */} + <div style={{ position: 'absolute', top: 10, left: '50%', transform: 'translateX(-50%)', zIndex: 16, + display: 'flex', alignItems: 'center', gap: 4, background: 'var(--glass)', backdropFilter: 'blur(10px)', WebkitBackdropFilter: 'blur(10px)', + border: '1px solid var(--glass-line)', borderRadius: 'var(--r-pill)', padding: '3px 5px' }}> + {presetBtn('in', 'input', 0.86)} + {presetBtn('dual', 'dual', 0.5)} + {presetBtn('out', 'output', 0.14)} + <span style={{ width: 1, height: 14, background: 'var(--glass-line)' }} /> + <span style={{ fontSize: 9, color: 'var(--fg-mute)', fontFamily: 'var(--font-mono)', fontVariantNumeric: 'tabular-nums', padding: '0 4px' }}> + in {pctIn} · out {100 - pctIn} + </span> + </div> + </div> + ); +} +window.CompositeStage = CompositeStage; diff --git a/docs/redesign/manifold-export/ui_kits/memlnaut/console/ConsoleApp.jsx b/docs/redesign/manifold-export/ui_kits/memlnaut/console/ConsoleApp.jsx new file mode 100644 index 0000000..2f99f1f --- /dev/null +++ b/docs/redesign/manifold-export/ui_kits/memlnaut/console/ConsoleApp.jsx @@ -0,0 +1,199 @@ +/* Console 2.0 — the reactive spine + layout. One input → ML → output flow that + every consumer (readout strip, manifold, health glow) reads. Pseudo-inference + stands in for the WASM MLP; the structure mirrors the rewrite plan. */ +const CA_DS = window.ManifoldDesignSystem_490915; + +/* ---- shared model (modes + inference) lives in model.jsx ---- */ +const MODES2 = window.MF_MODES; +const infer = window.MF_infer; +const seededGradient = window.MF_seededGradient; + +let SNAP_ID = 0; + +function ConsoleApp({ focus = 'in' }) { + const [modeId, setModeId] = React.useState('paf_synth'); + const mode = MODES2.find((m) => m.id === modeId); + const [params, setParams] = React.useState(() => mode.params.map((p) => ({ ...p }))); + const [pos, setPos] = React.useState([0.5, 0.5]); + const [seed, setSeed] = React.useState(0.4); + const [axes, setAxes] = React.useState({ boldness: 0.55, memory: 0.4, precision: 0.5 }); + const [preset, setPreset] = React.useState('Sculpt'); + const [noiseCap, setNoiseCap] = React.useState(0.12); + const [examples, setExamples] = React.useState(0); + const [addingExample, setAddingExample] = React.useState(false); + const [loss, setLoss] = React.useState([]); + const [busy, setBusy] = React.useState(false); + const [snapshots, setSnapshots] = React.useState([]); + const [ab, setAB] = React.useState('B'); + const [holdingA, setHoldingA] = React.useState(false); + const aRef = React.useRef(null); + const [spread, setSpread] = React.useState(false); + const [tame, setTame] = React.useState(0.85); + const [health, setHealth] = React.useState(0.8); + const [rev, setRev] = React.useState(1); + const [active, setActive] = React.useState('feel'); // auto-open Feel peek + const [depth, setDepth] = React.useState('peek'); + const [follow, setFollow] = React.useState(false); + const [split, setSplit] = React.useState(() => { + const v = parseFloat(localStorage.getItem('mf-composite-split')); + return Number.isFinite(v) ? Math.max(0, Math.min(1, v)) : 0.5; + }); + React.useEffect(() => { localStorage.setItem('mf-composite-split', String(split)); }, [split]); + const [stripPinned, setStripPinned] = React.useState(true); + const [firstSession, setFirstSession] = React.useState(true); + const [pins, setPins] = React.useState([]); + + // reset transient state on mode switch + React.useEffect(() => { + setParams(mode.params.map((p) => ({ ...p }))); setPos([0.5, 0.5]); setExamples(0); + setLoss([]); setSnapshots([]); setSeed(0.4); setFollow(false); setPins([]); + setActive('feel'); setDepth('peek'); + }, [modeId]); + + const effSeed = holdingA && aRef.current ? aRef.current.seed : seed; + const values = React.useMemo(() => infer(pos, effSeed, params, axes), [pos, effSeed, params, axes]); + const gradient = React.useMemo(() => seededGradient(rev), [rev]); + + const pushSnap = (tag) => setSnapshots((s) => [...s, { id: ++SNAP_ID, tag, noise: noiseCap, seed }].slice(-50)); + + const commit = () => { + setFirstSession(false); pushSnap('commit +'); setBusy(true); + setNoiseCap((n) => Math.max(0.02, n * 0.7)); + setHealth((h) => Math.min(1, h + 0.08)); + setRev((r) => r + 1); + setTimeout(() => { + setLoss((prev) => { const base = prev.length ? prev[prev.length - 1] : 0.5; return [...prev, Math.max(0.004, base * (0.8 + Math.random() * 0.08))].slice(-120); }); + setBusy(false); + }, 260); + }; + const perturb = () => { + setFirstSession(false); pushSnap('perturb −'); + setSeed((s) => s + (Math.random() - 0.5) * (noiseCap * 4 + 0.3)); + setNoiseCap((n) => Math.min(0.5, n + 0.06)); + setHealth((h) => Math.max(0.1, h - 0.06)); + setRev((r) => r + 1); + }; + const reroll = () => { setFirstSession(false); pushSnap('re-roll'); setSeed(Math.random() * 6); setNoiseCap(0.4); setHealth(0.5); setRev((r) => r + 1); }; + const undo = () => setSnapshots((s) => { + if (!s.length) return s; + const last = s[s.length - 1]; setSeed(last.seed); setNoiseCap(last.noise); setRev((r) => r + 1); + return s.slice(0, -1); + }); + const train = () => { setBusy(true); setTimeout(() => { setLoss((p) => { const b = p.length ? p[p.length - 1] : 0.5; return [...p, Math.max(0.004, b * 0.82)].slice(-120); }); setBusy(false); }, 260); }; + const addExample = () => { + if (!addingExample) { setAddingExample(true); return; } + setAddingExample(false); setExamples((e) => e + 1); pushSnap('example'); train(); + }; + + const setParam = (i, patch) => setParams((ps) => ps.map((p, j) => (j === i ? { ...p, ...patch } : p))); + const cycleStatus = (i) => setParams((ps) => ps.map((p, j) => (j === i ? { ...p, status: ({ off: 'fixed', fixed: 'live', live: 'off' })[p.status] } : p))); + + const toggleAB = () => { + if (ab === 'B') { aRef.current = { seed }; setAB('A'); } else { if (aRef.current) setSeed(aRef.current.seed); setAB('B'); } + }; + + // keyboard accelerators + React.useEffect(() => { + const onKey = (e) => { + if (e.target.tagName === 'INPUT') return; + const map = { '1': 'shape', '2': 'feel', '3': 'route', '4': 'health', '5': 'help' }; + if (map[e.key]) { setActive((a) => (a === map[e.key] ? null : map[e.key])); setDepth('peek'); } + else if (e.key === '\\') setDepth((d) => (d === 'full' ? 'peek' : 'full')); + else if (focus === 'composite' && e.key === '[') { e.preventDefault(); setSplit((s) => Math.max(0, s - 0.04)); } + else if (focus === 'composite' && e.key === ']') { e.preventDefault(); setSplit((s) => Math.min(1, s + 0.04)); } + else if (focus === 'composite' && (e.key === '=' || e.key === '0')) { e.preventDefault(); setSplit(0.5); } + else if (e.key === ' ' || e.key === 'ArrowUp') { e.preventDefault(); commit(); } + else if (e.key === 'ArrowDown') { e.preventDefault(); perturb(); } + else if (e.key.toLowerCase() === 'z') undo(); + }; + window.addEventListener('keydown', onKey); + return () => window.removeEventListener('keydown', onKey); + }); + + const ctx = { + modes: MODES2, modeId, setModeId, mode, + axes, setAxis: (k, v) => setAxes((s) => ({ ...s, [k]: v })), + preset, setPreset, offsetActive: preset !== 'Sculpt', + datasetCount: examples, loss, busy, addingExample, + onAddExample: addExample, onTrain: train, onClear: () => { setExamples(0); setLoss([]); }, + snapshots, onJump: (id) => { const s = snapshots.find((x) => x.id === id); if (s) { setSeed(s.seed); setNoiseCap(s.noise); setRev((r) => r + 1); } }, + params, cycleStatus, outputBackend: 'audio', setOutputBackend: () => {}, + health, gradient: gradient.norms, gradientStatus: gradient.status, weightsRevision: rev, + spread, setSpread, tame, setTame, noiseCap, setNoiseCap, + }; + + const healthColor = health > 0.66 ? 'rgba(107,194,107,' : health > 0.33 ? 'rgba(245,196,94,' : 'rgba(255,68,102,'; + const Input = mode.input; + + return ( + <div style={{ position: 'absolute', inset: 0, background: 'var(--bg)', overflow: 'hidden', fontFamily: 'var(--font-mono)' }}> + <style>{`@keyframes mfDrawerIn{from{transform:translateX(16px)}to{transform:translateX(0)}}`}</style> + + {/* ambient health glow at the screen edge */} + <div style={{ position: 'absolute', inset: 0, pointerEvents: 'none', zIndex: 25, + boxShadow: `inset 0 0 120px ${healthColor}${0.05 + (1 - health) * 0.12})`, transition: 'box-shadow var(--dur-slow) var(--ease-console)' }} /> + + {/* stage = manifold area (left of dock) */} + <div style={{ position: 'absolute', top: 0, left: 0, right: 48, bottom: 0 }}> + {/* readout strip (input-first only — promoted to the stage in output-first) */} + {focus === 'in' && (stripPinned || mode.cls !== 'Synth') && ( + <div style={{ position: 'absolute', top: 0, left: 0, right: 0, zIndex: 30 }}> + <window.ReadoutStrip params={params} values={values} onChange={setParam} pinned={stripPinned} onTogglePin={() => setStripPinned((p) => !p)} /> + </div> + )} + + {/* the stage: manifold (input-first) or output field (output-first) */} + <div style={{ position: 'absolute', top: focus === 'in' && stripPinned ? 76 : 0, left: 0, right: 0, bottom: 0 }}> + {focus === 'composite' ? ( + <window.CompositeStage split={split} onSplit={setSplit} mode={mode} + pos={pos} onMove={(x, y) => setPos([x, y])} noiseCap={noiseCap} pins={pins} follow={follow} + onLongPress={(p) => setPins((ps) => [...ps, { x: p[0], y: p[1], color: 'rgba(255,106,0,0.16)' }])} + params={params} values={values} onChange={setParam} /> + ) : focus === 'split' ? ( + <window.SplitStage pos={pos} onMove={(x, y) => setPos([x, y])} noiseCap={noiseCap} + pins={pins} follow={follow} + onLongPress={(p) => setPins((ps) => [...ps, { x: p[0], y: p[1], color: 'rgba(255,106,0,0.16)' }])} + params={params} values={values} onChange={setParam} /> + ) : focus === 'out' ? ( + <React.Fragment> + <window.OutputStage params={params} values={values} onChange={setParam} /> + <window.InputMini mode={mode} pos={pos} onMove={(x, y) => setPos([x, y])} noiseCap={noiseCap} corner="bottom-left" /> + </React.Fragment> + ) : ( + <window.Manifold pos={pos} onMove={(x, y) => setPos([x, y])} noiseCap={noiseCap} pins={pins} + frozen={false} follow={follow} + onLongPress={(p) => setPins((ps) => [...ps, { x: p[0], y: p[1], color: 'rgba(255,106,0,0.16)' }])} /> + )} + + {/* corner overlays */} + <div style={{ position: 'absolute', top: 12, left: 14, zIndex: 20, pointerEvents: 'none' }}> + <div style={{ display: 'flex', alignItems: 'baseline', gap: 8 }}> + <strong style={{ color: 'var(--accent)', fontSize: 'var(--fs-md)' }}>MEMLNaut</strong> + <span style={{ color: 'var(--fg-mute)', fontSize: 'var(--fs-sm)' }}>{mode.label}</span> + {mode.badge && <span style={{ fontSize: 9, color: 'var(--warn)', border: '1px solid var(--warn)', borderRadius: 'var(--r-pill)', padding: '1px 6px' }}>{mode.badge}</span>} + </div> + <div style={{ color: 'var(--fg-dim)', fontSize: 'var(--fs-xs)', marginTop: 2 }}>{mode.cls.toLowerCase()} · {mode.input} · {params.length} targets</div> + </div> + + <div style={{ position: 'absolute', bottom: 14, left: focus === 'out' ? 178 : 14, zIndex: 20, display: 'flex', gap: 8, fontFamily: 'var(--font-mono)' }}> + <button type="button" onClick={() => setFollow((f) => !f)} title="Follow mode (hands-free drift)" + style={{ fontSize: 'var(--fs-xs)', padding: '4px 10px', borderRadius: 'var(--r-pill)', cursor: 'pointer', + border: `1px solid ${follow ? 'var(--good)' : 'var(--glass-line)'}`, background: 'var(--glass)', + color: follow ? 'var(--good)' : 'var(--fg-mute)' }}>{follow ? '◉ follow' : '○ follow'}</button> + <span style={{ fontSize: 'var(--fs-xs)', color: 'var(--fg-dim)', alignSelf: 'center' }}> + input ({pos[0].toFixed(2)}, {pos[1].toFixed(2)}) · noise {noiseCap.toFixed(3)} {holdingA && <span style={{ color: 'var(--accent-2)' }}>· preview A</span>} + </span> + </div> + + <window.VerdictCluster onPerturb={perturb} onUndo={undo} onCommit={commit} onReroll={reroll} + canUndo={snapshots.length > 0} ab={ab} onToggleAB={toggleAB} onHoldA={setHoldingA} firstSession={firstSession} /> + + <window.AltitudeNav current="console" focus={focus} style={{ top: 'auto', bottom: 14, right: 14 }} /> + </div> + </div> + + <window.Dock ctx={ctx} active={active} setActive={setActive} depth={depth} setDepth={setDepth} /> + </div> + ); +} +window.ConsoleApp = ConsoleApp; diff --git a/docs/redesign/manifold-export/ui_kits/memlnaut/console/CurvePad.jsx b/docs/redesign/manifold-export/ui_kits/memlnaut/console/CurvePad.jsx new file mode 100644 index 0000000..859dad4 --- /dev/null +++ b/docs/redesign/manifold-export/ui_kits/memlnaut/console/CurvePad.jsx @@ -0,0 +1,59 @@ +/* Console 2.0 — CurvePad: a square response-curve plot. Click-and-vertical-drag + anywhere reshapes the curve (drag up = bulge / ease-out, down = ease-in); + `curve` is 0..1 where ~0.43 reads as linear. Mirrors the engine's applyCurve. */ +function CurvePad({ curve = 0.5, onChange, size = 116 }) { + const ref = React.useRef(null); + const drag = React.useRef({ active: false, startY: 0, startC: 0 }); + + React.useEffect(() => { + const cv = ref.current; if (!cv) return; + const dpr = window.devicePixelRatio || 1; + cv.width = size * dpr; cv.height = size * dpr; + const ctx = cv.getContext('2d'); + ctx.setTransform(dpr, 0, 0, dpr, 0, 0); + ctx.clearRect(0, 0, size, size); + // frame + grid + ctx.strokeStyle = 'rgba(255,255,255,0.06)'; ctx.lineWidth = 1; + ctx.strokeRect(0.5, 0.5, size - 1, size - 1); + for (const t of [0.25, 0.5, 0.75]) { + ctx.beginPath(); ctx.moveTo(t * size, 0); ctx.lineTo(t * size, size); ctx.stroke(); + ctx.beginPath(); ctx.moveTo(0, t * size); ctx.lineTo(size, t * size); ctx.stroke(); + } + // diagonal reference (linear) + ctx.strokeStyle = 'rgba(255,255,255,0.10)'; ctx.setLineDash([3, 3]); + ctx.beginPath(); ctx.moveTo(0, size); ctx.lineTo(size, 0); ctx.stroke(); ctx.setLineDash([]); + // the curve + const css = getComputedStyle(cv); + const accent = css.getPropertyValue('--accent').trim() || '#ff6a00'; + const e = 0.25 + curve * 1.75; + ctx.strokeStyle = accent; ctx.lineWidth = 2; ctx.beginPath(); + const pad = 3; + for (let p = 0; p <= 80; p++) { + const xv = p / 80, yv = Math.pow(xv, e); + const px = pad + xv * (size - 2 * pad), py = (size - pad) - yv * (size - 2 * pad); + p === 0 ? ctx.moveTo(px, py) : ctx.lineTo(px, py); + } + ctx.stroke(); + }, [curve, size]); + + const down = (e) => { e.currentTarget.setPointerCapture?.(e.pointerId); drag.current = { active: true, startY: e.clientY, startC: curve }; }; + const move = (e) => { + const d = drag.current; if (!d.active) return; + const dc = (e.clientY - d.startY) / size; // drag up (negative dy) → lower c → bulge up + onChange && onChange(Math.max(0, Math.min(1, d.startC + dc))); + }; + const up = (e) => { drag.current.active = false; e.currentTarget.releasePointerCapture?.(e.pointerId); }; + + return ( + <div style={{ display: 'flex', flexDirection: 'column', gap: 4 }}> + <div style={{ display: 'flex', justifyContent: 'space-between' }}> + <span style={{ fontSize: 10, color: 'var(--fg-mute)', textTransform: 'uppercase', letterSpacing: '0.08em' }}>curve</span> + <span style={{ fontSize: 10, color: 'var(--fg-dim)', fontVariantNumeric: 'tabular-nums' }}>{(0.25 + curve * 1.75).toFixed(2)}</span> + </div> + <canvas ref={ref} onPointerDown={down} onPointerMove={move} onPointerUp={up} onPointerCancel={up} + title="Drag vertically to reshape" + style={{ width: size, height: size, display: 'block', cursor: 'ns-resize', background: 'var(--bg-1)', border: '1px solid var(--line)', borderRadius: 'var(--r-1)', touchAction: 'none' }} /> + </div> + ); +} +window.CurvePad = CurvePad; diff --git a/docs/redesign/manifold-export/ui_kits/memlnaut/console/Dock.jsx b/docs/redesign/manifold-export/ui_kits/memlnaut/console/Dock.jsx new file mode 100644 index 0000000..4e0da14 --- /dev/null +++ b/docs/redesign/manifold-export/ui_kits/memlnaut/console/Dock.jsx @@ -0,0 +1,118 @@ +/* Console 2.0 — The Console: right-edge 48px dock rail + one mutually-exclusive + drawer with three depth states (Peek 320 / Expand 520 / Full modal). */ +function ModeSwitcher({ modes, modeId, setModeId }) { + const [open, setOpen] = React.useState(false); + const active = modes.find((m) => m.id === modeId) || modes[0]; + const classes = ['Synth', 'Sequencer', 'Controller', 'Visual']; + return ( + <div style={{ position: 'relative' }}> + <button type="button" onClick={() => setOpen((o) => !o)} title={`Mode: ${active.label}`} + style={{ width: 40, height: 40, borderRadius: 'var(--r-2)', border: '1px solid var(--accent)', + background: 'rgba(255,106,0,0.12)', color: 'var(--accent)', cursor: 'pointer', fontSize: 18, + display: 'flex', alignItems: 'center', justifyContent: 'center' }}>{active.glyph || '⊞'}</button> + {open && ( + <div style={{ position: 'absolute', top: 0, right: 'calc(100% + 8px)', width: 240, background: 'var(--glass)', + backdropFilter: 'blur(14px)', WebkitBackdropFilter: 'blur(14px)', border: '1px solid var(--glass-line)', + borderRadius: 'var(--r-2)', boxShadow: 'var(--shadow-2)', padding: 6, zIndex: 80 }}> + {classes.map((cls) => { + const items = modes.filter((m) => m.cls === cls); + if (!items.length) return null; + return ( + <div key={cls}> + <div style={{ fontSize: 10, color: 'var(--fg-dim)', textTransform: 'uppercase', letterSpacing: '0.1em', padding: '6px 8px 2px' }}>{cls}</div> + {items.map((m) => ( + <button key={m.id} type="button" disabled={m.placeholder} + onClick={() => { if (m.placeholder) return; setModeId(m.id); setOpen(false); }} + style={{ display: 'flex', alignItems: 'center', gap: 8, width: '100%', textAlign: 'left', + background: m.id === modeId ? 'var(--bg-3)' : 'transparent', border: 0, borderRadius: 'var(--r-1)', + padding: '6px 8px', cursor: m.placeholder ? 'not-allowed' : 'pointer', opacity: m.placeholder ? 0.4 : 1, + fontFamily: 'var(--font-mono)', fontSize: 'var(--fs-sm)', color: m.id === modeId ? 'var(--accent)' : 'var(--fg)' }}> + <span style={{ width: 16, textAlign: 'center' }}>{m.glyph || '·'}</span>{m.label} + {m.badge && <span style={{ marginLeft: 'auto', fontSize: 9, color: 'var(--warn)' }}>{m.badge}</span>} + </button> + ))} + </div> + ); + })} + </div> + )} + </div> + ); +} + +function Dock({ ctx, active, setActive, depth, setDepth }) { + const sections = window.DRAWERS; + const order = ['shape', 'feel', 'route', 'health', 'help']; + + const iconBtn = (key) => { + const s = sections[key]; + const on = active === key; + return ( + <button key={key} type="button" title={s.label} onClick={() => { setActive(on ? null : key); setDepth('peek'); }} + style={{ position: 'relative', width: 40, height: 40, borderRadius: 'var(--r-2)', cursor: 'pointer', + border: `1px solid ${on ? 'var(--accent)' : 'transparent'}`, + background: on ? 'rgba(255,106,0,0.14)' : 'transparent', color: on ? 'var(--accent)' : 'var(--fg-mute)', + fontSize: 18, display: 'flex', alignItems: 'center', justifyContent: 'center', + transition: 'background var(--dur-fast), color var(--dur-fast)' }} + onMouseEnter={(e) => { if (!on) e.currentTarget.style.color = 'var(--fg)'; }} + onMouseLeave={(e) => { if (!on) e.currentTarget.style.color = 'var(--fg-mute)'; }}> + {s.icon} + {key === 'feel' && active !== 'feel' && ( + <span style={{ position: 'absolute', bottom: 4, left: 6, right: 6, height: 2, borderRadius: 2, + background: `linear-gradient(90deg, var(--accent) ${ctx.axes.boldness * 100}%, var(--line) 0)` }} /> + )} + </button> + ); + }; + + const width = depth === 'expand' ? 520 : 320; + const full = depth === 'full'; + const section = active ? sections[active] : null; + + return ( + <React.Fragment> + {/* drawer */} + {section && ( + <aside style={full ? { + position: 'fixed', inset: 0, zIndex: 90, background: 'var(--glass)', backdropFilter: 'blur(16px)', WebkitBackdropFilter: 'blur(16px)', + padding: 'var(--sp-6)', display: 'flex', flexDirection: 'column', gap: 'var(--sp-3)', + } : { + position: 'absolute', top: 0, right: 48, bottom: 0, width, zIndex: 35, + background: 'var(--glass)', backdropFilter: 'blur(16px)', WebkitBackdropFilter: 'blur(16px)', + borderLeft: '1px solid var(--glass-line)', boxShadow: '-8px 0 24px rgba(0,0,0,0.4)', + padding: 'var(--sp-4)', display: 'flex', flexDirection: 'column', gap: 'var(--sp-2)', overflow: 'auto', + animation: 'mfDrawerIn var(--dur-med) var(--ease-console)', + }}> + <header style={{ display: 'flex', alignItems: 'center', gap: 'var(--sp-2)', borderBottom: '1px solid var(--glass-line)', paddingBottom: 'var(--sp-2)' }}> + <span style={{ fontSize: 'var(--fs-md)', color: 'var(--accent)' }}>{section.icon}</span> + <h3 style={{ margin: 0, fontSize: 'var(--fs-md)', color: 'var(--fg)' }}>{section.label}</h3> + <span style={{ fontSize: 10, color: 'var(--fg-dim)', textTransform: 'uppercase', letterSpacing: '0.1em', marginLeft: 4 }}>{depth}</span> + <div style={{ marginLeft: 'auto', display: 'flex', gap: 4 }}> + {!full && ( + <button type="button" title={depth === 'peek' ? 'More' : 'Peek'} onClick={() => setDepth(depth === 'peek' ? 'expand' : 'peek')} + style={dockMini}>⤢</button> + )} + <button type="button" title="Open full" onClick={() => setDepth(full ? 'peek' : 'full')} style={dockMini}>{full ? '⤡' : '⤢⤢'}</button> + <button type="button" title="Close" onClick={() => setActive(null)} style={dockMini}>✕</button> + </div> + </header> + <div style={{ display: 'flex', flexDirection: 'column', gap: 'var(--sp-2)', flex: 1, ...(full ? { maxWidth: 720 } : {}) }}> + {section.render(ctx, depth)} + </div> + </aside> + )} + + {/* rail */} + <nav style={{ position: 'absolute', top: 0, right: 0, bottom: 0, width: 48, zIndex: 36, + background: 'var(--glass)', backdropFilter: 'blur(12px)', WebkitBackdropFilter: 'blur(12px)', + borderLeft: '1px solid var(--glass-line)', display: 'flex', flexDirection: 'column', alignItems: 'center', + gap: 'var(--sp-2)', padding: '8px 0' }}> + <ModeSwitcher modes={ctx.modes} modeId={ctx.modeId} setModeId={ctx.setModeId} /> + <div style={{ width: 24, height: 1, background: 'var(--glass-line)', margin: '2px 0' }} /> + {order.map(iconBtn)} + </nav> + </React.Fragment> + ); +} +const dockMini = { width: 26, height: 24, borderRadius: 'var(--r-1)', border: '1px solid var(--line)', background: 'var(--bg-2)', color: 'var(--fg-mute)', cursor: 'pointer', fontSize: 11, lineHeight: 1 }; +window.Dock = Dock; diff --git a/docs/redesign/manifold-export/ui_kits/memlnaut/console/Drawers.jsx b/docs/redesign/manifold-export/ui_kits/memlnaut/console/Drawers.jsx new file mode 100644 index 0000000..f1b8c03 --- /dev/null +++ b/docs/redesign/manifold-export/ui_kits/memlnaut/console/Drawers.jsx @@ -0,0 +1,223 @@ +/* Console 2.0 — Drawer contents. Each renderer takes (ctx, depth) where depth + is 'peek' | 'expand' | 'full'; what shows is gated by depth (the schema-tier + disclosure mechanism, simulated here). + NOTE: these are the PLACEHOLDER drawers. The real dock contents are specified + in docs/redesign/dock-spec.md (six drawers: Learning-Behaviour / Inputs / + Outputs-Routing / Powerful Synth Engine / Particle-Visual / Help). */ +const DRW_DS = window.ManifoldDesignSystem_490915; + +function Chip({ children, tone }) { + return ( + <span style={{ + fontSize: 10, fontFamily: 'var(--font-mono)', textTransform: 'uppercase', letterSpacing: '0.06em', + color: tone || 'var(--fg-mute)', background: 'var(--bg-2)', border: '1px solid var(--line)', + borderRadius: 'var(--r-pill)', padding: '2px 8px', + }}>{children}</span> + ); +} +function SectionLabel({ children }) { + return <div style={{ fontSize: 10, color: 'var(--fg-dim)', textTransform: 'uppercase', letterSpacing: '0.1em', marginTop: 'var(--sp-2)' }}>{children}</div>; +} + +const PRESETS = ['Sculpt', 'Wander', 'Lock-in', 'Chaos', 'Glide', 'Pin & probe']; + +function FeelDrawer(ctx, depth) { + const { ControlAxis, PillToggle } = DRW_DS; + const axes = [ + { key: 'boldness', label: 'Boldness', endpoints: ['Caution', 'Bold'], accent: 'var(--accent)' }, + { key: 'memory', label: 'Memory', endpoints: ['Amnesia', 'Elephant'], accent: 'var(--accent-2)' }, + { key: 'precision', label: 'Precision', endpoints: ['Raw', 'Precise'], accent: 'var(--accent)' }, + ]; + return ( + <React.Fragment> + <div style={{ display: 'flex', gap: 6, flexWrap: 'wrap' }}> + <Chip tone="var(--accent)">3 axes</Chip> + <Chip>preset · {ctx.preset}</Chip> + {ctx.offsetActive && <Chip tone="var(--accent-2)">offset active</Chip>} + </div> + <SectionLabel>Compound axes</SectionLabel> + {axes.map((a) => ( + <ControlAxis key={a.key} label={a.label} endpoints={a.endpoints} accent={a.accent} + value={ctx.axes[a.key]} onChange={(v) => ctx.setAxis(a.key, v)} + preset={a.key === 'boldness' ? (ctx.axes.boldness > 0.66 ? 'explore' : 'steady') : undefined} /> + ))} + <SectionLabel>Control presets</SectionLabel> + <div style={{ display: 'flex', gap: 6, flexWrap: 'wrap' }}> + {PRESETS.map((p) => ( + <button key={p} type="button" onClick={() => ctx.setPreset(p)} + style={{ + padding: '5px 10px', fontSize: 'var(--fs-xs)', fontFamily: 'var(--font-mono)', cursor: 'pointer', + borderRadius: 'var(--r-pill)', border: `1px solid ${ctx.preset === p ? 'var(--accent)' : 'var(--line)'}`, + background: ctx.preset === p ? 'rgba(255,106,0,0.15)' : 'var(--bg-2)', + color: ctx.preset === p ? 'var(--accent)' : 'var(--fg-mute)', + }}>{p}</button> + ))} + </div> + {depth !== 'peek' && ( + <React.Fragment> + <SectionLabel>Trim-pot offsets</SectionLabel> + <p style={{ fontSize: 'var(--fs-xs)', color: 'var(--fg-dim)', margin: 0, lineHeight: 1.6 }}> + Axis tweaks layer as offsets over the active preset. Double-tap an axis to re-link. An <span style={{ color: 'var(--accent-2)' }}>offset-active</span> dot marks divergence. + </p> + </React.Fragment> + )} + </React.Fragment> + ); +} + +function ShapeDrawer(ctx, depth) { + const { Button, Sparkline, Badge } = DRW_DS; + return ( + <React.Fragment> + <div style={{ display: 'flex', gap: 6, flexWrap: 'wrap' }}> + <Chip tone="var(--accent)">examples {ctx.datasetCount}</Chip> + <Chip>loss {ctx.loss.length ? ctx.loss[ctx.loss.length - 1].toExponential(1) : '—'}</Chip> + {ctx.busy && <Chip tone="var(--accent-2)">training…</Chip>} + </div> + <SectionLabel>Reinforcement (default)</SectionLabel> + <p style={{ fontSize: 'var(--fs-xs)', color: 'var(--fg-dim)', margin: 0, lineHeight: 1.6 }}> + Use the + / − verdicts on the manifold. Every verdict auto-snapshots first — undo is consequence-free. + </p> + <SectionLabel>Examples (IML)</SectionLabel> + <div style={{ display: 'flex', gap: 6, flexWrap: 'wrap' }}> + <Button size="sm" variant={ctx.addingExample ? 'primary' : 'secondary'} onClick={ctx.onAddExample}> + {ctx.addingExample ? '② store target' : '① add example'} + </Button> + <Button size="sm" onClick={ctx.onTrain} disabled={ctx.busy}>Train</Button> + <Button size="sm" variant="ghost" onClick={ctx.onClear}>clear</Button> + </div> + <SectionLabel>Loss history</SectionLabel> + <Sparkline data={ctx.loss.length ? ctx.loss : [0.5]} log width={depth === 'peek' ? 284 : 480} height={56} /> + {depth !== 'peek' && ( + <React.Fragment> + <SectionLabel>Snapshot DAG · {ctx.snapshots.length}</SectionLabel> + <ul style={{ listStyle: 'none', margin: 0, padding: 0, display: 'flex', flexDirection: 'column', gap: 4, maxHeight: 160, overflow: 'auto' }}> + {ctx.snapshots.slice().reverse().map((s, idx) => ( + <li key={s.id}> + <button type="button" onClick={() => ctx.onJump(s.id)} style={{ + width: '100%', textAlign: 'left', display: 'flex', justifyContent: 'space-between', + background: idx === 0 ? 'var(--bg-3)' : 'var(--bg-2)', border: '1px solid var(--line)', + borderRadius: 'var(--r-1)', padding: '4px 8px', cursor: 'pointer', fontFamily: 'var(--font-mono)', fontSize: 'var(--fs-xs)', + }}> + <span style={{ color: idx === 0 ? 'var(--accent)' : 'var(--fg)' }}>{s.tag}</span> + <span style={{ color: 'var(--fg-dim)' }}>noise {s.noise.toFixed(2)}</span> + </button> + </li> + ))} + </ul> + </React.Fragment> + )} + </React.Fragment> + ); +} + +const STATUS_CYCLE = { off: 'fixed', fixed: 'live', live: 'off' }; +const STATUS_COLOR = { off: 'var(--fg-dim)', fixed: 'var(--accent-2)', live: 'var(--accent)' }; + +function RouteDrawer(ctx, depth) { + const { PillToggle, Badge } = DRW_DS; + const counts = ctx.params.reduce((a, p) => { a[p.status] = (a[p.status] || 0) + 1; return a; }, {}); + const rows = depth === 'peek' ? ctx.params.slice(0, 6) : ctx.params; + return ( + <React.Fragment> + <div style={{ display: 'flex', gap: 6, flexWrap: 'wrap' }}> + <Chip>sources {ctx.mode.inN}/48</Chip> + <Chip>targets {ctx.params.length}/{ctx.params.length}</Chip> + <Chip tone="var(--accent)">live {counts.live || 0}</Chip> + <Chip tone="var(--accent-2)">fixed {counts.fixed || 0}</Chip> + </div> + <SectionLabel>Input source</SectionLabel> + <Badge tone="info">{ctx.mode.input === 'joystick' ? 'joystick' : ctx.mode.input === 'audio_in' ? 'mic (1-input)' : 'xy pad'}</Badge> + <SectionLabel>Output backend</SectionLabel> + <PillToggle value={ctx.outputBackend} onChange={ctx.setOutputBackend} + options={[{ value: 'audio', label: 'Audio' }, { value: 'midi', label: 'MIDI' }, { value: 'osc', label: 'OSC' }, { value: 'cv', label: 'CV' }]} /> + <SectionLabel>Control points · off / fixed / live</SectionLabel> + <div style={{ display: 'flex', flexDirection: 'column', gap: 3, maxHeight: depth === 'peek' ? 150 : 320, overflow: 'auto' }}> + {rows.map((p, i) => ( + <div key={i} style={{ display: 'flex', alignItems: 'center', gap: 'var(--sp-2)', background: 'var(--bg-2)', border: '1px solid var(--line)', borderRadius: 'var(--r-1)', padding: '4px 8px' }}> + <span style={{ flex: 1, fontSize: 'var(--fs-xs)', color: 'var(--fg)', overflow: 'hidden', textOverflow: 'ellipsis', whiteSpace: 'nowrap' }}>{p.name}</span> + <span style={{ fontSize: 10, color: 'var(--fg-dim)' }}>{p.group}</span> + <button type="button" onClick={() => ctx.cycleStatus(ctx.params.indexOf(p))} + style={{ width: 52, fontSize: 10, fontFamily: 'var(--font-mono)', textTransform: 'uppercase', letterSpacing: '0.06em', + border: `1px solid ${STATUS_COLOR[p.status]}`, color: STATUS_COLOR[p.status], background: 'transparent', + borderRadius: 'var(--r-1)', padding: '2px 4px', cursor: 'pointer' }}>{p.status}</button> + </div> + ))} + </div> + {depth === 'full' && ( + <p style={{ fontSize: 'var(--fs-xs)', color: 'var(--fg-dim)', margin: '8px 0 0', lineHeight: 1.6 }}> + Full tier hosts the Emitters × Targets matrix + per-cell editor + staged-unlock opt-in (built for the modular mega-mode). + </p> + )} + </React.Fragment> + ); +} + +function GradientBars({ norms, status }) { + const COLOR = { vanishing: 'var(--info)', exploding: 'var(--danger)', converged: 'var(--fg-mute)', healthy: 'var(--good)' }; + const max = Math.max(...norms, 0.0001); + return ( + <div style={{ display: 'flex', alignItems: 'flex-end', gap: 3, height: 56, background: 'var(--bg-1)', border: '1px solid var(--line)', borderRadius: 'var(--r-2)', padding: 6 }}> + {norms.map((n, i) => ( + <div key={i} style={{ flex: 1, display: 'flex', flexDirection: 'column', alignItems: 'center', gap: 2, height: '100%', justifyContent: 'flex-end' }}> + <div style={{ width: '100%', height: `${(n / max) * 100}%`, background: COLOR[status[i]] || 'var(--good)', borderRadius: 1 }} /> + <span style={{ fontSize: 8, color: 'var(--fg-dim)' }}>L{i}</span> + </div> + ))} + </div> + ); +} + +function HealthDrawer(ctx, depth) { + const { Sparkline, Switch, Slider, Badge } = DRW_DS; + const healthTone = ctx.health > 0.66 ? 'good' : ctx.health > 0.33 ? 'warn' : 'bad'; + return ( + <React.Fragment> + <div style={{ display: 'flex', gap: 6, flexWrap: 'wrap', alignItems: 'center' }}> + <Badge tone={healthTone} dot>{healthTone === 'good' ? 'healthy' : healthTone === 'warn' ? 'drifting' : 'unstable'}</Badge> + <Chip>rev {ctx.weightsRevision}</Chip> + </div> + <SectionLabel>Loss history</SectionLabel> + <Sparkline data={ctx.loss.length ? ctx.loss : [0.5]} log width={depth === 'peek' ? 284 : 480} height={56} /> + <SectionLabel>Gradient flow · per layer</SectionLabel> + <GradientBars norms={ctx.gradient} status={ctx.gradientStatus} /> + {depth !== 'peek' && ( + <React.Fragment> + <SectionLabel>Lab controls</SectionLabel> + <Switch checked={ctx.spread} onChange={ctx.setSpread} label="spread (Xavier regime)" /> + <Slider label="tame · output limiter" value={ctx.tame} min={0} max={1} step={0.01} onChange={ctx.setTame} /> + <Slider label="noise cap" value={ctx.noiseCap} min={0} max={0.5} step={0.01} onChange={ctx.setNoiseCap} /> + </React.Fragment> + )} + </React.Fragment> + ); +} + +const KEYS = [['1–5', 'open drawers'], ['\\', 'full depth'], ['space / ↑', 'commit +'], ['↓', 'perturb −'], ['z', 'undo'], ['double-click dot', 'follow mode']]; +function HelpDrawer(ctx, depth) { + return ( + <React.Fragment> + <SectionLabel>Keyboard</SectionLabel> + <div style={{ display: 'flex', flexDirection: 'column', gap: 4 }}> + {KEYS.map(([k, v]) => ( + <div key={k} style={{ display: 'flex', justifyContent: 'space-between', fontSize: 'var(--fs-xs)' }}> + <kbd style={{ background: 'var(--bg-2)', border: '1px solid var(--line)', borderRadius: 'var(--r-1)', padding: '1px 6px', color: 'var(--accent)' }}>{k}</kbd> + <span style={{ color: 'var(--fg-mute)' }}>{v}</span> + </div> + ))} + </div> + <SectionLabel>The loop</SectionLabel> + <p style={{ fontSize: 'var(--fs-xs)', color: 'var(--fg-dim)', margin: 0, lineHeight: 1.7 }}> + Drag the manifold to explore. Hear something good → + to settle. Wrong → − to perturb. Went too far → ↺ undo. The screen reveals exactly as much machinery as you reach for. + </p> + </React.Fragment> + ); +} + +window.DRAWERS = { + shape: { icon: '◇', label: 'Shape', render: ShapeDrawer }, + feel: { icon: '◎', label: 'Feel', render: FeelDrawer }, + route: { icon: '⇄', label: 'Route', render: RouteDrawer }, + health: { icon: '♥', label: 'Health', render: HealthDrawer }, + help: { icon: '?', label: 'Help', render: HelpDrawer }, +}; diff --git a/docs/redesign/manifold-export/ui_kits/memlnaut/console/InputMini.jsx b/docs/redesign/manifold-export/ui_kits/memlnaut/console/InputMini.jsx new file mode 100644 index 0000000..5a8695d --- /dev/null +++ b/docs/redesign/manifold-export/ui_kits/memlnaut/console/InputMini.jsx @@ -0,0 +1,25 @@ +/* Console 2.0 — InputMini: the input demoted to a small secondary control for + the output-first views. A compact joy-map / XY pad in a glass card; still + drives inference, but no longer the hero. */ +function InputMini({ mode, pos, onMove, noiseCap, size = 132, corner = 'bottom-left' }) { + const DS = window.ManifoldDesignSystem_490915; + const Pad = mode.input === 'joystick' ? DS.VirtualJoystick : DS.XYPad; + const place = { + 'bottom-left': { bottom: 14, left: 14 }, + 'bottom-right': { bottom: 14, right: 14 }, + 'top-left': { top: 14, left: 14 }, + }[corner]; + return ( + <div style={{ position: 'absolute', zIndex: 22, ...place, + background: 'var(--glass)', backdropFilter: 'blur(12px)', WebkitBackdropFilter: 'blur(12px)', + border: '1px solid var(--glass-line)', borderRadius: 'var(--r-2)', padding: 'var(--sp-2)', + display: 'flex', flexDirection: 'column', gap: 6 }}> + <div style={{ display: 'flex', justifyContent: 'space-between', alignItems: 'center', gap: 12 }}> + <span style={{ fontSize: 10, color: 'var(--fg-dim)', textTransform: 'uppercase', letterSpacing: '0.1em' }}>input · {mode.input === 'joystick' ? 'joy' : 'xy'}</span> + <span style={{ fontSize: 10, color: 'var(--fg-mute)', fontVariantNumeric: 'tabular-nums' }}>{pos[0].toFixed(2)},{pos[1].toFixed(2)}</span> + </div> + <Pad size={size} position={pos} onMove={(x, y) => onMove(x, y)} showGrid /> + </div> + ); +} +window.InputMini = InputMini; diff --git a/docs/redesign/manifold-export/ui_kits/memlnaut/console/Manifold.jsx b/docs/redesign/manifold-export/ui_kits/memlnaut/console/Manifold.jsx new file mode 100644 index 0000000..aa9a404 --- /dev/null +++ b/docs/redesign/manifold-export/ui_kits/memlnaut/console/Manifold.jsx @@ -0,0 +1,163 @@ +/* Console 2.0 — The Manifold: full-bleed input surface + joy-map visualisation. + Pointer-down anywhere drives the joystick dot; a ~5s vanishing trail fades + behind it; noise rings breathe at the current cap; pins overlay regions. */ +function Manifold({ pos, onMove, noiseCap = 0.1, pins = [], frozen = false, follow = false, onLongPress }) { + const wrapRef = React.useRef(null); + const canvasRef = React.useRef(null); + const stateRef = React.useRef({ pos, noiseCap, pins, frozen, follow }); + const trailRef = React.useRef([]); // {x,y,t} + const draggingRef = React.useRef(false); + const driftRef = React.useRef({ vx: 0.0011, vy: 0.0008 }); + const lpTimer = React.useRef(null); + + stateRef.current = { pos, noiseCap, pins, frozen, follow }; + + // push trail point whenever pos changes + React.useEffect(() => { + trailRef.current.push({ x: pos[0], y: pos[1], t: performance.now() }); + if (trailRef.current.length > 240) trailRef.current.shift(); + }, [pos[0], pos[1]]); + + const setFromEvent = (e) => { + const el = wrapRef.current; if (!el) return; + const r = el.getBoundingClientRect(); + const x = Math.max(0, Math.min(1, (e.clientX - r.left) / r.width)); + const y = Math.max(0, Math.min(1, 1 - (e.clientY - r.top) / r.height)); + onMove(x, y); + }; + + const down = (e) => { + if (stateRef.current.frozen) return; + e.currentTarget.setPointerCapture?.(e.pointerId); + draggingRef.current = true; + setFromEvent(e); + clearTimeout(lpTimer.current); + lpTimer.current = setTimeout(() => { onLongPress && onLongPress(stateRef.current.pos); }, 600); + }; + const move = (e) => { if (draggingRef.current) { setFromEvent(e); clearTimeout(lpTimer.current); } }; + const up = (e) => { draggingRef.current = false; e.currentTarget.releasePointerCapture?.(e.pointerId); clearTimeout(lpTimer.current); }; + + React.useEffect(() => { + const canvas = canvasRef.current, wrap = wrapRef.current; + if (!canvas || !wrap) return; + const ctx = canvas.getContext('2d'); + let raf; + const css = getComputedStyle(document.documentElement); + const C = (n, f) => (css.getPropertyValue(n).trim() || f); + const accent = C('--accent', '#ff6a00'), cyan = C('--accent-2', '#00ccff'), line = C('--line-strong', '#3a3a3a'); + + const dpr = window.devicePixelRatio || 1; + let lastW = -1, lastH = -1; + const ensureSize = (W, H) => { + if (W === lastW && H === lastH) return; + canvas.width = W * dpr; canvas.height = H * dpr; + canvas.style.width = W + 'px'; canvas.style.height = H + 'px'; + ctx.setTransform(dpr, 0, 0, dpr, 0, 0); + lastW = W; lastH = H; + }; + + const draw = () => { + const W = wrap.clientWidth, H = wrap.clientHeight; + // Self-heal against deferred / zero initial sizing (e.g. inside a flex + // child that lays out after mount): size each frame, skip until non-zero, + // but always keep the rAF chain alive. + if (W === 0 || H === 0) return; + ensureSize(W, H); + const { pos, noiseCap, pins, frozen, follow } = stateRef.current; + const now = performance.now(); + + // drift in follow mode + if (follow && !draggingRef.current && !frozen) { + let [x, y] = pos; const d = driftRef.current; + x += d.vx; y += d.vy; + if (x < 0.05 || x > 0.95) d.vx *= -1; + if (y < 0.05 || y > 0.95) d.vy *= -1; + x = Math.max(0.05, Math.min(0.95, x)); y = Math.max(0.05, Math.min(0.95, y)); + onMove(x, y); + } + + ctx.clearRect(0, 0, W, H); + + // adaptive graph-paper grid (8 major, 4 minor subdivisions) + const minor = 32, major = 8; + ctx.lineWidth = 1; + for (let i = 0; i <= minor; i++) { + const t = i / minor; const isMajor = i % (minor / major) === 0; + ctx.strokeStyle = isMajor ? 'rgba(255,255,255,0.06)' : 'rgba(255,255,255,0.022)'; + ctx.beginPath(); ctx.moveTo(t * W, 0); ctx.lineTo(t * W, H); ctx.stroke(); + ctx.beginPath(); ctx.moveTo(0, t * H); ctx.lineTo(W, t * H); ctx.stroke(); + } + // center crosshair + ctx.strokeStyle = 'rgba(255,255,255,0.05)'; + ctx.beginPath(); ctx.moveTo(W / 2, 0); ctx.lineTo(W / 2, H); ctx.moveTo(0, H / 2); ctx.lineTo(W, H / 2); ctx.stroke(); + + const px = pos[0] * W, py = (1 - pos[1]) * H; + + // pins + for (const p of pins) { + const ppx = p.x * W, ppy = (1 - p.y) * H; + ctx.fillStyle = p.color || 'rgba(255,106,0,0.18)'; + ctx.beginPath(); ctx.arc(ppx, ppy, 34, 0, Math.PI * 2); ctx.fill(); + ctx.strokeStyle = 'rgba(255,255,255,0.18)'; ctx.lineWidth = 1; + ctx.beginPath(); ctx.arc(ppx, ppy, 34, 0, Math.PI * 2); ctx.stroke(); + } + + // vanishing trail (~5s) + const LIFE = 5000; + const pts = trailRef.current; + ctx.lineWidth = 2; + for (let i = 1; i < pts.length; i++) { + const a = pts[i - 1], b = pts[i]; + const age = now - b.t; if (age > LIFE) continue; + const alpha = (1 - age / LIFE) * 0.5; + ctx.strokeStyle = `rgba(0,204,255,${alpha})`; + ctx.beginPath(); ctx.moveTo(a.x * W, (1 - a.y) * H); ctx.lineTo(b.x * W, (1 - b.y) * H); ctx.stroke(); + } + + // noise rings — breathe at cap + if (noiseCap > 0.001) { + const breathe = 1 + Math.sin(now / 600) * 0.06; + const rCap = noiseCap * Math.min(W, H) * 0.5 * breathe; + const rCur = rCap * 0.55; + ctx.setLineDash([4, 5]); + ctx.strokeStyle = `rgba(255,106,0,0.4)`; ctx.lineWidth = 1.5; + ctx.beginPath(); ctx.arc(px, py, rCap, 0, Math.PI * 2); ctx.stroke(); + ctx.strokeStyle = `rgba(255,106,0,0.7)`; + ctx.beginPath(); ctx.arc(px, py, rCur, 0, Math.PI * 2); ctx.stroke(); + ctx.setLineDash([]); + } + + // the joystick dot + ctx.shadowColor = frozen ? cyan : accent; ctx.shadowBlur = 18; + ctx.fillStyle = frozen ? cyan : accent; + ctx.beginPath(); ctx.arc(px, py, 9, 0, Math.PI * 2); ctx.fill(); + ctx.shadowBlur = 0; + ctx.fillStyle = '#0d0d0d'; + ctx.beginPath(); ctx.arc(px, py, 3, 0, Math.PI * 2); ctx.fill(); + }; + // Paint immediately + on every resize (robust to deferred flex sizing and to + // rAF being throttled in background/preview contexts); rAF drives animation. + const loop = () => { draw(); raf = requestAnimationFrame(loop); }; + const ro = new ResizeObserver(() => draw()); + ro.observe(wrap); + // Initial paint via setTimeout retries (rAF and ResizeObserver are both + // throttled/withheld in some preview contexts; setTimeout is not). Keep + // retrying until the flex child has a non-zero width and the canvas is sized. + let timer = null, kicks = 0; + const kick = () => { draw(); if (lastW <= 0 && kicks++ < 80) timer = setTimeout(kick, 40); }; + kick(); + raf = requestAnimationFrame(loop); + return () => { cancelAnimationFrame(raf); ro.disconnect(); clearTimeout(timer); }; + }, []); + + return ( + <div + ref={wrapRef} + onPointerDown={down} onPointerMove={move} onPointerUp={up} onPointerCancel={up} + style={{ position: 'absolute', inset: 0, cursor: frozen ? 'not-allowed' : 'crosshair', touchAction: 'none', userSelect: 'none' }} + > + <canvas ref={canvasRef} style={{ display: 'block' }} /> + </div> + ); +} +window.Manifold = Manifold; diff --git a/docs/redesign/manifold-export/ui_kits/memlnaut/console/OutputEditor.jsx b/docs/redesign/manifold-export/ui_kits/memlnaut/console/OutputEditor.jsx new file mode 100644 index 0000000..8e6a490 --- /dev/null +++ b/docs/redesign/manifold-export/ui_kits/memlnaut/console/OutputEditor.jsx @@ -0,0 +1,59 @@ +/* Console 2.0 — OutputEditor: the per-output menu. Opens on hover over an + output (column / cell). Sets state (off / fixed / live), min, max, the static + value (only when deactivated), and the curve via the CurvePad. */ +const OE_STATUS = [ + { v: 'off', label: 'Off', color: 'var(--fg-dim)' }, + { v: 'fixed', label: 'Fixed', color: 'var(--accent-2)' }, + { v: 'live', label: 'Live', color: 'var(--accent)' }, +]; + +function MiniSlider({ label, value, onChange, disabled }) { + return ( + <label style={{ display: 'flex', flexDirection: 'column', gap: 2, opacity: disabled ? 0.4 : 1 }}> + <div style={{ display: 'flex', justifyContent: 'space-between' }}> + <span style={{ fontSize: 10, color: 'var(--fg-mute)', textTransform: 'uppercase', letterSpacing: '0.08em' }}>{label}</span> + <span style={{ fontSize: 10, color: 'var(--fg-dim)', fontVariantNumeric: 'tabular-nums' }}>{value.toFixed(2)}</span> + </div> + <input type="range" min="0" max="1" step="0.01" value={value} disabled={disabled} + onChange={(e) => onChange(parseFloat(e.target.value))} className="mf-slider-input" style={{ width: '100%' }} /> + </label> + ); +} + +function OutputEditor({ param, onChange, onHold, onLeave, place }) { + const isLive = param.status === 'live'; + return ( + <div onPointerEnter={onHold} onPointerLeave={onLeave} + style={{ + position: 'absolute', width: 196, zIndex: 80, ...place, + background: 'var(--glass)', backdropFilter: 'blur(14px)', WebkitBackdropFilter: 'blur(14px)', + border: '1px solid var(--glass-line)', borderRadius: 'var(--r-2)', boxShadow: 'var(--shadow-2)', + padding: 'var(--sp-3)', display: 'flex', flexDirection: 'column', gap: 'var(--sp-2)', fontFamily: 'var(--font-mono)', + }}> + <div style={{ display: 'flex', justifyContent: 'space-between', alignItems: 'baseline' }}> + <strong style={{ fontSize: 'var(--fs-sm)', color: 'var(--fg)' }}>{param.name}</strong> + <span style={{ fontSize: 10, color: 'var(--fg-dim)' }}>{param.group}</span> + </div> + + {/* tri-state */} + <div style={{ display: 'flex', background: 'var(--bg-2)', border: '1px solid var(--line)', borderRadius: 'var(--r-pill)', padding: 2, gap: 2 }}> + {OE_STATUS.map((s) => { + const on = param.status === s.v; + return ( + <button key={s.v} type="button" onClick={() => onChange({ status: s.v })} + style={{ flex: 1, border: 0, borderRadius: 'var(--r-pill)', padding: '4px 0', cursor: 'pointer', + fontFamily: 'var(--font-mono)', fontSize: 10, textTransform: 'uppercase', letterSpacing: '0.06em', + background: on ? s.color : 'transparent', color: on ? 'var(--bg)' : 'var(--fg-mute)' }}>{s.label}</button> + ); + })} + </div> + + <MiniSlider label="min" value={param.min} onChange={(v) => onChange({ min: v })} /> + <MiniSlider label="max" value={param.max} onChange={(v) => onChange({ max: v })} /> + <MiniSlider label={isLive ? 'value · live' : 'value · static'} value={param.val} onChange={(v) => onChange({ val: v })} disabled={isLive} /> + <window.CurvePad curve={param.curve} onChange={(c) => onChange({ curve: c })} size={170} /> + <span style={{ fontSize: 9, color: 'var(--fg-dim)', lineHeight: 1.4 }}>drag a bar to set value · ⌥/alt-click cycles state</span> + </div> + ); +} +window.OutputEditor = OutputEditor; diff --git a/docs/redesign/manifold-export/ui_kits/memlnaut/console/OutputStage.jsx b/docs/redesign/manifold-export/ui_kits/memlnaut/console/OutputStage.jsx new file mode 100644 index 0000000..76cbdf5 --- /dev/null +++ b/docs/redesign/manifold-export/ui_kits/memlnaut/console/OutputStage.jsx @@ -0,0 +1,83 @@ +/* Console 2.0 — OutputStage: the OUTPUT as the hero surface. A full-bleed field + of parameter columns. Per column: name on top, value below it, then the bar. + - drag / click a bar → set the value + - ⌥/alt-click a bar → cycle state (off → fixed → live) + - hover a column → open the OutputEditor (state · min · max · value · curve) */ +const OUT_GROUP_COLOR = { + formant: '--accent', pitch: '--accent-2', amp: '--good', filter: '--warn', fx: '--info', mod: '--accent-3', +}; +const OUT_NEXT = { off: 'fixed', fixed: 'live', live: 'off' }; + +function OutputStage({ params, values, onChange, compact = false }) { + const [open, setOpen] = React.useState(null); + const timers = React.useRef({ open: null, close: null }); + const drag = React.useRef({ i: -1, moved: false, startY: 0, el: null, alt: false }); + + const scheduleOpen = (i) => { clearTimeout(timers.current.close); clearTimeout(timers.current.open); timers.current.open = setTimeout(() => setOpen(i), 110); }; + const scheduleClose = () => { clearTimeout(timers.current.open); clearTimeout(timers.current.close); timers.current.close = setTimeout(() => setOpen(null), 280); }; + const hold = () => clearTimeout(timers.current.close); + + const valFromEvent = (el, clientY) => { const r = el.getBoundingClientRect(); return Math.max(0, Math.min(1, 1 - (clientY - r.top) / r.height)); }; + const down = (e, i) => { e.currentTarget.setPointerCapture?.(e.pointerId); drag.current = { i, moved: false, startY: e.clientY, el: e.currentTarget, alt: e.altKey || e.metaKey }; }; + const move = (e, i) => { + const d = drag.current; if (d.i !== i) return; + if (Math.abs(e.clientY - d.startY) > 3) d.moved = true; + if (d.moved && !d.alt) onChange(i, { val: valFromEvent(d.el, e.clientY) }); + }; + const up = (e, i) => { + const d = drag.current; if (d.i !== i) return; + if (d.alt && !d.moved) onChange(i, { status: OUT_NEXT[params[i].status] || 'live' }); + else if (!d.moved) onChange(i, { val: valFromEvent(d.el, e.clientY) }); + drag.current = { i: -1, moved: false, startY: 0, el: null, alt: false }; + }; + + return ( + <div style={{ position: 'absolute', inset: 0, display: 'flex', alignItems: 'stretch', gap: compact ? 3 : 6, padding: compact ? 12 : '20px 28px' }}> + {params.map((p, i) => { + const eff = values[i] ?? 0; + const gc = `var(${OUT_GROUP_COLOR[p.group] || '--accent'})`; + const dim = p.status === 'off'; + const placeRight = i > params.length - 4; + return ( + <div key={i} style={{ position: 'relative', flex: 1, minWidth: 0, display: 'flex', flexDirection: 'column', gap: compact ? 1 : 2 }} + onPointerLeave={scheduleClose}> + {/* hover target: name + number ONLY (not the bar) */} + <div onPointerEnter={() => scheduleOpen(i)} style={{ cursor: 'help' }}> + {/* name (top) */} + <div style={{ textAlign: 'center', fontSize: compact ? 8 : 10, fontFamily: 'var(--font-mono)', color: p.status === 'live' ? 'var(--fg-mute)' : `var(${OUT_GROUP_COLOR[p.group] || '--accent'})`, + overflow: 'hidden', whiteSpace: 'nowrap', textOverflow: 'ellipsis', letterSpacing: '0.02em' }}>{p.name}</div> + {/* number (below name) */} + <div style={{ textAlign: 'center', fontSize: compact ? 9 : 11, fontFamily: 'var(--font-mono)', fontVariantNumeric: 'tabular-nums', + color: dim ? 'var(--fg-dim)' : 'var(--fg)' }}>{eff.toFixed(2)}</div> + </div> + {/* bar */} + <div onPointerDown={(e) => down(e, i)} onPointerMove={(e) => move(e, i)} onPointerUp={(e) => up(e, i)} onPointerCancel={(e) => up(e, i)} + style={{ position: 'relative', flex: 1, background: 'var(--bg-1)', border: '1px solid var(--line)', borderRadius: compact ? 2 : 'var(--r-1)', overflow: 'hidden', cursor: 'ns-resize', opacity: dim ? 0.55 : 1, touchAction: 'none' }}> + {[0.25, 0.5, 0.75].map((t) => <div key={t} style={{ position: 'absolute', left: 0, right: 0, bottom: `${t * 100}%`, height: 1, background: 'rgba(255,255,255,0.04)' }} />)} + <div style={{ position: 'absolute', left: 0, right: 0, bottom: 0, height: `${eff * 100}%`, background: gc, opacity: 0.22 + eff * 0.6, transition: 'height 70ms linear' }} /> + <div style={{ position: 'absolute', left: 0, right: 0, bottom: `${eff * 100}%`, height: 2, marginBottom: -1, background: gc, boxShadow: `0 0 8px ${gc}`, opacity: dim ? 0.4 : 0.9 }} /> + {/* static-value ghost when live */} + {p.status === 'live' && ( + <div style={{ position: 'absolute', left: 1, right: 1, bottom: `${p.val * 100}%`, height: 0, borderTop: '1px dashed rgba(255,255,255,0.35)' }} /> + )} + {/* state glyph */} + <div style={{ position: 'absolute', top: 3, left: 0, right: 0, textAlign: 'center', fontSize: 9, + color: p.status === 'live' ? 'transparent' : p.status === 'fixed' ? 'var(--accent-2)' : 'var(--fg-dim)' }}> + {p.status === 'fixed' ? '⊟' : p.status === 'off' ? '∅' : ''} + </div> + </div> + {open === i && !compact && ( + <window.OutputEditor param={p} onChange={(patch) => onChange(i, patch)} onHold={hold} onLeave={scheduleClose} + place={{ top: 38, [placeRight ? 'right' : 'left']: 0 }} /> + )} + {open === i && compact && ( + <window.OutputEditor param={p} onChange={(patch) => onChange(i, patch)} onHold={hold} onLeave={scheduleClose} + place={{ top: 26, [placeRight ? 'right' : 'left']: 0 }} /> + )} + </div> + ); + })} + </div> + ); +} +window.OutputStage = OutputStage; diff --git a/docs/redesign/manifold-export/ui_kits/memlnaut/console/ReadoutStrip.jsx b/docs/redesign/manifold-export/ui_kits/memlnaut/console/ReadoutStrip.jsx new file mode 100644 index 0000000..3d94342 --- /dev/null +++ b/docs/redesign/manifold-export/ui_kits/memlnaut/console/ReadoutStrip.jsx @@ -0,0 +1,65 @@ +/* Console 2.0 — Readout strip: the output heatmap as a control surface (the + thin top strip in Console). Same model as OutputStage: + - drag / click a cell → set value + - ⌥/alt-click a cell → cycle state (off → fixed → live) + - hover a cell → open the OutputEditor */ +const RS_GROUP_COLOR = { + formant: '--accent', pitch: '--accent-2', amp: '--good', filter: '--warn', fx: '--info', mod: '--accent-3', +}; +const RS_NEXT = { off: 'fixed', fixed: 'live', live: 'off' }; + +function ReadoutStrip({ params, values, onChange, pinned, onTogglePin }) { + const [open, setOpen] = React.useState(null); + const timers = React.useRef({ open: null, close: null }); + const drag = React.useRef({ i: -1, moved: false, startY: 0, el: null, alt: false }); + + const scheduleOpen = (i) => { clearTimeout(timers.current.close); clearTimeout(timers.current.open); timers.current.open = setTimeout(() => setOpen(i), 110); }; + const scheduleClose = () => { clearTimeout(timers.current.open); clearTimeout(timers.current.close); timers.current.close = setTimeout(() => setOpen(null), 280); }; + const hold = () => clearTimeout(timers.current.close); + + const valFromEvent = (el, clientY) => { const r = el.getBoundingClientRect(); return Math.max(0, Math.min(1, 1 - (clientY - r.top) / r.height)); }; + const down = (e, i) => { e.currentTarget.setPointerCapture?.(e.pointerId); drag.current = { i, moved: false, startY: e.clientY, el: e.currentTarget, alt: e.altKey || e.metaKey }; }; + const move = (e, i) => { const d = drag.current; if (d.i !== i) return; if (Math.abs(e.clientY - d.startY) > 3) d.moved = true; if (d.moved && !d.alt) onChange(i, { val: valFromEvent(d.el, e.clientY) }); }; + const up = (e, i) => { + const d = drag.current; if (d.i !== i) return; + if (d.alt && !d.moved) onChange(i, { status: RS_NEXT[params[i].status] || 'live' }); + else if (!d.moved) onChange(i, { val: valFromEvent(d.el, e.clientY) }); + drag.current = { i: -1, moved: false, startY: 0, el: null, alt: false }; + }; + + return ( + <div style={{ display: 'flex', alignItems: 'stretch', gap: 2, height: 76, padding: '0 2px', + background: 'var(--glass)', backdropFilter: 'blur(12px)', WebkitBackdropFilter: 'blur(12px)', + borderBottom: '1px solid var(--glass-line)', position: 'relative', zIndex: 30 }}> + <button type="button" onClick={onTogglePin} title="Pin strip open" + style={{ flex: '0 0 auto', width: 30, border: 0, background: 'transparent', color: pinned ? 'var(--accent)' : 'var(--fg-dim)', cursor: 'pointer', fontSize: 'var(--fs-md)' }}> + {pinned ? '📌' : '▾'} + </button> + {params.map((p, i) => { + const eff = values[i] ?? 0; + const gc = `var(${RS_GROUP_COLOR[p.group] || '--accent'})`; + const dim = p.status === 'off'; + const placeRight = i > params.length - 5; + return ( + <div key={i} style={{ position: 'relative', flex: 1, minWidth: 0, display: 'flex', flexDirection: 'column' }} + onPointerLeave={scheduleClose}> + {/* hover target: name ONLY (not the bar) */} + <div onPointerEnter={() => scheduleOpen(i)} style={{ textAlign: 'center', fontSize: 8, fontFamily: 'var(--font-mono)', lineHeight: '11px', cursor: 'help', + color: p.status === 'live' ? 'var(--fg-dim)' : gc, overflow: 'hidden', whiteSpace: 'nowrap', textOverflow: 'ellipsis' }}>{p.name}</div> + <div onPointerDown={(e) => down(e, i)} onPointerMove={(e) => move(e, i)} onPointerUp={(e) => up(e, i)} onPointerCancel={(e) => up(e, i)} + style={{ position: 'relative', flex: 1, background: 'var(--bg)', borderRadius: 2, overflow: 'hidden', cursor: 'ns-resize', opacity: dim ? 0.5 : 1, touchAction: 'none' }}> + <div style={{ position: 'absolute', left: 0, right: 0, bottom: 0, height: `${eff * 100}%`, background: gc, opacity: 0.25 + eff * 0.6, transition: 'height 60ms linear' }} /> + {p.status === 'live' && <div style={{ position: 'absolute', left: 0, right: 0, bottom: `${p.val * 100}%`, height: 0, borderTop: '1px dashed rgba(255,255,255,0.3)' }} />} + {p.status !== 'live' && <div style={{ position: 'absolute', top: 1, left: 0, right: 0, textAlign: 'center', fontSize: 8, color: p.status === 'fixed' ? 'var(--accent-2)' : 'var(--fg-dim)' }}>{p.status === 'fixed' ? '⊟' : '∅'}</div>} + </div> + {open === i && ( + <window.OutputEditor param={p} onChange={(patch) => onChange(i, patch)} onHold={hold} onLeave={scheduleClose} + place={{ top: 'calc(100% + 6px)', [placeRight ? 'right' : 'left']: 0 }} /> + )} + </div> + ); + })} + </div> + ); +} +window.ReadoutStrip = ReadoutStrip; diff --git a/docs/redesign/manifold-export/ui_kits/memlnaut/console/SplitStage.jsx b/docs/redesign/manifold-export/ui_kits/memlnaut/console/SplitStage.jsx new file mode 100644 index 0000000..b6dee7c --- /dev/null +++ b/docs/redesign/manifold-export/ui_kits/memlnaut/console/SplitStage.jsx @@ -0,0 +1,16 @@ +/* Console 2.0 — SplitStage: input and output given EQUAL prominence, side by + side. Left half = the Manifold (input), right half = the OutputStage (output + field). Both fully interactive; neither dominates. */ +function SplitStage({ pos, onMove, noiseCap, pins, follow, onLongPress, params, values, onChange }) { + return ( + <div style={{ position: 'absolute', inset: 0, display: 'flex' }}> + <div style={{ flex: 1, position: 'relative', borderRight: '1px solid var(--line)', minWidth: 0 }}> + <window.Manifold pos={pos} onMove={onMove} noiseCap={noiseCap} pins={pins} follow={follow} onLongPress={onLongPress} /> + </div> + <div style={{ flex: 1, position: 'relative', minWidth: 0 }}> + <window.OutputStage params={params} values={values} onChange={onChange} compact /> + </div> + </div> + ); +} +window.SplitStage = SplitStage; diff --git a/docs/redesign/manifold-export/ui_kits/memlnaut/console/VerdictCluster.jsx b/docs/redesign/manifold-export/ui_kits/memlnaut/console/VerdictCluster.jsx new file mode 100644 index 0000000..8382c46 --- /dev/null +++ b/docs/redesign/manifold-export/ui_kits/memlnaut/console/VerdictCluster.jsx @@ -0,0 +1,76 @@ +/* Console 2.0 — Verdict cluster: floating bottom-center, the app's main control. + ▽ perturb · ↺ undo · △ commit, + A/B toggle. Long-press perturb = full re-roll. + Rests at low opacity, full on hover. */ +function ThumbIcon({ size = 24, down = false }) { + // Minimal line-icon thumb, hairline stroke in currentColor. + return ( + <svg width={size} height={size} viewBox="0 0 24 24" fill="none" + stroke="currentColor" strokeWidth="1.6" strokeLinecap="round" strokeLinejoin="round" + style={{ transform: down ? 'rotate(180deg)' : 'none', display: 'block' }} aria-hidden="true"> + <path d="M14 9V5a2.4 2.4 0 0 0-2.4-2.4L8 11v9h8.1a1.6 1.6 0 0 0 1.6-1.36l1.1-7.2A1.6 1.6 0 0 0 17.2 9z" /> + <path d="M8 20H5.6A1.6 1.6 0 0 1 4 18.4v-5.8A1.6 1.6 0 0 1 5.6 11H8" /> + </svg> + ); +} +window.ThumbIcon = ThumbIcon; + +function VerdictCluster({ onPerturb, onUndo, onCommit, onReroll, canUndo, ab, onToggleAB, onHoldA, firstSession }) { + const [hover, setHover] = React.useState(false); + const lp = React.useRef(null); + const firedReroll = React.useRef(false); + + const perturbDown = () => { + firedReroll.current = false; + lp.current = setTimeout(() => { firedReroll.current = true; onReroll(); }, 600); + }; + const perturbUp = () => { clearTimeout(lp.current); if (!firedReroll.current) onPerturb(); }; + + const big = (extra) => ({ + width: 64, height: 64, borderRadius: '50%', fontSize: 26, cursor: 'pointer', + display: 'flex', alignItems: 'center', justifyContent: 'center', fontFamily: 'var(--font-mono)', + border: '1px solid var(--glass-line)', transition: 'transform var(--dur-fast) var(--ease-console), background var(--dur-fast)', ...extra, + }); + + return ( + <div + onPointerEnter={() => setHover(true)} onPointerLeave={() => setHover(false)} + style={{ + position: 'absolute', bottom: 28, left: '50%', transform: 'translateX(-50%)', + display: 'flex', alignItems: 'center', gap: 'var(--sp-3)', padding: 'var(--sp-2) var(--sp-3)', + background: 'var(--glass)', backdropFilter: 'blur(14px)', WebkitBackdropFilter: 'blur(14px)', + border: '1px solid var(--glass-line)', borderRadius: 'var(--r-pill)', boxShadow: 'var(--shadow-2)', + opacity: hover || firstSession ? 1 : 0.55, transition: 'opacity var(--dur-med) var(--ease-console)', zIndex: 40, + }} + > + <button type="button" title="Perturb — thumbs down (hold to re-roll)" + onPointerDown={perturbDown} onPointerUp={perturbUp} onPointerLeave={() => clearTimeout(lp.current)} + style={big({ background: 'rgba(255,68,102,0.16)', color: 'var(--danger)' })} + onMouseEnter={(e) => e.currentTarget.style.transform = 'scale(1.08)'} + onMouseLeave={(e) => e.currentTarget.style.transform = 'scale(1)'}><ThumbIcon down /></button> + + <button type="button" title="Undo (z)" onClick={onUndo} disabled={!canUndo} + style={big({ width: 48, height: 48, fontSize: 20, background: 'var(--bg-2)', color: 'var(--fg-mute)', opacity: canUndo ? 1 : 0.4, cursor: canUndo ? 'pointer' : 'not-allowed' })}>↺</button> + + <button type="button" title="Commit — thumbs up" onClick={onCommit} + style={big({ background: 'rgba(255,106,0,0.18)', color: 'var(--accent)', boxShadow: '0 0 16px var(--glow-accent)' })} + onMouseEnter={(e) => e.currentTarget.style.transform = 'scale(1.08)'} + onMouseLeave={(e) => e.currentTarget.style.transform = 'scale(1)'}><ThumbIcon /></button> + + <div style={{ width: 1, height: 36, background: 'var(--glass-line)', margin: '0 2px' }} /> + + <div + title="A/B compare — hold to preview A" + onPointerDown={() => onHoldA(true)} onPointerUp={() => onHoldA(false)} onPointerLeave={() => onHoldA(false)} + onClick={onToggleAB} + style={{ display: 'flex', borderRadius: 'var(--r-pill)', overflow: 'hidden', border: '1px solid var(--glass-line)', cursor: 'pointer', userSelect: 'none' }}> + {['A', 'B'].map((k) => ( + <span key={k} style={{ + padding: '8px 14px', fontSize: 'var(--fs-sm)', fontWeight: 600, + background: ab === k ? 'var(--accent)' : 'transparent', color: ab === k ? 'var(--bg)' : 'var(--fg-mute)', + }}>{k}</span> + ))} + </div> + </div> + ); +} +window.VerdictCluster = VerdictCluster; diff --git a/docs/redesign/manifold-export/ui_kits/memlnaut/console/model.jsx b/docs/redesign/manifold-export/ui_kits/memlnaut/console/model.jsx new file mode 100644 index 0000000..ef1137d --- /dev/null +++ b/docs/redesign/manifold-export/ui_kits/memlnaut/console/model.jsx @@ -0,0 +1,122 @@ +/* Console 2.0 — shared instrument model. Pure data + inference used by all + three altitudes (Console / Perform / Zen) so they share one spine. */ +function mkParams(spec) { + const out = []; + for (const [group, names] of spec) { + names.forEach((name) => out.push({ name, group, status: 'live', val: 0.5, min: 0, max: 1, curve: 0.5 })); + } + return out; +} +const MF_MODES = [ + { id: 'paf_synth', label: 'PAF Synth', cls: 'Synth', glyph: '∿', input: 'xy', + params: mkParams([['formant', ['F1', 'F2', 'F3', 'tilt', 'spread', 'skirt']], ['pitch', ['root', 'glide', 'detune']], ['amp', ['gain', 'attack', 'decay']], ['filter', ['cutoff', 'res', 'env']], ['fx', ['drive', 'air', 'width']]]) }, + { id: 'channel_strip', label: 'Channel Strip', cls: 'Synth', glyph: '▤', input: 'joystick', + params: mkParams([['filter', ['lo', 'loMid', 'hiMid', 'hi']], ['amp', ['comp', 'gate', 'makeup']], ['fx', ['sat', 'width', 'glue', 'tilt', 'air']]]) }, + { id: 'verb_fx', label: 'Verb FX', cls: 'Synth', glyph: '◞', input: 'joystick', + params: mkParams([['fx', ['size', 'decay', 'damp', 'diff']], ['mod', ['rate', 'depth']], ['filter', ['lo', 'hi']]]) }, + { id: 'elysiamorf', label: 'Elysiamorf', cls: 'Synth', glyph: '❋', input: 'xy', + params: mkParams([['formant', ['grain', 'size', 'pos', 'spray']], ['mod', ['rate', 'depth', 'jitter']], ['amp', ['gain', 'env']], ['filter', ['cutoff', 'res']], ['fx', ['blur', 'shimmer', 'freeze', 'width']]]) }, + { id: 'memlcelium', label: 'MEML Celium', cls: 'Sequencer', glyph: '☷', input: 'xy', + params: mkParams([['mod', ['cvA', 'cvB', 'gate', 'div']], ['pitch', ['root', 'scale', 'oct']], ['amp', ['vca', 'slew']]]) }, + { id: 'breakor', label: 'Breakor', cls: 'Sequencer', glyph: '⊟', input: 'joystick', + params: mkParams([['mod', ['density', 'swing', 'fill', 'stutter']], ['amp', ['punch', 'decay']], ['filter', ['tone', 'crush']], ['fx', ['glitch', 'rev']]]) }, + { id: 'sound_analysis_midi', label: 'Sound Analysis → MIDI', cls: 'Controller', glyph: '⇉', input: 'audio_in', badge: '1-input', + params: mkParams([['mod', ['cc1', 'cc2', 'cc3', 'cc4']], ['pitch', ['note', 'bend']], ['amp', ['vel', 'press']]]) }, + { id: 'visualizer', label: 'Visualizer', cls: 'Visual', glyph: '◑', input: 'xy', + params: mkParams([['mod', ['hue', 'sat', 'flow', 'warp']], ['amp', ['bloom', 'fade']], ['fx', ['grain', 'trail']]]) }, + { id: 'c15', label: 'C15', cls: 'Synth', glyph: '◆', input: 'xy', placeholder: true, badge: 'soon', params: mkParams([['amp', ['a', 'b']]]) }, +]; + +function applyCurve(v, c) { const e = 0.25 + c * 1.75; return Math.pow(Math.max(0, Math.min(1, v)), e); } +function MF_infer(pos, seed, params) { + const [x, y] = pos; + return params.map((p, i) => { + if (p.status === 'off') return 0; // deactivated → muted + if (p.status === 'fixed') return p.val ?? 0.5; // deactivated → held static + // live → driven by the model, shaped by min/max/curve + const a = Math.sin(x * (1.2 + i * 0.19) * Math.PI + i * 0.7 + seed) * 0.5 + 0.5; + const b = Math.cos(y * (0.9 + i * 0.13) * Math.PI - i * 0.4 - seed * 0.5) * 0.5 + 0.5; + let v = a * 0.55 + b * 0.45; + v = p.min + applyCurve(v, p.curve) * (p.max - p.min); + return Math.max(0, Math.min(1, v)); + }); +} +function MF_seededGradient(rev) { + const n = 4, norms = [], status = []; + for (let i = 0; i < n; i++) { + const r = Math.abs(Math.sin((rev + 1) * (i + 1) * 12.9898) * 43758.5453 % 1); + norms.push(0.2 + r * 0.8); + status.push(r > 0.85 ? 'exploding' : r < 0.18 ? 'vanishing' : r < 0.3 ? 'converged' : 'healthy'); + } + return { norms, status }; +} + +/* useInstrument — the shared reactive spine + verdict/training actions. + Returns flat state + actions; each view renders as much of it as it needs. */ +let MF_SNAP_ID = 0; +function useInstrument(initialMode) { + const [modeId, setModeId] = React.useState(initialMode || 'paf_synth'); + const mode = MF_MODES.find((m) => m.id === modeId) || MF_MODES[0]; + const [params, setParams] = React.useState(() => mode.params.map((p) => ({ ...p }))); + const [pos, setPos] = React.useState([0.5, 0.5]); + const [seed, setSeed] = React.useState(0.4); + const [axes, setAxes] = React.useState({ boldness: 0.55, memory: 0.4, precision: 0.5 }); + const [noiseCap, setNoiseCap] = React.useState(0.12); + const [examples, setExamples] = React.useState(0); + const [loss, setLoss] = React.useState([]); + const [busy, setBusy] = React.useState(false); + const [snapshots, setSnapshots] = React.useState([]); + const [ab, setAB] = React.useState('B'); + const [holdingA, setHoldingA] = React.useState(false); + const aRef = React.useRef(null); + const [health, setHealth] = React.useState(0.8); + const [rev, setRev] = React.useState(1); + const [follow, setFollow] = React.useState(false); + const [pins, setPins] = React.useState([]); + const [firstSession, setFirstSession] = React.useState(true); + + React.useEffect(() => { + setParams(mode.params.map((p) => ({ ...p }))); setPos([0.5, 0.5]); setExamples(0); + setLoss([]); setSnapshots([]); setSeed(0.4); setFollow(false); setPins([]); setHealth(0.8); setRev(1); + }, [modeId]); + + const effSeed = holdingA && aRef.current ? aRef.current.seed : seed; + const values = React.useMemo(() => MF_infer(pos, effSeed, params), [pos, effSeed, params]); + const gradient = React.useMemo(() => MF_seededGradient(rev), [rev]); + + const pushSnap = (tag) => setSnapshots((s) => [...s, { id: ++MF_SNAP_ID, tag, noise: noiseCap, seed }].slice(-50)); + const commit = () => { + setFirstSession(false); pushSnap('commit +'); setBusy(true); + setNoiseCap((n) => Math.max(0.02, n * 0.7)); setHealth((h) => Math.min(1, h + 0.08)); setRev((r) => r + 1); + setTimeout(() => { setLoss((prev) => { const b = prev.length ? prev[prev.length - 1] : 0.5; return [...prev, Math.max(0.004, b * (0.8 + Math.random() * 0.08))].slice(-120); }); setBusy(false); }, 260); + }; + const perturb = () => { + setFirstSession(false); pushSnap('perturb −'); + setSeed((s) => s + (Math.random() - 0.5) * (noiseCap * 4 + 0.3)); + setNoiseCap((n) => Math.min(0.5, n + 0.06)); setHealth((h) => Math.max(0.1, h - 0.06)); setRev((r) => r + 1); + }; + const reroll = () => { setFirstSession(false); pushSnap('re-roll'); setSeed(Math.random() * 6); setNoiseCap(0.4); setHealth(0.5); setRev((r) => r + 1); }; + const undo = () => setSnapshots((s) => { if (!s.length) return s; const last = s[s.length - 1]; setSeed(last.seed); setNoiseCap(last.noise); setRev((r) => r + 1); return s.slice(0, -1); }); + const train = () => { setBusy(true); setTimeout(() => { setLoss((p) => { const b = p.length ? p[p.length - 1] : 0.5; return [...p, Math.max(0.004, b * 0.82)].slice(-120); }); setBusy(false); }, 260); }; + const setParam = (i, patch) => setParams((ps) => ps.map((p, j) => (j === i ? { ...p, ...patch } : p))); + const cycleStatus = (i) => setParams((ps) => ps.map((p, j) => (j === i ? { ...p, status: ({ off: 'fixed', fixed: 'live', live: 'off' })[p.status] } : p))); + const toggleAB = () => { if (ab === 'B') { aRef.current = { seed }; setAB('A'); } else { if (aRef.current) setSeed(aRef.current.seed); setAB('B'); } }; + const cycleMode = (dir = 1) => { + const live = MF_MODES.filter((m) => !m.placeholder); + const i = live.findIndex((m) => m.id === modeId); + setModeId(live[(i + dir + live.length) % live.length].id); + }; + + return { + MODES: MF_MODES, modeId, setModeId, cycleMode, mode, params, setParam, cycleStatus, + pos, setPos, seed, values, axes, setAxes, setAxis: (k, v) => setAxes((s) => ({ ...s, [k]: v })), + noiseCap, setNoiseCap, examples, loss, busy, train, + snapshots, commit, perturb, reroll, undo, ab, toggleAB, holdingA, setHoldingA, + health, rev, gradient, follow, setFollow, pins, setPins, firstSession, + }; +} + +window.MF_MODES = MF_MODES; +window.MF_infer = MF_infer; +window.MF_seededGradient = MF_seededGradient; +window.useInstrument = useInstrument; diff --git a/docs/redesign/manifold-export/ui_kits/memlnaut/console/shared-ui.jsx b/docs/redesign/manifold-export/ui_kits/memlnaut/console/shared-ui.jsx new file mode 100644 index 0000000..3132097 --- /dev/null +++ b/docs/redesign/manifold-export/ui_kits/memlnaut/console/shared-ui.jsx @@ -0,0 +1,65 @@ +/* Console 2.0 — shared chrome for the simpler altitudes. */ +const ALT_HREF = { + in: { console: 'index.html', perform: 'perform.html', zen: 'zen.html' }, + split: { console: 'console-split.html', perform: 'perform-split.html', zen: 'zen-split.html' }, + out: { console: 'console-output.html', perform: 'perform-output.html', zen: 'zen-output.html' }, + composite: { console: 'console-composite.html', perform: 'perform-split.html', zen: 'zen-split.html' }, +}; +function AltitudeNav({ current, focus = 'in', style }) { + const items = [ + { id: 'console', dots: '◆◆◆', label: 'Console' }, + { id: 'perform', dots: '◆◆', label: 'Perform' }, + { id: 'zen', dots: '◆', label: 'Zen' }, + ]; + const pill = (on) => ({ textDecoration: 'none', fontSize: 11, padding: '2px 8px', borderRadius: 'var(--r-pill)', + color: on ? 'var(--accent)' : 'var(--fg-dim)', background: on ? 'rgba(255,106,0,0.14)' : 'transparent' }); + const foc = [['in', 'IN'], ['split', 'DUAL'], ['out', 'OUT'], ['composite', 'FLEX']]; + return ( + <div style={{ position: 'absolute', top: 12, right: 14, zIndex: 70, display: 'flex', gap: 6, alignItems: 'center', + background: 'var(--glass)', backdropFilter: 'blur(10px)', WebkitBackdropFilter: 'blur(10px)', + border: '1px solid var(--glass-line)', borderRadius: 'var(--r-pill)', padding: '4px 6px', ...style }}> + {items.map((it) => ( + <a key={it.id} href={ALT_HREF[focus][it.id]} title={`${it.label} · ${focus}`} + style={pill(it.id === current)}>{it.dots}</a> + ))} + <span style={{ width: 1, height: 16, background: 'var(--glass-line)' }} /> + {foc.map(([f, label]) => ( + <a key={f} href={ALT_HREF[f][current]} title={f === 'split' ? 'Input + output equal' : f === 'out' ? 'Output-first' : f === 'composite' ? 'Composite — drag to rebalance' : 'Input-first'} + style={{ ...pill(focus === f), fontSize: 9, letterSpacing: '0.08em' }}>{label}</a> + ))} + </div> + ); +} + +/* MiniMeters — glanceable read-only output bars (no interaction). */ +function MiniMeters({ params, values }) { + const GC = { formant: '--accent', pitch: '--accent-2', amp: '--good', filter: '--warn', fx: '--info', mod: '--accent-3' }; + return ( + <div style={{ display: 'flex', alignItems: 'flex-end', gap: 2, height: 40 }}> + {values.map((v, i) => ( + <div key={i} title={`${params[i].name}: ${v.toFixed(2)}`} + style={{ width: 5, height: '100%', background: 'var(--bg-2)', borderRadius: 1, position: 'relative', overflow: 'hidden' }}> + <div style={{ position: 'absolute', left: 0, right: 0, bottom: 0, height: `${v * 100}%`, + background: `var(${GC[params[i].group] || '--accent'})`, opacity: 0.3 + v * 0.6 }} /> + </div> + ))} + </div> + ); +} + +/* CompactAxis — slim labelled feel slider for the Perform bar. */ +function CompactAxis({ label, value, onChange, accent = 'var(--accent)' }) { + return ( + <label style={{ display: 'flex', alignItems: 'center', gap: 'var(--sp-2)', fontFamily: 'var(--font-mono)' }}> + <span style={{ width: 64, fontSize: 10, color: 'var(--fg-mute)', textTransform: 'uppercase', letterSpacing: '0.08em' }}>{label}</span> + <input type="range" min="0" max="1" step="0.01" value={value} + onChange={(e) => onChange(parseFloat(e.target.value))} + className="mf-slider-input" style={{ width: 120, '--mf-axis-accent': accent }} /> + <span style={{ width: '3ch', fontSize: 10, color: 'var(--fg-dim)', fontVariantNumeric: 'tabular-nums', textAlign: 'right' }}>{value.toFixed(2)}</span> + </label> + ); +} + +window.AltitudeNav = AltitudeNav; +window.MiniMeters = MiniMeters; +window.CompactAxis = CompactAxis; diff --git a/docs/redesign/playground-2.0-rewrite-plan.md b/docs/redesign/playground-2.0-rewrite-plan.md new file mode 100644 index 0000000..3507e61 --- /dev/null +++ b/docs/redesign/playground-2.0-rewrite-plan.md @@ -0,0 +1,398 @@ +# NISPS Playground 2.0 — Clean-Room SolidJS Rewrite Plan + +*Status: sign-off-ready. Author: lead architect, synthesizing four designer proposals (P1–P4) against three judge panels (mission-fit, feasibility, UX coherence). Date: 2026-06-17.* + +--- + +## TL;DR / recommendation + +We rewrite the browser playground clean-room in SolidJS as **one fullscreen instrument** whose interaction model is **P3's "Console"** (full-bleed canvas + always-on Verdict cluster + a right-edge dock whose drawers have three depth states), driven internally by **P2's single-owned-reactive-spine architecture** (one pure `createMemo` chain from input → ML → output, read by every consumer; impure side-effects in one `createEffect`). The whole UI is **one schema-driven `<GenericMode>`** replacing today's 8–9 cloned mode files, so adding an engine is "a JSON schema + a C++ lambda + codegen, zero UI." We adopt **P4's concepts** — the snapshot DAG (unifying undo/A-B/trail/history) and the control-point `off/fixed/live` enum (collapsing the three overlapping mute/pin/expose systems) — but **explicitly reject P4's v1 commitment to a runtime-shaped MLP**: v1 ships an *honest fixed-2-input contract that fails the build on schema/arch mismatch*, deferring multi-input modes until the C++ core genuinely supports them behind a passing parity check. + +Why this is more mature than today's app: (1) the live-feedback regression that has haunted every prior session ("MLP output stops updating when I move the joystick") becomes **structurally impossible and e2e-asserted**, because there is exactly one reactive path from input to output and every consumer reads it; (2) the 663-line `mode-runtime` god-hook and 8 near-identical mode files collapse into composable hooks + one data-driven component — the orthogonality principle finally reaches the UI layer; (3) two real research-validity bugs (fake single-point loss history, training on post-pipeline outputs) are fixed; (4) the feature inventory is treated as a **capability budget to make reachable**, not a screen budget to fill — most features live one drawer/keystroke/probe-call deep rather than cluttering the canvas. + +--- + +## 1. Mission & design thesis + +NISPS is **a research platform for interactive ML control of audio** — a *controller* for synths, not a synth. The north star for 2.0: + +> **The network is the medium you shape; the gesture is the cursor; reactivity is the wire that must never break. You shape a mapping from embodied gesture to sound by listening and giving verdicts (👍/👎), and the screen reveals exactly as much machinery as you reach for.** + +This serves the five-axis **orthogonality principle** (parameter sets · modes · ML architectures · audio engines · UI/UX must vary independently) by making each axis a different layer that can be edited without disturbing the others — and it serves the **crystallization pipeline** (browser is where research happens; firmware is where findings ossify) by keeping the browser identical to firmware *by default* and treating divergence (e.g. the `spread`/Xavier regime) as an opt-in lab toggle. + +Three personas in priority order, all served: **Dimi-researcher** (wants diagnostics that answer "is it learning?"), **Dimi-performer** (embodied, eyes on the synth, thumb on the verdict), and **AI coding agents** (the synchronous debug probe + Playwright + parity scripts are first-class — new UX must remain headlessly drivable and observable). + +The design discipline, taken from P1 and endorsed by the mission-fit judge: **a capability budget, not a screen budget.** The brief's enormous inventory is something to make *reachable* (one gesture/keystroke/probe-call deep), not something to render all at once. This is the direct antidote to cargo-cult feature parity. + +--- + +## 2. Intended UX, locked + +### 2.1 Stance: desktop-first, touch-correct primitives + +Confirmed as decided (§7-H of the brief, pending Dimi's confirmation it still holds). Layout target **1280×800+ landscape**; the Console docks right, the manifold owns the rest. Every primitive that touches the manifold (joystick, zoom, trail, heatmap-scrub, verdict buttons) is built on **Pointer Events with `touch-action:none` + pointer-capture**, so the same code is finger-correct on a foldable without a separate mobile IA. Below ~720px wide, the Console degrades to a bottom-sheet variant of the *same* components (one responsive breakpoint). **Touch-correctness of joystick/zoom/trail is a v1 acceptance criterion**, not deferred. Pinch maps to the same zoom signal as wheel/momentum. + +### 2.2 Information architecture: three persistent zones + the Console + +There is **one screen** (`/`). The only other routes are the hidden `/dev/primitives` showcase and the `?debug=1`-gated probe. + +``` +┌─────────────────────────────────────────────────────────┐ +│ ❶ READOUT STRIP (top, interactive heatmap, auto-hides) │ +├──────────────────────────────────────────────┬──────────┤ +│ │ ❹ THE │ +│ ❷ THE MANIFOLD │ CONSOLE │ +│ (fullscreen canvas: visualizer/seq grid + │ (48px │ +│ joy-map + trail + noise rings + pins) │ dock + │ +│ │ drawer) │ +│ ❸ VERDICT CLUSTER (floating, bottom) │ │ +└──────────────────────────────────────────────┴──────────┘ + ◀ Drawer slides in here (left of dock, mutually exclusive) +``` + +- **❶ Readout strip** — the interactive output heatmap (§2.5). Auto-hides in pure-synth modes (the visualizer is the readout); pin glyph holds it open. +- **❷ The Manifold** — the always-on canvas: engine/visual/sequencer output, the zoom joy-map minimap (adaptive graph-paper grid), the clickable vanishing trail, noise rings, pinned-region overlays, the joystick dot. **The empty canvas is itself the primary input surface** (pointer-down drives the joystick in joy-map mode), with hit-test priority: trail points and pins capture before the joystick. +- **❸ Verdict cluster** — floating bottom-center, the most-used control in the app, never hidden on first session: **👎 perturb · ↺ undo · 👍 commit**, with A/B toggle to the right; Randomize demoted to long-press-👎 ("thumbs-down really hard", confirm-on-release). Rests at low opacity, full on hover/active. +- **❹ The Console** — right-edge dock (48px icon rail) + a single mutually-exclusive drawer sliding in to its left. **This resolves the dock-vs-floating-bar conflict in favor of the dock.** + +All HUD anchors are **derived from a reactive layout context** (`useStageMetrics()`), never hardcoded pixels — killing the "tied to a removed 88px bottom sheet" bug class at the source. Chrome auto-dissolves after ~4s of inactivity (Design-C heritage) via a single `chromeOpacity` signal, restored on any pointer/key/`__nisps` activity — **but never fades on the first session**, and dock icons never fully vanish (a thin orange edge persists). + +Visual language is **faithful to a-immersive, not the cyan drift**: accent orange `#ff6a00`, danger `#ff4466`, glass `rgba(13,13,13,0.65)` on dark, JetBrains Mono @13px, easing `cubic-bezier(0.22,1,0.36,1)`. A lint allowlist of CSS vars enforces it. + +### 2.3 The Console drawers — split by mental mode, not by data + +Five dock sections. The key UX call (from P3, endorsed by the UX-coherence judge as "the strongest anti-modal-confusion move in the set"): **separate the performer's control surface from the researcher's diagnostics** — never make someone playing share a panel with gradient-flow bars. + +| Icon | Drawer | Mental mode | Contents | +|---|---|---|---| +| **Shape** | RL feel + examples + history | shaping | add example, train, clear, dataset count; RL feel controls; snapshot DAG browser at Full depth | +| **Feel** | compound axes + presets | playing | Boldness / Memory / Precision axes; trim-pot overrides; six control-preset chips | +| **Route** | input/output + control-point matrix | wiring | input source, output backend (Audio/MIDI/OSC/controller), per-param control-point rows, per-group curves; Emitters×Targets matrix at Full depth (dense/modular modes) | +| **Health** | diagnostics | debugging | weight-health detail, gradient-flow, layer-stats, loss plot, `spread`/`tame` lab controls | +| **Help** | overlay + keymap | — | keyboard map, first-run guidance | + +`Mode/engine switching` lives in a compact switcher at the top of the dock (grouped by capability class — see §2.10), so it is always reachable without colonizing a drawer. + +### 2.4 Progressive disclosure: drawer depth = one learnable gesture + +Adopted from P3, backed by P4's data model. Each drawer has **three depth states**, and that *is* the disclosure mechanism — there is **no global "Advanced" toggle**: + +| Depth | Width | Trigger | Contents | +|---|---|---|---| +| **Peek** | ~320px | click dock icon | summary chips (`Sources 8/48 · Targets 6/10 · live 4`) + the one primary control + read-only sparklines | +| **Expand** | ~520px | "⤢ More" / drag-resize | active controls only + collapsible rails + "show all" footer | +| **Full** | fullscreen modal | "⤢⤢ Open" | the dense surface: full matrix, per-cell editor, all overrides, exposure controls | + +"What shows at each depth" is **data, not hand-wiring**: each schema param and each diagnostic declares a `tier: 0|1|2`, and the current depth reveals tiers ≤ depth. This makes Dimi's "some modes expert-only" wish (ALIGNMENT Q2) just a schema default for `tier`, and unifies the brief's five competing disclosure proposals (global toggle / per-mode expertise / three-tier / control-point / staged-unlock) into one mechanism. The **Feel drawer's Peek auto-opens** on first load and after mode-switch (so the compound axes are glanceable without a click — the "always-on bar" intent without permanently stealing canvas), and a **mini-axis ghost** persists on the Feel dock icon when collapsed. + +The dense **staged-unlock** model (Dimi's near-decided intent) lives inside the Route drawer's Full tier: a mode starts with a small set of `live` control points; the user opts more in one-by-one or swaps them via the "show all / opt-in" footer. + +### 2.5 The interactive heatmap (readout strip) + +Output readout becomes a control surface. Two interactions per cell, with the **load-bearing tactile constants reproduced exactly**: + +1. **Drag** (>3px motion threshold, pointer-capture, horizontal across a cell) → directly sets that param's normalized value to the x-position; output reroutes immediately. +2. **Click** (pointerup, no motion) → toggles a per-parameter **override popup** anchored to the cell: min slider, max slider, curve slider, **freeze toggle** (frozen → drag sets a fixed value, bypassing the model), with **300ms grace** dismissal so the pointer can travel cell→popup. + +Hover shows `{name}: {value}` + a `▾` clickability hint, live during drag. Frozen cells dimmed. The strip is hidden in synth mode (the visualizer is the readout). These constants (3px / 300ms / pointer-capture) are non-negotiable craft. + +### 2.6 The core interaction loop (explore → shape → train → evaluate → compare) + +**Explore.** Drag the manifold (or joystick/sliders/mic). On every pointer-move (reactive, *not* rAF), the input pipeline runs `deadzone → zoom → curve → smoothing → momentum-as-zoom`; the result drives inference; outputs flow to the engine *and* repaint the readout strip + visualizer in the same propagation. The joy-map shows the zoom window (grid subdivides 4×4→32×32); a ~5s vanishing trail fades behind the dot; noise rings breathe at the current cap. Slow deliberate moves auto-zoom-in (momentum); zoom-to-zero = freeze input (a detent, no separate toggle). **Follow mode** (double-click the dot) releases the pointer for hands-free drift. + +**Shape via RL (the default learning mode).** Hear something good → **👍**: `train()` on accumulated examples + decay noise (settle). Hear something wrong → **👎**: `moveWeights(noiseCap, spread, pinMask)` + grow noise (perturb). Feedback is **zoom-aware**: zoomed-in 👎 nudges, zoomed-out shakes. Went too far → **↺ undo** pops the snapshot DAG and flashes. Want chaos → long-press 👎 = full re-roll. Every destructive op auto-snapshots first. This is the resting state; the **fearless loop** (RL + consequence-free undo + A/B preview) is the path of least resistance. + +**Shape via examples (IML, secondary).** Open Shape drawer. The "Add example" two-step toggle: 1st press freezes inference so you dial the target (on the heatmap strip or sliders), 2nd press stores `(effectiveInput → rawTargetOutputs)`. **Targets are raw model-space values, not post-pipeline outputs** (the §2.6 research-validity fix). Examples can be pinned (always-included, FIFO-exempt = region-pinning approach A). + +**Train.** Implicit in 👍, or explicit in Shape; runs in a single pooled disposable Web Worker holding its own WASM instance, round-trips weights, and streams **real loss history** (new `nisps_ml_loss_history` C API) to a live sparkline. + +**Evaluate.** Ambient weight-health glow at the screen edge answers "healthy?" at a glance; the Health drawer shows loss-history, gradient-flow bars, layer-stats, and the input-space heatmap (throttled to weight-change ~200ms, not per-frame). Mostly, evaluation *is listening* — diagnostics are there for when the ears raise a question. + +**Compare.** A/B: tag the current state as **A** (weights + noise + zoom), keep exploring as **B**, toggle A↔B at the same input position, **press-and-hold to preview A** (preview-pedal), then Accept B or Revert to A. **Freeze Output** locks the audible result while inference keeps running — watch the readout shift to preview what *would* change before committing (distinct from Freeze Input). + +**Protect.** Set a few control points to `fixed` (excluded from `moveWeights`, target held) and 👎 explores the rest; long-press the joy-map to region-pin. This zoom↔train↔pin refinement workflow is the whole reason the instrument is more than a randomizer. + +### 2.7 Undo / history + +Undo, snapshots, A/B, and the trail are **one model** (P4's snapshot DAG, §3 architecture): a content-addressed DAG of `NetworkState` nodes with a `current` pointer. Undo = `current ← parent`; destructive ops = append child + advance; A/B = pin/swap pointers; trail tap-to-return = jump to a node. The Verdict cluster's ↺ undo is the single-step face (20-step practical depth, ring-bounded, default 20/max 50); long-press = the tagged-node history list; the Shape-Full DAG browser renders the tree for branchable exploration. RL-undo against a stub is the **first build wave** (de-risks the whole history system; matches Dimi's own instinct). + +### 2.8 Keyboard accelerators (layer, not primary door) + +Verb keys (`1`–`5` for drawers, `\` for Full, `space`/`↑`/`↓` for verdict, `z` for undo) are an **accelerator layer on top of a fully pointer-operable Console** — never the only door. This keeps the performer/newcomer path pointer-discoverable while giving the researcher fast iteration. + +--- + +## 3. Architecture + +### 3.1 The reactive spine (the live-feedback guarantee, made structural) + +The single highest-value structural fix. The brief's #1 recurring frustration — MLP output silently stops updating the UI when the joystick moves — is a reactivity-discipline failure, not a UI one. We make it impossible to regress by collapsing input → ML → output into **one pure `createMemo` chain that every consumer reads**, with the impure side-effects in a single `createEffect`: + +``` +inputRaw // Float32Array signal, {equals:false} (the ONLY input entry point) + → processedInput // createMemo: deadzone→zoom→curve→smoothing→momentum (pure, golden-tested) + → mlOutput // createMemo: WasmIML.infer(processedInput) into a REUSED buffer (pure read) + → routedOutput // createMemo: voice-space + output pipeline (global curve→smoothing→slew→freeze) +``` + +Then **one** `createEffect` reads `routedOutput` and performs the side-effects: `postMessage` to the worklet (with a *dedicated re-filled send-buffer*, because transferables neuter the source) + engine post. Every UI consumer — readout strip, visualizer, trail — **reads `mlOutput`/`routedOutput` directly**. There is no separate "push output to the UI" path that can rot. + +This is the union the feasibility judge mandated: **P3's pull-based memo chain** (no rot-able push path) corrected for **memo purity** (no `postMessage` inside a memo — that lives in the effect), with **P2's rigor**: signal/store split by update cadence, `HeapVec` re-derive-on-access, pointer-event coalescing to display cadence via `batch()`+microtask (reactive, not a rAF poll, but rate-limited). **rAF touches only canvas drawing**, never inference. We **reject P1's `createComputed` approach** (wrong primitive; the feasibility judge called it a load-bearing foot-gun). + +**E2E invariant, asserted on every mode in CI:** `__nisps.setInputs([x,y]); expect(getOutputs()).toChange() && expect(getEngineParams()).toChange()` in the same tick. + +### 3.2 Stores — module singletons, split by update cadence + +The split by *update cadence* is the correctness lever (P2): + +| Store | Kind | Why | +|---|---|---| +| `input-store` | `createStore` (config) + `Float32Array` signal `{equals:false}` (raw axes) | per-frame hot path → never a deep proxy | +| `ml-store` | `createStore` (status/arch/dataset) + `outputs: Float32Array` `{equals:false}` + `weightsRevision: number` | outputs change every frame; revision is the cheap "weights changed" tripwire; owns `WasmIML` + training-worker lifecycle | +| `control-store` | `createStore` (axes) + **per-param `createMemo`** | compound-axis → param fanout is `createMemo(() => axisTable(axis) + offset)`, **replacing the `control-routing.ts` `JSON.stringify`-inside-untracked-effect anti-pattern** | +| `routing-store` | `createStore` | the Emitters×Targets matrix; one **control-point status** per target: `off / fixed / live` (collapses mute/pin/expose into one toggle/row) | +| `output-store` | `createStore` | global curve→smoothing→slew→freeze gate; uses the reuse buffer (no per-frame alloc) | +| `history-store` | `createStore` | the snapshot DAG (undo / A-B / trail / snapshots) | +| `session-store` | `createStore` | presets + persistence (versioned schema, base64 weights), URL params, mode switch (restore-or-fresh) | +| `bus.ts` | kept as-is | typed synchronous pub/sub for genuinely cross-cutting events (`mode.switched`, `pin.changed`, `snap.push`) — **not** hot-path data | + +### 3.3 ML / audio hot paths + +- **Inference (main thread):** the `mlOutput` memo calls `WasmIML.infer` into a **reused output buffer** (fixing today's per-frame `Float32Array` alloc). All WASM memory access goes through **one** place — a `HeapVec` wrapper that **re-derives the `HEAPF32` view on every access** (closes the stale-view silent-corruption bug class by construction; strictly better than today's cached-view-plus-manual-`rebind()`). `WasmIML` exposes only typed methods; no raw heap leaks out. +- **Audio (worklet):** the **two-WASM architecture is kept wholesale** — main-thread ML + AudioWorklet engine loading the *same* `nisps.wasm` bytes via `WebAssembly.compile` with hand-rolled auto-discovered imports (no Emscripten glue in `AudioWorkletGlobalScope`). **Worklet imported via `?worker&url`** (the recent, correct fix — plain `new URL(...,import.meta.url)` ships raw `.ts`). The send-effect owns a dedicated re-filled transferable buffer. +- **Training:** one pooled disposable Worker with its own WASM instance, torn down on unmount/mode-switch. + +### 3.4 Decomposing the god-hook + +`mode-runtime.ts` (663 lines) → small named composables, each independently testable and probe-mockable: + +``` +src/runtime/ + control-graph.ts # the memo chain + the single send-effect (the spine, ~one tested file) + use-input-adapters.ts# pointer/joystick/gamepad/mic → input-store.setRaw + use-audio-lifecycle.ts # engine-host start/stop/teardown on mount/mode-switch + use-snapshots.ts # DAG ops + use-heatmap-sampler.ts # input-space heatmap, throttled to weightsRevision + use-trail.ts + use-auto-explore.ts +``` + +### 3.5 Schema-driven mode system — one `<GenericMode>` + +The 8 cloned `*Mode.tsx` + `C15Mode.tsx` collapse to **one `<GenericMode schema={schema} />`** (the orthogonality principle reaching the UI; universal across all four proposals and the baseline the judges mandate). It reads the generated `ModeSchema`: + +- `ui.primary_input` → mounts the input adapter (`xy_pad`/`joystick`/`sliders`/`audio_in`/`midi_in`/`none`). +- `ui.show_synth_visualizer` → `SynthVisualizer` vs `VisualEngine` vs `SequencerLane`. +- `ui.show_voice_space_selector` → renders the selector or not. +- `params[].{group,curve,label,min,max}` → drives the readout strip, the Route per-param rows, and a cross-mode `<ParamGroupCurve>` primitive (per-group curve editing everywhere, per Dimi's explicit ask). +- `capability_class` (new field) → mode taxonomy (§2.10). +- `params[].tier` + diagnostics `tier` (new) → disclosure depth. + +**Heterogeneous I/O is first-class via schema flags, not code branches:** `output_kind: event` (breakor/elysiamorf) → `SequencerLane` + MIDI/event backend, silent audio; `output_kind: controller` (`kRouteOutputsToEngine=false`) → raw output meters; `primary_input: audio_in` → mic analyser adapter; no `voice_spaces` → no selector. + +### 3.6 Backends as output adapters + +```ts +interface OutputBackend { send(engineParams: Float32Array): void; start(): Promise<void>; teardown(): void; } +``` + +Implementations: `WebAudioBackend` (worklet), `WebMidiBackend` (7-bit CC out), `OscBridgeBackend` (WS → salvaged Deno bridge), `SerialCvBackend` (stub behind a locked contract, "later"). The mode picks one+ via schema; the send-effect just calls `backend.send()`. Adding OSC never touches the loop (orthogonality + Dimi's v1-backends ask). + +### 3.7 Adding a new mode/engine (the extensibility story) + +1. Write the C++ engine in `nisps/engines/` satisfying the `AudioEngine` concept (+ voice-space lambdas). +2. Add `schemas/modes/<mode>.json` (params, groups, curves, `capability_class`, `ui`, voice-space names, tiers). +3. `bun run codegen/generate.ts` → C++ `constexpr` header + TS types (golden test enforces idempotent byte-identical regen). +4. Register the engine id in the worklet's engine switch + `modes/index.ts`. +5. **Zero new UI.** `<GenericMode>` renders it; it appears in the switcher under its class. + +### 3.8 Persistence / session model + +Replace today's bespoke per-store `Partial<>` merges, `Infinity↔null` slew encoding, and slow `Array.from()` weight JSON with **one `persist<T>(store, version, migrate)` helper**: versioned schema + base64 weight blobs, debounced 200ms. URL params (`tame`/`spread`/`preset`/`debug`) kept. Session presets are **composed layers** (control / synth / weights / mode independently saveable, soft-bundled on save) — see §8-I. Restore-or-fresh on mode switch. + +### 3.9 The arch contract (v1, honest) + +The schema declares `input_size` but WASM is hardcoded `MLP<2,10,14,18,126>`; today `WasmIML.init` warns-and-ignores caller sizes and `setInput` loops past the real arch, writing phantom channels 2–5 into OOB heap. **v1 resolution (P1's honest contract, mandated by the mission-fit judge):** codegen **fails the build** on schema/arch mismatch; `<GenericMode>` clamps `setInput` to the real arch; multi-input modes are explicitly out of v1 (see §8-D). The runtime-shaped MLP (P4's idea, Dimi's stated intent) is deferred behind a passing parity check — **never bundled into the UI rewrite**, because it rewrites the parity-tested (2.4e-7) zero-heap core that the crystallization mission and browser≡firmware contract both depend on. + +--- + +## 4. Feature treatment table + +Legend: **Keep** (port forward) · **Redesign** (changed home/shape) · **Drop/Defer** (out of v1). Every item from the brief's §6 inventory. + +| Feature | Verdict | Rationale | +|---|---|---| +| Explore via abstract input channels | Keep | The loop; unify behind `input-store.setRaw` + adapters | +| RL thumbs (default) | Keep | Resting state; Verdict cluster; zoom-aware; auto-snapshot | +| Examples/IML two-step toggle | Keep + fix | Secondary path in Shape; **fix label source to raw outputs**; inline hint | +| Randomise | Redesign | Demote to long-press-👎 (declutter; "thumbs-down really hard") | +| Output pin mask | Keep | Folds into control-point `fixed` | +| Async training | Keep + fix | Pooled worker; **real loss history** via new C API | +| Auto-snapshot before destructive ops | Keep | Centralized as DAG append | +| Canvas-first immersive layout | Keep | The Stage; the thesis | +| Right dock + mutually-exclusive drawers | Keep/build | Resolves dock-vs-bottom-sheet → dock | +| Mode shell + switcher | Redesign | 8–9 clones → one `<GenericMode>`; switcher grouped by capability class | +| Auto-dissolving chrome | Keep | 4s timeout; never fade verdict on first session; dock edge persists | +| Floating status pill | Redesign | Becomes ambient health glow + Feel mini-axis ghost; layout-derived anchors | +| Help overlay | Keep | Behind `?` / Help dock | +| Visual language (orange/JetBrains/glass) | Keep + fix | **Fix the cyan drift**; lint a CSS-var allowlist | +| Boldness/Memory/Precision axes | Keep | Feel drawer; per-param `createMemo` fanout | +| Stability↔Fluidity (stretch axis) | Defer | Clutter; behind Feel-Expand later if demanded | +| Trim-pot offsets + double-tap re-link | Keep | Default (§8-F); add "offset active" dot for visibility | +| Six control presets | Keep | Cheap axis-triplets; encode playing-postures | +| Per-group curve editing | Keep | Cross-mode `<ParamGroupCurve>` primitive (Dimi explicit) | +| Heatmap readout strip | Keep | Top of stage | +| Drag-to-scrub cell (3px, pointer-capture) | Keep | Load-bearing tactile detail; reproduce exactly | +| Click → override popup (min/max/curve/freeze, 300ms grace) | Keep | Reproduce exactly | +| Frozen-cell dimming; hidden in synth mode | Keep | — | +| Per-param overrides (unified) | Redesign | Control-point `off/fixed/live`, one toggle/row | +| deadzone→zoom→curve→smoothing→momentum | Keep | Pure-fn memo; golden-tested | +| Zoom (log, anchor modes, zoom-at-zero=freeze) | Keep | Core navigation; freeze-detent elegant | +| Momentum-as-zoom | Keep | — | +| Joy-map minimap + adaptive grid | Keep | Navigation legibility | +| Vanishing trail + tap-to-return | Keep | DAG node jump | +| Noise ring(s) | Keep | — | +| Region pinning (example-pinning A) | Keep | — | +| Parameter pinning (pin mask ≠ mute) | Keep | Control-point `fixed` | +| Snapshot stack | Redesign | Unified into the snapshot DAG | +| RL undo (20-step) | Keep | Verdict cluster; **first build wave** | +| A/B compare + hold-to-preview | Keep | DAG pointer ops; preview-pedal | +| `spread` master regime | Redesign | **Opt-in lab toggle** in Health (firmware-init default); effect surfaced via Boldness axis (§8-E) | +| noise floor/cap/growth/decay, weight decay, distribution (incl. cauchy), layer-aware | Keep | Health/Feel, tier-2 | +| Zoom-aware feedback scaling | Keep | Default on | +| Pressure / hold-duration feedback | Keep | Touch-correct primitive | +| Auto-Explore + Follow mode | Keep | Shape drawer; emerald toggle + progress ring | +| global curve→smoothing→slew→freeze gate | Keep + fix | Reuse buffer (no per-frame alloc) | +| Freeze output (preview-before-commit) | Keep | Feel/Route | +| Tame | Redesign | Promote URL → Health panel; labelled "output limiter" (safety) | +| Input-space heatmap | Keep | Health; throttled to weight-change ~200ms | +| Weight-health indicator | Keep | Ambient screen-edge glow + Health detail | +| Gradient-flow bars | Keep | Health, tier-2 | +| Per-layer stats | Keep | Health-Full | +| Loss / loss-history plot | Keep + fix | **Real history** via new C API | +| Progressive disclosure | Redesign | Drawer depth (Peek/Expand/Full) backed by schema `tier`; no global Advanced toggle | +| Three-tier responsive surface | Keep | Is the drawer-depth mechanism | +| Control-point status (off/fixed/live) | Keep | Unified per-target model | +| Exposure-spectrum staged-unlock + swap + opt-in | Keep | Inside Route-Full for dense modes | +| Emitters × Targets matrix (MLP = emitter bank) | Keep (scope-gate) | Route-Full; built to host modular mode (§8-G) | +| Synth modes (incl. modular 3-osc/4op/additive) | Keep / scope-gate | Modular mega-mode v1.5 (§8-G), matrix built to host it | +| Controller-only | Keep | `output_kind: controller` | +| Sequencer (breakor/elysiamorf, ShapeSeq freeze/delta) | Keep | `SequencerLane`; freeze/delta is a killer feature | +| Visual | Keep | `VisualEngine`; genesis use case | +| C15 | Defer | Placeholder, no schema/engine — don't scaffold UI for vapor (§8-G) | +| Mic modes (XIASRI / sound_analysis_midi) | Defer (honest) | Out of v1 until runtime-shaped MLP (§8-D); v1 shows "single-input in browser" badge | +| Web Audio backend | Keep | — | +| WebMIDI out | Keep | v1 adapter | +| OSC via bridge | Keep | v1 adapter; salvage Deno bridge | +| CV/gate serial | Defer | Stub behind locked contract | +| Session presets | Keep | Composed layers (§8-I) | +| bypass vs mute per param | Keep | Folds into control-point semantics | +| restore-or-fresh on mode switch | Keep | — | +| Debounced localStorage + URL params | Keep + fix | Versioned schema + base64 weights | +| Debug probe (`window.__nisps`) | Keep + fix | **Gate behind `?debug=1`** (honor the doc); on by default in dev builds | +| Two-WASM arch + custom worklet loader | Keep | Correct, load-bearing | +| Pure-fn pipelines + curve catalog | Keep + fix | **Unify the two diverging curve enums** (one source, golden-tested vs `math.hpp`) | +| Typed signal bus | Keep | Cross-cutting events only | +| Live-feedback guarantee | Keep (structural centerpiece) | One reactive path; e2e-asserted every mode | + +--- + +## 5. What 2.0 gains vs today, and what we deliberately drop + +**Gains** +- **Live feedback can't silently regress.** UI, audio, and visuals derive from the *same* memo chain; a desync is a failing e2e test, not a recurring prod bug. Directly kills Dimi's #1 frustration by construction. +- **8–9 mode files → 1 `<GenericMode>`; 663-line god-hook → composable hooks.** Adding an engine is schema + C++ + codegen, zero UI. Orthogonality is enforced, not aspirational. +- **One history model** (snapshot DAG) instead of three (snapshots/undo/A-B) — less code, branchable exploration. +- **One disclosure mechanism** (drawer depth backed by schema `tier`) instead of five competing proposals — one learnable gesture. +- **Performer and researcher are separated** (Feel vs Health drawers) — no gradient bars in the player's face, no hunting for diagnostics. +- **Honest, correct core contract:** no phantom inputs, real loss history, training on raw outputs (a research-validity fix), heap safety by construction, no per-frame allocation on the audio-param path. +- **Backends are adapters** — MIDI/OSC/CV slot in without touching the loop. +- **Faithful visual language** restored (orange, not cyan). + +**Deliberately dropped from v1** +- **Multi-input mic modes** (XIASRI / sound_analysis_midi as ML inputs) — until the runtime-shaped MLP lands behind a passing parity check. v1 ships 2-input-correct, not 10-input-broken. +- **Runtime-shaped MLP itself** — deferred; it touches the sacred parity-tested core and must never ride the UI-rewrite wave. +- **C15** — pure placeholder; no UI scaffolding for vapor. +- **Stability↔Fluidity 4th axis**, **bundled all-in-one session presets**, **CV/gate serial**, **modular mega-mode** — deferred as parity clutter, not v1 mission work (modular is v1.5; the matrix is built to host it). +- **Backwards compatibility** — none; clean rewrite, per operator preference. + +--- + +## 6. Build roadmap + +Each phase is shippable and testable (Playwright + `__nisps` probe). Parity checkpoints called out. Effort is rough (ideal-engineer-days for one focused agent/operator pair). + +**Phase 0 — Scaffold & contract (≈2–3d).** Vite + SolidJS + TS skeleton; theme tokens (orange, lint allowlist); codegen wired with the **build-fails-on-arch-mismatch** rule; `ModeSchema` extended with `capability_class` + `tier`; `bus.ts` ported; `persist<T>` helper. *Test:* codegen golden test; typecheck green. + +**Phase 1 — The reactive spine against a stub (≈3–4d, de-risk wave).** `control-graph.ts` memo chain + single send-effect with a **stubbed `mlOutput`**; `input-store` + `output-store` (pure-fn pipelines ported, golden-tested; curve enums unified). *Test:* e2e asserts `setInputs → getOutputs` changes; no per-frame alloc (heap-snapshot fuzz). + +**Phase 2 — Console shell + Verdict cluster + RL-undo against the stub (≈4–5d, de-risk wave).** Dock + three-depth drawers; Verdict cluster; snapshot DAG + undo + A/B all against the stub. **User-test the three drawer depths with Dimi before wiring engines** (P3's mandate, Dimi's instinct). *Test:* e2e drives undo/A-B via probe. + +**Phase 3 — WASM ML bridge (≈4–5d).** `HeapVec` re-derive-on-access; `WasmIML` typed wrapper; pooled training worker; **`nisps_ml_loss_history` C API + real loss plumbing**; train on raw outputs. **Parity checkpoint:** main-thread inference matches native within 1e-5. *Test:* probe `train`/`infer`/`getLayerStats`; loss plot draws a real curve. + +**Phase 4 — Audio worklet + first real mode (≈4–5d).** Two-WASM worklet (`?worker&url`, hand-rolled imports, dedicated send-buffer); `WebAudioBackend`; `<GenericMode>` rendering PAFSynth (synth class). **Parity checkpoint:** `parity-check.sh` green; browser audio equivalent to firmware for PAFSynth. *Test:* per-mode live-feedback e2e (audio params change on input move). + +**Phase 5 — Schema-driven mode coverage (≈4–6d).** All in-scope engines via `<GenericMode>` (synth + sequencer + controller + visual classes; `SequencerLane`, `VisualEngine`, controller meters); capability-class switcher; heterogeneous I/O flags. *Test:* every mode passes the live-feedback e2e in CI; parity per mode. + +**Phase 6 — Control surface + navigation + pinning (≈4–5d).** Compound axes (memo fanout) + presets + trim-pot; full input pipeline (zoom/anchor/momentum); joy-map + trail + noise rings; region + param pinning (control-point `fixed`); interactive heatmap (3px/300ms craft). *Test:* probe-driven axis/pin/zoom; tactile-constant e2e. + +**Phase 7 — Diagnostics + Route matrix + exposure (≈4–5d).** Health drawer (weight-health glow, gradient-flow, layer-stats, input-space heatmap throttled, `spread`/`tame` lab); Route Full matrix + control-point rows + staged-unlock; per-group curves. *Test:* diagnostics render from real stats; probe `getLayerStats`. + +**Phase 8 — Backends + persistence + polish (≈3–4d).** `WebMidiBackend` + `OscBridgeBackend`; versioned persistence + base64 weights; session preset layers; auto-dissolve chrome; keyboard accelerators; Help. *Test:* full `run-all-tests.sh` (cmake + ctest + WASM + parity + lint + Playwright) green = **Verification chokepoint E**. + +**Post-v1 (gated):** runtime-shaped MLP behind parity → unlocks multi-input mic modes; modular mega-mode + Emitters×Targets population; C15; Stability axis. + +--- + +## 7. Risks & mitigations + +| Risk | Mitigation | +|---|---| +| Live-feedback regression recurs | One reactive path + per-mode e2e invariant in CI; no leaked write path (weights only mutate via `ml-store` actions that bump `weightsRevision`) | +| Memo purity violated (side-effect creeps into a memo) | Single documented send-`createEffect`; lint/review rule; memos must be pure | +| High pointer-event rate over-fires inference | Coalesce to display cadence via `batch()`+microtask; still reactive, rate-limited | +| Transferable buffer neutering | Dedicated re-filled send-buffer owned by the effect, separate from signal buffers | +| Stale `HEAPF32` view corruption | `HeapVec` re-derives the view on every access; sole WASM-memory access point; fuzz-tested | +| Drawer-depth model novel/undiscoverable | Prototype Console against `/dev/primitives`; user-test depths with Dimi (Phase 2); clear "⤢" affordances | +| Auto-dissolving chrome hides the loop | Never fade Verdict on first session; dock edge always visible; `?` help | +| Canvas-as-input vs clicking trail/pins | Hit-test priority: trail points + pins capture before joystick | +| 3px/300ms constants fragile cross-device | Port exact constants; e2e-test them | +| Snapshot DAG memory (large weight blobs) | Ring-bound (20/50); base64-compress; GC by tag priority (pinned/A nodes survive) | +| Parity drift on any core touch | Every core-touching phase gated on `parity-check.sh` green before UI lands | +| Probe ships to prod | Gate behind `?debug=1`; on by default in dev builds only | + +--- + +## 8. Open questions for Dimi (decide BEFORE building) + +Each is a real fork. I give options + my recommendation; I need your call. + +**A. Canonical tree / identity.** Is 2.0 the laptop `~/src/manifold` (`manifold::`, fresh repo — possibly already partly executed) or this VPS `MEMLNaut-NISPS` tree (`nisps::`)? **Recommendation:** target this VPS tree + `nisps::` (it's the live tree the recon ran against); the architecture is identity-agnostic and a rename is trivial. **But this blocks namespace/repo/codegen identity — I need your answer first.** + +**B. Mode taxonomy.** Four capability classes (Synth / Controller / Sequencer / Visual) vs the 8 engines + C15. **Recommendation (adopted):** add a `capability_class` schema field; the 8 engines are instances under Synth/Sequencer; Controller = `kRouteOutputsToEngine=false`; Visual = an output target. Switcher groups by class. Confirm this hierarchy vs an engine-flat switcher. + +**C. Progressive-disclosure model.** **Recommendation (adopted):** per-section drawer depth (Peek/Expand/Full), backed by schema-declared per-param/diagnostic `tier`; no global Advanced toggle. This unifies all five corpus proposals. Confirm it's canonical. + +**D. Runtime-shaped MLP vs fixed arch (the one I most want you to rule on).** **Recommendation (strong):** v1 ships the **honest fixed-2-input contract** (build fails on schema/arch mismatch); the runtime-shaped `Mlp` (your stated intent) is **deferred to post-v1 behind a passing parity check**, never bundled into the UI rewrite. Consequence: XIASRI / sound_analysis_midi multi-input modes are **firmware-only in v1** and show a "single-input in browser" badge. This is a materially different product surface than "all modes work in browser" — confirm you accept the v1 contraction. (The mission-fit and feasibility judges both flagged bundling the core rewrite into v1 as the project's biggest scoping hazard.) + +**E. `spread` prominence.** **Recommendation (adopted):** opt-in **lab toggle** in Health (firmware-init is the browser default, per the crystallization philosophy), with its *effect* surfaced through the **Boldness** axis so the primary affordance is musical. Confirm `spread` doesn't deserve its own primary slider regardless. + +**F. Trim-pot vs detach for axis overrides.** **Recommendation (adopted):** trim-pot offset stays default ("most musical"), with an "offset active" dot for visibility. Confirm, or whether detach should be a per-axis/per-user choice. + +**G. v1 scope of heavy features.** **Recommendation:** defer C15, the modular mega-mode (3-osc/4op/additive + shared mod pool + Emitters×Targets), and firmware-equivalent mic analysis to post-v1; v1 ships the pragmatic 4-feature analyser (read-only, not as ML input — see D) and keeps the matrix *built to host* the modular mode. Sequencer ShapeSeq freeze/delta **is** in v1. Confirm the line. + +**H. Desktop-first confirmation.** **Recommendation:** treat "desktop-first, touch-correct core" as decided, with touch-correctness of joystick/zoom/trail as a v1 acceptance criterion. Confirm the chat decision still holds over ALIGNMENT Q4's "defer until user data." + +**I. Session / bundled presets.** **Recommendation (adopted):** keep control-presets and synth-presets orthogonal; session presets are **composed layers** (weights / control / synth / mode independently saveable, soft-bundled on save), not a hard bundle. Confirm. + +--- + +## Appendix: provenance + +**P1 — research-instrument-minimalist ("The Listening Loop").** Mission-fit judge's top pick (9): its "capability budget, not screen budget" framing and its **honest fixed-2-input §7-D resolution** (build fails on mismatch; defer multi-input) are the correct posture for a research instrument and the only arch story that respects the crystallization + parity contracts — both grafted into this plan as load-bearing. UX-coherence ranked it second (7.5) — strong restraint, but it bets fast-path ergonomics on keyboard muscle-memory (least discoverable). Feasibility ranked it last (5) with a near-fatal flaw: it builds its "structurally unbreakable" claim on `createComputed`, the wrong reactive primitive — **rejected** in favor of P2/P3's memo chain. Net contribution: the minimalist philosophy + the honest arch fork. + +**P2 — realtime-frontend-architect.** Feasibility judge's clear winner (9): store-split-by-update-cadence, the `HeapVec` re-derive-on-access wrapper, transferable-buffer and pointer-coalescing rigor, and the OutputBackend adapter are the ship-grade backbone — all adopted. Mission-fit (8) credited its **live-feedback-as-one-owned-effect** as the single highest-value structural fix (Dimi's #1 frustration) plus the **train-on-raw-not-processed-outputs** research-validity fix. UX-coherence ranked it last (5) for its cardinal sin: it kept **both** a floating control bar and the dock, reproducing the exact two-homes conflict the brief said to resolve — **rejected** (we use the dock). Net contribution: the architecture spine + two correctness fixes + backends. + +**P3 — interaction-ux-designer ("The Console").** UX-coherence judge's winner (9): the dock-with-three-depth-states as the single disclosure gesture, the **Feel-vs-Health drawer split** (performer vs researcher mental modes), the floating-bar resolution (auto-open Peek + mini-axis ghost), the interaction-hazard handling (hit-test priority, randomize-as-long-press, never-fade-on-first-session), and the de-risk sequence (Console + RL-undo against a stub, user-tested first) — all adopted as the interaction spine. Mission-fit (8) credited the fearless-RL psychology. Feasibility (8) flagged one easily-corrected impurity: it described `postMessage` inside a memo — **corrected** here (side-effects live in one `createEffect`). Its gap: it left §7-D unresolved — **filled** with P1's honest contract. Net contribution: the entire UX/IA spine. + +**P4 — systems-information-architect ("altitude").** Lowest mission-fit (6) and mid feasibility (6)/UX (6.5) — not for bad ideas but for one **fatal v1-scoping flaw**: it committed the runtime-shaped MLP as a wave-1 deliverable, coupling the frontend rewrite to a cross-language rewrite of the sacred parity-tested core. All three judges said: adopt the **concepts, defer the core change**. So we take P4's **snapshot DAG** (unifying undo/A-B/trail/history) and its **control-point `off/fixed/live` enum** (collapsing the three overlapping ML-control systems) and its **`capability_class` schema field** — while explicitly gating runtime-shaped MLP behind a passing parity check, post-v1. Net contribution: the data-model concepts (history DAG, control-point enum, capability classes), minus the scoping trap. diff --git a/docs/redesign/rl-feedback-design.md b/docs/redesign/rl-feedback-design.md new file mode 100644 index 0000000..5e4f9ac --- /dev/null +++ b/docs/redesign/rl-feedback-design.md @@ -0,0 +1,225 @@ +# RL / Feedback Learning-Engine Design — Manifold + +*Design sign-off. Dated 2026-06-27. Supersedes the AVOID-out-of-scope decision in `docs/redesign/feedback-modes-port-spec.md` §2.5/§7. Author: lead architect, synthesising a 4-candidate / 4-judge design panel against verified source.* + +--- + +## 0. Decision summary + +| Setting | Default | Also selectable | +|---|---|---| +| `FEEDBACK_MODE` | **Explore and place** (Mode 2, positive-only) | **Geometric dislike** (Mode 1) | +| `SOLO_MODE` | **MaskGradients** (final-layer column freeze) | **ZeroLoss**, **DontCareExample** | + +**Winner of the panel:** *Sift* (musician-UX judge topPick, skeptic topPick; 2nd on feasibility and ML). It is the only candidate that does UX design rather than only porting design, and it satisfies the "Mode 2 is NEVER a dislike" requirement at the level of the performer's mental model. + +**Grafted onto Sift's felt-loop spine:** +- *Minimal-Divergence* — the **only candidate whose tree-read is correct** (verified below): the feedback C API, `MLHandle.feedback`, the `0xFEEDBACC0DE` salt, `CMakeLists.txt:59` registration, and parity Stage 5 ALL already exist. Its **append-only `FeedbackAction` enum** discipline and **controller-returns-action / caller-owns-training** boundary contract are kept verbatim. +- *Anchored-Manifold* — the **only candidate with correct provenance** (the upstream tip is `0a541cc "highlighting"`, verified; not the `e291192`/`abe93ec` the others cite). Its **one-`ReplayStore`-three-features** compression, its **bit-identical-frozen-column solo ctest**, and its **explicit TrainBatch-vs-SGD divergence flag** are adopted. +- *Anchor-First* — **keep replay/geometric logic OUT of the MLP kernel** (only minimal backprop hooks go in), and **port the `posMemCount==0` cold-start fallback faithfully** so dislikes-before-any-like don't destabilise the net. + +--- + +## 1. Verified ground truth (read this before building) + +All claims below were read from source on 2026-06-27. Where the four candidates disagreed, the verified fact decides. + +### 1.1 The scaffold already exists (Minimal-Divergence was right) +- `nisps/wasm/bindings.cpp:95-116` — `struct MLHandle` already holds `nisps::ml::FeedbackController<DefaultMLP> feedback;` + `feedback_static_scratch{}`, constructed `feedback(seed ^ 0xFEEDBACC0DEull)`. +- `nisps/wasm/bindings.cpp:439-535` — the 9 `nisps_ml_feedback_*` C functions (`set_mode`/`get_mode`/`exploring`/`learning_paused`/`set_focus`/`down`/`up`/`drag`/`static_output`) are **already wired**. +- `nisps/CMakeLists.txt:58-59` — `test_mlp_rl.cpp` **and** `test_mlp_feedback.cpp` are **already registered** in `nisps_core_tests`. +- `tests/cpp/parity_check.cpp:86` — `kVersion = 2u` already; `:198-205` — **Stage 5 (feedback) already exists** seeding `FeedbackController<ParityMLP> fb(kSeed ^ kFeedbackSalt)`. + +> Candidates 2/3/4 proposed *adding* `MLHandle.feedback`, *registering* `test_mlp_feedback.cpp`, and *creating* Stage 5. That is phantom work — they already exist. This design **edits** the existing `on_down` cases and **extends** the existing enum/API/parity-stage; it does not re-scaffold them. + +### 1.2 The current `feedback.hpp` (untracked, verified `nisps/ml/feedback.hpp:1-214`) +- `enum FeedbackMode {Avoid=0, RandomiseOutputs=1, RandomiseMlp=2}` (`:47-51`); `enum FeedbackAction {None=0,…Restore=7}` (`:56-65`). +- `on_down` Avoid case (`:103-105`) calls `mlp.move_weights(speed, spread, pin_mask)` — the **undirected-diffusion placeholder** the requirement now forbids for Mode 1. +- `RandomiseMlp` `on_down` (`:113-119`): 1st press enters (`snapshot_` + `draw_weights`), 2nd press **cancels**. Snapshot/restore is a byte-exact `get_weights`/`set_weights` round-trip (`:186-193`). +- The controller has its **own per-instance `Rng rng_`** (`:210`), seeded independently. + +### 1.3 The latest upstream algorithm (Anchored-Manifold's provenance was right) +The actual upstream tip is **`0a541cc "highlighting"`** at `/home/w1n5t0n/src/MEMLNaut-NISPS-upstream/src/memllib/examples/InterfaceRL.{hpp,cpp}`. Constants verified: +- `InterfaceRL.hpp:293` `kGeometricPushScale = 0.5f`; `:294` `kMaxDislikeMultiplier = 16`; `:296` `kCentroidK = 4`. +- `InterfaceRL.cpp:42-66` `_perform_dislike_action()`: nearby negative within Euclidean `0.05f` of `controlInput` → `reward = max(reward-1, -16)`; else `storeExperience(-1, controlInput, action)`; then `dislikeMultiplier_ = min(*2, 16)`. +- `InterfaceRL.cpp:602-627` k-NN centroid: average the `kCentroidK=4` positive memories nearest `controlInput` → `meanPositiveAction`. +- `InterfaceRL.cpp:664` proportional decay `reward += 0.0025f * max(|reward|, 1.0f)`; expired items removed; `dislikeMultiplier_` halved per expiry (`:752-760`). +- `InterfaceRL.cpp:713` `pushStep = clamp(|avgRewardNeg|, 0.25f, 1.0f) * kGeometricPushScale`. +- `InterfaceRL.cpp:721-735` per negative: `dir[j] = neg_action[j] - meanPositiveAction[j]`; `len = ||dir||`; `useRandom = (len <= 1e-4f)`; `effectivePushStep = pushStep / (1 + len)`; for each `j` gated by `activeDims_`: `d = useRandom ? (rand()&0xFF/127.5 - 1) : dir[j]/len`; `target[j] = clamp(neg_action[j] + d*effectivePushStep, 0, 1)`. +- `InterfaceRL.cpp:742-743` `negLRRatio = 0.5f - 0.4f*negFraction`; train geometric targets at `lr*negLRRatio`. +- `InterfaceRL.cpp:746` **cold-start fallback** when `posMemCount==0`: `TrainBatch(tsNegative, lr * 0.1f * avgRewardNeg, …)` — negative-LR training, no geometric push. + +### 1.4 The deepest parity hazard (Anchored-Manifold / Anchor-First flagged it) +Upstream `optimise()` trains with **`TrainBatch`** (shuffled, `batchSize≈8`, separate positive/geometric batches at *two distinct* learning rates). The nisps `MLP::train()` (`mlp.hpp:256-300`) is **unshuffled per-sample SGD over the dataset insertion order** (its own comment: *"TrainBatch shuffles, but we're not implementing batch yet"*), has **no reward field** and **no train-toward-arbitrary-targets** entry. Reproducing the two-batch / dynamic-`negLRRatio` dynamics on a non-batch trainer is the real `browser != firmware` line. **We accept behavioural (not bitwise) parity with firmware here, and pin `native == WASM` at 1e-5.** Recorded in `ALIGNMENT.md`. + +### 1.5 The injection points the masking/solo mechanism needs +- `mlp.hpp:281` — the per-sample `std::array<float,NOut> deriv{}` is the **exact place** to zero the output-error of non-soloed / don't-care dims before `backprop_` (`:291`). One branch per output dim. +- `rl.hpp:74-99` — `move_weights_layer` already skips the final-layer weight **column + bias** when `is_final_layer && output_pin_mask[node] != 0`. Solo's column-freeze is the **bitwise inverse** of this mask. Note the existing rule (`:88-96`): the RNG is advanced even on skipped weights so the stream is pin-independent — keep that invariant. +- `mode-runtime.ts:548-565` `thumbsDown` currently builds a pin mask and calls `moveWeights` — **no example stored**, no training. `thumbsUp` (`:523-546`) computes the `(features, labels)` and calls `addExample` + `trainOnCurrent`. + +--- + +## 2. FEEDBACK_MODE = two operator-switchable modes + +The selector is `exploration-store.ts` state `feedbackMode: 'explore-and-place' | 'geometric-dislike'`, surfaced in the **learning-behaviour dock panel** (§7). British spelling in all copy ("randomise", "auditioning", "centre", "behaviour"). + +### 2.1 Mode 1 — "Geometric dislike" (negative example + geometric push-away) + +**Felt loop:** you have a region you like; thumbs-down carves the current sound *away* from the liked centroid. Directed repulsion, audibly directional — the fix for today's inaudible undirected diffusion (`findings-feedback-behaviour.md §5`). + +**Algorithm — port `InterfaceRL.cpp` (0a541cc) into nisps core, keeping parity.** Because nisps has no async `optimise()` driver, the press-time half (`_perform_dislike_action`) and the optimise half collapse into one synchronous core call. + +On thumbs-down at `controlInput x_neg` with heard action `a_neg`: +1. `replay.deepen_or_store_negative(x_neg, a_neg, radius=0.05f)` — nearby negative → `reward = max(reward-1, -16.f)`; else store `reward=-1`. `dislikeMultiplier_ = min(*2, 16)`. (`InterfaceRL.cpp:42-66`) +2. If `replay.positive_count() == 0` → **cold-start fallback**: `train_targets(x_neg → a_neg, lr * 0.1f * avgRewardNeg, …)` (`InterfaceRL.cpp:746`). Surface a one-time UI prompt *"like a few sounds first to teach the system what to move away from"* (closes the cold-start gap all four judges flagged). +3. Else: `replay.knn_positive_centroid(x_neg, k=4, &mean)` (`:602-627`); `pushStep = clamp(|avgRewardNeg|, 0.25f, 1.0f) * 0.5f`; per active/soloed dim `j`: `dir[j]=a_neg[j]-mean[j]`, `len=||dir||`, `effectivePushStep=pushStep/(1+len)`, `d = (len<=1e-4f) ? rng_.next_float_signed() : dir[j]/len`, `target[j]=clamp(a_neg[j]+d*effectivePushStep, 0, 1)`; non-active dims keep `a_neg[j]`. +4. `train_targets(x_neg → target, lr * negLRRatio, …)` with `negLRRatio = 0.5f - 0.4f*negFraction`. +5. `replay.decay_negatives()` each call: `reward += 0.0025f*max(|reward|,1)`, evict `reward > -0.01f`. + +**Determinism carve-out:** the upstream `useRandom` branch uses libc `rand()&0xFF`; we substitute `rng_.next_float_signed()` (the controller's own per-instance `nisps::Rng`). Both native and WASM run the same Rng from the same seed → 1e-5 parity holds. This branch only fires when a disliked action sits exactly on the centroid. Record as the single deliberate firmware divergence in `ALIGNMENT.md`. + +**On thumbs-up** (Mode 1): `replay.store(+1, x, a)` so the centroid sees positives; caller also runs the existing `addExample`+`train` like today. + +**Files touched:** edit `feedback.hpp` `on_down` Avoid case (`:103-105`) → route to the geometric path; add `replay_` member; `on_up` LikeStore → `replay.store(+1,…)`. New `nisps/ml/replay.hpp` + `nisps/ml/geo_push.hpp` (§4). `move_weights` stays reachable as a **legacy `Diffuse` sub-mode** for A/B comparison (cheap insurance, lets the A/B-compare feature contrast geometric vs diffusion — graft from Minimal-Divergence). + +### 2.2 Mode 2 — "Explore and place" (positive-only; NEVER a dislike) — **DEFAULT** + +**Felt loop (Sift's north star):** *"randomise, explore, oh I like that → put it in that corner; meh, randomise…"*. The performer never reasons about "away from what" — they collect sounds they like. Place-by-corner is the **highest audible-effect-per-action gesture in the system**: one drop = one heard anchor the real model then interpolates. + +Built on the existing `RandomiseMlp` scaffold (`feedback.hpp:113-119,183-201`) but to the stricter spec. Lifecycle state machine in the reworked `FeedbackController`, `FeedbackMode::ExploreAndPlace`: + +1. **ENTER** (verdict-down/explore): `enter_scratchpad(mlp, spread)` — snapshot the real trained net into `snapshot_` (the set-aside model; existing `get_weights` round-trip), set `learning_paused_=true` (gates `trainOnCurrent` + auto-explore, mirroring upstream `optimiseSometimes` early-return), then `draw_weights(spread)` → **random scratchpad net**. +2. **SCRATCHPAD OPS — exactly two, position-agnostic, never trained, never a dislike:** + - `reroll()` = `draw_weights(spread)` again ("meh, randomise…"). Re-press while exploring **re-rolls** (returns `Reroll`); a dedicated cancel gesture / mode-switch aborts. (This changes the old 2nd-press-cancels semantics — deliberate; rewrite the affected ctest.) + - `nudge(speed, spread)` = a small `move_weights` on the scratchpad with a templated undo ring `scratch_undo_` (depth `D`; **WASM `D=4`, firmware `D=2`** via template param — Sift's free SRAM tunable). `undo_nudge()` pops. Reuses the byte-exact `get_weights`/`set_weights` round-trip (graft: no new parity surface). New actions `ScratchNudge`, `ScratchUndo`. + - The scratchpad is **NEVER trained and NEVER reads replay** — auditioning is pure forward inference as the user sweeps the joystick. +3. **PLACE** (user likes a sound): press → **choose a location** in input space → release. `place_begin()` freezes the current scratchpad output vector into `placed_out_` (so the heard sound is held while the user aims; emitted via the existing `static_output` bypass hook so the audition doesn't change). The TS runtime owns the "aim" UI (a JoyMap drag). On release: caller sets the scratchpad input to `chosen_input`, runs `mlp.process()`, reads outputs, calls `place_commit(chosen_input, scratch_out_at_chosen)` → appends a **positive anchor** and returns `AnchorPlaced`. The controller does NOT itself run inference (keeps it pure — Minimal-Divergence's boundary contract). +4. **WARM-START INTERPOLATION** (explicit "Done"/finalise gesture): restore the set-aside net (`set_weights(snapshot_)`), then warm-start it to interpolate **all placed anchors**. **Decision (graft from all four risk-lists): warm-start is ADDITIVE — anchors are added to the existing dataset, NOT `clear_examples()`+refit** — so the user's prior thumbs-up likes are not clobbered. The caller loops `addExample(input, output)` over `replay.anchors()` then `train(lr, maxIter, minErr)` (caller owns training). "Warm-start" = begin from the *restored real net*, not a fresh draw, so prior structure is preserved. Returns `WarmStarted`; `learning_paused_` flips false. +5. **ABORT** (mode-switch / cancel): restore the set-aside net, discard anchors not yet finalised. + +**Anchor store** = the same `ReplayStore` instance, positives only (§4). **Coupling decision (Sift's flag):** unifying Mode-1 replay and Mode-2 anchors into one store means a placed anchor also feeds Mode-1's positive centroid. This is a *feature* (mixed-mode sessions), but expose an operator toggle `unifyMemories` (default on) so they can be isolated into two `ReplayStore` instances if surprising. + +**Why Mode 2 is the default:** highest audible-effect-per-action, no negative concept to learn, and it directly realises the operator's stated felt loop. Mode 1 is the precision/sculpting tool for when you already have a liked region. + +--- + +## 3. SOLO / arm per output + +State: `solo_mask_ : std::array<std::uint8_t, kNOut>` on the controller (`1`=armed/soloed, `0`=frozen; all-zero / empty ⇒ none soloed ⇒ normal training). API `set_solo_mask(span)` / `clear_solo_mask()`, C API `nisps_ml_feedback_set_solo(ml, mask, n)` (mirrors the existing `set_focus` at `bindings.cpp:485`). One operator setting `SOLO_MODE {MaskGradients=0, ZeroLoss=1, DontCareExample=2}`. + +All three flow through **one injection point** — the per-sample `deriv` array at `mlp.hpp:281`, fused per Sift's graft (zeroing the loss on non-soloed outputs *is* the don't-care mask). + +### 3.1 MaskGradients (DEFAULT) — column-freeze +Predictable: "only this output moves." Derive `pin_mask = bitwise-NOT(solo_mask)` over outputs. In a new `train_masked(lr, max_iter, min_err, out_mask)`: zero `deriv[j]` for non-soloed `j` **before** `backprop_`, **and** freeze hidden layers when any solo is active (skip `layer0_/1_/2_.apply_grad`; apply only `layer3_` for soloed columns). Result: non-soloed final-layer columns AND all hidden weights are **bit-identical** after training. This is the exact inverse of `rl.hpp:74-99`'s pin gating — minimal new code, reuses proven machinery. + +### 3.2 ZeroLoss (selectable) — expressive-but-bleeds +Zero `deriv[j]` for non-soloed `j` but **let hidden layers update**. The soloed output gets the full network's expressive capacity, but non-soloed outputs can drift via shared hidden features. One-line variant of 3.1 (drop the hidden-layer freeze). + +### 3.3 DontCareExample (selectable) — most faithful to "store with a mask" +Each `ReplayStore` item carries `uint8 mask[NOut]`. An example placed while output `k` is soloed stores `mask = {0…1@k…0}`; `train_masked` consumes the per-example mask so other outputs at that input are never pulled toward a stale label. Survives later training. Costs `Cap*NOut` bytes. + +### 3.4 Honest limit (Sift's correctness graft — into product copy + ALIGNMENT.md) +**On a shared-trunk MLP, NO realisation both fully isolates AND stays expressive.** MaskGradients is predictable but weak (only the final linear column moves; hidden capacity frozen). ZeroLoss is expressive but bleeds through shared hidden weights. State this to the operator in the dock panel ("solo freezes the rest as far as a shared network allows") rather than overselling "leaves others unchanged". MaskGradients is the safe default because its guarantee is *provable* (§6.3 bit-identical ctest). + +**Solo is honoured in both modes:** Mode-1 geometric `target` only pushes soloed dims (unify `solo_mask_` with the `activeDims_` gate at `InterfaceRL.cpp:730` — Anchor-First's graft); Mode-2 warm-start trains anchors only on soloed dims when solo is active. + +--- + +## 4. Core component changes (keep replay OUT of the MLP kernel — Anchor-First) + +### New `nisps/ml/replay.hpp` +`ReplayStore<NIn, NOut, Cap>` — fixed `std::array` ring (no heap; **WASM `Cap=64`, firmware `Cap=16-32` via template param** — Anchored-Manifold's free SRAM/centroid tradeoff). Item `{std::array<float,NIn> input; std::array<float,NOut> action; float reward; std::uint8_t mask[NOut];}`. Methods (all deterministic, per-instance `nisps::Rng`, **no libc rand**): +- `deepen_or_store_negative(x, a, radius=0.05f)` (`InterfaceRL.cpp:42-66`) +- `store(reward, x, a, mask)` / `positive_count()` +- `knn_positive_centroid(x, k=4, &mean, &count)` — linear scan, fixed top-k insertion (no `std::sort`, no heap). **Deterministic tie-break by index** and **fixed accumulation order** so native==WASM (the classic float-sum parity trap). +- `decay_negatives()` (`reward += 0.0025f*max(|reward|,1)`, evict `> -0.01f`) +- `anchors()` / `anchor_count()` accessors for Mode-2 warm-start replay. + +### New `nisps/ml/geo_push.hpp` +Pure free function `compute_push_targets(replay, control_input, solo_mask, rng, &target_buf)` implementing `InterfaceRL.cpp:602-738` exactly. Writes into a caller-supplied scratch buffer (no heap). Keeps the geometric math out of both MLP and controller. + +### `nisps/ml/mlp.hpp` — minimal backprop hooks only +- `train_targets(span inputs, span targets, span sample_weights, span out_mask, lr)` — trains toward *computed* target vectors (Mode-1 geometric batch; targets are computed, not stored labels). Reuses `forward_`/`backprop_`/`apply_grad`. +- `train_masked(lr, max_iter, min_err, span out_mask)` — the solo path: zeroes `deriv[j]` at `:281` for masked `j`; in MaskGradients freezes hidden `apply_grad`. +- Factor the inner forward→loss→backprop of `train()` into a `train_pair_` helper so both reuse the proven backprop (Minimal-Divergence graft). **No replay/centroid/anchor logic in MLP.** + +### `nisps/ml/feedback.hpp` — edits (append-only enum) +- Add `ReplayStore<NIn,NOut> replay_`, `solo_mask_`, `scratch_undo_` ring (templated depth), `placed_out_`, `SoloMode` field. +- Edit `on_down` Avoid case → geometric push via `replay_` + `geo_push` + `train_targets`; keep a `Diffuse` legacy sub-mode. +- Edit `RandomiseMlp`/`ExploreAndPlace` `on_down` so re-press = `Reroll`. +- Add `nudge`/`undo_nudge`, `place_begin`/`place_commit`, `finalise`/`abort`, `set_solo_mask`/`clear_solo_mask`, `set_solo_mode`. +- **Append** new `FeedbackAction` values keeping existing numeric values stable: `GeometricPush=8, ScratchNudge=9, ScratchUndo=10, AnchorPlaced=11, WarmStarted=12` (Minimal-Divergence's single most parity-safe decision — never renumber the TS↔C++ contract). + +### `nisps/wasm/bindings.cpp` — extend the existing block (`:439-535`) +Add: `nisps_ml_feedback_set_solo(ml,mask,n)`, `_set_solo_mode(ml,mode)`, `_nudge(ml,speed,spread)`, `_undo_nudge(ml)`, `_place_begin(ml,current_out)`, `_place_commit(ml,chosen_input,scratch_out)`, `_anchor_count(ml)`, `_get_anchors(ml,buf)`, `_finalise(ml)`, `_abort(ml)`. Existing `_down/_up/_drag/_static_output` stay; `_down` now returns the new action ints. Reuse the existing scratch buffers + guard/cast style. + +### TS +- `playground/src/ml/types.ts` — add the new `_nisps_ml_feedback_*` decls; extend `FeedbackMode`/`FeedbackAction`/add `SoloMode` enums (numeric parity with C++). +- `playground/src/ml/wasm-iml.ts` — thin wrappers (`feedbackNudge`, `feedbackPlaceCommit`, `feedbackFinalise`, `feedbackSetSolo`, `getAnchors`); reuse `pinMaskBuf` (`:225`); add a small anchor heap buffer. +- `playground/src/modes/mode-runtime.ts` — route `thumbsDown` (`:548`) per active `feedbackMode`; Mode-2 wires press→aim→release to `place_begin`/`place_commit`; a "Done" button → `finalise` → loop `getAnchors`→`addExample`→`train`. Gate `trainOnCurrent` (`:516`) and auto-explore on `learning_paused()`. +- `playground/src/stores/exploration-store.ts` — add `feedbackMode`, `soloMode`, `soloMask`, `exploring`, `anchorCount`, `unifyMemories` + selectors. + +--- + +## 5. Phased integration plan (TS-prototype-first; each step gated on `parity-check.sh`) + +**Audible validation is the gate, not green tests.** Prototype the arithmetic in pure TS — driven through the EXISTING `WasmIML` primitives (`drawWeights`, `moveWeights`, `addExample`, `train`, `getWeights`/`setWeights`, `setInput`/`process`/`getOutputs`, verified at `wasm-iml.ts:293-476`) — before any new C++. + +### Phase 0 — TS prototype (no new WASM) +A throwaway TS module mirrors `ReplayStore` + `geo_push` + `train_masked`, computed in TS and fed to the real net via existing calls. Oracles: +- **Mode 1:** like 2-3 sounds in distinct corners; dislike a sound near one like; confirm the heard 126-vector at that input moves **audibly AWAY** from the liked timbre (not random wobble). A/B against today's `moveWeights` thumbs-down using the existing A/B-compare feature — directionality must be **perceptible** (Sift's oracle). +- **Mode 2:** randomise → sweep-audition → "like that" → place in a corner → randomise → place opposite corner → Done → sweep the JoyMap and confirm a **smooth morph between the two placed timbres**, and that re-randomising no longer destroys them. +- **Solo:** solo output 0, train, confirm via the heatmap / weight-health views that only that param's mapping changed (MaskGradients: bit-stable elsewhere). +- Use `window.__nisps` (`setInputs`/`getOutputs`/`train`/`thumbsDown`/`getLayerStats`) for scripted Playwright checks. + +**Gate:** felt loops audibly correct → freeze `kCentroidK`, `pushStep`, warm-start iteration count, undo depth. Only then crystallise. + +### Phase 1 — C++ core +Add `replay.hpp`, `geo_push.hpp`, the `mlp.hpp` hooks, the `feedback.hpp` edits. Build `nisps_core_tests` (`-Wall -Wextra -Werror`). **Gate:** new ctest cases green (§6.1); `scripts/parity-check.sh` still 1e-5 (existing Stage 5 unchanged at this point). + +### Phase 2 — C API +Add the new `nisps_ml_feedback_*` exports. Rebuild WASM (`scripts/build-wasm.sh`). **Gate:** `parity-check.sh` green after Stage-5 extension (§6.2); `kVersion` bump. + +### Phase 3 — TS FFI +`types.ts` decls + enums, `wasm-iml.ts` wrappers. **Gate:** `bun run typecheck`; a TS↔WASM smoke test reproducing a Phase-0 loop through the real FFI matches the TS prototype's outputs; `parity-check.sh` green. + +### Phase 4 — UI selector in the learning-behaviour dock panel +`exploration-store` state + the dock-panel `FEEDBACK_MODE` selector ("Explore and place" / "Geometric dislike"), `SOLO_MODE` selector, exploring/anchor indicators, the cold-start prompt, the honest solo-limit copy (British spelling). **Gate:** `bunx playwright test` (the e2e felt-loop specs); full `scripts/run-all-tests.sh` (chokepoint E) green. + +### Phase 5 — firmware readiness (gated, not on the critical path) +Confirm RP2350 SRAM budget with firmware `Cap`/undo-depth template params (`ReplayStore<2,126,16>` + anchors + `scratch_undo_` depth 2). Verify chokepoint A (audio correct) / B (no perf regression) before flashing. Re-pin the orphaned memllib `4733ca0` to `0a541cc`. + +--- + +## 6. Parity + perf contract + +Native==WASM within 1e-5 is preserved because **every new operation is deterministic f32 arithmetic on the per-instance `nisps::Rng`** (no libc `rand()` anywhere). Both sides run the same Rng from the same seed (`kSeed ^ kFeedbackSalt`, matching the `MLHandle` `0xFEEDBACC0DE` salt). + +### 6.1 Host ctest (`tests/cpp/test_mlp_feedback.cpp`, already registered) +Rewrite the cases that encode the old 2nd-press-cancels state machine (deliberate break). Add: replay dedup/deepen at 0.05; k-NN centroid selection + deterministic tie-break; geometric push direction sign (target moves away from centroid); taper; cold-start `posMemCount==0` fallback; **solo bit-identical assertion** — after `train_masked` with MaskGradients, non-soloed final-layer columns are `==` (not 1e-5-near) to pre-train (Anchored-Manifold's stronger-than-parity oracle); scratchpad-never-trained invariant; place→warm-start reachability (both anchors reproduced at their corners); determinism under fixed seed. + +### 6.2 Native↔WASM parity (`parity_check.cpp` + `parity_wasm.mjs`, bump `kVersion`) +Extend Stage 5: **(S5a Mode 1)** seed `ParityMLP`+`ReplayStore`, store 3 fixed positives + 1 negative, run geometric push (centroid→target→`train_targets`), push 126 post-train probe outputs both sides. **(S5b Mode 2)** enter scratchpad, reroll×2, nudge, undo, place 2 anchors at fixed inputs, finalise (warm-start at fixed lr/iters/seed), push probe outputs + 12 weight probes. **(S5c Solo)** set `solo_mask={1,0,…}`, `train_masked`, assert non-soloed columns byte-identical pre/post. `scripts/parity-check.sh` float32-diffs at 1e-5 — new floats covered automatically. + +### 6.3 Perf contract +`replay.hpp` + anchors + `solo_mask_` + `scratch_undo_` are all fixed `std::array` (Cap/depth compile-time, zero heap). Hot loops O(Cap*NOut) over fixed arrays; `roll`/`centroid`/`train_pair_` marked `NISPS_FORCE_INLINE` where hot; `.f` on every literal (`0.0025f`, `0.25f`, `0.5f`, `16.f`, `0.05f`). No virtual dispatch (plain template). Feedback runs on the **control core, never the audio ISR** (mirrors firmware `loopCallback`) → chokepoint B structurally satisfied. SRAM budget tuned per-target via `Cap`/depth template params. + +--- + +## 7. UI — learning-behaviour dock panel + +- **`FEEDBACK_MODE` segmented control:** *Explore and place* (default) / *Geometric dislike*. One-line description per mode in the operator's language. +- **`SOLO_MODE` selector** (visible when any output is armed): *Mask gradients* (default) / *Zero loss* / *Don't-care example*, with the honest "as far as a shared network allows" caveat. +- **Exploring indicator** + **anchor count** during Mode-2 sessions; a **Done / finalise** button and a **cancel** gesture; nudge + undo controls. +- **Cold-start prompt** (Mode 1, zero positives): *"Like a few sounds first so the system knows what to move away from."* +- **`unifyMemories` toggle** (advanced): share likes/dislikes across modes (default on). +- Copy uses British spelling throughout. + +--- + +## 8. ALIGNMENT.md updates (same commit as the code) +1. **RETRACT** the `feedback-modes-port-spec.md §2.5/§7` "geometric push out of scope / AVOID = move_weights" accepted-divergence note — Mode 1 now ports the latest upstream. Without this, a future session will "rediscover" the geometric push as scope creep. +2. Record the **single deliberate firmware divergence**: the `useRandom` degenerate branch uses `nisps::Rng`, not libc `rand()` (value generated, never compared; native==WASM holds). +3. Record the **TrainBatch-vs-unshuffled-SGD** behavioural divergence: nisps trains per-sample SGD, not shuffled two-LR batches — `browser != firmware` behaviourally, by design (separate verification targets). +4. State the **shared-trunk solo limit** as accepted: no realisation both perfectly isolates and stays expressive. +5. Re-pin orphaned memllib `4733ca0` → `0a541cc`. \ No newline at end of file diff --git a/docs/redesign/upstream-firmware-survey.md b/docs/redesign/upstream-firmware-survey.md new file mode 100644 index 0000000..0bbca5c --- /dev/null +++ b/docs/redesign/upstream-firmware-survey.md @@ -0,0 +1,253 @@ +# Upstream Firmware Survey — git archaeology of the MusicallyEmbodiedML ecosystem + +_Dated 2026-06-27. Author: research agent (read-only). All SHAs from a fresh `git fetch --all` + memllib submodule-gitdir fetch (incl. operator's memllib fork `monkey-w1n5t0n/memllib`)._ + +## TL;DR — the divergence headline + +**origin/main and upstream/main are two different codebases sharing one ancestor.** They forked at +`6efbe9c` ("voicing", 2026-04-14) and have NOT touched a common file since. origin/main is the +**complete C++20 + SolidJS clean rewrite** (`nisps/`, `playground/`, `firmware/`, schemas/codegen). +upstream/main is the **original Arduino `.ino` + `modes/*.hpp` firmware tree**, which the lab kept +developing in parallel. The 49 commits upstream has that origin lacks are **real firmware features on +an architecture origin deliberately abandoned** — they are *ports*, not *merges*. There is no clean +fast-forward or 3-way merge path; every integration is a re-implementation against the new core. + +``` + 6efbe9c "voicing" (2026-04-14) ← merge-base, last shared tree + / \ + origin/main (193 ahead) upstream/main (49 ahead) + C++20/SolidJS rewrite old .ino/modes firmware tree + pins memllib 4733ca0 (orphan) pins memllib e291192 (+62 vs origin's line) +``` + +--- + +## 1. Branch map + +### 1.1 Superproject — `origin` (operator fork) and `upstream` (lab) + +| Branch | Remote | Last push | Author | vs its main¹ | Purpose | Judgment | +|---|---|---|---|---|---|---| +| `main` | origin | 2026-06-16 | monkey-w1n5t0n | — | The C++20/SolidJS rewrite. | canonical | +| `feat/feedback-explore-modes` | origin | 2026-06-08 | w1n5t0n | +42 / −193 | Pins memllib `abe93ec` = the `FEEDBACK_MODE` (avoid / randomise-outputs / randomise-MLP) commit. Source material for `docs/redesign/feedback-modes-port-spec.md`. | **real RL work to port** (already spec'd, not yet in main) | +| `feat/useq-celium-opus46` | origin | 2026-06-14 | w1n5t0n | +2 / −36 | MEMLCelium useq experiment (Opus-4.6 session). | experiment — superseded by opus47 | +| `useq-celium-opus47` | origin | 2026-04-20 | w1n5t0n | +6 / −36 | Later MEMLCelium useq iteration. | experiment | +| `feat/unified-preset-system` | origin | 2026-04-16 | w1n5t0n | +46 / −55 | Preset-system spike; heavily diverged, stale. | abandoned/spike | +| `port-solidjs` | origin | 2026-06-14 | w1n5t0n | +31 / −66 | Early SolidJS port scaffolding; main has moved well past it. | superseded by main | +| `main` | upstream | 2026-06-22 | chriskiefer | — | Old firmware tree; LiveFX/VerbFX/DJFX/staticmlp/jolts/noise. | **real firmware to port** | +| `SB2026` | upstream | 2026-05-15 | chriskiefer | +0 / −30² | Superbooth-2026 build: mode sample-rate, **DISLIKES**, screen layout, sample player. **Fully merged into upstream/main.** | merged — port via memllib | +| `SaxFX` | upstream | 2026-05-05 | chriskiefer | +0 / −42² | Sax FX app. **Fully merged into upstream/main.** | merged — historical | +| `bettysax` | upstream | 2026-02-21 | chriskiefer | +1 / −72² | Bettysax; 1 extra commit `c5f0314 "demo"` not in main. | novelty/demo | + +¹ "+X / −Y" = (commits the branch is ahead of its own main) / (commits behind). For `git rev-list --left-right --count main...branch` the **left** count is main-only (= how far the branch is *behind*) and the **right** count is branch-only (= *ahead*) — read them in that order. For upstream branches, main = upstream/main. +² SB2026 / SaxFX show ahead=0 because they are *contained in* upstream/main (`git rev-list upstream/main..branch` = 0). The "−30/−42" is just how far main has advanced past their tip. + +### 1.2 Submodule — `src/memllib` (lab `origin` = MusicallyEmbodiedML/memllib; `w1n5t0n` = operator fork) + +memllib is where the **RL interface (`InterfaceRL`) and the DISLIKES/feedback logic actually live** — the +superproject only pins it. Two pins matter: + +- **origin/main superproject pins memllib `4733ca0`** — an **orphaned commit**: not reachable from any + ref in the lab repo *or* the operator's fork. Likely a rebased/abandoned tip. Cannot be diffed by SHA; + treat origin/main's memllib as "stale, off-graph." +- **upstream/main superproject pins memllib `e291192`** ("l r input swap", 2026-06-22) — current lab tip, + and it **already contains the SB2026 dislikes/rlview work** (`80420d1 "rlview and dislike"`, + `d9bdcd6 "remove dislike opts screen"`). +- **origin/feat/feedback-explore-modes pins memllib `abe93ec`** ("InterfaceRL: add FEEDBACK_MODE setting", + 2026-06-08, authored by w1n5t0n on the operator's fork). This is the operator's own RL feedback commit, + branched off lab `616b8e7` ("cc select"); it is +1/−6 vs `e291192`. + +| memllib branch | Last push | Author | ahead of main³ | Purpose | Judgment | +|---|---|---|---|---|---| +| `main` (e291192) | 2026-06-22 | chriskiefer | — | Current lab RL lib (incl. dislikes). | canonical lib | +| `SB2026` (1997699) | 2026-05-15 | chriskiefer | +0 / −20 | Superbooth dislikes/rlview. **Fully merged into main.** | merged | +| `w1n5t0n/feat/feedback-explore-modes` (abe93ec) | 2026-06-08 | w1n5t0n | +1 / −6 | Operator's FEEDBACK_MODE 3-mode enum. | **real — already spec'd to port** | +| `maincandidate` | 2025-11-06 | chriskiefer | +54 | Pre-2026 staging. | historical | +| `displayview` | 2025-10-13 | chriskiefer | +82 / −11 | Display work. | historical | +| `bettysaxanalysis` | 2025-09-03 | A. Martelloni | +124 | Sax analysis. | historical | +| `display2`, `pafcarl`, `pafsynth`, `bunty_new`, `studiofx`, `xiasri`, `rlversion`, `sdcard`, `superbooth`, `display`, `imlzoom`, `tomchris`, `iceland`, `chris`, `features` | 2025-05…08 | chris / Martelloni | +120…+243 | 2025 app/feature branches. | historical/abandoned | + +³ `git rev-list --left-right --count main...branch` inside the memllib gitdir. + +### 1.3 Tags + +| Tag | Points at | Date | In upstream/main? | In origin/main? | Notes | +|---|---|---|---|---|---| +| `LiveFX_1.0.0` | `701f2d9` | 2026-06-22 | yes | **no** | Newest firmware release; old tree. | +| `VerbFX_1.1.0` | `a18d9c6` | 2026-06-16 | yes | **no** | Large-verb FX release. | +| `DJFX_TR6S_MICROQ` | `ed4ac6d` | 2026-05-25 | yes | **no** | DJFX + TR-6S + MicroQ firmware. | +| `MEMLCelium_0_1_alpha` | `1e441cb` | 2026-04-20 | yes | **no** | MEMLCelium alpha; old tree. | +| `PAF_1_1_0` | `bae80fb` | 2026-03-29 | yes | **yes** | Pre-fork; in both. | +| `VerbFX` | `bae80fb` | 2026-03-29 | yes | **yes** | Same commit as PAF_1_1_0; pre-fork. | + +Only the two pre-fork tags (`bae80fb`) exist in origin/main. Every release tag from `MEMLCelium_0_1_alpha` +onward marks **old-tree firmware origin/main never received**. + +### 1.4 Sibling org repos (`gh repo list MusicallyEmbodiedML`, 70 repos) + +Active / firmware-relevant: +- **`memlp`** (2026-06-24) — the underlying NN library for microcontrollers; memllib's dependency. Recently active. +- **`memllib`** (2026-06-22) — the RL/interface lib (the submodule). The live one. +- **`MEMLNaut`** (2025-11-13) — the hardware/board repo. +- **`memlp_tests` / `memlp_test` / `RLTests`** — test harnesses for the NN/RL libs; relevant if porting RL behavior and wanting reference vectors. +- **`musicallyembodiedml.github.io`** (2026-06-22) — the docs site referenced by CLAUDE.md. + +Everything else is a 2024–2025 per-app fork (PAF, Xiasri, studioFX, euclidean, FM, subtractive, XMOS-era +boards, etc.) — historical context, **not integration targets**. The whole "one repo per app" pattern is +exactly what the `nisps/` rewrite consolidates; these forks are the museum. + +--- + +## 2. Dated timeline + +| Date | Event | +|---|---| +| 2026-03-29 | `bae80fb` — `PAF_1_1_0` / `VerbFX` tags. Last point both trees share a tag. | +| **2026-04-14** | **`6efbe9c` "voicing" — fork point (merge-base of origin/main & upstream/main).** | +| 2026-04-20 | `MEMLCelium_0_1_alpha` (`1e441cb`); memllib `psram`. First upstream-only release. | +| 2026-05-05 | upstream `SaxFX` branch tip (`33b6542 "memlib sub"`). (NB: `fcef420 "sax fx ftw"` is a *2026-05-15* commit on upstream/main, **not** the SaxFX tip — earlier draft conflated them.) | +| 2026-05-15 | upstream `SB2026` tip (`c617fcc` "mode sample rate, **dislikes**, screen layout, sample player"). memllib `SB2026` (`1997699`). | +| 2026-05-25 | `DJFX_TR6S_MICROQ` (`ed4ac6d`). | +| 2026-06-08 | origin `feat/feedback-explore-modes` — operator's `FEEDBACK_MODE` work; memllib fork `abe93ec`. | +| 2026-06-16 | **origin/main HEAD** (`f256217`, AudioWorklet fix). `VerbFX_1.1.0` (`a18d9c6`) on upstream. | +| 2026-06-22 | **upstream/main HEAD** (`701f2d9` "livefx"); `LiveFX_1.0.0`. memllib main `e291192`. | +| 2026-06-27 | This survey. | + +--- + +## 3. Git relationships + +- **Merge-base (superproject):** `6efbe9c` ("voicing", 2026-04-14). +- **origin/main vs upstream/main:** `git rev-list --left-right --count` = **193 / 49** (origin 193 ahead = the + entire rewrite; upstream 49 ahead = continued old-tree firmware). The trees are disjoint above the base: + upstream/main's top-level is `*.ino`, `modes/`, `voicespaces/`, `*AudioApp.hpp`; origin/main's is `nisps/`, + `playground/`, `firmware/`, `schemas/`, `codegen/`. **Correction (verified):** it is *not* true that "no file + is co-modified." A `git merge-tree`/name-status comparison above the base shows ~20 paths the two sides both + touched — but for nearly all of them the relationship is **delete-on-origin / modify-on-upstream** (the old + `.ino`, `modes/*.hpp`, `voicespaces/VerbFX/basic.hpp`, `src/memlp`), i.e. delete/modify *conflicts*, not + content overlaps. The one genuine both-sides-`M` conflict is the **`src/memllib` gitlink** (origin → `4733ca0`, + upstream → `e291192`). So the practical conclusion stands — every path resolves to a conflict (delete/modify or + submodule-pin) with nothing cleanly 3-way-mergeable; upstream commits must be hand-ported — but the literal + "no file is co-modified" was wrong. +- **What origin/main is *behind* on (the 49):** the upstream-only firmware features — newest first: + `701f2d9 livefx`, `a18d9c6 verbfx`, `0538eb9 staticmlp`, `99bd9b1 jolts`, `7ceac18 noise` / + `a6e7af1 explore noise update`, `75e0f44 large verb`, `414590b tr8s home mapping` / `be6eac7 fx, tr8s` / + `5a34a4a djfx single-cc fix`, `d1182f1 djfx alpha`, `24c2ad8 d50, sysex`, `1436250 midi output select`, + `98c7449 input selection`, `fcef420 sax fx`, `c617fcc SB dislikes/sample-rate/sample-player`, + `ee2814e focus targets memlcelium`, `47fc77b/2429bcc geo push rlview`, `d301cc7 rlview and dislike`. +- **What origin/main is *ahead* on (the 193):** the whole platform — `nisps/` core, 8 engines as concepts, + CRTP modes, WASM bridge, SolidJS playground, schemas/codegen, verification harness (golden+parity+lint+CI), + a-immersive feature parity. None of this exists upstream. +- **memllib pin divergence:** origin/main → `4733ca0` (**orphaned**, off-graph); upstream/main → `e291192` + (current, +62 commits on the lab main line, includes dislikes). origin's memllib pin is effectively + abandoned and should be re-pinned regardless of what else is ported. +- **SB2026 / SaxFX are dead branches** (fully folded into upstream/main); the *content* worth taking is + reachable from upstream/main and from memllib main. **bettysax** has one stray `c5f0314 "demo"` commit — + ignorable. + +--- + +## 4. Ranked integration recommendations + +Because the architectures are disjoint, "integrate" means **port the behavior into `nisps/` + `playground/`**, +not `git merge`. Ranked by mission impact (RL/feedback richness and firmware-feature parity). + +### Rank 1 — Land the FEEDBACK / DISLIKES port (origin already did the hard part) +- **Source commits:** memllib `abe93ec` (operator's `FEEDBACK_MODE` 3-mode enum) on + `origin/feat/feedback-explore-modes`; cross-checked against memllib main `80420d1`/`d9bdcd6` (SB2026 dislikes + in `InterfaceRL`) which upstream/main pins via `e291192`. +- **Why first:** This is the negative-feedback / RL core of the whole project, and the work is *already + written up* as an implementation-ready spec at `docs/redesign/feedback-modes-port-spec.md` (new + `nisps::ml::FeedbackController<MLP_T>`, WASM C API, 11 ctest cases, parity Stage 5). The branch + `feat/feedback-explore-modes` and the half-present `nisps/ml/feedback.hpp` + `tests/cpp/test_mlp_feedback.cpp` + in the working tree show this is mid-flight. **Finish it and merge to main.** The spec's AVOID reconciliation + decision (route to existing `move_weights`, skip the firmware-only k-NN geometric centroid push) is sound and + should be recorded in `ALIGNMENT.md`. +- **Action:** complete `nisps/ml/feedback.hpp`, wire WASM bindings + `mode-runtime.ts`, land the tests, re-pin + memllib off the orphan. Do **not** try to port the old `ReplayMemory` geometric push now (out of scope per spec). + +### Rank 2 — Re-pin memllib off the orphaned `4733ca0` +- **Source:** memllib main `e291192` (or `abe93ec` if you want the feedback commit pinned directly). +- **Why:** origin/main pins an off-graph commit nothing can resolve; this is a latent footgun (a fresh + `submodule update` cannot check it out cleanly from the canonical remote). Pin to a real, reachable commit + as part of Rank 1. Low effort, removes a trap. + +### Rank 3 — Port the new upstream firmware engines as `nisps/` engines (selective) +The 49 upstream commits are firmware apps on the old tree. Port the ones with mission value, highest-value first: +- **`a18d9c6 verbfx` / `75e0f44 large verb`** (→ `VerbFX_1.1.0`): origin already has `verb_fx`; diff the upstream + large-verb params and bring improvements into the engine. **Medium effort, high audio value.** +- **`d1182f1 djfx alpha` + `ed4ac6d` (→ `DJFX_TR6S_MICROQ`) + `5a34a4a` single-CC fix:** DJFX is a *new* engine + not in origin's 8. Port as a new `nisps/engines/` engine + mode + schema if DJFX is wanted in the playground. + **Higher effort (new engine).** +- **`701f2d9/adbc058 livefx` (→ `LiveFX_1.0.0`):** newest app; evaluate whether it subsumes existing FX modes + before porting. +- **`0538eb9 staticmlp` + `99bd9b1 jolts` + `7ceac18 noise`/`a6e7af1 explore-noise`:** these are *RL/exploration* + changes (static-MLP hold, jolts, explore-noise) that overlap conceptually with the Rank-1 feedback work and + with origin's existing spread/noise model. Review them **alongside** the feedback port — some may already be + covered by `RandomiseMlp`/`RandomiseOutputs`; "explore noise" likely maps to the existing `spread`/OU-noise + knobs. **Audit before porting to avoid duplicating behavior.** +- **`414590b tr8s home mapping` / `be6eac7 fx, tr8s` / `10b8fca dynamic focus for TR8S` / `296bec6 target + selection`:** the TR-8S focus-mask / dynamic-focus work is the firmware analogue of the `activeDims_` focus + feature the feedback spec already accounts for. Useful reference for the focus-mask UI; port the *mask + semantics*, not the firmware display code. + +### Rank 4 — Take SB2026 sample-rate / sample-player only if needed +- **Source:** `c617fcc` (SB2026, already in upstream/main). Per-mode sample rate and a sample player are + firmware-platform features; the playground runs through an AudioWorklet, so most of this is N/A. Port only the + per-mode sample-rate concept if a mode genuinely needs it. **Low priority.** + +### Do NOT integrate +- `port-solidjs`, `feat/unified-preset-system`, `feat/useq-celium-opus46`, `useq-celium-opus47` (origin): + superseded/abandoned spikes — main has moved past them. +- upstream `SaxFX`, `bettysax` (incl. `c5f0314 "demo"`): app-specific / demo, no mission value to the rewrite. +- All 2024–2025 memllib feature branches and per-app org forks: historical museum. + +--- + +## 5. Surprises + +1. **upstream/main is the *old* tree, not "main ahead of the fork."** The naming invites the assumption that + upstream advanced the same codebase; it didn't — the lab kept shipping the pre-rewrite `.ino` firmware. The + 49 commits are a *parallel* line, never to be fast-forwarded. +2. **origin/main pins an orphaned memllib commit (`4733ca0`)** that exists in neither the lab repo nor the + operator's own memllib fork — an off-graph gitlink that a clean checkout cannot resolve. +3. **The dislikes/RL feedback work the task flagged as "to find" is already (a) merged into upstream via SB2026 + → memllib `e291192`, AND (b) re-implemented by the operator on `feat/feedback-explore-modes` (memllib + `abe93ec`), AND (c) fully spec'd for the new core** in `docs/redesign/feedback-modes-port-spec.md` — with + `nisps/ml/feedback.hpp` and `tests/cpp/test_mlp_feedback.cpp` already present (untracked) in the working tree. + The feedback integration is not a research question; it's a half-finished implementation to land. + +--- + +## Verification (checked against source) + +_Adversarial re-check, 2026-06-27, independent of the original author. Re-ran the git/gh commands and a fresh +bare clone of lab + operator-fork memllib._ + +**Confirmed correct:** +- Merge-base `6efbe9c` "voicing" 2026-04-14 (`git merge-base origin/main upstream/main`). +- origin/main HEAD `f256217` 2026-06-16; upstream/main HEAD `701f2d9` "livefx" 2026-06-22. +- origin 193 ahead / upstream 49 ahead (`git rev-list --left-right --count origin/main...upstream/main`); the 49 upstream-only commits and the specific SHAs cited in §3 (`701f2d9`, `a18d9c6`, `0538eb9`, `99bd9b1`, `7ceac18`, `a6e7af1`, `75e0f44`, `414590b`, `be6eac7`, `5a34a4a`, `d1182f1`, `24c2ad8`, `1436250`, `98c7449`, `fcef420`, `c617fcc`, `ee2814e`, `47fc77b`, `2429bcc`, `d301cc7`) all resolve with matching messages. +- **memllib `4733ca0` is genuinely orphaned** — `git cat-file` fails for it even after fetching BOTH `MusicallyEmbodiedML/memllib` and `monkey-w1n5t0n/memllib` into a bare clone. The survey's headline surprise holds. +- memllib main = `e291192` and contains `80420d1 "rlview and dislike"` + `d9bdcd6 "remove dislike opts screen"`. +- memllib `abe93ec` "InterfaceRL: add FEEDBACK_MODE setting" 2026-06-08 by w1n5t0n, branched off `616b8e7 "cc select"`, +1/−6 vs main; superproject `feat/feedback-explore-modes` pins it. +- All six tags (`LiveFX_1.0.0` `701f2d9`, `VerbFX_1.1.0` `a18d9c6`, `DJFX_TR6S_MICROQ` `ed4ac6d`, `MEMLCelium_0_1_alpha` `1e441cb`, `PAF_1_1_0`/`VerbFX` `bae80fb`) — SHAs, dates, and origin/upstream containment all match. +- Upstream branch ahead/behind: SB2026 `c617fcc` +0/−30, SaxFX +0/−42, bettysax +1/−72 with stray `c5f0314 "demo"`; memllib SB2026 = `1997699`. + +**Corrected (real errors found):** +1. **Origin-branch ahead/behind columns were inverted** in table 1.1 (the author read the `--left-right` left/right columns the wrong way round). Real values: `feat/feedback-explore-modes` +42/−193 (was +193/−42); `feat/useq-celium-opus46` +2/−36; `useq-celium-opus47` +6/−36; `feat/unified-preset-system` +46/−55; `port-solidjs` +31/−66. Fixed in table 1.1 and footnote ¹. (These branches are all *behind* main by ~the full rewrite, which is the opposite of what the original numbers implied.) +2. **SaxFX timeline tip was wrong** (§2): the SaxFX branch tip is `33b6542 "memlib sub"` (2026-05-05), not `fcef420 "sax fx ftw"` — `fcef420` is a 2026-05-15 upstream/main commit, not on SaxFX. Date was right, SHA/message wrong. Fixed. +3. **"No file is co-modified" (§3) was literally false.** ~20 paths are touched on both sides; for almost all the relation is delete-on-origin/modify-on-upstream (a conflict), and `src/memllib` is a genuine both-`M` submodule-pin conflict. The downstream conclusion (no clean 3-way merge; hand-port required) is unaffected. Fixed in §3. + +The substance of the survey — disjoint trees, hand-port-only integration, orphaned memllib pin, feedback work already spec'd/half-implemented, ranked recommendations — survives verification. The errors are in the branch-delta bookkeeping, one timeline SHA, and one overstated absolute. + +--- + +## Correction (2026-06-27, from the RL judge-panel skeptic, verified against source) + +The **latest upstream `InterfaceRL` tip is `0a541cc` "highlighting"** (990 lines; `optimise()` at `:601`, +`kCentroidK=4` at the hpp `:296`, decay `0.0025f` at `:664`, `negLRRatio` at `:742`) — NOT `e291192`/`abe93ec`, +which are older. **Mode 1 "Geometric dislike" must be ported to `0a541cc` parity.** Note: the in-tree submodule +gitdir resolves none of `4733ca0`/`e291192`/`abe93ec`/`0a541cc` — only the separate `-upstream` memllib +checkout has them; the orphaned-pin footgun stands and `0a541cc` is the re-pin/port target. See +`docs/redesign/rl-feedback-design.md` §1 for the verified-ground-truth list.