docs(redesign): Manifold research findings + design specs + build plan
Phase-1 audits (feedback-behaviour, engine-surface, design/Manifold), the judge-panelled rl-feedback-design, dock/backends/inputs specs, the upstream-firmware survey, the Manifold design mirror, and BUILD-PLAN.md (resume anchor + locked decisions).
This commit is contained in:
monkey-w1n5t0n
parent
f41d6ea1b7
commit
24057e7b5f
47 changed files with 6488 additions and 0 deletions
183
docs/redesign/BUILD-PLAN.md
Normal file
183
docs/redesign/BUILD-PLAN.md
Normal file
|
|
@ -0,0 +1,183 @@
|
|||
# Manifold — Build Plan & Locked Decisions (resume anchor)
|
||||
|
||||
*Dated 2026-06-27. This is the single resume anchor for the Manifold convertible-app build. Read this + the
|
||||
specs it points to before continuing. Mission: one working browser instrument putting the NEW Manifold
|
||||
"convertible" front-end on top of the REAL parity-tested ML+audio engine.*
|
||||
|
||||
## Locked decisions (operator-confirmed 2026-06-27)
|
||||
| Decision | Choice |
|
||||
|---|---|
|
||||
| App directory | `manifold/` (Vite + React + TS), beside `playground/` until parity + sign-off |
|
||||
| Engine to wire | The parity-tested **TS engine** (`playground/src/ml/wasm-iml.ts` + `audio/engine-host.ts` + worklet), same `nisps/` core the firmware builds from |
|
||||
| Staging deploy | `meml.lnfinitemonkeys.org/next` (server-scope COOP/COEP already set); live a-immersive stays at `/` |
|
||||
| Default FEEDBACK_MODE | **Explore & place** (Mode 2, positive-only); Geometric-dislike selectable |
|
||||
| Default SOLO/arm | **Mask-gradients / column-freeze** (variant a); ZeroLoss + DontCareExample selectable |
|
||||
| Modular N×M MLP | **Multiple WASM modules + warm-start** ({2,4} then {2,4,8}); NOT runtime-shaped, NOT padded |
|
||||
| VCV bridge transport | **WS↔OSC bridge server** (reuse existing Deno bridge; bidirectional training) |
|
||||
| Synth UI label | **"Powerful Synth Engine"** — the string "C15" must NEVER appear in the UI |
|
||||
| Product copy | British spelling (randomise, visualise, colour) |
|
||||
|
||||
## Spec docs (all in docs/redesign/)
|
||||
- `rl-feedback-design.md` — the learning engine (both modes + solo); **§1 = verified ground truth: the feedback
|
||||
C API, `MLHandle.feedback`, `0xFEEDBACC0DE` salt, parity Stage 5, and `CMakeLists.txt:59` registration ALL
|
||||
ALREADY EXIST** — edit/extend, don't re-scaffold. Mode 1 ports to upstream **`0a541cc`** InterfaceRL.
|
||||
- `dock-spec.md` — six drawers + per-output control row + tri-state semantics + advanced modals.
|
||||
- `backends-spec.md` — `OutputBackend` adapter interface; **VCV module already at `vcv/` (2-in/12-out)** →
|
||||
evolve to 8→16 + LED rings; faithful `visualizer.js` particle port (algorithm documented).
|
||||
- `inputs-spec.md` — modular XY/WebMIDI/gamepad; multiple-WASM-module reshape with warm-start.
|
||||
- `findings-{feedback-behaviour,engine-surface,design-and-manifold}.md` — Phase-1 audits.
|
||||
- `upstream-firmware-survey.md` — git topology; latest InterfaceRL = `0a541cc`.
|
||||
- `playground-2.0-rewrite-plan.md`, `engine-architecture.md`, `aimmersive-clone-spec.md`,
|
||||
`feedback-modes-port-spec.md` — the prior planning corpus.
|
||||
|
||||
## Reactive spine → React (load-bearing, from findings-design-and-manifold.md §4)
|
||||
Spine lives BELOW React in an external store: `setInput` action derives processed→ml→routed eagerly+synchronously
|
||||
and fires the single `backend.send` at its tail (off-render). React subscribes via `useSyncExternalStore`
|
||||
(version counter, not the array); canvases (Manifold/particles) read in one rAF loop. `EngineApi` exposed via
|
||||
React Context. Lint rule: skins may not import engine internals; engine may not import React.
|
||||
|
||||
## Phase-3 build sequence (each step keeps live a-immersive up; gate core touches on parity-check.sh @1e-5)
|
||||
1. **Pull Manifold JSX mirror** — DesignSync (session-bound) → `docs/redesign/manifold-export/` (only tokens on
|
||||
disk now; pull `components/**`, `ui_kits/memlnaut/**` incl. `console/**`).
|
||||
2. **Scaffold** `manifold/` — Vite + `@vitejs/plugin-react` + strict TS, COOP/COEP dev+preview headers,
|
||||
`base:'./'` + base-aware WASM URLs, worklet via `?worker&url`, copy `public/nisps.{wasm,js}`.
|
||||
3. **Engine adapter** — lift `WasmIML` + `EngineHost` + worklet + input/output pipelines from `playground/src`;
|
||||
replace Solid `mlStore`/`coreBus` side-effects with an injected emitter; build the framework-neutral
|
||||
`EngineApi` + the external-store reactive spine + React `EngineProvider`/`useEngine`.
|
||||
4. **Primitives** — port the 12 design primitives (Button/Slider/PillToggle/Panel/Badge/Switch/StatusLine/
|
||||
XYPad/VirtualJoystick/ControlAxis/CurvePlot/Sparkline) from window-global JSX to ES-module React/TS + tokens.
|
||||
5. **Console convertible shell** — ConsoleApp/CompositeStage/SplitStage/OutputStage/InputMini/Manifold/Dock/
|
||||
Drawers/VerdictCluster/ReadoutStrip/AltitudeNav (+Perform/Zen). Replace `MF_infer` with real engine. The
|
||||
single draggable divider with width-based representation demotion (full manifold → pad → minimap) is the centre.
|
||||
6. **Dock real contents** (dock-spec) — six drawers, per-output control rows, tri-state, advanced modals.
|
||||
7. **Modular inputs** (inputs-spec) — InputSource adapters (XY/WebMIDI/gamepad), N-channel pipeline, multi-module
|
||||
warm-start rebuild.
|
||||
8. **Backends** (backends-spec) — OutputBackend adapters: Powerful Synth Engine, particle visualiser (faithful
|
||||
port), WebMIDI, OSC (reuse Deno bridge), CV/gate.
|
||||
9. **Feedback engine** (rl-feedback-design) — TS-prototype-first (audible directional-away + interpolate-between-
|
||||
corners are pass/fail oracles via existing WasmIML primitives) → C++ core (ReplayStore + train-toward-targets
|
||||
+ solo column mask) → C API → TS FFI → dock selector. Every core step gated on parity.
|
||||
10. **VCV module** — evolve `vcv/` to 8→16 + LED-ring widget (palette from tokens) + WS↔OSC bridge +
|
||||
bidirectional training.
|
||||
11. **e2e** — `window.__nisps` synchronous probe + Playwright (spine invariant: setInputs → outputs+params
|
||||
change in one tick; no-per-frame-alloc fuzz).
|
||||
12. **Deploy** `/next`, MAP.md + ALIGNMENT.md + CLAUDE.md sync, hand operator URL. Archive playground only after
|
||||
parity + sign-off.
|
||||
|
||||
## Animations (workstream G, parallel to build) — Manim skill at ~/src/hermes-agent/skills/creative/manim-video
|
||||
Knob 1:1 · XY→2 numbers · dimensionality fan-out (knob/fader/touchpad/2D/3D joystick) · feedback contrast
|
||||
(geometric push-away vs explore-and-place). Plus interactive in-app onboarding demos. Feedback piece reflects
|
||||
the locked default (Explore & place).
|
||||
|
||||
## Manifold mirror status (docs/redesign/manifold-export/) — updated 2026-06-27
|
||||
PRESENT on disk: all 6 token CSS + styles.css; 12 primitives (components/{core,control,data}/*.jsx);
|
||||
console kit — ConsoleApp, CompositeStage, SplitStage, OutputStage, InputMini, Manifold, model, shared-ui,
|
||||
Dock, Drawers, OutputEditor, VerdictCluster, ReadoutStrip, CurvePad.
|
||||
STILL TO PULL via DesignSync (non-core; project id 49091559-c3ce-47cb-a14f-08d62c26fe09): console PerformApp.jsx,
|
||||
ZenApp.jsx; ui_kits/memlnaut top-level (App, ModeView, OutputBars, TopBar, TrainingPanel); output-routing/
|
||||
(Editor, model, parts, variants); support.js; the specimen *.html. These are altitude/alt-routing variants and
|
||||
specimens — not needed for the core convertible Console build (steps 2-9). Pull when building Perform/Zen
|
||||
altitudes or the routing-matrix Full modal.
|
||||
|
||||
Key port notes from the mirrored JSX:
|
||||
- CompositeStage IS the convertible: single `split` ratio ∈[0,1], SNAPS [0.14,0.33,0.5,0.66,0.86] w/ magnetism,
|
||||
SHUT=0.1 collapses a side to a draggable corner minimap; tier demotion by measured width (full→pad / field→list).
|
||||
- model.jsx `MF_infer` is the sin/cos placeholder to REPLACE with WasmIML; it still lists a `c15` mode labelled
|
||||
"C15" → relabel to "Powerful Synth Engine" (and the synth backend everywhere).
|
||||
- Drawers.jsx = the PLACEHOLDER drawers (Shape/Feel/Route/Health/Help) → replace with dock-spec's six real drawers.
|
||||
- OutputEditor/OutputStage/ReadoutStrip already implement the off/fixed/live tri-state + per-output min/max/curve
|
||||
+ alt-click cycle — the baseline the dock-spec per-output control row formalises (add mute + solo/arm).
|
||||
- Components use `window.*` globals + `window.ManifoldDesignSystem_490915` — convert to ES-module imports.
|
||||
|
||||
## Build status — 2026-06-27 (Phase-3 in progress)
|
||||
DONE + verified (typecheck+build green): scaffold `manifold/`; engine layer (framework-neutral EngineApi +
|
||||
reactive spine + EngineProvider/useEngine + window.__nisps probe; added the missing nisps_ml_feedback_* TS
|
||||
bindings — no C change); 12 primitives; convertible Console shell (ConsoleApp/CompositeStage/etc) wired to the
|
||||
real engine (MF_infer replaced; default mode = Explore-and-place; synth = "Powerful Synth Engine", 0 "C15" in
|
||||
bundle). **DEPLOYED to https://meml.lnfinitemonkeys.org/next/** (docroot subdir of meml-aimmersive, no nginx
|
||||
change; COOP/COEP inherited; live a-immersive untouched at /). To redeploy: `cd manifold && bun run build` then
|
||||
`rm -rf /home/w1n5t0n/deployments/meml-aimmersive/next && cp -r manifold/dist /home/w1n5t0n/deployments/meml-aimmersive/next && cp .../next/index.html .../next/a-immersive.html` (the a-immersive.html copy satisfies nginx `index a-immersive.html` so /next/ resolves).
|
||||
|
||||
CAVEAT — e2e not yet run on the VPS: headless Chromium needs `libnspr4`/`libnss3` which aren't installed and
|
||||
need operator sudo (`sudo bunx playwright install-deps chromium`, or run e2e on the laptop/CI). The smoke spec is
|
||||
written at `manifold/tests/e2e/smoke.spec.ts` (asserts: engine WASM loads, spine invariant setInputs→outputs
|
||||
change, feedback runs, console renders, no "C15"). Serving smoke (HTML+COOP/COEP+wasm) passed via curl. Operator
|
||||
should verify the live runtime in a real browser at /next/.
|
||||
|
||||
REMAINING Phase-3+: real dock contents (dock-spec) · modular inputs (inputs-spec) · output backends + particle
|
||||
port (backends-spec) · feedback C++ engine (rl-feedback-design, TS-prototype-first) · VCV module · full e2e ·
|
||||
animations (G). Then MAP.md/ALIGNMENT.md sync + commit + operator sign-off → archive playground.
|
||||
|
||||
## Build status update — 2026-06-28
|
||||
ADDED + verified (typecheck+build green, redeployed to /next): real dock contents (six drawers + OutputControlRow
|
||||
with off/fixed/live + mute + solo/arm + dual-range + curve, unified output-state store, per-backend advanced
|
||||
modals, feedback-mode + solo + audio wired) AND the feedback-engine TS prototype (manifold/src/feedback/:
|
||||
FeedbackController — Explore-and-place scratchpad loop snapshot→randomise/nudge→place-at-chosen-location→
|
||||
warm-start anchor interpolation, undo ring, geometric-dislike mode, example-level solo respect; SeededRng;
|
||||
mode-aware VerdictCluster + Manifold pick-location state). C++ crystallisation gaps tagged inline `--- C++ GAP ---`
|
||||
→ rl-feedback-design §4 (replay store, k-NN geo-push, train_masked column-freeze). ALIGNMENT.md got the
|
||||
output-state divergence note from the dock agent.
|
||||
STILL REMAINING: backends + particle visualiser port (E), modular MIDI/gamepad inputs (F), VCV module (E),
|
||||
C++ feedback crystallisation (B phase 2, gated on parity), full browser e2e (needs libnspr4 via operator sudo),
|
||||
Manim animations (G), MAP.md sync + commit + sign-off → archive playground.
|
||||
|
||||
## RUNTIME VERIFIED — 2026-06-28 ✅
|
||||
The /next deploy is browser-verified: the e2e smoke (manifold/tests/e2e/smoke.spec.ts) PASSES — engine WASM
|
||||
loads, spine invariant holds (setInputs→outputs change), feedback runs, console renders, no "C15", no errors.
|
||||
Two runtime bugs found+fixed in the lifted engine:
|
||||
1. assetUrl resolved against location.origin → dropped the /next sub-path. FIXED: resolve against document.baseURI
|
||||
(main thread) + pass the deploy base to the async-train worker via its init message.
|
||||
2. nisps.js is Emscripten MODULARIZE glue with NO ES6 exports — import() yields an empty namespace. FIXED:
|
||||
fetch + indirect-eval the glue to install global createNispsModule (works on main thread AND module worker).
|
||||
(The playground's identical import()-based loader has the same latent flaw — worth fixing there too.)
|
||||
E2E INVOCATION on this VPS (bun runs under snap confinement that hides system libs from its Chromium subprocess;
|
||||
run the test RUNNER via non-snap node, browsers under the bun cache path):
|
||||
cd manifold && (bun run preview &)
|
||||
PLAYWRIGHT_BROWSERS_PATH=/home/w1n5t0n/snap/bun-js/87/.cache/ms-playwright node node_modules/.bin/playwright test
|
||||
|
||||
## Build status update — 2026-06-28 (later)
|
||||
DEPLOYED + e2e-verified to /next, all green:
|
||||
- Dock RESTRUCTURED (operator): top Mode selector (Particle System[default]/MIDI/OSC/Built-in Synth/MEMLNaut
|
||||
Editor) + 5 vertically-centred drawers (Learning/Inputs/Outputs/Settings/Help). Old Synth/Visual drawers folded
|
||||
into per-Mode config in Outputs.
|
||||
- Settings drawer: monochrome icons (focused orange / unfocused off-white|white|orange), input-map rect↔circular
|
||||
toggle, corner-radius slider (default 2px, overrides --r-1/--r-2 on :root). Store: manifold/src/settings/.
|
||||
- Feedback markers on the 2D map: positive (filled dot) + negative (open red ring) at their input location.
|
||||
- Monochrome SVG icon set (manifold/src/console/icons.tsx, currentColor).
|
||||
- MEMLNaut Editor mode: Web Serial USB scaffold (manifold/src/serial/, stub).
|
||||
- MIDI + OSC backends REAL: manifold/src/backends/ (OutputBackend + BackendManager consuming the spine; WebMIDI
|
||||
out w/ per-output CC#/ch/name/range; OSC-over-WS to the Deno bridge in manifold/osc-bridge/). Outputs panel
|
||||
SPECIALISES per backend (dock/OutputsBackendConfig.tsx) + named presets (backends/presets.ts, per-backend
|
||||
localStorage). Audio gated via engine.audio.setMuted on non-synth modes.
|
||||
Naming: synth shows as "Built-in Synth" (operator's latest wording; still never "C15").
|
||||
REMAINING: faithful particle-system port (Particle System is default Mode — currently passthrough/no flow field
|
||||
yet), modular MIDI/gamepad INPUTS (F), VCV module (E), C++ feedback crystallisation (B ph2, parity-gated), Manim
|
||||
animations (G), MAP/ALIGNMENT sync + commit + sign-off → archive playground. Nothing committed yet.
|
||||
|
||||
## VCV module — BUILT 2026-06-28 ✅
|
||||
vcv/ evolved 2→12 → **8 inputs × 16 outputs**, **compiles + links** against Rack SDK 2.6.4
|
||||
(installed at ~/.local/share/Rack2/Rack-SDK; `cd vcv && RACK_DIR=$HOME/.local/share/Rack2/Rack-SDK make` →
|
||||
plugin.so 2.4MB). Per-output **LED rings** (src/LedRing.hpp, drawLayer+nvgArc, proportional fill + glow),
|
||||
**palette.hpp** from frontend tokens (16-step orange→cyan ramp). **WS↔OSC bridge** (src/osc_server.hpp) verbs
|
||||
/nisps/{input,output,feedback,weights,examples,state}, default UDP 7001+id%64, routed through the SAME
|
||||
doThumbsUp/Down/enqueueJob paths as the panel — trainable from BOTH VCV and the browser. Retired nisps-core
|
||||
resolved by **vendoring src/iml.hpp** (self-contained runtime IML, firmware-aligned semantics but not bit-identical
|
||||
— follow-up). Derived outputs moved behind a context-menu toggle. TODOs: browser-side /nisps/feedback emission;
|
||||
redraw panel SVG art for 8/16 (coords relaid in code); align vendored IML optimiser to nisps/ml bit-exactly.
|
||||
REMAINING overall: faithful particle-system port, modular MIDI/gamepad inputs (F), C++ feedback crystallisation
|
||||
(B ph2), Manim animations (G), MAP/ALIGNMENT sync (done this round) + commit + sign-off.
|
||||
|
||||
## Explore-and-place CRYSTALLISED to shared C++ core — 2026-06-28 ✅ (parity-verified independently)
|
||||
nisps/ml/feedback.hpp gained FeedbackMode::ExploreAndPlace + Idle/Exploring/Placing state machine (no-heap,
|
||||
nisps::Rng, .f). Granular methods (enter_explore/exit_explore/reroll/nudge/undo/begin_place/commit_place/
|
||||
cancel_place) for firmware button-mapping + on_down/on_up browser default policy. Controller owns weights+
|
||||
scratchpad+undo-ring+frozen placed_out_; caller owns add_example+train. C API + EXPORTED_FUNCTIONS added.
|
||||
VERIFIED (orchestrator re-ran, not agent's word): native ctest 4/4; fresh build-wasm + parity-check.sh = PASS
|
||||
821 floats within 1e-5 (max 2.4e-7), parity Stage 5d exercises enter→reroll→nudge→undo→place→commit. Fixed bug:
|
||||
set_mode(3) fell back to Avoid. nisps.wasm synced to playground/public + manifold/public (md5-identical).
|
||||
manifold/src/feedback/controller.ts REWIRED to call the shared core (TS scratchpad deleted) — app explore-and-place
|
||||
is now firmware-identical; e2e green; redeployed /next. Firmware glue (peripherals.hpp) wired to the hardware flow
|
||||
(down=enter/exit explore, reroll+nudge buttons, like=begin place, up=commit) — NOT compiled (no arduino-cli);
|
||||
Placing-audition static_output() hook is a flagged follow-up. Mode 1 (geometric dislike) untouched.
|
||||
REMAINING: faithful particle-system port, modular MIDI/gamepad inputs (F), Manim animations (G), firmware build
|
||||
+ flash verification, browser-side /nisps/feedback VCV bridge emission, commit + sign-off.
|
||||
350
docs/redesign/aimmersive-clone-spec.md
Normal file
350
docs/redesign/aimmersive-clone-spec.md
Normal file
|
|
@ -0,0 +1,350 @@
|
|||
I now have a thorough, complete understanding of the entire deployed app — DOM, CSS theme tokens, all interaction handlers, state model, engine call points, persistence, and the help-modal copy. I have everything needed to write the faithful clone spec. Let me write it now.
|
||||
|
||||
# NISPS Immersive (`a-immersive.html`) — SolidJS Clone Spec
|
||||
|
||||
This is a faithful, component-by-component rebuild spec for the deployed vanilla-JS app at `/home/w1n5t0n/deployments/meml-aimmersive/`. Source of truth: `a-immersive.html` (446-line DOM), `css/a-immersive.css` (2538 lines), `js/a-app.js` (4521-line module), and supporting modules under `js/nisps/`, `js/ui/`, `js/synth/`, `js/midi/`, `js/audio/`. Goal: a from-scratch SolidJS reimplementation that matches the look and behaviour 1:1, binding to a shared engine layer (the WASM IML + synth engines) rather than re-porting it.
|
||||
|
||||
---
|
||||
|
||||
## 0. Critical scoping note before you build
|
||||
|
||||
**The deployed app does NOT use the Boldness/Memory/Precision control surface.** `README.md` and `SPEC-controls.md` describe that system (`js/ui/control-surface.js`, `control-surface-ui.js`, `input-pipeline.js`), but **`js/a-app.js` never imports any of them**. (Verified: `grep "control-surface" js/a-app.js` → no hits.) Those modules belong to a different design and to the SolidJS migration target (`PLAN-solidjs-migration.md`).
|
||||
|
||||
What a-immersive *actually* exposes for ML tuning is a flat **"NISPS" params drawer** with 6 raw sliders: **Spread, Noise, RL Decay, Learn Rate, Max Iters, Convergence** (`buildEngineParams()`, a-app.js:3137). To clone a-immersive 1:1, build **that** drawer. If the new SolidJS app is meant to also gain Boldness/Memory/Precision, treat it as an additive feature on top of this spec, not a replacement.
|
||||
|
||||
Joystick input is fed **directly** (`joyX, joyY` ∈ [0,1]) into `iml.setInput(0/1, …)` — there is **no deadzone/zoom/curve/smoothing pipeline** in a-immersive. The only output-side shaping is the per-param override system (curve/min/max/freeze).
|
||||
|
||||
Many imports in a-app.js are for features visible only behind flags or that exist as dock entries (EOC Effects Chain, Modular engine, ShapeSeq behind `?shapeseq=1`, Faust additive/FM engines, audio-canvas, hand tracking). See §10 for what's core vs deferrable.
|
||||
|
||||
---
|
||||
|
||||
## 1. Component tree
|
||||
|
||||
Top-level provider tree wraps everything in store contexts (see §5). All components are absolutely/fixed-positioned over a full-viewport canvas — this is a single-screen HUD, not a flow layout.
|
||||
|
||||
```
|
||||
<ImmersiveShell> // root; owns rAF loop, resize, init, keyboard, gamepad
|
||||
├── <BackButton> // fixed top-left chevron → designs.html
|
||||
├── <SynthQuickControls> // shown only when outputMode==='synth'
|
||||
│ ├── <PlayButton> // play/pause icon, audio-needs-init pulse
|
||||
│ ├── <PlayDrawer> // hover flyout: Vol + BPM sliders
|
||||
│ └── <SynthPresetSelect> // tiered <select> (Manual/Beginner/.../Expert)
|
||||
├── <MidiCcQuickControls> // shown only when outputMode==='midi-cc'
|
||||
│ ├── enable button, output-device <select>, preset <select>, status span
|
||||
├── <AudioCanvasMount> // div, populated by AudioCanvas (deferrable)
|
||||
├── <ShapeSeqContainer> // behind ?shapeseq=1 (deferrable)
|
||||
│
|
||||
├── <FlowFieldCanvas> // #vis-canvas — visual mode particle system
|
||||
├── <SynthVisualizer> // #synth-vis-canvas — 126-bar synth chart (synth mode)
|
||||
│
|
||||
├── <HeatmapStrip> // top strip; hidden in synth mode
|
||||
│ ├── <HeatmapCell> × N // one per output param; drag-to-set, click-to-popup
|
||||
│ ├── <HeatmapTooltip> // hover quick-info
|
||||
│ └── <ParamOverridePopup> // portal: curve/min-max/freeze (+ CC editors)
|
||||
│
|
||||
├── <FloatingJoystick> // bottom-left; canvas knob + minimap + noise ring
|
||||
│ ├── <JoyMapCanvas> // #joy-map: bg grid, training dots, knob, crosshair
|
||||
│ ├── <NoiseRing> // RL exploration indicator (off/active/high)
|
||||
│ ├── <FollowBadge> // "FOLLOW" pill when follow mode on
|
||||
│ └── <GamepadStatus> // tiny orange status text
|
||||
├── <HandTrackingPip> // replaces joystick when input==='hands' (deferrable)
|
||||
├── <EocJoystick> // independent EOC mode only (deferrable)
|
||||
│
|
||||
├── <RlButtons> // bottom-center: −/+ cluster + undo + (linked label)
|
||||
│ ├── <RlDownButton> (−, key 1)
|
||||
│ ├── <RlUpButton> (+, key 2)
|
||||
│ └── <UndoButton> (Z)
|
||||
├── <EocRlButtons> // FX −/+ (keys 3/4), linked mode only (deferrable)
|
||||
│
|
||||
├── <StatusLine> // bottom: "N examples · loss X · noise Y"
|
||||
│
|
||||
├── <Dock> // right-side macOS dock
|
||||
│ └── <DockIcon> × 6 // Train, Mode, Synth, NISPS, FX, Help
|
||||
├── <DrawerStack> // stacked glass panels left of dock
|
||||
│ ├── <TrainingDrawer> // actions, visual presets, loss canvas
|
||||
│ ├── <ModeDrawer> // input + output pill toggles + follow pill
|
||||
│ ├── <SynthDrawer> // engine switcher + audio/arp controls
|
||||
│ ├── <MidiCcDrawer> // CC param management
|
||||
│ ├── <ParamsDrawer> // "NISPS" tuning sliders (the 6 above)
|
||||
│ └── <EocDrawer> // effects chain UI mount (deferrable)
|
||||
├── <GroupOverrideDrawer> // portal: per-group curve + per-param rows (synth mode)
|
||||
│
|
||||
├── <HelpModal> // overlay; auto-opens on first visit
|
||||
└── <Toast> // transient bottom message
|
||||
```
|
||||
|
||||
### What each owns / renders
|
||||
|
||||
- **`<ImmersiveShell>`** — orchestrator. Holds no visual markup of its own beyond children. Owns: the `requestAnimationFrame` loop (`animate()`, a-app.js:4251 — polls gamepad, draws whichever canvas is active), the global `resize` handler (`onResize()`:4217 — resizes both canvases + joymap, re-inits particles, hides popups), keyboard (`wireKeyboard()`:3675), gamepad wiring, init sequence (`init()`:1265), auto-save interval (10 s), and the `?debug=1` probe (`window.__nisps`). Mounts `<Toast>` lazily.
|
||||
- **`<FlowFieldCanvas>`** — wraps `FlowFieldVisualizer` (`js/ui/visualizer.js`). 400-particle Canvas2D flow field driven by 20 outputs (see §7). Visible when `outputMode==='visual'`; gets `.hidden-canvas` otherwise. Internal render loop is driven by the shell's rAF, not its own.
|
||||
- **`<SynthVisualizer>`** — wraps the `SynthVisualizer` class (defined inline in a-app.js:584). Draws all visible (non-muted) param bars grouped into labeled sections, with hover tooltip and drag-to-set. Hovering a **section label** opens `<GroupOverrideDrawer>`. Visible (`.active`) only in synth mode.
|
||||
- **`<HeatmapStrip>`** — 22 px top strip of equal-flex bars, one per output. Each bar's width = output value %. Bars: hover→tooltip, drag→set value, click→`<ParamOverridePopup>`. Hidden in synth mode (the SynthVisualizer replaces it there). Gets `.shared-mode` (adds "+ FX" label) when EOC shared mode is active.
|
||||
- **`<FloatingJoystick>`** — the primary input. A 160×160 canvas knob with integrated minimap (training-example dots + current position). Drag = relative move; tap = snap-to-position; double-tap = toggle follow. See §2.
|
||||
- **`<RlButtons>`** — the −/+/undo cluster (see §2). `<StatusLine>` is a read-only floating string.
|
||||
- **`<Dock>` + `<DrawerStack>`** — dock icons toggle the matching drawer (Help opens the modal instead). Each drawer is an independent glass panel that shows/hides via a `.hidden` class; multiple can be open at once.
|
||||
|
||||
---
|
||||
|
||||
## 2. Every control and interaction (exact behaviour)
|
||||
|
||||
### 2.1 Floating joystick (`wireJoystick()`, a-app.js:2280)
|
||||
Listeners are on `#joy-map` (pointerdown) + `window` (pointermove/up/cancel).
|
||||
- **Tap / pointerdown**: snap `joyX/joyY` to the tapped position inside the circle. Mapping: `dx = clientX - rectLeft - size/2`; `maxR = size/2 - 8`; `joyX = clamp(0.5 + (dx/maxR)*0.5)`; `joyY = clamp(0.5 - (dy/maxR)*0.5)` (Y inverted). Then `drawJoyMap()` + `onJoystickMove()`. Starts a drag.
|
||||
- **Double-tap** (two pointerdowns < 350 ms apart): `toggleFollowMode()`; consume (no drag).
|
||||
- **Drag / pointermove**: relative — `joyX = clamp(startJX + dx*scale*0.5)` where `scale = 1/maxR`; Y inverted. Redraw + `onJoystickMove()` each move.
|
||||
- **Follow mode**: when on, pointer motion anywhere over `#vis-canvas` (and during window pointermove) sets `joyX = clamp(clientX/innerWidth)`, `joyY = clamp(1 - clientY/innerHeight)` — i.e. the whole screen becomes the pad, no hold needed.
|
||||
- **`onJoystickMove()`** (2378): `iml.setInput(0,joyX); setInput(1,joyY); process()`; route + heatmap + `syncRawParamsFromOutputs`; push to `joyTrail` (cap 30, used by enhanced map / not strictly needed for base clone). Skips when `inputMode!=='joystick'`.
|
||||
- **`drawJoyMap()`** (2193): circular clip; bg `rgba(13,13,13,0.7)`; quarter grid lines `rgba(255,255,255,0.06)`; ring border; **training dots** at `(fx = features[i][0]*w, fy=(1-features[i][1])*h)`, hue = `(labels[i][3]||0)*360` (i.e. coloured by output #3 = Hue), radius 3; **knob** orange `rgba(255,106,0,0.9)` r=8 with shadowBlur 12 + white inner dot r=3; orange crosshair.
|
||||
- **Noise ring** (`updateNoiseRing()`:2879): `noiseLevel>0.15` → `.active.high` (6 px red border); `>0.01` → `.active` (3 px); else none.
|
||||
|
||||
### 2.2 RL +/−/undo cluster (`onThumbsUp`/`onThumbsDown`/`onUndo`)
|
||||
Wired in both `wireControls()` (2567) and via keyboard.
|
||||
- **`+` (Up, key `2` / Numpad2 / gamepad RB / hand "thumbsup")** — `onThumbsUp()` (2988):
|
||||
1. `pushUndoSnapshot()` (weights + noiseLevel + exampleCount; stack cap 20).
|
||||
2. `target.addExample(getCurrentInputs(), [...rawParamValues])` — **adds the current input→current-output pair as a training example** (i.e. "this mapping is good, keep it").
|
||||
3. `noiseLevel *= rlExplorationDecay (0.97)`, floored at 0.005.
|
||||
4. `flash('btn-thumbsup')`, `updateNoiseRing()`, then **async train** (`trainModelAsync()`); EOC target trains directly.
|
||||
- **`−` (Down, key `1` / Numpad1 / gamepad LB / hand "thumbsdown")** — `onThumbsDown()` (3012):
|
||||
1. `pushUndoSnapshot()`.
|
||||
2. `noiseCap = 0.3*(1-spread) + 0.05*spread`; `noiseLevel = min(noiseLevel*1.5, noiseCap)`.
|
||||
3. `target.moveWeights(noiseLevel, spread)` — **perturbs network weights to explore a new mapping** (no example added).
|
||||
4. Re-route outputs from the affected IML; `updateStatus()`, `updateNoiseRing()`, `flash('btn-thumbsdown')`.
|
||||
- **Undo (key `z`/`Z`, button between +/−)** — `onUndo()` (2949): pop snapshot, restore weights (`_setFlatWeights`) + noiseLevel, re-infer, route. Button gets `.has-undo` when stack non-empty (`updateUndoButton()`).
|
||||
- **`flash(id)`**: add `.flash` class for 250 ms (scale pop animation).
|
||||
|
||||
### 2.3 Keyboard accelerators (`wireKeyboard()`, 3675)
|
||||
Ignored when focus is in INPUT/SELECT/TEXTAREA, and on `e.repeat`. `preventDefault()` each.
|
||||
- `1`/Numpad1 → `onThumbsDown()`; `2`/Numpad2 → `onThumbsUp()`.
|
||||
- `3`/Numpad3 → FX thumbs-down (only if `imlEoc` && linked mode); `4`/Numpad4 → FX thumbs-up (same guard). These temporarily set `eocTrainingTarget='eoc'`, fire, restore, `flash('eoc-rl-minus/plus')`.
|
||||
- `z`/`Z` → `onUndo()`.
|
||||
|
||||
### 2.4 Interactive heatmap (`buildHeatmap()`:1728, `setHeatmapValue()`:1829, `<ParamOverridePopup>`:1930)
|
||||
Per cell:
|
||||
- **Hover** (pointerenter): tooltip `"<name>: <value.toFixed(3)> ▾"`, positioned at cell's left. pointerleave hides; if its popup is open, schedule hide in 300 ms.
|
||||
- **pointerdown**: record down position, set `_dragging`, `setPointerCapture`, add `.dragging` (→ `cursor:ew-resize`).
|
||||
- **pointermove**: if moved >3 px in x or y, mark `_didDrag` and call `setHeatmapValue(i,e,cell)` — `x = clamp((clientX-rectLeft)/rectWidth)`; writes `rawParamValues[i]=x`, routes, updates heatmap, syncs; if param frozen, drag updates its `fixedValue`. Tooltip follows.
|
||||
- **pointerup**: if **not** dragged → it's a click → toggle the override popup (`showParamPopup(i)` / `hideParamPopup()`).
|
||||
- **Override popup** (`<ParamOverridePopup>`): glass card positioned below the cell (260 px wide, clamped to viewport). Rows:
|
||||
- *(MIDI CC mode only)* editable **Name** (text), **CC#** (0–127; auto-renames from `CC_NAMES`), **Ch** (1–16) — each persists via `saveCCMap`.
|
||||
- **Curve**: 36×36 draggable canvas (`_drawCurveOnCanvas` + `_wireCurveDrag`), value label; vertical drag sets curve ∈ [0,1], 0.5 = linear. Routes live.
|
||||
- **Range (min/max)**: dual-thumb overlapping `<input type=range>` (min blue `#4488ff`, max orange `#ff6a00`) with a fill bar; clamps min≤max; label `"min–max"`. Routes live.
|
||||
- **Freeze**: button toggles `frozen`; when frozen, captures current output into `fixedValue`, reveals a value slider (blue), and the param is removed from NISPS control (its output is pinned to `fixedValue`). Frozen cells get a hatched overlay + dimmed bar (`.heatmap-cell-frozen`).
|
||||
- Popup stays open while hovered; closes 300 ms after leaving both popup and cell.
|
||||
|
||||
### 2.5 Synth visualizer + group drawer (`SynthVisualizer`:584, `showGroupDrawer()`:3953)
|
||||
- **Bars**: only non-muted params drawn, grouped with 2 px gaps between sections; each section gets a centered label at the top (`_drawSectionLabel`). Bar height = display value (lerped at 0.12/frame toward target). Hover any bar → in-canvas tooltip (name / Val / Range / Curve).
|
||||
- **Drag a bar** (`enableInteraction(true)` in synth mode): sets that param's raw value from Y (`yToValue`), routes + updates heatmap. Sliding across bars reassigns the dragged index.
|
||||
- **Hover/click a section label** → `<GroupOverrideDrawer>` (320 px glass card, portal). Contents:
|
||||
- **Group master curve**: 48×48 draggable canvas. Vertical drag applies a *relative delta* to the group curve **and to every per-param curve** in the section (preserving relative offsets). `delta = dy/80`.
|
||||
- **Per-param rows**: name, 28×28 per-param curve canvas (vertical drag), dual-range min/max slider, and an **M (mute)** button. Muted rows hide curve+range, show a grey value slider, and strike-through the name. Mute removes the param from NISPS control.
|
||||
- For C15 this is backed by `groupOverrides[si]` + `SYNTH_SECTIONS`; for Faust engines by `engineParamOverrides` + `nonC15Sections` (group curves remembered by name across engine swaps). Unify behind a `getSectionView(sectionIndex)` adapter (a-app.js:3916).
|
||||
|
||||
### 2.6 Output-mode tabs (`setOutputMode()`:2786)
|
||||
In `<ModeDrawer>`, pill toggle: **Visual / Synth / MIDI CC / Audio Canvas** (`#output-toggle-float`). On switch:
|
||||
- Compute `targetOutputs = outputCountForMode(mode)` (visual 20, synth = engine paramCount [126 for C15], midi-cc = `midiCCMap.length`, audio-canvas = `audioCanvas.getOutputCount()`).
|
||||
- If output count changes **and** there are training examples: `confirm()` a weight-reset warning; on cancel, revert pill. Then `resizeMLP()` (warm-starts joystick weights, clears examples, recreates hand IML).
|
||||
- Toggle canvases/strips: synth → hide `#vis-canvas` (`.hidden-canvas`), show `#synth-vis-canvas` (`.active`), hide heatmap strip, show synth quick controls, `synthVisualizer.enableInteraction(true)`. Other modes show heatmap strip, hide synth vis, disable interaction; midi-cc shows MIDI quick controls; audio-canvas shows its wrap.
|
||||
- `buildHeatmap()` + `routeOutputs()` + `buildEngineParams()` + sync.
|
||||
|
||||
### 2.7 Input toggle + follow (`wireInputToggle()`:2622, `toggleFollowMode()`:2360)
|
||||
- `<ModeDrawer>` "Input" pill toggle: **Joystick / Hands**. Switching to Hands lazily constructs `HandTracker`, requests camera, swaps `iml=imlHand`, hides joystick, shows PIP; on camera error reverts to joystick. (Hand tracking deferrable — see §10.)
|
||||
- **Follow pill** (`#follow-pill`) and the joystick double-tap both call `toggleFollowMode()`. `updateFollowUI()` shows the FOLLOW badge, adds `.follow-active` (pulsing border) and marks the pill active.
|
||||
|
||||
### 2.8 Dock icons → drawers (`wireDock()`:2521)
|
||||
- Click a `.dock-icon[data-drawer=X]`: if `X==='help'` open the modal; else toggle `#drawer-X` `.hidden` and the icon's `.active`. `.drawer-close` buttons close their drawer + deactivate the icon. Drawers are independent (stack scrolls). The 6 icons: **Training, Mode, Synth, NISPS (params), FX (eoc), Help**.
|
||||
|
||||
### 2.9 Presets
|
||||
Two distinct preset systems — keep them separate:
|
||||
- **Visual/RL presets** (`<TrainingDrawer>` chips: Calm/Chaos, Rainbow, Vortex, Spiral, Embers) — `loadPreset(name)` (3060): clears the **joystick** dataset, adds the preset's hardcoded input→output examples (`PRESETS`, a-app.js:59; each output is 20 floats padded to N), trains synchronously, re-infers. These teach the *network*.
|
||||
- **Synth presets** (`<SynthPresetSelect>` dropdown, tiered) — `applyPreset(presetId)` (460): sets which of the engine's params are active vs muted and their min/max/curve/fixedValue, via `groupOverrides` (C15) or `engineParamOverrides` (Faust). Tiers expose progressively more params (Beginner 15 → Expert full). These shape the *output mapping*, not the network. `?preset=<id>` URL param auto-applies on load.
|
||||
|
||||
### 2.10 Gamepad (`wireGamepad()`:3645)
|
||||
`GamepadInput` with `invertY:true`. Left stick → `joyX/joyY` + `onJoystickMove`. Buttons: LB→down, RB→up, A→train, X→randomize, B→clearExamples. Status text in `#gamepad-status`.
|
||||
|
||||
---
|
||||
|
||||
## 3. Theme / branding tokens and key copy
|
||||
|
||||
### CSS custom properties (`:root`, css:11) — reproduce verbatim
|
||||
```css
|
||||
--glass-bg: rgba(13, 13, 13, 0.65);
|
||||
--glass-border: rgba(255, 255, 255, 0.08);
|
||||
--glass-blur: 16px;
|
||||
--accent: #ff6a00; /* signature orange */
|
||||
--accent-dim: rgba(255, 106, 0, 0.25);
|
||||
--danger: #ff4466;
|
||||
--text: #e0e0e0;
|
||||
--text-dim: #888;
|
||||
--radius: 12px;
|
||||
--radius-sm: 8px;
|
||||
--safe-bottom: env(safe-area-inset-bottom, 0px);
|
||||
--dock-width: 48px;
|
||||
--dock-gap: 8px;
|
||||
```
|
||||
- **Background**: `#0d0d0d` everywhere; `html,body { overflow:hidden; touch-action:none; user-select:none }`.
|
||||
- **Font**: `'JetBrains Mono', 'SF Mono', 'Fira Code', 'Cascadia Code', monospace`; base `13px`. Loaded from Google Fonts (`JetBrains+Mono:wght@400;500;700`).
|
||||
- **Glass mixin**: `background: var(--glass-bg); backdrop-filter: blur(var(--glass-blur)); border: 1px solid var(--glass-border)`. Drawers use a denser `rgba(13,13,13,0.88)` + `blur(20px)`.
|
||||
- **RL up** = teal `#00c8a0` (border/bg tinted); **RL down** = red-orange `#dc3c14`. EOC RL up = `#4ecdc4`. Noise ring = `rgba(255,100,50,…)` → high `rgba(255,60,40,…)`.
|
||||
- **Dock**: macOS magnify — `.dock-icon:hover { transform: scale(1.35) }`, neighbours scale 1.12 via `:has()`/sibling selectors; active icon tinted orange.
|
||||
- **Animations to reproduce**: `drawerSlideIn` (0.2 s translateX), `follow-pulse` (1.5 s box-shadow), `rl-flash` / `btn-flash` (0.25 s scale), `audioInitPulse` (2 s, on play button when audio not started), `helpFadeIn`.
|
||||
- **Heatmap frozen cell**: 45° hatched `rgba(80,160,255,0.15)` overlay + bar opacity 0.4.
|
||||
|
||||
### Key copy to reproduce verbatim
|
||||
- **Title** (`<title>`): "NISPS Immersive". **Help modal H2**: "Welcome to NISPS"; subtitle: "Neural Interactive Shaping of Parameter Spaces".
|
||||
- **Status line** default: `"0 examples · untrained"`; live format: `"<N> example(s) · loss <0.00000> · noise <0.000>"` (or `· untrained` when never trained). Middots are U+00B7.
|
||||
- **RL buttons**: down has `−` + `1`; up has `+` + `2`; titles "Explore more" / "Keep this". Undo title "Undo last action".
|
||||
- **Help modal** is a long static block — reproduce sections **What is this? / How it works / What to expect / Controls (Touch/Mouse, Keyboard 1/2/3/4/Z, Gamepad) / Hand Tracking / Synth Controls**, plus the "Got it" button. (Full text is in `a-immersive.html` lines 354–441 — copy it wholesale; it's the authoritative onboarding copy.) Auto-opens when `localStorage['nisps-help-seen']` is unset; closing sets it.
|
||||
- **Dock labels**: Train, Mode, Synth, NISPS, FX, Help. **Drawer headers** (uppercase orange): Training, Mode, Synth, MIDI CC, NISPS, Effects Chain.
|
||||
- **Training drawer buttons**: Add Example, Train, Clear Ex, Clear All, Randomize; preset chips Calm/Chaos, Rainbow, Vortex, Spiral, Embers.
|
||||
- **Synth preset `<select>`**: option "Manual" (value `""`), then optgroups Beginner(1.1–1.4), Intermediate(2.1–2.4), Advanced(3.1–3.3), Expert(4.1–4.2).
|
||||
- **Back button** → `designs.html`.
|
||||
|
||||
---
|
||||
|
||||
## 4. SVG icons (inline, reproduce as components)
|
||||
All dock icons, the back chevron, the undo arrow, the play/pause/MIDI-keyboard icons, and the gesture ring are inline SVGs in the HTML. Reproduce as small SolidJS components (paths are in `a-immersive.html`):
|
||||
- Back: `<path d="M10 2L4 8l6 6"/>` (stroke).
|
||||
- Play: `<path d="M4 2l10 6-10 6z"/>` (fill); Pause: two rects.
|
||||
- Dock: Train (mixer faders), Mode (two circles + diagonal), Synth (sine wave `M2 10c2-4 4-4 6 0s4-4 6 0`), NISPS (gear), FX (infinity-ish loop), Help (question mark in circle).
|
||||
- Undo: `<path d="M3 7h6a4 4 0 0 1 0 8H7"/><path d="M3 7l3-3M3 7l3 3"/>`.
|
||||
|
||||
---
|
||||
|
||||
## 5. State model → SolidJS stores/signals
|
||||
|
||||
The deployed app keeps state in module-level `let`s. Map to stores/signals. **Performance rule** (from migration plan): ML outputs should be a single `Float32Array` signal updated once per frame, not a reactive array of 126 cells — drive canvases imperatively, drive the heatmap DOM widths via a cheap effect.
|
||||
|
||||
| Deployed state (a-app.js) | SolidJS home | Notes |
|
||||
|---|---|---|
|
||||
| `iml`/`imlJoy`/`imlHand`/`imlEoc` | non-reactive refs in an **EngineContext** | WASM instances; never put in a store |
|
||||
| `inputMode` ('joystick'\|'hands') | `inputStore` signal | drives joystick vs PIP |
|
||||
| `outputMode` ('visual'\|'synth'\|'midi-cc'\|'audio-canvas') | `modeStore` signal | gates canvases, heatmap, quick controls |
|
||||
| `joyX,joyY,joyDragging,joyFollowMode,joyTrail` | `inputStore` (x/y signals + follow signal) | x/y change every frame → keep as plain signals |
|
||||
| current outputs (`iml.getOutputs()`) | `outputStore` — **one `Float32Array` signal** | updated once per `routeOutputs` |
|
||||
| `rawParamValues` | `outputStore.raw` (Float32Array signal) | the "set sliders" buffer for examples/heatmap drag |
|
||||
| `noiseLevel,spreadLevel,rlExplorationDecay` | `mlStore` signals | spread also from `?spread`; default 0.6, noise 0.05, decay 0.97 |
|
||||
| `iml.learningRate / maxIterations / convergenceThreshold` | exposed via `mlStore` getters/setters | the 6 NISPS sliders write these |
|
||||
| `exampleCount,lastLoss,lossHistory` | `mlStore` (derived from iml after each op) | status line + loss canvas |
|
||||
| `undoStack` (cap 20) | `mlStore.undo` (non-reactive array + `hasUndo` signal) | snapshot = {weights,noiseLevel,exampleCount} |
|
||||
| `visualOverrides[20]` | `overrideStore.visual` (createStore) | {min,max,curve,frozen,fixedValue} |
|
||||
| `groupOverrides[18]` (C15 sections) | `overrideStore.group` (createStore) | {curve, params:[{min,max,curve,muted,fixedValue}]} |
|
||||
| `engineParamOverrides` (Faust, flat) | `overrideStore.engine` | null for C15 |
|
||||
| `midiCCMap,midiCCOverrides` | `midiStore` | per-engine storage key |
|
||||
| `audioCanvasOverrides` | `overrideStore.audioCanvas` | dynamic length |
|
||||
| `activeEngine,activeSynthPresetId` | `synthStore` signals | engine id + preset id |
|
||||
| `eocChain,eocTrainingTarget,nispsMode` | `eocStore` | deferrable |
|
||||
| drawer open flags, active dock icon | `uiStore` (per-drawer booleans) | independent drawers |
|
||||
| `activePopupParam,activeDrawerSection` | `uiStore` signals | popup/group-drawer routing |
|
||||
|
||||
**Persistence** (`saveState()`:4298, `loadState()`:4346): single `localStorage['nisps-a-immersive']` JSON blob. Saved fields: `features/labels` (joystick dataset), `handFeatures/handLabels`, `noiseLevel`, `outputMode`, `inputMode`, `joyX/joyY`, `groupOverrides`, `visualOverrides`, `midiCCOverrides`, `audioCanvasState`, `synthPresetId`, `engineId`, EOC module/mode state, `modularDspState`. On load: re-add examples and **train synchronously**, restore overrides, restore output mode (`skipConfirm:true`), but **never auto-restore `inputMode='hands'`** (camera permission). Auto-save fires on a 10 s interval and `saveState()` is also called after preset apply etc. In SolidJS, run a debounced effect (the migration plan uses 200 ms) that serializes the same shape; keep the storage key identical for migration continuity. Separate keys: `nisps-help-seen`, `nisps-midi-cc-map:<engineId>`, `nisps-modular-state`.
|
||||
|
||||
---
|
||||
|
||||
## 6. Engine call points (bind to the shared engine layer)
|
||||
|
||||
The shared engine layer is **`WasmIML`** (`js/nisps/nisps-wasm.js`) for ML, plus a **`SynthEngine`** (default `C15Adapter`) for audio. The new SolidJS app should transplant these (per migration plan they move under `core/` largely unchanged) and call them from store actions. Exact bind points:
|
||||
|
||||
**Construction (`init()` / `resizeMLP()`):**
|
||||
- `imlJoy = await WasmIML.create(2, N_OUTPUTS, [32,48,64], 1000, 1.0, 0.00001)` then `imlJoy.randomiseWeights(spreadLevel)`.
|
||||
- `imlHand = await WasmIML.create(14, N_OUTPUTS, [48,48,64], …)`.
|
||||
- On output-count change: `WasmIML.createWithWarmStart(snapshot, newCount, …)` for the joystick IML (preserves hidden weights, re-randomises new output nodes), fresh `create` for hand.
|
||||
- WASM module loaded from `wasm/nisps.js` (Emscripten). Note: the **engine MLP that runs audio in the worklet** is a separate WASM instance fixed at `MLP<2,10,14,18,126>` per CLAUDE.md — but in a-immersive the *inference path* is this `WasmIML`, sized to the active mode's output count.
|
||||
|
||||
**Per-frame inference (joystick/hand move, `onJoystickMove`/`onHandInput`):**
|
||||
- `iml.setInput(0,x); iml.setInput(1,y); iml.process(); const out = iml.getOutputs();` → `routeOutputs(out)`.
|
||||
|
||||
**`routeOutputs(out)` (2425) — the output fan-out** (bind one store action):
|
||||
- **synth**: apply `applyGroupOverrides(out[i], i)` per index → `synthVisualizer.setParams(overridden)`; throttled (≥50 ms) + dead-zone (>0.002) `activeEngine.setParam(i, v)` for indices < engine paramCount; shared-mode EOC params routed to `eocChain.setParam`.
|
||||
- **midi-cc**: per CC, `applyGroupOverride(out[i],curve,min,max)` → `midiOutput.sendBatch([{channel,cc,value:round(v*127)}])`.
|
||||
- **audio-canvas**: `audioCanvas.setOutputs(out)`.
|
||||
- **visual**: per-index `applyGroupOverride` (or `fixedValue` if frozen) → `visualizer.setParams(vis)`.
|
||||
|
||||
**RL / training engine calls** (the interaction→engine mapping the prompt cares about):
|
||||
| UI action | Engine call |
|
||||
|---|---|
|
||||
| **thumbsUp / + / key 2 / RB / hand-up** | `iml.addExample(inputs, rawOutputs)` **+** `iml.trainAsync(cb)` |
|
||||
| **thumbsDown / − / key 1 / LB / hand-down** | `iml.moveWeights(noiseLevel, spread)` *(no example)* |
|
||||
| **Add Example button** | `iml.addExample(getCurrentInputs(), [...rawParamValues])` |
|
||||
| **Train button / gamepad A** | `iml.trainAsync(cb)` (async; updates loss canvas) |
|
||||
| **Randomize / gamepad X** | `iml.randomiseWeights(spread)` → re-infer; resets noise to 0.05 |
|
||||
| **Clear Ex / gamepad B** | `iml.clearDataset()` |
|
||||
| **Clear All** | `clearDataset()` + reset `lossHistory/bestLoss/iterations` + `clearState()` |
|
||||
| **Undo / key Z** | `iml._setFlatWeights(snapshot.weights)` → re-infer |
|
||||
| **Visual preset chip** | clear joystick dataset, `addExample` × preset rows, `iml.train()` (sync) |
|
||||
| **NISPS sliders** | write `spreadLevel` / `noiseLevel` / `iml.learningRate` / `iml.maxIterations` / `iml.convergenceThreshold` |
|
||||
|
||||
**WasmIML public surface to rely on** (stable contract): `setInput/setInputs/getOutputs/process/inferBatch`, `addExample/clearDataset/exampleCount`, `train()/trainAsync(cb)/isTraining/evalLoss()`, `randomiseWeights(spread)`, `moveWeights(speed,spread,pinMask?)`, `extractWeights()/createWithWarmStart()`, `_getFlatWeights()/_setFlatWeights()`, `getLayerStats()`, `lastLoss/bestLoss/lossHistory/totalTrainingIterations`, `dataset.features/labels`, `destroy()`. Training runs in a disposable Web Worker (`nisps-wasm-worker.js`).
|
||||
|
||||
**SynthEngine (`C15Adapter`) surface**: `id`, `displayName`, `paramMeta` (array of `{id,name,min,max,init,curve,group}`), `paramCount`, `init(ctx?)`, `stop()`, `running`, `setParam(index, normalized)`, `noteOn(note,vel)`, `noteOff(note)`, `getOutputNode()`, `setMasterVolume(v)`, `panic()`. Audio is started lazily on the play button (`activeEngine.init()`), which also wires EOC and (behind flag) ShapeSeq. Arpeggiator (`Arpeggiator`) and `MIDIInput` are attached to the active engine.
|
||||
|
||||
### Where the NEW 3-mode feedback selector ("Down Action") slots in
|
||||
Today **`−` (thumbsDown) is hardcoded to `moveWeights`** (explore) while **`+` is hardcoded to `addExample`+`train`** (reinforce). The new feature wants the `−` action to be selectable among 3 feedback modes. Slot it as a **`<DownActionSelector>`** control:
|
||||
|
||||
- **Placement (UI)**: a small segmented control. Most faithful spot is inside the **Training drawer** (a new "Down Action" row above the example actions) and/or a compact 3-way toggle adjacent to the RL cluster (mirroring how the existing `rl-label`/EOC label sit above the buttons in linked mode). It must be reachable without opening a drawer if it's meant for live use — consider a tiny pill under `<StatusLine>` or next to the `−` button.
|
||||
- **State**: add `downAction: 'explore' | <mode2> | <mode3>` to `mlStore` (persist it in the state blob).
|
||||
- **Bind point**: `onThumbsDown()` is the single chokepoint. Branch on `mlStore.downAction`:
|
||||
- `explore` (current): `target.moveWeights(noiseLevel, spread)`.
|
||||
- the two new modes: call the corresponding engine primitive (e.g. a "negative example" path via `addExample` of a contrasting target, or a stronger/weighted `moveWeights`, or `randomiseWeights` — whatever the 3 modes are defined to do). `WasmIML.moveWeights` already accepts an `outputPinMask`, so a "pin-aware explore" mode is a natural third option.
|
||||
- Keep `pushUndoSnapshot()` + `updateNoiseRing()` + `flash` shared across all branches.
|
||||
- The existing `_rlTarget()` indirection (which lets EOC linked-mode redirect feedback to `imlEoc`) should compose with the new selector — i.e. `downAction` chooses *what* the feedback does, `_rlTarget()` chooses *which network* receives it.
|
||||
|
||||
---
|
||||
|
||||
## 7. Reference data tables (must match exactly)
|
||||
|
||||
### Visual mode: 20 named outputs (`VISUAL_PARAM_NAMES`, a-app.js:46 + `visualizer.setParams`:159)
|
||||
Names: `Flow, Scale, Speed, Hue, Spread, Size, Trail, Turb, Attract, Radius, DispRate, DispAmt, Lifetime, Respawn, Advection, Inertia, Drag, Repulse, RepCnt, RepRate`. Colors in `VISUAL_PARAM_COLORS` (a-app.js:51, e.g. `Flow=#ff6a00, Scale=#00ccff, Speed=#ff6600, Hue=#ff00cc, …`). Output→visual-param ranges (verbatim, for the FlowFieldCanvas): `angleOffset=out0*2π`, `scale=0.001+out1*0.009`, `speed=0.5+out2*4.5`, `hueBase=out3*360`, `hueSpread=out4*120`, `particleSize=1+out5*5`, `fadeRate=0.01+out6*0.14`, `turbulence=out7*2`, `attractStrength=0.1+out8*2.9`, `attractRadius=40+out9*420`, `dispersionRate=0.2+out10*8`, `dispersionAmount=out11*3`, `particleLifetime=30+out12*470`, `respawnStyle=out13`, `advectionMode=out14`, `inertia=out15*0.98`, `drag=out16*0.35`, `repulsorStrength=out17*4.5`, `repulsorCount=floor(out18*4.999)`, `repulsorOrbitRate=0.1+out19*2.9`. 400 particles, value-noise flow field with central attractor, dispersion pulses, and orbiting repulsors. Transplant `FlowFieldVisualizer` as-is.
|
||||
|
||||
### Synth mode: C15 sections (`SYNTH_SECTIONS`, a-app.js:217) — for grouping the 126 bars
|
||||
`Env A(7), Env B(7), Env C(6), Osc A(5), Osc B(5), Shp A(6), Shp B(6), Comb(8), SVF(9), Gap(6), FB Mix(9), Out Mix(14), Cabinet(8), Flanger(13), Echo(7), Reverb(6), Unison(3), Mono(1)` = 126, each with a section color. Param data (id/name/label/default/safeMin/safeMax/bipolar) is `SYNTH_PARAM_MAP` (`js/synth/param-map.js`, 126 entries) — transplant as data, unchanged.
|
||||
|
||||
### Curve math (`param-map.js`:287) — reproduce exactly
|
||||
```
|
||||
applyCurve(v, c) = c===0.5 ? v : v^(2^(4*(c-0.5))) // 0.5 = linear
|
||||
applyGroupOverride(v,c,min,max) = min + applyCurve(v,c) * (max-min)
|
||||
```
|
||||
|
||||
### `?tame` (default 1): seeds each C15 param's default override range to `[safeMin*tame, 1-(1-safeMax)*tame]` (a-app.js:248). `?spread` (default 0.6), `?preset`, `?debug`, `?shapeseq=1` are the URL params.
|
||||
|
||||
---
|
||||
|
||||
## 8. Layout / positioning cheat-sheet (fixed coordinates to match)
|
||||
- Back button: `top:45px; left:8px; 36×36`. Synth quick controls: `top:45px; left:52px` (row). Heatmap strip: `top:0; height:22px`. Joystick: `bottom:100px; left:24px; 160×160`. Hand PIP: same anchor, `180×135`. RL buttons: `bottom:36px; left:50%` centered, 64×64 circles, 12 px grid gap, undo 28×28 spanning. EOC RL: `bottom:36px; left:calc(50%+100px)`, 48×48. Status line: `bottom:8px; center`. Dock: `right:8px; top:50%` vertical, 36×36 icons, 16 px radius glass. Drawer stack: `right:calc(48px+16px); top:28px; bottom:28px; width:260px`, scrolls, `pointer-events:none` on stack / `auto` on each drawer. Drawer max-height 400 px. Group drawer 320 px, param popup 260 px — both portalled to body and positioned relative to the hovered element.
|
||||
- z-index ladder: canvases 0–1, heatmap strip 20, tooltip 25, joystick/back/quick 30, status 40, drawers/RL 45, dock 50, group-drawer/popup 50, dev panel 90, help overlay 100.
|
||||
|
||||
---
|
||||
|
||||
## 9. SolidJS implementation notes
|
||||
- **Canvases**: `<FlowFieldCanvas>` / `<SynthVisualizer>` / `<JoyMapCanvas>` are components with a `ref` and an internal draw method; the **shell's single rAF** calls the active one (don't give each its own loop — matches `animate()` and avoids double-drawing). Resize via a window listener that re-inits particles.
|
||||
- **Heatmap**: render `<For>` over `count` cells once per mode change; update bar widths via a `createEffect` reading the outputs signal — but throttle DOM writes (the deployed app writes width % directly each frame; with 126 cells prefer writing only changed cells).
|
||||
- **Drawers/overlays**: build headless primitives (`Drawer`, `Overlay`, `Popover`, `Dock`) per the migration plan; the dock/drawer toggle is just boolean signals in `uiStore`. Independent open state (multiple drawers can coexist).
|
||||
- **Pointer capture**: the joystick, heatmap cells, synth bars, curve canvases, and dual-range sliders all use `setPointerCapture` + manual drag math — replicate the exact thresholds (3 px click-vs-drag on heatmap; double-tap 350 ms; group-curve `dy/80`; per-param curve drag).
|
||||
- **Untracked debug probe**: keep `window.__nisps` (synchronous, bypasses reactivity via `untrack`/`batch`) for Playwright — its API is the table in §6 plus `getLayerStats/inferBatch/saveState/evalLoss`. Also `window.__nispsEoc` (training target). These are how E2E tests drive the app.
|
||||
|
||||
---
|
||||
|
||||
## 10. Gaps / risks for a faithful clone
|
||||
|
||||
**Core (must build to match the page):**
|
||||
- WASM IML inference + sync/async training + `moveWeights`/`randomiseWeights`/warm-start — the whole ML surface. Non-negotiable; transplant `nisps-wasm.js` + `nisps.wasm`.
|
||||
- C15 synth engine (`C15Adapter`/`C15Bridge` + `param-map.js` + `presets.js`) — runs via AudioWorklet + SharedArrayBuffer; needs COOP/COEP headers (the deployed app uses `serve-coop.py`). The 126-param map, tiered presets, group/param overrides, and the synth visualizer are all core to "synth mode."
|
||||
- Flow-field visualizer (20 outputs), heatmap + override popup, joystick + noise ring, RL cluster + keyboard + undo, dock/drawers, status line, help modal, the 6-slider NISPS drawer, visual presets, persistence. All core.
|
||||
- Arpeggiator + MIDI input (used by synth mode); MIDI CC output mode (`midi-output.js`, `midi-cc-map.js`, `midi-cc-presets.js`).
|
||||
|
||||
**Deferrable / flagged (render scaffolds but feature can lag):**
|
||||
- **Hand tracking / MediaPipe** (`hand-tracker.js`, PIP, gesture ring, 14-input `imlHand`) — heavy dep, camera permission; the help text and Mode toggle reference it but it can be a later add. Note as optional.
|
||||
- **EOC Effects Chain** (`eoc/`, `eoc-chain-ui.js`, `eoc-joystick.js`, linked/independent/shared NISPS modes, second `imlEoc`, FX RL buttons, keys 3/4) — large subsystem; the dock has an "FX" icon but it's a self-contained module. Deferrable.
|
||||
- **Modular engine** (512 params, `modular-engine.js`, `modular-ui.js` — 51 KB UI) and **Faust additive/FM engines** (`faust-engine-base.js`, additive/fm presets) — accessed via the engine switcher in the Synth drawer. The switcher + C15 are core; the alternative engines are deferrable (and Faust needs a running AudioContext + the Faust runtime). CLAUDE.md flags Faust engines as not-yet-wired in the new codebase.
|
||||
- **Audio Canvas mode** (`audio/audio-canvas.js`, 60 KB) — a fourth output mode with dynamic output count; scaffolded but can defer.
|
||||
- **ShapeSeq** (`shapeseq/`, behind `?shapeseq=1`) — sequencer; explicitly flag-gated, defer.
|
||||
- **OSC bridge** (`nisps/osc-client.js`, `synth/osc-output.js`, `osc-bridge/`) — desktop/hardware bridge, not used in the default browser path; defer.
|
||||
- **Gamepad** — small (`gamepad.js`), low-risk; include if cheap, else defer (it just mirrors joystick + RL).
|
||||
|
||||
**Behavioural risks to watch:**
|
||||
- Output-count changes trigger `confirm()` dialogs and weight warm-start — replicate the guard so mode switches don't silently wipe training.
|
||||
- Param throttling (50 ms / 0.002 dead-zone) on synth param sends is load-bearing (prevents ring-buffer flooding at 126×30 fps); keep it.
|
||||
- The two preset systems (visual examples vs synth override tiers) are easy to conflate — keep them as distinct stores/actions.
|
||||
- Frozen/muted semantics differ between the heatmap popup (`frozen`) and the group drawer (`muted`) but map to the same underlying field via the `getParamOverride` adapter — preserve that mapping (`muted`↔`frozen`).
|
||||
- Help/onboarding, `nisps-help-seen`, and the 10 s autosave are small but part of the felt experience.
|
||||
|
||||
**Files that ARE the engine layer to bind to** (absolute paths): `/home/w1n5t0n/deployments/meml-aimmersive/js/nisps/nisps-wasm.js`, `/home/w1n5t0n/deployments/meml-aimmersive/js/nisps/nisps-wasm-worker.js`, `/home/w1n5t0n/deployments/meml-aimmersive/js/nisps/dataset.js`, `/home/w1n5t0n/deployments/meml-aimmersive/js/synth/c15-adapter.js`, `/home/w1n5t0n/deployments/meml-aimmersive/js/synth/c15-bridge.js`, `/home/w1n5t0n/deployments/meml-aimmersive/js/synth/param-map.js`, `/home/w1n5t0n/deployments/meml-aimmersive/js/synth/presets.js`, `/home/w1n5t0n/deployments/meml-aimmersive/js/ui/visualizer.js`. UI behaviour source of truth: `/home/w1n5t0n/deployments/meml-aimmersive/js/a-app.js`. Theme: `/home/w1n5t0n/deployments/meml-aimmersive/css/a-immersive.css`. DOM + copy: `/home/w1n5t0n/deployments/meml-aimmersive/a-immersive.html`.
|
||||
502
docs/redesign/backends-spec.md
Normal file
502
docs/redesign/backends-spec.md
Normal file
|
|
@ -0,0 +1,502 @@
|
|||
# Output Backends — Specification (`docs/redesign/backends-spec.md`)
|
||||
|
||||
*Workstream E. Design-only, read-only audit 2026-06-27. The new app is Vite + React + TS in `manifold/`, wired to the parity-tested TS engine lifted from `playground/src`. British spelling in product copy. The built-in synth is the **"Powerful Synth Engine"** — the string "C15" must never reach the user.*
|
||||
|
||||
> **Naming guard (non-negotiable).** The codename `C15` survives only in internal module/file names that the user never sees (`c15-adapter.js`, `c15-bridge.js`). Every label, tooltip, dock entry, menu item, status string, and aria-label says **"Powerful Synth Engine"** (or just "Synth"). A lint allowlist + a Playwright assertion (`expect(page).not.toContainText('C15')`) enforce this across `manifold/` and the VCV panel SVG/strings.
|
||||
|
||||
---
|
||||
|
||||
## 0. The one idea: backends are adapters behind one interface
|
||||
|
||||
Today the deployed app (`js/a-app.js`) fans output out to four ad-hoc sinks inline in `routeOutputs()` (`a-app.js:2425`): synth (`activeEngine.setParam`), MIDI CC (`midiOutput.sendBatch`), audio-canvas, and visual (`visualizer.setParams`). Each has its own throttle, dead-zone, and override handling copy-pasted. That fan-out *is* the debt this workstream removes.
|
||||
|
||||
**Replace it with one `OutputBackend` interface and a registry. Exactly one backend is "active" at a time, chosen in the Console dock.** The reactive spine (per `engine-architecture.md` §2 and `findings-design-and-manifold.md` §4) ends in a single side-effect that calls `activeBackend.send(routedOutput)`. Swapping backends swaps nothing else — the input pipeline, ML, output pipeline, training loop, and verdict loop are all backend-agnostic.
|
||||
|
||||
```
|
||||
gesture → input pipeline → ML (WasmIML) → output pipeline → [ activeBackend.send(routed) ]
|
||||
│
|
||||
┌───────────────┬───────────────┬───────────────┬─────────────┼──────────────┐
|
||||
WebAudioBackend ParticleBackend WebMidiBackend OscBridgeBackend CvGateBackend VcvBridgeBackend
|
||||
(Powerful Synth) (flow field) (advanced CC) (paths+ranges) (1V/oct+gate) (8→model→16, LED rings)
|
||||
```
|
||||
|
||||
The active backend is a property of the **output dock** (the Console's right rail / a-immersive's Mode drawer). Backends self-describe (id, label, capability) so the dock renders a picker without hard-coding the list.
|
||||
|
||||
---
|
||||
|
||||
## 1. The `OutputBackend` adapter interface (TS)
|
||||
|
||||
Lives at `manifold/src/engine/backends/backend.ts`. The engine never imports a concrete backend; it imports the interface + the registry. Concrete backends may import the engine's pure helpers (curves, param-map data) but **never** React.
|
||||
|
||||
```ts
|
||||
// manifold/src/engine/backends/backend.ts
|
||||
|
||||
/** What a backend needs to know about the active mode to map outputs. */
|
||||
export interface BackendContext {
|
||||
modeId: string;
|
||||
outputCount: number; // model output dims actually in use (≤ 126)
|
||||
paramMeta: ReadonlyArray<ParamMeta>; // name/label/min/max/curve/group per output
|
||||
sampleRate?: number; // for audio backends
|
||||
audioContext?: AudioContext; // lazily provided; only audio backends use it
|
||||
}
|
||||
|
||||
export interface ParamMeta {
|
||||
id: string; // stable machine id, e.g. 'Env_A_Att'
|
||||
label: string; // user-facing
|
||||
min: number; // baseline range floor (normalised 0..1 maps here)
|
||||
max: number; // baseline range ceil
|
||||
curve: number; // 0..1, 0.5 = linear (see §3 universal mapping)
|
||||
group: string; // for colour grouping (LED rings, heatmap)
|
||||
}
|
||||
|
||||
export type BackendId =
|
||||
| 'synth' | 'particles' | 'midi' | 'osc' | 'cvgate' | 'vcv';
|
||||
|
||||
export interface BackendDescriptor {
|
||||
id: BackendId;
|
||||
label: string; // dock label — NEVER "C15"
|
||||
description: string;
|
||||
/** crossOriginIsolated / WebMIDI / WebSocket etc. availability probe. */
|
||||
isAvailable(): boolean;
|
||||
/** true when this backend can ALSO feed inputs back (VCV bridge, OSC return). */
|
||||
bidirectional?: boolean;
|
||||
}
|
||||
|
||||
export interface OutputBackend {
|
||||
readonly descriptor: BackendDescriptor;
|
||||
|
||||
/** Called once when this backend becomes active. May be async (audio start,
|
||||
* WS connect, MIDI access). Resolve only when ready to receive send(). */
|
||||
start(ctx: BackendContext): Promise<void>;
|
||||
|
||||
/** Hot per-frame path. `routed` is the post-pipeline Float32Array (0..1),
|
||||
* length = ctx.outputCount. MUST NOT allocate; MUST NOT mutate `routed`.
|
||||
* Throttling/dead-zone live INSIDE each backend (rates differ per sink). */
|
||||
send(routed: Float32Array): void;
|
||||
|
||||
/** Called when switching away or unmounting. Release WS/MIDI/audio/threads. */
|
||||
teardown(): Promise<void>;
|
||||
|
||||
/** Optional input return path for bidirectional backends. The engine
|
||||
* subscribes; values drive model inputs (e.g. VCV CV-in, OSC /nisps/input). */
|
||||
onInputs?(cb: (values: Float32Array) => void): () => void;
|
||||
|
||||
/** Optional: backends that own training transport (VCV/OSC bridge) expose
|
||||
* the remote verdict/example surface here. See §7. */
|
||||
remote?: RemoteTrainingBridge;
|
||||
}
|
||||
```
|
||||
|
||||
Registry (`manifold/src/engine/backends/registry.ts`): a `Map<BackendId, () => OutputBackend>` of lazy factories. The dock reads `descriptors` (filtered by `isAvailable()`); selecting one calls `engine.setBackend(id)`, which `teardown()`s the old and `start()`s the new with the current `BackendContext`.
|
||||
|
||||
**Why `send(Float32Array)` and not per-param events:** matches the spine's single transferable-buffer effect (`engine-architecture.md` §2.1), keeps the hot path allocation-free, and lets each backend decide its own decimation. The legacy code already proves the pattern — every sink takes the full output vector and self-throttles (synth 50 ms/0.002 dead-zone `a-app.js:2425`; MIDI 50 ms/Δ1 `midi-output.js:114`; OSC 50 ms/0.002 `osc-output.js:97`).
|
||||
|
||||
---
|
||||
|
||||
## 2. The backends
|
||||
|
||||
### 2.1 Built-in Synth — the "Powerful Synth Engine" (`web-audio.ts` + `synth.ts`)
|
||||
|
||||
Two cooperating backends, both labelled as the synth in the UI, but architecturally distinct:
|
||||
|
||||
- **`WebAudioBackend`** wraps the parity-tested repo engine: `EngineHost` (`playground/src/audio/engine-host.ts`) + the worklet `nisps-processor.ts` running `_nisps_engine_process_block` (`nisps/wasm/bindings.cpp`). This is the firmware-parity audio path — the engine *is* the sound. `send()` → `EngineHost.setParams(routed)` → worklet. This is the default and the one that satisfies browser-parity chokepoint C.
|
||||
- **`PowerfulSynthBackend`** (the C15 path) wraps `deployments/meml-aimmersive/js/synth/c15-adapter.js` → `c15-bridge.js` (SharedArrayBuffer ring + its own `c15_engine.wasm` worklet). Param mapping comes from `param-map.js` (`SYNTH_PARAM_MAP`, 126 entries) and `presets.js` (tiered presets). `setParam(index, normalised)` maps index→hardware id (`c15-adapter.js:121`). This is browser-only (firmware has no C15), and its 126-param surface + group/section overrides power the synth visualiser and the group-override drawer.
|
||||
|
||||
**Reuse, verbatim:** `c15-adapter.js`, `c15-bridge.js`, `param-map.js`, `presets.js` move under `manifold/src/engine/backends/synth/` unchanged (internal names keep "c15"; UI strings do not). The `engine-interface.js` `SynthEngine` base maps cleanly onto `OutputBackend`: `init(ctx)`→`start`, `setParam` loop driven by `send`, `stop`→`teardown`. Curve/override math is `applyCurve`/`applyGroupOverride` (`param-map.js:287`) — fold into the universal mapping (§3).
|
||||
|
||||
**Throttle (keep — load-bearing):** ≥50 ms send interval + 0.002 dead-zone per param prevents flooding the C15 ring buffer at 126×30 fps (`a-immersive.html` clone-spec §10 flags this).
|
||||
|
||||
`isAvailable()`: WebAudio + (for the C15 path) `crossOriginIsolated === true` (SAB needs COOP/COEP; already server-scoped per `findings-engine-surface.md`).
|
||||
|
||||
### 2.2 Particle System — faithful port of `visualizer.js` (`particles.ts`)
|
||||
|
||||
A `ParticleBackend` whose `send(routed)` calls a ported `FlowFieldVisualizer.setParams(routed)`. The visual canvas runs in its own `requestAnimationFrame` loop (per `findings-design-and-manifold.md` §4.3 — rAF touches drawing only, never inference); `send()` only updates the param struct. The port MUST look and behave **exactly** as the deployed version. The full algorithm is documented in §4 so the React port is byte-faithful.
|
||||
|
||||
`isAvailable()`: always (Canvas2D). This backend produces no audio — it is the "visual" output mode.
|
||||
|
||||
### 2.3 MIDI out — advanced CC config (`web-midi.ts`)
|
||||
|
||||
A `WebMidiBackend` wrapping the salvaged `midi-output.js` (Web MIDI API, `sendBatch`, per-CC dead-zone + 50 ms throttle, device hot-plug handling — `midi-output.js`). The **advanced CC config comes from workstream D's config model**, persisted as a CC map: per output dim → `{ name, cc (0–127), channel (1–16), min, max, curve, muted, fixedValue }`. The map shape and storage are already defined in `midi-cc-map.js` (`createCCParam`, `loadCCMap`/`saveCCMap`, well-known `CC_NAMES`, default 8-CC starter set). Lift that file verbatim into `manifold/src/engine/backends/midi/cc-map.ts`.
|
||||
|
||||
`send(routed)`: for each non-muted CC param, `value = round(applyGroupOverride(routed[i], curve, min, max) * 127)`; batch the changed ones; `midiOutput.sendBatch(...)`. Storage key stays engine-scoped (`nisps-midi-cc-map:<modeId>`) for migration continuity.
|
||||
|
||||
`isAvailable()`: `!!navigator.requestMIDIAccess`.
|
||||
|
||||
### 2.4 OSC out — paths + ranges (`osc-bridge.ts`)
|
||||
|
||||
An `OscBridgeBackend` that **salvages the existing OSC bridge** rather than reinventing it. Two pieces already exist and are good:
|
||||
|
||||
- **Browser client:** `deployments/meml-aimmersive/js/synth/osc-output.js` (param-named WS messages, 50 ms/0.002 dead-zone) and the richer `js/nisps/osc-client.js` (`NispsOscClient` — `EventTarget`, `sendState`/`sendWeights`/`sendParams`, `onOutputsReceived`/`onInputsReceived`, auto-reconnect with backoff). Lift `osc-client.js` as the transport (it already speaks the bridge protocol and is bidirectional).
|
||||
- **Bridge server:** `deployments/meml-aimmersive/osc-bridge/bridge.ts` — a Deno WebSocket↔UDP-OSC bridge, zero-dependency OSC encode/decode, bidirectional. Keep it as-is; it is the canonical transport between browser and any OSC target (VCV, SuperCollider).
|
||||
|
||||
**OSC path + range contract (salvaged from `bridge.ts`):**
|
||||
|
||||
| Direction | Address | Args | Meaning |
|
||||
|---|---|---|---|
|
||||
| browser→target | `/nisps/<param_name>` | `f` | one param, **post-baseline-mapping value** (see §3) |
|
||||
| browser→target | `/nisps/state` | `s` | full JSON state (weights + examples + config) |
|
||||
| browser→target | `/nisps/weights` | `s` | weights-only JSON |
|
||||
| target→browser | `/nisps/output` | `f…f` | output vector (visualisation / monitoring) |
|
||||
| target→browser | `/nisps/input` | `f…f` | input vector → drives model inputs |
|
||||
|
||||
**Ranges:** OSC floats are sent in the param's mapped range by default (`applyGroupOverride` applied before send, matching `osc-output.js`), with a per-backend toggle to send **raw normalised 0..1** instead (some OSC targets want 0..1 and do their own scaling). Address prefix (`/nisps`), target host/port (default `127.0.0.1:9000`), and listen port (default `9001`) are configurable — `bridge.ts` already exposes `--osc-host/--osc-port/--osc-prefix/--ws-port/--listen-port`.
|
||||
|
||||
`isAvailable()`: always (attempts WS to `ws://localhost:8765`; surfaces a "bridge not running" status if the connect fails — `osc-client.js` already reconnects with backoff). Bidirectional (`onInputs` wired to `/nisps/input`).
|
||||
|
||||
### 2.5 CV / gate backend (`cvgate.ts`)
|
||||
|
||||
For browser-side CV/gate there is no native hardware path, so this backend has **two transports** selectable in config:
|
||||
|
||||
1. **DC-coupled WebAudio CV** (browser-native): each output dim drives a `ConstantSourceNode` (or a sample-accurate `AudioWorklet` channel) whose `offset` = mapped voltage, summed/routed to the audio interface's output channels. Gate outputs are derived from a configurable threshold on a chosen dim (value > τ → high). Pitch (1V/oct) uses a per-output "voltage role" config: `{ role: 'cv' | 'gate' | 'voct', vmin, vmax, gateThreshold }`. This is the only way to emit real CV from a browser (DC-coupled interface required; surfaced as a caveat in the UI).
|
||||
2. **Bridged CV via VCV / OSC** (recommended default): reuse the OSC/VCV transport — the *VCV module's 16 CV outputs* (§5) are the real CV/gate jacks. In this mode `CvGateBackend` is a thin alias that delegates to `VcvBridgeBackend` with a "treat outputs as CV/gate" preset (per-output unipolar 0–10 V / bipolar ±5 V / 1V-oct, matching the VCV per-output range menu in `MEMLNaut.cpp:818`).
|
||||
|
||||
`isAvailable()`: WebAudio path always; native-CV quality flagged as "requires DC-coupled interface". **Open choice:** whether browser-native DC CV is worth shipping vs. making CV strictly a VCV-bridge concern (recommendation: ship the VCV-bridge alias first, defer DC-coupled WebAudio CV).
|
||||
|
||||
### 2.6 VCV Rack module — first-class (`vcv-bridge.ts` browser side + `vcv/` C++ side)
|
||||
|
||||
The headline new backend. A first-class **VCV Rack 2 module** (`MEMLNaut`) with **8 CV inputs → model → 16 CV outputs**, an **LED ring around each of the 16 outputs**, and a **browser↔VCV bridge** so the tool is controllable AND trainable from both inside Rack and entirely from the browser. Full design in §5–§7.
|
||||
|
||||
The browser-side adapter `VcvBridgeBackend` reuses `NispsOscClient` (§2.4) as transport. When active in **bridged mode**, the browser supplies inputs in real time (`/nisps/input`) and the verdict/example loop is mirrored over the bridge (`/nisps/state`, `/nisps/weights`, new `/nisps/feedback`).
|
||||
|
||||
---
|
||||
|
||||
## 3. Universal per-output baseline mapping
|
||||
|
||||
Every backend shares ONE baseline mapping from a normalised model output `v ∈ [0,1]` to a sink value, so behaviour is identical across sinks and the override UI (heatmap popup, group drawer) is backend-agnostic. This is the existing curve math (`param-map.js:287`), promoted to `manifold/src/engine/backends/mapping.ts`:
|
||||
|
||||
```ts
|
||||
// 0.5 = linear; <0.5 ease-in, >0.5 ease-out. Bit-identical to legacy + nisps/core/math.hpp.
|
||||
export function applyCurve(v: number, c: number): number {
|
||||
return c === 0.5 ? v : Math.pow(v, Math.pow(2, 4 * (c - 0.5)));
|
||||
}
|
||||
|
||||
/** Per-output baseline: curve, then scale into [min,max]; honour freeze/mute. */
|
||||
export function mapOutput(v: number, p: OutputMapping): number {
|
||||
if (p.frozen) return p.fixedValue; // pinned, ignores model
|
||||
return p.min + applyCurve(v, p.curve) * (p.max - p.min);
|
||||
}
|
||||
|
||||
export interface OutputMapping {
|
||||
min: number; max: number; curve: number; // baseline range + curve
|
||||
frozen: boolean; fixedValue: number; // pin
|
||||
muted?: boolean; // excluded from this sink
|
||||
}
|
||||
```
|
||||
|
||||
Backends then apply only their **sink-specific final transform** on top of the baseline:
|
||||
|
||||
| Backend | Baseline → sink transform |
|
||||
|---|---|
|
||||
| Powerful Synth | `mapOutput` → `setParam(i, value)` (value already in param range) |
|
||||
| Particles | **raw 0..1** (the visualiser owns its own param ranges — see §4); curve/freeze still applied, min/max default to 0..1 |
|
||||
| MIDI CC | `mapOutput` → `round(value * 127)` clamped 0–127 |
|
||||
| OSC | `mapOutput` (or raw 0..1 if "send raw" toggled) → `/nisps/<name> <f>` |
|
||||
| CV/gate | `mapOutput` → voltage by role (`cv`: `value*10` or `(value-0.5)*10`; `voct`: 1V/oct; `gate`: `value>τ ? high : 0`) |
|
||||
| VCV | model output `0..1` sent raw over bridge; the **module** applies its own per-output range + attenuverter (`MEMLNaut.cpp:342 outputToVoltage`) |
|
||||
|
||||
The override store (one per mode, `OutputMapping[]`) is owned by the engine and shared by all backends; freeze/mute/curve/range edits in the UI apply uniformly. Note the legacy split where the heatmap calls it `frozen` and the group drawer calls it `muted` but both map to the same field (`aimmersive-clone-spec.md` §10) — unify to the single `OutputMapping` above.
|
||||
|
||||
---
|
||||
|
||||
## 4. Particle system — faithful-port plan with the documented algorithm
|
||||
|
||||
Source of truth: `deployments/meml-aimmersive/js/ui/visualizer.js` (289 lines, read in full). The React port (`manifold/src/engine/backends/particles/flow-field.ts`) must reproduce this **exactly**. Below is the complete algorithm with line citations so the port is verifiable.
|
||||
|
||||
### 4.1 Constants & noise source
|
||||
- **Particle count: `numParticles = 400`** (`visualizer.js:53`).
|
||||
- **Noise: a self-contained 2-D value/Perlin-style noise**, not a library. A `Uint8Array(512)` permutation table `PERM` is built once at module load by Fisher–Yates shuffling `[0..255]` then duplicating (`:5–14`). `fade(t)=t³(t(6t−15)+10)` (`:16`), `lerp` (`:17`), `grad(hash,x,y)` using `hash & 3` (`:19–24`), `noise2D(x,y)` doing the standard 4-corner bilinear-with-fade interpolation (`:26–44`). **The shuffle uses `Math.random()` at module load**, so the field is non-deterministic per page load — the port must keep this (or seed it; flagged as an open choice if determinism is wanted for tests).
|
||||
- `TWO_PI = Math.PI*2` (`:46`); `this.time` advances `+= 0.003` per `draw()` (`:185`).
|
||||
|
||||
### 4.2 Output→param mapping (20 dims, `setParams`, `:159–181`) — reproduce verbatim
|
||||
Guard: returns early if `outputs.length < 20` (`:160`). Then:
|
||||
```
|
||||
angleOffset = out0 * TWO_PI
|
||||
scale = 0.001 + out1 * 0.009
|
||||
speed = 0.5 + out2 * 4.5
|
||||
hueBase = out3 * 360
|
||||
hueSpread = out4 * 120
|
||||
particleSize = 1 + out5 * 5
|
||||
fadeRate = 0.01 + out6 * 0.14
|
||||
turbulence = out7 * 2
|
||||
attractStrength = 0.1 + out8 * 2.9
|
||||
attractRadius = 40 + out9 * 420
|
||||
dispersionRate = 0.2 + out10 * 8
|
||||
dispersionAmount = out11 * 3
|
||||
particleLifetime = 30 + out12 * 470
|
||||
respawnStyle = out13 // 0=random,1=edge,2=center-burst
|
||||
advectionMode = out14 // flow→orbit→radial blend
|
||||
inertia = out15 * 0.98
|
||||
drag = out16 * 0.35
|
||||
repulsorStrength = out17 * 4.5
|
||||
repulsorCount = floor(out18 * 4.999) // 0..4
|
||||
repulsorOrbitRate = 0.1 + out19 * 2.9
|
||||
```
|
||||
The 20 output names (for labels/the heatmap colouring) are `VISUAL_PARAM_NAMES` in `a-app.js:46`: `Flow, Scale, Speed, Hue, Spread, Size, Trail, Turb, Attract, Radius, DispRate, DispAmt, Lifetime, Respawn, Advection, Inertia, Drag, Repulse, RepCnt, RepRate`.
|
||||
|
||||
### 4.3 Particle lifecycle (`:104–156`)
|
||||
- `makeParticle(id)`: random x,y in canvas; `age = floor(rand*particleLifetime)`; `life = computeLifetime()`; `vx=vy=0` (`:104–114`).
|
||||
- `computeLifetime()`: `max(10, floor(particleLifetime * (0.65 + rand*0.7)))` (`:116–119`).
|
||||
- `respawnParticle(p)`: `mode = min(2, floor(respawnStyle*2.999))` (`:122`):
|
||||
- **edge (1):** spawn on a random one of 4 sides; give inward impulse of magnitude 2.0 toward centre (`:125–137`).
|
||||
- **center-burst (2):** spawn within radius `min(w,h)*0.08` of centre at random angle; velocity `2.5` outward along that angle (`:138–145`).
|
||||
- **random (0):** random position; velocity `(rand*2−1)*0.5` each axis (`:146–152`).
|
||||
- reset `age=0`, `life=computeLifetime()`.
|
||||
|
||||
### 4.4 Per-frame `draw()` (`:183–287`) — the exact integration
|
||||
1. `time += 0.003`. **Trail fade:** fill whole canvas with `rgba(13,13,13, fadeRate)` (`:188`) — this is the trail length control, not a clear.
|
||||
2. For each particle, with centre `(cx,cy)=(w/2,h/2)`:
|
||||
- **Flow field:** `nx=p.x*scale`, `ny=p.y*scale`; `angle = noise2D(nx+time, ny)*TWO_PI + angleOffset`; `curl = noise2D(nx+100, ny+100+time*0.5)*turbulence` (`:196–199`).
|
||||
- **Three advection fields** (`:202–214`): flow `(cos(angle+curl), sin(angle+curl))*speed`; orbit = perpendicular to the radial-from-centre unit vector, `*speed`; radial = radial unit vector `*speed`.
|
||||
- **Blend by `advectionMode`** (`:216–225`): `modeBlend = advectionMode*2`; if `<1` lerp(flow→orbit, modeBlend) else lerp(orbit→radial, modeBlend−1).
|
||||
- **Inertia + drag** (`:226–229`): `vx = vx*inertia + targetVx*(1−inertia)` (same for vy); then `vx *= (1−drag)`.
|
||||
- `nextX = x+vx`, `nextY = y+vy`.
|
||||
- **Central attractor** (`:233–242`): pull toward centre with `falloff = 1/(1+normDist²)` where `normDist = min(dist/attractRadius, 2)`; add `nCenter * attractStrength * falloff`.
|
||||
- **Dispersion pulse** (`:244–248`): `pulse = 0.5+0.5*sin(time*dispersionRate + id*0.07)`; subtract `nCenter * dispersionAmount*pulse*falloff` (pushes outward near centre).
|
||||
- **Orbiting repulsors** (`:250–264`): `repulsorRadius = min(w,h)*0.28`; for each of `repulsorCount`: phase from `time*repulsorOrbitRate + (r/4)*TWO_PI`, `wobble = 0.6+0.15*r`, position uses `cos(phase*(1+wobble))`/`sin(phase*(1.3+wobble))`; inverse-square push `force = repulsorStrength*(650/distSq)` with `distSq = d²+160`.
|
||||
- **Wrap edges** (`:270–273`); `age += 1`; respawn when `age >= life` (`:275–276`).
|
||||
- **Colour** (`:279–282`): `hue = (hueBase + (id/numParticles)*hueSpread) % 360`; `lightness = 50 + sin(id*0.1 + time)*15`; fill `hsl(hue, 75%, lightness%)`; `arc(x,y,particleSize)`.
|
||||
|
||||
### 4.5 Port plan
|
||||
- **Class, not component.** `FlowFieldVisualizer` is a plain TS class taking a `<canvas>` ref — identical to today. The React `<ParticleCanvas>` mounts it in `onMount`, drives `draw()` from one `requestAnimationFrame` loop, and calls `resize()` on the window resize handler (DPR scaling at `:84–92`). React renders the canvas element; the class owns all pixels.
|
||||
- **`ParticleBackend.send(routed)`** → `viz.setParams(routed)` (no alloc; just field writes). Because the visualiser reads `outputs[0..19]`, the backend asserts `ctx.outputCount >= 20` and slices/pads to 20.
|
||||
- **Faithfulness gate (Playwright):** pixel-diff a fixed seed (seed the `PERM` shuffle behind a `?seed=` for tests) at fixed param vectors against a golden capture from the deployed app; assert SSIM ≥ threshold. Also unit-test `setParams` mapping numerically (the §4.2 table).
|
||||
- **Verbatim copy is allowed**: the noise + integration math is pure and has no DOM coupling beyond `ctx`/`canvas`; lift `:1–288` essentially unchanged into TS, add types, keep numeric constants exact.
|
||||
|
||||
---
|
||||
|
||||
## 5. VCV Rack module design (`MEMLNaut`, 8 in / 16 out)
|
||||
|
||||
There is **already a working VCV module** at `vcv/` (`MEMLNaut.cpp`, 959 lines; `plugin.json`; `SPEC.md`; `osc_server.hpp`; `Makefile`; `res/*.svg`) — but it is **2-in / 12-out + 5 derived**. The new requirement is **8-in / 16-out with LED rings**. This is an evolution of the existing module, reusing its threading model, OSC server, and serialization wholesale.
|
||||
|
||||
### 5.1 What changes vs. the existing module
|
||||
- `NUM_ML_INPUTS`: 2 → **8** (the existing `MAX_ML_INPUTS = 8` already anticipated this; `MEMLNaut.cpp:14`). All 8 are first-class jacks (not "reserved").
|
||||
- `NUM_ML_OUTPUTS`: 12 → **16**. The 5 derived outputs (MEAN/STD/DELTA/NOVELTY/CONFIDENCE) remain but become **optional context-menu extras** or move to the expander — the 16 raw outputs are the headline.
|
||||
- MLP shape: stays within "the modular N×M envelope" — `nisps::IML<float> iml{8, 16, {16, 24, 16}}` (the existing default hidden stack is fine; sized for real-time inference per `SPEC.md`). Inputs feed model input dims; the 16 outputs are the inference outputs.
|
||||
- **LED ring per output** replaces the single `SmallLight<WhiteLight>` next to each jack (`MEMLNaut.cpp:804`).
|
||||
|
||||
### 5.2 Panel layout (Wide, ~32–44 HP)
|
||||
```
|
||||
┌────────────────────────────────────────────┐
|
||||
│ MEMLNaut │ ← brand; "Powerful Synth" wording N/A (this is the CV mapper)
|
||||
│ ┌──────────────────────────────────────┐ │
|
||||
│ │ DISPLAY: 16 bars + XY dot + metrics │ │ ← NanoVG LedDisplay/drawLayer (existing :699)
|
||||
│ └──────────────────────────────────────┘ │
|
||||
│ SPREAD RATE LEARN● RAND CLEAR │ ← knobs + buttons (existing controls)
|
||||
│ [+] [−] (verdict buttons) │
|
||||
│ │
|
||||
│ INPUTS (8 jacks, 2 rows × 4) │
|
||||
│ IN1 IN2 IN3 IN4 │
|
||||
│ IN5 IN6 IN7 IN8 + LEARN_GATE +TRIG −TRIG│
|
||||
│ │
|
||||
│ OUTPUTS (16, 4 rows × 4), each: │
|
||||
│ ◉jack with an LED RING around the jack │
|
||||
│ [(◯1)(◯2)(◯3)(◯4)] │
|
||||
│ [(◯5)(◯6)(◯7)(◯8)] │
|
||||
│ [(◯9)(◯10)(◯11)(◯12)] │
|
||||
│ [(◯13)(◯14)(◯15)(◯16)] │
|
||||
│ (optional) MEAN STD DELTA NOVELTY CONF │
|
||||
└────────────────────────────────────────────┘
|
||||
```
|
||||
Each output is a `PJ301MPort` jack with a **`LedRingWidget`** drawn concentric around it (no separate attenuverter trimpot in the default skin — attenuverter moves to right-click/expander to make room for the ring; keep `PARAM_ATTEN_*` in the model for range scaling).
|
||||
|
||||
### 5.3 The LED-ring widget + palette mapping
|
||||
A custom widget that draws a ring whose **arc fill is proportional to the output value** and whose **colour comes from the frontend design tokens**. Per the Rack manual, self-illuminating custom widgets override `drawLayer(args, 1)` and draw on layer 1 (so they stay bright when room brightness is lowered) ([VCV custom lights](https://community.vcvrack.com/t/how-to-use-custom-lights/1941), [Migrate2](https://vcvrack.com/manual/Migrate2)).
|
||||
|
||||
```cpp
|
||||
// vcv/src/LedRing.hpp (new)
|
||||
struct LedRingWidget : Widget {
|
||||
MEMLNaut* module = nullptr;
|
||||
int outIdx = 0;
|
||||
NVGcolor ringColor = nvgRGB(0xff, 0x6a, 0x00); // default --accent
|
||||
float radius = 7.f; // mm-ish, around a PJ301M jack
|
||||
void drawLayer(const DrawArgs& args, int layer) override {
|
||||
if (layer != 1 || !module) return;
|
||||
float v = clamp(module->slewOutputs[outIdx], 0.f, 1.f); // 0..1
|
||||
Vec c = box.size.div(2);
|
||||
// track (dim full ring)
|
||||
nvgBeginPath(args.vg);
|
||||
nvgArc(args.vg, c.x, c.y, radius, -M_PI/2, -M_PI/2 + 2*M_PI, NVG_CW);
|
||||
nvgStrokeColor(args.vg, nvgRGBA(ringColor.r*255, ringColor.g*255, ringColor.b*255, 40));
|
||||
nvgStrokeWidth(args.vg, 1.4f); nvgStroke(args.vg);
|
||||
// value arc (proportional)
|
||||
nvgBeginPath(args.vg);
|
||||
nvgArc(args.vg, c.x, c.y, radius, -M_PI/2, -M_PI/2 + v*2*M_PI, NVG_CW);
|
||||
nvgStrokeColor(args.vg, ringColor);
|
||||
nvgStrokeWidth(args.vg, 1.8f); nvgStroke(args.vg);
|
||||
// glow halo (matches the frontend "glow not shadow" signature)
|
||||
nvgGlobalCompositeOperation(args.vg, NVG_LIGHTER); /* … bloom pass … */
|
||||
}
|
||||
};
|
||||
```
|
||||
|
||||
**Palette mapping — derived from `docs/redesign/manifold-export/tokens/colors.css` (read).** Ring colours come from the design tokens so VCV matches the frontend. A `kRingPalette[16]` table assigns each output a colour by its **parameter group**, cycling through the token accents and group/pin colours:
|
||||
|
||||
| Source token (colors.css) | Hex | Used for |
|
||||
|---|---|---|
|
||||
| `--accent` | `#ff6a00` | primary outputs / group 0 (orange — the live colour) |
|
||||
| `--accent-2` | `#00ccff` | data outputs / group 1 (cyan) |
|
||||
| `--accent-3` | `#ffa860` | group 2 (warm tint) |
|
||||
| `--pin-3` base `#b464ff` | `#b464ff` | group 3 (violet) |
|
||||
| `--good` | `#6bc26b` | group 4 (green) |
|
||||
| `--warn` | `#f5c45e` | group 5 (amber) |
|
||||
| `--info` | `#5b9eef` | group 6 (blue) |
|
||||
| `--danger` | `#ff4466` | bipolar / perturbed outputs (red) |
|
||||
|
||||
Mapping rule: `ringColor = kRingPalette[paramMeta[i].group % paletteLen]`, so outputs in the same mode-group glow the same colour, identical to the heatmap/Console grouping. Bipolar outputs (per-output range menu) tint toward `--danger`. The palette is a single header (`vcv/src/palette.hpp`) generated from `colors.css` so a token change propagates to both frontend and module (a small codegen step; **open choice** whether to automate or hand-sync).
|
||||
|
||||
### 5.4 Reused, unchanged from the existing module
|
||||
- **Threading model** (`MEMLNaut.cpp:67–312`): audio-thread `iml` + worker-thread `imlShadow`, job queue (Train/Perturb/Randomize/Clear), atomic weight hand-off, single-writer invariant. Already correct; just resize the I/O.
|
||||
- **Inference-rate decimation + slew + post-swap crossfade** (`:447–499`).
|
||||
- **Verdict loop** (`:412–445`): `+`/`−` buttons and `+TRIG`/`−TRIG` gated by LEARN. `+` → add example (current inputs→current outputs) + enqueue Train + decay noise ×0.97; `−` → bump noise (cap `0.3(1−s)+0.05s`) + enqueue Perturb. Identical semantics to the browser verdict loop.
|
||||
- **Serialization** (`dataToJson`/`dataFromJson`, `:548–696`) + **`.nisps` preset save/load** (`:862–924`) + **OSC server** (`osc_server.hpp`). The `.nisps` format is the shared interchange with the browser (`vcv/NISPS-FORMAT.md`).
|
||||
- Per-output / per-input **range menu** (uni/bipolar, attenuverter) (`:818–843`).
|
||||
|
||||
### 5.5 Plugin scaffold files
|
||||
```
|
||||
vcv/
|
||||
├── plugin.json # slug MEMLNaut (exists) — bump version; tags Controller/Utility/Random
|
||||
├── Makefile # VCV SDK Makefile (exists); RACK_DIR-driven
|
||||
├── Makefile.dist # dist packaging (exists)
|
||||
├── src/
|
||||
│ ├── plugin.{hpp,cpp} # plugin init / model registration (exist)
|
||||
│ ├── MEMLNaut.cpp # module — resize 8 in / 16 out (edit)
|
||||
│ ├── LedRing.hpp # NEW — LED-ring widget (§5.3)
|
||||
│ ├── palette.hpp # NEW — ring colour table from colors.css (§5.3)
|
||||
│ └── osc_server.hpp # OSC server (exists; extend with /nisps/feedback §7)
|
||||
├── res/
|
||||
│ ├── MEMLNaut.svg / -wide.svg / -expander.svg # panels (exist; redraw for 8/16 layout)
|
||||
├── test/smoke_test.cpp # exists
|
||||
├── plugin.json, SPEC.md, BUILDING.md, README.md, NISPS-FORMAT.md (exist)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. Browser↔VCV bridge design
|
||||
|
||||
The module must be **controllable AND trainable from BOTH inside Rack AND entirely from the browser**. Both ends operate on the same model; the bridge keeps them coherent.
|
||||
|
||||
### 6.1 Transport — propose options (operator open choice)
|
||||
The existing salvage path is **WebSocket↔UDP-OSC** (`osc-bridge/bridge.ts` + `osc-client.js` + `osc_server.hpp`). This is the recommended default and already works. Three options to flag:
|
||||
|
||||
| Option | How | Pros | Cons |
|
||||
|---|---|---|---|
|
||||
| **A. WS↔OSC bridge server (recommended)** | Browser ⇄ `bridge.ts` (Deno WS server, localhost:8765) ⇄ UDP OSC ⇄ module's `osc_server.hpp` (port 9000/9001) | Already built + bidirectional; standard OSC; works with SuperCollider/Max too; no browser perms | Needs a helper process running locally (Deno or the compiled `bridge.mjs`) |
|
||||
| **B. Direct WebMIDI** | Browser ⇄ Web MIDI ⇄ a virtual MIDI port ⇄ a tiny MIDI-in path in the module | No helper process if a virtual MIDI port exists; browser-native | 7-bit/14-bit only — too coarse for weights/state; really only for live CC; module would need MIDI parsing |
|
||||
| **C. Native (module hosts a WS server)** | The VCV module itself opens a WebSocket/HTTP server; browser connects directly | No external bridge process | Adds a WS/TLS stack inside the plugin; COOP/COEP + mixed-content (`https://` page → `ws://localhost`) friction; more attack surface in the audio plugin |
|
||||
|
||||
**Recommendation:** ship **A** (it exists, it's bidirectional, it already targets this very module — see `osc-client.js:1` "Connects the webapp to VCV Rack MEMLNaut module"). Keep **B** as a live-performance CC convenience only. Treat **C** as a future "no-helper" nicety. **This transport choice is an explicit operator open choice.**
|
||||
|
||||
### 6.2 Two modes
|
||||
- **Standalone:** module runs entirely inside Rack (CV in → model → CV out; verdict via panel buttons/triggers). No browser. Works today.
|
||||
- **Bridged:** browser connects via the bridge. In bridged mode **the browser supplies inputs in real time** — the Manifold pointer / joystick / pads stream `/nisps/input <f…f>` to the module, which uses them instead of (or blended with, configurable) the physical CV-in jacks. The module streams `/nisps/output` and `/nisps/input` back at ~100 ms (`MEMLNaut.cpp:536`) for browser visualisation. State/weights sync both ways via `/nisps/state` + `/nisps/weights` (already staged atomically into the audio thread, `:360–384`).
|
||||
|
||||
### 6.3 Coherence model
|
||||
One model, one owner of weights at a time. The bridge sends **whole-model snapshots** (`/nisps/state`, `/nisps/weights`) on any structural change (train completes, randomize, clear, load preset), and **continuous I/O vectors** (`/nisps/input`, `/nisps/output`) for live feel. Last-writer-wins on weights with a short "training in progress" lock (the module already coalesces jobs at queue depth 1). The browser's `WasmIML` and the module's `nisps::IML` use the **same `.nisps` weight layout** (`vcv/NISPS-FORMAT.md`), so a snapshot from either side loads losslessly — *provided the architectures match*. **Caveat / open choice:** the browser engine is fixed `MLP<2,…,126>` while the module is `8→{16,24,16}→16`. For true weight transfer the bridged session must run a **matched architecture** (e.g. a browser mode configured to 8-in/16-out, within the modular envelope) — otherwise the bridge degrades to I/O + example transfer only (no raw-weight sync). Flag this explicitly.
|
||||
|
||||
---
|
||||
|
||||
## 7. Training over the bridge — both directions
|
||||
|
||||
The verdict loop (place examples / thumbs-up / thumbs-down / undo) must work from **either** end and stay coherent. Add one OSC address and a small `RemoteTrainingBridge`:
|
||||
|
||||
```ts
|
||||
// manifold/src/engine/backends/backend.ts
|
||||
export interface RemoteTrainingBridge {
|
||||
thumbsUp(): void; // add current (input,output) example + train
|
||||
thumbsDown(): void; // perturb weights (explore)
|
||||
addExample(input: Float32Array, output: Float32Array): void;
|
||||
randomise(spread: number): void;
|
||||
clear(): void;
|
||||
undo(): void;
|
||||
onState(cb: (s: NispsState) => void): () => void; // remote → local sync
|
||||
}
|
||||
```
|
||||
|
||||
New OSC verb: `/nisps/feedback <s>` carrying `{"op":"up|down|rand|clear|undo","spread":f,"input":[…],"output":[…]}`. The module's `osc_server.hpp` gets an `onFeedback` callback (mirroring the existing `onState`/`onWeights` at `MEMLNaut.cpp:120–132`) that stages the op atomically for the audio thread, which routes it through the **same** `enqueueJob`/`add_example` path the panel buttons use (`:412–445`). So:
|
||||
|
||||
- **Browser → VCV training:** user clicks thumbs-up in the Manifold → `VcvBridgeBackend.remote.thumbsUp()` → `/nisps/feedback {op:up,input,output}` → module stages → worker trains `imlShadow` → atomic weight swap → module streams `/nisps/state` back → browser `WasmIML.setWeights` updates so the UI/heatmap reflect the new mapping. The browser need not run its own training in bridged mode (or runs it and pushes weights; configurable — see §6.3 caveat).
|
||||
- **VCV → browser training:** user presses `+`/`−` on the panel (or sends `+TRIG`) → module trains/perturbs → streams `/nisps/state` → browser applies, so the verdict placed in Rack appears in the browser's example list and weight-health view.
|
||||
- **Example placing over the bridge:** either side can `addExample`; examples ride in `/nisps/state` (the module already serializes `examples.features`/`examples.labels`, `:587–604`) so the dataset stays in sync. Undo is local-history on each side, but a remote undo can be sent as `/nisps/feedback {op:undo}` to roll the module's last job (module keeps a one-deep snapshot, matching the browser undo stack semantics).
|
||||
|
||||
**Result:** the tool is fully controllable and trainable from inside VCV Rack and entirely from the browser, with the verdict loop and example-placing working either way over the same bridge.
|
||||
|
||||
---
|
||||
|
||||
## 8. Build / install notes
|
||||
|
||||
### Browser side
|
||||
Backends live under `manifold/src/engine/backends/`; lifted JS (`c15-*`, `param-map`, `presets`, `midi-output`, `midi-cc-map`, `osc-client`, `visualizer`) ported to TS, parity-checked. No new build step — they ride the existing Vite `manifold` build. COOP/COEP stays server-scoped (needed by the Powerful Synth's SAB path).
|
||||
|
||||
### OSC bridge server
|
||||
```bash
|
||||
cd osc-bridge
|
||||
deno run --allow-net bridge.ts # or the compiled bridge.mjs
|
||||
# --osc-host 127.0.0.1 --osc-port 9000 --ws-port 8765 --listen-port 9001
|
||||
```
|
||||
Ship `bridge.mjs` (already compiled) + `compile.sh` so users without Deno can run it via Node. Surface "bridge not running" in the OSC/VCV backend status (the client auto-reconnects).
|
||||
|
||||
### VCV module
|
||||
```bash
|
||||
cd vcv
|
||||
export RACK_DIR=/path/to/Rack-SDK # VCV Rack 2 SDK
|
||||
make # builds plugin.so/.dylib/.dll
|
||||
make install # copies into the VCV user plugins dir
|
||||
# distribution: make dist (per Makefile.dist; produces the .vcvplugin)
|
||||
```
|
||||
Per the search, the standard flow is `export RACK_DIR=…; make clean; make dist` ([Plugin Development Tutorial](https://vcvrack.com/manual/PluginDevelopmentTutorial)). Requires the VCV Rack 2 SDK; nisps-core is header-only C++20 (symlinked under `vcv/dep/` per `SPEC.md`). Ship v2-only (rationale in `SPEC.md`'s v1-compat section). License caveat: VCV SDK is GPLv3, nisps-core is MPL-2.0 — combined binary is effectively GPL; not submitting to the VCV Library initially (`SPEC.md` §License).
|
||||
|
||||
---
|
||||
|
||||
## 9. Open choices for the operator
|
||||
|
||||
1. **Bridge transport (§6.1):** confirm **A — WS↔OSC bridge server** as default (recommended; already built and bidirectional), with WebMIDI as a live-CC-only convenience and a native in-module WS server deferred. This is the biggest call.
|
||||
2. **Bridged weight-sync vs I/O-only (§6.3):** the browser engine is fixed `MLP<2,…,126>`; the module is `8→16`. Either (a) run a **matched 8-in/16-out browser mode** for true raw-weight transfer, or (b) accept that bridged sessions sync **I/O + examples only** and each side trains its own weights. Recommendation: (b) for v1, (a) when the modular N×M browser MLP lands (workstream F).
|
||||
3. **CV/gate native path (§2.5):** ship only the **VCV-bridge CV alias** for v1, or also build browser-native DC-coupled WebAudio CV? Recommendation: VCV-bridge first; defer DC CV.
|
||||
4. **Particle noise determinism (§4.1):** keep the `Math.random()`-seeded permutation (non-deterministic per load, faithful to today) or add a `?seed=` for reproducible visuals/tests? Recommendation: keep default behaviour, add an opt-in test seed.
|
||||
5. **Ring palette sync (§5.3):** auto-generate `vcv/src/palette.hpp` from `colors.css` via a codegen step, or hand-sync? Recommendation: small codegen so a token change updates both surfaces.
|
||||
6. **Derived outputs on the 16-out module (§5.1):** keep MEAN/STD/DELTA/NOVELTY/CONFIDENCE as menu-toggled extras / move to the expander, or drop them? Recommendation: move to the expander; keep the 16 raw outputs + LED rings as the headline panel.
|
||||
7. **Default active backend:** confirm **WebAudioBackend (firmware-parity engine)** is the default, with the Powerful Synth (C15 path), particles, MIDI, OSC, CV, VCV selectable in the dock.
|
||||
|
||||
---
|
||||
|
||||
## 10. Cited source files (absolute paths)
|
||||
- Particle algorithm (faithful port): `/home/w1n5t0n/deployments/meml-aimmersive/js/ui/visualizer.js` (`:5–288`)
|
||||
- Built-in synth: `/home/w1n5t0n/deployments/meml-aimmersive/js/synth/c15-adapter.js`, `…/c15-bridge.js`, `…/param-map.js` (`:287` curve math), `…/presets.js`
|
||||
- MIDI: `/home/w1n5t0n/deployments/meml-aimmersive/js/midi/midi-output.js` (`:114` batch throttle), `…/midi/midi-cc-map.js`
|
||||
- OSC client: `/home/w1n5t0n/deployments/meml-aimmersive/js/nisps/osc-client.js`; param-named client `…/js/synth/osc-output.js` (`:97`)
|
||||
- OSC bridge server: `/home/w1n5t0n/deployments/meml-aimmersive/osc-bridge/bridge.ts` (addresses `:298–305`, encode/decode `:70–182`), `bridge.mjs`, `compile.sh`
|
||||
- VCV module: `/home/w1n5t0n/src/MEMLNaut-NISPS/vcv/src/MEMLNaut.cpp` (threading `:67–312`, verdict `:412–445`, ranges `:818–843`, display `:699`, LED `:804`), `…/vcv/src/osc_server.hpp`, `…/vcv/plugin.json`, `…/vcv/SPEC.md`, `…/vcv/Makefile`, `…/vcv/NISPS-FORMAT.md`, `…/vcv/res/*.svg`
|
||||
- Design tokens (ring palette): `/home/w1n5t0n/src/MEMLNaut-NISPS/docs/redesign/manifold-export/tokens/colors.css`
|
||||
- Spine/engine context: `/home/w1n5t0n/src/MEMLNaut-NISPS/docs/redesign/engine-architecture.md` (§2), `…/findings-design-and-manifold.md` (§4), `…/findings-engine-surface.md`, `…/aimmersive-clone-spec.md` (routeOutputs §6, §7 visual table, §10)
|
||||
|
||||
Sources (VCV SDK / widgets): [VCV custom lights](https://community.vcvrack.com/t/how-to-use-custom-lights/1941), [Migrate2 (drawLayer/layer 1)](https://vcvrack.com/manual/Migrate2), [Plugin Development Tutorial (RACK_DIR/make dist)](https://vcvrack.com/manual/PluginDevelopmentTutorial), [Plugin API Guide](https://vcvrack.com/manual/PluginGuide).
|
||||
---
|
||||
|
||||
## Verification corrections (adversarial pass, 2026-06-27) — verdict: minor-issues
|
||||
|
||||
Strongly grounded: every cited file exists, the particle algorithm (§4) is reproduced faithfully, all 8 token
|
||||
hex values verify against `colors.css`, the VCV cites + OSC bridge protocol + MIDI/OSC salvage files + C15
|
||||
adapter chain are accurate. Apply these fixes:
|
||||
|
||||
- **`applyCurve` must keep the input clamp.** Real code (`param-map.js:287-291`) clamps to [0,1] BEFORE the
|
||||
pow: `Math.pow(clamp01(value), exponent)`. Drop the "bit-identical" wording or add the clamp to the TS port.
|
||||
- **`ParamMeta.id` vs `name` conflation** — the example `id: 'Env_A_Att'` is actually the `name` field; keep
|
||||
`id` (slug) and `name` (param key) distinct when porting `param-map.js`.
|
||||
- VCV module already at `vcv/` (2-in/12-out) — evolve to 8→16, do not rebuild; reuse its threading/OSC/serialisation.
|
||||
|
||||
---
|
||||
|
||||
## Implementation status (2026-06-28)
|
||||
|
||||
MIDI + OSC backends are BUILT in `manifold/src/backends/` (OutputBackend interface + BackendManager consuming the
|
||||
spine; WebMIDI out with per-output CC#/ch/name/range; OSC-over-WS to the Deno bridge in `manifold/osc-bridge/`).
|
||||
The Outputs dock panel specialises per backend (`manifold/src/dock/OutputsBackendConfig.tsx`) with named presets
|
||||
(`manifold/src/backends/presets.ts`). Audio gated via `engine.audio.setMuted` on non-synth modes.
|
||||
|
||||
**VCV module** — see `vcv/SPEC.md` "⚠️ BUILD DELTAS (2026-06-28)" for the authoritative build target: 8 inputs ×
|
||||
16 outputs, an LED ring per output (drawLayer + nvgArc), palette from the frontend tokens, WS↔OSC bridge
|
||||
(browser OSC backend → `manifold/osc-bridge/` Deno relay → the module's OSC server), bidirectional training, and
|
||||
the `nisps-core`→`nisps/` core-path repoint. The existing `vcv/` module (2→12) is evolved, not rebuilt.
|
||||
|
||||
**Still TODO** (clear in-code): the VCV↔browser bridge browser-side wiring to the new `/nisps/feedback` verb;
|
||||
the particle backend is a no-op passthrough pending the `visualizer.js` faithful port; CV/gate transport.
|
||||
363
docs/redesign/dock-spec.md
Normal file
363
docs/redesign/dock-spec.md
Normal file
|
|
@ -0,0 +1,363 @@
|
|||
# Dock Spec — Console Right-Dock Drawers + Per-Output Controls
|
||||
|
||||
*Workstream D. Read-only design, 2026-06-27. Target app: `manifold/` (Vite + React + TS), wired to the parity-tested TS engine (`playground/src`) via the headless `EngineApi` boundary (`docs/redesign/engine-architecture.md` §2). Citations are `file:line`. British spelling in product copy. The built-in synth is the **"Powerful Synth Engine"** — the string "C15" MUST NEVER appear in the UI.*
|
||||
|
||||
This spec replaces the placeholder Shape/Feel/Route/Health/Help drawers in `ConsoleApp.jsx` (`docs/redesign/findings-design-and-manifold.md` §2 — keyboard `1-5` map drawers, `\` toggles depth) with the real dock contents.
|
||||
|
||||
---
|
||||
|
||||
## 0. The 3-depth dock model (frame the design)
|
||||
|
||||
The Console's right dock is a 48px rail (`--dock-width`, `aimmersive-clone-spec.md:181`) of icons; each icon owns one drawer with **three depths**:
|
||||
|
||||
| Depth | Width / placement | Purpose | Source vocabulary |
|
||||
|---|---|---|---|
|
||||
| **peek** | narrow flyout (~72px) hugging the rail | glanceable status + 1–2 primary toggles; no scrolling | `playground-2026.md` dock+drawer |
|
||||
| **expand** | glass panel ~300px (cf. a-immersive drawer-stack `width:260px`, `aimmersive-clone-spec.md:312`) | the working surface: the per-output rows, the live sliders, the mode toggles | a-immersive drawers |
|
||||
| **FULL** | near-fullscreen **modal** (`--z-modal`, glass over dimmed canvas) | the "advanced config page" for one entry — backend CC tables, full param matrix, weight-health lab | mission "advanced" page |
|
||||
|
||||
- **Depth is per-drawer state** in `uiStore` (`peek | expand | full`), independent across drawers (a-immersive allows multiple open at once, `aimmersive-clone-spec.md:84`). FULL is mutually exclusive — only one modal at a time (it captures the screen).
|
||||
- `\` toggles the *focused* drawer peek↔expand; a drawer's header **⤢ button** (or double-click a row) opens FULL.
|
||||
- Esc closes FULL→expand; closing FULL restores prior depth.
|
||||
- Drawers slide in via `drawerSlideIn` 0.2s translateX (`aimmersive-clone-spec.md:189`). FULL fades via `helpFadeIn`.
|
||||
- Dock icons get `.active` tint (`--accent #ff6a00`) when their drawer is open; macOS magnify on hover (`scale(1.35)`, `aimmersive-clone-spec.md:188`).
|
||||
|
||||
**Drawer roster** (six icons, top→bottom on the rail):
|
||||
|
||||
| Icon | Drawer | One-line role |
|
||||
|---|---|---|
|
||||
| 🧠 LEARN | **Learning-Behaviour** | Feedback-mode selector, SOLO/arm chooser, live training params |
|
||||
| 🎚 IN | **Inputs** | Input source + per-axis pipeline (workstream F territory — referenced, §6) |
|
||||
| 🔀 OUT | **Outputs / Routing** | The per-output control matrix + backend selector (workstream E + §3, §4) |
|
||||
| 🔊 SYNTH | **Powerful Synth Engine** | Engine switch + audio/arp controls + group overrides |
|
||||
| ✦ VISUAL | **Particle / Visual System** | Flow-field visualiser params + presets |
|
||||
| ? HELP | **Help** | Onboarding overlay (opens modal directly, not a drawer) |
|
||||
|
||||
---
|
||||
|
||||
## 1. LEARNING-BEHAVIOUR drawer
|
||||
|
||||
Owns *how the model learns from your gestures*: the feedback-mode selector, the SOLO/arm variant chooser, and the live training knobs. Binds to `EngineApi.feedback` (`engine-architecture.md` §2.2) and the feedback C API (`findings-engine-surface.md:52`, `nisps_ml_feedback_*`).
|
||||
|
||||
### 1.1 FEEDBACK_MODE selector ("Down Action")
|
||||
|
||||
The `+` (up) verdict is always "keep this" (`addExample` + train, `findings-feedback-behaviour.md:102`); the `−` (down) verdict is **selectable** among the ported feedback modes (`feedback-modes-port-spec.md` §1; `FeedbackController<MLP_T>`). The new 2-mode product surface (per the prompt's "Mode 1 / Mode 2"):
|
||||
|
||||
| Selector label (UI copy) | Engine `FeedbackMode` | Behaviour |
|
||||
|---|---|---|
|
||||
| **Push away** (Mode 1) | `Avoid` (`feedback.hpp` enum, `findings-feedback-behaviour.md:90`) | down → geometric-dislike: perturbs the mapping away from what you disliked. In the deployed core this routes to `move_weights(speed, spread, pinMask)` (`feedback-modes-port-spec.md` §2.5 — the true k-NN centroid push is firmware-only, out of scope). |
|
||||
| **Explore & place** (Mode 2) | `RandomiseMlp` (`findings-feedback-behaviour.md:122`) | down → snapshot + `draw_weights(spread)` re-rolls the whole net into a scratchpad you audition by moving the joystick; `+`/drag commits a `+1` example at the chosen input and **restores the real net** (`findings-feedback-behaviour.md:135-146`); down-again **cancels** (restore snapshot). |
|
||||
|
||||
- A third engine mode `RandomiseOutputs` (bypass MLP, hold a static random vector, `findings-feedback-behaviour.md:111`) exists in the core but is **not** surfaced as a product mode in v1 — expose it only behind `?debug=1` as "Static roll". **(Open choice 1.)**
|
||||
- **Peek**: a 2-segment pill (`Push away` / `Explore & place`), the current mode highlighted in `--accent`. This pill is *also* mirrored next to the Verdict cluster (`VerdictCluster`, `findings-design-and-manifold.md:50`) so it is reachable during live play without opening the drawer — matching how a-immersive puts the `rl-label` above the RL buttons (`aimmersive-clone-spec.md:283`).
|
||||
- **Expand**: the pill + a one-line plain-English description of the active mode + an **"exploring…" indicator** that lights when `engine.feedback.exploring()` is true (`findings-feedback-behaviour.md:237`), reusing the `<NoiseRing>` colour ramp (off/active/high, `aimmersive-clone-spec.md:98`). While exploring in Mode 2, training is paused (`learning_paused()`, `findings-feedback-behaviour.md:48`) — show a small "learning paused" badge.
|
||||
- **FULL**: the feedback lab — a diagram of the active state machine (idle → exploring → commit/cancel), the raw `FeedbackAction` log of the last presses (`findings-feedback-behaviour.md:90` enum), plus the spread/tame "Health lab" sliders (§1.3) for radical exploration tuning.
|
||||
- **Bind point**: `engine.feedback.setMode(mode)` → `nisps_ml_feedback_set_mode` (`feedback-modes-port-spec.md` §4). Switching mode while exploring auto-aborts and restores the net (`set_mode` calls `abort_explore`, `findings-feedback-behaviour.md:96`). Persist `feedbackMode` in the session blob.
|
||||
|
||||
### 1.2 SOLO / arm variant chooser
|
||||
|
||||
"Solo" and "arm" are the SAME concept (the prompt: *solo(=arm)*): **focus training on one output**. It maps to the engine's **focus mask** (`activeDims_` → `nisps_ml_feedback_set_focus`, `findings-feedback-behaviour.md:159-167`) AND to the **output-pin mask** used by `move_weights`/`buildPinMask` (`findings-feedback-behaviour.md:117`, `mode-runtime.ts:548`).
|
||||
|
||||
Two armable scopes (the chooser):
|
||||
|
||||
| Variant | Effect | Engine wiring |
|
||||
|---|---|---|
|
||||
| **Arm output** (per-row solo) | only the armed output(s) learn/re-roll; all others are frozen in the model | `set_focus(mask)` where `mask[i]=1` for armed dims; unmuted-but-unarmed dims set `0` so `move_weights`/`roll_static_outputs` leave them put (`findings-feedback-behaviour.md:71,103-107`) |
|
||||
| **Arm all** (default) | every live output learns (empty mask ⇒ all active, `findings-feedback-behaviour.md:161`) | `clear_focus_mask()` |
|
||||
|
||||
- Arming is a per-output toggle (the **S** button on each control row, §3) — arming any output flips the drawer into "arm-output" scope automatically; clicking the drawer's **"Arm all"** chip clears it.
|
||||
- Multiple outputs may be armed at once (mask is a vector). The armed set is highlighted on the ReadoutStrip and on the heatmap (border glow, `--glow-focus`).
|
||||
- **Peek**: shows "Arm: all" or "Arm: 3 outputs"; tapping cycles to "Arm all".
|
||||
- **Expand**: the scope chips + a compact list of currently-armed output names with quick-unarm.
|
||||
- **Persisted** as part of routing state (the focus mask is reconstructable from per-row `armed` flags).
|
||||
|
||||
### 1.3 Live training params
|
||||
|
||||
Six live knobs, lifted from a-immersive's "NISPS" drawer (`aimmersive-clone-spec.md:67` — the deployed app's *actual* tuning surface, six raw sliders) plus the spread/tame regime controls:
|
||||
|
||||
| Slider | Range | Default | Binds to |
|
||||
|---|---|---|---|
|
||||
| **Noise** | 0–noiseCap | 0.05 | `mlStore.noiseLevel`; the `−` perturbation magnitude (`findings-feedback-behaviour.md:110`) |
|
||||
| **Learning rate** | 1e-6 – 1e-2 (log) | 1e-5 | `iml.learningRate` (`aimmersive-clone-spec.md:226`) |
|
||||
| **Decay** | 0.8–1.0 | 0.97 | `rlExplorationDecay` (`findings-feedback-behaviour.md:103`) |
|
||||
| **Spread** | 0–1 | 0.6 | `spreadLevel` — master noise/Xavier regime (`findings-feedback-behaviour.md:108`; CLAUDE.md `spread`) |
|
||||
| **Tame** | 0–1 | 1 | output-range safety toward `[safeMin,safeMax]` (`param-map.js:261` `applyTame`) |
|
||||
| **Max iters / Convergence** | — | — | `iml.maxIterations`/`convergenceThreshold` (`aimmersive-clone-spec.md:274`) — FULL-depth only |
|
||||
|
||||
- **Peek**: Noise + Spread (the two most live).
|
||||
- **Expand**: Noise, Spread, Tame, Learning rate, Decay as `<Slider>`s.
|
||||
- **FULL**: all six + Max iters/Convergence + a real `<LossPlot>` (needs `nisps_ml_loss_history`, `findings-design-and-manifold.md:62`) + `<WeightHealth>` edge-glow + `<LayerStats>` + `<GradientFlow>` (the diagnostics suite, `findings-design-and-manifold.md:63`). This is the "advanced learning" page.
|
||||
- All writes are eager engine setters (no React-render coupling, `engine-architecture.md` §4); persisted in the session blob.
|
||||
|
||||
---
|
||||
|
||||
## 2. INPUTS drawer (workstream F territory — referenced)
|
||||
|
||||
**Owner: workstream F (modular N×M inputs + input pipeline).** This drawer is specified there; here we only fix its *dock shape* so the dock model is complete and the two specs dovetail.
|
||||
|
||||
- **Source**: input source selector — **Joystick / Hands / MIDI / Mic** (a-immersive Input pill, `aimmersive-clone-spec.md:150`). v1 browser is **fixed-2-input** (`findings-engine-surface.md:67`; `MLP<2,…>`) — modes needing >2 inputs show a "single-input in browser" badge (`engine-architecture.md` §6 open Q2).
|
||||
- **Per-axis pipeline** (deadzone→zoom→curve→smoothing→momentum, `input/pipeline.ts`, `findings-design-and-manifold.md:84`): one `<ControlAxis>` row per input axis with its pipeline knobs.
|
||||
- **Depth**: peek = source + zoom; expand = per-axis pipeline rows; FULL = the N×M input-routing matrix (workstream F).
|
||||
- **Bind**: `engine.setInput(x,y)` is the only input door (`engine-architecture.md` §2.2); pipeline config lives in `input-store`.
|
||||
- **Per-axis controls reuse the per-output row component** (§3) where sensible (an input axis has min/max/curve too), but mute/solo/freeze semantics differ — defer those to workstream F. **(Open choice 2: confirm input-axis rows share the §3 component or get their own.)**
|
||||
|
||||
---
|
||||
|
||||
## 3. OUTPUTS / ROUTING drawer + the per-output control row
|
||||
|
||||
The heart of this spec. **Owner of the routing matrix + per-output baseline is shared; backend-specific advanced layouts are workstream E (VCV/OSC/MIDI LED-ring backends, `findings-design-and-manifold.md:30`).**
|
||||
|
||||
### 3.1 The per-output baseline (EVERY output, EVERY backend)
|
||||
|
||||
Every output — synth param, MIDI CC, OSC path, VCV channel, visual param — shares one baseline control set. This faithfully ports the deployed vanilla-JS per-param override system (`aimmersive-clone-spec.md` §2.4 heatmap popup + §2.5 group drawer; `param-map.js` `applyGroupOverride`/`applyCurve`/`applyTame`). The baseline:
|
||||
|
||||
| Control | Glyph / widget | Semantics |
|
||||
|---|---|---|
|
||||
| **Mute** | **M** toggle | silenced *downstream* but still computed + visible (distinct from off) |
|
||||
| **Solo / Arm** | **S** toggle | focus training on this output (§1.2; focus mask + pin mask) |
|
||||
| **Tri-state** | **off / fixed / live** segmented | model-control state (table §3.3) |
|
||||
| **Min / Max** | `<DualRangeSlider>` (min blue `#4488ff`, max orange `#ff6a00`, `aimmersive-clone-spec.md:130`) | output range remap; `applyGroupOverride(v,curve,min,max)` (`param-map.js:301`) |
|
||||
| **Curve** | `<CurvePad>` 36×36 drag canvas, 0.5=linear (`aimmersive-clone-spec.md:128`; `applyCurve`, `param-map.js:287`) | per-output response curve |
|
||||
| **Value** | inline read-out / drag bar | live model value, or the held value when fixed |
|
||||
|
||||
### 3.2 The control-row component + state model
|
||||
|
||||
```ts
|
||||
// manifold/src/engine/routing/output-control.ts (engine-side, headless)
|
||||
type OutputState = 'off' | 'fixed' | 'live'; // the tri-state
|
||||
|
||||
interface OutputControl {
|
||||
index: number; // model output index (0..125 for synth)
|
||||
name: string; // schema label (NEVER "C15"; synth params from SYNTH_PARAM_MAP)
|
||||
group: string; // section (Env A, Osc B, …) for grouping
|
||||
state: OutputState; // off | fixed | live
|
||||
muted: boolean; // downstream silence; still computed
|
||||
armed: boolean; // solo / focus-training (=arm)
|
||||
min: number; // [0,1]
|
||||
max: number; // [0,1], min<=max enforced
|
||||
curve: number; // [0,1], 0.5 linear
|
||||
fixedValue: number; // held value when state==='fixed' (captured on freeze)
|
||||
// backend-specific, populated by the active backend adapter:
|
||||
backend?: MidiCcSpec | OscSpec | VcvSpec;
|
||||
}
|
||||
```
|
||||
|
||||
- Stored in `routing-store` as a `createStore` array indexed by output (`engine-architecture.md` §1; `overrideStore` shape, `aimmersive-clone-spec.md:230`). This unifies a-immersive's split `visualOverrides`/`groupOverrides`/`engineParamOverrides`/`midiCCOverrides` (`aimmersive-clone-spec.md:230-233`) behind one `getSectionView` adapter (`aimmersive-clone-spec.md:140`).
|
||||
- **Mute ↔ off are distinct fields** (the prompt insists): `muted` is a downstream gate; `state==='off'` excludes from model control. They compose (an output can be off AND muted).
|
||||
- **The deployed app conflates frozen↔muted via one field** (`aimmersive-clone-spec.md:347` — heatmap popup `frozen` ↔ group drawer `muted` map to the same underlying field). The new model **separates them deliberately**: `state` carries off/fixed/live, `muted` is its own boolean, `armed` its own boolean. This is the one deliberate divergence from the deployed semantics — record in `ALIGNMENT.md`. **(Open choice 3.)**
|
||||
- **Reactive binding**: the row's controls write the store eagerly; `routedOutput` memo (`engine-architecture.md` §2.1) reads `routing-store` to remap each output via `applyGroupOverride`. The single send-effect pushes to the active backend. A row never holds a second data path.
|
||||
|
||||
### 3.3 Tri-state semantics table (precise)
|
||||
|
||||
This matches the deployed per-param override system: `off`/`fixed` correspond to a-immersive's `frozen` (excluded/pinned, `aimmersive-clone-spec.md:131`), `live` to the default model-driven path.
|
||||
|
||||
| State | Computed by model? | Pin mask (`buildPinMask`) | Sent downstream? | Value emitted | UI |
|
||||
|---|---|---|---|---|---|
|
||||
| **off** | **No** — removed from model control; final-layer weights pinned so RL/train never touch it (`findings-feedback-behaviour.md:117`) | pinned (excluded) | yes, at last held value | `fixedValue` (held/excluded) | dimmed row, no bar motion |
|
||||
| **fixed** (freeze) | computed but result discarded; held at a static value | pinned (protect the held dim) | yes | `fixedValue` (captured on freeze, draggable, `aimmersive-clone-spec.md:131`) | hatched bar overlay (`aimmersive-clone-spec.md:190`), value slider shown |
|
||||
| **live** | **Yes** — model-driven | not pinned | yes | `applyGroupOverride(modelOut, curve, min, max)` | full bar, animates |
|
||||
|
||||
Orthogonal modifiers (compose with any state):
|
||||
|
||||
| Modifier | Computed? | Visible? | Sent downstream? | Training focus |
|
||||
|---|---|---|---|---|
|
||||
| **mute** | yes (still computed) | **yes** (bar visible) | **no** (silenced) | unaffected |
|
||||
| **off** (state) | no | yes (dimmed) | yes (held) | excluded |
|
||||
| **solo / arm** | yes | yes (focus glow) | yes | **this output only** (focus + pin mask) |
|
||||
|
||||
> Precise difference the prompt demands: **off** = the model no longer drives this output (excluded from learning, held/pinned). **fixed/freeze** = held at a static value (also pinned, but conceptually "I chose this value", with a draggable `fixedValue`). **mute** = silenced downstream but *still computed and visible* (distinct from off — you still see it move, you just don't hear it). **solo/arm** = focus training on this output.
|
||||
|
||||
### 3.4 Drawer depths
|
||||
|
||||
- **Peek**: backend badge (Synth / MIDI / OSC / VCV) + count of live/fixed/off/muted outputs + "Arm all" status.
|
||||
- **Expand**: the **routing matrix** — a scrollable `<For>` of `<OutputControlRow>`s grouped by `group` (collapsible section headers carry a **group master curve** + **mute-group** like a-immersive's group drawer, `aimmersive-clone-spec.md:137`). Each row: name · M · S · [off|fixed|live] · dual-range · curve pad · value. 126 rows for synth — virtualise / collapse non-live by default.
|
||||
- **FULL**: the **Advanced backend modal** (§4) — backend selector at the top, then the backend-specific editor over the full output set.
|
||||
- **Backend selector** (workstream E): **Web Audio (synth) / Web MIDI / OSC bridge / VCV** (`engine-architecture.md` §1 backends; `aimmersive-clone-spec.md:143` output-mode tabs). Switching backend may change output count → reuse a-immersive's `confirm()` weight-reset guard (`aimmersive-clone-spec.md:144,344`).
|
||||
|
||||
---
|
||||
|
||||
## 4. ADVANCED modal — backend-specific layouts (FULL depth)
|
||||
|
||||
All backends share the §3.1 baseline (M/S/off-fixed-live/min/max/curve). The FULL modal adds the backend-specific fields. **(Workstream E owns the VCV/OSC/MIDI backend internals; this fixes the modal layout + state contract.)**
|
||||
|
||||
### 4.1 MIDI backend
|
||||
|
||||
Per-output extra fields (ported from a-immersive's MIDI-CC popup rows, `aimmersive-clone-spec.md:128`):
|
||||
|
||||
```ts
|
||||
interface MidiCcSpec {
|
||||
cc: number; // 0..127, auto-named from CC_NAMES
|
||||
channel: number; // 1..16
|
||||
value: number; // last sent, round(v*127)
|
||||
// min/max/curve from baseline define the 0..127 range mapping
|
||||
}
|
||||
```
|
||||
|
||||
- **Top of modal**: **number of CCs** (the output count for the MIDI backend = `midiCCMap.length`, `aimmersive-clone-spec.md:144`), output-device `<select>`, MIDI preset `<select>` (`aimmersive-clone-spec.md:32`).
|
||||
- **Per-CC row**: editable Name (text), **CC#** (0–127 spinner, auto-renames from `CC_NAMES`), **Ch** (1–16), plus the baseline min/max/curve/state. 7-bit value = `round(applyGroupOverride(v,curve,min,max) * 127)` (`findings-engine-surface.md:36`). Persist via `nisps-midi-cc-map:<engineId>` key (`aimmersive-clone-spec.md:239`).
|
||||
- **Send**: batched per block via `WebMidiBackend.send` (`engine-architecture.md` §1; `midiOutput.sendBatch`, `aimmersive-clone-spec.md:258`).
|
||||
|
||||
### 4.2 OSC backend
|
||||
|
||||
```ts
|
||||
interface OscSpec {
|
||||
path: string; // e.g. "/synth/cutoff"
|
||||
min: number; // physical range lo (engineering units, not [0,1])
|
||||
max: number; // physical range hi
|
||||
// curve/state from baseline
|
||||
}
|
||||
```
|
||||
|
||||
- **Top of modal**: bridge connection status (WS, deferred contract — `engine-architecture.md` §1 `osc-bridge.ts`), add/remove path rows.
|
||||
- **Per-output row**: **OSC path** (text), **range min/max** (typed numeric, physical units — distinct from the [0,1] baseline dual-range; the baseline min/max selects the normalised window, the OSC range maps that window to engineering units), plus baseline curve/state.
|
||||
- v1: OSC bridge is **stubbed behind a locked contract** (`engine-architecture.md` §1) — the modal renders and persists config but emits only when a bridge is connected.
|
||||
|
||||
### 4.3 VCV backend
|
||||
|
||||
```ts
|
||||
interface VcvSpec {
|
||||
// per-output min/max/freeze ONLY — the simplest backend
|
||||
// baseline min/max ARE the VCV range; baseline state 'fixed' IS freeze
|
||||
}
|
||||
```
|
||||
|
||||
- VCV adds **nothing beyond the baseline** — per-output min/max + freeze (= baseline `state==='fixed'`). The LED-ring palette derives from theme tokens (`findings-design-and-manifold.md:30`).
|
||||
- **Top of modal**: per-channel min/max/freeze grid; that is the whole VCV advanced surface.
|
||||
|
||||
### 4.4 Web Audio (synth) — the default
|
||||
|
||||
The synth backend's advanced modal *is* the **group-override matrix** (§5 / the Powerful Synth Engine drawer's FULL depth): per-group master curve + per-param min/max/curve/mute over the 126 synth params (`aimmersive-clone-spec.md:137`). No extra per-output struct — the synth output *is* the baseline.
|
||||
|
||||
---
|
||||
|
||||
## 5. POWERFUL SYNTH ENGINE drawer
|
||||
|
||||
Owns the audio engine and its parameter shaping. **The string "C15" must never render** — the built-in engine is labelled **"Powerful Synth Engine"** (`findings-engine-surface.md:74`; CLAUDE.md). Engine display names come from `SynthEngine.displayName` (`aimmersive-clone-spec.md:278`).
|
||||
|
||||
- **Peek**: play/pause + master volume + the active engine name ("Powerful Synth Engine") + BPM.
|
||||
- **Expand**:
|
||||
- **Engine switcher** — Powerful Synth Engine (default) + other engines (PAFSynth, ChannelStrip, etc. from `mode_select`); alternative engines flagged where not-yet-wired (`findings-engine-surface.md:74`).
|
||||
- **Audio controls**: Vol, BPM (a-immersive play drawer, `aimmersive-clone-spec.md:31`).
|
||||
- **Arpeggiator** controls (`aimmersive-clone-spec.md:278`).
|
||||
- **Synth preset `<select>`** — tiered Manual/Beginner/Intermediate/Advanced/Expert (`aimmersive-clone-spec.md:199`); these set which params are active/muted + their min/max/curve (the synth-side override tier — *distinct* from the visual/RL example presets, keep separate, `aimmersive-clone-spec.md:157`).
|
||||
- **FULL**: the synth **group-override matrix** (§4.4) — 18 collapsible sections (Env A…Mono, `aimmersive-clone-spec.md:298`), each with a group master curve (48×48 drag, applies relative delta to all child curves, `aimmersive-clone-spec.md:138`) and per-param baseline rows (§3.2). This is `<GroupOverrideDrawer>` promoted to FULL depth.
|
||||
- **Bind**: `engine.audio.start/stop`, `SynthEngine.setParam(i, v)` throttled ≥50ms / dead-zone >0.002 (`aimmersive-clone-spec.md:257,345` — load-bearing, prevents ring-buffer flooding). Audio starts lazily on the play gesture (`aimmersive-clone-spec.md:278`).
|
||||
- COOP/COEP server-scoped (`findings-design-and-manifold.md:100`); SharedArrayBuffer only for the browser-only Powerful Synth Engine path (`findings-engine-surface.md:74`).
|
||||
|
||||
---
|
||||
|
||||
## 6. PARTICLE / VISUAL SYSTEM drawer
|
||||
|
||||
Owns the flow-field visualiser — a faithful port of `js/ui/visualizer.js` (`findings-design-and-manifold.md:120` — exact look + behaviour; workstream E). 400-particle Canvas2D flow field driven by 20 named outputs (`aimmersive-clone-spec.md:296`).
|
||||
|
||||
- **Peek**: visual on/off + the active visual preset name.
|
||||
- **Expand**: the 20 visual params as per-output rows (§3 baseline — Flow/Scale/Speed/Hue/Spread/… `aimmersive-clone-spec.md:296`), each with its `VISUAL_PARAM_COLORS` swatch; plus **visual/RL preset chips** (Calm/Chaos, Rainbow, Vortex, Spiral, Embers, `aimmersive-clone-spec.md:158`) which teach the *network* by clearing the dataset and adding hardcoded input→output examples (distinct from synth tiers — keep separate).
|
||||
- **FULL**: the full visual param matrix + the output→visual-param range table (verbatim ranges, `aimmersive-clone-spec.md:296`) as editable advanced mappings + particle-count / lifetime tuning.
|
||||
- **Bind**: when the visual backend is active, `routedOutput` fans the 20 outputs into `FlowFieldVisualizer.setParams` (`aimmersive-clone-spec.md:260`); the canvas reads `engine.routedOutput()` in one rAF loop, never inference (`engine-architecture.md` §2.1).
|
||||
|
||||
---
|
||||
|
||||
## 7. React component tree
|
||||
|
||||
The Console wraps everything in `<EngineProvider>` (`engine-architecture.md` §2.2). The dock + drawers read `EngineApi` only; no drawer imports engine internals (lint seam, `engine-architecture.md` §6).
|
||||
|
||||
```
|
||||
<ConsoleApp> // focus stage + dock; owns rAF for canvases
|
||||
├── <Dock> // 48px right rail
|
||||
│ └── <DockIcon> ×6 // LEARN, IN, OUT, SYNTH, VISUAL, HELP
|
||||
│ (data-drawer; click → uiStore.setDrawer(depth))
|
||||
│
|
||||
├── <DrawerHost> // renders the open drawer at its depth
|
||||
│ ├── <Drawer depth="peek|expand"> // generic glass panel; ⤢ → FULL
|
||||
│ │ ├── <LearningBehaviourDrawer>
|
||||
│ │ │ ├── <FeedbackModeSelector> // pill → engine.feedback.setMode (§1.1)
|
||||
│ │ │ │ └── <ExploringIndicator> // <NoiseRing> ramp; learning-paused badge
|
||||
│ │ │ ├── <ArmScopeChooser> // arm-all / arm-output (§1.2)
|
||||
│ │ │ └── <TrainingParams> // <Slider> ×5/6 (§1.3)
|
||||
│ │ ├── <InputsDrawer> // workstream F (§6); <ControlAxis> rows
|
||||
│ │ ├── <RoutingDrawer> // §3
|
||||
│ │ │ ├── <BackendSelector> // Web Audio/MIDI/OSC/VCV
|
||||
│ │ │ └── <For each=group>
|
||||
│ │ │ └── <OutputGroupSection> // header: group master curve + mute-group
|
||||
│ │ │ └── <For each=output>
|
||||
│ │ │ └── <OutputControlRow> // ← the shared baseline component (§3.2)
|
||||
│ │ │ ├── <MuteToggle/> M
|
||||
│ │ │ ├── <ArmToggle/> S
|
||||
│ │ │ ├── <TriStateSegmented/> off|fixed|live
|
||||
│ │ │ ├── <DualRangeSlider/> min/max
|
||||
│ │ │ ├── <CurvePad/> 36×36
|
||||
│ │ │ └── <ValueBar/> live | fixedValue
|
||||
│ │ ├── <SynthEngineDrawer> // §5 ("Powerful Synth Engine" — never "C15")
|
||||
│ │ │ ├── <EngineSwitcher/>
|
||||
│ │ │ ├── <AudioControls/> Vol/BPM/Arp
|
||||
│ │ │ └── <SynthPresetSelect/> // tiered
|
||||
│ │ └── <VisualSystemDrawer> // §6
|
||||
│ │ ├── <For each=visualOutput><OutputControlRow/></For>
|
||||
│ │ └── <VisualPresetChips/>
|
||||
│ │
|
||||
│ └── <AdvancedModal depth="full"> // near-fullscreen, --z-modal, glass over dim
|
||||
│ ├── <FeedbackLab/> // LEARN FULL (state-machine + diagnostics)
|
||||
│ ├── <InputMatrix/> // IN FULL (workstream F)
|
||||
│ ├── <BackendAdvanced/> // OUT FULL — switches on active backend:
|
||||
│ │ ├── <MidiCcEditor/> // CC#/Ch/Name + count (§4.1)
|
||||
│ │ ├── <OscPathEditor/> // path + physical range (§4.2)
|
||||
│ │ ├── <VcvChannelEditor/> // min/max/freeze grid (§4.3)
|
||||
│ │ └── <SynthGroupMatrix/> // 18 sections × params (§4.4 / §5 FULL)
|
||||
│ └── <VisualMatrix/> // VISUAL FULL
|
||||
│
|
||||
├── <VerdictCluster> // +/−/undo/reroll; mirrors <FeedbackModeSelector> pill
|
||||
└── <HelpModal> // HELP icon opens directly
|
||||
```
|
||||
|
||||
Shared leaf primitives live in `shared/primitives/` (`<Slider>`, `<DualRangeSlider>`, `<CurvePad>`, `<NoiseRing>`, `<LossPlot>`, `<WeightHealth>`, `<LayerStats>`, `<GradientFlow>`, `<PillToggle>`, `<ControlAxis>` — `findings-design-and-manifold.md:60`). `<OutputControlRow>` is the one new composite component this spec introduces, reused across the Routing, Synth, Visual, and (optionally, §6 open choice 2) Inputs drawers.
|
||||
|
||||
---
|
||||
|
||||
## 8. Binding to the reactive spine + engine API (summary)
|
||||
|
||||
| UI surface | EngineApi call | Engine route |
|
||||
|---|---|---|
|
||||
| Feedback-mode pill | `engine.feedback.setMode(m)` | `nisps_ml_feedback_set_mode` (`feedback-modes-port-spec.md` §4) |
|
||||
| `−` verdict | `engine.feedback.thumbsDown()` | dispatch on mode → `on_down` (`findings-feedback-behaviour.md:90`) |
|
||||
| `+` verdict | `engine.feedback.thumbsUp()` | `addExample` + train, or `CommitStore` when exploring |
|
||||
| Arm (S) | `engine.feedback.setFocus(mask)` | `nisps_ml_feedback_set_focus` + `buildPinMask` |
|
||||
| Tri-state off/fixed | `routing-store` write → `weightsRevision`/pin recompute | `routedOutput` memo + pin mask (`engine-architecture.md` §2.1) |
|
||||
| min/max/curve | `routing-store` write | `routedOutput` memo `applyGroupOverride` (`param-map.js:301`) |
|
||||
| mute | `routing-store.muted` | send-effect zeroes/holds downstream, value still in `mlOutput` |
|
||||
| training sliders | `mlStore` setters | live engine params |
|
||||
| backend select | `engine.setBackend(b)` | swaps `OutputBackend` adapter; output-count guard |
|
||||
| synth setParam | (internal) send-effect throttle | `SynthEngine.setParam` ≥50ms/>0.002 |
|
||||
|
||||
All writes are **eager, synchronous, off-React-render** (`engine-architecture.md` §4) — drawers mutate stores; the single `routedOutput`→send-effect carries it to the backend; canvases read accessors in rAF. No drawer creates a second data path (the live-feedback guarantee, `engine-architecture.md` §2.3).
|
||||
|
||||
---
|
||||
|
||||
## 9. Open choices for the operator
|
||||
|
||||
1. **Surface `RandomiseOutputs` (third feedback mode)?** Spec hides it behind `?debug=1` ("Static roll"). Confirm v1 ships only the two product modes (Push away / Explore & place) or all three.
|
||||
2. **Do input-axis rows reuse `<OutputControlRow>`?** They share min/max/curve but not mute/off/live semantics. Spec defers to workstream F; confirm whether to unify the component or fork it.
|
||||
3. **Separate `state`/`muted`/`armed` fields vs the deployed conflated `frozen`↔`muted` single field.** Spec deliberately splits them (cleaner tri-state); this is a divergence from the deployed override system to record in `ALIGNMENT.md`. Confirm.
|
||||
4. **Default backend output counts on switch** trigger a `confirm()` weight-reset guard (ported from a-immersive). Confirm you want that friction, or prefer silent warm-start (`createWithWarmStart`, `aimmersive-clone-spec.md:250`).
|
||||
5. **Group master curve relative-delta behaviour** (drag the group curve nudges all child curves preserving offsets, `aimmersive-clone-spec.md:138`) — keep this a-immersive behaviour, or switch to absolute group curve? Spec keeps relative.
|
||||
6. **Where does the feedback-mode pill live for live play** — only mirrored next to the Verdict cluster, or also a permanent sub-`<StatusLine>` pill (`aimmersive-clone-spec.md:283`)? Spec puts it on the Verdict cluster.
|
||||
|
||||
---
|
||||
|
||||
## Verification corrections (adversarial pass, 2026-06-27) — verdict: minor-issues
|
||||
|
||||
The design reasoning, drawer model, tri-state/mute/arm separation and per-output baseline are sound; most
|
||||
`aimmersive-clone-spec.md` citations verified line-by-line. Apply these fixes in the build:
|
||||
|
||||
- **`engine.feedback.setMode` / `setFocus` / `exploring()` are not yet a JS surface** — only the C ABI
|
||||
(`nisps_ml_feedback_set_mode` / `set_focus` / `exploring`) exists. The Manifold `EngineApi.feedback` must
|
||||
ADD these JS wrappers over the C ABI in Phase 3 (this is intended new surface, not a mistake — but it must be
|
||||
built, not assumed). Likewise `engine.setBackend` and synth `setParam` are internal, not on EngineApi today.
|
||||
- **`param-map.js` (`applyTame/applyCurve/applyGroupOverride`) lives only in `deployments/meml-aimmersive`**,
|
||||
not in `playground/src`. The curve/override math must be **ported into `manifold/`** (see backends-spec
|
||||
`mapping.ts`), not imported from the deployed snapshot.
|
||||
- **`--dock-width` and min-thumb `#4488ff` are deployed-a-immersive values, absent from the Manifold tokens** —
|
||||
use Manifold tokens (`manifold-export/tokens/`) or add the missing ones deliberately.
|
||||
- A few `findings-feedback-behaviour.md:NN` citations are out of range / point at `feedback-modes-port-spec.md`
|
||||
instead (`learning_paused()` = `feedback.hpp:85` / port-spec:238; `exploring()` = `feedback.hpp:84`).
|
||||
- The React `<EngineProvider>`/`useEngine` pattern is in `findings-design-and-manifold.md §4`, not
|
||||
`engine-architecture.md §2.2` (that doc is SolidJS).
|
||||
439
docs/redesign/engine-architecture.md
Normal file
439
docs/redesign/engine-architecture.md
Normal file
|
|
@ -0,0 +1,439 @@
|
|||
I now have everything I need. Key findings confirmed:
|
||||
|
||||
- The **engine layer is genuinely good and parity-tested**: `WasmIML`, `EngineHost` (already uses `?worker&url` + hand-rolled imports), `nisps-processor.ts`, the pure pipelines, the stores. These are reusable.
|
||||
- The **debt is concentrated** in `mode-runtime.ts` (663-line god-hook), the 9 cloned `*Mode.tsx`, `ModeShell`/`ModeSwitcher`, and `App.tsx` (toy nav). The pull-vs-push reactivity bug lives in the god-hook (output flows through imperative `setInput` → `mlStore.process()` → effects, not a clean memo chain).
|
||||
- **Serving**: nginx root is `meml-aimmersive` (the legacy vanilla a-immersive at `a-immersive.html`, served untouched). The auto-deploy already builds `playground/dist` into `/home/w1n5t0n/deployments/meml.lnfinitemonkeys.org/playground/dist` but it's **currently unserved** (no location block points to it). COOP/COEP is server-scoped. The deploy clone is separate from the dev repo.
|
||||
- `FeedbackController` = the RL feedback bindings (`thumbsUp`/`thumbsDown`/`moveWeights`/`drawWeights` against the 3-mode loop), not a named class.
|
||||
|
||||
I have enough to write the spec. It's a foundation-architecture document, not a code change, so I'll return it directly.
|
||||
|
||||
---
|
||||
|
||||
# NISPS Foundation Architecture — One Engine, Two Skins
|
||||
|
||||
*Status: implementation-ready spec. Scope: the foundation only — the headless engine layer plus the two-skin shell. The detailed UX of each skin (Console drawers, interactive heatmap craft, etc.) is owned by `docs/redesign/playground-2.0-rewrite-plan.md` and not re-litigated here.*
|
||||
|
||||
*Constraint from the operator: "lay solid ground that's simple and doesn't paint us into a corner." Build BOTH skins on ONE engine; don't port the engine twice; don't inherit existing tech debt.*
|
||||
|
||||
---
|
||||
|
||||
## 0. The one decision that drives everything
|
||||
|
||||
**Separate the engine from the skin by an explicit, headless boundary — `EngineApi` — and let both skins be pure consumers of it.** Today there is no such boundary: `mode-runtime.ts` fuses WASM lifecycle, the reactive graph, input adapters, audio host, snapshots, heatmap sampling, and per-mode component wiring into one 663-line hook that each cloned `*Mode.tsx` calls. That fusion is *the* debt. The foundation's whole job is to draw the line the god-hook erased, on the correct side:
|
||||
|
||||
```
|
||||
┌──────────────────────────── ONE ENGINE ────────────────────────────┐
|
||||
gesture │ input pipeline → ML (WASM) → output pipeline → backends │ sound
|
||||
────────► │ (the reactive spine: one memo chain + one side-effect) │ ────────►
|
||||
│ exposes a headless EngineApi { inputRaw, mlOutput, routedOutput, │
|
||||
│ verdict actions, history, status } — NO JSX, NO DOM │
|
||||
└────────────────────────────────────────────────────────────────────┘
|
||||
▲ ▲
|
||||
│ reads EngineApi │ reads EngineApi
|
||||
┌──────────┴───────────┐ ┌──────────┴───────────┐
|
||||
│ skin-a-immersive │ │ skin-2.0 (Console) │
|
||||
│ (faithful clone) │ │ (right dock+drawers) │
|
||||
│ served at / │ │ served at /2.0 │
|
||||
└──────────────────────┘ └──────────────────────┘
|
||||
```
|
||||
|
||||
The litmus test for every file: **if it imports `solid-js/web`, JSX, or touches the DOM, it is a skin; otherwise it is the engine.** The engine compiles and is testable headlessly (Node/`__nisps` probe) with zero presentation. Two skins on one engine is then trivially true: a skin is just a tree of components that read `EngineApi` accessors and call `EngineApi` actions. Neither skin can break live-feedback, because neither skin owns the spine.
|
||||
|
||||
---
|
||||
|
||||
## 1. Module / directory structure
|
||||
|
||||
**Build in a fresh tree: `playground2/`** (justification in §3). Inside it, the engine and the two skins are siblings; presentation never reaches into engine internals except via `EngineApi`.
|
||||
|
||||
```
|
||||
playground2/
|
||||
├── index.html # skin-a-immersive entry (default, root)
|
||||
├── 2.0.html # skin-2.0 entry (Console variant)
|
||||
├── vite.config.ts # two inputs; base resolved per-build (§4)
|
||||
├── package.json # solid-js, vite, vite-plugin-solid, @playwright/test
|
||||
├── public/
|
||||
│ ├── nisps.wasm nisps.js # copied from scripts/build-wasm.sh output
|
||||
│ └── c15.wasm c15.glue.js # (only if/when C15 lands; deferred)
|
||||
│
|
||||
├── src/
|
||||
│ ├── engine/ # ───────── HEADLESS. No JSX. No DOM. ─────────
|
||||
│ │ │
|
||||
│ │ ├── index.ts # createEngine(schema, opts): EngineApi ← THE boundary
|
||||
│ │ ├── types.ts # EngineApi, EngineOptions, EngineStatus
|
||||
│ │ │
|
||||
│ │ ├── spine/
|
||||
│ │ │ ├── control-graph.ts # the memo chain + the single send-effect (§2). ~1 file.
|
||||
│ │ │ └── coalesce.ts # pointer-rate → display-cadence batch()+microtask gate
|
||||
│ │ │
|
||||
│ │ ├── ml/ # LIFTED from playground/ (parity-tested), de-storified
|
||||
│ │ │ ├── wasm-iml.ts # WasmIML class — adapt: HeapVec, reused out-buffer, no store writes
|
||||
│ │ │ ├── heap-vec.ts # NEW: re-derive HEAPF32 view on every access (replaces rebind())
|
||||
│ │ │ ├── wasm-worker.ts # LIFT as-is: disposable training worker (own WASM instance)
|
||||
│ │ │ └── dataset.ts # LIFT as-is
|
||||
│ │ │
|
||||
│ │ ├── audio/ # LIFTED from playground/ — already correct
|
||||
│ │ │ ├── engine-host.ts # LIFT ~as-is: ?worker&url, fetch+post bytes, transferables
|
||||
│ │ │ └── worklet/
|
||||
│ │ │ ├── nisps-processor.ts # LIFT as-is: 2nd WASM, hand-rolled imports
|
||||
│ │ │ └── audioworklet-globals.d.ts # LIFT as-is
|
||||
│ │ │
|
||||
│ │ ├── pipelines/ # LIFTED — pure fns, golden-tested
|
||||
│ │ │ ├── input.ts # deadzone→zoom→curve→smoothing→momentum (was input/pipeline.ts)
|
||||
│ │ │ ├── output.ts # global curve→smoothing→slew→freeze (was output/pipeline.ts)
|
||||
│ │ │ └── curves.ts # ONE curve catalog, golden-tested vs nisps/core/math.hpp (§3)
|
||||
│ │ │
|
||||
│ │ ├── backends/ # OutputBackend adapters (plan §3.6)
|
||||
│ │ │ ├── backend.ts # interface OutputBackend { send, start, teardown }
|
||||
│ │ │ ├── web-audio.ts # wraps engine-host (v1)
|
||||
│ │ │ ├── web-midi.ts # 7-bit CC out (v1 adapter; stub-OK at foundation)
|
||||
│ │ │ └── osc-bridge.ts # WS bridge (deferred; behind locked contract)
|
||||
│ │ │
|
||||
│ │ ├── stores/ # module singletons, split by update cadence (plan §3.2)
|
||||
│ │ │ ├── ml-store.ts # status/arch/dataset + outputs F32 signal + weightsRevision
|
||||
│ │ │ ├── input-store.ts # config store + raw-axes F32 signal {equals:false}
|
||||
│ │ │ ├── output-store.ts # global gate config + reuse buffer
|
||||
│ │ │ ├── control-store.ts # compound axes + per-param createMemo fanout
|
||||
│ │ │ ├── routing-store.ts # control-point off/fixed/live matrix
|
||||
│ │ │ ├── history-store.ts # snapshot DAG (undo/A-B/trail/snapshots)
|
||||
│ │ │ ├── session-store.ts # presets, persistence, URL params, mode switch
|
||||
│ │ │ └── bus.ts # LIFT as-is: typed sync pub/sub, cross-cutting events only
|
||||
│ │ │
|
||||
│ │ ├── runtime/ # the decomposed god-hook (plan §3.4) — engine-side hooks
|
||||
│ │ │ ├── use-input-adapters.ts # pointer/joystick/gamepad/mic → input-store.setRaw
|
||||
│ │ │ ├── use-audio-lifecycle.ts # backend start/stop/teardown on mount/mode-switch
|
||||
│ │ │ ├── use-snapshots.ts # DAG ops (→ history-store)
|
||||
│ │ │ ├── use-heatmap-sampler.ts # input-space heatmap, throttled to weightsRevision
|
||||
│ │ │ ├── use-trail.ts
|
||||
│ │ │ └── use-auto-explore.ts
|
||||
│ │ │
|
||||
│ │ ├── feedback/
|
||||
│ │ │ └── controller.ts # the 3-mode FeedbackController: thumbsUp/thumbsDown/randomize
|
||||
│ │ │ # + undo, wrapping moveWeights/drawWeights/train + auto-snapshot
|
||||
│ │ │
|
||||
│ │ ├── persist.ts # persist<T>(store, version, migrate): versioned + base64 weights
|
||||
│ │ └── probe.ts # window.__nisps — reads EngineApi only; ?debug=1-gated
|
||||
│ │
|
||||
│ ├── shared/ # ───── presentation shared by BOTH skins ─────
|
||||
│ │ ├── primitives/ # LIFTED: Slider, JoyMap, Heatmap, XYPad, LossPlot,
|
||||
│ │ │ # GradientFlow, WeightHealth, ProgressRing, ... (+ .demo.tsx)
|
||||
│ │ ├── theme/
|
||||
│ │ │ └── tokens.css # orange #ff6a00, danger #ff4466, glass, JetBrains Mono;
|
||||
│ │ │ # lint allowlist enforced
|
||||
│ │ ├── EngineProvider.tsx # createContext(EngineApi); both skins wrap their tree
|
||||
│ │ └── GenericMode.tsx # ONE schema-driven mode component (plan §3.5)
|
||||
│ │
|
||||
│ ├── skin-a-immersive/ # ───── DEFAULT skin (faithful a-immersive) ─────
|
||||
│ │ ├── main.tsx # mounts <EngineProvider><AImmersiveApp/></EngineProvider>
|
||||
│ │ ├── AImmersiveApp.tsx
|
||||
│ │ └── components/ # bottom-sheet-faithful chrome, floating RL buttons, top strip
|
||||
│ │
|
||||
│ ├── skin-2.0/ # ───── Console skin (Playground 2.0) ─────
|
||||
│ │ ├── main.tsx # mounts <EngineProvider><ConsoleApp/></EngineProvider>
|
||||
│ │ ├── ConsoleApp.tsx # the Manifold + dock + drawers + Verdict cluster
|
||||
│ │ └── components/ # Dock, Drawer (3-depth), VerdictCluster, ReadoutStrip, ...
|
||||
│ │
|
||||
│ └── dev/
|
||||
│ └── PrimitivesShowcase.tsx # /dev/primitives (shared, skin-agnostic)
|
||||
│
|
||||
└── tests/e2e/ # Playwright: spine invariant, both skins, parity hooks
|
||||
```
|
||||
|
||||
### Why this shape is "simple and doesn't paint us into a corner"
|
||||
|
||||
- **One axis of variation per directory.** Adding an *engine* touches `nisps/` + `schemas/` + `public/*.wasm`; zero skin files. Adding a *backend* touches `engine/backends/`; zero spine, zero skin. Adding/altering a *skin* touches one `skin-*/` tree; zero engine. This is the orthogonality principle made physical.
|
||||
- **`EngineApi` is the only seam.** Skins import `useEngine()` (the context) and nothing from `engine/` internals. A lint rule (`no-restricted-imports`: skins may not import `engine/**` except `engine/index.ts`/`engine/types.ts`) keeps it honest. You could delete a whole skin and the engine + tests still pass.
|
||||
- **`GenericMode` lives in `shared/`, not in a skin** — both skins render the same mode bodies; they differ only in *chrome* (how drawers/strips/clusters frame the canvas), not in *what a mode is*. This is what makes "two skins, one engine" cheap rather than a fork.
|
||||
|
||||
---
|
||||
|
||||
## 2. The reactive spine, concretely
|
||||
|
||||
This is the load-bearing structural fix and the literal definition of "doesn't paint us into a corner": there is exactly one path from gesture to sound, every consumer reads it, and a desync is a failing test rather than a recurring prod bug.
|
||||
|
||||
### 2.1 The chain (in `engine/spine/control-graph.ts`)
|
||||
|
||||
```ts
|
||||
// One input entry point. {equals:false} so identical-reference writes still notify.
|
||||
const inputRaw: Accessor<Float32Array> // input-store's raw-axes signal
|
||||
|
||||
// ── pure memos ──────────────────────────────────────────────────────────────
|
||||
const processedInput = createMemo(() => // deadzone→zoom→curve→smoothing→momentum
|
||||
runInputPipeline(inputRaw(), inputStore.config, inputState)); // pure, golden-tested
|
||||
|
||||
const mlOutput = createMemo(() => { // WasmIML.infer INTO a reused buffer
|
||||
const inp = processedInput(); // (no per-frame Float32Array alloc)
|
||||
iml.setInputsClamped(inp); // clamps to REAL arch (no phantom channels — §6/D)
|
||||
return iml.inferInto(mlOutBuf); // returns the reused buffer; pure *read* of weights
|
||||
}, mlOutBuf, { equals: false });
|
||||
|
||||
const routedOutput = createMemo(() => // voice-space + global gate
|
||||
runOutputPipeline(mlOutput(), routingStore, outputStore.config, outputState));
|
||||
|
||||
// ── the ONE side-effect ──────────────────────────────────────────────────────
|
||||
createEffect(() => {
|
||||
const out = routedOutput(); // single dependency
|
||||
backend.send(fillSendBuffer(out)); // dedicated re-filled transferable (neutering-safe)
|
||||
}); // ← the ONLY postMessage / engine post in the app
|
||||
```
|
||||
|
||||
Rules made structural, not aspirational:
|
||||
- **Memos are pure.** No `postMessage`, no `backend.send`, no store writes inside a memo. (Lint/review rule; the plan explicitly rejects P3's "postMessage in a memo" and P1's `createComputed`.)
|
||||
- **Weights mutate only through `ml-store` actions** that bump `weightsRevision`. `mlOutput`'s memo reads weights; because RL/train actions bump the revision and the memo's input or revision is a dependency, output re-derives. There is no leaked write path that can update audio without updating the UI.
|
||||
- **`mlOutput` writes into `mlOutBuf` and returns it** with `{equals:false}` — fixes today's per-frame `new Float32Array(...)` alloc in `WasmIML.process()` (line 302) and the god-hook's `recomputeOutputs`.
|
||||
- **Pointer-rate is coalesced to display cadence** in `coalesce.ts` via `batch()` + a microtask, *reactively* (not a rAF poll). rAF touches **canvas drawing only**, never inference.
|
||||
- **The send buffer is dedicated and owned by the effect**, separate from any signal buffer, because transferring neuters the source (today's `EngineHost.setParams` transfers `params.buffer`).
|
||||
|
||||
### 2.2 How BOTH skins consume it identically
|
||||
|
||||
Every consumer **reads accessors**; nobody is *pushed to*. There is no "push outputs to the UI" path that can rot — the bug class is deleted, not patched.
|
||||
|
||||
```ts
|
||||
// shared/EngineProvider.tsx
|
||||
const EngineContext = createContext<EngineApi>();
|
||||
export const useEngine = () => useContext(EngineContext)!;
|
||||
|
||||
// EngineApi (engine/types.ts) — the entire skin-facing surface
|
||||
interface EngineApi {
|
||||
// live reactive reads (the spine)
|
||||
inputRaw: Accessor<Float32Array>;
|
||||
processedInput:Accessor<Float32Array>;
|
||||
mlOutput: Accessor<Float32Array>; // raw model space (heatmap, diagnostics)
|
||||
routedOutput: Accessor<Float32Array>; // post-pipeline (engine, readout strip, visualizer)
|
||||
status: EngineStatus; // ready, audioStarted, training, examples, ... (store proxy)
|
||||
weightsRevision: Accessor<number>;
|
||||
layerStats: () => Float32Array; // diagnostics pull (throttled by caller)
|
||||
lossHistory: Accessor<ReadonlyArray<number>>;
|
||||
// actions
|
||||
setInput: (x: number, y: number) => void; // the ONLY input door
|
||||
feedback: FeedbackController; // thumbsUp/thumbsDown/randomize/undo/canUndo
|
||||
train: () => void;
|
||||
history: HistoryApi; // DAG: A/B pin/swap, jump-to-node
|
||||
audio: { start(): Promise<void>; stop(): Promise<void>; setMuted(b): void };
|
||||
schema: ModeSchema;
|
||||
}
|
||||
```
|
||||
|
||||
- **skin-a-immersive**: a floating RL button reads `engine.feedback.thumbsUp`; the top strip reads `engine.routedOutput()`; the XY pad calls `engine.setInput(x,y)`.
|
||||
- **skin-2.0**: the Verdict cluster's 👍 reads the *same* `engine.feedback.thumbsUp`; the ReadoutStrip reads the *same* `engine.routedOutput()`; the Manifold's pointer-down calls the *same* `engine.setInput(x,y)`.
|
||||
|
||||
Identical bindings, different chrome. A skin cannot create a second reactive path because it has no access to `WasmIML`, `backend`, or the stores' internal setters — only to `EngineApi`.
|
||||
|
||||
### 2.3 Live-feedback guarantee + the e2e assertion
|
||||
|
||||
**Guarantee:** any change to `inputRaw` (gesture) *or* `weightsRevision` (RL/train) propagates — in the same synchronous tick — to `routedOutput`, which the single effect sends to the backend and which every UI consumer reads. There is no code path that updates audio without updating the readout, or vice versa.
|
||||
|
||||
**The e2e invariant, asserted on every mode, in both skins, in CI** (the centerpiece; mirrors plan §3.1):
|
||||
|
||||
```ts
|
||||
// tests/e2e/spine.spec.ts — runs for each {mode} × {skin: '/', '/2.0'}
|
||||
const before = await page.evaluate(() => window.__nisps.getOutputs());
|
||||
await page.evaluate(() => window.__nisps.setInputs([0.9, 0.1])); // synchronous probe
|
||||
const after = await page.evaluate(() => window.__nisps.getOutputs());
|
||||
expect(after).not.toEqual(before); // ML output changed
|
||||
const eng = await page.evaluate(() => window.__nisps.getEngineParams());
|
||||
expect(eng).toChange(); // engine params changed, same input
|
||||
```
|
||||
|
||||
`window.__nisps` (in `engine/probe.ts`) is exposed synchronously, bypasses Solid reactivity with `untrack`/`batch`, and **reads `EngineApi` only** — so the probe exercises the exact path both skins use. Because the probe lives in the engine layer (not a skin), the *same* spec file runs unchanged against both `index.html` and `2.0.html`; if either skin ever fails to read the spine, that skin's row goes red.
|
||||
|
||||
A complementary **no-per-frame-alloc** check (heap-snapshot fuzz around a `setInputs` storm) guards the reused-buffer discipline.
|
||||
|
||||
---
|
||||
|
||||
## 3. Reuse-vs-rebuild
|
||||
|
||||
**Recommendation: build the foundation in a FRESH directory `playground2/`, lifting the parity-tested engine modules wholesale and discarding the skin/runtime layer.** This is strictly simpler than refactoring `playground/` in place and carries the least debt, *because the debt and the value are cleanly separable in the existing tree* — the value is the headless engine modules (already DOM-free), the debt is the god-hook + cloned modes + toy `App.tsx`.
|
||||
|
||||
### Reuse table
|
||||
|
||||
| Existing `playground/` module | Verdict | Action / why |
|
||||
|---|---|---|
|
||||
| `audio/engine-host.ts` | **Lift ~as-is** | Already correct: `?worker&url`, fetch-on-main + post bytes, transferables, lazy AudioContext. Best code in the repo. Minor: wrap behind `WebAudioBackend`. |
|
||||
| `audio/worklet/nisps-processor.ts` | **Lift as-is** | Two-WASM, hand-rolled auto-discovered imports, 128-sample shuttle. Load-bearing and correct. |
|
||||
| `audio/worklet/audioworklet-globals.d.ts` | **Lift as-is** | — |
|
||||
| `ml/wasm-iml.ts` | **Lift + adapt** | Keep the C-API surface and lifecycle. **Adapt:** (1) replace `HeapBuffer.rebind()` (manual, error-prone via `Object.defineProperty`) with `HeapVec` re-derive-on-access; (2) `inferInto(buf)` instead of `process()` allocating; (3) **remove `mlStore.__set*` writes from the class** — the class becomes a pure WASM wrapper, the store observes it. (4) Plumb real `nisps_ml_loss_history` (replaces the `lossHistory=[loss]` fake at line 419). |
|
||||
| `ml/wasm-worker.ts` | **Lift as-is** | Disposable training worker w/ own WASM instance. |
|
||||
| `ml/dataset.ts` | **Lift as-is** | JS mirror of the C++ ring. |
|
||||
| `input/pipeline.ts` | **Lift as-is** → `engine/pipelines/input.ts` | Pure, golden-tested, bit-equivalent to legacy. Becomes the `processedInput` memo body. |
|
||||
| `output/pipeline.ts` | **Lift as-is** → `engine/pipelines/output.ts` | Pure; becomes `routedOutput` body. **Fix:** use reuse buffer. |
|
||||
| `output/curves.ts` | **Lift + fix** | **Unify with the diverging `Curve` enum** in `generated/types.ts` (`exp/log/square/sqrt/sigmoid/cubic`) into ONE catalog, golden-tested against `nisps/core/math.hpp`. (Plan §4 "Pure-fn pipelines" Keep+fix.) |
|
||||
| `stores/bus.ts` | **Lift as-is** | Typed sync pub/sub; cross-cutting events only. |
|
||||
| `stores/ml-store.ts` | **Adapt** | Keep the signal/store split (it's already right: `outputs` F32 `{equals:false}`, store for status). **Add** `weightsRevision`. **Invert ownership:** store observes `WasmIML`, class stops writing the store. |
|
||||
| `stores/input-store.ts`, `output-store.ts` | **Adapt** | Keep config-store + raw F32 signal pattern; align to spine (input-store's raw signal becomes `inputRaw`). |
|
||||
| `stores/control-store.ts` | **Adapt** | Keep axis tables/presets/`interpolateAxis`. **Replace** the routing mechanism with **per-param `createMemo` fanout** (kills the `control-routing.ts` `JSON.stringify`-in-untracked-effect anti-pattern). |
|
||||
| `stores/session-store.ts`, `mode-store.ts`, `exploration-store.ts` | **Salvage values, re-cut along plan §3.2** | Re-partition into `routing-store` (control-point `off/fixed/live`), `history-store` (DAG), `session-store` (persist+presets+URL+mode). Lift constants/preset data; rebuild the store shapes. |
|
||||
| `stores/persistence.ts` | **Rebuild small** → `engine/persist.ts` | Replace bespoke per-store `Partial<>` merges + `Infinity↔null` slew encoding + slow `Array.from()` weight JSON with one `persist<T>(store, version, migrate)`; base64 weights. |
|
||||
| `features/snapshots.ts`, `overrides.ts`, `trail.ts`, `heatmap-sampler.ts`, `weight-health.ts`, `mic-input.ts` | **Salvage logic into `engine/runtime/` hooks** | The *algorithms* are fine; the *wiring* (god-hook calling them imperatively) is the debt. Re-home as `use-snapshots`/`use-trail`/`use-heatmap-sampler` reading stores, not as side-effects of `setInput`. Snapshots fold into the DAG. |
|
||||
| `features/control-routing.ts` | **Discard** | The `JSON.stringify`-inside-`untrack` effect anti-pattern; replaced by control-store memo fanout. |
|
||||
| `features/session-preset.ts` | **Rebuild** | Composed-layers model (control/synth/weights/mode), per plan §3.8. |
|
||||
| `primitives/*` (16 + demos) | **Lift as-is** → `shared/primitives/` | Genuinely good, skin-agnostic, already have `.demo.tsx`. The reusable presentation layer. |
|
||||
| `dev/PrimitivesShowcase.tsx`, `debug/probe.ts` | **Lift + adapt** | Showcase as-is. Probe re-pointed at `EngineApi` (it currently pokes stores directly); `?debug=1`-gate it. |
|
||||
| **`modes/mode-runtime.ts` (663 lines)** | **DISCARD** | The god-hook. Its responsibilities are split across `engine/spine/`, `engine/runtime/`, `engine/feedback/`. **Do not inherit.** |
|
||||
| **`modes/*Mode.tsx` (9 cloned files, ~5.6k lines)** | **DISCARD** | Replaced by one `shared/GenericMode.tsx` driven by schema. The clones *are* the debt. |
|
||||
| `modes/ModeShell.tsx`, `ModeSwitcher.tsx`, `SettingsDrawer.tsx`, `mode-helpers.ts` | **Discard / re-design per skin** | Chrome belongs to skins now; a capability-class switcher is rebuilt minimally. |
|
||||
| `App.tsx`, `main.tsx` | **Discard** | Toy hash-router with a "home page." Replaced by two skin entries + `EngineProvider`. |
|
||||
| `modes/generated/*` | **Regenerate, don't lift** | Codegen output; add `capability_class` + `tier` fields (plan §3.5/§3.9) and emit into `playground2/`. |
|
||||
|
||||
**Verdict in one line:** lift the entire `engine/` + `shared/primitives/` (≈ the `ml/`, `audio/`, `input/`, `output/`, `primitives/`, `bus.ts` value — the parity-tested core) and rebuild only the ≈6.5k lines of god-hook + cloned modes + toy app that *are* the debt.
|
||||
|
||||
### Fresh dir vs refactor-in-place — justification
|
||||
|
||||
| | **Fresh `playground2/` (recommended)** | Refactor `playground/` in place |
|
||||
|---|---|---|
|
||||
| Debt inheritance | Zero — debt files never copied; engine lifted file-by-file on purpose | High — easy to leave a god-hook tendril; "delete later" rarely happens |
|
||||
| "Simple ground" | Clean import graph from day 1; lint seam enforceable immediately | Mixed old/new imports during migration; seam blurry for weeks |
|
||||
| Parity risk | None — engine files lifted verbatim, parity-check runs against the same `nisps.wasm` | None, but harder to prove which path a test exercised |
|
||||
| Live deploy | Untouched. `playground/dist` keeps building; we wire `playground2/dist` only when ready | Risk of breaking the live (if pointed) build mid-refactor |
|
||||
| Rollback | `rm -rf playground2/` | `git revert` across an entangled history |
|
||||
| Cost | One-time copy of ~10 good files | Ongoing vigilance against re-entanglement |
|
||||
|
||||
A fresh dir is the cheaper path to *less debt while reusing the parity-tested core* — exactly the operator's brief. `playground/` stays as the working reference (and keeps deploying) until `playground2/` reaches parity, then `playground/` is deleted and `playground2/`→`playground/` (a rename, by which point the engine seam is proven).
|
||||
|
||||
> **Note for the operator (open question A in the plan):** if 2.0 is actually destined for the laptop `~/src/manifold` tree, this whole `playground2/` lands there instead — the architecture is identity-agnostic and the dir name is the only thing that changes. This blocks namespace/repo identity; see §6.
|
||||
|
||||
---
|
||||
|
||||
## 4. Two-skins mechanism (build + serve)
|
||||
|
||||
**Decision: two Vite entries, one build, one `dist/` — NOT a runtime route.** Two HTML entry points compiled in a single `vite build` produce `dist/index.html` (skin A) and `dist/2.0.html` (skin B), sharing all common chunks (engine, primitives, wasm). Rationale against "simple, no corners":
|
||||
|
||||
- **A runtime route** (`/2.0` as a client route inside one bundle) would force both skins into one app shell, one router, and — crucially — risk a *shared mutable* engine instance and shared chrome assumptions, reintroducing exactly the cross-skin coupling we're eliminating. It also makes "delete a skin" hard.
|
||||
- **Two entries** keep the skins genuinely independent (each owns its `main.tsx` + `EngineProvider` mount) while Vite's `manualChunks`/automatic splitting means the engine + primitives are **one shared chunk loaded by both** — zero engine duplication on disk or over the wire. This is the cleanest expression of "one engine, two skins."
|
||||
|
||||
### Vite config
|
||||
|
||||
```ts
|
||||
// playground2/vite.config.ts
|
||||
import { defineConfig } from 'vite';
|
||||
import solid from 'vite-plugin-solid';
|
||||
import { resolve } from 'node:path';
|
||||
|
||||
const isolation = {
|
||||
'Cross-Origin-Opener-Policy': 'same-origin',
|
||||
'Cross-Origin-Embedder-Policy': 'require-corp',
|
||||
};
|
||||
|
||||
export default defineConfig(({ command }) => ({
|
||||
// Skin A is served at site root; skin B at /2.0/. We build ONCE with base '/2.0/'-aware
|
||||
// asset URLs? No — simpler: build relative, let nginx alias map. See note below.
|
||||
base: './', // relative asset URLs → same dist works at / and at /2.0/
|
||||
plugins: [solid()],
|
||||
server: { port: 5173, headers: isolation }, // dev: COOP/COEP for SharedArrayBuffer + worklet
|
||||
preview: { port: 4173, headers: isolation },
|
||||
build: {
|
||||
target: 'es2022',
|
||||
sourcemap: true,
|
||||
rollupOptions: {
|
||||
input: {
|
||||
main: resolve(__dirname, 'index.html'), // → dist/index.html (skin A, default)
|
||||
v2: resolve(__dirname, '2.0.html'), // → dist/2.0.html (skin B, Console)
|
||||
},
|
||||
},
|
||||
},
|
||||
}));
|
||||
```
|
||||
|
||||
**Why `base: './'` (relative) rather than a fixed `/2.0/` base.** The two skins share `dist/assets/*`. If skin B used absolute `base:'/2.0/'`, skin A (at `/`) would request the wrong asset prefix. Relative `base` makes **the same `dist/` mount correctly at both `/` and `/2.0/`** with no second build. The one caveat is the **AudioWorklet URL** and the **`?worker&url` chunk**: Vite emits these as module URLs resolved relative to the importing chunk, which works under a sub-path *as long as the chunk itself is loaded relatively* — relative `base` satisfies this. The worklet's WASM is fetched by absolute `/nisps.wasm`; under a sub-path we make that **origin-relative and configurable** (engine reads `import.meta.env.BASE_URL` or a runtime `<base>`), so `/2.0/` resolves `…/nisps.wasm` correctly. (Today `wasm-iml.ts`/`engine-host.ts` hardcode `/nisps.wasm` against `window.location.origin` — fine at root, must become base-aware for the sub-path. This is the single concrete code change the two-skins serving imposes.)
|
||||
|
||||
Two `package.json` scripts make dev ergonomic:
|
||||
```jsonc
|
||||
"dev": "vite", // serves both index.html and 2.0.html
|
||||
"dev:2.0": "vite --open /2.0.html",
|
||||
"build": "tsc --noEmit && vite build", // emits dist/index.html + dist/2.0.html
|
||||
```
|
||||
|
||||
### nginx (keeping the legacy a-immersive root untouched)
|
||||
|
||||
The current root is `meml-aimmersive` serving the **legacy vanilla** `a-immersive.html`. The new SolidJS app builds to `…/meml.lnfinitemonkeys.org/playground/dist` (currently unserved). The cleanest, no-corner serving plan: **add location blocks for the new app under explicit sub-paths, leave `/` on the legacy root for now**, and flip `/` to the new skin A only when the operator signs off.
|
||||
|
||||
```nginx
|
||||
# inside the existing 443 server { } for meml.lnfinitemonkeys.org
|
||||
# (COOP/COEP already set at server scope → inherited by all locations below)
|
||||
|
||||
set $pg2 /home/w1n5t0n/deployments/meml.lnfinitemonkeys.org/playground2/dist;
|
||||
|
||||
# ── Skin B: Console at /2.0 ────────────────────────────────────────────────
|
||||
location = /2.0 { return 301 /2.0/; }
|
||||
location /2.0/ {
|
||||
alias $pg2/;
|
||||
index 2.0.html;
|
||||
try_files $uri $uri/ /2.0/2.0.html; # SPA fallback to skin B's entry
|
||||
}
|
||||
|
||||
# ── Skin A: faithful clone, staged at /next while it's proven ──────────────
|
||||
location = /next { return 301 /next/; }
|
||||
location /next/ {
|
||||
alias $pg2/;
|
||||
index index.html;
|
||||
try_files $uri $uri/ /next/index.html; # SPA fallback to skin A's entry
|
||||
}
|
||||
|
||||
# ── Shared assets (one chunk set for both skins) ──────────────────────────
|
||||
location /assets/ { alias $pg2/assets/; } # relative-base requests resolve here
|
||||
location = /nisps.wasm { alias $pg2/nisps.wasm; }
|
||||
location = /nisps.js { alias $pg2/nisps.js; }
|
||||
|
||||
# legacy root stays:
|
||||
# root /home/w1n5t0n/deployments/meml-aimmersive; index a-immersive.html; (UNCHANGED)
|
||||
```
|
||||
|
||||
**Cutover (one-line change, when signed off):** point `/` at skin A by setting the server `root $pg2;` + `index index.html;` and a top-level `try_files $uri /index.html;`. Until then `/` serves the legacy artifact untouched, `/next/` is the new faithful skin, `/2.0/` is the Console. The auto-deploy script gains one line: after `playground/`'s `npm run build`, also `cd playground2 && npm install && npm run build` (or replace once `playground/` is retired). COOP/COEP is **server-scoped already**, so every sub-path inherits cross-origin isolation — no per-location header duplication needed (the one real correctness requirement for SharedArrayBuffer + the worklet under a sub-path is satisfied for free).
|
||||
|
||||
---
|
||||
|
||||
## 5. Build sequence (small, shippable, Playwright-testable)
|
||||
|
||||
Each step ends green and observable. Parity checkpoints (verification chokepoints C/E) called out. This mirrors the plan's Phase 0–4 but with the **two-skin seam established early** so neither skin can diverge.
|
||||
|
||||
| Step | Deliverable | Test gate | Parity |
|
||||
|---|---|---|---|
|
||||
| **S0 — Scaffold + seam** | `playground2/` Vite skeleton; two HTML entries; `tokens.css` + CSS-var lint allowlist; `EngineApi`/`EngineProvider` stubs; codegen re-emitting into `playground2/` with `capability_class`+`tier` and **build-fails-on-arch-mismatch**; `persist<T>` helper; `bus.ts` lifted. Two trivial skins each render "hello from skin A/B" reading a stub `EngineApi`. | `bun run typecheck` green; codegen golden test; Playwright loads `/` and `/2.0.html`, both mount. | Codegen idempotent (byte-identical regen). |
|
||||
| **S1 — Spine against a stub** | `control-graph.ts` memo chain + single send-effect with a **stubbed `mlOutput`** (deterministic fn of input); `input-store` + `output-store` with lifted pure pipelines (curve enums unified, golden-tested); `coalesce.ts`. `EngineApi.setInput`/`routedOutput` real. | **Spine invariant e2e** (`setInputs→getOutputs changes`) passes against the stub, **on both skins**; no-per-frame-alloc heap fuzz. | Pipeline golden vs legacy bit-equivalence. |
|
||||
| **S2 — WASM ML bridge** | Lift `wasm-iml.ts` + `wasm-worker.ts` + `dataset.ts`; add `HeapVec` re-derive; `inferInto(buf)`; `ml-store` observes the class (no class→store writes); `weightsRevision`; real `nisps_ml_loss_history` plumbed. `mlOutput` memo now calls real `WasmIML`. | Probe `infer`/`train`/`getLayerStats`; loss plot draws a *real* curve; spine invariant still green on both skins. | **Checkpoint:** main-thread inference matches native within 1e-5 (`parity-check.sh`). |
|
||||
| **S3 — Audio + first real mode via GenericMode** | Lift `engine-host.ts` + worklet behind `WebAudioBackend`; base-aware WASM URL; `GenericMode.tsx` rendering **PAFSynth** (synth class). Mount `GenericMode` inside *both* skin shells (minimal chrome each). | Per-mode live-feedback e2e (engine params change on input move) on both skins; "Start audio" gesture works. | **Checkpoint:** `parity-check.sh` green; browser PAFSynth audio equivalent to firmware (chokepoint C). |
|
||||
| **S4 — Skin A default, faithful** | Build out `skin-a-immersive/` to faithfully reproduce a-immersive chrome (bottom-sheet, floating RL buttons, top strip) over `GenericMode` + `EngineApi`. Feedback controller (`thumbsUp/Down/randomize/undo`) wired; snapshot DAG (`history-store`) behind undo. Serve at `/next/`. | Playwright drives the faithful loop via probe + via real DOM on `/next/`; visual-faithfulness check (key anchors, orange tokens); a-immersive feature-parity subset (chokepoint D start). | — |
|
||||
| **S5 — Skin B Console at /2.0** | Build out `skin-2.0/`: Manifold + right dock + 3-depth drawers + Verdict cluster + interactive ReadoutStrip, all reading the *same* `EngineApi`. RL-undo + A/B against the DAG. Serve at `/2.0/`. | The **same** spine + feedback e2e specs run against `/2.0/` unchanged and pass; Console-specific drawer/depth e2e. | — |
|
||||
| **S6 — Mode coverage + backends + persistence** | All in-scope engines via `GenericMode` (synth/sequencer/controller/visual classes; `SequencerLane`, `VisualEngine`, controller meters); capability-class switcher in both skins; `WebMidiBackend` + (stubbed) `OscBridgeBackend`; versioned persistence + base64 weights; control surface (axis memo fanout, presets, pinning). | Every mode passes live-feedback e2e in CI on both skins; tactile-constant e2e (3px/300ms) on the Console strip. | **Checkpoint:** parity per mode; full `run-all-tests.sh` green = chokepoint E. |
|
||||
| **S7 — Cutover** | Point nginx `/` at skin A; delete legacy `meml-aimmersive` serving (archive the artifact); retire `playground/`, rename `playground2/`→`playground/`; update deploy script. | Post-cutover smoke e2e against `/` and `/2.0/`. | — |
|
||||
|
||||
The deliberate ordering choice vs the plan: **establish `EngineApi` + both skin mounts at S0** (not after the Console is built), so from the very first stub the "two skins read one engine" property is *tested*, and no skin can quietly grow its own data path. The Console UX detail (drawers, depths) is still de-risked against `/dev/primitives` and user-tested with Dimi before S5, per the plan.
|
||||
|
||||
---
|
||||
|
||||
## 6. Risks & open questions for the operator
|
||||
|
||||
**Open questions that block or shape the build (need a call before S0/S2):**
|
||||
|
||||
1. **Canonical tree / identity (plan §8-A).** Is the foundation `playground2/` *in this VPS `MEMLNaut-NISPS` tree* (`nisps::`), or does it belong in the laptop `~/src/manifold` tree (`manifold::`)? This blocks the dir name, namespace, repo, and codegen identity. *Recommendation:* build here (the live tree the recon ran against; rename is trivial), but confirm.
|
||||
|
||||
2. **Fixed-2-input contract (plan §8-D) — the big one.** The foundation hard-codes the honest fixed `MLP<2,10,14,18,126>` contract: codegen **fails the build** on schema/arch mismatch and `GenericMode` clamps `setInput` to the real arch (killing today's phantom-channel OOB writes in `mode-runtime.ts` lines 224-230 and `wasm-iml` warn-and-ignore at lines 198-203). Consequence: **multi-input mic modes (XIASRI / sound_analysis_midi) are firmware-only in v1** and show a "single-input in browser" badge; the runtime-shaped MLP is deferred *behind a passing parity check, never bundled into this rewrite*. Confirm you accept this v1 contraction — it's a materially smaller browser surface than "all modes work in browser."
|
||||
|
||||
3. **Two entries vs runtime route — confirm.** I've chosen two Vite entries / one `dist` (cleanest "one engine, two skins"). If you'd rather a single bundle with a `/2.0` client route (e.g. to share an app shell), say so now — it changes the skin-mount and engine-instance model.
|
||||
|
||||
4. **Serving cutover policy.** I propose: legacy a-immersive stays at `/` untouched; new skins live at `/next/` (faithful) and `/2.0/` (Console) until you sign off, then `/`→skin A. Confirm you want the new default *staged* rather than replacing `/` immediately. (Also: do you want skin A to eventually *replace* the legacy vanilla a-immersive, or coexist?)
|
||||
|
||||
5. **`capability_class` + `tier` schema fields (plan §8-B/§8-C).** The foundation's `GenericMode` and switcher depend on these new schema fields. Confirm the four classes (Synth/Controller/Sequencer/Visual) and the per-param/diagnostic `tier: 0|1|2` disclosure model are canonical, since codegen and `EngineApi.schema` bake them in from S0.
|
||||
|
||||
**Risks carried into the build (mitigations baked into the architecture):**
|
||||
|
||||
| Risk | Mitigation |
|
||||
|---|---|
|
||||
| A skin quietly grows a second data path → live-feedback rot returns | `EngineApi` is the only seam; `no-restricted-imports` lint forbids skins importing `engine/**` internals; spine e2e runs on **both** skins in CI. |
|
||||
| Memo purity violated (side-effect creeps into a memo) | Single documented send-`createEffect`; lint/review rule; the chain is one tested file (`control-graph.ts`). |
|
||||
| WASM URL breaks under `/2.0/` sub-path | Base-aware asset resolution (`import.meta.env.BASE_URL`); the one concrete change the sub-path imposes; covered by a load-under-sub-path e2e. |
|
||||
| Shared `dist` + relative base mis-resolves worklet/`?worker&url` chunk | `base:'./'` + nginx `alias`; explicit load-both-skins e2e at S0 catches it immediately. |
|
||||
| Lifting `wasm-iml` re-imports its store-coupling debt | Adapt-on-lift: strip `mlStore.__set*` from the class; store observes the class. Enforced by the headless-engine lint (engine files may not import skin/JSX). |
|
||||
| Two builds in the deploy pipeline slow/again-unserved | One `dist`, one `vite build` (two entries); deploy script gains one block; `/2.0/` + `/next/` wired in nginx from the start so it isn't "built but unserved" like today's `playground/dist`. |
|
||||
| Parity drift on any core touch | Every engine-touching step (S2, S3, S6) gated on `parity-check.sh` green before the skin lands. |
|
||||
| Probe ships to prod | `engine/probe.ts` gated behind `?debug=1`; reads `EngineApi` only. |
|
||||
|
||||
---
|
||||
|
||||
### Relevant file paths
|
||||
- Authoritative UX/feature plan (owns the skins' detail): `/home/w1n5t0n/src/MEMLNaut-NISPS/docs/redesign/playground-2.0-rewrite-plan.md`
|
||||
- Design-intent reference for skin-a-immersive: `/home/w1n5t0n/src/MEMLNaut-NISPS/docs/redesign/playground-2026.md`
|
||||
- The god-hook to discard (the debt): `/home/w1n5t0n/src/MEMLNaut-NISPS/playground/src/modes/mode-runtime.ts`
|
||||
- Engine modules to lift: `/home/w1n5t0n/src/MEMLNaut-NISPS/playground/src/audio/engine-host.ts`, `…/audio/worklet/nisps-processor.ts`, `…/ml/wasm-iml.ts`, `…/ml/wasm-worker.ts`, `…/ml/dataset.ts`, `…/input/pipeline.ts`, `…/output/pipeline.ts`, `…/output/curves.ts`, `…/stores/bus.ts`, `…/stores/ml-store.ts`, `…/primitives/*`
|
||||
- Codegen TS types to extend (`capability_class`+`tier`): `/home/w1n5t0n/src/MEMLNaut-NISPS/playground/src/modes/generated/types.ts`
|
||||
- Serving facts: nginx `/etc/nginx/sites-available/meml.lnfinitemonkeys.org` (server-scope COOP/COEP, root `meml-aimmersive`); deploy script `/home/w1n5t0n/.config/webhooks/meml-deploy.sh` (builds `…/meml.lnfinitemonkeys.org/playground/dist`, currently unserved); webhook def `/home/w1n5t0n/.config/webhooks/hooks.json`
|
||||
- New foundation tree to create: `/home/w1n5t0n/src/MEMLNaut-NISPS/playground2/` (or the laptop `manifold` tree, pending open question 1)
|
||||
547
docs/redesign/feedback-modes-port-spec.md
Normal file
547
docs/redesign/feedback-modes-port-spec.md
Normal file
|
|
@ -0,0 +1,547 @@
|
|||
I have complete grounding now. I'll write the implementation-ready spec.
|
||||
|
||||
# Implementation Spec — "Down Action" Negative-Feedback Feature in the `nisps/` Core
|
||||
|
||||
## 0. Provenance
|
||||
|
||||
Every behavior below is read directly from local branch `fork-feedback` of the memllib submodule at `/home/w1n5t0n/src/MEMLNaut-NISPS-upstream/`:
|
||||
|
||||
- `examples/InterfaceRL.hpp` / `.cpp` — the FEEDBACK_MODE enum, all new state, all helpers.
|
||||
- `hardware/memlnaut/display/RLView.hpp` — display only (out of scope for the core; informs JS only).
|
||||
- `modes/MEMLNautModeTR8S.hpp` — selector + focus-mask wiring (`activeDims_` source).
|
||||
|
||||
Target core read from `/home/w1n5t0n/src/MEMLNaut-NISPS/`: `nisps/ml/{mlp,rl,init,training}.hpp`, `nisps/core/{concepts,perf,rng}.hpp`, `nisps/wasm/bindings.cpp`, plus the JS driver (`playground/src/modes/mode-runtime.ts`, `playground/src/ml/{wasm-iml.ts,types.ts}`, `playground/src/stores/exploration-store.ts`) and the test/parity harness (`tests/cpp/`).
|
||||
|
||||
---
|
||||
|
||||
## 1. Exact behavior of all 3 modes (from the real source)
|
||||
|
||||
The original feature replaces a single "thumbs-down" semantic with three selectable behaviors. The selector is `FEEDBACK_MODE` (`InterfaceRL.hpp`):
|
||||
|
||||
```cpp
|
||||
enum class FEEDBACK_MODE : uint8_t {
|
||||
AVOID = 0, // down = negative reward / geometric push away (current behaviour)
|
||||
RANDOMISE_OUTPUTS, // down = bypass MLP, hold static random outputs (re-roll on each down); up = keep at current input
|
||||
RANDOMISE_MLP // down = snapshot+randomise the live net (re-roll via left toggle); up/drag = keep & restore; down again = cancel
|
||||
};
|
||||
```
|
||||
|
||||
The exploration state, all `private` members in `InterfaceRL`:
|
||||
|
||||
```cpp
|
||||
FEEDBACK_MODE feedbackMode_ = FEEDBACK_MODE::AVOID;
|
||||
bool explore_active_ = false; // inside a RANDOMISE_OUTPUTS static hold or RANDOMISE_MLP temp net
|
||||
bool learningPaused_ = false; // optimise() suspended while exploring
|
||||
std::vector<float> staticRandomOut_; // held output in RANDOMISE_OUTPUTS
|
||||
MLP<float>::mlp_weights weightSnapshot_; // original net, saved in RANDOMISE_MLP
|
||||
std::vector<bool> activeDims_; // focus mask (set by the mode; empty == all active)
|
||||
```
|
||||
|
||||
There are exactly three event entry points, dispatched in `loopCallback` (thread context, never ISR):
|
||||
- **up** = thumbs-up (MomA1) → `_feedback_up()`
|
||||
- **down** = thumbs-down (MomA2) → `_feedback_down()`
|
||||
- **drag-store** = joystick-click / Tog-A1 release → `_feedback_drag_store()`
|
||||
|
||||
Plus a fourth implicit transition: **switching the mode** while exploring → `_abort_explore()`.
|
||||
|
||||
A critical cross-cutting fact: `optimiseSometimes()` early-returns when `learningPaused_` is true, so **training is fully suspended during any active exploration** in modes 2 and 3.
|
||||
|
||||
### 1.1 Mode AVOID (`FEEDBACK_MODE::AVOID`) — historical behavior, unchanged
|
||||
|
||||
State machine is trivial — `explore_active_`/`learningPaused_` are never touched.
|
||||
|
||||
| Event | Action |
|
||||
|-------|--------|
|
||||
| **up** | `_perform_like_action()` — store `+1` example at `(controlInput, action)`; reset `dislikeMultiplier_ = 1`. |
|
||||
| **down** | `_perform_dislike_action()` — store/strengthen a negative example (see below). |
|
||||
| **drag-store** | `storeExperience(1.f, controlInput, savedAction)` — place the frozen output as a `+1` example. |
|
||||
|
||||
`_perform_dislike_action()` (`InterfaceRL.cpp:41`): scan replay memory; if a *negative* item exists within Euclidean distance `0.05` of `controlInput`, **deepen** it (`reward = max(reward - 1, -kMaxDislikeMultiplier)` where `kMaxDislikeMultiplier = 16`) instead of adding a duplicate; otherwise `storeExperience(-1.f, ...)`. Then `dislikeMultiplier_ = min(dislikeMultiplier_ * 2, 16)`.
|
||||
|
||||
The actual "push away" happens later in `optimise()` (`InterfaceRL.cpp:717`), **not** at press time. The geometric-push formula (the semantic that matters for reconciliation, §2.5):
|
||||
|
||||
```cpp
|
||||
// meanPositiveAction = mean of the kCentroidK (=4) positive memories
|
||||
// nearest to controlInput (k-NN centroid).
|
||||
float pushStep = std::clamp(fabsf(avgRewardNeg), 0.25f, 1.0f) * kGeometricPushScale; // kGeometricPushScale = 0.5
|
||||
for (const auto& neg_action : tsNegative.second) {
|
||||
dir[j] = neg_action[j] - meanPositiveAction[j]; // direction AWAY from positive centroid
|
||||
len = ||dir||;
|
||||
const float effectivePushStep = pushStep / (1.0f + len); // taper when already far
|
||||
// useRandom when len <= 1e-4 (degenerate: neg sits on the centroid)
|
||||
for each dim j with activeDims_[j]:
|
||||
float d = useRandom ? (rand&0xFF/127.5 - 1) : (dir[j]/len);
|
||||
target[j] = clamp(neg_action[j] + d * effectivePushStep, 0, 1);
|
||||
// train MLP toward `target` (push the disliked action away from the liked centroid)
|
||||
}
|
||||
```
|
||||
|
||||
So **AVOID = "move the network's output at the disliked input geometrically away from the centroid of nearby liked outputs."** Focus-aware: dims with `activeDims_[j] == false` are left untouched (`if (!active) continue;`).
|
||||
|
||||
### 1.2 Mode RANDOMISE_OUTPUTS (`FEEDBACK_MODE::RANDOMISE_OUTPUTS`)
|
||||
|
||||
Bypasses the MLP entirely and holds a static, hand-rolled output vector. The MLP and replay memory are not modified until commit.
|
||||
|
||||
| Event | `explore_active_ == false` (idle) | `explore_active_ == true` (exploring) |
|
||||
|-------|-----------------------------------|----------------------------------------|
|
||||
| **down** (1st) | `_enter_randomise_outputs()` | — |
|
||||
| **down** (subsequent) | — | `_roll_static_outputs()` (re-roll) |
|
||||
| **up** | `_perform_like_action()` (normal like) | `_commit_explore(controlInput, action)` (keep) |
|
||||
| **drag-store** | normal `+1` store | normal `+1` store, **stays in explore** (keep re-rolling) |
|
||||
|
||||
`_enter_randomise_outputs()` (`InterfaceRL.cpp:150`):
|
||||
```cpp
|
||||
explore_active_ = true;
|
||||
learningPaused_ = true;
|
||||
staticRandomOut_ = action; // seed inactive dims with current value
|
||||
_roll_static_outputs();
|
||||
```
|
||||
|
||||
`_roll_static_outputs()` — **the randomisation formula** (`InterfaceRL.cpp:159`):
|
||||
```cpp
|
||||
if (staticRandomOut_.size() != action.size()) staticRandomOut_ = action;
|
||||
for (size_t i = 0; i < staticRandomOut_.size(); i++) {
|
||||
const bool active = activeDims_.empty() || (i < activeDims_.size() && activeDims_[i]);
|
||||
if (active) staticRandomOut_[i] = static_cast<float>(rand() & 0xFFFF) / 65535.f; // uniform [0,1]
|
||||
// inactive dims keep their seeded entry value
|
||||
}
|
||||
newInput = true;
|
||||
```
|
||||
|
||||
Per-dim: **focused dims → fresh uniform [0,1]; unfocused dims → frozen at the value `action` had on entry.** The held vector is emitted in `generateAction()` with **no inference and no OU noise** (`InterfaceRL.cpp:998`):
|
||||
```cpp
|
||||
if (feedbackMode_ == FEEDBACK_MODE::RANDOMISE_OUTPUTS && explore_active_) {
|
||||
mappingOutput = staticRandomOut_; // MLP bypassed
|
||||
} else { /* GetOutput + OU noise + reflect-at-bounds */ }
|
||||
```
|
||||
|
||||
`_commit_explore(inputPos, outputVal)` (`InterfaceRL.cpp:188`): `storeExperience(1.f, inputPos, outputVal)` then `_restore_after_explore()`. For OUTPUTS, `outputVal == action`, i.e. **the static sound as currently heard becomes a +1 example at the current input**, after which inference resumes.
|
||||
|
||||
`_restore_after_explore()` (`InterfaceRL.cpp:194`): for OUTPUTS it does **not** touch weights (no snapshot exists); it only `learningPaused_ = false; explore_active_ = false; newInput = true;`.
|
||||
|
||||
### 1.3 Mode RANDOMISE_MLP (`FEEDBACK_MODE::RANDOMISE_MLP`)
|
||||
|
||||
Snapshots the live weights, randomises the live net into a "temp net", and lets the user audition. Commit/restore swaps the original net back.
|
||||
|
||||
| Event | `explore_active_ == false` (idle) | `explore_active_ == true` (exploring) |
|
||||
|-------|-----------------------------------|----------------------------------------|
|
||||
| **down** (1st) | `_enter_randomise_mlp()` | — |
|
||||
| **down** (subsequent) | — | `_cancel_explore()` (discard temp net, restore original) |
|
||||
| **re-roll** | — | (firmware: left-toggle → `randomiseTheNetwork()`; **not a `_feedback_*` hook**) |
|
||||
| **up** | `_perform_like_action()` | `_commit_explore(controlInput, action)` (keep) |
|
||||
| **drag-store** | normal `+1` store | `+1` store **and** `_restore_after_explore()` (reposition-commit) |
|
||||
|
||||
`_enter_randomise_mlp()` (`InterfaceRL.cpp:170`):
|
||||
```cpp
|
||||
explore_active_ = true;
|
||||
weightSnapshot_ = synthMapping->GetWeights(); // stash the REAL net
|
||||
learningPaused_ = true;
|
||||
randomiseTheNetwork(); // RandomiseWeightsAndBiasesLin(-0.9,1.1, -0.9,0.3); sets newInput
|
||||
```
|
||||
|
||||
`_cancel_explore()` (`InterfaceRL.cpp:179`): `SetWeights(weightSnapshot_)` (discard temp), resume learning, clear flags.
|
||||
|
||||
`_commit_explore()` → `storeExperience(1.f, controlInput, action)` then `_restore_after_explore()`. For MLP, `_restore_after_explore()` **restores the original net** (`SetWeights(weightSnapshot_)`) — the just-stored example then trains the *original* net toward the auditioned output. So the temp net is never kept as-is; instead the example it produced is grafted onto the real net via training.
|
||||
|
||||
The drag-store interaction (`_feedback_drag_store()`, `InterfaceRL.cpp:139`) is the only place the drag is more than a plain store:
|
||||
```cpp
|
||||
storeExperience(1.f, controlInput, savedAction);
|
||||
_refresh_mem_counts();
|
||||
if (feedbackMode_ == FEEDBACK_MODE::RANDOMISE_MLP && explore_active_) {
|
||||
_restore_after_explore(); // drag == reposition-commit
|
||||
}
|
||||
```
|
||||
|
||||
### 1.4 Forced teardown — `_abort_explore()` and mode switching
|
||||
|
||||
`setFeedbackMode(m)` (`InterfaceRL.hpp`): `if (explore_active_) _abort_explore(); feedbackMode_ = m;`. `_abort_explore()` is `_restore_after_explore()` minus the display calls: it restores the snapshot **only in MLP mode**, resumes learning, clears flags. This guarantees switching modes mid-exploration leaves the real net intact and learning resumed.
|
||||
|
||||
### 1.5 Focus-awareness summary (`activeDims_`)
|
||||
|
||||
`activeDims_` is a `std::vector<bool>` set by the mode (`setActiveDims`). Empty ⇒ all dims active. It gates three places:
|
||||
- AVOID geometric push: unfocused dims not pushed.
|
||||
- RANDOMISE_OUTPUTS roll: unfocused dims frozen at entry value.
|
||||
- (RANDOMISE_MLP has no per-dim focus — randomising weights affects all outputs; focus is only relevant via the output transform downstream.)
|
||||
|
||||
In TR-8S (`MEMLNautModeTR8S.hpp`) the mask is derived from a `FocusManager<kN_Params, 12>` (one group per drum voice + FX). This is **mode/UI policy**, not core logic.
|
||||
|
||||
---
|
||||
|
||||
## 2. Clean design for the new core component
|
||||
|
||||
### 2.1 Where the semantics diverge between old and new (must reconcile first)
|
||||
|
||||
| Concept | Old (`fork-feedback`) | New core (`nisps/ml/`) |
|
||||
|---------|-----------------------|------------------------|
|
||||
| MLP type | `MLP<float>` runtime-sized, heap (`std::vector`, `std::make_shared`) | `MLP<NIn,H1,H2,H3,NOut,...>` compile-time, all `std::array`, zero heap |
|
||||
| Weight snapshot | `MLP<float>::mlp_weights` (nested `std::vector`) | flat `std::span<const float>` from `get_weights()` (size `weight_count()`) |
|
||||
| RNG | global libc `rand()` | per-instance `nisps::Rng` (xoshiro256+), deterministic |
|
||||
| Randomise net | `RandomiseWeightsAndBiasesLin(-0.9,1.1,-0.9,0.3)` (asymmetric uniform) | `draw_weights(spread)` (spread-aware Xavier↔uniform) |
|
||||
| "Avoid" | geometric push toward/away centroid, deferred in `optimise()` | `move_weights(speed, spread, pin_mask)` = decay + Gaussian perturbation |
|
||||
| Output buffer | `std::vector<float> staticRandomOut_` runtime-sized | fixed `std::array<float, NOut>` |
|
||||
| Training pause | `learningPaused_` short-circuits `optimise()` | core has no training loop driver; pause must be a queryable flag the *caller's* train path honors |
|
||||
|
||||
**Recommendation for "Avoid" in the new core:** Do **not** port the k-NN geometric centroid push. It depends on `ReplayMemory<trainStatelessRLItem>` (rewards, decay, accumulation) which is firmware-only and absent from `nisps/`. The new playground already implements AVOID as **`move_weights(cap, spread, pinMask)` + noise growth** (`mode-runtime.ts::thumbsDown`). Define **`Avoid` in the new core as "delegate to the existing `move_weights` Gaussian perturbation."** This is the documented parity contract in `rl.hpp`. The geometric-centroid behavior is a separate, richer algorithm that should be filed as a follow-up if desired — it is NOT part of this port, and the spec should say so explicitly. The FeedbackController's job for AVOID is purely to **route** to `move_weights`; it owns no AVOID-specific state.
|
||||
|
||||
This keeps the controller small: it only owns state for the two RANDOMISE modes.
|
||||
|
||||
### 2.2 Proposed component: `nisps::ml::FeedbackController<MLP_T>`
|
||||
|
||||
A header-only class template parameterized on the concrete MLP type, living at **`/home/w1n5t0n/src/MEMLNaut-NISPS/nisps/ml/feedback.hpp`**, namespace `nisps::ml`. It does **not** own the MLP; every mutating method takes `MLP_T&`. It owns only the exploration state. No virtual dispatch, no heap, per-instance RNG (its own, seeded independently from the MLP's so the static-output stream is reproducible without perturbing inference noise).
|
||||
|
||||
```cpp
|
||||
// nisps/ml/feedback.hpp
|
||||
#pragma once
|
||||
#include <array>
|
||||
#include <cstdint>
|
||||
#include <span>
|
||||
#include "../core/perf.hpp"
|
||||
#include "../core/rng.hpp"
|
||||
|
||||
namespace nisps::ml {
|
||||
|
||||
enum class FeedbackMode : std::uint8_t {
|
||||
Avoid = 0, // down → move_weights (Gaussian perturb). No internal state.
|
||||
RandomiseOutputs= 1, // down → bypass MLP, hold static random vector; re-roll each down.
|
||||
RandomiseMlp = 2, // down → snapshot+draw_weights live net; down-again cancels.
|
||||
};
|
||||
|
||||
// What a down/up/drag press resolved to — the caller (JS runtime / firmware glue)
|
||||
// acts on this (store example, run move_weights, refresh UI). Keeps side effects
|
||||
// that touch replay memory / training OUT of the core.
|
||||
enum class FeedbackAction : std::uint8_t {
|
||||
None,
|
||||
AvoidPerturb, // caller: run mlp.move_weights(speed, spread, pin) + grow noise
|
||||
LikeStore, // caller: add +1 example at (input, output) + train
|
||||
EnterExplore, // entered a RANDOMISE_* exploration (UI: show "exploring")
|
||||
Reroll, // re-rolled within an exploration (UI feedback)
|
||||
CommitStore, // caller: add +1 example at (input, current output), THEN explore ended
|
||||
Cancel, // exploration discarded; net restored
|
||||
Restore, // exploration kept (drag reposition path); net restored
|
||||
};
|
||||
|
||||
template <typename MLP_T>
|
||||
class FeedbackController {
|
||||
public:
|
||||
static constexpr std::size_t kNOut = MLP_T::kOutput;
|
||||
static constexpr std::size_t kWeights = MLP_T::weight_count();
|
||||
|
||||
explicit FeedbackController(std::uint64_t seed) noexcept : rng_(seed) {}
|
||||
|
||||
void set_mode(FeedbackMode m, MLP_T& mlp) noexcept {
|
||||
if (explore_active_) abort_explore(mlp); // clean teardown on switch
|
||||
mode_ = m;
|
||||
}
|
||||
FeedbackMode mode() const noexcept { return mode_; }
|
||||
|
||||
bool exploring() const noexcept { return explore_active_; }
|
||||
bool learning_paused() const noexcept { return learning_paused_; }
|
||||
|
||||
// Focus mask: 1 byte per output; mask[i]==0 means "frozen" (unfocused).
|
||||
// Empty span ⇒ all active. Copied into a fixed buffer (no heap, no dangling).
|
||||
void set_focus_mask(std::span<const std::uint8_t> mask) noexcept {
|
||||
focus_count_ = (mask.size() < kNOut) ? mask.size() : kNOut;
|
||||
for (std::size_t i = 0; i < focus_count_; ++i) focus_[i] = mask[i];
|
||||
}
|
||||
void clear_focus_mask() noexcept { focus_count_ = 0; }
|
||||
|
||||
// --- press handlers. `current_out` is the live post-pipeline output the
|
||||
// user is hearing (NOut floats). They return a FeedbackAction telling the
|
||||
// caller what replay-memory / training side effect to perform. ---
|
||||
FeedbackAction on_down(MLP_T& mlp, std::span<const float> current_out,
|
||||
float speed, float spread,
|
||||
std::span<const std::uint8_t> pin_mask) noexcept;
|
||||
FeedbackAction on_up (MLP_T& mlp) noexcept;
|
||||
FeedbackAction on_drag(MLP_T& mlp) noexcept;
|
||||
|
||||
// Inference hook: returns true and fills `out` with the held static vector
|
||||
// when RandomiseOutputs is bypassing the MLP; returns false otherwise
|
||||
// (caller should run mlp.process() normally).
|
||||
bool static_output(std::span<float> out) const noexcept {
|
||||
if (!(mode_ == FeedbackMode::RandomiseOutputs && explore_active_)) return false;
|
||||
for (std::size_t i = 0; i < kNOut; ++i) out[i] = static_out_[i];
|
||||
return true;
|
||||
}
|
||||
|
||||
void seed(std::uint64_t s) noexcept { rng_.seed(s); }
|
||||
|
||||
private:
|
||||
void enter_randomise_outputs(std::span<const float> seed_out) noexcept {
|
||||
explore_active_ = true; learning_paused_ = true;
|
||||
for (std::size_t i = 0; i < kNOut; ++i) static_out_[i] = seed_out[i];
|
||||
roll_static_outputs();
|
||||
}
|
||||
void roll_static_outputs() noexcept {
|
||||
for (std::size_t i = 0; i < kNOut; ++i) {
|
||||
const bool active = (focus_count_ == 0u) || (i < focus_count_ && focus_[i] != 0u);
|
||||
if (active) static_out_[i] = rng_.next_float_uniform(); // [0,1) — see §2.4
|
||||
}
|
||||
}
|
||||
void enter_randomise_mlp(MLP_T& mlp, float spread) noexcept {
|
||||
explore_active_ = true; learning_paused_ = true;
|
||||
auto w = mlp.get_weights(); // flat snapshot
|
||||
for (std::size_t i = 0; i < kWeights; ++i) snapshot_[i] = w[i];
|
||||
mlp.draw_weights(spread); // randomise live net
|
||||
}
|
||||
void restore_after_explore(MLP_T& mlp) noexcept {
|
||||
if (mode_ == FeedbackMode::RandomiseMlp)
|
||||
mlp.set_weights(std::span<const float>(snapshot_.data(), kWeights));
|
||||
learning_paused_ = false; explore_active_ = false;
|
||||
}
|
||||
void cancel_explore(MLP_T& mlp) noexcept {
|
||||
if (mode_ == FeedbackMode::RandomiseMlp)
|
||||
mlp.set_weights(std::span<const float>(snapshot_.data(), kWeights));
|
||||
learning_paused_ = false; explore_active_ = false;
|
||||
}
|
||||
void abort_explore(MLP_T& mlp) noexcept { cancel_explore(mlp); }
|
||||
|
||||
FeedbackMode mode_ = FeedbackMode::Avoid;
|
||||
bool explore_active_ = false;
|
||||
bool learning_paused_ = false;
|
||||
std::array<float, kNOut> static_out_{};
|
||||
std::array<float, kWeights> snapshot_{}; // flat weight snapshot, no heap
|
||||
std::array<std::uint8_t, kNOut> focus_{};
|
||||
std::size_t focus_count_ = 0; // 0 ⇒ all active
|
||||
Rng rng_;
|
||||
};
|
||||
} // namespace nisps::ml
|
||||
```
|
||||
|
||||
The press handlers mirror the old `_feedback_up/_down/_drag_store` switch exactly:
|
||||
|
||||
```cpp
|
||||
template <typename M>
|
||||
FeedbackAction FeedbackController<M>::on_down(M& mlp, std::span<const float> current_out,
|
||||
float speed, float spread, std::span<const std::uint8_t> pin_mask) noexcept {
|
||||
switch (mode_) {
|
||||
case FeedbackMode::Avoid:
|
||||
mlp.move_weights(speed, spread, pin_mask); // AVOID == perturb (see §2.5)
|
||||
return FeedbackAction::AvoidPerturb;
|
||||
case FeedbackMode::RandomiseOutputs:
|
||||
if (!explore_active_) { enter_randomise_outputs(current_out); return FeedbackAction::EnterExplore; }
|
||||
roll_static_outputs(); return FeedbackAction::Reroll;
|
||||
case FeedbackMode::RandomiseMlp:
|
||||
if (!explore_active_) { enter_randomise_mlp(mlp, spread); return FeedbackAction::EnterExplore; }
|
||||
cancel_explore(mlp); return FeedbackAction::Cancel;
|
||||
}
|
||||
return FeedbackAction::None;
|
||||
}
|
||||
|
||||
template <typename M>
|
||||
FeedbackAction FeedbackController<M>::on_up(M& mlp) noexcept {
|
||||
if ((mode_ == FeedbackMode::RandomiseOutputs || mode_ == FeedbackMode::RandomiseMlp) && explore_active_) {
|
||||
restore_after_explore(mlp); // caller stores +1 at (input, current_out) FIRST
|
||||
return FeedbackAction::CommitStore;
|
||||
}
|
||||
return FeedbackAction::LikeStore; // AVOID up, or idle RANDOMISE up
|
||||
}
|
||||
|
||||
template <typename M>
|
||||
FeedbackAction FeedbackController<M>::on_drag(M& mlp) noexcept {
|
||||
if (mode_ == FeedbackMode::RandomiseMlp && explore_active_) {
|
||||
restore_after_explore(mlp);
|
||||
return FeedbackAction::Restore; // caller already stored the +1 from savedAction
|
||||
}
|
||||
return FeedbackAction::LikeStore; // plain drag-store
|
||||
}
|
||||
```
|
||||
|
||||
> Note the ordering contract: in the old `_commit_explore`, `storeExperience` runs **before** `_restore_after_explore` (so in MLP mode the example is captured from the temp net's output, then the original net is restored and trained). The new core encodes this as: `on_up` returns `CommitStore` and the **caller must add the example using the output it captured before calling `on_up`** (which is exactly the live `current_out`). Document this loudly at the call site.
|
||||
|
||||
### 2.3 Why a "FeedbackController" and not an MLP/engine extension
|
||||
|
||||
- **The MLP must stay a pure numeric kernel.** It already satisfies `MLEngine` (`concepts.hpp`) with no notion of "exploration", "focus", or "modes". Bolting feedback state onto `MLP` would (a) inflate `weight_count()`-sized snapshot storage onto every MLP instance even when unused, and (b) couple the parity/golden vectors to UI policy. Keep it separate.
|
||||
- **The controller is the natural home for the snapshot** because the snapshot size is `MLP_T::weight_count()` — known at compile time, so a `std::array` works and the no-heap rule is honored.
|
||||
- It composes: firmware modes and the WASM handle both already hold an MLP; they add one `FeedbackController` next to it.
|
||||
|
||||
### 2.4 RNG reconciliation
|
||||
|
||||
Old `_roll_static_outputs` uses `rand() & 0xFFFF / 65535.f` (libc, non-deterministic, global). New core uses the per-instance `Rng`. Two choices for the static-output stream:
|
||||
- **Recommended:** the controller owns its **own** `Rng rng_` seeded separately. Rationale: re-rolling outputs must not advance the MLP's inference/move RNG (that would make inference noise depend on how many times the user pressed "down"), and a separate stream makes the OUTPUTS path independently reproducible in tests. `rng_.next_float_uniform()` already returns `[0,1)` — semantically equivalent to the old `[0, 65535]/65535` up to the endpoint and resolution, well within parity tolerance because this value is *generated*, not *compared against the MLP*.
|
||||
- For RANDOMISE_MLP, `draw_weights(spread)` uses the **MLP's** RNG (correct — it's the net being randomised), matching how `nisps_ml_draw_weights` already advances the MLP stream.
|
||||
|
||||
### 2.5 AVOID reconciliation, restated as a decision
|
||||
|
||||
`move_weights` (Gaussian decay+perturb) and the old geometric centroid push are **different algorithms**. The new playground already shipped AVOID-as-`move_weights`. Therefore:
|
||||
- **`FeedbackController::Avoid` routes `on_down` → `mlp.move_weights(speed, spread, pin_mask)`** and returns `AvoidPerturb`. It carries no replay memory and no centroid math.
|
||||
- The richer geometric-push avoidance (with `ReplayMemory`, reward decay, k-NN centroid) is **explicitly out of scope** for this port; if wanted later it belongs in a separate replay-memory component, not the FeedbackController. Flag this in `ALIGNMENT.md` as an accepted divergence so a future session doesn't "rediscover" it as a bug.
|
||||
|
||||
---
|
||||
|
||||
## 3. The C++/JS boundary — what lives where
|
||||
|
||||
**Lives in C++ (`FeedbackController`, shared with firmware):**
|
||||
- `mode_` enum, `explore_active_`, `learning_paused_`.
|
||||
- `static_out_` buffer (`NOut` floats) + the re-roll formula + the bypass-inference hook.
|
||||
- `snapshot_` flat weights + snapshot/restore/cancel via `get_weights`/`set_weights`.
|
||||
- focus mask (`NOut` bytes) + the focus-gated roll.
|
||||
- The on_down/up/drag state-transition logic and the `FeedbackAction` it returns.
|
||||
|
||||
**Stays in the SolidJS runtime (inherently UI / orchestration):**
|
||||
- *Which input/output vector* to store as the example (depends on the pipeline-processed outputs, overrides, mic features) — `mode-runtime.ts::thumbsUp/thumbsDown` already computes these.
|
||||
- *When* to train, the LR, snapshot stack for undo (`autoSnapshot`), noise growth/decay (`exploration-store.ts`).
|
||||
- Building the `pin_mask` (`buildPinMask` from overrides + param pins).
|
||||
- Display strings, "Down Action" selector widget, toasts. (`RLView` in firmware is the analogous display — out of core scope.)
|
||||
- The replay-memory / example dataset itself stays caller-side (the JS `Dataset` + the MLP's own ring buffer).
|
||||
|
||||
**Max-sharing principle honored:** all the *fiddly state-machine logic* (the part with off-by-one re-roll/cancel/commit bugs) is shared C++; only data the core cannot know (pipeline outputs, pins, UI) stays in JS. The boundary is "the controller decides *what transition happened*; the caller decides *what to persist*."
|
||||
|
||||
The JS runtime keeps `learning_paused()` honored by gating its `trainOnCurrent()` and auto-explore tick: when paused, skip training (mirrors `optimiseSometimes()`'s early return).
|
||||
|
||||
---
|
||||
|
||||
## 4. WASM C API additions
|
||||
|
||||
Add to `nisps/wasm/bindings.cpp`. The `MLHandle` gains a `FeedbackController` next to its `DefaultMLP` (`FeedbackController<DefaultMLP>`), seeded off the same JS seed XOR a salt so its stream is independent of the MLP's:
|
||||
|
||||
```cpp
|
||||
// inside struct MLHandle:
|
||||
nisps::ml::FeedbackController<DefaultMLP> feedback;
|
||||
std::array<float, kDefaultOutputs> static_out_scratch{}; // for nisps_ml_feedback_static_output
|
||||
explicit MLHandle(std::uint64_t seed) noexcept
|
||||
: mlp(seed), feedback(seed ^ 0xFEEDBACC0DEull) {}
|
||||
```
|
||||
|
||||
New `extern "C"` functions (all `EMSCRIPTEN_KEEPALIVE`):
|
||||
|
||||
```cpp
|
||||
// Mode + state.
|
||||
void nisps_ml_feedback_set_mode(void* ml, int mode); // 0=Avoid 1=RandOut 2=RandMlp
|
||||
int nisps_ml_feedback_get_mode(void* ml);
|
||||
int nisps_ml_feedback_exploring(void* ml); // 0/1
|
||||
int nisps_ml_feedback_learning_paused(void* ml); // 0/1
|
||||
|
||||
// Focus mask (NOut bytes; pass nullptr/0 to clear).
|
||||
void nisps_ml_feedback_set_focus(void* ml, const uint8_t* mask, int n);
|
||||
|
||||
// Press handlers. `current_out` = NOut floats the user is hearing.
|
||||
// `pin_mask` may be null. Return value is the FeedbackAction enum (int) so JS
|
||||
// knows what to persist (store example / perturb / commit / cancel).
|
||||
int nisps_ml_feedback_down(void* ml, const float* current_out,
|
||||
float speed, float spread, const uint8_t* pin_mask);
|
||||
int nisps_ml_feedback_up(void* ml);
|
||||
int nisps_ml_feedback_drag(void* ml);
|
||||
|
||||
// Inference hook: if returns 1, `out` (NOut floats) holds the static bypass
|
||||
// vector and the caller should NOT call nisps_ml_process(); if 0, run process().
|
||||
int nisps_ml_feedback_static_output(void* ml, float* out);
|
||||
```
|
||||
|
||||
Representative implementation (matches the existing guard/cast style in `bindings.cpp`):
|
||||
|
||||
```cpp
|
||||
EMSCRIPTEN_KEEPALIVE
|
||||
int nisps_ml_feedback_down(void* ml, const float* current_out,
|
||||
float speed, float spread, const uint8_t* pin_mask) {
|
||||
if (!ml) return 0;
|
||||
auto* h = static_cast<MLHandle*>(ml);
|
||||
std::span<const float> out = current_out
|
||||
? std::span<const float>(current_out, kDefaultOutputs) : std::span<const float>{};
|
||||
std::span<const std::uint8_t> mask = pin_mask
|
||||
? std::span<const std::uint8_t>(pin_mask, kDefaultOutputs) : std::span<const std::uint8_t>{};
|
||||
return static_cast<int>(h->feedback.on_down(h->mlp, out, speed, spread, mask));
|
||||
}
|
||||
```
|
||||
|
||||
**TS FFI declarations** to add to `playground/src/ml/types.ts` (`NispsModule` interface, mirroring the existing `_nisps_ml_*` block):
|
||||
|
||||
```ts
|
||||
// Feedback / Down-Action.
|
||||
_nisps_ml_feedback_set_mode(ml: number, mode: number): void;
|
||||
_nisps_ml_feedback_get_mode(ml: number): number;
|
||||
_nisps_ml_feedback_exploring(ml: number): number;
|
||||
_nisps_ml_feedback_learning_paused(ml: number): number;
|
||||
_nisps_ml_feedback_set_focus(ml: number, mask_ptr: number, n: number): void;
|
||||
_nisps_ml_feedback_down(ml: number, current_out_ptr: number, speed: number, spread: number, pin_mask_ptr: number): number;
|
||||
_nisps_ml_feedback_up(ml: number): number;
|
||||
_nisps_ml_feedback_drag(ml: number): number;
|
||||
_nisps_ml_feedback_static_output(ml: number, out_ptr: number): number;
|
||||
```
|
||||
|
||||
Plus a `FeedbackAction` / `FeedbackMode` TS enum in `types.ts` mirroring the C++ enums (numeric values must match). `WasmIML` (`wasm-iml.ts`) gains thin wrappers (`feedbackDown(currentOut, speed, spread, pinMask)`, etc.) that reuse the existing `pinMaskBuf` and add a small `static_out` heap buffer; the parity `.mjs` shim adds `cwrap` bindings for the same names.
|
||||
|
||||
---
|
||||
|
||||
## 5. Test plan
|
||||
|
||||
### 5.1 Host ctest — new file `tests/cpp/test_mlp_feedback.cpp`
|
||||
|
||||
Register it in the `nisps_core_tests` executable in `nisps/CMakeLists.txt` (it links `nisps_core`, builds under `-Wall -Wextra -Werror -Wpedantic`). Use the existing harness (`NISPS_TEST` / `NISPS_EXPECT` / `NISPS_EXPECT_NEAR`) and a small MLP like the existing RL test (`MLP<2,4,4,4,6,8,32>`). Because the controller takes `MLP_T&`, instantiate `FeedbackController<SmallMLP>`.
|
||||
|
||||
Cases (one `NISPS_TEST` each):
|
||||
|
||||
1. **`feedback_avoid_routes_to_move_weights`** — mode Avoid; snapshot weights; `on_down(...)` returns `AvoidPerturb` and weights changed (most distinct), mirroring `mlp_move_weights_changes_unpinned_weights`. `exploring()` stays false; `learning_paused()` stays false.
|
||||
2. **`feedback_randout_enter_roll_state`** — mode RandomiseOutputs; `on_down` (1st) → `EnterExplore`, `exploring()==true`, `learning_paused()==true`; `static_output(buf)` returns true and buf differs from `current_out` on focused dims.
|
||||
3. **`feedback_randout_reroll_changes_focused_only`** — set focus mask `{1,0,1,0,1,0}`; enter; capture static; `on_down` again → `Reroll`; assert unfocused dims **unchanged** (frozen at seed value), focused dims **changed**.
|
||||
4. **`feedback_randout_commit_clears_state`** — while exploring, `on_up` → `CommitStore`, then `exploring()==false`, `learning_paused()==false`, `static_output()` returns false (MLP no longer bypassed). Weights unchanged (OUTPUTS never touches the net).
|
||||
5. **`feedback_randmlp_snapshot_and_cancel_restores`** — mode RandomiseMlp; snapshot weights `W0` (via `get_weights`); `on_down` (1st) → `EnterExplore`, weights now differ from `W0`; `on_down` again → `Cancel`; `get_weights()` byte-equals `W0` (exact `==`).
|
||||
6. **`feedback_randmlp_commit_restores_then_caller_trains`** — enter; `on_up` → `CommitStore`; assert net restored to `W0` (the kept example is the caller's responsibility; here we assert the restore semantics: post-commit weights == `W0`).
|
||||
7. **`feedback_randmlp_drag_repositions`** — enter; `on_drag` → `Restore`, net restored to `W0`, `exploring()==false`.
|
||||
8. **`feedback_mode_switch_aborts_explore`** — enter RandomiseMlp; `set_mode(Avoid, mlp)` → net restored to `W0`, `exploring()==false`, `learning_paused()==false`.
|
||||
9. **`feedback_determinism_fixed_seed`** — two `FeedbackController`s with the same seed, same MLP-seed, identical press sequence (enter RandomiseOutputs, two re-rolls) → `static_output` buffers byte-identical. Proves per-instance RNG determinism (no libc `rand()`).
|
||||
10. **`feedback_focus_empty_means_all_active`** — no focus mask set; enter RandomiseOutputs; all NOut dims changed from a fixed seed-vector of constants.
|
||||
11. **`feedback_static_output_bypass_only_in_randout`** — `static_output()` returns false in Avoid and in idle RandomiseOutputs, true only when `RandomiseOutputs && exploring`.
|
||||
|
||||
### 5.2 Native↔WASM parity
|
||||
|
||||
Extend the standalone parity runner in lock-step (`parity_check.cpp` + `parity_wasm.mjs`), appending a feedback block to the existing payload **after** the current stages (bump `kVersion` to 2 in both files and `parity_diff.mjs` if it checks version). Deterministic sequence, identical on both sides:
|
||||
|
||||
```
|
||||
// Stage 5 (feedback): with the SAME ParityMLP, after stage-2 training:
|
||||
// feedback.set_mode(RandomiseOutputs)
|
||||
// feedback.on_down(current_out = stage-2 outputs, speed=0.1, spread=0.5, pin=null) → EnterExplore
|
||||
// feedback.static_output(buf) → push 126 floats
|
||||
// feedback.on_down(... ) → Reroll
|
||||
// feedback.static_output(buf) → push 126 floats
|
||||
// feedback.set_mode(RandomiseMlp); feedback.on_down(...) (enter, draws weights)
|
||||
// push 12 probed weights of the now-randomised temp net
|
||||
// feedback.on_up() → CommitStore (restores)
|
||||
// push 12 probed weights of the restored net (must equal pre-explore probes)
|
||||
```
|
||||
|
||||
The controller must be seeded **identically** in `parity_check.cpp` and `parity_wasm.mjs` (e.g. seed `kSeed ^ 0xFEEDBACC0DE`, matching the `MLHandle` salt). `scripts/parity-check.sh` already float32-diffs at 1e-5 — the new floats are covered automatically. The RandomiseOutputs static vectors are RNG output (not compared against the MLP), so as long as both sides run the *same* `nisps::Rng` from the *same* seed they match exactly; the RandomiseMlp restore-probes prove `get_weights`/`set_weights` round-trips identically across native/WASM.
|
||||
|
||||
### 5.3 Existing suites
|
||||
|
||||
`nisps_core_tests`, `nisps_dsp_engine_tests`, `nisps_modes_tests`, `nisps_golden_tests` are unaffected (controller is additive). `bash scripts/run-all-tests.sh` (chokepoint E) gains the new ctest case and the extended parity blob. **No hardware** is exercised.
|
||||
|
||||
---
|
||||
|
||||
## 6. Firmware note — feasibility
|
||||
|
||||
The same `FeedbackController<MLP_T>` drops into a firmware mode unchanged, because it honors every rule in the RP2350 performance contract (`CLAUDE.md`):
|
||||
- **No heap:** `static_out_`, `snapshot_`, `focus_` are all `std::array` sized from `MLP_T::weight_count()` / `kOutput` at compile time. No `std::vector`, no `make_shared` (unlike the old `mlp_weights`).
|
||||
- **No virtual dispatch:** plain class template; press handlers are direct calls.
|
||||
- **Deterministic per-instance RNG:** uses `nisps::Rng`, constructed with a seed.
|
||||
- **`.f` literals / `NISPS_*` attrs:** the controller's hot path is trivial (a loop over `NOut`); mark `roll_static_outputs`/`static_output` `NISPS_FORCE_INLINE` and any audio-touched members `NISPS_AUDIO_MEM` if a firmware mode reads `static_out_` from the audio core (it would not — feedback runs on the control/UI core, like the old `loopCallback`).
|
||||
|
||||
Wiring in a future firmware mode (analogous to the TR-8S `bind_RL_interface` callbacks):
|
||||
- MomA1 (up) handler → set a `volatile pendingUp_` flag; control loop calls `controller.on_up(mlp)` and, on `CommitStore`/`LikeStore`, adds the example to whatever replay/dataset the firmware mode keeps.
|
||||
- MomA2 (down) → `pendingDown_` → `controller.on_down(mlp, current_out, speed, spread, pin)`.
|
||||
- Joystick-click / Tog-A1 release → `pendingDrag_` → `controller.on_drag(mlp)`.
|
||||
- The mode's per-sample/synth output path checks `controller.static_output(buf)`: if true, emit `buf` (bypass), else run the MLP — exactly the `generateAction()` branch at `InterfaceRL.cpp:998`.
|
||||
- The mode's training tick checks `controller.learning_paused()` and skips training when true (mirrors `optimiseSometimes()` early-return).
|
||||
- The "Down Action" `RotarySelectView` calls `controller.set_mode(m, mlp)` (the old `addFeedbackModeView`).
|
||||
|
||||
All handlers run in thread/control context (never ISR), identical to the original which dispatched everything via deferred flags in `loopCallback`. No new locking is needed beyond the mode's existing `mlpActive` spin-lock around `get_weights`/`set_weights`/`draw_weights` — the same lock the old code held around `optimise()`/`generateAction()`.
|
||||
|
||||
---
|
||||
|
||||
## 7. File-touch summary (implementation-ready)
|
||||
|
||||
| File | Change |
|
||||
|------|--------|
|
||||
| `/home/w1n5t0n/src/MEMLNaut-NISPS/nisps/ml/feedback.hpp` | **NEW** — `FeedbackMode`, `FeedbackAction`, `FeedbackController<MLP_T>` (as in §2.2). |
|
||||
| `/home/w1n5t0n/src/MEMLNaut-NISPS/nisps/wasm/bindings.cpp` | Add `FeedbackController<DefaultMLP>` + `static_out_scratch` to `MLHandle`; add 9 `nisps_ml_feedback_*` C functions (§4). |
|
||||
| `/home/w1n5t0n/src/MEMLNaut-NISPS/playground/src/ml/types.ts` | Add 9 `_nisps_ml_feedback_*` to `NispsModule`; add `FeedbackMode`/`FeedbackAction` enums. |
|
||||
| `/home/w1n5t0n/src/MEMLNaut-NISPS/playground/src/ml/wasm-iml.ts` | Thin wrapper methods + a small static-output heap buffer; reuse `pinMaskBuf`. |
|
||||
| `/home/w1n5t0n/src/MEMLNaut-NISPS/playground/src/modes/mode-runtime.ts` | Route `thumbsDown` through the controller per active mode; gate training on `learning_paused()`. |
|
||||
| `/home/w1n5t0n/src/MEMLNaut-NISPS/playground/src/stores/exploration-store.ts` | Add `feedbackMode` + `exploring` UI state (selector + indicator). |
|
||||
| `/home/w1n5t0n/src/MEMLNaut-NISPS/tests/cpp/test_mlp_feedback.cpp` | **NEW** — 11 ctest cases (§5.1). |
|
||||
| `/home/w1n5t0n/src/MEMLNaut-NISPS/nisps/CMakeLists.txt` | Add `test_mlp_feedback.cpp` to `nisps_core_tests`. |
|
||||
| `/home/w1n5t0n/src/MEMLNaut-NISPS/tests/cpp/parity_check.cpp` + `parity_wasm.mjs` | Append feedback Stage 5; bump version to 2 (§5.2). |
|
||||
|
||||
### Key reconciliation decisions to record in `ALIGNMENT.md`
|
||||
1. **AVOID** in the new core = `move_weights` Gaussian perturbation (the new playground's existing behavior), **not** the old k-NN geometric centroid push (which depended on firmware-only `ReplayMemory` and is out of scope). Accepted divergence.
|
||||
2. RANDOMISE_OUTPUTS static vector uses the controller's **own** `nisps::Rng` (deterministic, per-instance) instead of libc `rand()` — endpoint/resolution differ trivially from `[0,65535]/65535` but the value is generated, not compared.
|
||||
3. RANDOMISE_MLP randomisation uses `draw_weights(spread)` (spread-aware) instead of the old asymmetric `RandomiseWeightsAndBiasesLin(-0.9,1.1,-0.9,0.3)`. If the asymmetric distribution matters perceptually, that's a follow-up `draw_weights_range(lo,hi)` primitive — not part of this port.
|
||||
120
docs/redesign/findings-design-and-manifold.md
Normal file
120
docs/redesign/findings-design-and-manifold.md
Normal file
|
|
@ -0,0 +1,120 @@
|
|||
# Findings — Design + Manifold Console: build-oriented brief (Phase-1)
|
||||
|
||||
*Read-only synthesis, 2026-06-27, of the five redesign docs + the on-disk Manifold token export + the
|
||||
`ConsoleApp.jsx` read directly from the design project. VERIFIED = read from a file. INFER = deduced.*
|
||||
|
||||
> **Note on the mirror:** as of this audit `docs/redesign/manifold-export/` contains ONLY the six token CSS
|
||||
> files + `styles.css` manifest. The component/console JSX (ConsoleApp, CompositeStage, Dock, …) is in the
|
||||
> claude.ai design project and must be pulled via DesignSync before Phase-3 build. `ConsoleApp.jsx` was read
|
||||
> directly and is summarised in §2.
|
||||
|
||||
## Source map (`docs/redesign/`)
|
||||
- `playground-2.0-rewrite-plan.md` (399ln) — authoritative SolidJS UX+architecture plan: Console IA, drawers,
|
||||
reactive spine, feature table, roadmap, open questions.
|
||||
- `engine-architecture.md` (440ln) — one-engine-two-skins: `EngineApi` headless boundary, reuse table, COOP/COEP,
|
||||
build sequence S0–S7.
|
||||
- `aimmersive-clone-spec.md` (351ln) — faithful clone spec of deployed `a-immersive.html`: feature inventory,
|
||||
exact tactile constants, state→store map, engine call points.
|
||||
- `feedback-modes-port-spec.md` (548ln) — 3-mode "Down Action" ported to `nisps/` as `FeedbackController<MLP_T>`.
|
||||
- `playground-2026.md` (341ln) — older design-intent doc (dock+drawer, interactive heatmap); reference-only.
|
||||
|
||||
> **Framework mismatch:** every planning doc targets **SolidJS**; the mission is a **React** port. The §4 spine
|
||||
> translation is the largest design decision and biggest risk (§7).
|
||||
|
||||
## 1. Manifold design language (VERIFIED — token files)
|
||||
Pure CSS-custom-property layer; React port imports `styles.css` and references vars directly.
|
||||
- **Colors:** `--bg #0d0d0d` / `--bg-1..3`. `--fg #e8e8e8` / `--fg-mute` / `--fg-dim`. `--line`/`--line-strong`.
|
||||
Accents `--accent #ff6a00` (warm primary), `--accent-2 #00ccff` (cool data/plots), `--accent-3 #ffa860`
|
||||
(hover). Semantic `--good #6bc26b`/`--warn #f5c45e`/`--bad #ef5b5b`/`--info #5b9eef`. Console 2.0:
|
||||
`--danger #ff4466`, `--glass rgba(13,13,13,0.65)`, `--glass-line rgba(255,255,255,0.07)`. Region pins
|
||||
`--pin-1..5`. Glow alphas `--glow-accent/-2/-focus`. **VCV LED-ring palette derives from these** (workstream E).
|
||||
- **Spacing:** 4px scale `--sp-0..8`=2/4/8/12/16/24/32/48/64. Radius `--r-1 4`/`--r-2 8`/`--r-3 14`/`--r-pill`.
|
||||
Z `--z-bg/content/overlay/drawer/modal`. `--control-h 48px`, `--hit-min 44px`.
|
||||
- **Typography:** mono hero `--font-mono: 'JetBrains Mono',…`; sizes `--fs-xs 11`…`--fs-3xl 48` (body 15);
|
||||
letter-spacing `--ls-label 0.08em`/`--ls-wide 0.12em`; `--label-transform uppercase`.
|
||||
- **Effects:** *"glow halos, not drop shadows, on live elements"* (signature). Easings `--ease`,`--ease-out`,
|
||||
`--ease-console`; durations 120/220/360ms; `--glow-sm/md/lg`, `--focus-ring`.
|
||||
- **Takeaway:** dark, dense, mono-everywhere, orange-primary/cyan-data, glow-halos, uppercase letter-spaced
|
||||
labels, glass chrome over a full-bleed canvas. A lint allowlist of these vars is mandated to prevent drift.
|
||||
|
||||
## 2. Console shape (ConsoleApp.jsx VERIFIED; rest INFER)
|
||||
One fullscreen instrument. `focus = in|out|composite|split` drives the convertible stage:
|
||||
- `focus='in'` → **Manifold** full-bleed stage (input-first), optional top **ReadoutStrip**.
|
||||
- `focus='out'` → **OutputStage** + **InputMini** corner PiP (output-first).
|
||||
- `focus='composite'` → **CompositeStage** with a draggable split persisted to `localStorage['mf-composite-split']`;
|
||||
`[`/`]` nudge split ±0.04, `=`/`0` reset to 0.5.
|
||||
- `focus='split'` → **SplitStage**.
|
||||
- Keyboard: `1-5`→drawers (shape/feel/route/health/help), `\` toggles drawer depth peek↔full, space/↑=commit,
|
||||
↓=perturb, z=undo.
|
||||
- **Dock** (right 48px rail) with peek/full drawers. **VerdictCluster** (perturb/undo/commit/reroll + A/B
|
||||
hold-preview). **AltitudeNav** console|perform|zen. Health glow at screen edge from a `health` signal.
|
||||
- **Reactive spine (pseudo today):** `infer(pos, seed, params, axes)` = `MF_infer` placeholder in `model.jsx`;
|
||||
`values` memo every consumer reads. **Replace `MF_infer` with WasmIML** — that is the core wiring task.
|
||||
|
||||
> The updated mission's CONVERTIBLE spec (§1): one continuous view, a single draggable divider where each panel
|
||||
> demotes its representation by measured width (full manifold → pad → minimap) and a fully-shut side pops out as
|
||||
> a draggable corner minimap. This refines `CompositeStage`/`OutputStage`/`InputMini` — build to the mission
|
||||
> spec, using the JSX as the structural starting point.
|
||||
|
||||
## 3. Component inventory (port to React)
|
||||
**Primitives** (12 in the design system, VERIFIED present in the cloud project): Button, Slider, PillToggle,
|
||||
Panel, Badge, Switch, StatusLine, XYPad, VirtualJoystick, ControlAxis, CurvePlot, Sparkline. Plus a-immersive
|
||||
craft pieces to add: Heatmap/HeatmapCell (3px drag threshold, pointer-capture, 300ms popup grace — non-negotiable),
|
||||
NoiseRing, LossPlot (real history → needs new `nisps_ml_loss_history` C API), GradientFlow, WeightHealth,
|
||||
LayerStats, CurvePad, DualRangeSlider.
|
||||
**Console shell:** ConsoleApp, Manifold, CompositeStage, SplitStage, OutputStage, InputMini, ReadoutStrip,
|
||||
VerdictCluster, Dock, Drawers, OutputEditor, AltitudeNav, PerformApp, ZenApp, model.jsx, shared-ui.jsx.
|
||||
**Schema-driven:** one `GenericMode` reading `ui.primary_input`, `params[].{group,curve,tier}`,
|
||||
`capability_class`, `output_kind` — replaces the 8-9 cloned mode files.
|
||||
|
||||
## 4. Reactive spine → React (load-bearing translation)
|
||||
SolidJS spine: `inputRaw (signal {equals:false}) → createMemo(processed: deadzone→zoom→curve→smoothing→momentum)
|
||||
→ createMemo(ml: WasmIML.infer into reused buffer) → createMemo(routed: voice-space + global
|
||||
curve→smoothing→slew→freeze) → ONE createEffect(backend.send)`. Every consumer reads `ml`/`routed`.
|
||||
|
||||
**React translation (recommended):** do NOT use `useMemo`/`useEffect` for the spine — they recompute on render
|
||||
and per-frame audio inference must not couple to React's scheduler.
|
||||
1. **Spine lives below React** in a tiny external store (Zustand / `useSyncExternalStore` / hand-rolled
|
||||
observable) holding `inputRaw`; derive `processed→ml→routed` **eagerly+synchronously inside the `setInput`
|
||||
action**; fire the single `backend.send` at the action tail (off-render).
|
||||
2. **Expose `EngineApi` via React Context** (`EngineProvider`/`useEngine`); components call `engine.setInput`/
|
||||
`engine.feedback.thumbsUp()` and **subscribe** via `useSyncExternalStore(subscribe, () => versionCounter)`
|
||||
(version counter, not the array); read the live `Float32Array` imperatively.
|
||||
3. **Canvas consumers (Manifold/ReadoutStrip/particle-visualiser) bypass React** — read `engine.routedOutput()`
|
||||
in one `requestAnimationFrame` loop; rAF touches drawing only, never inference.
|
||||
4. **Pointer-rate coalescing** stays in the engine (batch to display cadence via microtask).
|
||||
**E2E invariant (port verbatim):** `__nisps.setInputs([x,y]); expect(getOutputs()).toChange() &&
|
||||
expect(getEngineParams()).toChange()` in one tick, per mode in CI; + a no-per-frame-alloc heap fuzz.
|
||||
|
||||
## 5. a-immersive feature-parity checklist
|
||||
3 compound axes Boldness/Memory/Precision (per-axis table → ~6 params), trim-pot offsets + re-link, 6 control
|
||||
presets (Default/First Touch/Jazz Hands/Sculptor/Improviser/Microscope), snapshots→snapshot-DAG (unifies
|
||||
undo/A-B/trail), A/B compare + Freeze Output, region pins (long-press joymap, `--pin-1..5`), param pins (pin
|
||||
mask ≠ mute), interactive heatmap, weight health (edge glow), gradient flow, output pipeline (reuse buffer, no
|
||||
per-frame alloc), session presets, input pipeline (deadzone→zoom→curve→smoothing→momentum), zoom (log/anchor),
|
||||
joy-map+adaptive grid, vanishing trail+tap-to-return, noise rings, layer-stats, **real** loss plot,
|
||||
auto-explore+follow, spread/tame Health lab, WebMIDI+OSC-bridge backends, **3-mode→2-mode Down Action** via
|
||||
`FeedbackController`. **Research-validity fixes:** train on raw model-space outputs not post-pipeline; real
|
||||
loss history; no phantom input channels; heap-safe WASM.
|
||||
|
||||
## 6. COOP/COEP + Vite + React + TS scaffolding
|
||||
COOP `same-origin` + COEP `require-corp` (dev `server.headers`, `preview.headers`, prod server-scoped on the
|
||||
nginx vhost). `@vitejs/plugin-react`; `base:'./'` + **base-aware WASM URLs** (`import.meta.env.BASE_URL`, not
|
||||
hardcoded `/nisps.wasm`); `build.target es2022`, sourcemap. Worklet via **`?worker&url`** (verified fix, commit
|
||||
`f256217`); two-WASM architecture kept. Strict TS, `tsc --noEmit && vite build`, `@playwright/test`. Assets
|
||||
`public/nisps.{wasm,js}` from `scripts/build-wasm.sh`. Serve new app at `/next/` sub-path, legacy a-immersive
|
||||
stays at `/` until sign-off; SPA fallback `try_files`.
|
||||
|
||||
## 7. Open risks
|
||||
1. **Framework mismatch (biggest)** — keep the spine off React's render cycle or live-feedback rot returns.
|
||||
Lint-enforce "skins may not import engine internals; engine may not import React".
|
||||
2. **Console JSX must be pulled** before committing component contracts; `focus`/`zen` vs the plan's
|
||||
depth/auto-dissolve vocabulary must be reconciled.
|
||||
3. **Fixed-2-input contraction** vs the modular N×M mission requirement (F): hardcoded `MLP<2,10,14,18,126>`;
|
||||
runtime-shaped MLP deferred behind a passing parity check — confirm scope with Dimi.
|
||||
4. **Tactile-constant fidelity** (3px threshold, 300ms grace, pointer-capture, double-tap 350ms) — port exactly,
|
||||
e2e-test.
|
||||
5. **Parity drift on core touches** — new FeedbackController C API + real loss-history C API + heap-safe vec all
|
||||
touch the parity-tested core; gate every core-touching step on `parity-check.sh` green.
|
||||
6. **Two preset systems** (visual/RL examples vs synth override tiers) — keep distinct.
|
||||
7. **Particle visualiser** must be a faithful port of `js/ui/visualizer.js` (workstream E) — exact look+behaviour.
|
||||
82
docs/redesign/findings-engine-surface.md
Normal file
82
docs/redesign/findings-engine-surface.md
Normal file
|
|
@ -0,0 +1,82 @@
|
|||
# Findings — Engine Surface Audit (Phase-1)
|
||||
|
||||
*Read-only audit, 2026-06-27. VERIFIED from source unless `[INFER]`. Engine A = deployed vanilla JS live at
|
||||
meml.lnfinitemonkeys.org (`/home/w1n5t0n/deployments/meml-aimmersive/js/`, a snapshot — repo source has
|
||||
diverged). Engine B = parity-tested TS playground in this repo (`playground/src/`).*
|
||||
|
||||
## Recommendation: wire the new React app to Engine B (the TS playground stack)
|
||||
Confirmed by the operator's directive ("training must use the same core the firmware builds from"): Engine B's
|
||||
`nisps.wasm` is compiled from the `nisps/` C++20 core and **parity-locked to native within 1e-5**
|
||||
(`scripts/parity-check.sh`, `tests/cpp/parity_*`). Engine A's deployed WASM exports an **older C ABI**
|
||||
(`nisps_mlp_*`) the repo no longer builds (`nisps_ml_*` + `nisps_engine_*`). Lift Engine B, don't fork.
|
||||
|
||||
## Executive summary
|
||||
1. Engine B is the *current* codebase; Engine A's WASM is a stale ABI snapshot.
|
||||
2. Engine B parity-tested (native==WASM ≤1e-5); A has no such guarantee.
|
||||
3. Engine B is fully typed TS as ES modules (`WasmIML`, `EngineHost`); A is clean ESM JS but untyped.
|
||||
4. **Decisive:** in Engine B, `nisps.wasm` produces the *audio itself* (`_nisps_engine_process_block` in the
|
||||
worklet) — true firmware parity. In Engine A, `nisps.wasm` is MLP-only; audio is separate JS/Faust/C15.
|
||||
5. So "reuse the MLP" via A still leaves audio to reimplement; via B you reuse MLP **and** the 8 audio engines
|
||||
from one WASM.
|
||||
6. Engine B's worklet is clean: no Emscripten glue, raw `WebAssembly.compile`, 128-sample blocks, per-thread
|
||||
instance, no SharedArrayBuffer for nisps audio.
|
||||
7. Engine B exposes the richer ML surface (feedback modes C API, layer stats, RL move/pin, batch infer) and a
|
||||
synchronous Playwright probe `window.__nisps`.
|
||||
8. Caveats (none blocking): fixed MLP arch (2→126), single-element `lossHistory` in sync train, mic input not
|
||||
wired, Solid coupling in `WasmIML` must be abstracted for React.
|
||||
|
||||
## Engine A — deployed vanilla JS (LIVE)
|
||||
WASM at `meml-aimmersive/wasm/nisps.{js,wasm}`. `WasmIML` (`js/nisps/nisps-wasm.js`, 612L):
|
||||
`create(nInputs,nOutputs,hiddenLayers=[10,10,14],…)`, `createWithWarmStart(snapshot,newOutputCount,…)` (transfers
|
||||
hidden weights across output-count changes, used live), `process()` sync, `inferBatch`, dataset add/clear,
|
||||
`train(options)` sync **(captures real per-iter loss history)**, `trainAsync(onComplete)` worker, `evalLoss`,
|
||||
`randomiseWeights(spread)`, `moveWeights(speed,spread,outputPinMask)` (pin mask supported), weights get/set,
|
||||
`getLayerStats()`→{meanAbs,maxAbs,deadFrac,satFrac}. **No feedback-mode API.** Emscripten MODULARIZE glue +
|
||||
`cwrap`. Exported ABI: `nisps_mlp_{create,destroy,draw_weights_spread,eval_loss,get_layer_stats,get_weights,
|
||||
infer_batch,inference,move_weights_ex,move_weights_spread,set_weights,train,train_ex,weight_count}` — **MLP
|
||||
only, no audio**. Audio is separate `SynthEngine` impls: C15 (`c15-bridge.js`, own `c15_engine.wasm` +
|
||||
worklet + SAB ring), Faust, JS `modular-engine.js`. Data path: input → `WasmIML.process()` → `outputs[]` →
|
||||
`param-map.js` (`applyCurve`,`applyGroupOverride`,`SYNTH_PARAM_MAP`) → `engine.setParam` → audio. Live arch
|
||||
bigger than expected: hidden `[32,48,64]`/`[48,48,64]`, warm-start across output counts. COOP/COEP needed only
|
||||
by the C15 path; the nisps MLP path needs no SAB.
|
||||
|
||||
## Engine B — parity-tested TS playground (in repo)
|
||||
Files: `ml/wasm-iml.ts` (650), `audio/engine-host.ts` (232), `audio/worklet/nisps-processor.ts` (309),
|
||||
`input/pipeline.ts` (307), `output/pipeline.ts` (153), `output/curves.ts` (128), `debug/probe.ts`.
|
||||
`WasmIML`: `create(opts)` — arch **resolved from the WASM build** via `_nisps_ml_describe` (caller sizes
|
||||
warned+ignored); fixed **2→126**, hidden `[10,14,18]`, 4 layers. `setInput`, `process():Float32Array`
|
||||
(copies out + pushes to `mlStore`), `inferXY`, `inferBatch` (chunked at 4096), dataset add/clear, `train`
|
||||
sync, `trainAsync` worker, `evalLoss`, `randomiseWeights`, `moveWeights(speed,spread,pinMask)`, weights
|
||||
get/set, `getLayerStats()`→{meanAbs,maxAbs,deadFrac,saturatingFrac}, `getLayerStatsFlat`, `dispose`, `reset`,
|
||||
debounced `saveNow()`, `tryLoadFromStorage_`. **Richer ABI** (`nisps/wasm/bindings.cpp`): all of A's MLP ops
|
||||
PLUS `nisps_ml_feedback_{up,down,drag,exploring,set_focus,set_mode,get_mode,static_output,learning_paused}`,
|
||||
`nisps_ml_describe`, AND audio `nisps_engine_{create,destroy,set_params,process_block}`.
|
||||
|
||||
**Audio VIA nisps.wasm (key difference):** `EngineHost` lazy-creates `AudioContext`, registers worklet via
|
||||
Vite `?worker&url`, fetches `nisps.wasm` on main thread, transfers bytes to worklet. `NispsProcessor` holds a
|
||||
2nd nisps.wasm instance compiled **without Emscripten glue** (raw `WebAssembly.compile` + import-shape
|
||||
discovery); per 128-sample block calls `_nisps_engine_process_block(handle,inL,inR,outL,outR,128)`. Engine
|
||||
switch = `port` message → `_nisps_engine_create` with new ASCII id. Data path: input pipeline
|
||||
(invert→deadzone→circular clamp→zoom→curve→EMA→momentum, pure fn) → `WasmIML.inferXY` → 126-vec → output
|
||||
pipeline (global curve→EMA→slew→freeze) → `EngineHost.setParams(Float32Array)` → worklet. **Same C++ engines
|
||||
as firmware.** COOP/COEP set in `vite.config.ts` (`same-origin`/`require-corp`); nisps audio needs no SAB
|
||||
(separate per-thread instances); SAB only for the browser-only C15 path. Three isolated nisps.wasm instances
|
||||
(main infer/sync-train, worker async-train, worklet audio). Debug probe `window.__nisps` (`debug/probe.ts`):
|
||||
synchronous, bypasses Solid via `untrack`/`batch`; never throws.
|
||||
|
||||
## Gaps to budget for (none blocking)
|
||||
- **Fixed MLP arch** (2→126, hidden [10,14,18]) baked into WASM; the modular N×M requirement (workstream F)
|
||||
needs either multiple WASM modules or runtime-shaped MLP (deferred behind a passing parity check). Engine A's
|
||||
configurable layers + warm-start is the reference for how to add this.
|
||||
- **`lossHistory` single-element on sync `train()`** (`wasm-iml.ts:419`); `trainAsync` returns a real array;
|
||||
C API lacks `nisps_ml_loss_history`. Port A's per-iter capture if loss curves matter.
|
||||
- **Mic input not wired** through the worklet (XIASRI / SoundAnalysisMIDI) — UI scaffolds only.
|
||||
- **C15** is browser-only, own WASM + SAB; voice space a placeholder. (UI label: "Powerful Synth Engine".)
|
||||
- **Solid coupling** in `WasmIML`/probe (`mlStore.__set*`, `coreBus.emit`) must become an injected
|
||||
callback/emitter to make it framework-neutral for React.
|
||||
|
||||
## Reuse plan for the React app
|
||||
Import `WasmIML` for the MLP, `EngineHost` + the worklet for audio, lift `input/pipeline.ts`,
|
||||
`output/pipeline.ts`, `output/curves.ts` as pure functions. Replace the Solid `mlStore`/`coreBus` side-effects
|
||||
with an injected emitter so the engine is framework-neutral, then mount it under React via context (see
|
||||
`findings-design-and-manifold.md` §4 reactive spine).
|
||||
158
docs/redesign/findings-feedback-behaviour.md
Normal file
158
docs/redesign/findings-feedback-behaviour.md
Normal file
|
|
@ -0,0 +1,158 @@
|
|||
# Findings — Current RL / Feedback Behaviour (Phase-1 Audit)
|
||||
|
||||
*Read-only audit, 2026-06-27. Citations are `file:line`. "VERIFIED" = read in source; "INFER" = deduced.
|
||||
Produced by the Phase-1 research fan-out (workflow `wf_db146f21`), persisted by the orchestrator because a
|
||||
hook blocks subagent report-file writes. Adversarially spot-checked against source.*
|
||||
|
||||
## Executive summary
|
||||
|
||||
1. The operator's mental model ("negative feedback takes the n-D opposite vector and pushes weights the other
|
||||
way — geometric push-away") describes the **firmware `InterfaceRL`** path, NOT what is deployed in the
|
||||
browser/aimmersive build.
|
||||
2. The **deployed JS engine** and the **`nisps/` C++ core** both implement thumbs-down as
|
||||
`move_weights(speed, spread, pinMask)` = per-weight decay (`w *= 1-0.1·spread`) then additive Gaussian
|
||||
noise (`stddev = speed · layer_scale`). There is **no opposite vector, no centroid, no geometric direction**
|
||||
in deployed/core code (VERIFIED: grep for `centroid|geometric|meanPositive|neg_action` returns nothing in
|
||||
`deployments/meml-aimmersive/js/`).
|
||||
3. So "push the weights the other way" is, in the deployed system, actually **"jitter the whole network with
|
||||
random Gaussian noise"** — undirected diffusion, not directed avoidance.
|
||||
4. The true geometric push-away (move output away from k-NN centroid of liked examples) exists only in the
|
||||
firmware submodule (`memllib` InterfaceRL; "geo push" commits) and is documented in
|
||||
`docs/redesign/feedback-modes-port-spec.md` §1.1. **This is the algorithm Mode 1 must match** (workstream B).
|
||||
5. **Why often inaudible (deployed AVOID):** noise added across all 4 layers of `MLP<2,10,14,18,126>`,
|
||||
per-layer scaled `(1-spread)+spread/√fan_in`, capped at `noiseCap=0.12`, spread default 0.6 → tiny
|
||||
per-step perturbation that the sigmoid output + output smoothing/slew pipeline absorb. One press moves the
|
||||
heard 126-vector very little, in a random direction uncorrelated with what was disliked.
|
||||
6. **Three FEEDBACK_MODEs already scaffolded** in untracked working files `nisps/ml/feedback.hpp` +
|
||||
`tests/cpp/test_mlp_feedback.cpp` (`FeedbackController<MLP_T>`): `Avoid`, `RandomiseOutputs`,
|
||||
`RandomiseMlp`. Not yet wired into bindings/TS/UI; no CMake registration.
|
||||
7. `RandomiseMlp` = the operator's Mode 2 as an audition: snapshot weights → `draw_weights(spread)` → user
|
||||
moves joystick to hear → up commits a +1 example, down/switch restores the snapshot. **This is the seed of
|
||||
Mode 2** but the updated spec is stricter (scratchpad NEVER trained; explicit place-at-location; warm-start
|
||||
real model to interpolate anchors).
|
||||
8. `RandomiseOutputs` = bypass the MLP entirely, hold a static random output vector (focus-aware per-dim),
|
||||
re-roll on each down; up commits the held vector as a +1 example.
|
||||
9. The operator's "randomise → anchor → train" alternative maps onto `RandomiseMlp` (randomise) + thumbs-up
|
||||
`LikeStore` (anchor) + `trainOnCurrent` (train). Missing in deployed build: the randomise-audition-then-anchor
|
||||
loop; today thumbs-down only diffuses noise and never anchors.
|
||||
10. The full port spec (component, WASM C API, TS FFI, ~22 ctest cases, parity stage) already exists at
|
||||
`docs/redesign/feedback-modes-port-spec.md` and matches the untracked `feedback.hpp`.
|
||||
11. **Parity/perf constraints:** no heap / `std::array` only, per-instance deterministic `nisps::Rng`
|
||||
(no libc `rand()`), no virtual dispatch, `.f` literals, fixed WASM arch `MLP<2,10,14,18,126>`,
|
||||
native↔WASM parity within `1e-5`.
|
||||
12. **Bottom line:** deployed negative-feedback is undirected Gaussian noise (not geometric push-away), which is
|
||||
why it is weak/inaudible; the in-progress `FeedbackController` already implements the seed of the operator's
|
||||
randomise-anchor-train idea but is unplumbed past the header.
|
||||
|
||||
## 1. Current algorithm, precisely, per layer
|
||||
|
||||
### 1.1 DEPLOYED (browser/aimmersive) — NOT geometric
|
||||
Thumbs-down = `move_weights`. Pure-JS reference `deployments/meml-aimmersive/js/nisps/mlp.js:238-265`
|
||||
(VERIFIED): per layer/node/weight `j`: `decay = 1-0.1*spread`; `layerScale = (1-spread)+(1/√fanIn)*spread`;
|
||||
`node.weights[j] *= decay`; `accum = Σ_{n=0..2}(rand*2-1)`; `node.weights[j] += 3*accum*speed*layerScale`.
|
||||
Output-layer nodes with `outputPinMask[ni]` skipped (`mlp.js:248`). WASM wrapper `nisps-wasm.js:389-406` →
|
||||
`cwrap('nisps_mlp_move_weights_ex')`. IML `iml.js:166-170` forwards then re-infers — **no dislike-direction
|
||||
logic, no example stored on dislike**.
|
||||
|
||||
**Key finding (VERIFIED, exhaustive grep):** `deployments/meml-aimmersive/js/` has zero matches for
|
||||
`centroid|meanPositive|geometric|pushStep|dislikeMultiplier|neg_action|RANDOMISE|FEEDBACK_MODE|explore_active`.
|
||||
Deployed negative feedback is purely undirected Gaussian weight diffusion. The disliked 126-vector is never read.
|
||||
|
||||
### 1.2 `nisps/` C++ core — same move_weights, no geometric push
|
||||
`nisps/ml/rl.hpp:63-100` `move_weights_layer` (VERIFIED): `layer_scale = spread_scale(spread,fan_in) =
|
||||
(1-spread)+spread/√fan_in` (`init.hpp:31-35`); `noise_stddev = speed*layer_scale`; `decay = 1-0.1*spread`
|
||||
(`rl.hpp:49-51`). Bias: `biases[node] += rng.next_float_gaussian(noise_stddev)` unless pinned (`rl.hpp:83`).
|
||||
Each weight: `w = w*decay + rng.next_float_gaussian(noise_stddev)` (`rl.hpp:92-97`); RNG advanced even on skip.
|
||||
Pin mask only on `is_final_layer` (`rl.hpp:78-79`). `mlp.hpp:312-322` iterates 4 layers, only layer3
|
||||
final+masked. `draw_weights` (`mlp.hpp:324-332`→`init.hpp:42-54`): `w = rng.next_float_signed()*spread_scale`,
|
||||
biases zeroed. C API `nisps_ml_move_weights`/`nisps_ml_draw_weights` (`bindings.cpp:419-437`). TS
|
||||
`wasm-iml.ts:495-508`. Handler `mode-runtime.ts:548-565`: builds pin mask, `moveWeights(cap,spread,pinMask)`,
|
||||
`growNoise`, re-`setInput`. **No example stored on thumbs-down** (vs thumbs-up `mode-runtime.ts:523-546`).
|
||||
|
||||
### 1.3 TRUE geometric push-away (firmware only) — THE MODE-1 TARGET
|
||||
Per `feedback-modes-port-spec.md:61-79` (sourced from firmware `InterfaceRL.cpp`), applied in `optimise()` not
|
||||
at press time: `pushStep = clamp(|avgRewardNeg|,0.25,1.0)*0.5`; `dir[j] = neg_action[j] -
|
||||
meanPositiveAction[j]` (away from k-NN(+) centroid); `target[j] = clamp(neg_action[j] +
|
||||
(dir/||dir||)[j]*pushStep/(1+||dir||), 0, 1)`; then train toward `target`. Depends on firmware-only
|
||||
`ReplayMemory`. Lives in `src/memllib` submodule (commits `2429bcc` "geo push", `d301cc7` "rlview and dislike";
|
||||
upstream `SB2026`; memllib main `e291192`/`abe93ec`). **Workstream A must confirm the newest variant.**
|
||||
|
||||
## 2. FEEDBACK_MODEs implemented / scaffolded, and where
|
||||
|
||||
| Mode | Firmware memllib | nisps/ core (untracked) | Deployed JS | Playground TS |
|
||||
|---|---|---|---|---|
|
||||
| AVOID geometric push | VERIFIED (geo-push commits) | NOT ported | — | — |
|
||||
| AVOID = move_weights | — | `feedback.hpp:103-105` | `mlp.js`/`nisps-wasm.js` | `mode-runtime.ts:548` |
|
||||
| RANDOMISE_OUTPUTS | VERIFIED (spec §1.2) | `feedback.hpp:106-112,160-181` | — | — |
|
||||
| RANDOMISE_MLP | VERIFIED (spec §1.3) | `feedback.hpp:113-119,183-189` | — | — |
|
||||
|
||||
**Untracked `nisps/ml/feedback.hpp`** (214 lines, VERIFIED): `nisps::ml::FeedbackController<MLP_T>`.
|
||||
`enum FeedbackMode {Avoid=0,RandomiseOutputs=1,RandomiseMlp=2}` (`:47-51`); `enum FeedbackAction
|
||||
{None,AvoidPerturb,LikeStore,EnterExplore,Reroll,CommitStore,Cancel,Restore}` (`:56-65`). State all
|
||||
`std::array` (no heap), own `Rng` seeded separately (`:203-210`). `on_down`: Avoid→move_weights;
|
||||
RandOut→enter/roll; RandMlp→enter(snapshot+draw_weights)/cancel. `on_up`: exploring→restore+CommitStore else
|
||||
LikeStore. `on_drag`: RandMlp+exploring→restore+Restore else LikeStore. `static_output`: bypass hook only when
|
||||
RandomiseOutputs&&exploring. `roll_static_outputs`: focused→`rng_.next_float_uniform()`, unfocused frozen.
|
||||
`set_mode` aborts active exploration first.
|
||||
|
||||
**Untracked `tests/cpp/test_mlp_feedback.cpp`** (437 lines, ~22 NISPS_TEST, VERIFIED):
|
||||
`SmallMLP=MLP<2,4,4,4,6,8,32>`; covers all transitions, RNG determinism, focus edge cases, a golden RNG
|
||||
stream. **Not registered in `nisps/CMakeLists.txt`** (spec §5.1/§7 lists as TODO).
|
||||
|
||||
**Not yet wired (gaps):** no `nisps_ml_feedback_*` in `bindings.cpp`; no `FeedbackController` on `MLHandle`;
|
||||
no TS FFI in `types.ts`; no UI selector. Firmware parent branch `origin/feat/feedback-explore-modes` wires a
|
||||
TR-8S FEEDBACK_MODE selector (`5c2d4d2`) and adds `docs/dislike_system_{analysis,design_space}.md` (4 meanings
|
||||
of "dislike"; here/anywhere × point/radius scope matrix — VERIFIED).
|
||||
|
||||
## 3. Spread / noise / zoom machinery
|
||||
- **spread∈[0,1]** (`exploration-store.ts:35,74`, default 0.6): feeds per-layer scale and decay. spread=0:
|
||||
uniform weights, big noise, no decay (saturated); spread=1: Xavier, small noise, 10% decay (spread out).
|
||||
- **noiseCap∈[floor,1]** (default 0.12): the `speed`/`cap` magnitude for thumbs-down and auto-explore.
|
||||
- **growNoise/decayNoise** driven by pointer `pressureForce` (`mode-runtime.ts:491-499,544,562`).
|
||||
- **Zoom** scales auto-explore intensity: `scaledIntensity = intensity*(0.3+0.7*zoom)` — an input-pipeline
|
||||
gain, not output magnitude.
|
||||
- **Auto-explore loop** (`mode-runtime.ts:444-482`): periodic `move_weights`+`growNoise`, default 2000ms, off
|
||||
by default — same diffusion as thumbs-down.
|
||||
- **Pin mask** `buildPinMask(...)`: freezes final-layer weights for pinned/overridden params — the only
|
||||
directional gating today (protects dims, doesn't push them). *Relevant to the SOLO/arm design (B).*
|
||||
|
||||
## 4. Parity/perf constraints any core change must respect
|
||||
No heap in `nisps/` (`std::array`/`FixedBuffer`; `feedback.hpp` obeys); deterministic per-instance `nisps::Rng`,
|
||||
no libc `rand()`; no virtual dispatch; `.f` literals, no double promotion, `NISPS_*` mem attrs; fixed WASM arch
|
||||
`MLP<2,10,14,18,126>` (snapshot = `weight_count()` floats, compile-time fixed); native↔WASM parity within
|
||||
`1e-5`. RandomiseMlp snapshot/restore must round-trip `get_weights`/`set_weights` byte-identically;
|
||||
RandomiseOutputs static vectors need same-seed/same-Rng; golden tests pin RNG streams.
|
||||
|
||||
## 5. WHY the effect is inaudible — the mechanism
|
||||
1. **Undirected, not opposite.** Zero-mean Gaussian noise on every weight in all 4 layers; the disliked output
|
||||
is never read. Expected output change ≈ 0; only variance moves — wobbles randomly, doesn't move *away*.
|
||||
2. **Small magnitude, multiply-attenuated.** `speed=0.12, spread=0.6` → output layer `layer_scale≈0.54`,
|
||||
`stddev≈0.065` per weight, squashed by a near-saturated sigmoid, then damped by smoothing→slew→freeze.
|
||||
3. **Decay fights accumulation.** `w *= 0.94` each press pulls back toward the basin → equilibrium by design.
|
||||
4. **Pinned/override dims frozen** on the final layer → never respond.
|
||||
5. **No anchoring.** Thumbs-down stores no example and triggers no training, so there's no learning signal.
|
||||
|
||||
## 6. Operator's Mode 2 ("explore-and-place") vs what exists
|
||||
Already half-built as `RandomiseMlp`:
|
||||
- **Randomise** = `RandomiseMlp::on_down` enter → `snapshot + draw_weights(spread)`: whole net re-rolled →
|
||||
audibly different; user auditions via joystick.
|
||||
- **Anchor** = `on_up` → `restore_after_explore` + `CommitStore`: caller stores +1 example at
|
||||
`(current_input, heard_output)`, original net restored.
|
||||
- **Train** = existing `trainOnCurrent()` trains restored net toward the anchored example.
|
||||
- **Discard** = `on_down` again → `Cancel`, or mode-switch `abort_explore` restores snapshot.
|
||||
|
||||
**Updated-spec deltas to implement (workstream B):** the scratchpad must be NEVER trained (only randomise +
|
||||
undoable nudge); the place step must let the user CHOOSE the input location (press → pick location → release);
|
||||
the real model is warm-started to interpolate ALL placed anchors. The existing `RandomiseMlp` is the closest
|
||||
prototype but commits at the *current* input position, not a chosen one, and lacks the explicit nudge op.
|
||||
|
||||
### Key files
|
||||
- Deployed: `deployments/meml-aimmersive/js/nisps/{mlp.js,nisps-wasm.js,iml.js}`
|
||||
- Core RL: `nisps/ml/{rl.hpp,init.hpp,mlp.hpp}`
|
||||
- Core feedback (untracked): `nisps/ml/feedback.hpp`, `tests/cpp/test_mlp_feedback.cpp`
|
||||
- WASM API: `nisps/wasm/bindings.cpp:419-437`
|
||||
- TS: `playground/src/ml/wasm-iml.ts:488-508`; runtime `playground/src/modes/mode-runtime.ts:444-565`
|
||||
- Exploration knobs: `playground/src/stores/exploration-store.ts`
|
||||
- Port spec: `docs/redesign/feedback-modes-port-spec.md`
|
||||
- Firmware geo-push: `src/memllib` (commits `2429bcc`, `d301cc7`; upstream `SB2026`; main `e291192`/`abe93ec`);
|
||||
design docs on `origin/feat/feedback-explore-modes`: `docs/dislike_system_{analysis,design_space}.md`
|
||||
270
docs/redesign/inputs-spec.md
Normal file
270
docs/redesign/inputs-spec.md
Normal file
|
|
@ -0,0 +1,270 @@
|
|||
# Inputs Spec — Modular Input Layer (Workstream F)
|
||||
|
||||
*Status: implementation-ready spec for the `manifold/` React app. Scope: the modular input layer — sources (XY pad, MIDI input, gamepad single/double-stick), how they compose into an N-dimensional input vector, the MLP-rebuild-on-input-change mechanism, the binding to the reactive spine, and the dock INPUTS panel. Read alongside `docs/redesign/findings-engine-surface.md` (the fixed-2-input gap), `docs/redesign/engine-architecture.md` (the `EngineApi` seam — note that doc says SolidJS/`playground2`; this app is React/`manifold/`), and `docs/redesign/aimmersive-clone-spec.md` (existing gamepad/MIDI/joystick behaviour). British spelling in product copy. The built-in synth is always shown as the "Powerful Synth Engine" — never "C15".*
|
||||
|
||||
---
|
||||
|
||||
## 0. The problem this layer solves
|
||||
|
||||
Today input is hard-wired to exactly two channels. `WasmIML.inferXY(x, y)` calls `setInput(0,x); setInput(1,y); process()` (`playground/src/ml/wasm-iml.ts:307-312`); the whole input pipeline is a 2-tuple pure function `processInput(raw: readonly [number, number], …)` (`playground/src/input/pipeline.ts:225-230`); the WASM build is fixed at `MLP<2u, 10u, 14u, 18u, 126u>` (`nisps/wasm/bindings.cpp:88`). The mission wants the input set to be **modular**: the operator picks one source *or a combination* (XY pad, MIDI input, single-stick gamepad = 2 dims, double-stick gamepad = 4 dims), and the **MLP is (re)built for N inputs × M outputs** whenever that set changes. The same composed input vector drives all backends (synth / MIDI / OSC / VCV) — it is upstream of the backend choice, which is workstream E/output's concern.
|
||||
|
||||
This collides head-on with the fixed-2-input WASM (`findings-engine-surface.md:68-71`, "Gaps to budget for → Fixed MLP arch"). Section 4 resolves that collision and recommends an approach.
|
||||
|
||||
---
|
||||
|
||||
## 1. Architecture: where the input layer sits
|
||||
|
||||
The input layer is **engine-side and headless** — it produces an N-dim `Float32Array` that is the head of the reactive spine. It owns: the registry of active sources, each source's adapter lifecycle, the composition of per-source axes into one vector, and the contract that fires an MLP rebuild when the active-channel count changes. It exposes its surface through `EngineApi` (the React app's single seam to the engine). Skins (the dock INPUTS panel, the Manifold stage) are pure consumers.
|
||||
|
||||
```
|
||||
physical inputs InputLayer (engine-side, headless) spine
|
||||
┌──────────────┐ adapter ┌───────────────────────────────────────────┐
|
||||
│ XY pad (DOM) │──emit axes──►│ InputSource registry (ordered, enabled) │
|
||||
│ WebMIDI │──emit axes──►│ → compose into channel vector raw[N] │──► inputRaw: Float32Array (N)
|
||||
│ Gamepad API │──emit axes──►│ → per-channel pipeline (deadzone…) │ │
|
||||
└──────────────┘ │ → reshape signal on channelCount change │ ▼
|
||||
└───────────────────────────────────────────┘ processedInput → mlOutput → routedOutput → backends
|
||||
│ channelCount change
|
||||
▼
|
||||
MLP rebuild (§4): WasmIML.rebuild(N, M) with warm-start
|
||||
```
|
||||
|
||||
Litmus test (from `engine-architecture.md:40`): anything that touches the DOM/JSX is a skin; the InputLayer adapters read raw browser APIs (`PointerEvent`, `navigator.requestMIDIAccess()`, `navigator.getGamepads()`) but emit **plain numeric axes** into the engine — they hold no React state. The dock panel is the only React surface and it only calls `EngineApi` actions and reads `EngineApi` accessors.
|
||||
|
||||
### File layout (under `manifold/src/engine/input/`)
|
||||
|
||||
```
|
||||
engine/input/
|
||||
├── source.ts # InputSource interface + InputAxis types + SourceId
|
||||
├── registry.ts # active source set (ordered, enabled), channel layout derivation
|
||||
├── compose.ts # sources' axes → flat raw[N] vector (channel map)
|
||||
├── pipeline.ts # LIFTED from playground/src/input/pipeline.ts, generalised to N channels (§3.3)
|
||||
├── sources/
|
||||
│ ├── xy-pad.ts # XYPadSource — 2 axes, fed by the Manifold/XYPad pointer
|
||||
│ ├── midi-input.ts # MidiInputSource — WebMIDI, N learn-mapped CC/note channels
|
||||
│ └── gamepad.ts # GamepadSource — single (2) or double (4) stick, + buttons→actions
|
||||
└── reshape.ts # channelCount-change → MLP rebuild orchestration (§4)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. The `InputSource` adapter interface
|
||||
|
||||
Every source is an adapter that contributes an ordered list of **axes** (each a normalised scalar in `[0,1]`) and, optionally, **action events** (button-style: thumbs-up/down/train/etc., already an a-immersive convention — gamepad LB/RB/A/X/B, `aimmersive-clone-spec.md:161-162`). The interface is deliberately tiny so a fourth source (hand-tracking, OSC-in, sensor) drops in without touching compose/reshape.
|
||||
|
||||
```ts
|
||||
// engine/input/source.ts
|
||||
export type SourceId = 'xy' | 'midi' | 'gamepad';
|
||||
|
||||
/** A single named input dimension produced by a source. Value in [0,1]. */
|
||||
export interface InputAxis {
|
||||
/** Stable per-source key, e.g. 'x', 'y', 'stickL.x', 'cc:74'. Used for persistence + UI labels. */
|
||||
readonly key: string;
|
||||
/** Human label for the dock panel, British spelling. e.g. 'Stick L — X'. */
|
||||
readonly label: string;
|
||||
/** Current normalised value [0,1]; 0.5 = centre for bipolar axes. */
|
||||
value: number;
|
||||
}
|
||||
|
||||
/** Discrete control event a source can raise (maps to verdict/train actions). */
|
||||
export type InputAction =
|
||||
| 'thumbsUp' | 'thumbsDown' | 'train' | 'randomise' | 'clearExamples' | 'undo';
|
||||
|
||||
export interface InputSource {
|
||||
readonly id: SourceId;
|
||||
/** Stable, ordered axes this source contributes when enabled. Length is fixed
|
||||
* per source *configuration* (gamepad: 2 or 4 depending on stick mode;
|
||||
* midi: however many channels the user has learn-mapped). */
|
||||
axes(): readonly InputAxis[];
|
||||
|
||||
/** Begin producing values. `onAxes` is called when any axis changes (coalesced
|
||||
* to display cadence by the engine, not by the source). `onAction` fires discrete
|
||||
* events. Returns a teardown. Idempotent. */
|
||||
attach(sink: {
|
||||
onAxes: () => void; // pull model: engine reads axes() after notify
|
||||
onAction: (a: InputAction) => void;
|
||||
}): () => void;
|
||||
|
||||
/** Availability + status for the dock panel (e.g. 'no MIDI device', 'gamepad 0 connected'). */
|
||||
status(): { available: boolean; connected: boolean; detail: string };
|
||||
|
||||
/** Serialisable per-source config (mappings, invert, stick mode). */
|
||||
toJSON(): unknown;
|
||||
fromJSON(data: unknown): void;
|
||||
}
|
||||
```
|
||||
|
||||
Design notes:
|
||||
- **Pull, not push, for values.** `onAxes()` is a *notify* (a dirty flag); the engine then reads `source.axes()` during composition. This matches the spine's "every consumer reads accessors" rule (`engine-architecture.md:188`) and avoids each source independently writing into the spine — there is exactly one write of `inputRaw` per coalesced tick.
|
||||
- **Actions are separate from axes.** Gamepad buttons and (optionally) MIDI notes raise `InputAction`s routed to `EngineApi.feedback.*`/`train()`, exactly as the legacy gamepad does (`aimmersive-clone-spec.md:161-162`). They do **not** add input channels.
|
||||
- **Axis count per source is a function of its config**, not fixed: gamepad single-stick = 2 axes, double-stick = 4; MIDI = the number of learn-mapped controls. This variable count is what drives the rebuild.
|
||||
|
||||
---
|
||||
|
||||
## 3. The three sources
|
||||
|
||||
### 3.1 `XYPadSource` (2 axes)
|
||||
|
||||
The default, always-available source. Axes `x`, `y` in `[0,1]`, centre 0.5. Fed by the Manifold stage / dock XYPad pointer through `EngineApi.setInput(x, y)` — i.e. the XYPad does not poll a device; the skin pushes pointer coordinates into this source. This preserves the exact a-immersive joystick mapping (`aimmersive-clone-spec.md:92-94`: tap = snap, drag = relative, Y inverted) which lives in the **skin** XYPad component, not here; the source just receives `(x,y)`.
|
||||
|
||||
- `attach`: registers the setter the skin calls; no device polling.
|
||||
- `status`: always `{available:true, connected:true, detail:'XY pad'}`.
|
||||
- This is the only source where the skin is the producer; MIDI and gamepad poll the browser directly.
|
||||
|
||||
### 3.2 `MidiInputSource` (N learn-mapped axes) — NEW
|
||||
|
||||
WebMIDI input, the new required source. Uses `navigator.requestMIDIAccess({ sysex:false })`. The user **learn-maps** physical controls to input channels:
|
||||
|
||||
- **Continuous channels** from CC messages: each mapped CC# (on a chosen channel 1–16) becomes one axis, value = `cc/127` → `[0,1]`. This mirrors the *output* CC convention already in the codebase (`aimmersive-clone-spec.md:128`, CC 0–127, Ch 1–16) but inverted to *input*.
|
||||
- **Note channels (optional)**: a mapped note can act as a momentary axis (`velocity/127` while held, 0 on note-off) or be routed to an `InputAction` (e.g. a pad → thumbsUp). Default: pitch-bend → one bipolar axis (centre 0.5) is offered as a convenience mapping.
|
||||
- **Learn flow**: dock panel "+ Learn" → next incoming CC/note within a timeout binds a new axis; the axis appears in `axes()` and the channel count increases → triggers a rebuild (§4). Removing a mapping decreases the count → rebuild.
|
||||
|
||||
```ts
|
||||
// engine/input/sources/midi-input.ts (shape)
|
||||
interface MidiMapping {
|
||||
key: string; // 'cc:74@1' | 'note:36@10' | 'pb@1'
|
||||
kind: 'cc' | 'note-hold' | 'pitchbend';
|
||||
channel: number; // 1..16
|
||||
selector: number; // cc# | note# (ignored for pitchbend)
|
||||
label: string; // user-editable, British copy default 'MIDI CC 74'
|
||||
invert: boolean;
|
||||
}
|
||||
```
|
||||
|
||||
- `attach`: subscribes to `MIDIInput.onmidimessage`, decodes status bytes, updates the matching mapping's axis value, calls `onAxes()`. Hot-plug handled via `access.onstatechange`.
|
||||
- `status`: reports `access` state and connected input device names; `{available:false}` if `requestMIDIAccess` rejected or unsupported (Safari).
|
||||
- Persistence: `toJSON` serialises the mapping array under the per-app storage key (consistent with the legacy `nisps-midi-cc-map:<engineId>` pattern, `aimmersive-clone-spec.md:239`, but for the *input* side use `nisps:midi-in-map`).
|
||||
|
||||
> **Browser note (open choice):** WebMIDI input is Chromium/Edge/Opera and Firefox-with-flag; **Safari has no WebMIDI**. The dock panel must degrade gracefully (`status.available=false`, greyed source, explanatory copy). See §7.
|
||||
|
||||
### 3.3 `GamepadSource` (2 or 4 axes) — single OR double stick
|
||||
|
||||
Uses the Gamepad API (`navigator.getGamepads()`), polled inside the engine's single coalescing tick (not its own rAF — the engine already runs one display-cadence loop per `engine-architecture.md:183`). Generalises the legacy `GamepadInput` (`aimmersive-clone-spec.md:161-162`, currently left-stick-only with `invertY:true`).
|
||||
|
||||
- **Stick mode `single`** (default): left stick → 2 axes `stick.x`, `stick.y`, mapped `axis*0.5+0.5` → `[0,1]`, Y inverted (preserve legacy `invertY:true`).
|
||||
- **Stick mode `double`**: left + right stick → 4 axes `stickL.x`, `stickL.y`, `stickR.x`, `stickR.y`. This is the "double joystick = 4 input dims" requirement. Switching single↔double changes the axis count 2↔4 → triggers a rebuild (§4).
|
||||
- **Buttons → actions** (unchanged from legacy): LB→`thumbsDown`, RB→`thumbsUp`, A→`train`, X→`randomise`, B→`clearExamples`. Raised via `onAction`, never as axes.
|
||||
- `status`: `{available: 'getGamepads' in navigator, connected: <any pad index present>, detail: pad.id}`. Connection via `gamepadconnected`/`gamepaddisconnected` window events.
|
||||
|
||||
### 3.4 The shared per-channel pipeline (lift + generalise `input/pipeline.ts`)
|
||||
|
||||
`playground/src/input/pipeline.ts` is a parity-relevant, golden-tested pure function but hard-coded to a 2-tuple with **coupled** X/Y stages (circular clamp to a unit disk, momentum computed from 2D velocity — `pipeline.ts:258-268`, `164-198`). For N channels it must be generalised:
|
||||
|
||||
- **Per-channel stages stay**: invert, deadzone, zoom-around-anchor, centred power curve, EMA smoothing, freeze (these are all per-axis already — `pipeline.ts:251-290`).
|
||||
- **The 2D-coupled stages become opt-in pairing**: circular clamp and 2D-velocity momentum only make sense for a *pair* of axes that represent an XY plane (the XY pad, one gamepad stick). The generalised pipeline groups channels into **pairs declared by the source** (XYPad → one pair; gamepad double → two pairs; MIDI CCs → unpaired/independent). Paired channels run the circular-clamp + 2D-momentum path bit-for-bit identical to today; unpaired channels skip those stages (no behavioural change for the existing 2-input case → parity preserved).
|
||||
- Config becomes per-channel (`InputConfig[]` indexed by channel, defaults from `defaultInputConfig()` `pipeline.ts:107-125`), and state per-channel (`InputState[]`, `defaultInputState()` `pipeline.ts:127-135`).
|
||||
|
||||
**Critical research-validity rule** (`findings-design-and-manifold.md:97-98`): train on **raw model-space**, and feed the MLP the **processed** input — but the *training examples* must record the same channel vector the MLP infers on, and there must be **no phantom input channels**. With a true N-input MLP (§4) this is automatic; with padding (§4 option C) it is the central hazard.
|
||||
|
||||
---
|
||||
|
||||
## 4. The MLP-rebuild-on-input-change mechanism — the core collision
|
||||
|
||||
When the composed channel count changes (gamepad single→double, MIDI learn add/remove, source toggled on/off), the MLP's input dimension must change. The WASM build is `MLP<2,…>` fixed at compile time (`bindings.cpp:88`; `nisps_ml_describe` returns the baked dims, `bindings.cpp:578`; `WasmIML` resolves arch from the build and *warns-and-ignores* caller sizes, `wasm-iml.ts:195-203`). Three candidate approaches:
|
||||
|
||||
### Option A — Runtime-shaped MLP (one WASM, dynamic dims)
|
||||
Make the C++ MLP store layer sizes at runtime (heap-allocated weight buffers sized at `nisps_ml_create(input_size, output_size, hidden, n_hidden)`) instead of as template params. `nisps_ml_create` already *accepts* these args (`bindings.cpp:289`); today they're ignored.
|
||||
- **Pros**: one WASM module; honest N×M; no padding; cleanest semantics; warm-start across both input- and output-count changes is natural (legacy already has `createWithWarmStart` for output changes, `aimmersive-clone-spec.md:250`).
|
||||
- **Cons / parity + perf**: **directly contradicts the firmware performance contract** — the firmware path forbids heap in hot paths and relies on compile-time-sized `std::array`/`FixedBuffer` (`CLAUDE.md` "No heap"; `nisps/core`). A runtime-shaped MLP is a *second*, divergent ML implementation → it would **not be the parity-tested core** the operator mandated ("training must use the same core the firmware builds from", `findings-engine-surface.md:7`). Parity check `scripts/parity-check.sh` asserts native==WASM ≤1e-5 against the *fixed* templated MLP; a runtime MLP needs its own parity harness. High cost, high parity risk.
|
||||
|
||||
### Option B — Multiple WASM modules (one per common arch)
|
||||
Pre-build a small matrix of WASM modules: `MLP<2,…,126>`, `MLP<4,…,126>`, and (for MIDI-heavy) `MLP<8,…,126>`. On channel-count change, tear down and recreate `WasmIML` against the module matching N (rounding up to the nearest built arch).
|
||||
- **Pros**: every module is the *real templated, parity-tested core* — firmware-identical, parity harness unchanged per module. No heap. Honest input dims (up to the nearest built size).
|
||||
- **Cons / perf**: N is quantised to the built set; download weight ×K modules (each ~94KB, `bindings.cpp` build is small, so ~3 modules ≈ 280KB — acceptable). Warm-start across modules requires weight transplant between *different* WASM instances (hidden weights transfer; new input/output rows re-randomise — the `createWithWarmStart` logic generalised). Build/CI must produce and parity-check each module (`scripts/build-wasm.sh` gains a loop). Medium cost, **low parity risk**.
|
||||
|
||||
### Option C — Max-N padded input with active-channel masking
|
||||
Keep one WASM at a generous fixed arch, e.g. `MLP<8,…,126>`. Always run 8 inputs; "inactive" channels are pinned to a constant (0.5/centre) and masked. The composed vector fills the first N; the rest are held neutral.
|
||||
- **Pros**: one module; no rebuild ever (zero-latency source switching); single parity harness.
|
||||
- **Cons / research-validity**: this is exactly the **phantom-input-channel** anti-pattern the redesign explicitly calls a research-validity bug (`findings-design-and-manifold.md:98`, `engine-architecture.md:409` "killing today's phantom-channel OOB writes"). Held-constant inputs still consume weights and bias the network; training examples recorded at different active-channel sets become inconsistent; the model "sees" 8 dims always. Also wastes firmware budget if the same arch ships to hardware. Low rebuild cost but **high validity cost** — rejected on principle.
|
||||
|
||||
### Recommendation: **Option B (multiple WASM modules), with a phased rollout**
|
||||
|
||||
Option B is the only approach that keeps the **parity-tested, firmware-identical templated core** (the operator's hard constraint) while honestly representing N input dims. Concretely:
|
||||
|
||||
1. **Phase F0 (ships first):** build **two** modules — `MLP<2,…,126>` (existing) and `MLP<4,…,126>`. This covers XY pad (2), single-stick gamepad (2), and double-stick gamepad (4) — the gamepad requirement in full. MIDI maps onto these by capping learn-mapped continuous channels and/or rounding N up to 4. Both modules go through `scripts/parity-check.sh` in CI.
|
||||
2. **Phase F1:** add `MLP<8,…,126>` for richer MIDI rigs (up to 8 continuous channels). N rounds up to the nearest built arch; unused channels of the *chosen* module are still real channels fed real values where present, **never phantom-padded across the active set** — i.e. we only ever select a module whose input size ≥ N, and we feed exactly N real channels, leaving at most a few genuinely-unused trailing channels held at centre *with a recorded "module larger than active set" caveat* (the one residual padding, bounded and surfaced, not silent).
|
||||
3. **Rebuild orchestration** (`engine/input/reshape.ts`): on channel-count change → (a) snapshot current weights via `WasmIML.getWeights()`; (b) pick target module for new N; (c) `WasmIML.rebuild(targetModule, N, M)` which creates a fresh handle and **warm-starts**: hidden + output weights transfer where shapes match, the input→layer0 weight rows are preserved for surviving channels and re-randomised for new ones (generalise `createWithWarmStart`, `aimmersive-clone-spec.md:250`); (d) if training examples exist and the input dimension changed, the dataset's feature vectors are re-shaped (truncate/zero-extend with a **user confirm**, mirroring the legacy output-count-change confirm at `aimmersive-clone-spec.md:145`) — or cleared; (e) bump `weightsRevision` so the spine re-derives.
|
||||
4. **The fixed-2 contraction is explicitly time-boxed:** Phase F0/F1 deliver 2 and 4 (and later 8) — *not* arbitrary N. Arbitrary-N runtime shaping (Option A) is deferred and only revisited if a use case needs counts outside the built matrix, and only *behind a passing dedicated parity check* (per `findings-engine-surface.md:68-71` / `engine-architecture.md:409`).
|
||||
|
||||
This is a strictly smaller, safer change than Option A and avoids the validity landmine of Option C, while satisfying "the MLP is (re)built for N×M when the input set changes" for the concrete source matrix the operator named.
|
||||
|
||||
### `EngineApi` + `WasmIML` surface changes
|
||||
|
||||
```ts
|
||||
// EngineApi additions (engine/types.ts)
|
||||
interface EngineApi {
|
||||
// ... existing spine accessors/actions ...
|
||||
input: {
|
||||
sources: () => readonly { id: SourceId; enabled: boolean; axisCount: number; status: SourceStatus }[];
|
||||
setEnabled: (id: SourceId, on: boolean) => void; // toggling → may rebuild
|
||||
channelCount: Accessor<number>; // current N (drives rebuild)
|
||||
channelLabels: () => readonly string[]; // for dock + heatmap axes
|
||||
setInput: (x: number, y: number) => void; // XYPad fast-path (back-compat)
|
||||
gamepadStickMode: (m: 'single' | 'double') => void; // → rebuild on change
|
||||
midi: MidiInputApi; // learn / map / clear / list devices
|
||||
rebuilding: Accessor<boolean>; // UI disables training mid-rebuild
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
`WasmIML` (lifted from `wasm-iml.ts`) gains `rebuild(moduleUrl, inputSize, outputSize, { warmStart: boolean })`. Today `init_` hard-binds one module and ignores caller dims (`wasm-iml.ts:171-225`); the rebuild path instantiates the module whose `nisps_ml_describe` reports `inputSize===N` (or the smallest ≥ N), re-allocates the heap buffers sized to the new arch (`wasm-iml.ts:218-225`), and re-pushes weights. The per-module factory cache keys on module URL (extend `cachedFactory`, `wasm-iml.ts:45`).
|
||||
|
||||
---
|
||||
|
||||
## 5. Composition into the N-dim vector
|
||||
|
||||
`engine/input/compose.ts` builds the flat `raw[N]` consumed by the spine:
|
||||
|
||||
1. **Channel layout** = concatenation, in registry order, of each *enabled* source's `axes()`. So `[XY:x, XY:y]` then `[gamepad:stickL.x, …]` then `[midi:cc74, midi:cc71, …]`. The layout (ordered list of `{sourceId, axisKey, label}`) is the single source of truth for: the MLP input index of each channel, the dock panel rows, and the heatmap/diagnostics axis labels.
|
||||
2. On any source `onAxes()` notify, the engine (in its one coalescing tick, `coalesce.ts` per `engine-architecture.md:183-185`) reads every enabled source's `axes()`, writes their values into `raw[]` at their layout indices, and sets the `inputRaw` signal `{equals:false}`.
|
||||
3. **Channel-count change detection:** the layout length is compared to the live MLP input size each time the registry changes (source toggled, gamepad stick mode flipped, MIDI mapping added/removed). A mismatch enqueues a rebuild (§4) *before* the next infer — never mid-tick.
|
||||
4. **Per-channel pipeline** (§3.4) runs on `raw[]` to produce `processedInput[]`, which is what the MLP infers on and what training examples record (no phantom channels — the recorded feature vector length == live MLP input size).
|
||||
|
||||
Multiple simultaneous sources compose naturally: XY pad + a MIDI CC = 3 channels → module `MLP<4,…>` (round up), 3 real channels fed, 1 trailing channel held at centre with the bounded-padding caveat surfaced in the panel.
|
||||
|
||||
---
|
||||
|
||||
## 6. Binding to the reactive spine (React)
|
||||
|
||||
Per `findings-design-and-manifold.md:70-88`, the spine must live **below React** and not on the render cycle. The input layer is the head of that spine:
|
||||
|
||||
- `inputRaw` is an external-store signal (`Float32Array(N)`, `{equals:false}`), updated only inside the engine's coalesced tick. React components **never** write it directly except via `EngineApi.input.setInput`/source toggles.
|
||||
- The dock INPUTS panel subscribes via `useSyncExternalStore(subscribe, () => versionCounter)` (version counter, not the array — `findings-design-and-manifold.md:81-82`) to re-render on source-set / status / channel-count changes. It reads live axis values imperatively for meters.
|
||||
- The Manifold/visualiser reads channel values in its own `requestAnimationFrame` loop (drawing only; `findings-design-and-manifold.md:84`).
|
||||
- **Rebuild is an engine action**, not a render effect: `EngineApi.input.setEnabled`/`gamepadStickMode`/`midi.learn` run the §4 reshape synchronously off-render, bump `weightsRevision`, then flip `rebuilding()` back to false. The single send-effect (`engine-architecture.md:173-177`) re-fires once after rebuild.
|
||||
- **E2E invariant extension** (`findings-design-and-manifold.md:86-87`): the existing `setInputs([x,y]) → getOutputs() changes` probe assertion is extended with a per-source-set case — e.g. enable double-stick gamepad, assert `channelCount()===4`, assert `getOutputs()` length and that moving channel 3 changes outputs. `window.__nisps` (engine probe, `engine-architecture.md:237`) gains `setChannel(i, v)`, `getChannelCount()`, `setSources([...])`.
|
||||
|
||||
---
|
||||
|
||||
## 7. The dock INPUTS panel (skin)
|
||||
|
||||
A dedicated dock entry (its own rail icon + drawer), parallel to the other dock drawers (`findings-design-and-manifold.md:50`). Pure consumer of `EngineApi.input`. Manifold token language: dark glass chrome, orange `--accent #ff6a00` primary, cyan `--accent-2 #00ccff` for live data/meters, uppercase letter-spaced labels, glow halos on live elements (`findings-design-and-manifold.md:24-38`).
|
||||
|
||||
**Layout:**
|
||||
|
||||
- **Source toggles** (top): three `PillToggle`s — **XY Pad** / **MIDI** / **Gamepad** — each enableable independently (combination allowed). Each shows a `StatusLine`: connected device / "No MIDI device" / "No gamepad detected". A disabled/unavailable source (e.g. MIDI on Safari) is greyed with explanatory copy.
|
||||
- **Channel readout**: a live list of the composed channels in layout order, each row = label + a cyan value meter (Sparkline/bar) reading the live axis. Shows total **N** prominently ("4 INPUT CHANNELS"). A subtle badge when a module larger than the active set is in use ("1 channel held at centre").
|
||||
- **MIDI sub-panel** (when MIDI enabled): device picker; a **+ Learn** button (arm → next CC/note binds, with countdown); a list of mappings each with editable **Label**, **CC#/Note**, **Ch**, **Invert**, and a remove (✕). Add/remove warns when training examples exist (rebuild confirm).
|
||||
- **Gamepad sub-panel** (when gamepad enabled): a `PillToggle` **Single / Double** stick (the 2↔4 dim switch), an invert-Y switch (default on), and a small button-map legend (LB Down / RB Up / A Train / X Randomise / B Clear).
|
||||
- **XY Pad sub-panel**: per-axis invert switches and the existing input-shaping controls (deadzone / zoom / curve / smoothing / momentum) — these are the per-channel pipeline params (§3.4), reused for any paired axes.
|
||||
- **Rebuild affordance**: whenever a control would change `channelCount`, the panel shows a confirm if examples exist ("Changing inputs rebuilds the network and may clear training examples"), then disables interaction while `rebuilding()` is true.
|
||||
|
||||
**Keyboard / accessibility:** the dock drawer opens via its number key (consistent with `findings-design-and-manifold.md:48`). All hit targets ≥ `--hit-min 44px`.
|
||||
|
||||
---
|
||||
|
||||
## 8. Persistence & VCV/bridge note
|
||||
|
||||
- Per-source config persists via `InputSource.toJSON`/`fromJSON` under one versioned blob (use `engine/persist.ts`, `engine-architecture.md:113`); keys: `nisps:input-sources` (enabled set + per-source config), `nisps:midi-in-map` (mappings). The **active channel count is persisted** so the correct WASM module is selected on load *before* weights rehydrate (rebuild-then-load order, mirroring legacy load-then-train, `aimmersive-clone-spec.md:239`). Never auto-request MIDI/gamepad permission on load beyond what the browser grants silently.
|
||||
- **Bridge / VCV (workstream E):** the composed `inputRaw[N]` is the same vector the VCV module receives over the bridge — the bridge is just another *producer* into a source slot (a future `BridgeInputSource`) or a *mirror* of the composed vector, depending on direction. This spec keeps the input layer backend-agnostic: it produces channels; who consumes them (synth/MIDI/OSC/VCV) is downstream. The four-source interface (XY/MIDI/gamepad + a future bridge/OSC-in) is why `InputSource` is kept minimal.
|
||||
|
||||
---
|
||||
|
||||
## 9. Open choices for the operator
|
||||
|
||||
1. **MLP-reshape strategy — confirm Option B (multiple WASM modules).** Recommended because it keeps the parity-tested, firmware-identical templated core and avoids phantom channels. Trade-off: input dim is quantised to the built matrix (2/4, later 8), not arbitrary N. Confirm the initial matrix is `{2, 4}` for F0 and `{2, 4, 8}` for F1. If the operator wants *arbitrary* N now, that forces Option A (runtime-shaped MLP) — a divergent, separately-parity-tested core, larger cost, and a departure from "same core as firmware".
|
||||
2. **Trailing-channel handling when N < module input size.** Recommended: feed N real channels, hold the remaining built-but-unused channels at centre and surface a badge. Confirm this bounded, visible padding is acceptable (it is *not* the rejected silent phantom-channel pattern, but it is a small residual).
|
||||
3. **Browser support for MIDI/gamepad.** WebMIDI: Chromium/Edge/Opera (+ Firefox flag), **no Safari** → MIDI source degrades to unavailable. Gamepad API: broad including Safari. Confirm the supported-browser target (recommend "Chromium-first, graceful degradation elsewhere", matching the COOP/COEP-heavy build already in `vite.config.ts`).
|
||||
4. **MIDI note → action vs axis.** Default offered: pitch-bend as a bipolar axis; pads/notes optionally route to `thumbsUp`/`thumbsDown`. Confirm whether note-as-axis (velocity-hold) should be a first-class channel type or stay an advanced toggle.
|
||||
5. **Does enabling/disabling a source mid-session clear training examples?** Recommended: confirm-and-reshape (truncate/extend feature vectors) rather than hard-clear, to match the legacy warm-start ethos. Confirm acceptable, or prefer the simpler hard-clear-on-input-change.
|
||||
6. **Dock entry placement/icon** among the existing drawers (`shape/feel/route/health/help`, `findings-design-and-manifold.md:48`) — INPUTS as a new top-level drawer vs a section of an existing one.
|
||||
|
|
@ -0,0 +1,54 @@
|
|||
import React from 'react';
|
||||
|
||||
/**
|
||||
* Manifold ControlAxis — a named macro slider with bipolar endpoint labels
|
||||
* (e.g. Boldness: Caution ↔ Bold). Shows a live preset tag and value. The
|
||||
* track accent can be themed per-axis via `accent`.
|
||||
*/
|
||||
export function ControlAxis({ label, endpoints = ['', ''], value = 0.5, onChange, preset, accent = 'var(--accent)', disabled = false, style }) {
|
||||
return (
|
||||
<div
|
||||
style={{
|
||||
display: 'flex',
|
||||
flexDirection: 'column',
|
||||
gap: 'var(--sp-1)',
|
||||
background: 'var(--bg-1)',
|
||||
border: '1px solid var(--line)',
|
||||
borderRadius: 'var(--r-2)',
|
||||
padding: 'var(--sp-2) var(--sp-3)',
|
||||
fontFamily: 'var(--font-mono)',
|
||||
opacity: disabled ? 0.5 : 1,
|
||||
pointerEvents: disabled ? 'none' : 'auto',
|
||||
...style,
|
||||
}}
|
||||
>
|
||||
<div style={{ display: 'flex', alignItems: 'center', gap: 'var(--sp-2)', fontSize: 'var(--fs-sm)' }}>
|
||||
<span style={{ fontWeight: 600, textTransform: 'uppercase', letterSpacing: '0.08em', color: 'var(--fg)', flex: 1 }}>{label}</span>
|
||||
{preset && <span style={{ color: accent, fontSize: 'var(--fs-xs)', textTransform: 'uppercase', letterSpacing: '0.06em' }}>{preset}</span>}
|
||||
<span style={{ fontVariantNumeric: 'tabular-nums', color: 'var(--fg-mute)', fontSize: 'var(--fs-xs)', minWidth: '4ch', textAlign: 'right' }}>{value.toFixed(2)}</span>
|
||||
</div>
|
||||
<input
|
||||
type="range"
|
||||
min={0}
|
||||
max={1}
|
||||
step={0.01}
|
||||
value={value}
|
||||
disabled={disabled}
|
||||
onChange={(e) => onChange && onChange(parseFloat(e.target.value))}
|
||||
className="mf-axis-input"
|
||||
style={{ WebkitAppearance: 'none', appearance: 'none', width: '100%', height: 24, background: 'transparent', margin: 0, cursor: 'pointer', '--mf-axis-accent': accent }}
|
||||
/>
|
||||
<div style={{ display: 'flex', justifyContent: 'space-between', fontSize: 10, textTransform: 'uppercase', letterSpacing: '0.08em', color: 'var(--fg-dim)' }}>
|
||||
<span>{endpoints[0]}</span>
|
||||
<span>{endpoints[1]}</span>
|
||||
</div>
|
||||
<style dangerouslySetInnerHTML={{ __html: `
|
||||
.mf-axis-input::-webkit-slider-runnable-track { height: 6px; border-radius: 999px; background: var(--bg-3); }
|
||||
.mf-axis-input::-moz-range-track { height: 6px; border-radius: 999px; background: var(--bg-3); }
|
||||
.mf-axis-input::-webkit-slider-thumb { -webkit-appearance: none; appearance: none; width: 18px; height: 18px; border-radius: 50%; background: var(--mf-axis-accent, var(--accent)); margin-top: -6px; box-shadow: 0 0 10px var(--mf-axis-accent, var(--accent)); cursor: pointer; }
|
||||
.mf-axis-input::-moz-range-thumb { width: 18px; height: 18px; border-radius: 50%; background: var(--mf-axis-accent, var(--accent)); border: none; box-shadow: 0 0 10px var(--mf-axis-accent, var(--accent)); }
|
||||
.mf-axis-input:focus { outline: none; }
|
||||
` }} />
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
|
@ -0,0 +1,80 @@
|
|||
import React from 'react';
|
||||
|
||||
/**
|
||||
* Manifold VirtualJoystick — circular control. Drag the glowing orange knob;
|
||||
* motion is constrained to the circle. Emits normalized (x, y) in [0,1], y-up.
|
||||
*/
|
||||
export function VirtualJoystick({ size = 200, position, onMove, onGrab, onRelease, disabled = false, ariaLabel = 'virtual joystick', style }) {
|
||||
const [internal, setInternal] = React.useState([0.5, 0.5]);
|
||||
const [dragging, setDragging] = React.useState(false);
|
||||
const ref = React.useRef(null);
|
||||
const pos = position || internal;
|
||||
|
||||
const update = (e) => {
|
||||
const el = ref.current;
|
||||
if (!el) return;
|
||||
const r = el.getBoundingClientRect();
|
||||
let x = Math.max(0, Math.min(1, (e.clientX - r.left) / r.width));
|
||||
let y = Math.max(0, Math.min(1, 1 - (e.clientY - r.top) / r.height));
|
||||
const dx = x - 0.5, dy = y - 0.5;
|
||||
const dist = Math.sqrt(dx * dx + dy * dy);
|
||||
if (dist > 0.5 && dist > 1e-12) { x = 0.5 + (dx / dist) * 0.5; y = 0.5 + (dy / dist) * 0.5; }
|
||||
if (!position) setInternal([x, y]);
|
||||
onMove && onMove(x, y);
|
||||
};
|
||||
|
||||
const down = (e) => { if (disabled) return; e.currentTarget.setPointerCapture?.(e.pointerId); setDragging(true); onGrab && onGrab(); update(e); };
|
||||
const move = (e) => { if (dragging) update(e); };
|
||||
const up = (e) => { if (!dragging) return; e.currentTarget.releasePointerCapture?.(e.pointerId); setDragging(false); onRelease && onRelease(); };
|
||||
|
||||
const [x, y] = pos;
|
||||
return (
|
||||
<div
|
||||
ref={ref}
|
||||
role="application"
|
||||
aria-label={ariaLabel}
|
||||
tabIndex={disabled ? -1 : 0}
|
||||
onPointerDown={down}
|
||||
onPointerMove={move}
|
||||
onPointerUp={up}
|
||||
onPointerCancel={up}
|
||||
style={{
|
||||
position: 'relative',
|
||||
width: size,
|
||||
height: size,
|
||||
background: 'var(--bg-1)',
|
||||
border: '1px solid var(--line)',
|
||||
borderRadius: '50%',
|
||||
touchAction: 'none',
|
||||
cursor: dragging ? 'grabbing' : 'grab',
|
||||
outline: 'none',
|
||||
userSelect: 'none',
|
||||
overflow: 'hidden',
|
||||
opacity: disabled ? 0.4 : 1,
|
||||
pointerEvents: disabled ? 'none' : 'auto',
|
||||
...style,
|
||||
}}
|
||||
>
|
||||
<div style={{ position: 'absolute', inset: '6%', borderRadius: '50%', border: '1px dashed var(--line-strong)', pointerEvents: 'none' }} />
|
||||
<div aria-hidden="true" style={{ position: 'absolute', inset: 0, opacity: 0.4, pointerEvents: 'none' }}>
|
||||
<div style={{ position: 'absolute', left: 0, right: 0, top: '50%', height: 1, background: 'var(--line-strong)' }} />
|
||||
<div style={{ position: 'absolute', top: 0, bottom: 0, left: '50%', width: 1, background: 'var(--line-strong)' }} />
|
||||
</div>
|
||||
<div
|
||||
aria-hidden="true"
|
||||
style={{
|
||||
position: 'absolute',
|
||||
top: 0,
|
||||
left: 0,
|
||||
width: 24,
|
||||
height: 24,
|
||||
borderRadius: '50%',
|
||||
background: 'var(--accent)',
|
||||
boxShadow: '0 0 12px var(--glow-accent)',
|
||||
transform: `translate(${x * size}px, ${(1 - y) * size}px) translate(-50%, -50%)`,
|
||||
pointerEvents: 'none',
|
||||
}}
|
||||
/>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
83
docs/redesign/manifold-export/components/control/XYPad.jsx
Normal file
83
docs/redesign/manifold-export/components/control/XYPad.jsx
Normal file
|
|
@ -0,0 +1,83 @@
|
|||
import React from 'react';
|
||||
|
||||
/**
|
||||
* Manifold XYPad — square control surface. Drag the glowing cyan dot; emits
|
||||
* normalized (x, y) in [0,1] with y-up. Uncontrolled by default; pass
|
||||
* `position` + `onMove` to control it.
|
||||
*/
|
||||
export function XYPad({ size = 240, showGrid = true, position, onMove, onGrab, onRelease, disabled = false, ariaLabel = 'XY pad', style }) {
|
||||
const [internal, setInternal] = React.useState([0.5, 0.5]);
|
||||
const [dragging, setDragging] = React.useState(false);
|
||||
const ref = React.useRef(null);
|
||||
const pos = position || internal;
|
||||
|
||||
const update = (e) => {
|
||||
const el = ref.current;
|
||||
if (!el) return;
|
||||
const r = el.getBoundingClientRect();
|
||||
const x = Math.max(0, Math.min(1, (e.clientX - r.left) / r.width));
|
||||
const y = Math.max(0, Math.min(1, 1 - (e.clientY - r.top) / r.height));
|
||||
if (!position) setInternal([x, y]);
|
||||
onMove && onMove(x, y);
|
||||
};
|
||||
|
||||
const down = (e) => {
|
||||
if (disabled) return;
|
||||
e.currentTarget.setPointerCapture?.(e.pointerId);
|
||||
setDragging(true); onGrab && onGrab(); update(e);
|
||||
};
|
||||
const move = (e) => { if (dragging) update(e); };
|
||||
const up = (e) => { if (!dragging) return; e.currentTarget.releasePointerCapture?.(e.pointerId); setDragging(false); onRelease && onRelease(); };
|
||||
|
||||
const [x, y] = pos;
|
||||
return (
|
||||
<div
|
||||
ref={ref}
|
||||
role="application"
|
||||
aria-label={ariaLabel}
|
||||
tabIndex={disabled ? -1 : 0}
|
||||
onPointerDown={down}
|
||||
onPointerMove={move}
|
||||
onPointerUp={up}
|
||||
onPointerCancel={up}
|
||||
style={{
|
||||
position: 'relative',
|
||||
width: size,
|
||||
height: size,
|
||||
background: 'var(--bg-1)',
|
||||
border: '1px solid var(--line)',
|
||||
borderRadius: 'var(--r-2)',
|
||||
touchAction: 'none',
|
||||
cursor: 'crosshair',
|
||||
outline: 'none',
|
||||
userSelect: 'none',
|
||||
overflow: 'hidden',
|
||||
opacity: disabled ? 0.4 : 1,
|
||||
pointerEvents: disabled ? 'none' : 'auto',
|
||||
...style,
|
||||
}}
|
||||
>
|
||||
{showGrid && (
|
||||
<div aria-hidden="true" style={{ position: 'absolute', inset: 0, opacity: 0.5, pointerEvents: 'none' }}>
|
||||
<div style={{ position: 'absolute', left: 0, right: 0, top: '50%', height: 1, background: 'var(--line-strong)' }} />
|
||||
<div style={{ position: 'absolute', top: 0, bottom: 0, left: '50%', width: 1, background: 'var(--line-strong)' }} />
|
||||
</div>
|
||||
)}
|
||||
<div
|
||||
aria-hidden="true"
|
||||
style={{
|
||||
position: 'absolute',
|
||||
top: 0,
|
||||
left: 0,
|
||||
width: 14,
|
||||
height: 14,
|
||||
borderRadius: '50%',
|
||||
background: 'var(--accent-2)',
|
||||
boxShadow: '0 0 10px var(--glow-accent-2)',
|
||||
transform: `translate(${x * size}px, ${(1 - y) * size}px) translate(-50%, -50%)`,
|
||||
pointerEvents: 'none',
|
||||
}}
|
||||
/>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
41
docs/redesign/manifold-export/components/core/Badge.jsx
Normal file
41
docs/redesign/manifold-export/components/core/Badge.jsx
Normal file
|
|
@ -0,0 +1,41 @@
|
|||
import React from 'react';
|
||||
|
||||
const TONES = {
|
||||
neutral: { fg: 'var(--fg-mute)', bd: 'var(--line)', bg: 'var(--bg-2)' },
|
||||
accent: { fg: 'var(--accent)', bd: 'rgba(255,106,0,0.4)', bg: 'rgba(255,106,0,0.12)' },
|
||||
good: { fg: 'var(--good)', bd: 'rgba(107,194,107,0.4)', bg: 'rgba(107,194,107,0.14)' },
|
||||
warn: { fg: 'var(--warn)', bd: 'rgba(245,196,94,0.4)', bg: 'rgba(245,196,94,0.14)' },
|
||||
bad: { fg: 'var(--bad)', bd: 'rgba(239,91,91,0.4)', bg: 'rgba(239,91,91,0.14)' },
|
||||
info: { fg: 'var(--info)', bd: 'rgba(91,158,239,0.4)', bg: 'rgba(91,158,239,0.14)' },
|
||||
};
|
||||
|
||||
/**
|
||||
* Manifold Badge — small status capsule. `dot` prepends a status dot;
|
||||
* `tone` sets the color. Use for state labels (frozen, training, healthy).
|
||||
*/
|
||||
export function Badge({ children, tone = 'neutral', dot = false, style }) {
|
||||
const t = TONES[tone] || TONES.neutral;
|
||||
return (
|
||||
<span
|
||||
style={{
|
||||
display: 'inline-flex',
|
||||
alignItems: 'center',
|
||||
gap: 'var(--sp-1)',
|
||||
fontFamily: 'var(--font-mono)',
|
||||
fontSize: 'var(--fs-xs)',
|
||||
textTransform: 'uppercase',
|
||||
letterSpacing: '0.06em',
|
||||
color: t.fg,
|
||||
background: t.bg,
|
||||
border: `1px solid ${t.bd}`,
|
||||
borderRadius: 'var(--r-pill)',
|
||||
padding: '2px 10px',
|
||||
lineHeight: 1.6,
|
||||
...style,
|
||||
}}
|
||||
>
|
||||
{dot && <span style={{ width: 6, height: 6, borderRadius: '50%', background: t.fg, boxShadow: `0 0 6px ${t.fg}` }} />}
|
||||
{children}
|
||||
</span>
|
||||
);
|
||||
}
|
||||
92
docs/redesign/manifold-export/components/core/Button.jsx
Normal file
92
docs/redesign/manifold-export/components/core/Button.jsx
Normal file
|
|
@ -0,0 +1,92 @@
|
|||
import React from 'react';
|
||||
|
||||
/**
|
||||
* Manifold Button — terminal-styled action.
|
||||
* Variants: primary (solid orange), secondary (outlined raised), ghost (text).
|
||||
* Sizes: sm, md, lg. Optional leading glyph.
|
||||
*/
|
||||
export function Button({
|
||||
children,
|
||||
variant = 'secondary',
|
||||
size = 'md',
|
||||
disabled = false,
|
||||
glyph,
|
||||
active = false,
|
||||
type = 'button',
|
||||
onClick,
|
||||
style,
|
||||
...rest
|
||||
}) {
|
||||
const sizes = {
|
||||
sm: { padding: '4px 12px', fontSize: 'var(--fs-xs)', height: 28 },
|
||||
md: { padding: '8px 12px', fontSize: 'var(--fs-sm)', height: 34 },
|
||||
lg: { padding: '10px 18px', fontSize: 'var(--fs-md)', height: 44 },
|
||||
};
|
||||
const s = sizes[size] || sizes.md;
|
||||
|
||||
const base = {
|
||||
display: 'inline-flex',
|
||||
alignItems: 'center',
|
||||
justifyContent: 'center',
|
||||
gap: 'var(--sp-2)',
|
||||
fontFamily: 'var(--font-mono)',
|
||||
fontSize: s.fontSize,
|
||||
height: s.height,
|
||||
padding: s.padding,
|
||||
borderRadius: 'var(--r-1)',
|
||||
border: '1px solid var(--line)',
|
||||
cursor: disabled ? 'not-allowed' : 'pointer',
|
||||
userSelect: 'none',
|
||||
transition: 'background var(--dur-fast) var(--ease), border-color var(--dur-fast) var(--ease), color var(--dur-fast) var(--ease)',
|
||||
whiteSpace: 'nowrap',
|
||||
};
|
||||
|
||||
const variants = {
|
||||
primary: {
|
||||
background: 'var(--accent)',
|
||||
borderColor: 'var(--accent)',
|
||||
color: 'var(--bg)',
|
||||
fontWeight: 600,
|
||||
},
|
||||
secondary: {
|
||||
background: active ? 'var(--bg-3)' : 'var(--bg-2)',
|
||||
borderColor: active ? 'var(--accent)' : 'var(--line)',
|
||||
color: active ? 'var(--accent)' : 'var(--fg)',
|
||||
},
|
||||
ghost: {
|
||||
background: 'transparent',
|
||||
borderColor: 'transparent',
|
||||
color: active ? 'var(--accent)' : 'var(--fg-mute)',
|
||||
},
|
||||
};
|
||||
|
||||
const disabledStyle = disabled
|
||||
? { opacity: 0.45, color: 'var(--fg-dim)', boxShadow: 'none' }
|
||||
: null;
|
||||
|
||||
return (
|
||||
<button
|
||||
type={type}
|
||||
disabled={disabled}
|
||||
onClick={onClick}
|
||||
style={{ ...base, ...(variants[variant] || variants.secondary), ...disabledStyle, ...style }}
|
||||
onMouseEnter={(e) => {
|
||||
if (disabled) return;
|
||||
if (variant === 'secondary') { e.currentTarget.style.background = 'var(--bg-3)'; e.currentTarget.style.borderColor = 'var(--line-strong)'; }
|
||||
if (variant === 'ghost') e.currentTarget.style.color = 'var(--fg)';
|
||||
if (variant === 'primary') e.currentTarget.style.background = 'var(--accent-3)';
|
||||
}}
|
||||
onMouseLeave={(e) => {
|
||||
if (disabled) return;
|
||||
const v = variants[variant] || variants.secondary;
|
||||
e.currentTarget.style.background = v.background;
|
||||
e.currentTarget.style.borderColor = v.borderColor;
|
||||
e.currentTarget.style.color = v.color;
|
||||
}}
|
||||
{...rest}
|
||||
>
|
||||
{glyph && <span aria-hidden="true" style={{ fontSize: '1.1em', lineHeight: 1 }}>{glyph}</span>}
|
||||
{children}
|
||||
</button>
|
||||
);
|
||||
}
|
||||
30
docs/redesign/manifold-export/components/core/Panel.jsx
Normal file
30
docs/redesign/manifold-export/components/core/Panel.jsx
Normal file
|
|
@ -0,0 +1,30 @@
|
|||
import React from 'react';
|
||||
|
||||
/**
|
||||
* Manifold Panel — the house surface: bg-1 fill, 1px hairline border, 8px
|
||||
* radius, no shadow. Optional header row with an uppercase title + actions,
|
||||
* separated by a hairline.
|
||||
*/
|
||||
export function Panel({ title, label, actions, children, padding = 'var(--sp-3)', style }) {
|
||||
return (
|
||||
<section
|
||||
style={{
|
||||
background: 'var(--bg-1)',
|
||||
border: '1px solid var(--line)',
|
||||
borderRadius: 'var(--r-2)',
|
||||
fontFamily: 'var(--font-mono)',
|
||||
color: 'var(--fg)',
|
||||
...style,
|
||||
}}
|
||||
>
|
||||
{(title || label || actions) && (
|
||||
<header style={{ display: 'flex', alignItems: 'center', gap: 'var(--sp-2)', padding: 'var(--sp-2) var(--sp-3)', borderBottom: '1px solid var(--line)' }}>
|
||||
{label && <span style={{ fontSize: 'var(--fs-xs)', color: 'var(--fg-dim)', textTransform: 'uppercase', letterSpacing: '0.08em' }}>{label}</span>}
|
||||
{title && <h3 style={{ margin: 0, fontSize: 'var(--fs-sm)', fontWeight: 600, color: 'var(--fg)' }}>{title}</h3>}
|
||||
{actions && <div style={{ marginLeft: 'auto', display: 'flex', gap: 'var(--sp-2)' }}>{actions}</div>}
|
||||
</header>
|
||||
)}
|
||||
<div style={{ padding }}>{children}</div>
|
||||
</section>
|
||||
);
|
||||
}
|
||||
55
docs/redesign/manifold-export/components/core/PillToggle.jsx
Normal file
55
docs/redesign/manifold-export/components/core/PillToggle.jsx
Normal file
|
|
@ -0,0 +1,55 @@
|
|||
import React from 'react';
|
||||
|
||||
/**
|
||||
* Manifold PillToggle — segmented radio control in a pill capsule.
|
||||
* The selected segment fills solid orange. Options: [{value,label}].
|
||||
*/
|
||||
export function PillToggle({ options = [], value, onChange, ariaLabel = 'segmented control', disabled = false, style }) {
|
||||
return (
|
||||
<div
|
||||
role="radiogroup"
|
||||
aria-label={ariaLabel}
|
||||
style={{
|
||||
display: 'inline-flex',
|
||||
background: 'var(--bg-2)',
|
||||
border: '1px solid var(--line)',
|
||||
borderRadius: 'var(--r-pill)',
|
||||
padding: 2,
|
||||
gap: 2,
|
||||
opacity: disabled ? 0.5 : 1,
|
||||
pointerEvents: disabled ? 'none' : 'auto',
|
||||
...style,
|
||||
}}
|
||||
>
|
||||
{options.map((opt) => {
|
||||
const selected = value === opt.value;
|
||||
return (
|
||||
<button
|
||||
key={opt.value}
|
||||
type="button"
|
||||
role="radio"
|
||||
aria-checked={selected}
|
||||
onClick={() => onChange && onChange(opt.value)}
|
||||
style={{
|
||||
background: selected ? 'var(--accent)' : 'transparent',
|
||||
color: selected ? 'var(--bg)' : 'var(--fg-mute)',
|
||||
border: 0,
|
||||
borderRadius: 'var(--r-pill)',
|
||||
padding: '6px 14px',
|
||||
fontFamily: 'var(--font-mono)',
|
||||
fontSize: 'var(--fs-xs)',
|
||||
textTransform: 'uppercase',
|
||||
letterSpacing: '0.08em',
|
||||
cursor: 'pointer',
|
||||
transition: 'background var(--dur-fast) var(--ease), color var(--dur-fast) var(--ease)',
|
||||
}}
|
||||
onMouseEnter={(e) => { if (!selected) e.currentTarget.style.color = 'var(--fg)'; }}
|
||||
onMouseLeave={(e) => { if (!selected) e.currentTarget.style.color = 'var(--fg-mute)'; }}
|
||||
>
|
||||
{opt.label}
|
||||
</button>
|
||||
);
|
||||
})}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
63
docs/redesign/manifold-export/components/core/Slider.jsx
Normal file
63
docs/redesign/manifold-export/components/core/Slider.jsx
Normal file
|
|
@ -0,0 +1,63 @@
|
|||
import React from 'react';
|
||||
|
||||
/**
|
||||
* Manifold Slider — labeled horizontal range with a glowing orange thumb and
|
||||
* a tabular value readout. Controlled via value/onChange (0..max).
|
||||
*/
|
||||
export function Slider({
|
||||
label,
|
||||
value = 0,
|
||||
min = 0,
|
||||
max = 1,
|
||||
step = 0.01,
|
||||
unit = '',
|
||||
onChange,
|
||||
disabled = false,
|
||||
format,
|
||||
style,
|
||||
}) {
|
||||
const pct = max > min ? (value - min) / (max - min) : 0;
|
||||
const display = format ? format(value) : (Number.isInteger(step) ? value : value.toFixed(2));
|
||||
|
||||
return (
|
||||
<div style={{ display: 'flex', flexDirection: 'column', gap: 'var(--sp-1)', fontFamily: 'var(--font-mono)', userSelect: 'none', opacity: disabled ? 0.5 : 1, pointerEvents: disabled ? 'none' : 'auto', ...style }}>
|
||||
{label && (
|
||||
<span style={{ color: 'var(--fg-mute)', fontSize: 'var(--fs-xs)', textTransform: 'uppercase', letterSpacing: '0.06em' }}>{label}</span>
|
||||
)}
|
||||
<div style={{ display: 'flex', gap: 'var(--sp-3)', alignItems: 'center' }}>
|
||||
<input
|
||||
type="range"
|
||||
min={min}
|
||||
max={max}
|
||||
step={step}
|
||||
value={value}
|
||||
disabled={disabled}
|
||||
onChange={(e) => onChange && onChange(parseFloat(e.target.value))}
|
||||
style={{
|
||||
flex: 1,
|
||||
WebkitAppearance: 'none',
|
||||
appearance: 'none',
|
||||
background: 'transparent',
|
||||
height: 24,
|
||||
margin: 0,
|
||||
cursor: 'pointer',
|
||||
'--mf-pct': `${pct}`,
|
||||
}}
|
||||
className="mf-slider-input"
|
||||
/>
|
||||
<span style={{ fontVariantNumeric: 'tabular-nums', fontSize: 'var(--fs-xs)', color: 'var(--fg-mute)', minWidth: '4ch', textAlign: 'right' }}>
|
||||
{display}{unit && <span style={{ color: 'var(--fg-dim)', marginLeft: 2 }}>{unit}</span>}
|
||||
</span>
|
||||
</div>
|
||||
<style dangerouslySetInnerHTML={{ __html: `
|
||||
.mf-slider-input::-webkit-slider-runnable-track { height: 4px; border-radius: 999px; background: linear-gradient(to right, var(--accent) 0%, var(--accent) calc(var(--mf-pct) * 100%), var(--bg-3) 0%); }
|
||||
.mf-slider-input::-moz-range-track { height: 4px; border-radius: 999px; background: var(--bg-3); }
|
||||
.mf-slider-input::-webkit-slider-thumb { -webkit-appearance: none; appearance: none; width: 16px; height: 16px; border-radius: 50%; background: var(--accent); margin-top: -6px; box-shadow: 0 0 8px var(--glow-accent); cursor: pointer; transition: transform var(--dur-fast) var(--ease); }
|
||||
.mf-slider-input::-moz-range-thumb { width: 16px; height: 16px; border-radius: 50%; background: var(--accent); border: none; box-shadow: 0 0 8px var(--glow-accent); }
|
||||
.mf-slider-input:hover::-webkit-slider-thumb { transform: scale(1.15); }
|
||||
.mf-slider-input:focus { outline: none; }
|
||||
.mf-slider-input:focus::-webkit-slider-thumb { box-shadow: 0 0 0 3px var(--glow-focus); }
|
||||
` }} />
|
||||
</div>
|
||||
);
|
||||
}
|
||||
43
docs/redesign/manifold-export/components/core/StatusLine.jsx
Normal file
43
docs/redesign/manifold-export/components/core/StatusLine.jsx
Normal file
|
|
@ -0,0 +1,43 @@
|
|||
import React from 'react';
|
||||
|
||||
/**
|
||||
* Manifold StatusLine — the dim mono readout strip at the bottom of a mode.
|
||||
* Pass an array of items; strings render plain, {label,value,tone} render a
|
||||
* labeled readout. Items are joined with the house middle-dot separator.
|
||||
*/
|
||||
export function StatusLine({ items = [], style }) {
|
||||
return (
|
||||
<p
|
||||
style={{
|
||||
display: 'flex',
|
||||
flexWrap: 'wrap',
|
||||
alignItems: 'center',
|
||||
gap: 'var(--sp-2)',
|
||||
margin: 0,
|
||||
fontFamily: 'var(--font-mono)',
|
||||
fontSize: 'var(--fs-xs)',
|
||||
color: 'var(--fg-dim)',
|
||||
}}
|
||||
>
|
||||
{items.map((it, i) => {
|
||||
const isObj = it && typeof it === 'object';
|
||||
const toneColor = isObj && it.tone
|
||||
? { accent: 'var(--accent)', cyan: 'var(--accent-2)', good: 'var(--good)', warn: 'var(--warn)', bad: 'var(--bad)' }[it.tone]
|
||||
: null;
|
||||
return (
|
||||
<React.Fragment key={i}>
|
||||
{i > 0 && <span aria-hidden="true">·</span>}
|
||||
{isObj ? (
|
||||
<span style={{ color: toneColor || 'var(--fg-dim)' }}>
|
||||
{it.label && <span style={{ color: 'var(--fg-dim)' }}>{it.label} </span>}
|
||||
<span style={{ fontVariantNumeric: 'tabular-nums', color: toneColor || 'var(--fg-mute)' }}>{it.value}</span>
|
||||
</span>
|
||||
) : (
|
||||
<span>{it}</span>
|
||||
)}
|
||||
</React.Fragment>
|
||||
);
|
||||
})}
|
||||
</p>
|
||||
);
|
||||
}
|
||||
44
docs/redesign/manifold-export/components/core/Switch.jsx
Normal file
44
docs/redesign/manifold-export/components/core/Switch.jsx
Normal file
|
|
@ -0,0 +1,44 @@
|
|||
import React from 'react';
|
||||
|
||||
/**
|
||||
* Manifold Switch — compact toggle. On = orange track + glow. Optional label.
|
||||
*/
|
||||
export function Switch({ checked = false, onChange, label, disabled = false, style }) {
|
||||
return (
|
||||
<label style={{ display: 'inline-flex', alignItems: 'center', gap: 'var(--sp-2)', fontFamily: 'var(--font-mono)', fontSize: 'var(--fs-sm)', color: 'var(--fg)', cursor: disabled ? 'not-allowed' : 'pointer', opacity: disabled ? 0.5 : 1, userSelect: 'none', ...style }}>
|
||||
<button
|
||||
type="button"
|
||||
role="switch"
|
||||
aria-checked={checked}
|
||||
disabled={disabled}
|
||||
onClick={() => onChange && onChange(!checked)}
|
||||
style={{
|
||||
position: 'relative',
|
||||
width: 36,
|
||||
height: 20,
|
||||
padding: 0,
|
||||
borderRadius: 'var(--r-pill)',
|
||||
border: `1px solid ${checked ? 'var(--accent)' : 'var(--line)'}`,
|
||||
background: checked ? 'var(--accent)' : 'var(--bg-2)',
|
||||
cursor: disabled ? 'not-allowed' : 'pointer',
|
||||
transition: 'background var(--dur-fast) var(--ease), border-color var(--dur-fast) var(--ease)',
|
||||
boxShadow: checked ? '0 0 8px var(--glow-accent)' : 'none',
|
||||
}}
|
||||
>
|
||||
<span
|
||||
style={{
|
||||
position: 'absolute',
|
||||
top: 2,
|
||||
left: checked ? 18 : 2,
|
||||
width: 14,
|
||||
height: 14,
|
||||
borderRadius: '50%',
|
||||
background: checked ? 'var(--bg)' : 'var(--fg-mute)',
|
||||
transition: 'left var(--dur-fast) var(--ease), background var(--dur-fast) var(--ease)',
|
||||
}}
|
||||
/>
|
||||
</button>
|
||||
{label && <span>{label}</span>}
|
||||
</label>
|
||||
);
|
||||
}
|
||||
66
docs/redesign/manifold-export/components/data/CurvePlot.jsx
Normal file
66
docs/redesign/manifold-export/components/data/CurvePlot.jsx
Normal file
|
|
@ -0,0 +1,66 @@
|
|||
import React from 'react';
|
||||
|
||||
const clamp01 = (v) => (v < 0 ? 0 : v > 1 ? 1 : v);
|
||||
|
||||
const CURVES = {
|
||||
linear: (x) => x,
|
||||
exp: (x) => (Math.exp(4 * x) - 1) / (Math.exp(4) - 1),
|
||||
log: (x) => Math.log(1 + x * (Math.exp(4) - 1)) / 4,
|
||||
square: (x) => x * x,
|
||||
sqrt: (x) => Math.sqrt(clamp01(x)),
|
||||
sigmoid: (x) => { const s = (v) => 1 / (1 + Math.exp(-(v - 0.5) * 8)); const lo = s(0), hi = s(1); return (s(x) - lo) / (hi - lo); },
|
||||
cubic: (x) => { const v = clamp01(x); return v * v * (3 - 2 * v); },
|
||||
centered_power: (x) => { const o = x - 0.5, sg = o < 0 ? -1 : 1; return clamp01((sg * Math.pow(Math.abs(o) * 2, 0.5)) / 2 + 0.5); },
|
||||
};
|
||||
|
||||
/**
|
||||
* Manifold CurvePlot — renders one of the named response curves (or a custom
|
||||
* function f:[0,1]→[0,1]) on the dark grid. The brand's straight-line &
|
||||
* parabolic/bézier motif.
|
||||
*/
|
||||
export function CurvePlot({ curve = 'cubic', fn, width = 200, height = 120, color = 'var(--accent)', showAxes = true, ariaLabel, style }) {
|
||||
const ref = React.useRef(null);
|
||||
|
||||
React.useEffect(() => {
|
||||
const cv = ref.current;
|
||||
if (!cv) return;
|
||||
const dpr = window.devicePixelRatio || 1;
|
||||
const w = width * dpr, h = height * dpr;
|
||||
cv.width = w; cv.height = h;
|
||||
const ctx = cv.getContext('2d');
|
||||
if (!ctx) return;
|
||||
ctx.clearRect(0, 0, w, h);
|
||||
const cs = getComputedStyle(cv);
|
||||
const resolve = (c) => { ctx.fillStyle = c; return ctx.fillStyle; };
|
||||
const stroke = color.startsWith('var(') ? cs.getPropertyValue(color.slice(4, -1).trim()).trim() || '#ff6a00' : color;
|
||||
const pad = 6 * dpr;
|
||||
|
||||
if (showAxes) {
|
||||
ctx.strokeStyle = 'rgba(255,255,255,0.06)';
|
||||
ctx.lineWidth = 1;
|
||||
ctx.strokeRect(0.5, 0.5, w - 1, h - 1);
|
||||
ctx.beginPath(); ctx.moveTo(0, h / 2); ctx.lineTo(w, h / 2); ctx.stroke();
|
||||
ctx.beginPath(); ctx.moveTo(w / 2, 0); ctx.lineTo(w / 2, h); ctx.stroke();
|
||||
}
|
||||
const f = fn || CURVES[curve] || CURVES.linear;
|
||||
ctx.strokeStyle = stroke;
|
||||
ctx.lineWidth = 2 * dpr;
|
||||
ctx.beginPath();
|
||||
for (let p = 0; p <= 120; p++) {
|
||||
const x = p / 120, y = clamp01(f(x));
|
||||
const px = pad + x * (w - 2 * pad);
|
||||
const py = (h - pad) - y * (h - 2 * pad);
|
||||
p === 0 ? ctx.moveTo(px, py) : ctx.lineTo(px, py);
|
||||
}
|
||||
ctx.stroke();
|
||||
}, [curve, fn, width, height, color, showAxes]);
|
||||
|
||||
return (
|
||||
<canvas
|
||||
ref={ref}
|
||||
role="img"
|
||||
aria-label={ariaLabel || `${curve} curve`}
|
||||
style={{ display: 'block', width, height, background: 'var(--bg-1)', border: '1px solid var(--line)', borderRadius: 'var(--r-2)', ...style }}
|
||||
/>
|
||||
);
|
||||
}
|
||||
62
docs/redesign/manifold-export/components/data/Sparkline.jsx
Normal file
62
docs/redesign/manifold-export/components/data/Sparkline.jsx
Normal file
|
|
@ -0,0 +1,62 @@
|
|||
import React from 'react';
|
||||
|
||||
/**
|
||||
* Manifold Sparkline — a compact time-series trace (training loss, a feature
|
||||
* envelope). Cyan line on a faint grid, with an optional last-value readout.
|
||||
*/
|
||||
export function Sparkline({ data = [], width = 320, height = 70, color = 'var(--accent-2)', log = false, showLast = true, format, ariaLabel = 'time series', style }) {
|
||||
const ref = React.useRef(null);
|
||||
|
||||
React.useEffect(() => {
|
||||
const cv = ref.current;
|
||||
if (!cv) return;
|
||||
const dpr = window.devicePixelRatio || 1;
|
||||
const w = width * dpr, h = height * dpr;
|
||||
cv.width = w; cv.height = h;
|
||||
const ctx = cv.getContext('2d');
|
||||
if (!ctx) return;
|
||||
ctx.clearRect(0, 0, w, h);
|
||||
if (!data.length) return;
|
||||
|
||||
const cs = getComputedStyle(cv);
|
||||
const stroke = color.startsWith('var(') ? cs.getPropertyValue(color.slice(4, -1).trim()).trim() || '#00ccff' : color;
|
||||
|
||||
const ys = data.map((v) => (log ? Math.log(Math.max(1e-10, v) + 1) : v));
|
||||
let lo = Infinity, hi = -Infinity;
|
||||
for (const y of ys) { if (y < lo) lo = y; if (y > hi) hi = y; }
|
||||
if (hi === lo) hi = lo + 1e-6;
|
||||
|
||||
ctx.strokeStyle = 'rgba(255,255,255,0.05)';
|
||||
ctx.lineWidth = 1;
|
||||
for (let i = 1; i < 4; i++) { const y = (i / 4) * h; ctx.beginPath(); ctx.moveTo(0, y); ctx.lineTo(w, y); ctx.stroke(); }
|
||||
|
||||
ctx.strokeStyle = stroke;
|
||||
ctx.lineWidth = 1.5 * dpr;
|
||||
ctx.beginPath();
|
||||
for (let i = 0; i < ys.length; i++) {
|
||||
const x = (i / Math.max(1, ys.length - 1)) * w;
|
||||
const norm = (ys[i] - lo) / (hi - lo);
|
||||
const y = h - norm * h;
|
||||
i === 0 ? ctx.moveTo(x, y) : ctx.lineTo(x, y);
|
||||
}
|
||||
ctx.stroke();
|
||||
|
||||
if (showLast) {
|
||||
const last = data[data.length - 1];
|
||||
const txt = format ? format(last) : (typeof last === 'number' ? last.toExponential(2) : String(last));
|
||||
ctx.fillStyle = '#9a9a9a';
|
||||
ctx.font = `${10 * dpr}px ui-monospace, monospace`;
|
||||
ctx.textAlign = 'right';
|
||||
ctx.fillText(txt, w - 4 * dpr, 12 * dpr);
|
||||
}
|
||||
}, [data, width, height, color, log, showLast]);
|
||||
|
||||
return (
|
||||
<canvas
|
||||
ref={ref}
|
||||
role="img"
|
||||
aria-label={ariaLabel}
|
||||
style={{ display: 'block', width, height, background: 'var(--bg-1)', border: '1px solid var(--line)', borderRadius: 'var(--r-2)', ...style }}
|
||||
/>
|
||||
);
|
||||
}
|
||||
101
docs/redesign/manifold-export/readme.md
Normal file
101
docs/redesign/manifold-export/readme.md
Normal file
|
|
@ -0,0 +1,101 @@
|
|||
# Manifold — Design System
|
||||
|
||||
> Train machine learning, on the fly, to play your instruments.
|
||||
|
||||
Manifold is an **open-source project and a family of commercial products** that train and run on-the-fly machine learning (small MLPs and related techniques) to **program, control, patch, and perform with electronic musical instruments** — synths, samplers, sequencers, Eurorack, and software plugins.
|
||||
|
||||
The shape of the system is always the same three stages:
|
||||
|
||||
```
|
||||
INPUTS ENGINE OUTPUTS
|
||||
joystick ┐ ┌ MIDI
|
||||
XY pad ├──▶ on-the-fly ML ──▶ ├ CV / gate
|
||||
trackpad ┘ (MLP / regression) └ OSC → devices
|
||||
(hw + sw, every hardware input has a software visualisation)
|
||||
```
|
||||
|
||||
A performer moves an **input** (a joystick, an XY pad, a trackpad — some hardware, some software, all with live software visualisations). The signal runs through the **engine**, a tiny neural net trained live by the player's own thumbs-up / thumbs-down feedback. The result is sent to the **world** as MIDI, CV, or OSC to control other devices.
|
||||
|
||||
The reference application is the **MEMLNaut playground** — a browser shell hosting nine "modes", each pairing an input surface with a synthesis/effects engine (PAF formant synth, channel strip, granular morph, reverb, drum/breakbeat, sound-analysis→MIDI, and more).
|
||||
|
||||
## Sources given
|
||||
|
||||
- **Codebase** `playground/` — the MEMLNaut playground, a Vite + **SolidJS** + TypeScript app. ML inference runs in-browser via a WASM module (`public/nisps.wasm`); audio synthesis runs in an AudioWorklet. Key paths:
|
||||
- `src/styles/tokens.css` — the original design tokens (dark canvas, mono, warm+cool accents). **This system is a faithful port + extension of that file.**
|
||||
- `src/primitives/` — the control + visualisation primitives: `XYPad`, `VirtualJoystick`, `Slider`, `PillToggle`, `ControlAxis`, `TrainingControls`, `LossPlot`, `Heatmap`, `JoyMap`, `OutputDisplay`, `GradientFlow`, `ProgressRing`, `SliderBank`, `WeightHealth`, `Drawer`, `ParamEditor`.
|
||||
- `src/modes/` — nine modes + `ModeShell` (the shared scaffold) + `ModeSwitcher`.
|
||||
- `src/output/curves.ts` — the named-curve catalog (`linear, exp, log, square, sqrt, sigmoid, cubic, centered_power`) — a core brand motif (straight lines & parabolic / polynomial / bézier curves).
|
||||
|
||||
There were no Figma links, slide decks, or logo binaries in the source — the brand wordmark in the app is the plain text **`MEMLNaut`** + dim `playground`. See ICONOGRAPHY for how the mark is treated and Caveats at the bottom.
|
||||
|
||||
---
|
||||
|
||||
## CONTENT FUNDAMENTALS
|
||||
|
||||
How Manifold writes. The voice is a **terminal operator's notebook**: lowercase, terse, technical, unfussy. It trusts the reader is a musician *and* a tinkerer.
|
||||
|
||||
- **Casing.** UI labels and routes are **lowercase** (`home`, `/modes`, `/dev/primitives`). Micro-labels above controls are **UPPERCASE with wide letter-spacing** (`EXAMPLES`, `LOSS`, `BOLDNESS`). Mode names are Title Case (`PAF Synth`, `Channel Strip`). Sentences in prose are sentence case.
|
||||
- **Person.** Mostly **imperative, addressed to "you"** — *"pick a mode and start playing"*, *"Drag to sculpt formants"*, *"start it from inside any mode"*. Rarely first-person.
|
||||
- **Tone.** Plain, confident, slightly hacker. States facts and constraints without apology: *"Audio cannot autoplay (browser policy)."*, *"ML inference runs on the main thread via WASM."* Comments in code carry a dry wit (*"Amnesia ↔ Elephant"*, *"Stream 9: nine modes, one shell."*).
|
||||
- **Density.** Pack real values, not filler. Readouts are always shown with units and tabular numerals: `loss 4.2e-3`, `input (0.62, 0.41)`, `noise 0.018`, `33 in → 1 out`.
|
||||
- **Verbs of the domain.** train, randomise (British -ise), explore, pin, snapshot, undo, freeze, converge, route. Feedback is **thumbs-up / thumbs-down** (rendered `+` / `−`).
|
||||
- **Punctuation.** Middle dot `·` separates inline metadata (`engine: paf_synth · 33 in → 1 out`). Arrows `→` `↔` `▶` `⏹` `↶` carry meaning. Ellipsis `…` for in-progress states (`training…`).
|
||||
- **Emoji.** Used **only as functional glyphs** in dense toolbars where an icon font is absent — `🎤` mic, `⚙` settings, `📚` history, `▶ ⏹` transport. Never decorative, never in prose. Prefer geometric Unicode/SVG over emoji where possible (see ICONOGRAPHY).
|
||||
- **British spelling** (randomise, visualisation, colour-adjacent terms) — the project's origin.
|
||||
|
||||
Examples (verbatim from the product):
|
||||
|
||||
> *"Interactive ML control of audio. Stream 9: nine modes, one shell."*
|
||||
> *"Drag to sculpt formants. Voice space: **Bright**"*
|
||||
> *"pick a mode and start playing"*
|
||||
|
||||
---
|
||||
|
||||
## VISUAL FOUNDATIONS
|
||||
|
||||
The look is a **vintage-computer / instrument-panel terminal**: near-black canvas, monospace everywhere, hairline borders, and **glow** instead of drop-shadow to signal what's alive.
|
||||
|
||||
- **Palette.** A dark greyscale stack (`#0d0d0d → #141414 → #1c1c1c → #242424`) for canvas → panel → raised → track. Text steps down `#e8e8e8 → #9a9a9a → #5a5a5a`. **Two accents do all the work:** warm **orange `#ff6a00`** (the primary — actions, focus, the live control dot) and cool **cyan `#00ccff`** (data, plots, the secondary dot). Semantic green/amber/red/blue are reserved for training health (loss, gradient flow). Five translucent **pin** colors mark regions on the joy-map.
|
||||
- **Type.** **JetBrains Mono** is the hero face — used for ~everything, including titles. A system sans is the quiet fallback for long prose only. Fixed px scale (`11 / 13 / 15 / 18 / 24`). Values use `font-variant-numeric: tabular-nums`. Micro-labels are 11px uppercase, `letter-spacing: 0.08em`, in `--fg-mute`.
|
||||
- **Borders.** Everything is outlined with a **1px hairline** `#2a2a2a`. Stronger `#3a3a3a` for internal grid lines and dashed guides (the joystick boundary is a 1px **dashed** circle). Borders, not shadows, define cards.
|
||||
- **Backgrounds.** Flat solids only — **no gradients, no images, no texture** on surfaces. The only "imagery" is generated: canvas-drawn plots, heatmaps, and the named math curves. The motif is **line + curve** (a loss trace, a parabola, a bézier), never photography or illustration.
|
||||
- **Glow, not shadow.** Live, draggable elements carry a colored **box-shadow halo** in their accent (`0 0 8–12px rgba(accent, .45)`): the XY-pad dot (cyan), the joystick knob (orange), slider thumbs. Drop-shadows (`--shadow-1/2`) appear only on floating chrome (drawers, popovers). Depth is communicated by the bg-step ladder, not elevation.
|
||||
- **Radius.** Tight and consistent: `4px` (buttons, inputs, chips), `8px` (panels, pads, cards), `14px` (drawers), `999px` (pills/segmented controls). XY pad = 8px square; joystick = full circle.
|
||||
- **Cards / panels.** `--bg-1` fill + 1px `--line` border + 8px radius + ~12–16px padding. No shadow. A header row is separated by a 1px bottom border; uppercase micro-labels sit above their values.
|
||||
- **Hover.** Surfaces lighten one bg step (`--bg-2 → --bg-3`) and the border goes `--line → --line-strong`. Text-only controls brighten `--fg-mute → --fg`. ~120ms.
|
||||
- **Press / active.** The selected pill/tab fills **solid orange with `--bg` text**. Active nav gets an orange border + orange text. Slider thumbs scale `1.15` on hover. No bounce.
|
||||
- **Focus.** Accent border + a **3px orange glow ring** (`0 0 0 3px rgba(255,106,0,.3)`) — visible and on-brand, replacing the browser outline.
|
||||
- **Motion.** One house easing `cubic-bezier(.25,.8,.35,1)`; durations `120 / 220 / 360ms`. Transitions are **fades and slides** (drawers slide in from the side). No spring, no bounce, no infinite decorative loops — the only continuous motion is *data* (a loss plot advancing, a live readout updating).
|
||||
- **Transparency / blur.** Used sparingly: translucent pin fills on the joy-map; faint white grid lines on plots (`rgba(255,255,255,0.05)`). No backdrop-blur glass.
|
||||
- **Layout.** Dense grids. The mode shell is a 4-row grid: header / body (input + map) / controls (axes + training) / status line. Hit targets ≥ 44px; primary training buttons are 48px tall. Everything is left-aligned and grid-gapped, never centered-flowing.
|
||||
- **Imagery vibe.** Cool and synthetic — black background, neon-on-dark plots, pixelated heatmaps. No warmth, grain, or photography.
|
||||
|
||||
---
|
||||
|
||||
## ICONOGRAPHY
|
||||
|
||||
Manifold's iconography is **geometric and minimal — drawn from the control surfaces themselves**, not from a packaged icon set.
|
||||
|
||||
- **No bundled icon font or SVG sprite** exists in the codebase. The product leans on three things, in order of preference:
|
||||
1. **Functional Unicode glyphs** rendered in the mono face — transport `▶ ⏹`, undo `↶`, arrows `→ ↔`, math `+ − ×`, middle-dot `·`. These match the terminal aesthetic and need no assets.
|
||||
2. **A few emoji as utility icons** in dense toolbars only — `🎤` mic, `⚙` settings/drawer, `📚` snapshot history. Treat these as placeholders for a future geometric set; never use emoji decoratively or in prose.
|
||||
3. **Canvas-drawn "live" iconography** — the real visual language is the *instrument itself*: the XY-pad crosshair + glowing dot, the joystick's dashed boundary + knob, the loss sparkline, the layer-norm bars, the heatmap. When you need to represent a Manifold concept, draw the control, don't reach for a metaphor icon.
|
||||
- **Geometric system, if you must add icons.** Use a thin, single-weight, square-cornered line set. The closest CDN match is **[Lucide](https://lucide.dev)** (2px stroke, geometric, open) or **[Phosphor](https://phosphoricons.com)** (`regular` weight). This system **substitutes Lucide via CDN** for kit screens that need wayfinding icons (settings, close, chevrons) — flagged as a substitution; swap for a bespoke set when one exists. See `assets/icons/` for the curated subset and `assets/README.md`.
|
||||
- **The logo / wordmark.** The only mark in the source is the text lockup **`MEMLNaut`** (bold, `--accent` orange) beside a dim lowercase `playground`, set in JetBrains Mono. There is no symbol/glyph logo. The brand name **Manifold** is set the same way: bold mono wordmark, optionally with the orange dot of a live control as the "o". See `assets/` for the recreated lockups.
|
||||
- **Emoji policy:** functional-only, toolbar-only. Default to Unicode geometry or Lucide. Never emoji in headings, marketing, or body copy.
|
||||
|
||||
---
|
||||
|
||||
## Index / manifest
|
||||
|
||||
- `styles.css` — global entry (link this). `@import` manifest only.
|
||||
- `tokens/` — `colors.css`, `typography.css`, `spacing.css`, `effects.css`, `fonts.css`, `base.css`.
|
||||
- `assets/` — recreated wordmarks, curated Lucide icon subset, README. (See Caveats.)
|
||||
- `guidelines/` — foundation specimen cards (Type / Colors / Spacing / Brand) shown on the Design System tab.
|
||||
- `components/` — reusable React primitives (see list below) + per-group cards.
|
||||
- `ui_kits/memlnaut/` — full-screen recreation of the MEMLNaut playground mode shell. Plus `ui_kits/memlnaut/console/` — the **Playground 2.0 Console** main view (full-bleed manifold + verdict cluster + right dock with depth drawers).
|
||||
- `SKILL.md` — Agent-Skill front-matter for downloadable use.
|
||||
|
||||
**Components** (`window.ManifoldDesignSystem_490915.*`): Button, PillToggle, Slider, Panel, Badge, Switch, StatusLine · XYPad, VirtualJoystick, ControlAxis · CurvePlot, Sparkline.
|
||||
|
||||
**Caveats / open questions** — see the chat summary; the wordmark and icon set are recreations/substitutions pending real brand assets.
|
||||
12
docs/redesign/manifold-export/styles.css
Normal file
12
docs/redesign/manifold-export/styles.css
Normal file
|
|
@ -0,0 +1,12 @@
|
|||
/**
|
||||
* Manifold Design System — global entry point.
|
||||
* Consumers link THIS file. It is an @import manifest only; never put rules
|
||||
* directly here. Everything reachable from these imports ships to consumers.
|
||||
*/
|
||||
|
||||
@import url('./tokens/fonts.css');
|
||||
@import url('./tokens/colors.css');
|
||||
@import url('./tokens/typography.css');
|
||||
@import url('./tokens/spacing.css');
|
||||
@import url('./tokens/effects.css');
|
||||
@import url('./tokens/base.css');
|
||||
51
docs/redesign/manifold-export/tokens/base.css
Normal file
51
docs/redesign/manifold-export/tokens/base.css
Normal file
|
|
@ -0,0 +1,51 @@
|
|||
/**
|
||||
* Manifold — base element styles.
|
||||
* Mirrors the playground's global resets so specimen cards and UI kits read
|
||||
* like the real product even before a single component mounts.
|
||||
*/
|
||||
|
||||
* {
|
||||
box-sizing: border-box;
|
||||
}
|
||||
|
||||
html, body {
|
||||
margin: 0;
|
||||
padding: 0;
|
||||
background: var(--bg);
|
||||
color: var(--fg);
|
||||
font-family: var(--font-mono);
|
||||
font-size: var(--fs-md);
|
||||
line-height: var(--lh-normal);
|
||||
-webkit-tap-highlight-color: transparent;
|
||||
-webkit-font-smoothing: antialiased;
|
||||
}
|
||||
|
||||
a {
|
||||
color: var(--accent-2);
|
||||
text-decoration: none;
|
||||
}
|
||||
a:hover {
|
||||
text-decoration: underline;
|
||||
}
|
||||
|
||||
code, kbd {
|
||||
font-family: var(--font-mono);
|
||||
}
|
||||
|
||||
::selection {
|
||||
background: var(--selection-bg);
|
||||
color: var(--selection-text);
|
||||
}
|
||||
|
||||
/* Uppercase micro-label helper used across the system. */
|
||||
.mf-label {
|
||||
font-size: var(--fs-xs);
|
||||
color: var(--fg-mute);
|
||||
text-transform: uppercase;
|
||||
letter-spacing: var(--ls-label);
|
||||
}
|
||||
|
||||
/* Tabular numerals for any live readout. */
|
||||
.mf-num {
|
||||
font-variant-numeric: tabular-nums;
|
||||
}
|
||||
72
docs/redesign/manifold-export/tokens/colors.css
Normal file
72
docs/redesign/manifold-export/tokens/colors.css
Normal file
|
|
@ -0,0 +1,72 @@
|
|||
/**
|
||||
* Manifold — color tokens
|
||||
* Dark terminal canvas, warm-orange primary, cool-cyan secondary.
|
||||
* Ported from the MEMLNaut playground (src/styles/tokens.css) and extended
|
||||
* with semantic aliases.
|
||||
*/
|
||||
|
||||
:root {
|
||||
/* ---- Surfaces (dark, layered) ---- */
|
||||
--bg: #0d0d0d; /* app canvas */
|
||||
--bg-1: #141414; /* panel / card */
|
||||
--bg-2: #1c1c1c; /* raised control */
|
||||
--bg-3: #242424; /* hover / track */
|
||||
|
||||
/* ---- Foreground / text ---- */
|
||||
--fg: #e8e8e8; /* primary text */
|
||||
--fg-mute: #9a9a9a; /* secondary text / labels */
|
||||
--fg-dim: #5a5a5a; /* tertiary / disabled */
|
||||
|
||||
/* ---- Lines / borders ---- */
|
||||
--line: #2a2a2a; /* default 1px hairline */
|
||||
--line-strong: #3a3a3a; /* grid lines, dashed guides */
|
||||
|
||||
/* ---- Accents ---- */
|
||||
--accent: #ff6a00; /* warm primary — actions, focus, the live dot */
|
||||
--accent-2: #00ccff; /* cool secondary — data, plots, secondary dot */
|
||||
--accent-3: #ffa860; /* warm hover / tint */
|
||||
|
||||
/* ---- Semantic ---- */
|
||||
--good: #6bc26b;
|
||||
--warn: #f5c45e;
|
||||
--bad: #ef5b5b;
|
||||
--info: #5b9eef;
|
||||
|
||||
/* ---- Console 2.0 surface language ---- */
|
||||
--danger: #ff4466; /* the 2.0 verdict-perturb / destructive red */
|
||||
--glass: rgba(13, 13, 13, 0.65); /* frosted chrome over the manifold */
|
||||
--glass-line: rgba(255, 255, 255, 0.07); /* hairline on glass */
|
||||
|
||||
/* ---- Region pins (translucent map markers) ---- */
|
||||
--pin-1: rgba(255, 106, 0, 0.25);
|
||||
--pin-2: rgba(0, 204, 255, 0.25);
|
||||
--pin-3: rgba(180, 100, 255, 0.25);
|
||||
--pin-4: rgba(80, 200, 120, 0.25);
|
||||
--pin-5: rgba(255, 200, 80, 0.25);
|
||||
|
||||
/* ---- Glow alphas (for box-shadow halos on live controls) ---- */
|
||||
--glow-accent: rgba(255, 106, 0, 0.45);
|
||||
--glow-accent-2: rgba(0, 204, 255, 0.45);
|
||||
--glow-focus: rgba(255, 106, 0, 0.30);
|
||||
|
||||
/* ============ Semantic aliases ============ */
|
||||
--surface-app: var(--bg);
|
||||
--surface-panel: var(--bg-1);
|
||||
--surface-raised: var(--bg-2);
|
||||
--surface-track: var(--bg-3);
|
||||
|
||||
--text-primary: var(--fg);
|
||||
--text-secondary: var(--fg-mute);
|
||||
--text-tertiary: var(--fg-dim);
|
||||
--text-accent: var(--accent);
|
||||
--text-link: var(--accent-2);
|
||||
|
||||
--border-default: var(--line);
|
||||
--border-strong: var(--line-strong);
|
||||
--border-focus: var(--accent);
|
||||
|
||||
--action-primary: var(--accent);
|
||||
--action-primary-text: var(--bg);
|
||||
--selection-bg: var(--accent);
|
||||
--selection-text: var(--bg);
|
||||
}
|
||||
31
docs/redesign/manifold-export/tokens/effects.css
Normal file
31
docs/redesign/manifold-export/tokens/effects.css
Normal file
|
|
@ -0,0 +1,31 @@
|
|||
/**
|
||||
* Manifold — motion, shadow & glow tokens
|
||||
* Manifold rarely uses drop shadows for depth; instead it uses *glow halos*
|
||||
* on live, interactive elements (the dot on an XY pad, a slider thumb).
|
||||
*/
|
||||
|
||||
:root {
|
||||
/* ---- Motion ---- */
|
||||
--ease: cubic-bezier(.25, .8, .35, 1); /* @kind other */
|
||||
--ease-out: cubic-bezier(.16, 1, .3, 1); /* @kind other */
|
||||
--ease-console: cubic-bezier(0.22, 1, 0.36, 1); /* @kind other */ /* the 2.0 Console drawer/chrome easing */
|
||||
--dur-fast: 120ms; /* @kind other */
|
||||
--dur-med: 220ms; /* @kind other */
|
||||
--dur-slow: 360ms; /* @kind other */
|
||||
|
||||
/* ---- Glow halos (the signature) ---- */
|
||||
--glow-sm: 0 0 8px var(--glow-accent);
|
||||
--glow-md: 0 0 12px var(--glow-accent);
|
||||
--glow-lg: 0 0 18px var(--glow-accent);
|
||||
--glow-cyan: 0 0 10px var(--glow-accent-2);
|
||||
--focus-ring: 0 0 0 3px var(--glow-focus);
|
||||
|
||||
/* ---- Shadows (used sparingly: drawers, popovers) ---- */
|
||||
--shadow-1: 0 2px 8px rgba(0, 0, 0, 0.4);
|
||||
--shadow-2: 0 8px 24px rgba(0, 0, 0, 0.5);
|
||||
|
||||
/* ---- Borders ---- */
|
||||
--bw: 1px; /* default hairline */
|
||||
--border: var(--bw) solid var(--line);
|
||||
--border-strong-rule: var(--bw) solid var(--line-strong);
|
||||
}
|
||||
11
docs/redesign/manifold-export/tokens/fonts.css
Normal file
11
docs/redesign/manifold-export/tokens/fonts.css
Normal file
|
|
@ -0,0 +1,11 @@
|
|||
/**
|
||||
* Manifold — webfonts
|
||||
* JetBrains Mono is the brand face. The original codebase referenced it by
|
||||
* name without bundling binaries, so we load it from Google Fonts here.
|
||||
*
|
||||
* SUBSTITUTION NOTE: shipped via Google Fonts CDN (OFL licensed). To self-host,
|
||||
* drop the .woff2 files in assets/fonts/ and replace this @import with
|
||||
* local @font-face rules.
|
||||
*/
|
||||
|
||||
@import url('https://fonts.googleapis.com/css2?family=JetBrains+Mono:ital,wght@0,400;0,500;0,600;0,700;1,400&display=swap');
|
||||
34
docs/redesign/manifold-export/tokens/spacing.css
Normal file
34
docs/redesign/manifold-export/tokens/spacing.css
Normal file
|
|
@ -0,0 +1,34 @@
|
|||
/**
|
||||
* Manifold — spacing, radius, layout, z-index
|
||||
* Compact 4px-based scale (the playground is dense, instrument-panel UI).
|
||||
*/
|
||||
|
||||
:root {
|
||||
/* ---- Spacing (px) ---- */
|
||||
--sp-0: 2px;
|
||||
--sp-1: 4px;
|
||||
--sp-2: 8px;
|
||||
--sp-3: 12px;
|
||||
--sp-4: 16px;
|
||||
--sp-5: 24px;
|
||||
--sp-6: 32px;
|
||||
--sp-7: 48px;
|
||||
--sp-8: 64px;
|
||||
|
||||
/* ---- Radius ---- */
|
||||
--r-1: 4px; /* buttons, inputs, small chips */
|
||||
--r-2: 8px; /* panels, pads, cards */
|
||||
--r-3: 14px; /* large surfaces, drawers */
|
||||
--r-pill: 999px;
|
||||
|
||||
/* ---- Z layers ---- */
|
||||
--z-bg: 0; /* @kind other */
|
||||
--z-content: 10; /* @kind other */
|
||||
--z-overlay: 100; /* @kind other */
|
||||
--z-drawer: 200; /* @kind other */
|
||||
--z-modal: 1000; /* @kind other */
|
||||
|
||||
/* ---- Control sizing ---- */
|
||||
--control-h: 48px; /* training buttons, large hit targets */
|
||||
--hit-min: 44px; /* minimum touch target */
|
||||
}
|
||||
46
docs/redesign/manifold-export/tokens/typography.css
Normal file
46
docs/redesign/manifold-export/tokens/typography.css
Normal file
|
|
@ -0,0 +1,46 @@
|
|||
/**
|
||||
* Manifold — typography tokens
|
||||
* Monospace is the hero (terminal vibe). Sans is a quiet system fallback,
|
||||
* used rarely for long-form prose.
|
||||
*/
|
||||
|
||||
:root {
|
||||
/* ---- Families ---- */
|
||||
--font-mono: 'JetBrains Mono', 'Fira Code', 'SF Mono', Menlo, Consolas, monospace;
|
||||
--font-sans: -apple-system, BlinkMacSystemFont, 'Segoe UI', system-ui, sans-serif;
|
||||
|
||||
/* Hero family alias — Manifold reads almost entirely in mono. */
|
||||
--font-display: var(--font-mono);
|
||||
--font-body: var(--font-mono);
|
||||
--font-prose: var(--font-sans);
|
||||
|
||||
/* ---- Sizes (px, fixed scale from the playground) ---- */
|
||||
--fs-xs: 11px; /* labels, captions, status */
|
||||
--fs-sm: 13px; /* secondary UI text */
|
||||
--fs-md: 15px; /* body / default */
|
||||
--fs-lg: 18px; /* mode titles, emphasis */
|
||||
--fs-xl: 24px; /* page titles */
|
||||
--fs-2xl: 34px; /* hero / display (specimen / marketing) */
|
||||
--fs-3xl: 48px;
|
||||
|
||||
/* ---- Weights ---- */
|
||||
--fw-regular: 400; /* @kind font */
|
||||
--fw-medium: 500; /* @kind font */
|
||||
--fw-semibold: 600; /* @kind font */
|
||||
--fw-bold: 700; /* @kind font */
|
||||
|
||||
/* ---- Line heights ---- */
|
||||
--lh-tight: 1.1; /* @kind other */
|
||||
--lh-snug: 1.3; /* @kind other */
|
||||
--lh-normal: 1.5; /* @kind other */
|
||||
|
||||
/* ---- Letter spacing ---- */
|
||||
--ls-tight: -0.01em; /* @kind other */
|
||||
--ls-normal: 0; /* @kind other */
|
||||
--ls-label: 0.08em; /* @kind other */
|
||||
--ls-wide: 0.12em; /* @kind other */
|
||||
|
||||
/* ---- Semantic label style ---- */
|
||||
--label-transform: uppercase; /* @kind other */
|
||||
--label-spacing: var(--ls-label); /* @kind other */
|
||||
}
|
||||
|
|
@ -0,0 +1,254 @@
|
|||
/* Console 2.0 — CompositeStage: ONE continuous view that becomes inputs-first,
|
||||
outputs-first, or 50/50 by dragging a single divider. No discrete modes — the
|
||||
layout is a single ratio `split` ∈ [0,1] (= the input's share of the width):
|
||||
split → 1 inputs-first (output demotes to a slim readout list, then a minimap)
|
||||
split = 0.5 dual / 50-50
|
||||
split → 0 outputs-first (input demotes to a pad, then a minimap)
|
||||
Pull the seam ALL THE WAY to an edge and the small side snaps shut, popping
|
||||
out as a draggable minimap rectangle that lives in any corner. Each panel
|
||||
chooses its own representation from its MEASURED width, so it never becomes a
|
||||
useless sliver — it demotes. Handle snaps to 0.14·0.33·0.5·0.66·0.86 with
|
||||
light magnetism; presets tween the ratio. */
|
||||
function CompositeStage({ split, onSplit, mode, pos, onMove, noiseCap, pins, follow, onLongPress, params, values, onChange }) {
|
||||
const ref = React.useRef(null);
|
||||
const [size, setSize] = React.useState({ w: 0, h: 0 });
|
||||
const drag = React.useRef(false);
|
||||
const splitRef = React.useRef(split); splitRef.current = split;
|
||||
|
||||
const SNAPS = [0.14, 0.33, 0.5, 0.66, 0.86];
|
||||
const MAGNET = 0.026;
|
||||
const SHUT = 0.1; // pull past this and the small side collapses to a minimap
|
||||
|
||||
// which corner each minimap rests in (persisted)
|
||||
const [inCorner, setInCorner] = React.useState(() => localStorage.getItem('mf-mm-incorner') || 'tl');
|
||||
const [outCorner, setOutCorner] = React.useState(() => localStorage.getItem('mf-mm-outcorner') || 'tl');
|
||||
React.useEffect(() => { localStorage.setItem('mf-mm-incorner', inCorner); }, [inCorner]);
|
||||
React.useEffect(() => { localStorage.setItem('mf-mm-outcorner', outCorner); }, [outCorner]);
|
||||
|
||||
React.useEffect(() => {
|
||||
const el = ref.current; if (!el) return;
|
||||
// Only commit non-zero measurements; some hosts lay out the flex child after
|
||||
// mount, so the first read can be 0. Keep ResizeObserver AND a setTimeout
|
||||
// kick loop (RO is withheld in some preview contexts) until width is real.
|
||||
const read = () => { if (el.clientWidth) setSize({ w: el.clientWidth, h: el.clientHeight }); };
|
||||
const ro = new ResizeObserver(read); ro.observe(el); read();
|
||||
let t = null, k = 0;
|
||||
const kick = () => { read(); if (!el.clientWidth && k++ < 80) t = setTimeout(kick, 40); };
|
||||
kick();
|
||||
return () => { ro.disconnect(); clearTimeout(t); };
|
||||
}, []);
|
||||
|
||||
const setFromClientX = (clientX) => {
|
||||
const el = ref.current; if (!el) return;
|
||||
const r = el.getBoundingClientRect();
|
||||
let f = (clientX - r.left) / r.width;
|
||||
f = Math.max(0, Math.min(1, f));
|
||||
if (f < SHUT) f = 0; // collapse input → minimap
|
||||
else if (f > 1 - SHUT) f = 1; // collapse output → minimap
|
||||
else for (const s of SNAPS) if (Math.abs(f - s) < MAGNET) { f = s; break; }
|
||||
onSplit(f);
|
||||
};
|
||||
const down = (e) => { drag.current = true; e.currentTarget.setPointerCapture?.(e.pointerId); setFromClientX(e.clientX); };
|
||||
const move = (e) => { if (drag.current) setFromClientX(e.clientX); };
|
||||
const up = (e) => { drag.current = false; e.currentTarget.releasePointerCapture?.(e.pointerId); };
|
||||
|
||||
const tweenTo = (target) => {
|
||||
const start = splitRef.current, t0 = performance.now(), dur = 300;
|
||||
const ease = (p) => 1 - Math.pow(1 - p, 3);
|
||||
const step = () => {
|
||||
const p = Math.min(1, (performance.now() - t0) / dur);
|
||||
onSplit(start + (target - start) * ease(p));
|
||||
if (p < 1) requestAnimationFrame(step);
|
||||
};
|
||||
requestAnimationFrame(step);
|
||||
};
|
||||
|
||||
const { w, h } = size;
|
||||
const knownW = w > 0; // until measured, default both sides to their hero tier
|
||||
const collapsed = split <= 0.001 ? 'in' : split >= 0.999 ? 'out' : null;
|
||||
const wIn = w * split, wOut = w * (1 - split);
|
||||
const inTier = (!knownW || wIn >= 300) ? 'full' : 'pad';
|
||||
const outTier = (!knownW || wOut >= 230) ? 'field' : 'list';
|
||||
|
||||
const DS = window.ManifoldDesignSystem_490915;
|
||||
const Pad = mode && mode.input === 'joystick' ? DS.VirtualJoystick : DS.XYPad;
|
||||
const tag = (text, side) => (
|
||||
<div style={{ position: 'absolute', top: 10, [side]: 12, zIndex: 8, pointerEvents: 'none',
|
||||
fontSize: 9, letterSpacing: '0.14em', color: 'var(--fg-dim)', fontFamily: 'var(--font-mono)' }}>{text}</div>
|
||||
);
|
||||
|
||||
// ---- INPUT panel content by tier (non-collapsed) ----
|
||||
const renderInput = () => {
|
||||
if (inTier === 'full') {
|
||||
return <window.Manifold pos={pos} onMove={onMove} noiseCap={noiseCap} pins={pins} follow={follow} onLongPress={onLongPress} />;
|
||||
}
|
||||
const s = Math.max(88, Math.min(wIn - 28, h - 88));
|
||||
return (
|
||||
<div style={{ position: 'absolute', inset: 0, display: 'flex', flexDirection: 'column', alignItems: 'center', justifyContent: 'center', gap: 10 }}>
|
||||
<Pad size={s} position={pos} onMove={(x, y) => onMove(x, y)} showGrid />
|
||||
<span style={{ fontSize: 10, color: 'var(--fg-mute)', fontFamily: 'var(--font-mono)', fontVariantNumeric: 'tabular-nums' }}>
|
||||
{pos[0].toFixed(2)}, {pos[1].toFixed(2)}
|
||||
</span>
|
||||
</div>
|
||||
);
|
||||
};
|
||||
|
||||
// ---- OUTPUT panel content by tier (non-collapsed) ----
|
||||
const GC = { formant: '--accent', pitch: '--accent-2', amp: '--good', filter: '--warn', fx: '--info', mod: '--accent-3' };
|
||||
const renderOutput = () => {
|
||||
if (outTier === 'field') {
|
||||
return <window.OutputStage params={params} values={values} onChange={onChange} compact={wOut < 440} />;
|
||||
}
|
||||
return (
|
||||
<div style={{ position: 'absolute', inset: 0, padding: '34px 10px 12px', display: 'flex', flexDirection: 'column', gap: 4, overflowY: 'auto' }}>
|
||||
{params.map((p, i) => {
|
||||
const eff = values[i] ?? 0, gc = `var(${GC[p.group] || '--accent'})`, dim = p.status === 'off';
|
||||
const set = (cx, el) => { const r = el.getBoundingClientRect(); onChange(i, { val: Math.max(0, Math.min(1, (cx - r.left) / r.width)) }); };
|
||||
return (
|
||||
<div key={i} style={{ display: 'flex', flexDirection: 'column', gap: 2, opacity: dim ? 0.5 : 1 }}>
|
||||
<div style={{ display: 'flex', justifyContent: 'space-between', gap: 6, fontFamily: 'var(--font-mono)' }}>
|
||||
<span style={{ fontSize: 9, color: 'var(--fg-mute)', overflow: 'hidden', whiteSpace: 'nowrap', textOverflow: 'ellipsis' }}>{p.name}</span>
|
||||
<span style={{ fontSize: 9, color: 'var(--fg)', fontVariantNumeric: 'tabular-nums' }}>{eff.toFixed(2)}</span>
|
||||
</div>
|
||||
<div onPointerDown={(e) => { e.currentTarget.setPointerCapture?.(e.pointerId); e.currentTarget._d = true; set(e.clientX, e.currentTarget); }}
|
||||
onPointerMove={(e) => { if (e.currentTarget._d) set(e.clientX, e.currentTarget); }}
|
||||
onPointerUp={(e) => { e.currentTarget._d = false; e.currentTarget.releasePointerCapture?.(e.pointerId); }}
|
||||
style={{ position: 'relative', height: 6, background: 'var(--bg-1)', border: '1px solid var(--line)', borderRadius: 999, cursor: 'ew-resize', touchAction: 'none' }}>
|
||||
<div style={{ position: 'absolute', left: 0, top: 0, bottom: 0, width: `${eff * 100}%`, background: gc, opacity: 0.6, borderRadius: 999 }} />
|
||||
<div style={{ position: 'absolute', left: `${eff * 100}%`, top: -2, width: 2, height: 10, marginLeft: -1, background: gc, boxShadow: `0 0 6px ${gc}` }} />
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
})}
|
||||
</div>
|
||||
);
|
||||
};
|
||||
|
||||
// ---- minimap (the collapsed side as a draggable corner rectangle) ----
|
||||
const CORNERS = { tl: { top: 62, left: 14 }, tr: { top: 62, right: 14 }, bl: { bottom: 14, left: 14 }, br: { bottom: 14, right: 14 } };
|
||||
const [mmPos, setMmPos] = React.useState(null); // {x,y,side} while dragging
|
||||
const mmRef = React.useRef({ side: null });
|
||||
const mmDown = (side) => (e) => {
|
||||
const card = e.currentTarget.closest('[data-mm]'); if (!card || !ref.current) return;
|
||||
e.currentTarget.setPointerCapture?.(e.pointerId);
|
||||
const r = ref.current.getBoundingClientRect(), cr = card.getBoundingClientRect();
|
||||
mmRef.current = { side, dx: e.clientX - cr.left, dy: e.clientY - cr.top, cw: cr.width, ch: cr.height };
|
||||
setMmPos({ x: cr.left - r.left, y: cr.top - r.top, side });
|
||||
};
|
||||
const mmMove = (e) => {
|
||||
const m = mmRef.current; if (!m.side || !ref.current) return;
|
||||
const r = ref.current.getBoundingClientRect();
|
||||
const x = Math.max(8, Math.min(r.width - m.cw - 8, e.clientX - r.left - m.dx));
|
||||
const y = Math.max(8, Math.min(r.height - m.ch - 8, e.clientY - r.top - m.dy));
|
||||
setMmPos({ x, y, side: m.side });
|
||||
};
|
||||
const mmUp = (e) => {
|
||||
const m = mmRef.current; if (!m.side || !ref.current) return;
|
||||
const r = ref.current.getBoundingClientRect(), p = mmPos || { x: 0, y: 0 };
|
||||
const corner = (p.y + m.ch / 2 < r.height / 2 ? 't' : 'b') + (p.x + m.cw / 2 < r.width / 2 ? 'l' : 'r');
|
||||
(m.side === 'in' ? setInCorner : setOutCorner)(corner);
|
||||
mmRef.current = { side: null }; setMmPos(null);
|
||||
e.currentTarget.releasePointerCapture?.(e.pointerId);
|
||||
};
|
||||
|
||||
const miniCard = (side, corner, body) => {
|
||||
const dragging = mmPos && mmPos.side === side;
|
||||
const place = dragging ? { left: mmPos.x, top: mmPos.y } : CORNERS[corner];
|
||||
const restore = () => tweenTo(0.5);
|
||||
return (
|
||||
<div data-mm={side} style={{ position: 'absolute', zIndex: 65, ...place,
|
||||
background: 'var(--glass)', backdropFilter: 'blur(12px)', WebkitBackdropFilter: 'blur(12px)',
|
||||
border: '1px solid var(--glass-line)', borderRadius: 'var(--r-2)', boxShadow: 'var(--shadow-2)',
|
||||
padding: 8, display: 'flex', flexDirection: 'column', gap: 6 }}>
|
||||
{/* header = drag handle */}
|
||||
<div onPointerDown={mmDown(side)} onPointerMove={mmMove} onPointerUp={mmUp} onPointerCancel={mmUp}
|
||||
style={{ display: 'flex', alignItems: 'center', justifyContent: 'space-between', gap: 12, cursor: 'move', touchAction: 'none', userSelect: 'none' }}>
|
||||
<span style={{ fontSize: 9, letterSpacing: '0.12em', color: 'var(--fg-mute)', fontFamily: 'var(--font-mono)' }}>
|
||||
<span style={{ color: 'var(--fg-dim)' }}>⠿</span> {side === 'in' ? 'INPUT' : 'OUTPUT'}
|
||||
</span>
|
||||
<button type="button" onClick={restore} title="expand back to dual"
|
||||
style={{ border: 'none', background: 'transparent', color: 'var(--fg-dim)', cursor: 'pointer', fontSize: 12, lineHeight: 1, padding: 0 }}>⤢</button>
|
||||
</div>
|
||||
{body}
|
||||
</div>
|
||||
);
|
||||
};
|
||||
const miniInput = () => miniCard('in', inCorner, (
|
||||
<Pad size={118} position={pos} onMove={(x, y) => onMove(x, y)} showGrid />
|
||||
));
|
||||
const miniOutput = () => miniCard('out', outCorner, (
|
||||
<div style={{ width: 168 }}>
|
||||
<window.MiniMeters params={params} values={values} />
|
||||
</div>
|
||||
));
|
||||
|
||||
// ---- divider geometry ----
|
||||
const pctIn = Math.round(split * 100);
|
||||
const presetActive = Math.abs(split - 0.86) < 0.04 ? 'in' : Math.abs(split - 0.14) < 0.04 ? 'out' : Math.abs(split - 0.5) < 0.04 ? 'dual' : null;
|
||||
const presetBtn = (id, label, target) => (
|
||||
<button type="button" onClick={() => tweenTo(target)} title={label}
|
||||
style={{ fontSize: 9, letterSpacing: '0.08em', padding: '3px 9px', borderRadius: 'var(--r-pill)', cursor: 'pointer', fontFamily: 'var(--font-mono)',
|
||||
border: `1px solid ${presetActive === id ? 'var(--accent)' : 'transparent'}`,
|
||||
background: presetActive === id ? 'rgba(255,106,0,0.14)' : 'transparent',
|
||||
color: presetActive === id ? 'var(--accent)' : 'var(--fg-dim)' }}>{label}</button>
|
||||
);
|
||||
const handleLeft = split <= 0 ? '0%' : split >= 1 ? '100%' : `${split * 100}%`;
|
||||
const handleMargin = split <= 0 ? 0 : split >= 1 ? -18 : -9;
|
||||
|
||||
return (
|
||||
<div ref={ref} style={{ position: 'absolute', inset: 0, overflow: 'hidden' }}>
|
||||
{collapsed === 'in' ? (
|
||||
<React.Fragment>
|
||||
<div style={{ position: 'absolute', inset: 0 }}>{tag('OUTPUT', 'right')}{renderOutput()}</div>
|
||||
{miniInput()}
|
||||
</React.Fragment>
|
||||
) : collapsed === 'out' ? (
|
||||
<React.Fragment>
|
||||
<div style={{ position: 'absolute', inset: 0 }}>{tag('INPUT', 'left')}{renderInput()}</div>
|
||||
{miniOutput()}
|
||||
</React.Fragment>
|
||||
) : (
|
||||
<React.Fragment>
|
||||
<div style={{ position: 'absolute', top: 0, bottom: 0, left: 0, width: `${split * 100}%`, overflow: 'hidden' }}>
|
||||
{tag('INPUT', 'left')}{renderInput()}
|
||||
</div>
|
||||
<div style={{ position: 'absolute', top: 0, bottom: 0, right: 0, width: `${(1 - split) * 100}%`, borderLeft: '1px solid var(--line)', overflow: 'hidden' }}>
|
||||
{tag('OUTPUT', 'right')}{renderOutput()}
|
||||
</div>
|
||||
{SNAPS.map((s) => (
|
||||
<div key={s} style={{ position: 'absolute', top: 0, left: `${s * 100}%`, width: 1, height: 6, marginLeft: -0.5,
|
||||
background: 'var(--line-strong)', opacity: Math.abs(split - s) < 0.012 ? 0 : 0.6, pointerEvents: 'none', zIndex: 14 }} />
|
||||
))}
|
||||
</React.Fragment>
|
||||
)}
|
||||
|
||||
{/* the divider handle — at the seam, or an edge tab when collapsed */}
|
||||
<div onPointerDown={down} onPointerMove={move} onPointerUp={up} onPointerCancel={up}
|
||||
onDoubleClick={() => tweenTo(presetActive === 'dual' ? 0.86 : presetActive === 'in' ? 0.14 : 0.5)}
|
||||
title={collapsed ? 'pull to reveal' : 'drag to rebalance · double-click to cycle'}
|
||||
style={{ position: 'absolute', top: 0, bottom: 0, left: handleLeft, width: 18, marginLeft: handleMargin, zIndex: 26,
|
||||
cursor: 'col-resize', touchAction: 'none', display: 'flex', alignItems: 'center', justifyContent: 'center' }}>
|
||||
<div style={{ position: 'absolute', top: 0, bottom: 0, left: collapsed === 'out' ? 'auto' : 9, right: collapsed === 'out' ? 9 : 'auto',
|
||||
width: 1, background: 'var(--accent)', opacity: collapsed ? 0.5 : 0.35 }} />
|
||||
<div style={{ width: 6, height: 46, borderRadius: 999, background: 'var(--bg-3)', border: '1px solid var(--line-strong)',
|
||||
display: 'flex', flexDirection: 'column', alignItems: 'center', justifyContent: 'center', gap: 3, boxShadow: '0 0 0 4px var(--bg)' }}>
|
||||
{[0, 1, 2].map((k) => <span key={k} style={{ width: 2, height: 2, borderRadius: '50%', background: 'var(--fg-mute)' }} />)}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{/* presets + ratio readout, top-centre */}
|
||||
<div style={{ position: 'absolute', top: 10, left: '50%', transform: 'translateX(-50%)', zIndex: 16,
|
||||
display: 'flex', alignItems: 'center', gap: 4, background: 'var(--glass)', backdropFilter: 'blur(10px)', WebkitBackdropFilter: 'blur(10px)',
|
||||
border: '1px solid var(--glass-line)', borderRadius: 'var(--r-pill)', padding: '3px 5px' }}>
|
||||
{presetBtn('in', 'input', 0.86)}
|
||||
{presetBtn('dual', 'dual', 0.5)}
|
||||
{presetBtn('out', 'output', 0.14)}
|
||||
<span style={{ width: 1, height: 14, background: 'var(--glass-line)' }} />
|
||||
<span style={{ fontSize: 9, color: 'var(--fg-mute)', fontFamily: 'var(--font-mono)', fontVariantNumeric: 'tabular-nums', padding: '0 4px' }}>
|
||||
in {pctIn} · out {100 - pctIn}
|
||||
</span>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
window.CompositeStage = CompositeStage;
|
||||
|
|
@ -0,0 +1,199 @@
|
|||
/* Console 2.0 — the reactive spine + layout. One input → ML → output flow that
|
||||
every consumer (readout strip, manifold, health glow) reads. Pseudo-inference
|
||||
stands in for the WASM MLP; the structure mirrors the rewrite plan. */
|
||||
const CA_DS = window.ManifoldDesignSystem_490915;
|
||||
|
||||
/* ---- shared model (modes + inference) lives in model.jsx ---- */
|
||||
const MODES2 = window.MF_MODES;
|
||||
const infer = window.MF_infer;
|
||||
const seededGradient = window.MF_seededGradient;
|
||||
|
||||
let SNAP_ID = 0;
|
||||
|
||||
function ConsoleApp({ focus = 'in' }) {
|
||||
const [modeId, setModeId] = React.useState('paf_synth');
|
||||
const mode = MODES2.find((m) => m.id === modeId);
|
||||
const [params, setParams] = React.useState(() => mode.params.map((p) => ({ ...p })));
|
||||
const [pos, setPos] = React.useState([0.5, 0.5]);
|
||||
const [seed, setSeed] = React.useState(0.4);
|
||||
const [axes, setAxes] = React.useState({ boldness: 0.55, memory: 0.4, precision: 0.5 });
|
||||
const [preset, setPreset] = React.useState('Sculpt');
|
||||
const [noiseCap, setNoiseCap] = React.useState(0.12);
|
||||
const [examples, setExamples] = React.useState(0);
|
||||
const [addingExample, setAddingExample] = React.useState(false);
|
||||
const [loss, setLoss] = React.useState([]);
|
||||
const [busy, setBusy] = React.useState(false);
|
||||
const [snapshots, setSnapshots] = React.useState([]);
|
||||
const [ab, setAB] = React.useState('B');
|
||||
const [holdingA, setHoldingA] = React.useState(false);
|
||||
const aRef = React.useRef(null);
|
||||
const [spread, setSpread] = React.useState(false);
|
||||
const [tame, setTame] = React.useState(0.85);
|
||||
const [health, setHealth] = React.useState(0.8);
|
||||
const [rev, setRev] = React.useState(1);
|
||||
const [active, setActive] = React.useState('feel'); // auto-open Feel peek
|
||||
const [depth, setDepth] = React.useState('peek');
|
||||
const [follow, setFollow] = React.useState(false);
|
||||
const [split, setSplit] = React.useState(() => {
|
||||
const v = parseFloat(localStorage.getItem('mf-composite-split'));
|
||||
return Number.isFinite(v) ? Math.max(0, Math.min(1, v)) : 0.5;
|
||||
});
|
||||
React.useEffect(() => { localStorage.setItem('mf-composite-split', String(split)); }, [split]);
|
||||
const [stripPinned, setStripPinned] = React.useState(true);
|
||||
const [firstSession, setFirstSession] = React.useState(true);
|
||||
const [pins, setPins] = React.useState([]);
|
||||
|
||||
// reset transient state on mode switch
|
||||
React.useEffect(() => {
|
||||
setParams(mode.params.map((p) => ({ ...p }))); setPos([0.5, 0.5]); setExamples(0);
|
||||
setLoss([]); setSnapshots([]); setSeed(0.4); setFollow(false); setPins([]);
|
||||
setActive('feel'); setDepth('peek');
|
||||
}, [modeId]);
|
||||
|
||||
const effSeed = holdingA && aRef.current ? aRef.current.seed : seed;
|
||||
const values = React.useMemo(() => infer(pos, effSeed, params, axes), [pos, effSeed, params, axes]);
|
||||
const gradient = React.useMemo(() => seededGradient(rev), [rev]);
|
||||
|
||||
const pushSnap = (tag) => setSnapshots((s) => [...s, { id: ++SNAP_ID, tag, noise: noiseCap, seed }].slice(-50));
|
||||
|
||||
const commit = () => {
|
||||
setFirstSession(false); pushSnap('commit +'); setBusy(true);
|
||||
setNoiseCap((n) => Math.max(0.02, n * 0.7));
|
||||
setHealth((h) => Math.min(1, h + 0.08));
|
||||
setRev((r) => r + 1);
|
||||
setTimeout(() => {
|
||||
setLoss((prev) => { const base = prev.length ? prev[prev.length - 1] : 0.5; return [...prev, Math.max(0.004, base * (0.8 + Math.random() * 0.08))].slice(-120); });
|
||||
setBusy(false);
|
||||
}, 260);
|
||||
};
|
||||
const perturb = () => {
|
||||
setFirstSession(false); pushSnap('perturb −');
|
||||
setSeed((s) => s + (Math.random() - 0.5) * (noiseCap * 4 + 0.3));
|
||||
setNoiseCap((n) => Math.min(0.5, n + 0.06));
|
||||
setHealth((h) => Math.max(0.1, h - 0.06));
|
||||
setRev((r) => r + 1);
|
||||
};
|
||||
const reroll = () => { setFirstSession(false); pushSnap('re-roll'); setSeed(Math.random() * 6); setNoiseCap(0.4); setHealth(0.5); setRev((r) => r + 1); };
|
||||
const undo = () => setSnapshots((s) => {
|
||||
if (!s.length) return s;
|
||||
const last = s[s.length - 1]; setSeed(last.seed); setNoiseCap(last.noise); setRev((r) => r + 1);
|
||||
return s.slice(0, -1);
|
||||
});
|
||||
const train = () => { setBusy(true); setTimeout(() => { setLoss((p) => { const b = p.length ? p[p.length - 1] : 0.5; return [...p, Math.max(0.004, b * 0.82)].slice(-120); }); setBusy(false); }, 260); };
|
||||
const addExample = () => {
|
||||
if (!addingExample) { setAddingExample(true); return; }
|
||||
setAddingExample(false); setExamples((e) => e + 1); pushSnap('example'); train();
|
||||
};
|
||||
|
||||
const setParam = (i, patch) => setParams((ps) => ps.map((p, j) => (j === i ? { ...p, ...patch } : p)));
|
||||
const cycleStatus = (i) => setParams((ps) => ps.map((p, j) => (j === i ? { ...p, status: ({ off: 'fixed', fixed: 'live', live: 'off' })[p.status] } : p)));
|
||||
|
||||
const toggleAB = () => {
|
||||
if (ab === 'B') { aRef.current = { seed }; setAB('A'); } else { if (aRef.current) setSeed(aRef.current.seed); setAB('B'); }
|
||||
};
|
||||
|
||||
// keyboard accelerators
|
||||
React.useEffect(() => {
|
||||
const onKey = (e) => {
|
||||
if (e.target.tagName === 'INPUT') return;
|
||||
const map = { '1': 'shape', '2': 'feel', '3': 'route', '4': 'health', '5': 'help' };
|
||||
if (map[e.key]) { setActive((a) => (a === map[e.key] ? null : map[e.key])); setDepth('peek'); }
|
||||
else if (e.key === '\\') setDepth((d) => (d === 'full' ? 'peek' : 'full'));
|
||||
else if (focus === 'composite' && e.key === '[') { e.preventDefault(); setSplit((s) => Math.max(0, s - 0.04)); }
|
||||
else if (focus === 'composite' && e.key === ']') { e.preventDefault(); setSplit((s) => Math.min(1, s + 0.04)); }
|
||||
else if (focus === 'composite' && (e.key === '=' || e.key === '0')) { e.preventDefault(); setSplit(0.5); }
|
||||
else if (e.key === ' ' || e.key === 'ArrowUp') { e.preventDefault(); commit(); }
|
||||
else if (e.key === 'ArrowDown') { e.preventDefault(); perturb(); }
|
||||
else if (e.key.toLowerCase() === 'z') undo();
|
||||
};
|
||||
window.addEventListener('keydown', onKey);
|
||||
return () => window.removeEventListener('keydown', onKey);
|
||||
});
|
||||
|
||||
const ctx = {
|
||||
modes: MODES2, modeId, setModeId, mode,
|
||||
axes, setAxis: (k, v) => setAxes((s) => ({ ...s, [k]: v })),
|
||||
preset, setPreset, offsetActive: preset !== 'Sculpt',
|
||||
datasetCount: examples, loss, busy, addingExample,
|
||||
onAddExample: addExample, onTrain: train, onClear: () => { setExamples(0); setLoss([]); },
|
||||
snapshots, onJump: (id) => { const s = snapshots.find((x) => x.id === id); if (s) { setSeed(s.seed); setNoiseCap(s.noise); setRev((r) => r + 1); } },
|
||||
params, cycleStatus, outputBackend: 'audio', setOutputBackend: () => {},
|
||||
health, gradient: gradient.norms, gradientStatus: gradient.status, weightsRevision: rev,
|
||||
spread, setSpread, tame, setTame, noiseCap, setNoiseCap,
|
||||
};
|
||||
|
||||
const healthColor = health > 0.66 ? 'rgba(107,194,107,' : health > 0.33 ? 'rgba(245,196,94,' : 'rgba(255,68,102,';
|
||||
const Input = mode.input;
|
||||
|
||||
return (
|
||||
<div style={{ position: 'absolute', inset: 0, background: 'var(--bg)', overflow: 'hidden', fontFamily: 'var(--font-mono)' }}>
|
||||
<style>{`@keyframes mfDrawerIn{from{transform:translateX(16px)}to{transform:translateX(0)}}`}</style>
|
||||
|
||||
{/* ambient health glow at the screen edge */}
|
||||
<div style={{ position: 'absolute', inset: 0, pointerEvents: 'none', zIndex: 25,
|
||||
boxShadow: `inset 0 0 120px ${healthColor}${0.05 + (1 - health) * 0.12})`, transition: 'box-shadow var(--dur-slow) var(--ease-console)' }} />
|
||||
|
||||
{/* stage = manifold area (left of dock) */}
|
||||
<div style={{ position: 'absolute', top: 0, left: 0, right: 48, bottom: 0 }}>
|
||||
{/* readout strip (input-first only — promoted to the stage in output-first) */}
|
||||
{focus === 'in' && (stripPinned || mode.cls !== 'Synth') && (
|
||||
<div style={{ position: 'absolute', top: 0, left: 0, right: 0, zIndex: 30 }}>
|
||||
<window.ReadoutStrip params={params} values={values} onChange={setParam} pinned={stripPinned} onTogglePin={() => setStripPinned((p) => !p)} />
|
||||
</div>
|
||||
)}
|
||||
|
||||
{/* the stage: manifold (input-first) or output field (output-first) */}
|
||||
<div style={{ position: 'absolute', top: focus === 'in' && stripPinned ? 76 : 0, left: 0, right: 0, bottom: 0 }}>
|
||||
{focus === 'composite' ? (
|
||||
<window.CompositeStage split={split} onSplit={setSplit} mode={mode}
|
||||
pos={pos} onMove={(x, y) => setPos([x, y])} noiseCap={noiseCap} pins={pins} follow={follow}
|
||||
onLongPress={(p) => setPins((ps) => [...ps, { x: p[0], y: p[1], color: 'rgba(255,106,0,0.16)' }])}
|
||||
params={params} values={values} onChange={setParam} />
|
||||
) : focus === 'split' ? (
|
||||
<window.SplitStage pos={pos} onMove={(x, y) => setPos([x, y])} noiseCap={noiseCap}
|
||||
pins={pins} follow={follow}
|
||||
onLongPress={(p) => setPins((ps) => [...ps, { x: p[0], y: p[1], color: 'rgba(255,106,0,0.16)' }])}
|
||||
params={params} values={values} onChange={setParam} />
|
||||
) : focus === 'out' ? (
|
||||
<React.Fragment>
|
||||
<window.OutputStage params={params} values={values} onChange={setParam} />
|
||||
<window.InputMini mode={mode} pos={pos} onMove={(x, y) => setPos([x, y])} noiseCap={noiseCap} corner="bottom-left" />
|
||||
</React.Fragment>
|
||||
) : (
|
||||
<window.Manifold pos={pos} onMove={(x, y) => setPos([x, y])} noiseCap={noiseCap} pins={pins}
|
||||
frozen={false} follow={follow}
|
||||
onLongPress={(p) => setPins((ps) => [...ps, { x: p[0], y: p[1], color: 'rgba(255,106,0,0.16)' }])} />
|
||||
)}
|
||||
|
||||
{/* corner overlays */}
|
||||
<div style={{ position: 'absolute', top: 12, left: 14, zIndex: 20, pointerEvents: 'none' }}>
|
||||
<div style={{ display: 'flex', alignItems: 'baseline', gap: 8 }}>
|
||||
<strong style={{ color: 'var(--accent)', fontSize: 'var(--fs-md)' }}>MEMLNaut</strong>
|
||||
<span style={{ color: 'var(--fg-mute)', fontSize: 'var(--fs-sm)' }}>{mode.label}</span>
|
||||
{mode.badge && <span style={{ fontSize: 9, color: 'var(--warn)', border: '1px solid var(--warn)', borderRadius: 'var(--r-pill)', padding: '1px 6px' }}>{mode.badge}</span>}
|
||||
</div>
|
||||
<div style={{ color: 'var(--fg-dim)', fontSize: 'var(--fs-xs)', marginTop: 2 }}>{mode.cls.toLowerCase()} · {mode.input} · {params.length} targets</div>
|
||||
</div>
|
||||
|
||||
<div style={{ position: 'absolute', bottom: 14, left: focus === 'out' ? 178 : 14, zIndex: 20, display: 'flex', gap: 8, fontFamily: 'var(--font-mono)' }}>
|
||||
<button type="button" onClick={() => setFollow((f) => !f)} title="Follow mode (hands-free drift)"
|
||||
style={{ fontSize: 'var(--fs-xs)', padding: '4px 10px', borderRadius: 'var(--r-pill)', cursor: 'pointer',
|
||||
border: `1px solid ${follow ? 'var(--good)' : 'var(--glass-line)'}`, background: 'var(--glass)',
|
||||
color: follow ? 'var(--good)' : 'var(--fg-mute)' }}>{follow ? '◉ follow' : '○ follow'}</button>
|
||||
<span style={{ fontSize: 'var(--fs-xs)', color: 'var(--fg-dim)', alignSelf: 'center' }}>
|
||||
input ({pos[0].toFixed(2)}, {pos[1].toFixed(2)}) · noise {noiseCap.toFixed(3)} {holdingA && <span style={{ color: 'var(--accent-2)' }}>· preview A</span>}
|
||||
</span>
|
||||
</div>
|
||||
|
||||
<window.VerdictCluster onPerturb={perturb} onUndo={undo} onCommit={commit} onReroll={reroll}
|
||||
canUndo={snapshots.length > 0} ab={ab} onToggleAB={toggleAB} onHoldA={setHoldingA} firstSession={firstSession} />
|
||||
|
||||
<window.AltitudeNav current="console" focus={focus} style={{ top: 'auto', bottom: 14, right: 14 }} />
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<window.Dock ctx={ctx} active={active} setActive={setActive} depth={depth} setDepth={setDepth} />
|
||||
</div>
|
||||
);
|
||||
}
|
||||
window.ConsoleApp = ConsoleApp;
|
||||
|
|
@ -0,0 +1,59 @@
|
|||
/* Console 2.0 — CurvePad: a square response-curve plot. Click-and-vertical-drag
|
||||
anywhere reshapes the curve (drag up = bulge / ease-out, down = ease-in);
|
||||
`curve` is 0..1 where ~0.43 reads as linear. Mirrors the engine's applyCurve. */
|
||||
function CurvePad({ curve = 0.5, onChange, size = 116 }) {
|
||||
const ref = React.useRef(null);
|
||||
const drag = React.useRef({ active: false, startY: 0, startC: 0 });
|
||||
|
||||
React.useEffect(() => {
|
||||
const cv = ref.current; if (!cv) return;
|
||||
const dpr = window.devicePixelRatio || 1;
|
||||
cv.width = size * dpr; cv.height = size * dpr;
|
||||
const ctx = cv.getContext('2d');
|
||||
ctx.setTransform(dpr, 0, 0, dpr, 0, 0);
|
||||
ctx.clearRect(0, 0, size, size);
|
||||
// frame + grid
|
||||
ctx.strokeStyle = 'rgba(255,255,255,0.06)'; ctx.lineWidth = 1;
|
||||
ctx.strokeRect(0.5, 0.5, size - 1, size - 1);
|
||||
for (const t of [0.25, 0.5, 0.75]) {
|
||||
ctx.beginPath(); ctx.moveTo(t * size, 0); ctx.lineTo(t * size, size); ctx.stroke();
|
||||
ctx.beginPath(); ctx.moveTo(0, t * size); ctx.lineTo(size, t * size); ctx.stroke();
|
||||
}
|
||||
// diagonal reference (linear)
|
||||
ctx.strokeStyle = 'rgba(255,255,255,0.10)'; ctx.setLineDash([3, 3]);
|
||||
ctx.beginPath(); ctx.moveTo(0, size); ctx.lineTo(size, 0); ctx.stroke(); ctx.setLineDash([]);
|
||||
// the curve
|
||||
const css = getComputedStyle(cv);
|
||||
const accent = css.getPropertyValue('--accent').trim() || '#ff6a00';
|
||||
const e = 0.25 + curve * 1.75;
|
||||
ctx.strokeStyle = accent; ctx.lineWidth = 2; ctx.beginPath();
|
||||
const pad = 3;
|
||||
for (let p = 0; p <= 80; p++) {
|
||||
const xv = p / 80, yv = Math.pow(xv, e);
|
||||
const px = pad + xv * (size - 2 * pad), py = (size - pad) - yv * (size - 2 * pad);
|
||||
p === 0 ? ctx.moveTo(px, py) : ctx.lineTo(px, py);
|
||||
}
|
||||
ctx.stroke();
|
||||
}, [curve, size]);
|
||||
|
||||
const down = (e) => { e.currentTarget.setPointerCapture?.(e.pointerId); drag.current = { active: true, startY: e.clientY, startC: curve }; };
|
||||
const move = (e) => {
|
||||
const d = drag.current; if (!d.active) return;
|
||||
const dc = (e.clientY - d.startY) / size; // drag up (negative dy) → lower c → bulge up
|
||||
onChange && onChange(Math.max(0, Math.min(1, d.startC + dc)));
|
||||
};
|
||||
const up = (e) => { drag.current.active = false; e.currentTarget.releasePointerCapture?.(e.pointerId); };
|
||||
|
||||
return (
|
||||
<div style={{ display: 'flex', flexDirection: 'column', gap: 4 }}>
|
||||
<div style={{ display: 'flex', justifyContent: 'space-between' }}>
|
||||
<span style={{ fontSize: 10, color: 'var(--fg-mute)', textTransform: 'uppercase', letterSpacing: '0.08em' }}>curve</span>
|
||||
<span style={{ fontSize: 10, color: 'var(--fg-dim)', fontVariantNumeric: 'tabular-nums' }}>{(0.25 + curve * 1.75).toFixed(2)}</span>
|
||||
</div>
|
||||
<canvas ref={ref} onPointerDown={down} onPointerMove={move} onPointerUp={up} onPointerCancel={up}
|
||||
title="Drag vertically to reshape"
|
||||
style={{ width: size, height: size, display: 'block', cursor: 'ns-resize', background: 'var(--bg-1)', border: '1px solid var(--line)', borderRadius: 'var(--r-1)', touchAction: 'none' }} />
|
||||
</div>
|
||||
);
|
||||
}
|
||||
window.CurvePad = CurvePad;
|
||||
118
docs/redesign/manifold-export/ui_kits/memlnaut/console/Dock.jsx
Normal file
118
docs/redesign/manifold-export/ui_kits/memlnaut/console/Dock.jsx
Normal file
|
|
@ -0,0 +1,118 @@
|
|||
/* Console 2.0 — The Console: right-edge 48px dock rail + one mutually-exclusive
|
||||
drawer with three depth states (Peek 320 / Expand 520 / Full modal). */
|
||||
function ModeSwitcher({ modes, modeId, setModeId }) {
|
||||
const [open, setOpen] = React.useState(false);
|
||||
const active = modes.find((m) => m.id === modeId) || modes[0];
|
||||
const classes = ['Synth', 'Sequencer', 'Controller', 'Visual'];
|
||||
return (
|
||||
<div style={{ position: 'relative' }}>
|
||||
<button type="button" onClick={() => setOpen((o) => !o)} title={`Mode: ${active.label}`}
|
||||
style={{ width: 40, height: 40, borderRadius: 'var(--r-2)', border: '1px solid var(--accent)',
|
||||
background: 'rgba(255,106,0,0.12)', color: 'var(--accent)', cursor: 'pointer', fontSize: 18,
|
||||
display: 'flex', alignItems: 'center', justifyContent: 'center' }}>{active.glyph || '⊞'}</button>
|
||||
{open && (
|
||||
<div style={{ position: 'absolute', top: 0, right: 'calc(100% + 8px)', width: 240, background: 'var(--glass)',
|
||||
backdropFilter: 'blur(14px)', WebkitBackdropFilter: 'blur(14px)', border: '1px solid var(--glass-line)',
|
||||
borderRadius: 'var(--r-2)', boxShadow: 'var(--shadow-2)', padding: 6, zIndex: 80 }}>
|
||||
{classes.map((cls) => {
|
||||
const items = modes.filter((m) => m.cls === cls);
|
||||
if (!items.length) return null;
|
||||
return (
|
||||
<div key={cls}>
|
||||
<div style={{ fontSize: 10, color: 'var(--fg-dim)', textTransform: 'uppercase', letterSpacing: '0.1em', padding: '6px 8px 2px' }}>{cls}</div>
|
||||
{items.map((m) => (
|
||||
<button key={m.id} type="button" disabled={m.placeholder}
|
||||
onClick={() => { if (m.placeholder) return; setModeId(m.id); setOpen(false); }}
|
||||
style={{ display: 'flex', alignItems: 'center', gap: 8, width: '100%', textAlign: 'left',
|
||||
background: m.id === modeId ? 'var(--bg-3)' : 'transparent', border: 0, borderRadius: 'var(--r-1)',
|
||||
padding: '6px 8px', cursor: m.placeholder ? 'not-allowed' : 'pointer', opacity: m.placeholder ? 0.4 : 1,
|
||||
fontFamily: 'var(--font-mono)', fontSize: 'var(--fs-sm)', color: m.id === modeId ? 'var(--accent)' : 'var(--fg)' }}>
|
||||
<span style={{ width: 16, textAlign: 'center' }}>{m.glyph || '·'}</span>{m.label}
|
||||
{m.badge && <span style={{ marginLeft: 'auto', fontSize: 9, color: 'var(--warn)' }}>{m.badge}</span>}
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
);
|
||||
})}
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
function Dock({ ctx, active, setActive, depth, setDepth }) {
|
||||
const sections = window.DRAWERS;
|
||||
const order = ['shape', 'feel', 'route', 'health', 'help'];
|
||||
|
||||
const iconBtn = (key) => {
|
||||
const s = sections[key];
|
||||
const on = active === key;
|
||||
return (
|
||||
<button key={key} type="button" title={s.label} onClick={() => { setActive(on ? null : key); setDepth('peek'); }}
|
||||
style={{ position: 'relative', width: 40, height: 40, borderRadius: 'var(--r-2)', cursor: 'pointer',
|
||||
border: `1px solid ${on ? 'var(--accent)' : 'transparent'}`,
|
||||
background: on ? 'rgba(255,106,0,0.14)' : 'transparent', color: on ? 'var(--accent)' : 'var(--fg-mute)',
|
||||
fontSize: 18, display: 'flex', alignItems: 'center', justifyContent: 'center',
|
||||
transition: 'background var(--dur-fast), color var(--dur-fast)' }}
|
||||
onMouseEnter={(e) => { if (!on) e.currentTarget.style.color = 'var(--fg)'; }}
|
||||
onMouseLeave={(e) => { if (!on) e.currentTarget.style.color = 'var(--fg-mute)'; }}>
|
||||
{s.icon}
|
||||
{key === 'feel' && active !== 'feel' && (
|
||||
<span style={{ position: 'absolute', bottom: 4, left: 6, right: 6, height: 2, borderRadius: 2,
|
||||
background: `linear-gradient(90deg, var(--accent) ${ctx.axes.boldness * 100}%, var(--line) 0)` }} />
|
||||
)}
|
||||
</button>
|
||||
);
|
||||
};
|
||||
|
||||
const width = depth === 'expand' ? 520 : 320;
|
||||
const full = depth === 'full';
|
||||
const section = active ? sections[active] : null;
|
||||
|
||||
return (
|
||||
<React.Fragment>
|
||||
{/* drawer */}
|
||||
{section && (
|
||||
<aside style={full ? {
|
||||
position: 'fixed', inset: 0, zIndex: 90, background: 'var(--glass)', backdropFilter: 'blur(16px)', WebkitBackdropFilter: 'blur(16px)',
|
||||
padding: 'var(--sp-6)', display: 'flex', flexDirection: 'column', gap: 'var(--sp-3)',
|
||||
} : {
|
||||
position: 'absolute', top: 0, right: 48, bottom: 0, width, zIndex: 35,
|
||||
background: 'var(--glass)', backdropFilter: 'blur(16px)', WebkitBackdropFilter: 'blur(16px)',
|
||||
borderLeft: '1px solid var(--glass-line)', boxShadow: '-8px 0 24px rgba(0,0,0,0.4)',
|
||||
padding: 'var(--sp-4)', display: 'flex', flexDirection: 'column', gap: 'var(--sp-2)', overflow: 'auto',
|
||||
animation: 'mfDrawerIn var(--dur-med) var(--ease-console)',
|
||||
}}>
|
||||
<header style={{ display: 'flex', alignItems: 'center', gap: 'var(--sp-2)', borderBottom: '1px solid var(--glass-line)', paddingBottom: 'var(--sp-2)' }}>
|
||||
<span style={{ fontSize: 'var(--fs-md)', color: 'var(--accent)' }}>{section.icon}</span>
|
||||
<h3 style={{ margin: 0, fontSize: 'var(--fs-md)', color: 'var(--fg)' }}>{section.label}</h3>
|
||||
<span style={{ fontSize: 10, color: 'var(--fg-dim)', textTransform: 'uppercase', letterSpacing: '0.1em', marginLeft: 4 }}>{depth}</span>
|
||||
<div style={{ marginLeft: 'auto', display: 'flex', gap: 4 }}>
|
||||
{!full && (
|
||||
<button type="button" title={depth === 'peek' ? 'More' : 'Peek'} onClick={() => setDepth(depth === 'peek' ? 'expand' : 'peek')}
|
||||
style={dockMini}>⤢</button>
|
||||
)}
|
||||
<button type="button" title="Open full" onClick={() => setDepth(full ? 'peek' : 'full')} style={dockMini}>{full ? '⤡' : '⤢⤢'}</button>
|
||||
<button type="button" title="Close" onClick={() => setActive(null)} style={dockMini}>✕</button>
|
||||
</div>
|
||||
</header>
|
||||
<div style={{ display: 'flex', flexDirection: 'column', gap: 'var(--sp-2)', flex: 1, ...(full ? { maxWidth: 720 } : {}) }}>
|
||||
{section.render(ctx, depth)}
|
||||
</div>
|
||||
</aside>
|
||||
)}
|
||||
|
||||
{/* rail */}
|
||||
<nav style={{ position: 'absolute', top: 0, right: 0, bottom: 0, width: 48, zIndex: 36,
|
||||
background: 'var(--glass)', backdropFilter: 'blur(12px)', WebkitBackdropFilter: 'blur(12px)',
|
||||
borderLeft: '1px solid var(--glass-line)', display: 'flex', flexDirection: 'column', alignItems: 'center',
|
||||
gap: 'var(--sp-2)', padding: '8px 0' }}>
|
||||
<ModeSwitcher modes={ctx.modes} modeId={ctx.modeId} setModeId={ctx.setModeId} />
|
||||
<div style={{ width: 24, height: 1, background: 'var(--glass-line)', margin: '2px 0' }} />
|
||||
{order.map(iconBtn)}
|
||||
</nav>
|
||||
</React.Fragment>
|
||||
);
|
||||
}
|
||||
const dockMini = { width: 26, height: 24, borderRadius: 'var(--r-1)', border: '1px solid var(--line)', background: 'var(--bg-2)', color: 'var(--fg-mute)', cursor: 'pointer', fontSize: 11, lineHeight: 1 };
|
||||
window.Dock = Dock;
|
||||
|
|
@ -0,0 +1,223 @@
|
|||
/* Console 2.0 — Drawer contents. Each renderer takes (ctx, depth) where depth
|
||||
is 'peek' | 'expand' | 'full'; what shows is gated by depth (the schema-tier
|
||||
disclosure mechanism, simulated here).
|
||||
NOTE: these are the PLACEHOLDER drawers. The real dock contents are specified
|
||||
in docs/redesign/dock-spec.md (six drawers: Learning-Behaviour / Inputs /
|
||||
Outputs-Routing / Powerful Synth Engine / Particle-Visual / Help). */
|
||||
const DRW_DS = window.ManifoldDesignSystem_490915;
|
||||
|
||||
function Chip({ children, tone }) {
|
||||
return (
|
||||
<span style={{
|
||||
fontSize: 10, fontFamily: 'var(--font-mono)', textTransform: 'uppercase', letterSpacing: '0.06em',
|
||||
color: tone || 'var(--fg-mute)', background: 'var(--bg-2)', border: '1px solid var(--line)',
|
||||
borderRadius: 'var(--r-pill)', padding: '2px 8px',
|
||||
}}>{children}</span>
|
||||
);
|
||||
}
|
||||
function SectionLabel({ children }) {
|
||||
return <div style={{ fontSize: 10, color: 'var(--fg-dim)', textTransform: 'uppercase', letterSpacing: '0.1em', marginTop: 'var(--sp-2)' }}>{children}</div>;
|
||||
}
|
||||
|
||||
const PRESETS = ['Sculpt', 'Wander', 'Lock-in', 'Chaos', 'Glide', 'Pin & probe'];
|
||||
|
||||
function FeelDrawer(ctx, depth) {
|
||||
const { ControlAxis, PillToggle } = DRW_DS;
|
||||
const axes = [
|
||||
{ key: 'boldness', label: 'Boldness', endpoints: ['Caution', 'Bold'], accent: 'var(--accent)' },
|
||||
{ key: 'memory', label: 'Memory', endpoints: ['Amnesia', 'Elephant'], accent: 'var(--accent-2)' },
|
||||
{ key: 'precision', label: 'Precision', endpoints: ['Raw', 'Precise'], accent: 'var(--accent)' },
|
||||
];
|
||||
return (
|
||||
<React.Fragment>
|
||||
<div style={{ display: 'flex', gap: 6, flexWrap: 'wrap' }}>
|
||||
<Chip tone="var(--accent)">3 axes</Chip>
|
||||
<Chip>preset · {ctx.preset}</Chip>
|
||||
{ctx.offsetActive && <Chip tone="var(--accent-2)">offset active</Chip>}
|
||||
</div>
|
||||
<SectionLabel>Compound axes</SectionLabel>
|
||||
{axes.map((a) => (
|
||||
<ControlAxis key={a.key} label={a.label} endpoints={a.endpoints} accent={a.accent}
|
||||
value={ctx.axes[a.key]} onChange={(v) => ctx.setAxis(a.key, v)}
|
||||
preset={a.key === 'boldness' ? (ctx.axes.boldness > 0.66 ? 'explore' : 'steady') : undefined} />
|
||||
))}
|
||||
<SectionLabel>Control presets</SectionLabel>
|
||||
<div style={{ display: 'flex', gap: 6, flexWrap: 'wrap' }}>
|
||||
{PRESETS.map((p) => (
|
||||
<button key={p} type="button" onClick={() => ctx.setPreset(p)}
|
||||
style={{
|
||||
padding: '5px 10px', fontSize: 'var(--fs-xs)', fontFamily: 'var(--font-mono)', cursor: 'pointer',
|
||||
borderRadius: 'var(--r-pill)', border: `1px solid ${ctx.preset === p ? 'var(--accent)' : 'var(--line)'}`,
|
||||
background: ctx.preset === p ? 'rgba(255,106,0,0.15)' : 'var(--bg-2)',
|
||||
color: ctx.preset === p ? 'var(--accent)' : 'var(--fg-mute)',
|
||||
}}>{p}</button>
|
||||
))}
|
||||
</div>
|
||||
{depth !== 'peek' && (
|
||||
<React.Fragment>
|
||||
<SectionLabel>Trim-pot offsets</SectionLabel>
|
||||
<p style={{ fontSize: 'var(--fs-xs)', color: 'var(--fg-dim)', margin: 0, lineHeight: 1.6 }}>
|
||||
Axis tweaks layer as offsets over the active preset. Double-tap an axis to re-link. An <span style={{ color: 'var(--accent-2)' }}>offset-active</span> dot marks divergence.
|
||||
</p>
|
||||
</React.Fragment>
|
||||
)}
|
||||
</React.Fragment>
|
||||
);
|
||||
}
|
||||
|
||||
function ShapeDrawer(ctx, depth) {
|
||||
const { Button, Sparkline, Badge } = DRW_DS;
|
||||
return (
|
||||
<React.Fragment>
|
||||
<div style={{ display: 'flex', gap: 6, flexWrap: 'wrap' }}>
|
||||
<Chip tone="var(--accent)">examples {ctx.datasetCount}</Chip>
|
||||
<Chip>loss {ctx.loss.length ? ctx.loss[ctx.loss.length - 1].toExponential(1) : '—'}</Chip>
|
||||
{ctx.busy && <Chip tone="var(--accent-2)">training…</Chip>}
|
||||
</div>
|
||||
<SectionLabel>Reinforcement (default)</SectionLabel>
|
||||
<p style={{ fontSize: 'var(--fs-xs)', color: 'var(--fg-dim)', margin: 0, lineHeight: 1.6 }}>
|
||||
Use the + / − verdicts on the manifold. Every verdict auto-snapshots first — undo is consequence-free.
|
||||
</p>
|
||||
<SectionLabel>Examples (IML)</SectionLabel>
|
||||
<div style={{ display: 'flex', gap: 6, flexWrap: 'wrap' }}>
|
||||
<Button size="sm" variant={ctx.addingExample ? 'primary' : 'secondary'} onClick={ctx.onAddExample}>
|
||||
{ctx.addingExample ? '② store target' : '① add example'}
|
||||
</Button>
|
||||
<Button size="sm" onClick={ctx.onTrain} disabled={ctx.busy}>Train</Button>
|
||||
<Button size="sm" variant="ghost" onClick={ctx.onClear}>clear</Button>
|
||||
</div>
|
||||
<SectionLabel>Loss history</SectionLabel>
|
||||
<Sparkline data={ctx.loss.length ? ctx.loss : [0.5]} log width={depth === 'peek' ? 284 : 480} height={56} />
|
||||
{depth !== 'peek' && (
|
||||
<React.Fragment>
|
||||
<SectionLabel>Snapshot DAG · {ctx.snapshots.length}</SectionLabel>
|
||||
<ul style={{ listStyle: 'none', margin: 0, padding: 0, display: 'flex', flexDirection: 'column', gap: 4, maxHeight: 160, overflow: 'auto' }}>
|
||||
{ctx.snapshots.slice().reverse().map((s, idx) => (
|
||||
<li key={s.id}>
|
||||
<button type="button" onClick={() => ctx.onJump(s.id)} style={{
|
||||
width: '100%', textAlign: 'left', display: 'flex', justifyContent: 'space-between',
|
||||
background: idx === 0 ? 'var(--bg-3)' : 'var(--bg-2)', border: '1px solid var(--line)',
|
||||
borderRadius: 'var(--r-1)', padding: '4px 8px', cursor: 'pointer', fontFamily: 'var(--font-mono)', fontSize: 'var(--fs-xs)',
|
||||
}}>
|
||||
<span style={{ color: idx === 0 ? 'var(--accent)' : 'var(--fg)' }}>{s.tag}</span>
|
||||
<span style={{ color: 'var(--fg-dim)' }}>noise {s.noise.toFixed(2)}</span>
|
||||
</button>
|
||||
</li>
|
||||
))}
|
||||
</ul>
|
||||
</React.Fragment>
|
||||
)}
|
||||
</React.Fragment>
|
||||
);
|
||||
}
|
||||
|
||||
const STATUS_CYCLE = { off: 'fixed', fixed: 'live', live: 'off' };
|
||||
const STATUS_COLOR = { off: 'var(--fg-dim)', fixed: 'var(--accent-2)', live: 'var(--accent)' };
|
||||
|
||||
function RouteDrawer(ctx, depth) {
|
||||
const { PillToggle, Badge } = DRW_DS;
|
||||
const counts = ctx.params.reduce((a, p) => { a[p.status] = (a[p.status] || 0) + 1; return a; }, {});
|
||||
const rows = depth === 'peek' ? ctx.params.slice(0, 6) : ctx.params;
|
||||
return (
|
||||
<React.Fragment>
|
||||
<div style={{ display: 'flex', gap: 6, flexWrap: 'wrap' }}>
|
||||
<Chip>sources {ctx.mode.inN}/48</Chip>
|
||||
<Chip>targets {ctx.params.length}/{ctx.params.length}</Chip>
|
||||
<Chip tone="var(--accent)">live {counts.live || 0}</Chip>
|
||||
<Chip tone="var(--accent-2)">fixed {counts.fixed || 0}</Chip>
|
||||
</div>
|
||||
<SectionLabel>Input source</SectionLabel>
|
||||
<Badge tone="info">{ctx.mode.input === 'joystick' ? 'joystick' : ctx.mode.input === 'audio_in' ? 'mic (1-input)' : 'xy pad'}</Badge>
|
||||
<SectionLabel>Output backend</SectionLabel>
|
||||
<PillToggle value={ctx.outputBackend} onChange={ctx.setOutputBackend}
|
||||
options={[{ value: 'audio', label: 'Audio' }, { value: 'midi', label: 'MIDI' }, { value: 'osc', label: 'OSC' }, { value: 'cv', label: 'CV' }]} />
|
||||
<SectionLabel>Control points · off / fixed / live</SectionLabel>
|
||||
<div style={{ display: 'flex', flexDirection: 'column', gap: 3, maxHeight: depth === 'peek' ? 150 : 320, overflow: 'auto' }}>
|
||||
{rows.map((p, i) => (
|
||||
<div key={i} style={{ display: 'flex', alignItems: 'center', gap: 'var(--sp-2)', background: 'var(--bg-2)', border: '1px solid var(--line)', borderRadius: 'var(--r-1)', padding: '4px 8px' }}>
|
||||
<span style={{ flex: 1, fontSize: 'var(--fs-xs)', color: 'var(--fg)', overflow: 'hidden', textOverflow: 'ellipsis', whiteSpace: 'nowrap' }}>{p.name}</span>
|
||||
<span style={{ fontSize: 10, color: 'var(--fg-dim)' }}>{p.group}</span>
|
||||
<button type="button" onClick={() => ctx.cycleStatus(ctx.params.indexOf(p))}
|
||||
style={{ width: 52, fontSize: 10, fontFamily: 'var(--font-mono)', textTransform: 'uppercase', letterSpacing: '0.06em',
|
||||
border: `1px solid ${STATUS_COLOR[p.status]}`, color: STATUS_COLOR[p.status], background: 'transparent',
|
||||
borderRadius: 'var(--r-1)', padding: '2px 4px', cursor: 'pointer' }}>{p.status}</button>
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
{depth === 'full' && (
|
||||
<p style={{ fontSize: 'var(--fs-xs)', color: 'var(--fg-dim)', margin: '8px 0 0', lineHeight: 1.6 }}>
|
||||
Full tier hosts the Emitters × Targets matrix + per-cell editor + staged-unlock opt-in (built for the modular mega-mode).
|
||||
</p>
|
||||
)}
|
||||
</React.Fragment>
|
||||
);
|
||||
}
|
||||
|
||||
function GradientBars({ norms, status }) {
|
||||
const COLOR = { vanishing: 'var(--info)', exploding: 'var(--danger)', converged: 'var(--fg-mute)', healthy: 'var(--good)' };
|
||||
const max = Math.max(...norms, 0.0001);
|
||||
return (
|
||||
<div style={{ display: 'flex', alignItems: 'flex-end', gap: 3, height: 56, background: 'var(--bg-1)', border: '1px solid var(--line)', borderRadius: 'var(--r-2)', padding: 6 }}>
|
||||
{norms.map((n, i) => (
|
||||
<div key={i} style={{ flex: 1, display: 'flex', flexDirection: 'column', alignItems: 'center', gap: 2, height: '100%', justifyContent: 'flex-end' }}>
|
||||
<div style={{ width: '100%', height: `${(n / max) * 100}%`, background: COLOR[status[i]] || 'var(--good)', borderRadius: 1 }} />
|
||||
<span style={{ fontSize: 8, color: 'var(--fg-dim)' }}>L{i}</span>
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
function HealthDrawer(ctx, depth) {
|
||||
const { Sparkline, Switch, Slider, Badge } = DRW_DS;
|
||||
const healthTone = ctx.health > 0.66 ? 'good' : ctx.health > 0.33 ? 'warn' : 'bad';
|
||||
return (
|
||||
<React.Fragment>
|
||||
<div style={{ display: 'flex', gap: 6, flexWrap: 'wrap', alignItems: 'center' }}>
|
||||
<Badge tone={healthTone} dot>{healthTone === 'good' ? 'healthy' : healthTone === 'warn' ? 'drifting' : 'unstable'}</Badge>
|
||||
<Chip>rev {ctx.weightsRevision}</Chip>
|
||||
</div>
|
||||
<SectionLabel>Loss history</SectionLabel>
|
||||
<Sparkline data={ctx.loss.length ? ctx.loss : [0.5]} log width={depth === 'peek' ? 284 : 480} height={56} />
|
||||
<SectionLabel>Gradient flow · per layer</SectionLabel>
|
||||
<GradientBars norms={ctx.gradient} status={ctx.gradientStatus} />
|
||||
{depth !== 'peek' && (
|
||||
<React.Fragment>
|
||||
<SectionLabel>Lab controls</SectionLabel>
|
||||
<Switch checked={ctx.spread} onChange={ctx.setSpread} label="spread (Xavier regime)" />
|
||||
<Slider label="tame · output limiter" value={ctx.tame} min={0} max={1} step={0.01} onChange={ctx.setTame} />
|
||||
<Slider label="noise cap" value={ctx.noiseCap} min={0} max={0.5} step={0.01} onChange={ctx.setNoiseCap} />
|
||||
</React.Fragment>
|
||||
)}
|
||||
</React.Fragment>
|
||||
);
|
||||
}
|
||||
|
||||
const KEYS = [['1–5', 'open drawers'], ['\\', 'full depth'], ['space / ↑', 'commit +'], ['↓', 'perturb −'], ['z', 'undo'], ['double-click dot', 'follow mode']];
|
||||
function HelpDrawer(ctx, depth) {
|
||||
return (
|
||||
<React.Fragment>
|
||||
<SectionLabel>Keyboard</SectionLabel>
|
||||
<div style={{ display: 'flex', flexDirection: 'column', gap: 4 }}>
|
||||
{KEYS.map(([k, v]) => (
|
||||
<div key={k} style={{ display: 'flex', justifyContent: 'space-between', fontSize: 'var(--fs-xs)' }}>
|
||||
<kbd style={{ background: 'var(--bg-2)', border: '1px solid var(--line)', borderRadius: 'var(--r-1)', padding: '1px 6px', color: 'var(--accent)' }}>{k}</kbd>
|
||||
<span style={{ color: 'var(--fg-mute)' }}>{v}</span>
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
<SectionLabel>The loop</SectionLabel>
|
||||
<p style={{ fontSize: 'var(--fs-xs)', color: 'var(--fg-dim)', margin: 0, lineHeight: 1.7 }}>
|
||||
Drag the manifold to explore. Hear something good → + to settle. Wrong → − to perturb. Went too far → ↺ undo. The screen reveals exactly as much machinery as you reach for.
|
||||
</p>
|
||||
</React.Fragment>
|
||||
);
|
||||
}
|
||||
|
||||
window.DRAWERS = {
|
||||
shape: { icon: '◇', label: 'Shape', render: ShapeDrawer },
|
||||
feel: { icon: '◎', label: 'Feel', render: FeelDrawer },
|
||||
route: { icon: '⇄', label: 'Route', render: RouteDrawer },
|
||||
health: { icon: '♥', label: 'Health', render: HealthDrawer },
|
||||
help: { icon: '?', label: 'Help', render: HelpDrawer },
|
||||
};
|
||||
|
|
@ -0,0 +1,25 @@
|
|||
/* Console 2.0 — InputMini: the input demoted to a small secondary control for
|
||||
the output-first views. A compact joy-map / XY pad in a glass card; still
|
||||
drives inference, but no longer the hero. */
|
||||
function InputMini({ mode, pos, onMove, noiseCap, size = 132, corner = 'bottom-left' }) {
|
||||
const DS = window.ManifoldDesignSystem_490915;
|
||||
const Pad = mode.input === 'joystick' ? DS.VirtualJoystick : DS.XYPad;
|
||||
const place = {
|
||||
'bottom-left': { bottom: 14, left: 14 },
|
||||
'bottom-right': { bottom: 14, right: 14 },
|
||||
'top-left': { top: 14, left: 14 },
|
||||
}[corner];
|
||||
return (
|
||||
<div style={{ position: 'absolute', zIndex: 22, ...place,
|
||||
background: 'var(--glass)', backdropFilter: 'blur(12px)', WebkitBackdropFilter: 'blur(12px)',
|
||||
border: '1px solid var(--glass-line)', borderRadius: 'var(--r-2)', padding: 'var(--sp-2)',
|
||||
display: 'flex', flexDirection: 'column', gap: 6 }}>
|
||||
<div style={{ display: 'flex', justifyContent: 'space-between', alignItems: 'center', gap: 12 }}>
|
||||
<span style={{ fontSize: 10, color: 'var(--fg-dim)', textTransform: 'uppercase', letterSpacing: '0.1em' }}>input · {mode.input === 'joystick' ? 'joy' : 'xy'}</span>
|
||||
<span style={{ fontSize: 10, color: 'var(--fg-mute)', fontVariantNumeric: 'tabular-nums' }}>{pos[0].toFixed(2)},{pos[1].toFixed(2)}</span>
|
||||
</div>
|
||||
<Pad size={size} position={pos} onMove={(x, y) => onMove(x, y)} showGrid />
|
||||
</div>
|
||||
);
|
||||
}
|
||||
window.InputMini = InputMini;
|
||||
|
|
@ -0,0 +1,163 @@
|
|||
/* Console 2.0 — The Manifold: full-bleed input surface + joy-map visualisation.
|
||||
Pointer-down anywhere drives the joystick dot; a ~5s vanishing trail fades
|
||||
behind it; noise rings breathe at the current cap; pins overlay regions. */
|
||||
function Manifold({ pos, onMove, noiseCap = 0.1, pins = [], frozen = false, follow = false, onLongPress }) {
|
||||
const wrapRef = React.useRef(null);
|
||||
const canvasRef = React.useRef(null);
|
||||
const stateRef = React.useRef({ pos, noiseCap, pins, frozen, follow });
|
||||
const trailRef = React.useRef([]); // {x,y,t}
|
||||
const draggingRef = React.useRef(false);
|
||||
const driftRef = React.useRef({ vx: 0.0011, vy: 0.0008 });
|
||||
const lpTimer = React.useRef(null);
|
||||
|
||||
stateRef.current = { pos, noiseCap, pins, frozen, follow };
|
||||
|
||||
// push trail point whenever pos changes
|
||||
React.useEffect(() => {
|
||||
trailRef.current.push({ x: pos[0], y: pos[1], t: performance.now() });
|
||||
if (trailRef.current.length > 240) trailRef.current.shift();
|
||||
}, [pos[0], pos[1]]);
|
||||
|
||||
const setFromEvent = (e) => {
|
||||
const el = wrapRef.current; if (!el) return;
|
||||
const r = el.getBoundingClientRect();
|
||||
const x = Math.max(0, Math.min(1, (e.clientX - r.left) / r.width));
|
||||
const y = Math.max(0, Math.min(1, 1 - (e.clientY - r.top) / r.height));
|
||||
onMove(x, y);
|
||||
};
|
||||
|
||||
const down = (e) => {
|
||||
if (stateRef.current.frozen) return;
|
||||
e.currentTarget.setPointerCapture?.(e.pointerId);
|
||||
draggingRef.current = true;
|
||||
setFromEvent(e);
|
||||
clearTimeout(lpTimer.current);
|
||||
lpTimer.current = setTimeout(() => { onLongPress && onLongPress(stateRef.current.pos); }, 600);
|
||||
};
|
||||
const move = (e) => { if (draggingRef.current) { setFromEvent(e); clearTimeout(lpTimer.current); } };
|
||||
const up = (e) => { draggingRef.current = false; e.currentTarget.releasePointerCapture?.(e.pointerId); clearTimeout(lpTimer.current); };
|
||||
|
||||
React.useEffect(() => {
|
||||
const canvas = canvasRef.current, wrap = wrapRef.current;
|
||||
if (!canvas || !wrap) return;
|
||||
const ctx = canvas.getContext('2d');
|
||||
let raf;
|
||||
const css = getComputedStyle(document.documentElement);
|
||||
const C = (n, f) => (css.getPropertyValue(n).trim() || f);
|
||||
const accent = C('--accent', '#ff6a00'), cyan = C('--accent-2', '#00ccff'), line = C('--line-strong', '#3a3a3a');
|
||||
|
||||
const dpr = window.devicePixelRatio || 1;
|
||||
let lastW = -1, lastH = -1;
|
||||
const ensureSize = (W, H) => {
|
||||
if (W === lastW && H === lastH) return;
|
||||
canvas.width = W * dpr; canvas.height = H * dpr;
|
||||
canvas.style.width = W + 'px'; canvas.style.height = H + 'px';
|
||||
ctx.setTransform(dpr, 0, 0, dpr, 0, 0);
|
||||
lastW = W; lastH = H;
|
||||
};
|
||||
|
||||
const draw = () => {
|
||||
const W = wrap.clientWidth, H = wrap.clientHeight;
|
||||
// Self-heal against deferred / zero initial sizing (e.g. inside a flex
|
||||
// child that lays out after mount): size each frame, skip until non-zero,
|
||||
// but always keep the rAF chain alive.
|
||||
if (W === 0 || H === 0) return;
|
||||
ensureSize(W, H);
|
||||
const { pos, noiseCap, pins, frozen, follow } = stateRef.current;
|
||||
const now = performance.now();
|
||||
|
||||
// drift in follow mode
|
||||
if (follow && !draggingRef.current && !frozen) {
|
||||
let [x, y] = pos; const d = driftRef.current;
|
||||
x += d.vx; y += d.vy;
|
||||
if (x < 0.05 || x > 0.95) d.vx *= -1;
|
||||
if (y < 0.05 || y > 0.95) d.vy *= -1;
|
||||
x = Math.max(0.05, Math.min(0.95, x)); y = Math.max(0.05, Math.min(0.95, y));
|
||||
onMove(x, y);
|
||||
}
|
||||
|
||||
ctx.clearRect(0, 0, W, H);
|
||||
|
||||
// adaptive graph-paper grid (8 major, 4 minor subdivisions)
|
||||
const minor = 32, major = 8;
|
||||
ctx.lineWidth = 1;
|
||||
for (let i = 0; i <= minor; i++) {
|
||||
const t = i / minor; const isMajor = i % (minor / major) === 0;
|
||||
ctx.strokeStyle = isMajor ? 'rgba(255,255,255,0.06)' : 'rgba(255,255,255,0.022)';
|
||||
ctx.beginPath(); ctx.moveTo(t * W, 0); ctx.lineTo(t * W, H); ctx.stroke();
|
||||
ctx.beginPath(); ctx.moveTo(0, t * H); ctx.lineTo(W, t * H); ctx.stroke();
|
||||
}
|
||||
// center crosshair
|
||||
ctx.strokeStyle = 'rgba(255,255,255,0.05)';
|
||||
ctx.beginPath(); ctx.moveTo(W / 2, 0); ctx.lineTo(W / 2, H); ctx.moveTo(0, H / 2); ctx.lineTo(W, H / 2); ctx.stroke();
|
||||
|
||||
const px = pos[0] * W, py = (1 - pos[1]) * H;
|
||||
|
||||
// pins
|
||||
for (const p of pins) {
|
||||
const ppx = p.x * W, ppy = (1 - p.y) * H;
|
||||
ctx.fillStyle = p.color || 'rgba(255,106,0,0.18)';
|
||||
ctx.beginPath(); ctx.arc(ppx, ppy, 34, 0, Math.PI * 2); ctx.fill();
|
||||
ctx.strokeStyle = 'rgba(255,255,255,0.18)'; ctx.lineWidth = 1;
|
||||
ctx.beginPath(); ctx.arc(ppx, ppy, 34, 0, Math.PI * 2); ctx.stroke();
|
||||
}
|
||||
|
||||
// vanishing trail (~5s)
|
||||
const LIFE = 5000;
|
||||
const pts = trailRef.current;
|
||||
ctx.lineWidth = 2;
|
||||
for (let i = 1; i < pts.length; i++) {
|
||||
const a = pts[i - 1], b = pts[i];
|
||||
const age = now - b.t; if (age > LIFE) continue;
|
||||
const alpha = (1 - age / LIFE) * 0.5;
|
||||
ctx.strokeStyle = `rgba(0,204,255,${alpha})`;
|
||||
ctx.beginPath(); ctx.moveTo(a.x * W, (1 - a.y) * H); ctx.lineTo(b.x * W, (1 - b.y) * H); ctx.stroke();
|
||||
}
|
||||
|
||||
// noise rings — breathe at cap
|
||||
if (noiseCap > 0.001) {
|
||||
const breathe = 1 + Math.sin(now / 600) * 0.06;
|
||||
const rCap = noiseCap * Math.min(W, H) * 0.5 * breathe;
|
||||
const rCur = rCap * 0.55;
|
||||
ctx.setLineDash([4, 5]);
|
||||
ctx.strokeStyle = `rgba(255,106,0,0.4)`; ctx.lineWidth = 1.5;
|
||||
ctx.beginPath(); ctx.arc(px, py, rCap, 0, Math.PI * 2); ctx.stroke();
|
||||
ctx.strokeStyle = `rgba(255,106,0,0.7)`;
|
||||
ctx.beginPath(); ctx.arc(px, py, rCur, 0, Math.PI * 2); ctx.stroke();
|
||||
ctx.setLineDash([]);
|
||||
}
|
||||
|
||||
// the joystick dot
|
||||
ctx.shadowColor = frozen ? cyan : accent; ctx.shadowBlur = 18;
|
||||
ctx.fillStyle = frozen ? cyan : accent;
|
||||
ctx.beginPath(); ctx.arc(px, py, 9, 0, Math.PI * 2); ctx.fill();
|
||||
ctx.shadowBlur = 0;
|
||||
ctx.fillStyle = '#0d0d0d';
|
||||
ctx.beginPath(); ctx.arc(px, py, 3, 0, Math.PI * 2); ctx.fill();
|
||||
};
|
||||
// Paint immediately + on every resize (robust to deferred flex sizing and to
|
||||
// rAF being throttled in background/preview contexts); rAF drives animation.
|
||||
const loop = () => { draw(); raf = requestAnimationFrame(loop); };
|
||||
const ro = new ResizeObserver(() => draw());
|
||||
ro.observe(wrap);
|
||||
// Initial paint via setTimeout retries (rAF and ResizeObserver are both
|
||||
// throttled/withheld in some preview contexts; setTimeout is not). Keep
|
||||
// retrying until the flex child has a non-zero width and the canvas is sized.
|
||||
let timer = null, kicks = 0;
|
||||
const kick = () => { draw(); if (lastW <= 0 && kicks++ < 80) timer = setTimeout(kick, 40); };
|
||||
kick();
|
||||
raf = requestAnimationFrame(loop);
|
||||
return () => { cancelAnimationFrame(raf); ro.disconnect(); clearTimeout(timer); };
|
||||
}, []);
|
||||
|
||||
return (
|
||||
<div
|
||||
ref={wrapRef}
|
||||
onPointerDown={down} onPointerMove={move} onPointerUp={up} onPointerCancel={up}
|
||||
style={{ position: 'absolute', inset: 0, cursor: frozen ? 'not-allowed' : 'crosshair', touchAction: 'none', userSelect: 'none' }}
|
||||
>
|
||||
<canvas ref={canvasRef} style={{ display: 'block' }} />
|
||||
</div>
|
||||
);
|
||||
}
|
||||
window.Manifold = Manifold;
|
||||
|
|
@ -0,0 +1,59 @@
|
|||
/* Console 2.0 — OutputEditor: the per-output menu. Opens on hover over an
|
||||
output (column / cell). Sets state (off / fixed / live), min, max, the static
|
||||
value (only when deactivated), and the curve via the CurvePad. */
|
||||
const OE_STATUS = [
|
||||
{ v: 'off', label: 'Off', color: 'var(--fg-dim)' },
|
||||
{ v: 'fixed', label: 'Fixed', color: 'var(--accent-2)' },
|
||||
{ v: 'live', label: 'Live', color: 'var(--accent)' },
|
||||
];
|
||||
|
||||
function MiniSlider({ label, value, onChange, disabled }) {
|
||||
return (
|
||||
<label style={{ display: 'flex', flexDirection: 'column', gap: 2, opacity: disabled ? 0.4 : 1 }}>
|
||||
<div style={{ display: 'flex', justifyContent: 'space-between' }}>
|
||||
<span style={{ fontSize: 10, color: 'var(--fg-mute)', textTransform: 'uppercase', letterSpacing: '0.08em' }}>{label}</span>
|
||||
<span style={{ fontSize: 10, color: 'var(--fg-dim)', fontVariantNumeric: 'tabular-nums' }}>{value.toFixed(2)}</span>
|
||||
</div>
|
||||
<input type="range" min="0" max="1" step="0.01" value={value} disabled={disabled}
|
||||
onChange={(e) => onChange(parseFloat(e.target.value))} className="mf-slider-input" style={{ width: '100%' }} />
|
||||
</label>
|
||||
);
|
||||
}
|
||||
|
||||
function OutputEditor({ param, onChange, onHold, onLeave, place }) {
|
||||
const isLive = param.status === 'live';
|
||||
return (
|
||||
<div onPointerEnter={onHold} onPointerLeave={onLeave}
|
||||
style={{
|
||||
position: 'absolute', width: 196, zIndex: 80, ...place,
|
||||
background: 'var(--glass)', backdropFilter: 'blur(14px)', WebkitBackdropFilter: 'blur(14px)',
|
||||
border: '1px solid var(--glass-line)', borderRadius: 'var(--r-2)', boxShadow: 'var(--shadow-2)',
|
||||
padding: 'var(--sp-3)', display: 'flex', flexDirection: 'column', gap: 'var(--sp-2)', fontFamily: 'var(--font-mono)',
|
||||
}}>
|
||||
<div style={{ display: 'flex', justifyContent: 'space-between', alignItems: 'baseline' }}>
|
||||
<strong style={{ fontSize: 'var(--fs-sm)', color: 'var(--fg)' }}>{param.name}</strong>
|
||||
<span style={{ fontSize: 10, color: 'var(--fg-dim)' }}>{param.group}</span>
|
||||
</div>
|
||||
|
||||
{/* tri-state */}
|
||||
<div style={{ display: 'flex', background: 'var(--bg-2)', border: '1px solid var(--line)', borderRadius: 'var(--r-pill)', padding: 2, gap: 2 }}>
|
||||
{OE_STATUS.map((s) => {
|
||||
const on = param.status === s.v;
|
||||
return (
|
||||
<button key={s.v} type="button" onClick={() => onChange({ status: s.v })}
|
||||
style={{ flex: 1, border: 0, borderRadius: 'var(--r-pill)', padding: '4px 0', cursor: 'pointer',
|
||||
fontFamily: 'var(--font-mono)', fontSize: 10, textTransform: 'uppercase', letterSpacing: '0.06em',
|
||||
background: on ? s.color : 'transparent', color: on ? 'var(--bg)' : 'var(--fg-mute)' }}>{s.label}</button>
|
||||
);
|
||||
})}
|
||||
</div>
|
||||
|
||||
<MiniSlider label="min" value={param.min} onChange={(v) => onChange({ min: v })} />
|
||||
<MiniSlider label="max" value={param.max} onChange={(v) => onChange({ max: v })} />
|
||||
<MiniSlider label={isLive ? 'value · live' : 'value · static'} value={param.val} onChange={(v) => onChange({ val: v })} disabled={isLive} />
|
||||
<window.CurvePad curve={param.curve} onChange={(c) => onChange({ curve: c })} size={170} />
|
||||
<span style={{ fontSize: 9, color: 'var(--fg-dim)', lineHeight: 1.4 }}>drag a bar to set value · ⌥/alt-click cycles state</span>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
window.OutputEditor = OutputEditor;
|
||||
|
|
@ -0,0 +1,83 @@
|
|||
/* Console 2.0 — OutputStage: the OUTPUT as the hero surface. A full-bleed field
|
||||
of parameter columns. Per column: name on top, value below it, then the bar.
|
||||
- drag / click a bar → set the value
|
||||
- ⌥/alt-click a bar → cycle state (off → fixed → live)
|
||||
- hover a column → open the OutputEditor (state · min · max · value · curve) */
|
||||
const OUT_GROUP_COLOR = {
|
||||
formant: '--accent', pitch: '--accent-2', amp: '--good', filter: '--warn', fx: '--info', mod: '--accent-3',
|
||||
};
|
||||
const OUT_NEXT = { off: 'fixed', fixed: 'live', live: 'off' };
|
||||
|
||||
function OutputStage({ params, values, onChange, compact = false }) {
|
||||
const [open, setOpen] = React.useState(null);
|
||||
const timers = React.useRef({ open: null, close: null });
|
||||
const drag = React.useRef({ i: -1, moved: false, startY: 0, el: null, alt: false });
|
||||
|
||||
const scheduleOpen = (i) => { clearTimeout(timers.current.close); clearTimeout(timers.current.open); timers.current.open = setTimeout(() => setOpen(i), 110); };
|
||||
const scheduleClose = () => { clearTimeout(timers.current.open); clearTimeout(timers.current.close); timers.current.close = setTimeout(() => setOpen(null), 280); };
|
||||
const hold = () => clearTimeout(timers.current.close);
|
||||
|
||||
const valFromEvent = (el, clientY) => { const r = el.getBoundingClientRect(); return Math.max(0, Math.min(1, 1 - (clientY - r.top) / r.height)); };
|
||||
const down = (e, i) => { e.currentTarget.setPointerCapture?.(e.pointerId); drag.current = { i, moved: false, startY: e.clientY, el: e.currentTarget, alt: e.altKey || e.metaKey }; };
|
||||
const move = (e, i) => {
|
||||
const d = drag.current; if (d.i !== i) return;
|
||||
if (Math.abs(e.clientY - d.startY) > 3) d.moved = true;
|
||||
if (d.moved && !d.alt) onChange(i, { val: valFromEvent(d.el, e.clientY) });
|
||||
};
|
||||
const up = (e, i) => {
|
||||
const d = drag.current; if (d.i !== i) return;
|
||||
if (d.alt && !d.moved) onChange(i, { status: OUT_NEXT[params[i].status] || 'live' });
|
||||
else if (!d.moved) onChange(i, { val: valFromEvent(d.el, e.clientY) });
|
||||
drag.current = { i: -1, moved: false, startY: 0, el: null, alt: false };
|
||||
};
|
||||
|
||||
return (
|
||||
<div style={{ position: 'absolute', inset: 0, display: 'flex', alignItems: 'stretch', gap: compact ? 3 : 6, padding: compact ? 12 : '20px 28px' }}>
|
||||
{params.map((p, i) => {
|
||||
const eff = values[i] ?? 0;
|
||||
const gc = `var(${OUT_GROUP_COLOR[p.group] || '--accent'})`;
|
||||
const dim = p.status === 'off';
|
||||
const placeRight = i > params.length - 4;
|
||||
return (
|
||||
<div key={i} style={{ position: 'relative', flex: 1, minWidth: 0, display: 'flex', flexDirection: 'column', gap: compact ? 1 : 2 }}
|
||||
onPointerLeave={scheduleClose}>
|
||||
{/* hover target: name + number ONLY (not the bar) */}
|
||||
<div onPointerEnter={() => scheduleOpen(i)} style={{ cursor: 'help' }}>
|
||||
{/* name (top) */}
|
||||
<div style={{ textAlign: 'center', fontSize: compact ? 8 : 10, fontFamily: 'var(--font-mono)', color: p.status === 'live' ? 'var(--fg-mute)' : `var(${OUT_GROUP_COLOR[p.group] || '--accent'})`,
|
||||
overflow: 'hidden', whiteSpace: 'nowrap', textOverflow: 'ellipsis', letterSpacing: '0.02em' }}>{p.name}</div>
|
||||
{/* number (below name) */}
|
||||
<div style={{ textAlign: 'center', fontSize: compact ? 9 : 11, fontFamily: 'var(--font-mono)', fontVariantNumeric: 'tabular-nums',
|
||||
color: dim ? 'var(--fg-dim)' : 'var(--fg)' }}>{eff.toFixed(2)}</div>
|
||||
</div>
|
||||
{/* bar */}
|
||||
<div onPointerDown={(e) => down(e, i)} onPointerMove={(e) => move(e, i)} onPointerUp={(e) => up(e, i)} onPointerCancel={(e) => up(e, i)}
|
||||
style={{ position: 'relative', flex: 1, background: 'var(--bg-1)', border: '1px solid var(--line)', borderRadius: compact ? 2 : 'var(--r-1)', overflow: 'hidden', cursor: 'ns-resize', opacity: dim ? 0.55 : 1, touchAction: 'none' }}>
|
||||
{[0.25, 0.5, 0.75].map((t) => <div key={t} style={{ position: 'absolute', left: 0, right: 0, bottom: `${t * 100}%`, height: 1, background: 'rgba(255,255,255,0.04)' }} />)}
|
||||
<div style={{ position: 'absolute', left: 0, right: 0, bottom: 0, height: `${eff * 100}%`, background: gc, opacity: 0.22 + eff * 0.6, transition: 'height 70ms linear' }} />
|
||||
<div style={{ position: 'absolute', left: 0, right: 0, bottom: `${eff * 100}%`, height: 2, marginBottom: -1, background: gc, boxShadow: `0 0 8px ${gc}`, opacity: dim ? 0.4 : 0.9 }} />
|
||||
{/* static-value ghost when live */}
|
||||
{p.status === 'live' && (
|
||||
<div style={{ position: 'absolute', left: 1, right: 1, bottom: `${p.val * 100}%`, height: 0, borderTop: '1px dashed rgba(255,255,255,0.35)' }} />
|
||||
)}
|
||||
{/* state glyph */}
|
||||
<div style={{ position: 'absolute', top: 3, left: 0, right: 0, textAlign: 'center', fontSize: 9,
|
||||
color: p.status === 'live' ? 'transparent' : p.status === 'fixed' ? 'var(--accent-2)' : 'var(--fg-dim)' }}>
|
||||
{p.status === 'fixed' ? '⊟' : p.status === 'off' ? '∅' : ''}
|
||||
</div>
|
||||
</div>
|
||||
{open === i && !compact && (
|
||||
<window.OutputEditor param={p} onChange={(patch) => onChange(i, patch)} onHold={hold} onLeave={scheduleClose}
|
||||
place={{ top: 38, [placeRight ? 'right' : 'left']: 0 }} />
|
||||
)}
|
||||
{open === i && compact && (
|
||||
<window.OutputEditor param={p} onChange={(patch) => onChange(i, patch)} onHold={hold} onLeave={scheduleClose}
|
||||
place={{ top: 26, [placeRight ? 'right' : 'left']: 0 }} />
|
||||
)}
|
||||
</div>
|
||||
);
|
||||
})}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
window.OutputStage = OutputStage;
|
||||
|
|
@ -0,0 +1,65 @@
|
|||
/* Console 2.0 — Readout strip: the output heatmap as a control surface (the
|
||||
thin top strip in Console). Same model as OutputStage:
|
||||
- drag / click a cell → set value
|
||||
- ⌥/alt-click a cell → cycle state (off → fixed → live)
|
||||
- hover a cell → open the OutputEditor */
|
||||
const RS_GROUP_COLOR = {
|
||||
formant: '--accent', pitch: '--accent-2', amp: '--good', filter: '--warn', fx: '--info', mod: '--accent-3',
|
||||
};
|
||||
const RS_NEXT = { off: 'fixed', fixed: 'live', live: 'off' };
|
||||
|
||||
function ReadoutStrip({ params, values, onChange, pinned, onTogglePin }) {
|
||||
const [open, setOpen] = React.useState(null);
|
||||
const timers = React.useRef({ open: null, close: null });
|
||||
const drag = React.useRef({ i: -1, moved: false, startY: 0, el: null, alt: false });
|
||||
|
||||
const scheduleOpen = (i) => { clearTimeout(timers.current.close); clearTimeout(timers.current.open); timers.current.open = setTimeout(() => setOpen(i), 110); };
|
||||
const scheduleClose = () => { clearTimeout(timers.current.open); clearTimeout(timers.current.close); timers.current.close = setTimeout(() => setOpen(null), 280); };
|
||||
const hold = () => clearTimeout(timers.current.close);
|
||||
|
||||
const valFromEvent = (el, clientY) => { const r = el.getBoundingClientRect(); return Math.max(0, Math.min(1, 1 - (clientY - r.top) / r.height)); };
|
||||
const down = (e, i) => { e.currentTarget.setPointerCapture?.(e.pointerId); drag.current = { i, moved: false, startY: e.clientY, el: e.currentTarget, alt: e.altKey || e.metaKey }; };
|
||||
const move = (e, i) => { const d = drag.current; if (d.i !== i) return; if (Math.abs(e.clientY - d.startY) > 3) d.moved = true; if (d.moved && !d.alt) onChange(i, { val: valFromEvent(d.el, e.clientY) }); };
|
||||
const up = (e, i) => {
|
||||
const d = drag.current; if (d.i !== i) return;
|
||||
if (d.alt && !d.moved) onChange(i, { status: RS_NEXT[params[i].status] || 'live' });
|
||||
else if (!d.moved) onChange(i, { val: valFromEvent(d.el, e.clientY) });
|
||||
drag.current = { i: -1, moved: false, startY: 0, el: null, alt: false };
|
||||
};
|
||||
|
||||
return (
|
||||
<div style={{ display: 'flex', alignItems: 'stretch', gap: 2, height: 76, padding: '0 2px',
|
||||
background: 'var(--glass)', backdropFilter: 'blur(12px)', WebkitBackdropFilter: 'blur(12px)',
|
||||
borderBottom: '1px solid var(--glass-line)', position: 'relative', zIndex: 30 }}>
|
||||
<button type="button" onClick={onTogglePin} title="Pin strip open"
|
||||
style={{ flex: '0 0 auto', width: 30, border: 0, background: 'transparent', color: pinned ? 'var(--accent)' : 'var(--fg-dim)', cursor: 'pointer', fontSize: 'var(--fs-md)' }}>
|
||||
{pinned ? '📌' : '▾'}
|
||||
</button>
|
||||
{params.map((p, i) => {
|
||||
const eff = values[i] ?? 0;
|
||||
const gc = `var(${RS_GROUP_COLOR[p.group] || '--accent'})`;
|
||||
const dim = p.status === 'off';
|
||||
const placeRight = i > params.length - 5;
|
||||
return (
|
||||
<div key={i} style={{ position: 'relative', flex: 1, minWidth: 0, display: 'flex', flexDirection: 'column' }}
|
||||
onPointerLeave={scheduleClose}>
|
||||
{/* hover target: name ONLY (not the bar) */}
|
||||
<div onPointerEnter={() => scheduleOpen(i)} style={{ textAlign: 'center', fontSize: 8, fontFamily: 'var(--font-mono)', lineHeight: '11px', cursor: 'help',
|
||||
color: p.status === 'live' ? 'var(--fg-dim)' : gc, overflow: 'hidden', whiteSpace: 'nowrap', textOverflow: 'ellipsis' }}>{p.name}</div>
|
||||
<div onPointerDown={(e) => down(e, i)} onPointerMove={(e) => move(e, i)} onPointerUp={(e) => up(e, i)} onPointerCancel={(e) => up(e, i)}
|
||||
style={{ position: 'relative', flex: 1, background: 'var(--bg)', borderRadius: 2, overflow: 'hidden', cursor: 'ns-resize', opacity: dim ? 0.5 : 1, touchAction: 'none' }}>
|
||||
<div style={{ position: 'absolute', left: 0, right: 0, bottom: 0, height: `${eff * 100}%`, background: gc, opacity: 0.25 + eff * 0.6, transition: 'height 60ms linear' }} />
|
||||
{p.status === 'live' && <div style={{ position: 'absolute', left: 0, right: 0, bottom: `${p.val * 100}%`, height: 0, borderTop: '1px dashed rgba(255,255,255,0.3)' }} />}
|
||||
{p.status !== 'live' && <div style={{ position: 'absolute', top: 1, left: 0, right: 0, textAlign: 'center', fontSize: 8, color: p.status === 'fixed' ? 'var(--accent-2)' : 'var(--fg-dim)' }}>{p.status === 'fixed' ? '⊟' : '∅'}</div>}
|
||||
</div>
|
||||
{open === i && (
|
||||
<window.OutputEditor param={p} onChange={(patch) => onChange(i, patch)} onHold={hold} onLeave={scheduleClose}
|
||||
place={{ top: 'calc(100% + 6px)', [placeRight ? 'right' : 'left']: 0 }} />
|
||||
)}
|
||||
</div>
|
||||
);
|
||||
})}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
window.ReadoutStrip = ReadoutStrip;
|
||||
|
|
@ -0,0 +1,16 @@
|
|||
/* Console 2.0 — SplitStage: input and output given EQUAL prominence, side by
|
||||
side. Left half = the Manifold (input), right half = the OutputStage (output
|
||||
field). Both fully interactive; neither dominates. */
|
||||
function SplitStage({ pos, onMove, noiseCap, pins, follow, onLongPress, params, values, onChange }) {
|
||||
return (
|
||||
<div style={{ position: 'absolute', inset: 0, display: 'flex' }}>
|
||||
<div style={{ flex: 1, position: 'relative', borderRight: '1px solid var(--line)', minWidth: 0 }}>
|
||||
<window.Manifold pos={pos} onMove={onMove} noiseCap={noiseCap} pins={pins} follow={follow} onLongPress={onLongPress} />
|
||||
</div>
|
||||
<div style={{ flex: 1, position: 'relative', minWidth: 0 }}>
|
||||
<window.OutputStage params={params} values={values} onChange={onChange} compact />
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
window.SplitStage = SplitStage;
|
||||
|
|
@ -0,0 +1,76 @@
|
|||
/* Console 2.0 — Verdict cluster: floating bottom-center, the app's main control.
|
||||
▽ perturb · ↺ undo · △ commit, + A/B toggle. Long-press perturb = full re-roll.
|
||||
Rests at low opacity, full on hover. */
|
||||
function ThumbIcon({ size = 24, down = false }) {
|
||||
// Minimal line-icon thumb, hairline stroke in currentColor.
|
||||
return (
|
||||
<svg width={size} height={size} viewBox="0 0 24 24" fill="none"
|
||||
stroke="currentColor" strokeWidth="1.6" strokeLinecap="round" strokeLinejoin="round"
|
||||
style={{ transform: down ? 'rotate(180deg)' : 'none', display: 'block' }} aria-hidden="true">
|
||||
<path d="M14 9V5a2.4 2.4 0 0 0-2.4-2.4L8 11v9h8.1a1.6 1.6 0 0 0 1.6-1.36l1.1-7.2A1.6 1.6 0 0 0 17.2 9z" />
|
||||
<path d="M8 20H5.6A1.6 1.6 0 0 1 4 18.4v-5.8A1.6 1.6 0 0 1 5.6 11H8" />
|
||||
</svg>
|
||||
);
|
||||
}
|
||||
window.ThumbIcon = ThumbIcon;
|
||||
|
||||
function VerdictCluster({ onPerturb, onUndo, onCommit, onReroll, canUndo, ab, onToggleAB, onHoldA, firstSession }) {
|
||||
const [hover, setHover] = React.useState(false);
|
||||
const lp = React.useRef(null);
|
||||
const firedReroll = React.useRef(false);
|
||||
|
||||
const perturbDown = () => {
|
||||
firedReroll.current = false;
|
||||
lp.current = setTimeout(() => { firedReroll.current = true; onReroll(); }, 600);
|
||||
};
|
||||
const perturbUp = () => { clearTimeout(lp.current); if (!firedReroll.current) onPerturb(); };
|
||||
|
||||
const big = (extra) => ({
|
||||
width: 64, height: 64, borderRadius: '50%', fontSize: 26, cursor: 'pointer',
|
||||
display: 'flex', alignItems: 'center', justifyContent: 'center', fontFamily: 'var(--font-mono)',
|
||||
border: '1px solid var(--glass-line)', transition: 'transform var(--dur-fast) var(--ease-console), background var(--dur-fast)', ...extra,
|
||||
});
|
||||
|
||||
return (
|
||||
<div
|
||||
onPointerEnter={() => setHover(true)} onPointerLeave={() => setHover(false)}
|
||||
style={{
|
||||
position: 'absolute', bottom: 28, left: '50%', transform: 'translateX(-50%)',
|
||||
display: 'flex', alignItems: 'center', gap: 'var(--sp-3)', padding: 'var(--sp-2) var(--sp-3)',
|
||||
background: 'var(--glass)', backdropFilter: 'blur(14px)', WebkitBackdropFilter: 'blur(14px)',
|
||||
border: '1px solid var(--glass-line)', borderRadius: 'var(--r-pill)', boxShadow: 'var(--shadow-2)',
|
||||
opacity: hover || firstSession ? 1 : 0.55, transition: 'opacity var(--dur-med) var(--ease-console)', zIndex: 40,
|
||||
}}
|
||||
>
|
||||
<button type="button" title="Perturb — thumbs down (hold to re-roll)"
|
||||
onPointerDown={perturbDown} onPointerUp={perturbUp} onPointerLeave={() => clearTimeout(lp.current)}
|
||||
style={big({ background: 'rgba(255,68,102,0.16)', color: 'var(--danger)' })}
|
||||
onMouseEnter={(e) => e.currentTarget.style.transform = 'scale(1.08)'}
|
||||
onMouseLeave={(e) => e.currentTarget.style.transform = 'scale(1)'}><ThumbIcon down /></button>
|
||||
|
||||
<button type="button" title="Undo (z)" onClick={onUndo} disabled={!canUndo}
|
||||
style={big({ width: 48, height: 48, fontSize: 20, background: 'var(--bg-2)', color: 'var(--fg-mute)', opacity: canUndo ? 1 : 0.4, cursor: canUndo ? 'pointer' : 'not-allowed' })}>↺</button>
|
||||
|
||||
<button type="button" title="Commit — thumbs up" onClick={onCommit}
|
||||
style={big({ background: 'rgba(255,106,0,0.18)', color: 'var(--accent)', boxShadow: '0 0 16px var(--glow-accent)' })}
|
||||
onMouseEnter={(e) => e.currentTarget.style.transform = 'scale(1.08)'}
|
||||
onMouseLeave={(e) => e.currentTarget.style.transform = 'scale(1)'}><ThumbIcon /></button>
|
||||
|
||||
<div style={{ width: 1, height: 36, background: 'var(--glass-line)', margin: '0 2px' }} />
|
||||
|
||||
<div
|
||||
title="A/B compare — hold to preview A"
|
||||
onPointerDown={() => onHoldA(true)} onPointerUp={() => onHoldA(false)} onPointerLeave={() => onHoldA(false)}
|
||||
onClick={onToggleAB}
|
||||
style={{ display: 'flex', borderRadius: 'var(--r-pill)', overflow: 'hidden', border: '1px solid var(--glass-line)', cursor: 'pointer', userSelect: 'none' }}>
|
||||
{['A', 'B'].map((k) => (
|
||||
<span key={k} style={{
|
||||
padding: '8px 14px', fontSize: 'var(--fs-sm)', fontWeight: 600,
|
||||
background: ab === k ? 'var(--accent)' : 'transparent', color: ab === k ? 'var(--bg)' : 'var(--fg-mute)',
|
||||
}}>{k}</span>
|
||||
))}
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
window.VerdictCluster = VerdictCluster;
|
||||
122
docs/redesign/manifold-export/ui_kits/memlnaut/console/model.jsx
Normal file
122
docs/redesign/manifold-export/ui_kits/memlnaut/console/model.jsx
Normal file
|
|
@ -0,0 +1,122 @@
|
|||
/* Console 2.0 — shared instrument model. Pure data + inference used by all
|
||||
three altitudes (Console / Perform / Zen) so they share one spine. */
|
||||
function mkParams(spec) {
|
||||
const out = [];
|
||||
for (const [group, names] of spec) {
|
||||
names.forEach((name) => out.push({ name, group, status: 'live', val: 0.5, min: 0, max: 1, curve: 0.5 }));
|
||||
}
|
||||
return out;
|
||||
}
|
||||
const MF_MODES = [
|
||||
{ id: 'paf_synth', label: 'PAF Synth', cls: 'Synth', glyph: '∿', input: 'xy',
|
||||
params: mkParams([['formant', ['F1', 'F2', 'F3', 'tilt', 'spread', 'skirt']], ['pitch', ['root', 'glide', 'detune']], ['amp', ['gain', 'attack', 'decay']], ['filter', ['cutoff', 'res', 'env']], ['fx', ['drive', 'air', 'width']]]) },
|
||||
{ id: 'channel_strip', label: 'Channel Strip', cls: 'Synth', glyph: '▤', input: 'joystick',
|
||||
params: mkParams([['filter', ['lo', 'loMid', 'hiMid', 'hi']], ['amp', ['comp', 'gate', 'makeup']], ['fx', ['sat', 'width', 'glue', 'tilt', 'air']]]) },
|
||||
{ id: 'verb_fx', label: 'Verb FX', cls: 'Synth', glyph: '◞', input: 'joystick',
|
||||
params: mkParams([['fx', ['size', 'decay', 'damp', 'diff']], ['mod', ['rate', 'depth']], ['filter', ['lo', 'hi']]]) },
|
||||
{ id: 'elysiamorf', label: 'Elysiamorf', cls: 'Synth', glyph: '❋', input: 'xy',
|
||||
params: mkParams([['formant', ['grain', 'size', 'pos', 'spray']], ['mod', ['rate', 'depth', 'jitter']], ['amp', ['gain', 'env']], ['filter', ['cutoff', 'res']], ['fx', ['blur', 'shimmer', 'freeze', 'width']]]) },
|
||||
{ id: 'memlcelium', label: 'MEML Celium', cls: 'Sequencer', glyph: '☷', input: 'xy',
|
||||
params: mkParams([['mod', ['cvA', 'cvB', 'gate', 'div']], ['pitch', ['root', 'scale', 'oct']], ['amp', ['vca', 'slew']]]) },
|
||||
{ id: 'breakor', label: 'Breakor', cls: 'Sequencer', glyph: '⊟', input: 'joystick',
|
||||
params: mkParams([['mod', ['density', 'swing', 'fill', 'stutter']], ['amp', ['punch', 'decay']], ['filter', ['tone', 'crush']], ['fx', ['glitch', 'rev']]]) },
|
||||
{ id: 'sound_analysis_midi', label: 'Sound Analysis → MIDI', cls: 'Controller', glyph: '⇉', input: 'audio_in', badge: '1-input',
|
||||
params: mkParams([['mod', ['cc1', 'cc2', 'cc3', 'cc4']], ['pitch', ['note', 'bend']], ['amp', ['vel', 'press']]]) },
|
||||
{ id: 'visualizer', label: 'Visualizer', cls: 'Visual', glyph: '◑', input: 'xy',
|
||||
params: mkParams([['mod', ['hue', 'sat', 'flow', 'warp']], ['amp', ['bloom', 'fade']], ['fx', ['grain', 'trail']]]) },
|
||||
{ id: 'c15', label: 'C15', cls: 'Synth', glyph: '◆', input: 'xy', placeholder: true, badge: 'soon', params: mkParams([['amp', ['a', 'b']]]) },
|
||||
];
|
||||
|
||||
function applyCurve(v, c) { const e = 0.25 + c * 1.75; return Math.pow(Math.max(0, Math.min(1, v)), e); }
|
||||
function MF_infer(pos, seed, params) {
|
||||
const [x, y] = pos;
|
||||
return params.map((p, i) => {
|
||||
if (p.status === 'off') return 0; // deactivated → muted
|
||||
if (p.status === 'fixed') return p.val ?? 0.5; // deactivated → held static
|
||||
// live → driven by the model, shaped by min/max/curve
|
||||
const a = Math.sin(x * (1.2 + i * 0.19) * Math.PI + i * 0.7 + seed) * 0.5 + 0.5;
|
||||
const b = Math.cos(y * (0.9 + i * 0.13) * Math.PI - i * 0.4 - seed * 0.5) * 0.5 + 0.5;
|
||||
let v = a * 0.55 + b * 0.45;
|
||||
v = p.min + applyCurve(v, p.curve) * (p.max - p.min);
|
||||
return Math.max(0, Math.min(1, v));
|
||||
});
|
||||
}
|
||||
function MF_seededGradient(rev) {
|
||||
const n = 4, norms = [], status = [];
|
||||
for (let i = 0; i < n; i++) {
|
||||
const r = Math.abs(Math.sin((rev + 1) * (i + 1) * 12.9898) * 43758.5453 % 1);
|
||||
norms.push(0.2 + r * 0.8);
|
||||
status.push(r > 0.85 ? 'exploding' : r < 0.18 ? 'vanishing' : r < 0.3 ? 'converged' : 'healthy');
|
||||
}
|
||||
return { norms, status };
|
||||
}
|
||||
|
||||
/* useInstrument — the shared reactive spine + verdict/training actions.
|
||||
Returns flat state + actions; each view renders as much of it as it needs. */
|
||||
let MF_SNAP_ID = 0;
|
||||
function useInstrument(initialMode) {
|
||||
const [modeId, setModeId] = React.useState(initialMode || 'paf_synth');
|
||||
const mode = MF_MODES.find((m) => m.id === modeId) || MF_MODES[0];
|
||||
const [params, setParams] = React.useState(() => mode.params.map((p) => ({ ...p })));
|
||||
const [pos, setPos] = React.useState([0.5, 0.5]);
|
||||
const [seed, setSeed] = React.useState(0.4);
|
||||
const [axes, setAxes] = React.useState({ boldness: 0.55, memory: 0.4, precision: 0.5 });
|
||||
const [noiseCap, setNoiseCap] = React.useState(0.12);
|
||||
const [examples, setExamples] = React.useState(0);
|
||||
const [loss, setLoss] = React.useState([]);
|
||||
const [busy, setBusy] = React.useState(false);
|
||||
const [snapshots, setSnapshots] = React.useState([]);
|
||||
const [ab, setAB] = React.useState('B');
|
||||
const [holdingA, setHoldingA] = React.useState(false);
|
||||
const aRef = React.useRef(null);
|
||||
const [health, setHealth] = React.useState(0.8);
|
||||
const [rev, setRev] = React.useState(1);
|
||||
const [follow, setFollow] = React.useState(false);
|
||||
const [pins, setPins] = React.useState([]);
|
||||
const [firstSession, setFirstSession] = React.useState(true);
|
||||
|
||||
React.useEffect(() => {
|
||||
setParams(mode.params.map((p) => ({ ...p }))); setPos([0.5, 0.5]); setExamples(0);
|
||||
setLoss([]); setSnapshots([]); setSeed(0.4); setFollow(false); setPins([]); setHealth(0.8); setRev(1);
|
||||
}, [modeId]);
|
||||
|
||||
const effSeed = holdingA && aRef.current ? aRef.current.seed : seed;
|
||||
const values = React.useMemo(() => MF_infer(pos, effSeed, params), [pos, effSeed, params]);
|
||||
const gradient = React.useMemo(() => MF_seededGradient(rev), [rev]);
|
||||
|
||||
const pushSnap = (tag) => setSnapshots((s) => [...s, { id: ++MF_SNAP_ID, tag, noise: noiseCap, seed }].slice(-50));
|
||||
const commit = () => {
|
||||
setFirstSession(false); pushSnap('commit +'); setBusy(true);
|
||||
setNoiseCap((n) => Math.max(0.02, n * 0.7)); setHealth((h) => Math.min(1, h + 0.08)); setRev((r) => r + 1);
|
||||
setTimeout(() => { setLoss((prev) => { const b = prev.length ? prev[prev.length - 1] : 0.5; return [...prev, Math.max(0.004, b * (0.8 + Math.random() * 0.08))].slice(-120); }); setBusy(false); }, 260);
|
||||
};
|
||||
const perturb = () => {
|
||||
setFirstSession(false); pushSnap('perturb −');
|
||||
setSeed((s) => s + (Math.random() - 0.5) * (noiseCap * 4 + 0.3));
|
||||
setNoiseCap((n) => Math.min(0.5, n + 0.06)); setHealth((h) => Math.max(0.1, h - 0.06)); setRev((r) => r + 1);
|
||||
};
|
||||
const reroll = () => { setFirstSession(false); pushSnap('re-roll'); setSeed(Math.random() * 6); setNoiseCap(0.4); setHealth(0.5); setRev((r) => r + 1); };
|
||||
const undo = () => setSnapshots((s) => { if (!s.length) return s; const last = s[s.length - 1]; setSeed(last.seed); setNoiseCap(last.noise); setRev((r) => r + 1); return s.slice(0, -1); });
|
||||
const train = () => { setBusy(true); setTimeout(() => { setLoss((p) => { const b = p.length ? p[p.length - 1] : 0.5; return [...p, Math.max(0.004, b * 0.82)].slice(-120); }); setBusy(false); }, 260); };
|
||||
const setParam = (i, patch) => setParams((ps) => ps.map((p, j) => (j === i ? { ...p, ...patch } : p)));
|
||||
const cycleStatus = (i) => setParams((ps) => ps.map((p, j) => (j === i ? { ...p, status: ({ off: 'fixed', fixed: 'live', live: 'off' })[p.status] } : p)));
|
||||
const toggleAB = () => { if (ab === 'B') { aRef.current = { seed }; setAB('A'); } else { if (aRef.current) setSeed(aRef.current.seed); setAB('B'); } };
|
||||
const cycleMode = (dir = 1) => {
|
||||
const live = MF_MODES.filter((m) => !m.placeholder);
|
||||
const i = live.findIndex((m) => m.id === modeId);
|
||||
setModeId(live[(i + dir + live.length) % live.length].id);
|
||||
};
|
||||
|
||||
return {
|
||||
MODES: MF_MODES, modeId, setModeId, cycleMode, mode, params, setParam, cycleStatus,
|
||||
pos, setPos, seed, values, axes, setAxes, setAxis: (k, v) => setAxes((s) => ({ ...s, [k]: v })),
|
||||
noiseCap, setNoiseCap, examples, loss, busy, train,
|
||||
snapshots, commit, perturb, reroll, undo, ab, toggleAB, holdingA, setHoldingA,
|
||||
health, rev, gradient, follow, setFollow, pins, setPins, firstSession,
|
||||
};
|
||||
}
|
||||
|
||||
window.MF_MODES = MF_MODES;
|
||||
window.MF_infer = MF_infer;
|
||||
window.MF_seededGradient = MF_seededGradient;
|
||||
window.useInstrument = useInstrument;
|
||||
|
|
@ -0,0 +1,65 @@
|
|||
/* Console 2.0 — shared chrome for the simpler altitudes. */
|
||||
const ALT_HREF = {
|
||||
in: { console: 'index.html', perform: 'perform.html', zen: 'zen.html' },
|
||||
split: { console: 'console-split.html', perform: 'perform-split.html', zen: 'zen-split.html' },
|
||||
out: { console: 'console-output.html', perform: 'perform-output.html', zen: 'zen-output.html' },
|
||||
composite: { console: 'console-composite.html', perform: 'perform-split.html', zen: 'zen-split.html' },
|
||||
};
|
||||
function AltitudeNav({ current, focus = 'in', style }) {
|
||||
const items = [
|
||||
{ id: 'console', dots: '◆◆◆', label: 'Console' },
|
||||
{ id: 'perform', dots: '◆◆', label: 'Perform' },
|
||||
{ id: 'zen', dots: '◆', label: 'Zen' },
|
||||
];
|
||||
const pill = (on) => ({ textDecoration: 'none', fontSize: 11, padding: '2px 8px', borderRadius: 'var(--r-pill)',
|
||||
color: on ? 'var(--accent)' : 'var(--fg-dim)', background: on ? 'rgba(255,106,0,0.14)' : 'transparent' });
|
||||
const foc = [['in', 'IN'], ['split', 'DUAL'], ['out', 'OUT'], ['composite', 'FLEX']];
|
||||
return (
|
||||
<div style={{ position: 'absolute', top: 12, right: 14, zIndex: 70, display: 'flex', gap: 6, alignItems: 'center',
|
||||
background: 'var(--glass)', backdropFilter: 'blur(10px)', WebkitBackdropFilter: 'blur(10px)',
|
||||
border: '1px solid var(--glass-line)', borderRadius: 'var(--r-pill)', padding: '4px 6px', ...style }}>
|
||||
{items.map((it) => (
|
||||
<a key={it.id} href={ALT_HREF[focus][it.id]} title={`${it.label} · ${focus}`}
|
||||
style={pill(it.id === current)}>{it.dots}</a>
|
||||
))}
|
||||
<span style={{ width: 1, height: 16, background: 'var(--glass-line)' }} />
|
||||
{foc.map(([f, label]) => (
|
||||
<a key={f} href={ALT_HREF[f][current]} title={f === 'split' ? 'Input + output equal' : f === 'out' ? 'Output-first' : f === 'composite' ? 'Composite — drag to rebalance' : 'Input-first'}
|
||||
style={{ ...pill(focus === f), fontSize: 9, letterSpacing: '0.08em' }}>{label}</a>
|
||||
))}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
/* MiniMeters — glanceable read-only output bars (no interaction). */
|
||||
function MiniMeters({ params, values }) {
|
||||
const GC = { formant: '--accent', pitch: '--accent-2', amp: '--good', filter: '--warn', fx: '--info', mod: '--accent-3' };
|
||||
return (
|
||||
<div style={{ display: 'flex', alignItems: 'flex-end', gap: 2, height: 40 }}>
|
||||
{values.map((v, i) => (
|
||||
<div key={i} title={`${params[i].name}: ${v.toFixed(2)}`}
|
||||
style={{ width: 5, height: '100%', background: 'var(--bg-2)', borderRadius: 1, position: 'relative', overflow: 'hidden' }}>
|
||||
<div style={{ position: 'absolute', left: 0, right: 0, bottom: 0, height: `${v * 100}%`,
|
||||
background: `var(${GC[params[i].group] || '--accent'})`, opacity: 0.3 + v * 0.6 }} />
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
/* CompactAxis — slim labelled feel slider for the Perform bar. */
|
||||
function CompactAxis({ label, value, onChange, accent = 'var(--accent)' }) {
|
||||
return (
|
||||
<label style={{ display: 'flex', alignItems: 'center', gap: 'var(--sp-2)', fontFamily: 'var(--font-mono)' }}>
|
||||
<span style={{ width: 64, fontSize: 10, color: 'var(--fg-mute)', textTransform: 'uppercase', letterSpacing: '0.08em' }}>{label}</span>
|
||||
<input type="range" min="0" max="1" step="0.01" value={value}
|
||||
onChange={(e) => onChange(parseFloat(e.target.value))}
|
||||
className="mf-slider-input" style={{ width: 120, '--mf-axis-accent': accent }} />
|
||||
<span style={{ width: '3ch', fontSize: 10, color: 'var(--fg-dim)', fontVariantNumeric: 'tabular-nums', textAlign: 'right' }}>{value.toFixed(2)}</span>
|
||||
</label>
|
||||
);
|
||||
}
|
||||
|
||||
window.AltitudeNav = AltitudeNav;
|
||||
window.MiniMeters = MiniMeters;
|
||||
window.CompactAxis = CompactAxis;
|
||||
398
docs/redesign/playground-2.0-rewrite-plan.md
Normal file
398
docs/redesign/playground-2.0-rewrite-plan.md
Normal file
|
|
@ -0,0 +1,398 @@
|
|||
# NISPS Playground 2.0 — Clean-Room SolidJS Rewrite Plan
|
||||
|
||||
*Status: sign-off-ready. Author: lead architect, synthesizing four designer proposals (P1–P4) against three judge panels (mission-fit, feasibility, UX coherence). Date: 2026-06-17.*
|
||||
|
||||
---
|
||||
|
||||
## TL;DR / recommendation
|
||||
|
||||
We rewrite the browser playground clean-room in SolidJS as **one fullscreen instrument** whose interaction model is **P3's "Console"** (full-bleed canvas + always-on Verdict cluster + a right-edge dock whose drawers have three depth states), driven internally by **P2's single-owned-reactive-spine architecture** (one pure `createMemo` chain from input → ML → output, read by every consumer; impure side-effects in one `createEffect`). The whole UI is **one schema-driven `<GenericMode>`** replacing today's 8–9 cloned mode files, so adding an engine is "a JSON schema + a C++ lambda + codegen, zero UI." We adopt **P4's concepts** — the snapshot DAG (unifying undo/A-B/trail/history) and the control-point `off/fixed/live` enum (collapsing the three overlapping mute/pin/expose systems) — but **explicitly reject P4's v1 commitment to a runtime-shaped MLP**: v1 ships an *honest fixed-2-input contract that fails the build on schema/arch mismatch*, deferring multi-input modes until the C++ core genuinely supports them behind a passing parity check.
|
||||
|
||||
Why this is more mature than today's app: (1) the live-feedback regression that has haunted every prior session ("MLP output stops updating when I move the joystick") becomes **structurally impossible and e2e-asserted**, because there is exactly one reactive path from input to output and every consumer reads it; (2) the 663-line `mode-runtime` god-hook and 8 near-identical mode files collapse into composable hooks + one data-driven component — the orthogonality principle finally reaches the UI layer; (3) two real research-validity bugs (fake single-point loss history, training on post-pipeline outputs) are fixed; (4) the feature inventory is treated as a **capability budget to make reachable**, not a screen budget to fill — most features live one drawer/keystroke/probe-call deep rather than cluttering the canvas.
|
||||
|
||||
---
|
||||
|
||||
## 1. Mission & design thesis
|
||||
|
||||
NISPS is **a research platform for interactive ML control of audio** — a *controller* for synths, not a synth. The north star for 2.0:
|
||||
|
||||
> **The network is the medium you shape; the gesture is the cursor; reactivity is the wire that must never break. You shape a mapping from embodied gesture to sound by listening and giving verdicts (👍/👎), and the screen reveals exactly as much machinery as you reach for.**
|
||||
|
||||
This serves the five-axis **orthogonality principle** (parameter sets · modes · ML architectures · audio engines · UI/UX must vary independently) by making each axis a different layer that can be edited without disturbing the others — and it serves the **crystallization pipeline** (browser is where research happens; firmware is where findings ossify) by keeping the browser identical to firmware *by default* and treating divergence (e.g. the `spread`/Xavier regime) as an opt-in lab toggle.
|
||||
|
||||
Three personas in priority order, all served: **Dimi-researcher** (wants diagnostics that answer "is it learning?"), **Dimi-performer** (embodied, eyes on the synth, thumb on the verdict), and **AI coding agents** (the synchronous debug probe + Playwright + parity scripts are first-class — new UX must remain headlessly drivable and observable).
|
||||
|
||||
The design discipline, taken from P1 and endorsed by the mission-fit judge: **a capability budget, not a screen budget.** The brief's enormous inventory is something to make *reachable* (one gesture/keystroke/probe-call deep), not something to render all at once. This is the direct antidote to cargo-cult feature parity.
|
||||
|
||||
---
|
||||
|
||||
## 2. Intended UX, locked
|
||||
|
||||
### 2.1 Stance: desktop-first, touch-correct primitives
|
||||
|
||||
Confirmed as decided (§7-H of the brief, pending Dimi's confirmation it still holds). Layout target **1280×800+ landscape**; the Console docks right, the manifold owns the rest. Every primitive that touches the manifold (joystick, zoom, trail, heatmap-scrub, verdict buttons) is built on **Pointer Events with `touch-action:none` + pointer-capture**, so the same code is finger-correct on a foldable without a separate mobile IA. Below ~720px wide, the Console degrades to a bottom-sheet variant of the *same* components (one responsive breakpoint). **Touch-correctness of joystick/zoom/trail is a v1 acceptance criterion**, not deferred. Pinch maps to the same zoom signal as wheel/momentum.
|
||||
|
||||
### 2.2 Information architecture: three persistent zones + the Console
|
||||
|
||||
There is **one screen** (`/`). The only other routes are the hidden `/dev/primitives` showcase and the `?debug=1`-gated probe.
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────┐
|
||||
│ ❶ READOUT STRIP (top, interactive heatmap, auto-hides) │
|
||||
├──────────────────────────────────────────────┬──────────┤
|
||||
│ │ ❹ THE │
|
||||
│ ❷ THE MANIFOLD │ CONSOLE │
|
||||
│ (fullscreen canvas: visualizer/seq grid + │ (48px │
|
||||
│ joy-map + trail + noise rings + pins) │ dock + │
|
||||
│ │ drawer) │
|
||||
│ ❸ VERDICT CLUSTER (floating, bottom) │ │
|
||||
└──────────────────────────────────────────────┴──────────┘
|
||||
◀ Drawer slides in here (left of dock, mutually exclusive)
|
||||
```
|
||||
|
||||
- **❶ Readout strip** — the interactive output heatmap (§2.5). Auto-hides in pure-synth modes (the visualizer is the readout); pin glyph holds it open.
|
||||
- **❷ The Manifold** — the always-on canvas: engine/visual/sequencer output, the zoom joy-map minimap (adaptive graph-paper grid), the clickable vanishing trail, noise rings, pinned-region overlays, the joystick dot. **The empty canvas is itself the primary input surface** (pointer-down drives the joystick in joy-map mode), with hit-test priority: trail points and pins capture before the joystick.
|
||||
- **❸ Verdict cluster** — floating bottom-center, the most-used control in the app, never hidden on first session: **👎 perturb · ↺ undo · 👍 commit**, with A/B toggle to the right; Randomize demoted to long-press-👎 ("thumbs-down really hard", confirm-on-release). Rests at low opacity, full on hover/active.
|
||||
- **❹ The Console** — right-edge dock (48px icon rail) + a single mutually-exclusive drawer sliding in to its left. **This resolves the dock-vs-floating-bar conflict in favor of the dock.**
|
||||
|
||||
All HUD anchors are **derived from a reactive layout context** (`useStageMetrics()`), never hardcoded pixels — killing the "tied to a removed 88px bottom sheet" bug class at the source. Chrome auto-dissolves after ~4s of inactivity (Design-C heritage) via a single `chromeOpacity` signal, restored on any pointer/key/`__nisps` activity — **but never fades on the first session**, and dock icons never fully vanish (a thin orange edge persists).
|
||||
|
||||
Visual language is **faithful to a-immersive, not the cyan drift**: accent orange `#ff6a00`, danger `#ff4466`, glass `rgba(13,13,13,0.65)` on dark, JetBrains Mono @13px, easing `cubic-bezier(0.22,1,0.36,1)`. A lint allowlist of CSS vars enforces it.
|
||||
|
||||
### 2.3 The Console drawers — split by mental mode, not by data
|
||||
|
||||
Five dock sections. The key UX call (from P3, endorsed by the UX-coherence judge as "the strongest anti-modal-confusion move in the set"): **separate the performer's control surface from the researcher's diagnostics** — never make someone playing share a panel with gradient-flow bars.
|
||||
|
||||
| Icon | Drawer | Mental mode | Contents |
|
||||
|---|---|---|---|
|
||||
| **Shape** | RL feel + examples + history | shaping | add example, train, clear, dataset count; RL feel controls; snapshot DAG browser at Full depth |
|
||||
| **Feel** | compound axes + presets | playing | Boldness / Memory / Precision axes; trim-pot overrides; six control-preset chips |
|
||||
| **Route** | input/output + control-point matrix | wiring | input source, output backend (Audio/MIDI/OSC/controller), per-param control-point rows, per-group curves; Emitters×Targets matrix at Full depth (dense/modular modes) |
|
||||
| **Health** | diagnostics | debugging | weight-health detail, gradient-flow, layer-stats, loss plot, `spread`/`tame` lab controls |
|
||||
| **Help** | overlay + keymap | — | keyboard map, first-run guidance |
|
||||
|
||||
`Mode/engine switching` lives in a compact switcher at the top of the dock (grouped by capability class — see §2.10), so it is always reachable without colonizing a drawer.
|
||||
|
||||
### 2.4 Progressive disclosure: drawer depth = one learnable gesture
|
||||
|
||||
Adopted from P3, backed by P4's data model. Each drawer has **three depth states**, and that *is* the disclosure mechanism — there is **no global "Advanced" toggle**:
|
||||
|
||||
| Depth | Width | Trigger | Contents |
|
||||
|---|---|---|---|
|
||||
| **Peek** | ~320px | click dock icon | summary chips (`Sources 8/48 · Targets 6/10 · live 4`) + the one primary control + read-only sparklines |
|
||||
| **Expand** | ~520px | "⤢ More" / drag-resize | active controls only + collapsible rails + "show all" footer |
|
||||
| **Full** | fullscreen modal | "⤢⤢ Open" | the dense surface: full matrix, per-cell editor, all overrides, exposure controls |
|
||||
|
||||
"What shows at each depth" is **data, not hand-wiring**: each schema param and each diagnostic declares a `tier: 0|1|2`, and the current depth reveals tiers ≤ depth. This makes Dimi's "some modes expert-only" wish (ALIGNMENT Q2) just a schema default for `tier`, and unifies the brief's five competing disclosure proposals (global toggle / per-mode expertise / three-tier / control-point / staged-unlock) into one mechanism. The **Feel drawer's Peek auto-opens** on first load and after mode-switch (so the compound axes are glanceable without a click — the "always-on bar" intent without permanently stealing canvas), and a **mini-axis ghost** persists on the Feel dock icon when collapsed.
|
||||
|
||||
The dense **staged-unlock** model (Dimi's near-decided intent) lives inside the Route drawer's Full tier: a mode starts with a small set of `live` control points; the user opts more in one-by-one or swaps them via the "show all / opt-in" footer.
|
||||
|
||||
### 2.5 The interactive heatmap (readout strip)
|
||||
|
||||
Output readout becomes a control surface. Two interactions per cell, with the **load-bearing tactile constants reproduced exactly**:
|
||||
|
||||
1. **Drag** (>3px motion threshold, pointer-capture, horizontal across a cell) → directly sets that param's normalized value to the x-position; output reroutes immediately.
|
||||
2. **Click** (pointerup, no motion) → toggles a per-parameter **override popup** anchored to the cell: min slider, max slider, curve slider, **freeze toggle** (frozen → drag sets a fixed value, bypassing the model), with **300ms grace** dismissal so the pointer can travel cell→popup.
|
||||
|
||||
Hover shows `{name}: {value}` + a `▾` clickability hint, live during drag. Frozen cells dimmed. The strip is hidden in synth mode (the visualizer is the readout). These constants (3px / 300ms / pointer-capture) are non-negotiable craft.
|
||||
|
||||
### 2.6 The core interaction loop (explore → shape → train → evaluate → compare)
|
||||
|
||||
**Explore.** Drag the manifold (or joystick/sliders/mic). On every pointer-move (reactive, *not* rAF), the input pipeline runs `deadzone → zoom → curve → smoothing → momentum-as-zoom`; the result drives inference; outputs flow to the engine *and* repaint the readout strip + visualizer in the same propagation. The joy-map shows the zoom window (grid subdivides 4×4→32×32); a ~5s vanishing trail fades behind the dot; noise rings breathe at the current cap. Slow deliberate moves auto-zoom-in (momentum); zoom-to-zero = freeze input (a detent, no separate toggle). **Follow mode** (double-click the dot) releases the pointer for hands-free drift.
|
||||
|
||||
**Shape via RL (the default learning mode).** Hear something good → **👍**: `train()` on accumulated examples + decay noise (settle). Hear something wrong → **👎**: `moveWeights(noiseCap, spread, pinMask)` + grow noise (perturb). Feedback is **zoom-aware**: zoomed-in 👎 nudges, zoomed-out shakes. Went too far → **↺ undo** pops the snapshot DAG and flashes. Want chaos → long-press 👎 = full re-roll. Every destructive op auto-snapshots first. This is the resting state; the **fearless loop** (RL + consequence-free undo + A/B preview) is the path of least resistance.
|
||||
|
||||
**Shape via examples (IML, secondary).** Open Shape drawer. The "Add example" two-step toggle: 1st press freezes inference so you dial the target (on the heatmap strip or sliders), 2nd press stores `(effectiveInput → rawTargetOutputs)`. **Targets are raw model-space values, not post-pipeline outputs** (the §2.6 research-validity fix). Examples can be pinned (always-included, FIFO-exempt = region-pinning approach A).
|
||||
|
||||
**Train.** Implicit in 👍, or explicit in Shape; runs in a single pooled disposable Web Worker holding its own WASM instance, round-trips weights, and streams **real loss history** (new `nisps_ml_loss_history` C API) to a live sparkline.
|
||||
|
||||
**Evaluate.** Ambient weight-health glow at the screen edge answers "healthy?" at a glance; the Health drawer shows loss-history, gradient-flow bars, layer-stats, and the input-space heatmap (throttled to weight-change ~200ms, not per-frame). Mostly, evaluation *is listening* — diagnostics are there for when the ears raise a question.
|
||||
|
||||
**Compare.** A/B: tag the current state as **A** (weights + noise + zoom), keep exploring as **B**, toggle A↔B at the same input position, **press-and-hold to preview A** (preview-pedal), then Accept B or Revert to A. **Freeze Output** locks the audible result while inference keeps running — watch the readout shift to preview what *would* change before committing (distinct from Freeze Input).
|
||||
|
||||
**Protect.** Set a few control points to `fixed` (excluded from `moveWeights`, target held) and 👎 explores the rest; long-press the joy-map to region-pin. This zoom↔train↔pin refinement workflow is the whole reason the instrument is more than a randomizer.
|
||||
|
||||
### 2.7 Undo / history
|
||||
|
||||
Undo, snapshots, A/B, and the trail are **one model** (P4's snapshot DAG, §3 architecture): a content-addressed DAG of `NetworkState` nodes with a `current` pointer. Undo = `current ← parent`; destructive ops = append child + advance; A/B = pin/swap pointers; trail tap-to-return = jump to a node. The Verdict cluster's ↺ undo is the single-step face (20-step practical depth, ring-bounded, default 20/max 50); long-press = the tagged-node history list; the Shape-Full DAG browser renders the tree for branchable exploration. RL-undo against a stub is the **first build wave** (de-risks the whole history system; matches Dimi's own instinct).
|
||||
|
||||
### 2.8 Keyboard accelerators (layer, not primary door)
|
||||
|
||||
Verb keys (`1`–`5` for drawers, `\` for Full, `space`/`↑`/`↓` for verdict, `z` for undo) are an **accelerator layer on top of a fully pointer-operable Console** — never the only door. This keeps the performer/newcomer path pointer-discoverable while giving the researcher fast iteration.
|
||||
|
||||
---
|
||||
|
||||
## 3. Architecture
|
||||
|
||||
### 3.1 The reactive spine (the live-feedback guarantee, made structural)
|
||||
|
||||
The single highest-value structural fix. The brief's #1 recurring frustration — MLP output silently stops updating the UI when the joystick moves — is a reactivity-discipline failure, not a UI one. We make it impossible to regress by collapsing input → ML → output into **one pure `createMemo` chain that every consumer reads**, with the impure side-effects in a single `createEffect`:
|
||||
|
||||
```
|
||||
inputRaw // Float32Array signal, {equals:false} (the ONLY input entry point)
|
||||
→ processedInput // createMemo: deadzone→zoom→curve→smoothing→momentum (pure, golden-tested)
|
||||
→ mlOutput // createMemo: WasmIML.infer(processedInput) into a REUSED buffer (pure read)
|
||||
→ routedOutput // createMemo: voice-space + output pipeline (global curve→smoothing→slew→freeze)
|
||||
```
|
||||
|
||||
Then **one** `createEffect` reads `routedOutput` and performs the side-effects: `postMessage` to the worklet (with a *dedicated re-filled send-buffer*, because transferables neuter the source) + engine post. Every UI consumer — readout strip, visualizer, trail — **reads `mlOutput`/`routedOutput` directly**. There is no separate "push output to the UI" path that can rot.
|
||||
|
||||
This is the union the feasibility judge mandated: **P3's pull-based memo chain** (no rot-able push path) corrected for **memo purity** (no `postMessage` inside a memo — that lives in the effect), with **P2's rigor**: signal/store split by update cadence, `HeapVec` re-derive-on-access, pointer-event coalescing to display cadence via `batch()`+microtask (reactive, not a rAF poll, but rate-limited). **rAF touches only canvas drawing**, never inference. We **reject P1's `createComputed` approach** (wrong primitive; the feasibility judge called it a load-bearing foot-gun).
|
||||
|
||||
**E2E invariant, asserted on every mode in CI:** `__nisps.setInputs([x,y]); expect(getOutputs()).toChange() && expect(getEngineParams()).toChange()` in the same tick.
|
||||
|
||||
### 3.2 Stores — module singletons, split by update cadence
|
||||
|
||||
The split by *update cadence* is the correctness lever (P2):
|
||||
|
||||
| Store | Kind | Why |
|
||||
|---|---|---|
|
||||
| `input-store` | `createStore` (config) + `Float32Array` signal `{equals:false}` (raw axes) | per-frame hot path → never a deep proxy |
|
||||
| `ml-store` | `createStore` (status/arch/dataset) + `outputs: Float32Array` `{equals:false}` + `weightsRevision: number` | outputs change every frame; revision is the cheap "weights changed" tripwire; owns `WasmIML` + training-worker lifecycle |
|
||||
| `control-store` | `createStore` (axes) + **per-param `createMemo`** | compound-axis → param fanout is `createMemo(() => axisTable(axis) + offset)`, **replacing the `control-routing.ts` `JSON.stringify`-inside-untracked-effect anti-pattern** |
|
||||
| `routing-store` | `createStore` | the Emitters×Targets matrix; one **control-point status** per target: `off / fixed / live` (collapses mute/pin/expose into one toggle/row) |
|
||||
| `output-store` | `createStore` | global curve→smoothing→slew→freeze gate; uses the reuse buffer (no per-frame alloc) |
|
||||
| `history-store` | `createStore` | the snapshot DAG (undo / A-B / trail / snapshots) |
|
||||
| `session-store` | `createStore` | presets + persistence (versioned schema, base64 weights), URL params, mode switch (restore-or-fresh) |
|
||||
| `bus.ts` | kept as-is | typed synchronous pub/sub for genuinely cross-cutting events (`mode.switched`, `pin.changed`, `snap.push`) — **not** hot-path data |
|
||||
|
||||
### 3.3 ML / audio hot paths
|
||||
|
||||
- **Inference (main thread):** the `mlOutput` memo calls `WasmIML.infer` into a **reused output buffer** (fixing today's per-frame `Float32Array` alloc). All WASM memory access goes through **one** place — a `HeapVec` wrapper that **re-derives the `HEAPF32` view on every access** (closes the stale-view silent-corruption bug class by construction; strictly better than today's cached-view-plus-manual-`rebind()`). `WasmIML` exposes only typed methods; no raw heap leaks out.
|
||||
- **Audio (worklet):** the **two-WASM architecture is kept wholesale** — main-thread ML + AudioWorklet engine loading the *same* `nisps.wasm` bytes via `WebAssembly.compile` with hand-rolled auto-discovered imports (no Emscripten glue in `AudioWorkletGlobalScope`). **Worklet imported via `?worker&url`** (the recent, correct fix — plain `new URL(...,import.meta.url)` ships raw `.ts`). The send-effect owns a dedicated re-filled transferable buffer.
|
||||
- **Training:** one pooled disposable Worker with its own WASM instance, torn down on unmount/mode-switch.
|
||||
|
||||
### 3.4 Decomposing the god-hook
|
||||
|
||||
`mode-runtime.ts` (663 lines) → small named composables, each independently testable and probe-mockable:
|
||||
|
||||
```
|
||||
src/runtime/
|
||||
control-graph.ts # the memo chain + the single send-effect (the spine, ~one tested file)
|
||||
use-input-adapters.ts# pointer/joystick/gamepad/mic → input-store.setRaw
|
||||
use-audio-lifecycle.ts # engine-host start/stop/teardown on mount/mode-switch
|
||||
use-snapshots.ts # DAG ops
|
||||
use-heatmap-sampler.ts # input-space heatmap, throttled to weightsRevision
|
||||
use-trail.ts
|
||||
use-auto-explore.ts
|
||||
```
|
||||
|
||||
### 3.5 Schema-driven mode system — one `<GenericMode>`
|
||||
|
||||
The 8 cloned `*Mode.tsx` + `C15Mode.tsx` collapse to **one `<GenericMode schema={schema} />`** (the orthogonality principle reaching the UI; universal across all four proposals and the baseline the judges mandate). It reads the generated `ModeSchema`:
|
||||
|
||||
- `ui.primary_input` → mounts the input adapter (`xy_pad`/`joystick`/`sliders`/`audio_in`/`midi_in`/`none`).
|
||||
- `ui.show_synth_visualizer` → `SynthVisualizer` vs `VisualEngine` vs `SequencerLane`.
|
||||
- `ui.show_voice_space_selector` → renders the selector or not.
|
||||
- `params[].{group,curve,label,min,max}` → drives the readout strip, the Route per-param rows, and a cross-mode `<ParamGroupCurve>` primitive (per-group curve editing everywhere, per Dimi's explicit ask).
|
||||
- `capability_class` (new field) → mode taxonomy (§2.10).
|
||||
- `params[].tier` + diagnostics `tier` (new) → disclosure depth.
|
||||
|
||||
**Heterogeneous I/O is first-class via schema flags, not code branches:** `output_kind: event` (breakor/elysiamorf) → `SequencerLane` + MIDI/event backend, silent audio; `output_kind: controller` (`kRouteOutputsToEngine=false`) → raw output meters; `primary_input: audio_in` → mic analyser adapter; no `voice_spaces` → no selector.
|
||||
|
||||
### 3.6 Backends as output adapters
|
||||
|
||||
```ts
|
||||
interface OutputBackend { send(engineParams: Float32Array): void; start(): Promise<void>; teardown(): void; }
|
||||
```
|
||||
|
||||
Implementations: `WebAudioBackend` (worklet), `WebMidiBackend` (7-bit CC out), `OscBridgeBackend` (WS → salvaged Deno bridge), `SerialCvBackend` (stub behind a locked contract, "later"). The mode picks one+ via schema; the send-effect just calls `backend.send()`. Adding OSC never touches the loop (orthogonality + Dimi's v1-backends ask).
|
||||
|
||||
### 3.7 Adding a new mode/engine (the extensibility story)
|
||||
|
||||
1. Write the C++ engine in `nisps/engines/` satisfying the `AudioEngine` concept (+ voice-space lambdas).
|
||||
2. Add `schemas/modes/<mode>.json` (params, groups, curves, `capability_class`, `ui`, voice-space names, tiers).
|
||||
3. `bun run codegen/generate.ts` → C++ `constexpr` header + TS types (golden test enforces idempotent byte-identical regen).
|
||||
4. Register the engine id in the worklet's engine switch + `modes/index.ts`.
|
||||
5. **Zero new UI.** `<GenericMode>` renders it; it appears in the switcher under its class.
|
||||
|
||||
### 3.8 Persistence / session model
|
||||
|
||||
Replace today's bespoke per-store `Partial<>` merges, `Infinity↔null` slew encoding, and slow `Array.from()` weight JSON with **one `persist<T>(store, version, migrate)` helper**: versioned schema + base64 weight blobs, debounced 200ms. URL params (`tame`/`spread`/`preset`/`debug`) kept. Session presets are **composed layers** (control / synth / weights / mode independently saveable, soft-bundled on save) — see §8-I. Restore-or-fresh on mode switch.
|
||||
|
||||
### 3.9 The arch contract (v1, honest)
|
||||
|
||||
The schema declares `input_size` but WASM is hardcoded `MLP<2,10,14,18,126>`; today `WasmIML.init` warns-and-ignores caller sizes and `setInput` loops past the real arch, writing phantom channels 2–5 into OOB heap. **v1 resolution (P1's honest contract, mandated by the mission-fit judge):** codegen **fails the build** on schema/arch mismatch; `<GenericMode>` clamps `setInput` to the real arch; multi-input modes are explicitly out of v1 (see §8-D). The runtime-shaped MLP (P4's idea, Dimi's stated intent) is deferred behind a passing parity check — **never bundled into the UI rewrite**, because it rewrites the parity-tested (2.4e-7) zero-heap core that the crystallization mission and browser≡firmware contract both depend on.
|
||||
|
||||
---
|
||||
|
||||
## 4. Feature treatment table
|
||||
|
||||
Legend: **Keep** (port forward) · **Redesign** (changed home/shape) · **Drop/Defer** (out of v1). Every item from the brief's §6 inventory.
|
||||
|
||||
| Feature | Verdict | Rationale |
|
||||
|---|---|---|
|
||||
| Explore via abstract input channels | Keep | The loop; unify behind `input-store.setRaw` + adapters |
|
||||
| RL thumbs (default) | Keep | Resting state; Verdict cluster; zoom-aware; auto-snapshot |
|
||||
| Examples/IML two-step toggle | Keep + fix | Secondary path in Shape; **fix label source to raw outputs**; inline hint |
|
||||
| Randomise | Redesign | Demote to long-press-👎 (declutter; "thumbs-down really hard") |
|
||||
| Output pin mask | Keep | Folds into control-point `fixed` |
|
||||
| Async training | Keep + fix | Pooled worker; **real loss history** via new C API |
|
||||
| Auto-snapshot before destructive ops | Keep | Centralized as DAG append |
|
||||
| Canvas-first immersive layout | Keep | The Stage; the thesis |
|
||||
| Right dock + mutually-exclusive drawers | Keep/build | Resolves dock-vs-bottom-sheet → dock |
|
||||
| Mode shell + switcher | Redesign | 8–9 clones → one `<GenericMode>`; switcher grouped by capability class |
|
||||
| Auto-dissolving chrome | Keep | 4s timeout; never fade verdict on first session; dock edge persists |
|
||||
| Floating status pill | Redesign | Becomes ambient health glow + Feel mini-axis ghost; layout-derived anchors |
|
||||
| Help overlay | Keep | Behind `?` / Help dock |
|
||||
| Visual language (orange/JetBrains/glass) | Keep + fix | **Fix the cyan drift**; lint a CSS-var allowlist |
|
||||
| Boldness/Memory/Precision axes | Keep | Feel drawer; per-param `createMemo` fanout |
|
||||
| Stability↔Fluidity (stretch axis) | Defer | Clutter; behind Feel-Expand later if demanded |
|
||||
| Trim-pot offsets + double-tap re-link | Keep | Default (§8-F); add "offset active" dot for visibility |
|
||||
| Six control presets | Keep | Cheap axis-triplets; encode playing-postures |
|
||||
| Per-group curve editing | Keep | Cross-mode `<ParamGroupCurve>` primitive (Dimi explicit) |
|
||||
| Heatmap readout strip | Keep | Top of stage |
|
||||
| Drag-to-scrub cell (3px, pointer-capture) | Keep | Load-bearing tactile detail; reproduce exactly |
|
||||
| Click → override popup (min/max/curve/freeze, 300ms grace) | Keep | Reproduce exactly |
|
||||
| Frozen-cell dimming; hidden in synth mode | Keep | — |
|
||||
| Per-param overrides (unified) | Redesign | Control-point `off/fixed/live`, one toggle/row |
|
||||
| deadzone→zoom→curve→smoothing→momentum | Keep | Pure-fn memo; golden-tested |
|
||||
| Zoom (log, anchor modes, zoom-at-zero=freeze) | Keep | Core navigation; freeze-detent elegant |
|
||||
| Momentum-as-zoom | Keep | — |
|
||||
| Joy-map minimap + adaptive grid | Keep | Navigation legibility |
|
||||
| Vanishing trail + tap-to-return | Keep | DAG node jump |
|
||||
| Noise ring(s) | Keep | — |
|
||||
| Region pinning (example-pinning A) | Keep | — |
|
||||
| Parameter pinning (pin mask ≠ mute) | Keep | Control-point `fixed` |
|
||||
| Snapshot stack | Redesign | Unified into the snapshot DAG |
|
||||
| RL undo (20-step) | Keep | Verdict cluster; **first build wave** |
|
||||
| A/B compare + hold-to-preview | Keep | DAG pointer ops; preview-pedal |
|
||||
| `spread` master regime | Redesign | **Opt-in lab toggle** in Health (firmware-init default); effect surfaced via Boldness axis (§8-E) |
|
||||
| noise floor/cap/growth/decay, weight decay, distribution (incl. cauchy), layer-aware | Keep | Health/Feel, tier-2 |
|
||||
| Zoom-aware feedback scaling | Keep | Default on |
|
||||
| Pressure / hold-duration feedback | Keep | Touch-correct primitive |
|
||||
| Auto-Explore + Follow mode | Keep | Shape drawer; emerald toggle + progress ring |
|
||||
| global curve→smoothing→slew→freeze gate | Keep + fix | Reuse buffer (no per-frame alloc) |
|
||||
| Freeze output (preview-before-commit) | Keep | Feel/Route |
|
||||
| Tame | Redesign | Promote URL → Health panel; labelled "output limiter" (safety) |
|
||||
| Input-space heatmap | Keep | Health; throttled to weight-change ~200ms |
|
||||
| Weight-health indicator | Keep | Ambient screen-edge glow + Health detail |
|
||||
| Gradient-flow bars | Keep | Health, tier-2 |
|
||||
| Per-layer stats | Keep | Health-Full |
|
||||
| Loss / loss-history plot | Keep + fix | **Real history** via new C API |
|
||||
| Progressive disclosure | Redesign | Drawer depth (Peek/Expand/Full) backed by schema `tier`; no global Advanced toggle |
|
||||
| Three-tier responsive surface | Keep | Is the drawer-depth mechanism |
|
||||
| Control-point status (off/fixed/live) | Keep | Unified per-target model |
|
||||
| Exposure-spectrum staged-unlock + swap + opt-in | Keep | Inside Route-Full for dense modes |
|
||||
| Emitters × Targets matrix (MLP = emitter bank) | Keep (scope-gate) | Route-Full; built to host modular mode (§8-G) |
|
||||
| Synth modes (incl. modular 3-osc/4op/additive) | Keep / scope-gate | Modular mega-mode v1.5 (§8-G), matrix built to host it |
|
||||
| Controller-only | Keep | `output_kind: controller` |
|
||||
| Sequencer (breakor/elysiamorf, ShapeSeq freeze/delta) | Keep | `SequencerLane`; freeze/delta is a killer feature |
|
||||
| Visual | Keep | `VisualEngine`; genesis use case |
|
||||
| C15 | Defer | Placeholder, no schema/engine — don't scaffold UI for vapor (§8-G) |
|
||||
| Mic modes (XIASRI / sound_analysis_midi) | Defer (honest) | Out of v1 until runtime-shaped MLP (§8-D); v1 shows "single-input in browser" badge |
|
||||
| Web Audio backend | Keep | — |
|
||||
| WebMIDI out | Keep | v1 adapter |
|
||||
| OSC via bridge | Keep | v1 adapter; salvage Deno bridge |
|
||||
| CV/gate serial | Defer | Stub behind locked contract |
|
||||
| Session presets | Keep | Composed layers (§8-I) |
|
||||
| bypass vs mute per param | Keep | Folds into control-point semantics |
|
||||
| restore-or-fresh on mode switch | Keep | — |
|
||||
| Debounced localStorage + URL params | Keep + fix | Versioned schema + base64 weights |
|
||||
| Debug probe (`window.__nisps`) | Keep + fix | **Gate behind `?debug=1`** (honor the doc); on by default in dev builds |
|
||||
| Two-WASM arch + custom worklet loader | Keep | Correct, load-bearing |
|
||||
| Pure-fn pipelines + curve catalog | Keep + fix | **Unify the two diverging curve enums** (one source, golden-tested vs `math.hpp`) |
|
||||
| Typed signal bus | Keep | Cross-cutting events only |
|
||||
| Live-feedback guarantee | Keep (structural centerpiece) | One reactive path; e2e-asserted every mode |
|
||||
|
||||
---
|
||||
|
||||
## 5. What 2.0 gains vs today, and what we deliberately drop
|
||||
|
||||
**Gains**
|
||||
- **Live feedback can't silently regress.** UI, audio, and visuals derive from the *same* memo chain; a desync is a failing e2e test, not a recurring prod bug. Directly kills Dimi's #1 frustration by construction.
|
||||
- **8–9 mode files → 1 `<GenericMode>`; 663-line god-hook → composable hooks.** Adding an engine is schema + C++ + codegen, zero UI. Orthogonality is enforced, not aspirational.
|
||||
- **One history model** (snapshot DAG) instead of three (snapshots/undo/A-B) — less code, branchable exploration.
|
||||
- **One disclosure mechanism** (drawer depth backed by schema `tier`) instead of five competing proposals — one learnable gesture.
|
||||
- **Performer and researcher are separated** (Feel vs Health drawers) — no gradient bars in the player's face, no hunting for diagnostics.
|
||||
- **Honest, correct core contract:** no phantom inputs, real loss history, training on raw outputs (a research-validity fix), heap safety by construction, no per-frame allocation on the audio-param path.
|
||||
- **Backends are adapters** — MIDI/OSC/CV slot in without touching the loop.
|
||||
- **Faithful visual language** restored (orange, not cyan).
|
||||
|
||||
**Deliberately dropped from v1**
|
||||
- **Multi-input mic modes** (XIASRI / sound_analysis_midi as ML inputs) — until the runtime-shaped MLP lands behind a passing parity check. v1 ships 2-input-correct, not 10-input-broken.
|
||||
- **Runtime-shaped MLP itself** — deferred; it touches the sacred parity-tested core and must never ride the UI-rewrite wave.
|
||||
- **C15** — pure placeholder; no UI scaffolding for vapor.
|
||||
- **Stability↔Fluidity 4th axis**, **bundled all-in-one session presets**, **CV/gate serial**, **modular mega-mode** — deferred as parity clutter, not v1 mission work (modular is v1.5; the matrix is built to host it).
|
||||
- **Backwards compatibility** — none; clean rewrite, per operator preference.
|
||||
|
||||
---
|
||||
|
||||
## 6. Build roadmap
|
||||
|
||||
Each phase is shippable and testable (Playwright + `__nisps` probe). Parity checkpoints called out. Effort is rough (ideal-engineer-days for one focused agent/operator pair).
|
||||
|
||||
**Phase 0 — Scaffold & contract (≈2–3d).** Vite + SolidJS + TS skeleton; theme tokens (orange, lint allowlist); codegen wired with the **build-fails-on-arch-mismatch** rule; `ModeSchema` extended with `capability_class` + `tier`; `bus.ts` ported; `persist<T>` helper. *Test:* codegen golden test; typecheck green.
|
||||
|
||||
**Phase 1 — The reactive spine against a stub (≈3–4d, de-risk wave).** `control-graph.ts` memo chain + single send-effect with a **stubbed `mlOutput`**; `input-store` + `output-store` (pure-fn pipelines ported, golden-tested; curve enums unified). *Test:* e2e asserts `setInputs → getOutputs` changes; no per-frame alloc (heap-snapshot fuzz).
|
||||
|
||||
**Phase 2 — Console shell + Verdict cluster + RL-undo against the stub (≈4–5d, de-risk wave).** Dock + three-depth drawers; Verdict cluster; snapshot DAG + undo + A/B all against the stub. **User-test the three drawer depths with Dimi before wiring engines** (P3's mandate, Dimi's instinct). *Test:* e2e drives undo/A-B via probe.
|
||||
|
||||
**Phase 3 — WASM ML bridge (≈4–5d).** `HeapVec` re-derive-on-access; `WasmIML` typed wrapper; pooled training worker; **`nisps_ml_loss_history` C API + real loss plumbing**; train on raw outputs. **Parity checkpoint:** main-thread inference matches native within 1e-5. *Test:* probe `train`/`infer`/`getLayerStats`; loss plot draws a real curve.
|
||||
|
||||
**Phase 4 — Audio worklet + first real mode (≈4–5d).** Two-WASM worklet (`?worker&url`, hand-rolled imports, dedicated send-buffer); `WebAudioBackend`; `<GenericMode>` rendering PAFSynth (synth class). **Parity checkpoint:** `parity-check.sh` green; browser audio equivalent to firmware for PAFSynth. *Test:* per-mode live-feedback e2e (audio params change on input move).
|
||||
|
||||
**Phase 5 — Schema-driven mode coverage (≈4–6d).** All in-scope engines via `<GenericMode>` (synth + sequencer + controller + visual classes; `SequencerLane`, `VisualEngine`, controller meters); capability-class switcher; heterogeneous I/O flags. *Test:* every mode passes the live-feedback e2e in CI; parity per mode.
|
||||
|
||||
**Phase 6 — Control surface + navigation + pinning (≈4–5d).** Compound axes (memo fanout) + presets + trim-pot; full input pipeline (zoom/anchor/momentum); joy-map + trail + noise rings; region + param pinning (control-point `fixed`); interactive heatmap (3px/300ms craft). *Test:* probe-driven axis/pin/zoom; tactile-constant e2e.
|
||||
|
||||
**Phase 7 — Diagnostics + Route matrix + exposure (≈4–5d).** Health drawer (weight-health glow, gradient-flow, layer-stats, input-space heatmap throttled, `spread`/`tame` lab); Route Full matrix + control-point rows + staged-unlock; per-group curves. *Test:* diagnostics render from real stats; probe `getLayerStats`.
|
||||
|
||||
**Phase 8 — Backends + persistence + polish (≈3–4d).** `WebMidiBackend` + `OscBridgeBackend`; versioned persistence + base64 weights; session preset layers; auto-dissolve chrome; keyboard accelerators; Help. *Test:* full `run-all-tests.sh` (cmake + ctest + WASM + parity + lint + Playwright) green = **Verification chokepoint E**.
|
||||
|
||||
**Post-v1 (gated):** runtime-shaped MLP behind parity → unlocks multi-input mic modes; modular mega-mode + Emitters×Targets population; C15; Stability axis.
|
||||
|
||||
---
|
||||
|
||||
## 7. Risks & mitigations
|
||||
|
||||
| Risk | Mitigation |
|
||||
|---|---|
|
||||
| Live-feedback regression recurs | One reactive path + per-mode e2e invariant in CI; no leaked write path (weights only mutate via `ml-store` actions that bump `weightsRevision`) |
|
||||
| Memo purity violated (side-effect creeps into a memo) | Single documented send-`createEffect`; lint/review rule; memos must be pure |
|
||||
| High pointer-event rate over-fires inference | Coalesce to display cadence via `batch()`+microtask; still reactive, rate-limited |
|
||||
| Transferable buffer neutering | Dedicated re-filled send-buffer owned by the effect, separate from signal buffers |
|
||||
| Stale `HEAPF32` view corruption | `HeapVec` re-derives the view on every access; sole WASM-memory access point; fuzz-tested |
|
||||
| Drawer-depth model novel/undiscoverable | Prototype Console against `/dev/primitives`; user-test depths with Dimi (Phase 2); clear "⤢" affordances |
|
||||
| Auto-dissolving chrome hides the loop | Never fade Verdict on first session; dock edge always visible; `?` help |
|
||||
| Canvas-as-input vs clicking trail/pins | Hit-test priority: trail points + pins capture before joystick |
|
||||
| 3px/300ms constants fragile cross-device | Port exact constants; e2e-test them |
|
||||
| Snapshot DAG memory (large weight blobs) | Ring-bound (20/50); base64-compress; GC by tag priority (pinned/A nodes survive) |
|
||||
| Parity drift on any core touch | Every core-touching phase gated on `parity-check.sh` green before UI lands |
|
||||
| Probe ships to prod | Gate behind `?debug=1`; on by default in dev builds only |
|
||||
|
||||
---
|
||||
|
||||
## 8. Open questions for Dimi (decide BEFORE building)
|
||||
|
||||
Each is a real fork. I give options + my recommendation; I need your call.
|
||||
|
||||
**A. Canonical tree / identity.** Is 2.0 the laptop `~/src/manifold` (`manifold::`, fresh repo — possibly already partly executed) or this VPS `MEMLNaut-NISPS` tree (`nisps::`)? **Recommendation:** target this VPS tree + `nisps::` (it's the live tree the recon ran against); the architecture is identity-agnostic and a rename is trivial. **But this blocks namespace/repo/codegen identity — I need your answer first.**
|
||||
|
||||
**B. Mode taxonomy.** Four capability classes (Synth / Controller / Sequencer / Visual) vs the 8 engines + C15. **Recommendation (adopted):** add a `capability_class` schema field; the 8 engines are instances under Synth/Sequencer; Controller = `kRouteOutputsToEngine=false`; Visual = an output target. Switcher groups by class. Confirm this hierarchy vs an engine-flat switcher.
|
||||
|
||||
**C. Progressive-disclosure model.** **Recommendation (adopted):** per-section drawer depth (Peek/Expand/Full), backed by schema-declared per-param/diagnostic `tier`; no global Advanced toggle. This unifies all five corpus proposals. Confirm it's canonical.
|
||||
|
||||
**D. Runtime-shaped MLP vs fixed arch (the one I most want you to rule on).** **Recommendation (strong):** v1 ships the **honest fixed-2-input contract** (build fails on schema/arch mismatch); the runtime-shaped `Mlp` (your stated intent) is **deferred to post-v1 behind a passing parity check**, never bundled into the UI rewrite. Consequence: XIASRI / sound_analysis_midi multi-input modes are **firmware-only in v1** and show a "single-input in browser" badge. This is a materially different product surface than "all modes work in browser" — confirm you accept the v1 contraction. (The mission-fit and feasibility judges both flagged bundling the core rewrite into v1 as the project's biggest scoping hazard.)
|
||||
|
||||
**E. `spread` prominence.** **Recommendation (adopted):** opt-in **lab toggle** in Health (firmware-init is the browser default, per the crystallization philosophy), with its *effect* surfaced through the **Boldness** axis so the primary affordance is musical. Confirm `spread` doesn't deserve its own primary slider regardless.
|
||||
|
||||
**F. Trim-pot vs detach for axis overrides.** **Recommendation (adopted):** trim-pot offset stays default ("most musical"), with an "offset active" dot for visibility. Confirm, or whether detach should be a per-axis/per-user choice.
|
||||
|
||||
**G. v1 scope of heavy features.** **Recommendation:** defer C15, the modular mega-mode (3-osc/4op/additive + shared mod pool + Emitters×Targets), and firmware-equivalent mic analysis to post-v1; v1 ships the pragmatic 4-feature analyser (read-only, not as ML input — see D) and keeps the matrix *built to host* the modular mode. Sequencer ShapeSeq freeze/delta **is** in v1. Confirm the line.
|
||||
|
||||
**H. Desktop-first confirmation.** **Recommendation:** treat "desktop-first, touch-correct core" as decided, with touch-correctness of joystick/zoom/trail as a v1 acceptance criterion. Confirm the chat decision still holds over ALIGNMENT Q4's "defer until user data."
|
||||
|
||||
**I. Session / bundled presets.** **Recommendation (adopted):** keep control-presets and synth-presets orthogonal; session presets are **composed layers** (weights / control / synth / mode independently saveable, soft-bundled on save), not a hard bundle. Confirm.
|
||||
|
||||
---
|
||||
|
||||
## Appendix: provenance
|
||||
|
||||
**P1 — research-instrument-minimalist ("The Listening Loop").** Mission-fit judge's top pick (9): its "capability budget, not screen budget" framing and its **honest fixed-2-input §7-D resolution** (build fails on mismatch; defer multi-input) are the correct posture for a research instrument and the only arch story that respects the crystallization + parity contracts — both grafted into this plan as load-bearing. UX-coherence ranked it second (7.5) — strong restraint, but it bets fast-path ergonomics on keyboard muscle-memory (least discoverable). Feasibility ranked it last (5) with a near-fatal flaw: it builds its "structurally unbreakable" claim on `createComputed`, the wrong reactive primitive — **rejected** in favor of P2/P3's memo chain. Net contribution: the minimalist philosophy + the honest arch fork.
|
||||
|
||||
**P2 — realtime-frontend-architect.** Feasibility judge's clear winner (9): store-split-by-update-cadence, the `HeapVec` re-derive-on-access wrapper, transferable-buffer and pointer-coalescing rigor, and the OutputBackend adapter are the ship-grade backbone — all adopted. Mission-fit (8) credited its **live-feedback-as-one-owned-effect** as the single highest-value structural fix (Dimi's #1 frustration) plus the **train-on-raw-not-processed-outputs** research-validity fix. UX-coherence ranked it last (5) for its cardinal sin: it kept **both** a floating control bar and the dock, reproducing the exact two-homes conflict the brief said to resolve — **rejected** (we use the dock). Net contribution: the architecture spine + two correctness fixes + backends.
|
||||
|
||||
**P3 — interaction-ux-designer ("The Console").** UX-coherence judge's winner (9): the dock-with-three-depth-states as the single disclosure gesture, the **Feel-vs-Health drawer split** (performer vs researcher mental modes), the floating-bar resolution (auto-open Peek + mini-axis ghost), the interaction-hazard handling (hit-test priority, randomize-as-long-press, never-fade-on-first-session), and the de-risk sequence (Console + RL-undo against a stub, user-tested first) — all adopted as the interaction spine. Mission-fit (8) credited the fearless-RL psychology. Feasibility (8) flagged one easily-corrected impurity: it described `postMessage` inside a memo — **corrected** here (side-effects live in one `createEffect`). Its gap: it left §7-D unresolved — **filled** with P1's honest contract. Net contribution: the entire UX/IA spine.
|
||||
|
||||
**P4 — systems-information-architect ("altitude").** Lowest mission-fit (6) and mid feasibility (6)/UX (6.5) — not for bad ideas but for one **fatal v1-scoping flaw**: it committed the runtime-shaped MLP as a wave-1 deliverable, coupling the frontend rewrite to a cross-language rewrite of the sacred parity-tested core. All three judges said: adopt the **concepts, defer the core change**. So we take P4's **snapshot DAG** (unifying undo/A-B/trail/history) and its **control-point `off/fixed/live` enum** (collapsing the three overlapping ML-control systems) and its **`capability_class` schema field** — while explicitly gating runtime-shaped MLP behind a passing parity check, post-v1. Net contribution: the data-model concepts (history DAG, control-point enum, capability classes), minus the scoping trap.
|
||||
225
docs/redesign/rl-feedback-design.md
Normal file
225
docs/redesign/rl-feedback-design.md
Normal file
|
|
@ -0,0 +1,225 @@
|
|||
# RL / Feedback Learning-Engine Design — Manifold
|
||||
|
||||
*Design sign-off. Dated 2026-06-27. Supersedes the AVOID-out-of-scope decision in `docs/redesign/feedback-modes-port-spec.md` §2.5/§7. Author: lead architect, synthesising a 4-candidate / 4-judge design panel against verified source.*
|
||||
|
||||
---
|
||||
|
||||
## 0. Decision summary
|
||||
|
||||
| Setting | Default | Also selectable |
|
||||
|---|---|---|
|
||||
| `FEEDBACK_MODE` | **Explore and place** (Mode 2, positive-only) | **Geometric dislike** (Mode 1) |
|
||||
| `SOLO_MODE` | **MaskGradients** (final-layer column freeze) | **ZeroLoss**, **DontCareExample** |
|
||||
|
||||
**Winner of the panel:** *Sift* (musician-UX judge topPick, skeptic topPick; 2nd on feasibility and ML). It is the only candidate that does UX design rather than only porting design, and it satisfies the "Mode 2 is NEVER a dislike" requirement at the level of the performer's mental model.
|
||||
|
||||
**Grafted onto Sift's felt-loop spine:**
|
||||
- *Minimal-Divergence* — the **only candidate whose tree-read is correct** (verified below): the feedback C API, `MLHandle.feedback`, the `0xFEEDBACC0DE` salt, `CMakeLists.txt:59` registration, and parity Stage 5 ALL already exist. Its **append-only `FeedbackAction` enum** discipline and **controller-returns-action / caller-owns-training** boundary contract are kept verbatim.
|
||||
- *Anchored-Manifold* — the **only candidate with correct provenance** (the upstream tip is `0a541cc "highlighting"`, verified; not the `e291192`/`abe93ec` the others cite). Its **one-`ReplayStore`-three-features** compression, its **bit-identical-frozen-column solo ctest**, and its **explicit TrainBatch-vs-SGD divergence flag** are adopted.
|
||||
- *Anchor-First* — **keep replay/geometric logic OUT of the MLP kernel** (only minimal backprop hooks go in), and **port the `posMemCount==0` cold-start fallback faithfully** so dislikes-before-any-like don't destabilise the net.
|
||||
|
||||
---
|
||||
|
||||
## 1. Verified ground truth (read this before building)
|
||||
|
||||
All claims below were read from source on 2026-06-27. Where the four candidates disagreed, the verified fact decides.
|
||||
|
||||
### 1.1 The scaffold already exists (Minimal-Divergence was right)
|
||||
- `nisps/wasm/bindings.cpp:95-116` — `struct MLHandle` already holds `nisps::ml::FeedbackController<DefaultMLP> feedback;` + `feedback_static_scratch{}`, constructed `feedback(seed ^ 0xFEEDBACC0DEull)`.
|
||||
- `nisps/wasm/bindings.cpp:439-535` — the 9 `nisps_ml_feedback_*` C functions (`set_mode`/`get_mode`/`exploring`/`learning_paused`/`set_focus`/`down`/`up`/`drag`/`static_output`) are **already wired**.
|
||||
- `nisps/CMakeLists.txt:58-59` — `test_mlp_rl.cpp` **and** `test_mlp_feedback.cpp` are **already registered** in `nisps_core_tests`.
|
||||
- `tests/cpp/parity_check.cpp:86` — `kVersion = 2u` already; `:198-205` — **Stage 5 (feedback) already exists** seeding `FeedbackController<ParityMLP> fb(kSeed ^ kFeedbackSalt)`.
|
||||
|
||||
> Candidates 2/3/4 proposed *adding* `MLHandle.feedback`, *registering* `test_mlp_feedback.cpp`, and *creating* Stage 5. That is phantom work — they already exist. This design **edits** the existing `on_down` cases and **extends** the existing enum/API/parity-stage; it does not re-scaffold them.
|
||||
|
||||
### 1.2 The current `feedback.hpp` (untracked, verified `nisps/ml/feedback.hpp:1-214`)
|
||||
- `enum FeedbackMode {Avoid=0, RandomiseOutputs=1, RandomiseMlp=2}` (`:47-51`); `enum FeedbackAction {None=0,…Restore=7}` (`:56-65`).
|
||||
- `on_down` Avoid case (`:103-105`) calls `mlp.move_weights(speed, spread, pin_mask)` — the **undirected-diffusion placeholder** the requirement now forbids for Mode 1.
|
||||
- `RandomiseMlp` `on_down` (`:113-119`): 1st press enters (`snapshot_` + `draw_weights`), 2nd press **cancels**. Snapshot/restore is a byte-exact `get_weights`/`set_weights` round-trip (`:186-193`).
|
||||
- The controller has its **own per-instance `Rng rng_`** (`:210`), seeded independently.
|
||||
|
||||
### 1.3 The latest upstream algorithm (Anchored-Manifold's provenance was right)
|
||||
The actual upstream tip is **`0a541cc "highlighting"`** at `/home/w1n5t0n/src/MEMLNaut-NISPS-upstream/src/memllib/examples/InterfaceRL.{hpp,cpp}`. Constants verified:
|
||||
- `InterfaceRL.hpp:293` `kGeometricPushScale = 0.5f`; `:294` `kMaxDislikeMultiplier = 16`; `:296` `kCentroidK = 4`.
|
||||
- `InterfaceRL.cpp:42-66` `_perform_dislike_action()`: nearby negative within Euclidean `0.05f` of `controlInput` → `reward = max(reward-1, -16)`; else `storeExperience(-1, controlInput, action)`; then `dislikeMultiplier_ = min(*2, 16)`.
|
||||
- `InterfaceRL.cpp:602-627` k-NN centroid: average the `kCentroidK=4` positive memories nearest `controlInput` → `meanPositiveAction`.
|
||||
- `InterfaceRL.cpp:664` proportional decay `reward += 0.0025f * max(|reward|, 1.0f)`; expired items removed; `dislikeMultiplier_` halved per expiry (`:752-760`).
|
||||
- `InterfaceRL.cpp:713` `pushStep = clamp(|avgRewardNeg|, 0.25f, 1.0f) * kGeometricPushScale`.
|
||||
- `InterfaceRL.cpp:721-735` per negative: `dir[j] = neg_action[j] - meanPositiveAction[j]`; `len = ||dir||`; `useRandom = (len <= 1e-4f)`; `effectivePushStep = pushStep / (1 + len)`; for each `j` gated by `activeDims_`: `d = useRandom ? (rand()&0xFF/127.5 - 1) : dir[j]/len`; `target[j] = clamp(neg_action[j] + d*effectivePushStep, 0, 1)`.
|
||||
- `InterfaceRL.cpp:742-743` `negLRRatio = 0.5f - 0.4f*negFraction`; train geometric targets at `lr*negLRRatio`.
|
||||
- `InterfaceRL.cpp:746` **cold-start fallback** when `posMemCount==0`: `TrainBatch(tsNegative, lr * 0.1f * avgRewardNeg, …)` — negative-LR training, no geometric push.
|
||||
|
||||
### 1.4 The deepest parity hazard (Anchored-Manifold / Anchor-First flagged it)
|
||||
Upstream `optimise()` trains with **`TrainBatch`** (shuffled, `batchSize≈8`, separate positive/geometric batches at *two distinct* learning rates). The nisps `MLP::train()` (`mlp.hpp:256-300`) is **unshuffled per-sample SGD over the dataset insertion order** (its own comment: *"TrainBatch shuffles, but we're not implementing batch yet"*), has **no reward field** and **no train-toward-arbitrary-targets** entry. Reproducing the two-batch / dynamic-`negLRRatio` dynamics on a non-batch trainer is the real `browser != firmware` line. **We accept behavioural (not bitwise) parity with firmware here, and pin `native == WASM` at 1e-5.** Recorded in `ALIGNMENT.md`.
|
||||
|
||||
### 1.5 The injection points the masking/solo mechanism needs
|
||||
- `mlp.hpp:281` — the per-sample `std::array<float,NOut> deriv{}` is the **exact place** to zero the output-error of non-soloed / don't-care dims before `backprop_` (`:291`). One branch per output dim.
|
||||
- `rl.hpp:74-99` — `move_weights_layer` already skips the final-layer weight **column + bias** when `is_final_layer && output_pin_mask[node] != 0`. Solo's column-freeze is the **bitwise inverse** of this mask. Note the existing rule (`:88-96`): the RNG is advanced even on skipped weights so the stream is pin-independent — keep that invariant.
|
||||
- `mode-runtime.ts:548-565` `thumbsDown` currently builds a pin mask and calls `moveWeights` — **no example stored**, no training. `thumbsUp` (`:523-546`) computes the `(features, labels)` and calls `addExample` + `trainOnCurrent`.
|
||||
|
||||
---
|
||||
|
||||
## 2. FEEDBACK_MODE = two operator-switchable modes
|
||||
|
||||
The selector is `exploration-store.ts` state `feedbackMode: 'explore-and-place' | 'geometric-dislike'`, surfaced in the **learning-behaviour dock panel** (§7). British spelling in all copy ("randomise", "auditioning", "centre", "behaviour").
|
||||
|
||||
### 2.1 Mode 1 — "Geometric dislike" (negative example + geometric push-away)
|
||||
|
||||
**Felt loop:** you have a region you like; thumbs-down carves the current sound *away* from the liked centroid. Directed repulsion, audibly directional — the fix for today's inaudible undirected diffusion (`findings-feedback-behaviour.md §5`).
|
||||
|
||||
**Algorithm — port `InterfaceRL.cpp` (0a541cc) into nisps core, keeping parity.** Because nisps has no async `optimise()` driver, the press-time half (`_perform_dislike_action`) and the optimise half collapse into one synchronous core call.
|
||||
|
||||
On thumbs-down at `controlInput x_neg` with heard action `a_neg`:
|
||||
1. `replay.deepen_or_store_negative(x_neg, a_neg, radius=0.05f)` — nearby negative → `reward = max(reward-1, -16.f)`; else store `reward=-1`. `dislikeMultiplier_ = min(*2, 16)`. (`InterfaceRL.cpp:42-66`)
|
||||
2. If `replay.positive_count() == 0` → **cold-start fallback**: `train_targets(x_neg → a_neg, lr * 0.1f * avgRewardNeg, …)` (`InterfaceRL.cpp:746`). Surface a one-time UI prompt *"like a few sounds first to teach the system what to move away from"* (closes the cold-start gap all four judges flagged).
|
||||
3. Else: `replay.knn_positive_centroid(x_neg, k=4, &mean)` (`:602-627`); `pushStep = clamp(|avgRewardNeg|, 0.25f, 1.0f) * 0.5f`; per active/soloed dim `j`: `dir[j]=a_neg[j]-mean[j]`, `len=||dir||`, `effectivePushStep=pushStep/(1+len)`, `d = (len<=1e-4f) ? rng_.next_float_signed() : dir[j]/len`, `target[j]=clamp(a_neg[j]+d*effectivePushStep, 0, 1)`; non-active dims keep `a_neg[j]`.
|
||||
4. `train_targets(x_neg → target, lr * negLRRatio, …)` with `negLRRatio = 0.5f - 0.4f*negFraction`.
|
||||
5. `replay.decay_negatives()` each call: `reward += 0.0025f*max(|reward|,1)`, evict `reward > -0.01f`.
|
||||
|
||||
**Determinism carve-out:** the upstream `useRandom` branch uses libc `rand()&0xFF`; we substitute `rng_.next_float_signed()` (the controller's own per-instance `nisps::Rng`). Both native and WASM run the same Rng from the same seed → 1e-5 parity holds. This branch only fires when a disliked action sits exactly on the centroid. Record as the single deliberate firmware divergence in `ALIGNMENT.md`.
|
||||
|
||||
**On thumbs-up** (Mode 1): `replay.store(+1, x, a)` so the centroid sees positives; caller also runs the existing `addExample`+`train` like today.
|
||||
|
||||
**Files touched:** edit `feedback.hpp` `on_down` Avoid case (`:103-105`) → route to the geometric path; add `replay_` member; `on_up` LikeStore → `replay.store(+1,…)`. New `nisps/ml/replay.hpp` + `nisps/ml/geo_push.hpp` (§4). `move_weights` stays reachable as a **legacy `Diffuse` sub-mode** for A/B comparison (cheap insurance, lets the A/B-compare feature contrast geometric vs diffusion — graft from Minimal-Divergence).
|
||||
|
||||
### 2.2 Mode 2 — "Explore and place" (positive-only; NEVER a dislike) — **DEFAULT**
|
||||
|
||||
**Felt loop (Sift's north star):** *"randomise, explore, oh I like that → put it in that corner; meh, randomise…"*. The performer never reasons about "away from what" — they collect sounds they like. Place-by-corner is the **highest audible-effect-per-action gesture in the system**: one drop = one heard anchor the real model then interpolates.
|
||||
|
||||
Built on the existing `RandomiseMlp` scaffold (`feedback.hpp:113-119,183-201`) but to the stricter spec. Lifecycle state machine in the reworked `FeedbackController`, `FeedbackMode::ExploreAndPlace`:
|
||||
|
||||
1. **ENTER** (verdict-down/explore): `enter_scratchpad(mlp, spread)` — snapshot the real trained net into `snapshot_` (the set-aside model; existing `get_weights` round-trip), set `learning_paused_=true` (gates `trainOnCurrent` + auto-explore, mirroring upstream `optimiseSometimes` early-return), then `draw_weights(spread)` → **random scratchpad net**.
|
||||
2. **SCRATCHPAD OPS — exactly two, position-agnostic, never trained, never a dislike:**
|
||||
- `reroll()` = `draw_weights(spread)` again ("meh, randomise…"). Re-press while exploring **re-rolls** (returns `Reroll`); a dedicated cancel gesture / mode-switch aborts. (This changes the old 2nd-press-cancels semantics — deliberate; rewrite the affected ctest.)
|
||||
- `nudge(speed, spread)` = a small `move_weights` on the scratchpad with a templated undo ring `scratch_undo_` (depth `D`; **WASM `D=4`, firmware `D=2`** via template param — Sift's free SRAM tunable). `undo_nudge()` pops. Reuses the byte-exact `get_weights`/`set_weights` round-trip (graft: no new parity surface). New actions `ScratchNudge`, `ScratchUndo`.
|
||||
- The scratchpad is **NEVER trained and NEVER reads replay** — auditioning is pure forward inference as the user sweeps the joystick.
|
||||
3. **PLACE** (user likes a sound): press → **choose a location** in input space → release. `place_begin()` freezes the current scratchpad output vector into `placed_out_` (so the heard sound is held while the user aims; emitted via the existing `static_output` bypass hook so the audition doesn't change). The TS runtime owns the "aim" UI (a JoyMap drag). On release: caller sets the scratchpad input to `chosen_input`, runs `mlp.process()`, reads outputs, calls `place_commit(chosen_input, scratch_out_at_chosen)` → appends a **positive anchor** and returns `AnchorPlaced`. The controller does NOT itself run inference (keeps it pure — Minimal-Divergence's boundary contract).
|
||||
4. **WARM-START INTERPOLATION** (explicit "Done"/finalise gesture): restore the set-aside net (`set_weights(snapshot_)`), then warm-start it to interpolate **all placed anchors**. **Decision (graft from all four risk-lists): warm-start is ADDITIVE — anchors are added to the existing dataset, NOT `clear_examples()`+refit** — so the user's prior thumbs-up likes are not clobbered. The caller loops `addExample(input, output)` over `replay.anchors()` then `train(lr, maxIter, minErr)` (caller owns training). "Warm-start" = begin from the *restored real net*, not a fresh draw, so prior structure is preserved. Returns `WarmStarted`; `learning_paused_` flips false.
|
||||
5. **ABORT** (mode-switch / cancel): restore the set-aside net, discard anchors not yet finalised.
|
||||
|
||||
**Anchor store** = the same `ReplayStore` instance, positives only (§4). **Coupling decision (Sift's flag):** unifying Mode-1 replay and Mode-2 anchors into one store means a placed anchor also feeds Mode-1's positive centroid. This is a *feature* (mixed-mode sessions), but expose an operator toggle `unifyMemories` (default on) so they can be isolated into two `ReplayStore` instances if surprising.
|
||||
|
||||
**Why Mode 2 is the default:** highest audible-effect-per-action, no negative concept to learn, and it directly realises the operator's stated felt loop. Mode 1 is the precision/sculpting tool for when you already have a liked region.
|
||||
|
||||
---
|
||||
|
||||
## 3. SOLO / arm per output
|
||||
|
||||
State: `solo_mask_ : std::array<std::uint8_t, kNOut>` on the controller (`1`=armed/soloed, `0`=frozen; all-zero / empty ⇒ none soloed ⇒ normal training). API `set_solo_mask(span)` / `clear_solo_mask()`, C API `nisps_ml_feedback_set_solo(ml, mask, n)` (mirrors the existing `set_focus` at `bindings.cpp:485`). One operator setting `SOLO_MODE {MaskGradients=0, ZeroLoss=1, DontCareExample=2}`.
|
||||
|
||||
All three flow through **one injection point** — the per-sample `deriv` array at `mlp.hpp:281`, fused per Sift's graft (zeroing the loss on non-soloed outputs *is* the don't-care mask).
|
||||
|
||||
### 3.1 MaskGradients (DEFAULT) — column-freeze
|
||||
Predictable: "only this output moves." Derive `pin_mask = bitwise-NOT(solo_mask)` over outputs. In a new `train_masked(lr, max_iter, min_err, out_mask)`: zero `deriv[j]` for non-soloed `j` **before** `backprop_`, **and** freeze hidden layers when any solo is active (skip `layer0_/1_/2_.apply_grad`; apply only `layer3_` for soloed columns). Result: non-soloed final-layer columns AND all hidden weights are **bit-identical** after training. This is the exact inverse of `rl.hpp:74-99`'s pin gating — minimal new code, reuses proven machinery.
|
||||
|
||||
### 3.2 ZeroLoss (selectable) — expressive-but-bleeds
|
||||
Zero `deriv[j]` for non-soloed `j` but **let hidden layers update**. The soloed output gets the full network's expressive capacity, but non-soloed outputs can drift via shared hidden features. One-line variant of 3.1 (drop the hidden-layer freeze).
|
||||
|
||||
### 3.3 DontCareExample (selectable) — most faithful to "store with a mask"
|
||||
Each `ReplayStore` item carries `uint8 mask[NOut]`. An example placed while output `k` is soloed stores `mask = {0…1@k…0}`; `train_masked` consumes the per-example mask so other outputs at that input are never pulled toward a stale label. Survives later training. Costs `Cap*NOut` bytes.
|
||||
|
||||
### 3.4 Honest limit (Sift's correctness graft — into product copy + ALIGNMENT.md)
|
||||
**On a shared-trunk MLP, NO realisation both fully isolates AND stays expressive.** MaskGradients is predictable but weak (only the final linear column moves; hidden capacity frozen). ZeroLoss is expressive but bleeds through shared hidden weights. State this to the operator in the dock panel ("solo freezes the rest as far as a shared network allows") rather than overselling "leaves others unchanged". MaskGradients is the safe default because its guarantee is *provable* (§6.3 bit-identical ctest).
|
||||
|
||||
**Solo is honoured in both modes:** Mode-1 geometric `target` only pushes soloed dims (unify `solo_mask_` with the `activeDims_` gate at `InterfaceRL.cpp:730` — Anchor-First's graft); Mode-2 warm-start trains anchors only on soloed dims when solo is active.
|
||||
|
||||
---
|
||||
|
||||
## 4. Core component changes (keep replay OUT of the MLP kernel — Anchor-First)
|
||||
|
||||
### New `nisps/ml/replay.hpp`
|
||||
`ReplayStore<NIn, NOut, Cap>` — fixed `std::array` ring (no heap; **WASM `Cap=64`, firmware `Cap=16-32` via template param** — Anchored-Manifold's free SRAM/centroid tradeoff). Item `{std::array<float,NIn> input; std::array<float,NOut> action; float reward; std::uint8_t mask[NOut];}`. Methods (all deterministic, per-instance `nisps::Rng`, **no libc rand**):
|
||||
- `deepen_or_store_negative(x, a, radius=0.05f)` (`InterfaceRL.cpp:42-66`)
|
||||
- `store(reward, x, a, mask)` / `positive_count()`
|
||||
- `knn_positive_centroid(x, k=4, &mean, &count)` — linear scan, fixed top-k insertion (no `std::sort`, no heap). **Deterministic tie-break by index** and **fixed accumulation order** so native==WASM (the classic float-sum parity trap).
|
||||
- `decay_negatives()` (`reward += 0.0025f*max(|reward|,1)`, evict `> -0.01f`)
|
||||
- `anchors()` / `anchor_count()` accessors for Mode-2 warm-start replay.
|
||||
|
||||
### New `nisps/ml/geo_push.hpp`
|
||||
Pure free function `compute_push_targets(replay, control_input, solo_mask, rng, &target_buf)` implementing `InterfaceRL.cpp:602-738` exactly. Writes into a caller-supplied scratch buffer (no heap). Keeps the geometric math out of both MLP and controller.
|
||||
|
||||
### `nisps/ml/mlp.hpp` — minimal backprop hooks only
|
||||
- `train_targets(span inputs, span targets, span sample_weights, span out_mask, lr)` — trains toward *computed* target vectors (Mode-1 geometric batch; targets are computed, not stored labels). Reuses `forward_`/`backprop_`/`apply_grad`.
|
||||
- `train_masked(lr, max_iter, min_err, span out_mask)` — the solo path: zeroes `deriv[j]` at `:281` for masked `j`; in MaskGradients freezes hidden `apply_grad`.
|
||||
- Factor the inner forward→loss→backprop of `train()` into a `train_pair_` helper so both reuse the proven backprop (Minimal-Divergence graft). **No replay/centroid/anchor logic in MLP.**
|
||||
|
||||
### `nisps/ml/feedback.hpp` — edits (append-only enum)
|
||||
- Add `ReplayStore<NIn,NOut> replay_`, `solo_mask_`, `scratch_undo_` ring (templated depth), `placed_out_`, `SoloMode` field.
|
||||
- Edit `on_down` Avoid case → geometric push via `replay_` + `geo_push` + `train_targets`; keep a `Diffuse` legacy sub-mode.
|
||||
- Edit `RandomiseMlp`/`ExploreAndPlace` `on_down` so re-press = `Reroll`.
|
||||
- Add `nudge`/`undo_nudge`, `place_begin`/`place_commit`, `finalise`/`abort`, `set_solo_mask`/`clear_solo_mask`, `set_solo_mode`.
|
||||
- **Append** new `FeedbackAction` values keeping existing numeric values stable: `GeometricPush=8, ScratchNudge=9, ScratchUndo=10, AnchorPlaced=11, WarmStarted=12` (Minimal-Divergence's single most parity-safe decision — never renumber the TS↔C++ contract).
|
||||
|
||||
### `nisps/wasm/bindings.cpp` — extend the existing block (`:439-535`)
|
||||
Add: `nisps_ml_feedback_set_solo(ml,mask,n)`, `_set_solo_mode(ml,mode)`, `_nudge(ml,speed,spread)`, `_undo_nudge(ml)`, `_place_begin(ml,current_out)`, `_place_commit(ml,chosen_input,scratch_out)`, `_anchor_count(ml)`, `_get_anchors(ml,buf)`, `_finalise(ml)`, `_abort(ml)`. Existing `_down/_up/_drag/_static_output` stay; `_down` now returns the new action ints. Reuse the existing scratch buffers + guard/cast style.
|
||||
|
||||
### TS
|
||||
- `playground/src/ml/types.ts` — add the new `_nisps_ml_feedback_*` decls; extend `FeedbackMode`/`FeedbackAction`/add `SoloMode` enums (numeric parity with C++).
|
||||
- `playground/src/ml/wasm-iml.ts` — thin wrappers (`feedbackNudge`, `feedbackPlaceCommit`, `feedbackFinalise`, `feedbackSetSolo`, `getAnchors`); reuse `pinMaskBuf` (`:225`); add a small anchor heap buffer.
|
||||
- `playground/src/modes/mode-runtime.ts` — route `thumbsDown` (`:548`) per active `feedbackMode`; Mode-2 wires press→aim→release to `place_begin`/`place_commit`; a "Done" button → `finalise` → loop `getAnchors`→`addExample`→`train`. Gate `trainOnCurrent` (`:516`) and auto-explore on `learning_paused()`.
|
||||
- `playground/src/stores/exploration-store.ts` — add `feedbackMode`, `soloMode`, `soloMask`, `exploring`, `anchorCount`, `unifyMemories` + selectors.
|
||||
|
||||
---
|
||||
|
||||
## 5. Phased integration plan (TS-prototype-first; each step gated on `parity-check.sh`)
|
||||
|
||||
**Audible validation is the gate, not green tests.** Prototype the arithmetic in pure TS — driven through the EXISTING `WasmIML` primitives (`drawWeights`, `moveWeights`, `addExample`, `train`, `getWeights`/`setWeights`, `setInput`/`process`/`getOutputs`, verified at `wasm-iml.ts:293-476`) — before any new C++.
|
||||
|
||||
### Phase 0 — TS prototype (no new WASM)
|
||||
A throwaway TS module mirrors `ReplayStore` + `geo_push` + `train_masked`, computed in TS and fed to the real net via existing calls. Oracles:
|
||||
- **Mode 1:** like 2-3 sounds in distinct corners; dislike a sound near one like; confirm the heard 126-vector at that input moves **audibly AWAY** from the liked timbre (not random wobble). A/B against today's `moveWeights` thumbs-down using the existing A/B-compare feature — directionality must be **perceptible** (Sift's oracle).
|
||||
- **Mode 2:** randomise → sweep-audition → "like that" → place in a corner → randomise → place opposite corner → Done → sweep the JoyMap and confirm a **smooth morph between the two placed timbres**, and that re-randomising no longer destroys them.
|
||||
- **Solo:** solo output 0, train, confirm via the heatmap / weight-health views that only that param's mapping changed (MaskGradients: bit-stable elsewhere).
|
||||
- Use `window.__nisps` (`setInputs`/`getOutputs`/`train`/`thumbsDown`/`getLayerStats`) for scripted Playwright checks.
|
||||
|
||||
**Gate:** felt loops audibly correct → freeze `kCentroidK`, `pushStep`, warm-start iteration count, undo depth. Only then crystallise.
|
||||
|
||||
### Phase 1 — C++ core
|
||||
Add `replay.hpp`, `geo_push.hpp`, the `mlp.hpp` hooks, the `feedback.hpp` edits. Build `nisps_core_tests` (`-Wall -Wextra -Werror`). **Gate:** new ctest cases green (§6.1); `scripts/parity-check.sh` still 1e-5 (existing Stage 5 unchanged at this point).
|
||||
|
||||
### Phase 2 — C API
|
||||
Add the new `nisps_ml_feedback_*` exports. Rebuild WASM (`scripts/build-wasm.sh`). **Gate:** `parity-check.sh` green after Stage-5 extension (§6.2); `kVersion` bump.
|
||||
|
||||
### Phase 3 — TS FFI
|
||||
`types.ts` decls + enums, `wasm-iml.ts` wrappers. **Gate:** `bun run typecheck`; a TS↔WASM smoke test reproducing a Phase-0 loop through the real FFI matches the TS prototype's outputs; `parity-check.sh` green.
|
||||
|
||||
### Phase 4 — UI selector in the learning-behaviour dock panel
|
||||
`exploration-store` state + the dock-panel `FEEDBACK_MODE` selector ("Explore and place" / "Geometric dislike"), `SOLO_MODE` selector, exploring/anchor indicators, the cold-start prompt, the honest solo-limit copy (British spelling). **Gate:** `bunx playwright test` (the e2e felt-loop specs); full `scripts/run-all-tests.sh` (chokepoint E) green.
|
||||
|
||||
### Phase 5 — firmware readiness (gated, not on the critical path)
|
||||
Confirm RP2350 SRAM budget with firmware `Cap`/undo-depth template params (`ReplayStore<2,126,16>` + anchors + `scratch_undo_` depth 2). Verify chokepoint A (audio correct) / B (no perf regression) before flashing. Re-pin the orphaned memllib `4733ca0` to `0a541cc`.
|
||||
|
||||
---
|
||||
|
||||
## 6. Parity + perf contract
|
||||
|
||||
Native==WASM within 1e-5 is preserved because **every new operation is deterministic f32 arithmetic on the per-instance `nisps::Rng`** (no libc `rand()` anywhere). Both sides run the same Rng from the same seed (`kSeed ^ kFeedbackSalt`, matching the `MLHandle` `0xFEEDBACC0DE` salt).
|
||||
|
||||
### 6.1 Host ctest (`tests/cpp/test_mlp_feedback.cpp`, already registered)
|
||||
Rewrite the cases that encode the old 2nd-press-cancels state machine (deliberate break). Add: replay dedup/deepen at 0.05; k-NN centroid selection + deterministic tie-break; geometric push direction sign (target moves away from centroid); taper; cold-start `posMemCount==0` fallback; **solo bit-identical assertion** — after `train_masked` with MaskGradients, non-soloed final-layer columns are `==` (not 1e-5-near) to pre-train (Anchored-Manifold's stronger-than-parity oracle); scratchpad-never-trained invariant; place→warm-start reachability (both anchors reproduced at their corners); determinism under fixed seed.
|
||||
|
||||
### 6.2 Native↔WASM parity (`parity_check.cpp` + `parity_wasm.mjs`, bump `kVersion`)
|
||||
Extend Stage 5: **(S5a Mode 1)** seed `ParityMLP`+`ReplayStore`, store 3 fixed positives + 1 negative, run geometric push (centroid→target→`train_targets`), push 126 post-train probe outputs both sides. **(S5b Mode 2)** enter scratchpad, reroll×2, nudge, undo, place 2 anchors at fixed inputs, finalise (warm-start at fixed lr/iters/seed), push probe outputs + 12 weight probes. **(S5c Solo)** set `solo_mask={1,0,…}`, `train_masked`, assert non-soloed columns byte-identical pre/post. `scripts/parity-check.sh` float32-diffs at 1e-5 — new floats covered automatically.
|
||||
|
||||
### 6.3 Perf contract
|
||||
`replay.hpp` + anchors + `solo_mask_` + `scratch_undo_` are all fixed `std::array` (Cap/depth compile-time, zero heap). Hot loops O(Cap*NOut) over fixed arrays; `roll`/`centroid`/`train_pair_` marked `NISPS_FORCE_INLINE` where hot; `.f` on every literal (`0.0025f`, `0.25f`, `0.5f`, `16.f`, `0.05f`). No virtual dispatch (plain template). Feedback runs on the **control core, never the audio ISR** (mirrors firmware `loopCallback`) → chokepoint B structurally satisfied. SRAM budget tuned per-target via `Cap`/depth template params.
|
||||
|
||||
---
|
||||
|
||||
## 7. UI — learning-behaviour dock panel
|
||||
|
||||
- **`FEEDBACK_MODE` segmented control:** *Explore and place* (default) / *Geometric dislike*. One-line description per mode in the operator's language.
|
||||
- **`SOLO_MODE` selector** (visible when any output is armed): *Mask gradients* (default) / *Zero loss* / *Don't-care example*, with the honest "as far as a shared network allows" caveat.
|
||||
- **Exploring indicator** + **anchor count** during Mode-2 sessions; a **Done / finalise** button and a **cancel** gesture; nudge + undo controls.
|
||||
- **Cold-start prompt** (Mode 1, zero positives): *"Like a few sounds first so the system knows what to move away from."*
|
||||
- **`unifyMemories` toggle** (advanced): share likes/dislikes across modes (default on).
|
||||
- Copy uses British spelling throughout.
|
||||
|
||||
---
|
||||
|
||||
## 8. ALIGNMENT.md updates (same commit as the code)
|
||||
1. **RETRACT** the `feedback-modes-port-spec.md §2.5/§7` "geometric push out of scope / AVOID = move_weights" accepted-divergence note — Mode 1 now ports the latest upstream. Without this, a future session will "rediscover" the geometric push as scope creep.
|
||||
2. Record the **single deliberate firmware divergence**: the `useRandom` degenerate branch uses `nisps::Rng`, not libc `rand()` (value generated, never compared; native==WASM holds).
|
||||
3. Record the **TrainBatch-vs-unshuffled-SGD** behavioural divergence: nisps trains per-sample SGD, not shuffled two-LR batches — `browser != firmware` behaviourally, by design (separate verification targets).
|
||||
4. State the **shared-trunk solo limit** as accepted: no realisation both perfectly isolates and stays expressive.
|
||||
5. Re-pin orphaned memllib `4733ca0` → `0a541cc`.
|
||||
253
docs/redesign/upstream-firmware-survey.md
Normal file
253
docs/redesign/upstream-firmware-survey.md
Normal file
|
|
@ -0,0 +1,253 @@
|
|||
# Upstream Firmware Survey — git archaeology of the MusicallyEmbodiedML ecosystem
|
||||
|
||||
_Dated 2026-06-27. Author: research agent (read-only). All SHAs from a fresh `git fetch --all` + memllib submodule-gitdir fetch (incl. operator's memllib fork `monkey-w1n5t0n/memllib`)._
|
||||
|
||||
## TL;DR — the divergence headline
|
||||
|
||||
**origin/main and upstream/main are two different codebases sharing one ancestor.** They forked at
|
||||
`6efbe9c` ("voicing", 2026-04-14) and have NOT touched a common file since. origin/main is the
|
||||
**complete C++20 + SolidJS clean rewrite** (`nisps/`, `playground/`, `firmware/`, schemas/codegen).
|
||||
upstream/main is the **original Arduino `.ino` + `modes/*.hpp` firmware tree**, which the lab kept
|
||||
developing in parallel. The 49 commits upstream has that origin lacks are **real firmware features on
|
||||
an architecture origin deliberately abandoned** — they are *ports*, not *merges*. There is no clean
|
||||
fast-forward or 3-way merge path; every integration is a re-implementation against the new core.
|
||||
|
||||
```
|
||||
6efbe9c "voicing" (2026-04-14) ← merge-base, last shared tree
|
||||
/ \
|
||||
origin/main (193 ahead) upstream/main (49 ahead)
|
||||
C++20/SolidJS rewrite old .ino/modes firmware tree
|
||||
pins memllib 4733ca0 (orphan) pins memllib e291192 (+62 vs origin's line)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 1. Branch map
|
||||
|
||||
### 1.1 Superproject — `origin` (operator fork) and `upstream` (lab)
|
||||
|
||||
| Branch | Remote | Last push | Author | vs its main¹ | Purpose | Judgment |
|
||||
|---|---|---|---|---|---|---|
|
||||
| `main` | origin | 2026-06-16 | monkey-w1n5t0n | — | The C++20/SolidJS rewrite. | canonical |
|
||||
| `feat/feedback-explore-modes` | origin | 2026-06-08 | w1n5t0n | +42 / −193 | Pins memllib `abe93ec` = the `FEEDBACK_MODE` (avoid / randomise-outputs / randomise-MLP) commit. Source material for `docs/redesign/feedback-modes-port-spec.md`. | **real RL work to port** (already spec'd, not yet in main) |
|
||||
| `feat/useq-celium-opus46` | origin | 2026-06-14 | w1n5t0n | +2 / −36 | MEMLCelium useq experiment (Opus-4.6 session). | experiment — superseded by opus47 |
|
||||
| `useq-celium-opus47` | origin | 2026-04-20 | w1n5t0n | +6 / −36 | Later MEMLCelium useq iteration. | experiment |
|
||||
| `feat/unified-preset-system` | origin | 2026-04-16 | w1n5t0n | +46 / −55 | Preset-system spike; heavily diverged, stale. | abandoned/spike |
|
||||
| `port-solidjs` | origin | 2026-06-14 | w1n5t0n | +31 / −66 | Early SolidJS port scaffolding; main has moved well past it. | superseded by main |
|
||||
| `main` | upstream | 2026-06-22 | chriskiefer | — | Old firmware tree; LiveFX/VerbFX/DJFX/staticmlp/jolts/noise. | **real firmware to port** |
|
||||
| `SB2026` | upstream | 2026-05-15 | chriskiefer | +0 / −30² | Superbooth-2026 build: mode sample-rate, **DISLIKES**, screen layout, sample player. **Fully merged into upstream/main.** | merged — port via memllib |
|
||||
| `SaxFX` | upstream | 2026-05-05 | chriskiefer | +0 / −42² | Sax FX app. **Fully merged into upstream/main.** | merged — historical |
|
||||
| `bettysax` | upstream | 2026-02-21 | chriskiefer | +1 / −72² | Bettysax; 1 extra commit `c5f0314 "demo"` not in main. | novelty/demo |
|
||||
|
||||
¹ "+X / −Y" = (commits the branch is ahead of its own main) / (commits behind). For `git rev-list --left-right --count main...branch` the **left** count is main-only (= how far the branch is *behind*) and the **right** count is branch-only (= *ahead*) — read them in that order. For upstream branches, main = upstream/main.
|
||||
² SB2026 / SaxFX show ahead=0 because they are *contained in* upstream/main (`git rev-list upstream/main..branch` = 0). The "−30/−42" is just how far main has advanced past their tip.
|
||||
|
||||
### 1.2 Submodule — `src/memllib` (lab `origin` = MusicallyEmbodiedML/memllib; `w1n5t0n` = operator fork)
|
||||
|
||||
memllib is where the **RL interface (`InterfaceRL`) and the DISLIKES/feedback logic actually live** — the
|
||||
superproject only pins it. Two pins matter:
|
||||
|
||||
- **origin/main superproject pins memllib `4733ca0`** — an **orphaned commit**: not reachable from any
|
||||
ref in the lab repo *or* the operator's fork. Likely a rebased/abandoned tip. Cannot be diffed by SHA;
|
||||
treat origin/main's memllib as "stale, off-graph."
|
||||
- **upstream/main superproject pins memllib `e291192`** ("l r input swap", 2026-06-22) — current lab tip,
|
||||
and it **already contains the SB2026 dislikes/rlview work** (`80420d1 "rlview and dislike"`,
|
||||
`d9bdcd6 "remove dislike opts screen"`).
|
||||
- **origin/feat/feedback-explore-modes pins memllib `abe93ec`** ("InterfaceRL: add FEEDBACK_MODE setting",
|
||||
2026-06-08, authored by w1n5t0n on the operator's fork). This is the operator's own RL feedback commit,
|
||||
branched off lab `616b8e7` ("cc select"); it is +1/−6 vs `e291192`.
|
||||
|
||||
| memllib branch | Last push | Author | ahead of main³ | Purpose | Judgment |
|
||||
|---|---|---|---|---|---|
|
||||
| `main` (e291192) | 2026-06-22 | chriskiefer | — | Current lab RL lib (incl. dislikes). | canonical lib |
|
||||
| `SB2026` (1997699) | 2026-05-15 | chriskiefer | +0 / −20 | Superbooth dislikes/rlview. **Fully merged into main.** | merged |
|
||||
| `w1n5t0n/feat/feedback-explore-modes` (abe93ec) | 2026-06-08 | w1n5t0n | +1 / −6 | Operator's FEEDBACK_MODE 3-mode enum. | **real — already spec'd to port** |
|
||||
| `maincandidate` | 2025-11-06 | chriskiefer | +54 | Pre-2026 staging. | historical |
|
||||
| `displayview` | 2025-10-13 | chriskiefer | +82 / −11 | Display work. | historical |
|
||||
| `bettysaxanalysis` | 2025-09-03 | A. Martelloni | +124 | Sax analysis. | historical |
|
||||
| `display2`, `pafcarl`, `pafsynth`, `bunty_new`, `studiofx`, `xiasri`, `rlversion`, `sdcard`, `superbooth`, `display`, `imlzoom`, `tomchris`, `iceland`, `chris`, `features` | 2025-05…08 | chris / Martelloni | +120…+243 | 2025 app/feature branches. | historical/abandoned |
|
||||
|
||||
³ `git rev-list --left-right --count main...branch` inside the memllib gitdir.
|
||||
|
||||
### 1.3 Tags
|
||||
|
||||
| Tag | Points at | Date | In upstream/main? | In origin/main? | Notes |
|
||||
|---|---|---|---|---|---|
|
||||
| `LiveFX_1.0.0` | `701f2d9` | 2026-06-22 | yes | **no** | Newest firmware release; old tree. |
|
||||
| `VerbFX_1.1.0` | `a18d9c6` | 2026-06-16 | yes | **no** | Large-verb FX release. |
|
||||
| `DJFX_TR6S_MICROQ` | `ed4ac6d` | 2026-05-25 | yes | **no** | DJFX + TR-6S + MicroQ firmware. |
|
||||
| `MEMLCelium_0_1_alpha` | `1e441cb` | 2026-04-20 | yes | **no** | MEMLCelium alpha; old tree. |
|
||||
| `PAF_1_1_0` | `bae80fb` | 2026-03-29 | yes | **yes** | Pre-fork; in both. |
|
||||
| `VerbFX` | `bae80fb` | 2026-03-29 | yes | **yes** | Same commit as PAF_1_1_0; pre-fork. |
|
||||
|
||||
Only the two pre-fork tags (`bae80fb`) exist in origin/main. Every release tag from `MEMLCelium_0_1_alpha`
|
||||
onward marks **old-tree firmware origin/main never received**.
|
||||
|
||||
### 1.4 Sibling org repos (`gh repo list MusicallyEmbodiedML`, 70 repos)
|
||||
|
||||
Active / firmware-relevant:
|
||||
- **`memlp`** (2026-06-24) — the underlying NN library for microcontrollers; memllib's dependency. Recently active.
|
||||
- **`memllib`** (2026-06-22) — the RL/interface lib (the submodule). The live one.
|
||||
- **`MEMLNaut`** (2025-11-13) — the hardware/board repo.
|
||||
- **`memlp_tests` / `memlp_test` / `RLTests`** — test harnesses for the NN/RL libs; relevant if porting RL behavior and wanting reference vectors.
|
||||
- **`musicallyembodiedml.github.io`** (2026-06-22) — the docs site referenced by CLAUDE.md.
|
||||
|
||||
Everything else is a 2024–2025 per-app fork (PAF, Xiasri, studioFX, euclidean, FM, subtractive, XMOS-era
|
||||
boards, etc.) — historical context, **not integration targets**. The whole "one repo per app" pattern is
|
||||
exactly what the `nisps/` rewrite consolidates; these forks are the museum.
|
||||
|
||||
---
|
||||
|
||||
## 2. Dated timeline
|
||||
|
||||
| Date | Event |
|
||||
|---|---|
|
||||
| 2026-03-29 | `bae80fb` — `PAF_1_1_0` / `VerbFX` tags. Last point both trees share a tag. |
|
||||
| **2026-04-14** | **`6efbe9c` "voicing" — fork point (merge-base of origin/main & upstream/main).** |
|
||||
| 2026-04-20 | `MEMLCelium_0_1_alpha` (`1e441cb`); memllib `psram`. First upstream-only release. |
|
||||
| 2026-05-05 | upstream `SaxFX` branch tip (`33b6542 "memlib sub"`). (NB: `fcef420 "sax fx ftw"` is a *2026-05-15* commit on upstream/main, **not** the SaxFX tip — earlier draft conflated them.) |
|
||||
| 2026-05-15 | upstream `SB2026` tip (`c617fcc` "mode sample rate, **dislikes**, screen layout, sample player"). memllib `SB2026` (`1997699`). |
|
||||
| 2026-05-25 | `DJFX_TR6S_MICROQ` (`ed4ac6d`). |
|
||||
| 2026-06-08 | origin `feat/feedback-explore-modes` — operator's `FEEDBACK_MODE` work; memllib fork `abe93ec`. |
|
||||
| 2026-06-16 | **origin/main HEAD** (`f256217`, AudioWorklet fix). `VerbFX_1.1.0` (`a18d9c6`) on upstream. |
|
||||
| 2026-06-22 | **upstream/main HEAD** (`701f2d9` "livefx"); `LiveFX_1.0.0`. memllib main `e291192`. |
|
||||
| 2026-06-27 | This survey. |
|
||||
|
||||
---
|
||||
|
||||
## 3. Git relationships
|
||||
|
||||
- **Merge-base (superproject):** `6efbe9c` ("voicing", 2026-04-14).
|
||||
- **origin/main vs upstream/main:** `git rev-list --left-right --count` = **193 / 49** (origin 193 ahead = the
|
||||
entire rewrite; upstream 49 ahead = continued old-tree firmware). The trees are disjoint above the base:
|
||||
upstream/main's top-level is `*.ino`, `modes/`, `voicespaces/`, `*AudioApp.hpp`; origin/main's is `nisps/`,
|
||||
`playground/`, `firmware/`, `schemas/`, `codegen/`. **Correction (verified):** it is *not* true that "no file
|
||||
is co-modified." A `git merge-tree`/name-status comparison above the base shows ~20 paths the two sides both
|
||||
touched — but for nearly all of them the relationship is **delete-on-origin / modify-on-upstream** (the old
|
||||
`.ino`, `modes/*.hpp`, `voicespaces/VerbFX/basic.hpp`, `src/memlp`), i.e. delete/modify *conflicts*, not
|
||||
content overlaps. The one genuine both-sides-`M` conflict is the **`src/memllib` gitlink** (origin → `4733ca0`,
|
||||
upstream → `e291192`). So the practical conclusion stands — every path resolves to a conflict (delete/modify or
|
||||
submodule-pin) with nothing cleanly 3-way-mergeable; upstream commits must be hand-ported — but the literal
|
||||
"no file is co-modified" was wrong.
|
||||
- **What origin/main is *behind* on (the 49):** the upstream-only firmware features — newest first:
|
||||
`701f2d9 livefx`, `a18d9c6 verbfx`, `0538eb9 staticmlp`, `99bd9b1 jolts`, `7ceac18 noise` /
|
||||
`a6e7af1 explore noise update`, `75e0f44 large verb`, `414590b tr8s home mapping` / `be6eac7 fx, tr8s` /
|
||||
`5a34a4a djfx single-cc fix`, `d1182f1 djfx alpha`, `24c2ad8 d50, sysex`, `1436250 midi output select`,
|
||||
`98c7449 input selection`, `fcef420 sax fx`, `c617fcc SB dislikes/sample-rate/sample-player`,
|
||||
`ee2814e focus targets memlcelium`, `47fc77b/2429bcc geo push rlview`, `d301cc7 rlview and dislike`.
|
||||
- **What origin/main is *ahead* on (the 193):** the whole platform — `nisps/` core, 8 engines as concepts,
|
||||
CRTP modes, WASM bridge, SolidJS playground, schemas/codegen, verification harness (golden+parity+lint+CI),
|
||||
a-immersive feature parity. None of this exists upstream.
|
||||
- **memllib pin divergence:** origin/main → `4733ca0` (**orphaned**, off-graph); upstream/main → `e291192`
|
||||
(current, +62 commits on the lab main line, includes dislikes). origin's memllib pin is effectively
|
||||
abandoned and should be re-pinned regardless of what else is ported.
|
||||
- **SB2026 / SaxFX are dead branches** (fully folded into upstream/main); the *content* worth taking is
|
||||
reachable from upstream/main and from memllib main. **bettysax** has one stray `c5f0314 "demo"` commit —
|
||||
ignorable.
|
||||
|
||||
---
|
||||
|
||||
## 4. Ranked integration recommendations
|
||||
|
||||
Because the architectures are disjoint, "integrate" means **port the behavior into `nisps/` + `playground/`**,
|
||||
not `git merge`. Ranked by mission impact (RL/feedback richness and firmware-feature parity).
|
||||
|
||||
### Rank 1 — Land the FEEDBACK / DISLIKES port (origin already did the hard part)
|
||||
- **Source commits:** memllib `abe93ec` (operator's `FEEDBACK_MODE` 3-mode enum) on
|
||||
`origin/feat/feedback-explore-modes`; cross-checked against memllib main `80420d1`/`d9bdcd6` (SB2026 dislikes
|
||||
in `InterfaceRL`) which upstream/main pins via `e291192`.
|
||||
- **Why first:** This is the negative-feedback / RL core of the whole project, and the work is *already
|
||||
written up* as an implementation-ready spec at `docs/redesign/feedback-modes-port-spec.md` (new
|
||||
`nisps::ml::FeedbackController<MLP_T>`, WASM C API, 11 ctest cases, parity Stage 5). The branch
|
||||
`feat/feedback-explore-modes` and the half-present `nisps/ml/feedback.hpp` + `tests/cpp/test_mlp_feedback.cpp`
|
||||
in the working tree show this is mid-flight. **Finish it and merge to main.** The spec's AVOID reconciliation
|
||||
decision (route to existing `move_weights`, skip the firmware-only k-NN geometric centroid push) is sound and
|
||||
should be recorded in `ALIGNMENT.md`.
|
||||
- **Action:** complete `nisps/ml/feedback.hpp`, wire WASM bindings + `mode-runtime.ts`, land the tests, re-pin
|
||||
memllib off the orphan. Do **not** try to port the old `ReplayMemory` geometric push now (out of scope per spec).
|
||||
|
||||
### Rank 2 — Re-pin memllib off the orphaned `4733ca0`
|
||||
- **Source:** memllib main `e291192` (or `abe93ec` if you want the feedback commit pinned directly).
|
||||
- **Why:** origin/main pins an off-graph commit nothing can resolve; this is a latent footgun (a fresh
|
||||
`submodule update` cannot check it out cleanly from the canonical remote). Pin to a real, reachable commit
|
||||
as part of Rank 1. Low effort, removes a trap.
|
||||
|
||||
### Rank 3 — Port the new upstream firmware engines as `nisps/` engines (selective)
|
||||
The 49 upstream commits are firmware apps on the old tree. Port the ones with mission value, highest-value first:
|
||||
- **`a18d9c6 verbfx` / `75e0f44 large verb`** (→ `VerbFX_1.1.0`): origin already has `verb_fx`; diff the upstream
|
||||
large-verb params and bring improvements into the engine. **Medium effort, high audio value.**
|
||||
- **`d1182f1 djfx alpha` + `ed4ac6d` (→ `DJFX_TR6S_MICROQ`) + `5a34a4a` single-CC fix:** DJFX is a *new* engine
|
||||
not in origin's 8. Port as a new `nisps/engines/` engine + mode + schema if DJFX is wanted in the playground.
|
||||
**Higher effort (new engine).**
|
||||
- **`701f2d9/adbc058 livefx` (→ `LiveFX_1.0.0`):** newest app; evaluate whether it subsumes existing FX modes
|
||||
before porting.
|
||||
- **`0538eb9 staticmlp` + `99bd9b1 jolts` + `7ceac18 noise`/`a6e7af1 explore-noise`:** these are *RL/exploration*
|
||||
changes (static-MLP hold, jolts, explore-noise) that overlap conceptually with the Rank-1 feedback work and
|
||||
with origin's existing spread/noise model. Review them **alongside** the feedback port — some may already be
|
||||
covered by `RandomiseMlp`/`RandomiseOutputs`; "explore noise" likely maps to the existing `spread`/OU-noise
|
||||
knobs. **Audit before porting to avoid duplicating behavior.**
|
||||
- **`414590b tr8s home mapping` / `be6eac7 fx, tr8s` / `10b8fca dynamic focus for TR8S` / `296bec6 target
|
||||
selection`:** the TR-8S focus-mask / dynamic-focus work is the firmware analogue of the `activeDims_` focus
|
||||
feature the feedback spec already accounts for. Useful reference for the focus-mask UI; port the *mask
|
||||
semantics*, not the firmware display code.
|
||||
|
||||
### Rank 4 — Take SB2026 sample-rate / sample-player only if needed
|
||||
- **Source:** `c617fcc` (SB2026, already in upstream/main). Per-mode sample rate and a sample player are
|
||||
firmware-platform features; the playground runs through an AudioWorklet, so most of this is N/A. Port only the
|
||||
per-mode sample-rate concept if a mode genuinely needs it. **Low priority.**
|
||||
|
||||
### Do NOT integrate
|
||||
- `port-solidjs`, `feat/unified-preset-system`, `feat/useq-celium-opus46`, `useq-celium-opus47` (origin):
|
||||
superseded/abandoned spikes — main has moved past them.
|
||||
- upstream `SaxFX`, `bettysax` (incl. `c5f0314 "demo"`): app-specific / demo, no mission value to the rewrite.
|
||||
- All 2024–2025 memllib feature branches and per-app org forks: historical museum.
|
||||
|
||||
---
|
||||
|
||||
## 5. Surprises
|
||||
|
||||
1. **upstream/main is the *old* tree, not "main ahead of the fork."** The naming invites the assumption that
|
||||
upstream advanced the same codebase; it didn't — the lab kept shipping the pre-rewrite `.ino` firmware. The
|
||||
49 commits are a *parallel* line, never to be fast-forwarded.
|
||||
2. **origin/main pins an orphaned memllib commit (`4733ca0`)** that exists in neither the lab repo nor the
|
||||
operator's own memllib fork — an off-graph gitlink that a clean checkout cannot resolve.
|
||||
3. **The dislikes/RL feedback work the task flagged as "to find" is already (a) merged into upstream via SB2026
|
||||
→ memllib `e291192`, AND (b) re-implemented by the operator on `feat/feedback-explore-modes` (memllib
|
||||
`abe93ec`), AND (c) fully spec'd for the new core** in `docs/redesign/feedback-modes-port-spec.md` — with
|
||||
`nisps/ml/feedback.hpp` and `tests/cpp/test_mlp_feedback.cpp` already present (untracked) in the working tree.
|
||||
The feedback integration is not a research question; it's a half-finished implementation to land.
|
||||
|
||||
---
|
||||
|
||||
## Verification (checked against source)
|
||||
|
||||
_Adversarial re-check, 2026-06-27, independent of the original author. Re-ran the git/gh commands and a fresh
|
||||
bare clone of lab + operator-fork memllib._
|
||||
|
||||
**Confirmed correct:**
|
||||
- Merge-base `6efbe9c` "voicing" 2026-04-14 (`git merge-base origin/main upstream/main`).
|
||||
- origin/main HEAD `f256217` 2026-06-16; upstream/main HEAD `701f2d9` "livefx" 2026-06-22.
|
||||
- origin 193 ahead / upstream 49 ahead (`git rev-list --left-right --count origin/main...upstream/main`); the 49 upstream-only commits and the specific SHAs cited in §3 (`701f2d9`, `a18d9c6`, `0538eb9`, `99bd9b1`, `7ceac18`, `a6e7af1`, `75e0f44`, `414590b`, `be6eac7`, `5a34a4a`, `d1182f1`, `24c2ad8`, `1436250`, `98c7449`, `fcef420`, `c617fcc`, `ee2814e`, `47fc77b`, `2429bcc`, `d301cc7`) all resolve with matching messages.
|
||||
- **memllib `4733ca0` is genuinely orphaned** — `git cat-file` fails for it even after fetching BOTH `MusicallyEmbodiedML/memllib` and `monkey-w1n5t0n/memllib` into a bare clone. The survey's headline surprise holds.
|
||||
- memllib main = `e291192` and contains `80420d1 "rlview and dislike"` + `d9bdcd6 "remove dislike opts screen"`.
|
||||
- memllib `abe93ec` "InterfaceRL: add FEEDBACK_MODE setting" 2026-06-08 by w1n5t0n, branched off `616b8e7 "cc select"`, +1/−6 vs main; superproject `feat/feedback-explore-modes` pins it.
|
||||
- All six tags (`LiveFX_1.0.0` `701f2d9`, `VerbFX_1.1.0` `a18d9c6`, `DJFX_TR6S_MICROQ` `ed4ac6d`, `MEMLCelium_0_1_alpha` `1e441cb`, `PAF_1_1_0`/`VerbFX` `bae80fb`) — SHAs, dates, and origin/upstream containment all match.
|
||||
- Upstream branch ahead/behind: SB2026 `c617fcc` +0/−30, SaxFX +0/−42, bettysax +1/−72 with stray `c5f0314 "demo"`; memllib SB2026 = `1997699`.
|
||||
|
||||
**Corrected (real errors found):**
|
||||
1. **Origin-branch ahead/behind columns were inverted** in table 1.1 (the author read the `--left-right` left/right columns the wrong way round). Real values: `feat/feedback-explore-modes` +42/−193 (was +193/−42); `feat/useq-celium-opus46` +2/−36; `useq-celium-opus47` +6/−36; `feat/unified-preset-system` +46/−55; `port-solidjs` +31/−66. Fixed in table 1.1 and footnote ¹. (These branches are all *behind* main by ~the full rewrite, which is the opposite of what the original numbers implied.)
|
||||
2. **SaxFX timeline tip was wrong** (§2): the SaxFX branch tip is `33b6542 "memlib sub"` (2026-05-05), not `fcef420 "sax fx ftw"` — `fcef420` is a 2026-05-15 upstream/main commit, not on SaxFX. Date was right, SHA/message wrong. Fixed.
|
||||
3. **"No file is co-modified" (§3) was literally false.** ~20 paths are touched on both sides; for almost all the relation is delete-on-origin/modify-on-upstream (a conflict), and `src/memllib` is a genuine both-`M` submodule-pin conflict. The downstream conclusion (no clean 3-way merge; hand-port required) is unaffected. Fixed in §3.
|
||||
|
||||
The substance of the survey — disjoint trees, hand-port-only integration, orphaned memllib pin, feedback work already spec'd/half-implemented, ranked recommendations — survives verification. The errors are in the branch-delta bookkeeping, one timeline SHA, and one overstated absolute.
|
||||
|
||||
---
|
||||
|
||||
## Correction (2026-06-27, from the RL judge-panel skeptic, verified against source)
|
||||
|
||||
The **latest upstream `InterfaceRL` tip is `0a541cc` "highlighting"** (990 lines; `optimise()` at `:601`,
|
||||
`kCentroidK=4` at the hpp `:296`, decay `0.0025f` at `:664`, `negLRRatio` at `:742`) — NOT `e291192`/`abe93ec`,
|
||||
which are older. **Mode 1 "Geometric dislike" must be ported to `0a541cc` parity.** Note: the in-tree submodule
|
||||
gitdir resolves none of `4733ca0`/`e291192`/`abe93ec`/`0a541cc` — only the separate `-upstream` memllib
|
||||
checkout has them; the orphaned-pin footgun stands and `0a541cc` is the re-pin/port target. See
|
||||
`docs/redesign/rl-feedback-design.md` §1 for the verified-ground-truth list.
|
||||
Loading…
Reference in a new issue