9ad1f78ddd
Roughly 20k lines were deleted from this repo in the last week and much of the
corpus still described the pre-deletion world in the present tense. Executes
the §8 table: archive the retired, reclassify the executed, prune the stale.
aimmersive-clone-spec -> _archive/ with a deprecated-by note
feedback-modes-port-spec -> plans/, kind: plan, status: executed
manifold-parity-features -> plans/, kind: plan, status: active
playground-2.0-rewrite -> status: superseded
engine-architecture 434 -> ~120 lines; seam + spine kept, rewritten
present-tense against the shipped engine/
MAIN.md six contradicted claims fixed; registry resynced
vcv-module.md pruned to the current 8->16 contract and made the
single .nisps format spec
vcv/NISPS-FORMAT.md DELETED — documented a v1 format that no longer loads
vcv/README.md, BUILDING.md rewritten to the real contract, menu, OSC table
inputs/backends/dock trio grounding sections marked historical, dead cites fixed
Two rows of the §8 table were themselves wrong, corrected here: the deleted
full-state sync lives in backends-spec.md §6.3, not vcv-module.md (which has no
§6.3), and codegen/README.md was already a MAP pointer with no port-solidjs
trigger left to remove.
Beyond the table — found by sweeping every backticked path in the changed docs
against `git ls-files`, which is how these should have been caught before:
manifold/ONBOARDING.md documented a UI that Phase 1 deleted, as if current:
SplitStage, ReadoutStrip, InputMini, BackendAdvanced, AltitudeNav, and a
shot.spec.ts that does not exist. The whole stage table was keyed on a `focus`
axis that no longer exists — selection is now sandwich > particles >
composite. This matters more than the rest: CLAUDE.md tells every agent to
read ONBOARDING.md first for Manifold work, so it was actively teaching a
fiction. Rewritten against ConsoleApp.tsx.
MAP.md claimed the input layer reduces axes to the engine arity with an
"even/odd blend". input-layer.ts says the opposite in its own header: one
dedicated slot per axis, 1:1, into a 32-input over-provisioned head, and
mean-blending was removed deliberately because it diluted every source.
AGENT-REFERENCE.md still promised TS emission "returns at P5" (landed),
per-mode dims "become schema-real at P5" (landed at P5.3), and pointed at
nisps::FixedBuffer (deleted).
Doc-right/code-suspect, filed rather than fixed: VCV computes derivedMean/Std/
Delta and cachedNovelty behind a live context-menu toggle that nothing reads;
vcv/plugin.json points at the MusicallyEmbodiedML org rather than this repo's
origin; and the module defaults to UDP 7001+id%64 while bridge.ts defaults to
9000, so out of the box they do not meet.
Firmware-build docs are deliberately untouched — the PlatformIO migration
lands next and rewrites all of them.
363 lines
39 KiB
Markdown
363 lines
39 KiB
Markdown
---
|
||
kind: spec
|
||
deprecated-by: 2026-07-21
|
||
superseded-by: MAIN.md
|
||
---
|
||
|
||
# NISPS Immersive (`a-immersive.html`) — SolidJS Clone Spec (Archived)
|
||
|
||
**Note:** This specced a faithful SolidJS clone of the deployed vanilla a-immersive app — a
|
||
product that was never built. The SolidJS playground target was retired 2026-07-13 (branch
|
||
`archive/playground-solidjs`, tag `playground-solidjs-final`); the current browser app is the
|
||
Manifold convertible Console (`manifold/`), covered by the live corpus (`../MAIN.md`). The
|
||
vanilla a-immersive app it describes remains deployed at `meml.lnfinitemonkeys.org/`, so this
|
||
survives as a behavioural reference for that app, frozen as of 2026-06-27. File paths cited
|
||
below (e.g. `/home/w1n5t0n/deployments/meml-aimmersive/`) refer to that deployment, not this
|
||
repo.
|
||
|
||
This is a faithful, component-by-component rebuild spec for the deployed vanilla-JS app at `/home/w1n5t0n/deployments/meml-aimmersive/`. Source of truth: `a-immersive.html` (446-line DOM), `css/a-immersive.css` (2538 lines), `js/a-app.js` (4521-line module), and supporting modules under `js/nisps/`, `js/ui/`, `js/synth/`, `js/midi/`, `js/audio/`. Goal: a from-scratch SolidJS reimplementation that matches the look and behaviour 1:1, binding to a shared engine layer (the WASM IML + synth engines) rather than re-porting it.
|
||
|
||
---
|
||
|
||
## 0. Critical scoping note before you build
|
||
|
||
**The deployed app does NOT use the Boldness/Memory/Precision control surface.** `README.md` and `SPEC-controls.md` describe that system (`js/ui/control-surface.js`, `control-surface-ui.js`, `input-pipeline.js`), but **`js/a-app.js` never imports any of them**. (Verified: `grep "control-surface" js/a-app.js` → no hits.) Those modules belong to a different design and to the SolidJS migration target (`PLAN-solidjs-migration.md`).
|
||
|
||
What a-immersive *actually* exposes for ML tuning is a flat **"NISPS" params drawer** with 6 raw sliders: **Spread, Noise, RL Decay, Learn Rate, Max Iters, Convergence** (`buildEngineParams()`, a-app.js:3137). To clone a-immersive 1:1, build **that** drawer. If the new SolidJS app is meant to also gain Boldness/Memory/Precision, treat it as an additive feature on top of this spec, not a replacement.
|
||
|
||
Joystick input is fed **directly** (`joyX, joyY` ∈ [0,1]) into `iml.setInput(0/1, …)` — there is **no deadzone/zoom/curve/smoothing pipeline** in a-immersive. The only output-side shaping is the per-param override system (curve/min/max/freeze).
|
||
|
||
Many imports in a-app.js are for features visible only behind flags or that exist as dock entries (EOC Effects Chain, Modular engine, ShapeSeq behind `?shapeseq=1`, Faust additive/FM engines, audio-canvas, hand tracking). See §10 for what's core vs deferrable.
|
||
|
||
---
|
||
|
||
## 1. Component tree
|
||
|
||
Top-level provider tree wraps everything in store contexts (see §5). All components are absolutely/fixed-positioned over a full-viewport canvas — this is a single-screen HUD, not a flow layout.
|
||
|
||
```
|
||
<ImmersiveShell> // root; owns rAF loop, resize, init, keyboard, gamepad
|
||
├── <BackButton> // fixed top-left chevron → designs.html
|
||
├── <SynthQuickControls> // shown only when outputMode==='synth'
|
||
│ ├── <PlayButton> // play/pause icon, audio-needs-init pulse
|
||
│ ├── <PlayDrawer> // hover flyout: Vol + BPM sliders
|
||
│ └── <SynthPresetSelect> // tiered <select> (Manual/Beginner/.../Expert)
|
||
├── <MidiCcQuickControls> // shown only when outputMode==='midi-cc'
|
||
│ ├── enable button, output-device <select>, preset <select>, status span
|
||
├── <AudioCanvasMount> // div, populated by AudioCanvas (deferrable)
|
||
├── <ShapeSeqContainer> // behind ?shapeseq=1 (deferrable)
|
||
│
|
||
├── <FlowFieldCanvas> // #vis-canvas — visual mode particle system
|
||
├── <SynthVisualizer> // #synth-vis-canvas — 126-bar synth chart (synth mode)
|
||
│
|
||
├── <HeatmapStrip> // top strip; hidden in synth mode
|
||
│ ├── <HeatmapCell> × N // one per output param; drag-to-set, click-to-popup
|
||
│ ├── <HeatmapTooltip> // hover quick-info
|
||
│ └── <ParamOverridePopup> // portal: curve/min-max/freeze (+ CC editors)
|
||
│
|
||
├── <FloatingJoystick> // bottom-left; canvas knob + minimap + noise ring
|
||
│ ├── <JoyMapCanvas> // #joy-map: bg grid, training dots, knob, crosshair
|
||
│ ├── <NoiseRing> // RL exploration indicator (off/active/high)
|
||
│ ├── <FollowBadge> // "FOLLOW" pill when follow mode on
|
||
│ └── <GamepadStatus> // tiny orange status text
|
||
├── <HandTrackingPip> // replaces joystick when input==='hands' (deferrable)
|
||
├── <EocJoystick> // independent EOC mode only (deferrable)
|
||
│
|
||
├── <RlButtons> // bottom-center: −/+ cluster + undo + (linked label)
|
||
│ ├── <RlDownButton> (−, key 1)
|
||
│ ├── <RlUpButton> (+, key 2)
|
||
│ └── <UndoButton> (Z)
|
||
├── <EocRlButtons> // FX −/+ (keys 3/4), linked mode only (deferrable)
|
||
│
|
||
├── <StatusLine> // bottom: "N examples · loss X · noise Y"
|
||
│
|
||
├── <Dock> // right-side macOS dock
|
||
│ └── <DockIcon> × 6 // Train, Mode, Synth, NISPS, FX, Help
|
||
├── <DrawerStack> // stacked glass panels left of dock
|
||
│ ├── <TrainingDrawer> // actions, visual presets, loss canvas
|
||
│ ├── <ModeDrawer> // input + output pill toggles + follow pill
|
||
│ ├── <SynthDrawer> // engine switcher + audio/arp controls
|
||
│ ├── <MidiCcDrawer> // CC param management
|
||
│ ├── <ParamsDrawer> // "NISPS" tuning sliders (the 6 above)
|
||
│ └── <EocDrawer> // effects chain UI mount (deferrable)
|
||
├── <GroupOverrideDrawer> // portal: per-group curve + per-param rows (synth mode)
|
||
│
|
||
├── <HelpModal> // overlay; auto-opens on first visit
|
||
└── <Toast> // transient bottom message
|
||
```
|
||
|
||
### What each owns / renders
|
||
|
||
- **`<ImmersiveShell>`** — orchestrator. Holds no visual markup of its own beyond children. Owns: the `requestAnimationFrame` loop (`animate()`, a-app.js:4251 — polls gamepad, draws whichever canvas is active), the global `resize` handler (`onResize()`:4217 — resizes both canvases + joymap, re-inits particles, hides popups), keyboard (`wireKeyboard()`:3675), gamepad wiring, init sequence (`init()`:1265), auto-save interval (10 s), and the `?debug=1` probe (`window.__nisps`). Mounts `<Toast>` lazily.
|
||
- **`<FlowFieldCanvas>`** — wraps `FlowFieldVisualizer` (`js/ui/visualizer.js`). 400-particle Canvas2D flow field driven by 20 outputs (see §7). Visible when `outputMode==='visual'`; gets `.hidden-canvas` otherwise. Internal render loop is driven by the shell's rAF, not its own.
|
||
- **`<SynthVisualizer>`** — wraps the `SynthVisualizer` class (defined inline in a-app.js:584). Draws all visible (non-muted) param bars grouped into labeled sections, with hover tooltip and drag-to-set. Hovering a **section label** opens `<GroupOverrideDrawer>`. Visible (`.active`) only in synth mode.
|
||
- **`<HeatmapStrip>`** — 22 px top strip of equal-flex bars, one per output. Each bar's width = output value %. Bars: hover→tooltip, drag→set value, click→`<ParamOverridePopup>`. Hidden in synth mode (the SynthVisualizer replaces it there). Gets `.shared-mode` (adds "+ FX" label) when EOC shared mode is active.
|
||
- **`<FloatingJoystick>`** — the primary input. A 160×160 canvas knob with integrated minimap (training-example dots + current position). Drag = relative move; tap = snap-to-position; double-tap = toggle follow. See §2.
|
||
- **`<RlButtons>`** — the −/+/undo cluster (see §2). `<StatusLine>` is a read-only floating string.
|
||
- **`<Dock>` + `<DrawerStack>`** — dock icons toggle the matching drawer (Help opens the modal instead). Each drawer is an independent glass panel that shows/hides via a `.hidden` class; multiple can be open at once.
|
||
|
||
---
|
||
|
||
## 2. Every control and interaction (exact behaviour)
|
||
|
||
### 2.1 Floating joystick (`wireJoystick()`, a-app.js:2280)
|
||
Listeners are on `#joy-map` (pointerdown) + `window` (pointermove/up/cancel).
|
||
- **Tap / pointerdown**: snap `joyX/joyY` to the tapped position inside the circle. Mapping: `dx = clientX - rectLeft - size/2`; `maxR = size/2 - 8`; `joyX = clamp(0.5 + (dx/maxR)*0.5)`; `joyY = clamp(0.5 - (dy/maxR)*0.5)` (Y inverted). Then `drawJoyMap()` + `onJoystickMove()`. Starts a drag.
|
||
- **Double-tap** (two pointerdowns < 350 ms apart): `toggleFollowMode()`; consume (no drag).
|
||
- **Drag / pointermove**: relative — `joyX = clamp(startJX + dx*scale*0.5)` where `scale = 1/maxR`; Y inverted. Redraw + `onJoystickMove()` each move.
|
||
- **Follow mode**: when on, pointer motion anywhere over `#vis-canvas` (and during window pointermove) sets `joyX = clamp(clientX/innerWidth)`, `joyY = clamp(1 - clientY/innerHeight)` — i.e. the whole screen becomes the pad, no hold needed.
|
||
- **`onJoystickMove()`** (2378): `iml.setInput(0,joyX); setInput(1,joyY); process()`; route + heatmap + `syncRawParamsFromOutputs`; push to `joyTrail` (cap 30, used by enhanced map / not strictly needed for base clone). Skips when `inputMode!=='joystick'`.
|
||
- **`drawJoyMap()`** (2193): circular clip; bg `rgba(13,13,13,0.7)`; quarter grid lines `rgba(255,255,255,0.06)`; ring border; **training dots** at `(fx = features[i][0]*w, fy=(1-features[i][1])*h)`, hue = `(labels[i][3]||0)*360` (i.e. coloured by output #3 = Hue), radius 3; **knob** orange `rgba(255,106,0,0.9)` r=8 with shadowBlur 12 + white inner dot r=3; orange crosshair.
|
||
- **Noise ring** (`updateNoiseRing()`:2879): `noiseLevel>0.15` → `.active.high` (6 px red border); `>0.01` → `.active` (3 px); else none.
|
||
|
||
### 2.2 RL +/−/undo cluster (`onThumbsUp`/`onThumbsDown`/`onUndo`)
|
||
Wired in both `wireControls()` (2567) and via keyboard.
|
||
- **`+` (Up, key `2` / Numpad2 / gamepad RB / hand "thumbsup")** — `onThumbsUp()` (2988):
|
||
1. `pushUndoSnapshot()` (weights + noiseLevel + exampleCount; stack cap 20).
|
||
2. `target.addExample(getCurrentInputs(), [...rawParamValues])` — **adds the current input→current-output pair as a training example** (i.e. "this mapping is good, keep it").
|
||
3. `noiseLevel *= rlExplorationDecay (0.97)`, floored at 0.005.
|
||
4. `flash('btn-thumbsup')`, `updateNoiseRing()`, then **async train** (`trainModelAsync()`); EOC target trains directly.
|
||
- **`−` (Down, key `1` / Numpad1 / gamepad LB / hand "thumbsdown")** — `onThumbsDown()` (3012):
|
||
1. `pushUndoSnapshot()`.
|
||
2. `noiseCap = 0.3*(1-spread) + 0.05*spread`; `noiseLevel = min(noiseLevel*1.5, noiseCap)`.
|
||
3. `target.moveWeights(noiseLevel, spread)` — **perturbs network weights to explore a new mapping** (no example added).
|
||
4. Re-route outputs from the affected IML; `updateStatus()`, `updateNoiseRing()`, `flash('btn-thumbsdown')`.
|
||
- **Undo (key `z`/`Z`, button between +/−)** — `onUndo()` (2949): pop snapshot, restore weights (`_setFlatWeights`) + noiseLevel, re-infer, route. Button gets `.has-undo` when stack non-empty (`updateUndoButton()`).
|
||
- **`flash(id)`**: add `.flash` class for 250 ms (scale pop animation).
|
||
|
||
### 2.3 Keyboard accelerators (`wireKeyboard()`, 3675)
|
||
Ignored when focus is in INPUT/SELECT/TEXTAREA, and on `e.repeat`. `preventDefault()` each.
|
||
- `1`/Numpad1 → `onThumbsDown()`; `2`/Numpad2 → `onThumbsUp()`.
|
||
- `3`/Numpad3 → FX thumbs-down (only if `imlEoc` && linked mode); `4`/Numpad4 → FX thumbs-up (same guard). These temporarily set `eocTrainingTarget='eoc'`, fire, restore, `flash('eoc-rl-minus/plus')`.
|
||
- `z`/`Z` → `onUndo()`.
|
||
|
||
### 2.4 Interactive heatmap (`buildHeatmap()`:1728, `setHeatmapValue()`:1829, `<ParamOverridePopup>`:1930)
|
||
Per cell:
|
||
- **Hover** (pointerenter): tooltip `"<name>: <value.toFixed(3)> ▾"`, positioned at cell's left. pointerleave hides; if its popup is open, schedule hide in 300 ms.
|
||
- **pointerdown**: record down position, set `_dragging`, `setPointerCapture`, add `.dragging` (→ `cursor:ew-resize`).
|
||
- **pointermove**: if moved >3 px in x or y, mark `_didDrag` and call `setHeatmapValue(i,e,cell)` — `x = clamp((clientX-rectLeft)/rectWidth)`; writes `rawParamValues[i]=x`, routes, updates heatmap, syncs; if param frozen, drag updates its `fixedValue`. Tooltip follows.
|
||
- **pointerup**: if **not** dragged → it's a click → toggle the override popup (`showParamPopup(i)` / `hideParamPopup()`).
|
||
- **Override popup** (`<ParamOverridePopup>`): glass card positioned below the cell (260 px wide, clamped to viewport). Rows:
|
||
- *(MIDI CC mode only)* editable **Name** (text), **CC#** (0–127; auto-renames from `CC_NAMES`), **Ch** (1–16) — each persists via `saveCCMap`.
|
||
- **Curve**: 36×36 draggable canvas (`_drawCurveOnCanvas` + `_wireCurveDrag`), value label; vertical drag sets curve ∈ [0,1], 0.5 = linear. Routes live.
|
||
- **Range (min/max)**: dual-thumb overlapping `<input type=range>` (min blue `#4488ff`, max orange `#ff6a00`) with a fill bar; clamps min≤max; label `"min–max"`. Routes live.
|
||
- **Freeze**: button toggles `frozen`; when frozen, captures current output into `fixedValue`, reveals a value slider (blue), and the param is removed from NISPS control (its output is pinned to `fixedValue`). Frozen cells get a hatched overlay + dimmed bar (`.heatmap-cell-frozen`).
|
||
- Popup stays open while hovered; closes 300 ms after leaving both popup and cell.
|
||
|
||
### 2.5 Synth visualizer + group drawer (`SynthVisualizer`:584, `showGroupDrawer()`:3953)
|
||
- **Bars**: only non-muted params drawn, grouped with 2 px gaps between sections; each section gets a centered label at the top (`_drawSectionLabel`). Bar height = display value (lerped at 0.12/frame toward target). Hover any bar → in-canvas tooltip (name / Val / Range / Curve).
|
||
- **Drag a bar** (`enableInteraction(true)` in synth mode): sets that param's raw value from Y (`yToValue`), routes + updates heatmap. Sliding across bars reassigns the dragged index.
|
||
- **Hover/click a section label** → `<GroupOverrideDrawer>` (320 px glass card, portal). Contents:
|
||
- **Group master curve**: 48×48 draggable canvas. Vertical drag applies a *relative delta* to the group curve **and to every per-param curve** in the section (preserving relative offsets). `delta = dy/80`.
|
||
- **Per-param rows**: name, 28×28 per-param curve canvas (vertical drag), dual-range min/max slider, and an **M (mute)** button. Muted rows hide curve+range, show a grey value slider, and strike-through the name. Mute removes the param from NISPS control.
|
||
- For C15 this is backed by `groupOverrides[si]` + `SYNTH_SECTIONS`; for Faust engines by `engineParamOverrides` + `nonC15Sections` (group curves remembered by name across engine swaps). Unify behind a `getSectionView(sectionIndex)` adapter (a-app.js:3916).
|
||
|
||
### 2.6 Output-mode tabs (`setOutputMode()`:2786)
|
||
In `<ModeDrawer>`, pill toggle: **Visual / Synth / MIDI CC / Audio Canvas** (`#output-toggle-float`). On switch:
|
||
- Compute `targetOutputs = outputCountForMode(mode)` (visual 20, synth = engine paramCount [126 for C15], midi-cc = `midiCCMap.length`, audio-canvas = `audioCanvas.getOutputCount()`).
|
||
- If output count changes **and** there are training examples: `confirm()` a weight-reset warning; on cancel, revert pill. Then `resizeMLP()` (warm-starts joystick weights, clears examples, recreates hand IML).
|
||
- Toggle canvases/strips: synth → hide `#vis-canvas` (`.hidden-canvas`), show `#synth-vis-canvas` (`.active`), hide heatmap strip, show synth quick controls, `synthVisualizer.enableInteraction(true)`. Other modes show heatmap strip, hide synth vis, disable interaction; midi-cc shows MIDI quick controls; audio-canvas shows its wrap.
|
||
- `buildHeatmap()` + `routeOutputs()` + `buildEngineParams()` + sync.
|
||
|
||
### 2.7 Input toggle + follow (`wireInputToggle()`:2622, `toggleFollowMode()`:2360)
|
||
- `<ModeDrawer>` "Input" pill toggle: **Joystick / Hands**. Switching to Hands lazily constructs `HandTracker`, requests camera, swaps `iml=imlHand`, hides joystick, shows PIP; on camera error reverts to joystick. (Hand tracking deferrable — see §10.)
|
||
- **Follow pill** (`#follow-pill`) and the joystick double-tap both call `toggleFollowMode()`. `updateFollowUI()` shows the FOLLOW badge, adds `.follow-active` (pulsing border) and marks the pill active.
|
||
|
||
### 2.8 Dock icons → drawers (`wireDock()`:2521)
|
||
- Click a `.dock-icon[data-drawer=X]`: if `X==='help'` open the modal; else toggle `#drawer-X` `.hidden` and the icon's `.active`. `.drawer-close` buttons close their drawer + deactivate the icon. Drawers are independent (stack scrolls). The 6 icons: **Training, Mode, Synth, NISPS (params), FX (eoc), Help**.
|
||
|
||
### 2.9 Presets
|
||
Two distinct preset systems — keep them separate:
|
||
- **Visual/RL presets** (`<TrainingDrawer>` chips: Calm/Chaos, Rainbow, Vortex, Spiral, Embers) — `loadPreset(name)` (3060): clears the **joystick** dataset, adds the preset's hardcoded input→output examples (`PRESETS`, a-app.js:59; each output is 20 floats padded to N), trains synchronously, re-infers. These teach the *network*.
|
||
- **Synth presets** (`<SynthPresetSelect>` dropdown, tiered) — `applyPreset(presetId)` (460): sets which of the engine's params are active vs muted and their min/max/curve/fixedValue, via `groupOverrides` (C15) or `engineParamOverrides` (Faust). Tiers expose progressively more params (Beginner 15 → Expert full). These shape the *output mapping*, not the network. `?preset=<id>` URL param auto-applies on load.
|
||
|
||
### 2.10 Gamepad (`wireGamepad()`:3645)
|
||
`GamepadInput` with `invertY:true`. Left stick → `joyX/joyY` + `onJoystickMove`. Buttons: LB→down, RB→up, A→train, X→randomize, B→clearExamples. Status text in `#gamepad-status`.
|
||
|
||
---
|
||
|
||
## 3. Theme / branding tokens and key copy
|
||
|
||
### CSS custom properties (`:root`, css:11) — reproduce verbatim
|
||
```css
|
||
--glass-bg: rgba(13, 13, 13, 0.65);
|
||
--glass-border: rgba(255, 255, 255, 0.08);
|
||
--glass-blur: 16px;
|
||
--accent: #ff6a00; /* signature orange */
|
||
--accent-dim: rgba(255, 106, 0, 0.25);
|
||
--danger: #ff4466;
|
||
--text: #e0e0e0;
|
||
--text-dim: #888;
|
||
--radius: 12px;
|
||
--radius-sm: 8px;
|
||
--safe-bottom: env(safe-area-inset-bottom, 0px);
|
||
--dock-width: 48px;
|
||
--dock-gap: 8px;
|
||
```
|
||
- **Background**: `#0d0d0d` everywhere; `html,body { overflow:hidden; touch-action:none; user-select:none }`.
|
||
- **Font**: `'JetBrains Mono', 'SF Mono', 'Fira Code', 'Cascadia Code', monospace`; base `13px`. Loaded from Google Fonts (`JetBrains+Mono:wght@400;500;700`).
|
||
- **Glass mixin**: `background: var(--glass-bg); backdrop-filter: blur(var(--glass-blur)); border: 1px solid var(--glass-border)`. Drawers use a denser `rgba(13,13,13,0.88)` + `blur(20px)`.
|
||
- **RL up** = teal `#00c8a0` (border/bg tinted); **RL down** = red-orange `#dc3c14`. EOC RL up = `#4ecdc4`. Noise ring = `rgba(255,100,50,…)` → high `rgba(255,60,40,…)`.
|
||
- **Dock**: macOS magnify — `.dock-icon:hover { transform: scale(1.35) }`, neighbours scale 1.12 via `:has()`/sibling selectors; active icon tinted orange.
|
||
- **Animations to reproduce**: `drawerSlideIn` (0.2 s translateX), `follow-pulse` (1.5 s box-shadow), `rl-flash` / `btn-flash` (0.25 s scale), `audioInitPulse` (2 s, on play button when audio not started), `helpFadeIn`.
|
||
- **Heatmap frozen cell**: 45° hatched `rgba(80,160,255,0.15)` overlay + bar opacity 0.4.
|
||
|
||
### Key copy to reproduce verbatim
|
||
- **Title** (`<title>`): "NISPS Immersive". **Help modal H2**: "Welcome to NISPS"; subtitle: "Neural Interactive Shaping of Parameter Spaces".
|
||
- **Status line** default: `"0 examples · untrained"`; live format: `"<N> example(s) · loss <0.00000> · noise <0.000>"` (or `· untrained` when never trained). Middots are U+00B7.
|
||
- **RL buttons**: down has `−` + `1`; up has `+` + `2`; titles "Explore more" / "Keep this". Undo title "Undo last action".
|
||
- **Help modal** is a long static block — reproduce sections **What is this? / How it works / What to expect / Controls (Touch/Mouse, Keyboard 1/2/3/4/Z, Gamepad) / Hand Tracking / Synth Controls**, plus the "Got it" button. (Full text is in `a-immersive.html` lines 354–441 — copy it wholesale; it's the authoritative onboarding copy.) Auto-opens when `localStorage['nisps-help-seen']` is unset; closing sets it.
|
||
- **Dock labels**: Train, Mode, Synth, NISPS, FX, Help. **Drawer headers** (uppercase orange): Training, Mode, Synth, MIDI CC, NISPS, Effects Chain.
|
||
- **Training drawer buttons**: Add Example, Train, Clear Ex, Clear All, Randomize; preset chips Calm/Chaos, Rainbow, Vortex, Spiral, Embers.
|
||
- **Synth preset `<select>`**: option "Manual" (value `""`), then optgroups Beginner(1.1–1.4), Intermediate(2.1–2.4), Advanced(3.1–3.3), Expert(4.1–4.2).
|
||
- **Back button** → `designs.html`.
|
||
|
||
---
|
||
|
||
## 4. SVG icons (inline, reproduce as components)
|
||
All dock icons, the back chevron, the undo arrow, the play/pause/MIDI-keyboard icons, and the gesture ring are inline SVGs in the HTML. Reproduce as small SolidJS components (paths are in `a-immersive.html`):
|
||
- Back: `<path d="M10 2L4 8l6 6"/>` (stroke).
|
||
- Play: `<path d="M4 2l10 6-10 6z"/>` (fill); Pause: two rects.
|
||
- Dock: Train (mixer faders), Mode (two circles + diagonal), Synth (sine wave `M2 10c2-4 4-4 6 0s4-4 6 0`), NISPS (gear), FX (infinity-ish loop), Help (question mark in circle).
|
||
- Undo: `<path d="M3 7h6a4 4 0 0 1 0 8H7"/><path d="M3 7l3-3M3 7l3 3"/>`.
|
||
|
||
---
|
||
|
||
## 5. State model → SolidJS stores/signals
|
||
|
||
The deployed app keeps state in module-level `let`s. Map to stores/signals. **Performance rule** (from migration plan): ML outputs should be a single `Float32Array` signal updated once per frame, not a reactive array of 126 cells — drive canvases imperatively, drive the heatmap DOM widths via a cheap effect.
|
||
|
||
| Deployed state (a-app.js) | SolidJS home | Notes |
|
||
|---|---|---|
|
||
| `iml`/`imlJoy`/`imlHand`/`imlEoc` | non-reactive refs in an **EngineContext** | WASM instances; never put in a store |
|
||
| `inputMode` ('joystick'\|'hands') | `inputStore` signal | drives joystick vs PIP |
|
||
| `outputMode` ('visual'\|'synth'\|'midi-cc'\|'audio-canvas') | `modeStore` signal | gates canvases, heatmap, quick controls |
|
||
| `joyX,joyY,joyDragging,joyFollowMode,joyTrail` | `inputStore` (x/y signals + follow signal) | x/y change every frame → keep as plain signals |
|
||
| current outputs (`iml.getOutputs()`) | `outputStore` — **one `Float32Array` signal** | updated once per `routeOutputs` |
|
||
| `rawParamValues` | `outputStore.raw` (Float32Array signal) | the "set sliders" buffer for examples/heatmap drag |
|
||
| `noiseLevel,spreadLevel,rlExplorationDecay` | `mlStore` signals | spread also from `?spread`; default 0.6, noise 0.05, decay 0.97 |
|
||
| `iml.learningRate / maxIterations / convergenceThreshold` | exposed via `mlStore` getters/setters | the 6 NISPS sliders write these |
|
||
| `exampleCount,lastLoss,lossHistory` | `mlStore` (derived from iml after each op) | status line + loss canvas |
|
||
| `undoStack` (cap 20) | `mlStore.undo` (non-reactive array + `hasUndo` signal) | snapshot = {weights,noiseLevel,exampleCount} |
|
||
| `visualOverrides[20]` | `overrideStore.visual` (createStore) | {min,max,curve,frozen,fixedValue} |
|
||
| `groupOverrides[18]` (C15 sections) | `overrideStore.group` (createStore) | {curve, params:[{min,max,curve,muted,fixedValue}]} |
|
||
| `engineParamOverrides` (Faust, flat) | `overrideStore.engine` | null for C15 |
|
||
| `midiCCMap,midiCCOverrides` | `midiStore` | per-engine storage key |
|
||
| `audioCanvasOverrides` | `overrideStore.audioCanvas` | dynamic length |
|
||
| `activeEngine,activeSynthPresetId` | `synthStore` signals | engine id + preset id |
|
||
| `eocChain,eocTrainingTarget,nispsMode` | `eocStore` | deferrable |
|
||
| drawer open flags, active dock icon | `uiStore` (per-drawer booleans) | independent drawers |
|
||
| `activePopupParam,activeDrawerSection` | `uiStore` signals | popup/group-drawer routing |
|
||
|
||
**Persistence** (`saveState()`:4298, `loadState()`:4346): single `localStorage['nisps-a-immersive']` JSON blob. Saved fields: `features/labels` (joystick dataset), `handFeatures/handLabels`, `noiseLevel`, `outputMode`, `inputMode`, `joyX/joyY`, `groupOverrides`, `visualOverrides`, `midiCCOverrides`, `audioCanvasState`, `synthPresetId`, `engineId`, EOC module/mode state, `modularDspState`. On load: re-add examples and **train synchronously**, restore overrides, restore output mode (`skipConfirm:true`), but **never auto-restore `inputMode='hands'`** (camera permission). Auto-save fires on a 10 s interval and `saveState()` is also called after preset apply etc. In SolidJS, run a debounced effect (the migration plan uses 200 ms) that serializes the same shape; keep the storage key identical for migration continuity. Separate keys: `nisps-help-seen`, `nisps-midi-cc-map:<engineId>`, `nisps-modular-state`.
|
||
|
||
---
|
||
|
||
## 6. Engine call points (bind to the shared engine layer)
|
||
|
||
The shared engine layer is **`WasmIML`** (`js/nisps/nisps-wasm.js`) for ML, plus a **`SynthEngine`** (default `C15Adapter`) for audio. The new SolidJS app should transplant these (per migration plan they move under `core/` largely unchanged) and call them from store actions. Exact bind points:
|
||
|
||
**Construction (`init()` / `resizeMLP()`):**
|
||
- `imlJoy = await WasmIML.create(2, N_OUTPUTS, [32,48,64], 1000, 1.0, 0.00001)` then `imlJoy.randomiseWeights(spreadLevel)`.
|
||
- `imlHand = await WasmIML.create(14, N_OUTPUTS, [48,48,64], …)`.
|
||
- On output-count change: `WasmIML.createWithWarmStart(snapshot, newCount, …)` for the joystick IML (preserves hidden weights, re-randomises new output nodes), fresh `create` for hand.
|
||
- WASM module loaded from `wasm/nisps.js` (Emscripten). Note: the **engine MLP that runs audio in the worklet** is a separate WASM instance fixed at `MLP<2,10,14,18,126>` per CLAUDE.md — but in a-immersive the *inference path* is this `WasmIML`, sized to the active mode's output count.
|
||
|
||
**Per-frame inference (joystick/hand move, `onJoystickMove`/`onHandInput`):**
|
||
- `iml.setInput(0,x); iml.setInput(1,y); iml.process(); const out = iml.getOutputs();` → `routeOutputs(out)`.
|
||
|
||
**`routeOutputs(out)` (2425) — the output fan-out** (bind one store action):
|
||
- **synth**: apply `applyGroupOverrides(out[i], i)` per index → `synthVisualizer.setParams(overridden)`; throttled (≥50 ms) + dead-zone (>0.002) `activeEngine.setParam(i, v)` for indices < engine paramCount; shared-mode EOC params routed to `eocChain.setParam`.
|
||
- **midi-cc**: per CC, `applyGroupOverride(out[i],curve,min,max)` → `midiOutput.sendBatch([{channel,cc,value:round(v*127)}])`.
|
||
- **audio-canvas**: `audioCanvas.setOutputs(out)`.
|
||
- **visual**: per-index `applyGroupOverride` (or `fixedValue` if frozen) → `visualizer.setParams(vis)`.
|
||
|
||
**RL / training engine calls** (the interaction→engine mapping the prompt cares about):
|
||
| UI action | Engine call |
|
||
|---|---|
|
||
| **thumbsUp / + / key 2 / RB / hand-up** | `iml.addExample(inputs, rawOutputs)` **+** `iml.trainAsync(cb)` |
|
||
| **thumbsDown / − / key 1 / LB / hand-down** | `iml.moveWeights(noiseLevel, spread)` *(no example)* |
|
||
| **Add Example button** | `iml.addExample(getCurrentInputs(), [...rawParamValues])` |
|
||
| **Train button / gamepad A** | `iml.trainAsync(cb)` (async; updates loss canvas) |
|
||
| **Randomize / gamepad X** | `iml.randomiseWeights(spread)` → re-infer; resets noise to 0.05 |
|
||
| **Clear Ex / gamepad B** | `iml.clearDataset()` |
|
||
| **Clear All** | `clearDataset()` + reset `lossHistory/bestLoss/iterations` + `clearState()` |
|
||
| **Undo / key Z** | `iml._setFlatWeights(snapshot.weights)` → re-infer |
|
||
| **Visual preset chip** | clear joystick dataset, `addExample` × preset rows, `iml.train()` (sync) |
|
||
| **NISPS sliders** | write `spreadLevel` / `noiseLevel` / `iml.learningRate` / `iml.maxIterations` / `iml.convergenceThreshold` |
|
||
|
||
**WasmIML public surface to rely on** (stable contract): `setInput/setInputs/getOutputs/process/inferBatch`, `addExample/clearDataset/exampleCount`, `train()/trainAsync(cb)/isTraining/evalLoss()`, `randomiseWeights(spread)`, `moveWeights(speed,spread,pinMask?)`, `extractWeights()/createWithWarmStart()`, `_getFlatWeights()/_setFlatWeights()`, `getLayerStats()`, `lastLoss/bestLoss/lossHistory/totalTrainingIterations`, `dataset.features/labels`, `destroy()`. Training runs in a disposable Web Worker (`nisps-wasm-worker.js`).
|
||
|
||
**SynthEngine (`C15Adapter`) surface**: `id`, `displayName`, `paramMeta` (array of `{id,name,min,max,init,curve,group}`), `paramCount`, `init(ctx?)`, `stop()`, `running`, `setParam(index, normalized)`, `noteOn(note,vel)`, `noteOff(note)`, `getOutputNode()`, `setMasterVolume(v)`, `panic()`. Audio is started lazily on the play button (`activeEngine.init()`), which also wires EOC and (behind flag) ShapeSeq. Arpeggiator (`Arpeggiator`) and `MIDIInput` are attached to the active engine.
|
||
|
||
### Where the NEW 3-mode feedback selector ("Down Action") slots in
|
||
Today **`−` (thumbsDown) is hardcoded to `moveWeights`** (explore) while **`+` is hardcoded to `addExample`+`train`** (reinforce). The new feature wants the `−` action to be selectable among 3 feedback modes. Slot it as a **`<DownActionSelector>`** control:
|
||
|
||
- **Placement (UI)**: a small segmented control. Most faithful spot is inside the **Training drawer** (a new "Down Action" row above the example actions) and/or a compact 3-way toggle adjacent to the RL cluster (mirroring how the existing `rl-label`/EOC label sit above the buttons in linked mode). It must be reachable without opening a drawer if it's meant for live use — consider a tiny pill under `<StatusLine>` or next to the `−` button.
|
||
- **State**: add `downAction: 'explore' | <mode2> | <mode3>` to `mlStore` (persist it in the state blob).
|
||
- **Bind point**: `onThumbsDown()` is the single chokepoint. Branch on `mlStore.downAction`:
|
||
- `explore` (current): `target.moveWeights(noiseLevel, spread)`.
|
||
- the two new modes: call the corresponding engine primitive (e.g. a "negative example" path via `addExample` of a contrasting target, or a stronger/weighted `moveWeights`, or `randomiseWeights` — whatever the 3 modes are defined to do). `WasmIML.moveWeights` already accepts an `outputPinMask`, so a "pin-aware explore" mode is a natural third option.
|
||
- Keep `pushUndoSnapshot()` + `updateNoiseRing()` + `flash` shared across all branches.
|
||
- The existing `_rlTarget()` indirection (which lets EOC linked-mode redirect feedback to `imlEoc`) should compose with the new selector — i.e. `downAction` chooses *what* the feedback does, `_rlTarget()` chooses *which network* receives it.
|
||
|
||
---
|
||
|
||
## 7. Reference data tables (must match exactly)
|
||
|
||
### Visual mode: 20 named outputs (`VISUAL_PARAM_NAMES`, a-app.js:46 + `visualizer.setParams`:159)
|
||
Names: `Flow, Scale, Speed, Hue, Spread, Size, Trail, Turb, Attract, Radius, DispRate, DispAmt, Lifetime, Respawn, Advection, Inertia, Drag, Repulse, RepCnt, RepRate`. Colors in `VISUAL_PARAM_COLORS` (a-app.js:51, e.g. `Flow=#ff6a00, Scale=#00ccff, Speed=#ff6600, Hue=#ff00cc, …`). Output→visual-param ranges (verbatim, for the FlowFieldCanvas): `angleOffset=out0*2π`, `scale=0.001+out1*0.009`, `speed=0.5+out2*4.5`, `hueBase=out3*360`, `hueSpread=out4*120`, `particleSize=1+out5*5`, `fadeRate=0.01+out6*0.14`, `turbulence=out7*2`, `attractStrength=0.1+out8*2.9`, `attractRadius=40+out9*420`, `dispersionRate=0.2+out10*8`, `dispersionAmount=out11*3`, `particleLifetime=30+out12*470`, `respawnStyle=out13`, `advectionMode=out14`, `inertia=out15*0.98`, `drag=out16*0.35`, `repulsorStrength=out17*4.5`, `repulsorCount=floor(out18*4.999)`, `repulsorOrbitRate=0.1+out19*2.9`. 400 particles, value-noise flow field with central attractor, dispersion pulses, and orbiting repulsors. Transplant `FlowFieldVisualizer` as-is.
|
||
|
||
### Synth mode: C15 sections (`SYNTH_SECTIONS`, a-app.js:217) — for grouping the 126 bars
|
||
`Env A(7), Env B(7), Env C(6), Osc A(5), Osc B(5), Shp A(6), Shp B(6), Comb(8), SVF(9), Gap(6), FB Mix(9), Out Mix(14), Cabinet(8), Flanger(13), Echo(7), Reverb(6), Unison(3), Mono(1)` = 126, each with a section color. Param data (id/name/label/default/safeMin/safeMax/bipolar) is `SYNTH_PARAM_MAP` (`js/synth/param-map.js`, 126 entries) — transplant as data, unchanged.
|
||
|
||
### Curve math (`param-map.js`:287) — reproduce exactly
|
||
```
|
||
applyCurve(v, c) = c===0.5 ? v : v^(2^(4*(c-0.5))) // 0.5 = linear
|
||
applyGroupOverride(v,c,min,max) = min + applyCurve(v,c) * (max-min)
|
||
```
|
||
|
||
### `?tame` (default 1): seeds each C15 param's default override range to `[safeMin*tame, 1-(1-safeMax)*tame]` (a-app.js:248). `?spread` (default 0.6), `?preset`, `?debug`, `?shapeseq=1` are the URL params.
|
||
|
||
---
|
||
|
||
## 8. Layout / positioning cheat-sheet (fixed coordinates to match)
|
||
- Back button: `top:45px; left:8px; 36×36`. Synth quick controls: `top:45px; left:52px` (row). Heatmap strip: `top:0; height:22px`. Joystick: `bottom:100px; left:24px; 160×160`. Hand PIP: same anchor, `180×135`. RL buttons: `bottom:36px; left:50%` centered, 64×64 circles, 12 px grid gap, undo 28×28 spanning. EOC RL: `bottom:36px; left:calc(50%+100px)`, 48×48. Status line: `bottom:8px; center`. Dock: `right:8px; top:50%` vertical, 36×36 icons, 16 px radius glass. Drawer stack: `right:calc(48px+16px); top:28px; bottom:28px; width:260px`, scrolls, `pointer-events:none` on stack / `auto` on each drawer. Drawer max-height 400 px. Group drawer 320 px, param popup 260 px — both portalled to body and positioned relative to the hovered element.
|
||
- z-index ladder: canvases 0–1, heatmap strip 20, tooltip 25, joystick/back/quick 30, status 40, drawers/RL 45, dock 50, group-drawer/popup 50, dev panel 90, help overlay 100.
|
||
|
||
---
|
||
|
||
## 9. SolidJS implementation notes
|
||
- **Canvases**: `<FlowFieldCanvas>` / `<SynthVisualizer>` / `<JoyMapCanvas>` are components with a `ref` and an internal draw method; the **shell's single rAF** calls the active one (don't give each its own loop — matches `animate()` and avoids double-drawing). Resize via a window listener that re-inits particles.
|
||
- **Heatmap**: render `<For>` over `count` cells once per mode change; update bar widths via a `createEffect` reading the outputs signal — but throttle DOM writes (the deployed app writes width % directly each frame; with 126 cells prefer writing only changed cells).
|
||
- **Drawers/overlays**: build headless primitives (`Drawer`, `Overlay`, `Popover`, `Dock`) per the migration plan; the dock/drawer toggle is just boolean signals in `uiStore`. Independent open state (multiple drawers can coexist).
|
||
- **Pointer capture**: the joystick, heatmap cells, synth bars, curve canvases, and dual-range sliders all use `setPointerCapture` + manual drag math — replicate the exact thresholds (3 px click-vs-drag on heatmap; double-tap 350 ms; group-curve `dy/80`; per-param curve drag).
|
||
- **Untracked debug probe**: keep `window.__nisps` (synchronous, bypasses reactivity via `untrack`/`batch`) for Playwright — its API is the table in §6 plus `getLayerStats/inferBatch/saveState/evalLoss`. Also `window.__nispsEoc` (training target). These are how E2E tests drive the app.
|
||
|
||
---
|
||
|
||
## 10. Gaps / risks for a faithful clone
|
||
|
||
**Core (must build to match the page):**
|
||
- WASM IML inference + sync/async training + `moveWeights`/`randomiseWeights`/warm-start — the whole ML surface. Non-negotiable; transplant `nisps-wasm.js` + `nisps.wasm`.
|
||
- C15 synth engine (`C15Adapter`/`C15Bridge` + `param-map.js` + `presets.js`) — runs via AudioWorklet + SharedArrayBuffer; needs COOP/COEP headers (the deployed app uses `serve-coop.py`). The 126-param map, tiered presets, group/param overrides, and the synth visualizer are all core to "synth mode."
|
||
- Flow-field visualizer (20 outputs), heatmap + override popup, joystick + noise ring, RL cluster + keyboard + undo, dock/drawers, status line, help modal, the 6-slider NISPS drawer, visual presets, persistence. All core.
|
||
- Arpeggiator + MIDI input (used by synth mode); MIDI CC output mode (`midi-output.js`, `midi-cc-map.js`, `midi-cc-presets.js`).
|
||
|
||
**Deferrable / flagged (render scaffolds but feature can lag):**
|
||
- **Hand tracking / MediaPipe** (`hand-tracker.js`, PIP, gesture ring, 14-input `imlHand`) — heavy dep, camera permission; the help text and Mode toggle reference it but it can be a later add. Note as optional.
|
||
- **EOC Effects Chain** (`eoc/`, `eoc-chain-ui.js`, `eoc-joystick.js`, linked/independent/shared NISPS modes, second `imlEoc`, FX RL buttons, keys 3/4) — large subsystem; the dock has an "FX" icon but it's a self-contained module. Deferrable.
|
||
- **Modular engine** (512 params, `modular-engine.js`, `modular-ui.js` — 51 KB UI) and **Faust additive/FM engines** (`faust-engine-base.js`, additive/fm presets) — accessed via the engine switcher in the Synth drawer. The switcher + C15 are core; the alternative engines are deferrable (and Faust needs a running AudioContext + the Faust runtime). CLAUDE.md flags Faust engines as not-yet-wired in the new codebase.
|
||
- **Audio Canvas mode** (`audio/audio-canvas.js`, 60 KB) — a fourth output mode with dynamic output count; scaffolded but can defer.
|
||
- **ShapeSeq** (`shapeseq/`, behind `?shapeseq=1`) — sequencer; explicitly flag-gated, defer.
|
||
- **OSC bridge** (`nisps/osc-client.js`, `synth/osc-output.js`, `osc-bridge/`) — desktop/hardware bridge, not used in the default browser path; defer.
|
||
- **Gamepad** — small (`gamepad.js`), low-risk; include if cheap, else defer (it just mirrors joystick + RL).
|
||
|
||
**Behavioural risks to watch:**
|
||
- Output-count changes trigger `confirm()` dialogs and weight warm-start — replicate the guard so mode switches don't silently wipe training.
|
||
- Param throttling (50 ms / 0.002 dead-zone) on synth param sends is load-bearing (prevents ring-buffer flooding at 126×30 fps); keep it.
|
||
- The two preset systems (visual examples vs synth override tiers) are easy to conflate — keep them as distinct stores/actions.
|
||
- Frozen/muted semantics differ between the heatmap popup (`frozen`) and the group drawer (`muted`) but map to the same underlying field via the `getParamOverride` adapter — preserve that mapping (`muted`↔`frozen`).
|
||
- Help/onboarding, `nisps-help-seen`, and the 10 s autosave are small but part of the felt experience.
|
||
|
||
**Files that ARE the engine layer to bind to** (absolute paths): `/home/w1n5t0n/deployments/meml-aimmersive/js/nisps/nisps-wasm.js`, `/home/w1n5t0n/deployments/meml-aimmersive/js/nisps/nisps-wasm-worker.js`, `/home/w1n5t0n/deployments/meml-aimmersive/js/nisps/dataset.js`, `/home/w1n5t0n/deployments/meml-aimmersive/js/synth/c15-adapter.js`, `/home/w1n5t0n/deployments/meml-aimmersive/js/synth/c15-bridge.js`, `/home/w1n5t0n/deployments/meml-aimmersive/js/synth/param-map.js`, `/home/w1n5t0n/deployments/meml-aimmersive/js/synth/presets.js`, `/home/w1n5t0n/deployments/meml-aimmersive/js/ui/visualizer.js`. UI behaviour source of truth: `/home/w1n5t0n/deployments/meml-aimmersive/js/a-app.js`. Theme: `/home/w1n5t0n/deployments/meml-aimmersive/css/a-immersive.css`. DOM + copy: `/home/w1n5t0n/deployments/meml-aimmersive/a-immersive.html`.
|