367 lines
28 KiB
Markdown
367 lines
28 KiB
Markdown
# Manifold UI — Agent Onboarding
|
||
|
||
> **Purpose of this doc:** everything an agent needs to make a tweak or fix to the Manifold
|
||
> front-end without re-grepping the tree. Read this + the relevant section's source files; you
|
||
> should not need to read the 40KB design specs for routine work (they're pointers at the end).
|
||
>
|
||
> **Voice:** ground-truth map. If something here contradicts the code, the code wins — fix this doc
|
||
> in the same commit.
|
||
|
||
---
|
||
|
||
## 1. What Manifold is
|
||
|
||
A **convertible-mode React front-end** on the real, parity-tested NISPS ML+audio engine (same
|
||
`nisps/` C++ core that the firmware and the `playground/` use, compiled to WASM). It lets you drive
|
||
a neural net's mapping from a small input space (XY pad / gamepad / MIDI) to many outputs, shape
|
||
that mapping with interactive ML feedback ("explore & place" / dislike), and route the outputs to
|
||
several backends (built-in synth, MIDI, OSC, particles, VCV, MEMLNaut hardware).
|
||
|
||
- **Stack:** Vite + React 18 + TypeScript (strict). No CSS-in-JS — design-token CSS variables.
|
||
- **Lives at:** `manifold/` (beside `playground/`). Deploys to **https://meml.lnfinitemonkeys.org/next/**.
|
||
- **Synth is labelled "Powerful Synth Engine" / "Built-in Synth" — NEVER "C15".** British spelling.
|
||
|
||
---
|
||
|
||
## 2. Run / build / deploy / test
|
||
|
||
```bash
|
||
cd manifold
|
||
bun install
|
||
bun run dev # Vite dev server, port 5273 (COOP/COEP headers set)
|
||
bun run typecheck # tsc --noEmit ← run before every commit
|
||
bun run build # tsc --noEmit && vite build → dist/
|
||
bun run preview # serve dist/ on :4273 (honours COOP/COEP — needed by WASM/worklet)
|
||
bun run test:e2e # Playwright smoke (needs `bun run build` first; runs against preview)
|
||
```
|
||
|
||
- **Typecheck is the cheap gate.** It catches most regressions; run it after any edit.
|
||
- **E2E on the VPS** needs a non-snap node runner (bun is snap-confined and hides libs from
|
||
Chromium): `PLAYWRIGHT_BROWSERS_PATH=/home/w1n5t0n/snap/bun-js/87/.cache/ms-playwright node node_modules/.bin/playwright test`. Preview via bun is fine. The smoke spec (`tests/e2e/smoke.spec.ts`)
|
||
asserts: engine WASM loads, spine invariant (setInputs → outputs change), feedback runs, console
|
||
renders, **no "C15" in the bundle**, no console errors.
|
||
- **Deploy is automatic on push to GitHub `main`, but gated on CI** → webhook → waits for the
|
||
`CI` workflow to conclude `success` on that exact SHA → builds `manifold/` → rsyncs to the live
|
||
`/next/` subdir. A red or missing CI run aborts the deploy (fail-closed, 20 min timeout);
|
||
`MEML_SKIP_CI_GATE=1` bypasses it for an emergency hand-deploy. The gate lives VPS-side in
|
||
`~/.config/webhooks/meml-deploy.sh` (not in this repo) — added 2026-07-21 per the simplification
|
||
audit. See the `manifold-deploy-pipeline` memory for the full chain and gotchas (the
|
||
`cp index.html a-immersive.html` 403 workaround; git-ignored `bun.lock`).
|
||
- **`manifold/public/nisps.{js,wasm}` are tracked artifacts that ship to production** — the webhook
|
||
builds only `manifold/`, so vite copies whatever is committed. CI's *WASM freshness gate* runs
|
||
the parity harness against the committed artifact before rebuilding it, so a stale commit fails
|
||
loudly. Rebuild with `scripts/build-wasm.sh` and commit it whenever `nisps/` changes.
|
||
- **`?debug=1`** installs `window.__nisps` (synchronous probe for Playwright/console — see
|
||
`src/debug/probe.ts`). **`base: './'`** in `vite.config.ts` keeps asset URLs relative so one
|
||
`dist/` mounts at both `/` and `/next/`. **WASM URLs must resolve via `document.baseURI`**, never
|
||
hardcoded (see the assetUrl gotcha in §6).
|
||
|
||
---
|
||
|
||
## 3. The big picture — three layers
|
||
|
||
```
|
||
┌─────────────────────────────────────────────────────────────────┐
|
||
│ UI layer (React) src/console/ src/dock/ src/primitives/
|
||
│ ConsoleApp = spine of the UI: holds state, picks a Stage, │
|
||
│ renders the Dock. "Convertible" = swappable Stages. │
|
||
└───────────────┬─────────────────────────────────────────────────-┘
|
||
│ reads state/output versions (useSyncExternalStore), canvases read buffers imperatively
|
||
│ writes via engine.setInput / setParam / feedback.*
|
||
┌───────────────▼─────────────────────────────────────────────────┐
|
||
│ Engine layer (NO React) src/engine/ src/inputs/ src/feedback/ src/backends/
|
||
│ Spine = reactive store below React. Per-frame ML inference is │
|
||
│ eager + synchronous, OFF the render cycle. React watches state │
|
||
│ changes plus an opt-in 30 Hz output channel for DOM surfaces. │
|
||
└───────────────┬─────────────────────────────────────────────────┘
|
||
│ C ABI
|
||
┌───────────────▼─────────────────────────────────────────────────┐
|
||
│ WASM (nisps.wasm) — built from the C++ nisps/ core │
|
||
│ Loaded TWICE: main thread (ML inference + training + feedback) │
|
||
│ and AudioWorklet (audio DSP, separate instance). │
|
||
└──────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
**Golden rule of the reactivity model:** components subscribe to `engine.version()` via
|
||
`useEngineVersion()` and then read live `Float32Array`s **imperatively** (`engine.getOutputs()`,
|
||
`engine.routedOutput()`). They do **not** get outputs as React state — that was a deliberate perf
|
||
decision. Buffers are reused frame-to-frame; never assume a fresh array.
|
||
|
||
---
|
||
|
||
## 4. UI layer (where most tweaks happen)
|
||
|
||
### Entry & composition
|
||
- `src/App.tsx` → mounts `<EngineProvider>` (async WASM load, shows `Loading`) → `<ConsoleApp focus="composite" />`. Installs the `?debug=1` probe once the engine is live.
|
||
- `src/console/ConsoleApp.tsx` (**~1040 lines — the UI spine**). Holds nearly all UI state:
|
||
`focus`, `modeId`, `params` (`MFParam[]`), `pos` (2D input), feedback state
|
||
(`feedbackMode`/`soloMode`/`exploring`/`picking`/`anchorCount`), output config
|
||
(`outputMode`/`midiOutputId`/`oscUrl`/`vcvUrl`), and UI flags (`sandwich`, `split`, `stripPinned`,
|
||
`snapshots`, `markers`, `health`, `rev`). Builds the flat `ConsoleCtx` passed to the Dock + drawers.
|
||
|
||
### The Stages (one renders at a time)
|
||
**There is no `focus` axis any more.** The focus/altitude system (`AltitudeNav`, `SplitStage`,
|
||
`ReadoutStrip`, `InputMini`, `CompactAxis`) was deleted in the 2026-07 simplification audit —
|
||
Manifold ships a single "composite" altitude. Selection is now a plain three-way in `ConsoleApp`:
|
||
`sandwich` wins, else `outputMode==='particles'`, else CompositeStage.
|
||
|
||
| Stage | File | Renders when | What it is |
|
||
|---|---|---|---|
|
||
| CompositeStage | `CompositeStage.tsx` | **default / hero** | Draggable split-ratio; magnet-snaps to 0.14/0.33/0.5/0.66/0.86; collapses a side to a corner minimap at extremes. |
|
||
| SandwichStage | `SandwichStage.tsx` | `sandwich===true` (wins over the others) | Three-pane layout: `Manifold` input surface left, 3D parameter-landscape centre (input → MLP heatmap grid → outputs, drag to orbit), compact `OutputStage` right. |
|
||
| ParticleStage | `ParticleStage.tsx` | `outputMode==='particles'` | Flow-field visualiser (`flow-field.ts`, 400 particles batched into 32 contiguous colour paths per frame) + interactive output heatmap sliders with cursor tooltips + a larger, explicitly adjustable/repositionable circular Manifold pad (feedback markers + cursor trail); double-click anywhere in the stage enters whole-screen follow-mouse mode. |
|
||
|
||
`Manifold.tsx` and `OutputStage.tsx` are no longer top-level stages — they are panes composed by
|
||
CompositeStage/SandwichStage. `Manifold.tsx` is the full-bleed 2D input surface (canvas trail + pins
|
||
+ feedback markers; pointer → `onMove`; **double-click the input mark → follow-mouse mode**, a window
|
||
`pointermove` listener mapping the whole viewport onto this surface's space, Esc or a second
|
||
double-click exits). `OutputStage.tsx` is the output columns; click or drag a bar to temporarily
|
||
audition a clamped value (the next actual input change restores the live MLP), and it takes a `compact` prop
|
||
for the narrow pane.
|
||
|
||
- **Output modes** (the TOP dock selector, NOT the same axis as `focus`): `src/console/output-mode.ts`
|
||
defines `OUTPUT_MODES` = **particles** (default) / midi / osc / cv / synth / editor, each mapping to a
|
||
`BackendId`. `DEFAULT_OUTPUT_MODE='particles'`. `outputDisplayCount()` is the shared active-card
|
||
boundary for the stage, backend context, and routing rows. MIDI starts with eight cards; every
|
||
backend can add a card or delete any individual card in condensed and expanded Outputs drawers.
|
||
The `N outputs` chip always reports that same set. `MFParam.id` is semantic identity; array position
|
||
is not; backend-specific display names (including Particle System names) resolve from that identity,
|
||
so deleting a middle card cannot transfer its name or settings to the next card. Settings decides
|
||
whether edits retain spare network capacity or keep exact arity.
|
||
- `src/console/output-mode.ts`, `types.ts`, `model.ts` are the shared vocabulary — read these first
|
||
when touching anything cross-cutting:
|
||
- `types.ts`: `Focus`, `OutputMode`, `DrawerKey`, `DrawerDepth`, `FeedbackModeUI`, `SoloMode`,
|
||
`Pin`, `FeedbackMarker`, `Snapshot`, and **`ConsoleCtx`** (the flat context handed to the dock).
|
||
- `model.ts`: the instrument catalogue `MF_MODES`, `MFParam`, `ParamStatus`
|
||
(`off|fixed|live`), `ParamGroup`, plus `shapeValues()` (applies min/max/curve to raw engine
|
||
outputs) and `modeEngineId()`. (`seededGradient()` is GONE — it was the fabricated
|
||
gradient-health source, deleted in the 2026-07 sweep, S16.) **Schema-backed modes are DERIVED from the codegen
|
||
schemas in `src/modes/generated/`** (one-core-engine P5.2) — real param names/groups/count,
|
||
plus each mode's `ml` net shape (`MFMode.ml`) and schema `engine_id` (`MFMode.engineId`)
|
||
come from schema truth. A thin manifold OVERLAY (`SCHEMA_MODES` in model.ts) supplies only
|
||
label/glyph/ModeClass/input/ordering. Two schema-less manifold-only modes (`visualizer`,
|
||
`c15` placeholder) stay hand-written on `DEFAULT_MODE_ML`. Do NOT hand-edit
|
||
`src/modes/generated/` — it is codegen output (`bun run codegen/generate.ts`).
|
||
|
||
### The Dock (right-edge rail) — `src/console/Dock.tsx` + `Drawers.tsx`
|
||
- 48px right rail: **TOP** = mode selector (the 5 output modes, popover); **MIDDLE** = 5 drawer
|
||
icons, vertically centred macOS-dock style; **BOTTOM** = sandwich toggle.
|
||
- Five drawers (`DRAWERS` in `Drawers.tsx`, each has `.render(ctx, depth)` — condensed 360px panel
|
||
vs expanded 80vw×80vh modal):
|
||
- **learn** — feedback mode (explore-and-place / geometric-dislike) + solo mode + per-output arm,
|
||
plus a live model-architecture strip (inputs/outputs highlighted; hidden layers expand on click).
|
||
At `expanded` depth ONLY it also renders `TrainingHealth.tsx`: the real per-iteration loss
|
||
curve (`EngineApi.lossHistory()` ← `nisps_ml_loss_history` ← `MLPCore::loss_history`) plus
|
||
the per-layer weight-health table (`getLayerStats`). **`depth === 'expanded'` is Manifold's
|
||
advanced-surface flag** — there is no separate feature-flag mechanism, so put advanced
|
||
surface there rather than inventing one. The panel renders "no training run yet" when the
|
||
core has no history; it never synthesises a curve.
|
||
- **inputs** — enable/configure input sources (XY pad / MIDI / gamepad).
|
||
- **route** (label "Outputs") — per-output control matrix + per-backend config; centered prepend/append `+`
|
||
controls are available in both condensed and expanded views; the output rows fill the remaining drawer height
|
||
and scroll independently at both depths. The orange drawer tab uses `<` to expand and `>` to condense.
|
||
- **settings** — icon style (monochrome/colour), input-map shape (xy/joystick/rect/circular), corner radius.
|
||
- **help** — keymap pills + explainer link.
|
||
- `src/dock/` holds the output-routing internals used by the `route` drawer:
|
||
- `output-state.ts` — the per-output control model: `OutputControl` (state/muted/armed/min/max/
|
||
curve/fixedValue + backend specs `MidiCcSpec`/`OscSpec`/`VcvSpec`), `toOutputControl()`,
|
||
`buildArmMask()` (solo focus).
|
||
- `OutputControlRow.tsx` — one output card: editable name · M(mute) · S(solo/arm) · cycling
|
||
off/fixed/live status · dual-range · live value; the curve pad is a right-hand column and MIDI mode
|
||
adds CC#/channel fields. The delete `×` sits on the upper-right corner. **Writes eagerly to the shared
|
||
`MFParam` store via `onChange`.** The Outputs drawer has centered `+` controls that prepend before the first
|
||
visible card or append after the last.
|
||
- `OutputsBackendConfig.tsx` — preset bar (save/restore/rename/delete) + transport/device config (MIDI
|
||
port/templates, OSC path/range, VCV polarity); MIDI per-output fields live on `OutputControlRow`.
|
||
|
||
### Primitives — `src/primitives/` (barrel: `index.ts`)
|
||
Seven: `Button`, `Slider`, `PillToggle`, `Badge`, `Switch`, `XYPad`, `VirtualJoystick`. Dumb,
|
||
reusable, no engine knowledge. Side-effect import of `styles/primitives.css` styles the range
|
||
inputs. `Panel`/`StatusLine`/`ControlAxis`/`CurvePlot`/`Sparkline` were **deleted** in the 2026-07
|
||
sweep (L22, zero consumers) — don't cite them.
|
||
|
||
### Other shared UI files
|
||
- `shared-ui.tsx` — `MiniMeters` (read-only output bars). `AltitudeNav`/`CompactAxis` were deleted with the focus system.
|
||
- `icons.tsx` — monochrome inline-SVG icons (mode icons + drawer icons + `GLYPH_FALLBACK` for when monochrome is off).
|
||
- `VerdictCluster.tsx` — floating bottom-centre feedback UI (perturb ▽ / undo ↺ / commit △ + A/B);
|
||
labels and press timing adapt to feedback mode. Push away fires once on pointer-down and
|
||
never turns a hold into randomisation; the 600 ms long-press re-roll remains Explore-only.
|
||
- `CurvePad.tsx` — square canvas curve editor (vertical drag reshapes [0,1]; ~0.43 ≈ linear). Used in OutputEditor + OutputControlRow.
|
||
- `OutputEditor.tsx` — inline range/curve popup for a single output (hover/click on a bar), using
|
||
`DualRange.tsx` for the shared dual-thumb min/max control.
|
||
|
||
### Styling — `src/styles/`
|
||
CSS-variable design tokens, no CSS-in-JS. `tokens.css` `@import`s `tokens/{base,colors,fonts,
|
||
typography,spacing,effects}.css`. All vars on `:root`, consumed via `var(--name)`. Dark theme: deep
|
||
black bg, warm-orange `--accent` (#ff6a00), cyan `--accent-2` (#00ccff). Corner radius is driven by
|
||
a setting → `--r-*` tokens.
|
||
|
||
---
|
||
|
||
## 5. Engine / inputs / feedback / backends (touch when behaviour, not chrome, changes)
|
||
|
||
### Engine — `src/engine/` (no React except the two binding files)
|
||
- `spine.ts` — **the reactive store.** `setInput(x,y)`/`setInputs(arr)` drive raw input synchronously
|
||
through `WasmIML.processInput` (WASM input chain) → `processInto()` → `WasmIML.processOutput` (WASM
|
||
output chain) → backend, all off the render cycle, reusing buffers. Bumps a monotonic `version`.
|
||
`reprocess()` re-ticks the last input after a weight change (stores the full N-D vector so extra axes
|
||
survive). Pipeline config lives in `inputConfig_`/`outputConfig_` and is pushed C-side via
|
||
`setInputConfig`/`setOutputConfig` (state itself lives in the WASM pipeline handle since P4).
|
||
- `engine-api.ts` — **`EngineApi`, the framework-neutral facade** everything in the UI talks to:
|
||
`setInput/setInputs`, `getOutputs/routedOutput`, training (`addExample/train/trainAsync/evalLoss`),
|
||
weights (`getWeights/setWeights/process/randomise`), telemetry
|
||
(`architecture/weightCount/exampleCount/lossHistory/getLayerStats`), structural `subscribe/version`,
|
||
throttled `subscribeOutputs/outputVersion`, and `on`, plus nested `.feedback` and `.audio`
|
||
facades. **`lossHistory()` reads SPINE STATE, not the MLP handle** — an async train runs on the
|
||
worker's mirror net, so the main handle's own history is empty for those runs; both paths
|
||
publish to the spine.
|
||
- `EngineProvider.tsx` / `useEngine.ts` — the **only** React coupling. `useEngine()` returns the API
|
||
(null until WASM ready); `useEngineVersion()` watches structural/training state, while
|
||
`useEngineOutputVersion()` opts DOM consumers into the throttled 30 Hz output channel. Particle canvases
|
||
use neither output subscription nor React state for live values; they shape the reused buffer in rAF.
|
||
- `engine-host.ts` — main-thread audio wiring: AudioContext (user-gesture gated), fetch `nisps.wasm`,
|
||
register + feed the worklet.
|
||
- `wasm-iml.ts` (**~750 lines**) — the ML interface to `nisps.wasm`: one MLP handle, dataset, heap
|
||
buffers, feedback C-ABI bindings (`nisps_ml_feedback_*`), lazy training worker.
|
||
- `wasm-worker.ts` — off-thread training worker; returns weights, final loss, and the real
|
||
per-iteration loss curve read off its own mirror handle. `worklet/nisps-processor.ts` — the AudioWorklet's
|
||
separate WASM instance (raw `WebAssembly.instantiate`, no Emscripten glue; 128-sample blocks).
|
||
- **Input/output pipelines + curves live in the C++/WASM core (one-core-engine P4).** The input chain
|
||
(invert → deadzone → circular clamp → momentum-modulated zoom → centred power → EMA → momentum) and
|
||
output chain (global curve → per-output EMA → slew → freeze/mask) are `nisps/pipeline/*`, exposed via
|
||
`nisps_input_*` / `nisps_output_*` and driven by thin `WasmIML` wrappers (`processInput`,
|
||
`processOutput`, `setInputConfig`, `setOutputConfig`, `setOutputFreezeMask`, `reset*`). State lives
|
||
C++-side per pipeline handle. First 2 axes get the full pad pipeline; axes 2+ feed raw to the spine.
|
||
The old TS `input-pipeline.ts` / `output-pipeline.ts` / `curves.ts` are **deleted**; config TYPES are
|
||
`pipeline-types.ts`, the curve NAME↔id contract is `curve-catalog.ts`, and the curve MATHS is sampled
|
||
from the core via `EngineApi.curveApply` / `curveApplyBatch`.
|
||
- `dataset.ts` — JS-side example store + sample-weight modes (uniform/recency/spatial/combined).
|
||
`sink.ts` — `EngineSink` framework boundary. `types.ts` — C-ABI surface types.
|
||
- `exploration.ts` — `ExplorationController` adapter for the Jolt press + OU explore gestures
|
||
(Learning drawer). **As of one-core-engine P3 the maths lives in the shared C++/WASM core** — this
|
||
class is a thin driver that owns only the control-rate timers and calls `engine.explore.*`
|
||
(`joltPress`/`joltStep`/`joltRelease`/`joltActive` → `nisps_ml_jolt_*`; `setExploreIntensity`/
|
||
`exploreApply` → `nisps_ml_explore_*`). The held ~200 Hz driver calls `joltStep()` + `process()`;
|
||
the OU walk is the spine's inert-by-default `setOutputMorph` hook, now `exploreApply(routed)` (copy
|
||
into the WASM heap → advance+add in the core → copy back). The interim `jolt.ts` / `ou-explore.ts`
|
||
TS ports were deleted with the swap.
|
||
|
||
### Inputs — `src/inputs/`
|
||
- `input-layer.ts` — composition hub. One rAF loop polls sources, pulls all axes into a vector,
|
||
forwards N→engine. **`MAX_AXES = 32`** is the source-vector safety cap, not the model shape.
|
||
**Dedicated dimensions, NO mean-blending** — each active axis drives its own engine slot 1:1;
|
||
the ConsoleApp reshapes the runtime model when the selected 2/4-input capacity or active layout
|
||
requires it. Changing the layout uses the same persisted identity-aware I/O policy as output cards.
|
||
- `base-source.ts` + sources: `xy-pad-source.ts` (push, 2 axes), `gamepad-source.ts` (single=2 /
|
||
double=4 axes, deadzone 0.08), `midi-input-source.ts` (Web MIDI, batch CC-learn, multi-port).
|
||
- `useInputLayer.ts` — React binding; manages exclusive input mode + gamepad stick mode + MIDI
|
||
device/learn map; exposes `pushPad`, `sources`, `channelLayout`, etc.
|
||
- **I/O migration (P2.3, live):** the net is **runtime-shaped**. Normal Manifold modes boot at
|
||
2 inputs; the Inputs dock can select 4, and active layouts can grow the model further. `EngineApi.reshape({ inputSize, … })`
|
||
→ `WasmIML.reshape`. `engine/io-reshape.ts` owns the identity map: **Keep capacity** (default)
|
||
permutes weights/examples in place and reconstructs only when active I/O outgrows the net;
|
||
**Exact I/O** reconstructs whenever active arity changes. Existing examples either adapt by
|
||
deleting removed dimensions and inserting the saved neutral placeholders (input 0, output 0.5 by
|
||
default), or clear, according to the persistent Settings choice. Feedback replay/exploration state
|
||
resets because it has no stable-ID contract. The spine tolerates arity changes; the training worker
|
||
(`wasm-worker.ts`) carries the current dims in its train message and re-creates its mirror net to
|
||
match. The raw debug `window.__nisps.reshape(nIn)` remains a low-level reconstruct-and-clear call.
|
||
- **Per-mode net dims (P5.3):** switching INSTRUMENT mode reshapes the net to that mode's schema
|
||
hidden/output config plus Manifold's effective input arity via a `ConsoleApp` effect keyed on
|
||
`[engine, modeId, modelInputSize]`. No confirm modal (switching instrument is deliberate). Normal
|
||
modes boot at **2 inputs** (paf_synth → 2→[10,10,14]→33, weights 787); the Inputs dock's **4 inputs**
|
||
option is retained across mode switches. Audio-analysis keeps its schema-defined input feature count.
|
||
The reshape-offer effect reads the engine's CURRENT `inputSize`
|
||
live, so a mode switch that changes arity doesn't spuriously prompt (its baseline tracks axis
|
||
COUNT, unchanged by a pure dim change). Manifold-only modes use the same 2/4 input selector.
|
||
Debug seam for tests: under `?debug=1` ConsoleApp installs `window.__mf`
|
||
(`setMode`/`getModeId`/`paramCount`/`modeIds`/`getModelInputSize`/`setModelInputSize`) — the UI-level analogue of `__nisps`, since no
|
||
in-UI instrument picker exists yet (`ctx.modes`/`setModeId` are plumbed but unrendered).
|
||
Manifold passes `spread=0` for boot, mode-switch reshapes, direct re-rolls, explore-and-place
|
||
scratchpad rolls, and VCV-forwarded randomise gestures by default. Settings → Experimental
|
||
features → **Xavier / spread randomisation** restores the schema spread and reveals the centred
|
||
regime switch in the expanded Learning drawer.
|
||
|
||
### Feedback — `src/feedback/`
|
||
- `controller.ts` — `FeedbackController`, framework-neutral, owned by ConsoleApp. **As of one-core-
|
||
engine P3 it holds NO algorithm approximation** — a thin driver over the shared C++ core. Two modes:
|
||
**geometric-dislike** (default, Mode 1, "Push away") — `dislike()` calls `engine.feedback.
|
||
dislikeGeometric(heardVec)` (the k-NN centroid push-away in `nisps/ml/geo_push.hpp`; returns
|
||
FeedbackAction 14=push / 15=cold-start) then `process()`; `like()` runs the core's `thumbsUp`
|
||
(auto-stores the positive centroid in Avoid+Geometric) + `addExample` + `train`. **explore-and-
|
||
place** (Mode 2, positive-only) drives the core's snapshot/scratchpad/undo lifecycle; caller
|
||
accumulates anchors and trains on finalise with warm-start. Solo/arm via per-output mask.
|
||
- **The heard-vector rule:** the geometric dislike trains AWAY from the HEARD (post-pipeline, routed)
|
||
output — pass `engine.routedOutput()`, never the raw MLP output, or the cold-start MSE derivative is
|
||
zero (inert). The `EngineApi.feedback.thumbsDown` facade + the Mode-1 `dislike()` call site honour this.
|
||
- **Cold-start prompt:** a dislike with zero positives returns action 15 → ConsoleApp shows a one-time
|
||
"Like a few sounds first…" banner (dismissed on the next like or the dismiss button; rl-feedback §7).
|
||
- The interim `rng.ts` (`SeededRng`) and the `C++ GAP` approximation markers are **gone** — the seeded
|
||
RNG, geometric push, jolt, and OU all run in the core now.
|
||
|
||
### Backends — `src/backends/`
|
||
- `manager.ts` — `BackendManager`, the **single consumer of the engine spine** for output: subscribes,
|
||
reads `routedOutput()` each version bump, calls `active.send(routed)`. Gates audio (mutes synth when
|
||
a non-synth backend is active).
|
||
- Registered: `midi` (`midi-backend.ts`, CC out, ~20Hz throttle), `osc` (`osc-client.ts` → WebSocket
|
||
to the Deno bridge in `osc-bridge/bridge.ts`), `synth`/`cvgate` (`passthrough-backend.ts`),
|
||
`particles` (`particle-backend.ts`, no-op — the visualiser is a separate `flow-field.ts` consumer),
|
||
`vcv` (`vcv-backend.ts`, bidirectional). `mapping.ts`, `presets.ts`, `useBackendManager.ts` support
|
||
config. `backend.ts` defines the `OutputBackend` interface + `OutputMapping`.
|
||
|
||
### Misc
|
||
- `src/serial/memlnaut-serial.ts` — **STUB** Web Serial scaffold for the MEMLNaut Editor mode (protocol TODO). `EditorPanel.tsx` is its UI.
|
||
- `src/settings/settings-store.ts` — localStorage settings (`mf-settings`): icon style, input-map
|
||
shape, exact-vs-capacity network resizing, adapt-vs-clear examples + neutral new-dimension values,
|
||
corner radius, and the opt-in legacy Xavier/spread feature flag.
|
||
- `src/midi-devices/` — codegen'd external-synth device templates.
|
||
- `src/debug/probe.ts` — `window.__nisps` synchronous probe (engine/audio/bus). Some playground
|
||
feature-store methods are present-but-inert (not ported yet) to keep the surface stable.
|
||
|
||
---
|
||
|
||
## 6. Gotchas & non-obvious rules
|
||
|
||
1. **Outputs are read imperatively, not via React state.** Subscribe to `version`, then call
|
||
`getOutputs()`/`routedOutput()`. Buffers are reused — copy if you need to retain.
|
||
2. **Asset URLs must resolve against `document.baseURI`, not `location.origin`.** `base: './'` +
|
||
the `/next/` subpath mean a hardcoded `/nisps.wasm` 404s. The `assetUrl()` helper (in
|
||
`engine-host.ts`, `wasm-iml.ts`, `wasm-worker.ts`) handles this — use it.
|
||
3. **`nisps.js` is non-module Emscripten glue (no ES exports).** Workers/worklet fetch + indirect-eval
|
||
to install the global `createNispsModule`; the worklet uses raw `WebAssembly.instantiate`.
|
||
4. **WASM MLP is runtime-shaped (since one-core-engine P2).** `nisps_ml_create(in, out, hidden[])`
|
||
honours its dims (non-positive/null → the default `32→[10,14,18]→126` head, so pre-P2 callers are
|
||
bit-identical), and `nisps_ml_reshape` swaps in a warm-started net at new dims. Weights = 3148 at
|
||
the default shape; reshaping only the input arity shifts the first layer (e.g. →4 inputs = 2868).
|
||
The firmware MLP stays compile-time templated — only the WASM/browser build is dynamic.
|
||
5. **Curves + input/output pipelines are C++/WASM only (one-core-engine P4).** No TS curve/pipeline
|
||
maths remains; the browser samples `nisps/core/math.hpp` + `nisps/pipeline/*` via the WASM. The
|
||
golden test (`tests/pipeline-golden.test.ts`) drives the WASM chains against the frozen fixtures.
|
||
NOTE: `exp/log/sigmoid/cubic` deliberately changed to the firmware-exact maths at P4 (see
|
||
`tests/fixtures/README.md`); `linear/square/sqrt/centered_power` are unchanged. The 3 momentum input
|
||
configs carry a wide (`1e-2`) tolerance — proven-inherent f32 drift, documented in the test header.
|
||
6. **COOP/COEP headers are mandatory** for the WASM/worklet path — set in `vite.config.ts` for
|
||
dev+preview, and at nginx server scope in prod (inherited by `/next/`).
|
||
7. **Never emit "C15"** in code or bundle — the smoke test fails on it. Synth is "Powerful Synth
|
||
Engine"/"Built-in Synth".
|
||
8. **Two `MFParam` writers** (stage controls AND dock rows) share one store — edits in either re-render both.
|
||
|
||
---
|
||
|
||
## 7. Deeper references (only when this doc isn't enough)
|
||
|
||
All in `docs/specs/` (at the repo root, not under `manifold/`), with subdirectories:
|
||
- `plans/BUILD-PLAN.md` — locked decisions + the 12-step build sequence + spec pointers (the resume anchor).
|
||
- `engine-architecture.md` — full engine/spine/WASM design.
|
||
- `dock-spec.md` — dock + drawers spec. `inputs-spec.md` — mixed-input design. `backends-spec.md` — backends.
|
||
- `docs/adr/rl-feedback-design.md` + `plans/feedback-modes-port-spec.md` (executed) + `recon/findings-feedback-behaviour.md` — feedback modes (Mode 1/Mode 2).
|
||
- `_archive/aimmersive-clone-spec.md` / `recon/playground-2026.md` — the a-immersive feature parity target (archived reference).
|
||
- `src/backends/README.md` — backend wiring notes.
|
||
|
||
**Memories** (auto-loaded): `manifold-build` (status + locked decisions), `manifold-mixed-inputs`
|
||
(N-D input design), `manifold-deploy-pipeline` (the auto-deploy chain), `single-double-joystick-toggle`.
|
||
|
||
---
|
||
|
||
*Keep this in sync with the code — update it in the same commit as any change to UI structure, the
|
||
engine spine, the dock/drawers, or the build/deploy commands.*
|