2026-06-28 22:20:58 +02:00
# Manifold UI — Agent Onboarding
> **Purpose of this doc:** everything an agent needs to make a tweak or fix to the Manifold
> front-end without re-grepping the tree. Read this + the relevant section's source files; you
> should not need to read the 40KB design specs for routine work (they're pointers at the end).
>
> **Voice:** ground-truth map. If something here contradicts the code, the code wins — fix this doc
> in the same commit.
---
## 1. What Manifold is
A **convertible-mode React front-end** on the real, parity-tested NISPS ML+audio engine (same
`nisps/` C++ core that the firmware and the `playground/` use, compiled to WASM). It lets you drive
a neural net's mapping from a small input space (XY pad / gamepad / MIDI) to many outputs, shape
that mapping with interactive ML feedback ("explore & place" / dislike), and route the outputs to
several backends (built-in synth, MIDI, OSC, particles, VCV, MEMLNaut hardware).
- **Stack:** Vite + React 18 + TypeScript (strict). No CSS-in-JS — design-token CSS variables.
- **Lives at:** `manifold/` (beside `playground/` ). Deploys to **https://meml.lnfinitemonkeys.org/next/** .
- **Synth is labelled "Powerful Synth Engine" / "Built-in Synth" — NEVER "C15".** British spelling.
---
## 2. Run / build / deploy / test
```bash
cd manifold
bun install
bun run dev # Vite dev server, port 5273 (COOP/COEP headers set)
bun run typecheck # tsc --noEmit ← run before every commit
bun run build # tsc --noEmit & & vite build → dist/
bun run preview # serve dist/ on :4273 (honours COOP/COEP — needed by WASM/worklet)
bun run test:e2e # Playwright smoke (needs `bun run build` first; runs against preview)
```
- **Typecheck is the cheap gate.** It catches most regressions; run it after any edit.
- **E2E on the VPS** needs a non-snap node runner (bun is snap-confined and hides libs from
Chromium): `PLAYWRIGHT_BROWSERS_PATH=/home/w1n5t0n/snap/bun-js/87/.cache/ms-playwright node node_modules/.bin/playwright test` . Preview via bun is fine. The smoke spec (`tests/e2e/smoke.spec.ts`)
asserts: engine WASM loads, spine invariant (setInputs → outputs change), feedback runs, console
renders, **no "C15" in the bundle** , no console errors. `shot.spec.ts` takes screenshots.
- **Deploy is automatic on push to GitHub `main` ** → webhook → builds `manifold/` → rsyncs to the
live `/next/` subdir. See the `manifold-deploy-pipeline` memory for the full chain and gotchas
(the `cp index.html a-immersive.html` 403 workaround; git-ignored `bun.lock` ).
- **`?debug=1`** installs `window.__nisps` (synchronous probe for Playwright/console — see
`src/debug/probe.ts` ). ** `base: './'` ** in `vite.config.ts` keeps asset URLs relative so one
`dist/` mounts at both `/` and `/next/` . **WASM URLs must resolve via `document.baseURI`** , never
hardcoded (see the assetUrl gotcha in §6).
---
## 3. The big picture — three layers
```
┌─────────────────────────────────────────────────────────────────┐
│ UI layer (React) src/console/ src/dock/ src/primitives/
│ ConsoleApp = spine of the UI: holds state, picks a Stage, │
│ renders the Dock. "Convertible" = swappable Stages. │
└───────────────┬─────────────────────────────────────────────────-┘
│ reads engine.version (useSyncExternalStore), reads buffers imperatively
│ writes via engine.setInput / setParam / feedback.*
┌───────────────▼─────────────────────────────────────────────────┐
│ Engine layer (NO React) src/engine/ src/inputs/ src/feedback/ src/backends/
│ Spine = reactive store below React. Per-frame ML inference is │
│ eager + synchronous, OFF the render cycle. React only watches │
│ a monotonic version counter. │
└───────────────┬─────────────────────────────────────────────────┘
│ C ABI
┌───────────────▼─────────────────────────────────────────────────┐
│ WASM (nisps.wasm) — built from the C++ nisps/ core │
│ Loaded TWICE: main thread (ML inference + training + feedback) │
│ and AudioWorklet (audio DSP, separate instance). │
└──────────────────────────────────────────────────────────────────┘
```
**Golden rule of the reactivity model:** components subscribe to `engine.version()` via
`useEngineVersion()` and then read live `Float32Array` s **imperatively** (`engine.getOutputs()`,
`engine.routedOutput()` ). They do **not** get outputs as React state — that was a deliberate perf
decision. Buffers are reused frame-to-frame; never assume a fresh array.
---
## 4. UI layer (where most tweaks happen)
### Entry & composition
- `src/App.tsx` → mounts `<EngineProvider>` (async WASM load, shows `Loading` ) → `<ConsoleApp focus="composite" />` . Installs the `?debug=1` probe once the engine is live.
- `src/console/ConsoleApp.tsx` (**~1040 lines — the UI spine**). Holds nearly all UI state:
`focus` , `modeId` , `params` (`MFParam[]`), `pos` (2D input), feedback state
(`feedbackMode`/`soloMode`/`exploring`/`picking`/`anchorCount`), output config
(`outputMode`/`midiOutputId`/`oscUrl`/`vcvUrl`), and UI flags (`sandwich`, `split` , `stripPinned` ,
`snapshots` , `markers` , `health` , `rev` ). Builds the flat `ConsoleCtx` passed to the Dock + drawers.
### The "convertible" Stages (one renders at a time, chosen by `focus` + `outputMode`)
| Stage | File | Renders when | What it is |
|---|---|---|---|
2026-06-29 00:45:45 +02:00
| Manifold | `Manifold.tsx` | `focus==='in'` (default input view) | Full-bleed 2D input surface; canvas trail + pins + feedback markers; pointer → `onMove` . **Double-click the input mark → follow-mouse mode** (self-contained state; a window `pointermove` listener maps the whole viewport onto this surface's space so the knob tracks the cursor across the entire UI; Esc / second double-click exits). |
2026-06-28 22:20:58 +02:00
| OutputStage | `OutputStage.tsx` | `focus==='out'` | Full-bleed output columns; drag bars set value; `InputMini` docked in a corner. |
| SplitStage | `SplitStage.tsx` | `focus==='split'` | Manifold left, OutputStage right, equal width. |
| CompositeStage | `CompositeStage.tsx` | `focus==='composite'` (**app default / hero**) | Draggable split-ratio; magnet-snaps to 0.14/0.33/0.5/0.66/0.86; collapses a side to a corner minimap at extremes. |
| SandwichStage | `SandwichStage.tsx` | `sandwich===true` (overrides) | 3D parameter-landscape view (input → MLP heatmap grid → outputs); drag to orbit. |
| ParticleStage | `ParticleStage.tsx` | `outputMode==='particles'` | Flow-field visualiser (`flow-field.ts`, 400-particle Canvas2D port) + macro-axis bar + corner joystick. |
- **Output modes** (the TOP dock selector, NOT the same axis as `focus` ): `src/console/output-mode.ts`
defines `OUTPUT_MODES` = **particles** (default) / midi / osc / synth / editor, each mapping to a
`BackendId` . `DEFAULT_OUTPUT_MODE='particles'` .
- `src/console/output-mode.ts` , `types.ts` , `model.ts` are the shared vocabulary — read these first
when touching anything cross-cutting:
- `types.ts` : `Focus` , `OutputMode` , `DrawerKey` , `DrawerDepth` , `FeedbackModeUI` , `SoloMode` ,
`Pin` , `FeedbackMarker` , `Snapshot` , and ** `ConsoleCtx` ** (the flat context handed to the dock).
feat(manifold): P5.2/P5.3 — derive MF_MODES from schema truth + per-mode engine dims
Schema-backed modes in console/model.ts are now DERIVED from the codegen
schemas in src/modes/generated/ (source of truth): real param names/groups/
count, plus each mode's ml net shape (MFMode.ml) and schema engine_id. A thin
manifold OVERLAY supplies only label/glyph/ModeClass/input/ordering. New
browser-viable modes xiasri + slp_workshop get derived entries; schema-less
visualizer + c15 stay hand-written on DEFAULT_MODE_ML. Schema min/max/default/
label/curve surface as engine-unit metadata (schemaMin/... on MFParam) without
touching the 0..1 routing semantics.
Switching instrument mode reshapes the runtime-shaped WASM net to the mode's
schema ml config (ConsoleApp effect keyed on [engine, modeId]; no confirm
modal). Boot lands paf_synth dims (4->[10,10,14]->33) once WASM is ready. The
P2.3 axis-count reshape offer still reads the engine's live inputSize and does
not spuriously prompt on a mode switch.
Adds schema-modes.spec.ts (P5 gate): drives switches via a new window.__mf
debug seam and asserts describe() dims, getWeights count, output length/bounds,
and UI param count FROM the imported schemas; spot-checks trainAsync after a
switch. Updates reshape/probe-api/geo-dislike specs to assert from the boot
mode schema instead of the retired fixed 32/126 shape.
All gates green: typecheck, unit (9), build, e2e (33).
2026-07-18 12:45:06 +02:00
- `model.ts` : the instrument catalogue `MF_MODES` , `MFParam` , `ParamStatus`
2026-06-28 22:20:58 +02:00
(`off|fixed|live`), `ParamGroup` , plus `shapeValues()` (applies min/max/curve to raw engine
feat(manifold): P5.2/P5.3 — derive MF_MODES from schema truth + per-mode engine dims
Schema-backed modes in console/model.ts are now DERIVED from the codegen
schemas in src/modes/generated/ (source of truth): real param names/groups/
count, plus each mode's ml net shape (MFMode.ml) and schema engine_id. A thin
manifold OVERLAY supplies only label/glyph/ModeClass/input/ordering. New
browser-viable modes xiasri + slp_workshop get derived entries; schema-less
visualizer + c15 stay hand-written on DEFAULT_MODE_ML. Schema min/max/default/
label/curve surface as engine-unit metadata (schemaMin/... on MFParam) without
touching the 0..1 routing semantics.
Switching instrument mode reshapes the runtime-shaped WASM net to the mode's
schema ml config (ConsoleApp effect keyed on [engine, modeId]; no confirm
modal). Boot lands paf_synth dims (4->[10,10,14]->33) once WASM is ready. The
P2.3 axis-count reshape offer still reads the engine's live inputSize and does
not spuriously prompt on a mode switch.
Adds schema-modes.spec.ts (P5 gate): drives switches via a new window.__mf
debug seam and asserts describe() dims, getWeights count, output length/bounds,
and UI param count FROM the imported schemas; spot-checks trainAsync after a
switch. Updates reshape/probe-api/geo-dislike specs to assert from the boot
mode schema instead of the retired fixed 32/126 shape.
All gates green: typecheck, unit (9), build, e2e (33).
2026-07-18 12:45:06 +02:00
outputs) and `seededGradient()` . **Schema-backed modes are DERIVED from the codegen
schemas in `src/modes/generated/` ** (one-core-engine P5.2) — real param names/groups/count,
plus each mode's `ml` net shape (`MFMode.ml`) and schema `engine_id` (`MFMode.engineId`)
come from schema truth. A thin manifold OVERLAY (`SCHEMA_MODES` in model.ts) supplies only
label/glyph/ModeClass/input/ordering. Two schema-less manifold-only modes (`visualizer`,
`c15` placeholder) stay hand-written on `DEFAULT_MODE_ML` . Do NOT hand-edit
`src/modes/generated/` — it is codegen output (`bun run codegen/generate.ts`).
2026-06-28 22:20:58 +02:00
### The Dock (right-edge rail) — `src/console/Dock.tsx` + `Drawers.tsx`
- 48px right rail: **TOP** = mode selector (the 5 output modes, popover); **MIDDLE** = 5 drawer
icons, vertically centred macOS-dock style; **BOTTOM** = sandwich toggle.
- Five drawers (`DRAWERS` in `Drawers.tsx` , each has `.render(ctx, depth)` — condensed 360px panel
vs expanded 80vw× 80vh modal):
- **learn** — feedback mode (explore-and-place / geometric-dislike) + solo mode + per-output arm.
- **inputs** — enable/configure input sources (XY pad / MIDI / gamepad).
- **route** (label "Outputs") — per-output control matrix + per-backend config.
- **settings** — icon style (monochrome/colour), input-map shape (xy/joystick/rect/circular), corner radius.
- **help** — keymap pills + loop explanation.
- `src/dock/` holds the output-routing internals used by the `route` drawer:
- `output-state.ts` — the per-output control model: `OutputControl` (state/muted/armed/min/max/
curve/fixedValue + backend specs `MidiCcSpec` /`OscSpec`/`VcvSpec`), `toOutputControl()` ,
`buildArmMask()` (solo focus).
- `OutputControlRow.tsx` — one output row: name · M(mute) · S(solo/arm) · off|fixed|live · dual-range
· curve pad · live value. **Writes eagerly to the shared `MFParam` store via `onChange`.**
- `OutputsBackendConfig.tsx` — preset bar (save/restore/rename/delete) + per-backend config (MIDI
CC#/channel, OSC path/range, VCV polarity).
- `BackendAdvanced.tsx` — full-depth modal version of the same editors.
### Primitives — `src/primitives/` (barrel: `index.ts`)
`Button` , `Slider` , `PillToggle` , `Panel` , `Badge` , `Switch` , `StatusLine` , `XYPad` ,
`VirtualJoystick` , `ControlAxis` , `CurvePlot` , `Sparkline` . Dumb, reusable, no engine knowledge.
Side-effect import of `styles/primitives.css` styles the range inputs.
### Other shared UI files
- `shared-ui.tsx` — `AltitudeNav` (focus IN/DUAL/OUT/FLEX switcher) + `MiniMeters` (read-only output bars).
- `icons.tsx` — monochrome inline-SVG icons (mode icons + drawer icons + `GLYPH_FALLBACK` for when monochrome is off).
- `ReadoutStrip.tsx` — thin horizontal heatmap strip (pinned, `focus==='in'` ); same per-output control as OutputStage.
- `VerdictCluster.tsx` — floating bottom-centre feedback UI (perturb ▽ / undo ↺ / commit △ + A/B); labels adapt to feedback mode.
- `CurvePad.tsx` — square canvas curve editor (vertical drag reshapes [0,1]; ~0.43 ≈ linear). Used in OutputEditor + OutputControlRow.
- `InputMini.tsx` — compact XYPad/joystick docked in a corner when input is demoted.
- `OutputEditor.tsx` — inline min/max/curve popup for a single output (hover/click on a bar).
### Styling — `src/styles/`
CSS-variable design tokens, no CSS-in-JS. `tokens.css` `@import` s `tokens/{base,colors,fonts,
typography,spacing,effects}.css`. All vars on `:root` , consumed via `var(--name)` . Dark theme: deep
black bg, warm-orange `--accent` (#ff6a00), cyan `--accent-2` (#00ccff). Corner radius is driven by
a setting → `--r-*` tokens.
---
## 5. Engine / inputs / feedback / backends (touch when behaviour, not chrome, changes)
### Engine — `src/engine/` (no React except the two binding files)
- `spine.ts` — **the reactive store.** `setInput(x,y)` /`setInputs(arr)` drive raw input synchronously
feat(manifold)!: route input/output pipelines + curves through the WASM core (P4.3/P4.4)
One-core-engine P4.3/P4.4: the input/output pipeline processing and the curve
catalog now live in the C++/WASM core (nisps/pipeline/*, nisps/core/math.hpp).
The TS ports are deleted and the browser drives the WASM chains.
Engine:
- WasmIML owns a nisps_pipeline_create handle + bridge buffers and exposes
setInputConfig (TS InputConfig → 15-float wire), processInput, resetInput,
setOutputConfig (Infinity slew → 0), setOutputFreezeMask, processOutput
(in place), resetOutput, curveApply, curveApplyBatch (chunked). Handle +
buffers created in init_, freed in dispose, output-sized buffers realloc'd
on reshape.
- Spine routes setInputs through iml.processInput/processOutput (state lives
C++-side); config source-of-truth stays TS-side and is pushed on attach /
setInputConfig / setOutputConfig. Preserves ?debug=1 fixed-dt determinism
(same dt fed to the WASM calls). EngineApi gains setInputConfig/
setOutputConfig/curveApply/curveApplyBatch.
- New types-only modules: pipeline-types.ts (InputConfig/OutputConfig +
defaults + wire int mappers) and curve-catalog.ts (CurveName + name→id).
types.ts declares the pipeline/curve C ABI. engine barrel updated.
- DELETED src/engine/{input-pipeline,output-pipeline,curves}.ts.
Tests (P4.4 gate — recorded-gesture regression):
- pipeline-golden.test.ts now loads the built WASM (indirect-eval shim,
tests/wasm-load.ts) and drives the frozen gesture/output fixtures through the
C++ chains, honouring the per-event dt clock contract. Tolerance 1e-5
(non-momentum drift <5e-7). The 3 momentum configs carry 1e-2: proven-inherent
f32 drift (a byte-faithful f32 port of the exact original algorithm matches
the WASM to <6e-8 while both diverge from the f64 capture by ~7-9e-3), NOT a
core bug.
- curves-golden.json: linear/square/sqrt/centered_power kept as the original
f64 captures (C++ matches within <3e-8); exp/log/sigmoid/cubic RE-BASELINED
from the WASM (deliberate switch to firmware-exact k=1 exp/log, slope-6
sigmoid, true cubic x^3). Provenance recorded in-file.
- _generate.ts rebuilt as the WASM curve re-baseline tool; pipeline-golden-lib
trimmed to pure data builders.
Docs: fixtures/README.md + manifold/ONBOARDING.md updated.
Gates: typecheck, bun test (9), vite build, playwright e2e (27) all green.
2026-07-18 12:21:35 +02:00
through `WasmIML.processInput` (WASM input chain) → `processInto()` → `WasmIML.processOutput` (WASM
output chain) → backend, all off the render cycle, reusing buffers. Bumps a monotonic `version` .
`reprocess()` re-ticks the last input after a weight change (stores the full N-D vector so extra axes
survive). Pipeline config lives in `inputConfig_` /`outputConfig_` and is pushed C-side via
`setInputConfig` /`setOutputConfig` (state itself lives in the WASM pipeline handle since P4).
2026-06-28 22:20:58 +02:00
- `engine-api.ts` — ** `EngineApi` , the framework-neutral facade** everything in the UI talks to:
`setInput/setInputs` , `getOutputs/routedOutput` , training (`addExample/train/trainAsync/evalLoss`),
weights (`getWeights/setWeights/process/randomise`), `subscribe/version/on` , plus nested
`.feedback` and `.audio` facades.
- `EngineProvider.tsx` / `useEngine.ts` — the **only** React coupling. `useEngine()` returns the API
(null until WASM ready); `useEngineVersion()` = `useSyncExternalStore(subscribe, version)` .
- `engine-host.ts` — main-thread audio wiring: AudioContext (user-gesture gated), fetch `nisps.wasm` ,
register + feed the worklet.
- `wasm-iml.ts` (**~750 lines**) — the ML interface to `nisps.wasm` : one MLP handle, dataset, heap
buffers, feedback C-ABI bindings (`nisps_ml_feedback_*`), lazy training worker.
- `wasm-worker.ts` — off-thread training worker. `worklet/nisps-processor.ts` — the AudioWorklet's
separate WASM instance (raw `WebAssembly.instantiate` , no Emscripten glue; 128-sample blocks).
feat(manifold)!: route input/output pipelines + curves through the WASM core (P4.3/P4.4)
One-core-engine P4.3/P4.4: the input/output pipeline processing and the curve
catalog now live in the C++/WASM core (nisps/pipeline/*, nisps/core/math.hpp).
The TS ports are deleted and the browser drives the WASM chains.
Engine:
- WasmIML owns a nisps_pipeline_create handle + bridge buffers and exposes
setInputConfig (TS InputConfig → 15-float wire), processInput, resetInput,
setOutputConfig (Infinity slew → 0), setOutputFreezeMask, processOutput
(in place), resetOutput, curveApply, curveApplyBatch (chunked). Handle +
buffers created in init_, freed in dispose, output-sized buffers realloc'd
on reshape.
- Spine routes setInputs through iml.processInput/processOutput (state lives
C++-side); config source-of-truth stays TS-side and is pushed on attach /
setInputConfig / setOutputConfig. Preserves ?debug=1 fixed-dt determinism
(same dt fed to the WASM calls). EngineApi gains setInputConfig/
setOutputConfig/curveApply/curveApplyBatch.
- New types-only modules: pipeline-types.ts (InputConfig/OutputConfig +
defaults + wire int mappers) and curve-catalog.ts (CurveName + name→id).
types.ts declares the pipeline/curve C ABI. engine barrel updated.
- DELETED src/engine/{input-pipeline,output-pipeline,curves}.ts.
Tests (P4.4 gate — recorded-gesture regression):
- pipeline-golden.test.ts now loads the built WASM (indirect-eval shim,
tests/wasm-load.ts) and drives the frozen gesture/output fixtures through the
C++ chains, honouring the per-event dt clock contract. Tolerance 1e-5
(non-momentum drift <5e-7). The 3 momentum configs carry 1e-2: proven-inherent
f32 drift (a byte-faithful f32 port of the exact original algorithm matches
the WASM to <6e-8 while both diverge from the f64 capture by ~7-9e-3), NOT a
core bug.
- curves-golden.json: linear/square/sqrt/centered_power kept as the original
f64 captures (C++ matches within <3e-8); exp/log/sigmoid/cubic RE-BASELINED
from the WASM (deliberate switch to firmware-exact k=1 exp/log, slope-6
sigmoid, true cubic x^3). Provenance recorded in-file.
- _generate.ts rebuilt as the WASM curve re-baseline tool; pipeline-golden-lib
trimmed to pure data builders.
Docs: fixtures/README.md + manifold/ONBOARDING.md updated.
Gates: typecheck, bun test (9), vite build, playwright e2e (27) all green.
2026-07-18 12:21:35 +02:00
- **Input/output pipelines + curves live in the C++/WASM core (one-core-engine P4).** The input chain
(invert → deadzone → circular clamp → momentum-modulated zoom → centred power → EMA → momentum) and
output chain (global curve → per-output EMA → slew → freeze/mask) are `nisps/pipeline/*` , exposed via
`nisps_input_*` / `nisps_output_*` and driven by thin `WasmIML` wrappers (`processInput`,
`processOutput` , `setInputConfig` , `setOutputConfig` , `setOutputFreezeMask` , `reset*` ). State lives
C++-side per pipeline handle. First 2 axes get the full pad pipeline; axes 2+ feed raw to the spine.
The old TS `input-pipeline.ts` / `output-pipeline.ts` / `curves.ts` are **deleted** ; config TYPES are
`pipeline-types.ts` , the curve NAME↔id contract is `curve-catalog.ts` , and the curve MATHS is sampled
from the core via `EngineApi.curveApply` / `curveApplyBatch` .
2026-06-28 22:20:58 +02:00
- `dataset.ts` — JS-side example store + sample-weight modes (uniform/recency/spatial/combined).
feat(manifold)!: route input/output pipelines + curves through the WASM core (P4.3/P4.4)
One-core-engine P4.3/P4.4: the input/output pipeline processing and the curve
catalog now live in the C++/WASM core (nisps/pipeline/*, nisps/core/math.hpp).
The TS ports are deleted and the browser drives the WASM chains.
Engine:
- WasmIML owns a nisps_pipeline_create handle + bridge buffers and exposes
setInputConfig (TS InputConfig → 15-float wire), processInput, resetInput,
setOutputConfig (Infinity slew → 0), setOutputFreezeMask, processOutput
(in place), resetOutput, curveApply, curveApplyBatch (chunked). Handle +
buffers created in init_, freed in dispose, output-sized buffers realloc'd
on reshape.
- Spine routes setInputs through iml.processInput/processOutput (state lives
C++-side); config source-of-truth stays TS-side and is pushed on attach /
setInputConfig / setOutputConfig. Preserves ?debug=1 fixed-dt determinism
(same dt fed to the WASM calls). EngineApi gains setInputConfig/
setOutputConfig/curveApply/curveApplyBatch.
- New types-only modules: pipeline-types.ts (InputConfig/OutputConfig +
defaults + wire int mappers) and curve-catalog.ts (CurveName + name→id).
types.ts declares the pipeline/curve C ABI. engine barrel updated.
- DELETED src/engine/{input-pipeline,output-pipeline,curves}.ts.
Tests (P4.4 gate — recorded-gesture regression):
- pipeline-golden.test.ts now loads the built WASM (indirect-eval shim,
tests/wasm-load.ts) and drives the frozen gesture/output fixtures through the
C++ chains, honouring the per-event dt clock contract. Tolerance 1e-5
(non-momentum drift <5e-7). The 3 momentum configs carry 1e-2: proven-inherent
f32 drift (a byte-faithful f32 port of the exact original algorithm matches
the WASM to <6e-8 while both diverge from the f64 capture by ~7-9e-3), NOT a
core bug.
- curves-golden.json: linear/square/sqrt/centered_power kept as the original
f64 captures (C++ matches within <3e-8); exp/log/sigmoid/cubic RE-BASELINED
from the WASM (deliberate switch to firmware-exact k=1 exp/log, slope-6
sigmoid, true cubic x^3). Provenance recorded in-file.
- _generate.ts rebuilt as the WASM curve re-baseline tool; pipeline-golden-lib
trimmed to pure data builders.
Docs: fixtures/README.md + manifold/ONBOARDING.md updated.
Gates: typecheck, bun test (9), vite build, playwright e2e (27) all green.
2026-07-18 12:21:35 +02:00
`sink.ts` — `EngineSink` framework boundary. `types.ts` — C-ABI surface types.
2026-07-13 23:30:28 +02:00
- `exploration.ts` — `ExplorationController` adapter for the Jolt press + OU explore gestures
feat(manifold): geometric dislike + jolt/OU via shared core (one-core-engine P3)
TS half of P3.1/P3.2/P3.3: wire the new WASM exports and delete the TS
approximations now that geometric-dislike, jolt, OU, and the seeded RNG live
in the C++/WASM core.
- types.ts/wasm-iml.ts: bind + wrap dislike_geometric, store_positive,
positive/negative_count, set_avoid_style, jolt_press/step/release/active/
lr_scale/tick_lr_ramp, explore_intensity/get/apply. Weight-mutating wrappers
republish weights (version bump); explore_apply reuses feedbackBuf.
- engine-api: extend .feedback (dislikeGeometric/storePositive/counts/
setAvoidStyle) + new .explore facade. thumbsDown now passes the HEARD
(routed) vector, not the raw MLP output (raw == net output => inert cold-start).
- feedback/controller.ts: delete dislikes[] + applyDislikeBias + both C++ GAP
blocks + the SeededRng field; dislike() -> core dislikeGeometric (returns
action, 15 => cold-start prompt); like() feeds the centroid via the core
thumbsUp; getState() exposes positive/negative counts. Delete feedback/rng.ts.
- engine/exploration.ts: execute the P3 SWAP POINT -> drive engine.explore.*;
delete engine/jolt.ts + ou-explore.ts. UI surface unchanged.
- ConsoleApp: one-time cold-start banner (British spelling), routed heard vector
at the dislike call site.
- App.tsx/spine.ts: under ?debug=1 pin a fixed seed + fixed per-tick dt so the
probe/e2e are deterministic (production keeps time-seeded, real-time dt).
- tests: new geo-dislike.spec.ts; probe gains dislikeGeometric/storePositive/
feedbackCounts/setAvoidStyle; the thumbsDown probe test now drives a real
distinct-heard-vector dislike (bare thumbsDown on the net's own output is
correctly inert under the geometric core). 27 e2e + 9 unit green.
Docs: manifold/ONBOARDING.md engine+feedback sections synced.
2026-07-14 04:54:27 +02:00
(Learning drawer). **As of one-core-engine P3 the maths lives in the shared C++/WASM core** — this
class is a thin driver that owns only the control-rate timers and calls `engine.explore.*`
(`joltPress`/`joltStep`/`joltRelease`/`joltActive` → `nisps_ml_jolt_*` ; `setExploreIntensity` /
`exploreApply` → `nisps_ml_explore_*` ). The held ~200 Hz driver calls `joltStep()` + `process()` ;
the OU walk is the spine's inert-by-default `setOutputMorph` hook, now `exploreApply(routed)` (copy
into the WASM heap → advance+add in the core → copy back). The interim `jolt.ts` / `ou-explore.ts`
TS ports were deleted with the swap.
2026-06-28 22:20:58 +02:00
### Inputs — `src/inputs/`
- `input-layer.ts` — composition hub. One rAF loop polls sources, pulls all axes into a vector,
forwards N→engine. ** `MAX_AXES = 32` ** (WASM net over-provisioned to 32 inputs). **Dedicated
dimensions, NO mean-blending** — each active axis drives its own engine slot 1:1; unused slots
zero-padded (inert). Changing axis count requires a **net reset** (UI confirm modal).
- `base-source.ts` + sources: `xy-pad-source.ts` (push, 2 axes), `gamepad-source.ts` (single=2 /
double=4 axes, deadzone 0.08), `midi-input-source.ts` (Web MIDI, batch CC-learn, multi-port).
- `useInputLayer.ts` — React binding; manages exclusive input mode + gamepad stick mode + MIDI
device/learn map; exposes `pushPad` , `sources` , `channelLayout` , etc.
feat(manifold): runtime-shaped net reshape with confirm modal (P2.3)
Wire the runtime-shaped WASM MLP (one-core-engine P2) through the manifold:
- WasmIML.reshape(dims, spread): calls nisps_ml_reshape, re-describes the
instance, reallocates every dim-dependent heap buffer, refreshes weightCount,
clears the TS Dataset mirror (C-side resets), drops the stale training worker,
and pushes the new shape through the sink.
- EngineApi.reshape exposes it and re-ticks the spine so outputs/audio reflect
the new net. Spine already tolerates the arity change (buffers resize, version
bumps); documented.
- Training worker protocol carries the current hidden dims; the worker
ensureNet()s its mirror net to match after a reshape.
- ConsoleApp offers the reshape behind ReshapeModal on an active-layout CHANGE
(never on load; default 32-input over-provisioned head + zero-padding
preserved when declined). British copy, reset-on-reshape.
- Drawers: delete the stale even/odd blending note; honest dedicated-dimensions
line + net-arity chip.
- Probe: __nisps.reshape(nIn) / .describe(); e2e reshape.spec (default 32/126,
reshape to 4, describe reports 4, bounded outputs, weight count 3148→2868,
spine still propagates). All 25 e2e pass (20 existing + 5 new).
- ONBOARDING: refresh the reshape status + stale hardwired-arity gotcha.
2026-07-14 03:54:12 +02:00
- **Reshape (P2.3, live):** the net is now **runtime-shaped** . It boots at the default
over-provisioned 32-input head (zero-padding preserved), and `EngineApi.reshape({ inputSize, … })`
→ `WasmIML.reshape` swaps in a new net at the requested arity, **warm-started** from the overlapping
weights (`nisps_ml_reshape`; C-side dataset + feedback state RESET). When the active axis layout
CHANGES to a count ≠ the net's arity, `ConsoleApp` offers the swap behind `ReshapeModal.tsx`
(reset-on-reshape confirm; declining keeps the zero-padded head). Never offered on load. The
spine tolerates the arity change (buffers resize, version bumps); the training worker
(`wasm-worker.ts`) carries the current dims in its train message and re-creates its mirror net to
match. Debug: `window.__nisps.reshape(nIn)` / `.describe()` . See the `manifold-mixed-inputs` memory
for the locked design (adaptive slider viz when >2 dims is still pending).
feat(manifold): P5.2/P5.3 — derive MF_MODES from schema truth + per-mode engine dims
Schema-backed modes in console/model.ts are now DERIVED from the codegen
schemas in src/modes/generated/ (source of truth): real param names/groups/
count, plus each mode's ml net shape (MFMode.ml) and schema engine_id. A thin
manifold OVERLAY supplies only label/glyph/ModeClass/input/ordering. New
browser-viable modes xiasri + slp_workshop get derived entries; schema-less
visualizer + c15 stay hand-written on DEFAULT_MODE_ML. Schema min/max/default/
label/curve surface as engine-unit metadata (schemaMin/... on MFParam) without
touching the 0..1 routing semantics.
Switching instrument mode reshapes the runtime-shaped WASM net to the mode's
schema ml config (ConsoleApp effect keyed on [engine, modeId]; no confirm
modal). Boot lands paf_synth dims (4->[10,10,14]->33) once WASM is ready. The
P2.3 axis-count reshape offer still reads the engine's live inputSize and does
not spuriously prompt on a mode switch.
Adds schema-modes.spec.ts (P5 gate): drives switches via a new window.__mf
debug seam and asserts describe() dims, getWeights count, output length/bounds,
and UI param count FROM the imported schemas; spot-checks trainAsync after a
switch. Updates reshape/probe-api/geo-dislike specs to assert from the boot
mode schema instead of the retired fixed 32/126 shape.
All gates green: typecheck, unit (9), build, e2e (33).
2026-07-18 12:45:06 +02:00
- **Per-mode net dims (P5.3):** switching INSTRUMENT mode reshapes the net to that mode's schema
`ml` config (`MFMode.ml` — input/hidden/output + spread) via a `ConsoleApp` effect keyed on
`[engine, modeId]` . No confirm modal (switching instrument is deliberate); the axis-count
`ReshapeModal` above is for input-LAYOUT changes only. The effect depends on `engine` , so on boot
it fires once WASM is ready and lands the boot mode's dims (**paf_synth → 4→[10,10,14]→33**, weights
809 — NOT the 32→126 default). The reshape-offer effect reads the engine's CURRENT `inputSize`
live, so a mode switch that changes arity doesn't spuriously prompt (its baseline tracks axis
COUNT, unchanged by a pure dim change). Non-schema modes restore `DEFAULT_MODE_ML` (32→126).
Debug seam for tests: under `?debug=1` ConsoleApp installs `window.__mf`
(`setMode`/`getModeId`/`paramCount`/`modeIds`) — the UI-level analogue of `__nisps` , since no
in-UI instrument picker exists yet (`ctx.modes`/`setModeId` are plumbed but unrendered).
2026-06-28 22:20:58 +02:00
### Feedback — `src/feedback/`
feat(manifold): geometric dislike + jolt/OU via shared core (one-core-engine P3)
TS half of P3.1/P3.2/P3.3: wire the new WASM exports and delete the TS
approximations now that geometric-dislike, jolt, OU, and the seeded RNG live
in the C++/WASM core.
- types.ts/wasm-iml.ts: bind + wrap dislike_geometric, store_positive,
positive/negative_count, set_avoid_style, jolt_press/step/release/active/
lr_scale/tick_lr_ramp, explore_intensity/get/apply. Weight-mutating wrappers
republish weights (version bump); explore_apply reuses feedbackBuf.
- engine-api: extend .feedback (dislikeGeometric/storePositive/counts/
setAvoidStyle) + new .explore facade. thumbsDown now passes the HEARD
(routed) vector, not the raw MLP output (raw == net output => inert cold-start).
- feedback/controller.ts: delete dislikes[] + applyDislikeBias + both C++ GAP
blocks + the SeededRng field; dislike() -> core dislikeGeometric (returns
action, 15 => cold-start prompt); like() feeds the centroid via the core
thumbsUp; getState() exposes positive/negative counts. Delete feedback/rng.ts.
- engine/exploration.ts: execute the P3 SWAP POINT -> drive engine.explore.*;
delete engine/jolt.ts + ou-explore.ts. UI surface unchanged.
- ConsoleApp: one-time cold-start banner (British spelling), routed heard vector
at the dislike call site.
- App.tsx/spine.ts: under ?debug=1 pin a fixed seed + fixed per-tick dt so the
probe/e2e are deterministic (production keeps time-seeded, real-time dt).
- tests: new geo-dislike.spec.ts; probe gains dislikeGeometric/storePositive/
feedbackCounts/setAvoidStyle; the thumbsDown probe test now drives a real
distinct-heard-vector dislike (bare thumbsDown on the net's own output is
correctly inert under the geometric core). 27 e2e + 9 unit green.
Docs: manifold/ONBOARDING.md engine+feedback sections synced.
2026-07-14 04:54:27 +02:00
- `controller.ts` — `FeedbackController` , framework-neutral, owned by ConsoleApp. **As of one-core-
engine P3 it holds NO algorithm approximation** — a thin driver over the shared C++ core. Two modes:
**geometric-dislike** (default, Mode 1, "Push away") — `dislike()` calls `engine.feedback.
dislikeGeometric(heardVec)` (the k-NN centroid push-away in `nisps/ml/geo_push.hpp` ; returns
FeedbackAction 14=push / 15=cold-start) then `process()` ; `like()` runs the core's `thumbsUp`
(auto-stores the positive centroid in Avoid+Geometric) + `addExample` + `train` . **explore-and-
place** (Mode 2, positive-only) drives the core's snapshot/scratchpad/undo lifecycle; caller
accumulates anchors and trains on finalise with warm-start. Solo/arm via per-output mask.
- **The heard-vector rule:** the geometric dislike trains AWAY from the HEARD (post-pipeline, routed)
output — pass `engine.routedOutput()` , never the raw MLP output, or the cold-start MSE derivative is
zero (inert). The `EngineApi.feedback.thumbsDown` facade + the Mode-1 `dislike()` call site honour this.
- **Cold-start prompt:** a dislike with zero positives returns action 15 → ConsoleApp shows a one-time
"Like a few sounds first…" banner (dismissed on the next like or the dismiss button; rl-feedback §7).
- The interim `rng.ts` (`SeededRng`) and the `C++ GAP` approximation markers are **gone** — the seeded
RNG, geometric push, jolt, and OU all run in the core now.
2026-06-28 22:20:58 +02:00
### Backends — `src/backends/`
- `manager.ts` — `BackendManager` , the **single consumer of the engine spine** for output: subscribes,
reads `routedOutput()` each version bump, calls `active.send(routed)` . Gates audio (mutes synth when
a non-synth backend is active).
- Registered: `midi` (`midi-backend.ts`, CC out, ~20Hz throttle), `osc` (`osc-client.ts` → WebSocket
to the Deno bridge in `osc-bridge/bridge.ts` ), `synth` /`cvgate` (`passthrough-backend.ts`),
`particles` (`particle-backend.ts`, no-op — the visualiser is a separate `flow-field.ts` consumer),
`vcv` (`vcv-backend.ts`, bidirectional). `mapping.ts` , `presets.ts` , `useBackendManager.ts` support
config. `backend.ts` defines the `OutputBackend` interface + `OutputMapping` .
### Misc
- `src/serial/memlnaut-serial.ts` — **STUB** Web Serial scaffold for the MEMLNaut Editor mode (protocol TODO). `EditorPanel.tsx` is its UI.
- `src/settings/settings-store.ts` — localStorage settings (`mf-settings`): icon style, input-map shape, corner radius.
- `src/midi-devices/` — codegen'd external-synth device templates.
- `src/debug/probe.ts` — `window.__nisps` synchronous probe (engine/audio/bus). Some playground
feature-store methods are present-but-inert (not ported yet) to keep the surface stable.
---
## 6. Gotchas & non-obvious rules
1. **Outputs are read imperatively, not via React state.** Subscribe to `version` , then call
`getOutputs()` /`routedOutput()`. Buffers are reused — copy if you need to retain.
2. **Asset URLs must resolve against `document.baseURI`, not `location.origin`.** `base: './'` +
the `/next/` subpath mean a hardcoded `/nisps.wasm` 404s. The `assetUrl()` helper (in
`engine-host.ts` , `wasm-iml.ts` , `wasm-worker.ts` ) handles this — use it.
3. ** `nisps.js` is non-module Emscripten glue (no ES exports).** Workers/worklet fetch + indirect-eval
to install the global `createNispsModule` ; the worklet uses raw `WebAssembly.instantiate` .
feat(manifold): runtime-shaped net reshape with confirm modal (P2.3)
Wire the runtime-shaped WASM MLP (one-core-engine P2) through the manifold:
- WasmIML.reshape(dims, spread): calls nisps_ml_reshape, re-describes the
instance, reallocates every dim-dependent heap buffer, refreshes weightCount,
clears the TS Dataset mirror (C-side resets), drops the stale training worker,
and pushes the new shape through the sink.
- EngineApi.reshape exposes it and re-ticks the spine so outputs/audio reflect
the new net. Spine already tolerates the arity change (buffers resize, version
bumps); documented.
- Training worker protocol carries the current hidden dims; the worker
ensureNet()s its mirror net to match after a reshape.
- ConsoleApp offers the reshape behind ReshapeModal on an active-layout CHANGE
(never on load; default 32-input over-provisioned head + zero-padding
preserved when declined). British copy, reset-on-reshape.
- Drawers: delete the stale even/odd blending note; honest dedicated-dimensions
line + net-arity chip.
- Probe: __nisps.reshape(nIn) / .describe(); e2e reshape.spec (default 32/126,
reshape to 4, describe reports 4, bounded outputs, weight count 3148→2868,
spine still propagates). All 25 e2e pass (20 existing + 5 new).
- ONBOARDING: refresh the reshape status + stale hardwired-arity gotcha.
2026-07-14 03:54:12 +02:00
4. **WASM MLP is runtime-shaped (since one-core-engine P2).** `nisps_ml_create(in, out, hidden[])`
honours its dims (non-positive/null → the default `32→[10,14,18]→126` head, so pre-P2 callers are
bit-identical), and `nisps_ml_reshape` swaps in a warm-started net at new dims. Weights = 3148 at
the default shape; reshaping only the input arity shifts the first layer (e.g. →4 inputs = 2868).
The firmware MLP stays compile-time templated — only the WASM/browser build is dynamic.
feat(manifold)!: route input/output pipelines + curves through the WASM core (P4.3/P4.4)
One-core-engine P4.3/P4.4: the input/output pipeline processing and the curve
catalog now live in the C++/WASM core (nisps/pipeline/*, nisps/core/math.hpp).
The TS ports are deleted and the browser drives the WASM chains.
Engine:
- WasmIML owns a nisps_pipeline_create handle + bridge buffers and exposes
setInputConfig (TS InputConfig → 15-float wire), processInput, resetInput,
setOutputConfig (Infinity slew → 0), setOutputFreezeMask, processOutput
(in place), resetOutput, curveApply, curveApplyBatch (chunked). Handle +
buffers created in init_, freed in dispose, output-sized buffers realloc'd
on reshape.
- Spine routes setInputs through iml.processInput/processOutput (state lives
C++-side); config source-of-truth stays TS-side and is pushed on attach /
setInputConfig / setOutputConfig. Preserves ?debug=1 fixed-dt determinism
(same dt fed to the WASM calls). EngineApi gains setInputConfig/
setOutputConfig/curveApply/curveApplyBatch.
- New types-only modules: pipeline-types.ts (InputConfig/OutputConfig +
defaults + wire int mappers) and curve-catalog.ts (CurveName + name→id).
types.ts declares the pipeline/curve C ABI. engine barrel updated.
- DELETED src/engine/{input-pipeline,output-pipeline,curves}.ts.
Tests (P4.4 gate — recorded-gesture regression):
- pipeline-golden.test.ts now loads the built WASM (indirect-eval shim,
tests/wasm-load.ts) and drives the frozen gesture/output fixtures through the
C++ chains, honouring the per-event dt clock contract. Tolerance 1e-5
(non-momentum drift <5e-7). The 3 momentum configs carry 1e-2: proven-inherent
f32 drift (a byte-faithful f32 port of the exact original algorithm matches
the WASM to <6e-8 while both diverge from the f64 capture by ~7-9e-3), NOT a
core bug.
- curves-golden.json: linear/square/sqrt/centered_power kept as the original
f64 captures (C++ matches within <3e-8); exp/log/sigmoid/cubic RE-BASELINED
from the WASM (deliberate switch to firmware-exact k=1 exp/log, slope-6
sigmoid, true cubic x^3). Provenance recorded in-file.
- _generate.ts rebuilt as the WASM curve re-baseline tool; pipeline-golden-lib
trimmed to pure data builders.
Docs: fixtures/README.md + manifold/ONBOARDING.md updated.
Gates: typecheck, bun test (9), vite build, playwright e2e (27) all green.
2026-07-18 12:21:35 +02:00
5. **Curves + input/output pipelines are C++/WASM only (one-core-engine P4).** No TS curve/pipeline
maths remains; the browser samples `nisps/core/math.hpp` + `nisps/pipeline/*` via the WASM. The
golden test (`tests/pipeline-golden.test.ts`) drives the WASM chains against the frozen fixtures.
NOTE: `exp/log/sigmoid/cubic` deliberately changed to the firmware-exact maths at P4 (see
`tests/fixtures/README.md` ); `linear/square/sqrt/centered_power` are unchanged. The 3 momentum input
configs carry a wide (`1e-2`) tolerance — proven-inherent f32 drift, documented in the test header.
2026-06-28 22:20:58 +02:00
6. **COOP/COEP headers are mandatory** for the WASM/worklet path — set in `vite.config.ts` for
dev+preview, and at nginx server scope in prod (inherited by `/next/` ).
7. **Never emit "C15"** in code or bundle — the smoke test fails on it. Synth is "Powerful Synth
Engine"/"Built-in Synth".
8. **Two `MFParam` writers** (stage controls AND dock rows) share one store — edits in either re-render both.
---
## 7. Deeper references (only when this doc isn't enough)
2026-07-13 22:15:46 +02:00
All in `docs/specs/` (at the repo root, not under `manifold/` ), with subdirectories:
- `plans/BUILD-PLAN.md` — locked decisions + the 12-step build sequence + spec pointers (the resume anchor).
2026-06-28 22:20:58 +02:00
- `engine-architecture.md` — full engine/spine/WASM design.
- `dock-spec.md` — dock + drawers spec. `inputs-spec.md` — mixed-input design. `backends-spec.md` — backends.
2026-07-13 22:15:46 +02:00
- `docs/adr/rl-feedback-design.md` + `feedback-modes-port-spec.md` + `recon/findings-feedback-behaviour.md` — feedback modes (Mode 1/Mode 2).
- `aimmersive-clone-spec.md` / `recon/playground-2026.md` — the a-immersive feature parity target.
2026-06-28 22:20:58 +02:00
- `src/backends/README.md` — backend wiring notes.
**Memories** (auto-loaded): `manifold-build` (status + locked decisions), `manifold-mixed-inputs`
(N-D input design), `manifold-deploy-pipeline` (the auto-deploy chain), `single-double-joystick-toggle` .
---
*Keep this in sync with the code — update it in the same commit as any change to UI structure, the
engine spine, the dock/drawers, or the build/deploy commands.*