diff --git a/MAP.md b/MAP.md index 83120ac..fb0b999 100644 --- a/MAP.md +++ b/MAP.md @@ -66,6 +66,14 @@ anchor + locked decisions) and the `docs/redesign/*-spec.md` set. - `manifold/src/midi-devices/` — external-synth device templates. `generated/` is codegen output from `schemas/midi_devices/` (`MIDI_DEVICES` catalogue + `MIDI_DEVICES_BY_ID`, params by name+CC). The MIDI Outputs config (`dock/OutputsBackendConfig.tsx`) reads it for the device picker + param-select that fills the CC table. +- `manifold/src/inputs/` — modular INPUT layer feeding the ML head. The Inputs dock picks ONE exclusive mode + (`InputMode` = `internal` | `gamepad` | `midi`; Internal/XY-pad is default). `input-layer.ts` owns a single rAF + loop composing the active source's axes → reduced to the engine arity (fixed 2-in WASM → even/odd blend) → one + `setInputs`, plus an `onReducedInput` callback the manifold tracks. Sources: `xy-pad-source` (push-driven), + `gamepad-source` (sticks→axes single/double; buttons emit press+release actions, bound in `ConsoleApp` to + verdicts — LB/RB=down/up, X/Y/B=randomise/nudge/undo, A-hold=reposition), `midi-input-source` (device picker + + BATCH "MIDI Learn": every CC swept while armed becomes an axis, shown as read-only meters). `useInputLayer.ts` + is the React binding; `base-source.ts` shared status/action plumbing; `types.ts` the adapter contract. - `manifold/src/feedback/` — `controller.ts` (Explore-and-place scratchpad + geometric-dislike + solo, TS prototype), `rng.ts` (seeded). - `manifold/src/settings/` — `settings-store.ts` (monochrome icons, input-map shape, corner radius). diff --git a/docs/redesign/midi-gamepad-inputs-worklog.md b/docs/redesign/midi-gamepad-inputs-worklog.md new file mode 100644 index 0000000..34e487f --- /dev/null +++ b/docs/redesign/midi-gamepad-inputs-worklog.md @@ -0,0 +1,96 @@ +# Work log — MIDI + Game Controller inputs, N-D engine foundation + +*Scope: what was actually built on the `feat/midi-inputs` branch. This is a +description of the work, not a spec. The design intent lives in +`docs/redesign/inputs-spec.md`; where this branch diverges from or only partially +realises that spec, it is called out below.* + +## Summary + +This branch wires the modular input layer (which already existed as adapters in +`manifold/src/inputs/`) into the Console, adds the missing gamepad→verdict and +MIDI-device plumbing, and reshapes the browser ML engine so input axes are +genuine independent dimensions instead of being blended into two. It landed in +two passes: + +1. **Input methods** — a working Inputs dock with three sources (Internal XY + pad, Game Controller, MIDI), gamepad buttons bound to verdicts, and a batch + "MIDI Learn". +2. **Engine foundation for mixing** — the WASM net was widened from 2 inputs to + a 32-input maximum so each active axis gets its own dimension (no blending). + +The Inputs dock currently presents the three sources as an **exclusive** picker. +The engine groundwork for *mixing* sources (independent dimensions, no idle-bias) +is in place, but the dock toggles, the reshape-confirm modal, and the +>2-dimension slider visualisation described in `inputs-spec.md` are **not yet +wired** — see "Not done yet" below. + +## What changed + +### Input sources (`manifold/src/inputs/`) +- `gamepad-source.ts` — buttons now emit both press and release edges (with + standard-mapping labels A/B/X/Y/LB/RB/…), enabling hold-and-move gestures. + Single/double-stick (2/4 axes) was already present. +- `midi-input-source.ts` — added single-device selection (`selectDevice`, the + dock device picker; default still listens to all ports) and changed MIDI-Learn + from one-binding-per-arm to a **batch** capture: while armed, every distinct CC + that moves is appended as an axis; notes stay discrete actions and are not + auto-bound. Learned CCs are exposed as bindings for the dock. +- `types.ts` — `InputAction` gained an optional `phase` ('press' | 'release'); + added an `InputMode` ('internal' | 'gamepad' | 'midi') type. +- `input-layer.ts` — added `onReducedInput` so the on-screen manifold can track a + gamepad/MIDI-driven position. **`compose()` no longer mean-blends**: it + forwards each active axis 1:1 to its own engine input slot (the engine + zero-pads the rest; a zero input is inert, `0 × weight = 0`). +- `useInputLayer.ts` — the React binding; exposes the active mode, per-source + status, gamepad stick mode, MIDI device list/selection, batch-learn arm, and + learned bindings. (Currently exclusive — one mode at a time.) + +### Console wiring (`manifold/src/console/`) +- `ConsoleApp.tsx` — subscribes to gamepad actions and binds them to existing + verdict handlers: RB = thumbs-up, LB = thumbs-down, X = randomise, Y = nudge, + B = undo, A-hold = reposition (hold, move stick, release to place an example + at the stick position). Mirrors the composed input position onto the manifold + when a non-pad source is active (deduped to avoid per-frame re-renders). +- `Drawers.tsx` — rebuilt the Inputs drawer: a source picker, a gamepad stick + toggle + button legend, a MIDI device picker, the batch MIDI-Learn flow with + its "move every control, then Done" message, and learned controls rendered as + read-only meters styled distinctly from the output sliders. + +### Engine (`manifold/src/engine/`, `nisps/wasm/`) +- `nisps/wasm/bindings.cpp` — `DefaultMLP` widened `MLP<2,…>` → `MLP<32,…>` + (32 = `MAX_AXES`). Each active axis maps to a dedicated input slot; unused + slots are held at 0. Rebuilt `nisps.wasm` and synced to both + `playground/public/` and `manifold/public/` (the C ABI / `nisps.js` glue is + unchanged). +- `spine.ts` / `engine-api.ts` — `setInputs(arr)` now writes the full + N-dimensional vector (it previously dropped everything past `arr[1]`); the + primary pair still runs through the 2-D input pipeline so the pad keeps its + feel, axes 2+ are written raw, and `process()` re-ticks the whole vector after + weight changes via the new `spine.reprocess()`. + +### Tests / build +- `tests/cpp/parity_check.cpp` + `tests/cpp/parity_wasm.mjs` — `ParityMLP` + bumped to 32 inputs and the example/feature buffers widened to match the net's + arity (`add_example` requires `features.size() >= NIn`). +- `nisps/CMakeLists.txt` — the parity binary now builds with `-ffp-contract=off`. + Widening the input layer exposed a native↔WASM divergence: native clang/gcc + fuse multiply-adds (FMA) the WASM build has no instruction for, and the + training loop amplified the rounding difference past the 1e-5 parity tolerance. + Disabling FP contraction on the native parity build alone restores bit-equality + (max delta ~2.4e-7). + +## Verification +- C++ suites (4/4) pass; native↔WASM parity passes at 1e-5. +- `manifold` typechecks and builds; the Playwright smoke test (engine loads, + input→output propagates) passes. + +## Not done yet (vs `inputs-spec.md`) +- The Inputs dock is an **exclusive** picker; mixing several sources at once + (independent toggles) is not wired, though the engine and `compose()` now + support it. +- No reshape-confirm modal + net reset when the active input set changes. +- No swap to a slider visualisation when more than two input dimensions are + active (the 2-D manifold is always shown). +- The input pipeline (deadzone/zoom/curve) is applied only to the primary pair; + per-source conditioning for axes 2+ is left raw. diff --git a/manifold/public/nisps.wasm b/manifold/public/nisps.wasm index d2226c2..9ee47b2 100755 Binary files a/manifold/public/nisps.wasm and b/manifold/public/nisps.wasm differ diff --git a/manifold/src/console/ConsoleApp.tsx b/manifold/src/console/ConsoleApp.tsx index e6d6912..d0bf780 100644 --- a/manifold/src/console/ConsoleApp.tsx +++ b/manifold/src/console/ConsoleApp.tsx @@ -546,6 +546,59 @@ export function ConsoleApp({ focus: initialFocus = 'composite' }: ConsoleAppProp return () => window.removeEventListener('keydown', onKey); }); + // ---- Game-controller verdict bindings (inputs-spec) --------------------- + // The gamepad's sticks already feed the input layer (→ engine); its BUTTONS + // drive verdicts here. Standard-mapping indices: + // LB(4) = down/negative · RB(5) = up/positive · X(2) = randomise · + // Y(3) = nudge · B(1) = undo · A(0) hold-and-move = reposition an example + // (hold A, move the stick to a spot on the manifold, release to drop it). + // MIDI note actions are surfaced too but left unbound (MIDI mode learns CCs + // as INPUT axes; verdicts there stay on the on-screen / keyboard controls). + // The effect has no dep array (matching the keydown handler above) so each + // binding closes over the latest verdict functions + live `pos`. + useEffect(() => { + const unBtn = inputs.onAction((a) => { + if (a.source !== 'gamepad') return; + const phase = a.phase ?? 'press'; + if (phase === 'press') { + switch (a.id) { + case 'button:4': // LB → thumbs-down + perturb(); + break; + case 'button:5': // RB → thumbs-up + commit(); + break; + case 'button:2': // X → randomise / re-roll + reroll(); + break; + case 'button:3': // Y → nudge (scratchpad) + onScratchNudge(); + break; + case 'button:1': // B → undo + undo(); + break; + case 'button:0': // A (down) → begin repositioning an example + onPlace(); + break; + } + } else if (phase === 'release' && a.id === 'button:0') { + // A (up) → drop the example at the current (stick-driven) location. + onPickLocation(pos[0], pos[1]); + } + }); + // Mirror the composed gamepad/MIDI position onto the on-screen manifold so + // markers + readouts track the controller (the XY pad pushes its own pos). + // The callback fires every rAF frame — only re-render when it actually moves. + const unPos = inputs.onReducedInput((x, y) => { + if (inputs.inputMode === 'internal') return; + setPos((prev) => (Math.abs(prev[0] - x) < 1e-3 && Math.abs(prev[1] - y) < 1e-3 ? prev : [x, y])); + }); + return () => { + unBtn(); + unPos(); + }; + }); + const onToggleAudio = () => { if (!engine) return; if (engine.audio.isStarted) { diff --git a/manifold/src/console/Drawers.tsx b/manifold/src/console/Drawers.tsx index cd1561e..52f4aef 100644 --- a/manifold/src/console/Drawers.tsx +++ b/manifold/src/console/Drawers.tsx @@ -23,6 +23,7 @@ import type { ReactNode } from 'react'; import { Badge, Button, PillToggle, Slider, Switch } from '../primitives'; import type { ConsoleCtx, DrawerDepth, DrawerKey, FeedbackModeUI, SoloMode } from './types'; +import type { InputMode } from '../inputs'; import { OutputControlRow } from '../dock/OutputControlRow'; import { BackendAdvanced } from '../dock/BackendAdvanced'; import { OutputsBackendConfig, BackendStatusChip } from '../dock/OutputsBackendConfig'; @@ -236,165 +237,246 @@ const STATUS_TONE: Record = { idle: 'var(--fg-dim)', }; +const INPUT_MODE_OPTS: { value: InputMode; label: string }[] = [ + { value: 'internal', label: 'Internal' }, + { value: 'gamepad', label: 'Game Controller' }, + { value: 'midi', label: 'MIDI' }, +]; + +/** Standard-mapping gamepad button → verdict legend (mirrors ConsoleApp). */ +const GAMEPAD_LEGEND: { btn: string; action: string }[] = [ + { btn: 'RB', action: 'Up · positive feedback' }, + { btn: 'LB', action: 'Down · negative feedback' }, + { btn: 'X', action: 'Randomise' }, + { btn: 'Y', action: 'Nudge' }, + { btn: 'B', action: 'Undo' }, + { btn: 'A (hold)', action: 'Reposition — hold, move stick, release to place' }, +]; + +/** + * Read-only INPUT meter — shows a learned MIDI control's live value. Deliberately + * styled apart from the output Sliders (which are orange, interactive thumbs): + * these are inset bars on the secondary accent with an "in" tag, so the user can + * see at a glance that these feed the net rather than being driven by it. + */ +function MidiInputMeter({ label, value, onClear }: { label: string; value: number; onClear: () => void }) { + const pct = Math.max(0, Math.min(1, value)); + return ( +
+ + {label} + +
+
+
+ + {value.toFixed(2)} + + +
+ ); +} + function InputsDrawer(ctx: ConsoleCtx, depth: DrawerDepth) { const inp = ctx.inputs; - const enabledCount = inp.sources.filter((s) => s.enabled).length; const reshaping = inp.axisCount > inp.engineInputSize; + const active = inp.sources.find((s) => s.enabled); + const modeLabel = INPUT_MODE_OPTS.find((o) => o.value === inp.inputMode)?.label ?? 'Internal'; return ( <>
- - {enabledCount === 0 ? 'no source' : `${enabledCount} source${enabledCount > 1 ? 's' : ''}`} - - {inp.axisCount} axes - engine: {inp.engineInputSize}-in + {modeLabel} + {inp.inputMode !== 'internal' && {inp.axisCount} axes} {reshaping && blended → {inp.engineInputSize}} + {active && ( + {active.status.state} + )}
- Sources -
- {inp.sources.map((s) => ( -
-
- - {s.label} - {s.enabled ? ` · ${s.axisCount} ax` : ''} - - {depth !== 'peek' && ( - - {s.status.message} - - )} -
- inp.setEnabled(s.kind, v)} label="" /> -
- ))} -
+ Input source + + {active && depth !== 'peek' && ( + + {active.status.message} + + )} - {depth !== 'peek' && ( + {/* ---- Internal (XY pad / manifold) ---- */} + {inp.inputMode === 'internal' && depth !== 'peek' && ( +

+ Drag the on-screen manifold / XY pad. Two axes feed the net directly — this is the default. +

+ )} + + {/* ---- Game Controller ---- */} + {inp.inputMode === 'gamepad' && depth !== 'peek' && ( <> - {/* ---- Gamepad config ---- */} - {inp.sources.find((s) => s.kind === 'gamepad')?.enabled && ( - <> - Gamepad · sticks - - - )} - - {/* ---- MIDI learn-map ---- */} - {inp.sources.find((s) => s.kind === 'midi')?.enabled && ( - <> - MIDI · learn-map -
- - {inp.midiBindings.length > 0 && ( - - )} + Sticks + + Buttons +
+ {GAMEPAD_LEGEND.map((g) => ( +
+ {g.btn} + {g.action}
- {inp.midiBindings.length === 0 ? ( -

- No axes learned yet — arm Learn, then wiggle a knob or hit a pad. CCs map to a - continuous axis; notes map to a gate (1 while held). -

- ) : ( -
- {inp.midiBindings.map((b, i) => ( -
- - {b.label} · {b.value.toFixed(2)} - - -
- ))} -
- )} - {inp.midiInputs.length > 0 && ( -

- Listening on: {inp.midiInputs.map((p) => p.name).join(', ')} -

- )} - - )} + ))} +
+

+ Connect a controller and press any button to wake it. Sticks drive the input map; + buttons fire the verdicts above. +

+ + )} - {/* ---- Channel layout ---- */} - Channel layout - {inp.channelLayout.length === 0 ? ( + {/* ---- MIDI ---- */} + {inp.inputMode === 'midi' && depth !== 'peek' && ( + <> + Device + {inp.midiInputs.length === 0 ? (

- No active axes. Enable a source above. + No MIDI inputs detected. Connect a device — it appears here automatically.

) : ( -
- {inp.channelLayout.map((c, i) => ( - - {i}: {c.source}·{c.label} - +
+ + {inp.midiInputs.map((p) => ( + ))}
)} -

- The active sources concatenate into one input vector at the head of the spine. - {reshaping - ? ` The browser WASM is a fixed ${inp.engineInputSize}-input head (MLP<2,…>), so the - ${inp.axisCount} axes are blended down to ${inp.engineInputSize} (even→X / odd→Y mean).` - : ''}{' '} - {/* TODO(workstream F, docs/redesign/inputs-spec.md — "multiple WASM modules + - warm-start"): give every axis its own genuine input dimension by (re)loading a - WASM module whose MLP arity matches axisCount and warm-starting from the prior - net. Deferred — the reduction lives in InputLayer.compose(). */} - True per-axis dimensions land with the multi-WASM reshape (inputs-spec). -

+ MIDI Learn + {inp.midiLearnArmed ? ( +
+ + Move all of the controls you want to use, then click Done. Each knob or fader you + touch becomes an input. + + +
+ ) : ( +
+ + {inp.midiBindings.length > 0 && ( + + )} +
+ )} + + {inp.midiBindings.length > 0 && ( + <> + Learned controls +
+ {inp.midiBindings.map((b, i) => ( + inp.clearMidiBinding(i)} + /> + ))} +
+ + )} )} + + {/* ---- Reshape note (only when >2 axes feed the fixed WASM head) ---- */} + {reshaping && depth !== 'peek' && ( +

+ {/* TODO(workstream F, docs/redesign/inputs-spec.md — "multiple WASM modules + + warm-start"): give every axis its own genuine input dimension by (re)loading a + WASM module whose MLP arity matches axisCount and warm-starting from the prior + net. Deferred — the reduction lives in InputLayer.compose(). */} + The browser WASM is a fixed {inp.engineInputSize}-input head (MLP<2,…>), so the{' '} + {inp.axisCount} axes are blended down to {inp.engineInputSize} (even→X / odd→Y mean). True + per-axis dimensions land with the multi-WASM reshape. +

+ )} ); } diff --git a/manifold/src/engine/engine-api.ts b/manifold/src/engine/engine-api.ts index 3b19719..1b0373a 100644 --- a/manifold/src/engine/engine-api.ts +++ b/manifold/src/engine/engine-api.ts @@ -164,9 +164,13 @@ export class EngineApi { this.spine.setInput(x, y); } - /** Set an arbitrary input vector (first two used as XY for the fixed 2→N MLP). */ + /** + * Set the full N-dimensional input vector (one axis per active input source). + * The first two axes run through the 2-D input pipeline; axes 2+ are raw. + * Extra axes beyond the net's input arity are ignored; unused slots → 0. + */ setInputs(arr: ReadonlyArray): void { - this.spine.setInput(arr[0] ?? 0.5, arr[1] ?? 0.5); + this.spine.setInputs(arr); } /** Live post-ML output vector (reused buffer — read, don't retain). */ @@ -193,7 +197,7 @@ export class EngineApi { * state without the user having to move the controller. */ process(): void { - this.spine.setInput(this.spine.lastRawX, this.spine.lastRawY); + this.spine.reprocess(); } // ---- Training ------------------------------------------------------ diff --git a/manifold/src/engine/spine.ts b/manifold/src/engine/spine.ts index 70186a0..12ceda3 100644 --- a/manifold/src/engine/spine.ts +++ b/manifold/src/engine/spine.ts @@ -94,6 +94,8 @@ export class Spine implements EngineSink { // Last raw input, so `EngineApi.process()` can re-tick after a weight change. lastRawX = 0.5; lastRawY = 0.5; + // Full last raw input vector (N-D) for re-ticking without losing extra axes. + private lastRawInputs: Float32Array = new Float32Array(2); private mlBuf: F32 = new Float32Array(126); private routedBuf: F32 | null = null; @@ -164,34 +166,61 @@ export class Spine implements EngineSink { // ---- The hot action ------------------------------------------------ /** - * Drive a raw [0,1] XY input through processed → ml → routed eagerly and - * synchronously, then fire the single backend.send at the tail. Off render. - * Returns the routed buffer (live, reused — do not retain across calls). + * Drive a raw [0,1] XY input through processed → ml → routed. Convenience for + * the 2-D manifold / XY-pad path — delegates to {@link setInputs}. */ setInput(x: number, y: number): Float32Array | null { + return this.setInputs([x, y]); + } + + /** + * Drive an N-dimensional raw input vector (each ∈ [0,1]) through + * processed → ml → routed eagerly and synchronously, then fire the single + * backend.send at the tail. Off render. Returns the routed buffer (live, + * reused — do not retain across calls). + * + * The mix-and-match input layer composes one axis PER active input source + * (XY pad / gamepad sticks / learned MIDI CCs) into this vector. The first + * two axes run through the 2-D input pipeline (deadzone→zoom→curve→smoothing→ + * momentum) so the pad keeps its feel and the ≤2-D path is unchanged; axes 2+ + * are written raw (sources self-condition). Unused slots up to the net's input + * arity are held at 0 so a shrinking vector never leaves a stale dimension hot. + */ + setInputs(arr: ArrayLike): Float32Array | null { const iml = this.iml; if (!iml) return null; - const now = (typeof performance !== 'undefined' ? performance.now() : Date.now()); - const dt = this.lastTickMs > 0 ? (now - this.lastTickMs) / 1000 : 1 / 60; - this.lastTickMs = now; + const dt = this.dt_(); + const inSize = this.state_.inputSize; - // 1. processed (pure input pipeline) + // 1. primary pair through the pure input pipeline (pad feel / 2-D parity). + const x = arr.length > 0 ? arr[0] : 0.5; + const y = arr.length > 1 ? arr[1] : 0.5; this.rawInput[0] = x; this.rawInput[1] = y; this.lastRawX = x; this.lastRawY = y; const proc = processInput(this.rawInput, this.inputConfig, this.inputState, dt); this.inputState = proc.state; - - // 2. ml (inference into the reused buffer; no alloc) iml.setInput(0, proc.x); iml.setInput(1, proc.y); + + // 2. extra axes raw; unused slots cleared to 0. Remember the full raw vector + // so process() can re-tick after a weight change without losing dims. + if (this.lastRawInputs.length !== inSize) this.lastRawInputs = new Float32Array(inSize); + this.lastRawInputs[0] = x; + this.lastRawInputs[1] = y; + for (let i = 2; i < inSize; i++) { + const v = i < arr.length ? arr[i] : 0; + iml.setInput(i, v); + this.lastRawInputs[i] = v; + } + + // 3. ml (inference into the reused buffer; no alloc). iml.processInto(this.mlBuf); - // Mirror to liveOutputs for imperative reads + bump. this.liveOutputs.set(this.mlBuf.subarray(0, this.liveOutputs.length)); - // 3. routed (output pipeline → reused routedBuf) + // 4. routed (output pipeline → reused routedBuf). const routedRes = processOutput(this.mlBuf, this.outputConfig, this.outputState, dt * 1000); this.outputState = routedRes.state; const routed = routedRes.processed; @@ -201,13 +230,30 @@ export class Spine implements EngineSink { this.routedBuf = routed; } - // 4. single backend.send at the tail (off React render) + // 5. single backend.send at the tail (off React render). if (this.backendSend && this.routedBuf) this.backendSend(this.routedBuf); this.bump_(); return this.routedBuf; } + /** + * Re-run the LAST full raw input vector through the spine (after a weight + * change — train / randomise / feedback) so outputs + audio reflect the new + * net without the user touching a control. Preserves all N dimensions. + */ + reprocess(): Float32Array | null { + return this.setInputs(this.lastRawInputs); + } + + /** Monotonic per-tick dt in seconds (≈1/60 on the first tick). */ + private dt_(): number { + const now = (typeof performance !== 'undefined' ? performance.now() : Date.now()); + const dt = this.lastTickMs > 0 ? (now - this.lastTickMs) / 1000 : 1 / 60; + this.lastTickMs = now; + return dt; + } + // ---- Imperative reads (canvas consumers bypass React) -------------- /** Live post-ML output vector. Reused — read, don't retain. */ diff --git a/manifold/src/inputs/gamepad-source.ts b/manifold/src/inputs/gamepad-source.ts index 7ec4d0a..65bca26 100644 --- a/manifold/src/inputs/gamepad-source.ts +++ b/manifold/src/inputs/gamepad-source.ts @@ -25,6 +25,31 @@ export type StickMode = 'single' | 'double'; const DEADZONE = 0.08; +/** + * Standard-mapping button index → human label. The Inputs dock binds these to + * verdict ops (see useInputLayer / ConsoleApp): LB/RB = down/up feedback, the + * face buttons = randomise / nudge / undo, and a hold-and-move button drops a + * repositioned example. The labels here keep the dock legend honest. + */ +const BUTTON_LABELS: Record = { + 0: 'A', + 1: 'B', + 2: 'X', + 3: 'Y', + 4: 'LB', + 5: 'RB', + 6: 'LT', + 7: 'RT', + 8: 'Back', + 9: 'Start', + 10: 'L3', + 11: 'R3', + 12: 'D↑', + 13: 'D↓', + 14: 'D←', + 15: 'D→', +}; + export class GamepadSource extends BaseSource { readonly kind: InputSourceKind = 'gamepad'; readonly label = 'Gamepad'; @@ -117,8 +142,17 @@ export class GamepadSource extends BaseSource { this.emitAction({ source: this.kind, id: `button:${i}`, - label: `Button ${i}`, + label: BUTTON_LABELS[i] ?? `Button ${i}`, value: pad.buttons[i].value || 1, + phase: 'press', + }); + } else if (!pressed && this.buttonsDown[i]) { + this.emitAction({ + source: this.kind, + id: `button:${i}`, + label: BUTTON_LABELS[i] ?? `Button ${i}`, + value: 0, + phase: 'release', }); } this.buttonsDown[i] = pressed; diff --git a/manifold/src/inputs/index.ts b/manifold/src/inputs/index.ts index da993a6..75c3b18 100644 --- a/manifold/src/inputs/index.ts +++ b/manifold/src/inputs/index.ts @@ -1,14 +1,15 @@ /** * Modular INPUT layer (workstream F) — public surface. * - * The user picks the input SOURCE(s) feeding the ML head (XY pad / MIDI / - * gamepad, or a combination); the InputLayer composes their axes into one - * N-dim vector at the head of the reactive spine. See input-layer.ts for the + * The user picks ONE exclusive input MODE feeding the ML head (Internal XY pad / + * Game Controller / MIDI); the InputLayer composes the active source's axes into + * one N-dim vector at the head of the reactive spine. See input-layer.ts for the * arity-reduction + the documented multi-WASM reshape TODO. */ export type { InputSource, InputSourceKind, + InputMode, InputSourceState, InputSourceStatus, InputAction, diff --git a/manifold/src/inputs/input-layer.ts b/manifold/src/inputs/input-layer.ts index a59ed2b..3cfc0f9 100644 --- a/manifold/src/inputs/input-layer.ts +++ b/manifold/src/inputs/input-layer.ts @@ -13,24 +13,18 @@ * its value), but routing everything through one compose path keeps sources * composable and the channel layout coherent. * - * ── Arity mismatch (the WASM reshape TODO) ────────────────────────────────── - * The browser WASM is fixed at MLP<2, …, 126> — a TWO-input head. When the - * composed vector has > 2 axes (double-stick gamepad = 4, MIDI learn-map = many) - * we must reduce to 2 to feed today's engine. We do NOT fake a wider net. + * ── Dedicated dimensions (no blending) ────────────────────────────────────── + * The WASM net is over-provisioned to a 32-input head (= MAX_AXES; see + * nisps/wasm/bindings.cpp). Each active axis drives its OWN engine input slot + * 1:1 — a double-stick gamepad is 4 genuine dims, a learned MIDI surface is N + * genuine dims. `compose()` simply forwards the active axes; the engine + * zero-pads the remaining slots and a zero input is inert (0 × weight = 0), so + * unused dimensions never perturb the net. We do NOT mean-blend (the previous + * behaviour) — that diluted every source and biased the net toward idle + * sources' resting values. * - * chosen reduction (this pass): pairwise BLEND. - * inX = mean(axis[0], axis[2], axis[4], …) // even axes - * inY = mean(axis[1], axis[3], axis[5], …) // odd axes - * so a single stick passes straight through (axis0→X, axis1→Y), a double - * stick averages L/R into one XY, and MIDI axes fold into X/Y by parity. - * - * TODO(workstream F, docs/redesign/inputs-spec.md — "multiple WASM modules + - * warm-start"): the real fix is to (re)load a WASM module whose MLP input arity - * matches the composed axis count and warm-start its weights from the prior net, - * so every axis gets its own genuine input dimension instead of being blended. - * That is a larger build (multiple .wasm artefacts or a runtime-variadic head) - * and is deliberately deferred — this layer is wired so that swapping the - * reduction for a true reshape is a localised change in `compose()`. + * Changing the ACTIVE axis count is a reshape: the front-end resets the net + * (recreate-from-scratch, behind a confirm modal) since slot meanings change. */ import type { InputAction, InputSource } from './types'; @@ -50,6 +44,7 @@ export class InputLayer { private rafId: number | null = null; private actionListeners = new Set<(a: InputAction) => void>(); private layoutListeners = new Set<() => void>(); + private reducedListeners = new Set<(x: number, y: number) => void>(); private unsubActions = new Map void>(); attach(engine: InputEngineSink): void { @@ -144,50 +139,32 @@ export class InputLayer { // 4. one engine write. engine.setInputs(reduced); + + // 5. report the reduced 2D position so the on-screen manifold can track a + // gamepad/MIDI-driven input (the XY pad pushes its own position). + if (this.reducedListeners.size) { + const x = reduced[0] ?? 0.5; + const y = reduced[1] ?? reduced[0] ?? 0.5; + for (const cb of this.reducedListeners) cb(x, y); + } } /** - * Reduce the composed N-axis vector to the engine's input arity. + * Map the composed N active axes to the engine's input vector — DEDICATED + * DIMENSIONS, no blending. Each active axis i drives engine input slot i 1:1; + * the engine zero-pads the slots beyond `count` and a zero input is inert + * (0 × weight = 0), so unused dimensions never perturb the net. * - * For the fixed 2-input WASM, fold by parity (even→X, odd→Y) via mean. If a - * future multi-module engine reports inputSize >= n, this passes axes through - * 1:1 (truncated/padded) — the seam where the real reshape lands. + * The net's input arity is over-provisioned (32, = MAX_AXES), so `inputSize` + * is effectively always ≥ n; the `min` only guards a transient where more + * axes are active than the net can take. We deliberately do NOT mean-blend + * (the old behaviour) — that diluted every source and biased the net toward + * idle sources' resting values. */ private compose(n: number, inputSize: number): number[] { - if (inputSize >= n) { - // True passthrough path (future multi-module head). Pad with 0.5. - const out = new Array(inputSize); - for (let i = 0; i < inputSize; i++) out[i] = i < n ? this.vector[i] : 0.5; - return out; - } - if (inputSize === 2) { - let sx = 0; - let sy = 0; - let cx = 0; - let cy = 0; - for (let i = 0; i < n; i++) { - if ((i & 1) === 0) { - sx += this.vector[i]; - cx++; - } else { - sy += this.vector[i]; - cy++; - } - } - return [cx ? sx / cx : 0.5, cy ? sy / cy : 0.5]; - } - // Generic fallback for any other fixed arity: chunked mean. - const out = new Array(inputSize).fill(0.5); - const per = Math.ceil(n / inputSize); - for (let k = 0; k < inputSize; k++) { - let s = 0; - let c = 0; - for (let i = k * per; i < Math.min((k + 1) * per, n); i++) { - s += this.vector[i]; - c++; - } - if (c) out[k] = s / c; - } + const count = Math.min(n, inputSize); + const out = new Array(count); + for (let i = 0; i < count; i++) out[i] = this.vector[i]; return out; } @@ -207,6 +184,14 @@ export class InputLayer { }; } + /** Subscribe to the reduced 2D input each frame (composed → engine arity). */ + onReducedInput(cb: (x: number, y: number) => void): () => void { + this.reducedListeners.add(cb); + return () => { + this.reducedListeners.delete(cb); + }; + } + private fanAction(a: InputAction): void { for (const cb of this.actionListeners) cb(a); } @@ -221,5 +206,6 @@ export class InputLayer { this.unsubActions.clear(); this.actionListeners.clear(); this.layoutListeners.clear(); + this.reducedListeners.clear(); } } diff --git a/manifold/src/inputs/midi-input-source.ts b/manifold/src/inputs/midi-input-source.ts index d419470..353fb31 100644 --- a/manifold/src/inputs/midi-input-source.ts +++ b/manifold/src/inputs/midi-input-source.ts @@ -10,10 +10,17 @@ * note-off. Note-on ALSO surfaces a discrete action (so a pad can fire * commit/perturb without the keyboard). * - * **Learn-map.** When `armLearn()` is active, the NEXT distinct CC or note seen - * is bound to a new axis (appended). This is the standard "MIDI learn" gesture: - * arm → wiggle the knob/pad → it captures. Axes can be cleared individually. - * The bindings are exposed for the dock channel-layout view. + * **Batch learn ("MIDI Learn" mode).** When `armLearn(true)` is active, EVERY + * distinct CC that moves is captured as a new axis (appended, deduped). The + * gesture the dock presents: arm → wiggle ALL the knobs/faders you want → + * click "Done" (`armLearn(false)`). This differs from the one-shot learn other + * apps use — the user sweeps the whole control surface in one pass. Notes are + * NOT auto-bound as axes (they stay as discrete actions); the "controls" the + * user sweeps are continuous CCs. Axes can be cleared individually or all. + * + * **Device selection.** By default every connected input port is listened to. + * `selectDevice(id)` narrows to a single port (the dock device picker); `null` + * restores listen-all. * * Pull-based: messages latch the latest per-binding value into `values`; * `sample()` copies them out. Hot path performs no IO/allocation. @@ -51,6 +58,8 @@ export class WebMidiInputSource extends BaseSource { private inputs: MIDIInput[] = []; private bindings: MidiBinding[] = []; private learnArmed = false; + /** Restrict listening to this input port id; null = every connected port. */ + private selectedDeviceId: string | null = null; private bindingsListeners = new Set<(b: MidiBinding[]) => void>(); isAvailable(): boolean { @@ -73,12 +82,16 @@ export class WebMidiInputSource extends BaseSource { // ---- Learn-map API (consumed by the dock) ------------------------------- - /** Arm/disarm MIDI-learn: the next distinct CC/note is captured as an axis. */ + /** + * Enter/leave batch MIDI-Learn. While armed, every distinct CC that moves is + * appended as an axis (the user sweeps their whole control surface, then + * clicks Done). Disarming keeps whatever was captured. + */ armLearn(armed: boolean): void { this.learnArmed = armed; this.setStatus( armed - ? { state: 'ready', message: 'Learn armed — move a knob or hit a pad' } + ? { state: 'ready', message: 'MIDI Learn — move every control you want, then click Done' } : this.readyStatus(), ); } @@ -87,6 +100,16 @@ export class WebMidiInputSource extends BaseSource { return this.learnArmed; } + /** Narrow listening to one input port (dock device picker). null = all ports. */ + selectDevice(id: string | null): void { + this.selectedDeviceId = id; + this.rewire(); + } + + getSelectedDeviceId(): string | null { + return this.selectedDeviceId; + } + getBindings(): ReadonlyArray { return this.bindings; } @@ -149,11 +172,15 @@ export class WebMidiInputSource extends BaseSource { if (!this.access) return; for (const inp of this.inputs) inp.onmidimessage = null; this.inputs = []; - this.access.inputs.forEach((inp) => { + this.access.inputs.forEach((inp, id) => { + // Honour the device picker: when a port is selected, listen to it alone. + if (this.selectedDeviceId !== null && id !== this.selectedDeviceId) return; inp.onmidimessage = (e) => this.onMessage(e); this.inputs.push(inp); }); - if (this.statusState.state !== 'connecting') this.setStatus(this.readyStatus()); + if (this.statusState.state !== 'connecting' && !this.learnArmed) { + this.setStatus(this.readyStatus()); + } } private readyStatus(): { state: 'ready'; message: string } { @@ -173,41 +200,45 @@ export class WebMidiInputSource extends BaseSource { const d2 = data.length > 2 ? data[2] : 0; if (status === STATUS_BYTE_CC) { - this.handleBindable('cc', d1, channel, d2 / 127); + this.handleCc(d1, channel, d2 / 127); } else if (status === STATUS_BYTE_NOTE_ON && d2 > 0) { - this.handleBindable('note', d1, channel, 1); - this.emitAction({ source: this.kind, id: `note:${d1}`, label: `Note ${d1}`, value: d2 / 127 }); + // Notes drive a held-gate on any already-learned note axis + a discrete + // action (so a pad can fire commit/perturb). Batch learn binds CCs only. + this.updateNote(d1, channel, 1); + this.emitAction({ source: this.kind, id: `note:${d1}`, label: `Note ${d1}`, value: d2 / 127, phase: 'press' }); } else if (status === STATUS_BYTE_NOTE_OFF || (status === STATUS_BYTE_NOTE_ON && d2 === 0)) { - this.handleBindable('note', d1, channel, 0, /*onlyUpdate*/ true); + this.updateNote(d1, channel, 0); + this.emitAction({ source: this.kind, id: `note:${d1}`, label: `Note ${d1}`, value: 0, phase: 'release' }); } } /** - * Route an incoming bindable message: update a matching binding's value, or — - * if learn is armed — create a new axis binding for it. + * Route an incoming CC: update a matching binding's value, or — if batch + * learn is armed — capture it as a NEW axis (deduped). Learn stays armed so + * the user can sweep their whole control surface in one pass. */ - private handleBindable( - kind: MidiBindingKind, - number: number, - channel: number, - value: number, - onlyUpdate = false, - ): void { + private handleCc(number: number, channel: number, value: number): void { const existing = this.bindings.find( - (b) => b.kind === kind && b.number === number && b.channel === channel, + (b) => b.kind === 'cc' && b.number === number && b.channel === channel, ); if (existing) { existing.value = value; this.notifyBindings(); return; } - if (onlyUpdate) return; // note-off for an unbound note: ignore if (this.learnArmed) { - const label = - kind === 'cc' ? `CC${number} ch${channel}` : `Note ${number} ch${channel}`; - this.bindings.push({ kind, number, channel, value, label }); - this.learnArmed = false; // learn one binding per arm - this.setStatus(this.readyStatus()); + this.bindings.push({ kind: 'cc', number, channel, value, label: `CC${number} ch${channel}` }); + this.notifyBindings(); + } + } + + /** Update a learned note axis's gate value (note bindings are not auto-learned). */ + private updateNote(number: number, channel: number, value: number): void { + const existing = this.bindings.find( + (b) => b.kind === 'note' && b.number === number && b.channel === channel, + ); + if (existing) { + existing.value = value; this.notifyBindings(); } } diff --git a/manifold/src/inputs/types.ts b/manifold/src/inputs/types.ts index e14d62d..35efbc1 100644 --- a/manifold/src/inputs/types.ts +++ b/manifold/src/inputs/types.ts @@ -44,6 +44,17 @@ export interface InputSourceStatus { /** Stable identity of a source kind. */ export type InputSourceKind = 'xy-pad' | 'midi' | 'gamepad'; +/** + * The exclusive INPUT MODE the user picks in the Inputs dock. Unlike the + * lower-level {@link InputSourceKind} (which the InputLayer can compose), the + * dock surfaces exactly one mode at a time: + * + * - `internal` → the on-screen XY pad / manifold (default; today's behaviour). + * - `gamepad` → a physical game controller (sticks → axes, buttons → verdicts). + * - `midi` → a connected MIDI device (learned CCs → axes). + */ +export type InputMode = 'internal' | 'gamepad' | 'midi'; + /** * A momentary discrete action surfaced by a source (e.g. a MIDI note-on or a * gamepad face-button press). Fanned out to InputLayer action listeners so the @@ -57,6 +68,13 @@ export interface InputAction { label: string; /** 0..1 velocity / analogue value where meaningful (else 1 for a press). */ value: number; + /** + * Edge phase. `press` (the default) fires on the leading edge; `release` on + * the trailing edge. Hold-and-move bindings (e.g. "hold a button, move the + * stick, release to drop an example") need both edges — most consumers only + * care about `press`. + */ + phase?: 'press' | 'release'; } /** diff --git a/manifold/src/inputs/useInputLayer.ts b/manifold/src/inputs/useInputLayer.ts index 8db7700..476e4c7 100644 --- a/manifold/src/inputs/useInputLayer.ts +++ b/manifold/src/inputs/useInputLayer.ts @@ -2,20 +2,20 @@ * useInputLayer — the thin React binding over the framework-neutral * {@link InputLayer} + source adapters. * - * Owns: - * - ONE InputLayer + one instance of each source (XY pad / MIDI / gamepad), - * created per engine and attached to it. - * - Which sources are ENABLED (the dock toggles these); enabling starts a - * source (async for MIDI) and adds it to the layer's composed set. - * - Per-source config (gamepad stick mode; MIDI learn arm + bindings). - * - The composed channel layout + per-source status, surfaced for the drawer. + * The dock surfaces ONE exclusive input MODE at a time (inputs-spec): + * - `internal` → the on-screen XY pad / manifold (default; today's behaviour). + * - `gamepad` → a physical game controller (sticks → axes, buttons → verdicts). + * - `midi` → a connected MIDI device (CCs learned onto axes). * - * The XY pad source is the one consumers push into directly: `pushPad(x,y)` is - * called from ConsoleApp.onMove so the existing pad keeps working unchanged - * while still composing with the other sources. + * Switching mode stops the previous source and starts the chosen one, then sets + * the layer's composed source set to exactly that source. The XY pad is the one + * consumers push into directly (`pushPad` from ConsoleApp.onMove) so the manifold + * keeps working unchanged in `internal` mode. * - * Discrete actions (MIDI notes / gamepad buttons) are fanned out via - * `onAction` so the console can later bind them to verdicts (commit/perturb). + * Per-mode config (gamepad stick mode + button verdict legend; MIDI device pick, + * batch learn arm, learned bindings) and per-source status are surfaced for the + * drawer. Discrete actions (gamepad buttons / MIDI notes) are fanned out via + * `onAction` so the console can bind them to verdicts (commit/perturb/…). */ import { useCallback, useEffect, useMemo, useRef, useState } from 'react'; import type { EngineApi } from '../engine'; @@ -23,7 +23,13 @@ import { InputLayer } from './input-layer'; import { XYPadSource } from './xy-pad-source'; import { WebMidiInputSource, type MidiBinding } from './midi-input-source'; import { GamepadSource, type StickMode } from './gamepad-source'; -import type { InputAction, InputSource, InputSourceKind, InputSourceStatus } from './types'; +import type { + InputAction, + InputMode, + InputSource, + InputSourceKind, + InputSourceStatus, +} from './types'; export interface SourceView { kind: InputSourceKind; @@ -33,13 +39,25 @@ export interface SourceView { axisCount: number; } +/** Which source backs each exclusive input mode. */ +const MODE_SOURCE: Record = { + internal: 'xy-pad', + gamepad: 'gamepad', + midi: 'midi', +}; + export interface UseInputLayer { /** Push the on-screen XY pad position (∈ [0,1]) — call from onMove. */ pushPad: (x: number, y: number) => void; - /** Per-source enable + status + axis count for the dock. */ + + // ---- exclusive mode ---- + /** The active input mode (Internal / Game Controller / MIDI). */ + inputMode: InputMode; + /** Switch the exclusive input mode. */ + setInputMode: (m: InputMode) => void; + + /** Per-source status + axis count for the dock (the active mode's source is `enabled`). */ sources: SourceView[]; - /** Toggle a source on/off. */ - setEnabled: (kind: InputSourceKind, enabled: boolean) => void; /** Composed channel layout (per-axis source+label). */ channelLayout: { source: string; label: string }[]; /** Total composed axis count. */ @@ -51,16 +69,23 @@ export interface UseInputLayer { gamepadStickMode: StickMode; setGamepadStickMode: (m: StickMode) => void; - // ---- midi learn-map ---- + // ---- midi device + learn-map ---- + /** Available MIDI input ports. */ + midiInputs: { id: string; name: string }[]; + /** The selected MIDI input port (null = listen to all ports). */ + midiDeviceId: string | null; + selectMidiDevice: (id: string | null) => void; + /** True while batch MIDI-Learn is armed (sweep controls, then Done). */ midiLearnArmed: boolean; armMidiLearn: (armed: boolean) => void; midiBindings: MidiBinding[]; clearMidiBinding: (i: number) => void; clearMidiBindings: () => void; - midiInputs: { id: string; name: string }[]; /** Subscribe to discrete actions (notes/buttons). */ onAction: (cb: (a: InputAction) => void) => () => void; + /** Subscribe to the reduced 2D input each frame (for the on-screen manifold). */ + onReducedInput: (cb: (x: number, y: number) => void) => () => void; } export function useInputLayer(engine: EngineApi | null): UseInputLayer { @@ -81,12 +106,7 @@ export function useInputLayer(engine: EngineApi | null): UseInputLayer { const midi = midiRef.current!; const gamepad = gamepadRef.current!; - // Enabled set — pad on by default (parity with today's behaviour). - const [enabled, setEnabledSet] = useState>({ - 'xy-pad': true, - midi: false, - gamepad: false, - }); + const [inputMode, setInputModeState] = useState('internal'); const [statuses, setStatuses] = useState>({ 'xy-pad': pad.status(), midi: midi.status(), @@ -97,8 +117,9 @@ export function useInputLayer(engine: EngineApi | null): UseInputLayer { const [midiLearnArmed, setMidiLearnArmed] = useState(false); const [midiBindings, setMidiBindings] = useState([]); const [midiInputs, setMidiInputs] = useState<{ id: string; name: string }[]>([]); + const [midiDeviceId, setMidiDeviceId] = useState(null); - // Attach to engine; start the pad immediately. Wire status/binding listeners. + // Attach to engine; start in `internal` mode (the XY pad). Wire listeners. useEffect(() => { if (!engine) return; layer.attach(engine); @@ -108,13 +129,13 @@ export function useInputLayer(engine: EngineApi | null): UseInputLayer { const unsubs: (() => void)[] = []; const wireStatus = (s: InputSource) => - unsubs.push( - s.onStatusChange((st) => setStatuses((m) => ({ ...m, [s.kind]: st }))), - ); + unsubs.push(s.onStatusChange((st) => setStatuses((m) => ({ ...m, [s.kind]: st })))); wireStatus(pad); wireStatus(midi); wireStatus(gamepad); unsubs.push(layer.onLayoutChange(() => setLayoutTick((t) => t + 1))); + // Refresh the device-picker list when ports come and go (hot-plug). + unsubs.push(midi.onStatusChange(() => setMidiInputs(midi.listInputs()))); unsubs.push( midi.onBindingsChange((b) => { setMidiBindings(b); @@ -131,34 +152,37 @@ export function useInputLayer(engine: EngineApi | null): UseInputLayer { // eslint-disable-next-line react-hooks/exhaustive-deps }, [engine]); - // Recompose the active source set whenever the enabled set changes. + // Recompose the active source set whenever the mode changes. useEffect(() => { - const active: InputSource[] = []; - if (enabled['xy-pad']) active.push(pad); - if (enabled.midi) active.push(midi); - if (enabled.gamepad) active.push(gamepad); - layer.setSources(active); + const kind = MODE_SOURCE[inputMode]; + const src = kind === 'xy-pad' ? pad : kind === 'gamepad' ? gamepad : midi; + layer.setSources([src]); // eslint-disable-next-line react-hooks/exhaustive-deps - }, [enabled]); + }, [inputMode]); - const setEnabled = useCallback( - (kind: InputSourceKind, on: boolean) => { - setEnabledSet((m) => ({ ...m, [kind]: on })); - if (kind === 'midi') { - if (on) { - void midi.start().then(() => setMidiInputs(midi.listInputs())); - } else { - void midi.stop(); - } - } else if (kind === 'gamepad') { - if (on) gamepad.start(); - else gamepad.stop(); - } else if (kind === 'xy-pad') { - if (on) pad.start(); + const setInputMode = useCallback( + (mode: InputMode) => { + setInputModeState((prev) => { + if (prev === mode) return prev; + // Stop the outgoing source, start the incoming one. + if (prev === 'gamepad') gamepad.stop(); + else if (prev === 'midi') void midi.stop(); else pad.stop(); - } + + if (mode === 'gamepad') { + gamepad.start(); + } else if (mode === 'midi') { + void midi.start().then(() => { + setMidiInputs(midi.listInputs()); + setMidiDeviceId(midi.getSelectedDeviceId()); + }); + } else { + pad.start(); + } + return mode; + }); }, - [midi, gamepad, pad], + [pad, midi, gamepad], ); const setGamepadStickMode = useCallback( @@ -170,6 +194,15 @@ export function useInputLayer(engine: EngineApi | null): UseInputLayer { [gamepad], ); + const selectMidiDevice = useCallback( + (id: string | null) => { + midi.selectDevice(id); + setMidiDeviceId(id); + setMidiInputs(midi.listInputs()); + }, + [midi], + ); + const armMidiLearn = useCallback( (armed: boolean) => { midi.armLearn(armed); @@ -194,42 +227,50 @@ export function useInputLayer(engine: EngineApi | null): UseInputLayer { const pushPad = useCallback((x: number, y: number) => pad.pushAxes(x, y), [pad]); const onAction = useCallback((cb: (a: InputAction) => void) => layer.onAction(cb), [layer]); + const onReducedInput = useCallback( + (cb: (x: number, y: number) => void) => layer.onReducedInput(cb), + [layer], + ); const sources: SourceView[] = useMemo( () => ([pad, midi, gamepad] as InputSource[]).map((s) => ({ kind: s.kind, label: s.label, - enabled: enabled[s.kind], + enabled: MODE_SOURCE[inputMode] === s.kind, status: statuses[s.kind], - axisCount: enabled[s.kind] ? s.axisCount() : 0, + axisCount: MODE_SOURCE[inputMode] === s.kind ? s.axisCount() : 0, })), // layoutTick forces recompute when axis counts shift (learn-map / stick mode). // eslint-disable-next-line react-hooks/exhaustive-deps - [enabled, statuses, layoutTick, pad, midi, gamepad], + [inputMode, statuses, layoutTick, pad, midi, gamepad], ); const channelLayout = useMemo( () => layer.channelLayout(), // eslint-disable-next-line react-hooks/exhaustive-deps - [layoutTick, enabled], + [layoutTick, inputMode], ); return { pushPad, + inputMode, + setInputMode, sources, - setEnabled, channelLayout, axisCount: channelLayout.length, engineInputSize: engine?.architecture.inputSize ?? 2, gamepadStickMode, setGamepadStickMode, + midiInputs, + midiDeviceId, + selectMidiDevice, midiLearnArmed, armMidiLearn, midiBindings, clearMidiBinding, clearMidiBindings, - midiInputs, onAction, + onReducedInput, }; } diff --git a/nisps/CMakeLists.txt b/nisps/CMakeLists.txt index 8ff80d2..a6be410 100644 --- a/nisps/CMakeLists.txt +++ b/nisps/CMakeLists.txt @@ -165,8 +165,14 @@ if(NOT EMSCRIPTEN) if(CMAKE_CXX_COMPILER_ID MATCHES "GNU|Clang|AppleClang") target_compile_options(nisps_parity_check PRIVATE -Wall -Wextra -Werror -Wpedantic + # Disable FP multiply-add contraction so native matches the WASM + # build, which has no FMA instruction. Without this, native clang/gcc + # fuses MACs in the training backprop and the (chaotic) loop amplifies + # the rounding difference past the 1e-5 parity tolerance — pronounced + # since the input layer widened to 32 for mix-and-match inputs. + -ffp-contract=off ) elseif(MSVC) - target_compile_options(nisps_parity_check PRIVATE /W4 /WX) + target_compile_options(nisps_parity_check PRIVATE /W4 /WX /fp:precise) endif() endif() diff --git a/nisps/wasm/bindings.cpp b/nisps/wasm/bindings.cpp index 0404e68..fdfa3f3 100644 --- a/nisps/wasm/bindings.cpp +++ b/nisps/wasm/bindings.cpp @@ -5,24 +5,27 @@ // 2. AudioWorklet processor (playground/src/audio/worklet/...) — engine // calls. (Each instance owns its own WASM module instance.) // -// FIXED-ARCHITECTURE LIMITATION (VERY IMPORTANT) -// ---------------------------------------------- +// ARCHITECTURE (input dim is OVER-PROVISIONED for mix-and-match inputs) +// --------------------------------------------------------------------- // The C++ MLP class is templated on layer sizes (architecture.md §4.1, §6.2). // We instantiate ONE concrete configuration here: // -// using DefaultMLP = nisps::ml::MLP<2, 10, 14, 18, 126>; +// using DefaultMLP = nisps::ml::MLP<32, 10, 14, 18, 126>; // -// This was chosen as the union of the playground use case (2-D joystick → -// 126 synth params) and the largest hidden-layer footprint that still fits -// firmware budgets. `nisps_ml_create()` accepts caller-supplied input_size, -// output_size, hidden[], n_hidden but ONLY validates them against the -// compile-time defaults — extra inputs/outputs are clipped at the boundary. -// If the caller passes incompatible dimensions we still create the module: -// extra inputs are zero-padded, extra outputs are truncated, and the -// hidden-layer override is silently ignored. +// The 32-input dimension is the MAX number of composed input axes the manifold +// front-end can feed (matches MAX_AXES in manifold/src/inputs/input-layer.ts). +// The mix-and-match input layer (Internal XY pad + Game Controller + MIDI) gives +// each active axis its OWN dedicated input slot — NO mean-blending — and feeds +// the remaining (unused) slots a constant 0. The "active input dimension count" +// is a front-end concept: a 2-axis pad uses slots 0–1, a 4-axis pad+stick uses +// 0–3, etc. Because slot assignment is stable and unused slots are held at 0, +// the net behaves as an N-input net where N = active axes; changing N is a +// reshape, after which the front-end resets the weights (recreate-from-scratch, +// behind a confirm modal). 126 outputs cover C15 + any current schema. // -// Future work: ship multiple WASM modules (one per common architecture) or -// rebuild on demand. See architecture.md "open questions" — Stream 7 punts. +// `nisps_ml_create()` accepts caller-supplied input_size/output_size/hidden[] +// but only validates them against these compile-time defaults — extra +// inputs/outputs are clipped at the boundary and hidden overrides are ignored. // // WIRE FORMAT FOR WEIGHTS // ----------------------- @@ -85,7 +88,7 @@ namespace { // * 126 outputs — enough for the C15 mode and any current schema. // // The MLP also has dataset slots, loss history etc. — see mlp.hpp. -using DefaultMLP = nisps::ml::MLP<2u, 10u, 14u, 18u, 126u>; +using DefaultMLP = nisps::ml::MLP<32u, 10u, 14u, 18u, 126u>; constexpr std::size_t kDefaultInputs = DefaultMLP::kInput; constexpr std::size_t kDefaultOutputs = DefaultMLP::kOutput; diff --git a/playground/public/nisps.wasm b/playground/public/nisps.wasm index d2226c2..9ee47b2 100755 Binary files a/playground/public/nisps.wasm and b/playground/public/nisps.wasm differ diff --git a/tests/cpp/parity_check.cpp b/tests/cpp/parity_check.cpp index 99bf291..82da3c3 100644 --- a/tests/cpp/parity_check.cpp +++ b/tests/cpp/parity_check.cpp @@ -21,7 +21,7 @@ // 4. ChannelStrip engine: identical methodology. // // We use the EXACT SAME compile-time MLP architecture as the WASM build: -// MLP<2, 10, 14, 18, 126> +// MLP<32, 10, 14, 18, 126> (32-input max for mix-and-match; see bindings.cpp) // // Output blob format // ------------------ @@ -62,7 +62,7 @@ namespace { -using ParityMLP = nisps::ml::MLP<2u, 10u, 14u, 18u, 126u>; +using ParityMLP = nisps::ml::MLP<32u, 10u, 14u, 18u, 126u>; // The WASM bindings (nisps/wasm/bindings.cpp) sign-extend the 32-bit JS // seed via `s ^ (s << 32)`. To get bit-equal output between native and @@ -133,9 +133,14 @@ int main(int argc, char** argv) { } // ---- Stage 2: training ---- - constexpr std::array, 3u> features = {{ - {{0.1f, 0.9f}}, {{0.5f, 0.5f}}, {{0.9f, 0.1f}}, - }}; + // Feature vectors are NIn(32)-wide: two real axes + zero-pad (the front-end + // feeds the same shape — active axes in the low slots, unused slots at 0). + // add_example requires features.size() >= NIn, so the pad is mandatory. + constexpr std::size_t kNIn = ParityMLP::kInput; + std::array, 3u> features = {}; + features[0][0] = 0.1f; features[0][1] = 0.9f; + features[1][0] = 0.5f; features[1][1] = 0.5f; + features[2][0] = 0.9f; features[2][1] = 0.1f; auto label_for = [](std::size_t i) { std::array out{}; const float a = static_cast(i) * 0.3f + 0.05f; @@ -146,7 +151,7 @@ int main(int argc, char** argv) { }; for (std::size_t i = 0; i < features.size(); ++i) { const auto label = label_for(i); - mlp.add_example(std::span(features[i].data(), 2u), + mlp.add_example(std::span(features[i].data(), kNIn), std::span(label.data(), 126u)); } const float final_loss = mlp.train(0.3f, 50u, 0.0f); diff --git a/tests/cpp/parity_wasm.mjs b/tests/cpp/parity_wasm.mjs index 840242b..afa3f7b 100644 --- a/tests/cpp/parity_wasm.mjs +++ b/tests/cpp/parity_wasm.mjs @@ -191,8 +191,8 @@ async function main() { api.describe(dimsBuf); const dims = new Int32Array(Module.HEAP32.buffer, dimsBuf, 6).slice(); api.free(dimsBuf); - // Expect: [2, 10, 14, 18, 126, 4] - const expectedDims = [2, 10, 14, 18, 126, 4]; + // Expect: [32, 10, 14, 18, 126, 4] (32-input max for mix-and-match) + const expectedDims = [32, 10, 14, 18, 126, 4]; for (let i = 0; i < expectedDims.length; ++i) { if (dims[i] !== expectedDims[i]) { console.error(`[parity_wasm] WASM build has dim[${i}]=${dims[i]}, native expected ${expectedDims[i]}`); @@ -200,10 +200,11 @@ async function main() { process.exit(2); } } + const N_IN = dims[0]; const N_OUT = dims[4]; // --- Stage 1: ML inference --- - const ml = api.create(2, N_OUT, 0, 0, SEED); + const ml = api.create(N_IN, N_OUT, 0, 0, SEED); api.drawWeights(ml, 0.5); api.setInput(ml, 0, INPUT_X); api.setInput(ml, 1, INPUT_Y); @@ -226,10 +227,13 @@ async function main() { for (let j = 0; j < N_OUT; ++j) out[j] = a + 0.005 * j; return out; }; - const featBuf = api.malloc(2 * 4); - const featF32 = new Float32Array(api.HEAPF32.buffer, featBuf, 2); + // Feature buffer is NIn-wide (zero-padded): two real axes + unused slots at 0, + // matching the native side and the front-end's mix-and-match input shape. + const featBuf = api.malloc(N_IN * 4); + const featF32 = new Float32Array(api.HEAPF32.buffer, featBuf, N_IN); const labelBuf = api.malloc(N_OUT * 4); for (let i = 0; i < features.length; ++i) { + featF32.fill(0); featF32[0] = features[i][0]; featF32[1] = features[i][1]; const label = labelFor(i);