memlnaut-nisps/docs/specs/recon/midi-gamepad-inputs-worklog.md

kind	date	immutable
finding	2026-06-27	true

Work log — MIDI + Game Controller inputs, N-D engine foundation

Scope: what was actually built on the feat/midi-inputs branch. This is a description of the work, not a spec. The design intent lives in docs/specs/inputs-spec.md; where this branch diverges from or only partially realises that spec, it is called out below.

Summary

This branch wires the modular input layer (which already existed as adapters in manifold/src/inputs/) into the Console, adds the missing gamepad→verdict and MIDI-device plumbing, and reshapes the browser ML engine so input axes are genuine independent dimensions instead of being blended into two. It landed in two passes:

1. Input methods — a working Inputs dock with three sources (Internal XY pad, Game Controller, MIDI), gamepad buttons bound to verdicts, and a batch "MIDI Learn".
2. Engine foundation for mixing — the WASM net was widened from 2 inputs to a 32-input maximum so each active axis gets its own dimension (no blending).

The Inputs dock currently presents the three sources as an exclusive picker. The engine groundwork for mixing sources (independent dimensions, no idle-bias) is in place, but the dock toggles, the reshape-confirm modal, and the

2-dimension slider visualisation described in inputs-spec.md are not yet wired — see "Not done yet" below.

What changed

Input sources (manifold/src/inputs/)

• gamepad-source.ts — buttons now emit both press and release edges (with standard-mapping labels A/B/X/Y/LB/RB/…), enabling hold-and-move gestures. Single/double-stick (2/4 axes) was already present.
• midi-input-source.ts — added single-device selection (selectDevice, the dock device picker; default still listens to all ports) and changed MIDI-Learn from one-binding-per-arm to a batch capture: while armed, every distinct CC that moves is appended as an axis; notes stay discrete actions and are not auto-bound. Learned CCs are exposed as bindings for the dock.
• types.ts — InputAction gained an optional phase ('press' | 'release'); added an InputMode ('internal' | 'gamepad' | 'midi') type.
• input-layer.ts — added onReducedInput so the on-screen manifold can track a gamepad/MIDI-driven position. compose() no longer mean-blends: it forwards each active axis 1:1 to its own engine input slot (the engine zero-pads the rest; a zero input is inert, 0 × weight = 0).
• useInputLayer.ts — the React binding; exposes the active mode, per-source status, gamepad stick mode, MIDI device list/selection, batch-learn arm, and learned bindings. (Currently exclusive — one mode at a time.)

Console wiring (manifold/src/console/)

• ConsoleApp.tsx — subscribes to gamepad actions and binds them to existing verdict handlers: RB = thumbs-up, LB = thumbs-down, X = randomise, Y = nudge, B = undo, A-hold = reposition (hold, move stick, release to place an example at the stick position). Mirrors the composed input position onto the manifold when a non-pad source is active (deduped to avoid per-frame re-renders).
• Drawers.tsx — rebuilt the Inputs drawer: a source picker, a gamepad stick toggle + button legend, a MIDI device picker, the batch MIDI-Learn flow with its "move every control, then Done" message, and learned controls rendered as read-only meters styled distinctly from the output sliders.

Engine (manifold/src/engine/, nisps/wasm/)

• nisps/wasm/bindings.cpp — DefaultMLP widened MLP<2,…> → MLP<32,…> (32 = MAX_AXES). Each active axis maps to a dedicated input slot; unused slots are held at 0. Rebuilt nisps.wasm and synced to both playground/public/ and manifold/public/ (the C ABI / nisps.js glue is unchanged).
• spine.ts / engine-api.ts — setInputs(arr) now writes the full N-dimensional vector (it previously dropped everything past arr[1]); the primary pair still runs through the 2-D input pipeline so the pad keeps its feel, axes 2+ are written raw, and process() re-ticks the whole vector after weight changes via the new spine.reprocess().

Tests / build

• tests/cpp/parity_check.cpp + tests/cpp/parity_wasm.mjs — ParityMLP bumped to 32 inputs and the example/feature buffers widened to match the net's arity (add_example requires features.size() >= NIn).
• nisps/CMakeLists.txt — the parity binary now builds with -ffp-contract=off. Widening the input layer exposed a native↔WASM divergence: native clang/gcc fuse multiply-adds (FMA) the WASM build has no instruction for, and the training loop amplified the rounding difference past the 1e-5 parity tolerance. Disabling FP contraction on the native parity build alone restores bit-equality (max delta ~2.4e-7).

Verification

• C++ suites (4/4) pass; native↔WASM parity passes at 1e-5.
• manifold typechecks and builds; the Playwright smoke test (engine loads, input→output propagates) passes.

Not done yet (vs inputs-spec.md)

• The Inputs dock is an exclusive picker; mixing several sources at once (independent toggles) is not wired, though the engine and compose() now support it.
• No reshape-confirm modal + net reset when the active input set changes.
• No swap to a slider visualisation when more than two input dimensions are active (the 2-D manifold is always shown).
• The input pipeline (deadzone/zoom/curve) is applied only to the primary pair; per-source conditioning for axes 2+ is left raw.