feat(manifold): MIDI + game controller inputs; widen ML net to N-D
Wire the modular input layer into the Console and reshape the browser engine so input axes are genuine independent dimensions. Inputs (manifold/src/inputs/): - gamepad-source: emit press+release edges with standard-mapping labels (enables hold-and-move); single/double-stick already present. - midi-input-source: single-device selection + batch "MIDI Learn" (every CC swept while armed becomes an axis); notes stay discrete. - input-layer: compose() forwards each axis 1:1 (no mean-blend); add onReducedInput so the manifold tracks gamepad/MIDI position. - types: InputAction.phase, InputMode. Console (manifold/src/console/): - ConsoleApp: bind gamepad buttons to verdicts (RB up / LB down / X randomise / Y nudge / B undo / A-hold reposition); mirror composed position onto the manifold. - Drawers: rebuilt Inputs drawer (source picker, gamepad legend, MIDI device picker + batch-learn flow, learned-control meters). Engine (nisps/wasm, manifold/src/engine): - DefaultMLP widened MLP<2,..> -> MLP<32,..> (32 = MAX_AXES); each active axis gets a dedicated slot, unused slots held at 0 (inert). Rebuilt nisps.wasm (playground + manifold). - spine/engine-api: setInputs writes the full N-D vector (was dropping arr[2+]); primary pair keeps the 2-D pipeline; process() re-ticks the whole vector via spine.reprocess(). Tests: - parity_check/parity_wasm: ParityMLP -> 32 inputs, widen example bufs. - CMakeLists: build parity binary with -ffp-contract=off so native matches FMA-free WASM (training amplified the gap past 1e-5). Inputs dock is still an exclusive picker; mixing toggles, reshape modal, and the >2-D slider view (inputs-spec.md) are groundwork-laid but not yet wired. See docs/redesign/midi-gamepad-inputs-worklog.md.
This commit is contained in:
monkey-w1n5t0n
parent
115fd501b5
commit
9e59eb04ce
18 changed files with 739 additions and 321 deletions
8
MAP.md
8
MAP.md
|
|
@ -66,6 +66,14 @@ anchor + locked decisions) and the `docs/redesign/*-spec.md` set.
|
|||
- `manifold/src/midi-devices/` — external-synth device templates. `generated/` is codegen output from
|
||||
`schemas/midi_devices/` (`MIDI_DEVICES` catalogue + `MIDI_DEVICES_BY_ID`, params by name+CC). The MIDI Outputs
|
||||
config (`dock/OutputsBackendConfig.tsx`) reads it for the device picker + param-select that fills the CC table.
|
||||
- `manifold/src/inputs/` — modular INPUT layer feeding the ML head. The Inputs dock picks ONE exclusive mode
|
||||
(`InputMode` = `internal` | `gamepad` | `midi`; Internal/XY-pad is default). `input-layer.ts` owns a single rAF
|
||||
loop composing the active source's axes → reduced to the engine arity (fixed 2-in WASM → even/odd blend) → one
|
||||
`setInputs`, plus an `onReducedInput` callback the manifold tracks. Sources: `xy-pad-source` (push-driven),
|
||||
`gamepad-source` (sticks→axes single/double; buttons emit press+release actions, bound in `ConsoleApp` to
|
||||
verdicts — LB/RB=down/up, X/Y/B=randomise/nudge/undo, A-hold=reposition), `midi-input-source` (device picker +
|
||||
BATCH "MIDI Learn": every CC swept while armed becomes an axis, shown as read-only meters). `useInputLayer.ts`
|
||||
is the React binding; `base-source.ts` shared status/action plumbing; `types.ts` the adapter contract.
|
||||
- `manifold/src/feedback/` — `controller.ts` (Explore-and-place scratchpad + geometric-dislike + solo, TS
|
||||
prototype), `rng.ts` (seeded).
|
||||
- `manifold/src/settings/` — `settings-store.ts` (monochrome icons, input-map shape, corner radius).
|
||||
|
|
|
|||
96
docs/redesign/midi-gamepad-inputs-worklog.md
Normal file
96
docs/redesign/midi-gamepad-inputs-worklog.md
Normal file
|
|
@ -0,0 +1,96 @@
|
|||
# Work log — MIDI + Game Controller inputs, N-D engine foundation
|
||||
|
||||
*Scope: what was actually built on the `feat/midi-inputs` branch. This is a
|
||||
description of the work, not a spec. The design intent lives in
|
||||
`docs/redesign/inputs-spec.md`; where this branch diverges from or only partially
|
||||
realises that spec, it is called out below.*
|
||||
|
||||
## Summary
|
||||
|
||||
This branch wires the modular input layer (which already existed as adapters in
|
||||
`manifold/src/inputs/`) into the Console, adds the missing gamepad→verdict and
|
||||
MIDI-device plumbing, and reshapes the browser ML engine so input axes are
|
||||
genuine independent dimensions instead of being blended into two. It landed in
|
||||
two passes:
|
||||
|
||||
1. **Input methods** — a working Inputs dock with three sources (Internal XY
|
||||
pad, Game Controller, MIDI), gamepad buttons bound to verdicts, and a batch
|
||||
"MIDI Learn".
|
||||
2. **Engine foundation for mixing** — the WASM net was widened from 2 inputs to
|
||||
a 32-input maximum so each active axis gets its own dimension (no blending).
|
||||
|
||||
The Inputs dock currently presents the three sources as an **exclusive** picker.
|
||||
The engine groundwork for *mixing* sources (independent dimensions, no idle-bias)
|
||||
is in place, but the dock toggles, the reshape-confirm modal, and the
|
||||
>2-dimension slider visualisation described in `inputs-spec.md` are **not yet
|
||||
wired** — see "Not done yet" below.
|
||||
|
||||
## What changed
|
||||
|
||||
### Input sources (`manifold/src/inputs/`)
|
||||
- `gamepad-source.ts` — buttons now emit both press and release edges (with
|
||||
standard-mapping labels A/B/X/Y/LB/RB/…), enabling hold-and-move gestures.
|
||||
Single/double-stick (2/4 axes) was already present.
|
||||
- `midi-input-source.ts` — added single-device selection (`selectDevice`, the
|
||||
dock device picker; default still listens to all ports) and changed MIDI-Learn
|
||||
from one-binding-per-arm to a **batch** capture: while armed, every distinct CC
|
||||
that moves is appended as an axis; notes stay discrete actions and are not
|
||||
auto-bound. Learned CCs are exposed as bindings for the dock.
|
||||
- `types.ts` — `InputAction` gained an optional `phase` ('press' | 'release');
|
||||
added an `InputMode` ('internal' | 'gamepad' | 'midi') type.
|
||||
- `input-layer.ts` — added `onReducedInput` so the on-screen manifold can track a
|
||||
gamepad/MIDI-driven position. **`compose()` no longer mean-blends**: it
|
||||
forwards each active axis 1:1 to its own engine input slot (the engine
|
||||
zero-pads the rest; a zero input is inert, `0 × weight = 0`).
|
||||
- `useInputLayer.ts` — the React binding; exposes the active mode, per-source
|
||||
status, gamepad stick mode, MIDI device list/selection, batch-learn arm, and
|
||||
learned bindings. (Currently exclusive — one mode at a time.)
|
||||
|
||||
### Console wiring (`manifold/src/console/`)
|
||||
- `ConsoleApp.tsx` — subscribes to gamepad actions and binds them to existing
|
||||
verdict handlers: RB = thumbs-up, LB = thumbs-down, X = randomise, Y = nudge,
|
||||
B = undo, A-hold = reposition (hold, move stick, release to place an example
|
||||
at the stick position). Mirrors the composed input position onto the manifold
|
||||
when a non-pad source is active (deduped to avoid per-frame re-renders).
|
||||
- `Drawers.tsx` — rebuilt the Inputs drawer: a source picker, a gamepad stick
|
||||
toggle + button legend, a MIDI device picker, the batch MIDI-Learn flow with
|
||||
its "move every control, then Done" message, and learned controls rendered as
|
||||
read-only meters styled distinctly from the output sliders.
|
||||
|
||||
### Engine (`manifold/src/engine/`, `nisps/wasm/`)
|
||||
- `nisps/wasm/bindings.cpp` — `DefaultMLP` widened `MLP<2,…>` → `MLP<32,…>`
|
||||
(32 = `MAX_AXES`). Each active axis maps to a dedicated input slot; unused
|
||||
slots are held at 0. Rebuilt `nisps.wasm` and synced to both
|
||||
`playground/public/` and `manifold/public/` (the C ABI / `nisps.js` glue is
|
||||
unchanged).
|
||||
- `spine.ts` / `engine-api.ts` — `setInputs(arr)` now writes the full
|
||||
N-dimensional vector (it previously dropped everything past `arr[1]`); the
|
||||
primary pair still runs through the 2-D input pipeline so the pad keeps its
|
||||
feel, axes 2+ are written raw, and `process()` re-ticks the whole vector after
|
||||
weight changes via the new `spine.reprocess()`.
|
||||
|
||||
### Tests / build
|
||||
- `tests/cpp/parity_check.cpp` + `tests/cpp/parity_wasm.mjs` — `ParityMLP`
|
||||
bumped to 32 inputs and the example/feature buffers widened to match the net's
|
||||
arity (`add_example` requires `features.size() >= NIn`).
|
||||
- `nisps/CMakeLists.txt` — the parity binary now builds with `-ffp-contract=off`.
|
||||
Widening the input layer exposed a native↔WASM divergence: native clang/gcc
|
||||
fuse multiply-adds (FMA) the WASM build has no instruction for, and the
|
||||
training loop amplified the rounding difference past the 1e-5 parity tolerance.
|
||||
Disabling FP contraction on the native parity build alone restores bit-equality
|
||||
(max delta ~2.4e-7).
|
||||
|
||||
## Verification
|
||||
- C++ suites (4/4) pass; native↔WASM parity passes at 1e-5.
|
||||
- `manifold` typechecks and builds; the Playwright smoke test (engine loads,
|
||||
input→output propagates) passes.
|
||||
|
||||
## Not done yet (vs `inputs-spec.md`)
|
||||
- The Inputs dock is an **exclusive** picker; mixing several sources at once
|
||||
(independent toggles) is not wired, though the engine and `compose()` now
|
||||
support it.
|
||||
- No reshape-confirm modal + net reset when the active input set changes.
|
||||
- No swap to a slider visualisation when more than two input dimensions are
|
||||
active (the 2-D manifold is always shown).
|
||||
- The input pipeline (deadzone/zoom/curve) is applied only to the primary pair;
|
||||
per-source conditioning for axes 2+ is left raw.
|
||||
Binary file not shown.
|
|
@ -546,6 +546,59 @@ export function ConsoleApp({ focus: initialFocus = 'composite' }: ConsoleAppProp
|
|||
return () => window.removeEventListener('keydown', onKey);
|
||||
});
|
||||
|
||||
// ---- Game-controller verdict bindings (inputs-spec) ---------------------
|
||||
// The gamepad's sticks already feed the input layer (→ engine); its BUTTONS
|
||||
// drive verdicts here. Standard-mapping indices:
|
||||
// LB(4) = down/negative · RB(5) = up/positive · X(2) = randomise ·
|
||||
// Y(3) = nudge · B(1) = undo · A(0) hold-and-move = reposition an example
|
||||
// (hold A, move the stick to a spot on the manifold, release to drop it).
|
||||
// MIDI note actions are surfaced too but left unbound (MIDI mode learns CCs
|
||||
// as INPUT axes; verdicts there stay on the on-screen / keyboard controls).
|
||||
// The effect has no dep array (matching the keydown handler above) so each
|
||||
// binding closes over the latest verdict functions + live `pos`.
|
||||
useEffect(() => {
|
||||
const unBtn = inputs.onAction((a) => {
|
||||
if (a.source !== 'gamepad') return;
|
||||
const phase = a.phase ?? 'press';
|
||||
if (phase === 'press') {
|
||||
switch (a.id) {
|
||||
case 'button:4': // LB → thumbs-down
|
||||
perturb();
|
||||
break;
|
||||
case 'button:5': // RB → thumbs-up
|
||||
commit();
|
||||
break;
|
||||
case 'button:2': // X → randomise / re-roll
|
||||
reroll();
|
||||
break;
|
||||
case 'button:3': // Y → nudge (scratchpad)
|
||||
onScratchNudge();
|
||||
break;
|
||||
case 'button:1': // B → undo
|
||||
undo();
|
||||
break;
|
||||
case 'button:0': // A (down) → begin repositioning an example
|
||||
onPlace();
|
||||
break;
|
||||
}
|
||||
} else if (phase === 'release' && a.id === 'button:0') {
|
||||
// A (up) → drop the example at the current (stick-driven) location.
|
||||
onPickLocation(pos[0], pos[1]);
|
||||
}
|
||||
});
|
||||
// Mirror the composed gamepad/MIDI position onto the on-screen manifold so
|
||||
// markers + readouts track the controller (the XY pad pushes its own pos).
|
||||
// The callback fires every rAF frame — only re-render when it actually moves.
|
||||
const unPos = inputs.onReducedInput((x, y) => {
|
||||
if (inputs.inputMode === 'internal') return;
|
||||
setPos((prev) => (Math.abs(prev[0] - x) < 1e-3 && Math.abs(prev[1] - y) < 1e-3 ? prev : [x, y]));
|
||||
});
|
||||
return () => {
|
||||
unBtn();
|
||||
unPos();
|
||||
};
|
||||
});
|
||||
|
||||
const onToggleAudio = () => {
|
||||
if (!engine) return;
|
||||
if (engine.audio.isStarted) {
|
||||
|
|
|
|||
|
|
@ -23,6 +23,7 @@
|
|||
import type { ReactNode } from 'react';
|
||||
import { Badge, Button, PillToggle, Slider, Switch } from '../primitives';
|
||||
import type { ConsoleCtx, DrawerDepth, DrawerKey, FeedbackModeUI, SoloMode } from './types';
|
||||
import type { InputMode } from '../inputs';
|
||||
import { OutputControlRow } from '../dock/OutputControlRow';
|
||||
import { BackendAdvanced } from '../dock/BackendAdvanced';
|
||||
import { OutputsBackendConfig, BackendStatusChip } from '../dock/OutputsBackendConfig';
|
||||
|
|
@ -236,165 +237,246 @@ const STATUS_TONE: Record<string, string> = {
|
|||
idle: 'var(--fg-dim)',
|
||||
};
|
||||
|
||||
const INPUT_MODE_OPTS: { value: InputMode; label: string }[] = [
|
||||
{ value: 'internal', label: 'Internal' },
|
||||
{ value: 'gamepad', label: 'Game Controller' },
|
||||
{ value: 'midi', label: 'MIDI' },
|
||||
];
|
||||
|
||||
/** Standard-mapping gamepad button → verdict legend (mirrors ConsoleApp). */
|
||||
const GAMEPAD_LEGEND: { btn: string; action: string }[] = [
|
||||
{ btn: 'RB', action: 'Up · positive feedback' },
|
||||
{ btn: 'LB', action: 'Down · negative feedback' },
|
||||
{ btn: 'X', action: 'Randomise' },
|
||||
{ btn: 'Y', action: 'Nudge' },
|
||||
{ btn: 'B', action: 'Undo' },
|
||||
{ btn: 'A (hold)', action: 'Reposition — hold, move stick, release to place' },
|
||||
];
|
||||
|
||||
/**
|
||||
* Read-only INPUT meter — shows a learned MIDI control's live value. Deliberately
|
||||
* styled apart from the output Sliders (which are orange, interactive thumbs):
|
||||
* these are inset bars on the secondary accent with an "in" tag, so the user can
|
||||
* see at a glance that these feed the net rather than being driven by it.
|
||||
*/
|
||||
function MidiInputMeter({ label, value, onClear }: { label: string; value: number; onClear: () => void }) {
|
||||
const pct = Math.max(0, Math.min(1, value));
|
||||
return (
|
||||
<div style={{ display: 'flex', alignItems: 'center', gap: 8 }}>
|
||||
<span
|
||||
style={{
|
||||
fontSize: 9,
|
||||
fontFamily: 'var(--font-mono)',
|
||||
color: 'var(--accent-2)',
|
||||
textTransform: 'uppercase',
|
||||
letterSpacing: '0.06em',
|
||||
minWidth: 70,
|
||||
}}
|
||||
>
|
||||
{label}
|
||||
</span>
|
||||
<div
|
||||
style={{
|
||||
position: 'relative',
|
||||
flex: 1,
|
||||
height: 8,
|
||||
background: 'var(--bg-2)',
|
||||
border: '1px solid var(--line)',
|
||||
borderLeft: '2px solid var(--accent-2)',
|
||||
borderRadius: 'var(--r-1)',
|
||||
overflow: 'hidden',
|
||||
}}
|
||||
>
|
||||
<div
|
||||
style={{
|
||||
position: 'absolute',
|
||||
inset: 0,
|
||||
width: `${pct * 100}%`,
|
||||
background: 'var(--accent-2)',
|
||||
opacity: 0.55,
|
||||
}}
|
||||
/>
|
||||
</div>
|
||||
<span
|
||||
style={{
|
||||
fontSize: 9,
|
||||
fontFamily: 'var(--font-mono)',
|
||||
color: 'var(--fg-mute)',
|
||||
minWidth: 28,
|
||||
textAlign: 'right',
|
||||
}}
|
||||
>
|
||||
{value.toFixed(2)}
|
||||
</span>
|
||||
<button
|
||||
type="button"
|
||||
onClick={onClear}
|
||||
aria-label={`Remove ${label}`}
|
||||
style={{ background: 'transparent', border: 0, color: 'var(--danger)', cursor: 'pointer', fontSize: 'var(--fs-xs)' }}
|
||||
>
|
||||
✕
|
||||
</button>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
function InputsDrawer(ctx: ConsoleCtx, depth: DrawerDepth) {
|
||||
const inp = ctx.inputs;
|
||||
const enabledCount = inp.sources.filter((s) => s.enabled).length;
|
||||
const reshaping = inp.axisCount > inp.engineInputSize;
|
||||
const active = inp.sources.find((s) => s.enabled);
|
||||
const modeLabel = INPUT_MODE_OPTS.find((o) => o.value === inp.inputMode)?.label ?? 'Internal';
|
||||
|
||||
return (
|
||||
<>
|
||||
<div style={{ display: 'flex', gap: 6, flexWrap: 'wrap', alignItems: 'center' }}>
|
||||
<Badge tone="info">
|
||||
{enabledCount === 0 ? 'no source' : `${enabledCount} source${enabledCount > 1 ? 's' : ''}`}
|
||||
</Badge>
|
||||
<Chip tone="var(--accent)">{inp.axisCount} axes</Chip>
|
||||
<Chip>engine: {inp.engineInputSize}-in</Chip>
|
||||
<Badge tone="info">{modeLabel}</Badge>
|
||||
{inp.inputMode !== 'internal' && <Chip tone="var(--accent)">{inp.axisCount} axes</Chip>}
|
||||
{reshaping && <Chip tone="var(--warn)">blended → {inp.engineInputSize}</Chip>}
|
||||
{active && (
|
||||
<Chip tone={STATUS_TONE[active.status.state] ?? 'var(--fg-dim)'}>{active.status.state}</Chip>
|
||||
)}
|
||||
</div>
|
||||
|
||||
<SectionLabel>Sources</SectionLabel>
|
||||
<div style={{ display: 'flex', flexDirection: 'column', gap: 6 }}>
|
||||
{inp.sources.map((s) => (
|
||||
<div
|
||||
key={s.kind}
|
||||
style={{
|
||||
display: 'flex',
|
||||
alignItems: 'center',
|
||||
justifyContent: 'space-between',
|
||||
gap: 8,
|
||||
padding: '4px 8px',
|
||||
background: 'var(--bg-2)',
|
||||
border: '1px solid var(--line)',
|
||||
borderRadius: 'var(--r-1)',
|
||||
}}
|
||||
>
|
||||
<div style={{ display: 'flex', flexDirection: 'column', gap: 2 }}>
|
||||
<span style={{ fontSize: 'var(--fs-xs)', color: 'var(--fg-mute)' }}>
|
||||
{s.label}
|
||||
{s.enabled ? ` · ${s.axisCount} ax` : ''}
|
||||
</span>
|
||||
{depth !== 'peek' && (
|
||||
<span style={{ fontSize: 9, color: STATUS_TONE[s.status.state] ?? 'var(--fg-dim)' }}>
|
||||
{s.status.message}
|
||||
</span>
|
||||
)}
|
||||
</div>
|
||||
<Switch checked={s.enabled} onChange={(v) => inp.setEnabled(s.kind, v)} label="" />
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
<SectionLabel>Input source</SectionLabel>
|
||||
<Segmented value={inp.inputMode} onChange={inp.setInputMode} options={INPUT_MODE_OPTS} />
|
||||
{active && depth !== 'peek' && (
|
||||
<span style={{ fontSize: 9, color: STATUS_TONE[active.status.state] ?? 'var(--fg-dim)' }}>
|
||||
{active.status.message}
|
||||
</span>
|
||||
)}
|
||||
|
||||
{depth !== 'peek' && (
|
||||
{/* ---- Internal (XY pad / manifold) ---- */}
|
||||
{inp.inputMode === 'internal' && depth !== 'peek' && (
|
||||
<p style={{ fontSize: 'var(--fs-xs)', color: 'var(--fg-dim)', margin: 0, lineHeight: 1.6 }}>
|
||||
Drag the on-screen manifold / XY pad. Two axes feed the net directly — this is the default.
|
||||
</p>
|
||||
)}
|
||||
|
||||
{/* ---- Game Controller ---- */}
|
||||
{inp.inputMode === 'gamepad' && depth !== 'peek' && (
|
||||
<>
|
||||
{/* ---- Gamepad config ---- */}
|
||||
{inp.sources.find((s) => s.kind === 'gamepad')?.enabled && (
|
||||
<>
|
||||
<SectionLabel>Gamepad · sticks</SectionLabel>
|
||||
<Segmented
|
||||
value={inp.gamepadStickMode}
|
||||
onChange={inp.setGamepadStickMode}
|
||||
options={[
|
||||
{ value: 'single', label: 'Single (2 ax)' },
|
||||
{ value: 'double', label: 'Double (4 ax)' },
|
||||
]}
|
||||
/>
|
||||
</>
|
||||
)}
|
||||
|
||||
{/* ---- MIDI learn-map ---- */}
|
||||
{inp.sources.find((s) => s.kind === 'midi')?.enabled && (
|
||||
<>
|
||||
<SectionLabel>MIDI · learn-map</SectionLabel>
|
||||
<div style={{ display: 'flex', gap: 6, alignItems: 'center', flexWrap: 'wrap' }}>
|
||||
<Button
|
||||
size="sm"
|
||||
variant={inp.midiLearnArmed ? 'primary' : 'secondary'}
|
||||
onClick={() => inp.armMidiLearn(!inp.midiLearnArmed)}
|
||||
>
|
||||
{inp.midiLearnArmed ? 'learning… (move a control)' : 'Learn axis'}
|
||||
</Button>
|
||||
{inp.midiBindings.length > 0 && (
|
||||
<Button size="sm" variant="secondary" onClick={inp.clearMidiBindings}>
|
||||
Clear all
|
||||
</Button>
|
||||
)}
|
||||
<SectionLabel>Sticks</SectionLabel>
|
||||
<Segmented
|
||||
value={inp.gamepadStickMode}
|
||||
onChange={inp.setGamepadStickMode}
|
||||
options={[
|
||||
{ value: 'single', label: 'One stick (2 ax)' },
|
||||
{ value: 'double', label: 'Both sticks (4 ax)' },
|
||||
]}
|
||||
/>
|
||||
<SectionLabel>Buttons</SectionLabel>
|
||||
<div style={{ display: 'flex', flexDirection: 'column', gap: 3 }}>
|
||||
{GAMEPAD_LEGEND.map((g) => (
|
||||
<div
|
||||
key={g.btn}
|
||||
style={{ display: 'flex', alignItems: 'center', gap: 8, fontSize: 'var(--fs-xs)' }}
|
||||
>
|
||||
<Chip tone="var(--accent)">{g.btn}</Chip>
|
||||
<span style={{ color: 'var(--fg-mute)' }}>{g.action}</span>
|
||||
</div>
|
||||
{inp.midiBindings.length === 0 ? (
|
||||
<p style={{ fontSize: 9, color: 'var(--fg-dim)', margin: 0 }}>
|
||||
No axes learned yet — arm Learn, then wiggle a knob or hit a pad. CCs map to a
|
||||
continuous axis; notes map to a gate (1 while held).
|
||||
</p>
|
||||
) : (
|
||||
<div style={{ display: 'flex', flexDirection: 'column', gap: 3 }}>
|
||||
{inp.midiBindings.map((b, i) => (
|
||||
<div
|
||||
key={`${b.kind}-${b.number}-${b.channel}`}
|
||||
style={{
|
||||
display: 'flex',
|
||||
justifyContent: 'space-between',
|
||||
alignItems: 'center',
|
||||
fontSize: 'var(--fs-xs)',
|
||||
color: 'var(--fg-mute)',
|
||||
}}
|
||||
>
|
||||
<span>
|
||||
{b.label} · {b.value.toFixed(2)}
|
||||
</span>
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => inp.clearMidiBinding(i)}
|
||||
style={{
|
||||
background: 'transparent',
|
||||
border: 0,
|
||||
color: 'var(--danger)',
|
||||
cursor: 'pointer',
|
||||
fontSize: 'var(--fs-xs)',
|
||||
}}
|
||||
>
|
||||
✕
|
||||
</button>
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
)}
|
||||
{inp.midiInputs.length > 0 && (
|
||||
<p style={{ fontSize: 9, color: 'var(--fg-dim)', margin: 0 }}>
|
||||
Listening on: {inp.midiInputs.map((p) => p.name).join(', ')}
|
||||
</p>
|
||||
)}
|
||||
</>
|
||||
)}
|
||||
))}
|
||||
</div>
|
||||
<p style={{ fontSize: 9, color: 'var(--fg-dim)', margin: 0, lineHeight: 1.6 }}>
|
||||
Connect a controller and press any button to wake it. Sticks drive the input map;
|
||||
buttons fire the verdicts above.
|
||||
</p>
|
||||
</>
|
||||
)}
|
||||
|
||||
{/* ---- Channel layout ---- */}
|
||||
<SectionLabel>Channel layout</SectionLabel>
|
||||
{inp.channelLayout.length === 0 ? (
|
||||
{/* ---- MIDI ---- */}
|
||||
{inp.inputMode === 'midi' && depth !== 'peek' && (
|
||||
<>
|
||||
<SectionLabel>Device</SectionLabel>
|
||||
{inp.midiInputs.length === 0 ? (
|
||||
<p style={{ fontSize: 9, color: 'var(--fg-dim)', margin: 0 }}>
|
||||
No active axes. Enable a source above.
|
||||
No MIDI inputs detected. Connect a device — it appears here automatically.
|
||||
</p>
|
||||
) : (
|
||||
<div style={{ display: 'flex', flexWrap: 'wrap', gap: 4 }}>
|
||||
{inp.channelLayout.map((c, i) => (
|
||||
<Chip key={i}>
|
||||
{i}: {c.source}·{c.label}
|
||||
</Chip>
|
||||
<div style={{ display: 'flex', gap: 6, flexWrap: 'wrap' }}>
|
||||
<Button
|
||||
size="sm"
|
||||
variant={inp.midiDeviceId === null ? 'primary' : 'secondary'}
|
||||
onClick={() => inp.selectMidiDevice(null)}
|
||||
>
|
||||
All ports
|
||||
</Button>
|
||||
{inp.midiInputs.map((p) => (
|
||||
<Button
|
||||
key={p.id}
|
||||
size="sm"
|
||||
variant={inp.midiDeviceId === p.id ? 'primary' : 'secondary'}
|
||||
onClick={() => inp.selectMidiDevice(p.id)}
|
||||
>
|
||||
{p.name}
|
||||
</Button>
|
||||
))}
|
||||
</div>
|
||||
)}
|
||||
|
||||
<p style={{ fontSize: 9, color: 'var(--fg-dim)', margin: 0, lineHeight: 1.6 }}>
|
||||
The active sources concatenate into one input vector at the head of the spine.
|
||||
{reshaping
|
||||
? ` The browser WASM is a fixed ${inp.engineInputSize}-input head (MLP<2,…>), so the
|
||||
${inp.axisCount} axes are blended down to ${inp.engineInputSize} (even→X / odd→Y mean).`
|
||||
: ''}{' '}
|
||||
{/* TODO(workstream F, docs/redesign/inputs-spec.md — "multiple WASM modules +
|
||||
warm-start"): give every axis its own genuine input dimension by (re)loading a
|
||||
WASM module whose MLP arity matches axisCount and warm-starting from the prior
|
||||
net. Deferred — the reduction lives in InputLayer.compose(). */}
|
||||
True per-axis dimensions land with the multi-WASM reshape (inputs-spec).
|
||||
</p>
|
||||
<SectionLabel>MIDI Learn</SectionLabel>
|
||||
{inp.midiLearnArmed ? (
|
||||
<div
|
||||
style={{
|
||||
padding: '8px 10px',
|
||||
background: 'var(--bg-2)',
|
||||
border: '1px solid var(--accent-2)',
|
||||
borderRadius: 'var(--r-1)',
|
||||
display: 'flex',
|
||||
flexDirection: 'column',
|
||||
gap: 8,
|
||||
}}
|
||||
>
|
||||
<span style={{ fontSize: 'var(--fs-xs)', color: 'var(--accent-2)', lineHeight: 1.5 }}>
|
||||
Move all of the controls you want to use, then click Done. Each knob or fader you
|
||||
touch becomes an input.
|
||||
</span>
|
||||
<Button size="sm" variant="primary" onClick={() => inp.armMidiLearn(false)}>
|
||||
Done ({inp.midiBindings.length} learned)
|
||||
</Button>
|
||||
</div>
|
||||
) : (
|
||||
<div style={{ display: 'flex', gap: 6, alignItems: 'center', flexWrap: 'wrap' }}>
|
||||
<Button size="sm" variant="secondary" onClick={() => inp.armMidiLearn(true)}>
|
||||
{inp.midiBindings.length ? 'Learn more…' : 'Start MIDI Learn'}
|
||||
</Button>
|
||||
{inp.midiBindings.length > 0 && (
|
||||
<Button size="sm" variant="secondary" onClick={inp.clearMidiBindings}>
|
||||
Clear all
|
||||
</Button>
|
||||
)}
|
||||
</div>
|
||||
)}
|
||||
|
||||
{inp.midiBindings.length > 0 && (
|
||||
<>
|
||||
<SectionLabel>Learned controls</SectionLabel>
|
||||
<div style={{ display: 'flex', flexDirection: 'column', gap: 5 }}>
|
||||
{inp.midiBindings.map((b, i) => (
|
||||
<MidiInputMeter
|
||||
key={`${b.kind}-${b.number}-${b.channel}`}
|
||||
label={b.label}
|
||||
value={b.value}
|
||||
onClear={() => inp.clearMidiBinding(i)}
|
||||
/>
|
||||
))}
|
||||
</div>
|
||||
</>
|
||||
)}
|
||||
</>
|
||||
)}
|
||||
|
||||
{/* ---- Reshape note (only when >2 axes feed the fixed WASM head) ---- */}
|
||||
{reshaping && depth !== 'peek' && (
|
||||
<p style={{ fontSize: 9, color: 'var(--fg-dim)', margin: 0, lineHeight: 1.6 }}>
|
||||
{/* TODO(workstream F, docs/redesign/inputs-spec.md — "multiple WASM modules +
|
||||
warm-start"): give every axis its own genuine input dimension by (re)loading a
|
||||
WASM module whose MLP arity matches axisCount and warm-starting from the prior
|
||||
net. Deferred — the reduction lives in InputLayer.compose(). */}
|
||||
The browser WASM is a fixed {inp.engineInputSize}-input head (MLP<2,…>), so the{' '}
|
||||
{inp.axisCount} axes are blended down to {inp.engineInputSize} (even→X / odd→Y mean). True
|
||||
per-axis dimensions land with the multi-WASM reshape.
|
||||
</p>
|
||||
)}
|
||||
</>
|
||||
);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -164,9 +164,13 @@ export class EngineApi {
|
|||
this.spine.setInput(x, y);
|
||||
}
|
||||
|
||||
/** Set an arbitrary input vector (first two used as XY for the fixed 2→N MLP). */
|
||||
/**
|
||||
* Set the full N-dimensional input vector (one axis per active input source).
|
||||
* The first two axes run through the 2-D input pipeline; axes 2+ are raw.
|
||||
* Extra axes beyond the net's input arity are ignored; unused slots → 0.
|
||||
*/
|
||||
setInputs(arr: ReadonlyArray<number>): void {
|
||||
this.spine.setInput(arr[0] ?? 0.5, arr[1] ?? 0.5);
|
||||
this.spine.setInputs(arr);
|
||||
}
|
||||
|
||||
/** Live post-ML output vector (reused buffer — read, don't retain). */
|
||||
|
|
@ -193,7 +197,7 @@ export class EngineApi {
|
|||
* state without the user having to move the controller.
|
||||
*/
|
||||
process(): void {
|
||||
this.spine.setInput(this.spine.lastRawX, this.spine.lastRawY);
|
||||
this.spine.reprocess();
|
||||
}
|
||||
|
||||
// ---- Training ------------------------------------------------------
|
||||
|
|
|
|||
|
|
@ -94,6 +94,8 @@ export class Spine implements EngineSink {
|
|||
// Last raw input, so `EngineApi.process()` can re-tick after a weight change.
|
||||
lastRawX = 0.5;
|
||||
lastRawY = 0.5;
|
||||
// Full last raw input vector (N-D) for re-ticking without losing extra axes.
|
||||
private lastRawInputs: Float32Array = new Float32Array(2);
|
||||
private mlBuf: F32 = new Float32Array(126);
|
||||
private routedBuf: F32 | null = null;
|
||||
|
||||
|
|
@ -164,34 +166,61 @@ export class Spine implements EngineSink {
|
|||
// ---- The hot action ------------------------------------------------
|
||||
|
||||
/**
|
||||
* Drive a raw [0,1] XY input through processed → ml → routed eagerly and
|
||||
* synchronously, then fire the single backend.send at the tail. Off render.
|
||||
* Returns the routed buffer (live, reused — do not retain across calls).
|
||||
* Drive a raw [0,1] XY input through processed → ml → routed. Convenience for
|
||||
* the 2-D manifold / XY-pad path — delegates to {@link setInputs}.
|
||||
*/
|
||||
setInput(x: number, y: number): Float32Array | null {
|
||||
return this.setInputs([x, y]);
|
||||
}
|
||||
|
||||
/**
|
||||
* Drive an N-dimensional raw input vector (each ∈ [0,1]) through
|
||||
* processed → ml → routed eagerly and synchronously, then fire the single
|
||||
* backend.send at the tail. Off render. Returns the routed buffer (live,
|
||||
* reused — do not retain across calls).
|
||||
*
|
||||
* The mix-and-match input layer composes one axis PER active input source
|
||||
* (XY pad / gamepad sticks / learned MIDI CCs) into this vector. The first
|
||||
* two axes run through the 2-D input pipeline (deadzone→zoom→curve→smoothing→
|
||||
* momentum) so the pad keeps its feel and the ≤2-D path is unchanged; axes 2+
|
||||
* are written raw (sources self-condition). Unused slots up to the net's input
|
||||
* arity are held at 0 so a shrinking vector never leaves a stale dimension hot.
|
||||
*/
|
||||
setInputs(arr: ArrayLike<number>): Float32Array | null {
|
||||
const iml = this.iml;
|
||||
if (!iml) return null;
|
||||
|
||||
const now = (typeof performance !== 'undefined' ? performance.now() : Date.now());
|
||||
const dt = this.lastTickMs > 0 ? (now - this.lastTickMs) / 1000 : 1 / 60;
|
||||
this.lastTickMs = now;
|
||||
const dt = this.dt_();
|
||||
const inSize = this.state_.inputSize;
|
||||
|
||||
// 1. processed (pure input pipeline)
|
||||
// 1. primary pair through the pure input pipeline (pad feel / 2-D parity).
|
||||
const x = arr.length > 0 ? arr[0] : 0.5;
|
||||
const y = arr.length > 1 ? arr[1] : 0.5;
|
||||
this.rawInput[0] = x;
|
||||
this.rawInput[1] = y;
|
||||
this.lastRawX = x;
|
||||
this.lastRawY = y;
|
||||
const proc = processInput(this.rawInput, this.inputConfig, this.inputState, dt);
|
||||
this.inputState = proc.state;
|
||||
|
||||
// 2. ml (inference into the reused buffer; no alloc)
|
||||
iml.setInput(0, proc.x);
|
||||
iml.setInput(1, proc.y);
|
||||
|
||||
// 2. extra axes raw; unused slots cleared to 0. Remember the full raw vector
|
||||
// so process() can re-tick after a weight change without losing dims.
|
||||
if (this.lastRawInputs.length !== inSize) this.lastRawInputs = new Float32Array(inSize);
|
||||
this.lastRawInputs[0] = x;
|
||||
this.lastRawInputs[1] = y;
|
||||
for (let i = 2; i < inSize; i++) {
|
||||
const v = i < arr.length ? arr[i] : 0;
|
||||
iml.setInput(i, v);
|
||||
this.lastRawInputs[i] = v;
|
||||
}
|
||||
|
||||
// 3. ml (inference into the reused buffer; no alloc).
|
||||
iml.processInto(this.mlBuf);
|
||||
// Mirror to liveOutputs for imperative reads + bump.
|
||||
this.liveOutputs.set(this.mlBuf.subarray(0, this.liveOutputs.length));
|
||||
|
||||
// 3. routed (output pipeline → reused routedBuf)
|
||||
// 4. routed (output pipeline → reused routedBuf).
|
||||
const routedRes = processOutput(this.mlBuf, this.outputConfig, this.outputState, dt * 1000);
|
||||
this.outputState = routedRes.state;
|
||||
const routed = routedRes.processed;
|
||||
|
|
@ -201,13 +230,30 @@ export class Spine implements EngineSink {
|
|||
this.routedBuf = routed;
|
||||
}
|
||||
|
||||
// 4. single backend.send at the tail (off React render)
|
||||
// 5. single backend.send at the tail (off React render).
|
||||
if (this.backendSend && this.routedBuf) this.backendSend(this.routedBuf);
|
||||
|
||||
this.bump_();
|
||||
return this.routedBuf;
|
||||
}
|
||||
|
||||
/**
|
||||
* Re-run the LAST full raw input vector through the spine (after a weight
|
||||
* change — train / randomise / feedback) so outputs + audio reflect the new
|
||||
* net without the user touching a control. Preserves all N dimensions.
|
||||
*/
|
||||
reprocess(): Float32Array | null {
|
||||
return this.setInputs(this.lastRawInputs);
|
||||
}
|
||||
|
||||
/** Monotonic per-tick dt in seconds (≈1/60 on the first tick). */
|
||||
private dt_(): number {
|
||||
const now = (typeof performance !== 'undefined' ? performance.now() : Date.now());
|
||||
const dt = this.lastTickMs > 0 ? (now - this.lastTickMs) / 1000 : 1 / 60;
|
||||
this.lastTickMs = now;
|
||||
return dt;
|
||||
}
|
||||
|
||||
// ---- Imperative reads (canvas consumers bypass React) --------------
|
||||
|
||||
/** Live post-ML output vector. Reused — read, don't retain. */
|
||||
|
|
|
|||
|
|
@ -25,6 +25,31 @@ export type StickMode = 'single' | 'double';
|
|||
|
||||
const DEADZONE = 0.08;
|
||||
|
||||
/**
|
||||
* Standard-mapping button index → human label. The Inputs dock binds these to
|
||||
* verdict ops (see useInputLayer / ConsoleApp): LB/RB = down/up feedback, the
|
||||
* face buttons = randomise / nudge / undo, and a hold-and-move button drops a
|
||||
* repositioned example. The labels here keep the dock legend honest.
|
||||
*/
|
||||
const BUTTON_LABELS: Record<number, string> = {
|
||||
0: 'A',
|
||||
1: 'B',
|
||||
2: 'X',
|
||||
3: 'Y',
|
||||
4: 'LB',
|
||||
5: 'RB',
|
||||
6: 'LT',
|
||||
7: 'RT',
|
||||
8: 'Back',
|
||||
9: 'Start',
|
||||
10: 'L3',
|
||||
11: 'R3',
|
||||
12: 'D↑',
|
||||
13: 'D↓',
|
||||
14: 'D←',
|
||||
15: 'D→',
|
||||
};
|
||||
|
||||
export class GamepadSource extends BaseSource {
|
||||
readonly kind: InputSourceKind = 'gamepad';
|
||||
readonly label = 'Gamepad';
|
||||
|
|
@ -117,8 +142,17 @@ export class GamepadSource extends BaseSource {
|
|||
this.emitAction({
|
||||
source: this.kind,
|
||||
id: `button:${i}`,
|
||||
label: `Button ${i}`,
|
||||
label: BUTTON_LABELS[i] ?? `Button ${i}`,
|
||||
value: pad.buttons[i].value || 1,
|
||||
phase: 'press',
|
||||
});
|
||||
} else if (!pressed && this.buttonsDown[i]) {
|
||||
this.emitAction({
|
||||
source: this.kind,
|
||||
id: `button:${i}`,
|
||||
label: BUTTON_LABELS[i] ?? `Button ${i}`,
|
||||
value: 0,
|
||||
phase: 'release',
|
||||
});
|
||||
}
|
||||
this.buttonsDown[i] = pressed;
|
||||
|
|
|
|||
|
|
@ -1,14 +1,15 @@
|
|||
/**
|
||||
* Modular INPUT layer (workstream F) — public surface.
|
||||
*
|
||||
* The user picks the input SOURCE(s) feeding the ML head (XY pad / MIDI /
|
||||
* gamepad, or a combination); the InputLayer composes their axes into one
|
||||
* N-dim vector at the head of the reactive spine. See input-layer.ts for the
|
||||
* The user picks ONE exclusive input MODE feeding the ML head (Internal XY pad /
|
||||
* Game Controller / MIDI); the InputLayer composes the active source's axes into
|
||||
* one N-dim vector at the head of the reactive spine. See input-layer.ts for the
|
||||
* arity-reduction + the documented multi-WASM reshape TODO.
|
||||
*/
|
||||
export type {
|
||||
InputSource,
|
||||
InputSourceKind,
|
||||
InputMode,
|
||||
InputSourceState,
|
||||
InputSourceStatus,
|
||||
InputAction,
|
||||
|
|
|
|||
|
|
@ -13,24 +13,18 @@
|
|||
* its value), but routing everything through one compose path keeps sources
|
||||
* composable and the channel layout coherent.
|
||||
*
|
||||
* ── Arity mismatch (the WASM reshape TODO) ──────────────────────────────────
|
||||
* The browser WASM is fixed at MLP<2, …, 126> — a TWO-input head. When the
|
||||
* composed vector has > 2 axes (double-stick gamepad = 4, MIDI learn-map = many)
|
||||
* we must reduce to 2 to feed today's engine. We do NOT fake a wider net.
|
||||
* ── Dedicated dimensions (no blending) ──────────────────────────────────────
|
||||
* The WASM net is over-provisioned to a 32-input head (= MAX_AXES; see
|
||||
* nisps/wasm/bindings.cpp). Each active axis drives its OWN engine input slot
|
||||
* 1:1 — a double-stick gamepad is 4 genuine dims, a learned MIDI surface is N
|
||||
* genuine dims. `compose()` simply forwards the active axes; the engine
|
||||
* zero-pads the remaining slots and a zero input is inert (0 × weight = 0), so
|
||||
* unused dimensions never perturb the net. We do NOT mean-blend (the previous
|
||||
* behaviour) — that diluted every source and biased the net toward idle
|
||||
* sources' resting values.
|
||||
*
|
||||
* chosen reduction (this pass): pairwise BLEND.
|
||||
* inX = mean(axis[0], axis[2], axis[4], …) // even axes
|
||||
* inY = mean(axis[1], axis[3], axis[5], …) // odd axes
|
||||
* so a single stick passes straight through (axis0→X, axis1→Y), a double
|
||||
* stick averages L/R into one XY, and MIDI axes fold into X/Y by parity.
|
||||
*
|
||||
* TODO(workstream F, docs/redesign/inputs-spec.md — "multiple WASM modules +
|
||||
* warm-start"): the real fix is to (re)load a WASM module whose MLP input arity
|
||||
* matches the composed axis count and warm-start its weights from the prior net,
|
||||
* so every axis gets its own genuine input dimension instead of being blended.
|
||||
* That is a larger build (multiple .wasm artefacts or a runtime-variadic head)
|
||||
* and is deliberately deferred — this layer is wired so that swapping the
|
||||
* reduction for a true reshape is a localised change in `compose()`.
|
||||
* Changing the ACTIVE axis count is a reshape: the front-end resets the net
|
||||
* (recreate-from-scratch, behind a confirm modal) since slot meanings change.
|
||||
*/
|
||||
import type { InputAction, InputSource } from './types';
|
||||
|
||||
|
|
@ -50,6 +44,7 @@ export class InputLayer {
|
|||
private rafId: number | null = null;
|
||||
private actionListeners = new Set<(a: InputAction) => void>();
|
||||
private layoutListeners = new Set<() => void>();
|
||||
private reducedListeners = new Set<(x: number, y: number) => void>();
|
||||
private unsubActions = new Map<InputSource, () => void>();
|
||||
|
||||
attach(engine: InputEngineSink): void {
|
||||
|
|
@ -144,50 +139,32 @@ export class InputLayer {
|
|||
|
||||
// 4. one engine write.
|
||||
engine.setInputs(reduced);
|
||||
|
||||
// 5. report the reduced 2D position so the on-screen manifold can track a
|
||||
// gamepad/MIDI-driven input (the XY pad pushes its own position).
|
||||
if (this.reducedListeners.size) {
|
||||
const x = reduced[0] ?? 0.5;
|
||||
const y = reduced[1] ?? reduced[0] ?? 0.5;
|
||||
for (const cb of this.reducedListeners) cb(x, y);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Reduce the composed N-axis vector to the engine's input arity.
|
||||
* Map the composed N active axes to the engine's input vector — DEDICATED
|
||||
* DIMENSIONS, no blending. Each active axis i drives engine input slot i 1:1;
|
||||
* the engine zero-pads the slots beyond `count` and a zero input is inert
|
||||
* (0 × weight = 0), so unused dimensions never perturb the net.
|
||||
*
|
||||
* For the fixed 2-input WASM, fold by parity (even→X, odd→Y) via mean. If a
|
||||
* future multi-module engine reports inputSize >= n, this passes axes through
|
||||
* 1:1 (truncated/padded) — the seam where the real reshape lands.
|
||||
* The net's input arity is over-provisioned (32, = MAX_AXES), so `inputSize`
|
||||
* is effectively always ≥ n; the `min` only guards a transient where more
|
||||
* axes are active than the net can take. We deliberately do NOT mean-blend
|
||||
* (the old behaviour) — that diluted every source and biased the net toward
|
||||
* idle sources' resting values.
|
||||
*/
|
||||
private compose(n: number, inputSize: number): number[] {
|
||||
if (inputSize >= n) {
|
||||
// True passthrough path (future multi-module head). Pad with 0.5.
|
||||
const out = new Array<number>(inputSize);
|
||||
for (let i = 0; i < inputSize; i++) out[i] = i < n ? this.vector[i] : 0.5;
|
||||
return out;
|
||||
}
|
||||
if (inputSize === 2) {
|
||||
let sx = 0;
|
||||
let sy = 0;
|
||||
let cx = 0;
|
||||
let cy = 0;
|
||||
for (let i = 0; i < n; i++) {
|
||||
if ((i & 1) === 0) {
|
||||
sx += this.vector[i];
|
||||
cx++;
|
||||
} else {
|
||||
sy += this.vector[i];
|
||||
cy++;
|
||||
}
|
||||
}
|
||||
return [cx ? sx / cx : 0.5, cy ? sy / cy : 0.5];
|
||||
}
|
||||
// Generic fallback for any other fixed arity: chunked mean.
|
||||
const out = new Array<number>(inputSize).fill(0.5);
|
||||
const per = Math.ceil(n / inputSize);
|
||||
for (let k = 0; k < inputSize; k++) {
|
||||
let s = 0;
|
||||
let c = 0;
|
||||
for (let i = k * per; i < Math.min((k + 1) * per, n); i++) {
|
||||
s += this.vector[i];
|
||||
c++;
|
||||
}
|
||||
if (c) out[k] = s / c;
|
||||
}
|
||||
const count = Math.min(n, inputSize);
|
||||
const out = new Array<number>(count);
|
||||
for (let i = 0; i < count; i++) out[i] = this.vector[i];
|
||||
return out;
|
||||
}
|
||||
|
||||
|
|
@ -207,6 +184,14 @@ export class InputLayer {
|
|||
};
|
||||
}
|
||||
|
||||
/** Subscribe to the reduced 2D input each frame (composed → engine arity). */
|
||||
onReducedInput(cb: (x: number, y: number) => void): () => void {
|
||||
this.reducedListeners.add(cb);
|
||||
return () => {
|
||||
this.reducedListeners.delete(cb);
|
||||
};
|
||||
}
|
||||
|
||||
private fanAction(a: InputAction): void {
|
||||
for (const cb of this.actionListeners) cb(a);
|
||||
}
|
||||
|
|
@ -221,5 +206,6 @@ export class InputLayer {
|
|||
this.unsubActions.clear();
|
||||
this.actionListeners.clear();
|
||||
this.layoutListeners.clear();
|
||||
this.reducedListeners.clear();
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -10,10 +10,17 @@
|
|||
* note-off. Note-on ALSO surfaces a discrete action (so a pad can fire
|
||||
* commit/perturb without the keyboard).
|
||||
*
|
||||
* **Learn-map.** When `armLearn()` is active, the NEXT distinct CC or note seen
|
||||
* is bound to a new axis (appended). This is the standard "MIDI learn" gesture:
|
||||
* arm → wiggle the knob/pad → it captures. Axes can be cleared individually.
|
||||
* The bindings are exposed for the dock channel-layout view.
|
||||
* **Batch learn ("MIDI Learn" mode).** When `armLearn(true)` is active, EVERY
|
||||
* distinct CC that moves is captured as a new axis (appended, deduped). The
|
||||
* gesture the dock presents: arm → wiggle ALL the knobs/faders you want →
|
||||
* click "Done" (`armLearn(false)`). This differs from the one-shot learn other
|
||||
* apps use — the user sweeps the whole control surface in one pass. Notes are
|
||||
* NOT auto-bound as axes (they stay as discrete actions); the "controls" the
|
||||
* user sweeps are continuous CCs. Axes can be cleared individually or all.
|
||||
*
|
||||
* **Device selection.** By default every connected input port is listened to.
|
||||
* `selectDevice(id)` narrows to a single port (the dock device picker); `null`
|
||||
* restores listen-all.
|
||||
*
|
||||
* Pull-based: messages latch the latest per-binding value into `values`;
|
||||
* `sample()` copies them out. Hot path performs no IO/allocation.
|
||||
|
|
@ -51,6 +58,8 @@ export class WebMidiInputSource extends BaseSource {
|
|||
private inputs: MIDIInput[] = [];
|
||||
private bindings: MidiBinding[] = [];
|
||||
private learnArmed = false;
|
||||
/** Restrict listening to this input port id; null = every connected port. */
|
||||
private selectedDeviceId: string | null = null;
|
||||
private bindingsListeners = new Set<(b: MidiBinding[]) => void>();
|
||||
|
||||
isAvailable(): boolean {
|
||||
|
|
@ -73,12 +82,16 @@ export class WebMidiInputSource extends BaseSource {
|
|||
|
||||
// ---- Learn-map API (consumed by the dock) -------------------------------
|
||||
|
||||
/** Arm/disarm MIDI-learn: the next distinct CC/note is captured as an axis. */
|
||||
/**
|
||||
* Enter/leave batch MIDI-Learn. While armed, every distinct CC that moves is
|
||||
* appended as an axis (the user sweeps their whole control surface, then
|
||||
* clicks Done). Disarming keeps whatever was captured.
|
||||
*/
|
||||
armLearn(armed: boolean): void {
|
||||
this.learnArmed = armed;
|
||||
this.setStatus(
|
||||
armed
|
||||
? { state: 'ready', message: 'Learn armed — move a knob or hit a pad' }
|
||||
? { state: 'ready', message: 'MIDI Learn — move every control you want, then click Done' }
|
||||
: this.readyStatus(),
|
||||
);
|
||||
}
|
||||
|
|
@ -87,6 +100,16 @@ export class WebMidiInputSource extends BaseSource {
|
|||
return this.learnArmed;
|
||||
}
|
||||
|
||||
/** Narrow listening to one input port (dock device picker). null = all ports. */
|
||||
selectDevice(id: string | null): void {
|
||||
this.selectedDeviceId = id;
|
||||
this.rewire();
|
||||
}
|
||||
|
||||
getSelectedDeviceId(): string | null {
|
||||
return this.selectedDeviceId;
|
||||
}
|
||||
|
||||
getBindings(): ReadonlyArray<MidiBinding> {
|
||||
return this.bindings;
|
||||
}
|
||||
|
|
@ -149,11 +172,15 @@ export class WebMidiInputSource extends BaseSource {
|
|||
if (!this.access) return;
|
||||
for (const inp of this.inputs) inp.onmidimessage = null;
|
||||
this.inputs = [];
|
||||
this.access.inputs.forEach((inp) => {
|
||||
this.access.inputs.forEach((inp, id) => {
|
||||
// Honour the device picker: when a port is selected, listen to it alone.
|
||||
if (this.selectedDeviceId !== null && id !== this.selectedDeviceId) return;
|
||||
inp.onmidimessage = (e) => this.onMessage(e);
|
||||
this.inputs.push(inp);
|
||||
});
|
||||
if (this.statusState.state !== 'connecting') this.setStatus(this.readyStatus());
|
||||
if (this.statusState.state !== 'connecting' && !this.learnArmed) {
|
||||
this.setStatus(this.readyStatus());
|
||||
}
|
||||
}
|
||||
|
||||
private readyStatus(): { state: 'ready'; message: string } {
|
||||
|
|
@ -173,41 +200,45 @@ export class WebMidiInputSource extends BaseSource {
|
|||
const d2 = data.length > 2 ? data[2] : 0;
|
||||
|
||||
if (status === STATUS_BYTE_CC) {
|
||||
this.handleBindable('cc', d1, channel, d2 / 127);
|
||||
this.handleCc(d1, channel, d2 / 127);
|
||||
} else if (status === STATUS_BYTE_NOTE_ON && d2 > 0) {
|
||||
this.handleBindable('note', d1, channel, 1);
|
||||
this.emitAction({ source: this.kind, id: `note:${d1}`, label: `Note ${d1}`, value: d2 / 127 });
|
||||
// Notes drive a held-gate on any already-learned note axis + a discrete
|
||||
// action (so a pad can fire commit/perturb). Batch learn binds CCs only.
|
||||
this.updateNote(d1, channel, 1);
|
||||
this.emitAction({ source: this.kind, id: `note:${d1}`, label: `Note ${d1}`, value: d2 / 127, phase: 'press' });
|
||||
} else if (status === STATUS_BYTE_NOTE_OFF || (status === STATUS_BYTE_NOTE_ON && d2 === 0)) {
|
||||
this.handleBindable('note', d1, channel, 0, /*onlyUpdate*/ true);
|
||||
this.updateNote(d1, channel, 0);
|
||||
this.emitAction({ source: this.kind, id: `note:${d1}`, label: `Note ${d1}`, value: 0, phase: 'release' });
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Route an incoming bindable message: update a matching binding's value, or —
|
||||
* if learn is armed — create a new axis binding for it.
|
||||
* Route an incoming CC: update a matching binding's value, or — if batch
|
||||
* learn is armed — capture it as a NEW axis (deduped). Learn stays armed so
|
||||
* the user can sweep their whole control surface in one pass.
|
||||
*/
|
||||
private handleBindable(
|
||||
kind: MidiBindingKind,
|
||||
number: number,
|
||||
channel: number,
|
||||
value: number,
|
||||
onlyUpdate = false,
|
||||
): void {
|
||||
private handleCc(number: number, channel: number, value: number): void {
|
||||
const existing = this.bindings.find(
|
||||
(b) => b.kind === kind && b.number === number && b.channel === channel,
|
||||
(b) => b.kind === 'cc' && b.number === number && b.channel === channel,
|
||||
);
|
||||
if (existing) {
|
||||
existing.value = value;
|
||||
this.notifyBindings();
|
||||
return;
|
||||
}
|
||||
if (onlyUpdate) return; // note-off for an unbound note: ignore
|
||||
if (this.learnArmed) {
|
||||
const label =
|
||||
kind === 'cc' ? `CC${number} ch${channel}` : `Note ${number} ch${channel}`;
|
||||
this.bindings.push({ kind, number, channel, value, label });
|
||||
this.learnArmed = false; // learn one binding per arm
|
||||
this.setStatus(this.readyStatus());
|
||||
this.bindings.push({ kind: 'cc', number, channel, value, label: `CC${number} ch${channel}` });
|
||||
this.notifyBindings();
|
||||
}
|
||||
}
|
||||
|
||||
/** Update a learned note axis's gate value (note bindings are not auto-learned). */
|
||||
private updateNote(number: number, channel: number, value: number): void {
|
||||
const existing = this.bindings.find(
|
||||
(b) => b.kind === 'note' && b.number === number && b.channel === channel,
|
||||
);
|
||||
if (existing) {
|
||||
existing.value = value;
|
||||
this.notifyBindings();
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -44,6 +44,17 @@ export interface InputSourceStatus {
|
|||
/** Stable identity of a source kind. */
|
||||
export type InputSourceKind = 'xy-pad' | 'midi' | 'gamepad';
|
||||
|
||||
/**
|
||||
* The exclusive INPUT MODE the user picks in the Inputs dock. Unlike the
|
||||
* lower-level {@link InputSourceKind} (which the InputLayer can compose), the
|
||||
* dock surfaces exactly one mode at a time:
|
||||
*
|
||||
* - `internal` → the on-screen XY pad / manifold (default; today's behaviour).
|
||||
* - `gamepad` → a physical game controller (sticks → axes, buttons → verdicts).
|
||||
* - `midi` → a connected MIDI device (learned CCs → axes).
|
||||
*/
|
||||
export type InputMode = 'internal' | 'gamepad' | 'midi';
|
||||
|
||||
/**
|
||||
* A momentary discrete action surfaced by a source (e.g. a MIDI note-on or a
|
||||
* gamepad face-button press). Fanned out to InputLayer action listeners so the
|
||||
|
|
@ -57,6 +68,13 @@ export interface InputAction {
|
|||
label: string;
|
||||
/** 0..1 velocity / analogue value where meaningful (else 1 for a press). */
|
||||
value: number;
|
||||
/**
|
||||
* Edge phase. `press` (the default) fires on the leading edge; `release` on
|
||||
* the trailing edge. Hold-and-move bindings (e.g. "hold a button, move the
|
||||
* stick, release to drop an example") need both edges — most consumers only
|
||||
* care about `press`.
|
||||
*/
|
||||
phase?: 'press' | 'release';
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
|
|||
|
|
@ -2,20 +2,20 @@
|
|||
* useInputLayer — the thin React binding over the framework-neutral
|
||||
* {@link InputLayer} + source adapters.
|
||||
*
|
||||
* Owns:
|
||||
* - ONE InputLayer + one instance of each source (XY pad / MIDI / gamepad),
|
||||
* created per engine and attached to it.
|
||||
* - Which sources are ENABLED (the dock toggles these); enabling starts a
|
||||
* source (async for MIDI) and adds it to the layer's composed set.
|
||||
* - Per-source config (gamepad stick mode; MIDI learn arm + bindings).
|
||||
* - The composed channel layout + per-source status, surfaced for the drawer.
|
||||
* The dock surfaces ONE exclusive input MODE at a time (inputs-spec):
|
||||
* - `internal` → the on-screen XY pad / manifold (default; today's behaviour).
|
||||
* - `gamepad` → a physical game controller (sticks → axes, buttons → verdicts).
|
||||
* - `midi` → a connected MIDI device (CCs learned onto axes).
|
||||
*
|
||||
* The XY pad source is the one consumers push into directly: `pushPad(x,y)` is
|
||||
* called from ConsoleApp.onMove so the existing pad keeps working unchanged
|
||||
* while still composing with the other sources.
|
||||
* Switching mode stops the previous source and starts the chosen one, then sets
|
||||
* the layer's composed source set to exactly that source. The XY pad is the one
|
||||
* consumers push into directly (`pushPad` from ConsoleApp.onMove) so the manifold
|
||||
* keeps working unchanged in `internal` mode.
|
||||
*
|
||||
* Discrete actions (MIDI notes / gamepad buttons) are fanned out via
|
||||
* `onAction` so the console can later bind them to verdicts (commit/perturb).
|
||||
* Per-mode config (gamepad stick mode + button verdict legend; MIDI device pick,
|
||||
* batch learn arm, learned bindings) and per-source status are surfaced for the
|
||||
* drawer. Discrete actions (gamepad buttons / MIDI notes) are fanned out via
|
||||
* `onAction` so the console can bind them to verdicts (commit/perturb/…).
|
||||
*/
|
||||
import { useCallback, useEffect, useMemo, useRef, useState } from 'react';
|
||||
import type { EngineApi } from '../engine';
|
||||
|
|
@ -23,7 +23,13 @@ import { InputLayer } from './input-layer';
|
|||
import { XYPadSource } from './xy-pad-source';
|
||||
import { WebMidiInputSource, type MidiBinding } from './midi-input-source';
|
||||
import { GamepadSource, type StickMode } from './gamepad-source';
|
||||
import type { InputAction, InputSource, InputSourceKind, InputSourceStatus } from './types';
|
||||
import type {
|
||||
InputAction,
|
||||
InputMode,
|
||||
InputSource,
|
||||
InputSourceKind,
|
||||
InputSourceStatus,
|
||||
} from './types';
|
||||
|
||||
export interface SourceView {
|
||||
kind: InputSourceKind;
|
||||
|
|
@ -33,13 +39,25 @@ export interface SourceView {
|
|||
axisCount: number;
|
||||
}
|
||||
|
||||
/** Which source backs each exclusive input mode. */
|
||||
const MODE_SOURCE: Record<InputMode, InputSourceKind> = {
|
||||
internal: 'xy-pad',
|
||||
gamepad: 'gamepad',
|
||||
midi: 'midi',
|
||||
};
|
||||
|
||||
export interface UseInputLayer {
|
||||
/** Push the on-screen XY pad position (∈ [0,1]) — call from onMove. */
|
||||
pushPad: (x: number, y: number) => void;
|
||||
/** Per-source enable + status + axis count for the dock. */
|
||||
|
||||
// ---- exclusive mode ----
|
||||
/** The active input mode (Internal / Game Controller / MIDI). */
|
||||
inputMode: InputMode;
|
||||
/** Switch the exclusive input mode. */
|
||||
setInputMode: (m: InputMode) => void;
|
||||
|
||||
/** Per-source status + axis count for the dock (the active mode's source is `enabled`). */
|
||||
sources: SourceView[];
|
||||
/** Toggle a source on/off. */
|
||||
setEnabled: (kind: InputSourceKind, enabled: boolean) => void;
|
||||
/** Composed channel layout (per-axis source+label). */
|
||||
channelLayout: { source: string; label: string }[];
|
||||
/** Total composed axis count. */
|
||||
|
|
@ -51,16 +69,23 @@ export interface UseInputLayer {
|
|||
gamepadStickMode: StickMode;
|
||||
setGamepadStickMode: (m: StickMode) => void;
|
||||
|
||||
// ---- midi learn-map ----
|
||||
// ---- midi device + learn-map ----
|
||||
/** Available MIDI input ports. */
|
||||
midiInputs: { id: string; name: string }[];
|
||||
/** The selected MIDI input port (null = listen to all ports). */
|
||||
midiDeviceId: string | null;
|
||||
selectMidiDevice: (id: string | null) => void;
|
||||
/** True while batch MIDI-Learn is armed (sweep controls, then Done). */
|
||||
midiLearnArmed: boolean;
|
||||
armMidiLearn: (armed: boolean) => void;
|
||||
midiBindings: MidiBinding[];
|
||||
clearMidiBinding: (i: number) => void;
|
||||
clearMidiBindings: () => void;
|
||||
midiInputs: { id: string; name: string }[];
|
||||
|
||||
/** Subscribe to discrete actions (notes/buttons). */
|
||||
onAction: (cb: (a: InputAction) => void) => () => void;
|
||||
/** Subscribe to the reduced 2D input each frame (for the on-screen manifold). */
|
||||
onReducedInput: (cb: (x: number, y: number) => void) => () => void;
|
||||
}
|
||||
|
||||
export function useInputLayer(engine: EngineApi | null): UseInputLayer {
|
||||
|
|
@ -81,12 +106,7 @@ export function useInputLayer(engine: EngineApi | null): UseInputLayer {
|
|||
const midi = midiRef.current!;
|
||||
const gamepad = gamepadRef.current!;
|
||||
|
||||
// Enabled set — pad on by default (parity with today's behaviour).
|
||||
const [enabled, setEnabledSet] = useState<Record<InputSourceKind, boolean>>({
|
||||
'xy-pad': true,
|
||||
midi: false,
|
||||
gamepad: false,
|
||||
});
|
||||
const [inputMode, setInputModeState] = useState<InputMode>('internal');
|
||||
const [statuses, setStatuses] = useState<Record<InputSourceKind, InputSourceStatus>>({
|
||||
'xy-pad': pad.status(),
|
||||
midi: midi.status(),
|
||||
|
|
@ -97,8 +117,9 @@ export function useInputLayer(engine: EngineApi | null): UseInputLayer {
|
|||
const [midiLearnArmed, setMidiLearnArmed] = useState(false);
|
||||
const [midiBindings, setMidiBindings] = useState<MidiBinding[]>([]);
|
||||
const [midiInputs, setMidiInputs] = useState<{ id: string; name: string }[]>([]);
|
||||
const [midiDeviceId, setMidiDeviceId] = useState<string | null>(null);
|
||||
|
||||
// Attach to engine; start the pad immediately. Wire status/binding listeners.
|
||||
// Attach to engine; start in `internal` mode (the XY pad). Wire listeners.
|
||||
useEffect(() => {
|
||||
if (!engine) return;
|
||||
layer.attach(engine);
|
||||
|
|
@ -108,13 +129,13 @@ export function useInputLayer(engine: EngineApi | null): UseInputLayer {
|
|||
|
||||
const unsubs: (() => void)[] = [];
|
||||
const wireStatus = (s: InputSource) =>
|
||||
unsubs.push(
|
||||
s.onStatusChange((st) => setStatuses((m) => ({ ...m, [s.kind]: st }))),
|
||||
);
|
||||
unsubs.push(s.onStatusChange((st) => setStatuses((m) => ({ ...m, [s.kind]: st }))));
|
||||
wireStatus(pad);
|
||||
wireStatus(midi);
|
||||
wireStatus(gamepad);
|
||||
unsubs.push(layer.onLayoutChange(() => setLayoutTick((t) => t + 1)));
|
||||
// Refresh the device-picker list when ports come and go (hot-plug).
|
||||
unsubs.push(midi.onStatusChange(() => setMidiInputs(midi.listInputs())));
|
||||
unsubs.push(
|
||||
midi.onBindingsChange((b) => {
|
||||
setMidiBindings(b);
|
||||
|
|
@ -131,34 +152,37 @@ export function useInputLayer(engine: EngineApi | null): UseInputLayer {
|
|||
// eslint-disable-next-line react-hooks/exhaustive-deps
|
||||
}, [engine]);
|
||||
|
||||
// Recompose the active source set whenever the enabled set changes.
|
||||
// Recompose the active source set whenever the mode changes.
|
||||
useEffect(() => {
|
||||
const active: InputSource[] = [];
|
||||
if (enabled['xy-pad']) active.push(pad);
|
||||
if (enabled.midi) active.push(midi);
|
||||
if (enabled.gamepad) active.push(gamepad);
|
||||
layer.setSources(active);
|
||||
const kind = MODE_SOURCE[inputMode];
|
||||
const src = kind === 'xy-pad' ? pad : kind === 'gamepad' ? gamepad : midi;
|
||||
layer.setSources([src]);
|
||||
// eslint-disable-next-line react-hooks/exhaustive-deps
|
||||
}, [enabled]);
|
||||
}, [inputMode]);
|
||||
|
||||
const setEnabled = useCallback(
|
||||
(kind: InputSourceKind, on: boolean) => {
|
||||
setEnabledSet((m) => ({ ...m, [kind]: on }));
|
||||
if (kind === 'midi') {
|
||||
if (on) {
|
||||
void midi.start().then(() => setMidiInputs(midi.listInputs()));
|
||||
} else {
|
||||
void midi.stop();
|
||||
}
|
||||
} else if (kind === 'gamepad') {
|
||||
if (on) gamepad.start();
|
||||
else gamepad.stop();
|
||||
} else if (kind === 'xy-pad') {
|
||||
if (on) pad.start();
|
||||
const setInputMode = useCallback(
|
||||
(mode: InputMode) => {
|
||||
setInputModeState((prev) => {
|
||||
if (prev === mode) return prev;
|
||||
// Stop the outgoing source, start the incoming one.
|
||||
if (prev === 'gamepad') gamepad.stop();
|
||||
else if (prev === 'midi') void midi.stop();
|
||||
else pad.stop();
|
||||
}
|
||||
|
||||
if (mode === 'gamepad') {
|
||||
gamepad.start();
|
||||
} else if (mode === 'midi') {
|
||||
void midi.start().then(() => {
|
||||
setMidiInputs(midi.listInputs());
|
||||
setMidiDeviceId(midi.getSelectedDeviceId());
|
||||
});
|
||||
} else {
|
||||
pad.start();
|
||||
}
|
||||
return mode;
|
||||
});
|
||||
},
|
||||
[midi, gamepad, pad],
|
||||
[pad, midi, gamepad],
|
||||
);
|
||||
|
||||
const setGamepadStickMode = useCallback(
|
||||
|
|
@ -170,6 +194,15 @@ export function useInputLayer(engine: EngineApi | null): UseInputLayer {
|
|||
[gamepad],
|
||||
);
|
||||
|
||||
const selectMidiDevice = useCallback(
|
||||
(id: string | null) => {
|
||||
midi.selectDevice(id);
|
||||
setMidiDeviceId(id);
|
||||
setMidiInputs(midi.listInputs());
|
||||
},
|
||||
[midi],
|
||||
);
|
||||
|
||||
const armMidiLearn = useCallback(
|
||||
(armed: boolean) => {
|
||||
midi.armLearn(armed);
|
||||
|
|
@ -194,42 +227,50 @@ export function useInputLayer(engine: EngineApi | null): UseInputLayer {
|
|||
|
||||
const pushPad = useCallback((x: number, y: number) => pad.pushAxes(x, y), [pad]);
|
||||
const onAction = useCallback((cb: (a: InputAction) => void) => layer.onAction(cb), [layer]);
|
||||
const onReducedInput = useCallback(
|
||||
(cb: (x: number, y: number) => void) => layer.onReducedInput(cb),
|
||||
[layer],
|
||||
);
|
||||
|
||||
const sources: SourceView[] = useMemo(
|
||||
() =>
|
||||
([pad, midi, gamepad] as InputSource[]).map((s) => ({
|
||||
kind: s.kind,
|
||||
label: s.label,
|
||||
enabled: enabled[s.kind],
|
||||
enabled: MODE_SOURCE[inputMode] === s.kind,
|
||||
status: statuses[s.kind],
|
||||
axisCount: enabled[s.kind] ? s.axisCount() : 0,
|
||||
axisCount: MODE_SOURCE[inputMode] === s.kind ? s.axisCount() : 0,
|
||||
})),
|
||||
// layoutTick forces recompute when axis counts shift (learn-map / stick mode).
|
||||
// eslint-disable-next-line react-hooks/exhaustive-deps
|
||||
[enabled, statuses, layoutTick, pad, midi, gamepad],
|
||||
[inputMode, statuses, layoutTick, pad, midi, gamepad],
|
||||
);
|
||||
|
||||
const channelLayout = useMemo(
|
||||
() => layer.channelLayout(),
|
||||
// eslint-disable-next-line react-hooks/exhaustive-deps
|
||||
[layoutTick, enabled],
|
||||
[layoutTick, inputMode],
|
||||
);
|
||||
|
||||
return {
|
||||
pushPad,
|
||||
inputMode,
|
||||
setInputMode,
|
||||
sources,
|
||||
setEnabled,
|
||||
channelLayout,
|
||||
axisCount: channelLayout.length,
|
||||
engineInputSize: engine?.architecture.inputSize ?? 2,
|
||||
gamepadStickMode,
|
||||
setGamepadStickMode,
|
||||
midiInputs,
|
||||
midiDeviceId,
|
||||
selectMidiDevice,
|
||||
midiLearnArmed,
|
||||
armMidiLearn,
|
||||
midiBindings,
|
||||
clearMidiBinding,
|
||||
clearMidiBindings,
|
||||
midiInputs,
|
||||
onAction,
|
||||
onReducedInput,
|
||||
};
|
||||
}
|
||||
|
|
|
|||
|
|
@ -165,8 +165,14 @@ if(NOT EMSCRIPTEN)
|
|||
if(CMAKE_CXX_COMPILER_ID MATCHES "GNU|Clang|AppleClang")
|
||||
target_compile_options(nisps_parity_check PRIVATE
|
||||
-Wall -Wextra -Werror -Wpedantic
|
||||
# Disable FP multiply-add contraction so native matches the WASM
|
||||
# build, which has no FMA instruction. Without this, native clang/gcc
|
||||
# fuses MACs in the training backprop and the (chaotic) loop amplifies
|
||||
# the rounding difference past the 1e-5 parity tolerance — pronounced
|
||||
# since the input layer widened to 32 for mix-and-match inputs.
|
||||
-ffp-contract=off
|
||||
)
|
||||
elseif(MSVC)
|
||||
target_compile_options(nisps_parity_check PRIVATE /W4 /WX)
|
||||
target_compile_options(nisps_parity_check PRIVATE /W4 /WX /fp:precise)
|
||||
endif()
|
||||
endif()
|
||||
|
|
|
|||
|
|
@ -5,24 +5,27 @@
|
|||
// 2. AudioWorklet processor (playground/src/audio/worklet/...) — engine
|
||||
// calls. (Each instance owns its own WASM module instance.)
|
||||
//
|
||||
// FIXED-ARCHITECTURE LIMITATION (VERY IMPORTANT)
|
||||
// ----------------------------------------------
|
||||
// ARCHITECTURE (input dim is OVER-PROVISIONED for mix-and-match inputs)
|
||||
// ---------------------------------------------------------------------
|
||||
// The C++ MLP class is templated on layer sizes (architecture.md §4.1, §6.2).
|
||||
// We instantiate ONE concrete configuration here:
|
||||
//
|
||||
// using DefaultMLP = nisps::ml::MLP<2, 10, 14, 18, 126>;
|
||||
// using DefaultMLP = nisps::ml::MLP<32, 10, 14, 18, 126>;
|
||||
//
|
||||
// This was chosen as the union of the playground use case (2-D joystick →
|
||||
// 126 synth params) and the largest hidden-layer footprint that still fits
|
||||
// firmware budgets. `nisps_ml_create()` accepts caller-supplied input_size,
|
||||
// output_size, hidden[], n_hidden but ONLY validates them against the
|
||||
// compile-time defaults — extra inputs/outputs are clipped at the boundary.
|
||||
// If the caller passes incompatible dimensions we still create the module:
|
||||
// extra inputs are zero-padded, extra outputs are truncated, and the
|
||||
// hidden-layer override is silently ignored.
|
||||
// The 32-input dimension is the MAX number of composed input axes the manifold
|
||||
// front-end can feed (matches MAX_AXES in manifold/src/inputs/input-layer.ts).
|
||||
// The mix-and-match input layer (Internal XY pad + Game Controller + MIDI) gives
|
||||
// each active axis its OWN dedicated input slot — NO mean-blending — and feeds
|
||||
// the remaining (unused) slots a constant 0. The "active input dimension count"
|
||||
// is a front-end concept: a 2-axis pad uses slots 0–1, a 4-axis pad+stick uses
|
||||
// 0–3, etc. Because slot assignment is stable and unused slots are held at 0,
|
||||
// the net behaves as an N-input net where N = active axes; changing N is a
|
||||
// reshape, after which the front-end resets the weights (recreate-from-scratch,
|
||||
// behind a confirm modal). 126 outputs cover C15 + any current schema.
|
||||
//
|
||||
// Future work: ship multiple WASM modules (one per common architecture) or
|
||||
// rebuild on demand. See architecture.md "open questions" — Stream 7 punts.
|
||||
// `nisps_ml_create()` accepts caller-supplied input_size/output_size/hidden[]
|
||||
// but only validates them against these compile-time defaults — extra
|
||||
// inputs/outputs are clipped at the boundary and hidden overrides are ignored.
|
||||
//
|
||||
// WIRE FORMAT FOR WEIGHTS
|
||||
// -----------------------
|
||||
|
|
@ -85,7 +88,7 @@ namespace {
|
|||
// * 126 outputs — enough for the C15 mode and any current schema.
|
||||
//
|
||||
// The MLP also has dataset slots, loss history etc. — see mlp.hpp.
|
||||
using DefaultMLP = nisps::ml::MLP<2u, 10u, 14u, 18u, 126u>;
|
||||
using DefaultMLP = nisps::ml::MLP<32u, 10u, 14u, 18u, 126u>;
|
||||
|
||||
constexpr std::size_t kDefaultInputs = DefaultMLP::kInput;
|
||||
constexpr std::size_t kDefaultOutputs = DefaultMLP::kOutput;
|
||||
|
|
|
|||
Binary file not shown.
|
|
@ -21,7 +21,7 @@
|
|||
// 4. ChannelStrip engine: identical methodology.
|
||||
//
|
||||
// We use the EXACT SAME compile-time MLP architecture as the WASM build:
|
||||
// MLP<2, 10, 14, 18, 126>
|
||||
// MLP<32, 10, 14, 18, 126> (32-input max for mix-and-match; see bindings.cpp)
|
||||
//
|
||||
// Output blob format
|
||||
// ------------------
|
||||
|
|
@ -62,7 +62,7 @@
|
|||
|
||||
namespace {
|
||||
|
||||
using ParityMLP = nisps::ml::MLP<2u, 10u, 14u, 18u, 126u>;
|
||||
using ParityMLP = nisps::ml::MLP<32u, 10u, 14u, 18u, 126u>;
|
||||
|
||||
// The WASM bindings (nisps/wasm/bindings.cpp) sign-extend the 32-bit JS
|
||||
// seed via `s ^ (s << 32)`. To get bit-equal output between native and
|
||||
|
|
@ -133,9 +133,14 @@ int main(int argc, char** argv) {
|
|||
}
|
||||
|
||||
// ---- Stage 2: training ----
|
||||
constexpr std::array<std::array<float, 2u>, 3u> features = {{
|
||||
{{0.1f, 0.9f}}, {{0.5f, 0.5f}}, {{0.9f, 0.1f}},
|
||||
}};
|
||||
// Feature vectors are NIn(32)-wide: two real axes + zero-pad (the front-end
|
||||
// feeds the same shape — active axes in the low slots, unused slots at 0).
|
||||
// add_example requires features.size() >= NIn, so the pad is mandatory.
|
||||
constexpr std::size_t kNIn = ParityMLP::kInput;
|
||||
std::array<std::array<float, kNIn>, 3u> features = {};
|
||||
features[0][0] = 0.1f; features[0][1] = 0.9f;
|
||||
features[1][0] = 0.5f; features[1][1] = 0.5f;
|
||||
features[2][0] = 0.9f; features[2][1] = 0.1f;
|
||||
auto label_for = [](std::size_t i) {
|
||||
std::array<float, 126u> out{};
|
||||
const float a = static_cast<float>(i) * 0.3f + 0.05f;
|
||||
|
|
@ -146,7 +151,7 @@ int main(int argc, char** argv) {
|
|||
};
|
||||
for (std::size_t i = 0; i < features.size(); ++i) {
|
||||
const auto label = label_for(i);
|
||||
mlp.add_example(std::span<const float>(features[i].data(), 2u),
|
||||
mlp.add_example(std::span<const float>(features[i].data(), kNIn),
|
||||
std::span<const float>(label.data(), 126u));
|
||||
}
|
||||
const float final_loss = mlp.train(0.3f, 50u, 0.0f);
|
||||
|
|
|
|||
|
|
@ -191,8 +191,8 @@ async function main() {
|
|||
api.describe(dimsBuf);
|
||||
const dims = new Int32Array(Module.HEAP32.buffer, dimsBuf, 6).slice();
|
||||
api.free(dimsBuf);
|
||||
// Expect: [2, 10, 14, 18, 126, 4]
|
||||
const expectedDims = [2, 10, 14, 18, 126, 4];
|
||||
// Expect: [32, 10, 14, 18, 126, 4] (32-input max for mix-and-match)
|
||||
const expectedDims = [32, 10, 14, 18, 126, 4];
|
||||
for (let i = 0; i < expectedDims.length; ++i) {
|
||||
if (dims[i] !== expectedDims[i]) {
|
||||
console.error(`[parity_wasm] WASM build has dim[${i}]=${dims[i]}, native expected ${expectedDims[i]}`);
|
||||
|
|
@ -200,10 +200,11 @@ async function main() {
|
|||
process.exit(2);
|
||||
}
|
||||
}
|
||||
const N_IN = dims[0];
|
||||
const N_OUT = dims[4];
|
||||
|
||||
// --- Stage 1: ML inference ---
|
||||
const ml = api.create(2, N_OUT, 0, 0, SEED);
|
||||
const ml = api.create(N_IN, N_OUT, 0, 0, SEED);
|
||||
api.drawWeights(ml, 0.5);
|
||||
api.setInput(ml, 0, INPUT_X);
|
||||
api.setInput(ml, 1, INPUT_Y);
|
||||
|
|
@ -226,10 +227,13 @@ async function main() {
|
|||
for (let j = 0; j < N_OUT; ++j) out[j] = a + 0.005 * j;
|
||||
return out;
|
||||
};
|
||||
const featBuf = api.malloc(2 * 4);
|
||||
const featF32 = new Float32Array(api.HEAPF32.buffer, featBuf, 2);
|
||||
// Feature buffer is NIn-wide (zero-padded): two real axes + unused slots at 0,
|
||||
// matching the native side and the front-end's mix-and-match input shape.
|
||||
const featBuf = api.malloc(N_IN * 4);
|
||||
const featF32 = new Float32Array(api.HEAPF32.buffer, featBuf, N_IN);
|
||||
const labelBuf = api.malloc(N_OUT * 4);
|
||||
for (let i = 0; i < features.length; ++i) {
|
||||
featF32.fill(0);
|
||||
featF32[0] = features[i][0];
|
||||
featF32[1] = features[i][1];
|
||||
const label = labelFor(i);
|
||||
|
|
|
|||
Loading…
Reference in a new issue