feat(inputs): single/double controller toggle (shared core + manifold)

Toggle between one and two 2-D controllers without rebuilding the MLP.

Shared core (nisps/modes/base.hpp):
- ModeBase per-channel pin mask + pin_value (default 0.5). Pinned channels feed
  the neutral value to the MLP via effective_input(); tick_control/setup use it.
  Single-controller mode pins the second controller's two channels — the net is
  never resized, so trained state survives toggling.

Manifold/browser:
- Widen the WASM head MLP<2,…> -> MLP<4,10,14,18,126> (4 inputs = two 2-D
  controllers). All bindings are dimension-driven via DefaultMLP::kInput.
- Spine holds a full N-axis input vector: axes 0,1 = primary pad (existing input
  pipeline), axes 2+ = secondary controller, unused axes default to neutral 0.5.
  engine-api.setInputs feeds the whole vector; input-layer passthrough pads 0.5
  (the lossy 4->2 blend is now only a fallback for >4 axes).
- Inputs dock: Controllers Single/Double(+gamepad) toggle, default single.
- build-wasm.sh mirrors nisps.{js,wasm} into manifold/public.

Tests/docs:
- test_mode_paf_synth: pinned-inputs-neutralized covers the pin contract.
- parity_check.cpp + parity_wasm.mjs exercise all 4 inputs; native↔WASM parity
  PASS (max delta 2.4e-7).
- CLAUDE.md / MAP.md / nisps/wasm/README.md updated (no longer "fixed MLP<2,…>").

Firmware settings-menu toggle (Phase 3) still pending.
This commit is contained in:
monkey-w1n5t0n 2026-06-28 20:47:15 +02:00
parent e8f0866b47
commit ebb953a13f
15 changed files with 253 additions and 75 deletions

View file

@ -118,9 +118,9 @@ Two WASM instances at runtime:
1. **Main thread** (`playground/src/ml/wasm-iml.ts`): ML inference + sync training + RL primitives. Update store after each call. Async training via disposable Web Worker (`wasm-worker.ts`). 1. **Main thread** (`playground/src/ml/wasm-iml.ts`): ML inference + sync training + RL primitives. Update store after each call. Async training via disposable Web Worker (`wasm-worker.ts`).
2. **AudioWorklet** (`playground/src/audio/worklet/nisps-processor.ts`): runs engine `process_block` per audio block. Loads `nisps.wasm` directly via `WebAssembly.compile` (no Emscripten glue in worklet). Bytes posted from main thread. 2. **AudioWorklet** (`playground/src/audio/worklet/nisps-processor.ts`): runs engine `process_block` per audio block. Loads `nisps.wasm` directly via `WebAssembly.compile` (no Emscripten glue in worklet). Bytes posted from main thread.
C API is in `nisps/wasm/bindings.cpp`. Build: `bash scripts/build-wasm.sh` (~94KB output to `playground/public/`). C API is in `nisps/wasm/bindings.cpp`. Build: `bash scripts/build-wasm.sh` (~100KB output, mirrored to `playground/public/` and `manifold/public/`).
The WASM target is fixed at `MLP<2, 10, 14, 18, 126>`. Modes with smaller `output_size` use the first N outputs only. The WASM target is built as `MLP<4, 10, 14, 18, 126>` — 4 inputs = two 2-D controllers (primary pad on axes 0,1; optional second controller, e.g. gamepad, on axes 2,3). Single-controller mode pins axes 2,3 to the neutral 0.5 on the JS side (no rebuild). Modes with smaller `output_size` use the first N outputs only.
### Known limitations ### Known limitations

2
MAP.md
View file

@ -138,7 +138,7 @@ the "BUILD DELTAS" block at the top of `vcv/SPEC.md`). `src/MEMLNaut.cpp` (modul
- C++ identifiers: `PascalCase` types, `snake_case` functions/variables, `kPascalCase` constexpr. JSON keys `snake_case`. TS types `PascalCase`, components `PascalCase.tsx`, modules `kebab-case.ts`. - C++ identifiers: `PascalCase` types, `snake_case` functions/variables, `kPascalCase` constexpr. JSON keys `snake_case`. TS types `PascalCase`, components `PascalCase.tsx`, modules `kebab-case.ts`.
- `Curve` enum lives in `nisps/core/math.hpp` (lowercase: `linear/exp/log/square/sqrt/sigmoid/cubic`); generated mode headers re-export via `using Curve = ::nisps::Curve;`. TS mirror at `playground/src/output/curves.ts` with same names. - `Curve` enum lives in `nisps/core/math.hpp` (lowercase: `linear/exp/log/square/sqrt/sigmoid/cubic`); generated mode headers re-export via `using Curve = ::nisps::Curve;`. TS mirror at `playground/src/output/curves.ts` with same names.
- Modes are TSX components composed of primitives; mode parameter contracts are JSON schemas with codegen → C++/TS types. **No declarative JSON UI.** - Modes are TSX components composed of primitives; mode parameter contracts are JSON schemas with codegen → C++/TS types. **No declarative JSON UI.**
- WASM and firmware share the same C++; WASM is fixed at `MLP<2, 10, 14, 18, 126>` and modes use a slice of outputs based on schema's `output_size`. - WASM and firmware share the same C++; WASM is built as `MLP<4, 10, 14, 18, 126>` (4 inputs = two 2-D controllers; single-controller mode pins axes 2,3 to 0.5 in JS) and modes use a slice of outputs based on schema's `output_size`.
- Cross-platform parity: `scripts/parity-check.sh` enforces native vs WASM agreement within 1e-5. - Cross-platform parity: `scripts/parity-check.sh` enforces native vs WASM agreement within 1e-5.
## Gotchas ## Gotchas

Binary file not shown.

View file

@ -240,6 +240,7 @@ function InputsDrawer(ctx: ConsoleCtx, depth: DrawerDepth) {
const inp = ctx.inputs; const inp = ctx.inputs;
const enabledCount = inp.sources.filter((s) => s.enabled).length; const enabledCount = inp.sources.filter((s) => s.enabled).length;
const reshaping = inp.axisCount > inp.engineInputSize; const reshaping = inp.axisCount > inp.engineInputSize;
const gamepadOn = inp.sources.find((s) => s.kind === 'gamepad')?.enabled ?? false;
return ( return (
<> <>
@ -252,6 +253,22 @@ function InputsDrawer(ctx: ConsoleCtx, depth: DrawerDepth) {
{reshaping && <Chip tone="var(--warn)">blended → {inp.engineInputSize}</Chip>} {reshaping && <Chip tone="var(--warn)">blended → {inp.engineInputSize}</Chip>}
</div> </div>
{/* ---- Controllers: 1 (pad) vs 2 (pad + gamepad). Default single. ---- */}
<SectionLabel>Controllers</SectionLabel>
<Segmented
value={gamepadOn ? 'double' : 'single'}
onChange={(m) => inp.setEnabled('gamepad', m === 'double')}
options={[
{ value: 'single', label: 'Single' },
{ value: 'double', label: 'Double (+gamepad)' },
]}
/>
<p style={{ fontSize: 9, color: 'var(--fg-dim)', margin: '2px 0 0' }}>
{gamepadOn
? 'Two 2-D controllers feed the model (pad → axes 0,1; gamepad → axes 2,3).'
: 'One 2-D controller (the pad). The second controller’s axes stay neutral (0.5).'}
</p>
<SectionLabel>Sources</SectionLabel> <SectionLabel>Sources</SectionLabel>
<div style={{ display: 'flex', flexDirection: 'column', gap: 6 }}> <div style={{ display: 'flex', flexDirection: 'column', gap: 6 }}>
{inp.sources.map((s) => ( {inp.sources.map((s) => (
@ -382,16 +399,15 @@ function InputsDrawer(ctx: ConsoleCtx, depth: DrawerDepth) {
)} )}
<p style={{ fontSize: 9, color: 'var(--fg-dim)', margin: 0, lineHeight: 1.6 }}> <p style={{ fontSize: 9, color: 'var(--fg-dim)', margin: 0, lineHeight: 1.6 }}>
The active sources concatenate into one input vector at the head of the spine. The active sources concatenate into one input vector at the head of the spine. The
browser WASM is a {inp.engineInputSize}-input head (two 2-D controllers), so up to
{' '}{inp.engineInputSize} axes each get their own genuine input dimension; unused axes
stay neutral (0.5).
{reshaping {reshaping
? ` The browser WASM is a fixed ${inp.engineInputSize}-input head (MLP<2,…>), so the ? ` Right now ${inp.axisCount} axes exceed the ${inp.engineInputSize}-input head, so
${inp.axisCount} axes are blended down to ${inp.engineInputSize} (even→X / odd→Y mean).` the surplus is blended down (chunked mean) — drop a source to give every axis its
: ''}{' '} own dimension.`
{/* TODO(workstream F, docs/redesign/inputs-spec.md — "multiple WASM modules + : ''}
warm-start"): give every axis its own genuine input dimension by (re)loading a
WASM module whose MLP arity matches axisCount and warm-starting from the prior
net. Deferred — the reduction lives in InputLayer.compose(). */}
True per-axis dimensions land with the multi-WASM reshape (inputs-spec).
</p> </p>
</> </>
)} )}

View file

@ -164,9 +164,13 @@ export class EngineApi {
this.spine.setInput(x, y); this.spine.setInput(x, y);
} }
/** Set an arbitrary input vector (first two used as XY for the fixed 2→N MLP). */ /**
* Set the full composed input vector from the modular InputLayer. Axes 0,1
* are the primary controller; axes 2+ feed additional controllers (the WASM
* head is 4-wide = two 2-D controllers). Omitted axes stay neutral.
*/
setInputs(arr: ReadonlyArray<number>): void { setInputs(arr: ReadonlyArray<number>): void {
this.spine.setInput(arr[0] ?? 0.5, arr[1] ?? 0.5); this.spine.setInputs(arr);
} }
/** Live post-ML output vector (reused buffer — read, don't retain). */ /** Live post-ML output vector (reused buffer — read, don't retain). */

View file

@ -18,6 +18,7 @@
* Float32Array threaded through the output pipeline and handed to the backend. * Float32Array threaded through the output pipeline and handed to the backend.
*/ */
import { clamp } from './curves';
import { import {
defaultInputConfig, defaultInputConfig,
defaultInputState, defaultInputState,
@ -91,6 +92,11 @@ export class Spine implements EngineSink {
// Reused per-frame buffers — NO per-frame allocation in the hot path. // Reused per-frame buffers — NO per-frame allocation in the hot path.
private rawInput: [number, number] = [0.5, 0.5]; private rawInput: [number, number] = [0.5, 0.5];
// Full input vector fed to the MLP. Axes 0,1 = primary pad (post input
// pipeline); axes 2+ = secondary controller(s), or the neutral 0.5 when no
// second controller is active (single-controller mode). Sized to the WASM
// head's input arity; defaults all-neutral.
private inputs: number[] = [0.5, 0.5];
// Last raw input, so `EngineApi.process()` can re-tick after a weight change. // Last raw input, so `EngineApi.process()` can re-tick after a weight change.
lastRawX = 0.5; lastRawX = 0.5;
lastRawY = 0.5; lastRawY = 0.5;
@ -108,7 +114,11 @@ export class Spine implements EngineSink {
setState(patch: EngineStatePatch): void { setState(patch: EngineStatePatch): void {
let changed = false; let changed = false;
const s = this.state_; const s = this.state_;
if (patch.inputSize !== undefined && patch.inputSize !== s.inputSize) { s.inputSize = patch.inputSize; changed = true; } if (patch.inputSize !== undefined && patch.inputSize !== s.inputSize) {
s.inputSize = patch.inputSize;
this.resizeInputs_(patch.inputSize);
changed = true;
}
if (patch.outputSize !== undefined && patch.outputSize !== s.outputSize) { if (patch.outputSize !== undefined && patch.outputSize !== s.outputSize) {
s.outputSize = patch.outputSize; s.outputSize = patch.outputSize;
// Resize hot buffers to the resolved output size. // Resize hot buffers to the resolved output size.
@ -155,6 +165,16 @@ export class Spine implements EngineSink {
if (this.routedBuf === null || this.routedBuf.length !== iml.architecture.outputSize) { if (this.routedBuf === null || this.routedBuf.length !== iml.architecture.outputSize) {
this.routedBuf = new Float32Array(iml.architecture.outputSize); this.routedBuf = new Float32Array(iml.architecture.outputSize);
} }
this.resizeInputs_(iml.architecture.inputSize);
}
/** Resize the full input vector to the WASM head arity, neutral-filling. */
private resizeInputs_(size: number): void {
if (size < 1) size = 1;
if (this.inputs.length === size) return;
const next = new Array<number>(size).fill(0.5);
for (let i = 0; i < Math.min(size, this.inputs.length); i++) next[i] = this.inputs[i]!;
this.inputs = next;
} }
setBackendSend(backendSend: BackendSend | null): void { setBackendSend(backendSend: BackendSend | null): void {
@ -169,6 +189,22 @@ export class Spine implements EngineSink {
* Returns the routed buffer (live, reused — do not retain across calls). * Returns the routed buffer (live, reused — do not retain across calls).
*/ */
setInput(x: number, y: number): Float32Array | null { setInput(x: number, y: number): Float32Array | null {
return this.tick_(x, y, null);
}
/**
* Drive the full composed input vector (from the modular InputLayer). Axes
* 0,1 are the primary controller (run through the input pipeline); axes 2+
* are secondary controllers, written through with a plain [0,1] clamp. Any
* axis the caller omits keeps its previous value (neutral 0.5 by default), so
* single-controller mode — where only axes 0,1 are supplied — leaves the
* second pair pinned at the neutral value.
*/
setInputs(arr: ReadonlyArray<number>): Float32Array | null {
return this.tick_(arr[0] ?? 0.5, arr[1] ?? 0.5, arr);
}
private tick_(x: number, y: number, arr: ReadonlyArray<number> | null): Float32Array | null {
const iml = this.iml; const iml = this.iml;
if (!iml) return null; if (!iml) return null;
@ -176,17 +212,25 @@ export class Spine implements EngineSink {
const dt = this.lastTickMs > 0 ? (now - this.lastTickMs) / 1000 : 1 / 60; const dt = this.lastTickMs > 0 ? (now - this.lastTickMs) / 1000 : 1 / 60;
this.lastTickMs = now; this.lastTickMs = now;
// 1. processed (pure input pipeline) // 1. processed (pure input pipeline) — primary controller, axes 0,1.
this.rawInput[0] = x; this.rawInput[0] = x;
this.rawInput[1] = y; this.rawInput[1] = y;
this.lastRawX = x; this.lastRawX = x;
this.lastRawY = y; this.lastRawY = y;
const proc = processInput(this.rawInput, this.inputConfig, this.inputState, dt); const proc = processInput(this.rawInput, this.inputConfig, this.inputState, dt);
this.inputState = proc.state; this.inputState = proc.state;
this.inputs[0] = proc.x;
this.inputs[1] = proc.y;
// Secondary axes (2+): pass through with a [0,1] clamp. Omitted axes keep
// their prior value (neutral 0.5 in single-controller mode).
if (arr) {
for (let i = 2; i < this.inputs.length; i++) {
if (i < arr.length) this.inputs[i] = clamp(arr[i]!, 0, 1);
}
}
// 2. ml (inference into the reused buffer; no alloc) // 2. ml (inference into the reused buffer; no alloc)
iml.setInput(0, proc.x); for (let i = 0; i < this.inputs.length; i++) iml.setInput(i, this.inputs[i]!);
iml.setInput(1, proc.y);
iml.processInto(this.mlBuf); iml.processInto(this.mlBuf);
// Mirror to liveOutputs for imperative reads + bump. // Mirror to liveOutputs for imperative reads + bump.
this.liveOutputs.set(this.mlBuf.subarray(0, this.liveOutputs.length)); this.liveOutputs.set(this.mlBuf.subarray(0, this.liveOutputs.length));

View file

@ -13,24 +13,19 @@
* its value), but routing everything through one compose path keeps sources * its value), but routing everything through one compose path keeps sources
* composable and the channel layout coherent. * composable and the channel layout coherent.
* *
* ── Arity mismatch (the WASM reshape TODO) ────────────────────────────────── * ── Arity: 4-input head ─────────────────────────────────────────────────────
* The browser WASM is fixed at MLP<2, …, 126> — a TWO-input head. When the * The browser WASM head is MLP<4, …, 126> — TWO 2-D controllers. The primary
* composed vector has > 2 axes (double-stick gamepad = 4, MIDI learn-map = many) * controller (on-screen pad) occupies axes 0,1; an optional second controller
* we must reduce to 2 to feed today's engine. We do NOT fake a wider net. * (gamepad) occupies axes 2,3. `compose()` passes axes through 1:1 when the
* head is at least as wide as the composed axis count (the common cases —
* single pad = 2, pad + single-stick gamepad = 4), padding any unused axes with
* the neutral 0.5. Single-controller mode is therefore just "no second source":
* axes 2,3 stay neutral, slicing the 4-D manifold at the centre of controller 2.
* *
* chosen reduction (this pass): pairwise BLEND. * Over-subscription (composed axes > head width — e.g. pad + dual-stick gamepad
* inX = mean(axis[0], axis[2], axis[4], …) // even axes * = 6, or a large MIDI learn-map) still falls back to a chunked-mean blend; the
* inY = mean(axis[1], axis[3], axis[5], …) // odd axes * net is never rebuilt. That blend is the documented lossy path for unusual
* so a single stick passes straight through (axis0→X, axis1→Y), a double * source combinations, not the default.
* stick averages L/R into one XY, and MIDI axes fold into X/Y by parity.
*
* TODO(workstream F, docs/redesign/inputs-spec.md — "multiple WASM modules +
* warm-start"): the real fix is to (re)load a WASM module whose MLP input arity
* matches the composed axis count and warm-start its weights from the prior net,
* so every axis gets its own genuine input dimension instead of being blended.
* That is a larger build (multiple .wasm artefacts or a runtime-variadic head)
* and is deliberately deferred — this layer is wired so that swapping the
* reduction for a true reshape is a localised change in `compose()`.
*/ */
import type { InputAction, InputSource } from './types'; import type { InputAction, InputSource } from './types';
@ -147,15 +142,15 @@ export class InputLayer {
} }
/** /**
* Reduce the composed N-axis vector to the engine's input arity. * Map the composed N-axis vector to the engine's input arity.
* *
* For the fixed 2-input WASM, fold by parity (even→X, odd→Y) via mean. If a * Common case: the 4-input head is at least as wide as the composed axes, so
* future multi-module engine reports inputSize >= n, this passes axes through * pass through 1:1 and pad unused axes with the neutral 0.5. Over-subscription
* 1:1 (truncated/padded) — the seam where the real reshape lands. * (axes > head width) folds by parity / chunked mean — the lossy fallback.
*/ */
private compose(n: number, inputSize: number): number[] { private compose(n: number, inputSize: number): number[] {
if (inputSize >= n) { if (inputSize >= n) {
// True passthrough path (future multi-module head). Pad with 0.5. // Passthrough: each axis gets its own input dimension. Pad with 0.5.
const out = new Array<number>(inputSize); const out = new Array<number>(inputSize);
for (let i = 0; i < inputSize; i++) out[i] = i < n ? this.vector[i] : 0.5; for (let i = 0; i < inputSize; i++) out[i] = i < n ? this.vector[i] : 0.5;
return out; return out;

View file

@ -141,7 +141,7 @@ class ModeBase {
// Run an inference at default inputs so engine has params on first // Run an inference at default inputs so engine has params on first
// process() call, even if no input has been touched. // process() call, even if no input has been touched.
for (std::size_t i = 0u; i < NInputs; ++i) { for (std::size_t i = 0u; i < NInputs; ++i) {
ml_.set_input(i, input_channels_[i]); ml_.set_input(i, effective_input(i));
} }
ml_.process(); ml_.process();
if constexpr (kRouteOutputsToEngine) { if constexpr (kRouteOutputsToEngine) {
@ -160,13 +160,42 @@ class ModeBase {
input_dirty_ = true; input_dirty_ = true;
} }
// ---- Input neutralization (single/double controller toggle) ----
//
// A pinned channel feeds `pin_value_` (neutral, default 0.5) to the MLP
// instead of its live value, without rebuilding/resizing the network.
// Glue toggles which channels are pinned (e.g. single-joystick mode pins
// the second 2D controller's two channels). The stored live value is left
// untouched, so unpinning resumes from the controller's current position.
NISPS_FORCE_INLINE void set_input_pinned(std::size_t idx, bool pinned) noexcept {
if (idx >= NInputs) return;
input_pinned_[idx] = pinned;
input_dirty_ = true;
}
NISPS_FORCE_INLINE bool is_input_pinned(std::size_t idx) const noexcept {
return idx < NInputs && input_pinned_[idx];
}
NISPS_FORCE_INLINE void set_pin_value(float v) noexcept {
if (v < 0.f) v = 0.f;
else if (v > 1.f) v = 1.f;
pin_value_ = v;
input_dirty_ = true;
}
float pin_value() const noexcept { return pin_value_; }
// Effective value fed to the MLP for channel i (pin override applied).
NISPS_FORCE_INLINE float effective_input(std::size_t i) const noexcept {
return input_pinned_[i] ? pin_value_ : input_channels_[i];
}
NISPS_HOT void tick_control() noexcept { NISPS_HOT void tick_control() noexcept {
if constexpr (requires(Derived& d) { d.on_pre_inference(); }) { if constexpr (requires(Derived& d) { d.on_pre_inference(); }) {
static_cast<Derived&>(*this).on_pre_inference(); static_cast<Derived&>(*this).on_pre_inference();
} }
// Forward (possibly Derived-mutated) channels into the MLP. // Forward (possibly Derived-mutated) channels into the MLP, applying
// the per-channel pin override.
for (std::size_t i = 0u; i < NInputs; ++i) { for (std::size_t i = 0u; i < NInputs; ++i) {
ml_.set_input(i, input_channels_[i]); ml_.set_input(i, effective_input(i));
} }
ml_.process(); ml_.process();
if constexpr (kRouteOutputsToEngine) { if constexpr (kRouteOutputsToEngine) {
@ -235,6 +264,8 @@ class ModeBase {
EngineT engine_{}; EngineT engine_{};
MLPType ml_; MLPType ml_;
std::array<float, NInputs> input_channels_{}; std::array<float, NInputs> input_channels_{};
std::array<bool, NInputs> input_pinned_{}; // false => live
float pin_value_ = 0.5f; // neutral
bool input_dirty_ = false; bool input_dirty_ = false;
std::size_t voice_space_idx_ = 0u; std::size_t voice_space_idx_ = 0u;
RingBuffer<ControlEvent, kModeEventBufferSize> events_{}; RingBuffer<ControlEvent, kModeEventBufferSize> events_{};

View file

@ -32,10 +32,13 @@ The MLP class template is parametrised on `(input_size, hidden1, hidden2,
hidden3, output_size)`. WASM cannot recompile templates at runtime, so hidden3, output_size)`. WASM cannot recompile templates at runtime, so
this build instantiates exactly ONE configuration: this build instantiates exactly ONE configuration:
nisps::ml::MLP<2, 10, 14, 18, 126> nisps::ml::MLP<4, 10, 14, 18, 126>
That serves the playground use case (2-D joystick → up to 126 synth That serves the manifold/playground use case: two 2-D controllers (primary
parameters). `nisps_ml_create()` accepts caller-supplied dimensions for pad on axes 0,1; optional second controller — e.g. gamepad — on axes 2,3)
→ up to 126 synth parameters. Single-controller mode pins axes 2,3 to the
neutral 0.5 on the JS side, so the net is never rebuilt when toggling
between one and two controllers. `nisps_ml_create()` accepts caller-supplied dimensions for
forward compatibility but currently ignores them — see comment at the top forward compatibility but currently ignores them — see comment at the top
of `bindings.cpp`. The schemas in `schemas/modes/*.json` use up to of `bindings.cpp`. The schemas in `schemas/modes/*.json` use up to
`output_size=126`; modes whose output_size is < 126 simply ignore the `output_size=126`; modes whose output_size is < 126 simply ignore the

View file

@ -5,24 +5,26 @@
// 2. AudioWorklet processor (playground/src/audio/worklet/...) — engine // 2. AudioWorklet processor (playground/src/audio/worklet/...) — engine
// calls. (Each instance owns its own WASM module instance.) // calls. (Each instance owns its own WASM module instance.)
// //
// FIXED-ARCHITECTURE LIMITATION (VERY IMPORTANT) // COMPILE-TIME ARCHITECTURE
// ---------------------------------------------- // -------------------------
// The C++ MLP class is templated on layer sizes (architecture.md §4.1, §6.2). // The C++ MLP class is templated on layer sizes (architecture.md §4.1, §6.2).
// We instantiate ONE concrete configuration here: // We instantiate ONE concrete configuration here:
// //
// using DefaultMLP = nisps::ml::MLP<2, 10, 14, 18, 126>; // using DefaultMLP = nisps::ml::MLP<4, 10, 14, 18, 126>;
// //
// This was chosen as the union of the playground use case (2-D joystick → // FOUR inputs = two 2-D controllers (primary on-screen pad on axes 0,1; an
// 126 synth params) and the largest hidden-layer footprint that still fits // optional second controller — gamepad — on axes 2,3). Single-controller mode
// firmware budgets. `nisps_ml_create()` accepts caller-supplied input_size, // pins the second pair to the neutral value 0.5 on the JS side (see
// output_size, hidden[], n_hidden but ONLY validates them against the // manifold/src/engine/spine.ts), so the network is never rebuilt/resized when
// compile-time defaults — extra inputs/outputs are clipped at the boundary. // toggling between one and two controllers. 126 outputs covers C15 and every
// If the caller passes incompatible dimensions we still create the module: // current schema.
// extra inputs are zero-padded, extra outputs are truncated, and the
// hidden-layer override is silently ignored.
// //
// Future work: ship multiple WASM modules (one per common architecture) or // `nisps_ml_create()` accepts caller-supplied input_size, output_size,
// rebuild on demand. See architecture.md "open questions" — Stream 7 punts. // hidden[], n_hidden but ONLY validates them against the compile-time
// defaults — extra inputs/outputs are clipped at the boundary. If the caller
// passes incompatible dimensions we still create the module: extra inputs are
// zero-padded, extra outputs are truncated, and the hidden-layer override is
// silently ignored. The TS side reads the real dims back via nisps_ml_describe.
// //
// WIRE FORMAT FOR WEIGHTS // WIRE FORMAT FOR WEIGHTS
// ----------------------- // -----------------------
@ -78,14 +80,15 @@ namespace {
// Compile-time default architecture. See header comment. // Compile-time default architecture. See header comment.
// //
// Choice rationale: // Choice rationale:
// * 2 inputs — playground virtual joystick (X, Y). // * 4 inputs — two 2-D controllers (primary pad axes 0,1; optional
// second controller axes 2,3). Single-controller mode pins axes 2,3 to 0.5.
// * [10, 14, 18] hidden — covers the largest schema layouts in // * [10, 14, 18] hidden — covers the largest schema layouts in
// `schemas/modes/*.json` (channel_strip variants, verb_fx, breakor, // `schemas/modes/*.json` (channel_strip variants, verb_fx, breakor,
// elysiamorf, memlcelium). // elysiamorf, memlcelium).
// * 126 outputs — enough for the C15 mode and any current schema. // * 126 outputs — enough for the C15 mode and any current schema.
// //
// The MLP also has dataset slots, loss history etc. — see mlp.hpp. // The MLP also has dataset slots, loss history etc. — see mlp.hpp.
using DefaultMLP = nisps::ml::MLP<2u, 10u, 14u, 18u, 126u>; using DefaultMLP = nisps::ml::MLP<4u, 10u, 14u, 18u, 126u>;
constexpr std::size_t kDefaultInputs = DefaultMLP::kInput; constexpr std::size_t kDefaultInputs = DefaultMLP::kInput;
constexpr std::size_t kDefaultOutputs = DefaultMLP::kOutput; constexpr std::size_t kDefaultOutputs = DefaultMLP::kOutput;

Binary file not shown.

View file

@ -82,3 +82,12 @@ set -x
echo "[build-wasm] wrote $OUT/nisps.js + $OUT/nisps.wasm" echo "[build-wasm] wrote $OUT/nisps.js + $OUT/nisps.wasm"
ls -lh "$OUT/nisps.js" "$OUT/nisps.wasm" ls -lh "$OUT/nisps.js" "$OUT/nisps.wasm"
# Mirror into the manifold app's public/ — it serves its own copy (and its
# deploy webhook builds manifold/), so both must stay in lockstep with the
# parity-checked artifacts above.
MANIFOLD_OUT="$ROOT/manifold/public"
if [[ -d "$MANIFOLD_OUT" ]]; then
cp "$OUT/nisps.js" "$OUT/nisps.wasm" "$MANIFOLD_OUT/"
echo "[build-wasm] mirrored nisps.{js,wasm} → $MANIFOLD_OUT"
fi

View file

@ -13,7 +13,7 @@
// //
// What we cover // What we cover
// ------------- // -------------
// 1. ML: seed=42, draw_weights(0.5), set_input(0.25, 0.75), process. // 1. ML: seed=42, draw_weights(0.5), set_input(0.25, 0.75, 0.40, 0.60), process.
// → 126 outputs + 12 weights sampled at known offsets. // → 126 outputs + 12 weights sampled at known offsets.
// 2. ML training: 3 examples added, train(0.3, 50, 0), capture loss + outputs. // 2. ML training: 3 examples added, train(0.3, 50, 0), capture loss + outputs.
// 3. PAFSynth engine: seed-equivalent setup (params=0.5), 128-sample run on // 3. PAFSynth engine: seed-equivalent setup (params=0.5), 128-sample run on
@ -21,7 +21,7 @@
// 4. ChannelStrip engine: identical methodology. // 4. ChannelStrip engine: identical methodology.
// //
// We use the EXACT SAME compile-time MLP architecture as the WASM build: // We use the EXACT SAME compile-time MLP architecture as the WASM build:
// MLP<2, 10, 14, 18, 126> // MLP<4, 10, 14, 18, 126>
// //
// Output blob format // Output blob format
// ------------------ // ------------------
@ -62,7 +62,7 @@
namespace { namespace {
using ParityMLP = nisps::ml::MLP<2u, 10u, 14u, 18u, 126u>; using ParityMLP = nisps::ml::MLP<4u, 10u, 14u, 18u, 126u>;
// The WASM bindings (nisps/wasm/bindings.cpp) sign-extend the 32-bit JS // The WASM bindings (nisps/wasm/bindings.cpp) sign-extend the 32-bit JS
// seed via `s ^ (s << 32)`. To get bit-equal output between native and // seed via `s ^ (s << 32)`. To get bit-equal output between native and
@ -73,6 +73,9 @@ constexpr std::uint64_t kSeed = static_cast<std::uint64_t>(kSeed32)
^ (static_cast<std::uint64_t>(kSeed32) << 32); ^ (static_cast<std::uint64_t>(kSeed32) << 32);
constexpr float kInputX = 0.25f; constexpr float kInputX = 0.25f;
constexpr float kInputY = 0.75f; constexpr float kInputY = 0.75f;
// Second controller (axes 2,3) — exercised so the 4-input head is fully probed.
constexpr float kInputZ = 0.40f;
constexpr float kInputW = 0.60f;
constexpr float kSampleRate = 48000.0f; constexpr float kSampleRate = 48000.0f;
constexpr std::size_t kSynthFrames = 128u; constexpr std::size_t kSynthFrames = 128u;
@ -118,6 +121,8 @@ int main(int argc, char** argv) {
mlp.draw_weights(0.5f); mlp.draw_weights(0.5f);
mlp.set_input(0u, kInputX); mlp.set_input(0u, kInputX);
mlp.set_input(1u, kInputY); mlp.set_input(1u, kInputY);
mlp.set_input(2u, kInputZ);
mlp.set_input(3u, kInputW);
mlp.process(); mlp.process();
{ {
const auto outs = mlp.outputs(); const auto outs = mlp.outputs();
@ -133,8 +138,8 @@ int main(int argc, char** argv) {
} }
// ---- Stage 2: training ---- // ---- Stage 2: training ----
constexpr std::array<std::array<float, 2u>, 3u> features = {{ constexpr std::array<std::array<float, 4u>, 3u> features = {{
{{0.1f, 0.9f}}, {{0.5f, 0.5f}}, {{0.9f, 0.1f}}, {{0.1f, 0.9f, 0.3f, 0.7f}}, {{0.5f, 0.5f, 0.5f, 0.5f}}, {{0.9f, 0.1f, 0.7f, 0.3f}},
}}; }};
auto label_for = [](std::size_t i) { auto label_for = [](std::size_t i) {
std::array<float, 126u> out{}; std::array<float, 126u> out{};
@ -146,13 +151,15 @@ int main(int argc, char** argv) {
}; };
for (std::size_t i = 0; i < features.size(); ++i) { for (std::size_t i = 0; i < features.size(); ++i) {
const auto label = label_for(i); const auto label = label_for(i);
mlp.add_example(std::span<const float>(features[i].data(), 2u), mlp.add_example(std::span<const float>(features[i].data(), 4u),
std::span<const float>(label.data(), 126u)); std::span<const float>(label.data(), 126u));
} }
const float final_loss = mlp.train(0.3f, 50u, 0.0f); const float final_loss = mlp.train(0.3f, 50u, 0.0f);
mlp.set_input(0u, kInputX); mlp.set_input(0u, kInputX);
mlp.set_input(1u, kInputY); mlp.set_input(1u, kInputY);
mlp.set_input(2u, kInputZ);
mlp.set_input(3u, kInputW);
mlp.process(); mlp.process();
{ {
const auto outs = mlp.outputs(); const auto outs = mlp.outputs();

View file

@ -43,6 +43,8 @@ const VERSION = 3; // v3 adds stage 5d (ExploreAndPlace lifecycle)
const SEED = 42 >>> 0; const SEED = 42 >>> 0;
const INPUT_X = 0.25; const INPUT_X = 0.25;
const INPUT_Y = 0.75; const INPUT_Y = 0.75;
const INPUT_Z = 0.40; // second controller axes 2,3 (must match parity_check.cpp)
const INPUT_W = 0.60;
const SAMPLE_RATE = 48000; const SAMPLE_RATE = 48000;
const SYNTH_FRAMES = 128; const SYNTH_FRAMES = 128;
const PROBE_IDX = [0, 5, 19, 31, 73, 137, 251, 491, 999, 1583, 2401, 3289]; const PROBE_IDX = [0, 5, 19, 31, 73, 137, 251, 491, 999, 1583, 2401, 3289];
@ -191,8 +193,8 @@ async function main() {
api.describe(dimsBuf); api.describe(dimsBuf);
const dims = new Int32Array(Module.HEAP32.buffer, dimsBuf, 6).slice(); const dims = new Int32Array(Module.HEAP32.buffer, dimsBuf, 6).slice();
api.free(dimsBuf); api.free(dimsBuf);
// Expect: [2, 10, 14, 18, 126, 4] // Expect: [inputs, h1, h2, h3, outputs, numLayers] = [4, 10, 14, 18, 126, 4]
const expectedDims = [2, 10, 14, 18, 126, 4]; const expectedDims = [4, 10, 14, 18, 126, 4];
for (let i = 0; i < expectedDims.length; ++i) { for (let i = 0; i < expectedDims.length; ++i) {
if (dims[i] !== expectedDims[i]) { if (dims[i] !== expectedDims[i]) {
console.error(`[parity_wasm] WASM build has dim[${i}]=${dims[i]}, native expected ${expectedDims[i]}`); console.error(`[parity_wasm] WASM build has dim[${i}]=${dims[i]}, native expected ${expectedDims[i]}`);
@ -203,10 +205,12 @@ async function main() {
const N_OUT = dims[4]; const N_OUT = dims[4];
// --- Stage 1: ML inference --- // --- Stage 1: ML inference ---
const ml = api.create(2, N_OUT, 0, 0, SEED); const ml = api.create(4, N_OUT, 0, 0, SEED);
api.drawWeights(ml, 0.5); api.drawWeights(ml, 0.5);
api.setInput(ml, 0, INPUT_X); api.setInput(ml, 0, INPUT_X);
api.setInput(ml, 1, INPUT_Y); api.setInput(ml, 1, INPUT_Y);
api.setInput(ml, 2, INPUT_Z);
api.setInput(ml, 3, INPUT_W);
api.process(ml); api.process(ml);
const outsStage1 = getOutputsCopy(api, ml, N_OUT); const outsStage1 = getOutputsCopy(api, ml, N_OUT);
@ -216,9 +220,9 @@ async function main() {
// --- Stage 2: training --- // --- Stage 2: training ---
const features = [ const features = [
[0.1, 0.9], [0.1, 0.9, 0.3, 0.7],
[0.5, 0.5], [0.5, 0.5, 0.5, 0.5],
[0.9, 0.1], [0.9, 0.1, 0.7, 0.3],
]; ];
const labelFor = (i) => { const labelFor = (i) => {
const out = new Float32Array(N_OUT); const out = new Float32Array(N_OUT);
@ -226,12 +230,14 @@ async function main() {
for (let j = 0; j < N_OUT; ++j) out[j] = a + 0.005 * j; for (let j = 0; j < N_OUT; ++j) out[j] = a + 0.005 * j;
return out; return out;
}; };
const featBuf = api.malloc(2 * 4); const featBuf = api.malloc(4 * 4);
const featF32 = new Float32Array(api.HEAPF32.buffer, featBuf, 2); const featF32 = new Float32Array(api.HEAPF32.buffer, featBuf, 4);
const labelBuf = api.malloc(N_OUT * 4); const labelBuf = api.malloc(N_OUT * 4);
for (let i = 0; i < features.length; ++i) { for (let i = 0; i < features.length; ++i) {
featF32[0] = features[i][0]; featF32[0] = features[i][0];
featF32[1] = features[i][1]; featF32[1] = features[i][1];
featF32[2] = features[i][2];
featF32[3] = features[i][3];
const label = labelFor(i); const label = labelFor(i);
new Float32Array(api.HEAPF32.buffer, labelBuf, N_OUT).set(label); new Float32Array(api.HEAPF32.buffer, labelBuf, N_OUT).set(label);
api.addExample(ml, featBuf, labelBuf); api.addExample(ml, featBuf, labelBuf);
@ -243,6 +249,8 @@ async function main() {
api.setInput(ml, 0, INPUT_X); api.setInput(ml, 0, INPUT_X);
api.setInput(ml, 1, INPUT_Y); api.setInput(ml, 1, INPUT_Y);
api.setInput(ml, 2, INPUT_Z);
api.setInput(ml, 3, INPUT_W);
api.process(ml); api.process(ml);
const outsStage2 = getOutputsCopy(api, ml, N_OUT); const outsStage2 = getOutputsCopy(api, ml, N_OUT);
@ -301,6 +309,8 @@ async function main() {
api.feedbackUndo(ml); // pop nudge api.feedbackUndo(ml); // pop nudge
api.setInput(ml, 0, INPUT_X); api.setInput(ml, 0, INPUT_X);
api.setInput(ml, 1, INPUT_Y); api.setInput(ml, 1, INPUT_Y);
api.setInput(ml, 2, INPUT_Z);
api.setInput(ml, 3, INPUT_W);
api.process(ml); api.process(ml);
api.feedbackLike(ml); // begin place: freeze scratchpad output api.feedbackLike(ml); // begin place: freeze scratchpad output
{ {

View file

@ -88,6 +88,62 @@ NISPS_TEST(paf_synth_mode_input_clamping) {
m.set_input(99u, 0.5f); m.set_input(99u, 0.5f);
} }
NISPS_TEST(paf_synth_mode_pinned_inputs_neutralized) {
// Single-joystick mode pins the second 2D controller's channels (2,3).
// A pinned channel must feed the neutral pin value (default 0.5) to the
// MLP regardless of its live value, and pinning must not resize the model.
PAFSynthMode m;
m.setup(48000.f);
NISPS_EXPECT(m.pin_value() == 0.5f);
// Reference: channels 2,3 driven live to 0.5 (the neutral value).
m.set_input(0, 0.2f);
m.set_input(1, 0.8f);
m.set_input(2, 0.5f);
m.set_input(3, 0.5f);
m.tick_control();
std::array<float, 33> ref{};
{
const auto outs = m.ml().outputs();
for (std::size_t i = 0u; i < outs.size(); ++i) ref[i] = outs[i];
}
// Pin channels 2,3, then drive them to arbitrary values. Output must match
// the reference (they are neutralized to 0.5).
m.set_input_pinned(2, true);
m.set_input_pinned(3, true);
NISPS_EXPECT(m.is_input_pinned(2));
m.set_input(2, 0.0f);
m.set_input(3, 1.0f);
m.tick_control();
for (std::size_t i = 0u; i < 33u; ++i) {
NISPS_EXPECT(std::fabs(m.ml().outputs()[i] - ref[i]) < 1e-6f);
}
// The active controller (channels 0,1) still affects the output.
m.set_input(0, 0.9f);
m.tick_control();
bool changed = false;
for (std::size_t i = 0u; i < 33u; ++i) {
if (std::fabs(m.ml().outputs()[i] - ref[i]) > 1e-6f) { changed = true; break; }
}
NISPS_EXPECT(changed);
// Unpinning restores sensitivity on channels 2,3.
m.set_input_pinned(2, false);
m.set_input_pinned(3, false);
m.set_input(0, 0.2f); // restore active channels to the reference pose
m.set_input(1, 0.8f);
m.set_input(2, 0.0f);
m.set_input(3, 1.0f);
m.tick_control();
bool differs = false;
for (std::size_t i = 0u; i < 33u; ++i) {
if (std::fabs(m.ml().outputs()[i] - ref[i]) > 1e-6f) { differs = true; break; }
}
NISPS_EXPECT(differs);
}
NISPS_TEST(paf_synth_mode_engine_and_ml_accessors) { NISPS_TEST(paf_synth_mode_engine_and_ml_accessors) {
PAFSynthMode m; PAFSynthMode m;
m.setup(48000.f); m.setup(48000.f);