feat(inputs): single/double controller toggle (shared core + manifold)
Toggle between one and two 2-D controllers without rebuilding the MLP.
Shared core (nisps/modes/base.hpp):
- ModeBase per-channel pin mask + pin_value (default 0.5). Pinned channels feed
the neutral value to the MLP via effective_input(); tick_control/setup use it.
Single-controller mode pins the second controller's two channels — the net is
never resized, so trained state survives toggling.
Manifold/browser:
- Widen the WASM head MLP<2,…> -> MLP<4,10,14,18,126> (4 inputs = two 2-D
controllers). All bindings are dimension-driven via DefaultMLP::kInput.
- Spine holds a full N-axis input vector: axes 0,1 = primary pad (existing input
pipeline), axes 2+ = secondary controller, unused axes default to neutral 0.5.
engine-api.setInputs feeds the whole vector; input-layer passthrough pads 0.5
(the lossy 4->2 blend is now only a fallback for >4 axes).
- Inputs dock: Controllers Single/Double(+gamepad) toggle, default single.
- build-wasm.sh mirrors nisps.{js,wasm} into manifold/public.
Tests/docs:
- test_mode_paf_synth: pinned-inputs-neutralized covers the pin contract.
- parity_check.cpp + parity_wasm.mjs exercise all 4 inputs; native↔WASM parity
PASS (max delta 2.4e-7).
- CLAUDE.md / MAP.md / nisps/wasm/README.md updated (no longer "fixed MLP<2,…>").
Firmware settings-menu toggle (Phase 3) still pending.
This commit is contained in:
parent
e8f0866b47
commit
ebb953a13f
15 changed files with 253 additions and 75 deletions
|
|
@ -118,9 +118,9 @@ Two WASM instances at runtime:
|
|||
1. **Main thread** (`playground/src/ml/wasm-iml.ts`): ML inference + sync training + RL primitives. Update store after each call. Async training via disposable Web Worker (`wasm-worker.ts`).
|
||||
2. **AudioWorklet** (`playground/src/audio/worklet/nisps-processor.ts`): runs engine `process_block` per audio block. Loads `nisps.wasm` directly via `WebAssembly.compile` (no Emscripten glue in worklet). Bytes posted from main thread.
|
||||
|
||||
C API is in `nisps/wasm/bindings.cpp`. Build: `bash scripts/build-wasm.sh` (~94KB output to `playground/public/`).
|
||||
C API is in `nisps/wasm/bindings.cpp`. Build: `bash scripts/build-wasm.sh` (~100KB output, mirrored to `playground/public/` and `manifold/public/`).
|
||||
|
||||
The WASM target is fixed at `MLP<2, 10, 14, 18, 126>`. Modes with smaller `output_size` use the first N outputs only.
|
||||
The WASM target is built as `MLP<4, 10, 14, 18, 126>` — 4 inputs = two 2-D controllers (primary pad on axes 0,1; optional second controller, e.g. gamepad, on axes 2,3). Single-controller mode pins axes 2,3 to the neutral 0.5 on the JS side (no rebuild). Modes with smaller `output_size` use the first N outputs only.
|
||||
|
||||
### Known limitations
|
||||
|
||||
|
|
|
|||
2
MAP.md
2
MAP.md
|
|
@ -138,7 +138,7 @@ the "BUILD DELTAS" block at the top of `vcv/SPEC.md`). `src/MEMLNaut.cpp` (modul
|
|||
- C++ identifiers: `PascalCase` types, `snake_case` functions/variables, `kPascalCase` constexpr. JSON keys `snake_case`. TS types `PascalCase`, components `PascalCase.tsx`, modules `kebab-case.ts`.
|
||||
- `Curve` enum lives in `nisps/core/math.hpp` (lowercase: `linear/exp/log/square/sqrt/sigmoid/cubic`); generated mode headers re-export via `using Curve = ::nisps::Curve;`. TS mirror at `playground/src/output/curves.ts` with same names.
|
||||
- Modes are TSX components composed of primitives; mode parameter contracts are JSON schemas with codegen → C++/TS types. **No declarative JSON UI.**
|
||||
- WASM and firmware share the same C++; WASM is fixed at `MLP<2, 10, 14, 18, 126>` and modes use a slice of outputs based on schema's `output_size`.
|
||||
- WASM and firmware share the same C++; WASM is built as `MLP<4, 10, 14, 18, 126>` (4 inputs = two 2-D controllers; single-controller mode pins axes 2,3 to 0.5 in JS) and modes use a slice of outputs based on schema's `output_size`.
|
||||
- Cross-platform parity: `scripts/parity-check.sh` enforces native vs WASM agreement within 1e-5.
|
||||
|
||||
## Gotchas
|
||||
|
|
|
|||
Binary file not shown.
|
|
@ -240,6 +240,7 @@ function InputsDrawer(ctx: ConsoleCtx, depth: DrawerDepth) {
|
|||
const inp = ctx.inputs;
|
||||
const enabledCount = inp.sources.filter((s) => s.enabled).length;
|
||||
const reshaping = inp.axisCount > inp.engineInputSize;
|
||||
const gamepadOn = inp.sources.find((s) => s.kind === 'gamepad')?.enabled ?? false;
|
||||
|
||||
return (
|
||||
<>
|
||||
|
|
@ -252,6 +253,22 @@ function InputsDrawer(ctx: ConsoleCtx, depth: DrawerDepth) {
|
|||
{reshaping && <Chip tone="var(--warn)">blended → {inp.engineInputSize}</Chip>}
|
||||
</div>
|
||||
|
||||
{/* ---- Controllers: 1 (pad) vs 2 (pad + gamepad). Default single. ---- */}
|
||||
<SectionLabel>Controllers</SectionLabel>
|
||||
<Segmented
|
||||
value={gamepadOn ? 'double' : 'single'}
|
||||
onChange={(m) => inp.setEnabled('gamepad', m === 'double')}
|
||||
options={[
|
||||
{ value: 'single', label: 'Single' },
|
||||
{ value: 'double', label: 'Double (+gamepad)' },
|
||||
]}
|
||||
/>
|
||||
<p style={{ fontSize: 9, color: 'var(--fg-dim)', margin: '2px 0 0' }}>
|
||||
{gamepadOn
|
||||
? 'Two 2-D controllers feed the model (pad → axes 0,1; gamepad → axes 2,3).'
|
||||
: 'One 2-D controller (the pad). The second controller’s axes stay neutral (0.5).'}
|
||||
</p>
|
||||
|
||||
<SectionLabel>Sources</SectionLabel>
|
||||
<div style={{ display: 'flex', flexDirection: 'column', gap: 6 }}>
|
||||
{inp.sources.map((s) => (
|
||||
|
|
@ -382,16 +399,15 @@ function InputsDrawer(ctx: ConsoleCtx, depth: DrawerDepth) {
|
|||
)}
|
||||
|
||||
<p style={{ fontSize: 9, color: 'var(--fg-dim)', margin: 0, lineHeight: 1.6 }}>
|
||||
The active sources concatenate into one input vector at the head of the spine.
|
||||
The active sources concatenate into one input vector at the head of the spine. The
|
||||
browser WASM is a {inp.engineInputSize}-input head (two 2-D controllers), so up to
|
||||
{' '}{inp.engineInputSize} axes each get their own genuine input dimension; unused axes
|
||||
stay neutral (0.5).
|
||||
{reshaping
|
||||
? ` The browser WASM is a fixed ${inp.engineInputSize}-input head (MLP<2,…>), so the
|
||||
${inp.axisCount} axes are blended down to ${inp.engineInputSize} (even→X / odd→Y mean).`
|
||||
: ''}{' '}
|
||||
{/* TODO(workstream F, docs/redesign/inputs-spec.md — "multiple WASM modules +
|
||||
warm-start"): give every axis its own genuine input dimension by (re)loading a
|
||||
WASM module whose MLP arity matches axisCount and warm-starting from the prior
|
||||
net. Deferred — the reduction lives in InputLayer.compose(). */}
|
||||
True per-axis dimensions land with the multi-WASM reshape (inputs-spec).
|
||||
? ` Right now ${inp.axisCount} axes exceed the ${inp.engineInputSize}-input head, so
|
||||
the surplus is blended down (chunked mean) — drop a source to give every axis its
|
||||
own dimension.`
|
||||
: ''}
|
||||
</p>
|
||||
</>
|
||||
)}
|
||||
|
|
|
|||
|
|
@ -164,9 +164,13 @@ export class EngineApi {
|
|||
this.spine.setInput(x, y);
|
||||
}
|
||||
|
||||
/** Set an arbitrary input vector (first two used as XY for the fixed 2→N MLP). */
|
||||
/**
|
||||
* Set the full composed input vector from the modular InputLayer. Axes 0,1
|
||||
* are the primary controller; axes 2+ feed additional controllers (the WASM
|
||||
* head is 4-wide = two 2-D controllers). Omitted axes stay neutral.
|
||||
*/
|
||||
setInputs(arr: ReadonlyArray<number>): void {
|
||||
this.spine.setInput(arr[0] ?? 0.5, arr[1] ?? 0.5);
|
||||
this.spine.setInputs(arr);
|
||||
}
|
||||
|
||||
/** Live post-ML output vector (reused buffer — read, don't retain). */
|
||||
|
|
|
|||
|
|
@ -18,6 +18,7 @@
|
|||
* Float32Array threaded through the output pipeline and handed to the backend.
|
||||
*/
|
||||
|
||||
import { clamp } from './curves';
|
||||
import {
|
||||
defaultInputConfig,
|
||||
defaultInputState,
|
||||
|
|
@ -91,6 +92,11 @@ export class Spine implements EngineSink {
|
|||
|
||||
// Reused per-frame buffers — NO per-frame allocation in the hot path.
|
||||
private rawInput: [number, number] = [0.5, 0.5];
|
||||
// Full input vector fed to the MLP. Axes 0,1 = primary pad (post input
|
||||
// pipeline); axes 2+ = secondary controller(s), or the neutral 0.5 when no
|
||||
// second controller is active (single-controller mode). Sized to the WASM
|
||||
// head's input arity; defaults all-neutral.
|
||||
private inputs: number[] = [0.5, 0.5];
|
||||
// Last raw input, so `EngineApi.process()` can re-tick after a weight change.
|
||||
lastRawX = 0.5;
|
||||
lastRawY = 0.5;
|
||||
|
|
@ -108,7 +114,11 @@ export class Spine implements EngineSink {
|
|||
setState(patch: EngineStatePatch): void {
|
||||
let changed = false;
|
||||
const s = this.state_;
|
||||
if (patch.inputSize !== undefined && patch.inputSize !== s.inputSize) { s.inputSize = patch.inputSize; changed = true; }
|
||||
if (patch.inputSize !== undefined && patch.inputSize !== s.inputSize) {
|
||||
s.inputSize = patch.inputSize;
|
||||
this.resizeInputs_(patch.inputSize);
|
||||
changed = true;
|
||||
}
|
||||
if (patch.outputSize !== undefined && patch.outputSize !== s.outputSize) {
|
||||
s.outputSize = patch.outputSize;
|
||||
// Resize hot buffers to the resolved output size.
|
||||
|
|
@ -155,6 +165,16 @@ export class Spine implements EngineSink {
|
|||
if (this.routedBuf === null || this.routedBuf.length !== iml.architecture.outputSize) {
|
||||
this.routedBuf = new Float32Array(iml.architecture.outputSize);
|
||||
}
|
||||
this.resizeInputs_(iml.architecture.inputSize);
|
||||
}
|
||||
|
||||
/** Resize the full input vector to the WASM head arity, neutral-filling. */
|
||||
private resizeInputs_(size: number): void {
|
||||
if (size < 1) size = 1;
|
||||
if (this.inputs.length === size) return;
|
||||
const next = new Array<number>(size).fill(0.5);
|
||||
for (let i = 0; i < Math.min(size, this.inputs.length); i++) next[i] = this.inputs[i]!;
|
||||
this.inputs = next;
|
||||
}
|
||||
|
||||
setBackendSend(backendSend: BackendSend | null): void {
|
||||
|
|
@ -169,6 +189,22 @@ export class Spine implements EngineSink {
|
|||
* Returns the routed buffer (live, reused — do not retain across calls).
|
||||
*/
|
||||
setInput(x: number, y: number): Float32Array | null {
|
||||
return this.tick_(x, y, null);
|
||||
}
|
||||
|
||||
/**
|
||||
* Drive the full composed input vector (from the modular InputLayer). Axes
|
||||
* 0,1 are the primary controller (run through the input pipeline); axes 2+
|
||||
* are secondary controllers, written through with a plain [0,1] clamp. Any
|
||||
* axis the caller omits keeps its previous value (neutral 0.5 by default), so
|
||||
* single-controller mode — where only axes 0,1 are supplied — leaves the
|
||||
* second pair pinned at the neutral value.
|
||||
*/
|
||||
setInputs(arr: ReadonlyArray<number>): Float32Array | null {
|
||||
return this.tick_(arr[0] ?? 0.5, arr[1] ?? 0.5, arr);
|
||||
}
|
||||
|
||||
private tick_(x: number, y: number, arr: ReadonlyArray<number> | null): Float32Array | null {
|
||||
const iml = this.iml;
|
||||
if (!iml) return null;
|
||||
|
||||
|
|
@ -176,17 +212,25 @@ export class Spine implements EngineSink {
|
|||
const dt = this.lastTickMs > 0 ? (now - this.lastTickMs) / 1000 : 1 / 60;
|
||||
this.lastTickMs = now;
|
||||
|
||||
// 1. processed (pure input pipeline)
|
||||
// 1. processed (pure input pipeline) — primary controller, axes 0,1.
|
||||
this.rawInput[0] = x;
|
||||
this.rawInput[1] = y;
|
||||
this.lastRawX = x;
|
||||
this.lastRawY = y;
|
||||
const proc = processInput(this.rawInput, this.inputConfig, this.inputState, dt);
|
||||
this.inputState = proc.state;
|
||||
this.inputs[0] = proc.x;
|
||||
this.inputs[1] = proc.y;
|
||||
// Secondary axes (2+): pass through with a [0,1] clamp. Omitted axes keep
|
||||
// their prior value (neutral 0.5 in single-controller mode).
|
||||
if (arr) {
|
||||
for (let i = 2; i < this.inputs.length; i++) {
|
||||
if (i < arr.length) this.inputs[i] = clamp(arr[i]!, 0, 1);
|
||||
}
|
||||
}
|
||||
|
||||
// 2. ml (inference into the reused buffer; no alloc)
|
||||
iml.setInput(0, proc.x);
|
||||
iml.setInput(1, proc.y);
|
||||
for (let i = 0; i < this.inputs.length; i++) iml.setInput(i, this.inputs[i]!);
|
||||
iml.processInto(this.mlBuf);
|
||||
// Mirror to liveOutputs for imperative reads + bump.
|
||||
this.liveOutputs.set(this.mlBuf.subarray(0, this.liveOutputs.length));
|
||||
|
|
|
|||
|
|
@ -13,24 +13,19 @@
|
|||
* its value), but routing everything through one compose path keeps sources
|
||||
* composable and the channel layout coherent.
|
||||
*
|
||||
* ── Arity mismatch (the WASM reshape TODO) ──────────────────────────────────
|
||||
* The browser WASM is fixed at MLP<2, …, 126> — a TWO-input head. When the
|
||||
* composed vector has > 2 axes (double-stick gamepad = 4, MIDI learn-map = many)
|
||||
* we must reduce to 2 to feed today's engine. We do NOT fake a wider net.
|
||||
* ── Arity: 4-input head ─────────────────────────────────────────────────────
|
||||
* The browser WASM head is MLP<4, …, 126> — TWO 2-D controllers. The primary
|
||||
* controller (on-screen pad) occupies axes 0,1; an optional second controller
|
||||
* (gamepad) occupies axes 2,3. `compose()` passes axes through 1:1 when the
|
||||
* head is at least as wide as the composed axis count (the common cases —
|
||||
* single pad = 2, pad + single-stick gamepad = 4), padding any unused axes with
|
||||
* the neutral 0.5. Single-controller mode is therefore just "no second source":
|
||||
* axes 2,3 stay neutral, slicing the 4-D manifold at the centre of controller 2.
|
||||
*
|
||||
* chosen reduction (this pass): pairwise BLEND.
|
||||
* inX = mean(axis[0], axis[2], axis[4], …) // even axes
|
||||
* inY = mean(axis[1], axis[3], axis[5], …) // odd axes
|
||||
* so a single stick passes straight through (axis0→X, axis1→Y), a double
|
||||
* stick averages L/R into one XY, and MIDI axes fold into X/Y by parity.
|
||||
*
|
||||
* TODO(workstream F, docs/redesign/inputs-spec.md — "multiple WASM modules +
|
||||
* warm-start"): the real fix is to (re)load a WASM module whose MLP input arity
|
||||
* matches the composed axis count and warm-start its weights from the prior net,
|
||||
* so every axis gets its own genuine input dimension instead of being blended.
|
||||
* That is a larger build (multiple .wasm artefacts or a runtime-variadic head)
|
||||
* and is deliberately deferred — this layer is wired so that swapping the
|
||||
* reduction for a true reshape is a localised change in `compose()`.
|
||||
* Over-subscription (composed axes > head width — e.g. pad + dual-stick gamepad
|
||||
* = 6, or a large MIDI learn-map) still falls back to a chunked-mean blend; the
|
||||
* net is never rebuilt. That blend is the documented lossy path for unusual
|
||||
* source combinations, not the default.
|
||||
*/
|
||||
import type { InputAction, InputSource } from './types';
|
||||
|
||||
|
|
@ -147,15 +142,15 @@ export class InputLayer {
|
|||
}
|
||||
|
||||
/**
|
||||
* Reduce the composed N-axis vector to the engine's input arity.
|
||||
* Map the composed N-axis vector to the engine's input arity.
|
||||
*
|
||||
* For the fixed 2-input WASM, fold by parity (even→X, odd→Y) via mean. If a
|
||||
* future multi-module engine reports inputSize >= n, this passes axes through
|
||||
* 1:1 (truncated/padded) — the seam where the real reshape lands.
|
||||
* Common case: the 4-input head is at least as wide as the composed axes, so
|
||||
* pass through 1:1 and pad unused axes with the neutral 0.5. Over-subscription
|
||||
* (axes > head width) folds by parity / chunked mean — the lossy fallback.
|
||||
*/
|
||||
private compose(n: number, inputSize: number): number[] {
|
||||
if (inputSize >= n) {
|
||||
// True passthrough path (future multi-module head). Pad with 0.5.
|
||||
// Passthrough: each axis gets its own input dimension. Pad with 0.5.
|
||||
const out = new Array<number>(inputSize);
|
||||
for (let i = 0; i < inputSize; i++) out[i] = i < n ? this.vector[i] : 0.5;
|
||||
return out;
|
||||
|
|
|
|||
|
|
@ -141,7 +141,7 @@ class ModeBase {
|
|||
// Run an inference at default inputs so engine has params on first
|
||||
// process() call, even if no input has been touched.
|
||||
for (std::size_t i = 0u; i < NInputs; ++i) {
|
||||
ml_.set_input(i, input_channels_[i]);
|
||||
ml_.set_input(i, effective_input(i));
|
||||
}
|
||||
ml_.process();
|
||||
if constexpr (kRouteOutputsToEngine) {
|
||||
|
|
@ -160,13 +160,42 @@ class ModeBase {
|
|||
input_dirty_ = true;
|
||||
}
|
||||
|
||||
// ---- Input neutralization (single/double controller toggle) ----
|
||||
//
|
||||
// A pinned channel feeds `pin_value_` (neutral, default 0.5) to the MLP
|
||||
// instead of its live value, without rebuilding/resizing the network.
|
||||
// Glue toggles which channels are pinned (e.g. single-joystick mode pins
|
||||
// the second 2D controller's two channels). The stored live value is left
|
||||
// untouched, so unpinning resumes from the controller's current position.
|
||||
NISPS_FORCE_INLINE void set_input_pinned(std::size_t idx, bool pinned) noexcept {
|
||||
if (idx >= NInputs) return;
|
||||
input_pinned_[idx] = pinned;
|
||||
input_dirty_ = true;
|
||||
}
|
||||
NISPS_FORCE_INLINE bool is_input_pinned(std::size_t idx) const noexcept {
|
||||
return idx < NInputs && input_pinned_[idx];
|
||||
}
|
||||
NISPS_FORCE_INLINE void set_pin_value(float v) noexcept {
|
||||
if (v < 0.f) v = 0.f;
|
||||
else if (v > 1.f) v = 1.f;
|
||||
pin_value_ = v;
|
||||
input_dirty_ = true;
|
||||
}
|
||||
float pin_value() const noexcept { return pin_value_; }
|
||||
|
||||
// Effective value fed to the MLP for channel i (pin override applied).
|
||||
NISPS_FORCE_INLINE float effective_input(std::size_t i) const noexcept {
|
||||
return input_pinned_[i] ? pin_value_ : input_channels_[i];
|
||||
}
|
||||
|
||||
NISPS_HOT void tick_control() noexcept {
|
||||
if constexpr (requires(Derived& d) { d.on_pre_inference(); }) {
|
||||
static_cast<Derived&>(*this).on_pre_inference();
|
||||
}
|
||||
// Forward (possibly Derived-mutated) channels into the MLP.
|
||||
// Forward (possibly Derived-mutated) channels into the MLP, applying
|
||||
// the per-channel pin override.
|
||||
for (std::size_t i = 0u; i < NInputs; ++i) {
|
||||
ml_.set_input(i, input_channels_[i]);
|
||||
ml_.set_input(i, effective_input(i));
|
||||
}
|
||||
ml_.process();
|
||||
if constexpr (kRouteOutputsToEngine) {
|
||||
|
|
@ -235,6 +264,8 @@ class ModeBase {
|
|||
EngineT engine_{};
|
||||
MLPType ml_;
|
||||
std::array<float, NInputs> input_channels_{};
|
||||
std::array<bool, NInputs> input_pinned_{}; // false => live
|
||||
float pin_value_ = 0.5f; // neutral
|
||||
bool input_dirty_ = false;
|
||||
std::size_t voice_space_idx_ = 0u;
|
||||
RingBuffer<ControlEvent, kModeEventBufferSize> events_{};
|
||||
|
|
|
|||
|
|
@ -32,10 +32,13 @@ The MLP class template is parametrised on `(input_size, hidden1, hidden2,
|
|||
hidden3, output_size)`. WASM cannot recompile templates at runtime, so
|
||||
this build instantiates exactly ONE configuration:
|
||||
|
||||
nisps::ml::MLP<2, 10, 14, 18, 126>
|
||||
nisps::ml::MLP<4, 10, 14, 18, 126>
|
||||
|
||||
That serves the playground use case (2-D joystick → up to 126 synth
|
||||
parameters). `nisps_ml_create()` accepts caller-supplied dimensions for
|
||||
That serves the manifold/playground use case: two 2-D controllers (primary
|
||||
pad on axes 0,1; optional second controller — e.g. gamepad — on axes 2,3)
|
||||
→ up to 126 synth parameters. Single-controller mode pins axes 2,3 to the
|
||||
neutral 0.5 on the JS side, so the net is never rebuilt when toggling
|
||||
between one and two controllers. `nisps_ml_create()` accepts caller-supplied dimensions for
|
||||
forward compatibility but currently ignores them — see comment at the top
|
||||
of `bindings.cpp`. The schemas in `schemas/modes/*.json` use up to
|
||||
`output_size=126`; modes whose output_size is < 126 simply ignore the
|
||||
|
|
|
|||
|
|
@ -5,24 +5,26 @@
|
|||
// 2. AudioWorklet processor (playground/src/audio/worklet/...) — engine
|
||||
// calls. (Each instance owns its own WASM module instance.)
|
||||
//
|
||||
// FIXED-ARCHITECTURE LIMITATION (VERY IMPORTANT)
|
||||
// ----------------------------------------------
|
||||
// COMPILE-TIME ARCHITECTURE
|
||||
// -------------------------
|
||||
// The C++ MLP class is templated on layer sizes (architecture.md §4.1, §6.2).
|
||||
// We instantiate ONE concrete configuration here:
|
||||
//
|
||||
// using DefaultMLP = nisps::ml::MLP<2, 10, 14, 18, 126>;
|
||||
// using DefaultMLP = nisps::ml::MLP<4, 10, 14, 18, 126>;
|
||||
//
|
||||
// This was chosen as the union of the playground use case (2-D joystick →
|
||||
// 126 synth params) and the largest hidden-layer footprint that still fits
|
||||
// firmware budgets. `nisps_ml_create()` accepts caller-supplied input_size,
|
||||
// output_size, hidden[], n_hidden but ONLY validates them against the
|
||||
// compile-time defaults — extra inputs/outputs are clipped at the boundary.
|
||||
// If the caller passes incompatible dimensions we still create the module:
|
||||
// extra inputs are zero-padded, extra outputs are truncated, and the
|
||||
// hidden-layer override is silently ignored.
|
||||
// FOUR inputs = two 2-D controllers (primary on-screen pad on axes 0,1; an
|
||||
// optional second controller — gamepad — on axes 2,3). Single-controller mode
|
||||
// pins the second pair to the neutral value 0.5 on the JS side (see
|
||||
// manifold/src/engine/spine.ts), so the network is never rebuilt/resized when
|
||||
// toggling between one and two controllers. 126 outputs covers C15 and every
|
||||
// current schema.
|
||||
//
|
||||
// Future work: ship multiple WASM modules (one per common architecture) or
|
||||
// rebuild on demand. See architecture.md "open questions" — Stream 7 punts.
|
||||
// `nisps_ml_create()` accepts caller-supplied input_size, output_size,
|
||||
// hidden[], n_hidden but ONLY validates them against the compile-time
|
||||
// defaults — extra inputs/outputs are clipped at the boundary. If the caller
|
||||
// passes incompatible dimensions we still create the module: extra inputs are
|
||||
// zero-padded, extra outputs are truncated, and the hidden-layer override is
|
||||
// silently ignored. The TS side reads the real dims back via nisps_ml_describe.
|
||||
//
|
||||
// WIRE FORMAT FOR WEIGHTS
|
||||
// -----------------------
|
||||
|
|
@ -78,14 +80,15 @@ namespace {
|
|||
// Compile-time default architecture. See header comment.
|
||||
//
|
||||
// Choice rationale:
|
||||
// * 2 inputs — playground virtual joystick (X, Y).
|
||||
// * 4 inputs — two 2-D controllers (primary pad axes 0,1; optional
|
||||
// second controller axes 2,3). Single-controller mode pins axes 2,3 to 0.5.
|
||||
// * [10, 14, 18] hidden — covers the largest schema layouts in
|
||||
// `schemas/modes/*.json` (channel_strip variants, verb_fx, breakor,
|
||||
// elysiamorf, memlcelium).
|
||||
// * 126 outputs — enough for the C15 mode and any current schema.
|
||||
//
|
||||
// The MLP also has dataset slots, loss history etc. — see mlp.hpp.
|
||||
using DefaultMLP = nisps::ml::MLP<2u, 10u, 14u, 18u, 126u>;
|
||||
using DefaultMLP = nisps::ml::MLP<4u, 10u, 14u, 18u, 126u>;
|
||||
|
||||
constexpr std::size_t kDefaultInputs = DefaultMLP::kInput;
|
||||
constexpr std::size_t kDefaultOutputs = DefaultMLP::kOutput;
|
||||
|
|
|
|||
Binary file not shown.
|
|
@ -82,3 +82,12 @@ set -x
|
|||
|
||||
echo "[build-wasm] wrote $OUT/nisps.js + $OUT/nisps.wasm"
|
||||
ls -lh "$OUT/nisps.js" "$OUT/nisps.wasm"
|
||||
|
||||
# Mirror into the manifold app's public/ — it serves its own copy (and its
|
||||
# deploy webhook builds manifold/), so both must stay in lockstep with the
|
||||
# parity-checked artifacts above.
|
||||
MANIFOLD_OUT="$ROOT/manifold/public"
|
||||
if [[ -d "$MANIFOLD_OUT" ]]; then
|
||||
cp "$OUT/nisps.js" "$OUT/nisps.wasm" "$MANIFOLD_OUT/"
|
||||
echo "[build-wasm] mirrored nisps.{js,wasm} → $MANIFOLD_OUT"
|
||||
fi
|
||||
|
|
|
|||
|
|
@ -13,7 +13,7 @@
|
|||
//
|
||||
// What we cover
|
||||
// -------------
|
||||
// 1. ML: seed=42, draw_weights(0.5), set_input(0.25, 0.75), process.
|
||||
// 1. ML: seed=42, draw_weights(0.5), set_input(0.25, 0.75, 0.40, 0.60), process.
|
||||
// → 126 outputs + 12 weights sampled at known offsets.
|
||||
// 2. ML training: 3 examples added, train(0.3, 50, 0), capture loss + outputs.
|
||||
// 3. PAFSynth engine: seed-equivalent setup (params=0.5), 128-sample run on
|
||||
|
|
@ -21,7 +21,7 @@
|
|||
// 4. ChannelStrip engine: identical methodology.
|
||||
//
|
||||
// We use the EXACT SAME compile-time MLP architecture as the WASM build:
|
||||
// MLP<2, 10, 14, 18, 126>
|
||||
// MLP<4, 10, 14, 18, 126>
|
||||
//
|
||||
// Output blob format
|
||||
// ------------------
|
||||
|
|
@ -62,7 +62,7 @@
|
|||
|
||||
namespace {
|
||||
|
||||
using ParityMLP = nisps::ml::MLP<2u, 10u, 14u, 18u, 126u>;
|
||||
using ParityMLP = nisps::ml::MLP<4u, 10u, 14u, 18u, 126u>;
|
||||
|
||||
// The WASM bindings (nisps/wasm/bindings.cpp) sign-extend the 32-bit JS
|
||||
// seed via `s ^ (s << 32)`. To get bit-equal output between native and
|
||||
|
|
@ -73,6 +73,9 @@ constexpr std::uint64_t kSeed = static_cast<std::uint64_t>(kSeed32)
|
|||
^ (static_cast<std::uint64_t>(kSeed32) << 32);
|
||||
constexpr float kInputX = 0.25f;
|
||||
constexpr float kInputY = 0.75f;
|
||||
// Second controller (axes 2,3) — exercised so the 4-input head is fully probed.
|
||||
constexpr float kInputZ = 0.40f;
|
||||
constexpr float kInputW = 0.60f;
|
||||
constexpr float kSampleRate = 48000.0f;
|
||||
constexpr std::size_t kSynthFrames = 128u;
|
||||
|
||||
|
|
@ -118,6 +121,8 @@ int main(int argc, char** argv) {
|
|||
mlp.draw_weights(0.5f);
|
||||
mlp.set_input(0u, kInputX);
|
||||
mlp.set_input(1u, kInputY);
|
||||
mlp.set_input(2u, kInputZ);
|
||||
mlp.set_input(3u, kInputW);
|
||||
mlp.process();
|
||||
{
|
||||
const auto outs = mlp.outputs();
|
||||
|
|
@ -133,8 +138,8 @@ int main(int argc, char** argv) {
|
|||
}
|
||||
|
||||
// ---- Stage 2: training ----
|
||||
constexpr std::array<std::array<float, 2u>, 3u> features = {{
|
||||
{{0.1f, 0.9f}}, {{0.5f, 0.5f}}, {{0.9f, 0.1f}},
|
||||
constexpr std::array<std::array<float, 4u>, 3u> features = {{
|
||||
{{0.1f, 0.9f, 0.3f, 0.7f}}, {{0.5f, 0.5f, 0.5f, 0.5f}}, {{0.9f, 0.1f, 0.7f, 0.3f}},
|
||||
}};
|
||||
auto label_for = [](std::size_t i) {
|
||||
std::array<float, 126u> out{};
|
||||
|
|
@ -146,13 +151,15 @@ int main(int argc, char** argv) {
|
|||
};
|
||||
for (std::size_t i = 0; i < features.size(); ++i) {
|
||||
const auto label = label_for(i);
|
||||
mlp.add_example(std::span<const float>(features[i].data(), 2u),
|
||||
mlp.add_example(std::span<const float>(features[i].data(), 4u),
|
||||
std::span<const float>(label.data(), 126u));
|
||||
}
|
||||
const float final_loss = mlp.train(0.3f, 50u, 0.0f);
|
||||
|
||||
mlp.set_input(0u, kInputX);
|
||||
mlp.set_input(1u, kInputY);
|
||||
mlp.set_input(2u, kInputZ);
|
||||
mlp.set_input(3u, kInputW);
|
||||
mlp.process();
|
||||
{
|
||||
const auto outs = mlp.outputs();
|
||||
|
|
|
|||
|
|
@ -43,6 +43,8 @@ const VERSION = 3; // v3 adds stage 5d (ExploreAndPlace lifecycle)
|
|||
const SEED = 42 >>> 0;
|
||||
const INPUT_X = 0.25;
|
||||
const INPUT_Y = 0.75;
|
||||
const INPUT_Z = 0.40; // second controller axes 2,3 (must match parity_check.cpp)
|
||||
const INPUT_W = 0.60;
|
||||
const SAMPLE_RATE = 48000;
|
||||
const SYNTH_FRAMES = 128;
|
||||
const PROBE_IDX = [0, 5, 19, 31, 73, 137, 251, 491, 999, 1583, 2401, 3289];
|
||||
|
|
@ -191,8 +193,8 @@ async function main() {
|
|||
api.describe(dimsBuf);
|
||||
const dims = new Int32Array(Module.HEAP32.buffer, dimsBuf, 6).slice();
|
||||
api.free(dimsBuf);
|
||||
// Expect: [2, 10, 14, 18, 126, 4]
|
||||
const expectedDims = [2, 10, 14, 18, 126, 4];
|
||||
// Expect: [inputs, h1, h2, h3, outputs, numLayers] = [4, 10, 14, 18, 126, 4]
|
||||
const expectedDims = [4, 10, 14, 18, 126, 4];
|
||||
for (let i = 0; i < expectedDims.length; ++i) {
|
||||
if (dims[i] !== expectedDims[i]) {
|
||||
console.error(`[parity_wasm] WASM build has dim[${i}]=${dims[i]}, native expected ${expectedDims[i]}`);
|
||||
|
|
@ -203,10 +205,12 @@ async function main() {
|
|||
const N_OUT = dims[4];
|
||||
|
||||
// --- Stage 1: ML inference ---
|
||||
const ml = api.create(2, N_OUT, 0, 0, SEED);
|
||||
const ml = api.create(4, N_OUT, 0, 0, SEED);
|
||||
api.drawWeights(ml, 0.5);
|
||||
api.setInput(ml, 0, INPUT_X);
|
||||
api.setInput(ml, 1, INPUT_Y);
|
||||
api.setInput(ml, 2, INPUT_Z);
|
||||
api.setInput(ml, 3, INPUT_W);
|
||||
api.process(ml);
|
||||
const outsStage1 = getOutputsCopy(api, ml, N_OUT);
|
||||
|
||||
|
|
@ -216,9 +220,9 @@ async function main() {
|
|||
|
||||
// --- Stage 2: training ---
|
||||
const features = [
|
||||
[0.1, 0.9],
|
||||
[0.5, 0.5],
|
||||
[0.9, 0.1],
|
||||
[0.1, 0.9, 0.3, 0.7],
|
||||
[0.5, 0.5, 0.5, 0.5],
|
||||
[0.9, 0.1, 0.7, 0.3],
|
||||
];
|
||||
const labelFor = (i) => {
|
||||
const out = new Float32Array(N_OUT);
|
||||
|
|
@ -226,12 +230,14 @@ async function main() {
|
|||
for (let j = 0; j < N_OUT; ++j) out[j] = a + 0.005 * j;
|
||||
return out;
|
||||
};
|
||||
const featBuf = api.malloc(2 * 4);
|
||||
const featF32 = new Float32Array(api.HEAPF32.buffer, featBuf, 2);
|
||||
const featBuf = api.malloc(4 * 4);
|
||||
const featF32 = new Float32Array(api.HEAPF32.buffer, featBuf, 4);
|
||||
const labelBuf = api.malloc(N_OUT * 4);
|
||||
for (let i = 0; i < features.length; ++i) {
|
||||
featF32[0] = features[i][0];
|
||||
featF32[1] = features[i][1];
|
||||
featF32[2] = features[i][2];
|
||||
featF32[3] = features[i][3];
|
||||
const label = labelFor(i);
|
||||
new Float32Array(api.HEAPF32.buffer, labelBuf, N_OUT).set(label);
|
||||
api.addExample(ml, featBuf, labelBuf);
|
||||
|
|
@ -243,6 +249,8 @@ async function main() {
|
|||
|
||||
api.setInput(ml, 0, INPUT_X);
|
||||
api.setInput(ml, 1, INPUT_Y);
|
||||
api.setInput(ml, 2, INPUT_Z);
|
||||
api.setInput(ml, 3, INPUT_W);
|
||||
api.process(ml);
|
||||
const outsStage2 = getOutputsCopy(api, ml, N_OUT);
|
||||
|
||||
|
|
@ -301,6 +309,8 @@ async function main() {
|
|||
api.feedbackUndo(ml); // pop nudge
|
||||
api.setInput(ml, 0, INPUT_X);
|
||||
api.setInput(ml, 1, INPUT_Y);
|
||||
api.setInput(ml, 2, INPUT_Z);
|
||||
api.setInput(ml, 3, INPUT_W);
|
||||
api.process(ml);
|
||||
api.feedbackLike(ml); // begin place: freeze scratchpad output
|
||||
{
|
||||
|
|
|
|||
|
|
@ -88,6 +88,62 @@ NISPS_TEST(paf_synth_mode_input_clamping) {
|
|||
m.set_input(99u, 0.5f);
|
||||
}
|
||||
|
||||
NISPS_TEST(paf_synth_mode_pinned_inputs_neutralized) {
|
||||
// Single-joystick mode pins the second 2D controller's channels (2,3).
|
||||
// A pinned channel must feed the neutral pin value (default 0.5) to the
|
||||
// MLP regardless of its live value, and pinning must not resize the model.
|
||||
PAFSynthMode m;
|
||||
m.setup(48000.f);
|
||||
NISPS_EXPECT(m.pin_value() == 0.5f);
|
||||
|
||||
// Reference: channels 2,3 driven live to 0.5 (the neutral value).
|
||||
m.set_input(0, 0.2f);
|
||||
m.set_input(1, 0.8f);
|
||||
m.set_input(2, 0.5f);
|
||||
m.set_input(3, 0.5f);
|
||||
m.tick_control();
|
||||
std::array<float, 33> ref{};
|
||||
{
|
||||
const auto outs = m.ml().outputs();
|
||||
for (std::size_t i = 0u; i < outs.size(); ++i) ref[i] = outs[i];
|
||||
}
|
||||
|
||||
// Pin channels 2,3, then drive them to arbitrary values. Output must match
|
||||
// the reference (they are neutralized to 0.5).
|
||||
m.set_input_pinned(2, true);
|
||||
m.set_input_pinned(3, true);
|
||||
NISPS_EXPECT(m.is_input_pinned(2));
|
||||
m.set_input(2, 0.0f);
|
||||
m.set_input(3, 1.0f);
|
||||
m.tick_control();
|
||||
for (std::size_t i = 0u; i < 33u; ++i) {
|
||||
NISPS_EXPECT(std::fabs(m.ml().outputs()[i] - ref[i]) < 1e-6f);
|
||||
}
|
||||
|
||||
// The active controller (channels 0,1) still affects the output.
|
||||
m.set_input(0, 0.9f);
|
||||
m.tick_control();
|
||||
bool changed = false;
|
||||
for (std::size_t i = 0u; i < 33u; ++i) {
|
||||
if (std::fabs(m.ml().outputs()[i] - ref[i]) > 1e-6f) { changed = true; break; }
|
||||
}
|
||||
NISPS_EXPECT(changed);
|
||||
|
||||
// Unpinning restores sensitivity on channels 2,3.
|
||||
m.set_input_pinned(2, false);
|
||||
m.set_input_pinned(3, false);
|
||||
m.set_input(0, 0.2f); // restore active channels to the reference pose
|
||||
m.set_input(1, 0.8f);
|
||||
m.set_input(2, 0.0f);
|
||||
m.set_input(3, 1.0f);
|
||||
m.tick_control();
|
||||
bool differs = false;
|
||||
for (std::size_t i = 0u; i < 33u; ++i) {
|
||||
if (std::fabs(m.ml().outputs()[i] - ref[i]) > 1e-6f) { differs = true; break; }
|
||||
}
|
||||
NISPS_EXPECT(differs);
|
||||
}
|
||||
|
||||
NISPS_TEST(paf_synth_mode_engine_and_ml_accessors) {
|
||||
PAFSynthMode m;
|
||||
m.setup(48000.f);
|
||||
|
|
|
|||
Loading…
Reference in a new issue