Compare commits

...

4 commits

Author SHA1 Message Date
monkey-w1n5t0n
3e6757dea8 feat(firmware): on-device Joystick Dual/Single settings menu
Add glue/settings_view.hpp: wire_settings(mode) registers a "Joystick"
SingleSelectView on the MEMLNaut display carousel (TFT + rotary encoder, same
DisplayDriver as SystemView/SelfTest). For the 4-input two-joystick modes,
"Single" pins ML input channels 2,3 (the second joystick) to neutral via
ModeBase::set_input_pinned — the network is never rebuilt and trained state
survives toggling. Default is Dual. Only registered when input arity == 4.

Called from MEMLNaut-NISPS.ino setup() after addSystemInfoView(). MAP.md +
CLAUDE.md glue listings updated.

NOTE: compile-unverified — no arduino-cli/RP2350 toolchain on this host and no
hardware; needs scripts/build-firmware.sh + a flash test (chokepoint A).
2026-06-28 21:02:23 +02:00
monkey-w1n5t0n
9e4c751781 feat(manifold): parameter-sandwich centre-stage toggle
Fuse the /learn parameter-landscape sandbox into the main console:
- New SandwichStage: ports the 3D landscape renderer to React, driven by the
  REAL EngineApi (inferBatch samples each output channel over the 2D input grid;
  getOutputs feeds the live probe). Re-samples on engine version bumps; orbit
  camera (top-down = flat 2D, tilt to reveal the stack), centred + grows to fill.
- Dock: new bottom toggle (SandwichIcon) → ctx.sandwich.
- ConsoleApp: when on, the sandwich replaces the centre stage in a 3-zone flex —
  shrunken Manifold (left) · sandwich (centre, fills) · compact OutputStage
  (right) — reusing pos/markers/verdicts. Opening it auto-closes the drawer so
  the output zone is visible. Layers = the mode's params (capped 8).

typecheck + build green; verified headless (toggle on/off, sample, tilt).
2026-06-28 20:50:54 +02:00
monkey-w1n5t0n
ebb953a13f feat(inputs): single/double controller toggle (shared core + manifold)
Toggle between one and two 2-D controllers without rebuilding the MLP.

Shared core (nisps/modes/base.hpp):
- ModeBase per-channel pin mask + pin_value (default 0.5). Pinned channels feed
  the neutral value to the MLP via effective_input(); tick_control/setup use it.
  Single-controller mode pins the second controller's two channels — the net is
  never resized, so trained state survives toggling.

Manifold/browser:
- Widen the WASM head MLP<2,…> -> MLP<4,10,14,18,126> (4 inputs = two 2-D
  controllers). All bindings are dimension-driven via DefaultMLP::kInput.
- Spine holds a full N-axis input vector: axes 0,1 = primary pad (existing input
  pipeline), axes 2+ = secondary controller, unused axes default to neutral 0.5.
  engine-api.setInputs feeds the whole vector; input-layer passthrough pads 0.5
  (the lossy 4->2 blend is now only a fallback for >4 axes).
- Inputs dock: Controllers Single/Double(+gamepad) toggle, default single.
- build-wasm.sh mirrors nisps.{js,wasm} into manifold/public.

Tests/docs:
- test_mode_paf_synth: pinned-inputs-neutralized covers the pin contract.
- parity_check.cpp + parity_wasm.mjs exercise all 4 inputs; native↔WASM parity
  PASS (max delta 2.4e-7).
- CLAUDE.md / MAP.md / nisps/wasm/README.md updated (no longer "fixed MLP<2,…>").

Firmware settings-menu toggle (Phase 3) still pending.
2026-06-28 20:47:15 +02:00
monkey-w1n5t0n
e8f0866b47 feat(learn): engine-driven 3D parameter-landscape sandbox
Interactive /learn page rebuilt on the real NISPS engine (nisps.wasm):
- Stacked 3D landscapes = MLP output channels sampled over the 2D input grid
  (the N-dimensional 'sandwich'); orbitable yaw/pitch camera, top-down = flat 2D.
- Three zones L→R: XY control (with +/- feedback markers) → MLP stack → output faders.
- Real training: push-away (feedback_down) + explore-and-place (enter/reroll/nudge/
  commit), undo, randomise, nudge; landscapes re-sample and morph after each verdict.
- Params numberbox (1..8); new layers grow downward with the top slab pinned.
- Minimal-by-default with a discrete info switch revealing placeholder copy.

Standalone no-build page; loads ../nisps.{js,wasm}; deployed at /next/learn/.
2026-06-28 20:38:33 +02:00
22 changed files with 1109 additions and 79 deletions

View file

@ -58,7 +58,8 @@ firmware/MEMLNaut-NISPS/
│ ├── midi_io.hpp # MIDI in → mode handlers; drain ControlEvent ring → MIDI UART │ ├── midi_io.hpp # MIDI in → mode handlers; drain ControlEvent ring → MIDI UART
│ ├── mode_select.hpp # type aliases firmware mode name → nisps::modes::*Mode │ ├── mode_select.hpp # type aliases firmware mode name → nisps::modes::*Mode
│ ├── input_router.hpp # wire_inputs() entry point │ ├── input_router.hpp # wire_inputs() entry point
│ └── output_router.hpp # drain_outputs() entry point │ ├── output_router.hpp # drain_outputs() entry point
│ └── settings_view.hpp # wire_settings(): TFT/rotary menu (Joystick Dual/Single for 4-in modes)
└── src/{memllib,daisysp,nisps} # symlinks (Arduino-CLI sketch tree convention) └── src/{memllib,daisysp,nisps} # symlinks (Arduino-CLI sketch tree convention)
``` ```
@ -118,9 +119,9 @@ Two WASM instances at runtime:
1. **Main thread** (`playground/src/ml/wasm-iml.ts`): ML inference + sync training + RL primitives. Update store after each call. Async training via disposable Web Worker (`wasm-worker.ts`). 1. **Main thread** (`playground/src/ml/wasm-iml.ts`): ML inference + sync training + RL primitives. Update store after each call. Async training via disposable Web Worker (`wasm-worker.ts`).
2. **AudioWorklet** (`playground/src/audio/worklet/nisps-processor.ts`): runs engine `process_block` per audio block. Loads `nisps.wasm` directly via `WebAssembly.compile` (no Emscripten glue in worklet). Bytes posted from main thread. 2. **AudioWorklet** (`playground/src/audio/worklet/nisps-processor.ts`): runs engine `process_block` per audio block. Loads `nisps.wasm` directly via `WebAssembly.compile` (no Emscripten glue in worklet). Bytes posted from main thread.
C API is in `nisps/wasm/bindings.cpp`. Build: `bash scripts/build-wasm.sh` (~94KB output to `playground/public/`). C API is in `nisps/wasm/bindings.cpp`. Build: `bash scripts/build-wasm.sh` (~100KB output, mirrored to `playground/public/` and `manifold/public/`).
The WASM target is fixed at `MLP<2, 10, 14, 18, 126>`. Modes with smaller `output_size` use the first N outputs only. The WASM target is built as `MLP<4, 10, 14, 18, 126>` — 4 inputs = two 2-D controllers (primary pad on axes 0,1; optional second controller, e.g. gamepad, on axes 2,3). Single-controller mode pins axes 2,3 to the neutral 0.5 on the JS side (no rebuild). Modes with smaller `output_size` use the first N outputs only.
### Known limitations ### Known limitations

3
MAP.md
View file

@ -23,6 +23,7 @@ MEMLNaut-NISPS — Neural Interactive Shaping of Parameter Spaces. One C++20 cod
- `mode_select.hpp` — type aliases mapping firmware mode identifiers to `nisps::modes::*Mode` C++ types. Build script rewrites the active line. Includes the six `MEMLNautModeExtSynth*` external-synth variants (one per device template in `nisps/midi`, e.g. `MEMLNautModeExtSynthSub37`). Also defines the `MEMLNautModeSelfTest` pseudo-variant (tag type) + the `NISPS_ST_*`/`NISPS_ST_CAT` token-paste macros the `.ino` uses to compute `NISPS_SELFTEST`. Note: `src/nisps/` exposes each referenced top-level nisps subdir as a symlink — `midi` was added alongside `core/dsp/engines/ml/modes`. - `mode_select.hpp` — type aliases mapping firmware mode identifiers to `nisps::modes::*Mode` C++ types. Build script rewrites the active line. Includes the six `MEMLNautModeExtSynth*` external-synth variants (one per device template in `nisps/midi`, e.g. `MEMLNautModeExtSynthSub37`). Also defines the `MEMLNautModeSelfTest` pseudo-variant (tag type) + the `NISPS_ST_*`/`NISPS_ST_CAT` token-paste macros the `.ino` uses to compute `NISPS_SELFTEST`. Note: `src/nisps/` exposes each referenced top-level nisps subdir as a symlink — `midi` was added alongside `core/dsp/engines/ml/modes`.
- `selftest.hpp` — standalone guided hardware self-test rig (`SelfTest` variant; no engine/ML). Step-driven state machine on a `SelfTestView`: TFT prompts the operator through every control, auto-advances on detection, encoder-press skips. Ends with optional L/R/BOTH sine-sweep headphone check (core 1 block callback) + MIDI loopback-cable test. Lives firmware-side (touches TFT + raw pins) so it stays out of platform-agnostic `nisps/`. - `selftest.hpp` — standalone guided hardware self-test rig (`SelfTest` variant; no engine/ML). Step-driven state machine on a `SelfTestView`: TFT prompts the operator through every control, auto-advances on detection, encoder-press skips. Ends with optional L/R/BOTH sine-sweep headphone check (core 1 block callback) + MIDI loopback-cable test. Lives firmware-side (touches TFT + raw pins) so it stays out of platform-agnostic `nisps/`.
- `input_router.hpp`, `output_router.hpp` — top-level `wire_inputs()` / `drain_outputs()` entry points. - `input_router.hpp`, `output_router.hpp` — top-level `wire_inputs()` / `drain_outputs()` entry points.
- `settings_view.hpp` — `wire_settings(mode)`: adds on-device settings views to the MEMLNaut display carousel (TFT + rotary encoder). Joystick Dual/Single toggle for the 4-input ("two 2-D joystick") modes — "Single" pins ML input channels 2,3 to neutral via `ModeBase::set_input_pinned` (no net rebuild). Registered in the `.ino` after `addSystemInfoView()`.
- `firmware/MEMLNaut-NISPS/src/{memllib,daisysp,nisps}` — symlinks (Arduino-CLI requires sketch-tree includes; preprocessor refuses `..` in headers). - `firmware/MEMLNaut-NISPS/src/{memllib,daisysp,nisps}` — symlinks (Arduino-CLI requires sketch-tree includes; preprocessor refuses `..` in headers).
- `firmware/README.md` — structure + build instructions. - `firmware/README.md` — structure + build instructions.
@ -138,7 +139,7 @@ the "BUILD DELTAS" block at the top of `vcv/SPEC.md`). `src/MEMLNaut.cpp` (modul
- C++ identifiers: `PascalCase` types, `snake_case` functions/variables, `kPascalCase` constexpr. JSON keys `snake_case`. TS types `PascalCase`, components `PascalCase.tsx`, modules `kebab-case.ts`. - C++ identifiers: `PascalCase` types, `snake_case` functions/variables, `kPascalCase` constexpr. JSON keys `snake_case`. TS types `PascalCase`, components `PascalCase.tsx`, modules `kebab-case.ts`.
- `Curve` enum lives in `nisps/core/math.hpp` (lowercase: `linear/exp/log/square/sqrt/sigmoid/cubic`); generated mode headers re-export via `using Curve = ::nisps::Curve;`. TS mirror at `playground/src/output/curves.ts` with same names. - `Curve` enum lives in `nisps/core/math.hpp` (lowercase: `linear/exp/log/square/sqrt/sigmoid/cubic`); generated mode headers re-export via `using Curve = ::nisps::Curve;`. TS mirror at `playground/src/output/curves.ts` with same names.
- Modes are TSX components composed of primitives; mode parameter contracts are JSON schemas with codegen → C++/TS types. **No declarative JSON UI.** - Modes are TSX components composed of primitives; mode parameter contracts are JSON schemas with codegen → C++/TS types. **No declarative JSON UI.**
- WASM and firmware share the same C++; WASM is fixed at `MLP<2, 10, 14, 18, 126>` and modes use a slice of outputs based on schema's `output_size`. - WASM and firmware share the same C++; WASM is built as `MLP<4, 10, 14, 18, 126>` (4 inputs = two 2-D controllers; single-controller mode pins axes 2,3 to 0.5 in JS) and modes use a slice of outputs based on schema's `output_size`.
- Cross-platform parity: `scripts/parity-check.sh` enforces native vs WASM agreement within 1e-5. - Cross-platform parity: `scripts/parity-check.sh` enforces native vs WASM agreement within 1e-5.
## Gotchas ## Gotchas

View file

@ -0,0 +1,484 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Manifold — the parameter landscape</title>
<style>
:root{
--bg:#0d0d0d; --bg-1:#141414; --bg-2:#1c1c1c; --bg-3:#242424;
--fg:#e8e8e8; --fg-mute:#9a9a9a; --fg-dim:#5a5a5a;
--line:#2a2a2a; --line-strong:#3a3a3a;
--accent:#ff6a00; --accent-2:#00ccff; --good:#6bc26b; --bad:#5aa6c8;
--font-mono:'JetBrains Mono','Fira Code','SF Mono',Menlo,Consolas,monospace;
}
*{box-sizing:border-box}
html,body{margin:0;height:100%;background:var(--bg);color:var(--fg);
font-family:var(--font-mono);font-size:14px;-webkit-font-smoothing:antialiased;overflow:hidden}
#app{position:fixed;inset:0}
#scene{position:absolute;inset:0;width:100%;height:100%;display:block;touch-action:none;cursor:grab}
#scene:active{cursor:grabbing}
/* floating HUD blocks */
.hud{position:absolute;z-index:5}
/* three zones, left → right: input · MLP stack · outputs */
.hud-l{left:24px;top:50%;transform:translateY(-50%);display:flex;flex-direction:column;gap:10px;align-items:flex-start}
.hud-r{right:24px;top:50%;transform:translateY(-50%)}
.hud-bc{left:50%;bottom:22px;transform:translateX(-50%)}
.hud-tr{right:18px;top:18px}
.zonecap{font-size:9px;letter-spacing:.16em;text-transform:uppercase;color:var(--fg-dim);margin-bottom:2px}
/* output fader bank (1-D projection of each layer's value at the point) */
.outs{display:flex;gap:13px;align-items:flex-start;background:rgba(20,20,20,.55);
border:1px solid var(--line);border-radius:12px;padding:12px 14px;backdrop-filter:blur(4px)}
.fader{display:flex;flex-direction:column;align-items:center;gap:6px;width:30px}
.fader .track{position:relative;width:8px;height:200px;background:var(--bg-3);
border:1px solid var(--line);border-radius:999px;overflow:hidden}
.fader .fill{position:absolute;left:0;right:0;bottom:0;border-radius:999px;transition:height .05s linear}
.fader .handle{position:absolute;left:50%;width:16px;height:3px;border-radius:2px;
transform:translate(-50%,1.5px);box-shadow:0 0 8px currentColor}
.fader .flab{font-size:9px;letter-spacing:.03em;text-transform:uppercase;max-width:34px;text-align:center;line-height:1.1}
.fader .fval{font-size:10px;font-variant-numeric:tabular-nums;color:var(--fg)}
/* XY pad */
#ndpad{display:block;border:1px solid var(--line);border-radius:8px;background:var(--bg-2);
cursor:crosshair;touch-action:none}
.padcap{font-size:10px;letter-spacing:.1em;text-transform:uppercase;color:var(--fg-dim)}
/* parameter numberbox */
.kbox{display:flex;align-items:center;gap:8px;background:var(--bg-1);border:1px solid var(--line);
border-radius:8px;padding:5px 8px;width:fit-content}
.kbox .lab{font-size:10px;letter-spacing:.08em;text-transform:uppercase;color:var(--fg-mute)}
.kbox .val{min-width:1.4ch;text-align:center;color:var(--fg);font-size:15px;font-variant-numeric:tabular-nums}
.step{width:24px;height:24px;display:grid;place-items:center;padding:0;font-size:15px;line-height:1;
background:var(--bg-2);border:1px solid var(--line-strong);border-radius:6px;color:var(--fg);cursor:pointer}
.step:hover{background:var(--bg-3)}
.step:disabled{opacity:.3;cursor:default}
/* action bar */
.actionbar{display:flex;align-items:center;gap:8px;background:rgba(20,20,20,.82);
border:1px solid var(--line);border-radius:12px;padding:8px;backdrop-filter:blur(6px)}
.act{display:flex;flex-direction:column;align-items:center;gap:3px;min-width:58px;padding:8px 6px;
background:var(--bg-2);border:1px solid var(--line-strong);border-radius:9px;color:var(--fg);
cursor:pointer;font-family:inherit;transition:border-color .12s,background .12s}
.act:hover{background:var(--bg-3)}
.act .ic{font-size:18px;line-height:1}
.act .tx{font-size:9px;letter-spacing:.04em;color:var(--fg-mute);text-transform:uppercase}
.act.up:hover{border-color:var(--accent);box-shadow:0 0 12px rgba(255,106,0,.25)}
.act.down:hover{border-color:var(--accent-2);box-shadow:0 0 12px rgba(0,204,255,.22)}
.act:disabled{opacity:.35;cursor:default;box-shadow:none}
.sep{width:1px;align-self:stretch;background:var(--line);margin:2px 2px}
/* mode toggle */
.modeseg{display:inline-flex;border:1px solid var(--line-strong);border-radius:9px;overflow:hidden}
.modeseg button{border:0;border-radius:0;padding:0 12px;height:100%;min-height:54px;font-family:inherit;
font-size:11px;letter-spacing:.04em;background:var(--bg-2);color:var(--fg-mute);cursor:pointer}
.modeseg button.on{background:var(--accent);color:#0d0d0d}
/* info toggle */
#infoBtn{width:34px;height:34px;border-radius:50%;border:1px solid var(--line-strong);background:rgba(20,20,20,.82);
color:var(--fg-mute);font-size:15px;cursor:pointer;backdrop-filter:blur(6px)}
#infoBtn:hover{color:var(--accent);border-color:var(--accent)}
/* fading rotate hint */
#hint{position:absolute;left:50%;top:18px;transform:translateX(-50%);z-index:4;
font-size:11px;letter-spacing:.08em;color:var(--fg-dim);pointer-events:none;
transition:opacity 1s ease;opacity:.85}
#hint.gone{opacity:0}
/* loading */
#loading{position:absolute;inset:0;display:grid;place-items:center;z-index:9;background:var(--bg);
color:var(--fg-dim);font-size:13px;letter-spacing:.1em}
#loading.gone{display:none}
#loading b{color:var(--accent);letter-spacing:-.01em}
/* prose drawer (revealed by the info switch) */
#prose{position:absolute;top:0;right:0;bottom:0;width:min(440px,86vw);z-index:8;overflow:auto;
background:rgba(13,13,13,.96);border-left:1px solid var(--line);padding:40px 26px 64px;
transform:translateX(102%);transition:transform .28s ease}
body:not(.minimal) #prose{transform:none}
#prose h1{font-size:24px;margin:0 0 8px}
#prose h1 b{color:var(--accent)}
#prose p{color:var(--fg-mute);line-height:1.6;margin:0 0 12px}
#prose p b{color:var(--fg)}
#prose .lbl{font-size:11px;letter-spacing:.14em;text-transform:uppercase;color:var(--accent);margin:26px 0 8px}
#prose .ph{font-size:10px;letter-spacing:.1em;text-transform:uppercase;color:var(--fg-dim);
border:1px dashed var(--line-strong);border-radius:6px;padding:6px 9px;display:inline-block;margin-bottom:14px}
#prose .bars{display:flex;flex-direction:column;gap:7px;margin-top:6px}
#prose .ro{display:flex;align-items:center;gap:10px}
#prose .ro span{width:74px;font-size:10px;text-transform:uppercase;letter-spacing:.05em;color:var(--fg-mute)}
#prose .bar{flex:1;height:7px;background:var(--bg-3);border:1px solid var(--line);border-radius:999px;overflow:hidden;position:relative}
#prose .bar i{position:absolute;left:0;top:0;bottom:0;border-radius:999px}
#prose .num{width:3.2ch;text-align:right;font-variant-numeric:tabular-nums;color:var(--fg);font-size:12px}
#prose a{color:var(--accent-2);text-decoration:none}
</style>
</head>
<body class="minimal">
<div id="app">
<canvas id="scene"></canvas>
<div id="hint">drag to rotate · top-down view = the flat 2-D map</div>
<!-- ZONE 1 · input: the 2-D control + parameter count (left) -->
<div class="hud hud-l">
<div class="zonecap">input · 2-D control</div>
<canvas id="ndpad" width="140" height="140"></canvas>
<div class="kbox">
<span class="lab">params</span>
<button class="step" id="kdown" title="fewer parameters">−</button>
<span class="val" id="kval">1</span>
<button class="step" id="kup" title="more parameters">+</button>
</div>
</div>
<!-- ZONE 3 · outputs: vertical faders, one per layer (right) -->
<div class="hud hud-r">
<div class="zonecap">outputs</div>
<div class="outs" id="outs"></div>
</div>
<!-- action bar (bottom-center) -->
<div class="hud hud-bc actionbar">
<button class="act" id="aRand"><span class="ic">🎲</span><span class="tx">randomise</span></button>
<button class="act" id="aNudge"><span class="ic">〜</span><span class="tx">nudge</span></button>
<button class="act" id="aUndo"><span class="ic">↶</span><span class="tx">undo</span></button>
<div class="sep"></div>
<button class="act up" id="aUp"><span class="ic">👍</span><span class="tx">keep</span></button>
<button class="act down" id="aDown"><span class="ic">👎</span><span class="tx" id="downTx">push away</span></button>
<div class="sep"></div>
<div class="modeseg">
<button id="mPush" class="on">push&#8209;away</button>
<button id="mExp">explore</button>
</div>
</div>
<button id="infoBtn" class="hud hud-tr" title="show / hide explanation">ⓘ</button>
<div id="loading"><span>loading the <b>&nbsp;NISPS&nbsp;</b> engine…</span></div>
</div>
<!-- placeholder explanatory copy (revealed by the ⓘ switch; to be reworked) -->
<div id="prose">
<h1>The <b>parameter landscape</b></h1>
<p class="ph">placeholder copy — to be reworked</p>
<p>Two numbers go in — your <b>X / Y control position</b> — and a whole fistful of parameters comes
out. You can't picture a dozen dimensions at once, so picture them <b>stacked</b>: each <b>slab</b> is one
parameter, drawn as a 3-D landscape whose height at any spot is that parameter's value. <b>One probe</b>
reads every slab at the same control position.</p>
<p>Seen straight from above the slabs overlap into a single flat map — it looks 2-D. <b>Tilt it</b> and the
reveal lands: it was a stack of 3-D surfaces all along. Add parameters to grow the sandwich.</p>
<p>Every surface here is the <b>real NISPS network</b> sampled across the input space. <b>👍 keep</b> teaches
the net that this sound belongs at this spot and the landscape bends to honour it; <b>👎</b> either pushes the
mapping away from the current sound or (in explore mode) re-rolls a fresh candidate to audition. <b>nudge</b>
and <b>randomise</b> perturb the net; <b>undo</b> steps back.</p>
<div class="lbl">Values under the probe</div>
<div class="bars" id="ndbars"></div>
<p style="margin-top:24px"><a href="../">← back to the instrument</a> · <a href="../animations/">animated explainers →</a></p>
</div>
</div>
<script>
(() => {
'use strict';
const clamp=(v,a=0,b=1)=>v<a?a:v>b?b:v;
const cssv=n=>getComputedStyle(document.documentElement).getPropertyValue(n).trim();
const MONO=cssv('--font-mono')||'monospace';
const NAMES=['Cutoff','Resonance','Drive','Tone','Decay','Air','Pitch','Width'];
const ABBR =['Cut','Res','Drv','Ton','Dec','Air','Pit','Wid'];
const MAXK=NAMES.length;
const OUT=126, IN=2, GRID=12, N=GRID, PTS=(N+1)*(N+1);
let K=1; // parameters (slabs) visualised — from the same net
let cx=0.5, cy=0.5; // 2-D control position in [0,1]
let mode='pushaway'; // 'pushaway' | 'explore'
let exploring=false; // explore-and-place scratchpad session active
let ready=false, auto=true, autoT=0.6;
// camera — START TOP-DOWN (looks 2-D); drag tilts to reveal the 3-D stack
let yaw=0, pitch=Math.PI/2, dragRot=false, lastX=0, lastY=0;
// ===== engine (vanilla wrapper over nisps.wasm; same C ABI the manifold uses) =====
const E={
mod:null,h:0,bIn:0,bOut:0,bF:0,bL:0,bW:0,bFb:0,wc:0,
async init(){
// nisps.{js,wasm} sit one level up at /next/ in production; probe a few bases.
let base=null,src=null;
for(const b of ['../','','./']){
try{const r=await fetch(b+'nisps.js');if(r.ok){src=await r.text();base=b;break;}}catch(e){}
}
if(base===null) throw new Error('nisps.js not found');
if(!window.createNispsModule)(0,eval)(src);
this.mod=await window.createNispsModule({locateFile:p=>p.endsWith('.wasm')?base+'nisps.wasm':p});
const m=this.mod;
this.h=m._nisps_ml_create(IN,OUT,0,0,(Date.now()>>>0));
this.wc=m._nisps_ml_weight_count(this.h);
this.bF=m._malloc(IN*4); this.bL=m._malloc(OUT*4); this.bFb=m._malloc(OUT*4);
this.bW=m._malloc(this.wc*4);
this.bIn=m._malloc(PTS*IN*4); this.bOut=m._malloc(PTS*OUT*4);
},
inferXY(x,y){const m=this.mod;m._nisps_ml_set_input(this.h,0,x);m._nisps_ml_set_input(this.h,1,y);
m._nisps_ml_process(this.h);const p=m._nisps_ml_outputs(this.h);
return new Float32Array(m.HEAPF32.buffer,p,OUT);},
inferGrid(){const m=this.mod;const inv=new Float32Array(m.HEAPF32.buffer,this.bIn,PTS*IN);
let q=0;for(let j=0;j<=N;j++)for(let i=0;i<=N;i++){inv[q++]=i/N;inv[q++]=j/N;}
m._nisps_ml_infer_batch(this.h,this.bIn,PTS,this.bOut);
return new Float32Array(new Float32Array(m.HEAPF32.buffer,this.bOut,PTS*OUT));},
addExample(f,l){const m=this.mod;const fv=new Float32Array(m.HEAPF32.buffer,this.bF,IN);
const lv=new Float32Array(m.HEAPF32.buffer,this.bL,OUT);fv[0]=f[0];fv[1]=f[1];
for(let i=0;i<OUT;i++)lv[i]=l[i]||0;m._nisps_ml_add_example(this.h,this.bF,this.bL);},
train(lr,it,me){return this.mod._nisps_ml_train(this.h,lr,it,me,0);},
draw(s){this.mod._nisps_ml_draw_weights(this.h,s);},
move(sp,s){this.mod._nisps_ml_move_weights(this.h,sp,s,0);},
setMode(i){this.mod._nisps_ml_feedback_set_mode(this.h,i);},
down(out,sp,s){const m=this.mod;const ov=new Float32Array(m.HEAPF32.buffer,this.bFb,OUT);
ov.fill(0);for(let i=0;i<OUT;i++)ov[i]=out[i]||0;m._nisps_ml_feedback_down(this.h,this.bFb,sp,s,0);},
enterExplore(s){this.mod._nisps_ml_feedback_enter_explore(this.h,s);},
reroll(s){this.mod._nisps_ml_feedback_reroll(this.h,s);},
coreNudge(a){this.mod._nisps_ml_feedback_nudge(this.h,a);},
coreUndo(){this.mod._nisps_ml_feedback_undo(this.h);},
exitExplore(){this.mod._nisps_ml_feedback_exit_explore(this.h);},
getWeights(){this.mod._nisps_ml_get_weights(this.h,this.bW);
return new Float32Array(new Float32Array(this.mod.HEAPF32.buffer,this.bW,this.wc));},
setWeights(w){const v=new Float32Array(this.mod.HEAPF32.buffer,this.bW,this.wc);v.set(w);
this.mod._nisps_ml_set_weights(this.h,this.bW);},
};
// Low spread = polarised weights = strong spatial relief (measured); high spread
// flattens the surface toward 0.5. Keep it low so the landscapes show real terrain.
const SPREAD=0.1, NUDGE=0.05, BOOT_SPREAD=0.05;
let grid=null, probeOut=new Float32Array(OUT);
function H(k,i,j){return grid?grid[(j*(N+1)+i)*OUT+k]:0.5;}
function resample(){grid=E.inferGrid();}
function updateProbe(){const o=E.inferXY(cx,cy);probeOut=new Float32Array(o);}
// ---- weight-snapshot undo (push-away mode; explore mode uses the core ring) ----
const undoStack=[];
function pushUndo(){undoStack.push(E.getWeights());if(undoStack.length>24)undoStack.shift();}
function popUndo(){const w=undoStack.pop();if(w){E.setWeights(w);return true;}return false;}
// ---- feedback examples marked on the 2-D control plane (like the main app) ----
// {x,y,sign} — sign +1 kept (👍), -1 pushed away (👎). undo pops the last.
const marks=[];
// ===== actions (all backed by the real engine) =====
function act(name){
if(!ready)return;
if(mode==='pushaway'){
if(name==='rand'){pushUndo();E.draw(SPREAD);}
else if(name==='nudge'){pushUndo();E.move(0.04,SPREAD);}
else if(name==='undo'){popUndo();marks.pop();}
else if(name==='up'){pushUndo();const o=new Float32Array(E.inferXY(cx,cy));E.addExample([cx,cy],o);E.train(1.0,200,1e-4);marks.push({x:cx,y:cy,sign:1});}
else if(name==='down'){pushUndo();const o=new Float32Array(E.inferXY(cx,cy));E.down(o,0.15,SPREAD);marks.push({x:cx,y:cy,sign:-1});}
}else{ // explore-and-place
if(name==='rand'||name==='down'){
if(!exploring){E.setMode(3);E.enterExplore(SPREAD);exploring=true;}else E.reroll(SPREAD);
}else if(name==='nudge'){
if(!exploring){E.setMode(3);E.enterExplore(SPREAD);exploring=true;}else E.coreNudge(NUDGE);
}else if(name==='undo'){
if(exploring)E.coreUndo();else{popUndo();marks.pop();}
}else if(name==='up'){
const o=new Float32Array(E.inferXY(cx,cy));
if(exploring){E.exitExplore();exploring=false;}
E.addExample([cx,cy],o);E.train(1.0,200,1e-4);marks.push({x:cx,y:cy,sign:1});
}
}
resample();updateProbe();reflectButtons();redraw();
}
function setMode(m){
if(m===mode)return;
if(mode==='explore'&&exploring){E.exitExplore();exploring=false;}
mode=m;E.setMode(m==='explore'?3:0);
document.getElementById('mPush').classList.toggle('on',m==='pushaway');
document.getElementById('mExp').classList.toggle('on',m==='explore');
document.getElementById('downTx').textContent=m==='explore'?'skip':'push away';
if(ready){resample();updateProbe();reflectButtons();redraw();}
}
function reflectButtons(){
const ud=document.getElementById('aUndo');
ud.disabled=(mode==='explore'?false:undoStack.length===0)&&!exploring;
}
// ===== 3-D projection: yaw + pitch orbit, bounding-sphere fit, X inverted =====
const SX=300,SZ=300,LAYER_GAP=120,ZAMP=92,MARGIN=44;
// Anchor the view to the TOP slab (k=0) at a FIXED world Y and a FIXED screen
// position, independent of K. So adding parameters grows the stack DOWNWARD and
// the top slab always stays the top — it never re-centres onto new layers.
const AWY=ZAMP*0.5, ANCHOR_FY=0.42;
let CW=800,CH=600,scale=1;
function computeView(){
let R=1; // bounding-sphere radius about the top anchor
for(let k=0;k<K;k++)for(const z of[0,1])for(const gx of[-0.5,0.5])for(const gy of[-0.5,0.5]){
const X=gx*SX,Y=(-k*LAYER_GAP+z*ZAMP)-AWY,Z=gy*SZ;const d=Math.sqrt(X*X+Y*Y+Z*Z);if(d>R)R=d;}
scale=(Math.min(CW,CH)/2-MARGIN)/R;
}
function project(gx,gy,zv,k){
const X=gx*SX, Y=(-k*LAYER_GAP+zv*ZAMP)-AWY, Z=-gy*SZ; // Z negated: pad-up = map-up
const cyA=Math.cos(yaw),syA=Math.sin(yaw);
const x1=X*cyA-Z*syA, z1=X*syA+Z*cyA;
const cpA=Math.cos(pitch),spA=Math.sin(pitch);
const y2=Y*cpA-z1*spA, z2=Y*spA+z1*cpA;
return {x:CW/2+x1*scale,y:CH*ANCHOR_FY-y2*scale,depth:z2,wx:X,wy:Y,wz:Z};
}
const DPR=Math.max(1,Math.min(2,window.devicePixelRatio||1));
const scene=document.getElementById('scene'),sctx=scene.getContext('2d');
const pad=document.getElementById('ndpad'),pctx=pad.getContext('2d');
pad.width=140*DPR;pad.height=140*DPR;pctx.scale(DPR,DPR);
function resize(){
const r=scene.getBoundingClientRect();CW=Math.max(320,Math.round(r.width));CH=Math.max(320,Math.round(r.height));
scene.width=CW*DPR;scene.height=CH*DPR;sctx.setTransform(DPR,0,0,DPR,0,0);computeView();
}
const hueOf=k=>K>1?(26+(196-26)*k/(K-1)):26;
const colCss=(k,a)=>`hsla(${hueOf(k).toFixed(0)},85%,60%,${a})`;
const LIGHT=(()=>{const v=[0.35,0.86,0.36],m=Math.hypot(v[0],v[1],v[2]);return[v[0]/m,v[1]/m,v[2]/m];})();
function renderScene(){
const c=sctx;c.clearRect(0,0,CW,CH);
const quads=[],labels=[];
for(let k=0;k<K;k++){
const hue=hueOf(k),P=[];
for(let j=0;j<=N;j++){P[j]=[];for(let i=0;i<=N;i++)P[j][i]=project(i/N-0.5,j/N-0.5,H(k,i,j),k);}
for(let j=0;j<N;j++)for(let i=0;i<N;i++){
const a=P[j][i],b=P[j][i+1],d=P[j+1][i+1],e=P[j+1][i];
const ux=d.wx-a.wx,uy=d.wy-a.wy,uz=d.wz-a.wz,vx=b.wx-e.wx,vy=b.wy-e.wy,vz=b.wz-e.wz;
const nx=uy*vz-uz*vy,ny=uz*vx-ux*vz,nz=ux*vy-uy*vx,nm=Math.hypot(nx,ny,nz)||1;
const L=clamp(0.32+0.62*Math.abs((nx*LIGHT[0]+ny*LIGHT[1]+nz*LIGHT[2])/nm));
quads.push({a,b,d,e,hue,L,depth:(a.depth+b.depth+d.depth+e.depth)/4});
}
const lp=project(-0.5,0.5,H(k,0,N),k);labels.push({x:lp.x,y:lp.y,hue,name:NAMES[k]});
}
quads.sort((p,q)=>q.depth-p.depth);
for(const Q of quads){
c.fillStyle=`hsla(${Q.hue},80%,${(26+Q.L*44).toFixed(0)}%,0.66)`;
c.strokeStyle=`hsla(${Q.hue},80%,82%,0.18)`;c.lineWidth=0.6;
c.beginPath();c.moveTo(Q.a.x,Q.a.y);c.lineTo(Q.b.x,Q.b.y);c.lineTo(Q.d.x,Q.d.y);c.lineTo(Q.e.x,Q.e.y);c.closePath();
c.fill();c.stroke();
}
c.font='11px '+MONO;c.textAlign='right';c.textBaseline='middle';
for(const Lb of labels){c.fillStyle=`hsla(${Lb.hue},85%,74%,0.95)`;c.fillText(Lb.name,Lb.x-6,Lb.y);}
// probe column at (cx,cy) through every slab
const gx=cx-0.5,gy=cy-0.5;
const top=project(gx,gy,1.18,0),bot=project(gx,gy,-0.18,K-1);
c.strokeStyle='rgba(255,255,255,0.32)';c.lineWidth=1;c.setLineDash([3,4]);
c.beginPath();c.moveTo(top.x,top.y);c.lineTo(bot.x,bot.y);c.stroke();c.setLineDash([]);
for(let k=0;k<K;k++){
const p=project(gx,gy,clamp(probeOut[k]),k),fl=project(gx,gy,0,k);
c.strokeStyle=colCss(k,0.5);c.lineWidth=1;c.beginPath();c.moveTo(p.x,p.y);c.lineTo(fl.x,fl.y);c.stroke();
c.fillStyle=colCss(k,1);c.shadowColor=colCss(k,1);c.shadowBlur=12;
c.beginPath();c.arc(p.x,p.y,6,0,Math.PI*2);c.fill();c.shadowBlur=0;
c.fillStyle='#0d0d0d';c.beginPath();c.arc(p.x,p.y,2.2,0,Math.PI*2);c.fill();
}
}
function drawPad(){
pctx.clearRect(0,0,140,140);pctx.fillStyle='#1c1c1c';pctx.fillRect(0,0,140,140);
pctx.strokeStyle='rgba(255,255,255,0.06)';pctx.lineWidth=1;
for(let i=1;i<4;i++){const t=i/4*140;pctx.beginPath();pctx.moveTo(t,0);pctx.lineTo(t,140);pctx.stroke();
pctx.beginPath();pctx.moveTo(0,t);pctx.lineTo(140,t);pctx.stroke();}
// feedback examples (👍 +, 👎 −) — same convention as the main app
const TAU=Math.PI*2;
for(const m of marks){const mx=m.x*140,my=(1-m.y)*140,col=m.sign>0?'#ff6a00':'#00ccff';
pctx.globalAlpha=0.28;pctx.fillStyle=col;pctx.beginPath();pctx.arc(mx,my,8,0,TAU);pctx.fill();
pctx.globalAlpha=1;pctx.strokeStyle=col;pctx.lineWidth=1.5;pctx.beginPath();pctx.arc(mx,my,5,0,TAU);pctx.stroke();
pctx.fillStyle=col;pctx.font='9px '+MONO;pctx.textAlign='center';pctx.textBaseline='middle';
pctx.fillText(m.sign>0?'+':'–',mx,my+0.5);}
const dx=cx*140,dy=(1-cy)*140;
pctx.strokeStyle='rgba(255,255,255,0.1)';pctx.beginPath();pctx.moveTo(0,dy);pctx.lineTo(140,dy);
pctx.moveTo(dx,0);pctx.lineTo(dx,140);pctx.stroke();
pctx.fillStyle='#ff6a00';pctx.shadowColor='#ff6a00';pctx.shadowBlur=12;
pctx.beginPath();pctx.arc(dx,dy,6,0,Math.PI*2);pctx.fill();pctx.shadowBlur=0;
pctx.fillStyle='#0d0d0d';pctx.beginPath();pctx.arc(dx,dy,2.4,0,Math.PI*2);pctx.fill();
}
// readout bars (prose drawer only)
const barsEl=document.getElementById('ndbars');let barI=[],barN=[];
function buildBars(){barsEl.innerHTML='';barI=[];barN=[];
for(let k=0;k<K;k++){const ro=document.createElement('div');ro.className='ro';
ro.innerHTML=`<span style="color:${colCss(k,0.95)}">${NAMES[k]}</span>`+
`<div class="bar"><i style="background:${colCss(k,1)}"></i></div><div class="num">0.50</div>`;
barsEl.appendChild(ro);barI[k]=ro.querySelector('i');barN[k]=ro.querySelector('.num');}}
function updateBars(){if(!barI.length)return;for(let k=0;k<K;k++){const v=clamp(probeOut[k]);
barI[k].style.width=(v*100)+'%';barN[k].textContent=v.toFixed(2);}}
// output faders — the 1-D projection of each layer's value at the current point
const TRACK_H=200;
const outsEl=document.getElementById('outs');let fFill=[],fHan=[],fVal=[];
function buildOuts(){outsEl.innerHTML='';fFill=[];fHan=[];fVal=[];
for(let k=0;k<K;k++){const col=colCss(k,1);const f=document.createElement('div');f.className='fader';
f.innerHTML=`<div class="track"><div class="fill" style="background:${col}"></div>`+
`<div class="handle" style="background:${col};color:${col}"></div></div>`+
`<div class="flab" style="color:${colCss(k,0.95)}" title="${NAMES[k]}">${ABBR[k]}</div><div class="fval">0.50</div>`;
outsEl.appendChild(f);fFill[k]=f.querySelector('.fill');fHan[k]=f.querySelector('.handle');fVal[k]=f.querySelector('.fval');}}
function updateOuts(){if(!fFill.length)return;for(let k=0;k<K;k++){const v=clamp(probeOut[k]);
fFill[k].style.height=(v*100)+'%';fHan[k].style.bottom=(v*TRACK_H)+'px';fVal[k].textContent=v.toFixed(2);}}
function redraw(){renderScene();drawPad();updateBars();updateOuts();}
// ===== interaction =====
function setK(n){K=clamp(n,1,MAXK);document.getElementById('kval').textContent=K;
document.getElementById('kup').disabled=K>=MAXK;document.getElementById('kdown').disabled=K<=1;
computeView();buildBars();buildOuts();updateProbe&&(probeOut=new Float32Array(E.inferXY(cx,cy)));redraw();}
function stopAuto(){auto=false;document.getElementById('hint').classList.add('gone');}
// pad → control position
function setFromPad(e){const r=pad.getBoundingClientRect();
cx=clamp((e.clientX-r.left)/r.width);cy=clamp(1-(e.clientY-r.top)/r.height);}
let padDrag=false;
pad.addEventListener('pointerdown',e=>{padDrag=true;pad.setPointerCapture(e.pointerId);stopAuto();setFromPad(e);updateProbe();redraw();});
pad.addEventListener('pointermove',e=>{if(padDrag){setFromPad(e);updateProbe();redraw();}});
pad.addEventListener('pointerup',()=>padDrag=false);
// scene → orbit (pitch inverted; pitch up to π/2 = exact top-down)
scene.addEventListener('pointerdown',e=>{dragRot=true;scene.setPointerCapture(e.pointerId);stopAuto();lastX=e.clientX;lastY=e.clientY;});
scene.addEventListener('pointermove',e=>{if(!dragRot)return;
yaw+=(e.clientX-lastX)*0.01;
pitch=clamp(pitch+(e.clientY-lastY)*0.008,0.06,Math.PI/2); // vertical (Y) drag — inverted
lastX=e.clientX;lastY=e.clientY;redraw();});
scene.addEventListener('pointerup',()=>dragRot=false);
scene.addEventListener('pointercancel',()=>dragRot=false);
// buttons
const B=(id,fn)=>document.getElementById(id).addEventListener('click',()=>{stopAuto();fn();});
B('aRand',()=>act('rand'));B('aNudge',()=>act('nudge'));B('aUndo',()=>act('undo'));
B('aUp',()=>act('up'));B('aDown',()=>act('down'));
B('kup',()=>setK(K+1));B('kdown',()=>setK(K-1));
document.getElementById('mPush').addEventListener('click',()=>{stopAuto();setMode('pushaway');});
document.getElementById('mExp').addEventListener('click',()=>{stopAuto();setMode('explore');});
document.getElementById('infoBtn').addEventListener('click',()=>{
document.body.classList.toggle('minimal');
document.getElementById('infoBtn').textContent=document.body.classList.contains('minimal')?'ⓘ':'✕';});
window.addEventListener('resize',()=>{resize();redraw();});
function loop(){
if(auto&&ready&&!padDrag&&!dragRot){
autoT+=0.006;cx=0.5+0.42*Math.sin(autoT*0.9);cy=0.5+0.42*Math.sin(autoT*1.27+1.3);
updateProbe();redraw();
}
requestAnimationFrame(loop);
}
// ===== boot =====
(async()=>{
try{
await E.init();
E.draw(BOOT_SPREAD); // a fresh net is near-flat; draw for visible relief
window.__learn={E,setK,resample,redraw,
range(k){let lo=1,hi=0;for(let j=0;j<=N;j++)for(let i=0;i<=N;i++){const v=H(k,i,j);if(v<lo)lo=v;if(v>hi)hi=v;}return[lo,hi];}};
resize();setK(1);resample();updateProbe();reflectButtons();
ready=true;document.getElementById('loading').classList.add('gone');
redraw();requestAnimationFrame(loop);
// fade the hint after a while
setTimeout(()=>{if(auto)document.getElementById('hint').classList.add('gone');},6000);
}catch(err){
console.error('[learn] engine init failed:',err);
document.getElementById('loading').innerHTML='<span style="color:#c8553a">engine failed to load — see console</span>';
}
})();
})();
</script>
</body>
</html>

View file

@ -31,6 +31,7 @@
#include "glue/midi_io.hpp" #include "glue/midi_io.hpp"
#include "glue/mode_select.hpp" #include "glue/mode_select.hpp"
#include "glue/output_router.hpp" #include "glue/output_router.hpp"
#include "glue/settings_view.hpp"
// ---- Mode selection ---- // ---- Mode selection ----
// Build script rewrites which line is uncommented. // Build script rewrites which line is uncommented.
@ -141,6 +142,8 @@ void setup() {
} }
MEMLNaut::Instance()->addSystemInfoView(); MEMLNaut::Instance()->addSystemInfoView();
// Settings menu (e.g. Joystick: Dual/Single for the 4-input modes).
nisps_firmware::wire_settings(g_mode);
Serial.println("Finished initialising core 0."); Serial.println("Finished initialising core 0.");
} }

View file

@ -0,0 +1,64 @@
// glue/settings_view.hpp — on-device settings menu views (TFT + rotary encoder).
//
// Adds entries to the MEMLNaut display carousel (the same DisplayDriver the
// SystemView and SelfTest use). Navigation: rotate to move between views;
// press to focus a view; rotate-while-focused to change its value; press
// again to unfocus. See src/memllib/.../display/DisplayDriver.hpp.
//
// Currently provides:
// * Joystick: Dual / Single — for the 4-input ("two 2-D joystick") modes,
// toggles whether the SECOND joystick (input channels 2,3) feeds the ML
// engine. "Single" pins those channels to the neutral value (0.5) via the
// platform-agnostic ModeBase pin mechanism, so the network is never
// rebuilt and trained state survives toggling. Default is Dual (the 4-D
// variant naturally uses both joysticks).
//
// Only registered for modes whose input arity is exactly 4 (the two-joystick
// modes); other modes (e.g. SoundAnalysisMIDI, whose joystick channels are not
// 2,3) get no joystick toggle.
#pragma once
#include <array>
#include <cstddef>
#include <memory>
#include <span>
#include "../src/memllib/hardware/memlnaut/MEMLNaut.hpp"
#include "../src/memllib/hardware/memlnaut/display/SingleSelectView.hpp"
namespace nisps_firmware {
// The second 2-D joystick maps to ML input channels 2 and 3 (joy_z / joy_w).
inline constexpr std::size_t kSecondJoystickCh0 = 2u;
inline constexpr std::size_t kSecondJoystickCh1 = 3u;
// Add the settings views for `mode` to the display carousel. Must run AFTER
// MEMLNaut::Initialize() (so the display + TFT exist) and after
// addSystemInfoView(), on core 0.
template <typename ModeT>
inline void wire_settings(ModeT& mode) {
if constexpr (ModeT::input_channel_count() == 4u) {
MEMLNaut* meml = MEMLNaut::Instance();
if (!meml || !meml->disp) return;
// Kept alive for the program lifetime (the carousel holds a shared_ptr
// too, but this guarantees ownership regardless of view churn).
static std::shared_ptr<SingleSelectView> joystick_view;
joystick_view = std::make_shared<SingleSelectView>("Joystick");
meml->disp->AddView(joystick_view); // calls Setup() → creates the selector
// Index 0 = Dual (default, no pinning); index 1 = Single.
std::array<String, 2u> options = { String("Dual"), String("Single") };
joystick_view->setOptions(std::span<String>(options.data(), options.size()));
joystick_view->setNewVoiceCallback([&mode](std::size_t idx) {
const bool single = (idx == 1u);
mode.set_input_pinned(kSecondJoystickCh0, single);
mode.set_input_pinned(kSecondJoystickCh1, single);
});
} else {
(void)mode;
}
}
} // namespace nisps_firmware

Binary file not shown.

View file

@ -28,6 +28,7 @@ import { MF_MODES, modeEngineId, seededGradient, shapeValues } from './model';
import type { MFParam } from './model'; import type { MFParam } from './model';
import { CompositeStage } from './CompositeStage'; import { CompositeStage } from './CompositeStage';
import { ParticleStage } from './ParticleStage'; import { ParticleStage } from './ParticleStage';
import { SandwichStage } from './SandwichStage';
import { SplitStage } from './SplitStage'; import { SplitStage } from './SplitStage';
import { OutputStage } from './OutputStage'; import { OutputStage } from './OutputStage';
import { InputMini } from './InputMini'; import { InputMini } from './InputMini';
@ -107,6 +108,8 @@ export function ConsoleApp({ focus: initialFocus = 'composite' }: ConsoleAppProp
const [rev, setRev] = useState(1); const [rev, setRev] = useState(1);
const [active, setActive] = useState<DrawerKey | null>('learn'); const [active, setActive] = useState<DrawerKey | null>('learn');
const [depth, setDepth] = useState<DrawerDepth>('peek'); const [depth, setDepth] = useState<DrawerDepth>('peek');
// Sandwich (parameter-landscape) centre-stage toggle — dock-bottom layers icon.
const [sandwich, setSandwich] = useState(false);
// Learning-behaviour store (dock-spec §1; rl-feedback-design). Default // Learning-behaviour store (dock-spec §1; rl-feedback-design). Default
// feedback mode = "Explore and place"; default solo = "Mask gradients". // feedback mode = "Explore and place"; default solo = "Mask gradients".
@ -718,7 +721,55 @@ export function ConsoleApp({ focus: initialFocus = 'composite' }: ConsoleAppProp
bottom: 0, bottom: 0,
}} }}
> >
{outputMode === 'particles' ? ( {sandwich ? (
// Sandwich centre-stage: shrunken input (left) · landscape stack
// (centre, fills) · compact outputs (right). Replaces the Mode stage.
<div style={{ position: 'absolute', inset: 0, display: 'flex' }}>
<div
style={{
width: 'clamp(200px, 24%, 320px)',
flex: '0 0 auto',
position: 'relative',
borderRight: '1px solid var(--line)',
}}
>
<Manifold
pos={pos}
onMove={onMove}
noiseCap={noiseCap}
pins={pins}
markers={markers}
variant={inputMapVariant}
frozen={false}
follow={follow}
onLongPress={addPin}
picking={picking}
onPickLocation={onPickLocation}
/>
</div>
<div style={{ flex: 1, minWidth: 0, position: 'relative' }}>
{engine && (
<SandwichStage
engine={engine}
version={version}
pos={pos}
layerCount={Math.min(params.length, 8)}
names={params.map((p) => p.name)}
/>
)}
</div>
<div
style={{
width: 'clamp(200px, 24%, 320px)',
flex: '0 0 auto',
position: 'relative',
borderLeft: '1px solid var(--line)',
}}
>
<OutputStage params={params} values={values} onChange={setParam} compact />
</div>
</div>
) : outputMode === 'particles' ? (
<ParticleStage <ParticleStage
pos={pos} pos={pos}
onMove={onMove} onMove={onMove}
@ -903,7 +954,19 @@ export function ConsoleApp({ focus: initialFocus = 'composite' }: ConsoleAppProp
</div> </div>
</div> </div>
<Dock ctx={ctx} active={active} setActive={setActive} depth={depth} setDepth={setDepth} /> <Dock
ctx={ctx}
active={active}
setActive={setActive}
depth={depth}
setDepth={setDepth}
sandwich={sandwich}
setSandwich={(v) => {
setSandwich(v);
// reveal the full 3-zone layout: a drawer would overlay the output zone
if (v) setActive(null);
}}
/>
</div> </div>
); );
} }

View file

@ -28,6 +28,7 @@ import {
OscIcon, OscIcon,
SynthIcon, SynthIcon,
EditorIcon, EditorIcon,
SandwichIcon,
CloseIcon, CloseIcon,
ExpandIcon, ExpandIcon,
GLYPH_FALLBACK, GLYPH_FALLBACK,
@ -173,11 +174,14 @@ export interface DockProps {
setActive: (k: DrawerKey | null) => void; setActive: (k: DrawerKey | null) => void;
depth: DrawerDepth; depth: DrawerDepth;
setDepth: (d: DrawerDepth) => void; setDepth: (d: DrawerDepth) => void;
/** Sandwich (parameter-landscape) centre-stage toggle. */
sandwich: boolean;
setSandwich: (v: boolean) => void;
} }
const ORDER: DrawerKey[] = ['learn', 'inputs', 'route', 'settings', 'help']; const ORDER: DrawerKey[] = ['learn', 'inputs', 'route', 'settings', 'help'];
export function Dock({ ctx, active, setActive, depth, setDepth }: DockProps) { export function Dock({ ctx, active, setActive, depth, setDepth, sandwich, setSandwich }: DockProps) {
const { settings } = useSettings(); const { settings } = useSettings();
const mono = settings.monochromeIcons; const mono = settings.monochromeIcons;
const restColour = unfocusedIconCss(settings.unfocusedIconColour); const restColour = unfocusedIconCss(settings.unfocusedIconColour);
@ -368,6 +372,27 @@ export function Dock({ ctx, active, setActive, depth, setDepth }: DockProps) {
> >
{ORDER.map(iconBtn)} {ORDER.map(iconBtn)}
</div> </div>
{/* Sandwich toggle — pinned at the BOTTOM of the rail. */}
<button
type="button"
title="Parameter sandwich (layer view)"
onClick={() => setSandwich(!sandwich)}
style={{
width: 40,
height: 40,
borderRadius: 'var(--r-2)',
cursor: 'pointer',
border: `1px solid ${sandwich ? 'var(--accent)' : 'transparent'}`,
background: sandwich ? 'rgba(255,106,0,0.14)' : 'transparent',
color: sandwich ? 'var(--accent)' : mono ? restColour : 'var(--fg-mute)',
display: 'flex',
alignItems: 'center',
justifyContent: 'center',
transition: 'background var(--dur-fast), color var(--dur-fast)',
}}
>
{mono ? <SandwichIcon /> : '≣'}
</button>
</nav> </nav>
</> </>
); );

View file

@ -240,6 +240,7 @@ function InputsDrawer(ctx: ConsoleCtx, depth: DrawerDepth) {
const inp = ctx.inputs; const inp = ctx.inputs;
const enabledCount = inp.sources.filter((s) => s.enabled).length; const enabledCount = inp.sources.filter((s) => s.enabled).length;
const reshaping = inp.axisCount > inp.engineInputSize; const reshaping = inp.axisCount > inp.engineInputSize;
const gamepadOn = inp.sources.find((s) => s.kind === 'gamepad')?.enabled ?? false;
return ( return (
<> <>
@ -252,6 +253,22 @@ function InputsDrawer(ctx: ConsoleCtx, depth: DrawerDepth) {
{reshaping && <Chip tone="var(--warn)">blended → {inp.engineInputSize}</Chip>} {reshaping && <Chip tone="var(--warn)">blended → {inp.engineInputSize}</Chip>}
</div> </div>
{/* ---- Controllers: 1 (pad) vs 2 (pad + gamepad). Default single. ---- */}
<SectionLabel>Controllers</SectionLabel>
<Segmented
value={gamepadOn ? 'double' : 'single'}
onChange={(m) => inp.setEnabled('gamepad', m === 'double')}
options={[
{ value: 'single', label: 'Single' },
{ value: 'double', label: 'Double (+gamepad)' },
]}
/>
<p style={{ fontSize: 9, color: 'var(--fg-dim)', margin: '2px 0 0' }}>
{gamepadOn
? 'Two 2-D controllers feed the model (pad → axes 0,1; gamepad → axes 2,3).'
: 'One 2-D controller (the pad). The second controller’s axes stay neutral (0.5).'}
</p>
<SectionLabel>Sources</SectionLabel> <SectionLabel>Sources</SectionLabel>
<div style={{ display: 'flex', flexDirection: 'column', gap: 6 }}> <div style={{ display: 'flex', flexDirection: 'column', gap: 6 }}>
{inp.sources.map((s) => ( {inp.sources.map((s) => (
@ -382,16 +399,15 @@ function InputsDrawer(ctx: ConsoleCtx, depth: DrawerDepth) {
)} )}
<p style={{ fontSize: 9, color: 'var(--fg-dim)', margin: 0, lineHeight: 1.6 }}> <p style={{ fontSize: 9, color: 'var(--fg-dim)', margin: 0, lineHeight: 1.6 }}>
The active sources concatenate into one input vector at the head of the spine. The active sources concatenate into one input vector at the head of the spine. The
browser WASM is a {inp.engineInputSize}-input head (two 2-D controllers), so up to
{' '}{inp.engineInputSize} axes each get their own genuine input dimension; unused axes
stay neutral (0.5).
{reshaping {reshaping
? ` The browser WASM is a fixed ${inp.engineInputSize}-input head (MLP<2,…>), so the ? ` Right now ${inp.axisCount} axes exceed the ${inp.engineInputSize}-input head, so
${inp.axisCount} axes are blended down to ${inp.engineInputSize} (even→X / odd→Y mean).` the surplus is blended down (chunked mean) — drop a source to give every axis its
: ''}{' '} own dimension.`
{/* TODO(workstream F, docs/redesign/inputs-spec.md — "multiple WASM modules + : ''}
warm-start"): give every axis its own genuine input dimension by (re)loading a
WASM module whose MLP arity matches axisCount and warm-starting from the prior
net. Deferred — the reduction lives in InputLayer.compose(). */}
True per-axis dimensions land with the multi-WASM reshape (inputs-spec).
</p> </p>
</> </>
)} )}

View file

@ -0,0 +1,198 @@
/**
* SandwichStage — the N-dimensional "parameter sandwich" as a centre stage,
* driven by the REAL engine (the same `EngineApi` instance the rest of the
* console uses). Ported from the standalone /learn renderer.
*
* Each layer is one output channel of the MLP, sampled across the 2-D input
* grid via `engine.inferBatch` → that surface IS the parameter's landscape.
* The grid is re-sampled only when the net changes (`version`); a per-frame
* rAF reads `engine.getOutputs()` for the live probe at the current input.
*
* Three-zone metaphor (input → MLP stack → outputs): this component owns the
* MIDDLE zone. ConsoleApp docks a shrunken Manifold (left) and OutputStage
* (right) around it. Drag the canvas to orbit; it lands top-down (a flat 2-D
* map) and tilts into the 3-D stack. The stack is centred and grows to fill
* the available height as more parameters are shown.
*/
import { useEffect, useRef } from 'react';
import type { EngineApi } from '../engine';
const GRID = 12;
const N = GRID;
const SX = 300, SZ = 300, LAYER_GAP = 120, ZAMP = 92, MARGIN = 34;
// Reference scale: each layer adds a fixed ~px gap (so the stack GROWS with the
// layer count) until it would overflow, at which point the fit scale caps it.
const REF_SCALE = 0.8;
export interface SandwichStageProps {
engine: EngineApi;
/** Engine version counter — re-sample the landscape grid when it bumps. */
version: number;
/** Current 2-D input (drives the probe column). */
pos: [number, number];
/** How many output channels to render as layers (top→down). */
layerCount: number;
/** Per-layer labels (the mode's param names). */
names: string[];
}
export function SandwichStage({ engine, version, pos, layerCount, names }: SandwichStageProps) {
const wrapRef = useRef<HTMLDivElement>(null);
const canvasRef = useRef<HTMLCanvasElement>(null);
const gridRef = useRef<Float32Array | null>(null);
const posRef = useRef(pos);
posRef.current = pos;
const Kref = useRef(layerCount);
Kref.current = Math.max(1, layerCount);
const namesRef = useRef(names);
namesRef.current = names;
const yaw = useRef(0);
const pitch = useRef(Math.PI / 2); // top-down (looks 2-D) until you tilt it
const view = useRef({ cw: 800, ch: 600, scale: 1, ceny: 0 });
const OUT = engine.architecture.outputSize;
// ---- re-sample the landscape grid whenever the net changes ----
useEffect(() => {
const pts: [number, number][] = [];
for (let j = 0; j <= N; j++) for (let i = 0; i <= N; i++) pts.push([i / N, j / N]);
try {
gridRef.current = engine.inferBatch(pts);
} catch {
gridRef.current = null;
}
}, [engine, version]);
// ---- render loop ----
useEffect(() => {
const canvas = canvasRef.current;
const wrap = wrapRef.current;
if (!canvas || !wrap) return;
const ctx = canvas.getContext('2d');
if (!ctx) return;
const DPR = Math.max(1, Math.min(2, window.devicePixelRatio || 1));
const computeView = () => {
const v = view.current;
const K = Kref.current;
v.ceny = -(K - 1) * LAYER_GAP / 2 + ZAMP / 2; // centroid (centre vertically)
let R = 1;
for (let k = 0; k < K; k++) for (const z of [0, 1]) for (const gx of [-0.5, 0.5]) for (const gy of [-0.5, 0.5]) {
const X = gx * SX, Y = (-k * LAYER_GAP + z * ZAMP) - v.ceny, Z = gy * SZ;
const d = Math.sqrt(X * X + Y * Y + Z * Z);
if (d > R) R = d;
}
const fit = (Math.min(v.cw, v.ch) / 2 - MARGIN) / R;
v.scale = Math.min(REF_SCALE, fit); // grow with K, then cap to fit
};
const resize = () => {
const r = wrap.getBoundingClientRect();
view.current.cw = Math.max(160, Math.round(r.width));
view.current.ch = Math.max(160, Math.round(r.height));
canvas.width = view.current.cw * DPR;
canvas.height = view.current.ch * DPR;
ctx.setTransform(DPR, 0, 0, DPR, 0, 0);
computeView();
};
const ro = new ResizeObserver(resize);
ro.observe(wrap);
resize();
const project = (gx: number, gy: number, zv: number, k: number) => {
const v = view.current;
const X = gx * SX, Y = (-k * LAYER_GAP + zv * ZAMP) - v.ceny, Z = -gy * SZ;
const cyA = Math.cos(yaw.current), syA = Math.sin(yaw.current);
const x1 = X * cyA - Z * syA, z1 = X * syA + Z * cyA;
const cpA = Math.cos(pitch.current), spA = Math.sin(pitch.current);
const y2 = Y * cpA - z1 * spA, z2 = Y * spA + z1 * cpA;
return { x: v.cw / 2 + x1 * v.scale, y: v.ch / 2 - y2 * v.scale, depth: z2, wx: X, wy: Y, wz: Z };
};
const hueOf = (k: number, K: number) => (K > 1 ? 26 + (196 - 26) * k / (K - 1) : 26);
const colCss = (k: number, K: number, a: number) => `hsla(${hueOf(k, K).toFixed(0)},85%,60%,${a})`;
const LIGHT = (() => { const v = [0.35, 0.86, 0.36]; const m = Math.hypot(v[0], v[1], v[2]); return [v[0] / m, v[1] / m, v[2] / m]; })();
const clamp = (x: number) => (x < 0 ? 0 : x > 1 ? 1 : x);
let raf = 0;
const draw = () => {
const v = view.current;
const K = Kref.current;
const grid = gridRef.current;
computeView();
ctx.clearRect(0, 0, v.cw, v.ch);
const H = (k: number, i: number, j: number) => (grid ? grid[(j * (N + 1) + i) * OUT + k] : 0.5);
const quads: { a: any; b: any; d: any; e: any; hue: number; L: number; depth: number }[] = [];
const labels: { x: number; y: number; hue: number; name: string }[] = [];
for (let k = 0; k < K; k++) {
const hue = hueOf(k, K);
const P: { x: number; y: number; depth: number; wx: number; wy: number; wz: number }[][] = [];
for (let j = 0; j <= N; j++) { P[j] = []; for (let i = 0; i <= N; i++) P[j][i] = project(i / N - 0.5, j / N - 0.5, H(k, i, j), k); }
for (let j = 0; j < N; j++) for (let i = 0; i < N; i++) {
const a = P[j][i], b = P[j][i + 1], d = P[j + 1][i + 1], e = P[j + 1][i];
const ux = d.wx - a.wx, uy = d.wy - a.wy, uz = d.wz - a.wz, vx = b.wx - e.wx, vy = b.wy - e.wy, vz = b.wz - e.wz;
const nx = uy * vz - uz * vy, ny = uz * vx - ux * vz, nz = ux * vy - uy * vx, nm = Math.hypot(nx, ny, nz) || 1;
const L = clamp(0.32 + 0.62 * Math.abs((nx * LIGHT[0] + ny * LIGHT[1] + nz * LIGHT[2]) / nm));
quads.push({ a, b, d, e, hue, L, depth: (a.depth + b.depth + d.depth + e.depth) / 4 });
}
const lp = project(-0.5, 0.5, H(k, 0, N), k);
labels.push({ x: lp.x, y: lp.y, hue, name: namesRef.current[k] || `p${k}` });
}
quads.sort((p, q) => q.depth - p.depth);
for (const Q of quads) {
ctx.fillStyle = `hsla(${Q.hue},80%,${(26 + Q.L * 44).toFixed(0)}%,0.66)`;
ctx.strokeStyle = `hsla(${Q.hue},80%,82%,0.18)`; ctx.lineWidth = 0.6;
ctx.beginPath(); ctx.moveTo(Q.a.x, Q.a.y); ctx.lineTo(Q.b.x, Q.b.y); ctx.lineTo(Q.d.x, Q.d.y); ctx.lineTo(Q.e.x, Q.e.y); ctx.closePath();
ctx.fill(); ctx.stroke();
}
ctx.font = '11px var(--font-mono, monospace)'; ctx.textAlign = 'right'; ctx.textBaseline = 'middle';
for (const Lb of labels) { ctx.fillStyle = `hsla(${Lb.hue},85%,74%,0.95)`; ctx.fillText(Lb.name, Lb.x - 6, Lb.y); }
// probe column at the current input, marker on each slab from the LIVE output
const out = engine.getOutputs();
const gx = posRef.current[0] - 0.5, gy = posRef.current[1] - 0.5;
const top = project(gx, gy, 1.18, 0), bot = project(gx, gy, -0.18, K - 1);
ctx.strokeStyle = 'rgba(255,255,255,0.3)'; ctx.lineWidth = 1; ctx.setLineDash([3, 4]);
ctx.beginPath(); ctx.moveTo(top.x, top.y); ctx.lineTo(bot.x, bot.y); ctx.stroke(); ctx.setLineDash([]);
for (let k = 0; k < K; k++) {
const val = clamp(out[k] ?? 0.5);
const p = project(gx, gy, val, k), fl = project(gx, gy, 0, k);
ctx.strokeStyle = colCss(k, K, 0.5); ctx.lineWidth = 1;
ctx.beginPath(); ctx.moveTo(p.x, p.y); ctx.lineTo(fl.x, fl.y); ctx.stroke();
ctx.fillStyle = colCss(k, K, 1); ctx.shadowColor = colCss(k, K, 1); ctx.shadowBlur = 12;
ctx.beginPath(); ctx.arc(p.x, p.y, 6, 0, Math.PI * 2); ctx.fill(); ctx.shadowBlur = 0;
ctx.fillStyle = 'var(--bg, #0d0d0d)'; ctx.beginPath(); ctx.arc(p.x, p.y, 2.2, 0, Math.PI * 2); ctx.fill();
}
raf = requestAnimationFrame(draw);
};
raf = requestAnimationFrame(draw);
return () => { cancelAnimationFrame(raf); ro.disconnect(); };
}, [engine, OUT]);
// ---- drag to orbit (yaw + pitch; pitch up to π/2 = exact top-down) ----
const rotating = useRef(false);
const last = useRef({ x: 0, y: 0 });
return (
<div ref={wrapRef} style={{ position: 'absolute', inset: 0, overflow: 'hidden' }}>
<canvas
ref={canvasRef}
style={{ position: 'absolute', inset: 0, width: '100%', height: '100%', cursor: 'grab', touchAction: 'none' }}
onPointerDown={(e) => {
rotating.current = true;
(e.currentTarget as HTMLCanvasElement).setPointerCapture?.(e.pointerId);
last.current = { x: e.clientX, y: e.clientY };
}}
onPointerMove={(e) => {
if (!rotating.current) return;
yaw.current += (e.clientX - last.current.x) * 0.01;
pitch.current = Math.max(0.06, Math.min(Math.PI / 2, pitch.current + (e.clientY - last.current.y) * 0.008));
last.current = { x: e.clientX, y: e.clientY };
}}
onPointerUp={() => { rotating.current = false; }}
onPointerCancel={() => { rotating.current = false; }}
/>
<div style={{ position: 'absolute', top: 10, left: '50%', transform: 'translateX(-50%)', fontSize: 9, letterSpacing: '0.12em', color: 'var(--fg-dim)', pointerEvents: 'none', fontFamily: 'var(--font-mono)' }}>
PARAMETER SANDWICH · drag to rotate · top-down = the flat 2-D map
</div>
</div>
);
}

View file

@ -50,6 +50,19 @@ export function ModeIcon({ size = 18, style }: IconProps) {
); );
} }
/** Sandwich — stacked parameter-landscape slabs (the layer toggle). */
export function SandwichIcon({ size = 18, style }: IconProps) {
return svg(
size,
style,
<>
<path d="M3 6.5 12 3l9 3.5L12 10 3 6.5Z" />
<path d="M3 12 12 15.5 21 12" />
<path d="M3 17 12 20.5 21 17" />
</>,
);
}
/** Learning — a brain-ish node graph. */ /** Learning — a brain-ish node graph. */
export function LearningIcon({ size = 18, style }: IconProps) { export function LearningIcon({ size = 18, style }: IconProps) {
return svg( return svg(

View file

@ -164,9 +164,13 @@ export class EngineApi {
this.spine.setInput(x, y); this.spine.setInput(x, y);
} }
/** Set an arbitrary input vector (first two used as XY for the fixed 2→N MLP). */ /**
* Set the full composed input vector from the modular InputLayer. Axes 0,1
* are the primary controller; axes 2+ feed additional controllers (the WASM
* head is 4-wide = two 2-D controllers). Omitted axes stay neutral.
*/
setInputs(arr: ReadonlyArray<number>): void { setInputs(arr: ReadonlyArray<number>): void {
this.spine.setInput(arr[0] ?? 0.5, arr[1] ?? 0.5); this.spine.setInputs(arr);
} }
/** Live post-ML output vector (reused buffer — read, don't retain). */ /** Live post-ML output vector (reused buffer — read, don't retain). */

View file

@ -18,6 +18,7 @@
* Float32Array threaded through the output pipeline and handed to the backend. * Float32Array threaded through the output pipeline and handed to the backend.
*/ */
import { clamp } from './curves';
import { import {
defaultInputConfig, defaultInputConfig,
defaultInputState, defaultInputState,
@ -91,6 +92,11 @@ export class Spine implements EngineSink {
// Reused per-frame buffers — NO per-frame allocation in the hot path. // Reused per-frame buffers — NO per-frame allocation in the hot path.
private rawInput: [number, number] = [0.5, 0.5]; private rawInput: [number, number] = [0.5, 0.5];
// Full input vector fed to the MLP. Axes 0,1 = primary pad (post input
// pipeline); axes 2+ = secondary controller(s), or the neutral 0.5 when no
// second controller is active (single-controller mode). Sized to the WASM
// head's input arity; defaults all-neutral.
private inputs: number[] = [0.5, 0.5];
// Last raw input, so `EngineApi.process()` can re-tick after a weight change. // Last raw input, so `EngineApi.process()` can re-tick after a weight change.
lastRawX = 0.5; lastRawX = 0.5;
lastRawY = 0.5; lastRawY = 0.5;
@ -108,7 +114,11 @@ export class Spine implements EngineSink {
setState(patch: EngineStatePatch): void { setState(patch: EngineStatePatch): void {
let changed = false; let changed = false;
const s = this.state_; const s = this.state_;
if (patch.inputSize !== undefined && patch.inputSize !== s.inputSize) { s.inputSize = patch.inputSize; changed = true; } if (patch.inputSize !== undefined && patch.inputSize !== s.inputSize) {
s.inputSize = patch.inputSize;
this.resizeInputs_(patch.inputSize);
changed = true;
}
if (patch.outputSize !== undefined && patch.outputSize !== s.outputSize) { if (patch.outputSize !== undefined && patch.outputSize !== s.outputSize) {
s.outputSize = patch.outputSize; s.outputSize = patch.outputSize;
// Resize hot buffers to the resolved output size. // Resize hot buffers to the resolved output size.
@ -155,6 +165,16 @@ export class Spine implements EngineSink {
if (this.routedBuf === null || this.routedBuf.length !== iml.architecture.outputSize) { if (this.routedBuf === null || this.routedBuf.length !== iml.architecture.outputSize) {
this.routedBuf = new Float32Array(iml.architecture.outputSize); this.routedBuf = new Float32Array(iml.architecture.outputSize);
} }
this.resizeInputs_(iml.architecture.inputSize);
}
/** Resize the full input vector to the WASM head arity, neutral-filling. */
private resizeInputs_(size: number): void {
if (size < 1) size = 1;
if (this.inputs.length === size) return;
const next = new Array<number>(size).fill(0.5);
for (let i = 0; i < Math.min(size, this.inputs.length); i++) next[i] = this.inputs[i]!;
this.inputs = next;
} }
setBackendSend(backendSend: BackendSend | null): void { setBackendSend(backendSend: BackendSend | null): void {
@ -169,6 +189,22 @@ export class Spine implements EngineSink {
* Returns the routed buffer (live, reused — do not retain across calls). * Returns the routed buffer (live, reused — do not retain across calls).
*/ */
setInput(x: number, y: number): Float32Array | null { setInput(x: number, y: number): Float32Array | null {
return this.tick_(x, y, null);
}
/**
* Drive the full composed input vector (from the modular InputLayer). Axes
* 0,1 are the primary controller (run through the input pipeline); axes 2+
* are secondary controllers, written through with a plain [0,1] clamp. Any
* axis the caller omits keeps its previous value (neutral 0.5 by default), so
* single-controller mode — where only axes 0,1 are supplied — leaves the
* second pair pinned at the neutral value.
*/
setInputs(arr: ReadonlyArray<number>): Float32Array | null {
return this.tick_(arr[0] ?? 0.5, arr[1] ?? 0.5, arr);
}
private tick_(x: number, y: number, arr: ReadonlyArray<number> | null): Float32Array | null {
const iml = this.iml; const iml = this.iml;
if (!iml) return null; if (!iml) return null;
@ -176,17 +212,25 @@ export class Spine implements EngineSink {
const dt = this.lastTickMs > 0 ? (now - this.lastTickMs) / 1000 : 1 / 60; const dt = this.lastTickMs > 0 ? (now - this.lastTickMs) / 1000 : 1 / 60;
this.lastTickMs = now; this.lastTickMs = now;
// 1. processed (pure input pipeline) // 1. processed (pure input pipeline) — primary controller, axes 0,1.
this.rawInput[0] = x; this.rawInput[0] = x;
this.rawInput[1] = y; this.rawInput[1] = y;
this.lastRawX = x; this.lastRawX = x;
this.lastRawY = y; this.lastRawY = y;
const proc = processInput(this.rawInput, this.inputConfig, this.inputState, dt); const proc = processInput(this.rawInput, this.inputConfig, this.inputState, dt);
this.inputState = proc.state; this.inputState = proc.state;
this.inputs[0] = proc.x;
this.inputs[1] = proc.y;
// Secondary axes (2+): pass through with a [0,1] clamp. Omitted axes keep
// their prior value (neutral 0.5 in single-controller mode).
if (arr) {
for (let i = 2; i < this.inputs.length; i++) {
if (i < arr.length) this.inputs[i] = clamp(arr[i]!, 0, 1);
}
}
// 2. ml (inference into the reused buffer; no alloc) // 2. ml (inference into the reused buffer; no alloc)
iml.setInput(0, proc.x); for (let i = 0; i < this.inputs.length; i++) iml.setInput(i, this.inputs[i]!);
iml.setInput(1, proc.y);
iml.processInto(this.mlBuf); iml.processInto(this.mlBuf);
// Mirror to liveOutputs for imperative reads + bump. // Mirror to liveOutputs for imperative reads + bump.
this.liveOutputs.set(this.mlBuf.subarray(0, this.liveOutputs.length)); this.liveOutputs.set(this.mlBuf.subarray(0, this.liveOutputs.length));

View file

@ -13,24 +13,19 @@
* its value), but routing everything through one compose path keeps sources * its value), but routing everything through one compose path keeps sources
* composable and the channel layout coherent. * composable and the channel layout coherent.
* *
* ── Arity mismatch (the WASM reshape TODO) ────────────────────────────────── * ── Arity: 4-input head ─────────────────────────────────────────────────────
* The browser WASM is fixed at MLP<2, …, 126> — a TWO-input head. When the * The browser WASM head is MLP<4, …, 126> — TWO 2-D controllers. The primary
* composed vector has > 2 axes (double-stick gamepad = 4, MIDI learn-map = many) * controller (on-screen pad) occupies axes 0,1; an optional second controller
* we must reduce to 2 to feed today's engine. We do NOT fake a wider net. * (gamepad) occupies axes 2,3. `compose()` passes axes through 1:1 when the
* head is at least as wide as the composed axis count (the common cases —
* single pad = 2, pad + single-stick gamepad = 4), padding any unused axes with
* the neutral 0.5. Single-controller mode is therefore just "no second source":
* axes 2,3 stay neutral, slicing the 4-D manifold at the centre of controller 2.
* *
* chosen reduction (this pass): pairwise BLEND. * Over-subscription (composed axes > head width — e.g. pad + dual-stick gamepad
* inX = mean(axis[0], axis[2], axis[4], …) // even axes * = 6, or a large MIDI learn-map) still falls back to a chunked-mean blend; the
* inY = mean(axis[1], axis[3], axis[5], …) // odd axes * net is never rebuilt. That blend is the documented lossy path for unusual
* so a single stick passes straight through (axis0→X, axis1→Y), a double * source combinations, not the default.
* stick averages L/R into one XY, and MIDI axes fold into X/Y by parity.
*
* TODO(workstream F, docs/redesign/inputs-spec.md — "multiple WASM modules +
* warm-start"): the real fix is to (re)load a WASM module whose MLP input arity
* matches the composed axis count and warm-start its weights from the prior net,
* so every axis gets its own genuine input dimension instead of being blended.
* That is a larger build (multiple .wasm artefacts or a runtime-variadic head)
* and is deliberately deferred — this layer is wired so that swapping the
* reduction for a true reshape is a localised change in `compose()`.
*/ */
import type { InputAction, InputSource } from './types'; import type { InputAction, InputSource } from './types';
@ -147,15 +142,15 @@ export class InputLayer {
} }
/** /**
* Reduce the composed N-axis vector to the engine's input arity. * Map the composed N-axis vector to the engine's input arity.
* *
* For the fixed 2-input WASM, fold by parity (even→X, odd→Y) via mean. If a * Common case: the 4-input head is at least as wide as the composed axes, so
* future multi-module engine reports inputSize >= n, this passes axes through * pass through 1:1 and pad unused axes with the neutral 0.5. Over-subscription
* 1:1 (truncated/padded) — the seam where the real reshape lands. * (axes > head width) folds by parity / chunked mean — the lossy fallback.
*/ */
private compose(n: number, inputSize: number): number[] { private compose(n: number, inputSize: number): number[] {
if (inputSize >= n) { if (inputSize >= n) {
// True passthrough path (future multi-module head). Pad with 0.5. // Passthrough: each axis gets its own input dimension. Pad with 0.5.
const out = new Array<number>(inputSize); const out = new Array<number>(inputSize);
for (let i = 0; i < inputSize; i++) out[i] = i < n ? this.vector[i] : 0.5; for (let i = 0; i < inputSize; i++) out[i] = i < n ? this.vector[i] : 0.5;
return out; return out;

View file

@ -141,7 +141,7 @@ class ModeBase {
// Run an inference at default inputs so engine has params on first // Run an inference at default inputs so engine has params on first
// process() call, even if no input has been touched. // process() call, even if no input has been touched.
for (std::size_t i = 0u; i < NInputs; ++i) { for (std::size_t i = 0u; i < NInputs; ++i) {
ml_.set_input(i, input_channels_[i]); ml_.set_input(i, effective_input(i));
} }
ml_.process(); ml_.process();
if constexpr (kRouteOutputsToEngine) { if constexpr (kRouteOutputsToEngine) {
@ -160,13 +160,42 @@ class ModeBase {
input_dirty_ = true; input_dirty_ = true;
} }
// ---- Input neutralization (single/double controller toggle) ----
//
// A pinned channel feeds `pin_value_` (neutral, default 0.5) to the MLP
// instead of its live value, without rebuilding/resizing the network.
// Glue toggles which channels are pinned (e.g. single-joystick mode pins
// the second 2D controller's two channels). The stored live value is left
// untouched, so unpinning resumes from the controller's current position.
NISPS_FORCE_INLINE void set_input_pinned(std::size_t idx, bool pinned) noexcept {
if (idx >= NInputs) return;
input_pinned_[idx] = pinned;
input_dirty_ = true;
}
NISPS_FORCE_INLINE bool is_input_pinned(std::size_t idx) const noexcept {
return idx < NInputs && input_pinned_[idx];
}
NISPS_FORCE_INLINE void set_pin_value(float v) noexcept {
if (v < 0.f) v = 0.f;
else if (v > 1.f) v = 1.f;
pin_value_ = v;
input_dirty_ = true;
}
float pin_value() const noexcept { return pin_value_; }
// Effective value fed to the MLP for channel i (pin override applied).
NISPS_FORCE_INLINE float effective_input(std::size_t i) const noexcept {
return input_pinned_[i] ? pin_value_ : input_channels_[i];
}
NISPS_HOT void tick_control() noexcept { NISPS_HOT void tick_control() noexcept {
if constexpr (requires(Derived& d) { d.on_pre_inference(); }) { if constexpr (requires(Derived& d) { d.on_pre_inference(); }) {
static_cast<Derived&>(*this).on_pre_inference(); static_cast<Derived&>(*this).on_pre_inference();
} }
// Forward (possibly Derived-mutated) channels into the MLP. // Forward (possibly Derived-mutated) channels into the MLP, applying
// the per-channel pin override.
for (std::size_t i = 0u; i < NInputs; ++i) { for (std::size_t i = 0u; i < NInputs; ++i) {
ml_.set_input(i, input_channels_[i]); ml_.set_input(i, effective_input(i));
} }
ml_.process(); ml_.process();
if constexpr (kRouteOutputsToEngine) { if constexpr (kRouteOutputsToEngine) {
@ -235,6 +264,8 @@ class ModeBase {
EngineT engine_{}; EngineT engine_{};
MLPType ml_; MLPType ml_;
std::array<float, NInputs> input_channels_{}; std::array<float, NInputs> input_channels_{};
std::array<bool, NInputs> input_pinned_{}; // false => live
float pin_value_ = 0.5f; // neutral
bool input_dirty_ = false; bool input_dirty_ = false;
std::size_t voice_space_idx_ = 0u; std::size_t voice_space_idx_ = 0u;
RingBuffer<ControlEvent, kModeEventBufferSize> events_{}; RingBuffer<ControlEvent, kModeEventBufferSize> events_{};

View file

@ -32,10 +32,13 @@ The MLP class template is parametrised on `(input_size, hidden1, hidden2,
hidden3, output_size)`. WASM cannot recompile templates at runtime, so hidden3, output_size)`. WASM cannot recompile templates at runtime, so
this build instantiates exactly ONE configuration: this build instantiates exactly ONE configuration:
nisps::ml::MLP<2, 10, 14, 18, 126> nisps::ml::MLP<4, 10, 14, 18, 126>
That serves the playground use case (2-D joystick → up to 126 synth That serves the manifold/playground use case: two 2-D controllers (primary
parameters). `nisps_ml_create()` accepts caller-supplied dimensions for pad on axes 0,1; optional second controller — e.g. gamepad — on axes 2,3)
→ up to 126 synth parameters. Single-controller mode pins axes 2,3 to the
neutral 0.5 on the JS side, so the net is never rebuilt when toggling
between one and two controllers. `nisps_ml_create()` accepts caller-supplied dimensions for
forward compatibility but currently ignores them — see comment at the top forward compatibility but currently ignores them — see comment at the top
of `bindings.cpp`. The schemas in `schemas/modes/*.json` use up to of `bindings.cpp`. The schemas in `schemas/modes/*.json` use up to
`output_size=126`; modes whose output_size is < 126 simply ignore the `output_size=126`; modes whose output_size is < 126 simply ignore the

View file

@ -5,24 +5,26 @@
// 2. AudioWorklet processor (playground/src/audio/worklet/...) — engine // 2. AudioWorklet processor (playground/src/audio/worklet/...) — engine
// calls. (Each instance owns its own WASM module instance.) // calls. (Each instance owns its own WASM module instance.)
// //
// FIXED-ARCHITECTURE LIMITATION (VERY IMPORTANT) // COMPILE-TIME ARCHITECTURE
// ---------------------------------------------- // -------------------------
// The C++ MLP class is templated on layer sizes (architecture.md §4.1, §6.2). // The C++ MLP class is templated on layer sizes (architecture.md §4.1, §6.2).
// We instantiate ONE concrete configuration here: // We instantiate ONE concrete configuration here:
// //
// using DefaultMLP = nisps::ml::MLP<2, 10, 14, 18, 126>; // using DefaultMLP = nisps::ml::MLP<4, 10, 14, 18, 126>;
// //
// This was chosen as the union of the playground use case (2-D joystick → // FOUR inputs = two 2-D controllers (primary on-screen pad on axes 0,1; an
// 126 synth params) and the largest hidden-layer footprint that still fits // optional second controller — gamepad — on axes 2,3). Single-controller mode
// firmware budgets. `nisps_ml_create()` accepts caller-supplied input_size, // pins the second pair to the neutral value 0.5 on the JS side (see
// output_size, hidden[], n_hidden but ONLY validates them against the // manifold/src/engine/spine.ts), so the network is never rebuilt/resized when
// compile-time defaults — extra inputs/outputs are clipped at the boundary. // toggling between one and two controllers. 126 outputs covers C15 and every
// If the caller passes incompatible dimensions we still create the module: // current schema.
// extra inputs are zero-padded, extra outputs are truncated, and the
// hidden-layer override is silently ignored.
// //
// Future work: ship multiple WASM modules (one per common architecture) or // `nisps_ml_create()` accepts caller-supplied input_size, output_size,
// rebuild on demand. See architecture.md "open questions" — Stream 7 punts. // hidden[], n_hidden but ONLY validates them against the compile-time
// defaults — extra inputs/outputs are clipped at the boundary. If the caller
// passes incompatible dimensions we still create the module: extra inputs are
// zero-padded, extra outputs are truncated, and the hidden-layer override is
// silently ignored. The TS side reads the real dims back via nisps_ml_describe.
// //
// WIRE FORMAT FOR WEIGHTS // WIRE FORMAT FOR WEIGHTS
// ----------------------- // -----------------------
@ -78,14 +80,15 @@ namespace {
// Compile-time default architecture. See header comment. // Compile-time default architecture. See header comment.
// //
// Choice rationale: // Choice rationale:
// * 2 inputs — playground virtual joystick (X, Y). // * 4 inputs — two 2-D controllers (primary pad axes 0,1; optional
// second controller axes 2,3). Single-controller mode pins axes 2,3 to 0.5.
// * [10, 14, 18] hidden — covers the largest schema layouts in // * [10, 14, 18] hidden — covers the largest schema layouts in
// `schemas/modes/*.json` (channel_strip variants, verb_fx, breakor, // `schemas/modes/*.json` (channel_strip variants, verb_fx, breakor,
// elysiamorf, memlcelium). // elysiamorf, memlcelium).
// * 126 outputs — enough for the C15 mode and any current schema. // * 126 outputs — enough for the C15 mode and any current schema.
// //
// The MLP also has dataset slots, loss history etc. — see mlp.hpp. // The MLP also has dataset slots, loss history etc. — see mlp.hpp.
using DefaultMLP = nisps::ml::MLP<2u, 10u, 14u, 18u, 126u>; using DefaultMLP = nisps::ml::MLP<4u, 10u, 14u, 18u, 126u>;
constexpr std::size_t kDefaultInputs = DefaultMLP::kInput; constexpr std::size_t kDefaultInputs = DefaultMLP::kInput;
constexpr std::size_t kDefaultOutputs = DefaultMLP::kOutput; constexpr std::size_t kDefaultOutputs = DefaultMLP::kOutput;

Binary file not shown.

View file

@ -82,3 +82,12 @@ set -x
echo "[build-wasm] wrote $OUT/nisps.js + $OUT/nisps.wasm" echo "[build-wasm] wrote $OUT/nisps.js + $OUT/nisps.wasm"
ls -lh "$OUT/nisps.js" "$OUT/nisps.wasm" ls -lh "$OUT/nisps.js" "$OUT/nisps.wasm"
# Mirror into the manifold app's public/ — it serves its own copy (and its
# deploy webhook builds manifold/), so both must stay in lockstep with the
# parity-checked artifacts above.
MANIFOLD_OUT="$ROOT/manifold/public"
if [[ -d "$MANIFOLD_OUT" ]]; then
cp "$OUT/nisps.js" "$OUT/nisps.wasm" "$MANIFOLD_OUT/"
echo "[build-wasm] mirrored nisps.{js,wasm} → $MANIFOLD_OUT"
fi

View file

@ -13,7 +13,7 @@
// //
// What we cover // What we cover
// ------------- // -------------
// 1. ML: seed=42, draw_weights(0.5), set_input(0.25, 0.75), process. // 1. ML: seed=42, draw_weights(0.5), set_input(0.25, 0.75, 0.40, 0.60), process.
// → 126 outputs + 12 weights sampled at known offsets. // → 126 outputs + 12 weights sampled at known offsets.
// 2. ML training: 3 examples added, train(0.3, 50, 0), capture loss + outputs. // 2. ML training: 3 examples added, train(0.3, 50, 0), capture loss + outputs.
// 3. PAFSynth engine: seed-equivalent setup (params=0.5), 128-sample run on // 3. PAFSynth engine: seed-equivalent setup (params=0.5), 128-sample run on
@ -21,7 +21,7 @@
// 4. ChannelStrip engine: identical methodology. // 4. ChannelStrip engine: identical methodology.
// //
// We use the EXACT SAME compile-time MLP architecture as the WASM build: // We use the EXACT SAME compile-time MLP architecture as the WASM build:
// MLP<2, 10, 14, 18, 126> // MLP<4, 10, 14, 18, 126>
// //
// Output blob format // Output blob format
// ------------------ // ------------------
@ -62,7 +62,7 @@
namespace { namespace {
using ParityMLP = nisps::ml::MLP<2u, 10u, 14u, 18u, 126u>; using ParityMLP = nisps::ml::MLP<4u, 10u, 14u, 18u, 126u>;
// The WASM bindings (nisps/wasm/bindings.cpp) sign-extend the 32-bit JS // The WASM bindings (nisps/wasm/bindings.cpp) sign-extend the 32-bit JS
// seed via `s ^ (s << 32)`. To get bit-equal output between native and // seed via `s ^ (s << 32)`. To get bit-equal output between native and
@ -73,6 +73,9 @@ constexpr std::uint64_t kSeed = static_cast<std::uint64_t>(kSeed32)
^ (static_cast<std::uint64_t>(kSeed32) << 32); ^ (static_cast<std::uint64_t>(kSeed32) << 32);
constexpr float kInputX = 0.25f; constexpr float kInputX = 0.25f;
constexpr float kInputY = 0.75f; constexpr float kInputY = 0.75f;
// Second controller (axes 2,3) — exercised so the 4-input head is fully probed.
constexpr float kInputZ = 0.40f;
constexpr float kInputW = 0.60f;
constexpr float kSampleRate = 48000.0f; constexpr float kSampleRate = 48000.0f;
constexpr std::size_t kSynthFrames = 128u; constexpr std::size_t kSynthFrames = 128u;
@ -118,6 +121,8 @@ int main(int argc, char** argv) {
mlp.draw_weights(0.5f); mlp.draw_weights(0.5f);
mlp.set_input(0u, kInputX); mlp.set_input(0u, kInputX);
mlp.set_input(1u, kInputY); mlp.set_input(1u, kInputY);
mlp.set_input(2u, kInputZ);
mlp.set_input(3u, kInputW);
mlp.process(); mlp.process();
{ {
const auto outs = mlp.outputs(); const auto outs = mlp.outputs();
@ -133,8 +138,8 @@ int main(int argc, char** argv) {
} }
// ---- Stage 2: training ---- // ---- Stage 2: training ----
constexpr std::array<std::array<float, 2u>, 3u> features = {{ constexpr std::array<std::array<float, 4u>, 3u> features = {{
{{0.1f, 0.9f}}, {{0.5f, 0.5f}}, {{0.9f, 0.1f}}, {{0.1f, 0.9f, 0.3f, 0.7f}}, {{0.5f, 0.5f, 0.5f, 0.5f}}, {{0.9f, 0.1f, 0.7f, 0.3f}},
}}; }};
auto label_for = [](std::size_t i) { auto label_for = [](std::size_t i) {
std::array<float, 126u> out{}; std::array<float, 126u> out{};
@ -146,13 +151,15 @@ int main(int argc, char** argv) {
}; };
for (std::size_t i = 0; i < features.size(); ++i) { for (std::size_t i = 0; i < features.size(); ++i) {
const auto label = label_for(i); const auto label = label_for(i);
mlp.add_example(std::span<const float>(features[i].data(), 2u), mlp.add_example(std::span<const float>(features[i].data(), 4u),
std::span<const float>(label.data(), 126u)); std::span<const float>(label.data(), 126u));
} }
const float final_loss = mlp.train(0.3f, 50u, 0.0f); const float final_loss = mlp.train(0.3f, 50u, 0.0f);
mlp.set_input(0u, kInputX); mlp.set_input(0u, kInputX);
mlp.set_input(1u, kInputY); mlp.set_input(1u, kInputY);
mlp.set_input(2u, kInputZ);
mlp.set_input(3u, kInputW);
mlp.process(); mlp.process();
{ {
const auto outs = mlp.outputs(); const auto outs = mlp.outputs();

View file

@ -43,6 +43,8 @@ const VERSION = 3; // v3 adds stage 5d (ExploreAndPlace lifecycle)
const SEED = 42 >>> 0; const SEED = 42 >>> 0;
const INPUT_X = 0.25; const INPUT_X = 0.25;
const INPUT_Y = 0.75; const INPUT_Y = 0.75;
const INPUT_Z = 0.40; // second controller axes 2,3 (must match parity_check.cpp)
const INPUT_W = 0.60;
const SAMPLE_RATE = 48000; const SAMPLE_RATE = 48000;
const SYNTH_FRAMES = 128; const SYNTH_FRAMES = 128;
const PROBE_IDX = [0, 5, 19, 31, 73, 137, 251, 491, 999, 1583, 2401, 3289]; const PROBE_IDX = [0, 5, 19, 31, 73, 137, 251, 491, 999, 1583, 2401, 3289];
@ -191,8 +193,8 @@ async function main() {
api.describe(dimsBuf); api.describe(dimsBuf);
const dims = new Int32Array(Module.HEAP32.buffer, dimsBuf, 6).slice(); const dims = new Int32Array(Module.HEAP32.buffer, dimsBuf, 6).slice();
api.free(dimsBuf); api.free(dimsBuf);
// Expect: [2, 10, 14, 18, 126, 4] // Expect: [inputs, h1, h2, h3, outputs, numLayers] = [4, 10, 14, 18, 126, 4]
const expectedDims = [2, 10, 14, 18, 126, 4]; const expectedDims = [4, 10, 14, 18, 126, 4];
for (let i = 0; i < expectedDims.length; ++i) { for (let i = 0; i < expectedDims.length; ++i) {
if (dims[i] !== expectedDims[i]) { if (dims[i] !== expectedDims[i]) {
console.error(`[parity_wasm] WASM build has dim[${i}]=${dims[i]}, native expected ${expectedDims[i]}`); console.error(`[parity_wasm] WASM build has dim[${i}]=${dims[i]}, native expected ${expectedDims[i]}`);
@ -203,10 +205,12 @@ async function main() {
const N_OUT = dims[4]; const N_OUT = dims[4];
// --- Stage 1: ML inference --- // --- Stage 1: ML inference ---
const ml = api.create(2, N_OUT, 0, 0, SEED); const ml = api.create(4, N_OUT, 0, 0, SEED);
api.drawWeights(ml, 0.5); api.drawWeights(ml, 0.5);
api.setInput(ml, 0, INPUT_X); api.setInput(ml, 0, INPUT_X);
api.setInput(ml, 1, INPUT_Y); api.setInput(ml, 1, INPUT_Y);
api.setInput(ml, 2, INPUT_Z);
api.setInput(ml, 3, INPUT_W);
api.process(ml); api.process(ml);
const outsStage1 = getOutputsCopy(api, ml, N_OUT); const outsStage1 = getOutputsCopy(api, ml, N_OUT);
@ -216,9 +220,9 @@ async function main() {
// --- Stage 2: training --- // --- Stage 2: training ---
const features = [ const features = [
[0.1, 0.9], [0.1, 0.9, 0.3, 0.7],
[0.5, 0.5], [0.5, 0.5, 0.5, 0.5],
[0.9, 0.1], [0.9, 0.1, 0.7, 0.3],
]; ];
const labelFor = (i) => { const labelFor = (i) => {
const out = new Float32Array(N_OUT); const out = new Float32Array(N_OUT);
@ -226,12 +230,14 @@ async function main() {
for (let j = 0; j < N_OUT; ++j) out[j] = a + 0.005 * j; for (let j = 0; j < N_OUT; ++j) out[j] = a + 0.005 * j;
return out; return out;
}; };
const featBuf = api.malloc(2 * 4); const featBuf = api.malloc(4 * 4);
const featF32 = new Float32Array(api.HEAPF32.buffer, featBuf, 2); const featF32 = new Float32Array(api.HEAPF32.buffer, featBuf, 4);
const labelBuf = api.malloc(N_OUT * 4); const labelBuf = api.malloc(N_OUT * 4);
for (let i = 0; i < features.length; ++i) { for (let i = 0; i < features.length; ++i) {
featF32[0] = features[i][0]; featF32[0] = features[i][0];
featF32[1] = features[i][1]; featF32[1] = features[i][1];
featF32[2] = features[i][2];
featF32[3] = features[i][3];
const label = labelFor(i); const label = labelFor(i);
new Float32Array(api.HEAPF32.buffer, labelBuf, N_OUT).set(label); new Float32Array(api.HEAPF32.buffer, labelBuf, N_OUT).set(label);
api.addExample(ml, featBuf, labelBuf); api.addExample(ml, featBuf, labelBuf);
@ -243,6 +249,8 @@ async function main() {
api.setInput(ml, 0, INPUT_X); api.setInput(ml, 0, INPUT_X);
api.setInput(ml, 1, INPUT_Y); api.setInput(ml, 1, INPUT_Y);
api.setInput(ml, 2, INPUT_Z);
api.setInput(ml, 3, INPUT_W);
api.process(ml); api.process(ml);
const outsStage2 = getOutputsCopy(api, ml, N_OUT); const outsStage2 = getOutputsCopy(api, ml, N_OUT);
@ -301,6 +309,8 @@ async function main() {
api.feedbackUndo(ml); // pop nudge api.feedbackUndo(ml); // pop nudge
api.setInput(ml, 0, INPUT_X); api.setInput(ml, 0, INPUT_X);
api.setInput(ml, 1, INPUT_Y); api.setInput(ml, 1, INPUT_Y);
api.setInput(ml, 2, INPUT_Z);
api.setInput(ml, 3, INPUT_W);
api.process(ml); api.process(ml);
api.feedbackLike(ml); // begin place: freeze scratchpad output api.feedbackLike(ml); // begin place: freeze scratchpad output
{ {

View file

@ -88,6 +88,62 @@ NISPS_TEST(paf_synth_mode_input_clamping) {
m.set_input(99u, 0.5f); m.set_input(99u, 0.5f);
} }
NISPS_TEST(paf_synth_mode_pinned_inputs_neutralized) {
// Single-joystick mode pins the second 2D controller's channels (2,3).
// A pinned channel must feed the neutral pin value (default 0.5) to the
// MLP regardless of its live value, and pinning must not resize the model.
PAFSynthMode m;
m.setup(48000.f);
NISPS_EXPECT(m.pin_value() == 0.5f);
// Reference: channels 2,3 driven live to 0.5 (the neutral value).
m.set_input(0, 0.2f);
m.set_input(1, 0.8f);
m.set_input(2, 0.5f);
m.set_input(3, 0.5f);
m.tick_control();
std::array<float, 33> ref{};
{
const auto outs = m.ml().outputs();
for (std::size_t i = 0u; i < outs.size(); ++i) ref[i] = outs[i];
}
// Pin channels 2,3, then drive them to arbitrary values. Output must match
// the reference (they are neutralized to 0.5).
m.set_input_pinned(2, true);
m.set_input_pinned(3, true);
NISPS_EXPECT(m.is_input_pinned(2));
m.set_input(2, 0.0f);
m.set_input(3, 1.0f);
m.tick_control();
for (std::size_t i = 0u; i < 33u; ++i) {
NISPS_EXPECT(std::fabs(m.ml().outputs()[i] - ref[i]) < 1e-6f);
}
// The active controller (channels 0,1) still affects the output.
m.set_input(0, 0.9f);
m.tick_control();
bool changed = false;
for (std::size_t i = 0u; i < 33u; ++i) {
if (std::fabs(m.ml().outputs()[i] - ref[i]) > 1e-6f) { changed = true; break; }
}
NISPS_EXPECT(changed);
// Unpinning restores sensitivity on channels 2,3.
m.set_input_pinned(2, false);
m.set_input_pinned(3, false);
m.set_input(0, 0.2f); // restore active channels to the reference pose
m.set_input(1, 0.8f);
m.set_input(2, 0.0f);
m.set_input(3, 1.0f);
m.tick_control();
bool differs = false;
for (std::size_t i = 0u; i < 33u; ++i) {
if (std::fabs(m.ml().outputs()[i] - ref[i]) > 1e-6f) { differs = true; break; }
}
NISPS_EXPECT(differs);
}
NISPS_TEST(paf_synth_mode_engine_and_ml_accessors) { NISPS_TEST(paf_synth_mode_engine_and_ml_accessors) {
PAFSynthMode m; PAFSynthMode m;
m.setup(48000.f); m.setup(48000.f);