Compare commits
4 commits
main
...
clean-liza
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
3e6757dea8 | ||
|
|
9e4c751781 | ||
|
|
ebb953a13f | ||
|
|
e8f0866b47 |
22 changed files with 1109 additions and 79 deletions
|
|
@ -58,7 +58,8 @@ firmware/MEMLNaut-NISPS/
|
|||
│ ├── midi_io.hpp # MIDI in → mode handlers; drain ControlEvent ring → MIDI UART
|
||||
│ ├── mode_select.hpp # type aliases firmware mode name → nisps::modes::*Mode
|
||||
│ ├── input_router.hpp # wire_inputs() entry point
|
||||
│ └── output_router.hpp # drain_outputs() entry point
|
||||
│ ├── output_router.hpp # drain_outputs() entry point
|
||||
│ └── settings_view.hpp # wire_settings(): TFT/rotary menu (Joystick Dual/Single for 4-in modes)
|
||||
└── src/{memllib,daisysp,nisps} # symlinks (Arduino-CLI sketch tree convention)
|
||||
```
|
||||
|
||||
|
|
@ -118,9 +119,9 @@ Two WASM instances at runtime:
|
|||
1. **Main thread** (`playground/src/ml/wasm-iml.ts`): ML inference + sync training + RL primitives. Update store after each call. Async training via disposable Web Worker (`wasm-worker.ts`).
|
||||
2. **AudioWorklet** (`playground/src/audio/worklet/nisps-processor.ts`): runs engine `process_block` per audio block. Loads `nisps.wasm` directly via `WebAssembly.compile` (no Emscripten glue in worklet). Bytes posted from main thread.
|
||||
|
||||
C API is in `nisps/wasm/bindings.cpp`. Build: `bash scripts/build-wasm.sh` (~94KB output to `playground/public/`).
|
||||
C API is in `nisps/wasm/bindings.cpp`. Build: `bash scripts/build-wasm.sh` (~100KB output, mirrored to `playground/public/` and `manifold/public/`).
|
||||
|
||||
The WASM target is fixed at `MLP<2, 10, 14, 18, 126>`. Modes with smaller `output_size` use the first N outputs only.
|
||||
The WASM target is built as `MLP<4, 10, 14, 18, 126>` — 4 inputs = two 2-D controllers (primary pad on axes 0,1; optional second controller, e.g. gamepad, on axes 2,3). Single-controller mode pins axes 2,3 to the neutral 0.5 on the JS side (no rebuild). Modes with smaller `output_size` use the first N outputs only.
|
||||
|
||||
### Known limitations
|
||||
|
||||
|
|
|
|||
3
MAP.md
3
MAP.md
|
|
@ -23,6 +23,7 @@ MEMLNaut-NISPS — Neural Interactive Shaping of Parameter Spaces. One C++20 cod
|
|||
- `mode_select.hpp` — type aliases mapping firmware mode identifiers to `nisps::modes::*Mode` C++ types. Build script rewrites the active line. Includes the six `MEMLNautModeExtSynth*` external-synth variants (one per device template in `nisps/midi`, e.g. `MEMLNautModeExtSynthSub37`). Also defines the `MEMLNautModeSelfTest` pseudo-variant (tag type) + the `NISPS_ST_*`/`NISPS_ST_CAT` token-paste macros the `.ino` uses to compute `NISPS_SELFTEST`. Note: `src/nisps/` exposes each referenced top-level nisps subdir as a symlink — `midi` was added alongside `core/dsp/engines/ml/modes`.
|
||||
- `selftest.hpp` — standalone guided hardware self-test rig (`SelfTest` variant; no engine/ML). Step-driven state machine on a `SelfTestView`: TFT prompts the operator through every control, auto-advances on detection, encoder-press skips. Ends with optional L/R/BOTH sine-sweep headphone check (core 1 block callback) + MIDI loopback-cable test. Lives firmware-side (touches TFT + raw pins) so it stays out of platform-agnostic `nisps/`.
|
||||
- `input_router.hpp`, `output_router.hpp` — top-level `wire_inputs()` / `drain_outputs()` entry points.
|
||||
- `settings_view.hpp` — `wire_settings(mode)`: adds on-device settings views to the MEMLNaut display carousel (TFT + rotary encoder). Joystick Dual/Single toggle for the 4-input ("two 2-D joystick") modes — "Single" pins ML input channels 2,3 to neutral via `ModeBase::set_input_pinned` (no net rebuild). Registered in the `.ino` after `addSystemInfoView()`.
|
||||
- `firmware/MEMLNaut-NISPS/src/{memllib,daisysp,nisps}` — symlinks (Arduino-CLI requires sketch-tree includes; preprocessor refuses `..` in headers).
|
||||
- `firmware/README.md` — structure + build instructions.
|
||||
|
||||
|
|
@ -138,7 +139,7 @@ the "BUILD DELTAS" block at the top of `vcv/SPEC.md`). `src/MEMLNaut.cpp` (modul
|
|||
- C++ identifiers: `PascalCase` types, `snake_case` functions/variables, `kPascalCase` constexpr. JSON keys `snake_case`. TS types `PascalCase`, components `PascalCase.tsx`, modules `kebab-case.ts`.
|
||||
- `Curve` enum lives in `nisps/core/math.hpp` (lowercase: `linear/exp/log/square/sqrt/sigmoid/cubic`); generated mode headers re-export via `using Curve = ::nisps::Curve;`. TS mirror at `playground/src/output/curves.ts` with same names.
|
||||
- Modes are TSX components composed of primitives; mode parameter contracts are JSON schemas with codegen → C++/TS types. **No declarative JSON UI.**
|
||||
- WASM and firmware share the same C++; WASM is fixed at `MLP<2, 10, 14, 18, 126>` and modes use a slice of outputs based on schema's `output_size`.
|
||||
- WASM and firmware share the same C++; WASM is built as `MLP<4, 10, 14, 18, 126>` (4 inputs = two 2-D controllers; single-controller mode pins axes 2,3 to 0.5 in JS) and modes use a slice of outputs based on schema's `output_size`.
|
||||
- Cross-platform parity: `scripts/parity-check.sh` enforces native vs WASM agreement within 1e-5.
|
||||
|
||||
## Gotchas
|
||||
|
|
|
|||
484
assets/media/learn/index.html
Normal file
484
assets/media/learn/index.html
Normal file
|
|
@ -0,0 +1,484 @@
|
|||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
<title>Manifold — the parameter landscape</title>
|
||||
<style>
|
||||
:root{
|
||||
--bg:#0d0d0d; --bg-1:#141414; --bg-2:#1c1c1c; --bg-3:#242424;
|
||||
--fg:#e8e8e8; --fg-mute:#9a9a9a; --fg-dim:#5a5a5a;
|
||||
--line:#2a2a2a; --line-strong:#3a3a3a;
|
||||
--accent:#ff6a00; --accent-2:#00ccff; --good:#6bc26b; --bad:#5aa6c8;
|
||||
--font-mono:'JetBrains Mono','Fira Code','SF Mono',Menlo,Consolas,monospace;
|
||||
}
|
||||
*{box-sizing:border-box}
|
||||
html,body{margin:0;height:100%;background:var(--bg);color:var(--fg);
|
||||
font-family:var(--font-mono);font-size:14px;-webkit-font-smoothing:antialiased;overflow:hidden}
|
||||
|
||||
#app{position:fixed;inset:0}
|
||||
#scene{position:absolute;inset:0;width:100%;height:100%;display:block;touch-action:none;cursor:grab}
|
||||
#scene:active{cursor:grabbing}
|
||||
|
||||
/* floating HUD blocks */
|
||||
.hud{position:absolute;z-index:5}
|
||||
/* three zones, left → right: input · MLP stack · outputs */
|
||||
.hud-l{left:24px;top:50%;transform:translateY(-50%);display:flex;flex-direction:column;gap:10px;align-items:flex-start}
|
||||
.hud-r{right:24px;top:50%;transform:translateY(-50%)}
|
||||
.hud-bc{left:50%;bottom:22px;transform:translateX(-50%)}
|
||||
.hud-tr{right:18px;top:18px}
|
||||
.zonecap{font-size:9px;letter-spacing:.16em;text-transform:uppercase;color:var(--fg-dim);margin-bottom:2px}
|
||||
|
||||
/* output fader bank (1-D projection of each layer's value at the point) */
|
||||
.outs{display:flex;gap:13px;align-items:flex-start;background:rgba(20,20,20,.55);
|
||||
border:1px solid var(--line);border-radius:12px;padding:12px 14px;backdrop-filter:blur(4px)}
|
||||
.fader{display:flex;flex-direction:column;align-items:center;gap:6px;width:30px}
|
||||
.fader .track{position:relative;width:8px;height:200px;background:var(--bg-3);
|
||||
border:1px solid var(--line);border-radius:999px;overflow:hidden}
|
||||
.fader .fill{position:absolute;left:0;right:0;bottom:0;border-radius:999px;transition:height .05s linear}
|
||||
.fader .handle{position:absolute;left:50%;width:16px;height:3px;border-radius:2px;
|
||||
transform:translate(-50%,1.5px);box-shadow:0 0 8px currentColor}
|
||||
.fader .flab{font-size:9px;letter-spacing:.03em;text-transform:uppercase;max-width:34px;text-align:center;line-height:1.1}
|
||||
.fader .fval{font-size:10px;font-variant-numeric:tabular-nums;color:var(--fg)}
|
||||
|
||||
/* XY pad */
|
||||
#ndpad{display:block;border:1px solid var(--line);border-radius:8px;background:var(--bg-2);
|
||||
cursor:crosshair;touch-action:none}
|
||||
.padcap{font-size:10px;letter-spacing:.1em;text-transform:uppercase;color:var(--fg-dim)}
|
||||
|
||||
/* parameter numberbox */
|
||||
.kbox{display:flex;align-items:center;gap:8px;background:var(--bg-1);border:1px solid var(--line);
|
||||
border-radius:8px;padding:5px 8px;width:fit-content}
|
||||
.kbox .lab{font-size:10px;letter-spacing:.08em;text-transform:uppercase;color:var(--fg-mute)}
|
||||
.kbox .val{min-width:1.4ch;text-align:center;color:var(--fg);font-size:15px;font-variant-numeric:tabular-nums}
|
||||
.step{width:24px;height:24px;display:grid;place-items:center;padding:0;font-size:15px;line-height:1;
|
||||
background:var(--bg-2);border:1px solid var(--line-strong);border-radius:6px;color:var(--fg);cursor:pointer}
|
||||
.step:hover{background:var(--bg-3)}
|
||||
.step:disabled{opacity:.3;cursor:default}
|
||||
|
||||
/* action bar */
|
||||
.actionbar{display:flex;align-items:center;gap:8px;background:rgba(20,20,20,.82);
|
||||
border:1px solid var(--line);border-radius:12px;padding:8px;backdrop-filter:blur(6px)}
|
||||
.act{display:flex;flex-direction:column;align-items:center;gap:3px;min-width:58px;padding:8px 6px;
|
||||
background:var(--bg-2);border:1px solid var(--line-strong);border-radius:9px;color:var(--fg);
|
||||
cursor:pointer;font-family:inherit;transition:border-color .12s,background .12s}
|
||||
.act:hover{background:var(--bg-3)}
|
||||
.act .ic{font-size:18px;line-height:1}
|
||||
.act .tx{font-size:9px;letter-spacing:.04em;color:var(--fg-mute);text-transform:uppercase}
|
||||
.act.up:hover{border-color:var(--accent);box-shadow:0 0 12px rgba(255,106,0,.25)}
|
||||
.act.down:hover{border-color:var(--accent-2);box-shadow:0 0 12px rgba(0,204,255,.22)}
|
||||
.act:disabled{opacity:.35;cursor:default;box-shadow:none}
|
||||
.sep{width:1px;align-self:stretch;background:var(--line);margin:2px 2px}
|
||||
|
||||
/* mode toggle */
|
||||
.modeseg{display:inline-flex;border:1px solid var(--line-strong);border-radius:9px;overflow:hidden}
|
||||
.modeseg button{border:0;border-radius:0;padding:0 12px;height:100%;min-height:54px;font-family:inherit;
|
||||
font-size:11px;letter-spacing:.04em;background:var(--bg-2);color:var(--fg-mute);cursor:pointer}
|
||||
.modeseg button.on{background:var(--accent);color:#0d0d0d}
|
||||
|
||||
/* info toggle */
|
||||
#infoBtn{width:34px;height:34px;border-radius:50%;border:1px solid var(--line-strong);background:rgba(20,20,20,.82);
|
||||
color:var(--fg-mute);font-size:15px;cursor:pointer;backdrop-filter:blur(6px)}
|
||||
#infoBtn:hover{color:var(--accent);border-color:var(--accent)}
|
||||
|
||||
/* fading rotate hint */
|
||||
#hint{position:absolute;left:50%;top:18px;transform:translateX(-50%);z-index:4;
|
||||
font-size:11px;letter-spacing:.08em;color:var(--fg-dim);pointer-events:none;
|
||||
transition:opacity 1s ease;opacity:.85}
|
||||
#hint.gone{opacity:0}
|
||||
|
||||
/* loading */
|
||||
#loading{position:absolute;inset:0;display:grid;place-items:center;z-index:9;background:var(--bg);
|
||||
color:var(--fg-dim);font-size:13px;letter-spacing:.1em}
|
||||
#loading.gone{display:none}
|
||||
#loading b{color:var(--accent);letter-spacing:-.01em}
|
||||
|
||||
/* prose drawer (revealed by the info switch) */
|
||||
#prose{position:absolute;top:0;right:0;bottom:0;width:min(440px,86vw);z-index:8;overflow:auto;
|
||||
background:rgba(13,13,13,.96);border-left:1px solid var(--line);padding:40px 26px 64px;
|
||||
transform:translateX(102%);transition:transform .28s ease}
|
||||
body:not(.minimal) #prose{transform:none}
|
||||
#prose h1{font-size:24px;margin:0 0 8px}
|
||||
#prose h1 b{color:var(--accent)}
|
||||
#prose p{color:var(--fg-mute);line-height:1.6;margin:0 0 12px}
|
||||
#prose p b{color:var(--fg)}
|
||||
#prose .lbl{font-size:11px;letter-spacing:.14em;text-transform:uppercase;color:var(--accent);margin:26px 0 8px}
|
||||
#prose .ph{font-size:10px;letter-spacing:.1em;text-transform:uppercase;color:var(--fg-dim);
|
||||
border:1px dashed var(--line-strong);border-radius:6px;padding:6px 9px;display:inline-block;margin-bottom:14px}
|
||||
#prose .bars{display:flex;flex-direction:column;gap:7px;margin-top:6px}
|
||||
#prose .ro{display:flex;align-items:center;gap:10px}
|
||||
#prose .ro span{width:74px;font-size:10px;text-transform:uppercase;letter-spacing:.05em;color:var(--fg-mute)}
|
||||
#prose .bar{flex:1;height:7px;background:var(--bg-3);border:1px solid var(--line);border-radius:999px;overflow:hidden;position:relative}
|
||||
#prose .bar i{position:absolute;left:0;top:0;bottom:0;border-radius:999px}
|
||||
#prose .num{width:3.2ch;text-align:right;font-variant-numeric:tabular-nums;color:var(--fg);font-size:12px}
|
||||
#prose a{color:var(--accent-2);text-decoration:none}
|
||||
</style>
|
||||
</head>
|
||||
<body class="minimal">
|
||||
<div id="app">
|
||||
<canvas id="scene"></canvas>
|
||||
|
||||
<div id="hint">drag to rotate · top-down view = the flat 2-D map</div>
|
||||
|
||||
<!-- ZONE 1 · input: the 2-D control + parameter count (left) -->
|
||||
<div class="hud hud-l">
|
||||
<div class="zonecap">input · 2-D control</div>
|
||||
<canvas id="ndpad" width="140" height="140"></canvas>
|
||||
<div class="kbox">
|
||||
<span class="lab">params</span>
|
||||
<button class="step" id="kdown" title="fewer parameters">−</button>
|
||||
<span class="val" id="kval">1</span>
|
||||
<button class="step" id="kup" title="more parameters">+</button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- ZONE 3 · outputs: vertical faders, one per layer (right) -->
|
||||
<div class="hud hud-r">
|
||||
<div class="zonecap">outputs</div>
|
||||
<div class="outs" id="outs"></div>
|
||||
</div>
|
||||
|
||||
<!-- action bar (bottom-center) -->
|
||||
<div class="hud hud-bc actionbar">
|
||||
<button class="act" id="aRand"><span class="ic">🎲</span><span class="tx">randomise</span></button>
|
||||
<button class="act" id="aNudge"><span class="ic">〜</span><span class="tx">nudge</span></button>
|
||||
<button class="act" id="aUndo"><span class="ic">↶</span><span class="tx">undo</span></button>
|
||||
<div class="sep"></div>
|
||||
<button class="act up" id="aUp"><span class="ic">👍</span><span class="tx">keep</span></button>
|
||||
<button class="act down" id="aDown"><span class="ic">👎</span><span class="tx" id="downTx">push away</span></button>
|
||||
<div class="sep"></div>
|
||||
<div class="modeseg">
|
||||
<button id="mPush" class="on">push‑away</button>
|
||||
<button id="mExp">explore</button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<button id="infoBtn" class="hud hud-tr" title="show / hide explanation">ⓘ</button>
|
||||
|
||||
<div id="loading"><span>loading the <b> NISPS </b> engine…</span></div>
|
||||
</div>
|
||||
|
||||
<!-- placeholder explanatory copy (revealed by the ⓘ switch; to be reworked) -->
|
||||
<div id="prose">
|
||||
<h1>The <b>parameter landscape</b></h1>
|
||||
<p class="ph">placeholder copy — to be reworked</p>
|
||||
<p>Two numbers go in — your <b>X / Y control position</b> — and a whole fistful of parameters comes
|
||||
out. You can't picture a dozen dimensions at once, so picture them <b>stacked</b>: each <b>slab</b> is one
|
||||
parameter, drawn as a 3-D landscape whose height at any spot is that parameter's value. <b>One probe</b>
|
||||
reads every slab at the same control position.</p>
|
||||
<p>Seen straight from above the slabs overlap into a single flat map — it looks 2-D. <b>Tilt it</b> and the
|
||||
reveal lands: it was a stack of 3-D surfaces all along. Add parameters to grow the sandwich.</p>
|
||||
<p>Every surface here is the <b>real NISPS network</b> sampled across the input space. <b>👍 keep</b> teaches
|
||||
the net that this sound belongs at this spot and the landscape bends to honour it; <b>👎</b> either pushes the
|
||||
mapping away from the current sound or (in explore mode) re-rolls a fresh candidate to audition. <b>nudge</b>
|
||||
and <b>randomise</b> perturb the net; <b>undo</b> steps back.</p>
|
||||
<div class="lbl">Values under the probe</div>
|
||||
<div class="bars" id="ndbars"></div>
|
||||
<p style="margin-top:24px"><a href="../">← back to the instrument</a> · <a href="../animations/">animated explainers →</a></p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<script>
|
||||
(() => {
|
||||
'use strict';
|
||||
const clamp=(v,a=0,b=1)=>v<a?a:v>b?b:v;
|
||||
const cssv=n=>getComputedStyle(document.documentElement).getPropertyValue(n).trim();
|
||||
const MONO=cssv('--font-mono')||'monospace';
|
||||
|
||||
const NAMES=['Cutoff','Resonance','Drive','Tone','Decay','Air','Pitch','Width'];
|
||||
const ABBR =['Cut','Res','Drv','Ton','Dec','Air','Pit','Wid'];
|
||||
const MAXK=NAMES.length;
|
||||
const OUT=126, IN=2, GRID=12, N=GRID, PTS=(N+1)*(N+1);
|
||||
|
||||
let K=1; // parameters (slabs) visualised — from the same net
|
||||
let cx=0.5, cy=0.5; // 2-D control position in [0,1]
|
||||
let mode='pushaway'; // 'pushaway' | 'explore'
|
||||
let exploring=false; // explore-and-place scratchpad session active
|
||||
let ready=false, auto=true, autoT=0.6;
|
||||
|
||||
// camera — START TOP-DOWN (looks 2-D); drag tilts to reveal the 3-D stack
|
||||
let yaw=0, pitch=Math.PI/2, dragRot=false, lastX=0, lastY=0;
|
||||
|
||||
// ===== engine (vanilla wrapper over nisps.wasm; same C ABI the manifold uses) =====
|
||||
const E={
|
||||
mod:null,h:0,bIn:0,bOut:0,bF:0,bL:0,bW:0,bFb:0,wc:0,
|
||||
async init(){
|
||||
// nisps.{js,wasm} sit one level up at /next/ in production; probe a few bases.
|
||||
let base=null,src=null;
|
||||
for(const b of ['../','','./']){
|
||||
try{const r=await fetch(b+'nisps.js');if(r.ok){src=await r.text();base=b;break;}}catch(e){}
|
||||
}
|
||||
if(base===null) throw new Error('nisps.js not found');
|
||||
if(!window.createNispsModule)(0,eval)(src);
|
||||
this.mod=await window.createNispsModule({locateFile:p=>p.endsWith('.wasm')?base+'nisps.wasm':p});
|
||||
const m=this.mod;
|
||||
this.h=m._nisps_ml_create(IN,OUT,0,0,(Date.now()>>>0));
|
||||
this.wc=m._nisps_ml_weight_count(this.h);
|
||||
this.bF=m._malloc(IN*4); this.bL=m._malloc(OUT*4); this.bFb=m._malloc(OUT*4);
|
||||
this.bW=m._malloc(this.wc*4);
|
||||
this.bIn=m._malloc(PTS*IN*4); this.bOut=m._malloc(PTS*OUT*4);
|
||||
},
|
||||
inferXY(x,y){const m=this.mod;m._nisps_ml_set_input(this.h,0,x);m._nisps_ml_set_input(this.h,1,y);
|
||||
m._nisps_ml_process(this.h);const p=m._nisps_ml_outputs(this.h);
|
||||
return new Float32Array(m.HEAPF32.buffer,p,OUT);},
|
||||
inferGrid(){const m=this.mod;const inv=new Float32Array(m.HEAPF32.buffer,this.bIn,PTS*IN);
|
||||
let q=0;for(let j=0;j<=N;j++)for(let i=0;i<=N;i++){inv[q++]=i/N;inv[q++]=j/N;}
|
||||
m._nisps_ml_infer_batch(this.h,this.bIn,PTS,this.bOut);
|
||||
return new Float32Array(new Float32Array(m.HEAPF32.buffer,this.bOut,PTS*OUT));},
|
||||
addExample(f,l){const m=this.mod;const fv=new Float32Array(m.HEAPF32.buffer,this.bF,IN);
|
||||
const lv=new Float32Array(m.HEAPF32.buffer,this.bL,OUT);fv[0]=f[0];fv[1]=f[1];
|
||||
for(let i=0;i<OUT;i++)lv[i]=l[i]||0;m._nisps_ml_add_example(this.h,this.bF,this.bL);},
|
||||
train(lr,it,me){return this.mod._nisps_ml_train(this.h,lr,it,me,0);},
|
||||
draw(s){this.mod._nisps_ml_draw_weights(this.h,s);},
|
||||
move(sp,s){this.mod._nisps_ml_move_weights(this.h,sp,s,0);},
|
||||
setMode(i){this.mod._nisps_ml_feedback_set_mode(this.h,i);},
|
||||
down(out,sp,s){const m=this.mod;const ov=new Float32Array(m.HEAPF32.buffer,this.bFb,OUT);
|
||||
ov.fill(0);for(let i=0;i<OUT;i++)ov[i]=out[i]||0;m._nisps_ml_feedback_down(this.h,this.bFb,sp,s,0);},
|
||||
enterExplore(s){this.mod._nisps_ml_feedback_enter_explore(this.h,s);},
|
||||
reroll(s){this.mod._nisps_ml_feedback_reroll(this.h,s);},
|
||||
coreNudge(a){this.mod._nisps_ml_feedback_nudge(this.h,a);},
|
||||
coreUndo(){this.mod._nisps_ml_feedback_undo(this.h);},
|
||||
exitExplore(){this.mod._nisps_ml_feedback_exit_explore(this.h);},
|
||||
getWeights(){this.mod._nisps_ml_get_weights(this.h,this.bW);
|
||||
return new Float32Array(new Float32Array(this.mod.HEAPF32.buffer,this.bW,this.wc));},
|
||||
setWeights(w){const v=new Float32Array(this.mod.HEAPF32.buffer,this.bW,this.wc);v.set(w);
|
||||
this.mod._nisps_ml_set_weights(this.h,this.bW);},
|
||||
};
|
||||
|
||||
// Low spread = polarised weights = strong spatial relief (measured); high spread
|
||||
// flattens the surface toward 0.5. Keep it low so the landscapes show real terrain.
|
||||
const SPREAD=0.1, NUDGE=0.05, BOOT_SPREAD=0.05;
|
||||
let grid=null, probeOut=new Float32Array(OUT);
|
||||
function H(k,i,j){return grid?grid[(j*(N+1)+i)*OUT+k]:0.5;}
|
||||
function resample(){grid=E.inferGrid();}
|
||||
function updateProbe(){const o=E.inferXY(cx,cy);probeOut=new Float32Array(o);}
|
||||
|
||||
// ---- weight-snapshot undo (push-away mode; explore mode uses the core ring) ----
|
||||
const undoStack=[];
|
||||
function pushUndo(){undoStack.push(E.getWeights());if(undoStack.length>24)undoStack.shift();}
|
||||
function popUndo(){const w=undoStack.pop();if(w){E.setWeights(w);return true;}return false;}
|
||||
|
||||
// ---- feedback examples marked on the 2-D control plane (like the main app) ----
|
||||
// {x,y,sign} — sign +1 kept (👍), -1 pushed away (👎). undo pops the last.
|
||||
const marks=[];
|
||||
|
||||
// ===== actions (all backed by the real engine) =====
|
||||
function act(name){
|
||||
if(!ready)return;
|
||||
if(mode==='pushaway'){
|
||||
if(name==='rand'){pushUndo();E.draw(SPREAD);}
|
||||
else if(name==='nudge'){pushUndo();E.move(0.04,SPREAD);}
|
||||
else if(name==='undo'){popUndo();marks.pop();}
|
||||
else if(name==='up'){pushUndo();const o=new Float32Array(E.inferXY(cx,cy));E.addExample([cx,cy],o);E.train(1.0,200,1e-4);marks.push({x:cx,y:cy,sign:1});}
|
||||
else if(name==='down'){pushUndo();const o=new Float32Array(E.inferXY(cx,cy));E.down(o,0.15,SPREAD);marks.push({x:cx,y:cy,sign:-1});}
|
||||
}else{ // explore-and-place
|
||||
if(name==='rand'||name==='down'){
|
||||
if(!exploring){E.setMode(3);E.enterExplore(SPREAD);exploring=true;}else E.reroll(SPREAD);
|
||||
}else if(name==='nudge'){
|
||||
if(!exploring){E.setMode(3);E.enterExplore(SPREAD);exploring=true;}else E.coreNudge(NUDGE);
|
||||
}else if(name==='undo'){
|
||||
if(exploring)E.coreUndo();else{popUndo();marks.pop();}
|
||||
}else if(name==='up'){
|
||||
const o=new Float32Array(E.inferXY(cx,cy));
|
||||
if(exploring){E.exitExplore();exploring=false;}
|
||||
E.addExample([cx,cy],o);E.train(1.0,200,1e-4);marks.push({x:cx,y:cy,sign:1});
|
||||
}
|
||||
}
|
||||
resample();updateProbe();reflectButtons();redraw();
|
||||
}
|
||||
function setMode(m){
|
||||
if(m===mode)return;
|
||||
if(mode==='explore'&&exploring){E.exitExplore();exploring=false;}
|
||||
mode=m;E.setMode(m==='explore'?3:0);
|
||||
document.getElementById('mPush').classList.toggle('on',m==='pushaway');
|
||||
document.getElementById('mExp').classList.toggle('on',m==='explore');
|
||||
document.getElementById('downTx').textContent=m==='explore'?'skip':'push away';
|
||||
if(ready){resample();updateProbe();reflectButtons();redraw();}
|
||||
}
|
||||
function reflectButtons(){
|
||||
const ud=document.getElementById('aUndo');
|
||||
ud.disabled=(mode==='explore'?false:undoStack.length===0)&&!exploring;
|
||||
}
|
||||
|
||||
// ===== 3-D projection: yaw + pitch orbit, bounding-sphere fit, X inverted =====
|
||||
const SX=300,SZ=300,LAYER_GAP=120,ZAMP=92,MARGIN=44;
|
||||
// Anchor the view to the TOP slab (k=0) at a FIXED world Y and a FIXED screen
|
||||
// position, independent of K. So adding parameters grows the stack DOWNWARD and
|
||||
// the top slab always stays the top — it never re-centres onto new layers.
|
||||
const AWY=ZAMP*0.5, ANCHOR_FY=0.42;
|
||||
let CW=800,CH=600,scale=1;
|
||||
function computeView(){
|
||||
let R=1; // bounding-sphere radius about the top anchor
|
||||
for(let k=0;k<K;k++)for(const z of[0,1])for(const gx of[-0.5,0.5])for(const gy of[-0.5,0.5]){
|
||||
const X=gx*SX,Y=(-k*LAYER_GAP+z*ZAMP)-AWY,Z=gy*SZ;const d=Math.sqrt(X*X+Y*Y+Z*Z);if(d>R)R=d;}
|
||||
scale=(Math.min(CW,CH)/2-MARGIN)/R;
|
||||
}
|
||||
function project(gx,gy,zv,k){
|
||||
const X=gx*SX, Y=(-k*LAYER_GAP+zv*ZAMP)-AWY, Z=-gy*SZ; // Z negated: pad-up = map-up
|
||||
const cyA=Math.cos(yaw),syA=Math.sin(yaw);
|
||||
const x1=X*cyA-Z*syA, z1=X*syA+Z*cyA;
|
||||
const cpA=Math.cos(pitch),spA=Math.sin(pitch);
|
||||
const y2=Y*cpA-z1*spA, z2=Y*spA+z1*cpA;
|
||||
return {x:CW/2+x1*scale,y:CH*ANCHOR_FY-y2*scale,depth:z2,wx:X,wy:Y,wz:Z};
|
||||
}
|
||||
|
||||
const DPR=Math.max(1,Math.min(2,window.devicePixelRatio||1));
|
||||
const scene=document.getElementById('scene'),sctx=scene.getContext('2d');
|
||||
const pad=document.getElementById('ndpad'),pctx=pad.getContext('2d');
|
||||
pad.width=140*DPR;pad.height=140*DPR;pctx.scale(DPR,DPR);
|
||||
function resize(){
|
||||
const r=scene.getBoundingClientRect();CW=Math.max(320,Math.round(r.width));CH=Math.max(320,Math.round(r.height));
|
||||
scene.width=CW*DPR;scene.height=CH*DPR;sctx.setTransform(DPR,0,0,DPR,0,0);computeView();
|
||||
}
|
||||
|
||||
const hueOf=k=>K>1?(26+(196-26)*k/(K-1)):26;
|
||||
const colCss=(k,a)=>`hsla(${hueOf(k).toFixed(0)},85%,60%,${a})`;
|
||||
const LIGHT=(()=>{const v=[0.35,0.86,0.36],m=Math.hypot(v[0],v[1],v[2]);return[v[0]/m,v[1]/m,v[2]/m];})();
|
||||
|
||||
function renderScene(){
|
||||
const c=sctx;c.clearRect(0,0,CW,CH);
|
||||
const quads=[],labels=[];
|
||||
for(let k=0;k<K;k++){
|
||||
const hue=hueOf(k),P=[];
|
||||
for(let j=0;j<=N;j++){P[j]=[];for(let i=0;i<=N;i++)P[j][i]=project(i/N-0.5,j/N-0.5,H(k,i,j),k);}
|
||||
for(let j=0;j<N;j++)for(let i=0;i<N;i++){
|
||||
const a=P[j][i],b=P[j][i+1],d=P[j+1][i+1],e=P[j+1][i];
|
||||
const ux=d.wx-a.wx,uy=d.wy-a.wy,uz=d.wz-a.wz,vx=b.wx-e.wx,vy=b.wy-e.wy,vz=b.wz-e.wz;
|
||||
const nx=uy*vz-uz*vy,ny=uz*vx-ux*vz,nz=ux*vy-uy*vx,nm=Math.hypot(nx,ny,nz)||1;
|
||||
const L=clamp(0.32+0.62*Math.abs((nx*LIGHT[0]+ny*LIGHT[1]+nz*LIGHT[2])/nm));
|
||||
quads.push({a,b,d,e,hue,L,depth:(a.depth+b.depth+d.depth+e.depth)/4});
|
||||
}
|
||||
const lp=project(-0.5,0.5,H(k,0,N),k);labels.push({x:lp.x,y:lp.y,hue,name:NAMES[k]});
|
||||
}
|
||||
quads.sort((p,q)=>q.depth-p.depth);
|
||||
for(const Q of quads){
|
||||
c.fillStyle=`hsla(${Q.hue},80%,${(26+Q.L*44).toFixed(0)}%,0.66)`;
|
||||
c.strokeStyle=`hsla(${Q.hue},80%,82%,0.18)`;c.lineWidth=0.6;
|
||||
c.beginPath();c.moveTo(Q.a.x,Q.a.y);c.lineTo(Q.b.x,Q.b.y);c.lineTo(Q.d.x,Q.d.y);c.lineTo(Q.e.x,Q.e.y);c.closePath();
|
||||
c.fill();c.stroke();
|
||||
}
|
||||
c.font='11px '+MONO;c.textAlign='right';c.textBaseline='middle';
|
||||
for(const Lb of labels){c.fillStyle=`hsla(${Lb.hue},85%,74%,0.95)`;c.fillText(Lb.name,Lb.x-6,Lb.y);}
|
||||
// probe column at (cx,cy) through every slab
|
||||
const gx=cx-0.5,gy=cy-0.5;
|
||||
const top=project(gx,gy,1.18,0),bot=project(gx,gy,-0.18,K-1);
|
||||
c.strokeStyle='rgba(255,255,255,0.32)';c.lineWidth=1;c.setLineDash([3,4]);
|
||||
c.beginPath();c.moveTo(top.x,top.y);c.lineTo(bot.x,bot.y);c.stroke();c.setLineDash([]);
|
||||
for(let k=0;k<K;k++){
|
||||
const p=project(gx,gy,clamp(probeOut[k]),k),fl=project(gx,gy,0,k);
|
||||
c.strokeStyle=colCss(k,0.5);c.lineWidth=1;c.beginPath();c.moveTo(p.x,p.y);c.lineTo(fl.x,fl.y);c.stroke();
|
||||
c.fillStyle=colCss(k,1);c.shadowColor=colCss(k,1);c.shadowBlur=12;
|
||||
c.beginPath();c.arc(p.x,p.y,6,0,Math.PI*2);c.fill();c.shadowBlur=0;
|
||||
c.fillStyle='#0d0d0d';c.beginPath();c.arc(p.x,p.y,2.2,0,Math.PI*2);c.fill();
|
||||
}
|
||||
}
|
||||
function drawPad(){
|
||||
pctx.clearRect(0,0,140,140);pctx.fillStyle='#1c1c1c';pctx.fillRect(0,0,140,140);
|
||||
pctx.strokeStyle='rgba(255,255,255,0.06)';pctx.lineWidth=1;
|
||||
for(let i=1;i<4;i++){const t=i/4*140;pctx.beginPath();pctx.moveTo(t,0);pctx.lineTo(t,140);pctx.stroke();
|
||||
pctx.beginPath();pctx.moveTo(0,t);pctx.lineTo(140,t);pctx.stroke();}
|
||||
// feedback examples (👍 +, 👎 −) — same convention as the main app
|
||||
const TAU=Math.PI*2;
|
||||
for(const m of marks){const mx=m.x*140,my=(1-m.y)*140,col=m.sign>0?'#ff6a00':'#00ccff';
|
||||
pctx.globalAlpha=0.28;pctx.fillStyle=col;pctx.beginPath();pctx.arc(mx,my,8,0,TAU);pctx.fill();
|
||||
pctx.globalAlpha=1;pctx.strokeStyle=col;pctx.lineWidth=1.5;pctx.beginPath();pctx.arc(mx,my,5,0,TAU);pctx.stroke();
|
||||
pctx.fillStyle=col;pctx.font='9px '+MONO;pctx.textAlign='center';pctx.textBaseline='middle';
|
||||
pctx.fillText(m.sign>0?'+':'–',mx,my+0.5);}
|
||||
const dx=cx*140,dy=(1-cy)*140;
|
||||
pctx.strokeStyle='rgba(255,255,255,0.1)';pctx.beginPath();pctx.moveTo(0,dy);pctx.lineTo(140,dy);
|
||||
pctx.moveTo(dx,0);pctx.lineTo(dx,140);pctx.stroke();
|
||||
pctx.fillStyle='#ff6a00';pctx.shadowColor='#ff6a00';pctx.shadowBlur=12;
|
||||
pctx.beginPath();pctx.arc(dx,dy,6,0,Math.PI*2);pctx.fill();pctx.shadowBlur=0;
|
||||
pctx.fillStyle='#0d0d0d';pctx.beginPath();pctx.arc(dx,dy,2.4,0,Math.PI*2);pctx.fill();
|
||||
}
|
||||
|
||||
// readout bars (prose drawer only)
|
||||
const barsEl=document.getElementById('ndbars');let barI=[],barN=[];
|
||||
function buildBars(){barsEl.innerHTML='';barI=[];barN=[];
|
||||
for(let k=0;k<K;k++){const ro=document.createElement('div');ro.className='ro';
|
||||
ro.innerHTML=`<span style="color:${colCss(k,0.95)}">${NAMES[k]}</span>`+
|
||||
`<div class="bar"><i style="background:${colCss(k,1)}"></i></div><div class="num">0.50</div>`;
|
||||
barsEl.appendChild(ro);barI[k]=ro.querySelector('i');barN[k]=ro.querySelector('.num');}}
|
||||
function updateBars(){if(!barI.length)return;for(let k=0;k<K;k++){const v=clamp(probeOut[k]);
|
||||
barI[k].style.width=(v*100)+'%';barN[k].textContent=v.toFixed(2);}}
|
||||
|
||||
// output faders — the 1-D projection of each layer's value at the current point
|
||||
const TRACK_H=200;
|
||||
const outsEl=document.getElementById('outs');let fFill=[],fHan=[],fVal=[];
|
||||
function buildOuts(){outsEl.innerHTML='';fFill=[];fHan=[];fVal=[];
|
||||
for(let k=0;k<K;k++){const col=colCss(k,1);const f=document.createElement('div');f.className='fader';
|
||||
f.innerHTML=`<div class="track"><div class="fill" style="background:${col}"></div>`+
|
||||
`<div class="handle" style="background:${col};color:${col}"></div></div>`+
|
||||
`<div class="flab" style="color:${colCss(k,0.95)}" title="${NAMES[k]}">${ABBR[k]}</div><div class="fval">0.50</div>`;
|
||||
outsEl.appendChild(f);fFill[k]=f.querySelector('.fill');fHan[k]=f.querySelector('.handle');fVal[k]=f.querySelector('.fval');}}
|
||||
function updateOuts(){if(!fFill.length)return;for(let k=0;k<K;k++){const v=clamp(probeOut[k]);
|
||||
fFill[k].style.height=(v*100)+'%';fHan[k].style.bottom=(v*TRACK_H)+'px';fVal[k].textContent=v.toFixed(2);}}
|
||||
|
||||
function redraw(){renderScene();drawPad();updateBars();updateOuts();}
|
||||
|
||||
// ===== interaction =====
|
||||
function setK(n){K=clamp(n,1,MAXK);document.getElementById('kval').textContent=K;
|
||||
document.getElementById('kup').disabled=K>=MAXK;document.getElementById('kdown').disabled=K<=1;
|
||||
computeView();buildBars();buildOuts();updateProbe&&(probeOut=new Float32Array(E.inferXY(cx,cy)));redraw();}
|
||||
|
||||
function stopAuto(){auto=false;document.getElementById('hint').classList.add('gone');}
|
||||
|
||||
// pad → control position
|
||||
function setFromPad(e){const r=pad.getBoundingClientRect();
|
||||
cx=clamp((e.clientX-r.left)/r.width);cy=clamp(1-(e.clientY-r.top)/r.height);}
|
||||
let padDrag=false;
|
||||
pad.addEventListener('pointerdown',e=>{padDrag=true;pad.setPointerCapture(e.pointerId);stopAuto();setFromPad(e);updateProbe();redraw();});
|
||||
pad.addEventListener('pointermove',e=>{if(padDrag){setFromPad(e);updateProbe();redraw();}});
|
||||
pad.addEventListener('pointerup',()=>padDrag=false);
|
||||
|
||||
// scene → orbit (pitch inverted; pitch up to π/2 = exact top-down)
|
||||
scene.addEventListener('pointerdown',e=>{dragRot=true;scene.setPointerCapture(e.pointerId);stopAuto();lastX=e.clientX;lastY=e.clientY;});
|
||||
scene.addEventListener('pointermove',e=>{if(!dragRot)return;
|
||||
yaw+=(e.clientX-lastX)*0.01;
|
||||
pitch=clamp(pitch+(e.clientY-lastY)*0.008,0.06,Math.PI/2); // vertical (Y) drag — inverted
|
||||
lastX=e.clientX;lastY=e.clientY;redraw();});
|
||||
scene.addEventListener('pointerup',()=>dragRot=false);
|
||||
scene.addEventListener('pointercancel',()=>dragRot=false);
|
||||
|
||||
// buttons
|
||||
const B=(id,fn)=>document.getElementById(id).addEventListener('click',()=>{stopAuto();fn();});
|
||||
B('aRand',()=>act('rand'));B('aNudge',()=>act('nudge'));B('aUndo',()=>act('undo'));
|
||||
B('aUp',()=>act('up'));B('aDown',()=>act('down'));
|
||||
B('kup',()=>setK(K+1));B('kdown',()=>setK(K-1));
|
||||
document.getElementById('mPush').addEventListener('click',()=>{stopAuto();setMode('pushaway');});
|
||||
document.getElementById('mExp').addEventListener('click',()=>{stopAuto();setMode('explore');});
|
||||
document.getElementById('infoBtn').addEventListener('click',()=>{
|
||||
document.body.classList.toggle('minimal');
|
||||
document.getElementById('infoBtn').textContent=document.body.classList.contains('minimal')?'ⓘ':'✕';});
|
||||
|
||||
window.addEventListener('resize',()=>{resize();redraw();});
|
||||
|
||||
function loop(){
|
||||
if(auto&&ready&&!padDrag&&!dragRot){
|
||||
autoT+=0.006;cx=0.5+0.42*Math.sin(autoT*0.9);cy=0.5+0.42*Math.sin(autoT*1.27+1.3);
|
||||
updateProbe();redraw();
|
||||
}
|
||||
requestAnimationFrame(loop);
|
||||
}
|
||||
|
||||
// ===== boot =====
|
||||
(async()=>{
|
||||
try{
|
||||
await E.init();
|
||||
E.draw(BOOT_SPREAD); // a fresh net is near-flat; draw for visible relief
|
||||
window.__learn={E,setK,resample,redraw,
|
||||
range(k){let lo=1,hi=0;for(let j=0;j<=N;j++)for(let i=0;i<=N;i++){const v=H(k,i,j);if(v<lo)lo=v;if(v>hi)hi=v;}return[lo,hi];}};
|
||||
resize();setK(1);resample();updateProbe();reflectButtons();
|
||||
ready=true;document.getElementById('loading').classList.add('gone');
|
||||
redraw();requestAnimationFrame(loop);
|
||||
// fade the hint after a while
|
||||
setTimeout(()=>{if(auto)document.getElementById('hint').classList.add('gone');},6000);
|
||||
}catch(err){
|
||||
console.error('[learn] engine init failed:',err);
|
||||
document.getElementById('loading').innerHTML='<span style="color:#c8553a">engine failed to load — see console</span>';
|
||||
}
|
||||
})();
|
||||
})();
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
|
|
@ -31,6 +31,7 @@
|
|||
#include "glue/midi_io.hpp"
|
||||
#include "glue/mode_select.hpp"
|
||||
#include "glue/output_router.hpp"
|
||||
#include "glue/settings_view.hpp"
|
||||
|
||||
// ---- Mode selection ----
|
||||
// Build script rewrites which line is uncommented.
|
||||
|
|
@ -141,6 +142,8 @@ void setup() {
|
|||
}
|
||||
|
||||
MEMLNaut::Instance()->addSystemInfoView();
|
||||
// Settings menu (e.g. Joystick: Dual/Single for the 4-input modes).
|
||||
nisps_firmware::wire_settings(g_mode);
|
||||
Serial.println("Finished initialising core 0.");
|
||||
}
|
||||
|
||||
|
|
|
|||
64
firmware/MEMLNaut-NISPS/glue/settings_view.hpp
Normal file
64
firmware/MEMLNaut-NISPS/glue/settings_view.hpp
Normal file
|
|
@ -0,0 +1,64 @@
|
|||
// glue/settings_view.hpp — on-device settings menu views (TFT + rotary encoder).
|
||||
//
|
||||
// Adds entries to the MEMLNaut display carousel (the same DisplayDriver the
|
||||
// SystemView and SelfTest use). Navigation: rotate to move between views;
|
||||
// press to focus a view; rotate-while-focused to change its value; press
|
||||
// again to unfocus. See src/memllib/.../display/DisplayDriver.hpp.
|
||||
//
|
||||
// Currently provides:
|
||||
// * Joystick: Dual / Single — for the 4-input ("two 2-D joystick") modes,
|
||||
// toggles whether the SECOND joystick (input channels 2,3) feeds the ML
|
||||
// engine. "Single" pins those channels to the neutral value (0.5) via the
|
||||
// platform-agnostic ModeBase pin mechanism, so the network is never
|
||||
// rebuilt and trained state survives toggling. Default is Dual (the 4-D
|
||||
// variant naturally uses both joysticks).
|
||||
//
|
||||
// Only registered for modes whose input arity is exactly 4 (the two-joystick
|
||||
// modes); other modes (e.g. SoundAnalysisMIDI, whose joystick channels are not
|
||||
// 2,3) get no joystick toggle.
|
||||
|
||||
#pragma once
|
||||
|
||||
#include <array>
|
||||
#include <cstddef>
|
||||
#include <memory>
|
||||
#include <span>
|
||||
|
||||
#include "../src/memllib/hardware/memlnaut/MEMLNaut.hpp"
|
||||
#include "../src/memllib/hardware/memlnaut/display/SingleSelectView.hpp"
|
||||
|
||||
namespace nisps_firmware {
|
||||
|
||||
// The second 2-D joystick maps to ML input channels 2 and 3 (joy_z / joy_w).
|
||||
inline constexpr std::size_t kSecondJoystickCh0 = 2u;
|
||||
inline constexpr std::size_t kSecondJoystickCh1 = 3u;
|
||||
|
||||
// Add the settings views for `mode` to the display carousel. Must run AFTER
|
||||
// MEMLNaut::Initialize() (so the display + TFT exist) and after
|
||||
// addSystemInfoView(), on core 0.
|
||||
template <typename ModeT>
|
||||
inline void wire_settings(ModeT& mode) {
|
||||
if constexpr (ModeT::input_channel_count() == 4u) {
|
||||
MEMLNaut* meml = MEMLNaut::Instance();
|
||||
if (!meml || !meml->disp) return;
|
||||
|
||||
// Kept alive for the program lifetime (the carousel holds a shared_ptr
|
||||
// too, but this guarantees ownership regardless of view churn).
|
||||
static std::shared_ptr<SingleSelectView> joystick_view;
|
||||
joystick_view = std::make_shared<SingleSelectView>("Joystick");
|
||||
meml->disp->AddView(joystick_view); // calls Setup() → creates the selector
|
||||
|
||||
// Index 0 = Dual (default, no pinning); index 1 = Single.
|
||||
std::array<String, 2u> options = { String("Dual"), String("Single") };
|
||||
joystick_view->setOptions(std::span<String>(options.data(), options.size()));
|
||||
joystick_view->setNewVoiceCallback([&mode](std::size_t idx) {
|
||||
const bool single = (idx == 1u);
|
||||
mode.set_input_pinned(kSecondJoystickCh0, single);
|
||||
mode.set_input_pinned(kSecondJoystickCh1, single);
|
||||
});
|
||||
} else {
|
||||
(void)mode;
|
||||
}
|
||||
}
|
||||
|
||||
} // namespace nisps_firmware
|
||||
Binary file not shown.
|
|
@ -28,6 +28,7 @@ import { MF_MODES, modeEngineId, seededGradient, shapeValues } from './model';
|
|||
import type { MFParam } from './model';
|
||||
import { CompositeStage } from './CompositeStage';
|
||||
import { ParticleStage } from './ParticleStage';
|
||||
import { SandwichStage } from './SandwichStage';
|
||||
import { SplitStage } from './SplitStage';
|
||||
import { OutputStage } from './OutputStage';
|
||||
import { InputMini } from './InputMini';
|
||||
|
|
@ -107,6 +108,8 @@ export function ConsoleApp({ focus: initialFocus = 'composite' }: ConsoleAppProp
|
|||
const [rev, setRev] = useState(1);
|
||||
const [active, setActive] = useState<DrawerKey | null>('learn');
|
||||
const [depth, setDepth] = useState<DrawerDepth>('peek');
|
||||
// Sandwich (parameter-landscape) centre-stage toggle — dock-bottom layers icon.
|
||||
const [sandwich, setSandwich] = useState(false);
|
||||
|
||||
// Learning-behaviour store (dock-spec §1; rl-feedback-design). Default
|
||||
// feedback mode = "Explore and place"; default solo = "Mask gradients".
|
||||
|
|
@ -718,7 +721,55 @@ export function ConsoleApp({ focus: initialFocus = 'composite' }: ConsoleAppProp
|
|||
bottom: 0,
|
||||
}}
|
||||
>
|
||||
{outputMode === 'particles' ? (
|
||||
{sandwich ? (
|
||||
// Sandwich centre-stage: shrunken input (left) · landscape stack
|
||||
// (centre, fills) · compact outputs (right). Replaces the Mode stage.
|
||||
<div style={{ position: 'absolute', inset: 0, display: 'flex' }}>
|
||||
<div
|
||||
style={{
|
||||
width: 'clamp(200px, 24%, 320px)',
|
||||
flex: '0 0 auto',
|
||||
position: 'relative',
|
||||
borderRight: '1px solid var(--line)',
|
||||
}}
|
||||
>
|
||||
<Manifold
|
||||
pos={pos}
|
||||
onMove={onMove}
|
||||
noiseCap={noiseCap}
|
||||
pins={pins}
|
||||
markers={markers}
|
||||
variant={inputMapVariant}
|
||||
frozen={false}
|
||||
follow={follow}
|
||||
onLongPress={addPin}
|
||||
picking={picking}
|
||||
onPickLocation={onPickLocation}
|
||||
/>
|
||||
</div>
|
||||
<div style={{ flex: 1, minWidth: 0, position: 'relative' }}>
|
||||
{engine && (
|
||||
<SandwichStage
|
||||
engine={engine}
|
||||
version={version}
|
||||
pos={pos}
|
||||
layerCount={Math.min(params.length, 8)}
|
||||
names={params.map((p) => p.name)}
|
||||
/>
|
||||
)}
|
||||
</div>
|
||||
<div
|
||||
style={{
|
||||
width: 'clamp(200px, 24%, 320px)',
|
||||
flex: '0 0 auto',
|
||||
position: 'relative',
|
||||
borderLeft: '1px solid var(--line)',
|
||||
}}
|
||||
>
|
||||
<OutputStage params={params} values={values} onChange={setParam} compact />
|
||||
</div>
|
||||
</div>
|
||||
) : outputMode === 'particles' ? (
|
||||
<ParticleStage
|
||||
pos={pos}
|
||||
onMove={onMove}
|
||||
|
|
@ -903,7 +954,19 @@ export function ConsoleApp({ focus: initialFocus = 'composite' }: ConsoleAppProp
|
|||
</div>
|
||||
</div>
|
||||
|
||||
<Dock ctx={ctx} active={active} setActive={setActive} depth={depth} setDepth={setDepth} />
|
||||
<Dock
|
||||
ctx={ctx}
|
||||
active={active}
|
||||
setActive={setActive}
|
||||
depth={depth}
|
||||
setDepth={setDepth}
|
||||
sandwich={sandwich}
|
||||
setSandwich={(v) => {
|
||||
setSandwich(v);
|
||||
// reveal the full 3-zone layout: a drawer would overlay the output zone
|
||||
if (v) setActive(null);
|
||||
}}
|
||||
/>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -28,6 +28,7 @@ import {
|
|||
OscIcon,
|
||||
SynthIcon,
|
||||
EditorIcon,
|
||||
SandwichIcon,
|
||||
CloseIcon,
|
||||
ExpandIcon,
|
||||
GLYPH_FALLBACK,
|
||||
|
|
@ -173,11 +174,14 @@ export interface DockProps {
|
|||
setActive: (k: DrawerKey | null) => void;
|
||||
depth: DrawerDepth;
|
||||
setDepth: (d: DrawerDepth) => void;
|
||||
/** Sandwich (parameter-landscape) centre-stage toggle. */
|
||||
sandwich: boolean;
|
||||
setSandwich: (v: boolean) => void;
|
||||
}
|
||||
|
||||
const ORDER: DrawerKey[] = ['learn', 'inputs', 'route', 'settings', 'help'];
|
||||
|
||||
export function Dock({ ctx, active, setActive, depth, setDepth }: DockProps) {
|
||||
export function Dock({ ctx, active, setActive, depth, setDepth, sandwich, setSandwich }: DockProps) {
|
||||
const { settings } = useSettings();
|
||||
const mono = settings.monochromeIcons;
|
||||
const restColour = unfocusedIconCss(settings.unfocusedIconColour);
|
||||
|
|
@ -368,6 +372,27 @@ export function Dock({ ctx, active, setActive, depth, setDepth }: DockProps) {
|
|||
>
|
||||
{ORDER.map(iconBtn)}
|
||||
</div>
|
||||
{/* Sandwich toggle — pinned at the BOTTOM of the rail. */}
|
||||
<button
|
||||
type="button"
|
||||
title="Parameter sandwich (layer view)"
|
||||
onClick={() => setSandwich(!sandwich)}
|
||||
style={{
|
||||
width: 40,
|
||||
height: 40,
|
||||
borderRadius: 'var(--r-2)',
|
||||
cursor: 'pointer',
|
||||
border: `1px solid ${sandwich ? 'var(--accent)' : 'transparent'}`,
|
||||
background: sandwich ? 'rgba(255,106,0,0.14)' : 'transparent',
|
||||
color: sandwich ? 'var(--accent)' : mono ? restColour : 'var(--fg-mute)',
|
||||
display: 'flex',
|
||||
alignItems: 'center',
|
||||
justifyContent: 'center',
|
||||
transition: 'background var(--dur-fast), color var(--dur-fast)',
|
||||
}}
|
||||
>
|
||||
{mono ? <SandwichIcon /> : '≣'}
|
||||
</button>
|
||||
</nav>
|
||||
</>
|
||||
);
|
||||
|
|
|
|||
|
|
@ -240,6 +240,7 @@ function InputsDrawer(ctx: ConsoleCtx, depth: DrawerDepth) {
|
|||
const inp = ctx.inputs;
|
||||
const enabledCount = inp.sources.filter((s) => s.enabled).length;
|
||||
const reshaping = inp.axisCount > inp.engineInputSize;
|
||||
const gamepadOn = inp.sources.find((s) => s.kind === 'gamepad')?.enabled ?? false;
|
||||
|
||||
return (
|
||||
<>
|
||||
|
|
@ -252,6 +253,22 @@ function InputsDrawer(ctx: ConsoleCtx, depth: DrawerDepth) {
|
|||
{reshaping && <Chip tone="var(--warn)">blended → {inp.engineInputSize}</Chip>}
|
||||
</div>
|
||||
|
||||
{/* ---- Controllers: 1 (pad) vs 2 (pad + gamepad). Default single. ---- */}
|
||||
<SectionLabel>Controllers</SectionLabel>
|
||||
<Segmented
|
||||
value={gamepadOn ? 'double' : 'single'}
|
||||
onChange={(m) => inp.setEnabled('gamepad', m === 'double')}
|
||||
options={[
|
||||
{ value: 'single', label: 'Single' },
|
||||
{ value: 'double', label: 'Double (+gamepad)' },
|
||||
]}
|
||||
/>
|
||||
<p style={{ fontSize: 9, color: 'var(--fg-dim)', margin: '2px 0 0' }}>
|
||||
{gamepadOn
|
||||
? 'Two 2-D controllers feed the model (pad → axes 0,1; gamepad → axes 2,3).'
|
||||
: 'One 2-D controller (the pad). The second controller’s axes stay neutral (0.5).'}
|
||||
</p>
|
||||
|
||||
<SectionLabel>Sources</SectionLabel>
|
||||
<div style={{ display: 'flex', flexDirection: 'column', gap: 6 }}>
|
||||
{inp.sources.map((s) => (
|
||||
|
|
@ -382,16 +399,15 @@ function InputsDrawer(ctx: ConsoleCtx, depth: DrawerDepth) {
|
|||
)}
|
||||
|
||||
<p style={{ fontSize: 9, color: 'var(--fg-dim)', margin: 0, lineHeight: 1.6 }}>
|
||||
The active sources concatenate into one input vector at the head of the spine.
|
||||
The active sources concatenate into one input vector at the head of the spine. The
|
||||
browser WASM is a {inp.engineInputSize}-input head (two 2-D controllers), so up to
|
||||
{' '}{inp.engineInputSize} axes each get their own genuine input dimension; unused axes
|
||||
stay neutral (0.5).
|
||||
{reshaping
|
||||
? ` The browser WASM is a fixed ${inp.engineInputSize}-input head (MLP<2,…>), so the
|
||||
${inp.axisCount} axes are blended down to ${inp.engineInputSize} (even→X / odd→Y mean).`
|
||||
: ''}{' '}
|
||||
{/* TODO(workstream F, docs/redesign/inputs-spec.md — "multiple WASM modules +
|
||||
warm-start"): give every axis its own genuine input dimension by (re)loading a
|
||||
WASM module whose MLP arity matches axisCount and warm-starting from the prior
|
||||
net. Deferred — the reduction lives in InputLayer.compose(). */}
|
||||
True per-axis dimensions land with the multi-WASM reshape (inputs-spec).
|
||||
? ` Right now ${inp.axisCount} axes exceed the ${inp.engineInputSize}-input head, so
|
||||
the surplus is blended down (chunked mean) — drop a source to give every axis its
|
||||
own dimension.`
|
||||
: ''}
|
||||
</p>
|
||||
</>
|
||||
)}
|
||||
|
|
|
|||
198
manifold/src/console/SandwichStage.tsx
Normal file
198
manifold/src/console/SandwichStage.tsx
Normal file
|
|
@ -0,0 +1,198 @@
|
|||
/**
|
||||
* SandwichStage — the N-dimensional "parameter sandwich" as a centre stage,
|
||||
* driven by the REAL engine (the same `EngineApi` instance the rest of the
|
||||
* console uses). Ported from the standalone /learn renderer.
|
||||
*
|
||||
* Each layer is one output channel of the MLP, sampled across the 2-D input
|
||||
* grid via `engine.inferBatch` → that surface IS the parameter's landscape.
|
||||
* The grid is re-sampled only when the net changes (`version`); a per-frame
|
||||
* rAF reads `engine.getOutputs()` for the live probe at the current input.
|
||||
*
|
||||
* Three-zone metaphor (input → MLP stack → outputs): this component owns the
|
||||
* MIDDLE zone. ConsoleApp docks a shrunken Manifold (left) and OutputStage
|
||||
* (right) around it. Drag the canvas to orbit; it lands top-down (a flat 2-D
|
||||
* map) and tilts into the 3-D stack. The stack is centred and grows to fill
|
||||
* the available height as more parameters are shown.
|
||||
*/
|
||||
import { useEffect, useRef } from 'react';
|
||||
import type { EngineApi } from '../engine';
|
||||
|
||||
const GRID = 12;
|
||||
const N = GRID;
|
||||
const SX = 300, SZ = 300, LAYER_GAP = 120, ZAMP = 92, MARGIN = 34;
|
||||
// Reference scale: each layer adds a fixed ~px gap (so the stack GROWS with the
|
||||
// layer count) until it would overflow, at which point the fit scale caps it.
|
||||
const REF_SCALE = 0.8;
|
||||
|
||||
export interface SandwichStageProps {
|
||||
engine: EngineApi;
|
||||
/** Engine version counter — re-sample the landscape grid when it bumps. */
|
||||
version: number;
|
||||
/** Current 2-D input (drives the probe column). */
|
||||
pos: [number, number];
|
||||
/** How many output channels to render as layers (top→down). */
|
||||
layerCount: number;
|
||||
/** Per-layer labels (the mode's param names). */
|
||||
names: string[];
|
||||
}
|
||||
|
||||
export function SandwichStage({ engine, version, pos, layerCount, names }: SandwichStageProps) {
|
||||
const wrapRef = useRef<HTMLDivElement>(null);
|
||||
const canvasRef = useRef<HTMLCanvasElement>(null);
|
||||
const gridRef = useRef<Float32Array | null>(null);
|
||||
const posRef = useRef(pos);
|
||||
posRef.current = pos;
|
||||
const Kref = useRef(layerCount);
|
||||
Kref.current = Math.max(1, layerCount);
|
||||
const namesRef = useRef(names);
|
||||
namesRef.current = names;
|
||||
const yaw = useRef(0);
|
||||
const pitch = useRef(Math.PI / 2); // top-down (looks 2-D) until you tilt it
|
||||
const view = useRef({ cw: 800, ch: 600, scale: 1, ceny: 0 });
|
||||
|
||||
const OUT = engine.architecture.outputSize;
|
||||
|
||||
// ---- re-sample the landscape grid whenever the net changes ----
|
||||
useEffect(() => {
|
||||
const pts: [number, number][] = [];
|
||||
for (let j = 0; j <= N; j++) for (let i = 0; i <= N; i++) pts.push([i / N, j / N]);
|
||||
try {
|
||||
gridRef.current = engine.inferBatch(pts);
|
||||
} catch {
|
||||
gridRef.current = null;
|
||||
}
|
||||
}, [engine, version]);
|
||||
|
||||
// ---- render loop ----
|
||||
useEffect(() => {
|
||||
const canvas = canvasRef.current;
|
||||
const wrap = wrapRef.current;
|
||||
if (!canvas || !wrap) return;
|
||||
const ctx = canvas.getContext('2d');
|
||||
if (!ctx) return;
|
||||
const DPR = Math.max(1, Math.min(2, window.devicePixelRatio || 1));
|
||||
|
||||
const computeView = () => {
|
||||
const v = view.current;
|
||||
const K = Kref.current;
|
||||
v.ceny = -(K - 1) * LAYER_GAP / 2 + ZAMP / 2; // centroid (centre vertically)
|
||||
let R = 1;
|
||||
for (let k = 0; k < K; k++) for (const z of [0, 1]) for (const gx of [-0.5, 0.5]) for (const gy of [-0.5, 0.5]) {
|
||||
const X = gx * SX, Y = (-k * LAYER_GAP + z * ZAMP) - v.ceny, Z = gy * SZ;
|
||||
const d = Math.sqrt(X * X + Y * Y + Z * Z);
|
||||
if (d > R) R = d;
|
||||
}
|
||||
const fit = (Math.min(v.cw, v.ch) / 2 - MARGIN) / R;
|
||||
v.scale = Math.min(REF_SCALE, fit); // grow with K, then cap to fit
|
||||
};
|
||||
const resize = () => {
|
||||
const r = wrap.getBoundingClientRect();
|
||||
view.current.cw = Math.max(160, Math.round(r.width));
|
||||
view.current.ch = Math.max(160, Math.round(r.height));
|
||||
canvas.width = view.current.cw * DPR;
|
||||
canvas.height = view.current.ch * DPR;
|
||||
ctx.setTransform(DPR, 0, 0, DPR, 0, 0);
|
||||
computeView();
|
||||
};
|
||||
const ro = new ResizeObserver(resize);
|
||||
ro.observe(wrap);
|
||||
resize();
|
||||
|
||||
const project = (gx: number, gy: number, zv: number, k: number) => {
|
||||
const v = view.current;
|
||||
const X = gx * SX, Y = (-k * LAYER_GAP + zv * ZAMP) - v.ceny, Z = -gy * SZ;
|
||||
const cyA = Math.cos(yaw.current), syA = Math.sin(yaw.current);
|
||||
const x1 = X * cyA - Z * syA, z1 = X * syA + Z * cyA;
|
||||
const cpA = Math.cos(pitch.current), spA = Math.sin(pitch.current);
|
||||
const y2 = Y * cpA - z1 * spA, z2 = Y * spA + z1 * cpA;
|
||||
return { x: v.cw / 2 + x1 * v.scale, y: v.ch / 2 - y2 * v.scale, depth: z2, wx: X, wy: Y, wz: Z };
|
||||
};
|
||||
const hueOf = (k: number, K: number) => (K > 1 ? 26 + (196 - 26) * k / (K - 1) : 26);
|
||||
const colCss = (k: number, K: number, a: number) => `hsla(${hueOf(k, K).toFixed(0)},85%,60%,${a})`;
|
||||
const LIGHT = (() => { const v = [0.35, 0.86, 0.36]; const m = Math.hypot(v[0], v[1], v[2]); return [v[0] / m, v[1] / m, v[2] / m]; })();
|
||||
const clamp = (x: number) => (x < 0 ? 0 : x > 1 ? 1 : x);
|
||||
|
||||
let raf = 0;
|
||||
const draw = () => {
|
||||
const v = view.current;
|
||||
const K = Kref.current;
|
||||
const grid = gridRef.current;
|
||||
computeView();
|
||||
ctx.clearRect(0, 0, v.cw, v.ch);
|
||||
const H = (k: number, i: number, j: number) => (grid ? grid[(j * (N + 1) + i) * OUT + k] : 0.5);
|
||||
|
||||
const quads: { a: any; b: any; d: any; e: any; hue: number; L: number; depth: number }[] = [];
|
||||
const labels: { x: number; y: number; hue: number; name: string }[] = [];
|
||||
for (let k = 0; k < K; k++) {
|
||||
const hue = hueOf(k, K);
|
||||
const P: { x: number; y: number; depth: number; wx: number; wy: number; wz: number }[][] = [];
|
||||
for (let j = 0; j <= N; j++) { P[j] = []; for (let i = 0; i <= N; i++) P[j][i] = project(i / N - 0.5, j / N - 0.5, H(k, i, j), k); }
|
||||
for (let j = 0; j < N; j++) for (let i = 0; i < N; i++) {
|
||||
const a = P[j][i], b = P[j][i + 1], d = P[j + 1][i + 1], e = P[j + 1][i];
|
||||
const ux = d.wx - a.wx, uy = d.wy - a.wy, uz = d.wz - a.wz, vx = b.wx - e.wx, vy = b.wy - e.wy, vz = b.wz - e.wz;
|
||||
const nx = uy * vz - uz * vy, ny = uz * vx - ux * vz, nz = ux * vy - uy * vx, nm = Math.hypot(nx, ny, nz) || 1;
|
||||
const L = clamp(0.32 + 0.62 * Math.abs((nx * LIGHT[0] + ny * LIGHT[1] + nz * LIGHT[2]) / nm));
|
||||
quads.push({ a, b, d, e, hue, L, depth: (a.depth + b.depth + d.depth + e.depth) / 4 });
|
||||
}
|
||||
const lp = project(-0.5, 0.5, H(k, 0, N), k);
|
||||
labels.push({ x: lp.x, y: lp.y, hue, name: namesRef.current[k] || `p${k}` });
|
||||
}
|
||||
quads.sort((p, q) => q.depth - p.depth);
|
||||
for (const Q of quads) {
|
||||
ctx.fillStyle = `hsla(${Q.hue},80%,${(26 + Q.L * 44).toFixed(0)}%,0.66)`;
|
||||
ctx.strokeStyle = `hsla(${Q.hue},80%,82%,0.18)`; ctx.lineWidth = 0.6;
|
||||
ctx.beginPath(); ctx.moveTo(Q.a.x, Q.a.y); ctx.lineTo(Q.b.x, Q.b.y); ctx.lineTo(Q.d.x, Q.d.y); ctx.lineTo(Q.e.x, Q.e.y); ctx.closePath();
|
||||
ctx.fill(); ctx.stroke();
|
||||
}
|
||||
ctx.font = '11px var(--font-mono, monospace)'; ctx.textAlign = 'right'; ctx.textBaseline = 'middle';
|
||||
for (const Lb of labels) { ctx.fillStyle = `hsla(${Lb.hue},85%,74%,0.95)`; ctx.fillText(Lb.name, Lb.x - 6, Lb.y); }
|
||||
|
||||
// probe column at the current input, marker on each slab from the LIVE output
|
||||
const out = engine.getOutputs();
|
||||
const gx = posRef.current[0] - 0.5, gy = posRef.current[1] - 0.5;
|
||||
const top = project(gx, gy, 1.18, 0), bot = project(gx, gy, -0.18, K - 1);
|
||||
ctx.strokeStyle = 'rgba(255,255,255,0.3)'; ctx.lineWidth = 1; ctx.setLineDash([3, 4]);
|
||||
ctx.beginPath(); ctx.moveTo(top.x, top.y); ctx.lineTo(bot.x, bot.y); ctx.stroke(); ctx.setLineDash([]);
|
||||
for (let k = 0; k < K; k++) {
|
||||
const val = clamp(out[k] ?? 0.5);
|
||||
const p = project(gx, gy, val, k), fl = project(gx, gy, 0, k);
|
||||
ctx.strokeStyle = colCss(k, K, 0.5); ctx.lineWidth = 1;
|
||||
ctx.beginPath(); ctx.moveTo(p.x, p.y); ctx.lineTo(fl.x, fl.y); ctx.stroke();
|
||||
ctx.fillStyle = colCss(k, K, 1); ctx.shadowColor = colCss(k, K, 1); ctx.shadowBlur = 12;
|
||||
ctx.beginPath(); ctx.arc(p.x, p.y, 6, 0, Math.PI * 2); ctx.fill(); ctx.shadowBlur = 0;
|
||||
ctx.fillStyle = 'var(--bg, #0d0d0d)'; ctx.beginPath(); ctx.arc(p.x, p.y, 2.2, 0, Math.PI * 2); ctx.fill();
|
||||
}
|
||||
raf = requestAnimationFrame(draw);
|
||||
};
|
||||
raf = requestAnimationFrame(draw);
|
||||
return () => { cancelAnimationFrame(raf); ro.disconnect(); };
|
||||
}, [engine, OUT]);
|
||||
|
||||
// ---- drag to orbit (yaw + pitch; pitch up to π/2 = exact top-down) ----
|
||||
const rotating = useRef(false);
|
||||
const last = useRef({ x: 0, y: 0 });
|
||||
return (
|
||||
<div ref={wrapRef} style={{ position: 'absolute', inset: 0, overflow: 'hidden' }}>
|
||||
<canvas
|
||||
ref={canvasRef}
|
||||
style={{ position: 'absolute', inset: 0, width: '100%', height: '100%', cursor: 'grab', touchAction: 'none' }}
|
||||
onPointerDown={(e) => {
|
||||
rotating.current = true;
|
||||
(e.currentTarget as HTMLCanvasElement).setPointerCapture?.(e.pointerId);
|
||||
last.current = { x: e.clientX, y: e.clientY };
|
||||
}}
|
||||
onPointerMove={(e) => {
|
||||
if (!rotating.current) return;
|
||||
yaw.current += (e.clientX - last.current.x) * 0.01;
|
||||
pitch.current = Math.max(0.06, Math.min(Math.PI / 2, pitch.current + (e.clientY - last.current.y) * 0.008));
|
||||
last.current = { x: e.clientX, y: e.clientY };
|
||||
}}
|
||||
onPointerUp={() => { rotating.current = false; }}
|
||||
onPointerCancel={() => { rotating.current = false; }}
|
||||
/>
|
||||
<div style={{ position: 'absolute', top: 10, left: '50%', transform: 'translateX(-50%)', fontSize: 9, letterSpacing: '0.12em', color: 'var(--fg-dim)', pointerEvents: 'none', fontFamily: 'var(--font-mono)' }}>
|
||||
PARAMETER SANDWICH · drag to rotate · top-down = the flat 2-D map
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
|
@ -50,6 +50,19 @@ export function ModeIcon({ size = 18, style }: IconProps) {
|
|||
);
|
||||
}
|
||||
|
||||
/** Sandwich — stacked parameter-landscape slabs (the layer toggle). */
|
||||
export function SandwichIcon({ size = 18, style }: IconProps) {
|
||||
return svg(
|
||||
size,
|
||||
style,
|
||||
<>
|
||||
<path d="M3 6.5 12 3l9 3.5L12 10 3 6.5Z" />
|
||||
<path d="M3 12 12 15.5 21 12" />
|
||||
<path d="M3 17 12 20.5 21 17" />
|
||||
</>,
|
||||
);
|
||||
}
|
||||
|
||||
/** Learning — a brain-ish node graph. */
|
||||
export function LearningIcon({ size = 18, style }: IconProps) {
|
||||
return svg(
|
||||
|
|
|
|||
|
|
@ -164,9 +164,13 @@ export class EngineApi {
|
|||
this.spine.setInput(x, y);
|
||||
}
|
||||
|
||||
/** Set an arbitrary input vector (first two used as XY for the fixed 2→N MLP). */
|
||||
/**
|
||||
* Set the full composed input vector from the modular InputLayer. Axes 0,1
|
||||
* are the primary controller; axes 2+ feed additional controllers (the WASM
|
||||
* head is 4-wide = two 2-D controllers). Omitted axes stay neutral.
|
||||
*/
|
||||
setInputs(arr: ReadonlyArray<number>): void {
|
||||
this.spine.setInput(arr[0] ?? 0.5, arr[1] ?? 0.5);
|
||||
this.spine.setInputs(arr);
|
||||
}
|
||||
|
||||
/** Live post-ML output vector (reused buffer — read, don't retain). */
|
||||
|
|
|
|||
|
|
@ -18,6 +18,7 @@
|
|||
* Float32Array threaded through the output pipeline and handed to the backend.
|
||||
*/
|
||||
|
||||
import { clamp } from './curves';
|
||||
import {
|
||||
defaultInputConfig,
|
||||
defaultInputState,
|
||||
|
|
@ -91,6 +92,11 @@ export class Spine implements EngineSink {
|
|||
|
||||
// Reused per-frame buffers — NO per-frame allocation in the hot path.
|
||||
private rawInput: [number, number] = [0.5, 0.5];
|
||||
// Full input vector fed to the MLP. Axes 0,1 = primary pad (post input
|
||||
// pipeline); axes 2+ = secondary controller(s), or the neutral 0.5 when no
|
||||
// second controller is active (single-controller mode). Sized to the WASM
|
||||
// head's input arity; defaults all-neutral.
|
||||
private inputs: number[] = [0.5, 0.5];
|
||||
// Last raw input, so `EngineApi.process()` can re-tick after a weight change.
|
||||
lastRawX = 0.5;
|
||||
lastRawY = 0.5;
|
||||
|
|
@ -108,7 +114,11 @@ export class Spine implements EngineSink {
|
|||
setState(patch: EngineStatePatch): void {
|
||||
let changed = false;
|
||||
const s = this.state_;
|
||||
if (patch.inputSize !== undefined && patch.inputSize !== s.inputSize) { s.inputSize = patch.inputSize; changed = true; }
|
||||
if (patch.inputSize !== undefined && patch.inputSize !== s.inputSize) {
|
||||
s.inputSize = patch.inputSize;
|
||||
this.resizeInputs_(patch.inputSize);
|
||||
changed = true;
|
||||
}
|
||||
if (patch.outputSize !== undefined && patch.outputSize !== s.outputSize) {
|
||||
s.outputSize = patch.outputSize;
|
||||
// Resize hot buffers to the resolved output size.
|
||||
|
|
@ -155,6 +165,16 @@ export class Spine implements EngineSink {
|
|||
if (this.routedBuf === null || this.routedBuf.length !== iml.architecture.outputSize) {
|
||||
this.routedBuf = new Float32Array(iml.architecture.outputSize);
|
||||
}
|
||||
this.resizeInputs_(iml.architecture.inputSize);
|
||||
}
|
||||
|
||||
/** Resize the full input vector to the WASM head arity, neutral-filling. */
|
||||
private resizeInputs_(size: number): void {
|
||||
if (size < 1) size = 1;
|
||||
if (this.inputs.length === size) return;
|
||||
const next = new Array<number>(size).fill(0.5);
|
||||
for (let i = 0; i < Math.min(size, this.inputs.length); i++) next[i] = this.inputs[i]!;
|
||||
this.inputs = next;
|
||||
}
|
||||
|
||||
setBackendSend(backendSend: BackendSend | null): void {
|
||||
|
|
@ -169,6 +189,22 @@ export class Spine implements EngineSink {
|
|||
* Returns the routed buffer (live, reused — do not retain across calls).
|
||||
*/
|
||||
setInput(x: number, y: number): Float32Array | null {
|
||||
return this.tick_(x, y, null);
|
||||
}
|
||||
|
||||
/**
|
||||
* Drive the full composed input vector (from the modular InputLayer). Axes
|
||||
* 0,1 are the primary controller (run through the input pipeline); axes 2+
|
||||
* are secondary controllers, written through with a plain [0,1] clamp. Any
|
||||
* axis the caller omits keeps its previous value (neutral 0.5 by default), so
|
||||
* single-controller mode — where only axes 0,1 are supplied — leaves the
|
||||
* second pair pinned at the neutral value.
|
||||
*/
|
||||
setInputs(arr: ReadonlyArray<number>): Float32Array | null {
|
||||
return this.tick_(arr[0] ?? 0.5, arr[1] ?? 0.5, arr);
|
||||
}
|
||||
|
||||
private tick_(x: number, y: number, arr: ReadonlyArray<number> | null): Float32Array | null {
|
||||
const iml = this.iml;
|
||||
if (!iml) return null;
|
||||
|
||||
|
|
@ -176,17 +212,25 @@ export class Spine implements EngineSink {
|
|||
const dt = this.lastTickMs > 0 ? (now - this.lastTickMs) / 1000 : 1 / 60;
|
||||
this.lastTickMs = now;
|
||||
|
||||
// 1. processed (pure input pipeline)
|
||||
// 1. processed (pure input pipeline) — primary controller, axes 0,1.
|
||||
this.rawInput[0] = x;
|
||||
this.rawInput[1] = y;
|
||||
this.lastRawX = x;
|
||||
this.lastRawY = y;
|
||||
const proc = processInput(this.rawInput, this.inputConfig, this.inputState, dt);
|
||||
this.inputState = proc.state;
|
||||
this.inputs[0] = proc.x;
|
||||
this.inputs[1] = proc.y;
|
||||
// Secondary axes (2+): pass through with a [0,1] clamp. Omitted axes keep
|
||||
// their prior value (neutral 0.5 in single-controller mode).
|
||||
if (arr) {
|
||||
for (let i = 2; i < this.inputs.length; i++) {
|
||||
if (i < arr.length) this.inputs[i] = clamp(arr[i]!, 0, 1);
|
||||
}
|
||||
}
|
||||
|
||||
// 2. ml (inference into the reused buffer; no alloc)
|
||||
iml.setInput(0, proc.x);
|
||||
iml.setInput(1, proc.y);
|
||||
for (let i = 0; i < this.inputs.length; i++) iml.setInput(i, this.inputs[i]!);
|
||||
iml.processInto(this.mlBuf);
|
||||
// Mirror to liveOutputs for imperative reads + bump.
|
||||
this.liveOutputs.set(this.mlBuf.subarray(0, this.liveOutputs.length));
|
||||
|
|
|
|||
|
|
@ -13,24 +13,19 @@
|
|||
* its value), but routing everything through one compose path keeps sources
|
||||
* composable and the channel layout coherent.
|
||||
*
|
||||
* ── Arity mismatch (the WASM reshape TODO) ──────────────────────────────────
|
||||
* The browser WASM is fixed at MLP<2, …, 126> — a TWO-input head. When the
|
||||
* composed vector has > 2 axes (double-stick gamepad = 4, MIDI learn-map = many)
|
||||
* we must reduce to 2 to feed today's engine. We do NOT fake a wider net.
|
||||
* ── Arity: 4-input head ─────────────────────────────────────────────────────
|
||||
* The browser WASM head is MLP<4, …, 126> — TWO 2-D controllers. The primary
|
||||
* controller (on-screen pad) occupies axes 0,1; an optional second controller
|
||||
* (gamepad) occupies axes 2,3. `compose()` passes axes through 1:1 when the
|
||||
* head is at least as wide as the composed axis count (the common cases —
|
||||
* single pad = 2, pad + single-stick gamepad = 4), padding any unused axes with
|
||||
* the neutral 0.5. Single-controller mode is therefore just "no second source":
|
||||
* axes 2,3 stay neutral, slicing the 4-D manifold at the centre of controller 2.
|
||||
*
|
||||
* chosen reduction (this pass): pairwise BLEND.
|
||||
* inX = mean(axis[0], axis[2], axis[4], …) // even axes
|
||||
* inY = mean(axis[1], axis[3], axis[5], …) // odd axes
|
||||
* so a single stick passes straight through (axis0→X, axis1→Y), a double
|
||||
* stick averages L/R into one XY, and MIDI axes fold into X/Y by parity.
|
||||
*
|
||||
* TODO(workstream F, docs/redesign/inputs-spec.md — "multiple WASM modules +
|
||||
* warm-start"): the real fix is to (re)load a WASM module whose MLP input arity
|
||||
* matches the composed axis count and warm-start its weights from the prior net,
|
||||
* so every axis gets its own genuine input dimension instead of being blended.
|
||||
* That is a larger build (multiple .wasm artefacts or a runtime-variadic head)
|
||||
* and is deliberately deferred — this layer is wired so that swapping the
|
||||
* reduction for a true reshape is a localised change in `compose()`.
|
||||
* Over-subscription (composed axes > head width — e.g. pad + dual-stick gamepad
|
||||
* = 6, or a large MIDI learn-map) still falls back to a chunked-mean blend; the
|
||||
* net is never rebuilt. That blend is the documented lossy path for unusual
|
||||
* source combinations, not the default.
|
||||
*/
|
||||
import type { InputAction, InputSource } from './types';
|
||||
|
||||
|
|
@ -147,15 +142,15 @@ export class InputLayer {
|
|||
}
|
||||
|
||||
/**
|
||||
* Reduce the composed N-axis vector to the engine's input arity.
|
||||
* Map the composed N-axis vector to the engine's input arity.
|
||||
*
|
||||
* For the fixed 2-input WASM, fold by parity (even→X, odd→Y) via mean. If a
|
||||
* future multi-module engine reports inputSize >= n, this passes axes through
|
||||
* 1:1 (truncated/padded) — the seam where the real reshape lands.
|
||||
* Common case: the 4-input head is at least as wide as the composed axes, so
|
||||
* pass through 1:1 and pad unused axes with the neutral 0.5. Over-subscription
|
||||
* (axes > head width) folds by parity / chunked mean — the lossy fallback.
|
||||
*/
|
||||
private compose(n: number, inputSize: number): number[] {
|
||||
if (inputSize >= n) {
|
||||
// True passthrough path (future multi-module head). Pad with 0.5.
|
||||
// Passthrough: each axis gets its own input dimension. Pad with 0.5.
|
||||
const out = new Array<number>(inputSize);
|
||||
for (let i = 0; i < inputSize; i++) out[i] = i < n ? this.vector[i] : 0.5;
|
||||
return out;
|
||||
|
|
|
|||
|
|
@ -141,7 +141,7 @@ class ModeBase {
|
|||
// Run an inference at default inputs so engine has params on first
|
||||
// process() call, even if no input has been touched.
|
||||
for (std::size_t i = 0u; i < NInputs; ++i) {
|
||||
ml_.set_input(i, input_channels_[i]);
|
||||
ml_.set_input(i, effective_input(i));
|
||||
}
|
||||
ml_.process();
|
||||
if constexpr (kRouteOutputsToEngine) {
|
||||
|
|
@ -160,13 +160,42 @@ class ModeBase {
|
|||
input_dirty_ = true;
|
||||
}
|
||||
|
||||
// ---- Input neutralization (single/double controller toggle) ----
|
||||
//
|
||||
// A pinned channel feeds `pin_value_` (neutral, default 0.5) to the MLP
|
||||
// instead of its live value, without rebuilding/resizing the network.
|
||||
// Glue toggles which channels are pinned (e.g. single-joystick mode pins
|
||||
// the second 2D controller's two channels). The stored live value is left
|
||||
// untouched, so unpinning resumes from the controller's current position.
|
||||
NISPS_FORCE_INLINE void set_input_pinned(std::size_t idx, bool pinned) noexcept {
|
||||
if (idx >= NInputs) return;
|
||||
input_pinned_[idx] = pinned;
|
||||
input_dirty_ = true;
|
||||
}
|
||||
NISPS_FORCE_INLINE bool is_input_pinned(std::size_t idx) const noexcept {
|
||||
return idx < NInputs && input_pinned_[idx];
|
||||
}
|
||||
NISPS_FORCE_INLINE void set_pin_value(float v) noexcept {
|
||||
if (v < 0.f) v = 0.f;
|
||||
else if (v > 1.f) v = 1.f;
|
||||
pin_value_ = v;
|
||||
input_dirty_ = true;
|
||||
}
|
||||
float pin_value() const noexcept { return pin_value_; }
|
||||
|
||||
// Effective value fed to the MLP for channel i (pin override applied).
|
||||
NISPS_FORCE_INLINE float effective_input(std::size_t i) const noexcept {
|
||||
return input_pinned_[i] ? pin_value_ : input_channels_[i];
|
||||
}
|
||||
|
||||
NISPS_HOT void tick_control() noexcept {
|
||||
if constexpr (requires(Derived& d) { d.on_pre_inference(); }) {
|
||||
static_cast<Derived&>(*this).on_pre_inference();
|
||||
}
|
||||
// Forward (possibly Derived-mutated) channels into the MLP.
|
||||
// Forward (possibly Derived-mutated) channels into the MLP, applying
|
||||
// the per-channel pin override.
|
||||
for (std::size_t i = 0u; i < NInputs; ++i) {
|
||||
ml_.set_input(i, input_channels_[i]);
|
||||
ml_.set_input(i, effective_input(i));
|
||||
}
|
||||
ml_.process();
|
||||
if constexpr (kRouteOutputsToEngine) {
|
||||
|
|
@ -235,6 +264,8 @@ class ModeBase {
|
|||
EngineT engine_{};
|
||||
MLPType ml_;
|
||||
std::array<float, NInputs> input_channels_{};
|
||||
std::array<bool, NInputs> input_pinned_{}; // false => live
|
||||
float pin_value_ = 0.5f; // neutral
|
||||
bool input_dirty_ = false;
|
||||
std::size_t voice_space_idx_ = 0u;
|
||||
RingBuffer<ControlEvent, kModeEventBufferSize> events_{};
|
||||
|
|
|
|||
|
|
@ -32,10 +32,13 @@ The MLP class template is parametrised on `(input_size, hidden1, hidden2,
|
|||
hidden3, output_size)`. WASM cannot recompile templates at runtime, so
|
||||
this build instantiates exactly ONE configuration:
|
||||
|
||||
nisps::ml::MLP<2, 10, 14, 18, 126>
|
||||
nisps::ml::MLP<4, 10, 14, 18, 126>
|
||||
|
||||
That serves the playground use case (2-D joystick → up to 126 synth
|
||||
parameters). `nisps_ml_create()` accepts caller-supplied dimensions for
|
||||
That serves the manifold/playground use case: two 2-D controllers (primary
|
||||
pad on axes 0,1; optional second controller — e.g. gamepad — on axes 2,3)
|
||||
→ up to 126 synth parameters. Single-controller mode pins axes 2,3 to the
|
||||
neutral 0.5 on the JS side, so the net is never rebuilt when toggling
|
||||
between one and two controllers. `nisps_ml_create()` accepts caller-supplied dimensions for
|
||||
forward compatibility but currently ignores them — see comment at the top
|
||||
of `bindings.cpp`. The schemas in `schemas/modes/*.json` use up to
|
||||
`output_size=126`; modes whose output_size is < 126 simply ignore the
|
||||
|
|
|
|||
|
|
@ -5,24 +5,26 @@
|
|||
// 2. AudioWorklet processor (playground/src/audio/worklet/...) — engine
|
||||
// calls. (Each instance owns its own WASM module instance.)
|
||||
//
|
||||
// FIXED-ARCHITECTURE LIMITATION (VERY IMPORTANT)
|
||||
// ----------------------------------------------
|
||||
// COMPILE-TIME ARCHITECTURE
|
||||
// -------------------------
|
||||
// The C++ MLP class is templated on layer sizes (architecture.md §4.1, §6.2).
|
||||
// We instantiate ONE concrete configuration here:
|
||||
//
|
||||
// using DefaultMLP = nisps::ml::MLP<2, 10, 14, 18, 126>;
|
||||
// using DefaultMLP = nisps::ml::MLP<4, 10, 14, 18, 126>;
|
||||
//
|
||||
// This was chosen as the union of the playground use case (2-D joystick →
|
||||
// 126 synth params) and the largest hidden-layer footprint that still fits
|
||||
// firmware budgets. `nisps_ml_create()` accepts caller-supplied input_size,
|
||||
// output_size, hidden[], n_hidden but ONLY validates them against the
|
||||
// compile-time defaults — extra inputs/outputs are clipped at the boundary.
|
||||
// If the caller passes incompatible dimensions we still create the module:
|
||||
// extra inputs are zero-padded, extra outputs are truncated, and the
|
||||
// hidden-layer override is silently ignored.
|
||||
// FOUR inputs = two 2-D controllers (primary on-screen pad on axes 0,1; an
|
||||
// optional second controller — gamepad — on axes 2,3). Single-controller mode
|
||||
// pins the second pair to the neutral value 0.5 on the JS side (see
|
||||
// manifold/src/engine/spine.ts), so the network is never rebuilt/resized when
|
||||
// toggling between one and two controllers. 126 outputs covers C15 and every
|
||||
// current schema.
|
||||
//
|
||||
// Future work: ship multiple WASM modules (one per common architecture) or
|
||||
// rebuild on demand. See architecture.md "open questions" — Stream 7 punts.
|
||||
// `nisps_ml_create()` accepts caller-supplied input_size, output_size,
|
||||
// hidden[], n_hidden but ONLY validates them against the compile-time
|
||||
// defaults — extra inputs/outputs are clipped at the boundary. If the caller
|
||||
// passes incompatible dimensions we still create the module: extra inputs are
|
||||
// zero-padded, extra outputs are truncated, and the hidden-layer override is
|
||||
// silently ignored. The TS side reads the real dims back via nisps_ml_describe.
|
||||
//
|
||||
// WIRE FORMAT FOR WEIGHTS
|
||||
// -----------------------
|
||||
|
|
@ -78,14 +80,15 @@ namespace {
|
|||
// Compile-time default architecture. See header comment.
|
||||
//
|
||||
// Choice rationale:
|
||||
// * 2 inputs — playground virtual joystick (X, Y).
|
||||
// * 4 inputs — two 2-D controllers (primary pad axes 0,1; optional
|
||||
// second controller axes 2,3). Single-controller mode pins axes 2,3 to 0.5.
|
||||
// * [10, 14, 18] hidden — covers the largest schema layouts in
|
||||
// `schemas/modes/*.json` (channel_strip variants, verb_fx, breakor,
|
||||
// elysiamorf, memlcelium).
|
||||
// * 126 outputs — enough for the C15 mode and any current schema.
|
||||
//
|
||||
// The MLP also has dataset slots, loss history etc. — see mlp.hpp.
|
||||
using DefaultMLP = nisps::ml::MLP<2u, 10u, 14u, 18u, 126u>;
|
||||
using DefaultMLP = nisps::ml::MLP<4u, 10u, 14u, 18u, 126u>;
|
||||
|
||||
constexpr std::size_t kDefaultInputs = DefaultMLP::kInput;
|
||||
constexpr std::size_t kDefaultOutputs = DefaultMLP::kOutput;
|
||||
|
|
|
|||
Binary file not shown.
|
|
@ -82,3 +82,12 @@ set -x
|
|||
|
||||
echo "[build-wasm] wrote $OUT/nisps.js + $OUT/nisps.wasm"
|
||||
ls -lh "$OUT/nisps.js" "$OUT/nisps.wasm"
|
||||
|
||||
# Mirror into the manifold app's public/ — it serves its own copy (and its
|
||||
# deploy webhook builds manifold/), so both must stay in lockstep with the
|
||||
# parity-checked artifacts above.
|
||||
MANIFOLD_OUT="$ROOT/manifold/public"
|
||||
if [[ -d "$MANIFOLD_OUT" ]]; then
|
||||
cp "$OUT/nisps.js" "$OUT/nisps.wasm" "$MANIFOLD_OUT/"
|
||||
echo "[build-wasm] mirrored nisps.{js,wasm} → $MANIFOLD_OUT"
|
||||
fi
|
||||
|
|
|
|||
|
|
@ -13,7 +13,7 @@
|
|||
//
|
||||
// What we cover
|
||||
// -------------
|
||||
// 1. ML: seed=42, draw_weights(0.5), set_input(0.25, 0.75), process.
|
||||
// 1. ML: seed=42, draw_weights(0.5), set_input(0.25, 0.75, 0.40, 0.60), process.
|
||||
// → 126 outputs + 12 weights sampled at known offsets.
|
||||
// 2. ML training: 3 examples added, train(0.3, 50, 0), capture loss + outputs.
|
||||
// 3. PAFSynth engine: seed-equivalent setup (params=0.5), 128-sample run on
|
||||
|
|
@ -21,7 +21,7 @@
|
|||
// 4. ChannelStrip engine: identical methodology.
|
||||
//
|
||||
// We use the EXACT SAME compile-time MLP architecture as the WASM build:
|
||||
// MLP<2, 10, 14, 18, 126>
|
||||
// MLP<4, 10, 14, 18, 126>
|
||||
//
|
||||
// Output blob format
|
||||
// ------------------
|
||||
|
|
@ -62,7 +62,7 @@
|
|||
|
||||
namespace {
|
||||
|
||||
using ParityMLP = nisps::ml::MLP<2u, 10u, 14u, 18u, 126u>;
|
||||
using ParityMLP = nisps::ml::MLP<4u, 10u, 14u, 18u, 126u>;
|
||||
|
||||
// The WASM bindings (nisps/wasm/bindings.cpp) sign-extend the 32-bit JS
|
||||
// seed via `s ^ (s << 32)`. To get bit-equal output between native and
|
||||
|
|
@ -73,6 +73,9 @@ constexpr std::uint64_t kSeed = static_cast<std::uint64_t>(kSeed32)
|
|||
^ (static_cast<std::uint64_t>(kSeed32) << 32);
|
||||
constexpr float kInputX = 0.25f;
|
||||
constexpr float kInputY = 0.75f;
|
||||
// Second controller (axes 2,3) — exercised so the 4-input head is fully probed.
|
||||
constexpr float kInputZ = 0.40f;
|
||||
constexpr float kInputW = 0.60f;
|
||||
constexpr float kSampleRate = 48000.0f;
|
||||
constexpr std::size_t kSynthFrames = 128u;
|
||||
|
||||
|
|
@ -118,6 +121,8 @@ int main(int argc, char** argv) {
|
|||
mlp.draw_weights(0.5f);
|
||||
mlp.set_input(0u, kInputX);
|
||||
mlp.set_input(1u, kInputY);
|
||||
mlp.set_input(2u, kInputZ);
|
||||
mlp.set_input(3u, kInputW);
|
||||
mlp.process();
|
||||
{
|
||||
const auto outs = mlp.outputs();
|
||||
|
|
@ -133,8 +138,8 @@ int main(int argc, char** argv) {
|
|||
}
|
||||
|
||||
// ---- Stage 2: training ----
|
||||
constexpr std::array<std::array<float, 2u>, 3u> features = {{
|
||||
{{0.1f, 0.9f}}, {{0.5f, 0.5f}}, {{0.9f, 0.1f}},
|
||||
constexpr std::array<std::array<float, 4u>, 3u> features = {{
|
||||
{{0.1f, 0.9f, 0.3f, 0.7f}}, {{0.5f, 0.5f, 0.5f, 0.5f}}, {{0.9f, 0.1f, 0.7f, 0.3f}},
|
||||
}};
|
||||
auto label_for = [](std::size_t i) {
|
||||
std::array<float, 126u> out{};
|
||||
|
|
@ -146,13 +151,15 @@ int main(int argc, char** argv) {
|
|||
};
|
||||
for (std::size_t i = 0; i < features.size(); ++i) {
|
||||
const auto label = label_for(i);
|
||||
mlp.add_example(std::span<const float>(features[i].data(), 2u),
|
||||
mlp.add_example(std::span<const float>(features[i].data(), 4u),
|
||||
std::span<const float>(label.data(), 126u));
|
||||
}
|
||||
const float final_loss = mlp.train(0.3f, 50u, 0.0f);
|
||||
|
||||
mlp.set_input(0u, kInputX);
|
||||
mlp.set_input(1u, kInputY);
|
||||
mlp.set_input(2u, kInputZ);
|
||||
mlp.set_input(3u, kInputW);
|
||||
mlp.process();
|
||||
{
|
||||
const auto outs = mlp.outputs();
|
||||
|
|
|
|||
|
|
@ -43,6 +43,8 @@ const VERSION = 3; // v3 adds stage 5d (ExploreAndPlace lifecycle)
|
|||
const SEED = 42 >>> 0;
|
||||
const INPUT_X = 0.25;
|
||||
const INPUT_Y = 0.75;
|
||||
const INPUT_Z = 0.40; // second controller axes 2,3 (must match parity_check.cpp)
|
||||
const INPUT_W = 0.60;
|
||||
const SAMPLE_RATE = 48000;
|
||||
const SYNTH_FRAMES = 128;
|
||||
const PROBE_IDX = [0, 5, 19, 31, 73, 137, 251, 491, 999, 1583, 2401, 3289];
|
||||
|
|
@ -191,8 +193,8 @@ async function main() {
|
|||
api.describe(dimsBuf);
|
||||
const dims = new Int32Array(Module.HEAP32.buffer, dimsBuf, 6).slice();
|
||||
api.free(dimsBuf);
|
||||
// Expect: [2, 10, 14, 18, 126, 4]
|
||||
const expectedDims = [2, 10, 14, 18, 126, 4];
|
||||
// Expect: [inputs, h1, h2, h3, outputs, numLayers] = [4, 10, 14, 18, 126, 4]
|
||||
const expectedDims = [4, 10, 14, 18, 126, 4];
|
||||
for (let i = 0; i < expectedDims.length; ++i) {
|
||||
if (dims[i] !== expectedDims[i]) {
|
||||
console.error(`[parity_wasm] WASM build has dim[${i}]=${dims[i]}, native expected ${expectedDims[i]}`);
|
||||
|
|
@ -203,10 +205,12 @@ async function main() {
|
|||
const N_OUT = dims[4];
|
||||
|
||||
// --- Stage 1: ML inference ---
|
||||
const ml = api.create(2, N_OUT, 0, 0, SEED);
|
||||
const ml = api.create(4, N_OUT, 0, 0, SEED);
|
||||
api.drawWeights(ml, 0.5);
|
||||
api.setInput(ml, 0, INPUT_X);
|
||||
api.setInput(ml, 1, INPUT_Y);
|
||||
api.setInput(ml, 2, INPUT_Z);
|
||||
api.setInput(ml, 3, INPUT_W);
|
||||
api.process(ml);
|
||||
const outsStage1 = getOutputsCopy(api, ml, N_OUT);
|
||||
|
||||
|
|
@ -216,9 +220,9 @@ async function main() {
|
|||
|
||||
// --- Stage 2: training ---
|
||||
const features = [
|
||||
[0.1, 0.9],
|
||||
[0.5, 0.5],
|
||||
[0.9, 0.1],
|
||||
[0.1, 0.9, 0.3, 0.7],
|
||||
[0.5, 0.5, 0.5, 0.5],
|
||||
[0.9, 0.1, 0.7, 0.3],
|
||||
];
|
||||
const labelFor = (i) => {
|
||||
const out = new Float32Array(N_OUT);
|
||||
|
|
@ -226,12 +230,14 @@ async function main() {
|
|||
for (let j = 0; j < N_OUT; ++j) out[j] = a + 0.005 * j;
|
||||
return out;
|
||||
};
|
||||
const featBuf = api.malloc(2 * 4);
|
||||
const featF32 = new Float32Array(api.HEAPF32.buffer, featBuf, 2);
|
||||
const featBuf = api.malloc(4 * 4);
|
||||
const featF32 = new Float32Array(api.HEAPF32.buffer, featBuf, 4);
|
||||
const labelBuf = api.malloc(N_OUT * 4);
|
||||
for (let i = 0; i < features.length; ++i) {
|
||||
featF32[0] = features[i][0];
|
||||
featF32[1] = features[i][1];
|
||||
featF32[2] = features[i][2];
|
||||
featF32[3] = features[i][3];
|
||||
const label = labelFor(i);
|
||||
new Float32Array(api.HEAPF32.buffer, labelBuf, N_OUT).set(label);
|
||||
api.addExample(ml, featBuf, labelBuf);
|
||||
|
|
@ -243,6 +249,8 @@ async function main() {
|
|||
|
||||
api.setInput(ml, 0, INPUT_X);
|
||||
api.setInput(ml, 1, INPUT_Y);
|
||||
api.setInput(ml, 2, INPUT_Z);
|
||||
api.setInput(ml, 3, INPUT_W);
|
||||
api.process(ml);
|
||||
const outsStage2 = getOutputsCopy(api, ml, N_OUT);
|
||||
|
||||
|
|
@ -301,6 +309,8 @@ async function main() {
|
|||
api.feedbackUndo(ml); // pop nudge
|
||||
api.setInput(ml, 0, INPUT_X);
|
||||
api.setInput(ml, 1, INPUT_Y);
|
||||
api.setInput(ml, 2, INPUT_Z);
|
||||
api.setInput(ml, 3, INPUT_W);
|
||||
api.process(ml);
|
||||
api.feedbackLike(ml); // begin place: freeze scratchpad output
|
||||
{
|
||||
|
|
|
|||
|
|
@ -88,6 +88,62 @@ NISPS_TEST(paf_synth_mode_input_clamping) {
|
|||
m.set_input(99u, 0.5f);
|
||||
}
|
||||
|
||||
NISPS_TEST(paf_synth_mode_pinned_inputs_neutralized) {
|
||||
// Single-joystick mode pins the second 2D controller's channels (2,3).
|
||||
// A pinned channel must feed the neutral pin value (default 0.5) to the
|
||||
// MLP regardless of its live value, and pinning must not resize the model.
|
||||
PAFSynthMode m;
|
||||
m.setup(48000.f);
|
||||
NISPS_EXPECT(m.pin_value() == 0.5f);
|
||||
|
||||
// Reference: channels 2,3 driven live to 0.5 (the neutral value).
|
||||
m.set_input(0, 0.2f);
|
||||
m.set_input(1, 0.8f);
|
||||
m.set_input(2, 0.5f);
|
||||
m.set_input(3, 0.5f);
|
||||
m.tick_control();
|
||||
std::array<float, 33> ref{};
|
||||
{
|
||||
const auto outs = m.ml().outputs();
|
||||
for (std::size_t i = 0u; i < outs.size(); ++i) ref[i] = outs[i];
|
||||
}
|
||||
|
||||
// Pin channels 2,3, then drive them to arbitrary values. Output must match
|
||||
// the reference (they are neutralized to 0.5).
|
||||
m.set_input_pinned(2, true);
|
||||
m.set_input_pinned(3, true);
|
||||
NISPS_EXPECT(m.is_input_pinned(2));
|
||||
m.set_input(2, 0.0f);
|
||||
m.set_input(3, 1.0f);
|
||||
m.tick_control();
|
||||
for (std::size_t i = 0u; i < 33u; ++i) {
|
||||
NISPS_EXPECT(std::fabs(m.ml().outputs()[i] - ref[i]) < 1e-6f);
|
||||
}
|
||||
|
||||
// The active controller (channels 0,1) still affects the output.
|
||||
m.set_input(0, 0.9f);
|
||||
m.tick_control();
|
||||
bool changed = false;
|
||||
for (std::size_t i = 0u; i < 33u; ++i) {
|
||||
if (std::fabs(m.ml().outputs()[i] - ref[i]) > 1e-6f) { changed = true; break; }
|
||||
}
|
||||
NISPS_EXPECT(changed);
|
||||
|
||||
// Unpinning restores sensitivity on channels 2,3.
|
||||
m.set_input_pinned(2, false);
|
||||
m.set_input_pinned(3, false);
|
||||
m.set_input(0, 0.2f); // restore active channels to the reference pose
|
||||
m.set_input(1, 0.8f);
|
||||
m.set_input(2, 0.0f);
|
||||
m.set_input(3, 1.0f);
|
||||
m.tick_control();
|
||||
bool differs = false;
|
||||
for (std::size_t i = 0u; i < 33u; ++i) {
|
||||
if (std::fabs(m.ml().outputs()[i] - ref[i]) > 1e-6f) { differs = true; break; }
|
||||
}
|
||||
NISPS_EXPECT(differs);
|
||||
}
|
||||
|
||||
NISPS_TEST(paf_synth_mode_engine_and_ml_accessors) {
|
||||
PAFSynthMode m;
|
||||
m.setup(48000.f);
|
||||
|
|
|
|||
Loading…
Reference in a new issue