f26ec923e1
Stream 7 wires nisps/ml + nisps/engines into the playground via Emscripten.
Highlights:
- nisps/wasm/bindings.cpp: flat C API per architecture.md §6.2. Fixed-arch
MLP<2, 10, 14, 18, 126>; engine string→type dispatch table with NoOp
fallback.
- scripts/build-wasm.sh: emcc invocation, MODULARIZE=1, exports listed
explicitly; produces playground/public/nisps.{js,wasm}.
- playground/src/ml/wasm-iml.ts: main-thread MLP host (sync inference,
sync training, RL ops, weights I/O, layer stats, localStorage).
- playground/src/ml/wasm-worker.ts: disposable Web Worker for off-thread
async training, owns its own WASM instance.
- playground/src/ml/dataset.ts: Float32Array-backed FIFO with sample-weight
modes (uniform/global/local/combined). Port of legacy dataset.js.
- playground/src/audio/engine-host.ts: AudioContext + AudioWorkletNode
lifecycle, with start/stop/setEngine/setParams.
- playground/src/audio/worklet/nisps-processor.ts: WASM-loading
AudioWorkletProcessor that runs engine.process_block per 128-sample
block. Loads its own WASM instance from main-thread-supplied bytes
(no fetch in worklet).
- playground/src/stores/ml-store.ts: wired stub methods to WasmIML
singleton; lazy initialize().
- playground/src/debug/probe.ts: window.__nisps now calls real WasmIML
via the store; lazy-init on first use.
Verified:
- bash scripts/build-wasm.sh succeeds (94 KB nisps.wasm)
- bun run typecheck OK
- bun run build OK (production bundle)
- vite dev server serves /nisps.{js,wasm} with COOP/COEP
Known limitation: WASM is fixed at one MLP shape. Multi-arch deferred —
documented in nisps/wasm/README.md.
|
||
|---|---|---|
| .. | ||
| audioworklet-globals.d.ts | ||
| nisps-processor.ts | ||
| README.md | ||
Why two WASM instances?
The playground loads nisps.wasm twice:
- Main thread, via
playground/src/ml/wasm-iml.ts. Used for ML inference + sync training + RL operations + UI feedback. - AudioWorklet thread, via
nisps-processor.ts. Used for engine processing (per-128-sample-block).
Why not share?
- AudioWorklet runs on its own thread. Sharing memory across threads requires SharedArrayBuffer + locking on every heap access; far more expensive than two heaps.
- AudioWorklet has neither
fetchnor ESMimport, so it can't load the Emscripten glue (nisps.js). The main thread fetches the WASM bytes once and posts them here as a transferableArrayBuffer; the worklet thenWebAssembly.instantiates directly. - Engines and ML never interact in the audio path. The main thread
computes the parameter vector each frame and pushes it into the
worklet via
port.postMessage. The worklet pushes nothing back per block (analysis features, if needed, are batched and sent at low rate).
Per-frame data flow:
Joystick → input pipeline → mlStore.outputs (Float32Array, length=126)
↓ EngineHost.setParams()
↓ port.postMessage (transferable)
AudioWorklet ← WASM engine.process_block ← WASM engine.set_params
Custom WASM loader
We do not use the Emscripten JS glue inside the worklet — the glue
contains URL, Worker, and fetch references that don't exist in
AudioWorkletGlobalScope. Instead nisps-processor.ts calls
WebAssembly.instantiate directly with hand-rolled imports and
discovers exports by walking the export descriptors. This makes the
worklet bundle small (just the processor TS) and avoids touching the
Emscripten init path.
The trade-off: only the engine API is callable here, not the ML API. If you ever need ML inference inside the worklet (we don't), use the main thread copy and post params over.
Block size
AudioWorklet always calls process() with 128-sample blocks. Our
heap buffers in nisps-processor.ts are sized to match. Don't change
the block size without changing the buffer allocations.