The matrix-seeding edit from the previous commit added a second
`const destNames = engine.destNames || [];` inside rebuildMatrixGrid
while the same name was already declared earlier in the function body.
Chrome threw SyntaxError at parse time, which in turn prevented
a-app.js from wiring window.__nisps at all, which is what made the
earlier FaustWorkletProcessor error look like the root cause — it
was just the next thing that broke once the parse error let partial
modules load.
Verified end-to-end in headless Chromium via Playwright: engine
switches to modular, paramCount=32, audioCtx running, worklet node
live, zero console errors.
Also updates the Modular engine card description to reflect the new
default (32 params = 4 ADSR × 4 + 8 LFO × 2, matrix cells opt-in).
Two fixes driven by user reports:
1. MLP wasn't affecting the sound. Matrix cells are now opt-in to the
MLP output vector rather than always-driven. Default modular paramMeta
is 32 mod-source params (4 ADSR * A/D/S/R + 8 LFO * rate/morph), down
from 512. The default patch's MM_Matrix/s00_d08_amp=1.0 now survives
the first inference tick because it's not in paramMeta — ADSR1 stays
routed to amp and the MLP drives envelope shape per joystick position.
Matrix cells still clickable as direct-DSP knobs in modular-ui: setCell
now routes through engine._setRawByLabel(). A later UI pass can add a
"expose to MLP" menu entry that calls engine.setExposeMatrixCell(s,d).
Also removes the earlier exampleCount-based routing gate; no longer
needed now that the default patch is stable.
2. Per-group curve drawer (hover over section labels on the synth
visualizer) is now available for all synth engines, not just C15.
Refactored showGroupDrawer behind a getSectionView(sectionIndex)
helper that returns a uniform view for either C15 (via SYNTH_SECTIONS
+ groupOverrides) or non-C15 (via nonC15Sections + nonC15GroupCurves
+ engineParamOverrides).
Group-level curve persists across sub-engine swaps by group name, so
e.g. tuning the "ADSR 1" curve survives a switch between subtractive
and fm without being reset.
ModularEngine: adds setExposeMatrixCell(s,d,exposed) +
getExposedMatrixCells() + clears exposed cells on sub-engine swap.
Modular mode's default patch sets MM_Matrix/s00_d08_amp = 1.0 via
_setRawByLabel at engine init, but on the first inference tick
routeOutputs writes the untrained MLP's output (~0.5 normalised) to
every paramMeta index. For matrix cells the raw range is [-1, 1], so
normalised 0.5 denormalises to 0 — clobbering the default amp routing
and killing all audio.
Skip the setParam loop for modular when iml.exampleCount === 0, so the
worklet keeps running on the default patch we already pushed at init.
Once the user captures at least one training example the MLP has a
target to reproduce, and routing resumes normally.
Only affects modular mode; other engines are unchanged.
Two fixes for Modular mode:
1. faust-worklet-processor.js: explicitly attach FaustWorkletProcessor
to globalThis. Class declarations at the top of a classic script are
lexically scoped to that script's evaluation context and do not
propagate across separate addModule() calls, so the base class was
invisible to subclass processors when they loaded in AudioWorklet-
GlobalScope. Symptom: "FaustWorkletProcessor is not defined" at the
extends clause of modular-subtractive-processor.js.
2. modular-ui.js + a-app.js: wire live matrix cell updates. Phase C
wired matrix cells for writes (click cycles, precise editor) but
not for reads — the MLP-driven values never propagated to the DOM.
modular-ui now exposes updateLive(outputs), called from routeOutputs
on every inference tick. Throttled to ~20 fps internally to avoid
DOM thrashing. Only visible sources (within adsrCount/lfoCount) are
updated; muted slots stay dark.
showGroupDrawer indexes into the hardcoded SYNTH_SECTIONS table and
groupOverrides, which only exist for the shaper-feedback engine. For
additive/fm/modular engines, paramToSection is rebuilt dynamically from
paramMeta groups, so region.index points into a larger dynamic section
list and SYNTH_SECTIONS[region.index] is undefined. Hovering a section
label in Modular mode threw "Cannot read properties of undefined
(reading 'name')".
These engines have their own param UIs; the C15 group drawer doesn't
apply. Guard and early-return.
New "Modular" engine in a-immersive with three hot-swappable Faust
sub-engines (subtractive/additive/fm) sharing a common modulation pool:
16 ADSR slots + 32 LFO slots (single-knob sine->tri->square->saw
wavemorph) routed through a 48-source x 10-destination matrix per
engine. Per-connection scalar amounts in [-1, 1], summed at each
destination. Default MLP output count is 512 (32 mod-source params +
480 matrix cells); model reinits on sub-engine swap, count change, or
engine-param exposure toggle.
Faust layer:
- mod-pool.lib: shared ADSR/LFO/source-bus library
- gen-modular-dsp.py: byte-reproducible generator (source of truth)
- modular-subtractive: faithful Minimoog (3 osc, ladder filter, no envs)
- modular-additive: 64-partial, spectral shape + formants, no envs/LFOs
- modular-fm: 4-op matrix + self-feedback, no envs
- All three share d08=amp, d09=pan conventions
- MODULAR_DESTINATIONS.md: authoritative destination table
JS layer:
- ModularEngine: self-contained SynthEngine with getState/setState,
setSubEngine, setModSourceCount, setExposeEngineParam
- modular-ui: drawer with sub-engine toggle, ADSR/LFO count steppers,
per-slot enable switches, matrix grid editor (tap-cycle, long-press
precise, right-click menu, negative amounts), preset overlay
- modular-presets: 6 named presets (Slow pad, Plucky bass, Crystal,
DX bell, Morphing drone + default)
- a-app.js: Modular mode registered, paramMeta:change -> resizeMLP,
modular DSP state persisted under modularDspState, window.__nisps
debug hooks for programmatic control
Tests: tests/e2e/modular-mode.spec.js (11 Playwright tests, all passing
including DSP state survives reload, sub-engine swap keeps paramCount,
preset apply verification).
Also fixes a pre-existing build.sh bug where the -e flag caused faust
to overwrite .wasm outputs with expanded DSP source text, leaving
additive/fm-matrix/eoc-* committed as invalid WebAssembly. Rebuilt all
affected engines with the corrected script. Added an early-message
buffer to faust-worklet-processor.js so setParam calls arriving before
wasm instantiation are queued rather than dropped (needed when the user
configures modular state before clicking Start Audio).
Move the lazy AudioCanvas initialization above the outputCountForMode()
call in setOutputMode() so that the real output count is available
instead of the 12-output fallback.
- Per-file loop count (0-4) with stepper UI on each waveform cell;
dynamic MLP output sizing (8 range + N_loops × 4 position outputs)
- Beat-synced loop length: global bars/beats selector (2/4/8/16/32/64)
with per-loop overrides via long-press dropdown, persistent badges
- Global param range groups: dual-thumb range sliders (volume, pitch,
position, filter) with ML-driven min/max and per-loop position mapping
- Pitch lock toggle (1×) to freeze playback rate across all submodes
- Fix: _resizeOutputs now syncs _loopOutputs to prevent stale reads
- Fix: granular visuals consolidated per-clip to prevent multi-loop flicker
When switching from C15 to Additive or FM engines, the MLP was resized
to 0 outputs because paramMeta wasn't loaded until init() (which needs
AudioContext). The SynthVisualizer also disappeared because its section
map was hardcoded to C15's 18 param groups.
- Add FaustEngineBase.loadParamMeta() for eager JSON-only loading
- Call loadParamMeta() before setActiveEngine() in engine switch handler
- Add SynthVisualizer.rebuild(paramMeta) to derive sections from groups
- Add rebuildParamToSection() / restoreC15ParamToSection() for engine-agnostic
param-to-section mapping
- Fix tooltip to use engineParamOverrides for non-C15 engines
- Engine switcher now lives in the Synth drawer (was incorrectly in NISPS params drawer)
- Params drawer renamed to "NISPS" (it's for ML engine tuning, not synth engine selection)
- Output mode button says "Synth" (generic) not "C15 Shaper-Feedback" (engine-specific)
- Dock icon for params drawer now labelled "NISPS" with correct tooltip
Additive engine: 5 presets across 3 tiers (Spectral Basics, Formant Play,
Harmonic Sculptor, Phase Explorer, Full Spectrum). FM engine: 5 presets
across 3 tiers (Simple FM, Bell Tones, Matrix Explorer, Feedback Machine,
Full Matrix).
Engine-aware preset system: applyPreset() dispatches to C15 groupOverrides
or new flat engineParamOverrides depending on active engine. Preset selector
dropdown rebuilds on engine switch. URL ?preset param searches all engines.
- Add totalOutputCount() helper: engine params + EOC params when nispsMode === 'shared'
- setActiveEngine() calls resizeMLP(totalOutputCount()) and rebuilds heatmap with combined paramMeta
- eoc:change listener handles nispsMode-changed and module-* events: resizes MLP, rebuilds heatmap, toggles .shared-mode CSS class
- routeOutputs() slices outputs: indices < engineParamCount go to engine, remainder to eocChain.setParam() in Shared mode
- CSS: .heatmap-strip.shared-mode adds green bottom border and '+ FX' label to signal combined output space
- Add AdditiveEngine (48 params) and FMEngine (55 params) wrapping FaustEngineBase
- Remove comingSoon flags from ENGINES descriptors
- Replace stub onSwitch handler with real engine construction + init + setActiveEngine
- Fix setActiveEngine EOC rewire to work with both C15 (_bridge.audioContext) and Faust (_audioCtx)
- Fix _startEocChain audioCtx resolution to support Faust engines
- Fix synth param send loop: use N_OUTPUTS instead of hardcoded SYNTH_PARAM_MAP.length
- Add rebuildHeatmap(paramMeta): resets rawParamValues/_lastSentParams and rebuilds
#heatmap-cells from engine paramMeta; colors from SYNTH_PARAM_COLORS for C15,
_colorFromGroup() hash for other engines
- Replace static SYNTH_PARAM_NAMES/COLORS lookups in buildHeatmap, setHeatmapValue,
and showParamPopup with activeEngine.paramMeta[index] lookups
- Scope MIDI CC localStorage key per engine via midiCCStorageKey()
- Add reloadMidiCCMap() helper; called in setActiveEngine() after engine switch
- loadCCMap/saveCCMap in midi-cc-map.js accept optional key parameter
Adds EngineSwitcher module with clickable engine cards (active highlight,
coming-soon greyed state, loading bar animation, confirm-on-switch when
training data exists). Wired into the Engine drawer; C15 active,
Additive/FM marked coming-soon with toast. Dock title reflects active
engine; engineId persisted to/restored from localStorage.
- input-pipeline.js: guard circular clamp against div-by-zero when
input is exactly at center (0.5, 0.5) — dist=0 produced NaN
- snapshot-stack.js: jumpTo() used slice(0, index) which excluded the
target snapshot; fixed to slice(0, index + 1)
- a-app.js: _lastSentParams not resized when MLP output count changes
during mode switch, causing stale throttle state and potential param
flood on first frame after switch
Add WasmIML.extractWeights() and WasmIML.createWithWarmStart() to preserve
learned joystick mappings across output-count changes; resizeMLP() now
transfers hidden-layer weights and shared output nodes instead of cold-starting.
Add evalLoss, inferBatch, and getLayerStats to the window.__nisps
debug probe so Playwright tests and dev console can access the new
WasmIML capabilities. InputHeatmap (Phase 3) is not yet wired into
a-app.js, so batch inference heatmap integration is deferred.
- Add window.__nisps debug probe (gated on ?debug=1) exposing iml state,
getOutputs/getLoss/getWeights/getExampleCount, and action triggers
(thumbsUp/thumbsDown/train/randomise/clearExamples/saveState)
- Fix WasmIML bug: this.dataset was a plain object; import Dataset and
use new Dataset(100) so computeWeights() is available for training
- Fix WasmIML.addExample/clearDataset to use Dataset API methods
- 44 Playwright e2e tests across 4 spec files:
- ml-engine.spec.js: WASM inference bounds, training loss, thumbs
up/down behavior, async training, example capture semantics
- ui-interactions.spec.js: drawer open/close, mode switching,
heatmap bar counts, preset chips, keyboard shortcuts (1/2/Z)
- input-pipeline.spec.js: input→output variation, clamping, joystick
drag, post-training output bounds across the full input space
- persistence.spec.js: URL params (?preset, ?spread), localStorage
round-trip, saveState probe
Add MIDI CC preset system for bundled device configurations:
- midi-cc-presets.js: listPresets/loadPreset/loadPresetFromFile API
- presets/polybrute.json: Arturia PolyBrute CC map
- Preset selector in quick controls bar and MIDI CC drawer
- File import button for loading JSON presets from disk
Bug fixes in a-app.js:
- Destroy IML instances before resizeMLP() to free WASM memory
- Fix randomiseWeights() call (was drawWeights())
- Fix thumbs-up to store rawParamValues instead of post-pipeline outputs
Browser-side OSCOutput module sends parameter values over WebSocket to
a companion Deno bridge script that converts them to OSC/UDP messages.
Enables controlling SuperCollider, Max/MSP, Pure Data, TouchDesigner,
or any OSC-capable software from the NISPS playground.
- Browser module (js/synth/osc-output.js): WebSocket client with
auto-reconnect, throttle (~20fps), and dead-zone filtering
- Deno bridge (osc-bridge/bridge.ts): zero-dependency, compiles to
standalone binaries via deno compile for Linux/macOS/Windows
- Test receiver (osc-bridge/test-receive.ts): terminal dashboard
showing live OSC parameter values with bar charts
- OSC pill button in floating bar for quick connect/disconnect
- Help modal section with platform-aware download, setup guide,
and examples for SuperCollider/PD/Max
- GitHub Actions workflow for cross-platform binary builds
- OSC sends in both visual and synth modes
Restore the original arpeggiator as the default sequencer. ShapeSeq
code is preserved but gated behind a URL feature flag (?shapeseq=1):
- Arpeggiator import, state, and all wiring fully restored
- ShapeSeq modules loaded via dynamic import() only when flag is on
- ensureShapeSeqInit() creates engine/viz/UI lazily on first audio start
- animate loop routes inputs to ShapeSeq only when flag + playing
- setOutputMode shows/hides ShapeSeq container only when flag is on
- HTML: ShapeSeq container added but hidden by default
- CSS: ShapeSeq styles added (inert when container is hidden)
All 13 ShapeSeq modules in playground/js/shapeseq/ are untouched.
To test ShapeSeq: add ?shapeseq=1 to the URL.
- Remove Arpeggiator import and all arp references from a-app.js
- Add ShapeSeq imports (ShapeSeqEngine, StepVisualizer, ChainBuilderUI,
getDefaultBus) and lazy initialization on first audio start
- In synth mode: show ShapeSeq container (circular viz + chain builder),
hide flow-field particles. Visual mode unchanged.
- Route joystick or hand tracking inputs to shapeSeq.setSequenceInputs()
each frame; render step viz at 60fps
- Add HTML: #shapeseq-container with #shapeseq-viz canvas and
#shapeseq-chain wrapper, sequencer transport controls (play/stop,
tempo, step count)
- Add CSS: full-screen flex layout, viz takes upper 45%, chain builder
scrollable below, hidden by default
- Quick play button wires to ShapeSeq start/stop instead of arp
Closes meml-6ry. This completes ShapeSeq Phase 1.
Hand tracking at 30fps was writing 126 params per frame to the C15 ring
buffer (capacity 512), flooding it and starving arpeggiator noteOn/noteOff
messages — causing stuck/dropped notes.
Three-part fix:
- Arpeggiator now runs in a dedicated Web Worker with direct
SharedArrayBuffer access, bypassing the main thread entirely for
note timing. Worker setInterval isn't subject to main thread jank.
- RingBufferWriter uses CAS (Atomics.compareExchange) for multi-producer
safety — both main thread (params) and worker (notes) write safely.
- routeOutputs throttles synth parameter sends: dead-zone filter
(0.2% change threshold) + rate cap (~20fps), reducing ring buffer
pressure from ~3800 msg/s to ~50-100 msg/s of actual changes.
Hand tracking via webcam as alternative to joystick input:
- Right hand tracks 14 derived features (palm XY, finger curls, spread,
roll, pitch, pinch) through a separate 14-input MLP
- Left hand gesture recognition (1 finger = thumbs up, 2 = thumbs down)
with 400ms debounce hold
- Split-zone PIP display (no camera feed, skeleton only) with dashed
divider and cross-zone dimming
- Dual IML architecture: independent imlJoy (2 inputs) and imlHand
(14 inputs) with pointer swap, preserving training data per mode
- Dev panel (?devmode=true): draggable/collapsible floating panel with
sliders for MediaPipe confidence thresholds, smoothing, gesture hold
time, world landmarks toggle, and live feature bar monitor
- Visual presets always route to joystick IML (prevents dimension mismatch)
- Race condition guard on input mode switching
Compile nisps-core C++ MLP to WASM (36KB) and use it as the ML engine
in the playground, replacing the JavaScript port for inference, training,
and weight manipulation.
- Add extern "C" WASM bindings with spread-aware drawWeights/moveWeights
- WasmIML class is a drop-in replacement for the JS IML
- Inference runs on main thread via WASM (fast, synchronous)
- Training runs in a Web Worker with its own WASM instance (non-blocking)
- Interactive training (thumbs-up, train button) no longer freezes UI/audio
- Preset loading and state restore still use sync training
4 tiers of progressive complexity (Beginner 15 params → Expert 126),
13 presets total with per-param min/max/curve overrides that bias
distributions without clamping extremes. Preset dropdown in UI,
persisted to localStorage, supports ?preset= URL param.
SVF count was 7 (actual 9), Gap Filter section was missing (6 params),
FB Mix count was 10 (actual 9) — causing cascading mislabeling of all
synth sections from SVF onwards. Also sync volume/tempo changes from
bottom sheet back to quick-play controls.
- Glass-style modal explains NISPS, how learning works, and what to expect
- Documents all controls: touch, keyboard, and gamepad bindings
- Shows automatically on first visit (localStorage flag)
- Dismissible via close button, "Got it", overlay click, or Escape
- Reopenable via ? button at bottom right
- Map A=Train, B=Clear Examples, X=Randomize, LB=Thumbs Down, RB=Thumbs Up
- Add periodic gamepad polling fallback for environments where
gamepadconnected event doesn't fire reliably (Steam Deck, some Linux browsers)
- Show "Press any gamepad button to connect" hint on page load
When synth controls (volume, tempo, etc.) have focus after dragging,
keyboard shortcuts 1/2 for thumbs down/up were intercepted by the
range input. Now all four UIs skip shortcut handling when an input,
select, or textarea element has focus.
Relocated the output mode toggle from a floating top-right position into
the bottom floating bar where it's more discoverable. Removed the
duplicate toggle from the expanded sheet. Compact pill-toggle-sm style
fits the toolbar layout.
Merge Examples/RL tabs into single unified toolbar across all four UIs.
Users can now freely mix supervised learning (Add Example + Train) with
RL feedback (thumbs up/down) without switching modes. Param bar dragging
is always enabled.
Add Web MIDI input module (js/synth/midi-input.js) for external MIDI
controllers — routes note on/off to C15 synth, CC 1/2 to joystick.
Add gamepad module (js/ui/gamepad.js) with auto-detection, deadzone,
and axis normalization across all UIs.
Fix synth parameter sync: routeOutputs() now called after trainModel()
in onThumbsUp and loadState to prevent stale params on first joystick
move. Add separate Clear Examples button (dataset only, keeps weights).
