Route onMove through the XY-pad source's pushPad; instantiate useInputLayer
per engine and expose it on ConsoleCtx. The Inputs drawer now picks the
source(s) (XY pad / MIDI / gamepad toggles), configures each (gamepad
single/double stick; MIDI learn-map with per-binding clear), and shows the
composed channel layout + the blend-to-2 reshape notice referencing the
multi-WASM TODO (inputs-spec).
Workstream F core. Pull-based InputSource interface (N axes + discrete
actions), three sources (XYPadSource push-based, WebMidiInputSource with
learn-map CCs/notes, GamepadSource single/double stick), and an InputLayer
that composes active sources into one N-dim vector at the head of the
reactive spine. Arity reduction blends >2 axes down to the fixed 2-input
WASM head; the true multi-WASM reshape is a documented TODO (inputs-spec).
useInputLayer is the React binding (enable/config/status/channel layout).
Registers a named ParticleBackend in the BackendManager + barrel in place of
the inline particles PassthroughBackend. Transport is a no-op (the flow-field
canvas reads engine outputs directly via rAF); selecting it gates synth audio
(setMuted) like the other non-synth backends and reports a clear ready status.
Particle System is the default output Mode. When active it renders the
full-bleed ParticleStage (flow field driven by engine outputs each rAF) in
place of the manifold stage, regardless of the in/split/out/composite focus,
and suppresses the duplicate corner MEMLNaut label (the stage's top axis bar
already shows it).
Faithful TypeScript port of the a-immersive visualiser
(deployments/meml-aimmersive/js/ui/visualizer.js): 400 particles, identical
value-noise + 20-output param mapping + advection/attractor/dispersion/repulsor
integration. ParticleStage renders the full-bleed canvas (driven by engine
outputs each rAF) plus the macro-axis bar and a corner input pad.
Map enter/exit-explore, reroll, nudge, begin-place, commit-place to the
peripheral buttons via the shared FeedbackController. (Not compiled here —
no arduino-cli; Placing-audition static_output hook is a flagged follow-up.)
8 inputs x 16 outputs; per-output LED ring widget (drawLayer+nvgArc); palette
from frontend tokens; OSC bridge verbs for bidirectional browser training;
vendored self-contained iml.hpp (retired nisps-core); compiles against Rack
SDK 2.6.4. See SPEC.md BUILD DELTAS.
Adds the Manifold app (manifold/) — the convertible-mode React front-end on
the real NISPS ML+audio engine, served live at meml.lnfinitemonkeys.org/next/.
Console chrome trimmed per UI pass:
- drop mode label + subtitle from the top-left overlay (keep MEMLNaut wordmark)
- remove the composite split preset/ratio readout (top-centre)
- remove the OUTPUT corner tag above the bars
- remove the A/B compare toggle from the verdict cluster
- remove the follow button + input/noise readout (bottom-left)
- remove AltitudeNav focus switcher (bottom-right)
addModule() was handed new URL('./nisps-processor.ts', import.meta.url), which
Vite copies as raw TypeScript (served as video/mp2t) instead of compiling it —
that pattern only bundles for new Worker(). Import the worklet with
?worker&url so Vite emits compiled, hashed .js. Fixes the synth engine failing
to load on meml.lnfinitemonkeys.org.
Builds the SolidJS-side feature parity with the legacy a-immersive
playground. Every Phase 1–4 feature called out in
.local/recon/04-playground.md now flows through the runtime → stores →
primitives pipeline.
Phase 1 (compound axes / zoom / drawer / trail):
- control-routing.ts: resolves Boldness/Memory/Precision into
input/output/exploration store writes via createEffect.
- SettingsDrawer (new): tabbed sections for Input / Training /
Exploration / Output / Per-param overrides / Advanced. Per-param
ParamEditor list wires through modeStore.setOverride.
- JoyMap navigator added to ModeShell; tap-to-return + long-press
region pin handlers wired into the runtime.
- Anchor mode toggle (auto/sticky/center) exposed via drawer.
Phase 2 (pinning / history / A/B):
- features/snapshots.ts: bridge sessionStore push/pop to actual
Float32Array weights. Auto-snapshot before train/randomize/thumbs-down.
- Undo button + long-press snapshot list popup wired in ModeShell.
- A/B capture/toggle/accept/revert in SettingsDrawer.
- features/overrides.ts buildPinMask() composes override `pinned`
flags + sessionStore param pins; passed to mlStore.moveWeights.
- Region pinning long-press → addRegionPin + auto-snapshot tag
"pinned baseline".
Phase 3 (refinement / exploration):
- features/heatmap-sampler.ts: 16x16 grid via mlStore.inferBatch,
three colour modes, throttled 5/sec, refreshed on ml.trained /
ml.delta_update bus events.
- exploration-store.ts (new): RL noise level, floor/cap/growth/decay,
spread, learning rate, weight decay, auto-explore, pressure.
- Auto-explore timer in mode-runtime drives moveWeights at user
interval, zoom-scaled intensity.
- Pressure feedback: window pointerdown/up timestamps mapped to
explorationStore.setPressure; feeds growNoise/decayNoise.
Phase 4 (output / persistence / polish):
- features/weight-health.ts: histogram, status, per-layer L2 deltas,
vanishing/exploding/converged classification.
- features/session-preset.ts: full state capture/restore +
base64url URL sharing; main.tsx applies on load.
- SettingsDrawer Advanced tab renders WeightHealth, GradientFlow,
LayerStats, Heatmap, session preset save/load/share.
Override application:
- features/overrides.ts applies per-param mute/freeze/curve/range
between MLP outputs and the engine. Mode runtime exposes
`paramOutputs` (length = schema.params) which all modes now bind to
OutputDisplay so ranges and mutes are visible.
- Freeze flags push into outputStore.freezeMask reactively.
Mic input:
- features/mic-input.ts: getUserMedia + AnalyserNode-derived
{energy, brightness, pitch, aperiodicity}. Mode runtime feeds
them into channels 2..(input_size-1) when active. ModeShell shows
a Mic toggle for audio_in modes.
Debug probe (probe.ts):
- Synchronous bypass for snapshot, A/B, pins, overrides, axes,
spread, output freeze, heatmap, weight health, session presets,
URL params, bus emit/on. Untracked reads/writes throughout to
avoid SolidJS reactivity surprises in tests.
Modes:
- All 8 firmware mode TSX files now use the default SettingsDrawer
(no per-mode SliderBank scaffolding). They bind OutputDisplay to
runtime.paramOutputs (post-override) instead of processedOutputs.
Build status:
- bun run typecheck: clean
- bun run build: clean
- dev server smoke: index, /modes, mode-runtime, SettingsDrawer,
features/* all serve.
Note: bd close meml-5wg failed because Dolt server unreachable from
this worktree. Issue should be closed manually by orchestrator.
Move the Arduino sketch into firmware/MEMLNaut-NISPS/ and bridge the
hardware (memllib) to the platform-agnostic nisps/ library through a
slim glue layer. Delete the legacy root-level *AudioApp.hpp,
modes/MEMLNautMode*.hpp, voicespaces/, IMLInterface.hpp, XiasriAnalysis,
and the src/memlp submodule.
Glue layout (firmware/MEMLNaut-NISPS/glue/):
audio_driver.hpp - bridge memllib block callback to Mode::process
via per-Mode templated trampoline (no virtual dispatch)
peripherals.hpp - joystick/pots/buttons -> Mode::set_input + ML primitives
midi_io.hpp - MIDI in -> mode.note_on/update_bpm/set_playing,
drains mode ControlEvent ring -> MIDI UART
mode_select.hpp - using-aliases mapping MEMLNautMode<Name> to
nisps::modes::*Mode (build script rewrites the
#define MEMLNAUT_MODE_TYPE line)
input_router.hpp / output_router.hpp - top-level wire/drain entry points
The sketch tree uses src/{memllib,daisysp,nisps} symlinks because
Arduino-CLI rejects ".." in include paths from sketch-tree headers.
mode_select.hpp #undefs Arduino's sq/min/max/abs/round macros before
including nisps headers (some nisps engines use those identifiers as
method names). The audio bridge struct is extern in the header and
defined in the .ino because inline + __not_in_flash section attribute
collide at link time.
Verification: arduino-cli compile succeeds for PAFSynth, ChannelStrip,
and BreakOr (rp2040:rp2040:solderparty_rp2350_stamp_xl:opt=Optimize3,
-std=gnu++20). Host C++ tests under nisps/build still pass (3 binaries,
110+ tests). Build script (scripts/build-firmware.sh) updated to point
at the new sketch path; mode-rewrite logic unchanged.
Closes meml-gkm.
Stream 9 (part 1/2) — shared infrastructure for the per-mode TSX layer.
- ModeShell.tsx: common chrome (header, voice-space PillToggle, audio
start/stop, control axes, training controls, drawer, status line).
Modes only have to author primary input + output JSX.
- mode-runtime.ts: useModeRuntime(schema) hook. Owns input pipeline,
WASM ML calls, output pipeline, and a 50ms-throttled engine-host
bridge. Exposes setInput/processedOutputs/training/audio surfaces
so mode TSX stays declarative.
- mode-helpers.ts: schema → SliderBank config + Float32Array →
slider-range value mapping. Pure functions.
- ModeSwitcher.tsx: top-of-page <select> wired to modeStore.
ModeShell and the runtime read controlStore (Boldness/Memory/Precision)
to derive learning rate + RL noise cap so axes affect training without
needing per-mode plumbing.
Stream 7 wires nisps/ml + nisps/engines into the playground via Emscripten.
Highlights:
- nisps/wasm/bindings.cpp: flat C API per architecture.md §6.2. Fixed-arch
MLP<2, 10, 14, 18, 126>; engine string→type dispatch table with NoOp
fallback.
- scripts/build-wasm.sh: emcc invocation, MODULARIZE=1, exports listed
explicitly; produces playground/public/nisps.{js,wasm}.
- playground/src/ml/wasm-iml.ts: main-thread MLP host (sync inference,
sync training, RL ops, weights I/O, layer stats, localStorage).
- playground/src/ml/wasm-worker.ts: disposable Web Worker for off-thread
async training, owns its own WASM instance.
- playground/src/ml/dataset.ts: Float32Array-backed FIFO with sample-weight
modes (uniform/global/local/combined). Port of legacy dataset.js.
- playground/src/audio/engine-host.ts: AudioContext + AudioWorkletNode
lifecycle, with start/stop/setEngine/setParams.
- playground/src/audio/worklet/nisps-processor.ts: WASM-loading
AudioWorkletProcessor that runs engine.process_block per 128-sample
block. Loads its own WASM instance from main-thread-supplied bytes
(no fetch in worklet).
- playground/src/stores/ml-store.ts: wired stub methods to WasmIML
singleton; lazy initialize().
- playground/src/debug/probe.ts: window.__nisps now calls real WasmIML
via the store; lazy-init on first use.
Verified:
- bash scripts/build-wasm.sh succeeds (94 KB nisps.wasm)
- bun run typecheck OK
- bun run build OK (production bundle)
- vite dev server serves /nisps.{js,wasm} with COOP/COEP
Known limitation: WASM is fixed at one MLP shape. Multi-arch deferred —
documented in nisps/wasm/README.md.
One header per mode, each ~50-130 LOC built atop ModeBase:
- `paf_synth.hpp` (4 inputs → 33 outputs, 7 voice spaces, note_on/off)
- `channel_strip.hpp` (4 → 24, 6 voice spaces)
- `xiasri.hpp` (4 → 24, 1 'Direct' voice space)
- `verb_fx.hpp` (4 → 47, 12 voice spaces)
- `memlcelium.hpp` (4 → 56, dual synth + 2-track sequencer; set_playing/update_bpm)
- `breakor.hpp` (4 → 56, 8-track ratio sequencer; pumps engine NoteOn/Off/Clock
events into ControlEvent ring buffer)
- `elysiamorf.hpp` (4 → 40, 8-track FM-pair MIDI-CC sequencer; pumps CC events)
- `sound_analysis_midi.hpp` (10 → 8; owns AnalysisEngine for input feature
extraction PLUS NoOpEngine 'thru' for audio passthrough; ML outputs
converted to MIDI CC events via on_post_inference; opts out of
output→engine routing via ModeRoutesOutputsToEngine specialisation)
Every mode satisfies `nisps::Mode` (verified via `static_assert` in each
header). Schema `output_size` is verified against engine `param_count()`
at compile time inside ModeBase.
All hardware-specific glue (MEMLNaut::Instance, pico/util/queue, MIDIInOut,
display widgets, button callbacks) is intentionally absent — that lives in
firmware/glue and playground/src/modes per architecture.md §4.3.
Adds two foundational pieces shared by every concrete mode:
- `base.hpp`: defines `nisps::ParamSchema` (the aggregate type the
`Mode` concept's `param_schema()` returns by const-reference),
`nisps::ControlEvent`, and `ModeBase<Derived, EngineT, MLPType,
NInputs>` — a CRTP scaffold absorbing input forwarding, ML
inference + voice space mapping, engine.set_params() routing, and
a per-mode ControlEvent ring buffer for MIDI/I2C events.
`ModeRoutesOutputsToEngine<Derived>` lets a derived mode (e.g.
SoundAnalysisMIDI) opt out of routing ML outputs to engine
params when its outputs become MIDI CCs instead.
- `voice_space.hpp`: `VoiceSpaceEntry` + `make_voice_space_entries()`
+ `find_voice_space()` helpers. Voice-space *mapping code* lives
inside engines (mirroring firmware); mode layer just selects
which voice space the engine uses by index/name.
No heap, no virtuals, no Pico/Arduino dependencies; satisfies stream
4 of architecture.md.
Stream 3's NoOpEngine used engine_id()=='noop' but the schema for
sound_analysis_midi.json declares engine_id: 'thru' (matches firmware's
ThruAudioApp naming). The class name stays NoOpEngine — it accurately
describes what process() does — but the schema-facing identifier is now
'thru'. BreakOr/Elysiamorf compose NoOpEngine by type, not engine_id, so
they're unaffected.
Stream 5 shipped a temporary local Curve enum (PascalCase) while stream 1
was in flight. Stream 1 has now landed nisps/core/math.hpp with the
authoritative lowercase Curve enum per architecture spec. Generated C++
headers now include core/math.hpp and re-export the enum into the
generated namespace via 'using Curve = ::nisps::Curve;'.
- codegen/generate.ts: emit lowercase enum values + include math.hpp
- regenerated all 8 mode schema headers
- updated golden snapshot to match
Verified all 8 headers compile clean with -std=c++20 -Wall -Wextra.
