# Architecture How the MEMLNaut NISPS SolidJS app works. ## High-Level Architecture ``` index.html └── App.tsx (root component, provider tree) ├── BusProvider (signal bus context) ├── MLProvider (ML store context) ├── InputProvider (input store context) ├── OutputProvider (output store context) ├── SynthProvider (synth store context) ├── EOCProvider (EOC store context) └── SessionProvider (persistence) └── ImmersiveLayout.tsx ├── Background canvases (FlowField or SynthVisualizer) ├── HeatmapStrip ├── Joystick + JoyMap ├── RLButtons ├── Dock + DrawerStack └── Floating controls ``` ## Data Flow ``` User Input (drag/gamepad/hands) → Input Store (joyX, joyY) → Input Pipeline (deadzone, zoom, curve, smoothing) → ML Store (setInputs on IML) → IML.process() (WASM inference) → Outputs Signal (Float32Array, single signal write) → Output Routing (apply overrides, route by mode) ├── FlowField (visual mode, own rAF loop reads signal) ├── SynthEngine (synth mode, throttled 50ms) ├── MIDIOutput (midi-cc mode) └── AudioCanvas (audio-canvas mode) ``` ## Key Components ### Signal Bus (`bus/signal-bus.ts`) Unified event system replacing EventBus + CustomEvents. Topics with `equals: false` for event semantics. Wildcard matching via `match()`. ### ML Store (`stores/ml-store.ts`) - IML instances (WasmIML) live OUTSIDE the store (opaque handles) - `outputs` is a `createSignal` (NOT a store property — proxies break typed arrays) - Store tracks: loss, lossHistory, exampleCount, isTraining, spreadLevel, noiseLevel, undoStack ### Input Store (`stores/input-store.ts`) - joyX, joyY as store properties - Input pipeline config (deadzone, zoom, curve, etc.) ### Output Store (`stores/output-store.ts`) - Output mode (visual/synth/midi-cc/audio-canvas) - Per-mode overrides arrays - Output pipeline config (globalCurve, smoothing, slewRate, freezeGate) ### Dual IML System - `imlJoy`: 2 inputs (joystick), dynamic outputs - `imlHand`: 14 inputs (hand tracking), dynamic outputs - `activeIml`: points to whichever is active - EOC IML: separate instance for Linked/Independent modes ### WASM Integration - WASM files in `public/` (no Vite hashing) - Emscripten glue loaded via dynamic import - Training worker: lazy-loaded, own WASM instance - Heap management: persistent buffers for inference ### Canvas Components Each canvas owns its rAF render loop. The component reads the outputs signal each frame. No shared orchestrator. ### Debug Probe `window.__nisps` exposed when `?debug=1`. All methods synchronous, bypassing SolidJS batching. `window.__nispsEoc` always exposed. ## Transplanted Modules (from playground/js/) These are copied with minimal TypeScript adaptation: - `core/wasm/` — WASM binaries + Emscripten glue (unchanged) - `core/iml.ts` — WasmIML wrapper (from nisps/nisps-wasm.js) - `core/dataset.ts` — FIFO ring buffer (from nisps/dataset.js) - `core/synth/` — All synth engines (c15-bridge, c15-adapter, param-map, presets, additive, fm, faust) - `core/audio/` — Arpeggiator, AudioCanvas, MIDI I/O - `core/eoc/` — EOC chain + effect modules - `core/shapeseq/` — ShapeSeq sequencer - `core/ui/` — Pure math/logic modules (input-pipeline, output-pipeline, control-surface, visualizer) ## Rewritten Modules Everything else is rewritten in SolidJS: - All DOM manipulation → JSX components - Event wiring → signal bus + reactive effects - State management → SolidJS stores/signals - Layout/CSS → new CSS with same design system - Init/boot → provider tree + onMount hooks ## Key Constants - N_JOY_INPUTS = 2, N_HAND_INPUTS = 14 - N_VISUAL_OUTPUTS = 20, N_SYNTH_OUTPUTS = 126 (C15) - STORAGE_KEY = 'nisps-a-immersive' - MAX_UNDO = 20, PARAM_DEAD_ZONE = 0.002, PARAM_SEND_INTERVAL = 50 - 18 C15 synth sections (SYNTH_SECTIONS) - 20 visual param names/colors (VISUAL_PARAM_NAMES, VISUAL_PARAM_COLORS) ## COOP/COEP Requirement C15 synth requires SharedArrayBuffer. Vite dev server must send: ``` Cross-Origin-Opener-Policy: same-origin Cross-Origin-Embedder-Policy: require-corp ``` ## MLP Resize Cascade When output count changes (mode switch, engine switch, EOC mode change): 1. Warn user if training data exists → confirmation dialog 2. Destroy old MLP, create new with different output count 3. Warm-start: copy hidden layer weights, reset output layer 4. Update output signal, rebuild heatmap