memlnaut-nisps/.factory/library/architecture.md
monkey-w1n5t0n 13112cf447 Add mission infrastructure for SolidJS migration
- Worker skill: solidjs-worker (SKILL.md)
- Services manifest (services.yaml)
- Init script (init.sh)
- Library: architecture.md, environment.md, user-testing.md, solidjs-patterns.md

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
2026-04-07 08:49:52 +02:00

4.6 KiB

Architecture

How the MEMLNaut NISPS SolidJS app works.

High-Level Architecture

index.html
  └── App.tsx (root component, provider tree)
      ├── BusProvider (signal bus context)
      ├── MLProvider (ML store context)
      ├── InputProvider (input store context)
      ├── OutputProvider (output store context)
      ├── SynthProvider (synth store context)
      ├── EOCProvider (EOC store context)
      └── SessionProvider (persistence)
            └── ImmersiveLayout.tsx
                ├── Background canvases (FlowField or SynthVisualizer)
                ├── HeatmapStrip
                ├── Joystick + JoyMap
                ├── RLButtons
                ├── Dock + DrawerStack
                └── Floating controls

Data Flow

User Input (drag/gamepad/hands)
  → Input Store (joyX, joyY)
  → Input Pipeline (deadzone, zoom, curve, smoothing)
  → ML Store (setInputs on IML)
  → IML.process() (WASM inference)
  → Outputs Signal (Float32Array, single signal write)
  → Output Routing (apply overrides, route by mode)
  ├── FlowField (visual mode, own rAF loop reads signal)
  ├── SynthEngine (synth mode, throttled 50ms)
  ├── MIDIOutput (midi-cc mode)
  └── AudioCanvas (audio-canvas mode)

Key Components

Signal Bus (bus/signal-bus.ts)

Unified event system replacing EventBus + CustomEvents. Topics with equals: false for event semantics. Wildcard matching via match().

ML Store (stores/ml-store.ts)

  • IML instances (WasmIML) live OUTSIDE the store (opaque handles)
  • outputs is a createSignal<Float32Array> (NOT a store property — proxies break typed arrays)
  • Store tracks: loss, lossHistory, exampleCount, isTraining, spreadLevel, noiseLevel, undoStack

Input Store (stores/input-store.ts)

  • joyX, joyY as store properties
  • Input pipeline config (deadzone, zoom, curve, etc.)

Output Store (stores/output-store.ts)

  • Output mode (visual/synth/midi-cc/audio-canvas)
  • Per-mode overrides arrays
  • Output pipeline config (globalCurve, smoothing, slewRate, freezeGate)

Dual IML System

  • imlJoy: 2 inputs (joystick), dynamic outputs
  • imlHand: 14 inputs (hand tracking), dynamic outputs
  • activeIml: points to whichever is active
  • EOC IML: separate instance for Linked/Independent modes

WASM Integration

  • WASM files in public/ (no Vite hashing)
  • Emscripten glue loaded via dynamic import
  • Training worker: lazy-loaded, own WASM instance
  • Heap management: persistent buffers for inference

Canvas Components

Each canvas owns its rAF render loop. The component reads the outputs signal each frame. No shared orchestrator.

Debug Probe

window.__nisps exposed when ?debug=1. All methods synchronous, bypassing SolidJS batching. window.__nispsEoc always exposed.

Transplanted Modules (from playground/js/)

These are copied with minimal TypeScript adaptation:

  • core/wasm/ — WASM binaries + Emscripten glue (unchanged)
  • core/iml.ts — WasmIML wrapper (from nisps/nisps-wasm.js)
  • core/dataset.ts — FIFO ring buffer (from nisps/dataset.js)
  • core/synth/ — All synth engines (c15-bridge, c15-adapter, param-map, presets, additive, fm, faust)
  • core/audio/ — Arpeggiator, AudioCanvas, MIDI I/O
  • core/eoc/ — EOC chain + effect modules
  • core/shapeseq/ — ShapeSeq sequencer
  • core/ui/ — Pure math/logic modules (input-pipeline, output-pipeline, control-surface, visualizer)

Rewritten Modules

Everything else is rewritten in SolidJS:

  • All DOM manipulation → JSX components
  • Event wiring → signal bus + reactive effects
  • State management → SolidJS stores/signals
  • Layout/CSS → new CSS with same design system
  • Init/boot → provider tree + onMount hooks

Key Constants

  • N_JOY_INPUTS = 2, N_HAND_INPUTS = 14
  • N_VISUAL_OUTPUTS = 20, N_SYNTH_OUTPUTS = 126 (C15)
  • STORAGE_KEY = 'nisps-a-immersive'
  • MAX_UNDO = 20, PARAM_DEAD_ZONE = 0.002, PARAM_SEND_INTERVAL = 50
  • 18 C15 synth sections (SYNTH_SECTIONS)
  • 20 visual param names/colors (VISUAL_PARAM_NAMES, VISUAL_PARAM_COLORS)

COOP/COEP Requirement

C15 synth requires SharedArrayBuffer. Vite dev server must send:

Cross-Origin-Opener-Policy: same-origin
Cross-Origin-Embedder-Policy: require-corp

MLP Resize Cascade

When output count changes (mode switch, engine switch, EOC mode change):

  1. Warn user if training data exists → confirmation dialog
  2. Destroy old MLP, create new with different output count
  3. Warm-start: copy hidden layer weights, reset output layer
  4. Update output signal, rebuild heatmap