// The IML instances live OUTSIDE the store (mutable, non-proxy-safe).
// The store tracks their *state* reactively.
// After each inference: setOutputs(iml.getOutputs())
// After each train: setMlState({ loss, exampleCount, lossHistory })
```
**Key design choice**: `Float32Array` outputs are a signal, not a store property. Stores use proxies which don't play well with typed arrays. A signal holding the array reference, replaced each frame, gives us batch reactivity cheaply.
### Input Store (`stores/input-store.ts`)
```typescript
const [inputState, setInputState] = createStore({
mode: 'joystick' as 'joystick' | 'hands',
joyX: 0.5,
joyY: 0.5,
isDragging: false,
followMode: false,
// Input pipeline config
pipeline: {
deadzone: 0.02,
zoom: 1.0,
zoomAnchorX: 0.5,
zoomAnchorY: 0.5,
anchorMode: 'auto' as 'auto' | 'sticky' | 'center',
// 'seq.*' → derived signal merging all seq.* topics
const prefix = pattern.replace('*', '');
return createMemo(() => {
let latest: { name: string; data: any } | undefined;
for (const [name, [get]] of topics) {
if (name.startsWith(prefix)) {
const val = get();
if (val !== undefined) {
latest = { name, data: val };
}
}
}
return latest;
});
}
return { topic, emit, match };
}
```
**Usage equals: false** is critical — it means every `emit()` triggers subscribers even if the data is identical. This matches event semantics (every `seq.step` matters, even if the step number repeats).
---
## Inference Loop (`hooks/useInference.ts`)
```typescript
import { createEffect, onCleanup } from "solid-js";
export function useInference(mlStore, inputStore, outputStore, bus) {
// Apply overrides (pure function, no side effects)
const processed = applyOverrides(raw, overrides);
switch (mode) {
case 'visual':
bus.emit('output.visual', processed);
break;
case 'synth':
// Throttled: only send to engine at 20fps
bus.emit('output.synth', processed);
break;
case 'midi-cc':
bus.emit('output.midi', processed);
break;
case 'audio-canvas':
bus.emit('output.audiocanvas', processed);
break;
}
// Always update heatmap (unthrottled — it's just DOM)
bus.emit('output.heatmap', raw);
}
```
Each output component subscribes to its own bus topic. The FlowField listens to `output.visual`, the SynthVisualizer to `output.synth`, etc. Throttling for synth param sends lives inside the synth subscriber, not in the routing.
Each provider creates its store and exposes it via context. The `SessionProvider` handles persistence (auto-save on interval, load on mount, URL param parsing).
The C15 synth engine requires `SharedArrayBuffer` for its audio ring buffer. This means the server **must** send:
```
Cross-Origin-Opener-Policy: same-origin
Cross-Origin-Embedder-Policy: require-corp
```
The current codebase uses `serve-coop.py` for this. **Vite dev server must be configured** with these headers or C15 synth mode will fail silently. This should be validated in Phase 1, not Phase 3.
```typescript
// vite.config.ts
export default defineConfig({
plugins: [solidPlugin()],
server: {
headers: {
'Cross-Origin-Opener-Policy': 'same-origin',
'Cross-Origin-Embedder-Policy': 'require-corp',
},
},
});
```
### 2. Worker Lifecycle (Currently Leaked)
Workers are **never terminated** in the current code:
-`nisps-wasm-worker.js` — created lazily on first `trainAsync()`, lives forever
-`arpeggiator-worker.js` — created on arpeggiator init, lives forever
In a SolidJS app where components mount/unmount, workers must be cleaned up in `onCleanup()`. The current code leaks them because the page never changes — the SolidJS version will need explicit termination, especially if engine switching or mode changes recreate IML instances.
**Add to architecture**: `core/iml.ts` must expose a `dispose()` method that terminates the worker. `synth-store` cleanup must terminate the arpeggiator worker.
### 3. AudioContext Lifecycle Gotchas
- **C15Bridge creates its own AudioContext internally**; Faust engines require one passed in. This API inconsistency means `useAudioContext()` can't own the single context — C15 creates its own.
- **No `dispose()` on C15Bridge** — relies on GC of the AudioContext. Engine switching disconnects audio nodes (line 1121 in a-app.js) but doesn't explicitly close the context.
- **AudioWorklet module loading** via `addModule()` — browsers deduplicate, but if the context is recreated, modules must be re-registered.
**Add to architecture**: The `useAudioContext` hook needs to handle two patterns: "I own the context" (Faust) vs "the engine owns the context" (C15). Consider normalizing this in the transplant — make C15Bridge accept an external AudioContext.
### 4. Debug Probe Synchrony Contract
The Playwright tests depend on `window.__nisps` methods being **synchronous**:
-`thumbsUp()` → synchronously calls `addExample()` before returning
-`saveState()` → synchronously writes to localStorage
-`train()` → synchronously trains and returns loss (not the async variant)
In SolidJS, store updates are batched by default. If `setInputs()` writes to a signal but the effect that runs inference hasn't flushed yet, the probe will return stale data. **The probe must bypass SolidJS reactivity** and call imperative methods directly on the IML instance.
**Add to Phase 5 (debug probe)**: Build the probe as a direct imperative bridge to the IML/store internals, not as a reactive consumer. Use `batch()` or `untrack()` where needed.
### 5. MIDI CC Map — Engine-Scoped Dynamic Storage Keys
MIDI CC maps are stored with engine-scoped localStorage keys: `nisps-midi-cc-map:${activeEngine.id}`. When the engine switches:
1. Current map is saved to the old engine's key
2. New map is loaded from the new engine's key
3.`midiCCMap` and `midiCCOverrides` arrays are mutated in-place
In SolidJS, in-place array mutation (`arr.length = 0; arr.push(...)`) won't trigger reactivity. The `midi-store` must use `setStore(produce(...))` or replace arrays entirely. The engine-scoped key pattern needs to be replicated in `session-store.ts`.
When EOC modules are added/removed in "Shared" mode:
1. MLP output count changes → MLP must be destroyed and recreated
2. Training examples are lost (different output dimensionality)
3. Audio graph nodes must be reconnected
4. Heatmap must be rebuilt
This is a **cascade of side effects** triggered by a single user action. In the current code it's handled by a `window.addEventListener('eoc:change', ...)` handler that orchestrates everything imperatively.
In SolidJS, this should be modeled as: EOC module list is a store → derived signal computes total output count → `createEffect` watches output count and triggers MLP resize when it changes. But the MLP resize is async (WASM allocation) and has a confirmation dialog ("this will clear examples"). **Effects can't show dialogs**.
**Proposed pattern**: EOC store exposes a `pendingResize` signal. A component watches it and shows the confirmation UI. On confirm, an action triggers the actual resize. Don't try to make this fully reactive — keep it as an explicit action flow.
### 7. WASM + Emscripten Glue Loading in Vite
The WASM is loaded via Emscripten's `nisps.js` glue file, which does its own `fetch()` of `nisps.wasm` using a relative path. Vite's asset handling will hash filenames in production builds, breaking the hardcoded path.
**Options**:
- Configure Vite to copy WASM files to `public/` (no hashing, always available at known path)
- Modify the Emscripten glue to accept a custom `locateFile` override
- Use Vite's `?url` import to get the resolved asset path and pass it to the WASM loader
This must be spiked in Phase 1. Same issue applies to C15 WASM (`c15/c15_engine.wasm`), C15 parameters (`c15/parameters.json`), and Faust DSP files.
### 8. Session Presets Are Shared Across Apps
`nisps-session-presets` localStorage key is **shared across all three current apps**. Since we're consolidating to one app, this is fine — but the key should be documented, and the migration should handle importing presets saved by the old app.
### 9. Lazy Loading Needs Suspense Boundaries
Three features are lazily loaded:
- **ShapeSeq**: dynamic `import()` when `?shapeseq=1`
- **Hand tracking**: imports MediaPipe from CDN (`cdn.jsdelivr.net`) — external dependency that can fail
- **Audio Canvas**: created on first switch to audio-canvas mode
SolidJS `lazy()` + `<Suspense>` handles this naturally, but:
- MediaPipe CDN fetch failure needs a fallback UI (not just a blank screen)
- ShapeSeq lazy loading should show a loading state, not block the whole app
- Audio Canvas creation involves AudioContext (requires gesture) — can't be wrapped in Suspense naively
### 10. CSS Animations Are Stateful
Several CSS classes trigger animations that encode UI state:
If SolidJS re-renders a component (e.g., `<Show>` toggling), CSS animations restart from the beginning. For the pulse animation this is fine, but for the flash animations, a re-render mid-flash would cause visual glitches.
**Mitigation**: Use `classList` toggling on stable DOM nodes rather than conditional rendering for animation-bearing elements. Or use the Web Animations API for imperative control.
### 11. `window.__nispsEoc` Is Unconditional
Unlike `window.__nisps` (gated by `?debug=1`), `window.__nispsEoc` is **always exposed** (line 1528 in a-app.js). It provides `trainingTarget` getter/setter and `imlEoc` reference. If external code depends on this, it needs to be preserved in the SolidJS version unconditionally.
### 12. Inference Should NOT Be in rAF
The plan puts inference in a `requestAnimationFrame` loop. But inference only needs to run **when inputs change** (joystick drag, gamepad poll, hand tracking frame). Running it every frame when the joystick is idle wastes CPU.
**Better pattern**: Run inference reactively — `createEffect` watching `inputState.joyX` and `inputState.joyY`. When they change, run inference and update outputs. The canvas render loops (FlowField, SynthVisualizer) still run on rAF for smooth animation, but they just read the latest outputs signal — they don't trigger inference.
Exception: gamepad polling needs a rAF loop to read the Gamepad API, but that loop should only set input signals, not run inference directly.
