From 13112cf4479b0b1c861b96204c6ab09935be29d8 Mon Sep 17 00:00:00 2001 From: monkey-w1n5t0n Date: Tue, 7 Apr 2026 08:49:52 +0200 Subject: [PATCH] Add mission infrastructure for SolidJS migration - Worker skill: solidjs-worker (SKILL.md) - Services manifest (services.yaml) - Init script (init.sh) - Library: architecture.md, environment.md, user-testing.md, solidjs-patterns.md Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com> --- .factory/init.sh | 18 ++++ .factory/library/architecture.md | 123 +++++++++++++++++++++++ .factory/library/environment.md | 45 +++++++++ .factory/library/solidjs-patterns.md | 124 ++++++++++++++++++++++++ .factory/library/user-testing.md | 74 ++++++++++++++ .factory/services.yaml | 23 +++++ .factory/settings.json | 5 + .factory/skills/solidjs-worker/SKILL.md | 107 ++++++++++++++++++++ 8 files changed, 519 insertions(+) create mode 100644 .factory/init.sh create mode 100644 .factory/library/architecture.md create mode 100644 .factory/library/environment.md create mode 100644 .factory/library/solidjs-patterns.md create mode 100644 .factory/library/user-testing.md create mode 100644 .factory/services.yaml create mode 100644 .factory/settings.json create mode 100644 .factory/skills/solidjs-worker/SKILL.md diff --git a/.factory/init.sh b/.factory/init.sh new file mode 100644 index 0000000..0b4af9d --- /dev/null +++ b/.factory/init.sh @@ -0,0 +1,18 @@ +#!/usr/bin/env bash +# Idempotent environment setup for SolidJS migration +set -e + +REPO_ROOT="/home/w1n5t0n/src/MEMLNaut-NISPS-SOLIDJS" +cd "$REPO_ROOT" + +# Install root npm dependencies if needed +if [ ! -d "node_modules" ]; then + npm install +fi + +# Ensure Playwright browsers are available +if ! npx playwright install --dry-run chromium 2>/dev/null; then + npx playwright install chromium 2>/dev/null || true +fi + +echo "Environment ready. Vite dev server: npx vite --port 5174" diff --git a/.factory/library/architecture.md b/.factory/library/architecture.md new file mode 100644 index 0000000..9d516af --- /dev/null +++ b/.factory/library/architecture.md @@ -0,0 +1,123 @@ +# Architecture + +How the MEMLNaut NISPS SolidJS app works. + +## High-Level Architecture + +``` +index.html + └── App.tsx (root component, provider tree) + ├── BusProvider (signal bus context) + ├── MLProvider (ML store context) + ├── InputProvider (input store context) + ├── OutputProvider (output store context) + ├── SynthProvider (synth store context) + ├── EOCProvider (EOC store context) + └── SessionProvider (persistence) + └── ImmersiveLayout.tsx + ├── Background canvases (FlowField or SynthVisualizer) + ├── HeatmapStrip + ├── Joystick + JoyMap + ├── RLButtons + ├── Dock + DrawerStack + └── Floating controls +``` + +## Data Flow + +``` +User Input (drag/gamepad/hands) + → Input Store (joyX, joyY) + → Input Pipeline (deadzone, zoom, curve, smoothing) + → ML Store (setInputs on IML) + → IML.process() (WASM inference) + → Outputs Signal (Float32Array, single signal write) + → Output Routing (apply overrides, route by mode) + ├── FlowField (visual mode, own rAF loop reads signal) + ├── SynthEngine (synth mode, throttled 50ms) + ├── MIDIOutput (midi-cc mode) + └── AudioCanvas (audio-canvas mode) +``` + +## Key Components + +### Signal Bus (`bus/signal-bus.ts`) +Unified event system replacing EventBus + CustomEvents. Topics with `equals: false` for event semantics. Wildcard matching via `match()`. + +### ML Store (`stores/ml-store.ts`) +- IML instances (WasmIML) live OUTSIDE the store (opaque handles) +- `outputs` is a `createSignal` (NOT a store property — proxies break typed arrays) +- Store tracks: loss, lossHistory, exampleCount, isTraining, spreadLevel, noiseLevel, undoStack + +### Input Store (`stores/input-store.ts`) +- joyX, joyY as store properties +- Input pipeline config (deadzone, zoom, curve, etc.) + +### Output Store (`stores/output-store.ts`) +- Output mode (visual/synth/midi-cc/audio-canvas) +- Per-mode overrides arrays +- Output pipeline config (globalCurve, smoothing, slewRate, freezeGate) + +### Dual IML System +- `imlJoy`: 2 inputs (joystick), dynamic outputs +- `imlHand`: 14 inputs (hand tracking), dynamic outputs +- `activeIml`: points to whichever is active +- EOC IML: separate instance for Linked/Independent modes + +### WASM Integration +- WASM files in `public/` (no Vite hashing) +- Emscripten glue loaded via dynamic import +- Training worker: lazy-loaded, own WASM instance +- Heap management: persistent buffers for inference + +### Canvas Components +Each canvas owns its rAF render loop. The component reads the outputs signal each frame. No shared orchestrator. + +### Debug Probe +`window.__nisps` exposed when `?debug=1`. All methods synchronous, bypassing SolidJS batching. `window.__nispsEoc` always exposed. + +## Transplanted Modules (from playground/js/) + +These are copied with minimal TypeScript adaptation: +- `core/wasm/` — WASM binaries + Emscripten glue (unchanged) +- `core/iml.ts` — WasmIML wrapper (from nisps/nisps-wasm.js) +- `core/dataset.ts` — FIFO ring buffer (from nisps/dataset.js) +- `core/synth/` — All synth engines (c15-bridge, c15-adapter, param-map, presets, additive, fm, faust) +- `core/audio/` — Arpeggiator, AudioCanvas, MIDI I/O +- `core/eoc/` — EOC chain + effect modules +- `core/shapeseq/` — ShapeSeq sequencer +- `core/ui/` — Pure math/logic modules (input-pipeline, output-pipeline, control-surface, visualizer) + +## Rewritten Modules + +Everything else is rewritten in SolidJS: +- All DOM manipulation → JSX components +- Event wiring → signal bus + reactive effects +- State management → SolidJS stores/signals +- Layout/CSS → new CSS with same design system +- Init/boot → provider tree + onMount hooks + +## Key Constants + +- N_JOY_INPUTS = 2, N_HAND_INPUTS = 14 +- N_VISUAL_OUTPUTS = 20, N_SYNTH_OUTPUTS = 126 (C15) +- STORAGE_KEY = 'nisps-a-immersive' +- MAX_UNDO = 20, PARAM_DEAD_ZONE = 0.002, PARAM_SEND_INTERVAL = 50 +- 18 C15 synth sections (SYNTH_SECTIONS) +- 20 visual param names/colors (VISUAL_PARAM_NAMES, VISUAL_PARAM_COLORS) + +## COOP/COEP Requirement + +C15 synth requires SharedArrayBuffer. Vite dev server must send: +``` +Cross-Origin-Opener-Policy: same-origin +Cross-Origin-Embedder-Policy: require-corp +``` + +## MLP Resize Cascade + +When output count changes (mode switch, engine switch, EOC mode change): +1. Warn user if training data exists → confirmation dialog +2. Destroy old MLP, create new with different output count +3. Warm-start: copy hidden layer weights, reset output layer +4. Update output signal, rebuild heatmap diff --git a/.factory/library/environment.md b/.factory/library/environment.md new file mode 100644 index 0000000..f72f283 --- /dev/null +++ b/.factory/library/environment.md @@ -0,0 +1,45 @@ +# Environment + +Environment variables, external dependencies, and setup notes. + +**What belongs here:** Required env vars, external dependencies, platform-specific notes. +**What does NOT belong here:** Service ports/commands (use `.factory/services.yaml`). + +--- + +## External Dependencies + +- **Node.js**: v25.4.0 +- **npm**: v11.7.0 +- **Playwright**: v1.59.1 (chromium headless shell v1217 installed) +- **Emscripten**: NOT required — existing WASM binaries are transplanted as-is + +## WASM Binaries (transplanted from playground/) + +- `public/nisps.wasm` (36KB) — ML engine +- `public/nisps.js` (14KB) — Emscripten glue +- `public/nisps-wasm-worker.js` — Training worker (from playground/js/nisps/) +- `public/c15/c15_engine.wasm` (270KB) — C15 synthesizer +- `public/c15/parameters.json` (163KB) — C15 parameter definitions +- `public/c15/worklet-processor.js` — C15 AudioWorklet +- `public/faust/` — Faust DSP files (fetched at runtime) + +## CDN Dependencies + +- **MediaPipe** (cdn.jsdelivr.net): Hand tracking only, lazy-loaded on mode switch to Hands +- **Google Fonts**: JetBrains Mono (loaded in index.html) + +## Browser Requirements + +- SharedArrayBuffer support (requires COOP/COEP headers) +- WebMIDI support (MIDI CC mode) +- AudioWorklet support (C15 synth) +- getUserMedia (hand tracking camera) +- Gamepad API (gamepad input) + +## Platform Notes + +- Linux x86_64 (kernel 6.14.0) +- No Docker +- 62GB RAM, 12 CPU cores +- ffmpeg 7.1.1 available (not used by SolidJS app) diff --git a/.factory/library/solidjs-patterns.md b/.factory/library/solidjs-patterns.md new file mode 100644 index 0000000..6f55c08 --- /dev/null +++ b/.factory/library/solidjs-patterns.md @@ -0,0 +1,124 @@ +# SolidJS Patterns + +Patterns and conventions for the SolidJS migration. + +**What belongs here:** SolidJS-specific patterns, reactivity gotchas, store design patterns. +**What does NOT belong here:** General architecture (use architecture.md), environment details (use environment.md). + +--- + +## Store vs Signal Decision + +| Data Type | Use | Why | +|-----------|-----|-----| +| Float32Array outputs | `createSignal` | Store proxies break typed arrays | +| Structured config | `createStore` | Fine-grained reactivity on nested properties | +| Opaque handles (IML, engines) | Module-scope variables | Not reactive — referenced imperatively | +| Event streams | Signal bus topic | `equals: false` for event semantics | + +## Reactive Inference Pattern + +```typescript +// Inference runs when inputs change, NOT on rAF +createEffect(() => { + const x = inputState.joyX; + const y = inputState.joyY; + const iml = getActiveIml(); + if (!iml) return; + iml.setInput(0, x); + iml.setInput(1, y); + iml.process(); + setOutputs(new Float32Array(iml.getOutputs())); +}); +``` + +## Canvas Component Pattern + +```typescript +const MyCanvas = () => { + let canvasRef: HTMLCanvasElement; + const data = bus.topic('output.visual'); + + onMount(() => { + const ctx = canvasRef.getContext('2d')!; + let raf: number; + const loop = () => { + const d = data(); + if (d) renderFrame(ctx, d); + raf = requestAnimationFrame(loop); + }; + raf = requestAnimationFrame(loop); + onCleanup(() => cancelAnimationFrame(raf)); + }); + + return ; +}; +``` + +## Provider Pattern + +```typescript +const MLContext = createContext(); +export const MLProvider = (props: { children: any }) => { + const [outputs, setOutputs] = createSignal(new Float32Array(126)); + const [mlState, setMlState] = createStore({ /* ... */ }); + const store = { outputs, setOutputs, mlState, setMlState }; + return {props.children}; +}; +export const useML = () => useContext(MLContext)!; +``` + +## Debug Probe (Synchronous) + +```typescript +// Must bypass SolidJS batching for sync probe +if (urlParams.debug) { + window.__nisps = { + setInputs: (x, y) => { + batch(() => { + setInputState({ joyX: x, joyY: y }); + }); + // Imperative: run inference directly + iml.setInput(0, x); + iml.setInput(1, y); + iml.process(); + setOutputs(new Float32Array(iml.getOutputs())); + }, + getOutputs: () => Array.from(outputs()), + // ... etc + }; +} +``` + +## CSS Pattern + +```css +/* Per-component CSS files, no framework */ +.component-name { + /* Use CSS custom properties from root */ + background: var(--glass-bg); + backdrop-filter: blur(var(--glass-blur)); + border: 1px solid var(--glass-border); +} +``` + +## Worker Cleanup + +```typescript +// Always clean up workers +onCleanup(() => { + trainingWorker?.terminate(); + arpeggiatorWorker?.terminate(); + imlJoy?.destroy(); + imlHand?.destroy(); +}); +``` + +## Key Gotchas + +1. **Store proxies and typed arrays**: NEVER put Float32Array in a store. Use createSignal. +2. **equals: false**: Required for bus topics. Every emit triggers subscribers even with same data. +3. **CSS animations**: Use classList toggling on stable DOM nodes. Don't use `` for animated elements. +4. **EOC resize cascade**: Cannot be a reactive effect (needs confirmation dialog). Use pending action pattern. +5. **Probe synchrony**: All probe methods must be synchronous. Use batch() + imperative IML calls. +6. **Idle inference waste**: Don't run inference on rAF. Use reactive createEffect on inputs. diff --git a/.factory/library/user-testing.md b/.factory/library/user-testing.md new file mode 100644 index 0000000..895b631 --- /dev/null +++ b/.factory/library/user-testing.md @@ -0,0 +1,74 @@ +# User Testing + +Testing surface, required tools, and resource cost classification. + +--- + +## Validation Surface + +**Surface**: Browser (web application) +**URL**: http://localhost:5174 (Vite dev server) +**Comparison URL**: http://localhost:7331/a-immersive.html (old playground app) + +**Primary testing tool**: agent-browser +**Secondary tool**: Playwright e2e tests via `window.__nisps` debug probe + +### Debug Probe Interface + +When loaded with `?debug=1`, `window.__nisps` exposes: + +| Method | Returns | Sync/Async | +|--------|---------|------------| +| getOutputs() | Float32Array(126) | Sync | +| getLoss() | number \| null | Sync | +| getWeights() | Float32Array | Sync | +| getExampleCount() | number | Sync | +| setInputs(x, y) | void | Sync | +| thumbsUp() | void | Sync | +| thumbsDown() | void | Sync | +| train() | number (loss) | Sync | +| trainAsync() | Promise | Async | +| randomise() | void | Sync | +| clearExamples() | void | Sync | +| saveState() | void | Sync | +| evalLoss() | number \| null | Sync | +| inferBatch(points) | Float32Array[] | Sync | +| getLayerStats() | object[] | Sync | + +`window.__nispsEoc` is ALWAYS available (no ?debug=1 required): +- `trainingTarget` getter/setter ('synth' | 'eoc') +- `imlEoc` getter (WasmIML | null) + +### DOM Selectors for Validation + +Key CSS selectors for UI state assertions: +- `.dock` — right-side dock container +- `.drawer` — drawer panels (`.hidden` when closed) +- `.heatmap-cell` — heatmap parameter bars +- `.heatmap-cell-bar` — bar width element +- `#status-text` — status line text +- `.rl-btn` — RL thumbs up/down buttons +- `.pill-opt[data-mode]` — output mode pills +- `.joystick-container` — joystick wrapper +- `#joy-map` — joystick canvas +- `.noise-ring` — noise indicator + +### Skip List + +These assertions require physical hardware and cannot be validated in headless browser: +- VAL-JOY-006 (Gamepad) — requires physical gamepad +- VAL-MIDI-001 (MIDI output) — requires physical MIDI device +- VAL-HAND-002 (Camera) — requires camera permission + gesture + +## Validation Concurrency + +**Machine specs**: 62GB RAM, 12 CPUs, ~23GB available + +**agent-browser instances**: +- Each instance: ~300MB RAM (lightweight app) +- Dev server: ~200MB RAM +- Per-validator total: ~500MB +- Usable headroom (70% of 23GB): ~16GB +- Max concurrent: 5 validators (5 × 500MB = 2.5GB, well within budget) + +**Playwright tests**: Run serially (single browser, single server) diff --git a/.factory/services.yaml b/.factory/services.yaml new file mode 100644 index 0000000..76b4e2d --- /dev/null +++ b/.factory/services.yaml @@ -0,0 +1,23 @@ +commands: + install: npm install + install-solid: npm install && cd /home/w1n5t0n/src/MEMLNaut-NISPS-SOLIDJS && npm install + typecheck: npx tsc --noEmit + build: npx vite build + test-old: npx playwright test --reporter=line + lint: npx eslint src/ --ext .ts,.tsx 2>/dev/null || true + dev: npx vite --port 5174 + +services: + vite-dev: + start: npx vite --port 5174 --host + stop: lsof -ti :5174 | xargs kill 2>/dev/null || true + healthcheck: curl -sf http://localhost:5174 > /dev/null 2>&1 + port: 5174 + depends_on: [] + + old-playground: + start: cd playground && python3 -m http.server 7331 + stop: lsof -ti :7331 | xargs kill 2>/dev/null || true + healthcheck: curl -sf http://localhost:7331/a-immersive.html > /dev/null 2>&1 + port: 7331 + depends_on: [] diff --git a/.factory/settings.json b/.factory/settings.json new file mode 100644 index 0000000..565f14a --- /dev/null +++ b/.factory/settings.json @@ -0,0 +1,5 @@ +{ + "enabledPlugins": { + "core@factory-plugins": true + } +} \ No newline at end of file diff --git a/.factory/skills/solidjs-worker/SKILL.md b/.factory/skills/solidjs-worker/SKILL.md new file mode 100644 index 0000000..e2db6b7 --- /dev/null +++ b/.factory/skills/solidjs-worker/SKILL.md @@ -0,0 +1,107 @@ +--- +name: solidjs-worker +description: General-purpose SolidJS feature implementation worker for the NISPS migration +--- + +# SolidJS Worker + +NOTE: Startup and cleanup are handled by `worker-base`. This skill defines the WORK PROCEDURE. + +## When to Use This Skill + +All feature implementation for the SolidJS migration. This worker builds components, stores, hooks, and transplants core modules from the old playground app. + +## Required Skills + +- **agent-browser**: For manual verification of UI behavior in the browser. Use after implementation to verify component rendering and interactions. +- **tuistory**: NOT required for this mission. + +## Work Procedure + +### Step 1: Understand the Feature + +1. Read `mission.md` in the mission directory for overall mission context. +2. Read `.factory/library/architecture.md` for system architecture. +3. Read `.factory/library/environment.md` for environment details. +4. Read the assigned feature from `features.json` carefully — understand every `expectedBehavior` and `fulfills` assertion. +5. Read the relevant assertion descriptions from `validation-contract.md`. + +### Step 2: Reference the Old Code + +1. The old playground app is at `playground/` (READ-ONLY reference). +2. `playground/js/a-app.js` is the main app (~4300 lines, read in chunks). +3. Identify the exact behavior to replicate by reading the relevant old code sections. +4. For transplanted modules (listed in `architecture.md`), copy from the old code and adapt to TypeScript with minimal changes. + +### Step 3: Write Tests First (TDD) + +1. Write failing tests BEFORE implementation. Tests go in `tests/` at the repo root. +2. For SolidJS components: use vitest + solid-testing-library (install if needed). +3. For behavioral assertions matching the validation contract: write Playwright e2e tests that use the `window.__nisps` debug probe. +4. For transplanted core modules: write unit tests verifying the same behavior as the old JS code. +5. Run tests to confirm they fail (red). + +### Step 4: Implement + +1. Follow the architecture in `.factory/library/architecture.md`. +2. Use SolidJS patterns from `.factory/library/solidjs-patterns.md`. +3. For transplanted modules: copy from `playground/js/`, add TypeScript types, adapt imports. +4. For new SolidJS code: use stores (createStore) for structured config, signals (createSignal) for Float32Arrays. +5. Ensure all workers are cleaned up in `onCleanup()`. +6. Ensure CSS uses the glass morphism design system (custom properties from `a-immersive.css`). + +### Step 5: Verify + +1. Run all tests: `npm test` (or the appropriate test command from services.yaml). +2. Run typecheck: `npx tsc --noEmit`. +3. Start the Vite dev server: `npx vite --port 5174`. +4. Use **agent-browser** to visually verify the feature: + - Navigate to `http://localhost:5174?debug=1` + - Verify the UI renders correctly + - Interact with the component and verify behavior + - Take screenshots as evidence +5. Compare behavior against the old app at `http://localhost:7331/a-immersive.html?debug=1` if needed. +6. Stop the dev server when done. + +### Step 6: Commit and Hand Off + +1. Stage and commit all changes with a descriptive message. +2. Report all verification evidence in the handoff. + +## Example Handoff + +```json +{ + "salientSummary": "Implemented Joystick component with pointer events and follow mode. Created input-store with joyX/joyY signals. Verified dragging updates position [0,1] and follow mode toggles on double-tap. All 5 tests pass, typecheck clean.", + "whatWasImplemented": "src/components/input/Joystick.tsx (pointer events, position tracking, follow mode toggle), src/stores/input-store.ts (joyX, joyY, isDragging, followMode), src/components/input/JoyMap.tsx (canvas minimap scaffold), tests/e2e/joystick.spec.js (3 tests: drag, follow, clamp)", + "whatWasLeftUndone": "JoyMap trail rendering deferred to layout milestone", + "verification": { + "commandsRun": [ + { "command": "npx tsc --noEmit", "exitCode": 0, "observation": "No type errors" }, + { "command": "npx vitest run tests/e2e/joystick.spec.js", "exitCode": 0, "observation": "3 tests passing" } + ], + "interactiveChecks": [ + { "action": "Navigate to localhost:5174?debug=1, drag joystick", "observed": "Joystick dot follows pointer, probe.setInputs matches visual position" }, + { "action": "Double-tap joystick", "observed": "Follow badge appears, pulse animation starts" } + ] + }, + "tests": { + "added": [ + { "file": "tests/e2e/joystick.spec.js", "cases": [ + { "name": "drag updates position", "verifies": "VAL-JOY-001" }, + { "name": "double-tap toggles follow", "verifies": "VAL-JOY-002" }, + { "name": "input clamping", "verifies": "VAL-JOY-005" } + ]} + ] + }, + "discoveredIssues": [] +} +``` + +## When to Return to Orchestrator + +- Feature depends on infrastructure not yet built (e.g., a store that doesn't exist) +- Requirements are ambiguous — cannot determine correct behavior from old code or contract +- Existing bugs in the codebase block this feature +- Need to modify files in `playground/` (off-limits without orchestrator approval) +- Cannot complete within the session — return with `whatWasLeftUndone` detailed