diff --git a/ALIGNMENT.md b/ALIGNMENT.md index 2672e8e..08da0a3 100644 --- a/ALIGNMENT.md +++ b/ALIGNMENT.md @@ -49,7 +49,7 @@ Pick A. Update perf.hpp + every `nisps/` use site. ### Q1: Per-mode MLP architectures or one shared shape? (2026-04-29) -Schemas currently declare per-mode `hidden_layers` (some `[10, 10, 14]`, some `[10, 14, 18]`). Browser is fixed at one shape; firmware compiles per-mode. This works for now. Is the mission served by maintaining per-mode shapes (research diversity) or by collapsing to one (simpler ops)? +Schemas declare per-mode `input_size`/`hidden_layers`/`output_size` (some hidden `[10, 10, 14]`, some `[10, 14, 18]`; inputs 4 or 10; outputs 24–56). As of P5.3 BOTH targets honour them: firmware compiles per-mode, and the browser now reshapes the runtime-shaped WASM net to the active mode's `ml` config on mode switch (was fixed at one 32→126 shape). This works for now. Is the mission served by maintaining per-mode shapes (research diversity) or by collapsing to one (simpler ops)? ### Q2: How to express "advanced" features (gradient flow, weight health) without cluttering modes? (2026-04-29) diff --git a/MAP.md b/MAP.md index 4f602ea..a2b9584 100644 --- a/MAP.md +++ b/MAP.md @@ -48,7 +48,12 @@ anchor + locked decisions) and the `docs/specs/*-spec.md` set. with snap/magnetism/minimap-demotion), `SplitStage`/`OutputStage`/`InputMini`/`Manifold` (canvas, rect↔circular + feedback markers), `Dock` (top Mode selector + 5 vertically-centred drawers), `Drawers` (Learning/Inputs/ Outputs/Settings/Help), `VerdictCluster` (mode-aware), `ReadoutStrip`, `OutputEditor`/`CurvePad`, `icons.tsx` - (monochrome currentColor SVG), `model.ts`, `output-mode.ts`. + (monochrome currentColor SVG), `model.ts` (`MF_MODES` catalogue — schema-backed modes DERIVED from + `manifold/src/modes/generated/`; carries per-mode `ml` net shape + `engineId`), `output-mode.ts`. +- `manifold/src/modes/generated/` — codegen output (`*_schema.ts`, do NOT hand-edit): `ModeSchema` + consts (mode_id, engine_id, ml dims, params, voice_spaces, ui) — the SOURCE OF TRUTH for `MF_MODES`. + Switching mode reshapes the WASM net to the mode's `ml` dims (ConsoleApp P5.3; boot mode paf_synth → + 4→[10,10,14]→33). - `manifold/src/dock/` — `OutputControlRow` (off/fixed/live + mute + solo/arm + min/max/curve), `output-state.ts`, `OutputsBackendConfig.tsx` (per-backend specialised Outputs panel), `BackendAdvanced.tsx`. - `manifold/src/backends/` — `OutputBackend` adapter + `BackendManager` (spine consumer); `midi-backend.ts` @@ -95,7 +100,7 @@ the "BUILD DELTAS" block at the top of `docs/specs/vcv-module.md`). `src/MEMLNau - `schemas/midi_devices/.json` (×6) — CC-controllable external synths (Moog Sub 37 / Sub Phatty, Creamware Pro-12 ASB, Elektron Analog Keys, ASM Hydrasynth, Roland JD-800). Each param: `{id, cc, label, min, max, default, group}`. Canonical source for both firmware + browser device pickers. Verified-CC provenance + sources live in `synth-midi-cc.json` (repo root). ### `codegen/` — schema → C++/TS code -- `codegen/generate.ts` — Bun script: validates schemas via ajv (incl. the P5 firmware-fit check: exactly 3 hidden layers, dims ≤4096), emits per-mode `nisps/modes/generated/_schema.hpp` (`constexpr`, `nisps::modes::generated`) and `manifold/src/modes/generated/_schema.ts` (+ `types.ts`, `index.ts`). Idempotent; golden-tested in `run-all-tests.sh` stage 5. +- `codegen/generate.ts` — Bun script: validates schemas via ajv (incl. the P5 firmware-fit check: exactly 3 hidden layers, dims ≤4096), emits per-mode C++ `nisps/modes/generated/_schema.hpp` (`constexpr`, `nisps::modes::generated`) AND TS `manifold/src/modes/generated/_schema.ts` (+ `types.ts`, `index.ts`). Idempotent; golden-tested in `run-all-tests.sh` stage 5. The TS output is the SOURCE OF TRUTH consumed by `MF_MODES` (`manifold/src/console/model.ts`). - `codegen/generate-midi-devices.ts` — separate Bun script (isolated from the mode golden test): validates `schemas/midi_devices/` via ajv, emits `nisps/midi/generated/midi_devices.hpp` (no-heap `constexpr`) and `manifold/src/midi-devices/generated/{types,devices,index}.ts`. Idempotent. - `codegen/seed-midi-devices.ts` — one-time/idempotent seed deriving `schemas/midi_devices/*.json` from the `synth-midi-cc.json` research artifact (slugifies labels → ids, heuristic groups). - `codegen/templates/`, `codegen/tests/golden/` — reference templates + golden snapshot for paf_synth. @@ -136,7 +141,7 @@ the "BUILD DELTAS" block at the top of `docs/specs/vcv-module.md`). `src/MEMLNau - **Parity check**: `bash scripts/parity-check.sh`. - **All tests**: `bash scripts/run-all-tests.sh`. - **Playwright**: `cd manifold && node node_modules/.bin/playwright test` (non-snap node runner on the VPS — BUILD-PLAN gotcha; `bunx playwright test` works elsewhere). -- **Codegen**: `cd codegen && bun run generate.ts` (regenerates `nisps/modes/generated/` and `manifold/src/modes/generated/`). +- **Codegen**: `cd codegen && bun run generate.ts` (regenerates both `nisps/modes/generated/` C++ and `manifold/src/modes/generated/` TS). ## Conventions diff --git a/manifold/ONBOARDING.md b/manifold/ONBOARDING.md index 33afb81..dbad8b6 100644 --- a/manifold/ONBOARDING.md +++ b/manifold/ONBOARDING.md @@ -108,9 +108,15 @@ decision. Buffers are reused frame-to-frame; never assume a fresh array. when touching anything cross-cutting: - `types.ts`: `Focus`, `OutputMode`, `DrawerKey`, `DrawerDepth`, `FeedbackModeUI`, `SoloMode`, `Pin`, `FeedbackMarker`, `Snapshot`, and **`ConsoleCtx`** (the flat context handed to the dock). - - `model.ts`: the static instrument catalogue `MF_MODES`, `MFParam`, `ParamStatus` + - `model.ts`: the instrument catalogue `MF_MODES`, `MFParam`, `ParamStatus` (`off|fixed|live`), `ParamGroup`, plus `shapeValues()` (applies min/max/curve to raw engine - outputs) and `seededGradient()`. + outputs) and `seededGradient()`. **Schema-backed modes are DERIVED from the codegen + schemas in `src/modes/generated/`** (one-core-engine P5.2) — real param names/groups/count, + plus each mode's `ml` net shape (`MFMode.ml`) and schema `engine_id` (`MFMode.engineId`) + come from schema truth. A thin manifold OVERLAY (`SCHEMA_MODES` in model.ts) supplies only + label/glyph/ModeClass/input/ordering. Two schema-less manifold-only modes (`visualizer`, + `c15` placeholder) stay hand-written on `DEFAULT_MODE_ML`. Do NOT hand-edit + `src/modes/generated/` — it is codegen output (`bun run codegen/generate.ts`). ### The Dock (right-edge rail) — `src/console/Dock.tsx` + `Drawers.tsx` - 48px right rail: **TOP** = mode selector (the 5 output modes, popover); **MIDDLE** = 5 drawer @@ -214,6 +220,17 @@ a setting → `--r-*` tokens. (`wasm-worker.ts`) carries the current dims in its train message and re-creates its mirror net to match. Debug: `window.__nisps.reshape(nIn)` / `.describe()`. See the `manifold-mixed-inputs` memory for the locked design (adaptive slider viz when >2 dims is still pending). +- **Per-mode net dims (P5.3):** switching INSTRUMENT mode reshapes the net to that mode's schema + `ml` config (`MFMode.ml` — input/hidden/output + spread) via a `ConsoleApp` effect keyed on + `[engine, modeId]`. No confirm modal (switching instrument is deliberate); the axis-count + `ReshapeModal` above is for input-LAYOUT changes only. The effect depends on `engine`, so on boot + it fires once WASM is ready and lands the boot mode's dims (**paf_synth → 4→[10,10,14]→33**, weights + 809 — NOT the 32→126 default). The reshape-offer effect reads the engine's CURRENT `inputSize` + live, so a mode switch that changes arity doesn't spuriously prompt (its baseline tracks axis + COUNT, unchanged by a pure dim change). Non-schema modes restore `DEFAULT_MODE_ML` (32→126). + Debug seam for tests: under `?debug=1` ConsoleApp installs `window.__mf` + (`setMode`/`getModeId`/`paramCount`/`modeIds`) — the UI-level analogue of `__nisps`, since no + in-UI instrument picker exists yet (`ctx.modes`/`setModeId` are plumbed but unrendered). ### Feedback — `src/feedback/` - `controller.ts` — `FeedbackController`, framework-neutral, owned by ConsoleApp. **As of one-core- diff --git a/manifold/src/console/ConsoleApp.tsx b/manifold/src/console/ConsoleApp.tsx index 66617c0..0290814 100644 --- a/manifold/src/console/ConsoleApp.tsx +++ b/manifold/src/console/ConsoleApp.tsx @@ -58,6 +58,25 @@ import { useSettings, resolveInputMap } from '../settings/settings-store'; import { useBackendManager } from '../backends'; import { useInputLayer } from '../inputs'; +/** + * Instrument-mode debug seam, installed on `window.__mf` under `?debug=1` + * (see the effect in ConsoleApp). UI-level analogue of the engine + * `window.__nisps` probe — lets Playwright drive mode switches and read the + * rendered param count, since no in-UI mode picker exists yet. + */ +export interface MfDebugHook { + setMode: (id: string) => void; + getModeId: () => string; + paramCount: () => number; + modeIds: () => string[]; +} + +declare global { + interface Window { + __mf?: MfDebugHook; + } +} + let SNAP_ID = 0; /** Small pill-button style for the exploring-scratchpad banner controls. */ @@ -250,6 +269,20 @@ export function ConsoleApp({ focus: initialFocus = 'composite' }: ConsoleAppProp if (engine) engine.audio.setBackend(modeEngineId(modeId) as Parameters[0]); }, [engine, modeId]); + // Per-mode net dims (one-core-engine P5.3). On mode switch — and once WASM is + // ready on boot (this effect depends on `engine`, so it fires when the engine + // transitions null→ready with the boot mode) — reshape the runtime-shaped MLP + // to the active mode's schema `ml` config (warm-started; the C-side dataset + + // feedback reset, which the transient reset below also clears). NO confirm + // modal: switching instrument is already a deliberate act. The P2.3 axis-count + // modal stays for input-LAYOUT changes only (see the reshape-offer effect). + useEffect(() => { + if (!engine) return; + const { inputSize, outputSize, hidden, defaultSpread } = mode.ml; + engine.reshape({ inputSize, outputSize, hidden }, defaultSpread); + // eslint-disable-next-line react-hooks/exhaustive-deps + }, [engine, modeId]); + // reset transient state on mode switch useEffect(() => { setParams(mode.params.map((p) => ({ ...p }))); @@ -319,6 +352,27 @@ export function ConsoleApp({ focus: initialFocus = 'composite' }: ConsoleAppProp if (engine && n != null) engine.reshape({ inputSize: n }); }; + // ---- Debug seam for instrument-mode switching (`?debug=1`) ------------------ + // There is no instrument-mode picker in the UI yet (ctx.modes/setModeId are + // plumbed but unrendered), so Playwright drives mode switches through this + // window hook — the UI-level analogue of the engine `window.__nisps` probe. + // Exposes the current modeId, the rendered param count, and the mode ids so + // the schema-modes e2e can switch a mode and assert the derived shape. + useEffect(() => { + if (typeof window === 'undefined') return; + const q = new URLSearchParams(window.location.search); + if (q.get('debug') !== '1') return; + window.__mf = { + setMode: (id: string) => setModeId(id), + getModeId: () => modeId, + paramCount: () => params.length, + modeIds: () => MF_MODES.map((m) => m.id), + }; + return () => { + if (window.__mf) delete window.__mf; + }; + }, [modeId, params]); + // Drive a pad/joystick/manifold move through the input layer's XY-pad source, // then mirror the raw position into React state for readouts. The layer's loop // composes it with any other active sources and writes to the engine; we still diff --git a/manifold/src/console/model.ts b/manifold/src/console/model.ts index f17196a..1298214 100644 --- a/manifold/src/console/model.ts +++ b/manifold/src/console/model.ts @@ -1,6 +1,15 @@ /** - * Console — shared instrument model: the static modes catalogue + per-param - * shaping helpers. Ported from the window-global `model.jsx`. + * Console — shared instrument model: the modes catalogue + per-param shaping + * helpers. + * + * SOURCE OF TRUTH (one-core-engine P5.2): the schema-backed modes are DERIVED + * from the codegen-produced schemas in `src/modes/generated/` — real param + * names, groups, count, and each mode's `ml` config + `engine_id` come from + * schema truth, never hand-written. A thin manifold-side OVERLAY supplies only + * the display concerns a schema has no opinion on: label, glyph, ModeClass, + * input kind, and ordering. Two manifold-only modes with no schema + * (`visualizer`, `c15` placeholder) stay hand-written and use the default net + * shape. * * KEY CHANGE vs the JSX reference: the pseudo-inference `MF_infer` (sin/cos * placeholder) and the `useInstrument` hook are GONE. The `values` every @@ -13,11 +22,51 @@ * internal mode id). */ +import type { ModeSchema } from '../modes/generated/types'; +import { + BreakorSchema, + ChannelStripSchema, + ElysiamorfSchema, + MemlceliumSchema, + PafSynthSchema, + SlpWorkshopSchema, + SoundAnalysisMidiSchema, + VerbFxSchema, + XiasriSchema, +} from '../modes/generated'; + export type ParamStatus = 'off' | 'fixed' | 'live'; -export type ParamGroup = 'formant' | 'pitch' | 'amp' | 'filter' | 'fx' | 'mod'; +/** + * Param GROUP. Historically a small hand-picked union; now the group is the raw + * schema string (`'operators'`, `'envelope'`, `'kick'`, …) so the type is just + * `string`. Unknown groups fall back to the accent colour in the GROUP_COLOR + * maps that key off this field. + */ +export type ParamGroup = string; export type ModeClass = 'Synth' | 'Sequencer' | 'Controller' | 'Visual'; export type ModeInput = 'xy' | 'joystick' | 'audio_in'; +/** + * A mode's net shape — the engine dims the runtime-shaped WASM MLP is reshaped + * to when this mode is active (one-core-engine P5.3). Schema-backed modes carry + * their schema's `ml`; the manifold-only modes carry the default net shape. + */ +export interface ModeML { + inputSize: number; + outputSize: number; + hidden: [number, number, number]; + /** Spread used when (re)drawing the net's weights on reshape. */ + defaultSpread: number; +} + +/** The compiled default over-provisioned net (32→[10,14,18]→126). */ +export const DEFAULT_MODE_ML: ModeML = { + inputSize: 32, + outputSize: 126, + hidden: [10, 14, 18], + defaultSpread: 0.6, +}; + /** * Per-output control row — the unified store used by both the stage * (OutputStage / ReadoutStrip) and the Outputs/Routing dock. `status` is the @@ -35,6 +84,17 @@ export interface MFParam { min: number; max: number; curve: number; + /** + * Schema ENGINE-unit metadata (display/tooltips only — NOT the routing range). + * `min`/`max`/`curve`/`val` above stay the 0..1 routing-knob semantics; these + * carry the schema's real engine range, default, human label, and curve name + * for the disliked param so the UI can show what an output actually drives. + */ + schemaMin?: number; + schemaMax?: number; + schemaDefault?: number; + schemaLabel?: string; + schemaCurve?: string; /** Downstream silence — still computed + visible (distinct from `off`). */ muted?: boolean; /** Solo / arm — focus training on this output (dock-spec §1.2). */ @@ -56,6 +116,18 @@ export interface MFMode { glyph: string; input: ModeInput; params: MFParam[]; + /** + * The net shape this mode drives (one-core-engine P5.3). Reshaped into the + * engine on mode switch. Schema-backed modes carry their schema's `ml`; the + * manifold-only modes carry {@link DEFAULT_MODE_ML}. + */ + ml: ModeML; + /** + * The schema's `engine_id` (audio-engine metadata). NOTE: audio backend + * SELECTION still routes through {@link modeEngineId}, which is unchanged — + * this field is the schema-truth annotation, not the routing decision. + */ + engineId: string; placeholder?: boolean; badge?: string; } @@ -72,103 +144,110 @@ function mkParams(spec: Spec): MFParam[] { return out; } -export const MF_MODES: MFMode[] = [ +/** Derive a mode's net shape from its schema `ml` config. */ +function mlFromSchema(schema: ModeSchema): ModeML { + const h = schema.ml.hidden_layers; + return { + inputSize: schema.ml.input_size, + outputSize: schema.ml.output_size, + hidden: [h[0] ?? 10, h[1] ?? 14, h[2] ?? 18], + defaultSpread: schema.ml.default_spread, + }; +} + +/** + * Build the manifold param rows from a schema's params. The param NAME + GROUP + * are schema truth; the routing knobs (status/val/min/max/curve) keep their + * manifold defaults (0..1 routing range) — schema min/max/default/label/curve + * are surfaced as ENGINE-unit metadata for display only. + */ +function paramsFromSchema(schema: ModeSchema): MFParam[] { + return schema.params.map((p) => ({ + name: p.name, + group: p.group, + status: 'live' as ParamStatus, + val: 0.5, + min: 0, + max: 1, + curve: 0.5, + schemaMin: p.min, + schemaMax: p.max, + schemaDefault: p.default, + schemaLabel: p.label, + schemaCurve: p.curve, + })); +} + +/** + * Manifold-side display OVERLAY for a schema-backed mode — the only fields a + * schema has no opinion on. Everything else (params, ml, engineId) is derived. + */ +interface ModeOverlay { + label: string; + glyph: string; + cls: ModeClass; + input: ModeInput; + badge?: string; +} + +/** + * ORDERED list of schema-backed modes: `{ schema, overlay }`. Order here is the + * catalogue order. `xiasri` + `slp_workshop` are new browser-viable entries + * (they have schemas but weren't in the hand-written catalogue). The overlay is + * hand-picked display; the params/ml/engine_id come from the schema. + */ +const SCHEMA_MODES: ReadonlyArray<{ schema: ModeSchema; overlay: ModeOverlay }> = [ + { schema: PafSynthSchema, overlay: { label: 'PAF Synth', glyph: '∿', cls: 'Synth', input: 'xy' } }, { - id: 'paf_synth', - label: 'PAF Synth', - cls: 'Synth', - glyph: '∿', - input: 'xy', - params: mkParams([ - ['formant', ['F1', 'F2', 'F3', 'tilt', 'spread', 'skirt']], - ['pitch', ['root', 'glide', 'detune']], - ['amp', ['gain', 'attack', 'decay']], - ['filter', ['cutoff', 'res', 'env']], - ['fx', ['drive', 'air', 'width']], - ]), + schema: ChannelStripSchema, + overlay: { label: 'Channel Strip', glyph: '▤', cls: 'Synth', input: 'joystick' }, + }, + { schema: VerbFxSchema, overlay: { label: 'Verb FX', glyph: '◞', cls: 'Synth', input: 'joystick' } }, + { schema: ElysiamorfSchema, overlay: { label: 'Elysiamorf', glyph: '❋', cls: 'Synth', input: 'xy' } }, + { + schema: MemlceliumSchema, + overlay: { label: 'MEML Celium', glyph: '☷', cls: 'Sequencer', input: 'xy' }, + }, + { schema: BreakorSchema, overlay: { label: 'Breakor', glyph: '⊟', cls: 'Sequencer', input: 'joystick' } }, + { schema: XiasriSchema, overlay: { label: 'Xiasri', glyph: '✴', cls: 'Synth', input: 'joystick' } }, + { + schema: SlpWorkshopSchema, + overlay: { label: 'SLP Workshop', glyph: '☷', cls: 'Sequencer', input: 'xy' }, }, { - id: 'channel_strip', - label: 'Channel Strip', - cls: 'Synth', - glyph: '▤', - input: 'joystick', - params: mkParams([ - ['filter', ['lo', 'loMid', 'hiMid', 'hi']], - ['amp', ['comp', 'gate', 'makeup']], - ['fx', ['sat', 'width', 'glue', 'tilt', 'air']], - ]), - }, - { - id: 'verb_fx', - label: 'Verb FX', - cls: 'Synth', - glyph: '◞', - input: 'joystick', - params: mkParams([ - ['fx', ['size', 'decay', 'damp', 'diff']], - ['mod', ['rate', 'depth']], - ['filter', ['lo', 'hi']], - ]), - }, - { - id: 'elysiamorf', - label: 'Elysiamorf', - cls: 'Synth', - glyph: '❋', - input: 'xy', - params: mkParams([ - ['formant', ['grain', 'size', 'pos', 'spray']], - ['mod', ['rate', 'depth', 'jitter']], - ['amp', ['gain', 'env']], - ['filter', ['cutoff', 'res']], - ['fx', ['blur', 'shimmer', 'freeze', 'width']], - ]), - }, - { - id: 'memlcelium', - label: 'MEML Celium', - cls: 'Sequencer', - glyph: '☷', - input: 'xy', - params: mkParams([ - ['mod', ['cvA', 'cvB', 'gate', 'div']], - ['pitch', ['root', 'scale', 'oct']], - ['amp', ['vca', 'slew']], - ]), - }, - { - id: 'breakor', - label: 'Breakor', - cls: 'Sequencer', - glyph: '⊟', - input: 'joystick', - params: mkParams([ - ['mod', ['density', 'swing', 'fill', 'stutter']], - ['amp', ['punch', 'decay']], - ['filter', ['tone', 'crush']], - ['fx', ['glitch', 'rev']], - ]), - }, - { - id: 'sound_analysis_midi', - label: 'Sound Analysis → MIDI', - cls: 'Controller', - glyph: '⇉', - input: 'audio_in', - badge: '1-input', - params: mkParams([ - ['mod', ['cc1', 'cc2', 'cc3', 'cc4']], - ['pitch', ['note', 'bend']], - ['amp', ['vel', 'press']], - ]), + schema: SoundAnalysisMidiSchema, + overlay: { label: 'Sound Analysis → MIDI', glyph: '⇉', cls: 'Controller', input: 'audio_in' }, }, +]; + +function modeFromSchema(schema: ModeSchema, overlay: ModeOverlay): MFMode { + return { + id: schema.mode_id, + label: overlay.label, + cls: overlay.cls, + glyph: overlay.glyph, + input: overlay.input, + badge: overlay.badge, + params: paramsFromSchema(schema), + ml: mlFromSchema(schema), + engineId: schema.engine_id, + }; +} + +/** + * Manifold-only modes with NO schema — hand-written params on the DEFAULT net + * shape. `visualizer` is a pure browser visual; `c15` is the "Powerful Synth + * Engine" placeholder (id stays `c15`; the string "C15" must never surface). + */ +const MANIFOLD_ONLY_MODES: MFMode[] = [ { id: 'visualizer', label: 'Visualizer', cls: 'Visual', glyph: '◑', input: 'xy', + ml: DEFAULT_MODE_ML, + engineId: 'thru', params: mkParams([ ['mod', ['hue', 'sat', 'flow', 'warp']], ['amp', ['bloom', 'fade']], @@ -184,10 +263,17 @@ export const MF_MODES: MFMode[] = [ input: 'xy', placeholder: true, badge: 'soon', + ml: DEFAULT_MODE_ML, + engineId: 'thru', params: mkParams([['amp', ['a', 'b']]]), }, ]; +export const MF_MODES: MFMode[] = [ + ...SCHEMA_MODES.map(({ schema, overlay }) => modeFromSchema(schema, overlay)), + ...MANIFOLD_ONLY_MODES, +]; + /** Mirrors the engine's `applyCurve` (≈0.43 ≈ linear). */ export function applyCurve(v: number, c: number): number { const e = 0.25 + c * 1.75; @@ -201,7 +287,10 @@ export function applyCurve(v: number, c: number): number { * fixed → p.val (held static) * live → engine output[i], shaped by min/max/curve * - * The engine output is 126-dim; a mode with N params uses the first N. + * The engine output vector is now per-mode-sized (the net is reshaped to the + * mode's schema `ml.output_size` on switch), so a mode with N params maps 1:1 + * onto its own N outputs; the `i < engineOut.length` guard keeps it safe during + * the async reshape window. */ export function shapeValues(params: MFParam[], engineOut: Float32Array | null): number[] { return params.map((p, i) => { @@ -229,9 +318,10 @@ export function seededGradient(rev: number): { return { norms, status }; } -/** Map a mode's `input` kind → the engine backend id to drive audio. */ +/** Map a mode id → the audio-engine backend id. Mode ids align with engine ids + * except `slp_workshop` (runs the memlcelium engine), the analysis controller, + * and the relabelled `c15`. */ export function modeEngineId(modeId: string): string { - // Mode ids align with engine ids except the relabelled `c15`. switch (modeId) { case 'paf_synth': case 'channel_strip': @@ -239,7 +329,10 @@ export function modeEngineId(modeId: string): string { case 'elysiamorf': case 'memlcelium': case 'breakor': + case 'xiasri': return modeId; + case 'slp_workshop': + return 'memlcelium'; case 'sound_analysis_midi': return 'analysis'; default: diff --git a/manifold/tests/e2e/geo-dislike.spec.ts b/manifold/tests/e2e/geo-dislike.spec.ts index 07835fa..f96da48 100644 --- a/manifold/tests/e2e/geo-dislike.spec.ts +++ b/manifold/tests/e2e/geo-dislike.spec.ts @@ -10,9 +10,10 @@ */ import { test, expect } from '@playwright/test'; import { loadProbe, settleInputs, countChanged, allWithin } from './helpers'; +import { PafSynthSchema } from '../../src/modes/generated'; -// Fixed by the WASM build (`nisps/wasm/bindings.cpp`: MLP<32,10,14,18,126>). -const N_OUTPUTS = 126; +// The boot mode is paf_synth; the heard vector must match its output arity. +const N_OUTPUTS = PafSynthSchema.ml.output_size; // 33 // A heard vector deliberately distinct from any plausible net output. const HEARD = new Array(N_OUTPUTS).fill(0.9); diff --git a/manifold/tests/e2e/helpers.ts b/manifold/tests/e2e/helpers.ts index 1227415..42e3d8f 100644 --- a/manifold/tests/e2e/helpers.ts +++ b/manifold/tests/e2e/helpers.ts @@ -98,4 +98,21 @@ export function allWithin(xs: number[], lo = 0, hi = 1, tol = 1e-6): boolean { return true; } +/** + * The MLP weight count implied by an ml config — sum over consecutive layers of + * `(prev + 1) * next` (the +1 is the per-layer bias), matching nisps' + * `weight_count()`. Layers = `[input_size, ...hidden_layers, output_size]`. + * E.g. the default 32→[10,14,18]→126 head = 3148; paf_synth 4→[10,10,14]→33 = 809. + */ +export function weightCountFromMl(ml: { + readonly input_size: number; + readonly hidden_layers: readonly number[]; + readonly output_size: number; +}): number { + const layers = [ml.input_size, ...ml.hidden_layers, ml.output_size]; + let total = 0; + for (let i = 0; i < layers.length - 1; i++) total += (layers[i]! + 1) * layers[i + 1]!; + return total; +} + export type Probe = DebugProbe; diff --git a/manifold/tests/e2e/probe-api.spec.ts b/manifold/tests/e2e/probe-api.spec.ts index e050030..fc00b8a 100644 --- a/manifold/tests/e2e/probe-api.spec.ts +++ b/manifold/tests/e2e/probe-api.spec.ts @@ -6,8 +6,9 @@ * every probe accessor must return the documented shape and never throw. * * Adaptations vs. the playground original: - * - Manifold's WASM net is `MLP<32,10,14,18,126>` (playground was `<2,...>`), - * so `getWeights()` has 3148 elements, not 2848 (derivation below). + * - Manifold's net is runtime-shaped and (since P5.3) boots at the BOOT MODE's + * schema `ml` config, so dims + weight count are derived from the imported + * schema, never hard-coded. * - No `probe.__init()` / no `mlStore.iml` poke-through: Manifold's probe * exposes `addExample()` and `routedOutputs()` directly, so the training * tests drive the real public surface instead of an escape hatch. @@ -16,15 +17,13 @@ * genuine failure, not a pending-stream skip. */ import { test, expect } from '@playwright/test'; -import { loadProbe, getOutputs, countChanged, allWithin } from './helpers'; +import { loadProbe, getOutputs, countChanged, allWithin, weightCountFromMl } from './helpers'; +import { PafSynthSchema } from '../../src/modes/generated'; -// Fixed by the WASM build (`nisps/wasm/bindings.cpp`: MLP<32,10,14,18,126>). -const N_OUTPUTS = 126; -// weight_count = 32*10 + 10*14 + 14*18 + 18*126 (weights) -// + 10 + 14 + 18 + 126 (biases) -// = 320 + 140 + 252 + 2268 + 168 = 3148 -const WEIGHT_COUNT = 3148; -// DefaultMLP::kNumLayers (4) * 4 stats per layer. +// The boot mode is paf_synth; all dims derive from its schema `ml` config. +const N_OUTPUTS = PafSynthSchema.ml.output_size; // 33 +const WEIGHT_COUNT = weightCountFromMl(PafSynthSchema.ml); // 4→[10,10,14]→33 = 809 +// 4 layers (3 hidden + output) * 4 stats per layer. const LAYER_STATS = 16; const EXAMPLE_LOW = { input: [0.1, 0.9], output: new Array(N_OUTPUTS).fill(0.1) }; diff --git a/manifold/tests/e2e/reshape.spec.ts b/manifold/tests/e2e/reshape.spec.ts index b9bddb9..3cd0f61 100644 --- a/manifold/tests/e2e/reshape.spec.ts +++ b/manifold/tests/e2e/reshape.spec.ts @@ -1,76 +1,84 @@ /** - * Runtime-shaped net reshape (one-core-engine P2.3). + * Runtime-shaped net reshape (one-core-engine P2.3 + P5.3). * - * The WASM MLP is runtime-shaped: it boots at the default over-provisioned - * 32→[10,14,18]→126 head, and `engine.reshape({ inputSize })` swaps in a new net - * at the requested arity, warm-started from the overlapping weights. This spec - * drives the reshape through the debug probe (`__nisps.reshape`) and asserts: + * The WASM MLP is runtime-shaped. Since P5.3 the app reshapes the net to the + * BOOT MODE's schema `ml` config once WASM is ready, so the debug harness boots + * at the boot mode's dims (NOT the over-provisioned 32→126 default). This spec + * asserts FROM the imported schema — never hard-coded dim numbers — that: * - * 1. default dims are 32 inputs / 126 outputs (unchanged by P2.3); - * 2. reshape to 4 inputs succeeds and `describe()` reports 4/126; - * 3. post-ML outputs stay bounded in [0,1] after the reshape; - * 4. getWeights length shrinks by (32-4)*10 = 280 → 2868 (biases unchanged); - * 5. the spine still propagates — distinct inputs → distinct bounded outputs. + * 1. the net boots at the boot mode's schema dims + weight count; + * 2. `engine.reshape(n)` to a DIFFERENT arity succeeds and `describe()` reports + * the new input size while keeping the schema output size; + * 3. getWeights length tracks the new arity; + * 4. post-ML outputs stay bounded in [0,1] and the spine still propagates. */ import { test, expect } from '@playwright/test'; -import { loadProbe, getOutputs, settleInputs, countChanged, allWithin } from './helpers'; +import { loadProbe, getOutputs, settleInputs, countChanged, allWithin, weightCountFromMl } from './helpers'; +import { PafSynthSchema } from '../../src/modes/generated'; -// Default: 32*10 + 10*14 + 14*18 + 18*126 weights + (10+14+18+126) biases = 3148. -const DEFAULT_WEIGHT_COUNT = 3148; -// Reshaping to 4 inputs only shrinks the first layer: 3148 - (32-4)*10 = 2868. -const RESHAPED_WEIGHT_COUNT = 2868; +// The boot mode (ConsoleApp `modeId` initial state) is paf_synth. +const BOOT = PafSynthSchema.ml; +const BOOT_INPUT = BOOT.input_size; // 4 +const BOOT_OUTPUT = BOOT.output_size; // 33 +const BOOT_WEIGHTS = weightCountFromMl(BOOT); // 4→[10,10,14]→33 = 809 + +// A reshape target arity guaranteed to differ from the boot arity. +const OTHER_INPUT = BOOT_INPUT + 6; // 10 +// Reshaping only the input arity shifts the first layer: (Δin)*hidden0. +const OTHER_WEIGHTS = BOOT_WEIGHTS + (OTHER_INPUT - BOOT_INPUT) * BOOT.hidden_layers[0]!; test.beforeEach(async ({ page }) => { await loadProbe(page); }); test.describe('reshape — runtime-shaped MLP', () => { - test('boots at the default 32 / 126 shape', async ({ page }) => { + test('boots at the boot mode schema shape', async ({ page }) => { const arch = await page.evaluate(() => window.__nisps!.describe()); - expect(arch.inputSize).toBe(32); - expect(arch.outputSize).toBe(126); + expect(arch.inputSize).toBe(BOOT_INPUT); + expect(arch.outputSize).toBe(BOOT_OUTPUT); + expect(arch.hidden).toEqual([...BOOT.hidden_layers]); const len = await page.evaluate(() => window.__nisps!.getWeights().length); - expect(len).toBe(DEFAULT_WEIGHT_COUNT); + expect(len).toBe(BOOT_WEIGHTS); }); - test('reshape to 4 inputs succeeds and describe reports 4 / 126', async ({ page }) => { - const result = await page.evaluate(() => window.__nisps!.reshape(4)); + test('reshape to a new arity succeeds and describe reports it', async ({ page }) => { + const result = await page.evaluate((n) => window.__nisps!.reshape(n), OTHER_INPUT); expect(result).not.toBeNull(); - expect(result!.inputSize).toBe(4); - expect(result!.outputSize).toBe(126); + expect(result!.inputSize).toBe(OTHER_INPUT); + expect(result!.outputSize).toBe(BOOT_OUTPUT); const arch = await page.evaluate(() => window.__nisps!.describe()); - expect(arch.inputSize).toBe(4); - expect(arch.outputSize).toBe(126); + expect(arch.inputSize).toBe(OTHER_INPUT); + expect(arch.outputSize).toBe(BOOT_OUTPUT); }); test('getWeights length changes with the new arity', async ({ page }) => { const before = await page.evaluate(() => window.__nisps!.getWeights().length); - expect(before).toBe(DEFAULT_WEIGHT_COUNT); + expect(before).toBe(BOOT_WEIGHTS); - await page.evaluate(() => window.__nisps!.reshape(4)); + await page.evaluate((n) => window.__nisps!.reshape(n), OTHER_INPUT); const after = await page.evaluate(() => window.__nisps!.getWeights().length); - expect(after).toBe(RESHAPED_WEIGHT_COUNT); + expect(after).toBe(OTHER_WEIGHTS); expect(after).not.toBe(before); }); test('outputs stay bounded after reshape', async ({ page }) => { - await page.evaluate(() => window.__nisps!.reshape(4)); + await page.evaluate((n) => window.__nisps!.reshape(n), OTHER_INPUT); await page.evaluate(() => window.__nisps!.setInputs(0.3, 0.7)); const outs = await getOutputs(page); - expect(outs.length).toBe(126); + expect(outs.length).toBe(BOOT_OUTPUT); expect(allWithin(outs, 0, 1)).toBe(true); }); test('spine still propagates after reshape', async ({ page }) => { - await page.evaluate(() => window.__nisps!.reshape(4)); + await page.evaluate((n) => window.__nisps!.reshape(n), OTHER_INPUT); const a = await settleInputs(page, 0.2, 0.8); const b = await settleInputs(page, 0.9, 0.1); - expect(a.length).toBe(126); - expect(b.length).toBe(126); + expect(a.length).toBe(BOOT_OUTPUT); + expect(b.length).toBe(BOOT_OUTPUT); expect(allWithin(a, 0, 1)).toBe(true); expect(allWithin(b, 0, 1)).toBe(true); // Distinct inputs must still move the mapping through the reshaped net. diff --git a/manifold/tests/e2e/schema-modes.spec.ts b/manifold/tests/e2e/schema-modes.spec.ts new file mode 100644 index 0000000..7ca7fdf --- /dev/null +++ b/manifold/tests/e2e/schema-modes.spec.ts @@ -0,0 +1,117 @@ +/** + * Schema-driven per-mode dims — the one-core-engine P5 gate. + * + * The generated mode schemas (`src/modes/generated/`) are the SOURCE OF TRUTH: + * `MF_MODES` derives its params from them (P5.2) and the engine reshapes to each + * mode's `ml` config on switch (P5.3). This spec imports the schemas DIRECTLY + * and, for a representative set of modes, drives a mode switch through the UI + * debug seam (`window.__mf`), then asserts against the schema — never hard-coded + * numbers — that: + * + * - `describe()` reports the schema's exact input/hidden/output dims; + * - `getWeights().length` equals the schema-implied weight count; + * - post-ML outputs have length == output_size and stay bounded in [0,1]; + * - the rendered UI param count equals `schema.params.length`; + * - training still works after a mode switch (per-mode dims flow through the + * async training worker — one-core-engine P2.2/P2.3 buffer sizing). + */ +import { test, expect } from '@playwright/test'; +import type { Page } from '@playwright/test'; +import { loadProbe, getOutputs, allWithin, weightCountFromMl } from './helpers'; +import type { MfDebugHook } from '../../src/console/ConsoleApp'; +import type { ModeSchema } from '../../src/modes/generated/types'; +import { + PafSynthSchema, + ChannelStripSchema, + MemlceliumSchema, + XiasriSchema, +} from '../../src/modes/generated'; + +declare global { + interface Window { + __mf?: MfDebugHook; + } +} + +// Representative modes: an xy synth (33 out), a joystick synth (24 out), a +// sequencer (56 out), and a browser-viable NEW mode (xiasri, 24 out) that never +// had a hand-written catalogue entry. +const CASES: ReadonlyArray = [ + PafSynthSchema, + ChannelStripSchema, + MemlceliumSchema, + XiasriSchema, +]; + +/** + * Switch the instrument mode via the debug seam and wait until the switch has + * fully landed: the modeId, the rendered param count, and the reshaped net dims + * must all reflect the target schema before we assert. + */ +async function switchToMode(page: Page, schema: ModeSchema): Promise { + await page.evaluate((id) => window.__mf!.setMode(id), schema.mode_id); + await page.waitForFunction( + (s) => + window.__mf?.getModeId() === s.id && + window.__mf?.paramCount() === s.params && + window.__nisps?.describe().outputSize === s.out && + window.__nisps?.describe().inputSize === s.in, + { id: schema.mode_id, params: schema.params.length, out: schema.ml.output_size, in: schema.ml.input_size }, + { timeout: 10_000 }, + ); +} + +test.beforeEach(async ({ page }) => { + await loadProbe(page); +}); + +test.describe('schema-driven per-mode dims (P5 gate)', () => { + test('the debug seam exposes every catalogue mode id', async ({ page }) => { + const ids = await page.evaluate(() => window.__mf!.modeIds()); + for (const schema of CASES) expect(ids).toContain(schema.mode_id); + }); + + for (const schema of CASES) { + test(`${schema.mode_id}: engine + UI match the schema`, async ({ page }) => { + await switchToMode(page, schema); + + // describe() reports the schema's exact dims. + const arch = await page.evaluate(() => window.__nisps!.describe()); + expect(arch.inputSize).toBe(schema.ml.input_size); + expect(arch.outputSize).toBe(schema.ml.output_size); + expect(arch.hidden).toEqual([...schema.ml.hidden_layers]); + + // getWeights length equals the schema-implied weight count. + const weights = await page.evaluate(() => window.__nisps!.getWeights().length); + expect(weights).toBe(weightCountFromMl(schema.ml)); + + // Outputs have length == output_size and stay bounded. + await page.evaluate(() => window.__nisps!.setInputs(0.35, 0.65)); + const outs = await getOutputs(page); + expect(outs).toHaveLength(schema.ml.output_size); + expect(allWithin(outs, 0, 1)).toBe(true); + + // The rendered UI param count equals schema.params.length. + const paramCount = await page.evaluate(() => window.__mf!.paramCount()); + expect(paramCount).toBe(schema.params.length); + }); + } + + test('training works after a mode switch (per-mode dims flow to the worker)', async ({ page }) => { + // Switch to a mode with distinct dims from the boot mode, then add a couple + // of contrasting examples at the mode's output arity and train. A finite, + // non-negative loss proves the async worker re-created its mirror net at the + // reshaped dims (buffer sizing did not assume a fixed 126). + await switchToMode(page, MemlceliumSchema); + const outSize = MemlceliumSchema.ml.output_size; + const loss = await page.evaluate(async (n) => { + const p = window.__nisps!; + p.addExample([0.1, 0.9], new Array(n).fill(0.1)); + p.addExample([0.9, 0.1], new Array(n).fill(0.9)); + return p.trainAsync(); + }, outSize); + expect(typeof loss).toBe('number'); + expect(Number.isFinite(loss)).toBe(true); + expect(loss).toBeGreaterThanOrEqual(0); + }); +});