w1n5t0n/memlnaut-nisps
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
memlnaut-nisps/docs/specs/manifold-parity-features-spec.md

24 KiB
Raw Blame History

kind stability layer
spec aspirational behavioural

Manifold Parity Features — Session Presets · Pins · Jolt · OU-Explore · Control Surface

Status: draft — awaiting review. Date: 2026-07-12. Scope: prescriptive spec for porting five playground features into manifold/. Spec only — no implementation is authorised by this document; break into ergo issues after sign-off.

Traces up to: playground-2.0-rewrite-plan.md §2.3 (Feel drawer), §2.6 (pins), §3.8 (composed-layers presets); engine-architecture.md §3.2 (fanout, control-point tri-state), §3.8 (persist helper); docs/specs/slp-workshop-firmware.md §3–4 (Jolt / OU-explore, Part I shipped in firmware); dock-spec.md (drawer depths, tri-state semantics); BUILD-PLAN locked decisions (parity-tested engine; British spelling; the synth is "Powerful Synth Engine", never the forbidden string).

Reference implementations: the playground versions are the behavioural ground truth for constants and algorithms — playground/src/features/session-preset.ts, stores/session-store.ts (pins), ml/jolt.ts, output/ou-explore.ts, stores/control-store.ts + features/control-routing.ts. Where this spec and the playground disagree, this spec wins (each divergence is called out and justified inline).

0. Principles applied throughout

  1. C++ owns gesture math where a C++ class exists. nisps::ml::Jolt and nisps::ml::OUNoise<N> already exist (nisps/ml/jolt.hpp, nisps/ml/ou_noise.hpp) but are not exposed to WASM. Manifold binds them rather than re-porting to TS — this closes two documented --- C++ GAP --- items and buys deterministic, firmware-parity noise for free. (The playground's TS reimplementations used Math.random(); that shortcut is not carried over.)
  2. No new mechanisms where an existing one already expresses the idea. Param pins are the existing off|fixed|live tri-state + arm mask, not a parallel pin system (§2.2).
  3. Transparent defaults. Every default constant appears in this spec as a number with its source, and the control surface UI must show the derived per-param values live (§5.6) — the user should never wonder what an axis is secretly doing.
  4. One persistence pattern. All new persisted state goes through a single versioned persist<T>(key, version, migrate) helper modelled on settings-store.ts (localStorage, debounced 200 ms), per engine-architecture.md §3.8.
  5. Probe parity. Each feature activates its currently-inert window.__nisps methods (manifold/src/debug/probe.ts) so Playwright can drive it headlessly. New e2e specs are part of each feature's acceptance criteria.

1. Session presets (composed layers)

1.1 Model

Adopt the composed-layers model prescribed by engine-architecture.md §3.8 rather than the playground's monolithic SessionPresetPayload. A preset is a bundle of independently optional layers:

interface SessionPresetV1 {
  v: 1;
  id: string;                    // `preset-${epoch36}`
  name: string;
  createdAt: number;
  layers: {
    control?: ControlLayer;      // axes + offsets + presetId          (§5)
    pipelines?: PipelineLayer;   // input/output pipeline params
    routing?: RoutingLayer;      // per-output MFParam state/min/max/curve/mute + backend specs
    mode?: ModeLayer;            // modeId + outputMode
    weights?: string;            // base64url Float32Array — OPT-IN only
  };
}
  • ControlLayer = { boldness, memory, precision, presetId, offsets } exactly as §5.4 state.
  • PipelineLayer = the configurable params of engine/input-pipeline.ts and output-pipeline.ts (deadzone, zoom, invert, curve, smoothing, momentum; global curve, smoothing, slew, freeze).
  • RoutingLayer = serialised MFParam[] control fields + the per-backend specs already handled by the OutputsBackendConfig preset bar. The existing backend preset bar becomes a view onto the routing layer — one storage model, two entry points (see §1.4).
  • weights uses the playground's base64url Float32Array codec verbatim (session-preset.ts:63–98): standard base64 with +/ → -_, padding stripped; byte length rounded down to a 4-byte boundary on decode; size-mismatch on restore is silently skipped.

Restore is best-effort and layer-wise: absent layers leave current state untouched; unknown keys inside a layer are ignored (forward compatibility).

1.2 Persistence

  • Key: mf-session, via the shared persist<T> helper, version field v: 1, 200 ms debounce.
  • Stored shape: { v: 1, presets: SessionPresetV1[], regionPins: RegionPin[] } (pins ride in the same document, §2).
  • Weights are excluded by default on save; the save UI offers an explicit "include network weights" toggle (default off). Rationale: weight blobs are ~KB-scale × N presets and stale weights surprise users; matches playground withWeights=false default.

1.3 URL sharing

Carry over both playground URL forms (session-preset.ts:249–283):

  • Compact axis form — ?boldness=0.5&memory=0.5&precision=0.3: applies only the control layer; each value clamped to [0,1].
  • Full form — ?session=<base64url(JSON of SessionPresetV1.layers)>: applied on first load, after engine ready, before first render of the dock. Never include weights in a generated share URL (URL length); the share button produces the full form minus weights.

URL params are read once at boot in App.tsx (same place the ?debug=1 probe is installed) and then stripped from the address bar via history.replaceState so a reload doesn't re-apply over user changes.

1.4 UI

  • Save/restore/rename/delete lives in a preset bar at the top of the route drawer's expanded depth, reusing the exact interaction pattern of dock/OutputsBackendConfig.tsx (save-as / restore / rename / delete). The existing backend-scoped preset bar remains; a session preset is the superset bundle.
  • Share button beside the bar copies the URL to the clipboard and flashes confirmation.
  • Restoring a preset that contains a mode layer while a different mode is active shows the same confirm modal used for net-reset (input-layer reshape) — mode switches discard training state and must not be silent.

1.5 Probe + acceptance

  • Probe: saveSessionPreset(name, withWeights?), loadSessionPreset(id), listSessionPresets(), buildShareUrl(), applySession(json).
  • e2e: save → mutate axes/routing → restore → assert state round-trips; share URL → fresh page load with ?session= → assert layers applied; weights round-trip when opted in (getWeights() equality within exact bits).

2. Region & param pins

2.1 Region pins

Type (unchanged from playground session-store.ts:34–44):

interface RegionPin {
  id: string;          // `pin-${epoch36}-${rand}`
  x: number; y: number;          // bottom-left corner, [0,1] input space
  width: number; height: number; // [0,1]
  colourSlot: number;  // 0..4 → --pin-1..--pin-5 tokens
  createdAt: number;
}
  • Cap: 5 pins, FIFO (oldest evicted). Colour slot defaults to pins.length % 5.
  • Colour tokens join manifold/src/styles/tokens/colors.css with the playground values: fills --pin-1..5 at 0.25 alpha (orange 255,106,0 / cyan 0,204,255 / purple 180,100,255 / green 80,200,120 / tan 255,200,80), borders same hues at 0.7 alpha, 2 px logical border width.
  • Creation gesture: long-press (600 ms — the constant Manifold.tsx already uses) on the Manifold stage pins the currently visible region (whole surface when unzoomed; the zoom window once input zoom exists in manifold — until then, {x:0,y:0,w:1,h:1} is not useful, so the long-press instead opens a small radial affordance: "pin region around cursor" creating a 0.25×0.25 region centred on the press point, clamped to [0,1]²). Divergence from playground (which pins the zoom window) because manifold has no input zoom yet; revisit when the inputs workstream lands zoom.
  • Removal: tap a pin border → small popover with colour dot + "remove"; "clear pins" action in the settings drawer.
  • Rendering: Manifold.tsx already accepts pins: Pin[] — extend the Pin type in console/types.ts to the full RegionPin shape (it currently has only x/y/colour), and render rects behind trail/markers, in front of the grid. Long-press must not fire when the press initiates a drag > 8 px (existing drag threshold logic) or while picking is true (explore-and-place anchor picking takes priority).
  • Persistence: in the mf-session document (§1.2).
  • Semantics: purely annotational in v1 (visual memory of "good areas"). They do not constrain training. A future spec may bind snapshot anchors to pins.

2.2 Param pins = the existing tri-state (decision, not new code)

The playground's separate ParamPin list + paramPinMask() is not ported. Manifold already has a strictly more expressive mechanism, per dock-spec.md §3.3:

  • off — excluded from model control and pinned out of training;
  • fixed — held at a static value, pinned out of training;
  • live — model-driven, trainable.

"Pinning a param" in manifold = setting it to fixed (hold current value) via the existing OutputControlRow segmented control or by clicking the value readout in OutputStage.

Gap this spec closes: the tri-state must actually feed the training mask. Requirement: the arm/focus mask sent through _nisps_ml_feedback_set_focus (and any moveWeights-style perturbation mask) is computed as armed ∧ (state === 'live') — off/fixed columns are never perturbed and never accumulate gradient. Owner: dock/output-state.ts buildArmMask().

Probe: pinParam(i) / unpinParam(i) map to setParam(i, {state:'fixed'|'live'}); the inert playground-compat methods in probe.ts are rewired to these.

Acceptance: e2e asserts (a) a fixed output's value is bit-stable across training and jolt; (b) region pins survive reload; (c) 6th region pin evicts the 1st.

3. Jolt

3.1 What it is

A held-gesture weight morph: while held, N randomly-chosen weights each glide (EMA) toward their own random target, re-rolling the target on arrival — a continuous, directional "stirring" of the network. On release the net freezes where it landed, and the learning rate ramps back up from 0 so training doesn't immediately fight the new position. Ported from upstream InterfaceRL; C++ implementation shipped in firmware (SLP workshop Part I).

3.2 Binding, not porting

Expose the existing nisps::ml::Jolt through the C ABI. New bindings in nisps/wasm/bindings.cpp + scripts/build-wasm.sh EXPORTED_FUNCS:

_nisps_ml_jolt_press(ml)          // picks indices+targets from the MLP's weight count
_nisps_ml_jolt_step(ml)           // one tick: mutates weights in place, then caller reprocesses
_nisps_ml_jolt_release(ml)
_nisps_ml_jolt_active(ml) -> i32
_nisps_ml_jolt_lr_scale(ml) -> f32   // 0 while held; ramps 0→1 after release
_nisps_ml_jolt_tick_lr_ramp(ml)
_nisps_ml_jolt_seed(ml, u64)

The Jolt instance lives beside the MLP handle inside the WASM module (one per net), seeded from the session seed so runs are reproducible. Parity CI: extend tests/cpp/parity_check.cpp with a golden jolt sequence (seed → press → k steps → weight vector) asserted native-vs-WASM.

3.3 Constants (defaults — from nisps/ml/jolt.hpp:42–49, upstream-verbatim)

Constant Value Meaning
num_weights 40 weights morphed simultaneously (hard cap 64)
morph_rate 0.017 EMA per tick: w += 0.017·(target − w)
target_min / max −1.2 / 0.9 random target range
target_epsilon 0.05 re-roll target when within this distance
lr_ramp_step 0.001 post-release LR ramp per tick (~5 s at 200 Hz)
tick period 5 ms (200 Hz) JOLT_TICK_MS, matches firmware control rate

3.4 Manifold integration

  • Driver: a setInterval(5 ms) in FeedbackController while active: jolt_step → engine.reprocess() (the spine already re-ticks the last input after weight changes) → bump version. LR suppression: training calls multiply their LR by jolt_lr_scale(); the ramp is ticked from the same driver until it reaches 1.0.
  • Snapshot before press: entering jolt pushes an auto-snapshot (tag before jolt) so undo recovers the pre-jolt net — same behaviour as playground mode-runtime.ts:551.
  • UI: a press-and-hold ⚡ Jolt button in the learn drawer (condensed depth, beside the VerdictCluster-adjacent controls) and keyboard hold J. Visual: the Manifold noise ring pulses while active. Pointer-cancel/blur must release (never a stuck jolt).
  • Interlocks: disabled while exploring (explore-and-place owns the net during a scratchpad session); off/fixed outputs are unaffected by definition (§2.2 mask applies only to training — jolt perturbs weights, so v1 explicitly documents that jolt ignores per-output pins; a per-column jolt mask is out of scope).

3.5 Probe + acceptance

Probe: joltPress(), joltRelease(), joltActive(). e2e: press → 200 ms → assert outputs changed and getWeights() differs; release → assert weights stable; undo restores pre-jolt weights; determinism: same seed + same step count ⇒ identical weight vector twice.

4. OU-explore noise

4.1 What it is

Ornstein–Uhlenbeck drift added to the output vector (not the weights): each output channel carries a state that takes small random steps while being pulled back toward zero, producing smooth, temporally-correlated wander around whatever the net currently outputs. Learning stays live throughout — it is an audition aid, not a training mode. C++ OUNoise<N> shipped in firmware (SLP workshop Part I).

Discrete update per channel (Euler–Maruyama, μ = 0):

state += theta · (−state) · dt + noise_scale · N(0,1)
out    = clamp(out + state, 0, 1)
noise_scale = stationary_std · √(2·theta·dt),   stationary_std = intensity · 0.65

4.2 Binding

Instantiate nisps::ml::OUNoise<126> (the WASM output width) beside the MLP handle. Bindings:

_nisps_ml_ou_set_intensity(ml, f32)   // [0,1] → stationary_std = v·0.65; 0 disables
_nisps_ml_ou_intensity(ml) -> f32
_nisps_ml_ou_apply(ml, out_ptr, n)    // advance state + add + clamp, in place
_nisps_ml_ou_reset(ml)
_nisps_ml_ou_set_theta(ml, f32)  _nisps_ml_ou_set_dt(ml, f32)
_nisps_ml_ou_seed(ml, u64)

4.3 Constants (defaults — nisps/ml/ou_noise.hpp, upstream-verbatim)

Constant Value
theta (mean reversion) 0.02
dt 0.001
kOUMaxAmplitude (full-scale stationary std) 0.65
default intensity 0 (inert — apply() is a no-op, preserving current behaviour/parity)
idle tick period 30 ms (EXPLORE_TICK_MS)

4.4 Manifold integration

  • Where in the chain: in the spine, after the TS output-pipeline (curve→smoothing→slew→ freeze) and before the backend send — i.e. ou_apply on the routed buffer. This matches the playground ordering (post-pipeline, pre-overrides). fixed/off outputs are re-asserted after ou_apply so pinned values do not wander (this is the §2.2 guarantee).
  • Idle ticking: when intensity > 0 and no input events arrive, a 30 ms interval calls reprocess() so the wander is audible with a static joystick. Interval torn down and ou_reset() called when intensity returns to 0.
  • UI: an Explore slider (0–1) in the learn drawer condensed depth, next to the existing Noise/Spread knobs; the Manifold noise ring radius reflects noiseCap + ou_intensity·0.65 so the visual language stays truthful.
  • Interaction with Jolt: independent and composable (jolt stirs weights, OU wanders outputs); both default inert.

4.5 Probe + acceptance

Probe: setExploreIntensity(v), getExploreIntensity(). e2e: intensity 0.5 + static input ⇒ outputs vary over 500 ms with bounded step size; intensity 0 ⇒ outputs bit-stable; fixed output does not wander; determinism under seeded runs.

5. Boldness / Memory / Precision control surface

5.1 Concept

Three compound axes, each a [0,1] fader that drives ~5–6 underlying parameters through a piecewise-linear lookup table, with per-param trim-pot offsets on top. This is the "Feel" surface from playground-2.0-rewrite-plan.md §2.3, ported with the playground's tables as the transparent defaults — reproduced in full below so the mapping is reviewable here, not buried in code.

5.2 Axis tables (defaults — playground/src/stores/control-store.ts:38–54, verbatim)

Numeric params interpolate linearly between breakpoints; discrete params snap to the upper row at t ≥ 0.75.

Boldness — "how hard do moves hit"

axis zoom noiseCap noiseGrowth learningRate weightDecay noiseDistribution
0.0 0.1 0.02 1.1 0.1 0.15 gaussian
0.5 0.5 0.12 1.5 1.0 0.06 gaussian
1.0 1.0 0.30 2.5 3.0 0.00 cauchy

Memory — "how much does the net remember"

axis maxExamples exampleDecay memoryWeightDecay noiseDecay convergenceThreshold
0.0 5 0.3 0.20 0.85 1e-3
0.5 50 0.7 0.06 0.97 1e-5
1.0 500 1.0 0.00 0.995 1e-8

Precision — "how surgical is the gesture"

axis inputCurve deadzone smoothing slewRate momentumZoom
0.0 1.0 0.00 0.00 1.0 off
0.5 1.5 0.05 0.15 0.3 off
1.0 3.0 0.15 0.40 0.1 off

Routing targets in manifold: zoom/deadzone/inputCurve/smoothing/momentumZoom → engine/input-pipeline config; slewRate → engine/output-pipeline; noiseCap/noiseGrowth/ noiseDecay/learningRate/weightDecay → FeedbackController + training params; maxExamples/exampleDecay/convergenceThreshold → engine/dataset.ts + training loop. noiseDistribution and momentumZoom are carried in the schema but may no-op until their consumers exist in manifold (each no-op must be logged once at boot — no silent dead params). spread is deliberately not axis-driven (stays an independent lab knob, as in playground).

5.3 Presets (defaults — verbatim)

id label boldness memory precision
default Default 0.5 0.5 0.3
first-touch First Touch 0.2 0.7 0.6
jazz-hands Jazz Hands 0.8 0.2 0.0
sculptor Sculptor 0.3 0.9 0.8
improviser Improviser 0.6 0.3 0.2
microscope Microscope 0.1 1.0 1.0

Boot default = the default preset (0.5 / 0.5 / 0.3). This replaces ConsoleApp's current ad-hoc {0.55, 0.4, 0.5} initial state. Moving any axis manually clears presetId (chips show "custom").

5.4 State + trim-pot offsets

interface ControlState {
  boldness: number; memory: number; precision: number;   // [0,1]
  offsets: Record<'boldness'|'memory'|'precision', Record<string, number>>;
  presetId: string | null;
}

Resolution: interpolate all three tables, merge, then add each axis's offsets to numeric params (resolved[k] = table[k] + offset[k]), clamp to each param's legal range. Offsets are created when the user adjusts a derived param directly (§5.6) — the delta from the current table value becomes the offset ("trim-pot"). Double-tap on a derived param's readout clears its offset ("re-link"); a per-axis "re-link all" clears the axis's offset map. A dot marks any param with a non-zero offset.

5.5 Fanout (architecture requirement)

Per engine-architecture.md §3.2: resolution is a memoised derivation per target param — in React terms one useMemo producing the resolved record + an effect per target store that writes only on change. The playground's JSON.stringify-signature-inside-effect pattern (control-routing.ts:34) is explicitly not ported. Resolution runs off the render cycle for engine targets (direct setter calls on the engine config objects), with React state only for what the UI displays.

5.6 UI — transparency is the feature

  • Placement: the learn drawer gains a top "Feel" section (this keeps manifold at five drawers; if the section crowds the condensed depth, promoting Feel to its own drawer is a pre-approved fallback — note it in the implementation PR).
    • Condensed: three vertical ControlAxis faders (the existing primitive, finally wired) + six preset chips.
    • Expanded: under each fader, the live list of its derived params — name, resolved value (table + offset), and a mini trim-pot. This satisfies the "reasonable and transparent defaults" requirement: the mapping is always visible in the UI, not just in this document.
  • Axis moves are continuous (no apply button); param writes are rate-limited to one per frame.
  • The existing preset = 'Sculpt' placeholder state in ConsoleApp is removed in favour of presetId.

5.7 Persistence, probe, acceptance

  • Persisted as the control slice of mf-session (also the control layer of session presets, §1 — one shape, two containers).
  • Probe: setAxis(name, v), getAxes(), resolveControlParams(), applyControlPreset(id), setAxisOffset(axis, param, v).
  • e2e: boldness 0→1 sweeps noiseCap 0.02→0.30 and zoom 0.1→1.0 (assert via resolveControlParams() + engine config readback); preset chip sets all three axes; manual axis move clears presetId; offset survives axis movement (offset is additive, not absolute); double-tap re-links; state round-trips through reload and through a session preset.

6. Cross-cutting

6.1 Build & sequencing (proposed ergo breakdown, post-sign-off)

  1. W1 — WASM bindings: jolt + OU C ABI, EXPORTED_FUNCS, parity golden tests. (Blocks W4, W5.)
  2. W2 — persist<T> helper + mf-session document (control slice, pins, presets scaffold).
  3. W3 — Control surface: state, tables, fanout, Feel section, presets, offsets. (Needs W2.)
  4. W4 — Jolt + OU in FeedbackController + learn drawer UI. (Needs W1.)
  5. W5 — Region pins (tokens, gesture, rendering, popover) + tri-state training-mask guarantee (§2.2). (Needs W2.)
  6. W6 — Session presets: layers, preset bar, URL sharing, probe. (Needs W2, W3.)

Each lands with its e2e spec; smoke suite additions run in the same CI gate as the existing tests/e2e/smoke.spec.ts.

6.2 Out of scope (explicit)

  • Snapshot-with-weights / snapshot DAG, A/B compare (separate spec — the biggest remaining parity gap, but architecturally entangled with the C++ feedback snapshot lifecycle).
  • 2D heatmap sampler, weight health, gradient flow (blocked on loss-history C API plumbing).
  • Input zoom (inputs workstream) — noted where it interacts with region pins (§2.1).
  • Per-column jolt masking (§3.4).

6.3 Open questions for review

  1. §2.1 creation gesture: is the 0.25×0.25 press-centred region an acceptable stand-in until input zoom lands, or should region pins wait for the zoom feature?
  2. §5.6: Feel section inside the learn drawer vs a sixth drawer — reviewer's call if condensed depth gets crowded.
  3. §1.1 RoutingLayer: should backend connection details (OSC URL, MIDI port id) be included in session presets, or excluded as machine-specific? Proposal: exclude port/URL, include per-output mappings.
  4. Memory-axis params (maxExamples, exampleDecay, convergenceThreshold): confirm the manifold training loop actually consumes these; if any are dead in the current engine, they land as logged no-ops (§5.2) with an ergo follow-up.

Doc-sync rule applies: when implementation diverges from this spec, update the spec in the same commit. When a workstream ships and is verified, migrate its section to ONBOARDING.md ("what is") and prune it here.