diff --git a/playground/SPEC-shapeseq.md b/playground/SPEC-shapeseq.md new file mode 100644 index 0000000..bcfd754 --- /dev/null +++ b/playground/SPEC-shapeseq.md @@ -0,0 +1,434 @@ +# ShapeSeq — NISPS Generative Sequencing System + +## Overview + +ShapeSeq is a generative sequencing system for the NISPS playground where interactive ML (via the NISPS MLP engine) controls **parameters of algorithmic sequencing primitives** rather than raw note data. The user shapes sequences by navigating a learned parameter space with a joystick or hand tracking, can freeze sequences they like, then selectively re-expose specific parameters for further ML-driven exploration. + +ShapeSeq replaces the existing placeholder arpeggiator. + +## Core Architecture + +### Design Principles + +1. **MLP outputs are abstract [0,1] values** — musical meaning is applied downstream by a configurable projection layer +2. **Separate NISPS instances** for timbre control and sequence control, with architecture supporting future unification into a single instance +3. **Port-ready JS** — no closures in hot paths, explicit state, data structures that map cleanly to C++ for future RP2040 firmware porting +4. **Modular primitives** — small, combinable algorithmic building blocks that generate musical patterns from continuous parameters + +### System Diagram + +``` + ┌─────────────────┐ + │ Input Router │ + │ (configurable) │ + └──┬──────────┬───┘ + │ │ + ┌────────▼──┐ ┌───▼────────┐ + │ NISPS MLP │ │ NISPS MLP │ + │ (timbre) │ │ (sequence) │ + └────────┬───┘ └───┬────────┘ + │ │ + ┌────────▼──┐ ┌───▼────────────────────┐ + │ Synth │ │ Delta Controller │ + │ Param Map │ │ (frozen vals + deltas) │ + └────────┬───┘ └───┬────────────────────┘ + │ │ + │ ┌────▼──────────────┐ + │ │ Primitive Chain │ + │ │ Euclid→ProbGate→… │ + │ └────┬──────────────┘ + │ │ + │ ┌────▼──────────────┐ + │ │ Projection Layer │ + │ │ (scale quant, etc) │ + │ └────┬──────────────┘ + │ │ + │ ┌────▼──────────┐ + │ │ Clock Engine │ + │ │ (AudioContext) │ + │ └────┬──────────┘ + │ │ + ┌────▼──────────▼────┐ + │ Namespaced Event Bus│ + │ seq.* ml.* ui.* │ + └────────┬───────────┘ + │ + ┌────────▼───────┐ + │ C15 Synth │ + │ (noteOn/Off + │ + │ param changes)│ + └────────────────┘ +``` + +### Namespaced Event Bus + +A pub/sub event system with namespaced channels: + +| Namespace | Events | Purpose | +|-----------|--------|---------| +| `seq.*` | `seq.step`, `seq.noteOn`, `seq.noteOff`, `seq.paramChange`, `seq.loopStart` | Musical output from sequencer to synth and visualizer | +| `ml.*` | `ml.trained`, `ml.frozen`, `ml.unfrozen`, `ml.deltaUpdate` | ML state changes | +| `ui.*` | `ui.paramSelect`, `ui.chainEdit`, `ui.presetLoad`, `ui.freezeToggle` | User actions | + +All events carry a timestamp (AudioContext.currentTime for `seq.*`, performance.now() for others). + +### Input Routing Matrix + +A configurable routing layer that maps any input source to either NISPS instance's inputs: + +**Input sources:** +- Joystick X, Y (2 values) +- Hand tracking features (14 values) +- Gamepad axes (variable) + +**Routing targets:** +- Timbre NISPS input 0, input 1 +- Sequence NISPS input 0, input 1 + +Default: joystick → timbre NISPS, hand tracking features 0+1 → sequence NISPS. User-configurable via UI. + +## Sequencing Primitives + +Each primitive is a pure function (or stateful generator with explicit state) that accepts a parameter object and produces typed output. All parameters are normalized [0,1]. + +### 1. Euclidean Rhythm Generator + +**Params:** `steps` (int, from continuous), `pulses` (int), `rotation` (int) +**Output type:** trigger pattern (boolean array) +**Stateless:** yes + +Generates Bjorklund-distributed trigger patterns. The continuous [0,1] params are projected to integer ranges based on current step count. + +### 2. Probability Gate + +**Params:** `density` [0,1], `accentProbability` [0,1] +**Input type:** trigger pattern +**Output type:** filtered trigger pattern with accent flags +**Stateless:** yes (per-step coin flip using seeded PRNG) + +Each incoming trigger survives with probability `density`. Surviving triggers receive accent flag with probability `accentProbability`. + +### 3. Pitch Walker + +**Params:** `stepSize` [0,1], `directionBias` [0,1] (0.5=unbiased), `gravity` [0,1] (pull toward center), `range` [0,1] +**Input type:** trigger pattern +**Output type:** pitch values [0,1] per triggered step +**Stateful:** yes — maintains current position in pitch space + +Constrained random walk that generates melodic contour. `gravity` pulls the walk toward center (0.5), preventing it from getting stuck at extremes. State includes current position and PRNG state. + +### 4. Ratchet + +**Params:** `maxDivision` [0,1] (maps to 1-4 subdivisions), `probability` [0,1] +**Input type:** trigger pattern +**Output type:** trigger pattern with subdivision timing offsets +**Stateless:** yes (per-step coin flip) + +Subdivides triggered steps into rapid repeats. Division count determined by `maxDivision`, applied probabilistically. + +### 5. Swing / Groove + +**Params:** `swingAmount` [0,1] (0=straight, 1=full swing), `swingGrid` [0,1] (which subdivisions swing) +**Input type:** timing information +**Output type:** modified timing offsets +**Stateless:** yes + +Shifts timing of alternating steps. At `swingAmount=0.67` this produces classic 2:1 shuffle. `swingGrid` controls whether swing applies to 8th notes, 16th notes, or triplets. + +### 6. Density Morph + +**Params:** `density` [0,1], `clustering` [0,1] (0=spread evenly, 1=clustered together) +**Input type:** step count +**Output type:** trigger pattern +**Stateless:** yes + +Alternative to Euclidean — generates trigger patterns with controllable density and spatial distribution. At high clustering, triggers group together creating bursts; at low clustering, triggers spread evenly. + +### 7. Interval Lock (Scale Quantizer) + +**Params:** `root` [0,1] (maps to 0-11 semitones), `mode` [0,1] (maps to scale index), `octaveRange` [0,1] (1-4 octaves) +**Input type:** pitch values [0,1] +**Output type:** MIDI note numbers +**Stateless:** yes + +Available scales: chromatic, major, natural minor, harmonic minor, pentatonic major, pentatonic minor, blues, dorian, mixolydian, whole tone, diminished. + +### 8. Velocity Shaper + +**Params:** `curveType` [0,1] (maps to: flat, accent-every-N, crescendo, decrescendo, random), `depth` [0,1], `phase` [0,1] +**Input type:** trigger pattern with step indices +**Output type:** velocity values [0,1] per step +**Stateless:** yes + +Applies cyclic velocity patterns. `phase` rotates the pattern, `depth` controls contrast between quiet and loud. + +### Primitive Interface + +```javascript +// Port-ready: explicit state, no closures +class Primitive { + constructor(name, paramSchema) { ... } + + // paramSchema: array of { name, min, max, default, boundary } + // boundary: 'clamp' | 'wrap' | 'scaled' + // For 'scaled': operates within ±scaledRange of frozen value + + // Pure processing function + process(params, input, state, stepCount, rng) → { output, nextState } + + // State management for freeze + getState() → serializable object + setState(state) → void + getSeed() → number + setSeed(seed) → void +} +``` + +### Chain Connection Modes + +Three configurable modes for how primitives connect in a chain: + +**1. Sequential Pipeline** — each primitive transforms the previous output. Order matters. Signal flows left to right. + +**2. Parallel + Merge** — each primitive runs independently, outputs merged (OR for triggers, average for continuous values). Order doesn't matter. + +**3. Typed Routing** — primitives connect via typed ports. A primitive's output connects to the next primitive that accepts that type. Multiple primitives can feed the same type (merged). Most flexible, most complex. + +The chain connection mode is a global setting (per-chain), configurable via UI. Default: sequential pipeline. + +## Delta Control System + +### Freeze Workflow + +1. User plays with NISPS, finds a sequence they like +2. User activates **freeze** — all current parameter values are captured +3. User selects specific parameters to **re-expose** (mark as "live"): + - Click/tap parameters in the UI + - Or use hand tracking: point with index finger, pinch gesture to toggle +4. Live parameters receive **deltas** from NISPS MLP output +5. Frozen parameters hold their captured values + +### Freeze Modes + +**Freeze as Algorithm** — captures parameter values + PRNG seed. Stateful primitives (pitch walker) will replay identically. Re-exposing params resumes algorithmic generation with delta-modified params. + +**Freeze as Pattern** — captures the realized note pattern (snapshot of all step events for one full loop). The primitive chain is bypassed; the sequencer loops the frozen pattern directly. Re-exposing params requires switching back to algorithm mode. + +User chooses freeze mode via UI toggle. + +### Delta Boundary Behavior + +Each parameter declares its boundary behavior: + +| Behavior | Description | Good for | +|----------|-------------|----------| +| `clamp` | Delta result clamped to [0,1] | Velocity, volume, most continuous params | +| `wrap` | Values wrap around (1.1 → 0.1) | Rotation, phase, cyclic params | +| `scaled` | Delta operates within ±`scaledRange` centered on frozen value | Precision control near a sweet spot | + +Parameters also declare a `scaledRange` (default 0.3) for the scaled boundary mode. Example: frozen value 0.8 with scaledRange 0.3 → effective range [0.5, 1.0], clamped at boundaries. + +## Clock Engine + +Replaces setTimeout-based arpeggiator with AudioContext-scheduled timing. + +```javascript +class ClockEngine { + constructor(audioContext) { ... } + + // Properties + bpm // beats per minute + stepCount // total steps in sequence + subdivision // ticks per step (for ratchet/swing resolution) + + // Lookahead scheduling: schedule events slightly ahead of time + // using AudioContext.currentTime for sample-accurate timing + start() → void + stop() → void + setTempo(bpm) → void + + // Callback: called with { stepIndex, time, isAccent } + onStep(callback) → void +} +``` + +The clock uses the standard Web Audio lookahead pattern: +- A setInterval (~25ms) checks if any events need scheduling in the next ~100ms +- Events are scheduled using AudioContext.currentTime for sample-accurate timing +- This decouples visual updates (requestAnimationFrame) from audio timing + +## Projection Layer + +A composable chain of transform functions that convert raw [0,1] MLP outputs into musical values. Each transform is a small, independent module. + +### Available Transforms + +| Transform | Input | Output | Params | +|-----------|-------|--------|--------| +| Scale Quantizer | [0,1] | MIDI note | root, scale, octave range | +| Velocity Curve | [0,1] | [0,1] | curve shape (linear, exponential, S-curve) | +| Gate Threshold | [0,1] | boolean | threshold value | +| Range Map | [0,1] | [min,max] | min, max | +| Octave Folder | MIDI note | MIDI note | target octave range | +| Stutter Map | [0,1] | repeat count | max repeats | + +Transforms snap together: output type of one must match input type of next. The chain is validated on construction. + +### Projection Presets + +Pre-built chain configurations for common use cases: +- **Melodic Minor** — scale quant (A minor) → octave fold (2 oct) → velocity curve (exponential) +- **Pentatonic Drift** — scale quant (pentatonic) → octave fold (3 oct) → velocity curve (S) +- **Chromatic Chaos** — range map (full MIDI) → velocity curve (linear) +- **Rhythmic Only** — gate threshold (0.5) → velocity curve (accent) + +Users can edit any preset or build custom chains. + +## UI Design + +### Circular Step Visualizer + +Steps arranged in a circle with even angular spacing (7 steps = heptagon, 13 steps = 13-gon, etc.). No grid overlay — the ear provides rhythmic context. + +**Visual elements:** +- Each step is a node on the circle +- Active/triggered steps glow or pulse +- Current playback position shown with a rotating indicator +- Pitch mapped to node distance from center (low=outer, high=inner) +- Velocity mapped to node size +- Accents shown with brighter color + +**Interaction:** +- Tap a step to solo/mute it +- Long-press for step detail (all params for that step) + +### Primitive Chain Builder + +Vertical stack layout (like a guitar pedalboard): +- Each primitive is a card with its name and key params visible +- Drag to reorder +- Swipe left to delete +- "+" button at bottom opens primitive palette +- Each card expandable to show all params as sliders +- Params marked as "live" (NISPS-controlled) get a distinct visual indicator (e.g., pulsing border) + +### Parameter Selection (for freeze/re-expose) + +Two input modes: +1. **Mouse/touch** — tap a parameter slider to toggle it between frozen (dimmed) and live (highlighted) +2. **Hand tracking** — point index finger at parameter, pinch to toggle. Visual cursor follows index finger tip. + +Live params show their current NISPS delta as a secondary indicator on the slider. + +### Layout + +``` +┌──────────────────────────────────┐ +│ Circular Step Viz │ +│ (upper half of screen) │ +│ │ +│ ○ ○ │ +│ ○ ○ │ +│ ○ ▶ ○ │ +│ ○ ○ │ +│ ○ ○ │ +│ │ +├──────────────────────────────────┤ +│ Chain Builder (scrollable stack) │ +│ ┌──────────────────────────────┐ │ +│ │ Euclidean [steps][pulses] │ │ +│ │ [rotation] │ │ +│ ├──────────────────────────────┤ │ +│ │ Prob Gate [density][accent] │ │ +│ ├──────────────────────────────┤ │ +│ │ Pitch Walk [step][bias] │ │ +│ ├──────────────────────────────┤ │ +│ │ [ + Add Primitive ] │ │ +│ └──────────────────────────────┘ │ +├──────────────────────────────────┤ +│ [▶ Play] [❄ Freeze] [Chain:Seq] │ +│ BPM: 120 Steps: 8 Scale: Cm │ +└──────────────────────────────────┘ +``` + +## Phased Implementation Plan + +### Phase 1 — Foundation (MVP) + +**Goal:** All 8 primitives working, chain builder, basic UI, NISPS control. Full architecture with minimal polish. + +1. **Event bus** — namespaced pub/sub system +2. **Clock engine** — AudioContext-based precise timing +3. **Primitive framework** — base class, param schema, state management +4. **All 8 primitives** — implement each with their param schemas +5. **Sequential chain** — primitives connected in sequence (pipeline mode only) +6. **Projection layer** — scale quantizer + velocity curve (2 transforms minimum) +7. **Sequence NISPS instance** — second MLP controlling primitive chain params +8. **Basic circular viz** — step circle with playback indicator +9. **Basic chain UI** — vertical stack with sliders, add/remove primitives +10. **Bridge integration** — sequence events → C15 noteOn/noteOff via event bus +11. **Remove arpeggiator** — replace with ShapeSeq + +### Phase 2 — Freeze & Delta Control + +1. **Freeze system** — capture params + seed, capture pattern snapshot +2. **Parameter selection UI** — click to toggle frozen/live +3. **Delta controller** — applies MLP deltas to live params with boundary config +4. **Hand tracking param select** — pinch gesture to toggle params +5. **Freeze mode toggle** — algorithm vs pattern freeze + +### Phase 3 — Advanced Chain & Routing + +1. **Parallel + merge chain mode** +2. **Typed routing chain mode** +3. **Input routing matrix** — configurable input → NISPS instance mapping +4. **Projection chain builder** — user-editable transform chains +5. **Projection presets** + +### Phase 4 — Polish & Expansion + +1. **Preset chains** — pre-built primitive combinations for common genres +2. **Save/load** — persist chain configs, frozen sequences, NISPS state +3. **Unified NISPS mode** — single MLP controlling both timbre + sequence +4. **Additional primitives** as discovered through experimentation +5. **Freeform lasso param selection** (see Future Work) +6. **Per-track variable step counts** (polyrhythm) + +## Future Work + +- **Freeform lasso selection** — draw/lasso over the step visualization to select params spatially. Intuitive but complex to implement. (Create bd backlog issue.) +- **MIDI clock sync** — accept external MIDI clock for hardware sync +- **OSC output** — route sequencer events via OSC for external software/hardware +- **C++ port** — port primitive framework and chain system to nisps-core for RP2040 firmware +- **Multi-track** — multiple independent primitive chains running simultaneously with different step counts (polyrhythm) +- **Markov chain primitive** — transition-probability-based note selection +- **L-system primitive** — Lindenmayer system string rewriting for self-similar patterns +- **Cellular automata primitive** — 1D CA rules (e.g., Rule 30) generating trigger patterns + +## Technical Notes + +### Port-Ready JS Conventions + +To facilitate future C++ porting: +- No closures in primitive process functions — all state is explicit +- Use typed arrays (Float32Array) for parameter vectors where possible +- Primitives are pure functions with explicit state in/out +- Seeded PRNG (not Math.random()) for deterministic replay +- All time values in seconds (AudioContext convention), not milliseconds + +### PRNG + +Use a seedable PRNG (e.g., mulberry32 or xoshiro128) so that: +- Freeze-as-algorithm can replay identical sequences from seed +- Different primitives in a chain get independent PRNG streams (derived from a master seed) +- Deterministic behavior aids debugging and reproducibility + +### Performance Budget + +The sequencer tick runs at most every ~15ms (at 250 BPM with 16th note subdivision). Each tick must: +1. Query NISPS MLP (already fast — the MLP inference is <1ms) +2. Run primitive chain (8 primitives × simple math = negligible) +3. Apply projection transforms (simple lookups/math) +4. Emit events + +Total budget per tick: ~5ms. This is comfortable even on mobile.