memlnaut-nisps/playground/SPEC-shapeseq.md

ShapeSeq — NISPS Generative Sequencing System

Overview

ShapeSeq is a generative sequencing system for the NISPS playground where interactive ML (via the NISPS MLP engine) controls parameters of algorithmic sequencing primitives rather than raw note data. The user shapes sequences by navigating a learned parameter space with a joystick or hand tracking, can freeze sequences they like, then selectively re-expose specific parameters for further ML-driven exploration.

ShapeSeq replaces the existing placeholder arpeggiator.

Core Architecture

Design Principles

1. MLP outputs are abstract [0,1] values — musical meaning is applied downstream by a configurable projection layer
2. Separate NISPS instances for timbre control and sequence control, with architecture supporting future unification into a single instance
3. Port-ready JS — no closures in hot paths, explicit state, data structures that map cleanly to C++ for future RP2040 firmware porting
4. Modular primitives — small, combinable algorithmic building blocks that generate musical patterns from continuous parameters

System Diagram

                    ┌─────────────────┐
                    │  Input Router   │
                    │ (configurable)  │
                    └──┬──────────┬───┘
                       │          │
              ┌────────▼──┐  ┌───▼────────┐
              │ NISPS MLP  │  │ NISPS MLP  │
              │ (timbre)   │  │ (sequence) │
              └────────┬───┘  └───┬────────┘
                       │          │
              ┌────────▼──┐  ┌───▼────────────────────┐
              │ Synth      │  │ Delta Controller       │
              │ Param Map  │  │ (frozen vals + deltas) │
              └────────┬───┘  └───┬────────────────────┘
                       │          │
                       │     ┌────▼──────────────┐
                       │     │ Primitive Chain    │
                       │     │ Euclid→ProbGate→… │
                       │     └────┬──────────────┘
                       │          │
                       │     ┌────▼──────────────┐
                       │     │ Projection Layer   │
                       │     │ (scale quant, etc) │
                       │     └────┬──────────────┘
                       │          │
                       │     ┌────▼──────────┐
                       │     │ Clock Engine   │
                       │     │ (AudioContext)  │
                       │     └────┬──────────┘
                       │          │
                  ┌────▼──────────▼────┐
                  │ Namespaced Event Bus│
                  │ seq.* ml.* ui.*    │
                  └────────┬───────────┘
                           │
                  ┌────────▼───────┐
                  │ C15 Synth      │
                  │ (noteOn/Off +  │
                  │  param changes)│
                  └────────────────┘

Namespaced Event Bus

A pub/sub event system with namespaced channels:

Namespace	Events	Purpose
seq.*	seq.step, seq.noteOn, seq.noteOff, seq.paramChange, seq.loopStart	Musical output from sequencer to synth and visualizer
ml.*	ml.trained, ml.frozen, ml.unfrozen, ml.deltaUpdate	ML state changes
ui.*	ui.paramSelect, ui.chainEdit, ui.presetLoad, ui.freezeToggle	User actions

All events carry a timestamp (AudioContext.currentTime for seq.*, performance.now() for others).

Input Routing Matrix

A configurable routing layer that maps any input source to either NISPS instance's inputs:

Input sources:

• Joystick X, Y (2 values)
• Hand tracking features (14 values)
• Gamepad axes (variable)

Routing targets:

• Timbre NISPS input 0, input 1
• Sequence NISPS input 0, input 1

Default: joystick → timbre NISPS, hand tracking features 0+1 → sequence NISPS. User-configurable via UI.

Sequencing Primitives

Each primitive is a pure function (or stateful generator with explicit state) that accepts a parameter object and produces typed output. All parameters are normalized [0,1].

1. Euclidean Rhythm Generator

Params: steps (int, from continuous), pulses (int), rotation (int) Output type: trigger pattern (boolean array) Stateless: yes

Generates Bjorklund-distributed trigger patterns. The continuous [0,1] params are projected to integer ranges based on current step count.

2. Probability Gate

Params: density [0,1], accentProbability [0,1] Input type: trigger pattern Output type: filtered trigger pattern with accent flags Stateless: yes (per-step coin flip using seeded PRNG)

Each incoming trigger survives with probability density. Surviving triggers receive accent flag with probability accentProbability.

3. Pitch Walker

Params: stepSize [0,1], directionBias [0,1] (0.5=unbiased), gravity [0,1] (pull toward center), range [0,1] Input type: trigger pattern Output type: pitch values [0,1] per triggered step Stateful: yes — maintains current position in pitch space

Constrained random walk that generates melodic contour. gravity pulls the walk toward center (0.5), preventing it from getting stuck at extremes. State includes current position and PRNG state.

4. Ratchet

Params: maxDivision [0,1] (maps to 1-4 subdivisions), probability [0,1] Input type: trigger pattern Output type: trigger pattern with subdivision timing offsets Stateless: yes (per-step coin flip)

Subdivides triggered steps into rapid repeats. Division count determined by maxDivision, applied probabilistically.

5. Swing / Groove

Params: swingAmount [0,1] (0=straight, 1=full swing), swingGrid [0,1] (which subdivisions swing) Input type: timing information Output type: modified timing offsets Stateless: yes

Shifts timing of alternating steps. At swingAmount=0.67 this produces classic 2:1 shuffle. swingGrid controls whether swing applies to 8th notes, 16th notes, or triplets.

6. Density Morph

Params: density [0,1], clustering [0,1] (0=spread evenly, 1=clustered together) Input type: step count Output type: trigger pattern Stateless: yes

Alternative to Euclidean — generates trigger patterns with controllable density and spatial distribution. At high clustering, triggers group together creating bursts; at low clustering, triggers spread evenly.

7. Interval Lock (Scale Quantizer)

Params: root [0,1] (maps to 0-11 semitones), mode [0,1] (maps to scale index), octaveRange [0,1] (1-4 octaves) Input type: pitch values [0,1] Output type: MIDI note numbers Stateless: yes

Available scales: chromatic, major, natural minor, harmonic minor, pentatonic major, pentatonic minor, blues, dorian, mixolydian, whole tone, diminished.

8. Velocity Shaper

Params: curveType [0,1] (maps to: flat, accent-every-N, crescendo, decrescendo, random), depth [0,1], phase [0,1] Input type: trigger pattern with step indices Output type: velocity values [0,1] per step Stateless: yes

Applies cyclic velocity patterns. phase rotates the pattern, depth controls contrast between quiet and loud.

Primitive Interface

// Port-ready: explicit state, no closures
class Primitive {
  constructor(name, paramSchema) { ... }

  // paramSchema: array of { name, min, max, default, boundary }
  // boundary: 'clamp' | 'wrap' | 'scaled'
  // For 'scaled': operates within ±scaledRange of frozen value

  // Pure processing function
  process(params, input, state, stepCount, rng) → { output, nextState }

  // State management for freeze
  getState() → serializable object
  setState(state) → void
  getSeed() → number
  setSeed(seed) → void
}

Chain Connection Modes

Three configurable modes for how primitives connect in a chain:

1. Sequential Pipeline — each primitive transforms the previous output. Order matters. Signal flows left to right.

2. Parallel + Merge — each primitive runs independently, outputs merged (OR for triggers, average for continuous values). Order doesn't matter.

3. Typed Routing — primitives connect via typed ports. A primitive's output connects to the next primitive that accepts that type. Multiple primitives can feed the same type (merged). Most flexible, most complex.

The chain connection mode is a global setting (per-chain), configurable via UI. Default: sequential pipeline.

Delta Control System

Freeze Workflow

1. User plays with NISPS, finds a sequence they like
2. User activates freeze — all current parameter values are captured
3. User selects specific parameters to re-expose (mark as "live"):
   • Click/tap parameters in the UI
   • Or use hand tracking: point with index finger, pinch gesture to toggle
4. Live parameters receive deltas from NISPS MLP output
5. Frozen parameters hold their captured values

Freeze Modes

Freeze as Algorithm — captures parameter values + PRNG seed. Stateful primitives (pitch walker) will replay identically. Re-exposing params resumes algorithmic generation with delta-modified params.

Freeze as Pattern — captures the realized note pattern (snapshot of all step events for one full loop). The primitive chain is bypassed; the sequencer loops the frozen pattern directly. Re-exposing params requires switching back to algorithm mode.

User chooses freeze mode via UI toggle.

Delta Boundary Behavior

Each parameter declares its boundary behavior:

Behavior	Description	Good for
clamp	Delta result clamped to [0,1]	Velocity, volume, most continuous params
wrap	Values wrap around (1.1 → 0.1)	Rotation, phase, cyclic params
scaled	Delta operates within ±scaledRange centered on frozen value	Precision control near a sweet spot

Parameters also declare a scaledRange (default 0.3) for the scaled boundary mode. Example: frozen value 0.8 with scaledRange 0.3 → effective range [0.5, 1.0], clamped at boundaries.

Clock Engine

Replaces setTimeout-based arpeggiator with AudioContext-scheduled timing.

class ClockEngine {
  constructor(audioContext) { ... }

  // Properties
  bpm         // beats per minute
  stepCount   // total steps in sequence
  subdivision // ticks per step (for ratchet/swing resolution)

  // Lookahead scheduling: schedule events slightly ahead of time
  // using AudioContext.currentTime for sample-accurate timing
  start() → void
  stop() → void
  setTempo(bpm) → void

  // Callback: called with { stepIndex, time, isAccent }
  onStep(callback) → void
}

The clock uses the standard Web Audio lookahead pattern:

• A setInterval (~25ms) checks if any events need scheduling in the next ~100ms
• Events are scheduled using AudioContext.currentTime for sample-accurate timing
• This decouples visual updates (requestAnimationFrame) from audio timing

Projection Layer

A composable chain of transform functions that convert raw [0,1] MLP outputs into musical values. Each transform is a small, independent module.

Available Transforms

Transform	Input	Output	Params
Scale Quantizer	[0,1]	MIDI note	root, scale, octave range
Velocity Curve	[0,1]	[0,1]	curve shape (linear, exponential, S-curve)
Gate Threshold	[0,1]	boolean	threshold value
Range Map	[0,1]	[min,max]	min, max
Octave Folder	MIDI note	MIDI note	target octave range
Stutter Map	[0,1]	repeat count	max repeats

Transforms snap together: output type of one must match input type of next. The chain is validated on construction.

Projection Presets

Pre-built chain configurations for common use cases:

• Melodic Minor — scale quant (A minor) → octave fold (2 oct) → velocity curve (exponential)
• Pentatonic Drift — scale quant (pentatonic) → octave fold (3 oct) → velocity curve (S)
• Chromatic Chaos — range map (full MIDI) → velocity curve (linear)
• Rhythmic Only — gate threshold (0.5) → velocity curve (accent)

Users can edit any preset or build custom chains.

UI Design

Circular Step Visualizer

Steps arranged in a circle with even angular spacing (7 steps = heptagon, 13 steps = 13-gon, etc.). No grid overlay — the ear provides rhythmic context.

Visual elements:

• Each step is a node on the circle
• Active/triggered steps glow or pulse
• Current playback position shown with a rotating indicator
• Pitch mapped to node distance from center (low=outer, high=inner)
• Velocity mapped to node size
• Accents shown with brighter color

Interaction:

• Tap a step to solo/mute it
• Long-press for step detail (all params for that step)

Primitive Chain Builder

Vertical stack layout (like a guitar pedalboard):

• Each primitive is a card with its name and key params visible
• Drag to reorder
• Swipe left to delete
• "+" button at bottom opens primitive palette
• Each card expandable to show all params as sliders
• Params marked as "live" (NISPS-controlled) get a distinct visual indicator (e.g., pulsing border)

Parameter Selection (for freeze/re-expose)

Two input modes:

1. Mouse/touch — tap a parameter slider to toggle it between frozen (dimmed) and live (highlighted)
2. Hand tracking — point index finger at parameter, pinch to toggle. Visual cursor follows index finger tip.

Live params show their current NISPS delta as a secondary indicator on the slider.

Layout

┌──────────────────────────────────┐
│         Circular Step Viz        │
│        (upper half of screen)    │
│                                  │
│              ○   ○               │
│           ○         ○            │
│          ○     ▶     ○           │
│           ○         ○            │
│              ○   ○               │
│                                  │
├──────────────────────────────────┤
│ Chain Builder (scrollable stack) │
│ ┌──────────────────────────────┐ │
│ │ Euclidean  [steps][pulses]   │ │
│ │            [rotation]        │ │
│ ├──────────────────────────────┤ │
│ │ Prob Gate  [density][accent] │ │
│ ├──────────────────────────────┤ │
│ │ Pitch Walk [step][bias]      │ │
│ ├──────────────────────────────┤ │
│ │        [ + Add Primitive ]   │ │
│ └──────────────────────────────┘ │
├──────────────────────────────────┤
│ [▶ Play] [❄ Freeze] [Chain:Seq] │
│ BPM: 120   Steps: 8   Scale: Cm │
└──────────────────────────────────┘

Phased Implementation Plan

Phase 1 — Foundation (MVP)

Goal: All 8 primitives working, chain builder, basic UI, NISPS control. Full architecture with minimal polish.

1. Event bus — namespaced pub/sub system
2. Clock engine — AudioContext-based precise timing
3. Primitive framework — base class, param schema, state management
4. All 8 primitives — implement each with their param schemas
5. Sequential chain — primitives connected in sequence (pipeline mode only)
6. Projection layer — scale quantizer + velocity curve (2 transforms minimum)
7. Sequence NISPS instance — second MLP controlling primitive chain params
8. Basic circular viz — step circle with playback indicator
9. Basic chain UI — vertical stack with sliders, add/remove primitives
10. Bridge integration — sequence events → C15 noteOn/noteOff via event bus
11. Remove arpeggiator — replace with ShapeSeq

Phase 2 — Freeze & Delta Control

1. Freeze system — capture params + seed, capture pattern snapshot
2. Parameter selection UI — click to toggle frozen/live
3. Delta controller — applies MLP deltas to live params with boundary config
4. Hand tracking param select — pinch gesture to toggle params
5. Freeze mode toggle — algorithm vs pattern freeze

Phase 3 — Advanced Chain & Routing

1. Parallel + merge chain mode
2. Typed routing chain mode
3. Input routing matrix — configurable input → NISPS instance mapping
4. Projection chain builder — user-editable transform chains
5. Projection presets

Phase 4 — Polish & Expansion

1. Preset chains — pre-built primitive combinations for common genres
2. Save/load — persist chain configs, frozen sequences, NISPS state
3. Unified NISPS mode — single MLP controlling both timbre + sequence
4. Additional primitives as discovered through experimentation
5. Freeform lasso param selection (see Future Work)
6. Per-track variable step counts (polyrhythm)

Future Work

• Freeform lasso selection — draw/lasso over the step visualization to select params spatially. Intuitive but complex to implement. (Create bd backlog issue.)
• MIDI clock sync — accept external MIDI clock for hardware sync
• OSC output — route sequencer events via OSC for external software/hardware
• C++ port — port primitive framework and chain system to nisps-core for RP2040 firmware
• Multi-track — multiple independent primitive chains running simultaneously with different step counts (polyrhythm)
• Markov chain primitive — transition-probability-based note selection
• L-system primitive — Lindenmayer system string rewriting for self-similar patterns
• Cellular automata primitive — 1D CA rules (e.g., Rule 30) generating trigger patterns

Technical Notes

Port-Ready JS Conventions

To facilitate future C++ porting:

• No closures in primitive process functions — all state is explicit
• Use typed arrays (Float32Array) for parameter vectors where possible
• Primitives are pure functions with explicit state in/out
• Seeded PRNG (not Math.random()) for deterministic replay
• All time values in seconds (AudioContext convention), not milliseconds

PRNG

Use a seedable PRNG (e.g., mulberry32 or xoshiro128) so that:

• Freeze-as-algorithm can replay identical sequences from seed
• Different primitives in a chain get independent PRNG streams (derived from a master seed)
• Deterministic behavior aids debugging and reproducibility

Performance Budget

The sequencer tick runs at most every ~15ms (at 250 BPM with 16th note subdivision). Each tick must:

1. Query NISPS MLP (already fast — the MLP inference is <1ms)
2. Run primitive chain (8 primitives × simple math = negligible)
3. Apply projection transforms (simple lookups/math)
4. Emit events

Total budget per tick: ~5ms. This is comfortable even on mobile.