memlnaut-nisps/playground/SPEC-shapeseq.md

ShapeSeq — NISPS Generative Sequencing System

Overview

ShapeSeq is a generative sequencing system for the NISPS playground where interactive ML (via the NISPS MLP engine) controls parameters of algorithmic sequencing primitives rather than raw note data. The user shapes sequences by navigating a learned parameter space with a joystick or hand tracking, can freeze sequences they like, then selectively re-expose specific parameters for further ML-driven exploration.

ShapeSeq replaces the existing placeholder arpeggiator. In synth output mode, the ShapeSeq UI (circular step visualizer + chain builder + param sliders) replaces the flow-field particle system.

Core Architecture

Design Principles

1. MLP outputs are abstract [0,1] values — musical meaning is applied downstream by the primitive chain and its symbolic processing
2. Separate NISPS instances for timbre control and sequence control, following the existing imlJoy/imlHand dual-instance pattern in a-app.js. Architecture supports future unification into a single instance
3. Port-ready JS — no closures in hot paths, explicit state, data structures that map cleanly to C++ for future RP2040 firmware porting. Note: the event bus and clock orchestration are JS-only concerns and not expected to port directly; the primitives themselves are the portable layer
4. Modular primitives — small, combinable algorithmic building blocks that generate musical patterns from continuous parameters
5. Symbolic chain — primitives compose as transforms over a pattern description, not concrete values. Each primitive takes the previous "pattern-generating machine" specification and produces a new one. The chain is evaluated once per loop (or on param change) to produce a complete pattern, which the clock then steps through

System Diagram

              ┌──────────────────┐
              │   Clock Engine   │  ← drives everything
              │  (AudioContext)  │
              └──────┬───────────┘
                     │ tick
                     ▼
              ┌──────────────────┐     ┌─────────────────┐
              │  Sequencer Core  │◄────│  Input Router   │
              │  (orchestrator)  │     │ (configurable)  │
              └──┬───────────┬───┘     └──┬──────────┬───┘
                 │           │            │          │
                 │ query     │ query ┌────▼───┐ ┌───▼────────┐
                 │ pattern   │ MLP   │NISPS   │ │ NISPS MLP  │
                 │           │       │(timbre)│ │ (sequence)  │
                 │           │       └───┬────┘ └───┬────────┘
                 │           │           │          │
                 │    ┌──────▼────────┐  │   ┌─────▼──────────┐
                 │    │ Param Mapping │  │   │ Param Mapping   │
                 │    │ (16 MLP outs  │  │   │ (16 MLP outs →  │
                 │    │  → N prim     │  │   │  126 synth      │
                 │    │  params)      │  │   │  params)        │
                 │    └──────┬────────┘  │   └─────┬──────────┘
                 │           │           │         │
                 │    ┌──────▼────────┐  │  ┌─────▼──────────┐
                 │    │Delta Controller│ │  │ Synth Param Map │
                 │    │(frozen+deltas)│  │  └─────┬──────────┘
                 │    └──────┬────────┘  │        │
                 │           │           │        │
                 │    ┌──────▼────────┐  │        │
                 │    │Primitive Chain │  │        │
                 │    │(symbolic eval) │  │        │
                 │    └──────┬────────┘  │        │
                 │           │           │        │
          ┌──────▼───────────▼───┐       │        │
          │ Namespaced Event Bus │       │        │
          │ seq.* ml.* ui.*      │       │        │
          └──┬───────────────┬───┘       │        │
             │               │           │        │
      ┌──────▼───────┐ ┌────▼───────────▼────────▼──┐
      │ Circular Viz  │ │        C15 Synth           │
      │ + Chain UI    │ │   (noteOn/Off + params)    │
      └──────────────┘ └────────────────────────────┘

Fixed MLP + Param Mapping Layer

The WasmIML creates an MLP with a fixed output count at construction time — it cannot be resized. Since the primitive chain is dynamic (users add/remove primitives, changing total param count), the MLP cannot output directly to primitive params.

Solution: The sequence MLP always outputs a fixed number of values (e.g., 16). A param mapping layer fans these 16 outputs to however many primitive params the current chain requires. This is the same pattern used by the timbre MLP (which maps to 126 synth params via param-map.js).

The mapping can be:

• Automatic (default): outputs are distributed across primitive params in chain order. If there are 30 primitive params and 16 MLP outputs, each output influences ~2 params via interpolation.
• Future: configurable user-defined mapping.

Design note: The fixed-16-output approach is the simplest starting point. If experimentation reveals that 16 is too few (or too many), the MLP can be reconstructed with a different size — this is a one-time setup cost, not a per-frame cost. The mapping layer insulates the rest of the system from this choice. Revisit if the mapping layer becomes a bottleneck for expressiveness.

Namespaced Event Bus

A pub/sub event system with namespaced channels:

Namespace	Events	Purpose
seq.*	seq.step, seq.noteOn, seq.noteOff, seq.paramChange, seq.loopStart	Musical output from sequencer to synth and visualizer
ml.*	ml.trained, ml.frozen, ml.unfrozen, ml.deltaUpdate	ML state changes
ui.*	ui.paramSelect, ui.chainEdit, ui.presetLoad, ui.freezeToggle	User actions

All events carry a timestamp (AudioContext.currentTime for seq.*, performance.now() for others).

Note: the event bus is a JS-only orchestration concern (string-namespaced pub/sub). It does not need to be port-ready — the portable layer is the primitives themselves.

Input Routing Matrix

A configurable routing layer that maps any input source to either NISPS instance's inputs. Builds on the existing imlJoy/imlHand switching pattern in a-app.js.

Input sources:

• Joystick X, Y (2 values)
• Hand tracking features (14 values)
• Gamepad axes (variable)

Routing targets:

• Timbre NISPS input 0, input 1
• Sequence NISPS input 0, input 1

Default: joystick → timbre NISPS, hand tracking features 0+1 → sequence NISPS. User-configurable via UI.

Sequencing Primitives

Primitive Categories

Primitives are categorized by their role in the chain:

Category	Role	Examples
Generator	Produces data from params alone (no input required)	Euclidean, Density Morph, Pitch Walker
Processor	Transforms incoming data	Probability Gate, Velocity Shaper
Timing Modifier	Modulates the timing of events in the pattern description	Swing/Groove, Ratchet
Converter	Changes data type (e.g., continuous → discrete)	Interval Lock

Generator combination rule: When multiple generators appear in the same chain, their outputs combine according to the chain's combination mode (user-configurable in real time):

• Additive (OR) — triggers from any generator fire. Pitch/velocity values are averaged where multiple generators contribute.
• Multiplicative (AND) — only steps where ALL generators agree will fire. Creates sparser, more selective patterns.

Symbolic Chain Evaluation

Primitives do NOT process concrete note data step-by-step. Instead, each primitive takes the previous pattern description (a symbolic representation of the entire sequence) and produces a new one. The complete chain is evaluated to produce a full pattern, which the clock then steps through.

This means:

• Timing modifiers (Swing, Ratchet) work by annotating the pattern description with timing offsets and subdivisions before any concrete scheduling happens
• The clock reads the finalized pattern description and schedules all events (including ratchet subdivisions and swing offsets) using AudioContext.currentTime
• Re-evaluation happens when params change (MLP output updates, user edits), not on every tick

Pattern description structure:

// The symbolic output of the chain — a complete loop description
{
  steps: [
    {
      trigger: true,          // whether this step fires
      pitch: 0.72,            // [0,1] abstract pitch (pre-quantization)
      velocity: 0.85,         // [0,1]
      accent: false,          // accent flag
      timeOffset: 0.0,        // swing offset in fractions of a step (-0.5 to +0.5)
      subdivisions: 1,        // ratchet: 1 = normal, 2-4 = subdivided
    },
    // ... one per step
  ],
  stepCount: 8,
  metadata: { ... }           // chain-specific info for visualization
}

Primitive Definitions

Each primitive is a pure function (or stateful generator with explicit state) that accepts a parameter object and produces typed output. All parameters are normalized [0,1].

1. Euclidean Rhythm Generator

Category: Generator Params: steps (int, from continuous), pulses (int), rotation (int) Output: trigger pattern (boolean array) Stateless: yes

Generates Bjorklund-distributed trigger patterns. The continuous [0,1] params are projected to integer ranges based on current step count.

2. Probability Gate

Category: Processor Params: density [0,1], accentProbability [0,1] Input: trigger pattern Output: filtered trigger pattern with accent flags Stateless: yes (per-step coin flip using seeded PRNG)

Each incoming trigger survives with probability density. Surviving triggers receive accent flag with probability accentProbability.

3. Pitch Walker

Category: Generator Params: stepSize [0,1], directionBias [0,1] (0.5=unbiased), gravity [0,1] (pull toward center), range [0,1] Output: pitch values [0,1] per triggered step Stateful: yes — maintains current position in pitch space

Constrained random walk that generates melodic contour. gravity pulls the walk toward center (0.5), preventing it from getting stuck at extremes. State includes current position and PRNG state.

4. Ratchet

Category: Timing Modifier Params: maxDivision [0,1] (maps to 1-4 subdivisions), probability [0,1] Input: pattern description with triggers Output: pattern description with subdivisions field set per step Stateless: yes (per-step coin flip)

Annotates triggered steps with subdivision counts. The clock engine reads subdivisions and schedules rapid repeats within the step's time window. Division count determined by maxDivision, applied probabilistically.

5. Swing / Groove

Category: Timing Modifier Params: swingAmount [0,1] (0=straight, 1=full swing), swingGrid [0,1] (which subdivisions swing) Input: pattern description Output: pattern description with timeOffset field set per step Stateless: yes

Annotates alternating steps with timing offsets. At swingAmount=0.67 this produces classic 2:1 shuffle. swingGrid controls whether swing applies to 8th notes, 16th notes, or triplets. The clock engine reads timeOffset and adjusts scheduling accordingly.

6. Density Morph

Category: Generator Params: density [0,1], clustering [0,1] (0=spread evenly, 1=clustered together) Output: trigger pattern Stateless: yes

Alternative to Euclidean — generates trigger patterns with controllable density and spatial distribution. At high clustering, triggers group together creating bursts; at low clustering, triggers spread evenly.

7. Interval Lock (Scale Quantizer)

Category: Converter Params: root [0,1] (maps to 0-11 semitones), mode [0,1] (maps to scale index), octaveRange [0,1] (1-4 octaves) Input: pitch values [0,1] Output: MIDI note numbers Stateless: yes

The sole pitch quantization mechanism — the projection layer does NOT duplicate this. All pitch quantization goes through Interval Lock.

Available scales: chromatic, major, natural minor, harmonic minor, pentatonic major, pentatonic minor, blues, dorian, mixolydian, whole tone, diminished.

8. Velocity Shaper

Category: Processor Params: curveType [0,1] (maps to: flat, accent-every-N, crescendo, decrescendo, random), depth [0,1], phase [0,1] Input: trigger pattern with step indices Output: velocity values [0,1] per step Stateless: yes

Applies cyclic velocity patterns. phase rotates the pattern, depth controls contrast between quiet and loud.

Primitive Interface

// Port-ready: explicit state, no closures
class Primitive {
  constructor(name, paramSchema, category) { ... }

  // category: 'generator' | 'processor' | 'timing' | 'converter'

  // paramSchema: array of { name, min, max, default, boundary }
  // boundary: 'clamp' | 'wrap' | 'scaled'
  // For 'scaled': operates within ±scaledRange of frozen value

  // Symbolic processing: transforms a pattern description
  process(params, patternDesc, state, rng) → { patternDesc, nextState }

  // State management for freeze
  getState() → serializable object
  setState(state) → void
  getSeed() → number
  setSeed(seed) → void
}

Chain Connection Modes

Three configurable modes for how primitives connect in a chain:

1. Sequential Pipeline — each primitive transforms the pattern description in order. Generators create initial data, processors/timing modifiers transform it. If multiple generators appear, they combine according to the generator combination mode (additive/multiplicative, configurable in real time).

2. Parallel + Merge — each primitive runs independently and produces a pattern description. Descriptions merge (OR for triggers in additive mode, AND in multiplicative mode; average for continuous values). Order doesn't matter.

3. Typed Routing — primitives connect via typed ports. A primitive's output connects to the next primitive that accepts that type. Multiple primitives can feed the same type (merged). Most flexible, most complex.

The chain connection mode is a global setting (per-chain), configurable via UI. Default: sequential pipeline.

Generator combination mode (additive/multiplicative) is an independent setting, also configurable in real time via UI.

Delta Control System

Freeze Workflow

1. User plays with NISPS, finds a sequence they like
2. User activates freeze — all current parameter values are captured
3. User selects specific parameters to re-expose (mark as "live"):
   • Click/tap parameters in the UI
   • Or use hand tracking: point with index finger, pinch gesture to toggle
4. Live parameters receive deltas from NISPS MLP output
5. Frozen parameters hold their captured values

Freeze Modes

Freeze as Algorithm — captures parameter values + PRNG seed. Stateful primitives (pitch walker) will replay identically. Re-exposing params resumes algorithmic generation with delta-modified params.

Freeze as Pattern — captures the realized note pattern (snapshot of all step events for one full loop). The primitive chain is bypassed; the sequencer loops the frozen pattern directly. Re-exposing params requires switching back to algorithm mode.

User chooses freeze mode via UI toggle.

Delta Boundary Behavior

Each parameter declares its boundary behavior:

Behavior	Description	Good for
clamp	Delta result clamped to [0,1]	Velocity, volume, most continuous params
wrap	Values wrap around (1.1 → 0.1)	Rotation, phase, cyclic params
scaled	Delta operates within ±scaledRange centered on frozen value	Precision control near a sweet spot

Parameters also declare a scaledRange (default 0.3) for the scaled boundary mode. Example: frozen value 0.8 with scaledRange 0.3 → effective range [0.5, 1.0], clamped at boundaries.

Clock Engine

Replaces setTimeout-based arpeggiator with AudioContext-scheduled timing.

class ClockEngine {
  constructor(audioContext) { ... }

  // Properties
  bpm         // beats per minute
  stepCount   // total steps in sequence

  // Lookahead scheduling: schedule events slightly ahead of time
  // using AudioContext.currentTime for sample-accurate timing
  start() → void
  stop() → void
  setTempo(bpm) → void

  // The clock reads the finalized pattern description and schedules
  // all events, including:
  // - timeOffset per step (swing)
  // - subdivisions per step (ratchet)
  // - accent flags (velocity scaling)
  schedulePattern(patternDesc) → void

  // Callback: called with { stepIndex, time, velocity, pitch, isSubdivision }
  onEvent(callback) → void
}

The clock uses the standard Web Audio lookahead pattern:

• A setInterval (~25ms) checks if any events need scheduling in the next ~100ms
• Events are scheduled using AudioContext.currentTime for sample-accurate timing
• This decouples visual updates (requestAnimationFrame) from audio timing
• The clock handles ratchet subdivisions and swing offsets natively by reading the pattern description's per-step subdivisions and timeOffset fields

Projection Layer

A composable chain of transform functions that convert raw [0,1] primitive outputs into final musical values. Each transform is a small, independent module.

Note: pitch quantization is handled by the Interval Lock primitive, not the projection layer. The projection layer handles non-pitch transforms only.

Available Transforms

Transform	Input	Output	Params
Velocity Curve	[0,1]	[0,1]	curve shape (linear, exponential, S-curve)
Gate Threshold	[0,1]	boolean	threshold value
Range Map	[0,1]	[min,max]	min, max
Octave Folder	MIDI note	MIDI note	target octave range
Stutter Map	[0,1]	repeat count	max repeats

Transforms snap together: output type of one must match input type of next. The chain is validated on construction.

Projection Presets

Pre-built chain configurations for common use cases:

• Expressive — velocity curve (exponential) → range map (48-84)
• Percussive — gate threshold (0.5) → velocity curve (accent)
• Full Range — range map (24-96) → velocity curve (linear)

Users can edit any preset or build custom chains.

UI Design

Mode Integration

ShapeSeq activates in synth output mode. When synth mode is active:

• The flow-field particle visualizer is replaced by the ShapeSeq UI (circular step viz + chain builder + param sliders)
• The timbre NISPS instance continues to control C15 synth parameters as before
• The sequence NISPS instance drives the ShapeSeq primitive chain

In visual output mode, the particle system remains unchanged.

Circular Step Visualizer

Steps arranged in a circle with even angular spacing (7 steps = heptagon, 13 steps = 13-gon, etc.). No grid overlay — the ear provides rhythmic context.

Visual elements:

• Each step is a node on the circle
• Active/triggered steps glow or pulse
• Current playback position shown with a rotating indicator
• Pitch mapped to node distance from center (low=outer, high=inner)
• Velocity mapped to node size
• Accents shown with brighter color

Interaction:

• Tap a step to solo/mute it
• Long-press for step detail (all params for that step)

Primitive Chain Builder

Vertical stack layout (like a guitar pedalboard):

• Each primitive is a card with its name and key params visible
• Drag to reorder
• Swipe left to delete
• "+" button at bottom opens primitive palette
• Each card expandable to show all params as sliders
• Params marked as "live" (NISPS-controlled) get a distinct visual indicator (e.g., pulsing border)

Parameter Selection (for freeze/re-expose)

Two input modes:

1. Mouse/touch — tap a parameter slider to toggle it between frozen (dimmed) and live (highlighted)
2. Hand tracking — point index finger at parameter, pinch to toggle. Visual cursor follows index finger tip.

Live params show their current NISPS delta as a secondary indicator on the slider.

Layout

┌──────────────────────────────────┐
│         Circular Step Viz        │
│        (upper half of screen)    │
│                                  │
│              ○   ○               │
│           ○         ○            │
│          ○     ▶     ○           │
│           ○         ○            │
│              ○   ○               │
│                                  │
├──────────────────────────────────┤
│ Chain Builder (scrollable stack) │
│ ┌──────────────────────────────┐ │
│ │ Euclidean  [steps][pulses]   │ │
│ │            [rotation]        │ │
│ ├──────────────────────────────┤ │
│ │ Prob Gate  [density][accent] │ │
│ ├──────────────────────────────┤ │
│ │ Pitch Walk [step][bias]      │ │
│ ├──────────────────────────────┤ │
│ │        [ + Add Primitive ]   │ │
│ └──────────────────────────────┘ │
├──────────────────────────────────┤
│ [▶ Play] [❄ Freeze] [Chain:Seq] │
│ [+×] BPM:120  Steps:8  Gen:Add  │
└──────────────────────────────────┘

Phased Implementation Plan

Phase 1 — Foundation (MVP)

Goal: All 8 primitives working, chain builder, basic UI, NISPS control. Full architecture with minimal polish.

1. Event bus — namespaced pub/sub system
2. Clock engine — AudioContext-based precise timing with pattern description scheduling (handles swing offsets + ratchet subdivisions)
3. Primitive framework — base class, param schema, category system, state management, symbolic pattern description structure
4. All 8 primitives — implement each with their param schemas and categories
5. Sequential chain — primitives connected in sequence (pipeline mode only), with additive/multiplicative generator combination mode
6. Param mapping layer — fixed 16-output MLP → N primitive params, automatic distribution
7. Projection layer — velocity curve + gate threshold (2 transforms minimum, no scale quantizer — that's Interval Lock)
8. Sequence NISPS instance — second WasmIML (16 outputs), following existing imlJoy/imlHand pattern
9. Basic circular viz — step circle with playback indicator
10. Basic chain UI — vertical stack with sliders, add/remove primitives, generator combo mode toggle
11. Bridge integration — sequence events → C15 noteOn/noteOff via event bus
12. Replace arpeggiator — remove old arpeggiator, ShapeSeq takes over in synth mode

Phase 2 — Freeze & Delta Control

1. Freeze system — capture params + seed, capture pattern snapshot
2. Parameter selection UI — click to toggle frozen/live
3. Delta controller — applies MLP deltas to live params with boundary config
4. Hand tracking param select — pinch gesture to toggle params
5. Freeze mode toggle — algorithm vs pattern freeze

Phase 3 — Advanced Chain & Routing

1. Parallel + merge chain mode
2. Typed routing chain mode
3. Input routing matrix — configurable input → NISPS instance mapping
4. Projection chain builder — user-editable transform chains
5. Projection presets

Phase 4 — Polish & Expansion

1. Preset chains — pre-built primitive combinations for common genres
2. Save/load — persist chain configs, frozen sequences, NISPS state
3. Unified NISPS mode — single MLP controlling both timbre + sequence
4. Additional primitives as discovered through experimentation
5. Freeform lasso param selection (see Future Work)
6. Per-track variable step counts (polyrhythm)

Open Design Questions

These are deliberately deferred decisions to be revisited after experimentation:

1. MLP output count: Is 16 the right number for the sequence MLP? Too few may limit expressiveness; too many may make learning harder. The param mapping layer insulates the system, so this can be changed without architectural impact.
2. Param mapping strategy: Automatic distribution is the starting point. Should users be able to manually wire MLP outputs to specific primitive params? This could enable more intentional control but adds UI complexity.
3. Generator combination modes: Additive and multiplicative are the starting pair. Other modes worth exploring: weighted average, priority (first generator wins), XOR (one or the other but not both).
4. Chain evaluation frequency: Currently re-evaluates when params change. Should there be an option for per-loop re-evaluation (stateful primitives produce different patterns each loop)?

Future Work

• Freeform lasso selection — draw/lasso over the step visualization to select params spatially. Intuitive but complex to implement. (Backlog issue: meml-hud)
• MIDI clock sync — accept external MIDI clock for hardware sync
• OSC output — route sequencer events via OSC for external software/hardware
• C++ port — port primitive framework and chain system to nisps-core for RP2040 firmware
• Multi-track — multiple independent primitive chains running simultaneously with different step counts (polyrhythm)
• Markov chain primitive — transition-probability-based note selection
• L-system primitive — Lindenmayer system string rewriting for self-similar patterns
• Cellular automata primitive — 1D CA rules (e.g., Rule 30) generating trigger patterns

Technical Notes

Port-Ready JS Conventions

To facilitate future C++ porting of the primitive layer:

• No closures in primitive process functions — all state is explicit
• Use typed arrays (Float32Array) for parameter vectors where possible
• Primitives are pure functions with explicit state in/out
• Seeded PRNG (not Math.random()) for deterministic replay
• All time values in seconds (AudioContext convention), not milliseconds

The orchestration layer (event bus, clock, UI) is JS-only and not expected to port.

PRNG

Use a seedable PRNG (e.g., mulberry32 or xoshiro128) so that:

• Freeze-as-algorithm can replay identical sequences from seed
• Different primitives in a chain get independent PRNG streams (derived from a master seed)
• Deterministic behavior aids debugging and reproducibility

Performance Budget

The chain evaluates on param change, not per tick. The clock merely steps through the pre-computed pattern description. Per-tick cost is minimal: read the next step from the pattern, schedule the event. MLP inference (~<1ms) only runs when input changes.

Chain re-evaluation (all 8 primitives) happens when the MLP output changes. At ~60fps input update rate, this means ~16ms budget per evaluation. Each primitive is simple math, so 8 primitives is well within budget even on mobile.