w1n5t0n/memlnaut-nisps
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs(playground): revise ShapeSeq spec after fresh-eyes review

Browse source 
Major revisions addressing architectural gaps:
- Fix data flow: clock drives everything (top of diagram)
- Add fixed MLP + param mapping layer to handle dynamic chain sizing
- Introduce symbolic chain evaluation (pattern descriptions, not concrete values)
- Categorize primitives: generator, processor, timing modifier, converter
- Generator combination modes (additive/multiplicative, user-configurable)
- Swing and Ratchet are now timing modifiers on the pattern description
- Remove Scale Quantizer from projection layer (Interval Lock is the sole quantizer)
- Clarify mode integration: ShapeSeq replaces particle viz in synth mode
- Build on existing imlJoy/imlHand dual-instance pattern
- Add Open Design Questions section for deferred decisions
- Port-ready applies to primitive layer only, not orchestration
This commit is contained in:
w1n5t0n 2026-03-24 00:30:43 +02:00
parent 1a0f2c4d1c
commit 4fd0b39d26
1 changed files with 202 additions and 103 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

305
playground/SPEC-shapeseq.md
View file

@ -4,62 +4,77 @@
ShapeSeq is a generative sequencing system for the NISPS playground where interactive ML (via the NISPS MLP engine) controls **parameters of algorithmic sequencing primitives** rather than raw note data. The user shapes sequences by navigating a learned parameter space with a joystick or hand tracking, can freeze sequences they like, then selectively re-expose specific parameters for further ML-driven exploration. ShapeSeq is a generative sequencing system for the NISPS playground where interactive ML (via the NISPS MLP engine) controls **parameters of algorithmic sequencing primitives** rather than raw note data. The user shapes sequences by navigating a learned parameter space with a joystick or hand tracking, can freeze sequences they like, then selectively re-expose specific parameters for further ML-driven exploration.
ShapeSeq replaces the existing placeholder arpeggiator. ShapeSeq replaces the existing placeholder arpeggiator. In synth output mode, the ShapeSeq UI (circular step visualizer + chain builder + param sliders) replaces the flow-field particle system.
## Core Architecture ## Core Architecture
### Design Principles ### Design Principles
1. **MLP outputs are abstract [0,1] values** — musical meaning is applied downstream by a configurable projection layer 1. **MLP outputs are abstract [0,1] values** — musical meaning is applied downstream by the primitive chain and its symbolic processing
2. **Separate NISPS instances** for timbre control and sequence control, with architecture supporting future unification into a single instance 2. **Separate NISPS instances** for timbre control and sequence control, following the existing `imlJoy`/`imlHand` dual-instance pattern in `a-app.js`. Architecture supports future unification into a single instance
3. **Port-ready JS** — no closures in hot paths, explicit state, data structures that map cleanly to C++ for future RP2040 firmware porting 3. **Port-ready JS** — no closures in hot paths, explicit state, data structures that map cleanly to C++ for future RP2040 firmware porting. Note: the event bus and clock orchestration are JS-only concerns and not expected to port directly; the primitives themselves are the portable layer
4. **Modular primitives** — small, combinable algorithmic building blocks that generate musical patterns from continuous parameters 4. **Modular primitives** — small, combinable algorithmic building blocks that generate musical patterns from continuous parameters
5. **Symbolic chain** — primitives compose as transforms over a pattern *description*, not concrete values. Each primitive takes the previous "pattern-generating machine" specification and produces a new one. The chain is evaluated once per loop (or on param change) to produce a complete pattern, which the clock then steps through
### System Diagram ### System Diagram
``` ```
┌─────────────────┐ ┌──────────────────┐
│ Input Router │ │ Clock Engine │ ← drives everything
│ (configurable) │ │ (AudioContext) │
└──┬──────────┬───┘ └──────┬───────────┘
│ │ │ tick
┌────────▼──┐ ┌───▼────────┐ ▼
│ NISPS MLP │ │ NISPS MLP │ ┌──────────────────┐ ┌─────────────────┐
│ (timbre) │ │ (sequence) │ │ Sequencer Core │◄────│ Input Router │
└────────┬───┘ └───┬────────┘ │ (orchestrator) │ │ (configurable) │
│ │ └──┬───────────┬───┘ └──┬──────────┬───┘
┌────────▼──┐ ┌───▼────────────────────┐ │ │ │ │
│ Synth │ │ Delta Controller │ │ query │ query ┌────▼───┐ ┌───▼────────┐
│ Param Map │ │ (frozen vals + deltas) │ │ pattern │ MLP │NISPS │ │ NISPS MLP │
└────────┬───┘ └───┬────────────────────┘ │ │ │(timbre)│ │ (sequence) │
│ │ │ │ └───┬────┘ └───┬────────┘
│ ┌────▼──────────────┐ │ │ │ │
│ │ Primitive Chain │ │ ┌──────▼────────┐ │ ┌─────▼──────────┐
│ │ Euclid→ProbGate→… │ │ │ Param Mapping │ │ │ Param Mapping │
│ └────┬──────────────┘ │ │ (16 MLP outs │ │ │ (16 MLP outs → │
│ │ │ │ → N prim │ │ │ 126 synth │
│ ┌────▼──────────────┐ │ │ params) │ │ │ params) │
│ │ Projection Layer │ │ └──────┬────────┘ │ └─────┬──────────┘
│ │ (scale quant, etc) │ │ │ │ │
│ └────┬──────────────┘ │ ┌──────▼────────┐ │ ┌─────▼──────────┐
│ │ │ │Delta Controller│ │ │ Synth Param Map │
│ ┌────▼──────────┐ │ │(frozen+deltas)│ │ └─────┬──────────┘
│ │ Clock Engine │ │ └──────┬────────┘ │ │
│ │ (AudioContext) │ │ │ │ │
│ └────┬──────────┘ │ ┌──────▼────────┐ │ │
│ │ │ │Primitive Chain │ │ │
┌────▼──────────▼────┐ │ │(symbolic eval) │ │ │
│ Namespaced Event Bus│ │ └──────┬────────┘ │ │
│ seq.* ml.* ui.* │ │ │ │ │
└────────┬───────────┘ ┌──────▼───────────▼───┐ │ │
│ │ Namespaced Event Bus │ │ │
┌────────▼───────┐ │ seq.* ml.* ui.* │ │ │
│ C15 Synth │ └──┬───────────────┬───┘ │ │
│ (noteOn/Off + │ │ │ │ │
│ param changes)│ ┌──────▼───────┐ ┌────▼───────────▼────────▼──┐
└────────────────┘ │ Circular Viz │ │ C15 Synth │
│ + Chain UI │ │ (noteOn/Off + params) │
└──────────────┘ └────────────────────────────┘
``` ```
### Fixed MLP + Param Mapping Layer
The `WasmIML` creates an MLP with a **fixed output count** at construction time — it cannot be resized. Since the primitive chain is dynamic (users add/remove primitives, changing total param count), the MLP cannot output directly to primitive params.
**Solution:** The sequence MLP always outputs a fixed number of values (e.g., 16). A **param mapping layer** fans these 16 outputs to however many primitive params the current chain requires. This is the same pattern used by the timbre MLP (which maps to 126 synth params via `param-map.js`).
The mapping can be:
- **Automatic** (default): outputs are distributed across primitive params in chain order. If there are 30 primitive params and 16 MLP outputs, each output influences ~2 params via interpolation.
- Future: configurable user-defined mapping.
> **Design note:** The fixed-16-output approach is the simplest starting point. If experimentation reveals that 16 is too few (or too many), the MLP can be reconstructed with a different size — this is a one-time setup cost, not a per-frame cost. The mapping layer insulates the rest of the system from this choice. Revisit if the mapping layer becomes a bottleneck for expressiveness.
### Namespaced Event Bus ### Namespaced Event Bus
A pub/sub event system with namespaced channels: A pub/sub event system with namespaced channels:
@ -72,9 +87,11 @@ A pub/sub event system with namespaced channels:
All events carry a timestamp (AudioContext.currentTime for `seq.*`, performance.now() for others). All events carry a timestamp (AudioContext.currentTime for `seq.*`, performance.now() for others).
Note: the event bus is a JS-only orchestration concern (string-namespaced pub/sub). It does not need to be port-ready — the portable layer is the primitives themselves.
### Input Routing Matrix ### Input Routing Matrix
A configurable routing layer that maps any input source to either NISPS instance's inputs: A configurable routing layer that maps any input source to either NISPS instance's inputs. Builds on the existing `imlJoy`/`imlHand` switching pattern in `a-app.js`.
**Input sources:** **Input sources:**
- Joystick X, Y (2 values) - Joystick X, Y (2 values)
@ -89,75 +106,129 @@ Default: joystick → timbre NISPS, hand tracking features 0+1 → sequence NISP
## Sequencing Primitives ## Sequencing Primitives
### Primitive Categories
Primitives are categorized by their role in the chain:
| Category | Role | Examples |
|----------|------|----------|
| **Generator** | Produces data from params alone (no input required) | Euclidean, Density Morph, Pitch Walker |
| **Processor** | Transforms incoming data | Probability Gate, Velocity Shaper |
| **Timing Modifier** | Modulates the timing of events in the pattern description | Swing/Groove, Ratchet |
| **Converter** | Changes data type (e.g., continuous → discrete) | Interval Lock |
**Generator combination rule:** When multiple generators appear in the same chain, their outputs combine according to the chain's **combination mode** (user-configurable in real time):
- **Additive** (OR) — triggers from any generator fire. Pitch/velocity values are averaged where multiple generators contribute.
- **Multiplicative** (AND) — only steps where ALL generators agree will fire. Creates sparser, more selective patterns.
### Symbolic Chain Evaluation
Primitives do NOT process concrete note data step-by-step. Instead, each primitive takes the previous **pattern description** (a symbolic representation of the entire sequence) and produces a new one. The complete chain is evaluated to produce a full pattern, which the clock then steps through.
This means:
- **Timing modifiers** (Swing, Ratchet) work by annotating the pattern description with timing offsets and subdivisions *before* any concrete scheduling happens
- The clock reads the finalized pattern description and schedules all events (including ratchet subdivisions and swing offsets) using AudioContext.currentTime
- Re-evaluation happens when params change (MLP output updates, user edits), not on every tick
**Pattern description structure:**
```javascript
// The symbolic output of the chain — a complete loop description
{
steps: [
{
trigger: true, // whether this step fires
pitch: 0.72, // [0,1] abstract pitch (pre-quantization)
velocity: 0.85, // [0,1]
accent: false, // accent flag
timeOffset: 0.0, // swing offset in fractions of a step (-0.5 to +0.5)
subdivisions: 1, // ratchet: 1 = normal, 2-4 = subdivided
},
// ... one per step
],
stepCount: 8,
metadata: { ... } // chain-specific info for visualization
}
```
### Primitive Definitions
Each primitive is a pure function (or stateful generator with explicit state) that accepts a parameter object and produces typed output. All parameters are normalized [0,1]. Each primitive is a pure function (or stateful generator with explicit state) that accepts a parameter object and produces typed output. All parameters are normalized [0,1].
### 1. Euclidean Rhythm Generator #### 1. Euclidean Rhythm Generator
**Category:** Generator
**Params:** `steps` (int, from continuous), `pulses` (int), `rotation` (int) **Params:** `steps` (int, from continuous), `pulses` (int), `rotation` (int)
**Output type:** trigger pattern (boolean array) **Output:** trigger pattern (boolean array)
**Stateless:** yes **Stateless:** yes
Generates Bjorklund-distributed trigger patterns. The continuous [0,1] params are projected to integer ranges based on current step count. Generates Bjorklund-distributed trigger patterns. The continuous [0,1] params are projected to integer ranges based on current step count.
### 2. Probability Gate #### 2. Probability Gate
**Category:** Processor
**Params:** `density` [0,1], `accentProbability` [0,1] **Params:** `density` [0,1], `accentProbability` [0,1]
**Input type:** trigger pattern **Input:** trigger pattern
**Output type:** filtered trigger pattern with accent flags **Output:** filtered trigger pattern with accent flags
**Stateless:** yes (per-step coin flip using seeded PRNG) **Stateless:** yes (per-step coin flip using seeded PRNG)
Each incoming trigger survives with probability `density`. Surviving triggers receive accent flag with probability `accentProbability`. Each incoming trigger survives with probability `density`. Surviving triggers receive accent flag with probability `accentProbability`.
### 3. Pitch Walker #### 3. Pitch Walker
**Category:** Generator
**Params:** `stepSize` [0,1], `directionBias` [0,1] (0.5=unbiased), `gravity` [0,1] (pull toward center), `range` [0,1] **Params:** `stepSize` [0,1], `directionBias` [0,1] (0.5=unbiased), `gravity` [0,1] (pull toward center), `range` [0,1]
**Input type:** trigger pattern **Output:** pitch values [0,1] per triggered step
**Output type:** pitch values [0,1] per triggered step
**Stateful:** yes — maintains current position in pitch space **Stateful:** yes — maintains current position in pitch space
Constrained random walk that generates melodic contour. `gravity` pulls the walk toward center (0.5), preventing it from getting stuck at extremes. State includes current position and PRNG state. Constrained random walk that generates melodic contour. `gravity` pulls the walk toward center (0.5), preventing it from getting stuck at extremes. State includes current position and PRNG state.
### 4. Ratchet #### 4. Ratchet
**Category:** Timing Modifier
**Params:** `maxDivision` [0,1] (maps to 1-4 subdivisions), `probability` [0,1] **Params:** `maxDivision` [0,1] (maps to 1-4 subdivisions), `probability` [0,1]
**Input type:** trigger pattern **Input:** pattern description with triggers
**Output type:** trigger pattern with subdivision timing offsets **Output:** pattern description with `subdivisions` field set per step
**Stateless:** yes (per-step coin flip) **Stateless:** yes (per-step coin flip)
Subdivides triggered steps into rapid repeats. Division count determined by `maxDivision`, applied probabilistically. Annotates triggered steps with subdivision counts. The clock engine reads `subdivisions` and schedules rapid repeats within the step's time window. Division count determined by `maxDivision`, applied probabilistically.
### 5. Swing / Groove #### 5. Swing / Groove
**Category:** Timing Modifier
**Params:** `swingAmount` [0,1] (0=straight, 1=full swing), `swingGrid` [0,1] (which subdivisions swing) **Params:** `swingAmount` [0,1] (0=straight, 1=full swing), `swingGrid` [0,1] (which subdivisions swing)
**Input type:** timing information **Input:** pattern description
**Output type:** modified timing offsets **Output:** pattern description with `timeOffset` field set per step
**Stateless:** yes **Stateless:** yes
Shifts timing of alternating steps. At `swingAmount=0.67` this produces classic 2:1 shuffle. `swingGrid` controls whether swing applies to 8th notes, 16th notes, or triplets. Annotates alternating steps with timing offsets. At `swingAmount=0.67` this produces classic 2:1 shuffle. `swingGrid` controls whether swing applies to 8th notes, 16th notes, or triplets. The clock engine reads `timeOffset` and adjusts scheduling accordingly.
### 6. Density Morph #### 6. Density Morph
**Category:** Generator
**Params:** `density` [0,1], `clustering` [0,1] (0=spread evenly, 1=clustered together) **Params:** `density` [0,1], `clustering` [0,1] (0=spread evenly, 1=clustered together)
**Input type:** step count **Output:** trigger pattern
**Output type:** trigger pattern
**Stateless:** yes **Stateless:** yes
Alternative to Euclidean — generates trigger patterns with controllable density and spatial distribution. At high clustering, triggers group together creating bursts; at low clustering, triggers spread evenly. Alternative to Euclidean — generates trigger patterns with controllable density and spatial distribution. At high clustering, triggers group together creating bursts; at low clustering, triggers spread evenly.
### 7. Interval Lock (Scale Quantizer) #### 7. Interval Lock (Scale Quantizer)
**Category:** Converter
**Params:** `root` [0,1] (maps to 0-11 semitones), `mode` [0,1] (maps to scale index), `octaveRange` [0,1] (1-4 octaves) **Params:** `root` [0,1] (maps to 0-11 semitones), `mode` [0,1] (maps to scale index), `octaveRange` [0,1] (1-4 octaves)
**Input type:** pitch values [0,1] **Input:** pitch values [0,1]
**Output type:** MIDI note numbers **Output:** MIDI note numbers
**Stateless:** yes **Stateless:** yes
The sole pitch quantization mechanism — the projection layer does NOT duplicate this. All pitch quantization goes through Interval Lock.
Available scales: chromatic, major, natural minor, harmonic minor, pentatonic major, pentatonic minor, blues, dorian, mixolydian, whole tone, diminished. Available scales: chromatic, major, natural minor, harmonic minor, pentatonic major, pentatonic minor, blues, dorian, mixolydian, whole tone, diminished.
### 8. Velocity Shaper #### 8. Velocity Shaper
**Category:** Processor
**Params:** `curveType` [0,1] (maps to: flat, accent-every-N, crescendo, decrescendo, random), `depth` [0,1], `phase` [0,1] **Params:** `curveType` [0,1] (maps to: flat, accent-every-N, crescendo, decrescendo, random), `depth` [0,1], `phase` [0,1]
**Input type:** trigger pattern with step indices **Input:** trigger pattern with step indices
**Output type:** velocity values [0,1] per step **Output:** velocity values [0,1] per step
**Stateless:** yes **Stateless:** yes
Applies cyclic velocity patterns. `phase` rotates the pattern, `depth` controls contrast between quiet and loud. Applies cyclic velocity patterns. `phase` rotates the pattern, `depth` controls contrast between quiet and loud.
@ -167,14 +238,16 @@ Applies cyclic velocity patterns. `phase` rotates the pattern, `depth` controls
```javascript ```javascript
// Port-ready: explicit state, no closures // Port-ready: explicit state, no closures
class Primitive { class Primitive {
constructor(name, paramSchema) { ... } constructor(name, paramSchema, category) { ... }
// category: 'generator' | 'processor' | 'timing' | 'converter'
// paramSchema: array of { name, min, max, default, boundary } // paramSchema: array of { name, min, max, default, boundary }
// boundary: 'clamp' | 'wrap' | 'scaled' // boundary: 'clamp' | 'wrap' | 'scaled'
// For 'scaled': operates within ±scaledRange of frozen value // For 'scaled': operates within ±scaledRange of frozen value
// Pure processing function // Symbolic processing: transforms a pattern description
process(params, input, state, stepCount, rng) → { output, nextState } process(params, patternDesc, state, rng) → { patternDesc, nextState }
// State management for freeze // State management for freeze
getState() → serializable object getState() → serializable object
@ -188,14 +261,16 @@ class Primitive {
Three configurable modes for how primitives connect in a chain: Three configurable modes for how primitives connect in a chain:
**1. Sequential Pipeline** — each primitive transforms the previous output. Order matters. Signal flows left to right. **1. Sequential Pipeline** — each primitive transforms the pattern description in order. Generators create initial data, processors/timing modifiers transform it. If multiple generators appear, they combine according to the generator combination mode (additive/multiplicative, configurable in real time).
**2. Parallel + Merge** — each primitive runs independently, outputs merged (OR for triggers, average for continuous values). Order doesn't matter. **2. Parallel + Merge** — each primitive runs independently and produces a pattern description. Descriptions merge (OR for triggers in additive mode, AND in multiplicative mode; average for continuous values). Order doesn't matter.
**3. Typed Routing** — primitives connect via typed ports. A primitive's output connects to the next primitive that accepts that type. Multiple primitives can feed the same type (merged). Most flexible, most complex. **3. Typed Routing** — primitives connect via typed ports. A primitive's output connects to the next primitive that accepts that type. Multiple primitives can feed the same type (merged). Most flexible, most complex.
The chain connection mode is a global setting (per-chain), configurable via UI. Default: sequential pipeline. The chain connection mode is a global setting (per-chain), configurable via UI. Default: sequential pipeline.
**Generator combination mode** (additive/multiplicative) is an independent setting, also configurable in real time via UI.
## Delta Control System ## Delta Control System
### Freeze Workflow ### Freeze Workflow
@ -239,7 +314,6 @@ class ClockEngine {
// Properties // Properties
bpm // beats per minute bpm // beats per minute
stepCount // total steps in sequence stepCount // total steps in sequence
subdivision // ticks per step (for ratchet/swing resolution)
// Lookahead scheduling: schedule events slightly ahead of time // Lookahead scheduling: schedule events slightly ahead of time
// using AudioContext.currentTime for sample-accurate timing // using AudioContext.currentTime for sample-accurate timing
@ -247,8 +321,15 @@ class ClockEngine {
stop() → void stop() → void
setTempo(bpm) → void setTempo(bpm) → void
// Callback: called with { stepIndex, time, isAccent } // The clock reads the finalized pattern description and schedules
onStep(callback) → void // all events, including:
// - timeOffset per step (swing)
// - subdivisions per step (ratchet)
// - accent flags (velocity scaling)
schedulePattern(patternDesc) → void
// Callback: called with { stepIndex, time, velocity, pitch, isSubdivision }
onEvent(callback) → void
} }
``` ```
@ -256,16 +337,18 @@ The clock uses the standard Web Audio lookahead pattern:
- A setInterval (~25ms) checks if any events need scheduling in the next ~100ms - A setInterval (~25ms) checks if any events need scheduling in the next ~100ms
- Events are scheduled using AudioContext.currentTime for sample-accurate timing - Events are scheduled using AudioContext.currentTime for sample-accurate timing
- This decouples visual updates (requestAnimationFrame) from audio timing - This decouples visual updates (requestAnimationFrame) from audio timing
- The clock handles ratchet subdivisions and swing offsets natively by reading the pattern description's per-step `subdivisions` and `timeOffset` fields
## Projection Layer ## Projection Layer
A composable chain of transform functions that convert raw [0,1] MLP outputs into musical values. Each transform is a small, independent module. A composable chain of transform functions that convert raw [0,1] primitive outputs into final musical values. Each transform is a small, independent module.
Note: pitch quantization is handled by the **Interval Lock** primitive, not the projection layer. The projection layer handles non-pitch transforms only.
### Available Transforms ### Available Transforms
| Transform | Input | Output | Params | | Transform | Input | Output | Params |
|-----------|-------|--------|--------| |-----------|-------|--------|--------|
| Scale Quantizer | [0,1] | MIDI note | root, scale, octave range |
| Velocity Curve | [0,1] | [0,1] | curve shape (linear, exponential, S-curve) | | Velocity Curve | [0,1] | [0,1] | curve shape (linear, exponential, S-curve) |
| Gate Threshold | [0,1] | boolean | threshold value | | Gate Threshold | [0,1] | boolean | threshold value |
| Range Map | [0,1] | [min,max] | min, max | | Range Map | [0,1] | [min,max] | min, max |
@ -277,15 +360,23 @@ Transforms snap together: output type of one must match input type of next. The
### Projection Presets ### Projection Presets
Pre-built chain configurations for common use cases: Pre-built chain configurations for common use cases:
- **Melodic Minor** — scale quant (A minor) → octave fold (2 oct) → velocity curve (exponential) - **Expressive** — velocity curve (exponential) → range map (48-84)
- **Pentatonic Drift** — scale quant (pentatonic) → octave fold (3 oct) → velocity curve (S) - **Percussive** — gate threshold (0.5) → velocity curve (accent)
- **Chromatic Chaos** — range map (full MIDI) → velocity curve (linear) - **Full Range** — range map (24-96) → velocity curve (linear)
- **Rhythmic Only** — gate threshold (0.5) → velocity curve (accent)
Users can edit any preset or build custom chains. Users can edit any preset or build custom chains.
## UI Design ## UI Design
### Mode Integration
ShapeSeq activates in **synth output mode**. When synth mode is active:
- The flow-field particle visualizer is replaced by the ShapeSeq UI (circular step viz + chain builder + param sliders)
- The timbre NISPS instance continues to control C15 synth parameters as before
- The sequence NISPS instance drives the ShapeSeq primitive chain
In visual output mode, the particle system remains unchanged.
### Circular Step Visualizer ### Circular Step Visualizer
Steps arranged in a circle with even angular spacing (7 steps = heptagon, 13 steps = 13-gon, etc.). No grid overlay — the ear provides rhythmic context. Steps arranged in a circle with even angular spacing (7 steps = heptagon, 13 steps = 13-gon, etc.). No grid overlay — the ear provides rhythmic context.
@ -347,7 +438,7 @@ Live params show their current NISPS delta as a secondary indicator on the slide
│ └──────────────────────────────┘ │ │ └──────────────────────────────┘ │
├──────────────────────────────────┤ ├──────────────────────────────────┤
│ [▶ Play] [❄ Freeze] [Chain:Seq] │ │ [▶ Play] [❄ Freeze] [Chain:Seq] │
│ BPM: 120 Steps: 8 Scale: Cm │ │ [+×] BPM:120 Steps:8 Gen:Add │
└──────────────────────────────────┘ └──────────────────────────────────┘
``` ```
@ -358,16 +449,17 @@ Live params show their current NISPS delta as a secondary indicator on the slide
**Goal:** All 8 primitives working, chain builder, basic UI, NISPS control. Full architecture with minimal polish. **Goal:** All 8 primitives working, chain builder, basic UI, NISPS control. Full architecture with minimal polish.
1. **Event bus** — namespaced pub/sub system 1. **Event bus** — namespaced pub/sub system
2. **Clock engine** — AudioContext-based precise timing 2. **Clock engine** — AudioContext-based precise timing with pattern description scheduling (handles swing offsets + ratchet subdivisions)
3. **Primitive framework** — base class, param schema, state management 3. **Primitive framework** — base class, param schema, category system, state management, symbolic pattern description structure
4. **All 8 primitives** — implement each with their param schemas 4. **All 8 primitives** — implement each with their param schemas and categories
5. **Sequential chain** — primitives connected in sequence (pipeline mode only) 5. **Sequential chain** — primitives connected in sequence (pipeline mode only), with additive/multiplicative generator combination mode
6. **Projection layer** — scale quantizer + velocity curve (2 transforms minimum) 6. **Param mapping layer** — fixed 16-output MLP → N primitive params, automatic distribution
7. **Sequence NISPS instance** — second MLP controlling primitive chain params 7. **Projection layer** — velocity curve + gate threshold (2 transforms minimum, no scale quantizer — that's Interval Lock)
8. **Basic circular viz** — step circle with playback indicator 8. **Sequence NISPS instance** — second WasmIML (16 outputs), following existing `imlJoy`/`imlHand` pattern
9. **Basic chain UI** — vertical stack with sliders, add/remove primitives 9. **Basic circular viz** — step circle with playback indicator
10. **Bridge integration** — sequence events → C15 noteOn/noteOff via event bus 10. **Basic chain UI** — vertical stack with sliders, add/remove primitives, generator combo mode toggle
11. **Remove arpeggiator** — replace with ShapeSeq 11. **Bridge integration** — sequence events → C15 noteOn/noteOff via event bus
12. **Replace arpeggiator** — remove old arpeggiator, ShapeSeq takes over in synth mode
### Phase 2 — Freeze & Delta Control ### Phase 2 — Freeze & Delta Control
@ -394,9 +486,18 @@ Live params show their current NISPS delta as a secondary indicator on the slide
5. **Freeform lasso param selection** (see Future Work) 5. **Freeform lasso param selection** (see Future Work)
6. **Per-track variable step counts** (polyrhythm) 6. **Per-track variable step counts** (polyrhythm)
## Open Design Questions
These are deliberately deferred decisions to be revisited after experimentation:
1. **MLP output count:** Is 16 the right number for the sequence MLP? Too few may limit expressiveness; too many may make learning harder. The param mapping layer insulates the system, so this can be changed without architectural impact.
2. **Param mapping strategy:** Automatic distribution is the starting point. Should users be able to manually wire MLP outputs to specific primitive params? This could enable more intentional control but adds UI complexity.
3. **Generator combination modes:** Additive and multiplicative are the starting pair. Other modes worth exploring: weighted average, priority (first generator wins), XOR (one or the other but not both).
4. **Chain evaluation frequency:** Currently re-evaluates when params change. Should there be an option for per-loop re-evaluation (stateful primitives produce different patterns each loop)?
## Future Work ## Future Work
- **Freeform lasso selection** — draw/lasso over the step visualization to select params spatially. Intuitive but complex to implement. (Create bd backlog issue.) - **Freeform lasso selection** — draw/lasso over the step visualization to select params spatially. Intuitive but complex to implement. (Backlog issue: meml-hud)
- **MIDI clock sync** — accept external MIDI clock for hardware sync - **MIDI clock sync** — accept external MIDI clock for hardware sync
- **OSC output** — route sequencer events via OSC for external software/hardware - **OSC output** — route sequencer events via OSC for external software/hardware
- **C++ port** — port primitive framework and chain system to nisps-core for RP2040 firmware - **C++ port** — port primitive framework and chain system to nisps-core for RP2040 firmware
@ -409,13 +510,15 @@ Live params show their current NISPS delta as a secondary indicator on the slide
### Port-Ready JS Conventions ### Port-Ready JS Conventions
To facilitate future C++ porting: To facilitate future C++ porting of the **primitive layer**:
- No closures in primitive process functions — all state is explicit - No closures in primitive process functions — all state is explicit
- Use typed arrays (Float32Array) for parameter vectors where possible - Use typed arrays (Float32Array) for parameter vectors where possible
- Primitives are pure functions with explicit state in/out - Primitives are pure functions with explicit state in/out
- Seeded PRNG (not Math.random()) for deterministic replay - Seeded PRNG (not Math.random()) for deterministic replay
- All time values in seconds (AudioContext convention), not milliseconds - All time values in seconds (AudioContext convention), not milliseconds
The orchestration layer (event bus, clock, UI) is JS-only and not expected to port.
### PRNG ### PRNG
Use a seedable PRNG (e.g., mulberry32 or xoshiro128) so that: Use a seedable PRNG (e.g., mulberry32 or xoshiro128) so that:
@ -425,10 +528,6 @@ Use a seedable PRNG (e.g., mulberry32 or xoshiro128) so that:
### Performance Budget ### Performance Budget
The sequencer tick runs at most every ~15ms (at 250 BPM with 16th note subdivision). Each tick must: The chain evaluates on param change, not per tick. The clock merely steps through the pre-computed pattern description. Per-tick cost is minimal: read the next step from the pattern, schedule the event. MLP inference (~<1ms) only runs when input changes.
1. Query NISPS MLP (already fast — the MLP inference is <1ms)
2. Run primitive chain (8 primitives × simple math = negligible)
3. Apply projection transforms (simple lookups/math)
4. Emit events
Total budget per tick: ~5ms. This is comfortable even on mobile. Chain re-evaluation (all 8 primitives) happens when the MLP output changes. At ~60fps input update rate, this means ~16ms budget per evaluation. Each primitive is simple math, so 8 primitives is well within budget even on mobile.