4fd0b39d26
Major revisions addressing architectural gaps: - Fix data flow: clock drives everything (top of diagram) - Add fixed MLP + param mapping layer to handle dynamic chain sizing - Introduce symbolic chain evaluation (pattern descriptions, not concrete values) - Categorize primitives: generator, processor, timing modifier, converter - Generator combination modes (additive/multiplicative, user-configurable) - Swing and Ratchet are now timing modifiers on the pattern description - Remove Scale Quantizer from projection layer (Interval Lock is the sole quantizer) - Clarify mode integration: ShapeSeq replaces particle viz in synth mode - Build on existing imlJoy/imlHand dual-instance pattern - Add Open Design Questions section for deferred decisions - Port-ready applies to primitive layer only, not orchestration
533 lines
28 KiB
Markdown
533 lines
28 KiB
Markdown
# ShapeSeq — NISPS Generative Sequencing System
|
||
|
||
## Overview
|
||
|
||
ShapeSeq is a generative sequencing system for the NISPS playground where interactive ML (via the NISPS MLP engine) controls **parameters of algorithmic sequencing primitives** rather than raw note data. The user shapes sequences by navigating a learned parameter space with a joystick or hand tracking, can freeze sequences they like, then selectively re-expose specific parameters for further ML-driven exploration.
|
||
|
||
ShapeSeq replaces the existing placeholder arpeggiator. In synth output mode, the ShapeSeq UI (circular step visualizer + chain builder + param sliders) replaces the flow-field particle system.
|
||
|
||
## Core Architecture
|
||
|
||
### Design Principles
|
||
|
||
1. **MLP outputs are abstract [0,1] values** — musical meaning is applied downstream by the primitive chain and its symbolic processing
|
||
2. **Separate NISPS instances** for timbre control and sequence control, following the existing `imlJoy`/`imlHand` dual-instance pattern in `a-app.js`. Architecture supports future unification into a single instance
|
||
3. **Port-ready JS** — no closures in hot paths, explicit state, data structures that map cleanly to C++ for future RP2040 firmware porting. Note: the event bus and clock orchestration are JS-only concerns and not expected to port directly; the primitives themselves are the portable layer
|
||
4. **Modular primitives** — small, combinable algorithmic building blocks that generate musical patterns from continuous parameters
|
||
5. **Symbolic chain** — primitives compose as transforms over a pattern *description*, not concrete values. Each primitive takes the previous "pattern-generating machine" specification and produces a new one. The chain is evaluated once per loop (or on param change) to produce a complete pattern, which the clock then steps through
|
||
|
||
### System Diagram
|
||
|
||
```
|
||
┌──────────────────┐
|
||
│ Clock Engine │ ← drives everything
|
||
│ (AudioContext) │
|
||
└──────┬───────────┘
|
||
│ tick
|
||
▼
|
||
┌──────────────────┐ ┌─────────────────┐
|
||
│ Sequencer Core │◄────│ Input Router │
|
||
│ (orchestrator) │ │ (configurable) │
|
||
└──┬───────────┬───┘ └──┬──────────┬───┘
|
||
│ │ │ │
|
||
│ query │ query ┌────▼───┐ ┌───▼────────┐
|
||
│ pattern │ MLP │NISPS │ │ NISPS MLP │
|
||
│ │ │(timbre)│ │ (sequence) │
|
||
│ │ └───┬────┘ └───┬────────┘
|
||
│ │ │ │
|
||
│ ┌──────▼────────┐ │ ┌─────▼──────────┐
|
||
│ │ Param Mapping │ │ │ Param Mapping │
|
||
│ │ (16 MLP outs │ │ │ (16 MLP outs → │
|
||
│ │ → N prim │ │ │ 126 synth │
|
||
│ │ params) │ │ │ params) │
|
||
│ └──────┬────────┘ │ └─────┬──────────┘
|
||
│ │ │ │
|
||
│ ┌──────▼────────┐ │ ┌─────▼──────────┐
|
||
│ │Delta Controller│ │ │ Synth Param Map │
|
||
│ │(frozen+deltas)│ │ └─────┬──────────┘
|
||
│ └──────┬────────┘ │ │
|
||
│ │ │ │
|
||
│ ┌──────▼────────┐ │ │
|
||
│ │Primitive Chain │ │ │
|
||
│ │(symbolic eval) │ │ │
|
||
│ └──────┬────────┘ │ │
|
||
│ │ │ │
|
||
┌──────▼───────────▼───┐ │ │
|
||
│ Namespaced Event Bus │ │ │
|
||
│ seq.* ml.* ui.* │ │ │
|
||
└──┬───────────────┬───┘ │ │
|
||
│ │ │ │
|
||
┌──────▼───────┐ ┌────▼───────────▼────────▼──┐
|
||
│ Circular Viz │ │ C15 Synth │
|
||
│ + Chain UI │ │ (noteOn/Off + params) │
|
||
└──────────────┘ └────────────────────────────┘
|
||
```
|
||
|
||
### Fixed MLP + Param Mapping Layer
|
||
|
||
The `WasmIML` creates an MLP with a **fixed output count** at construction time — it cannot be resized. Since the primitive chain is dynamic (users add/remove primitives, changing total param count), the MLP cannot output directly to primitive params.
|
||
|
||
**Solution:** The sequence MLP always outputs a fixed number of values (e.g., 16). A **param mapping layer** fans these 16 outputs to however many primitive params the current chain requires. This is the same pattern used by the timbre MLP (which maps to 126 synth params via `param-map.js`).
|
||
|
||
The mapping can be:
|
||
- **Automatic** (default): outputs are distributed across primitive params in chain order. If there are 30 primitive params and 16 MLP outputs, each output influences ~2 params via interpolation.
|
||
- Future: configurable user-defined mapping.
|
||
|
||
> **Design note:** The fixed-16-output approach is the simplest starting point. If experimentation reveals that 16 is too few (or too many), the MLP can be reconstructed with a different size — this is a one-time setup cost, not a per-frame cost. The mapping layer insulates the rest of the system from this choice. Revisit if the mapping layer becomes a bottleneck for expressiveness.
|
||
|
||
### Namespaced Event Bus
|
||
|
||
A pub/sub event system with namespaced channels:
|
||
|
||
| Namespace | Events | Purpose |
|
||
|-----------|--------|---------|
|
||
| `seq.*` | `seq.step`, `seq.noteOn`, `seq.noteOff`, `seq.paramChange`, `seq.loopStart` | Musical output from sequencer to synth and visualizer |
|
||
| `ml.*` | `ml.trained`, `ml.frozen`, `ml.unfrozen`, `ml.deltaUpdate` | ML state changes |
|
||
| `ui.*` | `ui.paramSelect`, `ui.chainEdit`, `ui.presetLoad`, `ui.freezeToggle` | User actions |
|
||
|
||
All events carry a timestamp (AudioContext.currentTime for `seq.*`, performance.now() for others).
|
||
|
||
Note: the event bus is a JS-only orchestration concern (string-namespaced pub/sub). It does not need to be port-ready — the portable layer is the primitives themselves.
|
||
|
||
### Input Routing Matrix
|
||
|
||
A configurable routing layer that maps any input source to either NISPS instance's inputs. Builds on the existing `imlJoy`/`imlHand` switching pattern in `a-app.js`.
|
||
|
||
**Input sources:**
|
||
- Joystick X, Y (2 values)
|
||
- Hand tracking features (14 values)
|
||
- Gamepad axes (variable)
|
||
|
||
**Routing targets:**
|
||
- Timbre NISPS input 0, input 1
|
||
- Sequence NISPS input 0, input 1
|
||
|
||
Default: joystick → timbre NISPS, hand tracking features 0+1 → sequence NISPS. User-configurable via UI.
|
||
|
||
## Sequencing Primitives
|
||
|
||
### Primitive Categories
|
||
|
||
Primitives are categorized by their role in the chain:
|
||
|
||
| Category | Role | Examples |
|
||
|----------|------|----------|
|
||
| **Generator** | Produces data from params alone (no input required) | Euclidean, Density Morph, Pitch Walker |
|
||
| **Processor** | Transforms incoming data | Probability Gate, Velocity Shaper |
|
||
| **Timing Modifier** | Modulates the timing of events in the pattern description | Swing/Groove, Ratchet |
|
||
| **Converter** | Changes data type (e.g., continuous → discrete) | Interval Lock |
|
||
|
||
**Generator combination rule:** When multiple generators appear in the same chain, their outputs combine according to the chain's **combination mode** (user-configurable in real time):
|
||
- **Additive** (OR) — triggers from any generator fire. Pitch/velocity values are averaged where multiple generators contribute.
|
||
- **Multiplicative** (AND) — only steps where ALL generators agree will fire. Creates sparser, more selective patterns.
|
||
|
||
### Symbolic Chain Evaluation
|
||
|
||
Primitives do NOT process concrete note data step-by-step. Instead, each primitive takes the previous **pattern description** (a symbolic representation of the entire sequence) and produces a new one. The complete chain is evaluated to produce a full pattern, which the clock then steps through.
|
||
|
||
This means:
|
||
- **Timing modifiers** (Swing, Ratchet) work by annotating the pattern description with timing offsets and subdivisions *before* any concrete scheduling happens
|
||
- The clock reads the finalized pattern description and schedules all events (including ratchet subdivisions and swing offsets) using AudioContext.currentTime
|
||
- Re-evaluation happens when params change (MLP output updates, user edits), not on every tick
|
||
|
||
**Pattern description structure:**
|
||
```javascript
|
||
// The symbolic output of the chain — a complete loop description
|
||
{
|
||
steps: [
|
||
{
|
||
trigger: true, // whether this step fires
|
||
pitch: 0.72, // [0,1] abstract pitch (pre-quantization)
|
||
velocity: 0.85, // [0,1]
|
||
accent: false, // accent flag
|
||
timeOffset: 0.0, // swing offset in fractions of a step (-0.5 to +0.5)
|
||
subdivisions: 1, // ratchet: 1 = normal, 2-4 = subdivided
|
||
},
|
||
// ... one per step
|
||
],
|
||
stepCount: 8,
|
||
metadata: { ... } // chain-specific info for visualization
|
||
}
|
||
```
|
||
|
||
### Primitive Definitions
|
||
|
||
Each primitive is a pure function (or stateful generator with explicit state) that accepts a parameter object and produces typed output. All parameters are normalized [0,1].
|
||
|
||
#### 1. Euclidean Rhythm Generator
|
||
|
||
**Category:** Generator
|
||
**Params:** `steps` (int, from continuous), `pulses` (int), `rotation` (int)
|
||
**Output:** trigger pattern (boolean array)
|
||
**Stateless:** yes
|
||
|
||
Generates Bjorklund-distributed trigger patterns. The continuous [0,1] params are projected to integer ranges based on current step count.
|
||
|
||
#### 2. Probability Gate
|
||
|
||
**Category:** Processor
|
||
**Params:** `density` [0,1], `accentProbability` [0,1]
|
||
**Input:** trigger pattern
|
||
**Output:** filtered trigger pattern with accent flags
|
||
**Stateless:** yes (per-step coin flip using seeded PRNG)
|
||
|
||
Each incoming trigger survives with probability `density`. Surviving triggers receive accent flag with probability `accentProbability`.
|
||
|
||
#### 3. Pitch Walker
|
||
|
||
**Category:** Generator
|
||
**Params:** `stepSize` [0,1], `directionBias` [0,1] (0.5=unbiased), `gravity` [0,1] (pull toward center), `range` [0,1]
|
||
**Output:** pitch values [0,1] per triggered step
|
||
**Stateful:** yes — maintains current position in pitch space
|
||
|
||
Constrained random walk that generates melodic contour. `gravity` pulls the walk toward center (0.5), preventing it from getting stuck at extremes. State includes current position and PRNG state.
|
||
|
||
#### 4. Ratchet
|
||
|
||
**Category:** Timing Modifier
|
||
**Params:** `maxDivision` [0,1] (maps to 1-4 subdivisions), `probability` [0,1]
|
||
**Input:** pattern description with triggers
|
||
**Output:** pattern description with `subdivisions` field set per step
|
||
**Stateless:** yes (per-step coin flip)
|
||
|
||
Annotates triggered steps with subdivision counts. The clock engine reads `subdivisions` and schedules rapid repeats within the step's time window. Division count determined by `maxDivision`, applied probabilistically.
|
||
|
||
#### 5. Swing / Groove
|
||
|
||
**Category:** Timing Modifier
|
||
**Params:** `swingAmount` [0,1] (0=straight, 1=full swing), `swingGrid` [0,1] (which subdivisions swing)
|
||
**Input:** pattern description
|
||
**Output:** pattern description with `timeOffset` field set per step
|
||
**Stateless:** yes
|
||
|
||
Annotates alternating steps with timing offsets. At `swingAmount=0.67` this produces classic 2:1 shuffle. `swingGrid` controls whether swing applies to 8th notes, 16th notes, or triplets. The clock engine reads `timeOffset` and adjusts scheduling accordingly.
|
||
|
||
#### 6. Density Morph
|
||
|
||
**Category:** Generator
|
||
**Params:** `density` [0,1], `clustering` [0,1] (0=spread evenly, 1=clustered together)
|
||
**Output:** trigger pattern
|
||
**Stateless:** yes
|
||
|
||
Alternative to Euclidean — generates trigger patterns with controllable density and spatial distribution. At high clustering, triggers group together creating bursts; at low clustering, triggers spread evenly.
|
||
|
||
#### 7. Interval Lock (Scale Quantizer)
|
||
|
||
**Category:** Converter
|
||
**Params:** `root` [0,1] (maps to 0-11 semitones), `mode` [0,1] (maps to scale index), `octaveRange` [0,1] (1-4 octaves)
|
||
**Input:** pitch values [0,1]
|
||
**Output:** MIDI note numbers
|
||
**Stateless:** yes
|
||
|
||
The sole pitch quantization mechanism — the projection layer does NOT duplicate this. All pitch quantization goes through Interval Lock.
|
||
|
||
Available scales: chromatic, major, natural minor, harmonic minor, pentatonic major, pentatonic minor, blues, dorian, mixolydian, whole tone, diminished.
|
||
|
||
#### 8. Velocity Shaper
|
||
|
||
**Category:** Processor
|
||
**Params:** `curveType` [0,1] (maps to: flat, accent-every-N, crescendo, decrescendo, random), `depth` [0,1], `phase` [0,1]
|
||
**Input:** trigger pattern with step indices
|
||
**Output:** velocity values [0,1] per step
|
||
**Stateless:** yes
|
||
|
||
Applies cyclic velocity patterns. `phase` rotates the pattern, `depth` controls contrast between quiet and loud.
|
||
|
||
### Primitive Interface
|
||
|
||
```javascript
|
||
// Port-ready: explicit state, no closures
|
||
class Primitive {
|
||
constructor(name, paramSchema, category) { ... }
|
||
|
||
// category: 'generator' | 'processor' | 'timing' | 'converter'
|
||
|
||
// paramSchema: array of { name, min, max, default, boundary }
|
||
// boundary: 'clamp' | 'wrap' | 'scaled'
|
||
// For 'scaled': operates within ±scaledRange of frozen value
|
||
|
||
// Symbolic processing: transforms a pattern description
|
||
process(params, patternDesc, state, rng) → { patternDesc, nextState }
|
||
|
||
// State management for freeze
|
||
getState() → serializable object
|
||
setState(state) → void
|
||
getSeed() → number
|
||
setSeed(seed) → void
|
||
}
|
||
```
|
||
|
||
### Chain Connection Modes
|
||
|
||
Three configurable modes for how primitives connect in a chain:
|
||
|
||
**1. Sequential Pipeline** — each primitive transforms the pattern description in order. Generators create initial data, processors/timing modifiers transform it. If multiple generators appear, they combine according to the generator combination mode (additive/multiplicative, configurable in real time).
|
||
|
||
**2. Parallel + Merge** — each primitive runs independently and produces a pattern description. Descriptions merge (OR for triggers in additive mode, AND in multiplicative mode; average for continuous values). Order doesn't matter.
|
||
|
||
**3. Typed Routing** — primitives connect via typed ports. A primitive's output connects to the next primitive that accepts that type. Multiple primitives can feed the same type (merged). Most flexible, most complex.
|
||
|
||
The chain connection mode is a global setting (per-chain), configurable via UI. Default: sequential pipeline.
|
||
|
||
**Generator combination mode** (additive/multiplicative) is an independent setting, also configurable in real time via UI.
|
||
|
||
## Delta Control System
|
||
|
||
### Freeze Workflow
|
||
|
||
1. User plays with NISPS, finds a sequence they like
|
||
2. User activates **freeze** — all current parameter values are captured
|
||
3. User selects specific parameters to **re-expose** (mark as "live"):
|
||
- Click/tap parameters in the UI
|
||
- Or use hand tracking: point with index finger, pinch gesture to toggle
|
||
4. Live parameters receive **deltas** from NISPS MLP output
|
||
5. Frozen parameters hold their captured values
|
||
|
||
### Freeze Modes
|
||
|
||
**Freeze as Algorithm** — captures parameter values + PRNG seed. Stateful primitives (pitch walker) will replay identically. Re-exposing params resumes algorithmic generation with delta-modified params.
|
||
|
||
**Freeze as Pattern** — captures the realized note pattern (snapshot of all step events for one full loop). The primitive chain is bypassed; the sequencer loops the frozen pattern directly. Re-exposing params requires switching back to algorithm mode.
|
||
|
||
User chooses freeze mode via UI toggle.
|
||
|
||
### Delta Boundary Behavior
|
||
|
||
Each parameter declares its boundary behavior:
|
||
|
||
| Behavior | Description | Good for |
|
||
|----------|-------------|----------|
|
||
| `clamp` | Delta result clamped to [0,1] | Velocity, volume, most continuous params |
|
||
| `wrap` | Values wrap around (1.1 → 0.1) | Rotation, phase, cyclic params |
|
||
| `scaled` | Delta operates within ±`scaledRange` centered on frozen value | Precision control near a sweet spot |
|
||
|
||
Parameters also declare a `scaledRange` (default 0.3) for the scaled boundary mode. Example: frozen value 0.8 with scaledRange 0.3 → effective range [0.5, 1.0], clamped at boundaries.
|
||
|
||
## Clock Engine
|
||
|
||
Replaces setTimeout-based arpeggiator with AudioContext-scheduled timing.
|
||
|
||
```javascript
|
||
class ClockEngine {
|
||
constructor(audioContext) { ... }
|
||
|
||
// Properties
|
||
bpm // beats per minute
|
||
stepCount // total steps in sequence
|
||
|
||
// Lookahead scheduling: schedule events slightly ahead of time
|
||
// using AudioContext.currentTime for sample-accurate timing
|
||
start() → void
|
||
stop() → void
|
||
setTempo(bpm) → void
|
||
|
||
// The clock reads the finalized pattern description and schedules
|
||
// all events, including:
|
||
// - timeOffset per step (swing)
|
||
// - subdivisions per step (ratchet)
|
||
// - accent flags (velocity scaling)
|
||
schedulePattern(patternDesc) → void
|
||
|
||
// Callback: called with { stepIndex, time, velocity, pitch, isSubdivision }
|
||
onEvent(callback) → void
|
||
}
|
||
```
|
||
|
||
The clock uses the standard Web Audio lookahead pattern:
|
||
- A setInterval (~25ms) checks if any events need scheduling in the next ~100ms
|
||
- Events are scheduled using AudioContext.currentTime for sample-accurate timing
|
||
- This decouples visual updates (requestAnimationFrame) from audio timing
|
||
- The clock handles ratchet subdivisions and swing offsets natively by reading the pattern description's per-step `subdivisions` and `timeOffset` fields
|
||
|
||
## Projection Layer
|
||
|
||
A composable chain of transform functions that convert raw [0,1] primitive outputs into final musical values. Each transform is a small, independent module.
|
||
|
||
Note: pitch quantization is handled by the **Interval Lock** primitive, not the projection layer. The projection layer handles non-pitch transforms only.
|
||
|
||
### Available Transforms
|
||
|
||
| Transform | Input | Output | Params |
|
||
|-----------|-------|--------|--------|
|
||
| Velocity Curve | [0,1] | [0,1] | curve shape (linear, exponential, S-curve) |
|
||
| Gate Threshold | [0,1] | boolean | threshold value |
|
||
| Range Map | [0,1] | [min,max] | min, max |
|
||
| Octave Folder | MIDI note | MIDI note | target octave range |
|
||
| Stutter Map | [0,1] | repeat count | max repeats |
|
||
|
||
Transforms snap together: output type of one must match input type of next. The chain is validated on construction.
|
||
|
||
### Projection Presets
|
||
|
||
Pre-built chain configurations for common use cases:
|
||
- **Expressive** — velocity curve (exponential) → range map (48-84)
|
||
- **Percussive** — gate threshold (0.5) → velocity curve (accent)
|
||
- **Full Range** — range map (24-96) → velocity curve (linear)
|
||
|
||
Users can edit any preset or build custom chains.
|
||
|
||
## UI Design
|
||
|
||
### Mode Integration
|
||
|
||
ShapeSeq activates in **synth output mode**. When synth mode is active:
|
||
- The flow-field particle visualizer is replaced by the ShapeSeq UI (circular step viz + chain builder + param sliders)
|
||
- The timbre NISPS instance continues to control C15 synth parameters as before
|
||
- The sequence NISPS instance drives the ShapeSeq primitive chain
|
||
|
||
In visual output mode, the particle system remains unchanged.
|
||
|
||
### Circular Step Visualizer
|
||
|
||
Steps arranged in a circle with even angular spacing (7 steps = heptagon, 13 steps = 13-gon, etc.). No grid overlay — the ear provides rhythmic context.
|
||
|
||
**Visual elements:**
|
||
- Each step is a node on the circle
|
||
- Active/triggered steps glow or pulse
|
||
- Current playback position shown with a rotating indicator
|
||
- Pitch mapped to node distance from center (low=outer, high=inner)
|
||
- Velocity mapped to node size
|
||
- Accents shown with brighter color
|
||
|
||
**Interaction:**
|
||
- Tap a step to solo/mute it
|
||
- Long-press for step detail (all params for that step)
|
||
|
||
### Primitive Chain Builder
|
||
|
||
Vertical stack layout (like a guitar pedalboard):
|
||
- Each primitive is a card with its name and key params visible
|
||
- Drag to reorder
|
||
- Swipe left to delete
|
||
- "+" button at bottom opens primitive palette
|
||
- Each card expandable to show all params as sliders
|
||
- Params marked as "live" (NISPS-controlled) get a distinct visual indicator (e.g., pulsing border)
|
||
|
||
### Parameter Selection (for freeze/re-expose)
|
||
|
||
Two input modes:
|
||
1. **Mouse/touch** — tap a parameter slider to toggle it between frozen (dimmed) and live (highlighted)
|
||
2. **Hand tracking** — point index finger at parameter, pinch to toggle. Visual cursor follows index finger tip.
|
||
|
||
Live params show their current NISPS delta as a secondary indicator on the slider.
|
||
|
||
### Layout
|
||
|
||
```
|
||
┌──────────────────────────────────┐
|
||
│ Circular Step Viz │
|
||
│ (upper half of screen) │
|
||
│ │
|
||
│ ○ ○ │
|
||
│ ○ ○ │
|
||
│ ○ ▶ ○ │
|
||
│ ○ ○ │
|
||
│ ○ ○ │
|
||
│ │
|
||
├──────────────────────────────────┤
|
||
│ Chain Builder (scrollable stack) │
|
||
│ ┌──────────────────────────────┐ │
|
||
│ │ Euclidean [steps][pulses] │ │
|
||
│ │ [rotation] │ │
|
||
│ ├──────────────────────────────┤ │
|
||
│ │ Prob Gate [density][accent] │ │
|
||
│ ├──────────────────────────────┤ │
|
||
│ │ Pitch Walk [step][bias] │ │
|
||
│ ├──────────────────────────────┤ │
|
||
│ │ [ + Add Primitive ] │ │
|
||
│ └──────────────────────────────┘ │
|
||
├──────────────────────────────────┤
|
||
│ [▶ Play] [❄ Freeze] [Chain:Seq] │
|
||
│ [+×] BPM:120 Steps:8 Gen:Add │
|
||
└──────────────────────────────────┘
|
||
```
|
||
|
||
## Phased Implementation Plan
|
||
|
||
### Phase 1 — Foundation (MVP)
|
||
|
||
**Goal:** All 8 primitives working, chain builder, basic UI, NISPS control. Full architecture with minimal polish.
|
||
|
||
1. **Event bus** — namespaced pub/sub system
|
||
2. **Clock engine** — AudioContext-based precise timing with pattern description scheduling (handles swing offsets + ratchet subdivisions)
|
||
3. **Primitive framework** — base class, param schema, category system, state management, symbolic pattern description structure
|
||
4. **All 8 primitives** — implement each with their param schemas and categories
|
||
5. **Sequential chain** — primitives connected in sequence (pipeline mode only), with additive/multiplicative generator combination mode
|
||
6. **Param mapping layer** — fixed 16-output MLP → N primitive params, automatic distribution
|
||
7. **Projection layer** — velocity curve + gate threshold (2 transforms minimum, no scale quantizer — that's Interval Lock)
|
||
8. **Sequence NISPS instance** — second WasmIML (16 outputs), following existing `imlJoy`/`imlHand` pattern
|
||
9. **Basic circular viz** — step circle with playback indicator
|
||
10. **Basic chain UI** — vertical stack with sliders, add/remove primitives, generator combo mode toggle
|
||
11. **Bridge integration** — sequence events → C15 noteOn/noteOff via event bus
|
||
12. **Replace arpeggiator** — remove old arpeggiator, ShapeSeq takes over in synth mode
|
||
|
||
### Phase 2 — Freeze & Delta Control
|
||
|
||
1. **Freeze system** — capture params + seed, capture pattern snapshot
|
||
2. **Parameter selection UI** — click to toggle frozen/live
|
||
3. **Delta controller** — applies MLP deltas to live params with boundary config
|
||
4. **Hand tracking param select** — pinch gesture to toggle params
|
||
5. **Freeze mode toggle** — algorithm vs pattern freeze
|
||
|
||
### Phase 3 — Advanced Chain & Routing
|
||
|
||
1. **Parallel + merge chain mode**
|
||
2. **Typed routing chain mode**
|
||
3. **Input routing matrix** — configurable input → NISPS instance mapping
|
||
4. **Projection chain builder** — user-editable transform chains
|
||
5. **Projection presets**
|
||
|
||
### Phase 4 — Polish & Expansion
|
||
|
||
1. **Preset chains** — pre-built primitive combinations for common genres
|
||
2. **Save/load** — persist chain configs, frozen sequences, NISPS state
|
||
3. **Unified NISPS mode** — single MLP controlling both timbre + sequence
|
||
4. **Additional primitives** as discovered through experimentation
|
||
5. **Freeform lasso param selection** (see Future Work)
|
||
6. **Per-track variable step counts** (polyrhythm)
|
||
|
||
## Open Design Questions
|
||
|
||
These are deliberately deferred decisions to be revisited after experimentation:
|
||
|
||
1. **MLP output count:** Is 16 the right number for the sequence MLP? Too few may limit expressiveness; too many may make learning harder. The param mapping layer insulates the system, so this can be changed without architectural impact.
|
||
2. **Param mapping strategy:** Automatic distribution is the starting point. Should users be able to manually wire MLP outputs to specific primitive params? This could enable more intentional control but adds UI complexity.
|
||
3. **Generator combination modes:** Additive and multiplicative are the starting pair. Other modes worth exploring: weighted average, priority (first generator wins), XOR (one or the other but not both).
|
||
4. **Chain evaluation frequency:** Currently re-evaluates when params change. Should there be an option for per-loop re-evaluation (stateful primitives produce different patterns each loop)?
|
||
|
||
## Future Work
|
||
|
||
- **Freeform lasso selection** — draw/lasso over the step visualization to select params spatially. Intuitive but complex to implement. (Backlog issue: meml-hud)
|
||
- **MIDI clock sync** — accept external MIDI clock for hardware sync
|
||
- **OSC output** — route sequencer events via OSC for external software/hardware
|
||
- **C++ port** — port primitive framework and chain system to nisps-core for RP2040 firmware
|
||
- **Multi-track** — multiple independent primitive chains running simultaneously with different step counts (polyrhythm)
|
||
- **Markov chain primitive** — transition-probability-based note selection
|
||
- **L-system primitive** — Lindenmayer system string rewriting for self-similar patterns
|
||
- **Cellular automata primitive** — 1D CA rules (e.g., Rule 30) generating trigger patterns
|
||
|
||
## Technical Notes
|
||
|
||
### Port-Ready JS Conventions
|
||
|
||
To facilitate future C++ porting of the **primitive layer**:
|
||
- No closures in primitive process functions — all state is explicit
|
||
- Use typed arrays (Float32Array) for parameter vectors where possible
|
||
- Primitives are pure functions with explicit state in/out
|
||
- Seeded PRNG (not Math.random()) for deterministic replay
|
||
- All time values in seconds (AudioContext convention), not milliseconds
|
||
|
||
The orchestration layer (event bus, clock, UI) is JS-only and not expected to port.
|
||
|
||
### PRNG
|
||
|
||
Use a seedable PRNG (e.g., mulberry32 or xoshiro128) so that:
|
||
- Freeze-as-algorithm can replay identical sequences from seed
|
||
- Different primitives in a chain get independent PRNG streams (derived from a master seed)
|
||
- Deterministic behavior aids debugging and reproducibility
|
||
|
||
### Performance Budget
|
||
|
||
The chain evaluates on param change, not per tick. The clock merely steps through the pre-computed pattern description. Per-tick cost is minimal: read the next step from the pattern, schedule the event. MLP inference (~<1ms) only runs when input changes.
|
||
|
||
Chain re-evaluation (all 8 primitives) happens when the MLP output changes. At ~60fps input update rate, this means ~16ms budget per evaluation. Each primitive is simple math, so 8 primitives is well within budget even on mobile.
|