ShapeSeq is a generative sequencing system for the NISPS playground where interactive ML (via the NISPS MLP engine) controls **parameters of algorithmic sequencing primitives** rather than raw note data. The user shapes sequences by navigating a learned parameter space with a joystick or hand tracking, can freeze sequences they like, then selectively re-expose specific parameters for further ML-driven exploration.
1.**MLP outputs are abstract [0,1] values** — musical meaning is applied downstream by the primitive chain and its symbolic processing
2.**Separate NISPS instances** for timbre control and sequence control, following the existing `imlJoy`/`imlHand` dual-instance pattern in `a-app.js`. Architecture supports future unification into a single instance
3.**Port-ready JS** — no closures in hot paths, explicit state, data structures that map cleanly to C++ for future RP2040 firmware porting. Note: the event bus and clock orchestration are JS-only concerns and not expected to port directly; the primitives themselves are the portable layer
5.**Symbolic chain** — primitives compose as transforms over a pattern *description*, not concrete values. Each primitive takes the previous "pattern-generating machine" specification and produces a new one. The chain is evaluated once per loop (or on param change) to produce a complete pattern, which the clock then steps through
The `WasmIML` creates an MLP with a **fixed output count** at construction time — it cannot be resized. Since the primitive chain is dynamic (users add/remove primitives, changing total param count), the MLP cannot output directly to primitive params.
**Solution:** The sequence MLP always outputs a fixed number of values (e.g., 16). A **param mapping layer** fans these 16 outputs to however many primitive params the current chain requires. This is the same pattern used by the timbre MLP (which maps to 126 synth params via `param-map.js`).
The mapping can be:
- **Automatic** (default): outputs are distributed across primitive params in chain order. If there are 30 primitive params and 16 MLP outputs, each output influences ~2 params via interpolation.
- Future: configurable user-defined mapping.
> **Design note:** The fixed-16-output approach is the simplest starting point. If experimentation reveals that 16 is too few (or too many), the MLP can be reconstructed with a different size — this is a one-time setup cost, not a per-frame cost. The mapping layer insulates the rest of the system from this choice. Revisit if the mapping layer becomes a bottleneck for expressiveness.
Note: the event bus is a JS-only orchestration concern (string-namespaced pub/sub). It does not need to be port-ready — the portable layer is the primitives themselves.
A configurable routing layer that maps any input source to either NISPS instance's inputs. Builds on the existing `imlJoy`/`imlHand` switching pattern in `a-app.js`.
Primitives are categorized by their role in the chain:
| Category | Role | Examples |
|----------|------|----------|
| **Generator** | Produces data from params alone (no input required) | Euclidean, Density Morph, Pitch Walker |
| **Processor** | Transforms incoming data | Probability Gate, Velocity Shaper |
| **Timing Modifier** | Modulates the timing of events in the pattern description | Swing/Groove, Ratchet |
| **Converter** | Changes data type (e.g., continuous → discrete) | Interval Lock |
**Generator combination rule:** When multiple generators appear in the same chain, their outputs combine according to the chain's **combination mode** (user-configurable in real time):
- **Additive** (OR) — triggers from any generator fire. Pitch/velocity values are averaged where multiple generators contribute.
- **Multiplicative** (AND) — only steps where ALL generators agree will fire. Creates sparser, more selective patterns.
### Symbolic Chain Evaluation
Primitives do NOT process concrete note data step-by-step. Instead, each primitive takes the previous **pattern description** (a symbolic representation of the entire sequence) and produces a new one. The complete chain is evaluated to produce a full pattern, which the clock then steps through.
This means:
- **Timing modifiers** (Swing, Ratchet) work by annotating the pattern description with timing offsets and subdivisions *before* any concrete scheduling happens
- The clock reads the finalized pattern description and schedules all events (including ratchet subdivisions and swing offsets) using AudioContext.currentTime
- Re-evaluation happens when params change (MLP output updates, user edits), not on every tick
**Pattern description structure:**
```javascript
// The symbolic output of the chain — a complete loop description
Each primitive is a pure function (or stateful generator with explicit state) that accepts a parameter object and produces typed output. All parameters are normalized [0,1].
**Stateful:** yes — maintains current position in pitch space
Constrained random walk that generates melodic contour. `gravity` pulls the walk toward center (0.5), preventing it from getting stuck at extremes. State includes current position and PRNG state.
Annotates triggered steps with subdivision counts. The clock engine reads `subdivisions` and schedules rapid repeats within the step's time window. Division count determined by `maxDivision`, applied probabilistically.
Annotates alternating steps with timing offsets. At `swingAmount=0.67` this produces classic 2:1 shuffle. `swingGrid` controls whether swing applies to 8th notes, 16th notes, or triplets. The clock engine reads `timeOffset` and adjusts scheduling accordingly.
Alternative to Euclidean — generates trigger patterns with controllable density and spatial distribution. At high clustering, triggers group together creating bursts; at low clustering, triggers spread evenly.
**1. Sequential Pipeline** — each primitive transforms the pattern description in order. Generators create initial data, processors/timing modifiers transform it. If multiple generators appear, they combine according to the generator combination mode (additive/multiplicative, configurable in real time).
**2. Parallel + Merge** — each primitive runs independently and produces a pattern description. Descriptions merge (OR for triggers in additive mode, AND in multiplicative mode; average for continuous values). Order doesn't matter.
**3. Typed Routing** — primitives connect via typed ports. A primitive's output connects to the next primitive that accepts that type. Multiple primitives can feed the same type (merged). Most flexible, most complex.
The chain connection mode is a global setting (per-chain), configurable via UI. Default: sequential pipeline.
1. User plays with NISPS, finds a sequence they like
2. User activates **freeze** — all current parameter values are captured
3. User selects specific parameters to **re-expose** (mark as "live"):
- Click/tap parameters in the UI
- Or use hand tracking: point with index finger, pinch gesture to toggle
4. Live parameters receive **deltas** from NISPS MLP output
5. Frozen parameters hold their captured values
### Freeze Modes
**Freeze as Algorithm** — captures parameter values + PRNG seed. Stateful primitives (pitch walker) will replay identically. Re-exposing params resumes algorithmic generation with delta-modified params.
**Freeze as Pattern** — captures the realized note pattern (snapshot of all step events for one full loop). The primitive chain is bypassed; the sequencer loops the frozen pattern directly. Re-exposing params requires switching back to algorithm mode.
User chooses freeze mode via UI toggle.
### Delta Boundary Behavior
Each parameter declares its boundary behavior:
| Behavior | Description | Good for |
|----------|-------------|----------|
| `clamp` | Delta result clamped to [0,1] | Velocity, volume, most continuous params |
| `scaled` | Delta operates within ±`scaledRange` centered on frozen value | Precision control near a sweet spot |
Parameters also declare a `scaledRange` (default 0.3) for the scaled boundary mode. Example: frozen value 0.8 with scaledRange 0.3 → effective range [0.5, 1.0], clamped at boundaries.
## Clock Engine
Replaces setTimeout-based arpeggiator with AudioContext-scheduled timing.
```javascript
class ClockEngine {
constructor(audioContext) { ... }
// Properties
bpm // beats per minute
stepCount // total steps in sequence
// Lookahead scheduling: schedule events slightly ahead of time
// using AudioContext.currentTime for sample-accurate timing
- The clock handles ratchet subdivisions and swing offsets natively by reading the pattern description's per-step `subdivisions` and `timeOffset` fields
A composable chain of transform functions that convert raw [0,1] primitive outputs into final musical values. Each transform is a small, independent module.
Note: pitch quantization is handled by the **Interval Lock** primitive, not the projection layer. The projection layer handles non-pitch transforms only.
Steps arranged in a circle with even angular spacing (7 steps = heptagon, 13 steps = 13-gon, etc.). No grid overlay — the ear provides rhythmic context.
**Visual elements:**
- Each step is a node on the circle
- Active/triggered steps glow or pulse
- Current playback position shown with a rotating indicator
- Pitch mapped to node distance from center (low=outer, high=inner)
- Velocity mapped to node size
- Accents shown with brighter color
**Interaction:**
- Tap a step to solo/mute it
- Long-press for step detail (all params for that step)
### Primitive Chain Builder
Vertical stack layout (like a guitar pedalboard):
- Each primitive is a card with its name and key params visible
- Drag to reorder
- Swipe left to delete
- "+" button at bottom opens primitive palette
- Each card expandable to show all params as sliders
- Params marked as "live" (NISPS-controlled) get a distinct visual indicator (e.g., pulsing border)
### Parameter Selection (for freeze/re-expose)
Two input modes:
1.**Mouse/touch** — tap a parameter slider to toggle it between frozen (dimmed) and live (highlighted)
2.**Hand tracking** — point index finger at parameter, pinch to toggle. Visual cursor follows index finger tip.
Live params show their current NISPS delta as a secondary indicator on the slider.
These are deliberately deferred decisions to be revisited after experimentation:
1.**MLP output count:** Is 16 the right number for the sequence MLP? Too few may limit expressiveness; too many may make learning harder. The param mapping layer insulates the system, so this can be changed without architectural impact.
2.**Param mapping strategy:** Automatic distribution is the starting point. Should users be able to manually wire MLP outputs to specific primitive params? This could enable more intentional control but adds UI complexity.
3.**Generator combination modes:** Additive and multiplicative are the starting pair. Other modes worth exploring: weighted average, priority (first generator wins), XOR (one or the other but not both).
4.**Chain evaluation frequency:** Currently re-evaluates when params change. Should there be an option for per-loop re-evaluation (stateful primitives produce different patterns each loop)?
- **Freeform lasso selection** — draw/lasso over the step visualization to select params spatially. Intuitive but complex to implement. (Backlog issue: meml-hud)
The chain evaluates on param change, not per tick. The clock merely steps through the pre-computed pattern description. Per-tick cost is minimal: read the next step from the pattern, schedule the event. MLP inference (~<1ms)onlyrunswheninputchanges.
Chain re-evaluation (all 8 primitives) happens when the MLP output changes. At ~60fps input update rate, this means ~16ms budget per evaluation. Each primitive is simple math, so 8 primitives is well within budget even on mobile.
