Compare commits

...

105 commits

Author SHA1 Message Date
monkey-w1n5t0n
a6b8f87d32 fix(manifold): start push-away feedback on press
Some checks failed
CI / C++ tests + WASM + parity + lint (push) Has been cancelled
CI / Manifold typecheck + unit + e2e (push) Has been cancelled
CI / Firmware (RP2350, PlatformIO) (push) Has been cancelled
2026-07-25 17:43:52 +02:00
monkey-w1n5t0n
079720a572 test(manifold): align e2e with two-input boot shape 2026-07-25 17:22:13 +02:00
monkey-w1n5t0n
326deb9a17 tune(manifold): strengthen default geometric push 2026-07-25 17:18:44 +02:00
monkey-w1n5t0n
13013077d4 fix(manifold): complete input sizing and feedback coverage 2026-07-25 17:12:26 +02:00
monkey-w1n5t0n
856642a962 fix(manifold): smooth particle rendering 2026-07-25 17:11:24 +02:00
monkey-w1n5t0n
a1cd26ff68 feat(manifold): replay geometric dislikes over time 2026-07-25 16:14:35 +02:00
monkey-w1n5t0n
7190932191 fix(manifold): remove particle backend copy 2026-07-25 16:10:04 +02:00
monkey-w1n5t0n
3cf11ef9cf refactor(manifold): trim remaining explanatory copy 2026-07-25 16:08:18 +02:00
monkey-w1n5t0n
ae256ecfbd refactor(manifold): remove redundant UI copy 2026-07-25 15:58:44 +02:00
monkey-w1n5t0n
32c4825a69 fix(manifold): let outputs list fill drawer 2026-07-25 15:47:55 +02:00
monkey-w1n5t0n
7e308a4e62 feat(manifold): improve output controls and particle input 2026-07-25 15:45:40 +02:00
monkey-w1n5t0n
b5891a6ab9 refactor(manifold): streamline output cards 2026-07-25 15:36:50 +02:00
monkey-w1n5t0n
5ab4d18523 refactor(manifold): move MIDI controls onto output cards 2026-07-25 15:34:06 +02:00
monkey-w1n5t0n
5ed84cbf00 fix(manifold): make particle outputs interactive 2026-07-25 15:30:35 +02:00
monkey-w1n5t0n
a7b6230b8b fix(manifold): preserve output card names on delete 2026-07-25 15:28:26 +02:00
monkey-w1n5t0n
4db0f498eb feat(manifold): show live model architecture in docks 2026-07-25 15:24:50 +02:00
monkey-w1n5t0n
491820ac79 fix(manifold): preserve slider overrides until input moves 2026-07-25 15:20:11 +02:00
monkey-w1n5t0n
0010097d01 feat(manifold): edit I/O as identity-aware cards 2026-07-25 15:16:37 +02:00
monkey-w1n5t0n
3879d57b4c fix(manifold): clamp temporary output slider overrides 2026-07-25 15:16:01 +02:00
monkey-w1n5t0n
6e71ad7d35 fix(manifold): broaden randomisation by default 2026-07-25 15:07:35 +02:00
monkey-w1n5t0n
0bd65917c4 fix(manifold): share output range slider 2026-07-25 15:05:32 +02:00
monkey-w1n5t0n
766493fcdc fix(manifold): show active output count in drawer 2026-07-25 14:45:04 +02:00
monkey-w1n5t0n
f75f683f40 fix(manifold): sync output sliders with backend count 2026-07-25 13:55:19 +02:00
monkey-w1n5t0n
ec3118004d fix(ml): re-base the geometric dislike on upstream e291192 — delete the taper
geo_push.hpp and replay.hpp cited memllib @ 0a541cc. upstream/main pins
e291192, where the same code had been deliberately redesigned — and because
InterfaceRL was not in the tree (fixed one commit ago), we carried the
superseded version for months. Three changes, all upstream's:

  kGeometricPushScale   0.5 -> 1.0    (InterfaceRL.hpp:409)
  kNegLRBase            0.5 -> 1.5    (InterfaceRL.hpp:410)
  /(1+len) taper        deleted       (InterfaceRL.tpp:724)

Upstream's own comment on the taper: "a 'no' should clearly move the mapping
away even from a sound already far from the liked region (the taper used to
kill exactly that case)". The direction is already a unit vector, so the
taper only ever shrank the push for exactly the sounds a user is most likely
to be rejecting.

Cold start is folded into the same path. Upstream's useRandom is
`!havePositives || len <= 1e-4`: with nothing liked yet there is no centroid
to push away from, so every dim goes in a random direction. Ours instead
kept the older 0a541cc fallback — train AWAY from the heard action at a
NEGATIVE lr — which was inert whenever the heard action equalled the net's
own output, i.e. in the common case. One path now, and a "no" moves the
mapping before any likes exist (ml_bench E1: 0 -> 2.3e-3). The
GeometricColdStart action is still reported so callers keep their "like a
few sounds first" prompt; only the training changed.

Measured (ml_bench, one dislike at a point):
  A4  0.0157 -> 0.0533 at-point displacement (3.4x), so end to end across
      this and the RMSProp fix: 5.3e-5 -> 5.3e-2, ~1000x. The gap to the
      legacy Diffuse design closes from ~4100x to ~4.2x.
  D1  effective_lr 4.7e-4 -> 1.5e-3; 10 presses now reach 0.34, 100 reach
      the full intended push.
  A5  compounding 0.87 -> 0.96 (a second press at the same spot is no
      longer noticeably weaker than the first).
  A7  damage_ratio essentially unchanged (0.87-1.57) — the collateral
      damage to protected positives scales with the push and is NOT
      addressed here; it is the negative-feedback design question.

NOT adopted, deliberately: upstream's per-tick batch retraining over all
live negatives, and its fixed kDislikeLifetimeMs=2500 in place of our
proportional decay. Both need something the core does not have — a per-tick
call site and a millisecond clock inside nisps/ml — so they change
FeedbackControllerCore's interface rather than its constants. Recorded in
ALIGNMENT's deferred-debt entry alongside the existing one-press-one-step
divergence, and filed as its own task.

test_mlp_geo_dislike.cpp: the taper test now pins its ABSENCE (equal
displacement near and far), the cold-start test pins movement where it used
to pin inertness, and a new test covers the random-direction branch.

ALIGNMENT defect 6b resolved. Gates: build-cpp-tests 139 tests / ctest 4/4,
parity-check PASS (WASM rebuilt), lint-cpp clean, firmware slpworkshop
SUCCESS.
2026-07-25 11:22:15 +02:00
monkey-w1n5t0n
1f0eecfe78 docs(firmware): vendor InterfaceRL as read-only reference
InterfaceRL.{hpp,cpp,tpp} + InterfaceRLFileFormat.hpp are the upstream
reference implementation of the entire NISPS feedback subsystem —
nisps/ml/{geo_push,replay,feedback,jolt,ou_noise}.hpp are all ports of it,
several still carrying `// upstream InterfaceRL.hpp:NNN` line references.
The Phase-4 vendoring dropped examples/ because nothing compiled it. That
was correct for the build and wrong for the codebase: with the source of
truth out of tree, upstream redesigned the geometric dislike (deleted the
/(1+len) taper, doubled kGeometricPushScale, tripled the negative-LR base,
moved to batch training over all negatives every tick) and we did not notice
for months.

Copied verbatim from memllib @ e291192 — the same commit the rest of the
vendored tree pins — into lib/memllib/reference/, which sits OUTSIDE src/
and is therefore never compiled: PlatformIO's LDF only recursively builds an
Arduino-format library's src/ folder. Verified: slpworkshop still builds
(RAM 28.8%, flash 2.1%).

reference/README.md states the two rules (never compiled, never edited — a
divergence from upstream is a recorded decision, not an edit here) and
VENDORED.md's "what was dropped" section now tells the truth.

Resolves ALIGNMENT defect 6c.
2026-07-25 11:14:35 +02:00
monkey-w1n5t0n
f57cddc278 fix(ml): port RMSProp — ported learning rates were landing in SGD
Upstream memlp (github.com/MusicallyEmbodiedML/memlp @ ea777502, the commit
upstream/main pins) applies gradients with RMSProp everywhere: Layer.h:239
ApplyAccumulatedGradients, the m_sq_grad_avg running average at Layer.h:601,
StaticMLP.h:268. nisps/ml/training.hpp shipped SGD only and filed the
difference as an optimiser-choice research question. It was not one.

RMSProp divides each step by the running gradient magnitude, so an upstream
lr is a NORMALISED step; under SGD the same number multiplies the raw
gradient. Every learning rate ported from upstream therefore landed in an
optimiser that reads it differently — most visibly feedback.hpp's
`geo_lr_ = 0.001f  // upstream InterfaceRL.hpp:312`, an RMSProp LR pasted
into a single SGD step.

rmsprop_step() ports Layer.h:239 exactly: clip at +/-10, sq = min(0.9*sq +
0.1*g^2, 1e6), adj = min(lr/(sqrt(sq)+1e-6), 1.0), w -= adj*g. The
adjusted-LR clamp stays one-sided as upstream's std::min is, so the negative
lr used by train_targets' "train away from this target" path behaves as it
does upstream. The per-weight squared-gradient average is new persistent
state and lives in the storage policies (FixedStorage arrays /
DynamicStorage arena) so nisps/ stays allocation-free and the firmware's
zero-heap contract holds. It is optimiser state, not model state: excluded
from weight_count()/get_weights()/set_weights(), matching upstream, and
cleared by MLPCore::reset_optimizer_state() (upstream ResetOptimizerState).
draw_weights() deliberately does NOT clear it — upstream's DrawWeights
doesn't either.

Measured with tests/cpp/ml_bench.cpp:
  D1  one geometric dislike moves the mapping 1.6e-2, up from 5.3e-5 (~295x),
      and repeated presses now CONVERGE on the intended 0.5 push (0.12 at 10,
      0.56 at 100) instead of creeping linearly forever.
  A4  geometric-vs-Diffuse gap narrows from ~4100x to ~14x in one press.
  U4  the upstream-LR positive path actually trains now (range_util 0.71 at
      100 ticks/gesture, was 0.016 — it was inert under SGD).
Not fixed by this, and now tracked as ALIGNMENT defect 6d: the dose
asymmetry. lurch_max is still ~1.08 against a [0,1] output range.

Golden vector stages 2 and 3 re-captured; stages 0 and 1 are pre-training
and did not move, which is the cross-check that only the update rule
changed. manifold/public/nisps.wasm rebuilt so parity-check compares like
with like — it FAILED at up to 5e-2 against the stale artifact and PASSES at
2.4e-7 against a fresh one. parity-check.sh only builds the WASM when it is
missing, never when it is stale; noted in MAP.md and filed separately.

ALIGNMENT defect 6 resolved (moved to Recently resolved); 6b's optimiser
cross-reference updated; new defect 6d for the positive-training dose.

Gates: build-cpp-tests 138 tests / ctest 4/4, parity-check PASS, lint-cpp
clean, manifold typecheck clean.
2026-07-25 11:11:23 +02:00
monkey-w1n5t0n
1603ea798e test(ml): behavioural benchmark + 20 invariants for the control mapping
NISPS is a controller, not a synth: the object of study is the mapping
f: control-space -> parameter-space and how a musician's gestures deform
it. Loss measures fit to points the user dictated, which is the one thing
they never experience. So this measures geometry and gesture-response.

  tests/cpp/ml_bench.cpp          61 scenarios, REPORTS never asserts (same
                                  discipline as engine_bench.cpp). Shape-
                                  agnostic via MLPCore<DynamicStorage>
                                  (--shape, default 2,16,16,16,8), seeded
                                  RNG throughout, branch points replayed
                                  from scratch rather than snapshotted.
  tests/cpp/test_ml_behaviour.cpp 20 asserting invariants, wired into
                                  nisps_core_tests.
  scripts/bench-ml.sh             native + WASM from one source; --compare,
                                  --sweep-shape, --smoke, --scenario, --seed.

Documents two contracts that fail SILENTLY (both now pinned by tests):
a thumbs-up must call BOTH mlp.add_example() and fb.store_positive(),
since dislike_geometric k-NNs the replay buffer and not the MLP dataset;
and placed_output() is valid only while state == Placing, after which an
empty span whose l2() is 0 scores a broken lifecycle as a perfect place.

ALIGNMENT defect 6 re-ranked (SGD-vs-RMSProp is not a research axis - it
silently invalidated every ported hyperparameter) and split into 6b (the
geometric dislike was ported from a superseded upstream design) and 6c
(InterfaceRL, the reference impl, is not in the tree).

Gates: build-cpp-tests (138 tests, ctest 4/4), parity-check PASS,
lint-cpp clean, bench-ml.sh --smoke runs end to end.
2026-07-25 11:02:24 +02:00
monkey-w1n5t0n
60584606a8 docs(specs): P5 architectural specs — 5a, 5b, 5c, 5d
Plan §6 says each P5 item is spec-first with its own session. These are the
specs; no implementation is authorised by them and none was written.

  plans/mode-layer-reunification.md   5a — ALIGNMENT defect 1, the largest
                                      architectural gap. Storage-policies the
                                      ModeBase orchestration the way P2 did
                                      MLPCore, rather than binding monolithic
                                      mode objects into WASM (which would
                                      contradict the locked two-instance RT
                                      architecture).
  plans/browser-mode-coverage-spec.md 5b — an audio-topology notion so Manifold
                                      stops cataloguing 4 modes that
                                      structurally cannot run in a browser.
  plans/curated-presets-spec.md       5c — ALIGNMENT defect 2, built on the
                                      operator's §7.6 definition: a curated
                                      preset is configuration only, network
                                      untrained.
  plans/hardware-editor-spec.md       5d — ALIGNMENT defect 3, applying
                                      useq-celium's existing discipline (C
                                      header as wire truth + TS mirror + parity
                                      test) to a MEMLNaut serial protocol.

2434 lines. I spot-checked their path citations mechanically against
git ls-files: of 170 backticked paths, every unresolved one is either a file
the spec proposes to create or a cross-reference to a sibling spec in this same
commit. None describes deleted code as live — which is the failure mode that
made half the existing corpus untrustworthy, and the reason the §8 pass earlier
today had so much to do.

The preset spec is the most valuable byproduct: en route it found five places
where existing docs still describe a deleted world, including
manifold-parity-features-spec.md §1.1 specifying a PipelineLayer over
engine/input-pipeline.ts and output-pipeline.ts, both deleted at one-core P4.

Also carries the doc sync for the telemetry and benchmark work in a77770f:
AGENT-REFERENCE gains the throughput and loss-history entries, and dock-spec
§1.3 records that its long-deferred diagnostics suite shipped PARTLY — the loss
curve and weight-health table are real, GradientFlow is not built and is not
planned as drawn (the core records no per-layer gradient magnitudes, and the
fabricated version was deleted in Phase 1).
2026-07-21 22:03:39 +02:00
monkey-w1n5t0n
a77770f95d feat: curve truth, DriverConfig, real telemetry, engine benchmark
Four items from one workflow, committed together because their build and CI
wiring genuinely interleaves — nisps/CMakeLists.txt, run-all-tests.sh and
ci.yml each carry hunks from two of them, and the stage renumbering (1/5 ->
1/6) touches every line. Splitting would produce commits that do not build,
which is worse than a commit that does four things and says so.

S26 part 2 — the curve declaration now matches reality. params[].curve stays
the mode-wide DEFAULT; a voice_spaces entry may now be {name, curve_overrides}
declaring only the slots where THAT voice space deviates. The 6 modes with one
voice space are byte-identical. The values were derived MECHANICALLY by a new
codegen/curve-audit.ts that models the four idioms a p[N]*p[N] regex misses
(alias form, memlcelium's implicit-counter sq() lambda, loop-generated indices,
smooth_params_), inlines helpers, and RAISES rather than guessing when it
cannot reduce an expression. A drift gate cross-checks 1179 (voice space x
param) slots against engine source on every run and was proved to fail loudly
on three drift classes. Application stays in the engine: nisps/engines,
nisps/pipeline and nisps/core are untouched, generated output is pure insertion
(755 insertions, 0 deletions), and the rebuilt nisps.wasm was byte-identical.

S4 / 7.2 — firmware reads the active mode's driver config at mode start, and
mic/line is real. My brief assumed the engine owns this; the code disagreed and
the code was right. sound_analysis_midi's EngineT is NoOpEngine — the mic lives
on a separately-composed AnalysisEngine member — so engine-level wiring would
have compiled, passed every gate, and left the one mic mode on line input.
Hence a mode-level seam defaulting to engine().driver_config(). Separately,
DriverConfig's defaults (line_level 0, output_volume 1.0) had drifted from
memllib's actual 3/0.8 because nothing had ever read them; wiring them as-is
would have made every silent mode louder and its line input maximally
insensitive — a behaviour change disguised as plumbing. Now pinned by a test.
Also: GetSysClockSpeed() panic()s on unsupported sample rates and runs on the
first line of setup(), so sample_rate needed a fallback ahead of clock setup.
CI's firmware env list gains soundanalysismidi — it is the only mic variant and
nothing else compiles that path.

Plan 5e — telemetry is real. A loss_history C-API entry across the full 5-layer
chain lets the browser read the per-iteration loss the core already records.
The audit named one fabrication site; there were two — wasm-iml.ts's
synchronous train() published lossHistory: [loss] as well. A third, ctx.loss,
was not merely dead but actively synthetic (fallbacks of prev * 0.82 and a
literal 0.5, rendered by nothing) and is deleted. The firmware buffer stays
untouched, per the L25 call. EngineApi.lossHistory() reads spine state rather
than the MLP handle, because trainAsync() fits on the worker's mirror net and
the handle would give a subtly-wrong second answer.

Plan 5f — engine throughput is measurable. One source compiled twice (CMake
natively, emcc for WASM) so the targets compare directly and no WASM export is
added. Sequencers are driven into a working state, and every row prints its own
working-state evidence so a number produced by an idle engine is visible rather
than plausible. Reports, never asserts: a wall-clock threshold on shared
hardware is meaningless or flaky, same call as the firmware size job.

ALIGNMENT: the telemetry defect is deleted (built, not deferred); the
performance defect is rewritten to what is actually left — these are HOST
numbers, and nothing measures the RP2350 at 150 MHz, which is the target the
mission's constraint is about. Q4 (memllib ownership) and Q5 (legacy feedback
modes) are closed.

Corrections to my own earlier claims, both found by agents contradicting the
brief: manifold/ONBOARDING.md was NOT "now accurate" — its primitives list
still named five deleted primitives and cited a seededGradient() that does not
exist. And the parity harness misses the sequencer engines because it runs 128
frames while their sequencers evaluate every 400-500 samples, NOT because
all-params-0.5 fails to trigger them (it does trigger: 0.5 maps to ratio 2,
firing three times per bar). The fix is a longer window, not different params.

Gates: run-all-tests.sh ALL GREEN — 4/4 ctest, parity PASS, lint clean, curve
drift 1179 slots ok, 39 e2e (was 33). Firmware: 5 envs built including the mic
variant.
2026-07-21 22:02:23 +02:00
monkey-w1n5t0n
68d4cc4017 build(firmware): migrate to PlatformIO and vendor memllib (plan §5)
One cut, no dual path. Closes ALIGNMENT defect 3 ("Arduino-CLI build
machinery is actively hostile") and vision bullet 4.

platformio.ini carries 16 [env:], one per variant, each passing
-DMEMLNAUT_MODE_TYPE; selftest passes -DNISPS_SELFTEST=1 instead. The env list
IS the registry now — the .ino comment-registry and the NISPS_ST_* token-paste
table are deleted rather than migrated. L12 noted that table was already
silently missing the currently-shipped SLPWorkshop variant, which is the whole
argument against having a second list.

Also deleted: the Python/sed machinery that rewrote the COMMITTED .ino on every
build, the sketch symlink forest, the global TFT_eSPI User_Setup.h mutation
(now -D flags — TFT_eSPI's own documented PlatformIO recipe), the UF2
boot-mount detection stack (upload_protocol=picotool talks to the bootloader
directly), and build-firmware-arch.sh entirely. Scripts 683 -> 435 lines.

memllib is vendored at lib/memllib/ from upstream e291192; no submodules
remain. VENDORED.md records provenance and the re-sync procedure.

S9: a firmware-build CI job compiles three representative envs against a cached
toolchain and reports per-variant flash/RAM. Firmware is in an automated gate
for the FIRST time. The old ci.yml comment justified excluding it as "low
verification value" — an assessment that did not survive contact, since the
SelfTest variant sat broken for an unknown period calling a DisplayDriver
method that did not exist at the pinned memllib commit, and nothing noticed
because nothing built it.

Verified: all 16 envs build from an empty cache, each within ~520 bytes of the
arduino-cli binary it replaces, flash and RAM. Measured as .text+.rodata /
.data+.bss+vector+uninitialized — NOT PlatformIO's console line, which
double-counts .data on this board. This does not prove the hardware boots; no
flash+smoke test was possible and that stays an operator chokepoint.

  slpworkshop 248232/145028   pafsynth 256880/149716   selftest 216228/17960
  (all 16 in the CI log format; none exceeds 2% of a 16 MB flash)

Two traps recorded so nobody rediscovers them: vendoring memllib's subdirs
without a src/ wrapper makes PlatformIO's library builder silently compile
NOTHING while still linking; and project build_flags land BEFORE the
framework's own -std=gnu++17 -Os, so build_unflags is required.

CORRECTION carried in this commit: the firmware sizes in c19d846's message and
the first version of the memllib recon doc were wrong — SLPWorkshop 145348,
PAFSynth 145300, SelfTest 141840. They came from building variants in sequence
through a SHARED incremental arduino-cli build directory, which reused stale
objects and under-reported by ~75 KB. Clean-cache rebuilds of the identical
commit give 216736/18492 for SelfTest. The real cost of the memllib upstream
bump is +216 bytes flash, not +316. Never measure firmware size through a
reused build dir.

HISTORY NOTE: this commit and the docs commit before it were rebuilt (force-push,
2026-07-21) so that each contains only what its message describes. The first
versions had the firmware deletions stranded in the docs commit by a shared-index
race between concurrent agents; content is byte-identical to the originals.

Gates: run-all-tests.sh ALL GREEN (nisps/ untouched by this change beyond
include paths); 16/16 pio envs build.
2026-07-21 20:17:58 +02:00
monkey-w1n5t0n
9ad1f78ddd docs: the specs disposition pass (plan §8)
Roughly 20k lines were deleted from this repo in the last week and much of the
corpus still described the pre-deletion world in the present tense. Executes
the §8 table: archive the retired, reclassify the executed, prune the stale.

  aimmersive-clone-spec      -> _archive/ with a deprecated-by note
  feedback-modes-port-spec   -> plans/, kind: plan, status: executed
  manifold-parity-features   -> plans/, kind: plan, status: active
  playground-2.0-rewrite     -> status: superseded
  engine-architecture        434 -> ~120 lines; seam + spine kept, rewritten
                             present-tense against the shipped engine/
  MAIN.md                    six contradicted claims fixed; registry resynced
  vcv-module.md              pruned to the current 8->16 contract and made the
                             single .nisps format spec
  vcv/NISPS-FORMAT.md        DELETED — documented a v1 format that no longer loads
  vcv/README.md, BUILDING.md rewritten to the real contract, menu, OSC table
  inputs/backends/dock trio  grounding sections marked historical, dead cites fixed

Two rows of the §8 table were themselves wrong, corrected here: the deleted
full-state sync lives in backends-spec.md §6.3, not vcv-module.md (which has no
§6.3), and codegen/README.md was already a MAP pointer with no port-solidjs
trigger left to remove.

Beyond the table — found by sweeping every backticked path in the changed docs
against `git ls-files`, which is how these should have been caught before:

  manifold/ONBOARDING.md documented a UI that Phase 1 deleted, as if current:
  SplitStage, ReadoutStrip, InputMini, BackendAdvanced, AltitudeNav, and a
  shot.spec.ts that does not exist. The whole stage table was keyed on a `focus`
  axis that no longer exists — selection is now sandwich > particles >
  composite. This matters more than the rest: CLAUDE.md tells every agent to
  read ONBOARDING.md first for Manifold work, so it was actively teaching a
  fiction. Rewritten against ConsoleApp.tsx.

  MAP.md claimed the input layer reduces axes to the engine arity with an
  "even/odd blend". input-layer.ts says the opposite in its own header: one
  dedicated slot per axis, 1:1, into a 32-input over-provisioned head, and
  mean-blending was removed deliberately because it diluted every source.

  AGENT-REFERENCE.md still promised TS emission "returns at P5" (landed),
  per-mode dims "become schema-real at P5" (landed at P5.3), and pointed at
  nisps::FixedBuffer (deleted).

Doc-right/code-suspect, filed rather than fixed: VCV computes derivedMean/Std/
Delta and cachedNovelty behind a live context-menu toggle that nothing reads;
vcv/plugin.json points at the MusicallyEmbodiedML org rather than this repo's
origin; and the module defaults to UDP 7001+id%64 while bridge.ts defaults to
9000, so out of the box they do not meet.

Firmware-build docs are deliberately untouched — the PlatformIO migration
lands next and rewrites all of them.
2026-07-21 20:17:58 +02:00
monkey-w1n5t0n
b16f26e6ab refactor(ml): one runtime-configurable training default (S26)
The operator's call: "there should be one default learning rate and one
default max iterations and they should both be configurable at runtime."

There were SIX copies, not the four the audit described, and they did not
agree:

  nisps/ml/mlp.hpp        no-arg train() hardcoding 1.f / 1000u / 0.001f —
                          and firmware's ONLY training path calls exactly
                          this, so firmware had no runtime knob at all
  wasm-iml.ts             train() and trainAsync() TS default params (x2)
  engine-api.ts           learningRate ?? 1.0, with no maxIterations knob
  vcv/src/iml.hpp         200 / 0.1 / 0.00001 — silently divergent
  external_synth_midi.hpp its own kDefaultLearningRate/kDefaultMaxIterations
  schemas/modes/*.json    x9, identical, read by nobody at runtime

Now: schemas/ml_defaults.json is the single declaration (validated against a
sibling meta-schema, matching the midi_device.schema.json convention), codegen
emits it to C++ and TS in the same run, and MLPCore carries a TrainConfig whose
default member initialisers read the generated constant.
set_train_config()/nisps_ml_set_train_config() make it runtime-overridable on
every target; the explicit-argument train() overload is untouched. min_error
joins the tuple — it was duplicated identically and belongs with the other two.

The per-mode ml block loses default_learning_rate/default_max_iterations.
default_spread stays (genuinely wired on both targets) and input_channels stays
(codegen-time validated, real information for sound_analysis_midi).

VCV BEHAVIOUR CHANGE, deliberate: MEMLNaut.cpp constructs IML positionally and
relies on those defaults, so the module moves to 1000/1.0/0.001 — 5x the max
iterations, 10x the learning rate, and a 100x looser early-stop threshold. The
old values were never justified anywhere; they arrived with fbc68eb alongside
an unrelated module rewrite and no tuning rationale. Firmware and WASM have
shipped 1.0/1000 all along. It is now runtime-settable if this turns out worse.

The generated header lands in nisps/ml/generated/, not nisps/modes/generated/
where the rest of codegen output lives: training hyperparameters are an ML
fact, and nisps/ml sits below nisps/modes, so emitting them there would make
mlp.hpp include upward. The agent that built this flagged the directory-crossing
rather than hiding it; this is the fix. CI's generated-freshness gate learns the
new directory.

Gates: run-all-tests.sh ALL GREEN — 4/4 ctest, parity PASS (max delta 2.38e-7),
lint clean, manifold typecheck + 17 unit + 33 e2e (which exercise train() and
trainAsync() through a real browser).
2026-07-21 17:20:10 +02:00
monkey-w1n5t0n
b6be081cf0 docs: record the operator's L25 and telemetry decisions
L25: keep the firmware loss-history buffer. With §7.3's "make it real, browser
only, behind a feature flag", the telemetry decision set is now complete and
plan 5e has no judgement calls left in it — only work.

Also corrects the ALIGNMENT entry, which described four live fragments: the
fabricated gradient-health UI (L1) was already deleted in Phase 1, so three
remain. Fragment inventory re-verified against the tree rather than copied
forward from the audit.
2026-07-21 17:05:30 +02:00
monkey-w1n5t0n
1f38f1a7fb fix(engines): restore MEMLCelium's feedback tap (L7)
feedback_/fbzm1_/fb_smooth_alpha_ were read every process() call but nothing
ever wrote feedback_, so the path was silently inert. Git history shows this
was a *muted* feature, not dead code: `feedback = v0 * feedbackGain` with
feedbackGain 0.1f (MEMLCeliumAudioApp.hpp @ d095688), then "0; //0.1f" —
zeroed with the value kept in the comment — then the write was dropped
entirely when the engine was ported to nisps/engines/ at the 2026-04-29
rewrite (8d0d47b). Operator decision: port it back, live at 0.1f.

The tap is voice 0's own enveloped output, post-envelope and pre voice-1/mix/
shape — NOT the final output. This differs from PAFSynth's feedback shape;
where the sibling and the reference disagreed, the reference won.

Not schema-exposed: the reference hardcoded it, and MEMLCelium has no voice
spaces to vary it by. If feedback depth becomes a research axis it wants to be
a runtime parameter — noted, not built.

Verification, and a gap worth recording: parity-check.sh PASSES (max delta
2.38e-7) but that is NOT evidence for this change — its harness only exercises
PAFSynth and ChannelStrip, and the impulse baseline runs all-params-0.5, under
which voice 0's sequencer never triggers, so v0_env stays 0 and the new write
is 0 either way. That baseline is blind to this path, not a check on it. So it
was cross-checked directly instead: sequencer params forced to trigger early,
4800 samples native vs WASM through the nisps_engine_* C ABI —

  native vs WASM max abs diff  1.19e-7   (tolerance 1e-5)
  all 4800 samples nonzero, finite, max amplitude ~0.99 (tanh-bounded)
  same run with gain forced to 0 diverges by up to 1.49 — not a no-op

No fixture regenerated. nisps.wasm ships here because it is a tracked artifact
under the Phase 0 freshness gate.

Follow-up: the parity harness covers 2 of 8 engines. "Firmware and WASM share
the same engines" is asserted repo-wide but tested narrowly.
2026-07-21 17:04:35 +02:00
monkey-w1n5t0n
c19d84619e feat(firmware): bump memllib to current upstream and dissolve the fork
The §7.5 decision was "rebase our commits onto upstream, then vendor". On
inspection there was no rebase to do, and the inventory that recommended it
(7a30da9) was wrong on two counts — both corrected in this commit.

All three fork commits touch only `examples/`. That directory is not in the
sketch symlink forest (firmware/MEMLNaut-NISPS/src/ links exactly audio,
hardware, interface, synth, utils, PicoDefs.hpp), so it is never compiled.
They existed to let the RL code build against nisps/core *while it was being
ported*, and that port is done — nisps/ml/{jolt,ou_noise,feedback,geo_push}.hpp
cite the upstream sources directly. Two of the 31 commits I flagged as
"work we are missing", `9fcd459 jolts` and `d0d8a72 noise`, are precisely the
ones already absorbed.

So the question was never rebase-or-not but which snapshot to vendor. Current
upstream, because the staleness was already costing us:

  - upstream has DisplayDriver::NavigateToView; the pin does not. The SelfTest
    variant called it and had been failing to compile (worked around in
    b953681). It was written against a newer memllib, not against nothing.
  - e291192 "l r input swap" is a hardware bug fix: the physical L/R input
    sockets are wired to opposite codec ADC channels. Every mode on the old
    pin sees its stereo input backwards.

Verified by building all three variants with arduino-cli:

  SLPWorkshop  145348 flash (+320)  87388 RAM (+4)
  PAFSynth     145300 flash (+312) 107060 RAM (+4)
  SelfTest     141840 flash (+320)  12028 RAM (+4)

Exactly one compile error: the .ino used kSampleRate in a constexpr, and
upstream 1997699 made it a runtime `extern size_t` so a mode can pick its own
rate. constexpr -> const; it is a once-per-second diagnostic print. The uniform
+316 is the AudioDriver/DisplayDriver deltas — the bulky new upstream code
(GrainDelayI16, ReverbI16, ModFXI16, CCSelectView, RLView, VUMeterView,
PSRAMManager) is header-only and unreferenced, so the linker drops all of it.

.gitmodules points at upstream again: with the pin on an upstream commit, the
fork holds nothing the firmware compiles. Phase 0 pointed it at the fork only
because b37fc53 existed on no remote. The fork's feat/nisps-core-swap branch
stays pushed; nothing is destroyed.

Refs: plan §5, §7.5; ALIGNMENT Q4.
2026-07-21 17:04:10 +02:00
monkey-w1n5t0n
7a30da91a8 docs(recon): memllib usage inventory — the §7.5 prerequisite, with a wrinkle
Plan §5 names an inventory of the load-bearing memllib surface as the
prerequisite for the vendor-vs-fork-pin decision. Built it from the firmware
LINKER MAP rather than grep — the same method that would have caught daisysp
(S8) mechanically instead of by reading.

Result: all 24 compiled translation units contribute kept sections to the image;
no discarded-only TU exists. memllib is not a daisysp situation. "The
load-bearing subset" is ~1.8 MB across 84 files — essentially all of memllib
minus examples/. There is no small subset to lift; synth/ alone is 1.3 MB and
all three of its TUs link.

The wrinkle, which was not visible when §7.5 was decided: src/memllib is not a
third-party dependency, it is the LAB's shared library
(MusicallyEmbodiedML/memllib), and our fork is 3 commits ahead but 31 commits
BEHIND — +2034/-153 across exactly the files we link, including "new staticmlp",
"jolts", "noise", "verb". Vendoring as-is freezes that gap permanently.

Recommendation recorded: rebase the three small NISPS commits onto upstream main
FIRST, then vendor. Same self-contained end state, but it does not silently
discard work that sounds directly relevant, and it is the only moment that merge
is cheap — after vendoring there is no upstream history to merge against.

Flagged for an operator call before the PlatformIO cut, which this gates.
2026-07-21 14:12:55 +02:00
monkey-w1n5t0n
b953681157 fix(firmware): restore the SelfTest build — it called a DisplayDriver method that does not exist
Found by actually building the firmware, which nothing in this repo does: CI
skips it, run-all-tests.sh skips it, and lint skips it (S9). The SelfTest
variant did not compile:

  glue/selftest.hpp:650: error: 'class DisplayDriver' has no member named
  'NavigateToView'

Pre-existing, not caused by this audit's work: no commit in this series touched
selftest.hpp, and `NavigateToView` appears nowhere in memllib at the pinned
commit. DisplayDriver's actual surface is AddView / InsertViewAfter /
ChangeView(delta) over a carousel.

The call was redundant as well as wrong. DisplayDriver::Setup() leaves
currentViewIndex_ at 0 and the selftest adds exactly one view, so that view is
already the one on screen. Deleting the call restores the build with no
behavioural change on device; a comment records why, since the intent
("navigate to my view") reads as necessary.

Worth noting against Phase 1: SineOsc was KEPT there specifically because "the
firmware selftest uses it". That was still the right call — deleting it would
have added a second breakage to a variant that already had one — but the
justification was weaker than it looked, because the selftest had not compiled
for some time.

Verified by building three variants end to end with arduino-cli, which also
constitutes the first real check of this audit's firmware-side deletions
(daisysp, input_router.hpp, the perf macros, the peripherals helper extraction):

  SLPWorkshop  flash 145028 B   RAM  87384 B (16%)
  PAFSynth     flash 144988 B   RAM 107056 B (20%)
  SelfTest     flash 141520 B   RAM  12024 B  (2%)

Those numbers are also the first per-variant size baseline this project has had
(plan §6.5f wants exactly this as a gate).

Incidentally confirms A6 live: `scripts/build-firmware.sh` rewrote the committed
.ino to select each variant, and the working tree had to be restored after every
build. That is what the PlatformIO migration deletes.
2026-07-21 14:10:05 +02:00
monkey-w1n5t0n
fae519092f docs: sync MAP with Phase 3; burn down plan §4 and hand S26 to the operator
MAP.md: codegen's generated/ now owns ParamSchema and the per-mode MLP aliases;
new shared headers (dsp/ratio_seq.hpp, dsp/seq_clock.hpp, core/event_queue.hpp
with a note on why it is deliberately not RingBuffer); backends/base-backend.ts;
the seed script and codegen/templates bullets removed; the synth-midi-cc.json
path corrected to its sources/ subdir.

Plan §4 burned down. S26 is NOT done — it is now an operator decision with the
per-field inventory that makes it cheap, recorded in the plan: default_spread is
already wired (the audit's "zero consumers" was a quarter wrong);
default_learning_rate/default_max_iterations are unread but every schema carries
exactly the values already hardcoded, so wiring them is numerically a no-op
today; input_channels is codegen-time-validated and carries real meaning for
sound_analysis_midi; and per-param curve is a trap — it is descriptive of
squaring the engines already do internally, so wiring it would double-apply on
35 params.
2026-07-21 14:03:16 +02:00
monkey-w1n5t0n
7ce4598a0b chore(schemas): retire the clobbering seed script; fix the direction-of-truth text
Phase 3 (L39, ST13, ST9, ST10).

- L39: deleted codegen/seed-midi-devices.ts. It regenerated — and therefore
  silently CLOBBERED — the hand-tuned device schemas it had originally created.
  Its purpose is served and git retains it. Confirmed nothing invokes it (the
  root package.json that once might have was deleted in Phase 1).
- ST13: moved the provenance artifact out of the repo root. AUDIT CORRECTION:
  the finding's proposed destination, schemas/midi_devices/synth-midi-cc.sources.json,
  would have BROKEN codegen and CI — generate-midi-devices.ts treats every *.json
  directly under that directory as a device template and ajv-validates it, and
  the sources file has no device_id/midi/params, so the generator would exit
  non-zero. It went to schemas/midi_devices/sources/ instead.
- ST9: schemas/schema.json's own description stated the direction of truth
  BACKWARDS. Rewritten: schemas are canonical, codegen emits C++ and TypeScript.
  (The audit cited params_notes.md; the real path is schemas/modes/params_notes.md.)
- ST10: codegen/README.md rewritten against current reality — it described
  retired playground paths and a phantom temp dir.

Descriptions are never emitted, so this is behaviour-inert; proven by
regenerating all four generated directories byte-identically.
2026-07-21 14:03:07 +02:00
monkey-w1n5t0n
ae2657a01b refactor(manifold): one applyCurve, one GROUP_COLOR — and make the knob honest
Phase 3 (L38, ST4).

L38 is a REAL, VISIBLE behaviour change, and it is a bug fix rather than a wash.
There were two divergent applyCurve implementations: model.ts's
(`e = 0.25 + c*1.75`, linear at c≈0.43) drove the on-screen bars, while
mapping.ts's spec-anchored version (0.5 = exactly linear, symmetric in
log-exponent space) drove `mapOutput` — i.e. the value actually sent to MIDI/OSC/
VCV. So the display disagreed with the signal at every curve setting except the
shared endpoint. mapping.ts's survives; model.ts's is deleted and re-exported.
Displayed values now move visibly, and they now match what is transmitted.

That unification exposed a third copy the finding did not mention: CurvePad.tsx
inlined the same deleted formula to draw the curve-knob preview and its numeric
readout. Left alone it would have matched NEITHER implementation — the knob
would have drawn one response while the bars and the output used another.
CurvePad now calls the same applyCurve, so knob, bars and transmitted value are
one thing. The readout shows the actual exponent.

ST4: GROUP_COLOR was duplicated across four files (Phase 1's ReadoutStrip
deletion removed a fifth). All four now import the single map in model.ts —
including OutputStage.tsx's OUT_GROUP_COLOR, which was outside the sweep's scope
and would have been the last survivor.

NOT done, deliberately: ST4 also invited replacing the fixed map with a
hash-to-palette function, because its keys (formant/pitch/amp/filter/fx/mod) are
JSX-era placeholders that do not match the real schema group strings
(verb/sequencer/general/filterbank/delays/eq/midi/...), so most groups currently
fall through to the accent colour. That is a visual redesign, not a
deduplication, and it should be one deliberate change now that every consumer
reads one map.

Gates: typecheck clean, 17/17 unit tests, 33 e2e specs.
2026-07-21 14:02:48 +02:00
monkey-w1n5t0n
87daac4f0d refactor(manifold): extract BaseBackend mirroring the input layer's BaseSource
Phase 3 (L17). The midi/osc/vcv transports each carried their own copy of the
status plumbing, throttle gate and lastSent management; the inputs layer already
had this factored out as BaseSource, so BaseBackend mirrors its shape rather
than inventing a second convention.

AUDIT CORRECTION: L17 says "all four" backends duplicate lastSent. Only three
do. The uSEQ CV backend has a genuinely different frame-level dead-zone (an
Int32Array over 14 channels plus gate bits) and no per-output lastSent, so it
takes the status half only — forcing it into the shared per-output path would
have needed a special case, and a base class with a per-subclass escape hatch is
worse than three backends sharing one. Also: BaseSource itself has no throttle
or lastSent (it is status + action plumbing only), so the mirror is partial by
nature; the throttle/lastSent half is the duplication the backends actually had.

Three internal representation changes, none observable: the MIDI backend's
lastSent moves Int16Array -> Float32Array (stored values are integers -1..127,
exact in f32, so every dead-zone comparison is bit-identical); the CV backend
now inherits a setContext that also allocates a per-output buffer it never reads
(cold path, no effect); and onStatusChange's unsubscribe returns void rather
than Set.delete's boolean (the declared type was already `() => void` and no
caller used it).

Phase 2's fix to vcv-backend.ts — tracking and dead-zoning the full N-dim input
vector rather than 2 — is preserved.

Gates: typecheck clean, 17/17 unit tests.
2026-07-21 14:02:48 +02:00
monkey-w1n5t0n
96737a3d42 refactor(engines): extract the shared sequencer machinery
Phase 3 (L8). Pure statement-for-statement relocation into
dsp/ratio_seq.hpp, dsp/seq_clock.hpp and core/event_queue.hpp.

AUDIT CORRECTION: L8 says "breakor and elysiamorf duplicate ratio_seq". That is
false — ElysiamorfEngine has no ratio_seq at all; it triggers continuously via
FM operators. The real duplicate pair is BreakOrEngine and MEMLCeliumEngine,
whose copies are byte-for-byte identical. Elysiamorf did share the clock and
event-queue machinery, so it uses those. memlcelium now includes the shared
ratio_seq too, which is what actually closes this finding.

Deliberately NOT folded into core/ring_buffer.hpp: RingBuffer is an
atomics-based cross-core SPSC channel (its header says so), whereas the engines'
event queue is produced and drained on one thread. Reusing it would have meant
paying for atomics to serve a single-threaded FIFO. The distinction is now
recorded in MAP.md so the next audit does not read them as duplicates.

Bit-exactness: verified the MIDI-clock tick and bar-phasor tick preserve the
original operation order with no floating-point re-association, and that
EventQueue keeps the original `% N` indexing rather than adopting RingBuffer's
bitmask. The golden suite (nisps_golden_tests) and the native<->WASM parity blob
both pass unchanged — they are the check, and they were not re-baselined.
2026-07-21 14:02:23 +02:00
monkey-w1n5t0n
f5b571412f refactor(codegen): codegen owns mode identity, per-mode schemas and net dims
Phase 3 (S1, S5, S6, S25, L11, L37, ST11, ST12). Behaviour-preserving by
construction: the diff on the generated directories is PURELY ADDITIVE (217
insertions, 0 deletions), so no emitted constant changed value. This moves where
truth lives; it does not change what truth says.

- S5: nine mode headers each carried a mechanically identical 12-field
  positional ParamSchema aggregate. codegen now emits one
  `inline constexpr ParamSchema k<Mode>Schema` per mode, and the struct itself
  moved into generated/schema_types.hpp. Each param_schema() is a one-line
  return.
- S6 + S25: every mode hand-typed its net shape a second time as MLP template
  args, duplicating the schema's own dims. codegen emits a `<Mode>MLP` alias
  built from the already-emitted constants (not re-literalled), and all nine
  modes use it. NMaxExamples still defaults from kDefaultMaxExamples (Phase 2).
- S1: model.ts hand-imported all nine schemas by name and hand-paired each with
  its overlay — so the SET of modes was hand-maintained and could silently drift
  from codegen. codegen now emits ALL_MODE_SCHEMAS; SCHEMA_MODE_OVERLAYS is
  purely display truth (label/glyph/css/order), which stays hand-curated.
- L37: deleted the hand-written modeEngineId switch, which duplicated
  schema.engine_id and silently defaulted unknown modes to 'thru'. Routes on
  MFMode.engineId with exactly one documented exception: sound_analysis_midi
  declares engine_id 'thru' because its ModeBase audio slot really is
  NoOpEngine, while it separately drives the real AnalysisEngine.
- L11: ExternalSynthMIDIMode's shape was literal in two places; now named once
  in an ext_synth_defaults namespace with an ExtSynthMIDIMLP alias. Full folding
  into the JSON pipeline is NOT done and the reason is recorded in-file: it is a
  template family over an externally-supplied Device and variable NOut, with no
  single (device, NOut) schema to author.
- ST12: extracted codegen/lib.ts for the helpers both generators duplicated, and
  corrected the comment that claimed they had to be separate. Proof the
  extraction was behaviour-free: regenerating the MIDI-device outputs produces a
  byte-identical tree.
- ST11: deleted codegen/templates/ — dead "reference" files no generator reads,
  already drifted from the real emitters.

Gates: run-all-tests.sh ALL GREEN; codegen idempotent (re-running both
generators yields no further diff), which is what CI's dirty-diff gate checks.
2026-07-21 14:02:23 +02:00
monkey-w1n5t0n
a1a158b8d0 docs: burn down plan §3, record Phase 2's two deliberate deviations
Plan §3 marked burned down, noting where execution departed from the written
plan and why: L35 deleted rather than relocated (zero consumers), and S35's
describe ABI widening forcing two out-of-scope buffer fixes. Also records the
package.json test-glob bug found en route — new unit tests were silently not
being run.

ALIGNMENT defect 5's trailing sentence updated: the behaviour bugs it listed as
pending are fixed.
2026-07-21 13:23:11 +02:00
monkey-w1n5t0n
8c44a51220 fix(gates): close the no-heap lint's false negatives and parity's silent FAIL
Phase 2 (S30, L52, L51). These are the gates that are supposed to protect the
core's headline constraints, so the fix is demonstrated rather than asserted.

- S30: lint-cpp.sh's heap audit hardcoded dsp/engines/ml/modes, so nisps/core/
  and nisps/pipeline/ — the P4 control-rate hot path — were NEVER scanned. Its
  comment handling also post-filtered with `grep -v ' *//'`, which misses
  trailing comments. Coverage is now exclusion-based (everything under nisps/
  except wasm/, tests/, build/), comments are STRIPPED before matching using the
  same perl strip audit_float_suffix already used, and the pattern set is
  extended. dynamic_storage.hpp remains deliberately allowlisted — it is a real,
  documented heap user that #errors on RP2350.

  PROOF (run by me, not just reported): planted `new float[4]` in
  nisps/pipeline/input_chain.hpp plus a std::malloc in the trailing-comment form
  the old filter skipped.
      old lint: "[lint-cpp] clean", exit 0     <- the false negative, live
      new lint: FAIL, both lines named, exit 1
  Plants removed; lint clean and exit 0 again; tree verified unmodified.

- L52: parity-check.sh's FAIL branch was unreachable — under `set -e` the script
  died at the diff command before it could print anything. A mismatch therefore
  failed silently. The diff now runs inside the `if` condition, where set -e does
  not apply. Verified by forcing NISPS_PARITY_TOL=0: old script exited silently,
  new one prints FAIL with the exit code.

- L51: parity_check.cpp and parity_wasm.mjs headers documented 4 stages / blob
  v1 against a real 7-stage / v5 implementation. Comment-only rewrite; parity
  behaviour unchanged (re-ran the gate: PASS, 1273 floats within 1e-5).

Note for future work: the lint now permanently scans nisps/core/ and
nisps/pipeline/, so changes there are enforced that were not before.
2026-07-21 13:23:11 +02:00
monkey-w1n5t0n
1a78ed9597 fix(vcv): close the audio-thread race and remove JSON from process()
Phase 2 (L34, L35). Both findings' line citations were accurate this time.

- L34 data race: process() (Rack's audio thread) called
  imlShadow.get_example_features()/get_example_labels() directly on the WORKER
  thread's private engine — which the file's own THREADING INVARIANT comment
  says only the worker may touch — while workerLoop() concurrently
  clear/refills those same std::vector<std::vector<float>> members via
  load_examples(), train_(), randomise_weights and clear_dataset. Unsynchronised
  reader/writer on a non-atomic vector: undefined behaviour.
  Fix extends the staged handoff the file ALREADY uses for pendingWeights
  rather than adding a second mutex: the worker deep-copies features/labels
  into pendingFeatures/pendingLabels at the same instant it copies
  pendingWeights, immediately before weightsPending.store(true), and the flag is
  now released only after the whole batch is consumed — closing an early-release
  window the old code had. process() no longer references imlShadow at all
  (verified: the only surviving mention is a comment).
- L35: process() ran full jansson serialize on every weight-swap OSC push and
  full json_loads + dataFromJson on incoming OSC state — heap-heavy tree work
  on the audio thread. The plan said to move it to the worker. It is DELETED
  instead: reading the actual transport shows both directions talk to nobody —
  osc-client.ts only ever sends {params|input|feedback}, and bridge.ts has no
  state/weights case and explicitly drops other addresses. Relocating
  heap-heavy work to serve a confirmed-zero consumer is complexity without a
  requirement; removing the cause is the smaller coherent design.
  dataToJson/dataFromJson are UNTOUCHED — they remain the live consumers for
  Rack patch save/load and the .nisps preset menu, both off the audio thread.

Neither is empirically reproduced: a real race needs a live Rack engine under
TSan, which is not available here. Justified by reading, and verified by
`cd vcv && make -j4` (clean) plus the host suite including
test_vcv_iml_parity.cpp, which pins iml.hpp bit-exactly against the core MLP —
iml.hpp was not modified, and parity holds.

Known remaining, pre-existing and out of scope: process() still takes a brief
lock_guard on feedbackMutex to copy a small staged struct, and several config
fields (slewMs, oscPort, output/input range flags) are written by the UI thread
without atomics.
2026-07-21 13:23:11 +02:00
monkey-w1n5t0n
75e0e58067 fix(manifold): full-width input vector, dropped switches, MIDI churn, re-subscribe churn
Phase 2 (S10, L19, L18, L24).

- S10: EngineApi.inputVector() returned a freshly allocated [lastRawX, lastRawY]
  on every spine tick, so VCV bridged mode silently truncated gamepad/MIDI input
  to 2-D while the spine already held the full N-dim raw vector. It now returns
  spine.lastRawInputs (ArrayLike<number>, documented as a live reused buffer —
  copy, don't retain; VcvBackend already copies), and VcvBackend tracks and
  dead-zones the full length. Audit correction: "32-input head" is not a
  constant — 32 is DEFAULT_MODE_ML.inputSize, the over-provisioned default
  before any mode is chosen; real per-mode widths come from the schemas.
- L19: BackendManager.setActive silently dropped a switch requested while
  another was in flight. Now stores the latest requested id and re-runs it in
  the finally block (latest-caller-wins).
- L18: MIDI CC messages triggered a React state update plus a snapshot
  allocation each. notifyBindings now fires only when the binding LIST changes.
- L24: two ConsoleApp global-listener effects had no dependency array and so
  re-subscribed on every render, including every pointer frame. Both now read
  through a single ref assigned in the render body, matching the existing
  onMoveRef pattern. Audit correction: its suggested `[inputs]` dep would not
  have worked — useInputLayer returns a fresh object literal each call, so that
  dep changes every render too.

Regression tests: input-vector-truncation.test.ts, backend-manager-switch.test.ts
(a fake backend whose start() is held open, to make the in-flight switch real),
midi-notify-churn.test.ts (fail-before confirmed: 51 notifications vs 1).
L24 has no test — this repo has no DOM render harness to count re-subscriptions
against a mounted component; verified by reading and reference-stability tracing.

ALSO: manifold/package.json's test script named its test files explicitly
("bun test src tests/pipeline-golden.test.ts"), so the three new files were not
run by `bun run test` or CI — regression tests that never execute. Now a glob.
Deliberately `tests/*.test.ts` rather than `tests`: bun's discovery matches
*.spec.ts too, which would drag the Playwright e2e specs into the unit run
(verified — it fails). Unit tests go 9 -> 17.

Gates: run-all-tests.sh ALL GREEN.
2026-07-21 13:22:38 +02:00
monkey-w1n5t0n
dbe0f5d8ba fix(ml): one named example capacity; train() and trainAsync() no longer diverge
Phase 2, S35. Two real defects from one root cause, both confirmed by trace
rather than taken from the audit:

1. Divergence. WasmIML built its TS Dataset mirror with a cap of 100 while
   every addExample() ALSO pushed into the C++ FIFO ring, capped at 128. Since
   train() reads the C++ ring and trainAsync() reads the TS mirror, past 100
   examples the two trained on different datasets — silently.
2. Latent OOB read. nisps_ml_train sizes its sample-weight span by the C++
   side's example_count() (up to 128), but wasm-iml.ts allocates that heap
   buffer from the TS dataset's size (<=100). Once the ring exceeds the mirror,
   the span reads past the end of the caller's allocation.

Fix: name the capacity ONCE as nisps::ml::kDefaultMaxExamples = 128, used by
FixedStorage's default template arg, DynamicStorage's default ctor arg, and the
MLP<> alias (which is the only real FixedStorage instantiation path and carried
its own independent 128 literal — the last copy of this dual truth). Expose it
through nisps_ml_describe and have the TS side read it instead of hardcoding.
Dataset's constructor default is removed entirely: a default was what invited
this bug class, and the sole call site now always supplies the describe() value.

ABI NOTE: this extends nisps_ml_describe from a 6-int to a 7-int descriptor.
nisps_ml_describe always writes 7 ints regardless of the caller's buffer, so
every call site had to grow in the same change or it would overflow the WASM
heap by 4 bytes per call. All five sites updated: three in wasm-iml.ts (init
defaults, init per-instance, reshape re-describe — the finding said there were
two), one in wasm-worker.ts, one in tests/cpp/parity_wasm.mjs. The parity
harness's expected-dims check now also pins the new max_examples slot.

Regression test: tests/cpp/test_mlp_storage_defaults.cpp — pins the two storage
policies to one constant, and drives MLPCore<DynamicStorage> exactly as
bindings.cpp does past the old TS cap, asserting it saturates at 128 and not at
100. Fail-before/pass-after confirmed by temporarily setting the constant to
100: 2 failures, named. Reverted: green.

Audit correction: the cited dataset.ts:81 is the FIFO eviction check; the
hardcoded default was at dataset.ts:45.

Gates: run-all-tests.sh ALL GREEN, parity PASS.
2026-07-21 13:22:38 +02:00
monkey-w1n5t0n
bf3d088ff1 docs: sync MAP/ALIGNMENT/AGENT-REFERENCE with the Phase 1 sweep
The three top-level orienting docs are each falsified by several of the eight
preceding commits, so they land here as one sync rather than being split
across commits that would each leave them half-true. Same phase, same push.

MAP.md — removed fixed_buffer.hpp, voice_space.hpp, the src/daisysp entry and
its symlink from the sketch-tree list, input_router.hpp/wire_inputs (inputs are
wired by bind_peripherals now), test_fixed_buffer.cpp, SplitStage/ReadoutStrip/
InputMini, BackendAdvanced.tsx, and feedback/rng.ts; corrected the primitives
count 12 -> 7; replaced the "engine LIFTED from playground/src" provenance;
rewrote the perf-attribute convention and deleted the NISPS_AUDIO_FUNC gotcha
(both macros and gotcha are gone).

docs/AGENT-REFERENCE.md — the memory-section-attribute instruction now names
only the macros that exist; input_router.hpp dropped from the firmware tree;
symlink list and the submodule-init note no longer mention daisysp.

ALIGNMENT.md — defect 5 (dead mass and registry sprawl) rewritten: the deletion
half is done, so the entry now scopes to what actually remains, which is the
registry/dual-truth half (Phase 3) plus the stale specs (docs pass). Per the
ALIGNMENT convention this is a rewrite-to-current, not a checkbox.
2026-07-21 12:49:39 +02:00
monkey-w1n5t0n
ffac04090c refactor(firmware): delete vendored daisysp and the input_router layer
Phase 1 group 8 (S8, ST2, L14).

- S8: the vendored src/daisysp tree (96 files, 41 .cpp compiled into every
  firmware build) and its sketch-tree symlink. Zero consumers — nisps replaced
  daisysp's PitchShifter with a custom granular implementation. Corrected
  firmware/README.md and setup-firmware-toolchain.sh, which called it a live
  submodule. The attribution comment in nisps/dsp/pitch_shift.hpp stays: it is
  an honest provenance note about a port, not a dependency.
  Noted while verifying: src/memllib/examples/KassiaAudioApp includes
  ../../daisysp/..., but examples/ is not symlinked into the sketch tree and is
  never compiled by the firmware build, so the deletion stands.
- ST2: input_router.hpp was a zero-logic speculative layer with one consumer;
  the .ino now calls bind_peripherals directly.
- L14: peripherals.hpp — deleted kAnalogInputCount, PeripheralBindings and the
  unused spread local, and extracted the duplicated commit-and-train block into
  one helper. The audit's suggested commit_and_train(mode, feedback) signature
  could NOT be behaviour-identical: repositioning() implies placing() (both
  live in ExploreState::Placing), so dispatching on feedback state inside the
  helper would silently reroute a TogB2 press mid-reposition from commit_place
  to commit_reposition, which differs (no snapshot restore, clears
  reposition_). Implemented as commit_and_train(mode, feedback, bool
  reposition) with the branch decided at the call site, preserving behaviour.

Firmware is not compiled by any gate yet (that arrives with the PlatformIO
migration, plan §5/S9), so this group is verified by reading and grep only.
Gates: run-all-tests.sh ALL GREEN.
2026-07-21 12:49:25 +02:00
monkey-w1n5t0n
53da84c425 chore(vcv): delete the dead test rig
Phase 1 group 7 (L33). vcv/test/smoke_test.cpp included a header that no longer
exists (a retired nisps-core path), asserted the pre-P6 2x12 module shape, and
ran in no gate; a compiled smoke_test binary was tracked alongside it. Removed
the directory plus Makefile.dist, updated BUILDING.md's two references, and
deleted the unreachable reply-to-sender branch in osc_server.hpp.

Gates: run-all-tests.sh ALL GREEN.
2026-07-21 12:49:25 +02:00
monkey-w1n5t0n
232d51039d refactor(manifold): delete the OSC bridge twin and dead protocol legs
Phase 1 group 6 (S11, L16).

- S11: deleted osc-bridge/bridge.mjs. It was not compiled output but a separate
  hand-written Node port of bridge.ts (node:dgram + ws vs Deno.listenDatagram).
  The completeness critic settled which twin survives:
  .github/workflows/osc-bridge.yml deno-compiles ONLY bridge.ts into the
  released cross-platform binaries, so bridge.ts plus those binaries are the
  distribution and the .mjs had no consumer in any workflow.
- L16: dead protocol legs left over from the retired playground —
  sendState/sendWeights in osc-client.ts, the legacy bare-array branch in the
  surviving bridge, the unreceivable /nisps/state path, and the unused
  module-output listeners.

Note for the docs phase: docs/specs/backends-spec.md still calls bridge.mjs
"already compiled" (doubly false now), and vcv/ still pushes /nisps/state via
OscServer::sendState with no manifold-side counterpart — flagged, not touched.

Gates: run-all-tests.sh ALL GREEN.
2026-07-21 12:49:25 +02:00
monkey-w1n5t0n
9b686eb312 refactor(manifold): delete the dead console UI stratum
Phase 1 group 5 (S15, S16, S18, S19, L22, L23, L20, L21, L1 delete-half).

- S15: the four-way focus/altitude system. setFocus was never called anywhere,
  so only the 'composite' branch was reachable. Deleted SplitStage,
  ReadoutStrip, InputMini, AltitudeNav, CompactAxis, the UI Focus type/prop,
  the focus branches, stripPinned and the vacuous keyboard gates. MiniMeters
  kept; engine.feedback.setFocus (a different, live thing) untouched.
- S16 + L1: the decorative stratum that rendered real-looking controls driving
  nothing — A/B machinery, the fake seed, seededGradient + weightsRevision,
  snapshots, master volume, bpm, and the learningRate/decay/tame/spreadLevel
  sliders with their Drawers rows. Each was confirmed self-referential first.
  NOTE a real behaviour change falls out of dropping `snapshots`: Undo outside
  an active explore-and-place session used to pop a UI-only snapshot that
  restored noiseCap/seed. It is now simply inactive unless a genuine
  core-backed scratchpad undo exists. Geometric-dislike mode never had a real
  undo primitive in the core, so only the fake path is gone.
- S18: BackendAdvanced.tsx and its Drawers block. It self-described as a
  duplicate of the inline OutputsBackendConfig editor and BOTH rendered in the
  same expanded drawer. OutputsBackendConfig already covers every backend.
- S19: pruned ConsoleCtx to the fields Dock/Drawers/OutputsBackendConfig
  actually read; deleted the Axes type + axes/setAxis and the
  preset/setPreset/offsetActive chain (permanently 'Sculpt'/false).
  KEPT ctx.modes and ctx.setModeId despite having no reader today — the
  Phase 5 instrument picker (§7.6, adopted) is built on exactly that plumbing.
- L22: the 5 dead primitives (Panel, StatusLine, ControlAxis, CurvePlot,
  Sparkline) and their barrel exports, plus the now-dead .mf-axis-input CSS.
- L23: OutputControl/toOutputControl, ModeIconComponent and the BACKENDS
  catalogue; Drawers now reads modeDesc.label/description from OUTPUT_MODES,
  the surviving single catalogue.
- L20: the solo-mode selector's two unimplemented options no longer pretend to
  be selectable.
- L21: FeedbackController vestiges — seed/undoDepth options, maxUndo, and six
  ControllerEngine members nothing called (the finding named three; the other
  three are used on the real EngineApi by debug/probe.ts, a different
  interface, so removing them from ControllerEngine is safe).

Gates: run-all-tests.sh ALL GREEN (typecheck, 33 Playwright specs).
2026-07-21 12:49:25 +02:00
monkey-w1n5t0n
c98d25c255 refactor(wasm): delete 12 dead C-API entries and the weights-publish channel
Phase 1 group 4 (S33, S34, L54).

- S33: removed 12 dead entries across the full 5-layer registration chain
  (bindings.cpp KEEPALIVE -> EXPORTED_FUNCTIONS -> the NispsModule declaration
  table -> the WasmIML wrapper -> the EngineApi facade): nisps_ml_reset,
  example_count, move_weights, feedback_learning_paused, feedback_drag,
  jolt_lr_scale, jolt_tick_lr_ramp, pipeline_state_size/save_state/load_state,
  feedback_placing and feedback_state. Each was grepped against manifold/src,
  manifold/tests, the e2e specs, manifold/tests/wasm-load.ts and
  tests/cpp/parity_wasm.mjs — the parity gate builds its own API via cwrap and
  is a real consumer, so it counts.
  KEPT deliberately: EXPORTED_RUNTIME's heap views + ccall/cwrap (the parity
  harness and wasm-load.ts depend on them), and nisps_ml_feedback_static_output,
  whose C export IS called directly by parity_wasm.mjs even though no TS
  wrapper reaches it. Also dropped parity_wasm.mjs's moveWeights cwrap, which
  was declared but never invoked.
- S34: deleted the publishWeights_ channel — EngineSink.setWeights,
  Spine.setWeights/weights()/liveWeights and every call site. It ran a C->heap
  copy plus a fresh Float32Array allocation at up to 200 Hz into a field
  nothing read. getWeights survives for persistence and the debug probe.
- L54: worklet loader — deleted the unused imports object, the 'c' branch,
  exMap and the duplicate second loop, and replaced the silent `() => 0` stub
  with one that throws, so a missing import fails loudly instead of returning
  plausible zeros into the audio path.

manifold/public/nisps.{js,wasm} rebuilt with the trimmed export list (emcc
3.1.69, the CI-pinned version) and committed — the freshness gate added in
Phase 0 requires it, and the webhook ships this artifact to production.

Gates: run-all-tests.sh ALL GREEN, parity 1273 floats within 1e-5.
2026-07-21 12:48:50 +02:00
monkey-w1n5t0n
ea588e79cd refactor(nisps): delete dead engine/mode mass
Phase 1 group 3 (L3, L9, L4, L5, L6, ST1).

- L3: nisps/modes/voice_space.hpp — entirely dead, no includes anywhere.
- L4: deleted SawOsc and SquareOsc. KEPT SineOsc — a verifier caught that the
  original reviewer's grep missed its live consumer (the firmware selftest);
  re-confirmed here before touching the file.
- L9: removed the no-op VoiceSpace enum/table/setter boilerplate from the five
  engines with no real voice spaces; kept it on PAFSynth, VerbFX and
  ChannelStrip, which have real ones. Every engine member was checked against
  nisps/wasm/bindings.cpp, firmware/ and tests/ for callers first.
- L5: ModeBase::input_dirty_ was write-only state — deleted the flag rather
  than making it gate inference, which would have been a behaviour change.
- L6: VerbFXEngine's delay_to_verb_ (computed 12x per block, never read),
  enable_reverb_, enable_delay_to_reverb_ and the unused set_enable_* setters.
- ST1: rewrote the four engine header comment blocks that described
  implementations which do not exist.

L7 (MEMLCeliumEngine's inert feedback path) is deliberately NOT done — tracing
git history showed feedbackGain went 0.1f (live) -> "0; //0.1f" (explicitly
muted, value preserved) -> dropped entirely in the port. That is a muted
feature, not dead weight, and deleting it would silently lose it. Left intact
pending an operator decision; see the phase report.

Gates: run-all-tests.sh ALL GREEN.
2026-07-21 12:48:50 +02:00
monkey-w1n5t0n
e37f16739e refactor(nisps): delete dead core/ML mass; keep the legacy feedback modes
Phase 1 group 2 (L27, L26, L28, S21, L13, ST6, S20).

- L27: fixed_buffer.hpp + its test + the CMake entry — no consumers.
- L26: dislike_multiplier_ and its doubling/halving bookkeeping — upstream
  InterfaceRL residue that drove nothing. The audit pointed at the wrong test
  file for the surviving reference; the actual assert was in
  test_mlp_geo_dislike.cpp:211, removed here.
- L28: added copy_weights_to(std::span<float>) to FixedStorage and
  DynamicStorage and switched feedback.hpp's take_snapshot/push_undo/nudge to
  it. Drops the permanent whole-net flat_ scratch buffer from FixedStorage and
  the per-gesture double copy. Behaviour-identical: same source values, same
  write order, same RNG draw order in nudge().
- S21 + L13: deleted NISPS_AUDIO_MEM / NISPS_APP_SRAM / NISPS_AUDIO_FUNC —
  zero use sites outside perf.hpp and comments — and rewrote midi_io.hpp's one
  misshapen NISPS_AUDIO_FUNC use as a plain `inline void`. perf.hpp now
  documents only the inlining/hotness macros that actually exist, and
  audio_driver.hpp no longer claims an SRAM discipline the code never had.
- ST6: feedback.hpp's header now describes the four current modes and the
  Geometric default, dropping the retracted "geometric push NOT ported" claim.

S20 — OPERATOR DECISION (§7.1): the four legacy feedback behaviours
(RandomiseOutputs, RandomiseMlp, AvoidStyle::Diffuse, the RandomiseMlp branch of
on_drag) are KEPT, not deleted. They are wanted as building blocks for
experimenting with how different instruments feel under different behaviours.
Each is now marked at its definition as deliberately-retained research reserve
so future audits stop flagging it as dead code.

L25 (the 16 KB firmware loss-history buffer) is NOT done here — see the phase
report; it turned out to be coupled into the shared mlp.hpp, and its fate
belongs with the browser telemetry build (§7.3 / plan §6.5e).

Gates: run-all-tests.sh ALL GREEN.
2026-07-21 12:48:27 +02:00
monkey-w1n5t0n
abb569b287 chore: delete retired-playground artefacts and root relics
Phase 1 of the 2026-07 simplification audit, group 1. All verified dead by
independent grep across scripts/, .github/workflows/, codegen/, manifold/,
firmware/, vcv/, nisps/ and README.md before removal.

- S23 / L55: playground/ — only dist/ was tracked (1.2 MB of bundled JS,
  sourcemaps and a second copy of nisps.wasm). The app itself is preserved on
  branch archive/playground-solidjs and tag playground-solidjs-final.
- S29: the root Playwright rig — playwright.config.js, package.json,
  package-lock.json and tests/e2e/. It served ./playground, a page that no
  longer exists anywhere in the tree, and was invoked by nothing. NOTE: the
  live suite is manifold/tests/e2e/ with manifold/playwright.config.ts, which
  is untouched; every CI Playwright step runs with working-directory: manifold.
- L48: NISPS_CORE_EXTRACTION_PLAN.md + NISPS_CORE_TASKS.md (818 lines of
  extraction relics; docs/specs/plans/ is the sanctioned home for plans).
- L36: data/ — two CSVs from a retired era, plus a tracked LibreOffice lock file.
- L32: .claude/worktrees/agent-ae87fe47/ only. The audit's wording invites
  deleting .claude/worktrees/ wholesale; that would have been destructive —
  the directory holds 12 LIVE registered git worktrees (332 MB), four of them
  on branches with unpushed commits. Verified agent-ae87fe47 is NOT registered
  before removing it; every live worktree is left intact. Cleaning up the rest
  is a separate operator decision.
- ST10 fallout: dropped the dead `port-solidjs` branch trigger from ci.yml and
  the dead `feat/manifold-mission` / `feat/vcv-dist` triggers from
  vcv-plugin.yml — those branches are stale and the SolidJS target is retired.

tests/cpp/ is untouched. Gates: run-all-tests.sh ALL GREEN.
2026-07-21 12:47:58 +02:00
monkey-w1n5t0n
aa60fbb466 fix(build): resolve a bare EMCC command name in build-wasm.sh
Third failure uncovered by the restored pipeline. With checkout and the C++
build fixed, CI reached the WASM step and died on `[build-wasm] emcc not found
at emcc`: ci.yml passed `EMCC: emcc`, but the script's existence check
(`[[ ! -x "$EMCC" && ! -f "$EMCC" ]]`) only understands paths, so it looked for
a file literally named "emcc" in the working directory. This has been broken
for as long as the override existed; it was invisible because CI never got past
checkout to run it.

Resolve a bare command name through `command -v` before the check, and drop the
now-redundant EMCC override in ci.yml — setup-emsdk already puts emcc on PATH,
which is what the script's own default looks for.

Verified by invoking it the way CI does: `EMCC=emcc bash scripts/build-wasm.sh`
now builds. Incidentally confirms the new freshness gate is sound — the local
toolchain is the same pinned emcc 3.1.69, and the rebuild reproduced the
committed artifact byte-for-byte.

run-all-tests.sh ALL GREEN.
2026-07-21 12:14:21 +02:00
monkey-w1n5t0n
3552ee46f6 fix(core): order RingBuffer::buf_ before the atomics to unblock the CI compiler
First real signal from the restored pipeline: with checkout fixed, the cpp-tests
job got far enough to fail at `-Werror=stringop-overflow` in ring_buffer.hpp on
GitHub's GCC 13, a failure that had been invisible behind the broken submodule
checkout for a month. Reproduced locally against gcc 13.4.0 (local default is
gcc 14, which does not fire).

The warning is a false positive: GCC anchors the destination object to the
member at offset 0 and reports `buf_[head & kMask]` as writing past
`head_._M_i` (size 8) at offset [16, 268] — offsets that are precisely
buf_[0..63] of a 64-entry, 4-byte ControlEvent array. Adding an explicit
`__builtin_unreachable()` bound hint does not help, because the index range was
never what GCC got wrong. Declaring buf_ first anchors the analysis correctly.

Not a suppression, and behaviour-preserving: RingBuffer has exactly one
production user (ModeBase::events_) and is never serialized, copied, or sent
over a wire, so member order is unobservable. Reasoning recorded at the
declaration so nobody "tidies" the order back.

Verified: full ctest suite green under gcc 13.4.0 (the CI compiler) as well as
gcc 14.2, and scripts/run-all-tests.sh ALL GREEN (parity 1273 floats within
1e-5, lint, 33 Playwright specs).
2026-07-21 12:08:52 +02:00
monkey-w1n5t0n
8c249ea8af ci: restore verification — reachable submodule pin, codegen + WASM freshness gates
Phase 0 of the 2026-07 simplification audit (plan §1). CI has been 100% red on
main since 2026-07-13 and every "gates green" claim since rested on local runs.

- S7 / critic gap 2: push memllib `feat/nisps-core-swap` (3 commits incl. the
  pin b37fc53) to monkey-w1n5t0n/memllib and repoint .gitmodules at the fork.
  Those commits existed on exactly one disk; `git ls-remote` now resolves the
  pin, so `submodules: recursive` checkout and fresh clones work again. Drops
  the compensating unreachable-pin error paragraph in build-firmware-arch.sh.
- S24 / S31: the manifold-tests job regenerates from schemas/, runs the codegen
  golden test, and fails on a dirty diff — the "schema changes ship with both
  generated outputs" rule is now enforced rather than assumed.
- S32: a WASM freshness gate runs the parity harness against the *committed*
  manifold/public/nisps.{js,wasm} before the CI rebuild overwrites it. That
  artifact is what the webhook ships to production, so a stale commit now fails
  loudly instead of shipping.
- critic gap 3 / operator decision §7.4: the VPS webhook
  (~/.config/webhooks/meml-deploy.sh, not in this repo) waits for the `CI`
  workflow to conclude success on the pushed SHA before building. Fail-closed;
  MEML_SKIP_CI_GATE=1 for an emergency hand-deploy. Verified the gate query
  returns `failure` for fa37047, i.e. it would have blocked that deploy.
- S31: corrected run-all-tests.sh's false "single command CI invokes" header.

Docs moved with the code: ALIGNMENT defect 1 deleted (resolved) and the rest
renumbered; MAP.md's unreachable-pin warning replaced with the fork pin and a
pointer to the §7.5 vendoring decision; ONBOARDING documents the deploy gate
and the tracked-WASM-ships-to-prod hazard; plan §1 marked burned down.

Gates: scripts/run-all-tests.sh ALL GREEN (ctest 4/4, parity 1273 floats within
1e-5, lint, typecheck, 33 Playwright specs).
2026-07-21 11:57:32 +02:00
monkey-w1n5t0n
fd0aee2354 docs(audit): simplification audit 2026-07 — recon findings, phased plan, ALIGNMENT rewrite
66-agent adversarially-verified audit vs the five-bullet one-core vision.
113 findings: CI red since 2026-07-13 (memllib pin on no remote), ungated
deploys, unshared mode layer, no curated/advanced split, dead-mass inventory.
Recon: docs/specs/recon/simplification-audit-2026-07.md (immutable).
Plan: docs/specs/plans/simplification-plan.md (proposal; phases gated on
operator adoption, §7 decisions). ALIGNMENT rewritten; MAP flatly-false
lines fixed (phantom MEMLCelium-upstream entry, exploration.ts, daisysp
non-submodule, pre-P5 sentence, perf-attr claims); MAIN registry updated.
2026-07-21 01:24:35 +02:00
monkey-w1n5t0n
fa3704765c docs(plan): P6 burned down; execution wrap-up — all phases landed, hardware chokepoints remain operator items 2026-07-18 13:03:21 +02:00
monkey-w1n5t0n
f028f1256d feat(vcv): merge P6 — module reunified onto core MLP (closes vcv delta #5) 2026-07-18 13:02:28 +02:00
monkey-w1n5t0n
1b69254de2 feat(vcv): reunify module onto core MLP — thin iml.hpp adapter (P6)
Replace the vendored runtime MLP in vcv/src/iml.hpp (DetRng + 3D-weight-store
MLP + Dataset + IML) with a THIN, Rack-free adapter over the shared core:
nisps::ml::MLPCore<nisps::ml::DynamicStorage> (8->[16,24,16]->16, the P2 dynamic
case), nisps::Rng, and the core MLP's own FIFO dataset. Behaviour changes from
the vendored approximation to core-exact firmware/WASM semantics.

- MEMLNaut.cpp: staged/pending weight buffers and patch JSON now use the core's
  flat [weights..][biases..] vector (nisps::IML<float>::Weights); patch version
  bumped to 3. Double-buffer / single-writer threading discipline unchanged.
- New ctest tests/cpp/test_vcv_iml_parity.cpp: seeded train/infer/move_weights
  session through the adapter is memcmp-equal to a bare MLPCore<DynamicStorage>.
- Docs: vcv-module.md delta #5 marked CLOSED (2026-07-18); MAP.md vcv/ updated.

Closes vcv-module.md delta #5.
2026-07-18 13:01:28 +02:00
monkey-w1n5t0n
e1713ae41f docs(plan): P5 burned down 2026-07-18 12:47:15 +02:00
monkey-w1n5t0n
161ce155e4 feat(manifold): merge P5 — MF_MODES from schema truth; per-mode engine dims real 2026-07-18 12:46:24 +02:00
monkey-w1n5t0n
28e394ddf3 docs+ci: P5 doc sync; codegen golden wired into run-all-tests stage 5 2026-07-18 12:45:45 +02:00
monkey-w1n5t0n
6c499e6826 feat(manifold): P5.2/P5.3 — derive MF_MODES from schema truth + per-mode engine dims
Schema-backed modes in console/model.ts are now DERIVED from the codegen
schemas in src/modes/generated/ (source of truth): real param names/groups/
count, plus each mode's ml net shape (MFMode.ml) and schema engine_id. A thin
manifold OVERLAY supplies only label/glyph/ModeClass/input/ordering. New
browser-viable modes xiasri + slp_workshop get derived entries; schema-less
visualizer + c15 stay hand-written on DEFAULT_MODE_ML. Schema min/max/default/
label/curve surface as engine-unit metadata (schemaMin/... on MFParam) without
touching the 0..1 routing semantics.

Switching instrument mode reshapes the runtime-shaped WASM net to the mode's
schema ml config (ConsoleApp effect keyed on [engine, modeId]; no confirm
modal). Boot lands paf_synth dims (4->[10,10,14]->33) once WASM is ready. The
P2.3 axis-count reshape offer still reads the engine's live inputSize and does
not spuriously prompt on a mode switch.

Adds schema-modes.spec.ts (P5 gate): drives switches via a new window.__mf
debug seam and asserts describe() dims, getWeights count, output length/bounds,
and UI param count FROM the imported schemas; spot-checks trainAsync after a
switch. Updates reshape/probe-api/geo-dislike specs to assert from the boot
mode schema instead of the retired fixed 32/126 shape.

All gates green: typecheck, unit (9), build, e2e (33).
2026-07-18 12:45:06 +02:00
monkey-w1n5t0n
c45b91a5f9 feat(codegen): P5 — TS emission targets manifold/src/modes/generated/
- generate.ts re-activates the TS emitters against manifold (types.ts,
  per-mode <id>_schema.ts, index.ts); 9 schemas emitted
- firmware-fit check: exactly 3 hidden layers (fixed 4-layer topology) and
  every dim within (0, 4096] (the browser kMaxDim)
- golden test restores the TS case against the manifold path (byte-
  identical to the retained P1-era snapshot)
2026-07-18 12:28:16 +02:00
monkey-w1n5t0n
bd35432797 docs(plan): P4 burned down — pipelines/curves in core, browser firmware-exact
MAP: nisps/pipeline entry + manifold engine listing; ALIGNMENT: curve-maths
unification note; plan gate evidence incl. the proven-inherent f32 momentum
drift and parity stage 7.
2026-07-18 12:23:14 +02:00
monkey-w1n5t0n
a7c1f29aa6 feat(manifold): merge P4 — pipelines + curves run the WASM core; TS ports deleted 2026-07-18 12:22:21 +02:00
monkey-w1n5t0n
846a0c373a feat(manifold)!: route input/output pipelines + curves through the WASM core (P4.3/P4.4)
One-core-engine P4.3/P4.4: the input/output pipeline processing and the curve
catalog now live in the C++/WASM core (nisps/pipeline/*, nisps/core/math.hpp).
The TS ports are deleted and the browser drives the WASM chains.

Engine:
- WasmIML owns a nisps_pipeline_create handle + bridge buffers and exposes
  setInputConfig (TS InputConfig → 15-float wire), processInput, resetInput,
  setOutputConfig (Infinity slew → 0), setOutputFreezeMask, processOutput
  (in place), resetOutput, curveApply, curveApplyBatch (chunked). Handle +
  buffers created in init_, freed in dispose, output-sized buffers realloc'd
  on reshape.
- Spine routes setInputs through iml.processInput/processOutput (state lives
  C++-side); config source-of-truth stays TS-side and is pushed on attach /
  setInputConfig / setOutputConfig. Preserves ?debug=1 fixed-dt determinism
  (same dt fed to the WASM calls). EngineApi gains setInputConfig/
  setOutputConfig/curveApply/curveApplyBatch.
- New types-only modules: pipeline-types.ts (InputConfig/OutputConfig +
  defaults + wire int mappers) and curve-catalog.ts (CurveName + name→id).
  types.ts declares the pipeline/curve C ABI. engine barrel updated.
- DELETED src/engine/{input-pipeline,output-pipeline,curves}.ts.

Tests (P4.4 gate — recorded-gesture regression):
- pipeline-golden.test.ts now loads the built WASM (indirect-eval shim,
  tests/wasm-load.ts) and drives the frozen gesture/output fixtures through the
  C++ chains, honouring the per-event dt clock contract. Tolerance 1e-5
  (non-momentum drift <5e-7). The 3 momentum configs carry 1e-2: proven-inherent
  f32 drift (a byte-faithful f32 port of the exact original algorithm matches
  the WASM to <6e-8 while both diverge from the f64 capture by ~7-9e-3), NOT a
  core bug.
- curves-golden.json: linear/square/sqrt/centered_power kept as the original
  f64 captures (C++ matches within <3e-8); exp/log/sigmoid/cubic RE-BASELINED
  from the WASM (deliberate switch to firmware-exact k=1 exp/log, slope-6
  sigmoid, true cubic x^3). Provenance recorded in-file.
- _generate.ts rebuilt as the WASM curve re-baseline tool; pipeline-golden-lib
  trimmed to pure data builders.

Docs: fixtures/README.md + manifold/ONBOARDING.md updated.

Gates: typecheck, bun test (9), vite build, playwright e2e (27) all green.
2026-07-18 12:21:35 +02:00
monkey-w1n5t0n
1672fe3474 feat(pipeline)!: P4 core — input/output chains + curve catalog in nisps/
- nisps/pipeline/input_chain.hpp: faithful f32 port of the manifold input
  pipeline (invert→deadzone→circular clamp→momentum-modulated zoom→centred
  power→EMA→momentum update). Caller-supplied dt, internal accumulated
  clock (no wall clock — deterministic; matches the P1 fixtures' clock
  contract). Fixed-capacity velocity ring; serialisable state.
- nisps/pipeline/output_chain.hpp: curve→EMA→slew→freeze(+per-output mask)
  chain, capacity-templated (browser cap 4096; firmware would use NOut).
- nisps/core/math.hpp: + centered_power(x, exponent) (both chains use it).
- bindings: nisps_pipeline_create/destroy, nisps_input_set_config(15-float
  wire layout)/process/reset, nisps_output_set_config/set_freeze_mask/
  process/reset, pipeline state save/load, nisps_curve_apply(+batch)
  (ids 0-6 = Curve enum, 7 = centred power).
- parity v5 Stage 7: rational (transcendental-free) traces through both
  chains (2 configs each) + full curve catalog — 1273 floats PASS, the
  pipeline floats bit-identical native↔WASM.
- ctest test_pipeline.cpp: deadzone remap, circular clamp, zoom+freeze,
  sticky anchor, frame-rate-independent EMA, momentum zoom-out/recovery,
  state round-trip, slew, freeze gate/mask, reseed-on-length-change,
  centred-power endpoints.

Part of one-core-engine-refactor P4; the manifold TS switch follows.
2026-07-18 11:52:53 +02:00
monkey-w1n5t0n
1f4513802f docs: P3 software burned down — ADR §8 ALIGNMENT updates, MAP feedback/replay entries
- ALIGNMENT: retract the 2026-06-18 'geometric push not ported' accepted
  divergence (it IS ported); record the two deliberate divergences
  (useRandom via nisps::Rng; synchronous per-press SGD vs upstream's
  shuffled TrainBatch — behavioural parity with firmware, 1e-5 native↔WASM)
- MAP: nisps/ml gains replay/geo_push/warm_start; feedback is storage-
  policied with Geometric default + Diffuse legacy
- AGENT-REFERENCE: Jolt/OU/dislike TS-math limitation closed
- plan: P3 software marked landed; chokepoint A pending hardware + bug
  10c3e55c (explore wiring linker-GC'd out of PAFSynth ELF, pre-existing)
2026-07-14 04:56:18 +02:00
monkey-w1n5t0n
a36b58dce8 feat(manifold): merge P3 TS switch — geometric dislike + jolt/OU on the C++ core 2026-07-14 04:55:25 +02:00
monkey-w1n5t0n
7fd0dda3a2 feat(manifold): geometric dislike + jolt/OU via shared core (one-core-engine P3)
TS half of P3.1/P3.2/P3.3: wire the new WASM exports and delete the TS
approximations now that geometric-dislike, jolt, OU, and the seeded RNG live
in the C++/WASM core.

- types.ts/wasm-iml.ts: bind + wrap dislike_geometric, store_positive,
  positive/negative_count, set_avoid_style, jolt_press/step/release/active/
  lr_scale/tick_lr_ramp, explore_intensity/get/apply. Weight-mutating wrappers
  republish weights (version bump); explore_apply reuses feedbackBuf.
- engine-api: extend .feedback (dislikeGeometric/storePositive/counts/
  setAvoidStyle) + new .explore facade. thumbsDown now passes the HEARD
  (routed) vector, not the raw MLP output (raw == net output => inert cold-start).
- feedback/controller.ts: delete dislikes[] + applyDislikeBias + both C++ GAP
  blocks + the SeededRng field; dislike() -> core dislikeGeometric (returns
  action, 15 => cold-start prompt); like() feeds the centroid via the core
  thumbsUp; getState() exposes positive/negative counts. Delete feedback/rng.ts.
- engine/exploration.ts: execute the P3 SWAP POINT -> drive engine.explore.*;
  delete engine/jolt.ts + ou-explore.ts. UI surface unchanged.
- ConsoleApp: one-time cold-start banner (British spelling), routed heard vector
  at the dislike call site.
- App.tsx/spine.ts: under ?debug=1 pin a fixed seed + fixed per-tick dt so the
  probe/e2e are deterministic (production keeps time-seeded, real-time dt).
- tests: new geo-dislike.spec.ts; probe gains dislikeGeometric/storePositive/
  feedbackCounts/setAvoidStyle; the thumbsDown probe test now drives a real
  distinct-heard-vector dislike (bare thumbsDown on the net's own output is
  correctly inert under the geometric core). 27 e2e + 9 unit green.

Docs: manifold/ONBOARDING.md engine+feedback sections synced.
2026-07-14 04:54:27 +02:00
monkey-w1n5t0n
9490e20a7a feat(ml)!: P3 core — geometric dislike in nisps/, jolt/OU + geo ABI
Geometric dislike (rl-feedback-design §2.1/§4; upstream InterfaceRL @
0a541cc ported verbatim, constants included):

- nisps/ml/replay.hpp: ReplayView over storage-owned buffers — deepen-or-
  store negatives (dedup 0.05, clamp -16), k-NN positive centroid with
  deterministic index tie-break + fixed accumulation order, proportional
  decay (0.0025*max(|r|,1)) + eviction, order-preserving compaction.
- nisps/ml/geo_push.hpp: push-away target (pushStep clamp(|avgNeg|,.25,1)
  *0.5, taper /(1+len), useRandom on len<=1e-4 via nisps::Rng — the single
  deliberate divergence from libc rand()), negLRRatio 0.5-0.4*negFraction.
- mlp.hpp: train_targets(input, computed-target, lr, out_mask) — trains
  toward computed targets (negative lr = cold-start train-away); solo/
  focus gating zeroes masked derivs.
- feedback.hpp: AvoidStyle {Geometric (new default), Diffuse (legacy
  move_weights, kept for A/B)}; dislike_geometric() collapses upstream's
  press+optimise into one synchronous call; on_up in geometric Avoid
  feeds the positive centroid; dislike-multiplier bookkeeping. Storage
  gains replay buffers (Fixed: ReplayCap=32 firmware default ≈ +8KB SRAM;
  Dynamic arena: cap 64).
- bindings: nisps_ml_feedback_{dislike_geometric,store_positive,
  positive_count,negative_count,set_avoid_style} + P3.2 jolt/OU ABI:
  nisps_ml_jolt_{press,step,release,active,lr_scale,tick_lr_ramp},
  nisps_ml_explore_{intensity,get_intensity,apply} (OUNoise<4096>
  over-provisioned; same code the firmware ModeBase runs).
- parity v4: Stage 6 scripted geometric session (2 likes → 2 dislikes,
  f32-exact heard vectors via Math.fround) — 961 floats PASS at 2.4e-7.
- tests: test_mlp_geo_dislike.cpp (replay dedup/deepen/clamp, centroid
  tie-break, push direction/taper/mask/clamp, cold-start inertness +
  train-away, determinism, Diffuse legacy); legacy Avoid test pinned to
  Diffuse per the ADR's deliberate-break note.

Firmware: PAFSynth .text/.data unchanged (geometric path not referenced
by current glue). NOTE: discovered pre-existing bug 10c3e55c — the
explore/place wiring is linker-GC'd out of the PAFSynth ELF (predates
this refactor; evidence in the ergo task).
2026-07-14 04:16:21 +02:00
monkey-w1n5t0n
7e957457cd docs: P2 complete — burn down plan, close ALIGNMENT defect #3 (fixed WASM arity)
- plan P2 marked landed (hardware timing spot-check deferred to the
  chokepoint-B session on physical MEMLNaut)
- ALIGNMENT: defect #3 resolved by the runtime-shaped browser MLP
- MAP/AGENT-REFERENCE/nisps-wasm-README: describe the honoured-dims
  create + reshape ABI; note per-mode dims become schema-real at P5
- fresh wasm artifacts from the gate run
2026-07-14 03:56:04 +02:00
monkey-w1n5t0n
7e4dc26f6e feat(manifold): merge P2.3 — runtime reshape wiring + reset-on-reshape modal 2026-07-14 03:55:07 +02:00
monkey-w1n5t0n
3af92b625a feat(manifold): runtime-shaped net reshape with confirm modal (P2.3)
Wire the runtime-shaped WASM MLP (one-core-engine P2) through the manifold:

- WasmIML.reshape(dims, spread): calls nisps_ml_reshape, re-describes the
  instance, reallocates every dim-dependent heap buffer, refreshes weightCount,
  clears the TS Dataset mirror (C-side resets), drops the stale training worker,
  and pushes the new shape through the sink.
- EngineApi.reshape exposes it and re-ticks the spine so outputs/audio reflect
  the new net. Spine already tolerates the arity change (buffers resize, version
  bumps); documented.
- Training worker protocol carries the current hidden dims; the worker
  ensureNet()s its mirror net to match after a reshape.
- ConsoleApp offers the reshape behind ReshapeModal on an active-layout CHANGE
  (never on load; default 32-input over-provisioned head + zero-padding
  preserved when declined). British copy, reset-on-reshape.
- Drawers: delete the stale even/odd blending note; honest dedicated-dimensions
  line + net-arity chip.
- Probe: __nisps.reshape(nIn) / .describe(); e2e reshape.spec (default 32/126,
  reshape to 4, describe reports 4, bounded outputs, weight count 3148→2868,
  spine still propagates). All 25 e2e pass (20 existing + 5 new).
- ONBOARDING: refresh the reshape status + stale hardwired-arity gotcha.
2026-07-14 03:54:12 +02:00
monkey-w1n5t0n
b6819fd26f feat(wasm)!: P2.2 — nisps_ml_create honours dims; runtime-shaped browser MLP + reshape
Operator-approved ABI change (P2 stop-point). The WASM MLP is now
MLPCore<DynamicStorage>:

- nisps_ml_create(input, output, hidden[3], n, seed) honours its args;
  non-positive/null fall back to the historical 32→[10,14,18]→126, so
  every pre-P2 caller (manifold, worker, parity harness) stays
  bit-identical. Invalid/oversized dims (>4096) → null.
- NEW nisps_ml_reshape(ml, in, out, hidden, n, spread): fresh net at the
  new dims, warm-started via nisps/ml/warm_start.hpp (overlapping region
  copied; rest keeps spread init); feedback controller re-created (state
  resets — reset-on-reshape modal is the front-end contract). Failure
  leaves the old net untouched.
- nisps_ml_describe(ml, out): takes the handle; null reports defaults.
- FeedbackController got the same storage split: algorithms in
  FeedbackControllerCore<FbStorage>; FixedFeedbackStorage keeps firmware/
  tests source-identical via the old alias; DynamicFeedbackStorage (one
  arena) sizes to the runtime net. Firmware .text unchanged (122692).
- MLHandle: per-instance scratch vectors; dropped the dead 2MB
  batch_out_scratch.
- TS: types.ts decls (+_nisps_ml_reshape), wasm-iml re-describes the
  created instance, worker carries a shape-contract note for P2.3.

Verified: ctest 4/4 incl. new warm-start grow/shrink test; reshape ABI
smoke (dims honoured, overlap survives, invalid rejected, outputs
bounded); parity PASS unchanged (2.4e-7); lint clean; manifold 9 unit +
20 e2e green; firmware .text 122692 (+0.30% vs pre-P2 baseline).
2026-07-14 03:38:06 +02:00
monkey-w1n5t0n
8a19e5b52c refactor(ml)!: P2.1 storage-policy split — MLPCore<Storage>, fixed + dynamic models
Algorithms (forward, backprop/SGD, init, move_weights, diagnostics) now live
once in MLPCore<Storage> (nisps/ml/mlp.hpp). Storage models:

- FixedStorage (storage.hpp): template-sized std::array, zero heap. The
  classic MLP<NIn,H1,H2,H3,NOut,...> is an alias preserving kInput/kHidden*/
  kOutput/kNumLayers/weight_count() constexpr — firmware + bindings + modes
  compile unchanged.
- DynamicStorage (dynamic_storage.hpp): runtime dims, ONE arena allocation
  at construction, nothing per-call. #error under NISPS_TARGET_EMBEDDED
  (new macro in core/perf.hpp); sole lint-cpp.sh heap-allowlist entry, plus
  a lint check that fails if the #error guard disappears.

Verification:
- new ctest test_mlp_storage_parity: fixed↔dynamic BIT-identical across
  init/draw/inference/train(FIFO)/move_weights(pin mask)/eval_loss/
  layer_stats/set_weights/infer_batch/reset; invalid+moved-from inert
- golden ML vectors (pre-refactor constants) pass → bit-stable refactor
- native↔WASM parity PASS, max delta unchanged (2.4e-7)
- chokepoint B compile: PAFSynth .text 122324→122692 (+0.30%, ±1% budget);
  RAM +416B (eval scratch)
- fix: firmware-common.sh used bare 'python' (absent here) → ${PYTHON:-python3}

Part of one-core-engine-refactor P2. nisps_ml_create ABI untouched (P2.2 is
an operator stop-point).
2026-07-13 23:47:03 +02:00
monkey-w1n5t0n
43ef6d5dae chore(submodule): re-pin memllib gitlink to the real working commit b37fc53
The recorded 4733ca0 was unfetchable ('not our ref') and the tree has been
built against b37fc53 (feat/nisps-core-swap) since June. NOTE: b37fc53 is
not on the memllib remote either — fresh clones still need the MAP.md
gotcha; pushing feat/nisps-core-swap to a reachable remote is the real fix
(operator decision, flagged in session report).
2026-07-13 23:33:53 +02:00
monkey-w1n5t0n
d1f1f45f6b docs(plan): P1 burned down; wire golden fixtures test into bun run test
- manifold package.json test script now includes tests/pipeline-golden.test.ts
- MAP.md: manifold exploration modules + tests/fixtures entries
- ONBOARDING.md §engine: exploration.ts + spine setOutputMorph + P3 swap point
- plan P1 marked landed with gate evidence
2026-07-13 23:30:28 +02:00
monkey-w1n5t0n
0ee7f1472c feat(manifold): merge Jolt press + OU explore UI shells (interim TS math, P3 swap point) 2026-07-13 23:28:21 +02:00
monkey-w1n5t0n
b9f739a56d test(manifold): merge probe-API + spine-invariant e2e specs 2026-07-13 23:28:21 +02:00
monkey-w1n5t0n
21b223abd3 test(manifold): merge P4 golden parity fixtures (pre-P4 capture) 2026-07-13 23:28:21 +02:00
monkey-w1n5t0n
0d1a2102eb feat(manifold)!: P1 — retire playground/, manifold is the sole browser app
Playground archived at branch archive/playground-solidjs (tag
playground-solidjs-final) and deleted from main. Retargets:

- run-all-tests.sh stage 5 → manifold (typecheck + bun test + build +
  playwright via non-snap node runner, BUILD-PLAN gotcha)
- ci.yml playground-tests → manifold-tests; osc-bridge.yml → manifold/osc-bridge
  (was already broken: playground/osc-bridge no longer existed)
- codegen: TS emission target removed (returns at P5 → manifold); golden
  test now C++-only; TS emitters retained dormant
- .gitignore: manifold/osc-bridge paths; drop dead playground faust exception
- docs: AGENTS.md gates, README quickstart, MAP.md, ALIGNMENT.md (C15 now
  archive-only — defect #1 updated), AGENT-REFERENCE.md, specs/MAIN.md

Part of docs/specs/plans/one-core-engine-refactor.md P1.
2026-07-13 23:27:56 +02:00
monkey-w1n5t0n
9056ac3f5e feat(manifold): port Jolt + OU explore gestures as UI shells
Bring the two playground-only exploration UIs into manifold ahead of the
playground's retirement (one-core-engine refactor §P1):

- Jolt: press-and-hold continuous weight-morph, release to freeze.
- Explore: Ornstein-Uhlenbeck exploration intensity on the output vector.

Interim TS maths ported verbatim from the retired playground modules
(engine/jolt.ts, engine/ou-explore.ts). The ExplorationController
(engine/exploration.ts) is the single P3 swap boundary: it drives Jolt via the
existing EngineApi get/set-weights + process route, and OU via a new inert-by-
default output-morph hook on the spine. In §P3 only that module changes to call
nisps_ml_jolt_press/release + nisps_ml_explore_intensity.

UI lands in the Learning drawer (Jolt hold-button + Explore slider), monochrome-
consistent, British copy. Gates green: typecheck, build, Playwright smoke.
2026-07-13 23:27:40 +02:00
monkey-w1n5t0n
469a38ae16 test(manifold): port probe-API contract + spine-invariant e2e from playground
Migrate the specs worth keeping from playground/tests/e2e (retired in P1) into
manifold/tests/e2e, adapted to Manifold's probe surface:

- probe-api.spec.ts: the window.__nisps debug-probe contract (ready, bounded
  outputs, example count, randomise, setInputs inference, thumbsUp/Down,
  addExample, train loss non-increasing, async train, clearExamples, evalLoss,
  inferBatch, getLayerStats, getWeights). Retargeted to MLP<32,10,14,18,126>
  (weight_count 3148) and Manifold's direct addExample/routedOutputs surface;
  dropped the playground's __init/iml-poke escape hatches and stream-pending
  skips.
- spine.spec.ts: the spine invariant — setInputs -> processed -> ml -> routed
  yields bounded, consistent routed outputs; the probe stays alive across dock
  output-mode switches (driven via the real selector UI, replacing the
  playground's localStorage-reload mode cycling).
- helpers.ts: loadProbe(?debug=1 + cleared storage + __ready wait), settleInputs
  for EMA convergence, bounded/changed assertions.

Dropped playground UI specs (ui-interactions, persistence, mode-registry list)
that die with the playground chrome. No probe.ts changes needed.
2026-07-13 23:26:00 +02:00
monkey-w1n5t0n
fb0228e6c3 test(manifold): capture golden parity fixtures for TS pipelines/curves before P4
Records canonical gesture trace, curve catalog samples, and input/output
pipeline outputs from the current TS implementations, plus a bun-test drift
guard that re-runs them against the fixtures within 1e-9. Serves the P4
one-core-engine gate: same pointer trace -> same routed output pre/post the
C++/WASM migration.
2026-07-13 23:25:31 +02:00
monkey-w1n5t0n
29dc88be3a chore(build): P0 plumbing — WASM build/parity retarget to manifold/public
- build-wasm.sh emits to manifold/public/ (transitional copy to
  playground/public/ until P1 retires the playground)
- parity-check.sh + parity_wasm.mjs read the manifold artifact
- fix stale MLP<2,...> arity in AGENT-REFERENCE.md + nisps/wasm/README.md
- gitignore .claude/worktrees/
- plan one-core-engine-refactor.md: P0 marked landed

Gate: run-all-tests green; parity PASS from manifold artifact (2.4e-7);
manifold builds against freshly-built nisps.wasm.
2026-07-13 23:14:23 +02:00
monkey-w1n5t0n
c936bf75c9 docs(ergo): update task guidance 2026-07-13 22:58:26 +02:00
monkey-w1n5t0n
d466dd3c48 chore(agents): converge project instructions 2026-07-13 22:58:26 +02:00
monkey-w1n5t0n
a9623d71d7 docs(plan): one-core-engine refactor plan — firmware+manifold reunification
Locked (operator 2026-07-13): retire playground/, all algorithms into
nisps/ C++, storage-policy MLP (fixed template on RP2350 / runtime-shaped
in WASM+VCV), codegen serves manifold, VCV last phase. Supersedes
BUILD-PLAN 'multiple WASM modules' MLP decision; fixes stale MLP<2,...>
in MAP.md.
2026-07-13 23:38:05 +03:00
monkey-w1n5t0n
fa80a305d9 Merge remote-tracking branch 'origin/main' 2026-07-13 23:17:08 +03:00
monkey-w1n5t0n
19f9082dd7 wip(firmware): select SLPWorkshop mode variant 2026-07-13 23:17:08 +03:00
monkey-w1n5t0n
45f3ca5cae docs: restructure design docs into docs/specs (adr/plans/recon), update path references 2026-07-13 23:15:46 +03:00
monkey-w1n5t0n
c7056e20e8 feat(manifold): follow-mouse knob (dbl-click mark) + 1/2 verdict keys
- Double-click the input mark to enter follow-mouse mode: a window-level
  pointermove listener maps the whole viewport onto the surface's [0,1]²
  space so the knob tracks the cursor across the entire UI. Esc or a second
  double-click exits; a badge + hidden cursor signal the active state. Local
  pan/long-press and auto-drift are suppressed while following.
- Keys 1 = thumbs-down/explore (perturb), 2 = thumbs-up (commit), on the
  window keydown handler so they fire even in follow-mouse mode. Drawers
  learn/inputs lose their 1/2 shortcuts (3-5 keep route/settings/help).
- Sync help keymap (Drawers) + ONBOARDING stages table.
2026-06-29 00:45:45 +02:00
monkey-w1n5t0n
d2b0427ec6 feat(manifold): default to geometric push + add Clear examples button
Geometric-dislike ('Push away', Mode 1) is now the default feedback mode
instead of explore-and-place, which works poorly. Add a Clear button to
the Learning drawer that forgets all recorded examples and wipes the
on-map visuals (feedback markers + placed-anchor pins) via the existing
ctx.onClear, now also resetting pins.
2026-06-29 00:36:32 +02:00
monkey-w1n5t0n
031c7f97ff fix(manifold): keep input knob inside the circular area for mouse + gamepad
The on-screen circular input disc let both the mouse and the gamepad drive
the knob outside the visible circle.

- Gamepad: each stick axis was clamped to [0,1] independently, so a full
  diagonal push reached the square corner. Clamp the stick *vector* to the
  unit disc (radially symmetric) before mapping to [0,1]; covers both sticks.
- Mouse: the circular variant clamped in normalised [0,1]^2 but the canvas
  drew the knob across the full non-square panel against the inscribed circle,
  so the disc rendered as an ellipse that spilled past the rim. Map both the
  pointer and every drawn position (knob/pins/markers/trail/flash) through the
  inscribed-circle geometry, and disc-clamp the auto-drift.
2026-06-29 00:14:49 +02:00
monkey-w1n5t0n
c986377b4c feat(firmware): reposition gesture — relocate an existing example to a new input
Add a "grab → move → drop" gesture that moves an existing positive
example's output to a new input position, preserving the output. This is
the new core's home for upstream InterfaceRL's drag-store/reposition-commit,
distinct from Explore→Place (which places newly-auditioned scratchpad sounds).

- nisps/ml/feedback.hpp: begin_reposition()/commit_reposition()/repositioning().
  Reuses the Placing state (static_output holds the carried vector) but a
  reposition_ flag makes commit AND the mode-switch teardown SKIP the weight
  restore — the real net is never set aside here, so restoring snapshot_ would
  clobber the live trained net. Guards cancel_place + abort_explore_place.
- firmware glue: state-gate Toggle B. Exploring → reroll/nudge (unchanged);
  Idle → MomB1 grab, MomB2 drop (commit + add_example + train). The 4D variant
  has no joystick button, so the gesture lives on the momentary toggle. Also
  fix a stale top-of-file control-map comment that contradicted the bindings.
- tests: 4 reposition cases (hold without snapshot; commit stores carried
  output with no restore; mode-switch aborts without clobber; begin-only-Idle).
- MAP.md: document the full ExploreAndPlace lifecycle + reposition wiring.

Audio-hold (carrying the sound audibly during the move) remains the existing
unwired static_output() TODO and affects Explore→Place identically.
Firmware compile unverified (no arduino-cli); host tests + lint pass.
2026-06-28 23:45:27 +02:00
monkey-w1n5t0n
ac84ed5ed9 Merge remote-tracking branch 'origin/main' into workshop/synth-fw-audit 2026-06-28 23:09:50 +02:00
monkey-w1n5t0n
2e75194fbb docs(spec): merge the two SLP-Workshop specs into one
Consolidate docs/slp-workshop/SPEC.md into docs/specs/slp-workshop-
firmware.md so the project has a single SLP-Workshop spec. The unified
spec is now two parts:
- Part I (§1-8, shipped/stable): the mode + Jolt & OU-explore gestures.
- Part II (§9-13, planned): the output-mode evolution — Continuous/Rhythm
  stream model, gate sequences, CV/MIDI output config, Manifold UX, and
  the locked operator decisions (2026-06-28).

Frontmatter bumped stable -> evolving (the doc now spans shipped +
planned). Donor docs/slp-workshop/SPEC.md removed; README.md row and
MAP.md ## Specs note updated. No code change.
2026-06-28 23:09:49 +02:00
monkey-w1n5t0n
94b91f1cc0 fix(manifold): correct SandwichStage 3D depth sort so tilt occludes right
The painter's-algorithm sort was descending, but z2 (depth) increases
toward the camera in both the top-down and tilted-from-above views, so
nearest was painted first (underneath). This showed the last layer on
top in the default top-down view and made the stack occlude inside-out
when tilted back. Sort ascending: farthest first, nearest last.
2026-06-28 23:06:33 +02:00
659 changed files with 81233 additions and 42403 deletions

View file

@ -1,319 +0,0 @@
/**
* ShapeSeq Chain — sequential pipeline runner with generator combination modes
*
* Evaluates an ordered list of primitives as a sequential pipeline:
* 1. Generators run first, combined via additive or multiplicative merge
* 2. Processors transform the pattern in chain order
* 3. Converters run in chain order
* 4. Timing modifiers annotate last
*
* Params are distributed flat across primitives in chain order.
* Each primitive gets a deterministic PRNG stream via fork(masterPRNG, index).
*
* Port-ready: explicit state, typed arrays, no closures in hot path.
*
* @module shapeseq/chain
*/
import { createPattern, mergePatterns } from './pattern.js';
import { createPRNG, fork } from './prng.js';
// ── Category execution order ────────────────────────────────────────
const PHASE_ORDER = ['generator', 'processor', 'converter', 'timing'];
// ── Chain class ─────────────────────────────────────────────────────
export class Chain {
constructor() {
/** @private @type {Array<import('./primitive.js').Primitive>} */
this._primitives = [];
/** @type {'additive'|'multiplicative'} */
this.generatorCombineMode = 'additive';
/** @private @type {number} */
this._masterSeed = 0;
/**
* Per-primitive state objects, indexed by position in chain.
* Populated after evaluate() calls; used for freeze support.
* @private @type {Array<Object>}
*/
this._primitiveStates = [];
}
// ── Primitive management ────────────────────────────────────────
/**
* Append a primitive to the end of the chain.
* @param {import('./primitive.js').Primitive} primitive
*/
addPrimitive(primitive) {
this._primitives.push(primitive);
this._primitiveStates.push(primitive.getState());
}
/**
* Remove the primitive at the given index.
* @param {number} index
*/
removePrimitive(index) {
const idx = index | 0;
if (idx < 0 || idx >= this._primitives.length) {
throw new RangeError('removePrimitive: index ' + index + ' out of range [0, ' + (this._primitives.length - 1) + ']');
}
this._primitives.splice(idx, 1);
this._primitiveStates.splice(idx, 1);
}
/**
* Insert a primitive at the given index, shifting others right.
* @param {number} index
* @param {import('./primitive.js').Primitive} primitive
*/
insertPrimitive(index, primitive) {
const idx = index | 0;
if (idx < 0 || idx > this._primitives.length) {
throw new RangeError('insertPrimitive: index ' + index + ' out of range [0, ' + this._primitives.length + ']');
}
this._primitives.splice(idx, 0, primitive);
this._primitiveStates.splice(idx, 0, primitive.getState());
}
/**
* Move a primitive from one position to another.
* @param {number} fromIndex
* @param {number} toIndex
*/
movePrimitive(fromIndex, toIndex) {
const from = fromIndex | 0;
const to = toIndex | 0;
const len = this._primitives.length;
if (from < 0 || from >= len) {
throw new RangeError('movePrimitive: fromIndex ' + fromIndex + ' out of range [0, ' + (len - 1) + ']');
}
if (to < 0 || to >= len) {
throw new RangeError('movePrimitive: toIndex ' + toIndex + ' out of range [0, ' + (len - 1) + ']');
}
const [prim] = this._primitives.splice(from, 1);
const [state] = this._primitiveStates.splice(from, 1);
this._primitives.splice(to, 0, prim);
this._primitiveStates.splice(to, 0, state);
}
/**
* Get the current list of primitives (shallow copy).
* @returns {Array<import('./primitive.js').Primitive>}
*/
getPrimitives() {
return this._primitives.slice();
}
// ── Configuration ───────────────────────────────────────────────
/**
* Total parameter count across all primitives in the chain.
* @returns {number}
*/
get totalParamCount() {
let total = 0;
for (let i = 0; i < this._primitives.length; i++) {
total += this._primitives[i].paramCount;
}
return total;
}
/**
* Get a flat list of all param schemas across all primitives,
* annotated with their primitive and param indices.
*
* @returns {Array<{ primitiveIndex: number, paramIndex: number, schema: Object }>}
*/
getParamSchemas() {
const result = [];
for (let pi = 0; pi < this._primitives.length; pi++) {
const prim = this._primitives[pi];
const schema = prim.paramSchema;
for (let si = 0; si < schema.length; si++) {
result.push({
primitiveIndex: pi,
paramIndex: si,
schema: schema[si],
});
}
}
return result;
}
// ── Evaluation ──────────────────────────────────────────────────
/**
* Evaluate the chain, producing a pattern description.
*
* Pipeline order:
* 1. Generators — combined via generatorCombineMode
* 2. Processors — sequential transform
* 3. Converters — sequential transform
* 4. Timing modifiers — annotate last
*
* @param {Float32Array|Array<number>} params - flat param array distributed across primitives
* @param {number} stepCount - number of steps in the output pattern
* @param {number} masterSeed - seed for the master PRNG
* @returns {{ steps: Array, stepCount: number, metadata: Object }}
*/
evaluate(params, stepCount, masterSeed) {
const primitives = this._primitives;
const primCount = primitives.length;
// Create master PRNG from seed
const masterPRNG = createPRNG(masterSeed >>> 0);
// ── Bucket primitives by category, preserving chain order ──
/** @type {Array<{ index: number, prim: Object }>} */
const generators = [];
const processors = [];
const converters = [];
const timingMods = [];
for (let i = 0; i < primCount; i++) {
const entry = { index: i, prim: primitives[i] };
switch (primitives[i].category) {
case 'generator': generators.push(entry); break;
case 'processor': processors.push(entry); break;
case 'converter': converters.push(entry); break;
case 'timing': timingMods.push(entry); break;
}
}
// ── Compute param offsets per primitive ──
const paramOffsets = new Array(primCount);
let offset = 0;
for (let i = 0; i < primCount; i++) {
paramOffsets[i] = offset;
offset += primitives[i].paramCount;
}
// ── Helper: run a single primitive ──
const self = this;
function runPrimitive(entry, inputPattern) {
const idx = entry.index;
const prim = entry.prim;
const pOffset = paramOffsets[idx];
const pCount = prim.paramCount;
// Slice params for this primitive
const primParams = new Float32Array(pCount);
for (let p = 0; p < pCount; p++) {
primParams[p] = pOffset + p < params.length ? +params[pOffset + p] : prim.paramSchema[p].default;
}
// Fork a deterministic PRNG for this primitive
const primRNG = fork(masterPRNG, idx);
// Get current state
const state = self._primitiveStates[idx] || prim.getState();
// Process
const result = prim.process(primParams, inputPattern, state, primRNG);
// Store updated state
self._primitiveStates[idx] = result.nextState;
return result.patternDesc;
}
// ── Phase 1: Generators ──
let pattern;
if (generators.length === 0) {
// Default pattern: all steps triggered
pattern = createPattern(stepCount);
for (let i = 0; i < stepCount; i++) {
pattern.steps[i].trigger = true;
}
} else if (generators.length === 1) {
// Single generator — no merge needed
pattern = runPrimitive(generators[0], createPattern(stepCount));
} else {
// Multiple generators — run each, then merge
let merged = runPrimitive(generators[0], createPattern(stepCount));
for (let g = 1; g < generators.length; g++) {
const next = runPrimitive(generators[g], createPattern(stepCount));
merged = mergePatterns(merged, next, this.generatorCombineMode);
}
pattern = merged;
}
// ── Phase 2: Processors ──
for (let i = 0; i < processors.length; i++) {
pattern = runPrimitive(processors[i], pattern);
}
// ── Phase 3: Converters ──
for (let i = 0; i < converters.length; i++) {
pattern = runPrimitive(converters[i], pattern);
}
// ── Phase 4: Timing modifiers ──
for (let i = 0; i < timingMods.length; i++) {
pattern = runPrimitive(timingMods[i], pattern);
}
return pattern;
}
// ── State management (for freeze) ──────────────────────────────
/**
* Get serializable state for all primitives in the chain.
* @returns {Array<Object>}
*/
getState() {
const states = new Array(this._primitives.length);
for (let i = 0; i < this._primitives.length; i++) {
states[i] = this._primitiveStates[i] || this._primitives[i].getState();
}
return states;
}
/**
* Restore all primitive states from a previously serialized state array.
* @param {Array<Object>} states
*/
setState(states) {
if (!Array.isArray(states)) {
throw new TypeError('setState expects an array of state objects');
}
const len = Math.min(states.length, this._primitives.length);
for (let i = 0; i < len; i++) {
this._primitives[i].setState(states[i]);
this._primitiveStates[i] = states[i];
}
}
/**
* Get the master PRNG seed.
* @returns {number}
*/
getMasterSeed() {
return this._masterSeed;
}
/**
* Set the master PRNG seed.
* @param {number} seed - 32-bit integer seed
*/
setMasterSeed(seed) {
this._masterSeed = seed >>> 0;
}
}

View file

@ -1,246 +0,0 @@
/**
* ShapeSeq Primitive Base Class and Param Schema System
*
* Base class for all sequencing primitives (generators, processors,
* timing modifiers, converters). Defines the param schema format,
* symbolic process() interface, and state management for freeze support.
*
* Port-ready: explicit state, no closures, typed arrays where possible.
*
* @module shapeseq/primitive
*/
// ── Valid primitive categories ──────────────────────────────────────
export const CATEGORIES = Object.freeze([
'generator',
'processor',
'timing',
'converter',
]);
// ── Param schema defaults ───────────────────────────────────────────
const DEFAULT_SCALED_RANGE = 0.3;
// ── Boundary enforcement helpers ────────────────────────────────────
/**
* Clamp a value to [0, 1].
* @param {number} v
* @returns {number}
*/
function clamp01(v) {
return v < 0 ? 0 : v > 1 ? 1 : v;
}
/**
* Wrap a value into [0, 1) with modular arithmetic.
* @param {number} v
* @returns {number}
*/
function wrap01(v) {
const m = v % 1;
return m < 0 ? m + 1 : m;
}
/**
* Apply boundary enforcement to a raw param value.
*
* @param {number} value - raw [0,1] value (or delta-adjusted value)
* @param {{ boundary: string, scaledRange?: number }} schema - param schema entry
* @param {number|null} frozenValue - frozen value for 'scaled' boundary (null if not frozen)
* @returns {number}
*/
export function applyBoundary(value, schema, frozenValue) {
switch (schema.boundary) {
case 'wrap':
return wrap01(value);
case 'scaled': {
if (frozenValue === null || frozenValue === undefined) {
return clamp01(value);
}
const range = schema.scaledRange !== undefined ? schema.scaledRange : DEFAULT_SCALED_RANGE;
const lo = frozenValue - range;
const hi = frozenValue + range;
// Map [0,1] input to [lo, hi], then clamp to [0,1]
const mapped = lo + value * (hi - lo);
return clamp01(mapped);
}
case 'clamp':
default:
return clamp01(value);
}
}
// ── Param schema validation ─────────────────────────────────────────
/**
* Validate a single param schema entry.
* Throws on invalid entries for fast fail during development.
*
* @param {{ name: string, default: number, boundary: string, scaledRange?: number }} entry
* @param {number} index - position in schema array (for error messages)
*/
function validateSchemaEntry(entry, index) {
if (!entry || typeof entry !== 'object') {
throw new TypeError('paramSchema[' + index + '] must be an object');
}
if (typeof entry.name !== 'string' || entry.name.length === 0) {
throw new TypeError('paramSchema[' + index + '].name must be a non-empty string');
}
if (typeof entry.default !== 'number' || entry.default < 0 || entry.default > 1) {
throw new RangeError('paramSchema[' + index + '].default must be in [0,1], got ' + entry.default);
}
if (entry.boundary !== 'clamp' && entry.boundary !== 'wrap' && entry.boundary !== 'scaled') {
throw new TypeError(
"paramSchema[" + index + "].boundary must be 'clamp', 'wrap', or 'scaled', got '" + entry.boundary + "'"
);
}
if (entry.boundary === 'scaled') {
const sr = entry.scaledRange;
if (sr !== undefined && (typeof sr !== 'number' || sr <= 0 || sr > 1)) {
throw new RangeError('paramSchema[' + index + '].scaledRange must be in (0,1], got ' + sr);
}
}
}
// ── Primitive base class ────────────────────────────────────────────
export class Primitive {
/**
* @param {string} name - unique identifier for this primitive type
* @param {string} category - one of CATEGORIES
* @param {Array<{ name: string, default: number, boundary: string, scaledRange?: number }>} paramSchema
*/
constructor(name, category, paramSchema) {
if (typeof name !== 'string' || name.length === 0) {
throw new TypeError('Primitive name must be a non-empty string');
}
if (CATEGORIES.indexOf(category) === -1) {
throw new TypeError(
"Primitive category must be one of [" + CATEGORIES.join(', ') + "], got '" + category + "'"
);
}
if (!Array.isArray(paramSchema)) {
throw new TypeError('paramSchema must be an array');
}
// Validate each entry
for (let i = 0; i < paramSchema.length; i++) {
validateSchemaEntry(paramSchema[i], i);
}
/** @type {string} */
this.name = name;
/** @type {string} */
this.category = category;
/**
* Frozen copy of the param schema. Each entry:
* { name: string, default: number, boundary: 'clamp'|'wrap'|'scaled', scaledRange?: number }
* @type {Array<Object>}
*/
this.paramSchema = Object.freeze(paramSchema.map(function (entry) {
const frozen = {
name: entry.name,
default: entry.default,
boundary: entry.boundary,
};
if (entry.boundary === 'scaled') {
frozen.scaledRange = entry.scaledRange !== undefined ? entry.scaledRange : DEFAULT_SCALED_RANGE;
}
return Object.freeze(frozen);
}));
/** @private */
this._seed = 0;
}
// ── Param utilities ─────────────────────────────────────────────
/**
* Total number of parameters this primitive exposes.
* @returns {number}
*/
get paramCount() {
return this.paramSchema.length;
}
/**
* Get default param values as a Float32Array, one per schema entry.
* @returns {Float32Array}
*/
getDefaults() {
const count = this.paramSchema.length;
const defaults = new Float32Array(count);
for (let i = 0; i < count; i++) {
defaults[i] = this.paramSchema[i].default;
}
return defaults;
}
// ── Symbolic processing ─────────────────────────────────────────
/**
* Transform a pattern description. Subclasses MUST override this.
*
* - Generators ignore patternDesc and create a new one (using createPattern())
* - Processors/timing modifiers clone and transform patternDesc
* - The rng param is a PRNG state from prng.js; consume via next(rng)
* and return the consumed state in the result
*
* @param {Float32Array|Array<number>} params - param values, one per schema entry, each [0,1]
* @param {{ steps: Array, stepCount: number, metadata: Object }} patternDesc - input pattern
* @param {Object} state - primitive-specific state (from previous process() call or getState())
* @param {{ state: number }} rng - PRNG state object from prng.js
* @returns {{ patternDesc: { steps: Array, stepCount: number, metadata: Object }, nextState: Object }}
*/
process(params, patternDesc, state, rng) {
void params; void patternDesc; void state; void rng;
throw new Error(this.name + '.process() must be overridden by subclass');
}
// ── State management (for freeze) ──────────────────────────────
/**
* Get serializable state for this primitive.
* Stateless primitives return {}. Stateful primitives (e.g. Pitch Walker)
* override to include their internal state.
*
* @returns {Object}
*/
getState() {
return {};
}
/**
* Restore primitive state from a previously serialized state object.
* Stateless primitives are a no-op. Stateful primitives override.
*
* @param {Object} _state
*/
setState(_state) {
// no-op for stateless primitives
}
/**
* Get the PRNG seed associated with this primitive.
* Used by freeze-as-algorithm to replay identical sequences.
*
* @returns {number}
*/
getSeed() {
return this._seed;
}
/**
* Set the PRNG seed for this primitive.
*
* @param {number} seed - 32-bit integer seed
*/
setSeed(seed) {
this._seed = seed >>> 0;
}
}

View file

@ -1,505 +0,0 @@
/**
* ShapeSeq Sequencing Primitives
*
* All 8 primitives for the ShapeSeq generative sequencing system.
* Each extends Primitive and implements process(params, patternDesc, state, rng).
*
* Port-ready: explicit state, no closures, seeded PRNG, typed arrays.
*
* @module shapeseq/primitives
*/
import { Primitive } from './primitive.js';
import { createPattern, clonePattern, setStep } from './pattern.js';
import { next, nextInt } from './prng.js';
// ── Helper: map [0,1] float to integer range [lo, hi] ──────────────
function mapToInt(value, lo, hi) {
const clamped = value < 0 ? 0 : value > 1 ? 1 : value;
return lo + Math.floor(clamped * (hi - lo + 1 - 1e-9));
}
// ── 1. EuclideanRhythm ──────────────────────────────────────────────
/**
* Bjorklund algorithm: distribute `pulses` as evenly as possible
* across `steps`, then apply rotation.
*/
function bjorklund(steps, pulses) {
if (pulses >= steps) {
const result = new Array(steps);
for (let i = 0; i < steps; i++) result[i] = true;
return result;
}
if (pulses <= 0) {
const result = new Array(steps);
for (let i = 0; i < steps; i++) result[i] = false;
return result;
}
// Build pattern using Bjorklund's algorithm
let groups = [];
for (let i = 0; i < pulses; i++) groups.push([true]);
for (let i = 0; i < steps - pulses; i++) groups.push([false]);
while (true) {
const remainder = groups.length - pulses;
if (remainder <= 1) break;
const minLen = Math.min(pulses, remainder);
const newGroups = [];
for (let i = 0; i < minLen; i++) {
newGroups.push(groups[i].concat(groups[groups.length - 1 - i]));
}
// Keep any leftovers
const leftStart = minLen;
const leftEnd = groups.length - minLen;
for (let i = leftStart; i < leftEnd; i++) {
newGroups.push(groups[i]);
}
groups = newGroups;
pulses = minLen;
if (pulses <= 1) break;
}
// Flatten groups
const result = [];
for (let i = 0; i < groups.length; i++) {
for (let j = 0; j < groups[i].length; j++) {
result.push(groups[i][j]);
}
}
return result;
}
export class EuclideanRhythm extends Primitive {
constructor() {
super('EuclideanRhythm', 'generator', [
{ name: 'steps', default: 0.5, boundary: 'clamp' },
{ name: 'pulses', default: 0.5, boundary: 'clamp' },
{ name: 'rotation', default: 0.0, boundary: 'wrap' },
]);
}
process(params, patternDesc, state, rng) {
const stepCount = patternDesc.stepCount;
const steps = mapToInt(params[0], 2, stepCount);
const pulses = mapToInt(params[1], 0, steps);
const rotation = mapToInt(params[2], 0, steps - 1);
const rhythm = bjorklund(steps, pulses);
const pattern = createPattern(stepCount);
for (let i = 0; i < stepCount; i++) {
if (i < steps) {
const srcIdx = (i - rotation + steps) % steps;
if (rhythm[srcIdx]) {
setStep(pattern, i, { trigger: true });
}
}
// Steps beyond `steps` remain untriggered (default)
}
return { patternDesc: pattern, nextState: {} };
}
}
// ── 2. ProbabilityGate ──────────────────────────────────────────────
export class ProbabilityGate extends Primitive {
constructor() {
super('ProbabilityGate', 'processor', [
{ name: 'density', default: 0.7, boundary: 'clamp' },
{ name: 'accentProbability', default: 0.3, boundary: 'clamp' },
]);
}
process(params, patternDesc, state, rng) {
const density = params[0];
const accentProb = params[1];
const pattern = clonePattern(patternDesc);
let currentRng = rng;
for (let i = 0; i < pattern.stepCount; i++) {
const step = pattern.steps[i];
if (step.trigger) {
// Coin flip for survival
const r1 = next(currentRng);
currentRng = r1.nextState;
if (r1.value >= density) {
step.trigger = false;
step.accent = false;
} else {
// Accent coin flip
const r2 = next(currentRng);
currentRng = r2.nextState;
step.accent = r2.value < accentProb;
}
}
}
return { patternDesc: pattern, nextState: {} };
}
}
// ── 3. PitchWalker ──────────────────────────────────────────────────
export class PitchWalker extends Primitive {
constructor() {
super('PitchWalker', 'generator', [
{ name: 'stepSize', default: 0.3, boundary: 'clamp' },
{ name: 'directionBias', default: 0.5, boundary: 'clamp' },
{ name: 'gravity', default: 0.3, boundary: 'clamp' },
{ name: 'range', default: 0.8, boundary: 'clamp' },
]);
/** @private */
this._position = 0.5;
}
getState() {
return { position: this._position };
}
setState(savedState) {
if (savedState && typeof savedState.position === 'number') {
this._position = savedState.position;
}
}
process(params, patternDesc, state, rng) {
const stepSize = params[0];
const directionBias = params[1];
const gravity = params[2];
const range = params[3];
// Restore position from state if provided
let position = (state && typeof state.position === 'number')
? state.position
: this._position;
const pattern = createPattern(patternDesc.stepCount);
let currentRng = rng;
// Use incoming pattern's triggers if available, otherwise all triggered
const srcSteps = patternDesc.steps;
for (let i = 0; i < patternDesc.stepCount; i++) {
const triggered = srcSteps[i].trigger;
if (triggered) {
// Random walk step
const r1 = next(currentRng);
currentRng = r1.nextState;
// Direction: bias + gravity toward center
const gravityPull = (0.5 - position) * gravity;
const biasOffset = (directionBias - 0.5) * 2; // [-1, 1]
const direction = biasOffset + gravityPull;
// Random component: [-1, 1] scaled by stepSize
const randomComponent = (r1.value * 2 - 1) * stepSize * range;
const delta = direction * stepSize * 0.5 + randomComponent;
position = position + delta;
// Clamp to [0, 1]
if (position < 0) position = 0;
if (position > 1) position = 1;
setStep(pattern, i, { trigger: true, pitch: position });
}
// Untriggered steps keep default pitch, trigger=false
}
this._position = position;
return {
patternDesc: pattern,
nextState: { position: position },
};
}
}
// ── 4. Ratchet ──────────────────────────────────────────────────────
export class Ratchet extends Primitive {
constructor() {
super('Ratchet', 'timing', [
{ name: 'maxDivision', default: 0.5, boundary: 'clamp' },
{ name: 'probability', default: 0.5, boundary: 'clamp' },
]);
}
process(params, patternDesc, state, rng) {
const maxDiv = mapToInt(params[0], 1, 4);
const probability = params[1];
const pattern = clonePattern(patternDesc);
let currentRng = rng;
for (let i = 0; i < pattern.stepCount; i++) {
const step = pattern.steps[i];
if (step.trigger) {
const r1 = next(currentRng);
currentRng = r1.nextState;
if (r1.value < probability && maxDiv > 1) {
// Pick a subdivision count in [2, maxDiv]
const r2 = nextInt(currentRng, 2, maxDiv);
currentRng = r2.nextState;
step.subdivisions = r2.value;
}
}
}
return { patternDesc: pattern, nextState: {} };
}
}
// ── 5. SwingGroove ──────────────────────────────────────────────────
export class SwingGroove extends Primitive {
constructor() {
super('SwingGroove', 'timing', [
{ name: 'swingAmount', default: 0.0, boundary: 'clamp' },
{ name: 'swingGrid', default: 0.0, boundary: 'clamp' },
]);
}
process(params, patternDesc, state, rng) {
const swingAmount = params[0];
const pattern = clonePattern(patternDesc);
// Max swing = 0.33 (triplet feel)
const maxOffset = 0.33;
const offset = swingAmount * maxOffset;
// Apply swing to every other step (odd-indexed steps)
for (let i = 1; i < pattern.stepCount; i += 2) {
pattern.steps[i].timeOffset = offset;
}
return { patternDesc: pattern, nextState: {} };
}
}
// ── 6. DensityMorph ─────────────────────────────────────────────────
export class DensityMorph extends Primitive {
constructor() {
super('DensityMorph', 'generator', [
{ name: 'density', default: 0.5, boundary: 'clamp' },
{ name: 'clustering', default: 0.0, boundary: 'clamp' },
]);
}
process(params, patternDesc, state, rng) {
const density = params[0];
const clustering = params[1];
const stepCount = patternDesc.stepCount;
const pattern = createPattern(stepCount);
const numTriggers = Math.floor(density * stepCount);
if (numTriggers <= 0) {
return { patternDesc: pattern, nextState: {} };
}
if (numTriggers >= stepCount) {
for (let i = 0; i < stepCount; i++) {
setStep(pattern, i, { trigger: true });
}
return { patternDesc: pattern, nextState: {} };
}
let currentRng = rng;
if (clustering < 0.01) {
// Even spread: Euclidean-like placement
for (let i = 0; i < numTriggers; i++) {
const idx = Math.floor((i * stepCount) / numTriggers);
setStep(pattern, idx, { trigger: true });
}
} else if (clustering > 0.99) {
// Full clustering: contiguous burst
const r1 = nextInt(currentRng, 0, stepCount - 1);
currentRng = r1.nextState;
const startPos = r1.value;
for (let i = 0; i < numTriggers; i++) {
const idx = (startPos + i) % stepCount;
setStep(pattern, idx, { trigger: true });
}
} else {
// Interpolate: place triggers with clustering-dependent spread
// Use a "center of mass" approach:
// Pick a random center, then distribute triggers around it
// with spread inversely proportional to clustering
const r1 = next(currentRng);
currentRng = r1.nextState;
const center = r1.value * stepCount;
// Spread factor: low clustering = large spread, high = tight
const spreadRadius = (1 - clustering) * stepCount * 0.5;
// Score each step by distance from center (wrapping)
const scores = new Float32Array(stepCount);
for (let i = 0; i < stepCount; i++) {
// Wrapped distance from center
let dist = Math.abs(i - center);
if (dist > stepCount * 0.5) dist = stepCount - dist;
// Add small random jitter to break ties
const r2 = next(currentRng);
currentRng = r2.nextState;
scores[i] = dist / (spreadRadius + 0.001) + r2.value * 0.01;
}
// Select the numTriggers steps with lowest scores
const indices = new Array(stepCount);
for (let i = 0; i < stepCount; i++) indices[i] = i;
indices.sort(function (a, b) { return scores[a] - scores[b]; });
for (let i = 0; i < numTriggers; i++) {
setStep(pattern, indices[i], { trigger: true });
}
}
return { patternDesc: pattern, nextState: {} };
}
}
// ── 7. IntervalLock ─────────────────────────────────────────────────
const SCALES = [
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11], // chromatic
[0, 2, 4, 5, 7, 9, 11], // major
[0, 2, 3, 5, 7, 8, 10], // natural minor
[0, 2, 3, 5, 7, 8, 11], // harmonic minor
[0, 2, 4, 7, 9], // pentatonic major
[0, 3, 5, 7, 10], // pentatonic minor
[0, 3, 5, 6, 7, 10], // blues
[0, 2, 3, 5, 7, 9, 10], // dorian
[0, 2, 4, 5, 7, 9, 10], // mixolydian
[0, 2, 4, 6, 8, 10], // whole tone
[0, 2, 3, 5, 6, 8, 9, 11], // diminished
];
export class IntervalLock extends Primitive {
constructor() {
super('IntervalLock', 'converter', [
{ name: 'root', default: 0.0, boundary: 'clamp' },
{ name: 'mode', default: 0.0, boundary: 'clamp' },
{ name: 'octaveRange', default: 0.25, boundary: 'clamp' },
]);
}
process(params, patternDesc, state, rng) {
const root = mapToInt(params[0], 0, 11);
const scaleIdx = mapToInt(params[1], 0, SCALES.length - 1);
const octaveRange = mapToInt(params[2], 1, 4);
const scale = SCALES[scaleIdx];
const pattern = clonePattern(patternDesc);
// Build the full set of MIDI notes in this scale + root + range
const notes = [];
for (let oct = 0; oct < octaveRange; oct++) {
for (let i = 0; i < scale.length; i++) {
const midiNote = root + scale[i] + oct * 12;
if (midiNote <= 127) {
notes.push(midiNote);
}
}
}
if (notes.length === 0) {
return { patternDesc: pattern, nextState: {} };
}
for (let i = 0; i < pattern.stepCount; i++) {
const step = pattern.steps[i];
// Quantize pitch [0,1] to nearest note in our scale
const targetIdx = Math.round(step.pitch * (notes.length - 1));
const clampedIdx = targetIdx < 0 ? 0 : targetIdx >= notes.length ? notes.length - 1 : targetIdx;
// Store as MIDI note / 127 to stay in [0,1]
step.pitch = notes[clampedIdx] / 127;
}
return { patternDesc: pattern, nextState: {} };
}
}
// ── 8. VelocityShaper ───────────────────────────────────────────────
export class VelocityShaper extends Primitive {
constructor() {
super('VelocityShaper', 'processor', [
{ name: 'curveType', default: 0.0, boundary: 'clamp' },
{ name: 'depth', default: 0.5, boundary: 'clamp' },
{ name: 'phase', default: 0.0, boundary: 'wrap' },
]);
}
process(params, patternDesc, state, rng) {
const curveIdx = mapToInt(params[0], 0, 4);
const depth = params[1];
const phase = params[2];
const pattern = clonePattern(patternDesc);
const stepCount = pattern.stepCount;
let currentRng = rng;
for (let i = 0; i < stepCount; i++) {
const step = pattern.steps[i];
if (!step.trigger) continue;
// Phase-shifted position
const pos = ((i / stepCount) + phase) % 1;
let shapeValue;
switch (curveIdx) {
case 0: // flat
shapeValue = 1.0;
break;
case 1: // accent-every-N (accent every 4th step)
shapeValue = ((i + Math.floor(phase * stepCount)) % 4 === 0) ? 1.0 : 0.5;
break;
case 2: // crescendo
shapeValue = pos;
break;
case 3: // decrescendo
shapeValue = 1.0 - pos;
break;
case 4: { // random
const r1 = next(currentRng);
currentRng = r1.nextState;
shapeValue = r1.value;
break;
}
default:
shapeValue = 1.0;
}
// Apply depth: interpolate between uniform (1.0) and shaped
// depth=0 means all same velocity (base), depth=1 means full shape
const baseVelocity = 0.7;
const shaped = shapeValue;
step.velocity = baseVelocity * (1 - depth) + shaped * depth;
// Clamp
if (step.velocity < 0) step.velocity = 0;
if (step.velocity > 1) step.velocity = 1;
}
return { patternDesc: pattern, nextState: {} };
}
}
// ── Registry ────────────────────────────────────────────────────────
export const PRIMITIVE_REGISTRY = {
EuclideanRhythm: EuclideanRhythm,
ProbabilityGate: ProbabilityGate,
PitchWalker: PitchWalker,
Ratchet: Ratchet,
SwingGroove: SwingGroove,
DensityMorph: DensityMorph,
IntervalLock: IntervalLock,
VelocityShaper: VelocityShaper,
};

View file

@ -1,304 +0,0 @@
/**
* ShapeSeq Engine — central orchestrator
*
* Wires together the sequence MLP, param mapping, primitive chain,
* projection layer, clock engine, and C15 bridge.
*
* Main loop (triggered by setSequenceInputs):
* 1. Forward inputs to sequenceIML
* 2. Run MLP inference to get 16 outputs
* 3. Map 16 outputs to N primitive params via param-map
* 4. Evaluate the chain to produce a pattern description
* 5. Apply projection transforms
* 6. Schedule the pattern on the clock
*
* Bridge integration:
* - Subscribes to seq.noteOn / seq.noteOff on the event bus
* - Forwards to C15Bridge.noteOn / noteOff
* - Tracks active notes to avoid orphans
*
* @module shapeseq/sequencer
*/
import { createSequenceIML, SEQ_N_OUTPUTS } from './seq-iml.js';
import { Chain } from './chain.js';
import { ClockEngine } from './clock.js';
import { map } from './param-map.js';
import { createProjectionChain, applyProjection, PRESETS } from './projection.js';
import { SEQ } from './event-bus.js';
import {
EuclideanRhythm,
ProbabilityGate,
PitchWalker,
IntervalLock,
VelocityShaper,
} from './primitives.js';
// ── Defaults ─────────────────────────────────────────────────────────
const DEFAULT_BPM = 120;
const DEFAULT_STEP_COUNT = 8;
const DEFAULT_MASTER_SEED = 42;
const DEFAULT_SPREAD = 0.6;
// ── ShapeSeqEngine ───────────────────────────────────────────────────
export class ShapeSeqEngine {
/**
* @param {{ audioContext: AudioContext, eventBus: import('./event-bus.js').EventBus, c15Bridge: import('../synth/c15-bridge.js').C15Bridge }} opts
*/
constructor({ audioContext, eventBus, c15Bridge }) {
if (!audioContext) throw new TypeError('ShapeSeqEngine requires an audioContext');
if (!eventBus) throw new TypeError('ShapeSeqEngine requires an eventBus');
if (!c15Bridge) throw new TypeError('ShapeSeqEngine requires a c15Bridge');
/** @private */ this._audioCtx = audioContext;
/** @private */ this._bus = eventBus;
/** @private */ this._c15 = c15Bridge;
/** @private */ this._sequenceIML = null;
/** @private */ this._chain = null;
/** @private */ this._clock = null;
/** @private */ this._projectionChain = null;
/** @private */ this._stepCount = DEFAULT_STEP_COUNT;
/** @private */ this._masterSeed = DEFAULT_MASTER_SEED;
/** @private */ this._playing = false;
/** @private */ this._initialized = false;
// Track active notes for orphan prevention
/** @private @type {Set<number>} */
this._activeNotes = new Set();
// Bound handlers for event bus (stored for cleanup)
/** @private */
this._onNoteOn = (data) => this._handleNoteOn(data);
/** @private */
this._onNoteOff = (data) => this._handleNoteOff(data);
}
// ── Lifecycle ──────────────────────────────────────────────────────
/**
* Initialize all subsystems: create sequence IML, default chain,
* clock, and projection chain. Must be called before start().
*/
async init() {
// 1. Create the sequence MLP
this._sequenceIML = await createSequenceIML();
// Randomize weights with default spread
this._sequenceIML.drawWeights(DEFAULT_SPREAD);
// 2. Create the default primitive chain
this._chain = new Chain();
this._chain.addPrimitive(new EuclideanRhythm());
this._chain.addPrimitive(new ProbabilityGate());
this._chain.addPrimitive(new PitchWalker());
this._chain.addPrimitive(new IntervalLock());
this._chain.addPrimitive(new VelocityShaper());
this._chain.setMasterSeed(this._masterSeed);
// 3. Set up the clock
this._clock = new ClockEngine(this._audioCtx, this._bus);
this._clock.bpm = DEFAULT_BPM;
// 4. Create default projection chain (expressive preset)
const result = createProjectionChain(PRESETS.expressive);
if (!result.valid) {
throw new Error('Default projection chain invalid: ' + result.error);
}
this._projectionChain = result;
// 5. Subscribe to event bus for C15 bridge integration
this._bus.on(SEQ.NOTE_ON, this._onNoteOn);
this._bus.on(SEQ.NOTE_OFF, this._onNoteOff);
this._initialized = true;
}
/**
* Start the clock. Requires init() to have been called.
*/
start() {
if (!this._initialized) {
throw new Error('ShapeSeqEngine.start() called before init()');
}
if (this._playing) return;
this._playing = true;
this._clock.start();
}
/**
* Stop the clock and release all active notes.
*/
stop() {
if (!this._playing) return;
this._playing = false;
this._clock.stop();
this._releaseAllNotes();
}
/**
* Full cleanup: stop playback, unsubscribe from events, destroy IML.
*/
destroy() {
this.stop();
// Unsubscribe from event bus
this._bus.off(SEQ.NOTE_ON, this._onNoteOn);
this._bus.off(SEQ.NOTE_OFF, this._onNoteOff);
// Destroy the sequence IML instance
if (this._sequenceIML) {
this._sequenceIML.destroy();
this._sequenceIML = null;
}
this._chain = null;
this._clock = null;
this._projectionChain = null;
this._initialized = false;
}
// ── Configuration ──────────────────────────────────────────────────
/**
* Update the clock tempo.
* @param {number} bpm
*/
setTempo(bpm) {
if (this._clock) {
this._clock.setTempo(bpm);
}
}
/**
* Set the number of steps in the generated pattern.
* @param {number} count
*/
setStepCount(count) {
const c = Math.max(1, count | 0);
this._stepCount = c;
}
/**
* Set the projection preset by name.
* @param {'expressive'|'percussive'|'fullRange'} presetName
*/
setProjectionPreset(presetName) {
const preset = PRESETS[presetName];
if (!preset) {
throw new Error('Unknown projection preset: ' + presetName);
}
const result = createProjectionChain(preset);
if (!result.valid) {
throw new Error('Projection chain invalid: ' + result.error);
}
this._projectionChain = result;
}
// ── Chain access (for UI binding) ──────────────────────────────────
/** @returns {Chain} */
getChain() { return this._chain; }
/** @returns {ClockEngine} */
getClock() { return this._clock; }
/** @returns {WasmIML} */
getSequenceIML() { return this._sequenceIML; }
// ── Input routing ──────────────────────────────────────────────────
/**
* Feed new input values to the sequence MLP and run the full pipeline:
* MLP inference -> param mapping -> chain evaluation -> projection -> clock scheduling.
*
* Call this each frame with the routed input values (e.g., [x, y]).
*
* @param {number[]} values - input array (typically [x, y])
*/
setSequenceInputs(values) {
if (!this._initialized || !this._sequenceIML) return;
// 1. Forward inputs to the sequence IML
this._sequenceIML.setInputs(values);
// 2. Run MLP inference
this._sequenceIML.process();
// 3. Get the 16 MLP outputs
const mlpOutputs = this._sequenceIML.getOutputs();
// 4. Map 16 outputs to N primitive params
const paramCount = this._chain.totalParamCount;
const mappedParams = map(mlpOutputs, paramCount);
// 5. Evaluate the chain to produce a pattern description
const patternDesc = this._chain.evaluate(mappedParams, this._stepCount, this._masterSeed);
// 6. Apply projection transforms
const projectedPattern = applyProjection(this._projectionChain, patternDesc);
// 7. Schedule the pattern on the clock
this._clock.schedulePattern(projectedPattern);
}
// ── ML control ─────────────────────────────────────────────────────
/** @returns {boolean} */
get isPlaying() {
return this._playing;
}
// ── Bridge integration (private) ───────────────────────────────────
/**
* Handle seq.noteOn events from the event bus.
* Converts [0,1] pitch to MIDI note number and forwards to C15.
*
* @private
* @param {Object} data - { pitch, velocity, stepIndex, time, accent, isSubdivision }
*/
_handleNoteOn(data) {
// pitch comes from the projection layer; after RangeMap it's already
// in MIDI note range (e.g., 48-84). Round to nearest integer.
const midiNote = Math.round(data.pitch) | 0;
const velocity = data.velocity;
// Clamp to valid MIDI range
const note = midiNote < 0 ? 0 : midiNote > 127 ? 127 : midiNote;
const vel = velocity < 0 ? 0 : velocity > 1 ? 1 : velocity;
this._c15.noteOn(note, vel);
this._activeNotes.add(note);
}
/**
* Handle seq.noteOff events from the event bus.
*
* @private
* @param {Object} data - { pitch, velocity, stepIndex, time }
*/
_handleNoteOff(data) {
const midiNote = Math.round(data.pitch) | 0;
const note = midiNote < 0 ? 0 : midiNote > 127 ? 127 : midiNote;
this._c15.noteOff(note);
this._activeNotes.delete(note);
}
/**
* Release all currently active notes to avoid orphaned noteOns.
* @private
*/
_releaseAllNotes() {
for (const note of this._activeNotes) {
this._c15.noteOff(note);
}
this._activeNotes.clear();
}
}

View file

@ -1,328 +0,0 @@
/**
* ShapeSeq Circular Step Visualizer
*
* Renders steps arranged in a circle (heptagon, tridecagon, etc.)
* with pitch mapped to radial distance, velocity to node size,
* and accent to color brightness.
*
* Designed for 60fps rendering — no allocations in the render loop.
* Port-ready: explicit state, no closures in hot paths.
*
* @module shapeseq/step-viz
*/
import { SEQ } from './event-bus.js';
// ── Constants (pre-allocated, shared across instances) ──────────────
const TWO_PI = Math.PI * 2;
const HALF_PI = Math.PI * 0.5;
// Color constants
const COLOR_INACTIVE = 'rgba(255, 255, 255, 0.15)';
const COLOR_ACTIVE = '#00ccff';
const COLOR_CURRENT = '#ff6a00';
const COLOR_ACCENT = '#ffcc00';
const COLOR_BG = '#0d0d0d';
// Glow colors (pre-computed rgba strings)
const GLOW_ACTIVE = 'rgba(0, 204, 255, 0.3)';
const GLOW_CURRENT = 'rgba(255, 106, 0, 0.4)';
const GLOW_ACCENT = 'rgba(255, 204, 0, 0.35)';
// Layout
const PADDING_RATIO = 0.08; // canvas padding as fraction of min dimension
const OUTER_RADIUS_RATIO = 0.90; // outer ring at 90% of available radius
const INNER_RADIUS_RATIO = 0.30; // inner ring at 30% of available radius
// Node sizing
const NODE_MIN_RADIUS = 4;
const NODE_MAX_RADIUS = 18;
const NODE_OUTLINE_WIDTH = 1.5;
// Current-step indicator
const INDICATOR_EXTRA_RADIUS = 8;
const INDICATOR_LINE_WIDTH = 2;
// Center dot
const CENTER_DOT_RADIUS = 3;
// ── StepVisualizer ──────────────────────────────────────────────────
export class StepVisualizer {
/**
* @param {{ canvas: HTMLCanvasElement, eventBus: import('./event-bus.js').EventBus }} opts
*/
constructor({ canvas, eventBus }) {
this._canvas = canvas;
this._ctx = canvas.getContext('2d');
this._bus = eventBus;
// State
this._pattern = null; // current pattern description
this._currentStep = -1; // playback position (-1 = none)
this._width = 0;
this._height = 0;
this._cx = 0; // center x
this._cy = 0; // center y
this._maxRadius = 0; // max ring radius in pixels
// Pre-allocated arrays to avoid per-frame allocation.
// Sized lazily when pattern is set.
this._nodeX = null; // Float64Array — screen x per step
this._nodeY = null; // Float64Array — screen y per step
this._nodeR = null; // Float64Array — rendered radius per step
// Interaction
this._tapCallback = null;
this._onPointerDown = this._handlePointerDown.bind(this);
// Event bus subscription
this._onStep = this._handleStep.bind(this);
this._bus.on(SEQ.STEP, this._onStep);
// Canvas interaction
this._canvas.addEventListener('pointerdown', this._onPointerDown);
// Initial sizing
this.resize(canvas.getBoundingClientRect().width, canvas.getBoundingClientRect().height);
}
// ── Public API ──────────────────────────────────────────────────
/**
* Update the displayed pattern.
* @param {{ steps: Array, stepCount: number, metadata: Object }} patternDesc
*/
setPattern(patternDesc) {
this._pattern = patternDesc;
const count = patternDesc ? patternDesc.stepCount : 0;
// (Re)allocate coordinate buffers only when step count changes
if (!this._nodeX || this._nodeX.length !== count) {
this._nodeX = new Float64Array(count);
this._nodeY = new Float64Array(count);
this._nodeR = new Float64Array(count);
}
this._computeLayout();
}
/**
* Update the playback position.
* @param {number} index — step index (0-based), or -1 for none
*/
setCurrentStep(index) {
this._currentStep = index;
}
/**
* Draw one frame. Call from requestAnimationFrame.
*/
render() {
const ctx = this._ctx;
const w = this._width;
const h = this._height;
// Clear
ctx.fillStyle = COLOR_BG;
ctx.fillRect(0, 0, w, h);
if (!this._pattern || this._pattern.stepCount === 0) return;
const steps = this._pattern.steps;
const count = this._pattern.stepCount;
const cx = this._cx;
const cy = this._cy;
// Draw connecting ring (subtle guide circle at midpoint radius)
const midRadius = this._maxRadius * ((OUTER_RADIUS_RATIO + INNER_RADIUS_RATIO) * 0.5);
ctx.beginPath();
ctx.arc(cx, cy, midRadius, 0, TWO_PI);
ctx.strokeStyle = 'rgba(255, 255, 255, 0.06)';
ctx.lineWidth = 1;
ctx.stroke();
// Center dot
ctx.beginPath();
ctx.arc(cx, cy, CENTER_DOT_RADIUS, 0, TWO_PI);
ctx.fillStyle = 'rgba(255, 255, 255, 0.2)';
ctx.fill();
// Draw connector line from center to current step
if (this._currentStep >= 0 && this._currentStep < count) {
const si = this._currentStep;
ctx.beginPath();
ctx.moveTo(cx, cy);
ctx.lineTo(this._nodeX[si], this._nodeY[si]);
ctx.strokeStyle = 'rgba(255, 106, 0, 0.2)';
ctx.lineWidth = INDICATOR_LINE_WIDTH;
ctx.stroke();
}
// Draw step nodes
for (let i = 0; i < count; i++) {
const step = steps[i];
const nx = this._nodeX[i];
const ny = this._nodeY[i];
const nr = this._nodeR[i];
const isCurrent = i === this._currentStep;
if (isCurrent) {
// Outer glow for current step
ctx.beginPath();
ctx.arc(nx, ny, nr + INDICATOR_EXTRA_RADIUS, 0, TWO_PI);
ctx.fillStyle = GLOW_CURRENT;
ctx.fill();
}
if (step.trigger) {
// Glow behind active nodes
if (!isCurrent) {
const glowColor = step.accent ? GLOW_ACCENT : GLOW_ACTIVE;
ctx.beginPath();
ctx.arc(nx, ny, nr + 4, 0, TWO_PI);
ctx.fillStyle = glowColor;
ctx.fill();
}
// Filled node
ctx.beginPath();
ctx.arc(nx, ny, nr, 0, TWO_PI);
if (isCurrent) {
ctx.fillStyle = COLOR_CURRENT;
} else if (step.accent) {
ctx.fillStyle = COLOR_ACCENT;
} else {
ctx.fillStyle = COLOR_ACTIVE;
}
ctx.fill();
} else {
// Dim outline only for untriggered steps
ctx.beginPath();
ctx.arc(nx, ny, nr, 0, TWO_PI);
ctx.strokeStyle = isCurrent ? COLOR_CURRENT : COLOR_INACTIVE;
ctx.lineWidth = NODE_OUTLINE_WIDTH;
ctx.stroke();
}
}
}
/**
* Handle canvas resize.
* @param {number} width — CSS pixels
* @param {number} height — CSS pixels
*/
resize(width, height) {
const dpr = window.devicePixelRatio || 1;
this._canvas.width = width * dpr;
this._canvas.height = height * dpr;
this._ctx.setTransform(dpr, 0, 0, dpr, 0, 0);
this._width = width;
this._height = height;
this._cx = width * 0.5;
this._cy = height * 0.5;
const minDim = Math.min(width, height);
this._maxRadius = (minDim * 0.5) * (1 - PADDING_RATIO * 2);
this._computeLayout();
}
/**
* Register a tap callback.
* @param {function(number): void} callback — receives step index
*/
onStepTap(callback) {
this._tapCallback = callback;
}
/**
* Unsubscribe from event bus and remove DOM listeners.
*/
destroy() {
this._bus.off(SEQ.STEP, this._onStep);
this._canvas.removeEventListener('pointerdown', this._onPointerDown);
this._tapCallback = null;
this._pattern = null;
}
// ── Private ─────────────────────────────────────────────────────
/**
* Re-compute node positions from current pattern + canvas size.
* Called when pattern or size changes — NOT per frame.
*/
_computeLayout() {
if (!this._pattern || !this._nodeX) return;
const steps = this._pattern.steps;
const count = this._pattern.stepCount;
const cx = this._cx;
const cy = this._cy;
const maxR = this._maxRadius;
const outerR = maxR * OUTER_RADIUS_RATIO;
const innerR = maxR * INNER_RADIUS_RATIO;
const radiusRange = outerR - innerR;
// Angular step: start at top (-PI/2), go clockwise
const angleStep = TWO_PI / count;
for (let i = 0; i < count; i++) {
const step = steps[i];
const angle = -HALF_PI + angleStep * i;
// Pitch -> radial distance: low pitch = outer, high pitch = inner
const pitchNorm = step.pitch; // 0 = low (outer), 1 = high (inner)
const r = outerR - pitchNorm * radiusRange;
this._nodeX[i] = cx + Math.cos(angle) * r;
this._nodeY[i] = cy + Math.sin(angle) * r;
// Velocity -> node size
this._nodeR[i] = NODE_MIN_RADIUS + step.velocity * (NODE_MAX_RADIUS - NODE_MIN_RADIUS);
}
}
/**
* Handle seq.step events from the event bus.
*/
_handleStep(data) {
if (typeof data.stepIndex === 'number') {
this._currentStep = data.stepIndex;
}
}
/**
* Handle pointer down on the canvas for tap interaction.
*/
_handlePointerDown(e) {
if (!this._tapCallback || !this._pattern) return;
const rect = this._canvas.getBoundingClientRect();
const px = e.clientX - rect.left;
const py = e.clientY - rect.top;
const count = this._pattern.stepCount;
// Find closest step within hit radius
let bestIdx = -1;
let bestDistSq = Infinity;
for (let i = 0; i < count; i++) {
const dx = px - this._nodeX[i];
const dy = py - this._nodeY[i];
const distSq = dx * dx + dy * dy;
// Hit area is the node radius + some tolerance
const hitR = this._nodeR[i] + 12;
if (distSq < hitR * hitR && distSq < bestDistSq) {
bestDistSq = distSq;
bestIdx = i;
}
}
if (bestIdx >= 0) {
this._tapCallback(bestIdx);
}
}
}

View file

@ -2,22 +2,30 @@ name: CI
# Stream 11 verification pipeline.
#
# Two parallel jobs:
# * cpp-tests — builds nisps host C++ tests, builds nisps.wasm, runs
# the parity check, runs the lint script.
# * playground-tests — typechecks the SolidJS playground, builds the
# production bundle, runs Playwright e2e tests.
# Three parallel jobs:
# * cpp-tests — builds nisps host C++ tests, builds nisps.wasm, runs
# the parity check, runs the lint script, and reports
# per-engine throughput on both host targets.
# * manifold-tests — typechecks the React manifold app, runs bun unit
# tests, builds the production bundle, runs Playwright
# e2e tests.
#
# Firmware compilation is NOT included in this workflow. Arduino-cli +
# rp2040 board package add ~2 minutes per run, and the verification value
# is low compared to the time cost; firmware build is documented as a
# manual `scripts/build-firmware.sh` step in README.md / CLAUDE.md.
# * firmware-build — compiles three representative PlatformIO envs for the
# RP2350 target.
#
# Firmware entered CI for the first time with the Phase 4 PlatformIO migration
# (plan §5, S9). It was previously excluded because arduino-cli + the rp2040
# board package cost ~2 minutes per run for "low verification value" — an
# assessment that did not survive contact: the SelfTest variant sat broken for
# an unknown period, calling a DisplayDriver method that did not exist at the
# pinned memllib commit, and nothing noticed because nothing built it. PIO's
# toolchain caches cleanly, so the cost is now a cache restore.
on:
push:
branches: [main, port-solidjs]
branches: [main]
pull_request:
branches: [main, port-solidjs]
branches: [main]
workflow_dispatch:
jobs:
@ -27,8 +35,6 @@ jobs:
steps:
- name: Checkout
uses: actions/checkout@v4
with:
submodules: recursive
- name: Install build deps
run: |
@ -58,11 +64,19 @@ jobs:
CMAKE_BUILD_PARALLEL_LEVEL: '4'
run: bash scripts/build-cpp-tests.sh
- name: Build WASM
# Freshness gate. manifold/public/nisps.{js,wasm} are TRACKED artifacts:
# the VPS webhook builds only manifold/, so vite copies whatever is
# committed into the production bundle. Prove the committed artifact
# still agrees with the C++ we just built, BEFORE the rebuild below
# overwrites it. A stale commit fails here instead of silently shipping.
- name: WASM freshness gate (committed artifact vs native)
env:
# The script defaults to /usr/lib/emscripten/emcc; the runner gets
# emcc on PATH via setup-emsdk. Override.
EMCC: emcc
NISPS_PARITY_NO_BUILD: '1'
run: bash scripts/parity-check.sh
- name: Build WASM
# setup-emsdk puts emcc on PATH; build-wasm.sh picks it up via
# `command -v emcc`. No EMCC override needed.
run: bash scripts/build-wasm.sh
- name: Parity check (native vs WASM)
@ -73,6 +87,19 @@ jobs:
- name: Lint
run: bash scripts/lint-cpp.sh
# Throughput is reported, not asserted — the same call the firmware
# flash/RAM step makes below, for a stronger reason: a wall-clock
# threshold on a shared runner is either slack enough to be meaningless
# or tight enough to fail on someone else's noisy neighbour. What this
# buys is (a) a per-commit ns/sample + realtime-factor record in the log,
# where a 2-3x regression — the failure mode ALIGNMENT defect 5 names —
# is unmissable even at runner noise levels, (b) proof the bench still
# builds and runs on BOTH targets, and (c) the wasm/native ratio.
# Runner noise makes small moves unreadable; for a real comparison run
# scripts/bench-engines.sh --compare locally.
- name: Engine throughput report (native + WASM, not asserted)
run: bash scripts/bench-engines.sh --target-ms 60
- name: Upload parity blobs on failure
if: failure()
uses: actions/upload-artifact@v4
@ -84,8 +111,8 @@ jobs:
if-no-files-found: ignore
retention-days: 7
playground-tests:
name: Playground typecheck + e2e
manifold-tests:
name: Manifold typecheck + unit + e2e
runs-on: ubuntu-latest
steps:
- name: Checkout
@ -96,24 +123,50 @@ jobs:
with:
bun-version: latest
- name: Install playground deps
working-directory: playground
# Enforces the "schema changes ship with regenerated C++ AND TypeScript
# in the same change" rule. Regenerate from schemas/ and fail if the
# committed outputs differ; then run the codegen golden test.
- name: Codegen is committed and idempotent
run: |
cd codegen
bun install --frozen-lockfile
bun run generate.ts
bun run generate-midi-devices.ts
bun run tests/golden_test.ts
# The schemas' per-voice-space `curve` declarations are DESCRIPTIVE
# of nisps/engines/*.hpp. Prove they still describe it.
bun run tests/curve_drift_test.ts
cd ..
GEN_DIRS="nisps/modes/generated nisps/midi/generated nisps/ml/generated \
manifold/src/modes/generated manifold/src/midi-devices/generated"
git add -N $GEN_DIRS
if ! git diff --exit-code -- $GEN_DIRS; then
echo "::error::Generated outputs are stale. Run codegen and commit the result." >&2
exit 1
fi
- name: Install manifold deps
working-directory: manifold
run: bun install --frozen-lockfile
- name: Typecheck
working-directory: playground
working-directory: manifold
run: bun run typecheck
- name: Build playground bundle
working-directory: playground
- name: Unit tests
working-directory: manifold
run: bun run test
- name: Build manifold bundle
working-directory: manifold
run: bun run build
- name: Install Playwright browsers
working-directory: playground
working-directory: manifold
run: bunx playwright install --with-deps chromium
- name: Run Playwright tests
working-directory: playground
working-directory: manifold
run: bunx playwright test
- name: Upload Playwright report on failure
@ -121,5 +174,61 @@ jobs:
uses: actions/upload-artifact@v4
with:
name: playwright-report
path: playground/playwright-report/
path: manifold/playwright-report/
retention-days: 7
firmware-build:
name: Firmware (RP2350, PlatformIO)
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: '3.12'
# The platform wrapper and the arduino-pico framework are fetched from
# git and total ~1-2 GB. Key on platformio.ini because that file pins
# both versions — change a pin, get a fresh toolchain.
- name: Cache PlatformIO toolchain
uses: actions/cache@v4
with:
path: |
~/.platformio
~/.cache/pip
key: pio-${{ runner.os }}-${{ hashFiles('firmware/MEMLNaut-NISPS/platformio.ini') }}
restore-keys: pio-${{ runner.os }}-
- name: Install PlatformIO
run: pip install --upgrade platformio
# Four representative envs rather than all sixteen: they differ mostly in
# which mode type is instantiated, so a compile break is usually common to
# all. slpworkshop is the shipped default, pafsynth is the heaviest RAM
# user, selftest takes the separate NISPS_SELFTEST fork through main.cpp
# (the one that broke unnoticed before), and soundanalysismidi is the ONLY
# microphone variant — the sole mode whose DriverConfig differs from its
# engine's, via an on_driver_config() override. Drop it and the mic path
# stops being compiled by anything.
- name: Build representative firmware variants
working-directory: firmware/MEMLNaut-NISPS
run: pio run -e slpworkshop -e pafsynth -e selftest -e soundanalysismidi
# Sizes are reported, not asserted. A threshold would either be slack
# enough to be meaningless or tight enough to fail on unrelated work;
# this puts the numbers in the log so a jump is visible in review.
# Flash = .text + .rodata, RAM = .data + .bss + vector table +
# uninitialized — matching arduino-cli's convention, NOT PlatformIO's
# own console line, which double-counts .data on this board.
- name: Report flash/RAM per variant
working-directory: firmware/MEMLNaut-NISPS
run: |
SIZE=$(find ~/.platformio/packages -name 'arm-none-eabi-size' | head -1)
for e in slpworkshop pafsynth selftest soundanalysismidi; do
"$SIZE" -A ".pio/build/$e/firmware.elf" | awk -v e="$e" '
/^\.text/{t=$2} /^\.rodata/{r=$2} /^\.data/{d=$2}
/^\.bss/{b=$2} /^\.ram_vector_table/{v=$2}
/^\.uninitialized_data/{u=$2}
END{printf "%-14s flash=%-8d ram=%-8d\n", e, t+r, d+b+v+u}'
done

View file

@ -40,7 +40,7 @@ jobs:
- name: Compile
run: |
cd playground/osc-bridge
cd manifold/osc-bridge
deno compile --allow-net --unstable-net --target ${{ matrix.target }} --output ../../dist/nisps-osc-bridge-${{ matrix.suffix }}${{ matrix.os == 'windows-latest' && '.exe' || '' }} bridge.ts
- uses: actions/upload-artifact@v4

View file

@ -12,7 +12,7 @@ name: VCV Rack plugin (cross-platform)
on:
push:
branches: [main, feat/manifold-mission, feat/vcv-dist]
branches: [main]
paths:
- 'vcv/**'
- '.github/workflows/vcv-plugin.yml'

17
.gitignore vendored
View file

@ -25,10 +25,6 @@
*.la
*.a
*.lib
# Exception: Faust shared library files (text source, .lib extension by
# Faust convention) should be tracked.
!playground/faust/*.lib
# Executables
*.exe
*.out
@ -37,9 +33,15 @@
.ai
build
# PlatformIO build artifacts (firmware/MEMLNaut-NISPS) — bootstrapped platform,
# toolchain, and libraries live outside the repo in ~/.platformio; only
# .pio/build (per-env objects/elf/uf2) and .pio/libdeps (fetched lib_deps)
# land here, both disposable.
.pio
# OSC bridge compiled binaries (built via compile.sh or CI)
playground/osc-bridge/dist/
playground/osc-bridge/node_modules/
manifold/osc-bridge/dist/
manifold/osc-bridge/node_modules/
# Dolt database files (added by bd init)
.dolt/
@ -50,3 +52,6 @@ node_modules/
bun.lock
playwright-report/
test-results/
# Agent worktrees
.claude/worktrees/

3
.gitmodules vendored
View file

@ -1,3 +0,0 @@
[submodule "src/memllib"]
path = src/memllib
url = https://github.com/MusicallyEmbodiedML/memllib.git

143
AGENTS.md
View file

@ -1,126 +1,31 @@
# Agent Instructions
# Agent Instructions — MEMLNaut-NISPS
This project uses **bd** (beads) for issue tracking. Run `bd onboard` to get started.
One C++20 NISPS core targets RP2350 firmware and the Manifold React/WASM browser app. Read
`MAP.md`, `ALIGNMENT.md`, and `docs/AGENT-REFERENCE.md`; for Manifold UI work read
`manifold/ONBOARDING.md` first. Parameter JSON schemas generate both C++ and TypeScript
contracts.
## Quick Reference
## Gates
```bash
bd ready # Find available work
bd show <id> # View issue details
bd update <id> --status in_progress # Claim work
bd close <id> # Complete work
bd sync # Sync with git
bash scripts/build-cpp-tests.sh
bash scripts/parity-check.sh
bash scripts/lint-cpp.sh
scripts/build-firmware.sh [VARIANT]
cd manifold && bun run typecheck && bun run test && bun run build
```
<!-- BEGIN BEADS INTEGRATION -->
## Issue Tracking with bd (beads)
- `nisps/` must remain platform-neutral and allocation-free in hot paths: no heap, no
virtual dispatch, `.f` float literals, deterministic per-instance RNG, and the memory/
hot-path attributes from `nisps/core/perf.hpp`.
- Firmware and WASM use the same engines/modes. Cross-platform outputs must retain parity;
do not patch target-specific behavior around a core mismatch.
- Schema changes require codegen and both generated-language outputs in the same change.
- Audio-thread/WASM-worklet communication stays bounded and real-time safe.
- Preserve dual-core firmware ownership and SPSC synchronization; UI/control work must not
block the audio path.
- UI behavior changes need the targeted unit/E2E evidence and the verification chokepoints
documented in the detailed reference.
**IMPORTANT**: This project uses **bd (beads)** for ALL issue tracking. Do NOT use markdown TODOs, task lists, or other tracking methods.
### Why bd?
- Dependency-aware: Track blockers and relationships between issues
- Git-friendly: Auto-syncs to JSONL for version control
- Agent-optimized: JSON output, ready work detection, discovered-from links
- Prevents duplicate tracking systems and confusion
### Quick Start
**Check for ready work:**
```bash
bd ready --json
```
**Create new issues:**
```bash
bd create "Issue title" --description="Detailed context" -t bug|feature|task -p 0-4 --json
bd create "Issue title" --description="What this issue is about" -p 1 --deps discovered-from:bd-123 --json
```
**Claim and update:**
```bash
bd update bd-42 --status in_progress --json
bd update bd-42 --priority 1 --json
```
**Complete work:**
```bash
bd close bd-42 --reason "Completed" --json
```
### Issue Types
- `bug` - Something broken
- `feature` - New functionality
- `task` - Work item (tests, docs, refactoring)
- `epic` - Large feature with subtasks
- `chore` - Maintenance (dependencies, tooling)
### Priorities
- `0` - Critical (security, data loss, broken builds)
- `1` - High (major features, important bugs)
- `2` - Medium (default, nice-to-have)
- `3` - Low (polish, optimization)
- `4` - Backlog (future ideas)
### Workflow for AI Agents
1. **Check ready work**: `bd ready` shows unblocked issues
2. **Claim your task**: `bd update <id> --status in_progress`
3. **Work on it**: Implement, test, document
4. **Discover new work?** Create linked issue:
- `bd create "Found bug" --description="Details about what was found" -p 1 --deps discovered-from:<parent-id>`
5. **Complete**: `bd close <id> --reason "Done"`
### Auto-Sync
bd automatically syncs with git:
- Exports to `.beads/issues.jsonl` after changes (5s debounce)
- Imports from JSONL when newer (e.g., after `git pull`)
- No manual export/import needed!
### Important Rules
- ✅ Use bd for ALL task tracking
- ✅ Always use `--json` flag for programmatic use
- ✅ Link discovered work with `discovered-from` dependencies
- ✅ Check `bd ready` before asking "what should I work on?"
- ❌ Do NOT create markdown TODO lists
- ❌ Do NOT use external issue trackers
- ❌ Do NOT duplicate tracking systems
For more details, see README.md and docs/QUICKSTART.md.
<!-- END BEADS INTEGRATION -->
## Landing the Plane (Session Completion)
**When ending a work session**, you MUST complete ALL steps below. Work is NOT complete until `git push` succeeds.
**MANDATORY WORKFLOW:**
1. **File issues for remaining work** - Create issues for anything that needs follow-up
2. **Run quality gates** (if code changed) - Tests, linters, builds
3. **Update issue status** - Close finished work, update in-progress items
4. **PUSH TO REMOTE** - This is MANDATORY:
```bash
git pull --rebase
bd sync
git push
git status # MUST show "up to date with origin"
```
5. **Clean up** - Clear stashes, prune remote branches
6. **Verify** - All changes committed AND pushed
7. **Hand off** - Provide context for next session
**CRITICAL RULES:**
- Work is NOT complete until `git push` succeeds
- NEVER stop before pushing - that leaves work stranded locally
- NEVER say "ready to push when you are" - YOU must push
- If push fails, resolve and retry until it succeeds
Use `ergo` for coding tasks. Keep `MAP.md`, `ALIGNMENT.md`, and affected docs synchronized
with code.

View file

@ -6,96 +6,165 @@
A research platform for interactive ML control of audio. We're building it to figure out what works and what doesn't — different ergonomics and ergodynamics of parameter sets, modes, ML architectures, audio engines, UI, and UX. Therefore: keep most/all parameters tweakable, ML/engine/UI/UX should each be configurable on their own axis, and the codebase has to enable/assist agentic AI coding patterns (confident changes, verifiable without hardware).
The clean-slate rewrite (2026-04-29) consolidates everything into one C++20 codebase compiling to firmware AND WASM, with SolidJS playground primitives composed into TSX mode components, and JSON schemas as the firmware↔browser parameter contract.
**Target vision (operator, 2026-07-20):** (1) one C++20 NISPS core serving RP2350 firmware and the browser, performance-sensitive on the MCU; (2) firmware modes runnable as modes in Manifold; (3) Manifold defaults to curated presets/modes, with the maximalist surface behind an "advanced" dev mode used to author them; (4) PlatformIO for hardware, no more .ino; (5) Manifold doubles as interface/editor for the hardware MEMLNaut (settings, presets, training, examples, visualisation).
The clean-slate rewrite (2026-04-29) consolidated everything into one C++20 codebase compiling to firmware AND WASM. Since 2026-07-13 (P1) the sole browser app is the React Manifold. JSON schemas remain the firmware↔browser parameter contract. A full-repo audit (2026-07-21, `docs/specs/recon/simplification-audit-2026-07.md`) grounds the entries below; mitigations are phased in `docs/specs/plans/simplification-plan.md`.
## Top defects (ranked by mission impact)
### 1. Browser-only audio engines incomplete (2026-04-29)
### 1. The mode layer is not shared: WASM re-orchestrates modes by hand (2026-07-21)
**What.** Stream 9 stubbed `C15Mode` as a placeholder ("C15 mode TODO"). The C15 worklet/bridge from the legacy playground is in `.local/playground-archive/js/synth/` but not wired in. Mic input for XIASRI / SoundAnalysisMIDI is also TODO — the UI renders but doesn't capture audio.
**What.** `nisps/modes/` — the CRTP layer binding ML config, engine, voice-space and I/O — compiles only into firmware. `nisps/wasm/bindings.cpp` includes engines and ML primitives but zero mode headers, and Manifold re-assembles mode behaviour (jolt stepping, OU, routing) in TS. "Firmware and WASM share the same modes" is true only at the engine level; every ModeBase behaviour must be mirrored browser-side by hand.
**Why it blocks the mission.** "Browser engines ⊇ firmware engines" was a non-negotiable. Without C15 + mic input, the playground can't fully demonstrate the modes; users can't audition XIASRI or SoundAnalysisMIDI in the browser.
**Why it blocks the mission.** Vision bullet 2 is precisely this. Until the control-tick orchestration exists once in C++, every new mode behaviour is a dual implementation with drift risk.
**Rough cost.** ~1–2 days. C15 wiring is mostly straightforward porting from the archived bridge. Mic input requires the engine-host to expose an input stream to the worklet (small worklet refactor).
**Rough cost.** Spec first, then ~a week: storage-policy the ModeBase orchestration the way P2 did MLPCore (verified shape in plan §6.5a — *not* binding monolithic mode objects, which would contradict the locked two-instance RT architecture). Related honesty gap: Manifold currently catalogues 4 modes that structurally cannot run in the browser (no mic input, event-only engines) — plan §6.5b (absorbs the old C15/mic-input defect; C15 itself lives on `archive/playground-solidjs`).
### 2. Per-iteration loss curve not plumbed through WASM (2026-04-29)
### 2. No curated/advanced split and no in-UI mode picker — the UI fights vision 3 (2026-07-21)
**What.** Stream 7's WASM C API exposes `_nisps_ml_train` but only returns the final loss; `MLP::loss_history()` exists in C++ but isn't reached. `lossHistory` in `mlStore` is a single-element array per training run.
**What.** Manifold is 100% dev-maximalist: five drawers of everything, no preset data model to author against, and mode switching exists only via the debug hook — there is no instrument picker in the UI at all (the plumbing, `ctx.modes`/`setModeId`, already exists unused). A stratum of decorative controls (training-param sliders, master volume, bpm, A/B, snapshots, fabricated gradient health) renders real-looking UI that drives nothing.
**Why it blocks the mission.** Gradient flow / loss visualization is a core UX affordance for "is the network learning?" — a research-mode debugging tool that's been implemented end-to-end in C++ but stops at the WASM boundary.
**Why it blocks the mission.** The default experience is supposed to be curated presets; the advanced surface is the authoring tool. Neither exists, and the decorative stratum actively misleads research use.
**Rough cost.** Half a day. Add `nisps_ml_train_with_history` (or extend the existing call) returning a pointer to the loss array; copy on the JS side.
**Rough cost.** Product-model decision first (plan §7.6), then incremental: picker is days; the curated-preset model seeds from `backends/presets.ts` + schemas; disclosure via per-drawer depth levels — which as of §6.5e (2026-07-21) has its first genuinely advanced-only consumer, the training-health panel, so the mechanism is proven rather than theoretical. The decorative stratum itself went in the Phase-1 sweep.
### 3. WASM MLP architecture is fixed (2026-04-29)
### 3. Manifold-as-hardware-editor is a facade (2026-07-21)
**What.** WASM compiles `MLP<2, 10, 14, 18, 126>` only. Modes with smaller `output_size` use a slice; modes that want different hidden layer shapes (e.g. `[10, 10, 14]` for smaller modes) can't get them in the browser.
**What.** Vision bullet 5 exists as a 237-line Web Serial shell: sound connect lifecycle, zero protocol (`saveModel`/`restoreModel`/`getSettings` are literal stubs), and firmware has no serial command surface or on-device persistence to talk to.
**Why it blocks the mission.** Per-mode ML architecture variation is one of our four "research dimensions". The fix isn't urgent for the current mode set (all schemas are within the universal arch's capacity), but the moment we want to experiment with bigger networks or different shapes, this hits.
**Why it blocks the mission.** The hardware research loop (train on device, inspect/curate in browser) is closed only by this bridge.
**Rough cost.** Medium. Either compile multiple WASM modules (one per arch shape; load on demand) or move to a runtime-shaped MLP (loses some compile-time perf). Templates-vs-runtime is a research-vs-firmware-perf tradeoff worth a separate decision doc.
**Rough cost.** Week+, spec-first (plan §6.5d). The right discipline already exists in-repo: useq-celium's C-header wire truth + TS mirror + parity test; settings payloads should derive from schema codegen.
### 4. `NISPS_AUDIO_FUNC` host fallback is misshapen (2026-04-29)
### 4. Dead mass and registry sprawl across every layer (2026-07-21)
**What.** `nisps/core/perf.hpp` defines `NISPS_AUDIO_FUNC(decl) decl` for the host but the firmware path `__not_in_flash_func(name)` takes only a function name (it stringifies into a section attribute). The two forms don't match. Stream 6 (firmware glue) avoided the macro to dodge the inconsistency, but it's still a footgun.
**What.** *Phase 1 landed 2026-07-21 and removed the bulk of this:* the dead focus/altitude UI system, the decorative control stratum, 12 dead WASM API entries across the 5-file registration chain, the vendored daisysp tree, retired-playground artifacts and root planning relics, 5 unused primitives, the duplicate backend editor and catalogue, `voice_space.hpp`, `fixed_buffer.hpp`, the dead perf-macro regime, and the OSC bridge twin. What remains is the *registry* half: mode identity spread across ~6 hand-maintained registries with demonstrated drift, MLP dims typed twice, per-mode schema blocks hand-written in C++, and assorted stale specs presenting a deleted world as present tense.
**Why it blocks the mission.** Future agents touching `nisps/` will hit this. Either decoration form in the codebase is fine; what's wrong is that the same call site shape doesn't work both places.
**Why it blocks the mission.** The registries are dual-truth bugs waiting to fire (one already did: the selftest table). Stale specs are agent-confusing surface area.
**Rough cost.** Tiny. Pick one form and apply consistently:
- Option A: `NISPS_AUDIO_FUNC` decorates a function name (e.g. `void NISPS_AUDIO_FUNC(my_callback)(...) { ... }`). Host stub: `#define NISPS_AUDIO_FUNC(name) name`.
- Option B: separate `NISPS_AUDIO_FUNC_BEGIN` / `_END` markers around the function, or a different macro.
Pick A. Update perf.hpp + every `nisps/` use site.
**Rough cost.** Plan phase 3 (~2–3 days, codegen takes ownership) plus the docs disposition pass (§8). The behaviour bugs found en route (dataset-cap divergence, VCV 2-D input truncation, VCV audio-thread race and JSON) were fixed in phase 2 on 2026-07-21.
### 5. RMSProp deferred from `nisps/ml/` (2026-04-29)
### 5. Performance is measured on the host but not on the target that constrains it (2026-07-21)
**What.** The legacy MLP supported both SGD and RMSProp paths. Stream 2 shipped only SGD as MVP. The architecture spec called for both. Documented as "follow-up bd issue when needed".
**What.** *Mostly closed 2026-07-21.* Size: the Phase 4 firmware CI job reports per-variant
flash/RAM on every push. Time: `scripts/bench-engines.sh` now reports per-engine ns/sample,
blocks/s and realtime factor on native AND WASM from one source
(`tests/cpp/engine_bench.cpp`), engines driven into a working state, with `--compare` for
per-engine deltas and a report step in CI. An engine getting 3x slower is now visible.
**Why it blocks the mission.** Optimizer choice is one of the things research wants to vary. Not blocking for the current XOR-style fits, but as soon as we tune for harder loss landscapes, RMSProp will matter.
**What is left.** The numbers are HOST numbers. The mission's performance constraint is the
**RP2350 at 150 MHz**, and nothing measures there — a host realtime factor of 100x says
nothing about whether an engine fits in the MCU's per-block budget, and the two targets have
different FPU, cache and memory behaviour. The honest next step is an on-device timing report
(cycle counter around the audio callback, published over the existing display/serial surface),
which lands naturally with the hardware editor (defect 3) since that is what gives firmware a
command surface to report through.
**Rough cost.** A day. Port the firmware's RMSProp from `src/memlp/MLP.cpp:415-543` (decay 0.9, epsilon 1e-6, gradient accumulation, batch size). Add tests for batch training convergence.
**Rough cost.** Host half is done. On-device: ~a day, and it wants defect 3's serial protocol
to have somewhere to send the number.
### 6. bd Dolt remote sync flaky (2026-04-29)
### 6d. One like still heaves the whole mapping (2026-07-25)
**What.** During the rewrite, `bd close` repeatedly failed with "database `beads_meml` not found on Dolt server" or similar lock conflicts. Several agent-side bd closures could not be performed and have orchestrator-side closure notes instead. May leave stream issues in inconsistent states.
**What.** A thumbs-up trains at `lr 1.0 x 1000 iterations` on every gesture. `ml_bench`
U1/U4 measure **lurch** — how far the mapping the musician is playing moves per single
gesture, averaged over the field: `lurch_max` 1.08 against a [0,1] output range, i.e.
one thumbs-up can move the mapping somewhere in the space by more than the entire output
range. Retention (how much of the previous teaching survives) is 0.38; at `iters=1` it
is 0.80.
**Why it blocks the mission.** Beads is the canonical task tracker; if it can't reliably sync, future agents lose visibility into what's done vs in-progress.
**Why it blocks the mission.** RMSProp did NOT fix the positive lurch — normalising the
step size does not change its dose. The negative path now exposes upstream-style repeated
small steps (rate/lifetime/LR are live controls), so its old fixed ~70x comparison is no
longer current. Positive teaching remains one enormous blocking train, and the two doses
still need a matched head-to-head rather than independent tuning.
**Rough cost.** Investigate Dolt server config + lock semantics. Out of scope for the rewrite itself.
**Rough cost.** Cheap to change, expensive to choose: the tuning space is now measurable
(`ml_bench` U4 sweeps dose; U1 sweeps upstream's soft-target alpha, where alpha=1 is
NISPS today). It wants a matched-N head-to-head, not a guess.
## Open mission questions
### Q1: Per-mode MLP architectures or one shared shape? (2026-04-29)
Schemas currently declare per-mode `hidden_layers` (some `[10, 10, 14]`, some `[10, 14, 18]`). Browser is fixed at one shape; firmware compiles per-mode. This works for now. Is the mission served by maintaining per-mode shapes (research diversity) or by collapsing to one (simpler ops)?
Schemas declare per-mode dims and since P5.3 both targets honour them. Is the mission served by maintaining per-mode shapes (research diversity) or collapsing to one (simpler ops)? Note the audit found all 9 mode schemas share copy-pasted ML defaults and 20 params are anonymous placeholders — the per-mode diversity is currently nominal (plan L40).
### Q2: How to express "advanced" features (gradient flow, weight health) without cluttering modes? (2026-04-29)
### Q2: Engine event taxonomy (2026-04-29)
Current playground reproduces the a-immersive "Advanced" toggle. Power features hide behind it. Is this the right model, or should the mode UI itself decide what's exposed (some modes are "expert-only", some are simpler)?
`ControlEvent` is a flat enum consumed by the two sequencer modes. Revisit when a third event-emitting mode lands.
### Q3: Engine event taxonomy (2026-04-29)
### Q3: Should Manifold stay desktop-first? (2026-04-29)
`nisps/modes/base.hpp` exposes a `ControlEvent` ring buffer pop_events interface for sequencer modes (BreakOr, Elysiamorf). Currently events are a flat enum. As we add more event-emitting modes (custom MIDI mappings, lighting, networked control), how should the event vocabulary grow? Open question; revisit when we add the third event-emitting mode.
### Q4: Should the playground stay desktop-first? (2026-04-29)
The original a-immersive was mobile-first ("designed for touch / foldable phone use"). The SolidJS rewrite is desktop-first by default. If the research story is "the user holds a phone and pinches to zoom while a synth runs in their pocket", we'll need a responsive pass. Defer until we have user data.
Legacy a-immersive was mobile-first; Manifold is desktop-first. Defer until user data exists.
## Deferred / accepted debt
- **EOC effects chain integration** — out of v1 rewrite (recon flagged as legacy complexity).
- **ShapeSeq sequencer** — gated behind `?shapeseq=1` in legacy; out of v1.
- **Modular engine (Phase E)** — newer JS-side feature in legacy; out of v1.
- **Engine configuration panel** (SPEC-controls Part 8) — backlog. Would let users tune network architecture, loss, optimizer at runtime. Currently compile-time only.
- **VCV Rack module** — used to consume `nisps-core/`. Now gone. If revived, it'd consume `nisps/` directly via CMake; not currently maintained.
- **Negative-feedback "Avoid" uses `move_weights`, not the firmware's k-NN geometric centroid push** (2026-06-18) — porting the 3-mode "Down Action" feature (`nisps/ml/feedback.hpp`, from upstream `InterfaceRL` branch `feat/feedback-explore-modes`) deliberately mapped `Avoid` to the existing `MLP::move_weights` Gaussian perturbation rather than re-porting the old geometric-centroid-toward-liked-prototype push (which depended on a firmware-only `ReplayMemory` with reward accumulation/decay). `RandomiseMlp` likewise uses `draw_weights(spread)` rather than the old asymmetric `RandomiseWeightsAndBiasesLin(-0.9,1.1,-0.9,0.3)`. Accepted divergence — **not a bug**. If richer geometric/contrastive avoidance is wanted later, it belongs in a separate replay-memory component, not `FeedbackController`.
- **Manifold dock splits `state`/`muted`/`armed` into three fields, diverging from the deployed conflated `frozen`↔`muted`** (2026-06-28) — the deployed a-immersive override system maps the heatmap-popup `frozen` and the group-drawer `muted` onto ONE underlying field. The Manifold per-output model (`manifold/src/dock/output-state.ts`, folded onto `MFParam`) deliberately separates them: `status` carries the off/fixed/live tri-state, `muted` is downstream-silence (still computed + visible), `armed` is solo/focus-training. Cleaner semantics; intentional divergence (dock-spec §3.3, open choice 3). Note `muted`-downstream and the `soloMode` gradient-mask variants (mask-gradients / zero-loss / dont-care) are UI+state only so far — the engine C API exposes `set_focus` but not per-mode gradient masking nor a downstream mute gate yet (TODOs in `ConsoleApp.tsx` / `Drawers.tsx` reference rl-feedback-design §3 and dock-spec §3.3).
- **EOC effects chain, ShapeSeq sequencer, modular engine (Phase E)** — legacy features consciously out of the v1 rewrite; revisit only if a mode wants them.
- **Inputs multi-source composition** (2026-06-28, reaffirmed 2026-07-21) — mix-and-match pad+gamepad+MIDI is a recorded, unreversed decision; the UI currently enforces exclusive single-source and the composition machinery sits dormant *by design*. Schedule or keep dormant — but the inputs-spec must stop presenting composition as current behaviour (plan §8).
- **Schema content is partially placeholder** (2026-07-21) — 20 anonymous "Param NN" slots across paf_synth/channel_strip/xiasri and copy-pasted ML defaults across all 9 modes. Name them during the first curated-preset pass per mode (plan §6.5c), or shrink `output_size` where the engine allows.
- **Geometric-dislike deliberate divergences** (2026-07-25): (1) the degenerate-branch RNG draws from deterministic `nisps::Rng`, not libc `rand()`; (2) each live rejection computes its liked centroid at its own stored input, rather than upstream reinterpreting every old rejection around the cursor's current position; (3) live negatives are applied as deterministic per-item RMSProp steps rather than one shuffled `TrainBatch`; (4) a repeated nearby rejection refreshes its lifetime (upstream's current dedup path leaves the original timestamp untouched); (5) upstream's default removal of a nearby positive is not yet adopted because Manifold also has a separate positive MLP dataset that would keep pulling; (6) `RandomiseMlp` uses `draw_weights(spread)` rather than old asymmetric ranges. Native↔WASM parity covers the elapsed-time replay path.
- **Manifold dock splits `state`/`muted`/`armed`** (2026-06-28) — deliberate divergence from the deployed conflated `frozen`↔`muted` model (dock-spec §3.3). `muted`-downstream and the `soloMode` gradient-mask variants remain UI-only; the C API exposes `set_focus` but no per-mode gradient masking yet. (The audit found `soloMode` behaviourally inert in the controller — plan L20 trims it until `train_masked` exists.)
## Recently resolved (delete after a few weeks)
- 2026-04-29: Three-implementation ML duplication (firmware `memlp`, `nisps-core`, JS engine) collapsed to single `nisps/` C++ codebase.
- 2026-04-29: Firmware mode forks (~280–400 lines duplicated across 8 modes) collapsed via `nisps/modes/base.hpp` CRTP scaffold; concrete modes are now ~50–130 lines.
- 2026-04-29: meml-ues double-scaling MSE bug fixed in `nisps/ml/loss.hpp` + `mlp.hpp`.
- 2026-04-29: `nisps-core/` retired; firmware is the canonical source of truth for ML.
- 2026-04-29: `src/memlp/` submodule deleted.
- 2026-04-29: Legacy playground variants (a-immersive.html, b-workbench, c-journey, designs.html, all `js/`) deleted in favor of SolidJS scaffold.
- 2026-04-29: Native↔WASM parity verified within 1e-5 (max delta 2.4e-7) for representative ML + engine outputs.
- 2026-07-25: **The geometric dislike is re-based on upstream `e291192` (defect 6b).**
`kGeometricPushScale` 0.5 -> 1.0, `kNegLRBase` 0.5 -> 1.5, and the `/(1+len)` taper
deleted — upstream's own comment is that a "no" should clearly move the mapping away
even from a sound already far from the liked region, which is exactly the case the
taper killed. Cold start folded into the same path (random direction when nothing is
liked yet) instead of the superseded negative-LR branch, so a "no" now moves the
mapping before any likes exist (`ml_bench` E1: 0 -> 2.3e-3). One dislike moves the
mapping **5.3e-2**, up from 1.6e-2 after the RMSProp fix and 5.3e-5 before it — a
~1000x change end to end, and now within ~4x of the legacy Diffuse design instead of
~4100x (`ml_bench` A4). The follow-up now adopts upstream's repeated-all-negatives
schedule and full-strength wall-clock lifetime through a deterministic elapsed-time
core seam. After operator calibration, Manifold defaults to 0.003 LR, 200 Hz and
2500 ms, exposes all three in the expanded Learning panel, and allows rate/lifetime
zero as an explicit one-shot A/B. The shared core fallback and the panel's explicit
"Upstream defaults" preset remain 0.001 LR, 200 Hz and 2500 ms.
- 2026-07-25: **`InterfaceRL` is back in the tree (defect 6c).** Vendored verbatim from
memllib `e291192` at `firmware/MEMLNaut-NISPS/lib/memllib/reference/` — outside `src/`,
so PlatformIO never compiles it. Upstream drift in the feedback subsystem is a `diff`
again rather than an archaeology session.
- 2026-07-25: **The optimiser mismatch (defect 6) is fixed.** `nisps/ml/training.hpp` was
SGD-only while upstream `memlp` (`ea777502`) applies **RMSProp everywhere**, so every
learning rate we ported landed in an optimiser that reads it differently — an RMSProp
`lr` is a normalised step, an SGD `lr` multiplies the raw gradient. `rmsprop_step()`
now ports `Layer.h:239 ApplyAccumulatedGradients` exactly (decay 0.9, eps 1e-6,
sq-avg clamp 1e6, one-sided adjusted-LR clamp 1.0), with the per-weight running
squared-gradient average living in the storage policies so the zero-heap contract
holds. Measured on `ml_bench` D1: one geometric dislike moves the mapping **1.6e-2,
up from 5.3e-5**, and repeated presses now converge on the intended 0.5 push (0.12 at
10 presses, 0.56 at 100) instead of creeping linearly. Golden vector stages 2 and 3
were re-captured; stages 0 and 1 are pre-training and did not move. What this does NOT
fix: the dose asymmetry, now tracked as defect 6d.
- 2026-07-21: **Q4 (who owns memllib) closed.** Vendored at `firmware/MEMLNaut-NISPS/lib/memllib/`
from upstream `e291192`; the submodule and the fork are both gone. **Q5 (legacy feedback modes)
closed** — operator kept all four (`RandomiseOutputs`/`RandomiseMlp`/`Diffuse`/`on_drag`) as
building blocks for comparing how instruments feel under different behaviours, which upholds
rather than reverses `docs/adr/rl-feedback-design.md`.
- 2026-07-21: **Training-health telemetry (old defect 6) is gone** — the browser reads the real
per-iteration loss the core records, both fabrication sites are deleted (`wasm-worker.ts`'s
1-element array AND `wasm-iml.ts`'s sync-train twin, which the audit missed), and the display
sits behind the existing `expanded` drawer depth. The firmware buffer stays, per the L25 call.
- 2026-07-21: **Training-health telemetry (old defect 6) is real.** `nisps_ml_loss_history` now
crosses all five WASM registration layers, so the browser reads the SAME per-iteration curve the
firmware MLP records; `wasm-worker.ts`'s 1-element `new Float32Array([loss])` fake is gone from
both the sync and the async train paths; and the curve + the already-plumbed `get_layer_stats`
render in `manifold/src/console/TrainingHealth.tsx` at the Learning drawer's `expanded` depth.
The firmware buffer stays, per the operator call — it is the record the hardware editor
(defect 3) will read. One more fabrication went with it: `ConsoleCtx.loss`, a synthetic
`prev * 0.82` series no drawer read. With no history the panel says "no training run yet"
rather than drawing a plausible curve.
- 2026-07-21: **Arduino-CLI build machinery (old defect 3) is gone.** Phase 4 replaced it with a
PlatformIO project: one `[env:]` per variant is now the only variant registry, the `.ino`-mutating
Python/sed machinery and the `NISPS_ST_*` token-paste table and the sketch symlink forest and the
global TFT_eSPI mutation are all deleted, memllib is vendored (no submodule), and firmware finally
entered CI — three representative envs per run, which is what would have caught the SelfTest
variant sitting broken. All 16 envs build; sizes match arduino-cli within ~520 bytes.
- 2026-07-21: Full-repo simplification audit landed (recon + plan + this rewrite). Superseded entries removed: "browser-only engines incomplete" (→ defect 2/plan 5b), "loss curve not plumbed" (→ defect 7), "NISPS_AUDIO_FUNC misshapen" (→ plan Phase 1, S21/L13), stale "VCV not currently maintained" note (vcv/ is active and consumes `nisps/` directly post-P6).
- 2026-07-18: Browser curve maths unified onto the canonical `nisps/core/math.hpp` catalog at P4; four silently-divergent TS curves re-baselined.
- 2026-07-14: WASM MLP fixed-architecture defect resolved by P2 (`MLPCore<Storage>`; browser runtime-shaped, firmware zero-heap fixed).

200
CLAUDE.md
View file

@ -1,200 +0,0 @@
# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## Overview
MEMLNaut-NISPS — Neural Interactive Shaping of Parameter Spaces. A research platform for interactive ML control of audio. **One C++20 codebase** (`nisps/`) compiles to two targets:
1. **RP2350 firmware** for the MEMLNaut hardware platform (`firmware/`).
2. **WASM** in a SolidJS browser playground (`playground/`) — same engines + ML, run through an AudioWorklet.
Browser audio engines are a superset of firmware engines (C15 is browser-only). Parameter contracts are JSON schemas (`schemas/`) with codegen producing both C++ headers and TypeScript types.
Project documentation: https://musicallyembodiedml.github.io/memlnaut/approaches/nisps
For the codebase index, see `MAP.md`. For strategic gaps and open mission questions, see `ALIGNMENT.md`.
**For anything UI-related in the Manifold front-end (`manifold/`), read `manifold/ONBOARDING.md` first** — it's a single-file agent orientation (run/build/deploy/test, the UI/engine-spine/WASM layering, the convertible Stages, the Dock + drawers, and the non-obvious gotchas).
## The `nisps/` core
```
nisps/
├── core/ types, perf attrs, concepts (AudioEngine, MLEngine, Mode), fixed/ring buffers, deterministic RNG, math
├── ml/ MLP class template (4-layer, 3 hidden); SGD, gradient clipping, spread-aware Xavier init,
│ RL move_weights with output pin mask + per-layer scaling + weight decay
├── dsp/ biquad, delay, reverb, filter, env, osc, pitch_shift, dc_blocker
├── engines/ 8 audio engines (paf_synth, channel_strip, xiasri, verb_fx, memlcelium, breakor,
│ elysiamorf, analysis) + NoOpEngine. Each satisfies the AudioEngine concept.
├── modes/ 8 platform-agnostic modes binding {ML, engine, voice space, abstract I/O channels}.
│ CRTP base eliminates the duplication that plagued firmware modes.
└── wasm/ Emscripten C API bindings (compiled only for WASM target)
```
Build: `cmake -S nisps -B nisps/build -G Ninja && cmake --build nisps/build && ctest --test-dir nisps/build`.
Tests: 4 executables (`nisps_core_tests`, `nisps_dsp_engine_tests`, `nisps_modes_tests`, `nisps_golden_tests`). Run all: `bash scripts/build-cpp-tests.sh`. Parity vs WASM: `bash scripts/parity-check.sh` (asserts native and WASM produce identical outputs within 1e-5).
### Performance contract (RP2350)
These rules apply to **all** code under `nisps/`. They are inert in WASM but kept globally for consistency.
- **No heap.** No `new`, `malloc`, `std::vector` in hot paths. Use `nisps::FixedBuffer<T, N>` or `std::array<T, N>`.
- **Constants discipline.** Float literals >255 used in hot paths must be `static const float val = X.f;` not inline.
- **`.f` suffix on all float literals.** No double promotion in audio/inference paths.
- **Memory section attributes.** Apply `NISPS_AUDIO_MEM` / `NISPS_AUDIO_FUNC` / `NISPS_APP_SRAM` / `NISPS_HOT` / `NISPS_FORCE_INLINE` (from `nisps/core/perf.hpp`).
- **No virtual dispatch in audio path.** `AudioEngine` and `Mode` are C++20 concepts, not interfaces.
- **Deterministic RNG.** All RNG state is per-instance; constructors take a seed; cross-platform parity tests rely on this.
Lint: `bash scripts/lint-cpp.sh` warns on missing `.f` and fails on heap/`Arduino.h` use under `nisps/`.
## The `firmware/` target
```
firmware/MEMLNaut-NISPS/
├── MEMLNaut-NISPS.ino # Entry point; mode selected via #define MEMLNAUT_MODE_TYPE
├── glue/
│ ├── audio_driver.hpp # memllib AudioDriver block callback → Mode::process per-sample
│ ├── peripherals.hpp # joystick / pots / buttons → Mode::set_input + ML primitives
│ ├── midi_io.hpp # MIDI in → mode handlers; drain ControlEvent ring → MIDI UART
│ ├── mode_select.hpp # type aliases firmware mode name → nisps::modes::*Mode
│ ├── input_router.hpp # wire_inputs() entry point
│ ├── output_router.hpp # drain_outputs() entry point
│ └── settings_view.hpp # wire_settings(): TFT/rotary menu (Joystick Dual/Single for 4-in modes)
└── src/{memllib,daisysp,nisps} # symlinks (Arduino-CLI sketch tree convention)
```
Build: `scripts/build-firmware.sh [VARIANT]`. Verified compiling for PAFSynth, ChannelStrip, BreakOr on `rp2040:rp2040:solderparty_rp2350_stamp_xl:opt=Optimize3` with `-std=gnu++20`. Flash: `scripts/flash-firmware.sh`. One-shot: `scripts/build-and-flash-firmware.sh`.
### Dual-core orchestration (firmware)
- **Core 0**: UI loop, ML inference (`Mode::tick_control`), peripheral polling (5ms period).
- **Core 1**: Real-time audio processing (`Mode::process`), MIDI polling.
- **Sync**: `nisps::core::ring_buffer` (templated SPSC lock-free, replaces pico/util/queue) + memory barriers (`nisps::core::memory_barrier`, `write_volatile`/`read_volatile`).
## The `playground/` target
```
playground/ # Vite + SolidJS + TypeScript
├── src/
│ ├── primitives/ # 16 UI building blocks (Slider, JoyMap, Heatmap, …) + .demo.tsx for /dev/primitives
│ ├── modes/ # one TSX per firmware mode + C15Mode (browser-only); ModeShell + ModeSwitcher + mode-runtime
│ ├── stores/ # Solid stores (ml, input, output, mode, control, session, exploration, bus + persistence)
│ ├── audio/ # engine-host + AudioWorklet processor (loads nisps.wasm separately on each thread)
│ ├── ml/ # WasmIML class + disposable async-training Worker + dataset
│ ├── input/, output/ # pure-fn pipelines (deadzone→zoom→curve→smoothing→momentum, then global curve→smoothing→slew→freeze)
│ ├── features/ # heatmap, snapshots, A/B compare, region/param pin, trail, weight health, gradient flow
│ └── debug/probe.ts # synchronous window.__nisps for Playwright
├── public/ # nisps.{wasm,js}, c15.{wasm,glue}
└── tests/e2e/ # Playwright specs + helpers
```
Dev: `cd playground && bun install && bun run dev`. Build: `bun run build`. Typecheck: `bun run typecheck`. E2E: `bunx playwright test`.
### Stores + reactivity
All stores use SolidJS `createStore` for objects, `createSignal` for primitives. ML outputs are stored in a separate Float32Array signal (per the migration plan's perf guidance). Persistence (debounced 200ms localStorage round-trip) wired in `playground/src/stores/persistence.ts`. The signal bus (`bus.ts`) handles cross-store events (`ml.*`, `mode.*`, `pin.*`, `ui.*`).
### Control surface
Three compound axes (Boldness / Memory / Precision) interpolate per-axis tables to drive ~6 underlying parameters each, with offset overrides ("trim-pot" model). State is in `control-store`. Six built-in control presets (Default, First Touch, Jazz Hands, Sculptor, Improviser, Microscope) available via the ModeShell control bar.
### Debug probe (Playwright)
`window.__nisps` is exposed synchronously and bypasses Solid reactivity (uses `untrack`/`batch`). API matches the `.local/recon/04-playground.md` spec — `setInputs`, `getOutputs`, `getLoss`, `train`, `thumbsUp`/`thumbsDown`, `randomise`, `clearExamples`, `inferBatch`, `getLayerStats`, `saveState`, etc.
## The `schemas/` + `codegen/` contract
Each mode has a `schemas/modes/<mode>.json` describing its parameters (name, label, range, default, curve, group), ML config (input/output sizes, hidden layers), voice spaces (names — bodies are inline lambdas in the C++ engine), and UI config. The meta-schema at `schemas/schema.json` validates these.
Codegen (`bun run codegen/generate.ts`) emits:
- `nisps/modes/generated/<mode>_schema.hpp` — `constexpr` C++ data, namespace `nisps::modes::generated`, re-exports `nisps::Curve` from `nisps/core/math.hpp`.
- `playground/src/modes/generated/<mode>_schema.ts` — typed const objects + per-mode params interface.
Codegen is idempotent. Golden test ensures regenerating produces byte-identical output.
## WASM bridge
Two WASM instances at runtime:
1. **Main thread** (`playground/src/ml/wasm-iml.ts`): ML inference + sync training + RL primitives. Update store after each call. Async training via disposable Web Worker (`wasm-worker.ts`).
2. **AudioWorklet** (`playground/src/audio/worklet/nisps-processor.ts`): runs engine `process_block` per audio block. Loads `nisps.wasm` directly via `WebAssembly.compile` (no Emscripten glue in worklet). Bytes posted from main thread.
C API is in `nisps/wasm/bindings.cpp`. Build: `bash scripts/build-wasm.sh` (~94KB output to `playground/public/`).
The WASM target is fixed at `MLP<2, 10, 14, 18, 126>`. Modes with smaller `output_size` use the first N outputs only.
### Known limitations
- Loss history not yet plumbed through C API; `lossHistory` in the store is a single-element array per training run.
- Engine MLP architecture is fixed at compile time — supporting per-mode hidden-layer shapes would need either multiple WASM modules or runtime variation.
- Mic input through the worklet for XIASRI / SoundAnalysisMIDI is not wired; UI scaffolds render but feature is TODO.
- C15 voice space integration in C15Mode is a placeholder.
- The browser Jolt/OU controls (`playground/src/ml/jolt.ts`, `playground/src/output/ou-explore.ts`) reimplement the gesture math in TS rather than calling the C++ `ml::Jolt`/`ml::OUNoise` through WASM. They drive weights via the existing `nisps_ml_get/set_weights` bindings and use `Math.random()` (not the deterministic `Rng`) — fine for stochastic exploration aids, but firmware↔browser bit-parity of the noise itself is intentionally not guaranteed.
## URL parameters (playground)
| Param | Range | Default | Effect |
|-------|-------|---------|--------|
| `tame` | 0–1 | 1 | Constrains synth output ranges toward safe limits. |
| `spread` | 0–1 | 0.6 | Master noise regime (init scale, RL noise cap, per-layer Xavier scaling, weight decay). |
| `preset` | preset id | _(none)_ | Auto-loads a synth preset on first visit. |
| `debug` | 1 | _(off)_ | Exposes `window.__nisps` debug probe. |
### `spread` — sigmoid saturation control
The MLP uses ReLU hidden layers with a sigmoid output. With uniform [-1,1] weights, the sum of many weighted inputs at each layer drives sigmoid pre-activations far from zero (std dev ≈ √fan_in), causing outputs to saturate. The `spread` parameter addresses this:
- `spread=0` (polarised): uniform [-1,1] weights, RL noise cap 0.3, no decay. Outputs cluster at extremes — good for radical exploration.
- `spread=1` (centered): Xavier-scaled weights, RL noise cap 0.05, 10% weight decay per move. Outputs spread across [0,1] — better for fine-grained shaping.
- Intermediate values interpolate.
## Verification chokepoints (user-confirmed)
- **A. Hardware**: each firmware mode flashes and produces correct audio on RP2350.
- **B. RP2350 perf**: no regression vs current main.
- **C. Browser parity**: each firmware mode runs in browser via WASM, sounds equivalent.
- **D. a-immersive feature parity**: control surface, snapshots, A/B compare, region/param pins, heatmap, weight health, gradient flow, output pipeline, session presets.
- **E. CI green**: `bash scripts/run-all-tests.sh` (cmake build + ctest + WASM build + parity + lint + Playwright).
## Build system summary
```bash
# Initialize submodules (required for memllib + daisysp)
git submodule update --init --recursive
# Codegen (run after editing any schemas/modes/*.json)
cd codegen && bun install && bun run generate.ts
# C++ host tests
bash scripts/build-cpp-tests.sh
# WASM
bash scripts/build-wasm.sh
# Cross-platform parity
bash scripts/parity-check.sh
# Lint
bash scripts/lint-cpp.sh
# Firmware
scripts/build-firmware.sh PAFSynth # or any other variant
scripts/flash-firmware.sh
scripts/build-and-flash-firmware.sh
# Playground
cd playground && bun install
bun run dev # Vite dev (COOP/COEP enabled)
bun run typecheck
bun run build
bunx playwright test
# All tests
bash scripts/run-all-tests.sh
```
## Issue tracking
Coding-work tasks go in **ergo** (`ergo ready | show | claim | done | block`), over the Holon EAV core — see the `ergo` skill. **bd (beads) is RETIRED** (migrated 2026-06-15; frozen read-only). Do NOT use `bd`, TodoWrite, or markdown TODO lists.

1
CLAUDE.md Symbolic link
View file

@ -0,0 +1 @@
AGENTS.md

209
MAP.md
View file

@ -1,68 +1,70 @@
# MAP
MEMLNaut-NISPS — Neural Interactive Shaping of Parameter Spaces. One C++20 codebase (`nisps/`) compiles to two targets: (1) Arduino/RP2350 firmware for the MEMLNaut hardware, (2) WASM in a SolidJS browser playground that runs the same engines + ML through an AudioWorklet. Browser audio engines are a superset of firmware engines (C15 is browser-only). See `CLAUDE.md` for the long-form architecture narrative and `ALIGNMENT.md` for current strategic gaps.
MEMLNaut-NISPS — Neural Interactive Shaping of Parameter Spaces. One C++20 codebase (`nisps/`) compiles to two targets: (1) Arduino/RP2350 firmware for the MEMLNaut hardware, (2) WASM in the Manifold React browser app that runs the same engines + ML through an AudioWorklet. (The former SolidJS playground was retired 2026-07-13 — branch `archive/playground-solidjs`, tag `playground-solidjs-final`; the browser-only C15 engine lives only there for now.) See `CLAUDE.md` for the long-form architecture narrative and `ALIGNMENT.md` for current strategic gaps.
## Layout
### `nisps/` — platform-agnostic C++20 library (the only ML/DSP/engine code)
- `nisps/core/` — `perf.hpp` (memory section attrs), `types.hpp`, `concepts.hpp` (`MLEngine`, `AudioEngine`, `Mode`), `fixed_buffer.hpp`, `ring_buffer.hpp` (SPSC lock-free, replaces pico/util/queue), `rng.hpp` (xoshiro256+ deterministic), `math.hpp` (fast_sigmoid, `Curve` enum + `apply_curve`).
- `nisps/ml/` — MLP class template `MLP<NIn, NH1, NH2, NH3, NOut>`. Files: `mlp.hpp`, `activations.hpp`, `loss.hpp` (MSE, no double-scaling), `training.hpp` (SGD + grad clipping), `init.hpp` (spread-aware uniform↔Xavier), `rl.hpp` (`move_weights` with output pin mask + per-layer scaling + weight decay), `jolt.hpp` (`Jolt` — held continuous weight-morph over the flat weight buffer + post-release LR ramp; ported from upstream InterfaceRL), `ou_noise.hpp` (`OUNoise<N>` — Ornstein-Uhlenbeck exploration walk on the output vector; ported from upstream InterfaceRL), `feedback.hpp` (`FeedbackController<MLP_T>` — the 3-mode "Down Action" negative-feedback state machine: Avoid / RandomiseOutputs / RandomiseMlp; header-only, zero-heap, own deterministic RNG, exposed via `nisps_ml_feedback_*` C API), `stats.hpp`. Jolt + OU are inert by default and wired into `ModeBase`, so every mode exposes `jolt_press/jolt_release`, `jolt_lr_scale`, and `set_explore_intensity`.
- `nisps/dsp/` — `biquad.hpp`, `delay.hpp`, `reverb.hpp`, `filter.hpp`, `env.hpp`, `osc.hpp`, `pitch_shift.hpp`, `dc_blocker.hpp`. Lean primitives extracted from maximilian; daisysp PitchShifter replaced with custom granular impl.
- `nisps/core/` — `perf.hpp` (hot-path/inlining attrs), `types.hpp`, `concepts.hpp` (`MLEngine`, `AudioEngine`, `Mode`), `ring_buffer.hpp` (SPSC lock-free cross-core channel, replaces pico/util/queue), `event_queue.hpp` (single-threaded in-engine event FIFO — deliberately NOT RingBuffer, which is an atomics-based cross-thread channel), `rng.hpp` (xoshiro256+ deterministic), `math.hpp` (fast_sigmoid, `Curve` enum + `apply_curve`).
- `nisps/ml/` — the MLP core, written once against a storage policy (`mlp.hpp` `MLPCore<Storage>`): `storage.hpp` (`FixedStorage` — template-sized `std::array`, zero heap; `MLP<NIn,NH1,NH2,NH3,NOut>` alias preserves the classic compile-time surface) and `dynamic_storage.hpp` (`DynamicStorage` — runtime dims, single arena alloc at construction; `#error`s on RP2350 builds, sole lint heap-allowlist entry). Fixed↔dynamic bit-parity enforced by `tests/cpp/test_mlp_storage_parity.cpp`. Files: `mlp.hpp`, `activations.hpp`, `loss.hpp` (MSE, no double-scaling), `training.hpp` (**RMSProp** + grad clipping — `rmsprop_step()` is a line-for-line port of upstream memlp `Layer.h:239 ApplyAccumulatedGradients` @ `ea777502`; the per-weight running squared-gradient average is optimiser state held in the storage policies, NOT part of `weight_count()`/`get_weights()`, and `MLPCore::reset_optimizer_state()` clears it. `draw_weights()` deliberately does not, matching upstream `DrawWeights`), `init.hpp` (spread-aware uniform↔Xavier), `rl.hpp` (`move_weights` with output pin mask + per-layer scaling + weight decay), `jolt.hpp` (`Jolt` — held continuous weight-morph over the flat weight buffer + post-release LR ramp; ported from upstream InterfaceRL), `ou_noise.hpp` (`OUNoise<N>` — Ornstein-Uhlenbeck exploration walk on the output vector; ported from upstream InterfaceRL), `feedback.hpp` (`FeedbackControllerCore<FbStorage>` — the "Down Action" state machine: Avoid (geometric push-away default / Diffuse legacy) / RandomiseOutputs / RandomiseMlp / ExploreAndPlace; geometric replay is advanced deterministically by caller-supplied elapsed time with configurable rate/lifetime; storage-policied like the MLP, own deterministic RNG, exposed via `nisps_ml_feedback_*` C API), `replay.hpp` (`ReplayView` — reward-tagged memory: dedup/deepen, k-NN positive centroid with deterministic tie-break, wall-clock age/lifetime + eviction), `geo_push.hpp` (push-away target computation, upstream InterfaceRL @ `e291192` — re-based from `0a541cc` on 2026-07-25: `kGeometricPushScale` 1.0, `kNegLRBase` 1.5, and NO `/(1+len)` taper, so distance from the liked centroid does not shrink a "no". Cold start is the same push in a random direction, not a separate branch), `warm_start.hpp` (overlapping-weights copy for reshape), `stats.hpp`. `generated/ml_defaults.hpp` is codegen output (do not edit): `nisps::ml::generated::kMlTrainDefaults`, the ONE learning-rate / max-iterations / min-error default shared by firmware, WASM and VCV (source `schemas/ml_defaults.json`); `MLPCore::set_train_config()` and `nisps_ml_set_train_config()` override it at runtime. It lives under `ml/` rather than `modes/generated/` because `nisps/ml` sits below `nisps/modes` — mlp.hpp must not include upward. Jolt + OU are inert by default and wired into `ModeBase`, so every mode exposes `jolt_press/jolt_release`, `jolt_lr_scale`, and `set_explore_intensity`.
- `nisps/pipeline/` — the control-rate input/output processing chains (P4): `input_chain.hpp` (`InputChain` — invert→deadzone→circular clamp→momentum-modulated zoom→centred power→EMA→momentum; caller-supplied dt, internal clock, fixed velocity ring, serialisable state) and `output_chain.hpp` (`OutputChain<NMax>` — curve→EMA→slew→freeze(+mask), capacity-templated). Behaviour contract = the retired manifold TS pipelines, pinned by `manifold/tests/fixtures/` and parity stage 7.
- `nisps/dsp/` — `biquad.hpp`, `delay.hpp`, `reverb.hpp`, `filter.hpp`, `env.hpp`, `osc.hpp`, `pitch_shift.hpp`, `dc_blocker.hpp`, plus the sequencer primitives shared by the sequencer engines: `ratio_seq.hpp` and `seq_clock.hpp` (bar phasor + MIDI clock + bpm). Lean primitives extracted from maximilian; daisysp PitchShifter replaced with custom granular impl.
- `nisps/engines/` — eight audio engines, each satisfying `AudioEngine`: `paf_synth.hpp`, `channel_strip.hpp`, `xiasri.hpp`, `verb_fx.hpp`, `memlcelium.hpp`, `breakor.hpp` (sequencer, NoOp audio), `elysiamorf.hpp` (sequencer, NoOp audio), `analysis.hpp` (input-side spectral features). Plus `base.hpp` (`NoOpEngine`, engine_id "thru").
- `nisps/modes/` — platform-agnostic modes binding `{ML config, engine, voice space lambdas, abstract I/O channels}`. Files: `paf_synth.hpp`, `channel_strip.hpp`, `xiasri.hpp`, `verb_fx.hpp`, `memlcelium.hpp`, `slp_workshop.hpp` (`SLPWorkshopMode` — the Synth Library Portland workshop build; reuses the MEMLCelium engine + MLP shape, foregrounds the Jolt + OU explore gestures), `breakor.hpp`, `elysiamorf.hpp`, `sound_analysis_midi.hpp`, `external_synth_midi.hpp` (`ExternalSynthMIDIMode<const MidiDevice&, NOut>` — joystick→MLP→MIDI CC for an external synth; compile-time device from `nisps/midi`; `consteval pick_cc_slots` curates which params fill the NOut slots; NoOpEngine, `kRouteOutputsToEngine=false`). `base.hpp` provides a CRTP scaffold eliminating the duplication that previously plagued firmware modes. `voice_space.hpp` holds engine-side voice space dispatch helpers. `generated/` contains codegen output (do not edit by hand).
- `nisps/modes/` — platform-agnostic modes binding `{ML config, engine, voice space lambdas, abstract I/O channels}`. Files: `paf_synth.hpp`, `channel_strip.hpp`, `xiasri.hpp`, `verb_fx.hpp`, `memlcelium.hpp`, `slp_workshop.hpp` (`SLPWorkshopMode` — the Synth Library Portland workshop build; reuses the MEMLCelium engine + MLP shape, foregrounds the Jolt + OU explore gestures), `breakor.hpp`, `elysiamorf.hpp`, `sound_analysis_midi.hpp`, `external_synth_midi.hpp` (`ExternalSynthMIDIMode<const MidiDevice&, NOut>` — joystick→MLP→MIDI CC for an external synth; compile-time device from `nisps/midi`; `consteval pick_cc_slots` curates which params fill the NOut slots; NoOpEngine, `kRouteOutputsToEngine=false`). `base.hpp` provides a CRTP scaffold eliminating the duplication that previously plagued firmware modes; it also owns `driver_config()` — the audio-driver setup (mic vs line, gain staging, sample rate) the platform glue reads at mode start. Defaults to `engine().driver_config()`; a mode overrides it with an optional `on_driver_config()` hook only when its engine is not what consumes the audio input (`sound_analysis_midi`, whose analyser rather than its NoOp engine owns the mic). `generated/` contains codegen output (do not edit by hand): per-mode `k<Mode>Schema` ParamSchema instances, the `<Mode>MLP` type aliases built from the schema's own dims, and `schema_types.hpp` which now owns the `ParamSchema` struct itself. Mode headers no longer hand-write either their schema aggregate or their net shape.
- `nisps/wasm/bindings.cpp` — flat C API exported to WASM (Emscripten target only).
- `nisps/midi/generated/midi_devices.hpp` — codegen output: no-heap `constexpr` external-MIDI-synth templates (`nisps::midi::generated`; `MidiParam`/`MidiDevice` + `kMidiDevices` registry). Source = `schemas/midi_devices/`; do not edit by hand.
- `nisps/CMakeLists.txt` + `nisps/build/` — host-target builds + ctest.
### `firmware/` — Arduino sketch + hardware glue
- `firmware/MEMLNaut-NISPS/MEMLNaut-NISPS.ino` — entry point. Selects active mode at compile time via `#define MEMLNAUT_MODE_TYPE`. Forks on `NISPS_SELFTEST`: normal modes run the engine/ML path; the `SelfTest` variant delegates all four entry points to `glue/selftest.hpp`.
### `firmware/` — PlatformIO project + hardware glue
- `firmware/MEMLNaut-NISPS/platformio.ini` — **the variant registry**: one `[env:<alias>]` per firmware variant (16 of them), each passing `-DMEMLNAUT_MODE_TYPE=<alias>`; `selftest` passes `-DNISPS_SELFTEST=1` instead. There is no second list to keep in sync. Shared `[env]` base pins the platform wrapper + arduino-pico framework, sets `-std=gnu++20 -O3` (via `build_unflags`, because the framework appends its own `-std=gnu++17 -Os` AFTER project flags), reaches `nisps/` with `-I${PROJECT_DIR}/../..`, and carries the TFT_eSPI panel config as `-D` flags. Build: `pio run -e <alias>`, or `scripts/build-firmware.sh [--all]`.
- `firmware/MEMLNaut-NISPS/src/main.cpp` — entry point (was `MEMLNaut-NISPS.ino`). Forks on `NISPS_SELFTEST`: normal modes run the engine/ML path; the `SelfTest` variant delegates all four entry points to `glue/selftest.hpp`.
- `firmware/MEMLNaut-NISPS/glue/` — hardware bindings:
- `audio_driver.hpp` — bridges memllib `AudioDriver` callback → `Mode::process(stereosample_t)`.
- `peripherals.hpp` — joystick / pots / buttons → `Mode::set_input` and ML primitives. Also wires the shared adaptive-learning gestures: **TogB1** = Jolt (held weight morph), **RVX1** = exploration amount (OU output walk).
- `audio_driver.hpp` — bridges memllib `AudioDriver` callback → `Mode::process(stereosample_t)`, and brings the codec up on the **active mode's** `driver_config()` (`setup_audio_driver`) plus publishes its preferred sample rate before the system clock is derived from it (`apply_mode_sample_rate`). Mic vs line input is therefore a mode-level declaration, not a firmware constant.
- `codec_config.hpp` — pure, Arduino-free clamping of a `nisps::DriverConfig` to SGTL5000-representable values + sample-rate resolution (unsupported/"don't care" → 48 kHz, because `AudioDriver::GetSysClockSpeed()` `panic()`s otherwise). Host-tested by `tests/cpp/test_mode_driver_config.cpp`.
- `peripherals.hpp` — joystick / pots / buttons → `Mode::set_input` and ML primitives. Wires the shared `FeedbackController` ExploreAndPlace lifecycle (MomA1 = enter/exit explore, MomA2 = freeze/place, TogB2 = commit; MomB1/MomB2 = reroll/nudge while exploring **or** grab/drop *reposition* while idle) plus the adaptive-learning gestures: **TogB1** = Jolt (held weight morph), **RVX1** = exploration amount (OU output walk). Reposition relocates an existing positive example's output to a new input position (`feedback.hpp` `begin_reposition`/`commit_reposition`) — no scratchpad, no weight restore.
- `midi_io.hpp` — MIDI in → mode `note_on`/`update_bpm`/`set_playing`; drains `ControlEvent` ring → MIDI UART.
- `mode_select.hpp` — type aliases mapping firmware mode identifiers to `nisps::modes::*Mode` C++ types. Build script rewrites the active line. Includes the six `MEMLNautModeExtSynth*` external-synth variants (one per device template in `nisps/midi`, e.g. `MEMLNautModeExtSynthSub37`). Also defines the `MEMLNautModeSelfTest` pseudo-variant (tag type) + the `NISPS_ST_*`/`NISPS_ST_CAT` token-paste macros the `.ino` uses to compute `NISPS_SELFTEST`. Note: `src/nisps/` exposes each referenced top-level nisps subdir as a symlink — `midi` was added alongside `core/dsp/engines/ml/modes`.
- `mode_select.hpp` — type aliases mapping firmware mode identifiers to `nisps::modes::*Mode` C++ types, selected by the `-D` from platformio.ini. Includes the six `MEMLNautModeExtSynth*` external-synth variants (one per device template in `nisps/midi`, e.g. `MEMLNautModeExtSynthSub37`). The `NISPS_ST_*`/`NISPS_ST_CAT` token-paste table and the `SelfTestRig` tag type are GONE — selftest is now just an env with its own `-D`.
- `selftest.hpp` — standalone guided hardware self-test rig (`SelfTest` variant; no engine/ML). Step-driven state machine on a `SelfTestView`: TFT prompts the operator through every control, auto-advances on detection, encoder-press skips. Ends with optional L/R/BOTH sine-sweep headphone check (core 1 block callback) + MIDI loopback-cable test. Lives firmware-side (touches TFT + raw pins) so it stays out of platform-agnostic `nisps/`.
- `input_router.hpp`, `output_router.hpp` — top-level `wire_inputs()` / `drain_outputs()` entry points.
- `output_router.hpp` — top-level `drain_outputs()` entry point. (Inputs are wired directly by `peripherals.hpp`'s `bind_peripherals()`.)
- `settings_view.hpp` — `wire_settings(mode)`: adds on-device settings views to the MEMLNaut display carousel (TFT + rotary encoder). Joystick Dual/Single toggle for the 4-input ("two 2-D joystick") modes — "Single" pins ML input channels 2,3 to neutral via `ModeBase::set_input_pinned` (no net rebuild). Registered in the `.ino` after `addSystemInfoView()`.
- `firmware/MEMLNaut-NISPS/src/{memllib,daisysp,nisps}` — symlinks (Arduino-CLI requires sketch-tree includes; preprocessor refuses `..` in headers).
- `firmware/MEMLNaut-NISPS/lib/memllib/` — **vendored** memllib (was the `src/memllib` submodule): hardware abstraction (audio driver, TFT display, MIDI, peripherals), ~1.9 MB / 100 files, `examples/` dropped **except** `reference/InterfaceRL.{hpp,cpp,tpp}` + `InterfaceRLFileFormat.hpp` — upstream's reference implementation of the feedback subsystem `nisps/ml/{geo_push,replay,feedback,jolt,ou_noise}.hpp` were ported from, kept verbatim and NEVER compiled (`reference/` sits outside `src/`, which is the only thing PlatformIO builds). It is there so upstream drift is a `diff`; losing it is how the e291192 geometric-dislike redesign went unnoticed for months. `VENDORED.md` records the upstream commit and the re-sync procedure; `LICENSE` is MPL-2.0, copied verbatim. **Sources must sit under `lib/memllib/src/`** — PlatformIO's library builder falls back to a flat root-only scan without it and silently compiles nothing while still linking (see VENDORED.md).
- `firmware/README.md` — structure + build instructions.
- `firmware/useq-celium/` — standalone RP2040 firmware (PlatformIO, Arduino-Pico core) that turns a uSEQ module + CV expander into a USB→CV/gate converter driven by the manifold `cvgate` backend. `shared/protocol.h` is the v2 wire-protocol single source of truth (mirrored by `manifold/src/backends/useq-protocol.ts`); `main/` (USB serial → CV1–3 + GATE1–3, I2C → expander) and `expander/` (I2C slave → CV4–11). Wire spec: `docs/useq-celium/protocol.md`. Restored from the April-2026 "uSEQ-Celium" mode.
- `firmware/useq-celium/` — standalone RP2040 firmware (PlatformIO, Arduino-Pico core) that turns a uSEQ module + CV expander into a USB→CV/gate converter driven by the manifold `cvgate` backend. `shared/protocol.h` is the v2 wire-protocol single source of truth (mirrored by `manifold/src/backends/useq-protocol.ts`); `main/` (USB serial → CV1–3 + GATE1–3, I2C → expander) and `expander/` (I2C slave → CV4–11). Wire spec: `docs/specs/useq-cv-protocol.md`. Restored from the April-2026 "uSEQ-Celium" mode.
### `playground/` — SolidJS + Vite + TypeScript app
- `playground/index.html`, `vite.config.ts`, `tsconfig.json`, `package.json` — scaffold. COOP/COEP headers configured.
- `playground/src/main.tsx`, `App.tsx` — entry + router (`/`, `/dev/primitives`, `/modes`).
- `playground/src/primitives/` — 16 UI building blocks: `Slider`, `SliderBank`, `VirtualJoystick`, `XYPad`, `Heatmap`, `OutputDisplay`, `TrainingControls`, `Drawer`, `ControlAxis`, `ProgressRing`, `PillToggle`, `ParamEditor`, `JoyMap`, `WeightHealth`, `GradientFlow`, `LossPlot`. Each has a `.demo.tsx` showcased on `/dev/primitives`.
- `playground/src/modes/` — one TSX per firmware mode (+ `C15Mode` browser-only). `ModeShell.tsx` is the shared scaffold; `ModeSwitcher.tsx` picks the active mode; `mode-runtime.ts` is the schema → ML → audio wiring hook; `mode-helpers.ts` for SliderBank configs. `generated/` holds codegen-produced TS schemas (do not edit).
- `playground/src/stores/` — Solid stores: `ml-store`, `input-store`, `output-store`, `mode-store`, `control-store` (compound axes Boldness/Memory/Precision), `session-store` (snapshots, A/B, presets), `exploration-store`, `bus` (typed signal bus). `persistence.ts` debounces localStorage writes.
- `playground/src/audio/engine-host.ts`, `worklet/nisps-processor.ts` — main-thread engine host + AudioWorklet processor. WASM loaded twice (main thread for ML, worklet for engines).
- `playground/src/ml/wasm-iml.ts`, `wasm-worker.ts`, `dataset.ts`, `types.ts` — main-thread WasmIML class + disposable async-training worker + FIFO dataset.
- `playground/src/input/pipeline.ts`, `playground/src/output/pipeline.ts`, `playground/src/output/curves.ts` — pure-fn pipelines (deadzone→zoom→curve→smoothing→momentum, then global curve→smoothing→slew→freeze).
- `playground/src/features/` — additional feature modules (heatmap sampling, snapshot stack, A/B compare, region pin, param pin, trail, weight health, etc.).
- `playground/src/debug/probe.ts` — synchronous `window.__nisps` debug probe for Playwright.
- `playground/public/nisps.{wasm,js}`, `c15.wasm`, `c15-glue.js` — compiled WASM artifacts (built by `scripts/build-wasm.sh`).
- `playground/tests/e2e/` — Playwright specs (`ml-engine`, `modes`, `persistence`, `ui-interactions`) + `helpers.ts`.
- `playground/playwright.config.ts` — Vite preview server setup.
### `manifold/` — Vite + React + TS convertible-mode app (the NEW front-end, WIP)
### `manifold/` — Vite + React + TS convertible-mode app (the sole browser app)
The Manifold "convertible" Console on the real engine, deployed at `meml.lnfinitemonkeys.org/next` (staging,
alongside the live vanilla a-immersive at `/`). Built 2026-06-27/28; see `docs/redesign/BUILD-PLAN.md` (resume
anchor + locked decisions) and the `docs/redesign/*-spec.md` set.
- `manifold/src/engine/` — the parity-tested TS engine LIFTED from `playground/src` (same `nisps.wasm`), made
alongside the live vanilla a-immersive at `/`). Built 2026-06-27/28; see `docs/specs/plans/BUILD-PLAN.md` (resume
anchor + locked decisions) and the `docs/specs/*-spec.md` set.
- `manifold/src/engine/` — the parity-tested TS engine (same `nisps.wasm`), made
framework-neutral: `wasm-iml.ts` (rewired off Solid stores onto an injected `EngineSink`), `engine-host.ts` +
`worklet/nisps-processor.ts` (audio), `input-pipeline.ts`/`output-pipeline.ts`/`curves.ts`, `wasm-worker.ts`,
`spine.ts` (the reactive spine BELOW React — `setInput` derives processed→ml→routed eagerly off-render),
`engine-api.ts` (`EngineApi` façade incl. `feedback.*` wrappers over the `nisps_ml_feedback_*` C ABI),
`EngineProvider.tsx`/`useEngine.ts` (React binding via `useSyncExternalStore` version counter). nisps.js is
`worklet/nisps-processor.ts` (audio), thin WASM wrappers over the core pipelines + curve catalog (the TS
`input-pipeline`/`output-pipeline`/`curves` implementations died at P4), `wasm-worker.ts`,
`spine.ts` (the reactive spine BELOW React — `setInput` derives processed→ml→routed eagerly off-render;
structural/training state and throttled live-output notifications are separate channels),
`engine-api.ts` (`EngineApi` façade incl. live architecture/weight/example metrics and `feedback.*` wrappers over the `nisps_ml_feedback_*` C ABI),
`EngineProvider.tsx`/`useEngine.ts` (React bindings via `useSyncExternalStore`: state version plus an
opt-in 30 Hz output version for DOM consumers; canvases read live buffers directly). nisps.js is
loaded via fetch+indirect-eval (Emscripten MODULARIZE glue has no ES exports), base-aware via `document.baseURI`
for the `/next` sub-path.
- `manifold/src/primitives/` — the 12 design primitives as typed React.
- `manifold/src/primitives/` — the 7 design primitives as typed React (Badge, Button, PillToggle, Slider, Switch, VirtualJoystick, XYPad). Five unused ones were deleted in the 2026-07 sweep (L22).
- `manifold/src/console/` — the convertible Console: `ConsoleApp`, `CompositeStage` (single-divider convertible
with snap/magnetism/minimap-demotion), `SplitStage`/`OutputStage`/`InputMini`/`Manifold` (canvas, rect↔circular
+ feedback markers), `Dock` (top Mode selector + 5 vertically-centred drawers), `Drawers` (Learning/Inputs/
Outputs/Settings/Help), `VerdictCluster` (mode-aware), `ReadoutStrip`, `OutputEditor`/`CurvePad`, `icons.tsx`
(monochrome currentColor SVG), `model.ts`, `output-mode.ts`.
- `manifold/src/dock/` — `OutputControlRow` (off/fixed/live + mute + solo/arm + min/max/curve), `output-state.ts`,
`OutputsBackendConfig.tsx` (per-backend specialised Outputs panel), `BackendAdvanced.tsx`.
with snap/magnetism/minimap-demotion), `OutputStage`/`SandwichStage`/`ParticleStage`/`Manifold` (canvas,
rect↔circular + feedback markers; ParticleStage batches its 400 particles into 32 colour paths per frame,
has interactive cursor-labelled heatmap sliders, an adjustable circular Manifold pad with feedback markers and cursor trail,
and double-click whole-screen follow-mouse input), `Dock` (top Mode selector + 5 vertically-centred drawers), `Drawers`
(Learning/Inputs/Outputs/Settings/Help; Learning includes the live model-architecture inspector and
Outputs owns the remaining scroll height), `TrainingHealth` (real per-iteration loss curve from
`nisps_ml_loss_history` + per-layer weight health from `nisps_ml_get_layer_stats`; rendered only at
the Learning drawer's `expanded` depth — that IS the advanced-surface flag), `VerdictCluster`
(mode-aware), `OutputEditor`/`DualRange`/`CurvePad`, `icons.tsx`
(monochrome currentColor SVG), `model.ts` (`MF_MODES` catalogue — schema-backed modes DERIVED from
`manifold/src/modes/generated/`; carries per-mode `ml` net shape + `engineId`), `output-mode.ts`.
- `manifold/src/modes/generated/` — codegen output (`*_schema.ts`, do NOT hand-edit): `ModeSchema`
consts (mode_id, engine_id, ml dims, params, voice_spaces, ui) — the SOURCE OF TRUTH for `MF_MODES`.
Switching mode reshapes the WASM net to the mode's `ml` dims (ConsoleApp P5.3; boot mode paf_synth →
4→[10,10,14]→33).
- `manifold/src/dock/` — `OutputControlRow` (editable names, corner delete, cycling off/fixed/live status, mute + solo/arm + min/max/curve, plus MIDI card fields), `output-state.ts`,
`OutputsBackendConfig.tsx` (per-backend specialised Outputs panel — the sole per-backend editor; centered prepend/append add-card controls live above and below the rows).
- `manifold/src/backends/` — `OutputBackend` adapter + `BackendManager` (spine consumer); `midi-backend.ts`
(WebMIDI), `osc-backend.ts`+`osc-client.ts` (OSC-over-WS), `vcv-backend.ts` (VCV-over-WS), `cv-backend.ts`
(`UseqCvBackend` — uSEQ CV/gate over USB Web Serial, backend id `cvgate`) + `useq-protocol.ts` (v2 wire
@ -70,98 +72,126 @@ anchor + locked decisions) and the `docs/redesign/*-spec.md` set.
`particle-backend.ts`, `passthrough-backend.ts`, `presets.ts` (named presets), `manager.ts`.
- `manifold/src/midi-devices/` — external-synth device templates. `generated/` is codegen output from
`schemas/midi_devices/` (`MIDI_DEVICES` catalogue + `MIDI_DEVICES_BY_ID`, params by name+CC). The MIDI Outputs
config (`dock/OutputsBackendConfig.tsx`) reads it for the device picker + param-select that fills the CC table.
config (`dock/OutputsBackendConfig.tsx`) reads it for the device picker + param-select that fills the per-card MIDI fields.
- `manifold/src/inputs/` — modular INPUT layer feeding the ML head. The Inputs dock picks ONE exclusive mode
(`InputMode` = `internal` | `gamepad` | `midi`; Internal/XY-pad is default). `input-layer.ts` owns a single rAF
loop composing the active source's axes → reduced to the engine arity (fixed 2-in WASM → even/odd blend) → one
`setInputs`, plus an `onReducedInput` callback the manifold tracks. Sources: `xy-pad-source` (push-driven),
loop composing the active source's axes → **one dedicated engine input slot per axis, 1:1, no blending** → one
`setInputs`, plus an `onReducedInput` callback the manifold tracks. Normal Manifold models default to a
runtime-shaped 2-input head; the Inputs dock can select 4 inputs, and active sources can grow the head when
they need more slots. Mean-blending was removed deliberately — it diluted every source and biased the net
toward idle sources' resting values. Active-axis edits follow the persistent I/O policy:
keep-capacity permutes stable identities in place until more slots are required; exact-I/O reconstructs
to the active count. Surviving weights and (under adapt policy) examples are identity-remapped; feedback
scratch state resets. Sources: `xy-pad-source` (push-driven),
`gamepad-source` (sticks→axes single/double; buttons emit press+release actions, bound in `ConsoleApp` to
verdicts — LB/RB=down/up, X/Y/B=randomise/nudge/undo, A-hold=reposition), `midi-input-source` (device picker +
BATCH "MIDI Learn": every CC swept while armed becomes an axis, shown as read-only meters). `useInputLayer.ts`
is the React binding; `base-source.ts` shared status/action plumbing; `types.ts` the adapter contract.
- `manifold/src/feedback/` — `controller.ts` (Explore-and-place scratchpad + geometric-dislike + solo, TS
prototype), `rng.ts` (seeded).
- `manifold/src/settings/` — `settings-store.ts` (monochrome icons, input-map shape, corner radius).
`backends/base-backend.ts` is its output-side counterpart (status + throttle + lastSent) used by the midi/osc/vcv transports.
- `manifold/src/feedback/` — `controller.ts` (Explore-and-place scratchpad + geometric-dislike + solo; owns the
elapsed-time timer for geometric replay while every weight-affecting step stays in the shared C++ core).
- `manifold/src/settings/` — `settings-store.ts` (monochrome icons, input-map shape, I/O resize policy, corner radius, and the opt-in legacy Xavier/spread feature flag; Manifold randomisation is full-range uniform by default).
- `manifold/src/serial/` — `memlnaut-serial.ts` Web Serial scaffold + `EditorPanel.tsx` (MEMLNaut Editor mode).
- `manifold/src/debug/probe.ts` — `window.__nisps` (`?debug=1`). `manifold/tests/e2e/smoke.spec.ts`. E2E on the
VPS runs via non-snap node (see BUILD-PLAN). `manifold/osc-bridge/` — Deno WS↔UDP-OSC bridge.
- `manifold/src/engine/exploration.ts` — Jolt press + OU explore gestures (Learning drawer): a thin
timer-driver over the shared C++ core via the `nisps_ml_jolt_*`/`nisps_ml_ou_*` bindings (the interim
TS math and `jolt.ts`/`ou-explore.ts` were deleted when P3 landed).
- `manifold/src/engine/io-reshape.ts` — the deep identity-migration module for I/O card edits:
exact-vs-capacity reconstruction decisions, flat weight remapping, and example vector adaptation.
- `manifold/src/debug/probe.ts` — `window.__nisps` (`?debug=1`). `manifold/tests/e2e/` — `smoke`,
`probe-api` (engine-contract port), `spine` (spine invariant + probe-survives-mode-switch),
`geo-dislike`, `reshape`, `schema-modes`, `training-health` (the loss/layer-stats panel + its
expanded-depth gating). E2E on the VPS runs via non-snap node (see BUILD-PLAN).
`manifold/tests/fixtures/` — golden parity
fixtures (gesture trace, curves, input/output pipelines) captured 2026-07-13 pre-P4, guarded by
`tests/pipeline-golden.test.ts` (in `bun run test`). `tests/loss-history.test.ts` drives the
`nisps_ml_loss_history` C ABI straight at the committed WASM — the training path parity-check
never touches. `manifold/osc-bridge/` — Deno WS↔UDP-OSC bridge.
### `vcv/` — VCV Rack 2 plugin (MEMLNaut module, WIP)
Native C++ Rack module: ML CV-mapper with RL feedback + a browser bridge. Currently 2→12 (being evolved to
**8 inputs × 16 outputs + per-output LED rings**, palette from the frontend tokens, WS↔OSC browser bridge — see
the "BUILD DELTAS" block at the top of `vcv/SPEC.md`). `src/MEMLNaut.cpp` (module), `src/osc_server.hpp` (bridge),
`src/plugin.{hpp,cpp}`, `res/*.svg` (panels), `Makefile` (needs `RACK_DIR`). Was built against the retired
`nisps-core`; the core include path is being repointed.
Native C++ Rack module: ML CV-mapper with RL feedback + a browser bridge. **8 inputs × 16 outputs + per-output
LED rings**, palette from the frontend tokens, WS↔OSC browser bridge (see `docs/specs/vcv-module.md`).
`src/MEMLNaut.cpp` (module, 8→[16,24,16]→16), `src/iml.hpp` (**thin adapter over `nisps::ml::MLPCore<DynamicStorage>`
+ core `nisps::Rng`** — P6 reunification 2026-07-18, closes vcv-module.md delta #5; behaviour is now core-exact,
pinned by `tests/cpp/test_vcv_iml_parity.cpp`), `src/osc_server.hpp` (bridge, transport-only), `src/plugin.{hpp,cpp}`,
`res/*.svg` (panels), `Makefile` (needs `RACK_DIR`). Builds against the current `../nisps/` core via relative
includes; no `nisps-core`.
### `schemas/` — JSON parameter contracts (firmware/browser source of truth)
- `schemas/schema.json` — Draft 2020-12 meta-schema validating mode files.
- `schemas/modes/<mode>.json` (×9) — each mode's params, ranges, defaults, curves, voice spaces, ML config. (`slp_workshop.json` reuses `engine_id: memlcelium`.)
- `schemas/modes/params_notes.md` — provenance notes and judgement calls per mode.
- `schemas/modes/<mode>.json` (×9) — each mode's params, ranges, defaults, curves, voice spaces, ML config. (`slp_workshop.json` reuses `engine_id: memlcelium`.) `params[].curve` is the mode-wide DEFAULT response curve; a `voice_spaces` entry may be an object `{name, curve_overrides}` declaring only the slots where THAT voice space deviates (index i == `VoiceSpace` ordinal i). Descriptive throughout: the curve is applied exactly once, inside the engine's voice space.
- `schemas/modes/params_notes.md` — provenance notes and judgement calls per mode, plus the exact `square`/`sqrt`/`linear` predicate the drift check enforces.
- `schemas/midi_device.schema.json` — Draft 2020-12 meta-schema for external-MIDI-synth templates.
- `schemas/midi_devices/<device>.json` (×6) — CC-controllable external synths (Moog Sub 37 / Sub Phatty, Creamware Pro-12 ASB, Elektron Analog Keys, ASM Hydrasynth, Roland JD-800). Each param: `{id, cc, label, min, max, default, group}`. Canonical source for both firmware + browser device pickers. Verified-CC provenance + sources live in `synth-midi-cc.json` (repo root).
- `schemas/midi_devices/<device>.json` (×6) — CC-controllable external synths (Moog Sub 37 / Sub Phatty, Creamware Pro-12 ASB, Elektron Analog Keys, ASM Hydrasynth, Roland JD-800). Each param: `{id, cc, label, min, max, default, group}`. Canonical source for both firmware + browser device pickers. Verified-CC provenance + sources live in `schemas/midi_devices/sources/synth-midi-cc.json` (a `sources/` subdir, because the generator ajv-validates every `*.json` directly under `midi_devices/` as a device template).
### `codegen/` — schema → C++/TS code
- `codegen/generate.ts` — Bun script: validates schemas via ajv, emits per-mode `nisps/modes/generated/<mode>_schema.hpp` (`constexpr`, `nisps::modes::generated`) and `playground/src/modes/generated/<mode>_schema.ts`. Idempotent.
- `codegen/generate.ts` — Bun script: validates schemas via ajv (incl. the P5 firmware-fit check: exactly 3 hidden layers, dims ≤4096), emits per-mode C++ `nisps/modes/generated/<mode>_schema.hpp` (`constexpr`, `nisps::modes::generated`) AND TS `manifold/src/modes/generated/<mode>_schema.ts` (+ `types.ts`, `index.ts`). Idempotent; golden-tested in `run-all-tests.sh` stage 5. The TS output is the SOURCE OF TRUTH consumed by `MF_MODES` (`manifold/src/console/model.ts`).
- `codegen/generate-midi-devices.ts` — separate Bun script (isolated from the mode golden test): validates `schemas/midi_devices/` via ajv, emits `nisps/midi/generated/midi_devices.hpp` (no-heap `constexpr`) and `manifold/src/midi-devices/generated/{types,devices,index}.ts`. Idempotent.
- `codegen/seed-midi-devices.ts` — one-time/idempotent seed deriving `schemas/midi_devices/*.json` from the `synth-midi-cc.json` research artifact (slugifies labels → ids, heuristic groups).
- `codegen/templates/`, `codegen/tests/golden/` — reference templates + golden snapshot for paf_synth.
- `codegen/curve-audit.ts` — reads `nisps/engines/*.hpp` and derives, per voice space, which response curve the engine applies to each NN-output slot. Handles the four idioms a regex misses (`const float v = p[n]; v*v`, memlcelium's implicit-counter `sq()` lambda, loop-generated indices, `smooth_params_[n]`) and RAISES on anything it cannot reduce rather than defaulting to `linear`.
- `codegen/tests/curve_drift_test.ts` — the gate: schema-declared curves (JSON **and** generated TS) must equal what `curve-audit.ts` extracts, and schema `voice_spaces` order must equal the engine's `kVoiceSpaceNames`. Source-level by necessity — the curve is not recoverable from engine output. Run by `run-all-tests.sh` stage 5, CI, and `bun run test` in `codegen/`.
- `codegen/lib.ts` — helpers shared by both generators. `codegen/tests/golden/` — golden snapshot for paf_synth (C++ + TS).
### `tests/cpp/` — host C++ tests
- Per-component tests: `test_dsp_*.cpp`, `test_engine_*.cpp`, `test_mlp_*.cpp`, `test_mode_*.cpp`, `test_fixed_buffer.cpp`, `test_ring_buffer.cpp`, `test_rng.cpp`, `test_math.cpp`. Helpers in `test_helpers.hpp`.
- Per-component tests: `test_dsp_*.cpp`, `test_engine_*.cpp`, `test_mlp_*.cpp`, `test_mode_*.cpp`, `test_ring_buffer.cpp`, `test_rng.cpp`, `test_math.cpp`. Helpers in `test_helpers.hpp`.
- Verification: `ml_golden_vectors.cpp`, `engine_impulse.cpp` (+ `engine_impulse_baseline.bin`), `parity_check.cpp` + `parity_wasm.mjs` + `parity_diff.mjs` — native-vs-WASM bit-equivalence within 1e-5.
- Behaviour: `test_ml_behaviour.cpp` — 20 INVARIANTS of the interaction model (not tuning). Asserts what must hold however the knobs are set: a like is reachable and a re-like overwrites; dislike never yields non-finite weights/outputs in either `AvoidStyle`, and is safe at cold start; repeat dislikes inside `kReplayDedupRadius` deepen ONE negative while distant ones store separately; explore→reroll→undo→exit restores weights *bit-exactly* and over-undoing is safe; `RandomiseOutputs` leaves weights untouched (the outputs-vs-weights randomisation distinction); same seed + same gesture sequence ⇒ bit-identical weights (what makes any behavioural benchmark comparable); example-ring overflow keeps the newest; contradictory examples stay finite; the ExploreAndPlace accessor contract (`placed_output()` while Placing, `committed_output()` after commit — for both place and reposition); a FULLY-masked focus gate freezes every weight; switching mode mid-exploration never strands the net in a randomised scratchpad; and all of it holds at 1×1, 2×8, 1×33, 8×2 and 32×8 shapes.
- Measurement (asserts nothing): `ml_bench.cpp` — the BEHAVIOURAL benchmark. NISPS is a controller, so this measures the shape of the control→parameter mapping and how interaction journeys deform it, never loss alone. Shape-agnostic (`--shape N_IN,H1,H2,H3,N_OUT`, default `2,16,16,16,8`) via `MLPCore<DynamicStorage>`, so "does a wider/deeper net change the UX?" becomes a number; sample points come from a deterministic Kronecker low-discrepancy sequence rather than a raster, which is what makes it work at any input arity. Field metrics: local gain p50/p95, cliff index, dead fraction, range utilisation, rail occupancy, effective dimensionality (participation ratio — trace²/‖C‖²_F, no eigendecomposition). Displacement metrics: at-point, rings, global, **blast ratio**, and collateral damage at the protected positives. **61 scenarios.** Diagnostic (D1 — the geometric-dislike dose decomposed: intended push vs effective LR vs measured movement at 1/10/100/1000 presses). Atomic probes A1–A14: at/around/far-from an example; one dislike under BOTH candidate designs; twice at one point; adjacent-then-return across the dedup radius; near a protected positive; roll-a-patch-and-place; the full explore→audition→place→commit lifecycle; explore-then-cancel; reposition; like-then-dislike in place; dislike-then-repair; focus/solo mask leakage. Journeys J1–J11: positive-only retention curve, randomise-place-only, mixed, branch (one shared prefix, three divergent gestures, replayed from scratch per branch because that is exact under a deterministic RNG), explore-place-only, 120-gesture long session with drift + weight-norm checkpoints, dislike storm, revisit-after-wandering, two-region interference, sweep-and-teach along a continuous path, undo-heavy. Edge cases E1–E13: cold start, single example, contradictory, collinear, corners, capacity overflow, undo exhaustion, minimal shape, identical targets, rail targets, rapid like/dislike alternation, mode-switch mid-exploration, fully-masked dislike. Upstream comparison U1–U3: the older memllib `interfaceRL` (a DDPG actor-critic, recoverable from this repo's own git history at blob `755ff8b`) differs structurally — the user HEARS `actorTarget`, a soft copy updated `target += alpha*(online-target)` at alpha=0.005, and it trains a batch of 4 from replay only every `optimiseDivisor=40` gestures. The critic half is not reproducible here (MLPCore has `train_targets` but not the per-layer gradient extraction the policy-gradient step needs), but both OUTPUT-PATH ideas are: U1 sweeps the soft-target alpha (alpha=1 IS NISPS today, a free control), U2 sweeps the train-every-Nth divisor, U3 compares actor shapes, U4 sweeps the positive-path training dose. U4's numbers became comparable to upstream's on 2026-07-25, when `nisps/ml/training.hpp` stopped being SGD-only and ported upstream's RMSProp — before that an upstream LR meant something different here than there, which is what made the geometric dislike inert (D1: 5.3e-5 per press before, 1.6e-2 after). Their shared metric is **lurch** — how far the mapping the musician is playing moves per single gesture, averaged over the whole field. Knobs: `--spread` (1 = Xavier, 0 = uniform with NO fan_in coupling — i.e. the post-removal behaviour, measurable before paying for the refactor), `--geo-lr`, `--geo-iters`. Driven by `scripts/bench-ml.sh`. **Two contracts that fail SILENTLY and are pinned by tests:** (1) a thumbs-up must go through BOTH `mlp.add_example` AND `fb.store_positive` — `dislike_geometric` k-NNs the replay buffer, not the MLP dataset, so a harness that only calls `add_example` measures the cold-start branch instead; (2) `placed_output()` is valid ONLY while state is `Placing` — after `commit_place()`/`commit_reposition()` the vector moves to `committed_output()`, and reading the wrong one yields an empty span whose `l2()` is 0, i.e. a broken lifecycle scored as a perfect placement.
- Measurement (asserts nothing): `engine_bench.cpp` + `bench_report.mjs` — per-engine throughput (ns/sample, blocks/s, realtime factor) for the `process()` hot path. ONE source compiled twice (CMake `nisps_engine_bench` natively, emcc for WASM) so the two targets are comparable without adding a single export to `nisps/wasm/bindings.cpp`. Engines are driven into a working state (transport running + event drain for the sequencers, periodic `note_on` for paf_synth, a noise+sine input bed for the fx/analysis engines) and every row prints its own working-state evidence, so a number produced by an idle engine is visible rather than plausible. Driven by `scripts/bench-engines.sh`.
### `scripts/` — build + verify entry points
- `build-firmware.sh`, `flash-firmware.sh`, `build-and-flash-firmware.sh`, `firmware-common.sh` — Arduino-CLI wrapper for RP2350 target with C++20 flag.
- `build-wasm.sh` — Emscripten compile producing `playground/public/nisps.{wasm,js}`.
- `build-wasm.sh` — Emscripten compile producing `manifold/public/nisps.{wasm,js}`.
- `build-cpp-tests.sh` — CMake configure + build + ctest (Ninja).
- `parity-check.sh` — runs native + WASM and diffs binary outputs.
- `parity-check.sh` — runs native + WASM and diffs binary outputs. **Gotcha**: it only builds `manifold/public/nisps.{js,wasm}` when they are MISSING, never when they are stale, so after any change under `nisps/` you must run `build-wasm.sh` yourself or you are diffing fresh native against an old WASM — which reports a parity FAILURE that is really a staleness failure (this is how the RMSProp port first "broke" parity).
- `bench-ml.sh` — the ML BEHAVIOUR benchmark on native + WASM from one source (same trick as `bench-engines.sh`). `--shape`, `--scenario`, `--smoke`, `--seed`, `--compare`, and `--sweep-shape` (runs the corpus across a ladder of architectures and arities — the knob-sensitivity instrument). **Reports, never asserts**: a cliff index is a description, not a pass/fail. Invariants live in `tests/cpp/test_ml_behaviour.cpp` instead. Reports land in `nisps/build/bench-ml/`.
- `bench-engines.sh` — engine throughput on native + WASM; `--compare <report.json>` prints per-engine Δ%. **Reports, never asserts** (a wall-clock threshold on shared hardware is meaningless or flaky — same call as the firmware size job). Reports land in `nisps/build/bench/`.
- `lint-cpp.sh` — `.f` literal warn + heap/`Arduino.h` violation fail.
- `run-all-tests.sh` — master verification script.
### `.github/workflows/`
- `ci.yml` — GitHub Actions: cmake build + ctest + WASM build + parity check + lint + Playwright (cpp-tests + playground-tests jobs). Firmware compile is documented as manual.
- `ci.yml` — GitHub Actions: cmake build + ctest + WASM build + parity check + lint + Playwright (cpp-tests + manifold-tests jobs). Firmware compile is documented as manual.
### Submodules (in `src/`)
- `src/memllib/` — hardware abstraction (audio driver, peripherals, MIDI). **Not auto-initialized** — fresh clones need `git submodule update --init --recursive`.
- `src/daisysp/` — vendored DSP library. Used by some firmware glue; nisps replaced its PitchShifter with a custom granular impl.
### `src/` — submodule + vendored trees
- **There are no submodules.** `src/memllib` was one until the Phase 4 PlatformIO migration; it is now vendored at `firmware/MEMLNaut-NISPS/lib/memllib/`. Fresh clones need no `git submodule` step.
### Top-level docs
- `CLAUDE.md` — long-form architecture narrative.
- `MAP.md` — this file.
- `ALIGNMENT.md` — strategic gaps + open mission questions, dated, opinionated.
- `README.md` — short quickstart.
- `AGENTS.md` — beads/bd conventions.
- `AGENTS.md` — canonical agent contract: architecture, build/test, scope, and Ergo workflow.
## Entry points
- **Firmware**: `scripts/build-firmware.sh [VARIANT]` (interactive prompt if omitted), `scripts/flash-firmware.sh`, `scripts/build-and-flash-firmware.sh`. Target: `rp2040:rp2040:solderparty_rp2350_stamp_xl:opt=Optimize3`, `-std=gnu++20`.
- **Playground dev**: `cd playground && bun install && bun run dev` (Vite, port 5173, COOP/COEP headers).
- **Playground build**: `cd playground && bun run build`.
- **Manifold dev**: `cd manifold && bun install && bun run dev` (Vite, COOP/COEP headers).
- **Manifold build**: `cd manifold && bun run build`.
- **WASM rebuild**: `bash scripts/build-wasm.sh` (needs `emcc`).
- **Host C++ tests**: `bash scripts/build-cpp-tests.sh`.
- **Parity check**: `bash scripts/parity-check.sh`.
- **All tests**: `bash scripts/run-all-tests.sh`.
- **Playwright**: `cd playground && bunx playwright test`.
- **Codegen**: `cd codegen && bun run generate.ts` (regenerates `nisps/modes/generated/` and `playground/src/modes/generated/`).
- **Engine benchmark**: `bash scripts/bench-engines.sh` (add `--compare nisps/build/bench/latest.json` to diff against the previous run).
- **ML behaviour benchmark**: `bash scripts/bench-ml.sh` (`--smoke` for a fast run, `--shape 2,16,16,16,8`, `--sweep-shape` for the architecture/arity sweep, `--compare` to diff).
- **All tests**: `bash scripts/run-all-tests.sh` (stage 6 is a bench smoke report; it does not gate).
- **Playwright**: `cd manifold && node node_modules/.bin/playwright test` (non-snap node runner on the VPS — BUILD-PLAN gotcha; `bunx playwright test` works elsewhere).
- **Codegen**: `cd codegen && bun run generate.ts` (regenerates `nisps/modes/generated/` + `nisps/ml/generated/` C++ and `manifold/src/modes/generated/` TS).
## Conventions
- Firmware mode selection is compile-time only — `#define MEMLNAUT_MODE_TYPE` in the `.ino`.
- `nisps/` follows Chris's RP2350 perf rules globally: no heap, `static const float` for non-trivial constants, strict `.f` suffix, memory section attrs (`NISPS_AUDIO_MEM`, `NISPS_AUDIO_FUNC`, `NISPS_APP_SRAM`, `NISPS_HOT`, `NISPS_FORCE_INLINE`).
- Firmware mode selection is compile-time only — one `-DMEMLNAUT_MODE_TYPE` per `[env:]` in `platformio.ini`.
- `nisps/` follows Chris's RP2350 perf rules: no heap, `static const float` for non-trivial constants, strict `.f` suffix. `perf.hpp` now carries only `NISPS_HOT`/`NISPS_FORCE_INLINE`; the three dead/misshapen SRAM-section macros were deleted in the 2026-07 sweep (S21/L13).
- C++ identifiers: `PascalCase` types, `snake_case` functions/variables, `kPascalCase` constexpr. JSON keys `snake_case`. TS types `PascalCase`, components `PascalCase.tsx`, modules `kebab-case.ts`.
- `Curve` enum lives in `nisps/core/math.hpp` (lowercase: `linear/exp/log/square/sqrt/sigmoid/cubic`); generated mode headers re-export via `using Curve = ::nisps::Curve;`. TS mirror at `playground/src/output/curves.ts` with same names.
- Modes are TSX components composed of primitives; mode parameter contracts are JSON schemas with codegen → C++/TS types. **No declarative JSON UI.**
- WASM and firmware share the same C++; WASM is fixed at `MLP<2, 10, 14, 18, 126>` and modes use a slice of outputs based on schema's `output_size`.
- `Curve` enum lives in `nisps/core/math.hpp` (lowercase: `linear/exp/log/square/sqrt/sigmoid/cubic`, plus the parameterised `centered_power` free function); generated mode headers re-export via `using Curve = ::nisps::Curve;`. Since P4 there is NO TS mirror — the browser samples the WASM catalog (`nisps_curve_apply(+batch)`).
- Modes are TSX components composed of primitives; mode parameter contracts are JSON schemas with codegen → C++ **and** TS types (`MF_MODES` derives params/ml-config from the generated schemas since P5; labels/ordering stay a manifold overlay). **No declarative JSON UI.**
- WASM and firmware share the same C++; the browser MLP is runtime-shaped (`MLPCore<DynamicStorage>`, since P2): `nisps_ml_create` honours `(input, output, hidden[3])` with non-positive/null args defaulting to `32→[10,14,18]→126`; `nisps_ml_reshape` warm-starts a new shape. Firmware keeps compile-time `MLP<...>` (zero heap). Manifold normal modes use a 2-input UI default and can opt into 4 inputs; audio-analysis keeps its schema-defined feature arity. Firmware mode shapes remain schema-real.
- Cross-platform parity: `scripts/parity-check.sh` enforces native vs WASM agreement within 1e-5.
## Gotchas
- `src/memllib` submodule is not auto-checked-out.
- Firmware sketch path is `firmware/MEMLNaut-NISPS/MEMLNaut-NISPS.ino` (Arduino-CLI requires sketch dir name == sketch file name); `firmware/MEMLNaut-NISPS/src/{memllib,daisysp,nisps}` are symlinks because Arduino's preprocessor refuses `..` in includes from sketch headers.
- Firmware needs PlatformIO: `nix-shell -p platformio-core`. Use `platformio-core`, NOT `platformio` — the latter is nixpkgs' bubblewrap-wrapped FHS build and fails without a working user namespace. First build pulls ~1-2 GB into `~/.platformio`.
- `firmware/MEMLNaut-NISPS/glue/mode_select.hpp` `#undef`s Arduino macros (`sq`, `min`, `max`, `abs`, `round`) before pulling nisps headers — engines use those identifiers as method names.
- `nisps_firmware::g_active_mode_bridge` is `extern` in `glue/audio_driver.hpp` and defined in the `.ino`; combining `inline` with `__not_in_flash` produces a comdat conflict at link time.
- The host fallback of `NISPS_AUDIO_FUNC` in `nisps/core/perf.hpp` is misshapen for use as a function-name decorator (firmware path expands to `__not_in_flash_func` which takes only a name); firmware glue avoids the macro to dodge the inconsistency. See `ALIGNMENT.md`.
- `nisps_modes_tests` builds against generated schemas under `nisps/modes/generated/`; if you add a new mode, regenerate via `bun run codegen/generate.ts` before building.
- `nisps_firmware::g_active_mode_bridge` is `extern` in `glue/audio_driver.hpp` and defined in `src/main.cpp`; combining `inline` with `__not_in_flash` produces a comdat conflict at link time.
- `pio run`'s own "Flash: NN%" console line double-counts `.data` on this board (PlatformIO's generic size checker counts every PROGBITS+ALLOC section). Compare `arm-none-eabi-size -A` — flash = `.text+.rodata` — before believing a size regression.
- `nisps_modes_tests` builds against generated schemas under `nisps/modes/generated/`; if you add a new mode, regenerate via `bun run codegen/generate.ts` before building. It also compiles one firmware header (`glue/codec_config.hpp`), so the repo root is on its include path.
- `nisps::DriverConfig`'s member defaults are load-bearing: they reproduce memllib's historical hardcoded codec setup, so a mode that declares nothing gets exactly the pre-wiring behaviour. Changing them changes the codec setup of every mode that expresses no opinion (pinned by `tests/cpp/test_mode_driver_config.cpp`).
## Smells / strategic concerns
@ -170,9 +200,10 @@ See `ALIGNMENT.md`.
## Specs
- **Root**: `docs/specs/`
- **Entry**: `README.md`
- **Layout**: `flat`
- **Index**: none
- **Entry**: `MAIN.md`
- **Layout**: flat (with `plans/`, `recon/`, `_archive/` subdirs)
- **Index**: none (intentionally — generate when a consumer exists)
- **Skill**: invoke `/specs` to review/maintain/add/navigate.
- **Conventions**: Four-genre ontology — `kind: spec` (timeless contract, wins by intent), `kind: plan` (status: active|executed|superseded, never authority for behaviour), `kind: finding` (dated, immutable, exempt from drift lint), ADRs in `docs/adr/`.
The corpus is a tiered T0–T4 prescriptive plan ("what we are going to build"); only `README.md` plus a handful of files exist so far. Alongside the tiers it also holds **stable reference specs** for already-built features — e.g. `slp-workshop-firmware.md` (SLP-Workshop firmware mode + the Jolt / OU-explore adaptive-learning gestures). Before changing the behaviour a spec covers, find it via `/specs`; the spec wins by intent — if it's wrong, update it in the same commit as the code.
The corpus holds platform-level specs (engine architecture, I/O backends, feedback design), implementation specs (port specs, wire protocols), feature specs (e.g. slp-workshop-firmware.md), historical findings (dated research artifacts), and finite build plans. Before changing behaviour a spec covers, find it via `/specs`; the spec wins by intent — if it's wrong, update it in the same commit as the code.

View file

@ -1,140 +0,0 @@
# NISPS Core Extraction Plan
Extract a platform-agnostic C++20 controller library from MEMLNaut-NISPS. This is **not** a synth or audio engine - it's a parameter mapping engine: control data in → ML → control data out. Use it to drive synths, effects, lights, robots, whatever.
> **Note**: Originally planned as C++17, upgraded to C++20 during implementation to use `std::span` for efficient array views.
## What This Is
NISPS core takes N input parameters (joystick position, sensor data, audio features) and maps them to M output parameters through an interactively-trained neural network. Users teach it by example: "when I'm here in input space, I want these output values."
## Dependencies to Remove
| Dependency | Replacement |
|------------|-------------|
| `Serial.print*` | Optional log callback |
| `queue_t` (Pico SDK) | Not needed (single-threaded) |
| `WString.h` (Arduino) | `std::string` |
| `__force_inline`, `AUDIO_MEM` | No-op macros |
## Structure
```
nisps/
├── mlp.hpp # MLP implementation (from memlp, cleaned)
├── dataset.hpp # Training dataset
├── iml.hpp # Interactive ML engine (~200 lines)
└── voice_space.hpp # Optional: example parameter mappings
```
## Core API
```cpp
namespace nisps {
template<typename Float = float>
class IML {
public:
IML(size_t n_inputs, size_t n_outputs,
std::vector<size_t> hidden_layers = {10, 10, 14});
// Input
void set_input(size_t index, Float value);
void set_inputs(const Float* values, size_t count);
// Output (valid after process())
const Float* get_outputs() const;
size_t num_outputs() const;
// Runtime
void process(); // Run inference, call at control rate
// Training workflow
enum class Mode { Inference, Training };
void set_mode(Mode mode);
void save_example(); // Store current input→output as training pair
void clear_dataset();
void randomise_weights(); // For exploration in training mode
void train(); // Blocking, runs on current dataset
// Optional
using LogFn = void(*)(const char*);
void set_logger(LogFn fn);
};
} // namespace nisps
```
## Usage
```cpp
#include "nisps/iml.hpp"
nisps::IML<float> iml(3, 24); // 3 inputs (x,y,z), 24 outputs
// Control loop
void update(float x, float y, float z) {
iml.set_input(0, x);
iml.set_input(1, y);
iml.set_input(2, z);
iml.process();
const float* params = iml.get_outputs();
my_synth.set_filter_cutoff(params[0] * 10000.f);
my_synth.set_resonance(params[1]);
// ... etc
}
// Training (triggered by user interaction)
void on_user_saves_position() {
iml.save_example();
}
void on_user_exits_training_mode() {
iml.set_mode(nisps::IML<>::Mode::Inference);
// This triggers training internally
}
```
## Voice Spaces (Optional)
Voice spaces are just functions that interpret the raw 0-1 output parameters. Not part of core, but useful as examples:
```cpp
// User-defined mapping
void apply_neve_style(const float* params, MyChannelStrip& strip) {
strip.pre_gain = 0.5f + params[0] * params[0] * 4.f;
strip.hp_freq = 30.f + params[8] * params[8] * 270.f;
strip.comp_threshold = 20.f + params[10] * -40.f;
// ... etc
}
```
## Phases
### Phase 1: Get memlp building standalone (1-2 days)
1. Copy `memlp` source into `nisps/`
2. Remove `Serial.print` calls (or stub them)
3. Remove Arduino `String` usage
4. Verify it compiles with g++/clang
### Phase 2: Wrap in IML interface (2-3 days)
1. Create `iml.hpp` with the API above
2. Port state machine logic from `IMLInterface.hpp`
3. Simple test: train on XOR, verify inference works
### Phase 3: Example integration (1-2 days)
1. Command-line example that reads CSV input, outputs CSV
2. Or: minimal JUCE/SDL example with mouse input
**Total: ~1 week to something usable**
## Later (only if needed)
- Model serialization (save/load trained weights)
- Thread-safe parameter updates
- Python bindings
- WASM build

View file

@ -1,678 +0,0 @@
# NISPS Core Extraction - Task Graph
Atomic tasks for extracting nisps-core. Each task is standalone and requires no decisions.
---
## Repository Setup
### TASK-001: Create nisps-core directory structure
**Blocked by**: None
**Description**: Create the nisps-core directory with the required structure.
**Actions**:
1. Create directory: `nisps-core/`
2. Create directory: `nisps-core/include/nisps/`
3. Create directory: `nisps-core/test/`
4. Create directory: `nisps-core/examples/`
**Verification**: Directories exist.
### TASK-002: Create CMakeLists.txt for nisps-core
**Blocked by**: TASK-001
**Description**: Create a minimal CMakeLists.txt that builds the library as header-only with test target.
**Actions**: Create `nisps-core/CMakeLists.txt` with this exact content:
```cmake
cmake_minimum_required(VERSION 3.14)
project(nisps-core VERSION 0.1.0 LANGUAGES CXX)
set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD_REQUIRED ON)
# Header-only library
add_library(nisps INTERFACE)
target_include_directories(nisps INTERFACE
$<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/include>
$<INSTALL_INTERFACE:include>
)
# Tests
option(NISPS_BUILD_TESTS "Build tests" ON)
if(NISPS_BUILD_TESTS)
enable_testing()
add_subdirectory(test)
endif()
```
**Verification**: File exists with exact content.
### TASK-003: Create test CMakeLists.txt
**Blocked by**: TASK-002
**Description**: Create test/CMakeLists.txt for building tests.
**Actions**: Create `nisps-core/test/CMakeLists.txt` with this exact content:
```cmake
add_executable(nisps_test main.cpp)
target_link_libraries(nisps_test PRIVATE nisps)
add_test(NAME nisps_test COMMAND nisps_test)
```
**Verification**: File exists with exact content.
### TASK-004: Create placeholder test file
**Blocked by**: TASK-003
**Description**: Create a minimal test file that includes the main header.
**Actions**: Create `nisps-core/test/main.cpp` with this exact content:
```cpp
#include <nisps/nisps.hpp>
#include <iostream>
int main() {
std::cout << "nisps-core tests placeholder\n";
return 0;
}
```
**Verification**: File exists with exact content.
---
## Phase 1: Extract MLP Core
### TASK-010: Copy Utils.h with Arduino code removed
**Blocked by**: TASK-001
**Description**: Copy `src/memlp/Utils.h` to `nisps-core/include/nisps/utils.hpp`, removing Arduino-specific code.
**Actions**:
1. Copy the file
2. Remove the `#ifdef ARDUINO` block (lines related to ENABLE_SAVE)
3. Remove the `extern "C" int getentropy` declaration at the end
4. Change header guard from `UTILS_H` to `NISPS_UTILS_HPP`
5. Wrap everything in `namespace nisps { ... }`
**Changes to make** (be exact):
- Line 1: Change `#ifndef UTILS_H` to `#ifndef NISPS_UTILS_HPP`
- Line 2: Change `#define UTILS_H` to `#define NISPS_UTILS_HPP`
- Remove lines 38-45 (the `#if defined(_WIN32)...` and `#ifdef ARDUINO` ENABLE_SAVE blocks)
- After line 57 (`namespace utils {`), keep as-is (utils is a sub-namespace)
- Before the final `#endif`, add closing brace for nisps namespace
- Remove the last line `extern "C" int getentropy...`
- Add `namespace nisps {` after the includes, before `enum ACTIVATION_FUNCTIONS`
- Add `} // namespace nisps` before `#endif // NISPS_UTILS_HPP`
**Verification**: File compiles with `g++ -std=c++17 -fsyntax-only nisps-core/include/nisps/utils.hpp`
### TASK-011: Copy Loss.h
**Blocked by**: TASK-001
**Description**: Copy `src/memlp/Loss.h` to `nisps-core/include/nisps/loss.hpp`.
**Actions**:
1. Copy the file
2. Change header guard from `LOSS_H` to `NISPS_LOSS_HPP`
3. Wrap everything in `namespace nisps { ... }`
4. Update include from `"Utils.h"` to `"utils.hpp"`
**Verification**: File compiles with `g++ -std=c++17 -fsyntax-only -I nisps-core/include nisps-core/include/nisps/loss.hpp`
### TASK-012: Copy Node.h
**Blocked by**: TASK-010
**Description**: Copy `src/memlp/Node.h` to `nisps-core/include/nisps/node.hpp`.
**Actions**:
1. Copy the file
2. Change header guard from `NODE_H` to `NISPS_NODE_HPP`
3. Wrap everything in `namespace nisps { ... }`
4. Update include from `"Utils.h"` to `"utils.hpp"`
**Verification**: File compiles with `g++ -std=c++17 -fsyntax-only -I nisps-core/include nisps-core/include/nisps/node.hpp`
### TASK-013: Copy Layer.h
**Blocked by**: TASK-012
**Description**: Copy `src/memlp/Layer.h` to `nisps-core/include/nisps/layer.hpp`.
**Actions**:
1. Copy the file
2. Change header guard from `LAYER_H` to `NISPS_LAYER_HPP`
3. Wrap everything in `namespace nisps { ... }`
4. Update include from `"Node.h"` to `"node.hpp"`
**Verification**: File compiles with `g++ -std=c++17 -fsyntax-only -I nisps-core/include nisps-core/include/nisps/layer.hpp`
### TASK-014: Copy Sample.h
**Blocked by**: TASK-001
**Description**: Copy `src/memlp/Sample.h` to `nisps-core/include/nisps/sample.hpp`.
**Actions**:
1. Copy the file
2. Change header guard from `SAMPLE_H` to `NISPS_SAMPLE_HPP`
3. Wrap everything in `namespace nisps { ... }`
**Verification**: File compiles with `g++ -std=c++17 -fsyntax-only nisps-core/include/nisps/sample.hpp`
### TASK-015: Copy MLP.h with Arduino code removed
**Blocked by**: TASK-011, TASK-013, TASK-014
**Description**: Copy `src/memlp/MLP.h` to `nisps-core/include/nisps/mlp.hpp`, removing Arduino-specific code.
**Actions**:
1. Copy the file
2. Change header guard from `MLP_H` to `NISPS_MLP_HPP`
3. **Remove lines 17-51** (the entire `#ifdef ARDUINO` block including Serial debug macros and SD includes)
4. Add these lines after the header guard instead:
```cpp
// Debug macros (no-op by default, override before including if needed)
#ifndef NISPS_DEBUG_PRINT
#define NISPS_DEBUG_PRINT(...)
#define NISPS_DEBUG_PRINTLN(...)
#define NISPS_DEBUG_PRINTF(...)
#endif
```
5. Update includes:
- `"Layer.h"` → `"layer.hpp"`
- `"Utils.h"` → `"utils.hpp"`
- `"Loss.h"` → `"loss.hpp"`
- `"Sample.h"` → `"sample.hpp"`
6. Remove `#if ENABLE_SAVE` blocks (lines 94-96, 99-113, 115-119) - remove the conditionals but **keep** the function declarations
7. Remove `#if ENABLE_SAVE_SD` block entirely (lines 115-119)
8. Wrap everything in `namespace nisps { ... }`
9. Replace `MLP_DEBUG_PRINT` with `NISPS_DEBUG_PRINT`, `MLP_DEBUG_PRINTLN` with `NISPS_DEBUG_PRINTLN`, `MLP_DEBUG_PRINTF` with `NISPS_DEBUG_PRINTF`
**Verification**: File compiles with `g++ -std=c++17 -fsyntax-only -I nisps-core/include nisps-core/include/nisps/mlp.hpp`
### TASK-016: Copy MLP.cpp with Arduino code removed
**Blocked by**: TASK-015
**Description**: Copy `src/memlp/MLP.cpp` to `nisps-core/include/nisps/mlp_impl.hpp` as inline implementation.
**Actions**:
1. Copy the file
2. Remove `#include "MLP.h"` (it will be included from mlp.hpp)
3. Add header guard `#ifndef NISPS_MLP_IMPL_HPP` / `#define NISPS_MLP_IMPL_HPP`
4. Wrap everything in `namespace nisps { ... }`
5. Remove all `#ifdef ARDUINO` / `#if ENABLE_SAVE` / `#if ENABLE_SAVE_SD` conditional blocks
6. For functions inside removed conditionals: keep the functions but remove the `#if` guards
7. Replace any `Serial.print*` calls with `NISPS_DEBUG_PRINT*` equivalents
8. At the end of `nisps-core/include/nisps/mlp.hpp`, add: `#include "mlp_impl.hpp"`
**Verification**: A test file that instantiates `nisps::MLP<float>` compiles and links.
### TASK-017: Copy Dataset.hpp
**Blocked by**: TASK-001
**Description**: Copy `src/memlp/Dataset.hpp` to `nisps-core/include/nisps/dataset.hpp`.
**Actions**:
1. Copy the file
2. Change header guard from `__DATASET_HPP__` to `NISPS_DATASET_HPP`
3. Wrap everything in `namespace nisps { ... }`
**Verification**: File compiles with `g++ -std=c++17 -fsyntax-only nisps-core/include/nisps/dataset.hpp`
### TASK-018: Copy Dataset.cpp as inline implementation
**Blocked by**: TASK-017
**Description**: Copy `src/memlp/Dataset.cpp` to `nisps-core/include/nisps/dataset_impl.hpp`.
**Actions**:
1. Copy the file
2. Remove `#include "Dataset.hpp"`
3. Add header guard `#ifndef NISPS_DATASET_IMPL_HPP` / `#define NISPS_DATASET_IMPL_HPP`
4. Wrap everything in `namespace nisps { ... }`
5. Make all functions `inline`
6. At the end of `nisps-core/include/nisps/dataset.hpp`, add: `#include "dataset_impl.hpp"`
**Verification**: A test file that instantiates `nisps::Dataset` compiles and links.
---
## Phase 2: Create IML Interface
### TASK-020: Create iml.hpp header
**Blocked by**: TASK-016, TASK-018
**Description**: Create the main IML class header based on IMLInterface.hpp.
**Actions**: Create `nisps-core/include/nisps/iml.hpp` with this exact content:
```cpp
#ifndef NISPS_IML_HPP
#define NISPS_IML_HPP
#include "mlp.hpp"
#include "dataset.hpp"
#include <vector>
#include <cstddef>
#include <functional>
namespace nisps {
template<typename Float = float>
class IML {
public:
enum class Mode { Inference, Training };
using LogFn = void(*)(const char*);
IML(size_t n_inputs, size_t n_outputs,
std::vector<size_t> hidden_layers = {10, 10, 14},
size_t max_iterations = 1000,
Float learning_rate = 1.0f,
Float convergence_threshold = 0.00001f);
// Input
void set_input(size_t index, Float value);
void set_inputs(const Float* values, size_t count);
// Output (valid after process())
const Float* get_outputs() const;
size_t num_inputs() const { return n_inputs_; }
size_t num_outputs() const { return n_outputs_; }
// Runtime
void process();
// Training workflow
void set_mode(Mode mode);
Mode get_mode() const { return mode_; }
void save_example();
void clear_dataset();
void randomise_weights();
// Optional logging
void set_logger(LogFn fn) { log_fn_ = fn; }
private:
void log(const char* msg) const {
if (log_fn_) log_fn_(msg);
}
void train();
size_t n_inputs_;
size_t n_outputs_;
size_t max_iterations_;
Float learning_rate_;
Float convergence_threshold_;
Mode mode_ = Mode::Inference;
bool input_updated_ = false;
bool perform_inference_ = true;
std::vector<Float> input_state_;
std::vector<Float> output_state_;
std::unique_ptr<Dataset> dataset_;
std::unique_ptr<MLP<Float>> mlp_;
typename MLP<Float>::mlp_weights stored_weights_;
bool weights_randomised_ = false;
LogFn log_fn_ = nullptr;
};
} // namespace nisps
#include "iml_impl.hpp"
#endif // NISPS_IML_HPP
```
**Verification**: File exists with exact content.
### TASK-021: Create iml_impl.hpp implementation
**Blocked by**: TASK-020
**Description**: Create the IML implementation file based on IMLInterface.hpp logic.
**Actions**: Create `nisps-core/include/nisps/iml_impl.hpp` with this exact content:
```cpp
#ifndef NISPS_IML_IMPL_HPP
#define NISPS_IML_IMPL_HPP
namespace nisps {
template<typename Float>
IML<Float>::IML(size_t n_inputs, size_t n_outputs,
std::vector<size_t> hidden_layers,
size_t max_iterations,
Float learning_rate,
Float convergence_threshold)
: n_inputs_(n_inputs)
, n_outputs_(n_outputs)
, max_iterations_(max_iterations)
, learning_rate_(learning_rate)
, convergence_threshold_(convergence_threshold)
{
// Build layer sizes: input + hidden + output
const size_t kBias = 1;
std::vector<size_t> layer_sizes;
layer_sizes.push_back(n_inputs + kBias);
for (size_t h : hidden_layers) {
layer_sizes.push_back(h);
}
layer_sizes.push_back(n_outputs);
// Activation functions: RELU for hidden, SIGMOID for output
std::vector<ACTIVATION_FUNCTIONS> activations;
for (size_t i = 0; i < hidden_layers.size(); ++i) {
activations.push_back(RELU);
}
activations.push_back(SIGMOID);
dataset_ = std::make_unique<Dataset>();
mlp_ = std::make_unique<MLP<Float>>(
layer_sizes,
activations,
loss::LOSS_MSE,
false, // use_constant_weight_init
0.0f // constant_weight_init
);
input_state_.resize(n_inputs, static_cast<Float>(0.5));
output_state_.resize(n_outputs, static_cast<Float>(0));
}
template<typename Float>
void IML<Float>::set_input(size_t index, Float value) {
if (index >= n_inputs_) return;
if (value < 0) value = 0;
if (value > 1) value = 1;
input_state_[index] = value;
input_updated_ = true;
}
template<typename Float>
void IML<Float>::set_inputs(const Float* values, size_t count) {
for (size_t i = 0; i < count && i < n_inputs_; ++i) {
set_input(i, values[i]);
}
}
template<typename Float>
const Float* IML<Float>::get_outputs() const {
return output_state_.data();
}
template<typename Float>
void IML<Float>::process() {
if (!perform_inference_ || !input_updated_) return;
// Add bias term
std::vector<Float> input_with_bias = input_state_;
input_with_bias.push_back(static_cast<Float>(1.0));
// Run inference
std::vector<Float> output(n_outputs_);
mlp_->GetOutput(input_with_bias, &output);
output_state_ = output;
input_updated_ = false;
}
template<typename Float>
void IML<Float>::set_mode(Mode mode) {
if (mode == Mode::Inference && mode_ == Mode::Training) {
train();
}
mode_ = mode;
}
template<typename Float>
void IML<Float>::save_example() {
// First call: stop inference, user will position output
if (perform_inference_) {
perform_inference_ = false;
log("Move to desired output position...");
return;
}
// Second call: store the example
dataset_->Add(input_state_, output_state_);
perform_inference_ = true;
// Run inference with new example
std::vector<Float> input_with_bias = input_state_;
input_with_bias.push_back(static_cast<Float>(1.0));
std::vector<Float> output(n_outputs_);
mlp_->GetOutput(input_with_bias, &output);
output_state_ = output;
log("Example saved.");
}
template<typename Float>
void IML<Float>::clear_dataset() {
if (mode_ == Mode::Training) {
dataset_->Clear();
log("Dataset cleared.");
}
}
template<typename Float>
void IML<Float>::randomise_weights() {
if (mode_ == Mode::Training) {
stored_weights_ = mlp_->GetWeights();
mlp_->DrawWeights();
weights_randomised_ = true;
// Run inference to show effect
std::vector<Float> input_with_bias = input_state_;
input_with_bias.push_back(static_cast<Float>(1.0));
std::vector<Float> output(n_outputs_);
mlp_->GetOutput(input_with_bias, &output);
output_state_ = output;
log("Weights randomised.");
}
}
template<typename Float>
void IML<Float>::train() {
// Restore weights if they were randomised
if (weights_randomised_) {
mlp_->SetWeights(stored_weights_);
weights_randomised_ = false;
}
auto features = dataset_->GetFeatures(true); // with bias
auto& labels = dataset_->GetLabels();
if (features.empty() || labels.empty()) {
log("Empty dataset, skipping training.");
return;
}
typename MLP<Float>::training_pair_t training_data(features, labels);
log("Training...");
Float loss = mlp_->Train(
training_data,
learning_rate_,
static_cast<int>(max_iterations_),
convergence_threshold_,
false // output_log
);
// Run inference after training
std::vector<Float> input_with_bias = input_state_;
input_with_bias.push_back(static_cast<Float>(1.0));
std::vector<Float> output(n_outputs_);
mlp_->GetOutput(input_with_bias, &output);
output_state_ = output;
log("Training complete.");
}
} // namespace nisps
#endif // NISPS_IML_IMPL_HPP
```
**Verification**: File exists with exact content.
---
## Phase 3: Create Main Header and Test
### TASK-030: Create main nisps.hpp header
**Blocked by**: TASK-020, TASK-021
**Description**: Create the main include header that exposes the public API.
**Actions**: Create `nisps-core/include/nisps/nisps.hpp` with this exact content:
```cpp
#ifndef NISPS_HPP
#define NISPS_HPP
#include "iml.hpp"
#endif // NISPS_HPP
```
**Verification**: File exists with exact content.
### TASK-031: Create XOR training test
**Blocked by**: TASK-030
**Description**: Replace the placeholder test with an XOR training test.
**Actions**: Replace `nisps-core/test/main.cpp` with this exact content:
```cpp
#include <nisps/nisps.hpp>
#include <iostream>
#include <cmath>
void log_callback(const char* msg) {
std::cout << "[nisps] " << msg << "\n";
}
int main() {
std::cout << "=== NISPS Core Test: XOR Training ===\n\n";
// Create IML with 2 inputs, 1 output
nisps::IML<float> iml(2, 1, {4, 4}, 5000, 1.0f, 0.0001f);
iml.set_logger(log_callback);
// Enter training mode
iml.set_mode(nisps::IML<float>::Mode::Training);
// Train on XOR pattern
// (0,0) -> 0
iml.set_input(0, 0.0f);
iml.set_input(1, 0.0f);
iml.save_example(); // First call: stop inference
// Manually set output for this example (simulating user positioning)
// We access output_state_ indirectly by calling process after training
// For this test, we'll add examples directly to dataset
// This simulates the two-step save process
// Actually, let's test the full workflow properly:
// The IML class expects: save_example() twice per example
// 1. First call stops inference
// 2. User sets output position (we can't do this externally easily)
// 3. Second call stores input->output
// For testing, let's verify the basic inference works
iml.set_mode(nisps::IML<float>::Mode::Inference);
// Test inference
iml.set_input(0, 0.0f);
iml.set_input(1, 0.0f);
iml.process();
float out_00 = iml.get_outputs()[0];
iml.set_input(0, 1.0f);
iml.set_input(1, 0.0f);
iml.process();
float out_10 = iml.get_outputs()[0];
iml.set_input(0, 0.0f);
iml.set_input(1, 1.0f);
iml.process();
float out_01 = iml.get_outputs()[0];
iml.set_input(0, 1.0f);
iml.set_input(1, 1.0f);
iml.process();
float out_11 = iml.get_outputs()[0];
std::cout << "\nInference results (untrained):\n";
std::cout << " (0,0) -> " << out_00 << "\n";
std::cout << " (1,0) -> " << out_10 << "\n";
std::cout << " (0,1) -> " << out_01 << "\n";
std::cout << " (1,1) -> " << out_11 << "\n";
std::cout << "\n=== Test passed: nisps-core compiles and runs ===\n";
return 0;
}
```
**Verification**: Test compiles and runs successfully.
### TASK-032: Build and run tests
**Blocked by**: TASK-031
**Description**: Build the library and run the test.
**Actions**:
```bash
cd nisps-core
mkdir -p build && cd build
cmake ..
make
ctest --output-on-failure
```
**Verification**: All commands succeed, test outputs "Test passed".
---
## Dependency Graph (ASCII)
```
TASK-001 (create dirs)
│
├──> TASK-002 (CMakeLists.txt)
│ │
│ └──> TASK-003 (test CMakeLists.txt)
│ │
│ └──> TASK-004 (placeholder test)
│
├──> TASK-010 (utils.hpp)
│ │
│ └──> TASK-011 (loss.hpp)
│ │ │
│ └──> TASK-012 (node.hpp)
│ │
│ └──> TASK-013 (layer.hpp)
│
├──> TASK-014 (sample.hpp)
│
├──> TASK-017 (dataset.hpp)
│ │
│ └──> TASK-018 (dataset_impl.hpp)
│
└───────────────────────────────────────┐
│
TASK-011 + TASK-013 + TASK-014 ────────> TASK-015 (mlp.hpp)
│
└──> TASK-016 (mlp_impl.hpp)
│
TASK-016 + TASK-018 ───────────────────────────> TASK-020 (iml.hpp)
│
└──> TASK-021 (iml_impl.hpp)
│
TASK-020 + TASK-021 ───────────────────────────────────> TASK-030 (nisps.hpp)
│
└──> TASK-031 (test)
│
└──> TASK-032 (build & run)
```
---
## Critical Path
The minimum path to a working library:
1. TASK-001 → TASK-002 → TASK-003 → TASK-004 (setup)
2. TASK-010 → TASK-012 → TASK-013 (utils → node → layer)
3. TASK-011 (loss, parallel with above)
4. TASK-014 (sample, parallel)
5. TASK-015 → TASK-016 (mlp)
6. TASK-017 → TASK-018 (dataset, parallel with 3-5)
7. TASK-020 → TASK-021 (iml)
8. TASK-030 → TASK-031 → TASK-032 (header + test + build)
**Estimated time**: 4-6 hours for a capable agent.
---
## File Checklist
After all tasks complete, these files should exist:
```
nisps-core/
├── CMakeLists.txt
├── include/
│ └── nisps/
│ ├── nisps.hpp
│ ├── iml.hpp
│ ├── iml_impl.hpp
│ ├── mlp.hpp
│ ├── mlp_impl.hpp
│ ├── dataset.hpp
│ ├── dataset_impl.hpp
│ ├── layer.hpp
│ ├── node.hpp
│ ├── loss.hpp
│ ├── sample.hpp
│ └── utils.hpp
├── test/
│ ├── CMakeLists.txt
│ └── main.cpp
└── examples/
```

View file

@ -21,16 +21,22 @@ Notes:
- `build-firmware.sh` accepts an optional variant name such as `MEMLCelium` or `BreakOr`. Matching remains case-insensitive, so `memlcelium` still works. If you omit it in an interactive shell, the script parses `MEMLNaut-NISPS.ino`, prompts for a variant, and rewrites the active `MEMLNAUT_MODE_TYPE` before building.
- `flash-firmware.sh` accepts an optional mountpoint argument, or auto-detects common UF2 bootloader mounts such as `/run/media/$USER/RP2350` and `/run/media/$USER/RPI-RP2`.
## Web Playground
## Manifold (browser app)
Try NISPS in your browser — no hardware required:
Try NISPS in your browser — no hardware required. Manifold is the React front-end
running the same C++ engines + ML as the firmware, compiled to WASM:
```bash
cd playground
python3 -m http.server
# Open http://localhost:8000
cd manifold
bun install
bun run dev
```
Train a neural network to map joystick positions to generative visuals through interactive machine learning. Two learning modes: direct example mapping and reinforcement learning with thumbs up/down feedback.
Staging deployment: https://meml.lnfinitemonkeys.org/next/
The playground UI includes an **Expand** toggle on the visual surface so you can make the canvas nearly full-screen while compressing parameter/control panels into a minimal strip beneath it.
Train a neural network to map input gestures to synth parameters through interactive
machine learning: place examples, or use verdict-based feedback (explore-and-place,
geometric dislike).
(The former SolidJS playground was retired in July 2026 — archived on branch
`archive/playground-solidjs`, tag `playground-solidjs-final`.)

View file

@ -14,7 +14,7 @@ British spelling throughout the on-screen copy (randomise, centre, behaviour).
| `01-knob-mapping.mp4` | ~21s | A knob rotates while a paired 0–1 number rises/falls in lockstep. The point: a control is just a readout of one number. | **Strong** — a live knob bound to a number is the simplest possible interactive widget. | Onboarding step 1 / the very first "what is a control" beat. Pairs with the `Slider`/`Knob` primitives. |
| `02-xy-pad.mp4` | ~20s | A 2-D pad projects to two independent 0–1 values A and B, each shown as a slider, as the dot moves. | **Strong** — a `JoyMap` driving two live readouts is already close to an in-app primitive demo. | Onboarding step 2. Directly mirrors the `JoyMap` primitive + two `Slider`s. |
| `03-fanout.mp4` | ~27s | The same 2-D control driving 3 → 4 → 5 outputs (N sliders one side, control the other), then the control type cycled across knob, fader, touchpad, 2-D joystick, 3-D joystick. | **Medium** — the N-slider fan-out is demoable live; the control-type carousel is better as video. | Onboarding step 3 — the "why a network in the middle" moment, just before the mode UI. Connects to the dimensionality story in the mode schemas. |
| `04-feedback.mp4` | ~53s | Training on the mapping surface (projected to 2-D) and how a verdict reshapes it. **Contrasts the two feedback modes:** geometric push-away (directed repulsion from the liked centroid) vs explore-and-place (randomise → audition → place → interpolate). Ends on a side-by-side and the recommendation. | **Medium/low** — concept-heavy; the explore-and-place loop could become a guided interactive walkthrough, but the contrast slide is best as video. | The learning-behaviour onboarding / the dock-panel `FEEDBACK_MODE` selector help. Coordinates with `docs/redesign/rl-feedback-design.md` (explore-and-place is the recommended default). |
| `04-feedback.mp4` | ~53s | Training on the mapping surface (projected to 2-D) and how a verdict reshapes it. **Contrasts the two feedback modes:** geometric push-away (directed repulsion from the liked centroid) vs explore-and-place (randomise → audition → place → interpolate). Ends on a side-by-side and the recommendation. | **Medium/low** — concept-heavy; the explore-and-place loop could become a guided interactive walkthrough, but the contrast slide is best as video. | The learning-behaviour onboarding / the dock-panel `FEEDBACK_MODE` selector help. Coordinates with `docs/adr/rl-feedback-design.md` (explore-and-place is the recommended default). |
`04-feedback.mp4` is `s4_feedback.py`'s three scenes (`GeometricDislike`,
`ExploreAndPlace`, `FeedbackContrast`) concatenated.

View file

@ -1,67 +1,50 @@
# MEMLNaut Mode-Schema Codegen
# codegen — schemas → C++ + TypeScript
Bun + TypeScript tool that turns `schemas/modes/*.json` into:
Bun scripts that turn the JSON schemas under `schemas/` into generated code for both
targets. The per-file inventory lives in `MAP.md` (§ `codegen/`) — this README only
covers how to run it.
- C++ headers under `nisps/modes/generated/<mode_id>_schema.hpp` (`constexpr` data, no runtime cost).
- TypeScript modules under `playground/src/modes/generated/<mode_id>_schema.ts` (typed `ModeSchema` objects).
- `generate.ts` — validates `schemas/modes/*.json` against `schemas/schema.json`, emits
`nisps/modes/generated/<mode>_schema.hpp` (+ `schema_types.hpp`) and
`manifold/src/modes/generated/<mode>_schema.ts` (+ `types.ts`, `index.ts`).
- `generate-midi-devices.ts` — validates `schemas/midi_devices/*.json` against
`schemas/midi_device.schema.json`, emits `nisps/midi/generated/midi_devices.hpp` and
`manifold/src/midi-devices/generated/{types,devices,index}.ts`.
The schemas are validated against `schemas/schema.json` (JSON Schema Draft 2020-12) on every run. Codegen exits non-zero if any schema fails validation.
Both validate with ajv (Draft 2020-12), exit non-zero on failure, and are idempotent
(re-running with unchanged schemas is byte-identical).
## Run
```bash
cd codegen
bun install # one-shot, fetches ajv + types
bun run generate.ts # or: bun run generate
bun install
bun run generate.ts # mode schemas
bun run generate-midi-devices.ts # MIDI device templates
```
The script writes everything into the two output dirs in one shot. Re-running with no schema changes is a no-op (byte-identical output).
Schema or codegen changes ship with the regenerated C++ **and** TypeScript in the same
commit — CI re-runs both generators and fails on any diff.
## Tests
## Golden test
`tests/golden_test.ts` regenerates from the live schemas into a temp dir and diffs against `tests/golden/`. The golden directory contains a snapshot of `paf_synth_schema.{hpp,ts}`. To refresh after intentional codegen changes:
```bash
bun run generate.ts
cp ../nisps/modes/generated/paf_synth_schema.hpp tests/golden/
cp ../playground/src/modes/generated/paf_synth_schema.ts tests/golden/
```
Run the test:
`tests/golden_test.ts` re-runs `generate.ts` against the live output dirs, diffs
`paf_synth_schema.{hpp,ts}` against the snapshots in `tests/golden/`, then re-runs to
prove idempotency:
```bash
bun run test
```
## Adding a new mode
After an intentional emission change, refresh the goldens:
1. Drop a new `<mode_id>.json` into `schemas/modes/` (must validate against `schemas/schema.json`).
2. Run `bun run generate.ts`.
3. Commit the JSON + the regenerated C++/TS pair.
## Layout
```
codegen/
├── package.json # ajv (+ formats) + bun-types
├── tsconfig.json
├── generate.ts # ~400 lines, single entrypoint
├── README.md # this file
├── templates/ # reference templates (NOT consumed — for review)
│ ├── cpp_schema.hpp.template
│ └── ts_schema.ts.template
└── tests/
├── golden_test.ts # diffs latest output against tests/golden/
└── golden/
├── paf_synth_schema.hpp
└── paf_synth_schema.ts
```bash
bun run generate.ts
cp ../nisps/modes/generated/paf_synth_schema.hpp tests/golden/
cp ../manifold/src/modes/generated/paf_synth_schema.ts tests/golden/
```
## Conventions
## Adding a mode / device
- **C++ namespace**: `nisps::modes::generated`. All generated symbols are `inline constexpr` so `#include`-ing the header in multiple TUs is safe.
- **C++ types**: `std::array`, `std::string_view`, `std::size_t`. No `std::vector`, no heap.
- **`Curve` enum**: declared in `nisps/modes/generated/schema_types.hpp` as a temporary local copy. Once stream 1 lands `nisps/core/math.hpp`, replace the local enum with an `#include` (search for `TODO(stream-1)` in the generated header).
- **Float literals**: emitted with explicit `.f` suffix and decimal point, per the perf contract (architecture §3.3).
- **Order**: schemas are processed in alphabetical order of mode_id so output is stable.
- **Naming**: `mode_id` is `snake_case` in JSON; the generated C++ const prefix is `k` + PascalCase (e.g. `kPafSynthParams`); TS const is PascalCase + `Schema` (e.g. `PafSynthSchema`).
Drop the new JSON into `schemas/modes/` (or `schemas/midi_devices/`), run the matching
generator, and commit the JSON plus the regenerated outputs together.

885
codegen/curve-audit.ts Normal file
View file

@ -0,0 +1,885 @@
/**
* curve-audit.ts — mechanically derive, from `nisps/engines/*.hpp` source,
* which response curve each engine applies to each NN-output slot, per voice
* space.
*
* WHY THIS IS A SOURCE-LEVEL CHECK
* --------------------------------
* "The curve" is not observable from engine output. A voice space maps a
* normalised param into engine state as `base + f(p) * scale` and then that
* state goes through DSP. Given only audio out you cannot separate `f` from
* `base`/`scale`/the DSP, and the engines expose no accessor for the mapped
* state. So the only place the curve exists as a fact is the arithmetic in
* `apply_*()` / `set_params()` / `process()` — and that is what this module
* reads.
*
* The contract it enforces is deliberately narrow and total:
*
* square <=> the engine multiplies the param slot by itself
* sqrt <=> the engine passes the param slot through std::sqrt
* linear <=> anything else (including quantisation, sin() combination and
* stepped lookup tables — those are not expressible in the
* `Curve` enum and are declared `linear` by definition)
*
* Everything it cannot reduce is a HARD ERROR, never a silent "linear". That
* is the property that makes the derived table trustworthy: a new voice space,
* a new idiom, or a renamed helper trips the check instead of quietly
* under-reporting. A regex over `p[N] * p[N]` would miss the `const float v =
* p[…]; v * v` form (verb_fx), the `sq()` implicit-counter lambda
* (memlcelium), loop-generated indices (verb_fx) and `smooth_params_[N]`
* (xiasri) — every one of those is live in this codebase today.
*
* Consumed by codegen/tests/curve_drift_test.ts, which asserts the schemas'
* declared curves (params[].curve + per-voice-space overrides) match.
*/
import { readFileSync, readdirSync } from "node:fs";
import { join } from "node:path";
export type Curve =
| "linear"
| "exp"
| "log"
| "square"
| "sqrt"
| "sigmoid"
| "cubic";
/** Identifiers that alias the NN-output vector inside an engine. */
const ACCESSORS = ["params", "p", "smooth_params_", "nn_outputs_"] as const;
const ACCESSOR_RE = new RegExp(`\\b(${ACCESSORS.join("|")})\\s*\\[`);
export interface EngineCurves {
engineId: string;
file: string;
nParams: number;
/** Names from the engine's `kVoiceSpaceNames`, or null when it has none. */
voiceSpaceNames: string[] | null;
/** curves[voiceSpaceIndex][paramIndex]. One row when there is no enum. */
curves: Curve[][];
}
export class CurveAuditError extends Error {}
function fail(where: string, msg: string): never {
throw new CurveAuditError(`[curve-audit] ${where}: ${msg}`);
}
// ---------------------------------------------------------------------------
// Lexical helpers
// ---------------------------------------------------------------------------
function stripComments(src: string): string {
// Block comments first, then line comments. No string literals in these
// files contain `//` or `/*` (checked: the only literals are identifiers).
return src.replace(/\/\*[\s\S]*?\*\//g, " ").replace(/\/\/[^\n]*/g, " ");
}
/** Index of the `)`/`}`/`]` matching the opener at `open`. */
function matchBracket(src: string, open: number): number {
const pairs: Record<string, string> = { "(": ")", "{": "}", "[": "]" };
const close = pairs[src[open]!];
if (!close) fail("matchBracket", `not an opener at ${open}: ${src[open]}`);
let depth = 0;
for (let i = open; i < src.length; i++) {
const c = src[i]!;
if (c === src[open]) depth++;
else if (c === close) {
depth--;
if (depth === 0) return i;
}
}
fail("matchBracket", `unbalanced ${src[open]} at ${open}`);
}
/** Body text (between braces, exclusive) of `<name>(...) ... { ... }`. */
function functionBody(src: string, name: string): string | null {
const re = new RegExp(`\\b${name}\\s*\\(`, "g");
let m: RegExpExecArray | null;
while ((m = re.exec(src))) {
const argOpen = m.index + m[0].length - 1;
let argClose: number;
try {
argClose = matchBracket(src, argOpen);
} catch {
continue;
}
// Between `)` and `{` only qualifiers may appear (noexcept, const, ->…).
const between = src.slice(argClose + 1, src.indexOf("{", argClose) + 1);
if (!/^[\s\w:&*<>,]*\{$/.test(between)) continue;
const braceOpen = src.indexOf("{", argClose);
if (braceOpen < 0) continue;
return src.slice(braceOpen + 1, matchBracket(src, braceOpen));
}
return null;
}
/** Formal parameter names of `<name>(...)`, in order. */
function functionParams(src: string, name: string): string[] | null {
const re = new RegExp(`\\b${name}\\s*\\(`, "g");
let m: RegExpExecArray | null;
while ((m = re.exec(src))) {
const argOpen = m.index + m[0].length - 1;
let argClose: number;
try {
argClose = matchBracket(src, argOpen);
} catch {
continue;
}
const between = src.slice(argClose + 1, src.indexOf("{", argClose) + 1);
if (!/^[\s\w:&*<>,]*\{$/.test(between)) continue;
const args = src.slice(argOpen + 1, argClose).trim();
if (args === "") return [];
return args.split(",").map((a) => {
const t = a.trim().replace(/\[\s*\]$/, "");
const w = t.match(/([A-Za-z_]\w*)\s*$/);
return w ? w[1]! : t;
});
}
return null;
}
/** Split arguments of a call at top nesting level. */
function splitArgs(text: string): string[] {
const out: string[] = [];
let depth = 0;
let cur = "";
for (const c of text) {
if (c === "(" || c === "[" || c === "{" || c === "<") depth++;
else if (c === ")" || c === "]" || c === "}" || c === ">") depth--;
if (c === "," && depth === 0) {
out.push(cur);
cur = "";
} else cur += c;
}
if (cur.trim() !== "" || out.length > 0) out.push(cur);
return out.map((s) => s.trim());
}
// ---------------------------------------------------------------------------
// Integer expression evaluation (array indices, loop bounds, ternary guards)
// ---------------------------------------------------------------------------
/**
* Evaluate a compile-time integer expression. Constants from the engine
* (`static constexpr std::size_t kX = …`) are substituted first. Anything the
* strict character whitelist rejects is a hard error — never a guess.
*/
function evalInt(expr: string, consts: Map<string, number>, where: string): number {
let e = expr;
for (let pass = 0; pass < 8; pass++) {
const before = e;
for (const [k, v] of consts) {
e = e.replace(new RegExp(`\\b${k}\\b`, "g"), `(${v})`);
}
if (e === before) break;
}
e = e.replace(/(\d)[uU]\b/g, "$1");
e = e.replace(/static_cast<[^>]*>/g, "");
if (!/^[\d\s+\-*/%()]+$/.test(e)) {
fail(where, `non-constant integer expression ${JSON.stringify(expr)}`);
}
// eslint-disable-next-line no-new-func
const v = Function(`"use strict"; return (${e});`)() as number;
if (!Number.isInteger(v)) fail(where, `non-integer index ${expr} -> ${v}`);
return v;
}
/** Evaluate a boolean guard, or null when it is not compile-time constant. */
function evalBool(expr: string, consts: Map<string, number>): boolean | null {
let e = expr.trim();
while (e.startsWith("(") && matchBracket(e, 0) === e.length - 1) {
e = e.slice(1, -1).trim();
}
for (let pass = 0; pass < 8; pass++) {
const before = e;
for (const [k, v] of consts) e = e.replace(new RegExp(`\\b${k}\\b`, "g"), `(${v})`);
if (e === before) break;
}
e = e.replace(/(\d)[uU]\b/g, "$1");
if (!/^[\d\s+\-*/%()<>=!&|]+$/.test(e)) return null;
try {
// eslint-disable-next-line no-new-func
return Boolean(Function(`"use strict"; return (${e});`)());
} catch {
return null;
}
}
// ---------------------------------------------------------------------------
// Body normalisation: inline helpers -> unroll loops -> flatten braces
// ---------------------------------------------------------------------------
interface EngineFile {
src: string;
path: string;
consts: Map<string, number>;
/** member name -> std::array element count, for range-for unrolling. */
arrayLens: Map<string, number>;
}
function loadEngine(path: string): EngineFile {
const src = stripComments(readFileSync(path, "utf8"));
const consts = new Map<string, number>();
// `static constexpr std::size_t kFoo = <expr>;` — resolved in declaration
// order so later constants may refer to earlier ones.
const cre = /static\s+constexpr\s+std::size_t\s+(\w+)\s*=\s*([^;]+);/g;
let m: RegExpExecArray | null;
while ((m = cre.exec(src))) {
try {
consts.set(m[1]!, evalInt(m[2]!, consts, path));
} catch {
/* not an integer constant we can use; ignore */
}
}
const arrayLens = new Map<string, number>();
const are = /std::array\s*<\s*[^,<>]+(?:<[^>]*>)?\s*,\s*([^>]+)>\s*(\w+)/g;
while ((m = are.exec(src))) {
try {
arrayLens.set(m[2]!, evalInt(m[1]!, consts, path));
} catch {
/* dynamic length; ignore */
}
}
return { src, path, consts, arrayLens };
}
/**
* Replace `name(args);` statements whose `name` is a method defined in the
* same file with that method's body, substituting formals for actuals.
*/
const KEYWORD_CALLS = new Set(["if", "for", "while", "switch", "return", "sizeof", "static_cast"]);
/** Locate the next bare `name(args);` statement whose `name` is defined here. */
function findInlinableCall(
body: string,
eng: EngineFile,
from: number
): { start: number; end: number; name: string; args: string } | null {
const callRe = /(?:^|[;{}])\s*([a-z_]\w*)\s*\(/g;
callRe.lastIndex = from;
let m: RegExpExecArray | null;
while ((m = callRe.exec(body))) {
const name = m[1]!;
callRe.lastIndex = m.index + m[0].length - 1;
if (KEYWORD_CALLS.has(name)) continue;
const open = m.index + m[0].length - 1;
let close: number;
try {
close = matchBracket(body, open);
} catch {
continue;
}
const rest = body.slice(close + 1);
if (rest.trimStart()[0] !== ";") continue;
if (functionBody(eng.src, name) === null) continue;
const semi = body.indexOf(";", close);
return { start: m.index + (/[;{}]/.test(m[0][0]!) ? 1 : 0), end: semi + 1, name, args: body.slice(open + 1, close) };
}
return null;
}
function inlineCalls(body: string, eng: EngineFile, seen: Set<string>, where: string): string {
let text = body;
for (let guard = 0; guard < 256; guard++) {
const hit = findInlinableCall(text, eng, 0);
if (!hit) return text;
if (seen.has(hit.name)) fail(where, `recursive inline of ${hit.name}()`);
const callee = functionBody(eng.src, hit.name)!;
const formals = functionParams(eng.src, hit.name) ?? [];
const actuals = splitArgs(hit.args);
let inner = inlineCalls(callee, eng, new Set([...seen, hit.name]), `${where}>${hit.name}`);
formals.forEach((f, i) => {
const a = actuals[i];
if (a === undefined || f === a) return;
inner = inner.replace(new RegExp(`\\b${f}\\b`, "g"), `(${a})`);
});
text = text.slice(0, hit.start) + ` { ${inner} } ` + text.slice(hit.end);
}
fail(where, "helper inlining did not converge");
}
/** Unroll `for` loops with compile-time trip counts, brace-stripping bodies. */
function unrollLoops(body: string, eng: EngineFile, where: string): string {
let text = body;
for (let pass = 0; pass < 64; pass++) {
const idx = text.search(/\bfor\s*\(/);
if (idx < 0) return text;
const open = text.indexOf("(", idx);
const close = matchBracket(text, open);
const header = text.slice(open + 1, close);
// Body: either a braced block or a single statement.
let bodyStart = close + 1;
while (/\s/.test(text[bodyStart] ?? "")) bodyStart++;
let inner: string;
let bodyEnd: number;
if (text[bodyStart] === "{") {
const b = matchBracket(text, bodyStart);
inner = text.slice(bodyStart + 1, b);
bodyEnd = b + 1;
} else {
const semi = text.indexOf(";", bodyStart);
if (semi < 0) fail(where, "for-loop body has no terminator");
inner = text.slice(bodyStart, semi + 1);
bodyEnd = semi + 1;
}
let expansion = "";
const counted = header.match(
/^\s*(?:std::size_t|int|std::uint\d+_t|auto)\s+(\w+)\s*=\s*([^;]+);\s*\1\s*<\s*([^;]+);\s*\+\+\1\s*$/
);
const ranged = header.match(/^\s*(?:const\s+)?auto\s*[&*]?\s*\w+\s*:\s*(\w+)\s*$/);
if (counted) {
const v = counted[1]!;
const lo = evalInt(counted[2]!, eng.consts, where);
const hi = evalInt(counted[3]!, eng.consts, where);
for (let i = lo; i < hi; i++) {
expansion += ` ${inner.replace(new RegExp(`\\b${v}\\b`, "g"), `(${i})`)} `;
}
} else if (ranged) {
const n = eng.arrayLens.get(ranged[1]!);
if (n === undefined) fail(where, `range-for over ${ranged[1]} with unknown length`);
for (let i = 0; i < n; i++) expansion += ` ${inner} `;
} else {
fail(where, `unrecognised for-loop header ${JSON.stringify(header.trim())}`);
}
text = text.slice(0, idx) + expansion + text.slice(bodyEnd);
}
fail(where, "for-loop unrolling did not converge");
}
/**
* Remove `switch (voice_space_) { … }` — each voice space is analysed against
* its own dispatch target, so keeping the switch would merge all of them.
* Any OTHER switch is deliberately left in place: brace-flattening turns its
* arms into straight-line statements, and if two arms disagree about a param's
* curve that surfaces as a conflict rather than a silent pick.
*/
function dropVoiceSpaceSwitch(body: string, where: string): string {
let text = body;
for (;;) {
const m = text.match(/\bswitch\s*\(\s*voice_space_\s*\)/);
if (!m || m.index === undefined) return text;
const open = text.indexOf("(", m.index);
const close = matchBracket(text, open);
let braceOpen = close + 1;
while (/\s/.test(text[braceOpen] ?? "")) braceOpen++;
if (text[braceOpen] !== "{") fail(where, "switch (voice_space_) without a block");
text = text.slice(0, m.index) + " " + text.slice(matchBracket(text, braceOpen) + 1);
}
}
/**
* Extract `auto NAME = [&]() { const float X = params[i++]; return EXPR; };`
* lambdas (memlcelium's `sq()`), returning the curve each application yields.
* Any other lambda shape is a hard error.
*/
function extractCounterLambdas(body: string, where: string): { text: string; lambdas: Map<string, Curve> } {
const lambdas = new Map<string, Curve>();
let text = body;
for (;;) {
const m = text.match(/\bauto\s+(\w+)\s*=\s*\[[^\]]*\]\s*\(/);
if (!m || m.index === undefined) return { text, lambdas };
const name = m[1]!;
const parenOpen = text.indexOf("(", m.index + m[0].length - 1);
const parenClose = matchBracket(text, parenOpen);
let braceOpen = parenClose + 1;
while (/\s/.test(text[braceOpen] ?? "")) braceOpen++;
if (text[braceOpen] !== "{") fail(where, `lambda ${name} without a body`);
const braceClose = matchBracket(text, braceOpen);
const inner = text.slice(braceOpen + 1, braceClose).trim();
const shape = inner.match(
/^const\s+float\s+(\w+)\s*=\s*params\s*\[\s*i\+\+\s*\]\s*;\s*return\s+([^;]+);$/
);
if (!shape) {
fail(where, `lambda ${name} has an unrecognised body: ${JSON.stringify(inner)}`);
}
const v = shape[1]!;
const ret = shape[2]!.replace(/\s+/g, " ").trim();
let curve: Curve;
if (ret === `${v} * ${v}`) curve = "square";
else if (ret === `std::sqrt(${v})`) curve = "sqrt";
else if (ret === v) curve = "linear";
else fail(where, `lambda ${name} returns an unrecognised form: ${JSON.stringify(ret)}`);
lambdas.set(name, curve);
let end = braceClose + 1;
while (/[\s;]/.test(text[end] ?? "")) end++;
text = text.slice(0, m.index) + " " + text.slice(end);
}
}
/** Strip every remaining brace; loops are unrolled and switches dropped by now. */
function flattenBraces(text: string): string {
return text.replace(/[{}]/g, " ");
}
function splitStatements(text: string): string[] {
return text
.split(";")
.map((s) => s.replace(/\s+/g, " ").trim())
.filter((s) => s !== "");
}
// ---------------------------------------------------------------------------
// Ternary reduction + accessor classification
// ---------------------------------------------------------------------------
/** Collapse `cond ? a : b` where `cond` is compile-time constant. */
function reduceTernaries(stmt: string, consts: Map<string, number>): string {
let text = stmt;
for (let pass = 0; pass < 32; pass++) {
// Innermost `?` first: the one with no further `?` before its `:`.
const q = text.lastIndexOf("?");
if (q < 0) return text;
// Condition: scan left to the nearest unbalanced `(`, or `=`/`,`/start.
let depth = 0;
let condStart = 0;
for (let i = q - 1; i >= 0; i--) {
const c = text[i]!;
if (c === ")" || c === "]") depth++;
else if (c === "(" || c === "[") {
if (depth === 0) {
condStart = i + 1;
break;
}
depth--;
} else if (depth === 0 && (c === "=" || c === ",")) {
condStart = i + 1;
break;
}
}
// Matching `:` at the same nesting depth.
depth = 0;
let colon = -1;
for (let i = q + 1; i < text.length; i++) {
const c = text[i]!;
if (c === "(" || c === "[") depth++;
else if (c === ")" || c === "]") {
if (depth === 0) break;
depth--;
} else if (c === ":" && depth === 0 && text[i + 1] !== ":" && text[i - 1] !== ":") {
colon = i;
break;
}
}
if (colon < 0) return text;
// End of the false branch: unbalanced `)`/`]`/`,` or end of statement.
depth = 0;
let end = text.length;
for (let i = colon + 1; i < text.length; i++) {
const c = text[i]!;
if (c === "(" || c === "[") depth++;
else if (c === ")" || c === "]") {
if (depth === 0) {
end = i;
break;
}
depth--;
} else if (c === "," && depth === 0) {
end = i;
break;
}
}
const cond = text.slice(condStart, q);
const t = text.slice(q + 1, colon);
const f = text.slice(colon + 1, end);
const v = evalBool(cond, consts);
// When the guard is not constant, keep BOTH branches: an accessor that is
// squared in one and sqrt'd in the other then surfaces as a conflict.
const repl = v === null ? `( ${cond} ) * ( ${t} ) * ( ${f} )` : v ? `( ${t} )` : `( ${f} )`;
text = text.slice(0, condStart) + repl + text.slice(end);
}
return text;
}
/** Operand immediately to the left of `at` (balanced group, or a bare term). */
function leftOperand(text: string, at: number): { start: number; text: string } | null {
let i = at - 1;
while (i >= 0 && /\s/.test(text[i]!)) i--;
if (i < 0) return null;
if (text[i] === ")" || text[i] === "]") {
// Walk back over the balanced group, then over any leading identifier
// (so `p[3]` and `std::sqrt(x)` come back whole).
let depth = 0;
let j = i;
const open = text[i] === ")" ? "(" : "[";
for (; j >= 0; j--) {
if (text[j] === text[i]) depth++;
else if (text[j] === open) {
depth--;
if (depth === 0) break;
}
}
if (j < 0) return null;
let k = j - 1;
while (k >= 0 && /[\w:]/.test(text[k]!)) k--;
return { start: k + 1, text: text.slice(k + 1, i + 1) };
}
let j = i;
while (j >= 0 && /[\w.:]/.test(text[j]!)) j--;
if (j === i) return null;
return { start: j + 1, text: text.slice(j + 1, i + 1) };
}
/** Operand immediately to the right of `at`. */
function rightOperand(text: string, at: number): { end: number; text: string } | null {
let i = at + 1;
while (i < text.length && /\s/.test(text[i]!)) i++;
if (i >= text.length) return null;
let j = i;
while (j < text.length && /[\w:]/.test(text[j]!)) j++;
if (j < text.length && (text[j] === "(" || text[j] === "[")) {
const close = matchBracket(text, j);
return { end: close + 1, text: text.slice(i, close + 1) };
}
if (j === i) return null;
return { end: j, text: text.slice(i, j) };
}
/** True when `text` is exactly one param-slot read, modulo parens/whitespace. */
function isBareAccessor(text: string): boolean {
let t = text.trim();
while (t.startsWith("(") && matchBracket(t, 0) === t.length - 1) t = t.slice(1, -1).trim();
const m = t.match(new RegExp(`^(?:${ACCESSORS.join("|")})\\s*\\[`));
if (!m) return false;
return matchBracket(t, t.indexOf("[")) === t.length - 1;
}
/** Every accessor index appearing in `text`. */
function accessorIndices(text: string, consts: Map<string, number>, where: string): number[] {
const out: number[] = [];
const re = new RegExp(`\\b(${ACCESSORS.join("|")})\\s*\\[`, "g");
let m: RegExpExecArray | null;
while ((m = re.exec(text))) {
const open = m.index + m[0].length - 1;
const close = matchBracket(text, open);
out.push(evalInt(text.slice(open + 1, close), consts, where));
re.lastIndex = close;
}
return out;
}
/**
* Classify every accessor occurrence in one expression.
*
* `sqrt` is recognised as `std::sqrt(<expr containing exactly the accessors>)`;
* `square` as `X * X` for textually identical operands containing accessors.
* Recognised occurrences are blanked so they are not re-counted as linear.
*/
function classifyExpr(expr: string, consts: Map<string, number>, where: string): Map<number, Curve> {
const found = new Map<number, Curve>();
const note = (idx: number, c: Curve) => {
const prev = found.get(idx);
if (prev !== undefined && prev !== c) {
fail(where, `param ${idx} is both ${prev} and ${c} in one expression: ${expr}`);
}
found.set(idx, c);
};
let text = expr;
// 1. std::sqrt(...)
for (;;) {
const m = text.match(/\bstd::sqrt\s*\(/);
if (!m || m.index === undefined) break;
const open = text.indexOf("(", m.index);
const close = matchBracket(text, open);
const inner = text.slice(open + 1, close);
const idxs = accessorIndices(inner, consts, where);
if (idxs.length > 0) {
// A sqrt over a compound expression is not a per-param sqrt curve — the
// same reasoning as the self-product rule below. None exists today, so
// this is a hard error rather than a silent demotion; if one ever
// appears the author must decide what the declaration should say.
if (!isBareAccessor(inner)) fail(where, `std::sqrt over a compound expression: ${inner}`);
note(idxs[0]!, "sqrt");
text = text.slice(0, m.index) + " __X__ " + text.slice(close + 1);
} else {
text = text.slice(0, m.index) + " __X__ " + text.slice(close + 1);
}
}
// 2. X * X, where X is one bare param slot (possibly parenthesised because
// it arrived via an alias). A self-product over a COMPOUND expression —
// e.g. paf_synth Elderstar's `factor * factor` where
// `factor = 1.f + (p[17] + p[27] * 0.2f)` — is deliberately NOT a
// per-param square: no single slot is multiplied by itself, and the
// `Curve` enum has no way to say "this slot is one term inside a squared
// sum". Those slots fall through to `linear`, which is what the schemas
// declare. The rule is explicit, not a silent fallback.
for (let from = 0; ; ) {
let hit = false;
for (let i = from; i < text.length; i++) {
if (text[i] !== "*") continue;
const l = leftOperand(text, i);
const r = rightOperand(text, i);
if (!l || !r) continue;
if (l.text.replace(/\s+/g, "") !== r.text.replace(/\s+/g, "")) continue;
const idxs = accessorIndices(l.text, consts, where);
if (idxs.length === 0) continue;
if (idxs.length !== 1 || !isBareAccessor(l.text)) {
from = i + 1;
hit = true;
break;
}
note(idxs[0]!, "square");
text = text.slice(0, l.start) + " __X__ " + text.slice(r.end);
from = 0;
hit = true;
break;
}
if (!hit) break;
}
// 3. anything left is linear
for (const idx of accessorIndices(text, consts, where)) note(idx, "linear");
return found;
}
// ---------------------------------------------------------------------------
// Body analysis
// ---------------------------------------------------------------------------
interface Analysis {
/** assignment target -> curves contributed by the latest write to it. */
targets: Map<string, Map<number, Curve>>;
}
const ALIAS_DECL = /^(?:static\s+)?const\s+(?:float|std::size_t|int|auto)\s+(\w+)\s*=\s*(.*)$/;
const COUNTER_DECL = /^(?:std::size_t|int)\s+(\w+)\s*=\s*(\d+)u?$/;
function analyseBody(
rawBody: string,
eng: EngineFile,
where: string,
state: Analysis,
stmtSeq: { n: number }
): void {
let text = inlineCalls(rawBody, eng, new Set(), where);
text = dropVoiceSpaceSwitch(text, where);
text = unrollLoops(text, eng, where);
const { text: noLambda, lambdas } = extractCounterLambdas(text, where);
text = flattenBraces(noLambda);
const aliases = new Map<string, { rhs: string; used: boolean }>();
let counterName: string | null = null;
let counter = 0;
for (const rawStmt of splitStatements(text)) {
let stmt = rawStmt;
// Sequential dialect: `params[i++]` and counter lambdas consume the next
// slot, left to right, and are rewritten into the ordinary indexed form.
if (counterName !== null) {
const seqRe = new RegExp(
`params\\s*\\[\\s*${counterName}\\+\\+\\s*\\]|\\b(${[...lambdas.keys()].join("|") || "\\u0000"})\\s*\\(\\s*\\)`,
"g"
);
stmt = stmt.replace(seqRe, (whole, lam?: string) => {
const idx = counter++;
if (lam) {
const c = lambdas.get(lam)!;
if (c === "square") return `params[${idx}] * params[${idx}]`;
if (c === "sqrt") return `std::sqrt(params[${idx}])`;
return `params[${idx}]`;
}
return `params[${idx}]`;
});
}
if (/\+\+\s*\]/.test(stmt)) {
fail(where, `post-increment index with no active counter: ${stmt}`);
}
const counterM = stmt.match(COUNTER_DECL);
if (counterM && !ACCESSOR_RE.test(stmt)) {
counterName = counterM[1]!;
counter = Number(counterM[2]!);
continue;
}
// Substitute live aliases (longest name first so `p1v` beats `p`).
for (const [name, a] of [...aliases].sort((x, y) => y[0].length - x[0].length)) {
const re = new RegExp(`\\b${name}\\b`, "g");
if (re.test(stmt)) {
// Not a use if this statement redeclares it.
const decl = stmt.match(ALIAS_DECL);
if (decl && decl[1] === name && !new RegExp(`\\b${name}\\b`).test(decl[2]!)) continue;
stmt = stmt.replace(re, `(${a.rhs})`);
a.used = true;
}
}
const aliasM = stmt.match(ALIAS_DECL);
if (aliasM) {
const name = aliasM[1]!;
const prev = aliases.get(name);
if (prev && !prev.used && ACCESSOR_RE.test(prev.rhs)) {
fail(where, `alias ${name} carrying a param was shadowed before use`);
}
aliases.set(name, { rhs: aliasM[2]!, used: false });
continue;
}
if (!ACCESSOR_RE.test(stmt)) continue;
const reduced = reduceTernaries(stmt, eng.consts);
const eq = topLevelAssign(reduced);
if (eq === null) {
// A call statement or return: unique key, never overwritten.
state.targets.set(`${where}#${stmtSeq.n++}`, classifyExpr(reduced, eng.consts, where));
continue;
}
const lhs = reduced.slice(0, eq).trim();
const rhs = reduced.slice(eq + 1);
// Pure copy between two aliases of the NN vector carries no curve.
if (isPureCopy(lhs, rhs, eng.consts, where)) continue;
if (ACCESSOR_RE.test(lhs)) {
fail(where, `assignment INTO the param vector with arithmetic: ${reduced}`);
}
state.targets.set(lhs.replace(/^(?:const\s+)?(?:float|auto)\s+/, ""), classifyExpr(rhs, eng.consts, where));
}
for (const [name, a] of aliases) {
if (!a.used && ACCESSOR_RE.test(a.rhs)) {
fail(where, `alias ${name} carrying a param was never used`);
}
}
}
/** Index of a top-level `=` that is an assignment (not ==, +=, <=, …). */
function topLevelAssign(stmt: string): number | null {
let depth = 0;
for (let i = 0; i < stmt.length; i++) {
const c = stmt[i]!;
if (c === "(" || c === "[") depth++;
else if (c === ")" || c === "]") depth--;
else if (c === "=" && depth === 0) {
if (stmt[i + 1] === "=") return null;
if ("=!<>+-*/&|%".includes(stmt[i - 1] ?? "")) return null;
return i;
}
}
return null;
}
function isPureCopy(lhs: string, rhs: string, consts: Map<string, number>, where: string): boolean {
const l = lhs.trim();
const r = rhs.trim();
if (!ACCESSOR_RE.test(l) || !ACCESSOR_RE.test(r)) return false;
const li = accessorIndices(l, consts, where);
const ri = accessorIndices(r, consts, where);
if (li.length !== 1 || ri.length !== 1 || li[0] !== ri[0]) return false;
// RHS must be nothing but the accessor.
return new RegExp(`^(?:${ACCESSORS.join("|")})\\s*\\[[^\\]]*\\]$`).test(r);
}
// ---------------------------------------------------------------------------
// Engine-level extraction
// ---------------------------------------------------------------------------
function parseVoiceSpaces(eng: EngineFile): { enumNames: string[]; displayNames: string[] } | null {
const em = eng.src.match(/enum\s+class\s+VoiceSpace\s*:[^{]*\{/);
if (!em || em.index === undefined) return null;
const open = eng.src.indexOf("{", em.index);
const inner = eng.src.slice(open + 1, matchBracket(eng.src, open));
const enumNames = inner
.split(",")
.map((s) => s.split("=")[0]!.trim())
.filter((s) => s !== "" && s !== "Count");
const nm = eng.src.match(/kVoiceSpaceNames\s*=\s*\{/);
if (!nm || nm.index === undefined) fail(eng.path, "VoiceSpace enum without kVoiceSpaceNames");
const nOpen = eng.src.indexOf("{", nm.index);
const nInner = eng.src.slice(nOpen + 1, matchBracket(eng.src, nOpen));
const displayNames = [...nInner.matchAll(/"((?:[^"\\]|\\.)*)"/g)].map((m) => m[1]!);
if (displayNames.length !== enumNames.length) {
fail(eng.path, `VoiceSpace enum has ${enumNames.length} entries but kVoiceSpaceNames has ${displayNames.length}`);
}
return { enumNames, displayNames };
}
/** enum name -> the `apply_*` function the dispatch switch routes it to. */
function parseDispatch(eng: EngineFile, enumNames: string[]): Map<string, string> {
const out = new Map<string, string>();
const re = /case\s+VoiceSpace::(\w+)\s*:\s*(?:\{\s*)?(\w+)\s*\(/g;
let m: RegExpExecArray | null;
while ((m = re.exec(eng.src))) {
if (m[1] === "Count") continue;
out.set(m[1]!, m[2]!);
}
for (const n of enumNames) {
if (!out.has(n)) fail(eng.path, `VoiceSpace::${n} has no dispatch case`);
}
return out;
}
function collectCurves(state: Analysis, nParams: number, where: string): Curve[] {
const merged = new Map<number, Curve>();
for (const contrib of state.targets.values()) {
for (const [idx, c] of contrib) {
if (idx < 0 || idx >= nParams) fail(where, `param index ${idx} out of range (0..${nParams - 1})`);
const prev = merged.get(idx);
if (prev !== undefined && prev !== c) {
fail(where, `param ${idx} is mapped both ${prev} and ${c} within one voice space`);
}
merged.set(idx, c);
}
}
const row: Curve[] = new Array(nParams).fill("linear");
for (const [idx, c] of merged) row[idx] = c;
return row;
}
export function auditEngine(path: string): EngineCurves {
const eng = loadEngine(path);
const idm = eng.src.match(/engine_id\(\)\s*noexcept\s*\{\s*return\s*"([^"]+)"/);
if (!idm) fail(path, "no engine_id()");
const engineId = idm[1]!;
const nParams = eng.consts.get("kNParams") ?? 0;
const vs = parseVoiceSpaces(eng);
const setParams = functionBody(eng.src, "set_params");
const process = functionBody(eng.src, "process");
if (nParams === 0) {
return { engineId, file: path, nParams: 0, voiceSpaceNames: null, curves: [[]] };
}
if (vs === null) {
const state: Analysis = { targets: new Map() };
const seq = { n: 0 };
if (setParams) analyseBody(setParams, eng, `${engineId}:set_params`, state, seq);
if (process) analyseBody(process, eng, `${engineId}:process`, state, seq);
return {
engineId,
file: path,
nParams,
voiceSpaceNames: null,
curves: [collectCurves(state, nParams, engineId)],
};
}
const dispatch = parseDispatch(eng, vs.enumNames);
const curves: Curve[][] = [];
for (const name of vs.enumNames) {
const fn = dispatch.get(name)!;
const body = functionBody(eng.src, fn);
if (body === null) fail(path, `dispatch target ${fn}() has no body`);
const state: Analysis = { targets: new Map() };
const seq = { n: 0 };
const where = `${engineId}:${name}`;
if (setParams) analyseBody(setParams, eng, `${where}/set_params`, state, seq);
analyseBody(body, eng, where, state, seq);
curves.push(collectCurves(state, nParams, where));
}
return { engineId, file: path, nParams, voiceSpaceNames: vs.displayNames, curves };
}
export function auditAllEngines(enginesDir: string): Map<string, EngineCurves> {
const out = new Map<string, EngineCurves>();
for (const f of readdirSync(enginesDir).filter((f) => f.endsWith(".hpp")).sort()) {
const res = auditEngine(join(enginesDir, f));
if (out.has(res.engineId)) fail(enginesDir, `duplicate engine_id ${res.engineId}`);
out.set(res.engineId, res);
}
return out;
}

View file

@ -14,14 +14,18 @@
* by name, and selects which the ML drives over MIDI CC. Idempotent: regenerating the
* same schemas yields byte-identical output. Exits non-zero on validation failure.
*
* Deliberately separate from codegen/generate.ts (the mode-schema pipeline) so it cannot
* disturb the mode golden test.
* Deliberately a separate DRIVER from codegen/generate.ts (the mode-schema
* pipeline): different schema dir, different meta-schema, different output
* dirs, and generate.ts's golden test must not depend on MIDI-device schema
* state. That does not require duplicate helpers, though — the small
* string/fs helpers both drivers need live once in ./lib.ts (ST12).
*/
import { readFileSync, writeFileSync, readdirSync, mkdirSync, existsSync } from "node:fs";
import { writeFileSync, readdirSync, existsSync } from "node:fs";
import { join, dirname, resolve } from "node:path";
import { fileURLToPath } from "node:url";
import Ajv2020, { type AnySchemaObject } from "ajv/dist/2020.js";
import { ensureDir, readJSON, toPascalCase, cppStringLit, tsStringLit } from "./lib.ts";
// ----- Types ----------------------------------------------------------------
@ -62,22 +66,8 @@ const CPP_OUT_DIR = join(REPO_ROOT, "nisps", "midi", "generated");
const TS_OUT_DIR = join(REPO_ROOT, "manifold", "src", "midi-devices", "generated");
// ----- Helpers --------------------------------------------------------------
function ensureDir(d: string): void {
if (!existsSync(d)) mkdirSync(d, { recursive: true });
}
function readJSON<T>(path: string): T {
return JSON.parse(readFileSync(path, "utf8")) as T;
}
function toPascalCase(snake: string): string {
return snake.split("_").map(s => (s.length === 0 ? s : s[0].toUpperCase() + s.slice(1))).join("");
}
function cppStringLit(s: string): string {
return '"' + s.replace(/\\/g, "\\\\").replace(/"/g, '\\"') + '"';
}
function tsStringLit(s: string): string {
return "'" + s.replace(/\\/g, "\\\\").replace(/'/g, "\\'") + "'";
}
// ensureDir/readJSON/toPascalCase/cppStringLit/tsStringLit now live in
// ./lib.ts (ST12, shared with generate.ts).
const BANNER = "// AUTOGENERATED — do not edit. Source: schemas/midi_devices/*.json. " +
"Run `bun run codegen/generate-midi-devices.ts` to regenerate.";

View file

@ -7,23 +7,89 @@
*
* Writes: nisps/modes/generated/<mode_id>_schema.hpp
* nisps/modes/generated/schema_types.hpp
* playground/src/modes/generated/<mode_id>_schema.ts
* playground/src/modes/generated/types.ts
* playground/src/modes/generated/index.ts
* manifold/src/modes/generated/<mode_id>_schema.ts
* manifold/src/modes/generated/types.ts
* manifold/src/modes/generated/index.ts
*
* (The TS target moved playground → manifold at P5 of
* docs/specs/plans/one-core-engine-refactor.md.)
*
* Per-mode C++ headers also emit (simplification 2026-07, S1/S5/S6/S25):
* - `k<Mode>Schema` — the `nisps::ParamSchema` aggregate; each mode's
* `param_schema()` becomes a one-line `return generated::k<Mode>Schema;`
* instead of hand-assembling the same 12-field struct.
* - `<Mode>MLP` — the mode's `nisps::ml::MLP<...>` net-shape alias,
* built from the constants above instead of a second hand-typed copy of
* the dims in nisps/modes/*.hpp.
* The TS `index.ts` also emits `ALL_MODE_SCHEMAS` (every mode schema, in
* generation order) so manifold's mode catalogue no longer hand-imports each
* schema by name — see manifold/src/console/model.ts's `SCHEMA_MODES` overlay.
*
* Idempotent: regenerating the same schemas yields byte-identical output.
* Exits non-zero on validation failure.
*/
import { readFileSync, writeFileSync, readdirSync, mkdirSync, existsSync } from "node:fs";
import { writeFileSync, readdirSync, existsSync } from "node:fs";
import { join, dirname, resolve, basename } from "node:path";
import { fileURLToPath } from "node:url";
import Ajv2020, { type AnySchemaObject } from "ajv/dist/2020.js";
import { ensureDir, readJSON, toPascalCase, cppStringLit, tsStringLit } from "./lib.ts";
// ----- Types ----------------------------------------------------------------
type Curve = "linear" | "exp" | "log" | "square" | "sqrt" | "sigmoid" | "cubic";
/**
* A voice space is either a bare name (it applies every param's default
* `curve`) or a name plus the slots where it DEVIATES from those defaults.
*
* The curve is a property of (param x voice space), not of the mode: the same
* NN slot is squared by one voice space and passed through linearly by
* another. The mode-wide `params[].curve` remains the default; this is the
* delta. Application stays where it has always been — inside the engine —
* so declaring it here changes no emitted numeric value.
*/
type VoiceSpaceDecl = string | { name: string; curve_overrides: Record<string, Curve>; _note?: string };
/** Flat (voice space, param) -> curve deviation, as emitted to both languages. */
interface CurveOverrideRow {
voiceSpace: number;
param: number;
curve: Curve;
}
function voiceSpaceName(v: VoiceSpaceDecl): string {
return typeof v === "string" ? v : v.name;
}
/**
* Resolve a mode's declared deltas into the flat table both targets emit,
* sorted by (voice space, param) so output is order-independent.
* Throws on an override that names an unknown param or restates the default.
*/
function curveOverrideRows(schema: ModeSchema, source: string): CurveOverrideRow[] {
const paramIndex = new Map(schema.params.map((p, i) => [p.name, i] as const));
const rows: CurveOverrideRow[] = [];
schema.voice_spaces.forEach((vs, vi) => {
if (typeof vs === "string") return;
for (const [name, curve] of Object.entries(vs.curve_overrides)) {
const pi = paramIndex.get(name);
if (pi === undefined) {
throw new Error(`${source}: voice space ${vs.name} overrides unknown param ${JSON.stringify(name)}`);
}
if (schema.params[pi]!.curve === curve) {
throw new Error(
`${source}: voice space ${vs.name} restates param ${name}'s default curve ` +
`(${curve}) — declare only deviations`
);
}
rows.push({ voiceSpace: vi, param: pi, curve });
}
});
rows.sort((a, b) => a.voiceSpace - b.voiceSpace || a.param - b.param);
return rows;
}
interface ModeSchema {
$schema?: string;
_note?: string;
@ -35,8 +101,6 @@ interface ModeSchema {
hidden_layers: number[];
output_size: number;
default_spread: number;
default_learning_rate: number;
default_max_iterations: number;
};
params: Array<{
name: string;
@ -48,7 +112,7 @@ interface ModeSchema {
group: string;
_note?: string;
}>;
voice_spaces: string[];
voice_spaces: VoiceSpaceDecl[];
ui: {
primary_input: "xy_pad" | "joystick" | "sliders" | "audio_in" | "midi_in" | "none";
show_voice_space_selector: boolean;
@ -56,6 +120,24 @@ interface ModeSchema {
};
}
/**
* The ONE global training-hyperparameter default (schemas/ml_defaults.json,
* validated against schemas/ml_defaults.schema.json) — NOT per-mode, unlike
* everything else in ModeSchema. See S26 (docs/specs/recon/
* simplification-audit-2026-07.md): learning_rate/max_iterations/min_error
* used to be duplicated per-mode (identically, unread at runtime) plus
* hardcoded separately in nisps/ml/mlp.hpp, manifold/src/engine/wasm-iml.ts,
* and vcv/src/iml.hpp. Now declared once here and consumed by
* `nisps::ml::MLPCore`'s `TrainConfig` default member initialisers.
*/
interface MlTrainDefaults {
$schema?: string;
_note?: string;
learning_rate: number;
max_iterations: number;
min_error: number;
}
// ----- Path resolution ------------------------------------------------------
const __dirname = dirname(fileURLToPath(import.meta.url));
@ -63,31 +145,20 @@ const REPO_ROOT = resolve(__dirname, "..");
const SCHEMAS_DIR = join(REPO_ROOT, "schemas");
const MODES_DIR = join(SCHEMAS_DIR, "modes");
const META_SCHEMA_PATH = join(SCHEMAS_DIR, "schema.json");
const ML_DEFAULTS_PATH = join(SCHEMAS_DIR, "ml_defaults.json");
const ML_DEFAULTS_SCHEMA_PATH = join(SCHEMAS_DIR, "ml_defaults.schema.json");
const CPP_OUT_DIR = join(REPO_ROOT, "nisps", "modes", "generated");
const TS_OUT_DIR = join(REPO_ROOT, "playground", "src", "modes", "generated");
// The training defaults are an ML fact, not a mode fact, and `nisps/ml` sits
// BELOW `nisps/modes` in the layering MAP.md documents — emitting them into
// modes/generated/ would make mlp.hpp include upward. They get their own
// output dir on the C++ side. The TS side has no such layering to respect, so
// it keeps everything generated under one directory.
const CPP_ML_OUT_DIR = join(REPO_ROOT, "nisps", "ml", "generated");
const TS_OUT_DIR = join(REPO_ROOT, "manifold", "src", "modes", "generated");
// ----- Helpers --------------------------------------------------------------
function ensureDir(d: string): void {
if (!existsSync(d)) {
mkdirSync(d, { recursive: true });
}
}
function readJSON<T>(path: string): T {
return JSON.parse(readFileSync(path, "utf8")) as T;
}
/**
* Convert snake_case to PascalCase.
* "paf_synth" -> "PafSynth", "memlcelium" -> "Memlcelium"
*/
function toPascalCase(snake: string): string {
return snake
.split("_")
.map(s => s.length === 0 ? s : s[0].toUpperCase() + s.slice(1))
.join("");
}
// ensureDir/readJSON/toPascalCase/cppStringLit/tsStringLit now live in
// ./lib.ts (ST12, shared with generate-midi-devices.ts).
/**
* Convert mode_id to UPPER_SNAKE for #define guards.
@ -96,21 +167,6 @@ function toUpperSnake(snake: string): string {
return snake.toUpperCase();
}
/**
* Escape a string literal for embedding into a C++ string.
* We expect ASCII identifiers + label text only; quote and backslash are escaped.
*/
function cppStringLit(s: string): string {
return '"' + s.replace(/\\/g, "\\\\").replace(/"/g, '\\"') + '"';
}
/**
* Escape a string for a TS single-line single-quoted literal.
*/
function tsStringLit(s: string): string {
return "'" + s.replace(/\\/g, "\\\\").replace(/'/g, "\\'") + "'";
}
/**
* Format a float literal for C++ ensuring `.f` suffix and explicit decimal point.
* Required by the perf contract (architecture §3.3).
@ -158,11 +214,18 @@ function emitSchemaTypesHpp(): string {
"//",
"// `Curve` is the authoritative enum from nisps/core/math.hpp; we re-export it",
"// into this namespace so generated headers can refer to plain `Curve::linear`.",
"//",
"// `ParamSchema` lives in the top-level `nisps` namespace, not",
"// `nisps::modes::generated`: nisps/core/concepts.hpp forward-declares",
"// `nisps::ParamSchema` and requires `T::param_schema()` to return",
"// `const nisps::ParamSchema&`, so the definition has to match that",
"// forward declaration exactly (S5, one-core simplification 2026-07).",
"#ifndef NISPS_GENERATED_SCHEMA_TYPES_HPP",
"#define NISPS_GENERATED_SCHEMA_TYPES_HPP",
"",
"#include <array>",
"#include <cstddef>",
"#include <span>",
"#include <string_view>",
"",
"#include \"../../core/math.hpp\"",
@ -181,12 +244,21 @@ function emitSchemaTypesHpp(): string {
" std::string_view group;",
"};",
"",
"// One (voice space, param) slot where the engine applies a curve OTHER",
"// than that param's default. The curve is a property of the pair, not of",
"// the mode — apply_ssl4k() squares slot 11 where apply_neve66() does not.",
"// DESCRIPTIVE: the engine's voice space is still the only place a curve",
"// is applied, and it is applied exactly once.",
"struct CurveOverride {",
" std::size_t voice_space;",
" std::size_t param;",
" Curve curve;",
"};",
"",
"struct MLConfig {",
" std::size_t input_size;",
" std::size_t output_size;",
" float default_spread;",
" float default_learning_rate;",
" std::size_t default_max_iterations;",
"};",
"",
"enum class PrimaryInput : unsigned char {",
@ -206,6 +278,40 @@ function emitSchemaTypesHpp(): string {
"",
"} // namespace nisps::modes::generated",
"",
"namespace nisps {",
"",
"// View-style aggregate satisfying the `nisps::Mode` concept's",
"// `param_schema()` requirement. All members are spans/views into",
"// compile-time generated arrays; codegen emits one",
"// `inline constexpr ParamSchema k<Mode>Schema` per mode (see the",
"// per-mode <mode_id>_schema.hpp in this directory).",
"struct ParamSchema {",
" std::string_view mode_id;",
" std::string_view engine_id;",
" std::span<const std::string_view> input_channels;",
" std::size_t input_size;",
" std::span<const std::size_t> hidden_layers;",
" std::size_t output_size;",
" float default_spread;",
" std::span<const ::nisps::modes::generated::Param> params;",
" std::span<const std::string_view> voice_spaces;",
" std::span<const ::nisps::modes::generated::CurveOverride> curve_overrides;",
" ::nisps::modes::generated::UIConfig ui;",
"};",
"",
"// The curve voice space `vs` applies to output slot `param`: the param's",
"// default unless this mode declares a deviation for that voice space.",
"// Linear scan — the table has tens of rows and this is not a hot path.",
"constexpr ::nisps::Curve effective_curve(const ParamSchema& s, std::size_t vs,",
" std::size_t param) noexcept {",
" for (const auto& o : s.curve_overrides) {",
" if (o.voice_space == vs && o.param == param) return o.curve;",
" }",
" return s.params[param].curve;",
"}",
"",
"} // namespace nisps",
"",
"#endif // NISPS_GENERATED_SCHEMA_TYPES_HPP",
"",
].join("\n");
@ -243,14 +349,26 @@ function emitSharedTsTypes(): string {
" readonly group: string;",
"}",
"",
"/**",
" * One (voice space, param) slot where the engine applies a curve OTHER than",
" * that param's default. The curve is a property of the pair, not of the mode",
" * — apply_ssl4k() squares slot 11 where apply_neve66() does not. DESCRIPTIVE:",
" * the engine's voice space is still the only place a curve is applied, and it",
" * is applied exactly once. Indices match `ModeSchema.voice_spaces` /",
" * `ModeSchema.params`.",
" */",
"export interface CurveOverride {",
" readonly voice_space: number;",
" readonly param: number;",
" readonly curve: Curve;",
"}",
"",
"export interface MLConfig {",
" readonly input_channels: readonly string[];",
" readonly input_size: number;",
" readonly hidden_layers: readonly number[];",
" readonly output_size: number;",
" readonly default_spread: number;",
" readonly default_learning_rate: number;",
" readonly default_max_iterations: number;",
"}",
"",
"export interface UIConfig {",
@ -265,9 +383,25 @@ function emitSharedTsTypes(): string {
" readonly ml: MLConfig;",
" readonly params: readonly Param[];",
" readonly voice_spaces: readonly string[];",
" readonly curve_overrides: readonly CurveOverride[];",
" readonly ui: UIConfig;",
"}",
"",
"/**",
" * The curve voice space `voiceSpace` applies to output slot `param`: the",
" * param's default unless this mode declares a deviation for that voice space.",
" */",
"export function effectiveCurve(",
" schema: ModeSchema,",
" voiceSpace: number,",
" param: number,",
"): Curve {",
" for (const o of schema.curve_overrides) {",
" if (o.voice_space === voiceSpace && o.param === param) return o.curve;",
" }",
" return schema.params[param]!.curve;",
"}",
"",
].join("\n");
}
@ -283,6 +417,7 @@ function emitModeHpp(schema: ModeSchema, sourceFile: string): string {
lines.push(`#define ${guard}`);
lines.push("");
lines.push("#include \"schema_types.hpp\"");
lines.push("#include \"../../ml/mlp.hpp\"");
lines.push("");
lines.push("namespace nisps::modes::generated {");
lines.push("");
@ -312,8 +447,6 @@ function emitModeHpp(schema: ModeSchema, sourceFile: string): string {
lines.push(` ${schema.ml.input_size}u,`);
lines.push(` ${schema.ml.output_size}u,`);
lines.push(` ${cppFloatLit(schema.ml.default_spread)},`);
lines.push(` ${cppFloatLit(schema.ml.default_learning_rate)},`);
lines.push(` ${schema.ml.default_max_iterations}u,`);
lines.push("};");
lines.push("");
@ -339,7 +472,7 @@ function emitModeHpp(schema: ModeSchema, sourceFile: string): string {
if (schema.voice_spaces.length > 0) {
lines.push(`inline constexpr std::array<std::string_view, ${constName}VoiceSpaceCount> ${constName}VoiceSpaces = {{`);
for (const v of schema.voice_spaces) {
lines.push(` ${cppStringLit(v)},`);
lines.push(` ${cppStringLit(voiceSpaceName(v))},`);
}
lines.push("}};");
} else {
@ -348,6 +481,25 @@ function emitModeHpp(schema: ModeSchema, sourceFile: string): string {
}
lines.push("");
// Per-voice-space curve deviations (see CurveOverride in schema_types.hpp).
// Only the deltas from `params[].curve`; resolve with nisps::effective_curve.
const overrides = curveOverrideRows(schema, sourceFile);
if (overrides.length > 0) {
lines.push(
`inline constexpr std::array<CurveOverride, ${overrides.length}u> ${constName}CurveOverrides = {{`
);
for (const o of overrides) {
lines.push(
` CurveOverride{${o.voiceSpace}u, ${o.param}u, ${cppCurveEnum(o.curve)}},` +
` // ${voiceSpaceName(schema.voice_spaces[o.voiceSpace]!)}.${schema.params[o.param]!.name}`
);
}
lines.push("}};");
} else {
lines.push(`inline constexpr std::array<CurveOverride, 0> ${constName}CurveOverrides = {};`);
}
lines.push("");
// UI
let primary: string;
switch (schema.ui.primary_input) {
@ -365,6 +517,34 @@ function emitModeHpp(schema: ModeSchema, sourceFile: string): string {
lines.push("};");
lines.push("");
// Net-shape alias (S6/S25): the mode's MLP<> template args, built from the
// constants above rather than hand-typed a second time in nisps/modes/*.hpp.
const modeMlpName = `${toPascalCase(schema.mode_id)}MLP`;
lines.push(
`using ${modeMlpName} = ::nisps::ml::MLP<` +
`${constName}MLConfig.input_size, ` +
`${constName}HiddenLayers[0], ${constName}HiddenLayers[1], ${constName}HiddenLayers[2], ` +
`${constName}MLConfig.output_size>;`
);
lines.push("");
// ParamSchema aggregate (S5): the one-line `param_schema()` body every mode
// used to hand-assemble as a private `kSchema` positional-init block.
lines.push(`inline constexpr ::nisps::ParamSchema ${constName}Schema = {`);
lines.push(` ${constName}ModeId,`);
lines.push(` ${constName}EngineId,`);
lines.push(` std::span<const std::string_view>(${constName}InputChannels),`);
lines.push(` ${constName}MLConfig.input_size,`);
lines.push(` std::span<const std::size_t>(${constName}HiddenLayers),`);
lines.push(` ${constName}MLConfig.output_size,`);
lines.push(` ${constName}MLConfig.default_spread,`);
lines.push(` std::span<const Param>(${constName}Params),`);
lines.push(` std::span<const std::string_view>(${constName}VoiceSpaces),`);
lines.push(` std::span<const CurveOverride>(${constName}CurveOverrides),`);
lines.push(` ${constName}UI,`);
lines.push("};");
lines.push("");
lines.push("} // namespace nisps::modes::generated");
lines.push("");
lines.push(`#endif // ${guard}`);
@ -410,8 +590,6 @@ function emitModeTs(schema: ModeSchema, sourceFile: string): string {
lines.push(" ],");
lines.push(` output_size: ${schema.ml.output_size},`);
lines.push(` default_spread: ${schema.ml.default_spread},`);
lines.push(` default_learning_rate: ${schema.ml.default_learning_rate},`);
lines.push(` default_max_iterations: ${schema.ml.default_max_iterations},`);
lines.push(" },");
lines.push(" params: [");
for (const p of schema.params) {
@ -431,7 +609,22 @@ function emitModeTs(schema: ModeSchema, sourceFile: string): string {
} else {
lines.push(" voice_spaces: [");
for (const v of schema.voice_spaces) {
lines.push(` ${tsStringLit(v)},`);
lines.push(` ${tsStringLit(voiceSpaceName(v))},`);
}
lines.push(" ],");
}
// Per-voice-space curve deviations; resolve with effectiveCurve() from
// ./types. Only the deltas from params[].curve are listed.
const overrides = curveOverrideRows(schema, sourceFile);
if (overrides.length === 0) {
lines.push(" curve_overrides: [],");
} else {
lines.push(" curve_overrides: [");
for (const o of overrides) {
lines.push(
` { voice_space: ${o.voiceSpace}, param: ${o.param}, curve: ${tsStringLit(o.curve)} },` +
` // ${voiceSpaceName(schema.voice_spaces[o.voiceSpace]!)}.${schema.params[o.param]!.name}`
);
}
lines.push(" ],");
}
@ -451,14 +644,99 @@ function emitTsIndex(modeIds: string[]): string {
lines.push("// AUTOGENERATED — do not edit. Run `bun run codegen/generate.ts` to regenerate.");
lines.push("// Re-exports every generated mode schema.");
lines.push("");
lines.push("import type { ModeSchema } from './types';");
for (const id of modeIds) {
lines.push(`import { ${toPascalCase(id)}Schema } from './${id}_schema';`);
}
lines.push("");
lines.push("export * from './types';");
for (const id of modeIds) {
lines.push(`export * from './${id}_schema';`);
}
lines.push("");
// Mechanically-derived mode-identity registry (S1): every generated mode
// schema, in generation (mode_id-sorted) order. NOT display order — that
// ordering is hand-curated overlay truth (manifold/src/console/model.ts's
// `SCHEMA_MODES`).
lines.push(
"/** Every generated mode schema, mode_id-sorted. Mechanically-derived mode-identity truth. */"
);
lines.push("export const ALL_MODE_SCHEMAS: readonly ModeSchema[] = [");
for (const id of modeIds) {
lines.push(` ${toPascalCase(id)}Schema,`);
}
lines.push("];");
lines.push("");
return lines.join("\n");
}
// ----- Global ML-defaults emission (S26) ------------------------------------
// The ONE learning_rate/max_iterations/min_error default (schemas/ml_defaults.
// json), NOT per-mode — emitted once, alongside schema_types.hpp/types.ts,
// rather than once per mode file like everything else in this module.
const AUTOGEN_BANNER_ML_DEFAULTS = (lang: "C++" | "TS"): string =>
`// AUTOGENERATED (${lang}) — do not edit. Source: schemas/ml_defaults.json. ` +
"Run `bun run codegen/generate.ts` to regenerate.";
function emitMlDefaultsHpp(d: MlTrainDefaults): string {
return [
AUTOGEN_BANNER_ML_DEFAULTS("C++"),
"// The ONE global training-hyperparameter default, shared by every mode on",
"// every platform (firmware/WASM/VCV) — see docs/specs/recon/",
"// simplification-audit-2026-07.md S26. Consumed by",
"// nisps::ml::MLPCore::TrainConfig's default member initialisers",
"// (nisps/ml/mlp.hpp); nisps_ml_set_train_config() and",
"// nisps::ml::MLPCore::set_train_config() make it runtime-overridable.",
"#ifndef NISPS_ML_GENERATED_ML_DEFAULTS_HPP",
"#define NISPS_ML_GENERATED_ML_DEFAULTS_HPP",
"",
"#include <cstddef>",
"",
"namespace nisps::ml::generated {",
"",
"struct MlTrainDefaults {",
" float learning_rate;",
" std::size_t max_iterations;",
" float min_error;",
"};",
"",
"inline constexpr MlTrainDefaults kMlTrainDefaults = {",
` ${cppFloatLit(d.learning_rate)},`,
` ${d.max_iterations}u,`,
` ${cppFloatLit(d.min_error)},`,
"};",
"",
"} // namespace nisps::ml::generated",
"",
"#endif // NISPS_ML_GENERATED_ML_DEFAULTS_HPP",
"",
].join("\n");
}
function emitMlDefaultsTs(d: MlTrainDefaults): string {
return [
AUTOGEN_BANNER_ML_DEFAULTS("TS"),
"// The ONE global training-hyperparameter default, shared by every mode on",
"// every platform — see docs/specs/recon/simplification-audit-2026-07.md S26.",
"// Consumed by WasmIML's train()/trainAsync() default parameters and",
"// EngineApi's learningRate/maxIterations options (manifold/src/engine/).",
"",
"export interface MlTrainDefaults {",
" readonly learningRate: number;",
" readonly maxIterations: number;",
" readonly minError: number;",
"}",
"",
"export const ML_TRAIN_DEFAULTS: MlTrainDefaults = {",
` learningRate: ${d.learning_rate},`,
` maxIterations: ${d.max_iterations},`,
` minError: ${d.min_error},`,
"};",
"",
].join("\n");
}
// ----- Driver --------------------------------------------------------------
function main(): number {
@ -479,6 +757,29 @@ function main(): number {
});
const validate = ajv.compile<ModeSchema>(metaSchema);
// 1b. Load, compile, and validate the ONE global ML training default
// (schemas/ml_defaults.json against schemas/ml_defaults.schema.json — S26,
// NOT per-mode, so it lives outside the modeFiles loop below).
if (!existsSync(ML_DEFAULTS_SCHEMA_PATH)) {
console.error(`error: ml-defaults meta-schema not found at ${ML_DEFAULTS_SCHEMA_PATH}`);
return 1;
}
if (!existsSync(ML_DEFAULTS_PATH)) {
console.error(`error: ml-defaults data not found at ${ML_DEFAULTS_PATH}`);
return 1;
}
const mlDefaultsSchema = readJSON<AnySchemaObject>(ML_DEFAULTS_SCHEMA_PATH);
const validateMlDefaults = ajv.compile<MlTrainDefaults>(mlDefaultsSchema);
const mlDefaultsRaw = readJSON<unknown>(ML_DEFAULTS_PATH);
if (!validateMlDefaults(mlDefaultsRaw)) {
console.error(`error: ${ML_DEFAULTS_PATH}: schema validation failed:`);
for (const err of validateMlDefaults.errors ?? []) {
console.error(` ${err.instancePath || "<root>"} ${err.message}`);
}
return 1;
}
const mlDefaults = mlDefaultsRaw as MlTrainDefaults;
// 2. Discover all mode schemas
if (!existsSync(MODES_DIR)) {
console.error(`error: modes directory not found at ${MODES_DIR}`);
@ -530,6 +831,44 @@ function main(): number {
errorCount++;
continue;
}
// Firmware-fit check (one-core-engine P5): the fixed firmware MLP template
// is exactly 4 layers (3 hidden); the browser's runtime-shaped MLP caps
// every dimension at 4096 (bindings kMaxDim).
if (schema.ml.hidden_layers.length !== 3) {
console.error(
`error: ${f}: ml.hidden_layers must have exactly 3 entries ` +
`(fixed 4-layer topology); got ${schema.ml.hidden_layers.length}`
);
errorCount++;
continue;
}
{
const dims = [schema.ml.input_size, ...schema.ml.hidden_layers, schema.ml.output_size];
const bad = dims.find(d => d <= 0 || d > 4096);
if (bad !== undefined) {
console.error(`error: ${f}: ml dimension ${bad} outside (0, 4096]`);
errorCount++;
continue;
}
}
// Per-voice-space curve deviations must name real params and must be real
// deviations. Resolved here so a bad table fails BEFORE anything is written.
try {
curveOverrideRows(schema, f);
} catch (e) {
console.error(`error: ${(e as Error).message}`);
errorCount++;
continue;
}
{
const names = schema.voice_spaces.map(voiceSpaceName);
const dup = names.find((n, i) => names.indexOf(n) !== i);
if (dup !== undefined) {
console.error(`error: ${f}: duplicate voice space name ${JSON.stringify(dup)}`);
errorCount++;
continue;
}
}
schemas.push({ source: f, schema });
}
if (errorCount > 0) {
@ -540,6 +879,8 @@ function main(): number {
// 4. Emit C++ outputs
ensureDir(CPP_OUT_DIR);
writeFileSync(join(CPP_OUT_DIR, "schema_types.hpp"), emitSchemaTypesHpp());
ensureDir(CPP_ML_OUT_DIR);
writeFileSync(join(CPP_ML_OUT_DIR, "ml_defaults.hpp"), emitMlDefaultsHpp(mlDefaults));
for (const { source, schema } of schemas) {
const out = join(CPP_OUT_DIR, `${schema.mode_id}_schema.hpp`);
writeFileSync(out, emitModeHpp(schema, source));
@ -548,6 +889,7 @@ function main(): number {
// 5. Emit TS outputs
ensureDir(TS_OUT_DIR);
writeFileSync(join(TS_OUT_DIR, "types.ts"), emitSharedTsTypes());
writeFileSync(join(TS_OUT_DIR, "ml_defaults.ts"), emitMlDefaultsTs(mlDefaults));
for (const { source, schema } of schemas) {
const out = join(TS_OUT_DIR, `${schema.mode_id}_schema.ts`);
writeFileSync(out, emitModeTs(schema, source));

51
codegen/lib.ts Normal file
View file

@ -0,0 +1,51 @@
/**
* Shared helpers for the two codegen drivers (generate.ts — mode schemas —
* and generate-midi-devices.ts — MIDI device templates).
*
* The two drivers stay separate FILES on purpose: they read different schema
* directories, validate against different meta-schemas, write to different
* output dirs, and generate.ts's golden test (codegen/tests/golden_test.ts)
* must not depend on MIDI-device schema state. None of that requires
* duplicating these small string/fs helpers — they carry no per-driver
* behaviour, so they live here once (ST12, simplification 2026-07).
*/
import { readFileSync, existsSync, mkdirSync } from "node:fs";
/** Create a directory (and parents) if it doesn't already exist. */
export function ensureDir(d: string): void {
if (!existsSync(d)) {
mkdirSync(d, { recursive: true });
}
}
/** Read and JSON-parse a file. */
export function readJSON<T>(path: string): T {
return JSON.parse(readFileSync(path, "utf8")) as T;
}
/**
* Convert snake_case to PascalCase.
* "paf_synth" -> "PafSynth", "memlcelium" -> "Memlcelium"
*/
export function toPascalCase(snake: string): string {
return snake
.split("_")
.map(s => (s.length === 0 ? s : s[0].toUpperCase() + s.slice(1)))
.join("");
}
/**
* Escape a string literal for embedding into a C++ string.
* We expect ASCII identifiers + label text only; quote and backslash are escaped.
*/
export function cppStringLit(s: string): string {
return '"' + s.replace(/\\/g, "\\\\").replace(/"/g, '\\"') + '"';
}
/**
* Escape a string for a TS single-line single-quoted literal.
*/
export function tsStringLit(s: string): string {
return "'" + s.replace(/\\/g, "\\\\").replace(/'/g, "\\'") + "'";
}

View file

@ -6,7 +6,7 @@
"type": "module",
"scripts": {
"generate": "bun run generate.ts",
"test": "bun run tests/golden_test.ts"
"test": "bun run tests/golden_test.ts && bun run tests/curve_drift_test.ts"
},
"dependencies": {
"ajv": "^8.17.1",

View file

@ -1,121 +0,0 @@
#!/usr/bin/env bun
/**
* One-time seed: derive canonical MIDI device templates (schemas/midi_devices/*.json)
* from the verified research artifact synth-midi-cc.json.
*
* Only the CC-controllable devices are emitted (has_midi_cc === true). For each,
* every {cc, name} becomes a param with a stable snake_case `id` (slugified from the
* label, de-duplicated by appending _cc<NN>), min/max from the device value range,
* a neutral default, and a coarse `group` assigned by keyword heuristic.
*
* Re-runnable and idempotent. Edit the emitted schemas/midi_devices/*.json afterwards
* if you want to hand-tune labels, groups, ranges, or defaults — they are the canonical
* source consumed by codegen/generate-midi-devices.ts. synth-midi-cc.json stays as the
* human-readable provenance/sources document.
*/
import { readFileSync, writeFileSync, mkdirSync, existsSync } from "node:fs";
import { join, dirname, resolve } from "node:path";
import { fileURLToPath } from "node:url";
const __dirname = dirname(fileURLToPath(import.meta.url));
const REPO_ROOT = resolve(__dirname, "..");
const RESEARCH_PATH = join(REPO_ROOT, "synth-midi-cc.json");
const OUT_DIR = join(REPO_ROOT, "schemas", "midi_devices");
const DEFAULT_VALUE = 64; // neutral midpoint; only used when a param is "fixed", not ML-driven
// Map research `name` -> canonical device_id + default channel. Only these are seeded.
const DEVICE_MAP: Record<string, { device_id: string; channel: number }> = {
"Moog Sub 37": { device_id: "moog_sub37", channel: 1 },
"Creamware Pro-12 ASB": { device_id: "creamware_pro12_asb", channel: 1 },
"Elektron Analog Keys": { device_id: "elektron_analog_keys", channel: 1 },
"ASM Hydrasynth": { device_id: "asm_hydrasynth", channel: 1 },
"Roland JD-800": { device_id: "roland_jd800", channel: 1 },
"Moog Sub Phatty": { device_id: "moog_sub_phatty", channel: 1 },
};
function slugify(label: string): string {
// Drop parenthetical notes, lowercase, non-alnum -> _, collapse, trim.
let s = label.replace(/\([^)]*\)/g, " ").toLowerCase();
s = s.replace(/[^a-z0-9]+/g, "_").replace(/^_+|_+$/g, "").replace(/_+/g, "_");
if (s.length === 0) s = "param";
if (!/^[a-z]/.test(s)) s = "p_" + s;
return s;
}
function groupFor(label: string): string {
const l = label.toLowerCase();
const has = (...ks: string[]) => ks.some(k => l.includes(k));
if (has("arp", "arpeggiat")) return "arp";
if (has("env", "attack", "decay", "sustain", "release", " adsr", "adr", "hold", "delay time")) return "env";
if (has("lfo")) return "lfo";
if (has("filter", "cutoff", "resonance", "res ", " res", "multidrive", "poles", "keytrack", "kb track")) return "filter";
if (has("delay", "reverb", "chorus", "flanger", "fx", "wet", "dry")) return "fx";
if (has("macro", "mod ", "mutator", "ring mod", " mod", "modulation", "wheel", "aftertouch")) return "mod";
if (has("osc", "vco", "wave", "pitch", "tune", "freq", "sub", "noise", "detune", "sync", "cent", "octave", "glide", "portamento")) return "osc";
if (has("vol", "level", "pan", "mix", "amp", "output")) return "mix";
return "global";
}
interface ResearchParam { cc: number; name: string }
interface ResearchSynth {
name: string;
manufacturer: string;
year?: number;
has_midi_cc: boolean;
cc_parameters: ResearchParam[];
}
interface Research { synths: ResearchSynth[] }
function main(): number {
const research = JSON.parse(readFileSync(RESEARCH_PATH, "utf8")) as Research;
if (!existsSync(OUT_DIR)) mkdirSync(OUT_DIR, { recursive: true });
let count = 0;
for (const synth of research.synths) {
const map = DEVICE_MAP[synth.name];
if (!map) continue; // skip non-CC / out-of-scope devices
if (!synth.has_midi_cc || synth.cc_parameters.length === 0) continue;
const seenIds = new Set<string>();
const params = synth.cc_parameters.map(p => {
let id = slugify(p.name);
if (seenIds.has(id)) id = `${id}_cc${p.cc}`;
while (seenIds.has(id)) id = `${id}_x`;
seenIds.add(id);
return {
id,
cc: p.cc,
label: p.name,
min: 0,
max: 127,
default: DEFAULT_VALUE,
group: groupFor(p.name),
};
});
const device = {
$schema: "../midi_device.schema.json",
device_id: map.device_id,
name: synth.name,
manufacturer: synth.manufacturer,
...(synth.year ? { year: synth.year } : {}),
midi: {
default_channel: map.channel,
value_range: [0, 127],
},
params,
};
const out = join(OUT_DIR, `${map.device_id}.json`);
writeFileSync(out, JSON.stringify(device, null, 2) + "\n");
count++;
console.log(` ${map.device_id}: ${params.length} params`);
}
console.log(`OK seeded ${count} device template(s) -> ${OUT_DIR}`);
return 0;
}
process.exit(main());

View file

@ -1,53 +0,0 @@
// AUTOGENERATED — do not edit. Source: schemas/modes/{{SOURCE_FILE}}. Run `bun run codegen/generate.ts` to regenerate.
//
// This file is a reference template showing the shape of each generated C++
// header. It is NOT consumed at codegen time — `codegen/generate.ts` builds the
// strings programmatically (see emitModeHpp). The template lives here so that:
// 1. reviewers can see the intended structure at a glance,
// 2. anyone tweaking the codegen has a "what should this look like?" anchor.
//
// Placeholders use {{NAME}} syntax purely for documentation.
#ifndef NISPS_GENERATED_{{MODE_ID_UPPER}}_SCHEMA_HPP
#define NISPS_GENERATED_{{MODE_ID_UPPER}}_SCHEMA_HPP
#include "schema_types.hpp"
namespace nisps::modes::generated {
inline constexpr std::string_view k{{ModeId}}ModeId = "{{mode_id}}";
inline constexpr std::string_view k{{ModeId}}EngineId = "{{engine_id}}";
inline constexpr std::array<std::string_view, {{N_INPUTS}}> k{{ModeId}}InputChannels = {{
"{{input_channel_0}}",
/* ... */
}};
inline constexpr std::array<std::size_t, {{N_HIDDEN}}> k{{ModeId}}HiddenLayers = {{
/* ... */
}};
inline constexpr MLConfig k{{ModeId}}MLConfig = {
{{input_size}}u, {{output_size}}u,
{{default_spread}}f, {{default_lr}}f, {{default_iter}}u,
};
inline constexpr std::size_t k{{ModeId}}ParamCount = {{N_PARAMS}}u;
inline constexpr std::array<Param, k{{ModeId}}ParamCount> k{{ModeId}}Params = {{
Param{ "{{name}}", "{{label}}", {{min}}f, {{max}}f, {{default}}f, Curve::{{Curve}}, "{{group}}" },
/* ... */
}};
inline constexpr std::size_t k{{ModeId}}VoiceSpaceCount = {{N_VS}}u;
inline constexpr std::array<std::string_view, k{{ModeId}}VoiceSpaceCount> k{{ModeId}}VoiceSpaces = {{
/* ... */
}};
inline constexpr UIConfig k{{ModeId}}UI = {
PrimaryInput::{{PrimaryInput}},
{{show_voice_space_selector}}, {{show_synth_visualizer}},
};
} // namespace nisps::modes::generated
#endif // NISPS_GENERATED_{{MODE_ID_UPPER}}_SCHEMA_HPP

View file

@ -1,44 +0,0 @@
// AUTOGENERATED — do not edit. Source: schemas/modes/{{SOURCE_FILE}}. Run `bun run codegen/generate.ts` to regenerate.
//
// Reference template for the TypeScript codegen output. Not consumed at runtime
// (the codegen builds strings programmatically in emitModeTs). Lives here for
// human reviewers — placeholders use {{NAME}} syntax.
import type { ModeSchema } from './types';
export interface {{ModeId}}Params {
readonly {{param_name_0}}: number;
/* ... */
}
export const {{ModeId}}Schema: ModeSchema = {
mode_id: '{{mode_id}}',
engine_id: '{{engine_id}}',
ml: {
input_channels: ['{{input_channel_0}}', /* ... */],
input_size: {{input_size}},
hidden_layers: [/* ... */],
output_size: {{output_size}},
default_spread: {{default_spread}},
default_learning_rate: {{default_lr}},
default_max_iterations: {{default_iter}},
},
params: [
{
name: '{{name}}',
label: '{{label}}',
min: {{min}},
max: {{max}},
default: {{default}},
curve: '{{curve}}',
group: '{{group}}',
},
/* ... */
],
voice_spaces: [/* ... */],
ui: {
primary_input: '{{primary_input}}',
show_voice_space_selector: {{show_voice_space_selector}},
show_synth_visualizer: {{show_synth_visualizer}},
},
};

View file

@ -0,0 +1,185 @@
#!/usr/bin/env bun
/**
* Curve drift check — the schemas' declared response curves must equal what
* `nisps/engines/*.hpp` actually does, per voice space.
*
* WHY THIS CHECK IS SOURCE-LEVEL, NOT BEHAVIOURAL
* -----------------------------------------------
* You cannot observe "the curve" from engine output. A voice space maps a
* normalised slot into engine state as `base + f(p) * scale` and the state
* then disappears into DSP; without independently knowing `base`/`scale` (and
* the DSP transfer function) there is no way to recover `f` from audio. The
* engines expose no accessor for the mapped state, and adding one would mean
* editing engine internals to make a declaration checkable — the tail wagging
* the dog. So the fact lives in the arithmetic, and that is what we read.
*
* The upside of being source-level: it verifies the WHOLE table (9 modes, 26
* voice spaces, 344 params) rather than the handful of code paths any
* behavioural harness would reach. The parity harness, for contrast, only
* exercises PAFSynth + ChannelStrip at all-params-0.5.
*
* It fails loudly on three separate classes of drift:
* 1. a declared curve that disagrees with the engine,
* 2. a schema voice-space list that disagrees with the engine's
* `kVoiceSpaceNames` (order matters — the schema index IS the enum
* ordinal that `ModeBase::set_voice_space` casts to),
* 3. an engine idiom the extractor cannot reduce (codegen/curve-audit.ts
* raises rather than guessing "linear").
*
* It checks the JSON schemas AND the generated TypeScript, so a codegen bug
* that drops the table cannot pass.
*/
import { readFileSync, readdirSync } from "node:fs";
import { join, dirname, resolve } from "node:path";
import { fileURLToPath } from "node:url";
import { auditAllEngines, CurveAuditError, type Curve } from "../curve-audit.ts";
import { ALL_MODE_SCHEMAS, effectiveCurve } from "../../manifold/src/modes/generated/index.ts";
const __dirname = dirname(fileURLToPath(import.meta.url));
const REPO_ROOT = resolve(__dirname, "..", "..");
const MODES_DIR = join(REPO_ROOT, "schemas", "modes");
const ENGINES_DIR = join(REPO_ROOT, "nisps", "engines");
type VoiceSpaceDecl = string | { name: string; curve_overrides: Record<string, Curve> };
interface JsonModeSchema {
mode_id: string;
engine_id: string;
params: Array<{ name: string; curve: Curve }>;
voice_spaces: VoiceSpaceDecl[];
}
const vsName = (v: VoiceSpaceDecl): string => (typeof v === "string" ? v : v.name);
/** Resolve the JSON declaration into curves[voiceSpace][param]. */
function declaredCurves(s: JsonModeSchema): Curve[][] {
const defaults = s.params.map((p) => p.curve);
const byName = new Map(s.params.map((p, i) => [p.name, i] as const));
const rows = s.voice_spaces.length === 0 ? [null] : s.voice_spaces;
return rows.map((v) => {
const row = [...defaults];
if (v && typeof v !== "string") {
for (const [name, c] of Object.entries(v.curve_overrides)) {
row[byName.get(name)!] = c;
}
}
return row;
});
}
function run(): number {
const problems: string[] = [];
let engines;
try {
engines = auditAllEngines(ENGINES_DIR);
} catch (e) {
if (e instanceof CurveAuditError) {
console.error(`\nFAILED to read the engines' curve arithmetic:\n ${e.message}\n`);
console.error(
"The extractor refuses to guess. Either the engine grew an idiom\n" +
"codegen/curve-audit.ts does not model, or a voice space lost its\n" +
"dispatch case. Teach the extractor, do not weaken it.\n"
);
return 1;
}
throw e;
}
const tsByModeId = new Map(ALL_MODE_SCHEMAS.map((m) => [m.mode_id, m] as const));
let checkedModes = 0;
let checkedSlots = 0;
for (const f of readdirSync(MODES_DIR).filter((n) => n.endsWith(".json")).sort()) {
const s = JSON.parse(readFileSync(join(MODES_DIR, f), "utf8")) as JsonModeSchema;
const eng = engines.get(s.engine_id);
if (!eng) {
problems.push(`${f}: engine_id ${JSON.stringify(s.engine_id)} matches no engine in nisps/engines/`);
continue;
}
// (2) voice-space identity. Index i in the schema IS VoiceSpace ordinal i.
if (eng.voiceSpaceNames !== null) {
const declared = s.voice_spaces.map(vsName);
if (JSON.stringify(declared) !== JSON.stringify(eng.voiceSpaceNames)) {
problems.push(
`${f}: voice_spaces disagree with ${eng.file}'s kVoiceSpaceNames\n` +
` schema: ${JSON.stringify(declared)}\n` +
` engine: ${JSON.stringify(eng.voiceSpaceNames)}`
);
continue;
}
} else if (s.voice_spaces.length > 1) {
problems.push(
`${f}: declares ${s.voice_spaces.length} voice spaces but ${eng.file} has no VoiceSpace enum`
);
continue;
}
// An engine with no params of its own (NoOpEngine, engine_id "thru") maps
// nothing: the mode's outputs are MIDI CCs it emits itself, unshaped.
const allLinear: Curve[] = s.params.map(() => "linear");
const actual =
eng.nParams === 0
? [allLinear]
: eng.voiceSpaceNames !== null
? eng.curves
: [eng.curves[0]!];
if (eng.nParams !== 0 && eng.nParams !== s.params.length) {
problems.push(`${f}: ${s.params.length} params but ${eng.engineId} has kNParams = ${eng.nParams}`);
continue;
}
// (1) the declared table, from the JSON…
const declared = declaredCurves(s);
// …and independently from the generated TypeScript, so a codegen bug that
// drops or mis-indexes the table is caught too.
const ts = tsByModeId.get(s.mode_id);
if (!ts) {
problems.push(`${f}: no generated TypeScript schema for mode_id ${s.mode_id}`);
continue;
}
const nVs = Math.max(declared.length, 1);
for (let vs = 0; vs < nVs; vs++) {
const expect = actual.length === 1 ? actual[0]! : actual[vs]!;
for (let i = 0; i < s.params.length; i++) {
checkedSlots++;
const label = `${s.mode_id}[${vsName(s.voice_spaces[vs] ?? "-")}].${s.params[i]!.name} (slot ${i})`;
if (declared[vs]![i] !== expect[i]) {
problems.push(
`${label}: schema says ${declared[vs]![i]}, ${eng.file} applies ${expect[i]}`
);
}
const fromTs = effectiveCurve(ts, vs, i);
if (fromTs !== declared[vs]![i]) {
problems.push(
`${label}: generated TS says ${fromTs}, schemas/modes/${f} says ${declared[vs]![i]}`
);
}
}
}
checkedModes++;
}
if (problems.length > 0) {
console.error(`\ncurve drift: ${problems.length} problem(s)\n`);
for (const p of problems) console.error(` ${p}`);
console.error(
"\nThe schemas' `curve` fields are DESCRIPTIVE: they record what the\n" +
"engine already does. If an engine's arithmetic changed on purpose,\n" +
"update schemas/modes/*.json (params[].curve for the default, or the\n" +
"voice space's curve_overrides for a deviation) and re-run codegen.\n" +
"Do NOT change the engine to match the declaration.\n"
);
return 1;
}
console.log(
`curve drift: ok — ${checkedModes} modes, ${checkedSlots} (voice space x param) slots ` +
`cross-checked against nisps/engines/ source.`
);
return 0;
}
process.exit(run());

View file

@ -3,6 +3,7 @@
#define NISPS_GENERATED_PAF_SYNTH_SCHEMA_HPP
#include "schema_types.hpp"
#include "../../ml/mlp.hpp"
namespace nisps::modes::generated {
@ -26,8 +27,6 @@ inline constexpr MLConfig kPafSynthMLConfig = {
4u,
33u,
0.6f,
1.0f,
1000u,
};
inline constexpr std::size_t kPafSynthParamCount = 33u;
@ -342,12 +341,92 @@ inline constexpr std::array<std::string_view, kPafSynthVoiceSpaceCount> kPafSynt
"Ipeleiades",
}};
inline constexpr std::array<CurveOverride, 61u> kPafSynthCurveOverrides = {{
CurveOverride{0u, 8u, Curve::linear}, // Ellipticacacia.paf0_vib
CurveOverride{0u, 9u, Curve::linear}, // Ellipticacacia.paf1_vib
CurveOverride{0u, 11u, Curve::linear}, // Ellipticacacia.paf0_vfr
CurveOverride{0u, 12u, Curve::linear}, // Ellipticacacia.paf1_vfr
CurveOverride{0u, 14u, Curve::square}, // Ellipticacacia.paf0_shift
CurveOverride{0u, 17u, Curve::linear}, // Ellipticacacia.dl1mix
CurveOverride{0u, 18u, Curve::square}, // Ellipticacacia.p18
CurveOverride{0u, 19u, Curve::square}, // Ellipticacacia.dlfb
CurveOverride{0u, 20u, Curve::linear}, // Ellipticacacia.env_decay
CurveOverride{0u, 26u, Curve::linear}, // Ellipticacacia.shape_gain
CurveOverride{0u, 27u, Curve::linear}, // Ellipticacacia.shape_asym
CurveOverride{0u, 29u, Curve::linear}, // Ellipticacacia.rm_gain
CurveOverride{2u, 10u, Curve::square}, // Neemeda.p10
CurveOverride{2u, 13u, Curve::square}, // Neemeda.p13
CurveOverride{2u, 23u, Curve::square}, // Neemeda.p23
CurveOverride{2u, 24u, Curve::square}, // Neemeda.p24
CurveOverride{3u, 10u, Curve::square}, // Aquillow.p10
CurveOverride{3u, 13u, Curve::square}, // Aquillow.p13
CurveOverride{3u, 23u, Curve::square}, // Aquillow.p23
CurveOverride{3u, 24u, Curve::square}, // Aquillow.p24
CurveOverride{3u, 26u, Curve::linear}, // Aquillow.shape_gain
CurveOverride{3u, 27u, Curve::linear}, // Aquillow.shape_asym
CurveOverride{3u, 29u, Curve::linear}, // Aquillow.rm_gain
CurveOverride{3u, 32u, Curve::square}, // Aquillow.env_release
CurveOverride{4u, 5u, Curve::square}, // Magnetarch.paf0_bw
CurveOverride{4u, 8u, Curve::linear}, // Magnetarch.paf0_vib
CurveOverride{4u, 9u, Curve::linear}, // Magnetarch.paf1_vib
CurveOverride{4u, 11u, Curve::linear}, // Magnetarch.paf0_vfr
CurveOverride{4u, 12u, Curve::linear}, // Magnetarch.paf1_vfr
CurveOverride{4u, 17u, Curve::linear}, // Magnetarch.dl1mix
CurveOverride{4u, 20u, Curve::linear}, // Magnetarch.env_decay
CurveOverride{4u, 26u, Curve::linear}, // Magnetarch.shape_gain
CurveOverride{4u, 27u, Curve::linear}, // Magnetarch.shape_asym
CurveOverride{4u, 29u, Curve::linear}, // Magnetarch.rm_gain
CurveOverride{5u, 8u, Curve::linear}, // Elderstar.paf0_vib
CurveOverride{5u, 9u, Curve::linear}, // Elderstar.paf1_vib
CurveOverride{5u, 11u, Curve::linear}, // Elderstar.paf0_vfr
CurveOverride{5u, 12u, Curve::linear}, // Elderstar.paf1_vfr
CurveOverride{5u, 14u, Curve::square}, // Elderstar.paf0_shift
CurveOverride{5u, 17u, Curve::linear}, // Elderstar.dl1mix
CurveOverride{5u, 18u, Curve::square}, // Elderstar.p18
CurveOverride{5u, 19u, Curve::square}, // Elderstar.dlfb
CurveOverride{5u, 21u, Curve::square}, // Elderstar.p21
CurveOverride{5u, 22u, Curve::square}, // Elderstar.p22
CurveOverride{5u, 23u, Curve::square}, // Elderstar.p23
CurveOverride{5u, 26u, Curve::linear}, // Elderstar.shape_gain
CurveOverride{5u, 27u, Curve::linear}, // Elderstar.shape_asym
CurveOverride{5u, 29u, Curve::linear}, // Elderstar.rm_gain
CurveOverride{6u, 8u, Curve::linear}, // Ipeleiades.paf0_vib
CurveOverride{6u, 9u, Curve::linear}, // Ipeleiades.paf1_vib
CurveOverride{6u, 11u, Curve::linear}, // Ipeleiades.paf0_vfr
CurveOverride{6u, 12u, Curve::linear}, // Ipeleiades.paf1_vfr
CurveOverride{6u, 14u, Curve::square}, // Ipeleiades.paf0_shift
CurveOverride{6u, 17u, Curve::linear}, // Ipeleiades.dl1mix
CurveOverride{6u, 18u, Curve::square}, // Ipeleiades.p18
CurveOverride{6u, 21u, Curve::square}, // Ipeleiades.p21
CurveOverride{6u, 22u, Curve::square}, // Ipeleiades.p22
CurveOverride{6u, 23u, Curve::square}, // Ipeleiades.p23
CurveOverride{6u, 26u, Curve::linear}, // Ipeleiades.shape_gain
CurveOverride{6u, 27u, Curve::linear}, // Ipeleiades.shape_asym
CurveOverride{6u, 29u, Curve::linear}, // Ipeleiades.rm_gain
}};
inline constexpr UIConfig kPafSynthUI = {
PrimaryInput::XYPad,
true,
true,
};
using PafSynthMLP = ::nisps::ml::MLP<kPafSynthMLConfig.input_size, kPafSynthHiddenLayers[0], kPafSynthHiddenLayers[1], kPafSynthHiddenLayers[2], kPafSynthMLConfig.output_size>;
inline constexpr ::nisps::ParamSchema kPafSynthSchema = {
kPafSynthModeId,
kPafSynthEngineId,
std::span<const std::string_view>(kPafSynthInputChannels),
kPafSynthMLConfig.input_size,
std::span<const std::size_t>(kPafSynthHiddenLayers),
kPafSynthMLConfig.output_size,
kPafSynthMLConfig.default_spread,
std::span<const Param>(kPafSynthParams),
std::span<const std::string_view>(kPafSynthVoiceSpaces),
std::span<const CurveOverride>(kPafSynthCurveOverrides),
kPafSynthUI,
};
} // namespace nisps::modes::generated
#endif // NISPS_GENERATED_PAF_SYNTH_SCHEMA_HPP

View file

@ -55,8 +55,6 @@ export const PafSynthSchema: ModeSchema = {
],
output_size: 33,
default_spread: 0.6,
default_learning_rate: 1,
default_max_iterations: 1000,
},
params: [
{
@ -366,6 +364,69 @@ export const PafSynthSchema: ModeSchema = {
'Elderstar',
'Ipeleiades',
],
curve_overrides: [
{ voice_space: 0, param: 8, curve: 'linear' }, // Ellipticacacia.paf0_vib
{ voice_space: 0, param: 9, curve: 'linear' }, // Ellipticacacia.paf1_vib
{ voice_space: 0, param: 11, curve: 'linear' }, // Ellipticacacia.paf0_vfr
{ voice_space: 0, param: 12, curve: 'linear' }, // Ellipticacacia.paf1_vfr
{ voice_space: 0, param: 14, curve: 'square' }, // Ellipticacacia.paf0_shift
{ voice_space: 0, param: 17, curve: 'linear' }, // Ellipticacacia.dl1mix
{ voice_space: 0, param: 18, curve: 'square' }, // Ellipticacacia.p18
{ voice_space: 0, param: 19, curve: 'square' }, // Ellipticacacia.dlfb
{ voice_space: 0, param: 20, curve: 'linear' }, // Ellipticacacia.env_decay
{ voice_space: 0, param: 26, curve: 'linear' }, // Ellipticacacia.shape_gain
{ voice_space: 0, param: 27, curve: 'linear' }, // Ellipticacacia.shape_asym
{ voice_space: 0, param: 29, curve: 'linear' }, // Ellipticacacia.rm_gain
{ voice_space: 2, param: 10, curve: 'square' }, // Neemeda.p10
{ voice_space: 2, param: 13, curve: 'square' }, // Neemeda.p13
{ voice_space: 2, param: 23, curve: 'square' }, // Neemeda.p23
{ voice_space: 2, param: 24, curve: 'square' }, // Neemeda.p24
{ voice_space: 3, param: 10, curve: 'square' }, // Aquillow.p10
{ voice_space: 3, param: 13, curve: 'square' }, // Aquillow.p13
{ voice_space: 3, param: 23, curve: 'square' }, // Aquillow.p23
{ voice_space: 3, param: 24, curve: 'square' }, // Aquillow.p24
{ voice_space: 3, param: 26, curve: 'linear' }, // Aquillow.shape_gain
{ voice_space: 3, param: 27, curve: 'linear' }, // Aquillow.shape_asym
{ voice_space: 3, param: 29, curve: 'linear' }, // Aquillow.rm_gain
{ voice_space: 3, param: 32, curve: 'square' }, // Aquillow.env_release
{ voice_space: 4, param: 5, curve: 'square' }, // Magnetarch.paf0_bw
{ voice_space: 4, param: 8, curve: 'linear' }, // Magnetarch.paf0_vib
{ voice_space: 4, param: 9, curve: 'linear' }, // Magnetarch.paf1_vib
{ voice_space: 4, param: 11, curve: 'linear' }, // Magnetarch.paf0_vfr
{ voice_space: 4, param: 12, curve: 'linear' }, // Magnetarch.paf1_vfr
{ voice_space: 4, param: 17, curve: 'linear' }, // Magnetarch.dl1mix
{ voice_space: 4, param: 20, curve: 'linear' }, // Magnetarch.env_decay
{ voice_space: 4, param: 26, curve: 'linear' }, // Magnetarch.shape_gain
{ voice_space: 4, param: 27, curve: 'linear' }, // Magnetarch.shape_asym
{ voice_space: 4, param: 29, curve: 'linear' }, // Magnetarch.rm_gain
{ voice_space: 5, param: 8, curve: 'linear' }, // Elderstar.paf0_vib
{ voice_space: 5, param: 9, curve: 'linear' }, // Elderstar.paf1_vib
{ voice_space: 5, param: 11, curve: 'linear' }, // Elderstar.paf0_vfr
{ voice_space: 5, param: 12, curve: 'linear' }, // Elderstar.paf1_vfr
{ voice_space: 5, param: 14, curve: 'square' }, // Elderstar.paf0_shift
{ voice_space: 5, param: 17, curve: 'linear' }, // Elderstar.dl1mix
{ voice_space: 5, param: 18, curve: 'square' }, // Elderstar.p18
{ voice_space: 5, param: 19, curve: 'square' }, // Elderstar.dlfb
{ voice_space: 5, param: 21, curve: 'square' }, // Elderstar.p21
{ voice_space: 5, param: 22, curve: 'square' }, // Elderstar.p22
{ voice_space: 5, param: 23, curve: 'square' }, // Elderstar.p23
{ voice_space: 5, param: 26, curve: 'linear' }, // Elderstar.shape_gain
{ voice_space: 5, param: 27, curve: 'linear' }, // Elderstar.shape_asym
{ voice_space: 5, param: 29, curve: 'linear' }, // Elderstar.rm_gain
{ voice_space: 6, param: 8, curve: 'linear' }, // Ipeleiades.paf0_vib
{ voice_space: 6, param: 9, curve: 'linear' }, // Ipeleiades.paf1_vib
{ voice_space: 6, param: 11, curve: 'linear' }, // Ipeleiades.paf0_vfr
{ voice_space: 6, param: 12, curve: 'linear' }, // Ipeleiades.paf1_vfr
{ voice_space: 6, param: 14, curve: 'square' }, // Ipeleiades.paf0_shift
{ voice_space: 6, param: 17, curve: 'linear' }, // Ipeleiades.dl1mix
{ voice_space: 6, param: 18, curve: 'square' }, // Ipeleiades.p18
{ voice_space: 6, param: 21, curve: 'square' }, // Ipeleiades.p21
{ voice_space: 6, param: 22, curve: 'square' }, // Ipeleiades.p22
{ voice_space: 6, param: 23, curve: 'square' }, // Ipeleiades.p23
{ voice_space: 6, param: 26, curve: 'linear' }, // Ipeleiades.shape_gain
{ voice_space: 6, param: 27, curve: 'linear' }, // Ipeleiades.shape_asym
{ voice_space: 6, param: 29, curve: 'linear' }, // Ipeleiades.rm_gain
],
ui: {
primary_input: 'xy_pad',
show_voice_space_selector: true,

View file

@ -2,12 +2,13 @@
/**
* Golden test: re-runs codegen and asserts that the freshly-generated
* paf_synth_schema.{hpp,ts} files are byte-identical to the snapshots in
* codegen/tests/golden/.
* codegen/tests/golden/. (The TS target moved playground → manifold at P5
* of docs/specs/plans/one-core-engine-refactor.md.)
*
* If you change the codegen template intentionally, regenerate the goldens:
* bun run generate.ts
* cp ../nisps/modes/generated/paf_synth_schema.hpp tests/golden/
* cp ../playground/src/modes/generated/paf_synth_schema.ts tests/golden/
* cp ../manifold/src/modes/generated/paf_synth_schema.ts tests/golden/
*/
import { readFileSync, existsSync } from "node:fs";
@ -33,7 +34,7 @@ const CASES: Case[] = [
},
{
name: "paf_synth_schema.ts",
generatedPath: join(REPO_ROOT, "playground", "src", "modes", "generated", "paf_synth_schema.ts"),
generatedPath: join(REPO_ROOT, "manifold", "src", "modes", "generated", "paf_synth_schema.ts"),
goldenPath: join(GOLDEN_DIR, "paf_synth_schema.ts"),
},
];

View file

@ -1 +0,0 @@
,ck84,dharma-the-swan,04.11.2025 16:41,file:///home/ck84/snap/libreoffice/358/.config/libreoffice/4;

View file

@ -1,424 +0,0 @@
Name,Component_1,Component_2
Betelginkgo,Betelgeuse,Ginkgo
Pleiadedar,Pleiades,Cedar
Helixypress,Helix Nebula,Cypress
Rigelmwood,Rigel,Elmwood
Sombreroak,Sombrero Galaxy,Oak
Hyadehorn,Hyades,Hawthorn
Triangularch,Triangulum,Larch
Vegalder,Vega,Alder
Whirlpoplar,Whirpool Galaxy,Poplar
Siriuswood,Sirius,Basswood
Oortcacia,Oort Cloud,Acacia
Denebony,Deneb,Ebony
Cartwhillow,Cartwheel Galaxy,Willow
Polariswood,Polaris,Ironwood
Craborwood,Crab Nebula,Hornbeam
Aldebaranyan,Aldebaran,Banyan
Pinwheelnut,Pinwheel Galaxy,Walnut
Antaresnut,Antares,Chestnut
Kuiperlpa,Kuiper Belt,Catalpa
Arcturuspruce,Arcturus,Spruce
Pegasafras,Pegasus,Sassafras
Capellaberry,Capella,Mulberry
Lyracacia,Lyra,Acacia
Polluxwood,Pollux,Boxwood
Scorpiumac,Scorpius,Sumac
Castorwood,Castor,Cottonwood
Dracodar,Draco,Cedar
Regulushua,Regulus,Joshua
Aquilnut,Aquila,Walnut
Spicamore,Spica,Sycamore
Cygnusswood,Cygnus,Sasswood
Proceyon,Procyon,Pecan
Perseusqoia,Perseus,Sequoia
Sagittalpa,Sagittarius,Catalpa
Tauruspen,Taurus,Aspen
Geminowan,Gemini,Rowan
Canceria,Cancer,Wisteria
Leogany,Leo,Mahogany
Virgoak,Virgo,Oak
Libraobab,Libra,Baobab
Ariescalyptus,Aries,Eucalyptus
Capricornia,Capricorn,Magnolia
Aquariusqoia,Aquarius,Sequoia
Piscesimmon,Pisces,Persimmon
Omeganolia,Omega Centauri,Magnolia
Beehivewood,Beehive Cluster,Olivewood
Pillarspress,Pillars of Creation,Cypress
Greatwillow,Great Wall,Willow
Localboobao,Local Bubble,Baobab
Termishoak,Termination Shock,Oak
Bowshocknut,Bow Shock,Chestnut
Stellarix,Stellar Wind,Tamarix
Heliopaulownia,Heliopause,Paulownia
Rochemewood,Roche Limit,Limewood
Eventhornbeam,Event Horizon,Hornbeam
Accretioak,Accretion Disk,Oak
Protoplanetree,Protoplanetary Disk,Planetree
Lagranginkgo,Lagrange Point,Ginkgo
Trojaniper,Trojan,Juniper
Gammarchwood,Gamma Ray Burst,Birchwood
Hypernowan,Hypernova,Rowan
Whitedwillow,White Dwarf,Willow
Redgianter,Red Giant,Red Alder
Superclusternut,Supercluster,Butternut
Filamenteakwood,Filament,Teakwood
Voidglass,Void,Douglass Fir
Greywallnut,Great Wall,Wallnut
Comettree,Comet,Peppertree
Asteroider,Asteroid Belt,Elder
Helixelkova,Helix Nebula,Zelkova
Orionwood,Orion Nebula,Redwood
Cassiopaple,Cassiopeia,Apple
Ursajortree,Ursa Major,Majortree
Ursaminorwood,Ursa Minor,Dogwood
Bristlecrab,Bristlecone Pine,Crab Nebula
Metasequoyasar,Metasequoia,Quasar
Monkeypleiades,Monkey Puzzle,Pleiades
Dragontree,Dragon Tree,Draco
Boojumbrero,Boojum,Sombrero Galaxy
Quiverwheel,Quiver Tree,Pinwheel
Joshuapiter,Joshua Tree,Jupiter
Wolleminebula,Wollemi Pine,Nebula
Dawndromeda,Dawn Redwood,Andromeda
Baldcygnuss,Bald Cypress,Cygnus
Zelkovega,Zelkova,Vega
Hornbeamstar,Hornbeam,Barnstar
Rowantares,Rowan,Antares
Basswoodburst,Basswood,Gamma Burst
Calabashea,Calabash,Cassiopeia
Neemeda,Neem,Andromeda
Salirius,Sal,Sirius
Tooniverse,Toon,Universe
Cassiapeia,Cassia,Cassiopeia
Yellowoodrion,Yellowwood,Orion
Blackwoodhole,Blackwood,Black Hole
Ironwoodsar,Ironwood,Quasar
Satinwhirl,Satinwood,Whirlpool
Poplarsar,Poplar,Pulsar
Cottonwoodway,Cottonwood,Milky Way
Sumacnova,Sumac,Supernova
Hazeleius,Hazel,Pegasus
Elderstar,Elder,Barnstar
Rigelbark,Rigel,Bark
Sirillow,Sirius,Willow
Altairix,Altair,Tamarix
Denebula,Deneb,Nebula
Polarispine,Polaris,Pine
Betelpine,Betelgeuse,Bristlecone
Rigellarch,Rigel,Larch
Vegamore,Vega,Sycamore
Altairnut,Altair,Chestnut
Arcturuscacia,Arcturus,Acacia
Capellaple,Capella,Apple
Polluxdogwood,Pollux,Dogwood
Castorong,Castor,Camphorwood
Regulusqoia,Regulus,Sequoia
Spicape,Spica,Grape
Procyonwood,Procyon,Sandalwood
Aldebaranyan,Aldebaran,Banyan
Antarespress,Antares,Cypress
Pleiadwood,Pleiades,Cedarwood
Hyadesmac,Hyades,Sumac
Beehivewood,Beehive,Olivewood
Orionniper,Orion,Juniper
Cassiopear,Cassiopeia,Pear
Dracoroak,Draco,Cork Oak
Lyralnut,Lyra,Walnut
Cygnust,Cygnus,Locust
Aquillow,Aquila,Willow
Perseusnut,Perseus,Butternut
Pegasuspen,Pegasus,Aspen
Andromelm,Andromeda,Elm
Triangulfir,Triangulum,Fir
Sombreruce,Sombrero,Spruce
Whirlpinecone,Whirlpool,Pine
Cartwheelnut,Cartwheel,Hazelnut
Helixelm,Helix,Elm
Crabapple,Crab Nebula,Crabapple
Pillarspen,Pillars,Aspen
Sagittariuswood,Sagittarius,Sandalwood
Scorpiuspen,Scorpius,Aspen
Taurusqoia,Taurus,Sequoia
Geminoak,Gemini,Oak
Cancerdar,Cancer,Cedar
Leowood,Leo,Boxwood
Virgowan,Virgo,Rowan
Librasswood,Libra,Basswood
Ariesnut,Aries,Chestnut
Capricornbeam,Capricorn,Hornbeam
Aquariuslder,Aquarius,Alder
Piscespress,Pisces,Cypress
Magnetardwood,Magnetar,Hardwood
Pulsarwood,Pulsar,Rosewood
Quasarberry,Quasar,Elderberry
Supernovakia,Supernova,Plumeria
Hypernovania,Hypernova,Paulownia
Blackholly,Black Hole,Holly
Whitedwarfpine,White Dwarf,White Pine
Redgiantwood,Red Giant,Giantwood
Bluegiganticacia,Blue Giant,Acacia
Neutronia,Neutron Star,Catalpa
Browndwarfnut,Brown Dwarf,Butternut
Protostarfir,Protostar,Fir
Mainsequoia,Main Sequence,Sequoia
Subergiantree,Supergiant,Tree
Yorkcloud,Oort Cloud,Cork
Kuiperlpa,Kuiper Belt,Catalpa
Heliospherical,Heliosphere,Spherical Tree
Heliopaulogany,Heliopause,Mahogany
Terminationoak,Termination Shock,Oak
Bowshockberry,Bow Shock,Hackberry
Stellarwindwood,Stellar Wind,Driftwood
Solarwindowan,Solar Wind,Rowan
Interstellarch,Interstellar Medium,Larch
Circumstellarix,Circumstellar Disk,Tamarix
Protoplanetree,Protoplanetary Disk,Plane Tree
Accretionoak,Accretion Disk,Oak
Eventhorizonwood,Event Horizon,Ironwood
Schwarzschillow,Schwarzschild Radius,Willow
Singularsimmon,Singularity,Persimmon
Rochelimewood,Roche Limit,Limewood
Lagrangewood,Lagrange Point,Rangewood
Trojantree,Trojan Asteroids,Tree
Hildazel,Hilda Group,Hazel
Amoribark,Amor Asteroids,Bark
Atenberry,Aten Asteroids,Elderberry
Apollomeria,Apollo Asteroids,Plumeria
Gammaburst Ash,Gamma Ray Burst,Ash
Xraybirch,X-ray Binary,Birch
Millisecwood,Millisecond Pulsar,Redwood
Magnetarch,Magnetar,Larch
Softgammawood,Soft Gamma Repeater,Softwood
Superclusternut,Supercluster,Clusternut
Galaxyclusterpine,Galaxy Cluster,Pine
Galaxygroupfir,Galaxy Group,Fir
Localgrowan,Local Group,Rowan
Virgoclusteroak,Virgo Cluster,Oak
Comaberrynicaceae,Coma Berenices,Berry
Greatwallnut,Great Wall,Wallnut
Sloangreatwillow,Sloan Great Wall,Willow
Herculessuperia,Hercules Supercluster,Wisteria
Shapleypress,Shapley Supercluster,Cypress
Laniakeroak,Laniakea Supercluster,Kauri Oak
Cosmicwebwood,Cosmic Web,Webwood
Filamentree,Filament,Tree
Voidwood,Void,Driftwood
Boötesvoidwood,Boötes Void,Olivewood
Sculptornut,Sculptor Void,Coconut
Greywallash,Great Wall,Ash
Localbubbaobab,Local Bubble,Baobab
Loopicacia,Local Interstellar Cloud,Acacia
Gouldspen,Gould Belt,Aspen
Orionspurmac,Orion Spur,Sumac
Sagittariusarmwood,Sagittarius Arm,Armwood
Perseusarmpine,Perseus Arm,Pine
Outerarmoak,Outer Arm,Oak
Scutumcentauruswood,Scutum-Centaurus Arm,Scutumwood
Normamalder,Norma Arm,Alder
Barredspiralex,Barred Spiral,Ilex
Ellipticacacia,Elliptical Galaxy,Acacia
Irregularnut,Irregular Galaxy,Butternut
Lenticularcorn,Lenticular Galaxy,Acorn
Dwarfnut,Dwarf Galaxy,Chestnut
Satelliteelm,Satellite Galaxy,Elm
Starburstfir,Starburst Galaxy,Fir
Seyfertree,Seyfert Galaxy,Feather Tree
Blazarwood,Blazar,Rosewood
Radiogalaxypress,Radio Galaxy,Cypress
Activenutcleus,Active Galactic Nucleus,Nut
Quasartree,Quasar,Quandong Tree
Ultraluminouswood,Ultraluminous Galaxy,Luminous Tree
Luminousredgiant,Luminous Red Galaxy,Red Giant Sequoia
Greenpeachia,Green Pea Galaxy,Peach
Polarringwood,Polar Ring Galaxy,Ringwood
Shellgalaxash,Shell Galaxy,Ash
Ringalaxypress,Ring Galaxy,Cypress
Hoagsobjectoak,Hoag's Object,Oak
Mayallsobjectree,Mayall's Object,Tree
Tadpolgalaxillow,Tadpole Galaxy,Willow
Cometgalaxalder,Comet Galaxy,Alder
Sunflowernut,Sunflower Galaxy,Sunflower
Blackeyewood,Black Eye Galaxy,Blackwood
Bodesgalaxnut,Bode's Galaxy,Coconut
Cigargalaxpress,Cigar Galaxy,Cypress
Mousesgalaxypen,Mice Galaxies,Aspen
Antennaegalaxynt,Antennae Galaxies,Antwood
Polarisstwood,Polaris,Sasswood
Rigwood,Rigel,Dogwood
Canopusnut,Canopus,Coconut
Vegamore,Vega,Sycamore
Capellalnut,Capella,Walnut
Rigelberry,Rigel,Mulberry
Procyonwood,Procyon,Cottonwood
Achernarnut,Achernar,Butternut
Betelgeuswood,Betelgeuse,Geuswood
Hadarcedarwood,Hadar,Cedarwood
Altairix,Altair,Tamarix
Alderaminwood,Alderamin,Alderwood
Denebolive,Denebola,Olive
Mimosacia,Mimosa,Acacia
Regulenut,Regulus,Filbert
Adharalnut,Adhara,Walnut
Shahelmwood,Shaula,Elm
Gacruxifir,Gacrux,Fir
Bellatrixnut,Bellatrix,Hazelnut
Elnathnut,Elnath,Chestnut
Miaplacidiumwood,Miaplacidus,Palladium
Alnitaknut,Alnitak,Coconut
Alnilamspen,Alnilam,Aspen
Aliothogany,Alioth,Mahogany
Mirzamberry,Mirzam,Mulberry
Alkaidnut,Alkaid,Butternut
Atriarch,Atria,Larch
Alhenawood,Alhena,Alderwood
Peacockahogany,Peacock,Mahogany
Mirfaknut,Mirfak,Coconut
Wezenalder,Wezen,Alder
Sargassage,Sargas,Sage
Menkarberry,Menkar,Elderberry
Eniffir,Enif,Fir
Denebalgedinut,Deneb Algedi,Butternut
Zubenhaknutbi,Zuben Hakrabi,Butternut
Alshainalnut,Alshain,Walnut
Rastabantamarind,Rastaban,Tamarind
Markabwood,Markab,Markwood
Schedarogany,Schedar,Mahogany
Almaalnutak,Almaak,Walnut
Rasalgetash,Rasalgethi,Ash
Thubanyan,Thuban,Banyan
Kausafras,Kaus Australis,Sassafras
Nunkirnut,Nunki,Coconut
Sabikwood,Sabik,Rosewood
Menkalnutnan,Menkalnan,Butternut
Asteropewood,Asterope,Redwood
Atlaspen,Atlas,Aspen
Electralder,Electra,Alder
Maiagany,Maia,Mahogany
Meropewood,Merope,Olivewood
Taugetamarix,Taygeta,Tamarix
Pleionetree,Pleione,Tree
Celaenoak,Celaeno,Oak
Alcyonedar,Alcyone,Cedar
Sterropopelnut,Sterope,Pecanut
Proximaple,Proxima Centauri,Apple
Alphacenternut,Alpha Centauri,Butternut
Barnardstaroak,Barnard's Star,Oak
Wolfwood,Wolf 359,Dogwood
Lalandelarch,Lalande 21185,Larch
Siriuswood,Sirius,Basswood
Luytenbirch,Luyten 726-8,Birch
Rossnut,Ross 154,Coconut
Rossberry,Ross 248,Elderberry
Epsilonwood,Epsilon Eridani,Sandalwood
Lacaillemwood,Lacaille 9352,Elmwood
Rosswood,Ross 128,Cottonwood
EZaquariuspen,EZ Aquarii,Aspen
Procyonix,Procyon,Phoenix
Sixtyonecygnust,61 Cygni,Locust
Struvewood,Struve 2398,Rosewood
Groombridgefir,Groombridge 34,Fir
Epsilondiwood,Epsilon Indi,Indigowood
DXcancroak,DX Cancri,Oak
Taucetialder,Tau Ceti,Alder
Luytenut,Luyten's Star,Butternut
Teegardeenspen,Teegarden's Star,Aspen
Kapteynsnut,Kapteyn's Star,Coconut
Lacaillewood,Lacaille 8760,Lacewood
Krugernut,Krüger 60,Coconut
Rosspress,Ross 614,Cypress
Wolfwoodogwood,Wolf 1061,Dogwood
Vanmaanensnut,Van Maanen's Star,Chestnut
Gliesetree,Gliese 1,Tree
Gielenseginkgo,Gliese 876,Ginkgo
Giseewood,Gliese 581,Rosewood
TRappistree,TRAPPIST-1,Trapwood
Gliesecherry,Gliese 667,Cherry
Glisewood,Gliese 832,Walnutwood
Glaesewood,Gliese 163,Olivewood
Gleasewood,Gliese 180,Teaselwood
Lessewood,Gliese 221,Lesserwood
Sleewood,Gliese 317,Sleekwood
Liewood,Gliese 357,Tree
Gleetree,Gliese 433,Tree
Graywood,Gliese 436,Graywood
Greatwood,Gliese 504,Greatwood
Greenwood,Gliese 570,Greenwood
Greshwood,Gliese 625,Ashwood
Gladewood,Gliese 649,Glade
Glowwood,Gliese 674,Glowwood
Gradewood,Gliese 682,Gradewood
Gracewood,Gliese 686,Gracewood
Grandewood,Gliese 785,Grandewood
Glintwood,Gliese 806,Flintwood
Grimwood,Gliese 832,Grimwood
Horsehead Hickory,Horsehead Nebula,Hickory
Eaglenberry,Eagle Nebula,Elderberry
Trifidogwood,Trifid Nebula,Dogwood
Lagoonwood,Lagoon Nebula,Lagoonwood
Rosettamarisk,Rosette Nebula,Tamarisk
Carinapress,Carina Nebula,Cypress
Veilbark,Veil Nebula,Bark
Northamericoak,North America Nebula,Oak
Flamingowan,Flaming Star Nebula,Rowan
Owlnut,Owl Nebula,Walnut
Catseyelm,Cat's Eye Nebula,Elm
Ringebula,Ring Nebula,Nebula
Dumbbellnut,Dumbbell Nebula,Butternut
Boomerangalm,Boomerang Nebula,Palm
Butterflypress,Butterfly Nebula,Cypress
Calasash,Calabash Nebula,Ash
Redspiderash,Red Spider Nebula,Ash
Eskimogany,Eskimo Nebula,Mahogany
Ghostjuniper,Ghost of Jupiter,Juniper
Helixogwood,Helix Nebula,Dogwood
Spiralpress,Spiral Planetary,Cypress
Littlegembark,Little Gem Nebula,Bark
Necklacacia,Necklace Nebula,Acacia
Saturnbeech,Saturn Nebula,Beech
Bluesnowbark,Blue Snowball,Bark
Twinjet Fir,Twin Jet Nebula,Fir
Hourglasswood,Hourglass Nebula,Boxwood
Eggwood,Egg Nebula,Dogwood
Boxtree,Box Nebula,Boxtree
Crescentpress,Crescent Nebula,Cypress
Bubblebark,Bubble Nebula,Bark
Thorwood,Thor's Helmet,Thornwood
Snailgum,Snail Nebula,Sweetgum
Conepinepress,Cone Nebula,Pine
Irisbeech,Iris Nebula,Beech
Witchheadash,Witch Head Nebula,Ash
Barnardloopwood,Barnard's Loop,Loopwood
Californiapress,California Nebula,Cypress
Heartbark,Heart Nebula,Bark
Soulspuce,Soul Nebula,Spruce
Pacmangany,Pacman Nebula,Mahogany
Monkeyheadnut,Monkey Head Nebula,Monkeynut
Wizardlm,Wizard Nebula,Elm
Tulipwood,Tulip Nebula,Tulipwood
Crescentbark,Crescent Nebula,Bark
Pelicangum,Pelican Nebula,Gum
Elephantstrunkwood,Elephant's Trunk,Trunkwood
Tuningforkpress,Tuning Fork Galaxy,Cypress
Sombrerowood,Sombrero Galaxy,Sombrerowood
Blackeyesycamore,Black Eye Galaxy,Sycamore
Sunflowernut,Sunflower Galaxy,Coconut
Pinwheelder,Pinwheel Galaxy,Elder
Fireworksgalaxyalder,Fireworks Galaxy,Alder
Spindlepress,Spindle Galaxy,Cypress
Bodesgalaxnut,Bode's Galaxy,Butternut
Cigargalaxbirch,Cigar Galaxy,Birch
Whirlpoolepress,Whirlpool Galaxy,Cypress
Cartwheelalder,Cartwheel Galaxy,Alder
Tadpolewood,Tadpole Galaxy,Tadwood
Antenaennut,Antennae Galaxies,Nut
Mousesgalaxielm,Mice Galaxies,Elm
Atomspeachachy,Atom's Peace Galaxy,Peachy
Malinsgalaxnut,Malin 1,Nut
Hoagsobjecteakwood,Hoag's Object,Teakwood
Ringgalaxynut,Ring Galaxy,Donut
Polarringiron,Polar Ring Galaxy,Ironwood
Silversliverwood,Silver Sliver Galaxy,Silverwood
Needlepress,Needle Galaxy,Cypress
Hamburgerash,Hamburger Galaxy,Ash
1 Name Component_1 Component_2
2 Betelginkgo Betelgeuse Ginkgo
3 Pleiadedar Pleiades Cedar
4 Helixypress Helix Nebula Cypress
5 Rigelmwood Rigel Elmwood
6 Sombreroak Sombrero Galaxy Oak
7 Hyadehorn Hyades Hawthorn
8 Triangularch Triangulum Larch
9 Vegalder Vega Alder
10 Whirlpoplar Whirpool Galaxy Poplar
11 Siriuswood Sirius Basswood
12 Oortcacia Oort Cloud Acacia
13 Denebony Deneb Ebony
14 Cartwhillow Cartwheel Galaxy Willow
15 Polariswood Polaris Ironwood
16 Craborwood Crab Nebula Hornbeam
17 Aldebaranyan Aldebaran Banyan
18 Pinwheelnut Pinwheel Galaxy Walnut
19 Antaresnut Antares Chestnut
20 Kuiperlpa Kuiper Belt Catalpa
21 Arcturuspruce Arcturus Spruce
22 Pegasafras Pegasus Sassafras
23 Capellaberry Capella Mulberry
24 Lyracacia Lyra Acacia
25 Polluxwood Pollux Boxwood
26 Scorpiumac Scorpius Sumac
27 Castorwood Castor Cottonwood
28 Dracodar Draco Cedar
29 Regulushua Regulus Joshua
30 Aquilnut Aquila Walnut
31 Spicamore Spica Sycamore
32 Cygnusswood Cygnus Sasswood
33 Proceyon Procyon Pecan
34 Perseusqoia Perseus Sequoia
35 Sagittalpa Sagittarius Catalpa
36 Tauruspen Taurus Aspen
37 Geminowan Gemini Rowan
38 Canceria Cancer Wisteria
39 Leogany Leo Mahogany
40 Virgoak Virgo Oak
41 Libraobab Libra Baobab
42 Ariescalyptus Aries Eucalyptus
43 Capricornia Capricorn Magnolia
44 Aquariusqoia Aquarius Sequoia
45 Piscesimmon Pisces Persimmon
46 Omeganolia Omega Centauri Magnolia
47 Beehivewood Beehive Cluster Olivewood
48 Pillarspress Pillars of Creation Cypress
49 Greatwillow Great Wall Willow
50 Localboobao Local Bubble Baobab
51 Termishoak Termination Shock Oak
52 Bowshocknut Bow Shock Chestnut
53 Stellarix Stellar Wind Tamarix
54 Heliopaulownia Heliopause Paulownia
55 Rochemewood Roche Limit Limewood
56 Eventhornbeam Event Horizon Hornbeam
57 Accretioak Accretion Disk Oak
58 Protoplanetree Protoplanetary Disk Planetree
59 Lagranginkgo Lagrange Point Ginkgo
60 Trojaniper Trojan Juniper
61 Gammarchwood Gamma Ray Burst Birchwood
62 Hypernowan Hypernova Rowan
63 Whitedwillow White Dwarf Willow
64 Redgianter Red Giant Red Alder
65 Superclusternut Supercluster Butternut
66 Filamenteakwood Filament Teakwood
67 Voidglass Void Douglass Fir
68 Greywallnut Great Wall Wallnut
69 Comettree Comet Peppertree
70 Asteroider Asteroid Belt Elder
71 Helixelkova Helix Nebula Zelkova
72 Orionwood Orion Nebula Redwood
73 Cassiopaple Cassiopeia Apple
74 Ursajortree Ursa Major Majortree
75 Ursaminorwood Ursa Minor Dogwood
76 Bristlecrab Bristlecone Pine Crab Nebula
77 Metasequoyasar Metasequoia Quasar
78 Monkeypleiades Monkey Puzzle Pleiades
79 Dragontree Dragon Tree Draco
80 Boojumbrero Boojum Sombrero Galaxy
81 Quiverwheel Quiver Tree Pinwheel
82 Joshuapiter Joshua Tree Jupiter
83 Wolleminebula Wollemi Pine Nebula
84 Dawndromeda Dawn Redwood Andromeda
85 Baldcygnuss Bald Cypress Cygnus
86 Zelkovega Zelkova Vega
87 Hornbeamstar Hornbeam Barnstar
88 Rowantares Rowan Antares
89 Basswoodburst Basswood Gamma Burst
90 Calabashea Calabash Cassiopeia
91 Neemeda Neem Andromeda
92 Salirius Sal Sirius
93 Tooniverse Toon Universe
94 Cassiapeia Cassia Cassiopeia
95 Yellowoodrion Yellowwood Orion
96 Blackwoodhole Blackwood Black Hole
97 Ironwoodsar Ironwood Quasar
98 Satinwhirl Satinwood Whirlpool
99 Poplarsar Poplar Pulsar
100 Cottonwoodway Cottonwood Milky Way
101 Sumacnova Sumac Supernova
102 Hazeleius Hazel Pegasus
103 Elderstar Elder Barnstar
104 Rigelbark Rigel Bark
105 Sirillow Sirius Willow
106 Altairix Altair Tamarix
107 Denebula Deneb Nebula
108 Polarispine Polaris Pine
109 Betelpine Betelgeuse Bristlecone
110 Rigellarch Rigel Larch
111 Vegamore Vega Sycamore
112 Altairnut Altair Chestnut
113 Arcturuscacia Arcturus Acacia
114 Capellaple Capella Apple
115 Polluxdogwood Pollux Dogwood
116 Castorong Castor Camphorwood
117 Regulusqoia Regulus Sequoia
118 Spicape Spica Grape
119 Procyonwood Procyon Sandalwood
120 Aldebaranyan Aldebaran Banyan
121 Antarespress Antares Cypress
122 Pleiadwood Pleiades Cedarwood
123 Hyadesmac Hyades Sumac
124 Beehivewood Beehive Olivewood
125 Orionniper Orion Juniper
126 Cassiopear Cassiopeia Pear
127 Dracoroak Draco Cork Oak
128 Lyralnut Lyra Walnut
129 Cygnust Cygnus Locust
130 Aquillow Aquila Willow
131 Perseusnut Perseus Butternut
132 Pegasuspen Pegasus Aspen
133 Andromelm Andromeda Elm
134 Triangulfir Triangulum Fir
135 Sombreruce Sombrero Spruce
136 Whirlpinecone Whirlpool Pine
137 Cartwheelnut Cartwheel Hazelnut
138 Helixelm Helix Elm
139 Crabapple Crab Nebula Crabapple
140 Pillarspen Pillars Aspen
141 Sagittariuswood Sagittarius Sandalwood
142 Scorpiuspen Scorpius Aspen
143 Taurusqoia Taurus Sequoia
144 Geminoak Gemini Oak
145 Cancerdar Cancer Cedar
146 Leowood Leo Boxwood
147 Virgowan Virgo Rowan
148 Librasswood Libra Basswood
149 Ariesnut Aries Chestnut
150 Capricornbeam Capricorn Hornbeam
151 Aquariuslder Aquarius Alder
152 Piscespress Pisces Cypress
153 Magnetardwood Magnetar Hardwood
154 Pulsarwood Pulsar Rosewood
155 Quasarberry Quasar Elderberry
156 Supernovakia Supernova Plumeria
157 Hypernovania Hypernova Paulownia
158 Blackholly Black Hole Holly
159 Whitedwarfpine White Dwarf White Pine
160 Redgiantwood Red Giant Giantwood
161 Bluegiganticacia Blue Giant Acacia
162 Neutronia Neutron Star Catalpa
163 Browndwarfnut Brown Dwarf Butternut
164 Protostarfir Protostar Fir
165 Mainsequoia Main Sequence Sequoia
166 Subergiantree Supergiant Tree
167 Yorkcloud Oort Cloud Cork
168 Kuiperlpa Kuiper Belt Catalpa
169 Heliospherical Heliosphere Spherical Tree
170 Heliopaulogany Heliopause Mahogany
171 Terminationoak Termination Shock Oak
172 Bowshockberry Bow Shock Hackberry
173 Stellarwindwood Stellar Wind Driftwood
174 Solarwindowan Solar Wind Rowan
175 Interstellarch Interstellar Medium Larch
176 Circumstellarix Circumstellar Disk Tamarix
177 Protoplanetree Protoplanetary Disk Plane Tree
178 Accretionoak Accretion Disk Oak
179 Eventhorizonwood Event Horizon Ironwood
180 Schwarzschillow Schwarzschild Radius Willow
181 Singularsimmon Singularity Persimmon
182 Rochelimewood Roche Limit Limewood
183 Lagrangewood Lagrange Point Rangewood
184 Trojantree Trojan Asteroids Tree
185 Hildazel Hilda Group Hazel
186 Amoribark Amor Asteroids Bark
187 Atenberry Aten Asteroids Elderberry
188 Apollomeria Apollo Asteroids Plumeria
189 Gammaburst Ash Gamma Ray Burst Ash
190 Xraybirch X-ray Binary Birch
191 Millisecwood Millisecond Pulsar Redwood
192 Magnetarch Magnetar Larch
193 Softgammawood Soft Gamma Repeater Softwood
194 Superclusternut Supercluster Clusternut
195 Galaxyclusterpine Galaxy Cluster Pine
196 Galaxygroupfir Galaxy Group Fir
197 Localgrowan Local Group Rowan
198 Virgoclusteroak Virgo Cluster Oak
199 Comaberrynicaceae Coma Berenices Berry
200 Greatwallnut Great Wall Wallnut
201 Sloangreatwillow Sloan Great Wall Willow
202 Herculessuperia Hercules Supercluster Wisteria
203 Shapleypress Shapley Supercluster Cypress
204 Laniakeroak Laniakea Supercluster Kauri Oak
205 Cosmicwebwood Cosmic Web Webwood
206 Filamentree Filament Tree
207 Voidwood Void Driftwood
208 Boötesvoidwood Boötes Void Olivewood
209 Sculptornut Sculptor Void Coconut
210 Greywallash Great Wall Ash
211 Localbubbaobab Local Bubble Baobab
212 Loopicacia Local Interstellar Cloud Acacia
213 Gouldspen Gould Belt Aspen
214 Orionspurmac Orion Spur Sumac
215 Sagittariusarmwood Sagittarius Arm Armwood
216 Perseusarmpine Perseus Arm Pine
217 Outerarmoak Outer Arm Oak
218 Scutumcentauruswood Scutum-Centaurus Arm Scutumwood
219 Normamalder Norma Arm Alder
220 Barredspiralex Barred Spiral Ilex
221 Ellipticacacia Elliptical Galaxy Acacia
222 Irregularnut Irregular Galaxy Butternut
223 Lenticularcorn Lenticular Galaxy Acorn
224 Dwarfnut Dwarf Galaxy Chestnut
225 Satelliteelm Satellite Galaxy Elm
226 Starburstfir Starburst Galaxy Fir
227 Seyfertree Seyfert Galaxy Feather Tree
228 Blazarwood Blazar Rosewood
229 Radiogalaxypress Radio Galaxy Cypress
230 Activenutcleus Active Galactic Nucleus Nut
231 Quasartree Quasar Quandong Tree
232 Ultraluminouswood Ultraluminous Galaxy Luminous Tree
233 Luminousredgiant Luminous Red Galaxy Red Giant Sequoia
234 Greenpeachia Green Pea Galaxy Peach
235 Polarringwood Polar Ring Galaxy Ringwood
236 Shellgalaxash Shell Galaxy Ash
237 Ringalaxypress Ring Galaxy Cypress
238 Hoagsobjectoak Hoag's Object Oak
239 Mayallsobjectree Mayall's Object Tree
240 Tadpolgalaxillow Tadpole Galaxy Willow
241 Cometgalaxalder Comet Galaxy Alder
242 Sunflowernut Sunflower Galaxy Sunflower
243 Blackeyewood Black Eye Galaxy Blackwood
244 Bodesgalaxnut Bode's Galaxy Coconut
245 Cigargalaxpress Cigar Galaxy Cypress
246 Mousesgalaxypen Mice Galaxies Aspen
247 Antennaegalaxynt Antennae Galaxies Antwood
248 Polarisstwood Polaris Sasswood
249 Rigwood Rigel Dogwood
250 Canopusnut Canopus Coconut
251 Vegamore Vega Sycamore
252 Capellalnut Capella Walnut
253 Rigelberry Rigel Mulberry
254 Procyonwood Procyon Cottonwood
255 Achernarnut Achernar Butternut
256 Betelgeuswood Betelgeuse Geuswood
257 Hadarcedarwood Hadar Cedarwood
258 Altairix Altair Tamarix
259 Alderaminwood Alderamin Alderwood
260 Denebolive Denebola Olive
261 Mimosacia Mimosa Acacia
262 Regulenut Regulus Filbert
263 Adharalnut Adhara Walnut
264 Shahelmwood Shaula Elm
265 Gacruxifir Gacrux Fir
266 Bellatrixnut Bellatrix Hazelnut
267 Elnathnut Elnath Chestnut
268 Miaplacidiumwood Miaplacidus Palladium
269 Alnitaknut Alnitak Coconut
270 Alnilamspen Alnilam Aspen
271 Aliothogany Alioth Mahogany
272 Mirzamberry Mirzam Mulberry
273 Alkaidnut Alkaid Butternut
274 Atriarch Atria Larch
275 Alhenawood Alhena Alderwood
276 Peacockahogany Peacock Mahogany
277 Mirfaknut Mirfak Coconut
278 Wezenalder Wezen Alder
279 Sargassage Sargas Sage
280 Menkarberry Menkar Elderberry
281 Eniffir Enif Fir
282 Denebalgedinut Deneb Algedi Butternut
283 Zubenhaknutbi Zuben Hakrabi Butternut
284 Alshainalnut Alshain Walnut
285 Rastabantamarind Rastaban Tamarind
286 Markabwood Markab Markwood
287 Schedarogany Schedar Mahogany
288 Almaalnutak Almaak Walnut
289 Rasalgetash Rasalgethi Ash
290 Thubanyan Thuban Banyan
291 Kausafras Kaus Australis Sassafras
292 Nunkirnut Nunki Coconut
293 Sabikwood Sabik Rosewood
294 Menkalnutnan Menkalnan Butternut
295 Asteropewood Asterope Redwood
296 Atlaspen Atlas Aspen
297 Electralder Electra Alder
298 Maiagany Maia Mahogany
299 Meropewood Merope Olivewood
300 Taugetamarix Taygeta Tamarix
301 Pleionetree Pleione Tree
302 Celaenoak Celaeno Oak
303 Alcyonedar Alcyone Cedar
304 Sterropopelnut Sterope Pecanut
305 Proximaple Proxima Centauri Apple
306 Alphacenternut Alpha Centauri Butternut
307 Barnardstaroak Barnard's Star Oak
308 Wolfwood Wolf 359 Dogwood
309 Lalandelarch Lalande 21185 Larch
310 Siriuswood Sirius Basswood
311 Luytenbirch Luyten 726-8 Birch
312 Rossnut Ross 154 Coconut
313 Rossberry Ross 248 Elderberry
314 Epsilonwood Epsilon Eridani Sandalwood
315 Lacaillemwood Lacaille 9352 Elmwood
316 Rosswood Ross 128 Cottonwood
317 EZaquariuspen EZ Aquarii Aspen
318 Procyonix Procyon Phoenix
319 Sixtyonecygnust 61 Cygni Locust
320 Struvewood Struve 2398 Rosewood
321 Groombridgefir Groombridge 34 Fir
322 Epsilondiwood Epsilon Indi Indigowood
323 DXcancroak DX Cancri Oak
324 Taucetialder Tau Ceti Alder
325 Luytenut Luyten's Star Butternut
326 Teegardeenspen Teegarden's Star Aspen
327 Kapteynsnut Kapteyn's Star Coconut
328 Lacaillewood Lacaille 8760 Lacewood
329 Krugernut Krüger 60 Coconut
330 Rosspress Ross 614 Cypress
331 Wolfwoodogwood Wolf 1061 Dogwood
332 Vanmaanensnut Van Maanen's Star Chestnut
333 Gliesetree Gliese 1 Tree
334 Gielenseginkgo Gliese 876 Ginkgo
335 Giseewood Gliese 581 Rosewood
336 TRappistree TRAPPIST-1 Trapwood
337 Gliesecherry Gliese 667 Cherry
338 Glisewood Gliese 832 Walnutwood
339 Glaesewood Gliese 163 Olivewood
340 Gleasewood Gliese 180 Teaselwood
341 Lessewood Gliese 221 Lesserwood
342 Sleewood Gliese 317 Sleekwood
343 Liewood Gliese 357 Tree
344 Gleetree Gliese 433 Tree
345 Graywood Gliese 436 Graywood
346 Greatwood Gliese 504 Greatwood
347 Greenwood Gliese 570 Greenwood
348 Greshwood Gliese 625 Ashwood
349 Gladewood Gliese 649 Glade
350 Glowwood Gliese 674 Glowwood
351 Gradewood Gliese 682 Gradewood
352 Gracewood Gliese 686 Gracewood
353 Grandewood Gliese 785 Grandewood
354 Glintwood Gliese 806 Flintwood
355 Grimwood Gliese 832 Grimwood
356 Horsehead Hickory Horsehead Nebula Hickory
357 Eaglenberry Eagle Nebula Elderberry
358 Trifidogwood Trifid Nebula Dogwood
359 Lagoonwood Lagoon Nebula Lagoonwood
360 Rosettamarisk Rosette Nebula Tamarisk
361 Carinapress Carina Nebula Cypress
362 Veilbark Veil Nebula Bark
363 Northamericoak North America Nebula Oak
364 Flamingowan Flaming Star Nebula Rowan
365 Owlnut Owl Nebula Walnut
366 Catseyelm Cat's Eye Nebula Elm
367 Ringebula Ring Nebula Nebula
368 Dumbbellnut Dumbbell Nebula Butternut
369 Boomerangalm Boomerang Nebula Palm
370 Butterflypress Butterfly Nebula Cypress
371 Calasash Calabash Nebula Ash
372 Redspiderash Red Spider Nebula Ash
373 Eskimogany Eskimo Nebula Mahogany
374 Ghostjuniper Ghost of Jupiter Juniper
375 Helixogwood Helix Nebula Dogwood
376 Spiralpress Spiral Planetary Cypress
377 Littlegembark Little Gem Nebula Bark
378 Necklacacia Necklace Nebula Acacia
379 Saturnbeech Saturn Nebula Beech
380 Bluesnowbark Blue Snowball Bark
381 Twinjet Fir Twin Jet Nebula Fir
382 Hourglasswood Hourglass Nebula Boxwood
383 Eggwood Egg Nebula Dogwood
384 Boxtree Box Nebula Boxtree
385 Crescentpress Crescent Nebula Cypress
386 Bubblebark Bubble Nebula Bark
387 Thorwood Thor's Helmet Thornwood
388 Snailgum Snail Nebula Sweetgum
389 Conepinepress Cone Nebula Pine
390 Irisbeech Iris Nebula Beech
391 Witchheadash Witch Head Nebula Ash
392 Barnardloopwood Barnard's Loop Loopwood
393 Californiapress California Nebula Cypress
394 Heartbark Heart Nebula Bark
395 Soulspuce Soul Nebula Spruce
396 Pacmangany Pacman Nebula Mahogany
397 Monkeyheadnut Monkey Head Nebula Monkeynut
398 Wizardlm Wizard Nebula Elm
399 Tulipwood Tulip Nebula Tulipwood
400 Crescentbark Crescent Nebula Bark
401 Pelicangum Pelican Nebula Gum
402 Elephantstrunkwood Elephant's Trunk Trunkwood
403 Tuningforkpress Tuning Fork Galaxy Cypress
404 Sombrerowood Sombrero Galaxy Sombrerowood
405 Blackeyesycamore Black Eye Galaxy Sycamore
406 Sunflowernut Sunflower Galaxy Coconut
407 Pinwheelder Pinwheel Galaxy Elder
408 Fireworksgalaxyalder Fireworks Galaxy Alder
409 Spindlepress Spindle Galaxy Cypress
410 Bodesgalaxnut Bode's Galaxy Butternut
411 Cigargalaxbirch Cigar Galaxy Birch
412 Whirlpoolepress Whirlpool Galaxy Cypress
413 Cartwheelalder Cartwheel Galaxy Alder
414 Tadpolewood Tadpole Galaxy Tadwood
415 Antenaennut Antennae Galaxies Nut
416 Mousesgalaxielm Mice Galaxies Elm
417 Atomspeachachy Atom's Peace Galaxy Peachy
418 Malinsgalaxnut Malin 1 Nut
419 Hoagsobjecteakwood Hoag's Object Teakwood
420 Ringgalaxynut Ring Galaxy Donut
421 Polarringiron Polar Ring Galaxy Ironwood
422 Silversliverwood Silver Sliver Galaxy Silverwood
423 Needlepress Needle Galaxy Cypress
424 Hamburgerash Hamburger Galaxy Ash

View file

@ -1,420 +0,0 @@
Name,Component_1,Component_2
Pleiadeiba,Pleiades,Ceiba
Pau Brasilica,Pau Brasil,Crab Nebula
Araucarina,Araucaria,Carina Nebula
Quebracho X-1,Quebracho,Cygnus X-1
Lapachomeda,Lapacho,Andromeda
Guanacastellum,Guanacaste,Magellanic Cloud
Trianguanum,Triangulum,Guanacaste
Jacarandromeda,Jacaranda,Andromeda
Ceibalaxy,Ceiba,Galaxy
Horsehead Brasil,Horsehead,Pau Brasil
Monkeypuzzle Void,Monkey Puzzle,Cosmic Void
Quebrachion,Quebracho,Orion
Lapachosar,Lapacho,Quasar
Rubber Rosette,Rubber Tree,Rosette Nebula
Cashewmeda,Cashew,Andromeda
Brazil Nutnova,Brazil Nut,Supernova
Cecropiades,Cecropia,Pleiades
Balsombrero,Balsa,Sombrero Galaxy
Rosewoodway,Rosewood,Milky Way
Ipe Helix,Ipe,Helix Nebula
Kapoksagittarius,Kapok,Sagittarius
Cedrella Cluster,Cedrella,Star Cluster
Alerce Lyra,Alerce,Lyra
Mahogalactica,Mahogany,Galactic Center
Pinwheelnut,Pinwheel,Brazil Nut
Tarantularch,Tarantula Nebula,Larch
Ceibomeda,Ceiba,Andromeda
Pau Brassiopeia,Pau Brasil,Cassiopeia
Araucarion,Araucaria,Orion
Quebrachole,Quebracho,Black Hole
Lapachoway,Lapacho,Milky Way
Guanasar,Guanacaste,Quasar
Jacaranebula,Jacaranda,Nebula
Ceiba Centauri,Ceiba,Alpha Centauri
Brazilbeehive,Brazil Nut,Beehive Cluster
Monkeypuzzar,Monkey Puzzle,Quasar
Quebrachomeda,Quebracho,Andromeda
Lapachydes,Lapacho,Hyades
Rubberway,Rubber Tree,Milky Way
Cashewsar,Cashew,Quasar
Nutmeda,Brazil Nut,Andromeda
Cecropiagalaxy,Cecropia,Galaxy
Balsanova,Balsa,Supernova
Rosettewood,Rosette,Rosewood
Ipegalaxy,Ipe,Galaxy
Kapoknebula,Kapok,Nebula
Cedrellaxy,Cedrella,Galaxy
Alercentauri,Alerce,Centaurus
Mahogacrab,Mahogany,Crab Nebula
Eaglebrasil,Eagle Nebula,Pau Brasil
Tarantuleia,Tarantula Nebula,Ceiba
Whirlpoolnut,Whirlpool,Brazil Nut
Araucagalaxy,Araucaria,Galaxy
Quebrachiades,Quebracho,Pleiades
Lapachomeda,Lapacho,Andromeda
Guanacasteroid,Guanacaste,Asteroid
Jacarandromeda,Jacaranda,Andromeda
Ceibulsar,Ceiba,Pulsar
Tadpolebrasil,Tadpole,Pau Brasil
Monkeynova,Monkey Puzzle,Supernova
Quebrachygnus,Quebracho,Cygnus
Lapachoacentauri,Lapacho,Alpha Centauri
Rubberhole,Rubber Tree,Black Hole
Cashewnova,Cashew,Supernova
Brazilbutterfly,Brazil Nut,Butterfly Cluster
Cecropiomeda,Cecropia,Andromeda
Balsagittarius,Balsa,Sagittarius
Rosewoodrion,Rosewood,Orion
Ipeleiades,Ipe,Pleiades
Kapoksombrero,Kapok,Sombrero Galaxy
Cedrellipse,Cedrella,Elliptical Galaxy
Alercegalaxy,Alerce,Galaxy
Mahogaring,Mahogany,Ring Nebula
Cartwheeiba,Cartwheel,Ceiba
Pau Brasilian,Pau Brasil,Cassiopeian
Araucarionebula,Araucaria,Orion Nebula
Quebrasar,Quebracho,Quasar
Lapachomega,Lapacho,Omega Centauri
Guanacastway,Guanacaste,Milky Way
Jacaranovae,Jacaranda,Supernova
Ceibachomeda,Ceiba,Andromeda
Sunflowerbrasil,Sunflower Galaxy,Pau Brasil
Monkeypuzzalaxy,Monkey Puzzle,Galaxy
Quebrachoxima,Quebracho,Proxima
Lapachosar,Lapacho,Quasar
Rubberbula,Rubber Tree,Nebula
Cashewcluster,Cashew,Star Cluster
Nutway,Brazil Nut,Milky Way
Cecropianebula,Cecropia,Nebula
Balsagara,Balsa,Sagittarius A*
Rosewoodraco,Rosewood,Draco
Ipelipse,Ipe,Ellipse
Kapokagoon,Kapok,Lagoon Nebula
Cedrellaneta,Cedrella,Planetary Nebula
Alercerion,Alerce,Orion
Mahogacarina,Mahogany,Carina
Trifidnut,Trifid Nebula,Brazil Nut
Tarantulacho,Tarantula,Lapacho
Ceibanova,Ceiba,Supernova
Pau Brasilion,Pau Brasil,Orion
Araucariades,Araucaria,Pleiades
Quebrachoelix,Quebracho,Helix
Lapacholia,Lapacho,Magnolia
Guanacasar,Guanacaste,Quasar
Jacarandromeda,Jacaranda,Andromeda
Ceibulsar,Ceiba,Pulsar
Titan Brasil,Titan,Pau Brasil
Monkeynebula,Monkey Puzzle,Nebula
Quebrachomeda,Quebracho,Andromeda
Lapachogalaxy,Lapacho,Galaxy
Rubbersar,Rubber Tree,Quasar
Cashewmeda,Cashew,Andromeda
Nutcluster,Brazil Nut,Cluster
Cecropioway,Cecropia,Milky Way
Balsanebula,Balsa,Nebula
Rosewoodway,Rosewood,Milky Way
Ipegalaxy,Ipe,Galaxy
Kapoksar,Kapok,Quasar
Cedrellaxy,Cedrella,Galaxy
Alercenova,Alerce,Supernova
Mahogalactic,Mahogany,Galactic
Sculptorbrasil,Sculptor Galaxy,Pau Brasil
Ceibomeda,Ceiba,Andromeda
Araucarionway,Araucaria,Milky Way
Quebrachion,Quebracho,Orion
Lapachohole,Lapacho,Black Hole
Guanacastromeda,Guanacaste,Andromeda
Jacaranebula,Jacaranda,Nebula
Ceibagalaxy,Ceiba,Galaxy
Europa Brasil,Europa,Pau Brasil
Monkeypuzzar,Monkey Puzzle,Quasar
Quebrachnova,Quebracho,Supernova
Lapachoides,Lapacho,Hyades
Rubbermeda,Rubber Tree,Andromeda
Cashewnova,Cashew,Supernova
Nutway,Brazil Nut,Milky Way
Cecropiagalaxy,Cecropia,Galaxy
Balsaquila,Balsa,Aquila
Rosewoodsar,Rosewood,Quasar
Ipecentauri,Ipe,Alpha Centauri
Kapokgalaxy,Kapok,Galaxy
Cedrellaneta,Cedrella,Planetary Nebula
Alercemeda,Alerce,Andromeda
Mahoganova,Mahogany,Supernova
Boomerangbrasil,Boomerang Nebula,Pau Brasil
Ceibulsar,Ceiba,Pulsar
Pau Brasilades,Pau Brasil,Pleiades
Araucagalaxy,Araucaria,Galaxy
Quebrachomeda,Quebracho,Andromeda
Lapachosar,Lapacho,Quasar
Guanacastellum,Guanacaste,Magellanic
Jacaranovae,Jacaranda,Supernova
Ceibaway,Ceiba,Milky Way
Enceladnut,Enceladus,Brazil Nut
Monkeynebula,Monkey Puzzle,Nebula
Quebrachogalaxy,Quebracho,Galaxy
Lapachonova,Lapacho,Supernova
Rubberion,Rubber Tree,Orion
Cashewgalaxy,Cashew,Galaxy
Nutsagittarius,Brazil Nut,Sagittarius
Cecropia A*,Cecropia,Sagittarius A*
Balsanova,Balsa,Supernova
Rosewoodmeda,Rosewood,Andromeda
Ipesar,Ipe,Quasar
Kapokway,Kapok,Milky Way
Cedrellipse,Cedrella,Ellipse
Alercenebula,Alerce,Nebula
Mahogaring,Mahogany,Ring Nebula
Cat's Eye Brasil,Cat's Eye,Pau Brasil
Ceibomeda,Ceiba,Andromeda
Araucarionebula,Araucaria,Orion Nebula
Quebrachoides,Quebracho,Hyades
Lapachogalaxy,Lapacho,Galaxy
Guanacastar,Guanacaste,Quasar
Jacarandromeda,Jacaranda,Andromeda
Ceibaneta,Ceiba,Planetary Nebula
Ganymede Brasil,Ganymede,Pau Brasil
Monkeypuzzalaxy,Monkey Puzzle,Galaxy
Quebrachonova,Quebracho,Supernova
Lapachomeda,Lapacho,Andromeda
Rubbersar,Rubber Tree,Quasar
Cashewnebula,Cashew,Nebula
Nutgalaxy,Brazil Nut,Galaxy
Cecropiomeda,Cecropia,Andromeda
Balsagittarius,Balsa,Sagittarius
Rosewoodion,Rosewood,Orion
Ipeleiades,Ipe,Pleiades
Kapoknova,Kapok,Supernova
Cedrellaxy,Cedrella,Galaxy
Alercegalaxy,Alerce,Galaxy
Mahogacrab,Mahogany,Crab
Dumbbell Brasil,Dumbbell Nebula,Pau Brasil
Ceibulsar,Ceiba,Pulsar
Pau Brasilway,Pau Brasil,Milky Way
Araucarina,Araucaria,Carina
Quebrachogalaxy,Quebracho,Galaxy
Lapachosar,Lapacho,Quasar
Guanacastromeda,Guanacaste,Andromeda
Jacaranebula,Jacaranda,Nebula
Ceibanova,Ceiba,Supernova
Io Brasil,Io,Pau Brasil
Monkeypuzzar,Monkey Puzzle,Quasar
Quebrachomeda,Quebracho,Andromeda
Lapachogalaxy,Lapacho,Galaxy
Rubbernova,Rubber Tree,Supernova
Cashewmeda,Cashew,Andromeda
Nutnebula,Brazil Nut,Nebula
Cecropiagalaxy,Cecropia,Galaxy
Balsasar,Balsa,Quasar
Rosewoodway,Rosewood,Milky Way
Ipegalaxy,Ipe,Galaxy
Kapokmeda,Kapok,Andromeda
Cedrellaneta,Cedrella,Planetary Nebula
Alercenova,Alerce,Supernova
Mahogalagoon,Mahogany,Lagoon
Crescent Brasil,Crescent Nebula,Pau Brasil
Ceibagalaxy,Ceiba,Galaxy
Araucarionway,Araucaria,Milky Way
Quebrachion,Quebracho,Orion
Lapachohole,Lapacho,Black Hole
Guanacastar,Guanacaste,Quasar
Jacaranovo,Jacaranda,Supernova
Ceibomeda,Ceiba,Andromeda
Callisto Brasil,Callisto,Pau Brasil
Monkeynebula,Monkey Puzzle,Nebula
Quebrachogalaxy,Quebracho,Galaxy
Lapachonova,Lapacho,Supernova
Rubbermeda,Rubber Tree,Andromeda
Cashewgalaxy,Cashew,Galaxy
Nutway,Brazil Nut,Milky Way
Cecropiomeda,Cecropia,Andromeda
Balsanebula,Balsa,Nebula
Rosewoodsar,Rosewood,Quasar
Ipecentauri,Ipe,Centauri
Kapokgalaxy,Kapok,Galaxy
Cedrellaxy,Cedrella,Galaxy
Alercemeda,Alerce,Andromeda
Mahoganova,Mahogany,Supernova
Flame Brasil,Flame Nebula,Pau Brasil
Ceibulsar,Ceiba,Pulsar
Pau Brasilion,Pau Brasil,Orion
Araucagalaxy,Araucaria,Galaxy
Quebrachomeda,Quebracho,Andromeda
Lapachosar,Lapacho,Quasar
Guanacastway,Guanacaste,Milky Way
Jacarandromeda,Jacaranda,Andromeda
Ceibanova,Ceiba,Supernova
Veil Brasil,Veil Nebula,Pau Brasil
Monkeypuzzalaxy,Monkey Puzzle,Galaxy
Quebrachonova,Quebracho,Supernova
Lapachogalaxy,Lapacho,Galaxy
Rubberion,Rubber Tree,Orion
Cashewnova,Cashew,Supernova
Nutgalaxy,Brazil Nut,Galaxy
Cecropiagalaxy,Cecropia,Galaxy
Balsomeda,Balsa,Andromeda
Rosewoodway,Rosewood,Milky Way
Ipesar,Ipe,Quasar
Kapoknebula,Kapok,Nebula
Cedrellaneta,Cedrella,Planetary Nebula
Alercenebula,Alerce,Nebula
Mahogaring,Mahogany,Ring
Pelican Brasil,Pelican Nebula,Pau Brasil
Ceibagalaxy,Ceiba,Galaxy
Araucarionebula,Araucaria,Orion Nebula
Quebrachoides,Quebracho,Hyades
Lapachomeda,Lapacho,Andromeda
Guanacastar,Guanacaste,Quasar
Jacaranovo,Jacaranda,Supernova
Ceibulsar,Ceiba,Pulsar
Cone Brasil,Cone Nebula,Pau Brasil
Monkeypuzzar,Monkey Puzzle,Quasar
Quebrachogalaxy,Quebracho,Galaxy
Lapachosar,Lapacho,Quasar
Rubbernova,Rubber Tree,Supernova
Cashewmeda,Cashew,Andromeda
Nutway,Brazil Nut,Milky Way
Cecropiomeda,Cecropia,Andromeda
Balsagalaxy,Balsa,Galaxy
Rosewoodion,Rosewood,Orion
Ipeleiades,Ipe,Pleiades
Kapokgalaxy,Kapok,Galaxy
Cedrellaxy,Cedrella,Galaxy
Alercegalaxy,Alerce,Galaxy
Mahoganova,Mahogany,Supernova
Witch Head Brasil,Witch Head,Pau Brasil
Ceiboway,Ceiba,Milky Way
Pau Brasilmeda,Pau Brasil,Andromeda
Araucarina,Araucaria,Carina
Quebrachonebula,Quebracho,Nebula
Lapachogalaxy,Lapacho,Galaxy
Guanacastromeda,Guanacaste,Andromeda
Jacaranebula,Jacaranda,Nebula
Ceibanova,Ceiba,Supernova
Iris Brasil,Iris Nebula,Pau Brasil
Monkeynova,Monkey Puzzle,Supernova
Quebrachomeda,Quebracho,Andromeda
Lapachosar,Lapacho,Quasar
Rubbermeda,Rubber Tree,Andromeda
Cashewgalaxy,Cashew,Galaxy
Nutnebula,Brazil Nut,Nebula
Cecropiagalaxy,Cecropia,Galaxy
Balsanovo,Balsa,Supernova
Rosewoodway,Rosewood,Milky Way
Ipegalaxy,Ipe,Galaxy
Kapokmeda,Kapok,Andromeda
Cedrellaneta,Cedrella,Planetary Nebula
Alercenova,Alerce,Supernova
Mahogagalaxy,Mahogany,Galaxy
Bubble Brasil,Bubble Nebula,Pau Brasil
Ceibomeda,Ceiba,Andromeda
Araucarionway,Araucaria,Milky Way
Quebrachion,Quebracho,Orion
Lapachohole,Lapacho,Black Hole
Guanacastar,Guanacaste,Quasar
Jacaranovo,Jacaranda,Supernova
Ceibagalaxy,Ceiba,Galaxy
North America Brasil,North America,Pau Brasil
Monkeypuzzar,Monkey Puzzle,Quasar
Quebrachogalaxy,Quebracho,Galaxy
Lapachonova,Lapacho,Supernova
Rubbersar,Rubber Tree,Quasar
Cashewnebula,Cashew,Nebula
Nutgalaxy,Brazil Nut,Galaxy
Cecropiomeda,Cecropia,Andromeda
Balsoway,Balsa,Milky Way
Rosewoodsar,Rosewood,Quasar
Ipecentauri,Ipe,Centauri
Kapokgalaxy,Kapok,Galaxy
Cedrellaxy,Cedrella,Galaxy
Alercemeda,Alerce,Andromeda
Mahoganova,Mahogany,Supernova
Pacman Brasil,Pacman Nebula,Pau Brasil
Ceibulsar,Ceiba,Pulsar
Pau Brasilades,Pau Brasil,Pleiades
Araucagalaxy,Araucaria,Galaxy
Quebrachomeda,Quebracho,Andromeda
Lapachosar,Lapacho,Quasar
Guanacastway,Guanacaste,Milky Way
Jacarandromeda,Jacaranda,Andromeda
Ceibanova,Ceiba,Supernova
Soul Brasil,Soul Nebula,Pau Brasil
Monkeynebula,Monkey Puzzle,Nebula
Quebrachonova,Quebracho,Supernova
Lapachogalaxy,Lapacho,Galaxy
Rubberion,Rubber Tree,Orion
Cashewnova,Cashew,Supernova
Nutway,Brazil Nut,Milky Way
Cecropiagalaxy,Cecropia,Galaxy
Balsameda,Balsa,Andromeda
Rosewoodway,Rosewood,Milky Way
Ipesar,Ipe,Quasar
Kapoknebula,Kapok,Nebula
Cedrellaneta,Cedrella,Planetary Nebula
Alercenebula,Alerce,Nebula
Mahogaring,Mahogany,Ring
Heart Brasil,Heart Nebula,Pau Brasil
Ceibagalaxy,Ceiba,Galaxy
Araucarionebula,Araucaria,Orion Nebula
Quebrachoides,Quebracho,Hyades
Lapachomeda,Lapacho,Andromeda
Guanacastar,Guanacaste,Quasar
Jacaranovo,Jacaranda,Supernova
Ceiboway,Ceiba,Milky Way
Elephant Trunk Brasil,Elephant Trunk,Pau Brasil
Monkeypuzzalaxy,Monkey Puzzle,Galaxy
Quebrachogalaxy,Quebracho,Galaxy
Lapachosar,Lapacho,Quasar
Rubbernova,Rubber Tree,Supernova
Cashewmeda,Cashew,Andromeda
Nutgalaxy,Brazil Nut,Galaxy
Cecropiomeda,Cecropia,Andromeda
Balsagalaxy,Balsa,Galaxy
Rosewoodion,Rosewood,Orion
Ipeleiades,Ipe,Pleiades
Kapokgalaxy,Kapok,Galaxy
Cedrellaxy,Cedrella,Galaxy
Alercegalaxy,Alerce,Galaxy
Mahoganova,Mahogany,Supernova
Omega Brasil,Omega Nebula,Pau Brasil
Ceibulsar,Ceiba,Pulsar
Pau Brasilion,Pau Brasil,Orion
Araucarina,Araucaria,Carina
Quebrachonebula,Quebracho,Nebula
Lapachogalaxy,Lapacho,Galaxy
Guanacastromeda,Guanacaste,Andromeda
Jacaranebula,Jacaranda,Nebula
Ceibanova,Ceiba,Supernova
Blinking Brasil,Blinking Nebula,Pau Brasil
Monkeypuzzar,Monkey Puzzle,Quasar
Quebrachomeda,Quebracho,Andromeda
Lapachosar,Lapacho,Quasar
Rubbermeda,Rubber Tree,Andromeda
Cashewgalaxy,Cashew,Galaxy
Nutnebula,Brazil Nut,Nebula
Cecropiagalaxy,Cecropia,Galaxy
Balsanovo,Balsa,Supernova
Rosewoodway,Rosewood,Milky Way
Ipegalaxy,Ipe,Galaxy
Kapokmeda,Kapok,Andromeda
Cedrellaneta,Cedrella,Planetary Nebula
Alercenova,Alerce,Supernova
Mahogagalaxy,Mahogany,Galaxy
Ghost Brasil,Ghost Nebula,Pau Brasil
Ceibomeda,Ceiba,Andromeda
Araucarionway,Araucaria,Milky Way
Quebrachion,Quebracho,Orion
Lapachohole,Lapacho,Black Hole
Guanacastar,Guanacaste,Quasar
Jacaranovo,Jacaranda,Supernova
Ceibagalaxy,Ceiba,Galaxy
Eskimo Brasil,Eskimo Nebula,Pau Brasil
Monkeynebula,Monkey Puzzle,Nebula
Quebrachogalaxy,Quebracho,Galaxy
Lapachonova,Lapacho,Supernova
Rubbersar,Rubber Tree,Quasar
Cashewnebula,Cashew,Nebula
Nutgalaxy,Brazil Nut,Galaxy
Cecropioway,Cecropia,Milky Way
1 Name Component_1 Component_2
2 Pleiadeiba Pleiades Ceiba
3 Pau Brasilica Pau Brasil Crab Nebula
4 Araucarina Araucaria Carina Nebula
5 Quebracho X-1 Quebracho Cygnus X-1
6 Lapachomeda Lapacho Andromeda
7 Guanacastellum Guanacaste Magellanic Cloud
8 Trianguanum Triangulum Guanacaste
9 Jacarandromeda Jacaranda Andromeda
10 Ceibalaxy Ceiba Galaxy
11 Horsehead Brasil Horsehead Pau Brasil
12 Monkeypuzzle Void Monkey Puzzle Cosmic Void
13 Quebrachion Quebracho Orion
14 Lapachosar Lapacho Quasar
15 Rubber Rosette Rubber Tree Rosette Nebula
16 Cashewmeda Cashew Andromeda
17 Brazil Nutnova Brazil Nut Supernova
18 Cecropiades Cecropia Pleiades
19 Balsombrero Balsa Sombrero Galaxy
20 Rosewoodway Rosewood Milky Way
21 Ipe Helix Ipe Helix Nebula
22 Kapoksagittarius Kapok Sagittarius
23 Cedrella Cluster Cedrella Star Cluster
24 Alerce Lyra Alerce Lyra
25 Mahogalactica Mahogany Galactic Center
26 Pinwheelnut Pinwheel Brazil Nut
27 Tarantularch Tarantula Nebula Larch
28 Ceibomeda Ceiba Andromeda
29 Pau Brassiopeia Pau Brasil Cassiopeia
30 Araucarion Araucaria Orion
31 Quebrachole Quebracho Black Hole
32 Lapachoway Lapacho Milky Way
33 Guanasar Guanacaste Quasar
34 Jacaranebula Jacaranda Nebula
35 Ceiba Centauri Ceiba Alpha Centauri
36 Brazilbeehive Brazil Nut Beehive Cluster
37 Monkeypuzzar Monkey Puzzle Quasar
38 Quebrachomeda Quebracho Andromeda
39 Lapachydes Lapacho Hyades
40 Rubberway Rubber Tree Milky Way
41 Cashewsar Cashew Quasar
42 Nutmeda Brazil Nut Andromeda
43 Cecropiagalaxy Cecropia Galaxy
44 Balsanova Balsa Supernova
45 Rosettewood Rosette Rosewood
46 Ipegalaxy Ipe Galaxy
47 Kapoknebula Kapok Nebula
48 Cedrellaxy Cedrella Galaxy
49 Alercentauri Alerce Centaurus
50 Mahogacrab Mahogany Crab Nebula
51 Eaglebrasil Eagle Nebula Pau Brasil
52 Tarantuleia Tarantula Nebula Ceiba
53 Whirlpoolnut Whirlpool Brazil Nut
54 Araucagalaxy Araucaria Galaxy
55 Quebrachiades Quebracho Pleiades
56 Lapachomeda Lapacho Andromeda
57 Guanacasteroid Guanacaste Asteroid
58 Jacarandromeda Jacaranda Andromeda
59 Ceibulsar Ceiba Pulsar
60 Tadpolebrasil Tadpole Pau Brasil
61 Monkeynova Monkey Puzzle Supernova
62 Quebrachygnus Quebracho Cygnus
63 Lapachoacentauri Lapacho Alpha Centauri
64 Rubberhole Rubber Tree Black Hole
65 Cashewnova Cashew Supernova
66 Brazilbutterfly Brazil Nut Butterfly Cluster
67 Cecropiomeda Cecropia Andromeda
68 Balsagittarius Balsa Sagittarius
69 Rosewoodrion Rosewood Orion
70 Ipeleiades Ipe Pleiades
71 Kapoksombrero Kapok Sombrero Galaxy
72 Cedrellipse Cedrella Elliptical Galaxy
73 Alercegalaxy Alerce Galaxy
74 Mahogaring Mahogany Ring Nebula
75 Cartwheeiba Cartwheel Ceiba
76 Pau Brasilian Pau Brasil Cassiopeian
77 Araucarionebula Araucaria Orion Nebula
78 Quebrasar Quebracho Quasar
79 Lapachomega Lapacho Omega Centauri
80 Guanacastway Guanacaste Milky Way
81 Jacaranovae Jacaranda Supernova
82 Ceibachomeda Ceiba Andromeda
83 Sunflowerbrasil Sunflower Galaxy Pau Brasil
84 Monkeypuzzalaxy Monkey Puzzle Galaxy
85 Quebrachoxima Quebracho Proxima
86 Lapachosar Lapacho Quasar
87 Rubberbula Rubber Tree Nebula
88 Cashewcluster Cashew Star Cluster
89 Nutway Brazil Nut Milky Way
90 Cecropianebula Cecropia Nebula
91 Balsagara Balsa Sagittarius A*
92 Rosewoodraco Rosewood Draco
93 Ipelipse Ipe Ellipse
94 Kapokagoon Kapok Lagoon Nebula
95 Cedrellaneta Cedrella Planetary Nebula
96 Alercerion Alerce Orion
97 Mahogacarina Mahogany Carina
98 Trifidnut Trifid Nebula Brazil Nut
99 Tarantulacho Tarantula Lapacho
100 Ceibanova Ceiba Supernova
101 Pau Brasilion Pau Brasil Orion
102 Araucariades Araucaria Pleiades
103 Quebrachoelix Quebracho Helix
104 Lapacholia Lapacho Magnolia
105 Guanacasar Guanacaste Quasar
106 Jacarandromeda Jacaranda Andromeda
107 Ceibulsar Ceiba Pulsar
108 Titan Brasil Titan Pau Brasil
109 Monkeynebula Monkey Puzzle Nebula
110 Quebrachomeda Quebracho Andromeda
111 Lapachogalaxy Lapacho Galaxy
112 Rubbersar Rubber Tree Quasar
113 Cashewmeda Cashew Andromeda
114 Nutcluster Brazil Nut Cluster
115 Cecropioway Cecropia Milky Way
116 Balsanebula Balsa Nebula
117 Rosewoodway Rosewood Milky Way
118 Ipegalaxy Ipe Galaxy
119 Kapoksar Kapok Quasar
120 Cedrellaxy Cedrella Galaxy
121 Alercenova Alerce Supernova
122 Mahogalactic Mahogany Galactic
123 Sculptorbrasil Sculptor Galaxy Pau Brasil
124 Ceibomeda Ceiba Andromeda
125 Araucarionway Araucaria Milky Way
126 Quebrachion Quebracho Orion
127 Lapachohole Lapacho Black Hole
128 Guanacastromeda Guanacaste Andromeda
129 Jacaranebula Jacaranda Nebula
130 Ceibagalaxy Ceiba Galaxy
131 Europa Brasil Europa Pau Brasil
132 Monkeypuzzar Monkey Puzzle Quasar
133 Quebrachnova Quebracho Supernova
134 Lapachoides Lapacho Hyades
135 Rubbermeda Rubber Tree Andromeda
136 Cashewnova Cashew Supernova
137 Nutway Brazil Nut Milky Way
138 Cecropiagalaxy Cecropia Galaxy
139 Balsaquila Balsa Aquila
140 Rosewoodsar Rosewood Quasar
141 Ipecentauri Ipe Alpha Centauri
142 Kapokgalaxy Kapok Galaxy
143 Cedrellaneta Cedrella Planetary Nebula
144 Alercemeda Alerce Andromeda
145 Mahoganova Mahogany Supernova
146 Boomerangbrasil Boomerang Nebula Pau Brasil
147 Ceibulsar Ceiba Pulsar
148 Pau Brasilades Pau Brasil Pleiades
149 Araucagalaxy Araucaria Galaxy
150 Quebrachomeda Quebracho Andromeda
151 Lapachosar Lapacho Quasar
152 Guanacastellum Guanacaste Magellanic
153 Jacaranovae Jacaranda Supernova
154 Ceibaway Ceiba Milky Way
155 Enceladnut Enceladus Brazil Nut
156 Monkeynebula Monkey Puzzle Nebula
157 Quebrachogalaxy Quebracho Galaxy
158 Lapachonova Lapacho Supernova
159 Rubberion Rubber Tree Orion
160 Cashewgalaxy Cashew Galaxy
161 Nutsagittarius Brazil Nut Sagittarius
162 Cecropia A* Cecropia Sagittarius A*
163 Balsanova Balsa Supernova
164 Rosewoodmeda Rosewood Andromeda
165 Ipesar Ipe Quasar
166 Kapokway Kapok Milky Way
167 Cedrellipse Cedrella Ellipse
168 Alercenebula Alerce Nebula
169 Mahogaring Mahogany Ring Nebula
170 Cat's Eye Brasil Cat's Eye Pau Brasil
171 Ceibomeda Ceiba Andromeda
172 Araucarionebula Araucaria Orion Nebula
173 Quebrachoides Quebracho Hyades
174 Lapachogalaxy Lapacho Galaxy
175 Guanacastar Guanacaste Quasar
176 Jacarandromeda Jacaranda Andromeda
177 Ceibaneta Ceiba Planetary Nebula
178 Ganymede Brasil Ganymede Pau Brasil
179 Monkeypuzzalaxy Monkey Puzzle Galaxy
180 Quebrachonova Quebracho Supernova
181 Lapachomeda Lapacho Andromeda
182 Rubbersar Rubber Tree Quasar
183 Cashewnebula Cashew Nebula
184 Nutgalaxy Brazil Nut Galaxy
185 Cecropiomeda Cecropia Andromeda
186 Balsagittarius Balsa Sagittarius
187 Rosewoodion Rosewood Orion
188 Ipeleiades Ipe Pleiades
189 Kapoknova Kapok Supernova
190 Cedrellaxy Cedrella Galaxy
191 Alercegalaxy Alerce Galaxy
192 Mahogacrab Mahogany Crab
193 Dumbbell Brasil Dumbbell Nebula Pau Brasil
194 Ceibulsar Ceiba Pulsar
195 Pau Brasilway Pau Brasil Milky Way
196 Araucarina Araucaria Carina
197 Quebrachogalaxy Quebracho Galaxy
198 Lapachosar Lapacho Quasar
199 Guanacastromeda Guanacaste Andromeda
200 Jacaranebula Jacaranda Nebula
201 Ceibanova Ceiba Supernova
202 Io Brasil Io Pau Brasil
203 Monkeypuzzar Monkey Puzzle Quasar
204 Quebrachomeda Quebracho Andromeda
205 Lapachogalaxy Lapacho Galaxy
206 Rubbernova Rubber Tree Supernova
207 Cashewmeda Cashew Andromeda
208 Nutnebula Brazil Nut Nebula
209 Cecropiagalaxy Cecropia Galaxy
210 Balsasar Balsa Quasar
211 Rosewoodway Rosewood Milky Way
212 Ipegalaxy Ipe Galaxy
213 Kapokmeda Kapok Andromeda
214 Cedrellaneta Cedrella Planetary Nebula
215 Alercenova Alerce Supernova
216 Mahogalagoon Mahogany Lagoon
217 Crescent Brasil Crescent Nebula Pau Brasil
218 Ceibagalaxy Ceiba Galaxy
219 Araucarionway Araucaria Milky Way
220 Quebrachion Quebracho Orion
221 Lapachohole Lapacho Black Hole
222 Guanacastar Guanacaste Quasar
223 Jacaranovo Jacaranda Supernova
224 Ceibomeda Ceiba Andromeda
225 Callisto Brasil Callisto Pau Brasil
226 Monkeynebula Monkey Puzzle Nebula
227 Quebrachogalaxy Quebracho Galaxy
228 Lapachonova Lapacho Supernova
229 Rubbermeda Rubber Tree Andromeda
230 Cashewgalaxy Cashew Galaxy
231 Nutway Brazil Nut Milky Way
232 Cecropiomeda Cecropia Andromeda
233 Balsanebula Balsa Nebula
234 Rosewoodsar Rosewood Quasar
235 Ipecentauri Ipe Centauri
236 Kapokgalaxy Kapok Galaxy
237 Cedrellaxy Cedrella Galaxy
238 Alercemeda Alerce Andromeda
239 Mahoganova Mahogany Supernova
240 Flame Brasil Flame Nebula Pau Brasil
241 Ceibulsar Ceiba Pulsar
242 Pau Brasilion Pau Brasil Orion
243 Araucagalaxy Araucaria Galaxy
244 Quebrachomeda Quebracho Andromeda
245 Lapachosar Lapacho Quasar
246 Guanacastway Guanacaste Milky Way
247 Jacarandromeda Jacaranda Andromeda
248 Ceibanova Ceiba Supernova
249 Veil Brasil Veil Nebula Pau Brasil
250 Monkeypuzzalaxy Monkey Puzzle Galaxy
251 Quebrachonova Quebracho Supernova
252 Lapachogalaxy Lapacho Galaxy
253 Rubberion Rubber Tree Orion
254 Cashewnova Cashew Supernova
255 Nutgalaxy Brazil Nut Galaxy
256 Cecropiagalaxy Cecropia Galaxy
257 Balsomeda Balsa Andromeda
258 Rosewoodway Rosewood Milky Way
259 Ipesar Ipe Quasar
260 Kapoknebula Kapok Nebula
261 Cedrellaneta Cedrella Planetary Nebula
262 Alercenebula Alerce Nebula
263 Mahogaring Mahogany Ring
264 Pelican Brasil Pelican Nebula Pau Brasil
265 Ceibagalaxy Ceiba Galaxy
266 Araucarionebula Araucaria Orion Nebula
267 Quebrachoides Quebracho Hyades
268 Lapachomeda Lapacho Andromeda
269 Guanacastar Guanacaste Quasar
270 Jacaranovo Jacaranda Supernova
271 Ceibulsar Ceiba Pulsar
272 Cone Brasil Cone Nebula Pau Brasil
273 Monkeypuzzar Monkey Puzzle Quasar
274 Quebrachogalaxy Quebracho Galaxy
275 Lapachosar Lapacho Quasar
276 Rubbernova Rubber Tree Supernova
277 Cashewmeda Cashew Andromeda
278 Nutway Brazil Nut Milky Way
279 Cecropiomeda Cecropia Andromeda
280 Balsagalaxy Balsa Galaxy
281 Rosewoodion Rosewood Orion
282 Ipeleiades Ipe Pleiades
283 Kapokgalaxy Kapok Galaxy
284 Cedrellaxy Cedrella Galaxy
285 Alercegalaxy Alerce Galaxy
286 Mahoganova Mahogany Supernova
287 Witch Head Brasil Witch Head Pau Brasil
288 Ceiboway Ceiba Milky Way
289 Pau Brasilmeda Pau Brasil Andromeda
290 Araucarina Araucaria Carina
291 Quebrachonebula Quebracho Nebula
292 Lapachogalaxy Lapacho Galaxy
293 Guanacastromeda Guanacaste Andromeda
294 Jacaranebula Jacaranda Nebula
295 Ceibanova Ceiba Supernova
296 Iris Brasil Iris Nebula Pau Brasil
297 Monkeynova Monkey Puzzle Supernova
298 Quebrachomeda Quebracho Andromeda
299 Lapachosar Lapacho Quasar
300 Rubbermeda Rubber Tree Andromeda
301 Cashewgalaxy Cashew Galaxy
302 Nutnebula Brazil Nut Nebula
303 Cecropiagalaxy Cecropia Galaxy
304 Balsanovo Balsa Supernova
305 Rosewoodway Rosewood Milky Way
306 Ipegalaxy Ipe Galaxy
307 Kapokmeda Kapok Andromeda
308 Cedrellaneta Cedrella Planetary Nebula
309 Alercenova Alerce Supernova
310 Mahogagalaxy Mahogany Galaxy
311 Bubble Brasil Bubble Nebula Pau Brasil
312 Ceibomeda Ceiba Andromeda
313 Araucarionway Araucaria Milky Way
314 Quebrachion Quebracho Orion
315 Lapachohole Lapacho Black Hole
316 Guanacastar Guanacaste Quasar
317 Jacaranovo Jacaranda Supernova
318 Ceibagalaxy Ceiba Galaxy
319 North America Brasil North America Pau Brasil
320 Monkeypuzzar Monkey Puzzle Quasar
321 Quebrachogalaxy Quebracho Galaxy
322 Lapachonova Lapacho Supernova
323 Rubbersar Rubber Tree Quasar
324 Cashewnebula Cashew Nebula
325 Nutgalaxy Brazil Nut Galaxy
326 Cecropiomeda Cecropia Andromeda
327 Balsoway Balsa Milky Way
328 Rosewoodsar Rosewood Quasar
329 Ipecentauri Ipe Centauri
330 Kapokgalaxy Kapok Galaxy
331 Cedrellaxy Cedrella Galaxy
332 Alercemeda Alerce Andromeda
333 Mahoganova Mahogany Supernova
334 Pacman Brasil Pacman Nebula Pau Brasil
335 Ceibulsar Ceiba Pulsar
336 Pau Brasilades Pau Brasil Pleiades
337 Araucagalaxy Araucaria Galaxy
338 Quebrachomeda Quebracho Andromeda
339 Lapachosar Lapacho Quasar
340 Guanacastway Guanacaste Milky Way
341 Jacarandromeda Jacaranda Andromeda
342 Ceibanova Ceiba Supernova
343 Soul Brasil Soul Nebula Pau Brasil
344 Monkeynebula Monkey Puzzle Nebula
345 Quebrachonova Quebracho Supernova
346 Lapachogalaxy Lapacho Galaxy
347 Rubberion Rubber Tree Orion
348 Cashewnova Cashew Supernova
349 Nutway Brazil Nut Milky Way
350 Cecropiagalaxy Cecropia Galaxy
351 Balsameda Balsa Andromeda
352 Rosewoodway Rosewood Milky Way
353 Ipesar Ipe Quasar
354 Kapoknebula Kapok Nebula
355 Cedrellaneta Cedrella Planetary Nebula
356 Alercenebula Alerce Nebula
357 Mahogaring Mahogany Ring
358 Heart Brasil Heart Nebula Pau Brasil
359 Ceibagalaxy Ceiba Galaxy
360 Araucarionebula Araucaria Orion Nebula
361 Quebrachoides Quebracho Hyades
362 Lapachomeda Lapacho Andromeda
363 Guanacastar Guanacaste Quasar
364 Jacaranovo Jacaranda Supernova
365 Ceiboway Ceiba Milky Way
366 Elephant Trunk Brasil Elephant Trunk Pau Brasil
367 Monkeypuzzalaxy Monkey Puzzle Galaxy
368 Quebrachogalaxy Quebracho Galaxy
369 Lapachosar Lapacho Quasar
370 Rubbernova Rubber Tree Supernova
371 Cashewmeda Cashew Andromeda
372 Nutgalaxy Brazil Nut Galaxy
373 Cecropiomeda Cecropia Andromeda
374 Balsagalaxy Balsa Galaxy
375 Rosewoodion Rosewood Orion
376 Ipeleiades Ipe Pleiades
377 Kapokgalaxy Kapok Galaxy
378 Cedrellaxy Cedrella Galaxy
379 Alercegalaxy Alerce Galaxy
380 Mahoganova Mahogany Supernova
381 Omega Brasil Omega Nebula Pau Brasil
382 Ceibulsar Ceiba Pulsar
383 Pau Brasilion Pau Brasil Orion
384 Araucarina Araucaria Carina
385 Quebrachonebula Quebracho Nebula
386 Lapachogalaxy Lapacho Galaxy
387 Guanacastromeda Guanacaste Andromeda
388 Jacaranebula Jacaranda Nebula
389 Ceibanova Ceiba Supernova
390 Blinking Brasil Blinking Nebula Pau Brasil
391 Monkeypuzzar Monkey Puzzle Quasar
392 Quebrachomeda Quebracho Andromeda
393 Lapachosar Lapacho Quasar
394 Rubbermeda Rubber Tree Andromeda
395 Cashewgalaxy Cashew Galaxy
396 Nutnebula Brazil Nut Nebula
397 Cecropiagalaxy Cecropia Galaxy
398 Balsanovo Balsa Supernova
399 Rosewoodway Rosewood Milky Way
400 Ipegalaxy Ipe Galaxy
401 Kapokmeda Kapok Andromeda
402 Cedrellaneta Cedrella Planetary Nebula
403 Alercenova Alerce Supernova
404 Mahogagalaxy Mahogany Galaxy
405 Ghost Brasil Ghost Nebula Pau Brasil
406 Ceibomeda Ceiba Andromeda
407 Araucarionway Araucaria Milky Way
408 Quebrachion Quebracho Orion
409 Lapachohole Lapacho Black Hole
410 Guanacastar Guanacaste Quasar
411 Jacaranovo Jacaranda Supernova
412 Ceibagalaxy Ceiba Galaxy
413 Eskimo Brasil Eskimo Nebula Pau Brasil
414 Monkeynebula Monkey Puzzle Nebula
415 Quebrachogalaxy Quebracho Galaxy
416 Lapachonova Lapacho Supernova
417 Rubbersar Rubber Tree Quasar
418 Cashewnebula Cashew Nebula
419 Nutgalaxy Brazil Nut Galaxy
420 Cecropioway Cecropia Milky Way

203
docs/AGENT-REFERENCE.md Normal file
View file

@ -0,0 +1,203 @@
# Detailed Agent Reference — MEMLNaut-NISPS
Supplement to [`../AGENTS.md`](../AGENTS.md); deep target, schema, build, and verification
detail lives here.
This file provides guidance to coding agents working with this repository.
## Overview
MEMLNaut-NISPS — Neural Interactive Shaping of Parameter Spaces. A research platform for interactive ML control of audio. **One C++20 codebase** (`nisps/`) compiles to two targets:
1. **RP2350 firmware** for the MEMLNaut hardware platform (`firmware/`).
2. **WASM** in the Manifold React browser app (`manifold/`) — same engines + ML, run through an AudioWorklet.
(The former SolidJS playground was retired 2026-07-13 at P1 of `docs/specs/plans/one-core-engine-refactor.md`; archived on branch `archive/playground-solidjs`, tag `playground-solidjs-final`. The browser-only C15 engine currently lives only there.) Parameter contracts are JSON schemas (`schemas/`) with codegen producing the C++ headers AND the TypeScript modules (both live since P5; CI fails if either is stale).
Project documentation: https://musicallyembodiedml.github.io/memlnaut/approaches/nisps
For the codebase index, see `MAP.md`. For strategic gaps and open mission questions, see `ALIGNMENT.md`.
**For anything UI-related in the Manifold front-end (`manifold/`), read `manifold/ONBOARDING.md` first** — it's a single-file agent orientation (run/build/deploy/test, the UI/engine-spine/WASM layering, the convertible Stages, the Dock + drawers, and the non-obvious gotchas).
## The `nisps/` core
```
nisps/
├── core/ types, perf attrs, concepts (AudioEngine, MLEngine, Mode), fixed/ring buffers, deterministic RNG, math
├── ml/ MLP class template (4-layer, 3 hidden); SGD, gradient clipping, spread-aware Xavier init,
│ RL move_weights with output pin mask + per-layer scaling + weight decay
├── dsp/ biquad, delay, reverb, filter, env, osc, pitch_shift, dc_blocker
├── engines/ 8 audio engines (paf_synth, channel_strip, xiasri, verb_fx, memlcelium, breakor,
│ elysiamorf, analysis) + NoOpEngine. Each satisfies the AudioEngine concept.
├── modes/ 8 platform-agnostic modes binding {ML, engine, voice space, abstract I/O channels}.
│ CRTP base eliminates the duplication that plagued firmware modes.
└── wasm/ Emscripten C API bindings (compiled only for WASM target)
```
Build: `cmake -S nisps -B nisps/build -G Ninja && cmake --build nisps/build && ctest --test-dir nisps/build`.
Tests: 4 executables (`nisps_core_tests`, `nisps_dsp_engine_tests`, `nisps_modes_tests`, `nisps_golden_tests`). Run all: `bash scripts/build-cpp-tests.sh`. Parity vs WASM: `bash scripts/parity-check.sh` (asserts native and WASM produce identical outputs within 1e-5).
Throughput: `bash scripts/bench-engines.sh` (per-engine ns/sample, blocks/s, realtime factor on native + WASM; `--compare <report.json>` for deltas). It **reports and never asserts** — no threshold, no failure mode. Read it before and after any change to `nisps/dsp/` or `nisps/engines/`; nothing else in the repo will tell you an engine got slower.
### Performance contract (RP2350)
These rules apply to **all** code under `nisps/`. They are inert in WASM but kept globally for consistency.
- **No heap.** No `new`, `malloc`, `std::vector` in hot paths. Use `std::array<T, N>`. (`nisps::FixedBuffer` is gone — deleted in the Phase 1 sweep; `std::array` was doing the same job.)
- **Constants discipline.** Float literals >255 used in hot paths must be `static const float val = X.f;` not inline.
- **`.f` suffix on all float literals.** No double promotion in audio/inference paths.
- **Hot-path attributes.** Apply `NISPS_HOT` / `NISPS_FORCE_INLINE` (from `nisps/core/perf.hpp`). The SRAM-section macros were deleted in the 2026-07 sweep — they had no real use sites.
- **No virtual dispatch in audio path.** `AudioEngine` and `Mode` are C++20 concepts, not interfaces.
- **Deterministic RNG.** All RNG state is per-instance; constructors take a seed; cross-platform parity tests rely on this.
Lint: `bash scripts/lint-cpp.sh` warns on missing `.f` and fails on heap/`Arduino.h` use under `nisps/`.
## The `firmware/` target
```
firmware/MEMLNaut-NISPS/
├── MEMLNaut-NISPS.ino # Entry point; mode selected via #define MEMLNAUT_MODE_TYPE
├── glue/
│ ├── audio_driver.hpp # memllib AudioDriver block callback → Mode::process per-sample
│ ├── peripherals.hpp # joystick / pots / buttons → Mode::set_input + ML primitives
│ ├── midi_io.hpp # MIDI in → mode handlers; drain ControlEvent ring → MIDI UART
│ ├── mode_select.hpp # type aliases firmware mode name → nisps::modes::*Mode
│ ├── output_router.hpp # drain_outputs() entry point
│ └── settings_view.hpp # wire_settings(): TFT/rotary menu (Joystick Dual/Single for 4-in modes)
└── src/{memllib,nisps} # symlinks (Arduino-CLI sketch tree convention)
```
Build: `scripts/build-firmware.sh [VARIANT]`. Verified compiling for PAFSynth, ChannelStrip, BreakOr on `rp2040:rp2040:solderparty_rp2350_stamp_xl:opt=Optimize3` with `-std=gnu++20`. Flash: `scripts/flash-firmware.sh`. One-shot: `scripts/build-and-flash-firmware.sh`.
### Dual-core orchestration (firmware)
- **Core 0**: UI loop, ML inference (`Mode::tick_control`), peripheral polling (5ms period).
- **Core 1**: Real-time audio processing (`Mode::process`), MIDI polling.
- **Sync**: `nisps::core::ring_buffer` (templated SPSC lock-free, replaces pico/util/queue) + memory barriers (`nisps::core::memory_barrier`, `write_volatile`/`read_volatile`).
## The `manifold/` target
```
manifold/ # Vite + React + TypeScript (the sole browser app)
├── src/
│ ├── engine/ # framework-neutral TS engine spine: wasm-iml, engine-host + worklet,
│ │ # thin WASM wrappers over nisps/pipeline + the curve catalog (P4), spine.ts, EngineProvider
│ ├── primitives/ # 12 design primitives as typed React
│ ├── console/ # convertible Console (CompositeStage, Dock, Drawers, Manifold canvas, …)
│ ├── dock/, backends/, inputs/, feedback/, settings/, serial/, midi-devices/
│ └── debug/probe.ts # window.__nisps behind ?debug=1 for Playwright
├── public/ # nisps.{wasm,js} — canonical build-wasm.sh output
└── tests/ # e2e Playwright specs + fixtures/ (P4 golden parity fixtures)
```
Dev: `cd manifold && bun install && bun run dev`. Build: `bun run build`. Typecheck: `bun run typecheck`. Unit: `bun run test`. E2E: `bunx playwright test` (on the VPS run the runner via non-snap node — BUILD-PLAN gotcha).
Read `manifold/ONBOARDING.md` before touching manifold UI — layering, Stages, Dock, gotchas.
## The `schemas/` + `codegen/` contract
Each mode has a `schemas/modes/<mode>.json` describing its parameters (name, label, range, default, curve, group), ML config (input/output sizes, hidden layers), voice spaces (names — bodies are inline lambdas in the C++ engine), and UI config. The meta-schema at `schemas/schema.json` validates these.
Codegen (`bun run codegen/generate.ts`) emits:
- `nisps/modes/generated/<mode>_schema.hpp` — `constexpr` C++ data, namespace `nisps::modes::generated`, re-exports `nisps::Curve` from `nisps/core/math.hpp`.
- `manifold/src/modes/generated/<mode>_schema.ts` (+ `types.ts`, `index.ts`) — typed const schemas; `MF_MODES` derives params/ml-config from them (P5).
Codegen validates the firmware fit (exactly 3 hidden layers; dims ≤4096) and is idempotent. The golden test (both languages) runs in `run-all-tests.sh` stage 5.
## WASM bridge
Two WASM instances at runtime:
1. **Main thread** (`manifold/src/engine/wasm-iml.ts`): ML inference + sync training + RL primitives, reporting into an injected `EngineSink`. Async training via disposable Web Worker (`wasm-worker.ts`).
2. **AudioWorklet** (`manifold/src/engine/worklet/nisps-processor.ts`): runs engine `process_block` per audio block. Loads `nisps.wasm` directly via `WebAssembly.compile` (no Emscripten glue in worklet). Bytes posted from main thread.
C API is in `nisps/wasm/bindings.cpp`. Build: `bash scripts/build-wasm.sh` (~94KB output to `manifold/public/`).
The browser MLP is runtime-shaped since P2 (`MLPCore<DynamicStorage>`): `nisps_ml_create` honours `(input, output, hidden[3])`; non-positive/null args default to `32→[10,14,18]→126`. Raw `nisps_ml_reshape` reconstructs and prefix-warm-starts a new shape. Manifold's higher-level `engine/io-reshape.ts` seam adds stable input/output identity: it can permute weights and examples without reconstruction while capacity suffices, or reconstruct with arbitrary surviving-dimension remaps. Persistent settings select capacity-vs-exact arity and adapt-vs-clear examples (neutral new-input/output defaults 0/0.5); feedback/exploration scratch state resets on either identity edit. Per-mode dims have been schema-real since P5.3 on both targets — modes no longer slice a shared 126-wide default.
### Known limitations
- Mic input through the worklet for XIASRI / SoundAnalysisMIDI is not wired in manifold.
- C15 has no home on main (see `ALIGNMENT.md` defect 1, browser mode coverage).
- (P3, 2026-07-14) The browser Jolt/OU gestures and the geometric dislike run the C++ core through WASM: `nisps_ml_jolt_*`, `nisps_ml_explore_*`, `nisps_ml_feedback_dislike_geometric` — no TS gesture math remains.
- (§6.5e, 2026-07-21) The per-iteration loss curve IS plumbed: `nisps_ml_loss_history(ml, out, max)` returns the total entry count and fills `min(count, max)`, so `max=0` is a count probe. Both train paths publish it to the spine; `EngineApi.lossHistory()` reads it and `console/TrainingHealth.tsx` displays it at `expanded` drawer depth. `MLPCore::train()` resets the history per run; `train_targets()` (geometric dislike) does not record.
## URL parameters (manifold)
| Param | Range | Default | Effect |
|-------|-------|---------|--------|
| `debug` | 1 | _(off)_ | Exposes the `window.__nisps` debug probe. |
(The playground-era `tame`/`spread`/`preset` URL params died with the playground; `spread` survives as an opt-in engine concept — see below.)
### `spread` — sigmoid saturation control
The MLP uses ReLU hidden layers with a sigmoid output. With uniform [-1,1] weights, the sum of many weighted inputs at each layer drives sigmoid pre-activations far from zero (std dev ≈ √fan_in), causing outputs to saturate. The `spread` parameter addresses this:
- `spread=0` (polarised): uniform [-1,1] weights, RL noise cap 0.3, no decay. Outputs cluster at extremes — good for radical exploration.
- `spread=1` (centered): Xavier-scaled weights, RL noise cap 0.05, 10% weight decay per move. Outputs spread across [0,1] — better for fine-grained shaping.
- Intermediate values interpolate.
Manifold deliberately defaults to `spread=0` at every browser boundary: initial construction,
mode-switch reshape, direct re-roll, explore-and-place scratchpad re-roll, and forwarded VCV
randomise. The old schema spread and the expanded Learning-drawer centred switch are available only
after enabling Settings → Experimental features → **Xavier / spread randomisation**. This is a
Manifold compatibility flag; the shared C++ core and generated mode schemas still expose spread for
firmware, VCV, benchmarks, and explicit API callers.
## Verification chokepoints (user-confirmed)
- **A. Hardware**: each firmware mode flashes and produces correct audio on RP2350.
- **B. RP2350 perf**: no regression vs current main.
- **C. Browser parity**: each firmware mode runs in browser via WASM, sounds equivalent.
- **D. a-immersive feature parity**: control surface, snapshots, A/B compare, region/param pins, heatmap, weight health, gradient flow, output pipeline, session presets.
- **E. CI green**: `bash scripts/run-all-tests.sh` (cmake build + ctest + WASM build + parity + lint + Playwright).
## Build system summary
```bash
# Initialize submodules (required for memllib)
git submodule update --init --recursive
# Codegen (run after editing any schemas/modes/*.json)
cd codegen && bun install && bun run generate.ts
# C++ host tests
bash scripts/build-cpp-tests.sh
# WASM
bash scripts/build-wasm.sh
# Cross-platform parity
bash scripts/parity-check.sh
# Engine throughput (reports; never fails)
bash scripts/bench-engines.sh
# Lint
bash scripts/lint-cpp.sh
# Firmware
scripts/build-firmware.sh PAFSynth # or any other variant
scripts/flash-firmware.sh
scripts/build-and-flash-firmware.sh
# Manifold
cd manifold && bun install
bun run dev # Vite dev (COOP/COEP enabled)
bun run typecheck
bun run test # bun unit tests
bun run build
bunx playwright test # on the VPS: node node_modules/.bin/playwright test
# All tests
bash scripts/run-all-tests.sh
```
## Issue tracking
Use `ergo` exclusively for coding-work tasks over the Holon core; see the `ergo` skill.
Do not use TodoWrite or markdown task lists. Legacy tracker stores are retired/frozen
and reference-only.

View file

@ -1,9 +1,32 @@
# RL / Feedback Learning-Engine Design — Manifold
*Design sign-off. Dated 2026-06-27. Supersedes the AVOID-out-of-scope decision in `docs/redesign/feedback-modes-port-spec.md` §2.5/§7. Author: lead architect, synthesising a 4-candidate / 4-judge design panel against verified source.*
*Design sign-off. Dated 2026-06-27. Supersedes the AVOID-out-of-scope decision in `docs/specs/plans/feedback-modes-port-spec.md` §2.5/§7 (reclassified `kind: plan, status: executed` 2026-07-21). Author: lead architect, synthesising a 4-candidate / 4-judge design panel against verified source.*
---
### 2026-07-25 implementation amendment
The verified upstream reference is now memllib `e291192`, vendored under
`firmware/MEMLNaut-NISPS/lib/memllib/reference/`. Its geometric constants and schedule
supersede the older `0a541cc` constants and synchronous-one-shot wording below:
- push scale `1.0`, negative-LR base `1.5`, no distance taper; cold start pushes in a
deterministic random direction;
- a press stores the rejection and performs one immediate update, then
`FeedbackControllerCore::advance_geometric(dt)` replays **all** live negatives at a
configurable rate for a full-strength wall-clock lifetime;
- the shared core fallback and Manifold's "Upstream defaults" experiment preset match
upstream: LR `0.001`, `200 Hz`, `2500 ms`; rate or lifetime zero is explicit one-shot
mode;
- after operator calibration on 2026-07-25, Manifold starts at LR `0.003`, `200 Hz`,
`2500 ms`; this product default does not change the firmware/core fallback;
- the host supplies elapsed time, but target computation and every weight mutation stay
in the allocation-free shared C++ core. Native↔WASM parity covers this seam.
Manifold exposes LR/rate/lifetime at expanded Learning depth. Deliberate divergences are
kept current in `ALIGNMENT.md`; source comparison evidence is immutable in
`docs/specs/recon/findings-push-away-upstream-comparison.md`.
## 0. Decision summary
| Setting | Default | Also selectable |
@ -222,4 +245,4 @@ Extend Stage 5: **(S5a Mode 1)** seed `ParityMLP`+`ReplayStore`, store 3 fixed p
2. Record the **single deliberate firmware divergence**: the `useRandom` degenerate branch uses `nisps::Rng`, not libc `rand()` (value generated, never compared; native==WASM holds).
3. Record the **TrainBatch-vs-unshuffled-SGD** behavioural divergence: nisps trains per-sample SGD, not shuffled two-LR batches — `browser != firmware` behaviourally, by design (separate verification targets).
4. State the **shared-trunk solo limit** as accepted: no realisation both perfectly isolates and stays expressive.
5. Re-pin orphaned memllib `4733ca0` → `0a541cc`.
5. Re-pin orphaned memllib `4733ca0` → `0a541cc`.

3
docs/redesign/README.md Normal file
View file

@ -0,0 +1,3 @@
# Design Documentation
Design docs have moved to `docs/specs/` (see `docs/specs/MAIN.md` for the corpus entry point). Only `manifold-export/` (design-token and UI-kit asset export) remains here.

View file

@ -1,439 +0,0 @@
I now have everything I need. Key findings confirmed:
- The **engine layer is genuinely good and parity-tested**: `WasmIML`, `EngineHost` (already uses `?worker&url` + hand-rolled imports), `nisps-processor.ts`, the pure pipelines, the stores. These are reusable.
- The **debt is concentrated** in `mode-runtime.ts` (663-line god-hook), the 9 cloned `*Mode.tsx`, `ModeShell`/`ModeSwitcher`, and `App.tsx` (toy nav). The pull-vs-push reactivity bug lives in the god-hook (output flows through imperative `setInput` → `mlStore.process()` → effects, not a clean memo chain).
- **Serving**: nginx root is `meml-aimmersive` (the legacy vanilla a-immersive at `a-immersive.html`, served untouched). The auto-deploy already builds `playground/dist` into `/home/w1n5t0n/deployments/meml.lnfinitemonkeys.org/playground/dist` but it's **currently unserved** (no location block points to it). COOP/COEP is server-scoped. The deploy clone is separate from the dev repo.
- `FeedbackController` = the RL feedback bindings (`thumbsUp`/`thumbsDown`/`moveWeights`/`drawWeights` against the 3-mode loop), not a named class.
I have enough to write the spec. It's a foundation-architecture document, not a code change, so I'll return it directly.
---
# NISPS Foundation Architecture — One Engine, Two Skins
*Status: implementation-ready spec. Scope: the foundation only — the headless engine layer plus the two-skin shell. The detailed UX of each skin (Console drawers, interactive heatmap craft, etc.) is owned by `docs/redesign/playground-2.0-rewrite-plan.md` and not re-litigated here.*
*Constraint from the operator: "lay solid ground that's simple and doesn't paint us into a corner." Build BOTH skins on ONE engine; don't port the engine twice; don't inherit existing tech debt.*
---
## 0. The one decision that drives everything
**Separate the engine from the skin by an explicit, headless boundary — `EngineApi` — and let both skins be pure consumers of it.** Today there is no such boundary: `mode-runtime.ts` fuses WASM lifecycle, the reactive graph, input adapters, audio host, snapshots, heatmap sampling, and per-mode component wiring into one 663-line hook that each cloned `*Mode.tsx` calls. That fusion is *the* debt. The foundation's whole job is to draw the line the god-hook erased, on the correct side:
```
┌──────────────────────────── ONE ENGINE ────────────────────────────┐
gesture │ input pipeline → ML (WASM) → output pipeline → backends │ sound
────────► │ (the reactive spine: one memo chain + one side-effect) │ ────────►
│ exposes a headless EngineApi { inputRaw, mlOutput, routedOutput, │
│ verdict actions, history, status } — NO JSX, NO DOM │
└────────────────────────────────────────────────────────────────────┘
▲ ▲
│ reads EngineApi │ reads EngineApi
┌──────────┴───────────┐ ┌──────────┴───────────┐
│ skin-a-immersive │ │ skin-2.0 (Console) │
│ (faithful clone) │ │ (right dock+drawers) │
│ served at / │ │ served at /2.0 │
└──────────────────────┘ └──────────────────────┘
```
The litmus test for every file: **if it imports `solid-js/web`, JSX, or touches the DOM, it is a skin; otherwise it is the engine.** The engine compiles and is testable headlessly (Node/`__nisps` probe) with zero presentation. Two skins on one engine is then trivially true: a skin is just a tree of components that read `EngineApi` accessors and call `EngineApi` actions. Neither skin can break live-feedback, because neither skin owns the spine.
---
## 1. Module / directory structure
**Build in a fresh tree: `playground2/`** (justification in §3). Inside it, the engine and the two skins are siblings; presentation never reaches into engine internals except via `EngineApi`.
```
playground2/
├── index.html # skin-a-immersive entry (default, root)
├── 2.0.html # skin-2.0 entry (Console variant)
├── vite.config.ts # two inputs; base resolved per-build (§4)
├── package.json # solid-js, vite, vite-plugin-solid, @playwright/test
├── public/
│ ├── nisps.wasm nisps.js # copied from scripts/build-wasm.sh output
│ └── c15.wasm c15.glue.js # (only if/when C15 lands; deferred)
│
├── src/
│ ├── engine/ # ───────── HEADLESS. No JSX. No DOM. ─────────
│ │ │
│ │ ├── index.ts # createEngine(schema, opts): EngineApi ← THE boundary
│ │ ├── types.ts # EngineApi, EngineOptions, EngineStatus
│ │ │
│ │ ├── spine/
│ │ │ ├── control-graph.ts # the memo chain + the single send-effect (§2). ~1 file.
│ │ │ └── coalesce.ts # pointer-rate → display-cadence batch()+microtask gate
│ │ │
│ │ ├── ml/ # LIFTED from playground/ (parity-tested), de-storified
│ │ │ ├── wasm-iml.ts # WasmIML class — adapt: HeapVec, reused out-buffer, no store writes
│ │ │ ├── heap-vec.ts # NEW: re-derive HEAPF32 view on every access (replaces rebind())
│ │ │ ├── wasm-worker.ts # LIFT as-is: disposable training worker (own WASM instance)
│ │ │ └── dataset.ts # LIFT as-is
│ │ │
│ │ ├── audio/ # LIFTED from playground/ — already correct
│ │ │ ├── engine-host.ts # LIFT ~as-is: ?worker&url, fetch+post bytes, transferables
│ │ │ └── worklet/
│ │ │ ├── nisps-processor.ts # LIFT as-is: 2nd WASM, hand-rolled imports
│ │ │ └── audioworklet-globals.d.ts # LIFT as-is
│ │ │
│ │ ├── pipelines/ # LIFTED — pure fns, golden-tested
│ │ │ ├── input.ts # deadzone→zoom→curve→smoothing→momentum (was input/pipeline.ts)
│ │ │ ├── output.ts # global curve→smoothing→slew→freeze (was output/pipeline.ts)
│ │ │ └── curves.ts # ONE curve catalog, golden-tested vs nisps/core/math.hpp (§3)
│ │ │
│ │ ├── backends/ # OutputBackend adapters (plan §3.6)
│ │ │ ├── backend.ts # interface OutputBackend { send, start, teardown }
│ │ │ ├── web-audio.ts # wraps engine-host (v1)
│ │ │ ├── web-midi.ts # 7-bit CC out (v1 adapter; stub-OK at foundation)
│ │ │ └── osc-bridge.ts # WS bridge (deferred; behind locked contract)
│ │ │
│ │ ├── stores/ # module singletons, split by update cadence (plan §3.2)
│ │ │ ├── ml-store.ts # status/arch/dataset + outputs F32 signal + weightsRevision
│ │ │ ├── input-store.ts # config store + raw-axes F32 signal {equals:false}
│ │ │ ├── output-store.ts # global gate config + reuse buffer
│ │ │ ├── control-store.ts # compound axes + per-param createMemo fanout
│ │ │ ├── routing-store.ts # control-point off/fixed/live matrix
│ │ │ ├── history-store.ts # snapshot DAG (undo/A-B/trail/snapshots)
│ │ │ ├── session-store.ts # presets, persistence, URL params, mode switch
│ │ │ └── bus.ts # LIFT as-is: typed sync pub/sub, cross-cutting events only
│ │ │
│ │ ├── runtime/ # the decomposed god-hook (plan §3.4) — engine-side hooks
│ │ │ ├── use-input-adapters.ts # pointer/joystick/gamepad/mic → input-store.setRaw
│ │ │ ├── use-audio-lifecycle.ts # backend start/stop/teardown on mount/mode-switch
│ │ │ ├── use-snapshots.ts # DAG ops (→ history-store)
│ │ │ ├── use-heatmap-sampler.ts # input-space heatmap, throttled to weightsRevision
│ │ │ ├── use-trail.ts
│ │ │ └── use-auto-explore.ts
│ │ │
│ │ ├── feedback/
│ │ │ └── controller.ts # the 3-mode FeedbackController: thumbsUp/thumbsDown/randomize
│ │ │ # + undo, wrapping moveWeights/drawWeights/train + auto-snapshot
│ │ │
│ │ ├── persist.ts # persist<T>(store, version, migrate): versioned + base64 weights
│ │ └── probe.ts # window.__nisps — reads EngineApi only; ?debug=1-gated
│ │
│ ├── shared/ # ───── presentation shared by BOTH skins ─────
│ │ ├── primitives/ # LIFTED: Slider, JoyMap, Heatmap, XYPad, LossPlot,
│ │ │ # GradientFlow, WeightHealth, ProgressRing, ... (+ .demo.tsx)
│ │ ├── theme/
│ │ │ └── tokens.css # orange #ff6a00, danger #ff4466, glass, JetBrains Mono;
│ │ │ # lint allowlist enforced
│ │ ├── EngineProvider.tsx # createContext(EngineApi); both skins wrap their tree
│ │ └── GenericMode.tsx # ONE schema-driven mode component (plan §3.5)
│ │
│ ├── skin-a-immersive/ # ───── DEFAULT skin (faithful a-immersive) ─────
│ │ ├── main.tsx # mounts <EngineProvider><AImmersiveApp/></EngineProvider>
│ │ ├── AImmersiveApp.tsx
│ │ └── components/ # bottom-sheet-faithful chrome, floating RL buttons, top strip
│ │
│ ├── skin-2.0/ # ───── Console skin (Playground 2.0) ─────
│ │ ├── main.tsx # mounts <EngineProvider><ConsoleApp/></EngineProvider>
│ │ ├── ConsoleApp.tsx # the Manifold + dock + drawers + Verdict cluster
│ │ └── components/ # Dock, Drawer (3-depth), VerdictCluster, ReadoutStrip, ...
│ │
│ └── dev/
│ └── PrimitivesShowcase.tsx # /dev/primitives (shared, skin-agnostic)
│
└── tests/e2e/ # Playwright: spine invariant, both skins, parity hooks
```
### Why this shape is "simple and doesn't paint us into a corner"
- **One axis of variation per directory.** Adding an *engine* touches `nisps/` + `schemas/` + `public/*.wasm`; zero skin files. Adding a *backend* touches `engine/backends/`; zero spine, zero skin. Adding/altering a *skin* touches one `skin-*/` tree; zero engine. This is the orthogonality principle made physical.
- **`EngineApi` is the only seam.** Skins import `useEngine()` (the context) and nothing from `engine/` internals. A lint rule (`no-restricted-imports`: skins may not import `engine/**` except `engine/index.ts`/`engine/types.ts`) keeps it honest. You could delete a whole skin and the engine + tests still pass.
- **`GenericMode` lives in `shared/`, not in a skin** — both skins render the same mode bodies; they differ only in *chrome* (how drawers/strips/clusters frame the canvas), not in *what a mode is*. This is what makes "two skins, one engine" cheap rather than a fork.
---
## 2. The reactive spine, concretely
This is the load-bearing structural fix and the literal definition of "doesn't paint us into a corner": there is exactly one path from gesture to sound, every consumer reads it, and a desync is a failing test rather than a recurring prod bug.
### 2.1 The chain (in `engine/spine/control-graph.ts`)
```ts
// One input entry point. {equals:false} so identical-reference writes still notify.
const inputRaw: Accessor<Float32Array> // input-store's raw-axes signal
// ── pure memos ──────────────────────────────────────────────────────────────
const processedInput = createMemo(() => // deadzone→zoom→curve→smoothing→momentum
runInputPipeline(inputRaw(), inputStore.config, inputState)); // pure, golden-tested
const mlOutput = createMemo(() => { // WasmIML.infer INTO a reused buffer
const inp = processedInput(); // (no per-frame Float32Array alloc)
iml.setInputsClamped(inp); // clamps to REAL arch (no phantom channels — §6/D)
return iml.inferInto(mlOutBuf); // returns the reused buffer; pure *read* of weights
}, mlOutBuf, { equals: false });
const routedOutput = createMemo(() => // voice-space + global gate
runOutputPipeline(mlOutput(), routingStore, outputStore.config, outputState));
// ── the ONE side-effect ──────────────────────────────────────────────────────
createEffect(() => {
const out = routedOutput(); // single dependency
backend.send(fillSendBuffer(out)); // dedicated re-filled transferable (neutering-safe)
}); // ← the ONLY postMessage / engine post in the app
```
Rules made structural, not aspirational:
- **Memos are pure.** No `postMessage`, no `backend.send`, no store writes inside a memo. (Lint/review rule; the plan explicitly rejects P3's "postMessage in a memo" and P1's `createComputed`.)
- **Weights mutate only through `ml-store` actions** that bump `weightsRevision`. `mlOutput`'s memo reads weights; because RL/train actions bump the revision and the memo's input or revision is a dependency, output re-derives. There is no leaked write path that can update audio without updating the UI.
- **`mlOutput` writes into `mlOutBuf` and returns it** with `{equals:false}` — fixes today's per-frame `new Float32Array(...)` alloc in `WasmIML.process()` (line 302) and the god-hook's `recomputeOutputs`.
- **Pointer-rate is coalesced to display cadence** in `coalesce.ts` via `batch()` + a microtask, *reactively* (not a rAF poll). rAF touches **canvas drawing only**, never inference.
- **The send buffer is dedicated and owned by the effect**, separate from any signal buffer, because transferring neuters the source (today's `EngineHost.setParams` transfers `params.buffer`).
### 2.2 How BOTH skins consume it identically
Every consumer **reads accessors**; nobody is *pushed to*. There is no "push outputs to the UI" path that can rot — the bug class is deleted, not patched.
```ts
// shared/EngineProvider.tsx
const EngineContext = createContext<EngineApi>();
export const useEngine = () => useContext(EngineContext)!;
// EngineApi (engine/types.ts) — the entire skin-facing surface
interface EngineApi {
// live reactive reads (the spine)
inputRaw: Accessor<Float32Array>;
processedInput:Accessor<Float32Array>;
mlOutput: Accessor<Float32Array>; // raw model space (heatmap, diagnostics)
routedOutput: Accessor<Float32Array>; // post-pipeline (engine, readout strip, visualizer)
status: EngineStatus; // ready, audioStarted, training, examples, ... (store proxy)
weightsRevision: Accessor<number>;
layerStats: () => Float32Array; // diagnostics pull (throttled by caller)
lossHistory: Accessor<ReadonlyArray<number>>;
// actions
setInput: (x: number, y: number) => void; // the ONLY input door
feedback: FeedbackController; // thumbsUp/thumbsDown/randomize/undo/canUndo
train: () => void;
history: HistoryApi; // DAG: A/B pin/swap, jump-to-node
audio: { start(): Promise<void>; stop(): Promise<void>; setMuted(b): void };
schema: ModeSchema;
}
```
- **skin-a-immersive**: a floating RL button reads `engine.feedback.thumbsUp`; the top strip reads `engine.routedOutput()`; the XY pad calls `engine.setInput(x,y)`.
- **skin-2.0**: the Verdict cluster's 👍 reads the *same* `engine.feedback.thumbsUp`; the ReadoutStrip reads the *same* `engine.routedOutput()`; the Manifold's pointer-down calls the *same* `engine.setInput(x,y)`.
Identical bindings, different chrome. A skin cannot create a second reactive path because it has no access to `WasmIML`, `backend`, or the stores' internal setters — only to `EngineApi`.
### 2.3 Live-feedback guarantee + the e2e assertion
**Guarantee:** any change to `inputRaw` (gesture) *or* `weightsRevision` (RL/train) propagates — in the same synchronous tick — to `routedOutput`, which the single effect sends to the backend and which every UI consumer reads. There is no code path that updates audio without updating the readout, or vice versa.
**The e2e invariant, asserted on every mode, in both skins, in CI** (the centerpiece; mirrors plan §3.1):
```ts
// tests/e2e/spine.spec.ts — runs for each {mode} × {skin: '/', '/2.0'}
const before = await page.evaluate(() => window.__nisps.getOutputs());
await page.evaluate(() => window.__nisps.setInputs([0.9, 0.1])); // synchronous probe
const after = await page.evaluate(() => window.__nisps.getOutputs());
expect(after).not.toEqual(before); // ML output changed
const eng = await page.evaluate(() => window.__nisps.getEngineParams());
expect(eng).toChange(); // engine params changed, same input
```
`window.__nisps` (in `engine/probe.ts`) is exposed synchronously, bypasses Solid reactivity with `untrack`/`batch`, and **reads `EngineApi` only** — so the probe exercises the exact path both skins use. Because the probe lives in the engine layer (not a skin), the *same* spec file runs unchanged against both `index.html` and `2.0.html`; if either skin ever fails to read the spine, that skin's row goes red.
A complementary **no-per-frame-alloc** check (heap-snapshot fuzz around a `setInputs` storm) guards the reused-buffer discipline.
---
## 3. Reuse-vs-rebuild
**Recommendation: build the foundation in a FRESH directory `playground2/`, lifting the parity-tested engine modules wholesale and discarding the skin/runtime layer.** This is strictly simpler than refactoring `playground/` in place and carries the least debt, *because the debt and the value are cleanly separable in the existing tree* — the value is the headless engine modules (already DOM-free), the debt is the god-hook + cloned modes + toy `App.tsx`.
### Reuse table
| Existing `playground/` module | Verdict | Action / why |
|---|---|---|
| `audio/engine-host.ts` | **Lift ~as-is** | Already correct: `?worker&url`, fetch-on-main + post bytes, transferables, lazy AudioContext. Best code in the repo. Minor: wrap behind `WebAudioBackend`. |
| `audio/worklet/nisps-processor.ts` | **Lift as-is** | Two-WASM, hand-rolled auto-discovered imports, 128-sample shuttle. Load-bearing and correct. |
| `audio/worklet/audioworklet-globals.d.ts` | **Lift as-is** | — |
| `ml/wasm-iml.ts` | **Lift + adapt** | Keep the C-API surface and lifecycle. **Adapt:** (1) replace `HeapBuffer.rebind()` (manual, error-prone via `Object.defineProperty`) with `HeapVec` re-derive-on-access; (2) `inferInto(buf)` instead of `process()` allocating; (3) **remove `mlStore.__set*` writes from the class** — the class becomes a pure WASM wrapper, the store observes it. (4) Plumb real `nisps_ml_loss_history` (replaces the `lossHistory=[loss]` fake at line 419). |
| `ml/wasm-worker.ts` | **Lift as-is** | Disposable training worker w/ own WASM instance. |
| `ml/dataset.ts` | **Lift as-is** | JS mirror of the C++ ring. |
| `input/pipeline.ts` | **Lift as-is** → `engine/pipelines/input.ts` | Pure, golden-tested, bit-equivalent to legacy. Becomes the `processedInput` memo body. |
| `output/pipeline.ts` | **Lift as-is** → `engine/pipelines/output.ts` | Pure; becomes `routedOutput` body. **Fix:** use reuse buffer. |
| `output/curves.ts` | **Lift + fix** | **Unify with the diverging `Curve` enum** in `generated/types.ts` (`exp/log/square/sqrt/sigmoid/cubic`) into ONE catalog, golden-tested against `nisps/core/math.hpp`. (Plan §4 "Pure-fn pipelines" Keep+fix.) |
| `stores/bus.ts` | **Lift as-is** | Typed sync pub/sub; cross-cutting events only. |
| `stores/ml-store.ts` | **Adapt** | Keep the signal/store split (it's already right: `outputs` F32 `{equals:false}`, store for status). **Add** `weightsRevision`. **Invert ownership:** store observes `WasmIML`, class stops writing the store. |
| `stores/input-store.ts`, `output-store.ts` | **Adapt** | Keep config-store + raw F32 signal pattern; align to spine (input-store's raw signal becomes `inputRaw`). |
| `stores/control-store.ts` | **Adapt** | Keep axis tables/presets/`interpolateAxis`. **Replace** the routing mechanism with **per-param `createMemo` fanout** (kills the `control-routing.ts` `JSON.stringify`-in-untracked-effect anti-pattern). |
| `stores/session-store.ts`, `mode-store.ts`, `exploration-store.ts` | **Salvage values, re-cut along plan §3.2** | Re-partition into `routing-store` (control-point `off/fixed/live`), `history-store` (DAG), `session-store` (persist+presets+URL+mode). Lift constants/preset data; rebuild the store shapes. |
| `stores/persistence.ts` | **Rebuild small** → `engine/persist.ts` | Replace bespoke per-store `Partial<>` merges + `Infinity↔null` slew encoding + slow `Array.from()` weight JSON with one `persist<T>(store, version, migrate)`; base64 weights. |
| `features/snapshots.ts`, `overrides.ts`, `trail.ts`, `heatmap-sampler.ts`, `weight-health.ts`, `mic-input.ts` | **Salvage logic into `engine/runtime/` hooks** | The *algorithms* are fine; the *wiring* (god-hook calling them imperatively) is the debt. Re-home as `use-snapshots`/`use-trail`/`use-heatmap-sampler` reading stores, not as side-effects of `setInput`. Snapshots fold into the DAG. |
| `features/control-routing.ts` | **Discard** | The `JSON.stringify`-inside-`untrack` effect anti-pattern; replaced by control-store memo fanout. |
| `features/session-preset.ts` | **Rebuild** | Composed-layers model (control/synth/weights/mode), per plan §3.8. |
| `primitives/*` (16 + demos) | **Lift as-is** → `shared/primitives/` | Genuinely good, skin-agnostic, already have `.demo.tsx`. The reusable presentation layer. |
| `dev/PrimitivesShowcase.tsx`, `debug/probe.ts` | **Lift + adapt** | Showcase as-is. Probe re-pointed at `EngineApi` (it currently pokes stores directly); `?debug=1`-gate it. |
| **`modes/mode-runtime.ts` (663 lines)** | **DISCARD** | The god-hook. Its responsibilities are split across `engine/spine/`, `engine/runtime/`, `engine/feedback/`. **Do not inherit.** |
| **`modes/*Mode.tsx` (9 cloned files, ~5.6k lines)** | **DISCARD** | Replaced by one `shared/GenericMode.tsx` driven by schema. The clones *are* the debt. |
| `modes/ModeShell.tsx`, `ModeSwitcher.tsx`, `SettingsDrawer.tsx`, `mode-helpers.ts` | **Discard / re-design per skin** | Chrome belongs to skins now; a capability-class switcher is rebuilt minimally. |
| `App.tsx`, `main.tsx` | **Discard** | Toy hash-router with a "home page." Replaced by two skin entries + `EngineProvider`. |
| `modes/generated/*` | **Regenerate, don't lift** | Codegen output; add `capability_class` + `tier` fields (plan §3.5/§3.9) and emit into `playground2/`. |
**Verdict in one line:** lift the entire `engine/` + `shared/primitives/` (≈ the `ml/`, `audio/`, `input/`, `output/`, `primitives/`, `bus.ts` value — the parity-tested core) and rebuild only the ≈6.5k lines of god-hook + cloned modes + toy app that *are* the debt.
### Fresh dir vs refactor-in-place — justification
| | **Fresh `playground2/` (recommended)** | Refactor `playground/` in place |
|---|---|---|
| Debt inheritance | Zero — debt files never copied; engine lifted file-by-file on purpose | High — easy to leave a god-hook tendril; "delete later" rarely happens |
| "Simple ground" | Clean import graph from day 1; lint seam enforceable immediately | Mixed old/new imports during migration; seam blurry for weeks |
| Parity risk | None — engine files lifted verbatim, parity-check runs against the same `nisps.wasm` | None, but harder to prove which path a test exercised |
| Live deploy | Untouched. `playground/dist` keeps building; we wire `playground2/dist` only when ready | Risk of breaking the live (if pointed) build mid-refactor |
| Rollback | `rm -rf playground2/` | `git revert` across an entangled history |
| Cost | One-time copy of ~10 good files | Ongoing vigilance against re-entanglement |
A fresh dir is the cheaper path to *less debt while reusing the parity-tested core* — exactly the operator's brief. `playground/` stays as the working reference (and keeps deploying) until `playground2/` reaches parity, then `playground/` is deleted and `playground2/`→`playground/` (a rename, by which point the engine seam is proven).
> **Note for the operator (open question A in the plan):** if 2.0 is actually destined for the laptop `~/src/manifold` tree, this whole `playground2/` lands there instead — the architecture is identity-agnostic and the dir name is the only thing that changes. This blocks namespace/repo identity; see §6.
---
## 4. Two-skins mechanism (build + serve)
**Decision: two Vite entries, one build, one `dist/` — NOT a runtime route.** Two HTML entry points compiled in a single `vite build` produce `dist/index.html` (skin A) and `dist/2.0.html` (skin B), sharing all common chunks (engine, primitives, wasm). Rationale against "simple, no corners":
- **A runtime route** (`/2.0` as a client route inside one bundle) would force both skins into one app shell, one router, and — crucially — risk a *shared mutable* engine instance and shared chrome assumptions, reintroducing exactly the cross-skin coupling we're eliminating. It also makes "delete a skin" hard.
- **Two entries** keep the skins genuinely independent (each owns its `main.tsx` + `EngineProvider` mount) while Vite's `manualChunks`/automatic splitting means the engine + primitives are **one shared chunk loaded by both** — zero engine duplication on disk or over the wire. This is the cleanest expression of "one engine, two skins."
### Vite config
```ts
// playground2/vite.config.ts
import { defineConfig } from 'vite';
import solid from 'vite-plugin-solid';
import { resolve } from 'node:path';
const isolation = {
'Cross-Origin-Opener-Policy': 'same-origin',
'Cross-Origin-Embedder-Policy': 'require-corp',
};
export default defineConfig(({ command }) => ({
// Skin A is served at site root; skin B at /2.0/. We build ONCE with base '/2.0/'-aware
// asset URLs? No — simpler: build relative, let nginx alias map. See note below.
base: './', // relative asset URLs → same dist works at / and at /2.0/
plugins: [solid()],
server: { port: 5173, headers: isolation }, // dev: COOP/COEP for SharedArrayBuffer + worklet
preview: { port: 4173, headers: isolation },
build: {
target: 'es2022',
sourcemap: true,
rollupOptions: {
input: {
main: resolve(__dirname, 'index.html'), // → dist/index.html (skin A, default)
v2: resolve(__dirname, '2.0.html'), // → dist/2.0.html (skin B, Console)
},
},
},
}));
```
**Why `base: './'` (relative) rather than a fixed `/2.0/` base.** The two skins share `dist/assets/*`. If skin B used absolute `base:'/2.0/'`, skin A (at `/`) would request the wrong asset prefix. Relative `base` makes **the same `dist/` mount correctly at both `/` and `/2.0/`** with no second build. The one caveat is the **AudioWorklet URL** and the **`?worker&url` chunk**: Vite emits these as module URLs resolved relative to the importing chunk, which works under a sub-path *as long as the chunk itself is loaded relatively* — relative `base` satisfies this. The worklet's WASM is fetched by absolute `/nisps.wasm`; under a sub-path we make that **origin-relative and configurable** (engine reads `import.meta.env.BASE_URL` or a runtime `<base>`), so `/2.0/` resolves `…/nisps.wasm` correctly. (Today `wasm-iml.ts`/`engine-host.ts` hardcode `/nisps.wasm` against `window.location.origin` — fine at root, must become base-aware for the sub-path. This is the single concrete code change the two-skins serving imposes.)
Two `package.json` scripts make dev ergonomic:
```jsonc
"dev": "vite", // serves both index.html and 2.0.html
"dev:2.0": "vite --open /2.0.html",
"build": "tsc --noEmit && vite build", // emits dist/index.html + dist/2.0.html
```
### nginx (keeping the legacy a-immersive root untouched)
The current root is `meml-aimmersive` serving the **legacy vanilla** `a-immersive.html`. The new SolidJS app builds to `…/meml.lnfinitemonkeys.org/playground/dist` (currently unserved). The cleanest, no-corner serving plan: **add location blocks for the new app under explicit sub-paths, leave `/` on the legacy root for now**, and flip `/` to the new skin A only when the operator signs off.
```nginx
# inside the existing 443 server { } for meml.lnfinitemonkeys.org
# (COOP/COEP already set at server scope → inherited by all locations below)
set $pg2 /home/w1n5t0n/deployments/meml.lnfinitemonkeys.org/playground2/dist;
# ── Skin B: Console at /2.0 ────────────────────────────────────────────────
location = /2.0 { return 301 /2.0/; }
location /2.0/ {
alias $pg2/;
index 2.0.html;
try_files $uri $uri/ /2.0/2.0.html; # SPA fallback to skin B's entry
}
# ── Skin A: faithful clone, staged at /next while it's proven ──────────────
location = /next { return 301 /next/; }
location /next/ {
alias $pg2/;
index index.html;
try_files $uri $uri/ /next/index.html; # SPA fallback to skin A's entry
}
# ── Shared assets (one chunk set for both skins) ──────────────────────────
location /assets/ { alias $pg2/assets/; } # relative-base requests resolve here
location = /nisps.wasm { alias $pg2/nisps.wasm; }
location = /nisps.js { alias $pg2/nisps.js; }
# legacy root stays:
# root /home/w1n5t0n/deployments/meml-aimmersive; index a-immersive.html; (UNCHANGED)
```
**Cutover (one-line change, when signed off):** point `/` at skin A by setting the server `root $pg2;` + `index index.html;` and a top-level `try_files $uri /index.html;`. Until then `/` serves the legacy artifact untouched, `/next/` is the new faithful skin, `/2.0/` is the Console. The auto-deploy script gains one line: after `playground/`'s `npm run build`, also `cd playground2 && npm install && npm run build` (or replace once `playground/` is retired). COOP/COEP is **server-scoped already**, so every sub-path inherits cross-origin isolation — no per-location header duplication needed (the one real correctness requirement for SharedArrayBuffer + the worklet under a sub-path is satisfied for free).
---
## 5. Build sequence (small, shippable, Playwright-testable)
Each step ends green and observable. Parity checkpoints (verification chokepoints C/E) called out. This mirrors the plan's Phase 0–4 but with the **two-skin seam established early** so neither skin can diverge.
| Step | Deliverable | Test gate | Parity |
|---|---|---|---|
| **S0 — Scaffold + seam** | `playground2/` Vite skeleton; two HTML entries; `tokens.css` + CSS-var lint allowlist; `EngineApi`/`EngineProvider` stubs; codegen re-emitting into `playground2/` with `capability_class`+`tier` and **build-fails-on-arch-mismatch**; `persist<T>` helper; `bus.ts` lifted. Two trivial skins each render "hello from skin A/B" reading a stub `EngineApi`. | `bun run typecheck` green; codegen golden test; Playwright loads `/` and `/2.0.html`, both mount. | Codegen idempotent (byte-identical regen). |
| **S1 — Spine against a stub** | `control-graph.ts` memo chain + single send-effect with a **stubbed `mlOutput`** (deterministic fn of input); `input-store` + `output-store` with lifted pure pipelines (curve enums unified, golden-tested); `coalesce.ts`. `EngineApi.setInput`/`routedOutput` real. | **Spine invariant e2e** (`setInputs→getOutputs changes`) passes against the stub, **on both skins**; no-per-frame-alloc heap fuzz. | Pipeline golden vs legacy bit-equivalence. |
| **S2 — WASM ML bridge** | Lift `wasm-iml.ts` + `wasm-worker.ts` + `dataset.ts`; add `HeapVec` re-derive; `inferInto(buf)`; `ml-store` observes the class (no class→store writes); `weightsRevision`; real `nisps_ml_loss_history` plumbed. `mlOutput` memo now calls real `WasmIML`. | Probe `infer`/`train`/`getLayerStats`; loss plot draws a *real* curve; spine invariant still green on both skins. | **Checkpoint:** main-thread inference matches native within 1e-5 (`parity-check.sh`). |
| **S3 — Audio + first real mode via GenericMode** | Lift `engine-host.ts` + worklet behind `WebAudioBackend`; base-aware WASM URL; `GenericMode.tsx` rendering **PAFSynth** (synth class). Mount `GenericMode` inside *both* skin shells (minimal chrome each). | Per-mode live-feedback e2e (engine params change on input move) on both skins; "Start audio" gesture works. | **Checkpoint:** `parity-check.sh` green; browser PAFSynth audio equivalent to firmware (chokepoint C). |
| **S4 — Skin A default, faithful** | Build out `skin-a-immersive/` to faithfully reproduce a-immersive chrome (bottom-sheet, floating RL buttons, top strip) over `GenericMode` + `EngineApi`. Feedback controller (`thumbsUp/Down/randomize/undo`) wired; snapshot DAG (`history-store`) behind undo. Serve at `/next/`. | Playwright drives the faithful loop via probe + via real DOM on `/next/`; visual-faithfulness check (key anchors, orange tokens); a-immersive feature-parity subset (chokepoint D start). | — |
| **S5 — Skin B Console at /2.0** | Build out `skin-2.0/`: Manifold + right dock + 3-depth drawers + Verdict cluster + interactive ReadoutStrip, all reading the *same* `EngineApi`. RL-undo + A/B against the DAG. Serve at `/2.0/`. | The **same** spine + feedback e2e specs run against `/2.0/` unchanged and pass; Console-specific drawer/depth e2e. | — |
| **S6 — Mode coverage + backends + persistence** | All in-scope engines via `GenericMode` (synth/sequencer/controller/visual classes; `SequencerLane`, `VisualEngine`, controller meters); capability-class switcher in both skins; `WebMidiBackend` + (stubbed) `OscBridgeBackend`; versioned persistence + base64 weights; control surface (axis memo fanout, presets, pinning). | Every mode passes live-feedback e2e in CI on both skins; tactile-constant e2e (3px/300ms) on the Console strip. | **Checkpoint:** parity per mode; full `run-all-tests.sh` green = chokepoint E. |
| **S7 — Cutover** | Point nginx `/` at skin A; delete legacy `meml-aimmersive` serving (archive the artifact); retire `playground/`, rename `playground2/`→`playground/`; update deploy script. | Post-cutover smoke e2e against `/` and `/2.0/`. | — |
The deliberate ordering choice vs the plan: **establish `EngineApi` + both skin mounts at S0** (not after the Console is built), so from the very first stub the "two skins read one engine" property is *tested*, and no skin can quietly grow its own data path. The Console UX detail (drawers, depths) is still de-risked against `/dev/primitives` and user-tested with Dimi before S5, per the plan.
---
## 6. Risks & open questions for the operator
**Open questions that block or shape the build (need a call before S0/S2):**
1. **Canonical tree / identity (plan §8-A).** Is the foundation `playground2/` *in this VPS `MEMLNaut-NISPS` tree* (`nisps::`), or does it belong in the laptop `~/src/manifold` tree (`manifold::`)? This blocks the dir name, namespace, repo, and codegen identity. *Recommendation:* build here (the live tree the recon ran against; rename is trivial), but confirm.
2. **Fixed-2-input contract (plan §8-D) — the big one.** The foundation hard-codes the honest fixed `MLP<2,10,14,18,126>` contract: codegen **fails the build** on schema/arch mismatch and `GenericMode` clamps `setInput` to the real arch (killing today's phantom-channel OOB writes in `mode-runtime.ts` lines 224-230 and `wasm-iml` warn-and-ignore at lines 198-203). Consequence: **multi-input mic modes (XIASRI / sound_analysis_midi) are firmware-only in v1** and show a "single-input in browser" badge; the runtime-shaped MLP is deferred *behind a passing parity check, never bundled into this rewrite*. Confirm you accept this v1 contraction — it's a materially smaller browser surface than "all modes work in browser."
3. **Two entries vs runtime route — confirm.** I've chosen two Vite entries / one `dist` (cleanest "one engine, two skins"). If you'd rather a single bundle with a `/2.0` client route (e.g. to share an app shell), say so now — it changes the skin-mount and engine-instance model.
4. **Serving cutover policy.** I propose: legacy a-immersive stays at `/` untouched; new skins live at `/next/` (faithful) and `/2.0/` (Console) until you sign off, then `/`→skin A. Confirm you want the new default *staged* rather than replacing `/` immediately. (Also: do you want skin A to eventually *replace* the legacy vanilla a-immersive, or coexist?)
5. **`capability_class` + `tier` schema fields (plan §8-B/§8-C).** The foundation's `GenericMode` and switcher depend on these new schema fields. Confirm the four classes (Synth/Controller/Sequencer/Visual) and the per-param/diagnostic `tier: 0|1|2` disclosure model are canonical, since codegen and `EngineApi.schema` bake them in from S0.
**Risks carried into the build (mitigations baked into the architecture):**
| Risk | Mitigation |
|---|---|
| A skin quietly grows a second data path → live-feedback rot returns | `EngineApi` is the only seam; `no-restricted-imports` lint forbids skins importing `engine/**` internals; spine e2e runs on **both** skins in CI. |
| Memo purity violated (side-effect creeps into a memo) | Single documented send-`createEffect`; lint/review rule; the chain is one tested file (`control-graph.ts`). |
| WASM URL breaks under `/2.0/` sub-path | Base-aware asset resolution (`import.meta.env.BASE_URL`); the one concrete change the sub-path imposes; covered by a load-under-sub-path e2e. |
| Shared `dist` + relative base mis-resolves worklet/`?worker&url` chunk | `base:'./'` + nginx `alias`; explicit load-both-skins e2e at S0 catches it immediately. |
| Lifting `wasm-iml` re-imports its store-coupling debt | Adapt-on-lift: strip `mlStore.__set*` from the class; store observes the class. Enforced by the headless-engine lint (engine files may not import skin/JSX). |
| Two builds in the deploy pipeline slow/again-unserved | One `dist`, one `vite build` (two entries); deploy script gains one block; `/2.0/` + `/next/` wired in nginx from the start so it isn't "built but unserved" like today's `playground/dist`. |
| Parity drift on any core touch | Every engine-touching step (S2, S3, S6) gated on `parity-check.sh` green before the skin lands. |
| Probe ships to prod | `engine/probe.ts` gated behind `?debug=1`; reads `EngineApi` only. |
---
### Relevant file paths
- Authoritative UX/feature plan (owns the skins' detail): `/home/w1n5t0n/src/MEMLNaut-NISPS/docs/redesign/playground-2.0-rewrite-plan.md`
- Design-intent reference for skin-a-immersive: `/home/w1n5t0n/src/MEMLNaut-NISPS/docs/redesign/playground-2026.md`
- The god-hook to discard (the debt): `/home/w1n5t0n/src/MEMLNaut-NISPS/playground/src/modes/mode-runtime.ts`
- Engine modules to lift: `/home/w1n5t0n/src/MEMLNaut-NISPS/playground/src/audio/engine-host.ts`, `…/audio/worklet/nisps-processor.ts`, `…/ml/wasm-iml.ts`, `…/ml/wasm-worker.ts`, `…/ml/dataset.ts`, `…/input/pipeline.ts`, `…/output/pipeline.ts`, `…/output/curves.ts`, `…/stores/bus.ts`, `…/stores/ml-store.ts`, `…/primitives/*`
- Codegen TS types to extend (`capability_class`+`tier`): `/home/w1n5t0n/src/MEMLNaut-NISPS/playground/src/modes/generated/types.ts`
- Serving facts: nginx `/etc/nginx/sites-available/meml.lnfinitemonkeys.org` (server-scope COOP/COEP, root `meml-aimmersive`); deploy script `/home/w1n5t0n/.config/webhooks/meml-deploy.sh` (builds `…/meml.lnfinitemonkeys.org/playground/dist`, currently unserved); webhook def `/home/w1n5t0n/.config/webhooks/hooks.json`
- New foundation tree to create: `/home/w1n5t0n/src/MEMLNaut-NISPS/playground2/` (or the laptop `manifold` tree, pending open question 1)

View file

@ -1,231 +0,0 @@
# SLP-Workshop — output modes, gate sequences & config UX
**Status:** DRAFT / spec-only (2026-06-28). The SLP-Workshop firmware exists as
the `slp_workshop` mode (Synth Library Portland workshop; reuses the MEMLCelium
engine + MLP shape verbatim, foregrounds the Jolt / OU-noise learning gestures —
branch `workshop/synth-fw-audit`). The **three output modes** below are a planned
evolution: compile-time *slices* of that mode. This document specs the mode model,
the Manifold-side gate-sequence configuration, and a proposed UI so the firmware
and the browser app stay aligned. The output-mode slicing + Manifold config are
NOT implemented yet.
Companion docs: `docs/useq-celium/protocol.md` (the CV wire protocol), the
manifold backends (`manifold/src/backends/`), the input layer
(`manifold/src/inputs/`).
**Decisions locked (2026-06-28, operator):** (1) hardware keeps exactly 2 ratio
sequences (memlcelium verbatim); the browser may instantiate as many as the user
wants. (2) The 3 uSEQ gate-only jacks make gates *optional* — pure CV is valid;
if doing gates at all, use those 3 first, then convert CV jacks. (3) Split-net
input is per-input-channel engine routing (each pad/stick/CC tagged Continuous or
Rhythm). (4) Internal BPM clock now; external-MIDI-clock sync is a later nicety.
---
## 1. Ground truth: what `memlcelium` actually is
`memlcelium` is the **hybrid**, and it is **one MLP**, not two:
- Net: `MLP<4, [10,14,18], 56>` (`nisps/engines/memlcelium.hpp`).
- The 56 outputs are split:
- `[0..13]` — **sequencer**: 2 sequences × **7 ratio-seq params** each.
- `[14..55]` — **synthesis**: 42 continuous params (Voice 0 + Voice 1).
- The single net therefore produces **both** the continuous values **and** the
RatioSeq parameters. An internal RatioSeq tick turns the seq params into note
triggers; the engine already exposes `pop_events()` so those triggers can be
consumed externally (MIDI / CV / gate) — that is the hook for gate outputs.
So the **two-separate-networks** hybrid is a genuinely different topology (two MLP
heads), which is why the hardware does the shared-net version and only the browser
(more compute, dynamic) does the split one.
### 7 ratio-seq params per track
`ratios[0..2]` (3), `phasor_mul` (1), `phase_off` (1), `amp_ratios[0..1]` (2) = 7.
(The April browser uSEQ-Celium used 8 — it added a pulse-width param. Use **7**
for firmware parity; pulse width can be an optional 8th later.)
---
## 2. The unifying model: two output STREAMS
Everything below collapses to two output streams, each driven by an MLP head:
| Stream | MLP outputs | Generates | Routes to |
|----------------|----------------------------------|----------------------|----------------------|
| **Continuous** | 1 value per channel | smooth 0..1 values | MIDI CC / CV jack |
| **Rhythm** | 7 params per gate-sequence track | clock-driven gates | MIDI note / gate jack|
A **mode** is just *which streams are active* and *whether they share a network*.
### How the Rhythm stream generates gates (RatioSeq)
Per gate-sequence track, each control tick (`nisps/engines/*` `ratio_seq_*`):
1. A shared **internal-BPM clock** advances a bar phasor. (External-MIDI-clock
sync is a desirable later addition, not a launch requirement — operator.)
2. `seq_phasor = (bar_phasor × phasor_mul + phase_off) mod 1`.
3. `ratio_seq_3(seq_phasor, ratios, pw=0.5)` → **boolean gate** (the 3 ratios
carve the cycle into proportional segments; the phasor's segment + pulse-width
decide high/low).
4. `ratio_seq_2(amp_ratios)` → **2-level velocity** (127 / 64).
5. Rising edge → note-on / gate-high (with velocity); falling edge → note-off /
gate-low.
So a gate is **clock + learned pattern**, not a threshold on a continuous value —
this is what makes "Rhythm" a distinct stream, and why it needs its own 7-params
per track rather than one output per gate.
**Defaults = firmware parity** (pulse width fixed at 0.5; 2-level velocity). Two
deferred extensions, not blocking: a per-track **pulse-width** (the 8th param →
controllable gate length, matters for envelope vs trigger) and **continuous
velocity/accent** instead of the 127/64 two-level. Add later if the workshop
wants them.
> Where RatioSeq runs in the browser: when the active engine *is* the memlcelium /
> SLP WASM engine, consume `pop_events()`. For CV/MIDI modes that don't run that
> audio engine, a small **TS RatioSeq** fed by the Rhythm MLP's 7-params/track
> drives the gates. (Implementation note — not built yet.)
---
## 3. Modes
### Firmware (SLP-Workshop) — compile-time, one chosen at build
Always **exactly 2 ratio sequences** (memlcelium verbatim) — the firmware does not
vary sequence count.
| Mode | Streams | Net | MLP output_size |
|-----------------------|----------------------|-----|----------------------------|
| **Continuous only** | Continuous | 1 | continuous params only |
| **Continuous & Rhythm** (= memlcelium) | both, **shared** | 1 | 14 seq + 42 synth = 56 |
| **Rhythm only** | Rhythm | 1 | 2 × 7 = 14 seq params |
### Browser (Manifold) — dynamic, switchable live
Same three **plus** a fourth, and with **as many ratio sequences as the user wants**
(each gate sequence = its own independent 7-param track; the net reshapes to suit):
- **Continuous & Rhythm (split nets)** — Continuous MLP + a separate Rhythm MLP.
Browser-only (two MLP heads; hardware uses one). `output_size`: shared net =
`continuous + 7 × n_gates` (mind the 126-output WASM cap → ~16 gates max shared);
split net = the Rhythm net is sized independently, so it scales further.
The browser mode is implied by the Outputs config (§4), not a separate picker:
choosing continuous-count > 0 and gate-sequences = 0 ⇒ Continuous-only;
gates > 0 with Shared net ⇒ hybrid-shared; gates > 0 with Separate net ⇒
hybrid-split; continuous-count = 0 ⇒ Rhythm-only.
---
## 4. Configuring gate sequences (Manifold, CV **and** MIDI modes)
The user can add **gate-sequence outputs** in both CV and MIDI modes. Two new
controls only; everything else is automatic.
### 4.1 The two output kinds
- **Continuous** → CC (MIDI) / CV jack (CV).
- **Gate sequence** → note (MIDI) / gate jack (CV). Each = one RatioSeq track.
**MIDI mode** — two independent steppers (capped by the model output budget):
```
Continuous (CC): [ 8 ]
Gate sequences: [ 2 ] each → a rhythm track → note on/off
```
**CV mode** — hardware is fixed (11 PWM/CV-capable + 3 digital/gate-only), so the
two counts are *linked*. Gates are **optional** — pure CV (0 gates) is valid. One
control:
```
uSEQ jacks CV 11 · Gate 0
Gate sequences 0 ●───────────── 14
└ first 3 are FREE (the gate-only digital jacks); the 4th+ convert a CV jack
```
Rule: `gates ∈ [0, 14]`; `CV = gates ≤ 3 ? 11 : 14 − gates`. So the first 3 gate
sequences land on the dedicated gate-only jacks and cost no CV (gates 0→11 CV;
3→11 CV; 6→8 CV; 14→0 CV). "Add more gates by swapping CV outs to gate outs" only
kicks in past 3. (If you want gates at all, those 3 gate-only jacks are there to
use; if you don't, they sit idle and you keep all 11 CV.)
> Wire-protocol impact: **none**. A CV jack acting as a gate just carries 0/full
> (or the 2-level velocity) in its `u16` slot; the 3 dedicated gate bits stay the
> digital pins (`docs/useq-celium/protocol.md`). The CV backend's `CvSpec` already
> lets any output target a CV *or* a gate channel — this extends it so PWM jacks
> can be gate targets too.
### 4.2 The Rhythm network (shown only when gate sequences > 0)
```
Rhythm network: ( Shared ) ( Separate ◀ default )
```
- **Separate** (default): the Rhythm stream gets its **own MLP**, so each input
channel is **routed to one of the two engines** (Continuous or Rhythm). The
routing is automatic per source kind:
- **XY pad (internal)** → a **second on-screen XY pad** appears; pad 1 →
Continuous, pad 2 → Rhythm.
- **Gamepad** → **double-stick**: left stick → Continuous, right stick → Rhythm.
- **MIDI controller** → **per-CC routing**: each learned CC has a small
`Continuous | Rhythm` toggle, so the user assigns which knobs drive which
engine. (The pad/gamepad cases are just the pre-grouped 2-axis versions of
this same per-channel routing.)
- **Shared**: one MLP drives both streams; **all** input channels feed the single
net (one pad / one stick / all CCs). This is the hardware-parity hybrid.
That is the whole decision surface: **two numbers + one toggle.** The second
pad / double-stick / per-CC tag is a *consequence* of Separate, surfaced inline,
not a separate mode control. It reuses the existing input layer
(`InputSource.axisCount()` already supports gamepad single↔double-stick; per-axis
labels exist via `axisLabels()`) — "Separate" adds an engine tag per input axis
and composes a second `xy-pad` source / flips the gamepad to 4-axis as needed.
### 4.3 Per-output detail (advanced, optional)
In `BackendAdvanced` (full-depth modal), each output row can override the
defaults: which model output drives each CC/CV; which rhythm track drives each
note/gate; gate threshold (for continuous-derived gates); MIDI note#/channel; CV
polarity. Defaults (identity assignment) make this unnecessary for the common case.
---
## 5. Proposed UI summary (the "simple" target)
Outputs panel, top to bottom:
1. **Output kinds** — `Continuous [n]` + `Gate sequences [n]` (MIDI), or the
linked `Gate sequences 0–14` slider with the live `CV n · Gate n` readout (CV).
2. **Rhythm network** toggle `( Shared | Separate )` + the inline input surface
(2nd pad / double-stick / per-CC engine tags) — only when gate sequences > 0.
3. The existing per-output rows (off/fixed/live, mute, arm, min/max/curve) +
the per-backend specifics (CC#, CV jack, note#) as today.
No mode dropdown for the rhythm/continuous split — the counts + toggle *are* the
mode.
---
## 6. Open questions / deltas to build later
Build deltas (when we proceed):
- **Manifold engine**: today one MLP head on the spine. "Separate" needs a second
Rhythm MLP head + per-input-axis engine routing. Scope: engine + input-layer.
- **TS RatioSeq**: a browser-side RatioSeq (or a `pop_events()` bridge) to turn
Rhythm-MLP params into gate/note events for the CV & MIDI backends. 2 sequences
on firmware; arbitrary count in the browser.
- **MIDI backend**: gate sequences → note on/off (it currently only sends CC).
- **CvSpec**: allow PWM jacks (cv1..cv11) to be gate targets (incl. velocity-CV),
with the 3 digital pins as the first-used gate jacks.
- **Reshape**: shared-net `output_size` = continuous + 7 × gate_sequences (≤126
WASM cap) — confirm against the reshape/reset-on-reshape flow.
- **Firmware output_size** for "Continuous only" / "Rhythm only" — slices of the
`slp_workshop` mode (`workshop/synth-fw-audit`); the firmware agent's call. This
doc fixes the *shapes*, not exact counts.
Deferred niceties (explicitly not blocking): external-MIDI-clock sync; per-track
pulse-width (gate length); continuous velocity/accent beyond 127/64.

150
docs/specs/MAIN.md Normal file
View file

@ -0,0 +1,150 @@
---
kind: spec
stability: stable
layer: cross-cutting
---
# MEMLNaut-NISPS Spec Corpus
**Neural Interactive Shaping of Parameter Spaces** — a research platform for interactive ML control of audio. One C++20 codebase compiles to two targets: RP2350 firmware for the MEMLNaut hardware, and WASM in the Manifold React browser app running the same engines + ML through an AudioWorklet. (The SolidJS playground was retired 2026-07-13, P1 of plans/one-core-engine-refactor.md; archived on `archive/playground-solidjs`.)
---
## 0. Spec Corpus Ontology
This corpus distinguishes four document kinds by temporal stance and mutability:
- **`kind: spec`** (flat in `docs/specs/`) — timeless present-tense contracts. **Wins over code by intent.** Drift-lintable; updated when the spec itself changes or code breaks it.
- **`kind: plan`** (in `docs/specs/plans/`) — future-tense, finite prescriptions carrying `status: active | executed | superseded`. Burn down as work lands; NEVER citable as authority for shipped behaviour (executed items migrate to `MAP.md` per the repo's doc-sync rule).
- **`kind: finding`** (in `docs/specs/recon/`) — past-tense, dated, **IMMUTABLE**. Never edited, only superseded; exempt from drift lint. Research artifacts capturing ground truth at a specific moment.
- **ADRs** (in `docs/adr/`) — decision rationale ("why we chose X"). Outside the spec root; used as backlinks from specs that encode those decisions.
Archived documents live in `docs/specs/_archive/` with `deprecated-by`/`superseded-by` front matter.
---
## Source Files (Architectural Roots)
The codebase's most architecturally significant areas referenced by this corpus:
| Layer | Path | Purpose |
|-------|------|---------|
| **Shared C++20** | `nisps/core/`, `nisps/ml/`, `nisps/dsp/`, `nisps/engines/`, `nisps/modes/`, `nisps/pipeline/` | One audio+ML library compiling to both firmware and WASM |
| **Schema/Codegen** | `schemas/`, `codegen/` | Parameter contracts + code generation (C++ headers, TS types) |
| **Firmware** | `firmware/MEMLNaut-NISPS/glue/` | Hardware bindings (audio, peripherals, MIDI, settings) |
| **Browser (Manifold)** | `manifold/src/engine/`, `manifold/src/console/` | Headless TS engine + AudioWorklet + the convertible React Console |
---
## 1. Frame
**What it is.** MEMLNaut-NISPS is a single C++20 codebase that compiles once to two distinct targets:
1. **Firmware** — runs on RP2350 hardware (MEMLNaut instrument). Real-time audio engines + interactive ML (4-layer MLP, SGD + RL feedback) with deterministic RNG + dual-core orchestration (audio on core 1, control on core 0).
2. **Browser (WASM)** — runs in the React Manifold front-end via AudioWorklet. The **same eight `nisps/` engines** as firmware — neither target is a superset. (The browser-only C15 synth lives only on the retired-playground archive branch; browser audio-*input* — mic — is not wired, so the analysis-driven modes are hardware-only today.)
**The unifying constraint**: one source tree, one ML architecture, cross-platform parity (native ↔ WASM within 1e-5 numerical tolerance). Parameter shapes are JSON schemas with codegen producing both C++ headers and TypeScript types.
**Performance contract** (applies to `nisps/`, inert in WASM but kept for consistency):
- No heap in audio/inference paths; use `std::array` (the one deliberate exception: `nisps/ml/dynamic_storage.hpp`, browser/VCV-only, `#error`s on RP2350 builds, sole lint allowlist entry).
- Float literals have the `.f` suffix; non-trivial float constants are `static const`.
- No virtual dispatch in the audio path (C++20 concepts, not interfaces).
- Deterministic per-instance RNG; all random state is local and seeded at construction.
- Hot-path attributes from `nisps/core/perf.hpp`: `NISPS_HOT` / `NISPS_FORCE_INLINE` (the former SRAM-section macro regime was deleted 2026-07 as dead).
---
## 2. Cross-Cutting Contracts
### Parity discipline
**Native ↔ WASM bit-equivalence within 1e-5.** The C++ core and its WASM compilation are asserted to produce identical outputs given the same input + same seed. See `tests/cpp/parity_check.cpp` (native test) + `scripts/parity-check.sh` (verify runner).
### Schema/codegen contract
Each mode has a `schemas/modes/<mode>.json` describing parameters (name, label, range, default, curve, group), ML config (input/output sizes, hidden layers), voice spaces (names; bodies are inline lambdas in the engine), and UI config. The meta-schema at `schemas/schema.json` validates these. Codegen (`bun run codegen/generate.ts`) is idempotent and produces **both language outputs**:
- `nisps/modes/generated/<mode>_schema.hpp` — `constexpr` C++ data under `nisps::modes::generated` (+ `nisps/ml/generated/ml_defaults.hpp` from `schemas/ml_defaults.json`).
- `manifold/src/modes/generated/<mode>_schema.ts` (+ `types.ts`, `index.ts`) — the source of truth for the Manifold `MF_MODES` catalogue.
A second generator, `codegen/generate-midi-devices.ts`, emits `nisps/midi/generated/midi_devices.hpp` + `manifold/src/midi-devices/generated/` from `schemas/midi_devices/`.
Schema changes ship with the regenerated C++ **and** TypeScript in the same commit; CI re-runs both generators and fails on any diff (plus the golden test).
### No-heap + deterministic RNG
These are inert in WASM but kept globally to avoid surprises when porting. Lint catches violations: `bash scripts/lint-cpp.sh` warns on missing `.f` suffix and fails on heap use under `nisps/` (allowlist: `dynamic_storage.hpp`), plus `Arduino.h` use outside firmware glue.
---
## Sub-specs (Behavioural + Binding)
| File | Stability | Layer | Summary |
|------|-----------|-------|---------|
| `backends-spec.md` | evolving | binding | Output backends (synth, particles, MIDI, OSC, CV/gate, VCV) unified behind one `OutputBackend` interface + manager. Largely built (`manifold/src/backends/`); carries a 2026-07 grounding note — the full-state OSC sync legs it designed were deleted. |
| `dock-spec.md` | evolving | behavioural | Console dock drawers + the per-output control row (off/fixed/live tri-state, mute, solo/arm, min/max/curve). Built with operator restructuring (top Mode selector + 5 drawers, 2 depths); grounding note maps spec→shipped. |
| `engine-architecture.md` | stable | cross-cutting | The surviving browser-engine contract: the headless `EngineApi` seam + the reactive-spine invariant, as realised in `manifold/src/engine/`. Trimmed 2026-07; superseded framing documented in its header. |
| `inputs-spec.md` | evolving | binding | Modular input layer (XY pad, MIDI input, gamepad single/double-stick) composing into an N-dim vector. Built as `manifold/src/inputs/` (runtime-reshaped net, exclusive-mode UI; mixing groundwork laid — operator decision §7.7 pending). |
| `slp-workshop-firmware.md` | evolving | binding | SLP-Workshop (Synth Library Portland workshop build). Part I (Jolt + OU explore adaptive-learning gestures) shipped & stable. Part II (output-mode evolution, gate sequences, Manifold config) planned/evolving. |
| `useq-cv-protocol.md` | stable | binding | uSEQ-CV wire protocol v2 (USB Web Serial ↔ uSEQ main module → CV/gate jacks + I2C expander). Single source of truth: `firmware/useq-celium/shared/protocol.h`, mirrored in `manifold/src/backends/useq-protocol.ts`. |
| `vcv-module.md` | evolving | binding | MEMLNaut VCV Rack 2 module: CV-to-CV mapper (8 in × 16 out, LED rings) on the shared `nisps/` core (P6). Also the single `.nisps`/patch format spec (v3 flat weights). Pruned 2026-07-21 to the current contract. |
**Archived** (in `_archive/`): `aimmersive-clone-spec.md` — the SolidJS clone spec for the deployed vanilla a-immersive app; the clone was never built and its target framework was retired. Kept as a behavioural reference for the still-deployed vanilla app.
---
## Plans (Finite, Burn-Down)
| File | Status | Summary |
|------|--------|---------|
| `plans/BUILD-PLAN.md` | executed | Manifold build resume anchor (2026-06-27). The app shipped to `/next/`; survivors (non-snap-node e2e invocation, naming/copy decisions) noted in its header; deploy is now the CI-gated webhook. |
| `plans/feedback-modes-port-spec.md` | executed | The "Down Action" negative-feedback port into `nisps/`. Landed, then evolved past it — `docs/adr/rl-feedback-design.md` + `nisps/ml/feedback.hpp` are the surviving truth (the ADR explicitly supersedes its §2.5/§7 AVOID decision). |
| `plans/manifold-parity-features-spec.md` | active | Five playground features for Manifold. Jolt + OU-explore: executed by other means (one-core P3 core bindings). Session presets, pins, control surface: still the live prescription, feeding the curated/advanced split (simplification-plan §6.5c). |
| `plans/playground-2.0-rewrite-plan.md` | superseded | SolidJS clean-room rewrite plan (2026-06-17). Its target (the playground) was retired; many of its ideas shipped in Manifold instead. Still cited as design source by the parity-features plan. |
| `plans/one-core-engine-refactor.md` | active | Firmware+Manifold core reunification (2026-07-13). **All six phases landed on main by 2026-07-18**; stays `active` only for the two hardware-verification chokepoints (its §6) before flipping to executed. |
| `plans/simplification-plan.md` | active | Phased burn-down of the 2026-07-21 simplification audit. Phases 0 (CI/verification), 2 (behaviour bugs) and 3 (truth consolidation, minus S26) burned down 2026-07-21; Phase 1 deletions largely landed; PlatformIO (§5) in flight; §7 lists remaining operator decisions. |
---
## Findings (Immutable, Dated Research)
Exempt from drift lint.
| File | Date | Summary |
|------|------|---------|
| `recon/findings-design-and-manifold.md` | 2026-06-27 | Build-oriented brief synthesizing five redesign docs + Manifold token export + `ConsoleApp.jsx`. VERIFIED from source. |
| `recon/findings-engine-surface.md` | 2026-06-27 | Engine surface audit (deployed JS, WASM, C++ core). Uncovers fixed-2-input WASM gap, loss-history plumbing gap, C15 placeholder. |
| `recon/findings-feedback-behaviour.md` | 2026-06-27 | Current RL feedback behaviour audit. Deployed JS + C++ core use undirected Gaussian noise (not geometric push-away); firmware InterfaceRL implements the true geometric mode. |
| `recon/midi-gamepad-inputs-worklog.md` | 2026-06-27 | Work log of `feat/midi-inputs` branch. Modular input sources (MIDI, gamepad), 32-input WASM foundation, gamepad→verdict wiring. Exclusive picker (not mixing) shipped; mixing engine groundwork done but UI deferred. |
| `recon/upstream-firmware-survey.md` | 2026-06-27 | Git archaeology of the MusicallyEmbodiedML ecosystem. origin/main (C++20/SolidJS rewrite) and upstream/main (old `.ino` firmware) forked at `6efbe9c` (2026-04-14); 49 commits upstream not in origin are ports, not merges. |
| `recon/playground-2026.md` | 2026-04-12 | Design intent snapshot of an unfinished playground UI redesign. Reference-only for SolidJS rewrite; do NOT merge into vanilla playground (those files moved on independently). |
| `recon/simplification-audit-2026-07.md` | 2026-07-21 | Full-repo smell/bloat/spec audit vs the five-bullet one-core vision (66-agent workflow, adversarially verified). 113 findings: CI red since 2026-07-13, memllib pin unpushed, ungated deploys, mode layer unshared, no curated/advanced split, dead-mass inventory. Mitigations in `plans/simplification-plan.md`. |
| `recon/memllib-usage-inventory.md` | 2026-07-21 | Per-file inventory of the memllib surface the firmware actually compiles — the prerequisite for the §7.5 ownership decision. Result: all of memllib bar `examples/` is load-bearing (~1.8 MB / 84 files); no small subset to vendor. |
---
## ADRs (Decision Rationale)
| File | Decision | Audience |
|------|----------|----------|
| `adr/rl-feedback-design.md` | Rationale for the Explore-and-Place + Geometric-Dislike feedback model choices | Implementation team |
---
## Cross-References
- **`MAP.md`** — the neutral inventory of the codebase as it stands (ground truth for what exists).
- **`CLAUDE.md`** — agent contract (gates, hard constraints); `docs/AGENT-REFERENCE.md` — the long-form reference.
- **`ALIGNMENT.md`** — strategic gaps + open mission questions (dated, opinionated gap diagnosis).
- **`manifold/ONBOARDING.md`** — agent-oriented breakdown of the Manifold React app (run/build/deploy/test, UI/engine-spine/WASM layering, convertible Stages, Dock+drawers, gotchas).
- **`docs/redesign/manifold-export/`** — design-token and UI-kit asset export from Manifold design project (not part of this corpus; static asset reference, not migrated).
---
## Open / Deferred
- **Index generation (`.index.json`)**: intentionally not generated yet. No consumer defined; add when crawler/search infra lands.
- **Per-spec §-numbering normalisation**: deferred to a future maintainer pass. Current state mixes flat (§1.1, §1.2…) and three-level nesting (§6.x.y).
- **ADR corpus**: currently minimal (only RL feedback design). Strategic decisions captured in spec Open/Deferred sections; promotion to dedicated ADRs is a future pass.
- **Spec-touched validator**: git-aware lint rule that flags when a code diff touches spec-cited files without also touching the spec. Useful as a pre-commit hook or PR check; infrastructure TBD.

View file

@ -1,6 +1,19 @@
I now have a thorough, complete understanding of the entire deployed app — DOM, CSS theme tokens, all interaction handlers, state model, engine call points, persistence, and the help-modal copy. I have everything needed to write the faithful clone spec. Let me write it now.
---
kind: spec
deprecated-by: 2026-07-21
superseded-by: MAIN.md
---
# NISPS Immersive (`a-immersive.html`) — SolidJS Clone Spec
# NISPS Immersive (`a-immersive.html`) — SolidJS Clone Spec (Archived)
**Note:** This specced a faithful SolidJS clone of the deployed vanilla a-immersive app — a
product that was never built. The SolidJS playground target was retired 2026-07-13 (branch
`archive/playground-solidjs`, tag `playground-solidjs-final`); the current browser app is the
Manifold convertible Console (`manifold/`), covered by the live corpus (`../MAIN.md`). The
vanilla a-immersive app it describes remains deployed at `meml.lnfinitemonkeys.org/`, so this
survives as a behavioural reference for that app, frozen as of 2026-06-27. File paths cited
below (e.g. `/home/w1n5t0n/deployments/meml-aimmersive/`) refer to that deployment, not this
repo.
This is a faithful, component-by-component rebuild spec for the deployed vanilla-JS app at `/home/w1n5t0n/deployments/meml-aimmersive/`. Source of truth: `a-immersive.html` (446-line DOM), `css/a-immersive.css` (2538 lines), `js/a-app.js` (4521-line module), and supporting modules under `js/nisps/`, `js/ui/`, `js/synth/`, `js/midi/`, `js/audio/`. Goal: a from-scratch SolidJS reimplementation that matches the look and behaviour 1:1, binding to a shared engine layer (the WASM IML + synth engines) rather than re-porting it.

View file

@ -1,6 +1,13 @@
# Tiered Specs — MEMLNaut-NISPS
---
deprecated-by: 2026-07-13
superseded-by: MAIN.md
---
This corpus is the **prescriptive plan** for the project: what we are building and why, organised from
# Tiered Specs — MEMLNaut-NISPS (Deprecated)
**Note:** This tiered planning approach was not built out; the corpus adopted a flat MAIN.md layout instead (see `docs/specs/MAIN.md`).
This corpus was the **prescriptive plan** for the project: what we are building and why, organised from
high-level intent down to implementation specifics. It is distinct from the two orienting docs at the repo
root:
@ -20,7 +27,7 @@ T2 Architecture & Contracts
├ playground architecture
└ backends architecture
T3 Component Design → core | playground | firmware | backends (branches per product)
T4 Implementation specifics → (same branches) → spawn bd issues/epics
T4 Implementation specifics → (same branches) → spawn Ergo tasks/epics
```
Files:
@ -31,9 +38,9 @@ Files:
| T1 | `T1-capabilities-and-principles.md` | **draft — awaiting audit** |
| T2 | `T2-architecture-and-contracts.md` (+ per-product sections) | not started |
| T3 | `T3-component-design/{core,playground,firmware,backends}.md` | not started |
| T4 | `T4-implementation/{core,playground,firmware,backends}.md` → beads | not started |
| T4 | `T4-implementation/{core,playground,firmware,backends}.md` → Ergo | not started |
| — | `00-decisions-log.md` | the interview outcomes that seed these tiers |
| ref | `slp-workshop-firmware.md` | **stable reference** — SLP-Workshop firmware mode + Jolt / OU-explore gestures (built + merged) |
| ref | `slp-workshop-firmware.md` | **evolving** — SLP-Workshop: Part I shipped (mode + Jolt / OU-explore gestures), Part II planned (output modes, gate sequences, Manifold config UX) |
## Audit protocol (why tiers exist)
@ -42,7 +49,7 @@ only made *downstream* of a tier when that tier itself got something wrong. So:
1. T0 + T1 are reviewed and approved first (they constrain everything).
2. Only then is T2 authored; reviewed; approved.
3. Only then T3; then T4 → beads → execution.
3. Only then T3; then T4 → Ergo → execution.
If review of a lower tier reveals that an upper tier is wrong, **fix the upper tier first**, then propagate.
Every tier file ends with a "Traces up to" line citing the tier(s) above it that justify its content.

View file

@ -1,6 +1,26 @@
# Output Backends — Specification (`docs/redesign/backends-spec.md`)
---
kind: spec
stability: evolving
layer: binding
---
*Workstream E. Design-only, read-only audit 2026-06-27. The new app is Vite + React + TS in `manifold/`, wired to the parity-tested TS engine lifted from `playground/src`. British spelling in product copy. The built-in synth is the **"Powerful Synth Engine"** — the string "C15" must never reach the user.*
# Output Backends — Specification
*Workstream E. Designed 2026-06-27 against the pre-refactor tree; largely built (see the
Implementation status section at the end). British spelling in product copy. The built-in synth
is the **"Powerful Synth Engine"** — the string "C15" must never reach the user.*
> **Grounding note (2026-07-21).** Treat every `file:line` cite in the body as historical
> grounding, not a live pointer: `playground/*` died with the retired SolidJS playground (branch
> `archive/playground-solidjs`); `aimmersive-clone-spec.md` is archived at
> `_archive/aimmersive-clone-spec.md`; `/home/w1n5t0n/deployments/meml-aimmersive/*` cites refer
> to the still-deployed vanilla app *outside this repo*; VCV line numbers pre-date the P6
> reunification. The shipped backends live at `manifold/src/backends/` (not the proposed
> `engine/backends/`); the shipped bridge protocol is **`/nisps/input` + `/nisps/output` +
> `/nisps/feedback` only** — the `/nisps/state` / `/nisps/weights` / `/nisps/examples`
> full-state-sync legs described below were **deleted 2026-07-21** (zero consumers on either
> side; Rack patch save/load owns persistence). The current protocol contract is
> `vcv-module.md` §Browser Bridge.
> **Naming guard (non-negotiable).** The codename `C15` survives only in internal module/file names that the user never sees (`c15-adapter.js`, `c15-bridge.js`). Every label, tooltip, dock entry, menu item, status string, and aria-label says **"Powerful Synth Engine"** (or just "Synth"). A lint allowlist + a Playwright assertion (`expect(page).not.toContainText('C15')`) enforce this across `manifold/` and the VCV panel SVG/strings.
@ -126,18 +146,20 @@ A `WebMidiBackend` wrapping the salvaged `midi-output.js` (Web MIDI API, `sendBa
An `OscBridgeBackend` that **salvages the existing OSC bridge** rather than reinventing it. Two pieces already exist and are good:
- **Browser client:** `deployments/meml-aimmersive/js/synth/osc-output.js` (param-named WS messages, 50 ms/0.002 dead-zone) and the richer `js/nisps/osc-client.js` (`NispsOscClient` — `EventTarget`, `sendState`/`sendWeights`/`sendParams`, `onOutputsReceived`/`onInputsReceived`, auto-reconnect with backoff). Lift `osc-client.js` as the transport (it already speaks the bridge protocol and is bidirectional).
- **Bridge server:** `deployments/meml-aimmersive/osc-bridge/bridge.ts` — a Deno WebSocket↔UDP-OSC bridge, zero-dependency OSC encode/decode, bidirectional. Keep it as-is; it is the canonical transport between browser and any OSC target (VCV, SuperCollider).
- **Browser client:** `deployments/meml-aimmersive/js/synth/osc-output.js` (param-named WS messages, 50 ms/0.002 dead-zone) and the richer `js/nisps/osc-client.js` (`NispsOscClient` — `EventTarget`, auto-reconnect with backoff). The shipped transport is `manifold/src/backends/osc-client.ts` (`sendParams` / `sendInput` / `sendFeedback` — the legacy `sendState`/`sendWeights` legs were deleted with the full-state sync).
- **Bridge server:** the repo's canonical bridge is `manifold/osc-bridge/bridge.ts` — a Deno WebSocket↔UDP-OSC bridge, zero-dependency OSC encode/decode, bidirectional (WS verbs: `params`, `input`, `feedback`).
**OSC path + range contract (salvaged from `bridge.ts`):**
**OSC path + range contract (the live set):**
| Direction | Address | Args | Meaning |
|---|---|---|---|
| browser→target | `/nisps/<param_name>` | `f` | one param, **post-baseline-mapping value** (see §3) |
| browser→target | `/nisps/state` | `s` | full JSON state (weights + examples + config) |
| browser→target | `/nisps/weights` | `s` | weights-only JSON |
| browser→target | `/nisps/input` | `f…f` | input vector → drives model inputs (VCV bridged mode) |
| browser→target | `/nisps/feedback` | `s` | verdict op JSON (`up` / `down` / `rand` / `clear`) |
| target→browser | `/nisps/output` | `f…f` | output vector (visualisation / monitoring) |
| target→browser | `/nisps/input` | `f…f` | input vector → drives model inputs |
| target→browser | `/nisps/input` | `f…f` | input vector echo |
(The 2026-06 design also carried `/nisps/state` and `/nisps/weights` full-JSON sync — deleted, see the grounding note.)
**Ranges:** OSC floats are sent in the param's mapped range by default (`applyGroupOverride` applied before send, matching `osc-output.js`), with a per-backend toggle to send **raw normalised 0..1** instead (some OSC targets want 0..1 and do their own scaling). Address prefix (`/nisps`), target host/port (default `127.0.0.1:9000`), and listen port (default `9001`) are configurable — `bridge.ts` already exposes `--osc-host/--osc-port/--osc-prefix/--ws-port/--listen-port`.
@ -156,7 +178,7 @@ For browser-side CV/gate there is no native hardware path, so this backend has *
The headline new backend. A first-class **VCV Rack 2 module** (`MEMLNaut`) with **8 CV inputs → model → 16 CV outputs**, an **LED ring around each of the 16 outputs**, and a **browser↔VCV bridge** so the tool is controllable AND trainable from both inside Rack and entirely from the browser. Full design in §5–§7.
The browser-side adapter `VcvBridgeBackend` reuses `NispsOscClient` (§2.4) as transport. When active in **bridged mode**, the browser supplies inputs in real time (`/nisps/input`) and the verdict/example loop is mirrored over the bridge (`/nisps/state`, `/nisps/weights`, new `/nisps/feedback`).
The browser-side adapter (shipped as `manifold/src/backends/vcv-backend.ts`) reuses the OSC client (§2.4) as transport. When active in **bridged mode**, the browser supplies inputs in real time (`/nisps/input`) and forwards verdicts to the module's embedded learner (`/nisps/feedback`).
---
@ -348,26 +370,26 @@ Mapping rule: `ringColor = kRingPalette[paramMeta[i].group % paletteLen]`, so ou
- **Threading model** (`MEMLNaut.cpp:67–312`): audio-thread `iml` + worker-thread `imlShadow`, job queue (Train/Perturb/Randomize/Clear), atomic weight hand-off, single-writer invariant. Already correct; just resize the I/O.
- **Inference-rate decimation + slew + post-swap crossfade** (`:447–499`).
- **Verdict loop** (`:412–445`): `+`/`−` buttons and `+TRIG`/`−TRIG` gated by LEARN. `+` → add example (current inputs→current outputs) + enqueue Train + decay noise ×0.97; `−` → bump noise (cap `0.3(1−s)+0.05s`) + enqueue Perturb. Identical semantics to the browser verdict loop.
- **Serialization** (`dataToJson`/`dataFromJson`, `:548–696`) + **`.nisps` preset save/load** (`:862–924`) + **OSC server** (`osc_server.hpp`). The `.nisps` format is the shared interchange with the browser (`vcv/NISPS-FORMAT.md`).
- **Serialization** (`dataToJson`/`dataFromJson`) + **`.nisps` preset save/load** + **OSC server** (`osc_server.hpp`). The `.nisps` v3 format is specified in `vcv-module.md` §State Persistence (the separate `vcv/NISPS-FORMAT.md` was deleted).
- Per-output / per-input **range menu** (uni/bipolar, attenuverter) (`:818–843`).
### 5.5 Plugin scaffold files
### 5.5 Plugin scaffold files (as they exist today)
```
vcv/
├── plugin.json # slug MEMLNaut (exists) — bump version; tags Controller/Utility/Random
├── Makefile # VCV SDK Makefile (exists); RACK_DIR-driven
├── Makefile.dist # dist packaging (exists)
├── plugin.json # slug MEMLNaut; tags Controller/Utility/Random
├── Makefile # VCV SDK Makefile; RACK_DIR-driven (make dist via the SDK's plugin.mk)
├── src/
│ ├── plugin.{hpp,cpp} # plugin init / model registration (exist)
│ ├── MEMLNaut.cpp # module — resize 8 in / 16 out (edit)
│ ├── LedRing.hpp # NEW — LED-ring widget (§5.3)
│ ├── palette.hpp # NEW — ring colour table from colors.css (§5.3)
│ └── osc_server.hpp # OSC server (exists; extend with /nisps/feedback §7)
├── res/
│ ├── MEMLNaut.svg / -wide.svg / -expander.svg # panels (exist; redraw for 8/16 layout)
├── test/smoke_test.cpp # exists
├── plugin.json, SPEC.md, BUILDING.md, README.md, NISPS-FORMAT.md (exist)
│ ├── plugin.{hpp,cpp} # plugin init / model registration
│ ├── MEMLNaut.cpp # module — 8 in / 16 out
│ ├── iml.hpp # thin adapter over the shared nisps/ core (P6)
│ ├── LedRing.hpp # LED-ring widget (§5.3)
│ ├── palette.hpp # ring colour table from the design tokens (§5.3)
│ └── osc_server.hpp # OSC server (transport-only; /nisps/feedback per §7)
├── res/ # MEMLNaut.svg / -wide.svg / -expander.svg
├── BUILDING.md, DISTRIBUTION.md, README.md
```
(The 2026-06 draft also listed `Makefile.dist`, `test/smoke_test.cpp`, `SPEC.md` and
`NISPS-FORMAT.md` — all since deleted or absorbed; the module spec is `vcv-module.md`.)
---
@ -380,7 +402,7 @@ The existing salvage path is **WebSocket↔UDP-OSC** (`osc-bridge/bridge.ts` + `
| Option | How | Pros | Cons |
|---|---|---|---|
| **A. WS↔OSC bridge server (recommended)** | Browser ⇄ `bridge.ts` (Deno WS server, localhost:8765) ⇄ UDP OSC ⇄ module's `osc_server.hpp` (port 9000/9001) | Already built + bidirectional; standard OSC; works with SuperCollider/Max too; no browser perms | Needs a helper process running locally (Deno or the compiled `bridge.mjs`) |
| **A. WS↔OSC bridge server (recommended)** | Browser ⇄ `bridge.ts` (Deno WS server, localhost:8765) ⇄ UDP OSC ⇄ module's `osc_server.hpp` | Already built + bidirectional; standard OSC; works with SuperCollider/Max too; no browser perms | Needs a helper process running locally (Deno, or a compiled `nisps-osc-bridge` binary from `compile.sh`) |
| **B. Direct WebMIDI** | Browser ⇄ Web MIDI ⇄ a virtual MIDI port ⇄ a tiny MIDI-in path in the module | No helper process if a virtual MIDI port exists; browser-native | 7-bit/14-bit only — too coarse for weights/state; really only for live CC; module would need MIDI parsing |
| **C. Native (module hosts a WS server)** | The VCV module itself opens a WebSocket/HTTP server; browser connects directly | No external bridge process | Adds a WS/TLS stack inside the plugin; COOP/COEP + mixed-content (`https://` page → `ws://localhost`) friction; more attack surface in the audio plugin |
@ -388,37 +410,35 @@ The existing salvage path is **WebSocket↔UDP-OSC** (`osc-bridge/bridge.ts` + `
### 6.2 Two modes
- **Standalone:** module runs entirely inside Rack (CV in → model → CV out; verdict via panel buttons/triggers). No browser. Works today.
- **Bridged:** browser connects via the bridge. In bridged mode **the browser supplies inputs in real time** — the Manifold pointer / joystick / pads stream `/nisps/input <f…f>` to the module, which uses them instead of (or blended with, configurable) the physical CV-in jacks. The module streams `/nisps/output` and `/nisps/input` back at ~100 ms (`MEMLNaut.cpp:536`) for browser visualisation. State/weights sync both ways via `/nisps/state` + `/nisps/weights` (already staged atomically into the audio thread, `:360–384`).
- **Bridged:** browser connects via the bridge. In bridged mode **the browser supplies inputs in real time** — the Manifold pointer / joystick / pads stream `/nisps/input <f…f>` to the module, which uses them instead of the physical CV-in jacks. The module streams `/nisps/output` and `/nisps/input` back at ~100 ms for browser visualisation.
### 6.3 Coherence model
One model, one owner of weights at a time. The bridge sends **whole-model snapshots** (`/nisps/state`, `/nisps/weights`) on any structural change (train completes, randomize, clear, load preset), and **continuous I/O vectors** (`/nisps/input`, `/nisps/output`) for live feel. Last-writer-wins on weights with a short "training in progress" lock (the module already coalesces jobs at queue depth 1). The browser's `WasmIML` and the module's `nisps::IML` use the **same `.nisps` weight layout** (`vcv/NISPS-FORMAT.md`), so a snapshot from either side loads losslessly — *provided the architectures match*. **Caveat / open choice:** the browser engine is fixed `MLP<2,…,126>` while the module is `8→{16,24,16}→16`. For true weight transfer the bridged session must run a **matched architecture** (e.g. a browser mode configured to 8-in/16-out, within the modular envelope) — otherwise the bridge degrades to I/O + example transfer only (no raw-weight sync). Flag this explicitly.
**One model — the module's.** In bridged mode the browser acts as a remote controller and
trainer for the module's embedded learner: it streams inputs and forwards verdicts
(`/nisps/feedback`), and reads the module's live I/O back. There is **no weight or state sync**
between the browser's own WASM net and the module — the 2026-06 whole-model-snapshot design
(`/nisps/state` / `/nisps/weights`, last-writer-wins, matched-architecture weight transfer) was
deleted 2026-07-21 with zero consumers on either side. Persistence belongs to the Rack patch and
`.nisps` files (`vcv-module.md`).
---
## 7. Training over the bridge — both directions
The verdict loop (place examples / thumbs-up / thumbs-down / undo) must work from **either** end and stay coherent. Add one OSC address and a small `RemoteTrainingBridge`:
The verdict loop works from **either** end, on the module's model. The shipped surface:
```ts
// manifold/src/engine/backends/backend.ts
export interface RemoteTrainingBridge {
thumbsUp(): void; // add current (input,output) example + train
thumbsDown(): void; // perturb weights (explore)
addExample(input: Float32Array, output: Float32Array): void;
randomise(spread: number): void;
clear(): void;
undo(): void;
onState(cb: (s: NispsState) => void): () => void; // remote → local sync
}
```
- OSC verb `/nisps/feedback <s>` carrying a JSON op — **`up` / `down` / `rand` / `clear`**
(optional `spread`, `input`/`output` vectors). The module's `osc_server.hpp` `onFeedback`
callback stages the op atomically for the audio thread, which routes it through the **same**
job/`add_example` paths the panel buttons use.
- **Browser → VCV training:** thumbs-up in the Manifold → `vcv-backend.ts` `sendFeedback('up')`
→ the module's worker trains its shadow instance → atomic weight swap. The browser observes
the result through the live `/nisps/output` stream (not through weight sync — §6.3).
- **VCV panel training** works exactly as in standalone mode; the browser sees the new mapping
through the output stream.
New OSC verb: `/nisps/feedback <s>` carrying `{"op":"up|down|rand|clear|undo","spread":f,"input":[…],"output":[…]}`. The module's `osc_server.hpp` gets an `onFeedback` callback (mirroring the existing `onState`/`onWeights` at `MEMLNaut.cpp:120–132`) that stages the op atomically for the audio thread, which routes it through the **same** `enqueueJob`/`add_example` path the panel buttons use (`:412–445`). So:
- **Browser → VCV training:** user clicks thumbs-up in the Manifold → `VcvBridgeBackend.remote.thumbsUp()` → `/nisps/feedback {op:up,input,output}` → module stages → worker trains `imlShadow` → atomic weight swap → module streams `/nisps/state` back → browser `WasmIML.setWeights` updates so the UI/heatmap reflect the new mapping. The browser need not run its own training in bridged mode (or runs it and pushes weights; configurable — see §6.3 caveat).
- **VCV → browser training:** user presses `+`/`−` on the panel (or sends `+TRIG`) → module trains/perturbs → streams `/nisps/state` → browser applies, so the verdict placed in Rack appears in the browser's example list and weight-health view.
- **Example placing over the bridge:** either side can `addExample`; examples ride in `/nisps/state` (the module already serializes `examples.features`/`examples.labels`, `:587–604`) so the dataset stays in sync. Undo is local-history on each side, but a remote undo can be sent as `/nisps/feedback {op:undo}` to roll the module's last job (module keeps a one-deep snapshot, matching the browser undo stack semantics).
**Result:** the tool is fully controllable and trainable from inside VCV Rack and entirely from the browser, with the verdict loop and example-placing working either way over the same bridge.
The 2026-06 design's richer `RemoteTrainingBridge` (remote `addExample`, `undo`, `onState`
callbacks) was never built; its state-sync legs are deleted (§6.3).
---
@ -429,11 +449,13 @@ Backends live under `manifold/src/engine/backends/`; lifted JS (`c15-*`, `param-
### OSC bridge server
```bash
cd osc-bridge
deno run --allow-net bridge.ts # or the compiled bridge.mjs
cd manifold/osc-bridge
deno run --allow-net --unstable-net bridge.ts
# --osc-host 127.0.0.1 --osc-port 9000 --ws-port 8765 --listen-port 9001
```
Ship `bridge.mjs` (already compiled) + `compile.sh` so users without Deno can run it via Node. Surface "bridge not running" in the OSC/VCV backend status (the client auto-reconnects).
For users without Deno, `compile.sh` cross-compiles standalone `nisps-osc-bridge-<platform>`
binaries into `dist/` via `deno compile`. Surface "bridge not running" in the OSC/VCV backend
status (the client auto-reconnects).
### VCV module
```bash
@ -441,9 +463,12 @@ cd vcv
export RACK_DIR=/path/to/Rack-SDK # VCV Rack 2 SDK
make # builds plugin.so/.dylib/.dll
make install # copies into the VCV user plugins dir
# distribution: make dist (per Makefile.dist; produces the .vcvplugin)
# distribution: make dist (SDK plugin.mk; produces the .vcvplugin — see vcv/DISTRIBUTION.md)
```
Per the search, the standard flow is `export RACK_DIR=…; make clean; make dist` ([Plugin Development Tutorial](https://vcvrack.com/manual/PluginDevelopmentTutorial)). Requires the VCV Rack 2 SDK; nisps-core is header-only C++20 (symlinked under `vcv/dep/` per `SPEC.md`). Ship v2-only (rationale in `SPEC.md`'s v1-compat section). License caveat: VCV SDK is GPLv3, nisps-core is MPL-2.0 — combined binary is effectively GPL; not submitting to the VCV Library initially (`SPEC.md` §License).
Requires the VCV Rack 2 SDK; the shared `nisps/` core is header-only C++20, reached via relative
includes from `vcv/src/` (see `vcv/BUILDING.md`). Ship v2-only (rationale in `vcv-module.md`).
License caveat: VCV SDK is GPLv3 — the combined binary is effectively GPL; not submitting to the
VCV Library initially.
---
@ -465,9 +490,9 @@ Per the search, the standard flow is `export RACK_DIR=…; make clean; make dist
- MIDI: `/home/w1n5t0n/deployments/meml-aimmersive/js/midi/midi-output.js` (`:114` batch throttle), `…/midi/midi-cc-map.js`
- OSC client: `/home/w1n5t0n/deployments/meml-aimmersive/js/nisps/osc-client.js`; param-named client `…/js/synth/osc-output.js` (`:97`)
- OSC bridge server: `/home/w1n5t0n/deployments/meml-aimmersive/osc-bridge/bridge.ts` (addresses `:298–305`, encode/decode `:70–182`), `bridge.mjs`, `compile.sh`
- VCV module: `/home/w1n5t0n/src/MEMLNaut-NISPS/vcv/src/MEMLNaut.cpp` (threading `:67–312`, verdict `:412–445`, ranges `:818–843`, display `:699`, LED `:804`), `…/vcv/src/osc_server.hpp`, `…/vcv/plugin.json`, `…/vcv/SPEC.md`, `…/vcv/Makefile`, `…/vcv/NISPS-FORMAT.md`, `…/vcv/res/*.svg`
- VCV module: `/home/w1n5t0n/src/MEMLNaut-NISPS/vcv/src/MEMLNaut.cpp` (line numbers are pre-P6 grounding), `…/vcv/src/osc_server.hpp`, `…/vcv/plugin.json`, `…/docs/specs/vcv-module.md`, `…/vcv/Makefile`, `…/vcv/res/*.svg`
- Design tokens (ring palette): `/home/w1n5t0n/src/MEMLNaut-NISPS/docs/redesign/manifold-export/tokens/colors.css`
- Spine/engine context: `/home/w1n5t0n/src/MEMLNaut-NISPS/docs/redesign/engine-architecture.md` (§2), `…/findings-design-and-manifold.md` (§4), `…/findings-engine-surface.md`, `…/aimmersive-clone-spec.md` (routeOutputs §6, §7 visual table, §10)
- Spine/engine context: `/home/w1n5t0n/src/MEMLNaut-NISPS/docs/specs/engine-architecture.md` (trimmed 2026-07), `…/recon/findings-design-and-manifold.md` (§4), `…/recon/findings-engine-surface.md`, `…/_archive/aimmersive-clone-spec.md` (routeOutputs §6, §7 visual table, §10)
Sources (VCV SDK / widgets): [VCV custom lights](https://community.vcvrack.com/t/how-to-use-custom-lights/1941), [Migrate2 (drawLayer/layer 1)](https://vcvrack.com/manual/Migrate2), [Plugin Development Tutorial (RACK_DIR/make dist)](https://vcvrack.com/manual/PluginDevelopmentTutorial), [Plugin API Guide](https://vcvrack.com/manual/PluginGuide).
---
@ -493,10 +518,13 @@ spine; WebMIDI out with per-output CC#/ch/name/range; OSC-over-WS to the Deno br
The Outputs dock panel specialises per backend (`manifold/src/dock/OutputsBackendConfig.tsx`) with named presets
(`manifold/src/backends/presets.ts`). Audio gated via `engine.audio.setMuted` on non-synth modes.
**VCV module** — see `vcv/SPEC.md` "⚠️ BUILD DELTAS (2026-06-28)" for the authoritative build target: 8 inputs ×
16 outputs, an LED ring per output (drawLayer + nvgArc), palette from the frontend tokens, WS↔OSC bridge
(browser OSC backend → `manifold/osc-bridge/` Deno relay → the module's OSC server), bidirectional training, and
the `nisps-core`→`nisps/` core-path repoint. The existing `vcv/` module (2→12) is evolved, not rebuilt.
**VCV module** — built: 8 inputs × 16 outputs, an LED ring per output (drawLayer + nvgArc), palette from the
frontend tokens, WS↔OSC bridge (browser VCV backend → `manifold/osc-bridge/` Deno relay → the module's OSC
server), bidirectional training over `/nisps/feedback`, and (P6, 2026-07-18) the core reunification onto
`nisps::ml::MLPCore<DynamicStorage>`. Authoritative module contract: `vcv-module.md`.
**Still TODO** (clear in-code): the VCV↔browser bridge browser-side wiring to the new `/nisps/feedback` verb;
the particle backend is a no-op passthrough pending the `visualizer.js` faithful port; CV/gate transport.
**Update 2026-07-21:** the earlier TODOs closed — `/nisps/feedback` browser-side wiring shipped
(`vcv-backend.ts` `sendFeedback`); the flow-field port shipped (`manifold/src/console/flow-field.ts` +
`ParticleStage`; `particle-backend.ts` is a deliberate no-op transport because the visualiser reads the spine
directly, and selecting it gates the synth audio); CV/gate shipped as the `cvgate` uSEQ Web Serial backend
(`manifold/src/backends/cv-backend.ts`, protocol spec `useq-cv-protocol.md`) rather than the §2.5 designs.

View file

@ -1,8 +1,43 @@
---
kind: spec
stability: evolving
layer: behavioural
---
# Dock Spec — Console Right-Dock Drawers + Per-Output Controls
*Workstream D. Read-only design, 2026-06-27. Target app: `manifold/` (Vite + React + TS), wired to the parity-tested TS engine (`playground/src`) via the headless `EngineApi` boundary (`docs/redesign/engine-architecture.md` §2). Citations are `file:line`. British spelling in product copy. The built-in synth is the **"Powerful Synth Engine"** — the string "C15" MUST NEVER appear in the UI.*
*Workstream D. Designed 2026-06-27; built 2026-06-28 with operator-directed restructuring.
British spelling in product copy. The built-in synth is the **"Powerful Synth Engine"** — the
string "C15" MUST NEVER appear in the UI.*
This spec replaces the placeholder Shape/Feel/Route/Health/Help drawers in `ConsoleApp.jsx` (`docs/redesign/findings-design-and-manifold.md` §2 — keyboard `1-5` map drawers, `\` toggles depth) with the real dock contents.
> **Grounding note (2026-07-21).** Treat every `file:line` cite in the body as historical
> grounding: `playground/*` died with the retired SolidJS playground, `aimmersive-clone-spec.md`
> is archived at `_archive/aimmersive-clone-spec.md`, and `engine-architecture.md` was trimmed
> 2026-07 (its cited § numbers are gone). What shipped diverges from the prescription in shape
> but not in substance:
>
> - **Drawer roster**: the operator restructured the dock during the build — a top **Mode
> selector** plus five drawers (`learn`, `inputs`, `route`, `settings`, `help` —
> `manifold/src/console/Dock.tsx`/`Drawers.tsx`). The §0 six-icon roster (SYNTH/VISUAL as
> drawers) became per-Mode config inside the Outputs drawer instead.
> - **Depths**: two shipped, not three — `condensed` (360px side panel) and `expanded` (centred
> modal). The separate FULL "advanced backend modal" (§4, `<BackendAdvanced/>`) was built,
> then deleted as a duplicate editor (2026-07 sweep, S18); the sole per-backend editor is
> `manifold/src/dock/OutputsBackendConfig.tsx` in the Outputs drawer.
> - **The per-output control row** (§3, the heart of this spec) shipped essentially as designed:
> `manifold/src/dock/OutputControlRow.tsx` + `output-state.ts` (off/fixed/live + mute +
> solo/arm + min/max/curve).
> - The feedback surface shipped richer than §1 (Explore-and-Place with reroll/nudge/place, plus
> geometric dislike) — see `docs/adr/rl-feedback-design.md` and `nisps/ml/feedback.hpp`.
> - **§1.3's diagnostics suite shipped PARTLY, 2026-07-21** (simplification-plan §6.5e). The
> `nisps_ml_loss_history` C API the section waited on now exists, and
> `manifold/src/console/TrainingHealth.tsx` renders the real per-iteration loss curve plus the
> per-layer weight-health table (`nisps_ml_get_layer_stats`) at `expanded` depth — the advanced
> surface, since only two depths shipped. `<GradientFlow>` is NOT built and is not planned as
> drawn: the core records no per-layer gradient magnitudes, and the fabricated version of it was
> deleted in the Phase-1 sweep. `<WeightHealth>` as an edge-glow is likewise unbuilt; the same
> numbers are in the table. §1.3's six live sliders were deleted as decorative (S16) — `noise
> cap` and the Xavier switch are the survivors, because they drive real engine setters.
---
@ -41,18 +76,19 @@ Owns *how the model learns from your gestures*: the feedback-mode selector, the
### 1.1 FEEDBACK_MODE selector ("Down Action")
The `+` (up) verdict is always "keep this" (`addExample` + train, `findings-feedback-behaviour.md:102`); the `−` (down) verdict is **selectable** among the ported feedback modes (`feedback-modes-port-spec.md` §1; `FeedbackController<MLP_T>`). The new 2-mode product surface (per the prompt's "Mode 1 / Mode 2"):
The `+` (up) verdict is always "keep this" (`addExample` + train, `findings-feedback-behaviour.md:102`); the `−` (down) verdict is **selectable** among the ported feedback modes (`plans/feedback-modes-port-spec.md` §1; `FeedbackController<MLP_T>`). The new 2-mode product surface (per the prompt's "Mode 1 / Mode 2"):
| Selector label (UI copy) | Engine `FeedbackMode` | Behaviour |
|---|---|---|
| **Push away** (Mode 1) | `Avoid` (`feedback.hpp` enum, `findings-feedback-behaviour.md:90`) | down → geometric-dislike: perturbs the mapping away from what you disliked. In the deployed core this routes to `move_weights(speed, spread, pinMask)` (`feedback-modes-port-spec.md` §2.5 — the true k-NN centroid push is firmware-only, out of scope). |
| **Explore & place** (Mode 2) | `RandomiseMlp` (`findings-feedback-behaviour.md:122`) | down → snapshot + `draw_weights(spread)` re-rolls the whole net into a scratchpad you audition by moving the joystick; `+`/drag commits a `+1` example at the chosen input and **restores the real net** (`findings-feedback-behaviour.md:135-146`); down-again **cancels** (restore snapshot). |
| **Push away** (Mode 1) | `Avoid` (`nisps/ml/feedback.hpp`) | down → geometric dislike: stores the rejection and immediately pushes the mapping away from what you disliked; controller-owned elapsed-time replay then continues optimising live rejections (`docs/adr/rl-feedback-design.md`, 2026-07-25 amendment). |
| **Explore & place** (Mode 2) | `ExploreAndPlace` (`nisps/ml/feedback.hpp`) | down → snapshot + `draw_weights(spread)` re-rolls the whole net into a scratchpad you audition by moving the joystick; `+`/drag commits a positive anchor at the chosen input and **restores the real net**; re-roll and cancel remain distinct Explore-and-place actions (`docs/adr/rl-feedback-design.md` §2.2). |
- **Verdict pointer timing is mode-specific:** in **Push away**, pointer-down MUST dispatch exactly one geometric dislike immediately; holding, pointer-up, and release MUST NOT dispatch another dislike, re-roll/randomise, or cross-route into Explore-and-place. Controller-owned elapsed-time replay MAY continue after that single UI dispatch and is not a repeated held-pointer dispatch. **Explore & place alone** MAY interpret its existing long press as a re-roll.
- A third engine mode `RandomiseOutputs` (bypass MLP, hold a static random vector, `findings-feedback-behaviour.md:111`) exists in the core but is **not** surfaced as a product mode in v1 — expose it only behind `?debug=1` as "Static roll". **(Open choice 1.)**
- **Peek**: a 2-segment pill (`Push away` / `Explore & place`), the current mode highlighted in `--accent`. This pill is *also* mirrored next to the Verdict cluster (`VerdictCluster`, `findings-design-and-manifold.md:50`) so it is reachable during live play without opening the drawer — matching how a-immersive puts the `rl-label` above the RL buttons (`aimmersive-clone-spec.md:283`).
- **Expand**: the pill + a one-line plain-English description of the active mode + an **"exploring…" indicator** that lights when `engine.feedback.exploring()` is true (`findings-feedback-behaviour.md:237`), reusing the `<NoiseRing>` colour ramp (off/active/high, `aimmersive-clone-spec.md:98`). While exploring in Mode 2, training is paused (`learning_paused()`, `findings-feedback-behaviour.md:48`) — show a small "learning paused" badge.
- **FULL**: the feedback lab — a diagram of the active state machine (idle → exploring → commit/cancel), the raw `FeedbackAction` log of the last presses (`findings-feedback-behaviour.md:90` enum), plus the spread/tame "Health lab" sliders (§1.3) for radical exploration tuning.
- **Bind point**: `engine.feedback.setMode(mode)` → `nisps_ml_feedback_set_mode` (`feedback-modes-port-spec.md` §4). Switching mode while exploring auto-aborts and restores the net (`set_mode` calls `abort_explore`, `findings-feedback-behaviour.md:96`). Persist `feedbackMode` in the session blob.
- **Bind point**: `engine.feedback.setMode(mode)` → `nisps_ml_feedback_set_mode` (`plans/feedback-modes-port-spec.md` §4). Switching mode while exploring auto-aborts and restores the net (`set_mode` calls `abort_explore`, `findings-feedback-behaviour.md:96`). Persist `feedbackMode` in the session blob.
### 1.2 SOLO / arm variant chooser
@ -317,7 +353,7 @@ Shared leaf primitives live in `shared/primitives/` (`<Slider>`, `<DualRangeSlid
| UI surface | EngineApi call | Engine route |
|---|---|---|
| Feedback-mode pill | `engine.feedback.setMode(m)` | `nisps_ml_feedback_set_mode` (`feedback-modes-port-spec.md` §4) |
| Feedback-mode pill | `engine.feedback.setMode(m)` | `nisps_ml_feedback_set_mode` (`plans/feedback-modes-port-spec.md` §4) |
| `−` verdict | `engine.feedback.thumbsDown()` | dispatch on mode → `on_down` (`findings-feedback-behaviour.md:90`) |
| `+` verdict | `engine.feedback.thumbsUp()` | `addExample` + train, or `CommitStore` when exploring |
| Arm (S) | `engine.feedback.setFocus(mask)` | `nisps_ml_feedback_set_focus` + `buildPinMask` |
@ -357,7 +393,7 @@ The design reasoning, drawer model, tri-state/mute/arm separation and per-output
`mapping.ts`), not imported from the deployed snapshot.
- **`--dock-width` and min-thumb `#4488ff` are deployed-a-immersive values, absent from the Manifold tokens** —
use Manifold tokens (`manifold-export/tokens/`) or add the missing ones deliberately.
- A few `findings-feedback-behaviour.md:NN` citations are out of range / point at `feedback-modes-port-spec.md`
- A few `findings-feedback-behaviour.md:NN` citations are out of range / point at `plans/feedback-modes-port-spec.md`
instead (`learning_paused()` = `feedback.hpp:85` / port-spec:238; `exploring()` = `feedback.hpp:84`).
- The React `<EngineProvider>`/`useEngine` pattern is in `findings-design-and-manifold.md §4`, not
`engine-architecture.md §2.2` (that doc is SolidJS).

View file

@ -0,0 +1,89 @@
---
kind: spec
stability: stable
layer: cross-cutting
---
# Browser Engine Architecture — the EngineApi Seam and the Reactive Spine
*Supersession note (2026-07): this spec's original "One Engine, Two Skins" framing — a fresh
`playground2/` tree hosting a faithful a-immersive skin plus a Console skin — is superseded by
`plans/one-core-engine-refactor.md` (executed), which states: "the seam is built — Manifold is its
realisation; the axis of unification now runs vertically, core↔targets, not horizontally,
skin↔skin." The playground2 directory plan, reuse-vs-rebuild table, two-entry Vite/nginx serving
scheme, S0–S7 build sequence, and operator open questions were all consumed by the Manifold build
(`plans/BUILD-PLAN.md`, executed) and are deleted here; git history keeps them. What remains below
is the surviving contract: the headless `EngineApi` boundary and the reactive-spine invariant, as
realised in `manifold/src/engine/`.*
---
## 1. The seam: engine vs presentation
**The engine is separated from presentation by an explicit, headless boundary — `EngineApi` — and
the UI is a pure consumer of it.** The litmus test for every file: if it imports React, JSX, or
touches the DOM, it is presentation; otherwise it is the engine. The engine compiles and is
testable headlessly (the `window.__nisps` probe under `?debug=1`) with zero presentation.
```
┌──────────────────────────── ONE ENGINE ────────────────────────────┐
gesture │ input pipeline → ML (WASM) → output pipeline → backends │ sound
────────► │ (the reactive spine: one derivation path + one side-effect) │ ────────►
│ exposes a headless EngineApi — NO JSX, NO DOM │
└────────────────────────────────────────────────────────────────────┘
▲
│ reads EngineApi
┌──────────┴───────────┐
│ Manifold Console │
│ (React, manifold/) │
└──────────────────────┘
```
Realisation (see `MAP.md` §manifold for the per-file inventory):
- `manifold/src/engine/engine-api.ts` — the `EngineApi` class: the entire UI-facing surface
(inputs, training, feedback wrappers over the `nisps_ml_feedback_*` C ABI, audio lifecycle).
- `manifold/src/engine/spine.ts` — the reactive spine (§2).
- `manifold/src/engine/EngineProvider.tsx` + `useEngine.ts` — the React binding via
`useSyncExternalStore`. These two files are the only presentation-side entry to the engine.
- `manifold/src/debug/probe.ts` — `window.__nisps`, reads `EngineApi` only, gated by `?debug=1`.
The UI cannot create a second data path because it has no access to `WasmIML`, the backend, or the
spine's internals — only to `EngineApi`. That structural property, not vigilance, is what makes
the historic "MLP output stops updating when I move the joystick" bug class impossible.
---
## 2. The reactive spine
There is exactly one path from gesture to sound, every consumer reads it, and a desync is a
failing test rather than a recurring prod bug.
### 2.1 The derivation path (`manifold/src/engine/spine.ts`)
The spine is an external store living BELOW React (deliberately not on the render scheduler): the
`setInput` action derives processed → ml → routed **eagerly and synchronously** (the input
pipeline, inference, and output pipeline are all C++/WASM since one-core-engine P4) and fires the
single `backend.send` at the action's tail, off-render. Contract rules:
- **One side-effect.** `backend.send(routed)` at the tail of `setInput` is the only transport
call in the derivation path.
- **No per-frame allocation.** Buffers are reused (`routedBuf` threaded through the output
pipeline and handed to the backend).
- **Weights mutate only through engine actions** (train / feedback / reshape). Structural/training
state bumps `version`; every inference advances a separate `outputVersion`. Both expose explicit
subscriptions, so audio propagation never depends on React scheduling.
- **React state and live output are separate channels.** Structural/training consumers use
`useSyncExternalStore(subscribe, version)`. DOM output consumers opt into the throttled 30 Hz
`subscribeOutputs/outputVersion` channel. Canvases read and shape the live `Float32Array`s
imperatively in rAF and never re-render per frame.
### 2.2 The live-feedback guarantee and its e2e assertion
**Guarantee:** any change to the input (gesture) or the weights (train / feedback) propagates — in
the same synchronous action — to the routed output, which the single effect sends to the backend
and which every UI consumer reads.
The invariant is asserted in CI by `manifold/tests/e2e/spine.spec.ts` via the synchronous
`window.__nisps` probe: set inputs, read outputs, expect change (plus probe-survives-mode-switch).
Because the probe reads `EngineApi` only, it exercises the exact path the UI uses.

View file

@ -1,6 +1,39 @@
---
kind: spec
stability: evolving
layer: binding
---
# Inputs Spec — Modular Input Layer (Workstream F)
*Status: implementation-ready spec for the `manifold/` React app. Scope: the modular input layer — sources (XY pad, MIDI input, gamepad single/double-stick), how they compose into an N-dimensional input vector, the MLP-rebuild-on-input-change mechanism, the binding to the reactive spine, and the dock INPUTS panel. Read alongside `docs/redesign/findings-engine-surface.md` (the fixed-2-input gap), `docs/redesign/engine-architecture.md` (the `EngineApi` seam — note that doc says SolidJS/`playground2`; this app is React/`manifold/`), and `docs/redesign/aimmersive-clone-spec.md` (existing gamepad/MIDI/joystick behaviour). British spelling in product copy. The built-in synth is always shown as the "Powerful Synth Engine" — never "C15".*
*Scope: the modular input layer of the `manifold/` React app — sources (XY pad, MIDI input,
gamepad single/double-stick), how they compose into an N-dimensional input vector, the
MLP-reshape-on-input-change mechanism, the binding to the reactive spine, and the dock INPUTS
panel. British spelling in product copy. The built-in synth is always shown as the "Powerful
Synth Engine" — never "C15".*
> **Grounding note (2026-07-21).** This spec was written 2026-06-28 and implemented; the body is
> kept for the behavioural intent, but treat it as follows:
>
> - **Every `file:line` cite below is historical grounding, not a live pointer.** `playground/*`
> files died with the retired SolidJS playground (branch `archive/playground-solidjs`);
> `aimmersive-clone-spec.md` is archived at `_archive/aimmersive-clone-spec.md`;
> `engine-architecture.md` was trimmed 2026-07 so its cited line numbers no longer exist;
> `findings-*` recon docs are immutable snapshots of the pre-refactor world.
> - **What shipped** lives in `manifold/src/inputs/` (not the proposed `engine/input/`):
> `input-layer.ts` (single rAF compose loop), `xy-pad-source` / `gamepad-source` /
> `midi-input-source`, `useInputLayer.ts` React binding. See `MAP.md` §manifold.
> - **§4's option analysis was overtaken by events.** The recommendation (Option B, multiple
> fixed-shape WASM modules) never shipped: one-core-engine P2 made the ONE parity-tested core
> runtime-shaped in the browser (`MLPCore<DynamicStorage>`, `nisps_ml_reshape` warm-start) — so
> honest N-dim nets landed via Option A's shape *without* the divergent-second-core cost that
> §4 rejected it for. The shipped layer feeds axes into dedicated input slots of an
> over-provisioned 32-input head and offers a warm-started reshape when the active axis count
> changes (reset-on-reshape modal).
> - **Source composition**: the engine layer composes multiple sources, but the dock currently
> exposes an exclusive one-at-a-time picker (`InputMode` = internal | gamepad | midi).
> Mix-and-match UI is an unreversed decision with groundwork deliberately laid — operator
> decision pending (`plans/simplification-plan.md` §7.7).
---

View file

@ -1,8 +1,15 @@
---
kind: plan
status: executed
---
# Manifold — Build Plan & Locked Decisions (resume anchor)
*Dated 2026-06-27. This is the single resume anchor for the Manifold convertible-app build. Read this + the
specs it points to before continuing. Mission: one working browser instrument putting the NEW Manifold
"convertible" front-end on top of the REAL parity-tested ML+audio engine.*
*Dated 2026-06-27. **Executed** — the Manifold app was built, deployed to `/next/`, and is the sole
browser app; current ground truth is `MAP.md` §manifold + `manifold/ONBOARDING.md` (the manual
redeploy recipe below is superseded by the CI-gated push-to-main webhook). What survives from here:
the VPS non-snap-node Playwright invocation (now in `MAP.md` Entry points) and the locked
naming/copy decisions (never "C15", British spelling).*
## Locked decisions (operator-confirmed 2026-06-27)
| Decision | Choice |
@ -12,23 +19,23 @@ specs it points to before continuing. Mission: one working browser instrument pu
| Staging deploy | `meml.lnfinitemonkeys.org/next` (server-scope COOP/COEP already set); live a-immersive stays at `/` |
| Default FEEDBACK_MODE | **Explore & place** (Mode 2, positive-only); Geometric-dislike selectable |
| Default SOLO/arm | **Mask-gradients / column-freeze** (variant a); ZeroLoss + DontCareExample selectable |
| Modular N×M MLP | **Multiple WASM modules + warm-start** ({2,4} then {2,4,8}); NOT runtime-shaped, NOT padded |
| Modular N×M MLP | ~~**Multiple WASM modules + warm-start** ({2,4} then {2,4,8}); NOT runtime-shaped, NOT padded~~ **SUPERSEDED 2026-07-13** by `one-core-engine-refactor.md` §1: runtime-shaped MLP in browser (warm-start retained), fixed template on hardware |
| VCV bridge transport | **WS↔OSC bridge server** (reuse existing Deno bridge; bidirectional training) |
| Synth UI label | **"Powerful Synth Engine"** — the string "C15" must NEVER appear in the UI |
| Product copy | British spelling (randomise, visualise, colour) |
## Spec docs (all in docs/redesign/)
- `rl-feedback-design.md` — the learning engine (both modes + solo); **§1 = verified ground truth: the feedback
## Spec docs (now in docs/specs/ and subdirs)
- `docs/adr/rl-feedback-design.md` — the learning engine (both modes + solo); **§1 = verified ground truth: the feedback
C API, `MLHandle.feedback`, `0xFEEDBACC0DE` salt, parity Stage 5, and `CMakeLists.txt:59` registration ALL
ALREADY EXIST** — edit/extend, don't re-scaffold. Mode 1 ports to upstream **`0a541cc`** InterfaceRL.
- `dock-spec.md` — six drawers + per-output control row + tri-state semantics + advanced modals.
- `backends-spec.md` — `OutputBackend` adapter interface; **VCV module already at `vcv/` (2-in/12-out)** →
evolve to 8→16 + LED rings; faithful `visualizer.js` particle port (algorithm documented).
- `inputs-spec.md` — modular XY/WebMIDI/gamepad; multiple-WASM-module reshape with warm-start.
- `findings-{feedback-behaviour,engine-surface,design-and-manifold}.md` — Phase-1 audits.
- `upstream-firmware-survey.md` — git topology; latest InterfaceRL = `0a541cc`.
- `playground-2.0-rewrite-plan.md`, `engine-architecture.md`, `aimmersive-clone-spec.md`,
`feedback-modes-port-spec.md` — the prior planning corpus.
- `recon/findings-{feedback-behaviour,engine-surface,design-and-manifold}.md` — Phase-1 audits.
- `recon/upstream-firmware-survey.md` — git topology; latest InterfaceRL = `0a541cc`.
- `plans/playground-2.0-rewrite-plan.md`, `engine-architecture.md`, `_archive/aimmersive-clone-spec.md`,
`plans/feedback-modes-port-spec.md` — the prior planning corpus.
## Reactive spine → React (load-bearing, from findings-design-and-manifold.md §4)
Spine lives BELOW React in an external store: `setInput` action derives processed→ml→routed eagerly+synchronously

View file

@ -0,0 +1,524 @@
---
kind: plan
status: active
---
# Browser Mode Coverage Honesty — audio topology as declared, verified truth
*Dated 2026-07-21. Spec for `plans/simplification-plan.md` §6.5b (audit finding `A2`), written
spec-first per plan §6 — **proposal only, no phase adopted**. Companion decisions live in §8.
Absorbs the old ALIGNMENT "browser-only engines incomplete / mic input" defect.*
*Every claim below was checked against the tree at `fd0aee2`. Measurements are reproducible; the
exact commands are given. Where I could not verify something I say so instead of asserting it.*
---
## §0 What is actually true today (measured, not inferred)
Two independent probes, both run 2026-07-21:
1. **Native**, engines compiled straight from `nisps/engines/*.hpp` (`g++ -std=gnu++20 -O2 -I nisps`).
2. **The committed WASM artifact** `manifold/public/nisps.{js,wasm}`, driven through the exact C ABI
the AudioWorklet uses (`nisps_engine_create` / `set_params` / `process_block`), loaded with the
same glue dance as `manifold/tests/loss-history.test.ts`.
Both were run for 2 s at 48 kHz with all params 0.5, first with a silent input, then with a
0.3-amplitude 220 Hz sine. Output RMS (left channel):
| engine | silent in | sine in | native == wasm |
|---|---|---|---|
| `paf_synth` | **0.000000000** | *(ignores input)* | yes |
| `channel_strip` | 0.000000000 | 0.446555 | yes (≤1e-6) |
| `xiasri` | 0.000000000 | 0.848909 | yes (≤1e-6) |
| `verb_fx` | 0.000346808 | 0.267881 | yes (≤1e-6) |
| `memlcelium` | **0.399765** | *(ignores input)* | yes |
| `breakor` | 0.000000000 | — | yes |
| `elysiamorf` | 0.000000000 | — | yes |
| `analysis` | 0.000000000 | 0.000000000 | yes |
| `thru` | 0.000000000 | — | yes |
Reading that against what the browser host actually supplies:
- **Nothing is ever connected to the worklet's audio input.** `EngineHost` constructs the node with
`numberOfInputs: 1` (`manifold/src/engine/engine-host.ts:117`) and connects it to the destination
(`:121`), but there is no `getUserMedia`, `MediaStream`, or `createMediaStreamSource` anywhere in
`manifold/src` (grep: zero hits). The worklet already *reads* `inputs[0]`
(`manifold/src/engine/worklet/nisps-processor.ts:266-271`) and zero-fills when absent — so the
DSP side is done and the gap is one main-thread connection.
- **The engine C ABI has four entry points only**: create / destroy / set_params / process_block
(`nisps/wasm/bindings.cpp:1074-1108`, exported at `scripts/build-wasm.sh:70-71`). There is no
`note_on`, no `set_playing`, no `update_bpm`, no `pop_events`, no feature readback.
Therefore, **of the nine schema-backed modes, exactly two produce audio in the browser today**:
`memlcelium` and `slp_workshop` (which share the `memlcelium` engine).
| mode | why it is silent / inert in the browser |
|---|---|
| `paf_synth` | Note-gated. `process()` multiplies by `env_.play()` and the ADSR idles in `WaitToTrig` → exactly 0 (`nisps/engines/paf_synth.hpp:114`, `nisps/dsp/env.hpp`). Only `note_on` starts it (`:132-138`), and no browser path calls it. |
| `channel_strip` | Pure input processor; no audio input connected. |
| `verb_fx` | Pure input processor; RMS 3.5e-4 from a zero input is residual, not a signal. |
| `xiasri` | Pure input processor; no audio input connected. |
| `breakor` | Event-only by design (`nisps/engines/breakor.hpp:81-109` returns `{0,0}`). `playing_` defaults `true` (`:159`) so it *does* tick and enqueue, but `pop_events` is not exported, so the 64-slot `EventQueue` fills and silently drops (`nisps/core/event_queue.hpp:44-49`). |
| `elysiamorf` | Same shape (`nisps/engines/elysiamorf.hpp:100-104`, `playing_ = true` at `:136`). |
| `sound_analysis_midi` | Needs mic *and* a features readback. `AnalysisEngine::process` returns silence and exposes `features()` / `copy_features()` (`nisps/engines/analysis.hpp:129`, `:188-196`) — neither is exported. |
| `visualizer` (manifold-only) | Correctly silent: it is a visual mode on `thru`. |
| `c15` (manifold-only) | Already honest — `placeholder: true`, `badge: 'soon'` (`manifold/src/console/model.ts:301-313`). |
### How the table was produced
Both probes were throwaway scripts in `/tmp`, not committed — **§5 step 2 turns them into permanent
tests, which is the point of this document.** Their shape, so they can be rebuilt in ten minutes:
- **Native.** One `.cpp` including `nisps/engines/*.hpp`; per engine: default-construct,
`setup(48000.f)`, `set_params` with a `param_count()`-sized vector of `0.5f`, then 96 000 calls to
`process({s, s})` accumulating `y.L² + y.R²`; `s` is 0 or `0.3 · sin(2π·220·i/48000)`. Built with
`g++ -std=gnu++20 -O2 -I nisps`.
- **Committed WASM.** A `bun` script using the same glue-loading dance as
`manifold/tests/loss-history.test.ts:34-45` (`new Function(...)` over `public/nisps.js`, then
`factory({ wasmBinary: public/nisps.wasm })`), then `M.cwrap` for `nisps_engine_create`,
`nisps_engine_set_params`, `nisps_engine_process_block`, `nisps_engine_destroy`; engine id written
as NUL-terminated ASCII into a 32-byte `_malloc` buffer exactly as the worklet does
(`nisps-processor.ts:217-230`); 750 blocks × 128 frames.
The two agreed to ≤1e-6 on every engine — incidental corroboration that WASM parity holds on a path
`scripts/parity-check.sh` does not cover.
---
## §1 Where the brief and the audit are wrong
Trust the code over these documents, including this one.
1. **"4 modes that structurally cannot run" is an undercount.** Audit `A2` names `breakor`,
`elysiamorf`, `xiasri`, `sound_analysis_midi`. `channel_strip` and `verb_fx` are in exactly the
same position as `xiasri` — all three are pure input processors with no input connected — and
`paf_synth`, the catalogue's flagship, is silent for a *third* reason the audit's four-way enum
cannot even express. The real number is **7 of 9**.
2. **"The UI presents them as available" is false today.** There is no instrument-mode picker in
Manifold at all. `ConsoleApp.tsx:326-329` says so explicitly; `ctx.modes` / `ctx.setModeId` are
plumbed with no renderer (`manifold/src/console/types.ts:50-57, 64-66`); the Dock's "Mode"
selector is the **output backend** picker, not the instrument (`manifold/src/console/output-mode.ts:1-18`).
The only way to reach another mode is `window.__mf.setMode()` under `?debug=1`.
**Consequence for sequencing:** the dishonesty is *latent*. It fires the moment §6.5c builds the
picker. 5b is therefore a **prerequisite of 5c**, and a "badge" alone would ship a component no
user can see. What 5b must deliver is the *data* plus the *capabilities*, so that 5c's picker has
something true to render and most modes no longer need a warning at all.
3. **The parity harness's PAFSynth stage is vacuous.** `tests/cpp/parity_check.cpp:175-190` runs
PAFSynth for 128 frames with silent input and no `note_on`, then averages — both sides push
exactly `0.0, 0.0`. Stage 3 compares zero to zero. (Stage 4, ChannelStrip, at least feeds a
0.25 DC step.) This is on top of the known limits the brief states. A parity PASS is not evidence
for anything in this document.
4. **§7.2's DriverConfig work changed the firmware, not the browser.** `Mode::driver_config()` is
now part of the `nisps::Mode` concept (`nisps/core/concepts.hpp:74-80`), defaults to
`engine().driver_config()` in `ModeBase` (`nisps/modes/base.hpp:145-165`), and firmware brings
the codec up from it (`firmware/MEMLNaut-NISPS/glue/audio_driver.hpp`). **Mic vs line is real on
hardware.** Nothing in `manifold/` or `nisps/wasm/bindings.cpp` reads `DriverConfig` — the WASM
bridge never calls it. So yes, the browser side is the remaining gap for *routing*, but see the
next point, which is why 5b cannot simply reuse `DriverConfig`.
5. **`DriverConfig` is not a topology declaration and must not be pressed into service as one.** It
is a codec-negotiation record: `mic_input`, `mic_gain_db`, `line_level`, `output_volume`,
`sample_rate` (`nisps/core/types.hpp:50-71`, struct at `:65`). It *correlates* with topology by accident — the
three input-processing engines all set `line_level = 6u` — but `paf_synth` also deviates
(`output_volume = 0.9f`) while consuming no input, `breakor`/`elysiamorf` return a default
`{}` identical to a mode that expresses no opinion, and `analysis`'s `mic_input = true` is about
*which codec pin*, not about *whether audio input is required*. Inferring topology from
`line_level != 3` would be a guess dressed as a fact. Topology needs its own declaration.
---
## §2 The model
### §2.1 Why not the flat four-class enum
The plan proposes `generator | audio-in-fx | event-only | analysis`. That enum cannot express
`paf_synth`, which is a generator that produces nothing without note events — the single biggest
user-visible gap in the catalogue. Three independent facts are being conflated:
| axis | question | values |
|---|---|---|
| `input` | does the engine need an audio input signal to do its job? | `none` \| `required` |
| `output` | what does the engine emit? | `audio` \| `events` \| `none` |
| `trigger` | does it need note events to sound? | `free` \| `notes` |
The four familiar class names survive as a **derived label**, not stored data:
```
output === 'events' → "Event sequencer"
output === 'none' && input==='required'→ "Analysis"
input === 'required' → "Audio FX"
otherwise → "Generator"
```
### §2.2 The declared values
| mode | `input` | `output` | `trigger` | derived label |
|---|---|---|---|---|
| `paf_synth` | none | audio | **notes** | Generator |
| `channel_strip` | **required** | audio | free | Audio FX |
| `verb_fx` | **required** | audio | free | Audio FX |
| `xiasri` | **required** | audio | free | Audio FX |
| `memlcelium` | none | audio | free | Generator |
| `slp_workshop` | none | audio | free | Generator |
| `breakor` | none | **events** | free | Event sequencer |
| `elysiamorf` | none | **events** | free | Event sequencer |
| `sound_analysis_midi` | **required** | **none** | free | Analysis |
### §2.3 Host capability, and a derived verdict — not a hard-coded badge
The catalogue must never carry a hand-maintained "works in browser" flag: it would be wrong the day
a capability lands. Instead, the browser declares what it can currently provide, and support is
**computed**:
```ts
interface HostAudioCapabilities {
audioInput: boolean; // a MediaStream is connected to the worklet node
noteTrigger: boolean; // nisps_engine_note_on/off exist and are wired
eventDrain: boolean; // worklet → host event channel exists
analysisFeatures: boolean; // analysis feature readback exists
}
type ModeSupport =
| { ok: true }
| { ok: false; reason: 'needs-audio-input' | 'needs-note-source' | 'hardware-only'; detail: string };
```
`modeSupport(schema.audio, host)` is a pure function; the UI renders its result. As each capability
in §5 lands, its flag flips to `true` and modes stop being labelled — no catalogue edit, no badge
list to maintain. Today the flags are `{ audioInput: <runtime>, noteTrigger: false, eventDrain:
false, analysisFeatures: false }`.
---
## §3 Where the truth lives
**In `schemas/modes/<mode>.json`, as a new required `audio` object; codegen emits C++ and TS.**
Rejected alternatives, with reasons:
- *A hand-written topology map in `manifold/src/console/model.ts`* (the audit's suggested fix) —
this is a seventh hand-maintained mode registry, straight against Phase 3, which just finished
deleting six of them (`simplification-plan.md` §4). It also puts a fact about C++ engines in
TypeScript, where no C++ test can check it.
- *A WASM export `nisps_engine_topology(id)`* — `MF_MODES` is built at module load
(`model.ts:267-316`); making it await WASM would make the whole catalogue async for a static fact.
- *Deriving it from `DriverConfig`* — see §1.5.
### §3.1 Schema shape
Added to `schemas/schema.json` (`required` gains `"audio"`, `additionalProperties: false` preserved):
```json
"audio": {
"type": "object",
"required": ["input", "output", "trigger"],
"additionalProperties": false,
"properties": {
"input": { "enum": ["none", "required"] },
"output": { "enum": ["audio", "events", "none"] },
"trigger": { "enum": ["free", "notes"] },
"dsp_engine_id": {
"type": "string", "pattern": "^[a-z][a-z0-9_]*$",
"description": "The engine that occupies the audio path for this mode. Defaults to the mode's engine_id; declared only when the mode's ModeBase engine slot is a NoOp and a separately-composed engine owns the audio (sound_analysis_midi → analysis)."
}
}
}
```
`dsp_engine_id` is optional and is set on exactly one mode, `sound_analysis_midi: "analysis"`. It
exists to **delete** a hard-coded exception — see §6.
### §3.2 Codegen anchors
`codegen/generate.ts`, mirroring what `ui` already does — each of these is one line-group:
| what | anchor |
|---|---|
| input TS type for a schema file | `interface ModeSchema` at `:93-126` (its `ui` block is `:116-120`) |
| C++ enums + `struct AudioConfig` in `schema_types.hpp` | beside `enum class PrimaryInput` `:264-272` / `struct UIConfig` `:273-277` |
| `AudioConfig audio;` member on `ParamSchema` | `struct ParamSchema` at `:288-300` |
| per-mode `inline constexpr AudioConfig k<Mode>Audio` | beside the `// UI` block, `:503-518` |
| TS `AudioConfig` interface + `ModeSchema.audio` | `export interface UIConfig` at `:374-378` |
| TS per-mode emission | beside `:631-635` |
Regenerate with `cd codegen && bun run generate.ts`; the golden snapshot
`codegen/tests/golden/paf_synth_schema.{hpp,ts}` must be updated in the same change, and CI's
codegen dirty-diff gate (added in plan §1.3) will fail on anything stale.
---
## §4 Invariants
Beyond the repo-wide constraints (platform-neutral allocation-free `nisps/`, 1e-5 parity, both
codegen outputs in one change, RT-safe worklet/dual-core comms), this area has its own:
1. **Declared topology is verified by behaviour, never by comment.** Every value in §2.2 is asserted
by a test that drives the engine (§5 step 2). Adding an engine or a mode without a topology
declaration must fail to build; declaring one falsely must fail a test.
2. **The support verdict is derived, never stored.** No "browser-ok" boolean in a schema, a
catalogue, or an overlay. See §2.3.
3. **`nisps/` gains no browser-only concept.** `audio` is a schema/codegen fact consumed by the
generated `ParamSchema` and by tests. Firmware behaviour must not change: no engine or mode
source file changes in step 1, and `nisps_modes_tests` / `nisps_dsp_engine_tests` must stay green
with no edits to existing cases.
4. **Microphone input is opt-in, per session, and never auto-enabled.** Mic → worklet → destination
is an acoustic feedback loop through the room. Enabling requires an explicit user gesture and a
headphones warning; the mic is disconnected on `EngineHost.stop()` and on mode switch away from
an `input: 'required'` mode.
5. **No new audio-thread allocation.** Anything crossing worklet↔main (events, analysis features)
uses a pre-allocated channel. `postMessage` of a freshly-allocated typed array per render quantum
is not acceptable; COOP/COEP are already set for `manifold` (see `vite.config.ts` /
`ONBOARDING.md` §2), so `SharedArrayBuffer` is available and is the intended mechanism.
6. **The exclusive `InputMode` picker stays exclusive and stays ML-only.** `internal | gamepad |
midi` (`manifold/src/console/Drawers.tsx:284-288`) selects the *ML input source*. Microphone
audio is not an ML input source (except transitively for `sound_analysis_midi`, whose features
would enter through the engine, not through `input-layer.ts`). Do not add a fourth entry.
7. **Silence must be explained, not shipped.** If a mode is selectable and cannot sound, the UI says
which capability is missing. "It just doesn't make a noise" is the bug this document exists to
remove.
---
## §5 Implementation sequence
Each step is independently landable and independently verifiable. Steps 1–3 are the honest minimum;
4–6 are scoped by the decisions in §8.
### Step 1 — declare topology (schema + codegen, both languages)
Edit `schemas/schema.json` (§3.1) and all nine `schemas/modes/*.json` with the §2.2 values, plus
`dsp_engine_id: "analysis"` on `sound_analysis_midi`. Extend `codegen/generate.ts` at the §3.2
anchors. Regenerate; commit `nisps/modes/generated/`, `manifold/src/modes/generated/`, and the
codegen golden.
*No behaviour changes.* Note honestly: `ui.primary_input` / `show_voice_space_selector` /
`show_synth_visualizer` are generated into both languages and read by **nothing** outside codegen and
its golden (verified by grep across `manifold/src`, `nisps/`, `firmware/`). `audio` must not repeat
that — it acquires two consumers in steps 2 and 3 of this same plan.
**Verification:** `bash scripts/build-cpp-tests.sh` (generated headers still compile into
`nisps_modes_tests`); `cd codegen && bun run test` (golden + curve-drift); `cd manifold && bun run
typecheck`; `bun run codegen/generate.ts` twice → no diff (idempotence).
### Step 2 — make the declaration honest (two behavioural tests)
**2a. `tests/cpp/test_mode_audio_topology.cpp`**, added to the `nisps_modes_tests` target
(`nisps/CMakeLists.txt:119-144` — that target already has the repo root on its include path and
already links the generated schemas). For every generated mode schema, resolve
`audio.dsp_engine_id ?? engine_id` to its C++ engine type and assert:
| declared | assertion |
|---|---|
| `input: 'none'` | output is *identical* driven silent vs driven with a noise bed (the engine ignores its input). For `output: 'none'` engines, compare `features()` instead of audio. |
| `input: 'required'` | output RMS with the noise bed exceeds output RMS with silence by > 1e-2. |
| `output: 'audio'` | with the mode's sounding precondition satisfied (a `note_on` when `trigger === 'notes'`, a noise bed when `input === 'required'`), RMS > 1e-3. |
| `output: 'events'` | RMS is exactly 0 under both drives **and** `pop_events()` yields > 0 events within 2 s at the default tempo. |
| `output: 'none'` | RMS is exactly 0 under both drives. |
| `trigger: 'notes'` | RMS is exactly 0 before `note_on`, and > 1e-3 within 1 s after `note_on(60, 100)`. |
| `trigger: 'free'` | covered by the `output: 'audio'` row. |
Drive params pseudo-randomly in [0.05, 0.95] from a fixed-seed `nisps::Rng`, **not** all-0.5 — that
vector is a degenerate corner and is precisely why `parity_check.cpp` stage 3 tests nothing
(`simplification-plan.md` §6.5f(3) made the same call for the bench).
The engine_id → C++ type resolution is a `switch` local to the test. Guard it: assert the set of
resolved ids equals the set of ids across all generated schemas, so a new engine cannot be added
without the test noticing.
**2b. `manifold/tests/engine-topology.test.ts`** (`bun test`, picked up by the existing
`bun test src tests/*.test.ts` glob — do not change that script). The §0 WASM probe, promoted:
drive the **committed** `manifold/public/nisps.{js,wasm}` through `nisps_engine_*` and assert the
same table at the browser's own ABI. This is the layer parity-check does not cover and the layer the
deploy actually ships.
**Verification:** `bash scripts/build-cpp-tests.sh` (5 ctest targets green); `cd manifold && bun run
test`. Both are hardware-free and run in CI. If a §2.2 value is wrong, this is where it surfaces.
### Step 3 — surface the truth in Manifold
- `manifold/src/console/model.ts`: `MFMode` gains `audio: AudioConfig` (derived in
`modeFromSchema`, `:246-259` — one line, straight off the schema).
- New `manifold/src/console/mode-support.ts`: `HostAudioCapabilities`, `modeSupport()`, and
`modeClassLabel()` (§2.1/§2.3). Pure, unit-testable, no React.
- `ConsoleApp` builds the capability record from what the engine actually exposes (today:
`audioInput` from `engine.audio.hasAudioInput`, the other three literal `false` constants that get
deleted as steps 4–6 land) and puts it on `ConsoleCtx`.
- Render: a support chip in the synth `ModeConfig` block (`manifold/src/console/Drawers.tsx:549-570`,
the "Transport" section — it already names the active mode), stating why the active mode cannot
sound. When 5c builds the picker it reads the same `modeSupport()` for its per-entry badge and for
disabling `hardware-only` entries.
**Verification:** unit test for `modeSupport()` over all nine modes × the current capability record
(`bun test`); `bun run typecheck`; e2e — extend `manifold/tests/e2e/schema-modes.spec.ts`, which
already switches modes through `window.__mf`, to assert the chip's text for one mode per class.
### Step 4 — wire microphone input (makes the Audio FX class real)
This is the cheapest real capability in the document: the worklet already consumes `inputs[0]`
(§0), so **no `nisps/` change, no new WASM export, no worklet change**.
- `EngineHost`: `setAudioInput(stream: MediaStream | null)` — hold a
`MediaStreamAudioSourceNode`, `connect(this.node)` / `disconnect()`; handle "called before
`start()`" by stashing the stream and connecting once the node exists; drop it in `stop()`.
- `EngineApi.audio` gains `setAudioInput()` + `hasAudioInput` (`manifold/src/engine/engine-api.ts:95-102,
203-207`).
- UI in the synth Transport block beside the play button, rendered only when the active mode
declares `input: 'required'`: an "Audio input" toggle that calls
`navigator.mediaDevices.getUserMedia({ audio: { echoCancellation: false, noiseSuppression: false,
autoGainControl: false } })` — the three defaults must be off or the codec fights the DSP — with
a headphones warning (invariant 4) and a clear denied/unavailable state. Requires a secure
context: production is HTTPS, dev is `http://localhost:5273`, both fine.
- Device picker: **out of scope**, default device only. Note it.
**Verification, honestly graded:**
- *Strong:* step 2b already proves these three engines transform a driven input correctly at the
committed-WASM layer.
- *Medium:* an e2e with `--use-fake-device-for-media-stream --use-fake-ui-for-media-stream` and
`context.grantPermissions(['microphone'])` can assert `engine.audio.hasAudioInput === true` and
that the mode chip clears. That proves **wiring**, not audibility.
- *Weak / not automatable here:* that sound actually comes out of the speakers with a real mic. The
worklet's output is not observable from the page. This needs an operator listening check, once,
per engine. Say so in the commit rather than implying the gates cover it.
### Step 5 — `paf_synth` note triggering *(gated on decision §8.1)*
Two exports, `nisps_engine_note_on(void*, int note, int vel)` and `nisps_engine_note_off(void*,
int note)`, dispatched with `if constexpr (requires { e->note_on(n, v); })` so only engines that
have them respond (today: `paf_synth` alone). The chain is **worklet-side**, which differs from the
main-thread five-layer chain — the main thread never creates engines:
1. `EMSCRIPTEN_KEEPALIVE` in `nisps/wasm/bindings.cpp` (beside `:1089-1108`)
2. `EXPORTED_FUNCTIONS` in `scripts/build-wasm.sh:70-71`
3. `WasmInstance.exports` interface + a `pickFn(...)` in `init_` and the re-export in the
`this.instance = { exports: { … } }` shim (`nisps-processor.ts:29-42`, `:145-165`, `:176-188`)
4. `HostToWorkletMessage` union + `onMessage_` branch (`engine-host.ts:35-53`,
`nisps-processor.ts:72-90`)
5. `EngineHost.noteOn/noteOff` → `EngineAudioApi` (`engine-api.ts:95-102`, `:203`)
Miss a layer and it fails at runtime, not compile time. `manifold/src/engine/types.ts:159-167` also
declares the engine ABI for the main-thread module; it is unused by `wasm-iml.ts` — decide whether
to keep it symmetric or leave it alone, but do not assume it is load-bearing.
RT safety: notes arrive on the worklet's existing `port`, handled between render quanta exactly as
params already are — bounded, no allocation inside `process()`.
Trigger sources: `manifold/src/inputs/midi-input-source.ts:204-211` **already** emits `note:<n>`
press/release actions with velocity from WebMIDI; `ConsoleApp` binds actions to verdicts today, so
routing note actions to `engine.audio.noteOn/Off` is a small addition there. Add one on-screen
momentary "Note" button in the Transport block as the no-hardware fallback.
Note for context: on firmware, `paf_synth` is *also* note-driven — `glue/midi_io.hpp:41-50` routes
MIDI note-in to `mode.note_on`. The browser gap is not a divergence from hardware behaviour, it is
the absence of the same route.
**Verification:** extend `tests/cpp/test_mode_audio_topology.cpp`'s `trigger: 'notes'` row (already
specified in step 2a — it will already be green before this step, since it tests the C++ engine, not
the browser); add a `manifold/tests/engine-topology.test.ts` case asserting the *exported* note
functions exist and produce non-zero output through the committed WASM; e2e asserting the chip
clears after a synthetic note. Audibility again needs an ear.
### Step 6 — event-only and analysis modes *(gated on decisions §8.2 / §8.3)*
**Default position: label them `hardware-only` and stop.** That is honest, costs nothing, and the
label disappears by itself when a capability lands (§2.3).
If instead they are to be made real:
- **Event-only (`breakor`, `elysiamorf`)** needs (a) `nisps_engine_pop_events`,
`nisps_engine_set_playing`, `nisps_engine_update_bpm` across the step-5 chain; (b) a worklet→main
event channel — a `SharedArrayBuffer` ring written by the worklet and drained on the main thread's
rAF (invariant 5; COOP/COEP are already set); (c) transport UI (play/stop/bpm) and a route from
events to `midi-backend.ts` that bypasses the per-param output path entirely, because these modes
emit notes, not parameter values. **This overlaps 5a**: on firmware the mode layer owns the event
drain (`nisps/modes/breakor.hpp:42-46`, `glue/midi_io.hpp:53-63`), so building a second,
browser-only orchestration here is exactly the duplication 5a exists to remove. **Sequence it
after 5a.**
- **Analysis (`sound_analysis_midi`)** needs step 4 *plus* `nisps_engine_copy_features` *plus* a
worklet→main path for six floats per control tick (same SAB channel) *plus* routing those into ML
input slots 0–5 alongside `joy_*` in slots 6–9 — which is `input-layer.ts`'s composition contract
and touches the exclusive-source picker (invariant 6, and the dormant multi-source decision,
§7.7 of the plan). Not a small item; do not smuggle it into 5b.
---
## §6 Proposed deletion
One, with its consumer named:
**`modeEngineId()`'s hard-coded exception** — `manifold/src/console/model.ts:392-395`:
```ts
export function modeEngineId(modeId: string): string {
if (modeId === 'sound_analysis_midi') return 'analysis';
return MF_MODES.find((m) => m.id === modeId)?.engineId ?? 'thru';
}
```
Consumer: `ConsoleApp.tsx:242` (`engine.audio.setBackend(...)`). Replaced by
`schema.audio.dsp_engine_id ?? schema.engine_id` (§3.1). This is the same class of hand-exception
Phase 3 spent two days removing, and it is the only place in the app where a mode's audio engine is
decided by an `if`.
Nothing else in this document deletes anything. In particular, do **not** delete `nisps/engines/`
sequencer or analysis code because the browser cannot use it — firmware runs all of it.
---
## §7 Adjacent, verified, deliberately out of scope
Two findings from the same files. Recording them so the implementer does not rediscover them
mid-change; neither is part of 5b.
1. **`SCHEMA_MODE_OVERLAYS.input` duplicates `ui.primary_input` and has already drifted.**
`model.ts:229-244` hand-declares `breakor: { input: 'joystick' }` while
`schemas/modes/breakor.json` declares `"primary_input": "xy_pad"`. The overlay value is live —
`ConsoleApp.tsx:814` feeds it to `resolveInputMap()` (`settings/settings-store.ts:146-153`) to
pick the input-map shape — so the two sources disagree about a rendered behaviour. Fixing it
means deriving `MFMode.input` from the schema and deleting the overlay field.
2. **`ui` is generated into both languages and consumed by nothing.** `primary_input`,
`show_voice_space_selector`, `show_synth_visualizer` have zero readers outside `codegen/` and its
golden. Either wire them (item 1 above wires `primary_input`) or delete them. Doing this in the
same commit as step 1 is cheaper than separately — same files, same regeneration — but it is a
different decision and should be asked for explicitly.
---
## §8 Operator decisions
These genuinely need a call; everything else in this document I decided and justified above.
1. **`paf_synth` note triggering (step 5): build it, or label the flagship mode "needs a note
source"?** This is the most user-visible silence in the catalogue and the most likely to be read
as "the app is broken". Building it is two exports plus a UI button, and the MIDI note plumbing
already exists. Labelling it is free but leaves the best-developed engine unusable in the browser.
*Recommendation: build it.* Counter-argument worth weighing: note routing is arguably 5a's
territory (firmware does it in `ModeBase`/`midi_io.hpp`), so building it here creates a small
browser-only path that 5a will later absorb.
2. **Event-only modes (`breakor`, `elysiamorf`): label hardware-only, or build the event channel?**
*Recommendation: label now, build after 5a* — the browser-side orchestration is precisely what 5a
is meant to stop hand-writing.
3. **`sound_analysis_midi`: label hardware-only, or build the analysis path?** It needs mic +
feature readback + ML-input composition, and it collides with the dormant multi-source input
decision (plan §7.7). *Recommendation: label now.*
4. **`audio.dsp_engine_id` — adopt the field (and delete the `modeEngineId` exception), or keep the
exception and give the test a local special case?** Adopting it is the single-source answer and
is what §3.1/§6 assume; the cost is one more optional schema field used by exactly one mode.
5. **Bundle the §7 `ui` clean-up into step 1?** Same files and same regeneration, so bundling is
cheaper — but it widens a change that is otherwise purely additive.
6. **How much microphone UX?** Step 4 ships default-device-only with no picker and no input meter.
A device picker and a level meter are obvious follow-ons; say now whether they are wanted, since
a meter needs a tap on the input node that the current host does not have.
---
## §9 What this plan does not do
- It does not build the instrument-mode picker. That is §6.5c (`plans/curated-presets-spec.md`);
this plan gives it truthful data, and 5c should not ship a picker without it.
- It does not touch `nisps/modes/` orchestration. That is §6.5a
(`plans/mode-layer-reunification.md`), which is also where the browser event pump belongs — see
§5 step 6 and decision §8.2.
- It does not change firmware behaviour in any step. Every mode remains fully functional on hardware.
- It does not improve the parity harness, despite §1.3. Fixing stage 3 to trigger a note before
averaging is a one-line change with a golden-baseline consequence; it belongs with whoever next
touches `tests/cpp/parity_check.cpp`, and would bump `kVersion` (`:93`).

View file

@ -0,0 +1,780 @@
---
kind: plan
status: active
---
# Curated Presets, the Instrument Library, and Per-Surface Disclosure
*Dated 2026-07-21. The spec for `simplification-plan.md` §6.5c ("5c Curated/advanced split",
audit findings A3 + A7, `ALIGNMENT.md` defect 2). **Proposal only — no code is authorised until the
operator adopts it.** Every code claim below was read out of the tree on 2026-07-21 and is cited
with `file:line`; where a repo document disagrees with the code, the code is recorded as the truth
and the disagreement is called out in §1.6. British spelling in product copy. The built-in synth is
"Powerful Synth Engine" / "Built-in Synth" — the string "C15" must never reach the UI.*
**Reading order for an implementer:** §1 (ground truth) → §2 (product model) → §3–§6 (design) →
§9 (sequenced plan). §7 is the invariant list to check work against; §11 is what the operator still
has to decide.
---
## 0. Scope, and what this supersedes
**In scope** (one item, three limbs that only make sense together):
1. A **preset data model** — what a curated preset is as data, where it is stored, how it is
validated against schema truth, and how it is applied.
2. The **Instrument Library** — the missing in-UI instrument picker, built as a preset browser
(choosing a preset is how you choose an instrument).
3. **Progressive disclosure** via per-surface depth *levels*, replacing the accidental use of the
drawer geometry toggle as the advanced flag.
**Out of scope, deliberately:**
- Browser mode-coverage honesty / audio topology (`simplification-plan.md` §6.5b, spec'd in
`plans/browser-mode-coverage-spec.md`). The Library renders whatever signal 5b produces; it does
not invent one. **That spec's §0 calls itself a prerequisite of this item, and it is right** — the
picker is what makes the latent dishonesty visible. Sequencing recommendation: land 5b before §9
step 6. If it has not, the Library lists every catalogue entry with no viability claim (§5.4) —
honest by omission rather than wrong, but a worse first impression.
- The mode-layer reunification (§6.5a) and the hardware editor (§6.5d). Presets are browser-local
data in this item; §11.Q4 records the promotion path if they later have to reach firmware.
- Region/param pins and the "control surface" (boldness/memory/precision) of
`manifold-parity-features-spec.md` §2 and §5. Neither is implemented, neither is needed here.
**Supersedes** `plans/manifold-parity-features-spec.md` §1 ("Session presets (composed layers)").
That section has **no code consumer** — nothing under `manifold/src/` implements or references it
(verified: no `SessionPresetV1`, no `mf-session` key, no `persist<T>` helper in the tree). Its
composed-layer idea survives here in reduced form; three of its specifics do not, and the
divergences are explicit:
| §1 of manifold-parity-features-spec | Disposition here |
|---|---|
| `layers.weights?: string` (base64url Float32Array, opt-in) | **Deleted.** Contradicts the operator's §7.6 decision that a curated preset is configuration only, network untrained (§2.1). |
| `PipelineLayer` = params of `engine/input-pipeline.ts` / `output-pipeline.ts` | **Deleted as written** — both files were deleted at one-core-engine P4; the config now lives in `manifold/src/engine/pipeline-types.ts` and has **zero UI writers** (§1.5). Nothing can author it, so a preset cannot meaningfully carry it yet. |
| `ControlLayer` = `{ boldness, memory, precision, presetId, offsets }` | **Deferred** — the control surface (§5 of that spec) is unimplemented and out of scope. |
| URL sharing (`?session=`, `?boldness=`) | **Deferred** to §11.Q5; the model is designed not to preclude it. |
On adoption, `manifold-parity-features-spec.md`'s status block gains a line pointing §1 here. That
edit belongs to whoever lands step 3 of §9, not to this document.
---
## 1. Ground truth (read out of the tree, 2026-07-21)
### 1.1 There is no instrument picker, and the active instrument is invisible
- The instrument mode is a plain `useState` in the console spine:
`manifold/src/console/ConsoleApp.tsx:99` — `const [modeId, setModeId] = useState('paf_synth')`.
- `ConsoleCtx` carries `modes: MFMode[]` and `setModeId` (`manifold/src/console/types.ts:64-66`),
populated at `ConsoleApp.tsx:735-737`. **Neither has a single reader.** `grep -rn "ctx.modes|ctx.setModeId" manifold/src`
returns only the comment at `ConsoleApp.tsx:326` that says so.
- The only place the active instrument's name is rendered anywhere in the app is one sentence of
explanatory prose — `Drawers.tsx:566`, `The active engine follows the selected mode ({ctx.mode.label})` —
which is itself gated behind `depth === 'expanded'` *and* the synth output target. In the default
session (particles target, condensed drawers) the user cannot discover which instrument is loaded.
- `MFMode.glyph`, `MFMode.cls`, `MFMode.placeholder`, `MFMode.badge`
(`manifold/src/console/model.ts:136-160`) have **zero consumers**. They are display fields
maintained for a renderer that does not exist. The Library is their first consumer, or they should
be deleted; see §8.4.
- Mode switching today happens only through the `?debug=1` seam
(`ConsoleApp.tsx:331-345`, `window.__mf.setMode/getModeId/paramCount/modeIds`), which
`manifold/tests/e2e/schema-modes.spec.ts` drives. **That seam must survive** (§7.8).
### 1.2 Mode switching itself works, and reshapes the net
Three effects fire on a `modeId` change, in declaration order:
| Effect | Lines | Does |
|---|---|---|
| audio backend | `ConsoleApp.tsx:241-243` | `engine.audio.setBackend(modeEngineId(modeId))` |
| net reshape | `ConsoleApp.tsx:252-257` | `engine.reshape(mode.ml)` — schema dims, warm-started |
| transient reset | `ConsoleApp.tsx:260-271` | rebuilds `params` from the mode, recentres input, clears examples/pins/markers, closes drawers |
`engine.reshape` (`engine-api.ts:323` → `wasm-iml.ts:355` → `nisps_ml_reshape`,
`nisps/wasm/bindings.cpp:397`) constructs a fresh `MLPCore` at the new dims and swaps it in.
Because the dataset lives **inside** `MLPCore` (`nisps/ml/mlp.hpp:143`, `:441`, `:443`), a reshape
destroys every stored example; `wasm-iml.ts:431` clears the TS mirror to match. This is the single
most important fact for preset application ordering — see §4.2.
### 1.3 The only preset machinery is per-backend output routing
`manifold/src/backends/presets.ts` (162 lines) is real, consumed, and narrow:
- `OutputPreset` = `{ name, backend, rows: OutputPresetRow[], settings?, savedAt }` (`:16-37`).
- Storage is localStorage, **namespaced per backend**: `manifold-output-presets:<backend>` (`:38-41`).
- `OutputPresetRow` (`:16-29`) captures `name, status, muted, armed, min, max, curve, val, midi, osc, vcv`.
- The sole UI is `PresetBar` in `manifold/src/dock/OutputsBackendConfig.tsx:99-208`, rendered from
`:686`, itself gated behind `depth === 'expanded'` in `Drawers.tsx:634`.
- `PresetBar.backendSettings()` (`OutputsBackendConfig.tsx:113-118`) persists MIDI port/ccCount, OSC
url/sendRaw, VCV url/sendRaw — and `{}` for everything else.
**The `cv` bug is confirmed and is worse than "a missing field".** `MFParam.cv` (`model.ts:133`,
`{ channel, gateThreshold }`) is live state: `useBackendManager.ts:161` reads `p.cv` per output to
build the uSEQ channel map, and `CvConfig` (`OutputsBackendConfig.tsx:590`) edits it. But
`OutputPresetRow` has no `cv` (`presets.ts:16-29`), `rowsFromParams` does not copy it (`:72-86`),
and `applyPreset` does not restore it (`:141-162`). Saving a preset for the `cvgate` backend
therefore silently discards **the entire channel assignment**, which is the only thing a CV preset
is for. Fix = three one-line additions (§9 step 1), exactly as the audit said.
### 1.4 Two different things are called "Mode", and one backend is unreachable
- **Instrument mode**: `MF_MODES` / `modeId`, schema-backed (`model.ts:316`).
- **Output mode**: `OutputMode` (`types.ts:26`) / `OUTPUT_MODES` (`console/output-mode.ts:33-73`),
which is what the dock's top button is labelled — `Dock.tsx:126` renders the header
`Mode · output target`.
Adding an instrument picker beside a control already called "Mode" is a naming collision the design
has to resolve (§2.2).
Also observed, and not recorded anywhere: **`BackendId` includes `'vcv'`
(`dock/output-state.ts:31`) but no `OutputModeDescriptor` maps to it** (`output-mode.ts:33-73`
lists particles/midi/osc/cv/synth/editor). `outputBackend` is derived solely from the selected
output mode (`ConsoleApp.tsx:137`), so the VCV backend — `vcv-backend.ts`, 240 lines, registered at
`backends/manager.ts:68`, with its own config panel at `OutputsBackendConfig.tsx:691` — **cannot be
selected from the UI at all.** That is out of this item's scope to fix, but the Library work will
put the output-target list in front of the operator, so it should be reported rather than quietly
inherited.
### 1.5 What state a preset could capture, and what nothing can author
| Candidate state | Where it lives | Authorable in the UI today? |
|---|---|---|
| instrument mode id | `ConsoleApp.tsx:99` | only via `?debug=1` |
| output target | `ConsoleApp.tsx:136` | yes (dock top button) |
| per-output routing (`MFParam` status/muted/armed/min/max/curve/val + midi/osc/vcv/cv) | `ConsoleApp` `params` store; `OutputControlRow` | yes |
| backend settings (MIDI port + CC count, OSC/VCV url + sendRaw) | `ConsoleApp` state, `ConsoleCtx` | yes |
| input source + gamepad stick mode + MIDI learn bindings | `inputs/useInputLayer.ts:113-120` | yes |
| feedback mode, solo mode, explore intensity, noise cap, Xavier spread | `ConsoleApp.tsx:104-133` | yes |
| **input/output pipeline config** (zoom, deadzone, curve, smoothing, slew, freeze) | `engine/pipeline-types.ts` | **no** — `grep` for `setInputConfig|setOutputConfig|defaultInputConfig` outside `src/engine/` finds only `tests/`. There is no UI writer. |
| **voice space** | `nisps/modes/base.hpp:279-291` | **no** — there is no `nisps_*_voice_space` export in `nisps/wasm/bindings.cpp`, and the worklet reaches the audio engine through raw WASM exports (`worklet/nisps-processor.ts:33-36`), so exposing it means walking the full export chain twice. See §11.Q3. |
| network weights | `wasm-iml.ts` | yes (`randomise`, training) — **deliberately excluded**, §2.1 |
Schema fields `ui.show_voice_space_selector` and `ui.show_synth_visualizer` exist in every mode
schema (e.g. `schemas/modes/paf_synth.json`) and reach TS as `UIConfig`
(`manifold/src/modes/generated/types.ts:52-56`) but have **no consumer on either target**.
### 1.6 Where documents disagree with the code
Recorded here so the next session does not re-derive them. None of these are this item's job to fix
except where §8/§9 says so.
1. **`ONBOARDING.md` §4** says `OUTPUT_MODES` = "particles (default) / midi / osc / synth / editor" —
five. The code has six; `cv` (uSEQ CV/gate) is missing from the doc. `output-mode.ts:32`'s own
comment ("The five Modes") is wrong the same way, one line above a six-element array.
2. **`model.ts` overlay vs schema — a live dual truth with real drift.** `ModeOverlay.input`
(`model.ts:211-216`) is hand-written per mode (`:229-247`) while every schema already declares
`ui.primary_input` (`manifold/src/modes/generated/types.ts:52-56`). They agree for eight of nine
modes and **disagree for `breakor`**: the schema
says `xy_pad`, the overlay says `joystick` (`model.ts:235`). The overlay wins today, and it is
the input to `resolveInputMap(settings.inputMap, mode.input)` (`ConsoleApp.tsx:814`), so Breakor
renders a circular joystick surface where its schema asks for a rectangular pad. Single consumer,
five-line fix (§9 step 2).
3. **`dock-spec.md` §0** prescribes three depths (`peek | expand | FULL`) as **per-drawer** state.
Two shipped, and they are **one global state** shared by all five drawers
(`ConsoleApp.tsx:110`), reset to `condensed` on every drawer switch (`Dock.tsx:204`) and every
mode switch (`ConsoleApp.tsx:268`). Its own 2026-07-21 grounding note already records the
two-depth divergence; the *global-not-per-drawer* part is not recorded anywhere.
4. **`manifold-parity-features-spec.md` §1.1** describes a `PipelineLayer` over
`engine/input-pipeline.ts` and `output-pipeline.ts`. Both files were deleted at P4.
5. `simplification-plan.md` §7.6 still lists the curated-preset product model as an *open* operator
decision. The operator has since decided it (§2.1). Reconciling the plan document is the
orchestrator's, not this spec's.
---
## 2. The product model
### 2.1 What a curated preset is (operator decision, §7.6)
> **A curated preset is configuration only: mode + parameters + routing, with the network
> UNTRAINED — though it may sometimes ship seed training data.**
Consequences taken as binding:
- **A preset never carries weights.** Not optionally, not opt-in. The type has no weights field
(§3.1) and the apply routine never calls `setWeights`. This is testable — see §7.1.
- **A preset may carry seed examples**: `(input vector, output vector)` pairs at the mode's schema
arity. Loading them puts them in the dataset; it does **not** train (§4.4).
- Therefore applying a preset is *deterministic and cheap*: it reshapes the net to the mode's schema
dims (which mode switching already does) and writes UI/routing state. The sound you get is the
untrained net through the preset's routing — the preset defines the *instrument and the mapping
space*, and the user teaches it.
### 2.2 Naming: Instrument vs Output
To end the collision in §1.4, product copy uses two words and never the word "Mode":
| Concept | UI label | Code |
|---|---|---|
| which instrument/mode is loaded | **Instrument** | `modeId`, `MF_MODES` (unchanged — `mode_id` is schema truth) |
| where the outputs go | **Output** | `OutputMode` → renamed `OutputTarget` (§8.3) |
`Dock.tsx:126`'s `Mode · output target` header becomes `Output`. This is copy plus a mechanical
type rename; it changes no behaviour and is typecheck-verified.
### 2.3 The Library is the picker
There is no separate "instrument picker" and "preset browser". One surface — the **Library** —
lists curated presets grouped by instrument; choosing one selects its instrument. Below the presets,
at the advanced level, sits the bare instrument list (load an instrument with no preset). This is
one surface instead of two, and it makes the curated path the default path structurally rather than
by exhortation, which is what vision bullet 3 asks for.
---
## 3. Data model
### 3.1 Types
New module `manifold/src/presets/types.ts` — types + pure validation only, no React, no storage:
```ts
import type { ParamStatus } from '../console/model';
import type { OutputTarget } from '../console/types';
import type { CvSpec, MidiCcSpec, OscSpec, VcvSpec } from '../dock/output-state';
/** One output's deviation from the mode default. `i` indexes the mode's schema params. */
export interface OutputOverride {
i: number;
/** Display name override (MIDI/OSC outputs get renamed by users). */
label?: string;
status?: ParamStatus;
muted?: boolean;
armed?: boolean;
min?: number;
max?: number;
curve?: number;
/** Held value; only meaningful when status === 'fixed'. */
val?: number;
midi?: MidiCcSpec;
osc?: OscSpec;
vcv?: VcvSpec;
cv?: CvSpec;
}
export interface PresetRouting {
/** Status for every output NOT named in `outputs`. Omit to leave mode defaults alone. */
rest?: ParamStatus;
outputs: OutputOverride[];
}
export interface PresetInputs {
mode: 'internal' | 'gamepad' | 'midi';
gamepadStickMode?: 'single' | 'double';
}
export interface PresetLearning {
feedbackMode?: 'explore-and-place' | 'geometric-dislike';
exploreIntensity?: number;
noiseCap?: number;
spread?: boolean;
}
/** Seed examples, at the mode's schema arity. Validated in `validatePreset`. */
export interface PresetSeed {
inputs: number[][]; // each length === schema.ml.input_size
outputs: number[][]; // each length === schema.ml.output_size
}
export interface Preset {
v: 1;
id: string; // slug, e.g. 'paf-synth/glass-formants'
name: string;
/** Optional one-line description shown in the Library. */
blurb?: string;
modeId: string; // MUST be an MF_MODES id
target: OutputTarget;
routing?: PresetRouting;
backendSettings?: Record<string, unknown>; // same blob shape PresetBar already writes
inputs?: PresetInputs;
learning?: PresetLearning;
seed?: PresetSeed;
origin: 'builtin' | 'user';
savedAt?: number; // user presets only
}
```
**There is no `weights` field, by design.** A reviewer should treat any future addition of one as a
reversal of §2.1 requiring the operator.
Rationale for the choices that are not obvious:
- **`OutputOverride` carries its own index and layers are optional**, so a hand-authored curated
preset for a 56-output mode is a dozen lines, not 56 rows of boilerplate. Index (not name) is
canonical because `MFParam.name` is user-editable for MIDI/OSC (`presets.ts:147-149`) while the schema
param order is stable codegen output.
- **`rest`** lets a curated preset say "these six outputs are live; everything else is off" in one
field — which is what curation mostly *is*.
- **Backend settings stay an untyped blob**, matching `OutputPreset.settings`
(`presets.ts:35`) and the writer/reader pair at `OutputsBackendConfig.tsx:113-131`. Typing it
properly is a separate, larger job with no consumer asking for it.
- **The layer types are `Preset`-prefixed** because `InputLayer` is already taken: it is the input
composition hub class in `manifold/src/inputs/input-layer.ts`. Do not reuse the name.
### 3.2 Storage
- **Built-in (curated) presets**: TypeScript data modules, `manifold/src/presets/builtin/<mode_id>.ts`,
re-exported from `builtin/index.ts` as `BUILTIN_PRESETS: readonly Preset[]`. Typed, typechecked,
tree-shaken, no build step, no codegen.
*Why not `schemas/` + codegen?* Codegen exists to keep firmware and browser honest about a shared
contract. Presets have no firmware consumer today (firmware modes are compile-time
`-DMEMLNAUT_MODE_TYPE` selections with no on-device persistence — `ALIGNMENT.md` defect 3).
Adding a C++ emitter for data nothing in C++ reads would be complexity without a requirement.
**Removal condition:** when §6.5d (hardware editor) gives firmware a command surface and
persistence, promote presets to `schemas/presets/*.json` + codegen with both outputs in the same
change, per the standing schema rule.
- **User presets**: one localStorage document, key `mf-presets`, shape
`{ v: 1, presets: Preset[], lastAppliedId?: string }`, written through a store modelled on
`settings/settings-store.ts` (framework-neutral class + `useSyncExternalStore` hook + module-level
singleton). No new persistence mechanism.
- The per-backend namespaces `manifold-output-presets:<backend>` are **retired** (§8.1). See §11.Q1
— deleting them discards any presets the operator saved in the live app.
### 3.3 Validation against schema truth
`validatePreset(p, schemas): string[]` in `presets/types.ts`, pure, returns human-readable problems:
- `modeId` resolves in `MF_MODES`.
- every `OutputOverride.i` is `< schema.params.length`.
- `min <= max`, all of `min/max/curve/val` within `[0,1]` (the routing-knob space — `MFParam`'s
`min/max/curve/val` are 0..1 routing semantics; schema engine units live in the `schema*` fields,
`model.ts:107-121`).
- `seed.inputs[k].length === schema.ml.input_size`, `seed.outputs[k].length === schema.ml.output_size`,
`inputs.length === outputs.length`, and `inputs.length <= 128` — the shared dataset capacity
(`nisps::ml::kDefaultMaxExamples`, `nisps/ml/storage.hpp:57`, mirrored at `wasm-iml.ts:52` and
reported through `nisps_ml_describe`, `bindings.cpp:926`).
- `target` resolves in `OUTPUT_TARGETS`.
A unit test runs `validatePreset` over every entry of `BUILTIN_PRESETS` (§9 step 4). This is the
mechanism that stops curated data rotting when a schema changes — the failure is a red `bun test`,
not a silent misbehaviour.
---
## 4. Applying a preset
### 4.1 One imperative entry point, not an effect chain
`manifold/src/presets/apply.ts` exports:
```ts
export interface PresetApplier {
engine: EngineApi;
selectInstrument(modeId: string): void; // reshape + rebuild params + reset transients
setParams(next: MFParam[]): void;
setTarget(t: OutputTarget): void;
setBackendSettings(s: Record<string, unknown> | undefined): void;
setInputs(l: PresetInputs | undefined): void;
setLearning(l: PresetLearning | undefined): void;
}
export function applyPreset(a: PresetApplier, p: Preset): void;
```
`ConsoleApp` provides the applier; the sequencing logic lives in `apply.ts` where it can be unit
tested against a fake applier. `ConsoleApp.tsx` is already 1082 lines — this must not add to it.
### 4.2 The ordering invariant (the thing that will silently break)
`engine.reshape` destroys the dataset (§1.2). Today the reshape happens in an **effect**
(`ConsoleApp.tsx:252-257`), which runs *after* the event handler that changed `modeId`. So the naive
implementation — "handler sets modeId, then adds seed examples" — loads the seeds and then has them
wiped by the effect one tick later, silently, with a dataset counter that briefly reads right.
**Fix, and it is a simplification rather than a workaround:** collapse the three mode-switch effects
(`ConsoleApp.tsx:241-243`, `:252-257`, `:260-271`) into one imperative `selectInstrument(modeId)`
function that does, in order: `engine.audio.setBackend` → `engine.reshape` → rebuild `params` from
the mode → reset transients → `setModeId`. Keep exactly one effect, keyed on `[engine]`, that calls
`selectInstrument(modeId)` once when WASM becomes ready (the boot path the current effects rely on).
`applyPreset` then runs entirely inside one handler, in a guaranteed order, with no implicit
cross-effect contract.
`window.__mf.setMode` (`ConsoleApp.tsx:336`) repoints to `selectInstrument`.
**This is a refactor of live behaviour and must be verified as one** — see §9 step 3.
### 4.3 Apply order
1. `selectInstrument(p.modeId)` — reshape lands here, dataset is now empty.
2. `setTarget(p.target)`; `setBackendSettings(p.backendSettings)`.
3. Build the params array: start from the mode's schema defaults (what step 1 produced), apply
`routing.rest` to every output, then apply each `OutputOverride` by index. One `setParams`.
4. `setInputs(p.inputs)`, `setLearning(p.learning)`.
5. If `p.seed`: `engine.addExample(inputs[k], outputs[k])` for each pair. **No training.**
6. Record `lastAppliedId`.
Steps 3–5 must not be reordered: 5 after 1 (else wiped), 3 after 1 (else overwritten by the mode
rebuild).
### 4.4 Seeds are loaded, not trained
After apply, the Learning drawer shows `N examples` (its existing chip, `Drawers.tsx:219`, fed
by `ctx.datasetCount`), and the user trains or gives verdicts. Nothing auto-trains, because §2.1
says the network ships untrained and auto-training would make that false. §11.Q2 records the
alternative if the operator wants a preset to be able to opt into one training pass.
---
## 5. The Library (the picker)
### 5.1 Placement
A second pinned control at the top of the dock rail (`Dock.tsx:405`), **above** the existing output
selector, rendering the active instrument's glyph and opening a popover — the same interaction and
the same visual language as `ModeSelector` (`Dock.tsx:64-174`), which becomes `OutputSelector`. Two
labelled controls, top of rail: **Instrument**, **Output**.
Rejected alternatives, briefly: a sixth drawer (a preset browser is a chooser, not a workbench, and
drawers are mutually exclusive with the surface you are auditioning against); a stage-header title
bar (the stages are full-bleed by design and CompositeStage owns its own chrome).
### 5.2 Contents
At **basic** level:
- Curated presets, grouped by instrument, each row: preset name, instrument label + glyph, `blurb`.
- The active preset marked; applying another switches instrument as a side effect.
- A "seeds" chip on presets that carry `seed`, showing the example count — so the difference between
"config only" and "config + seeds" is visible before you commit to it, not after.
At **advanced** level, additionally:
- **Instruments** — the bare `MF_MODES` list (`ctx.modes`, finally consumed), each entry loading the
instrument with no preset. This is the authoring entry point: load a bare instrument, configure,
save as a user preset.
- **Save current as preset** / rename / delete for user presets, reusing `PresetBar`'s interaction
pattern (`OutputsBackendConfig.tsx:99-208`) rather than inventing a second one.
### 5.3 Applying is not silent
Applying a preset switches instrument, which resets the dataset and the net. Today that is already
true of a mode switch and is done without confirmation ("switching instrument is already a
deliberate act", `ConsoleApp.tsx:250`). Keep that: **no modal** — but the Library rows carry a
persistent one-line warning ("loading resets examples and the network") and, when
`ctx.datasetCount > 0`, the row click routes through the existing `ReshapeModal` pattern
(`console/ReshapeModal.tsx`) with preset-specific copy. Losing 40 hand-taught examples to a
mis-click is the failure this prevents; losing an untouched default session is not worth a modal.
The condition is deliberately `datasetCount` alone — "unsaved routing edits" would need a
dirty-tracking mechanism the app does not have and this item does not justify building.
### 5.4 Viability signalling
The Library renders `MFMode.badge` when present (`model.ts:158`) and disables entries with
`placeholder: true` (`:157`) — both fields exist and neither has a consumer today (§1.1). If §8.5
lands, `c15` is the only `placeholder: true` entry and the disable path becomes
dormant-but-correct; keep it, it costs one line.
The Library does **not** invent a viability model. `plans/browser-mode-coverage-spec.md` §2.3
proposes exactly the right shape — a pure `modeSupport(schema.audio, host)` returning
`{ ok: true } | { ok: false; reason; detail }`, computed from declared per-mode audio topology and
declared host capabilities rather than a hand-maintained flag — and the Library should render that
result verbatim: a chip carrying `detail`, the row disabled when `ok` is false. If 5b has not
landed, entries render unmarked. **Do not add a topology map to `model.ts` in the meantime** — that
is precisely the seventh hand-maintained mode registry Phase 3 spent two days eliminating, and 5b
rejects it by name.
### 5.5 Boot
Boot applies `lastAppliedId` if it resolves, else `DEFAULT_PRESET_ID`.
**`DEFAULT_PRESET_ID` must be a `paf_synth` preset.** The e2e suite clears localStorage before every
run (`tests/e2e/helpers.ts:31-41`), so boot always takes the default path, and `ONBOARDING.md` §5
plus several tests pin the boot shape at `4→[10,10,14]→33`, weights 809. Choosing a default on any
other instrument silently changes the boot contract several tests assert. If a different default is
wanted, the boot-shape claims in `ONBOARDING.md` and the affected specs move in the same change.
---
## 6. Progressive disclosure: per-surface levels
### 6.1 What `depth` actually is today
`DrawerDepth = 'condensed' | 'expanded'` (`types.ts:38`) is **one global state**
(`ConsoleApp.tsx:110`) that simultaneously controls panel geometry (360px side panel vs 80vw×80vh
centred modal, `Dock.tsx:265-299`) and content disclosure. §6.5e adopted `expanded` as "the
advanced-surface flag" for the training-health panel, and `ONBOARDING.md` now documents that as the
rule.
It is not a rule; it is an accident. Classifying every gate in `Drawers.tsx`:
| Site | Content | What the gate is really for |
|---|---|---|
| `:171` | feedback-mode explanation | **prose** |
| `:208` | explore/Jolt explanation | **prose** |
| `:229` | solo/arm scope, solo behaviour, noise-cap slider | **advanced** |
| `:254` | feedback lab, Xavier switch, `<TrainingHealth/>` | **advanced** |
| `:396` | active input source's status message | **status — should never have been hidden** |
| `:403` | internal-pad explanation | **prose** |
| `:410` | gamepad stick mode + button legend | **function** — you cannot pick one/two sticks without opening a modal |
| `:441` | MIDI device picker + MIDI-Learn | **function** — same |
| `:523` | dedicated-dimensions explanation | **prose** |
| `:564`, `:583`, `:592`, `:598` | per-target explanations | **prose** |
| `:620-621` | rows sliced to 6 when condensed | **geometry** |
| `:634` | `<OutputsBackendConfig/>` (CC table, CV channel map, preset bar) | **function + geometry** |
| `:641` | row list max-height 460 vs 220 | **geometry** |
| `:654` | `showCurve` on each row | **geometry** |
| `:659` | "+N more — expand to edit" | **geometry** |
| `:691` | unfocused icon colour + preview | **advanced (cosmetic detail)** |
| `:720`, `:737` | settings explanations | **prose** |
Three of twenty-one gates are genuinely about expertise. The rest are geometry, prose, or — in three
cases — real functionality hidden by a size switch. Using this as the advanced flag means the
advanced surface and the big-panel surface can never diverge, and it means beginners' explanatory
prose is hidden from beginners, which is exactly backwards.
### 6.2 The split
Two orthogonal axes, one existing mechanism each:
- **`presentation: 'panel' | 'modal'`** — today's geometry toggle, renamed. Per-drawer, transient,
driven by the ⤢ tab (`Dock.tsx:302-328`) and `\` (`ConsoleApp.tsx:642`). Behaviour unchanged.
- **`level: 'basic' | 'advanced'`** — the disclosure axis. **Per surface, persisted** in the existing
settings document (`settings-store.ts`, key `mf-settings`) as
`levels: Record<DrawerKey | 'library', DrawerLevel>`. No new store, no new persistence pattern,
no global boolean. The Library is a surface with a level for the same reason the drawers are, so
§5.2's advanced tier needs no second mechanism.
Two levels, not `dock-spec.md` §0's three: there is exactly one advanced tier of content to put
anywhere, and inventing a third empty level is the kind of speculative structure this repo has been
deleting all week. `dock-spec.md` §0's `peek` maps onto nothing that shipped.
Control: a small `basic | advanced` segmented toggle in each drawer's header
(`Dock.tsx:355-368`), beside the existing depth label at `:364` — which currently renders the
literal string `condensed`/`expanded` and becomes the presentation state's label.
### 6.3 Re-gating, item by item
Applying §6.1's classification:
- **advanced** (`:229`, `:254`, `:691`): gate on `level === 'advanced'`. `<TrainingHealth/>` keeps
its advanced gating — the §6.5e decision survives intact, only its mechanism changes.
- **function** (`:410`, `:441`, `:634`): ungate. Configuring your gamepad or your CV channel map is
not advanced and must not require a modal. They stay in whatever presentation the drawer is in.
- **status** (`:396`): ungate.
- **geometry** (`:620-621`, `:641`, `:654`, `:659`): gate on `presentation === 'modal'`. Unchanged
behaviour, renamed condition.
- **prose** (`:171`, `:208`, `:403`, `:523`, `:564`, `:583`, `:592`, `:598`, `:720`, `:737`):
gate on `level === 'basic'` — *shown* to beginners, *hidden* from the advanced surface, which is
the inversion of today. This is a judgement call, not a derivation; §12.3 records it as mine and
the cheap fallback (always show) if the operator disagrees.
### 6.4 Level does not reset
`depth` is reset to `condensed` on every drawer switch (`Dock.tsx:204`) and every mode switch
(`ConsoleApp.tsx:268`). `presentation` keeps that behaviour. `level` **must not** reset — a persisted
stance that resets when you change instrument is not a stance. This is the concrete reason level
lives in `settings-store` and not in `ConsoleApp` state.
---
## 7. Invariants
The repo-wide hard constraints (platform-neutral allocation-free `nisps/`, ≤1e-5 native↔WASM parity,
schema changes shipping both codegen outputs, bounded RT-safe worklet comms, dual-core ownership)
are the floor. This item's own:
1. **A preset never changes network weights.** Concretely: from a fixed starting state,
`applyPreset(p)` and `selectInstrument(p.modeId)` must leave `getWeights()` **bit-identical**.
(Weights *do* change across the reshape — `nisps_ml_reshape` redraws and warm-starts,
`bindings.cpp:397-418` — but the preset must contribute nothing beyond it.) This is the machine
check on §2.1 and belongs in `bun test`.
2. **Seeds load after the reshape or not at all** (§4.2). A preset whose examples vanish is worse
than one with no examples: the dataset counter lies.
3. **The engine's net shape always equals `MF_MODES[modeId].ml`** after any apply — the P5.3
contract that `tests/e2e/schema-modes.spec.ts` already enforces.
4. **`ctx.datasetCount` equals the engine's real example count** after apply. Same bar as §6.5e:
nothing may render a plausible number it did not get from the core.
5. **No C++, no schema, no codegen, no new WASM export in this item.** If a step appears to need one,
stop and re-spec — it means the scope was wrong. (Corollary: `scripts/parity-check.sh` is
*irrelevant* to this work; see §10.)
6. **Preset application is main-thread config only** — no worklet message, no audio-thread work
beyond the existing `setBackend`. `addExample` is a bounded heap copy per example, capped at 128
by validation (§3.3).
7. **`level` is persisted and never auto-reset** (§6.4); `presentation` is transient.
8. **`window.__mf` survives** with the same four methods (`ConsoleApp.tsx:331-345`), repointed at
`selectInstrument`. Playwright drives mode switches through it and must keep working *even after
a real picker exists* — the seam is faster and less brittle than clicking a popover.
9. **No "C15" in the bundle** (`tests/e2e/smoke.spec.ts:59`). The Library renders `MFMode.label`;
if the `c15` entry survives §8.5, its label is already "Powerful Synth Engine".
---
## 8. Deletions, with their consumers
| Delete | Consumers today | Replacement |
|---|---|---|
| **8.1** `backends/presets.ts` per-backend keying: `OutputPreset.backend`, `KEY_PREFIX:<backend>` namespaces, `listPresets/savePreset/getPreset/deletePreset/renamePreset(backend, …)` signatures, `OutputPresetRow` | `PresetBar` (`OutputsBackendConfig.tsx:99-208`, rendered `:686`) — the only one | `presets/store.ts` + `Preset`/`OutputOverride`. `PresetBar` becomes a view onto the unified store filtered by the active target, keeping its save/restore/rename/delete affordances. **Note the sequencing:** step 1 of §9 fixes the `cv` bug inside the old file; step 5 deletes the file. That is deliberate — the bug fix is independently valuable if 5c is never adopted. |
| **8.2** `DrawerDepth` as a disclosure flag | `Drawers.tsx` (21 sites, §6.1), `Dock.tsx`, `ConsoleApp.tsx:110/268/642` | `presentation` + `level` (§6.2) |
| **8.3** `OutputMode` / `outputMode` / `OUTPUT_MODES` / `output-mode.ts` names | `types.ts:26,110-111`, `output-mode.ts`, `Dock.tsx`, `Drawers.tsx:553,615`, `ConsoleApp.tsx:136-137,280,755-756,882,904` | `OutputTarget` / `target` / `OUTPUT_TARGETS` / `output-target.ts`. Mechanical rename; typecheck is the proof. Optional — §12.2. |
| **8.4** `ModeOverlay.input` + `MFMode.input` + `ModeInput` (`model.ts:41,141,211-216,229-247`) | `resolveInputMap(settings.inputMap, mode.input)` (`ConsoleApp.tsx:814`) — the only one | Derive from `schema.ui.primary_input` in `modeFromSchema` (`model.ts:246`); fixes the Breakor drift (§1.6.2). Manifold-only modes keep a literal. |
| **8.5** the `c15` placeholder mode (`model.ts:301-313`) | `MF_MODES` only; nothing renders it (§1.1) | Delete. It is a 2-fake-param entry on the default net shape that will become *visible* the moment a picker exists — the last survivor of the decorative stratum Phase 1 swept. Keep only if the operator wants roadmap signalling in the Library (§12.4). |
Not deleted, and why: `MFMode.glyph`/`cls`/`badge`/`placeholder` (§1.1) get their first real
consumer here. `visualizer` stays — it works, and its removal is a product call unrelated to this
item.
---
## 9. Sequenced implementation plan
Each step is independently landable and independently verifiable. Steps 1–2 are worth doing even if
the operator rejects everything after them.
**Step 1 — the `cv` preset bug** (§1.3). Three additions to `manifold/src/backends/presets.ts`:
`cv?: MFParam['cv']` in `OutputPresetRow` (`:16-29`), `cv: p.cv` in `rowsFromParams` (`:72-86`),
`cv: r.cv ?? p.cv` in `applyPreset` (`:141-162`).
*Verify:* a new `manifold/tests/presets.test.ts` case — build params with `cv` specs, round-trip
through `rowsFromParams`/`applyPreset`, assert channel + threshold survive. Runs in `bun run test`.
Red before, green after.
**Step 2 — schema-truth input kind** (§8.4, §1.6.2). Derive `MFMode.input` from
`schema.ui.primary_input` in `modeFromSchema`; drop `input` from `ModeOverlay`.
*Verify:* `bun run typecheck`; a unit test asserting `MF_MODES.find(m => m.id === 'breakor')!.input`
now follows the schema. Note this **changes visible behaviour** — Breakor's input surface flips from
circular to rectangular. That is the bug being fixed, and it belongs in the commit message.
**Step 3 — collapse mode switching into `selectInstrument`** (§4.2). Replace the three effects at
`ConsoleApp.tsx:241-243`, `:252-257`, `:260-271` with one function plus one boot effect keyed
`[engine]`. Repoint `window.__mf.setMode`.
*Verify:* this is the step with real regression risk and it already has coverage —
`tests/e2e/schema-modes.spec.ts` (four modes: dims, weight count, output arity, UI param count,
training after switch), `tests/e2e/reshape.spec.ts`, `tests/e2e/spine.spec.ts` (probe survives a
mode switch). Run the full Playwright suite, not just the smoke. On the VPS use the non-snap node
runner (`ONBOARDING.md` §2). **No new test proves the boot path** beyond the suite's own boot — if
this step misbehaves it will most likely be a mode switch that no longer reshapes, which
`schema-modes` catches, or a double-apply on boot, which nothing catches; add an explicit
"reshape called exactly once on boot" assertion via a probe counter if that risk is judged real.
**Step 4 — preset model + store + built-in catalogue** (§3). `presets/types.ts`, `presets/store.ts`,
`presets/builtin/`. Author **two or three** curated presets only — enough to exercise the model
(one config-only, one with `rest: 'off'` plus a handful of overrides, one with seeds). Authoring the
full curated set is a separate, per-mode pass — and note that pass is **not** governed by §7.5: the
20 anonymous "Param NN" slots (`ALIGNMENT.md` deferred debt; `schemas/modes/paf_synth.json` params
`p00`…) can only be named by editing the schemas, which means codegen and both generated outputs in
that change. Keep it out of this item.
*Verify:* `manifold/tests/presets.test.ts` — `validatePreset` over every `BUILTIN_PRESETS` entry
against `ALL_MODE_SCHEMAS`; store round-trip through a localStorage fake. `bun run test`.
**Step 5 — `applyPreset` + the weights invariant** (§4, §7.1). `presets/apply.ts` + `ConsoleApp`
wiring; retire the per-backend preset store (§8.1) and repoint `PresetBar`.
*Verify:* two parts, and neither alone is sufficient — say so in the commit rather than claiming
§7.1 is fully proven.
(a) **Unit, fake applier**: assert the call ORDER (seeds strictly after `selectInstrument`; routing
after the mode rebuild) and assert that no weight-mutating call — `setWeights`, `randomise`,
`train`, `trainAsync` — is made at all. This is the direct machine check on "configuration only".
(b) **C-ABI, real WASM**: in the style of `manifold/tests/loss-history.test.ts` (which drives the
committed `public/nisps.{js,wasm}` under `bun test` via `cwrap` — note it does NOT go through
`WasmIML`/`EngineApi`, which need a browser-ish environment), assert that
`nisps_ml_reshape` followed by N × `nisps_ml_add_example` leaves `nisps_ml_get_weights` bit-identical
to `nisps_ml_reshape` alone. That is the core-side half of the invariant.
**Residual gap:** nothing asserts the invariant end-to-end through the real `EngineApi` under
`bun test`. Step 6's e2e covers it indirectly (dims + dataset count after an apply). Closing it
properly would mean making `WasmIML` constructible headlessly, which is a bigger job than this
item.
**Step 6 — the Library** (§5). `console/InstrumentPicker.tsx` + the rail changes in `Dock.tsx`.
*Depends on* §6.5b (`plans/browser-mode-coverage-spec.md`) if its `modeSupport()` is to be rendered
(§5.4); buildable without it, at the cost of listing modes that cannot run.
*Verify:* a new `tests/e2e/library.spec.ts` — open the Library, apply a preset, assert (a) the
engine reshaped to that preset's mode dims via `window.__nisps.describe()`, (b) the rendered param
count matches the schema via `window.__mf.paramCount()`, (c) `datasetCount` equals the preset's seed
count, (d) no console errors. Plus `bun run test:e2e` for the smoke's no-"C15" assertion.
**Step 7 — presentation/level split** (§6). Rename `DrawerDepth`, add `levels` to `settings-store`,
re-gate all 21 sites per §6.3.
*Verify:* `tests/e2e/training-health.spec.ts` currently asserts the panel appears at `expanded`
depth — it must be updated to `advanced` level in the same change, and it is the direct proof the
mechanism swap preserved the §6.5e behaviour. Add one case: level persists across reload (the e2e
helper clears localStorage per test, so this needs an explicit in-test reload rather than a fresh
`loadProbe`). `bun run typecheck` catches the rename fallout.
**Step 8 — docs.** `manifold/ONBOARDING.md` (§4 drawer/dock description, the "`depth === 'expanded'`
is the advanced-surface flag" rule, the five-vs-six output modes error of §1.6.1), `MAP.md`
(`manifold/src/presets/`), `ALIGNMENT.md` defect 2, `dock-spec.md` (a grounding note recording the
depth model this replaces), `manifold-parity-features-spec.md` §1 (superseded pointer). Same commit
as the code that makes each true.
**Step 9 (optional) — `OutputTarget` rename** (§8.3). Pure mechanical; `bun run typecheck` is the
whole verification. Do it first or last, never in the middle of a behavioural step.
---
## 10. What the gates prove, and what they do not
- `bun run typecheck` / `bun run test` / `bun run test:e2e` are the real gates here. Baseline on
2026-07-21 before any of this work: typecheck clean, `bun test src tests/*.test.ts` → **23 pass,
0 fail, 6 files**.
- **`scripts/parity-check.sh` proves nothing about this item, in either direction.** It exercises
PAFSynth and ChannelStrip from an all-params-0.5 baseline; this item touches no C++ and no
schema, so a PASS is not evidence of correctness and could not be evidence of a regression. Do
not cite it. Do not skip it either — CI runs it regardless, and a red one means something else
broke.
- `bash scripts/build-cpp-tests.sh` and `scripts/lint-cpp.sh` likewise have no bearing; running them
is cheap insurance against having touched C++ by accident, which §7.5 forbids.
- **The weakest verification in this plan is step 3** (mode-switch refactor). It is covered by e2e,
which is slow, environment-sensitive on the VPS, and asserts outcomes rather than call counts. If
the operator wants stronger evidence, the cheap addition is a probe-visible reshape counter.
- **Nothing here needs hardware.** The one place hardware would matter — a preset reaching the
MEMLNaut — is explicitly out of scope (§0, §11.Q4).
---
## 11. Open questions for the operator
**Q1. Do you have saved output presets in the live app that must survive?**
§8.1 retires the `manifold-output-presets:<backend>` localStorage namespaces. If you have presets
saved at `meml.lnfinitemonkeys.org/next/` you care about, the answer is a one-off import shim
(~20 lines, deleted after one run); if not, they are simply orphaned and the code is cleaner. I
cannot check your browser storage.
**Q2. May a curated preset opt into one training pass over its seeds?**
§2.1 says the network ships untrained, so the default is no. But a preset with seeds and no training
sounds like an untrained net until the user presses Train — which may be exactly the intent
(the seeds are a *starting lesson*, not a *sound*), or may be a footgun. A `seed.autotrain?: boolean`
is three lines. Default proposed: absent, i.e. never.
**Q3. Should voice space be part of a preset?**
Every schema declares `voice_spaces` and `ui.show_voice_space_selector`, `nisps/modes/base.hpp:279`
implements selection, and **the browser cannot reach it** — there is no WASM export, and the audio
engine lives in the worklet behind raw exports (`worklet/nisps-processor.ts:33-36`), so this costs
the full 5-layer export chain *plus* worklet + engine-host message plumbing. It is a real
per-instrument character control that curation would obviously want. Out of scope as specified; say
if it should be pulled in, because it changes this item from "no C++" to "C++ and a new export".
**Q4. Where does curation ultimately live — browser only, or shared with firmware?**
§3.2 chooses browser-local TS data with a stated promotion condition (the hardware editor, §6.5d).
If you expect to author a preset in Manifold and send it to the MEMLNaut, say so now: the format
should then be JSON under `schemas/` from the start, because retrofitting codegen later means
rewriting the catalogue.
**Q5. URL sharing?** `manifold-parity-features-spec.md` §1.3 specified `?session=<base64url>` and a
compact axis form. Deferred here (nothing depends on it, and the axis form's control surface does
not exist). Worth scheduling, or drop it from the corpus?
---
## 12. Decisions I made (not questions)
Recorded so they can be reversed knowingly rather than rediscovered.
1. **The picker *is* the preset browser** (§2.3), one surface not two. Rationale: vision 3 wants the
curated path to be the default path; a separate mode list beside a preset list would make the raw
instrument the front door again.
2. **`OutputMode` → `OutputTarget`** (§8.3) is proposed but optional and sequenced last. It is pure
naming hygiene for a collision that only becomes painful once two "Mode" controls sit adjacent on
the rail.
3. **Explanatory prose shows at `basic` and hides at `advanced`** (§6.3), inverting today's
accidental behaviour where beginners' help is behind a modal. If you dislike it, the fallback is
one line: show prose unconditionally.
4. **Delete the `c15` placeholder mode** (§8.5), because a picker makes it visible and it does
nothing. Trivially reversible by keeping the entry and rendering its `badge: 'soon'` disabled.
5. **Two disclosure levels, not three** (§6.2) — `dock-spec.md` §0's `peek` never shipped and there
is exactly one tier of advanced content in the tree.
6. **Index-keyed output overrides** (§3.1), because `MFParam.name` is user-editable while schema
param order is codegen-stable.
7. **Two or three built-in presets in step 4, not a full catalogue** (§9). Authoring curated presets
per instrument is real design work that wants ears and hardware; the model must be proven first.

View file

@ -1,7 +1,18 @@
I have complete grounding now. I'll write the implementation-ready spec.
---
kind: plan
status: executed
---
# Implementation Spec — "Down Action" Negative-Feedback Feature in the `nisps/` Core
**Executed 2026-06/07; not authority for current behaviour.** The port landed, then evolved past
this document: the surviving truth is `docs/adr/rl-feedback-design.md` (which explicitly
supersedes §2.5/§7's AVOID-out-of-scope decision — the geometric push WAS ported, via
`nisps/ml/geo_push.hpp` + `nisps/ml/replay.hpp`) plus the code itself, `nisps/ml/feedback.hpp`
(now `FeedbackControllerCore<FbStorage>`, storage-policied, with ExploreAndPlace added). The
`playground/` paths below refer to the retired SolidJS app (branch `archive/playground-solidjs`);
the browser driver is now `manifold/src/feedback/controller.ts`.
## 0. Provenance
Every behavior below is read directly from local branch `fork-feedback` of the memllib submodule at `/home/w1n5t0n/src/MEMLNaut-NISPS-upstream/`:

View file

@ -0,0 +1,465 @@
---
kind: plan
status: active
---
# Manifold as Hardware Editor — MEMLNaut USB-Serial Protocol, Firmware Command Surface, On-Device Persistence
*Dated 2026-07-21. Spec for plan item **§6.5d** (`simplification-plan.md:130`), audit findings **A4** / **S14** /
**L29**, ALIGNMENT **defect 3** (and the live half of **defect 5**). Vision bullet 5: "Manifold doubles as
interface/editor for the hardware MEMLNaut (settings, presets, training, examples, visualisation)."*
**Nothing here is implemented. No code was written for this document.** Every `file:line` below was read from
the working tree on 2026-07-21 and is marked ✓ (verified by reading the file) or ✗ (could not verify — stated
as unknown, never asserted). The tree was mid-flight with concurrent edits from other sessions
(`git status` showed 20+ modified files including `src/main.cpp`, `codegen/generate.ts`, `ConsoleApp.tsx`);
re-check citations in those files before relying on an exact line number.
---
## 1. Ground truth — what exists today
### 1.1 The browser end is a connect-lifecycle shell
| Fact | Evidence |
|---|---|
| `MemlnautSerial` handles `requestPort`/`open`/`close` and a 5-state status store, nothing else | ✓ `manifold/src/serial/memlnaut-serial.ts` (140 lines, not the 237 the audit reports — stale) |
| `saveModel` returns `false`, `restoreModel` returns `null`, `getSettings` returns `{}`, each with a `TODO: real protocol` | ✓ same file, the three methods under `// ---- Protocol stubs ----` |
| Baud 115200 is explicitly a placeholder | ✓ same file, `TODO(memlnaut-serial): negotiate the real baud rate` |
| The only consumer is `EditorPanel.tsx`; the only consumer of *that* is the Settings-drawer `editor` case | ✓ `manifold/src/console/Drawers.tsx:33` (import), `:579` (render). Repo-wide grep for `memlnaut-serial`/`getMemlnautSerial` finds no other importer. |
| `editor` is a top-level dock **output mode**, `audio: false`, description already promises "configure / save / restore" | ✓ `manifold/src/console/output-mode.ts:69-75` |
| The panel already carries an honest "not yet wired" note | ✓ `EditorPanel.tsx`, final `<p>` |
`manifold/src/serial/web-serial.d.ts` is an **ambient** (global) declaration file. Its header comment claims it
declares "just the surface memlnaut-serial.ts uses" — false: `manifold/src/backends/cv-backend.ts:56,95,123,133`
type-checks against the same global `SerialPort`/`navigator.serial`. There is one Web Serial *type* surface and
two *transport* implementations, not three (✓ grep for `navigator.serial|SerialPort` over `manifold/src` returns
exactly `web-serial.d.ts`, `cv-backend.ts`, `memlnaut-serial.ts`).
### 1.2 The firmware end has a USB CDC port, and nothing listening on it
| Fact | Evidence |
|---|---|
| `Serial` (USB CDC) is opened at boot and used **only** for diagnostics | ✓ `firmware/MEMLNaut-NISPS/src/main.cpp:125` (`Serial.begin(115200)`) |
| Core 0's `loop()` prints `"."` ~10×/s and a perf line ~1×/s onto that same stream | ✓ `src/main.cpp:166`, `:172` (`Serial.printf("ml: %d, aud: %d, q: %f\n", …)`) |
| **The USB CDC port is otherwise free**: MIDI runs on `Serial2` (UART1), not USB | ✓ `lib/memllib/src/interface/MIDIInOut.cpp:15` `MIDI_CREATE_CUSTOM_INSTANCE(HardwareSerial, Serial2, MIDI, …)`, `:81-88` |
| USB-MIDI (TinyUSB composite) is compiled out — `MIDI_USB_CLIENT` is defined nowhere | ✓ `MIDIInOut.cpp:55` is `#ifdef MIDI_USB_CLIENT`; grep across `platformio.ini`, `src/`, `glue/` finds no definition. The device enumerates as a plain CDC port. |
| There is no serial *reader* anywhere in firmware — no `Serial.read()`, no parser, no command dispatch | ✓ grep over `src/` + `glue/` |
| Core 0's control cadence is 5 ms (`ML_INFERENCE_PERIOD_US 5000`), core 1's drain 1 ms | ✓ `src/main.cpp:152-160`, `:208-216` |
### 1.3 There is no on-device persistence, but two mechanisms are already compiled in
- **LittleFS** (internal flash). memllib wraps it in `lib/memllib/src/hardware/FlashFS.hpp` ✓ — which has **zero
consumers** in this repo (✓ grep for `FlashFS` hits only vendored TFT_eSPI example sketches under `.pio/`).
Two problems with that wrapper: it defines `void begin()` and `bool exists()` as **non-`inline` free functions
in a header** (✓ read in full — 22 lines), so including it from two translation units is a duplicate-symbol
link error; and using it at all buys nothing over calling `LittleFS` directly.
- **The filesystem is currently sized to zero.** `platformio.ini` does not set `board_build.filesystem_size`
(✓ grep), and the platform's default is `"0MB"` (✓ `~/.platformio/platforms/raspberrypi/builder/main.py:63`).
With `_size == 0`, `LittleFS.begin()` returns `false` immediately (✓
`~/.platformio/packages/framework-arduinopico/libraries/LittleFS/src/LittleFS.h:173`). **On-device flash
persistence therefore requires a `platformio.ini` change that alters the flash layout of all 16 envs.**
- **LittleFS writes park core 1.** ✓ `…/LittleFS/src/LittleFS.cpp:186-193` and `:203-209`:
`noInterrupts(); rp2040.idleOtherCore(); flash_range_program/flash_range_erase(...); rp2040.resumeOtherCore();
interrupts();`. Core 1 is the audio core. **Any flash write stalls audio for the duration of the erase +
program.** This is the single hardest constraint in this document.
- **SD card.** memllib ships `interface/SDCard.{hpp,cpp}` (SdFat over hardware SPI) ✓, the MEMLNaut board has
dedicated SD pins ✓ `lib/memllib/src/hardware/memlnaut/Pins.hpp:87-90` (`SD_CS 13`, `SD_SCK 14`, `SD_MISO 12`,
`SD_MOSI 15`), and `SDCard.cpp` **already compiles into every variant** ✓ (`.pio/build/pafsynth/…/interface/
SDCard.cpp.o` exists). SPI writes do **not** lock out core 1. It needs a physical card fitted; whether the
operator's units have one is ✗ unknown.
### 1.4 The model *is* transferable — the shapes already agree
- Firmware instantiates a compile-time `MLP<…>` whose dims come from the mode's generated schema; the browser
reshapes its `MLPCore<DynamicStorage>` to **the same schema dims** on every mode switch ✓
`manifold/src/console/ConsoleApp.tsx:252-257` (`engine.reshape({ inputSize, outputSize, hidden }, defaultSpread)`
from `mode.ml`). So for a given `mode_id` the two nets have the *same* `weight_count` by construction.
- **But not unconditionally.** The browser also offers an axis-count reshape that changes `inputSize` alone ✓
`ConsoleApp.tsx:290-322`, and `nisps_ml_create`'s default shape is a 32-input head ✓
`manifold/src/inputs/input-layer.ts:17,38`. A live browser net can therefore be at `6→[10,10,14]→33` while the
device is at `4→[10,10,14]→33`. **The protocol must carry the shape and refuse mismatches; it must never infer
compatibility from `mode_id` alone.**
- Flat weight layout is already specified and identical on both targets: `[l0_w][l1_w][l2_w][l3_w][l0_b][l1_b]
[l2_b][l3_b]`, little-endian `float32` ✓ `nisps/wasm/bindings.cpp:25-34` (comment), `nisps/ml/mlp.hpp:385-414`
(`get_weights`/`set_weights`).
- Per-mode blob sizes (computed from `schemas/modes/*.json` ✓):
| mode | dims | weights+biases | bytes |
|---|---|---|---|
| `sound_analysis_midi` | 10→[10,10,14]→8 | 494 | 1 976 |
| `channel_strip`, `xiasri` | 4→[10,10,14]→24 | 674 | 2 696 |
| `paf_synth` | 4→[10,10,14]→33 | 809 | 3 236 |
| `elysiamorf` | 4→[10,14,18]→40 | 1 234 | 4 936 |
| `verb_fx` | 4→[10,14,18]→47 | 1 367 | 5 468 |
| `breakor`, `memlcelium`, `slp_workshop` | 4→[10,14,18]→56 | 1 538 | **6 152** |
Worst case is ~6 KB. Small — but not small enough to sit in one frame without either a 6 KB RX buffer or
chunking.
### 1.5 Training examples are **not** readable out of the core today
`MLPCore` exposes `example_count()` ✓ `nisps/ml/mlp.hpp:441` and `clear_examples()` ✓ `:443`, but the per-example
accessors `sample_features_(s)` / `sample_labels_(s)` are **private** ✓ `:577`, `:581`. Exporting the training
set from the device is therefore a real (small) addition to `nisps/ml/mlp.hpp`, not a wiring job.
Worse for the editor: the firmware's `FeedbackController` — which owns explore/place state and drives
`add_example` — is a **function-local static inside `bind_peripherals()`** ✓
`firmware/MEMLNaut-NISPS/glue/peripherals.hpp:152`. It is unreachable from `main.cpp` or any command handler.
Any command that touches feedback state requires hoisting it first.
### 1.6 The discipline to copy already exists in-repo
`firmware/useq-celium/shared/protocol.h` ✓ is a C header of `static const` constants + `static inline` helpers,
mirrored constant-for-constant by `manifold/src/backends/useq-protocol.ts` ✓, pinned by
`manifold/src/backends/useq-protocol.test.ts` ✓ (`test('frame sizes match the C header')`), documented as
`kind: spec` in `docs/specs/useq-cv-protocol.md` ✓. The device-side parser is a ~20-line resync state machine ✓
`firmware/useq-celium/main/src/main.cpp:93-110`. **Verified live**: `bun test src tests/*.test.ts` in `manifold/`
→ `23 pass, 0 fail, 6 files, 65 ms`.
### 1.7 L29 — `InputChain`/`OutputChain` have no firmware consumer
✓ Repo-wide grep: the only non-test, non-self references are `nisps/wasm/bindings.cpp:76-77,194-195,968,1011`
and `tests/cpp/{test_pipeline,parity_check}.cpp`. `nisps/pipeline/output_chain.hpp:13` says "firmware would pick
its mode's NOut" — aspirational, present-tense-adjacent, and currently false. `InputChain` already serialises
(`state_size()`, round-trip test ✓ `tests/cpp/test_pipeline.cpp:131-150`), which matters below.
---
## 2. Invariants
Beyond the repo-wide hard constraints (platform-neutral allocation-free `nisps/`; native↔WASM parity ≤1e-5;
schema changes ship both codegen outputs; RT-safe worklet + SPSC dual-core discipline), this area adds:
- **I1 — The wire format has exactly one source of truth.** A single C header under `firmware/`, mirrored in TS,
pinned by a test that fails when they diverge. No constant is written twice by hand in the same language.
- **I2 — No hand-maintained per-mode table anywhere in the protocol.** Mode identity, net shape, param count and
ordering reach the wire from codegen output on both sides, or they do not reach it at all.
- **I3 — A model blob is applied only against a proven-identical contract.** Shape *and* schema fingerprint must
match. On mismatch the device NAKs; it never partially applies, never truncates, never zero-pads.
- **I4 — Serial servicing is bounded.** The core-0 pump reads at most a fixed byte budget per `loop()` iteration.
The 5 ms `tick_control()` cadence ✓ (`src/main.cpp:152,160`) is not to be perturbed by a host that streams.
- **I5 — Serial handling never runs on core 1.** Core 1 is audio + MIDI drain. All parsing, buffering, filesystem
and ML mutation happen on core 0.
- **I6 — Flash writes are an explicit, announced, user-initiated act.** Because they park core 1 (§1.3), a
persist is never automatic, never periodic, and never a side effect of another command.
- **I7 — Diagnostics and protocol do not share the stream unframed.** Either the diagnostics are silenced for the
session or they are carried inside a frame. Never raw ASCII interleaved with binary payloads.
- **I8 — The device is authoritative about itself.** The browser asks and believes; it never assumes a firmware
build from a mode name, a port name, or a USB descriptor.
- **I9 — No compat shim.** The stubs are deleted, not deprecated. `EditorPanel` is the only consumer (§1.1).
---
## 3. Design
### 3.1 Transport and channel
USB CDC (`Serial`), 8N1; the CDC baud parameter is ignored by the hardware but the host must supply one —
**115200**, matching what the firmware already calls and what `cv-backend.ts:133` uses. No second UART, no
composite USB device, no TinyUSB reconfiguration.
**The diagnostics collision (I7) is resolved by session state.** On `HELLO` the firmware sets an
`editor_session` flag; while set, `loop()`'s `Serial.println(".")` and perf `printf` (✓ `src/main.cpp:166,172`)
are suppressed and the same information is emitted as a `TELEMETRY` frame instead. `BYE`, a host disconnect
(DTR drop), or a 5 s silence timeout clears the flag and restores the prints. Boot-time prints
(`"Serial initialised."`, `"Bound peripherals to mode."`) happen before any session and are harmless — the
host-side parser drops non-sync bytes anyway.
*Rejected:* moving diagnostics to a second UART (costs pins and a second cable) and CDC-composite dual ports
(complicates the Web Serial port picker for one debug convenience).
### 3.2 Framing
Variable-length, because payloads range from 0 bytes to a 6 KB model (§1.4). Fixed-length-per-type (the useq
choice) does not survive that.
```
byte 0 sync 0xA5 host→device, 0x5A device→host
byte 1 type u8
bytes 2..3 length u16 LE, payload bytes, 0..MEMLED_MAX_PAYLOAD
bytes 4.. payload
last 2 crc u16 LE, CRC-16/CCITT-FALSE over bytes [1 .. 3+length]
```
- **CRC-16, not XOR-8.** useq's XOR-8 is right for a 26-byte frame streamed at 100 Hz where a corrupt frame is
discarded and replaced 10 ms later. It is wrong for a 6 KB model transfer, where a 1-in-256 undetected
corruption silently installs wrong weights and the user hears an inexplicable instrument. Bitwise CRC-16 is
~15 lines, table-free, and identical on both sides.
- **`MEMLED_MAX_PAYLOAD = 512`.** Bounds the device RX buffer at 518 bytes and each frame's service time at
~45 ms of wire time worst case — hence I4's byte budget. Larger objects chunk (§3.4).
- Both directions use the same frame shape; only the sync byte differs, so one parser implementation serves
both ends.
Home: **`firmware/shared/memlnaut-editor-protocol.h`** — deliberately *not* under `firmware/MEMLNaut-NISPS/`,
because it is shared with the browser exactly as `firmware/useq-celium/shared/protocol.h` is. Mirror:
**`manifold/src/serial/memlnaut-protocol.ts`**. Parity test: **`manifold/src/serial/memlnaut-protocol.test.ts`**
(picked up by the existing `bun test src tests/*.test.ts` glob — do not touch that script).
### 3.3 Identity: a codegen-emitted schema fingerprint (discharges I2, I3)
The one genuinely new idea in this spec, and the thing that makes "derive from schema codegen, not hand-defined
tables" concrete.
`codegen/generate.ts` gains one emitted constant per mode: a **FNV-1a 64 hash over a canonical serialisation of
the mode's contract-bearing schema fields** — `mode_id`, `engine_id`, `ml` (`input_size`, `hidden_layers`,
`output_size`, `default_spread`), `params[]` as `(name, min, max, default, curve, group)` in order, and
`voice_spaces` (names + curve overrides) in order. Deliberately **excluded**: `label`, `_note`, `ui` — cosmetic,
and a label edit must not invalidate a user's trained model.
Emitted into both generated worlds in the same change (hard constraint):
- `nisps/modes/generated/<mode>_schema.hpp` → `inline constexpr std::uint64_t k<Mode>SchemaHash = 0x…ull;`
alongside the existing `k<Mode>ModeId` ✓ (`nisps/modes/generated/paf_synth_schema.hpp:10`)
- `manifold/src/modes/generated/<mode>_schema.ts` → `schema_hash: '0x…'` on the existing `ModeSchema` const
CI already fails on a stale generated tree (`.github/workflows/ci.yml`, step *"Codegen is committed and
idempotent"* ✓ `:129`), and `codegen/tests/golden/paf_synth_schema.{hpp,ts}` ✓ will need regenerating in the
same commit.
The device reports its hash in `DESCRIBE`; the browser compares against **its own** generated hash for that
mode. Equal ⇒ weights, settings and examples are interchangeable. Unequal ⇒ the UI says *"this MEMLNaut is
running a different build of `paf_synth`"* and disables transfer. No table, no version negotiation matrix, no
per-mode special case — and the check is exact rather than heuristic.
### 3.4 Command set (protocol v1)
Types are `static const uint8_t` in the shared header. `ACK`/`NAK` are the universal replies; `NAK` carries a
one-byte reason code from an enum in the same header.
**Host → device**
| Type | Payload | Reply |
|---|---|---|
| `HELLO` | `u16 host_proto_ver` | `DESCRIBE` — opens the session, silences diagnostics |
| `BYE` | — | `ACK` — closes the session, restores diagnostics |
| `GET_MODEL` | — | `MODEL_BEGIN`, then `MODEL_CHUNK`×N |
| `PUT_MODEL_BEGIN` | `u64 schema_hash, u16 dims[5], u32 weight_count, u32 crc32` | `ACK`/`NAK` |
| `PUT_MODEL_CHUNK` | `u16 index, bytes` | `ACK` |
| `PUT_MODEL_END` | — | `ACK` (applied) / `NAK` (CRC or count mismatch — **nothing applied**) |
| `GET_SETTINGS` | — | `SETTINGS` |
| `SET_SETTING` | `u16 id, f32 value` | `ACK`/`NAK` |
| `PERSIST` | `u8 slot` | `ACK`/`NAK` — see I6 |
| `LOAD_PERSISTED` | `u8 slot` | `ACK`/`NAK` |
| `CLEAR_EXAMPLES` | — | `ACK` |
| `SUBSCRIBE` | `u16 mask, u8 rate_hz` | `ACK` — telemetry stream on/off |
**Device → host**
`DESCRIBE`, `MODEL_BEGIN`, `MODEL_CHUNK`, `SETTINGS`, `TELEMETRY`, `LOG` (UTF-8 text, framed — the session
replacement for the raw prints), `ACK`, `NAK`.
`DESCRIBE` payload: `u16 proto_ver`, `u64 schema_hash`, `mode_id` (fixed 24-byte NUL-padded ASCII, from
`generated::k<Mode>ModeId` ✓), `u16 dims[5]`, `u32 weight_count`, `u16 max_examples`, `u16 capability_bits`,
`char fw_build[16]`. The capability bits are what let one protocol serve 16 firmware variants without a
per-variant table: `HAS_PERSIST`, `HAS_SD`, `HAS_EXAMPLE_EXPORT`, `HAS_TELEMETRY`, `IS_SEQUENCER`.
**PUT_MODEL is atomic (I3).** Chunks land in a staging buffer sized `weight_count * 4` — for the largest mode
6 152 bytes of core-0 RAM, allocated as a `static` array sized from `MLPType::weight_count()` at compile time,
so no heap. `set_weights` is called once, from `PUT_MODEL_END`, only after the CRC-32 over the whole blob
matches and `weight_count` and `dims[5]` equal the device's own.
**Settings (I2, honestly).** Not everything can come from codegen, and pretending otherwise would be the
hand-defined table under a different name. The split:
- *Derived from codegen, on both sides:* the training triple `learning_rate` / `max_iterations` / `min_error`
(defaults + identity from `schemas/ml_defaults.json` → `nisps::ml::generated::kMlTrainDefaults` ✓ and
`manifold/src/modes/generated/ml_defaults.ts` ✓ — both mirrors already exist), and `voice_space_index` whose
legal range is the generated `voice_spaces` list ✓.
- *Genuinely device-side, enumerated in the shared header:* `pin_value`, `joystick_single` (the Dual/Single
toggle ✓ `glue/settings_view.hpp:54-58`), `explore_intensity` (RVX1 ✓ `glue/peripherals.hpp:124-126`),
`master_volume` (RVGain1 ✓ `:115-117` — note it drives `AudioDriver::SetMasterVolume` directly, not the mode). These are `ModeBase`/peripheral concepts with no schema home; there
are four of them; they live in one enum in the shared header and nowhere else.
**Telemetry closes ALIGNMENT defect 5's live half.** The `TELEMETRY` payload carries `PERF_GET_MEAN(MLSTATS)`
and `AUDIOLOOP_MEAN` — numbers the firmware **already computes and prints** ✓ `src/main.cpp:172` — plus
`example_count()`, `eval_loss()`, and (paged) the loss history the operator deliberately kept on device
(audit L25). That is the on-device timing report ALIGNMENT defect 5 asks for, at near-zero marginal cost,
displayed in a browser panel that already knows how to draw a loss curve (`console/TrainingHealth.tsx` ✓).
### 3.5 Persistence
**Primary: LittleFS on internal flash**, because it needs no accessory and every unit has it.
Required changes: `board_build.filesystem_size` in `platformio.ini`'s `[env]` block (§1.3 — currently absent,
default `0MB`, `begin()` fails), a `glue/persistence.hpp` that calls `LittleFS` **directly** (not memllib's
`FlashFS.hpp`, whose non-`inline` header functions are a duplicate-symbol trap and which adds nothing — §1.3),
and a fixed on-disk record:
```
/memlnaut/<mode_id>/<slot>.mdl magic, u16 record_ver, u64 schema_hash, u16 dims[5],
u32 weight_count, f32 weights[], settings block, u32 crc32
```
`schema_hash` on disk is checked on load exactly as on the wire (I3), so a firmware reflash that changes the
schema invalidates old saves loudly instead of loading garbage.
**The audio stall is real and must be surfaced, not hidden.** Sequence for `PERSIST`: mute via
`AudioDriver::SetMasterVolume(0)`, emit `LOG "saving…"`, write, restore volume, `ACK`. The browser shows a
determinate "Saving to device — audio will pause" state. The alternative — pretending a flash erase is free —
would be exactly the kind of plausible-looking lie this repo has been deleting all week.
**Not chosen, but noted:** the SD path (`interface/SDCard.hpp` ✓, already compiled ✓, pins ✓) does not park
core 1 and is the right home for bulk artefacts (full example sets, session recordings) if the operator's units
have cards fitted. Left behind the `HAS_SD` capability bit rather than built speculatively.
### 3.6 Browser side
`manifold/src/serial/memlnaut-serial.ts` is **rewritten, not extended**: the three stub methods and their
`TODO`s are deleted (I9 — sole consumer is `EditorPanel.tsx`, §1.1). The class becomes a framed-transport
driver: a read loop over `port.readable`, a resync parser identical in shape to the TS mirror's encoder, a
promise-keyed request/reply map, and the same `subscribe()` store it already has (which `EditorPanel` already
consumes via `useSyncExternalStore` ✓).
`EditorPanel.tsx` grows: a device card (mode, build, dims, compatibility verdict), Save-to-device /
Load-from-device with the compatibility gate and a progress state, a settings section, and a live telemetry
strip. **The "not yet wired" note is deleted in the same commit as the code that wires it** — a note that
outlives its truth is the failure mode this repo keeps finding.
`output-mode.ts:72`'s description ("configure / save / restore") becomes true and needs no edit. Two small
hygiene items ride along: fix `web-serial.d.ts`'s false header comment (§1.1) and move it to a shared location
now that two modules provably depend on it.
---
## 4. Deletions
| Deleted | Named consumers | Why safe |
|---|---|---|
| `MemlnautSerial.saveModel` / `.restoreModel` / `.getSettings` bodies + `TODO`s | `EditorPanel.tsx` only (✓ grep) | Replaced in the same change; no external caller |
| `EditorPanel`'s "not yet wired" `<p>` | none | Becomes false the moment step 6 lands |
| The unconditional `Serial.println(".")` / perf `printf` in `src/main.cpp:166-175` | none (human eyeball on a serial monitor) | Not deleted — made session-conditional. Outside a session behaviour is unchanged. |
| `lib/memllib/src/hardware/FlashFS.hpp` — *not* deleted | none (✓ zero consumers) | Vendored upstream code; deleting it diverges the vendor tree for no gain (`VENDORED.md` re-sync). We simply do not use it, and `persistence.hpp` says why. |
Nothing else. In particular the `cvgate` backend's Web Serial code stays untouched — different device, different
protocol, no shared transport worth extracting for two implementations.
---
## 5. Sequenced implementation plan
Each step is independently landable and independently verifiable. Steps 1–4 are agent-safe with the existing
gates. Steps 5–8 need hardware and are honestly marked.
**Step 0 — L29 disposition (do first, it is one line either way).**
`nisps/pipeline/output_chain.hpp:13` and `input_chain.hpp`'s header currently imply a firmware consumer that does
not exist (§1.7). If the operator answers **Q4** (below) with "not now", soften both comments to say the chains
are browser-side today, in this step. If "yes", they stay and step 8 fulfils them. Either way L29 stops being an
open lie after step 0.
*Verification:* `bash scripts/lint-cpp.sh` (comment-only change; the lint strips comments before matching ✓).
**Step 1 — the shared header + TS mirror + parity test. No behaviour anywhere.**
`firmware/shared/memlnaut-editor-protocol.h`, `manifold/src/serial/memlnaut-protocol.ts`,
`manifold/src/serial/memlnaut-protocol.test.ts`. Encoder/decoder + CRC-16 in both languages; the test asserts
frame offsets, `MEMLED_MAX_PAYLOAD`, every type byte, and CRC-16 against fixed vectors.
*Verification:* `cd manifold && bun run typecheck && bun run test` (the new test is picked up by the existing
glob — **do not change the test script**). Additionally register a host C++ test that includes the header and
asserts the same vectors: add a source to `nisps_modes_tests`, which already puts the repo root on its include
path for exactly this reason ✓ (`nisps/CMakeLists.txt:134`). Then `bash scripts/build-cpp-tests.sh` proves the C
and TS encoders agree on the same vectors on both sides.
*Note:* `scripts/parity-check.sh` says nothing about any of this — it exercises PAFSynth and ChannelStrip at
all-params-0.5. A green parity run is not evidence for this step.
**Step 2 — codegen emits the schema fingerprint.**
`codegen/generate.ts` + `codegen/lib.ts`; C++ and TS in the same commit; `codegen/tests/golden/*` regenerated.
*Verification:* `cd codegen && bun run generate.ts` twice (idempotence), `bun run test` (golden + curve-drift),
then `bash scripts/build-cpp-tests.sh` (the generated headers are compiled by `nisps_modes_tests` ✓). CI's
"Codegen is committed and idempotent" step ✓ is the backstop.
**Step 3 — firmware command surface, read-only commands.**
`glue/editor_serial.hpp`: the bounded pump (I4), the parser, `HELLO`/`DESCRIBE`/`GET_SETTINGS`/`GET_MODEL`/
`SUBSCRIBE`/`BYE`, the session flag gating the diagnostics. Called from `loop()` on core 0 next to
`tick_control()`. Deliberately no mutation yet.
*Verification (weak, be honest):* `pio run -e slpworkshop -e pafsynth -e selftest` compiles and the flash/RAM
delta is reported (CI's firmware job already does this ✓ `.github/workflows/ci.yml:213,223`). Compilation is
**all** that is verified without hardware. The parser itself should be structured so its byte-level core is
Arduino-free and host-testable — the precedent is `glue/codec_config.hpp`, deliberately Arduino-free and covered
by `tests/cpp/test_mode_driver_config.cpp` ✓. Follow it: put the state machine in the shared header or a pure
sibling, and the `Serial` reads in the glue.
**Step 4 — browser transport + read-only editor UI.**
Rewrite `memlnaut-serial.ts` onto the framed protocol; `EditorPanel` renders `DESCRIBE` + telemetry + the
compatibility verdict. Save/Load still absent.
*Verification:* `bun run typecheck && bun run test && bun run build`. A Playwright e2e can cover the panel's
**disconnected** rendering only — Web Serial cannot be driven headlessly. Do not claim more. The frame codec is
covered by step 1's unit test; the *round trip* is not covered by anything until hardware.
**Step 5 — firmware mutating commands.**
`PUT_MODEL_*` (staging buffer + atomic apply), `SET_SETTING`, `CLEAR_EXAMPLES`. Requires hoisting the
`FeedbackController` out of `bind_peripherals`'s function-local static (§1.5) if any command touches feedback
state; if v1 avoids feedback entirely, say so and leave it.
*Verification:* compile + size. The atomicity logic (CRC-32 accumulate, count check, single `set_weights`) is
host-testable if written as a pure function over spans — do that, and cover it in `nisps_modes_tests`.
**Step 6 — Save-to-device / Load-from-device in the UI**, gated on the schema hash + dims match, with progress
and the explicit incompatibility message. Delete the "not yet wired" note here.
*Verification:* typecheck + unit + build; e2e for the **gate** (a fake `DESCRIBE` with a wrong hash must disable
the buttons) is worth writing because it is pure logic.
**Step 7 — persistence.**
`board_build.filesystem_size` in `platformio.ini` (**changes the flash layout of all 16 envs — rebuild and
re-report sizes**), `glue/persistence.hpp` on `LittleFS` directly, the record format, `PERSIST`/`LOAD_PERSISTED`,
the mute-around-write sequence.
*Verification:* all 16 envs build and the size table is re-baselined (`scripts/build-firmware.sh --all`). The
record encode/decode is pure and host-testable; the flash behaviour, the stall duration, and whether the audio
interruption is acceptable are **operator chokepoints — no automated gate reaches them.**
**Step 8 — (conditional on Q4) `InputChain`/`OutputChain` firmware wiring.**
`InputChain` in `bind_peripherals` between the joystick callbacks and `mode.set_input`; `OutputChain<MLPType::
kOutput>` in `ModeBase::tick_control` before `engine_.set_params`. Both configs become editor settings, which is
the only reason firmware would want them configurable at all. Note the cost honestly: `OutputChain<NMax>` adds an
`NMax`-float state array to every mode instance and a per-tick pass at 200 Hz.
*Verification:* host tests already exist for both chains ✓ (`tests/cpp/test_pipeline.cpp`) and parity stage 7
covers them ✓ (`tests/cpp/parity_check.cpp:341-394`) — but **neither proves the firmware wiring**, and the thing
that actually changes (how the joystick *feels*) can only be judged by ear on hardware.
---
## 6. Open questions — operator only
1. **Persistence medium and the audio stall.** Internal flash (always present, but every save parks core 1 and
interrupts audio for the erase+program — §1.3) or SD (no stall, but needs a card fitted, and it is ✗ unknown
whether your units have one)? If flash: is a brief, announced audio interruption on an explicit save
acceptable, or does that rule flash out?
2. **What "presets" means for the device.** Vision bullet 5 says *settings, presets, training, examples,
visualisation*. This spec covers settings, model transfer, telemetry, and persistence slots. It does **not**
define a preset object — and §6.5c (curated/advanced split) is supposed to define exactly that, and is
scheduled *before* this item. Does the editor wait for 5c's preset model, or ship slots-of-weights now and
absorb presets later?
3. **Do examples need to leave the device?** Exporting the training set requires a new public accessor on
`MLPCore` (§1.5) and probably hoisting the `FeedbackController` (§1.5). "Train on device, curate in browser"
implies yes; "save/restore a model" implies no. This is the difference between a small v1 and a much larger
one.
4. **L29 / step 8: should firmware gain the browser's input pipeline?** Deadzone, circular clamp, momentum zoom
and EMA would change the feel of every hardware mode, and only your ears can judge it. If the answer is "not
now", step 0 softens the comments and the question closes cleanly.
5. **Whose model wins on connect?** When the browser connects and both ends hold a trained net for the same
mode, does the editor do nothing until told (proposed default), or offer/auto-pull the device's?
## 7. Decisions taken here (not questions)
- CRC-16 for frames, CRC-32 for whole-model transfers, not useq's XOR-8 — §3.2.
- Variable-length frames with a 512-byte cap and chunked models, not one large frame — §3.2, I4.
- Diagnostics silenced by session flag rather than moved to another port — §3.1, I7.
- Compatibility by codegen-emitted schema fingerprint + explicit dims, never by `mode_id` alone — §3.3, I3.
- `LittleFS` called directly; memllib's `FlashFS.hpp` not used and not deleted — §3.5, §4.
- The four device-side settings live in an enum in the shared header, and the spec says plainly that they are
not codegen-derived rather than pretending — §3.4.
- Protocol header at `firmware/shared/`, not inside `firmware/MEMLNaut-NISPS/`, mirroring useq-celium.
## 8. Where verification is weak
Say this out loud in any PR description that lands these steps:
- **`scripts/parity-check.sh` proves nothing here.** It exercises PAFSynth and ChannelStrip with all params at
0.5. It touches no serial code, no framing, no persistence.
- **Nothing in CI executes firmware.** The firmware job compiles three envs and reports sizes ✓
(`.github/workflows/ci.yml:213,223`). A protocol that compiles is not a protocol that works.
- **Web Serial cannot be exercised headlessly.** Playwright can cover the panel's disconnected and
incompatible-device states, and the frame codec is unit-tested on both sides — the *round trip over a real
cable* is untested until someone plugs a MEMLNaut in.
- **Mitigation worth building early:** a tiny host-side loopback harness — the TS encoder feeding the C parser
compiled natively into `nisps_modes_tests` — turns "the two ends agree about bytes" from a hope into a gate,
and costs about an hour. It still says nothing about USB CDC, flash timing, or feel.
- **Step 7 has no automated gate at all** beyond "16 envs still build". Flash layout, stall duration and
recovery-after-power-loss are hardware chokepoints.

View file

@ -0,0 +1,486 @@
---
kind: plan
status: active
---
# Manifold Parity Features — Session Presets · Pins · Jolt · OU-Explore · Control Surface
*Date: 2026-07-12; status updated 2026-07-21. Prescriptive plan for porting five playground
features into `manifold/`. Now part-executed:*
- ***§3 Jolt and §4 OU-explore: EXECUTED by other means*** *— one-core-engine P3 landed the core
bindings (this document's own preferred route, §0 principle 1). The shipped C ABI is
`nisps_ml_jolt_{press,step,release,active}` and `nisps_ml_explore_{intensity,get_intensity,apply}`
— not the §3.2/§4.2 names proposed below — driven by `manifold/src/engine/exploration.ts`.
§3/§4 are kept as behavioural reference; the code wins.*
- ***§1 session presets, §2 pins, §5 control surface: NOT implemented.*** *These remain the live
prescription, feeding the curated/advanced-split work (`simplification-plan.md` §6.5c).*
**Traces up to:** `playground-2.0-rewrite-plan.md` (superseded plan — its target died but these
sections are still the cited design source) §2.3 (Feel drawer), §2.6 (pins), §3.2 (fanout,
control-point tri-state), §3.8 (composed-layers presets, persist helper);
`../slp-workshop-firmware.md` §3–4 (Jolt / OU-explore, Part I shipped in firmware);
`../dock-spec.md` (drawer depths, tri-state semantics); BUILD-PLAN locked decisions
(parity-tested engine; British spelling; the synth is "Powerful Synth Engine", never the forbidden
string).
**Reference implementations:** the playground versions are the behavioural ground truth for
constants and algorithms — `src/features/session-preset.ts`, `src/stores/session-store.ts` (pins),
`src/ml/jolt.ts`, `src/output/ou-explore.ts`, `src/stores/control-store.ts` +
`src/features/control-routing.ts`, all on the retired-playground archive (branch
`archive/playground-solidjs`, tag `playground-solidjs-final` — these files no longer exist on
main). Where this spec and the playground disagree, this spec wins (each divergence is called out
and justified inline).
---
## 0. Principles applied throughout
1. **C++ owns gesture math where a C++ class exists.** `nisps::ml::Jolt` and
`nisps::ml::OUNoise<N>` (`nisps/ml/jolt.hpp`, `nisps/ml/ou_noise.hpp`) are bound to WASM
rather than re-ported to TS — deterministic, firmware-parity noise for free. *(Done — this
landed via one-core-engine P3; see the status block above.)*
2. **No new mechanisms where an existing one already expresses the idea.** Param pins are the
existing `off|fixed|live` tri-state + arm mask, not a parallel pin system (§2.2).
3. **Transparent defaults.** Every default constant appears in this spec as a number with its
source, and the control surface UI must *show* the derived per-param values live (§5.6) —
the user should never wonder what an axis is secretly doing.
4. **One persistence pattern.** All new persisted state goes through a single versioned
`persist<T>(key, version, migrate)` helper modelled on `settings-store.ts`
(localStorage, debounced 200 ms), per `playground-2.0-rewrite-plan.md` §3.8.
5. **Probe parity.** Each feature activates its currently-inert `window.__nisps` methods
(`manifold/src/debug/probe.ts`) so Playwright can drive it headlessly. New e2e specs are part
of each feature's acceptance criteria.
---
## 1. Session presets (composed layers)
### 1.1 Model
Adopt the composed-layers model prescribed by `playground-2.0-rewrite-plan.md` §3.8 rather than
the playground's monolithic `SessionPresetPayload`. A preset is a bundle of independently optional
layers:
```ts
interface SessionPresetV1 {
v: 1;
id: string; // `preset-${epoch36}`
name: string;
createdAt: number;
layers: {
control?: ControlLayer; // axes + offsets + presetId (§5)
pipelines?: PipelineLayer; // input/output pipeline params
routing?: RoutingLayer; // per-output MFParam state/min/max/curve/mute + backend specs
mode?: ModeLayer; // modeId + outputMode
weights?: string; // base64url Float32Array — OPT-IN only
};
}
```
- `ControlLayer` = `{ boldness, memory, precision, presetId, offsets }` exactly as §5.4 state.
- `PipelineLayer` = the configurable params of `engine/input-pipeline.ts` and
`output-pipeline.ts` (deadzone, zoom, invert, curve, smoothing, momentum; global curve,
smoothing, slew, freeze).
- `RoutingLayer` = serialised `MFParam[]` control fields + the per-backend specs already handled
by the `OutputsBackendConfig` preset bar. The existing backend preset bar becomes a *view onto
the routing layer* — one storage model, two entry points (see §1.4).
- `weights` uses the playground's base64url `Float32Array` codec verbatim
(`session-preset.ts:63–98`): standard base64 with `+/` → `-_`, padding stripped; byte length
rounded down to a 4-byte boundary on decode; size-mismatch on restore is silently skipped.
Restore is best-effort and layer-wise: absent layers leave current state untouched; unknown keys
inside a layer are ignored (forward compatibility).
### 1.2 Persistence
- Key: `mf-session`, via the shared `persist<T>` helper, version field `v: 1`, 200 ms debounce.
- Stored shape: `{ v: 1, presets: SessionPresetV1[], regionPins: RegionPin[] }` (pins ride in
the same document, §2).
- Weights are **excluded by default** on save; the save UI offers an explicit
"include network weights" toggle (default off). Rationale: weight blobs are ~KB-scale ×
N presets and stale weights surprise users; matches playground `withWeights=false` default.
### 1.3 URL sharing
Carry over both playground URL forms (`session-preset.ts:249–283`):
- **Compact axis form** — `?boldness=0.5&memory=0.5&precision=0.3`: applies only the control
layer; each value clamped to [0,1].
- **Full form** — `?session=<base64url(JSON of SessionPresetV1.layers)>`: applied on first load,
after engine ready, before first render of the dock. Never include weights in a generated
share URL (URL length); the share button produces the full form minus `weights`.
URL params are read once at boot in `App.tsx` (same place the `?debug=1` probe is installed) and
then stripped from the address bar via `history.replaceState` so a reload doesn't re-apply over
user changes.
### 1.4 UI
- **Save/restore/rename/delete** lives in a preset bar at the top of the **route drawer's
expanded depth**, reusing the exact interaction pattern of
`dock/OutputsBackendConfig.tsx` (save-as / restore / rename / delete). The existing
backend-scoped preset bar remains; a session preset is the superset bundle.
- **Share** button beside the bar copies the URL to the clipboard and flashes confirmation.
- Restoring a preset that contains a `mode` layer while a different mode is active shows the
same confirm modal used for net-reset (input-layer reshape) — mode switches discard training
state and must not be silent.
### 1.5 Probe + acceptance
- Probe: `saveSessionPreset(name, withWeights?)`, `loadSessionPreset(id)`, `listSessionPresets()`,
`buildShareUrl()`, `applySession(json)`.
- e2e: save → mutate axes/routing → restore → assert state round-trips; share URL → fresh page
load with `?session=` → assert layers applied; weights round-trip when opted in
(`getWeights()` equality within exact bits).
---
## 2. Region & param pins
### 2.1 Region pins
**Type (unchanged from playground `session-store.ts:34–44`):**
```ts
interface RegionPin {
id: string; // `pin-${epoch36}-${rand}`
x: number; y: number; // bottom-left corner, [0,1] input space
width: number; height: number; // [0,1]
colourSlot: number; // 0..4 → --pin-1..--pin-5 tokens
createdAt: number;
}
```
- **Cap: 5 pins, FIFO** (oldest evicted). Colour slot defaults to `pins.length % 5`.
- **Colour tokens** join `manifold/src/styles/tokens/colors.css` with the playground values:
fills `--pin-1..5` at 0.25 alpha (orange `255,106,0` / cyan `0,204,255` / purple
`180,100,255` / green `80,200,120` / tan `255,200,80`), borders same hues at 0.7 alpha,
2 px logical border width.
- **Creation gesture:** long-press (600 ms — the constant Manifold.tsx already uses) on the
Manifold stage pins the *currently visible region* (whole surface when unzoomed; the zoom
window once input zoom exists in manifold — until then, `{x:0,y:0,w:1,h:1}` is not useful, so
the long-press instead opens a small radial affordance: "pin region around cursor"
creating a 0.25×0.25 region centred on the press point, clamped to [0,1]²).
*Divergence from playground (which pins the zoom window) because manifold has no input zoom
yet; revisit when the inputs workstream lands zoom.*
- **Removal:** tap a pin border → small popover with colour dot + "remove"; "clear pins" action
in the settings drawer.
- **Rendering:** `Manifold.tsx` already accepts `pins: Pin[]` — extend the `Pin` type in
`console/types.ts` to the full `RegionPin` shape (it currently has only `x/y/colour`), and
render rects behind trail/markers, in front of the grid. Long-press must not fire when the
press initiates a drag > 8 px (existing drag threshold logic) or while `picking` is true
(explore-and-place anchor picking takes priority).
- **Persistence:** in the `mf-session` document (§1.2).
- **Semantics:** purely annotational in v1 (visual memory of "good areas"). They do **not**
constrain training. A future spec may bind snapshot anchors to pins.
### 2.2 Param pins = the existing tri-state (decision, not new code)
The playground's separate `ParamPin` list + `paramPinMask()` is **not ported**. Manifold already
has a strictly more expressive mechanism, per `dock-spec.md` §3.3:
- `off` — excluded from model control *and* pinned out of training;
- `fixed` — held at a static value, pinned out of training;
- `live` — model-driven, trainable.
"Pinning a param" in manifold = setting it to `fixed` (hold current value) via the existing
`OutputControlRow` segmented control or by clicking the value readout in `OutputStage`.
**Gap this spec closes:** the tri-state must actually feed the training mask. Requirement: the
arm/focus mask sent through `_nisps_ml_feedback_set_focus` (and any `moveWeights`-style
perturbation mask) is computed as `armed ∧ (state === 'live')` — `off`/`fixed` columns are never
perturbed and never accumulate gradient. Owner: `dock/output-state.ts` `buildArmMask()`.
Probe: `pinParam(i)` / `unpinParam(i)` map to `setParam(i, {state:'fixed'|'live'})`; the inert
playground-compat methods in `probe.ts` are rewired to these.
Acceptance: e2e asserts (a) a `fixed` output's value is bit-stable across training and jolt;
(b) region pins survive reload; (c) 6th region pin evicts the 1st.
---
## 3. Jolt
### 3.1 What it is
A held-gesture weight morph: while held, N randomly-chosen weights each glide (EMA) toward their
own random target, re-rolling the target on arrival — a continuous, directional "stirring" of
the network. On release the net freezes where it landed, and the learning rate ramps back up
from 0 so training doesn't immediately fight the new position. Ported from upstream
`InterfaceRL`; C++ implementation shipped in firmware (SLP workshop Part I).
### 3.2 Binding, not porting
Expose the existing `nisps::ml::Jolt` through the C ABI. New bindings in
`nisps/wasm/bindings.cpp` + `scripts/build-wasm.sh` EXPORTED_FUNCS:
```
_nisps_ml_jolt_press(ml) // picks indices+targets from the MLP's weight count
_nisps_ml_jolt_step(ml) // one tick: mutates weights in place, then caller reprocesses
_nisps_ml_jolt_release(ml)
_nisps_ml_jolt_active(ml) -> i32
_nisps_ml_jolt_lr_scale(ml) -> f32 // 0 while held; ramps 0→1 after release
_nisps_ml_jolt_tick_lr_ramp(ml)
_nisps_ml_jolt_seed(ml, u64)
```
The Jolt instance lives beside the MLP handle inside the WASM module (one per net), seeded from
the session seed so runs are reproducible. Parity CI: extend `tests/cpp/parity_check.cpp` with a
golden jolt sequence (seed → press → k steps → weight vector) asserted native-vs-WASM.
### 3.3 Constants (defaults — from `nisps/ml/jolt.hpp` `JoltParams`, upstream-verbatim)
| Constant | Value | Meaning |
|---|---|---|
| `num_weights` | 40 | weights morphed simultaneously (hard cap 64) |
| `morph_rate` | 0.017 | EMA per tick: `w += 0.017·(target − w)` |
| `target_min / max` | −1.2 / 0.9 | random target range |
| `target_epsilon` | 0.05 | re-roll target when within this distance |
| `lr_ramp_step` | 0.001 | post-release LR ramp per tick (~5 s at 200 Hz) |
| tick period | 5 ms (200 Hz) | `JOLT_TICK_MS`, matches firmware control rate |
### 3.4 Manifold integration
- **Driver:** a `setInterval(5 ms)` in `FeedbackController` while active: `jolt_step` →
`engine.reprocess()` (the spine already re-ticks the last input after weight changes) → bump
version. LR suppression: training calls multiply their LR by `jolt_lr_scale()`; the ramp is
ticked from the same driver until it reaches 1.0.
- **Snapshot before press:** entering jolt pushes an auto-snapshot (tag `before jolt`) so undo
recovers the pre-jolt net — same behaviour as playground `mode-runtime.ts:551`.
- **UI:** a press-and-hold **⚡ Jolt** button in the **learn drawer** (condensed depth, beside
the VerdictCluster-adjacent controls) and keyboard hold `J`. Visual: the Manifold noise ring
pulses while active. Pointer-cancel/blur must release (never a stuck jolt).
- **Interlocks:** disabled while `exploring` (explore-and-place owns the net during a scratchpad
session); `off`/`fixed` outputs are unaffected by definition (§2.2 mask applies only to
training — jolt perturbs *weights*, so v1 explicitly documents that jolt ignores per-output
pins; a per-column jolt mask is out of scope).
### 3.5 Probe + acceptance
Probe: `joltPress()`, `joltRelease()`, `joltActive()`. e2e: press → 200 ms → assert outputs
changed and `getWeights()` differs; release → assert weights stable; undo restores pre-jolt
weights; determinism: same seed + same step count ⇒ identical weight vector twice.
---
## 4. OU-explore noise
### 4.1 What it is
Ornstein–Uhlenbeck drift added to the *output vector* (not the weights): each output channel
carries a state that takes small random steps while being pulled back toward zero, producing
smooth, temporally-correlated wander around whatever the net currently outputs. Learning stays
live throughout — it is an audition aid, not a training mode. C++ `OUNoise<N>` shipped in
firmware (SLP workshop Part I).
Discrete update per channel (Euler–Maruyama, μ = 0):
```
state += theta · (−state) · dt + noise_scale · N(0,1)
out = clamp(out + state, 0, 1)
noise_scale = stationary_std · √(2·theta·dt), stationary_std = intensity · 0.65
```
### 4.2 Binding
Instantiate `nisps::ml::OUNoise<126>` (the WASM output width) beside the MLP handle. Bindings:
```
_nisps_ml_ou_set_intensity(ml, f32) // [0,1] → stationary_std = v·0.65; 0 disables
_nisps_ml_ou_intensity(ml) -> f32
_nisps_ml_ou_apply(ml, out_ptr, n) // advance state + add + clamp, in place
_nisps_ml_ou_reset(ml)
_nisps_ml_ou_set_theta(ml, f32) _nisps_ml_ou_set_dt(ml, f32)
_nisps_ml_ou_seed(ml, u64)
```
### 4.3 Constants (defaults — `nisps/ml/ou_noise.hpp`, upstream-verbatim)
| Constant | Value |
|---|---|
| `theta` (mean reversion) | 0.02 |
| `dt` | 0.001 |
| `kOUMaxAmplitude` (full-scale stationary std) | 0.65 |
| default intensity | 0 (**inert** — `apply()` is a no-op, preserving current behaviour/parity) |
| idle tick period | 30 ms (`EXPLORE_TICK_MS`) |
### 4.4 Manifold integration
- **Where in the chain:** in the spine, after the TS output-pipeline (curve→smoothing→slew→
freeze) and before the backend send — i.e. `ou_apply` on the routed buffer. This matches the
playground ordering (post-pipeline, pre-overrides). `fixed`/`off` outputs are re-asserted
*after* `ou_apply` so pinned values do not wander (this is the §2.2 guarantee).
- **Idle ticking:** when intensity > 0 and no input events arrive, a 30 ms interval calls
`reprocess()` so the wander is audible with a static joystick. Interval torn down and
`ou_reset()` called when intensity returns to 0.
- **UI:** an **Explore** slider (0–1) in the **learn drawer** condensed depth, next to the
existing Noise/Spread knobs; the Manifold noise ring radius reflects
`noiseCap + ou_intensity·0.65` so the visual language stays truthful.
- **Interaction with Jolt:** independent and composable (jolt stirs weights, OU wanders
outputs); both default inert.
### 4.5 Probe + acceptance
Probe: `setExploreIntensity(v)`, `getExploreIntensity()`. e2e: intensity 0.5 + static input ⇒
outputs vary over 500 ms with bounded step size; intensity 0 ⇒ outputs bit-stable; `fixed`
output does not wander; determinism under seeded runs.
---
## 5. Boldness / Memory / Precision control surface
### 5.1 Concept
Three compound axes, each a `[0,1]` fader that drives ~5–6 underlying parameters through a
piecewise-linear lookup table, with per-param trim-pot offsets on top. This is the "Feel"
surface from `playground-2.0-rewrite-plan.md` §2.3, ported with the playground's tables as the
**transparent defaults** — reproduced in full below so the mapping is reviewable here, not
buried in code.
### 5.2 Axis tables (defaults — `playground/src/stores/control-store.ts:38–54`, verbatim)
Numeric params interpolate linearly between breakpoints; discrete params snap to the upper row
at t ≥ 0.75.
**Boldness** — "how hard do moves hit"
| axis | zoom | noiseCap | noiseGrowth | learningRate | weightDecay | noiseDistribution |
|---|---|---|---|---|---|---|
| 0.0 | 0.1 | 0.02 | 1.1 | 0.1 | 0.15 | gaussian |
| 0.5 | 0.5 | 0.12 | 1.5 | 1.0 | 0.06 | gaussian |
| 1.0 | 1.0 | 0.30 | 2.5 | 3.0 | 0.00 | cauchy |
**Memory** — "how much does the net remember"
| axis | maxExamples | exampleDecay | memoryWeightDecay | noiseDecay | convergenceThreshold |
|---|---|---|---|---|---|
| 0.0 | 5 | 0.3 | 0.20 | 0.85 | 1e-3 |
| 0.5 | 50 | 0.7 | 0.06 | 0.97 | 1e-5 |
| 1.0 | 500 | 1.0 | 0.00 | 0.995 | 1e-8 |
**Precision** — "how surgical is the gesture"
| axis | inputCurve | deadzone | smoothing | slewRate | momentumZoom |
|---|---|---|---|---|---|
| 0.0 | 1.0 | 0.00 | 0.00 | 1.0 | off |
| 0.5 | 1.5 | 0.05 | 0.15 | 0.3 | off |
| 1.0 | 3.0 | 0.15 | 0.40 | 0.1 | off |
Routing targets in manifold: `zoom/deadzone/inputCurve/smoothing/momentumZoom` →
`engine/input-pipeline` config; `slewRate` → `engine/output-pipeline`; `noiseCap/noiseGrowth/
noiseDecay/learningRate/weightDecay` → `FeedbackController` + training params;
`maxExamples/exampleDecay/convergenceThreshold` → `engine/dataset.ts` + training loop.
`noiseDistribution` and `momentumZoom` are carried in the schema but may no-op until their
consumers exist in manifold (each no-op must be logged once at boot — no silent dead params).
`spread` is deliberately **not** axis-driven (stays an independent lab knob, as in playground).
### 5.3 Presets (defaults — verbatim)
| id | label | boldness | memory | precision |
|---|---|---|---|---|
| `default` | Default | 0.5 | 0.5 | 0.3 |
| `first-touch` | First Touch | 0.2 | 0.7 | 0.6 |
| `jazz-hands` | Jazz Hands | 0.8 | 0.2 | 0.0 |
| `sculptor` | Sculptor | 0.3 | 0.9 | 0.8 |
| `improviser` | Improviser | 0.6 | 0.3 | 0.2 |
| `microscope` | Microscope | 0.1 | 1.0 | 1.0 |
Boot default = the `default` preset (0.5 / 0.5 / 0.3). **This replaces ConsoleApp's current
ad-hoc `{0.55, 0.4, 0.5}` initial state.** Moving any axis manually clears `presetId` (chips
show "custom").
### 5.4 State + trim-pot offsets
```ts
interface ControlState {
boldness: number; memory: number; precision: number; // [0,1]
offsets: Record<'boldness'|'memory'|'precision', Record<string, number>>;
presetId: string | null;
}
```
Resolution: interpolate all three tables, merge, then add each axis's offsets to numeric params
(`resolved[k] = table[k] + offset[k]`), clamp to each param's legal range. Offsets are created
when the user adjusts a *derived* param directly (§5.6) — the delta from the current table value
becomes the offset ("trim-pot"). Double-tap on a derived param's readout clears its offset
("re-link"); a per-axis "re-link all" clears the axis's offset map. A dot marks any param with a
non-zero offset.
### 5.5 Fanout (architecture requirement)
Per `playground-2.0-rewrite-plan.md` §3.2: resolution is a memoised derivation per target param — in
React terms one `useMemo` producing the resolved record + an effect per target store that
writes only on change. The playground's `JSON.stringify`-signature-inside-effect pattern
(`control-routing.ts:34`) is explicitly **not** ported. Resolution runs off the render cycle for
engine targets (direct setter calls on the engine config objects), with React state only for
what the UI displays.
### 5.6 UI — transparency is the feature
- **Placement:** the **learn drawer** gains a top "Feel" section (this keeps manifold at five
drawers; if the section crowds the condensed depth, promoting Feel to its own drawer is a
pre-approved fallback — note it in the implementation PR).
- *Condensed:* three vertical `ControlAxis` faders (the existing primitive, finally wired) +
six preset chips.
- *Expanded:* under each fader, the live list of its derived params — name, resolved value
(table + offset), and a mini trim-pot. This satisfies the "reasonable and transparent
defaults" requirement: the mapping is always visible in the UI, not just in this document.
- Axis moves are continuous (no apply button); param writes are rate-limited to one per frame.
- The existing `preset = 'Sculpt'` placeholder state in ConsoleApp is removed in favour of
`presetId`.
### 5.7 Persistence, probe, acceptance
- Persisted as the `control` slice of `mf-session` (also the `control` layer of session
presets, §1 — one shape, two containers).
- Probe: `setAxis(name, v)`, `getAxes()`, `resolveControlParams()`, `applyControlPreset(id)`,
`setAxisOffset(axis, param, v)`.
- e2e: boldness 0→1 sweeps noiseCap 0.02→0.30 and zoom 0.1→1.0 (assert via
`resolveControlParams()` + engine config readback); preset chip sets all three axes; manual
axis move clears presetId; offset survives axis movement (offset is additive, not absolute);
double-tap re-links; state round-trips through reload and through a session preset.
---
## 6. Cross-cutting
### 6.1 Build & sequencing (proposed ergo breakdown, post-sign-off)
1. **W1 — WASM bindings**: jolt + OU C ABI, EXPORTED_FUNCS, parity golden tests. (Blocks W4, W5.)
2. **W2 — `persist<T>` helper + `mf-session` document** (control slice, pins, presets scaffold).
3. **W3 — Control surface**: state, tables, fanout, Feel section, presets, offsets. (Needs W2.)
4. **W4 — Jolt + OU in FeedbackController + learn drawer UI.** (Needs W1.)
5. **W5 — Region pins** (tokens, gesture, rendering, popover) + tri-state training-mask
guarantee (§2.2). (Needs W2.)
6. **W6 — Session presets**: layers, preset bar, URL sharing, probe. (Needs W2, W3.)
Each lands with its e2e spec; smoke suite additions run in the same CI gate as the existing
`tests/e2e/smoke.spec.ts`.
### 6.2 Out of scope (explicit)
- Snapshot-with-weights / snapshot DAG, A/B compare (separate spec — the biggest remaining
parity gap, but architecturally entangled with the C++ feedback snapshot lifecycle).
- 2D heatmap sampler, weight health, gradient flow (blocked on loss-history C API plumbing).
- Input zoom (inputs workstream) — noted where it interacts with region pins (§2.1).
- Per-column jolt masking (§3.4).
### 6.3 Open questions for review
1. §2.1 creation gesture: is the 0.25×0.25 press-centred region an acceptable stand-in until
input zoom lands, or should region pins wait for the zoom feature?
2. §5.6: Feel section inside the learn drawer vs a sixth drawer — reviewer's call if condensed
depth gets crowded.
3. §1.1 RoutingLayer: should backend connection details (OSC URL, MIDI port id) be included in
session presets, or excluded as machine-specific? Proposal: exclude port/URL, include
per-output mappings.
4. Memory-axis params (`maxExamples`, `exampleDecay`, `convergenceThreshold`): confirm the
manifold training loop actually consumes these; if any are dead in the current engine, they
land as logged no-ops (§5.2) with an ergo follow-up.
---
*Doc-sync rule applies: when implementation diverges from this spec, update the spec in the same
commit. When a workstream ships and is verified, migrate its section to ONBOARDING.md ("what
is") and prune it here.*

View file

@ -0,0 +1,665 @@
---
kind: plan
status: active
---
# Mode-Layer Reunification — the shared control tick (P5 item 5a)
*Dated 2026-07-21. Spec-first, per `simplification-plan.md` §6 ("each item spec-first, own
session"). Closes `ALIGNMENT.md` defect 1; audit findings A1 + A5
(`../recon/simplification-audit-2026-07.md`). **This document authorises no code.** It fixes
the seam, the invariants, the sequence and the verification so an implementation session can
start cold. §8 lists the decisions that must come from the operator before step 3 lands.*
**Every code claim below was re-read against the working tree on 2026-07-21** (`fd0aee2` plus
the uncommitted in-flight work of parallel sessions — `nisps/modes/base.hpp`,
`nisps/wasm/bindings.cpp` and several `manifold/src/engine/` files are all dirty relative to
HEAD, so line numbers are working-tree numbers and may shift by a few lines once those land).
Where a cited document (audit, plan, ALIGNMENT) disagreed with the code, the code won and the
disagreement is called out inline.
Sibling P5 specs written in the same session: `browser-mode-coverage-spec.md` (5b),
`curated-presets-spec.md` (5c), `hardware-editor-spec.md` (5d). This one is 5a.
---
## 0. The gap, stated precisely
`nisps/modes/` compiles into exactly one binary: the firmware. Its consumers are
`firmware/MEMLNaut-NISPS/glue/{mode_select,midi_io,output_router,peripherals,settings_view}.hpp`
and the host tests in `tests/cpp/test_mode_*.cpp`. `nisps/wasm/bindings.cpp` includes eight
engine headers (lines 54–62) and six `nisps/ml/` headers (65–72) and **zero** mode headers —
verified by `grep -n modes nisps/wasm/bindings.cpp`, whose only hit is the word "modes" inside
a comment about `schemas/modes/*.json` (line 90).
The consequence is not that the browser lacks modes. It is that the **control tick** —
`ModeBase::tick_control()`, `nisps/modes/base.hpp:198–234` — exists twice: once in C++ for the
firmware, once hand-assembled in TypeScript for the browser out of `Spine.setInputs`
(`manifold/src/engine/spine.ts:254–303`) plus `ExplorationController`'s two timers
(`manifold/src/engine/exploration.ts:40–42, 57, 84–120`). The primitives underneath (MLP, Jolt,
OU, feedback, pipelines, curves) are already shared; the **orchestration** is not.
That is the whole defect. Everything in this document serves one outcome: *the control tick is
written once, in C++, and both targets call it.*
---
## 1. Verified ground truth
### 1.1 What `ModeBase` actually does today
`nisps/modes/base.hpp`, `ModeBase<Derived, EngineT, MLPType, NInputs>`:
| Responsibility | Where | Notes |
|---|---|---|
| Own the engine, the MLP, the input channels | members, `:322–341` | `input_channels_`, `input_pinned_`, `pin_value_`, `out_buf_`, `jolt_buf_` |
| `setup(sr)` — engine setup, **all channels to 0.5f**, one inference, route to engine, `on_setup` hook | `:126–142` | |
| `set_input(idx, v)` — bounds-drop + clamp to [0,1] | `:165–170` | |
| Input pinning (single/double-joystick neutralise) | `:179–196` | `set_input_pinned` / `set_pin_value` / `effective_input`; driven by `glue/settings_view.hpp` |
| `tick_control()` — the control tick | `:198–234` | order: `on_pre_inference` → jolt step (if active) → `jolt_.tick_lr_ramp()` → forward pinned channels → `ml_.process()` → OU on a copy (if enabled) → `engine_.set_params()` → `on_post_inference` |
| `apply_jolt_()` — get→glide→set of the flat weights | `:312–320` | out-of-line by design |
| Jolt / OU gesture surface | `:241–255` | `jolt_press/release/active/lr_scale`, `set_explore_intensity`, `jolt()`, `ou_noise()` |
| `driver_config()` | `:157–163` | firmware-only concept (codec / mic vs line) |
| Voice-space selection | `:279–290` | delegates to the engine, re-applies params |
| `ControlEvent` ring (`push`/`pop_control_events`) | `:294–305` | drained by `glue/midi_io.hpp:69–98` |
| `process(stereosample_t)` | `:257–259` | pure engine forward, RT |
Firmware drives it from two cores: `tick_control()` on core 0 every 5 ms
(`firmware/MEMLNaut-NISPS/src/main.cpp:154–160`, `ML_INFERENCE_PERIOD_US 5000`), `process()` on
core 1 from the audio callback, and `drain_outputs()` on core 1 at ~1 kHz
(`main.cpp:208–212`).
### 1.2 What the browser does instead
`Spine.setInputs` (`manifold/src/engine/spine.ts:254–303`) is the browser's control tick:
1. `iml.processInput(x, y, dt)` — the 2-D `InputChain` (**no firmware counterpart**, see 1.3).
2. `iml.setInput(i, v)` for every axis → `nisps_ml_set_input` → `MLPCore::set_input`.
3. `iml.processInto(mlBuf)` → `nisps_ml_process`.
4. `iml.processOutput(routedBuf, dt)` — the `OutputChain`.
5. `this.outputMorph(routedBuf)` — the OU walk, registered by
`ExplorationController`'s constructor (`exploration.ts:57`) → `nisps_ml_explore_apply`.
6. `backendSend(routedBuf)` → `EngineHost.setParams` → `postMessage` → the **worklet's own
WASM instance** → `nisps_engine_set_params`.
Jolt is driven by a separate 5 ms `setInterval` (`exploration.ts:40, 84–91`) calling
`nisps_ml_jolt_step` + `engine.process()`; the OU walk gets a second 30 ms interval
(`exploration.ts:42, 107–113`).
**Three WASM instances exist**, not one: main thread (`WasmIML` — MLP, feedback, pipelines),
AudioWorklet (`worklet/nisps-processor.ts` — engine only, four exports), and an on-demand
training worker (`wasm-worker.ts` — a mirror MLP). `manifold/ONBOARDING.md` §3 says "loaded
TWICE"; the training worker is a third. This is why a monolithic mode object cannot be bound
(see §2).
### 1.3 The divergences this hand-mirroring has produced
Each row is a fact about the current tree, verified today.
| # | Divergence | Evidence |
|---|---|---|
| D1 | **`nisps_ml_jolt_step` is a line-for-line duplicate of `ModeBase::apply_jolt_`** | `bindings.cpp:842–849` vs `base.hpp:312–320` — same get→copy→`jolt.step`→set sequence, two copies of the code |
| D2 | **Jolt/OU RNG salts differ between targets** | `base.hpp:122–123` uses `0x91E10C5E` / `0x0CEA0FF5`; `bindings.cpp:101–102` uses `0xB01DFACE` / `0x0DDBA11`. The bindings comment claims it "mirrors the firmware ModeBase seeding" — it mirrors the *practice*, not the values. Same seed ⇒ different jolt/OU streams. (`kFeedbackSalt` *is* matched across `bindings.cpp:98` and `glue/peripherals.hpp:58`.) |
| D3 | **OU is applied at a different point in the chain.** Firmware: ML → OU → engine (`base.hpp:216–229`). Browser: ML → OutputChain → OU (`spine.ts:292, 296`). | Consequence: in the browser the per-output **freeze mask holds a value that the OU walk then moves again** — `OutputChain::process` writes `prev_[i]` for frozen dims (`output_chain.hpp:116–119`) and `nisps_ml_explore_apply` perturbs every entry up to `n_out` (`bindings.cpp:880–887`). Freeze is defeated whenever Explore > 0. |
| D4 | **Input clamping and pinning do not exist in the browser.** `nisps_ml_set_input` bounds-checks the index only (`bindings.cpp:434–441`); `MLPCore::set_input` does not clamp (`mlp.hpp:119–122`). No browser path calls anything equivalent to `set_input_pinned`. | The single/double-joystick neutralise is a firmware-only feature today (`glue/settings_view.hpp`), although the operator decision recorded in memory covers "firmware + manifold input toggle" |
| D5 | **Initial state differs.** Firmware `setup()` sets every channel to `0.5f` before its first inference (`base.hpp:129`). The browser's boot inference comes from `ConsoleApp`'s mount effect calling `engine.setInput(0.5, 0.5)` (`ConsoleApp.tsx:268`), and `Spine.setInputs` zero-fills axes 2..N (`spine.ts:277–281`). | For any mode with >2 inputs the two targets boot from different points in input space |
| D6 | **Stateful gestures advance per *call*, not per unit of time.** `Jolt::morph_rate` is documented "EMA per tick, ~1s @200Hz" (`jolt.hpp:44`); `Jolt::tick_lr_ramp` adds a fixed `0.001` per call (`jolt.hpp:107–111`); `OUNoise` uses a fixed internal `dt_ = 0.001f`, *not* the caller's dt (`ou_noise.hpp:74, 96`). Firmware calls the tick at exactly 200 Hz. The browser advances the OU walk on **every spine tick** — i.e. faster when the user moves the pad faster — plus a 33 Hz idle driver. | The same Explore setting produces materially different roam speeds on the two targets, and a browser-side rate that depends on user gesture speed |
| D7 | **`jolt_lr_scale()` has zero consumers on either target.** | `grep -rn 'jolt_lr_scale\|lr_scale' nisps/ firmware/ manifold/src tests/ vcv/src` → only `base.hpp:246`, `jolt.hpp:65`, and tests. The firmware's two `train()` call sites (`glue/peripherals.hpp:89, 225`) do not scale the LR. The documented "training eases back in after a jolt" behaviour is **not wired anywhere**. |
| D8 | **`nisps/pipeline/` is WASM-only.** `InputChain`/`OutputChain` are included by `bindings.cpp:76–77` and `tests/cpp/parity_check.cpp:67–68` and by nothing else. | Firmware has no input conditioning and no output smoothing/slew/freeze. (`simplification-plan` §6.5d already assigns this wiring to the hardware-editor item, L29.) |
D1, D2, D6 are what this document fixes. D3, D4, D5 are fixed as a side effect and are
behaviour changes that need the operator's eyes (§5, §8). D7 and D8 are named here so an
implementer does not "discover" them mid-flight and widen scope; both are out of scope (§9).
---
## 2. What this design deliberately does **not** do
- **It does not bind mode objects into WASM.** A `ModeBase` instance owns an MLP *and* an
engine. In the browser those live in different threads and different WASM instances
(§1.2): the MLP on the main thread, the engine in the AudioWorklet, joined by a
`postMessage` of a `Float32Array` (`engine-host.ts:162–168`). Binding a mode object would
require either running the engine on the main thread (kills the RT guarantee) or the MLP in
the worklet (kills training and the whole feedback surface). The two-instance split is a
locked decision from the one-core refactor; this design respects it.
- **It does not delete the per-primitive C API.** `nisps_ml_*` stays exactly as it is —
`simplification-plan` §6.5a and audit A5 both say so explicitly, and the parity harness,
the training worker, the feedback controller and the probe all consume it.
- **It does not move the audio engine, the voice-space surface, the `ControlEvent` ring, or
`driver_config()`.** Those are the *other* half of a mode and belong to items 5b (browser
mode coverage / audio topology) and 5d (hardware editor). Mixing them in is how this becomes
a month.
- **It does not touch `nisps/pipeline/`'s ownership.** The input chain stays a caller-side
pre-stage and the output chain stays a caller-side post-stage, both on the existing
`nisps_pipeline_*` handle. Only their *ordering relative to OU* changes (§5).
---
## 3. Invariants
The repo-wide constraints (`CLAUDE.md`) are the floor. These are the ones specific to this area:
1. **`nisps/` stays platform-neutral and heap-free.** The new shared code lives under
`nisps/modes/`, and its fixed storage policy must be `std::array`-only. The dynamic policy is
a second deliberate exception to the no-heap rule and must carry the same
`#if defined(NISPS_TARGET_EMBEDDED) → #error` guard as `nisps/ml/dynamic_storage.hpp:22–24`,
and must be added to **both** the allowlist branch (`scripts/lint-cpp.sh:146`) and the
guard-survival check (`:194–198`). Today that script hardcodes exactly one path; generalise
it to a list rather than adding a second special case.
2. **Firmware behaviour is bit-identical at the current cadence.** With the mode ticked at
5 ms, every mode's audio output must be unchanged. `tests/cpp/test_mode_learning.cpp`'s
inert-parity test and `test_mode_paf_synth.cpp`'s reference-vector tests are the existing
guards; step 1 of §6 adds a stronger one.
3. **The `nisps::Mode` concept keeps its current shape** (`nisps/core/concepts.hpp:70–88`).
`tick_control()` must remain callable with no arguments, or every mode `static_assert` and
all of `firmware/glue/` breaks.
4. **The control tick stays bounded and allocation-free.** It runs on core 0 on firmware and on
the main thread in the browser; neither may block. Any catch-up loop introduced for
rate-independence (§4.3) must be hard-capped.
5. **Dual-core ownership and SPSC discipline are preserved.** `engine_.set_params()` is written
from core 0 while `engine_.process()` runs on core 1 — that is the existing arrangement and
this change must not add a second writer. The `ControlEvent` ring has exactly one producer
per mode (core 0 for the CC-emitting modes via `on_post_inference`, core 1 for the sequencer
modes via `pump_engine_events`); the ring must not move into shared code where a mode could
acquire two producers.
6. **Deterministic per-instance RNG.** Jolt and OU keep their own `nisps::Rng` streams,
independent of the MLP's and the feedback controller's. After this change the *salts must
match across targets* (D2) — that is the point.
7. **Native↔WASM parity ≤1e-5 must cover the new code.** The existing harness does not
(§7). Extending it is part of the work, not a follow-up.
8. **A WASM C-API addition walks all five layers**: `EMSCRIPTEN_KEEPALIVE` →
`EXPORTED_FUNCS` in `scripts/build-wasm.sh:40–72` → `NispsModule` in
`manifold/src/engine/types.ts` → `WasmIML` → `EngineApi`. A missed layer fails at runtime.
---
## 4. The design
### 4.1 The seam: `ControlCore<CtlStorage>`
A new header `nisps/modes/control_core.hpp` defining `nisps::ControlCore<CtlStorage>` — the
control tick and nothing else. It is storage-policied exactly the way `MLPCore<Storage>`
(`nisps/ml/mlp.hpp:91–92`) and `FeedbackControllerCore<FbStorage>` (`nisps/ml/feedback.hpp:42–51`)
already are, including the `ControlCore : public CtlStorage` + forwarded-`StorageArgs`
constructor idiom (`mlp.hpp:103–106`).
It **owns**: the input channels, the pin mask and pin value, a `ml::Jolt`, the OU walk and its
state, the produced parameter buffer, and the fixed-rate accumulator.
It **does not own** the MLP. Like `FeedbackControllerCore`, every method that needs the network
takes it by reference as a method-level template, so a `MLPCore<FixedStorage>` and a
`MLPCore<DynamicStorage>` both work:
```cpp
namespace nisps {
template <typename CtlStorage>
class ControlCore : public CtlStorage {
public:
template <typename... StorageArgs>
explicit ControlCore(std::uint64_t seed, StorageArgs&&... args) noexcept;
bool valid() const noexcept; // dynamic storage may fail to allocate
// ---- input channels (was ModeBase :165–196) ----
void set_input(std::size_t idx, float v) noexcept; // bounds-drop + clamp [0,1]
void set_input_pinned(std::size_t idx, bool p) noexcept;
bool is_input_pinned(std::size_t idx) const noexcept;
void set_pin_value(float v) noexcept;
float pin_value() const noexcept;
float effective_input(std::size_t i) const noexcept;
std::span<const float> input_channels() const noexcept;
std::span<float> mutable_input_channels() noexcept; // SoundAnalysisMIDI's hook
// ---- gestures (was ModeBase :241–255) ----
void jolt_press(std::size_t weight_count) noexcept;
void jolt_release() noexcept;
bool jolt_active() const noexcept;
float jolt_lr_scale() const noexcept;
void set_explore_intensity(float level) noexcept;
float explore_intensity() const noexcept;
// ---- the tick ----
// Runs `n` fixed-rate control steps (n derived from dt_s, see 4.3), one
// inference, and the OU walk. Returns the produced parameter vector — a
// STABLE span into this object's own buffer, valid until the next reshape.
template <typename MLPT>
std::span<const float> tick(MLPT& ml, float dt_s) noexcept;
std::span<const float> params() const noexcept; // same span, without ticking
void reset() noexcept; // clears OU state + accumulator
};
} // namespace nisps
```
### 4.2 Storage policies
Surface (documented in the header the way `nisps/ml/storage.hpp:20–28` documents the MLP's):
```
dims: n_in(), n_out()
buffers: input_channels(), input_pinned(), out_buf(), ou_state()
validity: valid()
```
- `nisps/modes/control_storage.hpp` — `FixedControlStorage<NIn, NOut>`: four `std::array`
members, all dims `constexpr`-foldable, zero heap. Firmware + host tests.
- `nisps/modes/dynamic_control_storage.hpp` — `DynamicControlStorage`: dims at construction,
**one arena allocation in the constructor and none after**, `#error` under
`NISPS_TARGET_EMBEDDED`. Browser only. Mirrors `nisps/ml/dynamic_storage.hpp` file-for-file
in shape so the lint allowlist reads as one pattern rather than two exceptions.
`FixedControlStorage`'s buffers are ~`2*NIn + 2*NOut` floats — for MEMLCelium (4 in, 56 out)
about 480 bytes.
**`jolt_buf_` is deleted, not moved.** `ModeBase::jolt_buf_` is a
`std::array<float, MLPType::weight_count()>` (`base.hpp:341`) — **6,152 bytes of SRAM on every
MEMLCelium-class mode** (`MLP<4,10,14,18,56>` ⇒ 1,538 weights). It exists only to hold a copy of
the flat weights across `jolt_.step()`. But `MLPCore`'s storage already owns exactly such a
buffer: `flat_buf()` (`nisps/ml/storage.hpp:164`, `nisps/ml/dynamic_storage.hpp:162`), which is
what `get_weights()` fills (`mlp.hpp:385–400`). Add a `std::span<float> weights_scratch()`
accessor to `MLPCore` and the jolt step becomes:
```cpp
const auto w = ml.get_weights(); // fills the storage's flat buffer
jolt_.step(ml.weights_scratch().first(w.size()));
ml.set_weights(ml.weights_scratch().first(w.size()));
```
`MLHandle::jolt_scratch` (`bindings.cpp:167`, a `std::vector<float>`) goes the same way.
**`OUNoise` loses its template parameter and splits `apply()` in two.**
`ml::OUNoise<N>` (`nisps/ml/ou_noise.hpp:44–99`) holds `std::array<float, N> state_` and its
`apply()` advances the walk and adds it to the output in one loop (`:68–81`). Two changes:
(i) the state span comes from the caller, so `OUNoise` becomes a plain class and `ControlCore`'s
storage owns the state; (ii) split into `advance(std::span<float> state)` and
`add_to(std::span<float> out, std::span<const float> state)`, because a catch-up tick must
advance N times and add once (§4.3a). Both keep the existing early-out at intensity 0, so the
inert path stays free and parity-safe. This also deletes the browser's
`OUNoise<kMaxDim>` — a 16 KB fixed array (`bindings.cpp:166`) sized for a net that is never
4096-wide. Three call sites: `base.hpp`, `bindings.cpp`, `tests/cpp/test_mlp_ou_noise.cpp`.
*Lower-risk fallback if this proves noisy: keep `OUNoise<N>` and have each storage policy expose
an `OU` typedef at its own N. It costs the browser its 16 KB and leaves a template where none is
needed, but it is strictly smaller.*
### 4.3 The canonical tick
```
tick(ml, dt_s):
steps = accumulate_steps(dt_s) # see below; firmware @5ms => exactly 1
for s in 0..steps-1:
if jolt.active(): jolt.step(ml's flat weight scratch) ; ml.set_weights(...)
jolt.tick_lr_ramp()
for i in 0..n_in-1: ml.set_input(i, effective_input(i))
ml.process()
copy ml.outputs() -> out_buf # ALWAYS, see below
for s in 0..steps-1: ou.advance(ou_state) # no-op while disabled
ou.add_to(out_buf, ou_state) # no-op while disabled
return out_buf
```
Four decisions are baked in here; each is a change from at least one target's current behaviour.
**(a) Fixed-rate stepping (fixes D6).** `ControlCore` owns a `float accum_` and a compile-time
`kControlPeriodS = 1.f / 200.f`. `accumulate_steps(dt_s)` adds `dt_s`, takes out whole periods,
and **caps the result at `kMaxCatchUpSteps` (8)** — invariant 4. At the firmware's exact 5 ms
cadence this yields exactly one step with zero residual, so firmware is unchanged; in the browser
a 30 ms driver call performs six steps and the walk becomes wall-clock-correct regardless of how
fast the user moves the pad. The rate constant, not wall-clock dt, stays the calibration for
`Jolt::morph_rate`, `Jolt::lr_ramp_step` and `OUNoise::dt_` — those are shape constants tuned at
200 Hz upstream. Re-deriving them from wall-clock dt is *not* equivalent: `OUNoise::dt_` is
`0.001`, not the firmware's 5 ms period, so feeding wall-clock dt would multiply firmware's
mean-reversion term by 5 and its per-step noise by √5.
**(b) Inference runs once per `tick()`, not once per control step.** Only the stateful walks are
rate-sensitive; a forward pass is idempotent given the same inputs and weights. This keeps a
catch-up tick cheap.
**(c) The output copy is unconditional.** Today firmware skips the copy when OU is disabled and
hands `ml_.outputs()` straight to the engine (`base.hpp:228`). If `tick()` did that, the returned
span would sometimes point into the MLP's output buffer and sometimes into `out_buf` — a
pointer-stability footgun for the C API, which caches the pointer JS-side. Always copying costs
`n_out` floats per tick (56 at 200 Hz for MEMLCelium: ~11 k float copies/s against a 1,538-weight
forward pass at the same rate) and is bit-identical.
**(d) OU is applied to the mode's parameter vector, before any caller-side output chain
(changes the browser — D3).** Firmware order is preserved; the browser's `OutputChain` moves from
*before* the OU walk to *after* it. Rationale: OU is part of what the mapping *produces*; the
output chain is how the *destination* conditions it. This is also what makes per-output freeze
mean something again. It is user-audible — see §5 and §8-Q1.
### 4.4 `ModeBase` afterwards
`ModeBase` keeps its entire public surface (invariant 3) and becomes a composition:
```cpp
NISPS_HOT void tick_control(float dt_s = kControlPeriodS) noexcept {
if constexpr (requires(Derived& d) { d.on_pre_inference(); })
static_cast<Derived&>(*this).on_pre_inference();
const auto params = control_.tick(ml_, dt_s);
if constexpr (kRouteOutputsToEngine) engine_.set_params(params);
if constexpr (requires(Derived& d) { d.on_post_inference(); })
static_cast<Derived&>(*this).on_post_inference();
}
```
with `control_` a `ControlCore<FixedControlStorage<NInputs, MLPType::kOutput>>`, and
`set_input` / `set_input_pinned` / `pin_value` / `input_channels` / `mutable_input_channels` /
`jolt_*` / `set_explore_intensity` / `explore_intensity` all one-line forwards. The default
argument preserves the concept and every existing call site (`main.cpp:157`,
`test_mode_*.cpp`).
What stays in `ModeBase`: the engine, the CRTP hooks, `setup()`, `driver_config()`,
`set_voice_space()`, the `ControlEvent` ring, `process()`. What leaves: the input-channel state,
`jolt_`, `ou_`, `out_buf_`, `jolt_buf_`, `apply_jolt_()`.
Two accessors need a call: `ml::Jolt& jolt()` (`base.hpp:247–248`) has **no consumer outside
`base.hpp`** — delete it. `ml::OUNoise& ou_noise()` (`:254–255`) is used by
`tests/cpp/test_mode_learning.cpp:89–103` — keep it (forwarding into `control_`) or rewrite those
three assertions against `explore_intensity()`; implementer's call.
The seeding fix (D2) rides along: `ControlCore`'s constructor takes the salts, and both targets
pass the same ones. Name them once, in `control_core.hpp`, as
`kJoltSalt = 0x91E10C5Eull` / `kOUSalt = 0x0CEA0FF5ull` (the firmware's current values, so
firmware weight streams are unchanged; the browser's jolt/OU streams change, which no committed
artifact or golden depends on — verified: no fixture, parity stage or e2e drives jolt/OU).
### 4.5 The browser afterwards
`MLHandle` (`bindings.cpp:148–185`) drops `jolt`, `ou` and `jolt_scratch` and gains one
`ControlCore<DynamicControlStorage>` member, constructed at `(d.n_in, d.n_out)` and rebuilt by
`nisps_ml_reshape` alongside the feedback controller (`bindings.cpp:396–421`).
The existing `nisps_ml_jolt_*` and `nisps_ml_explore_intensity` entries stay in place and
**forward into the head** — so the per-primitive C API survives (§2) while its *implementation*
stops being a second copy (kills D1).
`Spine.setInputs` becomes:
1. `iml.processInput(x, y, dt)` — unchanged.
2. `iml.setControlInput(i, v)` for each axis → `nisps_control_set_input` (clamped + pin-aware).
3. `iml.controlTick(dt)` → `nisps_control_tick` — replaces today's steps 3 *and* 5.
4. read the raw ML outputs (`nisps_ml_outputs`) into `mlBuf` for `liveOutputs`, and the head's
params (`nisps_control_params`) into `routedBuf`.
5. `iml.processOutput(routedBuf, dt)` — unchanged call, now *after* the OU walk.
6. `backendSend(routedBuf)` — unchanged.
`Spine.setOutputMorph` and the `outputMorph` field are **deleted** (`spine.ts:105–109, 221–229,
294–296`); its only consumer is `ExplorationController`'s constructor and `dispose`
(`exploration.ts:57, 131`). `ExplorationController` collapses from two timers to one ~60 Hz
driver that exists while `joltActive() || exploreIntensity() > 0` and simply calls
`engine.process()` — the fixed-rate accumulator makes the driver's exact period irrelevant.
### 4.6 The C API additions — the five-layer walk
| Export | `bindings.cpp` | `build-wasm.sh` `EXPORTED_FUNCS` | `types.ts` `NispsModule` | `WasmIML` | `EngineApi` |
|---|---|---|---|---|---|
| `void nisps_control_set_input(void* ml, int idx, float v)` | new | add | add | `setControlInput` | via `spine` |
| `void nisps_control_set_input_pinned(void* ml, int idx, int pinned)` | new | add | add | `setControlInputPinned` | `EngineApi.control.setInputPinned` |
| `void nisps_control_set_pin_value(void* ml, float v)` | new | add | add | `setControlPinValue` | `EngineApi.control.setPinValue` |
| `int nisps_control_tick(void* ml, float dt_s)` → n params | new | add | add | `controlTick` | via `spine` |
| `const float* nisps_control_params(void* ml)` | new | add | add | (internal) | — |
| `void nisps_ml_explore_apply(void* ml, float*, int)` | **delete** (`:880–887`) | **remove** (`:63`) | **remove** (`:133`) | **remove** `exploreApply` | **remove** `explore.exploreApply` |
Deleting `nisps_ml_explore_apply` is safe: its consumers are `WasmIML.exploreApply` →
`EngineApi.explore.exploreApply` → `exploration.ts:57`, all of which this change removes. It is
**not** touched by `tests/cpp/parity_wasm.mjs`, `manifold/tests/`, or `src/debug/probe.ts`
(verified by grep). Re-verify before deleting.
`nisps_ml_set_input` **stays**: `tests/cpp/parity_wasm.mjs` drives parity stage 1 through it, and
it is the raw-MLP write that `infer_batch` and the training worker semantics assume. The two are
distinguished in the header comment: `nisps_ml_set_input` = raw network input; `nisps_control_set_input`
= the mode control path (clamped, pinnable).
### 4.7 Deletions, with named consumers
| Deleted | Named consumers, and what happens to them |
|---|---|
| `ModeBase::jolt_buf_` (`base.hpp:341`) + `apply_jolt_` (`:312–320`) | `tick_control` only → replaced by `ControlCore::tick` over `MLPCore::weights_scratch()` |
| `MLHandle::jolt_scratch` (`bindings.cpp:167`) | `nisps_ml_jolt_step` only → same |
| `MLHandle::jolt`, `MLHandle::ou` (`:165–166`) | the `nisps_ml_jolt_*` / `nisps_ml_explore_*` bodies → forward into `control` |
| `ModeBase::ou_` / `out_buf_` / `input_channels_` / `input_pinned_` / `pin_value_` | `ModeBase` methods only → forwarded into `control_` |
| `ml::OUNoise<N>`'s template parameter + `state_` | `base.hpp`, `bindings.cpp:166`, `tests/cpp/test_mlp_ou_noise.cpp` → state moves to `ControlCore`'s storage |
| `ModeBase::jolt()` accessor (`:247–248`) | none — verified by grep across `nisps/ firmware/ manifold/ tests/ vcv/` |
| `nisps_ml_explore_apply` + its 4 downstream layers | see 4.6 |
| `Spine.outputMorph` + `setOutputMorph` (`spine.ts:105–109, 221–229, 294–296`) | `exploration.ts:57, 131` only |
| `ExplorationController`'s `joltTimer`/`exploreTimer` pair | internal → one driver |
---
## 5. Behaviour changes this lands
Stated plainly, because two of them are audible.
1. **Browser: the output chain now runs after the OU walk** (D3). With Explore > 0, per-output
freeze actually freezes, and the EMA/slew smooth the OU walk instead of the walk overwriting
smoothed values. Audible whenever Explore > 0 *and* smoothing/slew/freeze are non-default.
2. **Browser: the OU roam rate becomes wall-clock-correct and gesture-independent** (D6).
Today the walk advances once per spine tick, so it roams faster while the user is moving.
After: a fixed 200 steps/second. At the current 33 Hz idle driver the browser's walk is
~6× slower than firmware's; after the change they match. **Existing Explore slider positions
will feel different** — most users will want a lower setting.
3. **Browser: inputs are clamped to [0,1] and honour a pin mask** (D4). No current caller feeds
out-of-range values, so this is latent until the pin mask is wired into the UI (5c).
4. **Browser: jolt/OU RNG streams change** (D2), because the salts become the firmware's. No
committed artifact depends on them.
5. **Firmware: no behaviour change.** The tick order, the constants, the cadence and the
arithmetic are preserved by construction; §6 step 1 makes that a test rather than a claim.
The one *near*-exception is `jolt_lr_scale()`, whose ramp accumulates identically per step —
and which has no consumers on either target (D7).
---
## 6. Sequenced implementation plan
Each step is separately landable and separately verifiable. Step 1 is pure C++ and changes no
behaviour anywhere. Step 2 adds the browser-side plumbing and, because the jolt/OU salts and the
fixed-rate accumulator take effect as soon as `bindings.cpp` forwards into the head, it already
carries §5.2 and §5.4. Step 3 is the browser swap and carries §5.1 and §5.3.
**Sequencing hazard:** `nisps_ml_explore_apply` must be deleted in **step 3**, not step 2 — it is
still called by `WasmIML.exploreApply` until the TS swap lands, and removing the export first
leaves a runtime failure that no compiler catches (invariant 8, in reverse).
### Step 0 — capture a firmware-behaviour baseline (half a day)
Add `tests/cpp/test_mode_control_tick.cpp` to the `nisps_modes_tests` target
(`nisps/CMakeLists.txt:119–128`) that pins the *current* `tick_control` for at least PAFSynth,
MEMLCelium and SoundAnalysisMIDI: N ticks with a scripted input trace, asserting the exact
engine-param vector and the exact ML output vector, with Jolt and Explore both exercised.
Land it **before** touching `base.hpp` so it is a genuine before/after.
*Verification:* `bash scripts/build-cpp-tests.sh` (baseline today: 4/4 ctest suites pass in
0.03 s — run and confirmed 2026-07-21).
### Step 1 — `ControlCore` + fixed storage; `ModeBase` composes it (1–2 days)
New: `nisps/modes/control_storage.hpp`, `nisps/modes/control_core.hpp`. Modified:
`nisps/modes/base.hpp` (compose + forward), `nisps/ml/mlp.hpp` (`weights_scratch()`),
`nisps/ml/ou_noise.hpp` (externalise state). Concrete mode headers should need **no changes** —
if one does, the forwarding surface is wrong.
*Verification:* step 0's baseline test must pass **unchanged** — that is the whole point of doing
it first. Plus `nisps_modes_tests`, `nisps_core_tests`, `scripts/lint-cpp.sh`, and a firmware
compile of at least `slpworkshop`, `pafsynth`, `soundanalysismidi` via `scripts/build-firmware.sh`
(needs `nix-shell -p platformio-core`; first build pulls 1–2 GB). Compare flash/RAM against the
pre-change build — the expectation is a **reduction** of roughly `weight_count() * 4` bytes of
RAM (6,152 for MEMLCelium-class modes) from the `jolt_buf_` deletion. A size *increase* means
the composition failed to inline and wants investigating.
*Not verifiable here:* that the hardware still boots and sounds right. Operator chokepoint,
same as every firmware change in this repo.
### Step 2 — dynamic storage + the C API + parity coverage (1–2 days)
New: `nisps/modes/dynamic_control_storage.hpp`. Modified: `nisps/wasm/bindings.cpp` (MLHandle
member, reshape, the five new `nisps_control_*` exports, the `nisps_ml_jolt_*` /
`nisps_ml_explore_intensity` bodies forwarding into the head — but **not** the
`nisps_ml_explore_apply` deletion, which waits for step 3), `scripts/build-wasm.sh`,
`scripts/lint-cpp.sh` (allowlist → list).
**Add parity stage 8** to `tests/cpp/parity_check.cpp` and `tests/cpp/parity_wasm.mjs`: drive a
`ControlCore` over the harness's `MLP<32,10,14,18,126>` through a scripted trace — inputs set,
some channels pinned, a jolt pressed and released across several ticks, Explore raised, at least
one catch-up tick with `dt_s` > one period — and push the parameter vector plus a weight probe.
Bump `kVersion` 5 → 6 (`parity_check.cpp:93`) in both drivers, and extend the header's
authoritative stage list (`:14–33`) and `parity_diff.mjs`'s section names.
This is the step that converts "a parity PASS is not evidence for this change" into "it is".
Without stage 8, `scripts/parity-check.sh` exercises PAFSynth and ChannelStrip under an
all-params-0.5 vector and says nothing whatsoever about the control tick.
*Verification:* `bash scripts/build-wasm.sh && bash scripts/parity-check.sh` (both native and
WASM at 1e-5, now including stage 8); `bash scripts/lint-cpp.sh`;
`bash scripts/build-cpp-tests.sh`. `emcc`, `bun`, `cmake` and `ninja` are all on PATH here, so
this step is fully verifiable locally.
### Step 3 — the browser swap (1–2 days)
Modified: `manifold/src/engine/types.ts`, `wasm-iml.ts`, `engine-api.ts`, `spine.ts`,
`exploration.ts`. Deleted: `Spine.setOutputMorph` + the `outputMorph` field, `exploreApply`
through all four TS layers, one of the two exploration timers.
Rebuild and **commit** `manifold/public/nisps.{js,wasm}` in the same change — the webhook builds
only `manifold/` and ships whatever is committed (`manifold/ONBOARDING.md` §2), and CI's WASM
freshness gate runs the parity harness against the committed artifact before rebuilding it.
*Verification:* `cd manifold && bun run typecheck && bun run test && bun run build`, then
Playwright. Note that `manifold`'s test script is deliberately `bun test src tests/*.test.ts` —
do not "fix" it to `tests`, that drags the e2e suite into the unit run. On the VPS, e2e needs the
non-snap node runner (`ONBOARDING.md` §2).
Add e2e coverage for the two changes users can see, in `manifold/tests/e2e/`:
- Explore > 0 with a freeze mask set ⇒ the frozen outputs do **not** move (this fails today).
- Explore > 0 with the pad held still ⇒ the routed vector's drift over 2 s is within a
tolerance band, i.e. wall-clock-rated rather than tick-rated.
A unit test in the `bun test` set driving `nisps_control_tick` straight at the committed WASM —
the way `manifold/tests/loss-history.test.ts` drives `nisps_ml_loss_history` — is the cheapest
guard on the C ABI and should exist too.
### Step 4 — documentation sync (same commits)
`MAP.md` (`nisps/modes/` and `nisps/wasm/` bullets), `docs/specs/MAIN.md` (registry row +
"Loaded TWICE" is wrong — it is three instances), `manifold/ONBOARDING.md` §3 (same),
`ALIGNMENT.md` defect 1 (delete it when step 3 lands, per the repo's doc-sync rule),
`simplification-plan.md` §6.5a (burn down), and this file's `status:` → `executed`.
---
## 7. What the verification actually covers — and what it does not
| Claim | Covered by | Confidence |
|---|---|---|
| Firmware tick behaviour unchanged | step 0's pinned baseline + `nisps_modes_tests` | **High** — it is a before/after on the exact byte values |
| Fixed↔dynamic control head agree | parity stage 8 (step 2) | **High**, once written |
| Native↔WASM agreement of the new code | `scripts/parity-check.sh` **after** stage 8 exists | High after; **zero** before — the current harness touches neither modes nor jolt nor OU |
| The C API is reachable end-to-end | the `bun test` C-ABI test + Playwright | Medium-high — the 5-layer chain fails at runtime, so a browser-level test is the only real proof |
| Freeze is no longer defeated by Explore | new e2e case | High |
| OU roam rate is wall-clock | new e2e case (tolerance band) | Medium — timing assertions in a browser are inherently soft; keep the band wide |
| Firmware flash/RAM does not regress | `firmware-build` CI job's per-variant report + local `pio run` | High for size |
| **The hardware still boots and sounds correct** | *nothing* | **None.** This is an operator chokepoint, as it was for the PlatformIO cut (`simplification-plan` §5) |
| Engines other than PAFSynth/ChannelStrip behave identically | `nisps_dsp_engine_tests` + `engine_impulse` baseline; unchanged by this work | High, and unaffected — no engine code is touched |
Honest summary: everything except on-device behaviour can be verified in CI, **provided stage 8
is written**. If the implementation session runs out of time and skips stage 8, the change should
not land: the parity gate would go green while covering none of it.
---
## 8. Open questions for the operator
**Q1. The output-chain / OU ordering (§4.3d, §5.1) — confirm.** The recommendation is the
firmware's order: OU is part of the mapping, the output chain conditions what the destination
receives, and per-output freeze starts working. It changes the browser's sound whenever Explore
is up and smoothing/slew/freeze are non-default. The alternative — keeping each target's current
order — means the tick is not actually unified and needs a per-target branch, which is the thing
this whole item exists to remove. **Recommend: adopt the firmware order.**
**Q2. Fixed 200 Hz control rate as a shared contract (§4.3a) — confirm.** This makes the
browser's Explore feel roughly 6× faster at the same slider position. Options: (a) adopt it and
accept that existing slider positions read differently; (b) adopt it and rescale
`kOUMaxAmplitude` or the slider mapping so the browser's *current* feel is preserved — which
then changes the *firmware's* feel; (c) adopt it and expose the rate as a per-target constant,
which re-forks the behaviour. **Recommend (a)**, with a note in the Learning drawer copy.
**Q3. `jolt_lr_scale()` (D7) — wire it or delete it?** It is documented on both `Jolt` and
`ModeBase` as gating the learning rate after a jolt, and *nothing on either target multiplies by
it*. Wiring it means touching the firmware's two `train()` call sites and the browser's train
path — a behaviour change to training, not to this seam. Deleting it removes a documented
feature. It is cheap either way once `ControlCore` exists. **Not blocking**; needed before this
item can be called done rather than parked.
**Q4. Does the browser's input pin mask (D4) get UI in this item or in 5c?** The C API and the
core behaviour come free with `ControlCore`. Rendering a Single/Double control in the Inputs
drawer is UI work that belongs with the curated/advanced split
(`plans/curated-presets-spec.md`). **Recommend: ship the API in this item, the UI in 5c**, and
say so in `inputs-spec.md`.
**Q5. `DynamicControlStorage`'s home.** Recommended: `nisps/modes/`, with `lint-cpp.sh`'s
allowlist generalised from one hardcoded path to a list. The alternative — defining the dynamic
storage policy inside `nisps/wasm/bindings.cpp`, where heap is already legitimate and the lint
does not look — needs no lint change at all, but puts a storage policy outside `nisps/` where a
future consumer (VCV, a native fixture) cannot reach it. **Recommend `nisps/modes/`**; it is a
~10-line shell change and it keeps the "one auditable exception list" discipline.
Not asked, because they were decided here and are recorded rather than open: `ControlCore` does
not own the MLP (§4.1, matching `FeedbackControllerCore`); the parameter copy is unconditional
(§4.3c); the jolt scratch buffer is deleted in favour of `MLPCore`'s existing flat scratch
(§4.2); `OUNoise` loses its template parameter (§4.2); the salts become the firmware's (§4.4).
---
## 9. Explicitly out of scope
- **The engine half of a mode in the browser** — voice-space selection, `note_on`, transport
(`set_playing`/`update_bpm`), and the `ControlEvent` drain have no browser surface at all
(verified: `grep -rn 'voiceSpace\|noteOn\|setPlaying\|popEvents' manifold/src` returns only
generated-schema data). That is item **5b** — see `plans/browser-mode-coverage-spec.md` — and
it is the reason four catalogued modes cannot actually run in the browser.
- **Firmware `InputChain`/`OutputChain` wiring (D8, L29)** — assigned to **5d** by
`simplification-plan` §6.5d; see `plans/hardware-editor-spec.md`. This item only fixes where OU
sits relative to a chain the caller already owns.
- **Feedback-controller ownership.** Firmware keeps it as a function-local static in
`glue/peripherals.hpp:152`; the browser keeps it inside `MLHandle`. Both already call the same
shared core, so it is an asymmetry, not a duplication. The related `FOLLOW-UP` comment at
`glue/peripherals.hpp:33–39` (hold `feedback.static_output()` while Placing instead of running
fresh inference) *would* have a natural home in `ControlCore::tick` once it exists — note it,
do not build it here.
- **The browser's multi-anchor ExploreAndPlace session policy**
(`manifold/src/feedback/controller.ts:200–290`) differs from the firmware's freeze-and-carry
gesture. That is an application-policy divergence above the core, not a core duplication.
- **`jolt_lr_scale` wiring** (Q3) and **the pin-mask UI** (Q4).
---
## 10. Source-file index
Read in this order.
**The gap**
- `nisps/modes/base.hpp` — `ModeBase`; the tick is `:198–234`, the jolt copy `:312–320`
- `nisps/wasm/bindings.cpp` — `MLHandle` `:148–185`, jolt/OU C API `:827–886`, reshape `:396–421`
- `manifold/src/engine/spine.ts` — `setInputs` `:254–303`, the morph hook `:105–109, 221–229`
- `manifold/src/engine/exploration.ts` — the two timers and the morph registration
**The pattern being copied**
- `nisps/ml/mlp.hpp:91–116` + `nisps/ml/storage.hpp` + `nisps/ml/dynamic_storage.hpp` —
`MLPCore<Storage>`, the storage-surface documentation style, the embedded `#error` guard
- `nisps/ml/feedback.hpp:42–61` — `FeedbackControllerCore<FbStorage>`, and the
"does NOT own the MLP" convention this design follows
**Consumers that must keep compiling unchanged**
- `firmware/MEMLNaut-NISPS/glue/peripherals.hpp`, `settings_view.hpp`, `midi_io.hpp`,
`output_router.hpp`, `src/main.cpp:154–160`
- `tests/cpp/test_mode_{concepts,paf_synth,voice_space,breakor_events,learning,curve_overrides,driver_config}.cpp`
- `nisps/modes/sound_analysis_midi.hpp:86–111` and `external_synth_midi.hpp` — the only modes
with `on_pre_inference` / `on_post_inference` hooks, i.e. the sharpest test of the forwarding
surface
**Gates**
- `scripts/build-cpp-tests.sh`, `scripts/parity-check.sh`, `scripts/lint-cpp.sh`,
`scripts/build-wasm.sh`, `scripts/build-firmware.sh`, `.github/workflows/ci.yml`
- `tests/cpp/parity_check.cpp:14–33` (the authoritative stage list) + `parity_wasm.mjs` +
`parity_diff.mjs`

View file

@ -0,0 +1,221 @@
---
kind: plan
status: active
---
# One Core Engine — Firmware + Manifold Reunification Plan
*Dated 2026-07-13. Operator-confirmed scope (see §1). Principle: **one core engine (`nisps/`) for both the
resource-constrained RP2350 hardware and the Manifold browser app.** Everything with behaviour — ML, feedback,
exploration, RNG, DSP, input/output processing, curves — lives once, in C++. TypeScript keeps only UI,
reactivity, and transport. This plan supersedes the browser-side "one engine, two skins" framing of
`../engine-architecture.md` (that seam is built — Manifold is its realisation); the axis of unification now
runs vertically, core↔targets, not horizontally, skin↔skin.*
## 0. Where duplication actually lives today (audited 2026-07-13)
| Component | Canonical | Duplicate / divergence | Disposition |
|---|---|---|---|
| MLP, training, init, RL `move_weights` | `nisps/ml/` | `vcv/src/iml.hpp` full runtime reimpl ("delta #5") | Reunify (P2, P6) |
| Feedback explore-and-place | `nisps/ml/feedback.hpp` | none — firmware + manifold both call it ✓ | Keep |
| Geometric dislike (Mode 1) | *(missing from core)* | `manifold/src/feedback/controller.ts` TS approximation, explicit `C++ GAP` markers (lines ~369, ~436) | Port to core (P3) |
| Jolt weight-morph | `nisps/ml/jolt.hpp` | `playground/src/ml/jolt.ts` TS port; absent in manifold | Expose via WASM (P3) |
| OU exploration noise | `nisps/ml/ou_noise.hpp` | `playground/src/output/ou-explore.ts` TS port; absent in manifold | Expose via WASM (P3) |
| Deterministic RNG | `nisps/core/rng.hpp` | `manifold/src/feedback/rng.ts` (`SeededRng`, self-described stand-in) | Replace with core stream (P3) |
| Input pipeline | *(TS-only concept)* | `playground/src/input/pipeline.ts` ≡ `manifold/src/engine/input-pipeline.ts` | Port to core (P4) |
| Output pipeline + curves | curves mirror `nisps/core/math.hpp` | byte-identical TS pairs in playground + manifold | Port to core (P4) |
| Audio DSP engines (×8) | `nisps/engines/` | none — both targets run the WASM ✓ | Keep |
| TS engine spine (wasm-iml, worker, host, worklet) | — | manifold forked from playground; several files byte-identical | Single home in manifold (P1) |
| Compiled `nisps.wasm` | `scripts/build-wasm.sh` → `playground/public/` | `manifold/public/` byte-identical **manual copy** | Retarget build (P0) |
| Mode param schemas | `schemas/modes/*.json` → codegen | manifold bypasses codegen; hardcodes `MF_MODES` in `console/model.ts` | Codegen for manifold (P5) |
| WASM MLP arity | `bindings.cpp` `MLP<32,10,14,18,126>`; `nisps_ml_create` ignores requested dims | forces clamping, blocks per-mode arch, motivated the VCV reimpl | Runtime-shaped browser MLP (P2) |
## 1. Locked decisions (operator, 2026-07-13)
| Decision | Choice |
|---|---|
| `playground/` | **Retired.** Archived to branch `archive/playground-solidjs`, deleted from main. Manifold is the sole browser app; WASM build, parity-check, codegen all retarget manifold. |
| Depth of "one core" | **All algorithms in C++.** Feedback (incl. geometric dislike), Jolt, OU, RNG, input/output pipelines, curves move into `nisps/`; TS keeps UI/reactivity/transport only. |
| MLP arity | **Template the hardware, dynamic browser.** Firmware keeps compile-time `MLP<...>` (zero-heap contract intact); WASM/VCV targets get a runtime-shaped MLP sharing the same algorithm code. Heap allowed at construction time on non-RP2350 targets only. |
| VCV | **In scope, final phase** (P6); may slip without blocking anything else. |
**Supersedes:** `BUILD-PLAN.md` locked decision "Modular N×M MLP: multiple WASM modules + warm-start; NOT
runtime-shaped" (2026-06-27). The runtime-shaped MLP is now the chosen mechanism; warm-start on reshape is
retained as a feature of it.
## 2. Target architecture
```
┌───────────────────────── nisps/ (ONE CORE) ─────────────────────────┐
│ core/ rng, math+curves, buffers, perf attrs │
│ ml/ algorithms templated on StoragePolicy: │
│ Fixed<NIn,H1,H2,H3,NOut> (arrays, zero-heap → RP2350) │
│ Dynamic (sized at construction → WASM,VCV)│
│ mlp / training / init / rl / jolt / ou_noise / feedback / │
│ geo_push / replay (geometric dislike lands here) │
│ pipeline/ input chain (deadzone→zoom→curve→smooth→momentum) │
│ output chain (curve→smooth→slew→freeze) ← NEW │
│ dsp/ engines/ modes/ (unchanged) │
└──────────────────────────────────────────────────────────────────────┘
▲ direct #include ▲ nisps/wasm/bindings.cpp (C ABI)
┌────────┴─────────┐ ┌─────────┴──────────┐ ┌───────────────┐
│ firmware/ glue │ │ manifold/ (React) │ │ vcv/ (P6) │
│ RP2350, Fixed │ │ UI + reactive spine│ │ Dynamic │
│ storage, no heap │ │ + transports only │ │ storage │
└──────────────────┘ └────────────────────┘ └───────────────┘
```
Litmus test per file: **if it computes behaviour that affects sound, weights, or parameter values, it is C++
in `nisps/`; if it schedules, renders, or transports, it may be TS.** The existing browser litmus (JSX/DOM =
skin) still applies inside manifold; this plan adds the vertical one.
## 3. Phases
Each phase ends green on its test gate and is independently landable. File phases as **ergo** tasks
(one per phase, sub-tasks per bullet); do not start a phase before its predecessor's gate is green.
### P0 — Plumbing hygiene (hours, no behaviour change) — ✅ landed 2026-07-13
- ✅ `scripts/build-wasm.sh` emits to `manifold/public/` (keep `playground/public/` copy only until P1 lands).
- ✅ `scripts/parity-check.sh` (and `tests/cpp/parity_wasm.mjs`) read the manifold artifact.
- ✅ Fix stale doc: the `MLP<2,…>` line was in `docs/AGENT-REFERENCE.md` + `nisps/wasm/README.md` (MAP.md was already correct); both now say `MLP<32u,10u,14u,18u,126u>`.
- ✅ `ml-debug.log`/`graphify-out/` were already absent from the tree; `.claude/worktrees/` gitignored.
- **Gate met:** `run-all-tests.sh` green (Playwright leg via the BUILD-PLAN non-snap-node VPS runner); parity PASS reading `manifold/public/` (max delta 2.4e-7); manifold typecheck+build green against the freshly-built artifact.
### P1 — Retire playground, single TS home (≈1 day) — ✅ landed 2026-07-13
- ✅ Branch `archive/playground-solidjs` + tag `playground-solidjs-final`; `playground/` deleted from main.
Parity fixtures for P4 captured FIRST: `manifold/tests/fixtures/` (288-event gesture trace, curve
goldens, input/output pipeline goldens under 14+8 configs) + drift-guard `pipeline-golden.test.ts`.
Note: input pipeline's momentum path reads `performance.now()` — fixtures pin a clock contract (see
fixtures README). C15 (stub + c15.wasm) now lives only on the archive branch (ALIGNMENT defect #1).
- ✅ Jolt press + OU explore UI shells in the Learning drawer; interim TS math via get/set_weights in
`manifold/src/engine/{exploration,jolt,ou-explore}.ts`, marked `P3 SWAP POINT`.
- ✅ Keeper Playwright specs ported: `probe-api.spec.ts` (15 tests), `spine.spec.ts` (4). Playground UI/
persistence specs dropped with the playground.
- ✅ References retargeted: run-all-tests stage 5 → manifold, ci.yml manifold-tests job, osc-bridge.yml
(was already broken), codegen TS target removed (returns P5), AGENTS/README/MAP/ALIGNMENT/
AGENT-REFERENCE/specs-MAIN. The VPS deploy webhook already built only manifold — no change needed.
- **Gate met:** full `run-all-tests.sh` green with playground gone (ctest 4/4, parity PASS, lint clean,
manifold 9 unit + 20 e2e).
### P2 — Storage-policy split: templated hardware, dynamic browser (the structural centre, ≈1 wk) — ✅ landed 2026-07-14 (hardware timing spot-check of chokepoint B pending physical MEMLNaut)
- ✅ (landed 2026-07-13) Refactor `nisps/ml/` so algorithms (forward, backprop/SGD, init, `move_weights`)
are written once against a storage policy (`mlp.hpp` `MLPCore<Storage>`; jolt/OU/feedback already operate
on the MLP surface and needed no change). Two models:
- `FixedStorage<NIn,H1,H2,H3,NOut>` (`nisps/ml/storage.hpp`) — `std::array`, zero heap; `MLP<...>` is an
alias preserving the full compile-time surface. **RP2350 contract verified: PAFSynth `.text`
122324→122692 = +0.30% (±1% budget); ctest + golden vectors + WASM parity bit-stable.**
- `DynamicStorage` (`nisps/ml/dynamic_storage.hpp`) — sizes at construction, single arena allocation,
nothing per-call. `#error`s under `NISPS_TARGET_EMBEDDED`; sole `lint-cpp.sh` heap-allowlist entry, with
a lint check that fails if the guard is ever removed.
- ✅ (landed 2026-07-14, operator-approved) `nisps_ml_create(input, output, hidden[])` honours its
arguments (non-positive/null → the historical 32→[10,14,18]→126 defaults, keeping pre-P2 callers
bit-identical). `nisps_ml_reshape` = new instance + warm-start copy of overlapping weights
(`nisps/ml/warm_start.hpp`); feedback controller re-created (state resets — front-end modal).
`FeedbackController` got the same storage split (`FeedbackControllerCore<FbStorage>`, fixed alias for
firmware/tests, `DynamicFeedbackStorage` for the browser). `nisps_ml_describe` now takes the handle
(null → default shape). Verified: reshape ABI smoke (dims honoured, overlap survives, invalid dims
rejected), warm-start ctest (grow+shrink), parity PASS unchanged, firmware `.text` unchanged.
- ✅ (landed 2026-07-14) Manifold reshape wiring: `WasmIML.reshape` + `EngineApi.reshape` (buffers/worker/
spine follow the new dims; training-worker protocol carries `hidden`); reset-on-reshape confirm modal
offered on genuine input-layout changes (default 32-in over-provisioned head untouched on load/decline);
stale even/odd-blend copy deleted. The mean-blend clamping itself was already gone (dedicated dims);
the `min(n, inputSize)` guard + zero-padding stay as the legitimate declined-reshape behaviour.
XIASRI/sound-analysis multi-input modes become browser-viable (schema-real dims arrive at P5).
- **Gate met:** fixed↔dynamic bit-identical ctest (+ warm-start grow/shrink ctest + reshape ABI smoke);
`parity-check.sh` native↔WASM PASS unchanged (2.4e-7); PAFSynth `.text` 122324→122692 (+0.30%, within
±1%); manifold 9 unit + 25 e2e. Hardware audio-callback timing check deferred to the chokepoint-B
hardware session (operator, physical MEMLNaut).
### P3 — Exploration + feedback fully in core (≈3–4 days) — ✅ software landed 2026-07-14; ⏳ chokepoint A hardware spot-check pending (operator, physical MEMLNaut)
- ✅ **Geometric dislike:** firmware k-NN centroid push (upstream `InterfaceRL` @ `0a541cc`, verbatim
constants) ported into `nisps/ml/geo_push.hpp` + `replay.hpp` + `mlp.train_targets`; the press+optimise
halves collapsed into one synchronous `dislike_geometric()` per the ADR. `AvoidStyle::Geometric` is the
Avoid default; `Diffuse` (legacy move_weights) kept for A/B. Exposed via
`nisps_ml_feedback_dislike_geometric` (+ store_positive/counts/set_avoid_style). TS approximation +
both `C++ GAP` blocks deleted; two call-site bugs fixed en route (raw vs HEARD post-pipeline vector —
raw gives a zero-derivative no-op). Cold-start prompt per ADR §7.
- ✅ **Jolt / OU:** exposed through bindings (`nisps_ml_jolt_press/step/release/lr_scale/...`,
`nisps_ml_explore_intensity/apply`; browser `OUNoise<4096>` over-provisioned); manifold's P1 UI shells
switched at the marked swap point; interim TS ports (`jolt.ts`, `ou-explore.ts`) deleted. OU applies
through the spine's output morph hook.
- ✅ **RNG:** `manifold/src/feedback/rng.ts` deleted; all feedback/exploration randomness is the core's
seeded per-instance `Rng` streams. (Debug-only fixed seed/dt under `?debug=1` for deterministic e2e.)
- **Gate:** ✅ parity Stage 6 (v4) — scripted seeded feedback session (2 likes → centroid → 2 dislikes)
produces identical weight trajectories native↔WASM (961 floats, 2.4e-7); ctest geo suite; manifold 9
unit + 27 e2e. ⏳ Hardware spot-check (chokepoint A) BLOCKED on the physical MEMLNaut — and on ergo bug
`10c3e55c`: the explore/place wiring is linker-GC'd out of the PAFSynth ELF (pre-existing, discovered
during P3; the gestures are dead code on that build until fixed).
### P4 — Input/output pipelines + curves into core (≈3 days) — ✅ landed 2026-07-18
- ✅ New `nisps/pipeline/`: `input_chain.hpp` + `output_chain.hpp` (capacity-templated), perf-contract
structs, caller-supplied dt with an internal clock (no wall time — deterministic), serialisable state.
- ✅ Curves: `nisps/core/math.hpp` is the single catalog (+ parameterised `centered_power`); bindings expose
`nisps_curve_apply(id, x, param)` + batch. TS `curves.ts` deleted. NOTE: the TS mirror's `exp/log/
sigmoid/cubic` maths silently diverged from the canonical catalog — the browser now behaves
firmware-exact (re-baselined in `curves-golden.json` with a provenance note; ALIGNMENT entry).
- ✅ Manifold pipelines are thin WASM calls (one `nisps_pipeline_create` handle per WasmIML; state C++-side;
config stays TS-side and is pushed over the 15-float wire layout).
- ✅ Golden TS-vs-C++ tests retired → the fixture suite drives the WASM directly.
- **Gate met:** spine e2e unchanged (27 e2e green); recorded-gesture regression against the P1 pre-migration
fixtures passes — non-momentum configs at ≤5e-7, momentum configs at 1e-2 with the drift PROVEN inherent
f32-vs-f64 (a byte-faithful f32 reference reproduces the WASM to <6e-8; documented in the test + fixtures
README). Parity stage 7 (v5): native↔WASM pipeline/curve floats bit-identical.
### P5 — Schema/codegen serves manifold (≈2 days) — ✅ landed 2026-07-18
- ✅ `codegen/generate.ts` emits `manifold/src/modes/generated/` alongside the C++ headers. `MF_MODES` is
derived from the generated schemas (real param names/groups/counts + per-mode `ml` config + `engine_id`);
labels/glyphs/class/ordering stay a manifold overlay; `xiasri` + `slp_workshop` join the browser catalog;
schema-less `visualizer`/`c15` remain hand-written overlays on the default shape.
- ✅ Per-mode `ml.input_size/output_size/hidden` is REAL in the browser: mode switch reshapes the runtime
MLP to the schema config (boot mode paf_synth runs 4→[10,10,14]→33, not the old fixed 32→126). Codegen
gained the firmware-fit check (exactly 3 hidden layers; dims ≤4096).
- **Gate met:** codegen idempotence golden covers both languages and runs in `run-all-tests.sh` stage 5;
`schema-modes.spec.ts` asserts dims/weight-count/param-count FROM the imported schemas for 4 modes; the
existing shape-assuming specs were re-derived from schema constants. 33 e2e green.
### P6 — VCV reunification (later; may slip) — ✅ landed 2026-07-18
- ✅ The vendored `vcv/src/iml.hpp` (DetRng + 3D-weight runtime MLP + Dataset) is now a THIN Rack-free
adapter over `MLPCore<DynamicStorage>` + `nisps::Rng` + the core dataset; hidden `[16,24,16]` kept
(maps 1:1 onto the 3-hidden topology). Patch JSON weights went flat (version 2→3). The module's
double-buffered audio/worker threading discipline is unchanged (two self-contained adapter instances).
Closes vcv-module.md delta #5.
- ✅ OSC server stayed vcv-local.
- **Gate met:** `plugin.so` builds/links clean against the Rack SDK; new ctest
`test_vcv_iml_parity.cpp` — the adapter is memcmp-identical to driving the core directly across
construction/draw/train/inference/move_weights(pin mask).
## 6. Execution wrap-up (2026-07-18)
All six phases landed on main with green gates (P0 2026-07-13 → P6 2026-07-18). The plan flips to
`status: executed` once the two OPERATOR items close (physical MEMLNaut required):
- **Chokepoint A** — geometric-dislike feel spot-check on hardware (ergo `425f4a8f`), blocked on ergo bug
`10c3e55c` (explore/place wiring linker-GC'd out of the PAFSynth ELF — pre-existing, discovered in P3).
- **Chokepoint B (timing half)** — audio-callback timing on hardware; the compile half (`.text` ±1%)
passed at P2.
## 4. Risks
| Risk | Mitigation |
|---|---|
| P2 template refactor destabilises firmware perf (chokepoint B) | Fixed-storage path stays `std::array` + same attrs; compare `.text`/RAM and audio-callback timing before/after; land behind a ctest parity gate. |
| Dynamic storage leaks heap use into firmware paths | Compile-time guard (`#if NISPS_TARGET_EMBEDDED` excludes `DynamicStorage`); `lint-cpp.sh` already fails heap under `nisps/` — carve an explicit allowlist for the dynamic TU only. |
| Per-event WASM calls for pipelines add latency | Pointer-rate ≈120 Hz, one call each — negligible; batch API as fallback. rAF/canvas never calls WASM. |
| Geometric-dislike port changes feel vs firmware | Port from upstream source verbatim, seeded-trajectory parity test, then hardware A/B (chokepoint A) before deleting the TS fallback. |
| Retiring playground loses a working reference | Archive branch + tag; parity fixtures (recorded gestures, golden outputs) extracted *before* deletion in P1. |
| Manifold worklet + main thread now both depend on retargeted wasm build | P0 gate builds manifold from fresh artifact in CI, killing the manual-copy hazard permanently. |
## 5. Doc-sync obligations
- On each phase landing: update `MAP.md` (it is the shipped-behaviour authority) and burn down this plan's
phase table; when all phases land, set `status: executed`.
- `MAIN.md` registry: this file added; `engine-architecture.md` remains authoritative for the *browser-side*
seam only; BUILD-PLAN.md's MLP row annotated as superseded (done 2026-07-13).
- `ALIGNMENT.md` defect #3 (fixed WASM arity) closes at P2; the accepted-divergence log entries for Jolt/OU
TS ports and vcv delta #5 close at P3/P6.

View file

@ -1,6 +1,11 @@
---
kind: plan
status: superseded
---
# NISPS Playground 2.0 — Clean-Room SolidJS Rewrite Plan
*Status: sign-off-ready. Author: lead architect, synthesizing four designer proposals (P1–P4) against three judge panels (mission-fit, feasibility, UX coherence). Date: 2026-06-17.*
*Author: lead architect, synthesizing four designer proposals (P1–P4) against three judge panels (mission-fit, feasibility, UX coherence). Date: 2026-06-17. **Superseded**: its target — the SolidJS playground — was retired 2026-07-13 (branch `archive/playground-solidjs`, tag `playground-solidjs-final`). Many of its ideas (Console interaction model, reactive spine, right-edge dock, off/fixed/live tri-state, schema-driven modes) shipped instead in the Manifold React app; see `BUILD-PLAN.md` (executed) and `MAP.md` §manifold.*
---

View file

@ -0,0 +1,227 @@
---
kind: plan
status: active
---
# Simplification Plan 2026-07 — burning down the audit
*Dated 2026-07-21. Derived from `../recon/simplification-audit-2026-07.md` (finding IDs `A*`/`S*`/`L*`/`ST*` refer to its inventory; every mitigation here was adversarially verified against the code). Companion: `ALIGNMENT.md` carries the strategic tier. Proposal only — no phase is adopted until the operator says so; §7 lists the decisions that gate specific items.*
## §0 Principles and ordering rationale
- **Deletion over refactor, refactor over addition.** Nothing gets a compat shim without a named consumer.
- **Hard constraints hold throughout**: `nisps/` platform-neutral + allocation-free hot paths; native↔WASM parity ≤1e-5; schema changes ship with both codegen outputs; RT-safe worklet/dual-core comms. Every phase ends with `run-all-tests.sh` green.
- **Docs move in the same commits as code** (MAP/ALIGNMENT/spec-corpus sync is part of "done").
- Order: *restore verification first* — until CI runs, nothing else can be trusted to have landed; *then delete dead mass* — it shrinks every later diff; *then bugs, then truth-consolidation, then the vision builds*.
- On adoption of a phase, decompose it into ergo tasks (one per bullet-group below); this doc is the map, not the tracker.
## §1 Phase 0 — Restore verification (URGENT, ~half day) — S7, S24, S31, S32; critic gaps 1–3
**BURNED DOWN 2026-07-21.** All five items landed: memllib `feat/nisps-core-swap` pushed to
`monkey-w1n5t0n/memllib` and `.gitmodules` repointed (pin `b37fc53` now reachable — data-loss risk
closed); the compensating error paragraph in `build-firmware-arch.sh` deleted; codegen dirty-diff +
golden test added to the `manifold-tests` CI job; a WASM freshness gate (parity harness vs the
*committed* artifact) added to `cpp-tests` before the rebuild; and the VPS webhook
(`~/.config/webhooks/meml-deploy.sh`, not in this repo) now blocks the deploy unless the `CI`
workflow concluded `success` on that exact SHA — fail-closed, `MEML_SKIP_CI_GATE=1` to override.
CI has been 100% red on main since 2026-07-13; the cause and the data-loss risk are the same object.
1. **Push `feat/nisps-core-swap` to the `monkey-w1n5t0n/memllib` fork** (3 commits incl. pin `b37fc53` currently existing only on this disk); repoint `.gitmodules` to the fork URL; verify a fresh clone + `submodules: recursive` checkout succeeds; delete the compensating error-message paragraph in `scripts/build-firmware-arch.sh` (S7).
2. Confirm CI goes green end-to-end; only then trust any subsequent gate claims.
3. **Codegen enforcement in CI** (S24, S31): in the manifold-tests job (bun already present) run both generators + the golden test and fail on dirty diff — makes the "regenerated outputs in the same change" rule real.
4. **WASM freshness gate** (S32): before CI's own WASM build overwrites it, run the existing parity harness against the *committed* `manifold/public/nisps.{js,wasm}` so a stale committed artifact fails loudly. (The same artifact is what the VPS webhook ships to production.)
5. **Deploy gating** (critic gap 3, operator decision §7.4): today push-to-main deploys to `meml.lnfinitemonkeys.org/next/` with no gate at all. Cheapest real fix is VPS-side (webhook checks the commit's CI status before building); repo-side minimum is documenting the risk in `manifold/ONBOARDING.md`.
## §2 Phase 1 — Dead-mass deletion sweep (~2–3 days, one commit per group)
All verifier-checked deletions; protected exceptions noted. Rough net effect: thousands of lines removed with zero live-behaviour change.
- **Repo root / hygiene**: retired-playground dist + root Playwright rig + root `package.json`/lockfile/`node_modules` (S23, S29, L55); `NISPS_CORE_EXTRACTION_PLAN.md` + `NISPS_CORE_TASKS.md` (L48/L36); `data/` (L36); committed `.claude/worktrees/` fragment (L32); prune stale `worktree-*` branches.
- **nisps core/ml**: `fixed_buffer.hpp` + its test (L27); `dislike_multiplier_` (L26); `copy_weights_to(span)` to drop `flat_` from FixedStorage and the per-gesture double copy (L28); perf-macro regime — delete the 3/5 dead SRAM macros, fix the one misshapen `NISPS_AUDIO_FUNC` use in `midi_io.hpp` (S21, L13, closes old ALIGNMENT #4). The 16 KB loss-history buffer (L25) waits on the telemetry decision (§7.3).
- **engines/modes**: `voice_space.hpp` (L3); no-op VoiceSpace boilerplate on the five engines without real voice spaces (L9); `SawOsc`/`SquareOsc` — **keep `SineOsc`**, the selftest uses it (L4); `input_dirty_` (L5); VerbFX dead fields/setters (L6); MEMLCelium inert feedback path after checking the upstream app for a missing write (L7). `DriverConfig` (S4) is **wired, not deleted** — §7.2 resolved 2026-07-21; see below.
- **wasm bridge**: dead C-API entries through the full 5-layer chain — **keep** `EXPORTED_RUNTIME` `cwrap`/heaps (parity + wasm-load tests build their API via `Module.cwrap`) (S33); `publishWeights_` 200 Hz weight-copy channel (S34); worklet loader dead scaffolding + throwing import stubs (L54).
- **manifold UI**: dead focus/altitude system — SplitStage, ReadoutStrip, InputMini, AltitudeNav, CompactAxis, keep MiniMeters and **don't touch `engine.feedback.setFocus`** (S15); decorative controls — A/B, fake seed/gradient, snapshots, master volume, bpm, training-param sliders per verifier notes (S16, absorbs L1's delete-half); `BackendAdvanced.tsx` duplicate editor table (S18); ConsoleCtx prune to consumed fields (S19); 5 dead primitives (L22); dual backend catalogues (L23); inert soloMode selector + FeedbackController vestiges (L20, L21).
- **backends**: dead protocol legs — sendState/sendWeights, legacy array format, unreceivable `/nisps/state`, unused module-output listeners (L16); delete `bridge.mjs`, keep `bridge.ts` + compiled binaries as distribution (S11).
- **vcv**: `vcv/test/` + `Makefile.dist` + tracked ELF (L33).
- **firmware**: daisysp — 41 dead .cpp per build, delete vendored tree + symlink, fix docs that call it a submodule (S8); `input_router.hpp` zero-logic layer (ST2); peripherals dead declarations + extracted `commit_and_train` helper (L14).
## §3 Phase 2 — Behaviour bugs (~2 days)
**BURNED DOWN 2026-07-21.** All nine landed with regression tests where testable. Two deviations
from the text below, both deliberate: **L35** was resolved by DELETING the OSC full-state bridge
rather than moving its JSON to the worker — reading the current transport showed both directions
have zero consumers (manifold's osc-client.ts never sends `/nisps/state`|`/nisps/weights` and
bridge.ts drops them), so relocating heap-heavy work to serve nobody was the wrong shape;
`dataToJson`/`dataFromJson` are untouched and still serve Rack patch save/load off the audio
thread. **S35** required extending `nisps_ml_describe` from a 6-int to a 7-int wire format, which
forced matching buffer-size fixes in two call sites outside the finding's scope
(`wasm-worker.ts`, `parity_wasm.mjs`) that would otherwise have overflowed the WASM heap by 4
bytes on every call. Also fixed en route: `manifold/package.json`'s test script listed test files
explicitly, so new unit tests were silently not run — now a glob.
- **S35** Dual example store: name `kDefaultMaxExamples = 128` once in nisps, expose via `nisps_ml_describe`, align the TS mirror (currently 100) — fixes `train()`/`trainAsync()` diverging past 100 examples and the latent OOB sample-weight read.
- **S10** VCV bridged mode truncates inputs to 2-D: `EngineApi.inputVector()` returns the spine's full N-dim raw vector; VcvBackend tracks/dead-zones the full length.
- **L34/L35** VCV plugin RT-safety: worker deep-copies example vectors into staging before flagging; move full-state JSON off Rack's audio thread onto the existing worker (reuse the staged-weights pattern).
- **L18** MIDI input flooding React per CC message → notify only on binding-list changes; **L19** BackendManager dropping a switch requested mid-switch; **L24** ConsoleApp global-listener effects re-subscribing every render.
- **S30** lint-cpp: cover `nisps/pipeline/` + `core/`, strip comments before matching, extend the heap pattern set — the no-heap gate currently has proven false-negative modes; **L52** parity-check.sh unreachable FAIL branch under `set -e`.
- **L51** parity runner header/stage-list drift (4-stage/v1 documented vs 7-stage/v5 real).
## §4 Phase 3 — Single-source-of-truth consolidation (~2–3 days)
**BURNED DOWN 2026-07-21**, except **S26**, which is now an operator decision backed by a
per-field inventory (see below). Every generated numeric value is byte-identical to what was
committed — the diff on the generated dirs is purely additive, which is what a relocation of truth
should look like. Three audit claims were wrong and are corrected in the commits: L8's "breakor and
elysiamorf duplicate ratio_seq" (elysiamorf has no ratio_seq at all; the real duplicate pair is
breakor + memlcelium), ST13's proposed destination would have broken codegen (the generator
ajv-validates every `*.json` directly under `schemas/midi_devices/`, so the provenance file went to
a `sources/` subdir), and ST4's remaining-copy count.
**S26 — decision now cheap.** `ml.default_spread` is already wired on both targets (the audit's
"zero consumers" claim was a quarter wrong). `ml.default_learning_rate` and
`ml.default_max_iterations` are unread on both targets, but every schema carries exactly 1.0/1000,
which are precisely the values hardcoded in `mlp.hpp` and `wasm-iml.ts` — so **wiring them is
numerically a no-op today** and simply makes the schema the source of truth it claims to be.
`ml.input_channels` is read by nobody at runtime but is validated at codegen time and carries real
information for `sound_analysis_midi`. Per-param `curve` is a **trap**: `params_notes.md` says it is
DESCRIPTIVE — it documents that engine voice spaces already square the value internally — so
"wiring" it would double-apply squaring on 35 params.
- **S1** Mode identity: codegen emits an `ALL_MODE_SCHEMAS` registry; `SCHEMA_MODES` becomes a mode_id→display-overlay map (labels/glyphs stay legitimately hand-curated); **L37** delete the hand-`switch`ed `modeEngineId` (route on schema `engine_id`, single named exception for sound_analysis_midi).
- **S5** codegen emits per-mode `kXSchema` ParamSchema constants (deletes 9 hand-written 12-field blocks); **S6/S25** MLP template args come from generated constexpr dims (or `static_assert` parity), killing the hand-typed-dims dual truth; **L11** ext-synth ML config folds into the same pipeline.
- **S26** Schema surface honesty: per-field wire-or-delete pass (default_learning_rate/max_iterations both-platforms-or-neither; input_channels; per-param curves) per the verifier's corrected list.
- **L38** one TS `applyCurve` (mapping.ts's spec-anchored formula survives); **ST4** one GROUP_COLOR source; **L8** extract shared `ratio_seq`/SeqClock/EventQueue for the sequencer engines; **L17** BaseBackend mirroring BaseSource for status/throttle plumbing; **ST12** shared codegen `lib.ts`; **L39** delete the clobbering seed script (git keeps it); **ST13** move `synth-midi-cc.json` under `schemas/midi_devices/`.
## §5 Phase 4 — PlatformIO migration (vision 4) — A6, S2, L12, S9
**BURNED DOWN 2026-07-21.** One cut, no dual path, as specified.
- `firmware/MEMLNaut-NISPS/platformio.ini`: 16 `[env:]`, one per variant, each passing
`-DMEMLNAUT_MODE_TYPE`; `selftest` passes `-DNISPS_SELFTEST=1`. **The env list IS the registry** —
the `.ino` comment-registry and the `NISPS_ST_*` token-paste table (L12, already silently missing
the shipped variant) are gone, not migrated.
- Deleted: the Python/sed `.ino`-mutation machinery, the sketch symlink forest, the global TFT_eSPI
`User_Setup.h` mutation (now `-D` flags, TFT_eSPI's own documented recipe), the UF2 boot-mount
detection stack (`upload_protocol = picotool` talks to the bootloader directly), and
`build-firmware-arch.sh` entirely. Scripts 683 → 435 lines.
- memllib **vendored** at `lib/memllib/` from upstream `e291192`; the submodule is gone. Provenance
and re-sync procedure in `VENDORED.md`.
- **S9 done**: a `firmware-build` CI job compiles three representative envs with a cached toolchain
and reports per-variant flash/RAM. Firmware is in an automated gate for the first time.
Verified: **all 16 envs build from an empty cache**, and every one lands within ~520 bytes of the
arduino-cli binary it replaces (flash and RAM), measured as `.text+.rodata` / `.data+.bss+…` rather
than PlatformIO's console line, which double-counts `.data` on this board. What this does NOT prove
is that the hardware boots — no flash+smoke test was possible. That remains an operator chokepoint.
Two traps worth remembering, both recorded in `VENDORED.md` / `platformio.ini`: vendoring memllib's
subdirs without a `src/` wrapper makes PlatformIO's library builder silently compile **nothing**
while still linking; and project `build_flags` land *before* the framework's own `-std=gnu++17 -Os`,
so `build_unflags` is required — appending our own flags is not enough.
## §6 Phase 5 — Vision-facing architecture (each item spec-first, own session)
- **5a Mode-layer reunification (A1, A5).** The verified shape: do *not* bind monolithic mode objects into WASM and do not delete the per-primitive C API (contradicts the locked P2/P3 architecture and the two-instance RT split). Instead storage-policy the ModeBase orchestration the way P2 did MLPCore, so control-tick behaviour (jolt stepping, OU, routing, event pump) exists once in C++ and the browser stops hand-mirroring it. Needs its own spec before code.
- **5b Browser mode coverage honesty (A2).** Add an audio-topology notion (generator / audio-in-fx / event-only / analysis) so Manifold stops cataloguing 4 modes that structurally cannot run; wire mic input for the audio-in class (absorbs old ALIGNMENT #1); event-only modes need transport/MIDI-out UI, or explicit "hardware-only" labelling.
- **5c Curated/advanced split (A3, A7).** Product model first (§7.6): what is a "curated preset" — schema + backend preset + input map + trained net? Then: instrument picker rendered from the already-plumbed `ctx.modes`/`setModeId` (engine reshape-on-switch already works); per-drawer depth levels as the disclosure mechanism rather than one global boolean; `backends/presets.ts` + schemas seed the data model (add the missing `cv` backend to presets — small bug from A3's verification).
- **5d Hardware editor (A4, S14).** The repo already contains the right discipline: `useq-celium`'s C-header wire-protocol truth + TS mirror + parity test. Apply it to a MEMLNaut USB-serial protocol; give firmware an actual command surface + on-device persistence; settings/training payloads derive from schema codegen, not hand-defined tables. `InputChain`/`OutputChain` firmware wiring (L29) lands here or gets its comment softened now.
- **5e Training-health telemetry (critic gap 5; §7.3) — BURNED DOWN 2026-07-21.** `int
nisps_ml_loss_history(ml, out, max)` landed across all five layers (KEEPALIVE →
`build-wasm.sh` exports → `NispsModule` decl → `WasmIML.lossHistory()` → `EngineApi
.lossHistory()`); it returns the TOTAL entry count and fills `min(count, max)`, so a
`max=0` probe sizes the JS buffer from C++ truth instead of mirroring the history cap.
The worker's `new Float32Array([loss])` is replaced by a readback off the worker's OWN
mirror handle, so async fits publish a real curve too. Display: `manifold/src/console/
TrainingHealth.tsx` at the Learning drawer's `expanded` depth — the flag mechanism is the
existing `DrawerDepth`, not a new one. The firmware buffer stays untouched (operator, L25).
**Two deviations.** (1) `ConsoleCtx.loss` was deleted as well: it was a synthetic series
(`evalLoss()` when finite, else `prev * 0.82`, else literal `0.5`) that no drawer read —
the §6.5e bar ("nothing may still fabricate a number") reaches it even though the audit
filed it as merely-unfinished plumbing under S19. (2) `<GradientFlow>` from dock-spec §1.3
is NOT built and should not be: the core records no per-layer gradient magnitudes, so it
could only be fabricated. Evidence beyond the standard gates, which do not cover this
path: `manifold/tests/loss-history.test.ts` (C-ABI contract driven straight at the
committed WASM under `bun test`) and two new `probe-api` cases + `tests/e2e/
training-health.spec.ts` (browser, both train paths, panel content).
- **5f Performance measurement (critic gap 4) — BURNED DOWN 2026-07-21.** The size half landed
with Phase 4 (firmware CI reports per-variant flash/RAM). The time half is
`tests/cpp/engine_bench.cpp` + `scripts/bench-engines.sh`: per-engine ns/sample, blocks/s and
realtime factor for the `process()` hot path, on native and WASM.
**Four decisions worth recording.** (1) *One source, compiled twice.* The bench is compiled
natively by CMake (`nisps_engine_bench`) and by emcc for WASM, from the same file — so the two
targets are comparable, `nisps/wasm/bindings.cpp` gains no export, and the 5-layer WASM export
chain is not walked at all. Nothing under `nisps/` changed except `CMakeLists.txt`: the hot path
is not perturbed by measuring it. (2) *Reporting, not asserting.* No threshold anywhere. On
shared CI hardware a wall-clock threshold is either slack enough to be meaningless or tight
enough to flake — the same call the firmware size job made. A regression gets noticed three
ways: `--compare <report.json>` prints per-engine Δ% (noise floor ~±3% at default settings,
measured; the failure mode this exists to catch is 2-3x); CI prints the table per commit on
both targets; and `run-all-tests.sh` stage 6 runs a ~0.15 s native smoke so the bench cannot rot
the way the SelfTest variant did. (3) *Engines are driven.* Sequencers run with transport on and
their event queues drained per block (an undrained 64-slot queue makes `push` take a cheaper
path than production); paf_synth gets a `note_on` every 0.25 s (its envelope gates it to silence
otherwise, and silence in the delay line decays to denormals); the fx/analysis engines are fed a
noise+sine bed. Params are pseudo-random in [0.05, 0.95], not the parity harness's all-0.5 —
that vector is a degenerate corner and at 128 frames never reaches a sequencer tick. (4) *Each
row carries its own working-state evidence* (output RMS / event count / analysis feature sum,
chosen by engine kind, since breakor/elysiamorf/analysis emit silence by design), so a number
produced by an idle engine is visible in the table instead of merely plausible.
**Not done, and now the live half of ALIGNMENT defect 5:** these are HOST numbers. The
constraint is the RP2350 at 150 MHz, and no on-device timing exists.
## §7 Operator decisions needed
1. **S20 legacy feedback modes** (RandomiseOutputs/RandomiseMlp/Diffuse/on_drag): deletion reverses the explicit "keep for A/B comparison" in `docs/adr/rl-feedback-design.md` — delete (and amend the ADR) or keep?
2. **S4 DriverConfig** — **DECIDED + LANDED 2026-07-21.** Operator: "wire it up; firmware reads active mode's
config at mode start; mic/line becomes real." Done: `Mode::driver_config()` (now part of the `nisps::Mode`
concept) defaults to `engine().driver_config()` in `ModeBase`, overridable per mode via an optional
`on_driver_config()` CRTP hook — SoundAnalysisMIDIMode is the one user, because its audio *engine* is a silent
NoOp while its `AnalysisEngine` is what owns the microphone. `src/main.cpp` publishes the mode's sample rate
before the system clock is derived from it, and `setup1()` calls `AudioDriver::Setup(...)` with the mode's
config instead of the parameterless overload. `DriverConfig`'s member defaults were changed to memllib's
historical hardcoded values (line_level 3, output_volume 0.8) so a mode that declares nothing is a genuine
no-op. Codec clamping + sample-rate resolution live in the Arduino-free `glue/codec_config.hpp` and are
host-tested in `tests/cpp/test_mode_driver_config.cpp`.
3. **Telemetry** (§6.5e) — **DECIDED + LANDED 2026-07-21.** Operator: feature, browser-only,
behind the advanced-surface flag; fakes deleted; firmware loss-history buffer kept (L25).
Built as described in §6.5e above.
4. **Deploy gating** (§1.5): gate the webhook on CI, or accept ungated deploys knowingly.
5. **memllib ownership** (§5): fork-pin vs vendored subset vs upstreaming to MusicallyEmbodiedML.
6. **Curated-preset product model** (§6.5c): what a preset bundles; where curation lives.
7. **S12 inputs composition**: mix-and-match multi-source remains an unreversed 2026-06-28 decision with groundwork deliberately laid — schedule the UI, or keep dormant (spec relabelled either way).
## §8 Docs/specs disposition (executes alongside phases; ~1 day)
| Doc | Disposition |
|---|---|
| `aimmersive-clone-spec.md` | Archive (`_archive/`, deprecated-by note) — specs a retired product; strip chat residue (L42) |
| `feedback-modes-port-spec.md` | Reclassify `kind: plan, status: executed`; ADR + `feedback.hpp` are the truth (L43) |
| `plans/BUILD-PLAN.md` | `status: executed`; two-line survivors note (L44) |
| `plans/playground-2.0-rewrite-plan.md` | `status: superseded` (retired target) (S3) |
| `engine-architecture.md` | Excise dead playground2 sections; keep EngineApi-seam + spine contract with supersession header (S3, L45) |
| `MAIN.md` | Fix the ≥6 contradicted claims; update registry rows (S27) |
| `manifold-parity-features-spec.md` | Reclassify plan; mark Jolt/OU executed-by-other-means; presets/pins sections stay live feeding 5c (S28) |
| `vcv-module.md` + `vcv/README.md` + `NISPS-FORMAT.md` | Prune to the current 8→16 contract; one format spec, not two (S22, L46) |
| `inputs/backends/dock` spec trio | Mark grounding sections historical; fix dead cites; drop aimmersive counterpart links (L50) |
| `MAP.md` | Flatly-false lines fixed with this commit (phantom `MEMLCelium-upstream`, exploration.ts, daisysp "submodule", pre-P5 sentence, perf-attr "globally" claim); the rest moves with the phases that change the code (L2, L15/L31/L41, L49, ST7, ST15) |
| `AGENT-REFERENCE.md` | Pre-P5 sentences updated (L49) |
| `codegen/README.md` | Rewrite or shrink to a MAP pointer; drop `port-solidjs` CI trigger (ST10) |
| Root `NISPS_CORE_*` relics | Deleted in Phase 1 (L48) |
| `ALIGNMENT.md` | Rewritten with this commit (L30, L47, ST6 fold into their code phases) |
## §9 Effort summary
| Phase | Cost | Depends on |
|---|---|---|
| 0 Restore verification | ~half day | — (do first, includes the only data-loss risk) |
| 1 Dead-mass deletion | 2–3 days | 0 (green gates to delete against) |
| 2 Behaviour bugs | ~2 days | 0; independent of 1 |
| 3 Truth consolidation | 2–3 days | 1 (less mass to consolidate) |
| 4 PlatformIO | 2–3 days | 0.1 (memllib remote); §7.5 |
| 5 Vision builds | spec-first, weeks, incremental | 1–4 recommended; §7 decisions |
| Docs disposition | ~1 day | interleaved |
Phases 1–3 are safe, mostly-mechanical, and could be largely agent-driven with the existing gates. Phase 4 is bounded and proven in-repo. Phase 5 is where the product decisions live — each item deserves its own spec + session, in the order 5c → 5a → 5d (curation is the user-visible payoff; reunification derisks everything mode-shaped; the editor closes the hardware loop).

View file

@ -1,3 +1,9 @@
---
kind: finding
date: 2026-06-27
immutable: true
---
# Findings — Design + Manifold Console: build-oriented brief (Phase-1)
*Read-only synthesis, 2026-06-27, of the five redesign docs + the on-disk Manifold token export + the
@ -8,15 +14,15 @@
> claude.ai design project and must be pulled via DesignSync before Phase-3 build. `ConsoleApp.jsx` was read
> directly and is summarised in §2.
## Source map (`docs/redesign/`)
- `playground-2.0-rewrite-plan.md` (399ln) — authoritative SolidJS UX+architecture plan: Console IA, drawers,
## Source map (now in docs/specs/ and subdirs)
- `plans/playground-2.0-rewrite-plan.md` (399ln) — authoritative SolidJS UX+architecture plan: Console IA, drawers,
reactive spine, feature table, roadmap, open questions.
- `engine-architecture.md` (440ln) — one-engine-two-skins: `EngineApi` headless boundary, reuse table, COOP/COEP,
build sequence S0–S7.
- `aimmersive-clone-spec.md` (351ln) — faithful clone spec of deployed `a-immersive.html`: feature inventory,
exact tactile constants, state→store map, engine call points.
- `feedback-modes-port-spec.md` (548ln) — 3-mode "Down Action" ported to `nisps/` as `FeedbackController<MLP_T>`.
- `playground-2026.md` (341ln) — older design-intent doc (dock+drawer, interactive heatmap); reference-only.
- `recon/playground-2026.md` (341ln) — older design-intent doc (dock+drawer, interactive heatmap); reference-only.
> **Framework mismatch:** every planning doc targets **SolidJS**; the mission is a **React** port. The §4 spine
> translation is the largest design decision and biggest risk (§7).

View file

@ -1,3 +1,9 @@
---
kind: finding
date: 2026-06-27
immutable: true
---
# Findings — Engine Surface Audit (Phase-1)
*Read-only audit, 2026-06-27. VERIFIED from source unless `[INFER]`. Engine A = deployed vanilla JS live at

View file

@ -1,3 +1,9 @@
---
kind: finding
date: 2026-06-27
immutable: true
---
# Findings — Current RL / Feedback Behaviour (Phase-1 Audit)
*Read-only audit, 2026-06-27. Citations are `file:line`. "VERIFIED" = read in source; "INFER" = deduced.
@ -18,7 +24,7 @@ hook blocks subagent report-file writes. Adversarially spot-checked against sour
random Gaussian noise"** — undirected diffusion, not directed avoidance.
4. The true geometric push-away (move output away from k-NN centroid of liked examples) exists only in the
firmware submodule (`memllib` InterfaceRL; "geo push" commits) and is documented in
`docs/redesign/feedback-modes-port-spec.md` §1.1. **This is the algorithm Mode 1 must match** (workstream B).
`docs/specs/feedback-modes-port-spec.md` §1.1. **This is the algorithm Mode 1 must match** (workstream B).
5. **Why often inaudible (deployed AVOID):** noise added across all 4 layers of `MLP<2,10,14,18,126>`,
per-layer scaled `(1-spread)+spread/√fan_in`, capped at `noiseCap=0.12`, spread default 0.6 → tiny
per-step perturbation that the sigmoid output + output smoothing/slew pipeline absorb. One press moves the
@ -36,7 +42,7 @@ hook blocks subagent report-file writes. Adversarially spot-checked against sour
`LikeStore` (anchor) + `trainOnCurrent` (train). Missing in deployed build: the randomise-audition-then-anchor
loop; today thumbs-down only diffuses noise and never anchors.
10. The full port spec (component, WASM C API, TS FFI, ~22 ctest cases, parity stage) already exists at
`docs/redesign/feedback-modes-port-spec.md` and matches the untracked `feedback.hpp`.
`docs/specs/feedback-modes-port-spec.md` and matches the untracked `feedback.hpp`.
11. **Parity/perf constraints:** no heap / `std::array` only, per-instance deterministic `nisps::Rng`
(no libc `rand()`), no virtual dispatch, `.f` literals, fixed WASM arch `MLP<2,10,14,18,126>`,
native↔WASM parity within `1e-5`.
@ -153,6 +159,6 @@ prototype but commits at the *current* input position, not a chosen one, and lac
- WASM API: `nisps/wasm/bindings.cpp:419-437`
- TS: `playground/src/ml/wasm-iml.ts:488-508`; runtime `playground/src/modes/mode-runtime.ts:444-565`
- Exploration knobs: `playground/src/stores/exploration-store.ts`
- Port spec: `docs/redesign/feedback-modes-port-spec.md`
- Port spec: `docs/specs/feedback-modes-port-spec.md`
- Firmware geo-push: `src/memllib` (commits `2429bcc`, `d301cc7`; upstream `SB2026`; main `e291192`/`abe93ec`);
design docs on `origin/feat/feedback-explore-modes`: `docs/dislike_system_{analysis,design_space}.md`

View file

@ -0,0 +1,179 @@
---
kind: finding
date: 2026-07-25
immutable: true
---
# Findings — Manifold “Push away” vs upstream geometric dislike
_Read-only comparison, 2026-07-25. “Confirmed” claims cite source; “Inference” labels
interpretation. No implementation decision is made here._
## Scope and source identity
The hardware repository named in the question,
[`MusicallyEmbodiedML/MEMLNaut`](https://github.com/MusicallyEmbodiedML/MEMLNaut),
does not contain the active ML implementation. The firmware repository is
[`MusicallyEmbodiedML/MEMLNaut-NISPS`](https://github.com/MusicallyEmbodiedML/MEMLNaut-NISPS/tree/701f2d9f1b4698e0ddfa147193928489de12601f),
whose `src/memllib` submodule pins
[`MusicallyEmbodiedML/memllib` at `e291192`](https://github.com/MusicallyEmbodiedML/memllib/tree/e291192d8e4f2fca7b79670c4df9c2ec8bdf03cd).
The local read-only copies of upstream `InterfaceRL.hpp`, `.tpp`, and `.cpp` have Git
blob hashes `03e9255`, `9ec762b`, and `60204d1`, respectively—the same blobs returned
by GitHub for that pin. The comparison below is therefore against the exact upstream
source, not an approximation. Provenance is recorded locally in
`firmware/MEMLNaut-NISPS/lib/memllib/reference/README.md:1-15`.
## Confirmed behaviour
### The mental model is substantially right, but “different” needs a target
A negative verdict has no supervised label by itself. Upstream defines “different” as:
take the output that was heard, find the mean output of the four liked positions nearest
the current control input, and create a new target one unit farther from that liked
centroid in output space. With no likes—or a degenerate zero-length direction—it uses a
random direction. The target is clamped to `[0,1]`
([upstream `InterfaceRL.tpp:698-761`](https://github.com/MusicallyEmbodiedML/memllib/blob/e291192d8e4f2fca7b79670c4df9c2ec8bdf03cd/examples/InterfaceRL.tpp#L698-L761);
local mirror `firmware/MEMLNaut-NISPS/lib/memllib/reference/InterfaceRL.tpp:698-761`).
That target is then passed to `synthMapping.TrainBatch`. Inference subsequently reads
the same MLP through `synthMapping.GetOutput`. Therefore current upstream push-away is
**weight training on the mapping MLP**, not a separate rejection layer after inference
([upstream training call](https://github.com/MusicallyEmbodiedML/memllib/blob/e291192d8e4f2fca7b79670c4df9c2ec8bdf03cd/examples/InterfaceRL.tpp#L757-L761);
[upstream inference path](https://github.com/MusicallyEmbodiedML/memllib/blob/e291192d8e4f2fca7b79670c4df9c2ec8bdf03cd/examples/InterfaceRL.tpp#L869-L890)).
OU noise and `paramTransformHook` do exist after MLP inference, but are independent of
the dislike algorithm (`InterfaceRL.tpp:877-890`).
The Manifold path has the same basic semantics. The UI passes its current post-output-
pipeline vector to `FeedbackController.dislike`, which calls the WASM geometric-dislike
entry point (`manifold/src/console/ConsoleApp.tsx:509-535`;
`manifold/src/feedback/controller.ts:347-373`;
`manifold/src/engine/wasm-iml.ts:956-978`). The C++ controller computes the target and
calls `MLPCore::train_targets`, which runs forward propagation, backpropagation and one
RMSProp update on the same network weights (`nisps/ml/feedback.hpp:395-477`;
`nisps/ml/mlp.hpp:250-287`).
### The target maths now matches current upstream, but the training schedule does not
Current Manifold main now matches upstream’s untapered target formula and constants:
`kGeometricPushScale=1.0`, `kNegLRBase=1.5`, no `/(1+distance)` taper
(`nisps/ml/geo_push.hpp:1-110`; upstream `InterfaceRL.hpp:406-414` and
`InterfaceRL.tpp:723-761`). It also now uses the upstream RMSProp update rather than
interpreting an upstream RMSProp learning rate as plain SGD
(`nisps/ml/training.hpp:1-92`).
The remaining major divergence is dose:
- Upstream stores the dislike at press time, then its main loop calls `optimise()` on
subsequent cycles (`InterfaceRL.tpp:38-54,186-232`).
- Every optimisation scans **all** live negatives and batch-trains them again
(`InterfaceRL.tpp:673-761`).
- Upstream computes one liked centroid around the **current live control input** and
applies it to every negative in that cycle, even if the user has moved away from the
original disliked position (`InterfaceRL.tpp:698-755`). Manifold instead computes the
centroid at the just-pressed negative’s stored input (`nisps/ml/feedback.hpp:431-467`).
- A negative remains at full strength for 2500 ms, then expires; the number of updates
depends on the mode’s loop rate (`InterfaceRL.hpp:406-414`).
- Manifold collapses press and optimisation into **one synchronous
`train_targets` call for only the just-pressed negative**, then proportionally decays
stored negatives (`nisps/ml/feedback.hpp:395-477`; `nisps/ml/replay.hpp:165-185`).
There is no background/per-frame feedback optimiser.
Thus a Manifold click names a strongly displaced target, but takes only one step toward
it. Upstream keeps walking toward its target for the next 2.5 seconds. This is the most
direct explanation for a remaining perceptual strength difference.
As checked on 2026-07-25, the WASM served by
`https://meml.lnfinitemonkeys.org/next/nisps.wasm` has SHA-256
`d1c58a59517a00c6f51870ea1ec21194561b81058e22bbfb2e11de4af45c645a`, exactly matching
`manifold/public/nisps.wasm` on current main. The reported live behaviour therefore
cannot be explained by production still serving the pre-RMSProp or tapered binary.
### Upstream also cancels a nearby positive; Manifold does not
Upstream’s default replay policy is `REPLACE_10_PERCENT`
(`InterfaceRL.hpp:404-405`). When a negative is stored, a positive within input-space
distance `0.10` is removed before the negative is added
(`InterfaceRL.tpp:904-929,950-979`). Manifold’s replay method only deepens or adds a
negative and leaves positives intact (`nisps/ml/replay.hpp:102-121`). Manifold also
keeps liked examples in the separate MLP dataset
(`manifold/src/feedback/controller.ts:375-390`).
**Inference:** this is less about the first click’s amplitude than persistence. A later
positive training run can pull the mapping back toward a sound rejected near an
existing like, whereas upstream removes that local positive from its continuously
trained replay set.
### Current measured scale
On current main, the native behavioural benchmark at shape `2→16→16→16→8`, seed
`24301`, reports:
- one geometric dislike: at-point L2 movement `0.05335`;
- one legacy undirected Diffuse dislike: `0.22626`;
- repeated geometric dislikes: `0.05314` after 1, `0.33748` after 10, `1.09504`
after 100.
Commands:
```bash
scripts/bench-ml.sh --native-only --scenario A4_negative_once
scripts/bench-ml.sh --native-only --scenario D1_geo_anatomy
```
These numbers confirm that the current path is no longer inert, but also that a single
geometric press is still about 4.2× smaller than the legacy random-diffusion gesture in
this benchmark. They do not by themselves establish the right musical feel.
At Manifold’s current default PAF shape (`4→10→10→14→33`), the same seeded scenarios
report one-click movement `0.09654` geometric versus `0.20788` Diffuse (about 2.2×
smaller), and geometric movement `0.55935` after ten presses. These are vector L2
distances across 33 parameters, so they establish that weights move; they do not prove
that the affected parameters produce a perceptually obvious timbral change.
The more revealing PAF-shape `A12_like_then_dislike` journey dislikes exactly where a
liked target was taught. With one update, distance from that rejected liked target
changes from `0.38291` to `0.36030` (`rejection_moved=-0.02261`): the mapping moves, but
slightly **toward** the particular target the user just rejected. Ten updates change the
distance to `0.67843` (`rejection_moved=+0.29552`). This is deterministic evidence that
one update is not sufficient to realise the user-facing semantic in an important
contradictory-feedback case; it also motivates testing upstream’s nearby-like removal
separately.
## Why it likely felt weak
1. **Fixed today: optimiser mismatch.** Before `f57cddc`, the port used a tiny upstream
RMSProp learning rate inside plain SGD, reducing one press to roughly `5.3e-5`
movement.
2. **Fixed today: superseded push formula.** Before `ec31180`, the port halved the target
step, used one-third the negative-LR base, and tapered the step by distance—the exact
case upstream says made “no” ineffective.
3. **Still present: one update versus a time window of updates.** Manifold performs one
weight update per click; upstream replays all dislikes repeatedly for 2.5 seconds.
4. **Still present: highly asymmetric teaching dose.** A Manifold like trains the whole
positive dataset at `lr=1.0` for up to 1000 iterations, while a dislike takes one
roughly `0.0015` RMSProp step. Relative to the lurch caused by a like, correction
still feels small (`ALIGNMENT.md:66-84`).
5. **Potential persistence mismatch:** nearby likes are retained locally but removed by
upstream’s default replay policy, so future positive retraining may partly undo the
rejection.
## Recommended next experiment
Do not start by increasing `geo_lr` blindly. That can make a click stronger but does not
answer whether the intended interaction is one discrete correction or a short-lived
repulsive constraint.
Add a deterministic “advance feedback by `dt`” core seam and benchmark three matched
variants from the same seeded prefix:
1. current one-shot update;
2. upstream-style replay of all negatives for 2500 ms, while separately testing whether
centroid lookup follows each stored negative or upstream’s current live input;
3. a bounded discrete equivalent (for example 10/25/50 steps at press time) that avoids
wall-clock ownership in the core.
For each, report at-point movement, neighbourhood rings, global blast ratio, collateral
movement at liked positions, and like→dislike→like persistence. Then perform an
in-Manifold blind A/B at the real per-mode output arities and choose the smallest dose
that makes one rejection obvious without damaging liked regions. Test the nearby-like
removal policy as a separate axis rather than coupling it to dose.

View file

@ -0,0 +1,126 @@
---
kind: finding
date: 2026-07-21
---
# memllib usage inventory — the prerequisite for the §7.5 ownership decision
_Prerequisite named by `../plans/simplification-plan.md` §5: "inventory which memllib surface
(AudioDriver, MEMLNaut board, display/menu, MIDI) is load-bearing" before choosing between a
fork-pin and vendoring. The operator's §7.5 decision is **vendor the parts we use** ("everything
should be self contained in this repo"). This document is the evidence that decision needs, plus one
consequence that was not visible when it was made._
## Method
Not grep. The firmware was built for real (`arduino-cli`, `rp2040:rp2040:solderparty_rp2350_stamp_xl`,
SLPWorkshop variant) and the **linker map** was read: `MEMLNaut-NISPS.ino.map` distinguishes
translation units that merely compiled from those contributing sections to the final image. This is
the same method that would have caught the daisysp finding (S8) mechanically instead of by reading.
## Result: the compile surface is 24 TUs, and all 24 are linked
| subdir | .cpp linked / present | headers | size |
|---|---|---|---|
| `audio/` | 4 / 4 | 12 | 132K |
| `hardware/` | 7 / 7 | 21 | 176K |
| `interface/` | 8 / 8 | 8 | 108K |
| `synth/` | 3 / 3 | 10 | 1.3M |
| `utils/` | 2 / 2 | 9 | 80K |
| **total** | **24 / 24** | **60** | **~1.8M** |
Every translation unit reachable through the sketch-tree symlinks contributes kept sections to the
image — **no discarded-only TU exists**. memllib is therefore *not* a daisysp situation (41 .cpp
compiled, zero consumers). It is genuinely load-bearing.
The library has 30 `.cpp` in total; the 6 not counted above live outside the five symlinked subdirs
(e.g. `examples/`) and are never compiled by the firmware build. Those are the only part safe to
drop outright.
Firmware's direct include surface is just six entry points — `audio/AudioDriver.hpp`,
`hardware/memlnaut/MEMLNaut.hpp`, `hardware/memlnaut/Pins.hpp`,
`hardware/memlnaut/display/{View,SingleSelectView}.hpp`, `interface/MIDIInOut.hpp`, `PicoDefs.hpp`,
`utils/perf.hpp` — but the transitive closure is the whole of those five subdirs.
**Consequence for vendoring: "the load-bearing subset" is ~1.8 MB / 84 files, i.e. essentially all
of memllib minus `examples/`.** There is no small subset to lift. `synth/` alone is 1.3 MB (mostly
`maximilian.cpp`) and all three of its TUs link.
## The fork divergence — and the correction to this document's first version
`src/memllib` is not a third-party dependency. It is **the lab's shared library**
(`MusicallyEmbodiedML/memllib`), and our fork was 3 commits ahead / 31 behind.
**The first version of this document (commit `7a30da9`) drew the wrong conclusion from that, and
recommended a rebase-then-vendor on the strength of it. Two facts checked afterwards overturn it:**
1. **All three of our commits touch only `examples/`** — `bf9691c` (swap memlp includes for
nisps/core, inline RL utilities), `32cc831` (newlib RNG fix), `b37fc53` (seed-helper rename).
`examples/` is **not in the sketch symlink forest** (`firmware/MEMLNaut-NISPS/src/` symlinks
exactly `audio hardware interface synth utils PicoDefs.hpp`), so it is never compiled. Those
commits existed to let the RL code build against `nisps/core` **while it was being ported**, and
that port is finished: `nisps/ml/{jolt,ou_noise,feedback,geo_push}.hpp` cite the upstream sources
directly. So there is nothing of ours to carry forward, and **no rebase to perform** — vendoring
the five linked subdirs drops `examples/` and our three commits with it.
2. **Two of the "31 commits of work we are missing" were already absorbed.** `nisps/ml/jolt.hpp:3`
cites `9fcd459 "jolts"` and `nisps/ml/ou_noise.hpp:3` cites `d0d8a72 "noise"` as their sources.
The alarm in the first version — "the 31 commits contain work that sounds directly relevant" —
was therefore overstated for exactly the two commits it named.
So the real question was never "rebase or not" but **"which snapshot do we vendor: the pinned
`188496d` we build today, or current upstream?"**
## The staleness was already costing us
Upstream `main` **has `DisplayDriver::NavigateToView`** (`display/DisplayDriver.hpp:54`); the pinned
commit does not. The SelfTest firmware variant called it and had been failing to compile — fixed in
`b953681` by routing around the missing method. That variant was not written against broken code; it
was written against a **newer memllib than the pin**. The gap is not theoretical.
Upstream also carries `e291192 "l r input swap"`, a **hardware bug fix**: the physical L/R input
sockets are wired to the opposite codec ADC channels, so `AudioDriver.cpp` now swaps them at the
lowest level. Every mode on the pinned commit sees its stereo input backwards.
## Verified: current upstream builds, and costs almost nothing
Submodule moved to `e291192` (upstream `main`).
**CORRECTION (2026-07-21, same day).** The first version of this table, and the numbers in commit
`c19d846`'s message, were WRONG — SLPWorkshop 145348, PAFSynth 145300, SelfTest 141840, with a
"+316 uniform delta". They were produced by building three variants in a row through a **shared
incremental `arduino-cli` build directory** (`/tmp/memlnaut-firmware-build`), which reused stale
objects and under-reported by roughly 75 KB. The error surfaced when the PlatformIO migration could
not reproduce them; a clean-cache rebuild of the identical commit in a throwaway worktree confirmed
it. **Never measure firmware size through a reused build directory.**
Measured properly (clean build dir per measurement, SelfTest variant):
| memllib pin | flash | RAM |
|---|---|---|
| `b37fc53` (old pin, pre-bump) | 216520 | 18480 |
| `e291192` (upstream main) | 216736 | 18492 |
| **delta** | **+216** | **+12** |
The conclusion is unchanged and if anything stronger: the 31-commit upstream bump costs ~216 bytes
of a 16 MB flash. The bulky new upstream code (`GrainDelayI16`, `ReverbI16`, `ModFXI16`,
`CCSelectView`, `NameInputView`, `RLView`, `VUMeterView`, `PSRAMManager`) is header-only and
unreferenced, so the linker drops all of it; the delta is the AudioDriver/DisplayDriver changes.
Absolute sizes for every variant, post-bump, are in the Phase 4 migration commit — all 16 build,
and none exceeds 2% of flash.
Exactly **one** compile error had to be fixed: `MEMLNaut-NISPS.ino:169` used `kSampleRate` in a
`constexpr`, and upstream `1997699 "mode sample rate"` made it a runtime `extern size_t` so a mode
can choose its own rate. `constexpr` → `const`; it is a once-per-second diagnostic print.
## Where this leaves the vendoring
Vendor **from current upstream**, not from the old pin. Take the five linked subdirs +
`PicoDefs.hpp`; drop `examples/` (~1.8 MB → the vendored surface). Record the exact upstream commit
so a future re-sync is a documented diff rather than an archaeology exercise.
The submodule now points at **upstream** rather than the fork: with the pin moved to an upstream
commit, the fork has nothing the firmware compiles. The fork's `feat/nisps-core-swap` branch remains
pushed, so nothing is destroyed.
This unblocks the PlatformIO migration (plan §5).

View file

@ -1,8 +1,14 @@
---
kind: finding
date: 2026-06-27
immutable: true
---
# Work log — MIDI + Game Controller inputs, N-D engine foundation
*Scope: what was actually built on the `feat/midi-inputs` branch. This is a
description of the work, not a spec. The design intent lives in
`docs/redesign/inputs-spec.md`; where this branch diverges from or only partially
`docs/specs/inputs-spec.md`; where this branch diverges from or only partially
realises that spec, it is called out below.*
## Summary

View file

@ -1,10 +1,16 @@
---
kind: finding
date: 2026-04-12
immutable: true
---
# Playground Redesign — 2026 (Design Intent)
> **Purpose of this doc.** Captures design intent of an unfinished playground UI
> redesign that lived as uncommitted changes in a `MEMLNaut-NISPS_rewound`
> working copy. The implementation will not be merged into the current vanilla
> playground — the playground is being rewritten in SolidJS — but the *design*
> shouldn't be lost. Read this as a feature spec for the SolidJS rewrite, not as
> shouldn't be lost. Read this as a reference for the SolidJS rewrite, not as
> a description of current code.
>
> **Source snapshot.** `archive/playground-redesign-2026-snapshot` branch in
@ -12,7 +18,7 @@
> `playground/a-immersive.html`, `playground/css/a-immersive.css`,
> `playground/js/a-app.js`. ~1200 lines net of changes.
>
> **Status.** Reference-only. Do not merge into the vanilla playground — those
> **Status.** Reference-only, immutable. Do not merge into the vanilla playground — those
> files have moved on independently in `main` and a textual merge would be
> meaningless.

View file

@ -0,0 +1,298 @@
---
kind: finding
date: 2026-07-21
immutable: true
---
# Simplification Audit 2026-07 — full-repo smell/bloat/spec review vs the one-core vision
_Dated 2026-07-21. Method: a 66-agent workflow — 11 parallel subsystem reviewers (ml-core, dsp-engines-modes, wasm-bridge, manifold-ui, manifold-backends-inputs, firmware, schemas-codegen, specs-docs, tests-build, satellites-hygiene, architecture-gap) → adversarial verification (per-finding skeptics for architectural/structural claims, batch spot-checks for local/style) → a completeness critic. 113 findings survived, 0 were refuted outright, 22 were downgraded/corrected by verifiers. Yardstick: the operator's five-bullet target vision (2026-07-20): (1) one C++20 NISPS core serving RP2350 firmware and the browser, performance-sensitive on the MCU; (2) firmware modes runnable as modes in Manifold; (3) curated-presets default with a maximalist "advanced" dev mode; (4) PlatformIO, no more .ino; (5) Manifold as interface/editor for the hardware. Actionable mitigations are phased in `../plans/simplification-plan.md`; the strategic tier is promoted into `ALIGNMENT.md` (both updated in the same commit as this finding)._
## Headline
1. **CI has been 100% red on main since 2026-07-13** and nothing noticed. The memllib submodule pin (`b37fc53`, branch `feat/nisps-core-swap`) is reachable from **no remote** — GitHub's `submodules: recursive` checkout fails before any gate runs, so every P4–P6 "gates green" claim rests on local runs only, and fresh clones cannot build firmware. Verified live via `gh run list` (all recent main runs `failure`) and `git branch -r --contains b37fc53` (empty).
2. **Those three memllib commits exist only on one disk.** A single disk failure loses firmware-critical code. (The `w1n5t0n` fork remote exists but does not contain them.)
3. **Production deploys are gated by nothing.** Push to main → VPS webhook → live at `meml.lnfinitemonkeys.org/next/`, with CI red and the committed `manifold/public/nisps.wasm` having no freshness gate — what users run is whatever was last committed, unverified.
4. **Real behaviour bugs surfaced**, not just smells: the TS example-store mirror caps at 100 while the C++ store caps at 128, so `train()` and `trainAsync()` silently train on different datasets past 100 examples (plus a latent OOB read on sample weights); VCV bridged mode truncates the input vector to 2-D against a 32-input head; the VCV plugin has an audio-thread data race and does full-state JSON on the audio thread.
5. **The five-bullet vision has zero footprint in ALIGNMENT.md or the spec corpus** — nothing records PlatformIO, curated/advanced, or the hardware-editor role as goals, so nothing was steering toward them.
6. **No performance measurement exists anywhere** — the headline "performance-sensitive" constraint is enforced entirely by static discipline (no heap/virtual, section attrs) and vibes; no benchmark, no size report, no CPU-load assertion on either target.
The codebase itself is in **better shape than its vibe-coded provenance suggests**: the `nisps/` core's storage-policy split is justified and parity-pinned, hot paths are clean, the CRTP mode scaffold genuinely centralises, the backend/input contracts are uniform. The dominant rot is *vestigial mass* (dead subsystems, decorative UI, executed-plan specs presented as live contracts, hand-maintained registries that codegen should own) — most fixes are deletions.
## Area verdicts
**ml-core** — The ml-core area (nisps/core, nisps/ml, nisps/pipeline) is genuinely one of the healthier parts of this codebase: the MLPCore<Storage> policy split is justified (two real consumers — zero-heap firmware FixedStorage, arena-allocated browser/VCV DynamicStorage — sharing one algorithm body, pinned by a bit-parity test), hot paths are heap-free, virtual-free, .f-consistent, and RNG is deterministic per instance. Jolt/OU/replay/geo_push are small, inert-by-default, and cleanly ported. The rot is concentrated in vestigial baggage rather than wrong abstractions: feedback.hpp carries two whole legacy […]
**dsp-engines-modes** — The dsp/engines/modes stack is in better shape than the vibe-coded provenance suggests: the DSP primitives are lean, faithful ports with no maximilian/daisysp vestiges beyond honest attribution comments; all 8 engines have real consumers on both targets; the CRTP ModeBase genuinely centralizes orchestration (inputs, jolt, OU, events) rather than hiding duplication. The biggest gap versus the vision is architectural: the modes/ layer is consumed ONLY by firmware — the WASM bindings include engines but zero mode headers and re-assemble mode behavior (jolt stepping, OU, routing) in bindings.cpp […]
**wasm-bridge** — The wasm-bridge is in better shape than its history suggests: the P3 'SWAP POINT' duplication is genuinely resolved (jolt.ts/ou-explore.ts are deleted; exploration.ts is now a thin timer-driver over the shared C++ core), the flat C API in bindings.cpp is stylistically coherent (consistent null-guards, grouped, documented wire layouts), and worklet messaging is RT-acceptable (postMessage between render quanta, fixed-size param buffer, no allocation inside process() beyond trivial subarray views). The real problems are accretion and dead mass: roughly a dozen API entries exist across a 5-file […]
**manifold-ui** — The manifold UI is functional and the engine wiring (FeedbackController → shared C++ core, backend manager, input layer) is genuinely live, but the console layer carries heavy sediment from its window-global JSX ancestry: a dead four-way "focus/altitude" system whose only reachable branch is 'composite' (SplitStage, ReadoutStrip, InputMini, AltitudeNav all unreachable), a large stratum of decorative state that renders real-looking controls doing nothing (learning-rate/decay/spread/tame sliders, master volume, bpm, A/B toggle, snapshots, health/gradient visuals), and a ConsoleCtx god-object […]
**manifold-backends-inputs** — This is one of the cleaner vibe-coded areas: the OutputBackend contract is genuinely uniform (start/send/teardown/status with in-backend throttle+dead-zone), mapping.ts is a single shared truth, presets.ts is real and consumed by OutputsBackendConfig (not a stub), and midi-devices/ is proper schema→codegen with one import path. The real debts are: (1) the VCV bridged-training path is pinned to an obsolete 2-D input assumption while the engine has a 32-input head and the spine already stores the full N-dim raw vector — so VCV mode silently truncates gamepad/MIDI inputs; (2) the OSC bridge […]
**firmware** — The firmware C++ glue itself is in decent shape — it is thin (~1.4k lines), template-based with no virtual dispatch in the audio path, the SPSC/dual-core discipline is respected, and peripherals.hpp/selftest.hpp are unusually well-commented and readable for vibe-coded work. The rot is concentrated in the build system, which is exactly what vision bullet 4 targets: the Arduino-CLI toolchain forces a symlink forest, a build script that mutates the committed .ino (variant flips pollute git history), a triple-bookkept mode list (the .ino comment block, mode_select.hpp aliases, and a […]
**schemas-codegen** — The schemas→codegen pipeline is one of the better-built parts of this vibe-coded repo: both generators validate via ajv, emit deterministic/idempotent output, and the generated C++ is genuinely sane (inline constexpr arrays, string_views, .f literals, no heap — spot-checked against the perf contract). But the "schemas are the single source of truth" story is only about half true in practice: net dimensions are hand-duplicated in every C++ mode's MLP template args with no compile-time tie back to the generated constants (browser reshapes from schema, firmware uses the hand-written dims — the […]
**specs-docs** — The corpus has an unusually good ontology (spec/plan/finding/ADR split, _archive dir, front-matter kinds) and MAP.md is ~90% accurate and current — but the July one-core refactor (P4–P6, landed 2026-07-14/18) rotted the registry layer faster than the doc-sync discipline caught it. MAIN.md, the corpus entry point, now makes at least six false or self-contradictory claims; two 2026-06-era specs describe the RETIRED SolidJS playground world as present tense; both plans that are done still carry status:active in ways the corpus's own ontology forbids (BUILD-PLAN) or defensibly (one-core, gated on […]
**tests-build** — The test estate is in genuinely good shape at its core: 42 focused C++ test files with a documented hand-rolled harness, a 7-stage native-vs-WASM parity blob, committed golden baselines (impulse, ML vectors), fixed↔dynamic storage parity, VCV-adapter parity, and post-P4 pipeline-golden fixtures whose provenance and re-baselining are exemplary — the fixtures are meaningful, not frozen accidents. The problems cluster at the edges: an entirely dead root-level Playwright rig left over from the retired playground (config points at an HTML file that no longer exists anywhere in the tree, and […]
**satellites-hygiene** — The satellites are in better shape than their paperwork. Post-P6, vcv/src/iml.hpp genuinely is a thin adapter over MLPCore<DynamicStorage> (pinned bit-exact by tests/cpp/test_vcv_iml_parity.cpp, wired into CMake), and keeping the module in-repo is coherent since it builds against ../../nisps directly; the one defensible wart is the insertion-order example mirror (features_/labels_), which carries a named requirement (JSON persistence + cross-thread staging) and is acceptable. firmware/useq-celium is the cleanest corner of the repo: I byte-verified the protocol.h ↔ useq-protocol.ts mirror […]
**architecture-gap** — Scored against the 5-bullet vision: (1) "one C++ core" is ~80% real — the algorithm layer (ML, feedback, jolt/OU, pipelines, curves) genuinely lives once in nisps/ after the P0–P6 refactor, but the MODE layer does not: nisps/modes/ never compiles to WASM, so the browser recomposes engine+ML in TS and every ModeBase behaviour (event ring, voice spaces, note_on/BPM, analysis feature-splicing, pinning) is firmware-only. (2) Consequently "firmware modes runnable in Manifold" is ~40%: 4 of the 9 catalogued browser modes are non-functional (sequencers emit events nobody drains; audio-in modes have […]
## Completeness-critic gaps
_A final agent asked "what did the audit miss?". Verdict: file-tree coverage was strong; the systematic blind spot was everywhere the repo touches the outside world._
**CI has been entirely red on main since 2026-07-13 and no auditor noticed.** Verified live: `gh run list` shows 8/8 recent runs on main FAILED, spanning every P3–P6 push (2026-07-14 through 2026-07-18). Cause confirmed from run 29641902005: the cpp-tests job dies at the Checkout step — 'remote error: upload-pack: not our ref b37fc53…' — because ci.yml uses `submodules: recursive` and the memllib gitlink was re-pinned (commit 43ef6d5, 2026-07-13) to a commit that exists on no remote. The vcv-plugin workflow fails identically. The firmware area found the unreachable pin ('fresh clones cannot build firmware') and tests-build criticized CI *content* drift, but the compound fact is worse than either: the entire C++ gate […]
**The unpushed memllib fork is a data-loss single point of failure, and its ownership question is unasked.** The three commits on memllib's local `feat/nisps-core-swap` branch (bf9691c 'swap memlp includes for nisps/core, inline RL utilities', 32cc831 newlib RNG fix, b37fc53 seed-helper rename) are NISPS-load-bearing source that exists ONLY in this working copy — `git ls-remote` of MusicallyEmbodiedML/memllib shows no ref containing them, and the operator has no fork remote configured. One disk failure loses firmware-critical code. The audit framed this purely as a build blocker; the trivial mitigation (push the branch to a monkey-w1n5t0n/w1n5t0n Forgejo or GitHub fork and repoint .gitmodules) was never proposed. The deeper unasked question matters […]
**The production deploy pipeline was audited by nobody — live deploys are gated by nothing.** manifold/ONBOARDING.md:43 documents it: push to GitHub main → VPS webhook → builds manifold/ → rsyncs to the live meml.lnfinitemonkeys.org/next/. Three consequences no area examined: (a) deploys do not wait for CI — and CI is red anyway (gap 1), so every P4–P6 push went straight to the live site with zero automated gate between a nisps/ change and production; (b) the webhook builds only manifold/, so vite copies the COMMITTED manifold/public/nisps.wasm into dist — the 'stale artifact, no freshness gate' finding that tests-build scoped to browser tests actually extends to what users run in production; (c) the webhook config itself (on the […]
**No performance measurement exists anywhere, though 'super performance-sensitive on the microcontroller' is the vision's first bullet.** Every area checked perf DISCIPLINE statically (no heap, no virtual, .f literals, section attrs) but nobody asked how the project would ever KNOW the audio path meets budget or regressed: grep across scripts/ and tests/cpp/ finds zero benchmark harness, no cycle/CPU-load assertions, no blocks-per-second smoke for engine_process_block (native or WASM), and no firmware flash/RAM budget report in any gate — the 16 KB dead loss-history buffer was found by reading, not by a size report that would have caught it mechanically. The only perf scaffolding is on-device (src/memllib/utils/perf.hpp, included by the .ino) with no automated consumer. A […]
**Training-health telemetry is one product decision fragmented into four uncoordinated findings.** Four areas each hold one limb of the same feature: ml-core flags the 16 KB loss-history buffer in every firmware MLP (nothing reads it); specs-docs verifies wasm-worker.ts:304 fakes a 1-element lossHistory; manifold-ui flags the health/gradient visuals as decorative; architecture-gap notes the real get_layer_stats API is plumbed end-to-end and unused. Nobody asked the single question that resolves all four: IS training-health telemetry a feature of this instrument? If yes (plausible — 'interactive control visualisation' is in vision bullet 5 and 'advanced mode as preset-authoring dev tool' in bullet 3 wants training introspection), the fix […]
**The osc-bridge release workflow already names the surviving bridge — and two of three workflows were reviewed by nobody.** tests-build audited only ci.yml, but .github/workflows/ contains three workflows. osc-bridge.yml (tag-triggered) deno-compiles ONLY manifold/osc-bridge/bridge.ts into cross-platform release binaries — that is the named external consumer that settles the manifold-backends 'two hand-maintained bridge implementations' finding: bridge.ts has a release pipeline, bridge.mjs has no consumer in any workflow, so the mjs twin is the deletion candidate. vcv-plugin.yml was also unreviewed: it fails on the same unreachable submodule pin as ci.yml (verified in the 2026-07-18 run), and still triggers on stale feat/manifold-mission / feat/vcv-dist branch […]
## Findings inventory
_`confirmed` = an adversarial verifier re-read the cited code and the claim held; `downgraded` = the claim held but severity or fix needed correction (fix shown is the verifier's). IDs are referenced from `../plans/simplification-plan.md`; evidence lives at the cited lines._
### Architectural (8)
**A1. Mode layer is not shared: nisps::modes never compiles to WASM; Manifold rebuilds modes in TS** (architecture-gap, downgraded, week+)
`nisps/wasm/bindings.cpp:54`, `nisps/modes/base.hpp:113`, `manifold/src/console/model.ts:199`, `manifold/src/engine/spine.ts:1`
bindings.cpp includes nisps/engines/* and nisps/ml/* but nothing from nisps/modes/ (includes at lines 54-77; engine dispatch at 267-275 instantiates bare engines). grep for events/voice_space/note_on/update_bpm in bindings.cpp returns nothing. So ModeBase behaviours — initial inference at setup […]
*Fix:* Reunify mode BEHAVIOUR in C++ without pretending the browser can host one monolithic mode object: (1) storage-policy ModeBase exactly as P2 did for MLPCore — ModeBase<Derived, EngineT, MLCore<Storage>, ...> with Fixed for firmware (zero-heap intact) and Dynamic for WASM (heap at construction only), so browser modes stay runtime-reshapeable; (2) split the mode C ABI along the […]
**A2. Manifold catalogues 4 modes that structurally cannot work in the browser** (architecture-gap, downgraded, days)
`manifold/src/console/model.ts:206`, `nisps/engines/breakor.hpp:123`, `manifold/src/engine/engine-host.ts:1`, `manifold/src/engine/worklet/nisps-processor.ts:287`
SCHEMA_MODES (model.ts:199-221) lists all 9 schema modes including breakor, elysiamorf (sequencers: engine audio is silence; their whole output is pop_events, breakor.hpp:123-128) and xiasri, sound_analysis_midi (audio-in). But: (a) no WASM binding drains engine events and no manifold code […]
*Fix:* Immediate (hours, manifold-only): add a manifold-side audio-topology map keyed by engineId in model.ts — 'generator' (paf_synth, memlcelium→also slp_workshop, elysiamorf? no: event-only) vs 'audio-in-fx' (channel_strip, verb_fx, xiasri) vs 'event-only' (breakor, elysiamorf) vs 'analysis' (sound_analysis_midi) — and surface it as a 'firmware-only'/'needs audio-in' badge on […]
**A3. No curated/advanced split exists; Manifold is 100% dev-maximalist with no preset data model to author against** (architecture-gap, downgraded, week+)
`manifold/src/settings/settings-store.ts:24`, `manifold/src/backends/presets.ts:30`, `manifold/src/console/ConsoleApp.tsx:105`
Grep for advanced/curated across manifold/src finds only BackendAdvanced.tsx (per-backend transport settings) — no global mode toggle. settings-store.ts:24-40 holds only cosmetic prefs (icon colour, input-map shape, corner radius). The only preset machinery is presets.ts: per-BACKEND output-routing […]
*Fix:* Split into three differently-sized items. (a) NOW, small bug: add `cv` to OutputPresetRow, rowsFromParams, and applyPreset in manifold/src/backends/presets.ts (three one-line additions). (b) IF/WHEN the curated-default direction is actually adopted, first record it in ALIGNMENT.md (it currently contradicts open question Q2) — then build the unified preset artifact by EXTENDING […]
**A4. Hardware-editor (vision 5) is a facade: serial stub in the UI, zero firmware command surface, zero on-device persistence** (architecture-gap, downgraded, week+)
`manifold/src/serial/memlnaut-serial.ts:102`, `manifold/src/console/Drawers.tsx:590`, `firmware/MEMLNaut-NISPS/MEMLNaut-NISPS.ino:126`
memlnaut-serial.ts saveModel/restoreModel/getSettings are explicit no-op stubs (lines 102-132: 'TODO: real protocol', returns false/null/{}), and git log shows manifold/src/serial untouched since the June initial commit. EditorPanel is shipped live in the Settings drawer (Drawers.tsx:590) with […]
*Fix:* Drop the short-term step — the 'not wired' badge already exists in EditorPanel.tsx:88-94; at most soften the dock description in output-mode.ts:79 ('configure / save / restore — protocol not yet implemented'). Real path stands as proposed: spec the USB-serial protocol the useq-celium way (shared versioned header under firmware/, mirrored TS + frame-size unit test), add a […]
**A5. modes/ layer has one consumer: WASM bindings bypass it and re-orchestrate modes from engines + ml primitives** (dsp-engines-modes, downgraded, days)
`nisps/wasm/bindings.cpp:54`, `nisps/wasm/bindings.cpp:844`, `nisps/modes/base.hpp:311`, `firmware/MEMLNaut-NISPS/glue/mode_select.hpp:41`
bindings.cpp includes all 8 engine headers (lines 54-62) and ml/jolt.hpp, ml/ou_noise.hpp directly, but not a single nisps/modes/*.hpp; only firmware glue (mode_select.hpp:41-50) consumes the mode classes. bindings.cpp:844-851 (nisps_ml_jolt_step) is a line-for-line duplicate of […]
*Fix:* Do NOT bind the mode classes into WASM and do NOT delete the per-primitive jolt/OU C API — both contradict the locked one-core-engine architecture (runtime-shaped browser MLP, P2; primitives-via-C-ABI, P3) and the two-WASM-instance RT split. Instead: (1) collapse the one true code duplication by moving the copy→morph→writeback sequence into the shared primitive — e.g. a […]
**A6. Arduino-CLI/.ino build machinery: build mutates committed source, triple-bookkept mode list, symlink forest — PlatformIO migration deletes ~400 lines of hackery** (firmware, confirmed, days)
`scripts/firmware-common.sh:192-208`, `scripts/firmware-common.sh:298-337`, `firmware/MEMLNaut-NISPS/MEMLNaut-NISPS.ino:37-59`, `firmware/MEMLNaut-NISPS/glue/mode_select.hpp:90-106`
Mode selection lives in THREE hand-synced places: the .ino comment block (lines 38-54) that firmware-common.sh greps (load_firmware_variants:192-208), the mode_select.hpp using-aliases (54-85), and the NISPS_ST_* macro table (90-104). set_firmware_variant (298-337) runs an inline Python script that […]
*Fix:* Migrate to PlatformIO in one cut, no dual path: platformio.ini with maxgerhardt/platform-raspberrypi + board_build.core=earlephilhower (the FQBN solderparty_rp2350_stamp_xl proves the variant exists in arduino-pico; verify the PIO board def or add a small custom board JSON). One [env] per firmware variant with build_flags = -std=gnu++20 -DMEMLNAUT_MODE_TYPE=<alias> (keep […]
**A7. No curated-default/advanced split and no in-UI instrument picker — UI structure fights the core vision** (manifold-ui, downgraded, days)
`manifold/src/console/ConsoleApp.tsx:355`, `manifold/src/console/ConsoleApp.tsx:61`, `manifold/src/settings/settings-store.ts:24`, `manifold/src/console/Drawers.tsx:1`
The vision requires curated presets/modes by default with maximalist settings behind an 'advanced' gate. Settings (settings-store.ts:24-38) contains only monochromeIcons/unfocusedIconColour/inputMap/cornerRadius — no advanced flag exists anywhere. The default surface is the full five-drawer console […]
*Fix:* Render the instrument-mode picker from the already-plumbed ctx.modes/setModeId (pure UI; engine reshape-on-mode-switch already works per P5; keep the window.__mf seam for e2e). Drop the global advancedMode boolean: progressive disclosure is already designed and partly built as the per-drawer depth ladder (dock-spec §0 peek/expand/FULL with FULL as the advanced page). If more […]
**A8. The operator's target vision (PlatformIO, curated-presets default vs advanced mode, Manifold as hardware editor) has zero footprint in ALIGNMENT or the spec corpus** (specs-docs, confirmed, hours)
`ALIGNMENT.md:48`, `docs/specs/MAIN.md:140`, `MAP.md:150`, `manifold/src/serial/memlnaut-serial.ts`
No spec, plan, ALIGNMENT defect, or open question mentions PlatformIO anywhere (grep -ri platformio docs/ MAP.md ALIGNMENT.md hits only firmware/useq-celium, which already uses it — proving the toolchain is in-house viable). MAP.md:150 codifies the .ino/Arduino-CLI flow as a Convention; ALIGNMENT […]
*Fix:* Add three dated ALIGNMENT entries (PlatformIO migration — killing the .ino sketch-tree symlink hacks MAP.md:161 documents; curated-presets/advanced-mode product model — promoting Q2 from question to defect and linking the pending manifold-parity presets plan; hardware-editor role for manifold/src/serial). Specs follow when each is picked up.
### Structural (35)
**S1. Mode identity is ~6 hand-maintained registries with demonstrated drift, not one source of truth** (architecture-gap, downgraded, day) — `manifold/src/console/model.ts:199`, `manifold/src/console/model.ts:324`
Adding a mode touches: schema JSON (canonical) + hand C++ mode (legit) PLUS (a) mode_select.hpp using-alias, (b) the NISPS_ST_* zero-table, […] *Fix:* Keep points (1), (4), (5) as proposed: codegen an ALL_MODE_SCHEMAS array (TS-side codegen template change; SCHEMA_MODES becomes a mode_id→overlay map, keeping hand label/glyph/cls/ordering, which are legitimately display truth); collapse […]
**S2. Vision 4 (PlatformIO) not started; the .ino build machinery is actively hostile (symlink farm, sed-mutates-source mode selection, macro undef dance)** (architecture-gap, downgraded, days) — `scripts/firmware-common.sh:311`, `firmware/MEMLNaut-NISPS/glue/mode_select.hpp:19`
Mode selection = build scripts regex-rewriting the .ino's #define line in place (firmware-common.sh:194-205 grep/sed discovery, :311 python […] *Fix:* Migrate firmware/MEMLNaut-NISPS to one platformio.ini with an env per variant passing -DMEMLNAUT_MODE_TYPE=<alias> (grep/sed/python rewrite machinery, the .ino comment registry, and the NISPS_ST_* SelfTest grep-discovery tag all deleted; […]
**S3. Two 'authoritative' specs still prescribe the retired SolidJS/playground2 world** (architecture-gap, downgraded, hours) — `docs/specs/engine-architecture.md:38`, `docs/specs/aimmersive-clone-spec.md:1`
engine-architecture.md (kind: spec — 'wins over code by intent' per MAIN.md §0) mandates building in a fresh `playground2/` tree with […] *Fix:* One commit, three coordinated edits. (1) engine-architecture.md: do not flatten to one page — excise the dead sections (§1 playground2 tree, §3 reuse-vs-rebuild, §4 two-entry Vite/nginx serving, §5 S0–S7, stale §6 open questions) and […]
**S4. DriverConfig is a dead contract: required by the AudioEngine concept, implemented by all 8 engines, consumed by nobody** (dsp-engines-modes, confirmed, day) — `nisps/core/concepts.hpp:62`, `nisps/core/types.hpp:59`
Repo-wide grep for driver_config/DriverConfig finds only the struct (types.hpp:59), the concept requirement (concepts.hpp:62), and the 8 […] *Fix:* Decide once: either wire the firmware audio driver to read the active mode's engine().driver_config() at mode start (makes the mic/line settings real), or delete DriverConfig, the concept line, and all 8 implementations. Do not keep the […]
**S5. Per-mode kSchema aggregates are hand-written 12-field boilerplate that codegen should emit** (dsp-engines-modes, confirmed, hours) — `nisps/modes/breakor.hpp:76`, `nisps/modes/memlcelium.hpp:48`
Nine mode headers each contain a mechanically identical positional ParamSchema aggregate wiring 12 generated constants (kXModeId, […] *Fix:* Move struct ParamSchema into generated/schema_types.hpp and have codegen emit 'inline constexpr ParamSchema kXSchema{...}' per mode; each mode's param_schema() becomes a one-line return of the generated constant. Delete the 9 hand-written […]
**S6. MLP topology is a dual source of truth: hand-typed template args vs generated hidden_layers** (dsp-engines-modes, confirmed, hours) — `nisps/modes/memlcelium.hpp:28`, `nisps/modes/generated/memlcelium_schema.hpp:19`
Every mode hardcodes its net shape (e.g. ml::MLP<4u,10u,14u,18u,56u> at memlcelium.hpp:28) while the generated header carries the same dims […] *Fix:* Have codegen emit 'using MemlceliumMLP = ::nisps::ml::MLP<4u,10u,14u,18u,56u>;' (or a static_assert block comparing MLP dims to kXHiddenLayers) into each generated header, and make modes use the alias. One source of truth, compile-time […]
**S7. memllib submodule pin is unreachable from any remote — fresh clones cannot build firmware** (firmware, confirmed, hours) — `.gitmodules:1-3`, `scripts/build-firmware-arch.sh:74-79`
git submodule status → b37fc53 (heads/feat/nisps-core-swap). Inside src/memllib, `git branch -r --contains HEAD` returns EMPTY — the pinned […] *Fix:* Push feat/nisps-core-swap to the monkey-w1n5t0n/memllib fork and change the .gitmodules URL to the fork (or upstream the changes to MusicallyEmbodiedML and re-pin). Then delete the compensating error-message paragraph in […]
**S8. daisysp is fully dead weight: 41 .cpp files compiled into every firmware build with zero consumers, and docs mislabel it a live submodule** (firmware, confirmed, hours) — `firmware/MEMLNaut-NISPS/src/daisysp`, `nisps/dsp/pitch_shift.hpp:3`
grep for daisysp across firmware glue, the .ino, nisps/, and the symlinked memllib subdirs (audio/hardware/interface/synth/utils) finds […] *Fix:* Delete the firmware/MEMLNaut-NISPS/src/daisysp symlink and the vendored src/daisysp tree; correct MAP.md, firmware/README.md, and setup-firmware-toolchain.sh in the same commit. (Answers the focus question: yes, the granular pitch-shift […]
**S9. Firmware compilation is in no automated gate: CI, run-all-tests.sh, and lint all skip firmware/** (firmware, confirmed, day) — `scripts/run-all-tests.sh`, `.github/workflows/ci.yml:14-15`
grep 'firmware' scripts/run-all-tests.sh → no hits. ci.yml comments that 'firmware build is documented as a manual […] *Fix:* After the PlatformIO migration, add a CI job running `pio run -e slpworkshop -e selftest -e pafsynth` with a cached toolchain (arduino-pico core download is the only heavy step). Pre-migration stopgap: a weekly/manual-dispatch arduino-cli […]
**S10. VCV bridged mode truncates the input vector to 2-D against a 32-input engine head** (manifold-backends-inputs, downgraded, day) — `manifold/src/engine/engine-api.ts:264`, `manifold/src/backends/vcv-backend.ts:67`
engine-api.ts inputVector() returns `[this.spine.lastRawX, this.spine.lastRawY]` (a fresh 2-element array allocated on EVERY spine tick […] *Fix:* In manifold TS only: (1) change EngineApi.inputVector() to return spine.lastRawInputs (signature ArrayLike<number>; document "live reused buffer — copy, don't retain"; VcvBackend.setInputVector already copies). (2) Make VcvBackend […]
**S11. Two hand-maintained OSC bridge implementations (bridge.ts Deno + bridge.mjs Node)** (manifold-backends-inputs, confirmed, hours) — `manifold/osc-bridge/bridge.ts:1`, `manifold/osc-bridge/bridge.mjs:1`
bridge.mjs is not compiled output — it is a separate hand-written Node port (node:dgram + ws dependency vs Deno.listenDatagram; deno […] *Fix:* Delete bridge.mjs; keep bridge.ts as the single source and compile.sh binaries as the no-Deno distribution. Fix the spec line that calls .mjs 'compiled'.
**S12. Inputs: spec promises source composition, code enforces exclusive single-source; composition machinery unreachable** (manifold-backends-inputs, downgraded, day) — `docs/specs/inputs-spec.md:16`, `docs/specs/inputs-spec.md:230`
inputs-spec.md §0 says the operator picks 'one source *or a combination*' and §223-230 specifies multi-source composition ('XY pad + a MIDI […] *Fix:* Do not prune the composition machinery or rewrite the spec to exclusive — mix-and-match is a recorded, unreversed operator decision (2026-06-28 memory; commit 9e59eb0 and docs/specs/recon/midi-gamepad-inputs-worklog.md both mark the […]
**S13. Per-backend config side-channel + instanceof special-casing in the manager's hot path** (manifold-backends-inputs, downgraded, day) — `manifold/src/backends/manager.ts:69`, `manifold/src/backends/manager.ts:78`
The OutputBackend contract carries mappings/names via BackendContext, but each real backend ALSO needs an index-aligned parallel specs […] *Fix:* Two independent fixes, narrower than proposed. (a) Replace the per-tick instanceof (manager.ts:69) with an optional `setInputVector?(vec: ReadonlyArray<number>): void` on OutputBackend, implemented only by VcvBackend — this alone removes […]
**S14. MEMLNaut Editor (vision bullet 5) is a 237-line shell: sound connect lifecycle, zero protocol, and a third unshared Web Serial implementation** (manifold-backends-inputs, downgraded, week+) — `manifold/src/serial/memlnaut-serial.ts:102`, `manifold/src/serial/EditorPanel.tsx:72`
memlnaut-serial.ts is honestly labelled STUB: saveModel returns false, restoreModel returns null, getSettings returns {} (lines 102-133); […] *Fix:* When bullet 5 is scheduled: (1) define the wire FRAMING as a C header under firmware/ following the useq-celium discipline (header single source of truth → TS mirror → parity test asserting frame sizes/offsets) — this part of the original […]
**S15. Dead focus/altitude system: SplitStage, ReadoutStrip, InputMini, AltitudeNav, CompactAxis are unreachable** (manifold-ui, confirmed, hours) — `manifold/src/console/ConsoleApp.tsx:105`, `manifold/src/App.tsx:78`
ConsoleApp.tsx:105 `const [focus, setFocus] = useState<Focus>(initialFocus)` — setFocus is never called anywhere in the repo (grep […] *Fix:* As proposed (delete SplitStage.tsx, ReadoutStrip.tsx, InputMini.tsx, AltitudeNav, CompactAxis, the UI Focus type/prop, the focus branches + stripPinned + vacuous keyboard gates in ConsoleApp, and the barrel exports; keep MiniMeters), with […]
**S16. Decorative/fake controls: training-param sliders, master volume, bpm, A/B toggle, snapshots and health/gradient visuals drive nothing** (manifold-ui, downgraded, day) — `manifold/src/console/ConsoleApp.tsx:113`, `manifold/src/console/Drawers.tsx:242`
Verified by grep: learningRate/decay/tame/spreadLevel (ConsoleApp.tsx:126,149-151) are read ONLY by their own Drawers.tsx:242-255 sliders — […] *Fix:* Deletion-first, with three corrections. (1) Delete the A/B machinery (ab/aRef/setHoldingA/toggleAB, the VerdictCluster props, and the '+ A/B toggle' header comment), the fake seed, seededGradient + rev/weightsRevision, the unconsumed […]
**S17. VCV backend UI is unreachable: no OutputMode maps to BackendId 'vcv'** (manifold-ui, confirmed, hours) — `manifold/src/console/output-mode.ts:40`, `manifold/src/console/types.ts:27`
OutputMode type (types.ts:27) is 'particles'|'midi'|'osc'|'cv'|'synth'|'editor' and OUTPUT_MODES (output-mode.ts:40-83) maps these to […] *Fix:* Decide which: add `{ id: 'vcv', backend: 'vcv' }` to OUTPUT_MODES + OutputMode + MODE_ICON (three small edits) to make the path live, or delete VcvConfig/VcvChannelEditor/forwardVcvFeedback/vcv ctx state. The first is likely intended given […]
**S18. Per-backend config editors duplicated in two files and BOTH rendered in the same expanded drawer** (manifold-ui, confirmed, hours) — `manifold/src/console/Drawers.tsx:646`, `manifold/src/console/Drawers.tsx:677`
BackendAdvanced.tsx:8-9 self-describes as 'the full-depth duplicate of the inline config in OutputsBackendConfig; both write the same […] *Fix:* Delete BackendAdvanced.tsx and the Drawers.tsx:674-679 block (keep SynthGroupNote inline in Drawers if its one paragraph is wanted). OutputsBackendConfig already covers every backend.
**S19. ConsoleCtx god-object: ~25 of its fields have zero consumers outside ConsoleApp** (manifold-ui, downgraded, day) — `manifold/src/console/types.ts:56`, `manifold/src/console/ConsoleApp.tsx:789`
Grep across console/, dock/, serial/ (excluding ConsoleApp/types): ctx.modes, setModeId, axes, setAxis, preset, setPreset, offsetActive, […] *Fix:* Prune ConsoleCtx to the 57 fields Dock/Drawers/OutputsBackendConfig actually read (plus mode). Delete outright: the Axes type + axes/setAxis state, the preset/setPreset/offsetActive chain (permanently 'Sculpt'/false), and the dead ctx […]
**S20. Dead legacy feedback modes: RandomiseOutputs, RandomiseMlp, AvoidStyle::Diffuse, on_drag have no product consumer** (ml-core, downgraded, day) — `nisps/ml/feedback.hpp:59-73`, `nisps/ml/feedback.hpp:258-271`
The only product callers of the feedback controller select two behaviours: firmware sets only FeedbackMode::ExploreAndPlace […] *Fix:* Treat as a spec-governed decision, not dead-code removal. First surface to the operator: deleting AvoidStyle::Diffuse reverses the explicit "keep for A/B comparison" decision in docs/adr/rl-feedback-design.md, and deleting […]
**S21. perf.hpp SRAM-section macro regime is mostly dead; the one real NISPS_AUDIO_FUNC use is shape-mismatched; docs claim discipline that doesn't exist** (ml-core, confirmed, hours) — `nisps/core/perf.hpp:26-35`, `firmware/MEMLNaut-NISPS/glue/midi_io.hpp:72`
Repo-wide grep: NISPS_AUDIO_MEM and NISPS_APP_SRAM are used ZERO times outside perf.hpp and comments (the .ino uses memllib's own AUDIO_MEM […] *Fix:* Delete NISPS_AUDIO_MEM/NISPS_APP_SRAM/NISPS_AUDIO_FUNC from perf.hpp; change midi_io.hpp:72 to plain 'inline void ...' (or Pico's __not_in_flash_func(name) applied correctly); rewrite MAP.md:151/164 and drop ALIGNMENT #4. If SRAM placement […]
**S22. All vcv/ user-facing docs describe the pre-P6 nisps-core-era module (2×12, v1 3D weights, dead paths)** (satellites-hygiene, confirmed, hours) — `vcv/README.md:3`, `vcv/README.md:31`
README.md:3 says '2 inputs … 12 outputs' (module is 8×16 per MEMLNaut.cpp:16-17); README.md:31+265 and BUILDING.md:29+73 reference […] *Fix:* Rewrite README.md's numbers/paths (small diff), delete the playground-sharing sections, and either rewrite NISPS-FORMAT.md against the v3 flat layout or delete it and point at docs/specs/vcv-module.md (which the /specs corpus already owns) […]
**S23. Retired-playground build output and its root E2E harness still tracked on main** (satellites-hygiene, confirmed, hours) — `playground/dist/index.html`, `playground/dist/nisps.wasm`
git ls-files shows playground/dist/* (1.2 MB: bundled JS+sourcemaps, a second nisps.wasm and nisps.js) tracked, though the playground app […] *Fix:* git rm -r playground/ tests/e2e/ package.json package-lock.json playwright.config.js; delete root node_modules and the playground dir from disk. The archive branch already preserves the app; nothing else names these paths.
**S24. CI never runs codegen — the 'schema changes require regenerated outputs in the same change' gate is unenforced** (schemas-codegen, confirmed, hours) — `.github/workflows/ci.yml:24`, `scripts/run-all-tests.sh:51`
ci.yml's two jobs (cpp-tests, manifold-tests) contain no codegen step at all. The golden test runs only in local run-all-tests.sh stage 5, […] *Fix:* Add a CI step (in the manifold-tests job, where bun is already set up, before typecheck): `cd codegen && bun install --frozen-lockfile && bun run generate.ts && bun run generate-midi-devices.ts && cd .. && git add -N nisps/modes/generated […]
**S25. MLP net dimensions are a dual source of truth: hand-written template args in every C++ mode vs schema dims used by the browser** (schemas-codegen, confirmed, hours) — `nisps/modes/paf_synth.hpp:31`, `nisps/modes/memlcelium.hpp:28`
All 9 modes hand-write dims, e.g. paf_synth.hpp:31 `ml::MLP<4u, 10u, 10u, 14u, 33u>` duplicating schemas/modes/paf_synth.json ml {4, […] *Fix:* Use the generated constexpr constants as the template args directly — they are valid constant expressions today: ml::MLP<generated::kPafSynthMLConfig.input_size, generated::kPafSynthHiddenLayers[0], generated::kPafSynthHiddenLayers[1], […]
**S26. Large fraction of the schema surface has zero consumers — the single-source-of-truth claim is ~half freight** (schemas-codegen, downgraded, day) — `schemas/schema.json:45`, `codegen/generate.ts:195`
Verified by repo-wide grep excluding generated dirs: (a) ml.input_channels names — read nowhere (input_router.hpp:4 admits it may 'grow […] *Fix:* Keep the per-field wire-or-delete structure, with three corrections. (1) default_learning_rate/default_max_iterations: wire BOTH platforms in one change or delete from both — if wiring, firmware glue passes […]
**S27. MAIN.md (corpus entry point) carries at least six claims the code or its own table contradicts** (specs-docs, confirmed, hours) — `docs/specs/MAIN.md:42`, `docs/specs/MAIN.md:67`
(1) :67 'TS schema emission is dormant since P1; it returns at P5' — P5 landed 2026-07-18; codegen/generate.ts:8-12 writes […] *Fix:* Keep the original editing pass over docs/specs/MAIN.md (fix §1.2 superset claim per ALIGNMENT #1 — same eight nisps engines on both targets, C15 archived, mic input unwired; fix §2 to state TS emission is live at […]
**S28. manifold-parity-features-spec.md is part-executed, part-pending, but still labelled 'no implementation authorised' with ground-truth references deleted from main** (specs-docs, confirmed, day) — `docs/specs/manifold-parity-features-spec.md:9`, `docs/specs/manifold-parity-features-spec.md:20`
Front matter: aspirational; body :9-11 'Status: draft — awaiting review... no implementation is authorised'. But its §0 principle 1 (bind […] *Fix:* Reclassify as kind: plan; mark §on Jolt/OU as executed-by-other-means (one-core P3, core bindings — its own preferred route); repoint playground references to the archive branch/tag; keep presets/pins/control-surface sections as the active […]
**S29. Root Playwright rig is dead: tests a page that no longer exists, invoked by nothing** (tests-build, confirmed, hours) — `playwright.config.js:13-22`, `package.json:1-13`
playwright.config.js webServer serves cwd './playground' and testDir './tests/e2e'; tests/e2e/helpers.js:19 does […] *Fix:* Delete playwright.config.js, package.json, package-lock.json, and tests/e2e/ at the repo root; rm -rf root node_modules/, playwright-report/, test-results/ (already gitignored). No consumer exists: CI, run-all-tests.sh, MAP.md's entry […]
**S30. lint-cpp.sh heap audit has proven false-negative modes and never covers nisps/pipeline/** (tests-build, confirmed, hours) — `scripts/lint-cpp.sh:121`, `scripts/lint-cpp.sh:122`
(1) Line 121 hardcodes subdirs dsp/engines/ml/modes — nisps/pipeline/ (the P4 control-rate hot path) and nisps/core/ are never scanned for […] *Fix:* In lint-cpp.sh: add "$NISPS_DIR/pipeline" and "$NISPS_DIR/core" to the subdirs array; replace the `grep -v ' *//'` post-filter with comment-stripping before matching (same perl strip already used by audit_float_suffix at lines 78-98); […]
**S31. CI re-implements run-all-tests.sh and has drifted: codegen golden test not gated in CI** (tests-build, confirmed, hours) — `.github/workflows/ci.yml:24-131`, `scripts/run-all-tests.sh:3`
run-all-tests.sh:3 says 'Designed to be the single command CI invokes', but ci.yml never calls it — it re-lists the stages as discrete […] *Fix:* Add one step to the manifold-tests job (bun is already installed): `working-directory: codegen; run: bun install --frozen-lockfile && bun run tests/golden_test.ts`. Fix the false 'single command CI invokes' header comment in […]
**S32. Committed manifold/public/nisps.{js,wasm} has no freshness gate — all browser tests validate a potentially stale artifact** (tests-build, downgraded, day) — `.github/workflows/ci.yml:88-122`, `manifold/tests/wasm-load.ts:48-49`
manifold/public/nisps.{js,wasm} are tracked build artifacts, last rebuilt/committed at 1672fe3 (P4). The manifold-tests CI job […] *Fix:* Replace the byte-diff with a toolchain-agnostic behavioral freshness gate: in the cpp-tests job, BEFORE the 'Build WASM' step (i.e. while manifold/public/ still holds the committed artifact), run the existing wasm parity harness against it […]
**S33. Dead API entries threaded through the full 5-layer registration chain** (wasm-bridge, downgraded, day) — `nisps/wasm/bindings.cpp:643`, `nisps/wasm/bindings.cpp:868`
Grepped all of manifold/src, manifold/tests, e2e, and tests/cpp/parity_wasm.mjs (the WASM parity driver — I extracted its full list of […] *Fix:* Delete the dead entries at every layer as proposed, with two corrections. (1) EXPORTED_RUNTIME must remain ["HEAP32","HEAPU8","HEAPF32","cwrap"] — tests/cpp/parity_wasm.mjs (parity-check.sh gate) and manifold/tests/wasm-load.ts […]
**S34. publishWeights_ channel: full weight-vector copy at up to 200 Hz into a field nothing reads** (wasm-bridge, confirmed, hours) — `manifold/src/engine/wasm-iml.ts:1093`, `manifold/src/engine/wasm-iml.ts:929`
publishWeights_() (wasm-iml.ts:1093-1096) runs getWeights() — a C→heap copy plus a fresh Float32Array allocation — then sink.setWeights, […] *Fix:* Delete EngineSink.setWeights, Spine.setWeights/weights()/liveWeights, and all publishWeights_ call sites (keep getWeights for persistence and the debug probe). If a weight-health view ever materialises, it should pull getLayerStats on […]
**S35. Dual example store with mismatched caps: train() and trainAsync() diverge past 100 examples; latent OOB read on sample weights** (wasm-bridge, downgraded, hours) — `manifold/src/engine/wasm-iml.ts:167`, `manifold/src/engine/dataset.ts:81`
WasmIML keeps a TS Dataset mirror (default cap 100, wasm-iml.ts:167) AND pushes every example into the C++ FIFO ring (DynamicStorage […] *Fix:* Keep the cap-alignment core of the proposal: name the constant once in nisps (e.g. constexpr kDefaultMaxExamples = 128 shared by storage.hpp:55 and dynamic_storage.hpp:43 defaults), expose it through nisps_ml_describe (extend the dims […]
### Local (55)
| ID | Area | Finding | Where | Fix |
|----|------|---------|-------|-----|
| L1 | architecture-gap | Console health UI shows fabricated gradient data while the real layer-stats API sits plumbed […] | `manifold/src/console/model.ts:305` | Delete seededGradient; wire the health readout to engine.getLayerStats() (already reactive via the version counter). Separately land the half-day […] *(verifier: All cited lines verify: seededGradient (model.ts:305-319) fabricates norms/statuses from the rev counter, is […])* |
| L2 | architecture-gap | MAP.md drift: swap points and TS prototypes described as live are already swapped/deleted | `MAP.md:79` | Fix the two MAP.md entries (exploration = thin WASM wrappers, controller = thin driver, drop rng.ts and the phantom files). Delete the dead […] |
| L3 | dsp-engines-modes | nisps/modes/voice_space.hpp is entirely dead code | `nisps/modes/voice_space.hpp:27` | Delete the file. |
| L4 | dsp-engines-modes | Dead oscillators: SineOsc/SawOsc/SquareOsc have no consumers | `nisps/dsp/osc.hpp:28` | Delete the three classes; keep detail::kPi/kTwoPi, PAFOperator, FMOp. *(verifier: SawOsc and SquareOsc are genuinely dead, but SineOsc has a live consumer the reviewer's grep scope missed: […])* |
| L5 | dsp-engines-modes | ModeBase::input_dirty_ is write-only state | `nisps/modes/base.hpp:165` | Delete the flag, or actually gate tick_control's inference on it (measure first; deletion is the safe default). |
| L6 | dsp-engines-modes | VerbFXEngine dead fields and setters: delay_to_verb_ computed 12 times, never read; […] | `nisps/engines/verb_fx.hpp:242` | Delete delay_to_verb_, enable_reverb_, enable_delay_to_reverb_, and the unused set_enable_* setters (keep the ones a named consumer appears for). If […] |
| L7 | dsp-engines-modes | MEMLCeliumEngine feedback path is inert: feedback_ never written in process() | `nisps/engines/memlcelium.hpp:180` | Check the original MEMLCeliumAudioApp: if it writes feedback, port the missing write; if not, delete feedback_, fbzm1_, fb_smooth_alpha_ and the […] |
| L8 | dsp-engines-modes | Sequencer machinery duplicated across engines: ratio_seq, event queue, bar/MIDI-clock phasors, […] | `nisps/engines/breakor.hpp:178` | Extract dsp/ratio_seq.hpp (free function template), a small SeqClock struct (bar phasor + MIDI clock + counter + update_bpm), and EventQueue<E,64> […] |
| L9 | dsp-engines-modes | Dead no-op VoiceSpace boilerplate on five engines | `nisps/engines/breakor.hpp:37` | Delete the VoiceSpace enum/table/setter from the five engines with zero real voice spaces; keep them only on PAFSynth, VerbFX, ChannelStrip. ModeBase […] |
| L10 | dsp-engines-modes | VerbFXEngine recomputes the full 47-param voice-space mapping every audio sample (and its […] | `nisps/engines/verb_fx.hpp:77` | Run smoothing + voice-space mapping at a shared control rate (e.g. every 16-32 samples, same divisor on both targets so native↔WASM parity holds), […] |
| L11 | dsp-engines-modes | ExternalSynthMIDIMode hand-rolls its schema and ML config outside codegen | `nisps/modes/external_synth_midi.hpp:132` | Fold the ext-synth ML config into the schema/codegen pipeline (a shared 'ext_synth' base schema emitting the constants), or at minimum define the […] |
| L12 | firmware | NISPS_ST_ selftest token-paste table silently out of date: SLPWorkshop (the currently active […] | `firmware/MEMLNaut-NISPS/glue/mode_select.hpp:90-106` | Fold into the PlatformIO migration: delete the entire NISPS_ST_ table and the SelfTestRig tag type, and select the selftest build with […] |
| L13 | firmware | midi_io.hpp uses the misshapen NISPS_AUDIO_FUNC macro that MAP.md and ALIGNMENT.md both claim […] | `firmware/MEMLNaut-NISPS/glue/midi_io.hpp:72` | Apply ALIGNMENT #4 option A: make NISPS_AUDIO_FUNC(name) decorate a function NAME on both targets, update the handful of use sites (midi_io.hpp here; […] |
| L14 | firmware | peripherals.hpp: dead declarations, duplicated commit-and-train block, and the documented […] | `firmware/MEMLNaut-NISPS/glue/peripherals.hpp:67` | Delete kAnalogInputCount, PeripheralBindings, and the unused spread local; extract one commit_and_train(mode, feedback) helper used by both branches; […] |
| L15 | firmware | MAP.md documents a vendored firmware/MEMLCelium-upstream/ directory that does not exist on […] | `MAP.md:30-31` | Delete the MAP.md entry (or, if the snapshot lives untracked on another machine and is genuinely wanted, commit it — but name the consumer first; the […] |
| L16 | manifold-backends-inputs | Dead protocol legs from the retired playground: sendState/sendWeights, legacy array format, […] | `manifold/src/backends/osc-client.ts:116` | Delete sendState/sendWeights, the legacy-array branch in both bridges (or the one surviving bridge), […] |
| L17 | manifold-backends-inputs | Status/throttle boilerplate duplicated across all four real backends while inputs/ already has […] | `manifold/src/backends/midi-backend.ts:163` | Extract a small BaseBackend (status plumbing + throttle gate + lastSent management), mirroring BaseSource. Pure deletion-by-refactor, no behaviour […] |
| L18 | manifold-backends-inputs | MIDI input floods React with per-CC-message state updates and snapshot allocations | `manifold/src/inputs/midi-input-source.ts:224` | Only call notifyBindings on binding-LIST changes (learn capture, clearBinding, clearAllBindings); if the drawer wants live values, read them on the […] |
| L19 | manifold-backends-inputs | BackendManager.setActive silently drops a switch requested while another is in flight | `manifold/src/backends/manager.ts:131` | Store the latest requested id when switching is true and re-run setActive with it in the finally block (3-4 lines). |
| L20 | manifold-ui | Solo-mode selector offers three behaviours but soloMode is behaviourally inert | `manifold/src/console/Drawers.tsx:142` | Collapse to a single non-selectable label until train_masked lands in the C API, or disable the two unimplemented options with a 'pending core […] |
| L21 | manifold-ui | FeedbackController vestiges: unused seed/undoDepth options, never-called engine-interface […] | `manifold/src/feedback/controller.ts:57` | Delete seed/undoDepth options and maxUndo; trim ControllerEngine to the members actually called; change dislike to (output: Float32Array) and update […] |
| L22 | manifold-ui | 5 of 12 design primitives are dead (Panel, StatusLine, ControlAxis, CurvePlot, Sparkline) | `manifold/src/primitives/index.ts:1` | Delete the five unused primitives (Sparkline/CurvePlot may return with the deferred diagnostics suite — re-add then). Optionally promote […] |
| L23 | manifold-ui | Dual backend catalogues: BACKENDS duplicates OUTPUT_MODES; OutputControl/toOutputControl dead | `manifold/src/dock/output-state.ts:40` | Delete OutputControl/toOutputControl and ModeIconComponent; delete BACKENDS and use modeDesc.label/description in Drawers (or derive BACKENDS from […] |
| L24 | manifold-ui | ConsoleApp global-listener effects run without dependency arrays — re-subscribed on every […] | `manifold/src/console/ConsoleApp.tsx:684` | Hold the verdict handlers in refs (or useEffectEvent) and give both effects [] / [inputs] deps, matching Manifold's existing onMoveRef pattern. |
| L25 | ml-core | 16 KB loss-history buffer baked into every firmware MLP instance with zero product consumers | `nisps/ml/storage.hpp:200` | Decide plumb-or-delete now. Smallest: delete lh_/loss_hist_buf/loss_history()/loss_history_count_ (train already returns final loss) and the […] |
| L26 | ml-core | dislike_multiplier_ is upstream-InterfaceRL bookkeeping that drives nothing | `nisps/ml/feedback.hpp:406-407` | Delete dislike_multiplier_, its update/halving logic, the accessor, and the test line — or, if the upstream behaviour it was meant to drive is […] |
| L27 | ml-core | nisps/core/fixed_buffer.hpp is dead code | `nisps/core/fixed_buffer.hpp:1-77` | Delete fixed_buffer.hpp, test_fixed_buffer.cpp, the CMake/ctest entry, and the MAP.md:8 mention. Trivially revivable from git if a consumer ever […] |
| L28 | ml-core | get_weights() forces a permanent whole-net scratch copy (flat_buf) and double-copies on every […] | `nisps/ml/storage.hpp:199` | Add a copy_weights_to(std::span<float> dst) (and set from span, which already exists) and switch feedback.hpp's take_snapshot/push_undo/nudge to it; […] |
| L29 | ml-core | OutputChain's firmware story is aspirational — pipeline chains have browser-only consumers | `nisps/pipeline/output_chain.hpp:12-13` | Either wire InputChain/OutputChain into the firmware control tick (peripherals/ModeBase) as the named vision-5 requirement, or soften the header […] |
| L30 | satellites-hygiene | ALIGNMENT.md declares the VCV module dead while vcv/ is actively maintained | `ALIGNMENT.md:72` | Delete the ALIGNMENT.md:72 entry (the module now DOES consume nisps/ directly — the entry's own 'if revived' condition is met and closed). While […] |
| L31 | satellites-hygiene | MAP.md documents firmware/MEMLCelium-upstream/ which does not exist on disk or in any git ref | `MAP.md:30` | Delete the MAP.md:30 entry (the port provenance is already captured in ALIGNMENT.md's geometric-dislike entry and nisps/ml file headers; upstream […] |
| L32 | satellites-hygiene | Agent-worktree fragment committed to main (.claude/worktrees/agent-ae87fe47/…) | `.claude/worktrees/agent-ae87fe47/playground/js/shapeseq […]` | git rm -r .claude/worktrees/ (the ignore rule then holds for the future). Also consider pruning the 30+ stale worktree-agent-*/worktree-wf_* local […] |
| L33 | satellites-hygiene | vcv/test/ is dead: smoke test includes a nonexistent header, tests the retired 2×12 shape, and […] | `vcv/test/smoke_test.cpp:6` | git rm -r vcv/test/ and vcv/Makefile.dist (update BUILDING.md's two references). In osc_server.hpp, delete the unreachable reply-to-sender branch and […] |
| L34 | satellites-hygiene | Data race: audio thread reads imlShadow's example vectors while the worker may be mutating them | `vcv/src/MEMLNaut.cpp:514` | Have the worker deep-copy features/labels into pending staging buffers (alongside pendingWeights, e.g. under stagingMutex) BEFORE setting […] |
| L35 | satellites-hygiene | Full-state JSON serialize/parse runs on Rack's audio thread | `vcv/src/MEMLNaut.cpp:474` | Move JSON build/parse to the existing worker thread: stage the raw string in/out with the same atomic-flag pattern already used for oscStagedJson, […] |
| L36 | satellites-hygiene | Dead root planning docs and dead data/ directory from retired eras | `NISPS_CORE_EXTRACTION_PLAN.md` | git rm both NISPS_CORE_* files (history preserves them; docs/specs/plans/ is the sanctioned home for plans) and git rm -r data/. If the CSVs matter […] |
| L37 | schemas-codegen | modeEngineId switch in manifold hand-duplicates schema engine_id and silently drops new modes […] | `manifold/src/console/model.ts:324` | Delete the switch; route on mode.engineId with the single named exception (sound_analysis_midi → 'analysis', or add an optional analysis-engine […] |
| L38 | schemas-codegen | Two divergent TS applyCurve implementations for the per-output curve knob | `manifold/src/console/model.ts:278` | Pick one formula (mapping.ts's, which is spec-anchored and has a sane 0.5=linear midpoint), delete the other, and re-export the survivor; update […] |
| L39 | schemas-codegen | seed-midi-devices.ts silently clobbers the canonical hand-tunable device schemas it created | `codegen/seed-midi-devices.ts:11` | Delete the script (its purpose is served; it lives in git history and synth-midi-cc.json keeps provenance), or make it skip existing files. Drop the […] |
| L40 | schemas-codegen | Mode schema content is partially placeholder: 20 params across paf_synth/channel_strip/xiasri […] | `schemas/modes/paf_synth.json:16` | Accept as dated debt but record it (ALIGNMENT.md entry): name the placeholder slots during the first preset-curation pass per mode, or shrink […] |
| L41 | specs-docs | MAP.md documents a phantom vendored firmware (firmware/MEMLCelium-upstream/) that exists […] | `MAP.md:30` | Delete the MAP.md:30 bullet (or, if the snapshot exists untracked on another machine and is genuinely wanted, commit it — but the smallest fix is […] |
| L42 | specs-docs | aimmersive-clone-spec.md specs a retired product (SolidJS clone of a-immersive) and still […] | `docs/specs/aimmersive-clone-spec.md:7` | Move to docs/specs/_archive/ with deprecated-by front matter (or reclassify kind: finding, dated, noting it describes the deployed vanilla app); […] |
| L43 | specs-docs | feedback-modes-port-spec.md is an executed port plan masquerading as an evolving spec, with […] | `docs/specs/feedback-modes-port-spec.md:6` | Archive it (kind: plan, status: executed, deprecated-by: docs/adr/rl-feedback-design.md + code). The ADR + feedback.hpp are the surviving truth; […] |
| L44 | specs-docs | BUILD-PLAN.md still status:active though executed, with stale locked-decision facts | `docs/specs/plans/BUILD-PLAN.md:3` | Flip to status: executed, add a two-line 'what survives' note (non-snap-node gotcha → MAP), update the MAIN registry row. |
| L45 | specs-docs | engine-architecture.md's 'One Engine, Two Skins' framing is superseded but the file carries no […] | `docs/specs/engine-architecture.md:7` | Add a supersession header pointing at one-core-engine-refactor §0, retitle/trim to what remains authoritative: the browser-side EngineApi seam […] |
| L46 | specs-docs | vcv-module.md is a three-era palimpsest whose 'retained for reference' sections state stale […] | `docs/specs/vcv-module.md:169` | Prune pass: delete the 2→12-era architecture/prerequisite/Phase-0 text (git keeps it), keep the current 8→16 contract + threading/persistence […] |
| L47 | specs-docs | ALIGNMENT.md staleness: dead VCV deferred-debt entry, mitigation pointing at a deleted file, […] | `ALIGNMENT.md:72` | Delete :72; repoint #5's source reference to upstream MusicallyEmbodiedML memlp (or the archive branch); prune the 2026-04-29 resolved entries. |
| L48 | specs-docs | 818 lines of dead nisps-core extraction relics at repo root | `NISPS_CORE_EXTRACTION_PLAN.md` | git rm both files (history retains them; the executed outcome is already narrated in ALIGNMENT's resolved log). |
| L49 | specs-docs | MAP.md and AGENT-REFERENCE.md still carry pre-P5 'modes slice the default 126 outputs / TS […] | `MAP.md:155` | Update the three sentences to past-tense schema-real reality (or delete the convention bullet — MAP's manifold section already states it correctly). |
| L50 | specs-docs | backends/dock/inputs spec trio grounds itself in a deleted world (playground paths, a-app.js […] | `docs/specs/inputs-spec.md:14` | One pass across the three files: mark the audit/grounding sections as historical ('as of 2026-06-27, pre-refactor'), fix the handful of dead-path […] |
| L51 | tests-build | Parity runner headers document 4 stages / blob v1 while the code implements 7 stages / v5 | `tests/cpp/parity_check.cpp:14-45` | Rewrite both file headers' payload/stage lists to match the v5 blob (or replace the duplicated prose lists with a one-line pointer to the stage […] |
| L52 | tests-build | parity-check.sh FAIL branch is unreachable under set -e | `scripts/parity-check.sh:18` | Wrap: `if node "$TESTS_DIR/parity_diff.mjs" ...; then echo PASS; else status=$?; echo FAIL; exit $status; fi` — or delete the dead branch and let the […] |
| L53 | wasm-bridge | fetch + indirect-eval glue loading duplicated in two files; build could emit an ES module […] | `manifold/src/engine/wasm-iml.ts:56` | Add `-sEXPORT_ES6=1` (and `-sENVIRONMENT=web,worker` stays) to build-wasm.sh, replace both eval blocks with a single shared `await import(/* […] |
| L54 | wasm-bridge | Worklet WASM loader carries dead scaffolding and a silent-stub failure trap | `manifold/src/engine/worklet/nisps-processor.ts:104` | Delete the unused `imports` object, the 'c' branch, `exMap`, and the duplicate second loop. Replace the generic `() => 0` stub with a throwing stub […] |
| L55 | wasm-bridge | Retired playground's dist/ build artifacts still tracked in git | `playground/dist` | git rm -r playground/dist and remove the leftover untracked directory; add playground/ to .gitignore or delete the folder entirely. |
### Style (15)
| ID | Area | Finding | Where | Fix |
|----|------|---------|-------|-----|
| ST1 | dsp-engines-modes | Engine header comments describe implementations that don't exist (doc drift cluster) | `nisps/engines/breakor.hpp:4` | Rewrite the four comment blocks to describe the actual code; drop stream-plan tense. |
| ST2 | firmware | input_router.hpp is a zero-logic speculative layer with one consumer | `firmware/MEMLNaut-NISPS/glue/input_router.hpp:18-21` | Delete input_router.hpp; have the .ino call bind_peripherals directly (or rename bind_peripherals to wire_inputs inside peripherals.hpp). Drop the […] |
| ST3 | manifold-backends-inputs | Stale comments/docs: 'inputs-spec.md is absent', 'fixed WASM head = 2', 'Editor → BackendId' | `manifold/src/inputs/types.ts:10` | Delete/correct the three claims; one small comment pass. |
| ST4 | manifold-ui | GROUP_COLOR maps duplicated in four files and keyed on obsolete JSX-era group names | `manifold/src/console/shared-ui.tsx:83` | One shared map in model.ts (or a hash-to-palette function over arbitrary group strings, which would actually work for schema groups); delete the […] |
| ST5 | manifold-ui | Minor prop/doc lint: duplicate onReroll/onRandomise, unused CompositeStage mode prop, probe […] | `manifold/src/console/ConsoleApp.tsx:1092` | Collapse to one onReroll prop; drop the mode prop; rewrite the probe header to describe the actual surface. |
| ST6 | ml-core | feedback.hpp header comment asserts the retracted 'geometric push NOT ported' claim, […] | `nisps/ml/feedback.hpp:8-12` | Rewrite the header's mode summary to match the four current modes and the Geometric default (fold into the dead-modes deletion commit). |
| ST7 | ml-core | MAP.md and math.hpp still describe the pre-P3/P4 world (interim TS jolt/OU, playground curve […] | `MAP.md:79-81` | Update MAP.md:79-81 to describe exploration.ts as a thin timer-driver over the WASM core, and replace math.hpp's playground-mirror paragraph with the […] |
| ST8 | satellites-hygiene | License story is self-contradictory: MPL-2.0 repo, 'proprietary' VCV plugin manifest | `LICENSE:1` | Set plugin.json license to MPL-2.0 (or GPL-3.0-or-later per VCV convention) and drop the 'in spirit' hedge in iml.hpp — one licence, stated once. |
| ST9 | schemas-codegen | schema.json's own description states the direction of truth backwards | `schemas/schema.json:5` | Rewrite the description sentence: schemas are the canonical contract; codegen emits C++ and TS; provenance notes live in params_notes.md. |
| ST10 | schemas-codegen | codegen/README.md is wholesale stale (playground paths, phantom temp-dir, resolved stream-1 […] | `codegen/README.md:6` | Rewrite codegen/README.md against current reality (or shrink it to a pointer at MAP.md §codegen); drop port-solidjs from ci.yml triggers. |
| ST11 | schemas-codegen | codegen/templates/ are dead 'reference' files already drifting from the real emitters | `codegen/templates/cpp_schema.hpp.template:4` | Delete codegen/templates/ and the README line pointing at it. |
| ST12 | schemas-codegen | Duplicated helper block across the two codegen scripts, justified by a false rationale comment | `codegen/generate-midi-devices.ts:17` | Extract a tiny codegen/lib.ts with the shared helpers and correct the comment (keep both drivers). Or just fix the comment and accept the 40 lines. |
| ST13 | schemas-codegen | synth-midi-cc.json lives at repo root; its only machine consumer is the seed script | `synth-midi-cc.json:1` | Move to schemas/midi_devices/synth-midi-cc.sources.json (one-line path update in the seed script — moot if that script is deleted per the clobber […] |
| ST14 | tests-build | test_helpers.hpp documents NISPS_EXPECT as non-fatal but it throws; NISPS_ASSERT is a plain […] | `tests/cpp/test_helpers.hpp:10-11` | Smallest fix: correct the comment to say both are fatal and delete the NISPS_ASSERT alias (or keep the alias, one line). Only implement true […] |
| ST15 | wasm-bridge | Doc/comment drift lagging the P1–P4 refactors | `MAP.md:79` | One doc-sync pass: fix MAP.md:79, rewrite the nisps/wasm/README paragraph and either complete or delete its API table (point at bindings.cpp as the […] |

View file

@ -1,3 +1,9 @@
---
kind: finding
date: 2026-06-27
immutable: true
---
# Upstream Firmware Survey — git archaeology of the MusicallyEmbodiedML ecosystem
_Dated 2026-06-27. Author: research agent (read-only). All SHAs from a fresh `git fetch --all` + memllib submodule-gitdir fetch (incl. operator's memllib fork `monkey-w1n5t0n/memllib`)._
@ -29,7 +35,7 @@ fast-forward or 3-way merge path; every integration is a re-implementation again
| Branch | Remote | Last push | Author | vs its main¹ | Purpose | Judgment |
|---|---|---|---|---|---|---|
| `main` | origin | 2026-06-16 | monkey-w1n5t0n | — | The C++20/SolidJS rewrite. | canonical |
| `feat/feedback-explore-modes` | origin | 2026-06-08 | w1n5t0n | +42 / −193 | Pins memllib `abe93ec` = the `FEEDBACK_MODE` (avoid / randomise-outputs / randomise-MLP) commit. Source material for `docs/redesign/feedback-modes-port-spec.md`. | **real RL work to port** (already spec'd, not yet in main) |
| `feat/feedback-explore-modes` | origin | 2026-06-08 | w1n5t0n | +42 / −193 | Pins memllib `abe93ec` = the `FEEDBACK_MODE` (avoid / randomise-outputs / randomise-MLP) commit. Source material for `docs/specs/feedback-modes-port-spec.md`. | **real RL work to port** (already spec'd, not yet in main) |
| `feat/useq-celium-opus46` | origin | 2026-06-14 | w1n5t0n | +2 / −36 | MEMLCelium useq experiment (Opus-4.6 session). | experiment — superseded by opus47 |
| `useq-celium-opus47` | origin | 2026-04-20 | w1n5t0n | +6 / −36 | Later MEMLCelium useq iteration. | experiment |
| `feat/unified-preset-system` | origin | 2026-04-16 | w1n5t0n | +46 / −55 | Preset-system spike; heavily diverged, stale. | abandoned/spike |
@ -157,7 +163,7 @@ not `git merge`. Ranked by mission impact (RL/feedback richness and firmware-fea
`origin/feat/feedback-explore-modes`; cross-checked against memllib main `80420d1`/`d9bdcd6` (SB2026 dislikes
in `InterfaceRL`) which upstream/main pins via `e291192`.
- **Why first:** This is the negative-feedback / RL core of the whole project, and the work is *already
written up* as an implementation-ready spec at `docs/redesign/feedback-modes-port-spec.md` (new
written up* as an implementation-ready spec at `docs/specs/feedback-modes-port-spec.md` (new
`nisps::ml::FeedbackController<MLP_T>`, WASM C API, 11 ctest cases, parity Stage 5). The branch
`feat/feedback-explore-modes` and the half-present `nisps/ml/feedback.hpp` + `tests/cpp/test_mlp_feedback.cpp`
in the working tree show this is mid-flight. **Finish it and merge to main.** The spec's AVOID reconciliation
@ -213,7 +219,7 @@ The 49 upstream commits are firmware apps on the old tree. Port the ones with mi
operator's own memllib fork — an off-graph gitlink that a clean checkout cannot resolve.
3. **The dislikes/RL feedback work the task flagged as "to find" is already (a) merged into upstream via SB2026
→ memllib `e291192`, AND (b) re-implemented by the operator on `feat/feedback-explore-modes` (memllib
`abe93ec`), AND (c) fully spec'd for the new core** in `docs/redesign/feedback-modes-port-spec.md` — with
`abe93ec`), AND (c) fully spec'd for the new core** in `docs/specs/feedback-modes-port-spec.md` — with
`nisps/ml/feedback.hpp` and `tests/cpp/test_mlp_feedback.cpp` already present (untracked) in the working tree.
The feedback integration is not a research question; it's a half-finished implementation to land.
@ -250,4 +256,4 @@ The **latest upstream `InterfaceRL` tip is `0a541cc` "highlighting"** (990 lines
which are older. **Mode 1 "Geometric dislike" must be ported to `0a541cc` parity.** Note: the in-tree submodule
gitdir resolves none of `4733ca0`/`e291192`/`abe93ec`/`0a541cc` — only the separate `-upstream` memllib
checkout has them; the orphaned-pin footgun stands and `0a541cc` is the re-pin/port target. See
`docs/redesign/rl-feedback-design.md` §1 for the verified-ground-truth list.
`docs/adr/rl-feedback-design.md` §1 for the verified-ground-truth list.

View file

@ -1,26 +1,35 @@
---
stability: stable
kind: spec
stability: evolving
layer: binding
---
# SLP-Workshop Firmware & Adaptive-Learning Gestures
# SLP-Workshop Firmware, Learning Gestures & Output Modes
> Reference spec for the **SLP-Workshop** firmware mode (built for a workshop at
> Synth Library Portland) and the two adaptive-learning gestures it introduced —
> **Jolt** (held continuous weight morph) and **OU explore** (Ornstein-Uhlenbeck
> output walk). Both gestures live in the shared mode base, so they are available
> to every mode; SLP-Workshop is the mode that surfaces them.
> The single spec for **SLP-Workshop** (built for a workshop at Synth Library
> Portland). It has two parts:
>
> This describes a feature that is already built and merged. It is a stable
> reference, not a forward plan: it records what is, and why the load-bearing
> decisions were made the way they were.
> - **Part I (§1–8) — shipped & stable.** The `slp_workshop` mode and the two
> adaptive-learning gestures it introduced — **Jolt** (held continuous weight
> morph) and **OU explore** (Ornstein-Uhlenbeck output walk). These live in the
> shared mode base, so every mode gains them; SLP-Workshop is the mode that
> surfaces them. Built and merged.
> - **Part II (§9–13) — planned / evolving.** The output-mode evolution: the
> Continuous/Rhythm stream model, gate sequences, and the Manifold-side
> configuration UX so firmware and browser stay aligned. **Not implemented
> yet** — this part fixes the model and shapes, not the final counts.
>
> Provenance: landed on branch `workshop/synth-fw-audit` (commits `4e60d01`
> core + firmware, `57c9ede` browser controls); merged to `main` at `527b8fc`.
> The file is marked `evolving` because Part II is still being figured out; treat
> Part I as the stable baseline and Part II as the agreed direction.
>
> Migrated from `docs/slp-workshop/SPEC.md` (folded in 2026-06-28; donor removed).
> Provenance of Part I: branch `workshop/synth-fw-audit` (commits `4e60d01` core +
> firmware, `57c9ede` browser controls); merged to `main` at `527b8fc`.
## Source files
- `nisps/modes/slp_workshop.hpp` — `nisps::modes::SLPWorkshopMode` (mode_id `slp_workshop`); reuses the MEMLCelium engine + MLP shape verbatim.
- `nisps/engines/memlcelium.hpp` — the shared engine: one `MLP<4,[10,14,18],56>`, 56 outputs split into sequencer + synthesis; exposes `pop_events()`.
- `nisps/ml/jolt.hpp` — `ml::Jolt`: held-gesture continuous weight morph + post-release LR ramp.
- `nisps/ml/ou_noise.hpp` — `ml::OUNoise<N>`: Ornstein-Uhlenbeck random walk on the output vector.
- `nisps/modes/base.hpp` — `nisps::ModeBase` CRTP base; owns `jolt_`/`ou_` and the guarded integration points.
@ -33,16 +42,19 @@ layer: binding
- `playground/src/modes/SLPWorkshopMode.tsx`, `playground/src/modes/mode-runtime.ts` — browser controls.
- `playground/src/ml/jolt.ts`, `playground/src/output/ou-explore.ts` — browser-side gesture math.
- `tests/cpp/test_mlp_jolt.cpp`, `tests/cpp/test_mlp_ou_noise.cpp`, `tests/cpp/test_mode_learning.cpp`, `tests/cpp/test_mode_concepts.cpp` — anchoring tests.
- **Part II companions (planned work):** `docs/specs/useq-cv-protocol.md` (CV wire protocol), `manifold/src/backends/` (CV/MIDI output backends), `manifold/src/inputs/` (input layer).
---
# Part I — Shipped: mode + adaptive-learning gestures
## 1. Frame
1.1 SLP-Workshop is a distinct nisps mode (`nisps::modes::SLPWorkshopMode`, mode_id `slp_workshop`) that **MUST** reuse the MEMLCelium engine and MLP shape verbatim: engine_id stays `memlcelium`, and the MLP is `MLP<4, 10, 14, 18, 56>`.
**Why:** the workshop wanted its own identity (preset directory, display name) and a UI that foregrounds the two adaptive-learning gestures, but no new synthesis behaviour — so the synthesis mapping is byte-for-byte MEMLCelium's, and only the mode wrapper, identity, and surfaced controls differ.
1.2 The synthesis mapping in SLP-Workshop **MUST** stay identical to `MEMLCeliumMode` for the same seed and inputs.
**Why:** the two adaptive-learning gestures live in the shared base (§2), not in this mode; SLP-Workshop adds only identity and control surfacing. A test asserts SLP-Workshop ≡ MEMLCelium (same seed, identical audio) with the new features off.
**Why:** the two adaptive-learning gestures live in the shared base (§5), not in this mode; SLP-Workshop adds only identity and control surfacing. A test asserts SLP-Workshop ≡ MEMLCelium (same seed, identical audio) with the new features off.
1.3 Firmware compiles exactly one mode at a time (`MEMLNAUT_MODE_TYPE`), so the per-mode code cost of an extra mode is negligible.
**Why:** modes are selected at compile time, not linked together; an unused mode contributes nothing to the firmware image.
@ -149,9 +161,105 @@ layer: binding
---
# Part II — Planned: output modes, gate sequences & config UX
> **Status: planned, not implemented.** This part fixes the *model and shapes*, not
> the final counts. **Locked operator decisions (2026-06-28):** (1) hardware keeps
> exactly **2** ratio sequences (memlcelium verbatim); the browser may instantiate
> as many as the user wants. (2) The 3 uSEQ gate-only jacks make gates **optional**
> — pure CV is valid; if doing gates at all, use those 3 first, then convert CV
> jacks. (3) Split-net input is **per-input-channel** engine routing (each
> pad/stick/CC tagged Continuous or Rhythm). (4) **Internal BPM clock** now;
> external-MIDI-clock sync is a later nicety.
## 9. Ground truth: what `memlcelium` is
9.1 `memlcelium` is **one** MLP (`MLP<4, [10,14,18], 56>`), not two. Its 56 outputs split: `[0..13]` **sequencer** (2 sequences × 7 ratio-seq params), `[14..55]` **synthesis** (42 continuous params, Voice 0 + Voice 1). The single net produces both the continuous values and the RatioSeq params; an internal RatioSeq tick turns the seq params into note triggers, exposed via `pop_events()`.
**Why:** the planned output modes are *slices/reshapes of this one net*, and the gate stream already exists inside the engine — so gate outputs are a routing/consumption feature, not new DSP. (A true two-MLP-head "split" topology is genuinely different and therefore browser-only; §11.2.)
9.2 The 7 ratio-seq params per track are: `ratios[0..2]` (3), `phasor_mul` (1), `phase_off` (1), `amp_ratios[0..1]` (2).
**Why:** firmware parity. (The April browser uSEQ-Celium used 8, adding a pulse-width param; pulse width is an optional 8th later — §13.)
---
## 10. The unifying model: two output streams
10.1 Everything collapses to two output **streams**, each driven by an MLP head:
| Stream | MLP outputs | Generates | Routes to |
|----------------|----------------------------------|--------------------|-----------------------|
| **Continuous** | 1 value per channel | smooth 0..1 values | MIDI CC / CV jack |
| **Rhythm** | 7 params per gate-sequence track | clock-driven gates | MIDI note / gate jack |
A **mode** is just *which streams are active* and *whether they share a network*.
10.2 The Rhythm stream **SHOULD** generate gates via RatioSeq, per track per control tick: (1) a shared internal-BPM clock advances a bar phasor; (2) `seq_phasor = (bar_phasor × phasor_mul + phase_off) mod 1`; (3) `ratio_seq_3(seq_phasor, ratios, pw=0.5)` → boolean gate; (4) `ratio_seq_2(amp_ratios)` → 2-level velocity (127/64); (5) rising edge → note-on / gate-high, falling edge → note-off / gate-low.
**Why:** a gate is **clock + learned pattern**, not a threshold on a continuous value — this is what makes Rhythm a distinct stream and why it needs its own 7 params per track rather than one output per gate. Defaults (pulse-width 0.5, 2-level velocity) match firmware.
10.3 Where RatioSeq runs (browser): when the active engine *is* the memlcelium/SLP WASM engine, consume `pop_events()`; for CV/MIDI modes that don't run that audio engine, a small **TS RatioSeq** fed by the Rhythm MLP's 7-params/track drives the gates.
**Why:** the gate generator must work whether or not the WASM audio engine is the active backend.
---
## 11. Modes
11.1 **Firmware (compile-time, one chosen at build)** **MUST** always use exactly 2 ratio sequences (memlcelium verbatim). Three slices of the `slp_workshop` mode:
| Mode | Streams | Nets | MLP output_size |
|----------------------------------------|-------------------|------|---------------------------|
| Continuous only | Continuous | 1 | continuous params only |
| Continuous & Rhythm (= memlcelium) | both, **shared** | 1 | 14 seq + 42 synth = 56 |
| Rhythm only | Rhythm | 1 | 2 × 7 = 14 seq params |
**Why:** the firmware does not vary sequence count (locked decision); exact output_size counts for the sliced modes are the firmware agent's call — this spec fixes the shapes.
11.2 **Browser (Manifold, dynamic, switchable live)** **MAY** offer the same three plus a **split-nets** variant (a separate Continuous MLP and Rhythm MLP), with **as many ratio sequences as the user wants** (each gate sequence = its own 7-param track; the net reshapes to suit). The browser mode is **implied by the Outputs config** (§12), not a separate picker: continuous>0 & gates=0 ⇒ Continuous-only; gates>0 & Shared net ⇒ hybrid-shared; gates>0 & Separate net ⇒ hybrid-split; continuous=0 ⇒ Rhythm-only.
**Why:** the browser has the compute for two MLP heads and dynamic reshape; making the mode a consequence of the config avoids a redundant mode dropdown. Shared-net `output_size = continuous + 7 × n_gates` must respect the 126-output WASM cap (~16 gates max shared); a separate Rhythm net is sized independently and scales further.
---
## 12. Configuring gate sequences (Manifold, CV **and** MIDI)
12.1 Two output kinds: **Continuous** → CC (MIDI) / CV jack (CV); **Gate sequence** → note (MIDI) / gate jack (CV), each = one RatioSeq track.
12.2 **MIDI mode** **SHOULD** expose two independent counts — Continuous (CC) and Gate-sequences — capped by the model output budget.
12.3 **CV mode** hardware is fixed (11 PWM/CV-capable jacks + 3 digital gate-only jacks), so the two counts are **linked** and gates are optional. The rule **MUST** be `gates ∈ [0, 14]`; `CV = gates ≤ 3 ? 11 : 14 − gates`.
**Why:** the first 3 gate sequences land on the dedicated gate-only jacks and cost no CV (0→11 CV, 3→11 CV, 6→8 CV, 14→0 CV); converting CV jacks only begins past 3. Pure CV (0 gates) is valid.
12.4 Wire-protocol impact **MUST** be none: a CV jack acting as a gate carries 0/full (or the 2-level velocity) in its `u16` slot; the 3 dedicated gate bits stay digital pins (`docs/specs/useq-cv-protocol.md`). `CvSpec` extends so PWM jacks can also be gate targets.
**Why:** reusing the existing protocol slots avoids a protocol revision for a routing feature.
12.5 The **Rhythm network** toggle (shown only when gate sequences > 0) selects **Separate** (default) or **Shared**. Separate gives the Rhythm stream its own MLP and routes each input channel to one engine, automatically by source kind: XY pad → a second on-screen pad (pad 1 → Continuous, pad 2 → Rhythm); gamepad → double-stick (left → Continuous, right → Rhythm); MIDI controller → per-CC `Continuous | Rhythm` toggle. Shared uses one MLP for both streams (all inputs feed it; the hardware-parity hybrid).
**Why:** the decision surface stays **two numbers + one toggle** — the second pad / double-stick / per-CC tag is a *consequence* of Separate, surfaced inline, reusing the existing input layer (`InputSource.axisCount()` / `axisLabels()`).
12.6 `BackendAdvanced` (full-depth modal) **MAY** override per-output defaults: which model output drives each CC/CV; which rhythm track drives each note/gate; gate threshold; MIDI note#/channel; CV polarity.
**Why:** identity defaults cover the common case, so advanced overrides stay optional.
---
## 13. Proposed UI (the "simple" target)
13.1 The Outputs panel **SHOULD** read, top to bottom: (1) **Output kinds** — `Continuous [n]` + `Gate sequences [n]` (MIDI) or the linked `Gate sequences 0–14` slider with a live `CV n · Gate n` readout (CV); (2) the **Rhythm network** toggle `( Shared | Separate )` plus its inline input surface (2nd pad / double-stick / per-CC tags), shown only when gate sequences > 0; (3) the existing per-output rows (off/fixed/live, mute, arm, min/max/curve) and per-backend specifics (CC#, CV jack, note#).
**Why:** the counts + toggle *are* the mode, so no mode dropdown is needed for the rhythm/continuous split.
---
## Open / Deferred
Part I (shipped) caveats:
- **Browser↔firmware noise parity is intentionally NOT guaranteed.** `playground/src/ml/jolt.ts` and `playground/src/output/ou-explore.ts` reimplement the gesture math in TypeScript (driving weights via the existing `nisps_ml_get/set_weights` bindings) rather than calling C++ `ml::Jolt` / `ml::OUNoise` through WASM, and they use `Math.random()` not the deterministic `Rng`. Acceptable for stochastic exploration aids, but the noise itself will not be bit-identical across firmware and browser.
- **The post-release LR ramp (§3.4) is not ported to the browser.** The browser SLP mode trains only on explicit gestures, so there is no resumed continuous trainer to ease back in.
- **Verified in the dev environment:** C++ host tests (`tests/cpp/test_mlp_jolt.cpp`, `test_mlp_ou_noise.cpp`, `test_mode_learning.cpp`, `test_mode_concepts.cpp`), lint, WASM build, native↔WASM parity, codegen golden, playground typecheck.
- **NOT verifiable in the dev environment (hardware/toolchain-bound; corresponds to verification chokepoints A/B):** firmware compile (no arduino-cli / submodules), Playwright e2e, on-hardware audio. To close: run `scripts/build-firmware.sh SLPWorkshop`, then flash on the RP2350.
- **NOT verifiable in the dev environment (hardware/toolchain-bound; verification chokepoints A/B):** firmware compile (no arduino-cli / submodules), Playwright e2e, on-hardware audio. To close: run `scripts/build-firmware.sh SLPWorkshop`, then flash on the RP2350.
Part II (planned) build deltas, when we proceed:
- **Manifold engine:** today one MLP head on the spine. "Separate" needs a second Rhythm MLP head + per-input-axis engine routing (engine + input-layer scope).
- **TS RatioSeq:** a browser-side RatioSeq (or a `pop_events()` bridge) to turn Rhythm-MLP params into gate/note events for the CV & MIDI backends (2 sequences on firmware; arbitrary count in the browser).
- **MIDI backend:** gate sequences → note on/off (it currently sends CC only).
- **CvSpec:** allow PWM jacks (`cv1..cv11`) to be gate targets (incl. velocity-CV), with the 3 digital pins as the first-used gate jacks.
- **Reshape:** shared-net `output_size = continuous + 7 × gate_sequences` (≤126 WASM cap) — confirm against the reshape / reset-on-reshape flow.
- **Firmware output_size** for the "Continuous only" / "Rhythm only" slices of `slp_workshop` — the firmware agent's call; this spec fixes the shapes, not exact counts.
- **Deferred niceties (explicitly not blocking):** external-MIDI-clock sync; per-track pulse-width (gate length, the optional 8th ratio-seq param); continuous velocity/accent beyond the 127/64 two-level.

View file

@ -1,4 +1,10 @@
# uSEQ-CV wire protocol v2
---
kind: spec
stability: stable
layer: binding
---
# uSEQ-CV Wire Protocol v2
The protocol the Manifold **CV output backend** (`manifold/src/backends/cv-backend.ts`)
speaks over USB Web Serial to the uSEQ main module, which drives CV/gate jacks and

321
docs/specs/vcv-module.md Normal file
View file

@ -0,0 +1,321 @@
---
kind: spec
stability: evolving
layer: binding
counterpart: backends-spec.md
---
# MEMLNaut VCV Rack Module — Specification
**Revision note:** pruned 2026-07-21 to the current 8→16 contract. The 2→12-era design
sections (phase plan, per-node cost tables, nisps-core prerequisites, dead OSC verbs) are in git
history. This file is also the single `.nisps`/patch format spec — the separate
`vcv/NISPS-FORMAT.md` (which documented the retired v1 nested-weights format) was deleted with
this prune.
---
## Overview
A VCV Rack module that embeds the NISPS interactive ML engine as a CV-to-CV mapper. Users explore
high-dimensional parameter spaces via reinforcement-learning feedback, producing **16 raw CV
outputs** (each with a custom LED-ring indicator) from **8 CV inputs**.
The module does **not** produce sound. It maps input CVs through a trained neural network to
output CVs, which the user patches into other modules (VCOs, VCFs, VCAs, etc.). The result: a
learned, nonlinear, high-dimensional modulation source shaped by the user's aesthetic preferences.
**Plugin name:** MEMLNaut (`vcv/plugin.json`, currently v0.2.0, license field "proprietary")
**Target:** VCV Rack 2 only (see the v2-only decision at the end)
**Distribution:** not submitted to the VCV Library; `.vcvplugin` bundles published at
`meml.lnfinitemonkeys.org/next/vcv/` (see `vcv/DISTRIBUTION.md`; local builds land in the
git-ignored `vcv/dist/`). Note: VCV SDK is GPLv3 — linking against it effectively makes the
combined binary GPL.
---
## Architecture
### Core Stack
```
┌─────────────────────────────────────┐
│ VCV Module (UI + I/O) │
│ Panel, knobs, ports, display │
├─────────────────────────────────────┤
│ VCV process() callback │
│ Reads CV inputs, writes CV outputs │
│ Decimated inference trigger │
├─────────────────────────────────────┤
│ src/iml.hpp — thin adapter over │
│ nisps::ml::MLPCore<DynamicStorage> │
│ (the shared C++20 core in nisps/) │
├─────────────────────────────────────┤
│ State Manager │
│ Serialize/deserialize weights, │
│ examples, config to JSON │
└─────────────────────────────────────┘
```
### Core Library Integration
**Delta #5 CLOSED (2026-07-18, one-core-engine-refactor P6).** The module no longer
vendors its own MLP. `vcv/src/iml.hpp` is a THIN, Rack-free adapter over the shared
core: `nisps::ml::MLPCore<nisps::ml::DynamicStorage>` (the runtime-shaped branch of the one
core MLP — fixed 4-layer ReLU×3 + Sigmoid topology, three runtime hidden sizes) with the
module's real `[16, 24, 16]` shape, `nisps::Rng` (nisps/core/rng.hpp) replacing the vendored
`DetRng`, and the core MLP's own FIFO dataset replacing the vendored `Dataset`. The adapter
includes only nisps headers + the standard library (no Rack includes) so the host ctest can
compile it. Behaviour is **core-exact**: weight init, RL `move_weights`, SGD training,
activations and RNG are bit-identical to the firmware/WASM engine — pinned by
`tests/cpp/test_vcv_iml_parity.cpp` (adapter == bare `MLPCore<DynamicStorage>`, memcmp-equal).
### Threading Model
- **Audio thread** (`process()`): reads input CVs, runs MLP inference (decimated), writes output
CVs. Never blocks.
- **Worker thread**: handles training (thumbs-up) and weight perturbation (thumbs-down). On
completion, signals the audio thread to swap in the new weights and crossfade.
- **Widget thread**: draws UI, reads output values for display.
**Threading invariant** (documented at `vcv/src/MEMLNaut.cpp` `iml`/`imlShadow`): only the audio
thread touches `iml`; the worker thread operates exclusively on `imlShadow`. Hand-off is through
atomic-flagged staging buffers (`pendingWeights`, staged example copies) — the worker deep-copies
example vectors into staging before flagging, and the audio thread never reads `imlShadow`
directly. Feedback taps arriving mid-job are coalesced (latest pending state wins).
**Post-swap output crossfade**: when weights are swapped, outputs may jump discontinuously. The
context-menu slew setting (default 10 ms) crossfades old→new output vectors to prevent clicks.
**Multiple instances**: each module instance is fully independent (own adapter instances, own
worker thread, own OSC port = 7001 + instance-id % 64).
### Input Signal Handling
- **Polyphonic inputs**: channel 0 only (monophonic).
- **Input clipping**: CV inputs are hard-clamped to their configured range before normalisation
to [0, 1]. Unipolar (default): clamp [0, 10 V], ÷10. Bipolar: clamp [−5, +5 V], +5, ÷10.
---
## I/O Specification
### Inputs (8 CV inputs + control ports)
| Port | Description |
|------|-------------|
| IN 1–8 | CV inputs feeding the 8-input MLP |
| SPREAD CV | CV modulation of SPREAD knob (added to knob value, /10 V) |
| LEARN | Gate input: high accepts RL feedback (OR'd with the LEARN toggle) |
| + TRIG | Trigger input: thumbs-up |
| − TRIG | Trigger input: thumbs-down |
### Outputs
| Port | Description |
|------|-------------|
| OUT 1–16 | Raw MLP outputs scaled to the configured CV range; each jack has an LED ring and an attenuverter trimpot (−100%..+100%, default +100%) |
There are **no derived-output jacks** in the 8→16 module (the 2→12-era MEAN/STD/DELTA/NOVELTY/
CONFIDENCE jacks were dropped in the redesign). A context-menu toggle "Compute derived stats
(Mean/Std/Delta)" computes the stats on the audio thread, and a novelty distance is cached after
training — but as of 2026-07-21 **no jack, display, or OSC verb consumes these values** (see Open
Questions).
### Panel Controls
| Control | Type | Description |
|---------|------|-------------|
| SPREAD | Knob (default 60%) | Weight-init scale, RL noise cap, weight decay |
| RATE | Knob | Inference rate: block-rate (~170 Hz) … audio-rate (44.1 kHz) |
| + | Momentary | Thumbs up (requires LEARN) |
| − | Momentary | Thumbs down (requires LEARN) |
| LEARN | Toggle + green LED | Enable/disable learning; inference always runs |
| RAND | Momentary | Randomise network weights (uses SPREAD) |
| CLEAR | Momentary (long-press ~1 s) | Clear all examples and reset network |
A yellow TRAIN LED flashes during worker-thread training. The display strip draws the 16 output
bars (palette-coloured), the noise level (`N:`), the example count (`n/100`), and `TRAIN` while
training.
### Context Menu (the real, current one)
| Setting | Description |
|---------|-------------|
| Output ranges | Per-output Unipolar (0–10 V, default) / Bipolar (±5 V) |
| Input ranges | Per-input Unipolar (0–10 V, default) / Bipolar (±5 V) |
| Compute derived stats (Mean/Std/Delta) | Toggle; see note above — currently unconsumed |
| Output slew | 0 / 5 / 10 (default) / 20 / 50 / 100 ms |
| Presets (.nisps) | Save / Load `.nisps` preset files (format below) |
| Browser bridge (WS↔OSC) | Enable OSC server; listen port ∈ {7001, 7002, 7003, 9000, 9001} |
---
## Visual Feedback — Per-Output LED Rings
Each of the 16 output jacks is surrounded by a custom LED ring (`vcv/src/LedRing.hpp`, NanoVG on
`drawLayer()` layer 1). The ring arc fills proportionally to the output's current value; the
track ring is dimly visible always. Colours come from `vcv/src/palette.hpp` (hand-written from
the frontend design tokens — `--accent` #ff6a00 etc.): the 16 outputs read as a clean
orange→cyan-anchored ramp across the jacks.
---
## MLP Configuration
```
Inputs: 8
Hidden: [16, 24, 16] (3 hidden layers, ReLU)
Output: 16 (sigmoid, values in [0, 1])
```
`vcv/src/MEMLNaut.cpp`: `NUM_ML_INPUTS = 8`, `NUM_ML_OUTPUTS = 16`,
`nisps::IML<float> iml{NUM_ML_INPUTS, NUM_ML_OUTPUTS, {16, 24, 16}}`. The topology is fixed; the
core stores biases explicitly (no phantom "+1 bias node" in the layer counts).
### Spread Parameter
Identical to the browser/firmware engine (the core methods):
1. **Weight initialisation**: `draw_weights(spread)` — uniform [−1,1] (spread=0) → Xavier (spread=1)
2. **RL perturbation**: `move_weights(speed, spread)` — per-layer scaling + weight decay
3. **Noise cap**: `0.3·(1−spread) + 0.05·spread`
### Dataset Capacity
100 examples (`vcv/src/iml.hpp` `kMaxExamples = 100`, kept equal to the old vendored capacity so
patch round-trips preserve counts). FIFO: oldest example dropped when full. Note the nisps core's
own default is 128 (`kDefaultMaxExamples`); the VCV adapter pins 100 explicitly.
---
## Inference Rate
RATE knob, exponential mapping `period = 256 · (1/256)^rate` samples:
| Position | Rate | Behavior |
|----------|------|----------|
| Full CCW | ~170 Hz | Once per 256 samples. Cheapest. |
| 12 o'clock | ~2.8 kHz | Every 16 samples. Good for CV-rate modulation. |
| Full CW | 44.1 kHz | Every sample. Audio-rate CV. |
Between inference steps, outputs are linearly interpolated; the post-training crossfade (slew)
composes on top.
---
## RL Feedback Workflow
**Thumbs up (+)**: capture current input/output pair as a training example → enqueue training on
the worker → `noiseLevel *= 0.97`.
**Thumbs down (−)**: `noiseLevel = min(noiseLevel · 1.5, noiseCap)` → enqueue perturbation
(`move_weights(noiseLevel, spread)`) on the worker → audio thread crossfades to the new weights.
**Learn gate**: when LEARN is off (gate low AND toggle off), +/− are ignored; inference always
runs ("learn off = play mode").
---
## State Persistence — the ONE `.nisps`/patch format (version 3)
Both the VCV patch state and menu-saved `.nisps` files use the same JSON, produced by
`MEMLNaut::dataToJson()`:
```json
{
"version": 3,
"inputCount": 8,
"outputCount": 16,
"noiseLevel": 0.1,
"slewMs": 10.0,
"computeDerived": false,
"oscEnabled": false,
"oscPort": 7001,
"outputRangeUnipolar": [true, "… ×16"],
"inputRangeUnipolar": [true, "… ×8"],
"weights": ["… flat float array"],
"examples": { "features": [["… ×8"]], "labels": [["… ×16"]] },
"mlpConfig": { "layers": [8, 16, 24, 16, 16] },
"params": ["only in menu-saved .nisps files: all param values incl. attenuverters"]
}
```
- **`weights` is the core's FLAT vector**: `[layer0_w … layer3_w][layer0_b … layer3_b]` — the
exact `MLPCore` weight layout, biases included. The old v1 nested 3-D
`weights[layer][node][weight]` blobs (bias not serialised) do **not** load: `set_weights`
rejects them on a size guard.
- **`mlpConfig.layers`** are true node counts `[8, 16, 24, 16, 16]` — no "+1 bias" first entry.
- **`examples`**: parallel `features` (×8 floats) / `labels` (×16 floats) arrays, ≤100 entries.
- The menu "Load .nisps preset…" rejects files with `version < 1`; unknown extra fields are
ignored. `params` is appended only by the menu save (patch save round-trips params natively).
---
## Browser Bridge (WS ↔ UDP OSC)
The module runs a UDP OSC server (`vcv/src/osc_server.hpp`, transport-only). The browser
(`manifold/src/backends/vcv-backend.ts`) connects via the Deno WS↔UDP bridge
(`manifold/osc-bridge/bridge.ts`; WS port default 8765 — pass `--osc-port` to match the module's
port, the bridge's own default is 9000).
### Live OSC verbs (the complete set)
| Verb | Direction | Payload |
|------|-----------|---------|
| `/nisps/input` | Browser → VCV | float array — input vector; while streaming, the module runs in **bridged mode** (browser drives the model inputs instead of the jacks) |
| `/nisps/output` | VCV → Browser (~100 ms throttle) | 16 floats — live outputs (alive-proof + visualisation) |
| `/nisps/input` | VCV → Browser (~100 ms throttle) | 8 floats — live inputs (echo) |
| `/nisps/feedback` | Browser → VCV | JSON string op (`up` / `down` / `rand` / `clear`, optional `spread`, `input`/`output` vectors), routed through the same paths as the panel buttons |
The 2→12-era `/nisps/weights`, `/nisps/examples`, and `/nisps/state` full-state sync verbs were
**deleted** (both directions had zero consumers; Rack patch save/load owns persistence).
Bidirectional *training* remains: both the browser verdict loop and the module panel drive the
same MLP.
**Port assignment:** default UDP 7001 + per-instance offset (`7001 + id % 64`); the context menu
offers {7001, 7002, 7003, 9000, 9001}.
---
## Panel Layout
The module is **44HP** (`res/MEMLNaut-wide.svg`, 223.52 mm — the SVG the widget actually loads).
`res/MEMLNaut.svg` is an unused 30HP variant and `res/MEMLNaut-expander.svg` an unused 8HP
expander panel; no expander module is implemented.
- Top: display strip (16 output bars, noise level, example count, TRAIN)
- Upper: SPREAD + RATE knobs, LEARN toggle + LED, RAND, CLEAR, +/− buttons
- Middle: 8 input jacks, SPREAD CV, LEARN gate, +/− trigger inputs
- Lower: 16 output jacks (2 rows of 8) with LED rings and per-output attenuverter trimpots
---
## Build
```bash
cd vcv
export RACK_DIR=/path/to/Rack-SDK # default: ~/.local/share/Rack2/Rack-SDK
make
make install
```
Files: `src/plugin.{hpp,cpp}`, `src/MEMLNaut.cpp` (module + widget + serialization + OSC wiring),
`src/iml.hpp` (core adapter), `src/osc_server.hpp`, `src/LedRing.hpp`, `src/palette.hpp`,
`res/*.svg`, `Makefile` (adds `-std=c++20`; the core is reached via relative `../../nisps/…`
includes from `src/`, no extra `-I`). Cross-platform bundles are produced by `build-mac.sh` /
`build-win.sh` into `dist/` (see `vcv/BUILDING.md`, `vcv/DISTRIBUTION.md`).
**Rack version decision: ship v2-only.** The v1 SDK lacks the menu/tooltip APIs used throughout
and may not support the C++20 the core requires; the port (~8 h) can be done later if demand
materialises.
---
## Open Questions
1. **Derived stats are computed but unconsumed** — the context-menu toggle computes Mean/Std/
Delta (and novelty is cached after training) with no jack, display, or OSC consumer. Either
wire a consumer or delete the compute path.
2. **Hidden-layer sizing**: [16, 24, 16] carried over from the 2→12 era; untuned for 8→16.
3. **Expander**: an 8HP expander SVG exists with no module behind it — build or delete.

View file

@ -12,19 +12,66 @@
// The two `stereosample_t` types — the pre-existing firmware POD and the new
// nisps namespaced one — have identical memory layout (two floats L,R) so the
// bridge does an explicit field-wise copy. No reinterpret_cast.
//
// All hot code lives in flash-resident SRAM via NISPS_AUDIO_FUNC.
#pragma once
#include <Arduino.h>
#include "../src/nisps/core/perf.hpp"
#include "../src/nisps/core/types.hpp"
#include "../src/memllib/audio/AudioDriver.hpp"
#include "nisps/core/perf.hpp"
#include "nisps/core/types.hpp"
#include "audio/AudioDriver.hpp"
#include "codec_config.hpp"
namespace nisps_firmware {
// ---------------------------------------------------------------------------
// Driver configuration — the active mode decides how the codec is set up.
//
// `Mode::driver_config()` (nisps/modes/base.hpp) returns the mode's engine's
// `nisps::DriverConfig`, or the mode's own override when the engine isn't what
// consumes the audio input. Nothing here knows which mode is compiled in: the
// two entry points below are the whole of the glue, and a mode that expresses
// no opinion gets `nisps::DriverConfig{}`'s defaults, which reproduce the
// firmware's historical hardcoded codec setup.
// ---------------------------------------------------------------------------
// Field-for-field translation into memllib's codec struct, after clamping to
// what the SGTL5000 can represent (see codec_config.hpp — that part is
// host-tested in tests/cpp/test_mode_driver_config.cpp).
inline AudioDriver::codec_config_t to_codec_config(const nisps::DriverConfig& cfg) noexcept {
const nisps::DriverConfig c = clamp_driver_config(cfg);
AudioDriver::codec_config_t out{};
out.mic_input = c.mic_input;
out.line_level = static_cast<size_t>(c.line_level);
out.mic_gain_dB = static_cast<size_t>(c.mic_gain_db);
out.output_volume = c.output_volume;
return out;
}
// Publish the mode's preferred sample rate to the driver.
//
// MUST run before `set_sys_clock_khz(AudioDriver::GetSysClockSpeed(), ...)` —
// the system clock is derived from the rate, and `GetSysClockSpeed()` panics on
// a rate it has no divider for. `select_sample_rate` therefore resolves
// "don't care" (0) and anything unsupported to 48 kHz rather than letting it
// reach the driver. Called on core 0 in setup(), i.e. before core 1 gets past
// its `g_serial_ready` handshake and reads `GetSampleRate()`.
template <typename Mode>
inline void apply_mode_sample_rate(const Mode& mode) noexcept {
AudioDriver::SetSampleRate(
static_cast<size_t>(select_sample_rate(mode.driver_config().sample_rate)));
}
// Bring the audio driver up configured for the active mode: codec input source
// (mic vs line), input gain step, mic pre-amp gain, analog output volume.
// Replaces the old parameterless `AudioDriver::Setup()`, which hardcoded line
// input for every variant regardless of what the mode's engine asked for.
template <typename Mode>
inline bool setup_audio_driver(const Mode& mode) {
return AudioDriver::Setup(to_codec_config(mode.driver_config()));
}
// Pointer to the active mode. The audio block callback reads through this.
// Set during setup1() before AudioDriver::Setup() is called. Marked
// `__not_in_flash("audio")` so the audio ISR path does not pay flash latency.

View file

@ -0,0 +1,67 @@
// firmware/glue/codec_config.hpp — Pure, host-testable translation of a
// mode's `nisps::DriverConfig` into what the SGTL5000 codec can actually be
// asked for.
//
// Deliberately Arduino-free and memllib-free so `tests/cpp/` can exercise it
// on the host: this is the only part of the mic/line wiring that has logic in
// it, and it is the part that can panic the device if it gets the sample rate
// wrong. `glue/audio_driver.hpp` does the remaining (field-for-field) copy
// into `AudioDriver::codec_config_t`.
//
// Codec limits are read off the vendored driver, not invented:
// - `AudioControlSGTL5000::lineInLevel` (control_sgtl5000.cpp) clamps to 15;
// the register is 4 bits per channel.
// - `AudioControlSGTL5000::micGain` saturates at preamp step 3 (+40 dB) plus
// input_gain 15, i.e. 40 + ceil(15 * 3 / 2) = 63 dB. Above that the codec
// setting is identical, so we clamp to the point of saturation.
// - `AudioDriver::Setup` already truncates output volume above 0.99; it does
// NOT guard against a negative, which would run through
// `AudioControlSGTL5000::calcVol` — so clamp both ends here.
// - `AudioDriver::GetSysClockSpeed` `panic()`s on any sample rate outside
// {24000, 32000, 44100, 48000}. An engine asking for anything else must
// NOT brick the boot, so we fall back to 48 kHz.
#pragma once
#include <cstdint>
#include "nisps/core/types.hpp"
namespace nisps_firmware {
inline constexpr std::uint8_t kMaxLineLevel = 15u;
inline constexpr std::uint8_t kMaxMicGainDb = 63u;
inline constexpr float kMaxOutputVolume = 0.99f;
inline constexpr std::uint32_t kDefaultSampleRate = 48000u;
// The rates `AudioDriver::GetSysClockSpeed()` knows a system-clock for.
inline constexpr std::uint32_t kSupportedSampleRates[] = {
24000u, 32000u, 44100u, 48000u,
};
// Fold a mode's requested config into the codec's representable range.
// Everything out of range is clamped, never wrapped or ignored.
inline constexpr nisps::DriverConfig clamp_driver_config(nisps::DriverConfig c) noexcept {
if (c.line_level > kMaxLineLevel) c.line_level = kMaxLineLevel;
if (c.mic_gain_db > kMaxMicGainDb) c.mic_gain_db = kMaxMicGainDb;
if (c.output_volume < 0.f) c.output_volume = 0.f;
else if (c.output_volume > kMaxOutputVolume) c.output_volume = kMaxOutputVolume;
return c;
}
// Resolve the rate the driver should actually run at.
// requested <= 0 ⇒ "don't care" ⇒ 48 kHz
// requested unsupported ⇒ would panic() ⇒ 48 kHz
// otherwise ⇒ the requested rate
// Matched with a small tolerance so a float literal like 44100.f that does not
// round-trip exactly still resolves.
inline constexpr std::uint32_t select_sample_rate(float requested) noexcept {
if (!(requested > 0.f)) return kDefaultSampleRate; // also catches NaN
for (const std::uint32_t rate : kSupportedSampleRates) {
const float diff = requested - static_cast<float>(rate);
if (diff > -0.5f && diff < 0.5f) return rate;
}
return kDefaultSampleRate;
}
} // namespace nisps_firmware

View file

@ -1,23 +0,0 @@
// firmware/glue/input_router.hpp — High-level input wiring entry point.
//
// Currently a thin re-export over `peripherals.hpp`. The intent is for this
// to grow into a schema-aware router (read mode.param_schema().input_channels
// names, look up matching hardware sources by name, attach callbacks). For
// now the trivial positional mapping in `bind_peripherals()` is sufficient
// since every existing mode's first N channels are joystick X/Y/Z(/W).
//
// Kept as its own header so growth of routing logic doesn't bloat
// peripherals.hpp.
#pragma once
#include "peripherals.hpp"
namespace nisps_firmware {
template <typename Mode>
inline void wire_inputs(Mode& mode) {
bind_peripherals(mode);
}
} // namespace nisps_firmware

View file

@ -26,10 +26,8 @@
#include <cstddef>
#include <cstdint>
#include <memory>
#include "../src/nisps/core/perf.hpp"
#include "../src/nisps/modes/base.hpp"
#include "../src/memllib/interface/MIDIInOut.hpp"
#include "nisps/modes/base.hpp"
#include "interface/MIDIInOut.hpp"
namespace nisps_firmware {
@ -69,7 +67,7 @@ inline void bind_midi_input(std::shared_ptr<MIDIInOut> midi, Mode& mode) {
// Drain the mode's ControlEvent ring and dispatch each event to MIDI out.
// Called at ~1 kHz from loop1(). Non-blocking; no allocations.
template <typename Mode>
NISPS_AUDIO_FUNC(inline void drain_mode_events(std::shared_ptr<MIDIInOut> midi, Mode& mode)) {
inline void drain_mode_events(std::shared_ptr<MIDIInOut> midi, Mode& mode) {
if (!midi) return;
std::array<::nisps::ControlEvent, 32u> buf{};
const std::size_t n = mode.pop_control_events(std::span<::nisps::ControlEvent>(buf));

View file

@ -1,18 +1,12 @@
// firmware/glue/mode_select.hpp — Compile-time mode selection.
//
// The active firmware mode is chosen at compile time via `MEMLNAUT_MODE_TYPE`
// (the .ino's master macro). The macro expands to one of the canonical mode
// type aliases below; build scripts rewrite the active line.
// The active firmware mode is chosen at compile time via `MEMLNAUT_MODE_TYPE`,
// defined per-variant as a PlatformIO build flag (one `[env:...]` per variant
// in platformio.ini, e.g. `-DMEMLNAUT_MODE_TYPE=MEMLNautModePAFSynth`) — there
// is no in-source registry and no build-time file rewriting.
//
// Each alias maps a short human-readable name (`MEMLNautModePAFSynth`) to a
// concrete `nisps::modes::*Mode` C++ type. The legacy
// `modes/MEMLNautMode*.hpp` wrappers are gone; the same identifier now refers
// to the new platform-agnostic mode type.
//
// This indirection (via using-aliases) lets the build script's mode-rewrite
// logic stay near-identical: it still sees lines of the form
// #define MEMLNAUT_MODE_TYPE MEMLNautModePAFSynth
// and rewrites between alternatives.
// concrete `nisps::modes::*Mode` C++ type.
#pragma once
@ -38,16 +32,16 @@
# undef round
#endif
#include "../src/nisps/modes/breakor.hpp"
#include "../src/nisps/modes/channel_strip.hpp"
#include "../src/nisps/modes/elysiamorf.hpp"
#include "../src/nisps/modes/external_synth_midi.hpp"
#include "../src/nisps/modes/memlcelium.hpp"
#include "../src/nisps/modes/paf_synth.hpp"
#include "../src/nisps/modes/slp_workshop.hpp"
#include "../src/nisps/modes/sound_analysis_midi.hpp"
#include "../src/nisps/modes/verb_fx.hpp"
#include "../src/nisps/modes/xiasri.hpp"
#include "nisps/modes/breakor.hpp"
#include "nisps/modes/channel_strip.hpp"
#include "nisps/modes/elysiamorf.hpp"
#include "nisps/modes/external_synth_midi.hpp"
#include "nisps/modes/memlcelium.hpp"
#include "nisps/modes/paf_synth.hpp"
#include "nisps/modes/slp_workshop.hpp"
#include "nisps/modes/sound_analysis_midi.hpp"
#include "nisps/modes/verb_fx.hpp"
#include "nisps/modes/xiasri.hpp"
// ---- Public name → nisps type aliases ----
// Build script greps for `MEMLNautMode*` lines, so we keep the prefix.
@ -75,32 +69,9 @@ using MEMLNautModeExtSynthAnalogKeys = ::nisps::modes::ExternalSynthMIDIMode<::n
using MEMLNautModeExtSynthHydrasynth = ::nisps::modes::ExternalSynthMIDIMode<::nisps::midi::generated::kAsmHydrasynth, 8u>;
using MEMLNautModeExtSynthJD800 = ::nisps::modes::ExternalSynthMIDIMode<::nisps::midi::generated::kRolandJd800, 8u>;
// ---- SelfTest pseudo-variant ----
// A standalone guided hardware self-test (see glue/selftest.hpp). It is NOT a
// nisps Mode — it drives the display + raw peripherals directly. We expose a
// tiny tag type so the `MEMLNautModeSelfTest` alias type-checks and the build
// script's `MEMLNautMode*` grep discovers the variant; the .ino forks on
// `NISPS_SELFTEST` and never instantiates this tag.
namespace nisps_firmware { namespace selftest { struct SelfTestRig {}; } }
using MEMLNautModeSelfTest = ::nisps_firmware::selftest::SelfTestRig;
// Compile-time guard: NISPS_SELFTEST expands to 1 iff MEMLNAUT_MODE_TYPE is
// MEMLNautModeSelfTest, else 0. Computed in the .ino *after* the mode #define.
// Two-level CAT so MEMLNAUT_MODE_TYPE expands before the paste.
#define NISPS_ST_MEMLNautModePAFSynth 0
#define NISPS_ST_MEMLNautModeChannelStrip 0
#define NISPS_ST_MEMLNautModeXIASRI 0
#define NISPS_ST_MEMLNautModeSoundAnalysisMIDI 0
#define NISPS_ST_MEMLNautModeBreakOr 0
#define NISPS_ST_MEMLNautModeVerbFX 0
#define NISPS_ST_MEMLNautModeElysiamorfs 0
#define NISPS_ST_MEMLNautModeMEMLCelium 0
#define NISPS_ST_MEMLNautModeExtSynthSub37 0
#define NISPS_ST_MEMLNautModeExtSynthSubPhatty 0
#define NISPS_ST_MEMLNautModeExtSynthPro12 0
#define NISPS_ST_MEMLNautModeExtSynthAnalogKeys 0
#define NISPS_ST_MEMLNautModeExtSynthHydrasynth 0
#define NISPS_ST_MEMLNautModeExtSynthJD800 0
#define NISPS_ST_MEMLNautModeSelfTest 1
#define NISPS_ST_CAT_(x) NISPS_ST_##x
#define NISPS_ST_CAT(x) NISPS_ST_CAT_(x)
// ---- SelfTest ----
// The guided hardware self-test (see glue/selftest.hpp) is NOT a nisps Mode —
// it drives the display + raw peripherals directly and is selected by its own
// `selftest` PlatformIO env, which defines `NISPS_SELFTEST=1` and leaves
// `MEMLNAUT_MODE_TYPE` undefined (src/main.cpp's `#if !NISPS_SELFTEST` fork
// never instantiates `ActiveMode` in that env, so no alias is needed here).

View file

@ -16,13 +16,14 @@
// `[0, N)` — that's the trivial mapping that matches every concrete mode's
// schema today (joystick first, optional 4D extra pot for joy_w).
//
// Buttons drive the InteractiveML primitives directly:
// MomA1 (TA up) : randomise / draw weights
// MomA2 (TA down) : clear examples (reset dataset)
// MomB1 (MA up) : randomise (synonym, deliberate)
// MomB2 (MA down) : nudge (small bounded perturbation)
// Buttons/toggles drive the SHARED ExploreAndPlace lifecycle. The authoritative
// per-control mapping is documented at the binding site below; in summary:
// MomA1 (TA up) : enter / exit explore (toggle)
// MomA2 (TA down) : like — freeze current scratchpad output (begin place)
// MomB1 (MA up) : Exploring → reroll scratchpad; Idle → grab (reposition)
// MomB2 (MA down) : Exploring → nudge scratchpad; Repositioning → drop+train
// TogB1 : Jolt — held continuous weight morph (up=morph, down=freeze)
// TogB2 : train (rising-edge → call ml.train())
// TogB2 : commit place (rising-edge) — store +1 example + train
// RVX1 : exploration amount (Ornstein-Uhlenbeck output walk)
//
// The button block below wires the SHARED ExploreAndPlace lifecycle
@ -45,9 +46,9 @@
#include <cstddef>
#include <cstdint>
#include "../src/nisps/core/perf.hpp"
#include "../src/nisps/ml/feedback.hpp"
#include "../src/memllib/hardware/memlnaut/MEMLNaut.hpp"
#include "nisps/core/perf.hpp"
#include "nisps/ml/feedback.hpp"
#include "hardware/memlnaut/MEMLNaut.hpp"
namespace nisps_firmware {
@ -60,16 +61,34 @@ inline constexpr std::uint64_t kFeedbackSalt = 0xFEEDBACC0DEull;
// D=4, firmware D=2 — SRAM budget).
inline constexpr std::size_t kFirmwareUndoDepth = 2u;
// Number of analog input channels we forward to the mode. Modes with fewer
// inputs ignore the surplus (set_input(idx, ...) silently rejects out-of-
// range idx in ModeBase).
inline constexpr std::size_t kAnalogInputCount = 4u; // X, Y, Z, plus one pot
// A small POD that the .ino owns. Captured by lambda below.
template <typename Mode>
struct PeripheralBindings {
Mode* mode = nullptr;
};
// Shared commit path for the two "commit the held output at the CURRENT
// joystick input" gestures (MomB2 drop-reposition, TogB2 commit-place).
// Captures the joystick input BEFORE the commit (commit_place restores the
// real net), runs the caller-named commit, then stores the +1 example
// (input → committed output) and warm-start trains — the CALLER owns
// training. The commit is picked by the call site, not by feedback state:
// dispatching on repositioning() here would change what TogB2 does when
// pressed mid-reposition (today it takes the commit_place path).
template <typename Mode, typename FB>
inline void commit_and_train(Mode& mode, FB& feedback, bool reposition) {
// Capture the current joystick input — the DESTINATION position.
const auto in = mode.input_channels();
std::array<float, Mode::ML::kInput> features{};
for (std::size_t i = 0; i < Mode::ML::kInput; ++i) {
features[i] = (i < in.size()) ? in[i] : 0.f;
}
if (reposition) {
feedback.commit_reposition(); // no weight restore (net never set aside)
} else {
feedback.commit_place(mode.ml()); // restores the real net
}
const auto label = feedback.committed_output(); // valid post-commit
if (!label.empty()) {
mode.ml().add_example(
std::span<const float>(features.data(), Mode::ML::kInput), label);
(void)mode.ml().train();
}
}
// Wire all hardware → mode bindings. Must be called *after*
// MEMLNaut::Initialize() (so MEMLNaut::Instance() is valid). The Mode
@ -115,8 +134,12 @@ inline void bind_peripherals(Mode& mode) {
//
// MomA1 (TA up) : down — enter explore (Idle→Exploring) /
// exit explore (Exploring→Idle) TOGGLE
// MomB1 (MA up) : randomise — reroll the scratchpad (Exploring)
// MomB2 (MA down) : nudge — small bounded perturbation (Exploring)
// MomB1 (MA up) : STATE-GATED — Exploring: reroll the scratchpad;
// Idle: GRAB (begin reposition — freeze the
// current trained-net output to carry it).
// MomB2 (MA down) : STATE-GATED — Exploring: nudge (small perturbation);
// Repositioning: DROP (commit the carried output
// at the current joystick input + train).
// MomA2 (TA down) : like — begin place (Exploring→Placing, freeze output)
// TogB2 (rising) : up — commit place (Placing→Idle) at the CURRENT
// joystick input, then store the +1 example and
@ -129,8 +152,6 @@ inline void bind_peripherals(Mode& mode) {
static FB feedback(static_cast<std::uint64_t>(0xC0FFEEu) ^ kFeedbackSalt);
feedback.set_mode(nisps::ml::FeedbackMode::ExploreAndPlace, mode.ml());
const float spread = mode.param_schema().default_spread;
// MomA1: enter/exit explore toggle.
meml->setMomA1Callback([&mode]() {
if (feedback.explore_state() == nisps::ml::ExploreState::Idle) {
@ -140,14 +161,40 @@ inline void bind_peripherals(Mode& mode) {
}
});
// MomB1: reroll the scratchpad (only in Exploring; no-op otherwise).
// MomB1: STATE-GATED.
// Exploring → reroll the scratchpad (a fresh random sound).
// Idle → GRAB: begin a reposition. Freezes the output currently heard
// from the trained net so the user can carry it to a new input
// location (the 4D variant has no joystick button, so the
// grab/drop gesture lives on this momentary toggle). The real
// net is NOT set aside — only the heard output is frozen.
meml->setMomB1Callback([&mode]() {
feedback.reroll(mode.ml(), mode.param_schema().default_spread);
switch (feedback.explore_state()) {
case nisps::ml::ExploreState::Exploring:
feedback.reroll(mode.ml(), mode.param_schema().default_spread);
break;
case nisps::ml::ExploreState::Idle:
feedback.begin_reposition(mode.ml()); // process + capture + hold
break;
case nisps::ml::ExploreState::Placing:
break; // already holding — ignore
}
});
// MomB2: nudge the scratchpad (small bounded perturbation; undoable).
// MomB2: STATE-GATED.
// Exploring → nudge the scratchpad (small bounded perturbation).
// Repositioning → DROP: commit the carried output at the CURRENT joystick
// input, store the +1 example (new input → carried output)
// and warm-start train. This is the "move an existing
// positive example to a new position" gesture.
meml->setMomB2Callback([&mode]() {
feedback.nudge(mode.ml(), 0.05f);
if (feedback.explore_state() == nisps::ml::ExploreState::Exploring) {
feedback.nudge(mode.ml(), 0.05f);
return;
}
if (feedback.repositioning()) {
commit_and_train(mode, feedback, /*reposition=*/true);
}
});
// MomA2: like → begin place. Freezes the current scratchpad output so the
@ -173,20 +220,7 @@ inline void bind_peripherals(Mode& mode) {
meml->setTogB2Callback([&mode](bool state) {
if (!state) return;
if (feedback.placing()) {
// Capture the current joystick input BEFORE the restore.
const auto in = mode.input_channels();
std::array<float, Mode::ML::kInput> features{};
for (std::size_t i = 0; i < Mode::ML::kInput; ++i) {
features[i] = (i < in.size()) ? in[i] : 0.f;
}
feedback.commit_place(mode.ml()); // restores the real net
// CALLER owns training: add the +1 example then warm-start train.
const auto label = feedback.committed_output(); // valid post-commit
if (!label.empty()) {
mode.ml().add_example(
std::span<const float>(features.data(), Mode::ML::kInput), label);
(void)mode.ml().train();
}
commit_and_train(mode, feedback, /*reposition=*/false);
} else {
(void)mode.ml().train();
}

View file

@ -15,8 +15,9 @@
//
// This logic is INHERENTLY hardware-coupled (it draws to the TFT and reads raw
// pins), so it lives firmware-side here in glue/, NOT under platform-agnostic
// nisps/. It is selected at build time via the `SelfTest` variant (see
// mode_select.hpp + the NISPS_SELFTEST fork in MEMLNaut-NISPS.ino).
// nisps/. It is selected at build time via the `selftest` PlatformIO env (see
// platformio.ini, which defines NISPS_SELFTEST=1; the fork lives in
// src/main.cpp).
//
// Threading: core 0 owns the step state machine (driven from MEMLNaut's
// loopCallback, ~5 ms) and the display. core 1 owns the audio sweep block
@ -48,14 +49,14 @@
#include <cstdio>
#include <functional>
#include "../src/memllib/PicoDefs.hpp"
#include "../src/memllib/utils/perf.hpp"
#include "../src/memllib/audio/AudioDriver.hpp"
#include "../src/memllib/interface/MIDIInOut.hpp"
#include "../src/memllib/hardware/memlnaut/MEMLNaut.hpp"
#include "../src/memllib/hardware/memlnaut/Pins.hpp"
#include "../src/memllib/hardware/memlnaut/display/View.hpp"
#include "../src/nisps/dsp/osc.hpp"
#include "PicoDefs.hpp"
#include "utils/perf.hpp"
#include "audio/AudioDriver.hpp"
#include "interface/MIDIInOut.hpp"
#include "hardware/memlnaut/MEMLNaut.hpp"
#include "hardware/memlnaut/Pins.hpp"
#include "hardware/memlnaut/display/View.hpp"
#include "nisps/dsp/osc.hpp"
// Inter-core boot-handshake flags. Defined in the .ino (shared by both the
// normal-mode and self-test build paths); declared here so this header is
@ -646,8 +647,11 @@ inline void setup() {
});
g_view = std::make_shared<SelfTestView>("Self Test");
// AddView appends to the carousel and DisplayDriver::Setup() leaves
// currentViewIndex_ at 0, so this — the selftest's only view — is already
// the one on screen. (There is no NavigateToView in DisplayDriver; a call
// to one used to sit here and had broken this variant's build.)
M->disp->AddView(g_view);
M->disp->NavigateToView(g_view);
M->setLoopCallback([] { tickStateMachine(); });
WRITE_VOLATILE(g_iface_ready, true);

View file

@ -3,7 +3,7 @@
// Adds entries to the MEMLNaut display carousel (the same DisplayDriver the
// SystemView and SelfTest use). Navigation: rotate to move between views;
// press to focus a view; rotate-while-focused to change its value; press
// again to unfocus. See src/memllib/.../display/DisplayDriver.hpp.
// again to unfocus. See lib/memllib/hardware/memlnaut/display/DisplayDriver.hpp.
//
// Currently provides:
// * Joystick: Dual / Single — for the 4-input ("two 2-D joystick") modes,
@ -24,8 +24,8 @@
#include <memory>
#include <span>
#include "../src/memllib/hardware/memlnaut/MEMLNaut.hpp"
#include "../src/memllib/hardware/memlnaut/display/SingleSelectView.hpp"
#include "hardware/memlnaut/MEMLNaut.hpp"
#include "hardware/memlnaut/display/SingleSelectView.hpp"
namespace nisps_firmware {

View file

@ -0,0 +1,373 @@
Mozilla Public License Version 2.0
==================================
1. Definitions
--------------
1.1. "Contributor"
means each individual or legal entity that creates, contributes to
the creation of, or owns Covered Software.
1.2. "Contributor Version"
means the combination of the Contributions of others (if any) used
by a Contributor and that particular Contributor's Contribution.
1.3. "Contribution"
means Covered Software of a particular Contributor.
1.4. "Covered Software"
means Source Code Form to which the initial Contributor has attached
the notice in Exhibit A, the Executable Form of such Source Code
Form, and Modifications of such Source Code Form, in each case
including portions thereof.
1.5. "Incompatible With Secondary Licenses"
means
(a) that the initial Contributor has attached the notice described
in Exhibit B to the Covered Software; or
(b) that the Covered Software was made available under the terms of
version 1.1 or earlier of the License, but not also under the
terms of a Secondary License.
1.6. "Executable Form"
means any form of the work other than Source Code Form.
1.7. "Larger Work"
means a work that combines Covered Software with other material, in
a separate file or files, that is not Covered Software.
1.8. "License"
means this document.
1.9. "Licensable"
means having the right to grant, to the maximum extent possible,
whether at the time of the initial grant or subsequently, any and
all of the rights conveyed by this License.
1.10. "Modifications"
means any of the following:
(a) any file in Source Code Form that results from an addition to,
deletion from, or modification of the contents of Covered
Software; or
(b) any new file in Source Code Form that contains any Covered
Software.
1.11. "Patent Claims" of a Contributor
means any patent claim(s), including without limitation, method,
process, and apparatus claims, in any patent Licensable by such
Contributor that would be infringed, but for the grant of the
License, by the making, using, selling, offering for sale, having
made, import, or transfer of either its Contributions or its
Contributor Version.
1.12. "Secondary License"
means either the GNU General Public License, Version 2.0, the GNU
Lesser General Public License, Version 2.1, the GNU Affero General
Public License, Version 3.0, or any later versions of those
licenses.
1.13. "Source Code Form"
means the form of the work preferred for making modifications.
1.14. "You" (or "Your")
means an individual or a legal entity exercising rights under this
License. For legal entities, "You" includes any entity that
controls, is controlled by, or is under common control with You. For
purposes of this definition, "control" means (a) the power, direct
or indirect, to cause the direction or management of such entity,
whether by contract or otherwise, or (b) ownership of more than
fifty percent (50%) of the outstanding shares or beneficial
ownership of such entity.
2. License Grants and Conditions
--------------------------------
2.1. Grants
Each Contributor hereby grants You a world-wide, royalty-free,
non-exclusive license:
(a) under intellectual property rights (other than patent or trademark)
Licensable by such Contributor to use, reproduce, make available,
modify, display, perform, distribute, and otherwise exploit its
Contributions, either on an unmodified basis, with Modifications, or
as part of a Larger Work; and
(b) under Patent Claims of such Contributor to make, use, sell, offer
for sale, have made, import, and otherwise transfer either its
Contributions or its Contributor Version.
2.2. Effective Date
The licenses granted in Section 2.1 with respect to any Contribution
become effective for each Contribution on the date the Contributor first
distributes such Contribution.
2.3. Limitations on Grant Scope
The licenses granted in this Section 2 are the only rights granted under
this License. No additional rights or licenses will be implied from the
distribution or licensing of Covered Software under this License.
Notwithstanding Section 2.1(b) above, no patent license is granted by a
Contributor:
(a) for any code that a Contributor has removed from Covered Software;
or
(b) for infringements caused by: (i) Your and any other third party's
modifications of Covered Software, or (ii) the combination of its
Contributions with other software (except as part of its Contributor
Version); or
(c) under Patent Claims infringed by Covered Software in the absence of
its Contributions.
This License does not grant any rights in the trademarks, service marks,
or logos of any Contributor (except as may be necessary to comply with
the notice requirements in Section 3.4).
2.4. Subsequent Licenses
No Contributor makes additional grants as a result of Your choice to
distribute the Covered Software under a subsequent version of this
License (see Section 10.2) or under the terms of a Secondary License (if
permitted under the terms of Section 3.3).
2.5. Representation
Each Contributor represents that the Contributor believes its
Contributions are its original creation(s) or it has sufficient rights
to grant the rights to its Contributions conveyed by this License.
2.6. Fair Use
This License is not intended to limit any rights You have under
applicable copyright doctrines of fair use, fair dealing, or other
equivalents.
2.7. Conditions
Sections 3.1, 3.2, 3.3, and 3.4 are conditions of the licenses granted
in Section 2.1.
3. Responsibilities
-------------------
3.1. Distribution of Source Form
All distribution of Covered Software in Source Code Form, including any
Modifications that You create or to which You contribute, must be under
the terms of this License. You must inform recipients that the Source
Code Form of the Covered Software is governed by the terms of this
License, and how they can obtain a copy of this License. You may not
attempt to alter or restrict the recipients' rights in the Source Code
Form.
3.2. Distribution of Executable Form
If You distribute Covered Software in Executable Form then:
(a) such Covered Software must also be made available in Source Code
Form, as described in Section 3.1, and You must inform recipients of
the Executable Form how they can obtain a copy of such Source Code
Form by reasonable means in a timely manner, at a charge no more
than the cost of distribution to the recipient; and
(b) You may distribute such Executable Form under the terms of this
License, or sublicense it under different terms, provided that the
license for the Executable Form does not attempt to limit or alter
the recipients' rights in the Source Code Form under this License.
3.3. Distribution of a Larger Work
You may create and distribute a Larger Work under terms of Your choice,
provided that You also comply with the requirements of this License for
the Covered Software. If the Larger Work is a combination of Covered
Software with a work governed by one or more Secondary Licenses, and the
Covered Software is not Incompatible With Secondary Licenses, this
License permits You to additionally distribute such Covered Software
under the terms of such Secondary License(s), so that the recipient of
the Larger Work may, at their option, further distribute the Covered
Software under the terms of either this License or such Secondary
License(s).
3.4. Notices
You may not remove or alter the substance of any license notices
(including copyright notices, patent notices, disclaimers of warranty,
or limitations of liability) contained within the Source Code Form of
the Covered Software, except that You may alter any license notices to
the extent required to remedy known factual inaccuracies.
3.5. Application of Additional Terms
You may choose to offer, and to charge a fee for, warranty, support,
indemnity or liability obligations to one or more recipients of Covered
Software. However, You may do so only on Your own behalf, and not on
behalf of any Contributor. You must make it absolutely clear that any
such warranty, support, indemnity, or liability obligation is offered by
You alone, and You hereby agree to indemnify every Contributor for any
liability incurred by such Contributor as a result of warranty, support,
indemnity or liability terms You offer. You may include additional
disclaimers of warranty and limitations of liability specific to any
jurisdiction.
4. Inability to Comply Due to Statute or Regulation
---------------------------------------------------
If it is impossible for You to comply with any of the terms of this
License with respect to some or all of the Covered Software due to
statute, judicial order, or regulation then You must: (a) comply with
the terms of this License to the maximum extent possible; and (b)
describe the limitations and the code they affect. Such description must
be placed in a text file included with all distributions of the Covered
Software under this License. Except to the extent prohibited by statute
or regulation, such description must be sufficiently detailed for a
recipient of ordinary skill to be able to understand it.
5. Termination
--------------
5.1. The rights granted under this License will terminate automatically
if You fail to comply with any of its terms. However, if You become
compliant, then the rights granted under this License from a particular
Contributor are reinstated (a) provisionally, unless and until such
Contributor explicitly and finally terminates Your grants, and (b) on an
ongoing basis, if such Contributor fails to notify You of the
non-compliance by some reasonable means prior to 60 days after You have
come back into compliance. Moreover, Your grants from a particular
Contributor are reinstated on an ongoing basis if such Contributor
notifies You of the non-compliance by some reasonable means, this is the
first time You have received notice of non-compliance with this License
from such Contributor, and You become compliant prior to 30 days after
Your receipt of the notice.
5.2. If You initiate litigation against any entity by asserting a patent
infringement claim (excluding declaratory judgment actions,
counter-claims, and cross-claims) alleging that a Contributor Version
directly or indirectly infringes any patent, then the rights granted to
You by any and all Contributors for the Covered Software under Section
2.1 of this License shall terminate.
5.3. In the event of termination under Sections 5.1 or 5.2 above, all
end user license agreements (excluding distributors and resellers) which
have been validly granted by You or Your distributors under this License
prior to termination shall survive termination.
************************************************************************
* *
* 6. Disclaimer of Warranty *
* ------------------------- *
* *
* Covered Software is provided under this License on an "as is" *
* basis, without warranty of any kind, either expressed, implied, or *
* statutory, including, without limitation, warranties that the *
* Covered Software is free of defects, merchantable, fit for a *
* particular purpose or non-infringing. The entire risk as to the *
* quality and performance of the Covered Software is with You. *
* Should any Covered Software prove defective in any respect, You *
* (not any Contributor) assume the cost of any necessary servicing, *
* repair, or correction. This disclaimer of warranty constitutes an *
* essential part of this License. No use of any Covered Software is *
* authorized under this License except under this disclaimer. *
* *
************************************************************************
************************************************************************
* *
* 7. Limitation of Liability *
* -------------------------- *
* *
* Under no circumstances and under no legal theory, whether tort *
* (including negligence), contract, or otherwise, shall any *
* Contributor, or anyone who distributes Covered Software as *
* permitted above, be liable to You for any direct, indirect, *
* special, incidental, or consequential damages of any character *
* including, without limitation, damages for lost profits, loss of *
* goodwill, work stoppage, computer failure or malfunction, or any *
* and all other commercial damages or losses, even if such party *
* shall have been informed of the possibility of such damages. This *
* limitation of liability shall not apply to liability for death or *
* personal injury resulting from such party's negligence to the *
* extent applicable law prohibits such limitation. Some *
* jurisdictions do not allow the exclusion or limitation of *
* incidental or consequential damages, so this exclusion and *
* limitation may not apply to You. *
* *
************************************************************************
8. Litigation
-------------
Any litigation relating to this License may be brought only in the
courts of a jurisdiction where the defendant maintains its principal
place of business and such litigation shall be governed by laws of that
jurisdiction, without reference to its conflict-of-law provisions.
Nothing in this Section shall prevent a party's ability to bring
cross-claims or counter-claims.
9. Miscellaneous
----------------
This License represents the complete agreement concerning the subject
matter hereof. If any provision of this License is held to be
unenforceable, such provision shall be reformed only to the extent
necessary to make it enforceable. Any law or regulation which provides
that the language of a contract shall be construed against the drafter
shall not be used to construe this License against a Contributor.
10. Versions of the License
---------------------------
10.1. New Versions
Mozilla Foundation is the license steward. Except as provided in Section
10.3, no one other than the license steward has the right to modify or
publish new versions of this License. Each version will be given a
distinguishing version number.
10.2. Effect of New Versions
You may distribute the Covered Software under the terms of the version
of the License under which You originally received the Covered Software,
or under the terms of any subsequent version published by the license
steward.
10.3. Modified Versions
If you create software not governed by this License, and you want to
create a new license for such software, you may create and use a
modified version of this License if you rename the license and remove
any references to the name of the license steward (except to note that
such modified license differs from this License).
10.4. Distributing Source Code Form that is Incompatible With Secondary
Licenses
If You choose to distribute Source Code Form that is Incompatible With
Secondary Licenses under the terms of this version of the License, the
notice described in Exhibit B of this License must be attached.
Exhibit A - Source Code Form License Notice
-------------------------------------------
This Source Code Form is subject to the terms of the Mozilla Public
License, v. 2.0. If a copy of the MPL was not distributed with this
file, You can obtain one at http://mozilla.org/MPL/2.0/.
If it is not possible or desirable to put the notice in a particular
file, then You may include the notice in a location (such as a LICENSE
file in a relevant directory) where a recipient would be likely to look
for such a notice.
You may add additional accurate notices of copyright ownership.
Exhibit B - "Incompatible With Secondary Licenses" Notice
---------------------------------------------------------
This Source Code Form is "Incompatible With Secondary Licenses", as
defined by the Mozilla Public License, v. 2.0.

View file

@ -0,0 +1,92 @@
# memllib — vendored, not a submodule
This directory is a **vendored copy** of the load-bearing subset of
[`MusicallyEmbodiedML/memllib`](https://github.com/MusicallyEmbodiedML/memllib), the
hardware-abstraction library for the MEMLNaut board (audio driver, TFT display, MIDI
I/O, peripherals). It replaced the `src/memllib` git submodule during the Phase 4
PlatformIO migration (2026-07-21).
## Provenance
- **Upstream repo**: `https://github.com/MusicallyEmbodiedML/memllib.git`
- **Vendored at commit**: `e291192d8e4f2fca7b79670c4df9c2ec8bdf03cd` (upstream `main`,
"l r input swap")
- **License**: MPL-2.0 (`LICENSE` in this directory, copied verbatim from upstream)
## What was copied, what was dropped
Copied verbatim, directory structure unchanged, under `src/`: `audio/`, `hardware/`,
`interface/`, `synth/`, `utils/`, `PicoDefs.hpp`. `LICENSE` sits at this directory's
root (metadata, not source). These are exactly the subdirectories the firmware sketch
used to reach via its symlink forest (`firmware/MEMLNaut-NISPS/src/memllib` before this
migration) — only the wrapping `src/` folder and the `library.properties` manifest are
new, both required for PlatformIO to discover and recursively compile this tree (see
below).
Dropped: `examples/` (never compiled; the firmware never referenced it, and its content
that mattered was already ported into `nisps/ml/{jolt,ou_noise,feedback,geo_push}.hpp`
per the pre-Phase-4 submodule-bump decision) — **except** `InterfaceRL.{hpp,cpp,tpp}` and
`InterfaceRLFileFormat.hpp`, which were added back on 2026-07-25 under `reference/`.
Dropping them was correct for the build and wrong for the codebase: `InterfaceRL` is the
source of truth for the whole feedback subsystem we ported, and with it out of tree we
missed upstream's redesign of the geometric dislike for months. `reference/` sits outside
`src/`, so PlatformIO does not compile it; see `reference/README.md`. Also dropped:
`.git` (submodule gitlink),
`.gitignore` (build-artifact patterns, meaningless once vendored — this repo's own
`.gitignore` covers it), `README.md` (described the old Arduino-IDE TFT_eSPI
`User_Setup_Select.h` copy-paste workflow, which PlatformIO replaces with
`-D USER_SETUP_LOADED=1` + explicit build flags in `platformio.ini` — see there).
98 files, ~1.9 MB total — all of memllib bar `examples/`; there is no smaller subset to
lift (every one of the 24 `.cpp` translation units here is reached by at least one
compiled firmware variant).
## Internal include convention (do not break)
Files inside this tree include each other with paths relative to `src/` as the root
(e.g. `src/hardware/memlnaut/MEMLNaut.cpp` does `#include "../PicoDefs.hpp"`,
`src/hardware/memlnaut/display/View.cpp` does `#include "../../PicoDefs.hpp"`).
This directory is consumed as a PlatformIO Arduino-format library (`lib/memllib/`, with
`library.properties` + a `src/` subfolder — the standard 1.5 Arduino library layout).
PlatformIO's Library Dependency Finder therefore adds `lib/memllib/src` (not
`lib/memllib` itself) to the include search path and recursively compiles every source
file under `src/`.
**Do not vendor these five subdirectories directly under `lib/memllib/`** (i.e. without
the `src/` wrapper) — that was tried first and silently compiles nothing: PlatformIO's
`ArduinoLibBuilder`, when it finds no `src/` subfolder, falls back to a *non-recursive*
"files directly in the library root" scan (the historical Arduino 1.0 library format,
which only special-cases a `utility/` subfolder). Nested folders like `audio/` or
`hardware/` are silently invisible to the build under that fallback — it links, or
rather fails to link, with `undefined reference to MEMLNaut::...` for every symbol in
this library. The `src/` subfolder switches PlatformIO onto the recursive path.
Firmware code outside this tree (`../../src/main.cpp`, `../../glue/*.hpp`) includes
headers here relative to `src/` as the root, e.g. `#include "audio/AudioDriver.hpp"`,
`#include "hardware/memlnaut/MEMLNaut.hpp"` — no `memllib/` or `src/` prefix, because
`lib/memllib/src/` *is* the include root PlatformIO adds.
## Re-syncing with upstream
There is no submodule to bump anymore, so a re-sync is a manual, documented diff:
1. Clone upstream at the desired commit: `git clone
https://github.com/MusicallyEmbodiedML/memllib.git /tmp/memllib-upstream`
2. Diff the five subdirs + `PicoDefs.hpp` against this directory's `src/`, e.g.:
```
diff -ru /tmp/memllib-upstream/audio firmware/MEMLNaut-NISPS/lib/memllib/src/audio
# ...repeat for hardware/ interface/ synth/ utils/ PicoDefs.hpp
```
3. Copy over the changed files (`cp -a`), re-run `diff -ru` both ways to confirm nothing
outside the tracked subset leaked in and nothing was silently dropped.
4. Update the "Vendored at commit" line above to the new upstream SHA + its subject
line.
5. Rebuild every `platformio.ini` env (`pio run`) and diff flash/RAM sizes against the
previous vendored commit's numbers — a size jump with no corresponding upstream
feature is a signal something unexpected changed.
6. Commit the vendored-file changes and this doc update together.
If upstream ever restructures these directories (renames, new cross-subdir relative
includes), the internal-include convention above may need re-verification — grep for
`#include "\.\./` inside this tree and confirm every relative path still resolves.

View file

@ -0,0 +1,9 @@
name=memllib
version=0.0.0
author=MusicallyEmbodiedML
maintainer=MusicallyEmbodiedML <https://github.com/MusicallyEmbodiedML/memllib>
sentence=Hardware abstraction for the MEMLNaut board (audio driver, TFT display, MIDI I/O, peripherals).
paragraph=Vendored subset of MusicallyEmbodiedML/memllib at commit e291192d8e4f2fca7b79670c4df9c2ec8bdf03cd. See VENDORED.md in this directory for provenance and the re-sync procedure.
category=Device Control
url=https://github.com/MusicallyEmbodiedML/memllib
architectures=rp2040

View file

@ -0,0 +1,6 @@
// InterfaceRL is now a class template (InterfaceRL<N_OUTPUTS>); its definitions
// live in InterfaceRL.tpp, included from InterfaceRL.hpp so each per-mode
// instantiation is available at its use site. This translation unit is
// intentionally empty (kept so existing build scripts that reference it still
// find a file).
#include "InterfaceRL.hpp"

View file

@ -0,0 +1,492 @@
#ifndef INTERFACERL_HPP
#define INTERFACERL_HPP
#include "../interface/InterfaceBase.hpp"
#include "../../memlp/StaticMLP.h"
#include "../../memlp/ReplayMemory.hpp"
#include "../../memlp/OrnsteinUhlenbeckNoise.h"
#include <memory>
#include "../utils/sharedMem.hpp"
#include "../PicoDefs.hpp"
//#include "../hardware/memlnaut/display.hpp"
#include "../hardware/memlnaut/display/MessageView.hpp"
#include "../interface/UARTInput.hpp"
#include "../interface/MIDIInOut.hpp"
#include "../hardware/memlnaut/display/MessageView.hpp"
#include "../hardware/memlnaut/display/BarGraphView.hpp"
#include "../hardware/memlnaut/display/RLView.hpp"
#include "../hardware/memlnaut/display/BlockSelectView.hpp"
#include "../hardware/memlnaut/display/SingleSelectView.hpp"
#include "../hardware/memlnaut/display/RotarySelectView.hpp"
#include "../hardware/memlnaut/display/NameInputView.hpp"
#include "../hardware/memlnaut/display/CCSelectView.hpp"
#include "InterfaceRLFileFormat.hpp"
#define RL_MEM __not_in_flash("rlmem")
struct trainStatelessRLItem {
std::vector<float> input ;
std::vector<float> action;
float reward;
};
// Non-template base holding the nested types + constants that callers reference
// without a template argument (e.g. InterfaceRLBase::INPUT_MODES in the modes
// and the .ino). Because this (and InterfaceBase) are *non-dependent* bases of
// InterfaceRL<N_OUTPUTS>, the template sees every inherited member by ordinary
// lookup — no this->/using-declarations required.
class InterfaceRLBase : public InterfaceBase
{
public:
using OnMIDICtrlCallback = std::function<void(uint8_t)>;
// Training data type (independent of the network's output width).
using training_pair_t = std::pair<std::vector<std::vector<float>>,
std::vector<std::vector<float>>>;
static constexpr size_t kMaxNNInputs = 10;
enum class INPUT_MODES {
JOYSTICK,
MACHINE_LISTENING,
JOYSTICK_AND_MACHINE_LISTENING,
SERIAL_INPUT
};
enum class INPUT_SOURCE : uint8_t {
JOYSTICK_3D = 0,
JOYSTICK_4D,
MACHINE_LISTENING,
MIDI_1CC,
MIDI_3CC,
MIDI_8CC,
COMBINED,
COUNT
};
enum class MEMORY_STORE_MODES {
ADD,
REPLACE_5_PERCENT,
REPLACE_10_PERCENT,
REPLACE_15_PERCENT,
REWARD_DECAY_10_PERCENT,
REWARD_DECAY_20_PERCENT
};
// Input-source state is independent of the network's output width, so it
// lives in the base: this lets N-agnostic helpers (e.g. MachineListeningMixin)
// hold an InterfaceRLBase* and still query/configure the input source.
INPUT_SOURCE getInputSource() const { return input_source_; }
void setHasMachineListening(bool v) { hasMachineListening_ = v; }
protected:
INPUT_SOURCE input_source_ = INPUT_SOURCE::JOYSTICK_3D;
bool hasMachineListening_ = false;
};
// The RL interface. N_OUTPUTS (the active mode's parameter count) is fixed at
// compile time, so the synth-mapping network is a static-memory StaticMLP with
// no heap allocation. The mode declares e.g. InterfaceRL<MyApp::kN_Params>.
template<size_t N_OUTPUTS>
class InterfaceRL : public InterfaceRLBase
{
public:
// Compile-time mapping network: kMaxNNInputs -> 16 -> 16 -> N_OUTPUTS.
using SynthMLP = smlp::StaticMLP<float,
smlp::Layout<kMaxNNInputs, 16, 16, N_OUTPUTS>,
smlp::Activations<RELU, RELU, HARDSIGMOID>,
loss::LOSS_FUNCTIONS::LOSS_MSE>;
InterfaceRL() : InterfaceRLBase()
// , ou_noise(0.02f, 0.0f, 0.2f, 0.001f, 0.0f)
{
}
void setup(size_t n_inputs, size_t n_outputs, bool addMessageView = true);
void optimise();
inline void setState(const size_t index, float value) {
controlInput[index] = value;
newInput = true;
}
// Force the next loop to regenerate + re-send the action, even if no input changed.
// Use when an output-stage parameter (e.g. a fade/home value) changes.
inline void markInputDirty() { newInput = true; }
void readAnalysisParameters(std::vector<float> params) override;
void generateAction(bool donthesitate=false);
inline void optimiseSometimes() {
if (optimiseCounter>=optimiseDivisor) {
optimise();
optimiseCounter=0;
newInput = true;
}else{
optimiseCounter++;
}
}
void storeExperience(float reward, std::vector<float> &experienceState, std::vector<float> &experienceAction );
#define randomWeightVariance 1.f
inline void randomiseTheNetwork()
{
synthMapping.RandomiseWeightsAndBiasesLin(-0.9f,1.1f, -0.9f, 0.3f);
newInput = true;
resetMinMaxFlag = true;
}
inline void setOptimiseDivisor(size_t newDiv) {
optimiseDivisor = newDiv;
}
void setOptimiseDivisorInterf(float value);
inline void forgetMemory() {
replayMem.clear();
}
inline void setRewardScale(float scale) {
rewardScale = scale;
}
inline void setLRScale(const float scale) {
learningRateScaled = learningRate * scale; // knob at 0 -> LR 0 -> training off (intended)
String msg = "LR scale: " + String(scale);
if (msgView) msgView->post(msg);
}
void setRewardScaleInterf(float value);
inline void setNoiseLevel(float level) {
// Knob [0,1] -> roaming amplitude (the OU walk's stationary std) in param space.
// theta/dt (set in setup) fix the smoothness; this only sets how far each param
// drifts from the mapping output. Low = gentle local wander; full = slow sweeps
// across the whole [0,1] range. kMaxAmplitude sets the "depth": higher reaches
// deeper into the param space (and interacts more with the [0,1] rails via the
// gentle reflection in generateAction), lower keeps it shallower/more local.
constexpr float kMaxAmplitude = 0.65f;
float amplitude = level * kMaxAmplitude;
if (amplitude < 0.01f) {
amplitude = 0.f;
if (msgView) msgView->post("Noise off");
} else {
String msg = "Explore amount: " + String(amplitude, 3);
if (msgView) msgView->post(msg);
}
for(auto& ou_noise: ou_noises) {
ou_noise->setStationaryStd(amplitude);
}
// Learning stays active during exploration on purpose: likes/dislikes given while
// the noise roams are what steer the network toward sounds the player wants.
if (nnOutputsGraphView) nnOutputsGraphView->setNoiseActive(amplitude > 0.f);
}
// Jolt = permanent weight modulation (B2, held). At press, pick a random subset of
// weights scattered across the net and roll a bounded random target for each. While
// held, EMA-glide each toward its target (stepJolt, called per loop); release just
// freezes them, so the change persists. Bounded by construction (interpolation toward
// targets in the weight-init range can't run away) and smooth (no per-tick jitter).
// Runs on the main loop / under the mlpActive lock, so touching the MLP here is safe.
inline float randomJoltTarget() const {
return kJoltWeightMin + (static_cast<float>(rand()) / RAND_MAX) * (kJoltWeightMax - kJoltWeightMin);
}
inline void startJolt() {
joltActive_ = true;
joltWeightLoc_.clear();
joltTarget_.clear();
// StaticMLP exposes a flat view over all weights (layer 0 first); pick
// random global indices to modulate.
const size_t total = SynthMLP::TotalWeights();
if (total == 0) return;
for (size_t i = 0; i < kJoltNumWeights; i++) {
joltWeightLoc_.push_back(rand() % total);
joltTarget_.push_back(randomJoltTarget());
}
if (nnOutputsGraphView) nnOutputsGraphView->setLastAction("jolt");
if (msgView) msgView->post("Jolt: morphing weights");
}
inline void stepJolt() {
const size_t total = SynthMLP::TotalWeights();
for (size_t i = 0; i < joltWeightLoc_.size(); i++) {
const size_t idx = joltWeightLoc_[i];
if (idx >= total) continue; // stale after a model load
float* wp = synthMapping.WeightPtrAt(idx);
if (!wp) continue;
float& w = *wp;
w += kJoltMorphRate * (joltTarget_[i] - w);
// Reached this target (EMA only asymptotes, so use a threshold) -> roll a new
// one, keeping the weight in motion for as long as the button is held.
float gap = joltTarget_[i] - w;
if (gap < 0.f) gap = -gap;
if (gap < kJoltTargetEpsilon) joltTarget_[i] = randomJoltTarget();
}
markInputDirty(); // weights changed -> regenerate + re-send the action
}
inline void stopJolt() {
joltActive_ = false; // weights stay where they morphed to (permanent)
joltLRRamp_ = 0.f; // resume learning from 0, ramping back to full over ~5s
}
void bind_RL_interface(INPUT_MODES input_mode = INPUT_MODES::JOYSTICK, bool joystick4D = false);
void bindInterface(INPUT_MODES input_mode = INPUT_MODES::JOYSTICK,bool joystick4D = false) {
bind_RL_interface(input_mode, joystick4D);
}
void bindInterface(bool disable_joystick=false, bool joystick4D = false) {
bind_RL_interface(disable_joystick ? INPUT_MODES::MACHINE_LISTENING : INPUT_MODES::JOYSTICK, joystick4D);
}
void bindUARTInput(std::shared_ptr<UARTInput> uart_input,
const std::vector<size_t>& kUARTListenInputs)
{
uart_input->SetCallback([this](size_t channel, float value) {
// Serial.println("UART input: " + String(channel) + " value: " + String(value));
if (channel < controlInput.size()) {
setState(channel, value);
}
});
}
void bindMIDI(std::shared_ptr<MIDIInOut> midi_interf, bool enableFootcontroller=false);
void setModeInfo(const String& modeRoot, const String& modeTag);
using ExtraSaveDataFn = std::function<std::vector<uint8_t>()>;
using ExtraLoadDataFn = std::function<void(const uint8_t*, uint16_t, uint16_t)>;
void setExtraSaveCallback(ExtraSaveDataFn fn) { _extraSaveFn = fn; }
void setExtraLoadCallback(ExtraLoadDataFn fn) { _extraLoadFn = fn; }
using RVCallback = std::function<void(float)>;
void setRVX1Override(RVCallback fn) { rvX1Override = std::move(fn); }
void setRVY1Override(RVCallback fn) { rvY1Override = std::move(fn); }
void setRVZ1Override(RVCallback fn) { rvZ1Override = std::move(fn); }
void setActiveDims(std::vector<bool> dims) { activeDims_ = std::move(dims); }
std::function<void(std::vector<float>&)> inputInjectionHook;
void trigger_like();
void trigger_dislike();
inline void getAction(std::vector<float> &out_action) {
out_action = action;
}
size_t getActiveInputCount() const {
switch (input_source_) {
case INPUT_SOURCE::JOYSTICK_3D: return 3;
case INPUT_SOURCE::JOYSTICK_4D: return 4;
case INPUT_SOURCE::MACHINE_LISTENING: return 6;
case INPUT_SOURCE::MIDI_1CC: return 1;
case INPUT_SOURCE::MIDI_3CC: return 3;
case INPUT_SOURCE::MIDI_8CC: return 8;
case INPUT_SOURCE::COMBINED: return kMaxNNInputs;
default: return kMaxNNInputs;
}
}
const std::vector<float>& getControlInput() const { return controlInput; }
// Recompute the unused-input pad value. Done only when the input mode changes (not per
// frame). Scales as 1.1/n_unused so the total constant injected into layer 1 stays
// bounded regardless of how many dims are unused — avoids over-driving the net.
void updateUnusedInputDefault() {
const size_t used = getActiveInputCount();
const size_t unused = (used < kMaxNNInputs) ? (kMaxNNInputs - used) : 0;
unusedInputDefault_ = (unused > 0) ? (1.1f / static_cast<float>(unused)) : 0.f;
}
// persist=false applies the change in memory + updates the bar graph without writing
// flash. A flash write stalls XIP execution on the RP2040 (blanking the display), so
// the rotary-driven path applies immediately but debounces the save (see loopCallback).
void setInputSource(INPUT_SOURCE src, bool persist = true) {
input_source_ = src;
updateUnusedInputDefault();
if (persist) saveInputSource();
if (nnInputsGraphView) nnInputsGraphView->setNumDisplayBars(getActiveInputCount());
}
// ISR-safe entry point (the rotary-encoder dispatch runs in interrupt context).
// setInputSource() does heap allocation (bar-graph resize), SPI (fillRect) and flash
// file IO — all unsafe in an ISR — so only record the request here and let the main
// loop apply it via the pendingInputSourceChange_ handler in bind_RL_interface().
void requestInputSource(INPUT_SOURCE src) {
pendingInputSource_ = src;
pendingInputSourceChange_ = true;
}
void addInputSourceView(bool includeCCSelect = true);
void SetMIDI5Callback(OnMIDICtrlCallback _cb_) {
midi5cb = _cb_;
}
void SetMIDI6Callback(OnMIDICtrlCallback _cb_) {
midi6cb = _cb_;
}
// Display views
std::shared_ptr<MessageView> msgView;
std::shared_ptr<BlockSelectView> fileSaveView;
std::shared_ptr<BlockSelectView> fileLoadView;
std::shared_ptr<NameInputView> nameInputView;
std::shared_ptr<BarGraphView> nnInputsGraphView;
std::shared_ptr<RLView> nnOutputsGraphView;
std::shared_ptr<SingleSelectView> memoryStoreModeView;
std::shared_ptr<CCSelectView> ccSelectView;
const std::vector<float>& getLastAction() const { return action; }
protected:
// Helper methods for trigger actions
void _perform_like_action();
void _perform_dislike_action();
void _perform_randomiseRL_action();
bool _save_RL_to_SD(String id);
bool _load_RL_from_SD(String id);
void _forget_replay_mem_interf();
void _saveSlotNames();
void _loadSlotNames();
static constexpr int kNumSlots = 12;
String slotNames[kNumSlots];
int pendingSaveSlot = -1;
private:
OnMIDICtrlCallback midi5cb = nullptr;
OnMIDICtrlCallback midi6cb = nullptr;
RVCallback rvX1Override;
RVCallback rvY1Override;
RVCallback rvZ1Override;
static constexpr size_t bias=1;
size_t optimiseDivisor = 1;
size_t optimiseCounter = 0;
bool newInput=false;
bool actionBeingDragged=false;
std::vector<size_t> itemsToRemove;
float raw_joystick_[4] = {};
float raw_ml_[6] = {};
float raw_midi_[8] = {};
// Constant used to pad the unused NN input dims; recomputed only on input-mode change.
float unusedInputDefault_ = 0.5f;
static constexpr const char* kInputSourceFile = "/input_source.bin";
void assembleInputs();
void copyAndZero(const float* src, size_t n);
void saveInputSource();
void loadInputSource();
void saveCCNumbers();
void loadCCNumbers();
size_t analysisParamsOffset = 0;
MEMORY_STORE_MODES memoryStoreMode = MEMORY_STORE_MODES::REPLACE_10_PERCENT;
std::array<String, 6> memOptions = {"Add", "Replace 5%", "Replace 10%", "Replace 15%", "Reward Decay 10%", "Reward Decay 20%"};
// Dislike repulsion: how far a 'no' moves the disliked action's training target away
// from the liked region (untapered), and the negative-batch LR base. Bigger = a 'no'
// slides the sound clearly further away; >1 tends to push params to the [0,1] rails.
static constexpr float kGeometricPushScale = 1.0f;
static constexpr float kNegLRBase = 1.5f; // negLRRatio = kNegLRBase - 0.4*negFraction
// A 'no' pushes at full strength for this long (wall-clock), then expires — no decay.
// The number of optimise cycles within the window depends on the mode's NN update rate.
static constexpr uint32_t kDislikeLifetimeMs = 2500;
static constexpr size_t kCentroidK = 4;
std::vector<bool> activeDims_;
bool removeItemsAtDistance(std::vector<float> &experienceState, const float distThreshold, const float reward);
void decayItemsAtDistance(std::vector<float> &experienceState, const float distThreshold);
std::vector<size_t> layers_nodes;
const bool use_constant_weight_init = false;
const float constant_weight_init = 0;
SynthMLP synthMapping; // value member -> lives in the (static) mode object: zero heap
float learningRate = 1e-3;
float learningRateScaled = learningRate;
std::vector<float> action;
ReplayMemory<trainStatelessRLItem> replayMem;
static constexpr size_t memoryLimit = 64;
static constexpr size_t batchSize = 8;
std::vector<float> mappingOutput;
std::vector<float> controlInput;
std::vector<float> savedAction;
float rewardScale = 1.0f;
// OrnsteinUhlenbeckNoise ou_noise;
std::vector<std::unique_ptr<OrnsteinUhlenbeckNoise>> ou_noises;
// Exploration-noise travel speed.
static constexpr float kNoiseDt = 0.004f; // normal OU travel speed (set in setup)
// Jolt = permanent weight modulation while B2 is held (see startJolt/stepJolt).
static constexpr size_t kJoltNumWeights = 40; // random weights perturbed per press
static constexpr float kJoltMorphRate = 0.017f; // EMA per tick (~1s to target @200Hz)
static constexpr float kJoltWeightMin = -1.2f; // target range == weight-init range
static constexpr float kJoltWeightMax = 0.9f;
static constexpr float kJoltTargetEpsilon = 0.05f; // re-roll target once within this
static constexpr float kJoltLRRampStep = 0.001f; // LR recovery rate: 1/(5s * 200Hz)
std::vector<size_t> joltWeightLoc_; // global flat weight indices (StaticMLP::WeightPtrAt)
std::vector<float> joltTarget_; // per-selected-weight target value
bool joltActive_ = false;
// After a jolt releases, learning resumes gently: effective LR *= joltLRRamp_, which
// climbs 0 -> 1 over ~5s so fresh training doesn't immediately drag the net off the
// jolted sound. 1.0 = normal (full LR).
float joltLRRamp_ = 1.0f;
bool resetMinMaxFlag = false;
// Deferred actions: set from ISR, consumed in main-loop loopCallback before optimise()
volatile bool pendingLike_{false};
volatile bool pendingDislike_{false};
volatile bool pendingDragStore_{false}; // drag-release: store savedAction
volatile bool pendingInputSourceChange_{false}; // input-source change: deferred from rotary ISR
INPUT_SOURCE pendingInputSource_{INPUT_SOURCE::JOYSTICK_3D};
// Debounced flash persistence: a save is scheduled this many ms after the last change,
// so scrolling through sources doesn't trigger a flash write (XIP stall) per detent.
static constexpr uint32_t kInputSourceSaveDelayMs = 600;
uint32_t inputSourceSaveDueMs_ = 0; // 0 = no save pending
spin_lock_t *mlpActive;
String _modeRoot{"mlp_rl"};
String _modeTag{"Unknown"};
ExtraSaveDataFn _extraSaveFn;
ExtraLoadDataFn _extraLoadFn;
};
// Template method definitions (header-only so the per-mode instantiation is
// available wherever InterfaceRL<N> is used).
#include "InterfaceRL.tpp"
#endif // INTERFACERL_HPP

File diff suppressed because it is too large Load diff

View file

@ -0,0 +1,19 @@
#ifndef INTERFACE_RL_FILE_FORMAT_HPP
#define INTERFACE_RL_FILE_FORMAT_HPP
#include <stdint.h>
static constexpr uint16_t MEML_FILE_FORMAT_VERSION = 1;
// Binary file layout:
// MEMLFileHeader (26 bytes)
// extra_size bytes of mode-specific data (may be 0)
// MLP binary (existing format)
struct MEMLFileHeader {
char magic[4]; // always "MEML"
uint16_t format_version; // MEML_FILE_FORMAT_VERSION
char mode_tag[16]; // null-padded mode identifier, e.g. "VerbFX"
uint16_t extra_size; // bytes of mode-specific data immediately following
};
#endif // INTERFACE_RL_FILE_FORMAT_HPP

View file

@ -0,0 +1,33 @@
# `reference/` — upstream source of truth, never compiled
These four files are `examples/InterfaceRL.{hpp,cpp,tpp}` +
`examples/InterfaceRLFileFormat.hpp`, copied verbatim from
[`MusicallyEmbodiedML/memllib`](https://github.com/MusicallyEmbodiedML/memllib) at
commit `e291192d8e4f2fca7b79670c4df9c2ec8bdf03cd` — the same commit the rest of this
vendored tree pins (see `../VENDORED.md`).
## Why they are here
`InterfaceRL` is the reference implementation of the whole NISPS feedback subsystem.
`nisps/ml/geo_push.hpp`, `nisps/ml/replay.hpp`, `nisps/ml/feedback.hpp`,
`nisps/ml/jolt.hpp` and `nisps/ml/ou_noise.hpp` are all ports of it, and several of them
still carry `// upstream InterfaceRL.hpp:NNN` line references.
The Phase-4 vendoring dropped `examples/` because nothing compiled it. That was correct
for the build and wrong for the codebase: with the source of truth out of tree, upstream
redesigned the geometric dislike (deleted the `/(1+len)` taper, doubled
`kGeometricPushScale`, tripled the negative-LR base, moved to batch training over all
negatives every tick) and we did not notice for months. Keeping these files in-tree turns
the next upstream drift into a `diff` instead of an archaeology session.
## Rules
- **Never compiled.** This directory sits OUTSIDE `../src/`, and PlatformIO's Library
Dependency Finder only recursively compiles an Arduino-format library's `src/` folder
(`../VENDORED.md` explains that mechanism at length). Do not move these under `src/`
and do not add them to any build.
- **Never edited.** They are upstream's bytes. Our behaviour lives in `nisps/ml/`. A
divergence from upstream is a decision recorded in `ALIGNMENT.md` or a task, not an
edit here.
- **Re-sync with the rest of the tree**, in the same step and to the same commit —
`../VENDORED.md` § "Re-syncing with upstream".

View file

@ -0,0 +1,85 @@
#ifndef __MEML_PICO_HPP__
#define __MEML_PICO_HPP__
#include "pico.h"
#include <memory>
#define AUDIO_FUNC(x) __not_in_flash_func(x) ///< Macro to make audio function load from mem
#define AUDIO_MEM __not_in_flash("audio") ///< Macro to make variable load from mem
//#define AUDIO_MEM_2 __not_in_flash("audio2")
#define APP_SRAM __not_in_flash("app")
#define ML_BUFFER_MEM __attribute__((section(".scratch_x")))
#define AUDIO_BUFFER_MEM __attribute__((section(".scratch_y")))
#define PERIODIC_DEBUG(COUNT, FUNC) \
static size_t ct=0; \
if (ct++ % COUNT == 0) { \
FUNC \
}
// Add these macros near other globals
#define MEMORY_BARRIER() __sync_synchronize()
#define WRITE_VOLATILE(var, val) do { MEMORY_BARRIER(); (var) = (val); MEMORY_BARRIER(); } while (0)
#define READ_VOLATILE(var) ({ MEMORY_BARRIER(); typeof(var) __temp = (var); MEMORY_BARRIER(); __temp; })
template<typename T>
inline void write_volatile_struct(volatile T& dest, const T& src) {
MEMORY_BARRIER();
memcpy((void*)&dest, &src, sizeof(T));
MEMORY_BARRIER();
}
#define WRITE_VOLATILE_STRUCT(var, val) write_volatile_struct((var), (val))
// Add this template function after write_volatile_struct:
template<typename T>
inline T read_volatile_struct(const volatile T& src) {
MEMORY_BARRIER();
T temp;
memcpy(&temp, (const void*)&src, sizeof(T));
MEMORY_BARRIER();
return temp;
}
#define READ_VOLATILE_STRUCT(var) read_volatile_struct(var)
//#define ALLOW_DEBUG
#ifdef ALLOW_DEBUG
#define DEBUG_PRINTF(...) Serial.printf(__VA_ARGS__); Serial.flush()
#define DEBUG_PRINT(...) Serial.print(__VA_ARGS__); Serial.flush()
#define DEBUG_PRINTLN(...) Serial.println(__VA_ARGS__); Serial.flush()
#else
#define DEBUG_PRINT(...)
#define DEBUG_PRINTLN(...)
#define DEBUG_PRINTF(...)
#endif
#define PERIODIC_RUN(code, freq_ms) \
{ \
static size_t lastUpdate = 0; \
size_t now = millis(); \
if (now - lastUpdate > (freq_ms)) { \
lastUpdate = now; \
code; \
} \
}
#define PERIODIC_RUN_US(code, freq_us) \
{ \
static size_t lastUpdate = 0; \
size_t now = micros(); \
if (now - lastUpdate > (freq_us)) { \
lastUpdate = now; \
code; \
} \
}
template<typename T>
std::shared_ptr<T> make_non_owning(T& obj) {
return std::shared_ptr<T>(&obj, [](T*){});
}
#endif // __MEML_PICO_HPP__

View file

@ -0,0 +1,33 @@
#include "AnalysisParams.hpp"
#include "pico/mutex.h"
#include "CoreMutex.h"
#include "Arduino.h"
#include "../PicoDefs.hpp"
static mutex_t mutex_0to1_;
static std::vector<float> params_mem_;
void AnalysisParamsSetup(size_t n_params) {
params_mem_.resize(n_params, -1.f);
mutex_init(&mutex_0to1_);
};
void AUDIO_FUNC(AnalysisParamsWrite)(std::vector<float> &params) {
{ // Acquire
CoreMutex acquire_1to0(&mutex_0to1_);
for (unsigned int n=0; n < params_mem_.size(); n++) {
if (n >= params.size()) {
DEBUG_PRINTLN("PANIK! Too many params for AnalysisParams");
}
params_mem_[n] = params[n];
}
} // Release
}
void AnalysisParamsRead(std::vector<float> &params) {
{ // Acquire
CoreMutex acquire_1to0(&mutex_0to1_);
params = params_mem_;
} // Release
}

View file

@ -0,0 +1,12 @@
#ifndef __ANALYSIS_PARAMS_HPP__
#define __ANALYSIS_PARAMS_HPP__
#include <stddef.h>
#include <vector>
void AnalysisParamsSetup(size_t n_params);
void AnalysisParamsWrite(std::vector<float> &params);
void AnalysisParamsRead(std::vector<float> &params);
#endif // __ANALYSIS_PARAMS_HPP__

View file

@ -0,0 +1,67 @@
#ifndef __AUDIO_APP_BASE_HPP__
#define __AUDIO_APP_BASE_HPP__
#include "AudioDriver.hpp"
#include "../interface/InterfaceBase.hpp"
#include <functional>
#include <memory>
template<size_t NPARAMS>
class AudioAppBase {
protected:
float sample_rate_;
std::shared_ptr<InterfaceBase> interface_;
static std::function<stereosample_t(stereosample_t)> callback_;
std::array<float, NPARAMS> paramsFromQueue;
public:
virtual AudioDriver::codec_config_t GetDriverConfig() const {
return {
.mic_input = false,
.line_level = 3,
.mic_gain_dB = 0,
.output_volume = 0.55f
};
}
AudioAppBase() = default;
virtual ~AudioAppBase() = default;
virtual stereosample_t Process(const stereosample_t x) {
return x;
}
static stereosample_t audioCallback(const stereosample_t x) {
return callback_(x);
}
virtual void Setup(float sample_rate, std::shared_ptr<InterfaceBase> interface) {
sample_rate_ = sample_rate;
interface_ = interface;
callback_ = [this](stereosample_t x) { return Process(x); };
AudioDriver::SetCallback(audioCallback);
}
virtual void ProcessParams(const std::array<float, NPARAMS>& params) {
// Default implementation does nothing
}
virtual void loop() {
if (!interface_) {
DEBUG_PRINTLN("AudioAppBase::loop - Error: Interface is null");
return; // Early return if interface is null
}
// std::vector<float> x;
if (interface_->ReceiveParamsFromQueue(paramsFromQueue.data())) {
ProcessParams(paramsFromQueue);
}
}
};
template<size_t NPARAMS>
std::function<stereosample_t(stereosample_t)> AudioAppBase<NPARAMS>::callback_ = nullptr;
#endif // __AUDIO_APP_BASE_HPP__

View file

@ -0,0 +1,48 @@
/* Audio Library for Teensy 3.X
* Copyright (c) 2014, Paul Stoffregen, paul@pjrc.com
*
* Development of this audio library was funded by PJRC.COM, LLC by sales of
* Teensy and Audio Adaptor boards. Please support PJRC's efforts to develop
* open source software by purchasing Teensy or other PJRC products.
*
* Permission is hereby granted, free of charge, to any person obtaining a copy
* of this software and associated documentation files (the "Software"), to deal
* in the Software without restriction, including without limitation the rights
* to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
* copies of the Software, and to permit persons to whom the Software is
* furnished to do so, subject to the following conditions:
*
* The above copyright notice, development funding notice, and this permission
* notice shall be included in all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
* THE SOFTWARE.
*/
#ifndef AudioControl_h_
#define AudioControl_h_
#include <stdint.h>
// A base class for all Codecs, DACs and ADCs, so at least the
// most basic functionality is consistent.
#define AUDIO_INPUT_LINEIN 0
#define AUDIO_INPUT_MIC 1
class AudioControl
{
public:
virtual bool enable(void) = 0;
virtual bool disable(void) = 0;
virtual bool volume(float volume) = 0; // volume 0.0 to 1.0
virtual bool inputLevel(float volume) = 0; // volume 0.0 to 1.0
virtual bool inputSelect(int n) = 0;
};
#endif

View file

@ -0,0 +1,298 @@
#include "AudioDriver.hpp"
#include <Arduino.h>
#include "Wire.h"
#include "control_sgtl5000.h"
#include "../synth/maximilian.h"
#include "hardware/dma.h"
#include "hardware/clocks.h"
#include "../PicoDefs.hpp"
#include "../utils/perf.hpp"
#define TEST_TONES 0
#define PASSTHROUGH 0
extern "C" {
size_t kSampleRate = 48000;
float kSampleRateRcpr = 1.0f / 48000.0f;
}
PERF_DECLARE(AUDIOLOOP);
AUDIO_MEM uint32_t AUDIOLOOP_MEAN=0;
int sampleRate = (int)kSampleRate;
constexpr int bitsPerSample = 32;
constexpr float amplitude = 1 << (bitsPerSample - 2); // amplitude of square wave = 1/2 of maximum
constexpr float neg_amplitude = -amplitude; // amplitude of square wave = 1/2 of maximum
constexpr float one_over_amplitude = 1.f / amplitude;
static int32_t AUDIO_MEM sample = amplitude; // current sample value
static AUDIO_MEM AudioControlSGTL5000 codecCtl;
audiocallback_fptr_t AUDIO_MEM audio_callback_ = nullptr;
audiocallback_block_fptr_t AUDIO_MEM audio_callback_block_ = nullptr;
//define AUDIO_BUFFER_MEM externally
static AUDIO_BUFFER_MEM float input_buffer[kNChannels][kBufferSize];
static AUDIO_BUFFER_MEM float output_buffer[kNChannels][kBufferSize];
static __attribute__((aligned(8))) AUDIO_MEM pio_i2s i2s;
volatile bool AUDIO_MEM dsp_overload;
float master_volume_ = 0;
#if TEST_TONES
maxiOsc osc, osc2;
float f1=20, f2=2000;
#endif // TEST_TONES
inline float __attribute__((always_inline)) _scale_down(float x) {
// Convert from int32 range to [-1.0, 1.0]
return x * (1.0f / (float)(1LL << 31)); // Divide by 2^31 for proper scaling
}
inline float __attribute__((always_inline)) _scale_and_saturate(float x) {
// Convert from [-1.0, 1.0] back to int32 range with saturation
const float scaled = x * (float)(1LL << 31);
if (scaled > INT32_MAX) return INT32_MAX;
if (scaled < INT32_MIN) return INT32_MIN;
return scaled;
}
#if TEST_TONES
static inline __attribute__((always_inline)) void AUDIO_FUNC(process_test_tones)(
int32_t* output, size_t i) {
output[i*2] = osc.sinewave(f1) * sample;
output[(i*2) + 1] = osc2.sinewave(f2) * sample;
f1 *= 1.00001;
f2 *= 1.00001;
if (f1 > 15000) {
f1 = 20.0;
}
if (f2 > 15000) {
f2 = 20.0;
}
}
#endif
#if PASSTHROUGH
static inline __attribute__((always_inline)) void AUDIO_FUNC(process_passthrough)(
const int32_t* input, int32_t* output, size_t i) {
output[i*2] = input[i*2];
output[(i*2) + 1] = input[(i*2) + 1];
}
#endif
static inline __attribute__((always_inline)) void AUDIO_FUNC(process_normal)(
const int32_t* input, int32_t* output, size_t i,
const size_t indexL, const size_t indexR) {
stereosample_t y {
_scale_down(static_cast<float>(input[indexL])),
_scale_down(static_cast<float>(input[indexR]))
};
y = audio_callback_(y); // y should now be in [-1.0, 1.0] range
output[indexL] = static_cast<int32_t>(_scale_and_saturate(y.L * master_volume_));
output[indexR] = static_cast<int32_t>(_scale_and_saturate(y.R * master_volume_));
}
static void AUDIO_FUNC(process_audio)(const int32_t* input, int32_t* output, size_t num_frames) {
PERF_BEGIN(AUDIOLOOP);
if (audio_callback_block_ != nullptr) {
// Convert from interleaved int32_t to deinterleaved float.
// Channels are swapped here to correct a hardware layout issue: the physical
// L/R input sockets map to the opposite codec ADC channels. Swapping at this
// lowest level means every mode sees x.L/x.R matching the labelled sockets.
for (size_t i = 0; i < num_frames; i++) {
const size_t indexL = i << 1;
const size_t indexR = indexL + 1;
input_buffer[0][i] = _scale_down(static_cast<float>(input[indexR]));
input_buffer[1][i] = _scale_down(static_cast<float>(input[indexL]));
}
// Serial.println(input_buffer[0][0]);
// Serial.println(input[0]);
// Call block callback
audio_callback_block_(input_buffer, output_buffer, kNChannels, num_frames);
// Convert from deinterleaved float to interleaved int32_t
for (size_t i = 0; i < num_frames; i++) {
const size_t indexL = i << 1;
const size_t indexR = indexL + 1;
// TODO find way to perform only one for loop
output[indexL] = static_cast<int32_t>(_scale_and_saturate(output_buffer[0][i] * master_volume_));
output[indexR] = static_cast<int32_t>(_scale_and_saturate(output_buffer[1][i] * master_volume_));
}
} else {
for (size_t i = 0; i < num_frames; i++) {
const size_t indexL = i << 1;
const size_t indexR = indexL + 1;
#if TEST_TONES
process_test_tones(output, i);
#else
#if PASSTHROUGH
process_passthrough(input, output, i);
#else
process_normal(input, output, i, indexL, indexR);
#endif
#endif
}
}
PERF_END(AUDIOLOOP);
AUDIOLOOP_MEAN = PERF_GET_MEAN(AUDIOLOOP);
}
static void __isr dma_i2s_in_handler(void) {
/* We're double buffering using chained TCBs. By checking which buffer the
* DMA is currently reading from, we can identify which buffer it has just
* finished reading (the completion of which has triggered this interrupt).
*/
if (*(int32_t**)dma_hw->ch[i2s.dma_ch_in_ctrl].read_addr == i2s.input_buffer) {
// It is inputting to the second buffer so we can overwrite the first
process_audio(i2s.input_buffer, i2s.output_buffer, AUDIO_BUFFER_FRAMES);
} else {
// It is currently inputting the first buffer, so we write to the second
process_audio(&i2s.input_buffer[STEREO_BUFFER_SIZE], &i2s.output_buffer[STEREO_BUFFER_SIZE], AUDIO_BUFFER_FRAMES);
}
dma_hw->ints0 = 1u << i2s.dma_ch_in_data; // clear the IRQ
}
void __isr AudioDriver::i2sOutputCallback() {
// for(size_t i=0; i < kBufferSize; i++) {
// stereosample_t y { 0 };
// y = audio_callback_(y);
// stereosample_t y_scaled {
// _scale_and_saturate(y.L),
// _scale_and_saturate(y.R),
// };
// i2s.write32(static_cast<int32_t>(y_scaled.L), static_cast<int32_t>(y_scaled.R));
// }
// Timing end
// auto elapsed = micros() - ts;
// static constexpr float quantumLength = 1.f/
// ((static_cast<float>(kBufferSize)/static_cast<float>(kSampleRate))
// * 1000000.f);
// float dspload = elapsed * quantumLength;
// // Report DSP overload if needed
// static volatile bool dsp_overload = false;
// if (dspload > 0.95 and !dsp_overload) {
// dsp_overload = true;
// } else if (dspload < 0.9) {
// dsp_overload = false;
// }
}
bool AudioDriver::Setup(const codec_config_t &config) {
if (nullptr == audio_callback_) {
audio_callback_ = &silence_;
}
DEBUG_PRINTF("AUDIO- Setup - Audio callback address: 0x%x\n", audio_callback_);
dsp_overload = false;
master_volume_ = 0;
// Zero out float buffers
for (size_t ch = 0; ch < kNChannels; ch++) {
for (size_t i = 0; i < kBufferSize; i++) {
input_buffer[ch][i] = 0;
output_buffer[ch][i] = 0;
}
}
maxiSettings::setup(kSampleRate, 2, kBufferSize);
if (!Wire.setSDA(i2c_sgt5000Data) ||
!Wire.setSCL(i2c_sgt5000Clk)) {
DEBUG_PRINTLN("AUDIO- Failed to setup I2C with codec!");
}
// set_sys_clock_khz(132000*2, true);
// set_sys_clock_khz(129600, true);
DEBUG_PRINTF("System Clock: %lu\n", clock_get_hz(clk_sys));
size_t sys_clk_hz = clock_get_hz(clk_sys);
if (sys_clk_hz != AudioDriver::GetSysClockSpeed() * 1000) {
DEBUG_PRINTLN("Error: audio driver: system clock must be set externally (see ::GetDesiredClockSpeed)");
DEBUG_PRINTLN("After 'setup()) {', add: 'set_sys_clock_khz(AudioDriver::GetSysClockSpeed(), true);'");
}
i2s_config picoI2SConfig {
kSampleRate, // 48000,
256,
bitsPerSample, // 32,
i2s_pMCLK, // 10,
i2s_pDIN, // 6,
i2s_pDOUT, // 7,
i2s_pBCLK, // 8,
true
};
i2s_program_start_synched(pio0, &picoI2SConfig, dma_i2s_in_handler, &i2s);
setDACVolume(3.0f);
// init i2c
codecCtl.enable();
DEBUG_PRINTLN("AUDIO - Codec enabled");
DEBUG_PRINTF("config.output_volume = %f\n", config.output_volume);
codecCtl.volume(config.output_volume > 0.99 ? 0.99 : config.output_volume);
DEBUG_PRINTF("config.mic_input = %d\n", config.mic_input);
codecCtl.inputSelect(config.mic_input ? AUDIO_INPUT_MIC : AUDIO_INPUT_LINEIN);
DEBUG_PRINTF("config.line_level = %d\n", config.line_level);
codecCtl.lineInLevel(config.line_level);
DEBUG_PRINTF("config.mic_gain_dB = %d\n", config.mic_gain_dB);
if (config.mic_input) {
codecCtl.micGain(config.mic_gain_dB);
}
codecCtl.lineOutLevel(20);
return true;
}
bool AudioDriver::Setup() {
codec_config_t config;
config.mic_input = false;
config.line_level = 3;
config.mic_gain_dB = 0;
config.output_volume = 0.8;
return Setup(config);
}
stereosample_t AudioDriver::silence_(stereosample_t x) {
x.L = 0;
x.R = 0;
return x;
}
//breaking change
void AudioDriver::setDACVolume(float n) {
codecCtl.dacVolume(n);
}
void AudioDriver::SetSampleRate(size_t rate) {
kSampleRate = rate;
kSampleRateRcpr = 1.0f / (float)rate;
}

View file

@ -0,0 +1,132 @@
#ifndef __AUDIO_DRIVER_HPP__
#define __AUDIO_DRIVER_HPP__
#include <Arduino.h>
#include <stddef.h>
#include "../PicoDefs.hpp"
#include "i2s_pio/i2s.h"
extern "C" {
const size_t kBufferSize = AUDIO_BUFFER_FRAMES;
extern size_t kSampleRate;
extern float kSampleRateRcpr;
const size_t kNChannels = 2;
struct stereosample_t {
float L;
float R;
__force_inline stereosample_t operator+(const stereosample_t& other) const {
return {L + other.L, R + other.R};
}
__force_inline stereosample_t& operator+=(const stereosample_t& other) {
L += other.L;
R += other.R;
return *this;
}
__force_inline stereosample_t operator*(float scalar) const {
return {L * scalar, R * scalar};
}
__force_inline stereosample_t& operator*=(float scalar) {
L *= scalar;
R *= scalar;
return *this;
}
__force_inline stereosample_t operator-() const {
return {-L, -R};
}
__force_inline stereosample_t operator-(const stereosample_t& other) const {
return {L - other.L, R - other.R};
}
__force_inline stereosample_t& operator-=(const stereosample_t& other) {
L -= other.L;
R -= other.R;
return *this;
}
__force_inline float operator[](size_t index) const {
return index == 0 ? L : R;
}
};
using audiocallback_fptr_t = stereosample_t (*)(stereosample_t);
using audiocallback_block_fptr_t = void (*)(float[][kBufferSize], float[][kBufferSize], size_t, size_t);
}
extern audiocallback_fptr_t audio_callback_;
extern audiocallback_block_fptr_t audio_callback_block_;
extern uint32_t AUDIOLOOP_MEAN;
enum PinConfig_i2c {
i2c_sgt5000Data = 0,
i2c_sgt5000Clk = 1,
i2s_pDIN = 6,
i2s_pDOUT = 7,
i2s_pBCLK = 8,
i2s_pWS = 9,
i2s_pMCLK = 10,
};
extern volatile bool AUDIO_MEM dsp_overload;
extern float master_volume_;
class AudioDriver {
public:
typedef struct {
bool mic_input;
size_t line_level;
size_t mic_gain_dB;
float output_volume;
} codec_config_t;
static bool Setup();
static bool Setup(const codec_config_t& config);
static inline void SetCallback(audiocallback_fptr_t callback) {
audio_callback_ = callback;
DEBUG_PRINT("AUDIO_DRIVER - Callback address: ");
DEBUG_PRINTF("%p\n", audio_callback_);
}
static inline void SetBlockCallback(audiocallback_block_fptr_t callback) {
audio_callback_block_ = callback;
DEBUG_PRINT("AUDIO_DRIVER - Block Callback address: ");
DEBUG_PRINTF("%p\n", audio_callback_block_);
}
static inline void SetMasterVolume(float volume) {
if (volume > 1.0f) {
volume = 1.0f;
} else if (volume < 0) {
volume = 0;
}
master_volume_ = volume;
}
static void SetSampleRate(size_t rate);
static inline size_t GetSampleRate() { return kSampleRate; }
static size_t GetSysClockSpeed() {
if (kSampleRate == 48000) {
return 132000 * 2;
} else if (kSampleRate == 44100) {
return 135475 * 2;
} else if (kSampleRate == 32000) {
return 132000 * 2;
} else if (kSampleRate == 24000) {
return 132000 * 2;
} else {
panic("Unsupported sample rate for SGTL5000");
}
}
AudioDriver() = delete;
static void i2sOutputCallback(void);
static stereosample_t silence_(stereosample_t);
private:
static void setDACVolume(float n);
};
#endif // __AUDIO_DRIVER_HPP__

View file

@ -0,0 +1,71 @@
#pragma once
#include <array>
#include <cstddef>
#include <cstdint>
#include <vector>
#include <WString.h>
// FocusManager — selective parameter latching for the Focus system.
//
// Interaction model:
// selectedMask == 0 → all params live (default, no-op)
// selectedMask != 0 → a param is live if any of its groups is in selectedMask,
// otherwise it uses its latch buffer value.
//
// Latch snapshot: setFocus() only captures params transitioning live→latched,
// so the freeze point is the exact moment a group leaves the selection.
//
// Multi-group membership: assign (kGroupA | kGroupB) to a param — it stays
// live if either group is selected.
template<size_t NPARAMS, size_t NGROUPS>
class FocusManager {
public:
std::array<String, NGROUPS> groupNames = {};
std::array<uint32_t, NPARAMS> paramGroupMask = {};
FocusManager() {
latchBuffer.fill(0.f);
}
void setGroupName(size_t groupIdx, const String& name) {
if (groupIdx < NGROUPS) groupNames[groupIdx] = name;
}
void setParamGroups(const std::array<uint32_t, NPARAMS>& masks) {
paramGroupMask = masks;
}
// Change focus selection. Snapshots only params transitioning live→latched.
void setFocus(uint32_t newMask, const std::vector<float>& live) {
for (size_t i = 0; i < NPARAMS && i < live.size(); i++) {
const bool wasLive = (selectedMask == 0) || ((paramGroupMask[i] & selectedMask) != 0);
const bool willBeLive = (newMask == 0) || ((paramGroupMask[i] & newMask) != 0);
if (wasLive && !willBeLive) {
latchBuffer[i] = live[i];
}
}
selectedMask = newMask;
}
// Apply focus filter in place. No-op when selectedMask == 0.
void applyInPlace(std::vector<float>& params) const {
if (selectedMask == 0) return;
for (size_t i = 0; i < NPARAMS && i < params.size(); i++) {
if ((paramGroupMask[i] & selectedMask) == 0) {
params[i] = latchBuffer[i];
}
}
}
uint32_t getSelectedMask() const { return selectedMask; }
bool isGroupSelected(size_t groupIdx) const {
return (selectedMask & (1u << groupIdx)) != 0;
}
private:
std::array<float, NPARAMS> latchBuffer = {};
uint32_t selectedMask = 0;
};

File diff suppressed because it is too large Load diff

View file

@ -0,0 +1,145 @@
/* Audio Library for Teensy 3.X
* Copyright (c) 2014, Paul Stoffregen, paul@pjrc.com
*
* Development of this audio library was funded by PJRC.COM, LLC by sales of
* Teensy and Audio Adaptor boards. Please support PJRC's efforts to develop
* open source software by purchasing Teensy or other PJRC products.
*
* Permission is hereby granted, free of charge, to any person obtaining a copy
* of this software and associated documentation files (the "Software"), to deal
* in the Software without restriction, including without limitation the rights
* to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
* copies of the Software, and to permit persons to whom the Software is
* furnished to do so, subject to the following conditions:
*
* The above copyright notice, development funding notice, and this permission
* notice shall be included in all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
* THE SOFTWARE.
*/
#ifndef control_sgtl5000_h_
#define control_sgtl5000_h_
// #include <AudioStream.h> // github.com/PaulStoffregen/cores/blob/master/teensy4/AudioStream.h
#include "AudioControl.h"
// SGTL5000-specific defines for headphones
#define AUDIO_HEADPHONE_DAC 0
#define AUDIO_HEADPHONE_LINEIN 1
#define AUDIO_SAMPLE_RATE_EXACT 48000
class AudioControlSGTL5000 : public AudioControl
{
public:
AudioControlSGTL5000(void) : i2c_addr(0x0A) { }
void setAddress(uint8_t level);
bool enable(void);//For Teensy LC the SGTL acts as master, for all other Teensys as slave.
bool enable(const unsigned extMCLK, const uint32_t pllFreq = (4096.0l * AUDIO_SAMPLE_RATE_EXACT) ); //With extMCLK > 0, the SGTL acts as Master
bool disable(void) { return false; }
bool volume(float n) { return volumeInteger(n * 129 + 0.499f); }
bool inputLevel(float n) {return false;}
bool muteHeadphone(void) { return write(0x0024, ana_ctrl | (1<<4)); }
bool unmuteHeadphone(void) { return write(0x0024, ana_ctrl & ~(1<<4)); }
bool muteLineout(void) { return write(0x0024, ana_ctrl | (1<<8)); }
bool unmuteLineout(void) { return write(0x0024, ana_ctrl & ~(1<<8)); }
bool inputSelect(int n) {
if (n == AUDIO_INPUT_LINEIN) {
return write(0x0020, 0x055) // +7.5dB gain (1.3Vp-p full scale)
&& write(0x0024, ana_ctrl | (1<<2)); // enable linein
} else if (n == AUDIO_INPUT_MIC) {
return write(0x002A, 0x0173) // mic preamp gain = +40dB
&& write(0x0020, 0x088) // input gain +12dB (is this enough?)
&& write(0x0024, ana_ctrl & ~(1<<2)); // enable mic
} else {
return false;
}
}
bool headphoneSelect(int n) {
if (n == AUDIO_HEADPHONE_DAC) {
return write(0x0024, ana_ctrl | (1<<6)); // route DAC to headphones out
} else if (n == AUDIO_HEADPHONE_LINEIN) {
return write(0x0024, ana_ctrl & ~(1<<6)); // route linein to headphones out
} else {
return false;
}
}
bool volume(float left, float right);
bool micGain(unsigned int dB);
bool lineInLevel(uint8_t n) { return lineInLevel(n, n); }
bool lineInLevel(uint8_t left, uint8_t right);
unsigned short lineOutLevel(uint8_t n);
unsigned short lineOutLevel(uint8_t left, uint8_t right);
unsigned short dacVolume(float n);
unsigned short dacVolume(float left, float right);
bool dacVolumeRamp();
bool dacVolumeRampLinear();
bool dacVolumeRampDisable();
unsigned short adcHighPassFilterEnable(void);
unsigned short adcHighPassFilterFreeze(void);
unsigned short adcHighPassFilterDisable(void);
unsigned short audioPreProcessorEnable(void);
unsigned short audioPostProcessorEnable(void);
unsigned short audioProcessorDisable(void);
unsigned short eqFilterCount(uint8_t n);
unsigned short eqSelect(uint8_t n);
unsigned short eqBand(uint8_t bandNum, float n);
void eqBands(float bass, float mid_bass, float midrange, float mid_treble, float treble);
void eqBands(float bass, float treble);
void eqFilter(uint8_t filterNum, int *filterParameters);
unsigned short autoVolumeControl(uint8_t maxGain, uint8_t lbiResponse, uint8_t hardLimit, float threshold, float attack, float decay);
unsigned short autoVolumeEnable(void);
unsigned short autoVolumeDisable(void);
unsigned short enhanceBass(float lr_lev, float bass_lev);
unsigned short enhanceBass(float lr_lev, float bass_lev, uint8_t hpf_bypass, uint8_t cutoff);
unsigned short enhanceBassEnable(void);
unsigned short enhanceBassDisable(void);
unsigned short surroundSound(uint8_t width);
unsigned short surroundSound(uint8_t width, uint8_t select);
unsigned short surroundSoundEnable(void);
unsigned short surroundSoundDisable(void);
void killAutomation(void) { semi_automated=false; }
void setMasterMode(uint32_t freqMCLK_in);
protected:
bool muted;
bool volumeInteger(unsigned int n); // range: 0x00 to 0x80
uint16_t ana_ctrl;
uint8_t i2c_addr;
unsigned char calcVol(float n, unsigned char range);
unsigned int read(unsigned int reg);
bool write(unsigned int reg, unsigned int val);
unsigned int modify(unsigned int reg, unsigned int val, unsigned int iMask);
unsigned short dap_audio_eq_band(uint8_t bandNum, float n);
private:
bool semi_automated;
void automate(uint8_t dap, uint8_t eq);
void automate(uint8_t dap, uint8_t eq, uint8_t filterCount);
};
//For Filter Type: 0 = LPF, 1 = HPF, 2 = BPF, 3 = NOTCH, 4 = PeakingEQ, 5 = LowShelf, 6 = HighShelf
#define FILTER_LOPASS 0
#define FILTER_HIPASS 1
#define FILTER_BANDPASS 2
#define FILTER_NOTCH 3
#define FILTER_PARAEQ 4
#define FILTER_LOSHELF 5
#define FILTER_HISHELF 6
//For frequency adjustment
#define FLAT_FREQUENCY 0
#define PARAMETRIC_EQUALIZER 1
#define TONE_CONTROLS 2
#define GRAPHIC_EQUALIZER 3
void calcBiquad(uint8_t filtertype, float fC, float dB_Gain, float Q, uint32_t quantization_unit, uint32_t fS, int *coef);
#endif

View file

@ -0,0 +1,236 @@
/* i2s.c
*
* Author: Daniel Collins
* Date: 2022-02-25
*
* Copyright (c) 2022 Daniel Collins
*
* This file is part of rp2040_i2s_example.
*
* rp2040_i2s_example is free software: you can redistribute it and/or modify it under
* the terms of the GNU General Public License, version 3 as published by the
* Free Software Foundation.
*
* rp2040_i2s_example is distributed in the hope that it will
* be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
* See the GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License along with
* rp2040_i2s_example. If not, see <https://www.gnu.org/licenses/>.
*/
#include "i2s.h"
#include <math.h>
#include "hardware/clocks.h"
#include "hardware/dma.h"
#include "hardware/irq.h"
#include "i2s_pio_lib.h"
#include <Arduino.h>
// const i2s_config i2s_config_default = {48000, 256, 32, 10, 6, 7, 8, true};
static float pio_div(float freq, uint16_t* div, uint8_t* frac) {
float clk = (float)clock_get_hz(clk_sys);
float ratio = clk / freq;
float d;
float f = modff(ratio, &d);
*div = (uint16_t)d;
*frac = (uint8_t)(f * 256.0f);
// Use post-converted values to get actual freq after any rounding
float result = clk / ((float)*div + ((float)*frac / 256.0f));
return result;
}
static void calc_clocks(const i2s_config* config, pio_i2s_clocks* clocks) {
// Try to get a precise ratio between SCK and BCK regardless of how
// perfect the system_clock divides. First, see what sck we can actually get:
float sck_desired = (float)config->fs * (float)config->sck_mult * (float)i2s_sck_program_pio_mult;
float sck_attained = pio_div(sck_desired, &clocks->sck_d, &clocks->sck_f);
clocks->fs_attained = sck_attained / (float)config->sck_mult / (float)i2s_sck_program_pio_mult;
// Now that we have the closest fs our dividers will give us, we can
// re-calculate SCK and BCK as correct ratios of this adjusted fs:
float sck_hz = clocks->fs_attained * (float)config->sck_mult;
clocks->sck_pio_hz = pio_div(sck_hz * (float)i2s_sck_program_pio_mult, &clocks->sck_d, &clocks->sck_f);
float bck_hz = clocks->fs_attained * (float)config->bit_depth * 2.0f;
clocks->bck_pio_hz = pio_div(bck_hz * (float)i2s_out_master_program_pio_mult, &clocks->bck_d, &clocks->bck_f);
}
static bool validate_sck_bck_sync(pio_i2s_clocks* clocks) {
float ratio = clocks->sck_pio_hz / clocks->bck_pio_hz;
float actual_sck = clocks->sck_pio_hz / (float)i2s_sck_program_pio_mult;
float actual_bck = clocks->bck_pio_hz / (float)i2s_out_master_program_pio_mult;
DEBUG_PRINTF("Clock speed for SCK: %f (PIO %f Hz with divider %d.%d)\n", actual_sck, clocks->sck_pio_hz, clocks->sck_d, clocks->sck_f);
DEBUG_PRINTF("Clock speed for BCK: %f (PIO %f Hz with divider %d.%d)\n", actual_bck, clocks->bck_pio_hz, clocks->bck_d, clocks->bck_f);
DEBUG_PRINTF("Clock Ratio: %f\n", ratio);
float whole_ratio;
float fractional_ratio = modff(ratio, &whole_ratio);
return (fractional_ratio == 0.0f);
}
static void dma_double_buffer_init(pio_i2s* i2s, void (*dma_handler)(void)) {
// Set up DMA for PIO I2s - two channels, in and out
i2s->dma_ch_in_ctrl = dma_claim_unused_channel(true);
i2s->dma_ch_out_ctrl = dma_claim_unused_channel(true);
i2s->dma_ch_out_data = dma_claim_unused_channel(true);
i2s->dma_ch_in_data = dma_claim_unused_channel(true);
// Control blocks support double-buffering with interrupts on buffer change
i2s->in_ctrl_blocks[0] = i2s->input_buffer;
i2s->in_ctrl_blocks[1] = &i2s->input_buffer[STEREO_BUFFER_SIZE];
i2s->out_ctrl_blocks[0] = i2s->output_buffer;
i2s->out_ctrl_blocks[1] = &i2s->output_buffer[STEREO_BUFFER_SIZE];
// DMA I2S OUT control channel - wrap read address every 8 bytes (2 words)
// Transfer 1 word at a time, to the out channel read address and trigger.
dma_channel_config c = dma_channel_get_default_config(i2s->dma_ch_out_ctrl);
channel_config_set_read_increment(&c, true);
channel_config_set_write_increment(&c, false);
channel_config_set_ring(&c, false, 3);
channel_config_set_transfer_data_size(&c, DMA_SIZE_32);
dma_channel_configure(i2s->dma_ch_out_ctrl, &c, &dma_hw->ch[i2s->dma_ch_out_data].al3_read_addr_trig, i2s->out_ctrl_blocks, 1, false);
c = dma_channel_get_default_config(i2s->dma_ch_out_data);
channel_config_set_read_increment(&c, true);
channel_config_set_write_increment(&c, false);
channel_config_set_chain_to(&c, i2s->dma_ch_out_ctrl);
channel_config_set_dreq(&c, pio_get_dreq(i2s->pio, i2s->sm_dout, true));
dma_channel_configure(i2s->dma_ch_out_data,
&c,
&i2s->pio->txf[i2s->sm_dout], // Destination pointer
NULL, // Source pointer, will be set by ctrl channel
STEREO_BUFFER_SIZE, // Number of transfers
false // Start immediately
);
c = dma_channel_get_default_config(i2s->dma_ch_in_ctrl);
channel_config_set_read_increment(&c, true);
channel_config_set_write_increment(&c, false);
channel_config_set_ring(&c, false, 3);
channel_config_set_transfer_data_size(&c, DMA_SIZE_32);
dma_channel_configure(i2s->dma_ch_in_ctrl, &c, &dma_hw->ch[i2s->dma_ch_in_data].al2_write_addr_trig, i2s->in_ctrl_blocks, 1, false);
c = dma_channel_get_default_config(i2s->dma_ch_in_data);
channel_config_set_read_increment(&c, false);
channel_config_set_write_increment(&c, true);
channel_config_set_chain_to(&c, i2s->dma_ch_in_ctrl);
channel_config_set_dreq(&c, pio_get_dreq(i2s->pio, i2s->sm_din, false));
dma_channel_configure(i2s->dma_ch_in_data,
&c,
NULL, // Will be set by ctrl chan
&i2s->pio->rxf[i2s->sm_din], // Source pointer
STEREO_BUFFER_SIZE, // Number of transfers
false // Don't start yet
);
// Input channel triggers the DMA interrupt handler, hopefully these stay
// in perfect sync with the output.
dma_channel_set_irq0_enabled(i2s->dma_ch_in_data, true);
irq_set_exclusive_handler(DMA_IRQ_0, dma_handler);
irq_set_enabled(DMA_IRQ_0, true);
// Enable all the dma channels
dma_channel_start(i2s->dma_ch_out_ctrl); // This will trigger-start the out chan
dma_channel_start(i2s->dma_ch_in_ctrl); // This will trigger-start the in chan
}
/* Initializes an I2S block (of 3 state machines) on the designated PIO.
* NOTE! This does NOT START the PIO units. You must call i2s_program_start
* with the resulting i2s object!
*/
static void i2s_slave_program_init(PIO pio, const i2s_config* config, pio_i2s* i2s) {
DEBUG_PRINTLN("Slave init");
uint offset = 0;
i2s->pio = pio;
i2s->sm_mask = 0;
pio_i2s_clocks clocks;
calc_clocks(config, &clocks);
if (config->sck_enable) {
// SCK block
i2s->sm_sck = pio_claim_unused_sm(pio, true);
i2s->sm_mask |= (1u << i2s->sm_sck);
offset = pio_add_program(pio, &i2s_sck_program);
i2s_sck_program_init(pio, i2s->sm_sck, offset, config->sck_pin);
pio_sm_set_clkdiv_int_frac(pio, i2s->sm_sck, clocks.sck_d, clocks.sck_f);
}
// Bi-Di I2S block, clocked with SCK
i2s->sm_din = pio_claim_unused_sm(pio, true);
i2s->sm_dout = i2s->sm_din;
i2s->sm_mask |= (1u << i2s->sm_din);
offset = pio_add_program(pio, &i2s_bidi_slave_program);
i2s_bidi_slave_program_init(pio, i2s->sm_din, offset, config->dout_pin, config->din_pin);
pio_sm_set_clkdiv_int_frac(pio, i2s->sm_din, clocks.sck_d, clocks.sck_f);
}
/* Initializes an I2S block (of 3 state machines) on the designated PIO.
* NOTE! This does NOT START the PIO units. You must call i2s_program_start
* with the resulting i2s object!
*/
static void i2s_sync_program_init(PIO pio, const i2s_config* config, pio_i2s* i2s) {
uint offset = 0;
i2s->pio = pio;
i2s->sm_mask = 0;
pio_i2s_clocks clocks;
calc_clocks(config, &clocks);
if (config->sck_enable) {
// Check that SCK and BCK are in perfect whole ratio
if (!validate_sck_bck_sync(&clocks)) {
/* There are lots of possible causes for this, a few are:
* - You are running a system clock frequency that doesn't divide well at all into SCK or BCK
* - You are running a 24-bit I2S with a 256x SCK multiplier (RP2040 cannot support this)
* - You have mucked with the PIO ratios or done something silly.
*/
DEBUG_PRINTLN("SCK and BCK are not in sync.");
}
// SCK block
i2s->sm_sck = pio_claim_unused_sm(pio, true);
i2s->sm_mask |= (1u << i2s->sm_sck);
offset = pio_add_program(pio, &i2s_sck_program);
i2s_sck_program_init(pio, i2s->sm_sck, offset, config->sck_pin);
pio_sm_set_clkdiv_int_frac(pio, i2s->sm_sck, clocks.sck_d, clocks.sck_f);
}
// In block, clocked with SCK
i2s->sm_din = pio_claim_unused_sm(pio, true);
i2s->sm_mask |= (1u << i2s->sm_din);
offset = pio_add_program(pio, &i2s_in_slave_program);
i2s_in_slave_program_init(pio, i2s->sm_din, offset, config->din_pin);
pio_sm_set_clkdiv_int_frac(pio, i2s->sm_din, clocks.sck_d, clocks.sck_f);
// Out block, clocked with BCK
i2s->sm_dout = pio_claim_unused_sm(pio, true);
i2s->sm_mask |= (1u << i2s->sm_dout);
offset = pio_add_program(pio, &i2s_out_master_program);
i2s_out_master_program_init(pio, i2s->sm_dout, offset, config->bit_depth, config->dout_pin, config->clock_pin_base);
pio_sm_set_clkdiv_int_frac(pio, i2s->sm_dout, clocks.bck_d, clocks.bck_f);
}
void i2s_program_start_slaved(PIO pio, const i2s_config* config, void (*dma_handler)(void), pio_i2s* i2s) {
if (((uint32_t)i2s & 0x7) != 0) {
DEBUG_PRINTLN("pio_i2s argument must be 8-byte aligned!");
}
i2s_slave_program_init(pio, config, i2s);
dma_double_buffer_init(i2s, dma_handler);
pio_enable_sm_mask_in_sync(i2s->pio, i2s->sm_mask);
}
void i2s_program_start_synched(PIO pio, const i2s_config* config, void (*dma_handler)(void), pio_i2s* i2s) {
// DEBUG_PRINTLN("Start synched");
if (((uint32_t)i2s & 0x7) != 0) {
DEBUG_PRINTLN("pio_i2s argument must be 8-byte aligned!");
}
i2s_sync_program_init(pio, config, i2s);
dma_double_buffer_init(i2s, dma_handler);
pio_enable_sm_mask_in_sync(i2s->pio, i2s->sm_mask);
}

View file

@ -0,0 +1,82 @@
/* i2s.h
*
* Author: Daniel Collins
* Date: 2022-02-25
*
* Copyright (c) 2022 Daniel Collins
*
* This file is part of rp2040_i2s_example.
*
* rp2040_i2s_example is free software: you can redistribute it and/or modify it under
* the terms of the GNU General Public License, version 3 as published by the
* Free Software Foundation.
*
* rp2040_i2s_example is distributed in the hope that it will
* be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
* See the GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License along with
* rp2040_i2s_example. If not, see <https://www.gnu.org/licenses/>.
*/
#include <stdio.h>
#include "hardware/pio.h"
#include "../../PicoDefs.hpp"
#ifndef I2S_TEST_I2S_H
#define I2S_TEST_I2S_H
#define AUDIO_BUFFER_FRAMES 48
#define STEREO_BUFFER_SIZE AUDIO_BUFFER_FRAMES * 2 // roughly 1ms, 48 L + R words
typedef struct i2s_config {
uint32_t fs;
uint32_t sck_mult;
uint8_t bit_depth;
uint8_t sck_pin;
uint8_t dout_pin;
uint8_t din_pin;
uint8_t clock_pin_base;
bool sck_enable;
} i2s_config;
typedef struct pio_i2s_clocks {
// Clock computation results
float fs_attained;
float sck_pio_hz;
float bck_pio_hz;
// PIO divider ratios to obtain the computed clocks above
uint16_t sck_d;
uint8_t sck_f;
uint16_t bck_d;
uint8_t bck_f;
} pio_i2s_clocks;
// NOTE: Use __attribute__ ((aligned(8))) on this struct or the DMA wrap won't work!
typedef struct pio_i2s {
PIO pio;
uint8_t sm_mask;
uint8_t sm_sck;
uint8_t sm_dout;
uint8_t sm_din;
uint dma_ch_in_ctrl;
uint dma_ch_in_data;
uint dma_ch_out_ctrl;
uint dma_ch_out_data;
int32_t* in_ctrl_blocks[2]; // Control blocks MUST have 8-byte alignment.
int32_t* out_ctrl_blocks[2];
int32_t input_buffer[STEREO_BUFFER_SIZE * 2];
int32_t output_buffer[STEREO_BUFFER_SIZE * 2];
i2s_config config;
} pio_i2s;
// extern const i2s_config i2s_config_default;
const i2s_config i2s_config_default = {48000, 256, 32, 10, 6, 7, 8, true};
void i2s_program_start_slaved(PIO pio, const i2s_config* config, void (*dma_handler)(void), pio_i2s* i2s);
void i2s_program_start_synched(PIO pio, const i2s_config* config, void (*dma_handler)(void), pio_i2s* i2s);
#endif // I2S_TEST_I2S_H

View file

@ -0,0 +1,73 @@
// -------------------------------------------------- //
// This file is autogenerated by pioasm; do not edit! //
// -------------------------------------------------- //
#pragma once
/*
; I2S audio bidirectional (input/output) block, both in slave mode.
; Requires external BCK and LRCK, usually from the codec directly.
; This block provides both the output and input components together,
; so should not be used in combination with either of the other output or
; input units on the same I2S bus.
;
; Input pin order: DIN, BCK, LRCK
; Set JMP pin to LRCK.
;
; Clock synchronously with the system clock, or *at least* 4x the usual
; bit clock for a given fs (e.g. for 48kHz 24-bit, clock at at least
; (48000 * 24 * 2 (stereo)) * 4 = 9.216 MHz. Ideally 8x or more.
*/
#if !PICO_NO_HARDWARE
#include "hardware/pio.h"
#endif
// -------------- //
// i2s_bidi_slave //
// -------------- //
#define i2s_bidi_slave_wrap_target 0
#define i2s_bidi_slave_wrap 23
static const uint16_t i2s_bidi_slave_program_instructions[] = {
// .wrap_target
0x20a1, // 0: wait 1 pin, 1
0x4001, // 1: in pins, 1
0x8080, // 2: pull noblock
0x8000, // 3: push noblock
0x2021, // 4: wait 0 pin, 1
0x2022, // 5: wait 0 pin, 2
0x6001, // 6: out pins, 1
0x20a1, // 7: wait 1 pin, 1
0x4001, // 8: in pins, 1
0x2021, // 9: wait 0 pin, 1
0x6001, // 10: out pins, 1
0x00cd, // 11: jmp pin, 13
0x0007, // 12: jmp 7
0x20a1, // 13: wait 1 pin, 1
0x4001, // 14: in pins, 1
0x8080, // 15: pull noblock
0x8000, // 16: push noblock
0x2021, // 17: wait 0 pin, 1
0x6001, // 18: out pins, 1
0x20a1, // 19: wait 1 pin, 1
0x4001, // 20: in pins, 1
0x2021, // 21: wait 0 pin, 1
0x6001, // 22: out pins, 1
0x00d3, // 23: jmp pin, 19
// .wrap
};
#if !PICO_NO_HARDWARE
static const struct pio_program i2s_bidi_slave_program = {
.instructions = i2s_bidi_slave_program_instructions,
.length = 24,
.origin = -1,
};
static inline pio_sm_config i2s_bidi_slave_program_get_default_config(uint offset) {
pio_sm_config c = pio_get_default_sm_config();
sm_config_set_wrap(&c, offset + i2s_bidi_slave_wrap_target, offset + i2s_bidi_slave_wrap);
return c;
}
#endif

View file

@ -0,0 +1,65 @@
// -------------------------------------------------- //
// This file is autogenerated by pioasm; do not edit! //
// -------------------------------------------------- //
#pragma once
/*
; I2S Audio Input - Slave or Synchronous with Output Master
; Inputs must be sequential in order: DIN, BCK, LRCK
; Must run at same speed of SCK block, or at least 4x BCK.
;
; NOTE: Set JMP pin to LRCK pin.
; NOTE: The very first word read is potentially corrupt, since there is a
; chance to start in the middle of an L frame and only read part of it.
; Nevertheless, the frame order should be synchronized (first word is L,
; second is R, etc.
*/
#if !PICO_NO_HARDWARE
#include "hardware/pio.h"
#endif
// ------------ //
// i2s_in_slave //
// ------------ //
#define i2s_in_slave_wrap_target 0
#define i2s_in_slave_wrap 17
static const uint16_t i2s_in_slave_program_instructions[] = {
// .wrap_target
0x20a1, // 0: wait 1 pin, 1
0x4001, // 1: in pins, 1
0x8000, // 2: push noblock
0x2021, // 3: wait 0 pin, 1
0x2022, // 4: wait 0 pin, 2
0x20a1, // 5: wait 1 pin, 1
0x4001, // 6: in pins, 1
0x2021, // 7: wait 0 pin, 1
0x00ca, // 8: jmp pin, 10
0x0005, // 9: jmp 5
0x20a1, // 10: wait 1 pin, 1
0x4001, // 11: in pins, 1
0x8000, // 12: push noblock
0x2021, // 13: wait 0 pin, 1
0x20a1, // 14: wait 1 pin, 1
0x4001, // 15: in pins, 1
0x2021, // 16: wait 0 pin, 1
0x00ce, // 17: jmp pin, 14
// .wrap
};
#if !PICO_NO_HARDWARE
static const struct pio_program i2s_in_slave_program = {
.instructions = i2s_in_slave_program_instructions,
.length = 18,
.origin = -1,
};
static inline pio_sm_config i2s_in_slave_program_get_default_config(uint offset) {
pio_sm_config c = pio_get_default_sm_config();
sm_config_set_wrap(&c, offset + i2s_in_slave_wrap_target, offset + i2s_in_slave_wrap);
return c;
}
#endif

Some files were not shown because too many files have changed in this diff Show more