9ad1f78ddd
Roughly 20k lines were deleted from this repo in the last week and much of the
corpus still described the pre-deletion world in the present tense. Executes
the §8 table: archive the retired, reclassify the executed, prune the stale.
aimmersive-clone-spec -> _archive/ with a deprecated-by note
feedback-modes-port-spec -> plans/, kind: plan, status: executed
manifold-parity-features -> plans/, kind: plan, status: active
playground-2.0-rewrite -> status: superseded
engine-architecture 434 -> ~120 lines; seam + spine kept, rewritten
present-tense against the shipped engine/
MAIN.md six contradicted claims fixed; registry resynced
vcv-module.md pruned to the current 8->16 contract and made the
single .nisps format spec
vcv/NISPS-FORMAT.md DELETED — documented a v1 format that no longer loads
vcv/README.md, BUILDING.md rewritten to the real contract, menu, OSC table
inputs/backends/dock trio grounding sections marked historical, dead cites fixed
Two rows of the §8 table were themselves wrong, corrected here: the deleted
full-state sync lives in backends-spec.md §6.3, not vcv-module.md (which has no
§6.3), and codegen/README.md was already a MAP pointer with no port-solidjs
trigger left to remove.
Beyond the table — found by sweeping every backticked path in the changed docs
against `git ls-files`, which is how these should have been caught before:
manifold/ONBOARDING.md documented a UI that Phase 1 deleted, as if current:
SplitStage, ReadoutStrip, InputMini, BackendAdvanced, AltitudeNav, and a
shot.spec.ts that does not exist. The whole stage table was keyed on a `focus`
axis that no longer exists — selection is now sandwich > particles >
composite. This matters more than the rest: CLAUDE.md tells every agent to
read ONBOARDING.md first for Manifold work, so it was actively teaching a
fiction. Rewritten against ConsoleApp.tsx.
MAP.md claimed the input layer reduces axes to the engine arity with an
"even/odd blend". input-layer.ts says the opposite in its own header: one
dedicated slot per axis, 1:1, into a 32-input over-provisioned head, and
mean-blending was removed deliberately because it diluted every source.
AGENT-REFERENCE.md still promised TS emission "returns at P5" (landed),
per-mode dims "become schema-real at P5" (landed at P5.3), and pointed at
nisps::FixedBuffer (deleted).
Doc-right/code-suspect, filed rather than fixed: VCV computes derivedMean/Std/
Delta and cachedNovelty behind a live context-menu toggle that nothing reads;
vcv/plugin.json points at the MusicallyEmbodiedML org rather than this repo's
origin; and the module defaults to UDP 7001+id%64 while bridge.ts defaults to
9000, so out of the box they do not meet.
Firmware-build docs are deliberately untouched — the PlatformIO migration
lands next and rewrites all of them.
150 lines
13 KiB
Markdown
150 lines
13 KiB
Markdown
---
|
||
kind: spec
|
||
stability: stable
|
||
layer: cross-cutting
|
||
---
|
||
|
||
# MEMLNaut-NISPS Spec Corpus
|
||
|
||
**Neural Interactive Shaping of Parameter Spaces** — a research platform for interactive ML control of audio. One C++20 codebase compiles to two targets: RP2350 firmware for the MEMLNaut hardware, and WASM in the Manifold React browser app running the same engines + ML through an AudioWorklet. (The SolidJS playground was retired 2026-07-13, P1 of plans/one-core-engine-refactor.md; archived on `archive/playground-solidjs`.)
|
||
|
||
---
|
||
|
||
## 0. Spec Corpus Ontology
|
||
|
||
This corpus distinguishes four document kinds by temporal stance and mutability:
|
||
|
||
- **`kind: spec`** (flat in `docs/specs/`) — timeless present-tense contracts. **Wins over code by intent.** Drift-lintable; updated when the spec itself changes or code breaks it.
|
||
- **`kind: plan`** (in `docs/specs/plans/`) — future-tense, finite prescriptions carrying `status: active | executed | superseded`. Burn down as work lands; NEVER citable as authority for shipped behaviour (executed items migrate to `MAP.md` per the repo's doc-sync rule).
|
||
- **`kind: finding`** (in `docs/specs/recon/`) — past-tense, dated, **IMMUTABLE**. Never edited, only superseded; exempt from drift lint. Research artifacts capturing ground truth at a specific moment.
|
||
- **ADRs** (in `docs/adr/`) — decision rationale ("why we chose X"). Outside the spec root; used as backlinks from specs that encode those decisions.
|
||
|
||
Archived documents live in `docs/specs/_archive/` with `deprecated-by`/`superseded-by` front matter.
|
||
|
||
---
|
||
|
||
## Source Files (Architectural Roots)
|
||
|
||
The codebase's most architecturally significant areas referenced by this corpus:
|
||
|
||
| Layer | Path | Purpose |
|
||
|-------|------|---------|
|
||
| **Shared C++20** | `nisps/core/`, `nisps/ml/`, `nisps/dsp/`, `nisps/engines/`, `nisps/modes/`, `nisps/pipeline/` | One audio+ML library compiling to both firmware and WASM |
|
||
| **Schema/Codegen** | `schemas/`, `codegen/` | Parameter contracts + code generation (C++ headers, TS types) |
|
||
| **Firmware** | `firmware/MEMLNaut-NISPS/glue/` | Hardware bindings (audio, peripherals, MIDI, settings) |
|
||
| **Browser (Manifold)** | `manifold/src/engine/`, `manifold/src/console/` | Headless TS engine + AudioWorklet + the convertible React Console |
|
||
|
||
---
|
||
|
||
## 1. Frame
|
||
|
||
**What it is.** MEMLNaut-NISPS is a single C++20 codebase that compiles once to two distinct targets:
|
||
|
||
1. **Firmware** — runs on RP2350 hardware (MEMLNaut instrument). Real-time audio engines + interactive ML (4-layer MLP, SGD + RL feedback) with deterministic RNG + dual-core orchestration (audio on core 1, control on core 0).
|
||
2. **Browser (WASM)** — runs in the React Manifold front-end via AudioWorklet. The **same eight `nisps/` engines** as firmware — neither target is a superset. (The browser-only C15 synth lives only on the retired-playground archive branch; browser audio-*input* — mic — is not wired, so the analysis-driven modes are hardware-only today.)
|
||
|
||
**The unifying constraint**: one source tree, one ML architecture, cross-platform parity (native ↔ WASM within 1e-5 numerical tolerance). Parameter shapes are JSON schemas with codegen producing both C++ headers and TypeScript types.
|
||
|
||
**Performance contract** (applies to `nisps/`, inert in WASM but kept for consistency):
|
||
|
||
- No heap in audio/inference paths; use `std::array` (the one deliberate exception: `nisps/ml/dynamic_storage.hpp`, browser/VCV-only, `#error`s on RP2350 builds, sole lint allowlist entry).
|
||
- Float literals have the `.f` suffix; non-trivial float constants are `static const`.
|
||
- No virtual dispatch in the audio path (C++20 concepts, not interfaces).
|
||
- Deterministic per-instance RNG; all random state is local and seeded at construction.
|
||
- Hot-path attributes from `nisps/core/perf.hpp`: `NISPS_HOT` / `NISPS_FORCE_INLINE` (the former SRAM-section macro regime was deleted 2026-07 as dead).
|
||
|
||
---
|
||
|
||
## 2. Cross-Cutting Contracts
|
||
|
||
### Parity discipline
|
||
|
||
**Native ↔ WASM bit-equivalence within 1e-5.** The C++ core and its WASM compilation are asserted to produce identical outputs given the same input + same seed. See `tests/cpp/parity_check.cpp` (native test) + `scripts/parity-check.sh` (verify runner).
|
||
|
||
### Schema/codegen contract
|
||
|
||
Each mode has a `schemas/modes/<mode>.json` describing parameters (name, label, range, default, curve, group), ML config (input/output sizes, hidden layers), voice spaces (names; bodies are inline lambdas in the engine), and UI config. The meta-schema at `schemas/schema.json` validates these. Codegen (`bun run codegen/generate.ts`) is idempotent and produces **both language outputs**:
|
||
|
||
- `nisps/modes/generated/<mode>_schema.hpp` — `constexpr` C++ data under `nisps::modes::generated` (+ `nisps/ml/generated/ml_defaults.hpp` from `schemas/ml_defaults.json`).
|
||
- `manifold/src/modes/generated/<mode>_schema.ts` (+ `types.ts`, `index.ts`) — the source of truth for the Manifold `MF_MODES` catalogue.
|
||
|
||
A second generator, `codegen/generate-midi-devices.ts`, emits `nisps/midi/generated/midi_devices.hpp` + `manifold/src/midi-devices/generated/` from `schemas/midi_devices/`.
|
||
|
||
Schema changes ship with the regenerated C++ **and** TypeScript in the same commit; CI re-runs both generators and fails on any diff (plus the golden test).
|
||
|
||
### No-heap + deterministic RNG
|
||
|
||
These are inert in WASM but kept globally to avoid surprises when porting. Lint catches violations: `bash scripts/lint-cpp.sh` warns on missing `.f` suffix and fails on heap use under `nisps/` (allowlist: `dynamic_storage.hpp`), plus `Arduino.h` use outside firmware glue.
|
||
|
||
---
|
||
|
||
## Sub-specs (Behavioural + Binding)
|
||
|
||
| File | Stability | Layer | Summary |
|
||
|------|-----------|-------|---------|
|
||
| `backends-spec.md` | evolving | binding | Output backends (synth, particles, MIDI, OSC, CV/gate, VCV) unified behind one `OutputBackend` interface + manager. Largely built (`manifold/src/backends/`); carries a 2026-07 grounding note — the full-state OSC sync legs it designed were deleted. |
|
||
| `dock-spec.md` | evolving | behavioural | Console dock drawers + the per-output control row (off/fixed/live tri-state, mute, solo/arm, min/max/curve). Built with operator restructuring (top Mode selector + 5 drawers, 2 depths); grounding note maps spec→shipped. |
|
||
| `engine-architecture.md` | stable | cross-cutting | The surviving browser-engine contract: the headless `EngineApi` seam + the reactive-spine invariant, as realised in `manifold/src/engine/`. Trimmed 2026-07; superseded framing documented in its header. |
|
||
| `inputs-spec.md` | evolving | binding | Modular input layer (XY pad, MIDI input, gamepad single/double-stick) composing into an N-dim vector. Built as `manifold/src/inputs/` (runtime-reshaped net, exclusive-mode UI; mixing groundwork laid — operator decision §7.7 pending). |
|
||
| `slp-workshop-firmware.md` | evolving | binding | SLP-Workshop (Synth Library Portland workshop build). Part I (Jolt + OU explore adaptive-learning gestures) shipped & stable. Part II (output-mode evolution, gate sequences, Manifold config) planned/evolving. |
|
||
| `useq-cv-protocol.md` | stable | binding | uSEQ-CV wire protocol v2 (USB Web Serial ↔ uSEQ main module → CV/gate jacks + I2C expander). Single source of truth: `firmware/useq-celium/shared/protocol.h`, mirrored in `manifold/src/backends/useq-protocol.ts`. |
|
||
| `vcv-module.md` | evolving | binding | MEMLNaut VCV Rack 2 module: CV-to-CV mapper (8 in × 16 out, LED rings) on the shared `nisps/` core (P6). Also the single `.nisps`/patch format spec (v3 flat weights). Pruned 2026-07-21 to the current contract. |
|
||
|
||
**Archived** (in `_archive/`): `aimmersive-clone-spec.md` — the SolidJS clone spec for the deployed vanilla a-immersive app; the clone was never built and its target framework was retired. Kept as a behavioural reference for the still-deployed vanilla app.
|
||
|
||
---
|
||
|
||
## Plans (Finite, Burn-Down)
|
||
|
||
| File | Status | Summary |
|
||
|------|--------|---------|
|
||
| `plans/BUILD-PLAN.md` | executed | Manifold build resume anchor (2026-06-27). The app shipped to `/next/`; survivors (non-snap-node e2e invocation, naming/copy decisions) noted in its header; deploy is now the CI-gated webhook. |
|
||
| `plans/feedback-modes-port-spec.md` | executed | The "Down Action" negative-feedback port into `nisps/`. Landed, then evolved past it — `docs/adr/rl-feedback-design.md` + `nisps/ml/feedback.hpp` are the surviving truth (the ADR explicitly supersedes its §2.5/§7 AVOID decision). |
|
||
| `plans/manifold-parity-features-spec.md` | active | Five playground features for Manifold. Jolt + OU-explore: executed by other means (one-core P3 core bindings). Session presets, pins, control surface: still the live prescription, feeding the curated/advanced split (simplification-plan §6.5c). |
|
||
| `plans/playground-2.0-rewrite-plan.md` | superseded | SolidJS clean-room rewrite plan (2026-06-17). Its target (the playground) was retired; many of its ideas shipped in Manifold instead. Still cited as design source by the parity-features plan. |
|
||
| `plans/one-core-engine-refactor.md` | active | Firmware+Manifold core reunification (2026-07-13). **All six phases landed on main by 2026-07-18**; stays `active` only for the two hardware-verification chokepoints (its §6) before flipping to executed. |
|
||
| `plans/simplification-plan.md` | active | Phased burn-down of the 2026-07-21 simplification audit. Phases 0 (CI/verification), 2 (behaviour bugs) and 3 (truth consolidation, minus S26) burned down 2026-07-21; Phase 1 deletions largely landed; PlatformIO (§5) in flight; §7 lists remaining operator decisions. |
|
||
|
||
---
|
||
|
||
## Findings (Immutable, Dated Research)
|
||
|
||
Exempt from drift lint.
|
||
|
||
| File | Date | Summary |
|
||
|------|------|---------|
|
||
| `recon/findings-design-and-manifold.md` | 2026-06-27 | Build-oriented brief synthesizing five redesign docs + Manifold token export + `ConsoleApp.jsx`. VERIFIED from source. |
|
||
| `recon/findings-engine-surface.md` | 2026-06-27 | Engine surface audit (deployed JS, WASM, C++ core). Uncovers fixed-2-input WASM gap, loss-history plumbing gap, C15 placeholder. |
|
||
| `recon/findings-feedback-behaviour.md` | 2026-06-27 | Current RL feedback behaviour audit. Deployed JS + C++ core use undirected Gaussian noise (not geometric push-away); firmware InterfaceRL implements the true geometric mode. |
|
||
| `recon/midi-gamepad-inputs-worklog.md` | 2026-06-27 | Work log of `feat/midi-inputs` branch. Modular input sources (MIDI, gamepad), 32-input WASM foundation, gamepad→verdict wiring. Exclusive picker (not mixing) shipped; mixing engine groundwork done but UI deferred. |
|
||
| `recon/upstream-firmware-survey.md` | 2026-06-27 | Git archaeology of the MusicallyEmbodiedML ecosystem. origin/main (C++20/SolidJS rewrite) and upstream/main (old `.ino` firmware) forked at `6efbe9c` (2026-04-14); 49 commits upstream not in origin are ports, not merges. |
|
||
| `recon/playground-2026.md` | 2026-04-12 | Design intent snapshot of an unfinished playground UI redesign. Reference-only for SolidJS rewrite; do NOT merge into vanilla playground (those files moved on independently). |
|
||
| `recon/simplification-audit-2026-07.md` | 2026-07-21 | Full-repo smell/bloat/spec audit vs the five-bullet one-core vision (66-agent workflow, adversarially verified). 113 findings: CI red since 2026-07-13, memllib pin unpushed, ungated deploys, mode layer unshared, no curated/advanced split, dead-mass inventory. Mitigations in `plans/simplification-plan.md`. |
|
||
| `recon/memllib-usage-inventory.md` | 2026-07-21 | Per-file inventory of the memllib surface the firmware actually compiles — the prerequisite for the §7.5 ownership decision. Result: all of memllib bar `examples/` is load-bearing (~1.8 MB / 84 files); no small subset to vendor. |
|
||
|
||
---
|
||
|
||
## ADRs (Decision Rationale)
|
||
|
||
| File | Decision | Audience |
|
||
|------|----------|----------|
|
||
| `adr/rl-feedback-design.md` | Rationale for the Explore-and-Place + Geometric-Dislike feedback model choices | Implementation team |
|
||
|
||
---
|
||
|
||
## Cross-References
|
||
|
||
- **`MAP.md`** — the neutral inventory of the codebase as it stands (ground truth for what exists).
|
||
- **`CLAUDE.md`** — agent contract (gates, hard constraints); `docs/AGENT-REFERENCE.md` — the long-form reference.
|
||
- **`ALIGNMENT.md`** — strategic gaps + open mission questions (dated, opinionated gap diagnosis).
|
||
- **`manifold/ONBOARDING.md`** — agent-oriented breakdown of the Manifold React app (run/build/deploy/test, UI/engine-spine/WASM layering, convertible Stages, Dock+drawers, gotchas).
|
||
- **`docs/redesign/manifold-export/`** — design-token and UI-kit asset export from Manifold design project (not part of this corpus; static asset reference, not migrated).
|
||
|
||
---
|
||
|
||
## Open / Deferred
|
||
|
||
- **Index generation (`.index.json`)**: intentionally not generated yet. No consumer defined; add when crawler/search infra lands.
|
||
- **Per-spec §-numbering normalisation**: deferred to a future maintainer pass. Current state mixes flat (§1.1, §1.2…) and three-level nesting (§6.x.y).
|
||
- **ADR corpus**: currently minimal (only RL feedback design). Strategic decisions captured in spec Open/Deferred sections; promotion to dedicated ADRs is a future pass.
|
||
- **Spec-touched validator**: git-aware lint rule that flags when a code diff touches spec-cited files without also touching the spec. Useful as a pre-commit hook or PR check; infrastructure TBD.
|