docs: P2 complete — burn down plan, close ALIGNMENT defect #3 (fixed WASM arity)
- plan P2 marked landed (hardware timing spot-check deferred to the chokepoint-B session on physical MEMLNaut) - ALIGNMENT: defect #3 resolved by the runtime-shaped browser MLP - MAP/AGENT-REFERENCE/nisps-wasm-README: describe the honoured-dims create + reshape ABI; note per-mode dims become schema-real at P5 - fresh wasm artifacts from the gate run
This commit is contained in:
parent
7e4dc26f6e
commit
7e957457cd
5 changed files with 30 additions and 39 deletions
10
ALIGNMENT.md
10
ALIGNMENT.md
|
|
@ -26,14 +26,6 @@ The clean-slate rewrite (2026-04-29) consolidated everything into one C++20 code
|
||||||
|
|
||||||
**Rough cost.** Half a day. Add `nisps_ml_train_with_history` (or extend the existing call) returning a pointer to the loss array; copy on the JS side.
|
**Rough cost.** Half a day. Add `nisps_ml_train_with_history` (or extend the existing call) returning a pointer to the loss array; copy on the JS side.
|
||||||
|
|
||||||
### 3. WASM MLP architecture is fixed (2026-04-29)
|
|
||||||
|
|
||||||
**What.** WASM compiles `MLP<2, 10, 14, 18, 126>` only. Modes with smaller `output_size` use a slice; modes that want different hidden layer shapes (e.g. `[10, 10, 14]` for smaller modes) can't get them in the browser.
|
|
||||||
|
|
||||||
**Why it blocks the mission.** Per-mode ML architecture variation is one of our four "research dimensions". The fix isn't urgent for the current mode set (all schemas are within the universal arch's capacity), but the moment we want to experiment with bigger networks or different shapes, this hits.
|
|
||||||
|
|
||||||
**Rough cost.** Medium. Either compile multiple WASM modules (one per arch shape; load on demand) or move to a runtime-shaped MLP (loses some compile-time perf). Templates-vs-runtime is a research-vs-firmware-perf tradeoff worth a separate decision doc.
|
|
||||||
|
|
||||||
### 4. `NISPS_AUDIO_FUNC` host fallback is misshapen (2026-04-29)
|
### 4. `NISPS_AUDIO_FUNC` host fallback is misshapen (2026-04-29)
|
||||||
|
|
||||||
**What.** `nisps/core/perf.hpp` defines `NISPS_AUDIO_FUNC(decl) decl` for the host but the firmware path `__not_in_flash_func(name)` takes only a function name (it stringifies into a section attribute). The two forms don't match. Stream 6 (firmware glue) avoided the macro to dodge the inconsistency, but it's still a footgun.
|
**What.** `nisps/core/perf.hpp` defines `NISPS_AUDIO_FUNC(decl) decl` for the host but the firmware path `__not_in_flash_func(name)` takes only a function name (it stringifies into a section attribute). The two forms don't match. Stream 6 (firmware glue) avoided the macro to dodge the inconsistency, but it's still a footgun.
|
||||||
|
|
@ -84,6 +76,8 @@ The original a-immersive was mobile-first ("designed for touch / foldable phone
|
||||||
|
|
||||||
## Recently resolved (delete after a few weeks)
|
## Recently resolved (delete after a few weeks)
|
||||||
|
|
||||||
|
- 2026-07-14: Defect "WASM MLP architecture is fixed" resolved by one-core-engine P2: `nisps/ml/` is storage-policied (`MLPCore<Storage>`); the browser MLP is runtime-shaped (`DynamicStorage`), `nisps_ml_create` honours dims, `nisps_ml_reshape` warm-starts. Firmware keeps the zero-heap fixed template (`.text` +0.30%, within contract).
|
||||||
|
|
||||||
- 2026-04-29: Three-implementation ML duplication (firmware `memlp`, `nisps-core`, JS engine) collapsed to single `nisps/` C++ codebase.
|
- 2026-04-29: Three-implementation ML duplication (firmware `memlp`, `nisps-core`, JS engine) collapsed to single `nisps/` C++ codebase.
|
||||||
- 2026-04-29: Firmware mode forks (~280–400 lines duplicated across 8 modes) collapsed via `nisps/modes/base.hpp` CRTP scaffold; concrete modes are now ~50–130 lines.
|
- 2026-04-29: Firmware mode forks (~280–400 lines duplicated across 8 modes) collapsed via `nisps/modes/base.hpp` CRTP scaffold; concrete modes are now ~50–130 lines.
|
||||||
- 2026-04-29: meml-ues double-scaling MSE bug fixed in `nisps/ml/loss.hpp` + `mlp.hpp`.
|
- 2026-04-29: meml-ues double-scaling MSE bug fixed in `nisps/ml/loss.hpp` + `mlp.hpp`.
|
||||||
|
|
|
||||||
2
MAP.md
2
MAP.md
|
|
@ -143,7 +143,7 @@ the "BUILD DELTAS" block at the top of `docs/specs/vcv-module.md`). `src/MEMLNau
|
||||||
- C++ identifiers: `PascalCase` types, `snake_case` functions/variables, `kPascalCase` constexpr. JSON keys `snake_case`. TS types `PascalCase`, components `PascalCase.tsx`, modules `kebab-case.ts`.
|
- C++ identifiers: `PascalCase` types, `snake_case` functions/variables, `kPascalCase` constexpr. JSON keys `snake_case`. TS types `PascalCase`, components `PascalCase.tsx`, modules `kebab-case.ts`.
|
||||||
- `Curve` enum lives in `nisps/core/math.hpp` (lowercase: `linear/exp/log/square/sqrt/sigmoid/cubic`); generated mode headers re-export via `using Curve = ::nisps::Curve;`. TS mirror at `manifold/src/engine/curves.ts` with same names (moves into core at P4).
|
- `Curve` enum lives in `nisps/core/math.hpp` (lowercase: `linear/exp/log/square/sqrt/sigmoid/cubic`); generated mode headers re-export via `using Curve = ::nisps::Curve;`. TS mirror at `manifold/src/engine/curves.ts` with same names (moves into core at P4).
|
||||||
- Modes are TSX components composed of primitives; mode parameter contracts are JSON schemas with codegen → C++ types (TS codegen returns at P5). **No declarative JSON UI.**
|
- Modes are TSX components composed of primitives; mode parameter contracts are JSON schemas with codegen → C++ types (TS codegen returns at P5). **No declarative JSON UI.**
|
||||||
- WASM and firmware share the same C++; WASM is fixed at `MLP<32, 10, 14, 18, 126>` (`nisps_ml_create` ignores requested dims — see `plans/one-core-engine-refactor.md` P2) and modes use a slice of outputs based on schema's `output_size`.
|
- WASM and firmware share the same C++; the browser MLP is runtime-shaped (`MLPCore<DynamicStorage>`, since P2): `nisps_ml_create` honours `(input, output, hidden[3])` with non-positive/null args defaulting to `32→[10,14,18]→126`; `nisps_ml_reshape` warm-starts a new shape. Firmware keeps compile-time `MLP<...>` (zero heap). Modes currently still use a slice of the default 126 outputs (per-mode dims become schema-real at P5).
|
||||||
- Cross-platform parity: `scripts/parity-check.sh` enforces native vs WASM agreement within 1e-5.
|
- Cross-platform parity: `scripts/parity-check.sh` enforces native vs WASM agreement within 1e-5.
|
||||||
|
|
||||||
## Gotchas
|
## Gotchas
|
||||||
|
|
|
||||||
|
|
@ -114,12 +114,11 @@ Two WASM instances at runtime:
|
||||||
|
|
||||||
C API is in `nisps/wasm/bindings.cpp`. Build: `bash scripts/build-wasm.sh` (~94KB output to `manifold/public/`).
|
C API is in `nisps/wasm/bindings.cpp`. Build: `bash scripts/build-wasm.sh` (~94KB output to `manifold/public/`).
|
||||||
|
|
||||||
The WASM target is fixed at `MLP<32u, 10u, 14u, 18u, 126u>` (`nisps_ml_create` ignores requested dims — see `docs/specs/plans/one-core-engine-refactor.md` P2). Modes with smaller `input_size`/`output_size` use a slice.
|
The browser MLP is runtime-shaped since P2 (`MLPCore<DynamicStorage>`): `nisps_ml_create` honours `(input, output, hidden[3])`; non-positive/null args default to `32→[10,14,18]→126`. `nisps_ml_reshape` swaps in a new shape warm-started from the overlapping weights (examples + feedback state reset). Modes currently still slice the default 126 outputs; per-mode dims become schema-real at P5.
|
||||||
|
|
||||||
### Known limitations
|
### Known limitations
|
||||||
|
|
||||||
- Loss history not yet plumbed through C API; only the final loss of a training run reaches TS.
|
- Loss history not yet plumbed through C API; only the final loss of a training run reaches TS.
|
||||||
- Engine MLP architecture is fixed at compile time — runtime-shaped browser MLP arrives at P2 of `docs/specs/plans/one-core-engine-refactor.md`.
|
|
||||||
- Mic input through the worklet for XIASRI / SoundAnalysisMIDI is not wired in manifold.
|
- Mic input through the worklet for XIASRI / SoundAnalysisMIDI is not wired in manifold.
|
||||||
- C15 has no home on main (see `ALIGNMENT.md` defect #1).
|
- C15 has no home on main (see `ALIGNMENT.md` defect #1).
|
||||||
- The browser Jolt/OU controls in manifold reimplement the gesture math in TS (interim, ported from the retired playground) rather than calling the C++ `ml::Jolt`/`ml::OUNoise` through WASM. They drive weights via the existing `nisps_ml_get/set_weights` bindings — the P3 phase of the one-core-engine plan replaces them with `nisps_ml_jolt_press/release` + `nisps_ml_explore_intensity` bindings.
|
- The browser Jolt/OU controls in manifold reimplement the gesture math in TS (interim, ported from the retired playground) rather than calling the C++ `ml::Jolt`/`ml::OUNoise` through WASM. They drive weights via the existing `nisps_ml_get/set_weights` bindings — the P3 phase of the one-core-engine plan replaces them with `nisps_ml_jolt_press/release` + `nisps_ml_explore_intensity` bindings.
|
||||||
|
|
|
||||||
|
|
@ -99,7 +99,7 @@ Each phase ends green on its test gate and is independently landable. File phase
|
||||||
- **Gate met:** full `run-all-tests.sh` green with playground gone (ctest 4/4, parity PASS, lint clean,
|
- **Gate met:** full `run-all-tests.sh` green with playground gone (ctest 4/4, parity PASS, lint clean,
|
||||||
manifold 9 unit + 20 e2e).
|
manifold 9 unit + 20 e2e).
|
||||||
|
|
||||||
### P2 — Storage-policy split: templated hardware, dynamic browser (the structural centre, ≈1 wk)
|
### P2 — Storage-policy split: templated hardware, dynamic browser (the structural centre, ≈1 wk) — ✅ landed 2026-07-14 (hardware timing spot-check of chokepoint B pending physical MEMLNaut)
|
||||||
|
|
||||||
- ✅ (landed 2026-07-13) Refactor `nisps/ml/` so algorithms (forward, backprop/SGD, init, `move_weights`)
|
- ✅ (landed 2026-07-13) Refactor `nisps/ml/` so algorithms (forward, backprop/SGD, init, `move_weights`)
|
||||||
are written once against a storage policy (`mlp.hpp` `MLPCore<Storage>`; jolt/OU/feedback already operate
|
are written once against a storage policy (`mlp.hpp` `MLPCore<Storage>`; jolt/OU/feedback already operate
|
||||||
|
|
@ -118,11 +118,16 @@ Each phase ends green on its test gate and is independently landable. File phase
|
||||||
firmware/tests, `DynamicFeedbackStorage` for the browser). `nisps_ml_describe` now takes the handle
|
firmware/tests, `DynamicFeedbackStorage` for the browser). `nisps_ml_describe` now takes the handle
|
||||||
(null → default shape). Verified: reshape ABI smoke (dims honoured, overlap survives, invalid dims
|
(null → default shape). Verified: reshape ABI smoke (dims honoured, overlap survives, invalid dims
|
||||||
rejected), warm-start ctest (grow+shrink), parity PASS unchanged, firmware `.text` unchanged.
|
rejected), warm-start ctest (grow+shrink), parity PASS unchanged, firmware `.text` unchanged.
|
||||||
- Manifold drops input clamping/phantom-channel handling; XIASRI/sound-analysis multi-input modes become
|
- ✅ (landed 2026-07-14) Manifold reshape wiring: `WasmIML.reshape` + `EngineApi.reshape` (buffers/worker/
|
||||||
browser-viable.
|
spine follow the new dims; training-worker protocol carries `hidden`); reset-on-reshape confirm modal
|
||||||
- **Gate:** parity — fixed and dynamic storage produce bit-identical outputs for identical shapes/seeds
|
offered on genuine input-layout changes (default 32-in over-provisioned head untouched on load/decline);
|
||||||
(new ctest); `parity-check.sh` native↔WASM ≤1e-5 unchanged; firmware builds byte-comparable (chokepoint B:
|
stale even/odd-blend copy deleted. The mean-blend clamping itself was already gone (dedicated dims);
|
||||||
compile PAFSynth, compare `.text` size ±1%).
|
the `min(n, inputSize)` guard + zero-padding stay as the legitimate declined-reshape behaviour.
|
||||||
|
XIASRI/sound-analysis multi-input modes become browser-viable (schema-real dims arrive at P5).
|
||||||
|
- **Gate met:** fixed↔dynamic bit-identical ctest (+ warm-start grow/shrink ctest + reshape ABI smoke);
|
||||||
|
`parity-check.sh` native↔WASM PASS unchanged (2.4e-7); PAFSynth `.text` 122324→122692 (+0.30%, within
|
||||||
|
±1%); manifold 9 unit + 25 e2e. Hardware audio-callback timing check deferred to the chokepoint-B
|
||||||
|
hardware session (operator, physical MEMLNaut).
|
||||||
|
|
||||||
### P3 — Exploration + feedback fully in core (≈3–4 days)
|
### P3 — Exploration + feedback fully in core (≈3–4 days)
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -28,30 +28,23 @@ Both files are committed (so the browser apps work from a fresh clone
|
||||||
without a C++ toolchain). Re-run `build-wasm.sh` after changes to
|
without a C++ toolchain). Re-run `build-wasm.sh` after changes to
|
||||||
`nisps/{core,ml,engines,wasm}`.
|
`nisps/{core,ml,engines,wasm}`.
|
||||||
|
|
||||||
## Architecture limit (read this)
|
## Architecture (runtime-shaped since one-core-engine P2)
|
||||||
|
|
||||||
The MLP class template is parametrised on `(input_size, hidden1, hidden2,
|
The browser MLP is `MLPCore<DynamicStorage>` (`nisps/ml/dynamic_storage.hpp`):
|
||||||
hidden3, output_size)`. WASM cannot recompile templates at runtime, so
|
`nisps_ml_create(input, output, hidden[3], n, seed)` HONOURS its dimensions.
|
||||||
this build instantiates exactly ONE configuration:
|
The 4-layer topology (ReLU×3 + Sigmoid) is fixed; only the dimensions are
|
||||||
|
runtime, capped at 4096 per dim. Non-positive/null arguments fall back to the
|
||||||
|
historical defaults:
|
||||||
|
|
||||||
nisps::ml::MLP<32, 10, 14, 18, 126>
|
32 inputs → [10, 14, 18] hidden → 126 outputs
|
||||||
|
|
||||||
That serves the browser use case (up to 32 input axes → up to 126 synth
|
`nisps_ml_reshape(ml, in, out, hidden, n, spread)` constructs a new net at
|
||||||
parameters). `nisps_ml_create()` accepts caller-supplied dimensions for
|
the requested shape, warm-starts it by copying the overlapping weight region
|
||||||
forward compatibility but currently ignores them — see comment at the top
|
(`nisps/ml/warm_start.hpp`), and swaps it in. The C-side dataset and the
|
||||||
of `bindings.cpp`. The schemas in `schemas/modes/*.json` use up to
|
feedback controller state RESET on reshape (front-end shows a confirm modal).
|
||||||
`output_size=126`; modes whose output_size is < 126 simply ignore the
|
Heap is used only at create/reshape time, never per-call; the firmware target
|
||||||
trailing entries.
|
never compiles the dynamic storage at all (`#error` under
|
||||||
|
`NISPS_TARGET_EMBEDDED`).
|
||||||
To support additional architectures, either:
|
|
||||||
|
|
||||||
1. Compile multiple wasm modules (`nisps_small.wasm`,
|
|
||||||
`nisps_default.wasm`, …) and let the playground load the right one
|
|
||||||
based on the active mode.
|
|
||||||
2. Add a runtime-shape MLP variant to `nisps/ml` (heap allocation only at
|
|
||||||
`create()`; no impact on hot paths).
|
|
||||||
|
|
||||||
Both options are deferred to a future stream.
|
|
||||||
|
|
||||||
## C API surface
|
## C API surface
|
||||||
|
|
||||||
|
|
@ -59,7 +52,7 @@ See `bindings.cpp` for the full list. Summary:
|
||||||
|
|
||||||
| Group | Functions |
|
| Group | Functions |
|
||||||
|-----------|------------------------------------------------------------------------------|
|
|-----------|------------------------------------------------------------------------------|
|
||||||
| ML life | `nisps_ml_create`, `nisps_ml_destroy`, `nisps_ml_reset` |
|
| ML life | `nisps_ml_create`, `nisps_ml_destroy`, `nisps_ml_reset`, `nisps_ml_reshape` |
|
||||||
| ML I/O | `nisps_ml_set_input`, `nisps_ml_process`, `nisps_ml_outputs`, `nisps_ml_infer_batch` |
|
| ML I/O | `nisps_ml_set_input`, `nisps_ml_process`, `nisps_ml_outputs`, `nisps_ml_infer_batch` |
|
||||||
| Training | `nisps_ml_add_example`, `nisps_ml_train`, `nisps_ml_eval_loss`, `nisps_ml_clear_examples`, `nisps_ml_example_count` |
|
| Training | `nisps_ml_add_example`, `nisps_ml_train`, `nisps_ml_eval_loss`, `nisps_ml_clear_examples`, `nisps_ml_example_count` |
|
||||||
| Weights | `nisps_ml_weight_count`, `nisps_ml_get_weights`, `nisps_ml_set_weights`, `nisps_ml_draw_weights`, `nisps_ml_move_weights` |
|
| Weights | `nisps_ml_weight_count`, `nisps_ml_get_weights`, `nisps_ml_set_weights`, `nisps_ml_draw_weights`, `nisps_ml_move_weights` |
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue