164 lines
12 KiB
Markdown
164 lines
12 KiB
Markdown
---
|
||
kind: finding
|
||
date: 2026-06-27
|
||
immutable: true
|
||
---
|
||
|
||
# Findings — Current RL / Feedback Behaviour (Phase-1 Audit)
|
||
|
||
*Read-only audit, 2026-06-27. Citations are `file:line`. "VERIFIED" = read in source; "INFER" = deduced.
|
||
Produced by the Phase-1 research fan-out (workflow `wf_db146f21`), persisted by the orchestrator because a
|
||
hook blocks subagent report-file writes. Adversarially spot-checked against source.*
|
||
|
||
## Executive summary
|
||
|
||
1. The operator's mental model ("negative feedback takes the n-D opposite vector and pushes weights the other
|
||
way — geometric push-away") describes the **firmware `InterfaceRL`** path, NOT what is deployed in the
|
||
browser/aimmersive build.
|
||
2. The **deployed JS engine** and the **`nisps/` C++ core** both implement thumbs-down as
|
||
`move_weights(speed, spread, pinMask)` = per-weight decay (`w *= 1-0.1·spread`) then additive Gaussian
|
||
noise (`stddev = speed · layer_scale`). There is **no opposite vector, no centroid, no geometric direction**
|
||
in deployed/core code (VERIFIED: grep for `centroid|geometric|meanPositive|neg_action` returns nothing in
|
||
`deployments/meml-aimmersive/js/`).
|
||
3. So "push the weights the other way" is, in the deployed system, actually **"jitter the whole network with
|
||
random Gaussian noise"** — undirected diffusion, not directed avoidance.
|
||
4. The true geometric push-away (move output away from k-NN centroid of liked examples) exists only in the
|
||
firmware submodule (`memllib` InterfaceRL; "geo push" commits) and is documented in
|
||
`docs/specs/feedback-modes-port-spec.md` §1.1. **This is the algorithm Mode 1 must match** (workstream B).
|
||
5. **Why often inaudible (deployed AVOID):** noise added across all 4 layers of `MLP<2,10,14,18,126>`,
|
||
per-layer scaled `(1-spread)+spread/√fan_in`, capped at `noiseCap=0.12`, spread default 0.6 → tiny
|
||
per-step perturbation that the sigmoid output + output smoothing/slew pipeline absorb. One press moves the
|
||
heard 126-vector very little, in a random direction uncorrelated with what was disliked.
|
||
6. **Three FEEDBACK_MODEs already scaffolded** in untracked working files `nisps/ml/feedback.hpp` +
|
||
`tests/cpp/test_mlp_feedback.cpp` (`FeedbackController<MLP_T>`): `Avoid`, `RandomiseOutputs`,
|
||
`RandomiseMlp`. Not yet wired into bindings/TS/UI; no CMake registration.
|
||
7. `RandomiseMlp` = the operator's Mode 2 as an audition: snapshot weights → `draw_weights(spread)` → user
|
||
moves joystick to hear → up commits a +1 example, down/switch restores the snapshot. **This is the seed of
|
||
Mode 2** but the updated spec is stricter (scratchpad NEVER trained; explicit place-at-location; warm-start
|
||
real model to interpolate anchors).
|
||
8. `RandomiseOutputs` = bypass the MLP entirely, hold a static random output vector (focus-aware per-dim),
|
||
re-roll on each down; up commits the held vector as a +1 example.
|
||
9. The operator's "randomise → anchor → train" alternative maps onto `RandomiseMlp` (randomise) + thumbs-up
|
||
`LikeStore` (anchor) + `trainOnCurrent` (train). Missing in deployed build: the randomise-audition-then-anchor
|
||
loop; today thumbs-down only diffuses noise and never anchors.
|
||
10. The full port spec (component, WASM C API, TS FFI, ~22 ctest cases, parity stage) already exists at
|
||
`docs/specs/feedback-modes-port-spec.md` and matches the untracked `feedback.hpp`.
|
||
11. **Parity/perf constraints:** no heap / `std::array` only, per-instance deterministic `nisps::Rng`
|
||
(no libc `rand()`), no virtual dispatch, `.f` literals, fixed WASM arch `MLP<2,10,14,18,126>`,
|
||
native↔WASM parity within `1e-5`.
|
||
12. **Bottom line:** deployed negative-feedback is undirected Gaussian noise (not geometric push-away), which is
|
||
why it is weak/inaudible; the in-progress `FeedbackController` already implements the seed of the operator's
|
||
randomise-anchor-train idea but is unplumbed past the header.
|
||
|
||
## 1. Current algorithm, precisely, per layer
|
||
|
||
### 1.1 DEPLOYED (browser/aimmersive) — NOT geometric
|
||
Thumbs-down = `move_weights`. Pure-JS reference `deployments/meml-aimmersive/js/nisps/mlp.js:238-265`
|
||
(VERIFIED): per layer/node/weight `j`: `decay = 1-0.1*spread`; `layerScale = (1-spread)+(1/√fanIn)*spread`;
|
||
`node.weights[j] *= decay`; `accum = Σ_{n=0..2}(rand*2-1)`; `node.weights[j] += 3*accum*speed*layerScale`.
|
||
Output-layer nodes with `outputPinMask[ni]` skipped (`mlp.js:248`). WASM wrapper `nisps-wasm.js:389-406` →
|
||
`cwrap('nisps_mlp_move_weights_ex')`. IML `iml.js:166-170` forwards then re-infers — **no dislike-direction
|
||
logic, no example stored on dislike**.
|
||
|
||
**Key finding (VERIFIED, exhaustive grep):** `deployments/meml-aimmersive/js/` has zero matches for
|
||
`centroid|meanPositive|geometric|pushStep|dislikeMultiplier|neg_action|RANDOMISE|FEEDBACK_MODE|explore_active`.
|
||
Deployed negative feedback is purely undirected Gaussian weight diffusion. The disliked 126-vector is never read.
|
||
|
||
### 1.2 `nisps/` C++ core — same move_weights, no geometric push
|
||
`nisps/ml/rl.hpp:63-100` `move_weights_layer` (VERIFIED): `layer_scale = spread_scale(spread,fan_in) =
|
||
(1-spread)+spread/√fan_in` (`init.hpp:31-35`); `noise_stddev = speed*layer_scale`; `decay = 1-0.1*spread`
|
||
(`rl.hpp:49-51`). Bias: `biases[node] += rng.next_float_gaussian(noise_stddev)` unless pinned (`rl.hpp:83`).
|
||
Each weight: `w = w*decay + rng.next_float_gaussian(noise_stddev)` (`rl.hpp:92-97`); RNG advanced even on skip.
|
||
Pin mask only on `is_final_layer` (`rl.hpp:78-79`). `mlp.hpp:312-322` iterates 4 layers, only layer3
|
||
final+masked. `draw_weights` (`mlp.hpp:324-332`→`init.hpp:42-54`): `w = rng.next_float_signed()*spread_scale`,
|
||
biases zeroed. C API `nisps_ml_move_weights`/`nisps_ml_draw_weights` (`bindings.cpp:419-437`). TS
|
||
`wasm-iml.ts:495-508`. Handler `mode-runtime.ts:548-565`: builds pin mask, `moveWeights(cap,spread,pinMask)`,
|
||
`growNoise`, re-`setInput`. **No example stored on thumbs-down** (vs thumbs-up `mode-runtime.ts:523-546`).
|
||
|
||
### 1.3 TRUE geometric push-away (firmware only) — THE MODE-1 TARGET
|
||
Per `feedback-modes-port-spec.md:61-79` (sourced from firmware `InterfaceRL.cpp`), applied in `optimise()` not
|
||
at press time: `pushStep = clamp(|avgRewardNeg|,0.25,1.0)*0.5`; `dir[j] = neg_action[j] -
|
||
meanPositiveAction[j]` (away from k-NN(+) centroid); `target[j] = clamp(neg_action[j] +
|
||
(dir/||dir||)[j]*pushStep/(1+||dir||), 0, 1)`; then train toward `target`. Depends on firmware-only
|
||
`ReplayMemory`. Lives in `src/memllib` submodule (commits `2429bcc` "geo push", `d301cc7` "rlview and dislike";
|
||
upstream `SB2026`; memllib main `e291192`/`abe93ec`). **Workstream A must confirm the newest variant.**
|
||
|
||
## 2. FEEDBACK_MODEs implemented / scaffolded, and where
|
||
|
||
| Mode | Firmware memllib | nisps/ core (untracked) | Deployed JS | Playground TS |
|
||
|---|---|---|---|---|
|
||
| AVOID geometric push | VERIFIED (geo-push commits) | NOT ported | — | — |
|
||
| AVOID = move_weights | — | `feedback.hpp:103-105` | `mlp.js`/`nisps-wasm.js` | `mode-runtime.ts:548` |
|
||
| RANDOMISE_OUTPUTS | VERIFIED (spec §1.2) | `feedback.hpp:106-112,160-181` | — | — |
|
||
| RANDOMISE_MLP | VERIFIED (spec §1.3) | `feedback.hpp:113-119,183-189` | — | — |
|
||
|
||
**Untracked `nisps/ml/feedback.hpp`** (214 lines, VERIFIED): `nisps::ml::FeedbackController<MLP_T>`.
|
||
`enum FeedbackMode {Avoid=0,RandomiseOutputs=1,RandomiseMlp=2}` (`:47-51`); `enum FeedbackAction
|
||
{None,AvoidPerturb,LikeStore,EnterExplore,Reroll,CommitStore,Cancel,Restore}` (`:56-65`). State all
|
||
`std::array` (no heap), own `Rng` seeded separately (`:203-210`). `on_down`: Avoid→move_weights;
|
||
RandOut→enter/roll; RandMlp→enter(snapshot+draw_weights)/cancel. `on_up`: exploring→restore+CommitStore else
|
||
LikeStore. `on_drag`: RandMlp+exploring→restore+Restore else LikeStore. `static_output`: bypass hook only when
|
||
RandomiseOutputs&&exploring. `roll_static_outputs`: focused→`rng_.next_float_uniform()`, unfocused frozen.
|
||
`set_mode` aborts active exploration first.
|
||
|
||
**Untracked `tests/cpp/test_mlp_feedback.cpp`** (437 lines, ~22 NISPS_TEST, VERIFIED):
|
||
`SmallMLP=MLP<2,4,4,4,6,8,32>`; covers all transitions, RNG determinism, focus edge cases, a golden RNG
|
||
stream. **Not registered in `nisps/CMakeLists.txt`** (spec §5.1/§7 lists as TODO).
|
||
|
||
**Not yet wired (gaps):** no `nisps_ml_feedback_*` in `bindings.cpp`; no `FeedbackController` on `MLHandle`;
|
||
no TS FFI in `types.ts`; no UI selector. Firmware parent branch `origin/feat/feedback-explore-modes` wires a
|
||
TR-8S FEEDBACK_MODE selector (`5c2d4d2`) and adds `docs/dislike_system_{analysis,design_space}.md` (4 meanings
|
||
of "dislike"; here/anywhere × point/radius scope matrix — VERIFIED).
|
||
|
||
## 3. Spread / noise / zoom machinery
|
||
- **spread∈[0,1]** (`exploration-store.ts:35,74`, default 0.6): feeds per-layer scale and decay. spread=0:
|
||
uniform weights, big noise, no decay (saturated); spread=1: Xavier, small noise, 10% decay (spread out).
|
||
- **noiseCap∈[floor,1]** (default 0.12): the `speed`/`cap` magnitude for thumbs-down and auto-explore.
|
||
- **growNoise/decayNoise** driven by pointer `pressureForce` (`mode-runtime.ts:491-499,544,562`).
|
||
- **Zoom** scales auto-explore intensity: `scaledIntensity = intensity*(0.3+0.7*zoom)` — an input-pipeline
|
||
gain, not output magnitude.
|
||
- **Auto-explore loop** (`mode-runtime.ts:444-482`): periodic `move_weights`+`growNoise`, default 2000ms, off
|
||
by default — same diffusion as thumbs-down.
|
||
- **Pin mask** `buildPinMask(...)`: freezes final-layer weights for pinned/overridden params — the only
|
||
directional gating today (protects dims, doesn't push them). *Relevant to the SOLO/arm design (B).*
|
||
|
||
## 4. Parity/perf constraints any core change must respect
|
||
No heap in `nisps/` (`std::array`/`FixedBuffer`; `feedback.hpp` obeys); deterministic per-instance `nisps::Rng`,
|
||
no libc `rand()`; no virtual dispatch; `.f` literals, no double promotion, `NISPS_*` mem attrs; fixed WASM arch
|
||
`MLP<2,10,14,18,126>` (snapshot = `weight_count()` floats, compile-time fixed); native↔WASM parity within
|
||
`1e-5`. RandomiseMlp snapshot/restore must round-trip `get_weights`/`set_weights` byte-identically;
|
||
RandomiseOutputs static vectors need same-seed/same-Rng; golden tests pin RNG streams.
|
||
|
||
## 5. WHY the effect is inaudible — the mechanism
|
||
1. **Undirected, not opposite.** Zero-mean Gaussian noise on every weight in all 4 layers; the disliked output
|
||
is never read. Expected output change ≈ 0; only variance moves — wobbles randomly, doesn't move *away*.
|
||
2. **Small magnitude, multiply-attenuated.** `speed=0.12, spread=0.6` → output layer `layer_scale≈0.54`,
|
||
`stddev≈0.065` per weight, squashed by a near-saturated sigmoid, then damped by smoothing→slew→freeze.
|
||
3. **Decay fights accumulation.** `w *= 0.94` each press pulls back toward the basin → equilibrium by design.
|
||
4. **Pinned/override dims frozen** on the final layer → never respond.
|
||
5. **No anchoring.** Thumbs-down stores no example and triggers no training, so there's no learning signal.
|
||
|
||
## 6. Operator's Mode 2 ("explore-and-place") vs what exists
|
||
Already half-built as `RandomiseMlp`:
|
||
- **Randomise** = `RandomiseMlp::on_down` enter → `snapshot + draw_weights(spread)`: whole net re-rolled →
|
||
audibly different; user auditions via joystick.
|
||
- **Anchor** = `on_up` → `restore_after_explore` + `CommitStore`: caller stores +1 example at
|
||
`(current_input, heard_output)`, original net restored.
|
||
- **Train** = existing `trainOnCurrent()` trains restored net toward the anchored example.
|
||
- **Discard** = `on_down` again → `Cancel`, or mode-switch `abort_explore` restores snapshot.
|
||
|
||
**Updated-spec deltas to implement (workstream B):** the scratchpad must be NEVER trained (only randomise +
|
||
undoable nudge); the place step must let the user CHOOSE the input location (press → pick location → release);
|
||
the real model is warm-started to interpolate ALL placed anchors. The existing `RandomiseMlp` is the closest
|
||
prototype but commits at the *current* input position, not a chosen one, and lacks the explicit nudge op.
|
||
|
||
### Key files
|
||
- Deployed: `deployments/meml-aimmersive/js/nisps/{mlp.js,nisps-wasm.js,iml.js}`
|
||
- Core RL: `nisps/ml/{rl.hpp,init.hpp,mlp.hpp}`
|
||
- Core feedback (untracked): `nisps/ml/feedback.hpp`, `tests/cpp/test_mlp_feedback.cpp`
|
||
- WASM API: `nisps/wasm/bindings.cpp:419-437`
|
||
- TS: `playground/src/ml/wasm-iml.ts:488-508`; runtime `playground/src/modes/mode-runtime.ts:444-565`
|
||
- Exploration knobs: `playground/src/stores/exploration-store.ts`
|
||
- Port spec: `docs/specs/feedback-modes-port-spec.md`
|
||
- Firmware geo-push: `src/memllib` (commits `2429bcc`, `d301cc7`; upstream `SB2026`; main `e291192`/`abe93ec`);
|
||
design docs on `origin/feat/feedback-explore-modes`: `docs/dislike_system_{analysis,design_space}.md`
|