feat(manifold): replay geometric dislikes over time
This commit is contained in:
monkey-w1n5t0n
parent
7190932191
commit
a1cd26ff68
24 changed files with 720 additions and 114 deletions
19
ALIGNMENT.md
19
ALIGNMENT.md
|
|
@ -72,12 +72,11 @@ one thumbs-up can move the mapping somewhere in the space by more than the entir
|
|||
range. Retention (how much of the previous teaching survives) is 0.38; at `iters=1` it
|
||||
is 0.80.
|
||||
|
||||
**Why it blocks the mission.** This is the other half of the feedback asymmetry, and
|
||||
RMSProp did NOT fix it — normalising the step size does not change the dose. Upstream
|
||||
keeps both directions on small repeated steps; we take one enormous positive step and
|
||||
one small negative one, so teaching feels like a lurch and correcting feels like
|
||||
nothing. The two numbers now differ by ~70x rather than ~2e6x, which is progress and
|
||||
still not a design.
|
||||
**Why it blocks the mission.** RMSProp did NOT fix the positive lurch — normalising the
|
||||
step size does not change its dose. The negative path now exposes upstream-style repeated
|
||||
small steps (rate/lifetime/LR are live controls), so its old fixed ~70x comparison is no
|
||||
longer current. Positive teaching remains one enormous blocking train, and the two doses
|
||||
still need a matched head-to-head rather than independent tuning.
|
||||
|
||||
**Rough cost.** Cheap to change, expensive to choose: the tuning space is now measurable
|
||||
(`ml_bench` U4 sweeps dose; U1 sweeps upstream's soft-target alpha, where alpha=1 is
|
||||
|
|
@ -102,7 +101,7 @@ Legacy a-immersive was mobile-first; Manifold is desktop-first. Defer until user
|
|||
- **EOC effects chain, ShapeSeq sequencer, modular engine (Phase E)** — legacy features consciously out of the v1 rewrite; revisit only if a mode wants them.
|
||||
- **Inputs multi-source composition** (2026-06-28, reaffirmed 2026-07-21) — mix-and-match pad+gamepad+MIDI is a recorded, unreversed decision; the UI currently enforces exclusive single-source and the composition machinery sits dormant *by design*. Schedule or keep dormant — but the inputs-spec must stop presenting composition as current behaviour (plan §8).
|
||||
- **Schema content is partially placeholder** (2026-07-21) — 20 anonymous "Param NN" slots across paf_synth/channel_strip/xiasri and copy-pasted ML defaults across all 9 modes. Name them during the first curated-preset pass per mode (plan §6.5c), or shrink `output_size` where the engine allows.
|
||||
- **Geometric-dislike deliberate divergences** (2026-07-14, one-core P3; reaffirmed 2026-07-25): (1) the degenerate-branch RNG draws from the controller's deterministic `nisps::Rng`, not libc `rand()` — native==WASM parity holds; (2) upstream's async shuffled two-LR `optimise()` is collapsed into one synchronous `dislike_geometric()` training only the pressed negative's target — behavioural, not bitwise, parity with firmware upstream, by design; (3) `RandomiseMlp` uses `draw_weights(spread)` rather than the old asymmetric ranges. All intentional. The 2026-07-25 re-base onto `e291192` took upstream's constants and deleted the taper but did NOT adopt (2)'s counterpart — upstream retrains a batch over ALL live negatives on every 200 Hz tick and holds each negative at full strength for a fixed `kDislikeLifetimeMs = 2500` instead of decaying it proportionally. Adopting that needs a per-tick call site and a millisecond clock inside `nisps/ml`, which the core does not have; it is a real interface decision, not a constant, and is tracked as its own task.
|
||||
- **Geometric-dislike deliberate divergences** (2026-07-25): (1) the degenerate-branch RNG draws from deterministic `nisps::Rng`, not libc `rand()`; (2) each live rejection computes its liked centroid at its own stored input, rather than upstream reinterpreting every old rejection around the cursor's current position; (3) live negatives are applied as deterministic per-item RMSProp steps rather than one shuffled `TrainBatch`; (4) a repeated nearby rejection refreshes its lifetime (upstream's current dedup path leaves the original timestamp untouched); (5) upstream's default removal of a nearby positive is not yet adopted because Manifold also has a separate positive MLP dataset that would keep pulling; (6) `RandomiseMlp` uses `draw_weights(spread)` rather than old asymmetric ranges. Native↔WASM parity covers the elapsed-time replay path.
|
||||
- **Manifold dock splits `state`/`muted`/`armed`** (2026-06-28) — deliberate divergence from the deployed conflated `frozen`↔`muted` model (dock-spec §3.3). `muted`-downstream and the `soloMode` gradient-mask variants remain UI-only; the C API exposes `set_focus` but no per-mode gradient masking yet. (The audit found `soloMode` behaviourally inert in the controller — plan L20 trims it until `train_masked` exists.)
|
||||
|
||||
## Recently resolved (delete after a few weeks)
|
||||
|
|
@ -116,8 +115,10 @@ Legacy a-immersive was mobile-first; Manifold is desktop-first. Defer until user
|
|||
mapping before any likes exist (`ml_bench` E1: 0 -> 2.3e-3). One dislike moves the
|
||||
mapping **5.3e-2**, up from 1.6e-2 after the RMSProp fix and 5.3e-5 before it — a
|
||||
~1000x change end to end, and now within ~4x of the legacy Diffuse design instead of
|
||||
~4100x (`ml_bench` A4). Not adopted: upstream's per-tick batch retraining and fixed
|
||||
2500 ms dislike lifetime — see "Deferred / accepted debt".
|
||||
~4100x (`ml_bench` A4). The follow-up now adopts upstream's repeated-all-negatives
|
||||
schedule and full-strength wall-clock lifetime through a deterministic elapsed-time
|
||||
core seam. Manifold defaults to 0.001 LR, 200 Hz and 2500 ms, exposes all three in the
|
||||
expanded Learning panel, and allows rate/lifetime zero as an explicit one-shot A/B.
|
||||
- 2026-07-25: **`InterfaceRL` is back in the tree (defect 6c).** Vendored verbatim from
|
||||
memllib `e291192` at `firmware/MEMLNaut-NISPS/lib/memllib/reference/` — outside `src/`,
|
||||
so PlatformIO never compiles it. Upstream drift in the feedback subsystem is a `diff`
|
||||
|
|
|
|||
6
MAP.md
6
MAP.md
|
|
@ -6,7 +6,7 @@ MEMLNaut-NISPS — Neural Interactive Shaping of Parameter Spaces. One C++20 cod
|
|||
|
||||
### `nisps/` — platform-agnostic C++20 library (the only ML/DSP/engine code)
|
||||
- `nisps/core/` — `perf.hpp` (hot-path/inlining attrs), `types.hpp`, `concepts.hpp` (`MLEngine`, `AudioEngine`, `Mode`), `ring_buffer.hpp` (SPSC lock-free cross-core channel, replaces pico/util/queue), `event_queue.hpp` (single-threaded in-engine event FIFO — deliberately NOT RingBuffer, which is an atomics-based cross-thread channel), `rng.hpp` (xoshiro256+ deterministic), `math.hpp` (fast_sigmoid, `Curve` enum + `apply_curve`).
|
||||
- `nisps/ml/` — the MLP core, written once against a storage policy (`mlp.hpp` `MLPCore<Storage>`): `storage.hpp` (`FixedStorage` — template-sized `std::array`, zero heap; `MLP<NIn,NH1,NH2,NH3,NOut>` alias preserves the classic compile-time surface) and `dynamic_storage.hpp` (`DynamicStorage` — runtime dims, single arena alloc at construction; `#error`s on RP2350 builds, sole lint heap-allowlist entry). Fixed↔dynamic bit-parity enforced by `tests/cpp/test_mlp_storage_parity.cpp`. Files: `mlp.hpp`, `activations.hpp`, `loss.hpp` (MSE, no double-scaling), `training.hpp` (**RMSProp** + grad clipping — `rmsprop_step()` is a line-for-line port of upstream memlp `Layer.h:239 ApplyAccumulatedGradients` @ `ea777502`; the per-weight running squared-gradient average is optimiser state held in the storage policies, NOT part of `weight_count()`/`get_weights()`, and `MLPCore::reset_optimizer_state()` clears it. `draw_weights()` deliberately does not, matching upstream `DrawWeights`), `init.hpp` (spread-aware uniform↔Xavier), `rl.hpp` (`move_weights` with output pin mask + per-layer scaling + weight decay), `jolt.hpp` (`Jolt` — held continuous weight-morph over the flat weight buffer + post-release LR ramp; ported from upstream InterfaceRL), `ou_noise.hpp` (`OUNoise<N>` — Ornstein-Uhlenbeck exploration walk on the output vector; ported from upstream InterfaceRL), `feedback.hpp` (`FeedbackControllerCore<FbStorage>` — the "Down Action" state machine: Avoid (geometric push-away default / Diffuse legacy) / RandomiseOutputs / RandomiseMlp / ExploreAndPlace; storage-policied like the MLP, own deterministic RNG, exposed via `nisps_ml_feedback_*` C API), `replay.hpp` (`ReplayView` — reward-tagged memory: dedup/deepen, k-NN positive centroid with deterministic tie-break, proportional decay+eviction), `geo_push.hpp` (push-away target computation, upstream InterfaceRL @ `e291192` — re-based from `0a541cc` on 2026-07-25: `kGeometricPushScale` 1.0, `kNegLRBase` 1.5, and NO `/(1+len)` taper, so distance from the liked centroid does not shrink a "no". Cold start is the same push in a random direction, not a separate branch), `warm_start.hpp` (overlapping-weights copy for reshape), `stats.hpp`. `generated/ml_defaults.hpp` is codegen output (do not edit): `nisps::ml::generated::kMlTrainDefaults`, the ONE learning-rate / max-iterations / min-error default shared by firmware, WASM and VCV (source `schemas/ml_defaults.json`); `MLPCore::set_train_config()` and `nisps_ml_set_train_config()` override it at runtime. It lives under `ml/` rather than `modes/generated/` because `nisps/ml` sits below `nisps/modes` — mlp.hpp must not include upward. Jolt + OU are inert by default and wired into `ModeBase`, so every mode exposes `jolt_press/jolt_release`, `jolt_lr_scale`, and `set_explore_intensity`.
|
||||
- `nisps/ml/` — the MLP core, written once against a storage policy (`mlp.hpp` `MLPCore<Storage>`): `storage.hpp` (`FixedStorage` — template-sized `std::array`, zero heap; `MLP<NIn,NH1,NH2,NH3,NOut>` alias preserves the classic compile-time surface) and `dynamic_storage.hpp` (`DynamicStorage` — runtime dims, single arena alloc at construction; `#error`s on RP2350 builds, sole lint heap-allowlist entry). Fixed↔dynamic bit-parity enforced by `tests/cpp/test_mlp_storage_parity.cpp`. Files: `mlp.hpp`, `activations.hpp`, `loss.hpp` (MSE, no double-scaling), `training.hpp` (**RMSProp** + grad clipping — `rmsprop_step()` is a line-for-line port of upstream memlp `Layer.h:239 ApplyAccumulatedGradients` @ `ea777502`; the per-weight running squared-gradient average is optimiser state held in the storage policies, NOT part of `weight_count()`/`get_weights()`, and `MLPCore::reset_optimizer_state()` clears it. `draw_weights()` deliberately does not, matching upstream `DrawWeights`), `init.hpp` (spread-aware uniform↔Xavier), `rl.hpp` (`move_weights` with output pin mask + per-layer scaling + weight decay), `jolt.hpp` (`Jolt` — held continuous weight-morph over the flat weight buffer + post-release LR ramp; ported from upstream InterfaceRL), `ou_noise.hpp` (`OUNoise<N>` — Ornstein-Uhlenbeck exploration walk on the output vector; ported from upstream InterfaceRL), `feedback.hpp` (`FeedbackControllerCore<FbStorage>` — the "Down Action" state machine: Avoid (geometric push-away default / Diffuse legacy) / RandomiseOutputs / RandomiseMlp / ExploreAndPlace; geometric replay is advanced deterministically by caller-supplied elapsed time with configurable rate/lifetime; storage-policied like the MLP, own deterministic RNG, exposed via `nisps_ml_feedback_*` C API), `replay.hpp` (`ReplayView` — reward-tagged memory: dedup/deepen, k-NN positive centroid with deterministic tie-break, wall-clock age/lifetime + eviction), `geo_push.hpp` (push-away target computation, upstream InterfaceRL @ `e291192` — re-based from `0a541cc` on 2026-07-25: `kGeometricPushScale` 1.0, `kNegLRBase` 1.5, and NO `/(1+len)` taper, so distance from the liked centroid does not shrink a "no". Cold start is the same push in a random direction, not a separate branch), `warm_start.hpp` (overlapping-weights copy for reshape), `stats.hpp`. `generated/ml_defaults.hpp` is codegen output (do not edit): `nisps::ml::generated::kMlTrainDefaults`, the ONE learning-rate / max-iterations / min-error default shared by firmware, WASM and VCV (source `schemas/ml_defaults.json`); `MLPCore::set_train_config()` and `nisps_ml_set_train_config()` override it at runtime. It lives under `ml/` rather than `modes/generated/` because `nisps/ml` sits below `nisps/modes` — mlp.hpp must not include upward. Jolt + OU are inert by default and wired into `ModeBase`, so every mode exposes `jolt_press/jolt_release`, `jolt_lr_scale`, and `set_explore_intensity`.
|
||||
- `nisps/pipeline/` — the control-rate input/output processing chains (P4): `input_chain.hpp` (`InputChain` — invert→deadzone→circular clamp→momentum-modulated zoom→centred power→EMA→momentum; caller-supplied dt, internal clock, fixed velocity ring, serialisable state) and `output_chain.hpp` (`OutputChain<NMax>` — curve→EMA→slew→freeze(+mask), capacity-templated). Behaviour contract = the retired manifold TS pipelines, pinned by `manifold/tests/fixtures/` and parity stage 7.
|
||||
- `nisps/dsp/` — `biquad.hpp`, `delay.hpp`, `reverb.hpp`, `filter.hpp`, `env.hpp`, `osc.hpp`, `pitch_shift.hpp`, `dc_blocker.hpp`, plus the sequencer primitives shared by the sequencer engines: `ratio_seq.hpp` and `seq_clock.hpp` (bar phasor + MIDI clock + bpm). Lean primitives extracted from maximilian; daisysp PitchShifter replaced with custom granular impl.
|
||||
- `nisps/engines/` — eight audio engines, each satisfying `AudioEngine`: `paf_synth.hpp`, `channel_strip.hpp`, `xiasri.hpp`, `verb_fx.hpp`, `memlcelium.hpp`, `breakor.hpp` (sequencer, NoOp audio), `elysiamorf.hpp` (sequencer, NoOp audio), `analysis.hpp` (input-side spectral features). Plus `base.hpp` (`NoOpEngine`, engine_id "thru").
|
||||
|
|
@ -85,8 +85,8 @@ anchor + locked decisions) and the `docs/specs/*-spec.md` set.
|
|||
BATCH "MIDI Learn": every CC swept while armed becomes an axis, shown as read-only meters). `useInputLayer.ts`
|
||||
is the React binding; `base-source.ts` shared status/action plumbing; `types.ts` the adapter contract.
|
||||
`backends/base-backend.ts` is its output-side counterpart (status + throttle + lastSent) used by the midi/osc/vcv transports.
|
||||
- `manifold/src/feedback/` — `controller.ts` (Explore-and-place scratchpad + geometric-dislike + solo; a thin
|
||||
driver over the shared C++ core).
|
||||
- `manifold/src/feedback/` — `controller.ts` (Explore-and-place scratchpad + geometric-dislike + solo; owns the
|
||||
elapsed-time timer for geometric replay while every weight-affecting step stays in the shared C++ core).
|
||||
- `manifold/src/settings/` — `settings-store.ts` (monochrome icons, input-map shape, I/O resize policy, corner radius, and the opt-in legacy Xavier/spread feature flag; Manifold randomisation is full-range uniform by default).
|
||||
- `manifold/src/serial/` — `memlnaut-serial.ts` Web Serial scaffold + `EditorPanel.tsx` (MEMLNaut Editor mode).
|
||||
- `manifold/src/engine/exploration.ts` — Jolt press + OU explore gestures (Learning drawer): a thin
|
||||
|
|
|
|||
|
|
@ -4,6 +4,26 @@
|
|||
|
||||
---
|
||||
|
||||
### 2026-07-25 implementation amendment
|
||||
|
||||
The verified upstream reference is now memllib `e291192`, vendored under
|
||||
`firmware/MEMLNaut-NISPS/lib/memllib/reference/`. Its geometric constants and schedule
|
||||
supersede the older `0a541cc` constants and synchronous-one-shot wording below:
|
||||
|
||||
- push scale `1.0`, negative-LR base `1.5`, no distance taper; cold start pushes in a
|
||||
deterministic random direction;
|
||||
- a press stores the rejection and performs one immediate update, then
|
||||
`FeedbackControllerCore::advance_geometric(dt)` replays **all** live negatives at a
|
||||
configurable rate for a full-strength wall-clock lifetime;
|
||||
- defaults match upstream: LR `0.001`, `200 Hz`, `2500 ms`; rate or lifetime zero is
|
||||
explicit one-shot mode;
|
||||
- the host supplies elapsed time, but target computation and every weight mutation stay
|
||||
in the allocation-free shared C++ core. Native↔WASM parity covers this seam.
|
||||
|
||||
Manifold exposes LR/rate/lifetime at expanded Learning depth. Deliberate divergences are
|
||||
kept current in `ALIGNMENT.md`; source comparison evidence is immutable in
|
||||
`docs/specs/recon/findings-push-away-upstream-comparison.md`.
|
||||
|
||||
## 0. Decision summary
|
||||
|
||||
| Setting | Default | Also selectable |
|
||||
|
|
|
|||
179
docs/specs/recon/findings-push-away-upstream-comparison.md
Normal file
179
docs/specs/recon/findings-push-away-upstream-comparison.md
Normal file
|
|
@ -0,0 +1,179 @@
|
|||
---
|
||||
kind: finding
|
||||
date: 2026-07-25
|
||||
immutable: true
|
||||
---
|
||||
|
||||
# Findings — Manifold “Push away” vs upstream geometric dislike
|
||||
|
||||
_Read-only comparison, 2026-07-25. “Confirmed” claims cite source; “Inference” labels
|
||||
interpretation. No implementation decision is made here._
|
||||
|
||||
## Scope and source identity
|
||||
|
||||
The hardware repository named in the question,
|
||||
[`MusicallyEmbodiedML/MEMLNaut`](https://github.com/MusicallyEmbodiedML/MEMLNaut),
|
||||
does not contain the active ML implementation. The firmware repository is
|
||||
[`MusicallyEmbodiedML/MEMLNaut-NISPS`](https://github.com/MusicallyEmbodiedML/MEMLNaut-NISPS/tree/701f2d9f1b4698e0ddfa147193928489de12601f),
|
||||
whose `src/memllib` submodule pins
|
||||
[`MusicallyEmbodiedML/memllib` at `e291192`](https://github.com/MusicallyEmbodiedML/memllib/tree/e291192d8e4f2fca7b79670c4df9c2ec8bdf03cd).
|
||||
The local read-only copies of upstream `InterfaceRL.hpp`, `.tpp`, and `.cpp` have Git
|
||||
blob hashes `03e9255`, `9ec762b`, and `60204d1`, respectively—the same blobs returned
|
||||
by GitHub for that pin. The comparison below is therefore against the exact upstream
|
||||
source, not an approximation. Provenance is recorded locally in
|
||||
`firmware/MEMLNaut-NISPS/lib/memllib/reference/README.md:1-15`.
|
||||
|
||||
## Confirmed behaviour
|
||||
|
||||
### The mental model is substantially right, but “different” needs a target
|
||||
|
||||
A negative verdict has no supervised label by itself. Upstream defines “different” as:
|
||||
take the output that was heard, find the mean output of the four liked positions nearest
|
||||
the current control input, and create a new target one unit farther from that liked
|
||||
centroid in output space. With no likes—or a degenerate zero-length direction—it uses a
|
||||
random direction. The target is clamped to `[0,1]`
|
||||
([upstream `InterfaceRL.tpp:698-761`](https://github.com/MusicallyEmbodiedML/memllib/blob/e291192d8e4f2fca7b79670c4df9c2ec8bdf03cd/examples/InterfaceRL.tpp#L698-L761);
|
||||
local mirror `firmware/MEMLNaut-NISPS/lib/memllib/reference/InterfaceRL.tpp:698-761`).
|
||||
|
||||
That target is then passed to `synthMapping.TrainBatch`. Inference subsequently reads
|
||||
the same MLP through `synthMapping.GetOutput`. Therefore current upstream push-away is
|
||||
**weight training on the mapping MLP**, not a separate rejection layer after inference
|
||||
([upstream training call](https://github.com/MusicallyEmbodiedML/memllib/blob/e291192d8e4f2fca7b79670c4df9c2ec8bdf03cd/examples/InterfaceRL.tpp#L757-L761);
|
||||
[upstream inference path](https://github.com/MusicallyEmbodiedML/memllib/blob/e291192d8e4f2fca7b79670c4df9c2ec8bdf03cd/examples/InterfaceRL.tpp#L869-L890)).
|
||||
OU noise and `paramTransformHook` do exist after MLP inference, but are independent of
|
||||
the dislike algorithm (`InterfaceRL.tpp:877-890`).
|
||||
|
||||
The Manifold path has the same basic semantics. The UI passes its current post-output-
|
||||
pipeline vector to `FeedbackController.dislike`, which calls the WASM geometric-dislike
|
||||
entry point (`manifold/src/console/ConsoleApp.tsx:509-535`;
|
||||
`manifold/src/feedback/controller.ts:347-373`;
|
||||
`manifold/src/engine/wasm-iml.ts:956-978`). The C++ controller computes the target and
|
||||
calls `MLPCore::train_targets`, which runs forward propagation, backpropagation and one
|
||||
RMSProp update on the same network weights (`nisps/ml/feedback.hpp:395-477`;
|
||||
`nisps/ml/mlp.hpp:250-287`).
|
||||
|
||||
### The target maths now matches current upstream, but the training schedule does not
|
||||
|
||||
Current Manifold main now matches upstream’s untapered target formula and constants:
|
||||
`kGeometricPushScale=1.0`, `kNegLRBase=1.5`, no `/(1+distance)` taper
|
||||
(`nisps/ml/geo_push.hpp:1-110`; upstream `InterfaceRL.hpp:406-414` and
|
||||
`InterfaceRL.tpp:723-761`). It also now uses the upstream RMSProp update rather than
|
||||
interpreting an upstream RMSProp learning rate as plain SGD
|
||||
(`nisps/ml/training.hpp:1-92`).
|
||||
|
||||
The remaining major divergence is dose:
|
||||
|
||||
- Upstream stores the dislike at press time, then its main loop calls `optimise()` on
|
||||
subsequent cycles (`InterfaceRL.tpp:38-54,186-232`).
|
||||
- Every optimisation scans **all** live negatives and batch-trains them again
|
||||
(`InterfaceRL.tpp:673-761`).
|
||||
- Upstream computes one liked centroid around the **current live control input** and
|
||||
applies it to every negative in that cycle, even if the user has moved away from the
|
||||
original disliked position (`InterfaceRL.tpp:698-755`). Manifold instead computes the
|
||||
centroid at the just-pressed negative’s stored input (`nisps/ml/feedback.hpp:431-467`).
|
||||
- A negative remains at full strength for 2500 ms, then expires; the number of updates
|
||||
depends on the mode’s loop rate (`InterfaceRL.hpp:406-414`).
|
||||
- Manifold collapses press and optimisation into **one synchronous
|
||||
`train_targets` call for only the just-pressed negative**, then proportionally decays
|
||||
stored negatives (`nisps/ml/feedback.hpp:395-477`; `nisps/ml/replay.hpp:165-185`).
|
||||
There is no background/per-frame feedback optimiser.
|
||||
|
||||
Thus a Manifold click names a strongly displaced target, but takes only one step toward
|
||||
it. Upstream keeps walking toward its target for the next 2.5 seconds. This is the most
|
||||
direct explanation for a remaining perceptual strength difference.
|
||||
|
||||
As checked on 2026-07-25, the WASM served by
|
||||
`https://meml.lnfinitemonkeys.org/next/nisps.wasm` has SHA-256
|
||||
`d1c58a59517a00c6f51870ea1ec21194561b81058e22bbfb2e11de4af45c645a`, exactly matching
|
||||
`manifold/public/nisps.wasm` on current main. The reported live behaviour therefore
|
||||
cannot be explained by production still serving the pre-RMSProp or tapered binary.
|
||||
|
||||
### Upstream also cancels a nearby positive; Manifold does not
|
||||
|
||||
Upstream’s default replay policy is `REPLACE_10_PERCENT`
|
||||
(`InterfaceRL.hpp:404-405`). When a negative is stored, a positive within input-space
|
||||
distance `0.10` is removed before the negative is added
|
||||
(`InterfaceRL.tpp:904-929,950-979`). Manifold’s replay method only deepens or adds a
|
||||
negative and leaves positives intact (`nisps/ml/replay.hpp:102-121`). Manifold also
|
||||
keeps liked examples in the separate MLP dataset
|
||||
(`manifold/src/feedback/controller.ts:375-390`).
|
||||
|
||||
**Inference:** this is less about the first click’s amplitude than persistence. A later
|
||||
positive training run can pull the mapping back toward a sound rejected near an
|
||||
existing like, whereas upstream removes that local positive from its continuously
|
||||
trained replay set.
|
||||
|
||||
### Current measured scale
|
||||
|
||||
On current main, the native behavioural benchmark at shape `2→16→16→16→8`, seed
|
||||
`24301`, reports:
|
||||
|
||||
- one geometric dislike: at-point L2 movement `0.05335`;
|
||||
- one legacy undirected Diffuse dislike: `0.22626`;
|
||||
- repeated geometric dislikes: `0.05314` after 1, `0.33748` after 10, `1.09504`
|
||||
after 100.
|
||||
|
||||
Commands:
|
||||
|
||||
```bash
|
||||
scripts/bench-ml.sh --native-only --scenario A4_negative_once
|
||||
scripts/bench-ml.sh --native-only --scenario D1_geo_anatomy
|
||||
```
|
||||
|
||||
These numbers confirm that the current path is no longer inert, but also that a single
|
||||
geometric press is still about 4.2× smaller than the legacy random-diffusion gesture in
|
||||
this benchmark. They do not by themselves establish the right musical feel.
|
||||
|
||||
At Manifold’s current default PAF shape (`4→10→10→14→33`), the same seeded scenarios
|
||||
report one-click movement `0.09654` geometric versus `0.20788` Diffuse (about 2.2×
|
||||
smaller), and geometric movement `0.55935` after ten presses. These are vector L2
|
||||
distances across 33 parameters, so they establish that weights move; they do not prove
|
||||
that the affected parameters produce a perceptually obvious timbral change.
|
||||
|
||||
The more revealing PAF-shape `A12_like_then_dislike` journey dislikes exactly where a
|
||||
liked target was taught. With one update, distance from that rejected liked target
|
||||
changes from `0.38291` to `0.36030` (`rejection_moved=-0.02261`): the mapping moves, but
|
||||
slightly **toward** the particular target the user just rejected. Ten updates change the
|
||||
distance to `0.67843` (`rejection_moved=+0.29552`). This is deterministic evidence that
|
||||
one update is not sufficient to realise the user-facing semantic in an important
|
||||
contradictory-feedback case; it also motivates testing upstream’s nearby-like removal
|
||||
separately.
|
||||
|
||||
## Why it likely felt weak
|
||||
|
||||
1. **Fixed today: optimiser mismatch.** Before `f57cddc`, the port used a tiny upstream
|
||||
RMSProp learning rate inside plain SGD, reducing one press to roughly `5.3e-5`
|
||||
movement.
|
||||
2. **Fixed today: superseded push formula.** Before `ec31180`, the port halved the target
|
||||
step, used one-third the negative-LR base, and tapered the step by distance—the exact
|
||||
case upstream says made “no” ineffective.
|
||||
3. **Still present: one update versus a time window of updates.** Manifold performs one
|
||||
weight update per click; upstream replays all dislikes repeatedly for 2.5 seconds.
|
||||
4. **Still present: highly asymmetric teaching dose.** A Manifold like trains the whole
|
||||
positive dataset at `lr=1.0` for up to 1000 iterations, while a dislike takes one
|
||||
roughly `0.0015` RMSProp step. Relative to the lurch caused by a like, correction
|
||||
still feels small (`ALIGNMENT.md:66-84`).
|
||||
5. **Potential persistence mismatch:** nearby likes are retained locally but removed by
|
||||
upstream’s default replay policy, so future positive retraining may partly undo the
|
||||
rejection.
|
||||
|
||||
## Recommended next experiment
|
||||
|
||||
Do not start by increasing `geo_lr` blindly. That can make a click stronger but does not
|
||||
answer whether the intended interaction is one discrete correction or a short-lived
|
||||
repulsive constraint.
|
||||
|
||||
Add a deterministic “advance feedback by `dt`” core seam and benchmark three matched
|
||||
variants from the same seeded prefix:
|
||||
|
||||
1. current one-shot update;
|
||||
2. upstream-style replay of all negatives for 2500 ms, while separately testing whether
|
||||
centroid lookup follows each stored negative or upstream’s current live input;
|
||||
3. a bounded discrete equivalent (for example 10/25/50 steps at press time) that avoids
|
||||
wall-clock ownership in the core.
|
||||
|
||||
For each, report at-point movement, neighbourhood rings, global blast ratio, collateral
|
||||
movement at liked positions, and like→dislike→like persistence. Then perform an
|
||||
in-Manifold blind A/B at the real per-mode output arities and choose the smallest dose
|
||||
that makes one rejection obvious without damaging liked regions. Test the nearby-like
|
||||
removal policy as a separate axis rather than coupling it to dose.
|
||||
File diff suppressed because one or more lines are too long
Binary file not shown.
|
|
@ -29,7 +29,13 @@
|
|||
*/
|
||||
import { useEffect, useMemo, useRef, useState } from 'react';
|
||||
import type { CSSProperties } from 'react';
|
||||
import { useEngine, useEngineVersion, ExplorationController } from '../engine';
|
||||
import {
|
||||
useEngine,
|
||||
useEngineVersion,
|
||||
ExplorationController,
|
||||
DEFAULT_GEOMETRIC_FEEDBACK_CONFIG,
|
||||
type GeometricFeedbackConfig,
|
||||
} from '../engine';
|
||||
import { MF_MODES, createOutputParam, modeEngineId, shapeValues } from './model';
|
||||
import type { MFParam } from './model';
|
||||
import { CompositeStage } from './CompositeStage';
|
||||
|
|
@ -110,6 +116,9 @@ export function ConsoleApp() {
|
|||
const [pos, setPos] = useState<[number, number]>([0.5, 0.5]);
|
||||
|
||||
const [noiseCap, setNoiseCap] = useState(0.12);
|
||||
const [geometricConfig, setGeometricConfig] = useState<GeometricFeedbackConfig>(() => ({
|
||||
...DEFAULT_GEOMETRIC_FEEDBACK_CONFIG,
|
||||
}));
|
||||
const [examples, setExamples] = useState(0);
|
||||
const [addingExample, setAddingExample] = useState(false);
|
||||
const [busy, setBusy] = useState(false);
|
||||
|
|
@ -186,6 +195,7 @@ export function ConsoleApp() {
|
|||
if (engine && !controllerRef.current) {
|
||||
controllerRef.current = new FeedbackController(engine, {
|
||||
spread: randomisationSpread,
|
||||
geometricConfig,
|
||||
});
|
||||
}
|
||||
|
||||
|
|
@ -210,6 +220,8 @@ export function ConsoleApp() {
|
|||
}
|
||||
useEffect(
|
||||
() => () => {
|
||||
controllerRef.current?.dispose();
|
||||
controllerRef.current = null;
|
||||
explorationRef.current?.dispose();
|
||||
explorationRef.current = null;
|
||||
},
|
||||
|
|
@ -248,6 +260,10 @@ export function ConsoleApp() {
|
|||
controllerRef.current?.setSpread(randomisationSpread);
|
||||
}, [engine, randomisationSpread]);
|
||||
|
||||
useEffect(() => {
|
||||
controllerRef.current?.setGeometricConfig(geometricConfig);
|
||||
}, [engine, geometricConfig]);
|
||||
|
||||
// Push the selected solo-mode + the arm mask into the controller whenever they
|
||||
// change (dock-spec §1.2). The controller RESPECTS the arm mask at the example
|
||||
// level in BOTH modes and forwards it to engine.feedback.setFocus.
|
||||
|
|
@ -968,6 +984,8 @@ export function ConsoleApp() {
|
|||
xavierSpreadEnabled: settings.xavierSpreadEnabled,
|
||||
noiseCap,
|
||||
setNoiseCap,
|
||||
geometricConfig,
|
||||
setGeometricConfig,
|
||||
// learning-behaviour
|
||||
feedbackMode,
|
||||
setFeedbackMode,
|
||||
|
|
|
|||
|
|
@ -31,7 +31,11 @@ import { outputModeDescriptor } from './output-mode';
|
|||
import { useSettings, unfocusedIconCss } from '../settings/settings-store';
|
||||
import type { UnfocusedIconColour, InputMapMode } from '../settings/settings-store';
|
||||
import type { ExampleResizePolicy, NetworkResizePolicy } from '../engine/io-reshape';
|
||||
import { useEngine, useEngineVersion } from '../engine';
|
||||
import {
|
||||
useEngine,
|
||||
useEngineVersion,
|
||||
DEFAULT_GEOMETRIC_FEEDBACK_CONFIG,
|
||||
} from '../engine';
|
||||
import { EditorPanel } from '../serial/EditorPanel';
|
||||
import { TrainingHealth } from './TrainingHealth';
|
||||
import {
|
||||
|
|
@ -315,7 +319,76 @@ function LearningDrawer(ctx: ConsoleCtx, depth: DrawerDepth) {
|
|||
</span>
|
||||
</div>
|
||||
<SectionLabel>Live training params</SectionLabel>
|
||||
<Slider label="noise cap" value={ctx.noiseCap} min={0} max={0.5} step={0.01} onChange={ctx.setNoiseCap} />
|
||||
{ctx.feedbackMode === 'geometric-dislike' ? (
|
||||
<>
|
||||
<Slider
|
||||
label="push · learning rate"
|
||||
value={ctx.geometricConfig.learningRate}
|
||||
min={0.0001}
|
||||
max={0.005}
|
||||
step={0.0001}
|
||||
format={(v) => v.toFixed(4)}
|
||||
onChange={(learningRate) =>
|
||||
ctx.setGeometricConfig({ ...ctx.geometricConfig, learningRate })
|
||||
}
|
||||
/>
|
||||
<Slider
|
||||
label="push · updates / second"
|
||||
value={ctx.geometricConfig.updatesPerSecond}
|
||||
min={0}
|
||||
max={400}
|
||||
step={10}
|
||||
format={(v) => `${Math.round(v)} Hz`}
|
||||
onChange={(updatesPerSecond) =>
|
||||
ctx.setGeometricConfig({ ...ctx.geometricConfig, updatesPerSecond })
|
||||
}
|
||||
/>
|
||||
<Slider
|
||||
label="push · lifetime"
|
||||
value={ctx.geometricConfig.lifetimeMs / 1000}
|
||||
min={0}
|
||||
max={5}
|
||||
step={0.1}
|
||||
format={(v) => `${v.toFixed(1)} s`}
|
||||
onChange={(seconds) =>
|
||||
ctx.setGeometricConfig({
|
||||
...ctx.geometricConfig,
|
||||
lifetimeMs: seconds * 1000,
|
||||
})
|
||||
}
|
||||
/>
|
||||
<div style={{ display: 'flex', gap: 8, alignItems: 'center', flexWrap: 'wrap' }}>
|
||||
<Button
|
||||
size="sm"
|
||||
variant="secondary"
|
||||
onClick={() =>
|
||||
ctx.setGeometricConfig({ ...DEFAULT_GEOMETRIC_FEEDBACK_CONFIG })
|
||||
}
|
||||
>
|
||||
Upstream defaults
|
||||
</Button>
|
||||
<span style={{ fontSize: 'var(--fs-xs)', color: 'var(--fg-dim)' }}>
|
||||
{ctx.geometricConfig.updatesPerSecond <= 0 ||
|
||||
ctx.geometricConfig.lifetimeMs <= 0
|
||||
? 'one immediate update per press'
|
||||
: `≈ ${Math.round(
|
||||
(ctx.geometricConfig.updatesPerSecond *
|
||||
ctx.geometricConfig.lifetimeMs) /
|
||||
1000,
|
||||
)} replay updates + the press`}
|
||||
</span>
|
||||
</div>
|
||||
</>
|
||||
) : (
|
||||
<Slider
|
||||
label="noise cap"
|
||||
value={ctx.noiseCap}
|
||||
min={0}
|
||||
max={0.5}
|
||||
step={0.01}
|
||||
onChange={ctx.setNoiseCap}
|
||||
/>
|
||||
)}
|
||||
</>
|
||||
)}
|
||||
|
||||
|
|
|
|||
|
|
@ -5,6 +5,7 @@ import type { MFMode, MFParam } from './model';
|
|||
import type { FeedbackMode } from '../engine/types';
|
||||
import type { BackendStatus } from '../backends/backend';
|
||||
import type { UseInputLayer } from '../inputs';
|
||||
import type { GeometricFeedbackConfig } from '../engine';
|
||||
|
||||
/** The two product feedback modes (dock-spec §1.1; rl-feedback-design §0). */
|
||||
export type FeedbackModeUI = 'explore-and-place' | 'geometric-dislike';
|
||||
|
|
@ -124,6 +125,8 @@ export interface ConsoleCtx {
|
|||
xavierSpreadEnabled: boolean;
|
||||
noiseCap: number;
|
||||
setNoiseCap: (v: number) => void;
|
||||
geometricConfig: GeometricFeedbackConfig;
|
||||
setGeometricConfig: (config: GeometricFeedbackConfig) => void;
|
||||
|
||||
// ---- Learning-behaviour (dock-spec §1; rl-feedback-design) ----
|
||||
feedbackMode: FeedbackModeUI;
|
||||
|
|
|
|||
|
|
@ -52,6 +52,8 @@ export interface DebugProbe {
|
|||
* int (14=GeometricPush, 15=GeometricColdStart).
|
||||
*/
|
||||
dislikeGeometric(heardVec: ReadonlyArray<number>, lr?: number): number;
|
||||
setGeometricConfig(learningRate: number, updatesPerSecond: number, lifetimeMs: number): void;
|
||||
advanceGeometric(dtSeconds: number): number;
|
||||
/** Feed a positive into the k-NN centroid (omit vec → live MLP output). */
|
||||
storePositive(vec?: ReadonlyArray<number>): void;
|
||||
/** Replay-memory sizes (Mode 1). */
|
||||
|
|
@ -174,6 +176,20 @@ function makeProbe(engine: EngineApi): DebugProbe {
|
|||
return a;
|
||||
},
|
||||
|
||||
setGeometricConfig(
|
||||
learningRate: number,
|
||||
updatesPerSecond: number,
|
||||
lifetimeMs: number,
|
||||
): void {
|
||||
engine.feedback.setGeometricConfig({ learningRate, updatesPerSecond, lifetimeMs });
|
||||
},
|
||||
|
||||
advanceGeometric(dtSeconds: number): number {
|
||||
const steps = engine.feedback.advanceGeometric(dtSeconds);
|
||||
if (steps > 0) engine.process();
|
||||
return steps;
|
||||
},
|
||||
|
||||
storePositive(vec?: ReadonlyArray<number>): void {
|
||||
engine.feedback.storePositive(vec ? Float32Array.from(vec) : undefined);
|
||||
},
|
||||
|
|
|
|||
|
|
@ -23,6 +23,18 @@ import type { EngineId, FeedbackMode, LayerStats } from './types';
|
|||
import { WasmIML } from './wasm-iml';
|
||||
import type { IoMigration } from './io-reshape';
|
||||
|
||||
export interface GeometricFeedbackConfig {
|
||||
learningRate: number;
|
||||
updatesPerSecond: number;
|
||||
lifetimeMs: number;
|
||||
}
|
||||
|
||||
export const DEFAULT_GEOMETRIC_FEEDBACK_CONFIG: Readonly<GeometricFeedbackConfig> = {
|
||||
learningRate: 0.001,
|
||||
updatesPerSecond: 200,
|
||||
lifetimeMs: 2500,
|
||||
};
|
||||
|
||||
export interface EngineFeedbackApi {
|
||||
/** Positive feedback (thumbs-up). Returns the FeedbackAction int. */
|
||||
thumbsUp(): number;
|
||||
|
|
@ -65,6 +77,10 @@ export interface EngineFeedbackApi {
|
|||
* FeedbackAction int (14=GeometricPush, 15=GeometricColdStart).
|
||||
*/
|
||||
dislikeGeometric(heardVec?: Float32Array, lr?: number): number;
|
||||
/** Configure upstream-style replay dose and wall-clock lifetime. */
|
||||
setGeometricConfig(config: GeometricFeedbackConfig): void;
|
||||
/** Advance replay by elapsed wall-clock time; returns optimise cycles run. */
|
||||
advanceGeometric(dtSeconds: number): number;
|
||||
/** Feed a positive (like) into the k-NN centroid (null → live MLP output). */
|
||||
storePositive(vec?: Float32Array): void;
|
||||
positiveCount(): number;
|
||||
|
|
@ -185,6 +201,13 @@ export class EngineApi {
|
|||
placedOutput: () => this.iml.feedbackPlacedOutput(),
|
||||
dislikeGeometric: (heardVec?: Float32Array, lr = 0) =>
|
||||
this.iml.feedbackDislikeGeometric(heardVec, lr),
|
||||
setGeometricConfig: (config) =>
|
||||
this.iml.feedbackSetGeometricConfig(
|
||||
config.learningRate,
|
||||
config.updatesPerSecond,
|
||||
config.lifetimeMs,
|
||||
),
|
||||
advanceGeometric: (dtSeconds) => this.iml.feedbackAdvanceGeometric(dtSeconds),
|
||||
storePositive: (vec?: Float32Array) => this.iml.feedbackStorePositive(vec),
|
||||
positiveCount: () => this.iml.feedbackPositiveCount(),
|
||||
negativeCount: () => this.iml.feedbackNegativeCount(),
|
||||
|
|
|
|||
|
|
@ -5,12 +5,17 @@
|
|||
* (everything except EngineProvider/useEngine) imports NO React.
|
||||
*/
|
||||
|
||||
export { EngineApi, createEngine } from './engine-api';
|
||||
export {
|
||||
EngineApi,
|
||||
createEngine,
|
||||
DEFAULT_GEOMETRIC_FEEDBACK_CONFIG,
|
||||
} from './engine-api';
|
||||
export type {
|
||||
EngineApiOptions,
|
||||
EngineAudioApi,
|
||||
EngineFeedbackApi,
|
||||
EngineExploreApi,
|
||||
GeometricFeedbackConfig,
|
||||
} from './engine-api';
|
||||
|
||||
export { Spine } from './spine';
|
||||
|
|
|
|||
|
|
@ -114,6 +114,13 @@ export interface NispsModule {
|
|||
// controller default (1e-3). Returns the FeedbackAction int (14=GeometricPush,
|
||||
// 15=GeometricColdStart when no positives exist yet).
|
||||
_nisps_ml_feedback_dislike_geometric(ml: number, current_out_ptr: number, lr: number): number;
|
||||
_nisps_ml_feedback_set_geometric_config(
|
||||
ml: number,
|
||||
lr: number,
|
||||
update_hz: number,
|
||||
lifetime_ms: number,
|
||||
): void;
|
||||
_nisps_ml_feedback_advance_geometric(ml: number, dt_seconds: number): number;
|
||||
// Store a positive (like) into the replay memory so the k-NN centroid sees it.
|
||||
// current_out may be null (live output used). Caller still runs addExample+train.
|
||||
_nisps_ml_feedback_store_positive(ml: number, current_out_ptr: number): void;
|
||||
|
|
|
|||
|
|
@ -978,6 +978,24 @@ export class WasmIML {
|
|||
return action;
|
||||
}
|
||||
|
||||
feedbackSetGeometricConfig(lr: number, updateHz: number, lifetimeMs: number): void {
|
||||
this.module._nisps_ml_feedback_set_geometric_config(
|
||||
this.mlHandle,
|
||||
lr,
|
||||
updateHz,
|
||||
lifetimeMs,
|
||||
);
|
||||
}
|
||||
|
||||
feedbackAdvanceGeometric(dtSeconds: number): number {
|
||||
const steps = this.module._nisps_ml_feedback_advance_geometric(this.mlHandle, dtSeconds);
|
||||
if (steps > 0) {
|
||||
this.sink.emit('ml.delta_update', { reason: 'geometric-replay' });
|
||||
this.scheduleSave_();
|
||||
}
|
||||
return steps;
|
||||
}
|
||||
|
||||
/**
|
||||
* Feed a positive (like) into the replay memory so the k-NN centroid sees it.
|
||||
* `vec` is the heard output at the liked input (null → the live MLP output).
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
/**
|
||||
* FeedbackController — framework-neutral learning-engine behaviour for the two
|
||||
* feedback modes plus solo/arm, prototyped in pure TS on the EXISTING engine
|
||||
* primitives (NO C++/WASM change).
|
||||
* FeedbackController — framework-neutral browser driver for the two feedback
|
||||
* modes plus solo/arm. Weight-affecting behaviour lives in the shared C++ core;
|
||||
* this layer owns UI scheduling and caller-owned example storage.
|
||||
*
|
||||
* Authoritative design: docs/adr/rl-feedback-design.md (Mode 2 default;
|
||||
* Mode 1 selectable; SOLO default MaskGradients). Engine primitives audited in
|
||||
|
|
@ -14,7 +14,7 @@
|
|||
* setInput(x,y) / getOutputs() — synchronous forward inference (the spine)
|
||||
* process() — re-run last input after a weight change
|
||||
* addExample([x,y], outVec) — append a training example
|
||||
* train() — SGD over the dataset
|
||||
* train() — supervised training over the dataset
|
||||
* feedback.{setFocus,thumbsUp,dislikeGeometric,…}
|
||||
* — the SHARED C++ core's RL primitives
|
||||
*
|
||||
|
|
@ -27,6 +27,10 @@
|
|||
*/
|
||||
|
||||
import type { FeedbackMode } from '../engine/types';
|
||||
import {
|
||||
DEFAULT_GEOMETRIC_FEEDBACK_CONFIG,
|
||||
type GeometricFeedbackConfig,
|
||||
} from '../engine/engine-api';
|
||||
|
||||
/** The two product feedback modes (rl-feedback-design §0). */
|
||||
export type ProtoFeedbackMode = 'explore-and-place' | 'geometric-dislike';
|
||||
|
|
@ -67,6 +71,8 @@ export interface ControllerEngine {
|
|||
// `heardVec` is the post-pipeline (HEARD) output; returns the FeedbackAction
|
||||
// int (14=GeometricPush, 15=GeometricColdStart).
|
||||
dislikeGeometric(heardVec?: Float32Array, lr?: number): number;
|
||||
setGeometricConfig(config: GeometricFeedbackConfig): void;
|
||||
advanceGeometric(dtSeconds: number): number;
|
||||
positiveCount(): number;
|
||||
negativeCount(): number;
|
||||
// ExploreAndPlace lifecycle — the SHARED C++ core (mode 'explore_and_place').
|
||||
|
|
@ -112,12 +118,16 @@ export interface FeedbackControllerOptions {
|
|||
spread?: number;
|
||||
/** Nudge perturbation standard deviation (small bounded weight jitter). */
|
||||
nudgeStddev?: number;
|
||||
geometricConfig?: GeometricFeedbackConfig;
|
||||
}
|
||||
|
||||
export class FeedbackController {
|
||||
private engine: ControllerEngine;
|
||||
private spread: number;
|
||||
private nudgeStddev: number;
|
||||
private geometricConfig: GeometricFeedbackConfig;
|
||||
private geometricTimer: ReturnType<typeof setInterval> | null = null;
|
||||
private geometricLastTickMs = 0;
|
||||
|
||||
private mode: ProtoFeedbackMode = 'explore-and-place';
|
||||
private soloMode: ProtoSoloMode = 'mask-gradients';
|
||||
|
|
@ -147,6 +157,10 @@ export class FeedbackController {
|
|||
this.engine = engine;
|
||||
this.spread = opts.spread ?? 0;
|
||||
this.nudgeStddev = opts.nudgeStddev ?? 0.05;
|
||||
this.geometricConfig = {
|
||||
...(opts.geometricConfig ?? DEFAULT_GEOMETRIC_FEEDBACK_CONFIG),
|
||||
};
|
||||
this.engine.feedback.setGeometricConfig(this.geometricConfig);
|
||||
}
|
||||
|
||||
// ===================================================================
|
||||
|
|
@ -158,6 +172,7 @@ export class FeedbackController {
|
|||
// Switching mode aborts any active scratchpad session. For explore-and-place
|
||||
// the SHARED C++ core owns the scratchpad, so delegate the teardown to it.
|
||||
if (this.exploringFlag) this.cancel();
|
||||
if (mode !== 'geometric-dislike') this.stopGeometricReplay();
|
||||
this.mode = mode;
|
||||
// Keep the C++ core's feedback mode in lockstep so the shared explore-and-
|
||||
// place lifecycle is active when this mode is selected.
|
||||
|
|
@ -176,11 +191,29 @@ export class FeedbackController {
|
|||
this.spread = spread;
|
||||
}
|
||||
|
||||
setGeometricConfig(config: GeometricFeedbackConfig): void {
|
||||
this.geometricConfig = {
|
||||
learningRate: Math.max(0.000001, config.learningRate),
|
||||
updatesPerSecond: Math.max(0, config.updatesPerSecond),
|
||||
lifetimeMs: Math.max(0, config.lifetimeMs),
|
||||
};
|
||||
this.engine.feedback.setGeometricConfig(this.geometricConfig);
|
||||
if (this.geometricTimer) {
|
||||
this.stopGeometricReplay();
|
||||
if (this.engine.feedback.negativeCount() > 0) this.startGeometricReplay();
|
||||
}
|
||||
}
|
||||
|
||||
getGeometricConfig(): Readonly<GeometricFeedbackConfig> {
|
||||
return this.geometricConfig;
|
||||
}
|
||||
|
||||
/**
|
||||
* An I/O identity edit resets the core's index-aligned scratch/replay state.
|
||||
* Mirror that reset locally without issuing another core transition.
|
||||
*/
|
||||
resetAfterIoChange(): void {
|
||||
this.stopGeometricReplay();
|
||||
this.exploringFlag = false;
|
||||
this.pickingFlag = false;
|
||||
this.anchors = [];
|
||||
|
|
@ -351,14 +384,15 @@ export class FeedbackController {
|
|||
/**
|
||||
* DISLIKE (thumbs-down in Mode 1). Push the current mapping away from the liked
|
||||
* centroid — the SHARED C++ core's geometric push (nisps/ml/geo_push.hpp +
|
||||
* replay.hpp + mlp.train_targets, ported from upstream InterfaceRL 0a541cc).
|
||||
* replay.hpp + mlp.train_targets, ported from upstream InterfaceRL e291192).
|
||||
*
|
||||
* The core uses the MLP's CURRENT input and the passed HEARD output vector:
|
||||
* 1. stores the negative (input, a_neg) in the ReplayStore (dedup within 0.05)
|
||||
* 2. k-NN(k=4) centroid of positives near the input
|
||||
* 3. target[j] = clamp(a_neg[j] + dir/||dir|| · pushStep/(1+||dir||), 0, 1)
|
||||
* 4. trains toward that target at lr·negLRRatio
|
||||
* 5. cold-start fallback (negative-LR) when there are no positives yet.
|
||||
* 3. target[j] = clamp(a_neg[j] + dir/||dir|| · pushStep, 0, 1)
|
||||
* 4. trains all live negatives once immediately, then at the configured
|
||||
* replay rate until each rejection reaches its wall-clock lifetime
|
||||
* 5. uses a deterministic random direction when there are no positives yet.
|
||||
* Soloed/active dims come from the core's focus mask (set via setArmMask).
|
||||
*
|
||||
* @param output the HEARD (post-pipeline) output vector a_neg. MUST be the
|
||||
|
|
@ -369,9 +403,44 @@ export class FeedbackController {
|
|||
dislike(output: Float32Array): number {
|
||||
const action = this.engine.feedback.dislikeGeometric(output);
|
||||
this.engine.process();
|
||||
this.startGeometricReplay();
|
||||
return action;
|
||||
}
|
||||
|
||||
private startGeometricReplay(): void {
|
||||
this.stopGeometricReplay();
|
||||
if (
|
||||
this.mode !== 'geometric-dislike' ||
|
||||
this.geometricConfig.updatesPerSecond <= 0 ||
|
||||
this.geometricConfig.lifetimeMs <= 0 ||
|
||||
this.engine.feedback.negativeCount() <= 0
|
||||
) {
|
||||
return;
|
||||
}
|
||||
this.geometricLastTickMs = Date.now();
|
||||
const intervalMs = Math.max(
|
||||
5,
|
||||
Math.min(50, 1000 / this.geometricConfig.updatesPerSecond),
|
||||
);
|
||||
this.geometricTimer = setInterval(() => {
|
||||
const now = Date.now();
|
||||
const dtSeconds = Math.max(0, (now - this.geometricLastTickMs) / 1000);
|
||||
this.geometricLastTickMs = now;
|
||||
const steps = this.engine.feedback.advanceGeometric(dtSeconds);
|
||||
if (steps > 0) this.engine.process();
|
||||
if (this.engine.feedback.negativeCount() <= 0) this.stopGeometricReplay();
|
||||
}, intervalMs);
|
||||
}
|
||||
|
||||
private stopGeometricReplay(): void {
|
||||
if (this.geometricTimer !== null) clearInterval(this.geometricTimer);
|
||||
this.geometricTimer = null;
|
||||
}
|
||||
|
||||
dispose(): void {
|
||||
this.stopGeometricReplay();
|
||||
}
|
||||
|
||||
/**
|
||||
* LIKE + train (thumbs-up in Mode 1). Feeds the positive into the C++ core's
|
||||
* k-NN centroid (via the core's thumbsUp auto-store, ADR §2.1) AND stores the
|
||||
|
|
|
|||
|
|
@ -70,4 +70,36 @@ test.describe('geometric dislike (Mode 1) — core-backed', () => {
|
|||
expect(countChanged(before, after, 1e-4)).toBeGreaterThan(0);
|
||||
expect(allWithin(after, 0, 1)).toBe(true);
|
||||
});
|
||||
|
||||
test('parameterised replay applies configured dose then expires by wall time', async ({ page }) => {
|
||||
const result = await page.evaluate((heard) => {
|
||||
const p = window.__nisps!;
|
||||
p.setFeedbackMode('avoid');
|
||||
p.setAvoidStyle(0);
|
||||
p.setGeometricConfig(0.001, 20, 100);
|
||||
p.setInputs(0.4, 0.6);
|
||||
p.dislikeGeometric(heard);
|
||||
const first = p.advanceGeometric(0.05);
|
||||
const live = p.feedbackCounts().negative;
|
||||
const second = p.advanceGeometric(0.05);
|
||||
const expired = p.feedbackCounts().negative;
|
||||
return { first, live, second, expired };
|
||||
}, HEARD);
|
||||
|
||||
expect(result).toEqual({ first: 1, live: 1, second: 1, expired: 0 });
|
||||
});
|
||||
|
||||
test('expanded Learning panel exposes the upstream-default experiment controls', async ({ page }) => {
|
||||
await page.getByTitle('Learning', { exact: true }).click();
|
||||
await page.getByTitle('Expand', { exact: true }).click();
|
||||
|
||||
await expect(page.getByText('push · learning rate', { exact: true })).toBeVisible();
|
||||
await expect(page.getByText('push · updates / second', { exact: true })).toBeVisible();
|
||||
await expect(page.getByText('push · lifetime', { exact: true })).toBeVisible();
|
||||
await expect(page.getByText('0.0010', { exact: true })).toBeVisible();
|
||||
await expect(page.getByText('200 Hz', { exact: true })).toBeVisible();
|
||||
await expect(page.getByText('2.5 s', { exact: true })).toBeVisible();
|
||||
await expect(page.getByRole('button', { name: 'Upstream defaults' })).toBeVisible();
|
||||
await expect(page.getByText('≈ 500 replay updates + the press')).toBeVisible();
|
||||
});
|
||||
});
|
||||
|
|
|
|||
|
|
@ -249,6 +249,7 @@ class DynamicFeedbackStorage {
|
|||
+ replay_cap * n_in // replay inputs
|
||||
+ replay_cap * n_out // replay actions
|
||||
+ replay_cap // replay rewards
|
||||
+ replay_cap // replay ages (ms)
|
||||
+ n_out * 2u // centroid + target
|
||||
+ focus_floats;
|
||||
arena_ = new (std::nothrow) float[total]();
|
||||
|
|
@ -260,7 +261,8 @@ class DynamicFeedbackStorage {
|
|||
off_replay_in_ = off_undo_ + n_weights_ * undo_cap_;
|
||||
off_replay_a_ = off_replay_in_ + replay_cap_ * n_in_;
|
||||
off_replay_r_ = off_replay_a_ + replay_cap_ * n_out_;
|
||||
off_centroid_ = off_replay_r_ + replay_cap_;
|
||||
off_replay_age_ = off_replay_r_ + replay_cap_;
|
||||
off_centroid_ = off_replay_age_ + replay_cap_;
|
||||
off_target_ = off_centroid_ + n_out_;
|
||||
off_focus_ = off_target_ + n_out_;
|
||||
}
|
||||
|
|
@ -302,6 +304,7 @@ class DynamicFeedbackStorage {
|
|||
std::span<float> replay_inputs() noexcept { return {arena_ + off_replay_in_, replay_cap_ * n_in_}; }
|
||||
std::span<float> replay_actions() noexcept { return {arena_ + off_replay_a_, replay_cap_ * n_out_}; }
|
||||
std::span<float> replay_rewards() noexcept { return {arena_ + off_replay_r_, replay_cap_}; }
|
||||
std::span<float> replay_ages_ms() noexcept { return {arena_ + off_replay_age_, replay_cap_}; }
|
||||
std::span<float> centroid_buf() noexcept { return {arena_ + off_centroid_, n_out_}; }
|
||||
std::span<float> target_buf() noexcept { return {arena_ + off_target_, n_out_}; }
|
||||
std::span<std::uint8_t> focus() noexcept {
|
||||
|
|
@ -318,7 +321,8 @@ class DynamicFeedbackStorage {
|
|||
off_placed_ = o.off_placed_; off_snap_ = o.off_snap_;
|
||||
off_scratch_ = o.off_scratch_; off_undo_ = o.off_undo_; off_focus_ = o.off_focus_;
|
||||
off_replay_in_ = o.off_replay_in_; off_replay_a_ = o.off_replay_a_;
|
||||
off_replay_r_ = o.off_replay_r_; off_centroid_ = o.off_centroid_;
|
||||
off_replay_r_ = o.off_replay_r_; off_replay_age_ = o.off_replay_age_;
|
||||
off_centroid_ = o.off_centroid_;
|
||||
off_target_ = o.off_target_;
|
||||
arena_ = o.arena_;
|
||||
o.arena_ = nullptr;
|
||||
|
|
@ -328,7 +332,7 @@ class DynamicFeedbackStorage {
|
|||
std::size_t off_placed_ = 0u, off_snap_ = 0u, off_scratch_ = 0u,
|
||||
off_undo_ = 0u, off_focus_ = 0u, off_replay_in_ = 0u,
|
||||
off_replay_a_ = 0u, off_replay_r_ = 0u, off_centroid_ = 0u,
|
||||
off_target_ = 0u;
|
||||
off_replay_age_ = 0u, off_target_ = 0u;
|
||||
float* arena_ = nullptr;
|
||||
};
|
||||
|
||||
|
|
|
|||
|
|
@ -18,9 +18,9 @@
|
|||
// - Geometric (DEFAULT) — the ported firmware k-NN
|
||||
// centroid push-away, backed by the controller's
|
||||
// own ReplayMemory (see dislike_geometric() below).
|
||||
// Cold-starts with a negative-LR fallback until the
|
||||
// first positive is stored. The old "geometric push
|
||||
// not ported" note is stale — it IS ported.
|
||||
// Replays all live negatives for a parameterised
|
||||
// wall-clock window; cold start uses a deterministic
|
||||
// random direction until a positive is stored.
|
||||
// - Diffuse — the pre-P3 undirected MLP::move_weights
|
||||
// perturb. Deliberately-retained research reserve,
|
||||
// not the product default (see its definition below
|
||||
|
|
@ -138,7 +138,7 @@ enum class FeedbackAction : std::uint8_t {
|
|||
CancelPlace = 13, // Placing→Exploring; backed out of placing (no store).
|
||||
// ---- Geometric dislike (append-only) ----
|
||||
GeometricPush = 14, // dislike trained toward the computed push-away target.
|
||||
GeometricColdStart = 15, // no positives yet: negative-LR fallback ran; UI shows the cold-start prompt.
|
||||
GeometricColdStart = 15, // no positives yet: random-direction push ran; UI shows the cold-start prompt.
|
||||
};
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
|
|
@ -180,6 +180,7 @@ class FixedFeedbackStorage {
|
|||
NISPS_FORCE_INLINE std::span<float> replay_inputs() noexcept { return replay_in_; }
|
||||
NISPS_FORCE_INLINE std::span<float> replay_actions() noexcept { return replay_act_; }
|
||||
NISPS_FORCE_INLINE std::span<float> replay_rewards() noexcept { return replay_rew_; }
|
||||
NISPS_FORCE_INLINE std::span<float> replay_ages_ms() noexcept { return replay_age_ms_; }
|
||||
// Centroid + push-target scratch (n_out each).
|
||||
NISPS_FORCE_INLINE std::span<float> centroid_buf() noexcept { return centroid_; }
|
||||
NISPS_FORCE_INLINE std::span<float> target_buf() noexcept { return target_; }
|
||||
|
|
@ -194,6 +195,7 @@ class FixedFeedbackStorage {
|
|||
std::array<float, ReplayCap * NIn> replay_in_{};
|
||||
std::array<float, ReplayCap * NOut> replay_act_{};
|
||||
std::array<float, ReplayCap> replay_rew_{};
|
||||
std::array<float, ReplayCap> replay_age_ms_{};
|
||||
std::array<float, NOut> centroid_{};
|
||||
std::array<float, NOut> target_{};
|
||||
};
|
||||
|
|
@ -230,6 +232,17 @@ class FeedbackControllerCore : public FbStorage {
|
|||
// InterfaceRL default 1e-3, pre-scaling).
|
||||
void set_geo_lr(float lr) noexcept { geo_lr_ = lr; }
|
||||
float geo_lr() const noexcept { return geo_lr_; }
|
||||
void set_geo_update_hz(float hz) noexcept {
|
||||
geo_update_hz_ = (hz > 0.f) ? hz : 0.f;
|
||||
geo_step_accum_ = 0.f;
|
||||
if (!(geo_update_hz_ > 0.f)) replay_().remove_all_negatives();
|
||||
}
|
||||
float geo_update_hz() const noexcept { return geo_update_hz_; }
|
||||
void set_geo_lifetime_ms(float ms) noexcept {
|
||||
geo_lifetime_ms_ = (ms > 0.f) ? ms : 0.f;
|
||||
if (!(geo_lifetime_ms_ > 0.f)) replay_().remove_all_negatives();
|
||||
}
|
||||
float geo_lifetime_ms() const noexcept { return geo_lifetime_ms_; }
|
||||
|
||||
// `exploring()` is true whenever a scratchpad net is live and learning is
|
||||
// paused — for the legacy RANDOMISE_* modes, AND for ExploreAndPlace in
|
||||
|
|
@ -393,9 +406,9 @@ class FeedbackControllerCore : public FbStorage {
|
|||
void seed(std::uint64_t s) noexcept { rng_.seed(s); }
|
||||
|
||||
// =========================================================================
|
||||
// Geometric dislike (rl-feedback-design §2.1) — the press-time half and
|
||||
// the async optimise() half of upstream InterfaceRL collapsed into ONE
|
||||
// synchronous call (nisps has no background optimise driver).
|
||||
// Geometric dislike (rl-feedback-design §2.1): press stores the rejection
|
||||
// and performs one immediate update; advance_geometric() supplies the
|
||||
// repeated, wall-clock-bounded optimise dose used by current upstream.
|
||||
// =========================================================================
|
||||
|
||||
std::size_t replay_size() const noexcept { return replay_count_; }
|
||||
|
|
@ -416,18 +429,12 @@ class FeedbackControllerCore : public FbStorage {
|
|||
// Thumbs-down at the MLP's CURRENT input with heard action `current_out`
|
||||
// (empty ⇒ the MLP's live outputs). Runs the full upstream sequence:
|
||||
// 1. deepen-or-store the negative (dedup radius 0.05).
|
||||
// 2. k-NN(4) positive centroid — or zeros when nothing is liked yet —
|
||||
// → push-away target → train toward it at lr * negLRRatio, gated by
|
||||
// the focus/solo mask. With no positives the push direction is
|
||||
// random per dim, which is upstream's cold start (there is no
|
||||
// separate negative-LR branch any more; see the body).
|
||||
// 3. proportional decay + eviction of expired negatives.
|
||||
// 2. immediately optimise all live negatives once. Subsequent updates
|
||||
// come from advance_geometric().
|
||||
template <typename M>
|
||||
FeedbackAction dislike_geometric(M& mlp, std::span<const float> current_out,
|
||||
float lr) noexcept {
|
||||
auto replay = replay_();
|
||||
const std::size_t n_out = this->n_out();
|
||||
|
||||
std::span<const float> a_neg = current_out.empty()
|
||||
? std::span<const float>(mlp.outputs())
|
||||
: current_out;
|
||||
|
|
@ -436,44 +443,46 @@ class FeedbackControllerCore : public FbStorage {
|
|||
// 1. store/deepen the negative (InterfaceRL.cpp:42-66).
|
||||
replay.deepen_or_store_negative(x_neg, a_neg);
|
||||
|
||||
const std::size_t pos_total = replay.positive_count();
|
||||
const std::size_t neg_total = replay.negative_count();
|
||||
const float avg_neg = replay.avg_negative_reward();
|
||||
|
||||
// 2+3. ONE path, as upstream (InterfaceRL.tpp:696-761). With no likes
|
||||
// stored there is no centroid to push away from, so the centroid is
|
||||
// zeros and `have_positives=false` sends every dim in a random
|
||||
// direction — upstream's `useRandom = !havePositives || ...`. We still
|
||||
// report GeometricColdStart so the caller can show its "like a few
|
||||
// sounds first" prompt, but the TRAINING is the same push at the same
|
||||
// LR either way. (Until 2026-07-25 the cold start instead trained AWAY
|
||||
// from the heard action at a NEGATIVE lr — a port of the older
|
||||
// `0a541cc` fallback that upstream has since replaced.)
|
||||
auto mean = this->centroid_buf();
|
||||
const bool have_positives = (pos_total > 0u);
|
||||
if (have_positives) {
|
||||
(void)replay.knn_positive_centroid(x_neg, kCentroidK, mean);
|
||||
} else {
|
||||
for (std::size_t j = 0; j < n_out; ++j) mean[j] = 0.f;
|
||||
const bool have_positives = replay.positive_count() > 0u;
|
||||
(void)optimise_geometric_once_(mlp, lr);
|
||||
if (!(geo_update_hz_ > 0.f) || !(geo_lifetime_ms_ > 0.f)) {
|
||||
replay.remove_all_negatives();
|
||||
}
|
||||
|
||||
auto target = this->target_buf();
|
||||
compute_push_target(a_neg.subspan(0, (a_neg.size() < n_out) ? a_neg.size() : n_out),
|
||||
std::span<const float>(mean.data(), n_out),
|
||||
focus_span_(), geo_push_step(avg_neg), have_positives,
|
||||
rng_, target);
|
||||
const float ratio = geo_neg_lr_ratio(neg_total, pos_total);
|
||||
mlp.train_targets(x_neg, std::span<const float>(target.data(), n_out),
|
||||
lr * ratio, focus_span_());
|
||||
|
||||
const FeedbackAction action = have_positives
|
||||
return have_positives
|
||||
? FeedbackAction::GeometricPush
|
||||
: FeedbackAction::GeometricColdStart;
|
||||
}
|
||||
|
||||
// 4. decay + evict expired negatives (InterfaceRL.cpp:752-760).
|
||||
replay.decay_negatives();
|
||||
// Advance the upstream-style replay optimiser by elapsed wall-clock time.
|
||||
// Returns the number of optimisation cycles applied. A long scheduler gap
|
||||
// does not create an unbounded catch-up burst: upstream also cannot execute
|
||||
// missed loop iterations while blocked. Ages still advance by the full dt.
|
||||
template <typename M>
|
||||
std::size_t advance_geometric(M& mlp, float dt_seconds) noexcept {
|
||||
if (mode_ != FeedbackMode::Avoid || avoid_style_ != AvoidStyle::Geometric ||
|
||||
!(dt_seconds > 0.f)) {
|
||||
return 0u;
|
||||
}
|
||||
auto replay = replay_();
|
||||
if (replay.negative_count() == 0u) {
|
||||
geo_step_accum_ = 0.f;
|
||||
return 0u;
|
||||
}
|
||||
|
||||
return action;
|
||||
constexpr float kMaxDoseDtSeconds = 0.1f;
|
||||
const float dose_dt = (dt_seconds < kMaxDoseDtSeconds)
|
||||
? dt_seconds
|
||||
: kMaxDoseDtSeconds;
|
||||
geo_step_accum_ += dose_dt * geo_update_hz_;
|
||||
std::size_t steps = static_cast<std::size_t>(geo_step_accum_);
|
||||
geo_step_accum_ -= static_cast<float>(steps);
|
||||
|
||||
for (std::size_t i = 0; i < steps; ++i) {
|
||||
if (!optimise_geometric_once_(mlp, geo_lr_)) break;
|
||||
}
|
||||
replay.advance_negative_ages(dt_seconds * 1000.f, geo_lifetime_ms_);
|
||||
if (replay.negative_count() == 0u) geo_step_accum_ = 0.f;
|
||||
return steps;
|
||||
}
|
||||
|
||||
// =========================================================================
|
||||
|
|
@ -661,8 +670,43 @@ class FeedbackControllerCore : public FbStorage {
|
|||
// The replay view over the storage-owned buffers.
|
||||
ReplayView replay_() noexcept {
|
||||
return ReplayView(this->replay_inputs(), this->replay_actions(),
|
||||
this->replay_rewards(), this->n_in(), this->n_out(),
|
||||
this->replay_cap(), replay_count_);
|
||||
this->replay_rewards(), this->replay_ages_ms(),
|
||||
this->n_in(), this->n_out(), this->replay_cap(),
|
||||
replay_count_);
|
||||
}
|
||||
|
||||
// One upstream-style optimise cycle over ALL live negatives. Unlike the
|
||||
// upstream cursor-coupled implementation, the positive centroid is looked
|
||||
// up at each negative's own stored input, so moving the cursor cannot
|
||||
// reinterpret an older rejection.
|
||||
template <typename M>
|
||||
bool optimise_geometric_once_(M& mlp, float lr) noexcept {
|
||||
auto replay = replay_();
|
||||
const std::size_t pos_total = replay.positive_count();
|
||||
const std::size_t neg_total = replay.negative_count();
|
||||
if (neg_total == 0u) return false;
|
||||
const float avg_neg = replay.avg_negative_reward();
|
||||
const std::size_t n_out = this->n_out();
|
||||
const float ratio = geo_neg_lr_ratio(neg_total, pos_total);
|
||||
|
||||
for (std::size_t i = 0; i < replay.size(); ++i) {
|
||||
if (replay.reward(i) > 0.f) continue;
|
||||
const auto x_neg = replay.input(i);
|
||||
const auto a_neg = replay.action(i);
|
||||
auto mean = this->centroid_buf();
|
||||
const bool have_positives =
|
||||
replay.knn_positive_centroid(x_neg, kCentroidK, mean) > 0u;
|
||||
if (!have_positives) {
|
||||
for (std::size_t j = 0; j < n_out; ++j) mean[j] = 0.f;
|
||||
}
|
||||
auto target = this->target_buf();
|
||||
compute_push_target(a_neg, std::span<const float>(mean.data(), n_out),
|
||||
focus_span_(), geo_push_step(avg_neg),
|
||||
have_positives, rng_, target);
|
||||
mlp.train_targets(x_neg, std::span<const float>(target.data(), n_out),
|
||||
lr * ratio, focus_span_());
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
// The focus mask as the geometric active-dims gate (empty ⇒ all active).
|
||||
|
|
@ -795,6 +839,9 @@ class FeedbackControllerCore : public FbStorage {
|
|||
// ---- Geometric dislike state ---------------------------------------------
|
||||
std::size_t replay_count_ = 0u;
|
||||
float geo_lr_ = 0.001f; // upstream InterfaceRL.hpp:312
|
||||
float geo_update_hz_ = 200.f; // upstream default optimise loop
|
||||
float geo_lifetime_ms_ = 2500.f; // upstream kDislikeLifetimeMs
|
||||
float geo_step_accum_ = 0.f;
|
||||
|
||||
// ---- ExploreAndPlace state ----------------------------------------------
|
||||
ExploreState ep_state_ = ExploreState::Idle;
|
||||
|
|
|
|||
|
|
@ -5,7 +5,7 @@
|
|||
// * `_perform_dislike_action()` (InterfaceRL.cpp:42-66) — nearby-negative
|
||||
// deepening within Euclidean 0.05, else store reward=-1.
|
||||
// * `optimise()` k-NN positive centroid (InterfaceRL.cpp:602-627).
|
||||
// * proportional negative decay + eviction (InterfaceRL.cpp:664, :752-760).
|
||||
// * wall-clock negative lifetime + eviction (current upstream e291192).
|
||||
//
|
||||
// STORAGE: the buffers live in the feedback controller's storage policy
|
||||
// (nisps/ml/feedback.hpp — fixed std::array on firmware, arena slice in the
|
||||
|
|
@ -29,22 +29,21 @@
|
|||
|
||||
namespace nisps::ml {
|
||||
|
||||
// Upstream constants (InterfaceRL.hpp:293-296, .cpp:42-66,664).
|
||||
// Upstream constants (InterfaceRL.hpp / InterfaceRL.tpp @ e291192).
|
||||
inline constexpr float kReplayDedupRadius = 0.05f;
|
||||
inline constexpr float kReplayDecayStep = 0.0025f;
|
||||
inline constexpr float kReplayEvictThreshold = -0.01f;
|
||||
inline constexpr float kMaxDislikeMagnitude = 16.f;
|
||||
inline constexpr std::size_t kCentroidK = 4u;
|
||||
|
||||
// A non-owning view over the replay buffers (inputs: cap×n_in, actions:
|
||||
// cap×n_out, rewards: cap) plus the live count. All methods deterministic,
|
||||
// cap×n_out, rewards/ages: cap) plus the live count. All methods deterministic,
|
||||
// allocation-free.
|
||||
class ReplayView {
|
||||
public:
|
||||
ReplayView(std::span<float> inputs, std::span<float> actions,
|
||||
std::span<float> rewards, std::size_t n_in, std::size_t n_out,
|
||||
std::size_t cap, std::size_t& count) noexcept
|
||||
: inputs_(inputs), actions_(actions), rewards_(rewards),
|
||||
std::span<float> rewards, std::span<float> ages_ms,
|
||||
std::size_t n_in, std::size_t n_out, std::size_t cap,
|
||||
std::size_t& count) noexcept
|
||||
: inputs_(inputs), actions_(actions), rewards_(rewards), ages_ms_(ages_ms),
|
||||
n_in_(n_in), n_out_(n_out), cap_(cap), count_(count) {}
|
||||
|
||||
std::size_t size() const noexcept { return count_; }
|
||||
|
|
@ -57,6 +56,7 @@ class ReplayView {
|
|||
return actions_.subspan(i * n_out_, n_out_);
|
||||
}
|
||||
float reward(std::size_t i) const noexcept { return rewards_[i]; }
|
||||
float age_ms(std::size_t i) const noexcept { return ages_ms_[i]; }
|
||||
|
||||
std::size_t positive_count() const noexcept {
|
||||
std::size_t n = 0u;
|
||||
|
|
@ -113,6 +113,7 @@ class ReplayView {
|
|||
auto act = actions_.subspan(i * n_out_, n_out_);
|
||||
const std::size_t n = (a.size() < n_out_) ? a.size() : n_out_;
|
||||
for (std::size_t j = 0; j < n; ++j) act[j] = a[j];
|
||||
ages_ms_[i] = 0.f;
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
|
@ -162,21 +163,21 @@ class ReplayView {
|
|||
return used;
|
||||
}
|
||||
|
||||
// Proportional decay of every negative (`reward += 0.0025 * max(|r|, 1)`)
|
||||
// and in-place eviction of items decayed past -0.01. Returns the number
|
||||
// evicted (the caller halves its dislike multiplier per expiry, matching
|
||||
// upstream InterfaceRL.cpp:752-760).
|
||||
std::size_t decay_negatives() noexcept {
|
||||
// Advance wall-clock age for every negative and remove those which have
|
||||
// lived their configured full-strength window. Positives do not expire.
|
||||
// Current upstream uses a timestamp and kDislikeLifetimeMs=2500; explicit
|
||||
// elapsed time keeps the core deterministic on firmware, native, and WASM.
|
||||
std::size_t advance_negative_ages(float dt_ms, float lifetime_ms) noexcept {
|
||||
if (!(dt_ms > 0.f) || !(lifetime_ms > 0.f)) return 0u;
|
||||
std::size_t evicted = 0u;
|
||||
std::size_t i = 0u;
|
||||
while (i < count_) {
|
||||
if (rewards_[i] <= 0.f) {
|
||||
const float mag = (rewards_[i] < 0.f) ? -rewards_[i] : rewards_[i];
|
||||
rewards_[i] += kReplayDecayStep * ((mag > 1.f) ? mag : 1.f);
|
||||
if (rewards_[i] > kReplayEvictThreshold) {
|
||||
ages_ms_[i] += dt_ms;
|
||||
if (ages_ms_[i] >= lifetime_ms) {
|
||||
evict_(i);
|
||||
++evicted;
|
||||
continue; // same index now holds the next item
|
||||
continue;
|
||||
}
|
||||
}
|
||||
++i;
|
||||
|
|
@ -184,6 +185,17 @@ class ReplayView {
|
|||
return evicted;
|
||||
}
|
||||
|
||||
void remove_all_negatives() noexcept {
|
||||
std::size_t i = 0u;
|
||||
while (i < count_) {
|
||||
if (rewards_[i] <= 0.f) {
|
||||
evict_(i);
|
||||
continue;
|
||||
}
|
||||
++i;
|
||||
}
|
||||
}
|
||||
|
||||
void clear() noexcept { count_ = 0u; }
|
||||
|
||||
private:
|
||||
|
|
@ -207,6 +219,7 @@ class ReplayView {
|
|||
for (std::size_t j = 0; j < n_in_; ++j) in[j] = (j < nx) ? x[j] : 0.f;
|
||||
for (std::size_t j = 0; j < n_out_; ++j) act[j] = (j < na) ? a[j] : 0.f;
|
||||
rewards_[slot] = reward;
|
||||
ages_ms_[slot] = 0.f;
|
||||
}
|
||||
|
||||
// Remove item i, shifting everything after it down one slot.
|
||||
|
|
@ -219,6 +232,7 @@ class ReplayView {
|
|||
auto src_act = actions_.subspan(m * n_out_, n_out_);
|
||||
for (std::size_t j = 0; j < n_out_; ++j) dst_act[j] = src_act[j];
|
||||
rewards_[m - 1u] = rewards_[m];
|
||||
ages_ms_[m - 1u] = ages_ms_[m];
|
||||
}
|
||||
--count_;
|
||||
}
|
||||
|
|
@ -226,6 +240,7 @@ class ReplayView {
|
|||
std::span<float> inputs_;
|
||||
std::span<float> actions_;
|
||||
std::span<float> rewards_;
|
||||
std::span<float> ages_ms_;
|
||||
std::size_t n_in_;
|
||||
std::size_t n_out_;
|
||||
std::size_t cap_;
|
||||
|
|
|
|||
|
|
@ -403,6 +403,9 @@ int nisps_ml_reshape(void* ml, int input_size, int output_size,
|
|||
|
||||
const auto feedback_mode = h->feedback.mode();
|
||||
const auto avoid_style = h->feedback.avoid_style();
|
||||
const float geo_lr = h->feedback.geo_lr();
|
||||
const float geo_update_hz = h->feedback.geo_update_hz();
|
||||
const float geo_lifetime_ms = h->feedback.geo_lifetime_ms();
|
||||
BrowserMLP fresh(h->seed64, d.n_in, std::span<const std::size_t>(d.hidden, 3u), d.n_out);
|
||||
if (!fresh.valid()) return 0;
|
||||
fresh.draw_weights(spread);
|
||||
|
|
@ -412,6 +415,9 @@ int nisps_ml_reshape(void* ml, int input_size, int output_size,
|
|||
kFeedbackUndoDepth, d.n_in, kFeedbackReplayCap);
|
||||
if (!fb.valid()) return 0;
|
||||
fb.set_avoid_style(avoid_style);
|
||||
fb.set_geo_lr(geo_lr);
|
||||
fb.set_geo_update_hz(geo_update_hz);
|
||||
fb.set_geo_lifetime_ms(geo_lifetime_ms);
|
||||
fb.set_mode(feedback_mode, fresh);
|
||||
|
||||
h->mlp = static_cast<BrowserMLP&&>(fresh);
|
||||
|
|
@ -627,10 +633,16 @@ void nisps_ml_feedback_reset(void* ml) {
|
|||
auto* h = static_cast<MLHandle*>(ml);
|
||||
const auto feedback_mode = h->feedback.mode();
|
||||
const auto avoid_style = h->feedback.avoid_style();
|
||||
const float geo_lr = h->feedback.geo_lr();
|
||||
const float geo_update_hz = h->feedback.geo_update_hz();
|
||||
const float geo_lifetime_ms = h->feedback.geo_lifetime_ms();
|
||||
BrowserFeedback fb(h->seed64 ^ kFeedbackSalt, h->n_out(), h->mlp.weight_count(),
|
||||
kFeedbackUndoDepth, h->n_in(), kFeedbackReplayCap);
|
||||
if (!fb.valid()) return;
|
||||
fb.set_avoid_style(avoid_style);
|
||||
fb.set_geo_lr(geo_lr);
|
||||
fb.set_geo_update_hz(geo_update_hz);
|
||||
fb.set_geo_lifetime_ms(geo_lifetime_ms);
|
||||
fb.set_mode(feedback_mode, h->mlp);
|
||||
h->feedback = static_cast<BrowserFeedback&&>(fb);
|
||||
h->feedback_static_scratch.assign(h->n_out(), 0.f);
|
||||
|
|
@ -817,6 +829,29 @@ int nisps_ml_feedback_dislike_geometric(void* ml, const float* current_out, floa
|
|||
return static_cast<int>(h->feedback.dislike_geometric(h->mlp, out, use_lr));
|
||||
}
|
||||
|
||||
// Configure the replay dose. Non-negative values are accepted; zero update
|
||||
// rate or lifetime gives one-shot press behaviour. Defaults are upstream:
|
||||
// 1e-3, 200 Hz, 2500 ms.
|
||||
EMSCRIPTEN_KEEPALIVE
|
||||
void nisps_ml_feedback_set_geometric_config(void* ml, float lr,
|
||||
float update_hz,
|
||||
float lifetime_ms) {
|
||||
if (!ml) return;
|
||||
auto* h = static_cast<MLHandle*>(ml);
|
||||
if (lr > 0.f) h->feedback.set_geo_lr(lr);
|
||||
h->feedback.set_geo_update_hz(update_hz);
|
||||
h->feedback.set_geo_lifetime_ms(lifetime_ms);
|
||||
}
|
||||
|
||||
// Deterministic elapsed-time seam. The host owns the clock; every weight-
|
||||
// affecting step remains in the shared C++ core. Returns optimise cycles run.
|
||||
EMSCRIPTEN_KEEPALIVE
|
||||
int nisps_ml_feedback_advance_geometric(void* ml, float dt_seconds) {
|
||||
if (!ml) return 0;
|
||||
auto* h = static_cast<MLHandle*>(ml);
|
||||
return static_cast<int>(h->feedback.advance_geometric(h->mlp, dt_seconds));
|
||||
}
|
||||
|
||||
// Store a positive (like) into the replay memory so the k-NN centroid sees
|
||||
// it. current_out may be null (live output used). The caller still runs its
|
||||
// usual addExample + train.
|
||||
|
|
|
|||
|
|
@ -56,6 +56,7 @@ EXPORTED_FUNCS='[
|
|||
"_nisps_ml_feedback_undo_depth",
|
||||
"_nisps_ml_feedback_placed_output",
|
||||
"_nisps_ml_feedback_dislike_geometric","_nisps_ml_feedback_store_positive",
|
||||
"_nisps_ml_feedback_set_geometric_config","_nisps_ml_feedback_advance_geometric",
|
||||
"_nisps_ml_feedback_positive_count","_nisps_ml_feedback_negative_count",
|
||||
"_nisps_ml_feedback_set_avoid_style",
|
||||
"_nisps_ml_jolt_press","_nisps_ml_jolt_step","_nisps_ml_jolt_release",
|
||||
|
|
|
|||
|
|
@ -281,10 +281,10 @@ int main(int argc, char** argv) {
|
|||
// ---- Stage 6: geometric dislike (one-core-engine P3) ----
|
||||
// Scripted feedback session — likes at two corners feed the replay
|
||||
// positives (via the Avoid+Geometric on_up path), then two dislikes at
|
||||
// a probed input: the first stores the negative and trains toward the
|
||||
// computed push-away target; the second deepens and pushes again. The
|
||||
// weight trajectory must match native↔WASM within 1e-5 (the useRandom
|
||||
// branch never fires here; the controller Rng is untouched).
|
||||
// a probed input. Eight 5ms driver ticks cover the parameterised
|
||||
// upstream-style 200 Hz replay seam over all live negatives. The weight
|
||||
// trajectory must match native↔WASM within 1e-5 (the useRandom branch
|
||||
// never fires here; the controller Rng is untouched).
|
||||
fb.set_mode(nisps::ml::FeedbackMode::Avoid, mlp);
|
||||
|
||||
auto like_at = [&](float x, float y) {
|
||||
|
|
@ -316,6 +316,9 @@ int main(int argc, char** argv) {
|
|||
};
|
||||
dislike_at(0.25f, 0.75f);
|
||||
dislike_at(0.26f, 0.74f); // within dedup radius → deepen + push
|
||||
for (int i = 0; i < 8; ++i) {
|
||||
fb.advance_geometric(mlp, 0.005f);
|
||||
}
|
||||
|
||||
payload.push_back(static_cast<float>(fb.positive_count()));
|
||||
payload.push_back(static_cast<float>(fb.negative_count()));
|
||||
|
|
|
|||
|
|
@ -104,6 +104,7 @@ function bind(Module) {
|
|||
feedbackLike: cwrap('nisps_ml_feedback_like', null, ['number']),
|
||||
feedbackCommitPlace: cwrap('nisps_ml_feedback_commit_place', null, ['number']),
|
||||
feedbackPlacedOutput: cwrap('nisps_ml_feedback_placed_output', 'number', ['number','number']),
|
||||
feedbackAdvanceGeometric: cwrap('nisps_ml_feedback_advance_geometric', 'number', ['number','number']),
|
||||
feedbackPositiveCount: cwrap('nisps_ml_feedback_positive_count', 'number', ['number']),
|
||||
feedbackNegativeCount: cwrap('nisps_ml_feedback_negative_count', 'number', ['number']),
|
||||
describe: cwrap('nisps_ml_describe', null, ['number','number']),
|
||||
|
|
@ -333,9 +334,9 @@ async function main() {
|
|||
|
||||
// --- Stage 6: geometric dislike (one-core-engine P3) ---
|
||||
// Mirrors parity_check.cpp stage 6: two likes feed the replay positives via
|
||||
// the Avoid+Geometric on_up path, then two dislikes (second deepens) train
|
||||
// toward the computed push-away target. f32 arithmetic for the "heard"
|
||||
// vector via Math.fround to match native float ops exactly.
|
||||
// the Avoid+Geometric on_up path, then two dislikes (second deepens) plus
|
||||
// eight 5ms replay ticks train toward the computed push-away target. f32
|
||||
// arithmetic for the "heard" vector via Math.fround matches native exactly.
|
||||
const FB_AVOID = 0;
|
||||
api.feedbackSetMode(ml, FB_AVOID);
|
||||
|
||||
|
|
@ -368,6 +369,7 @@ async function main() {
|
|||
};
|
||||
dislikeAt(0.25, 0.75);
|
||||
dislikeAt(0.26, 0.74); // within dedup radius: deepen + push
|
||||
for (let i = 0; i < 8; i++) api.feedbackAdvanceGeometric(ml, 0.005);
|
||||
|
||||
feedbackFloats.push(api.feedbackPositiveCount(ml));
|
||||
feedbackFloats.push(api.feedbackNegativeCount(ml));
|
||||
|
|
|
|||
|
|
@ -3,9 +3,8 @@
|
|||
//
|
||||
// Covers: replay dedup/deepen at radius 0.05, k-NN centroid selection with
|
||||
// deterministic index tie-break, push direction sign (target moves AWAY from
|
||||
// the liked centroid), taper, cold-start posMemCount==0 fallback, decay/
|
||||
// eviction + dislike-multiplier bookkeeping, solo/focus gating, and fixed-seed
|
||||
// determinism.
|
||||
// the liked centroid), cold-start posMemCount==0 fallback, parameterised replay
|
||||
// dose/lifetime, solo/focus gating, and fixed-seed determinism.
|
||||
|
||||
#include <array>
|
||||
#include <cmath>
|
||||
|
|
@ -36,10 +35,11 @@ struct RawReplay {
|
|||
std::array<float, kCap * kNIn> in{};
|
||||
std::array<float, kCap * kNOut> act{};
|
||||
std::array<float, kCap> rew{};
|
||||
std::array<float, kCap> age_ms{};
|
||||
std::size_t count = 0u;
|
||||
|
||||
ReplayView view() {
|
||||
return ReplayView(in, act, rew, kNIn, kNOut, kCap, count);
|
||||
return ReplayView(in, act, rew, age_ms, kNIn, kNOut, kCap, count);
|
||||
}
|
||||
};
|
||||
|
||||
|
|
@ -118,24 +118,27 @@ NISPS_TEST(replay_knn_centroid_deterministic_tie_break) {
|
|||
NISPS_EXPECT(used_after_neg == 3u);
|
||||
}
|
||||
|
||||
NISPS_TEST(replay_decay_and_evict) {
|
||||
NISPS_TEST(replay_wall_clock_lifetime_and_evict) {
|
||||
RawReplay raw;
|
||||
auto r = raw.view();
|
||||
const float x[kNIn] = {0.1f, 0.1f};
|
||||
const float a[kNOut] = {};
|
||||
// A shallow negative just above the evict threshold decays out in a few
|
||||
// calls; rewards move by +0.0025*max(|r|,1) per call.
|
||||
r.store(-0.012f, std::span<const float>(x), std::span<const float>(a));
|
||||
r.store(-1.f, std::span<const float>(x), std::span<const float>(a));
|
||||
NISPS_ASSERT(r.size() == 1u);
|
||||
std::size_t evicted = r.decay_negatives(); // -0.012 + 0.0025 = -0.0095 > -0.01 → evict
|
||||
std::size_t evicted = r.advance_negative_ages(2499.f, 2500.f);
|
||||
NISPS_EXPECT(evicted == 0u);
|
||||
NISPS_EXPECT(r.size() == 1u);
|
||||
NISPS_EXPECT_NEAR(r.age_ms(0), 2499.f, 1e-6);
|
||||
evicted = r.advance_negative_ages(1.f, 2500.f);
|
||||
NISPS_EXPECT(evicted == 1u);
|
||||
NISPS_EXPECT(r.size() == 0u);
|
||||
|
||||
// Positives are never decayed/evicted.
|
||||
// Positives do not age or expire.
|
||||
r.store(1.f, std::span<const float>(x), std::span<const float>(a));
|
||||
evicted = r.decay_negatives();
|
||||
evicted = r.advance_negative_ages(5000.f, 2500.f);
|
||||
NISPS_EXPECT(evicted == 0u);
|
||||
NISPS_EXPECT(r.size() == 1u);
|
||||
NISPS_EXPECT(r.age_ms(0) == 0.f);
|
||||
}
|
||||
|
||||
// -- compute_push_target --------------------------------------------------------
|
||||
|
|
@ -302,6 +305,38 @@ NISPS_TEST(geo_dislike_deterministic_under_fixed_seed) {
|
|||
}
|
||||
}
|
||||
|
||||
NISPS_TEST(geo_dislike_replay_rate_and_lifetime_are_parameterised) {
|
||||
GeoMLP m(88ull);
|
||||
m.draw_weights(0.5f);
|
||||
GeoFB fb(99ull);
|
||||
NISPS_EXPECT(fb.geo_lr() == 0.001f);
|
||||
NISPS_EXPECT(fb.geo_update_hz() == 200.f);
|
||||
NISPS_EXPECT(fb.geo_lifetime_ms() == 2500.f);
|
||||
|
||||
fb.set_geo_lr(0.002f);
|
||||
fb.set_geo_update_hz(20.f);
|
||||
fb.set_geo_lifetime_ms(100.f);
|
||||
set_inputs(m, 0.4f, 0.6f);
|
||||
m.process();
|
||||
fb.on_down(m, {}, 0.1f, 0.5f, {}); // one immediate update
|
||||
NISPS_ASSERT(fb.negative_count() == 1u);
|
||||
|
||||
// 20 Hz gives one replay update per 50 ms. At 100 ms the rejection
|
||||
// expires after its second scheduled update.
|
||||
NISPS_EXPECT(fb.advance_geometric(m, 0.025f) == 0u);
|
||||
NISPS_EXPECT(fb.advance_geometric(m, 0.025f) == 1u);
|
||||
NISPS_EXPECT(fb.negative_count() == 1u);
|
||||
NISPS_EXPECT(fb.advance_geometric(m, 0.05f) == 1u);
|
||||
NISPS_EXPECT(fb.negative_count() == 0u);
|
||||
NISPS_EXPECT(fb.advance_geometric(m, 0.05f) == 0u);
|
||||
|
||||
// Zero rate is an explicit one-shot mode: the press still updates once,
|
||||
// but no stale negative remains to affect a later press.
|
||||
fb.set_geo_update_hz(0.f);
|
||||
fb.on_down(m, {}, 0.1f, 0.5f, {});
|
||||
NISPS_EXPECT(fb.negative_count() == 0u);
|
||||
}
|
||||
|
||||
NISPS_TEST(geo_dislike_diffuse_style_preserves_legacy_path) {
|
||||
GeoMLP m(9ull);
|
||||
GeoFB fb(9ull);
|
||||
|
|
|
|||
Loading…
Reference in a new issue