validation(user-testing): skeleton-wasm milestone — all 6 assertions passed

User testing validation for skeleton-wasm milestone. Tested 6 assertions
via agent-browser flow validators: COOP/COEP headers (VAL-PROJ-001),
SolidJS rendering (VAL-PROJ-002), WASM engine loading (VAL-PROJ-003),
signal bus emit/receive/wildcard (VAL-PROJ-004), debug probe with
?debug=1 (VAL-PROJ-005), debug probe absent without ?debug=1 (VAL-PROJ-006).
Added Flow Validator Guidance section to user-testing.md.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
This commit is contained in:
monkey-w1n5t0n 2026-04-07 17:10:24 +02:00
parent 4db11849e1
commit 447ab54b54
4 changed files with 295 additions and 0 deletions

View file

@ -72,3 +72,31 @@ These assertions require physical hardware and cannot be validated in headless b
- Max concurrent: 5 validators (5 × 500MB = 2.5GB, well within budget)
**Playwright tests**: Run serially (single browser, single server)
---
## Flow Validator Guidance: Browser
**Surface**: Browser-based web application served by Vite dev server
**Isolation rules**:
- Each validator gets its own agent-browser session
- All validators share the same Vite dev server at http://localhost:5174
- localStorage is per-browser-profile — since each agent-browser session is independent, there's no state leakage between validators
- The debug probe (`window.__nisps`) is the primary testing interface; use JavaScript evaluation to call probe methods
- The app state is ephemeral per page load — clearing localStorage or reloading resets state
**Shared state concerns**:
- The Vite dev server is shared — if one validator causes a crash, it affects all validators
- No database or server-side state to worry about
- WASM binary is loaded fresh per page load
**Resource constraints**:
- Max 5 concurrent agent-browser sessions
- Each session: ~300MB RAM + browser process
- Keep screenshots minimal to avoid disk pressure
**Navigation patterns**:
- Load app with `?debug=1` to enable debug probe: `http://localhost:5174/?debug=1`
- Load without query params for production behavior: `http://localhost:5174`
- Use `localStorage.clear()` before tests requiring fresh state

View file

@ -0,0 +1,124 @@
{
"groupId": "group-1",
"testedAt": "2026-04-07T00:00:00Z",
"isolation": {
"appUrl": "http://localhost:5174/?debug=1",
"session": "9776ea6c39e9__g1",
"localStorageCleared": true
},
"toolsUsed": ["agent-browser"],
"assertions": [
{
"id": "VAL-PROJ-001",
"title": "Vite dev server starts with COOP/COEP headers",
"status": "pass",
"steps": [
{
"action": "Navigate to http://localhost:5174/?debug=1",
"expected": "Page loads without errors",
"observed": "Page loaded successfully with title 'MEMLNaut NISPS'"
},
{
"action": "Check console errors",
"expected": "No JavaScript errors",
"observed": "No console errors"
},
{
"action": "Check SharedArrayBuffer availability via typeof SharedArrayBuffer !== 'undefined'",
"expected": "true (SharedArrayBuffer available)",
"observed": "true"
},
{
"action": "Check window.crossOriginIsolated",
"expected": "true (cross-origin isolation active)",
"observed": "true"
},
{
"action": "Fetch '/' and check response headers",
"expected": "COOP: same-origin, COEP: require-corp",
"observed": "COOP: same-origin, COEP: require-corp"
}
],
"evidence": {
"screenshots": ["skeleton-wasm/group-1/VAL-PROJ-001-app-loaded.png"],
"consoleErrors": "none",
"network": "GET / -> 200, headers: {coop: 'same-origin', coep: 'require-corp'}"
},
"issues": null
},
{
"id": "VAL-PROJ-002",
"title": "SolidJS app renders without errors",
"status": "pass",
"steps": [
{
"action": "Clear localStorage and set nisps-help-seen, reload with ?debug=1",
"expected": "Page loads fresh without help modal",
"observed": "Page loaded cleanly, no help modal"
},
{
"action": "Wait for page to fully load and WASM to initialize",
"expected": "window.__nisps becomes available",
"observed": "window.__nisps available after ~3s"
},
{
"action": "Check DOM for heading and status",
"expected": "Heading 'MEMLNaut NISPS' and status 'Ready — 126 outputs loaded'",
"observed": "heading 'MEMLNaut NISPS' [ref=e1], paragraph 'Ready — 126 outputs loaded'"
},
{
"action": "Check console errors",
"expected": "No JavaScript errors",
"observed": "No console errors"
},
{
"action": "Take screenshot",
"expected": "App renders with immersive layout visible",
"observed": "Screenshot shows app rendered correctly with heading and status text"
}
],
"evidence": {
"screenshots": ["skeleton-wasm/group-1/VAL-PROJ-002-app-rendered.png"],
"consoleErrors": "none"
},
"issues": null
},
{
"id": "VAL-PROJ-003",
"title": "WASM ML engine loads",
"status": "pass",
"steps": [
{
"action": "Wait for WASM initialization (window.__nisps !== undefined)",
"expected": "Debug probe available",
"observed": "window.__nisps defined and available"
},
{
"action": "Call window.__nisps.getOutputs() and check length, bounds",
"expected": "Float32Array length 126, all values in [0,1]",
"observed": "length=126, min=2.066e-13, max=1, allBounded=true, sample=[0.300, 0.835, 0.688, 8.96e-8, 0.9999]"
},
{
"action": "Check console errors",
"expected": "No JavaScript errors",
"observed": "No console errors"
}
],
"evidence": {
"screenshots": ["skeleton-wasm/group-1/VAL-PROJ-003-wasm-loaded.png"],
"consoleErrors": "none",
"probeOutput": {
"length": 126,
"min": "2.066e-13",
"max": 1,
"allBounded": true,
"sample": [0.30002, 0.83461, 0.68781, 8.96e-8, 0.99994]
}
},
"issues": null
}
],
"frictions": [],
"blockers": [],
"summary": "Tested 3 assertions: all 3 passed. VAL-PROJ-001: Vite dev server correctly serves COOP/COEP headers with SharedArrayBuffer available and crossOriginIsolated=true. VAL-PROJ-002: SolidJS app renders without errors showing 'MEMLNaut NISPS' heading and 'Ready — 126 outputs loaded' status. VAL-PROJ-003: WASM ML engine loads successfully, debug probe returns 126 bounded outputs in [0,1]."
}

View file

@ -0,0 +1,114 @@
{
"groupId": "group-2",
"testedAt": "2026-04-07T00:00:00Z",
"isolation": {
"appUrl": "http://localhost:5174",
"session": "9776ea6c39e9__g2",
"localStorageClearedBeforeTests": true
},
"toolsUsed": ["agent-browser"],
"assertions": [
{
"id": "VAL-PROJ-004",
"title": "Signal bus creates topics and emits/receives",
"status": "pass",
"steps": [
{
"action": "Navigate to http://localhost:5174/?debug=1, clear localStorage, reload",
"expected": "Page loads without errors, __nispsBus available",
"observed": "Page loaded successfully, window.__nispsBus is an object with createTopic, getTopic, match methods"
},
{
"action": "Create topics test.a and test.b, subscribe to each, emit values 42 and 99",
"expected": "Subscribers receive emitted values",
"observed": "topicResults = [{sub:'a',data:42},{sub:'b',data:99},{sub:'a',data:100},{sub:'b',data:200}] — subscribers received all emitted values"
},
{
"action": "Subscribe to wildcard pattern 'test.*' via bus.match(), then emit 100 and 200 to test.a and test.b",
"expected": "Wildcard subscriber captures emits after match() subscription",
"observed": "wildcardResults = [100, 200, 1, 1] — captured emits from test.a, test.b, and test.same matching test.*"
},
{
"action": "Subscribe to test.same, emit 1 twice",
"expected": "sameDataCount is 2 (equals:false ensures every emit triggers subscriber)",
"observed": "sameDataCount = 2 — both emits triggered the subscriber even with identical data"
}
],
"evidence": {
"screenshots": [
"skeleton-wasm/group-2/VAL-PROJ-004-initial-load.png",
"skeleton-wasm/group-2/VAL-PROJ-004-after-test.png"
],
"consoleErrors": "none",
"javascriptOutput": "{\"topicResults\":[{\"sub\":\"a\",\"data\":42},{\"sub\":\"b\",\"data\":99},{\"sub\":\"a\",\"data\":100},{\"sub\":\"b\",\"data\":200}],\"wildcardResults\":[100,200,1,1],\"sameDataCount\":2}"
},
"issues": null
},
{
"id": "VAL-PROJ-005",
"title": "Debug probe exposed with ?debug=1",
"status": "pass",
"steps": [
{
"action": "Navigate to http://localhost:5174/?debug=1 (already loaded from VAL-PROJ-004)",
"expected": "window.__nisps is an object",
"observed": "window.__nisps exists and is an object (not null)"
},
{
"action": "Check typeof for all 15 required methods",
"expected": "All methods are functions",
"observed": "All 15 methods present: getOutputs, getLoss, getWeights, getExampleCount, setInputs, thumbsUp, thumbsDown, train, trainAsync, randomise, clearExamples, saveState, evalLoss, inferBatch, getLayerStats — allPresent: true"
}
],
"evidence": {
"screenshots": [
"skeleton-wasm/group-2/VAL-PROJ-005-debug-probe-present.png"
],
"consoleErrors": "none",
"javascriptOutput": "{\"exists\":true,\"methods\":[{\"name\":\"getOutputs\",\"exists\":true},{\"name\":\"getLoss\",\"exists\":true},{\"name\":\"getWeights\",\"exists\":true},{\"name\":\"getExampleCount\",\"exists\":true},{\"name\":\"setInputs\",\"exists\":true},{\"name\":\"thumbsUp\",\"exists\":true},{\"name\":\"thumbsDown\",\"exists\":true},{\"name\":\"train\",\"exists\":true},{\"name\":\"trainAsync\",\"exists\":true},{\"name\":\"randomise\",\"exists\":true},{\"name\":\"clearExamples\",\"exists\":true},{\"name\":\"saveState\",\"exists\":true},{\"name\":\"evalLoss\",\"exists\":true},{\"name\":\"inferBatch\",\"exists\":true},{\"name\":\"getLayerStats\",\"exists\":true}],\"allPresent\":true}"
},
"issues": null
},
{
"id": "VAL-PROJ-006",
"title": "Debug probe NOT exposed without ?debug=1",
"status": "pass",
"steps": [
{
"action": "Navigate to http://localhost:5174/ (without ?debug=1)",
"expected": "Page loads successfully",
"observed": "Page loaded without errors"
},
{
"action": "Check typeof window.__nisps",
"expected": "typeof window.__nisps === 'undefined'",
"observed": "nispsType: 'undefined', isUndefined: true"
}
],
"evidence": {
"screenshots": [
"skeleton-wasm/group-2/VAL-PROJ-006-no-debug-probe.png"
],
"consoleErrors": "none",
"javascriptOutput": "{\"nispsType\":\"undefined\",\"isUndefined\":true}"
},
"issues": null
}
],
"frictions": [
{
"description": "The validation contract task description referenced bus.topic() API but the actual implementation uses bus.createTopic()/getTopic(). The match() method takes (pattern, callback) directly rather than returning a subscribable object. Required reading the source code to discover the correct API.",
"resolved": true,
"resolution": "Read src/bus/signal-bus.ts to identify the actual API: createTopic(name), getTopic(name), match(pattern, fn)",
"affectedAssertions": ["VAL-PROJ-004"]
},
{
"description": "Arrow function callbacks in eval() caused 'ms.fn is not a function' error on first attempt after page reload with cleared localStorage. Switching to function declarations resolved it, suggesting a possible serialization issue with arrow functions in the browser eval context.",
"resolved": true,
"resolution": "Used function declarations instead of arrow functions in eval() JavaScript code",
"affectedAssertions": ["VAL-PROJ-004"]
}
],
"blockers": [],
"summary": "Tested 3 assertions: all 3 passed. VAL-PROJ-004 (signal bus): topics emit/receive correctly, wildcard matching works, equals:false ensures duplicate data triggers subscribers. VAL-PROJ-005 (debug probe with ?debug=1): all 15 required methods present. VAL-PROJ-006 (no probe without ?debug=1): window.__nisps is undefined."
}

View file

@ -0,0 +1,29 @@
{
"milestone": "skeleton-wasm",
"round": 1,
"status": "pass",
"assertionsSummary": {
"total": 6,
"passed": 6,
"failed": 0,
"blocked": 0
},
"passedAssertions": [
"VAL-PROJ-001",
"VAL-PROJ-002",
"VAL-PROJ-003",
"VAL-PROJ-004",
"VAL-PROJ-005",
"VAL-PROJ-006"
],
"failedAssertions": [],
"blockedAssertions": [],
"appliedUpdates": [
{
"target": "user-testing.md",
"description": "Added 'Flow Validator Guidance: Browser' section with isolation rules, shared state concerns, resource constraints, and navigation patterns for agent-browser flow validators.",
"source": "setup"
}
],
"previousRound": null
}