From ae37721072460687fc39fb2946289cd0374cf75c Mon Sep 17 00:00:00 2001 From: Christopher Nelson Date: Tue, 22 Sep 2026 14:09:07 -0400 Subject: [PATCH] fix(game-api): compute live swarm experiment metrics Live World Lab experiment metrics always read zero: the snapshot passed an empty resolved-action array to calculateExperimentMetrics even though the retained swarm ticks carry every committed action. Derive them instead through a new calculateRetainedExperimentMetrics, which uses the same resolved-action, provider-attempt and control-change derivation as an all-agents, entire-retained export, so live and exported metrics cannot drift apart. It scopes to the initial and current roster, as the export does, so a captured agent's history still counts. movementDirectionDistribution, longestRepeatedDirectionStreak and recentCellRevisits were declared in the metrics schema but never assigned after the legacy turn records were removed, so they always fell back to their defaults. Compute them from accepted agent-moved events, which already carry both cells, using the existing geographicDirectionBetweenCells classifier. The legacy implementation walked every scoped move as one sequence, so an aggregate streak ran across interleaved agents; each agent's path is now walked separately, with aggregates summing counts and revisits and reporting the longest single-agent streak. Both new tests fail against the previous code. Co-Authored-By: Claude Opus 5.5 --- ROADMAP.md | 26 ++-- apps/game-api/src/experiment-export.test.ts | 125 ++++++++++++++++++ apps/game-api/src/experiment-export.ts | 105 ++++++++++++++- .../src/simulation-service.swarm.test.ts | 44 ++++++ apps/game-api/src/simulation-service.ts | 13 +- docs/ARCHITECTURE.md | 8 +- docs/TESTING.md | 7 + 7 files changed, 303 insertions(+), 25 deletions(-) create mode 100644 apps/game-api/src/experiment-export.test.ts diff --git a/ROADMAP.md b/ROADMAP.md index cc3b097..182d408 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -30,6 +30,14 @@ architecture and removed all legacy infrastructure: brought README, this roadmap, architecture, gameplay foundation, security, testing, and the experiment-archive guide in line with the delivered architecture. See ADR 0033. +- **Live metrics fix** (`fix(game-api): compute live swarm experiment metrics`): + live World Lab metrics had read zero because the snapshot passed no resolved + actions to the metrics calculation; they now come from the retained swarm + ticks through the same derivation as an all-agents, entire-retained export. + The movement-pattern metrics (`movementDirectionDistribution`, + `longestRepeatedDirectionStreak`, `recentCellRevisits`), which nothing had + ever assigned since the migration, are computed per agent from accepted + moves. The retirement case is structural rather than measured: the legacy path made one full generative provider call per active agent per tick, so provider attempts, @@ -44,22 +52,6 @@ call) is retained as the ablation control for the swarm comparisons, not as a second production architecture. Historical milestones below remain as implementation history. -## Known open work - -The following issues are known and owned by a follow-on pull request: - -- `simulation-service.ts` passes an empty resolved-action array to - `calculateExperimentMetrics`, so all live World Lab experiment metrics read - zero even though `swarmTicks` now carries the data needed to populate them. -- `movementDirectionDistribution`, `longestRepeatedDirectionStreak`, and - `recentCellRevisits` are declared in the shared metrics schema but nothing - ever assigns them, so they always fall back to their schema defaults. The - direction helper itself already exists - (`geographicDirectionBetweenCells` in `apps/game-api/src/geographic-direction.ts`, - used by swarm pressure and reflex execution); what is missing is the - originating cell on the resolved-action record — which carries only the move - target — and the metric computation itself. - ## Agent Zero planner Agent Zero is the sole generative planner. It makes one OpenRouter call per @@ -182,6 +174,4 @@ operator safety boundary, and their schema-v12 safe ledger can be exported to the analysis archive even when no turn committed. This is not active runtime persistence, restart recovery, or provider-account balance enforcement. -This milestone also owns the two known open metrics defects noted above. - Player development begins only after these agent milestones demonstrate compelling behavior. diff --git a/apps/game-api/src/experiment-export.test.ts b/apps/game-api/src/experiment-export.test.ts new file mode 100644 index 0000000..c988e93 --- /dev/null +++ b/apps/game-api/src/experiment-export.test.ts @@ -0,0 +1,125 @@ +import { gridDisk, latLngToCell } from 'h3-js'; +import { describe, expect, it } from 'vitest'; +import { + agentIdSchema, + eventIdSchema, + h3CellSchema, + type AgentId, + type H3Cell, + type WorldAction, + type WorldActionResult, +} from '@hexzero/shared'; +import { calculateExperimentMetrics } from './experiment-export'; +import { geographicDirectionBetweenCells } from './geographic-direction'; + +const agentX = agentIdSchema.parse('00000000-0000-4000-8000-000000000001'); +const agentY = agentIdSchema.parse('00000000-0000-4000-8000-000000000002'); +const origin = h3CellSchema.parse(latLngToCell(41.6528, -83.5379, 9)); + +function neighbors(cell: H3Cell): H3Cell[] { + return gridDisk(cell, 1) + .filter((candidate) => candidate !== cell) + .map((candidate) => h3CellSchema.parse(candidate)); +} + +/** + * Finds origin -> a -> b where both outbound steps share one direction and + * both return steps share another. + */ +function straightLine() { + for (const a of neighbors(origin)) { + const direction = geographicDirectionBetweenCells(origin, a); + const opposite = geographicDirectionBetweenCells(a, origin); + const b = neighbors(a).find( + (candidate) => + candidate !== origin && + geographicDirectionBetweenCells(a, candidate) === direction && + geographicDirectionBetweenCells(candidate, a) === opposite, + ); + if (b) return { a, b, direction, opposite }; + } + throw new Error('No straight two-step H3 line from the origin.'); +} + +let eventCount = 0; +function moved(tickNumber: number, agentId: AgentId, from: H3Cell, to: H3Cell) { + eventCount += 1; + const action: WorldAction = { type: 'move', targetCell: to }; + const actionResult: WorldActionResult = { + accepted: true, + event: { + id: eventIdSchema.parse( + `00000000-0000-4000-8000-${String(eventCount).padStart(12, '0')}`, + ), + agentId, + occurredAt: '2026-08-13T12:00:00.000Z', + type: 'agent-moved', + fromCell: from, + toCell: to, + }, + }; + return { tickNumber, agentId, action, actionResult }; +} + +function rejectedMove(tickNumber: number, agentId: AgentId, to: H3Cell) { + const action: WorldAction = { type: 'move', targetCell: to }; + const actionResult: WorldActionResult = { + accepted: false, + reason: 'not-adjacent', + details: 'Target is not adjacent.', + }; + return { tickNumber, agentId, action, actionResult }; +} + +describe('movement-pattern metrics', () => { + it('walks each agent path separately for direction streaks and revisits', () => { + const { a, b, direction, opposite } = straightLine(); + const yTarget = neighbors(origin).find( + (cell) => geographicDirectionBetweenCells(origin, cell) === direction, + )!; + + const metrics = calculateExperimentMetrics( + [ + moved(1, agentX, origin, a), + // Agent Y moves in the same direction between X's two moves. A single + // interleaved walk would report a streak of three. + moved(1, agentY, origin, yTarget), + rejectedMove(2, agentX, origin), + moved(3, agentX, a, b), + moved(4, agentX, b, a), + moved(5, agentX, a, origin), + ], + [agentX, agentY], + ); + + const x = metrics.byAgent.find(({ agentId }) => agentId === agentX)!; + expect(x.metrics.longestRepeatedDirectionStreak).toBe(2); + expect(x.metrics.recentCellRevisits).toBe(2); + expect(x.metrics.movementDirectionDistribution).toEqual( + expect.arrayContaining([ + { direction, count: 2 }, + { direction: opposite, count: 2 }, + ]), + ); + + expect(metrics.aggregate.longestRepeatedDirectionStreak).toBe(2); + expect(metrics.aggregate.recentCellRevisits).toBe(2); + expect(metrics.aggregate.movementDirectionDistribution).toEqual( + expect.arrayContaining([ + { direction, count: 3 }, + { direction: opposite, count: 2 }, + ]), + ); + }); + + it('reports no movement pattern when no move was accepted', () => { + const metrics = calculateExperimentMetrics( + [rejectedMove(1, agentX, origin)], + [agentX], + ); + + expect(metrics.aggregate.movementDirectionDistribution).toEqual([]); + expect(metrics.aggregate.longestRepeatedDirectionStreak).toBe(0); + expect(metrics.aggregate.recentCellRevisits).toBe(0); + }); +}); diff --git a/apps/game-api/src/experiment-export.ts b/apps/game-api/src/experiment-export.ts index 58d3f41..2aad5cd 100644 --- a/apps/game-api/src/experiment-export.ts +++ b/apps/game-api/src/experiment-export.ts @@ -26,6 +26,10 @@ import { type WorldActionResult, type WorldSnapshot, } from '@hexzero/shared'; +import { + geographicDirectionBetweenCells, + type GeographicDirection, +} from './geographic-direction'; export interface ExperimentSource { schemaVersion: 12; @@ -411,7 +415,14 @@ function filterControlChanges( !request.actions.includes('capture') ) return []; - return requestFiltered.flatMap(({ tickNumber, actionResult }) => { + return capturesAffecting(requestFiltered, selected); +} + +function capturesAffecting( + actions: readonly ResolvedWorldAction[], + selected: Set, +): ExportedControlChange[] { + return actions.flatMap(({ tickNumber, actionResult }) => { if (!actionResult.accepted || actionResult.event.type !== 'hex-captured') return []; const event = actionResult.event; @@ -575,6 +586,64 @@ function attemptMetrics(attempts: readonly ProviderAttemptRecord[]) { }; } +const movementDirections: readonly GeographicDirection[] = [ + 'N', + 'NE', + 'SE', + 'S', + 'SW', + 'NW', +]; + +/** + * Direction streaks and revisits only mean something along one agent's own + * path, so each agent's accepted moves are walked separately. A scope spanning + * several agents sums direction counts and revisits and reports the longest + * single-agent streak. A revisit is an accepted move into any cell the agent + * has already occupied within the scope, including its first move's origin. + */ +function movementMetrics(scopeActions: readonly ResolvedWorldAction[]) { + const counts = new Map(); + const paths = new Map< + AgentId, + { + previous: GeographicDirection | null; + streak: number; + visited: Set; + } + >(); + let longestRepeatedDirectionStreak = 0; + let recentCellRevisits = 0; + for (const { agentId, actionResult } of scopeActions) { + if (!actionResult.accepted || actionResult.event.type !== 'agent-moved') + continue; + const { fromCell, toCell } = actionResult.event; + const direction = geographicDirectionBetweenCells(fromCell, toCell); + let path = paths.get(agentId); + if (!path) { + path = { previous: null, streak: 0, visited: new Set([fromCell]) }; + paths.set(agentId, path); + } + counts.set(direction, (counts.get(direction) ?? 0) + 1); + path.streak = direction === path.previous ? path.streak + 1 : 1; + path.previous = direction; + longestRepeatedDirectionStreak = Math.max( + longestRepeatedDirectionStreak, + path.streak, + ); + if (path.visited.has(toCell)) recentCellRevisits += 1; + path.visited.add(toCell); + } + return { + movementDirectionDistribution: movementDirections.flatMap((direction) => { + const count = counts.get(direction); + return count ? [{ direction, count }] : []; + }), + longestRepeatedDirectionStreak, + recentCellRevisits, + }; +} + function metricCountsFor( scopeActions: readonly ResolvedWorldAction[], scopeAttempts: readonly ProviderAttemptRecord[], @@ -654,6 +723,7 @@ function metricCountsFor( territoryGainedThroughCapture, territoryLostThroughCapture, uniqueVisitedCells: visited.size, + ...movementMetrics(scopeActions), ...attemptMetrics(scopeAttempts), }; } @@ -684,6 +754,39 @@ export function calculateExperimentMetrics( }); } +/** + * Metrics over every known agent and the entire retained experiment: the same + * values an all-agents, entire-retained export reports, so live World Lab + * metrics cannot drift from exported ones. + */ +export function calculateRetainedExperimentMetrics( + source: Pick< + ExperimentSource, + | 'swarmTicks' + | 'scenario' + | 'initialAgents' + | 'currentAgents' + | 'providerAttempts' + >, +): ExperimentMetrics { + const agentIds = [ + ...new Set( + [...source.initialAgents, ...source.currentAgents].map(({ id }) => id), + ), + ]; + const selected = new Set(agentIds); + const resolved = resolvedActionsFromTicks( + source.swarmTicks, + source.scenario.patientZeroAgentId, + ); + return calculateExperimentMetrics( + resolved.filter(({ agentId }) => selected.has(agentId)), + agentIds, + source.providerAttempts.filter(({ agentId }) => selected.has(agentId)), + capturesAffecting(resolved, selected), + ); +} + export function serializeExperimentExport( document: ExperimentExportDocument, ): string { diff --git a/apps/game-api/src/simulation-service.swarm.test.ts b/apps/game-api/src/simulation-service.swarm.test.ts index 3ff3b8d..5ecf13d 100644 --- a/apps/game-api/src/simulation-service.swarm.test.ts +++ b/apps/game-api/src/simulation-service.swarm.test.ts @@ -633,6 +633,50 @@ describe('zero-swarm SimulationService tick', () => { expect(tick.swarmTick?.tickNumber).toBe(1); }); + it('reports live experiment metrics equal to an all-agents entire-retained export', async () => { + const simulation = setup( + new InspectingPlanner(), + new ScriptedReflexProvider( + Array.from({ length: 21 }, () => ({ chosenCandidateId: 'action_0' })), + ), + ); + + await simulation.executeNextTick(); + await simulation.executeNextTick(); + await simulation.executeNextTick(); + + const snapshot = simulation.getSnapshot(); + const resolvedActionCount = (snapshot.swarmTicks ?? []).reduce( + (count, tick) => + count + + (tick.zeroAction && tick.zeroActionResult ? 1 : 0) + + tick.workers.filter( + ({ action, actionResult }) => action && actionResult, + ).length, + 0, + ); + expect(resolvedActionCount).toBeGreaterThan(0); + expect(snapshot.experiment.metrics.aggregate.totalTurns).toBe( + resolvedActionCount, + ); + + const exported = simulation.generateExperimentExport({ + agents: { mode: 'all' }, + turns: { mode: 'entire-retained' }, + outcomes: [ + 'accepted', + 'rejected', + 'lost-tick', + 'provider-error', + 'operator-skipped', + ], + actions: ['move', 'infect', 'capture', 'wait'], + level: 'full-safe', + serialization: 'compact', + }); + expect(snapshot.experiment.metrics).toEqual(exported.metrics); + }); + it('freezes player-advanced facts for Zero, uses only reflex choices, and resolves physical actions in engine order', async () => { const planner = new InspectingPlanner(); const simulation = setup( diff --git a/apps/game-api/src/simulation-service.ts b/apps/game-api/src/simulation-service.ts index cd753e9..d4c8b7c 100644 --- a/apps/game-api/src/simulation-service.ts +++ b/apps/game-api/src/simulation-service.ts @@ -64,7 +64,7 @@ import { type WorldState, } from '@hexzero/world-engine'; import { - calculateExperimentMetrics, + calculateRetainedExperimentMetrics, createExperimentExport, createExperimentPreview, type ExperimentSource, @@ -417,10 +417,13 @@ export class SimulationService { id: this.#experimentId, startedAt: this.#experimentStartedAt, attemptAccounting: this.#attemptAccounting.snapshot(), - metrics: calculateExperimentMetrics( - [], - agents.map(({ id }) => id), - ), + metrics: calculateRetainedExperimentMetrics({ + swarmTicks: this.#experimentSwarmTicks, + scenario: this.#scenario, + initialAgents: this.#initialExperimentAgents, + currentAgents: agents, + providerAttempts: this.#attemptAccounting.ledger(), + }), currentTerritory: this.#territoryScoreboard(), simulatedPlayerMetrics: this.#state.simulatedPlayer?.metrics ?? { movements: 0, diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index d0cc2ba..96542de 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -261,7 +261,13 @@ archive never becomes simulation authority. The active experiment has a runtime-validated UUID, start time, versioned authoritative scenario and ordered initial roster, immutable configuration events, initial world, and up to 5,000 complete safe turns. The browser snapshot and world-event list remain capped at 120. Reset creates a new experiment from the current scenario and clears telemetry/cost; no previous experiments survive reset or process restart. -Metrics and filtering are deterministic Game API responsibilities. All exports +Metrics and filtering are deterministic Game API responsibilities. The live +snapshot's experiment metrics use the same derivation as an all-agents, +entire-retained export over the retained swarm ticks, so World Lab and exported +metrics cannot drift apart. Movement-pattern metrics walk each agent's accepted +moves separately, classifying each step with `geographicDirectionBetweenCells`; +aggregates sum direction counts and revisits and report the longest +single-agent streak. All exports use schema version 12, which carries `swarmArchitectureVersion: "zero-swarm-v1"` and independent provider-attempt accounting unconditionally. Pre-swarm exports (schema versions 9, 10, and 11) are rejected outright; there is no migration diff --git a/docs/TESTING.md b/docs/TESTING.md index 5720075..3b67b3c 100644 --- a/docs/TESTING.md +++ b/docs/TESTING.md @@ -154,6 +154,13 @@ swarm summary showing no player pressure and fallback actions. boundaries with north wraparound, initial bearings across longitude wraparound, and same-cell/same-coordinate rejection. +`experiment-export.test.ts` covers movement-pattern metrics: direction streaks +and cell revisits walk each agent's accepted moves separately, so an interleaved +move by another agent neither extends nor breaks a streak; aggregates sum +direction counts and revisits and report the longest single-agent streak. +`simulation-service.swarm.test.ts` additionally asserts that live snapshot +metrics equal the metrics of an all-agents, entire-retained export. + ### World Lab (`apps/world-lab`) `swarm-view.test.tsx` covers swarm telemetry panels: inactive pressure/action/