Skip to content

feat: add experiment telemetry and safe exports - #11

Merged
christopherjnelson merged 1 commit into
mainfrom
feat/0006-experiment-telemetry-export
Aug 14, 2026
Merged

christopherjnelson merged 1 commit into
mainfrom
feat/0006-experiment-telemetry-export

Conversation

@christopherjnelson

Copy link
Copy Markdown
Member

Implementation

  • add a server-owned, runtime-validated in-memory experiment record with configurable 5,000-turn retention, independent from the existing 120-turn browser snapshot
  • normalize optional OpenRouter usage and actual charged cost while preserving known safe metadata through malformed model decisions
  • calculate aggregate and per-agent metrics, including known-versus-unknown costs, exact tiny-cost aggregation, tokens, latency, movement, infection, and unique-cell activity
  • add validated preview and generation APIs with agent, turn, outcome, and action filtering plus Minimal, Standard, Full safe, and Custom export levels
  • make compact JSON the AI-sharing default while supporting pretty JSON from the same schema-validated document
  • add World Lab usage meters, selected-agent usage, accessible export controls, copy/download feedback, reset telemetry warnings, and fully-infected playback auto-pause

Architecture and security

  • keep authoritative telemetry, filtering, metrics, and export construction in the loopback-only Game API
  • keep the deterministic world engine unaware of telemetry, HTTP, providers, and billing; the agent runtime only normalizes provider usage
  • retain the richest safe structured record once and filter it at export time without changing prompts or inference usage
  • exclude API keys, authorization headers, fixed system prompts, raw provider bodies, unbounded diagnostics, and private reasoning from records and exports
  • use export-specific state-only initialWorld and currentWorld snapshots containing all agents and hex states but no embedded event history
  • keep filtered top-level worldEvents as the single canonical event stream, respecting selected agents, turn ranges, outcomes, and actions
  • keep experiments memory-only; reset creates a new identity and clears telemetry/cost while preserving active personalities, and process restart loses the experiment

Testing

The repository owner completed the full local validation gate successfully:

node --version                         v24.18.0
pnpm --version                         11.21.0
pnpm install --frozen-lockfile         passed
pnpm format:check                      passed
pnpm lint                              passed
pnpm typecheck                         passed
pnpm test                              passed (125 tests)
pnpm build                             passed
pnpm exec playwright install chromium conditional; Chromium available
pnpm test:e2e                           passed

Coverage includes shared contracts, OpenRouter usage normalization and failure preservation, scripted zero-cost behavior, retention/reset/configuration history, filters and all export levels, subset metrics, concurrency/recovery, API validation and security exclusions, World Lab export interactions, fully infected playback behavior, and scripted-provider Playwright export/download validation.

Manual export inspection

  • inspected genuine OpenRouter usage/cost visibility and tiered export behavior
  • confirmed compact serialization materially reduces AI-sharing size and exact decimal aggregation avoids floating-point display artifacts
  • found and corrected the Full safe world-snapshot event duplication/scope issue
  • regression coverage proves a one-agent Full safe export has state-only world snapshots and only that agent's events in canonical worldEvents; all-agent Full safe exports remain supported
  • the inspected experiment had no personality edits, so its configurationEvents array was empty; deterministic tests cover custom edits and restore-default events

Documentation

README, roadmap, architecture, security, testing guidance, and ADR 0004 document retention boundaries, reset semantics, actual-known versus unknown cost, token estimation, security exclusions, state-only export snapshots, and the fully infected auto-pause safeguard.

@christopherjnelson
christopherjnelson marked this pull request as ready for review August 14, 2026 10:16
@christopherjnelson
christopherjnelson merged commit 085c46c into main Aug 14, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant