diff --git a/README.md b/README.md index 7396a9ea..93b4c1f3 100644 --- a/README.md +++ b/README.md @@ -54,6 +54,10 @@ · Open public reviewer demo · + RPG evaluation lab + · + Open deployed evaluation lab + · Open MP4 demo · Inspect source evidence @@ -66,6 +70,14 @@ source, tests, reviewer run, and system map. The public reviewer demo is a deterministic GitHub Pages app path: no API key, no live backend, and no generation claim beyond the committed source evidence. +The [RPG Runtime Evaluation Lab](./docs/RPG_EVALUATION_LAB.md) adds a portable, +backend-free reviewer surface for comparing another RPG agent's memory, +consequences, agency, trajectory, entities, choices, and player-facing boundary +hygiene through `rpg_evaluation_bundle.v1`. It is a deterministic diagnostic, +not a calibrated academic metric. + +Public deployment: + --- ## Reviewer Run diff --git a/README.zh.md b/README.zh.md index 17a100aa..bb302765 100644 --- a/README.zh.md +++ b/README.zh.md @@ -50,10 +50,17 @@ - [观看 75 秒 demo](https://youtu.be/RRJ7uyjW_nA) - [打开 MP4 备份](./docs/demo-video/tiny-stories-admissions-demo-readme.mp4) - [公开链接检查通过后再打开 GitHub Pages 展示页](https://lishehao.github.io/RPG_Demo/) +- [查看 RPG Runtime Evaluation Lab 接入规范](./docs/RPG_EVALUATION_LAB.md) +- [打开已部署的 RPG Runtime Evaluation Lab](https://tiny-stories-rpg-evaluation.vercel.app) 视频用于快速理解玩家看到的 loop;真正的申请材料证据仍然在 source、tests、 `#/portfolio`、`#/reviewer`、system map 和 engineering evidence packet 里. +RPG Runtime Evaluation Lab 是一个不依赖后端的公开评测前端。其他 RPG Agent +只需输出 `rpg_evaluation_bundle.v1`,即可用同一界面检查记忆连续性、后果可见性、 +玩家能动性、目标推进、人物一致性、选择多样性和边界卫生。该分数是确定性产品诊断, +不是已经校准的学术质量指标。 + 如果要把公开链接发给招生、推荐人或招聘 reviewer,先运行公开链接检查: `python3 tools/portfolio_public_evidence_preflight.py`.如果它提示本地 `HEAD` 领先 `origin/main`,或 GitHub Pages 缺少当前 marker,这些证据只能算 diff --git a/docs/RESEARCH_RUNTIME_BRIEF.md b/docs/RESEARCH_RUNTIME_BRIEF.md new file mode 100644 index 00000000..7a1866fd --- /dev/null +++ b/docs/RESEARCH_RUNTIME_BRIEF.md @@ -0,0 +1,97 @@ +# Tiny Stories Research Runtime Brief + +## Decision + +Tiny Stories is scoped as a research-grade technical product: + +> A controllable and evaluated LLM interactive-narrative runtime, delivered as +> a research specification, a runnable reference implementation, and a +> repeatable evaluation harness. + +Interactive storytelling is the experimental carrier. The project does not +claim consumer-market validation, general narrative robustness, or a novel +foundation model. + +## Primary Question + +How can an underspecified, revisable user intent be compiled into a bounded, +stateful, playable, recoverable, and evaluable interactive narrative under +stochastic LLM outputs? + +## Approved Defaults + +1. Primary audience: applied-AI R&D, product-systems, and graduate-project reviewers. +2. Secondary audience: interactive-narrative creators and bounded test participants. +3. Public delivery: static evidence and evaluation UI, with invite-controlled or quota-bounded live runs. +4. Content boundary: a small set of modern social-pressure, backstage, workplace, and mystery scenarios. +5. Evidence: deterministic fixtures, bounded live-provider acceptance, structured judges, and small formative human review. +6. Provider strategy: one primary live provider with only bounded comparison runs. +7. Post-research direction: freeze evidence first; reopen consumer beta or creator SDK only if the research reveals a clear opportunity. + +## In Scope + +- arbitrary-input Story Butler routing and compressed context; +- corrections and superseded-fact handling; +- Story Brief compilation and opening fidelity; +- a 12-turn playable runtime with explicit goals, pressure, people, clues, forecasts, and impacts; +- bounded failure, retry, repair, fallback, idempotency, and persistence; +- reviewer-only telemetry and deterministic/LLM-assisted evaluation; +- Chinese/English and desktop/mobile robustness; +- a portable RPG evaluation bundle and public evaluation frontend. + +## Explicit Non-goals + +- open UGC marketplace; +- deep emotional-companion positioning; +- adult visual content; +- subscriptions, growth loops, or consumer retention claims; +- multiplayer/social graph; +- production-scale multi-tenant infrastructure; +- calibrated academic metrics or SOTA claims. + +## Completion Evidence + +The research artifact is complete only when a third party can: + +1. understand the claim and its limits; +2. inspect a deterministic success and failure case; +3. run a bounded live path with real telemetry; +4. inspect memory, state, consequences, and evaluator evidence; +5. reproduce the tests and static public demo without secrets. + +## Reference Data Path + +```text +arbitrary input + -> intent-aware Story Butler + -> bounded current-truth memory + -> Story Brief + -> persisted template research context + -> live opening + -> 12-turn Action Runtime + -> ending + -> portable session bundle + -> deterministic evaluator + Reviewer evidence +``` + +Corrections are first-class events: the replacement becomes active truth while +the old value remains only in `superseded_facts`. Play and ending prompts use a +sanitized Story Contract rather than the unbounded Create transcript. The +portable evaluator receives explicit terminal-turn and progress-provenance +fields so it can distinguish runtime facts from turn-budget proxies. + +The public Vercel lab is deliberately static and backend-free. Live acceptance +is proven separately through a configured local gateway, strict telemetry, and +a 12-turn golden-path artifact. This division keeps the demo reproducible while +preventing a static portfolio surface from being mistaken for live-provider +evidence. + +## Current Reference Acceptance + +As of 2026-08-24, the strict reference run completes a corrected Story Butler +conversation, Story Brief, live opening, 12 live Play turns, ending, persisted +session export, Reviewer evidence, and portable evaluation with no required +operation falling back. English and Chinese bounded runs verify that current +facts, superseded corrections, story language, and responsive UI state survive +the same chain. The full records live under `docs/evidence/` and intentionally +retain latency, token, source, fallback, provenance, and consistency caveats. diff --git a/docs/RPG_EVALUATION_LAB.md b/docs/RPG_EVALUATION_LAB.md new file mode 100644 index 00000000..df13affd --- /dev/null +++ b/docs/RPG_EVALUATION_LAB.md @@ -0,0 +1,201 @@ +# RPG Runtime Evaluation Lab + +## Purpose + +The RPG Runtime Evaluation Lab is a backend-free reviewer frontend for +comparing stateful RPG-agent runs. It is intentionally broader than Tiny +Stories: another RPG can export one portable JSON bundle and use the same +memory, consequence, agency, trajectory, entity, choice, and boundary checks. + +Public route: + +```text +#/lab/rpg-evaluation +``` + +Public deployment: + +The lab is a product reliability diagnostic. Its score is deterministic and +inspectable, but it is not a calibrated academic quality metric. Narrative +appeal and emotional quality still require bounded human review. + +## Portable Contract + +The input root is `rpg_evaluation_bundle.v1`: + +```json +{ + "schema_version": "rpg_evaluation_bundle.v1", + "run_id": "run-123", + "system_label": "Candidate runtime", + "locale": "en", + "scenario": { + "scenario_id": "backstage-01", + "title": "Vanished Before Curtain", + "genre": "social-pressure mystery", + "objective": "Find the singer before air.", + "turn_budget": 12, + "entity_ids": ["singer", "producer", "publicist"], + "boundaries": ["No violence"] + }, + "turns": [] +} +``` + +Each turn contains: + +- the player action; +- the player-visible world response; +- the real next-action set; +- typed state deltas; +- clue and opportunity unlocks; +- referenced entity ids; +- bounded objective progress; +- one `rpg_memory.v1` snapshot. + +The frontend deliberately does not accept raw prompts, private reasoning, +provider credentials, or tool arguments. + +## Memory Contract + +`rpg_backend.research_runtime.memory` reduces portable events into a bounded, +auditable snapshot: + +- active facts have one current value per namespace/key; +- corrections move the previous value to `superseded_facts`; +- non-story input is retained as conversational evidence but never becomes a + story fact; +- open threads can be opened and resolved; +- person state is merged by entity id; +- recent events and active facts have explicit budgets; +- diagnostics report event, active, superseded, non-story, dropped, and + compaction counts. + +`project_story_guide_memory()` projects the existing Story Butler compressed +context into the same portable contract. The live Story Butler prompt and +runtime behavior remain unchanged. + +## Deterministic Criteria + +| Criterion | What it checks | +| --- | --- | +| Memory continuity | No duplicate active fact or active/superseded conflict | +| Memory boundedness | Fact and recency budgets remain bounded | +| Consequence visibility | Each move yields a typed delta, clue, or opportunity | +| Player agency | Each turn preserves at least two distinct next actions | +| Trajectory progress | Objective progress advances without unexplained regression | +| Entity coherence | Referenced people/factions exist in the scenario registry | +| Choice diversity | Full next-action sets change across the trajectory | +| Boundary hygiene | Player text contains no protocol or private-reasoning leakage | + +The aggregate score is a readable summary, not a replacement for criterion +rows or evidence. + +## CLI + +Export a bundle from the lab, then run the same backend evaluator: + +```bash +python3.11 -m tools.rpg_eval.portable_runtime run.json --out report.json +``` + +No provider call is made. Invalid or extra fields fail closed through Pydantic +contracts. + +## Frontend Views + +- **Overview**: aggregate result, criterion rows, evidence, and baseline comparison. +- **Run trace**: action, world reaction, explicit changes, and next actions per turn. +- **Memory**: active facts, superseded facts, people, open threads, and recent events. +- **Adapter**: compact input shape plus bundle/report downloads. + +The built-in candidate and prose-only baseline make the lab usable without a +backend. Imported bundles never leave the browser. + +When the full local Tiny Stories backend is running, the same route also accepts +`?session=`. An authenticated owner can load a completed live +session, export its server-built bundle, and run the authoritative deterministic +evaluator without copying JSON. This local-only loader is omitted from the +public static build, so the Vercel artifact remains backend-free and cannot +silently claim access to live sessions. + +## End-to-End Provenance + +The reference runtime keeps one auditable contract across the product chain: + +1. Story Butler compacts current facts, corrections, boundaries, open questions, + and non-story intents. +2. Template creation persists the Story Brief plus a sanitized Story Guide + context. Raw chat turns are not stored in this research seed. +3. Play turns and endings receive a compact `story_contract` containing current + truth only. Superseded facts, meta conversation, planner internals, and raw + model messages are excluded. +4. Session export projects the persisted seed and every completed Play turn into + `rpg_evaluation_bundle.v1`. +5. The frontend lab and backend evaluator share the same terminal-turn and + progress-provenance rules. A reported progress value derived from the turn + budget is labeled `turn_budget_proxy`, not presented as engine ground truth. + +The owner-only Reviewer surface links to the session-scoped lab after the first +completed turn. Normal Play remains free of evaluator, token, provider, and +private-reasoning details. + +## Vercel Delivery + +`frontend2/vercel.json` builds the same React application with +`VITE_PUBLIC_LAB_DEFAULT=true`. The deployment therefore opens the evaluation +lab at the root and skips the initial auth request. No LLM key or backend URL is +needed for the public artifact. + +The live Tiny Stories acceptance path remains a separate, configured-backend +validation. A static Vercel demo must not be cited as live-provider proof. + +## Strict Live Gate + +`tools/rpg_eval/tiny_stories_golden_path_harness.py` is the strict completion +gate. It runs a corrected Story Butler memory sequence, Story Brief, live +opening, 12 live Play turns, ending, persisted session export, and backend +evaluation. Every required provider operation must be `live` or +`live_repaired`, with no fallback reason. The gate preserves exact telemetry +and a deterministic quality summary; neither is a substitute for human +narrative judgment. + +## Latest Bounded Live Acceptance + +The strict 2026-08-24 run used an isolated runtime database and a configured +live text gateway. It corrected an initially wrong player role, compiled the +current truth into a Story Brief, generated a live opening, completed all 12 +Play turns, produced an ending, exported the persisted session, and evaluated +the portable bundle. The sanitized evidence is available as +[`evidence/tiny-stories-12-turn-live-research-runtime-2026-08-24.json`](./evidence/tiny-stories-12-turn-live-research-runtime-2026-08-24.json) +and a compact +[`Markdown report`](./evidence/tiny-stories-12-turn-live-research-runtime-2026-08-24.md). + +| Gate | Result | +| --- | --- | +| Required live operations | Story Butler, Story Brief, Opening, 12 Play turns | +| Provider classification | all `live/success`; zero fallback, repair, or retry | +| Play latency | 11,403 ms median; 14,858 ms max | +| Play usage | 99,705 input; 71,680 cached input; 7,665 output; 107,370 total tokens | +| Ending | completed at 12/12, canonical label `共谋` | +| Step / Contract Judge | 12 / 12 rows, all pass | +| Deterministic quality package | all 6 bounded criteria pass | +| Portable evaluator | pass, 100/100, progress provenance `turn_budget_proxy` | +| Memory | 13 active facts; 3 superseded facts; corrected role absent from current truth | + +Two shorter bilingual runs separately exercise the same current-truth memory, +Brief, opening, Play, persistence, and evaluator path: + +- [English live acceptance](./evidence/rpg-evaluation-live-memory-acceptance-2026-08-24.json): + pass, portable score 90, opening consistency pass. +- [Chinese live acceptance](./evidence/rpg-evaluation-live-zh-acceptance-2026-08-24.json): + pass, portable score 90, Chinese Story Butler/Brief/Opening/Play output, opening + consistency warn retained as a quality limitation. + +These runs prove that the bounded reference path can complete through the +configured provider without deterministic fallback. They do not prove general +story quality, calibrated fun, consumer demand, or robustness outside the +tested scenarios. The Browser acceptance also covered English and Chinese +Home, Create, Play, Replay, Portfolio, About, Evaluation, Reviewer, and local QA +routes at desktop and 390 px widths; normal player surfaces had no provider, +schema, token, trace, fallback, raw JSON, or private-reasoning copy. diff --git a/docs/evidence/rpg-evaluation-live-acceptance-2026-08-24.json b/docs/evidence/rpg-evaluation-live-acceptance-2026-08-24.json new file mode 100644 index 00000000..2d84aaf5 --- /dev/null +++ b/docs/evidence/rpg-evaluation-live-acceptance-2026-08-24.json @@ -0,0 +1,76 @@ +{ + "schema_version": "tiny_stories_live_acceptance_evidence.v1", + "generated_at": "2026-08-23T16:42:37.169185+00:00", + "local_acceptance_date": "2026-08-24", + "status": "pass", + "scope": [ + "story_butler_turn", + "story_brief", + "opening", + "play_turn", + "reviewer_telemetry" + ], + "health": { + "text_llm": "configured", + "create_story_butler": "configured", + "story_brief": "configured", + "opening": "configured", + "play_turns": "configured" + }, + "operations": [ + { + "operation": "create.story_butler_turn", + "source_label": "live", + "status": "success", + "latency_ms": 1281, + "input_tokens": 2389, + "cached_input_tokens": 0, + "output_tokens": 31, + "total_tokens": 2420, + "fallback_reason": null + }, + { + "operation": "narrative.story_brief", + "source_label": "live", + "status": "success", + "latency_ms": 2207, + "input_tokens": 1477, + "cached_input_tokens": 0, + "output_tokens": 156, + "total_tokens": 1633, + "fallback_reason": null + }, + { + "operation": "narrative.opening", + "source_label": "live", + "status": "success", + "latency_ms": 12635, + "input_tokens": 2686, + "cached_input_tokens": 0, + "output_tokens": 808, + "total_tokens": 3494, + "fallback_reason": null + }, + { + "operation": "narrative.advance_turn", + "source_label": "live", + "status": "success", + "latency_ms": 9312, + "input_tokens": 6863, + "cached_input_tokens": 0, + "output_tokens": 447, + "total_tokens": 7310, + "fallback_reason": null + } + ], + "quality_signals": { + "opening_recovery": null, + "opening_consistency_status": "warn", + "play_turn_complete": false, + "judge_events_present": [ + "step_judge", + "contract_judge" + ] + }, + "claim_boundary": "This bounded run proves live-path execution and telemetry for one reference case. It does not establish calibrated narrative quality, general robustness, or consumer-market validation." +} diff --git a/docs/evidence/rpg-evaluation-live-memory-acceptance-2026-08-24.json b/docs/evidence/rpg-evaluation-live-memory-acceptance-2026-08-24.json new file mode 100644 index 00000000..184c2b8d --- /dev/null +++ b/docs/evidence/rpg-evaluation-live-memory-acceptance-2026-08-24.json @@ -0,0 +1,284 @@ +{ + "schema_version": "tiny_stories_reliability_live_summary.v1", + "mode": "live_acceptance", + "generated_at": "2026-08-23T18:57:51.391453+00:00", + "run_started_at": "2026-08-23T18:57:14.249111+00:00", + "status": "pass", + "base_url": "http://127.0.0.1:8450", + "username": "portfolio_reviewer", + "language": "en", + "seed_excerpt": "At an awards gala, a publicist, a singer, and a sponsor discover the live trophy reveal is rigged. The player is the publicist who must protect the singer before the host walks onstage. No gore.", + "template_id": "tmpl_3a081028c1f0", + "session_id": "sess_39113231e58b", + "required_operations": [ + "create.story_butler_turn", + "narrative.advance_turn", + "narrative.opening", + "narrative.story_brief" + ], + "health_required": [ + "create_story_butler", + "opening", + "play_turns", + "story_brief", + "text_llm" + ], + "api_latencies_ms": { + "health_ms": 4, + "login_ms": 2, + "story_guide_ms": 6581, + "story_guide_correction_ms": 6272, + "story_brief_ms": 2337, + "template_opening_ms": 13020, + "story_with_trace_ms": 7, + "advance_turn_ms": 8877, + "evaluation_bundle_ms": 10, + "evaluation_report_ms": 3, + "llm_events_ms": 3 + }, + "stage_sources": { + "story_guide": { + "initial_source": "live", + "correction_source": "live", + "status": "needs_field", + "canShapeBrief": false, + "reply_excerpt": "Noted—you're the publicist. The rigged reveal is about to go live. What hits first: an accusation, a disappearance, a forced decision, or an exposed secret?", + "player_role": "publicist", + "active_cast": [ + "singer", + "sponsor", + "host" + ], + "superseded_count": 1 + }, + "story_brief": { + "source": "live_hybrid_v1", + "runtime_source": "live", + "can_generate": true + }, + "opening": { + "template_id": "tmpl_3a081028c1f0", + "session_id": "sess_39113231e58b", + "opening_recovery": null, + "consistency_status": "pass" + }, + "advance_turn": { + "is_complete": false, + "agent_event_types": [ + "agent_plan", + "step_judge", + "contract_judge" + ] + }, + "portable_evaluation": { + "bundle_schema": "rpg_evaluation_bundle.v1", + "report_schema": "rpg_evaluation_report.v1", + "status": "pass", + "score": 90, + "progress_basis": "turn_budget_proxy", + "active_fact_count": 13, + "superseded_fact_count": 3 + } + }, + "failures": [], + "operation_events": { + "create.story_butler_context": [ + { + "event_id": 60, + "operation": "create.story_butler_context", + "status": "success", + "source_label": "live", + "latency_ms": 4783, + "operation_latency_ms": 4783, + "input_tokens": 1294, + "cached_input_tokens": 1280, + "output_tokens": 321, + "total_tokens": 1615, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": null, + "session_id": null, + "created_at": "2026-08-23T18:57:19.060862+00:00" + }, + { + "event_id": 62, + "operation": "create.story_butler_context", + "status": "success", + "source_label": "live", + "latency_ms": 4038, + "operation_latency_ms": 4038, + "input_tokens": 1783, + "cached_input_tokens": 640, + "output_tokens": 309, + "total_tokens": 2092, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": null, + "session_id": null, + "created_at": "2026-08-23T18:57:24.899236+00:00" + } + ], + "create.story_butler_turn": [ + { + "event_id": 61, + "operation": "create.story_butler_turn", + "status": "success", + "source_label": "live", + "latency_ms": 1784, + "operation_latency_ms": 1786, + "input_tokens": 2407, + "cached_input_tokens": 2304, + "output_tokens": 38, + "total_tokens": 2445, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": null, + "session_id": null, + "created_at": "2026-08-23T18:57:20.851787+00:00" + }, + { + "event_id": 63, + "operation": "create.story_butler_turn", + "status": "success", + "source_label": "live", + "latency_ms": 2221, + "operation_latency_ms": 2221, + "input_tokens": 2674, + "cached_input_tokens": 512, + "output_tokens": 42, + "total_tokens": 2716, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": null, + "session_id": null, + "created_at": "2026-08-23T18:57:27.125423+00:00" + } + ], + "narrative.advance_turn": [ + { + "event_id": 66, + "operation": "narrative.advance_turn", + "status": "success", + "source_label": "live", + "latency_ms": 7852, + "operation_latency_ms": 8860, + "input_tokens": 7115, + "cached_input_tokens": 5376, + "output_tokens": 445, + "total_tokens": 7560, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": "tmpl_3a081028c1f0", + "session_id": "sess_39113231e58b", + "created_at": "2026-08-23T18:57:51.362260+00:00" + } + ], + "narrative.opening": [ + { + "event_id": 65, + "operation": "narrative.opening", + "status": "success", + "source_label": "live", + "latency_ms": 13000, + "operation_latency_ms": 13014, + "input_tokens": 2856, + "cached_input_tokens": 1536, + "output_tokens": 847, + "total_tokens": 3703, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": "tmpl_3a081028c1f0", + "session_id": "sess_39113231e58b", + "created_at": "2026-08-23T18:57:42.485195+00:00" + } + ], + "narrative.story_brief": [ + { + "event_id": 64, + "operation": "narrative.story_brief", + "status": "success", + "source_label": "live", + "latency_ms": 2319, + "operation_latency_ms": 2319, + "input_tokens": 1763, + "cached_input_tokens": 1664, + "output_tokens": 155, + "total_tokens": 1918, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": null, + "session_id": null, + "created_at": "2026-08-23T18:57:29.458211+00:00" + } + ] + }, + "session_llm_events": [ + { + "event_id": 65, + "operation": "narrative.opening", + "status": "success", + "source_label": "live", + "latency_ms": 13000, + "operation_latency_ms": 13014, + "input_tokens": 2856, + "cached_input_tokens": 1536, + "output_tokens": 847, + "total_tokens": 3703, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": "tmpl_3a081028c1f0", + "session_id": "sess_39113231e58b", + "created_at": "2026-08-23T18:57:42.485195+00:00" + }, + { + "event_id": 66, + "operation": "narrative.advance_turn", + "status": "success", + "source_label": "live", + "latency_ms": 7852, + "operation_latency_ms": 8860, + "input_tokens": 7115, + "cached_input_tokens": 5376, + "output_tokens": 445, + "total_tokens": 7560, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": "tmpl_3a081028c1f0", + "session_id": "sess_39113231e58b", + "created_at": "2026-08-23T18:57:51.362260+00:00" + }, + { + "event_id": 67, + "operation": "narrative.judge_failure", + "status": "success", + "source_label": "live", + "latency_ms": 998, + "operation_latency_ms": 8860, + "input_tokens": 718, + "cached_input_tokens": 256, + "output_tokens": 22, + "total_tokens": 740, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": "tmpl_3a081028c1f0", + "session_id": "sess_39113231e58b", + "created_at": "2026-08-23T18:57:51.367614+00:00" + } + ], + "runtime_db": "/tmp/tiny-stories-research-runtime-live-20260824.sqlite3", + "notes": [ + "This is the live acceptance path: it drives Story Butler, Story Brief, opening/template, Play turn, and reviewer telemetry APIs.", + "A pass requires accepted live/live_repaired telemetry for all required operations and no fallback_reason on those rows.", + "Protocol/fixture checks are separate and cheaper; they are not the main acceptance evidence." + ] +} diff --git a/docs/evidence/rpg-evaluation-live-zh-acceptance-2026-08-24.json b/docs/evidence/rpg-evaluation-live-zh-acceptance-2026-08-24.json new file mode 100644 index 00000000..c80ad45f --- /dev/null +++ b/docs/evidence/rpg-evaluation-live-zh-acceptance-2026-08-24.json @@ -0,0 +1,284 @@ +{ + "schema_version": "tiny_stories_reliability_live_summary.v1", + "mode": "live_acceptance", + "generated_at": "2026-08-23T19:09:47.395537+00:00", + "run_started_at": "2026-08-23T19:09:10.742142+00:00", + "status": "pass", + "base_url": "http://127.0.0.1:8450", + "username": "portfolio_reviewer", + "language": "zh", + "seed_excerpt": "在一场直播颁奖礼上，公关、歌手和赞助商发现奖杯揭晓被人操纵。玩家是公关，必须在主持人上台前保护歌手。不要出现血腥暴力。", + "template_id": "tmpl_4b98a27d2fb2", + "session_id": "sess_fce44799b64a", + "required_operations": [ + "create.story_butler_turn", + "narrative.advance_turn", + "narrative.opening", + "narrative.story_brief" + ], + "health_required": [ + "create_story_butler", + "opening", + "play_turns", + "story_brief", + "text_llm" + ], + "api_latencies_ms": { + "health_ms": 4, + "login_ms": 2, + "story_guide_ms": 6063, + "story_guide_correction_ms": 5768, + "story_brief_ms": 3092, + "template_opening_ms": 11503, + "story_with_trace_ms": 9, + "advance_turn_ms": 10171, + "evaluation_bundle_ms": 8, + "evaluation_report_ms": 3, + "llm_events_ms": 3 + }, + "stage_sources": { + "story_guide": { + "initial_source": "live", + "correction_source": "live", + "status": "ready_to_brief", + "canShapeBrief": true, + "reply_excerpt": "公关身份已确认——直播颁奖礼、被操纵的奖杯、主持上台前护住歌手。Story Brief 已在下方就绪，可直接进入故事，或再发一条修正。", + "player_role": "公关", + "active_cast": [ + "歌手", + "赞助", + "主持人" + ], + "superseded_count": 5 + }, + "story_brief": { + "source": "live_hybrid_v1", + "runtime_source": "live", + "can_generate": true + }, + "opening": { + "template_id": "tmpl_4b98a27d2fb2", + "session_id": "sess_fce44799b64a", + "opening_recovery": null, + "consistency_status": "warn" + }, + "advance_turn": { + "is_complete": false, + "agent_event_types": [ + "agent_plan", + "step_judge", + "contract_judge" + ] + }, + "portable_evaluation": { + "bundle_schema": "rpg_evaluation_bundle.v1", + "report_schema": "rpg_evaluation_report.v1", + "status": "pass", + "score": 90, + "progress_basis": "turn_budget_proxy", + "active_fact_count": 12, + "superseded_fact_count": 5 + } + }, + "failures": [], + "operation_events": { + "create.story_butler_context": [ + { + "event_id": 104, + "operation": "create.story_butler_context", + "status": "success", + "source_label": "live", + "latency_ms": 3987, + "operation_latency_ms": 3988, + "input_tokens": 1435, + "cached_input_tokens": 1408, + "output_tokens": 326, + "total_tokens": 1761, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": null, + "session_id": null, + "created_at": "2026-08-23T19:09:14.758989+00:00" + }, + { + "event_id": 106, + "operation": "create.story_butler_context", + "status": "success", + "source_label": "live", + "latency_ms": 3927, + "operation_latency_ms": 3927, + "input_tokens": 2116, + "cached_input_tokens": 1536, + "output_tokens": 329, + "total_tokens": 2445, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": null, + "session_id": null, + "created_at": "2026-08-23T19:09:20.763563+00:00" + } + ], + "create.story_butler_turn": [ + { + "event_id": 105, + "operation": "create.story_butler_turn", + "status": "success", + "source_label": "live", + "latency_ms": 2055, + "operation_latency_ms": 2058, + "input_tokens": 2754, + "cached_input_tokens": 640, + "output_tokens": 40, + "total_tokens": 2794, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": null, + "session_id": null, + "created_at": "2026-08-23T19:09:16.824831+00:00" + }, + { + "event_id": 107, + "operation": "create.story_butler_turn", + "status": "success", + "source_label": "live", + "latency_ms": 1823, + "operation_latency_ms": 1824, + "input_tokens": 3335, + "cached_input_tokens": 512, + "output_tokens": 44, + "total_tokens": 3379, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": null, + "session_id": null, + "created_at": "2026-08-23T19:09:22.592816+00:00" + } + ], + "narrative.advance_turn": [ + { + "event_id": 110, + "operation": "narrative.advance_turn", + "status": "success", + "source_label": "live", + "latency_ms": 8159, + "operation_latency_ms": 10155, + "input_tokens": 6518, + "cached_input_tokens": 5376, + "output_tokens": 462, + "total_tokens": 6980, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": "tmpl_4b98a27d2fb2", + "session_id": "sess_fce44799b64a", + "created_at": "2026-08-23T19:09:47.369379+00:00" + } + ], + "narrative.opening": [ + { + "event_id": 109, + "operation": "narrative.opening", + "status": "success", + "source_label": "live", + "latency_ms": 11484, + "operation_latency_ms": 11497, + "input_tokens": 2149, + "cached_input_tokens": 768, + "output_tokens": 776, + "total_tokens": 2925, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": "tmpl_4b98a27d2fb2", + "session_id": "sess_fce44799b64a", + "created_at": "2026-08-23T19:09:37.195036+00:00" + } + ], + "narrative.story_brief": [ + { + "event_id": 108, + "operation": "narrative.story_brief", + "status": "success", + "source_label": "live", + "latency_ms": 3075, + "operation_latency_ms": 3075, + "input_tokens": 1802, + "cached_input_tokens": 1792, + "output_tokens": 181, + "total_tokens": 1983, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": null, + "session_id": null, + "created_at": "2026-08-23T19:09:25.684190+00:00" + } + ] + }, + "session_llm_events": [ + { + "event_id": 109, + "operation": "narrative.opening", + "status": "success", + "source_label": "live", + "latency_ms": 11484, + "operation_latency_ms": 11497, + "input_tokens": 2149, + "cached_input_tokens": 768, + "output_tokens": 776, + "total_tokens": 2925, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": "tmpl_4b98a27d2fb2", + "session_id": "sess_fce44799b64a", + "created_at": "2026-08-23T19:09:37.195036+00:00" + }, + { + "event_id": 110, + "operation": "narrative.advance_turn", + "status": "success", + "source_label": "live", + "latency_ms": 8159, + "operation_latency_ms": 10155, + "input_tokens": 6518, + "cached_input_tokens": 5376, + "output_tokens": 462, + "total_tokens": 6980, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": "tmpl_4b98a27d2fb2", + "session_id": "sess_fce44799b64a", + "created_at": "2026-08-23T19:09:47.369379+00:00" + }, + { + "event_id": 111, + "operation": "narrative.judge_failure", + "status": "success", + "source_label": "live", + "latency_ms": 1981, + "operation_latency_ms": 10155, + "input_tokens": 698, + "cached_input_tokens": 256, + "output_tokens": 22, + "total_tokens": 720, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": "tmpl_4b98a27d2fb2", + "session_id": "sess_fce44799b64a", + "created_at": "2026-08-23T19:09:47.374207+00:00" + } + ], + "runtime_db": "/tmp/tiny-stories-research-runtime-live-20260824.sqlite3", + "notes": [ + "This is the live acceptance path: it drives Story Butler, Story Brief, opening/template, Play turn, and reviewer telemetry APIs.", + "A pass requires accepted live/live_repaired telemetry for all required operations and no fallback_reason on those rows.", + "Protocol/fixture checks are separate and cheaper; they are not the main acceptance evidence." + ] +} diff --git a/docs/evidence/tiny-stories-12-turn-live-research-runtime-2026-08-24.json b/docs/evidence/tiny-stories-12-turn-live-research-runtime-2026-08-24.json new file mode 100644 index 00000000..2913b46e --- /dev/null +++ b/docs/evidence/tiny-stories-12-turn-live-research-runtime-2026-08-24.json @@ -0,0 +1,1300 @@ +{ + "schema_version": "tiny_stories_live_golden_path.v1", + "mode": "live_golden_path_12_turn", + "generated_at": "2026-08-23T19:02:12.038317+00:00", + "run_started_at": "2026-08-23T18:59:11.541460+00:00", + "status": "pass", + "base_url": "http://127.0.0.1:8450", + "username": "portfolio_reviewer", + "seed_excerpt": "At an awards gala, a publicist, a singer, and a sponsor discover the live trophy reveal is rigged. The player is the publicist who must protect the singer before the host walks onstage. No gore.", + "turn_budget": 12, + "completed_turns": 12, + "template_id": "tmpl_741cac2776e9", + "session_id": "sess_0b6f5a82cc81", + "health_required": [ + "create_story_butler", + "opening", + "play_turns", + "story_brief", + "text_llm" + ], + "required_operations": [ + "create.story_butler_turn", + "narrative.advance_turn", + "narrative.opening", + "narrative.story_brief" + ], + "api_latencies_ms": { + "health_ms": 1, + "login_ms": 2, + "story_guide_ms": 5049, + "story_guide_correction_ms": 4419, + "story_brief_ms": 2042, + "template_opening_ms": 13461, + "initial_story_ms": 6, + "evaluation_bundle_ms": 11, + "evaluation_report_ms": 4, + "llm_events_ms": 2 + }, + "stage_sources": { + "story_guide": { + "initial_source": "live", + "correction_source": "live", + "status": "needs_field", + "canShapeBrief": false, + "reply_excerpt": "The gala is live, and eyes are on you. What pressure hits first: an accusation, a disappearance, a decision, or an exposed secret?", + "player_role": "publicist", + "active_cast": [ + "singer", + "sponsor", + "host" + ], + "superseded_count": 1 + }, + "story_brief": { + "source": "live_hybrid_v1", + "runtime_source": "live", + "can_generate": true + }, + "opening": { + "template_id": "tmpl_741cac2776e9", + "session_id": "sess_0b6f5a82cc81", + "opening_recovery": null, + "consistency_status": "pass" + }, + "portable_evaluation": { + "bundle_schema": "rpg_evaluation_bundle.v1", + "report_schema": "rpg_evaluation_report.v1", + "status": "pass", + "score": 100, + "progress_basis": "turn_budget_proxy", + "active_fact_count": 13, + "superseded_fact_count": 3 + } + }, + "turns": [ + { + "turn_number": 1, + "api_latency_ms": 9989, + "story_refresh_latency_ms": 5, + "chosen_option_index": 0, + "chosen_option_label": "[Confront] Tell Arthur the rigging stops now.", + "narrator_ord": 2, + "is_complete": false, + "ending_present": false, + "next_option_count": 3, + "step_judge_status": "pass", + "contract_judge_status": "pass" + }, + { + "turn_number": 2, + "api_latency_ms": 11546, + "story_refresh_latency_ms": 6, + "chosen_option_index": 1, + "chosen_option_label": "[Yield] Step back and let the rigged reveal proceed to protect Elena.", + "narrator_ord": 4, + "is_complete": false, + "ending_present": false, + "next_option_count": 3, + "step_judge_status": "pass", + "contract_judge_status": "pass" + }, + { + "turn_number": 3, + "api_latency_ms": 9890, + "story_refresh_latency_ms": 6, + "chosen_option_index": 2, + "chosen_option_label": "[Counter] Pull out the burner phone and threaten to expose Arthur.", + "narrator_ord": 6, + "is_complete": false, + "ending_present": false, + "next_option_count": 3, + "step_judge_status": "pass", + "contract_judge_status": "pass" + }, + { + "turn_number": 4, + "api_latency_ms": 9715, + "story_refresh_latency_ms": 6, + "chosen_option_index": 0, + "chosen_option_label": "[Counter] Show Arthur the screen and threaten to send the texts to the press", + "narrator_ord": 8, + "is_complete": false, + "ending_present": false, + "next_option_count": 3, + "step_judge_status": "pass", + "contract_judge_status": "pass" + }, + { + "turn_number": 5, + "api_latency_ms": 10840, + "story_refresh_latency_ms": 9, + "chosen_option_index": 1, + "chosen_option_label": "[Yield] Lower the phone and negotiate a cover-up", + "narrator_ord": 10, + "is_complete": false, + "ending_present": false, + "next_option_count": 3, + "step_judge_status": "pass", + "contract_judge_status": "pass" + }, + { + "turn_number": 6, + "api_latency_ms": 11602, + "story_refresh_latency_ms": 7, + "chosen_option_index": 2, + "chosen_option_label": "[Counter] Reveal that you know about Arthur's past sponsorship scandals and demand he handle the journalist himself.", + "narrator_ord": 12, + "is_complete": false, + "ending_present": false, + "next_option_count": 3, + "step_judge_status": "pass", + "contract_judge_status": "pass" + }, + { + "turn_number": 7, + "api_latency_ms": 12081, + "story_refresh_latency_ms": 6, + "chosen_option_index": 0, + "chosen_option_label": "[Ally] Propose to Elena that you both find leverage against Arthur together.", + "narrator_ord": 14, + "is_complete": false, + "ending_present": false, + "next_option_count": 3, + "step_judge_status": "pass", + "contract_judge_status": "pass" + }, + { + "turn_number": 8, + "api_latency_ms": 14877, + "story_refresh_latency_ms": 10, + "chosen_option_index": 1, + "chosen_option_label": "[Counter] Show Elena the texts and plan to leak them together.", + "narrator_ord": 16, + "is_complete": false, + "ending_present": false, + "next_option_count": 3, + "step_judge_status": "pass", + "contract_judge_status": "pass" + }, + { + "turn_number": 9, + "api_latency_ms": 12193, + "story_refresh_latency_ms": 8, + "chosen_option_index": 2, + "chosen_option_label": "[Ally] Suggest to Elena that you both confront Arthur together, presenting a united front.", + "narrator_ord": 18, + "is_complete": false, + "ending_present": false, + "next_option_count": 3, + "step_judge_status": "pass", + "contract_judge_status": "pass" + }, + { + "turn_number": 10, + "api_latency_ms": 13189, + "story_refresh_latency_ms": 9, + "chosen_option_index": 0, + "chosen_option_label": "[Confront] Step forward and lay out the evidence, demanding Arthur back off.", + "narrator_ord": 20, + "is_complete": false, + "ending_present": false, + "next_option_count": 3, + "step_judge_status": "pass", + "contract_judge_status": "pass" + }, + { + "turn_number": 11, + "api_latency_ms": 11305, + "story_refresh_latency_ms": 9, + "chosen_option_index": 1, + "chosen_option_label": "[Ally] Plan with Elena to release the evidence later, when you can control the narrative.", + "narrator_ord": 22, + "is_complete": false, + "ending_present": false, + "next_option_count": 3, + "step_judge_status": "pass", + "contract_judge_status": "pass" + }, + { + "turn_number": 12, + "api_latency_ms": 28138, + "story_refresh_latency_ms": 7, + "chosen_option_index": 2, + "chosen_option_label": "[Ally] Leave the gala with Elena and plan the leak from a safe distance.", + "narrator_ord": 24, + "is_complete": true, + "ending_present": true, + "next_option_count": 3, + "step_judge_status": "pass", + "contract_judge_status": "pass" + } + ], + "turn_telemetry": [ + { + "event_id": 81, + "operation": "narrative.advance_turn", + "status": "success", + "source_label": "live", + "latency_ms": 9969, + "operation_latency_ms": 9977, + "input_tokens": 7141, + "cached_input_tokens": 5376, + "output_tokens": 543, + "total_tokens": 7684, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": "tmpl_741cac2776e9", + "session_id": "sess_0b6f5a82cc81", + "created_at": "2026-08-23T18:59:46.530862+00:00" + }, + { + "event_id": 82, + "operation": "narrative.advance_turn", + "status": "success", + "source_label": "live", + "latency_ms": 11522, + "operation_latency_ms": 11537, + "input_tokens": 7364, + "cached_input_tokens": 5376, + "output_tokens": 674, + "total_tokens": 8038, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": "tmpl_741cac2776e9", + "session_id": "sess_0b6f5a82cc81", + "created_at": "2026-08-23T18:59:58.083862+00:00" + }, + { + "event_id": 83, + "operation": "narrative.advance_turn", + "status": "success", + "source_label": "live", + "latency_ms": 9870, + "operation_latency_ms": 9882, + "input_tokens": 7602, + "cached_input_tokens": 5888, + "output_tokens": 585, + "total_tokens": 8187, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": "tmpl_741cac2776e9", + "session_id": "sess_0b6f5a82cc81", + "created_at": "2026-08-23T19:00:07.981294+00:00" + }, + { + "event_id": 84, + "operation": "narrative.advance_turn", + "status": "success", + "source_label": "live", + "latency_ms": 9690, + "operation_latency_ms": 9703, + "input_tokens": 7856, + "cached_input_tokens": 6144, + "output_tokens": 510, + "total_tokens": 8366, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": "tmpl_741cac2776e9", + "session_id": "sess_0b6f5a82cc81", + "created_at": "2026-08-23T19:00:17.703285+00:00" + }, + { + "event_id": 85, + "operation": "narrative.advance_turn", + "status": "success", + "source_label": "live", + "latency_ms": 10819, + "operation_latency_ms": 10831, + "input_tokens": 8092, + "cached_input_tokens": 6272, + "output_tokens": 553, + "total_tokens": 8645, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": "tmpl_741cac2776e9", + "session_id": "sess_0b6f5a82cc81", + "created_at": "2026-08-23T19:00:28.550856+00:00" + }, + { + "event_id": 86, + "operation": "narrative.advance_turn", + "status": "success", + "source_label": "live", + "latency_ms": 11578, + "operation_latency_ms": 11594, + "input_tokens": 8368, + "cached_input_tokens": 6528, + "output_tokens": 716, + "total_tokens": 9084, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": "tmpl_741cac2776e9", + "session_id": "sess_0b6f5a82cc81", + "created_at": "2026-08-23T19:00:40.163985+00:00" + }, + { + "event_id": 87, + "operation": "narrative.advance_turn", + "status": "success", + "source_label": "live", + "latency_ms": 12060, + "operation_latency_ms": 12070, + "input_tokens": 8605, + "cached_input_tokens": 6784, + "output_tokens": 639, + "total_tokens": 9244, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": "tmpl_741cac2776e9", + "session_id": "sess_0b6f5a82cc81", + "created_at": "2026-08-23T19:00:52.253386+00:00" + }, + { + "event_id": 88, + "operation": "narrative.advance_turn", + "status": "success", + "source_label": "live", + "latency_ms": 14858, + "operation_latency_ms": 14868, + "input_tokens": 8843, + "cached_input_tokens": 7040, + "output_tokens": 911, + "total_tokens": 9754, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": "tmpl_741cac2776e9", + "session_id": "sess_0b6f5a82cc81", + "created_at": "2026-08-23T19:01:07.138146+00:00" + }, + { + "event_id": 89, + "operation": "narrative.advance_turn", + "status": "success", + "source_label": "live", + "latency_ms": 12174, + "operation_latency_ms": 12184, + "input_tokens": 8899, + "cached_input_tokens": 5376, + "output_tokens": 667, + "total_tokens": 9566, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": "tmpl_741cac2776e9", + "session_id": "sess_0b6f5a82cc81", + "created_at": "2026-08-23T19:01:19.342688+00:00" + }, + { + "event_id": 90, + "operation": "narrative.advance_turn", + "status": "success", + "source_label": "live", + "latency_ms": 13166, + "operation_latency_ms": 13181, + "input_tokens": 8922, + "cached_input_tokens": 5632, + "output_tokens": 779, + "total_tokens": 9701, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": "tmpl_741cac2776e9", + "session_id": "sess_0b6f5a82cc81", + "created_at": "2026-08-23T19:01:32.541823+00:00" + }, + { + "event_id": 91, + "operation": "narrative.advance_turn", + "status": "success", + "source_label": "live", + "latency_ms": 11284, + "operation_latency_ms": 11297, + "input_tokens": 8987, + "cached_input_tokens": 5632, + "output_tokens": 571, + "total_tokens": 9558, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": "tmpl_741cac2776e9", + "session_id": "sess_0b6f5a82cc81", + "created_at": "2026-08-23T19:01:43.857931+00:00" + }, + { + "event_id": 92, + "operation": "narrative.advance_turn", + "status": "success", + "source_label": "live", + "latency_ms": 9957, + "operation_latency_ms": 28125, + "input_tokens": 9026, + "cached_input_tokens": 5632, + "output_tokens": 517, + "total_tokens": 9543, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": "tmpl_741cac2776e9", + "session_id": "sess_0b6f5a82cc81", + "created_at": "2026-08-23T19:02:12.003074+00:00" + } + ], + "operation_events": { + "create.story_butler_turn": [ + { + "event_id": 76, + "operation": "create.story_butler_turn", + "status": "success", + "source_label": "live", + "latency_ms": 1423, + "operation_latency_ms": 1424, + "input_tokens": 2407, + "cached_input_tokens": 2304, + "output_tokens": 26, + "total_tokens": 2433, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": null, + "session_id": null, + "created_at": "2026-08-23T18:59:16.606195+00:00" + }, + { + "event_id": 78, + "operation": "create.story_butler_turn", + "status": "success", + "source_label": "live", + "latency_ms": 1036, + "operation_latency_ms": 1037, + "input_tokens": 2653, + "cached_input_tokens": 512, + "output_tokens": 38, + "total_tokens": 2691, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": null, + "session_id": null, + "created_at": "2026-08-23T18:59:21.028119+00:00" + } + ], + "narrative.advance_turn": [ + { + "event_id": 81, + "operation": "narrative.advance_turn", + "status": "success", + "source_label": "live", + "latency_ms": 9969, + "operation_latency_ms": 9977, + "input_tokens": 7141, + "cached_input_tokens": 5376, + "output_tokens": 543, + "total_tokens": 7684, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": "tmpl_741cac2776e9", + "session_id": "sess_0b6f5a82cc81", + "created_at": "2026-08-23T18:59:46.530862+00:00" + }, + { + "event_id": 82, + "operation": "narrative.advance_turn", + "status": "success", + "source_label": "live", + "latency_ms": 11522, + "operation_latency_ms": 11537, + "input_tokens": 7364, + "cached_input_tokens": 5376, + "output_tokens": 674, + "total_tokens": 8038, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": "tmpl_741cac2776e9", + "session_id": "sess_0b6f5a82cc81", + "created_at": "2026-08-23T18:59:58.083862+00:00" + }, + { + "event_id": 83, + "operation": "narrative.advance_turn", + "status": "success", + "source_label": "live", + "latency_ms": 9870, + "operation_latency_ms": 9882, + "input_tokens": 7602, + "cached_input_tokens": 5888, + "output_tokens": 585, + "total_tokens": 8187, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": "tmpl_741cac2776e9", + "session_id": "sess_0b6f5a82cc81", + "created_at": "2026-08-23T19:00:07.981294+00:00" + }, + { + "event_id": 84, + "operation": "narrative.advance_turn", + "status": "success", + "source_label": "live", + "latency_ms": 9690, + "operation_latency_ms": 9703, + "input_tokens": 7856, + "cached_input_tokens": 6144, + "output_tokens": 510, + "total_tokens": 8366, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": "tmpl_741cac2776e9", + "session_id": "sess_0b6f5a82cc81", + "created_at": "2026-08-23T19:00:17.703285+00:00" + }, + { + "event_id": 85, + "operation": "narrative.advance_turn", + "status": "success", + "source_label": "live", + "latency_ms": 10819, + "operation_latency_ms": 10831, + "input_tokens": 8092, + "cached_input_tokens": 6272, + "output_tokens": 553, + "total_tokens": 8645, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": "tmpl_741cac2776e9", + "session_id": "sess_0b6f5a82cc81", + "created_at": "2026-08-23T19:00:28.550856+00:00" + }, + { + "event_id": 86, + "operation": "narrative.advance_turn", + "status": "success", + "source_label": "live", + "latency_ms": 11578, + "operation_latency_ms": 11594, + "input_tokens": 8368, + "cached_input_tokens": 6528, + "output_tokens": 716, + "total_tokens": 9084, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": "tmpl_741cac2776e9", + "session_id": "sess_0b6f5a82cc81", + "created_at": "2026-08-23T19:00:40.163985+00:00" + }, + { + "event_id": 87, + "operation": "narrative.advance_turn", + "status": "success", + "source_label": "live", + "latency_ms": 12060, + "operation_latency_ms": 12070, + "input_tokens": 8605, + "cached_input_tokens": 6784, + "output_tokens": 639, + "total_tokens": 9244, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": "tmpl_741cac2776e9", + "session_id": "sess_0b6f5a82cc81", + "created_at": "2026-08-23T19:00:52.253386+00:00" + }, + { + "event_id": 88, + "operation": "narrative.advance_turn", + "status": "success", + "source_label": "live", + "latency_ms": 14858, + "operation_latency_ms": 14868, + "input_tokens": 8843, + "cached_input_tokens": 7040, + "output_tokens": 911, + "total_tokens": 9754, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": "tmpl_741cac2776e9", + "session_id": "sess_0b6f5a82cc81", + "created_at": "2026-08-23T19:01:07.138146+00:00" + }, + { + "event_id": 89, + "operation": "narrative.advance_turn", + "status": "success", + "source_label": "live", + "latency_ms": 12174, + "operation_latency_ms": 12184, + "input_tokens": 8899, + "cached_input_tokens": 5376, + "output_tokens": 667, + "total_tokens": 9566, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": "tmpl_741cac2776e9", + "session_id": "sess_0b6f5a82cc81", + "created_at": "2026-08-23T19:01:19.342688+00:00" + }, + { + "event_id": 90, + "operation": "narrative.advance_turn", + "status": "success", + "source_label": "live", + "latency_ms": 13166, + "operation_latency_ms": 13181, + "input_tokens": 8922, + "cached_input_tokens": 5632, + "output_tokens": 779, + "total_tokens": 9701, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": "tmpl_741cac2776e9", + "session_id": "sess_0b6f5a82cc81", + "created_at": "2026-08-23T19:01:32.541823+00:00" + }, + { + "event_id": 91, + "operation": "narrative.advance_turn", + "status": "success", + "source_label": "live", + "latency_ms": 11284, + "operation_latency_ms": 11297, + "input_tokens": 8987, + "cached_input_tokens": 5632, + "output_tokens": 571, + "total_tokens": 9558, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": "tmpl_741cac2776e9", + "session_id": "sess_0b6f5a82cc81", + "created_at": "2026-08-23T19:01:43.857931+00:00" + }, + { + "event_id": 92, + "operation": "narrative.advance_turn", + "status": "success", + "source_label": "live", + "latency_ms": 9957, + "operation_latency_ms": 28125, + "input_tokens": 9026, + "cached_input_tokens": 5632, + "output_tokens": 517, + "total_tokens": 9543, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": "tmpl_741cac2776e9", + "session_id": "sess_0b6f5a82cc81", + "created_at": "2026-08-23T19:02:12.003074+00:00" + } + ], + "narrative.branches": [ + { + "event_id": 94, + "operation": "narrative.branches", + "status": "success", + "source_label": "live", + "latency_ms": 7358, + "operation_latency_ms": 28125, + "input_tokens": 6579, + "cached_input_tokens": 0, + "output_tokens": 365, + "total_tokens": 6944, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": "tmpl_741cac2776e9", + "session_id": "sess_0b6f5a82cc81", + "created_at": "2026-08-23T19:02:12.005383+00:00" + } + ], + "narrative.ending": [ + { + "event_id": 93, + "operation": "narrative.ending", + "status": "success", + "source_label": "live", + "latency_ms": 10414, + "operation_latency_ms": 28125, + "input_tokens": 4826, + "cached_input_tokens": 0, + "output_tokens": 429, + "total_tokens": 5255, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": "tmpl_741cac2776e9", + "session_id": "sess_0b6f5a82cc81", + "created_at": "2026-08-23T19:02:12.004240+00:00" + } + ], + "narrative.highlights": [ + { + "event_id": 95, + "operation": "narrative.highlights", + "status": "success", + "source_label": "live", + "latency_ms": 7726, + "operation_latency_ms": 28125, + "input_tokens": 5048, + "cached_input_tokens": 0, + "output_tokens": 585, + "total_tokens": 5633, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": "tmpl_741cac2776e9", + "session_id": "sess_0b6f5a82cc81", + "created_at": "2026-08-23T19:02:12.006343+00:00" + } + ], + "narrative.opening": [ + { + "event_id": 80, + "operation": "narrative.opening", + "status": "success", + "source_label": "live", + "latency_ms": 13444, + "operation_latency_ms": 13456, + "input_tokens": 2852, + "cached_input_tokens": 1536, + "output_tokens": 856, + "total_tokens": 3708, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": "tmpl_741cac2776e9", + "session_id": "sess_0b6f5a82cc81", + "created_at": "2026-08-23T18:59:36.534582+00:00" + } + ], + "narrative.story_brief": [ + { + "event_id": 79, + "operation": "narrative.story_brief", + "status": "success", + "source_label": "live", + "latency_ms": 2031, + "operation_latency_ms": 2031, + "input_tokens": 1763, + "cached_input_tokens": 1664, + "output_tokens": 151, + "total_tokens": 1914, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": null, + "session_id": null, + "created_at": "2026-08-23T18:59:23.068528+00:00" + } + ] + }, + "session_llm_events": [ + { + "event_id": 80, + "operation": "narrative.opening", + "status": "success", + "source_label": "live", + "latency_ms": 13444, + "operation_latency_ms": 13456, + "input_tokens": 2852, + "cached_input_tokens": 1536, + "output_tokens": 856, + "total_tokens": 3708, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": "tmpl_741cac2776e9", + "session_id": "sess_0b6f5a82cc81", + "created_at": "2026-08-23T18:59:36.534582+00:00" + }, + { + "event_id": 81, + "operation": "narrative.advance_turn", + "status": "success", + "source_label": "live", + "latency_ms": 9969, + "operation_latency_ms": 9977, + "input_tokens": 7141, + "cached_input_tokens": 5376, + "output_tokens": 543, + "total_tokens": 7684, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": "tmpl_741cac2776e9", + "session_id": "sess_0b6f5a82cc81", + "created_at": "2026-08-23T18:59:46.530862+00:00" + }, + { + "event_id": 82, + "operation": "narrative.advance_turn", + "status": "success", + "source_label": "live", + "latency_ms": 11522, + "operation_latency_ms": 11537, + "input_tokens": 7364, + "cached_input_tokens": 5376, + "output_tokens": 674, + "total_tokens": 8038, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": "tmpl_741cac2776e9", + "session_id": "sess_0b6f5a82cc81", + "created_at": "2026-08-23T18:59:58.083862+00:00" + }, + { + "event_id": 83, + "operation": "narrative.advance_turn", + "status": "success", + "source_label": "live", + "latency_ms": 9870, + "operation_latency_ms": 9882, + "input_tokens": 7602, + "cached_input_tokens": 5888, + "output_tokens": 585, + "total_tokens": 8187, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": "tmpl_741cac2776e9", + "session_id": "sess_0b6f5a82cc81", + "created_at": "2026-08-23T19:00:07.981294+00:00" + }, + { + "event_id": 84, + "operation": "narrative.advance_turn", + "status": "success", + "source_label": "live", + "latency_ms": 9690, + "operation_latency_ms": 9703, + "input_tokens": 7856, + "cached_input_tokens": 6144, + "output_tokens": 510, + "total_tokens": 8366, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": "tmpl_741cac2776e9", + "session_id": "sess_0b6f5a82cc81", + "created_at": "2026-08-23T19:00:17.703285+00:00" + }, + { + "event_id": 85, + "operation": "narrative.advance_turn", + "status": "success", + "source_label": "live", + "latency_ms": 10819, + "operation_latency_ms": 10831, + "input_tokens": 8092, + "cached_input_tokens": 6272, + "output_tokens": 553, + "total_tokens": 8645, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": "tmpl_741cac2776e9", + "session_id": "sess_0b6f5a82cc81", + "created_at": "2026-08-23T19:00:28.550856+00:00" + }, + { + "event_id": 86, + "operation": "narrative.advance_turn", + "status": "success", + "source_label": "live", + "latency_ms": 11578, + "operation_latency_ms": 11594, + "input_tokens": 8368, + "cached_input_tokens": 6528, + "output_tokens": 716, + "total_tokens": 9084, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": "tmpl_741cac2776e9", + "session_id": "sess_0b6f5a82cc81", + "created_at": "2026-08-23T19:00:40.163985+00:00" + }, + { + "event_id": 87, + "operation": "narrative.advance_turn", + "status": "success", + "source_label": "live", + "latency_ms": 12060, + "operation_latency_ms": 12070, + "input_tokens": 8605, + "cached_input_tokens": 6784, + "output_tokens": 639, + "total_tokens": 9244, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": "tmpl_741cac2776e9", + "session_id": "sess_0b6f5a82cc81", + "created_at": "2026-08-23T19:00:52.253386+00:00" + }, + { + "event_id": 88, + "operation": "narrative.advance_turn", + "status": "success", + "source_label": "live", + "latency_ms": 14858, + "operation_latency_ms": 14868, + "input_tokens": 8843, + "cached_input_tokens": 7040, + "output_tokens": 911, + "total_tokens": 9754, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": "tmpl_741cac2776e9", + "session_id": "sess_0b6f5a82cc81", + "created_at": "2026-08-23T19:01:07.138146+00:00" + }, + { + "event_id": 89, + "operation": "narrative.advance_turn", + "status": "success", + "source_label": "live", + "latency_ms": 12174, + "operation_latency_ms": 12184, + "input_tokens": 8899, + "cached_input_tokens": 5376, + "output_tokens": 667, + "total_tokens": 9566, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": "tmpl_741cac2776e9", + "session_id": "sess_0b6f5a82cc81", + "created_at": "2026-08-23T19:01:19.342688+00:00" + }, + { + "event_id": 90, + "operation": "narrative.advance_turn", + "status": "success", + "source_label": "live", + "latency_ms": 13166, + "operation_latency_ms": 13181, + "input_tokens": 8922, + "cached_input_tokens": 5632, + "output_tokens": 779, + "total_tokens": 9701, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": "tmpl_741cac2776e9", + "session_id": "sess_0b6f5a82cc81", + "created_at": "2026-08-23T19:01:32.541823+00:00" + }, + { + "event_id": 91, + "operation": "narrative.advance_turn", + "status": "success", + "source_label": "live", + "latency_ms": 11284, + "operation_latency_ms": 11297, + "input_tokens": 8987, + "cached_input_tokens": 5632, + "output_tokens": 571, + "total_tokens": 9558, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": "tmpl_741cac2776e9", + "session_id": "sess_0b6f5a82cc81", + "created_at": "2026-08-23T19:01:43.857931+00:00" + }, + { + "event_id": 92, + "operation": "narrative.advance_turn", + "status": "success", + "source_label": "live", + "latency_ms": 9957, + "operation_latency_ms": 28125, + "input_tokens": 9026, + "cached_input_tokens": 5632, + "output_tokens": 517, + "total_tokens": 9543, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": "tmpl_741cac2776e9", + "session_id": "sess_0b6f5a82cc81", + "created_at": "2026-08-23T19:02:12.003074+00:00" + }, + { + "event_id": 93, + "operation": "narrative.ending", + "status": "success", + "source_label": "live", + "latency_ms": 10414, + "operation_latency_ms": 28125, + "input_tokens": 4826, + "cached_input_tokens": 0, + "output_tokens": 429, + "total_tokens": 5255, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": "tmpl_741cac2776e9", + "session_id": "sess_0b6f5a82cc81", + "created_at": "2026-08-23T19:02:12.004240+00:00" + }, + { + "event_id": 94, + "operation": "narrative.branches", + "status": "success", + "source_label": "live", + "latency_ms": 7358, + "operation_latency_ms": 28125, + "input_tokens": 6579, + "cached_input_tokens": 0, + "output_tokens": 365, + "total_tokens": 6944, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": "tmpl_741cac2776e9", + "session_id": "sess_0b6f5a82cc81", + "created_at": "2026-08-23T19:02:12.005383+00:00" + }, + { + "event_id": 95, + "operation": "narrative.highlights", + "status": "success", + "source_label": "live", + "latency_ms": 7726, + "operation_latency_ms": 28125, + "input_tokens": 5048, + "cached_input_tokens": 0, + "output_tokens": 585, + "total_tokens": 5633, + "retry_count": 0, + "repair_count": 0, + "fallback_reason": null, + "template_id": "tmpl_741cac2776e9", + "session_id": "sess_0b6f5a82cc81", + "created_at": "2026-08-23T19:02:12.006343+00:00" + } + ], + "quality_fun_gate": { + "schema_version": "tiny_stories_golden_path_quality.v2", + "status": "pass", + "criteria": { + "consequence_clarity": { + "status": "pass", + "evidence": { + "step_judge": { + "pass": 12, + "warn": 0, + "fail": 0, + "unknown": 0 + }, + "contract_judge": { + "pass": 12, + "warn": 0, + "fail": 0, + "unknown": 0 + } + } + }, + "choice_diversity": { + "status": "pass", + "evidence": { + "unique_chosen_option_labels": 12, + "chosen_option_labels": [ + "[confront] tell arthur the rigging stops now.", + "[yield] step back and let the rigged reveal proceed to protect elena.", + "[counter] pull out the burner phone and threaten to expose arthur.", + "[counter] show arthur the screen and threaten to send the texts to the press", + "[yield] lower the phone and negotiate a cover-up", + "[counter] reveal that you know about arthur's past sponsorship scandals and demand he handle the journalist himself.", + "[ally] propose to elena that you both find leverage against arthur together.", + "[counter] show elena the texts and plan to leak them together.", + "[ally] suggest to elena that you both confront arthur together, presenting a united front.", + "[confront] step forward and lay out the evidence, demanding arthur back off.", + "[ally] plan with elena to release the evidence later, when you can control the narrative.", + "[ally] leave the gala with elena and plan the leak from a safe distance." + ] + } + }, + "escalation": { + "status": "pass", + "evidence": { + "stage_phases": [ + "hook", + "pressure", + "pressure", + "pressure", + "pressure", + "reversal", + "climax", + "climax", + "climax", + "climax", + "pre_finale", + "pre_finale_open" + ], + "distinct_stage_phases": [ + "climax", + "hook", + "pre_finale", + "pre_finale_open", + "pressure", + "reversal" + ] + } + }, + "character_intent": { + "status": "pass", + "evidence": { + "mode": "story", + "active_npc_turns": 0, + "active_npc_turns_required": 0, + "focused_npc_turns": 12, + "responsive_npc_turns": 12, + "responsive_npc_turns_required": 9, + "shifted_npc_turns": 12, + "shifted_npc_turns_required": 6, + "distinct_npc_ids": [ + "host", + "singer", + "sponsor" + ], + "distinct_npc_ids_required": 2, + "plan_count": 12 + } + }, + "brief_payoff": { + "status": "pass", + "evidence": { + "ending_present": true, + "payoff_terms_seen": [ + "gala", + "award", + "sponsor", + "trophy" + ] + } + }, + "playable_options": { + "status": "pass", + "evidence": { + "non_final_option_counts": [ + 3, + 3, + 3, + 3, + 3, + 3, + 3, + 3, + 3, + 3, + 3 + ], + "min_non_final_option_count": 3 + } + } + }, + "rationale": "Deterministic quality packaging only: it checks judge pass/fail rows, choice variety, stage escalation, mode-aware character response, ending payoff, and option availability. It is not a calibrated fun metric." + }, + "ending": { + "label": "共谋", + "subtitle": "我握着证据，等天亮，等火烧起来。", + "passage": "The service door clicks shut behind you, sealing away the gala's golden noise. The alley is a tunnel of cold concrete and the distant wail of a siren, but Elena's hand is warm in yours. You don't look back. You've left Arthur in the ballroom, his laughter still ringing false, his threats still hanging in the air. But out here, under the sodium-orange streetlights, the city feels enormous and indifferent—and for the first time in hours, you can breathe.\n\nIn your apartment, the curtains drawn against the night, you and Elena sit on the floor with the burner phone between you. The texts glow like embers, each one a match that could set Arthur's empire ablaze. She reads them again, her finger tracing the screen, her face a study in fear and resolve. 'We can't just sit on this,' she whispers. 'He'll come for us.'\n\nYou know she's right. Arthur Sterling doesn't lose. He waits, he watches, and he strikes when you least expect it. But tonight, you have something he doesn't: the truth, and the will to use it.\n\nYou draft the message to the journalist from the Daily Ledger—the one who was already sniffing around. You don't send it. Not yet. You save it as a draft, a loaded gun on the nightstand. Then you lean back against the couch, Elena's head on your shoulder, the city's hum a lullaby of distant traffic. The award sits in her bag, tarnished by the lie, but yours now. Not as a trophy, but as a weapon.\n\nTomorrow, the narrative will be yours. Tonight, you are not Arthur's fixer. You are not the keeper of his secrets. You are the one who holds the match, and the whole house of cards is waiting to burn.", + "tier": "compromised", + "early_terminated": false, + "failure_trigger": null, + "highlights": [ + { + "beat_ord": 2, + "headline": "You refuse the rigging and con", + "body_excerpt": "You step toward Arthur, your voice low but sharp. \"The rigging stops now. We're not going through with this.\" Arthur's jaw tightens; he glances over his shoulder at the stage, where Marcus is now holding the sealed envelope, the live audience's applause swelling.", + "why_pivotal": "Your first open defiance shifts the power dynamic, revealing your leverage over Arthur and setting the stage for the escalating conflict." + }, + { + "beat_ord": 6, + "headline": "You reveal the burner phone", + "body_excerpt": "You pull out the burner phone, thumb hovering over the screen. Arthur's eyes drop to it, his smirk faltering. \"You wouldn't,\" he says, but his voice cracks, and you know you've hit something.", + "why_pivotal": "You brandish your key asset—the incriminating texts—turning the tables and exposing Arthur's vulnerability." + }, + { + "beat_ord": 14, + "headline": "Elena joins your side", + "body_excerpt": "You pull Elena deeper into the shadows behind the curtain, your voice a whisper. \"We find something on Arthur. Something he can't bury. Then we're not his puppets anymore.\" Elena stares at you, her eyes wide with a mix of fear and a spark of desperate hope.", + "why_pivotal": "Elena shifts from wary to aligned, becoming your ally and strengthening your position against Arthur." + }, + { + "beat_ord": 18, + "headline": "United front against Arthur", + "body_excerpt": "You step out of the shadows, Elena at your side, and Arthur's eyes narrow as you approach. \"What's this? A mutiny?\" His voice is a cold blade, but you see the flicker of unease behind his bravado.", + "why_pivotal": "You and Elena confront Arthur together, solidifying your alliance and delivering a direct challenge to his control." + }, + { + "beat_ord": 20, + "headline": "You threaten to expose everyth", + "body_excerpt": "You step forward, the phone raised like a shield, the screen's glow casting your face in cold blue. 'Back off, Arthur. I have everything—the texts, the payment trails, the names. One tap and it's over.'", + "why_pivotal": "You escalate to an ultimatum, leveraging your full inventory to force Arthur's retreat and secure your freedom." + } + ], + "branches": [ + { + "pivot_beat_ord": 2, + "chosen_path_summary": "Yielded and let the rigged reveal proceed.", + "alternate_path_summary": "Show the burner phone and threaten to leak the texts.", + "alternate_ending_label": "反噬", + "alternate_ending_tier": "collapsed", + "rationale": "If you had countered with the phone, Arthur likely would have called your bluff, leading to an immediate public scandal that destroys Elena's career and leaves you blacklisted—a chaotic implosion." + }, + { + "pivot_beat_ord": 6, + "chosen_path_summary": "Pulled out the phone and threatened Arthur.", + "alternate_path_summary": "Put the phone away and let Arthur think he's won.", + "alternate_ending_label": "沉沦", + "alternate_ending_tier": "collapsed", + "rationale": "By yielding there, you would have surrendered all leverage. Arthur would tighten his grip, forcing you into deeper cover-ups and compromises, dragging you and Elena into a permanent state of." + }, + { + "pivot_beat_ord": 12, + "chosen_path_summary": "Proposed to Elena to find leverage together.", + "alternate_path_summary": "Follow Arthur to see if he's really handling the journalist.", + "alternate_ending_label": "孤狼", + "alternate_ending_tier": "compromised", + "rationale": "If you had followed Arthur alone, you would have left Elena behind and pursued the truth independently. That path likely leads to you uncovering his schemes but acting without allies, ending in a." + } + ] + }, + "reviewer_evidence": { + "agent_event_count": 36, + "step_judge_count": 12, + "contract_judge_count": 12, + "session_llm_event_count": 16 + }, + "runtime_db": "/tmp/tiny-stories-research-runtime-live-20260824.sqlite3", + "failures": [], + "notes": [ + "This is strict live acceptance evidence for one canonical 12-turn Tiny Stories run.", + "Required Story Butler, Story Brief, opening, and Play turn operations must be live/live_repaired with no fallback_reason.", + "The quality/fun gate is deterministic product evidence, not a calibrated research metric." + ] +} diff --git a/docs/evidence/tiny-stories-12-turn-live-research-runtime-2026-08-24.md b/docs/evidence/tiny-stories-12-turn-live-research-runtime-2026-08-24.md new file mode 100644 index 00000000..18144560 --- /dev/null +++ b/docs/evidence/tiny-stories-12-turn-live-research-runtime-2026-08-24.md @@ -0,0 +1,37 @@ +# Tiny Stories 12-Turn Live Golden Path + +- Generated: `2026-08-23T19:02:12.038317+00:00` +- Status: `pass` +- Session: `sess_0b6f5a82cc81` +- Template: `tmpl_741cac2776e9` +- Turn result: `12/12` +- Ending reached: `True` +- Quality gate: `pass` + +## Live Turn Telemetry + +| # | source | status | latency_ms | op_latency_ms | input | cached | output | total | fallback | +|---:|---|---|---:|---:|---:|---:|---:|---:|---| +| 1 | `live` | `success` | 9969 | 9977 | 7141 | 5376 | 543 | 7684 | none | +| 2 | `live` | `success` | 11522 | 11537 | 7364 | 5376 | 674 | 8038 | none | +| 3 | `live` | `success` | 9870 | 9882 | 7602 | 5888 | 585 | 8187 | none | +| 4 | `live` | `success` | 9690 | 9703 | 7856 | 6144 | 510 | 8366 | none | +| 5 | `live` | `success` | 10819 | 10831 | 8092 | 6272 | 553 | 8645 | none | +| 6 | `live` | `success` | 11578 | 11594 | 8368 | 6528 | 716 | 9084 | none | +| 7 | `live` | `success` | 12060 | 12070 | 8605 | 6784 | 639 | 9244 | none | +| 8 | `live` | `success` | 14858 | 14868 | 8843 | 7040 | 911 | 9754 | none | +| 9 | `live` | `success` | 12174 | 12184 | 8899 | 5376 | 667 | 9566 | none | +| 10 | `live` | `success` | 13166 | 13181 | 8922 | 5632 | 779 | 9701 | none | +| 11 | `live` | `success` | 11284 | 11297 | 8987 | 5632 | 571 | 9558 | none | +| 12 | `live` | `success` | 9957 | 28125 | 9026 | 5632 | 517 | 9543 | none | + +## Quality / Fun Gate + +Deterministic quality packaging only: it checks judge pass/fail rows, choice variety, stage escalation, mode-aware character response, ending payoff, and option availability. It is not a calibrated fun metric. + +- `consequence_clarity`: `pass` +- `choice_diversity`: `pass` +- `escalation`: `pass` +- `character_intent`: `pass` +- `brief_payoff`: `pass` +- `playable_options`: `pass` diff --git a/frontend2/.gitignore b/frontend2/.gitignore new file mode 100644 index 00000000..e985853e --- /dev/null +++ b/frontend2/.gitignore @@ -0,0 +1 @@ +.vercel diff --git a/frontend2/.vercelignore b/frontend2/.vercelignore new file mode 100644 index 00000000..b7aaa735 --- /dev/null +++ b/frontend2/.vercelignore @@ -0,0 +1,6 @@ +node_modules/ +dist/ +.vite/ +test-results/ +.vercel/ +*.log diff --git a/frontend2/src/api/client.ts b/frontend2/src/api/client.ts index 0a96d04d..f4e166f5 100644 --- a/frontend2/src/api/client.ts +++ b/frontend2/src/api/client.ts @@ -88,6 +88,8 @@ export type FrontendApiClient = { options?: NarrativeAgentTraceOptions, ): Promise getNarrativeLLMEvents(sessionId: string): Promise + getNarrativeEvaluationBundle(sessionId: string): Promise + evaluatePortableRpgRun(bundle: unknown): Promise advanceNarrativeTurn( sessionId: string, request: NarrativeAdvanceTurnRequest, diff --git a/frontend2/src/api/contracts.ts b/frontend2/src/api/contracts.ts index 2307ba52..1baba786 100644 --- a/frontend2/src/api/contracts.ts +++ b/frontend2/src/api/contracts.ts @@ -1111,6 +1111,7 @@ export type NarrativePublicReplayResponse = { template_seed: string template_title_i18n?: NarrativeLocalizedText | null template_summary_i18n?: NarrativeLocalizedText | null + language?: NarrativeTemplateLanguage cover_image_url?: string | null cast: NarrativeCastMember[] advisor_persona: string @@ -1133,6 +1134,7 @@ export type NarrativeCreateTemplateRequest = { difficulty?: NarrativeDifficulty language?: NarrativeTemplateLanguage story_brief?: NarrativeStoryBrief | null + story_guide_context?: NarrativeStoryGuideCompressedContext | null } export type NarrativeStartSessionRequest = { diff --git a/frontend2/src/api/http-client.ts b/frontend2/src/api/http-client.ts index 69912ce4..31aa9c9b 100644 --- a/frontend2/src/api/http-client.ts +++ b/frontend2/src/api/http-client.ts @@ -396,6 +396,18 @@ export function createHttpApiClient(baseUrl: string): FrontendApiClient { ) }, + getNarrativeEvaluationBundle(sessionId: string) { + return requestJson("getNarrativeEvaluationBundle", { + params: { session_id: sessionId }, + }) + }, + + evaluatePortableRpgRun(bundle: unknown) { + return requestJson("evaluatePortableRpgRun", { + body: bundle, + }) + }, + advanceNarrativeTurn( sessionId: string, request: NarrativeAdvanceTurnRequest, diff --git a/frontend2/src/api/route-map.ts b/frontend2/src/api/route-map.ts index 05f58090..d2b4c906 100644 --- a/frontend2/src/api/route-map.ts +++ b/frontend2/src/api/route-map.ts @@ -42,6 +42,14 @@ export const BACKEND_ROUTE_MAP = { method: "GET", path: "/narrative/sessions/:session_id/llm-events", }, + getNarrativeEvaluationBundle: { + method: "GET", + path: "/narrative/sessions/:session_id/evaluation-bundle", + }, + evaluatePortableRpgRun: { + method: "POST", + path: "/research/rpg-evaluations", + }, advanceNarrativeTurn: { method: "POST", path: "/narrative/sessions/:session_id/story/turns", diff --git a/frontend2/src/app/app.tsx b/frontend2/src/app/app.tsx index 25dd2091..08c9e63c 100644 --- a/frontend2/src/app/app.tsx +++ b/frontend2/src/app/app.tsx @@ -21,6 +21,7 @@ import { ReplayFixture } from "../pages/replay/replay-fixture" import { TemplateDetailPage } from "../pages/world/world-detail-page" import { PortfolioPage } from "../pages/portfolio/portfolio-page" import { ReviewerPage } from "../pages/portfolio/reviewer-page" +import { RpgEvaluationPage } from "../pages/evaluation/rpg-evaluation-page" function NotFoundRedirect({ navigate }: { navigate: (next: AppRoute) => void }) { useEffect(() => { @@ -136,6 +137,7 @@ function renderRoute(route: AppRoute, navigate: (next: AppRoute) => void) { onOpenCreate={() => navigate({ name: "create" })} onOpenReviewer={() => navigate({ name: "reviewer" })} onOpenPublicDemo={() => navigate({ name: "publicReviewerDemo" })} + onOpenEvaluation={() => navigate({ name: "rpgEvaluation" })} /> ) case "reviewer": @@ -150,6 +152,8 @@ function renderRoute(route: AppRoute, navigate: (next: AppRoute) => void) { ) case "publicReviewerDemo": return navigate({ name: "portfolio" })} /> + case "rpgEvaluation": + return navigate({ name: "portfolio" })} /> case "about": return ( = { portfolio: 1, reviewer: 1, publicReviewerDemo: 1, + rpgEvaluation: 1, playAdvisorFixture: 1, playActionFixture: 1, playEndingFixture: 1, @@ -67,7 +69,9 @@ function parseRoute(hash: string): AppRoute { const params = new URLSearchParams(search) if (segments.length === 0) { - return { name: "home" } + return import.meta.env.VITE_PUBLIC_LAB_DEFAULT === "true" + ? { name: "rpgEvaluation" } + : { name: "home" } } if (segments[0] === "login") { return { name: "login", next: params.get("next") ?? undefined } @@ -115,6 +119,9 @@ function parseRoute(hash: string): AppRoute { if (segments[0] === "demo" && segments[1] === "reviewer") { return { name: "publicReviewerDemo" } } + if (segments[0] === "lab" && segments[1] === "rpg-evaluation") { + return { name: "rpgEvaluation", sessionId: params.get("session") ?? undefined } + } if (segments[0] === "about") { return { name: "about" } } @@ -169,6 +176,10 @@ export function buildHash(route: AppRoute): string { return "#/reviewer" case "publicReviewerDemo": return "#/demo/reviewer" + case "rpgEvaluation": + return route.sessionId + ? `#/lab/rpg-evaluation?session=${encodeURIComponent(route.sessionId)}` + : "#/lab/rpg-evaluation" case "about": return "#/about" } @@ -196,9 +207,12 @@ export function useAppRoute() { } window.addEventListener("hashchange", onChange) if (!window.location.hash) { - window.history.replaceState(null, "", buildHash({ name: "home" })) - setRoute({ name: "home" }) - prevRouteRef.current = { name: "home" } + const defaultRoute: AppRoute = import.meta.env.VITE_PUBLIC_LAB_DEFAULT === "true" + ? { name: "rpgEvaluation" } + : { name: "home" } + window.history.replaceState(null, "", buildHash(defaultRoute)) + setRoute(defaultRoute) + prevRouteRef.current = defaultRoute } return () => window.removeEventListener("hashchange", onChange) }, []) diff --git a/frontend2/src/app/theme.css b/frontend2/src/app/theme.css index e83fa443..63e48954 100644 --- a/frontend2/src/app/theme.css +++ b/frontend2/src/app/theme.css @@ -2083,3 +2083,220 @@ kbd { font-size: 12px; } } + +/* RPG Runtime Evaluation Lab + * Public, backend-free reviewer surface. It shares Tiny Stories' graphite, + * ivory, and amber language while keeping evaluation data denser than Play. + */ +.rpg-lab-page { + --lab-panel: #111318; + --lab-panel-raised: #171a20; + --lab-ink: #f2eee5; + --lab-muted: rgba(242, 238, 229, 0.62); + --lab-line: rgba(242, 238, 229, 0.13); + --lab-amber: #d6aa5d; + --lab-green: #72b99a; + --lab-red: #d46f61; + min-height: 100vh; + color: var(--lab-ink); + background: + linear-gradient(rgba(255, 255, 255, 0.018) 1px, transparent 1px), + #090b0e; + background-size: 100% 56px; +} + +.rpg-lab-topbar { + position: sticky; + top: 0; + z-index: 10; + display: grid; + grid-template-columns: 48px minmax(0, 1fr) auto; + align-items: center; + min-height: 68px; + padding: 0 28px; + border-bottom: 1px solid var(--lab-line); + background: rgba(9, 11, 14, 0.94); + backdrop-filter: blur(18px); +} + +.rpg-lab-back { + width: 34px; + height: 34px; + color: var(--lab-muted); + font-size: 22px; + transition: color 150ms ease, transform 150ms ease; +} +.rpg-lab-back:hover { color: var(--lab-ink); transform: translateX(-2px); } +.rpg-lab-brand { display: grid; gap: 2px; } +.rpg-lab-brand strong { font-family: var(--font-narrative); font-size: 18px; font-weight: 600; } +.rpg-lab-brand span { color: var(--lab-muted); font-size: 10px; letter-spacing: 0.12em; text-transform: uppercase; } +.rpg-lab-language { display: flex; gap: 14px; } +.rpg-lab-language button { padding: 6px 0; color: var(--lab-muted); font-size: 12px; border-bottom: 1px solid transparent; } +.rpg-lab-language button.is-active { color: var(--lab-amber); border-bottom-color: var(--lab-amber); } + +.rpg-lab-main { + width: min(1320px, calc(100% - 56px)); + margin: 0 auto; + padding: 54px 0 88px; +} + +.rpg-lab-hero { max-width: 940px; padding-bottom: 34px; } +.rpg-lab-hero > span { color: var(--lab-amber); font-size: 11px; font-weight: 700; letter-spacing: 0.14em; text-transform: uppercase; } +.rpg-lab-hero h1 { margin: 14px 0 12px; font-family: var(--font-narrative); font-size: clamp(38px, 5vw, 68px); font-weight: 500; line-height: 1.02; letter-spacing: 0; } +.rpg-lab-hero > p { max-width: 760px; margin: 0; color: var(--lab-muted); font-size: 16px; line-height: 1.65; } +.rpg-lab-boundary { margin-top: 22px; padding: 12px 0 12px 16px; border-left: 2px solid var(--lab-amber); color: rgba(242, 238, 229, 0.75); font-size: 12px; line-height: 1.55; } + +.rpg-lab-controls { + display: grid; + grid-template-columns: minmax(0, 1fr) auto; + gap: 18px; + align-items: end; + padding: 20px 0; + border-block: 1px solid var(--lab-line); +} +.rpg-lab-sample-switcher { display: grid; grid-template-columns: repeat(2, minmax(230px, 1fr)); gap: 1px; background: var(--lab-line); } +.rpg-lab-sample-switcher button { min-height: 74px; padding: 14px 16px; text-align: left; background: var(--lab-panel); transition: background 150ms ease; } +.rpg-lab-sample-switcher button:hover { background: var(--lab-panel-raised); } +.rpg-lab-sample-switcher button.is-active { box-shadow: inset 0 -2px 0 var(--lab-amber); background: #1a1814; } +.rpg-lab-sample-switcher span { display: block; color: var(--lab-ink); font-size: 14px; font-weight: 700; } +.rpg-lab-sample-switcher small { display: block; margin-top: 6px; color: var(--lab-muted); font-size: 11px; } +.rpg-lab-import input { display: none; } +.rpg-lab-import button, +.rpg-lab-adapter-actions button { min-height: 42px; padding: 0 16px; border: 1px solid var(--lab-line); color: var(--lab-ink); background: var(--lab-panel); font-size: 12px; font-weight: 700; } +.rpg-lab-import button:hover, +.rpg-lab-adapter-actions button:hover { border-color: rgba(214, 170, 93, 0.55); color: var(--lab-amber); } +.rpg-lab-import-error { grid-column: 1 / -1; margin: 0; color: #f3a398; font-size: 12px; } +.rpg-lab-session-loader { grid-column: 1 / -1; display: grid; gap: 8px; } +.rpg-lab-session-loader > label { color: var(--lab-muted); font-size: 10px; font-weight: 700; letter-spacing: 0.1em; text-transform: uppercase; } +.rpg-lab-session-loader > div { display: grid; grid-template-columns: minmax(0, 1fr) auto; gap: 1px; background: var(--lab-line); } +.rpg-lab-session-loader input { min-width: 0; min-height: 42px; padding: 0 14px; border: 0; color: var(--lab-ink); background: #0e100f; font-size: 13px; outline: none; } +.rpg-lab-session-loader input:focus { box-shadow: inset 0 -2px 0 var(--lab-amber); } +.rpg-lab-session-loader button { min-height: 42px; padding: 0 18px; color: #11110f; background: var(--lab-amber); font-size: 12px; font-weight: 800; } +.rpg-lab-session-loader button:disabled { color: var(--lab-muted); background: var(--lab-panel); cursor: not-allowed; } + +.rpg-lab-scoreboard { display: grid; grid-template-columns: 220px minmax(0, 1fr); min-height: 150px; border-bottom: 1px solid var(--lab-line); } +.rpg-lab-score { display: grid; grid-template-columns: 1fr auto; align-content: center; gap: 4px 14px; padding: 24px 28px 24px 0; border-right: 1px solid var(--lab-line); } +.rpg-lab-score > span { grid-column: 1 / -1; color: var(--lab-muted); font-size: 10px; letter-spacing: 0.12em; text-transform: uppercase; } +.rpg-lab-score > strong { font-family: var(--font-narrative); font-size: 58px; font-weight: 500; line-height: 1; } +.rpg-lab-score > em { align-self: end; margin-bottom: 7px; } +.rpg-lab-scoreboard dl { display: grid; grid-template-columns: 0.8fr 0.8fr 1fr 2fr; margin: 0; } +.rpg-lab-scoreboard dl > div { display: grid; align-content: center; gap: 9px; padding: 22px; border-right: 1px solid var(--lab-line); } +.rpg-lab-scoreboard dl > div:last-child { border-right: none; } +.rpg-lab-scoreboard dt { color: var(--lab-muted); font-size: 10px; letter-spacing: 0.1em; text-transform: uppercase; } +.rpg-lab-scoreboard dd { margin: 0; overflow-wrap: anywhere; font-size: 15px; font-weight: 700; } +.rpg-lab-status { display: inline-flex; width: fit-content; padding: 3px 0; font-size: 10px; font-style: normal; font-weight: 800; letter-spacing: 0.1em; text-transform: uppercase; } +.rpg-lab-status--pass { color: var(--lab-green); } +.rpg-lab-status--warn { color: var(--lab-amber); } +.rpg-lab-status--fail { color: var(--lab-red); } + +.rpg-lab-tabs { display: flex; gap: 28px; min-height: 62px; align-items: end; border-bottom: 1px solid var(--lab-line); } +.rpg-lab-tabs button { min-height: 44px; padding: 0 0 12px; color: var(--lab-muted); font-size: 12px; font-weight: 700; border-bottom: 2px solid transparent; } +.rpg-lab-tabs button.is-active { color: var(--lab-ink); border-bottom-color: var(--lab-amber); } +.rpg-lab-view { padding-top: 30px; animation: rpgLabEnter 180ms ease-out; } +@keyframes rpgLabEnter { from { opacity: 0; transform: translateY(4px); } to { opacity: 1; transform: translateY(0); } } +.rpg-lab-objective { display: grid; grid-template-columns: minmax(0, 1fr) 240px; gap: 28px; align-items: end; padding-bottom: 26px; border-bottom: 1px solid var(--lab-line); } +.rpg-lab-objective > div { display: grid; gap: 8px; } +.rpg-lab-objective > div > span { color: var(--lab-amber); font-size: 10px; letter-spacing: 0.12em; text-transform: uppercase; } +.rpg-lab-objective > div > strong { max-width: 880px; font-family: var(--font-narrative); font-size: 29px; font-weight: 500; line-height: 1.24; } +.rpg-lab-objective > div > p { margin: 0; color: var(--lab-muted); font-size: 12px; } +.rpg-lab-objective figure { position: relative; min-width: 0; height: 128px; margin: 0; overflow: hidden; border-left: 1px solid rgba(214, 170, 93, 0.5); } +.rpg-lab-objective img { width: 100%; height: 100%; object-fit: cover; filter: grayscale(0.35) brightness(0.62) contrast(1.08); } +.rpg-lab-objective figcaption { position: absolute; inset: auto 0 0; padding: 18px 10px 8px; color: rgba(255, 255, 255, 0.72); background: linear-gradient(transparent, rgba(0, 0, 0, 0.78)); font-size: 9px; letter-spacing: 0.08em; text-transform: uppercase; } + +.rpg-lab-overview-grid { display: grid; grid-template-columns: minmax(0, 1.7fr) minmax(280px, 0.8fr); gap: 38px; padding-top: 28px; } +.rpg-lab-section-head { display: flex; align-items: center; gap: 12px; min-height: 34px; margin-bottom: 12px; } +.rpg-lab-section-head > span { color: var(--lab-amber); font-size: 10px; font-weight: 800; } +.rpg-lab-section-head h2 { margin: 0; font-size: 14px; letter-spacing: 0; } +.rpg-lab-criteria article { display: grid; grid-template-columns: minmax(0, 1fr) 54px; gap: 6px 18px; padding: 16px 0; border-top: 1px solid var(--lab-line); } +.rpg-lab-criteria article > div { display: flex; gap: 14px; align-items: center; } +.rpg-lab-criteria article > div strong { font-size: 14px; } +.rpg-lab-criteria article > b { text-align: right; font-family: var(--font-narrative); font-size: 24px; font-weight: 500; } +.rpg-lab-criteria article > p { grid-column: 1 / -1; margin: 0; color: var(--lab-muted); font-size: 12px; line-height: 1.5; } +.rpg-lab-criteria details { grid-column: 1 / -1; color: var(--lab-muted); font-size: 11px; } +.rpg-lab-criteria summary { cursor: pointer; width: fit-content; color: rgba(214, 170, 93, 0.78); } +.rpg-lab-criteria details span { display: block; margin-top: 6px; padding-left: 12px; border-left: 1px solid var(--lab-line); } +.rpg-lab-comparison { border-left: 1px solid var(--lab-line); padding-left: 28px; } +.rpg-lab-comparison > div { display: grid; grid-template-columns: minmax(0, 1fr) auto; gap: 7px 18px; padding: 18px 0; border-top: 1px solid var(--lab-line); opacity: 0.68; } +.rpg-lab-comparison > div.is-active { opacity: 1; } +.rpg-lab-comparison > div > span { font-family: var(--font-narrative); font-size: 28px; } +.rpg-lab-comparison i { grid-column: 1 / -1; height: 3px; background: rgba(255, 255, 255, 0.07); } +.rpg-lab-comparison i b { display: block; height: 100%; background: var(--lab-amber); } +.rpg-lab-comparison small, +.rpg-lab-comparison > p { color: var(--lab-muted); font-size: 11px; line-height: 1.5; } +.rpg-lab-comparison > p { margin: 28px 0 0; padding-top: 16px; border-top: 1px solid var(--lab-line); } + +.rpg-lab-trace { display: grid; gap: 0; } +.rpg-lab-trace article { display: grid; grid-template-columns: 72px minmax(0, 1fr) minmax(260px, 0.62fr); gap: 18px 28px; padding: 26px 0; border-bottom: 1px solid var(--lab-line); } +.rpg-lab-trace article > header { grid-row: 1 / 5; display: flex; flex-direction: column; gap: 10px; padding-right: 18px; border-right: 1px solid var(--lab-line); } +.rpg-lab-trace article > header span { color: var(--lab-amber); font-family: var(--font-narrative); font-size: 30px; } +.rpg-lab-trace article > header strong { color: var(--lab-muted); font-size: 11px; } +.rpg-lab-trace-copy { grid-column: 2; } +.rpg-lab-trace-copy > span { color: var(--lab-muted); font-size: 9px; letter-spacing: 0.1em; text-transform: uppercase; } +.rpg-lab-trace-copy p { margin: 5px 0 0; font-size: 14px; line-height: 1.56; } +.rpg-lab-deltas { grid-column: 2; display: flex; flex-wrap: wrap; gap: 8px; } +.rpg-lab-deltas span { padding: 5px 8px; border-left: 2px solid var(--lab-amber); color: #eed09a; background: rgba(214, 170, 93, 0.07); font-size: 10px; } +.rpg-lab-trace ol { grid-column: 3; grid-row: 1 / 5; margin: 0; padding: 0; list-style: none; counter-reset: action; } +.rpg-lab-trace li { counter-increment: action; padding: 11px 0 11px 34px; border-bottom: 1px solid var(--lab-line); color: var(--lab-muted); font-size: 12px; line-height: 1.4; position: relative; } +.rpg-lab-trace li::before { content: counter(action); position: absolute; left: 0; color: var(--lab-amber); font-family: var(--font-narrative); font-size: 18px; } + +.rpg-lab-turn-selector { display: flex; gap: 10px; margin-bottom: 22px; } +.rpg-lab-turn-selector button { width: 42px; height: 34px; border-bottom: 1px solid var(--lab-line); color: var(--lab-muted); font-size: 11px; } +.rpg-lab-turn-selector button.is-active { color: var(--lab-amber); border-color: var(--lab-amber); } +.rpg-lab-memory-grid { display: grid; grid-template-columns: repeat(2, minmax(0, 1fr)); border-top: 1px solid var(--lab-line); border-left: 1px solid var(--lab-line); } +.rpg-lab-memory-grid > section { min-height: 260px; padding: 22px; border-right: 1px solid var(--lab-line); border-bottom: 1px solid var(--lab-line); } +.rpg-lab-memory-grid p { display: grid; gap: 3px; margin: 0; padding: 10px 0; border-top: 1px solid var(--lab-line); font-size: 12px; } +.rpg-lab-memory-grid p b { color: var(--lab-amber); font-size: 10px; text-transform: uppercase; } +.rpg-lab-memory-grid p span, +.rpg-lab-memory-grid em, +.rpg-lab-memory-grid small { color: var(--lab-muted); font-style: normal; line-height: 1.5; } +.rpg-lab-memory-grid h3 { margin: 22px 0 8px; color: var(--lab-muted); font-size: 10px; letter-spacing: 0.1em; text-transform: uppercase; } +.rpg-lab-memory-grid small { display: block; padding: 5px 0; } + +.rpg-lab-adapter { max-width: 960px; } +.rpg-lab-adapter > p { max-width: 720px; color: var(--lab-muted); line-height: 1.6; } +.rpg-lab-adapter-flow { display: grid; grid-template-columns: 1fr auto 1fr auto 1fr auto 1fr; align-items: center; gap: 12px; margin: 28px 0; } +.rpg-lab-adapter-flow span { padding: 16px; border-block: 1px solid var(--lab-line); text-align: center; font-size: 12px; font-weight: 700; } +.rpg-lab-adapter-flow i { color: var(--lab-amber); font-style: normal; } +.rpg-lab-adapter pre { max-width: 100%; overflow: auto; padding: 20px; border: 1px solid var(--lab-line); color: #d9e7dc; background: #0a0d0c; font: 12px/1.6 Menlo, Consolas, monospace; } +.rpg-lab-adapter-actions { display: flex; flex-wrap: wrap; gap: 10px; margin-top: 18px; } + +@media (max-width: 760px) { + .rpg-lab-topbar { min-height: 60px; padding: 0 16px; grid-template-columns: 36px minmax(0, 1fr) auto; } + .rpg-lab-main { width: min(100% - 30px, 680px); padding: 34px 0 64px; } + .rpg-lab-hero { padding-bottom: 24px; } + .rpg-lab-hero h1 { font-size: 38px; } + .rpg-lab-hero > p { font-size: 14px; } + .rpg-lab-controls { grid-template-columns: 1fr; } + .rpg-lab-sample-switcher { grid-template-columns: 1fr; } + .rpg-lab-import button { width: 100%; } + .rpg-lab-session-loader > div { grid-template-columns: 1fr; } + .rpg-lab-session-loader button { width: 100%; } + .rpg-lab-scoreboard { grid-template-columns: 1fr; } + .rpg-lab-score { grid-template-columns: 1fr auto; padding: 22px 0; border-right: none; border-bottom: 1px solid var(--lab-line); } + .rpg-lab-scoreboard dl { grid-template-columns: repeat(2, minmax(0, 1fr)); } + .rpg-lab-scoreboard dl > div { min-height: 92px; padding: 16px 12px; border-bottom: 1px solid var(--lab-line); } + .rpg-lab-scoreboard dl > div:nth-child(2) { border-right: none; } + .rpg-lab-tabs { gap: 20px; overflow-x: auto; } + .rpg-lab-tabs button { flex: 0 0 auto; } + .rpg-lab-overview-grid { grid-template-columns: 1fr; gap: 28px; } + .rpg-lab-objective { grid-template-columns: 1fr; } + .rpg-lab-objective figure { height: 112px; } + .rpg-lab-comparison { padding-left: 0; border-left: none; } + .rpg-lab-trace article { grid-template-columns: 48px minmax(0, 1fr); gap: 14px 18px; } + .rpg-lab-trace article > header { grid-row: 1 / 5; padding-right: 10px; } + .rpg-lab-trace-copy, + .rpg-lab-deltas, + .rpg-lab-trace ol { grid-column: 2; } + .rpg-lab-trace ol { grid-row: auto; } + .rpg-lab-memory-grid { grid-template-columns: 1fr; } + .rpg-lab-memory-grid > section { min-height: 0; } + .rpg-lab-adapter-flow { grid-template-columns: 1fr; } + .rpg-lab-adapter-flow i { transform: rotate(90deg); text-align: center; } + .rpg-lab-adapter-actions { display: grid; } +} + +@media (prefers-reduced-motion: reduce) { + .rpg-lab-view { animation: none; } + .rpg-lab-back, + .rpg-lab-sample-switcher button { transition: none; } +} diff --git a/frontend2/src/pages/create/create-page.tsx b/frontend2/src/pages/create/create-page.tsx index 9f81ce21..1eee8bd5 100644 --- a/frontend2/src/pages/create/create-page.tsx +++ b/frontend2/src/pages/create/create-page.tsx @@ -421,6 +421,7 @@ export function CreatePage({ difficulty, language: storyLanguage, story_brief: activeBriefResponse?.can_generate ? activeBriefResponse.brief : null, + story_guide_context: guideLoopState.context, }) const openingElapsedMs = Date.now() - startedAt const handoffLabelKey: StringKey | null = diff --git a/frontend2/src/pages/evaluation/rpg-evaluation-contract.ts b/frontend2/src/pages/evaluation/rpg-evaluation-contract.ts new file mode 100644 index 00000000..87f965ad --- /dev/null +++ b/frontend2/src/pages/evaluation/rpg-evaluation-contract.ts @@ -0,0 +1,252 @@ +export type EvaluationStatus = "pass" | "warn" | "fail" + +export type RpgMemoryFact = { + fact_id: string + namespace: string + key: string + value: string + status: "active" | "superseded" + source_turn: number + source: "user" | "runtime" | "system" | "imported" + confidence: number + superseded_by?: string | null +} + +export type RpgMemoryEntity = { + entity_id: string + name: string + state: Record + last_updated_turn: number +} + +export type RpgMemorySnapshot = { + schema_version: "rpg_memory.v1" + run_id: string + turn_index: number + objective: string + active_facts: RpgMemoryFact[] + superseded_facts: RpgMemoryFact[] + open_threads: string[] + entities: RpgMemoryEntity[] + recent_events: string[] + episodic_summary: string + diagnostics: { + event_count: number + active_fact_count: number + superseded_fact_count: number + non_story_event_count: number + dropped_recent_event_count: number + last_compacted_turn: number + } +} + +export type RpgStateDelta = { + target: string + kind: "increase" | "decrease" | "set" | "unlock" | "spend" | "shift" + label: string + value?: string + evidence?: string +} + +export type RpgTurnObservation = { + turn_index: number + player_action: string + world_response: string + options: string[] + state_deltas: RpgStateDelta[] + clue_unlocks: string[] + opportunity_unlocks: string[] + referenced_entity_ids: string[] + terminal?: boolean + objective_progress: number + progress_basis?: "runtime_reported" | "turn_budget_proxy" | "unknown" + memory: RpgMemorySnapshot +} + +export type RpgEvaluationBundle = { + schema_version: "rpg_evaluation_bundle.v1" + run_id: string + system_label: string + locale: "en" | "zh" | "mixed" + scenario: { + scenario_id: string + title: string + genre: string + objective: string + turn_budget: number + entity_ids: string[] + boundaries: string[] + } + turns: RpgTurnObservation[] +} + +export type RpgCriterion = { + criterion: + | "memory_continuity" + | "memory_boundedness" + | "consequence_visibility" + | "player_agency" + | "trajectory_progress" + | "entity_coherence" + | "choice_diversity" + | "boundary_hygiene" + status: EvaluationStatus + score: number + summary: string + evidence: string[] +} + +export type RpgEvaluationReport = { + schema_version: "rpg_evaluation_report.v1" + run_id: string + system_label: string + status: EvaluationStatus + score: number + criteria: RpgCriterion[] + limitations: string[] +} + +const TECHNICAL_LEAK = /\b(provider|model id|api key|schema|token count|chain[- ]of[- ]thought|scratchpad|raw json|fallback_used)\b/i + +function statusFor(score: number): EvaluationStatus { + if (score >= 80) return "pass" + if (score >= 55) return "warn" + return "fail" +} + +function criterion( + name: RpgCriterion["criterion"], + score: number, + summary: string, + evidence: string[], +): RpgCriterion { + const boundedScore = Math.max(0, Math.min(100, Math.round(score))) + return { criterion: name, status: statusFor(boundedScore), score: boundedScore, summary, evidence: evidence.slice(0, 6) } +} + +export function evaluateRpgBundle(bundle: RpgEvaluationBundle): RpgEvaluationReport { + const turns = bundle.turns + const memoryConflicts: string[] = [] + const memoryOverflow: string[] = [] + const consequenceGaps: string[] = [] + const agencyGaps: string[] = [] + const entityGaps: string[] = [] + const leakage: string[] = [] + const optionSignatures = new Set() + let previousProgress = 0 + let progressAdvances = 0 + let progressRegressions = 0 + const progressBases = new Set() + + for (const turn of turns) { + progressBases.add(turn.progress_basis ?? "unknown") + const activeKeys = new Set() + const supersededIds = new Set(turn.memory.superseded_facts.map((fact) => fact.fact_id)) + for (const fact of turn.memory.active_facts) { + const key = `${fact.namespace.toLowerCase()}:${fact.key.toLowerCase()}` + if (activeKeys.has(key)) memoryConflicts.push(`Turn ${turn.turn_index}: duplicate active fact ${fact.namespace}.${fact.key}`) + activeKeys.add(key) + if (fact.status !== "active" || supersededIds.has(fact.fact_id)) { + memoryConflicts.push(`Turn ${turn.turn_index}: superseded fact remained active`) + } + } + if (turn.memory.active_facts.length > 24 || turn.memory.recent_events.length > 8) { + memoryOverflow.push(`Turn ${turn.turn_index}: memory exceeded the portable budget`) + } + if (turn.state_deltas.length + turn.clue_unlocks.length + turn.opportunity_unlocks.length === 0) { + consequenceGaps.push(`Turn ${turn.turn_index}: no typed visible consequence`) + } + const options = turn.options.map((option) => option.trim().toLowerCase()) + if (!turn.terminal && new Set(options).size < 2) agencyGaps.push(`Turn ${turn.turn_index}: fewer than two distinct options`) + if (!turn.terminal && options.length > 0) optionSignatures.add(options.join("|")) + const unknown = turn.referenced_entity_ids.filter((id) => !bundle.scenario.entity_ids.includes(id)) + if (unknown.length > 0) entityGaps.push(`Turn ${turn.turn_index}: unknown entities ${unknown.join(", ")}`) + if (TECHNICAL_LEAK.test([turn.player_action, turn.world_response, ...turn.options].join(" "))) { + leakage.push(`Turn ${turn.turn_index}: technical wording appeared in player-facing text`) + } + if (turn.objective_progress + 0.001 < previousProgress) progressRegressions += 1 + else if (turn.objective_progress > previousProgress + 0.05) progressAdvances += 1 + previousProgress = Math.max(previousProgress, turn.objective_progress) + } + + const denominator = Math.max(1, turns.length) + const nonTerminalTurns = turns.filter((turn) => !turn.terminal).length + const choiceDiversityScore = nonTerminalTurns > 0 + ? Math.min(100, 100 * optionSignatures.size / nonTerminalTurns) + : 55 + const playerAgencyScore = nonTerminalTurns > 0 + ? 100 * (nonTerminalTurns - agencyGaps.length) / nonTerminalTurns + : 55 + const criteria: RpgCriterion[] = [ + criterion("memory_continuity", 100 - memoryConflicts.length * 35, + memoryConflicts.length === 0 ? "Corrections remain auditable without conflicting active facts." : "Active and superseded facts conflict.", + memoryConflicts.length > 0 ? memoryConflicts : [`${turns.length} snapshots checked with no active fact conflict.`]), + criterion("memory_boundedness", 100 - memoryOverflow.length * 40, + memoryOverflow.length === 0 ? "Memory stays inside the portable fact and recency budgets." : "At least one memory snapshot exceeded its budget.", + memoryOverflow.length > 0 ? memoryOverflow : ["Active facts <= 24 and recent events <= 8 for every turn."]), + criterion("consequence_visibility", 100 * (turns.length - consequenceGaps.length) / denominator, + consequenceGaps.length === 0 ? "Moves resolve into visible state changes, clues, or opportunities." : "Some moves produced prose without a visible game-state consequence.", + consequenceGaps.length > 0 ? consequenceGaps : [`All ${turns.length} turns expose a typed consequence.`]), + criterion("player_agency", playerAgencyScore, + nonTerminalTurns === 0 + ? "A terminal-only run has insufficient evidence for player agency." + : agencyGaps.length === 0 + ? "Every turn preserves at least two distinct next actions." + : "At least one turn collapsed into duplicated or singular choice.", + agencyGaps.length > 0 ? agencyGaps : [`Distinct choices preserved across ${nonTerminalTurns} non-terminal turns.`]), + criterion("trajectory_progress", previousProgress * 70 + Math.min(30, progressAdvances * 10) - progressRegressions * 25, + progressRegressions === 0 ? "Reported or explicitly proxied progress advances without unexplained regression." : "Objective progress regressed on at least one turn.", + [`Final observed progress: ${Math.round(previousProgress * 100)}%.`, `Meaningful advances: ${progressAdvances}; regressions: ${progressRegressions}.`, `Progress basis: ${[...progressBases].sort().join(", ")}.`]), + criterion("entity_coherence", 100 - entityGaps.length * 35, + entityGaps.length === 0 ? "People and factions stay inside the scenario registry." : "The run referenced entities outside the scenario contract.", + entityGaps.length > 0 ? entityGaps : [`Validated against ${bundle.scenario.entity_ids.length} registered entities.`]), + criterion("choice_diversity", choiceDiversityScore, + nonTerminalTurns === 0 + ? "A terminal-only run has insufficient evidence for choice diversity." + : optionSignatures.size === nonTerminalTurns + ? "Next-action sets change with the trajectory." + : "Some turns repeated the same full action set.", + [`${optionSignatures.size} distinct action sets across ${nonTerminalTurns} non-terminal turns.`]), + criterion("boundary_hygiene", 100 - leakage.length * 50, + leakage.length === 0 ? "Player-facing text is free of protocol and private-reasoning language." : "Technical implementation wording leaked into player-facing text.", + leakage.length > 0 ? leakage : ["No provider, schema, token, raw JSON, or private-reasoning terms detected."]), + ] + const score = Math.round(criteria.reduce((sum, item) => sum + item.score, 0) / criteria.length) + return { + schema_version: "rpg_evaluation_report.v1", + run_id: bundle.run_id, + system_label: bundle.system_label, + status: statusFor(score), + score, + criteria, + limitations: [ + "This deterministic report is a product reliability diagnostic, not a calibrated research metric.", + "Narrative appeal and emotional quality still require bounded human review.", + "Imported state deltas are only as trustworthy as the source adapter.", + ...(progressBases.has("turn_budget_proxy") + ? ["Trajectory progress for this run uses elapsed turn budget as a proxy, not model-reported goal completion."] + : []), + ], + } +} + +export function isRpgEvaluationBundle(value: unknown): value is RpgEvaluationBundle { + if (!value || typeof value !== "object") return false + const candidate = value as Partial + return candidate.schema_version === "rpg_evaluation_bundle.v1" + && typeof candidate.run_id === "string" + && typeof candidate.system_label === "string" + && Boolean(candidate.scenario && typeof candidate.scenario === "object") + && Array.isArray(candidate.turns) + && candidate.turns.length > 0 +} + +export function isRpgEvaluationReport(value: unknown): value is RpgEvaluationReport { + if (!value || typeof value !== "object") return false + const candidate = value as Partial + return candidate.schema_version === "rpg_evaluation_report.v1" + && typeof candidate.run_id === "string" + && typeof candidate.score === "number" + && Array.isArray(candidate.criteria) + && candidate.criteria.length === 8 +} diff --git a/frontend2/src/pages/evaluation/rpg-evaluation-page.tsx b/frontend2/src/pages/evaluation/rpg-evaluation-page.tsx new file mode 100644 index 00000000..099535f9 --- /dev/null +++ b/frontend2/src/pages/evaluation/rpg-evaluation-page.tsx @@ -0,0 +1,472 @@ +import { useEffect, useMemo, useRef, useState, type ChangeEvent, type FormEvent } from "react" +import { useApi } from "../../app/api-context" +import { useLanguage } from "../../shared/lib/i18n" +import { + evaluateRpgBundle, + isRpgEvaluationBundle, + isRpgEvaluationReport, + type EvaluationStatus, + type RpgEvaluationBundle, + type RpgEvaluationReport, +} from "./rpg-evaluation-contract" +import { RPG_EVALUATION_SAMPLES } from "./rpg-evaluation-samples" + +type LabView = "overview" | "run" | "memory" | "adapter" + +const CRITERION_LABELS = { + memory_continuity: ["记忆连续性", "Memory continuity"], + memory_boundedness: ["记忆有界性", "Memory boundedness"], + consequence_visibility: ["后果可见性", "Consequence visibility"], + player_agency: ["玩家能动性", "Player agency"], + trajectory_progress: ["轨迹推进", "Trajectory progress"], + entity_coherence: ["实体一致性", "Entity coherence"], + choice_diversity: ["选择多样性", "Choice diversity"], + boundary_hygiene: ["边界与界面卫生", "Boundary hygiene"], +} as const + +const UI = { + zh: { + kicker: "研究型运行时 · 公开评测前端", + title: "RPG Runtime Evaluation Lab", + lede: "把不同 RPG Agent 的回合轨迹转成同一个可比较合同：记忆、选择、后果、人物、目标推进与边界。", + boundary: "这是确定性产品诊断，不是已校准的学术质量分。叙事吸引力仍需要有界的人类评审。", + back: "返回作品页", + overview: "总览", + run: "回合轨迹", + memory: "记忆", + adapter: "接入规范", + benchmark: "基准运行", + score: "综合诊断", + turns: "回合", + facts: "有效事实", + corrections: "已覆盖事实", + status: "状态", + objective: "本局目标", + compare: "对照结果", + criteria: "逐项评测", + action: "玩家行动", + reaction: "世界响应", + changes: "明确变化", + next: "下一动作", + active: "当前事实", + superseded: "被覆盖但可审计", + threads: "开放问题", + entities: "人物状态", + recent: "最近事件", + import: "导入 JSON 运行包", + exportBundle: "下载运行包", + exportReport: "下载评测报告", + reset: "恢复内置样例", + invalid: "文件不是 rpg_evaluation_bundle.v1，或缺少回合数据。", + sessionLabel: "评测真实会话", + sessionPlaceholder: "输入 Tiny Stories session id", + loadSession: "载入会话", + loadingSession: "正在整理运行证据...", + sessionError: "无法载入该运行。请确认已用运行所有者登录，并至少完成一个回合。", + adapterTitle: "让其他 RPG 接入", + adapterCopy: "适配器只需输出场景、逐回合行动/响应、选项、结构化变化和记忆快照。评测前端不要求接入 Tiny Stories 后端。", + noValue: "无", + }, + en: { + kicker: "Research runtime · public evaluation frontend", + title: "RPG Runtime Evaluation Lab", + lede: "Normalize RPG-agent runs into one comparable contract: memory, choices, consequences, people, objective progress, and boundaries.", + boundary: "This is a deterministic product diagnostic, not a calibrated academic quality score. Narrative appeal still needs bounded human review.", + back: "Back to portfolio", + overview: "Overview", + run: "Run trace", + memory: "Memory", + adapter: "Adapter", + benchmark: "Benchmark run", + score: "Diagnostic score", + turns: "Turns", + facts: "Active facts", + corrections: "Superseded facts", + status: "Status", + objective: "Episode objective", + compare: "Comparison", + criteria: "Criterion review", + action: "Player action", + reaction: "World response", + changes: "Visible changes", + next: "Next actions", + active: "Current facts", + superseded: "Superseded, still auditable", + threads: "Open threads", + entities: "Person state", + recent: "Recent events", + import: "Import JSON run bundle", + exportBundle: "Download run bundle", + exportReport: "Download evaluation report", + reset: "Restore built-in sample", + invalid: "The file is not an rpg_evaluation_bundle.v1 bundle, or has no turns.", + sessionLabel: "Evaluate a live session", + sessionPlaceholder: "Enter a Tiny Stories session id", + loadSession: "Load session", + loadingSession: "Compiling run evidence...", + sessionError: "This run could not be loaded. Sign in as its owner and complete at least one turn.", + adapterTitle: "Connect another RPG", + adapterCopy: "An adapter only needs to emit the scenario, per-turn action/response, options, typed changes, and memory snapshots. The lab does not require the Tiny Stories backend.", + noValue: "None", + }, +} as const + +function downloadJson(filename: string, value: unknown) { + const blob = new Blob([`${JSON.stringify(value, null, 2)}\n`], { type: "application/json" }) + const url = URL.createObjectURL(blob) + const anchor = document.createElement("a") + anchor.href = url + anchor.download = filename + document.body.appendChild(anchor) + anchor.click() + anchor.remove() + URL.revokeObjectURL(url) +} + +function scoreTone(status: EvaluationStatus) { + return `rpg-lab-status rpg-lab-status--${status}` +} + +export function RpgEvaluationPage({ + onBack, + sessionId, +}: { + onBack: () => void + sessionId?: string +}) { + const api = useApi() + const { lang, setLang } = useLanguage() + const copy = UI[lang] + const [bundle, setBundle] = useState(RPG_EVALUATION_SAMPLES[0]) + const [view, setView] = useState("overview") + const [selectedTurn, setSelectedTurn] = useState(RPG_EVALUATION_SAMPLES[0].turns.length - 1) + const [importError, setImportError] = useState("") + const [sessionInput, setSessionInput] = useState(sessionId ?? "") + const [sessionBusy, setSessionBusy] = useState(false) + const [sessionError, setSessionError] = useState("") + const [authoritativeReport, setAuthoritativeReport] = useState(null) + const fileInputRef = useRef(null) + const loadedSessionRef = useRef("") + const sessionRequestRef = useRef("") + const liveSessionEnabled = import.meta.env.VITE_PUBLIC_LAB_DEFAULT !== "true" + const localReport = useMemo(() => evaluateRpgBundle(bundle), [bundle]) + const report = authoritativeReport?.run_id === bundle.run_id ? authoritativeReport : localReport + const comparison = useMemo( + () => RPG_EVALUATION_SAMPLES.map((sample) => ({ sample, report: evaluateRpgBundle(sample) })), + [], + ) + const latestMemory = bundle.turns[Math.min(selectedTurn, bundle.turns.length - 1)]?.memory + + const chooseBundle = (next: RpgEvaluationBundle) => { + if (next.locale !== "mixed") setLang(next.locale) + setBundle(next) + setSelectedTurn(Math.max(0, next.turns.length - 1)) + setImportError("") + setSessionError("") + setAuthoritativeReport(null) + } + + const loadLiveSession = async (rawSessionId: string) => { + const nextSessionId = rawSessionId.trim() + if (!nextSessionId || !liveSessionEnabled || sessionRequestRef.current === nextSessionId) return + sessionRequestRef.current = nextSessionId + setSessionBusy(true) + setSessionError("") + try { + const value = await api.getNarrativeEvaluationBundle(nextSessionId) + if (!isRpgEvaluationBundle(value)) throw new Error("invalid evaluation bundle") + const serverReport = await api.evaluatePortableRpgRun(value) + if (!isRpgEvaluationReport(serverReport)) throw new Error("invalid evaluation report") + chooseBundle(value) + setAuthoritativeReport(serverReport) + setView("overview") + loadedSessionRef.current = nextSessionId + } catch { + setSessionError(copy.sessionError) + } finally { + sessionRequestRef.current = "" + setSessionBusy(false) + } + } + + useEffect(() => { + if (!sessionId || !liveSessionEnabled || loadedSessionRef.current === sessionId) return + setSessionInput(sessionId) + void loadLiveSession(sessionId) + }, [sessionId]) + + const handleSessionLoad = (event: FormEvent) => { + event.preventDefault() + void loadLiveSession(sessionInput) + } + + const handleImport = async (event: ChangeEvent) => { + const file = event.target.files?.[0] + if (!file) return + try { + const value: unknown = JSON.parse(await file.text()) + if (!isRpgEvaluationBundle(value)) throw new Error("invalid bundle") + chooseBundle(value) + } catch { + setImportError(copy.invalid) + } finally { + event.target.value = "" + } + } + + return ( +
+
+ +
+ Tiny Stories + Runtime Lab / v1 +
+
+ + +
+
+ +
+
+ {copy.kicker} +

{copy.title}

+

{copy.lede}

+
{copy.boundary}
+
+ +
+
+ {RPG_EVALUATION_SAMPLES.map((sample) => ( + + ))} +
+
+ void handleImport(event)} /> + +
+ {liveSessionEnabled ? ( +
+ +
+ setSessionInput(event.target.value)} + placeholder={copy.sessionPlaceholder} + autoComplete="off" + /> + +
+
+ ) : null} + {importError ?

{importError}

: null} + {sessionError ?

{sessionError}

: null} +
+ +
+
+ {copy.score} + {report.score} + {report.status} +
+
+
{copy.turns}
{bundle.turns.length}/{bundle.scenario.turn_budget}
+
{copy.facts}
{latestMemory?.active_facts.length ?? 0}
+
{copy.corrections}
{latestMemory?.superseded_facts.length ?? 0}
+
{copy.status}
{bundle.system_label}
+
+
+ + + + {view === "overview" ? ( + + ) : null} + {view === "run" ? : null} + {view === "memory" && latestMemory ? ( + + ) : null} + {view === "adapter" ? ( + downloadJson(`${bundle.run_id}.json`, bundle)} + onDownloadReport={() => downloadJson(`${bundle.run_id}-report.json`, report)} + onReset={() => chooseBundle(RPG_EVALUATION_SAMPLES[0])} + /> + ) : null} +
+
+ ) +} + +function Overview({ + report, + bundle, + comparison, + lang, + copy, +}: { + report: RpgEvaluationReport + bundle: RpgEvaluationBundle + comparison: Array<{ sample: RpgEvaluationBundle; report: RpgEvaluationReport }> + lang: "zh" | "en" + copy: typeof UI.zh | typeof UI.en +}) { + const labelIndex = lang === "zh" ? 0 : 1 + return ( +
+
+
+ {copy.objective} + {bundle.scenario.objective} +

{bundle.scenario.genre} · {bundle.scenario.boundaries.join(" · ")}

+
+ {bundle.scenario.scenario_id === "awards-livestream" ? ( +
+ Backstage awards livestream reference scene +
Reference scene / Awards Livestream
+
+ ) : null} +
+
+
+
01

{copy.criteria}

+ {report.criteria.map((item) => ( +
+
+ {item.status} + {CRITERION_LABELS[item.criterion][labelIndex]} +
+ {item.score} +

{item.summary}

+
{item.evidence.length} evidence{item.evidence.map((entry) => {entry})}
+
+ ))} +
+ +
+
+ ) +} + +function RunTrace({ bundle, copy }: { bundle: RpgEvaluationBundle; copy: typeof UI.zh | typeof UI.en }) { + return ( +
+ {bundle.turns.map((turn) => ( +
+
{String(turn.turn_index).padStart(2, "0")}{Math.round(turn.objective_progress * 100)}%
+
{copy.action}

{turn.player_action}

+
{copy.reaction}

{turn.world_response}

+
+ {[...turn.state_deltas.map((delta) => delta.label), ...turn.clue_unlocks, ...turn.opportunity_unlocks].map((item) => {item})} +
+
    {turn.options.map((option) =>
  1. {option}
    2. )}
+
+ ))} +
+ ) +} + +function MemoryView({ + bundle, + selectedTurn, + setSelectedTurn, + copy, +}: { + bundle: RpgEvaluationBundle + selectedTurn: number + setSelectedTurn: (turn: number) => void + copy: typeof UI.zh | typeof UI.en +}) { + const memory = bundle.turns[selectedTurn]?.memory + if (!memory) return null + return ( +
+
+ {bundle.turns.map((turn, index) => ( + + ))} +
+
+
A

{copy.active}

{memory.active_facts.map((fact) =>

{fact.key}{fact.value}

)}
+
B

{copy.superseded}

{memory.superseded_facts.length ? memory.superseded_facts.map((fact) =>

{fact.key}{fact.value}

) : {copy.noValue}}
+
C

{copy.entities}

{memory.entities.map((entity) =>

{entity.name}{Object.entries(entity.state).map(([key, value]) => `${key}: ${value}`).join(" · ")}

)}
+
D

{copy.threads}

{memory.open_threads.length ? memory.open_threads.map((thread) =>

{thread}

) : {copy.noValue}}

{copy.recent}

{memory.recent_events.map((event) => {event})}
+
+
+ ) +} + +function AdapterView({ + copy, + onDownloadBundle, + onDownloadReport, + onReset, +}: { + copy: typeof UI.zh | typeof UI.en + onDownloadBundle: () => void + onDownloadReport: () => void + onReset: () => void +}) { + return ( +
+
API

{copy.adapterTitle}

+

{copy.adapterCopy}

+
+ Scenario→Turns→Memory→Report +
+ 
{`{
+  "schema_version": "rpg_evaluation_bundle.v1",
+  "scenario": { "objective": "...", "entity_ids": [] },
+  "turns": [{
+    "player_action": "...",
+    "world_response": "...",
+    "options": ["..."],
+    "state_deltas": [],
+    "memory": { "schema_version": "rpg_memory.v1" }
+  }]
+}`}
+
+ + + +
+
+ ) +} diff --git a/frontend2/src/pages/evaluation/rpg-evaluation-samples.ts b/frontend2/src/pages/evaluation/rpg-evaluation-samples.ts new file mode 100644 index 00000000..08de70cc --- /dev/null +++ b/frontend2/src/pages/evaluation/rpg-evaluation-samples.ts @@ -0,0 +1,130 @@ +import type { RpgEvaluationBundle, RpgMemoryFact, RpgMemorySnapshot } from "./rpg-evaluation-contract" + +function fact(id: string, key: string, value: string, turn: number, namespace = "story"): RpgMemoryFact { + return { + fact_id: id, + namespace, + key, + value, + status: "active", + source_turn: turn, + source: "runtime", + confidence: 1, + } +} + +function memory(turn: number, active: RpgMemoryFact[], recent: string[], progress: string): RpgMemorySnapshot { + return { + schema_version: "rpg_memory.v1", + run_id: "awards-candidate-01", + turn_index: turn, + objective: "Find the missing singer before the livestream countdown ends.", + active_facts: active, + superseded_facts: turn >= 2 ? [{ + ...fact("fact-old-role", "player_role", "Backup dancer", 0), + status: "superseded", + superseded_by: "fact-role", + }] : [], + open_threads: turn < 3 ? ["Who used the green-room badge after Mina vanished?"] : [], + entities: [ + { entity_id: "lena", name: "Lena", state: { trust: turn >= 2 ? "warming" : "guarded" }, last_updated_turn: turn }, + { entity_id: "producer_han", name: "Producer Han", state: { pressure: turn >= 3 ? "cornered" : "wary" }, last_updated_turn: turn }, + { entity_id: "dana", name: "Dana Vale", state: { role: "off-the-record advisor" }, last_updated_turn: turn }, + ], + recent_events: recent.slice(-8), + episodic_summary: `Objective: find Mina before air. ${progress}`, + diagnostics: { + event_count: recent.length + active.length, + active_fact_count: active.length, + superseded_fact_count: turn >= 2 ? 1 : 0, + non_story_event_count: 1, + dropped_recent_event_count: 0, + last_compacted_turn: turn, + }, + } +} + +const baseFacts = [ + fact("fact-role", "player_role", "Publicist", 0), + fact("fact-pressure", "pressure", "Three minutes until air", 0), + fact("fact-boundary", "content_boundary", "No violence or blackmail", 0, "boundary"), +] + +export const CANDIDATE_RPG_BUNDLE: RpgEvaluationBundle = { + schema_version: "rpg_evaluation_bundle.v1", + run_id: "awards-candidate-01", + system_label: "Stateful runtime candidate", + locale: "en", + scenario: { + scenario_id: "awards-livestream", + title: "Awards Livestream", + genre: "social-pressure mystery", + objective: "Find the missing singer before the livestream countdown ends.", + turn_budget: 12, + entity_ids: ["lena", "producer_han", "dana"], + boundaries: ["No violence", "No blackmail", "Keep pressure social and investigative"], + }, + turns: [ + { + turn_index: 1, + player_action: "Freeze the sponsor feed and ask Lena to verify the badge timestamp.", + world_response: "Lena checks the feed log. The badge was used after Mina vanished, and Producer Han can no longer dismiss the discrepancy.", + options: ["Question Producer Han", "Send Lena to the corridor", "Protect the feed log"], + state_deltas: [{ target: "lena", kind: "increase", label: "Lena trust +1", evidence: "You gave her verifiable proof." }], + clue_unlocks: ["Green-room badge timestamp"], + opportunity_unlocks: [], + referenced_entity_ids: ["lena", "producer_han"], + objective_progress: 0.3, + memory: memory(1, baseFacts, ["Player: froze the sponsor feed", "Clue: badge timestamp"], "The badge is now evidence."), + }, + { + turn_index: 2, + player_action: "Use the timestamp to make Producer Han name the last badge holder.", + world_response: "Han admits the badge crossed the service corridor. Lena moves to cover the exit while the room waits for your next call.", + options: ["Search the service corridor", "Ask Dana to map the sponsor risk", "Call the stage manager as witness"], + state_deltas: [ + { target: "producer_han", kind: "shift", label: "Han is cornered", evidence: "The timestamp removed his plausible denial." }, + { target: "pressure", kind: "increase", label: "Public pressure rising", evidence: "The feed cannot stay frozen much longer." }, + ], + clue_unlocks: [], + opportunity_unlocks: ["Service corridor search"], + referenced_entity_ids: ["lena", "producer_han", "dana"], + objective_progress: 0.62, + memory: memory(2, [...baseFacts, fact("fact-corridor", "search_area", "Service corridor", 2)], ["Player: confronted Han", "World: service corridor identified"], "The search is narrowed."), + }, + { + turn_index: 3, + player_action: "Take Lena and the stage manager into the service corridor before the countdown resumes.", + world_response: "They find Mina's in-ear monitor beside an unlocked rehearsal room. A voice inside asks whether the sponsor has left.", + options: ["Answer Mina honestly", "Let Lena speak first", "Secure the rehearsal room door"], + state_deltas: [{ target: "objective", kind: "increase", label: "Singer located", evidence: "Mina answered from the rehearsal room." }], + clue_unlocks: ["Mina's in-ear monitor"], + opportunity_unlocks: ["Private negotiation with Mina"], + referenced_entity_ids: ["lena", "producer_han"], + objective_progress: 0.9, + memory: memory(3, [...baseFacts, fact("fact-location", "singer_location", "Rehearsal room", 3)], ["Player: searched corridor", "World: Mina answered", "Clue: in-ear monitor"], "Mina is found; sponsor pressure remains."), + }, + ], +} + +const weakMemory = memory(1, [fact("weak-role", "player_role", "Someone in the room", 0)], ["World: tension changed"], "No concrete progress recorded.") + +export const BASELINE_RPG_BUNDLE: RpgEvaluationBundle = { + ...CANDIDATE_RPG_BUNDLE, + run_id: "awards-baseline-01", + system_label: "Prose-only baseline", + turns: [1, 2, 3].map((turn) => ({ + turn_index: turn, + player_action: "Wait and see what happens.", + world_response: "The room grows more tense, but no one makes a clear move.", + options: ["Wait", "Wait"], + state_deltas: [], + clue_unlocks: [], + opportunity_unlocks: [], + referenced_entity_ids: [], + objective_progress: 0, + memory: { ...weakMemory, run_id: "awards-baseline-01", turn_index: turn }, + })), +} + +export const RPG_EVALUATION_SAMPLES = [CANDIDATE_RPG_BUNDLE, BASELINE_RPG_BUNDLE] diff --git a/frontend2/src/pages/play/components/play-editorial-primitives.tsx b/frontend2/src/pages/play/components/play-editorial-primitives.tsx index 73f4d24c..9f932483 100644 --- a/frontend2/src/pages/play/components/play-editorial-primitives.tsx +++ b/frontend2/src/pages/play/components/play-editorial-primitives.tsx @@ -88,27 +88,28 @@ export function MoodPlate({ compact: boolean isComplete?: boolean }) { + const t = useT() const imageUrl = sceneUrl || coverUrl const safeTurnBudget = Math.max(turnBudget, 1) const turnsPlayed = Math.max(0, Math.min(turnsCompleted, safeTurnBudget)) const progress = turnsPlayed <= 0 - ? `${safeTurnBudget} turns to play` - : `${turnsPlayed}/${safeTurnBudget} turns played` + ? t("play.mood_progress_empty", { count: safeTurnBudget }) + : t("play.mood_progress_played", { current: turnsPlayed, total: safeTurnBudget }) const stage = isComplete - ? "Complete" + ? t("play.mood_stage_complete") : turnsRemaining <= 2 - ? "Final stretch" + ? t("play.mood_stage_final") : turnsCompleted <= 0 - ? "Opening" - : "In motion" + ? t("play.mood_stage_opening") + : t("play.mood_stage_motion") const context = isComplete - ? "This run is finished. Review the ending, then replay or share it." + ? t("play.mood_context_complete") : turnsCompleted <= 0 - ? "First shot is live. Choose the pressure you step into." + ? t("play.mood_context_opening") : turnsRemaining <= 2 - ? "The room is close to its final break." - : "The room is waiting." + ? t("play.mood_context_final") + : t("play.mood_context_waiting") return (
- +
- + {pressure} - +
{actors.map((actor) => { const focused = focusedActorId === actor.id diff --git a/frontend2/src/pages/play/components/runtime-inspector.tsx b/frontend2/src/pages/play/components/runtime-inspector.tsx index 08da9da4..b17738a4 100644 --- a/frontend2/src/pages/play/components/runtime-inspector.tsx +++ b/frontend2/src/pages/play/components/runtime-inspector.tsx @@ -316,7 +316,18 @@ export function RuntimeInspector({
- Session {story.session.turn_count}/{story.session.turn_budget} · {inventoryState} · ending {endingLabel} · {turnsRemaining} left + + Session {story.session.turn_count}/{story.session.turn_budget} · {inventoryState} · ending {endingLabel} · {turnsRemaining} left + + {story.session.turn_count > 0 ? ( + + Open portable run evaluation + + ) : null}
) diff --git a/frontend2/src/pages/play/play-page.tsx b/frontend2/src/pages/play/play-page.tsx index 8e35426e..847e7455 100644 --- a/frontend2/src/pages/play/play-page.tsx +++ b/frontend2/src/pages/play/play-page.tsx @@ -286,6 +286,48 @@ function resourceActionCountsForOptions( return counts } +function gameplayTrackLabel(t: ReturnType, id: string, fallback: string): string { + if (id === "time") return t("play.gameplay_track_time") + if (id === "pressure") return t("play.gameplay_track_pressure") + if (id === "people") return t("play.gameplay_track_people") + if (id === "evidence") return t("play.gameplay_track_evidence") + return fallback +} + +function gameplayTrackValue(t: ReturnType, value: string): string { + const normalized = value.trim().toLowerCase() + if (normalized === "none" || normalized === "none yet") return t("play.gameplay_track_none") + if (normalized === "rising") return t("play.gameplay_track_rising") + if (normalized === "easing") return t("play.gameplay_track_easing") + if (normalized === "held") return t("play.gameplay_track_held") + if (normalized === "watching") return t("play.gameplay_track_watching") + const heldMatch = normalized.match(/^(\d+)\s+(?:items?\s+)?held$/) + if (heldMatch) return t("play.gameplay_track_items_held", { count: heldMatch[1] }) + return value +} + +function gameplayForecastLabel(t: ReturnType, label: string): string { + const normalized = label.replace(/\s+/g, " ").trim().toLowerCase() + if (normalized === "read the room") return t("play.gameplay_forecast_read_room") + if (normalized === "time -1") return t("play.gameplay_forecast_time_cost") + if (normalized === "pressure +1") return t("play.gameplay_forecast_pressure_cost") + if (normalized === "trust +1") return t("play.gameplay_forecast_trust_gain") + if (normalized === "may reveal evidence") return t("play.gameplay_forecast_evidence") + if (normalized === "use leverage") return t("play.gameplay_forecast_leverage") + if (normalized === "risk +1") return t("play.gameplay_forecast_risk_cost") + return label +} + +function localizedActionForecasts( + t: ReturnType, + rows: GameplayEnvelope["actionForecasts"], +): GameplayEnvelope["actionForecasts"] { + return rows.map((row) => row.map((chip) => ({ + ...chip, + label: gameplayForecastLabel(t, chip.label), + }))) +} + function GameplayStatePanel({ envelope, focusedResourceId, @@ -319,6 +361,8 @@ function GameplayStatePanel({ aria-label={t("play.gameplay_tracks_label")} > {envelope.tracks.map((track) => { + const displayLabel = gameplayTrackLabel(t, track.id, track.label) + const displayValue = gameplayTrackValue(t, track.value) const focusableTrackId = isGameplayResourceFocusId(track.id) ? track.id : null const isFocused = focusedResourceId === focusableTrackId && focusableTrackId !== null const focusTitle = focusableTrackId ? gameplayResourceFocusTitle(t, focusableTrackId) : "" @@ -345,8 +389,8 @@ function GameplayStatePanel({ } const content = ( <> - {track.label} - {track.value} + {displayLabel} + {displayValue} {focusableTrackId && resourceActionLabel ? ( {resourceActionLabel} @@ -365,7 +409,7 @@ function GameplayStatePanel({ data-gameplay-resource-focus={isFocused ? "true" : undefined} data-gameplay-resource-action-count={resourceMatchCount} aria-pressed={isFocused} - aria-label={`${track.label}: ${track.value}. ${resourceActionLabel ? `${resourceActionLabel}. ` : ""}${focusTitle}`} + aria-label={`${displayLabel}: ${displayValue}. ${resourceActionLabel ? `${resourceActionLabel}. ` : ""}${focusTitle}`} title={focusTitle} onClick={() => onFocusResource?.(focusableTrackId)} > @@ -840,6 +884,7 @@ export function PlayPage({ const api = apiClient ?? defaultApi const auth = useAuth() const t = useT() + const { setLang } = useLanguage() const [story, setStory] = useState(null) const [ending, setEnding] = useState(null) const [latestAgentPlan, setLatestAgentPlan] = useState(null) @@ -896,6 +941,7 @@ export function PlayPage({ .getNarrativeStory(sessionId, canRequestAgentTrace ? { agentTrace: true } : undefined) .then(async (response) => { if (cancelled) return + if (response.template.language) setLang(response.template.language) setStory(response) const agentEvents = canRequestAgentTrace ? response.agent_events ?? [] : [] setLatestAgentEvents(agentEvents) @@ -1277,6 +1323,7 @@ export function PlayPage({ castNameById, backendEnvelope: story.gameplay_envelope ?? null, }) + const displayActionForecasts = localizedActionForecasts(t, gameplayEnvelope.actionForecasts) const showGameplayImpactSummary = actionAreaVisible && turnsCompleted > 0 && gameplayEnvelope.impact.length > 0 const nextChoiceTargets = lastNarrator @@ -1656,7 +1703,7 @@ export function PlayPage({ // state, so remount doesn't drop user typing. key={`actions-${lastNarrator.ord}`} options={lastNarrator.options} - actionForecasts={gameplayEnvelope.actionForecasts} + actionForecasts={displayActionForecasts} leverageCards={leverageCards} roleHasNoLeverage={Boolean(story.session.player_role) && leverageCards.length === 0} latestNpcPulses={lastNarrator.npc_pulse ?? []} diff --git a/frontend2/src/pages/play/play-styles.ts b/frontend2/src/pages/play/play-styles.ts index 8583b5fb..0d7e2005 100644 --- a/frontend2/src/pages/play/play-styles.ts +++ b/frontend2/src/pages/play/play-styles.ts @@ -1293,6 +1293,17 @@ export const ppStyles: Record = { color: "rgba(244,239,230,0.48)", fontSize: 11.5, lineHeight: 1.4, + display: "flex", + alignItems: "center", + justifyContent: "space-between", + gap: 12, + flexWrap: "wrap", + }, + evaluationFooterLink: { + color: "rgba(245,200,120,0.92)", + textDecoration: "none", + borderBottom: "1px solid rgba(245,200,120,0.42)", + paddingBottom: 2, }, roleInvAcquired: { diff --git a/frontend2/src/pages/portfolio/portfolio-page.tsx b/frontend2/src/pages/portfolio/portfolio-page.tsx index 98ff5b0f..8486b3f9 100644 --- a/frontend2/src/pages/portfolio/portfolio-page.tsx +++ b/frontend2/src/pages/portfolio/portfolio-page.tsx @@ -1,7 +1,6 @@ -import { useEffect, useState } from "react" +import { useState } from "react" import { motion } from "motion/react" import { Header } from "../../shared/ui/header" -import { useLanguage } from "../../shared/lib/i18n" import { CASE_STUDY_POINTS, EVIDENCE_PACKET_URL, @@ -122,21 +121,17 @@ export function PortfolioPage({ onOpenCreate, onOpenReviewer, onOpenPublicDemo, + onOpenEvaluation, }: { onBackHome: () => void onOpenCreate: () => void onOpenReviewer: () => void onOpenPublicDemo: () => void + onOpenEvaluation: () => void }) { const [activeStep, setActiveStep] = useState(0) const step = PIPELINE_STEPS[activeStep] const localQaAvailable = canOpenLocalQaRoute() - const { setLang } = useLanguage() - - useEffect(() => { - setLang("en") - }, [setLang]) - return (
@@ -178,6 +173,9 @@ export function PortfolioPage({ + diff --git a/frontend2/src/pages/portfolio/reviewer-page.tsx b/frontend2/src/pages/portfolio/reviewer-page.tsx index 5c375c95..9f160989 100644 --- a/frontend2/src/pages/portfolio/reviewer-page.tsx +++ b/frontend2/src/pages/portfolio/reviewer-page.tsx @@ -117,10 +117,6 @@ export function ReviewerPage({ const launchErrorRef = useRef(null) const localQaAvailable = canOpenLocalQaRoute() - useEffect(() => { - setLang("en") - }, [setLang]) - useEffect(() => { if (!busy || launchPhase === "ready") return const frame = window.requestAnimationFrame(() => { diff --git a/frontend2/src/pages/replay/replay-page.tsx b/frontend2/src/pages/replay/replay-page.tsx index e9fd8b8c..6343bc19 100644 --- a/frontend2/src/pages/replay/replay-page.tsx +++ b/frontend2/src/pages/replay/replay-page.tsx @@ -39,7 +39,7 @@ export function ReplayPage({ const defaultApi = useApi() const api = apiClient ?? defaultApi const t = useT() - const { lang } = useLanguage() + const { lang, setLang } = useLanguage() const [replay, setReplay] = useState(null) const [error, setError] = useState(null) // Entry mode for shared replay links: @@ -75,16 +75,17 @@ export function ReplayPage({ .getNarrativePublicReplay(sessionId) .then((r) => { if (cancelled) return + if (r.language) setLang(r.language) setReplay(r) }) .catch(() => { if (cancelled) return - setError(t("replay.error_load_failed")) + setError("load_failed") }) return () => { cancelled = true } - }, [api, sessionId, t]) + }, [api, sessionId, setLang]) if (!replay) { return ( @@ -114,7 +115,9 @@ export function ReplayPage({ } // Build a synthetic template-like object so we can reuse the cover helper. - const displayTitle = getReplayDisplayTitle(replay, lang) + const displayTitle = replay.template_forkable + ? getReplayDisplayTitle(replay, lang) + : t("replay.private_title") const displaySummary = getReplayDisplaySummary(replay, lang) const templateLike = { template_id: sessionId, // stable hash on session_id for the cover pick diff --git a/frontend2/src/shared/lib/i18n.ts b/frontend2/src/shared/lib/i18n.ts index 18a57b22..9ce550ca 100644 --- a/frontend2/src/shared/lib/i18n.ts +++ b/frontend2/src/shared/lib/i18n.ts @@ -18,7 +18,7 @@ // union, list it in LANGUAGE_OPTIONS, and add a prompt-language branch // in `rpg_backend/narrative/engine.py`. -import { createContext, createElement, useContext, useEffect, useMemo, useState, type ReactNode } from "react" +import { createContext, createElement, useCallback, useContext, useEffect, useMemo, useState, type ReactNode } from "react" export type Lang = "zh" | "en" @@ -598,6 +598,38 @@ export const STRINGS_ZH = { "play.gameplay_loop_react_detail": "人物和局势变化", "play.gameplay_loop_update_label": "用变化", "play.gameplay_loop_update_detail": "线索、压力、下一步", + "play.mood_progress_empty": "还有 {count} 段可行动", + "play.mood_progress_played": "已行动 {current}/{total} 段", + "play.mood_stage_complete": "已完结", + "play.mood_stage_final": "最后阶段", + "play.mood_stage_opening": "第一幕", + "play.mood_stage_motion": "局面推进中", + "play.mood_context_complete": "本局已经结束。可以查看结局、回放或分享。", + "play.mood_context_opening": "第一幕已经开始。选择你要踏入的压力。", + "play.mood_context_final": "局面正在逼近最后的断点。", + "play.mood_context_waiting": "房间在等你的下一步。", + "play.gameplay_track_time": "时间", + "play.gameplay_track_pressure": "压力", + "play.gameplay_track_people": "人物", + "play.gameplay_track_evidence": "证据", + "play.gameplay_track_rising": "上升", + "play.gameplay_track_easing": "缓和", + "play.gameplay_track_held": "稳住", + "play.gameplay_track_watching": "正在观察", + "play.gameplay_track_none": "暂无", + "play.gameplay_forecast_read_room": "观察局面", + "play.gameplay_forecast_time_cost": "时间 -1", + "play.gameplay_forecast_pressure_cost": "压力 +1", + "play.gameplay_forecast_trust_gain": "信任 +1", + "play.gameplay_forecast_evidence": "可能发现证据", + "play.gameplay_forecast_leverage": "使用筹码", + "play.gameplay_forecast_risk_cost": "风险 +1", + "play.gameplay_track_items_held": "持有 {count} 项", + "play.rail_aria": "场景支持", + "play.rail_you_are": "你是", + "play.rail_you_fallback": "你", + "play.rail_pressure_now": "当前压力", + "play.rail_people": "可介入的人物", "play.runtime_inspector_label": "运行检查器", "play.runtime_inspector_kicker": "评审运行检查器", "play.runtime_inspector_title": "系统视角", @@ -851,7 +883,7 @@ export const STRINGS_ZH = { "play.inner_motive_submit_cta": "带着动机提交", "play.inner_motive_submit_disabled_hint": "先写一句真实动机，才能带着动机提交。", "play.advisor_card_title": "场外朋友", - "play.advisor_card_name": "Dana Vale", + "play.advisor_card_name": "黛娜·维尔", "play.advisor_card_role": "危机旁批", "play.advisor_card_background": "懂你的压力点，只帮你看清下一步，不替你行动。", "play.advisor_card_ask": "问一句", @@ -1117,6 +1149,7 @@ export const STRINGS_ZH = { "replay.loading_label": "正在还原这一局…", "replay.crumb_back_home": "← 故事入口", "replay.badge": "回放", + "replay.private_title": "私密故事回放", "replay.completed_meta": "已完成的一局", "replay.role_meta": "扮演 {role}", "replay.turns_meta": "这局 {current} / {total} 段", @@ -1684,6 +1717,38 @@ export const STRINGS_EN: Record = { "play.gameplay_loop_react_detail": "People and stakes shift", "play.gameplay_loop_update_label": "Use what changed", "play.gameplay_loop_update_detail": "Clues, pressure, next move", + "play.mood_progress_empty": "{count} turns to play", + "play.mood_progress_played": "{current}/{total} turns played", + "play.mood_stage_complete": "Complete", + "play.mood_stage_final": "Final stretch", + "play.mood_stage_opening": "Opening", + "play.mood_stage_motion": "In motion", + "play.mood_context_complete": "This run is finished. Review the ending, then replay or share it.", + "play.mood_context_opening": "First shot is live. Choose the pressure you step into.", + "play.mood_context_final": "The room is close to its final break.", + "play.mood_context_waiting": "The room is waiting.", + "play.gameplay_track_time": "Time", + "play.gameplay_track_pressure": "Pressure", + "play.gameplay_track_people": "People", + "play.gameplay_track_evidence": "Evidence", + "play.gameplay_track_rising": "rising", + "play.gameplay_track_easing": "easing", + "play.gameplay_track_held": "held", + "play.gameplay_track_watching": "watching", + "play.gameplay_track_none": "none", + "play.gameplay_forecast_read_room": "Read the room", + "play.gameplay_forecast_time_cost": "Time -1", + "play.gameplay_forecast_pressure_cost": "Pressure +1", + "play.gameplay_forecast_trust_gain": "Trust +1", + "play.gameplay_forecast_evidence": "May reveal evidence", + "play.gameplay_forecast_leverage": "Use leverage", + "play.gameplay_forecast_risk_cost": "Risk +1", + "play.gameplay_track_items_held": "{count} held", + "play.rail_aria": "Scene support", + "play.rail_you_are": "You are", + "play.rail_you_fallback": "You", + "play.rail_pressure_now": "Pressure now", + "play.rail_people": "People you can involve", "play.runtime_inspector_label": "Runtime inspector", "play.runtime_inspector_kicker": "Reviewer runtime inspector", "play.runtime_inspector_title": "System lens", @@ -2192,6 +2257,7 @@ export const STRINGS_EN: Record = { "replay.loading_label": "Restoring this run…", "replay.crumb_back_home": "← Story Desk", "replay.badge": "Replay", + "replay.private_title": "Shared private story", "replay.completed_meta": "Completed run", "replay.role_meta": "Played as {role}", "replay.turns_meta": "{current} / {total} turns in this run", @@ -2298,15 +2364,16 @@ export function LanguageProvider({ children }: { children: ReactNode }) { document.documentElement.lang = lang }, [lang]) - const value = useMemo(() => { - const setLang = (next: Lang) => { - setLangState(next) - try { - window.localStorage.setItem(STORAGE_KEY, next) - } catch { - // localStorage unavailable (private browsing) — fail silently. - } + const setLang = useCallback((next: Lang) => { + setLangState(next) + try { + window.localStorage.setItem(STORAGE_KEY, next) + } catch { + // localStorage unavailable (private browsing) — fail silently. } + }, []) + + const value = useMemo(() => { const t: TFn = (key, paramsOrFallback, fallback) => { const params = typeof paramsOrFallback === "object" ? paramsOrFallback : undefined const stringFallback = typeof paramsOrFallback === "string" ? paramsOrFallback : fallback @@ -2318,7 +2385,7 @@ export function LanguageProvider({ children }: { children: ReactNode }) { return stringFallback ?? String(key) } return { lang, setLang, t } - }, [lang]) + }, [lang, setLang]) return createElement(LanguageContext.Provider, { value }, children) } diff --git a/frontend2/vercel.json b/frontend2/vercel.json new file mode 100644 index 00000000..57d13181 --- /dev/null +++ b/frontend2/vercel.json @@ -0,0 +1,16 @@ +{ + "$schema": "https://openapi.vercel.sh/vercel.json", + "framework": "vite", + "buildCommand": "VITE_PUBLIC_LAB_DEFAULT=true npm run build", + "outputDirectory": "dist", + "headers": [ + { + "source": "/(.*)", + "headers": [ + { "key": "X-Content-Type-Options", "value": "nosniff" }, + { "key": "Referrer-Policy", "value": "strict-origin-when-cross-origin" }, + { "key": "Permissions-Policy", "value": "camera=(), microphone=(), geolocation=()" } + ] + } + ] +} diff --git a/frontend2/vite.config.ts b/frontend2/vite.config.ts index 48f3f6b9..4c9e8a3a 100644 --- a/frontend2/vite.config.ts +++ b/frontend2/vite.config.ts @@ -14,7 +14,7 @@ export default defineConfig(({ mode }) => { port: 8001, strictPort: true, proxy: { - "^/(health|me|auth|author|stories|play|benchmark|narrative)": { + "^/(health|me|auth|author|stories|play|benchmark|narrative|research)": { target: backendTarget, changeOrigin: true, }, diff --git a/rpg_backend/main.py b/rpg_backend/main.py index 74487d4b..c1be1ec2 100644 --- a/rpg_backend/main.py +++ b/rpg_backend/main.py @@ -77,6 +77,8 @@ ) from rpg_backend.narrative.service import NarrativeServiceError, get_narrative_service from rpg_backend.quotas import DailyQuotaLimiter, QuotaExceededError +from rpg_backend.research_runtime.contracts import RpgEvaluationBundleV1, RpgEvaluationReportV1 +from rpg_backend.research_runtime.evaluator import evaluate_rpg_bundle app = FastAPI(title="rpg-demo-rebuild") settings = get_settings() @@ -723,6 +725,32 @@ def get_narrative_story( ) +@app.get( + "/narrative/sessions/{session_id}/evaluation-bundle", + response_model=RpgEvaluationBundleV1, +) +def get_narrative_evaluation_bundle( + session_id: str, + user=Depends(get_required_request_user), +) -> RpgEvaluationBundleV1: + """Export an owned run without exposing prompts, keys, or raw traces.""" + + return narrative_service.get_rpg_evaluation_bundle( + session_id, + player_user_id=user.user_id, + ) + + +@app.post("/research/rpg-evaluations", response_model=RpgEvaluationReportV1) +def evaluate_portable_rpg_run( + payload: RpgEvaluationBundleV1, + _user=Depends(get_required_request_user), +) -> RpgEvaluationReportV1: + """Run the deterministic portable rubric for an authenticated reviewer.""" + + return evaluate_rpg_bundle(payload) + + @app.get("/narrative/sessions/{session_id}/llm-events", response_model=LLMCallEventListResponse) def get_narrative_llm_events( session_id: str, diff --git a/rpg_backend/narrative/contracts.py b/rpg_backend/narrative/contracts.py index 3fb040e7..cef615d8 100644 --- a/rpg_backend/narrative/contracts.py +++ b/rpg_backend/narrative/contracts.py @@ -1006,6 +1006,7 @@ class PublicReplayResponse(BaseModel): template_seed: str template_title_i18n: LocalizedText | None = None template_summary_i18n: LocalizedText | None = None + language: TemplateLanguage = "en" cast: list[CastMember] advisor_persona: str cover_image_url: str | None = Field(default=None, max_length=1000) @@ -1037,10 +1038,14 @@ class CreateTemplateRequest(BaseModel): # all sessions forking this template share the same language. language: TemplateLanguage = DEFAULT_TEMPLATE_LANGUAGE # Optional create-time plan returned by Story Brief Advisor. This is - # reviewed by the user before generation, then injected into the opening - # payload so planning and generation use the same facts. It is not - # persisted on the template in this MVP. + # reviewed by the user before generation, injected into the opening, and + # persisted as an internal research/audit seed. It is not exposed in the + # public template summary. story_brief: StoryBrief | None = None + # Sanitized Story Butler memory at the exact generation boundary. Keeping + # this separate from the raw transcript preserves corrections and intent + # routing without persisting unbounded chat or private model reasoning. + story_guide_context: StoryGuideCompressedContext | None = None @field_validator("seed") @classmethod diff --git a/rpg_backend/narrative/engine.py b/rpg_backend/narrative/engine.py index 05d6c31e..504a5960 100644 --- a/rpg_backend/narrative/engine.py +++ b/rpg_backend/narrative/engine.py @@ -23,6 +23,7 @@ PlayerLeverageOverNPC, PlayerRole, StoryBrief, + StoryGuideCompressedContext, StoryMessage, StoryOption, STORY_OPTION_LABEL_MAX_LENGTH, @@ -253,7 +254,7 @@ def _language_directive(language: TemplateLanguage | None) -> str: _TURN_SYSTEM_PROMPT = """\ 你是一名擅长写关系剧的剧作家。玩家正在玩一个互动故事，你负责续写下一段。 -每个回合你会收到：故事种子、cast 名单（每个 NPC 都带有 `hidden_objective`、`leverage_over_player`、以及 `leverages_over_other_npcs` —— **这是 NPC 之间的相互把柄网络**）、最近若干段故事历史、玩家这一回合的动作、**当前所处的故事阶段**（关键！）、`difficulty` 字段（`story` 或 `gauntlet`）、**玩家的角色卡 `player_role`**（玩家这局选了谁来扮演）、**`current_inventory`**（玩家当前手里的所有物件/情报，包括 starting_assets 和过去几回合获得的）、可选的 `npc_agenda_this_turn`（gauntlet 主动调度）、**可选的 `twist_directive`**（reversal 阶段强制翻转指令）、**可选的 `player_diary`**（玩家私下对自己说的话，NPC 不可见）和 `recent_consequences`（上一回合结构化回响）。 +每个回合你会收到：故事种子、cast 名单（每个 NPC 都带有 `hidden_objective`、`leverage_over_player`、以及 `leverages_over_other_npcs` —— **这是 NPC 之间的相互把柄网络**）、最近若干段故事历史、玩家这一回合的动作、**当前所处的故事阶段**（关键！）、`difficulty` 字段（`story` 或 `gauntlet`）、**玩家的角色卡 `player_role`**（玩家这局选了谁来扮演）、**`current_inventory`**（玩家当前手里的所有物件/情报，包括 starting_assets 和过去几回合获得的）、可选的 `story_contract`（Create 阶段确认并压缩过的当前事实、边界和语气）、可选的 `npc_agenda_this_turn`（gauntlet 主动调度）、**可选的 `twist_directive`**（reversal 阶段强制翻转指令）、**可选的 `player_diary`**（玩家私下对自己说的话，NPC 不可见）和 `recent_consequences`（上一回合结构化回响）。 你的任务是续写**一段**叙述（200-400 字），并给出**3 个新选项**，同时输出每个 NPC 的当下反应（`npc_pulse`）。 @@ -305,6 +306,7 @@ def _language_directive(language: TemplateLanguage | None) -> str: 写作要求： - 第二人称 - 必须**真正承接**玩家的动作，让 TA 看见自己的选择带来了什么 +- 如果有 `story_contract`，把它视为当前有效事实与边界；不要恢复被用户改掉的旧角色/设定，也不要违反其中的 boundaries - 节奏感：每段聚焦一个戏剧瞬间，**根据 stage_phase 调整事件密度** **故事阶段（stage_phase）会随回合推进，请严格依照阶段调度节奏:** @@ -1165,6 +1167,45 @@ def build_agent_plan( ) +def _build_turn_story_contract( + story_brief: StoryBrief | None, + story_guide_context: StoryGuideCompressedContext | None, +) -> dict[str, object] | None: + """Compact Create-time truth for turn generation. + + Raw chat, non-story input, superseded values, planner internals, and model + source labels are intentionally excluded. + """ + + payload: dict[str, object] = {} + if story_brief is not None: + payload.update( + { + "premise": story_brief.premise_summary, + "story_kernel": story_brief.story_kernel, + "player_role": story_brief.player_role, + "tone": story_brief.genre_tone, + "boundaries": story_brief.preserved_constraints[:8], + } + ) + if story_guide_context is not None: + if story_guide_context.scene_summary: + payload["scene_summary"] = story_guide_context.scene_summary + if story_guide_context.player_role: + payload["player_role"] = story_guide_context.player_role + if story_guide_context.cast_or_factions: + payload["cast_or_factions"] = story_guide_context.cast_or_factions[:8] + if story_guide_context.pressure: + payload["pressure"] = story_guide_context.pressure + if story_guide_context.constraints: + payload["boundaries"] = story_guide_context.constraints[:8] + if story_guide_context.tone: + payload["tone"] = story_guide_context.tone + if story_guide_context.confirmed_facts: + payload["confirmed_facts"] = story_guide_context.confirmed_facts[:12] + return payload or None + + def advance_turn( *, gateway: NarrativeLLMGateway, @@ -1182,6 +1223,8 @@ def advance_turn( current_inventory: list[str] | None = None, player_diary: str | None = None, played_leverage: PlayedLeverageCard | None = None, + story_brief: StoryBrief | None = None, + story_guide_context: StoryGuideCompressedContext | None = None, language: TemplateLanguage = "en", ) -> TurnResult: """Advance one turn.""" @@ -1221,6 +1264,9 @@ def advance_turn( user_payload["player_diary"] = player_diary if played_leverage is not None: user_payload["played_leverage_card"] = played_leverage.model_dump() + story_contract = _build_turn_story_contract(story_brief, story_guide_context) + if story_contract is not None: + user_payload["story_contract"] = story_contract # Active scheduling: tell the LLM which NPC should actively push their # agenda this turn. Empty in story mode and during the hook phase. @@ -1354,6 +1400,8 @@ def synthesize_early_ending( failure_trigger: str, failure_reason: str, player_role: PlayerRole | None = None, + story_brief: StoryBrief | None = None, + story_guide_context: StoryGuideCompressedContext | None = None, language: TemplateLanguage = "en", ) -> EndingResult: """Generate a 'collapsed' ending when judge_failure flagged a trigger. @@ -1370,6 +1418,9 @@ def synthesize_early_ending( } if player_role is not None: user_payload["player_role"] = player_role.model_dump() + story_contract = _build_turn_story_contract(story_brief, story_guide_context) + if story_contract is not None: + user_payload["story_contract"] = story_contract response = gateway.invoke_json( system_prompt=_EARLY_ENDING_SYSTEM_PROMPT, user_payload=user_payload, @@ -1765,6 +1816,8 @@ def synthesize_ending( history: list[StoryMessage], turn_count: int, player_role: PlayerRole | None = None, + story_brief: StoryBrief | None = None, + story_guide_context: StoryGuideCompressedContext | None = None, language: TemplateLanguage = "en", ) -> EndingResult: """Generate a 400-600 word ending + label + first-person subtitle. @@ -1788,6 +1841,9 @@ def synthesize_ending( } if player_role is not None: user_payload["player_role"] = player_role.model_dump() + story_contract = _build_turn_story_contract(story_brief, story_guide_context) + if story_contract is not None: + user_payload["story_contract"] = story_contract system_prompt = _ENDING_SYSTEM_PROMPT_TEMPLATE.format( turn_count=turn_count, labels_list=" / ".join(ENDING_LABELS), diff --git a/rpg_backend/narrative/repository.py b/rpg_backend/narrative/repository.py index 8914748b..90bbfdd6 100644 --- a/rpg_backend/narrative/repository.py +++ b/rpg_backend/narrative/repository.py @@ -31,6 +31,8 @@ PlayerGoal, PlayerRole, StepJudgeResult, + StoryBrief, + StoryGuideCompressedContext, StoryMessage, StoryOption, TemplateLanguage, @@ -154,6 +156,8 @@ def _ensure_schema(self, connection: sqlite3.Connection) -> None: ("cover_image_url", "ALTER TABLE narrative_templates ADD COLUMN cover_image_url TEXT"), ("title_i18n_json", "ALTER TABLE narrative_templates ADD COLUMN title_i18n_json TEXT"), ("summary_i18n_json", "ALTER TABLE narrative_templates ADD COLUMN summary_i18n_json TEXT"), + ("story_brief_json", "ALTER TABLE narrative_templates ADD COLUMN story_brief_json TEXT"), + ("story_guide_context_json", "ALTER TABLE narrative_templates ADD COLUMN story_guide_context_json TEXT"), # Pre-i18n templates default to "zh"; this matches the # historic behavior where every template was generated in # Chinese. @@ -306,18 +310,21 @@ def create_template( cover_image_url: str | None = None, title_i18n: LocalizedText | None = None, summary_i18n: LocalizedText | None = None, + story_brief: StoryBrief | None = None, + story_guide_context: StoryGuideCompressedContext | None = None, ) -> NarrativeTemplate: created_at = _utc_now() with self._connect() as conn: conn.execute( """ INSERT INTO narrative_templates - (template_id, owner_user_id, seed, title, title_i18n_json, summary_i18n_json, cast_json, + (template_id, owner_user_id, seed, title, title_i18n_json, summary_i18n_json, + story_brief_json, story_guide_context_json, cast_json, advisor_persona, opening_passage, opening_options_json, player_goals_json, failure_conditions_json, player_role_options_json, cover_image_url, visibility, language, play_count, created_at) - VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 0, ?) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 0, ?) """, ( template_id, @@ -326,6 +333,12 @@ def create_template( title, _dump_localized_text(title_i18n), _dump_localized_text(summary_i18n), + json.dumps(story_brief.model_dump(mode="json"), ensure_ascii=False) + if story_brief is not None + else None, + json.dumps(story_guide_context.model_dump(mode="json"), ensure_ascii=False) + if story_guide_context is not None + else None, json.dumps([c.model_dump() for c in cast], ensure_ascii=False), advisor_persona, opening_passage, @@ -361,6 +374,42 @@ def create_template( created_at=created_at, ) + def get_template_research_context( + self, + template_id: str, + ) -> tuple[StoryBrief | None, StoryGuideCompressedContext | None]: + """Return validated create-time memory seeds for reviewer evaluation. + + Legacy rows and malformed additive metadata deliberately downgrade to + ``None`` instead of making the playable template unreadable. + """ + + with self._connect() as conn: + row = conn.execute( + """ + SELECT story_brief_json, story_guide_context_json + FROM narrative_templates + WHERE template_id = ? + """, + (template_id,), + ).fetchone() + if row is None: + raise NarrativeNotFoundError(f"narrative template not found: {template_id}") + + def load(model: type[StoryBrief] | type[StoryGuideCompressedContext], raw: str | None): + if not raw: + return None + try: + payload = json.loads(raw) + return model.model_validate(payload) if isinstance(payload, dict) else None + except Exception: # noqa: BLE001 + return None + + return ( + load(StoryBrief, row["story_brief_json"]), + load(StoryGuideCompressedContext, row["story_guide_context_json"]), + ) + def get_template(self, template_id: str) -> NarrativeTemplate: with self._connect() as conn: row = conn.execute( diff --git a/rpg_backend/narrative/service.py b/rpg_backend/narrative/service.py index cb14f0f7..97605aeb 100644 --- a/rpg_backend/narrative/service.py +++ b/rpg_backend/narrative/service.py @@ -90,6 +90,11 @@ from rpg_backend.narrative.judges import judge_contract, judge_step from rpg_backend.narrative.repository import NarrativeNotFoundError, NarrativeRepository from rpg_backend.narrative.story_guide import advance_story_guide_loop, story_butler_voice_policy +from rpg_backend.research_runtime.contracts import RpgEvaluationBundleV1 +from rpg_backend.research_runtime.session_export import ( + SessionEvaluationNotReadyError, + build_session_evaluation_bundle, +) PRIVATE_REPLAY_TITLE = "Shared private story" @@ -994,6 +999,12 @@ def create_template( status_code=422, ) + opening = _complete_opening_identity_from_brief( + opening, + brief=request.story_brief, + language=request.language, + ) + template_id = _generate_template_id() display_title, display_intro = _template_display_metadata( seed=seed, @@ -1017,6 +1028,12 @@ def create_template( language=request.language, title_i18n=_localized_text_for_language(display_title, request.language), summary_i18n=_localized_text_for_language(display_intro, request.language), + story_brief=request.story_brief, + story_guide_context=( + request.story_guide_context.model_copy(update={"recent_turns": []}) + if request.story_guide_context is not None + else None + ), ) # Auto-create the creator's session with the requested difficulty. @@ -1198,6 +1215,34 @@ def list_llm_call_events( self._load_session_for_player(session_id, player_user_id) return LLMCallEventListResponse(items=self._repo.list_llm_call_events_for_session(session_id)) + def get_rpg_evaluation_bundle( + self, + session_id: str, + *, + player_user_id: str, + ) -> RpgEvaluationBundleV1: + """Export one owned session into the portable research contract.""" + + session = self._load_session_for_player(session_id, player_user_id) + template = self._repo.get_template(session.template_id) + story_brief, story_guide_context = self._repo.get_template_research_context( + template.template_id + ) + try: + return build_session_evaluation_bundle( + template=template, + session=session, + messages=self._repo.list_story_messages(session_id), + story_brief=story_brief, + story_guide_context=story_guide_context, + ) + except SessionEvaluationNotReadyError as exc: + raise NarrativeServiceError( + code="evaluation_not_ready", + message="Play at least one complete turn before evaluating this run.", + status_code=409, + ) from exc + # ------------------------------------------------------------------ # Advance a turn # ------------------------------------------------------------------ @@ -1218,6 +1263,9 @@ def advance( status_code=409, ) template = self._repo.get_template(session.template_id) + story_brief, story_guide_context = self._repo.get_template_research_context( + template.template_id + ) active_role = _resolve_player_role(template, session.selected_player_role_id) if self._finalize_if_budget_exhausted(session, template, player_role=active_role): raise NarrativeServiceError( @@ -1312,6 +1360,8 @@ def build_deterministic_turn() -> TurnResult: current_inventory=current_inventory or None, player_diary=diary_text, played_leverage=played_leverage, + story_brief=story_brief, + story_guide_context=story_guide_context, language=template.language, ) except NarrativeServiceError as exc: @@ -1652,6 +1702,9 @@ def _finalize_session( budget. """ full_history = self._repo.list_story_messages(session_id) + story_brief, story_guide_context = self._repo.get_template_research_context( + template.template_id + ) try: if self._gateway is None: raise NarrativeServiceError( @@ -1667,6 +1720,8 @@ def _finalize_session( history=full_history, turn_count=len([m for m in full_history if m.role == "narrator"]) - 1, player_role=player_role, + story_brief=story_brief, + story_guide_context=story_guide_context, language=template.language, ) except (NarrativeGatewayError, NarrativeServiceError, ValueError, AttributeError) as exc: @@ -1740,6 +1795,9 @@ def _finalize_session_early( turn. Generate a 'collapsed' ending right now, regardless of turn_budget.""" full_history = self._repo.list_story_messages(session_id) + story_brief, story_guide_context = self._repo.get_template_research_context( + template.template_id + ) try: if self._gateway is None: raise NarrativeServiceError( @@ -1756,6 +1814,8 @@ def _finalize_session_early( failure_trigger=failure_trigger, failure_reason=failure_reason, player_role=player_role, + story_brief=story_brief, + story_guide_context=story_guide_context, language=template.language, ) except (NarrativeGatewayError, NarrativeServiceError, ValueError, AttributeError) as exc: @@ -1911,6 +1971,7 @@ def get_public_replay(self, session_id: str) -> PublicReplayResponse: template_seed=template.seed if is_shareable_template else "", template_title_i18n=template.title_i18n if is_shareable_template else None, template_summary_i18n=template.summary_i18n if is_shareable_template else None, + language=template.language, cast=_public_replay_cast(template.cast) if is_shareable_template else [], advisor_persona=template.advisor_persona if is_shareable_template else "", cover_image_url=template.cover_image_url if is_shareable_template else None, @@ -2918,6 +2979,52 @@ def _story_brief_fallback_opening(brief: StoryBrief, *, language: str) -> Openin ) +def _complete_opening_identity_from_brief( + opening: OpeningResult, + *, + brief: StoryBrief, + language: str, +) -> OpeningResult: + """Fill an omitted live role card from the reviewed Brief contract. + + Role cards are optional in the live opening response so their absence must + not fail an otherwise useful opening. The Play UI still needs a stable + player identity, though. This completion keeps the live passage, cast, + choices, goals, and failure conditions intact and derives only the missing + identity from the already reviewed Story Brief. + """ + if opening.player_role_options or not brief.player_role or not opening.cast: + return opening + if language == "zh": + target = opening.cast[0] + player_roles = [ + PlayerRole( + role_id=_fallback_slug(brief.player_role)[:32] or "role_1", + label=_fallback_label(brief.player_role, limit=24), + public_persona=f"你是{brief.player_role}，需要在当前压力下稳住局面。", + hidden_objective="在局势定型前，让关键人物、压力与线索浮出水面。", + leverages_over_npcs=[ + PlayerLeverageOverNPC( + npc_id=target.character_id, + leverage=f"你知道为什么必须先听到{target.display_name}的立场。", + ) + ], + starting_assets=["行动筹码"], + ) + ] + else: + player_roles = _fallback_player_roles(brief, opening.cast) + return OpeningResult( + title=opening.title, + advisor_persona=opening.advisor_persona, + cast=opening.cast, + opening_message=opening.opening_message, + player_goals=opening.player_goals, + failure_conditions=opening.failure_conditions, + player_role_options=player_roles, + ) + + def _fallback_cast_members( names: list[str], *, diff --git a/rpg_backend/narrative/story_guide.py b/rpg_backend/narrative/story_guide.py index 3be60b50..cc468284 100644 --- a/rpg_backend/narrative/story_guide.py +++ b/rpg_backend/narrative/story_guide.py @@ -484,12 +484,37 @@ def _compress_story_context( previous = state.context player_role = _slot_evidence(state, "player_role") or previous.player_role cast = _compact_list(_split_context_terms(_slot_evidence(state, "active_cast")) or previous.cast_or_factions, limit=8) + role_only_cast_entries = { + role.casefold().removeprefix("the ").strip() + for role in (previous.player_role, player_role) + if role + } + cast = [ + member + for member in cast + if member.casefold().removeprefix("the ").strip() not in role_only_cast_entries + ] pressure = _slot_evidence(state, "pressure") or previous.pressure constraints = _compact_list(_split_context_terms(_slot_evidence(state, "boundaries")) or previous.constraints, limit=8) tone = _slot_evidence(state, "tone") or (settings.tensionProfile or "") or previous.tone scene_summary = _slot_evidence(state, "first_scene_hook") or previous.scene_summary if not scene_summary and state.acceptedTurns: scene_summary = _short_evidence(state.acceptedTurns[0]) + if ( + previous.player_role + and player_role + and previous.player_role.casefold() != player_role.casefold() + ): + scene_summary = _rewrite_player_role_reference( + scene_summary, previous.player_role, player_role + ) + pressure = _rewrite_player_role_reference( + pressure, previous.player_role, player_role + ) + constraints = [ + _rewrite_player_role_reference(item, previous.player_role, player_role) + for item in constraints + ] changed = list(previous.rejected_or_changed_facts) _append_superseded(changed, "player_role", previous.player_role, player_role) @@ -579,6 +604,33 @@ def _append_superseded(changed: list[str], label: str, previous: str, current: s changed.append(f"superseded {label}: {previous}") +def _rewrite_player_role_reference(value: str, previous: str, current: str) -> str: + """Update explicit player-identity clauses without rewriting unrelated cast prose.""" + + if not value or not previous or not current: + return value + old = previous.strip() + new = current.strip() + rewritten = value.replace(f"玩家是{old}", f"玩家是{new}") + rewritten = rewritten.replace(f"我是{old}", f"我是{new}") + old_without_article = re.sub(r"^(?:a|an|the)\s+", "", old, flags=re.I).strip() + if old_without_article: + role_pattern = rf"(?:a\s+|an\s+|the\s+)?{re.escape(old_without_article)}" + rewritten = re.sub( + rf"((?:the\s+)?player\s+is\s+){role_pattern}", + lambda match: f"{match.group(1)}{new}", + rewritten, + flags=re.I, + ) + rewritten = re.sub( + rf"(i(?:'m|\s+am)\s+){role_pattern}", + lambda match: f"{match.group(1)}{new}", + rewritten, + flags=re.I, + ) + return rewritten + + def _fact(label: str, value: str) -> list[str]: cleaned = _short_evidence(value) return [f"{label}: {cleaned}"] if cleaned else [] diff --git a/rpg_backend/research_runtime/__init__.py b/rpg_backend/research_runtime/__init__.py new file mode 100644 index 00000000..f01c09c5 --- /dev/null +++ b/rpg_backend/research_runtime/__init__.py @@ -0,0 +1,31 @@ +"""Portable, reviewer-facing contracts for RPG runtime research.""" + +from rpg_backend.research_runtime.contracts import ( + RpgEvaluationBundleV1, + RpgEvaluationReportV1, + RpgMemoryEventV1, + RpgMemorySnapshotV1, +) +from rpg_backend.research_runtime.evaluator import evaluate_rpg_bundle +from rpg_backend.research_runtime.memory import ( + project_story_guide_memory, + reduce_memory_events, + story_guide_memory_events, +) +from rpg_backend.research_runtime.session_export import ( + SessionEvaluationNotReadyError, + build_session_evaluation_bundle, +) + +__all__ = [ + "RpgEvaluationBundleV1", + "RpgEvaluationReportV1", + "RpgMemoryEventV1", + "RpgMemorySnapshotV1", + "evaluate_rpg_bundle", + "build_session_evaluation_bundle", + "SessionEvaluationNotReadyError", + "project_story_guide_memory", + "reduce_memory_events", + "story_guide_memory_events", +] diff --git a/rpg_backend/research_runtime/contracts.py b/rpg_backend/research_runtime/contracts.py new file mode 100644 index 00000000..69137c1e --- /dev/null +++ b/rpg_backend/research_runtime/contracts.py @@ -0,0 +1,180 @@ +from __future__ import annotations + +from typing import Literal + +from pydantic import BaseModel, ConfigDict, Field, field_validator + + +MemoryEventKind = Literal[ + "objective_set", + "fact_asserted", + "fact_corrected", + "non_story_input", + "thread_opened", + "thread_resolved", + "entity_changed", + "player_action", + "world_consequence", + "clue_unlocked", +] +MemoryFactStatus = Literal["active", "superseded"] +EvaluationStatus = Literal["pass", "warn", "fail"] + + +class RpgMemoryEventV1(BaseModel): + """One bounded state-changing observation from any RPG runtime.""" + + model_config = ConfigDict(extra="forbid") + + event_id: str = Field(min_length=1, max_length=120) + turn_index: int = Field(ge=0, le=500) + kind: MemoryEventKind + namespace: str = Field(default="story", min_length=1, max_length=40) + key: str = Field(default="", max_length=80) + value: str = Field(default="", max_length=600) + entity_id: str | None = Field(default=None, max_length=80) + entity_name: str | None = Field(default=None, max_length=120) + state: dict[str, str] = Field(default_factory=dict, max_length=12) + source: Literal["user", "runtime", "system", "imported"] = "runtime" + confidence: float = Field(default=1.0, ge=0.0, le=1.0) + + @field_validator("event_id", "namespace", "key", "value", "entity_id", "entity_name", mode="before") + @classmethod + def _strip_text(cls, value: object) -> object: + return value.strip() if isinstance(value, str) else value + + +class RpgMemoryFactV1(BaseModel): + model_config = ConfigDict(extra="forbid") + + fact_id: str = Field(min_length=1, max_length=160) + namespace: str = Field(min_length=1, max_length=40) + key: str = Field(min_length=1, max_length=80) + value: str = Field(min_length=1, max_length=600) + status: MemoryFactStatus = "active" + source_turn: int = Field(ge=0, le=500) + source: Literal["user", "runtime", "system", "imported"] = "runtime" + confidence: float = Field(default=1.0, ge=0.0, le=1.0) + superseded_by: str | None = Field(default=None, max_length=160) + + +class RpgMemoryEntityV1(BaseModel): + model_config = ConfigDict(extra="forbid") + + entity_id: str = Field(min_length=1, max_length=80) + name: str = Field(min_length=1, max_length=120) + state: dict[str, str] = Field(default_factory=dict, max_length=12) + last_updated_turn: int = Field(ge=0, le=500) + + +class RpgMemoryDiagnosticsV1(BaseModel): + model_config = ConfigDict(extra="forbid") + + event_count: int = Field(ge=0) + active_fact_count: int = Field(ge=0) + superseded_fact_count: int = Field(ge=0) + non_story_event_count: int = Field(ge=0) + dropped_recent_event_count: int = Field(ge=0) + last_compacted_turn: int = Field(ge=0) + + +class RpgMemorySnapshotV1(BaseModel): + model_config = ConfigDict(extra="forbid") + + schema_version: Literal["rpg_memory.v1"] = "rpg_memory.v1" + run_id: str = Field(min_length=1, max_length=120) + turn_index: int = Field(ge=0, le=500) + objective: str = Field(default="", max_length=600) + active_facts: list[RpgMemoryFactV1] = Field(default_factory=list, max_length=32) + superseded_facts: list[RpgMemoryFactV1] = Field(default_factory=list, max_length=32) + open_threads: list[str] = Field(default_factory=list, max_length=16) + entities: list[RpgMemoryEntityV1] = Field(default_factory=list, max_length=16) + recent_events: list[str] = Field(default_factory=list, max_length=10) + episodic_summary: str = Field(default="", max_length=1200) + diagnostics: RpgMemoryDiagnosticsV1 + + +class RpgStateDeltaV1(BaseModel): + model_config = ConfigDict(extra="forbid") + + target: str = Field(min_length=1, max_length=100) + kind: Literal["increase", "decrease", "set", "unlock", "spend", "shift"] + label: str = Field(min_length=1, max_length=160) + value: str = Field(default="", max_length=120) + evidence: str = Field(default="", max_length=300) + + +class RpgTurnObservationV1(BaseModel): + model_config = ConfigDict(extra="forbid") + + turn_index: int = Field(ge=1, le=500) + player_action: str = Field(min_length=1, max_length=800) + world_response: str = Field(min_length=1, max_length=2400) + # A terminal turn can legitimately have no next actions. The evaluator + # distinguishes that from an agency failure using scenario completion + # evidence instead of rejecting the bundle at parse time. + options: list[str] = Field(default_factory=list, max_length=6) + state_deltas: list[RpgStateDeltaV1] = Field(default_factory=list, max_length=8) + clue_unlocks: list[str] = Field(default_factory=list, max_length=4) + opportunity_unlocks: list[str] = Field(default_factory=list, max_length=4) + referenced_entity_ids: list[str] = Field(default_factory=list, max_length=12) + terminal: bool = False + objective_progress: float = Field(default=0.0, ge=0.0, le=1.0) + progress_basis: Literal["runtime_reported", "turn_budget_proxy", "unknown"] = "unknown" + memory: RpgMemorySnapshotV1 + + +class RpgEvaluationScenarioV1(BaseModel): + model_config = ConfigDict(extra="forbid") + + scenario_id: str = Field(min_length=1, max_length=100) + title: str = Field(min_length=1, max_length=160) + genre: str = Field(default="interactive_rpg", max_length=80) + objective: str = Field(min_length=1, max_length=600) + turn_budget: int = Field(default=12, ge=1, le=80) + entity_ids: list[str] = Field(default_factory=list, max_length=24) + boundaries: list[str] = Field(default_factory=list, max_length=16) + + +class RpgEvaluationBundleV1(BaseModel): + """Portable adapter input for deterministic RPG runtime evaluation.""" + + model_config = ConfigDict(extra="forbid") + + schema_version: Literal["rpg_evaluation_bundle.v1"] = "rpg_evaluation_bundle.v1" + run_id: str = Field(min_length=1, max_length=120) + system_label: str = Field(min_length=1, max_length=120) + locale: Literal["en", "zh", "mixed"] = "en" + scenario: RpgEvaluationScenarioV1 + turns: list[RpgTurnObservationV1] = Field(default_factory=list, min_length=1, max_length=80) + + +class RpgCriterionResultV1(BaseModel): + model_config = ConfigDict(extra="forbid") + + criterion: Literal[ + "memory_continuity", + "memory_boundedness", + "consequence_visibility", + "player_agency", + "trajectory_progress", + "entity_coherence", + "choice_diversity", + "boundary_hygiene", + ] + status: EvaluationStatus + score: int = Field(ge=0, le=100) + summary: str = Field(min_length=1, max_length=240) + evidence: list[str] = Field(default_factory=list, max_length=6) + + +class RpgEvaluationReportV1(BaseModel): + model_config = ConfigDict(extra="forbid") + + schema_version: Literal["rpg_evaluation_report.v1"] = "rpg_evaluation_report.v1" + run_id: str + system_label: str + status: EvaluationStatus + score: int = Field(ge=0, le=100) + criteria: list[RpgCriterionResultV1] = Field(min_length=8, max_length=8) + limitations: list[str] = Field(default_factory=list, max_length=8) diff --git a/rpg_backend/research_runtime/evaluator.py b/rpg_backend/research_runtime/evaluator.py new file mode 100644 index 00000000..b77b2a5d --- /dev/null +++ b/rpg_backend/research_runtime/evaluator.py @@ -0,0 +1,190 @@ +from __future__ import annotations + +import re +from statistics import mean + +from rpg_backend.research_runtime.contracts import ( + EvaluationStatus, + RpgCriterionResultV1, + RpgEvaluationBundleV1, + RpgEvaluationReportV1, +) + + +TECHNICAL_LEAK_RE = re.compile( + r"\b(provider|model id|api key|schema|token count|chain[- ]of[- ]thought|scratchpad|raw json|fallback_used)\b", + re.I, +) + + +def _status(score: int) -> EvaluationStatus: + if score >= 80: + return "pass" + if score >= 55: + return "warn" + return "fail" + + +def _criterion(name: str, score: int, summary: str, evidence: list[str]) -> RpgCriterionResultV1: + return RpgCriterionResultV1( + criterion=name, + status=_status(score), + score=score, + summary=summary, + evidence=evidence[:6], + ) + + +def evaluate_rpg_bundle(bundle: RpgEvaluationBundleV1) -> RpgEvaluationReportV1: + turns = bundle.turns + memory_conflicts: list[str] = [] + memory_overflow: list[str] = [] + consequence_gaps: list[str] = [] + agency_gaps: list[str] = [] + entity_gaps: list[str] = [] + leakage: list[str] = [] + option_signatures: set[tuple[str, ...]] = set() + + previous_progress = 0.0 + progress_regressions = 0 + progress_advances = 0 + progress_bases: set[str] = set() + for turn in turns: + progress_bases.add(turn.progress_basis) + active_keys: set[tuple[str, str]] = set() + superseded_ids = {fact.fact_id for fact in turn.memory.superseded_facts} + for fact in turn.memory.active_facts: + lookup = (fact.namespace.casefold(), fact.key.casefold()) + if lookup in active_keys: + memory_conflicts.append(f"Turn {turn.turn_index}: duplicate active fact {fact.namespace}.{fact.key}") + active_keys.add(lookup) + if fact.fact_id in superseded_ids or fact.status != "active": + memory_conflicts.append(f"Turn {turn.turn_index}: superseded fact remained active") + if len(turn.memory.active_facts) > 24 or len(turn.memory.recent_events) > 8: + memory_overflow.append(f"Turn {turn.turn_index}: memory exceeded the portable budget") + + if not turn.state_deltas and not turn.clue_unlocks and not turn.opportunity_unlocks: + consequence_gaps.append(f"Turn {turn.turn_index}: no typed visible consequence") + normalized_options = tuple(" ".join(option.casefold().split()) for option in turn.options) + if not turn.terminal and len(set(normalized_options)) < 2: + agency_gaps.append(f"Turn {turn.turn_index}: fewer than two distinct options") + if not turn.terminal and normalized_options: + option_signatures.add(normalized_options) + + unknown_entities = set(turn.referenced_entity_ids) - set(bundle.scenario.entity_ids) + if unknown_entities: + entity_gaps.append(f"Turn {turn.turn_index}: unknown entities {', '.join(sorted(unknown_entities))}") + + visible_text = " ".join([turn.player_action, turn.world_response, *turn.options]) + if TECHNICAL_LEAK_RE.search(visible_text): + leakage.append(f"Turn {turn.turn_index}: technical wording appeared in player-facing text") + + if turn.objective_progress + 0.001 < previous_progress: + progress_regressions += 1 + elif turn.objective_progress > previous_progress + 0.05: + progress_advances += 1 + previous_progress = max(previous_progress, turn.objective_progress) + + non_terminal_turn_count = sum(not turn.terminal for turn in turns) + player_agency_score = ( + max(0, round(100 * (non_terminal_turn_count - len(agency_gaps)) / non_terminal_turn_count)) + if non_terminal_turn_count + else 55 + ) + choice_diversity_score = ( + min(100, round(100 * len(option_signatures) / non_terminal_turn_count)) + if non_terminal_turn_count + else 55 + ) + criteria = [ + _criterion( + "memory_continuity", + max(0, 100 - 35 * len(memory_conflicts)), + "Active facts remain singular and corrections are retained as superseded evidence." + if not memory_conflicts + else "Memory contains active/superseded conflicts that can distort later turns.", + memory_conflicts or [f"{len(turns)} snapshots checked with no active fact conflict."], + ), + _criterion( + "memory_boundedness", + max(0, 100 - 40 * len(memory_overflow)), + "Memory stays within the portable fact and recency budgets." + if not memory_overflow + else "One or more snapshots exceeded the bounded memory budget.", + memory_overflow or ["Active facts <= 24 and recent events <= 8 for every turn."], + ), + _criterion( + "consequence_visibility", + round(100 * (len(turns) - len(consequence_gaps)) / len(turns)), + "Moves resolve into explicit deltas, clues, or opportunities." + if not consequence_gaps + else "Some moves only produced prose without a visible game-state consequence.", + consequence_gaps or [f"All {len(turns)} turns expose at least one typed consequence."], + ), + _criterion( + "player_agency", + player_agency_score, + "Each turn preserves at least two distinct next actions." + if non_terminal_turn_count and not agency_gaps + else "A terminal-only run has insufficient evidence for player agency." + if not non_terminal_turn_count + else "At least one turn collapsed into a single or duplicated action path.", + agency_gaps or [f"Distinct choices preserved across {non_terminal_turn_count} non-terminal turns."], + ), + _criterion( + "trajectory_progress", + max(0, min(100, round(previous_progress * 70) + min(30, progress_advances * 10) - progress_regressions * 25)), + "Reported or explicitly proxied progress advances without unexplained regression." + if progress_regressions == 0 + else "Objective progress regressed on at least one turn.", + [ + f"Final observed progress: {previous_progress:.0%}.", + f"Meaningful advances: {progress_advances}; regressions: {progress_regressions}.", + f"Progress basis: {', '.join(sorted(progress_bases))}.", + ], + ), + _criterion( + "entity_coherence", + max(0, 100 - 35 * len(entity_gaps)), + "Referenced people and factions remain inside the scenario registry." + if not entity_gaps + else "The run referenced entities absent from the scenario contract.", + entity_gaps or [f"Validated against {len(bundle.scenario.entity_ids)} registered entities."], + ), + _criterion( + "choice_diversity", + choice_diversity_score, + "Next-action sets change as the trajectory changes." + if non_terminal_turn_count and len(option_signatures) == non_terminal_turn_count + else "A terminal-only run has insufficient evidence for choice diversity." + if not non_terminal_turn_count + else "Some turns repeated the same full action set.", + [f"{len(option_signatures)} distinct action sets across {non_terminal_turn_count} non-terminal turns."], + ), + _criterion( + "boundary_hygiene", + max(0, 100 - 50 * len(leakage)), + "Player-facing text is free of protocol and private-reasoning language." + if not leakage + else "Technical implementation wording leaked into player-facing text.", + leakage or ["No provider, schema, token, raw JSON, or private-reasoning terms detected."], + ), + ] + score = round(mean(item.score for item in criteria)) + limitations = [ + "This deterministic report is a product reliability diagnostic, not a calibrated research metric.", + "Narrative appeal and emotional quality still require bounded human review.", + "Imported state deltas are trusted only to the extent that the source adapter is trustworthy.", + ] + if "turn_budget_proxy" in progress_bases: + limitations.append( + "Trajectory progress for this run uses elapsed turn budget as a proxy, not model-reported goal completion." + ) + return RpgEvaluationReportV1( + run_id=bundle.run_id, + system_label=bundle.system_label, + status=_status(score), + score=score, + criteria=criteria, + limitations=limitations, + ) diff --git a/rpg_backend/research_runtime/memory.py b/rpg_backend/research_runtime/memory.py new file mode 100644 index 00000000..05e8e645 --- /dev/null +++ b/rpg_backend/research_runtime/memory.py @@ -0,0 +1,242 @@ +from __future__ import annotations + +import hashlib +import re +from collections.abc import Iterable + +from rpg_backend.narrative.contracts import StoryGuideCompressedContext +from rpg_backend.research_runtime.contracts import ( + RpgMemoryDiagnosticsV1, + RpgMemoryEntityV1, + RpgMemoryEventV1, + RpgMemoryFactV1, + RpgMemorySnapshotV1, +) + + +def _fact_id(namespace: str, key: str, value: str, turn_index: int) -> str: + digest = hashlib.sha1(f"{namespace}:{key}:{value}:{turn_index}".encode("utf-8")).hexdigest()[:12] + return f"fact_{digest}" + + +def _clean(value: str, *, limit: int) -> str: + return " ".join(value.split())[:limit].strip() + + +def reduce_memory_events( + run_id: str, + events: Iterable[RpgMemoryEventV1], + *, + max_active_facts: int = 24, + max_recent_events: int = 8, +) -> RpgMemorySnapshotV1: + """Reduce portable events into a bounded, auditable memory snapshot.""" + + ordered = sorted(events, key=lambda event: (event.turn_index, event.event_id)) + objective = "" + active: dict[tuple[str, str], RpgMemoryFactV1] = {} + superseded: list[RpgMemoryFactV1] = [] + open_threads: list[str] = [] + entities: dict[str, RpgMemoryEntityV1] = {} + recent: list[str] = [] + non_story_count = 0 + + for event in ordered: + if event.kind == "objective_set": + objective = _clean(event.value, limit=600) + recent.append(f"Objective: {objective}") + continue + if event.kind in {"fact_asserted", "fact_corrected"}: + key = _clean(event.key, limit=80) + value = _clean(event.value, limit=600) + if not key or not value: + continue + lookup = (event.namespace, key) + fact_id = _fact_id(event.namespace, key, value, event.turn_index) + previous = active.get(lookup) + if previous and previous.value.casefold() != value.casefold(): + superseded.append(previous.model_copy(update={"status": "superseded", "superseded_by": fact_id})) + active[lookup] = RpgMemoryFactV1( + fact_id=fact_id, + namespace=event.namespace, + key=key, + value=value, + source_turn=event.turn_index, + source=event.source, + confidence=event.confidence, + ) + recent.append(f"{key}: {value}") + continue + if event.kind == "non_story_input": + non_story_count += 1 + recent.append(f"Conversation only: {_clean(event.value, limit=180)}") + continue + if event.kind == "thread_opened": + thread = _clean(event.value or event.key, limit=180) + if thread and thread.casefold() not in {item.casefold() for item in open_threads}: + open_threads.append(thread) + continue + if event.kind == "thread_resolved": + target = _clean(event.value or event.key, limit=180).casefold() + open_threads = [item for item in open_threads if item.casefold() != target] + continue + if event.kind == "entity_changed" and event.entity_id: + current = entities.get(event.entity_id) + state = dict(current.state) if current else {} + state.update({_clean(key, limit=60): _clean(value, limit=160) for key, value in event.state.items()}) + entities[event.entity_id] = RpgMemoryEntityV1( + entity_id=event.entity_id, + name=event.entity_name or (current.name if current else event.entity_id), + state=state, + last_updated_turn=event.turn_index, + ) + recent.append(f"{entities[event.entity_id].name}: {', '.join(state.values())}") + continue + label = { + "player_action": "Player", + "world_consequence": "World", + "clue_unlocked": "Clue", + }.get(event.kind, "Event") + recent.append(f"{label}: {_clean(event.value, limit=240)}") + + active_facts = sorted(active.values(), key=lambda fact: (fact.source_turn, fact.fact_id))[-max_active_facts:] + superseded_facts = superseded[-32:] + dropped = max(0, len(recent) - max_recent_events) + recent_events = recent[-max_recent_events:] + last_turn = max((event.turn_index for event in ordered), default=0) + summary_parts = [objective] if objective else [] + summary_parts.extend(f"{fact.key}: {fact.value}" for fact in active_facts[-6:]) + if open_threads: + summary_parts.append(f"Open: {'; '.join(open_threads[:4])}") + return RpgMemorySnapshotV1( + run_id=run_id, + turn_index=last_turn, + objective=objective, + active_facts=active_facts, + superseded_facts=superseded_facts, + open_threads=open_threads[-16:], + entities=sorted(entities.values(), key=lambda entity: entity.entity_id), + recent_events=recent_events, + episodic_summary=_clean(" | ".join(summary_parts), limit=1200), + diagnostics=RpgMemoryDiagnosticsV1( + event_count=len(ordered), + active_fact_count=len(active_facts), + superseded_fact_count=len(superseded_facts), + non_story_event_count=non_story_count, + dropped_recent_event_count=dropped, + last_compacted_turn=last_turn, + ), + ) + + +def story_guide_memory_events( + context: StoryGuideCompressedContext, + *, + turn_index: int, +) -> list[RpgMemoryEventV1]: + """Convert bounded Story Butler state into portable memory events.""" + + events: list[RpgMemoryEventV1] = [] + sequence = 0 + + def add( + kind: str, + *, + key: str = "", + value: str = "", + namespace: str = "story", + event_turn: int | None = None, + ) -> None: + nonlocal sequence + sequence += 1 + events.append( + RpgMemoryEventV1( + event_id=f"guide_{turn_index}_{sequence}", + turn_index=turn_index if event_turn is None else event_turn, + kind=kind, + namespace=namespace, + key=key, + value=value, + source="runtime", + ) + ) + + superseded_keys: set[str] = set() + for item in context.rejected_or_changed_facts: + match = re.match(r"superseded\s+([^:]+):\s*(.+)", item, re.I) + if not match: + continue + key, value = match.groups() + clean_key = key.strip() + superseded_keys.add(clean_key.casefold()) + add( + "fact_asserted", + key=clean_key, + value=value.strip(), + event_turn=max(0, turn_index - 1), + ) + + if context.scene_summary: + add("objective_set", value=context.scene_summary) + for key, value in ( + ("scene_summary", context.scene_summary), + ("player_role", context.player_role), + ("pressure", context.pressure), + ("tone", context.tone), + ): + if value: + add( + "fact_corrected" if key.casefold() in superseded_keys else "fact_asserted", + key=key, + value=value, + ) + for index, value in enumerate(context.cast_or_factions): + add("fact_asserted", key=f"cast_{index + 1}", value=value, namespace="cast") + for index, value in enumerate(context.constraints): + add("fact_asserted", key=f"constraint_{index + 1}", value=value, namespace="boundary") + existing_values = { + value.casefold() + for value in ( + context.scene_summary, + context.player_role, + context.pressure, + context.tone, + *context.cast_or_factions, + *context.constraints, + ) + if value + } + for index, value in enumerate(context.confirmed_facts): + if value.casefold() not in existing_values: + add("fact_asserted", key=f"confirmed_{index + 1}", value=value) + for item in context.non_story_user_intents: + add("non_story_input", value=item, namespace="conversation") + for item in context.open_questions: + add("thread_opened", value=item, namespace="question") + if context.planner_job: + add("world_consequence", value=f"Planner focus: {context.planner_job}", namespace="planner") + + return events + + +def project_story_guide_memory( + context: StoryGuideCompressedContext, + *, + run_id: str, + turn_index: int, +) -> RpgMemorySnapshotV1: + """Project the existing Story Butler memory into the portable contract.""" + + events = story_guide_memory_events(context, turn_index=turn_index) + snapshot = reduce_memory_events(run_id, events) + return snapshot.model_copy( + update={ + "turn_index": turn_index, + "objective": context.scene_summary, + "diagnostics": snapshot.diagnostics.model_copy( + update={ + "last_compacted_turn": turn_index, + } + ), + } + ) diff --git a/rpg_backend/research_runtime/session_export.py b/rpg_backend/research_runtime/session_export.py new file mode 100644 index 00000000..1e598bdf --- /dev/null +++ b/rpg_backend/research_runtime/session_export.py @@ -0,0 +1,327 @@ +from __future__ import annotations + +from collections.abc import Iterable + +from rpg_backend.narrative.contracts import ( + GameplayChip, + NarrativeSession, + NarrativeTemplate, + NPCPulse, + PlayerRole, + StoryBrief, + StoryGuideCompressedContext, + StoryMessage, +) +from rpg_backend.research_runtime.contracts import ( + RpgEvaluationBundleV1, + RpgEvaluationScenarioV1, + RpgMemoryEventV1, + RpgStateDeltaV1, + RpgTurnObservationV1, +) +from rpg_backend.research_runtime.memory import reduce_memory_events, story_guide_memory_events + + +class SessionEvaluationNotReadyError(ValueError): + """Raised only when a session has no completed player/world turn.""" + + +def _clean(value: str, limit: int) -> str: + return " ".join(value.split())[:limit].strip() + + +def _active_role(template: NarrativeTemplate, session: NarrativeSession) -> PlayerRole | None: + if not template.player_role_options: + return None + if session.selected_player_role_id: + for role in template.player_role_options: + if role.role_id == session.selected_player_role_id: + return role + return template.player_role_options[0] + + +def _objective( + template: NarrativeTemplate, + session: NarrativeSession, + story_brief: StoryBrief | None, +) -> str: + if template.player_goals: + return _clean(template.player_goals[0].goal, 600) + role = _active_role(template, session) + if role is not None: + return _clean(role.hidden_objective, 600) + if story_brief is not None: + return _clean(story_brief.story_kernel or story_brief.premise_summary, 600) + return _clean(template.seed or template.title, 600) + + +def _append_event( + events: list[RpgMemoryEventV1], + *, + turn_index: int, + kind: str, + value: str = "", + key: str = "", + namespace: str = "story", + entity_id: str | None = None, + entity_name: str | None = None, + state: dict[str, str] | None = None, +) -> None: + events.append( + RpgMemoryEventV1( + event_id=f"session_{turn_index}_{len(events) + 1}", + turn_index=turn_index, + kind=kind, + value=_clean(value, 600), + key=_clean(key, 80), + namespace=_clean(namespace, 40), + entity_id=_clean(entity_id, 80) if entity_id else None, + entity_name=_clean(entity_name, 120) if entity_name else None, + state={_clean(k, 80): _clean(v, 600) for k, v in (state or {}).items()}, + source="runtime", + ) + ) + + +def _seed_memory_events( + *, + template: NarrativeTemplate, + session: NarrativeSession, + story_brief: StoryBrief | None, + story_guide_context: StoryGuideCompressedContext | None, +) -> list[RpgMemoryEventV1]: + events = ( + story_guide_memory_events(story_guide_context, turn_index=0) + if story_guide_context is not None + else [] + ) + _append_event(events, turn_index=0, kind="objective_set", value=_objective(template, session, story_brief)) + if story_brief is not None: + for key, value in ( + ("premise", story_brief.premise_summary), + ("player_role", story_brief.player_role or ""), + ("tone", story_brief.genre_tone), + ("story_kernel", story_brief.story_kernel), + ): + if value: + _append_event(events, turn_index=0, kind="fact_asserted", key=key, value=value) + for index, constraint in enumerate(story_brief.preserved_constraints[:8]): + _append_event( + events, + turn_index=0, + kind="fact_asserted", + namespace="boundary", + key=f"constraint_{index + 1}", + value=constraint, + ) + role = _active_role(template, session) + if role is not None: + _append_event(events, turn_index=0, kind="fact_asserted", key="play_role", value=role.label) + return events + + +def _pulse_delta(pulse: NPCPulse, names: dict[str, str]) -> RpgStateDeltaV1: + kind = { + "warmer": "increase", + "colder": "decrease", + "steady": "set", + "wary": "shift", + "broken": "decrease", + }[pulse.shift] + name = names.get(pulse.npc_id, pulse.npc_id) + return RpgStateDeltaV1( + target=pulse.npc_id, + kind=kind, + label=_clean(f"{name}: {pulse.shift}", 160), + value=_clean(pulse.state, 120), + evidence=_clean(pulse.reason or pulse.state, 300), + ) + + +def _chip_delta(chip: GameplayChip, *, target: str = "scene") -> RpgStateDeltaV1: + kind = { + "gain": "increase", + "cost": "decrease", + "unlock": "unlock", + "shift": "shift", + }[chip.tone] + return RpgStateDeltaV1( + target=target, + kind=kind, + label=_clean(chip.label, 160), + evidence=_clean(chip.detail or chip.label, 300), + ) + + +def _dedupe(values: Iterable[str], *, limit: int) -> list[str]: + result: list[str] = [] + seen: set[str] = set() + for value in values: + cleaned = _clean(value, 160) + key = cleaned.casefold() + if not cleaned or key in seen: + continue + seen.add(key) + result.append(cleaned) + if len(result) >= limit: + break + return result + + +def build_session_evaluation_bundle( + *, + template: NarrativeTemplate, + session: NarrativeSession, + messages: list[StoryMessage], + story_brief: StoryBrief | None = None, + story_guide_context: StoryGuideCompressedContext | None = None, +) -> RpgEvaluationBundleV1: + """Export one persisted Tiny Stories session into the portable contract. + + The adapter uses accepted structured metadata when present and conservative + persisted-state derivation otherwise. It never invokes an LLM. + """ + + ordered = sorted(messages, key=lambda message: message.ord) + player_messages = [message for message in ordered if message.role == "player"] + names = {member.character_id: member.display_name for member in template.cast} + objective = _objective(template, session, story_brief) + memory_events = _seed_memory_events( + template=template, + session=session, + story_brief=story_brief, + story_guide_context=story_guide_context, + ) + observations: list[RpgTurnObservationV1] = [] + + for turn_index, player in enumerate(player_messages, start=1): + narrator = next( + ( + message + for message in ordered + if message.role == "narrator" and message.ord > player.ord + ), + None, + ) + if narrator is None: + continue + + _append_event( + memory_events, + turn_index=turn_index, + kind="player_action", + value=player.content, + namespace="episode", + ) + _append_event( + memory_events, + turn_index=turn_index, + kind="world_consequence", + value=narrator.content, + namespace="episode", + ) + + state_deltas = [_pulse_delta(pulse, names) for pulse in narrator.npc_pulse] + referenced_entities = [pulse.npc_id for pulse in narrator.npc_pulse if pulse.npc_id in names] + for pulse in narrator.npc_pulse: + _append_event( + memory_events, + turn_index=turn_index, + kind="entity_changed", + namespace="entity", + entity_id=pulse.npc_id, + entity_name=names.get(pulse.npc_id, pulse.npc_id), + state={"state": pulse.state, "shift": pulse.shift}, + ) + + inventory_added = narrator.inventory_delta.added if narrator.inventory_delta else [] + inventory_removed = narrator.inventory_delta.removed if narrator.inventory_delta else [] + for item in inventory_added: + state_deltas.append( + RpgStateDeltaV1( + target="inventory", + kind="unlock", + label=_clean(f"Evidence: {item}", 160), + evidence=_clean(narrator.inventory_delta.reason if narrator.inventory_delta else item, 300), + ) + ) + _append_event(memory_events, turn_index=turn_index, kind="clue_unlocked", value=item) + for item in inventory_removed: + state_deltas.append( + RpgStateDeltaV1( + target="inventory", + kind="spend", + label=_clean(f"Spent: {item}", 160), + evidence=_clean(narrator.inventory_delta.reason if narrator.inventory_delta else item, 300), + ) + ) + + metadata = narrator.gameplay_metadata + clue_unlocks = list(inventory_added) + opportunity_unlocks: list[str] = [] + if metadata is not None: + state_deltas.extend(_chip_delta(chip) for chip in metadata.state_deltas) + if metadata.motive_effect is not None: + state_deltas.append(_chip_delta(metadata.motive_effect, target="motive")) + clue_unlocks.extend(chip.label for chip in metadata.clue_unlocks) + opportunity_unlocks.extend(chip.label for chip in metadata.opportunity_unlocks) + state_deltas.extend(_chip_delta(chip, target="clue") for chip in metadata.clue_unlocks) + state_deltas.extend( + _chip_delta(chip, target="opportunity") for chip in metadata.opportunity_unlocks + ) + for clue in metadata.clue_unlocks: + _append_event(memory_events, turn_index=turn_index, kind="clue_unlocked", value=clue.label) + + unique_deltas: list[RpgStateDeltaV1] = [] + delta_labels: set[str] = set() + for delta in state_deltas: + key = delta.label.casefold() + if key not in delta_labels: + delta_labels.add(key) + unique_deltas.append(delta) + + terminal = bool(session.ending_label) and turn_index == len(player_messages) + progress = 1.0 if terminal else min(0.95, turn_index / max(1, session.turn_budget)) + observations.append( + RpgTurnObservationV1( + turn_index=turn_index, + player_action=_clean(player.content, 800), + world_response=_clean(narrator.content, 2400), + options=[_clean(option.label, 800) for option in narrator.options[:6]], + state_deltas=unique_deltas[:8], + clue_unlocks=_dedupe(clue_unlocks, limit=4), + opportunity_unlocks=_dedupe(opportunity_unlocks, limit=4), + referenced_entity_ids=_dedupe(referenced_entities, limit=12), + terminal=terminal, + objective_progress=progress, + progress_basis="turn_budget_proxy", + memory=reduce_memory_events(session.session_id, memory_events), + ) + ) + + if not observations: + raise SessionEvaluationNotReadyError( + "Session has no completed player/narrator turn to evaluate." + ) + + boundaries: list[str] = [] + if story_guide_context is not None: + boundaries.extend(story_guide_context.constraints) + if story_brief is not None: + boundaries.extend(story_brief.preserved_constraints) + + return RpgEvaluationBundleV1( + run_id=session.session_id, + system_label="Tiny Stories runtime", + locale=template.language, + scenario=RpgEvaluationScenarioV1( + scenario_id=template.template_id, + title=template.title, + genre=story_brief.genre_tone if story_brief is not None else "interactive_rpg", + objective=objective, + turn_budget=session.turn_budget, + entity_ids=[member.character_id for member in template.cast], + boundaries=_dedupe(boundaries, limit=16), + ), + turns=observations, + ) diff --git a/tests/test_korean_agent_chat_reviewer_contract.py b/tests/test_korean_agent_chat_reviewer_contract.py index b7539c6d..814d747a 100644 --- a/tests/test_korean_agent_chat_reviewer_contract.py +++ b/tests/test_korean_agent_chat_reviewer_contract.py @@ -54,27 +54,21 @@ def test_reviewer_launch_explains_async_progress_to_external_reviewers() -> None assert ".reviewer-launch-plan__head" in theme -def test_reviewer_page_normalizes_direct_entry_to_english() -> None: +def test_reviewer_page_preserves_ui_language_until_locked_english_launch() -> None: source = (ROOT / "frontend2/src/pages/portfolio/reviewer-page.tsx").read_text() - mount_language_idx = source.index('useEffect(() => {\n setLang("en")') handle_start_idx = source.index("const handleStart = async") - assert mount_language_idx < handle_start_idx - assert '}, [setLang])' in source[mount_language_idx:handle_start_idx] + assert 'setLang("en")' not in source[:handle_start_idx] assert 'setLang("en")' in source[source.index("const handleStart = async") :] assert "A locked English demo path designed for portfolio review" in source -def test_portfolio_page_normalizes_application_entry_to_english() -> None: +def test_portfolio_page_preserves_the_selected_application_language() -> None: source = (ROOT / "frontend2/src/pages/portfolio/portfolio-page.tsx").read_text() - mount_language_idx = source.index('useEffect(() => {\n setLang("en")') - hero_idx = source.index("

Tiny Stories") - - assert mount_language_idx < hero_idx - assert '}, [setLang])' in source[mount_language_idx:hero_idx] - assert 'import { useLanguage } from "../../shared/lib/i18n"' in source + assert 'setLang("en")' not in source + assert 'import { useLanguage } from "../../shared/lib/i18n"' not in source assert "Portfolio Case Study" in source assert "portfolio-grade AI product-system evidence" in source assert "not a launched consumer adoption claim" in source diff --git a/tests/test_narrative_create_prompt_shape.py b/tests/test_narrative_create_prompt_shape.py index 5976607c..76605f84 100644 --- a/tests/test_narrative_create_prompt_shape.py +++ b/tests/test_narrative_create_prompt_shape.py @@ -207,6 +207,63 @@ def test_generate_opening_injects_reviewed_story_brief() -> None: assert "Do not convert a comedy/cozy brief" in payload["story_brief_generation_rules"] +def test_missing_live_player_role_is_completed_from_reviewed_brief() -> None: + brief = build_story_brief( + seed=( + "颁奖礼直播前歌手失踪，玩家是公关，制作人、赞助商和主持人都在控制室里施压。" + "倒计时已经开始，不要血腥。" + ), + language="zh", + desired_tension_profile="high_drama", + ).brief.model_copy(update={"player_role": "公关"}) + opening = narrative_service_module.OpeningResult( + title="直播暗箱", + advisor_persona="一位场外顾问守着电话。", + cast=[ + CastMember( + character_id="singer", + display_name="歌手", + role="失踪的表演者", + relation_to_protagonist="等待玩家找到她。", + ), + CastMember( + character_id="producer", + display_name="制作人", + role="直播负责人", + relation_to_protagonist="要求节目继续。", + ), + CastMember( + character_id="host", + display_name="主持人", + role="台前主持", + relation_to_protagonist="等待新的说辞。", + ), + ], + opening_message=StoryMessage( + ord=0, + role="narrator", + content="倒计时还在走，控制室里所有人都看向你。", + options=[StoryOption(label="查看监控", hint="寻找线索", handle="查监控")], + ), + player_goals=[], + failure_conditions=[], + player_role_options=[], + ) + + completed = narrative_service_module._complete_opening_identity_from_brief( + opening, + brief=brief, + language="zh", + ) + + assert completed.opening_message is opening.opening_message + assert completed.cast is opening.cast + assert completed.player_goals == [] + assert [role.label for role in completed.player_role_options] == ["公关"] + assert completed.player_role_options[0].hidden_objective.startswith("在局势定型前") + assert completed.player_role_options[0].starting_assets == ["行动筹码"] + + def test_create_template_blocks_explicit_small_cast_prompt_before_llm( tmp_path, monkeypatch: pytest.MonkeyPatch, diff --git a/tests/test_navigation_mental_model_contract.py b/tests/test_navigation_mental_model_contract.py index 45a775f3..be2efb4a 100644 --- a/tests/test_navigation_mental_model_contract.py +++ b/tests/test_navigation_mental_model_contract.py @@ -1466,6 +1466,25 @@ def test_replay_fixture_uses_real_replay_page_for_local_evidence() -> None: assert "local-only application evidence until the public-link check passes" in code_map +def test_story_language_drives_play_replay_and_evaluation_without_portfolio_side_effects() -> None: + play = (ROOT / "frontend2/src/pages/play/play-page.tsx").read_text() + replay = (ROOT / "frontend2/src/pages/replay/replay-page.tsx").read_text() + evaluation = (ROOT / "frontend2/src/pages/evaluation/rpg-evaluation-page.tsx").read_text() + portfolio = (ROOT / "frontend2/src/pages/portfolio/portfolio-page.tsx").read_text() + reviewer = (ROOT / "frontend2/src/pages/portfolio/reviewer-page.tsx").read_text() + contracts = (ROOT / "rpg_backend/narrative/contracts.py").read_text() + strings = (ROOT / "frontend2/src/shared/lib/i18n.ts").read_text() + + assert "setLang(response.template.language)" in play + assert "setLang(r.language)" in replay + assert 'if (next.locale !== "mixed") setLang(next.locale)' in evaluation + assert "language: TemplateLanguage" in contracts + assert 'setLang("en")\n }, [setLang])' not in portfolio + assert 'setLang("en")\n }, [setLang])' not in reviewer + assert '"play.mood_context_waiting": "房间在等你的下一步。"' in strings + assert '"play.gameplay_track_pressure": "压力"' in strings + + def test_source_maps_use_current_story_butler_product_framing() -> None: code_map = (ROOT / "docs/tiny-stories-code-map.md").read_text() frontend_readme = (ROOT / "frontend2/src/README.md").read_text() diff --git a/tests/test_play_direction_a_editorial_primitives_contract.py b/tests/test_play_direction_a_editorial_primitives_contract.py index ccd9141f..e44f4e17 100644 --- a/tests/test_play_direction_a_editorial_primitives_contract.py +++ b/tests/test_play_direction_a_editorial_primitives_contract.py @@ -424,7 +424,7 @@ def test_run_context_stage_label_module_owns_passive_fallback_stage_copy() -> No assert 'return "Coda"' not in stage_label assert '"stage_bar.pre_finale": "Final stretch"' in strings assert '"stage_bar.pre_finale": "Coda"' not in strings - assert '"Final stretch"' in mood_plate + assert 't("play.mood_stage_final")' in mood_plate assert '"Coda"' not in mood_plate assert 'return stage.replace(/_/g, " ")' in stage_label assert "function stageDisplayName" not in panels @@ -785,7 +785,7 @@ def test_scene_support_rail_uses_webtoon_portrait_images() -> None: assert "onFocusActor={focusSceneActor}" in play_page assert "onClearActorFocus={() => setFocusedActorId(null)}" in play_page assert "onAskAdvisor={openAdvisor}" in play_page - assert 'title="People you can involve"' in primitives + assert 'title={t("play.rail_people")}' in primitives assert "play.advisor_card_name" in primitives assert "play.advisor_card_background" in primitives assert "advisorAskTitle" in primitives @@ -1026,15 +1026,15 @@ def test_finish_mode_normal_play_reduces_top_metadata_density() -> None: assert "moodPlateComplete" in mood_plate assert "moodPlateCopyComplete" in mood_plate assert "moodTitleComplete" in mood_plate - assert "This run is finished. Review the ending, then replay or share it." in mood_plate + assert 't("play.mood_context_complete")' in mood_plate assert "{!isComplete ? (" in mood_plate assert "cast={story.template.cast.map" not in play_page assert "const castLine" not in mood_plate assert "story.template.seed" not in mood_plate - assert "First shot is live" in mood_plate - assert "The room is waiting." in mood_plate - assert "turns to play" in mood_plate - assert "turns played" in mood_plate + assert 't("play.mood_context_opening")' in mood_plate + assert 't("play.mood_context_waiting")' in mood_plate + assert 't("play.mood_progress_empty"' in mood_plate + assert 't("play.mood_progress_played"' in mood_plate assert "Turn ${turnsCompleted} of ${turnBudget}" not in mood_plate assert "The latest beat is ready for your next move." not in mood_plate assert '' not in scene_rail @@ -1591,7 +1591,7 @@ def test_normal_play_prefers_backend_gameplay_envelope_with_derived_backup() -> assert 'data-gameplay-evidence-resource={focusableTrackId === "evidence" ? "true" : undefined}' in play_page assert "data-gameplay-resource-focus={isFocused ? \"true\" : undefined}" in play_page assert "data-gameplay-resource-action-count={resourceMatchCount}" in play_page - assert 'aria-label={`${track.label}: ${track.value}. ${resourceActionLabel ? `${resourceActionLabel}. ` : ""}${focusTitle}`}' in play_page + assert 'aria-label={`${displayLabel}: ${displayValue}. ${resourceActionLabel ? `${resourceActionLabel}. ` : ""}${focusTitle}`}' in play_page assert 'aria-label={`${track.label}: ${track.value}. ${focusTitle}`}' not in play_page assert 't("play.resource_focus_active_count_one")' in play_page assert 't("play.resource_focus_active_count_many", { count: resourceMatchCount })' in play_page @@ -1650,7 +1650,10 @@ def test_normal_play_prefers_backend_gameplay_envelope_with_derived_backup() -> assert "focusedActorId={focusedActorId}" in play_page assert "onFocusActor={focusSceneActor}" in play_page assert "!isComplete && !busy && turnsCompleted > 0" in play_page - assert "actionForecasts={gameplayEnvelope.actionForecasts}" in play_page + assert "const displayActionForecasts = localizedActionForecasts(t, gameplayEnvelope.actionForecasts)" in play_page + assert "actionForecasts={displayActionForecasts}" in play_page + assert "function gameplayForecastLabel(" in play_page + assert 't("play.gameplay_forecast_read_room")' in play_page assert "actorFocus={actorFocus}" in play_page assert "resourceFocus={focusedResourceId && focusedResourceTrack" in play_page assert "const compactTracks = useCompactLayout(\"(max-width: 680px)\")" in play_page @@ -2008,6 +2011,10 @@ def test_normal_play_prefers_backend_gameplay_envelope_with_derived_backup() -> assert '"play.gameplay_tracks_label": "风险与资源"' in strings assert '"play.gameplay_tracks_hint": "每次行动都在交换时间、压力、信任或证据"' in strings assert '"play.gameplay_tracks_hint": "选择行动时盯住这些变化"' not in strings + assert '"play.gameplay_track_none": "暂无"' in strings + assert '"play.gameplay_forecast_read_room": "观察局面"' in strings + assert '"play.gameplay_forecast_evidence": "可能发现证据"' in strings + assert '"play.advisor_card_name": "黛娜·维尔"' in strings assert "gameplayStakesHeader" in styles assert '"play.feedback_source_move_label": "From your move"' in strings assert '"play.gameplay_decision_forecast_label": "Likely impact"' in strings diff --git a/tests/test_rpg_evaluation_lab_contract.py b/tests/test_rpg_evaluation_lab_contract.py new file mode 100644 index 00000000..108a3c1e --- /dev/null +++ b/tests/test_rpg_evaluation_lab_contract.py @@ -0,0 +1,87 @@ +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[1] + + +def test_public_rpg_evaluation_lab_is_backend_free_and_routable() -> None: + routes = (ROOT / "frontend2/src/app/routes.ts").read_text(encoding="utf-8") + app = (ROOT / "frontend2/src/app/app.tsx").read_text(encoding="utf-8") + auth = (ROOT / "frontend2/src/app/auth-context.tsx").read_text(encoding="utf-8") + page = (ROOT / "frontend2/src/pages/evaluation/rpg-evaluation-page.tsx").read_text(encoding="utf-8") + + assert 'name: "rpgEvaluation"' in routes + assert '#/lab/rpg-evaluation' in routes + assert 'VITE_PUBLIC_LAB_DEFAULT === "true"' in routes + assert " None: + page = (ROOT / "frontend2/src/pages/evaluation/rpg-evaluation-page.tsx").read_text(encoding="utf-8") + contracts = (ROOT / "frontend2/src/pages/evaluation/rpg-evaluation-contract.ts").read_text(encoding="utf-8") + styles = (ROOT / "frontend2/src/app/theme.css").read_text(encoding="utf-8") + + for hook in ( + "data-rpg-evaluation-summary", + "data-rpg-evaluation-criterion", + "data-rpg-evaluation-turn", + "data-rpg-evaluation-memory", + "data-rpg-evaluation-comparison", + "data-rpg-evaluation-boundary", + ): + assert hook in page + + assert '"rpg_evaluation_bundle.v1"' in contracts + assert '"rpg_memory.v1"' in contracts + assert "memory_continuity" in contracts + assert "consequence_visibility" in contracts + assert "player_agency" in contracts + assert "boundary_hygiene" in contracts + assert "@media (max-width: 760px)" in styles + assert "@media (prefers-reduced-motion: reduce)" in styles + + +def test_vercel_build_defaults_to_static_evaluation_lab() -> None: + config = (ROOT / "frontend2/vercel.json").read_text(encoding="utf-8") + + assert '"framework": "vite"' in config + assert "VITE_PUBLIC_LAB_DEFAULT=true npm run build" in config + assert '"outputDirectory": "dist"' in config + + +def test_live_session_export_connects_create_memory_play_and_reviewer_lab() -> None: + create_page = (ROOT / "frontend2/src/pages/create/create-page.tsx").read_text(encoding="utf-8") + routes = (ROOT / "frontend2/src/api/route-map.ts").read_text(encoding="utf-8") + client = (ROOT / "frontend2/src/api/http-client.ts").read_text(encoding="utf-8") + page = (ROOT / "frontend2/src/pages/evaluation/rpg-evaluation-page.tsx").read_text(encoding="utf-8") + inspector = (ROOT / "frontend2/src/pages/play/components/runtime-inspector.tsx").read_text(encoding="utf-8") + backend = (ROOT / "rpg_backend/main.py").read_text(encoding="utf-8") + repository = (ROOT / "rpg_backend/narrative/repository.py").read_text(encoding="utf-8") + + assert "story_guide_context: guideLoopState.context" in create_page + assert "story_brief_json" in repository + assert "story_guide_context_json" in repository + assert "/narrative/sessions/:session_id/evaluation-bundle" in routes + assert '"/research/rpg-evaluations"' in routes + assert "getNarrativeEvaluationBundle(sessionId" in client + assert 'data-rpg-live-session-loader="true"' in page + assert 'VITE_PUBLIC_LAB_DEFAULT !== "true"' in page + assert 'data-reviewer-open-portable-evaluation="true"' in inspector + assert '"/narrative/sessions/{session_id}/evaluation-bundle"' in backend + assert '"/research/rpg-evaluations"' in backend + vite_config = (ROOT / "frontend2/vite.config.ts").read_text(encoding="utf-8") + assert "narrative|research" in vite_config + + +def test_frontend_evaluator_handles_terminal_turns_and_progress_provenance() -> None: + contract = (ROOT / "frontend2/src/pages/evaluation/rpg-evaluation-contract.ts").read_text(encoding="utf-8") + + assert "terminal?: boolean" in contract + assert '"turn_budget_proxy"' in contract + assert "!turn.terminal" in contract + assert "nonTerminalTurns" in contract + assert "Progress basis:" in contract diff --git a/tests/test_rpg_research_runtime.py b/tests/test_rpg_research_runtime.py new file mode 100644 index 00000000..5336db31 --- /dev/null +++ b/tests/test_rpg_research_runtime.py @@ -0,0 +1,215 @@ +from __future__ import annotations + +from rpg_backend.narrative.contracts import StoryGuideCompressedContext +from rpg_backend.research_runtime.contracts import ( + RpgEvaluationBundleV1, + RpgEvaluationScenarioV1, + RpgMemoryEventV1, + RpgStateDeltaV1, + RpgTurnObservationV1, +) +from rpg_backend.research_runtime.evaluator import evaluate_rpg_bundle +from rpg_backend.research_runtime.memory import project_story_guide_memory, reduce_memory_events + + +def _event(event_id: str, turn_index: int, kind: str, **kwargs: object) -> RpgMemoryEventV1: + return RpgMemoryEventV1(event_id=event_id, turn_index=turn_index, kind=kind, **kwargs) + + +def test_memory_reducer_supersedes_corrections_and_bounds_recent_events() -> None: + events = [ + _event("e0", 0, "objective_set", value="Find the singer before air."), + _event("e1", 1, "fact_asserted", key="player_role", value="Backup dancer", source="user"), + _event("e2", 2, "fact_corrected", key="player_role", value="Publicist", source="user"), + _event("e3", 2, "non_story_input", value="who are you", source="user"), + *[ + _event(f"e{index + 4}", index + 3, "world_consequence", value=f"Beat {index}") + for index in range(12) + ], + ] + + snapshot = reduce_memory_events("run-1", events) + + assert snapshot.objective == "Find the singer before air." + assert [(fact.key, fact.value) for fact in snapshot.active_facts] == [("player_role", "Publicist")] + assert snapshot.superseded_facts[0].value == "Backup dancer" + assert snapshot.superseded_facts[0].superseded_by == snapshot.active_facts[0].fact_id + assert snapshot.diagnostics.non_story_event_count == 1 + assert len(snapshot.recent_events) == 8 + assert snapshot.diagnostics.dropped_recent_event_count > 0 + + +def test_story_guide_projection_keeps_non_story_intent_out_of_active_facts() -> None: + context = StoryGuideCompressedContext( + scene_summary="Awards livestream with the singer missing.", + player_role="Publicist", + cast_or_factions=["Producer Han", "Sponsor team"], + pressure="Three minutes until air.", + constraints=["No violence"], + tone="high_drama", + open_questions=["Who last saw the singer?"], + confirmed_facts=["Awards livestream"], + rejected_or_changed_facts=["superseded player_role: Backup dancer"], + non_story_user_intents=["meta_assistant: who are you"], + last_user_intent="story_seed", + readiness_score=0.67, + planner_skill="cast_focus", + planner_job="clarify the person closest to the pressure", + ) + + snapshot = project_story_guide_memory(context, run_id="guide-1", turn_index=4) + + assert any(fact.key == "player_role" and fact.value == "Publicist" for fact in snapshot.active_facts) + assert any(fact.value == "Backup dancer" for fact in snapshot.superseded_facts) + assert all("who are you" not in fact.value for fact in snapshot.active_facts) + assert snapshot.diagnostics.non_story_event_count == 1 + assert snapshot.open_threads == ["Who last saw the singer?"] + + +def test_portable_evaluator_scores_specific_stateful_run() -> None: + events = [ + _event("objective", 0, "objective_set", value="Find the singer before air."), + _event("role", 0, "fact_asserted", key="player_role", value="Publicist"), + _event("clue", 1, "clue_unlocked", value="Green-room badge"), + ] + memory_one = reduce_memory_events("run-eval", events) + memory_two = reduce_memory_events( + "run-eval", + [ + *events, + _event("pressure", 2, "fact_asserted", key="pressure", value="Sponsor feed frozen"), + ], + ) + bundle = RpgEvaluationBundleV1( + run_id="run-eval", + system_label="Candidate runtime", + scenario=RpgEvaluationScenarioV1( + scenario_id="awards", + title="Awards livestream", + objective="Find the singer before air.", + entity_ids=["producer_han", "lena"], + ), + turns=[ + RpgTurnObservationV1( + turn_index=1, + player_action="Freeze the sponsor feed and show Lena the badge.", + world_response="Lena verifies the badge while Producer Han loses control of the room.", + options=["Question Producer Han", "Check the corridor", "Protect the witness"], + state_deltas=[ + RpgStateDeltaV1( + target="lena", + kind="increase", + label="Lena trust +1", + evidence="She verifies the badge.", + ) + ], + clue_unlocks=["Green-room badge"], + referenced_entity_ids=["producer_han", "lena"], + objective_progress=0.35, + memory=memory_one, + ), + RpgTurnObservationV1( + turn_index=2, + player_action="Use the badge log to confront Producer Han.", + world_response="The log exposes a corridor handoff and narrows the search.", + options=["Search the corridor", "Ask Lena for cover", "Call the stage manager"], + state_deltas=[ + RpgStateDeltaV1( + target="objective", + kind="increase", + label="Search narrowed", + evidence="The badge log identifies the corridor.", + ) + ], + opportunity_unlocks=["Corridor search"], + referenced_entity_ids=["producer_han", "lena"], + objective_progress=0.9, + memory=memory_two, + ), + ], + ) + + report = evaluate_rpg_bundle(bundle) + + assert report.status == "pass" + assert report.score >= 80 + assert len(report.criteria) == 8 + assert all(item.status == "pass" for item in report.criteria) + + +def test_terminal_options_do_not_overcount_choice_diversity() -> None: + memory = reduce_memory_events( + "terminal-options", + [_event("objective", 0, "objective_set", value="Resolve the gala reveal.")], + ) + bundle = RpgEvaluationBundleV1( + run_id="terminal-options", + system_label="Terminal option compatibility", + scenario=RpgEvaluationScenarioV1( + scenario_id="gala", + title="Gala", + objective="Resolve the gala reveal.", + ), + turns=[ + RpgTurnObservationV1( + turn_index=1, + player_action="Show the timestamp.", + world_response="The room accepts the evidence.", + options=["Question the sponsor", "Protect the singer"], + state_deltas=[RpgStateDeltaV1(target="pressure", kind="decrease", label="Pressure eased")], + objective_progress=0.5, + memory=memory, + ), + RpgTurnObservationV1( + turn_index=2, + player_action="Close the broadcast.", + world_response="The gala ends with the truth on record.", + options=["Replay", "Go home"], + terminal=True, + state_deltas=[RpgStateDeltaV1(target="objective", kind="increase", label="Truth recorded")], + objective_progress=1.0, + memory=memory, + ), + ], + ) + + report = evaluate_rpg_bundle(bundle) + choice = next(item for item in report.criteria if item.criterion == "choice_diversity") + + assert choice.score == 100 + assert choice.evidence == ["1 distinct action sets across 1 non-terminal turns."] + + +def test_portable_evaluator_warns_for_prose_only_repeated_choices() -> None: + memory = reduce_memory_events( + "weak-run", + [_event("objective", 0, "objective_set", value="Escape the tower.")], + ) + repeated = ["Wait", "Wait"] + bundle = RpgEvaluationBundleV1( + run_id="weak-run", + system_label="Prose-only baseline", + scenario=RpgEvaluationScenarioV1( + scenario_id="tower", + title="Tower", + objective="Escape the tower.", + ), + turns=[ + RpgTurnObservationV1( + turn_index=1, + player_action="Wait.", + world_response="The room remains tense.", + options=repeated, + objective_progress=0.0, + memory=memory, + ) + ], + ) + + report = evaluate_rpg_bundle(bundle) + by_name = {item.criterion: item for item in report.criteria} + + assert report.status != "pass" + assert by_name["consequence_visibility"].status == "fail" + assert by_name["player_agency"].status == "fail" + assert by_name["trajectory_progress"].status == "fail" diff --git a/tests/test_rpg_session_evaluation_export.py b/tests/test_rpg_session_evaluation_export.py new file mode 100644 index 00000000..49268a39 --- /dev/null +++ b/tests/test_rpg_session_evaluation_export.py @@ -0,0 +1,274 @@ +from __future__ import annotations + +import sqlite3 + +from fastapi.testclient import TestClient + +import rpg_backend.main as main_module +from rpg_backend.narrative.brief import build_story_brief +from rpg_backend.narrative.contracts import ( + AdvanceTurnRequest, + CastMember, + GameplayChip, + InventoryDelta, + NPCPulse, + PlayerGoal, + PlayerRole, + StoryGuideCompressedContext, + StoryMessage, + StoryOption, + TurnGameplayMetadata, +) +from rpg_backend.narrative.repository import NarrativeRepository +from rpg_backend.narrative.service import NarrativeService +from rpg_backend.research_runtime.evaluator import evaluate_rpg_bundle +from rpg_backend.responses_transport import ResponsesJSONResponse + + +class _TurnContractCapture: + def __init__(self) -> None: + self.calls: list[dict[str, object]] = [] + + def invoke_json( + self, + *, + system_prompt: str, + user_payload: dict[str, object], + operation_name: str, + max_output_tokens: int | None = None, + ) -> ResponsesJSONResponse: + del system_prompt, max_output_tokens + self.calls.append({"operation_name": operation_name, "user_payload": user_payload}) + return ResponsesJSONResponse( + payload={ + "passage": "Marcus freezes when the publicist names the timestamp on camera.", + "options": [ + {"label": "Ask Elena to verify the envelope"}, + {"label": "Move Marcus away from the host"}, + ], + "npc_pulse": [ + { + "npc_id": "marcus", + "state": "His denial is losing the room.", + "shift": "cornered", + "reason": "The timestamp is now public.", + } + ], + }, + response_id="turn-contract-capture", + usage={}, + input_characters=len(str(user_payload)), + ) + + +def _seed_runtime(repo: NarrativeRepository, *, owner_user_id: str = "owner") -> tuple[str, str]: + template_id = "tmpl_research_export" + session_id = "sess_research_export" + context = StoryGuideCompressedContext( + scene_summary="An awards livestream is seconds from exposing a rigged result.", + player_role="Publicist", + cast_or_factions=["Elena", "Marcus", "Dana"], + pressure="The host goes live in three minutes.", + constraints=["No violence"], + tone="high_drama", + confirmed_facts=["The trophy envelope was swapped."], + rejected_or_changed_facts=["superseded player_role: Backup dancer"], + non_story_user_intents=["meta_assistant: who are you"], + readiness_score=0.92, + ) + brief = build_story_brief( + seed=( + "At an awards livestream, Elena, Marcus, and Dana collide over a rigged trophy " + "three minutes before air. The player is the publicist. No violence." + ), + desired_tension_profile="high_drama", + ).brief + cast = [ + CastMember( + character_id="elena", + display_name="Elena", + role="award nominee", + relation_to_protagonist="client", + ), + CastMember( + character_id="marcus", + display_name="Marcus", + role="event sponsor", + relation_to_protagonist="adversary", + ), + CastMember( + character_id="dana", + display_name="Dana", + role="crisis advisor", + relation_to_protagonist="mentor", + ), + ] + repo.create_template( + template_id=template_id, + owner_user_id=owner_user_id, + seed=brief.original_seed, + title="Rigged Trophy Reveal", + cast=cast, + advisor_persona="A precise crisis editor.", + opening_passage="The host reaches for the rigged envelope.", + opening_options=[StoryOption(label="Show Elena the proof"), StoryOption(label="Confront Marcus")], + player_goals=[PlayerGoal(goal="Protect Elena before the reveal", stakes="Her career and your reputation")], + failure_conditions=[], + player_role_options=[ + PlayerRole( + role_id="publicist", + label="The Publicist", + public_persona="Elena's crisis manager.", + hidden_objective="Protect Elena without burying the truth.", + starting_assets=["Photo of the swapped envelope"], + ) + ], + visibility="private", + language="en", + story_brief=brief, + story_guide_context=context, + ) + repo.create_session( + session_id=session_id, + template_id=template_id, + player_user_id=owner_user_id, + turn_budget=8, + selected_player_role_id="publicist", + ) + repo.append_story_message( + session_id, + StoryMessage( + ord=0, + role="narrator", + content="The host reaches for the rigged envelope.", + options=[StoryOption(label="Show Elena the proof"), StoryOption(label="Confront Marcus")], + ), + ) + repo.append_story_message( + session_id, + StoryMessage(ord=1, role="player", content="Show Elena the proof before Marcus can intervene."), + ) + repo.append_story_message( + session_id, + StoryMessage( + ord=2, + role="narrator", + content=( + "Elena checks the timestamp and steps away from Marcus as Dana locks the press-room door. " + + "The live audience closes in around the disputed envelope. " * 12 + ), + options=[StoryOption(label="Question Marcus"), StoryOption(label="Secure the original envelope")], + npc_pulse=[ + NPCPulse( + npc_id="elena", + state="She trusts your warning.", + shift="warmer", + reason="You showed verifiable proof.", + ) + ], + inventory_delta=InventoryDelta( + added=["Timestamped envelope photo"], + reason="Elena verified the image.", + ), + gameplay_metadata=TurnGameplayMetadata( + state_deltas=[GameplayChip(label="Public pressure contained", tone="gain")], + clue_unlocks=[GameplayChip(label="Envelope timestamp", tone="unlock")], + opportunity_unlocks=[GameplayChip(label="Press-room interview", tone="unlock")], + ), + ), + ) + repo.touch_session(session_id, increment_turns=1) + return template_id, session_id + + +def test_create_memory_seeds_persist_and_export_into_live_session_bundle(tmp_path) -> None: + db_path = tmp_path / "runtime.sqlite3" + repo = NarrativeRepository(str(db_path)) + template_id, session_id = _seed_runtime(repo) + + brief, context = repo.get_template_research_context(template_id) + assert brief is not None + assert context is not None + assert context.player_role == "Publicist" + + bundle = NarrativeService(repository=repo, gateway=None).get_rpg_evaluation_bundle( + session_id, + player_user_id="owner", + ) + report = evaluate_rpg_bundle(bundle) + + assert bundle.schema_version == "rpg_evaluation_bundle.v1" + assert bundle.turns[0].progress_basis == "turn_budget_proxy" + assert bundle.turns[0].memory.diagnostics.non_story_event_count == 1 + assert any(fact.key == "player_role" and fact.value == "Publicist" for fact in bundle.turns[0].memory.active_facts) + assert any(fact.value == "Backup dancer" for fact in bundle.turns[0].memory.superseded_facts) + assert any(delta.target == "elena" for delta in bundle.turns[0].state_deltas) + assert "Envelope timestamp" in bundle.turns[0].clue_unlocks + assert "Press-room interview" in bundle.turns[0].opportunity_unlocks + assert report.status == "pass" + assert any("turn budget as a proxy" in item for item in report.limitations) + + +def test_malformed_research_context_downgrades_without_breaking_template(tmp_path) -> None: + db_path = tmp_path / "runtime.sqlite3" + repo = NarrativeRepository(str(db_path)) + template_id, _ = _seed_runtime(repo) + with sqlite3.connect(db_path) as connection: + connection.execute( + "UPDATE narrative_templates SET story_guide_context_json = ? WHERE template_id = ?", + ("{not-valid-json", template_id), + ) + connection.commit() + + brief, context = repo.get_template_research_context(template_id) + + assert repo.get_template(template_id).title == "Rigged Trophy Reveal" + assert brief is not None + assert context is None + + +def test_live_turn_prompt_uses_current_create_memory_without_superseded_or_chat_data(tmp_path) -> None: + repo = NarrativeRepository(str(tmp_path / "runtime.sqlite3")) + _, session_id = _seed_runtime(repo) + gateway = _TurnContractCapture() + + NarrativeService(repository=repo, gateway=gateway).advance( + session_id, + AdvanceTurnRequest(chosen_option_index=0), + player_user_id="owner", + ) + + payload = gateway.calls[-1]["user_payload"] + assert isinstance(payload, dict) + contract = payload["story_contract"] + assert isinstance(contract, dict) + assert contract["player_role"] == "Publicist" + assert contract["boundaries"] == ["No violence"] + assert "The trophy envelope was swapped." in contract["confirmed_facts"] + assert "Backup dancer" not in str(contract) + assert "meta_assistant" not in str(contract) + assert "recent_turns" not in contract + + +def test_session_export_and_portable_evaluator_are_reachable_through_authenticated_api(tmp_path) -> None: + repo = NarrativeRepository(str(tmp_path / "runtime.sqlite3")) + client = TestClient(main_module.app) + login = client.post("/auth/login", json={"username": "evaluation_owner"}) + assert login.status_code == 200 + user_id = login.json()["user"]["user_id"] + _, session_id = _seed_runtime(repo, owner_user_id=user_id) + service = NarrativeService(repository=repo, gateway=None) + original_service = main_module.narrative_service + main_module.narrative_service = service + + try: + bundle_response = client.get(f"/narrative/sessions/{session_id}/evaluation-bundle") + report_response = client.post("/research/rpg-evaluations", json=bundle_response.json()) + finally: + main_module.narrative_service = original_service + + assert bundle_response.status_code == 200 + assert bundle_response.json()["run_id"] == session_id + assert report_response.status_code == 200 + assert report_response.json()["schema_version"] == "rpg_evaluation_report.v1" + assert report_response.json()["status"] == "pass" diff --git a/tests/test_unified_live_llm_pipeline_contract.py b/tests/test_unified_live_llm_pipeline_contract.py index d6ffd58c..1b5958a1 100644 --- a/tests/test_unified_live_llm_pipeline_contract.py +++ b/tests/test_unified_live_llm_pipeline_contract.py @@ -495,6 +495,53 @@ def test_story_guide_context_tracks_superseded_facts_and_delegated_choices() -> assert "Who is the player" not in delegated.reply +def test_player_role_correction_removes_role_only_residue_from_active_cast() -> None: + initial = advance_story_guide_loop( + None, + ( + "At an awards gala, a publicist, a singer, and a sponsor discover the reveal is rigged. " + "The player is a backup dancer who must protect the singer before air." + ), + "en", + ) + corrected = advance_story_guide_loop( + initial.state, + "Correction: I am the publicist, not the backup dancer.", + "en", + ) + + assert corrected.state.context.player_role == "publicist" + assert "backup dancer" not in corrected.state.context.cast_or_factions + assert "publicist" not in corrected.state.context.cast_or_factions + assert "backup dancer" not in corrected.state.context.scene_summary.lower() + assert "backup dancer" not in corrected.state.context.pressure.lower() + assert all("backup dancer" not in item.lower() for item in corrected.state.context.constraints) + assert "singer" in corrected.state.context.cast_or_factions + assert any( + "superseded player_role: backup dancer" in fact + for fact in corrected.state.context.rejected_or_changed_facts + ) + + zh_initial = advance_story_guide_loop( + None, + "直播颁奖礼被人操纵。玩家是伴舞，必须在主持人上台前保护歌手。", + "zh", + ) + zh_state = zh_initial.state.model_copy( + update={ + "context": zh_initial.state.context.model_copy(update={"player_role": "伴舞"}) + } + ) + zh_corrected = advance_story_guide_loop( + zh_state, + "修正：我是公关，不是伴舞。", + "zh", + ) + assert zh_corrected.state.context.player_role == "公关" + assert "玩家是伴舞" not in zh_corrected.state.context.scene_summary + assert "玩家是公关" in zh_corrected.state.context.scene_summary + + def test_story_guide_compresses_rich_seed_into_specific_playable_facts() -> None: response = advance_story_guide_loop( None, diff --git a/tools/rpg_eval/portable_runtime.py b/tools/rpg_eval/portable_runtime.py new file mode 100644 index 00000000..adbf3b45 --- /dev/null +++ b/tools/rpg_eval/portable_runtime.py @@ -0,0 +1,37 @@ +from __future__ import annotations + +import argparse +import json +from pathlib import Path +from typing import Any + +from rpg_backend.research_runtime.contracts import RpgEvaluationBundleV1 +from rpg_backend.research_runtime.evaluator import evaluate_rpg_bundle + + +def evaluate_payload(payload: dict[str, Any]) -> dict[str, Any]: + bundle = RpgEvaluationBundleV1.model_validate(payload) + return evaluate_rpg_bundle(bundle).model_dump(mode="json") + + +def main() -> int: + parser = argparse.ArgumentParser( + description="Evaluate a portable RPG turn bundle without provider calls.", + ) + parser.add_argument("input", type=Path, help="Path to an rpg_evaluation_bundle.v1 JSON file.") + parser.add_argument("--out", type=Path, help="Optional report output path. Defaults to stdout.") + args = parser.parse_args() + + payload = json.loads(args.input.read_text(encoding="utf-8")) + report = evaluate_payload(payload) + rendered = json.dumps(report, ensure_ascii=False, indent=2) + "\n" + if args.out: + args.out.parent.mkdir(parents=True, exist_ok=True) + args.out.write_text(rendered, encoding="utf-8") + else: + print(rendered, end="") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tools/rpg_eval/tiny_stories_golden_path_harness.py b/tools/rpg_eval/tiny_stories_golden_path_harness.py index f7ead12f..0e415146 100644 --- a/tools/rpg_eval/tiny_stories_golden_path_harness.py +++ b/tools/rpg_eval/tiny_stories_golden_path_harness.py @@ -456,28 +456,115 @@ def run_live_golden_path( if not user_id: failures.append(_live_failure("environment", "auth", "login did not return a user id")) + initial_guide_seed = seed.replace( + "The player is the publicist", + "The player is a backup dancer", + ) guide, api_latencies["story_guide_ms"] = client.request_json( "POST", "/narrative/story-guide/turns", - payload={"message": seed, "language": "en"}, + payload={"message": initial_guide_seed, "language": "en"}, + timeout=timeout, + ) + corrected_guide, api_latencies["story_guide_correction_ms"] = client.request_json( + "POST", + "/narrative/story-guide/turns", + payload={ + "message": "Correction: I am the publicist, not the backup dancer.", + "language": "en", + "current_seed": initial_guide_seed, + "previous_assistant_reply": guide.get("reply"), + "state": guide.get("state"), + }, timeout=timeout, ) + guide_context = ((corrected_guide.get("state") or {}).get("context") or {}) stage_sources["story_guide"] = { - "source": guide.get("source"), - "status": guide.get("status"), - "canShapeBrief": guide.get("canShapeBrief"), - "reply_excerpt": str(guide.get("reply") or "")[:180], + "initial_source": guide.get("source"), + "correction_source": corrected_guide.get("source"), + "status": corrected_guide.get("status"), + "canShapeBrief": corrected_guide.get("canShapeBrief"), + "reply_excerpt": str(corrected_guide.get("reply") or "")[:180], + "player_role": guide_context.get("player_role"), + "active_cast": guide_context.get("cast_or_factions"), + "superseded_count": len(guide_context.get("rejected_or_changed_facts") or []), } - if guide.get("source") not in LIVE_ACCEPTED_SOURCE_LABELS: + if ( + guide.get("source") not in LIVE_ACCEPTED_SOURCE_LABELS + or corrected_guide.get("source") not in LIVE_ACCEPTED_SOURCE_LABELS + ): failures.append( _live_failure( "provider", "create.story_butler_turn", - "Story Butler guide response was not live-backed", - observed_source=guide.get("source"), + "One or more Story Butler guide responses were not live-backed", + initial_source=guide.get("source"), + correction_source=corrected_guide.get("source"), + ) + ) + if "publicist" not in str(guide_context.get("player_role") or "").lower(): + failures.append( + _live_failure( + "story_guide_intent", + "create.story_butler_turn", + "Correction did not replace the initial player role", + observed_role=guide_context.get("player_role"), + ) + ) + if not any( + "superseded player_role" in str(item).lower() + for item in guide_context.get("rejected_or_changed_facts") or [] + ): + failures.append( + _live_failure( + "story_guide_intent", + "create.story_butler_turn", + "Correction did not preserve superseded role evidence", + ) + ) + active_cast = { + str(item).strip().casefold() + for item in guide_context.get("cast_or_factions") or [] + } + if {"backup dancer", "publicist"}.intersection(active_cast): + failures.append( + _live_failure( + "story_guide_intent", + "create.story_butler_turn", + "Player-role identities leaked into the active NPC cast after correction", + observed_cast=sorted(active_cast), + ) + ) + active_truth = " ".join( + [ + str(guide_context.get("scene_summary") or ""), + str(guide_context.get("pressure") or ""), + *[str(item) for item in guide_context.get("constraints") or []], + *[str(item) for item in guide_context.get("confirmed_facts") or []], + ] + ).casefold() + if "backup dancer" in active_truth: + failures.append( + _live_failure( + "story_guide_intent", + "create.story_butler_turn", + "Superseded player role remained in active compressed story facts", ) ) + brief_guide_context = { + key: guide_context.get(key) + for key in ( + "scene_summary", + "player_role", + "cast_or_factions", + "pressure", + "constraints", + "tone", + "confirmed_facts", + ) + } + brief, api_latencies["story_brief_ms"] = client.request_json( "POST", "/narrative/story-briefs", @@ -485,6 +572,7 @@ def run_live_golden_path( "seed": seed, "language": "en", "desired_tension_profile": "high_drama", + "guide_context": brief_guide_context, }, timeout=timeout, ) @@ -515,6 +603,7 @@ def run_live_golden_path( "difficulty": "story", "language": "en", "story_brief": brief.get("brief"), + "story_guide_context": guide_context, }, timeout=timeout, ) @@ -639,6 +728,44 @@ def run_live_golden_path( if ending_payload is None: failures.append(_live_failure("trajectory_judge", "ending", "ending payload was not available")) + evaluation_bundle, api_latencies["evaluation_bundle_ms"] = client.request_json( + "GET", + f"/narrative/sessions/{session_id}/evaluation-bundle", + timeout=timeout, + ) + evaluation_report, api_latencies["evaluation_report_ms"] = client.request_json( + "POST", + "/research/rpg-evaluations", + payload=evaluation_bundle, + timeout=timeout, + ) + latest_memory = ((evaluation_bundle.get("turns") or [{}])[-1].get("memory") or {}) + stage_sources["portable_evaluation"] = { + "bundle_schema": evaluation_bundle.get("schema_version"), + "report_schema": evaluation_report.get("schema_version"), + "status": evaluation_report.get("status"), + "score": evaluation_report.get("score"), + "progress_basis": ((evaluation_bundle.get("turns") or [{}])[-1].get("progress_basis")), + "active_fact_count": len(latest_memory.get("active_facts") or []), + "superseded_fact_count": len(latest_memory.get("superseded_facts") or []), + } + if evaluation_bundle.get("schema_version") != "rpg_evaluation_bundle.v1": + failures.append(_live_failure("schema", "research.rpg_evaluation", "portable bundle schema missing")) + if evaluation_report.get("schema_version") != "rpg_evaluation_report.v1": + failures.append(_live_failure("schema", "research.rpg_evaluation", "portable report schema missing")) + if not any( + "backup dancer" in str(fact.get("value") or "").lower() + for fact in latest_memory.get("superseded_facts") or [] + if isinstance(fact, dict) + ): + failures.append( + _live_failure( + "artifact", + "research.rpg_evaluation", + "Create correction was not retained in the 12-turn evaluation memory", + ) + ) + session_event_response, api_latencies["llm_events_ms"] = client.request_json( "GET", f"/narrative/sessions/{session_id}/llm-events", @@ -754,6 +881,135 @@ def run_live_golden_path( return payload +def resume_live_golden_path_evidence( + *, + base_url: str, + artifact: Path, + report: Path, + runtime_db: Path, + username: str = "portfolio_reviewer", + timeout: float = 180.0, +) -> dict[str, Any]: + """Finish deterministic/reviewer evidence for an already completed live run. + + This recovery path never submits a Play turn or invokes an LLM. It is for + evaluator/telemetry failures that happen after the live trajectory has + already reached its persisted ending. + """ + + payload = json.loads(artifact.read_text()) + session_id = str(payload.get("session_id") or "") + if not session_id or int(payload.get("completed_turns") or 0) != int(payload.get("turn_budget") or 0): + raise ValueError("Resume requires an artifact with a complete persisted trajectory.") + + client = LiveHarnessClient(base_url, timeout=timeout) + failures = [ + failure + for failure in payload.get("failures") or [] + if str(failure.get("stage") or "") != "/research/rpg-evaluations" + ] + api_latencies = dict(payload.get("api_latencies_ms") or {}) + stage_sources = dict(payload.get("stage_sources") or {}) + + health, api_latencies["resume_health_ms"] = client.request_json("GET", "/health", timeout=30) + failures.extend(_health_failures(health)) + login, api_latencies["resume_login_ms"] = client.request_json( + "POST", "/auth/login", payload={"username": username}, timeout=30, + ) + user = login.get("user") if isinstance(login.get("user"), dict) else {} + user_id = str(user.get("user_id") or "") + story, api_latencies["resume_story_ms"] = client.request_json( + "GET", f"/narrative/sessions/{session_id}/story", query={"agent_trace": True}, timeout=timeout, + ) + ending, api_latencies["resume_ending_ms"] = client.request_json( + "GET", f"/narrative/sessions/{session_id}/ending", timeout=timeout, + ) + bundle, api_latencies["resume_evaluation_bundle_ms"] = client.request_json( + "GET", f"/narrative/sessions/{session_id}/evaluation-bundle", timeout=timeout, + ) + evaluation, api_latencies["resume_evaluation_report_ms"] = client.request_json( + "POST", "/research/rpg-evaluations", payload=bundle, timeout=timeout, + ) + event_response, api_latencies["resume_llm_events_ms"] = client.request_json( + "GET", f"/narrative/sessions/{session_id}/llm-events", timeout=timeout, + ) + session_events = [item for item in event_response.get("items") or [] if isinstance(item, dict)] + all_events = _load_live_events_from_runtime_db( + runtime_db, + user_id=user_id, + started_at=str(payload.get("run_started_at") or payload.get("generated_at") or ""), + ) + failures.extend(_session_event_failures(session_events)) + failures.extend( + _telemetry_failures( + events=all_events, + session_id=session_id, + turn_budget=int(payload.get("turn_budget") or GOLDEN_PATH_TURN_BUDGET), + ) + ) + + latest_memory = ((bundle.get("turns") or [{}])[-1].get("memory") or {}) + stage_sources["portable_evaluation"] = { + "bundle_schema": bundle.get("schema_version"), + "report_schema": evaluation.get("schema_version"), + "status": evaluation.get("status"), + "score": evaluation.get("score"), + "progress_basis": ((bundle.get("turns") or [{}])[-1].get("progress_basis")), + "active_fact_count": len(latest_memory.get("active_facts") or []), + "superseded_fact_count": len(latest_memory.get("superseded_facts") or []), + } + if bundle.get("schema_version") != "rpg_evaluation_bundle.v1" or evaluation.get("schema_version") != "rpg_evaluation_report.v1": + failures.append(_live_failure("schema", "research.rpg_evaluation", "portable evaluation schema missing")) + if not any( + "backup dancer" in str(fact.get("value") or "").lower() + for fact in latest_memory.get("superseded_facts") or [] + if isinstance(fact, dict) + ): + failures.append(_live_failure("artifact", "research.rpg_evaluation", "Create correction missing from evaluation memory")) + + agent_events = [event for event in story.get("agent_events") or [] if isinstance(event, dict)] + quality = _quality_summary( + turn_summaries=list(payload.get("turns") or []), + agent_events=agent_events, + story=story, + ending=ending, + seed=str(payload.get("seed_excerpt") or ""), + ) + if quality["status"] == "fail": + failures.append(_live_failure("trajectory_judge", "quality_fun_gate", "deterministic quality gate failed")) + + payload.update({ + "status": "fail" if failures else "pass", + "base_url": base_url, + "api_latencies_ms": api_latencies, + "stage_sources": stage_sources, + "turn_telemetry": _turn_telemetry_summaries(all_events, session_id), + "operation_events": { + operation: [_event_summary(event) for event in events] + for operation, events in sorted(_events_by_operation(all_events).items()) + if operation in GOLDEN_PATH_REQUIRED_OPERATIONS + or operation in {"narrative.ending", "narrative.highlights", "narrative.branches"} + }, + "session_llm_events": [_event_summary(event) for event in session_events], + "quality_fun_gate": quality, + "ending": ending, + "reviewer_evidence": { + "agent_event_count": len(agent_events), + "step_judge_count": len(_agent_event_payloads(agent_events, "step_judge")), + "contract_judge_count": len(_agent_event_payloads(agent_events, "contract_judge")), + "session_llm_event_count": len(session_events), + }, + "runtime_db": str(runtime_db), + "failures": failures, + }) + payload.setdefault("notes", []).append( + "Post-run evaluator and telemetry evidence was resumed without additional LLM or Play calls." + ) + _write_json(artifact, payload) + _write_markdown_report(report, payload) + return payload + + def main() -> int: parser = argparse.ArgumentParser(description="Run the Tiny Stories strict 12-turn live golden path gate.") parser.add_argument("--base-url", default="http://127.0.0.1:8350", help="Backend base URL.") @@ -764,6 +1020,11 @@ def main() -> int: parser.add_argument("--turn-budget", type=int, default=GOLDEN_PATH_TURN_BUDGET, help="Expected live turn count.") parser.add_argument("--output", default=str(DEFAULT_OUTPUT), help="Summary JSON path.") parser.add_argument("--report", default=str(DEFAULT_REPORT), help="Markdown report path.") + parser.add_argument( + "--resume-existing", + action="store_true", + help="Resume evaluator/telemetry evidence from --output without submitting more live turns.", + ) args = parser.parse_args() runtime_db = Path(args.runtime_db) if args.runtime_db else None @@ -772,16 +1033,28 @@ def main() -> int: runtime_db_env = os.environ.get("APP_RUNTIME_STATE_DB_PATH") runtime_db = Path(runtime_db_env) if runtime_db_env else None - payload = run_live_golden_path( - base_url=args.base_url, - output=Path(args.output), - report=Path(args.report), - username=args.username, - seed=args.seed, - runtime_db=runtime_db, - timeout=args.timeout, - turn_budget=args.turn_budget, - ) + if args.resume_existing: + if runtime_db is None: + parser.error("--resume-existing requires --runtime-db or APP_RUNTIME_STATE_DB_PATH") + payload = resume_live_golden_path_evidence( + base_url=args.base_url, + artifact=Path(args.output), + report=Path(args.report), + runtime_db=runtime_db, + username=args.username, + timeout=args.timeout, + ) + else: + payload = run_live_golden_path( + base_url=args.base_url, + output=Path(args.output), + report=Path(args.report), + username=args.username, + seed=args.seed, + runtime_db=runtime_db, + timeout=args.timeout, + turn_budget=args.turn_budget, + ) print(json.dumps({ "status": payload["status"], "mode": payload["mode"], diff --git a/tools/rpg_eval/tiny_stories_reliability_harness.py b/tools/rpg_eval/tiny_stories_reliability_harness.py index 0d0b8863..6cc539e8 100644 --- a/tools/rpg_eval/tiny_stories_reliability_harness.py +++ b/tools/rpg_eval/tiny_stories_reliability_harness.py @@ -43,6 +43,10 @@ "At an awards gala, a publicist, a singer, and a sponsor discover the live trophy reveal is rigged. " "The player is the publicist who must protect the singer before the host walks onstage. No gore." ) +LIVE_ACCEPTANCE_SEED_ZH = ( + "在一场直播颁奖礼上，公关、歌手和赞助商发现奖杯揭晓被人操纵。" + "玩家是公关，必须在主持人上台前保护歌手。不要出现血腥暴力。" +) REQUIRED_CASES = { "arbitrary_input_smalltalk", "meta_help_input", @@ -423,6 +427,7 @@ def run_live_acceptance( seed: str = LIVE_ACCEPTANCE_SEED, runtime_db: Path | None = None, timeout: float = 150.0, + language: str = "en", ) -> dict[str, Any]: run_started_at = _utc_now_iso() client = LiveHarnessClient(base_url, timeout=timeout) @@ -450,35 +455,131 @@ def run_live_acceptance( if not user_id: failures.append(_live_failure("environment", "auth", "login did not return a user id")) + if language == "zh": + initial_guide_seed = seed.replace("玩家是公关", "玩家是伴舞") + correction_message = "修正：我是公关，不是伴舞。" + else: + initial_guide_seed = seed.replace( + "The player is the publicist", + "The player is a backup dancer", + ) + correction_message = "Correction: I am the publicist, not the backup dancer." guide, api_latencies["story_guide_ms"] = client.request_json( "POST", "/narrative/story-guide/turns", - payload={"message": seed, "language": "en"}, + payload={"message": initial_guide_seed, "language": language}, + timeout=timeout, + ) + corrected_guide, api_latencies["story_guide_correction_ms"] = client.request_json( + "POST", + "/narrative/story-guide/turns", + payload={ + "message": correction_message, + "language": language, + "current_seed": initial_guide_seed, + "previous_assistant_reply": guide.get("reply"), + "state": guide.get("state"), + }, timeout=timeout, ) + guide_context = ((corrected_guide.get("state") or {}).get("context") or {}) stage_sources["story_guide"] = { - "source": guide.get("source"), - "status": guide.get("status"), - "canShapeBrief": guide.get("canShapeBrief"), - "reply_excerpt": str(guide.get("reply") or "")[:180], + "initial_source": guide.get("source"), + "correction_source": corrected_guide.get("source"), + "status": corrected_guide.get("status"), + "canShapeBrief": corrected_guide.get("canShapeBrief"), + "reply_excerpt": str(corrected_guide.get("reply") or "")[:180], + "player_role": guide_context.get("player_role"), + "active_cast": guide_context.get("cast_or_factions"), + "superseded_count": len(guide_context.get("rejected_or_changed_facts") or []), } - if guide.get("source") not in LIVE_ACCEPTED_SOURCE_LABELS: + if ( + guide.get("source") not in LIVE_ACCEPTED_SOURCE_LABELS + or corrected_guide.get("source") not in LIVE_ACCEPTED_SOURCE_LABELS + ): failures.append( _live_failure( "provider", "create.story_butler_turn", - "Story Butler guide response was not live-backed", - observed_source=guide.get("source"), + "One or more Story Butler guide responses were not live-backed", + initial_source=guide.get("source"), + correction_source=corrected_guide.get("source"), + ) + ) + expected_role = "公关" if language == "zh" else "publicist" + superseded_role = "伴舞" if language == "zh" else "backup dancer" + if expected_role not in str(guide_context.get("player_role") or "").lower(): + failures.append( + _live_failure( + "story_guide_intent", + "create.story_butler_turn", + "Correction did not replace the initial player role", + observed_role=guide_context.get("player_role"), + ) + ) + if not any( + "superseded player_role" in str(item).lower() + and superseded_role in str(item).lower() + for item in guide_context.get("rejected_or_changed_facts") or [] + ): + failures.append( + _live_failure( + "story_guide_intent", + "create.story_butler_turn", + "Correction did not preserve the superseded role as audit evidence", + ) + ) + active_cast = { + str(item).strip().casefold() + for item in guide_context.get("cast_or_factions") or [] + } + if {superseded_role, expected_role}.intersection(active_cast): + failures.append( + _live_failure( + "story_guide_intent", + "create.story_butler_turn", + "Player-role identities leaked into the active NPC cast after correction", + observed_cast=sorted(active_cast), ) ) + active_truth = " ".join( + [ + str(guide_context.get("scene_summary") or ""), + str(guide_context.get("pressure") or ""), + *[str(item) for item in guide_context.get("constraints") or []], + *[str(item) for item in guide_context.get("confirmed_facts") or []], + ] + ).casefold() + if superseded_role in active_truth: + failures.append( + _live_failure( + "story_guide_intent", + "create.story_butler_turn", + "Superseded player role remained in active compressed story facts", + ) + ) + + brief_guide_context = { + key: guide_context.get(key) + for key in ( + "scene_summary", + "player_role", + "cast_or_factions", + "pressure", + "constraints", + "tone", + "confirmed_facts", + ) + } brief, api_latencies["story_brief_ms"] = client.request_json( "POST", "/narrative/story-briefs", payload={ "seed": seed, - "language": "en", + "language": language, "desired_tension_profile": "high_drama", + "guide_context": brief_guide_context, }, timeout=timeout, ) @@ -507,8 +608,9 @@ def run_live_acceptance( "visibility": "private", "turn_budget": 8, "difficulty": "gauntlet", - "language": "en", + "language": language, "story_brief": brief.get("brief"), + "story_guide_context": guide_context, }, timeout=timeout, ) @@ -558,6 +660,48 @@ def run_live_acceptance( ) ) + evaluation_bundle, api_latencies["evaluation_bundle_ms"] = client.request_json( + "GET", + f"/narrative/sessions/{session_id}/evaluation-bundle", + timeout=timeout, + ) + evaluation_report, api_latencies["evaluation_report_ms"] = client.request_json( + "POST", + "/research/rpg-evaluations", + payload=evaluation_bundle, + timeout=timeout, + ) + latest_memory = ((evaluation_bundle.get("turns") or [{}])[-1].get("memory") or {}) + stage_sources["portable_evaluation"] = { + "bundle_schema": evaluation_bundle.get("schema_version"), + "report_schema": evaluation_report.get("schema_version"), + "status": evaluation_report.get("status"), + "score": evaluation_report.get("score"), + "progress_basis": ((evaluation_bundle.get("turns") or [{}])[-1].get("progress_basis")), + "active_fact_count": len(latest_memory.get("active_facts") or []), + "superseded_fact_count": len(latest_memory.get("superseded_facts") or []), + } + if evaluation_bundle.get("schema_version") != "rpg_evaluation_bundle.v1": + failures.append( + _live_failure("schema", "research.rpg_evaluation", "Portable evaluation bundle schema was missing") + ) + if evaluation_report.get("schema_version") != "rpg_evaluation_report.v1": + failures.append( + _live_failure("schema", "research.rpg_evaluation", "Portable evaluation report schema was missing") + ) + if not any( + superseded_role in str(fact.get("value") or "").lower() + for fact in latest_memory.get("superseded_facts") or [] + if isinstance(fact, dict) + ): + failures.append( + _live_failure( + "artifact", + "research.rpg_evaluation", + "Create-time correction was not preserved in the Play evaluation memory", + ) + ) + session_event_response, api_latencies["llm_events_ms"] = client.request_json( "GET", f"/narrative/sessions/{session_id}/llm-events", @@ -613,6 +757,7 @@ def run_live_acceptance( "status": "fail" if failures else "pass", "base_url": base_url, "username": username, + "language": language, "seed_excerpt": seed[:220], "template_id": template_id, "session_id": session_id, @@ -645,6 +790,7 @@ def main() -> int: parser.add_argument("--base-url", default="http://127.0.0.1:8350", help="Backend base URL for live_acceptance.") parser.add_argument("--username", default="portfolio_reviewer", help="Reviewer username for live_acceptance.") parser.add_argument("--seed", default=LIVE_ACCEPTANCE_SEED, help="Story seed for live_acceptance.") + parser.add_argument("--language", choices=("en", "zh"), default="en", help="Live acceptance language.") parser.add_argument( "--runtime-db", default=None, @@ -664,13 +810,17 @@ def main() -> int: runtime_db_env = os.environ.get("APP_RUNTIME_STATE_DB_PATH") runtime_db = Path(runtime_db_env) if runtime_db_env else None + live_seed = args.seed + if args.language == "zh" and live_seed == LIVE_ACCEPTANCE_SEED: + live_seed = LIVE_ACCEPTANCE_SEED_ZH payload = run_live_acceptance( base_url=args.base_url, output=output, username=args.username, - seed=args.seed, + seed=live_seed, runtime_db=runtime_db, timeout=args.timeout, + language=args.language, ) else: payload = run_protocol_contract(output)