diff --git a/AGENTS.md b/AGENTS.md index 36eae16a..19650917 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -9,3 +9,40 @@ - Utiliser `supermemory_add_memory` avec `action: "save"`. Ne PAS garder le résumé uniquement dans le contexte de session — il doit être persistant dans supermemory pour les sessions suivantes. + + + + +## Gw2Analytics + +Plateforme d'analyse de combats WvW. Les contrats et conventions détaillés vivent dans `README.md`, `CONTRIBUTING.md` et `docs/`; la planification BMAD vit dans `_bmad-output/`. + +## Policy + +- Ne jamais pousser directement sur `main`; passer par une PR, conserver un historique linéaire et squash-merger. +- Signer chaque commit avec le trailer DCO `Signed-off-by:`. +- Ne jamais créer de commit sans demande explicite du mainteneur. +- Ne jamais committer les logs EVTC ou exports Elite Insights privés; ne versionner que leur manifeste et leurs empreintes. + +## Where things are + +- Comparaison EI canonique: `libs/gw2_analytics/src/gw2_analytics/ei_compare.py`; pilotes et corpus local: `scripts/ei-parity/`. +- Décisions d'architecture acceptées: `docs/adr/`; contrat BMAD actif: `_bmad-output/specs/`. + +## Running and verifying + +- Exécuter les outils Python via `uv run`; une invocation Python nue contourne l'environnement du workspace. +- Itérer avec les tests ciblés; la suite Python complète impose une couverture globale de 90 % et peut nécessiter les services Docker pour les tests d'intégration. + +## Conventions that differ from defaults + +- `libs/gw2_core` est l'unique contrat partagé et reste sans I/O; le frontend consomme OpenAPI, jamais les structures EVTC ou ORM. +- Dans l'API, respecter `routes -> services -> repositories -> ORM`; les repositories ne commitent jamais, les services possèdent les transactions. + +## Known pitfalls + +- Comparer chaque entrée joueur EI à sa fenêtre `firstAware`/`lastAware`, pas aux totaux du combat entier; un même compte peut avoir plusieurs slices contiguës. +- Résoudre propriétaires, changements de personnage et identifiants d'agent avec le temps; une table globale `instance_id -> owner` produit de fausses attributions. +- Distinguer canal arcdps et classification EI pour les dégâts d'altération; les effets de vol de vie ne sont pas des altérations EI. + + diff --git a/docs/ei-parity-workbench.md b/docs/ei-parity-workbench.md index 9ff85009..048ad6e2 100644 --- a/docs/ei-parity-workbench.md +++ b/docs/ei-parity-workbench.md @@ -95,6 +95,37 @@ intentional parser result change: uv run python scripts/ei-parity/ei_diff.py --json scripts/ei-parity/corpus-baseline.json ``` +### Detailed local report and known-delta registry + +`compare_elite_insights` also emits one atomic result per comparison — key, +status (`PASS`/`FAIL`), expected, actual, numeric `delta = actual - expected` +(otherwise `null`), rule and dimensions — alongside the unchanged +`matches`/`compared`/`differences` contract. `--report-json` requires the full +corpus and writes a **local, private** report that never lands in Git: + +```bash +uv run python scripts/ei-parity/ei_diff.py \ + --report-json /tmp/ei-certification-report.json +``` + +The report root carries `schema_version` (1), the EI reference, the manifest +SHA-256, a summary by status and by status/bucket, and a deterministically +sorted result list. Player/slice identifiers appear only when available and +are never copied into `corpus-baseline.json`, which stays aggregate. + +`scripts/ei-parity/known-deltas.json` is the versioned, initially-empty +registry of exact, bounded exemptions (schema v1). A rule flips a `FAIL` to +`KNOWN_DELTA` only when every selector field (stem, account, slice, bucket, +skill id, buff id, full key) matches exactly and the delta is within the rule's +`max_abs_delta` bound. Rules never touch a `PASS` row, and when several rules +match one row the first listed rule wins. Fields are fixed rules: `id`, +`selector`, `constraint`, `reason`, `remove_when`; the registry rejects unknown +keys, empty selectors or constraints, wrong schema versions, and malformed +rules (empty or duplicate ids, null selector values, non-positive or +non-finite bounds). No regex ever masks a result, and the current 133 +differences stay `FAIL` with the empty registry. Use the registry to track a +confirmed parity gap with a removal condition — never to silence a row. + Supporting probes in `scripts/ei-parity/`: | Script | Answers | diff --git a/libs/gw2_analytics/src/gw2_analytics/ei_compare.py b/libs/gw2_analytics/src/gw2_analytics/ei_compare.py index 7b690b68..f7731324 100644 --- a/libs/gw2_analytics/src/gw2_analytics/ei_compare.py +++ b/libs/gw2_analytics/src/gw2_analytics/ei_compare.py @@ -2,6 +2,7 @@ from __future__ import annotations +import math from collections import Counter, defaultdict from collections.abc import Sequence from typing import Any @@ -138,23 +139,65 @@ ) +def _atomic_result( + key: str, + matched: bool, + expected: object, + actual: object, + rule: str, + dimensions: dict[str, object] | None, +) -> dict[str, object]: + delta: object = None + if ( + isinstance(expected, (int, float)) + and isinstance(actual, (int, float)) + and not isinstance(expected, bool) + and not isinstance(actual, bool) + ): + computed = actual - expected + if math.isfinite(computed): + delta = computed + return { + "key": key, + "status": "PASS" if matched else "FAIL", + "expected": expected, + "actual": actual, + "delta": delta, + "rule": rule, + "dimensions": dict(dimensions or {}), + } + + def _compare_fields( prefix: str, expected: dict[str, Any], actual: dict[str, Any], differences: dict[str, object], fields: Sequence[str] | None = None, + *, + results: list[dict[str, object]] | None = None, + rule: str = "field-equal", + dimensions: dict[str, object] | None = None, ) -> None: for field in fields or tuple(actual): - if field in expected and expected[field] != actual.get(field): - differences[f"{prefix}.{field}"] = { - "expected": expected[field], - "actual": actual.get(field), - } + if field in expected: + exp = expected[field] + act = actual.get(field) + if exp != act: + differences[f"{prefix}.{field}"] = {"expected": exp, "actual": act} + if results is not None: + results.append( + _atomic_result(f"{prefix}.{field}", exp == act, exp, act, rule, dimensions) + ) def _rotation_unmatched( - expected: Sequence[tuple[int, int, int]], actual: Sequence[tuple[int, int, int]] + expected: Sequence[tuple[int, int, int]], + actual: Sequence[tuple[int, int, int]], + *, + prefix: str, + results: list[dict[str, object]], + dimensions: dict[str, object] | None = None, ) -> tuple[list[tuple[int, int, int]], list[tuple[int, int, int]]]: remaining = list(actual) missing: list[tuple[int, int, int]] = [] @@ -173,9 +216,41 @@ def _rotation_unmatched( ) if match is None: missing.append((skill_id, cast_time, duration)) + results.append( + _atomic_result( + f"{prefix}.rotation[castTime={cast_time}][duration={duration}][skillID={skill_id}]", + False, + (skill_id, cast_time, duration), + None, + "rotation-cast-match", + {**(dimensions or {}), "skill_id": skill_id}, + ) + ) else: - remaining.pop(match) - return missing, remaining + matched = remaining.pop(match) + results.append( + _atomic_result( + f"{prefix}.rotation[castTime={cast_time}][duration={duration}][skillID={skill_id}]", + True, + (skill_id, cast_time, duration), + matched, + "rotation-cast-match", + {**(dimensions or {}), "skill_id": skill_id}, + ) + ) + extra = remaining + for skill_id, cast_time, duration in extra: + results.append( + _atomic_result( + f"{prefix}.rotation[castTime={cast_time}][duration={duration}][skillID={skill_id}]", + False, + None, + (skill_id, cast_time, duration), + "rotation-cast-match", + {**(dimensions or {}), "skill_id": skill_id}, + ) + ) + return missing, extra def _defense_landed_hit(event: DamageEvent) -> bool: @@ -412,12 +487,29 @@ def compare_elite_insights( # noqa: PLR0912, PLR0915 "eiEncounterID": fight.ei_encounter_id, } differences: dict[str, object] = {} - _compare_fields("", expected, actual, differences) + results: list[dict[str, object]] = [] + _compare_fields( + "", + expected, + actual, + differences, + fields=tuple(field for field in tuple(actual) if field != "players"), + results=results, + rule="header-field", + ) differences = {key.removeprefix("."): value for key, value in differences.items()} + for result in results: + key = result.get("key") + result["key"] = key.removeprefix(".") if isinstance(key, str) else key expected_players = expected.get("players") if not isinstance(expected_players, list): - return {"matches": not differences, "compared": actual, "differences": differences} + return { + "matches": not differences, + "compared": actual, + "differences": differences, + "results": results, + } event_list = list(events) origin = ( @@ -684,6 +776,16 @@ def player_agent_ids(agent: Agent) -> set[int]: if agent is None: compared_players[account] = None differences[f"players[{account}]"] = {"expected": "present", "actual": None} + results.append( + _atomic_result( + f"players[{account}]", + False, + "present", + None, + "player-present", + {"account": account, "slice": player.get("firstAware")}, + ) + ) continue slice_lo, slice_hi = _slice_bounds(player, origin) # Several EI entries can share an account; key the report by the @@ -693,6 +795,7 @@ def player_agent_ids(agent: Agent) -> set[int]: if account_entry_count[account] < 2 else f"players[{account}@{player.get('firstAware')}]" ) + player_dims: dict[str, object] = {"account": account, "slice": player.get("firstAware")} anonymous = agent.account_name is None agent_ids = agent_ids_by_instance.get(agent.instance_id, {agent.id}) @@ -720,7 +823,15 @@ def in_slice(event: Event, _lo: int = slice_lo, _hi: int = slice_hi) -> bool: agent.team_id, ), } - _compare_fields(prefix, player, values, differences) + _compare_fields( + prefix, + player, + values, + differences, + results=results, + rule="player-field", + dimensions=player_dims, + ) # v0.17.0: exclude self-inflicted damage (src == dst) from the # DEALT side. A self-condition tick (e.g. skill 19426 on # wvw-large-fight) is recorded by arcdps with src == dst == the @@ -758,7 +869,15 @@ def in_slice(event: Event, _lo: int = slice_lo, _hi: int = slice_hi) -> bool: "powerDamage": damage - condition, } values["dpsAll"] = dps_values - _compare_fields(f"{prefix}.dpsAll", dps_all[0], dps_values, differences) + _compare_fields( + f"{prefix}.dpsAll", + dps_all[0], + dps_values, + differences, + results=results, + rule="dpsAll-field", + dimensions=player_dims, + ) defenses = player.get("defenses") if isinstance(defenses, list) and defenses and isinstance(defenses[0], dict): @@ -819,7 +938,15 @@ def in_slice(event: Event, _lo: int = slice_lo, _hi: int = slice_hi) -> bool: ), } values["defenses"] = defense_values - _compare_fields(f"{prefix}.defenses", defenses[0], defense_values, differences) + _compare_fields( + f"{prefix}.defenses", + defenses[0], + defense_values, + differences, + results=results, + rule="defenses-field", + dimensions=player_dims, + ) stats_all = player.get("statsAll") if isinstance(stats_all, list) and stats_all and isinstance(stats_all[0], dict): @@ -834,7 +961,14 @@ def in_slice(event: Event, _lo: int = slice_lo, _hi: int = slice_hi) -> bool: ) values["statsAll"] = stats_values _compare_fields( - prefix + ".statsAll", stats_all[0], stats_values, differences, _STAT_FIELDS + prefix + ".statsAll", + stats_all[0], + stats_values, + differences, + _STAT_FIELDS, + results=results, + rule="statsAll-field", + dimensions=player_dims, ) expected_buffs = player.get("buffUptimes") @@ -880,15 +1014,26 @@ def in_slice(event: Event, _lo: int = slice_lo, _hi: int = slice_hi) -> bool: expected_uptime = account_expected.get(buff_id) if (account, buff_id) in reported_buff_diffs: continue - if ( - expected_uptime is None - or abs(expected_uptime - compared_buffs[buff_id]) > 0.005 - ): - reported_buff_diffs.add((account, buff_id)) + reported_buff_diffs.add((account, buff_id)) + matched = ( + expected_uptime is not None + and abs(expected_uptime - compared_buffs[buff_id]) <= 0.005 + ) + if not matched: differences[f"{prefix}.buffUptimes[{buff_id}].uptime"] = { "expected": expected_uptime, "actual": compared_buffs[buff_id], } + results.append( + _atomic_result( + f"{prefix}.buffUptimes[{buff_id}].uptime", + matched, + expected_uptime, + compared_buffs[buff_id], + "buff-uptime-tolerance", + {**player_dims, "buff_id": buff_id}, + ) + ) values["buffUptimes"] = compared_buffs expected_rotation = player.get("rotation") @@ -930,7 +1075,13 @@ def in_slice(event: Event, _lo: int = slice_lo, _hi: int = slice_hi) -> bool: key=lambda item: (item[1], item[0]), ) values["rotation"] = actual_casts - missing_casts, extra_casts = _rotation_unmatched(expected_casts, actual_casts) + missing_casts, extra_casts = _rotation_unmatched( + expected_casts, + actual_casts, + prefix=prefix, + results=results, + dimensions=player_dims, + ) if missing_casts or extra_casts: differences[f"{prefix}.rotation"] = { "expected": missing_casts, @@ -964,6 +1115,27 @@ def in_slice(event: Event, _lo: int = slice_lo, _hi: int = slice_hi) -> bool: "expected": expected_consumables, "actual": actual_consumables, } + results.append( + _atomic_result( + f"{prefix}.consumables", + False, + expected_consumables, + actual_consumables, + "consumables-match", + player_dims, + ) + ) + else: + results.append( + _atomic_result( + f"{prefix}.consumables", + True, + expected_consumables, + actual_consumables, + "consumables-match", + player_dims, + ) + ) dps_targets = player.get("dpsTargets") stats_targets = player.get("statsTargets") @@ -1061,6 +1233,12 @@ def in_slice(event: Event, _lo: int = slice_lo, _hi: int = slice_hi) -> bool: dps_targets[index][0], target_values, differences, + results=results, + rule="dpsTargets-field", + dimensions={ + **player_dims, + "target_instance_id": target["instanceID"], + }, ) if ( isinstance(stats_targets, list) @@ -1073,6 +1251,12 @@ def in_slice(event: Event, _lo: int = slice_lo, _hi: int = slice_hi) -> bool: target_stats, differences, _STAT_FIELDS, + results=results, + rule="statsTargets-field", + dimensions={ + **player_dims, + "target_instance_id": target["instanceID"], + }, ) values["targets"] = compared_targets @@ -1095,12 +1279,20 @@ def in_slice(event: Event, _lo: int = slice_lo, _hi: int = slice_hi) -> bool: ), differences, ("totalDamage", "connectedHits", "crit", "critDamage"), + results=results, + rule="totalDamageDist-field", + dimensions={**player_dims, "skill_id": skill_id}, ) compared_players[account] = values actual["players"] = compared_players - return {"matches": not differences, "compared": actual, "differences": differences} + return { + "matches": not differences, + "compared": actual, + "differences": differences, + "results": results, + } __all__ = ["compare_elite_insights"] diff --git a/libs/gw2_analytics/tests/test_ei_compare.py b/libs/gw2_analytics/tests/test_ei_compare.py index 5aec6b10..539d1ee1 100644 --- a/libs/gw2_analytics/tests/test_ei_compare.py +++ b/libs/gw2_analytics/tests/test_ei_compare.py @@ -1,6 +1,6 @@ from __future__ import annotations -from typing import Any +from typing import Any, cast from gw2_analytics.ei_compare import _skill_stats, compare_elite_insights from gw2_core import ( @@ -13,9 +13,14 @@ EvtcHeader, Fight, Profession, + Skill, ) +def _rows(result: dict[str, object]) -> dict[str, dict[str, Any]]: + return {str(row["key"]): row for row in cast("list[dict[str, Any]]", result["results"])} + + def test_compare_elite_insights_keeps_first_anonymous_agent_for_shared_instance() -> None: fight = Fight( id="fight", @@ -404,3 +409,233 @@ def test_compare_elite_insights_excludes_breakbar_from_damage_taken_count() -> N result = compare_elite_insights(fight, expected, [_dmg(10, True)]) assert result["differences"] == {} + + +def test_compare_elite_insights_emits_atomic_results_for_players_and_header() -> None: + fight = Fight( + id="fight", + header=EvtcHeader(build_version="20260224", agent_count=1, duration_ms=10_000), + agents=[ + Agent( + id=1, + name="Player", + profession=Profession.MESMER, + elite=EliteSpec.CHRONOMANCER, + is_player=True, + account_name=":Player.1234", + subgroup="1", + instance_id=1111, + ) + ], + ) + expected: dict[str, Any] = { + "players": [ + { + "account": "Player.1234", + "instanceID": 1111, + "firstAware": 0, + "name": "Player", + "group": 1, + } + ] + } + + result = compare_elite_insights(fight, expected, []) + + assert result["differences"] == {} + rows = _rows(result) + player = rows["players[Player.1234].name"] + assert player["status"] == "PASS" + assert player["rule"] == "player-field" + assert player["dimensions"] == {"account": "Player.1234", "slice": 0} + assert player["delta"] is None + + +def test_compare_elite_insights_reports_fail_with_numeric_delta() -> None: + fight = Fight( + id="fight", + header=EvtcHeader(build_version="20260224", agent_count=1, duration_ms=10_000), + agents=[ + Agent( + id=1, + name="Player", + profession=Profession.MESMER, + elite=EliteSpec.CHRONOMANCER, + is_player=True, + account_name=":Player.1234", + instance_id=1111, + ) + ], + ) + expected: dict[str, Any] = { + "players": [ + { + "account": "Player.1234", + "instanceID": 1111, + "firstAware": 0, + "name": "Wrong Name", + "group": 99, + } + ] + } + + result = compare_elite_insights(fight, expected, []) + + assert len(cast("list[dict[str, object]]", result["differences"])) == 2 + rows = _rows(result) + name = rows["players[Player.1234].name"] + assert name["status"] == "FAIL" + assert name["expected"] == "Wrong Name" + assert name["actual"] == "Player" + assert name["delta"] is None + group = rows["players[Player.1234].group"] + assert group["status"] == "FAIL" + assert group["delta"] == 0 - 99 + + +def test_compare_elite_insights_emits_player_present_result_when_agent_missing() -> None: + fight = Fight(id="fight", agents=[]) + expected: dict[str, Any] = { + "players": [{"account": "Ghost.1234", "instanceID": 1111, "name": "Ghost"}] + } + + result = compare_elite_insights(fight, expected, []) + + rows = _rows(result) + row = rows["players[Ghost.1234]"] + assert row["status"] == "FAIL" + assert row["rule"] == "player-present" + assert row["dimensions"] == {"account": "Ghost.1234", "slice": None} + + +def test_compare_elite_insights_emits_buff_uptime_tolerance_result() -> None: + fight = Fight( + id="fight", + header=EvtcHeader(build_version="20260224", agent_count=1, duration_ms=10_000), + agents=[ + Agent( + id=1, + name="Player", + profession=Profession.ENGINEER, + elite=EliteSpec.SCRAPPER, + is_player=True, + account_name=":Player.1234", + instance_id=1111, + ) + ], + ) + expected: dict[str, Any] = { + "players": [ + { + "account": "Player.1234", + "instanceID": 1111, + "firstAware": 0, + "buffUptimes": [{"id": 1187, "buffData": [{"uptime": 100.0}]}], + } + ] + } + events = [ + BoonApplyEvent( + time_ms=0, + source_agent_id=1, + target_agent_id=1, + skill_id=1187, + duration_ms=10_000, + stacks=1, + kind="apply", + ) + ] + + result = compare_elite_insights(fight, expected, events) + + assert result["differences"] == {} + rows = _rows(result) + row = rows["players[Player.1234].buffUptimes[1187].uptime"] + assert row["status"] == "PASS" + assert row["rule"] == "buff-uptime-tolerance" + assert row["dimensions"] == {"account": "Player.1234", "slice": 0, "buff_id": 1187} + + +def test_compare_elite_insights_emits_per_cast_rotation_results() -> None: + fight = Fight( + id="fight", + header=EvtcHeader(build_version="20260224", agent_count=1, duration_ms=10_000), + agents=[ + Agent( + id=1, + name="Player", + profession=Profession.MESMER, + elite=EliteSpec.CHRONOMANCER, + is_player=True, + account_name=":Player.1234", + instance_id=1111, + ) + ], + ) + expected: dict[str, Any] = { + "players": [ + { + "account": "Player.1234", + "instanceID": 1111, + "firstAware": 0, + "rotation": [{"id": 42, "skills": [{"castTime": 100, "duration": 500}]}], + } + ] + } + + result = compare_elite_insights(fight, expected, []) + + rows = _rows(result) + row = rows["players[Player.1234].rotation[castTime=100][duration=500][skillID=42]"] + assert row["status"] == "FAIL" + assert row["rule"] == "rotation-cast-match" + assert row["dimensions"] == {"account": "Player.1234", "slice": 0, "skill_id": 42} + + +def test_compare_elite_insights_reports_dps_all_scalar_results() -> None: + fight = Fight( + id="fight", + header=EvtcHeader(build_version="20260224", agent_count=1, duration_ms=10_000), + agents=[ + Agent( + id=1, + name="Player", + profession=Profession.MESMER, + elite=EliteSpec.CHRONOMANCER, + is_player=True, + account_name=":Player.1234", + instance_id=1111, + ) + ], + skills=[Skill(id=42, name="Blurred Frenzy")], + ) + damage = DamageEvent( + time_ms=500, + source_agent_id=1, + target_agent_id=2, + skill_id=42, + damage=1000, + connected=True, + ) + expected: dict[str, Any] = { + "players": [ + { + "account": "Player.1234", + "instanceID": 1111, + "firstAware": 0, + "lastAware": 10000, + "dpsAll": [{"damage": 2000, "condiDamage": 0, "powerDamage": 2000}], + } + ] + } + + result = compare_elite_insights(fight, expected, [damage]) + + rows = _rows(result) + row = rows["players[Player.1234].dpsAll.damage"] + assert row["status"] == "FAIL" + assert row["rule"] == "dpsAll-field" + assert row["expected"] == 2000 + assert row["actual"] == 1000 + assert row["delta"] == -1000 + assert row["dimensions"] == {"account": "Player.1234", "slice": 0} diff --git a/scripts/ei-parity/ei_diff.py b/scripts/ei-parity/ei_diff.py index fee9c14f..5ed5a053 100644 --- a/scripts/ei-parity/ei_diff.py +++ b/scripts/ei-parity/ei_diff.py @@ -10,6 +10,7 @@ import argparse import hashlib import json +import math import os import re import sys @@ -23,12 +24,20 @@ LOGS = ROOT / "zevtc files" EI_OUT = ROOT / ".tooling" / "ei-out" CORPUS = Path(__file__).resolve().parent / "corpus.txt" +CORPUS_BASELINE = Path(__file__).resolve().parent / "corpus-baseline.json" MANIFEST = Path(__file__).resolve().parent / "corpus-manifest.json" +KNOWN_DELTAS = Path(__file__).resolve().parent / "known-deltas.json" EI_CLI = ROOT / ".tooling" / "GW2EICLI" / "GuildWars2EliteInsights-CLI.dll" EI_VERSION = "3.26.0.0" EXPORT_TYPE = "detailed_wvw_kill" +REPORT_SCHEMA_VERSION = 1 +KNOWN_DELTAS_SCHEMA_VERSION = 1 MANIFEST_KEYS = {"schema_version", "reference", "tag_vocabulary", "entries"} +KNOWN_DELTAS_KEYS = {"schema_version", "rules"} +KNOWN_DELTA_RULE_KEYS = {"id", "selector", "constraint", "reason", "remove_when"} +KNOWN_DELTA_SELECTOR_KEYS = {"stem", "account", "slice", "bucket", "skill_id", "buff_id", "key"} +KNOWN_DELTA_CONSTRAINT_KEYS = {"max_abs_delta"} REFERENCE_KEYS = {"ei_version", "cli_sha256", "export_type"} ENTRY_KEYS = {"stem", "date", "evtc_sha256", "export_sha256", "tags"} SHA256 = re.compile(r"[0-9a-f]{64}") @@ -199,7 +208,13 @@ def validate_corpus() -> tuple[list[str], str, dict[str, object]]: # noqa: PLR0 def _validate_baseline_destination(path: Path, corpus: list[str]) -> Path: destination = path.resolve() - inputs = {MANIFEST.resolve(), CORPUS.resolve(), EI_CLI.resolve()} + inputs = { + MANIFEST.resolve(), + CORPUS.resolve(), + CORPUS_BASELINE.resolve(), + KNOWN_DELTAS.resolve(), + EI_CLI.resolve(), + } inputs.update((LOGS / f"{stem}.zevtc").resolve() for stem in corpus) inputs.update((EI_OUT / f"{stem}_{EXPORT_TYPE}.json").resolve() for stem in corpus) if destination in inputs: @@ -207,7 +222,7 @@ def _validate_baseline_destination(path: Path, corpus: list[str]) -> Path: return destination -def _write_baseline(destination: Path, content: str) -> None: +def _write_atomic(destination: Path, content: str) -> None: temporary: Path | None = None try: with tempfile.NamedTemporaryFile( @@ -225,7 +240,62 @@ def _write_baseline(destination: Path, content: str) -> None: except OSError: if temporary is not None: temporary.unlink(missing_ok=True) - _fail("unable to write baseline") + _fail("unable to write output") + + +def _validate_known_deltas(data: object) -> list[dict[str, object]]: # noqa: PLR0912 + if not isinstance(data, dict): + _fail("known-deltas.json must be a mapping") + if set(data) != KNOWN_DELTAS_KEYS: + _fail("known-deltas.json has unexpected top-level keys") + if data.get("schema_version") != KNOWN_DELTAS_SCHEMA_VERSION: + _fail("known-deltas.json has an unsupported schema version") + rules = data.get("rules") + if not isinstance(rules, list): + _fail("known-deltas.json rules must be a list") + seen_ids: set[str] = set() + for rule in rules: + if not isinstance(rule, dict): + _fail("known-deltas.json rule must be a mapping") + if set(rule) != KNOWN_DELTA_RULE_KEYS: + _fail("known-deltas.json rule has unexpected keys") + rule_id = rule.get("id") + if not isinstance(rule_id, str) or not rule_id: + _fail("known-deltas.json rule id must be a non-empty string") + if rule_id in seen_ids: + _fail(f"known-deltas.json duplicate rule id: {rule_id}") + seen_ids.add(rule_id) + selector = rule.get("selector") + if not isinstance(selector, dict) or not selector: + _fail("known-deltas.json rule selector must be a non-empty mapping") + if not set(selector) <= KNOWN_DELTA_SELECTOR_KEYS: + _fail("known-deltas.json rule selector has unexpected keys") + if any(value is None for value in selector.values()): + _fail("known-deltas.json rule selector must not contain null values") + if not all(isinstance(v, str | int | float | bool) for v in selector.values()): + _fail("known-deltas.json rule selector values must be scalar") + constraint = rule.get("constraint") + if not isinstance(constraint, dict) or not constraint: + _fail("known-deltas.json rule constraint must be a non-empty mapping") + if not set(constraint) <= KNOWN_DELTA_CONSTRAINT_KEYS: + _fail("known-deltas.json rule constraint has unexpected keys") + bound = constraint.get("max_abs_delta") + if ( + not isinstance(bound, int | float) + or isinstance(bound, bool) + or not math.isfinite(bound) + or bound <= 0 + ): + _fail("known-deltas.json rule max_abs_delta must be a positive number") + return rules + + +def _load_known_deltas() -> list[dict[str, object]]: + if not KNOWN_DELTAS.is_file(): + return [] + return _validate_known_deltas( + _load_json(KNOWN_DELTAS, "known-deltas.json", reject_duplicates=True) + ) def _rotation_deltas(diffs: dict[str, object]) -> tuple[Counter[int], Counter[int]]: @@ -325,14 +395,138 @@ def run_one(stem: str) -> dict[str, object]: "ei_targets": len(expected.get("targets", [])), "n_diffs": len(diffs), "differences": diffs, + "results": result["results"], "buckets": Counter(bucket(k) for k in diffs), } -def main() -> int: # noqa: PLR0912 +def _selector_matches( + selector: dict[str, object], + stem: str, + result: dict[str, object], +) -> bool: + dimensions = result.get("dimensions") + if not isinstance(dimensions, dict): + dimensions = {} + exact = { + "stem": stem, + "account": dimensions.get("account"), + "slice": dimensions.get("slice"), + "bucket": bucket(result.get("key", "")), + "skill_id": dimensions.get("skill_id"), + "buff_id": dimensions.get("buff_id"), + "key": result.get("key"), + } + return all(value == exact[key] for key, value in selector.items()) + + +def _constraint_matches( + constraint: dict[str, object], + result: dict[str, object], +) -> bool: + delta = result.get("delta") + bound = constraint.get("max_abs_delta") + if bound is not None: + if not isinstance(delta, int | float) or not isinstance(bound, int | float): + return False + if abs(delta) > abs(bound): + return False + return True + + +def _classify_results( + reports: list[dict[str, object]], + rules: list[dict[str, object]], +) -> list[dict[str, object]]: + rows: list[dict[str, object]] = [] + for rep in reports: + stem = rep["stem"] + for result in rep["results"]: + if not isinstance(result, dict): + continue + status = result.get("status") + if status not in {"PASS", "FAIL"}: + _fail(f"{stem}: unexpected result status {status!r}") + known: object = None + if status == "FAIL": + for rule in rules: + if not _selector_matches(rule["selector"], stem, result): + continue + if not _constraint_matches(rule["constraint"], result): + continue + known = { + "rule_id": rule["id"], + "reason": rule["reason"], + "remove_when": rule["remove_when"], + } + break + if known is not None: + status = "KNOWN_DELTA" + rows.append( + { + "stem": stem, + "key": result["key"], + "bucket": bucket(result["key"]), + "status": status, + "expected": result.get("expected"), + "actual": result.get("actual"), + "delta": result.get("delta"), + "rule": result.get("rule"), + "dimensions": result.get("dimensions"), + "known_delta": known, + } + ) + rows.sort(key=lambda row: (row["stem"], row["key"])) + return rows + + +def _build_report( + reports: list[dict[str, object]], + rules: list[dict[str, object]], + reference: str, + manifest_sha256: str, +) -> dict[str, object]: + rows = _classify_results(reports, rules) + summary_by_status: Counter[str] = Counter(row["status"] for row in rows) + summary_by_status_bucket: dict[str, Counter[str]] = {} + for row in rows: + per_status = summary_by_status_bucket.setdefault(row["status"], Counter()) + per_status[row["bucket"]] += 1 + return { + "schema_version": REPORT_SCHEMA_VERSION, + "reference": reference, + "manifest_sha256": manifest_sha256, + "log_count": len(reports), + "summary_by_status": dict(sorted(summary_by_status.items())), + "summary_by_status_bucket": { + status: dict(sorted(buckets.items())) + for status, buckets in sorted(summary_by_status_bucket.items()) + }, + "results": rows, + } + + +def _validate_report_destination(path: Path, corpus: list[str]) -> Path: + destination = path.resolve() + inputs = { + MANIFEST.resolve(), + CORPUS.resolve(), + CORPUS_BASELINE.resolve(), + KNOWN_DELTAS.resolve(), + EI_CLI.resolve(), + } + inputs.update((LOGS / f"{stem}.zevtc").resolve() for stem in corpus) + inputs.update((EI_OUT / f"{stem}_{EXPORT_TYPE}.json").resolve() for stem in corpus) + if destination in inputs: + _fail("report destination is a certification input") + return destination + + +def main() -> int: # noqa: PLR0912, PLR0915 ap = argparse.ArgumentParser() ap.add_argument("stems", nargs="*") ap.add_argument("--json", dest="json_out") + ap.add_argument("--report-json", dest="report_out") ap.add_argument("--top", type=int, default=40) ap.add_argument( "--rotation-skills", @@ -361,6 +555,13 @@ def main() -> int: # noqa: PLR0912 json_out = ( _validate_baseline_destination(Path(args.json_out), corpus) if args.json_out else None ) + if args.report_out and args.stems: + _fail("report output requires the complete corpus") + report_out = ( + _validate_report_destination(Path(args.report_out), corpus) if args.report_out else None + ) + if json_out is not None and report_out is not None and json_out == report_out: + _fail("json and report destinations must differ") stems = args.stems or corpus reports = [] grand = Counter() @@ -405,7 +606,7 @@ def main() -> int: # noqa: PLR0912 shown += 1 if json_out is not None: - _write_baseline( + _write_atomic( json_out, json.dumps( { @@ -415,9 +616,18 @@ def main() -> int: # noqa: PLR0912 "buckets": dict(sorted(grand.items())), }, indent=2, + allow_nan=False, ) + "\n", ) + if report_out is not None: + report = _build_report( + reports, + _load_known_deltas(), + reference, + manifest_sha256, + ) + _write_atomic(report_out, json.dumps(report, indent=2, allow_nan=False) + "\n") return 0 diff --git a/scripts/ei-parity/known-deltas.json b/scripts/ei-parity/known-deltas.json new file mode 100644 index 00000000..f69920aa --- /dev/null +++ b/scripts/ei-parity/known-deltas.json @@ -0,0 +1,4 @@ +{ + "schema_version": 1, + "rules": [] +} diff --git a/tests/scripts/test_ei_diff.py b/tests/scripts/test_ei_diff.py index baae4f90..573b47c0 100644 --- a/tests/scripts/test_ei_diff.py +++ b/tests/scripts/test_ei_diff.py @@ -274,6 +274,17 @@ def report(stem): "rotation_extra": 0, "differences": {"players[Private].z": {"expected": "A", "actual": "B"}}, "buckets": ei_diff.Counter({"z.bucket": 1, "a.bucket": 1}), + "results": [ + { + "key": "players[Private].dpsAll.damage", + "status": "FAIL", + "expected": 1, + "actual": 2, + "delta": 1, + "rule": "dpsAll-field", + "dimensions": {"account": "Private", "slice": 0}, + } + ], } monkeypatch.setattr(ei_diff, "run_one", report) @@ -289,3 +300,400 @@ def report(stem): assert replaced and replaced[0][1] == output assert replaced[0][0].parent == output.parent assert not any(word in output.read_text() for word in ("Private", "expected", "actual")) + + +def _single_result_report(stem): + return { + "stem": stem, + "n_diffs": 1, + "events": 1, + "agents": 1, + "ei_players": 1, + "ei_targets": 1, + "parse_seconds": 0, + "rotation_missing": 0, + "rotation_extra": 0, + "differences": {"players[Private].dpsAll.damage": {"expected": 1, "actual": 2}}, + "buckets": ei_diff.Counter({"players.dpsAll.damage": 1}), + "results": [ + { + "key": "players[Private].dpsAll.damage", + "status": "FAIL", + "expected": 1, + "actual": 2, + "delta": 1, + "rule": "dpsAll-field", + "dimensions": {"account": "Private", "slice": 0}, + } + ], + } + + +def test_report_requires_complete_corpus(monkeypatch, tmp_path): + files = _certification_files(tmp_path) + _configure(monkeypatch, files) + output = tmp_path / "report.json" + monkeypatch.setattr( + ei_diff.sys, "argv", [str(SCRIPT), "--report-json", str(output), files[-1][0]] + ) + monkeypatch.setattr(ei_diff, "run_one", lambda _stem: pytest.fail("parsed a log")) + + with pytest.raises(SystemExit, match="report output requires the complete corpus"): + ei_diff.main() + + +def test_report_destination_cannot_replace_certification_input(monkeypatch, tmp_path): + files = _certification_files(tmp_path) + _configure(monkeypatch, files) + monkeypatch.setattr(ei_diff.sys, "argv", [str(SCRIPT), "--report-json", str(files[4])]) + monkeypatch.setattr(ei_diff, "run_one", lambda _stem: pytest.fail("parsed a log")) + + with pytest.raises(SystemExit, match="report destination is a certification input"): + ei_diff.main() + + +def test_report_is_atomic_sorted_and_summarized(monkeypatch, tmp_path): + files = _certification_files(tmp_path) + _configure(monkeypatch, files) + output = tmp_path / "report.json" + replaced = [] + real_replace = ei_diff.os.replace + + def replace(source, destination): + replaced.append((Path(source), Path(destination))) + real_replace(source, destination) + + def report(stem): + return _single_result_report(stem) + + monkeypatch.setattr(ei_diff, "run_one", report) + monkeypatch.setattr(ei_diff.os, "replace", replace) + monkeypatch.setattr(ei_diff, "KNOWN_DELTAS", tmp_path / "known-deltas.json") + monkeypatch.setattr( + ei_diff.sys, "argv", [str(SCRIPT), "--report-json", str(output), "--rotation-skills", "0"] + ) + + assert ei_diff.main() == 0 + data = json.loads(output.read_text()) + assert data["schema_version"] == 1 + assert data["log_count"] == 35 + assert data["summary_by_status"] == {"FAIL": 35} + assert data["summary_by_status_bucket"] == {"FAIL": {"players.dpsAll.damage": 35}} + assert data["results"][0]["bucket"] == "players.dpsAll.damage" + assert replaced and replaced[0][1] == output + + +def test_known_delta_rule_flips_fail_to_known_delta(monkeypatch, tmp_path): + files = _certification_files(tmp_path) + _configure(monkeypatch, files) + output = tmp_path / "report.json" + rules = tmp_path / "known-deltas.json" + rules.write_text( + json.dumps( + { + "schema_version": 1, + "rules": [ + { + "id": "dpsall-rounding", + "selector": { + "bucket": "players.dpsAll.damage", + "key": "players[Private].dpsAll.damage", + }, + "constraint": {"max_abs_delta": 1}, + "reason": "EI rounds contribution to integers", + "remove_when": "EI fixes rounding", + } + ], + } + ) + ) + + def report(stem): + return _single_result_report(stem) + + monkeypatch.setattr(ei_diff, "run_one", report) + monkeypatch.setattr(ei_diff, "KNOWN_DELTAS", rules) + monkeypatch.setattr( + ei_diff.sys, "argv", [str(SCRIPT), "--report-json", str(output), "--rotation-skills", "0"] + ) + + assert ei_diff.main() == 0 + data = json.loads(output.read_text()) + assert data["summary_by_status"] == {"KNOWN_DELTA": 35} + row = data["results"][0] + assert row["status"] == "KNOWN_DELTA" + assert row["known_delta"]["rule_id"] == "dpsall-rounding" + + +def test_known_delta_rule_never_flips_a_pass_row(monkeypatch, tmp_path): + files = _certification_files(tmp_path) + _configure(monkeypatch, files) + output = tmp_path / "report.json" + rules = tmp_path / "known-deltas.json" + rules.write_text( + json.dumps( + { + "schema_version": 1, + "rules": [ + { + "id": "catches-everything", + "selector": {"bucket": "players.dpsAll.damage"}, + "constraint": {"max_abs_delta": 1000}, + "reason": "wild", + "remove_when": "never", + } + ], + } + ) + ) + + def report(stem): + rep = _single_result_report(stem) + rep["results"] = [ + { + "key": "players[Private].dpsAll.damage", + "status": "PASS", + "expected": 1, + "actual": 1, + "delta": 0, + "rule": "dpsAll-field", + "dimensions": {"account": "Private", "slice": None}, + } + ] + rep["differences"] = {} + rep["n_diffs"] = 0 + rep["buckets"] = ei_diff.Counter() + return rep + + monkeypatch.setattr(ei_diff, "run_one", report) + monkeypatch.setattr(ei_diff, "KNOWN_DELTAS", rules) + monkeypatch.setattr( + ei_diff.sys, "argv", [str(SCRIPT), "--report-json", str(output), "--rotation-skills", "0"] + ) + + assert ei_diff.main() == 0 + data = json.loads(output.read_text()) + assert data["summary_by_status"] == {"PASS": 35} + assert all(row["status"] == "PASS" for row in data["results"]) + + +@pytest.mark.parametrize( + "payload", + [ + { + "id": "", + "selector": {"bucket": "players.dpsAll.damage"}, + "constraint": {"max_abs_delta": 1}, + "reason": "why", + "remove_when": "never", + }, + { + "id": "neg-bound", + "selector": {"bucket": "players.dpsAll.damage"}, + "constraint": {"max_abs_delta": -1}, + "reason": "why", + "remove_when": "never", + }, + { + "id": "zero-bound", + "selector": {"bucket": "players.dpsAll.damage"}, + "constraint": {"max_abs_delta": 0}, + "reason": "why", + "remove_when": "never", + }, + { + "id": "null-selector", + "selector": {"account": None}, + "constraint": {"max_abs_delta": 1}, + "reason": "why", + "remove_when": "never", + }, + { + "id": "non-scalar-selector", + "selector": {"bucket": ["players.dpsAll.damage"]}, + "constraint": {"max_abs_delta": 1}, + "reason": "why", + "remove_when": "never", + }, + { + "id": "nan-constraint", + "selector": {"bucket": "players.dpsAll.damage"}, + "constraint": {"max_abs_delta": float("nan")}, + "reason": "why", + "remove_when": "never", + }, + ], +) +def test_invalid_known_delta_rule_is_rejected(monkeypatch, tmp_path, payload): + files = _certification_files(tmp_path) + _configure(monkeypatch, files) + rules = tmp_path / "known-deltas.json" + rules.write_text(json.dumps({"schema_version": 1, "rules": [payload]})) + + monkeypatch.setattr(ei_diff, "KNOWN_DELTAS", rules) + monkeypatch.setattr(ei_diff, "run_one", _single_result_report) + monkeypatch.setattr( + ei_diff.sys, + "argv", + [str(SCRIPT), "--report-json", str(tmp_path / "r.json"), "--rotation-skills", "0"], + ) + + with pytest.raises(SystemExit, match=r"known-deltas\.json"): + ei_diff.main() + + +def test_duplicate_rule_ids_are_rejected(monkeypatch, tmp_path): + files = _certification_files(tmp_path) + _configure(monkeypatch, files) + rules = tmp_path / "known-deltas.json" + rules.write_text( + json.dumps( + { + "schema_version": 1, + "rules": [ + { + "id": "dup", + "selector": {"bucket": "players.dpsAll.damage"}, + "constraint": {"max_abs_delta": 1}, + "reason": "a", + "remove_when": "never", + }, + { + "id": "dup", + "selector": {"bucket": "players.rotation"}, + "constraint": {"max_abs_delta": 1}, + "reason": "b", + "remove_when": "never", + }, + ], + } + ) + ) + + monkeypatch.setattr(ei_diff, "KNOWN_DELTAS", rules) + monkeypatch.setattr(ei_diff, "run_one", _single_result_report) + monkeypatch.setattr( + ei_diff.sys, + "argv", + [str(SCRIPT), "--report-json", str(tmp_path / "r.json"), "--rotation-skills", "0"], + ) + + with pytest.raises(SystemExit, match="duplicate rule id"): + ei_diff.main() + + +def test_report_destination_cannot_replace_known_deltas(monkeypatch, tmp_path): + files = _certification_files(tmp_path) + _configure(monkeypatch, files) + known = tmp_path / "known-deltas.json" + known.write_text('{"schema_version": 1, "rules": []}') + monkeypatch.setattr(ei_diff, "KNOWN_DELTAS", known) + monkeypatch.setattr(ei_diff.sys, "argv", [str(SCRIPT), "--report-json", str(known)]) + monkeypatch.setattr(ei_diff, "run_one", lambda _stem: pytest.fail("parsed a log")) + + with pytest.raises(SystemExit, match="report destination is a certification input"): + ei_diff.main() + + +def test_report_destination_cannot_replace_corpus_baseline(monkeypatch, tmp_path): + files = _certification_files(tmp_path) + _configure(monkeypatch, files) + baseline = tmp_path / "corpus-baseline.json" + baseline.write_text("{}") + monkeypatch.setattr(ei_diff, "CORPUS_BASELINE", baseline) + monkeypatch.setattr( + ei_diff.sys, "argv", [str(SCRIPT), "--report-json", str(baseline), "--rotation-skills", "0"] + ) + monkeypatch.setattr(ei_diff, "run_one", lambda _stem: pytest.fail("parsed a log")) + + with pytest.raises(SystemExit, match="report destination is a certification input"): + ei_diff.main() + + +def test_known_deltas_invalid_json_is_rejected_cleanly(monkeypatch, tmp_path): + files = _certification_files(tmp_path) + _configure(monkeypatch, files) + rules = tmp_path / "known-deltas.json" + rules.write_text("{not valid json") + monkeypatch.setattr(ei_diff, "KNOWN_DELTAS", rules) + monkeypatch.setattr(ei_diff, "run_one", _single_result_report) + monkeypatch.setattr( + ei_diff.sys, + "argv", + [str(SCRIPT), "--report-json", str(tmp_path / "r.json"), "--rotation-skills", "0"], + ) + + with pytest.raises(SystemExit, match=r"invalid known-deltas\.json"): + ei_diff.main() + + +def test_known_delta_rule_outside_selector_stays_fail(monkeypatch, tmp_path): + files = _certification_files(tmp_path) + _configure(monkeypatch, files) + output = tmp_path / "report.json" + rules = tmp_path / "known-deltas.json" + rules.write_text( + json.dumps( + { + "schema_version": 1, + "rules": [ + { + "id": "other-bucket", + "selector": {"bucket": "players.rotation"}, + "constraint": {"max_abs_delta": 100}, + "reason": "nope", + "remove_when": "never", + } + ], + } + ) + ) + + def report(stem): + return _single_result_report(stem) + + monkeypatch.setattr(ei_diff, "run_one", report) + monkeypatch.setattr(ei_diff, "KNOWN_DELTAS", rules) + monkeypatch.setattr( + ei_diff.sys, "argv", [str(SCRIPT), "--report-json", str(output), "--rotation-skills", "0"] + ) + + assert ei_diff.main() == 0 + data = json.loads(output.read_text()) + assert data["summary_by_status"] == {"FAIL": 35} + + +def test_known_delta_rule_above_bound_stays_fail(monkeypatch, tmp_path): + files = _certification_files(tmp_path) + _configure(monkeypatch, files) + output = tmp_path / "report.json" + rules = tmp_path / "known-deltas.json" + rules.write_text( + json.dumps( + { + "schema_version": 1, + "rules": [ + { + "id": "tight-bound", + "selector": {"bucket": "players.dpsAll.damage"}, + "constraint": {"max_abs_delta": 0.5}, + "reason": "too tight", + "remove_when": "never", + } + ], + } + ) + ) + + def report(stem): + return _single_result_report(stem) + + monkeypatch.setattr(ei_diff, "run_one", report) + monkeypatch.setattr(ei_diff, "KNOWN_DELTAS", rules) + monkeypatch.setattr( + ei_diff.sys, "argv", [str(SCRIPT), "--report-json", str(output), "--rotation-skills", "0"] + ) + + assert ei_diff.main() == 0 + data = json.loads(output.read_text()) + assert data["summary_by_status"] == {"FAIL": 35} diff --git a/web/pnpm-lock.yaml b/web/pnpm-lock.yaml index ce1a6ce9..cf39e006 100644 --- a/web/pnpm-lock.yaml +++ b/web/pnpm-lock.yaml @@ -1791,8 +1791,8 @@ packages: ms@2.1.3: resolution: {integrity: sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==} - nanoid@3.3.17: - resolution: {integrity: sha512-xQLf0A3HOMlgHq0n247/LRuAOYmB7dXJ/DvAxGvsSBij45XtBSmQycu+F8ODbHwns/XyFZagyL1+J0Offw1E0g==} + nanoid@3.3.18: + resolution: {integrity: sha512-DTg4MJbGMWkfi6VZFdNt2/caMbQy4Ou+Op/hJQvGEWcnVfoA1QA+xzRKAzw9jD6+GVOOeYr/mIcuDSdug6F6+w==} engines: {node: ^10 || ^12 || ^13.7 || ^14 || >=15.0.1} hasBin: true @@ -3875,7 +3875,7 @@ snapshots: ms@2.1.3: {} - nanoid@3.3.17: {} + nanoid@3.3.18: {} natural-compare@1.4.0: {} @@ -3985,7 +3985,7 @@ snapshots: postcss@8.5.25: dependencies: - nanoid: 3.3.17 + nanoid: 3.3.18 picocolors: 1.1.1 source-map-js: 1.2.1