diff --git a/_bmad-output/specs/spec-ei-parity-certification/stories/1-figer-la-reference-et-le-corpus.md b/_bmad-output/specs/spec-ei-parity-certification/stories/1-figer-la-reference-et-le-corpus.md new file mode 100644 index 00000000..ddc8cae8 --- /dev/null +++ b/_bmad-output/specs/spec-ei-parity-certification/stories/1-figer-la-reference-et-le-corpus.md @@ -0,0 +1,106 @@ +--- +title: 'Figer la référence et le corpus' +type: 'chore' +created: '2026-08-13' +status: 'done' +review_loop_iteration: 0 +baseline_commit: '553b40c73a650dadae0b48028a564a8fa7c32c0d' +context: + - '{project-root}/AGENTS.md' +--- + + + +## Intent + +**Problème :** Le corpus EI actuel contient 35 stems, mais son exécution ignore silencieusement des références absentes et ne prouve ni l'intégrité des fichiers locaux ni la version EI ayant produit les exports. La baseline n'est pas versionnée sous une forme agrégée et non personnelle. + +**Approche :** Versionner un manifeste pour les 35 couples privés, le valider avant toute comparaison dans `ei_diff.py`, puis produire une baseline complète par bucket sans détails joueur. Attester EI v3.26.0.0 par une version déclarée et l'empreinte SHA-256 du binaire CLI local utilisé. + +## Boundaries & Constraints + +**Always:** Conserver `corpus.txt` comme liste ordonnée canonique; exiger exactement une entrée de manifeste par stem; vérifier avant parsing la présence et le SHA-256 du `.zevtc`, de l'export EI et du binaire CLI; fixer le type `detailed_wvw_kill`, EI `3.26.0.0` et un vocabulaire fermé de tags; n'écrire dans la baseline que les compteurs agrégés des buckets normalisés effectivement observés. + +**Ask First:** Toute modification du corpus de 35 stems, de la version EI, du type d'export, du vocabulaire des tags ou des buckets inclus; toute impossibilité d'attester le binaire local par empreinte. + +**Never:** Déplacer, renommer ou versionner un log, un export EI ou le binaire; inclure chemin absolu, compte, personnage, `recordedBy` ou différence brute dans un artefact versionné; créer un comparateur ou un générateur de corpus parallèle; accepter le fallback ambigu `_detailed_wvw*.json` pour une exécution certifiée. + +## I/O & Edge-Case Matrix + +| Scenario | Input / State | Expected Output / Behavior | Error Handling | +|----------|--------------|---------------------------|----------------| +| Corpus valide | 35 entrées uniques, fichiers présents, empreintes conformes | Les 35 logs sont comparés et la baseline expose les buckets observés, triés par clé | N/A | +| Fichier absent ou altéré | Couple, CLI ou SHA-256 manquant/invalide | Aucun log n'est parsé et aucun rapport trompeur n'est produit | Échec explicite nommant seulement le stem et l'artefact | +| Manifeste incohérent | Stem absent, dupliqué, supplémentaire, mauvaise version/type/tag | Exécution refusée avant comparaison | Échec explicite sans donnée privée | +| Sortie baseline | Résultats des 35 logs | JSON stable: référence, empreinte du manifeste, nombre de logs et compte par bucket | Aucun détail attendu/réel ni identifiant joueur | + + + +## Code Map + +- `scripts/ei-parity/corpus.txt` -- ordre canonique des 35 stems; ne pas dupliquer cette sélection dans le code. +- `scripts/ei-parity/corpus-manifest.json` -- nouvel inventaire versionné: métadonnées non personnelles, empreintes des artefacts privés et attestation EI. +- `scripts/ei-parity/corpus-baseline.json` -- baseline agrégée versionnée produite sur les 35 logs: 133 différences dans 13 buckets normalisés. +- `scripts/ei-parity/ei_diff.py` -- preflight strict UTF-8/JSON/timestamps/empreintes avant parsing, chargement des couples et écriture atomique de la baseline; aucun fallback de référence. +- `scripts/ei-parity/ei.conf:3-15,25-28` -- preuve de configuration du type d'export détaillé WvW et timeline brute; lecture seule. +- `libs/gw2_analytics/src/gw2_analytics/ei_compare.py:385-416,403-413,749-1098` -- surface réelle des comparaisons et buckets; lecture seule pour cette story. +- `tests/scripts/test_ei_diff.py` -- nouveau test isolé sur fichiers temporaires, sans EVTC/export réel. +- `docs/ei-parity-workbench.md:19-24,40-66,79-89` -- procédure locale à mettre à jour avec validation du manifeste et génération de baseline. +- `.gitignore:130-136` et `AGENTS.md:20-25` -- garde-fous de confidentialité; les artefacts privés restent hors Git. + +## Tasks & Acceptance + +**Execution:** +- [x] `scripts/ei-parity/corpus-manifest.json` -- inventorier les 35 stems, dates, SHA-256 EVTC/export, version et empreinte CLI, type d'export et tags de couverture sans donnée personnelle. +- [x] `scripts/ei-parity/corpus-baseline.json` -- versionner la baseline complète agrégée des 35 logs sans donnée personnelle ni différence brute. +- [x] `scripts/ei-parity/ei_diff.py` -- charger et valider strictement manifeste, corpus et artefacts avant parsing; ajouter une sortie baseline stable des buckets observés sans différences brutes. +- [x] `tests/scripts/test_ei_diff.py` -- protéger l'alignement corpus/manifeste, les erreurs avant parsing, le contrat fermé des métadonnées et l'absence de détails privés dans la baseline. +- [x] `docs/ei-parity-workbench.md` -- documenter l'attestation EI, la vérification préalable et la commande reproductible de baseline. + +**Acceptance Criteria:** +- Given les artefacts locaux correspondant au manifeste, when le corpus complet est lancé, then les 35 stems sont vérifiés puis comparés sans omission silencieuse et le JSON agrégé est reproductible. +- Given un manifeste versionné, when il est inspecté, then chaque stem de `corpus.txt` apparaît exactement une fois avec les champs requis et aucune donnée personnelle ou chemin local. +- Given la baseline produite, when elle est inspectée, then chaque bucket normalisé observé apparaît une fois avec un compteur entier et aucune différence brute n'est présente. + +## Spec Change Log + +## Design Notes + +Le JSON d'export EI ne contient pas de preuve fiable de la release génératrice. L'attestation minimale est donc double: `ei_version: "3.26.0.0"` comme contrat humain et SHA-256 du `GuildWars2EliteInsights-CLI.dll` local comme preuve de l'artefact exécuté. La baseline doit également inclure le SHA-256 des octets du manifeste afin de lier ses compteurs au corpus exact. + +## Verification + +**Commands:** +- `uv run pytest tests/scripts/test_ei_diff.py -q` -- expected: contrat du manifeste et cas d'échec couverts sans artefact privé. +- `uv run ruff check scripts/ei-parity/ei_diff.py tests/scripts/test_ei_diff.py` -- expected: aucune erreur. +- `uv run ruff format --check scripts/ei-parity/ei_diff.py tests/scripts/test_ei_diff.py` -- expected: format conforme. +- `uv run python scripts/ei-parity/ei_diff.py --json scripts/ei-parity/corpus-baseline.json` -- expected: 35 logs vérifiés et baseline versionnée régénérée atomiquement avec 133 différences dans 13 buckets. + +## Suggested Review Order + +**Validation et sécurité** + +- Le preflight bloque tout corpus incohérent avant le premier parsing. + [`ei_diff.py:130`](../../../../../scripts/ei-parity/ei_diff.py#L130) + +- L'écriture atomique protège baseline et entrées contre l'écrasement. + [`ei_diff.py:210`](../../../../../scripts/ei-parity/ei_diff.py#L210) + +- L'entrée CLI impose corpus complet et destination sûre. + [`ei_diff.py:332`](../../../../../scripts/ei-parity/ei_diff.py#L332) + +**Référence figée** + +- Le manifeste atteste précisément les 35 couples privés et EI 3.26.0.0. + [`corpus-manifest.json:1`](../../../../../scripts/ei-parity/corpus-manifest.json#L1) + +- La baseline versionnée lie 133 écarts au manifeste exact. + [`corpus-baseline.json:1`](../../../../../scripts/ei-parity/corpus-baseline.json#L1) + +**Vérification et usage** + +- Les tests couvrent succès, altérations, incohérences et confidentialité. + [`test_ei_diff.py:1`](../../../../../tests/scripts/test_ei_diff.py#L1) + +- Le workbench documente régénération et contrat non personnel. + [`ei-parity-workbench.md:26`](../../../../../docs/ei-parity-workbench.md#L26) diff --git a/docs/ei-parity-workbench.md b/docs/ei-parity-workbench.md index 8867982e..9ff85009 100644 --- a/docs/ei-parity-workbench.md +++ b/docs/ei-parity-workbench.md @@ -23,6 +23,16 @@ mkdir -p .tooling && curl -sL -o .tooling/GW2EICLI.zip https://github.com/baaron `.tooling/` and `zevtc files/` are gitignored: the vendored CLI, the raw logs and the reference JSON are all local artefacts. +`scripts/ei-parity/corpus-manifest.json` is the versioned, non-personal +inventory that attests EI `3.26.0.0`, the SHA-256 of +`GuildWars2EliteInsights-CLI.dll`, and each private EVTC/export pair. Verify +the downloaded binary against its `reference.cli_sha256` before generating +exports: + +```bash +sha256sum .tooling/GW2EICLI/GuildWars2EliteInsights-CLI.dll +``` + Elite Insights' own sources answer questions its JSON cannot — above all the `InstantCastFinder` rules behind `rotation`, which are declarative and cannot be recovered by correlating event streams. A shallow sparse clone is enough: @@ -52,18 +62,38 @@ Output lands in `.tooling/ei-out/_detailed_wvw_kill.json`. ## 3. Diffing ```bash -uv run python scripts/ei-parity/ei_diff.py +uv run python scripts/ei-parity/ei_diff.py --json /tmp/ei-certification-baseline.json ``` -Runs `gw2_analytics.ei_compare.compare_elite_insights` over every corpus log -that has a reference, and prints per-log difference counts plus a histogram of -difference kinds with the per-player and per-target subscripts collapsed +Before parsing any log, this validates the exact 35-entry alignment between +`corpus.txt` and the manifest, the fixed EI version/export type/tag vocabulary, +strict timestamp stems, and the presence and SHA-256 of the CLI, EVTC files, +and exact `_detailed_wvw_kill.json` exports. The manifest, corpus, and exports +must be valid UTF-8; the manifest rejects duplicate JSON keys, and every export +must parse as JSON during preflight. Any failure refuses the complete run; no +`_detailed_wvw*.json` fallback is accepted for certification. + +It then runs `gw2_analytics.ei_compare.compare_elite_insights` over every +corpus log and prints per-log difference counts plus a histogram of difference +kinds with the per-player and per-target subscripts collapsed (`players[x].statsAll.totalDmg` → `players.statsAll.totalDmg`), so a systemic error reads as one large bucket instead of hundreds of unrelated rows. Useful flags: `--show ` prints raw expected/actual pairs for matching -buckets, `--json ` dumps the full report, and positional arguments -restrict the run to named log stems. +buckets for local investigation, and positional arguments restrict a local run +to named corpus stems. `--json ` requires the full corpus and writes only +the reference, manifest SHA-256, log count, and key-sorted aggregate bucket +counts; it never writes raw differences or player identifiers. It refuses any +certification input as its destination and atomically replaces the output only +after the complete run succeeds. + +The checked-in `scripts/ei-parity/corpus-baseline.json` is the current certified +baseline. Regenerate it only after an approved reference/corpus change or an +intentional parser result change: + +```bash +uv run python scripts/ei-parity/ei_diff.py --json scripts/ei-parity/corpus-baseline.json +``` Supporting probes in `scripts/ei-parity/`: @@ -84,6 +114,11 @@ month. The spread matters because arcdps changes the wire format between builds: the corpus covers ten distinct `arcVersion` values, including the 2026-05-07 break described in `docs/EVTC2025_FORMAT.md`. +`corpus.txt` is the canonical order. The manifest must contain exactly one +entry per stem in that order. Changing the corpus, EI version, export type, tag +vocabulary, or included bucket surface requires explicit approval and a new +baseline. + Re-running EI over the whole 7 758-log sink is possible but produces roughly 100 GB of JSON; the stratified corpus is the working set, and a full sweep is only worth it to confirm a release. diff --git a/scripts/ei-parity/corpus-baseline.json b/scripts/ei-parity/corpus-baseline.json new file mode 100644 index 00000000..a3bd9c88 --- /dev/null +++ b/scripts/ei-parity/corpus-baseline.json @@ -0,0 +1,24 @@ +{ + "reference": { + "ei_version": "3.26.0.0", + "cli_sha256": "5ffe8565a55f2f80a88dc28f4ec0f383fd84a64954c360c47264ecafed7faa38", + "export_type": "detailed_wvw_kill" + }, + "manifest_sha256": "ac66f2f7531dc9402592c8c7fe17918f71adfc5baa90b806af00105a7c39fce9", + "log_count": 35, + "buckets": { + "players.buffUptimes.uptime": 52, + "players.group": 6, + "players.rotation": 52, + "players.statsAll.appliedCrowdControl": 1, + "players.statsAll.appliedCrowdControlDownContribution": 2, + "players.statsAll.appliedCrowdControlDuration": 1, + "players.statsAll.appliedCrowdControlDurationDownContribution": 2, + "players.statsAll.downContribution": 6, + "players.statsTargets.appliedCrowdControl": 1, + "players.statsTargets.appliedCrowdControlDownContribution": 2, + "players.statsTargets.appliedCrowdControlDuration": 1, + "players.statsTargets.appliedCrowdControlDurationDownContribution": 2, + "players.statsTargets.downContribution": 5 + } +} diff --git a/scripts/ei-parity/corpus-manifest.json b/scripts/ei-parity/corpus-manifest.json new file mode 100644 index 00000000..c2105e3a --- /dev/null +++ b/scripts/ei-parity/corpus-manifest.json @@ -0,0 +1,54 @@ +{ + "schema_version": 1, + "reference": { + "ei_version": "3.26.0.0", + "cli_sha256": "5ffe8565a55f2f80a88dc28f4ec0f383fd84a64954c360c47264ecafed7faa38", + "export_type": "detailed_wvw_kill" + }, + "tag_vocabulary": [ + "monthly-size-1", + "monthly-size-2", + "monthly-size-3", + "monthly-size-4", + "monthly-supplement", + "pre-2026-05-07-format", + "post-2026-05-07-format" + ], + "entries": [ + {"stem": "20260128-160105", "date": "2026-01-28", "evtc_sha256": "0ab8329c6fcd1ec6b50cf2a463615d22b4bb7437bdfb6154a311fb009ff8dd56", "export_sha256": "cfff3c4a0996faab96c77455692d026162ba04460f0f111a702a61c6fe014b5d", "tags": ["monthly-size-1", "pre-2026-05-07-format"]}, + {"stem": "20260122-223116", "date": "2026-01-22", "evtc_sha256": "9df07559a828096e0fd7883dd94ef6495e325750b695f9323f87b2e74fb0f450", "export_sha256": "70c77245288afdad2ea1fadfa46f404e1279b34640270a881da4e86d9dfd40b1", "tags": ["monthly-size-2", "pre-2026-05-07-format"]}, + {"stem": "20260125-194936", "date": "2026-01-25", "evtc_sha256": "9f5537d107eaa2e01bcded500a33fbdfeff5c4fda109820d6b8b392ed502d19f", "export_sha256": "b9ddd56d1b0bfd42f2f6ab719e8210362a28cfcb1cd5792a1701158f26fcdb67", "tags": ["monthly-size-3", "pre-2026-05-07-format"]}, + {"stem": "20260129-110256", "date": "2026-01-29", "evtc_sha256": "4ec3d39fedd8acc4fec7fcbae7fde957e73cfb0fc1565ace796df51d536a5716", "export_sha256": "6c485dd7dc142667ed70dea0a2f6f54d93948468a983923fb640662e68f3d825", "tags": ["monthly-size-4", "pre-2026-05-07-format"]}, + {"stem": "20260213-213832", "date": "2026-02-13", "evtc_sha256": "6fbf851690f78988af28623cf62c7892ef8730d633775325ef58f1ea47e340fa", "export_sha256": "b1baa12654610204422548c55deb80e29db424a459f39e71a29be3c6a0530911", "tags": ["monthly-size-1", "pre-2026-05-07-format"]}, + {"stem": "20260210-235749", "date": "2026-02-10", "evtc_sha256": "128ec8f15e09e95e68835bbdee5ff8b36625efc5ecadaad2f442de35a28a38c1", "export_sha256": "a0c922e8a56b4535b545960a222a5ef33ff94c1dbaf51353e2089c6ee417624f", "tags": ["monthly-size-2", "pre-2026-05-07-format"]}, + {"stem": "20260205-231349", "date": "2026-02-05", "evtc_sha256": "1906b70a0605db07e7c2f38e90f9a9f459f10ed6e6cd3fc0455a158aeb4c7f8e", "export_sha256": "e636d45e0259382463748f59a7d975ee304cbcf54ed5e15b3fc594f7bf305913", "tags": ["monthly-size-3", "pre-2026-05-07-format"]}, + {"stem": "20260224-233019", "date": "2026-02-24", "evtc_sha256": "f74b311e10f75acaea41890097e53ffcb05954612185c064276e8fe3833a03d2", "export_sha256": "3833a321be1806825aa222feb300329fb946d445cf01599615ff35ebd2d22345", "tags": ["monthly-size-4", "pre-2026-05-07-format"]}, + {"stem": "20260316-231209", "date": "2026-03-16", "evtc_sha256": "7e48ffdb9465f5a666128ca32ef15cd6ef89a58cf577bc84fcca5aaef122b75d", "export_sha256": "47cda13d912c86342f00ab35424e9e45a5ca23f326b6ea51bba42713debf43a6", "tags": ["monthly-size-1", "pre-2026-05-07-format"]}, + {"stem": "20260309-233818", "date": "2026-03-09", "evtc_sha256": "3d5453d90969cbb6ce796bde9ce3e34769504d00805ac5eb8eb2ae6d45a387e8", "export_sha256": "ecfbb99c899abd165da5b36a4845b724bd3dfeaccaf7adea7386d8a20871a80b", "tags": ["monthly-size-2", "pre-2026-05-07-format"]}, + {"stem": "20260324-013930", "date": "2026-03-24", "evtc_sha256": "a5b8f62cc3fe1c254c0a0b6dbba2eb5c4d9bbf3d0238d781b8b5ca8c31e50e78", "export_sha256": "a5fc8b60f5747e66f901e750b646e97624d17841139efd39a25bb1c44ad43453", "tags": ["monthly-size-3", "pre-2026-05-07-format"]}, + {"stem": "20260314-234454", "date": "2026-03-14", "evtc_sha256": "70c692803838cb42afb3b1c6fcc5430ba065a0be648c93382229e33e36f76b3b", "export_sha256": "4a2654ff893bbb7f04db1adfe63118ca0a949b34feaaec2c1b3f567d7dd223be", "tags": ["monthly-size-4", "pre-2026-05-07-format"]}, + {"stem": "20260409-004339", "date": "2026-04-09", "evtc_sha256": "2e09ccb36f5ffead46c9bc4549fc6b62a0732ed3c790df5bb72cb8c2089bbfce", "export_sha256": "9768a8fbe2f68bfa6929106a10ce223ec29d8e491603a7db2c5b1607c1f5f3c2", "tags": ["monthly-size-1", "pre-2026-05-07-format"]}, + {"stem": "20260429-222732", "date": "2026-04-29", "evtc_sha256": "e7357e78d4c4fbf0a0990ca344c74beabcf90fdf28b126e572b7874bfc1b55df", "export_sha256": "0f332cea038ec425bfc355df8915cdf0ccaa68bbb37f4cc7bca939ad8cecbb31", "tags": ["monthly-size-2", "pre-2026-05-07-format"]}, + {"stem": "20260412-220632", "date": "2026-04-12", "evtc_sha256": "281441131dd4fff47533b3bf3e3a6c661053a55a5e088e02d0c8cbc9b13af4a6", "export_sha256": "9aeae27447ec36134daf1fb265a5ce818d94cb097011e589130a0dc35ebc4c7f", "tags": ["monthly-size-3", "pre-2026-05-07-format"]}, + {"stem": "20260424-204954", "date": "2026-04-24", "evtc_sha256": "3acbea5c38c0517cfd23acf5af38f5390d15597b8fb87168deb785cdafd74f5c", "export_sha256": "069a56acc3a72768c135c4273863f5f27763710118684e4d43252d04de48df1c", "tags": ["monthly-size-4", "pre-2026-05-07-format"]}, + {"stem": "20260519-112129", "date": "2026-05-19", "evtc_sha256": "502aa4810485cb613ad6d078b6296baa65b334aea2cedafd1bb9db8e1acbfd7c", "export_sha256": "c194fc9457262499915fb6bc550a5cf8cea8b0af233da0b11ea9c5fd51849cd1", "tags": ["monthly-size-1", "post-2026-05-07-format"]}, + {"stem": "20260512-181500", "date": "2026-05-12", "evtc_sha256": "51881dbb6e428c92a4b739f0ed8e5755dae5d5053c3c790a394bb51db18f7496", "export_sha256": "02a2668a15b4d9363583feaeb6ed5cf66730fbd34fdbb1c9af7537b4f54b0253", "tags": ["monthly-size-2", "post-2026-05-07-format"]}, + {"stem": "20260526-202841", "date": "2026-05-26", "evtc_sha256": "f493b1eccec4f4efab6c796e36c6a4ac74fa4249a8a1bbc55cd3edb2303a04fe", "export_sha256": "d459a959ed3d18acb94ca3d6c8d68ab26106f2dcdd142e5852f8a65f7dac4c74", "tags": ["monthly-size-3", "post-2026-05-07-format"]}, + {"stem": "20260508-001302", "date": "2026-05-08", "evtc_sha256": "b423025d29b3621fa90c61c9e1464edc2518e145d0d93caa5f90a11252f143a9", "export_sha256": "7d3a1407ae138caf5fad3ecdfc97f61cd939e55cee921c451d538ef041abf798", "tags": ["monthly-size-4", "pre-2026-05-07-format"]}, + {"stem": "20260608-011208", "date": "2026-06-08", "evtc_sha256": "785e94ef2d6eda75a0410d9ccbf2a0cb814d3652c59d4070d68efb0e4181e813", "export_sha256": "90357f0f97e8021eb399a2f908c9ca7f22ff29b409b13d148607f0c8325e9e0c", "tags": ["monthly-size-1", "post-2026-05-07-format"]}, + {"stem": "20260618-182428", "date": "2026-06-18", "evtc_sha256": "ec299025f4c2de8ec4cda5c23ddc4963478d6bbc62009fa110557c33ee36a86a", "export_sha256": "ecf15b80711a3374b1466987618d6a673b45a8ae3dacf3c207fad457cdf77983", "tags": ["monthly-size-2", "post-2026-05-07-format"]}, + {"stem": "20260621-225056", "date": "2026-06-21", "evtc_sha256": "0841cde9cf608759d52a4feaadded2a47e64145732cc196eb0eb789edf1cd398", "export_sha256": "000ac99fddb3cb49270d65282d0a53b2b468bdd1b3ca52971543f708c875e16f", "tags": ["monthly-size-3", "post-2026-05-07-format"]}, + {"stem": "20260625-215943", "date": "2026-06-25", "evtc_sha256": "a7fc9595896e293a078b40b9774893019e0c3682aae26c86c2da99eca141b8db", "export_sha256": "eb09f04bddbd0717175ddbae8293250c7d54ccd64e7887651df64a10c6fbed29", "tags": ["monthly-size-4", "post-2026-05-07-format"]}, + {"stem": "20260717-225125", "date": "2026-07-17", "evtc_sha256": "7da6211479329feb8ff4b97eb3edca549987bcd886875650786ac16eb44e6b6a", "export_sha256": "fa6a97a9bd60a9eeea0d592bbc9ec24c015ad84b64bc491ab8c777e8697b98dd", "tags": ["monthly-size-1", "post-2026-05-07-format"]}, + {"stem": "20260705-234656", "date": "2026-07-05", "evtc_sha256": "5e351898b3580bffbc40ab09bad9d3b3a6abcd836df910273ddec3c55d43b7ca", "export_sha256": "12455e839c7c861520709cc8b33d7b43bc82231133d8ca2390d4e0c5bff558b5", "tags": ["monthly-size-2", "post-2026-05-07-format"]}, + {"stem": "20260712-203400", "date": "2026-07-12", "evtc_sha256": "f9a98079163147dbcd8e5c9b22eff1721933c16937bd89804e74c102f2e0a098", "export_sha256": "a28fa6e926bb9b5c27c9efc991a630478c2fea522c1691b16ade11705f83fd0d", "tags": ["monthly-size-3", "post-2026-05-07-format"]}, + {"stem": "20260708-215736", "date": "2026-07-08", "evtc_sha256": "9fb6096501759911edba3062efb3c27dc5c7f5d7f6d575bc8d5d7a5e6095d0ed", "export_sha256": "ebac71e573e09d7664b986cdf36fb043a96c9cd0cd9fed1f30eafbf9cf2dd388", "tags": ["monthly-size-4", "post-2026-05-07-format"]}, + {"stem": "20260118-020548", "date": "2026-01-18", "evtc_sha256": "88e06fd301a8982c8ee3f5db1cd8563a721aace2f5c5ac30e393a2841eb542cb", "export_sha256": "823b3c2a86ecf837bb0898cc2cb1e39ab5608099d37509e6bbd0c91fa5873a9d", "tags": ["monthly-supplement", "pre-2026-05-07-format"]}, + {"stem": "20260217-210926", "date": "2026-02-17", "evtc_sha256": "a2b40e3a98b96a5dd426d615b2dabc2451a17ecf13dbbc2ffd0ec46ec4da9d3a", "export_sha256": "d7659c24d761a4eaa10e42db22a8523662d9497f715fd49118d8a8b73565bf63", "tags": ["monthly-supplement", "pre-2026-05-07-format"]}, + {"stem": "20260327-223844", "date": "2026-03-27", "evtc_sha256": "6a17ada92630f6fee0a4902e5924f21c80136a767447535a0d5940827c252390", "export_sha256": "c73964bd2ee61e21daa4952905074c52d1ab7c5d737af64ea4b025bcf81e518f", "tags": ["monthly-supplement", "pre-2026-05-07-format"]}, + {"stem": "20260428-222420", "date": "2026-04-28", "evtc_sha256": "19e6ddc7fee048ff4f2aa5e0dd8420b44a937474ffbc8da7284e80b656d8c47d", "export_sha256": "e512522a08feae96d356924605c559fe688782028165d7ae69183b777e064c6d", "tags": ["monthly-supplement", "pre-2026-05-07-format"]}, + {"stem": "20260506-211522", "date": "2026-05-06", "evtc_sha256": "ccbb295acb40d825a2a27dc2a7cd472b4a1b6c2d8263c7f72e3de2c3b6a06dab", "export_sha256": "ffa5ca596c4dcb2310ad6c2b3003770859f01fd786d739d003b482547d254cbe", "tags": ["monthly-supplement", "pre-2026-05-07-format"]}, + {"stem": "20260610-212627", "date": "2026-06-10", "evtc_sha256": "f5bb746d504ad80eed211a1bf030b4cf12a034e7446a0970ecbfebb01f09734b", "export_sha256": "a249c26fa269ba93e1a26b0725ce37ffafe1a42c639d265f269d7300f9030366", "tags": ["monthly-supplement", "post-2026-05-07-format"]}, + {"stem": "20260718-154555", "date": "2026-07-18", "evtc_sha256": "9e63f3cbd8c9f9a4c813d617656ebd949a65fb18357e3d7c89c7939343737837", "export_sha256": "d7c22f8f5534fd476e1ab4435aa5021171e6a0236ddb227635c4f05f60752be4", "tags": ["monthly-supplement", "post-2026-05-07-format"]} + ] +} diff --git a/scripts/ei-parity/ei_diff.py b/scripts/ei-parity/ei_diff.py index ed92d72f..fee9c14f 100644 --- a/scripts/ei-parity/ei_diff.py +++ b/scripts/ei-parity/ei_diff.py @@ -8,17 +8,40 @@ from __future__ import annotations import argparse +import hashlib import json +import os import re import sys +import tempfile import time from collections import Counter +from datetime import datetime from pathlib import Path ROOT = Path(__file__).resolve().parents[2] LOGS = ROOT / "zevtc files" EI_OUT = ROOT / ".tooling" / "ei-out" CORPUS = Path(__file__).resolve().parent / "corpus.txt" +MANIFEST = Path(__file__).resolve().parent / "corpus-manifest.json" +EI_CLI = ROOT / ".tooling" / "GW2EICLI" / "GuildWars2EliteInsights-CLI.dll" + +EI_VERSION = "3.26.0.0" +EXPORT_TYPE = "detailed_wvw_kill" +MANIFEST_KEYS = {"schema_version", "reference", "tag_vocabulary", "entries"} +REFERENCE_KEYS = {"ei_version", "cli_sha256", "export_type"} +ENTRY_KEYS = {"stem", "date", "evtc_sha256", "export_sha256", "tags"} +SHA256 = re.compile(r"[0-9a-f]{64}") +STEM = re.compile(r"[0-9]{8}-[0-9]{6}") +TAG_VOCABULARY = [ + "monthly-size-1", + "monthly-size-2", + "monthly-size-3", + "monthly-size-4", + "monthly-supplement", + "pre-2026-05-07-format", + "post-2026-05-07-format", +] KNOWN_ROTATION_DEAD_ENDS = { -41, @@ -54,6 +77,157 @@ def bucket(key: str) -> str: return _BRACKET.sub("", key) +def _sha256(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as stream: + for chunk in iter(lambda: stream.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() + + +def _fail(message: str) -> None: + raise SystemExit(f"certification refused: {message}") + + +def _unique_object(pairs: list[tuple[str, object]]) -> dict[str, object]: + result = {} + for key, value in pairs: + if key in result: + raise ValueError("duplicate JSON key") + result[key] = value + return result + + +def _load_json(path: Path, label: str, *, reject_duplicates: bool = False) -> object: + try: + text = path.read_bytes().decode("utf-8") + return json.loads(text, object_pairs_hook=_unique_object if reject_duplicates else None) + except (OSError, UnicodeDecodeError, json.JSONDecodeError, ValueError): + _fail(f"invalid {label}") + + +def _validate_stem(stem: object) -> str: + if not isinstance(stem, str) or not STEM.fullmatch(stem): + _fail("invalid stem") + try: + parsed = datetime.strptime(stem, "%Y%m%d-%H%M%S") + except ValueError: + _fail(f"{stem}: invalid stem") + if parsed.strftime("%Y%m%d-%H%M%S") != stem: + _fail(f"{stem}: invalid stem") + return stem + + +def _check_artifact(stem: str, artifact: str, path: Path, expected_sha256: object) -> None: + if not isinstance(expected_sha256, str) or not SHA256.fullmatch(expected_sha256): + _fail(f"{stem}: invalid {artifact} SHA-256") + if not path.is_file(): + _fail(f"{stem}: missing {artifact}") + if _sha256(path) != expected_sha256: + _fail(f"{stem}: altered {artifact}") + + +def validate_corpus() -> tuple[list[str], str, dict[str, object]]: # noqa: PLR0912 + """Validate the complete certification corpus before any log is parsed.""" + try: + manifest_bytes = MANIFEST.read_bytes() + manifest = json.loads(manifest_bytes.decode("utf-8"), object_pairs_hook=_unique_object) + except (OSError, UnicodeDecodeError, json.JSONDecodeError, ValueError): + _fail("invalid corpus manifest") + if not isinstance(manifest, dict) or set(manifest) != MANIFEST_KEYS: + _fail("invalid corpus manifest fields") + if manifest["schema_version"] != 1: + _fail("invalid corpus manifest schema version") + + reference = manifest["reference"] + if not isinstance(reference, dict) or set(reference) != REFERENCE_KEYS: + _fail("invalid reference fields") + if reference["ei_version"] != EI_VERSION: + _fail("invalid EI version") + if reference["export_type"] != EXPORT_TYPE: + _fail("invalid export type") + _check_artifact("corpus", "EI CLI", EI_CLI, reference["cli_sha256"]) + + vocabulary = manifest["tag_vocabulary"] + if vocabulary != TAG_VOCABULARY: + _fail("invalid tag vocabulary") + allowed_tags = set(vocabulary) + + try: + corpus = [ + line.strip() + for line in CORPUS.read_bytes().decode("utf-8").splitlines() + if line.strip() + ] + except (OSError, UnicodeDecodeError): + _fail("invalid corpus.txt") + corpus = [_validate_stem(stem) for stem in corpus] + if len(corpus) != 35 or len(corpus) != len(set(corpus)): + _fail("corpus.txt must contain 35 unique stems") + entries = manifest["entries"] + if not isinstance(entries, list) or len(entries) != len(corpus): + _fail("manifest must contain exactly one entry per corpus stem") + + seen: set[str] = set() + for index, entry in enumerate(entries): + if not isinstance(entry, dict) or set(entry) != ENTRY_KEYS: + _fail("invalid manifest entry fields") + stem = _validate_stem(entry["stem"]) + if stem in seen: + _fail(f"{stem}: duplicate manifest entry") + seen.add(stem) + if stem != corpus[index]: + _fail(f"{stem}: manifest order differs from corpus.txt") + if entry["date"] != f"{stem[:4]}-{stem[4:6]}-{stem[6:8]}": + _fail(f"{stem}: invalid date") + tags = entry["tags"] + if ( + not isinstance(tags, list) + or not tags + or any(not isinstance(tag, str) for tag in tags) + or len(tags) != len(set(tags)) + or not set(tags) <= allowed_tags + ): + _fail(f"{stem}: invalid tags") + _check_artifact(stem, "EVTC", LOGS / f"{stem}.zevtc", entry["evtc_sha256"]) + export_path = EI_OUT / f"{stem}_{EXPORT_TYPE}.json" + _check_artifact(stem, "EI export", export_path, entry["export_sha256"]) + _load_json(export_path, f"{stem}: EI export JSON") + + return corpus, hashlib.sha256(manifest_bytes).hexdigest(), reference + + +def _validate_baseline_destination(path: Path, corpus: list[str]) -> Path: + destination = path.resolve() + inputs = {MANIFEST.resolve(), CORPUS.resolve(), EI_CLI.resolve()} + inputs.update((LOGS / f"{stem}.zevtc").resolve() for stem in corpus) + inputs.update((EI_OUT / f"{stem}_{EXPORT_TYPE}.json").resolve() for stem in corpus) + if destination in inputs: + _fail("baseline destination is a certification input") + return destination + + +def _write_baseline(destination: Path, content: str) -> None: + temporary: Path | None = None + try: + with tempfile.NamedTemporaryFile( + "w", + encoding="utf-8", + dir=destination.parent, + prefix=f".{destination.name}.", + delete=False, + ) as stream: + temporary = Path(stream.name) + stream.write(content) + stream.flush() + os.fsync(stream.fileno()) + os.replace(temporary, destination) # noqa: PTH105 - atomic replacement is required + except OSError: + if temporary is not None: + temporary.unlink(missing_ok=True) + _fail("unable to write baseline") + + def _rotation_deltas(diffs: dict[str, object]) -> tuple[Counter[int], Counter[int]]: missing: Counter[int] = Counter() extra: Counter[int] = Counter() @@ -104,12 +278,7 @@ def _print_rotation_skill_deltas( def run_one(stem: str) -> dict[str, object]: log_path = LOGS / f"{stem}.zevtc" - ei_path = EI_OUT / f"{stem}_detailed_wvw_kill.json" - if not ei_path.exists(): - candidates = sorted(EI_OUT.glob(f"{stem}_detailed_wvw*.json")) - if not candidates: - raise SystemExit(f"no EI reference for {stem}") - ei_path = candidates[0] + ei_path = EI_OUT / f"{stem}_{EXPORT_TYPE}.json" started = time.monotonic() raw = read_zevtc_archive(log_path) @@ -160,7 +329,7 @@ def run_one(stem: str) -> dict[str, object]: } -def main() -> int: +def main() -> int: # noqa: PLR0912 ap = argparse.ArgumentParser() ap.add_argument("stems", nargs="*") ap.add_argument("--json", dest="json_out") @@ -179,11 +348,20 @@ def main() -> int: ap.add_argument("--show", default=None, help="print raw diffs whose bucket matches this regex") args = ap.parse_args() - stems = args.stems or [ - s - for s in (line.strip() for line in CORPUS.read_text().splitlines()) - if s and (EI_OUT / f"{s}_detailed_wvw_kill.json").exists() - ] + corpus, manifest_sha256, reference = validate_corpus() + if len(args.stems) != len(set(args.stems)): + _fail("duplicate positional stem") + for stem in args.stems: + _validate_stem(stem) + unknown_stems = set(args.stems) - set(corpus) + if unknown_stems: + _fail("requested stem is not in corpus.txt") + if args.json_out and args.stems: + _fail("baseline output requires the complete corpus") + json_out = ( + _validate_baseline_destination(Path(args.json_out), corpus) if args.json_out else None + ) + stems = args.stems or corpus reports = [] grand = Counter() for stem in stems: @@ -226,13 +404,19 @@ def main() -> int: print(f"[{rep['stem']}] {key}\n {json.dumps(value)[:400]}") shown += 1 - if args.json_out: - Path(args.json_out).write_text( + if json_out is not None: + _write_baseline( + json_out, json.dumps( - [{**r, "buckets": dict(r["buckets"])} for r in reports], + { + "reference": reference, + "manifest_sha256": manifest_sha256, + "log_count": len(reports), + "buckets": dict(sorted(grand.items())), + }, indent=2, - default=str, ) + + "\n", ) return 0 diff --git a/tests/scripts/test_ei_diff.py b/tests/scripts/test_ei_diff.py new file mode 100644 index 00000000..baae4f90 --- /dev/null +++ b/tests/scripts/test_ei_diff.py @@ -0,0 +1,291 @@ +import hashlib +import importlib.util +import json +from datetime import datetime, timedelta +from pathlib import Path + +import pytest + +ROOT = Path(__file__).resolve().parents[2] +SCRIPT = ROOT / "scripts" / "ei-parity" / "ei_diff.py" +SPEC = importlib.util.spec_from_file_location("ei_diff", SCRIPT) +assert SPEC and SPEC.loader +ei_diff = importlib.util.module_from_spec(SPEC) +SPEC.loader.exec_module(ei_diff) + + +def _sha256(path): + return hashlib.sha256(path.read_bytes()).hexdigest() + + +def _certification_files(tmp_path): + logs = tmp_path / "logs" + exports = tmp_path / "exports" + logs.mkdir() + exports.mkdir() + cli = tmp_path / "GuildWars2EliteInsights-CLI.dll" + cli.write_bytes(b"cli") + start = datetime(2026, 1, 1) + stems = [(start + timedelta(days=index)).strftime("%Y%m%d-000000") for index in range(35)] + corpus = tmp_path / "corpus.txt" + corpus.write_text("\n".join(stems) + "\n") + entries = [] + for stem in stems: + evtc = logs / f"{stem}.zevtc" + export = exports / f"{stem}_detailed_wvw_kill.json" + evtc.write_bytes(stem.encode()) + export.write_text("{}") + entries.append( + { + "stem": stem, + "date": f"{stem[:4]}-{stem[4:6]}-{stem[6:8]}", + "evtc_sha256": _sha256(evtc), + "export_sha256": _sha256(export), + "tags": [ei_diff.TAG_VOCABULARY[0]], + } + ) + manifest = tmp_path / "corpus-manifest.json" + manifest.write_text( + json.dumps( + { + "schema_version": 1, + "reference": { + "ei_version": "3.26.0.0", + "cli_sha256": _sha256(cli), + "export_type": "detailed_wvw_kill", + }, + "tag_vocabulary": ei_diff.TAG_VOCABULARY, + "entries": entries, + } + ) + ) + return logs, exports, cli, corpus, manifest, stems + + +def _configure(monkeypatch, files): + logs, exports, cli, corpus, manifest, _ = files + monkeypatch.setattr(ei_diff, "LOGS", logs) + monkeypatch.setattr(ei_diff, "EI_OUT", exports) + monkeypatch.setattr(ei_diff, "EI_CLI", cli) + monkeypatch.setattr(ei_diff, "CORPUS", corpus) + monkeypatch.setattr(ei_diff, "MANIFEST", manifest) + + +def _manifest(files): + return json.loads(files[4].read_text()) + + +def _write_manifest(files, data): + files[4].write_text(json.dumps(data)) + + +def _assert_before_parsing(monkeypatch, message): + monkeypatch.setattr(ei_diff, "run_one", lambda _stem: pytest.fail("parsed a log")) + monkeypatch.setattr(ei_diff.sys, "argv", [str(SCRIPT)]) + with pytest.raises(SystemExit, match=message): + ei_diff.main() + + +def test_manifest_and_versioned_baseline_match_canonical_corpus(): + manifest = json.loads((ROOT / "scripts/ei-parity/corpus-manifest.json").read_text()) + baseline = json.loads((ROOT / "scripts/ei-parity/corpus-baseline.json").read_text()) + corpus = (ROOT / "scripts/ei-parity/corpus.txt").read_text().splitlines() + + assert len(corpus) == len(set(corpus)) == baseline["log_count"] == 35 + assert [entry["stem"] for entry in manifest["entries"]] == corpus + assert baseline["reference"] == manifest["reference"] + assert baseline["manifest_sha256"] == _sha256(ROOT / "scripts/ei-parity/corpus-manifest.json") + assert sum(baseline["buckets"].values()) == 133 + assert len(baseline["buckets"]) == 13 + + +@pytest.mark.parametrize( + ("artifact", "change", "message"), + [ + ("cli", "missing", "missing EI CLI"), + ("cli", "altered", "altered EI CLI"), + ("evtc", "missing", "missing EVTC"), + ("evtc", "altered", "altered EVTC"), + ("export", "missing", "missing EI export"), + ("export", "altered", "altered EI export"), + ], +) +def test_missing_or_altered_artifacts_fail_before_parsing( + monkeypatch, tmp_path, artifact, change, message +): + files = _certification_files(tmp_path) + _configure(monkeypatch, files) + path = { + "cli": files[2], + "evtc": files[0] / f"{files[-1][-1]}.zevtc", + "export": files[1] / f"{files[-1][-1]}_detailed_wvw_kill.json", + }[artifact] + path.unlink() if change == "missing" else path.write_bytes(b"altered") + + _assert_before_parsing(monkeypatch, message) + + +@pytest.mark.parametrize( + ("field", "value", "message"), + [ + ("ei_version", "3.27.0.0", "invalid EI version"), + ("export_type", "summary", "invalid export type"), + ], +) +def test_wrong_reference_contract_is_rejected(monkeypatch, tmp_path, field, value, message): + files = _certification_files(tmp_path) + _configure(monkeypatch, files) + data = _manifest(files) + data["reference"][field] = value + _write_manifest(files, data) + + _assert_before_parsing(monkeypatch, message) + + +@pytest.mark.parametrize( + "stem", + ["../private", "20260230-000000", "\uff12\uff10\uff12\uff16\uff10\uff11\uff10\uff11-000000"], +) +def test_invalid_stem_is_rejected_before_path_construction(monkeypatch, tmp_path, stem): + files = _certification_files(tmp_path) + _configure(monkeypatch, files) + data = _manifest(files) + files[3].write_text(stem + "\n" + "\n".join(files[-1][1:]) + "\n") + data["entries"][0]["stem"] = stem + _write_manifest(files, data) + + _assert_before_parsing(monkeypatch, "invalid stem") + + +@pytest.mark.parametrize("change", ["missing", "reordered", "duplicate", "tag"]) +def test_manifest_and_corpus_incoherence_is_rejected(monkeypatch, tmp_path, change): + files = _certification_files(tmp_path) + _configure(monkeypatch, files) + data = _manifest(files) + if change == "missing": + data["entries"].pop() + elif change == "reordered": + data["entries"][0], data["entries"][1] = data["entries"][1], data["entries"][0] + elif change == "duplicate": + data["entries"][-1] = data["entries"][0] + else: + data["entries"][0]["tags"] = ["unknown"] + _write_manifest(files, data) + + _assert_before_parsing(monkeypatch, "manifest|duplicate|invalid tags") + + +@pytest.mark.parametrize("source", ["manifest", "corpus", "export"]) +def test_invalid_unicode_is_rejected(monkeypatch, tmp_path, source): + files = _certification_files(tmp_path) + _configure(monkeypatch, files) + path = { + "manifest": files[4], + "corpus": files[3], + "export": files[1] / f"{files[-1][0]}_detailed_wvw_kill.json", + }[source] + path.write_bytes(b"\xff") + if source == "export": + data = _manifest(files) + data["entries"][0]["export_sha256"] = _sha256(path) + _write_manifest(files, data) + + _assert_before_parsing(monkeypatch, "invalid") + + +def test_duplicate_manifest_key_is_rejected(monkeypatch, tmp_path): + files = _certification_files(tmp_path) + _configure(monkeypatch, files) + text = ( + files[4] + .read_text() + .replace('"schema_version": 1', '"schema_version": 1, "schema_version": 1') + ) + files[4].write_text(text) + + _assert_before_parsing(monkeypatch, "invalid corpus manifest") + + +def test_unreadable_export_json_is_rejected_before_parsing(monkeypatch, tmp_path): + files = _certification_files(tmp_path) + _configure(monkeypatch, files) + export = files[1] / f"{files[-1][0]}_detailed_wvw_kill.json" + export.write_text("{") + data = _manifest(files) + data["entries"][0]["export_sha256"] = _sha256(export) + _write_manifest(files, data) + + _assert_before_parsing(monkeypatch, "invalid .* EI export JSON") + + +@pytest.mark.parametrize( + "args", + [["20260101-000000", "20260101-000000"], ["--json", "out.json", "20260101-000000"]], +) +def test_duplicate_or_partial_positional_run_is_rejected(monkeypatch, tmp_path, args): + files = _certification_files(tmp_path) + _configure(monkeypatch, files) + monkeypatch.setattr(ei_diff.sys, "argv", [str(SCRIPT), *args]) + monkeypatch.setattr(ei_diff, "run_one", lambda _stem: pytest.fail("parsed a log")) + + with pytest.raises(SystemExit, match=r"duplicate positional stem|complete corpus"): + ei_diff.main() + + +@pytest.mark.parametrize("input_name", ["cli", "corpus", "manifest", "evtc", "export"]) +def test_json_destination_cannot_replace_certification_input(monkeypatch, tmp_path, input_name): + files = _certification_files(tmp_path) + _configure(monkeypatch, files) + destination = { + "cli": files[2], + "corpus": files[3], + "manifest": files[4], + "evtc": files[0] / f"{files[-1][0]}.zevtc", + "export": files[1] / f"{files[-1][0]}_detailed_wvw_kill.json", + }[input_name] + monkeypatch.setattr(ei_diff.sys, "argv", [str(SCRIPT), "--json", str(destination)]) + monkeypatch.setattr(ei_diff, "run_one", lambda _stem: pytest.fail("parsed a log")) + + with pytest.raises(SystemExit, match="destination is a certification input"): + ei_diff.main() + + +def test_json_baseline_is_atomic_aggregate_only_and_sorted(monkeypatch, tmp_path): + files = _certification_files(tmp_path) + _configure(monkeypatch, files) + output = tmp_path / "baseline.json" + replaced = [] + real_replace = ei_diff.os.replace + + def replace(source, destination): + replaced.append((Path(source), Path(destination))) + real_replace(source, destination) + + def report(stem): + return { + "stem": stem, + "n_diffs": 2, + "events": 1, + "agents": 1, + "ei_players": 1, + "ei_targets": 1, + "parse_seconds": 0, + "rotation_missing": 0, + "rotation_extra": 0, + "differences": {"players[Private].z": {"expected": "A", "actual": "B"}}, + "buckets": ei_diff.Counter({"z.bucket": 1, "a.bucket": 1}), + } + + monkeypatch.setattr(ei_diff, "run_one", report) + monkeypatch.setattr(ei_diff.os, "replace", replace) + monkeypatch.setattr( + ei_diff.sys, "argv", [str(SCRIPT), "--json", str(output), "--rotation-skills", "0"] + ) + + assert ei_diff.main() == 0 + baseline = json.loads(output.read_text()) + assert list(baseline["buckets"]) == ["a.bucket", "z.bucket"] + assert baseline["buckets"] == {"a.bucket": 35, "z.bucket": 35} + assert replaced and replaced[0][1] == output + assert replaced[0][0].parent == output.parent + assert not any(word in output.read_text() for word in ("Private", "expected", "actual"))