NVIDIA · chrisknvidia · Aug 25, 2026 · Aug 28, 2026
@@ -68,7 +68,7 @@ jobs:
         run: python scripts/check_oss_boundary.py --root . --allowlist config/oss_boundary_allowlist.json
       - name: Lint committed BENCHMARK.md fixtures for leak patterns
         if: ${{ needs.classify-changes.outputs.docs_only != 'true' }}
-        run: python scripts/ci/check_public_benchmarks.py --require-files tests/golden
+        run: uv run python scripts/ci/check_public_benchmarks.py --require-files tests/golden
       - name: Lint
         if: ${{ needs.classify-changes.outputs.docs_only != 'true' }}
         run: uv run ruff check .
@@ -276,6 +276,7 @@ jobs:
           tests/test_cli.py
           tests/test_harbor_output_provenance.py
           tests/test_harbor_runtime_skill_isolation.py
+          tests/test_publication_identity.py
           tests/test_harbor_secure_copy.py::test_checked_windows_fallback_accepts_crt_descriptor_identity
           tests/test_harbor_secure_copy.py::test_checked_windows_fallback_verifies_portable_chmod_semantics
           tests/test_harbor_secure_copy.py::test_native_windows_fallback_copies_tree_and_file

@@ -29,6 +29,28 @@ All notable changes to SkillEvaluator are documented in this file.
   conversion limit, preserves nonzero Wilson interval widths and paired-effect
   directions at large case counts, and documents exact-rational omission
   markers.
+- `BENCHMARK.md` publication verdicts now require completed Tier 1 and Tier 2
+  execution evidence by default, conservatively resolve conflicting peer
+  policy and Tier 3 result metadata, bind every tier and policy claim to one
+  versioned source-tree digest, preserve the Tier 3 run ID, persist an explicit
+  publication status in JSON and HTML, and reject
+  publication `PASS` cards whose decision evidence is missing, incomplete,
+  linked, or hidden in raw HTML. Custom Tier 3 result roots inside the skill are
+  rejected; use the canonical `evals/results` path or an external results root.
+- Publication source identity now seals forward, reverse, and final source reads,
+  normalizes filesystem case aliases, aligns Tier 1/Tier 2 and Tier 3 runtime
+  projections with the v2 generated-artifact exclusions, and carries a bounded
+  source-change marker through rerendered Tier 3 evidence.
+- Report output is required outside the publication target. A default
+  in-target `reports/` location relocates to an authenticated sibling
+  `<skill>-reports` (or `<catalog>-reports`), refuses unowned collisions, and is
+  excluded from Tier 3 full-repository staging.
+- Public benchmark provenance now rejects placeholder identities, malformed or
+  future calendar dates, missing duplicated run IDs, hostile benchmark-policy
+  metadata, and environment-label substitutions in required proof fields.
+- Hardened Tier 3 HTML and Markdown reporting against recursive or malformed
+  display metadata, Markdown structure injection, and inline-JavaScript
+  injection through untrusted agent or skill names.
 
 ## 0.2.1 - 2026-08-24
 

@@ -10,3 +10,46 @@ records the exact resolved dependency set used for this release.
 | LLM | Anthropic (MIT), Boto3 (Apache-2.0), LiteLLM (MIT), OpenAI (Apache-2.0) |
 | Tier 3 | Harbor (Apache-2.0) |
 | Security | Bandit (Apache-2.0), pip-audit (Apache-2.0) |
+
+## Unicode security data
+
+`src/skillevaluator/publication_text.py` contains Unicode 15.1 general-category
+ranges and a generated subset of the Unicode 17.0.0 `confusables.txt` data used
+by Unicode Technical Standard #39.
+
+UNICODE LICENSE V3
+
+COPYRIGHT AND PERMISSION NOTICE
+
+Copyright © 1991-2026 Unicode, Inc.
+
+NOTICE TO USER: Carefully read the following legal agreement. BY DOWNLOADING,
+INSTALLING, COPYING OR OTHERWISE USING DATA FILES, AND/OR SOFTWARE, YOU
+UNEQUIVOCALLY ACCEPT, AND AGREE TO BE BOUND BY, ALL OF THE TERMS AND CONDITIONS
+OF THIS AGREEMENT. IF YOU DO NOT AGREE, DO NOT DOWNLOAD, INSTALL, COPY,
+DISTRIBUTE OR USE THE DATA FILES OR SOFTWARE.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of
+data files and any associated documentation (the "Data Files") or software and
+any associated documentation (the "Software") to deal in the Data Files or
+Software without restriction, including without limitation the rights to use,
+copy, modify, merge, publish, distribute, and/or sell copies of the Data Files
+or Software, and to permit persons to whom the Data Files or Software are
+furnished to do so, provided that either (a) this copyright and permission
+notice appear with all copies of the Data Files or Software, or (b) this
+copyright and permission notice appear in associated Documentation.
+
+THE DATA FILES AND SOFTWARE ARE PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OF THIRD
+PARTY RIGHTS. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR HOLDERS INCLUDED IN THIS
+NOTICE BE LIABLE FOR ANY CLAIM, OR ANY SPECIAL INDIRECT OR CONSEQUENTIAL
+DAMAGES, OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS,
+WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING
+OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THE DATA FILES OR
+SOFTWARE.
+
+Except as contained in this notice, the name of a copyright holder shall not be
+used in advertising or otherwise to promote the sale, use or other dealings in
+these Data Files or Software without prior written authorization of the
+copyright holder.
@@ -56,13 +56,43 @@ skillevaluator validate ./skills/example-skill \
   --output-dir ./benchmark-backfill/example-skill
 ```
 
-Tier 3 is publication-required by default. Omitting `--agent-eval` therefore
-produces an `INCOMPLETE` card with Tier 3 marked `NOT RUN`, and never a
-publication recommendation. If a catalog intentionally makes Tier 3 optional,
-its orchestration must persist `benchmark_policy.tier3_required = false`; the
-card then discloses that policy. Do not reuse a prior live score while changing
-only its label or date. If a baseline was not run, keep the explicit “uplift
-unavailable” state.
+Tier 1 is always publication-required. Tier 2 and Tier 3 are
+publication-required by default. Omitting Tier 2 with `--no-dedup` or
+`--tiers 1,3`, or omitting Tier 3 by leaving out `--agent-eval`, therefore
+produces an `INCOMPLETE` card with the omitted tier marked `NOT RUN` and never a
+publication recommendation. `--no-block-on-dedup` changes the command exit
+gate; it is not a publication waiver.
+
+If a catalog intentionally makes Tier 2 or Tier 3 optional, its orchestration
+must persist `benchmark_policy.tier2_required = false` or
+`benchmark_policy.tier3_required = false`. The card then discloses that policy.
+An optional tier that produced incomplete evidence still makes the card
+`INCOMPLETE`; only absent or cleanly skipped evidence can use the waiver.
+Do not reuse a prior live score while changing only its label or date. If a
+baseline was not run, keep the explicit “uplift unavailable” state.
+
+When an orchestrator combines tiers from separate jobs, merge the actual result
+objects, including each result's `publication_target` and the built-in Tier 1
+or Tier 2 `publication_evidence` producer marker, and preserve the explicit
+`benchmark_policy` in the combined artifact. Every contributing result must
+carry the same exact NFC filesystem-entry name and
+`skill-evaluator-source-tree/2` digest. Completed Tier 3 evidence
+must also preserve its run-owned `run_id` in the payload and summary. Anonymous
+legacy results, a changed source tree, or inconsistent run identity make the
+combined publication status `INCOMPLETE`.
+Do not infer a waiver from an unselected tier or a non-blocking job. A missing
+result or policy key defaults to required, so an incomplete split-tier merge
+fails closed. Peer result objects have equal precedence: conflicting required
+flags resolve to `true` independent of merge order. Agent-evaluation payload
+metadata and its summary are duplicated higher-precedence policy claims that
+must agree. Those claims are used only when the containing result is bound to
+the same exact source digest; a foreign or anonymous payload cannot create a
+waiver.
+
+Use the combined JSON report's top-level `publication_status` or
+`publication.status` for publication automation. Do not substitute
+`overall_status`, which records the process gate and can remain `passed` when
+publication evidence is incomplete.
 
 The candidate card is:
 
@@ -87,7 +117,7 @@ Expected presentation changes include:
 - removal of the `Num` column;
 - explicit `NOT RUN`, `SKIPPED`, and `INCOMPLETE` tier states;
 - collapsible methodology and non-blocking observations;
-- evaluator version, dataset digest, task composition, Tier 3 requirement,
+- evaluator version, dataset digest, task composition, Tier 2 and Tier 3 requirements,
   isolation wording, and freshness copy.
 
 Investigate any numerical change. The canonical mapping is Security=`security`,
@@ -105,15 +135,16 @@ and does not override that gate.
 Scan all candidates before copying them into skills:
 
 ```bash
-python scripts/ci/check_public_benchmarks.py \
+uv run python scripts/ci/check_public_benchmarks.py \
   --require-files \
   ./benchmark-backfill
 ```
 
 The linter rejects configured known leak patterns such as retired product
 identities, any validation-profile metadata line, common absolute home paths,
 ambiguous legacy uplift cells, the old `Num` column, missing metadata/decision
-sections, and a publication `PASS` without completed required Tier 3 evidence.
+sections, and a publication `PASS` without completed required Tier 1, Tier 2,
+or Tier 3 evidence.
 A clean result means none of those configured patterns matched; it is a fixture
 regression guard, not proof that a card is safe to publish. Keep human review
 and the repository's broader boundary and security checks in the promotion
@@ -128,7 +159,7 @@ catalog:
 cp ./benchmark-backfill/example-skill/BENCHMARK.md \
   ./skills/example-skill/BENCHMARK.md
 
-python scripts/ci/check_public_benchmarks.py --require-files ./skills
+uv run python scripts/ci/check_public_benchmarks.py --require-files ./skills
 git diff --check
 git diff -- ./skills/example-skill/BENCHMARK.md
 ```
@@ -149,6 +180,8 @@ Regenerate a card when any of these inputs changes:
 - attempt count/pass threshold;
 - execution environment or isolation mode.
 
-The generated evaluation date must come from the live run artifact. Older
-artifacts without an unambiguous timestamp remain “not recorded”; never replace
-that state with the report-generation date.
+The generated evaluation date must come from the live run artifact as a
+timezone-aware ISO timestamp. Date-only, timezone-naive, malformed, or
+materially future-dated values are not publication evidence. Older artifacts
+without an unambiguous timestamp remain “not recorded”; never replace that state
+with the report-generation date.
@@ -194,11 +194,22 @@ When the exit code is not enough — dashboards, custom thresholds, badge genera
 | --- | --- | --- |
 | `overall_passed` | boolean | `true` when every check passed |
 | `overall_status` | string | `passed`, `failed`, or `incomplete` (a required scanner produced no evidence) |
+| `publication_status` | string | `pass`, `neutral`, `fail`, or `incomplete`; use this field for publication eligibility |
+| `publication` | object | Publication decision, eligibility, reasons, and Tier 3 evidence status |
+| `benchmark_policy` | object | Persisted `tier2_required` and `tier3_required` evidence policy used for the publication decision |
+| `results[].publication_target` | object | Exact NFC filesystem-entry name, versioned source-tree digest, and digest algorithm used to bind split-tier evidence |
+| `results[].publication_evidence` | object | Validated built-in Tier 1 or Tier 2 producer contract: schema version, producer, tier, and canonical check ID |
+| `tier3.run_id` | string | Run-owned Tier 3 identity; duplicated in the Tier 3 summary for consistency checks |
 | `severity_counts` | object | Totals for `critical`, `high`, `medium`, `low` |
 | `total_errors`, `total_warnings` | number | Aggregate counts across all validators |
 | `skills` | array | Per-skill `{ name, passed, issue_count }` |
 | `quality_summary` | array | Quality details with `overall_score` (0–100) and `grade` (A–F); a folder target reports the collection average plus a `skill_count` |
 
+`overall_status` and `overall_passed` describe the configured command gate. They
+can be green when an advisory tier did not run, so do not use them as a
+publication signal. Require `publication_status == "pass"` (or
+`publication.eligible == true`) before publishing a benchmark.
+
 A step that enforces a stricter score than the built-in gate:
 
 ```bash title="Gate on quality score with jq"

@@ -100,7 +100,7 @@ execution environments are covered in
 
 | Variable | Default | Effect |
 | --- | --- | --- |
-| `SKILLEVALUATOR_RESULTS_DIR` | `<skill>/evals/results` | External root for run results. Precedence for writes: the `--results-dir` flag, then this variable, then the legacy in-skill location. Read commands (`view`, `compare`) honor the same order and also fall back to the legacy location so older runs stay visible. |
+| `SKILLEVALUATOR_RESULTS_DIR` | `<skill>/evals/results` | External root for run results. Precedence for writes: the `--results-dir` flag, then this variable, then the canonical in-skill location. Configured roots must remain outside the skill; `evals/results` is the only supported in-skill results root so generated output cannot change the skill's publication identity. Read commands (`view`, `compare`) honor the same order and also fall back to the legacy location so older runs stay visible. |
 | `SKILLEVALUATOR_LOCAL_SANDBOX` | `require` | Local-mode sandbox policy. `require` fails closed when no OS sandbox backend (Bubblewrap on Linux, Seatbelt on macOS) is usable; `prefer` degrades to advisory-only guardrails with a loud warning; `off` skips sandbox probing entirely — for skills you fully trust. No value enables native Windows: local mode fails closed there before anything runs — use WSL2 or `--env-mode docker`. |
 | `SKILLEVALUATOR_LOCAL_ALLOW_NET` | `true` | Network egress for local-mode trials. Set to `0` to airgap a skill that must not reach the network. Incompatible with the `nv_build` provider — NVIDIA Build local agents require network access, so airgapped runs are rejected up front. |
 | `SKILLEVALUATOR_LOCAL_STRICT_READS` | `false` | Tightens the sandbox's read-only view of the host system to a stricter path set. |