Make agent evaluation evidence filterable in a verified HF casebook - #4
Merged
Merged
Conversation
noteflowai
added a commit
that referenced
this pull request
Sep 15, 2026
Carried out on the maintainer's explicit instruction: issue #4 on the hack-verifiable environments repository, and EvalArc's post in a finding-led X thread. The note keeps the distinction sharp instead of implying an equivalence, and says twice that this does not measure reward hacking. What it offers the authors is a number on the assumption their measurement rests on: an exploitation rate is meaningful because a planted hack is detectable by construction, and here six of twenty-one planted faults are detectable through exactly one check each, at a mutation score of 1.0. In their terms those six would become undetected hacks if the single check weakened. It asks a question back rather than only presenting, namely whether a planted hack's own detection margin belongs beside the hacking rate, and it volunteers the error that produced a wrong number first, since another implementer would hit it: counting case runs rather than distinct cases makes every margin double per added seed. Numbers re-verified against current main before submission, not copied from the earlier session. No link, citation or mention requested, and the issue offers to move rather than duplicate if a discussion thread suits the authors better.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The evidence lab previously required visitors to inspect nested reports. Add a Hugging Face casebook with three separately loadable development configurations: 167 audit cases, six repeated attempts and three suite jobs. The default table exposes the same 93.75% score with different acceptance decisions, while preserving full resolution separately.
The exporter validates source evaluations, recomputes repetitions and suite gates, checks JUnit, and preserves unchanged evidence files with hashes and exact JSON Pointers. The card explains row units, historical grader versions, synthetic fixtures and interpretation limits. Both READMEs, the Space and the site link the dataset.
CI retains a checked dataset artifact alongside the site. The existing guarded publication job publishes only after tests, Docker audits and browser checks pass, uses the expected Hub parent commit and reads every uploaded file back without credentials. An existing unrelated dataset is rejected.
Validation: 177 Python tests; Ruff lint and formatting; actual
datasetsloading of all three local configurations and gate assertions; validated HF dataset card; browser checks at 1440 and 390 px. This publishes existing scripted development evidence; it adds no model benchmark results or CLI version bump.