Skip to content

chore(corpus): CHAOS-5721 Rescore six refuse rows as any_of under a corpus version bump - #76

Merged
chrisgeo merged 1 commit into
mainfrom
chore/5721-corpus-anyof-rescore
Sep 13, 2026
Merged

chrisgeo merged 1 commit into
mainfrom
chore/5721-corpus-anyof-rescore

Conversation

@chrisgeo

@chrisgeo chrisgeo commented Sep 13, 2026

Copy link
Copy Markdown
Contributor

Ticket: CHAOS-5721

TEST-EVIDENCE

Scope: data-only change to corpus/corpus.py (7 rows' expect), corpus/test_corpus.py (pin), corpus/README.md (version history). No scorer/engine code touched.

Where the corpus version lives: this corpus has no separate numeric version constant — the corpus version IS this file's git commit sha (git show <sha>:corpus/corpus.py), which is what a scoring run's corpus_version provenance field records (corpus/semantic_verdict.py:build_verdict). This PR's own commit sha is the bump; corpus/README.md's new version-history section says so explicitly rather than implying a constant changed.

BASELINE GUARD disclosure (every row this PR changes; row expectations are chris's data-semantics call):

row id old expect new expect bar movement
basis-discovered-incidents refuse any_of{serve(family=discovered_cohort_ranking), refuse} LOOSENS: a served answer, which always scored disagree under the old scalar refuse, can now score agree/agree_weak via the added serve branch; a refusal still matches (unchanged).
basis-discovered-repo-count refuse any_of{serve(family=discovered_cohort_ranking), refuse} LOOSENS, same shape as above.
cv-org-count-projects refuse any_of{serve(family=discovered_cohort_ranking), refuse} LOOSENS, same shape as above.
basis-grouped-metric-by-repo refuse any_of{serve(family=grouped_cohort_status), refuse} LOOSENS, same shape as above.
basis-grouped-pr-by-project refuse any_of{serve(family=grouped_cohort_status), refuse} LOOSENS, same shape as above.
pos-grouped-per-phrasing refuse any_of{serve(family=grouped_cohort_status), refuse} LOOSENS, same shape as above.
cv-scoped-projects-by-team-bounded any_of{serve(family=scoped_cohort_status), refuse} any_of{serve(family=scoped_cohort_status)} TIGHTENS: a refusal (previously agreeing via the dropped branch, including the observed role-blind declared-kind terminal error) no longer matches any alternative.

Six rows widen acceptance to admit a serve branch that a prior servable-kind gap made unreachable; one row narrows acceptance by removing a refuse branch that had certified a terminal-role error as a match. Old scoring series stays in git history (git show <sha>:corpus/corpus.py), never overwritten; corpus/README.md gets a new ## Version history entry naming both moves. No other row's expect changed.

Red-first (pin fails at origin/main, a9a72b1e): copied origin/main's corpus/corpus.py into this branch's corpus/test_corpus.py and ran it —

$ python3 corpus/test_corpus.py
CorpusValidationError: refuse/decline row set changed since the pinned scalar tranches: added=['basis-discovered-incidents', 'basis-discovered-repo-count', 'basis-grouped-metric-by-repo', 'basis-grouped-pr-by-project', 'cv-org-count-projects', 'pos-grouped-per-phrasing'], removed=[]

(literal message elides the pin dict's own ticket-id substring per this body's one-ticket-token rule; otherwise verbatim.)

Green at this PR's tip (5758d48), fresh pnpm install first (ajv needed by scripts/validate_json_schema.mjs):

$ pnpm install --frozen-lockfile
Done in 1.9s using pnpm v11.21.0

$ python3 corpus/test_corpus.py
PASS: 36 corpus rows validated
PASS: 7 first-tranche + 4 D24 refuse/decline expect values pinned and matched
PASS: 7 this-PR any_of row branch sets pinned and matched
PASS: every corpus row scored except ['cv-named-project-completion', 'neg-single-subject-why']
PASS: baseline sha256 pin matched, 36 baseline rows shape-checked (0 historical-only, not in the current corpus)

PINNED_5721_ANY_OF pins all 7 rows by branch SHAPE (outcome + serve family), never by question text; its own red controls (drifted branch, reverted-to-scalar, deleted row) and green control run inside the same file.

Full pnpm test:corpus (schema validation, semantic_verdict, schema-driven shape generation): rc=0, all PASS lines, no FAIL.

Full ask-dev quality gate, all rc=0:

$ pnpm exec prettier --check .   -> All matched files use Prettier code style!
$ pnpm run lint                  -> eslint . (clean)
$ pnpm run typecheck             -> tsc --noEmit (clean)
$ pnpm run test:unit             -> Test Files 60 passed (60), Tests 977 passed (977)
$ pnpm run build                 -> next build: Compiled successfully, 4/4 pages generated

CI at this PR's tip (5758d48, PR #76): all 12 checks pass — Analyze (actions/javascript-typescript/python), Build, CodeQL, Dependency Audit, E2E smoke, Gitleaks Secret Scan, Quality, Unit tests, run_checks.sh self-test, test.

PR-ref CodeQL at 5758d48, results_count=0 on every analysis:

$ gh api "repos/full-chaos/ask-dev/code-scanning/analyses?ref=refs/pull/76/head"
/language:javascript-typescript  results_count=0
/language:python                 results_count=0
/language:actions                results_count=0

No codex round: data/expectation PR, no scorer code touched.

RISK-NOTES

  • Pure data/expectation change: no product, scorer, or engine code touched; expect_schema.py's SCHEMA_VERSION is unchanged (branch shape itself did not change, only which rows use it).
  • The rescoring run against these seven rows' new expect is ordered separately, not part of this PR.
  • A related in-flight PR changes corpus/semantic_verdict.py only (scoring any_of serve branches from persisted semantic state); this PR does not depend on it and does not touch that file.
  • cv-scoped-projects-by-team-bounded's narrowed acceptance means a future rep that still hits the role-blind terminal error will now score as a disagreement instead of silently agreeing — expected and intended by this PR, not a regression to chase.

…on bump

Six rows whose scalar refuse had been overtaken by the servable-kind
admissions rescore to any_of {serve with the row's own family, refuse}.
cv-scoped-projects-by-team-bounded's any_of drops its refuse alternative
now that the role-aware declared-kind terminal landed (it had certified
a role-blind terminal error as a match). Old scoring series kept as
history per corpus/README.md ## Versioning; version-history entry added.
Adds a branch-shape pin (PINNED_5721_ANY_OF) for all seven rows.
@linear-code

linear-code Bot commented Sep 13, 2026

Copy link
Copy Markdown

CHAOS-5721

@chrisgeo
chrisgeo merged commit dbf936d into main Sep 13, 2026
13 checks passed
@chrisgeo
chrisgeo deleted the chore/5721-corpus-anyof-rescore branch September 13, 2026 22:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant