Repository navigation
[TK-001] Make QC/report coverage, exit semantics, and finding lifecycle authoritative #1
Description
Priority: P0
Labels: bug, correctness, reporting, release-blocker, table-kit
Confidence: 100%
Evidence
- Documented
tablekit qcdoes not recordqc.runwithout a secondary flag. - Report “NOT CHECKED” exits 0.
- One early QC run can bless 77 later unchecked beats as “Defects — none,” exit 0.
- Detector exceptions are swallowed; an old run can support a clean report.
- Minimum-sample handling can hide a confirmed defect.
- JSON-valid but schema-invalid ledger rows are not validated on read and can affect denominators, crash consumers, or be misinterpreted; syntactically malformed rows themselves are excluded by
Ledger.beats().
Scope / specification
Adopt EvaluationResult v1. Each QC run records immutable evaluator inventory/version/config, input digest, event range, compatible/eligible/evaluated counts, skipped/errors, and checked-through event ID. Report reconciles the current ledger with those runs and cannot infer coverage from mere presence of qc.run.
Acceptance criteria
- The documented QC command records coverage or docs/command are changed unambiguously.
NOT CHECKED, stale/post-run events, gaps, detector errors, syntactically malformed or schema-invalid rows, or incomplete eligibility return non-authoritative status/exit 2.- Confirmed findings are always surfaced even below advisory minimum sample.
- Minimum sample affects confidence/aggregate inference, not visibility of observed defects.
- Report lists unchecked ranges, failed/disabled detectors, and exact checked-through cursor.
- “Defects — none” appears only when categorical-defect coverage is authoritative; any non-blocking attention signals are separately and prominently labeled.
- QC and report apply one explicit advisory exit contract; an attention-only session produces parity between commands instead of QC exit 1 versus report exit 0.
- Findings use immutable IDs independent of mutable
detailand haveopen | resolved | supersededtransitions; reports distinguish current from historical findings and never infer “still open” from repeated records alone.
Required tests: zero QC, early QC + later beats, detector exception, syntactically malformed and JSON-valid/schema-invalid rows, finding below minimum, attention-only QC/report parity, finding on run N then absent on N+1, mutable detail with stable ID, partial evaluator set, config/tool drift, exact exit/status snapshots.
Dependencies: repo-local stale-QC/detector/sample-floor/exit/lifecycle patches have no portfolio dependency and land immediately; migrate to PORT-001 afterward.
Activity
chaoz23 commentedon Aug 3, 2026
Status reconciliation (2026-08-03): keep open; substantial local QC is merged, but authority is not complete.
PR #10 delivered fail-closed QC, exact snapshot coverage, stale/partial detection, stable finding lifecycle, and typed clean/advisory/incomplete/error states. PRs #11–#15 added the shared result contract and installed four-tool conformance.
Remaining: an independent host must own the protected descriptor, canonical cursor, evaluator invocation, and atomic result commit; observed-table evidence must establish precision, false negatives, DM workload, and gap recovery. Implementation is specified in #17 and the evidence gate in #18. Close only after host-attested shadow operation and pilot evidence satisfy the original authoritative-report intent.
chaoz23 commentedon Aug 18, 2026
Audit evidence for the "exit semantics" half of this issue.
SKILL.md:33-37states the exit-2 lane incompletelyWhat it does state is true —
0= clean and1= findings both verified forbeat,qc, andreport, and its two exit-2 examples (init: file exists;consumed: id not open) are accurate.The defect is omission. It reduces exit 2 to "can't do that" and illustrates it only with usage errors, never documenting the genuine honest lane that
qcandreportactually emit:tablekit/detector.py:614-619— exit 2 for incomplete evaluator coveragetablekit/report.py:496-503— exit 2 for insufficient data / incomplete coverageVerified:
reporton an empty ledger → exit 2;qcwithout a config → exit 2;qc/reporton a clean covered session → exit 0.So an agent reading
SKILL.mdtreats "coverage was incomplete, I cannot adjudicate this session" as "you called me wrong" — which is exactly backwards, and it is the casereporthits most often in practice.Two adjacent
SKILL.mddefects found in the same passreportnever appears as an invocation anywhere in the file.grep -n 'report' SKILL.md→ lines 9, 31, 56 only, none of them a command. The frontmatter promises it as the primary close-out verb.Related, filed separately:
tool.jsondeclares 16 subcommands against 18 registered, omittingverdictandpark.Evidence from a conformance audit of the check-family four against
FAMILY.mdv1, 2026-08-17. Commenting here rather than opening a duplicate, since this issue already owns the area.ci: run the family conformance gate on every PR (#28)
chore: drop both HONEST_LANE_OVERLOAD waivers (#88)