[PORT-001][RFC][P0] Define EvaluationResult v1 and fail closed when coverage is incomplete

Decision requested

Define a shared table.evaluation/1.0 result envelope that proves whether evaluation coverage is complete and fails closed when it is not. This RFC defines a contract and links repo-local containment work; it does not commit all four sibling packages to immediate migration.

Priority: P0 — blocks trustworthy agent/live use.

Evidence confidence: 99%. The precise shared design remains an RFC decision.

Problem / evidence

Process success is currently confused with evidentiary success:

dmcheck can return clean for empty or zero-effective-GM input.
table-kit can report “Defects — none” from an old QC run while later beats remain unchecked.
table-kit “NOT CHECKED” can exit 0, and detector exceptions can disappear.
Passing the current table-kit ledger directly to dmcheck yields zero compatible messages and can return clean.

Existing CharacterCheck and srdcheck result contracts are design inputs and constraints, not an assumption that those packages must immediately adopt this envelope.

Proposed contract

The envelope should include:

tool/version and schema version;
session and evaluation IDs;
status: checked_clean | checked_with_advisories | findings | incomplete | unsupported | invalid | internal_error;
authority_status: self_attested | host_attested or an equivalent explicit field;
input, compatible, eligible, evaluated, and skipped counts globally and per evaluator/obligation class;
structured skip reasons;
checked-through event ID, gap state, and input digest;
roster, policy, config, and source-set digests;
typed findings, advisories, warnings, and errors.

Exit 0 means checked_clean only. Findings or advisories worth operator attention use a nonzero success/finding convention that remains consistent with the documented family contract; any non-authoritative result exits 2. Agents consume the typed status, not prose or exit code alone.

Acceptance criteria

Publish versioned JSON Schema and generated Python/TypeScript types.
Every evaluation/report path, including MCP where exposed, returns the same envelope. Non-evaluation commands such as init/schema/rules are out of scope.
Empty input, zero compatible rows, zero eligible rows where evidence is required, transport gaps, detector exceptions, stale QC, invalid input, and incomplete per-evaluator coverage cannot return checked_clean.
Coverage identifies every disabled or skipped evaluator and its reason.
Coverage proves all required eligible obligations were adjudicated; a global count cannot hide evaluator-specific gaps.
Every finding carries the effective machine-readable policy subset, policy/charter version, and digest.
Reports visibly distinguish clean, advisories, findings, incomplete, unsupported, and failed.
Self-attested results cannot be represented as host-authoritative.
DMC-001 and TK-001 can land repo-local containment first and later adapt to this contract.

Required tests

Contract fixture for every status and authority state.
Property test: exit 0 implies checked_clean, no gaps/errors, complete required coverage, and an honestly labeled authority state.
Cross-language schema round trip.
Golden zero-input, incompatible-input, stale-QC, and detector-crash cases.
Installed-artifact parity for every exposed evaluation/report surface.

Dependencies

Schema design can start immediately. host_attested and any claim of independent authority cannot close until the authority-boundary RFC is resolved.

Suggested labels: P0, RFC, cross-repo, architecture, agent-contract, correctness, diligence, blocks-live-use.

Scope note: This is a cross-repository architecture proposal hosted temporarily in table-kit because this repository currently owns the session ledger and documents sibling-tool compatibility. It does not assert that table-kit already provides a suite host/control plane, or that dmcheck, charactercheck, and srdcheck currently consume a common event protocol. Repo-local containment work should proceed independently.

Foundation is implemented in draft PR #11: the packaged table.evaluation/1.0 schema, all status/authority fixtures, generated Python/TypeScript declarations, cross-field fail-closed validators, CLI discovery, and artifact CI. Keep this RFC open for native dmcheck/table-kit projections, remaining sibling-tool joins, and installed cross-tool conformance.

Implementation evidence: chaoz23/dmcheck#9 now projects dmcheck's native
TableEvent result into the table.evaluation/1.0 contract proposed in #11.

The draft PR is stacked on dmcheck#8 and is hosted-green on Python 3.10 and
3.14 at 17e3a42cbca9182727a168c795f72c4a7b64eac1. It covers deterministic
identity, canonical input digest, aggregate/per-rule coverage, cursor state,
exact event evidence references, typed status/exit mapping, and fail-closed
handling when evidence cannot support a public finding. Output is deliberately
limited to self_attested; protected-host attestation remains external.

Cross-schema checks exercised checked_clean, findings, incomplete,
unsupported, and invalid outputs against table-kit's actual JSON Schema
and semantic validator. No release or merge is implied.

Hosted cross-repository evidence is now implemented in green draft
#13 at
cefc204ec018c3254a53899b56250ae5b2cf3b30.

The job builds the table-kit candidate and the exact dmcheck#9 commit as
wheels, cold-installs both outside either checkout, and validates dmcheck's
shared outputs with table-kit's installed JSON Schema and semantic validator.
It covers checked_clean, exact-evidence findings, and fail-closed
incomplete outcomes. The new hosted job passes alongside Python 3.9/3.12 and
Node. Output remains self_attested; no host authority or release is implied.

The first sibling evaluator join is now hosted-green.

feat: project rules verdicts to table.evaluation/1.0 srdcheck#60 at
94f87fc54b011ce9048a13aefa96f05d8708711a projects scoped native verdicts
into deterministic self-attested table.evaluation/1.0 envelopes.
ci: enforce installed srdcheck result conformance #14 at
fd69cb2bd5f5128e82c6149df78422ee50993e45 builds and cold-installs the exact
srdcheck candidate with table-kit, outside both checkouts, and validates six
cases with the installed JSON Schema and semantic validator.

The cases cover checked-clean, a failed-but-completely-adjudicated saving throw,
an exact-rule-evidence illegal action, missing facts, unsupported content, and
invalid input. Both PRs are draft, clean, mergeable, and hosted-green. Rules
results remain advisory and self_attested; no release or host authority is
implied.

The first sibling evaluator join is now hosted-green.

feat: project rules verdicts to table.evaluation/1.0 srdcheck#60 at
21a43d8c2a6791465be9e0fb9856ce05888292d0 projects scoped native verdicts
into deterministic self-attested table.evaluation/1.0 envelopes.
ci: enforce installed srdcheck result conformance #14 at
7cf2d0f51b9248bb050d88b56eee7c88816a1135 builds and cold-installs the exact
srdcheck candidate with table-kit, outside both checkouts, and validates six
cases with the installed JSON Schema and semantic validator.

The cases cover checked-clean, a failed-but-completely-adjudicated saving throw,
an exact-rule-evidence illegal action, missing facts, unsupported content, and
invalid input. Both PRs are draft, clean, mergeable, and hosted-green. Rules
results remain advisory and self_attested; no release or host authority is
implied. This supersedes the earlier head references after srdcheck #59 and #61
merged and the two draft branches were rebased/pinned to current history.

The first four-tool installed contract gate is now hosted-green.

feat: project character trust to table.evaluation/1.0 charactercheck#32 at
e6ef29f5beb8f7bae972865e724988c932ebf708 projects canonical character field
assessments into deterministic, value-free, self-attested
table.evaluation/1.0 envelopes.
ci: enforce installed charactercheck result conformance #15 at
79434c20c2e2bcdcc4540e86c1ed3ce57bf817f4 builds and cold-installs that
exact candidate alongside the table-kit schema. The same hosted workflow
also keeps the exact dmcheck and srdcheck installed gates green.

The CharacterCheck cases cover unsupported mechanics, complete static coverage
with player-authority advisories, unknown coverage, invalid mechanics,
privacy-safe invalid references, and deterministic replay. Character values,
names, and refs are not copied into the shared envelope. All four producer PRs
remain draft/self-attested and unreleased; the protected suite host is still a
separate PORT-003 deliverable.

The first four-tool installed contract gate is now hosted-green.

feat: project character trust to table.evaluation/1.0 charactercheck#32 at
e6ef29f5beb8f7bae972865e724988c932ebf708 projects canonical character field
assessments into deterministic, value-free, self-attested
table.evaluation/1.0 envelopes.
ci: enforce installed charactercheck result conformance #15 at
4da56f44b0dfcb8f6652e5b29c4f3f23b20eeaf2 builds and cold-installs that
exact candidate alongside the table-kit schema. The same hosted workflow
also keeps the exact dmcheck and current srdcheck #60 commit
4bb87ee31f69f0dd70f6ceb176392c97b3147f33 installed gates green.

The CharacterCheck cases cover unsupported mechanics, complete static coverage
with player-authority advisories, unknown coverage, invalid mechanics,
privacy-safe invalid references, and deterministic replay. Character values,
names, and refs are not copied into the shared envelope. All four producer PRs
remain draft/self-attested and unreleased; the protected suite host is still a
separate PORT-003 deliverable.

This exact-head update supersedes the earlier #15 head after the concurrent
srdcheck #60 improvement was pinned and the stack rebased.

The first four-tool installed contract gate is now hosted-green.

feat: project character trust to table.evaluation/1.0 charactercheck#32 at
e6ef29f5beb8f7bae972865e724988c932ebf708 projects canonical character field
assessments into deterministic, value-free, self-attested
table.evaluation/1.0 envelopes.
ci: enforce installed charactercheck result conformance #15 at
283fbfe626c891d847b84158fd1e6cb8d5b2d490 builds and cold-installs that
exact candidate alongside the table-kit schema. The same hosted workflow
also keeps the exact dmcheck and current srdcheck #60 commit
4bb87ee31f69f0dd70f6ceb176392c97b3147f33 installed gates green.

The CharacterCheck cases cover unsupported mechanics, complete static coverage
with player-authority advisories, unknown coverage, invalid mechanics,
privacy-safe invalid references, and deterministic replay. Character values,
names, and refs are not copied into the shared envelope. All four producer PRs
remain draft/self-attested and unreleased; the protected suite host is still a
separate PORT-003 deliverable.

This exact-head update supersedes the earlier #15 head after the concurrent
srdcheck #60 improvement was pinned and the stack rebased.

Status reconciliation (2026-08-03): keep open; the contract and first four-tool adoption gate are complete, host-attested operation is not.

PR #11 delivered schema, semantic validation, fixtures, generated types, and authority/coverage rules. dmcheck #8–#9 and table-kit #13–#15 deliver installed dmcheck, SRDCheck, and CharacterCheck conformance.

The chain correctly remains self-attested. Remaining: independent host implementation, full intended surface adoption, adversarial attestation tests, and observed-table evidence. Tracked by #17 and #18; schema adoption alone is not closure.

Navigation Menu

Repository navigation

[PORT-001][RFC][P0] Define EvaluationResult v1 and fail closed when coverage is incomplete #7

Decision requested

Problem / evidence

Proposed contract

Acceptance criteria

Required tests

Dependencies

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

[PORT-001][RFC][P0] Define EvaluationResult v1 and fail closed when coverage is incomplete #7

Description

Decision requested

Problem / evidence

Proposed contract

Acceptance criteria

Required tests

Dependencies

Activity

chaoz23 commented on Aug 2, 2026

chaoz23 commented on Aug 2, 2026

chaoz23 commented on Aug 2, 2026

chaoz23 commented on Aug 2, 2026

chaoz23 commented on Aug 2, 2026

chaoz23 commented on Aug 2, 2026

chaoz23 commented on Aug 2, 2026

chaoz23 commented on Aug 2, 2026

chaoz23 commented on Aug 3, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions