Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 47 additions & 0 deletions DESIGN.md
Original file line number Diff line number Diff line change
Expand Up @@ -400,6 +400,53 @@ draft of this section:
tier ladder preserves this difference and the receipt never states more
than the probe demonstrated.

## The policy layer (schema 0.0.11): evidence floors per path

The end state's second commitment, delivered: `correctful.json` at the repo
root declares rules — paths plus a floor (`min_tier`, optionally a required
`mechanism` and a required `scope`). Evaluation is per changed file, and
the tie between a file and its evidence is STRUCTURAL, never assumed: a
verified claim speaks for a file only when the claim was sourced from it
(LLM claims with confirmed edges, spec-ids in code) or a reference site in
shipped code names it. A matched file nothing demonstrably ties evidence
to is a miss stated exactly that way — which makes floors honest and also
scopes where they are USEFUL: repos that annotate code with claim ids, or
run the LLM extractor. A floor on unannotated code fails, and should.

Design decisions worth recording:

- **Misses block the gate**, same as refutations — a floor that only
informs is not a floor. The exit-gate line in the receipt says which legs
block.
- **Test files are exempt and the exemption is counted** — a `_test.go` is
evidence, not an evidence subject. Silent exemption would be a coverage
lie; the receipt shows the count.
- **The policy digest is the second chain field** (after the tool
version): SHA-256 over the policy file's exact bytes, rendered short
beside the change. A policy change — the trust base changing — is
visible in the receipt chain, which is the review trigger the end
state's first commitment asks for.
- **A malformed policy fails loudly before any probe runs.** A broken
floor must never fail open; a missing file simply means no policy.
- **The LLM edge gate applies identically here** — `Evidence.CountsFor`
moved to the schema so weighing and policy evaluation share one
definition and can never disagree. A pass on an unconfirmed
model-proposed edge satisfies no floor.
- **Each rule stands alone**: a file matched by two rules must satisfy
both floors; the miss row names the violated rule and the best tied
evidence, so the reader sees the gap, not just the verdict.

Measured (first live run, 2026-08, on a real annotated repo's feature
branch): a T1 floor over the changed command directory matched 26 files —
8 exempt as test files (disclosed), 18 evaluated, ALL 18 missed with "no
verified claim ties to this file", and the gate blocked. Correctly: the
branch's 153 verified claims are all test-name-sourced, and none of the
changed code files carries a reconciled claim id, so no verified evidence
structurally speaks for them. The strictness is the finding — a floor
demands the tie discipline (id-annotated code reconciled with id-named
tests, or LLM extraction with confirmed edges), and states exactly what is
missing when a repo has not adopted it.

## Known limitations (found by dogfooding, stated honestly)

correctful was run on itself and on a real 101-file production change on its
Expand Down
38 changes: 37 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,10 +99,46 @@ shows the reference configuration:
- The workflow writes the receipt as a comment on the pull request. The
marker `<!-- correctful-receipt -->` identifies the comment. The workflow
updates the same comment after each push.
- The gate fails only when a probe refuted a claim.
- The gate fails when a probe refuted a claim, or when the change missed a
declared policy floor.

This repository uses this gate for each of its own pull requests.

## Policy floors (optional)

You can declare evidence floors for the paths that matter most. Write a
`correctful.json` file in the repository root:

```json
{
"policy_version": 1,
"rules": [
{
"name": "auth-floor",
"paths": ["internal/auth/..."],
"min_tier": 2,
"mechanism": "go-test-pair"
}
]
}
```

The rule reads: each changed file under `internal/auth/` must have one
verified claim that connects to that file, at tier T2 or higher, from an
accept/reject test pair. A rule can also demand a measured execution scope
(`"scope": "cross-package"`).

- A connection is structural. The claim's source file, or a reference site
in the code, must name the changed file. The tool does not guess.
- Test files are exempt. They supply evidence; they do not need evidence.
The receipt shows the count of exempt files.
- A missed floor blocks the gate, in the same way as a refuted claim. The
receipt shows each miss with the best found evidence and the floor.
- The receipt shows the SHA-256 digest of the policy file. A policy change
is visible in the receipt chain.
- No policy file means no policy. A malformed policy file stops the run
with an error. It never fails open.

## What correctful examines

| You write | correctful harvests | The probe | Tier |
Expand Down
20 changes: 16 additions & 4 deletions cmd/correctful/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,9 @@
// -concurrency max probes to run at once. Default: 4.
// -timeout overall probe budget. Default: 5m.
//
// Exit status: 0 when no claim was refuted; 1 when a probe ran and a claim did
// not hold (merge-gate semantics). The remainder never fails the run — it is an
// honest report, not a defect.
// Exit status: 0 when no claim was refuted and every declared policy floor was
// met; 1 on a refutation or a policy miss (merge-gate semantics). The
// remainder never fails the run — it is an honest report, not a defect.
package main

import (
Expand All @@ -32,6 +32,7 @@ import (
"github.com/joshft/correctful/internal/gitdiff"
"github.com/joshft/correctful/internal/harvest"
"github.com/joshft/correctful/internal/llmextract"
"github.com/joshft/correctful/internal/policy"
"github.com/joshft/correctful/internal/probe"
"github.com/joshft/correctful/internal/receipt"
)
Expand Down Expand Up @@ -94,6 +95,14 @@ func run(base, repo, format string, concurrency int, timeout time.Duration, useL
// a mid-branch receipt harvests the working tree.
change.InputDigest = gitdiff.InputDigest(root, change.Files)

// Load the policy BEFORE any probe runs: a malformed policy fails loudly
// here (a broken floor must never fail open), and a missing file simply
// means no policy.
pol, err := policy.Load(root)
if err != nil {
return err
}

// Harvest claims, then dispatch probes against them.
harvesters := harvest.Default()
if useLLM {
Expand Down Expand Up @@ -130,6 +139,9 @@ func run(base, repo, format string, concurrency int, timeout time.Duration, useL
Dispatch(ctx, root, claims)

r := receipt.Assemble(change, claims, evidence, coverage)
if pol != nil {
r.Policy = policy.Evaluate(pol, r)
}

switch format {
case "json":
Expand All @@ -142,7 +154,7 @@ func run(base, repo, format string, concurrency int, timeout time.Duration, useL
receipt.WriteText(os.Stdout, r)
}

if r.Summary.Refuted > 0 {
if r.Summary.Refuted > 0 || (r.Policy != nil && len(r.Policy.Misses) > 0) {
os.Exit(1)
}
return nil
Expand Down
Loading
Loading