Maintainer Eval Harness

Regression tests for Builder Loops maintainer behavior.

The goal is to make maintainer prompt and policy changes measurable before they reach the live GitHub App. The harness keeps a small set of labeled historical cases and scores whether a candidate planner or reviewer would make the right product-management decision.

What This Tests

• Planner behavior: chooses the right next active step, does not repeat bad targets, replans prerequisites after repeated failures.
• Reviewer behavior: rejects fake completion, demo-only paths, mock outputs, and PRs that pass tests while leaving the real product workflow broken.
• Controller behavior: preserves one active issue, avoids stale duplicates, and treats failure reports as planning evidence.

Run

npm install
npm run check

Case Format

Cases live in fixtures/cases/*.json.

Each case has:

• kind: planner, reviewer, or controller
• input: compact scenario data
• expected: labels and must-match expectations
• rubric: human-readable scoring notes

The current evaluator is deterministic and rule-based. That is intentional for v0: it lets us preserve known failures cheaply. The next layer should run incumbent and challenger LLM prompts against the same cases and compare scores before deployment.

Builder Loops Policy

The maintainer should behave like a hands-on product lead supervising a build loop:

• One active product goal at a time.
• Acceptance must be visible through the normal product workflow.
• Demo-only, mock-only, test-only, or probe-specific completion does not count.
• Repeated PR failures should produce a failure report and a prerequisite step, not endless close cycles.
• Old failure counters should be tied to an evaluator version so stale judgments do not poison a new evaluator.