Skip to content

Make evaluation trigger and baseline gates explicit - #139

Merged
dfrysinger merged 6 commits into
mainfrom
dfrysinger/evaluation-trigger-contract
Nov 10, 2026
Merged

dfrysinger merged 6 commits into
mainfrom
dfrysinger/evaluation-trigger-contract

Conversation

@dfrysinger

@dfrysinger dfrysinger commented Nov 6, 2026 •

Copy link
Copy Markdown
Owner

Change

Put skill evaluation on one ordered path: declare the real trigger and
observable outcome, establish the unchanged baseline, compare the declared
change, and report only what the evidence supports.

New negative examples default to preventing the failure before human
intervention. Explicit response-only requests select that narrower goal. When
prevention is unavailable, propose the least-intervention alternative and
agree before replacing the goal; continue independent relevant preparation.

Reuse applicable operational proof when only skill bytes or binding identities
change. Verify bindings mechanically and observe instruction exposure. Expanded
skill visibility, forced loads and context-limit changes remain separate
runtime interventions. Keep method reviews focused on material changes rather
than restarting them for mechanical repairs with unchanged inputs and meaning.

Historical repairs use capture-time skills and context as their primary
control. The baseline must reproduce the specific bad agent behavior before
the treatment phase opens. A synthetic fixture, failed setup or response-only
proposal cannot substitute for historical executed recovery.

Recover historical skills from Git at the selected replay cutoff, check them
against recorded skill loads, and freeze the full repository. Keep missing
host or task state separate from recoverable repo files. If the proposed
skill package is unchanged, establish the baseline rather than running a
same-byte improvement comparison.

The playbook covers scheduled wakes, objective or schedule creation, source
production, user requests and drift during ongoing work. Preserve authentic
prompt wording and skill names; added diagnosis or loading requests change
the intervention.

Separate clean replacement from recovery after an update. The latter can keep
identical old loaded history in both arms while changing future available
skills. Reloading availability does not prove automatic invocation or removal
of earlier instructions.

Require pre-run independent Claude and GPT method review using five scored
rules. Both families must grade a phase READY; missing prerequisites and
disagreements keep it closed. The reusable reviewer addendum grades the user's
actual question, not an author's narrower substitute.

Dashboard setup is removed from the normal path. The report template leads
with the question, actual work, controls and supported conclusion. Unrun
comparisons remain NOT RUN.

Scope and evidence

Instructions and references only. No evaluator implementation, permission,
dashboard, plugin-version or installed-package change.

Both independent source-review families completed the focused rule/rubric
review and verified the required fixes. Skill shape, plugin manifests,
relative links/anchors, the existing no-enqueue guard and whitespace checks
pass. The restoration recipe's shell commands reproduce the verified
historical repository archives and preserve an existing archive.

Source review does not prove improved future instruction following. Historical
causal validation remains unexecuted; prior synthetic observations are
supplemental diagnostics, not a substitute.

Fake Copilot and others added 3 commits November 5, 2026 17:30
Require the unchanged agent to reproduce a reported failure before a repair comparison. Separate scheduled recovery, upstream artifact production and long-session observation, and report missing controls explicitly. Keep one ordered procedure and remove dashboard setup from the evaluation path.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: cf89240b-a65d-414a-9123-5c1dc9c40c7d
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: cf89240b-a65d-414a-9123-5c1dc9c40c7d
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: cf89240b-a65d-414a-9123-5c1dc9c40c7d
Fake Copilot and others added 3 commits November 8, 2026 21:15
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: cf89240b-a65d-414a-9123-5c1dc9c40c7d
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: cf89240b-a65d-414a-9123-5c1dc9c40c7d
Preserve explicit response-only requests and propose the least-intervention alternative when prevention is unavailable. Reuse applicable operational proof and distinguish runtime visibility changes from skill effects.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 6f7726ad-c851-4da9-bd16-ffbd015dd0f1
@dfrysinger
dfrysinger merged commit f58eb28 into main Nov 10, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant