Repository navigation
Make evaluation trigger and baseline gates explicit - #139
Merged
Merged
Conversation
Require the unchanged agent to reproduce a reported failure before a repair comparison. Separate scheduled recovery, upstream artifact production and long-session observation, and report missing controls explicitly. Keep one ordered procedure and remove dashboard setup from the evaluation path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: cf89240b-a65d-414a-9123-5c1dc9c40c7d
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: cf89240b-a65d-414a-9123-5c1dc9c40c7d
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: cf89240b-a65d-414a-9123-5c1dc9c40c7d
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: cf89240b-a65d-414a-9123-5c1dc9c40c7d
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: cf89240b-a65d-414a-9123-5c1dc9c40c7d
Preserve explicit response-only requests and propose the least-intervention alternative when prevention is unavailable. Reuse applicable operational proof and distinguish runtime visibility changes from skill effects. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6f7726ad-c851-4da9-bd16-ffbd015dd0f1
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Change
Put skill evaluation on one ordered path: declare the real trigger and
observable outcome, establish the unchanged baseline, compare the declared
change, and report only what the evidence supports.
New negative examples default to preventing the failure before human
intervention. Explicit response-only requests select that narrower goal. When
prevention is unavailable, propose the least-intervention alternative and
agree before replacing the goal; continue independent relevant preparation.
Reuse applicable operational proof when only skill bytes or binding identities
change. Verify bindings mechanically and observe instruction exposure. Expanded
skill visibility, forced loads and context-limit changes remain separate
runtime interventions. Keep method reviews focused on material changes rather
than restarting them for mechanical repairs with unchanged inputs and meaning.
Historical repairs use capture-time skills and context as their primary
control. The baseline must reproduce the specific bad agent behavior before
the treatment phase opens. A synthetic fixture, failed setup or response-only
proposal cannot substitute for historical executed recovery.
Recover historical skills from Git at the selected replay cutoff, check them
against recorded skill loads, and freeze the full repository. Keep missing
host or task state separate from recoverable repo files. If the proposed
skill package is unchanged, establish the baseline rather than running a
same-byte improvement comparison.
The playbook covers scheduled wakes, objective or schedule creation, source
production, user requests and drift during ongoing work. Preserve authentic
prompt wording and skill names; added diagnosis or loading requests change
the intervention.
Separate clean replacement from recovery after an update. The latter can keep
identical old loaded history in both arms while changing future available
skills. Reloading availability does not prove automatic invocation or removal
of earlier instructions.
Require pre-run independent Claude and GPT method review using five scored
rules. Both families must grade a phase READY; missing prerequisites and
disagreements keep it closed. The reusable reviewer addendum grades the user's
actual question, not an author's narrower substitute.
Dashboard setup is removed from the normal path. The report template leads
with the question, actual work, controls and supported conclusion. Unrun
comparisons remain NOT RUN.
Scope and evidence
Instructions and references only. No evaluator implementation, permission,
dashboard, plugin-version or installed-package change.
Both independent source-review families completed the focused rule/rubric
review and verified the required fixes. Skill shape, plugin manifests,
relative links/anchors, the existing no-enqueue guard and whitespace checks
pass. The restoration recipe's shell commands reproduce the verified
historical repository archives and preserve an existing archive.
Source review does not prove improved future instruction following. Historical
causal validation remains unexecuted; prior synthetic observations are
supplemental diagnostics, not a substitute.