Skip to content

feat(tooling): enforce explicit environments and exact-head CI evidence - #452

Merged
cloga merged 9 commits into
mainfrom
cloga-environment-evidence-guardrails
Oct 25, 2026
Merged

cloga merged 9 commits into
mainfrom
cloga-environment-evidence-guardrails

Conversation

@cloga

@cloga cloga commented Oct 25, 2026 •

Copy link
Copy Markdown
Owner

Summary

Fixes #451. Concrete engineering safeguards, independently deliverable without #442's evaluator platform, daemon or multi-agent loop. Repository tooling only; no product/Core/runtime change.

  • Explicit machine-readable environment/task selection; prohibited local dependency-network actions fail before execution. Fixed dependency-free local checks; full dependency/build/type/integration/package validation uses existing GitHub CI.
  • Persistent two-execution-failure budget per task/operation, unchanged across continuation/success. No guard polling, registry switching or permission escalation. Pending/rejected CI stays blocked but does not consume execution-failure budget.
  • Real bounded gh api reads bind repository, clean exact HEAD, workflow identity and five required jobs. Newer runs precede old rerun attempts; normal skipped Release is permitted.
  • Existing acceptance schema 2 combines real exact-head CI with strict, content-free reported live observations. Separate installed/loaded/version/source facts, required-action matching, bounded numeric times and fresh post-action observations. Unknown/wrong loaded version, missing/live-not-run, error/timeout and unapproved actions cannot become acceptance.
  • Current app lifecycle: fresh discovery; one explicitly authorized controlled launch only when stopped; inspect official safe welcome skip rather than infer auth failure; rediscover replacement windows; fresh final current app plus target-panel/workspace. Old success, action done, no immediate diff or launch receipt does not establish readiness. Interruption remains blocked pending fresh UI. Two distinct navigation methods and one panel handoff remain bounded.
  • GitHub-first formal upgrade sources: exact canonical published Release asset URL preferred; verified GitHub-downloaded tgz cache is a distinct digest-matched fallback with a fixed native-URL failure reason. Reject local builds/source/link/worktree as formal upgrade sources.
  • Separate mandatory completeness from optional explicit residual-risk disposition. A specific fix, controlled regressions, required exact CI/live/version/source gates and user risk acceptance can recommend closure with untested production faults listed and reopen-on-recurrence. Known failures, critical gaps and mandatory unmet checks cannot be waived. Untested remains untested; original root cause is not asserted proven; no automatic issue closure.
  • All live/source/authorization records remain report consistency only, not host execution, provenance or authenticated consent attestation. No Desktop controller, installer, auto-upload, new evaluator or success receipt. Existing iteration-review preflight and validation gates remain intact.

Expected vs Actual

Expected: forbidden operations never execute; budgets persist/exhaust; latest non-green CI cannot be masked; pending can later complete. Required field acceptance cannot be replaced by CI-green, installed artifacts, unobserved/stale windows, permissions inferred from user absence, wrong actions or nonformal sources. Optional risk acceptance cannot rewrite required results or excuse known reproduction.

Final HEAD: c35435d50b130799e75e0602cd06188d48c480b8.

Authoritative full CI run 40157832380: completed/success for that exact HEAD. Windows/Linux verify, both Published adapter jobs and release-ready all succeeded; release normally skipped.

  • node scripts/agent.mjs guard --task 451 --environment github-actions-evidence --operation ci-evidence --head c35435d50b130799e75e0602cd06188d48c480b8 — passed against actual GitHub, CI_EVIDENCE_VERIFIED, CI execution-failure count 0, all required jobs successful.
  • node scripts/agent.mjs guard --task 451 --environment local-auxiliary --operation scripts-test — local validation incomplete / budget exhausted. First lifecycle edit exposed five schema-1 regression failures, repaired before push. Second invocation passed those assertions and lifecycle cases but hit the unchanged 120-second command timeout before suite completion. Persistent task 451/scripts-test ledger retained two failures, remaining 0. No reset, alternate task ID, direct-shell rerun, expanded timeout or dependency recovery was used. Final complete validation ran in existing CI.
  • Prior fully completed local guarded run before lifecycle increment: 137/137 passed on 857e4194; not represented as current-head proof. Final CI executes all current source/tooling regressions, including lifecycle and residual disposition cases.
  • git -c core.whitespace=cr-at-eol diff --check — passed before final commits; checkout clean after push.
  • Initial CI detected the removed iteration-review integration and was repaired without weakening baseline/CI. Historical successes are not substituted for final HEAD.

The checked commands below ran successfully in final GitHub CI, not locally:

  • pnpm install --frozen-lockfile
  • pnpm verify (source/test types, Agent contract, tooling tests, clean build, import smoke)
  • pnpm pack --pack-destination artifacts
  • pnpm verify:tarball -- artifacts/dsh-github-copilot-<package-version>.tgz
  • All Windows/Linux exact current DSH 0.2.1-rc.3 tagged-source and published-artifact CI checks passed (earlier pins are historical evidence only)

Acceptance evidence

This PR extends the checker; synthetic fixtures are not live acceptance. Schema 1 remains non-live checking; browser/manual requirements require schema 2. Schema 2 invokes the same issue/ci-evidence ledger, never a report-supplied green boolean. Final CI covers current-app startup/skip/replacement/final freshness, incorrect auth/launch assumptions, strict sources and bounded risk disposition. Execution/environment/delivery/provenance/authorization flags remain unverified. A close recommendation is reviewable report consistency, never a permission grant or issue-closing operation.

Evidence limits

Only routed operations are enforced; arbitrary shell bypass, fabricated observations/digests and metadata tampering are not intercepted. Final lifecycle freshness validates supplied times, not host truth or continued future liveness. The checker never launches/controls Desktop, downloads/installs assets, authenticates consent/provenance, or changes system networking/accounts to inject faults. No real app paths, window identifiers, workspace titles, raw errors, credentials, account identity or conversation content belong in schema 2 reports; no automatic uploads.

This coding session performed no Desktop actions, live model/search/account calls, install/restart, Core changes, issue #346 operations, merge or publication. Coordinator field observations are not this PR's execution evidence. Production fault injection is separate from controlled regression coverage: untested remains explicit, but may be user-accepted residual risk once all mandatory conditions pass. CI-green is not business/field acceptance or roadmap completion.

Iteration evidence

N/A: repository-only engineering workflow; no Auto/model/continuation/assistance policy changes. Mandatory read-only iteration-review preflight preserved separately from guarded verification commands.

Contract checklist

  • plugin-only: no Core source/artifact changes, node_modules patches, private-registry/prototype/shared-catalog mutation, or Core commit/PR/release work.
  • Uses existing published public APIs; delivery does not depend on a new Core export or Core patch. Missing capabilities have an explicit limitation or plugin-local alternative.
  • No second general adapter/catalog, credential store, gateway or sandbox owner.
  • Credential payloads stay Host-only; no real grant, device code or sensitive response appears in fixtures, logs, screenshots or this PR.
  • Both READMEs, Agent contract, baseline and tests match changed behavior.
    • Product setup/behavior/baseline unchanged; source-checkout rules live in AGENTS, Agent contract/verifier, existing acceptance/guard docs and related regressions. No README product-contract change required.
  • Release metadata and URLs stay aligned if version changed.
    • No version change or release.
  • Attribution names the actual tool (Assisted-by: DeepSeek Harness (DSH) for DSH); no copied vendor bot or fabricated Co-authored-by identity.
    • Actual tool: GitHub Copilot App.

Risks and rollback

Tooling/policy/docs/tests only. Shared AGENTS/agent-contract/verifier edits preserve previous constraints; other concurrent installation-smoke/replay-diagnostic implementation files untouched. Schema 1 browser/manual callers must explicitly use schema 2; its lifecycle shape is strict and fails closed. Reported controls are not arbitrary host interception. Ledger remains per-worktree Git metadata and local exhausted count is retained. Revert task commits to roll back; no product/data migration.

Release delivery

  • Classification: internal-only.
  • Prepared version and release channel: N/A; unchanged.
  • User delivery restrictions, if any: update existing PR and verify existing GitHub CI; no merge/tag/publish/install/live business operations.
  • After reviewed merge: important updates continue through green required CI, a fresh tag and verified Release without another release prompt.
  • Published Release URL + tag/commit/assets/SHA-256, or concrete blocker and pending step (update after release): N/A; publication explicitly out of scope.

Approval boundary

This focused PR and existing CI only. No merge/tag/release, profile installation, Desktop actions or Session interruption by this coding task. Broad acceptance requests, unavailable users or hand-entered consent fields never replace action-specific real tool authorization. Disposition recommendations never close issues automatically.

Fixes #451

cloga and others added 9 commits October 25, 2026 10:09
Add explicit local auxiliary and GitHub Actions evidence gates with persistent task/operation failure budgets and strict CI job validation. Document enforcement boundaries and keep full verification on CI.

Refs #451

Assisted-by: GitHub Copilot App

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Order distinct workflow runs before rerun attempts so an old green cannot mask a newer non-green run. Keep pending and rejected evidence blocked without consuming the persistent execution-failure budget; retain bounded API failures and no automatic polling.

Assisted-by: GitHub Copilot App

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Preserve the existing required read-only review validator separately from environment-selected verification commands.

Assisted-by: GitHub Copilot App

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…face (#451)

Keep exact validator argv and mandatory unexecuted review assertions while checking verification commands use the guarded environment selector. Full suite execution remains in GitHub CI.

Assisted-by: GitHub Copilot App

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…451)

Extend the existing acceptance contract with content-free UI action and fresh observation records, separate installed/loaded version evidence, bounded supported navigation and explicit permission/handoff blockers. Reuse the real CI reader and persistent operation budget without claiming UI execution attestation or controlling Desktop.

Assisted-by: GitHub Copilot App

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Do not let a panel observation stand in for a required refresh or reopen.

Assisted-by: GitHub Copilot App

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Prefer exact canonical Release URLs and distinguish digest-matched GitHub download cache fallbacks from local builds, source directories and links. Preserve report-only provenance limits and independent installation permissions.

Assisted-by: GitHub Copilot App

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Preserve controlled launch consent, official safe welcome skip, window rediscovery and final current-surface observation as bounded report consistency. Do not treat prior success, launch receipts or welcome UI alone as runtime/authentication proof.

Assisted-by: GitHub Copilot App

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…tance (#451)

Allow explicit risk-accepted closure recommendations for untested production faults after mandatory checks pass, without changing test truth, waiving critical failures or automatically closing issues. Local scripts-test budget remains exhausted; full validation uses existing GitHub CI.

Assisted-by: GitHub Copilot App

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add deterministic environment and CI evidence guardrails

1 participant