Skip to content

S25 wiring: policy inheritance in shadow mode (SHIP_POLICY_SHADOW) - #43

Merged
im-tyler merged 2 commits into
mainfrom
claude/w3-policy-shadow
Oct 4, 2026
Merged

im-tyler merged 2 commits into
mainfrom
claude/w3-policy-shadow

Conversation

@im-tyler

@im-tyler im-tyler commented Oct 4, 2026

Copy link
Copy Markdown
Contributor

Changes

Shadow mode only. Nothing is enforced and no outcome changes.

  • src/shadow-hook.ts (new, leaf, type-only imports): an observer seam. The decision points report their outcome after deciding. The observer is null unless SHIP_POLICY_SHADOW=on installs one, and its errors are swallowed.
  • src/policy-shadow.ts (new):
    • Layer loader. org: an open ceiling plus the governance grants as action rules, plus SHIP_DAILY_BUDGET_USD when it is set. project: the project record plus the daily cap actually applied. user / service_account: the principal. Optional declared layers come from policy-layers.json (SHIP_POLICY_LAYERS_FILE).
    • Comparator that runs resolvePolicy next to each decision.
    • JSONL sink (policy-shadow.jsonl, SHIP_POLICY_SHADOW_FILE) that records disagreements and shadow errors.
    • summariseShadow and renderShadowReport.
  • Hooks, each one line or a thin wrapper: mayDo (this also covers web may() and the delivery re-check), effectiveAuthority, resolveNetworkTier, normalizeEgressAllow, assertRepoAllowed (a wrapper that observes both the allow and the throw), and assertDailyBudget (both the daily and the weekly comparison).
  • fileRuntime and nucleusRuntime call installPolicyShadowFromEnv(), which does nothing unless the flag is on.
  • CLI: teploy-ship policy shadow-report [--json].
  • Service accounts are id-only matches. The role is not decided, and the report says so. Records are tagged [service-account].

Checks

  • pnpm run lint is clean.
  • pnpm test: 1871 of 1871 pass in the main suite (20 new tests in policy-shadow.test.ts).
  • The swebench/ and scripts/ suites show 3 failures in scripts/grader-sensitivity.test.mjs (pj-b-deploy-recovery, deploy.sh). They fail identically on a clean origin/main checkout and this PR does not touch them.
  • Default-off equivalence: with the flag off, no observer is installed and no file is created. The same calls return byte-identical results with the shadow on and off.
  • Negative controls:
    • A union/max resolver gives zero records under the correct merge and 2 visible disagreements under the buggy one.
    • Mutation 1, removing the mayDo hook: 6 tests fail.
    • Mutation 2, swapping the disagreement direction: 10 tests fail.
    • Mutation 3, a resolver that always returns the loosest authority and budget: 4 tests fail.
    • All three mutations were restored afterwards.
  • web/ is untouched, so I did not run the web tests or the web build.

Not verified / still open

  • No live system was exercised. Only fake layers and fake runtimes.
  • Agreements are not counted, so the report gives no disagreement rate. A missing record file reads as unknown, not zero.
  • Derived layers mirror today's configuration, so a deployment with no declared layers records few disagreements. Weekly budget and repo host against egress are the exceptions.
  • Weekly spend is lower-bounded by today's committed spend. Nothing reads a weekly total.
  • assertRepoAllowed is compared through a proxy: the clone host against the policy's egress rules.
  • Network and egress points have no project context, so only org-level layers apply to them.
  • The worker.ts intake budget check is not hooked, to avoid conflicts with other agents. Only the enqueue budget check is.
  • Service-account role is still undecided.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VqsBNqvaWezf1DwQrAnVgX


Generated by Claude Code

claude added 2 commits October 4, 2026 02:43
…th disagreement report

Record-only comparison of resolvePolicy against mayDo, effectiveAuthority,
resolveNetworkTier, normalizeEgressAllow, assertRepoAllowed and the enqueue
daily budget. Default off installs nothing. Adds 'policy shadow-report'.
Service accounts are id-only matches; their role is still undecided.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VqsBNqvaWezf1DwQrAnVgX
Resolves imports in cli.ts (policy shadow report and stack detect) and the
budget path in runtime.ts (shadow observation first, then the budget gate).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VqsBNqvaWezf1DwQrAnVgX
@im-tyler
im-tyler merged commit 2a51d92 into main Oct 4, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants