Skip to content

Make C12 qualification deterministic - #16

Merged
KeyffMS merged 4 commits into
mainfrom
fix/c12-deterministic-runtime-probe
Sep 2, 2026
Merged

Make C12 qualification deterministic#16
KeyffMS merged 4 commits into
mainfrom
fix/c12-deterministic-runtime-probe

Conversation

@KeyffMS

@KeyffMS KeyffMS commented Sep 2, 2026

Copy link
Copy Markdown
Owner

Summary

Replaces the model-planned C12 setup with a deterministic runtime probe so project_doc_max_bytes is tested independently of planner behavior.

C12 automatic-loading probe

  • creates a 4,320-byte AGENTS.md with a secret head marker inside the first 1,024 bytes and a secret tail marker at byte 3,072;
  • sets project_doc_max_bytes = 1024 in project config and redundantly passes -c project_doc_max_bytes=1024 to codex exec;
  • requires a tool-free read-only response: command-tool use blocks the probe;
  • the prompt contains marker names but never marker values, so the tail value cannot be guessed from the prompt;
  • outer harness marks C12 FAILED if the secret tail marker beyond the budget is actually visible.

PlanAnvil explicit-read assertion

  • runs the real map_instructions.py implementation outside the model;
  • verifies the full 4,320-byte size, complete SHA-256, full_read=true, and truncation_risk=true at the same 1,024-byte automatic budget;
  • computes the final C12 decision deterministically without an evaluator-agent.

Scope / safety

  • C12 is the only newly overridden capability;
  • existing C01–C11 and C13–C16 deterministic fixes are inherited unchanged;
  • no sandbox bypass or danger-full-access;
  • full qualification remains workflow_dispatch-only from main;
  • this PR uses only standard hosted CI.

@KeyffMS
KeyffMS merged commit f6beffe into main Sep 2, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant