Agent Cabinet genuine-path dogfood (one local receipt) - #65
Conversation
Reuse AgentAdapter, run_agent_trial, OpenAICompatClient, and publication_receipt for one inspectable local product receipt. Scripted transports stay injected-transport / dry_run; paid OpenRouter and Ollama Cloud calls are refused without authorization. Co-authored-by: RaapTechllc <RaapTechllc@users.noreply.github.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2ce33b78cb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| _SYSTEM = ( | ||
| "You edit files in a local workspace. Read the user request and the file " | ||
| "listing. Reply with JSON only: " | ||
| '{"writes": {"relative/path": "full file contents"}}' | ||
| ) |
There was a problem hiding this comment.
For live-local artifacts, this is the real system prompt, but write_agent_artifact still derives prompt_config_sha256 from only the manifest's public prompt and the literal agent-harness; neither _SYSTEM nor the workspace-listing/output-format template is included. Changing this prompt or comparing against another adapter using the same manifest and harness can therefore leave every COMPARABILITY_FIELDS value equal and incorrectly classify prompt-different results as exactly comparable, so the actual model prompt template needs to be threaded into the provenance hash.
Useful? React with 👍 / 👎.
| if artifact["provenance"].get("self_check") == "failed": | ||
| return 1 | ||
| if not all(trial.get("workspace_disposed") for trial in artifact["trials"]): | ||
| return 1 | ||
| # Injected/fake dogfood is a path proof, not a leaderboard publish. | ||
| return 0 |
There was a problem hiding this comment.
When the Ollama probe succeeds but the subsequent chat call fails—for example because the requested model is absent or the daemon disappears—run_agent_trial records execution_failed, and the receipt refuses the artifact for infrastructure_errors; however, the self-check and disposal checks here still pass, so the CLI returns 0 even though no valid model evaluation occurred. Check the trial or summary infrastructure status before returning success so shell automation cannot treat this as a completed run.
Useful? React with 👍 / 👎.
| [ChatMessage(role="system", content=_SYSTEM), ChatMessage(role="user", content=user)], | ||
| temperature=0.0, | ||
| top_p=1.0, | ||
| max_tokens=min(budget.max_tokens, self.max_output_tokens) or 1, |
There was a problem hiding this comment.
AgentBudget.from_dict explicitly permits max_tokens == 0, but this expression converts that limit to max_tokens=1 and still performs the provider call before budget.consume reports an overage. A valid zero-token manifest therefore executes a model request despite forbidding all token use; reject that mode or return a timeout before invoking client.chat rather than overriding the configured budget.
Useful? React with 👍 / 👎.
2 participants
Why
PRD #30 infrastructure is on main. The remaining gap is a genuine evaluation path from execution to an inspectable product receipt — not another fake/gold dry-run sold as live evidence.
This PR does not merge, does not publish to production, and does not enable a paid campaign.
Preflight (reuse, no new adapter family)
Documented in
docs/validation/agent-eval-reuse-preflight.md.Reused as-is:
AgentAdapter+run_agent_trial+OfflineTextEnvironmentOpenAICompatClient(injectableTransport)build_agent_artifact/apply_agent_cabinet_to_artifactpublication_receipt(still refusesdry_run)offline-text-repair@1fixturepython -m agentbench.agent_evalis a binding of that contract. Family CLIs stay offline gold/fake runners.What landed
FilesystemModelAgentinspects the workspace, calls the existing client, applies JSONwrites, and leaves hidden verification in the environmentLocal product receipt: scorecard (completion, category completion, cost, latency) + publication gate + technician provenance
Honest labels:
dry_runoffline-referenceinjected-transportlive-localTests prove the path, including “wrong model write fails hidden verify” (not a gold agent)
CI runs the injected-transport dogfood and asserts
dry_run,published: false, andpublishable: falseLive-provider blocker
This environment has no
OPENROUTER_API_KEY, no local Ollama daemon, and this CLI does not useOLLAMA_API_KEY(Ollama Cloud is treated as paid). A live-local receipt needs an operator machine with Ollama. A paid campaign is a separate authorized task. Do not flipdry_run.Dogfood locally
python -m agentbench.agent_eval \ --manifest agentbench/tasks/minibench-agent-v1-dogfood.json \ --agent model \ --scripted-reply-file agentbench/tasks/dogfood-scripted-reply.json \ --trials 2 \ --out /tmp/minibench-agent-eval.json \ --receipt-out /tmp/minibench-agent-eval-receipt.jsonInspect the receipt. Then:
python -m agentbench.import_results /tmp/minibench-agent-eval.json --check # expected: REFUSED … dry_run … destination=/api/v1/agent-cabinet/runsOptional live-local (unpublished):
python -m agentbench.agent_eval \ --manifest agentbench/tasks/minibench-agent-v1-dogfood.json \ --agent model --provider ollama --model llama3.2:1b \ --trials 1 \ --out /tmp/minibench-agent-eval-ollama.json \ --receipt-out /tmp/minibench-agent-eval-ollama-receipt.jsonOperator notes:
docs/operators/agent-cabinet.md(Genuine dogfood).Verification
cd agentbench && pytest -q→ 503 passed, 3 skippedevaluation_class=injected-transport,dry_run=true,published=falseimport_results --checkrefuses the dogfood artifact--provider openrouterexits 2 and writes nothingagent_taskssmoke and MoA--dry-runstill passDoes not close #30. Does not touch Folio/FabOps, secrets, or production publication flags.