Skip to content

Provision a ready Odinn benchmark agent fixture - #10

Merged
jason-allen-oneal merged 1 commit into
mainfrom
fix/odinn-ready-agent-fixture
Aug 2, 2026
Merged

Provision a ready Odinn benchmark agent fixture#10
jason-allen-oneal merged 1 commit into
mainfrom
fix/odinn-ready-agent-fixture

Conversation

@jason-allen-oneal

Copy link
Copy Markdown
Member

Summary

  • add a deterministic Odinn benchmark-state preparer
  • create a completed nonblank main agent identity and valid integrity-bound manifest/registry
  • remove stale BOOTSTRAP.md so trials cannot enter onboarding
  • preserve existing provider configuration and protected OAuth state without creating or printing secrets
  • install the required workspace/tool policy grants and explicit unconfined-process acknowledgement
  • fail closed before model preflight when Odinn advertises write/exec capabilities without a ready fixture

Root cause

The adapter advertised workspace.write and process.exec, but the benchmark had no enforced contract proving that the Odinn state fixture actually contained a completed identity, matching runtime grants, and an enabled main-agent registry. The mismatch allowed every tool-dependent Odinn trial to enter scoring with capabilities the runtime could not provide.

Current-head proof

  • HEAD SHA: 4edd035333de9447e969117294b3a833740b4cc8
  • Exact command: pnpm typecheck && pnpm test
  • Log: /tmp/agent-benchmarks-pr-odinn-ready-agent.log
  • Result: typecheck passed; 24/24 tests passed, 0 failed or skipped
  • Relevant output: Odinn state preparation, missing-grant/identity rejection, and missing unsafe-acknowledgement rejection all passed

Operator usage

After configuring the isolated Odinn provider and protected OAuth state:

pnpm prepare:odinn-state -- --state benchmarks/state/odinn

Not tested

  • No live provider-backed 105-trial benchmark matrix was rerun from this branch.
  • Protected production OAuth material was deliberately not copied into this worktree.

@jason-allen-oneal
jason-allen-oneal merged commit fa56a19 into main Aug 2, 2026
3 checks passed
@jason-allen-oneal
jason-allen-oneal deleted the fix/odinn-ready-agent-fixture branch August 2, 2026 14:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant