You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Oct 23, 2026. It is now read-only.
Repository navigation
This repository was archived by the owner on Oct 23, 2026. It is now read-only.
Add offline fixtures and side-effect-safe probe infrastructure #9
Several probes rely on live arXiv access and actual GitHub issue creation. Network availability, account state, and changing external data contaminate model scores; live issue creation also creates unwanted side effects.
Goal
Run all benchmark probes against deterministic local substitutes with explicit setup, reset, and cleanup behavior.
Scope
Provide fixed arXiv-like search/detail fixtures that exercise the research skill interface.
Provide a disposable local GitHub-compatible issue target or a narrow mock that records created issues.
Add deterministic fixture data for each probe, including expected facts to propagate through workflow chains.
Add lifecycle commands for setup, reset, and cleanup.
Document the supported isolation boundary and prohibit calls to production external services during a benchmark run.
Acceptance Criteria
The full suite runs with outbound network access disabled after images/dependencies are available.
No probe can create or modify a real GitHub issue or repository.
Fixture responses are versioned and have stable IDs/content across runs.
Workflow-chain probes can validate citations and conclusions against fixture facts.
Setup/reset/cleanup leave no state that affects a subsequent run.
A test verifies that external service endpoints are not contacted.
Non-Goals
Perfect emulation of arXiv or GitHub.
Evaluating browsing quality against the live web.
Implementation Notes
Favor lightweight stdlib services or static fixtures over new runtime dependencies. Fixture interfaces should match the tools exposed to Hermes closely enough to preserve the intended skill behavior.
Activity
magnus919 commentedon Jul 12, 2026
Resolved in 314b84d: versioned offline arXiv fixtures, a local-only gh shim, isolated Docker fixture network, lifecycle commands, and fixture tests.