Skip to content
This repository was archived by the owner on Oct 23, 2026. It is now read-only.
This repository was archived by the owner on Oct 23, 2026. It is now read-only.

Add offline fixtures and side-effect-safe probe infrastructure #9

Closed

Description

@magnus919

Problem

Several probes rely on live arXiv access and actual GitHub issue creation. Network availability, account state, and changing external data contaminate model scores; live issue creation also creates unwanted side effects.

Goal

Run all benchmark probes against deterministic local substitutes with explicit setup, reset, and cleanup behavior.

Scope

  • Provide fixed arXiv-like search/detail fixtures that exercise the research skill interface.
  • Provide a disposable local GitHub-compatible issue target or a narrow mock that records created issues.
  • Add deterministic fixture data for each probe, including expected facts to propagate through workflow chains.
  • Add lifecycle commands for setup, reset, and cleanup.
  • Document the supported isolation boundary and prohibit calls to production external services during a benchmark run.

Acceptance Criteria

  • The full suite runs with outbound network access disabled after images/dependencies are available.
  • No probe can create or modify a real GitHub issue or repository.
  • Fixture responses are versioned and have stable IDs/content across runs.
  • Workflow-chain probes can validate citations and conclusions against fixture facts.
  • Setup/reset/cleanup leave no state that affects a subsequent run.
  • A test verifies that external service endpoints are not contacted.

Non-Goals

  • Perfect emulation of arXiv or GitHub.
  • Evaluating browsing quality against the live web.

Implementation Notes

Favor lightweight stdlib services or static fixtures over new runtime dependencies. Fixture interfaces should match the tools exposed to Hermes closely enough to preserve the intended skill behavior.

Dependencies

Activity

  1. magnus919 commented on Jul 12, 2026

    @magnus919
    ContributorAuthor

    Resolved in 314b84d: versioned offline arXiv fixtures, a local-only gh shim, isolated Docker fixture network, lifecycle commands, and fixture tests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions