feat(evaluation): benchmark PowerContext across agent harnesses and models

[Teingi]

Feature description

Publish reproducible PowerContext OFF/ON benchmarks across 3–4 agent harnesses and multiple model families, so users can assess whether the benefits hold for their own agent and model combination.

Here, agent harness means the agent host/runtime (such as Codex CLI, Goose, or Pi), distinct from the benchmark's execution and grading infrastructure.

Problem and proposed solution

The current benchmark page describes a paired SWE-bench Pro public v2 run using Codex and gpt-5.6-sol. This provides evidence for that configuration; broader coverage is needed to assess generality across agent hosts and models.

Initial coverage

Target at least three agent harnesses and three model families, with a fourth combination if practical:

Candidate agent harness	Candidate model family
Codex CLI	GPT / Sol
Goose	Qwen
Pi	GLM
Optional fourth host, such as Claude Code or OpenCode	An additional supported model family

These are candidate combinations, not claims of verified compatibility. Validate model access, PowerContext integration, unattended execution, and trace/usage capture before selecting the final matrix. Record exact host versions, provider/model IDs, reasoning settings, and integration versions.

Experimental protocol

• Run paired PowerContext OFF versus ON within every selected host/model combination, using the same task IDs, grader, host configuration, model settings, permissions, and execution budgets. Keep the host's normal native capabilities enabled in both arms.
• Pin the dataset, task revisions, PowerContext commit, and environment. Document precisely what ON enables and how its initial context is constructed.
• Isolate each arm and trial. Permit only declared pre-task history/context; prevent leakage from reference answers, hidden tests, or prior evaluation attempts.
• Use a shared task set where capabilities allow. Include real task-completion outcomes and a bounded cross-session continuation workload relevant to PowerContext. Preserve source-native grading semantics and disclose any incompatible or excluded tasks.
• Repeat trials to characterize variability. Predeclare the repetition plan, include failures/timeouts and integration failures in reporting, and report paired deltas with uncertainty. A single run per arm should be identified as preliminary.
• Interpret each OFF/ON delta within its own combination. Cross-combination absolute scores alone cannot identify the effects of the model or harness. Where feasible, add one overlapping configuration, such as the same model in two hosts.
• Reuse existing evaluation/E2E infrastructure where appropriate, following Tracking Issue: expand benchmark coverage #1263. Start with a small runnable pilot; maintainers can then fund and execute the declared evaluation scope.

Public results

Publish a per-combination table containing task counts, OFF/ON success rates, paired changes, uncertainty, latency, and observed token usage/cost with explicit accounting boundaries and missing-data disclosure. Token reduction is an outcome to measure, not a prerequisite for reporting a combination.

Provide runnable commands, non-secret configuration manifests, dataset hashes, exact version identities, per-task results, sanitized traces, and analysis scripts. Clearly identify pilot/subset results and distinguish integration validation from measured task performance.

Acceptance criteria

• Final matrix covers at least three agent harnesses and three model families, with exact versions and compatibility documented.
• A small OFF/ON pilot is reproducible for each selected combination.
• Shared tasks, grading, budgets, initial context, isolation, and repetition policy are documented before the main runs.
• The declared evaluation scope is completed and published for every selected combination, including unsuccessful and neutral results.
• Reports include per-combination paired outcomes, variability, latency, and available usage/cost data without implying unmeasured savings.
• A contributor can reproduce a bounded run and regenerate its report from the published instructions and artifacts.
• English and Chinese benchmark documentation links to the protocol, results, artifacts, and limitations.

Alternatives considered

• More runs on one host/model improve confidence in that configuration but do not establish broader applicability.
• A full host × model Cartesian matrix would be expensive. Start with 3–4 representative combinations and expand based on observed gaps.
• Integration smoke tests establish compatibility but do not demonstrate better task outcomes.

Additional context

Related tracking issue: #1263. This issue focuses on cross-host/model evidence and does not require migrating the existing full LoCoMo or SWE-bench Pro workflows.

[Fengzdadi]

Hi, I'd like to work on this issue, starting with the runnable pilot rather than the full matrix.

Plan:

• Build on the Harbor-based e2e/ path (per Tracking Issue: expand benchmark coverage #1263), not the evaluation/ console. Harbor 0.16 already provides installed agents for codex, claude_code, opencode, goose and pi, so the work is the layer above: a per-host adapter for attaching PowerContext in the ON arm, OFF/ON pairing with treatment validation, a paired-trial evidence model, and a per-combination report with uncertainty.
• First host: Codex, using the existing harbor-tasks/ samples plus a small Terminal-Bench 2.0 subset. Then Claude Code and OpenCode, which already have integrations in the repo. Goose has no integration today, so its ON arm would be MCP-only; I'd treat Goose and Pi as a second batch.

Two questions before start:

1. Is building on e2e/ (instead of evaluation/) the right direction?
2. Is a host-specific ON definition acceptable (plugin + hooks for Codex/Claude Code/OpenCode, MCP-only for Goose), as long as it is stated in the results?

P.S. Is there a time limit for resolving this issue? By when does it need to be completed?

[Fengzdadi]

@Teingi @PsiACE Following #1263 ("add more benchmark and evaluation workloads through the Harbor-backed E2E harness"), I'm building this on e2e/. Planned steps, one small PR each:

1. Extract a host adapter from the Bub harness, no behavior change: refactor(e2e): extract a host adapter from the Bub harness #1743
2. Bub OFF/ON pairs on cross-session tasks
3. Codex, then Claude Code, then OpenCode

One question is still open and matters from step 3 on: is it fine for the ON arm to use each host's own integration (plugin + hooks), stated in the results?