feat(e2e): compare PowerContext off and on for Bub continuation workloads

[Fengzdadi]

Which issue or RFC does this PR close?

Part of #1705, step 2 of the plan. Builds on #1743.

Rationale for this change

#1705 asks for paired PowerContext OFF/ON runs that include a cross-session continuation workload. The E2E harness evaluates Memory quality for a single arm, and its existing tasks use Bub-internal commands that bypass the model. Those tasks cannot show an OFF/ON difference and cannot run on other hosts.

What changes are included in this PR?

• paired command (make harness-paired): runs each continuation workload with PowerContext off and on, in separate containers, for --trials trials with the arm order alternating. It writes per-arm evidence and a paired report.
• Arms:
  • OFF installs Bub without the PowerContext plugin and passes no POWERCONTEXT_* settings.
  • ON binds the plugin to a new Scope with capture_events enabled. This is not the plugin's default; it makes Bub capture what the user says, as the other host integrations do, without relying on the model to call a memory tool.
  • HostAdapter.agent_config takes scope_id=None for OFF.
• Between sessions: a Harbor on_agent_ended hook records a Server statistics snapshot of the ON Scope after each session, flushing it first when another session follows. Harbor runs agent.run() under asyncio.wait_for, and the hook fires after it, so the flush does not use the agent's time budget. Harbor awaits the hook in a finally block, so a failed flush or read is recorded as a treatment failure rather than replacing the agent's own outcome, such as a timeout.
• Treatment check: an ON run counts only if the snapshots show Sources captured before the recall session and a context request during it. Otherwise it is an integration failure. Whether a flush creates Memory and whether recall returns content are PowerContext's own behavior under the treatment, so they are recorded but do not exclude a run: an ON run in which PowerContext keeps or returns nothing is a scored failure, not a missing measurement. For example, under the Server's default coding extraction profile, a pasted conversation between two other people was captured and queried but kept no Memory of it; excluding such runs would inflate the ON success rate.
• Scoring: integration failures and infrastructure errors are reported but not scored. Agent timeouts count as failures in both arms.
• First continuation task (project-decision-continuation): the fact appears only in conversation, next to an unrelated typo fix, and a later session asks for it. The answer key lives only in the recall step's tests, because Harbor keeps each uploaded test directory in the container for later steps.
• Isolation from acceptance: paired manifests live in e2e/bub/paired-tasks/, and acceptance rejects continuation workloads, so the acceptance catalog and CI are unchanged.
• README section describing both arms, the flush, the treatment check and the limits.

Are there any user-facing changes?

A new paired command and harness-paired Makefile target. Existing commands, manifests and evidence are unchanged.

How was this change tested?

Pilot run

make harness-paired ARGS='--trials 2' against a real Server, Harbor, Bub and model:

Trial	Order	OFF answer	ON answer
1	OFF, ON	unknown (0)	We decided to use OceanBase with a shard count of 12 for the ticket service. (1)
2	ON, OFF	unknown (0)	We decided to use OceanBase with a shard count of 12. (1)

• Report: OFF 0/2, ON 2/2, 2 scored pairs, mean ON minus OFF +1.00, no errors or integration failures. Single-trial smoke runs before the pilot and on the final commit gave the same outcome.
• Treatment evidence (ON): the capture session left 8–12 Sources in the Scope, and context requests rose during the recall session. The flush produced one Memory entry: "The ticket service database will run on OceanBase, with shard count fixed at 12, to remain compatible with existing MySQL clients." Ready context preparations also rose during the recall session (4 → 8 and 2 → 6).
• Why the check compares sessions: requests and ready preparations already occurred during the capture session. Bub's plugin flushes every five captured events, so later model calls in that session received the new Memory. Only the increase during the recall session shows what the recall session did.
• OFF behavior: in both trials the agent searched Bub's own history (tape_search) four times, found nothing, and wrote unknown. Bub's tape does not carry across these sessions, so OFF keeps Bub's native behavior.
• Configuration: the pilot ran on b02b8ab6. Later commits change how settling handles failures and the final session, the README timeouts, and which snapshot fields gate the treatment check; the pilot's runs classify the same under the final rule. Server extraction profile: default coding; Harbor 0.16.1; Bub 0.4.2 with bub-acp-server 0.0.2; agent model openai:gpt-6-luna through a ChatGPT login; local Server on SQLite with openrouter:deepseek/deepseek-v4-pro for generation and openrouter:qwen/qwen3-embedding-4b for embeddings; OrbStack on macOS. The Server API does not report model identities, so they are recorded here rather than in the evidence.
• Cost and time: Server-side inference for the smoke and pilot runs cost about $0.02. Each arm took 1–2.5 minutes.

Checks

• make check and make harness-check (71 tests) pass.
• A regression test covers an ON run that was captured and queried but kept and returned nothing: it counts as a scored attempt, not an integration failure.
• New tests cover the grader, the treatment check, outcome classification, the paired summary, the flush loop, settling across sessions and its failures, the OFF/ON job configurations, the OFF install command, the task invariants, and acceptance rejecting continuation workloads.
• The job-config tests from refactor(e2e): extract a host adapter from the Bub harness #1743 pass unchanged, so acceptance runs are unaffected.
• make unit-test: tests/builtin/test_native_code*.py fails locally on unmodified master too; nothing under tests/ imports the harness.

Limits

• One task and two trials per arm validate the pipeline and its evidence; they do not measure PowerContext's effect.
• Not run: the fixed Compose harness, other hosts, and other models. The report does not yet estimate uncertainty, check the Default Scope for leaks, or record latency and token usage.

AI usage statement

This PR was developed with Claude Code (Claude Opus 5.5), which designed and wrote the change and tests, ran the checks and the pilot above, and analyzed the evidence. The author reviewed the change and provided the pilot environment.

🤖 Generated with Claude Code