Benchmark task-contract continuation against recency and relevance baselines

[szmyty]

Outcome

Run a bounded, falsifiable experiment to determine whether an explicit task-contract representation improves continuation-task success at lower context cost than broad recent-history or ordinary relevance-based retention.

This evaluates rather than presumes the value of the continuation-state model in #31.

Hypothesis

For prompts such as “continue,” “do the next one,” or “finish this,” retaining only confirmed goals, constraints, decisions, unresolved questions, current state, and artifact pointers may preserve task continuity with fewer tokens and fewer constraint violations than broad recent history.

Frozen conditions

Before implementing an extension, freeze:

1. No continuity layer — only the immediate continuation prompt.
2. Recency/relevance baseline — a documented stock selection method.
3. Task-contract condition — bounded confirmed state with provenance and explicit supersession/expiration.

Use the same agent/model configuration and paired repository tasks.

Task-contract fields

Extract only:

• confirmed goals and success conditions;
• hard constraints and explicit exclusions;
• accepted decisions and rationale;
• current implementation/work state;
• unresolved questions, blockers, and risks;
• stable artifact and issue references;
• provenance to source turn/file;
• superseded, expired, uncertain, or conflicting state.

Do not silently convert guesses or proposed next steps into confirmed constraints.

Primary outcomes

• task success and tests/checks passed;
• constraint violations;
• stale-contract and supersession errors;
• incorrect artifact/reference use;
• input tokens and provider-cache effects;
• retrieval count;
• added extraction/retrieval latency;
• human correction or clarification required.

Evidence and reporting

Publish results through the provenance-preserving benchmark model tracked in Observatory. Retain:

• task definitions and frozen metrics;
• prompts/contracts supplied to each condition;
• transcripts and raw results;
• exclusions and failed runs;
• negative, null, and inconclusive outcomes;
• represented versions and environment.

Stop/reject conditions

Revise or reject the mechanism when:

• the stock baseline does not exhibit a reproducible limitation;
• the task-contract condition increases stale-state or constraint errors;
• latency or token overhead outweighs measured task benefit;
• results depend on an untestable semantic oracle;
• the evaluation cannot separate extraction quality from retention value.

Acceptance criteria

• The baseline limitation, tasks, conditions, primary metrics, and analysis plan are frozen before extension results are inspected.
• No-continuity, recency/relevance, and task-contract conditions are compared on paired tasks.
• The same agent/model configuration is used or differences are explicitly controlled.
• Provenance, supersession, expiration, uncertainty, and conflict behavior are tested.
• All transcripts, contracts, raw results, exclusions, failures, and negative outcomes are retained.
• Task success, constraint/stale-state errors, tokens/cache, retrievals, and latency are reported separately.
• The conclusion may be keep, revise, or reject.
• Findings feed back into Mindcap Add full and compact continuation bootstraps to conversation archives #31 without prematurely changing the archive contract.

Dependencies

• continuation bootstrap contract: Add full and compact continuation bootstraps to conversation archives #31
• benchmark/evaluation reporting: tracked in Observatory

Non-goals

• Claiming general memory superiority from one task set.
• Retaining unrestricted personal history.
• Treating relevance scoring as the only baseline.
• Changing Mindgarden routing or archive-capture semantics.

Source

Captured from the Headroom architecture/model/evaluation crossover review.