LongMemEval: answer-construction headroom (retrieved-but-unused facts)

[jack-arturo]

Context

Splitting this out of #158 and #159, both of which are now closed: the server-side recall work for LongMemEval shipped in #194 (date-aware ranking, recency_bias=off|on|auto, latest-fact selection, deterministic timestamp tiebreak, configurable recency decay, and the #183 failure-mode diagnosis harness). Retrieval is effectively saturated — recall@5 sits at ~90% on preference and ~97% on the temporal/multi-session set on the judged 500q run.

The residual gap is not retrieval anymore. It's answer construction: the correct fact is in the top-5 context, but the answerer doesn't use it.

Evidence

From the diagnosis harness on the ship run (failure_modes_ship_llm_20260612.json):

• Preference: 7 of 9 remaining misses are answer-construction (retrieved in top-5, unused) — only 2 are ranking/retrieval.
• Full set failure-mode distribution: answer-construction 39, retrieval-gap-rank-6-10 7, ranking 3. The answer-construction bucket dominates and is stable across runs.

Why this is its own issue

The lever here is context presentation / answer prompting, which is harness-side (the answerer's prompt + how retrieved memories are formatted into context) — not server recall ordering. Keeping it tangled up with the recall-ranking issues made both look perpetually "not done" when the server half is actually finished.

Acceptance criteria

• Characterize the answer-construction failures: is it ordering within the top-5, formatting, distractor memories crowding the signal, or the answerer ignoring a present fact?
• Prototype context-presentation changes (e.g. surfacing the latest-fact selection explicitly, trimming distractors, or a tighter answer prompt) and measure on the mini (--per-type 5) then full LongMemEval.
• Decide what, if anything, belongs server-side (e.g. a recall response hint about which memory is the current/authoritative fact) vs purely in the answerer/harness.

Related

• Closed: LongMemEval: improve preference memory update and recall semantics #158 (preference), LongMemEval: improve multi-session and temporal conflict resolution #159 (multi-session/temporal) — server-side portions done in feat(recall): ranking release — recency config, tag-score cap, relevance gate, date-aware ranking (#182, #193, #186, #187, #183, #184, #188) #194
• Diagnosis harness: feat(benchmarks): LongMemEval failure-mode diagnosis harness + judge quota preflight #183
• Current-state suppression: recall: respect invalidation and current-state semantics #169 / fix(recall): respect current memory state #170

[jack-arturo]

#200 update — evidence-first prompt probe

Implemented and measured the harness-only evidence-answer prompt probe locally against the healthy localhost:8001 AutoMem API. No production /recall, vector, graph, or scoring behavior was changed.

Mini A/B, 30-question stratified LongMemEval, gpt-5-mini answerer + pinned gpt-5.4-mini-2026-03-17 judge:

Config	Accuracy	Recall@5	Judge errors
temporal-answer control	80.00% (24/30)	100.00% (30/30)	0
evidence-answer v1	66.67% (20/30)	100.00% (30/30)	0
evidence-answer v2	66.67% (20/30)	100.00% (30/30)	0
evidence-answer v3	73.33% (22/30)	100.00% (30/30)	0

LLM failure diagnosis on the same mini slice:

• temporal-answer: 6 failed-but-retrieved; LLM labeled 5 as answer-construction, 1 as ranking; rank-1 abstentions: 2.
• Best evidence-answer v3: 8 failed-but-retrieved; LLM labeled 7 as answer-construction, 1 as missing-date-use; rank-1 abstentions: 1.

Decision: the evidence-first prompt is clearly worse on mini, so I did not run full LongMemEval and this should not close #200. The useful finding is that strict evidence-first prompting reduces some abstention shape but increases total answer-construction misses, especially preference/recommendation cases. Next direction should likely be more targeted answer assembly for preference/recommendation and multi-session count questions, not a blanket stricter evidence prompt.

Local artifacts are under ignored benchmarks/results/issue200/; summary was recorded in benchmarks/EXPERIMENT_LOG.md.

[jack-arturo]

Typed-answer probe update for #200:

• Implemented a harness-only typed-answer LongMemEval preset on fix/200-typed-answer-prompts.
• No server /recall, ranking, vector fetch, or graph behavior changed.
• Same-day mini A/B against healthy local API, using direct Python harness because Docker is unavailable in this shell:
  • temporal-answer control: 76.67% (23/30), recall@5 30/30, judge_errors=0
  • typed-answer: 80.00% (24/30), recall@5 30/30, judge_errors=0
• Diagnosis:
  • total LLM-labeled answer-construction misses stayed 6 -> 6
  • rank-1 abstentions improved 3 -> 0
  • multi-session accuracy improved 40% -> 60%

Conclusion: this is a better direction than the broad evidence prompt, but it does not meet the planned full-run gate because total answer-construction misses did not drop. I skipped full LongMemEval and #200 should remain open.

[jack-arturo]

Typed-answer v4 full validation is complete. My interpretation: keep #200 open.

Results:

• Mini gate looked promising: typed-answer v4 reached 83.33% (25/30), recall@5 30/30, and LLM-labeled answer-construction misses dropped 6 -> 5 versus same-day temporal-answer control.
• Full run did not generalize: typed-answer v4 finished at 85.80% (429/500), recall@5 96.60% (483/500), judge_errors=0, memory_ingest_failures=0, publishable=true.
• Historical full temporal-answer ship comparison was 86.00% (430/500), recall@5 96.60% (483/500), so the aggregate is effectively flat/down at identical retrieval.
• Full failure diagnosis characterized 48 answer-construction misses: 24 multi-session and 11 single-session-preference. That is worse than the prior full forensic range of 39-41 answer-construction cases.
• The two prompt-targeted full categories regressed versus the historical note: multi-session 77% vs 81%, single-session-preference 53% vs 63%.

Conclusion: v4 is useful negative evidence. The mini signal was real enough to justify the full run, but it appears overfit/noisy and did not reduce the full answer-construction failure mode. I updated benchmarks/EXPERIMENT_LOG.md and docs/TESTING.md with the full numbers and no-promote interpretation. No server recall API, ranking, vector fetch, graph, or production behavior changed.