Following up from mem0ai/mem0#5614, where memory-quality signals were redirected from SDK search() into this benchmark suite.
I'd like to contribute a small LongMemEval-first diagnostic for temporal retrieval validity. LongMemEval seems like the cleanest starting point because each question has a question_date, while retrieved memories may include created_at / updated_at.
Problem
Today LongMemEval reports answer/retrieval quality by cutoff, but it does not expose whether retrieved memories are temporally valid relative to the question date.
That matters because benchmark ingestion can include haystack sessions on both sides of the question timestamp. A retrieved memory may be semantically relevant but dated after the question, which makes it invalid evidence for that question.
Current shape
compute_longmemeval_metrics() currently reports cutoff-level quality like:
metrics_by_cutoff[label] = {
"overall": {
"total": total,
"correct": correct,
"accuracy": correct / total * 100 if total else 0.0,
"avg_score": statistics.mean(scores) * 100 if scores else 0.0,
},
"by_question_type": type_metrics,
}
Per-question outputs already carry the fields needed for temporal diagnostics:
{
"question_date": "2023/04/10 (Mon) 10:15",
"retrieval": {
"search_results": [
{
"memory": "...",
"score": 0.82,
"created_at": "2023-04-10T21:32:00Z",
"updated_at": "2023-04-10T21:32:00Z",
}
]
},
}
Possible output shape
A first pass could add a deterministic diagnostic block per cutoff:
metrics_by_cutoff[label] = {
"overall": {...},
"by_question_type": type_metrics,
"temporal_retrieval": {
"timestamp_coverage": 0.94,
"future_retrieval_rate": 0.02,
"avg_memory_age_days": 18.7,
},
}
Where:
timestamp_coverage = fraction of retrieved memories with a parseable timestamp;
future_retrieval_rate = fraction of retrieved memories dated after question_date;
avg_memory_age_days = average age of retrieved memories relative to question_date, when timestamps are available.
I would avoid calling this semantic "staleness" in the first version. This is only timestamp-based retrieval validity, not proof that an old memory is obsolete.
Proposed first scope
- LongMemEval only.
- Pure benchmark-side metric computation.
- No changes to Mem0 search behavior.
- No LLM judge for temporal validity.
- Handle missing/invalid timestamps without failing the benchmark run.
This seems complementary to PR #12: that PR adds deterministic failure-mode scenarios, while this would add aggregate temporal retrieval diagnostics to LongMemEval benchmark outputs.
Open question
Would maintainers prefer this to start as:
- a shared helper in
benchmarks/common/metrics.py, wired first into LongMemEval, or
- a LongMemEval-local helper first, generalized later if useful?
If this direction sounds right, I can pick up the first PR.
Following up from mem0ai/mem0#5614, where memory-quality signals were redirected from SDK
search()into this benchmark suite.I'd like to contribute a small LongMemEval-first diagnostic for temporal retrieval validity. LongMemEval seems like the cleanest starting point because each question has a
question_date, while retrieved memories may includecreated_at/updated_at.Problem
Today LongMemEval reports answer/retrieval quality by cutoff, but it does not expose whether retrieved memories are temporally valid relative to the question date.
That matters because benchmark ingestion can include haystack sessions on both sides of the question timestamp. A retrieved memory may be semantically relevant but dated after the question, which makes it invalid evidence for that question.
Current shape
compute_longmemeval_metrics()currently reports cutoff-level quality like:Per-question outputs already carry the fields needed for temporal diagnostics:
{ "question_date": "2023/04/10 (Mon) 10:15", "retrieval": { "search_results": [ { "memory": "...", "score": 0.82, "created_at": "2023-04-10T21:32:00Z", "updated_at": "2023-04-10T21:32:00Z", } ] }, }Possible output shape
A first pass could add a deterministic diagnostic block per cutoff:
Where:
timestamp_coverage= fraction of retrieved memories with a parseable timestamp;future_retrieval_rate= fraction of retrieved memories dated afterquestion_date;avg_memory_age_days= average age of retrieved memories relative toquestion_date, when timestamps are available.I would avoid calling this semantic "staleness" in the first version. This is only timestamp-based retrieval validity, not proof that an old memory is obsolete.
Proposed first scope
This seems complementary to PR #12: that PR adds deterministic failure-mode scenarios, while this would add aggregate temporal retrieval diagnostics to LongMemEval benchmark outputs.
Open question
Would maintainers prefer this to start as:
benchmarks/common/metrics.py, wired first into LongMemEval, orIf this direction sounds right, I can pick up the first PR.