Skip to content

Add LongMemEval temporal retrieval diagnostics #13

Description

@D0odi

Following up from mem0ai/mem0#5614, where memory-quality signals were redirected from SDK search() into this benchmark suite.

I'd like to contribute a small LongMemEval-first diagnostic for temporal retrieval validity. LongMemEval seems like the cleanest starting point because each question has a question_date, while retrieved memories may include created_at / updated_at.

Problem

Today LongMemEval reports answer/retrieval quality by cutoff, but it does not expose whether retrieved memories are temporally valid relative to the question date.

That matters because benchmark ingestion can include haystack sessions on both sides of the question timestamp. A retrieved memory may be semantically relevant but dated after the question, which makes it invalid evidence for that question.

Current shape

compute_longmemeval_metrics() currently reports cutoff-level quality like:

metrics_by_cutoff[label] = {
    "overall": {
        "total": total,
        "correct": correct,
        "accuracy": correct / total * 100 if total else 0.0,
        "avg_score": statistics.mean(scores) * 100 if scores else 0.0,
    },
    "by_question_type": type_metrics,
}

Per-question outputs already carry the fields needed for temporal diagnostics:

{
    "question_date": "2023/04/10 (Mon) 10:15",
    "retrieval": {
        "search_results": [
            {
                "memory": "...",
                "score": 0.82,
                "created_at": "2023-04-10T21:32:00Z",
                "updated_at": "2023-04-10T21:32:00Z",
            }
        ]
    },
}

Possible output shape

A first pass could add a deterministic diagnostic block per cutoff:

metrics_by_cutoff[label] = {
    "overall": {...},
    "by_question_type": type_metrics,
    "temporal_retrieval": {
        "timestamp_coverage": 0.94,
        "future_retrieval_rate": 0.02,
        "avg_memory_age_days": 18.7,
    },
}

Where:

  • timestamp_coverage = fraction of retrieved memories with a parseable timestamp;
  • future_retrieval_rate = fraction of retrieved memories dated after question_date;
  • avg_memory_age_days = average age of retrieved memories relative to question_date, when timestamps are available.

I would avoid calling this semantic "staleness" in the first version. This is only timestamp-based retrieval validity, not proof that an old memory is obsolete.

Proposed first scope

  • LongMemEval only.
  • Pure benchmark-side metric computation.
  • No changes to Mem0 search behavior.
  • No LLM judge for temporal validity.
  • Handle missing/invalid timestamps without failing the benchmark run.

This seems complementary to PR #12: that PR adds deterministic failure-mode scenarios, while this would add aggregate temporal retrieval diagnostics to LongMemEval benchmark outputs.

Open question

Would maintainers prefer this to start as:

  1. a shared helper in benchmarks/common/metrics.py, wired first into LongMemEval, or
  2. a LongMemEval-local helper first, generalized later if useful?

If this direction sounds right, I can pick up the first PR.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions