Skip to content

bench: add NDCG metrics and direct retrieval objective #172

Description

@Lucas-Bur

Parent: #166

Add rank-sensitive evaluation without weakening coverage guardrails.

Acceptance criteria:

  • Report NDCG@5, NDCG@10, NDCG@20, and NDCG@50 by query form, repository, model, fusion, aggregate, and holdout.
  • Define binary gain for resolved exact file-plus-symbol targets, duplicate/overlap handling, and zero-ideal-DCG behavior.
  • Compare NDCG@5-first direct optimization with the current Recall-first objective under identical folds, guardrails, and seeds.
  • Keep Recall@20/50 and ContextRecall@4096 as direct-retrieval guardrails.
  • Keep reranker objectives recall-oriented unless a separate ordering contract is introduced.

This issue remains benchmark-only and does not change Production retrieval behavior.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions