Skip to content
jerecoderPublic

Repository files navigation

IReranker

RAG reranking with IR.

Structure

├─ LICENSE
├─ Makefile
├─ README.md
├─ pyproject.toml
├─ config/
│  ├─ beir_eval.json       # Evaluation config (datasets, k_values, output_dir, etc.)
│  └─ beir_loader.json     # BEIR loader config (base_url, cache_subdir)
├─ data/
│  └─ external/            # BEIR datasets cache
├─ notebooks/
├─ references/
├─ reports/
│  └─ beir-metrics/        # Example outputs per dataset (CSV); regenerated
├─ ireranker/
│  ├─ __init__.py
│  ├─ config.py            # Paths and logging
│  ├─ types.py             # RankingTask, RankingDataset
│  ├─ plots.py             # NDCG@k plotting from summary.csv
│  ├─ run_beir_eval.py     # CLI/runner for BEIR evaluation
│  ├─ rankers/
│  │  ├─ ranker.py
│  │  ├─ bubble_ranker.py
│  │  ├─ mohajer_ranker.py
│  │  ├─ random_ranker.py
│  │  └─ registry.py
│  ├─ data/
│  │  └─ loaders.py        # Loader BEIR
│  └─ evaluation/
│     └─ beir.py           # BEIR evaluation
└─ tests/
   ├─ test_beir_eval_module.py
   ├─ test_metrics.py
   └─ ...

Installation with conda (recommended)

  • make create_environment (creates IReranker env with Python 3.10)
  • conda activate IReranker

Useful commands inside the IReranker env:

  • Install light dev deps for lint/tests: make requirements or make requirements-test (installs .[tests,lint])
  • Install full stack (tests/lint/notebooks + BEIR extras): make requirements-dev (installs .[dev,beir])
  • Lint/format: make lint / make format
  • Tests: make test
  • BEIR evaluation (uses configs): make beir-eval
  • Runtime-only install (no dev tools): pip install .

Configuration

  • config/beir_eval.json

    • datasets: list of BEIR datasets to evaluate (canonical names)

    • split: BEIR split (e.g., "test")

    • light_exclude: datasets to skip when running with --light

    • seed: RNG seed

    • k_values: evaluation cutoffs, e.g., [1,3,5,10,100]

    • output_dir: results destination (absolute or relative to REPORTS_DIR)

    • matrix_models: list of rerank matrix model filters (matches rerank filename/path, e.g., "flan-t5-large"). Each model is evaluated separately and always gets its own output folder/README tables. If omitted, models are auto-discovered from the available rerank matrices.

    • rankers: ["all"] or a list of names

    • Current example (config/beir_eval.json):

      {
        "datasets": [
          "webis-touche2020",
          "nfcorpus",
          "trec-covid",
          "scifact",
          "fiqa",
          "dbpedia-entity"
        ],
        "split": "test",
        "seed": 42,
        "k_values": [1, 3, 5, 10, 100],
        "light_exclude": ["dbpedia-entity", "fiqa"],
        "output_dir": "beir-metrics",
        "matrix_models": ["flan-t5-large", "flan-t5-xl"],
        "rankers": ["all"]
      }
  • config/beir_loader.json

    • base_url: BEIR base URL
    • cache_subdir: sub-folder under data/external/

Usage

  • View/adjust datasets via config/beir_eval.json.

  • Run evaluation (uses config):

    • make beir-eval
    • or python -m ireranker.run_beir_eval
  • Overrides:

    • make beir-eval ARGS="--dataset webis-touche2020"
    • make beir-eval ARGS="--light" to skip datasets from light_exclude
    • make beir-eval ARGS="--max-queries 20 --rankers random" for a quick/safe run
    • make beir-eval ARGS="--profile-out reports/profiles/beir_eval.prof --skip-readme-update" to profile without touching docs
    • make beir-eval ARGS="--matrix-models flan-t5-large,flan-t5-xl" to evaluate specific rerank matrix models (one output folder/table per model); omit only if models are set in config or auto-discovery suffices
    • python -m ireranker.run_beir_eval --config /path/to/custom.json

Output

Per dataset: CSV summary.csv with one row per ranker & k

  • Columns: ranker,k,NDCG,MAP,Recall,Precision,Comparisons,NDCG_per_comp (NDCG_per_comp is NDCG divided by comparisons)
  • Directory: always reports/beir-metrics/<model>/<dataset>/ (or <output_dir>/<model>/<dataset>/) so results are separated per rerank model

Notes

  • Existing dataset directories are reused (no re-download).
  • After extraction the ZIP is removed to save space.
  • Download errors are logged and the dataset creates an ERROR.txt so the run continues.
  • Low-memory runs: pass --max-queries <N> (and optionally --dataset .../--light) to prune rerank matrices to that many queries and avoid loading multiple copies in RAM.
  • BubbleRanker is implemented but intentionally not registered by default because it is expensive; import and instantiate it directly if you need it.
  • reports/beir-metrics/ contains example outputs; you can delete or regenerate them with make beir-eval.
  • Plot NDCG@k from a summary.csv: python -m ireranker.plots ndcg-bar nfcorpus --k 10.

BEIR results

Tables auto-updated after each BEIR evaluation.

Limit Comparisons Experiment

Results from limit comparisons experiments will appear here.

Releases

Packages

Contributors

Languages