RAG reranking with IR.
├─ LICENSE
├─ Makefile
├─ README.md
├─ pyproject.toml
├─ config/
│ ├─ beir_eval.json # Evaluation config (datasets, k_values, output_dir, etc.)
│ └─ beir_loader.json # BEIR loader config (base_url, cache_subdir)
├─ data/
│ └─ external/ # BEIR datasets cache
├─ notebooks/
├─ references/
├─ reports/
│ └─ beir-metrics/ # Example outputs per dataset (CSV); regenerated
├─ ireranker/
│ ├─ __init__.py
│ ├─ config.py # Paths and logging
│ ├─ types.py # RankingTask, RankingDataset
│ ├─ plots.py # NDCG@k plotting from summary.csv
│ ├─ run_beir_eval.py # CLI/runner for BEIR evaluation
│ ├─ rankers/
│ │ ├─ ranker.py
│ │ ├─ bubble_ranker.py
│ │ ├─ mohajer_ranker.py
│ │ ├─ random_ranker.py
│ │ └─ registry.py
│ ├─ data/
│ │ └─ loaders.py # Loader BEIR
│ └─ evaluation/
│ └─ beir.py # BEIR evaluation
└─ tests/
├─ test_beir_eval_module.py
├─ test_metrics.py
└─ ...
make create_environment(createsIRerankerenv with Python 3.10)conda activate IReranker
Useful commands inside the IReranker env:
- Install light dev deps for lint/tests:
make requirementsormake requirements-test(installs.[tests,lint]) - Install full stack (tests/lint/notebooks + BEIR extras):
make requirements-dev(installs.[dev,beir]) - Lint/format:
make lint/make format - Tests:
make test - BEIR evaluation (uses configs):
make beir-eval - Runtime-only install (no dev tools):
pip install .
-
config/beir_eval.json-
datasets: list of BEIR datasets to evaluate (canonical names) -
split: BEIR split (e.g., "test") -
light_exclude: datasets to skip when running with--light -
seed: RNG seed -
k_values: evaluation cutoffs, e.g., [1,3,5,10,100] -
output_dir: results destination (absolute or relative to REPORTS_DIR) -
matrix_models: list of rerank matrix model filters (matches rerank filename/path, e.g.,"flan-t5-large"). Each model is evaluated separately and always gets its own output folder/README tables. If omitted, models are auto-discovered from the available rerank matrices. -
rankers: ["all"] or a list of names -
Current example (
config/beir_eval.json):{ "datasets": [ "webis-touche2020", "nfcorpus", "trec-covid", "scifact", "fiqa", "dbpedia-entity" ], "split": "test", "seed": 42, "k_values": [1, 3, 5, 10, 100], "light_exclude": ["dbpedia-entity", "fiqa"], "output_dir": "beir-metrics", "matrix_models": ["flan-t5-large", "flan-t5-xl"], "rankers": ["all"] }
-
-
config/beir_loader.jsonbase_url: BEIR base URLcache_subdir: sub-folder underdata/external/
-
View/adjust datasets via
config/beir_eval.json. -
Run evaluation (uses config):
make beir-eval- or
python -m ireranker.run_beir_eval
-
Overrides:
make beir-eval ARGS="--dataset webis-touche2020"make beir-eval ARGS="--light"to skip datasets fromlight_excludemake beir-eval ARGS="--max-queries 20 --rankers random"for a quick/safe runmake beir-eval ARGS="--profile-out reports/profiles/beir_eval.prof --skip-readme-update"to profile without touching docsmake beir-eval ARGS="--matrix-models flan-t5-large,flan-t5-xl"to evaluate specific rerank matrix models (one output folder/table per model); omit only if models are set in config or auto-discovery sufficespython -m ireranker.run_beir_eval --config /path/to/custom.json
Per dataset: CSV summary.csv with one row per ranker & k
- Columns:
ranker,k,NDCG,MAP,Recall,Precision,Comparisons,NDCG_per_comp(NDCG_per_compis NDCG divided by comparisons) - Directory: always
reports/beir-metrics/<model>/<dataset>/(or<output_dir>/<model>/<dataset>/) so results are separated per rerank model
- Existing dataset directories are reused (no re-download).
- After extraction the ZIP is removed to save space.
- Download errors are logged and the dataset creates an
ERROR.txtso the run continues. - Low-memory runs: pass
--max-queries <N>(and optionally--dataset .../--light) to prune rerank matrices to that many queries and avoid loading multiple copies in RAM. BubbleRankeris implemented but intentionally not registered by default because it is expensive; import and instantiate it directly if you need it.reports/beir-metrics/contains example outputs; you can delete or regenerate them withmake beir-eval.- Plot NDCG@k from a
summary.csv:python -m ireranker.plots ndcg-bar nfcorpus --k 10.
Tables auto-updated after each BEIR evaluation.
Results from limit comparisons experiments will appear here.