report cited arXiv preprints that have since been published by ayush1298 · Pull Request #5722 · embeddings-benchmark/mteb

ayush1298 · 2026-10-20T04:00:12Z

Adds a monthly check that lists works we cite as arXiv preprints that have since been published, and opens or updates an issue listing them.

scripts/tools/check_preprints.py: finds the citations that only cite arXiv (387 on main today) and looks them all up in one Semantic Scholar batch request. Preprints Semantic Scholar knows no published version of are also looked up in OpenAlex, since neither index links every preprint to its published version.
- A DOI is reported only if Crossref holds it under the cited title, which rejects OpenAlex's wrong merges and conference-talk DOIs.
- A venue without a DOI (e.g. ICLR, NeurIPS) is reported only if the titles match.
- Links point to the DOI, the ACL Anthology, or Semantic Scholar, in that order.
- It reuses citations() from check_citations.py and refaudit's Crossref and OpenAlex clients.
check_citations.py, citation_cache.json and citation_allowlist.json move from tests/test_validate_metadata/ to scripts/tools/, since they are scripts rather than pytest tests. The JSON files are moved unchanged. Both checks run as modules (python -m scripts.tools.check_citations).
citation_check.yml: new preprint-check and report-preprints jobs instead of a separate workflow. They run monthly and on manual trigger, never on PRs, and use the SEMANTIC_SCHOLAR_API_KEY secret. A separate job with issues: write opens or updates the issue.
scripts/tools/preprint_allowlist.json: citations that should stay on the preprint. It lists MS MARCO and Jina Embeddings 2.
pyproject.toml: refaudit moves from the test group to a new citations group (part of dev and the default groups), used by make citation-check, make preprint-check and the citation workflow.
make preprint-check runs the check locally.

A local run found about 210 published preprints. Their citations are updated in #5724.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

Samoed · 2026-10-20T06:08:47Z

A local run finds about 200 published preprints. Their citations will be updated in follow-up PRs.

I think better to update before pr

ayush1298 · 2026-10-20T09:51:30Z

I think better to update before pr

open PR for this #5724

KennethEnevoldsen · 2026-10-20T15:20:27Z

+      with:
+        name: preprints
+
+    - name: Open or update the issue


instead of making an issue should we instead make a PR?

I kept issue because there is some part which can be automated and some which cant:

What it automates: for findings with a DOI or ACL Anthology id, it would fetch the BibTeX, keep each key, run the formatter and the citation check, and open a PR.

What it can't automate: findings with only a venue (ICLR, NeurIPS). OpenReview and DBLP now block automated requests, so those would be listed in the PR description and we have to fix them again

So, I was leaning towards more on keeping issue, but will change to PR if you want

emrehangorgec · 2026-10-20T16:22:03Z

Hi @ayush1298, would offline tests for check_preprints.py be useful here? I can cover arXiv DOI filtering, title mismatch rejection and API retry/error handling with mocked Semantic Scholar and Crossref responses, so the tests would run without API keys or network access.

ayush1298 · 2026-10-21T06:48:21Z

Hi @ayush1298, would offline tests for check_preprints.py be useful here? I can cover arXiv DOI filtering, title mismatch rejection and API retry/error handling with mocked Semantic Scholar and Crossref responses, so the tests would run without API keys or network access.

I dont think mock is required here, I have added some basic tests, but no HTTP call and use of API Key

Samoed · 2026-10-27T13:51:10Z

    "tabulate>=0.9.0",
    "bibtexparser>=2.0.0,<3.0.0", # used for tests/test_citation_formatting.py
-    "refaudit>=0.4.9", # used for tests/test_validate_metadata/check_citations.py
+    "refaudit>=0.4.9", # used for scripts/tools/check_citations.py and scripts/tools/check_preprints.py


Small note. Overall I think this is a good library, but I'm a bit afraid that it won't work after some time, because it have only one maintainer and not have any stars on github

check_citations.py uses most part of this library, maybe we can refactor it then if possible

Samoed · 2026-10-27T13:51:51Z

+from scripts.tools.check_preprints import (
+    Version,
+    confirm,
+    parse_openalex_work,
+    parse_semantic_scholar_paper,
+    preprint_arxiv_id,
+)


I don't think that tests should import from scripts

where we should add it then?

Do we need tests for that?

I guess not necessary, mostly those are sanity checks kind of tests

I have remove the test file

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

ayush1298 added 3 commits October 18, 2026 18:42

ci: report cited arXiv preprints that have since been published

9d44dca

update to use SEMANTIC_SCHOLAR_API_KEY

26f9a5e

update the script to get links of published version

2a76e62

Copilot AI balanced review requested due to automatic review settings October 20, 2026 04:00

Copilot AI reviewed Oct 20, 2026

ayush1298 requested review from KennethEnevoldsen and Samoed October 20, 2026 04:11

Samoed reviewed Oct 20, 2026

View reviewed changes

Comment thread scripts/tools/check_preprints.py

Comment thread .github/workflows/preprint_check.yml Outdated

ayush1298 mentioned this pull request Oct 20, 2026

fix: cite the published versions of preprints that have been published #5724

Merged

combine workflows, update allowlist and moved to scripts/tools

8a88b6f

KennethEnevoldsen reviewed Oct 20, 2026

View reviewed changes

add OpenAlex fallback and move other file under scripts/tools

d8d1948

added tests

1d1d627

ayush1298 requested review from KennethEnevoldsen and Samoed October 24, 2026 14:49

Samoed reviewed Oct 27, 2026

View reviewed changes

ayush1298 and others added 5 commits October 27, 2026 19:32

Merge origin/main into add_citation_update_test

5e75459

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

add citation group

614637f

update lock file

751382d

skip test

2245f84

remove test

09094b8

Conversation

ayush1298 commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Samoed commented Oct 20, 2026

Uh oh!

Uh oh!

Uh oh!

ayush1298 commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

emrehangorgec commented Oct 20, 2026

Uh oh!

ayush1298 commented Oct 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ayush1298 commented Oct 20, 2026 •

edited

Loading

ayush1298 commented Oct 20, 2026 •

edited

Loading

ayush1298 commented Oct 21, 2026 •

edited

Loading