Lineage 2.0

Deterministic local citation graph construction from research PDFs.

PDFs → bibliography extraction → reference splitting → evidence parsing → paper resolution → citation graph

The PDF is the source of truth. Lineage does not use an LLM, Semantic Scholar, embeddings, fuzzy title matching, or network lookup.

Pipeline

1. Document ingest

ingest.py reads PDFs from docs/ and identifies each exact document by:

sha256(pdf bytes)

Document identity and paper identity are separate. Renaming a PDF does not change its document identity, while different PDF versions can reconcile to the same paper.

Document-to-paper reconciliation is deterministic, in this order:

1. exact DOI
2. exact arXiv base ID
3. exact normalized title + compatible ordered authors

arXiv versions such as v1 and v3 belong to document/reference evidence; the arXiv base ID is the paper identity.

2. Bibliography extraction

bib_extract.py reads the selectable PDF text with pypdf, locates a References or Bibliography heading, and extracts the bibliography up to a recognized following appendix/section boundary.

3. Reference splitting

split_refs.py turns the bibliography into individual raw references.

It supports:

• bracket-numbered bibliographies such as [1] ... [2] ...
• author-year bibliographies using deterministic line-head detection

Raw reference text is retained in .cache/refs.json.

4. Evidence parsing

For each reference, ingest.py derives locally available evidence including:

• DOI
• arXiv base ID and version
• publication year
• title
• ordered authors
• whether the author list is complete

Reference parsing includes deterministic cleanup for PDF extraction artifacts such as line-wrap hyphenation. Raw stored reference text remains unchanged; cleanup affects derived evidence only.

5. Reference resolution

References with strong identifiers resolve by exact arXiv or DOI evidence.

A reference without a strong identifier may resolve through title_authors_anchor only when all of the following hold:

• the reference has an exact normalized title
• the reference has a complete ordered author list
• exactly one existing strong-ID-anchored paper has that title and compatible ordered authors

Title/author anchoring reuses an existing strong paper identity; it does not create a paper identity from title and authors alone.

Strong DOI/arXiv conflicts are never merged through title/author matching.

6. Graph rebuild

The graph is rebuilt from the current resolved references. Each directed edge is:

source paper → cited paper

Only current valid strong-ID or anchored resolutions contribute edges.

Identity normalization

Title normalization applies Unicode NFKC normalization, dash normalization, case folding, whitespace normalization, and removal of presentation punctuation while preserving meaningful technical punctuation.

Author normalization preserves ordered surname + first-given-name/initial evidence. Full given names must agree when both are available; an initial may match the corresponding full given name.

Local stores

All generated state lives under .cache/:

• lineage2.json — exact PDF documents and their paper assignments
• papers.json — reconciled paper identities and attached document hashes
• refs.json — split references, parsed evidence, and resolution state
• graph.json — current citation graph

The PDF corpus in docs/ and the generated cache are local and git-ignored.

Project layout

• bib_extract.py — bibliography extraction
• split_refs.py — deterministic reference splitting
• reconcile.py — document/paper identity normalization and reconciliation
• ingest.py — ingest, evidence parsing, reference resolution, cache migration, graph rebuild
• tests/ — regression and invariants tests

Usage

Python 3.10+ and pypdf are required.

pip install pypdf

python ingest.py            # refresh docs/
python ingest.py docs/      # explicit PDF directory
python ingest.py --clear    # rebuild local stores from scratch

python -m unittest discover -s tests

Repeated ingest of unchanged PDF bytes is idempotent: documents are not duplicated, current reference state is reused or migrated by cache version, and the citation graph is rebuilt from current evidence.