Skip to content

No ingest path extracts equations, so find_equation's TED route never fires #550

Open

Description

@chris-dare-dev

Severity medium · Subsystem ingest

Problem

find_equation's tree-edit-distance route (retrieval_mode="ted_fused") needs an equations table whose rows carry mathml_tree_json. Two steps produce that:

  1. ingest/extract_equations.py extracts equation atoms from a paper's LaTeXML HTML, one paper at a time from its CLI.
  2. python -m ingest.index_equations (feat(ingest): give the derived indexes working CLIs #541) builds the trees.

Neither run_bulk_ingest, run_delta nor tools/notebook_ingest.py calls step 1. On a corpus built the normal way there is no equations table, so find_equation always degrades to dense_only_fallback, and the TED route the eval gate (tests/eval/test_equations_parity.py) measures is unreachable in production.

Found while scoping #310's second half (#549), which builds the definitions index during notebook ingest but deliberately not equation trees, since there is nothing to build them from.

Suggested remedy

Run equation extraction per paper inside the ingest loop: it needs the same LaTeXML HTML the chunker reads. Then build the trees once at the end of a notebook ingest, as #549 does for definitions.

For the staging bulk path, the equations table lives inside the LanceDB dataset, so it is promoted by cutover with the chunks. Unlike theorem names, it has no cutover hazard.

Done when

  • A freshly ingested notebook has an equations table with trees.
  • find_equation on MathML input reports ted_fused on it.
  • A test drives an ingest end to end and checks both.

Activity

  1. added
    type:findingAdversary/critique finding to remediate
    area:ingestIngest pipeline: fetch, parse, chunk, embed, index
    area:retrievalRetrieval lanes and the 3-tier cache
    on Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:ingestIngest pipeline: fetch, parse, chunk, embed, indexarea:retrievalRetrieval lanes and the 3-tier cachesev:mediumMedium severitytype:findingAdversary/critique finding to remediate

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions