Skip to content

Open Tree of Life taxonomic backbone and curated role tooling - #3

Open
vbeefy wants to merge 1 commit into
mainfrom
publish/ott-taxonomy
Open

vbeefy wants to merge 1 commit into
mainfrom
publish/ott-taxonomy

Conversation

@vbeefy

@vbeefy vbeefy commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Adds an external taxonomic reference — the Open Tree of Life taxonomy — and the curation tooling built on top of it.

Why an external reference

AEDIN stores six denormalized rank columns per entity (kingdom, phylum, taxon_class, taxon_order, family, genus) with no parent–child edges, so it has no taxonomic tree of its own.

Three earlier attempts to find taxonomy defects by checking the data against itself failed the same way: the field used as the reference turned out to be no more reliable than the field under test. A field that was never derived from AEDIN's own values cannot inherit their errors, so OTT is loaded as a pinned, rebuildable, gitignored third database and used strictly as a comparison reference — the resolution pass writes only ott_id and ott_status and corrects nothing on its own.

What's here

Backbonelib/ott-{release,tsv,lookup,lineage,resolve,tnrs-fallback}.js

A pinned-release loader (never a current/ symlink — a citable knowledge base must be able to say which reference version it reconciled against) and a disambiguate-or-abstain resolver.

Both taxonomy.tsv and synonyms.tsv are mandatory, and this is a correctness requirement rather than extra coverage: the plant Ficus variegata is reachable only through the synonyms file, while the bare binomial in taxonomy.tsv resolves to a mollusc. Searching taxonomy alone would return that mollusc as a single confident candidate with no ambiguity signal — manufacturing the exact genus-collision error the subsystem exists to detect.

Collision-breaking uses curated genus name lists only — never bio_category, the stored kingdom, or claim context, since in precisely the collision cases that matter those fields are derived from the value under test. Where a candidate's kingdom cannot be determined, the resolver abstains rather than guessing.

Kingdom reconciliationlib/{kingdom-align,kingdom-consensus}.js

Compares stored bio_category against OTT's kingdom and splits the result into two classes that are deliberately not treated alike. Blank values are filled. Contradictions are not auto-applied: a genus name can sit in two kingdoms, and an external authority can resolve the wrong namesake as readily as we can (Graphium is both a swallowtail butterfly and an anamorphic ascomycete; Sphaeroderma both a flea beetle and a fungus).

The consensus gate blocks on any dissent rather than taking a majority — on collision rows a lone dissenting reviewer is usually recognising the other namesake, which makes it the most informative signal rather than noise to be averaged away.

Metazoa is resolved to vertebrate/invertebrate by walking the ancestry for the named clade Vertebrata. Testing on Chordata instead would classify tunicates and lancelets as vertebrates.

Rank reconciliationlib/{order-canonical,order-consistency,subfamily-source}.js

Handles the case where two sources carry the same taxon at different ranks because they follow classifications of different ages — and where a demotion renames as well as re-ranks (-idae-inae), so looking the unchanged string up at another rank finds nothing.

Role curationlib/{family-triage,family-verdicts,rule-reach,stale-role}.js, role-engine.js

Family-rank role rules are gated on domain review rather than on the tooling: triage produces evidence, never authority. verdictToRules refuses any verdict carrying exceptions, because the rule store cannot express "assert for the family except these genera" without a genus rule that would pre-empt evidence-based classification entirely.

role-engine gains a subfamily tier ranked above family, for facts that are true at subfamily rank and false at family rank — Miletinae larvae are predators of aphids and scale insects inside the otherwise herbivorous Lycaenidae, so a family-keyed rule files a biological control agent as a crop pest.

rule-reach audits how many entities each rule actually reaches, in both directions: rules that have gone inert because their taxon was demoted, and rules that silently widened when a family absorbed another group.

Safety properties

Every mutating script is dry-run by default, writes exactly one column, logs to revision_log, and supports --undo.

Tests

834 tests, 0 failures.

Adds an external taxonomic reference and the curation tooling built on it.

AEDIN stores six denormalized rank columns per entity with no parent-child
edges, so it has no tree of its own. Three previous attempts to find taxonomy
defects by checking the data against ITSELF failed the same way: the reference
field turned out to be no more reliable than the field under test. A field that
was never derived from AEDIN's own values cannot inherit their errors, so the
Open Tree Taxonomy is loaded as a pinned, rebuildable, gitignored third
database and used strictly as a comparison reference.

  lib/ott-{release,tsv,lookup,lineage,resolve,tnrs-fallback}.js
      Pinned-release loader and a disambiguate-or-abstain resolver. Both
      taxonomy.tsv and synonyms.tsv are mandatory: the plant Ficus variegata is
      reachable only through synonyms, while the bare binomial in taxonomy.tsv
      resolves to a mollusc. Collision-breaking uses curated genus NAME lists
      only - never bio_category, stored kingdom or claim context, all of which
      are derived from the value under test in exactly the collision cases that
      matter. Where the kingdom cannot be determined, it abstains.

  lib/{kingdom-align,kingdom-consensus}.js
      Compare stored bio_category against OTT's kingdom. Blank values are
      filled; contradictions are NOT auto-applied but routed to independent
      domain critics, because a genus name can sit in two kingdoms and an
      external authority can resolve the wrong namesake as readily as we can.
      The consensus gate blocks on ANY dissent rather than taking a majority:
      on collision rows a lone dissenter is usually recognising the other
      namesake. Metazoa is resolved to vertebrate/invertebrate by walking the
      ancestry for the named clade Vertebrata - testing on Chordata instead
      would label tunicates vertebrates.

  lib/{order-canonical,order-consistency,subfamily-source}.js
      Order-name and subfamily reconciliation. Handles the case where AEDIN and
      the reference carry the SAME taxon at two different ranks because they
      follow classifications of different ages, and a demotion renames as well
      as re-ranks (-idae -> -inae).

  lib/{family-triage,family-verdicts,rule-reach,stale-role}.js, role-engine.js
      Family-rank role curation, gated on domain critics rather than on the
      tooling: triage produces evidence and never authority. verdictToRules
      REFUSES a verdict carrying exceptions, because role_rules cannot express
      "assert for the family except these genera" without a genus rule that
      would pre-empt evidence-based classification. role-engine gains a
      subfamily tier above family, for facts true at subfamily rank and false
      at family rank. rule-reach audits how many entities each rule actually
      reaches, in both directions: rules inert because their taxon was demoted,
      and rules that silently WIDENED when a family absorbed another group.

Every mutating script is dry-run by default, writes one column, logs to
revision_log, and supports --undo.

834 tests, 0 failures.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant