Conversation
Adds an external taxonomic reference and the curation tooling built on it.
AEDIN stores six denormalized rank columns per entity with no parent-child
edges, so it has no tree of its own. Three previous attempts to find taxonomy
defects by checking the data against ITSELF failed the same way: the reference
field turned out to be no more reliable than the field under test. A field that
was never derived from AEDIN's own values cannot inherit their errors, so the
Open Tree Taxonomy is loaded as a pinned, rebuildable, gitignored third
database and used strictly as a comparison reference.
lib/ott-{release,tsv,lookup,lineage,resolve,tnrs-fallback}.js
Pinned-release loader and a disambiguate-or-abstain resolver. Both
taxonomy.tsv and synonyms.tsv are mandatory: the plant Ficus variegata is
reachable only through synonyms, while the bare binomial in taxonomy.tsv
resolves to a mollusc. Collision-breaking uses curated genus NAME lists
only - never bio_category, stored kingdom or claim context, all of which
are derived from the value under test in exactly the collision cases that
matter. Where the kingdom cannot be determined, it abstains.
lib/{kingdom-align,kingdom-consensus}.js
Compare stored bio_category against OTT's kingdom. Blank values are
filled; contradictions are NOT auto-applied but routed to independent
domain critics, because a genus name can sit in two kingdoms and an
external authority can resolve the wrong namesake as readily as we can.
The consensus gate blocks on ANY dissent rather than taking a majority:
on collision rows a lone dissenter is usually recognising the other
namesake. Metazoa is resolved to vertebrate/invertebrate by walking the
ancestry for the named clade Vertebrata - testing on Chordata instead
would label tunicates vertebrates.
lib/{order-canonical,order-consistency,subfamily-source}.js
Order-name and subfamily reconciliation. Handles the case where AEDIN and
the reference carry the SAME taxon at two different ranks because they
follow classifications of different ages, and a demotion renames as well
as re-ranks (-idae -> -inae).
lib/{family-triage,family-verdicts,rule-reach,stale-role}.js, role-engine.js
Family-rank role curation, gated on domain critics rather than on the
tooling: triage produces evidence and never authority. verdictToRules
REFUSES a verdict carrying exceptions, because role_rules cannot express
"assert for the family except these genera" without a genus rule that
would pre-empt evidence-based classification. role-engine gains a
subfamily tier above family, for facts true at subfamily rank and false
at family rank. rule-reach audits how many entities each rule actually
reaches, in both directions: rules inert because their taxon was demoted,
and rules that silently WIDENED when a family absorbed another group.
Every mutating script is dry-run by default, writes one column, logs to
revision_log, and supports --undo.
834 tests, 0 failures.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds an external taxonomic reference — the Open Tree of Life taxonomy — and the curation tooling built on top of it.
Why an external reference
AEDIN stores six denormalized rank columns per entity (
kingdom,phylum,taxon_class,taxon_order,family,genus) with no parent–child edges, so it has no taxonomic tree of its own.Three earlier attempts to find taxonomy defects by checking the data against itself failed the same way: the field used as the reference turned out to be no more reliable than the field under test. A field that was never derived from AEDIN's own values cannot inherit their errors, so OTT is loaded as a pinned, rebuildable, gitignored third database and used strictly as a comparison reference — the resolution pass writes only
ott_idandott_statusand corrects nothing on its own.What's here
Backbone —
lib/ott-{release,tsv,lookup,lineage,resolve,tnrs-fallback}.jsA pinned-release loader (never a
current/symlink — a citable knowledge base must be able to say which reference version it reconciled against) and a disambiguate-or-abstain resolver.Both
taxonomy.tsvandsynonyms.tsvare mandatory, and this is a correctness requirement rather than extra coverage: the plant Ficus variegata is reachable only through the synonyms file, while the bare binomial intaxonomy.tsvresolves to a mollusc. Searching taxonomy alone would return that mollusc as a single confident candidate with no ambiguity signal — manufacturing the exact genus-collision error the subsystem exists to detect.Collision-breaking uses curated genus name lists only — never
bio_category, the stored kingdom, or claim context, since in precisely the collision cases that matter those fields are derived from the value under test. Where a candidate's kingdom cannot be determined, the resolver abstains rather than guessing.Kingdom reconciliation —
lib/{kingdom-align,kingdom-consensus}.jsCompares stored
bio_categoryagainst OTT's kingdom and splits the result into two classes that are deliberately not treated alike. Blank values are filled. Contradictions are not auto-applied: a genus name can sit in two kingdoms, and an external authority can resolve the wrong namesake as readily as we can (Graphium is both a swallowtail butterfly and an anamorphic ascomycete; Sphaeroderma both a flea beetle and a fungus).The consensus gate blocks on any dissent rather than taking a majority — on collision rows a lone dissenting reviewer is usually recognising the other namesake, which makes it the most informative signal rather than noise to be averaged away.
Metazoais resolved to vertebrate/invertebrate by walking the ancestry for the named cladeVertebrata. Testing onChordatainstead would classify tunicates and lancelets as vertebrates.Rank reconciliation —
lib/{order-canonical,order-consistency,subfamily-source}.jsHandles the case where two sources carry the same taxon at different ranks because they follow classifications of different ages — and where a demotion renames as well as re-ranks (
-idae→-inae), so looking the unchanged string up at another rank finds nothing.Role curation —
lib/{family-triage,family-verdicts,rule-reach,stale-role}.js,role-engine.jsFamily-rank role rules are gated on domain review rather than on the tooling: triage produces evidence, never authority.
verdictToRulesrefuses any verdict carrying exceptions, because the rule store cannot express "assert for the family except these genera" without a genus rule that would pre-empt evidence-based classification entirely.role-enginegains a subfamily tier ranked above family, for facts that are true at subfamily rank and false at family rank — Miletinae larvae are predators of aphids and scale insects inside the otherwise herbivorous Lycaenidae, so a family-keyed rule files a biological control agent as a crop pest.rule-reachaudits how many entities each rule actually reaches, in both directions: rules that have gone inert because their taxon was demoted, and rules that silently widened when a family absorbed another group.Safety properties
Every mutating script is dry-run by default, writes exactly one column, logs to
revision_log, and supports--undo.Tests
834 tests, 0 failures.