TaxonBodyMass_DB

A compiled, harmonized database of species body masses (in grams) drawn from many published sources spanning the full diversity of heterotrophic life — from microbes to large vertebrates. Developed in support of FracFeed_DB and TaxonBodyMassML.

Data Sources

Body mass data are compiled from primary sources and databases covering organisms across all major taxonomic groups, including bacteria, protists, marine zooplankton and invertebrates, spiders, fish, amphibians, reptiles, birds, and mammals (modern and fossil). Sources include both ecological databases (food-web, metabolic-scaling, predator-prey, movement) and taxonomic trait databases (life-history, body size). Live downloads from FishBase and SeaLifeBase are supported via the rfishbase package.

Bibliographic information for all sources is in Bib/TaxonBodyMass_Citations.bib.

Data licences

The repository LICENSE is CC BY-NC 4.0 (Creative Commons Attribution-NonCommercial 4.0 International; owner decision of 2026-10-06, replacing the CC BY 4.0 of earlier commits and releases, which keep the licence they carried) and covers what the lab wrote: the pipeline code, the per-source parse scripts and READMEs, the parsed and transcribed tables, the compiled outputs (TaxonBodyMass.csv, TaxonBodyMass_GenusLevel.csv, TaxonBodyMass_Provenance.csv.gz) and the bibliographies, to the extent the lab holds rights in them. The NonCommercial term applies to all of it: the compiled tables, the bibliographies, the provenance table and the code in this repository may be shared and adapted with attribution for non-commercial purposes only; a commercial use needs separate permission from the owner. It does not relicense any third-party file: every raw source file kept under sources/databases/ and sources/conversion_factors/ stays under the terms of its licensor, recorded in the Licence: line of the folder README, in the tracked manifest sources/source_files.csv (one row per tracked file with size, SHA-256, role, licence class, attribution text and the page to check; regenerated by Rscript R/library/licence_manifest.r) and in sources/LICENSES.md, which is generated from the manifest (issue #6, licence audit of 2026-10-06). Three carve-outs from the repository licence: (1) the non-commercial and share-alike sources, whose files are redistributed here under their own terms only: FishBase and SeaLifeBase (CC BY-NC 4.0; downloaded at run time, nothing stored), Lane 2019 (CC BY-NC 4.0), Cai et al. 2025 and Hoehler et al. 2023 (CC BY-NC-ND 4.0), Animal Diversity Web through Quaardvark (CC BY-NC-SA 3.0), Pata & Hunt 2025 (CC BY-SA 4.0) and Hechinger et al. 2011 (non-commercial scientific use); the compiled species means that incorporate their values are published under CC BY-NC 4.0 as aggregated facts with attribution, which aligns the repository with the non-commercial term of these sources, and whether a downstream use is also bound by the share-alike (Pata & Hunt) or no-derivatives (Cai, Hoehler) terms is for the user of the values to decide (the sources behind every value are listed in source_mass and the provenance table, so they can be excluded); (2) publisher files (article PDFs and journal supplements of Wiley, Springer Nature, PNAS, Elsevier and Cambridge University Press), whose copyright is with the publishers: where the source data is available in a structured format (xlsx, csv, or a PDF whose tables are parseable), the data is extracted to a CSV file that is committed in place of the PDF; the publisher PDF itself is not committed but documented in the folder README (owner rule, issue #124, 2026-10-11; their copyright stays with the publishers, see sources/LICENSES.md); (3) the unpublished MOM v10.2 workbook (issue #4). Sources whose terms are not confirmed are marked Licence: unknown; owner to check <URL> (not yet reviewed) or Licence: unknown; owner unable to confirm <URL> (reviewed but no explicit licence found) and listed in sources/LICENSES.md and in reports/warnings_licences.md at every run; nothing is asserted from a deposit page that is not quoted on disk. CheckSourceDocs() stops a run when a source folder has no Licence: line or no row in the manifest. None of this is legal advice.

Pipeline

The pipeline is written in R. The main entry point is R/RunMe.r, which orchestrates all steps:

1. Per-source parsing — each source has its own script at sources/databases/<SourceName>/BodyMass_<SourceName>.r that reads raw data files, extracts taxon names and body masses, converts units to grams, and saves a cached .Rdata file to sources/Rdata/. Sources that report dry, ash-free dry or carbon mass are converted to wet grams with the group-specific factors in R/library/mass_conversion.r (ToWetMass()); each such script's README.md states the mass type reported by the source and the conversion group applied, and the CiteIDs of the conversion references are appended to the record's source_mass label (e.g. Kendall_etal_2019; Studier_1992) so they are cited alongside the source. Rows that the source itself flags as imputed, genus-averaged or taken from another species are removed in the parse scripts (DropImputed() in R/library/helpers.r) and logged to audit/imputed_rows.csv together with the records the raw-name rules of step 2 drop (life stages and small size classes) and the entries of the live download frames, which the download scripts save in audit/imputed_rows_live.csv (issue #40; the file is written after step 2, only when the parse scripts ran, and is the same whatever the download flags were); each source's README.md records this in an Imputed rows: line, which R/library/check_source_docs.r checks at every run, together with a Licence: line stating the terms of the folder's raw files (Licence: <class> (<where stated>; <URL>). <note>, or Licence: unknown; owner to check <URL>), which must agree with the manifest sources/source_files.csv (Data licences above; a folder without the line or without a manifest row stops the run, an unknown licence is warned about and written to reports/warnings_licences.md; exercised without network access by Rscript R/library/tests/test_licence_docs.R). These cached files are git-ignored and regenerated by setting recompile = TRUE in RunMe.r.

2. Taxon name cleaning — applied uniformly to every source before merging. The rule of the database is that a cleaned name has two parts, Genus_species, or one, Genus, and nothing but letters; every raw name that does not already look so is reduced to that form by an explicit rule, and a raw name no rule covers stops the run (issue #38):

   • R/library/fix_formatting.r — FixFormatting() first normalizes encoding (only strings that are not valid UTF-8 are converted from Latin-1; diacritics are then transliterated to ASCII, so Nausithoë rubra becomes Nausithoe_rubra) and strips the sex signs ♀/♂, keeping the record as the species' value (the two Tetrao urogallus ♀ records of Makarieva_2008 are Tetrao_urogallus records; issue #37). It then parses the name into its bracket groups and tokens and applies the tracked vocabulary audit/raw_name_patterns.csv (one regex per row with its scope, class, action, note and date; see audit/README.md): a capitalised word in brackets between genus and epithet is a subgenus and is removed (Jaera (Jaera) albifrons -> Jaera_albifrons; a genus written twice, Castor Castor canadensis, loses the repeat); sex marks ((F), Female, males, the signs) and form, strain, region ((colonial), UAB CTIP, Bulgaria) annotations are stripped and the record kept (the form written in place of the epithet, Brose_2005's Conochilus_colonial, folds to the genus-level record Conochilus; issue #48); of the Brose soil-web size classes the medium and large ones ({m}, {l}, {xl}, {xxl}, {xxxl}, large) are stripped and the record kept while the small ones ({xs}, {s}, small) are non-adult records and are dropped through DropImputed() (owner decision 2026-10-04); a species group marker (Polypedilum halterale group, Hylaeus modestus grp) is stripped and the record credited to the nominal species; a synonym or common name in brackets after the binomial and an authority (L., (Gmelin, 1789)) are stripped; a lowercase third token, with or without a rank marker (var., ssp.), is a subspecies or variety epithet and folds into the species (trinomial: Acanthiza pusilla apicalis -> Acanthiza_pusilla); a placeholder epithet (sp., spp., spec., indet., ssp, undefinable, undetermined, morphospecies codes such as sp2 or lassp8, species A, Unidentified) or an identification qualifier (cf., aff., nr.) leaves the function as the marker Genus_sp / Genus_cf (or the word as written: Genus_spp, Genus_spec, Genus_indet, Genus_nr), which fix_nontaxa.r removes, as before; life-stage annotations ((N1), (IV), (megalops), Juveniles, immature) make the record a non-adult and it is dropped through DropImputed() (scope rule of issue #8); hybrids (Anas platyrhynchos x rubripes -> Anas_platyrhynchos) and names listing two or more ambiguous alternatives (Empidonax traillii/alnorum -> Empidonax_traillii, Pipilo maculatus, ocai -> Pipilo_maculatus) are cut at the separator and credited to the first name written, a bare genus before the separator giving a genus-level record (owner decision 2026-10-04); the word hybrid in place of the epithet (Anas hybrid, VertNet) is a hybrid of unrecorded parentage and is credited to the bare genus the same way (issue #48; until then GBIF matched Anas_hybrid to Chloephaga hybrida); characters that cannot occur in a Latin name (symbols) are removed. Whitespace and capitalization are normalized throughout. Every name a rule touched is listed by class and source in reports/warnings_raw_names.md; a bracket group or trailing token no rule covers is an error: the name keeps its brackets and CheckRawNames() (below) stops the run, so a new pattern is classified by hand instead of being folded into a plausible binomial (which is how Crithidia (Strigomonas) oncopelti became the unresolvable Crithidia_strigomonas). Exercised without network access by Rscript R/library/tests/test_raw_name_rules.R (one fixture per class, the report, the format of the rename tables, 60 pinned ordinary names) and Rscript R/library/tests/test_fix_formatting_encoding.R
   • R/library/fix_misspellings.r — corrects known misspellings
   • R/library/fix_nontaxa.r — removes placeholder identifiers (the Genus_sp, _spp, _spec, _indet, _type, _cf, _aff, _nr, _unk markers, COMBINE's Not_recognised and the _fragment(s) / _immature(s) group labels of Hrycik_2024), functional-group labels and common names that the food-web compilations use as names (Bacteria, Ciliates, Plankton, the Chichewa fish names of the Lake Malawi web, ...; a one-token label would otherwise be filed as a genus-level record, issue #44), and other non-taxon entries. Exercised without network access by Rscript R/library/tests/test_nontaxa_labels.R
   • R/library/filter_extinct.r — removes extinct species (list in audit/extinct_taxa.csv, compiled by hand with R/library/build_extinct_taxa.r (not sourced by RunMe.r) from the status columns of MOM v10.2, PHYLACINE 1.2 and AVONET); the MOM and PHYLACINE scripts also drop their own extinct rows
   • R/library/check_taxon_names.r — three guards. CheckRawNames() runs right after FixFormatting() and stops the run with the list of raw names (and their sources and row counts) that carried brackets or trailing tokens the vocabulary does not cover (issue #38). CheckTaxonNames() runs after the chain and stops the run if any cleaned name is not Genus_species or Genus (whitespace, brackets, digits, a third token), which only a malformed replacement value in a rename table can produce; RunMe.r tells species-level from genus-level records by the underscore when it merges the sources, so such a name would be filed as a genus (issue #28). A unit test asserts that every key and value of fix_misspellings.r and every entry of fix_nontaxa.r has that format and that no key is duplicated (the fix_taxonomy_ranks.r keys match the species column, Genus epithet with a space, and are checked as such). CheckSheetTaxa() runs in step 3 on the rows of the lab Sheet, which are bound after the chain and so pass neither FixFormatting() nor CheckTaxonNames(): every Sheet taxon with a mass must be Genus_species or Genus, since the override matches Sheet names against the cleaned source names by equality; a three-part name (a subspecies) folds to the species with a warning, as the source trinomials do, and a cell with brackets, blanks, a digit, a lowercase genus or no name stops the run with the offending rows (taxon, mass, source) and a request to correct the Sheet (issue #48). Exercised without network access by Rscript R/library/tests/test_taxon_name_guard.R and test_raw_name_rules.R

3. Google Sheet override — lab-curated values (BM_data tab) take priority over the compiled sources. The tab is read from its tracked copy sources/BM_data_snapshot.csv (the Sheet's rows and columns, every cell as text; R/library/sheet_snapshots.r): a run with DataSheets = TRUE at the top of RunMe.r first downloads the three Sheet tabs the pipeline uses and rewrites their copies, printing for each the rows added, removed and changed against the committed file, and the run goes on; with DataSheets = FALSE (the committed default) the copies are read as they are, no Google token is needed, and the run says when each copy was last committed, so a stale copy is visible (issue #118). The outputs are the same whichever mode wrote the copies. The Sheet rows with a mass are first checked by CheckSheetTaxa() (R/library/check_taxon_names.r; section 4 of RunMe.r): every taxon must be a cleaned name, Genus_species or Genus, a three-part name folds to the species with a warning, and anything else stops the run with the offending rows (issue #48). The rows are then split by the rule that tells the two kinds of record apart throughout the pipeline, the underscore (R/library/sheet_override.r, issue #57): a species-level row (Genus_species) replaces all compiled records of that species, so the Sheet value is the species' only record; a genus-level row (a bare Genus) is a genus-only record: it replaces the sources' genus-only rows written as that bare name (matched, like the species rows, by equality of the cleaned name) and joins the genus-only path of step 6, where it is resolved at genus rank, filtered, combined per genus and source and de-duplicated like the sources' rows. A genus-level Sheet row does not touch the species-level values of the genus (Lepidostoma replaces the sources' Lepidostoma rows and leaves Lepidostoma_hirtum alone; the genus mean of step 6 averages both). Until #57 every Sheet row entered the species path, where a bare name was dropped as unresolved or, for Hypopomus, resolved by ITIS to the species Hypopomus artedi; the rows such names left in the species enrichment cache are dropped when the cache is loaded (step 4). Exercised without network access by Rscript R/library/tests/test_sheet_override.R.

4. Taxonomy enrichment and autotroph removal — every unique name is resolved once to an accepted species and its full kingdom-to-species classification through GBIF, NCBI, WoRMS, Catalogue of Life (the ChecklistBank name-usage matcher of the COL checklist, dataset/3LR/match/nameusage; until issue #103 the stage called the names-index endpoint and never resolved a name), ITIS and Wikidata in turn, with a final GBIF backfill of missing higher ranks (R/library/enrich_taxonomy.r; details in sources/passes/enrichment_pipeline.md). A later stage never replaces a species an earlier one set; the names Catalogue of Life resolves (names GBIF, NCBI and WoRMS lack) are listed in reports/warnings_taxonomy.md with the matched name, match type and status. The Catalogue of Life stage is exercised without network access by Rscript R/library/tests/test_col_stage.R on recorded responses. Results are cached in sources/enrich_cache.Rdata (object enrich_cache; the genus-rank resolutions of step 6 are cached in the same file as genus_cache) so re-runs only query new names; the cache holds species-level names only, and a bare name an earlier run left there (step 3, issue #57) is dropped when it is loaded. Manual corrections live in R/library/fix_taxonomy_ranks.r. Its Part 2b guards the autotroph filter that follows (issue #81): when an authority's kingdom is Plantae, Viridiplantae or Fungi but the lower ranks are an animal's (an exact homonym in two kingdoms, where GBIF abstains and NCBI answers with the plant, Myrmecia pyriformis the bull ant and a green alga; or a GBIF fuzzy match below confidence 90 into the other kingdom, Parus humilis -> Pyrus humilis), the row keeps the animal's ranks: the authority's kingdom is recorded in the cache column kingdom_conflict, the kingdom becomes Animalia, the plant ranks are cleared for rank inference, and the species is kept only when it is the input binomial (otherwise it is cleared and the name is unresolved). Such rows are listed in reports/warnings_taxonomy.md under Kingdom conflicts corrected before the autotroph filter, with their sources and record counts; before #81 the filter dropped them on the kingdom alone and no report saw them. The known cases have full Part 7 overwrites. Exercised without network access by Rscript R/library/tests/test_kingdom_conflicts.R. R/library/filter_autotrophs.r then removes autotrophs: the kingdoms Plantae, Viridiplantae and Fungi; predominantly photosynthetic phyla (Ochrophyta, Bacillariophyta, Haptophyta, Cryptophyta, Chlorophyta, Rhodophyta, Charophyta, Glaucophyta, Streptophyta, Euglenophyta, Cyanobacteria, Cyanobacteriota); and a curated list of photosynthetic or mixotrophic dinoflagellate and euglenid genera. Heterotrophic protists (ciliates, foraminifera, apicomplexans, kinetoplastids, amoebae, heterotrophic dinoflagellates and euglenids) are retained.

5. Merging, de-duplication and range filter — within each source, body masses are summarised as the geometric mean across records resolving to the same accepted species. Records are grouped by the source label (the first token of source_mass; the conversion CiteIDs appended after ; do not split a source), and the six VertNet dumps are pooled as one source because the traits dump re-exports specimens of the class dumps (every dump label stays in source_mass). Before this first pass the records are put in a fixed order (input name, source label, source_mass, value; byte order, OrderForPass1() in R/library/helpers.r): merge() leaves the records of a name in the order of the bound source frames, and the floating-point sum behind a geometric mean depends on that order, so without it the last digit of a few log10_range values moved whenever a source or a cleaning rule reordered rows (issue #53). Exercised without network access by Rscript R/library/tests/test_output_stability.R.

   The same datum often reaches the database through several compilations (AVONET and EltonTraits both reproduce Dunning's CRC Handbook; COMBINE reproduces Amniote, PHYLACINE and MOM; Hoehler et al. 2023 reproduces Makarieva et al. 2008, which reproduces Chown et al. 2007; Meiri 2018 applies the Feldman et al. 2016 equations to the same lengths, and so on). Averaging such copies as independent estimates inflates n and biases the cross-source mean, so R/library/dedupe_sources.r collapses them before Pass 2 (issue #5). Two layers combine: the hand-maintained dependency registry Bib/source_dependencies.csv (one row per edge child_label -> parent_label, see below) collapses a child value into its parent, or into a sibling that shares a parent outside the database, only when the two values also agree within the edge's tolerance (tol_log10: 1e-6 for verbatim copies, 0.0105 where the child re-rounded or re-derived the value); edges are transitive, and provenance_only edges document a relation whose values generally differ: they grant no tolerance-based (registry) collapse. A blind rule then collapses values that are the same number to at least three significant digits in any two sources of a species, whatever the registry says about the pair (identical values are the same datum; on a provenance_only pair such a collapse is labelled blind (provenance_only edge) in the summary report); values of one or two significant digits, round numbers such as 20 or 1500 g, are left alone as ambiguous. Matching values form a component per species; the registry ancestor (or the lowest-priority sibling) is kept and the others are recorded in source_dependencies as dropped<kept. reports/dedupe_summary.md lists every edge with the species it shares and collapsed, the species left with a single independent value, and the residual identical pairs the registry does not explain (candidates for new edges or shared primary literature).

   Across sources, the independent values are combined as the arithmetic mean (owner decision 2026-10-03, issue #16): the arithmetic mean is the published convention of the database, so outputs stay comparable with earlier releases and with the manuscript; within-source skew is already handled by the geometric mean of the first pass, and the cross-source step combines per-source central values, where the choice matters less once duplicated values are collapsed; switching to the geometric mean would have moved 795 species by more than 0.1 log10, always downward, a larger and systematic shift than de-duplication, and would entangle two effects that should stay separable; neither choice affects log10_range or the range filter. log10_range records the spread of the values in the mean (log10 of the largest over the smallest, written to six decimals; the range filter below and the QC reports use the unrounded value), source_mass lists every contributing source, n counts every record, and n_sources, n_independent and source_dependencies record the de-duplication. R/library/check_enriched.r writes QC reports to reports/ (warnings_taxonomy.md, which also lists every cleaned name no enrichment stage resolved, species-level names and genus-only names (step 6) alike, with its sources and row counts; warnings_mass_values.md; warnings_name_change.md), and species whose values in the mean disagree by more than one order of magnitude (log10_range > 1) are removed (R/library/remove_high_range.r). Counts from this step (including nValuesCollapsed, nSpeciesSingleDatum, nDependencyEdges, nValuesExcluded, nSpeciesRescued and nRemovedHighRange) are written to ../TaxonBodyMassML/ms/numbers_db.tex for the manuscript.

   Record-level range rule (issue #34, owner decisions of 2026-10-06; R/library/exclude_discordant.r, run between the de-duplication and the cross-source mean). Until #34 a species whose independent values spanned more than an order of magnitude was removed whole; in four of five such multi-value species a single value caused the spread (a VertNet juvenile, a dry mass taken as wet, a jellyfish ephyra, a web-level food-web datum) while the others agreed closely. ExcludeDiscordantValues() now judges each such species by a short cascade and excludes one value (or one side) from its mean when the cascade singles it out: (A) leave-one-out, loo_unique: exactly one value whose exclusion brings the rest within 1 log10, provided the rest span at least two evidence groups (sources the dependency registry relates, directly or through a common parent, count as one group, so two near-copies cannot outvote one independent value); (A2) several such values: the one of lowest trust tier when the tiers single it out (tier_tiebreak), else the one farthest from the median of the others (distance_tiebreak); (D) two values, or one evidence group against one value: the lower-trust side when the tiers differ (two_value_tier), the kept side itself spanning at most 1 log10; (E) otherwise nothing is excluded and the species falls to the log10_range > 1 filter as before (unresolved: same_tier, maximum_cannot_decide, one_group, no_single_culprit). Trust tiers come from the owner-editable value_tier column of Bib/source_provenance_classes.csv: 1 for compiled or measured adult values (the default), 2 for maxima-based or allometric sources (Feldman_etal_2016, Meiri_2018, fishbase, sealifebase), 3 for single specimens, individuals, converted or web-level values (the VertNet dumps, Brose_2005, Brose_etal_2018, Vanni_2017, Gonzalez_2025, Castro_2025, DeLong_etal_2010, Kiorboe_2013, Kiorboe_2014, Pata_2025); a value converted from dry, ash-free dry, carbon or energy mass (a conversion CiteID after the label in source_mass, step 1) is tier 3 whatever its source's tier. The tiers are direction-blind (a low and a high outlier are judged alike), and a tier-2 (maxima-based) value may be excluded but never decides against another value: in (A2) it never outranks a tier-3 candidate (the tie falls to the distance) and in (D) a tier-2 side never excludes a tier-3 side. No record-count tie-break is used, and no per-species rule exists (the per-species rule files of 2026-08 were retired under #24). Copies the de-duplication collapsed into an excluded value are excluded with it. An excluded value stays in source_mass and n, is left out of mass_g, n_independent and log10_range, and is named in sources_excluded (label<rule, n_excluded counting them); its provenance rows stay in TaxonBodyMass_Provenance.csv.gz with record_status excluded_<rule>. reports/excluded_records.csv is the live list of a run's exclusions (species and genus level, one row per excluded value with the kept values beside it; single_value_rescue marks the species left with one kept value, which the tiers rather than the data decided and which are flagged SUSPICIOUS for review, owner decision 2026-10-06) and reports/warnings_mass_values.md lists them in its own section with a sub-section for the single-value rescues; the register audit/flagged_species.csv holds the one append-only batch of the adoption (method range_rule_2026-10-06: one row per excluded value and one SUSPICIOUS row per single-value rescue; audit/README.md). Genus-only records are judged by the same module in step 6. Exercised without network access by Rscript R/library/tests/test_exclude_discordant.R.

6. Genus-level aggregation — TaxonBodyMass_GenusLevel.csv gives, for every accepted genus, the arithmetic mean of the cross-source means of its species (step 5) and of its genus-only record, each with the weight of one value; synonymous or misspelt input genera therefore fold into the accepted name. A genus-only record is a value for an organism identified to genus only, measured or compiled as the source reports it (population means included, issue #29): a name written as a bare genus (placeholder names such as Genus sp. or Genus spec. are dropped instead, issue #43), including the genus-level rows of the lab Sheet, which replace the sources' rows of the same bare name and leave the species-level values of the genus alone (step 3, issue #57). Such records get the machinery of the species path (R/library/enrich_genus.r, issue #49). Every bare name is resolved once at genus rank, through the accepted genera of the enrichment cache and the GBIF backbone exact-name lookup, with the class/order/family hints the source gives (VertNet, Castro_2025, Pata_2025, Makarieva_2008): synonyms and misspellings fold into the accepted genus (Dendroica -> Setophaga, Clethrionomys -> Myodes), homonyms are settled by the hints or, failing those, by preferring the heterotroph kingdoms (GBIF's name matcher files Coleoptera and Oligochaeta as plant genera), and names of ranks above genus leave the table: the backbone's own family, order and class usages (Staphylinidae, Araneae, Oligochaeta), the majority reading of the other GBIF checklists for names the backbone lacks or misfiles (Crustacea, Heteroptera, Tanytarsini, Polychaeta), a two-name curated list (Anisoptera, Hydracarina) and the family-group suffixes; fuzzy matches are accepted only for names of six letters or more with no exact usage at any rank, at confidence 80 or more and at most two letters apart. The answers are cached as genus_cache in sources/enrich_cache.Rdata; names no stage resolved are listed in reports/warnings_taxonomy.md with their sources and rows, and their rows are dropped. The resolved records are filtered with FilterAutotrophs() (the extinct list is species-level and does not apply), combined as one value per genus and source (geometric mean, VertNet dumps pooled), de-duplicated with the registry of step 5 (Brose_etal_2018 -> Brose_2005, -> Hechinger_etal_2011), and combined as the arithmetic mean of the independent per-source values: that single record enters the genus mean with the weight of one species, and n_independent counts the species' independent values plus one per independent source of genus-only records (owner decision 2026-10-04; one record per source instead would have moved 48 genera by at most 0.17 log10). Names of ranks above genus are dropped without a separate output (167 names and 9,249 rows at the 2026-10-04 run; owner decision). A genus-only record is judged against the genus's species cross-source means by the record-level range rule of step 5 (ExcludeDiscordantGenusRecords(), issue #34): more than 1 log10 from the median of two or more species it is excluded from the genus mean (genus_only_discordant); against a single species the two-value tier rule applies, the record being the only side that can be excluded (two_value_tier; a same-tier pair, or a species value that is a tier-2 maximum against a tier-3 record, leaves the record in and listed). An excluded record stays in the genus row's source_mass but not in mass_g, n or n_independent; a genus with no other contributor has no row. reports/genus_only_records.md lists, at every run, the names resolved above genus, the autotroph genera removed, the fuzzy matches and homonym choices to check by eye, the synonyms folded, the values collapsed as copies, and the genus-only records more than an order of magnitude from the genus's species values with their status. Exercised without network access by Rscript R/library/tests/test_genus_only_records.R.

7. Citations and provenance (issue #1) — R/RunMe.r section 8 reads both bibliographies, the hand-curated Bib/TaxonBodyMass_Citations.bib (BibDesk) and the generated Bib/TaxonBodyMass_PrimaryCitations.bib (built by R/library/citations/run_citations.r --bib from Crossref metadata or owner-approved fields, never edited by hand; a key present in both files stops the run), and both citation tabs of the lab Sheet from their tracked copies Bib/BM_citations_snapshot.csv and Bib/BM_primary_citations_snapshot.csv (BM_citations: source labels and conversion references -> bib keys; BM_primary_citations: the verified primary references appended by run_citations.r --sheet, which refreshes both copies, as does a RunMe.r run with DataSheets = TRUE; step 3, issue #118). Sheet rows whose key is in neither bib are listed in reports/warnings_citations.md instead of being dropped silently. Bib/TaxonBodyMass_CitationCiteIDs.csv keeps Bibcite, CiteID as its first two columns and gains doi, role and bib_file. Every record's source_mass is split before step 5 into the source label and the conversion CiteIDs appended by LabelWithConversion(), the parse scripts' per-record reference keys (ref_keys) travel through the first averaging pass, and TaxonBodyMass_Provenance.csv.gz links every species of TaxonBodyMass.csv to its source labels and, where a source's reference list has been ingested and verified (sources/databases/<Src>/primary_references.csv), to the study that measured the animal, with the hop and provenance type of each reference; records without a reference get the default of the registry Bib/source_provenance_classes.csv. A reference that is itself a source label of the database (recognised by CiteID, bib key or DOI; Herberstein et al. 2022 citing Chown et al. 2007) is an intermediate compilation: where the intermediate's own records of the species resolve to primary references the row goes through it (hop 2, compiled_via_compilation, via_cite_id the intermediate), where the intermediate measured the species itself the row cites it as the measurement (compiled_from), and otherwise the intermediate is the terminal citation (compilation_terminal) until its reference list is ingested; no third hop is attempted. The checks (every primary bib key in a bib file, every primary CiteID in a Sheet tab, every accepted reference with a DOI or an approval, per-source coverage) are written to reports/warnings_citations.md. The tooling, its decision rules and its hallucination-robustness rules are described in R/library/citations/README.md; RunMe.r sources only its offline part and never calls Crossref, OpenAlex or Scite.

Source dependency registry

Bib/source_dependencies.csv records which sources copy or derive their body masses from which others. (The provenance registry Bib/source_provenance_classes.csv, which classifies every source for the primary-source attribution of issue #1, is described in R/library/citations/README.md.) Columns: child_label (a source_mass label), parent_label (a label, or the name of a compilation outside the database such as Dunning_2008, White_2006, Nagy_1999, CareyJudge_2000), parent_in_db (TRUE/FALSE), relation (copies, via an intermediate compilation, shared_primary, derived_same_input, or provenance_only), tol_log10 (the largest |log10 difference| at which a child value counts as the parent's), priority (which of several siblings of an external parent is kept: the lowest number; one value per label), status (confirmed by the source's own documentation or suspected from the values alone), n_rows_child (records the child attributes to the parent where it does so per row), evidence and added. Where several routes connect two labels the loosest tolerance applies. LoadSourceDependencies() stops the pipeline when a child or an in-database parent is not a label of any source frame, when an external parent is one, on a cycle, or on conflicting priorities, so a renamed or removed source has to be reflected in the registry. Add an edge when a source's documentation or a high fraction of identical values (see the residual table of reports/dedupe_summary.md) shows that it reproduces another; use provenance_only when the values differ (the pair then gets no tolerance, but values identical to three significant digits are still collapsed by the blind rule and counted per edge in the report). The helper is tested without network access by

Rscript R/library/tests/test_dedupe_sources.R

and Rscript R/library/dedupe_audit.r [--before <earlier TaxonBodyMass.csv>] replicates Pass 1, the de-duplication and Pass 2 from sources/Rdata and checks the released CSV against it (invariants, agreement, per-edge fractions, the shifts the de-duplication causes).

Outputs

File	Description
TaxonBodyMass.csv	One row per accepted species. Columns: genus, species, taxon (a representative input name), taxon_provided (all input names, ;-separated), log10_range (spread of the independent per-source values: log10 of the largest over the smallest, six decimals), mass_g (grams, arithmetic mean of the independent values, four significant figures), source_mass (;-separated labels of every contributing source and conversion reference), n (records in all sources), n_sources (per-source values, the VertNet dumps counted once), n_independent (values entering the mean), source_dependencies (values collapsed as copies, dropped<kept pairs ;-separated, empty when none; pooled VertNet dumps appear joined by +), n_excluded and sources_excluded (per-source values the record-level range rule left out of the mean, step 5, issue #34: the count, and label<rule pairs ;-separated, a collapsed copy carrying its parent's rule; empty when none), kingdom, phylum, class, order, family, taxonomy_source, gbif_confidence, gbif_status, gbif_family, gbif_order, species_changed
TaxonBodyMass_GenusLevel.csv	One row per accepted genus (pipeline step 6): mass_g (grams, the arithmetic mean of the species' cross-source means and of the genus-only record, from the sources or the lab Sheet, four significant figures), source_mass (;-separated distinct labels and conversion references of every contributor, as in TaxonBodyMass.csv; joined by - before issue #1, which hyphenated labels made ambiguous), n (records) and n_independent (independent values: the species' plus one per independent source of genus-only records). Names of ranks above genus, unresolved bare names and autotroph genera are not in the table
Bib/source_dependencies.csv	The source dependency registry used for de-duplication (see pipeline step 5)
reports/excluded_records.csv	The live list of the values the record-level range rule excluded in the run (pipeline step 5, issue #34): level (species or genus), genus, species, taxon, source_label, source_mass, mass_g (the excluded Pass-1 value), n, value_tier, rule, distance_log10 (log10 of the value over the median of the kept values, signed), copy_of (the parent of a collapsed copy excluded with it), n_kept, single_value_rescue (TRUE when the species, or the genus, rests on one kept value), kept_mass_g (the mean of the kept values; for a genus row the median of the species means), kept_values. No timestamp; rewritten at every run
Bib/source_provenance_classes.csv	The registry of every source_mass label for the primary-source attribution of issue #1: class (primary, compilation, derived, database, live, unknown), default_provenance_type (the provenance type a record gets when it carries no resolved primary reference), equation_bibcite (allometry-derived sources) and notes; R/RunMe.r stops on an unregistered label (R/library/citations/provenance.r)
Bib/pending_citations.csv	The review queue of the citation tooling (R/library/citations/README.md): references the automated verification could not settle, with their candidates; the owner's committed decision approves them
Bib/scite_checks.csv	The retraction / correction screen of accepted DOIs (checked_by: scite-mcp, consensus-mcp or none), written in Claude Code sessions and read by the citation tooling
reports/dedupe_summary.md	Per-edge and residual summary of the de-duplication, regenerated at every run
reports/genus_only_records.md	How the genus-only records resolved at genus rank (pipeline step 6): the names above genus and the autotroph genera excluded, the fuzzy matches and homonym choices to check by eye, the synonyms folded, the values collapsed as copies, and the records more than an order of magnitude from the genus's species values (listed, not removed: issue #34); regenerated at every run (issue #49)
reports/warnings_raw_names.md	Every raw taxon name that FixFormatting() changed beyond blank -> underscore or classified by a rule of audit/raw_name_patterns.csv, by class and source, with the names dropped and the names no rule covers (empty when the run passes); regenerated at every run (issue #38)
audit/raw_name_patterns.csv	The vocabulary of raw-name rules FixFormatting() applies (pipeline step 2): pattern, scope, class, action, note, added; maintained by hand, validated on loading (see audit/README.md)
audit/imputed_rows.csv	Log of the rows dropped as imputed by the parse scripts, as life stages or small size classes by the raw-name rules and as group-level placeholders by the live DataRetriever download, one row per DropImputed() call (source, reason, n_dropped, n_taxa, n_kept), ordered by source; rewritten by RunMe.r at every run with recompile = TRUE, merging the saved entries of the live download frames (audit/imputed_rows_live.csv), so that a run with the download flags off reproduces it byte for byte (issue #40; see audit/README.md)
audit/imputed_rows_live.csv	The DropImputed() entries of the four live download frames, with frame, written (download date) and frame_md5; saved by R/library/data_retrieve.r, data_vertnet.r and data_fishbase.r when a frame is downloaded (each replacing its own frame's rows) and merged into audit/imputed_rows.csv at every run; checked against the cached frames by R/library/check_cache.r (see audit/README.md)
audit/extinct_taxa.csv	The extinct-taxa list applied by RemoveExtinct() (pipeline step 2), rebuilt by hand with R/library/build_extinct_taxa.r from the MOM, PHYLACINE and AVONET status columns
TaxonBodyMass_Provenance.csv.gz	One row per species x source label x reference for the species of TaxonBodyMass.csv (pipeline step 7, issue #1): genus, species, taxon, source_mass (one label), source_bibcite, origin (pipeline or BM_data), hop (0: the source measured the animal or a conversion factor; 1: the source cites the reference; 2: through an intermediate compilation that is itself a source label, whose per-record reference is the primary), via_cite_id (the intermediate's CiteID on hop-2 rows), ref_role (measurement, compilation, equation, conversion, self), provenance_type (measured_in_source, compiled_from, compiled_via_compilation, compilation_terminal, derived_allometry, derived_imputed, database_record, conversion_factor, unknown), primary_cite_id, primary_bibcite, primary_doi, match_status (the verification status of the reference; unmatched_key for a key the source's reference list lacks, uningested for a source whose reference list has not been ingested yet), n_records (records behind the row), record_status (kept when the value behind the row is in the species' mean, collapsed copies included; excluded_<rule> when the record-level range rule left it out, step 5, issue #34; the excluded records keep their rows). Gzipped CSV, deterministic (no timestamp); rows in a fixed order
sources/source_files.csv	The licence manifest (Data licences above): one row per tracked file under sources/databases/ and sources/conversion_factors/ plus one live row per source downloaded at run time: folder, label, path, role (raw, parsed, derived, code, doc, live), bytes, sha256, licence_class, statement_source, redistributable, attribution, check_url, notes; sizes and hashes regenerated and hand columns kept by Rscript R/library/licence_manifest.r, which also writes sources/LICENSES.md
sources/LICENSES.md	Generated from the manifest: the scope of the repository licence, the position on non-commercial and share-alike sources, a table per licence class with attribution texts, the carve-outs, the publisher files, the sources not redistributed and the "unknown; owner to check" list
reports/warnings_licences.md	Written by CheckSourceDocs() at every run: the source folders whose licence is unknown (with the URL to check), the non-commercial and share-alike carve-outs, README Licence: lines that disagree with the manifest, and folders without the line or absent from the manifest
reports/warnings_citations.md	The checks of pipeline step 7: totals, problems (bib keys, CiteID rows, accepted references without DOI or bib entry), the Sheet rows whose Bibcite is in neither bib, the labels without a CiteID row, and the per-source coverage of the provenance table (record links resolved to a primary reference, reference counts by status); regenerated at every run
Bib/TaxonBodyMass_CitationCiteIDs.csv	Maps BibTeX keys to source_mass labels for citation tracing: Bibcite, CiteID (the first two columns, read by TaxonBodyMassML), doi (from the bib entry, when it has one), role (source, conversion or primary), bib_file (Citations or PrimaryCitations)
Bib/TaxonBodyMass_Citations.bib	Hand-curated BibTeX bibliography (BibDesk) of the sources, the conversion references and the lab Sheet's citations
Bib/TaxonBodyMass_PrimaryCitations.bib	Generated bibliography of the verified primary references (run_citations.r --bib; %% GENERATED, never edited by hand, re-derivable from the primary_references.csv files and the response cache)
sources/BM_data_snapshot.csv, Bib/BM_citations_snapshot.csv, Bib/BM_primary_citations_snapshot.csv	The three tabs of the lab Sheet that the pipeline reads (steps 3 and 7), as last imported by RunMe.r with DataSheets = TRUE or, the two citation tabs, by run_citations.r --sheet: the Sheet's rows and columns in its order, every cell as text (an empty cell stays empty; a number as a 15-significant-digit decimal, which reads back exactly for every hand-typed value and within 1e-15 of the Sheet's value for the results of Sheet formulas; owner decision 2026-10-05), written by SnapshotSheetTab() in R/library/sheet_snapshots.r. The pipeline always reads these copies, so a run with DataSheets = FALSE needs no Google token and reproduces the outputs of the import that wrote them; run_citations.r --sheet is also the record of every append (issue #118)

The tables are reproducible byte for byte: for a given cache (sources/Rdata, sources/enrich_cache.Rdata) and given tracked copies of the lab Sheet tabs (above), two runs write identical TaxonBodyMass.csv, TaxonBodyMass_GenusLevel.csv, TaxonBodyMass_Provenance.csv.gz, reports/excluded_records.csv, audit/imputed_rows.csv and Bib/TaxonBodyMass_CitationCiteIDs.csv, because the records are put in a fixed order before the first averaging pass and the derived floating-point columns are rounded at write time (log10_range to six decimals, mass_g to four significant figures; issue #53). The reports under reports/ differ between runs only in their timestamp line.

Usage

Prerequisites

Install the required R packages:

install.packages(c("plyr", "dplyr", "googlesheets4", "stringr", "readxl"))
install.packages("rfishbase")
install.packages("rdataretriever")

The rdataretriever package requires Python and the retriever Python package. It is used only when DataRetrieve = TRUE in RunMe.r. RunMe.r points reticulate at a dedicated venv at ~/.local/share/r-rdataretriever pinned to Python 3.11 and retriever 2.4.0 (the compatibility patch in R/library/data_retrieve.r targets the 2.x API; retriever 3.x is not supported). Create it once with uv (brew install uv):

uv venv --python 3.11 ~/.local/share/r-rdataretriever
uv pip install --python ~/.local/share/r-rdataretriever "retriever==2.4.0" "setuptools<70"

retriever 2.4.0 fails on import with packaging.version.InvalidVersion: Invalid version: '' when any cached dataset script has an empty minimum-version field. Fix it by adding an and mod_ver guard in the venv's retriever/lib/scripts.py:

sed -i '' 's/if hasattr(module, "retriever_minimum_version"):/if hasattr(module, "retriever_minimum_version") and mod_ver:/' \
  ~/.local/share/r-rdataretriever/lib/python3.11/site-packages/retriever/lib/scripts.py

A message such as tidycensus is supported by Retriever version 3.0.1-dev on load is harmless; it only means that one dataset script is skipped.

Reproduce the database

To regenerate all per-source .Rdata files and recompile the database from scratch, set recompile = TRUE at the top of R/RunMe.r before running it; the download flags DataRetrieve, DataVertNet and DataFishbase control the four live-download frames, and DataSheets whether the lab Google Sheet is imported: TRUE downloads its three tabs through googlesheets4 (a Google token: a browser prompt on first use, the cached token without a prompt afterwards) and refreshes the tracked copies, printing what changed; FALSE, the committed default, reads the copies and needs no token, so a fresh clone runs as it is (the run stops at once, naming the files, if a copy is missing). Contact mark.novak@oregonstate.edu to contribute new sources or curated values.

The run stops before any cached frame is loaded (step 2) when:

• a parse script fails (recompile = TRUE) -- every failure is printed as an ERROR: line and all of them are listed in the stop message, so one run reports them all. Fix the scripts and rerun with recompile = TRUE.
• a cached frame was not rewritten (recompile = TRUE) -- a frame in sources/Rdata/ is older than the loop, e.g. an orphan left by a renamed or removed source, or a script that ran without saving. Delete the orphan or fix the script.
• an expected frame is missing (whatever recompile is) -- R/library/check_cache.r expects BodyMass_<Source>.Rdata for every sources/databases/<Source>/BodyMass_<Source>.r plus the four download frames, and the stop message names the flag to set for each missing frame (recompile, DataRetrieve, DataVertNet or DataFishbase). Frames that no current script writes only raise a warning. The three scripts whose save() target differs from their folder name are listed in frame_overrides in check_cache.r; a new source whose script saves under another name must be added there, otherwise the check stops and says so.

These guards are exercised on a throwaway source tree, without network access, by

Rscript R/library/tests/test_recompile_guard.R

After the completeness check, CheckImputedLive() (R/library/check_cache.r) warns, without stopping, when a live frame in sources/Rdata/ is not the file whose download saved its rows in audit/imputed_rows_live.csv (the frame was downloaded again without saving its entries, or the cache was copied from a run with a different frame); rerun with that frame's flag set to TRUE to refresh the frame and its entries together (issue #40). The merge of the saved entries into audit/imputed_rows.csv and this check are exercised by

Rscript R/library/tests/test_imputed_live.R

Use the compiled output

TaxonBodyMass.csv and TaxonBodyMass_GenusLevel.csv are tracked at the repository root and can be used directly without running the pipeline.

Reproducibility note

Steps 3 and 7 of the pipeline take the lab-curated mass overrides and the citation maps from a lab-internal Google Sheet. The three tabs the pipeline reads are tracked in this repository as CSV copies (sources/BM_data_snapshot.csv, Bib/BM_citations_snapshot.csv, Bib/BM_primary_citations_snapshot.csv; Outputs above), and the pipeline reads the copies whatever DataSheets is set to, so the full pipeline, lab-curated values included, is reproducible offline from a clone with DataSheets = FALSE (the committed default) and without access to the Sheet: the compiled output CSVs committed here are the output of the copies committed with them. DataSheets = TRUE is for the lab: it downloads the tabs with the lab's Google token, rewrites the copies, reports the rows added, removed and changed, and the refreshed copies are committed together with the outputs they produce (issue #118).

Citation

Manuscript in preparation.