Column-level reference for the committed datasets that the analysis package
loads and the reference outputs under results/. Fetched upstream data
(data/stale-blocks/), raw extracts, and other scratch artifacts are not
committed and are out of scope here; see the input data notes
for the data-boundary rules and research contracts
for the research semantics these columns encode.
- Hash byte order. All
*_hash/*_prev_hashcolumns are big-endian display order (the0000...form).btc_header_hexis the raw 80-byte header in wire order, hex-encoded (160 characters); double-SHA256 of those bytes, reversed, equals the display hash. - Line endings. All committed CSVs are LF.
- Encoding of the gate verdict. The chain-specific publication gate runs at
classification time and its verdict is persisted per row as
validation_status/expected_nbits. For chains exposing the required evidence, it checks header hash and self-PoW, active-parent linkage, expectednBits, median-time-past, Bitcoin's historical minimum block versions, coinbase scriptSig length, and BIP34's coinbase-height rule. RSK has no recoverable parent coinbase scriptSig, so it cannot independently apply the two coinbase-dependent checks; exact-key cross-chain exclusions protect known shared failures. Loaders read and filter the persisted verdict; they never recompute it. - Validity scope. Either exact accepted direct-stale status means publication-gate accepted under the checks available for that chain and row. It does not mean a complete Bitcoin block was available or fully consensus-validated. See the data validity contract.
- Dedup key. Stale events are deduplicated by
(height, hash), not height alone, so competing same-height stale hashes are preserved as distinct rows.
One row per recovered direct-stale BTC header candidate for a merge-mined
chain. Every committed row is classification = stale and
validation_status is exactly VALID or
VALID (post-BCH, difficulty matches BTC). These files are the
publication-gate-accepted output; no other VALID-prefixed spelling is
accepted. An accepted status is a header-profile verdict, not a full-block
validity assertion. The F2Pool blocks at heights 783,426 and 784,121 passed
that profile, but independent body evidence now places them in the error
catalogue and excludes them from these files. Every file begins with the same
16-column core layout: normalized
Bitcoin-parent fields, the registered child-height slot, the four child-header
fields, and the verdict fields. Source-specific variants (btc_stale_height, btc_hash,
btc_bits_hex) are normalized into this layout. Chain-specific research
columns trail that shared core:
| Column | Notes |
|---|---|
btc_height |
Candidate Bitcoin height inferred as active-parent height + 1. |
btc_header_hash |
The candidate header's own hash, display order. |
btc_prev_hash |
Parent on the canonical chain. |
btc_time |
Header nTime (unix seconds). |
btc_bits |
Header nBits, lowercase 8-char hex. |
coinbase_scriptsig_hex |
Preserved coinbase evidence for later pool-attribution research. |
coinbase_outputs |
The BTC parent coinbase's payout list, semicolon-joined in coinbase order. Each entry is <payout> or <payout>:<value_sats>, where the amount appears only where the extraction preserved it. <payout> is the Bitcoin mainnet address for the address-bearing standard templates (P2PKH, P2SH, P2WPKH, P2WSH, P2TR) and raw scriptPubKey hex for every other script; see "Coinbase output rendering" below. Empty where the extraction preserved none (RSK). |
btc_header_hex |
Full 80-byte header, hex (160 chars). |
<chain>_height (dvc_height, nmc_height, child_height, ...) |
Height on the merge-mined sibling chain where independently resolved. The column occupies the same position in every validated schema and remains blank when unavailable. I0coin's refreshed validated rows carry heights derived from complete authenticated header ancestry to genesis. Namecoin classifier inventory heights are historical block-file scan order, so normalization blanks them and hydrates only the exact node-verified identity height; see docs/chains/namecoin.md. |
child_block_hash |
Authenticated child block hash for the 17 refreshed historical source families. |
child_header_hex |
Authenticated serialized 80-byte child header for the 17 refreshed historical source families. |
child_block_time |
Unsigned decimal timestamp decoded from child_header_hex. |
child_nbits |
Lowercase 8-character child target field, decoded from the same header except for Xaya's authenticated external PowData target. |
classification |
stale for these files. |
validation_status |
Exactly VALID or VALID (post-BCH, difficulty matches BTC): passed the declared publication gate, not a full-block validity assertion. |
expected_nbits |
Canonical nBits at that BTC height (the gate's reference). |
Chain-specific research columns trail the shared layout where retained:
namecoin's nbits_match / post_bch_fork gate detail,
btc_bip34_height, and btc_parent_height (its btc_header_hex
is populated for all 1,641 rows; the 228 historically header-less rows were
back-filled from the committed monitor evidence, byte-verified against each
row's committed hash and decoded fields),
coiledcoin's eligius_attack_window, and i0coin's full_coinbase_hex.
The latter preserves the complete serialized Bitcoin parent coinbase for
all 191 accepted I0coin witnesses; its decoded scriptSig and ordered outputs,
including amounts, agree with the shared columns.
coinbase_outputs describes Bitcoin payouts, so it is rendered as Bitcoin
regardless of which child chain witnessed the block. Two script categories
deliberately stay as raw scriptPubKey hex rather than becoming an address:
- P2PK, because it has no address. Rendering it as the base58 address of
its pubkey hash would assert a P2PKH template the bytes do not carry and
make the two script types indistinguishable in the committed evidence.
For the same reason a payout that reached us only as a child node's
decoded address renders as
pkh(<hash160>): that node derives the same address from a P2PKH and a P2PK output, so the recipient hash is established but the script behind it is not. Historical Namecoin projections, Syscoin, and Terracoin carry that form; a P2SH address is unambiguous and stays an address. - Nulldata, because the payload is the point. Almost every modern coinbase
puts the segwit witness commitment there, and an
OP_RETURNlabel would discard it.
A leading ~ on every entry marks the list as an ordered filtered
projection, meaning the acquisition kept only some outputs so an entry's
ordinal is its order in the surviving list rather than its transaction
position. Namecoin's original acquisition produced 1,476 such lists; issue #52
recovered complete raw-script vectors for all 1,649 then-accepted loader rows, so
none of those loader cells now carries ~ or recipient-only pkh(...) claims.
Historical inventories and their full-evidence exports retain source-level
claims, including these weaker forms. The refreshed Monitor uses the restored
validated-loader vectors for accepted Namecoin rows. Keep filtered semantics
where the source warrants them: the marker is decided per row rather than
removed for the entire chain. Terracoin and Syscoin also
decoded through the child node but kept a placeholder for every output the
decode could not name, so their positions stay exact even where a payout
survives only as pkh(...). Terracoin's committed cells predate its
hex-reading backfill, and the decode provably collapses P2PK: for BTC
...67a249, Devcoin and Unobtanium hold output 1's raw P2PK script while the
Terracoin decode rendered it as the P2PKH-form address of the same hash160.
Bitcoin Vault binary-parses raw block hex, so its entries are exact scripts. Completeness is stated with this
marker rather than inferred from the rendering, because under one contract an
address-only list no longer implies an address-filtered source.
Producers emit this contract directly. The blkdat classifier requires complete
exact output scripts before RPC or writes and normalizes every output split,
so raw-script inventories need no subsequent rendering or recovery pass.
just validate-coinbase-outputs verifies committed cells without rewriting them;
a committed-data test also pins the invariant. Rendering preserves payout
identity, order and amount.
Committed rows from the raw-script acquisitions (devcoin, ixcoin, groupcoin,
and their blkdat-format peers) predate amount retention, so they are
payout-only; the current producers emit <payout>:<value_sats> where the
source preserves the value, so a regeneration would add amounts to those
rows. That is a pure refinement (positions and payout identities unchanged,
amounts only gained), which the publication floor accepts.
A zero-length scriptPubKey renders as an amount-only entry, which holds the
output's position but reads downstream as an unknown script rather than an
exact empty one. Distinguishing the two would need a new token in this
vocabulary, which upstreaming.md makes a lockstep contract with the
merge-mining-monitor importer; no committed row carries such an output.
Acquisition limitations are not repairable by rendering alone:
- Terracoin, Bitcoin Vault, and Syscoin recorded a bare
OP_RETURNlabel instead of the nulldata script, so those witness-commitment payloads were discarded before commit. The checker tolerates the label as a legacy value. - Namecoin's former address-only projection omitted 784 outputs across 455 accepted rows. The recorded raw-script recovery restored them with an exact ordered-subset consistency check. Normal classifier reruns consume complete extracted scripts directly. The complete Monitor rebuild uses the restored validated loader for accepted rows; full evidence retains the selected classifier inventory's source-level claims.
Exact (height, hash) keys of consensus-invalid full-proof-of-work Bitcoin
blocks that must be removed from stale publication surfaces. Every row has
classification=error_block; blank or unknown classifications fail closed
when the dataset is loaded. The current 53 rows include the 946,213 and 957,780
time_below_mtp blocks, the 717,696 nbits_retarget_not_applied block, the
Hathor-witnessed 649,674 bip34_coinbase_height_missing block, and four
stale-ancestry candidates at 331,673, 331,674, 402,610, and 422,059 whose bytes
re-derive bip34_coinbase_height_mismatch. Of the invalid keys, 25 carry a
mismatched BIP34 coinbase height, one omits the
required BIP34 height, three fail BIP66's minimum version 3 rule, five
fail BIP65's minimum version 4 rule, three are time-too-old against
median-time-past (380,992, 946,213 and 957,780), one carries a
103-byte coinbase scriptSig above Bitcoin's 100-byte limit, and one failed to
apply the difficulty retarget at an epoch boundary. Four entries at 363,732, 363,733, 363,735 and 363,736 inherit the invalid
BIP66 root at 363,731 through authenticated header paths. Ten further entries have
independently verified body-rule failures bound by body_evidence.csv. The dataset retains the signed
header version, child-chain provenance, raw coinbase scriptSig, rejection
reason, and the named rules violated. It is a compact audit record rather
than a self-contained merge-mining proof. It is applied by public loaders, upstream
novelty calculations, the upstream sidecar builder, full-evidence and
relevance exports, and unknown-ancestry reconciliation.
The rejection_reason / rules_violated token nbits_retarget_not_applied
means the block sits at a retarget-boundary height (height % 2016 == 0, an
epoch start) but carries the PREVIOUS epoch's nBits instead of the
newly-retargeted value the epoch start must use: the miner failed to apply
the difficulty retarget, a hard consensus violation. It is offline
re-checkable against the committed
data/bitcoin-epoch-reference/btc_nbits_by_epoch.json table: the row's
btc_bits must differ from the table value at its height and equal the table
value at height - 2016. This is distinct from a plain nBits mismatch
rejection at a non-boundary height, which is the contamination gate's target
class (a share/near row), not an error block: the error-block case is
specifically the retarget-not-applied at an epoch boundary where the header
still meets full proof of work.
The consensus_invalid_parent rule requires a reviewed complete header path
from the catalogue parent to a root whose minimum-version violation can be
re-derived. Each exact (height, hash) sidecar row contains
path_header_hexes (own header through root, pipe-separated), the root
identity/height/rule, and its canonical predecessor's identity, height,
serialized header and stable bitcoin-core-rpc:<label> receipt source.
root_evidence_url identifies the exact commit-pinned invalid-blocks JSONL
record. Missing, duplicate, malformed or unused sidecar records fail closed.
Validation binds the first header to the catalogue, authenticates every edge
and endpoint, checks height placement and full Bitcoin work with the required
epoch targets, and re-derives the root's contemporaneous minimum-version
failure. Canonical placement of the root predecessor rests on the reviewed
Bitcoin Core receipt; offline validation checks its serialized identity and
context rather than replaying the active chain. No body/UTXO validation is
asserted. The root is not published as a child observation without a genuine
merge-mining witness. The four current rows admit 13 child events at Bitcoin
363,732, 363,733,
363,735 and 363,736. The frozen BTC Relay fixture supplies intermediate
363,734, without fabricating a child witness. Eight prior strict observations
move to errors, and three formerly private 363,736 witnesses are added.
The original source unknown classifications remain intact while Monitor
publishes the stronger invalid verdict.
Reviewed external invalidity verdicts for catalogue entries whose body rule is not re-derived by Research. The sidecar replaces the former annotation-only F2Pool overlay; its parents are excluded from accepted stale publication.
| Column | Notes |
|---|---|
height, hash |
Exact catalogue parent identity; hash in display order. |
rule |
Registered external body-rule token matching rules_violated. |
block_file |
blocks/<height>-<hash>.bin in the pinned invalid-blocks clone (INVALID_BLOCKS_DIR). |
block_sha256 |
Digest of the complete body authenticated locally. |
evidence_url |
Exact invalid-blocks JSONL row at a full commit hash. |
just validate-error-blocks checks sidecar membership, required body bytes,
header identity, transaction merkle root and applicable witness commitment.
Admission review establishes the external verdict's independence; the offline
validator does not fetch the reference or repeat its consensus-rule checks.
Observation-ledger paths prefixed git/<commit>/ identify source bytes in that
retained repository revision; their row numbers and digests refer to that
snapshot, even after the current validated input removes the parent.
RSK cannot be attributed by parsing a BTC coinbase because the proof does not
preserve it. Like every other chain, the committed file is
VALID-stales-only and carries the shared validated-stales layout (btc_height,
btc_header_hash, btc_prev_hash, btc_time, btc_bits,
coinbase_scriptsig_hex and coinbase_outputs as empty placeholders - the
extraction preserves only the midstate-compressed coinbase, not the decoded
fields - btc_header_hex, rsk_height, classification,
validation_status, expected_nbits), followed by RSK-specific evidence and
historical evidence columns:
| Column | Meaning |
|---|---|
rsk_height, rsk_timestamp |
Position and time on the RSK chain. |
rsk_miner |
RSK miner address. |
pool_label |
Historical label retained for provenance; ignored by the current loader. |
merge_mining_hash |
RSK merge-mining tag committed in the BTC coinbase. |
coinbase_op_return, coinbase_ascii_strings |
Raw coinbase evidence. |
is_uncle, uncle_index, uncle_parent_height |
RSK uncle (stale) metadata. |
The full stale/unknown inventory (rsk_stales.csv, exposed for discovery as
rsk_stale_blocks.csv, 354
stale-labelled observations + 37,410 unknown rows after the 2026-09-08
height-zero acquisition) and its 236,073-row canonical companion are private.
Four consensus-invalid parents route to the error-block sibling output, and
the exact-key gate excludes the stale-labelled BTC 789,038, 783,426 and
784,121, leaving 351 direct-stale rows in the committed validated file. Three RSK observations of
stale-descendant parents, including BTC 656,478, enter publication through the
authenticated parent/witness module.
The stale-descendant module separates parent verdicts from child-chain witness provenance:
data/stale_descendants.csvcontains accepted Bitcoin parent verdicts. Loaders requireclassification=stale_descendantandvalidation_status=VALID_STALE_DESCENDANT, authenticate the serialized parent header, require the exact persisted schema and accepting gate cells (expected_nbits,pow_valid,header_hash_match, and the height-and-version-appropriate BIP34 status), require the persisted Bitcoin Core off-active verdicts, and reconcile the root height, fork depth, path endpoints, and serialized predecessor links. The terminal identity must also occur in the selected data tree's accepted per-chain or pinned upstream direct-stale inputs after the error-block exclusion. An intermediate node without a recovered parent verdict must have an exact height/hash entry and matching 80-byte predecessor header in that tree's pinned upstream census, after the same error-block exclusion. Such nodes authenticate path edges without adding a child-chain witness or parent row.data/stale_descendant_observations.csvcontains authenticated child-chain witnesses for those parents. Exact source coordinates, source SHA-256, and child identity bind each row to the recovered evidence. The canonical loader requires the exact ordered witness schema and complete row widths and rejects non-canonical POSIX source paths. A row without a serialized child header must match the exact chain, parent, child height, and child hash in the selecteddata/tree'schild-identity/index; any populated child time, header, ornBitsevidence must agree. One authenticated child event can witness exactly one Bitcoin parent, while distinct child events from the same chain may witness the same parent.
Reconciliation evaluates every authenticated candidate, begins only from the
declared trusted-root set, and verifies the full predecessor path. A direct
stale root is accepted only when its predecessor is on Bitcoin's active main
chain. A consensus-invalid full-proof-of-work candidate is classified as an
error block before stale-descendant publication. source_classification in the
witness ledger records the archive bucket for audit; it does not decide the
parent verdict. Attribution requires this ledger and validates it against the
parent table before admitting any descendant. The four ancestry-derived error
blocks are reviewed members of the canonical error_blocks.csv and
error_block_observations.csv module.
Reconciliation treats their hashes as terminal error verdicts and fails closed
if it finds any additional consensus-invalid candidate. Children and deeper
descendants of a catalogued error terminal inherit that invalid verdict and go
only to the disposable error-candidate diagnostic. The diagnostic also retains
incomplete or low-work rows as publication blockers, but only an authenticated
full-PoW violation is classified for canonical error-module admission. A
candidate is likewise unpublishable when the committed epoch reference cannot
supply canonical expected_nbits for its inferred height.
An unresolved observation remains in its raw source; withdrawing an unsupported accepted verdict removes its parent and witness claims from the accepted tables without erasing the observed header or establishing consensus invalidity.
Key columns beyond the standard BTC header fields include
active_mainchain_status and root_active_mainchain_status for the placement
gates; ancestry_relation; root_stale_hash / root_stale_height /
root_stale_sources / root_stale_chains for the trusted stale root;
stale_fork_depth, path_hashes, and path_chain_sets for the walked ancestry;
observed_btc_heights, observed_btc_times, observed_btc_bits,
observed_chains, and source_observation_count for the aggregated witnesses;
and pow_valid, header_hash_match, bip34_height_status, and notes for the
validation result.
The witness ledger follows the error-observation provenance shape: parent and
child identities, child header evidence, source coordinates and SHA-256,
height/identity provenance, parent row number, and a composite provenance
field. A split _unknown_blocks.csv row retains that exact path while its
source_kind identifies the shared full_inventory schema family, including
the family's child-height trust rules. Source paths are publication
coordinates, not classification rules.
Huntercoin and Xaya use one complete current classifier family. The selected
archive root must supply stale_blocks, canonical_blocks and unknown_blocks
CSVs together in one directory, including header-only empty splits. The first
root containing any family member is authoritative; a missing or ambiguous
member fails discovery rather than being filled from another archive or the
local data directory. Compact validated-stale inputs remain usable on their
own when no classifier family is selected.
Ancestry reconciliation checks the selected data tree's family before loading
candidates, including when producing disposable partial diagnostics.
When a never-split inventory and its canonical companion contain the same authenticated child event, reconciliation keeps one deterministic source coordinate only after their parent evidence is compatible. Conflicting copies fail closed, and distinct authenticated events remain distinct witnesses.
Upstream contribution sidecars: data/new_stale_blocks_for_upstream.csv and data/upstream_header_fills.csv
new_stale_blocks_for_upstream.csv is the set of publication-gate-accepted
stale (height, hash) pairs this project
contributes that are not already in bitcoin-data/stale-blocks.
upstream_header_fills.csv is the companion fill set: the accepted candidates
whose (height, hash) already exists upstream but is recorded without a
header, so the committed header can be contributed onto the existing row.
Both are rebuilt together from the committed inputs
above by scripts/reports/build_upstream_stale_sidecar.py.
The builder reads direct stales from the validated per-chain inputs and
descendants through the canonical parent loader. It never uses a previous
sidecar as an input.
Because these sidecars intentionally combine direct stales and accepted stale
descendants without a classification column, ancestry reconciliation never
uses them as a stale-root input. Direct roots come only from effective upstream
and accepted per-chain direct-stale inventories.
Both files share one schema:
| Column | Meaning |
|---|---|
height |
Canonical BTC height. |
hash |
Stale block hash, display order. |
header |
Full 80-byte BTC header, hex (160 chars). |
The full-evidence export is generated on demand by
scripts/reports/build_auxpow_full_evidence.py or just full-evidence. It is
not a stale census input and is gitignored because private archive runs can
write bulky row-level inventories. The export is for downstream consumers that
need candidate evidence before rechecking it against Bitcoin Core.
When an external publication is assembled through a staging directory,
--reported-output-dir records the logical final location in its manifests
without changing where the files are written.
The 2026-07-30 complete refresh retained 30 external artifacts. Its historical coverage report authenticated 2,933,154 of 2,933,154 rows and all 6 accepted stale-descendant observations belonging to those 17 sources, with zero unrecoverable rows.
The September 2026 I0coin/RSK coverage report authenticated 3,019,416 of 3,019,416 historical source rows and all 6 accepted stale-descendant observations belonging to those 17 sources, with zero unrecoverable rows. The larger I0coin inventory adds 86,267 rows; current catalogue exclusions remove two Devcoin, two Ixcoin and one Emercoin row from the earlier coverage population.
The combined 9 September 2026 coverage report adds one independently reviewed ROD canonical observation to that generation. Its separate complete native scan covers 4,127,690 blocks through height 4,127,689, including 1,058,017 SHA256d proofs. The private extraction and Bitcoin-context ledger preserve every observation; only the one eligible canonical witness enters the standard publication. The verified combined coverage authenticates all 3,019,417 observations across 18 source rows, with zero unrecoverable rows and all 6 accepted stale-descendant observations hydrated. The existing 17 sources retain the September I0coin/RSK generation and are not replaced with the smaller Monitor projection. See ROD provenance.
Each <chain>_evidence.csv uses this normalized schema:
| Column | Meaning |
|---|---|
chain, source_kind, source_path, source_row_number |
Source identity and row provenance. External archive roots are redacted as <chain-archive>/.... |
artifact_scope |
full_classifier_inventory, stale_only_publication, or stale_descendant_parent_verdicts. |
child_height, child_block_hash, child_header_hex, child_block_time, child_nbits |
Child-chain location and authenticated header evidence when available. child_block_hash is the internal/wire-order double-SHA256 digest, child_header_hex is the 80-byte canonical header serialization, child_block_time is the header timestamp, and child_nbits is eight lowercase hexadecimal characters. For Xaya and ROD, the header and timestamp come from CPureBlockHeader, while effective child_nbits comes from the adjacent PowData wrapper. For the five identity-hydration chains the existing hash and timestamp remain hydrated from data/child-identity/ (see below); their historical data is not re-derived by this work. Hathor publishes its source-authenticated block identity and timestamp directly. |
btc_height, btc_header_hash, btc_prev_hash, btc_time, btc_bits, btc_nonce, btc_header_hex |
Normalized Bitcoin parent header fields. |
coinbase_scriptsig_hex, coinbase_outputs, full_coinbase_hex |
Coinbase evidence. Hathor-style full rows with only full_coinbase_hex are parsed during export. |
classification |
Preserved source classification: canonical, stale, unknown (normalized from the historical orphan spelling on read), stale_descendant, near, or source-specific values. |
validation_status, expected_nbits, rejection_reason |
Persisted gate verdicts or rejection details when present. |
auxpow-full-evidence-counts.csv and
auxpow-full-evidence-manifest.* summarize counts for canonical, stale,
unknown, stale_descendant, near, rejected rows, malformed rows, and missing
evidence fields. The manifest's canonical_evidence_status distinguishes
canonical_retained, no_canonical_rows_in_discovered_artifact,
canonical_rows_not_represented_stale_only_publication_path, and
not_checked_missing_source.
Canonical Bitcoin parents recovered from a chain's merge-mining commitments
are retained evidence: they record which pool was building on which canonical
block, via which chain, at what time. The classifier partitions its results by
the primary classification into four files sharing one column schema (via the
shared write_classifier_outputs): <chain>_canonical_blocks.csv
(classification = canonical, empty validation_status/expected_nbits, with
Bitcoin Core's height authoritative), <chain>_stale_blocks.csv (stale rows
only), <chain>_unknown_blocks.csv (unknown rows), and the committed
<chain>_validated_stales.csv (the VALID-stale subset the loaders read). The
scratch paths under data/ are gitignored (_stale_blocks, _canonical_blocks,
_unknown_blocks). Canonical and unknown storage normally remains in the
private archive's classified/ family.
Every available canonical row is part of the normal publication and lives in
the corresponding
results/monitor-evidence/<chain>_monitor_evidence.csv output. There is no
per-chain canonical allowlist. A source's own coverage limit remains visible
in its provenance: VCash, for example, is a 68-row partial canonical subset
rather than a complete chain recovery, so no stale, strict, or weak total may
be inferred from it.
RSK keeps its miner-address parallel schema in a private stale/unknown inventory
plus the separately discovered rsk_canonical_blocks.csv companion. Fresh
full-inventory and canonical-companion rows retain the source RSK height, block hash, timestamp, miner,
merge-mining hash, uncle placement, mapped merkle proof, and compressed
coinbase tail. That source bundle normalizes without a historical sidecar. A
verified sidecar row for another RSK child observation of the same Bitcoin
parent is unrelated and is not applied. Only an exact height, hash, and time
match can replace the source bundle; every populated source cell must then
agree, so a field-by-field hybrid is never published. The raw CSV, exact
fallback ledger, completed extraction checkpoint, and classifier-family hash
manifest remain private. Publication verifies the manifest and every staged
output before a fresh RSK inventory is read. Classification permits Bitcoin
tip growth only when header and parent placements remain unchanged after a
final recheck. A changed placement, reorg through the starting tip, or tip
movement during that final check requires classification to be rerun.
Repository dependencies in the
version-4 classifier manifest resolve relative to the consuming research
checkout and must retain their recorded byte digests. Custom dependencies
outside the checkout remain relative to the manifest and must travel with the
run in the same relative layout. The compact rsk_validated_stales.csv keeps
its established schema and still requires the separate child-identity ledger
when used without the full private inventory. Hathor's unified classifier
writes the standard terminal category files for canonical, stale, unknown,
near, and error-block rows.
Qbit's complete native scan through child height 80,986 publishes 2,536
canonical parents and four accepted direct-stale observations. It records
the exact pure child header, native height and proof at acquisition, so no
separate identity hydration is needed. Its 24 zero-predecessor synthetic
parent proofs remain private unknown evidence: their own targets pass, but
the Bitcoin epoch targets do not. The original 44-byte commitment prefix is
not a BIP34 height. just acquire-qbit --from-acquisition ... verifies the
sealed captures and reruns the normal producer into a fresh output directory.
See Qbit provenance.
Five active child chains (Namecoin, RSK, Syscoin, Elastos, and Fractal) were
recovered without child block identity: their acquisition artifacts carry the
child height but no child block hash or timestamp. Hathor is active too, but
its API acquisition retains the source-authenticated block identity and
timestamp directly, so it is not a separate hydration target.
merge-mining-monitor keys live-captured events by
(source, child_height, child_block_hash), so imported rows need the exact
identity to deduplicate against live capture.
scripts/extract/recover_child_identity.py recovers each target child event
from its exact source hash when available, falling back to height -> hash only
for an unambiguous slot. Distinct events on one chain may commit to the same
Bitcoin parent; exact repeated events, unresolved same-height slots, or one
child event assigned to different parents are rejected.
The RSK work list also includes error-observation ledger parents, which
ordinary inventories exclude; when classified uncle metadata is absent, only
an unambiguous height-only target tries the recorded height as a canonical
child.
Every row is verified by re-deriving the Bitcoin
parent linkage from the fetched child block and requiring it to match the
row's own btc_header_hash: the decoded CAuxPow parent for Namecoin/Syscoin,
the serialized AuxPoW tail for Elastos, the getblockheader (hash, false, true) proof for Fractal, and sha256d(bitcoinMergedMiningHeader) for RSK
(uncle rows resolve through eth_getUncleByBlockNumberAndIndex). All 2,365
chain/header observations across the five active hydration chains verify
against today's reachable child nodes (1,940 distinct Bitcoin parent
headers; 246 parents were observed by more than one chain, a single miner
attaching one Bitcoin parent to several merge-mined chains at once).
The directory also contains five-row historical identity ledgers for Devcoin and Ixcoin. Those rows pin source-row-authenticated 80-byte child headers for archive observations; the four new BIP34 error witnesses additionally record independent node RPC verification. They authenticate historical evidence and do not imply that the archive rows were live monitor events. Across all seven generic ledgers the directory contains 2,375 observations for 1,941 distinct Bitcoin parents, 252 of them observed by more than one chain.
Each <chain>_child_identity.csv carries chain, btc_header_hash,
child_height, child_block_hash, child_block_time, verification, note,
child_header_hex, and child_nbits. A verified row is usable only when
child_height is a nonnegative integer and the child hash and time are valid.
When a serialized child header is present, its hash, timestamp, and nBits
must agree with the normalized fields. Xaya is the exception for nBits: its
effective value comes from the adjacent PowData wrapper rather than the pure
header. btc_header_hash stays in
display (RPC) order like every other
pipeline artifact; child_block_hash follows the pipeline's established
column contract of internal (wire) byte order for the Bitcoin-family chains,
and forward order for RSK, whose keccak block hashes have no
reversed display convention -- the same orders merge-mining-monitor stores,
so its importer decodes the column without re-reversing. The RSK file adds
the fields merge-mining-monitor's rsk_merge_mining_evidence sidecar
requires: rsk_miner, merge_mining_hash, is_uncle, uncle_index,
uncle_parent_height, rsk_merkle_proof, rsk_coinbase_tail.
The monitor-evidence export groups these files by (chain, btc_header_hash) and
selects the exact authenticated child height/hash before filling the
child_block_hash / child_block_time columns (and, for RSK, the sidecar
columns). A row without a verified identity stays empty, a different child
height is refused, and an ambiguous multi-event join fails closed. The
per-chain counts note records the join coverage as
child_identity_hydration=hydrated:N.
The strict and weak BTC-orphan gates use a compact, committed view of
Bitcoin's canonical retarget history rather than an operator's private node or
runtime cache. btc_nbits_by_epoch.json maps every retarget-boundary height to
its compact target. btc_epoch_headers.json carries the corresponding block
hash, timestamp, median time, and target, followed by a six-confirmation
horizon block that closes timestamp coverage through the refresh point.
manifest.json records the public Esplora source and exact finalized horizon.
Regenerate the three files with just refresh-bitcoin-epoch-reference. The
refresh is incremental, verifies the last retained epoch against the public
canonical chain, and refuses to continue if that identity changes. Scheduled
CI runs it weekly, proposes changes as a pull request, and merges that pull
request once CI passes.
A final-category projection of the full-evidence export, generated by
scripts/reports/build_monitor_evidence.py or just monitor-evidence. It
keeps only the categories the merge-mining-monitor ingests as final: every
available canonical row, publication-gate-accepted direct-stale candidates,
accepted stale descendants, and strict/weak BTC orphans. Unknown rows survive
only when the relevance inventory (see the vocabularies below) supplies a
strict_btc_orphan / weak_btc_orphan verdict for their header hash;
excluded and pending rows are omitted. Normal publication includes canonical
evidence uniformly for every chain. --skip-canonical is permitted only in an
explicit --allow-partial diagnostic build with a disposable output
directory.
--output-dir selects the physical destination. For an isolated publication,
--reported-output-dir selects the logical final directory recorded in the
manifest, counts CSV and artifact map, using the same convention as the
full-evidence producer. Omit it to report the physical destination. The writer
sanitises the logical directory once and appends each artifact filename, so
ordinary chain files, the error aggregate and both metadata files describe
one publication location. A partial diagnostic cannot report the committed
Monitor directory as its logical destination.
This flag does not select the preservation baseline or bypass publication
gates. The baseline remains the runtime checkout's committed
results/monitor-evidence/ files. When assembling a generation separately,
materialise the complete previous publication there before running the normal
producer. --data-dir selects the new canonical data inputs; original
classifier dependency bytes must still remain available at the paths bound
by a source-family manifest. Keep the complete RSK raw/checkpoint/skip-ledger
bundle and its classified siblings together, and verify them without
rewriting their dependency fingerprints.
The complete ancestry publisher also accepts --data-dir. It validates the selected tree's error module, epoch reference and pinned invalid-block bodies, reconciles the full candidate population, then installs both ancestry CSVs into that tree with the existing rollback protection. This lets a normal staged rebuild refresh publication inputs without changing retained classifier dependencies in the runtime checkout.
The per-chain *_monitor_evidence.csv payloads are stored in Git LFS. Run
git lfs pull --include="results/monitor-evidence/*_monitor_evidence.csv"
before reading or rebuilding them. monitor-evidence-counts.csv and
monitor-evidence-manifest.json remain ordinary Git files so counts,
provenance, and validation-contract changes are directly reviewable. The
shared evidence writer emits LF explicitly because LFS objects do not pass
through Git's text-normalization filter.
error-block-observations_monitor_evidence.csv is a separate 121-row aggregate
for the 53 catalogue parents. It uses the 34-column union schema: the shared
27 monitor-evidence columns plus the seven RSK sidecar columns
(rsk_miner, merge_mining_hash, is_uncle, uncle_index,
uncle_parent_height, rsk_merkle_proof, rsk_coinbase_tail). Non-RSK rows
leave those sidecar cells blank. RSK rows must carry a semantically valid
sidecar (20-byte miner, 32-byte merge-mining hash, is_uncle 0/1, uncle
placement present only for uncles, proof/tail blank or hex) copied from
data/child-identity/rsk_child_identity.csv without replacing the ledger
child hash. RSK keccak child hashes stay in forward node order; Bitcoin-family
child_block_hash_order=display hashes are exported in internal order. The
seven RSK error-observation parents are excluded from ordinary RSK publication,
so this aggregate is the only published place those sidecars live.
Its rows retain original archive, commit-pinned validated-input or
live-observation source coordinates but use classification=error_block and
the catalogue's consensus rejection reason. expected_nbits is the required
epoch target, so it can intentionally differ from the header's btc_bits for
a retarget violation. The compact recovered witness ledger lives at
data/error-blocks/error_block_observations.csv; it is checked against the
current parent catalogue before publication. The catalogue declares each
(chain, child_height, parent_hash) summary, while the ledger owns the exact
(chain, child_height, child_block_hash, parent_hash) event identity. This
preserves distinct sibling blocks from the same chain and height. Each
canonical source coordinate may authenticate only one exact event. Every
ledger row must have the exact canonical field count and identify its child
either with a well-formed hash or with a serialized child header from which
that hash can be authenticated. Staged publication applies the ordinary
parent/child evidence checks to the aggregate and requires every
catalogue/ledger-derived field and identity to match the canonical 121-row
module exactly; source-derived coinbase output enrichment may add evidence but
cannot replace it. The release path stages every ordinary artifact, the error
aggregate, counts, and manifest as one coherent transaction after validating
them against the current schemas and source contracts.
The September I0coin snapshot initially supplied three exact error
observations. The recovered snapshot
adds witnesses at Bitcoin heights 331,673 and 331,674, with child heights
1,315,879 and 1,315,884, and corrects the existing height-367,047 witness's
child height from 1,546,542 to 1,546,541. The latter is the same authenticated
child identity with corrected ancestry metadata, not an additional event.
The two new source rows remain unknown with blank source_btc_height;
their ledger btc_height comes from the catalogue's verified ancestry and
uses btc_height_provenance=catalogue-authenticated-ancestry-placement.
Together with catalogue-active-parent-placement, this is an explicit
provenance case allowing a blank source height. The catalogued Bitcoin height,
exact witness identity, authenticated child header, source coordinates and
consensus-failure gates remain required.
Each <chain>_monitor_evidence.csv uses the full-evidence schema plus two
columns the monitor's importer parses verbatim. The current schema includes
child_header_hex, child_block_time, and child_nbits; rows from the 17
historical child-header refresh pipelines and the reviewed ROD canonical
companion carry a complete authenticated bundle. The five identity-hydration
chains additionally publish child_block_hash / child_block_time hydrated
from data/child-identity/ (coverage recorded in the counts notes as
child_identity_hydration=hydrated:N; a publication build fails closed when
a non-canonical final row lacks a verified identity). Canonical rows remain
publishable when their historical source never recorded exact child identity;
the counts notes disclose that limit as canonical_unhydrated=N. The RSK
export appends the seven rsk_merge_mining_evidence sidecar columns listed in
the child-identity section. Sources outside those historical-header pipelines
and the ROD companion emit the child evidence their native source proves; absent fields stay empty
rather than being inferred. The standalone stale-descendants export contains
one row per accepted parent and therefore leaves child identity blank. Its
counts notes are child_height=unavailable and
parent_verdicts_only_witnesses_in_observation_ledger. The
exact source-chain witnesses come from
data/stale_descendant_observations.csv and enter their chain artifacts once
with classification=stale_descendant,
validation_status=VALID_STALE_DESCENDANT, and
relevance_reason=valid_stale_descendant. Their source-bucket classifications
remain audit fields in the witness ledger and never override the accepted
parent verdict. The ordinary artifacts, error-observation aggregate, and
both metadata files describe the same source generation. The six historical
observations carry complete authenticated child headers; live-chain
observations use the independently verified identities in
data/child-identity/. The committed ledger is the sole witness interface, so
all accepted observations are projected. A complete publication fails closed
when a required child identity is missing or disagrees with its generic ledger; a
partial diagnostic records that hydration shortfall instead of claiming a
usable identity.
i0coin and CoiledCoin publish their authenticated child hash, header, time, and
nBits. Their full_inventory and canonical_blocks source families retain
the existing normalization contract that leaves child_height empty, rather
than trusting historical block-file scan counters. I0coin's complete snapshot
now supports independently verified genesis-linked heights in its private
research records, accepted direct-stale rows and error-observation ledger.
Those verified heights do not change the normalized full-inventory or
canonical-companion contract. Doichain's retained
historical inventories likewise leave their unauthenticated file-order values blank; a
future run can fill exact heights through its documented RPC normalization
pass. The old block-file scan counters have been removed rather than presented
as heights. Height columns remain present uniformly. Any non-empty export with
no exact child height records child_height=unavailable; downstream importers
that require a height must skip those rows until an exact height source becomes
available.
Huntercoin and Xaya use the Kraft native classifier families as their foundation
inputs. Previous Arweave/CDN families are historical provenance and are not
fallback sources. Xaya's xaya_height records the coinbase-derived native
height; ordinary shared height resolution publishes it as child_height.
Both current classifier families use the ordinary
<chain-archive>/<chain>/classified/ layout. Xaya uses the existing
canonical_blocks source kind. Current exact heights remain
protected by the normal publication checks; there is no scan-position repair
or preservation exception for superseded Xaya inputs.
The child-header coverage report cross-references those accepted observations by source chain and Bitcoin header hash so their coverage is explicit:
| Column | Meaning |
|---|---|
btc_stale_relevance |
Empty for accepted stales/descendants, whose state is already represented on the primary classification plus validation_status axes, and for canonical rows in canonical-bearing local runs; strict_btc_orphan / weak_btc_orphan for admitted unknown rows. |
relevance_reason |
valid_direct_stale, valid_stale_descendant, strict_height_nbits_match, or timestamp_epoch_nbits_match. |
Namecoin's monitor export previously required partial header hydration for
its stale rows. The validated loader now carries btc_header_hex for all
1,641 accepted rows (back-filled from the committed monitor evidence), so
stale-row hydration has no remaining targets. The complete September Monitor
rebuild uses those loader headers directly; its manifest no longer carries
the earlier namecoin_header_hydration=hydrated:228 note. The 21
published strict/weak unknown rows already carry 160-character headers from
their private full-inventory source rows, so reproducing them requires that
private inventory but not the hydration inputs. A recovered header is
written only when its double-SHA256 verifies against the row's
btc_header_hash; a missing, malformed, or mismatching candidate is never
written. Hydration touches only Namecoin's btc_header_hex column.
monitor-evidence-counts.csv and monitor-evidence-manifest.json record
per-chain category counts, publication-projected source rows across the main
source and any replacement or companion sources, whether
unknown rows were present without a relevance inventory to judge them, and --
for namecoin -- the header-hydration coverage in the notes field
(namecoin_header_hydration=hydrated:N/still_missing:M/hash_mismatch_rejected:K,
with no_namecoin_header_hydration_sources_found appended when no extract file
was available). A non-empty chain export also records
child_height=unavailable when one or more rows lack an exact height. The JSON
manifest also publishes observation_chain_counts, the canonical per-chain
inventories for the error-block and stale-descendant observation ledgers. These
are witness-observation counts, not parent counts, so consumers do not need to
reopen those lower-level ledgers to reconstruct that metadata. Ordinary stale,
stale-descendant, and strict/weak orphan rows remain
covered by the existing per-chain category counts. Each inventory includes only
chains with observations; an absent chain means zero. Complete publication
builds carry both observation-ledger keys, while partial diagnostic builds may
omit error-block-observations. The manifest also carries a machine-readable
validation_contracts object. Its ordinary_monitor_evidence contract's
valid_token_scope=publication_gate_accepted_not_full_block_validity value is
the normative interpretation of both exact accepted direct-stale statuses for
ordinary evidence. The error-block-observations contract is separate: its
VALID_ERROR_BLOCK token means an authenticated witness of a
consensus-invalid Bitcoin parent, not a valid Bitcoin block. Consumers must
select the contract for the artifact they are reading rather than applying
the ordinary stale contract globally.
For monitor projection of full-inventory sources, source_rows replaces raw stale-classified rows
with the committed gate-accepted direct-stale rows. It can therefore
be smaller than the full-evidence or child-header coverage total when a stale
candidate was rejected. A canonical row present in both the main inventory and
its canonical companion is likewise counted once, matching the deduplicated
publication projection. Across the 27 source families represented by both
producers in the retained comparison before the body-invalid cutover, only
Groupcoin had this difference: 4,867
full-evidence source rows versus 4,866 Monitor source rows. This records one
rejected candidate, not missing source acquisition. Full evidence includes
that rejected row in its 31 stale-classified Groupcoin rows; rejected=1 is
an overlapping diagnostic count, not an additional row.
In that retained comparison, the aggregate source populations also differed because Monitor included the 68-row VCash canonical subset and the separate 88-row error-observation ledger, neither of which appears in full-evidence discovery. Full evidence therefore contained 4,560,214 source rows, while Monitor reported 4,560,369: 68 VCash rows plus 88 error observations, less the rejected Groupcoin row. These source totals are distinct from final Monitor row counts. Monitor now admits 26 strict/weak observations from the unknown bucket and projects all 33 accepted descendant witnesses into their observing chains, alongside the separate 21-row descendant parent table.
The committed strict/weak slice of the evidence, produced per chain like
the validated-stales loader inputs: each chain's file is its
monitor-evidence export filtered to the classification=unknown rows,
which are exactly the rows admitted with a strict_btc_orphan or
weak_btc_orphan verdict (same schema and row order as the export). The
verdict is per header, not per observation: a header seen by several chains
carries the strongest verdict any chain's evidence establishes, so the same
header can appear in more than one chain's file with the same verdict.
Every complete-coverage per-chain monitor-evidence export gets a file; a
header-only file means the chain has no admitted rows. Canonical-only partial
sources such as VCash and stale-only publication inputs are omitted because
they cannot establish a zero result. Hathor's complete retained corpus
publication therefore carries a header-only zero-result file.
strict-weak-orphans-counts.csv
summarizes the per-chain bucket counts. Generated by
scripts/reports/build_strict_weak_orphans.py (just strict-weak-orphans);
the underlying row-level relevance inventory
(results/analysis/btc-stale-relevance/btc-stale-relevance-inventory.csv) is gitignored for
size.
The generated report owns current upstream overlap and
chronological novelty for all registered chains. Run just novelty after a
compact input or upstream-pin change; just novelty-check detects drift without
writing. scripts/compute_chain_novelty.py --output <path> writes a disposable
review copy. The former positional chain and --no-csv options are removed.
The generator requires complete compact inputs, valid CSV schemas and a clean
upstream clone at the declared data-sources.tsv pin. STALE_BLOCKS_DIR may
select another clone at that same pin. Missing or mismatched inputs fail before
replacement; nothing is fetched automatically. A header-only validated input
is a verified zero-row result. Canonical-only chains have no validated CSV and
are labelled separately; an unexpected file requires a registry transition.
The report records the pin and a deterministic fingerprint of validated CSVs,
the upstream CSV, the error catalogue and ordered chronology. It uses the
existing gated loaders and exact (height, hash) identity. Direct-stale union
totals exclude descendants; contribution-sidecar totals can therefore differ.
Row-level evidence remains in the compact inputs, without a second per-chain
novelty CSV inventory.
The taxonomy has two axes. The primary classification records what the
evidence shows about a recovered BTC parent header; the derived
btc_stale_relevance refines the unknown bucket with a final verdict. The
merge-mining-monitor uses the same two-axis design (its BTC-orphan classifier
is a port of this repo's relevance classifier), and its importer reads the
relevance strings verbatim - renames must land in lockstep with the monitor.
"Orphan" is used only in the strict/weak relevance sense throughout this
repository; the broad evidence state is unknown.
classification (primary axis): canonical (Bitcoin Core reported the
header on the active chain at classification time), stale (an operational
direct-stale header candidate: the recovered BTC parent header was not active
and its prev hash was active), unknown (neither the header nor its prev hash is
known to the active chain; stays unknown on this axis; historical private
artifacts may use orphan, and readers normalize that spelling), stale_descendant
(a stale-fork continuation chaining back to a known stale root), near (the
header fails Bitcoin's PoW target - a child-chain artifact that was never a
Bitcoin block and remains outside the Bitcoin-parent population), error_block
(a consensus-invalid full-proof-of-work Bitcoin block witnessed via
merge-mining evidence: the header hash meets the Bitcoin target in force at
the claimed position and the merge-mining commitments are authentic, but the
block violates at least one named, mechanically re-checkable consensus rule, so it
was never a stale/orphan contender). Error blocks are catalogued in
data/error-blocks/error_blocks.csv and excluded from stale publication by
exact parent identity. Blocks that merely fail the PoW target are not error
blocks; they remain near. Adding error_block is a
breaking vocabulary change for the merge-mining-monitor importer and its
ported relevance classifier: renames and additions must land in lockstep with
the monitor.
btc_stale_relevance (derived refinement of unknown rows, emitted by
scripts/analysis/classify_btc_stale_relevance.py; constants in
stale_blocks_analysis.config). An accepted direct-stale or stale-descendant
candidate is already represented on the primary classification plus
validation_status axes, so it carries an EMPTY btc_stale_relevance here
(with relevance_reason valid_direct_stale / valid_stale_descendant);
this derived axis holds only the unknown-row refinement values:
strict_btc_orphan- unknown row with valid self-PoW plus strict height evidence whose nBits match Bitcoin's at that height. This derived classifier conservatively admits coinbase-height evidence only from the universal BIP34 boundary at 227,931, avoiding version-dependent inference in the earlier transition window.weak_btc_orphan- unknown row with valid self-PoW and timestamp-selected BTC nBits consistency within plus/minus one retarget epoch (no strict height).excluded- everything ruled out: canonical/near sources, PoW or nBits failures, known stale/mainchain membership, insufficient evidence.pending- the row's height or timestamp lies beyond the nBits table's coverage horizon; no final verdict yet, re-classifiable when the table is extended.
validation_status: VALID or
VALID (post-BCH, difficulty matches BTC) (passed the available chain-specific
publication gates, not full block validation), VALID_STALE_DESCENDANT
(descendant accepted under its declared profile), REJECTED: ... /
REJECTED_... (failed a gate, for example
an nBits mismatch with the canonical BTC height or a BIP34 height mismatch),
UNKNOWN: ... (required context or evidence unavailable at gate time).
Committed per-chain validated CSVs contain only the two exact accepted
direct-stale statuses. Two rejected spellings exist historically - the nBits
gates write REJECTED: <detail> and
the descendant reconciler writes REJECTED_<reason> - so consumers must match
on the REJECTED prefix, never on an exact string.
Either accepted direct-stale spelling means the declared publication profile passed. It is not a claim that Bitcoin Core accepted a complete block. The compact direct-stale rows do not contain the complete Bitcoin transaction set, so the project cannot check the candidate's full merkle tree, block transaction rules, weight, witness commitment, sigops, finality, scripts, subsidy, fees, or UTXO effects.
Chain status (used in the per-chain docs): the data-availability axis is
live / historical / catalogued (does this project have live, recovered,
or merely catalogued evidence), and the chain-liveness axis is active /
dead / dormant / zombie (a zombie chain still produces blocks at
negligible sub-Bitcoin difficulty). These are independent axes; the
merge-mining-monitor uses the same vocabulary.
Pool labels: pool attribution is implemented as a separate layer
(see pool-attribution.md): just attribute matches
coinbase evidence against the pinned bitcoin-data/mining-pools registry and
labels the merge-mining-recovered stales, writing an export with pool,
template_producer, and
attribution_basis columns as a gitignored run product. The upstream census
is not labelled in this repo; the combined census-plus-recovered set is
assembled by the companion analysis repo. No committed dataset
carries a shared pool-label column. RSK's committed
source data is a historical exception because it already carries miner-address
labels; consumers should not treat that chain-specific field as a completed
cross-chain attribution dataset.