diff --git a/docs/packages/overlays/overlay-discovery-services.md b/docs/packages/overlays/overlay-discovery-services.md index 1fb0ff1b4..0aa1bbd52 100644 --- a/docs/packages/overlays/overlay-discovery-services.md +++ b/docs/packages/overlays/overlay-discovery-services.md @@ -4,7 +4,7 @@ title: '@bsv/overlay-discovery-services' kind: package domain: overlays npm: '@bsv/overlay-discovery-services' -version: '2.2.2' +version: '2.2.3' last_updated: '2026-09-18' last_verified: '2026-09-18' review_cadence_days: 30 diff --git a/docs/packages/overlays/overlay-express.md b/docs/packages/overlays/overlay-express.md index 3af920759..24c1ff7ea 100644 --- a/docs/packages/overlays/overlay-express.md +++ b/docs/packages/overlays/overlay-express.md @@ -4,7 +4,7 @@ title: '@bsv/overlay-express' kind: package domain: overlays npm: '@bsv/overlay-express' -version: '2.6.2' +version: '2.7.0' last_updated: '2026-09-18' last_verified: '2026-09-18' review_cadence_days: 30 diff --git a/docs/packages/overlays/overlay-topics.md b/docs/packages/overlays/overlay-topics.md index ac636f130..63205b89f 100644 --- a/docs/packages/overlays/overlay-topics.md +++ b/docs/packages/overlays/overlay-topics.md @@ -4,7 +4,7 @@ title: '@bsv/overlay-topics' kind: package domain: overlays npm: '@bsv/overlay-topics' -version: '1.8.1' +version: '1.8.2' last_updated: '2026-09-18' last_verified: '2026-09-18' review_cadence_days: 30 diff --git a/docs/packages/overlays/overlay.md b/docs/packages/overlays/overlay.md index 1fc0675a6..312d65026 100644 --- a/docs/packages/overlays/overlay.md +++ b/docs/packages/overlays/overlay.md @@ -4,7 +4,7 @@ title: '@bsv/overlay' kind: package domain: overlays npm: '@bsv/overlay' -version: '2.4.0' +version: '2.5.0' last_updated: '2026-09-18' last_verified: '2026-09-18' review_cadence_days: 30 diff --git a/docs/packages/sdk/bsv-sdk.md b/docs/packages/sdk/bsv-sdk.md index f0fc7e200..24a6a630e 100644 --- a/docs/packages/sdk/bsv-sdk.md +++ b/docs/packages/sdk/bsv-sdk.md @@ -15,6 +15,12 @@ repo: 'https://github.com/bsv-blockchain/ts-stack/tree/main/packages/sdk' # @bsv/sdk +The current unpublished candidate corrects BUMP offset arithmetic above 32 bits +through `Number.MAX_SAFE_INTEGER`, preserving existing wire encodings. Root +calculation, extraction, combination and trimming use the same exact numeric +domain; malformed non-integer and unsafe offsets fail explicitly. No consumer +migration is needed for this correction. + The foundational cryptographic and transaction library for the BSV blockchain. Zero external dependencies — all cryptographic primitives have been validated by a third-party auditor. Every other library in the ts-stack builds on top of `@bsv/sdk`. Provides low-level primitives (keys, signatures, hashing), script construction and execution, transaction creation and signing, and integration interfaces for wallets and overlay networks. diff --git a/docs/reference/package-api-migrations.md b/docs/reference/package-api-migrations.md index 40ec9a8b5..0c370b6bb 100644 --- a/docs/reference/package-api-migrations.md +++ b/docs/reference/package-api-migrations.md @@ -42,13 +42,13 @@ and clean-consumer tests remain the executable type authority. | `@bsv/gasp` | `1.3.1` | `1.3.6` | patch | [API and usage](../packages/overlays/gasp.md) | No consumer migration is required; existing constructor calls, imports, synchronization behavior, and wire semantics are unchanged. Distributors must retain THIRD_PARTY_NOTICES.md and LICENSES/ with the package. | | `@bsv/lch` | `0.0.0` | `0.1.0` | minor | [API and usage](../packages/content/lch.md) | No consumer migration is required; this is the first release of a new additive package. Applications must give quote() the verified Asset encryption descriptor and selected Offer key-delivery mechanism, keep createPayment behind explicit wallet authorization, persist the funded transaction and every partial settlement proof through recovery, retry with that same transaction, distinguish finalized from accepted evidence, fail closed on unknown settlement or evidence profiles, and supply an address-pinning connector for DNS endpoints. Payees should select receipt-complete unless they explicitly accept authorized-output provider, privacy, and pre-internalization key-release tradeoffs. Distributors must retain THIRD_PARTY_NOTICES.md with the package; the new profile adds no dependency. Published BRC-170 remains authoritative if the implementation and standard differ. | | `@bsv/message-box-client` | `2.4.0` | `2.5.1` | minor | [API and usage](../packages/messaging/message-box-client.md) | No API migration is required. socketOptions is optional and nothing is forwarded by default, so socket transport negotiation and every HTTP code path are unchanged for existing callers; wallet and originator stay client-owned and cannot be overridden through it. managerOptions excludes autoConnect and retries: JavaScript callers receive an immediate error for autoConnect: false or nonzero message retries because the live socket starts immediately and AuthSocket does not send Socket.IO acknowledgements. Connection reconnection options remain supported. Upgrade @bsv/sdk and @bsv/message-box-client together; historical number-array wallets, current Uint8Array substrates, and already-pending numeric-key messages interoperate through the same portable transaction form. Distributors who copy the UMD file must keep THIRD_PARTY_NOTICES.md and LICENSES/ with it. No migration is required for the bundled payment optimization. Module consumers can use SDK 2.5.0 or later to enable the same optional extension; existing compatible SDK peer versions remain supported. Recipients must advertise already-validated transaction IDs through x-bsv-payment-known-txids, an optional SDK extension rather than a standardized BRC-105 header; services that omit it retain existing payment behavior. Existing return shapes and payment envelopes remain unchanged. Failed or incomplete notification payments stay queued. Resolve uncertain refund-send outcomes before retrying; this patch adds ordering checks, not an exactly-once refund journal. listMessages/listMessagesLite envelope behavior and basket-insertion semantics remain tracked in issue #503. | -| `@bsv/overlay` | `2.2.1` | `2.4.0` | minor | [API and usage](../packages/overlays/overlay.md) | Existing Engine and TopicManager implementations remain valid. Lookup results default to 1,000 formulas; pass -1 only when an equivalent deployment bound exists. Topic managers whose validation creates provisional external state should implement abortAdmissibleOutputs, while read-only managers require no change. Distributors must retain THIRD_PARTY_NOTICES.md and LICENSES/ with the package. No migration is required for the additive storage capability. Current Engine/Knex behavior is unchanged; no production adapter provides this atomic receipt yet. MongoDB is an optional peer: applications that import a Mongo storage deep entry point must install mongodb ^7.5.0 and explicitly operate the replica-set foundation. No Engine activation, default-adapter change, SQL-to-Mongo migration, or mixed-writer database topology is included. | -| `@bsv/overlay-discovery-services` | `2.1.1` | `2.2.2` | minor | [API and usage](../packages/overlays/overlay-discovery-services.md) | Existing mainnet and testnet advertisers are unchanged. TTN operators pass chain ttn and provision the staging storage and overlay endpoints before advertising. Distributors must retain THIRD_PARTY_NOTICES.md and LICENSES/ with the package. No migration is required for this dependency candidate update; default storage selection and existing runtime behavior remain unchanged. | -| `@bsv/overlay-express` | `2.5.0` | `2.6.2` | minor | [API and usage](../packages/overlays/overlay-express.md) | Existing mainnet and testnet servers are unchanged. TTN servers call configureNetwork('ttn'), configureArcade with the TTN endpoint, and configureChaintracks or configureChainTracker before engine initialization. Distributors must retain THIRD_PARTY_NOTICES.md and LICENSES/ with the package. No migration is required for this dependency candidate update; default storage selection and existing runtime behavior remain unchanged. | -| `@bsv/overlay-topics` | `1.6.10` | `1.8.1` | minor | [API and usage](../packages/overlays/overlay-topics.md) | Existing topic and lookup identifiers remain unchanged. Production UMP overlays must give UMPTopicManager and the UMP lookup service Mongo-backed stores that use the same database, then roll out before updated wallet clients; the no-argument manager is bounded but intended only for isolated single-process use. The reservation and bootstrap-marker collections are additive and initialize from currently indexed UMP UTXOs; take a MongoDB backup before rollout. Legacy ambiguous rows remain visible and can be resolved with WAB pinning rather than deleted. Valid uora-anchor-v3 outputs retain their bytes, signatures, and admission result. Readers now reject nonconforming key, tail, and text shapes. Coordinate reader upgrades across nodes serving tm_uora_dpp and audit any previously indexed nonconforming outputs before rebuilding that topic. No wire prefix, export, or persisted schema changes are required. Mandala custom state adapters must implement isAdminOutpoint against admitted per-asset admin history and retain matching owner rows during admission. Registration must omit assetId or use an empty string. Back up and audit historical admin and ownership records before replay; missing records must be restored from verified admission evidence. Valid wire fields and encodings are unchanged, and no collection migration is needed. Wallets that reveal input linkage must reveal the spender's own child key. No migration is required for this dependency candidate update; default storage selection and existing runtime behavior remain unchanged. | +| `@bsv/overlay` | `2.2.1` | `2.5.0` | minor | [API and usage](../packages/overlays/overlay.md) | Existing Engine and TopicManager implementations remain valid. Lookup results default to 1,000 formulas; pass -1 only when an equivalent deployment bound exists. Topic managers whose validation creates provisional external state should implement abortAdmissibleOutputs, while read-only managers require no change. Distributors must retain THIRD_PARTY_NOTICES.md and LICENSES/ with the package. No migration is required for the additive storage capability. Current Engine/Knex behavior is unchanged; no production adapter provides this atomic receipt yet. MongoDB is an optional peer: applications that import a Mongo storage deep entry point must install mongodb ^7.5.0 and explicitly operate the replica-set foundation. No Engine activation, default-adapter change, SQL-to-Mongo migration, or mixed-writer database topology is included. Existing Engine, TopicManager and header providers remain valid. BASM reconciliation requires a canonical header resolver; blockTransactionCount is optional and must come from the trusted provider for the same block hash. Header-only proof position assurance is reported explicitly. A remote admitted list is bound to its compound Merkle path before it is treated as divergence or repair evidence. Proof inclusion checks the canonical root at the claimed height and does not apply coinbase 100-block spendability; historical submit still runs standard SPV. BASM admission therefore submits in the historical-tx-no-spv mode, so topic managers that branch on the submission mode observe historical-tx-no-spv rather than historical-tx for BASM-admitted transactions. Untrusted bootstrap prefixes and over-limit proof/raw requests fail closed pending durable recovery work; automatic sync remains off and no schema migration is included. | +| `@bsv/overlay-discovery-services` | `2.1.1` | `2.2.3` | minor | [API and usage](../packages/overlays/overlay-discovery-services.md) | Existing mainnet and testnet advertisers are unchanged. TTN operators pass chain ttn and provision the staging storage and overlay endpoints before advertising. Distributors must retain THIRD_PARTY_NOTICES.md and LICENSES/ with the package. No migration is required for this dependency candidate update; default storage selection and existing runtime behavior remain unchanged. No migration is required for this dependency candidate update. | +| `@bsv/overlay-express` | `2.5.0` | `2.7.0` | minor | [API and usage](../packages/overlays/overlay-express.md) | Existing mainnet and testnet servers are unchanged. TTN servers call configureNetwork('ttn'), configureArcade with the TTN endpoint, and configureChaintracks or configureChainTracker before engine initialization. Distributors must retain THIRD_PARTY_NOTICES.md and LICENSES/ with the package. No migration is required for this dependency candidate update; default storage selection and existing runtime behavior remain unchanged. Valid legacy JSON shapes, numeric-string heights, public CORS, topic-free raw transactions, configured limits and synchronization defaults remain supported. No storage migration is required. Malformed/coercive inputs are rejected and missing BASM engine/storage capabilities return HTTP 400 with code BASM_UNSUPPORTED. Existing header providers remain valid without a blockTransactionCount field. | +| `@bsv/overlay-topics` | `1.6.10` | `1.8.2` | minor | [API and usage](../packages/overlays/overlay-topics.md) | Existing topic and lookup identifiers remain unchanged. Production UMP overlays must give UMPTopicManager and the UMP lookup service Mongo-backed stores that use the same database, then roll out before updated wallet clients; the no-argument manager is bounded but intended only for isolated single-process use. The reservation and bootstrap-marker collections are additive and initialize from currently indexed UMP UTXOs; take a MongoDB backup before rollout. Legacy ambiguous rows remain visible and can be resolved with WAB pinning rather than deleted. Valid uora-anchor-v3 outputs retain their bytes, signatures, and admission result. Readers now reject nonconforming key, tail, and text shapes. Coordinate reader upgrades across nodes serving tm_uora_dpp and audit any previously indexed nonconforming outputs before rebuilding that topic. No wire prefix, export, or persisted schema changes are required. Mandala custom state adapters must implement isAdminOutpoint against admitted per-asset admin history and retain matching owner rows during admission. Registration must omit assetId or use an empty string. Back up and audit historical admin and ownership records before replay; missing records must be restored from verified admission evidence. Valid wire fields and encodings are unchanged, and no collection migration is needed. Wallets that reveal input linkage must reveal the spender's own child key. No migration is required for this dependency candidate update; default storage selection and existing runtime behavior remain unchanged. No migration is required for this dependency candidate update. | | `@bsv/paymail` | `2.4.2` | `2.4.8` | patch | [API and usage](../packages/messaging/paymail.md) | Existing Paymail client APIs and protocol semantics are retained. Consumers provide one Express 4.18 or 5 runtime and matching type graph; browser bundles continue to exclude the server router implementation. Consumers of the former bundled Money Button or Tokenized specification documents must follow the authoritative links in docs/specs/README.md. Distributors must retain THIRD_PARTY_NOTICES.md and LICENSES/ with the package. | | `@bsv/payment-express-middleware` | `2.1.1` | `2.1.6` | patch | [API and usage](../packages/middleware/payment-express-middleware.md) | No consumer migration is required; legacy x-bsv-payment JSON behavior remains supported, and Express 4 and 5 applications use their own peer-provided Express installation. Distributors must retain THIRD_PARTY_NOTICES.md and LICENSES/ with the package. | -| `@bsv/sdk` | `2.7.0` | `2.7.1` | patch | [API and usage](../packages/sdk/bsv-sdk.md) | No API or wire migration is required. Upgrade to @bsv/sdk 2.7.1 or later. Valid BRC-103 v0.1 peers keep the same message fields, payload bytes, nonces, and signature derivation; authenticated messages whose transport identity differs from the established session now fail closed. | +| `@bsv/sdk` | `2.7.0` | `2.7.1` | patch | [API and usage](../packages/sdk/bsv-sdk.md) | No API or wire migration is required. Upgrade to @bsv/sdk 2.7.1 or later. Valid BRC-103 v0.1 peers keep the same message fields, payload bytes, nonces, and signature derivation; authenticated messages whose transport identity differs from the established session now fail closed. No API or valid BUMP wire migration is required for the offset correction. | | `@bsv/simple` | `0.4.1` | `0.5.3` | minor | [API and usage](../packages/helpers/simple.md) | Existing overlay configurations and number-array behavior are unchanged. TTN consumers select network teratestnet; all consumers should upgrade to @bsv/sdk 2.4.2 or later. The Message Box Client dependency advance requires no helper API migration and preserves the default HTTP and live-socket behavior. Distributors must retain THIRD_PARTY_NOTICES.md and LICENSES/. | | `@bsv/templates` | `1.9.1` | `1.10.1` | minor | [API and usage](../packages/helpers/templates.md) | No existing consumer migration is required; existing template APIs and generated scripts are unchanged. New R1K1Wallet consumers await lock(), retain each private 32-byte salt, and provide a PIV signer that signs the supplied digest directly without hashing it again. Distributors must retain THIRD_PARTY_NOTICES.md and LICENSES/ with the package. | | `@bsv/teranode-listener` | `1.1.1` | `1.1.5` | patch | [API and usage](../packages/network/teranode-listener.md) | No consumer migration is required; listener APIs, topics, and network configuration are unchanged. Distributors must retain THIRD_PARTY_NOTICES.md and LICENSES/ with the package. | @@ -277,8 +277,8 @@ CLI entry points: `{"lch":"./dist/cli.js"}`. - Package documentation: [docs/packages/overlays/overlay.md](../packages/overlays/overlay.md) - Source: [packages/overlays/overlay](https://github.com/bsv-blockchain/ts-stack/tree/main/packages/overlays/overlay) -- Release note: Adds an engine lookup-result cardinality ceiling before transaction/proof hydration and an optional TopicManager abort hook for provisional admission state when strict transaction broadcast fails. Retains the hash-pinned pre-uniformization Open BSV License version 4 grant as a scoped continuity notice. Adds optional v1 admission/storage capability types, deterministic semantic identity and exact integer helpers, recovery fences and cursor-publication predicates, with shared reference conformance fixtures. Adds an opt-in MongoDB foundation for validated scoped schema bootstrap, content-addressed payload publication, transactionally guarded references, and recoverable payload collection. -- Migration: Existing Engine and TopicManager implementations remain valid. Lookup results default to 1,000 formulas; pass -1 only when an equivalent deployment bound exists. Topic managers whose validation creates provisional external state should implement abortAdmissibleOutputs, while read-only managers require no change. Distributors must retain THIRD_PARTY_NOTICES.md and LICENSES/ with the package. No migration is required for the additive storage capability. Current Engine/Knex behavior is unchanged; no production adapter provides this atomic receipt yet. MongoDB is an optional peer: applications that import a Mongo storage deep entry point must install mongodb ^7.5.0 and explicitly operate the replica-set foundation. No Engine activation, default-adapter change, SQL-to-Mongo migration, or mixed-writer database topology is included. +- Release note: Adds an engine lookup-result cardinality ceiling before transaction/proof hydration and an optional TopicManager abort hook for provisional admission state when strict transaction broadcast fails. Retains the hash-pinned pre-uniformization Open BSV License version 4 grant as a scoped continuity notice. Adds optional v1 admission/storage capability types, deterministic semantic identity and exact integer helpers, recovery fences and cursor-publication predicates, with shared reference conformance fixtures. Adds an opt-in MongoDB foundation for validated scoped schema bootstrap, content-addressed payload publication, transactionally guarded references, and recoverable payload collection. Adds bounded BASM peer JSON validation, canonical header/proof/raw binding, optional trusted full-block-count position checks, and classified capability/resource errors. Forward sync pages fit the standard 1,000-anchor HTTP limit. Claimed admitted-list indices are bound to the compound path even when every remote txid is already local, and inclusion uses chain-tracker root/height rather than coinbase maturity. +- Migration: Existing Engine and TopicManager implementations remain valid. Lookup results default to 1,000 formulas; pass -1 only when an equivalent deployment bound exists. Topic managers whose validation creates provisional external state should implement abortAdmissibleOutputs, while read-only managers require no change. Distributors must retain THIRD_PARTY_NOTICES.md and LICENSES/ with the package. No migration is required for the additive storage capability. Current Engine/Knex behavior is unchanged; no production adapter provides this atomic receipt yet. MongoDB is an optional peer: applications that import a Mongo storage deep entry point must install mongodb ^7.5.0 and explicitly operate the replica-set foundation. No Engine activation, default-adapter change, SQL-to-Mongo migration, or mixed-writer database topology is included. Existing Engine, TopicManager and header providers remain valid. BASM reconciliation requires a canonical header resolver; blockTransactionCount is optional and must come from the trusted provider for the same block hash. Header-only proof position assurance is reported explicitly. A remote admitted list is bound to its compound Merkle path before it is treated as divergence or repair evidence. Proof inclusion checks the canonical root at the claimed height and does not apply coinbase 100-block spendability; historical submit still runs standard SPV. BASM admission therefore submits in the historical-tx-no-spv mode, so topic managers that branch on the submission mode observe historical-tx-no-spv rather than historical-tx for BASM-admitted transactions. Untrusted bootstrap prefixes and over-limit proof/raw requests fail closed pending durable recovery work; automatic sync remains off and no schema migration is included. | Public subpath | Runtime target(s) | Declaration target(s) | | ---------------- | -------------------------------------------------------------------------- | -------------------------------------------------------------------------------- | @@ -292,8 +292,8 @@ CLI entry points: `{"lch":"./dist/cli.js"}`. - Package documentation: [docs/packages/overlays/overlay-discovery-services.md](../packages/overlays/overlay-discovery-services.md) - Source: [packages/overlays/overlay-discovery-services](https://github.com/bsv-blockchain/ts-stack/tree/main/packages/overlays/overlay-discovery-services) -- Release note: Extends WalletAdvertiser to TerraTestNet with chain-isolated SHIP/SLAP resolution and TTN wallet services. Retains the hash-pinned pre-uniformization Open BSV License version 4 grant as a scoped continuity notice. Updates the packed workspace dependency candidate for the additive overlay persistence contracts without changing this package runtime behavior. -- Migration: Existing mainnet and testnet advertisers are unchanged. TTN operators pass chain ttn and provision the staging storage and overlay endpoints before advertising. Distributors must retain THIRD_PARTY_NOTICES.md and LICENSES/ with the package. No migration is required for this dependency candidate update; default storage selection and existing runtime behavior remain unchanged. +- Release note: Extends WalletAdvertiser to TerraTestNet with chain-isolated SHIP/SLAP resolution and TTN wallet services. Retains the hash-pinned pre-uniformization Open BSV License version 4 grant as a scoped continuity notice. Updates the packed workspace dependency candidate for the additive overlay persistence contracts without changing this package runtime behavior. Advances the packed overlay dependency candidate for BASM validation hardening without changing this package runtime behavior. +- Migration: Existing mainnet and testnet advertisers are unchanged. TTN operators pass chain ttn and provision the staging storage and overlay endpoints before advertising. Distributors must retain THIRD_PARTY_NOTICES.md and LICENSES/ with the package. No migration is required for this dependency candidate update; default storage selection and existing runtime behavior remain unchanged. No migration is required for this dependency candidate update. | Public subpath | Runtime target(s) | Declaration target(s) | | -------------- | ---------------------------------------------- | ---------------------------------------------------- | @@ -304,8 +304,8 @@ CLI entry points: `{"lch":"./dist/cli.js"}`. - Package documentation: [docs/packages/overlays/overlay-express.md](../packages/overlays/overlay-express.md) - Source: [packages/overlays/overlay-express](https://github.com/bsv-blockchain/ts-stack/tree/main/packages/overlays/overlay-express) -- Release note: Adds a TerraTestNet runtime mode with TTN discovery roots, wallet advertising, and explicit ChainTracks requirements while retaining Arcade-first propagation. Retains the hash-pinned pre-uniformization Open BSV License version 4 grant as a scoped continuity notice. Updates the packed workspace dependency candidate for the additive overlay persistence contracts without changing this package runtime behavior. -- Migration: Existing mainnet and testnet servers are unchanged. TTN servers call configureNetwork('ttn'), configureArcade with the TTN endpoint, and configureChaintracks or configureChainTracker before engine initialization. Distributors must retain THIRD_PARTY_NOTICES.md and LICENSES/ with the package. No migration is required for this dependency candidate update; default storage selection and existing runtime behavior remain unchanged. +- Release note: Adds a TerraTestNet runtime mode with TTN discovery roots, wallet advertising, and explicit ChainTracks requirements while retaining Arcade-first propagation. Retains the hash-pinned pre-uniformization Open BSV License version 4 grant as a scoped continuity notice. Updates the packed workspace dependency candidate for the additive overlay persistence contracts without changing this package runtime behavior. Validates BASM heights, hashes and txid sets; exposes unsupported capabilities explicitly and accepts an optional trusted full-block count from injected canonical header resolvers. +- Migration: Existing mainnet and testnet servers are unchanged. TTN servers call configureNetwork('ttn'), configureArcade with the TTN endpoint, and configureChaintracks or configureChainTracker before engine initialization. Distributors must retain THIRD_PARTY_NOTICES.md and LICENSES/ with the package. No migration is required for this dependency candidate update; default storage selection and existing runtime behavior remain unchanged. Valid legacy JSON shapes, numeric-string heights, public CORS, topic-free raw transactions, configured limits and synchronization defaults remain supported. No storage migration is required. Malformed/coercive inputs are rejected and missing BASM engine/storage capabilities return HTTP 400 with code BASM_UNSUPPORTED. Existing header providers remain valid without a blockTransactionCount field. | Public subpath | Runtime target(s) | Declaration target(s) | | -------------- | ---------------------------------------------- | ---------------------------------------------------- | @@ -316,8 +316,8 @@ CLI entry points: `{"lch":"./dist/cli.js"}`. - Package documentation: [docs/packages/overlays/overlay-topics.md](../packages/overlays/overlay-topics.md) - Source: [packages/overlays/topics](https://github.com/bsv-blockchain/ts-stack/tree/main/packages/overlays/topics) -- Release note: Adds persistent first-writer reservations for UMP presentation and recovery hashes, aborts provisional claims after strict broadcast failure, keeps confirmed owners protected until successor indexing, retries transient initialization, marks one-time legacy bootstrap, and returns the newest bounded legacy candidates for verified lineage or an operator pin. It also retains the collection-index resilience and opt-in repair controls from the prior candidate. Standardizes first-party author metadata on the current BSV Association name. Aligns the UORA v3 reader with the versioned format, including compressed locking keys, exact drop tails, and printable UTF-8 fields. Version 1.8.0 requires verified per-asset admin history and matching stored token ownership for Mandala admission, assigns registrations to their own genesis outpoint, rejects ambiguous linkage indices, and normalizes in-memory sanctions key casing. Retains the complete token-output linkage verification from 1.7.3. Updates the packed workspace dependency candidate for the additive overlay persistence contracts without changing this package runtime behavior. -- Migration: Existing topic and lookup identifiers remain unchanged. Production UMP overlays must give UMPTopicManager and the UMP lookup service Mongo-backed stores that use the same database, then roll out before updated wallet clients; the no-argument manager is bounded but intended only for isolated single-process use. The reservation and bootstrap-marker collections are additive and initialize from currently indexed UMP UTXOs; take a MongoDB backup before rollout. Legacy ambiguous rows remain visible and can be resolved with WAB pinning rather than deleted. Valid uora-anchor-v3 outputs retain their bytes, signatures, and admission result. Readers now reject nonconforming key, tail, and text shapes. Coordinate reader upgrades across nodes serving tm_uora_dpp and audit any previously indexed nonconforming outputs before rebuilding that topic. No wire prefix, export, or persisted schema changes are required. Mandala custom state adapters must implement isAdminOutpoint against admitted per-asset admin history and retain matching owner rows during admission. Registration must omit assetId or use an empty string. Back up and audit historical admin and ownership records before replay; missing records must be restored from verified admission evidence. Valid wire fields and encodings are unchanged, and no collection migration is needed. Wallets that reveal input linkage must reveal the spender's own child key. No migration is required for this dependency candidate update; default storage selection and existing runtime behavior remain unchanged. +- Release note: Adds persistent first-writer reservations for UMP presentation and recovery hashes, aborts provisional claims after strict broadcast failure, keeps confirmed owners protected until successor indexing, retries transient initialization, marks one-time legacy bootstrap, and returns the newest bounded legacy candidates for verified lineage or an operator pin. It also retains the collection-index resilience and opt-in repair controls from the prior candidate. Standardizes first-party author metadata on the current BSV Association name. Aligns the UORA v3 reader with the versioned format, including compressed locking keys, exact drop tails, and printable UTF-8 fields. Version 1.8.0 requires verified per-asset admin history and matching stored token ownership for Mandala admission, assigns registrations to their own genesis outpoint, rejects ambiguous linkage indices, and normalizes in-memory sanctions key casing. Retains the complete token-output linkage verification from 1.7.3. Updates the packed workspace dependency candidate for the additive overlay persistence contracts without changing this package runtime behavior. Advances the packed overlay dependency candidate for BASM validation hardening without changing this package runtime behavior. +- Migration: Existing topic and lookup identifiers remain unchanged. Production UMP overlays must give UMPTopicManager and the UMP lookup service Mongo-backed stores that use the same database, then roll out before updated wallet clients; the no-argument manager is bounded but intended only for isolated single-process use. The reservation and bootstrap-marker collections are additive and initialize from currently indexed UMP UTXOs; take a MongoDB backup before rollout. Legacy ambiguous rows remain visible and can be resolved with WAB pinning rather than deleted. Valid uora-anchor-v3 outputs retain their bytes, signatures, and admission result. Readers now reject nonconforming key, tail, and text shapes. Coordinate reader upgrades across nodes serving tm_uora_dpp and audit any previously indexed nonconforming outputs before rebuilding that topic. No wire prefix, export, or persisted schema changes are required. Mandala custom state adapters must implement isAdminOutpoint against admitted per-asset admin history and retain matching owner rows during admission. Registration must omit assetId or use an empty string. Back up and audit historical admin and ownership records before replay; missing records must be restored from verified admission evidence. Valid wire fields and encodings are unchanged, and no collection migration is needed. Wallets that reveal input linkage must reveal the spender's own child key. No migration is required for this dependency candidate update; default storage selection and existing runtime behavior remain unchanged. No migration is required for this dependency candidate update. | Public subpath | Runtime target(s) | Declaration target(s) | | -------------- | -------------------------------------- | --------------------- | @@ -359,8 +359,8 @@ CLI entry points: `{"lch":"./dist/cli.js"}`. - Package documentation: [docs/packages/sdk/bsv-sdk.md](../packages/sdk/bsv-sdk.md) - Source: [packages/sdk](https://github.com/bsv-blockchain/ts-stack/tree/main/packages/sdk) -- Release note: Binds authenticated general messages, certificate requests, and certificate responses to the identity stored in the nonce-selected session. Mismatched transport identity metadata is rejected, and application callbacks receive only the identity used for signature verification. -- Migration: No API or wire migration is required. Upgrade to @bsv/sdk 2.7.1 or later. Valid BRC-103 v0.1 peers keep the same message fields, payload bytes, nonces, and signature derivation; authenticated messages whose transport identity differs from the established session now fail closed. +- Release note: Binds authenticated general messages, certificate requests, and certificate responses to the identity stored in the nonce-selected session. Mismatched transport identity metadata is rejected, and application callbacks receive only the identity used for signature verification. The current unpublished candidate also corrects Merkle-path arithmetic for the full safe-integer offset domain, including positions above 32 bits, and rejects nonrepresentable offsets. +- Migration: No API or wire migration is required. Upgrade to @bsv/sdk 2.7.1 or later. Valid BRC-103 v0.1 peers keep the same message fields, payload bytes, nonces, and signature derivation; authenticated messages whose transport identity differs from the established session now fail closed. No API or valid BUMP wire migration is required for the offset correction. | Public subpath | Runtime target(s) | Declaration target(s) | | ---------------------------------- | ---------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | diff --git a/docs/reference/stack-facts.md b/docs/reference/stack-facts.md index 1e0dbf86d..7f8878d17 100644 --- a/docs/reference/stack-facts.md +++ b/docs/reference/stack-facts.md @@ -58,10 +58,10 @@ authorized release action. | network | `@bsv/chirp` | `0.1.1` | browser-library | browser-bundler, browser-esm, cli, node-esm | browser, node | `>=22` | [packages/network/chirp](https://github.com/bsv-blockchain/ts-stack/tree/main/packages/network/chirp) | | network | `@bsv/teranode-listener` | `1.1.5` | node-library | node-esm | node | `>=22` | [packages/network/ts-p2p](https://github.com/bsv-blockchain/ts-stack/tree/main/packages/network/ts-p2p) | | overlays | `@bsv/gasp` | `1.3.6` | browser-library | browser-bundler, browser-esm, node-cjs, node-esm | browser, node | `>=22` | [packages/overlays/gasp-core](https://github.com/bsv-blockchain/ts-stack/tree/main/packages/overlays/gasp-core) | -| overlays | `@bsv/overlay` | `2.4.0` | node-library | node-cjs, node-esm | node | `>=22` | [packages/overlays/overlay](https://github.com/bsv-blockchain/ts-stack/tree/main/packages/overlays/overlay) | -| overlays | `@bsv/overlay-discovery-services` | `2.2.2` | node-library | node-cjs, node-esm | node | `>=22` | [packages/overlays/overlay-discovery-services](https://github.com/bsv-blockchain/ts-stack/tree/main/packages/overlays/overlay-discovery-services) | -| overlays | `@bsv/overlay-express` | `2.6.2` | node-library | node-cjs, node-esm | node | `>=22` | [packages/overlays/overlay-express](https://github.com/bsv-blockchain/ts-stack/tree/main/packages/overlays/overlay-express) | -| overlays | `@bsv/overlay-topics` | `1.8.1` | node-library | node-esm | node | `>=22` | [packages/overlays/topics](https://github.com/bsv-blockchain/ts-stack/tree/main/packages/overlays/topics) | +| overlays | `@bsv/overlay` | `2.5.0` | node-library | node-cjs, node-esm | node | `>=22` | [packages/overlays/overlay](https://github.com/bsv-blockchain/ts-stack/tree/main/packages/overlays/overlay) | +| overlays | `@bsv/overlay-discovery-services` | `2.2.3` | node-library | node-cjs, node-esm | node | `>=22` | [packages/overlays/overlay-discovery-services](https://github.com/bsv-blockchain/ts-stack/tree/main/packages/overlays/overlay-discovery-services) | +| overlays | `@bsv/overlay-express` | `2.7.0` | node-library | node-cjs, node-esm | node | `>=22` | [packages/overlays/overlay-express](https://github.com/bsv-blockchain/ts-stack/tree/main/packages/overlays/overlay-express) | +| overlays | `@bsv/overlay-topics` | `1.8.2` | node-library | node-esm | node | `>=22` | [packages/overlays/topics](https://github.com/bsv-blockchain/ts-stack/tree/main/packages/overlays/topics) | | sdk | `@bsv/sdk` | `2.7.1` | browser-library | browser-bundler, browser-esm, node-cjs, node-esm, umd-global | browser, node, umd | `>=22` | [packages/sdk](https://github.com/bsv-blockchain/ts-stack/tree/main/packages/sdk) | | sdk | `@bsv/verifast` | `0.3.5` | wasm-library | browser-bundler, browser-esm, node-cjs, node-esm, umd-global, wasm-worker | browser, node, umd, wasm, worker | `>=22` | [packages/verifast](https://github.com/bsv-blockchain/ts-stack/tree/main/packages/verifast) | | wallet | `@bsv/btms` | `1.2.2` | node-library | node-cjs, node-esm | node | `>=22` | [packages/wallet/btms](https://github.com/bsv-blockchain/ts-stack/tree/main/packages/wallet/btms) | diff --git a/governance/package-release-notes.json b/governance/package-release-notes.json index f61810131..e9990915c 100644 --- a/governance/package-release-notes.json +++ b/governance/package-release-notes.json @@ -126,29 +126,29 @@ "name": "@bsv/overlay", "publishedVersion": "2.2.1", "releaseType": "minor", - "summary": "Adds an engine lookup-result cardinality ceiling before transaction/proof hydration and an optional TopicManager abort hook for provisional admission state when strict transaction broadcast fails. Retains the hash-pinned pre-uniformization Open BSV License version 4 grant as a scoped continuity notice. Adds optional v1 admission/storage capability types, deterministic semantic identity and exact integer helpers, recovery fences and cursor-publication predicates, with shared reference conformance fixtures. Adds an opt-in MongoDB foundation for validated scoped schema bootstrap, content-addressed payload publication, transactionally guarded references, and recoverable payload collection.", - "migration": "Existing Engine and TopicManager implementations remain valid. Lookup results default to 1,000 formulas; pass -1 only when an equivalent deployment bound exists. Topic managers whose validation creates provisional external state should implement abortAdmissibleOutputs, while read-only managers require no change. Distributors must retain THIRD_PARTY_NOTICES.md and LICENSES/ with the package. No migration is required for the additive storage capability. Current Engine/Knex behavior is unchanged; no production adapter provides this atomic receipt yet. MongoDB is an optional peer: applications that import a Mongo storage deep entry point must install mongodb ^7.5.0 and explicitly operate the replica-set foundation. No Engine activation, default-adapter change, SQL-to-Mongo migration, or mixed-writer database topology is included." + "summary": "Adds an engine lookup-result cardinality ceiling before transaction/proof hydration and an optional TopicManager abort hook for provisional admission state when strict transaction broadcast fails. Retains the hash-pinned pre-uniformization Open BSV License version 4 grant as a scoped continuity notice. Adds optional v1 admission/storage capability types, deterministic semantic identity and exact integer helpers, recovery fences and cursor-publication predicates, with shared reference conformance fixtures. Adds an opt-in MongoDB foundation for validated scoped schema bootstrap, content-addressed payload publication, transactionally guarded references, and recoverable payload collection. Adds bounded BASM peer JSON validation, canonical header/proof/raw binding, optional trusted full-block-count position checks, and classified capability/resource errors. Forward sync pages fit the standard 1,000-anchor HTTP limit. Claimed admitted-list indices are bound to the compound path even when every remote txid is already local, and inclusion uses chain-tracker root/height rather than coinbase maturity.", + "migration": "Existing Engine and TopicManager implementations remain valid. Lookup results default to 1,000 formulas; pass -1 only when an equivalent deployment bound exists. Topic managers whose validation creates provisional external state should implement abortAdmissibleOutputs, while read-only managers require no change. Distributors must retain THIRD_PARTY_NOTICES.md and LICENSES/ with the package. No migration is required for the additive storage capability. Current Engine/Knex behavior is unchanged; no production adapter provides this atomic receipt yet. MongoDB is an optional peer: applications that import a Mongo storage deep entry point must install mongodb ^7.5.0 and explicitly operate the replica-set foundation. No Engine activation, default-adapter change, SQL-to-Mongo migration, or mixed-writer database topology is included. Existing Engine, TopicManager and header providers remain valid. BASM reconciliation requires a canonical header resolver; blockTransactionCount is optional and must come from the trusted provider for the same block hash. Header-only proof position assurance is reported explicitly. A remote admitted list is bound to its compound Merkle path before it is treated as divergence or repair evidence. Proof inclusion checks the canonical root at the claimed height and does not apply coinbase 100-block spendability; historical submit still runs standard SPV. BASM admission therefore submits in the historical-tx-no-spv mode, so topic managers that branch on the submission mode observe historical-tx-no-spv rather than historical-tx for BASM-admitted transactions. Untrusted bootstrap prefixes and over-limit proof/raw requests fail closed pending durable recovery work; automatic sync remains off and no schema migration is included." }, { "name": "@bsv/overlay-discovery-services", "publishedVersion": "2.1.1", "releaseType": "minor", - "summary": "Extends WalletAdvertiser to TerraTestNet with chain-isolated SHIP/SLAP resolution and TTN wallet services. Retains the hash-pinned pre-uniformization Open BSV License version 4 grant as a scoped continuity notice. Updates the packed workspace dependency candidate for the additive overlay persistence contracts without changing this package runtime behavior.", - "migration": "Existing mainnet and testnet advertisers are unchanged. TTN operators pass chain ttn and provision the staging storage and overlay endpoints before advertising. Distributors must retain THIRD_PARTY_NOTICES.md and LICENSES/ with the package. No migration is required for this dependency candidate update; default storage selection and existing runtime behavior remain unchanged." + "summary": "Extends WalletAdvertiser to TerraTestNet with chain-isolated SHIP/SLAP resolution and TTN wallet services. Retains the hash-pinned pre-uniformization Open BSV License version 4 grant as a scoped continuity notice. Updates the packed workspace dependency candidate for the additive overlay persistence contracts without changing this package runtime behavior. Advances the packed overlay dependency candidate for BASM validation hardening without changing this package runtime behavior.", + "migration": "Existing mainnet and testnet advertisers are unchanged. TTN operators pass chain ttn and provision the staging storage and overlay endpoints before advertising. Distributors must retain THIRD_PARTY_NOTICES.md and LICENSES/ with the package. No migration is required for this dependency candidate update; default storage selection and existing runtime behavior remain unchanged. No migration is required for this dependency candidate update." }, { "name": "@bsv/overlay-express", "publishedVersion": "2.5.0", "releaseType": "minor", - "summary": "Adds a TerraTestNet runtime mode with TTN discovery roots, wallet advertising, and explicit ChainTracks requirements while retaining Arcade-first propagation. Retains the hash-pinned pre-uniformization Open BSV License version 4 grant as a scoped continuity notice. Updates the packed workspace dependency candidate for the additive overlay persistence contracts without changing this package runtime behavior.", - "migration": "Existing mainnet and testnet servers are unchanged. TTN servers call configureNetwork('ttn'), configureArcade with the TTN endpoint, and configureChaintracks or configureChainTracker before engine initialization. Distributors must retain THIRD_PARTY_NOTICES.md and LICENSES/ with the package. No migration is required for this dependency candidate update; default storage selection and existing runtime behavior remain unchanged." + "summary": "Adds a TerraTestNet runtime mode with TTN discovery roots, wallet advertising, and explicit ChainTracks requirements while retaining Arcade-first propagation. Retains the hash-pinned pre-uniformization Open BSV License version 4 grant as a scoped continuity notice. Updates the packed workspace dependency candidate for the additive overlay persistence contracts without changing this package runtime behavior. Validates BASM heights, hashes and txid sets; exposes unsupported capabilities explicitly and accepts an optional trusted full-block count from injected canonical header resolvers.", + "migration": "Existing mainnet and testnet servers are unchanged. TTN servers call configureNetwork('ttn'), configureArcade with the TTN endpoint, and configureChaintracks or configureChainTracker before engine initialization. Distributors must retain THIRD_PARTY_NOTICES.md and LICENSES/ with the package. No migration is required for this dependency candidate update; default storage selection and existing runtime behavior remain unchanged. Valid legacy JSON shapes, numeric-string heights, public CORS, topic-free raw transactions, configured limits and synchronization defaults remain supported. No storage migration is required. Malformed/coercive inputs are rejected and missing BASM engine/storage capabilities return HTTP 400 with code BASM_UNSUPPORTED. Existing header providers remain valid without a blockTransactionCount field." }, { "name": "@bsv/overlay-topics", "publishedVersion": "1.6.10", "releaseType": "minor", - "summary": "Adds persistent first-writer reservations for UMP presentation and recovery hashes, aborts provisional claims after strict broadcast failure, keeps confirmed owners protected until successor indexing, retries transient initialization, marks one-time legacy bootstrap, and returns the newest bounded legacy candidates for verified lineage or an operator pin. It also retains the collection-index resilience and opt-in repair controls from the prior candidate. Standardizes first-party author metadata on the current BSV Association name. Aligns the UORA v3 reader with the versioned format, including compressed locking keys, exact drop tails, and printable UTF-8 fields. Version 1.8.0 requires verified per-asset admin history and matching stored token ownership for Mandala admission, assigns registrations to their own genesis outpoint, rejects ambiguous linkage indices, and normalizes in-memory sanctions key casing. Retains the complete token-output linkage verification from 1.7.3. Updates the packed workspace dependency candidate for the additive overlay persistence contracts without changing this package runtime behavior.", - "migration": "Existing topic and lookup identifiers remain unchanged. Production UMP overlays must give UMPTopicManager and the UMP lookup service Mongo-backed stores that use the same database, then roll out before updated wallet clients; the no-argument manager is bounded but intended only for isolated single-process use. The reservation and bootstrap-marker collections are additive and initialize from currently indexed UMP UTXOs; take a MongoDB backup before rollout. Legacy ambiguous rows remain visible and can be resolved with WAB pinning rather than deleted. Valid uora-anchor-v3 outputs retain their bytes, signatures, and admission result. Readers now reject nonconforming key, tail, and text shapes. Coordinate reader upgrades across nodes serving tm_uora_dpp and audit any previously indexed nonconforming outputs before rebuilding that topic. No wire prefix, export, or persisted schema changes are required. Mandala custom state adapters must implement isAdminOutpoint against admitted per-asset admin history and retain matching owner rows during admission. Registration must omit assetId or use an empty string. Back up and audit historical admin and ownership records before replay; missing records must be restored from verified admission evidence. Valid wire fields and encodings are unchanged, and no collection migration is needed. Wallets that reveal input linkage must reveal the spender's own child key. No migration is required for this dependency candidate update; default storage selection and existing runtime behavior remain unchanged." + "summary": "Adds persistent first-writer reservations for UMP presentation and recovery hashes, aborts provisional claims after strict broadcast failure, keeps confirmed owners protected until successor indexing, retries transient initialization, marks one-time legacy bootstrap, and returns the newest bounded legacy candidates for verified lineage or an operator pin. It also retains the collection-index resilience and opt-in repair controls from the prior candidate. Standardizes first-party author metadata on the current BSV Association name. Aligns the UORA v3 reader with the versioned format, including compressed locking keys, exact drop tails, and printable UTF-8 fields. Version 1.8.0 requires verified per-asset admin history and matching stored token ownership for Mandala admission, assigns registrations to their own genesis outpoint, rejects ambiguous linkage indices, and normalizes in-memory sanctions key casing. Retains the complete token-output linkage verification from 1.7.3. Updates the packed workspace dependency candidate for the additive overlay persistence contracts without changing this package runtime behavior. Advances the packed overlay dependency candidate for BASM validation hardening without changing this package runtime behavior.", + "migration": "Existing topic and lookup identifiers remain unchanged. Production UMP overlays must give UMPTopicManager and the UMP lookup service Mongo-backed stores that use the same database, then roll out before updated wallet clients; the no-argument manager is bounded but intended only for isolated single-process use. The reservation and bootstrap-marker collections are additive and initialize from currently indexed UMP UTXOs; take a MongoDB backup before rollout. Legacy ambiguous rows remain visible and can be resolved with WAB pinning rather than deleted. Valid uora-anchor-v3 outputs retain their bytes, signatures, and admission result. Readers now reject nonconforming key, tail, and text shapes. Coordinate reader upgrades across nodes serving tm_uora_dpp and audit any previously indexed nonconforming outputs before rebuilding that topic. No wire prefix, export, or persisted schema changes are required. Mandala custom state adapters must implement isAdminOutpoint against admitted per-asset admin history and retain matching owner rows during admission. Registration must omit assetId or use an empty string. Back up and audit historical admin and ownership records before replay; missing records must be restored from verified admission evidence. Valid wire fields and encodings are unchanged, and no collection migration is needed. Wallets that reveal input linkage must reveal the spender's own child key. No migration is required for this dependency candidate update; default storage selection and existing runtime behavior remain unchanged. No migration is required for this dependency candidate update." }, { "name": "@bsv/paymail", @@ -168,8 +168,8 @@ "name": "@bsv/sdk", "publishedVersion": "2.7.0", "releaseType": "patch", - "summary": "Binds authenticated general messages, certificate requests, and certificate responses to the identity stored in the nonce-selected session. Mismatched transport identity metadata is rejected, and application callbacks receive only the identity used for signature verification.", - "migration": "No API or wire migration is required. Upgrade to @bsv/sdk 2.7.1 or later. Valid BRC-103 v0.1 peers keep the same message fields, payload bytes, nonces, and signature derivation; authenticated messages whose transport identity differs from the established session now fail closed." + "summary": "Binds authenticated general messages, certificate requests, and certificate responses to the identity stored in the nonce-selected session. Mismatched transport identity metadata is rejected, and application callbacks receive only the identity used for signature verification. The current unpublished candidate also corrects Merkle-path arithmetic for the full safe-integer offset domain, including positions above 32 bits, and rejects nonrepresentable offsets.", + "migration": "No API or wire migration is required. Upgrade to @bsv/sdk 2.7.1 or later. Valid BRC-103 v0.1 peers keep the same message fields, payload bytes, nonces, and signature derivation; authenticated messages whose transport identity differs from the established session now fail closed. No API or valid BUMP wire migration is required for the offset correction." }, { "name": "@bsv/simple", diff --git a/governance/repository-health/baselines.json b/governance/repository-health/baselines.json index d4aa601dc..b439a1bed 100644 --- a/governance/repository-health/baselines.json +++ b/governance/repository-health/baselines.json @@ -318,10 +318,10 @@ "@bsv/payment-express-middleware": "2.1.6", "@bsv/teranode-listener": "1.1.5", "@bsv/gasp": "1.3.6", - "@bsv/overlay": "2.4.0", - "@bsv/overlay-discovery-services": "2.2.2", - "@bsv/overlay-express": "2.6.2", - "@bsv/overlay-topics": "1.8.1", + "@bsv/overlay": "2.5.0", + "@bsv/overlay-discovery-services": "2.2.3", + "@bsv/overlay-express": "2.7.0", + "@bsv/overlay-topics": "1.8.2", "@bsv/sdk": "2.7.1", "@bsv/verifast": "0.3.5", "@bsv/btms": "1.2.2", diff --git a/packages/overlays/overlay-discovery-services/CHANGELOG.md b/packages/overlays/overlay-discovery-services/CHANGELOG.md index c6dd37b12..d0a03723f 100644 --- a/packages/overlays/overlay-discovery-services/CHANGELOG.md +++ b/packages/overlays/overlay-discovery-services/CHANGELOG.md @@ -10,6 +10,8 @@ All notable changes to this project will be documented in this file. The format ## [Unreleased] - Updates the packed workspace dependency candidate for the additive overlay persistence contract. Runtime behavior and defaults are unchanged; no consumer migration is required. +- Advances the packed overlay dependency candidate for BASM validation hardening. + Package runtime behavior is unchanged; no consumer migration is required. ### Added diff --git a/packages/overlays/overlay-discovery-services/package.json b/packages/overlays/overlay-discovery-services/package.json index 610bb5080..fa870977a 100644 --- a/packages/overlays/overlay-discovery-services/package.json +++ b/packages/overlays/overlay-discovery-services/package.json @@ -1,6 +1,6 @@ { "name": "@bsv/overlay-discovery-services", - "version": "2.2.2", + "version": "2.2.3", "sideEffects": false, "engines": { "node": ">=22" diff --git a/packages/overlays/overlay-express/CHANGELOG.md b/packages/overlays/overlay-express/CHANGELOG.md index ed0259671..30a2fb2d7 100644 --- a/packages/overlays/overlay-express/CHANGELOG.md +++ b/packages/overlays/overlay-express/CHANGELOG.md @@ -12,6 +12,11 @@ All notable changes to this project will be documented in this file. The format ## [Unreleased] - Updates the packed workspace dependency candidate for the additive overlay persistence contract. Runtime behavior and defaults are unchanged; no consumer migration is required. +- Validated BASM heights, hashes and txid sets while retaining numeric-string + heights, public CORS, topic-free raw transactions, configured limits and sync + defaults. Added explicit unsupported-capability codes and optional trusted + full-block transaction counts on injected header resolvers. No deployment or + storage migration is required. ### Added diff --git a/packages/overlays/overlay-express/README.md b/packages/overlays/overlay-express/README.md index ffa615e9d..b75f11aed 100644 --- a/packages/overlays/overlay-express/README.md +++ b/packages/overlays/overlay-express/README.md @@ -36,6 +36,20 @@ import OverlayExpress, { OverlayMonitor } from '@bsv/overlay-express' const { default: OverlayExpress, OverlayMonitor } = require('@bsv/overlay-express') ``` +The five public BASM JSON POST routes validate nonnegative safe-integer heights +(including existing numeric strings), 32-byte hexadecimal hashes/txids, and +request count limits. Empty raw-transaction requests remain valid; compound +proof requests require txids. The raw-transaction route does not require +`x-bsv-topic`. Missing engine/storage BASM capabilities retain HTTP 400 with +`{ status: 'error', message, code: 'BASM_UNSUPPORTED' }`. CORS, access rules, +configured limits, and automatic synchronization defaults are unchanged. + +An injected topic-anchor header resolver may additionally return +`blockTransactionCount` obtained independently for the same canonical block +hash. Existing Chaintracks/provider adapters remain header-only; they do not +claim this stronger position evidence. See the core engine's +[BASM validation and recovery limits](../overlay/README.md#basm-peer-validation-and-current-recovery-limits). + ## Example Usage Here's a quick example: diff --git a/packages/overlays/overlay-express/package.json b/packages/overlays/overlay-express/package.json index a8d093072..27c767b98 100644 --- a/packages/overlays/overlay-express/package.json +++ b/packages/overlays/overlay-express/package.json @@ -1,6 +1,6 @@ { "name": "@bsv/overlay-express", - "version": "2.6.2", + "version": "2.7.0", "sideEffects": false, "engines": { "node": ">=22" diff --git a/packages/overlays/overlay-express/src/OverlayExpress.ts b/packages/overlays/overlay-express/src/OverlayExpress.ts index 888d78d31..24ab06ae9 100644 --- a/packages/overlays/overlay-express/src/OverlayExpress.ts +++ b/packages/overlays/overlay-express/src/OverlayExpress.ts @@ -206,6 +206,8 @@ export type TopicAnchorHeaderResolver = (blockHeight: number) => Promise< blockHeight: number blockHash: string merkleRoot?: string + /** Independent full block count bound to blockHash; never an overlay subset count. */ + blockTransactionCount?: number } | undefined > @@ -251,6 +253,15 @@ class PublicRequestError extends Error { } } +class UnsupportedBasmCapabilityError extends PublicRequestError { + readonly code = 'BASM_UNSUPPORTED' + + constructor() { + super('BASM capability is not supported by this Overlay engine') + this.name = 'UnsupportedBasmCapabilityError' + } +} + function publicErrorMessage( error: unknown, fallback: string = 'Request could not be processed' @@ -2249,6 +2260,13 @@ export default class OverlayExpress { return res.status(200).json(await handler(req)) } catch (error) { console.error(chalk.red(`Error in ${path}:`), error) + if (error instanceof Error && 'code' in error && error.code === 'BASM_UNSUPPORTED') { + return res.status(400).json({ + status: 'error', + message: 'BASM capability is not supported by this Overlay engine', + code: error.code + }) + } return res.status(400).json({ status: 'error', message: publicErrorMessage(error) @@ -2261,26 +2279,73 @@ export default class OverlayExpress { ) } - const requireTxids = (value: unknown): string[] => { - if (!Array.isArray(value) || !value.every(txid => typeof txid === 'string')) { - throw new PublicRequestError('txids must be an array of strings') + type BasmCapability = + | 'provideTopicAnchorTip' + | 'provideTopicAnchorRange' + | 'provideAdmittedList' + | 'provideCompoundMerklePath' + | 'provideRawTransactions' + const requireBasmCapability = (capability: BasmCapability): void => { + if (typeof (engine as Partial)[capability] !== 'function') { + throw new UnsupportedBasmCapabilityError() + } + } + + const requireBasmHeight = (value: unknown, field: string): number => { + if ( + (typeof value !== 'number' && typeof value !== 'string') || + (typeof value === 'string' && value.trim().length === 0) + ) { + throw new PublicRequestError(`${field} must be a nonnegative safe integer`) + } + const height = Number(value) + if (!Number.isSafeInteger(height) || height < 0) { + throw new PublicRequestError(`${field} must be a nonnegative safe integer`) + } + return height + } + + const requireBlockHash = (value: unknown): string | undefined => { + if (value === undefined) return undefined + if (typeof value !== 'string' || !/^[0-9a-fA-F]{64}$/.test(value)) { + throw new PublicRequestError('blockHash must be a 32-byte hexadecimal string') + } + return value.toLowerCase() + } + + const requireTxids = (value: unknown, requireAtLeastOne: boolean = true): string[] => { + if (!Array.isArray(value) || (requireAtLeastOne && value.length === 0)) { + throw new PublicRequestError('txids must be a non-empty array') } if (maxBasmTxids !== -1 && value.length > maxBasmTxids) { throw new PublicRequestError(`txids must contain at most ${maxBasmTxids} entries`) } - return value + const seen = new Set() + const txids: string[] = [] + for (const txid of value) { + if (typeof txid !== 'string' || !/^[0-9a-fA-F]{64}$/.test(txid)) { + throw new PublicRequestError('txids must contain 32-byte hexadecimal transaction IDs') + } + const normalized = txid.toLowerCase() + if (seen.has(normalized)) { + throw new PublicRequestError('txids must not contain duplicates') + } + seen.add(normalized) + txids.push(normalized) + } + return txids } - registerJsonRoute( - '/requestTopicAnchorTip', - async req => await basmEngine.provideTopicAnchorTip(readBasmTopic(req)) - ) + registerJsonRoute('/requestTopicAnchorTip', async req => { + requireBasmCapability('provideTopicAnchorTip') + return await basmEngine.provideTopicAnchorTip(readBasmTopic(req)) + }) registerJsonRoute('/requestTopicAnchorRange', async req => { const { fromHeight, toHeight } = req.body - const from = Number(fromHeight) - const to = Number(toHeight) - if (!Number.isSafeInteger(from) || !Number.isSafeInteger(to) || from < 0 || to < from) { + const from = requireBasmHeight(fromHeight, 'fromHeight') + const to = requireBasmHeight(toHeight, 'toHeight') + if (to < from) { throw new PublicRequestError('fromHeight and toHeight must define a valid ascending range') } if (maxBasmAnchorRange !== -1 && to - from + 1 > maxBasmAnchorRange) { @@ -2288,32 +2353,32 @@ export default class OverlayExpress { `topic anchor range must contain at most ${maxBasmAnchorRange} blocks` ) } + requireBasmCapability('provideTopicAnchorRange') return await basmEngine.provideTopicAnchorRange(readBasmTopic(req), from, to) }) registerJsonRoute('/requestAdmittedList', async req => { const { blockHeight, blockHash } = req.body - return await basmEngine.provideAdmittedList( - readBasmTopic(req), - Number(blockHeight), - typeof blockHash === 'string' ? blockHash : undefined - ) + const height = requireBasmHeight(blockHeight, 'blockHeight') + const hash = requireBlockHash(blockHash) + requireBasmCapability('provideAdmittedList') + return await basmEngine.provideAdmittedList(readBasmTopic(req), height, hash) }) registerJsonRoute('/requestCompoundMerklePath', async req => { const topic = readBasmTopic(req) const { blockHeight, txids } = req.body - return await basmEngine.provideCompoundMerklePath( - topic, - Number(blockHeight), - requireTxids(txids) - ) + const height = requireBasmHeight(blockHeight, 'blockHeight') + const requestedTxids = requireTxids(txids) + requireBasmCapability('provideCompoundMerklePath') + return await basmEngine.provideCompoundMerklePath(topic, height, requestedTxids) }) - registerJsonRoute( - '/requestRawTransactions', - async req => await basmEngine.provideRawTransactions(requireTxids(req.body.txids)) - ) + registerJsonRoute('/requestRawTransactions', async req => { + const txids = requireTxids(req.body.txids, false) + requireBasmCapability('provideRawTransactions') + return await basmEngine.provideRawTransactions(txids) + }) /** * ============== ADMIN ROUTES ============== diff --git a/packages/overlays/overlay-express/src/__tests__/OverlayExpress.test.ts b/packages/overlays/overlay-express/src/__tests__/OverlayExpress.test.ts index 3cb087802..5c64c3215 100644 --- a/packages/overlays/overlay-express/src/__tests__/OverlayExpress.test.ts +++ b/packages/overlays/overlay-express/src/__tests__/OverlayExpress.test.ts @@ -94,6 +94,7 @@ describe('OverlayExpress', () => { expect(overlayExpress.port).toBe(3000) expect(overlayExpress.network).toBe('main') expect(overlayExpress.enableGASPSync).toBe(true) + expect(overlayExpress.enableBASMSync).toBe(false) expect(overlayExpress.verboseRequestLogging).toBe(false) expect(overlayExpress.managers).toEqual({}) expect(overlayExpress.services).toEqual({}) @@ -1692,20 +1693,54 @@ describe('OverlayExpress', () => { }) await invokeCapturedRoute(postSpy, '/requestAdmittedList', { ...topicRequest, - body: { blockHeight: 2, blockHash: 'hash' } + body: { blockHeight: '2', blockHash: 'aa'.repeat(32) } }) await invokeCapturedRoute(postSpy, '/requestCompoundMerklePath', { ...topicRequest, - body: { blockHeight: 2, txids: ['01', '02'] } + body: { blockHeight: '2', txids: ['01'.repeat(32), '02'.repeat(32)] } }) await invokeCapturedRoute(postSpy, '/requestRawTransactions', { - body: { txids: ['01'] } + body: { txids: ['01'.repeat(32)] } }) expect(mockEngine.provideTopicAnchorTip).toHaveBeenCalledWith('tm_test') expect(mockEngine.provideTopicAnchorRange).toHaveBeenCalledWith('tm_test', 1, 3) - expect(mockEngine.provideAdmittedList).toHaveBeenCalledWith('tm_test', 2, 'hash') - expect(mockEngine.provideCompoundMerklePath).toHaveBeenCalledWith('tm_test', 2, ['01', '02']) - expect(mockEngine.provideRawTransactions).toHaveBeenCalledWith(['01']) + expect(mockEngine.provideAdmittedList).toHaveBeenCalledWith('tm_test', 2, 'aa'.repeat(32)) + expect(mockEngine.provideCompoundMerklePath).toHaveBeenCalledWith('tm_test', 2, [ + '01'.repeat(32), + '02'.repeat(32) + ]) + expect(mockEngine.provideRawTransactions).toHaveBeenCalledWith(['01'.repeat(32)]) + + const tipRoute = postSpy.mock.calls.find( + (call: any[]) => call[0] === '/requestTopicAnchorTip' + ) + const rawRoute = postSpy.mock.calls.find( + (call: any[]) => call[0] === '/requestRawTransactions' + ) + const adminRoute = postSpy.mock.calls.find( + (call: any[]) => call[0] === '/admin/startBASMSync' + ) + expect(tipRoute).toHaveLength(2) + expect(rawRoute).toHaveLength(2) + expect(adminRoute.length).toBeGreaterThan(2) + + await invokeCapturedRoute(postSpy, '/requestAdmittedList', { + ...topicRequest, + body: { blockHeight: 3, blockHash: 'BB'.repeat(32) } + }) + await invokeCapturedRoute(postSpy, '/requestAdmittedList', { + ...topicRequest, + body: { blockHeight: 4 } + }) + await invokeCapturedRoute(postSpy, '/requestCompoundMerklePath', { + ...topicRequest, + body: { blockHeight: 3, txids: ['AB'.repeat(32)] } + }) + expect(mockEngine.provideAdmittedList).toHaveBeenCalledWith('tm_test', 3, 'bb'.repeat(32)) + expect(mockEngine.provideAdmittedList).toHaveBeenCalledWith('tm_test', 4, undefined) + expect(mockEngine.provideCompoundMerklePath).toHaveBeenCalledWith('tm_test', 3, [ + 'ab'.repeat(32) + ]) for (const [path, request] of [ ['/requestTopicAnchorTip', { headers: {} }], @@ -1722,6 +1757,141 @@ describe('OverlayExpress', () => { consoleError.mockRestore() }) + it('rejects malformed BASM JSON requests before invoking the engine', async () => { + const consoleError = jest.spyOn(console, 'error').mockImplementation(() => {}) + const { postSpy } = await startAndCaptureRoutes() + const topicRequest = { headers: { 'x-bsv-topic': 'tm_test' } } + + for (const [path, request, message] of [ + [ + '/requestTopicAnchorRange', + { ...topicRequest, body: { fromHeight: null, toHeight: 1 } }, + 'fromHeight must be a nonnegative safe integer' + ], + [ + '/requestTopicAnchorRange', + { ...topicRequest, body: { fromHeight: false, toHeight: 1 } }, + 'fromHeight must be a nonnegative safe integer' + ], + [ + '/requestAdmittedList', + { ...topicRequest, body: { blockHeight: {}, blockHash: 'aa'.repeat(32) } }, + 'blockHeight must be a nonnegative safe integer' + ], + [ + '/requestAdmittedList', + { ...topicRequest, body: { blockHeight: -1 } }, + 'blockHeight must be a nonnegative safe integer' + ], + [ + '/requestAdmittedList', + { ...topicRequest, body: { blockHeight: 1.5 } }, + 'blockHeight must be a nonnegative safe integer' + ], + [ + '/requestTopicAnchorRange', + { + ...topicRequest, + body: { fromHeight: Number.MAX_SAFE_INTEGER + 1, toHeight: 1 } + }, + 'fromHeight must be a nonnegative safe integer' + ], + [ + '/requestAdmittedList', + { ...topicRequest, body: { blockHeight: '2', blockHash: 'not-a-hash' } }, + 'blockHash must be a 32-byte hexadecimal string' + ], + [ + '/requestCompoundMerklePath', + { ...topicRequest, body: { blockHeight: '', txids: ['01'.repeat(32)] } }, + 'blockHeight must be a nonnegative safe integer' + ], + [ + '/requestCompoundMerklePath', + { ...topicRequest, body: { blockHeight: 2, txids: [] } }, + 'txids must be a non-empty array' + ], + [ + '/requestRawTransactions', + { body: { txids: ['not-a-txid'] } }, + 'txids must contain 32-byte hexadecimal transaction IDs' + ], + [ + '/requestRawTransactions', + { body: { txids: ['01'.repeat(32), '01'.repeat(32).toUpperCase()] } }, + 'txids must not contain duplicates' + ] + ] as const) { + const response = await invokeCapturedRoute(postSpy, path, request) + expect(response.status).toHaveBeenCalledWith(400) + expect(response.json).toHaveBeenCalledWith({ status: 'error', message }) + } + + expect(mockEngine.provideTopicAnchorRange).not.toHaveBeenCalled() + expect(mockEngine.provideAdmittedList).not.toHaveBeenCalled() + expect(mockEngine.provideCompoundMerklePath).not.toHaveBeenCalled() + expect(mockEngine.provideRawTransactions).not.toHaveBeenCalled() + consoleError.mockRestore() + }) + + it('preserves empty BASM responses and reports unsupported capabilities', async () => { + const consoleError = jest.spyOn(console, 'error').mockImplementation(() => {}) + const emptyTip = { + topic: 'tm_test', + blockHeight: -1, + tac: '00'.repeat(32) + } + mockEngine.provideTopicAnchorTip.mockResolvedValue(emptyTip) + const { postSpy } = await startAndCaptureRoutes() + + const tip = await invokeCapturedRoute(postSpy, '/requestTopicAnchorTip', { + headers: { 'x-bsv-topic': 'tm_test' } + }) + expect(tip.status).toHaveBeenCalledWith(200) + expect(tip.json).toHaveBeenCalledWith(emptyTip) + + const raw = await invokeCapturedRoute(postSpy, '/requestRawTransactions', { + body: { txids: [] } + }) + expect(raw.status).toHaveBeenCalledWith(200) + expect(mockEngine.provideRawTransactions).toHaveBeenCalledWith([]) + + delete mockEngine.provideRawTransactions + const unsupported = await invokeCapturedRoute(postSpy, '/requestRawTransactions', { + body: { txids: ['01'.repeat(32)] } + }) + expect(unsupported.status).toHaveBeenCalledWith(400) + expect(unsupported.json).toHaveBeenCalledWith({ + status: 'error', + message: 'BASM capability is not supported by this Overlay engine', + code: 'BASM_UNSUPPORTED' + }) + consoleError.mockRestore() + }) + + it('honors the unlimited BASM transaction limit override', async () => { + const previousLimit = process.env.OVERLAY_MAX_BASM_TXIDS + process.env.OVERLAY_MAX_BASM_TXIDS = '-1' + try { + const { postSpy } = await startAndCaptureRoutes() + const txids = Array.from({ length: 1001 }, (_, index) => + index.toString(16).padStart(64, '0') + ) + const response = await invokeCapturedRoute(postSpy, '/requestRawTransactions', { + body: { txids } + }) + + expect(response.status).toHaveBeenCalledWith(200) + expect(mockEngine.provideRawTransactions).toHaveBeenCalledWith(txids) + } finally { + if (previousLimit === undefined) { + delete process.env.OVERLAY_MAX_BASM_TXIDS + } else { + process.env.OVERLAY_MAX_BASM_TXIDS = previousLimit + } + } + }) + it('enforces admin authentication and executes bounded record and ban operations', async () => { instance.configureAdminIdentityKey('admin-identity') const banService = { diff --git a/packages/overlays/overlay/CHANGELOG.md b/packages/overlays/overlay/CHANGELOG.md index 8660374bb..0f74ad78b 100644 --- a/packages/overlays/overlay/CHANGELOG.md +++ b/packages/overlays/overlay/CHANGELOG.md @@ -26,6 +26,20 @@ All notable changes to this project will be documented in this file. The format payload-GC foundation behind an optional `mongodb` peer. It does not activate MongoDB in Engine, select a new default adapter, or provide an admission receipt path. +- Added bounded BASM JSON peer validation, classified capability/resource errors, + canonical header and optional full-block-count position checks, and explicit + position assurance in sync reports. Fixed default forward pages to fit the + public server's 1,000-anchor limit. Claimed admitted-list indices are bound to + the compound path even when every remote txid is already local. Inclusion uses + chain-tracker root/height rather than coinbase maturity, so BASM admission + submits in the `historical-tx-no-spv` mode and topic managers that branch on + the submission mode observe it for BASM-admitted transactions; the public + `historical-tx` mode keeps full SPV verification. Heights where the topic + admitted nothing (BRC-136 `k = 0`, zero BASM root) are checked without a proof + request instead of aborting the sync. Historical sync + refuses an untrusted TAC prefix; durable bootstrap, chunked recovery and topic + status remain pending. No storage migration or automatic sync activation is + included. ### Added - (Include new features or significant user-visible enhancements here.) diff --git a/packages/overlays/overlay/README.md b/packages/overlays/overlay/README.md index 734052799..3f9bbef58 100644 --- a/packages/overlays/overlay/README.md +++ b/packages/overlays/overlay/README.md @@ -163,6 +163,59 @@ For production deployments: `@bsv/overlay-express` supplies these standard HTTP controls while preserving public protocol access by default. +### BASM peer validation and current recovery limits + +BASM uses the current BRC-136 ordered admitted subset and block-anchored TAC. +The five existing JSON POST routes remain compatible; empty tips remain +`{ topic, blockHeight: -1, tac: }`. Unsupported storage capabilities +are errors, not empty histories. The remote client validates response shape, +topic/height/hash binding, ordered unique admitted positions, contiguous +returned ranges, and complete proof/raw response ID sets before use. + +`BASMRemote` retains its injectable third `fetch` argument and accepts optional +limits as a fourth argument. Defaults are 64 MiB per decoded response, 8 MiB +per proof, 32 MiB per raw transaction, 100,000 admitted entries, 1,000 requested +txids, 1,024 requested anchor heights, and 30 seconds per request including its +body. Aggregate response limits also apply to hex-encoded transactions. These +are configurable local acceptance limits, not consensus rules. Standard fetch +bodies are bounded while streaming; legacy injected `text()` implementations +are checked after buffering. Classified errors expose `code`, including +`BASM_UNSUPPORTED`, `BASM_RESOURCE_LIMIT`, and `BASM_TIMEOUT`. + +Reconciliation requires a canonical header resolver as well as a ChainTracker. +An optional `TopicAnchorHeader.blockTransactionCount` must come independently +from the trusted canonical provider and refer to that exact `blockHash`. +It enables full-block count/index bounds and odd-duplication checks. The sync +report's `positionValidation` is `canonical-count` only when that evidence was +available for every checked proof; legacy providers yield `encoded-offset-only`. +A Merkle root plus an encoded offset alone cannot disambiguate Bitcoin's +duplicate-last-leaf position ambiguity. No provider is required to add the +field, and the engine does not download full blocks to infer it. + +Forward sync pages now contain at most 1,000 anchors to fit the standard HTTP +server. Proof height, requested original index, canonical hash/root, raw byte +identity, TAC continuity, and repeated peer anchors are checked before historical +submission. Claimed admitted-list indices are bound to the compound path whenever +a remote list is used as evidence, including when every remote txid is already +local. Inclusion uses the chain tracker root/height check rather than +`MerklePath.verify`, which also enforces coinbase 100-block spendability. +Because inclusion is proven independently, admission submits in the +`historical-tx-no-spv` mode so `Transaction.verify` does not re-apply that +coinbase rule; the public `historical-tx` mode keeps full SPV verification. +Admission still applies the local TopicManager, and because every admitted +transaction carries its extracted Merkle path, neither network broadcast nor +overlay propagation occurs. Automatic BASM sync remains disabled by default. + +This is bounded protocol hardening, not durable recovery. An empty local node +whose topic genesis precedes the recent bootstrap window now refuses the +untrusted TAC prefix; this intentionally replaces the old unchecked tail +behavior. A block above 1,000 admitted entries reaches a request-limit error +until proof/raw chunking is implemented. Equal-height/local-ahead divergence, +whole-target bootstrap, durable cursors/leases, atomic revision fencing, and +truthful per-topic agreement status remain required follow-up work. A successful +legacy report does not establish global completeness, current unspentness, or +durable recovery completion. See [BASM details](./docs/BRC-136-BASM.md). + ## Development From the repository root: diff --git a/packages/overlays/overlay/docs/BRC-136-BASM.md b/packages/overlays/overlay/docs/BRC-136-BASM.md index 4cd1ed46d..649a75834 100644 --- a/packages/overlays/overlay/docs/BRC-136-BASM.md +++ b/packages/overlays/overlay/docs/BRC-136-BASM.md @@ -7,6 +7,17 @@ topic**. The two systems are deliberately decoupled. Read the final section ("How BASM and removal interact") before operating a node that runs the janitor or removes tokens via the admin API. +The protocol's binary-search reconciliation is a design capability, not an +implemented durable recovery scheduler in this release. The current engine +performs bounded forward work and refuses historical divergence or an untrusted +bootstrap prefix. See the [current validation and recovery limits](../README.md#basm-peer-validation-and-current-recovery-limits) +before interpreting a sync report. Optional canonical full-block counts provide +stronger original-position evidence; header-only providers retain explicitly +weaker evidence. A remote admitted list is bound to its compound Merkle path +even when every claimed txid is already local. Inclusion is canonical +root/height, not coinbase maturity. Peer anchor equality is not durable or +global completeness. + ## What BRC-136 gives you GASP ([Synchronization.md](./Synchronization.md)) replicates transaction *data* diff --git a/packages/overlays/overlay/package.json b/packages/overlays/overlay/package.json index 406b7e56e..de23ed672 100644 --- a/packages/overlays/overlay/package.json +++ b/packages/overlays/overlay/package.json @@ -1,6 +1,6 @@ { "name": "@bsv/overlay", - "version": "2.4.0", + "version": "2.5.0", "sideEffects": false, "engines": { "node": ">=22" diff --git a/packages/overlays/overlay/src/BASM.ts b/packages/overlays/overlay/src/BASM.ts index 470894d8b..3ec4d61fa 100644 --- a/packages/overlays/overlay/src/BASM.ts +++ b/packages/overlays/overlay/src/BASM.ts @@ -44,9 +44,13 @@ export interface TopicAnchorHeader { blockHeight: number blockHash: string merkleRoot?: string + /** Independently obtained full block transaction count, bound to this blockHash. */ + blockTransactionCount?: number } -export type TopicAnchorHeaderResolver = (blockHeight: number) => Promise +export type TopicAnchorHeaderResolver = ( + blockHeight: number +) => Promise export interface TopicAnchorRangeRequest { fromHeight: number @@ -106,6 +110,10 @@ export interface BASMPeerSyncReport { checkedHeights: number[] missingTxids: string[] fetchedTxCount: number + /** Weakest position evidence used by this attempt; absent when no proof was checked. */ + positionValidation?: 'canonical-count' | 'encoded-offset-only' + /** Present for classified peer protocol or transport failures. */ + errorCode?: string message?: string } @@ -190,7 +198,10 @@ export function computeTac(prevTac: string, blockHash: string, basmRoot: string) return internalToDisplayHex(sha256d(input)) } -export function extractMerkleProofMetadata(txid: string, proof?: MerklePath): MerkleProofMetadata | undefined { +export function extractMerkleProofMetadata( + txid: string, + proof?: MerklePath +): MerkleProofMetadata | undefined { if (proof === undefined) { return undefined } diff --git a/packages/overlays/overlay/src/BASMRemote.ts b/packages/overlays/overlay/src/BASMRemote.ts index d30a58807..bcb09ffd9 100644 --- a/packages/overlays/overlay/src/BASMRemote.ts +++ b/packages/overlays/overlay/src/BASMRemote.ts @@ -5,35 +5,230 @@ import type { TopicAnchorRangeResponse, TopicAnchorTip } from './BASM.js' +import { + BASMProtocolError, + DEFAULT_BASM_REMOTE_LIMITS, + basmAdmitted, + basmAnchor, + basmHash, + basmHex, + basmInteger, + basmObject, + basmTip, + basmTxids, + requireBASM, + requireBASMLimit +} from './BASMValidation.js' +import type { BASMRemoteLimits } from './BASMValidation.js' + +export { BASMProtocolError } from './BASMValidation.js' +export type { BASMRemoteLimits } from './BASMValidation.js' export class BASMRemote { + private readonly limits: Readonly + constructor( private readonly endpoint: string, private readonly topic: string, - private readonly fetchImpl: typeof fetch = fetch.bind(globalThis) - ) { } + private readonly fetchImpl: typeof fetch = fetch.bind(globalThis), + limits: Partial = {} + ) { + this.limits = Object.freeze({ ...DEFAULT_BASM_REMOTE_LIMITS, ...limits }) + for (const [key, value] of Object.entries(this.limits)) { + if (!Number.isSafeInteger(value) || value <= 0) + throw new TypeError(`Invalid BASM limit: ${key}`) + } + if (this.limits.timeoutMs > 2147483647) throw new TypeError('BASM timeout exceeds timer range') + } async requestTopicAnchorTip(): Promise { - return await this.post('/requestTopicAnchorTip', {}) + return basmTip(await this.post('/requestTopicAnchorTip', {}), this.topic) } - async requestTopicAnchorRange(fromHeight: number, toHeight: number): Promise { - return await this.post('/requestTopicAnchorRange', { fromHeight, toHeight }) + async requestTopicAnchorRange( + fromHeight: number, + toHeight: number + ): Promise { + basmInteger(fromHeight, 'range start') + basmInteger(toHeight, 'range end', fromHeight) + requireBASMLimit( + toHeight - fromHeight < this.limits.maxAnchorRange, + 'BASM anchor range exceeds limit' + ) + const obj = basmObject(await this.post('/requestTopicAnchorRange', { fromHeight, toHeight })) + requireBASM(obj.topic === this.topic && Array.isArray(obj.anchors), 'Invalid BASM anchor range') + requireBASMLimit( + obj.anchors.length <= toHeight - fromHeight + 1, + 'BASM anchor count exceeds requested range' + ) + const anchors = obj.anchors.map(value => basmAnchor(value, this.topic)) + let previousHeight: number | undefined + for (const anchor of anchors) { + requireBASM( + anchor.blockHeight >= fromHeight && anchor.blockHeight <= toHeight, + 'BASM anchor outside requested range' + ) + requireBASM( + previousHeight === undefined || anchor.blockHeight === previousHeight + 1, + 'BASM anchor range has a gap or is unordered' + ) + previousHeight = anchor.blockHeight + } + return { topic: this.topic, anchors } } - async requestAdmittedList(blockHeight: number, blockHash?: string): Promise { - return await this.post('/requestAdmittedList', { blockHeight, blockHash }) + async requestAdmittedList( + blockHeight: number, + blockHash?: string + ): Promise { + basmInteger(blockHeight, 'height') + if (blockHash !== undefined) basmHash(blockHash, 'block hash') + const obj = basmObject(await this.post('/requestAdmittedList', { blockHeight, blockHash })) + this.requireCoordinate(obj, blockHeight) + const responseHash = + obj.blockHash === undefined ? undefined : basmHash(obj.blockHash, 'block hash') + requireBASM( + blockHash === undefined || responseHash === undefined || responseHash === blockHash, + 'BASM admitted block hash mismatch' + ) + return { + topic: this.topic, + blockHeight, + ...(responseHash === undefined ? {} : { blockHash: responseHash }), + admitted: basmAdmitted(obj.admitted, this.limits.maxAdmittedTxids) + } } - async requestCompoundMerklePath(blockHeight: number, txids: string[]): Promise { - return await this.post('/requestCompoundMerklePath', { blockHeight, txids }) + async requestCompoundMerklePath( + blockHeight: number, + txids: string[] + ): Promise { + basmInteger(blockHeight, 'height') + const requested = basmTxids(txids, this.limits.maxRequestedTxids) + requireBASM(requested.length > 0, 'At least one BASM proof txid is required') + const obj = basmObject( + await this.post('/requestCompoundMerklePath', { blockHeight, txids: requested }) + ) + this.requireCoordinate(obj, blockHeight) + const returned = basmTxids(obj.txids, requested.length) + const requestedSet = new Set(requested) + requireBASM( + returned.length === requested.length && returned.every(txid => requestedSet.has(txid)), + 'BASM proof txids do not match request' + ) + return { + topic: this.topic, + blockHeight, + txids: returned, + merklePath: basmHex(obj.merklePath, 'Merkle path', this.limits.maxProofBytes) + } } async requestRawTransactions(txids: string[]): Promise { - return await this.post('/requestRawTransactions', { txids }) + const requested = new Set(basmTxids(txids, this.limits.maxRequestedTxids)) + const obj = basmObject(await this.post('/requestRawTransactions', { txids: [...requested] })) + requireBASM(Array.isArray(obj.transactions), 'Invalid BASM raw transactions') + requireBASMLimit( + obj.transactions.length <= requested.size, + 'BASM raw transaction count exceeds request' + ) + const seen = new Set() + const transactions = obj.transactions.map(value => { + const record = basmObject(value) + const txid = basmHash(record.txid, 'txid') + requireBASM( + requested.has(txid) && !seen.has(txid), + 'Unexpected or duplicate BASM raw transaction' + ) + seen.add(txid) + return { + txid, + rawTx: basmHex(record.rawTx, 'raw transaction', this.limits.maxRawTransactionBytes) + } + }) + const missing = basmTxids(obj.missing, requested.size) + for (const txid of missing) { + requireBASM( + requested.has(txid) && !seen.has(txid), + 'Unexpected or conflicting BASM missing txid' + ) + seen.add(txid) + } + requireBASM(seen.size === requested.size, 'BASM raw response omits requested txids') + return { transactions, missing } + } + + private requireCoordinate(obj: Record, blockHeight: number): void { + requireBASM( + obj.topic === this.topic && obj.blockHeight === blockHeight, + 'BASM response topic or height mismatch' + ) } - private async post(path: string, body: unknown): Promise { + private async readResponse(response: Response, signal: AbortSignal): Promise { + const advertisedLength = response.headers?.get('content-length') + if (advertisedLength !== null && advertisedLength !== undefined) { + if (Number(advertisedLength) > this.limits.maxResponseBytes) { + void response.body?.cancel().catch(() => {}) + requireBASMLimit(false, 'BASM response exceeds byte limit') + } + } + // Preserve injected fetch implementations that expose only text(). Real HTTP + // responses are bounded while streaming, including decoded chunked bodies. + if (response.body === undefined || response.body === null) { + const text = await response.text() + requireBASMLimit( + Buffer.byteLength(text) <= this.limits.maxResponseBytes, + 'BASM response exceeds byte limit' + ) + return text + } + const reader = response.body.getReader() + const chunks: Uint8Array[] = [] + let size = 0 + const cancel = (): void => { + void reader.cancel().catch(() => {}) + } + signal.addEventListener('abort', cancel, { once: true }) + try { + while (true) { + signal.throwIfAborted() + const { done, value } = await reader.read() + if (done) break + size += value.byteLength + requireBASMLimit(size <= this.limits.maxResponseBytes, 'BASM response exceeds byte limit') + chunks.push(value) + } + signal.throwIfAborted() + return Buffer.concat(chunks, size).toString('utf8') + } catch (error) { + cancel() + throw error + } finally { + signal.removeEventListener('abort', cancel) + reader.releaseLock() + } + } + + private async post(path: string, body: unknown): Promise { + const controller = new AbortController() + let timer: ReturnType | undefined + const deadline = new Promise((_resolve, reject) => { + timer = setTimeout(() => { + const error = new BASMProtocolError('BASM_TIMEOUT', 'BASM request timed out') + controller.abort(error) + reject(error) + }, this.limits.timeoutMs) + }) + try { + return await Promise.race([this.postResponse(path, body, controller.signal), deadline]) + } finally { + clearTimeout(timer) + controller.abort() + } + } + + private async postResponse(path: string, body: unknown, signal: AbortSignal): Promise { const response = await this.fetchImpl(new URL(path, this.endpoint).toString(), { method: 'POST', headers: { @@ -41,14 +236,32 @@ export class BASMRemote { 'Content-Type': 'application/json', 'x-bsv-topic': this.topic }, - body: JSON.stringify(body) + body: JSON.stringify(body), + signal }) - - const text = await response.text() + if (signal.aborted) { + void response.body?.cancel().catch(() => {}) + signal.throwIfAborted() + } + const text = await this.readResponse(response, signal) + let value: unknown + try { + value = JSON.parse(text) + } catch { + if (response.ok) + throw new BASMProtocolError('BASM_INVALID_RESPONSE', 'Invalid BASM JSON response') + } if (!response.ok) { - throw new Error(`BASM peer ${this.endpoint} returned ${response.status}: ${text}`) + const code = + typeof value === 'object' && value !== null && 'code' in value ? value.code : undefined + if (response.status === 501 || code === 'BASM_UNSUPPORTED') { + throw new BASMProtocolError( + 'BASM_UNSUPPORTED', + `BASM peer returned HTTP ${response.status}: unsupported BASM capability` + ) + } + throw new BASMProtocolError('BASM_HTTP_ERROR', `BASM peer returned HTTP ${response.status}`) } - - return (text.length === 0 ? {} : JSON.parse(text)) as T + return value } } diff --git a/packages/overlays/overlay/src/BASMValidation.ts b/packages/overlays/overlay/src/BASMValidation.ts new file mode 100644 index 000000000..fa6721cb2 --- /dev/null +++ b/packages/overlays/overlay/src/BASMValidation.ts @@ -0,0 +1,131 @@ +import { BASM_ZERO_HASH } from './BASM.js' +import type { AdmittedTxRef, TopicAnchorTip, TopicBlockAnchor } from './BASM.js' + +/** Local acceptance limits, not Bitcoin consensus limits. */ +export interface BASMRemoteLimits { + maxResponseBytes: number + maxProofBytes: number + maxRawTransactionBytes: number + maxAdmittedTxids: number + maxRequestedTxids: number + maxAnchorRange: number + timeoutMs: number +} + +export const DEFAULT_BASM_REMOTE_LIMITS: Readonly = Object.freeze({ + maxResponseBytes: 64 * 1024 * 1024, + maxProofBytes: 8 * 1024 * 1024, + maxRawTransactionBytes: 32 * 1024 * 1024, + maxAdmittedTxids: 100000, + maxRequestedTxids: 1000, + maxAnchorRange: 1024, + timeoutMs: 30000 +}) + +export class BASMProtocolError extends Error { + constructor( + public readonly code: + | 'BASM_INVALID_RESPONSE' + | 'BASM_RESOURCE_LIMIT' + | 'BASM_UNSUPPORTED' + | 'BASM_TIMEOUT' + | 'BASM_HTTP_ERROR', + message: string + ) { + super(message) + this.name = 'BASMProtocolError' + } +} + +export function requireBASM(condition: boolean, message: string): asserts condition { + if (!condition) throw new BASMProtocolError('BASM_INVALID_RESPONSE', message) +} + +export function requireBASMLimit(condition: boolean, message: string): void { + if (!condition) throw new BASMProtocolError('BASM_RESOURCE_LIMIT', message) +} + +export function basmObject(value: unknown): Record { + requireBASM( + typeof value === 'object' && value !== null && !Array.isArray(value), + 'Expected a BASM JSON object' + ) + return value as Record +} + +export function basmInteger(value: unknown, label: string, minimum = 0): number { + requireBASM( + typeof value === 'number' && Number.isSafeInteger(value) && value >= minimum, + `Invalid BASM ${label}` + ) + return value +} + +export function basmHash(value: unknown, label: string): string { + requireBASM(typeof value === 'string' && /^[0-9a-f]{64}$/.test(value), `Invalid BASM ${label}`) + return value +} + +export function basmHex(value: unknown, label: string, maxBytes: number): string { + requireBASM(typeof value === 'string', `Invalid BASM ${label}`) + requireBASMLimit(value.length <= maxBytes * 2, `BASM ${label} exceeds byte limit`) + requireBASM( + value.length > 0 && value.length % 2 === 0 && /^[0-9a-fA-F]+$/.test(value), + `Invalid BASM ${label}` + ) + return value +} + +export function basmTxids(value: unknown, maximum: number): string[] { + requireBASM(Array.isArray(value), 'Invalid BASM txids') + requireBASMLimit(value.length <= maximum, 'BASM txid count exceeds limit') + const txids = value.map(txid => basmHash(txid, 'txid')) + requireBASM(new Set(txids).size === txids.length, 'Duplicate BASM txid') + return txids +} + +export function basmTip(value: unknown, topic: string): TopicAnchorTip { + const obj = basmObject(value) + requireBASM(obj.topic === topic, 'BASM topic mismatch') + const blockHeight = basmInteger(obj.blockHeight, 'tip height', -1) + const tac = basmHash(obj.tac, 'TAC') + const tip: TopicAnchorTip = { topic, blockHeight, tac } + if (obj.blockHash !== undefined) tip.blockHash = basmHash(obj.blockHash, 'block hash') + if (obj.basmRoot !== undefined) tip.basmRoot = basmHash(obj.basmRoot, 'root') + if (obj.admittedCount !== undefined) + tip.admittedCount = basmInteger(obj.admittedCount, 'admitted count') + if (blockHeight === -1) requireBASM(tac === BASM_ZERO_HASH, 'Empty BASM tip must have zero TAC') + return tip +} + +export function basmAnchor(value: unknown, topic: string): TopicBlockAnchor { + const obj = basmObject(value) + const tip = basmTip(value, topic) + return { + topic, + blockHeight: basmInteger(tip.blockHeight, 'anchor height'), + blockHash: basmHash(obj.blockHash, 'block hash'), + basmRoot: basmHash(obj.basmRoot, 'root'), + admittedCount: basmInteger(obj.admittedCount, 'admitted count'), + tac: tip.tac + } +} + +export function basmAdmitted(value: unknown, maximum: number): AdmittedTxRef[] { + requireBASM(Array.isArray(value), 'Invalid BASM admitted list') + requireBASMLimit(value.length <= maximum, 'BASM admitted count exceeds limit') + let previousIndex = -1 + const seen = new Set() + return value.map(item => { + const obj = basmObject(item) + const txid = basmHash(obj.txid, 'txid') + const blockIndex = basmInteger(obj.blockIndex, 'block index') + requireBASM( + blockIndex > previousIndex && !seen.has(txid), + 'BASM admitted list must have unique txids in increasing block order' + ) + previousIndex = blockIndex + seen.add(txid) + return { txid, blockIndex } + }) +} diff --git a/packages/overlays/overlay/src/Engine.ts b/packages/overlays/overlay/src/Engine.ts index 5022b010d..2f0481ee0 100644 --- a/packages/overlays/overlay/src/Engine.ts +++ b/packages/overlays/overlay/src/Engine.ts @@ -27,11 +27,13 @@ import { OverlayGASPRemote } from './GASP/OverlayGASPRemote.js' import { OverlayGASPStorage } from './GASP/OverlayGASPStorage.js' import { BASM_ZERO_HASH, + type AdmittedTxRef, type AdmittedListResponse, type BASMPeerSyncReport, type CompoundMerklePathResponse, type RawTransactionResponse, type ReorgReport, + type TopicAnchorHeader, type TopicAnchorHeaderResolver, type TopicAnchorRangeResponse, type TopicAnchorTip, @@ -41,10 +43,13 @@ import { extractMerkleProofMetadata } from './BASM.js' import { BASMRemote } from './BASMRemote.js' +import { basmHash, basmInteger, requireBASM } from './BASMValidation.js' import { serializeErrorForLog, serializeLogValue } from './SafeLog.js' const DEFAULT_GASP_SYNC_LIMIT = 10000 const DEFAULT_BASM_RANGE_LIMIT = 1024 +// The public Overlay Express transport defaults to 1,000 anchors per request. +const DEFAULT_BASM_SYNC_PAGE_SIZE = 1000 type UTXOHistoryHydrationContext = { outputCache: Map> @@ -111,6 +116,12 @@ function findSpendingInputIndex(tx: Transaction, output: Output): number { }) } +function requireBASMDefined(value: T | undefined, aligned: boolean, message: string): T { + requireBASM(aligned, message) + requireBASM(value !== undefined, message) + return value +} + /** * An engine for running BSV Overlay Services (topic managers and lookup services). */ @@ -1060,6 +1071,10 @@ export class Engine { const txid = tx.id('hex') this.startTime(`submit_${txid}`) + // Every submission is SPV-verified except 'historical-tx-no-spv', which is + // reserved for callers that have already proven inclusion independently + // (GASP graph finalization and BASM reconciliation). 'historical-tx' is a + // public submission mode, so it keeps the full SPV check. if (mode !== 'historical-tx-no-spv') { this.startTime(`chainTracker_${txid.substring(0, 10)}`) const txValid = await tx.verify(this.chainTracker) @@ -1541,6 +1556,9 @@ export class Engine { } async provideTopicAnchorTip(topic: string): Promise { + if (typeof this.storage.findTopicAnchorTip !== 'function') { + throw Object.assign(new TypeError('Storage does not support BASM topic anchor tips'), { code: 'BASM_UNSUPPORTED' }) + } const tip = await this.storage.findTopicAnchorTip?.(topic) return tip ?? { topic, @@ -1551,7 +1569,7 @@ export class Engine { async provideTopicAnchorRange(topic: string, fromHeight: number, toHeight: number): Promise { if (typeof this.storage.findTopicBlockAnchors !== 'function') { - throw new TypeError('Storage does not support BASM topic anchor ranges') + throw Object.assign(new TypeError('Storage does not support BASM topic anchor ranges'), { code: 'BASM_UNSUPPORTED' }) } if (!Number.isInteger(fromHeight) || !Number.isInteger(toHeight) || fromHeight < 0 || toHeight < fromHeight) { throw new Error('Invalid topic anchor range') @@ -1568,7 +1586,7 @@ export class Engine { async provideAdmittedList(topic: string, blockHeight: number, blockHash?: string): Promise { if (typeof this.storage.findAdmittedTransactionsForBlock !== 'function') { - throw new TypeError('Storage does not support BASM admitted lists') + throw Object.assign(new TypeError('Storage does not support BASM admitted lists'), { code: 'BASM_UNSUPPORTED' }) } return { @@ -1581,7 +1599,7 @@ export class Engine { async provideCompoundMerklePath(topic: string, blockHeight: number, txids: string[]): Promise { if (typeof this.storage.findTransactionMerklePaths !== 'function') { - throw new TypeError('Storage does not support direct Merkle path lookup') + throw Object.assign(new TypeError('Storage does not support direct Merkle path lookup'), { code: 'BASM_UNSUPPORTED' }) } if (txids.length === 0) { throw new Error('At least one txid is required') @@ -1632,7 +1650,7 @@ export class Engine { async provideRawTransactions(txids: string[]): Promise { if (typeof this.storage.findRawTransactions !== 'function') { - throw new TypeError('Storage does not support raw transaction lookup') + throw Object.assign(new TypeError('Storage does not support raw transaction lookup'), { code: 'BASM_UNSUPPORTED' }) } const transactions = await this.storage.findRawTransactions(txids) @@ -1677,30 +1695,143 @@ export class Engine { ]) report.localTip = localTip report.remoteTip = remoteTip - + await this.requireMatchingRemoteBASMTip(remote, remoteTip) if (localTip.blockHeight >= remoteTip.blockHeight) { - report.status = localTip.tac === remoteTip.tac && localTip.blockHeight === remoteTip.blockHeight ? 'matched' : 'diverged' - report.message = report.status === 'matched' - ? 'Topic anchor tips match' - : 'Remote tip is not ahead; historical divergence needs manual or binary-search reconciliation' - return report + return await this.finishBASMWhenRemoteIsNotAhead(topic, localTip, remoteTip, report) } + return await this.advanceBASMWithRemoteAnchors(topic, remote, localTip, remoteTip, report) + } catch (error) { + return this.markBASMPeerSyncError(report, topic, endpoint, error) + } + } - const fromHeight = Math.max(localTip.blockHeight + 1, remoteTip.blockHeight - DEFAULT_BASM_RANGE_LIMIT + 1, 0) - const range = await remote.requestTopicAnchorRange(fromHeight, remoteTip.blockHeight) - for (const remoteAnchor of range.anchors) { - await this.reconcileRemoteAnchor(topic, remote, remoteAnchor, report) - } + private async requireMatchingRemoteBASMTip(remote: BASMRemote, remoteTip: TopicAnchorTip): Promise { + if (remoteTip.blockHeight < 0) { + return + } + const tipRange = await remote.requestTopicAnchorRange(remoteTip.blockHeight, remoteTip.blockHeight) + const remoteTipAnchor = tipRange.anchors[0] + const tipAnchor = requireBASMDefined( + remoteTipAnchor, + remoteTipAnchor?.tac === remoteTip.tac, + 'BASM tip does not match its anchor' + ) + for (const field of ['blockHash', 'basmRoot', 'admittedCount'] as const) { + requireBASM(remoteTip[field] === undefined || remoteTip[field] === tipAnchor[field], 'BASM tip metadata does not match its anchor') + } + await this.requireCanonicalBASMAnchor(tipAnchor) + } - const refreshedTip = await this.provideTopicAnchorTip(topic) - report.localTip = refreshedTip - report.status = refreshedTip.blockHeight >= remoteTip.blockHeight && refreshedTip.tac === remoteTip.tac ? 'matched' : 'advanced' - return report - } catch (error) { - report.status = 'error' - report.message = error instanceof Error ? error.message : String(error) - this.logger.error(`[BASM SYNC] Sync failed for topic "${topic}" with peer "${endpoint}"`, error) - return report + private async finishBASMWhenRemoteIsNotAhead( + topic: string, + localTip: TopicAnchorTip, + remoteTip: TopicAnchorTip, + report: BASMPeerSyncReport + ): Promise { + const tipsMatch = localTip.tac === remoteTip.tac && localTip.blockHeight === remoteTip.blockHeight + if (localTip.blockHeight >= 0 && tipsMatch) { + const localAnchor = await this.storage.findTopicBlockAnchor?.(topic, localTip.blockHeight) + await this.requireCanonicalBASMAnchor( + requireBASMDefined(localAnchor, localAnchor?.tac === localTip.tac, 'Local BASM tip lacks its anchor') + ) + } + report.status = tipsMatch ? 'matched' : 'diverged' + report.message = tipsMatch + ? 'Topic anchor tips match' + : 'Remote tip is not ahead; historical divergence needs manual or binary-search reconciliation' + return report + } + + private requireBASMRangePrefix( + localTip: TopicAnchorTip, + remoteTip: TopicAnchorTip, + fromHeight: number, + toHeight: number, + anchors: TopicBlockAnchor[] + ): void { + requireBASM(anchors.length > 0 && anchors.at(-1)?.blockHeight === toHeight, 'BASM range omits its requested target') + requireBASM(localTip.blockHeight < 0 || anchors[0].blockHeight === fromHeight, 'BASM range omits its next height') + let previousTac = localTip.tac + for (const anchor of anchors) { + requireBASM(anchor.tac === computeTac(previousTac, anchor.blockHash, anchor.basmRoot), 'BASM range TAC is inconsistent with its prefix') + previousTac = anchor.tac + } + if (toHeight === remoteTip.blockHeight) requireBASM(previousTac === remoteTip.tac, 'BASM range differs from its tip') + } + + private async advanceBASMWithRemoteAnchors( + topic: string, + remote: BASMRemote, + localTip: TopicAnchorTip, + remoteTip: TopicAnchorTip, + report: BASMPeerSyncReport + ): Promise { + const fromHeight = localTip.blockHeight < 0 + ? Math.max(remoteTip.blockHeight - DEFAULT_BASM_SYNC_PAGE_SIZE + 1, 0) + : localTip.blockHeight + 1 + const toHeight = Math.min(fromHeight + DEFAULT_BASM_SYNC_PAGE_SIZE - 1, remoteTip.blockHeight) + const range = await remote.requestTopicAnchorRange(fromHeight, toHeight) + this.requireBASMRangePrefix(localTip, remoteTip, fromHeight, toHeight, range.anchors) + for (const remoteAnchor of range.anchors) { + await this.reconcileRemoteAnchor(topic, remote, remoteAnchor, report) + if (report.status === 'diverged') return report + } + + const finalRemoteTip = await remote.requestTopicAnchorTip() + requireBASM(finalRemoteTip.blockHeight === remoteTip.blockHeight && finalRemoteTip.tac === remoteTip.tac, 'BASM peer history changed during reconciliation') + + const refreshedTip = await this.provideTopicAnchorTip(topic) + report.localTip = refreshedTip + report.status = refreshedTip.blockHeight === remoteTip.blockHeight && refreshedTip.tac === remoteTip.tac ? 'matched' : 'advanced' + return report + } + + private markBASMPeerSyncError( + report: BASMPeerSyncReport, + topic: string, + endpoint: string, + error: unknown + ): BASMPeerSyncReport { + report.status = 'error' + if (error instanceof Error && 'code' in error && typeof error.code === 'string') report.errorCode = error.code + report.message = error instanceof Error ? error.message : String(error) + this.logger.error(`[BASM SYNC] Sync failed for topic "${topic}" with peer "${endpoint}"`, error) + return report + } + + private async requireCanonicalBASMAnchor(anchor: TopicBlockAnchor, proofRoot?: string): Promise { + if (this.chainTracker === 'scripts only' || this.topicAnchorHeaderResolver === undefined) { + throw new Error('BASM reconciliation requires a ChainTracker and canonical header resolver') + } + const resolvedHeader = await this.topicAnchorHeaderResolver(anchor.blockHeight) + const header = requireBASMDefined( + resolvedHeader, + resolvedHeader?.blockHeight === anchor.blockHeight, + 'BASM canonical header is unavailable or has the wrong height' + ) + requireBASM(basmHash(header.blockHash.toLowerCase(), 'canonical block hash') === anchor.blockHash, 'BASM anchor block hash is not canonical') + if (proofRoot !== undefined && header.merkleRoot !== undefined) { + requireBASM(header.merkleRoot.toLowerCase() === proofRoot, 'BASM proof root differs from its canonical header') + } + if (header.blockTransactionCount !== undefined) { + basmInteger(header.blockTransactionCount, 'canonical block transaction count', 1) + requireBASM(anchor.admittedCount <= header.blockTransactionCount, 'BASM admitted count exceeds canonical block transaction count') + } + return header + } + + private validateBASMProofPositions(path: MerklePath, admitted: AdmittedTxRef[], count: number): void { + requireBASM(admitted.every(item => item.blockIndex < count), 'BASM admitted index exceeds canonical block transaction count') + let width = count + for (let height = 0; height < path.path.length; height++) { + requireBASM(height === 0 || width > 1, 'BASM proof exceeds canonical tree depth') + for (const node of path.path[height]) { + requireBASM( + node.duplicate === true ? width % 2 === 1 && node.offset === width : node.offset < width, + 'BASM proof node is outside canonical block positions' + ) + } + width = Math.ceil(width / 2) } } @@ -1710,6 +1841,7 @@ export class Engine { remoteAnchor: TopicBlockAnchor, report: BASMPeerSyncReport ): Promise { + await this.requireCanonicalBASMAnchor(remoteAnchor) report.checkedHeights.push(remoteAnchor.blockHeight) const localAnchor = await this.storage.findTopicBlockAnchor?.(topic, remoteAnchor.blockHeight, remoteAnchor.blockHash) if (localAnchor?.tac === remoteAnchor.tac) { @@ -1726,18 +1858,28 @@ export class Engine { } const localAdmitted = await this.storage.findAdmittedTransactionsForBlock?.(topic, remoteAnchor.blockHeight, remoteAnchor.blockHash) ?? [] + // BRC-136: an empty topic at a height has k = 0 and R = 32 zero bytes, which + // the root/count check above has already enforced. There is no transaction + // to bind to a proof or to fetch, so the height is checked without a proof + // round trip; it only diverges when this node admitted something there. + if (admittedResponse.admitted.length === 0) { + if (localAdmitted.length > 0) report.status = 'diverged' + return + } const localTxids = new Set(localAdmitted.map(item => item.txid)) const missingTxids = admittedResponse.admitted .map(item => item.txid) .filter(txid => !localTxids.has(txid)) report.missingTxids.push(...missingTxids) + // The BASM root only commits to txid order. Bind claimed original indices to + // the compound path even when every remote txid is already local. + const assurance = await this.fetchBASMMissingTransactions(remote, topic, remoteAnchor, admittedResponse.admitted, missingTxids) + if (report.positionValidation !== 'encoded-offset-only') report.positionValidation = assurance if (missingTxids.length === 0) { report.status = 'diverged' return } - - await this.fetchBASMMissingTransactions(remote, topic, remoteAnchor, missingTxids) report.fetchedTxCount += missingTxids.length } @@ -1745,19 +1887,38 @@ export class Engine { remote: BASMRemote, topic: string, anchor: TopicBlockAnchor, + admitted: AdmittedTxRef[], txids: string[] - ): Promise { + ): Promise<'canonical-count' | 'encoded-offset-only'> { if (this.chainTracker === 'scripts only') { throw new Error('BASM reconciliation requires a ChainTracker capable of validating BUMP proofs') } - const proofResponse = await remote.requestCompoundMerklePath(anchor.blockHeight, txids) + // Validate the whole claimed ordered subset, including entries already local: + // the BASM root alone does not bind the peer's claimed original positions. + const proofResponse = await remote.requestCompoundMerklePath(anchor.blockHeight, admitted.map(item => item.txid)) const compoundPath = MerklePath.fromHex(proofResponse.merklePath) - for (const txid of txids) { - const valid = await compoundPath.verify(txid, this.chainTracker) - if (!valid) { - throw new Error(`Peer supplied invalid compound Merkle path for ${txid} at height ${anchor.blockHeight}`) - } + requireBASM(compoundPath.blockHeight === anchor.blockHeight, 'BASM proof height does not match its anchor') + requireBASM(compoundPath.toHex() === proofResponse.merklePath.toLowerCase(), 'BASM proof is not canonically encoded') + const proofRoot = compoundPath.computeRoot() + const proofHeader = await this.requireCanonicalBASMAnchor(anchor, proofRoot) + if (proofHeader.blockTransactionCount !== undefined) { + this.validateBASMProofPositions(compoundPath, admitted, proofHeader.blockTransactionCount) + } + for (const { txid, blockIndex } of admitted) { + const leaf = compoundPath.path[0]?.find(item => item.hash === txid) + requireBASM(leaf?.offset === blockIndex, 'BASM proof does not bind the admitted block index') + requireBASM(compoundPath.path[0].length !== 1 || compoundPath.path.length !== 1 || blockIndex === 0, 'BASM singleton proof has a nonzero block index') + requireBASM(compoundPath.computeRoot(txid) === proofRoot, 'BASM proof root does not match the admitted transaction') + } + // Inclusion is root/height, not coinbase maturity. MerklePath.verify also + // applies the 100-block spendability rule at offset 0. + const valid = await this.chainTracker.isValidRootForHeight(proofRoot, compoundPath.blockHeight) + if (!valid) { + throw new Error(`Peer supplied invalid compound Merkle path at height ${anchor.blockHeight}`) + } + if (txids.length === 0) { + return proofHeader.blockTransactionCount === undefined ? 'encoded-offset-only' : 'canonical-count' } const rawResponse = await remote.requestRawTransactions(txids) @@ -1765,18 +1926,38 @@ export class Engine { throw new Error(`Peer did not return raw transactions for txids: ${rawResponse.missing.join(',')}`) } - for (const record of rawResponse.transactions) { + const transactions = rawResponse.transactions.map(record => { const tx = Transaction.fromHex(record.rawTx) - if (tx.id('hex') !== record.txid) { + if (tx.id('hex') !== record.txid || tx.toHex() !== record.rawTx.toLowerCase()) { throw new Error(`Raw transaction txid mismatch: expected ${record.txid}, got ${tx.id('hex')}`) } - try { - tx.merklePath = compoundPath.extract([record.txid]) - } catch { - tx.merklePath = compoundPath - } - await this.submit({ beef: tx.toBEEF(), topics: [topic] }, undefined, 'historical-tx') + tx.merklePath = compoundPath.extract([record.txid]) + return tx + }) + const refreshedAnchor = (await remote.requestTopicAnchorRange(anchor.blockHeight, anchor.blockHeight)).anchors[0] + requireBASM( + refreshedAnchor?.blockHash === anchor.blockHash && + refreshedAnchor?.basmRoot === anchor.basmRoot && + refreshedAnchor?.admittedCount === anchor.admittedCount && + refreshedAnchor?.tac === anchor.tac, + 'BASM peer anchor changed before admission' + ) + const commitHeader = await this.requireCanonicalBASMAnchor(anchor, proofRoot) + requireBASM(commitHeader.blockTransactionCount === proofHeader.blockTransactionCount, 'BASM canonical block transaction count changed before admission') + // Apply in the independently checked block order, regardless of raw response order. + // Admit with 'historical-tx-no-spv': inclusion is already proven above by + // isValidRootForHeight plus the canonical-header binding. Re-running + // Transaction.verify would apply MerklePath.verify's coinbase 100-block + // spendability rule and reject an admitted coinbase from a recent block. + const transactionById = new Map(transactions.map(tx => [tx.id('hex'), tx])) + for (const txid of txids) { + const tx = transactionById.get(txid) + requireBASM(tx !== undefined, 'BASM raw response omits a requested transaction') + await this.submit({ beef: tx.toBEEF(), topics: [topic] }, undefined, 'historical-tx-no-spv') } + const finalHeader = await this.requireCanonicalBASMAnchor(anchor, proofRoot) + requireBASM(finalHeader.blockTransactionCount === proofHeader.blockTransactionCount, 'BASM canonical block transaction count changed during admission') + return proofHeader.blockTransactionCount === undefined ? 'encoded-offset-only' : 'canonical-count' } async evictUnprovenTransactions(options: { diff --git a/packages/overlays/overlay/src/__tests/BASM.conformance.test.ts b/packages/overlays/overlay/src/__tests/BASM.conformance.test.ts new file mode 100644 index 000000000..3117517a5 --- /dev/null +++ b/packages/overlays/overlay/src/__tests/BASM.conformance.test.ts @@ -0,0 +1,165 @@ +import { execFileSync } from 'node:child_process' +import { createHash } from 'node:crypto' +import { readFileSync } from 'node:fs' +import { join } from 'node:path' +import { computeBasmRoot, computeTac, BASM_ZERO_HASH } from '../BASM' +import { basmAdmitted } from '../BASMValidation' + +interface MerkleVector { + name: string + txids: string[] + root: string + admissionListValid: boolean +} + +interface TacAnchor { + blockHeight: number + blockHash: string + basmRoot: string + admittedCount: number + rootSource: string + expectedTac: string +} + +interface TacVector { + name: string + genesisHeight: number + anchors: TacAnchor[] +} + +interface ByteOrderVector { + name: string + display: string + internal: string +} + +const FIXTURE_PATH = join(__dirname, 'fixtures', 'brc136-independent.json') +const FIXTURE_SHA256 = '51983432bb561e3031fb7c983947dc8bfdcfd0ef828ef9f24093666ff4665c8a' +const fixtureBytes = readFileSync(FIXTURE_PATH) +const fixture = JSON.parse(fixtureBytes.toString('utf8')) as { + specRevision: { brc136: string; repoHead: string } + hashEncoding: string + byteOrder: ByteOrderVector[] + merkle: MerkleVector[] + tac: TacVector[] +} +const merkleVectors = fixture.merkle +const tacVectors = fixture.tac + +function displayToInternal(hash: string): Buffer { + if (!/^[0-9a-f]{64}$/.test(hash)) { + throw new Error(`expected lowercase 32-byte hex, got ${hash}`) + } + return Buffer.from(hash, 'hex').reverse() +} + +function internalToDisplay(value: Buffer): string { + if (value.length !== 32) throw new Error('expected 32 bytes') + return Buffer.from(value).reverse().toString('hex') +} + +function sha256d(value: Buffer): Buffer { + return createHash('sha256').update(createHash('sha256').update(value).digest()).digest() +} + +function opensslSha256d(value: Buffer): Buffer { + const first = execFileSync('openssl', ['dgst', '-sha256', '-binary'], { input: value }) + return execFileSync('openssl', ['dgst', '-sha256', '-binary'], { input: first }) +} + +function independentBasmRoot(txids: string[], digest: (value: Buffer) => Buffer): string { + if (txids.length === 0) return BASM_ZERO_HASH + let layer = txids.map(displayToInternal) + while (layer.length > 1) { + const next: Buffer[] = [] + for (let index = 0; index < layer.length; index += 2) { + const left = layer[index] + const right = index + 1 < layer.length ? layer[index + 1] : left + next.push(digest(Buffer.concat([left, right]))) + } + layer = next + } + return internalToDisplay(layer[0]) +} + +function independentTac( + previous: string, + blockHash: string, + root: string, + digest: (value: Buffer) => Buffer +): string { + return internalToDisplay( + digest( + Buffer.concat([ + displayToInternal(previous), + displayToInternal(blockHash), + displayToInternal(root) + ]) + ) + ) +} + +const opensslAvailable = ((): boolean => { + try { + execFileSync('openssl', ['version'], { stdio: 'ignore' }) + return true + } catch { + return false + } +})() + +describe('independent BRC-136 conformance vectors', () => { + it('pins the BRC revision and frozen fixture bytes', () => { + expect(fixture.specRevision.brc136).toBe('2733cd2950a739b3c977b95d652ff63e3773c40b') + expect(fixture.specRevision.repoHead).toBe('39a643ff148a8dcd23ec08986a8ddeb7d5713743') + expect(fixture.hashEncoding).toBe('lowercase display-order hex') + expect(createHash('sha256').update(fixtureBytes).digest('hex')).toBe(FIXTURE_SHA256) + expect(merkleVectors.map(vector => vector.name)).toEqual( + expect.arrayContaining(['even-four', 'odd-five', 'unsorted-asymmetric-byte-values']) + ) + }) + + it.each(fixture.byteOrder)('reverses display/internal bytes for %s', vector => { + expect(displayToInternal(vector.display).toString('hex')).toBe(vector.internal) + expect(internalToDisplay(Buffer.from(vector.internal, 'hex'))).toBe(vector.display) + }) + + it.each(merkleVectors)('independently hashes the ordered admitted list for %s', vector => { + const independent = independentBasmRoot(vector.txids, sha256d) + expect(independent).toBe(vector.root) + expect(computeBasmRoot(vector.txids)).toBe(independent) + if (vector.txids.length >= 2 && opensslAvailable) { + expect(independentBasmRoot(vector.txids, opensslSha256d)).toBe(independent) + } + const admitted = vector.txids.map((txid, blockIndex) => ({ txid, blockIndex })) + if (vector.admissionListValid) { + expect(basmAdmitted(admitted, admitted.length + 1)).toEqual(admitted) + } else { + expect(() => basmAdmitted(admitted, admitted.length + 1)).toThrow('unique txids') + } + }) + + it.each(tacVectors)('independently chains every TAC step for %s', vector => { + const rootsByName = new Map(merkleVectors.map(root => [root.name, root])) + let previousTac = BASM_ZERO_HASH + let expectedHeight = vector.genesisHeight + + for (const anchor of vector.anchors) { + expect(anchor.blockHeight).toBe(expectedHeight) + const source = rootsByName.get(anchor.rootSource) + expect(source).toBeDefined() + expect(anchor.basmRoot).toBe(source?.root) + expect(anchor.admittedCount).toBe(source?.txids.length) + const independent = independentTac(previousTac, anchor.blockHash, anchor.basmRoot, sha256d) + expect(independent).toBe(anchor.expectedTac) + expect(computeTac(previousTac, anchor.blockHash, anchor.basmRoot)).toBe(independent) + if (opensslAvailable) { + expect(independentTac(previousTac, anchor.blockHash, anchor.basmRoot, opensslSha256d)).toBe( + independent + ) + } + previousTac = independent + expectedHeight += 1 + } + }) +}) diff --git a/packages/overlays/overlay/src/__tests/BASMEmptyBlock.test.ts b/packages/overlays/overlay/src/__tests/BASMEmptyBlock.test.ts new file mode 100644 index 000000000..efa6f4f3a --- /dev/null +++ b/packages/overlays/overlay/src/__tests/BASMEmptyBlock.test.ts @@ -0,0 +1,93 @@ +import { BASM_ZERO_HASH, computeBasmRoot, computeTac } from '../BASM' +import type { TopicBlockAnchor } from '../BASM' +import { blockHash, blockHeight, fixture, topic } from './BASMReconciliationFixture' + +afterEach(() => jest.restoreAllMocks()) + +// BRC-136: "Empty topic at this height. If k = 0, then R = 0x00…00 (32 zero bytes)." +// A peer anchors every height, so most anchors a lagging node sees admit nothing. +describe('BASM reconciliation of heights where the topic admitted nothing', () => { + const localHeight = blockHeight - 1 + const localHash = 'cd'.repeat(32) + + function emptyHeightFixture() { + const f = fixture() + const localAnchor: TopicBlockAnchor = { + topic, + blockHeight: localHeight, + blockHash: localHash, + basmRoot: BASM_ZERO_HASH, + admittedCount: 0, + tac: computeTac(BASM_ZERO_HASH, localHash, BASM_ZERO_HASH) + } + const emptyAnchor: TopicBlockAnchor = { + topic, + blockHeight, + blockHash, + basmRoot: BASM_ZERO_HASH, + admittedCount: 0, + tac: computeTac(localAnchor.tac, blockHash, BASM_ZERO_HASH) + } + f.storage.findTopicAnchorTip.mockResolvedValue({ + topic, + blockHeight: localHeight, + tac: localAnchor.tac + }) + f.storage.findTopicBlockAnchor.mockImplementation(async (...args: unknown[]) => + args[1] === localHeight ? localAnchor : undefined + ) + f.engine.topicAnchorHeaderResolver = jest.fn(async (height: number) => ({ + blockHeight: height, + blockHash: height === localHeight ? localHash : blockHash, + merkleRoot: f.root + })) + f.responses['/requestTopicAnchorTip'] = emptyAnchor + f.responses['/requestTopicAnchorRange'] = { topic, anchors: [emptyAnchor] } + f.responses['/requestAdmittedList'] = { topic, blockHeight, blockHash, admitted: [] } + return { f, emptyAnchor } + } + + it('defines the empty BASM root as 32 zero bytes', () => { + expect(computeBasmRoot([])).toBe(BASM_ZERO_HASH) + }) + + it('checks an empty height without requesting a proof or raw transactions', async () => { + const { f } = emptyHeightFixture() + const [report] = await f.engine.startBASMSync() + expect(report.message).toBeUndefined() + expect(report.status).not.toBe('error') + expect(report.status).not.toBe('diverged') + expect(report.checkedHeights).toEqual([blockHeight]) + expect(report.missingTxids).toEqual([]) + const paths = f.requests.map(request => request.path) + expect(paths).not.toContain('/requestCompoundMerklePath') + expect(paths).not.toContain('/requestRawTransactions') + expect(f.submit).not.toHaveBeenCalled() + }) + + it('reports divergence when the peer claims an empty height that is populated locally', async () => { + const { f } = emptyHeightFixture() + f.storage.findAdmittedTransactionsForBlock.mockResolvedValue([ + { txid: f.ids[1], blockIndex: 1 } + ]) + const [report] = await f.engine.startBASMSync() + expect(report.status).toBe('diverged') + expect(f.requests.map(request => request.path)).not.toContain('/requestCompoundMerklePath') + expect(f.submit).not.toHaveBeenCalled() + }) + + it('still rejects a zero-count anchor whose root is not the zero hash', async () => { + const { f, emptyAnchor } = emptyHeightFixture() + const forged = { ...emptyAnchor, basmRoot: 'ee'.repeat(32) } + forged.tac = computeTac( + computeTac(BASM_ZERO_HASH, localHash, BASM_ZERO_HASH), + blockHash, + forged.basmRoot + ) + f.responses['/requestTopicAnchorTip'] = forged + f.responses['/requestTopicAnchorRange'] = { topic, anchors: [forged] } + const [report] = await f.engine.startBASMSync() + expect(report.status).toBe('error') + expect(report.message).toContain('inconsistent with its anchor') + }) +}) diff --git a/packages/overlays/overlay/src/__tests/BASMGoInterop.test.ts b/packages/overlays/overlay/src/__tests/BASMGoInterop.test.ts new file mode 100644 index 000000000..65ac52073 --- /dev/null +++ b/packages/overlays/overlay/src/__tests/BASMGoInterop.test.ts @@ -0,0 +1,225 @@ +import { spawn, spawnSync, type ChildProcessWithoutNullStreams } from 'node:child_process' +import { createHash } from 'node:crypto' +import { copyFileSync, existsSync, mkdtempSync, writeFileSync } from 'node:fs' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { MerklePath, Transaction } from '@bsv/sdk' +import { BASMRemote } from '../BASMRemote' +import { computeBasmRoot, computeTac, BASM_ZERO_HASH } from '../BASM' + +const TOPIC = 'tm_interop' +const GENESIS_TXID = '4a5e1e4baab89f3a32518a88c31bc87f618f76673e2cc77ab2127b7afdeda33b' +const GENESIS_HEX = + '01000000010000000000000000000000000000000000000000000000000000000000000000ffffffff4d04ffff001d0104455468652054696d65732030332f4a616e2f32303039204368616e63656c6c6f72206f6e206272696e6b206f66207365636f6e64206261696c6f757420666f722062616e6b73ffffffff0100f2052a01000000434104678afdb0fe5548271967f1a67130b7105cd6a828e03909a67962e0ea1f61deb649f6bc3f4cef38c4f35504e51ec112de5c384df7ba0b8d578a4c702b6bf11d5fac00000000' +const GENESIS_BLOCK = '000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f' +const MISSING = 'ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff' +const GO_SOURCE = join(__dirname, 'fixtures', 'basm-go-read-server.go') + +function displayToInternal(hash: string): Buffer { + return Buffer.from(hash, 'hex').reverse() +} + +function independentTac(previous: string, blockHash: string, root: string): string { + const input = Buffer.concat([ + displayToInternal(previous), + displayToInternal(blockHash), + displayToInternal(root) + ]) + const first = createHash('sha256').update(input).digest() + return Buffer.from(createHash('sha256').update(first).digest()).reverse().toString('hex') +} + +const GO_CHECKOUT_MARKERS = [ + 'pkg/server/server_http_basm_interop_test.go', + 'pkg/core/engine/basm-read-service.go' +] + +/** + * Resolves the go-overlay-services checkout to run interop against. The only + * source is BASM_GO_OVERLAY_SERVICES: no workstation path is assumed, so this + * suite never silently depends on one machine's layout. When the variable is + * set but does not point at a usable checkout we fail rather than skip, so a + * job that claims Go interop cannot pass without running it. + */ +function resolveGoWorktree(): string | undefined { + const candidate = process.env.BASM_GO_OVERLAY_SERVICES + if (candidate === undefined || candidate === '') return undefined + const missing = GO_CHECKOUT_MARKERS.filter(marker => !existsSync(join(candidate, marker))) + if (missing.length > 0) { + throw new Error( + `BASM_GO_OVERLAY_SERVICES is set to "${candidate}" but that is not a go-overlay-services checkout (missing: ${missing.join(', ')})` + ) + } + return candidate +} + +async function waitForUrls( + child: ChildProcessWithoutNullStreams, + timeoutMs: number +): Promise<{ ready: string; unsupported: string }> { + let stdout = '' + let stderr = '' + return await new Promise((resolve, reject) => { + let settled = false + const finish = (error?: Error, urls?: { ready: string; unsupported: string }): void => { + if (settled) return + settled = true + clearTimeout(timer) + child.stdout.off('data', onStdout) + child.stderr.off('data', onStderr) + child.off('error', onError) + child.off('exit', onExit) + if (error !== undefined) reject(error) + else resolve(urls as { ready: string; unsupported: string }) + } + const timer = setTimeout(() => { + finish( + new Error( + `Go BASM host did not become ready: stdout=${stdout.trim()} stderr=${stderr.trim()}` + ) + ) + }, timeoutMs) + const onStdout = (chunk: Buffer | string): void => { + stdout += String(chunk) + const ready = stdout.match(/^READY (http:\/\/127\.0\.0\.1:\d+)/m)?.[1] + const unsupported = stdout.match(/^UNSUPPORTED (http:\/\/127\.0\.0\.1:\d+)/m)?.[1] + if (ready !== undefined && unsupported !== undefined) { + finish(undefined, { ready, unsupported }) + } + } + const onStderr = (chunk: Buffer | string): void => { + stderr += String(chunk) + } + const onError = (error: Error): void => { + finish(error) + } + const onExit = (code: number | null, signal: NodeJS.Signals | null): void => { + finish( + new Error( + `Go BASM host exited code=${String(code)} signal=${String(signal)} stderr=${stderr.trim()}` + ) + ) + } + child.stdout.on('data', onStdout) + child.stderr.on('data', onStderr) + child.on('error', onError) + child.on('exit', onExit) + }) +} + +function startGoHost(goRoot: string): ChildProcessWithoutNullStreams { + const dir = mkdtempSync(join(tmpdir(), 'basm-go-interop-')) + copyFileSync(GO_SOURCE, join(dir, 'main.go')) + writeFileSync( + join(dir, 'go.mod'), + [ + 'module ts-basm-go-interop', + '', + 'go 1.26.0', + '', + 'require github.com/bsv-blockchain/go-overlay-services v0.0.0', + '', + `replace github.com/bsv-blockchain/go-overlay-services => ${goRoot}`, + '' + ].join('\n') + ) + const env = { ...process.env, GOTOOLCHAIN: 'local' } + const tidy = spawnSync('go', ['mod', 'tidy'], { cwd: dir, encoding: 'utf8', env }) + if (tidy.status !== 0) { + throw new Error(`go mod tidy failed: ${tidy.stderr || tidy.stdout}`) + } + const build = spawnSync('go', ['build', '-o', 'basm-host', '.'], { + cwd: dir, + encoding: 'utf8', + env + }) + if (build.status !== 0) { + throw new Error(`go build failed: ${build.stderr || build.stdout}`) + } + const child = spawn(join(dir, 'basm-host'), [], { + cwd: dir, + env, + stdio: ['ignore', 'pipe', 'pipe'] + }) + child.unref() + return child +} + +const goRoot = resolveGoWorktree() +const describeInterop = goRoot === undefined ? describe.skip : describe +const suiteName = + goRoot === undefined + ? 'BASMRemote localhost interop against Go read/serving (skipped: set BASM_GO_OVERLAY_SERVICES to a go-overlay-services checkout to run it)' + : 'BASMRemote localhost interop against Go read/serving' + +describeInterop(suiteName, () => { + let child: ChildProcessWithoutNullStreams + let readyURL = '' + let unsupportedURL = '' + + beforeAll(async () => { + child = startGoHost(goRoot as string) + const urls = await waitForUrls(child, 120000) + readyURL = urls.ready + unsupportedURL = urls.unsupported + }, 130000) + + afterAll(() => { + child?.kill('SIGTERM') + }) + + it('exchanges the five BRC-136 read methods over credential-free HTTP', async () => { + const remote = new BASMRemote(readyURL, TOPIC) + const tip = await remote.requestTopicAnchorTip() + const range = await remote.requestTopicAnchorRange(0, 0) + const admitted = await remote.requestAdmittedList(0, GENESIS_BLOCK) + const proof = await remote.requestCompoundMerklePath(0, [GENESIS_TXID]) + const raw = await remote.requestRawTransactions([GENESIS_TXID, MISSING]) + + expect(tip.topic).toBe(TOPIC) + expect(tip.blockHeight).toBe(0) + expect(tip.blockHash).toBe(GENESIS_BLOCK) + expect(tip.basmRoot).toBe(GENESIS_TXID) + expect(tip.admittedCount).toBe(1) + expect(computeBasmRoot([{ txid: GENESIS_TXID, blockIndex: 0 }])).toBe(GENESIS_TXID) + expect(independentTac(BASM_ZERO_HASH, GENESIS_BLOCK, GENESIS_TXID)).toBe(tip.tac) + expect(computeTac(BASM_ZERO_HASH, GENESIS_BLOCK, GENESIS_TXID)).toBe(tip.tac) + + expect(range.anchors).toHaveLength(1) + expect(range.anchors[0]?.tac).toBe(tip.tac) + expect(range.anchors[0]?.blockHash).toBe(GENESIS_BLOCK) + expect(admitted.admitted).toEqual([{ txid: GENESIS_TXID, blockIndex: 0 }]) + + const parsedProof = MerklePath.fromHex(proof.merklePath) + expect(parsedProof.blockHeight).toBe(0) + expect(parsedProof.path[0]?.[0]?.hash).toBe(GENESIS_TXID) + expect(parsedProof.path[0]?.[0]?.offset).toBe(0) + expect(parsedProof.computeRoot(GENESIS_TXID)).toBe(GENESIS_TXID) + expect(parsedProof.toHex()).toBe(proof.merklePath.toLowerCase()) + + const parsedRaw = Transaction.fromHex(raw.transactions[0].rawTx) + expect(raw.transactions[0].txid).toBe(GENESIS_TXID) + expect(raw.transactions[0].rawTx).toBe(GENESIS_HEX) + expect(parsedRaw.id('hex')).toBe(GENESIS_TXID) + expect(raw.missing).toEqual([MISSING]) + + const cors = await fetch(new URL('/requestTopicAnchorTip', readyURL), { + method: 'POST', + headers: { + Origin: 'https://unknown-wallet.example', + Accept: 'application/json', + 'Content-Type': 'application/json', + 'x-bsv-topic': TOPIC + }, + body: '{}' + }) + expect(cors.ok).toBe(true) + expect(cors.headers.get('access-control-allow-origin')).toBe('*') + + await expect( + new BASMRemote(unsupportedURL, TOPIC).requestTopicAnchorTip() + ).rejects.toMatchObject({ + code: 'BASM_UNSUPPORTED' + }) + }, 30000) +}) diff --git a/packages/overlays/overlay/src/__tests/BASMGoInteropGating.test.ts b/packages/overlays/overlay/src/__tests/BASMGoInteropGating.test.ts new file mode 100644 index 000000000..e5746a32c --- /dev/null +++ b/packages/overlays/overlay/src/__tests/BASMGoInteropGating.test.ts @@ -0,0 +1,35 @@ +import { readFileSync } from 'node:fs' +import { join } from 'node:path' + +const INTEROP_SOURCE = join(__dirname, 'BASMGoInterop.test.ts') + +/** + * The Go read/serving interop suite can only run where a go-overlay-services + * checkout exists, so it is gated. The gate itself has to be asserted from the + * source text: a behavioural test would agree with a workstation-path fallback + * on every machine that lacks that path, which is exactly the environment (CI) + * where such a fallback hides the fact that the suite never runs. + */ +describe('BASM Go interop gating', () => { + const source = readFileSync(INTEROP_SOURCE, 'utf8') + + it('does not hard-code a developer workstation checkout path', () => { + expect(source).not.toMatch(/\/(?:Users|home)\/[A-Za-z0-9._-]+\//) + }) + + it('resolves the Go checkout only from BASM_GO_OVERLAY_SERVICES', () => { + const resolver = /\nfunction resolveGoWorktree\(\)[\s\S]*?\n}\n/.exec(source)?.[0] + expect(resolver).toBeDefined() + expect(resolver).toContain('process.env.BASM_GO_OVERLAY_SERVICES') + expect(resolver?.match(/process\.env\.[A-Za-z_]+/g)).toEqual([ + 'process.env.BASM_GO_OVERLAY_SERVICES' + ]) + }) + + it('names the environment variable in the skipped suite title', () => { + const skipTitle = /goRoot === undefined\s*\?\s*'([^']*)'/.exec(source)?.[1] + expect(skipTitle).toBeDefined() + expect(skipTitle).toContain('skipped') + expect(skipTitle).toContain('BASM_GO_OVERLAY_SERVICES') + }) +}) diff --git a/packages/overlays/overlay/src/__tests/BASMProofBinding.test.ts b/packages/overlays/overlay/src/__tests/BASMProofBinding.test.ts new file mode 100644 index 000000000..5d1bcbb46 --- /dev/null +++ b/packages/overlays/overlay/src/__tests/BASMProofBinding.test.ts @@ -0,0 +1,193 @@ +import { MerklePath } from '@bsv/sdk' +import { BASM_ZERO_HASH, computeTac } from '../BASM' +import type { TopicBlockAnchor } from '../BASM' +import { blockHash, blockHeight, fixture, hashPair, topic } from './BASMReconciliationFixture' +import type { BASMFixture } from './BASMReconciliationFixture' + +afterEach(() => jest.restoreAllMocks()) + +/** + * Re-points the fake peer at a single-block anchor whose admitted list is + * proven by `path`, and tells the chain tracker which root is canonical. + */ +function serveBlock( + f: BASMFixture, + options: { + admitted: Array<{ txid: string; blockIndex: number }> + basmRoot: string + path: MerklePath + merkleRoot: string + rawTxIndexes: number[] + blockTransactionCount?: number + } +): TopicBlockAnchor { + const anchor: TopicBlockAnchor = { + topic, + blockHeight, + blockHash, + basmRoot: options.basmRoot, + admittedCount: options.admitted.length, + tac: computeTac(BASM_ZERO_HASH, blockHash, options.basmRoot) + } + f.responses['/requestTopicAnchorTip'] = anchor + f.responses['/requestTopicAnchorRange'] = { topic, anchors: [anchor] } + f.responses['/requestAdmittedList'] = { + topic, + blockHeight, + blockHash, + admitted: options.admitted + } + f.responses['/requestCompoundMerklePath'] = { + topic, + blockHeight, + txids: options.admitted.map(item => item.txid), + merklePath: options.path.toHex() + } + f.responses['/requestRawTransactions'] = { + transactions: options.rawTxIndexes.map(index => ({ + txid: f.ids[index], + rawTx: f.transactions[index].toHex() + })), + missing: [] + } + f.tracker.isValidRootForHeight.mockImplementation( + async (candidate: string, height: number) => + candidate === options.merkleRoot && height === blockHeight + ) + f.engine.topicAnchorHeaderResolver = async height => ({ + blockHeight: height, + blockHash, + merkleRoot: options.merkleRoot, + blockTransactionCount: options.blockTransactionCount + }) + return anchor +} + +/** A three-transaction block: offset 3 is the phantom duplicate of offset 2. */ +function oddWidthBlock(f: BASMFixture): { path: MerklePath; merkleRoot: string } { + return { + path: new MerklePath(blockHeight, [ + [ + { offset: 2, hash: f.ids[2], txid: true }, + { offset: 3, duplicate: true } + ], + [{ offset: 0, hash: hashPair(f.ids[0], f.ids[1]) }] + ]), + merkleRoot: hashPair(hashPair(f.ids[0], f.ids[1]), hashPair(f.ids[2], f.ids[2])) + } +} + +describe('BASM canonical proof position validation', () => { + it('accepts a duplicate node at the right edge of an odd-width block', async () => { + const f = fixture() + const block = oddWidthBlock(f) + serveBlock(f, { + admitted: [{ txid: f.ids[2], blockIndex: 2 }], + basmRoot: f.ids[2], + path: block.path, + merkleRoot: block.merkleRoot, + rawTxIndexes: [2], + blockTransactionCount: 3 + }) + const [report] = await f.engine.startBASMSync() + expect(report.status).toBe('advanced') + expect(report.fetchedTxCount).toBe(1) + expect(report.positionValidation).toBe('canonical-count') + expect(f.submit).toHaveBeenCalledTimes(1) + expect(f.tracker.isValidRootForHeight).toHaveBeenCalledWith(block.merkleRoot, blockHeight) + }) + + it('rejects the same duplicate node when the canonical block width is even', async () => { + const f = fixture() + const block = oddWidthBlock(f) + serveBlock(f, { + admitted: [{ txid: f.ids[2], blockIndex: 2 }], + basmRoot: f.ids[2], + path: block.path, + merkleRoot: block.merkleRoot, + rawTxIndexes: [2], + // Four transactions leaves no odd right edge, so offset 3 is a real + // position the peer would have to prove rather than duplicate. + blockTransactionCount: 4 + }) + const [report] = await f.engine.startBASMSync() + expect(report.status).toBe('error') + expect(report.errorCode).toBe('BASM_INVALID_RESPONSE') + expect(report.message).toBe('BASM proof node is outside canonical block positions') + expect(f.requests.some(request => request.path === '/requestRawTransactions')).toBe(false) + expect(f.submit).not.toHaveBeenCalled() + }) + + it('rejects a compound path that silently omits one admitted transaction', async () => { + const f = fixture() + // Claims both txids in its response envelope, but the BUMP only carries a + // leaf for the first one. + f.responses['/requestCompoundMerklePath'] = { + topic, + blockHeight, + txids: f.admitted.map(item => item.txid), + merklePath: new MerklePath(blockHeight, [ + [ + { offset: 0, hash: f.ids[0] }, + { offset: 1, hash: f.ids[1], txid: true } + ], + [{ offset: 1, hash: hashPair(f.ids[2], f.ids[3]) }] + ]).toHex() + } + const [report] = await f.engine.startBASMSync() + expect(report.status).toBe('error') + expect(report.errorCode).toBe('BASM_INVALID_RESPONSE') + expect(report.message).toBe('BASM proof does not bind the admitted block index') + expect(f.requests.some(request => request.path === '/requestRawTransactions')).toBe(false) + expect(f.submit).not.toHaveBeenCalled() + }) + + it('rejects a well-formed proof whose root is not canonical for the height', async () => { + const f = fixture() + f.tracker.isValidRootForHeight.mockResolvedValue(false) + const [report] = await f.engine.startBASMSync() + expect(report.status).toBe('error') + expect(report.errorCode).toBeUndefined() + expect(report.message).toBe( + `Peer supplied invalid compound Merkle path at height ${blockHeight}` + ) + expect(f.tracker.isValidRootForHeight).toHaveBeenCalledWith(f.root, blockHeight) + expect(f.requests.some(request => request.path === '/requestRawTransactions')).toBe(false) + expect(f.submit).not.toHaveBeenCalled() + }) + + it('records canonical-count assurance even when nothing needs fetching', async () => { + const f = fixture() + f.storage.findAdmittedTransactionsForBlock.mockResolvedValue([...f.admitted]) + f.engine.topicAnchorHeaderResolver = async height => ({ + blockHeight: height, + blockHash, + merkleRoot: f.root, + blockTransactionCount: 4 + }) + const [report] = await f.engine.startBASMSync() + expect(report.status).toBe('diverged') + expect(report.positionValidation).toBe('canonical-count') + expect(report.missingTxids).toEqual([]) + expect(f.requests.some(request => request.path === '/requestCompoundMerklePath')).toBe(true) + expect(f.requests.some(request => request.path === '/requestRawTransactions')).toBe(false) + expect(f.submit).not.toHaveBeenCalled() + }) + + it('refuses to admit when the anchor disappears from the peer before admission', async () => { + const f = fixture() + const originalFetch = f.fetchMock.getMockImplementation()! + f.fetchMock.mockImplementation(async (url, init) => { + const response = await originalFetch(url, init) + if (new URL(String(url)).pathname === '/requestRawTransactions') { + f.responses['/requestTopicAnchorRange'] = { topic, anchors: [] } + } + return response + }) + const [report] = await f.engine.startBASMSync() + expect(report.status).toBe('error') + expect(report.errorCode).toBe('BASM_INVALID_RESPONSE') + expect(report.message).toBe('BASM peer anchor changed before admission') + expect(f.submit).not.toHaveBeenCalled() + }) +}) diff --git a/packages/overlays/overlay/src/__tests/BASMProviderGuards.test.ts b/packages/overlays/overlay/src/__tests/BASMProviderGuards.test.ts new file mode 100644 index 000000000..14aebff6d --- /dev/null +++ b/packages/overlays/overlay/src/__tests/BASMProviderGuards.test.ts @@ -0,0 +1,77 @@ +import { Engine } from '../Engine' +import type { Storage } from '../storage/Storage' + +const topic = 'tm_basm_guards' +const txid = 'ab'.repeat(32) + +/** A storage backend that implements none of the optional BASM read methods. */ +const engineWithout = (methods: Record = {}): Engine => + new Engine({}, {}, methods as unknown as Storage, 'scripts only') + +const expectUnsupported = async (operation: Promise, message: string): Promise => { + await expect(operation).rejects.toThrow(TypeError) + await expect(operation).rejects.toThrow(message) + await expect(operation).rejects.toMatchObject({ code: 'BASM_UNSUPPORTED' }) +} + +describe('BASM provider capability guards', () => { + it('refuses to serve a topic anchor tip without findTopicAnchorTip', async () => { + await expectUnsupported( + engineWithout().provideTopicAnchorTip(topic), + 'Storage does not support BASM topic anchor tips' + ) + }) + + it('refuses to serve a topic anchor range without findTopicBlockAnchors', async () => { + await expectUnsupported( + engineWithout().provideTopicAnchorRange(topic, 0, 0), + 'Storage does not support BASM topic anchor ranges' + ) + }) + + it('refuses to serve an admitted list without findAdmittedTransactionsForBlock', async () => { + await expectUnsupported( + engineWithout().provideAdmittedList(topic, 100), + 'Storage does not support BASM admitted lists' + ) + }) + + it('refuses to serve a compound Merkle path without findTransactionMerklePaths', async () => { + await expectUnsupported( + engineWithout().provideCompoundMerklePath(topic, 100, [txid]), + 'Storage does not support direct Merkle path lookup' + ) + }) + + it('refuses to serve raw transactions without findRawTransactions', async () => { + await expectUnsupported( + engineWithout().provideRawTransactions([txid]), + 'Storage does not support raw transaction lookup' + ) + }) + + it('checks capability before the empty-txid guard on compound Merkle paths', async () => { + // Order matters: an unsupported backend must report BASM_UNSUPPORTED rather + // than the argument-shape error a caller could mistake for a bad request. + await expectUnsupported( + engineWithout().provideCompoundMerklePath(topic, 100, []), + 'Storage does not support direct Merkle path lookup' + ) + }) + + it('still serves a tip and a range once the backend supports them', async () => { + const engine = engineWithout({ + findTopicAnchorTip: jest.fn(async () => undefined), + findTopicBlockAnchors: jest.fn(async () => []) + }) + await expect(engine.provideTopicAnchorTip(topic)).resolves.toEqual({ + topic, + blockHeight: -1, + tac: '00'.repeat(32) + }) + await expect(engine.provideTopicAnchorRange(topic, 5, 7)).resolves.toEqual({ + topic, + anchors: [] + }) + }) +}) diff --git a/packages/overlays/overlay/src/__tests/BASMRangePrefix.test.ts b/packages/overlays/overlay/src/__tests/BASMRangePrefix.test.ts new file mode 100644 index 000000000..a3b270ebf --- /dev/null +++ b/packages/overlays/overlay/src/__tests/BASMRangePrefix.test.ts @@ -0,0 +1,249 @@ +import { LockingScript, MerklePath, Transaction } from '@bsv/sdk' +import { Engine } from '../Engine' +import { BASM_ZERO_HASH, computeBasmRoot, computeTac } from '../BASM' +import type { AdmittedTxRef, TopicAnchorHeader, TopicAnchorTip, TopicBlockAnchor } from '../BASM' +import type { Storage } from '../storage/Storage' + +const topic = 'tm_basm_range' +const endpoint = 'https://peer.example' + +/** A distinct, well-formed 32-byte block hash per height. */ +const hashAt = (height: number): string => + `${height.toString(16).padStart(8, '0')}${'bb'.repeat(28)}` + +interface RangeBody { + fromHeight: number + toHeight: number +} + +/** + * A fake BASM peer plus a local overlay whose stored anchors, canonical + * headers and chain tracker are supplied per test. + */ +function harness(options: { + localTip?: TopicAnchorTip + localAnchors?: TopicBlockAnchor[] + header?: (height: number) => TopicAnchorHeader + isCanonicalRoot?: (root: string, height: number) => boolean + handle: (path: string, body: Record) => unknown +}) { + const localAnchors = new Map( + (options.localAnchors ?? []).map(anchor => [anchor.blockHeight, anchor]) + ) + const storage = { + findTopicAnchorTip: jest.fn(async (topicName: string): Promise => + topicName === topic ? options.localTip : undefined + ), + findTopicBlockAnchor: jest.fn( + async (topicName: string, height: number): Promise => + topicName === topic ? localAnchors.get(height) : undefined + ), + findAdmittedTransactionsForBlock: jest.fn(async (): Promise => []) + } + const tracker = { + currentHeight: jest.fn(async () => 5000), + isValidRootForHeight: jest.fn( + async (root: string, height: number) => options.isCanonicalRoot?.(root, height) ?? false + ) + } + const engine = new Engine({}, {}, storage as unknown as Storage, tracker) + engine.syncConfiguration = { [topic]: [endpoint] } + engine.topicAnchorHeaderResolver = jest.fn( + async (height: number) => + options.header?.(height) ?? { blockHeight: height, blockHash: hashAt(height) } + ) + engine.logger = { ...console, error: jest.fn() } + const submit = jest.spyOn(engine, 'submit').mockResolvedValue({}) + const requests: Array<{ path: string; body: Record }> = [] + jest.spyOn(globalThis, 'fetch').mockImplementation(async (url, init) => { + const path = new URL(String(url)).pathname + const body = JSON.parse(String(init?.body)) as Record + requests.push({ path, body }) + return Response.json(options.handle(path, body)) + }) + return { engine, storage, tracker, submit, requests } +} + +/** A contiguous run of zero-admission anchors chained onto `startTac`. */ +function chain(fromHeight: number, count: number, startTac: string): TopicBlockAnchor[] { + const anchors: TopicBlockAnchor[] = [] + let tac = startTac + for (let index = 0; index < count; index++) { + const blockHeight = fromHeight + index + tac = computeTac(tac, hashAt(blockHeight), BASM_ZERO_HASH) + anchors.push({ + topic, + blockHeight, + blockHash: hashAt(blockHeight), + basmRoot: BASM_ZERO_HASH, + admittedCount: 0, + tac + }) + } + return anchors +} + +const inRange = ( + anchors: TopicBlockAnchor[], + body: Record +): TopicBlockAnchor[] => { + const { fromHeight, toHeight } = body as unknown as RangeBody + return anchors.filter( + anchor => anchor.blockHeight >= fromHeight && anchor.blockHeight <= toHeight + ) +} + +afterEach(() => jest.restoreAllMocks()) + +describe('BASM forward-page prefix validation', () => { + it('rejects a page that stops short of the height it was asked for', async () => { + const [tip] = chain(1200, 1, BASM_ZERO_HASH) + const served = [...chain(201, 5, BASM_ZERO_HASH), tip] + const h = harness({ + handle: (path, body) => { + if (path === '/requestTopicAnchorTip') return tip + // A peer that silently caps every page at five anchors. + return { topic, anchors: inRange(served, body).slice(0, 5) } + } + }) + const [report] = await h.engine.startBASMSync() + expect(report.status).toBe('error') + expect(report.errorCode).toBe('BASM_INVALID_RESPONSE') + expect(report.message).toBe('BASM range omits its requested target') + // The page starts one page below the remote tip, not at genesis. + expect(h.requests.at(-1)).toEqual({ + path: '/requestTopicAnchorRange', + body: { fromHeight: 201, toHeight: 1200 } + }) + expect(h.submit).not.toHaveBeenCalled() + }) + + it('rejects a page that skips the height immediately after the local tip', async () => { + const localTac = computeTac(BASM_ZERO_HASH, hashAt(100), BASM_ZERO_HASH) + const [, tip] = chain(101, 2, localTac) + const h = harness({ + localTip: { topic, blockHeight: 100, tac: localTac }, + handle: (path, body) => { + if (path === '/requestTopicAnchorTip') return tip + // Height 101 is withheld, so the prefix cannot be chained. + return { topic, anchors: inRange([tip], body) } + } + }) + const [report] = await h.engine.startBASMSync() + expect(report.status).toBe('error') + expect(report.errorCode).toBe('BASM_INVALID_RESPONSE') + expect(report.message).toBe('BASM range omits its next height') + expect(h.submit).not.toHaveBeenCalled() + }) + + it('advances one capped page at a time when the peer is many pages ahead', async () => { + const localTac = computeTac(BASM_ZERO_HASH, hashAt(0), BASM_ZERO_HASH) + const page = chain(1, 1000, localTac) + const [tip] = chain(2000, 1, BASM_ZERO_HASH) + const h = harness({ + localTip: { topic, blockHeight: 0, tac: localTac }, + // Every anchor in the page is already stored locally with the same TAC. + localAnchors: page, + handle: (path, body) => { + if (path === '/requestTopicAnchorTip') return tip + return { topic, anchors: inRange([...page, tip], body) } + } + }) + const [report] = await h.engine.startBASMSync() + expect(report.status).toBe('advanced') + expect(report.checkedHeights).toHaveLength(1000) + expect(report.checkedHeights[0]).toBe(1) + expect(report.checkedHeights.at(-1)).toBe(1000) + expect(report.fetchedTxCount).toBe(0) + expect(report.missingTxids).toEqual([]) + // A page that ends below the remote tip is not required to reproduce it, + // and anchors already held locally are never re-downloaded. + expect( + h.requests.filter(request => request.path === '/requestTopicAnchorRange').map(r => r.body) + ).toEqual([ + { fromHeight: 2000, toHeight: 2000 }, + { fromHeight: 1, toHeight: 1000 } + ]) + expect(h.requests.some(request => request.path === '/requestAdmittedList')).toBe(false) + expect(h.submit).not.toHaveBeenCalled() + }) +}) + +describe('BASM position-assurance reporting across a page', () => { + it('keeps the weakest assurance when only some blocks bind a canonical count', async () => { + const transactions = [10, 20].map( + satoshis => + new Transaction(1, [], [{ satoshis, lockingScript: LockingScript.fromASM('OP_TRUE') }], 0) + ) + const ids = transactions.map(tx => tx.id('hex')) + const localTac = computeTac(BASM_ZERO_HASH, hashAt(99), BASM_ZERO_HASH) + const anchors: TopicBlockAnchor[] = [] + let tac = localTac + ids.forEach((txid, index) => { + const blockHeight = 100 + index + const basmRoot = computeBasmRoot([{ txid, blockIndex: 0 }]) + tac = computeTac(tac, hashAt(blockHeight), basmRoot) + anchors.push({ + topic, + blockHeight, + blockHash: hashAt(blockHeight), + basmRoot, + admittedCount: 1, + tac + }) + }) + const txidAt = (height: number): string => ids[height - 100] + const rawByTxid = new Map(ids.map((txid, index) => [txid, transactions[index].toHex()])) + const h = harness({ + localTip: { topic, blockHeight: 99, tac: localTac }, + header: height => ({ + blockHeight: height, + blockHash: hashAt(height), + merkleRoot: txidAt(height), + // Only the second block's header carries a trusted transaction count. + ...(height === 101 ? { blockTransactionCount: 1 } : {}) + }), + isCanonicalRoot: (root, height) => root === txidAt(height), + handle: (path, body) => { + if (path === '/requestTopicAnchorTip') return anchors[1] + if (path === '/requestTopicAnchorRange') return { topic, anchors: inRange(anchors, body) } + const { blockHeight, txids } = body as unknown as { + blockHeight: number + txids: string[] + } + if (path === '/requestAdmittedList') { + return { + topic, + blockHeight, + blockHash: hashAt(blockHeight), + admitted: [{ txid: txidAt(blockHeight), blockIndex: 0 }] + } + } + if (path === '/requestCompoundMerklePath') { + return { + topic, + blockHeight, + txids, + merklePath: new MerklePath(blockHeight, [ + [{ offset: 0, hash: txidAt(blockHeight), txid: true }] + ]).toHex() + } + } + return { + transactions: txids.map(txid => ({ txid, rawTx: rawByTxid.get(txid) })), + missing: [] + } + } + }) + const [report] = await h.engine.startBASMSync() + expect(report.status).toBe('advanced') + expect(report.checkedHeights).toEqual([100, 101]) + expect(report.fetchedTxCount).toBe(2) + // Height 100 could only be checked against encoded offsets, so the report + // must not claim the stronger canonical-count assurance of height 101. + expect(report.positionValidation).toBe('encoded-offset-only') + expect( + h.submit.mock.calls.map(([tagged]) => Transaction.fromBEEF(tagged.beef).id('hex')) + ).toEqual(ids) + }) +}) diff --git a/packages/overlays/overlay/src/__tests/BASMReconciliation.test.ts b/packages/overlays/overlay/src/__tests/BASMReconciliation.test.ts new file mode 100644 index 000000000..9cb71b7f9 --- /dev/null +++ b/packages/overlays/overlay/src/__tests/BASMReconciliation.test.ts @@ -0,0 +1,440 @@ +import { LockingScript, MerklePath, Transaction } from '@bsv/sdk' +import { BASM_ZERO_HASH, computeBasmRoot, computeTac } from '../BASM' +import type { TopicBlockAnchor } from '../BASM' +import { blockHash, blockHeight, fixture, hashPair, topic } from './BASMReconciliationFixture' + +afterEach(() => jest.restoreAllMocks()) + +describe('BASM reconciliation evidence binding', () => { + it('verifies a multi-level subset with original index gaps and submits historically in block order', async () => { + const f = fixture() + const [report] = await f.engine.startBASMSync() + expect(report.status).toBe('advanced') + expect(report.fetchedTxCount).toBe(2) + expect(report.positionValidation).toBe('encoded-offset-only') + expect(f.submit).toHaveBeenCalledTimes(2) + expect( + f.submit.mock.calls.map(([tagged]) => Transaction.fromBEEF(tagged.beef).id('hex')) + ).toEqual([f.ids[1], f.ids[3]]) + for (const [, callback, mode] of f.submit.mock.calls) { + expect(callback).toBeUndefined() + expect(mode).toBe('historical-tx-no-spv') + } + expect(f.tracker.isValidRootForHeight).toHaveBeenCalledTimes(1) + expect(f.tracker.isValidRootForHeight).toHaveBeenCalledWith(f.root, blockHeight) + }) + + it('reports stronger position validation only when a trusted count is bound to the canonical header', async () => { + const f = fixture() + f.engine.topicAnchorHeaderResolver = async height => ({ + blockHeight: height, + blockHash, + blockTransactionCount: 4 + }) + const [report] = await f.engine.startBASMSync() + expect(report.status).toBe('advanced') + expect(report.positionValidation).toBe('canonical-count') + expect(f.submit).toHaveBeenCalledTimes(2) + }) + + it.each([0, -1, 1, 1.5, 3, Number.MAX_SAFE_INTEGER + 1, Number.NaN])( + 'rejects inconsistent canonical full-block count %s', + async blockTransactionCount => { + const f = fixture() + f.engine.topicAnchorHeaderResolver = async height => ({ + blockHeight: height, + blockHash, + blockTransactionCount + }) + const [report] = await f.engine.startBASMSync() + expect(report.status).toBe('error') + expect(f.submit).not.toHaveBeenCalled() + } + ) + + it('rejects an admitted phantom duplicate at the right edge of an odd-width block', async () => { + const f = fixture() + const duplicated = [f.ids[0], f.ids[1], f.ids[2], f.ids[2]] + const root = hashPair( + hashPair(duplicated[0], duplicated[1]), + hashPair(duplicated[2], duplicated[3]) + ) + const proof = new MerklePath(blockHeight, [ + duplicated.map((hash, offset) => ({ hash, offset, txid: true })), + [] + ]) + const admitted = [{ txid: f.ids[2], blockIndex: 3 }] + const anchor = { + ...f.anchor, + basmRoot: f.ids[2], + admittedCount: 1, + tac: computeTac(BASM_ZERO_HASH, blockHash, f.ids[2]) + } + f.responses['/requestTopicAnchorTip'] = anchor + f.responses['/requestTopicAnchorRange'] = { topic, anchors: [anchor] } + f.responses['/requestAdmittedList'] = { topic, blockHeight, blockHash, admitted } + f.responses['/requestCompoundMerklePath'] = { + topic, + blockHeight, + txids: [f.ids[2]], + merklePath: proof.toHex() + } + f.engine.topicAnchorHeaderResolver = async height => ({ + blockHeight: height, + blockHash, + merkleRoot: root, + blockTransactionCount: 3 + }) + const [report] = await f.engine.startBASMSync() + expect(report.status).toBe('error') + expect(report.message).toContain('canonical block transaction count') + expect(f.submit).not.toHaveBeenCalled() + }) + + it.each([ + 'block hash', + 'header height', + 'header unavailable', + 'header root', + 'proof height', + 'proof index', + 'raw identity', + 'raw trailing bytes', + 'range TAC', + 'range gap' + ])('rejects invalid %s before admission', async failure => { + const f = fixture() + switch (failure) { + case 'block hash': + f.engine.topicAnchorHeaderResolver = async height => ({ + blockHeight: height, + blockHash: 'cd'.repeat(32) + }) + break + case 'header height': + f.engine.topicAnchorHeaderResolver = async height => ({ + blockHeight: height + 1, + blockHash + }) + break + case 'header unavailable': + f.engine.topicAnchorHeaderResolver = async () => undefined + break + case 'header root': + f.engine.topicAnchorHeaderResolver = async height => ({ + blockHeight: height, + blockHash, + merkleRoot: BASM_ZERO_HASH + }) + break + case 'proof height': + f.path.blockHeight += 1 + f.responses['/requestCompoundMerklePath'] = { + topic, + blockHeight, + txids: f.admitted.map(item => item.txid), + merklePath: f.path.toHex() + } + break + case 'proof index': + f.admitted[0].blockIndex = 0 + break + case 'raw identity': + case 'raw trailing bytes': + f.responses['/requestRawTransactions'] = { + transactions: f.admitted.map((item, i) => ({ + txid: item.txid, + rawTx: + f.transactions[failure === 'raw identity' ? 0 : i * 2 + 1].toHex() + + (failure === 'raw trailing bytes' ? '00' : '') + })), + missing: [] + } + break + case 'range TAC': + f.anchor.tac = 'cd'.repeat(32) + break + case 'range gap': + f.responses['/requestTopicAnchorRange'] = { topic, anchors: [] } + break + } + const [report] = await f.engine.startBASMSync() + expect(report.status).toBe('error') + expect(f.submit).not.toHaveBeenCalled() + }) + + it('requires canonical header capability even for equal-height matching tips', async () => { + const f = fixture() + jest.spyOn(f.engine, 'provideTopicAnchorTip').mockResolvedValue(f.anchor) + f.engine.topicAnchorHeaderResolver = undefined + const [report] = await f.engine.startBASMSync() + expect(report.status).toBe('error') + expect(report.message).toContain('canonical header resolver') + expect(f.submit).not.toHaveBeenCalled() + }) + + it('detects a peer anchor changing while raw transactions are fetched', async () => { + const f = fixture() + const originalFetch = f.fetchMock.getMockImplementation()! + f.fetchMock.mockImplementation(async (url, init) => { + const response = await originalFetch(url, init) + if (new URL(String(url)).pathname === '/requestRawTransactions') { + f.responses['/requestTopicAnchorRange'] = { + topic, + anchors: [{ ...f.anchor, tac: 'ef'.repeat(32) }] + } + } + return response + }) + const [report] = await f.engine.startBASMSync() + expect(report.status).toBe('error') + expect(report.message).toContain('changed before admission') + expect(f.submit).not.toHaveBeenCalled() + }) + + it('starts the next forward page immediately after the local tip when remote is over a page ahead', async () => { + const f = fixture() + jest + .spyOn(f.engine, 'provideTopicAnchorTip') + .mockResolvedValue({ topic, blockHeight: 1, tac: BASM_ZERO_HASH }) + f.anchor.blockHeight = 3000 + await f.engine.startBASMSync() + const ranges = f.requests.filter(request => request.path === '/requestTopicAnchorRange') + expect(ranges.map(request => request.body)).toEqual([ + { fromHeight: 3000, toHeight: 3000 }, + { fromHeight: 2, toHeight: 1001 } + ]) + expect(f.submit).not.toHaveBeenCalled() + }) + + it('refuses an old-topic bootstrap tail without a trusted TAC prefix (B02 continuation required)', async () => { + const f = fixture() + const anchors: TopicBlockAnchor[] = [] + let tac = BASM_ZERO_HASH + for (let height = 100; height <= 1100; height++) { + const basmRoot = height === 100 ? f.anchor.basmRoot : BASM_ZERO_HASH + tac = computeTac(tac, blockHash, basmRoot) + anchors.push({ + topic, + blockHeight: height, + blockHash, + basmRoot, + admittedCount: height === 100 ? 2 : 0, + tac + }) + } + f.fetchMock.mockImplementation(async (url, init) => { + const endpoint = new URL(String(url)).pathname + if (endpoint === '/requestTopicAnchorTip') return Response.json(anchors.at(-1)) + const body = JSON.parse(String(init?.body)) as { fromHeight: number; toHeight: number } + return Response.json({ + topic, + anchors: anchors.filter( + anchor => anchor.blockHeight >= body.fromHeight && anchor.blockHeight <= body.toHeight + ) + }) + }) + const [report] = await f.engine.startBASMSync() + expect(report.status).toBe('error') + expect(report.message).toContain('TAC is inconsistent with its prefix') + expect(f.submit).not.toHaveBeenCalled() + }) + + it('binds claimed indices to the compound path when every remote txid is already local', async () => { + const f = fixture() + f.storage.findAdmittedTransactionsForBlock.mockResolvedValue([...f.admitted]) + f.admitted[0].blockIndex = 0 + f.admitted[1].blockIndex = 2 + f.anchor.basmRoot = computeBasmRoot(f.admitted) + f.anchor.tac = computeTac(BASM_ZERO_HASH, blockHash, f.anchor.basmRoot) + const [report] = await f.engine.startBASMSync() + expect(report.status).toBe('error') + expect(report.message).toContain('admitted block index') + expect(f.requests.some(request => request.path === '/requestCompoundMerklePath')).toBe(true) + expect(f.requests.some(request => request.path === '/requestRawTransactions')).toBe(false) + expect(f.submit).not.toHaveBeenCalled() + }) + + it('still requests a compound path before reporting local-superset divergence', async () => { + const f = fixture() + f.storage.findAdmittedTransactionsForBlock.mockResolvedValue([...f.admitted]) + const [report] = await f.engine.startBASMSync() + expect(report.status).toBe('diverged') + expect(report.positionValidation).toBe('encoded-offset-only') + expect(f.requests.some(request => request.path === '/requestCompoundMerklePath')).toBe(true) + expect(f.requests.some(request => request.path === '/requestRawTransactions')).toBe(false) + expect(f.submit).not.toHaveBeenCalled() + }) + + it('accepts an admitted coinbase in a block younger than 100 confirmations', async () => { + const f = fixture() + const coinbase = new Transaction( + 1, + [], + [{ satoshis: 50, lockingScript: LockingScript.fromASM('OP_TRUE') }], + 0 + ) + const txid = coinbase.id('hex') + const path = new MerklePath(blockHeight, [[{ offset: 0, hash: txid, txid: true }]]) + const admitted = [{ txid, blockIndex: 0 }] + const anchor: TopicBlockAnchor = { + topic, + blockHeight, + blockHash, + basmRoot: txid, + admittedCount: 1, + tac: computeTac(BASM_ZERO_HASH, blockHash, txid) + } + f.tracker.currentHeight.mockResolvedValue(blockHeight + 50) + f.tracker.isValidRootForHeight.mockImplementation( + async (candidate: string, height: number) => candidate === txid && height === blockHeight + ) + f.engine.topicAnchorHeaderResolver = jest.fn(async height => ({ + blockHeight: height, + blockHash, + merkleRoot: txid + })) + f.responses['/requestTopicAnchorTip'] = anchor + f.responses['/requestTopicAnchorRange'] = { topic, anchors: [anchor] } + f.responses['/requestAdmittedList'] = { topic, blockHeight, blockHash, admitted } + f.responses['/requestCompoundMerklePath'] = { + topic, + blockHeight, + txids: [txid], + merklePath: path.toHex() + } + f.responses['/requestRawTransactions'] = { + transactions: [{ txid, rawTx: coinbase.toHex() }], + missing: [] + } + const manager = { + identifyAdmissibleOutputs: jest.fn(async () => ({ outputsToAdmit: [0], coinsToRetain: [] })), + getDocumentation: jest.fn(async () => ''), + getMetaData: jest.fn(async () => ({ name: 'test', shortDescription: 'test' })) + } + f.engine.managers[topic] = manager + Object.assign(f.storage, { + doesAppliedTransactionExist: jest.fn(async () => false), + findOutput: jest.fn(async () => null), + insertOutput: jest.fn(async () => undefined), + insertAppliedTransaction: jest.fn(async () => undefined), + upsertTransactionRecord: jest.fn(async () => undefined) + }) + // Exercise the production submit path: the young coinbase must survive it. + f.submit.mockRestore() + const submit = jest.spyOn(f.engine, 'submit') + const verify = jest.spyOn(MerklePath.prototype, 'verify') + const [report] = await f.engine.startBASMSync() + expect(report.status).toBe('advanced') + expect(report.fetchedTxCount).toBe(1) + expect(verify).not.toHaveBeenCalled() + expect(f.tracker.isValidRootForHeight).toHaveBeenCalledWith(txid, blockHeight) + expect(submit).toHaveBeenCalledTimes(1) + expect(submit.mock.calls[0][2]).toBe('historical-tx-no-spv') + expect(manager.identifyAdmissibleOutputs).toHaveBeenCalledWith( + expect.anything(), + [], + undefined, + 'historical-tx-no-spv' + ) + expect(f.storage.insertAppliedTransaction).toHaveBeenCalledWith( + expect.objectContaining({ txid, topic, blockHeight, blockIndex: 0, proven: true }) + ) + // The only chain-height read is the firstSeen bookkeeping in + // buildAppliedTransactionRecord; no coinbase spendability check ran. + expect(f.tracker.currentHeight).toHaveBeenCalledTimes(1) + }) + + it('admits a BASM-proven historical coinbase through the real submit path', async () => { + const f = fixture() + const coinbase = new Transaction( + 1, + [], + [{ satoshis: 50, lockingScript: LockingScript.fromASM('OP_TRUE') }], + 0 + ) + const beef = coinbase.toBEEF() + const txid = coinbase.id('hex') + const manager = { + identifyAdmissibleOutputs: jest.fn(async () => ({ outputsToAdmit: [0], coinsToRetain: [] })), + getDocumentation: jest.fn(async () => ''), + getMetaData: jest.fn(async () => ({ name: 'test', shortDescription: 'test' })) + } + f.engine.managers[topic] = manager + Object.assign(f.storage, { + doesAppliedTransactionExist: jest.fn(async () => false), + findOutput: jest.fn(async () => null), + insertOutput: jest.fn(async () => undefined), + insertAppliedTransaction: jest.fn(async () => undefined), + upsertTransactionRecord: jest.fn(async () => undefined) + }) + f.submit.mockRestore() + + await expect( + f.engine.submit({ beef, topics: [topic] }, undefined, 'historical-tx-no-spv') + ).resolves.toEqual({ + [topic]: { outputsToAdmit: [0], coinsToRetain: [], coinsRemoved: [] } + }) + expect(manager.identifyAdmissibleOutputs).toHaveBeenCalledWith( + beef, + [], + undefined, + 'historical-tx-no-spv' + ) + expect(f.storage.insertAppliedTransaction).toHaveBeenCalledWith( + expect.objectContaining({ txid, topic, proven: false }) + ) + }) + + it('still runs SPV verification for a public historical-tx submission', async () => { + const f = fixture() + const proven = f.transactions[1] + // A proof whose root is not the canonical root for this height: the chain + // tracker rejects it, so MerklePath.verify fails and submit must throw. + proven.merklePath = new MerklePath(blockHeight, [ + [ + { offset: 0, hash: f.ids[0], txid: true }, + { offset: 1, hash: f.ids[1], txid: true } + ] + ]) + const beef = proven.toBEEF() + const manager = { + identifyAdmissibleOutputs: jest.fn(async () => ({ outputsToAdmit: [0], coinsToRetain: [] })), + getDocumentation: jest.fn(async () => ''), + getMetaData: jest.fn(async () => ({ name: 'test', shortDescription: 'test' })) + } + f.engine.managers[topic] = manager + Object.assign(f.storage, { + doesAppliedTransactionExist: jest.fn(async () => false), + findOutput: jest.fn(async () => null), + insertOutput: jest.fn(async () => undefined), + insertAppliedTransaction: jest.fn(async () => undefined), + upsertTransactionRecord: jest.fn(async () => undefined) + }) + f.submit.mockRestore() + + await expect( + f.engine.submit({ beef, topics: [topic] }, undefined, 'historical-tx') + ).rejects.toThrow(/Invalid merkle path/) + expect(f.tracker.isValidRootForHeight).toHaveBeenCalled() + expect(manager.identifyAdmissibleOutputs).not.toHaveBeenCalled() + expect(f.storage.insertAppliedTransaction).not.toHaveBeenCalled() + }) + + it('reports a finite proof request limit for a block above 1000 admissions (B02 chunking required)', async () => { + const f = fixture() + const admitted = Array.from({ length: 1001 }, (_, blockIndex) => ({ + txid: blockIndex.toString(16).padStart(64, '0'), + blockIndex + })) + f.anchor.admittedCount = admitted.length + f.anchor.basmRoot = computeBasmRoot(admitted) + f.anchor.tac = computeTac(BASM_ZERO_HASH, blockHash, f.anchor.basmRoot) + f.responses['/requestAdmittedList'] = { topic, blockHeight, blockHash, admitted } + const [report] = await f.engine.startBASMSync() + expect(report.status).toBe('error') + expect(report.errorCode).toBe('BASM_RESOURCE_LIMIT') + expect(f.requests.some(request => request.path === '/requestCompoundMerklePath')).toBe(false) + expect(f.submit).not.toHaveBeenCalled() + }) +}) diff --git a/packages/overlays/overlay/src/__tests/BASMReconciliationFixture.ts b/packages/overlays/overlay/src/__tests/BASMReconciliationFixture.ts new file mode 100644 index 000000000..556d777f5 --- /dev/null +++ b/packages/overlays/overlay/src/__tests/BASMReconciliationFixture.ts @@ -0,0 +1,116 @@ +import { createHash } from 'node:crypto' +import { LockingScript, MerklePath, Transaction } from '@bsv/sdk' +import { Engine } from '../Engine' +import { BASM_ZERO_HASH, computeBasmRoot, computeTac } from '../BASM' +import type { AdmittedTxRef, TopicAnchorTip, TopicBlockAnchor } from '../BASM' +import type { Storage } from '../storage/Storage' + +export const topic = 'tm_basm_test' +export const blockHeight = 100 +export const blockHash = 'ab'.repeat(32) + +export const hashPair = (left: string, right: string): string => { + const bytes = Buffer.concat([ + Buffer.from(left, 'hex').reverse(), + Buffer.from(right, 'hex').reverse() + ]) + const first = createHash('sha256').update(bytes).digest() + return createHash('sha256').update(first).digest().reverse().toString('hex') +} + +export function fixture() { + const transactions = [1, 2, 3, 4].map( + satoshis => + new Transaction(1, [], [{ satoshis, lockingScript: LockingScript.fromASM('OP_TRUE') }], 0) + ) + const ids = transactions.map(tx => tx.id('hex')) + const root = hashPair(hashPair(ids[0], ids[1]), hashPair(ids[2], ids[3])) + const path = new MerklePath(blockHeight, [ + ids.map((hash, offset) => ({ offset, hash, txid: true })), + [] + ]) + const admitted = [ + { txid: ids[1], blockIndex: 1 }, + { txid: ids[3], blockIndex: 3 } + ] + const anchor: TopicBlockAnchor = { + topic, + blockHeight, + blockHash, + basmRoot: computeBasmRoot(admitted), + admittedCount: 2, + tac: '' + } + anchor.tac = computeTac(BASM_ZERO_HASH, blockHash, anchor.basmRoot) + const responses: Record = { + '/requestTopicAnchorTip': anchor, + '/requestTopicAnchorRange': { topic, anchors: [anchor] }, + '/requestAdmittedList': { topic, blockHeight, blockHash, admitted }, + '/requestCompoundMerklePath': { + topic, + blockHeight, + txids: admitted.map(item => item.txid), + merklePath: path.toHex() + }, + '/requestRawTransactions': { + // Deliberately reversed: admission must still follow original block order. + transactions: [3, 1].map(index => ({ txid: ids[index], rawTx: transactions[index].toHex() })), + missing: [] + } + } + const storage = { + findTopicAnchorTip: jest.fn(async (): Promise => undefined), + findTopicBlockAnchor: jest.fn(async (): Promise => undefined), + findAdmittedTransactionsForBlock: jest.fn(async (): Promise => []) + } + const tracker = { + currentHeight: jest.fn(async () => 250), + isValidRootForHeight: jest.fn( + async (candidate: string, height: number) => candidate === root && height === blockHeight + ) + } + const engine = new Engine({}, {}, storage as unknown as Storage, tracker) + engine.syncConfiguration = { [topic]: ['https://peer.example'] } + engine.topicAnchorHeaderResolver = jest.fn(async height => ({ + blockHeight: height, + blockHash, + merkleRoot: root + })) + engine.logger = { ...console, error: jest.fn() } + const submit = jest.spyOn(engine, 'submit').mockResolvedValue({}) + const requests: Array<{ path: string; body: Record }> = [] + if (typeof globalThis.fetch !== 'function') { + Object.defineProperty(globalThis, 'fetch', { + configurable: true, + writable: true, + value: async () => { + throw new Error('unexpected network request') + } + }) + } + const fetchMock = jest.spyOn(globalThis, 'fetch').mockImplementation(async (url, init) => { + const endpoint = new URL(String(url)).pathname + requests.push({ + path: endpoint, + body: JSON.parse(String(init?.body)) as Record + }) + return new Response(JSON.stringify(responses[endpoint]), { status: 200 }) + }) + return { + engine, + responses, + anchor, + admitted, + transactions, + ids, + path, + root, + submit, + storage, + tracker, + requests, + fetchMock + } +} + +export type BASMFixture = ReturnType diff --git a/packages/overlays/overlay/src/__tests/BASMRemote.test.ts b/packages/overlays/overlay/src/__tests/BASMRemote.test.ts new file mode 100644 index 000000000..6c18c6023 --- /dev/null +++ b/packages/overlays/overlay/src/__tests/BASMRemote.test.ts @@ -0,0 +1,583 @@ +import { BASMProtocolError, BASMRemote } from '../BASMRemote' + +const ENDPOINT = 'https://peer.example/overlay' +const TOPIC = 'tm_example' +const ZERO = '0000000000000000000000000000000000000000000000000000000000000000' +const TXID_1 = '0101010101010101010101010101010101010101010101010101010101010101' +const TXID_2 = '0202020202020202020202020202020202020202020202020202020202020202' +const TXID_3 = '0303030303030303030303030303030303030303030303030303030303030303' +const BLOCK_HASH = '0404040404040404040404040404040404040404040404040404040404040404' +const BASM_ROOT = '0505050505050505050505050505050505050505050505050505050505050505' +const TAC = '0606060606060606060606060606060606060606060606060606060606060606' + +function response(value: unknown, status = 200): Response { + return new Response(JSON.stringify(value), { + status, + headers: { 'content-type': 'application/json' } + }) +} + +function remoteFor(value: unknown, limits = {}): BASMRemote { + return new BASMRemote(ENDPOINT, TOPIC, async () => response(value), limits) +} + +function anchor(blockHeight: number): Record { + return { + topic: TOPIC, + blockHeight, + blockHash: BLOCK_HASH, + basmRoot: BASM_ROOT, + admittedCount: 0, + tac: TAC + } +} + +describe('BASMRemote', () => { + it('preserves the BASM request URL and wire shape while accepting legacy minimal responses', async () => { + const calls: Array<[RequestInfo | URL, RequestInit | undefined]> = [] + const injectedFetch: typeof fetch = async (input, init) => { + calls.push([input, init]) + if (String(input).endsWith('/requestTopicAnchorTip')) { + return response({ topic: TOPIC, blockHeight: -1, tac: ZERO }) + } + return response({ topic: TOPIC, blockHeight: 42, admitted: [] }) + } + const remote = new BASMRemote(ENDPOINT, TOPIC, injectedFetch) + + await expect(remote.requestTopicAnchorTip()).resolves.toEqual({ + topic: TOPIC, + blockHeight: -1, + tac: ZERO + }) + await expect(remote.requestAdmittedList(42)).resolves.toEqual({ + topic: TOPIC, + blockHeight: 42, + admitted: [] + }) + expect(calls).toEqual([ + [ + 'https://peer.example/requestTopicAnchorTip', + { + method: 'POST', + headers: { + Accept: 'application/json', + 'Content-Type': 'application/json', + 'x-bsv-topic': TOPIC + }, + body: '{}', + signal: expect.any(AbortSignal) + } + ], + [ + 'https://peer.example/requestAdmittedList', + { + method: 'POST', + headers: { + Accept: 'application/json', + 'Content-Type': 'application/json', + 'x-bsv-topic': TOPIC + }, + body: JSON.stringify({ blockHeight: 42 }), + signal: expect.any(AbortSignal) + } + ] + ]) + }) + + it.each([ + ['wrong topic', { topic: 'tm_other', blockHeight: 7, tac: TAC }], + ['wrong height', { topic: TOPIC, blockHeight: 8, admitted: [] }], + [ + 'conflicting block hash', + { topic: TOPIC, blockHeight: 7, blockHash: BLOCK_HASH, admitted: [] } + ] + ])('rejects an admitted response with a %s', async (_label, value) => { + const remote = remoteFor(value) + await expect(remote.requestAdmittedList(7, TXID_1)).rejects.toMatchObject({ + name: 'BASMProtocolError', + code: 'BASM_INVALID_RESPONSE' + }) + }) + + it('caps the admitted list before accepting unbounded peer data', async () => { + const remote = remoteFor( + { + topic: TOPIC, + blockHeight: 7, + admitted: [ + { txid: TXID_1, blockIndex: 0 }, + { txid: TXID_2, blockIndex: 1 } + ] + }, + { maxAdmittedTxids: 1 } + ) + + await expect(remote.requestAdmittedList(7)).rejects.toMatchObject({ + code: 'BASM_RESOURCE_LIMIT' + }) + }) + + it.each([ + ['unordered anchors', [anchor(5), anchor(4)]], + ['a gap between anchors', [anchor(4), anchor(6)]] + ])('rejects an anchor range with %s', async (_label, anchors) => { + const remote = remoteFor({ topic: TOPIC, anchors }) + await expect(remote.requestTopicAnchorRange(4, 6)).rejects.toMatchObject({ + code: 'BASM_INVALID_RESPONSE' + }) + }) + + it('rejects an anchor response with more entries than the requested range', async () => { + const remote = remoteFor({ topic: TOPIC, anchors: [anchor(4), anchor(5), anchor(6)] }) + await expect(remote.requestTopicAnchorRange(4, 5)).rejects.toMatchObject({ + code: 'BASM_RESOURCE_LIMIT' + }) + }) + + it.each([ + [ + 'a duplicate txid', + [ + { txid: TXID_1, blockIndex: 0 }, + { txid: TXID_1, blockIndex: 1 } + ] + ], + [ + 'a duplicate block index', + [ + { txid: TXID_1, blockIndex: 0 }, + { txid: TXID_2, blockIndex: 0 } + ] + ], + [ + 'nonmonotonic block indices', + [ + { txid: TXID_1, blockIndex: 2 }, + { txid: TXID_2, blockIndex: 1 } + ] + ] + ])('rejects an admitted list with %s', async (_label, admitted) => { + const remote = remoteFor({ topic: TOPIC, blockHeight: 7, admitted }) + await expect(remote.requestAdmittedList(7)).rejects.toMatchObject({ + code: 'BASM_INVALID_RESPONSE' + }) + }) + + it('accepts a proof response whose txids are reordered but exactly match the request', async () => { + const remote = remoteFor({ + topic: TOPIC, + blockHeight: 7, + txids: [TXID_2, TXID_1], + merklePath: 'aabb' + }) + await expect(remote.requestCompoundMerklePath(7, [TXID_1, TXID_2])).resolves.toEqual({ + topic: TOPIC, + blockHeight: 7, + txids: [TXID_2, TXID_1], + merklePath: 'aabb' + }) + }) + + it('rejects a proof response that does not contain exactly the requested txid set', async () => { + const remote = remoteFor({ + topic: TOPIC, + blockHeight: 7, + txids: [TXID_1, TXID_3], + merklePath: 'aabb' + }) + await expect(remote.requestCompoundMerklePath(7, [TXID_1, TXID_2])).rejects.toMatchObject({ + code: 'BASM_INVALID_RESPONSE' + }) + }) + + it.each([ + [ + 'an extra transaction', + { transactions: [{ txid: TXID_3, rawTx: 'aabb' }], missing: [TXID_1, TXID_2] } + ], + ['an omitted transaction', { transactions: [{ txid: TXID_1, rawTx: 'aabb' }], missing: [] }], + [ + 'a duplicate transaction', + { + transactions: [ + { txid: TXID_1, rawTx: 'aabb' }, + { txid: TXID_1, rawTx: 'ccdd' } + ], + missing: [] + } + ], + [ + 'a transaction also reported missing', + { transactions: [{ txid: TXID_1, rawTx: 'aabb' }], missing: [TXID_1, TXID_2] } + ] + ])('rejects a raw transaction response containing %s', async (_label, value) => { + const remote = remoteFor(value) + await expect(remote.requestRawTransactions([TXID_1, TXID_2])).rejects.toMatchObject({ + code: 'BASM_INVALID_RESPONSE' + }) + }) + + it('rejects a chunked response once its streamed bytes cross the configured cap', async () => { + const stream = new ReadableStream({ + start(controller) { + controller.enqueue(Buffer.from('{"topic"')) + controller.enqueue(Buffer.from(':"tm_example"}')) + controller.close() + } + }) + const remote = new BASMRemote(ENDPOINT, TOPIC, async () => new Response(stream), { + maxResponseBytes: 10 + }) + + await expect(remote.requestTopicAnchorTip()).rejects.toMatchObject({ + code: 'BASM_RESOURCE_LIMIT' + }) + }) + + it('rejects an advertised response length above the configured cap before reading it', async () => { + const remote = new BASMRemote( + ENDPOINT, + TOPIC, + async () => new Response('{"topic":"tm_example"}', { headers: { 'content-length': '11' } }), + { maxResponseBytes: 10 } + ) + + await expect(remote.requestTopicAnchorTip()).rejects.toMatchObject({ + code: 'BASM_RESOURCE_LIMIT' + }) + }) + + it('reports malformed successful JSON as a protocol error', async () => { + const remote = new BASMRemote(ENDPOINT, TOPIC, async () => new Response('{'), {}) + await expect(remote.requestTopicAnchorTip()).rejects.toBeInstanceOf(BASMProtocolError) + await expect(remote.requestTopicAnchorTip()).rejects.toMatchObject({ + code: 'BASM_INVALID_RESPONSE' + }) + }) + + it('times out a fetch that observes the abort signal', async () => { + let abortObserved = false + const remote = new BASMRemote( + ENDPOINT, + TOPIC, + async (_input, init) => + await new Promise((_resolve, reject) => { + init?.signal?.addEventListener('abort', () => { + abortObserved = true + reject(init.signal?.reason) + }) + }), + { timeoutMs: 20 } + ) + + await expect(remote.requestTopicAnchorTip()).rejects.toMatchObject({ code: 'BASM_TIMEOUT' }) + expect(abortObserved).toBe(true) + }) + + it('times out a fetch that ignores the abort signal', async () => { + let signal: AbortSignal | undefined + const remote = new BASMRemote( + ENDPOINT, + TOPIC, + async (_input, init) => + await new Promise(() => { + signal = init?.signal ?? undefined + }), + { timeoutMs: 20 } + ) + + await expect(remote.requestTopicAnchorTip()).rejects.toMatchObject({ code: 'BASM_TIMEOUT' }) + expect(signal?.aborted).toBe(true) + }) + + it('times out while a real response body stalls', async () => { + const stalled = new ReadableStream({ + pull: async () => await new Promise(() => {}) + }) + const remote = new BASMRemote(ENDPOINT, TOPIC, async () => new Response(stalled), { + timeoutMs: 20 + }) + + await expect(remote.requestTopicAnchorTip()).rejects.toMatchObject({ code: 'BASM_TIMEOUT' }) + }) + + it.each([ + ['a 501 response', response({}, 501), 'BASM_UNSUPPORTED'], + [ + 'an explicit unsupported response', + response({ code: 'BASM_UNSUPPORTED' }, 400), + 'BASM_UNSUPPORTED' + ], + ['a generic 503 response', response({}, 503), 'BASM_HTTP_ERROR'] + ])('classifies %s', async (_label, peerResponse, code) => { + const remote = new BASMRemote(ENDPOINT, TOPIC, async () => peerResponse) + await expect(remote.requestTopicAnchorTip()).rejects.toMatchObject({ code }) + }) + + it('accepts a contiguous range and a complete raw-transaction partition', async () => { + const remote = new BASMRemote(ENDPOINT, TOPIC, async input => { + if (String(input).endsWith('/requestTopicAnchorRange')) { + return response({ topic: TOPIC, anchors: [anchor(4), anchor(5)] }) + } + return response({ + transactions: [{ txid: TXID_2, rawTx: 'ccdd' }], + missing: [TXID_1] + }) + }) + + await expect(remote.requestTopicAnchorRange(4, 5)).resolves.toEqual({ + topic: TOPIC, + anchors: [anchor(4), anchor(5)] + }) + await expect(remote.requestRawTransactions([TXID_1, TXID_2])).resolves.toEqual({ + transactions: [{ txid: TXID_2, rawTx: 'ccdd' }], + missing: [TXID_1] + }) + }) + + it.each([ + [ + 'an uppercase TAC', + async (remote: BASMRemote) => await remote.requestTopicAnchorTip(), + { topic: TOPIC, blockHeight: -1, tac: 'ab'.repeat(32).toUpperCase() } + ], + [ + 'a numeric-string height', + async (remote: BASMRemote) => await remote.requestTopicAnchorTip(), + { topic: TOPIC, blockHeight: '7', tac: TAC } + ], + [ + 'a mixed-case block hash', + async (remote: BASMRemote) => await remote.requestTopicAnchorRange(4, 4), + { topic: TOPIC, anchors: [{ ...anchor(4), blockHash: 'cd'.repeat(32).toUpperCase() }] } + ] + ])('rejects %s rather than coercing untrusted peer JSON', async (_label, invoke, value) => { + await expect(invoke(remoteFor(value))).rejects.toMatchObject({ + code: 'BASM_INVALID_RESPONSE' + }) + }) + + it('rejects a duplicated height as an unordered or gapped range', async () => { + const remote = remoteFor({ topic: TOPIC, anchors: [anchor(4), anchor(4)] }) + await expect(remote.requestTopicAnchorRange(4, 5)).rejects.toMatchObject({ + code: 'BASM_INVALID_RESPONSE' + }) + }) + + it('ignores unknown additive JSON fields on a legacy tip', async () => { + const remote = remoteFor({ + topic: TOPIC, + blockHeight: -1, + tac: ZERO, + extra: 'ignored' + }) + await expect(remote.requestTopicAnchorTip()).resolves.toEqual({ + topic: TOPIC, + blockHeight: -1, + tac: ZERO + }) + }) + + it.each([ + ['a non-integer timeout', { timeoutMs: 20.5 }, 'Invalid BASM limit: timeoutMs'], + ['a zero timeout', { timeoutMs: 0 }, 'Invalid BASM limit: timeoutMs'], + ['a negative response cap', { maxResponseBytes: -1 }, 'Invalid BASM limit: maxResponseBytes'] + ])('rejects %s', (_label, limits, message) => { + expect(() => new BASMRemote(ENDPOINT, TOPIC, async () => response({}), limits)).toThrow( + new TypeError(message) + ) + }) + + it('rejects a timeout that exceeds the platform timer range', () => { + expect( + () => + new BASMRemote(ENDPOINT, TOPIC, async () => response({}), { + timeoutMs: 2147483648 + }) + ).toThrow(new TypeError('BASM timeout exceeds timer range')) + }) + + it('reads a response whose advertised length is within the configured cap', async () => { + const payload = JSON.stringify({ topic: TOPIC, blockHeight: -1, tac: ZERO }) + const remote = new BASMRemote( + ENDPOINT, + TOPIC, + async () => + new Response(payload, { + headers: { 'content-length': String(Buffer.byteLength(payload)) } + }), + { maxResponseBytes: Buffer.byteLength(payload) } + ) + + await expect(remote.requestTopicAnchorTip()).resolves.toEqual({ + topic: TOPIC, + blockHeight: -1, + tac: ZERO + }) + }) + + it('reads a body-less response through text() when headers are absent', async () => { + const payload = JSON.stringify({ topic: TOPIC, blockHeight: -1, tac: ZERO }) + const remote = new BASMRemote(ENDPOINT, TOPIC, async () => { + return { + ok: true, + status: 200, + body: null, + text: async () => payload + } as unknown as Response + }) + + await expect(remote.requestTopicAnchorTip()).resolves.toEqual({ + topic: TOPIC, + blockHeight: -1, + tac: ZERO + }) + }) + + it('rejects a body-less response whose decoded text exceeds the byte cap', async () => { + const remote = new BASMRemote( + ENDPOINT, + TOPIC, + async () => { + return { + ok: true, + status: 200, + headers: { get: () => null }, + body: undefined, + text: async () => '{"topic":"tm_example"}' + } as unknown as Response + }, + { maxResponseBytes: 4 } + ) + + await expect(remote.requestTopicAnchorTip()).rejects.toMatchObject({ + code: 'BASM_RESOURCE_LIMIT' + }) + }) + + it('cancels an oversized advertised body even when cancel rejects', async () => { + const remote = new BASMRemote( + ENDPOINT, + TOPIC, + async () => { + return { + ok: true, + status: 200, + headers: { get: () => '64' }, + body: { + cancel: async () => { + throw new Error('cancel failed') + } + }, + text: async () => '{"topic":"tm_example"}' + } as unknown as Response + }, + { maxResponseBytes: 8 } + ) + + await expect(remote.requestTopicAnchorTip()).rejects.toMatchObject({ + code: 'BASM_RESOURCE_LIMIT' + }) + }) + + it('skips body cancel when an oversized advertised response has no body', async () => { + const remote = new BASMRemote( + ENDPOINT, + TOPIC, + async () => { + return { + ok: true, + status: 200, + headers: { get: () => '64' }, + body: null, + text: async () => '{"topic":"tm_example"}' + } as unknown as Response + }, + { maxResponseBytes: 8 } + ) + + await expect(remote.requestTopicAnchorTip()).rejects.toMatchObject({ + code: 'BASM_RESOURCE_LIMIT' + }) + }) + + it('classifies malformed JSON on a failed HTTP response as an HTTP error', async () => { + const remote = new BASMRemote(ENDPOINT, TOPIC, async () => new Response('{', { status: 503 })) + await expect(remote.requestTopicAnchorTip()).rejects.toMatchObject({ + code: 'BASM_HTTP_ERROR' + }) + }) + + it.each([ + ['resolves', async () => {}], + [ + 'rejects', + async () => { + throw new Error('cancel failed') + } + ] + ])('cancels a late response body whose cancel %s after abort', async (_label, cancel) => { + let completeFetch: ((value: Response) => void) | undefined + let cancelled = false + const remote = new BASMRemote( + ENDPOINT, + TOPIC, + async () => + await new Promise(resolve => { + completeFetch = resolve + }), + { timeoutMs: 20 } + ) + + await expect(remote.requestTopicAnchorTip()).rejects.toMatchObject({ + code: 'BASM_TIMEOUT' + }) + + completeFetch?.({ + ok: true, + status: 200, + headers: { get: () => null }, + body: { + cancel: async () => { + cancelled = true + await cancel() + } + }, + text: async () => '' + } as unknown as Response) + + const deadline = Date.now() + 1000 + while (!cancelled && Date.now() < deadline) { + await new Promise(resolve => setTimeout(resolve, 10)) + } + expect(cancelled).toBe(true) + }) + + it('does not require a body when cancelling a late aborted response', async () => { + let completeFetch: ((value: Response) => void) | undefined + const remote = new BASMRemote( + ENDPOINT, + TOPIC, + async () => + await new Promise(resolve => { + completeFetch = resolve + }), + { timeoutMs: 20 } + ) + + await expect(remote.requestTopicAnchorTip()).rejects.toMatchObject({ + code: 'BASM_TIMEOUT' + }) + expect(completeFetch).toBeDefined() + + completeFetch?.({ + ok: true, + status: 200, + headers: { get: () => null }, + body: null, + text: async () => '' + } as unknown as Response) + + await new Promise(resolve => setTimeout(resolve, 20)) + }) +}) diff --git a/packages/overlays/overlay/src/__tests/BASMTipReconciliation.test.ts b/packages/overlays/overlay/src/__tests/BASMTipReconciliation.test.ts new file mode 100644 index 000000000..27bfd4f8d --- /dev/null +++ b/packages/overlays/overlay/src/__tests/BASMTipReconciliation.test.ts @@ -0,0 +1,141 @@ +import { BASM_ZERO_HASH } from '../BASM' +import { blockHash, blockHeight, fixture, topic } from './BASMReconciliationFixture' + +afterEach(() => jest.restoreAllMocks()) + +describe('BASM reconciliation when the peer is not ahead', () => { + it('matches two empty topics without asking the peer for anchors', async () => { + const f = fixture() + f.responses['/requestTopicAnchorTip'] = { topic, blockHeight: -1, tac: BASM_ZERO_HASH } + const [report] = await f.engine.startBASMSync() + expect(report.status).toBe('matched') + expect(report.message).toBe('Topic anchor tips match') + expect(report.localTip).toEqual({ topic, blockHeight: -1, tac: BASM_ZERO_HASH }) + expect(report.remoteTip).toEqual({ topic, blockHeight: -1, tac: BASM_ZERO_HASH }) + // An empty remote tip has no anchor to cross-check, so the range endpoint + // must not be called and no local anchor lookup may happen either. + expect(f.requests.map(request => request.path)).toEqual(['/requestTopicAnchorTip']) + expect(f.storage.findTopicBlockAnchor).not.toHaveBeenCalled() + expect(f.submit).not.toHaveBeenCalled() + }) + + it('re-verifies the local anchor behind an equal, matching tip', async () => { + const f = fixture() + f.storage.findTopicAnchorTip.mockResolvedValue({ topic, blockHeight, tac: f.anchor.tac }) + f.storage.findTopicBlockAnchor.mockResolvedValue(f.anchor) + const [report] = await f.engine.startBASMSync() + expect(report.status).toBe('matched') + expect(report.message).toBe('Topic anchor tips match') + expect(f.storage.findTopicBlockAnchor).toHaveBeenCalledWith(topic, blockHeight) + expect(f.engine.topicAnchorHeaderResolver).toHaveBeenCalledWith(blockHeight) + expect(f.requests.some(request => request.path === '/requestAdmittedList')).toBe(false) + expect(f.submit).not.toHaveBeenCalled() + }) + + it('reports historical divergence when the local tip is ahead with a different TAC', async () => { + const f = fixture() + f.storage.findTopicAnchorTip.mockResolvedValue({ + topic, + blockHeight: blockHeight + 1, + tac: 'cd'.repeat(32) + }) + const [report] = await f.engine.startBASMSync() + expect(report.status).toBe('diverged') + expect(report.message).toContain('Remote tip is not ahead') + // Divergence is not an anchor failure: the local tip is never re-verified. + expect(f.storage.findTopicBlockAnchor).not.toHaveBeenCalled() + expect(f.submit).not.toHaveBeenCalled() + }) + + it('fails closed when the local tip has no stored anchor', async () => { + const f = fixture() + f.storage.findTopicAnchorTip.mockResolvedValue({ topic, blockHeight, tac: f.anchor.tac }) + f.storage.findTopicBlockAnchor.mockResolvedValue(undefined) + const [report] = await f.engine.startBASMSync() + expect(report.status).toBe('error') + expect(report.errorCode).toBe('BASM_INVALID_RESPONSE') + expect(report.message).toBe('Local BASM tip lacks its anchor') + expect(f.submit).not.toHaveBeenCalled() + }) + + it('fails closed when the backend reports a tip it cannot produce an anchor for', async () => { + const f = fixture() + f.storage.findTopicAnchorTip.mockResolvedValue({ topic, blockHeight, tac: f.anchor.tac }) + // A backend with tip support but no per-block anchor lookup must not have + // its equal-height tip accepted on the peer's word alone. + delete (f.storage as { findTopicBlockAnchor?: unknown }).findTopicBlockAnchor + const [report] = await f.engine.startBASMSync() + expect(report.status).toBe('error') + expect(report.errorCode).toBe('BASM_INVALID_RESPONSE') + expect(report.message).toBe('Local BASM tip lacks its anchor') + expect(f.submit).not.toHaveBeenCalled() + }) + + it('fails closed when the stored anchor disagrees with the local tip TAC', async () => { + const f = fixture() + f.storage.findTopicAnchorTip.mockResolvedValue({ topic, blockHeight, tac: f.anchor.tac }) + f.storage.findTopicBlockAnchor.mockResolvedValue({ ...f.anchor, tac: 'ef'.repeat(32) }) + const [report] = await f.engine.startBASMSync() + expect(report.status).toBe('error') + expect(report.errorCode).toBe('BASM_INVALID_RESPONSE') + expect(report.message).toBe('Local BASM tip lacks its anchor') + expect(f.submit).not.toHaveBeenCalled() + }) + + it('fails closed when the equal-height local anchor is not canonical', async () => { + const f = fixture() + f.storage.findTopicAnchorTip.mockResolvedValue({ topic, blockHeight, tac: f.anchor.tac }) + f.storage.findTopicBlockAnchor.mockResolvedValue(f.anchor) + f.engine.topicAnchorHeaderResolver = async height => ({ + blockHeight: height, + blockHash: 'cd'.repeat(32) + }) + const [report] = await f.engine.startBASMSync() + expect(report.status).toBe('error') + expect(report.message).toBe('BASM anchor block hash is not canonical') + expect(f.submit).not.toHaveBeenCalled() + }) +}) + +describe('BASM reconciliation outcome reporting', () => { + it('reports "matched" when the admitted page brings the local tip up to the remote tip', async () => { + const f = fixture() + f.storage.findTopicAnchorTip + .mockResolvedValueOnce(undefined) + .mockResolvedValue({ topic, blockHeight, tac: f.anchor.tac, blockHash }) + const [report] = await f.engine.startBASMSync() + expect(report.status).toBe('matched') + expect(report.fetchedTxCount).toBe(2) + expect(report.localTip).toEqual({ topic, blockHeight, tac: f.anchor.tac, blockHash }) + expect(f.submit).toHaveBeenCalledTimes(2) + }) + + it('reports "advanced" when the local tip still trails after the page is applied', async () => { + const f = fixture() + const [report] = await f.engine.startBASMSync() + expect(report.status).toBe('advanced') + expect(report.localTip).toEqual({ topic, blockHeight: -1, tac: BASM_ZERO_HASH }) + expect(f.submit).toHaveBeenCalledTimes(2) + }) + + it('stringifies a non-Error failure without inventing an error code', async () => { + const f = fixture() + f.storage.findTopicAnchorTip.mockRejectedValue('peer storage is offline') + const [report] = await f.engine.startBASMSync() + expect(report.status).toBe('error') + expect(report.message).toBe('peer storage is offline') + expect(report.errorCode).toBeUndefined() + expect(f.submit).not.toHaveBeenCalled() + }) + + it('surfaces a BASM_UNSUPPORTED storage gap as a classified sync error', async () => { + const f = fixture() + // A backend that cannot answer tip queries at all. + delete (f.storage as { findTopicAnchorTip?: unknown }).findTopicAnchorTip + const [report] = await f.engine.startBASMSync() + expect(report.status).toBe('error') + expect(report.errorCode).toBe('BASM_UNSUPPORTED') + expect(report.message).toBe('Storage does not support BASM topic anchor tips') + expect(f.submit).not.toHaveBeenCalled() + }) +}) diff --git a/packages/overlays/overlay/src/__tests/fixtures/README.md b/packages/overlays/overlay/src/__tests/fixtures/README.md new file mode 100644 index 000000000..48ddcce21 --- /dev/null +++ b/packages/overlays/overlay/src/__tests/fixtures/README.md @@ -0,0 +1,35 @@ +# Independent BRC-136 fixture + +brc136-independent.json is copied byte-for-byte from the Go BASM +foundation fixture in the `go-overlay-services` repository: + +pkg/core/basm/testdata/vectors.json + +Source commit: d99216814a4b9dca5f9f4d04a602ef2d48bdc4a7 +Go worktree revision used for inventory: faaf69d372fd5e9974eaadbae9b8e26d761f0c86 +BRC-136 revision: 2733cd2950a739b3c977b95d652ff63e3773c40b +Fixture SHA-256: 51983432bb561e3031fb7c983947dc8bfdcfd0ef828ef9f24093666ff4665c8a + +The vectors cover ordered admitted subsets, multi-level Merkle roots, +Bitcoin odd-node duplication, original display/internal byte order, and +contiguous TAC chains through empty heights. The +repeated-last-four-illustrative-invalid vector intentionally shares a root +with the three-leaf illustrative vector. Primitive root calculation accepts +that input, while admitted-list validation must reject duplicate transaction +IDs. This fixture does not define multiproof wire bytes. + +TypeScript tests recompute every root and TAC with Node crypto SHA-256d and, +when present, OpenSSL `dgst -sha256`; they do not treat the Go expected +values as authoritative without that independent check. +`basm-go-read-server.go` is a local B01 interop host compiled into a temporary +module. It replaces onto the Go overlay-services worktree and must not modify +that tree's uncommitted S04 files. + +The Go interop suite (`BASMGoInterop.test.ts`) does not run by default: it is +skipped unless `BASM_GO_OVERLAY_SERVICES` points at a `go-overlay-services` +checkout, and no workstation path is assumed. When the variable is set but does +not resolve to such a checkout the suite fails rather than skips, so a job that +claims Go interop cannot pass without running it. + +The temporary interop server binds and advertises only `127.0.0.1`; its HTTP +transport never exposes the fixture on an external network interface. diff --git a/packages/overlays/overlay/src/__tests/fixtures/basm-go-read-server.go b/packages/overlays/overlay/src/__tests/fixtures/basm-go-read-server.go new file mode 100644 index 000000000..57eba7cc3 --- /dev/null +++ b/packages/overlays/overlay/src/__tests/fixtures/basm-go-read-server.go @@ -0,0 +1,212 @@ +// Temporary B01 interop host. Compiled from the TypeScript test into a temp +// module that replaces onto the local go-overlay-services worktree. It does not +// live in that worktree and must not be used to edit S04 files. +package main + +import ( + "context" + "encoding/hex" + "fmt" + "net" + "os" + "os/signal" + "syscall" + + "github.com/bsv-blockchain/go-sdk/chainhash" + "github.com/bsv-blockchain/go-sdk/overlay" + "github.com/bsv-blockchain/go-sdk/overlay/lookup" + "github.com/bsv-blockchain/go-sdk/transaction" + "github.com/gofiber/fiber/v2" + + "github.com/bsv-blockchain/go-overlay-services/pkg/core/basm" + "github.com/bsv-blockchain/go-overlay-services/pkg/core/engine" + "github.com/bsv-blockchain/go-overlay-services/pkg/core/gasp" + "github.com/bsv-blockchain/go-overlay-services/pkg/server" +) + +const ( + interopTopic = "tm_interop" + interopGenesisID = "4a5e1e4baab89f3a32518a88c31bc87f618f76673e2cc77ab2127b7afdeda33b" + interopGenesisHex = "01000000010000000000000000000000000000000000000000000000000000000000000000ffffffff4d04ffff001d0104455468652054696d65732030332f4a616e2f32303039204368616e63656c6c6f72206f6e206272696e6b206f66207365636f6e64206261696c6f757420666f722062616e6b73ffffffff0100f2052a01000000434104678afdb0fe5548271967f1a67130b7105cd6a828e03909a67962e0ea1f61deb649f6bc3f4cef38c4f35504e51ec112de5c384df7ba0b8d578a4c702b6bf11d5fac00000000" + interopGenesisBlock = "000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f" +) + +func main() { + if err := run(); err != nil { + fmt.Fprintln(os.Stderr, err) + os.Exit(1) + } +} + +func run() error { + ready, err := newReadyService() + if err != nil { + return err + } + readyURL, err := listen(ready) + if err != nil { + return err + } + unsupportedURL, err := listen(nil) + if err != nil { + return err + } + fmt.Printf("READY %s\n", readyURL) + fmt.Printf("UNSUPPORTED %s\n", unsupportedURL) + _ = os.Stdout.Sync() + + signals := make(chan os.Signal, 1) + signal.Notify(signals, syscall.SIGINT, syscall.SIGTERM) + <-signals + return nil +} + +func listen(provider engine.BASMProvider) (string, error) { + app := server.RegisterRoutesWithErrorHandler(fiber.New(fiber.Config{DisableStartupMessage: true}), &server.RegisterRoutesConfig{ + AdminBearerToken: "interop-admin-token", + Engine: stubEngine{}, + BASMProvider: provider, + BASMLimits: basm.DefaultReadLimits(), + }) + listener, err := (&net.ListenConfig{}).Listen(context.Background(), "tcp", "127.0.0.1:0") + if err != nil { + return "", err + } + go func() { + if serveErr := app.Listener(listener); serveErr != nil { + fmt.Fprintln(os.Stderr, serveErr) + } + }() + // Keep the advertised host explicitly loopback, matching the bound listener. + return fmt.Sprintf("http://127.0.0.1:%d", listener.Addr().(*net.TCPAddr).Port), nil +} + +func newReadyService() (*engine.BASMReadService, error) { + txid, err := basm.ParseHash(interopGenesisID) + if err != nil { + return nil, err + } + blockHash, err := basm.ParseHash(interopGenesisBlock) + if err != nil { + return nil, err + } + txidFlag := true + transactionHash := chainhash.Hash(txid) + proof := transaction.NewMerklePath(0, [][]*transaction.PathElement{{{ + Offset: 0, + Hash: &transactionHash, + Txid: &txidFlag, + }}}).Bytes() + anchor := basm.Anchor{TopicBlockAnchor: basm.TopicBlockAnchor{ + Topic: interopTopic, + BlockHeight: 0, + BlockHash: blockHash, + BASMRoot: txid, + AdmittedCount: 1, + }} + anchor.TAC = basm.HashTACStep(basm.Hash{}, anchor.BlockHash, anchor.BASMRoot) + storage := interopStorage{view: interopView{anchor: anchor, txid: txid, proof: proof}} + headers := interopHeaders{header: engine.BASMCanonicalHeader{ + Height: 0, + BlockHash: blockHash, + MerkleRoot: txid, + TransactionCount: 1, + }} + return engine.NewBASMReadService(storage, headers, basm.DefaultReadLimits()) +} + +type interopStorage struct{ view interopView } + +func (s interopStorage) OpenBASMRead(_ context.Context, topic string, _ basm.ReadLimits) (engine.BASMReadView, error) { + if topic != "" && topic != interopTopic { + return nil, engine.ErrBASMNotFound + } + return s.view, nil +} + +type interopView struct { + anchor basm.Anchor + txid basm.Hash + proof []byte +} + +func (v interopView) Tip(context.Context) (*basm.Anchor, error) { + anchor := v.anchor + return &anchor, nil +} + +func (v interopView) Anchors(_ context.Context, from, to, _ uint32) ([]basm.Anchor, error) { + if from != 0 || to != 0 { + return nil, engine.ErrBASMNotReady + } + return []basm.Anchor{v.anchor}, nil +} + +func (v interopView) Admitted(_ context.Context, height, _ uint32) ([]basm.AdmittedTxRef, error) { + if height != 0 { + return nil, engine.ErrBASMNotReady + } + return []basm.AdmittedTxRef{{TxID: v.txid, BlockIndex: 0}}, nil +} + +func (v interopView) MerklePath(_ context.Context, txid basm.Hash, _ uint32) ([]byte, error) { + if txid != v.txid { + return nil, engine.ErrBASMNotFound + } + return append([]byte(nil), v.proof...), nil +} + +func (v interopView) RawTx(_ context.Context, txid basm.Hash, _ uint32) ([]byte, error) { + if txid != v.txid { + return nil, engine.ErrBASMNotFound + } + return hex.DecodeString(interopGenesisHex) +} + +func (interopView) CheckCurrent(context.Context) error { return nil } +func (interopView) Close() error { return nil } + +type interopHeaders struct{ header engine.BASMCanonicalHeader } + +func (h interopHeaders) CanonicalBASMHeader(_ context.Context, height uint32) (engine.BASMCanonicalHeader, error) { + if height != h.header.Height { + return engine.BASMCanonicalHeader{}, engine.ErrBASMNotReady + } + return h.header, nil +} + +type stubEngine struct{} + +func (stubEngine) Submit(_ context.Context, _ overlay.TaggedBEEF, _ engine.SumbitMode, onSteakReady engine.OnSteakReady) (overlay.Steak, error) { + if onSteakReady != nil { + onSteakReady(&overlay.Steak{}) + } + return overlay.Steak{}, nil +} +func (stubEngine) Lookup(_ context.Context, _ *lookup.LookupQuestion) (*lookup.LookupAnswer, error) { + return &lookup.LookupAnswer{}, nil +} +func (stubEngine) GetUTXOHistory(_ context.Context, _ *engine.Output, _ func(beef *transaction.Beef, outputIndex, currentDepth uint32) bool, _ uint32) (*engine.Output, error) { + return &engine.Output{}, nil +} +func (stubEngine) SyncAdvertisements(context.Context) error { return nil } +func (stubEngine) StartGASPSync(context.Context) error { return nil } +func (stubEngine) ProvideForeignSyncResponse(_ context.Context, _ *gasp.InitialRequest, _ string) (*gasp.InitialResponse, error) { + return &gasp.InitialResponse{}, nil +} +func (stubEngine) ProvideForeignGASPNode(_ context.Context, _, _ *transaction.Outpoint, _ string) (*gasp.Node, error) { + return &gasp.Node{}, nil +} +func (stubEngine) ListTopicManagers() map[string]*overlay.MetaData { + return map[string]*overlay.MetaData{} +} +func (stubEngine) ListLookupServiceProviders() map[string]*overlay.MetaData { + return map[string]*overlay.MetaData{} +} +func (stubEngine) GetDocumentationForLookupServiceProvider(string) (string, error) { + return "", nil +} +func (stubEngine) GetDocumentationForTopicManager(string) (string, error) { return "", nil } +func (stubEngine) HandleNewMerkleProof(context.Context, *chainhash.Hash, *transaction.MerklePath) error { + return nil +} diff --git a/packages/overlays/overlay/src/__tests/fixtures/brc136-independent.json b/packages/overlays/overlay/src/__tests/fixtures/brc136-independent.json new file mode 100644 index 000000000..1498e2a10 --- /dev/null +++ b/packages/overlays/overlay/src/__tests/fixtures/brc136-independent.json @@ -0,0 +1,185 @@ +{ + "specRevision": { + "brc136": "2733cd2950a739b3c977b95d652ff63e3773c40b", + "repoHead": "39a643ff148a8dcd23ec08986a8ddeb7d5713743" + }, + "hashEncoding": "lowercase display-order hex", + "byteOrder": [ + { + "name": "asymmetric-display-to-internal", + "display": "000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f", + "internal": "1f1e1d1c1b1a191817161514131211100f0e0d0c0b0a09080706050403020100" + } + ], + "merkle": [ + { + "name": "empty", + "txids": [], + "root": "0000000000000000000000000000000000000000000000000000000000000000", + "admissionListValid": true + }, + { + "name": "singleton-genesis-bitcoin-txid", + "txids": [ + "4a5e1e4baab89f3a32518a88c31bc87f618f76673e2cc77ab2127b7afdeda33b" + ], + "root": "4a5e1e4baab89f3a32518a88c31bc87f618f76673e2cc77ab2127b7afdeda33b", + "admissionListValid": true + }, + { + "name": "singleton-asymmetric-byte-order", + "txids": [ + "000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f" + ], + "root": "000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f", + "admissionListValid": true + }, + { + "name": "even-two-asymmetric", + "txids": [ + "000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f", + "ffeeddccbbaa99887766554433221100deadbeefcafebabe0123456789abcdef" + ], + "root": "668066643818b3695ed31c7cfd09dcc8aaeba2f6eb46e6360b041fb3254ccffc", + "admissionListValid": true + }, + { + "name": "even-four", + "txids": [ + "4a5e1e4baab89f3a32518a88c31bc87f618f76673e2cc77ab2127b7afdeda33b", + "000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f", + "ffeeddccbbaa99887766554433221100deadbeefcafebabe0123456789abcdef", + "112233445566778899aabbccddeeff00123456789abcdef00fedcba987654321" + ], + "root": "4c59e81df05211158cb925226168b4536aef5b6c7a7575197e09a77234a63114", + "admissionListValid": true + }, + { + "name": "odd-three", + "txids": [ + "000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f", + "ffeeddccbbaa99887766554433221100deadbeefcafebabe0123456789abcdef", + "112233445566778899aabbccddeeff00123456789abcdef00fedcba987654321" + ], + "root": "82853ce87598bf885ef5fb2e288d92581a1addc09bc651ea5e3bc2834867f7a0", + "admissionListValid": true + }, + { + "name": "tac-three-asymmetric", + "txids": [ + "ffeeddccbbaa99887766554433221100deadbeefcafebabe0123456789abcdef", + "112233445566778899aabbccddeeff00123456789abcdef00fedcba987654321", + "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef" + ], + "root": "bd162ecb4feb15a6489b8e7a6112dadfb1a699e691077b53dabf4cf79462173c", + "admissionListValid": true + }, + { + "name": "odd-five", + "txids": [ + "4a5e1e4baab89f3a32518a88c31bc87f618f76673e2cc77ab2127b7afdeda33b", + "000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f", + "ffeeddccbbaa99887766554433221100deadbeefcafebabe0123456789abcdef", + "112233445566778899aabbccddeeff00123456789abcdef00fedcba987654321", + "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef" + ], + "root": "a6ff6556e3c63ba45b8872ec460ac1e8c322574512d94f061e1639e669b74b7e", + "admissionListValid": true + }, + { + "name": "unsorted-asymmetric-byte-values", + "txids": [ + "ffeeddccbbaa99887766554433221100deadbeefcafebabe0123456789abcdef", + "4a5e1e4baab89f3a32518a88c31bc87f618f76673e2cc77ab2127b7afdeda33b", + "000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f" + ], + "root": "e1c6afbb7257b4a4de469c22d715825b983cbb80b175fa6837244d3b575fbdaa", + "admissionListValid": true + }, + { + "name": "duplicate-looking-three-illustrative", + "txids": [ + "000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f", + "ffeeddccbbaa99887766554433221100deadbeefcafebabe0123456789abcdef", + "112233445566778899aabbccddeeff00123456789abcdef00fedcba987654321" + ], + "root": "82853ce87598bf885ef5fb2e288d92581a1addc09bc651ea5e3bc2834867f7a0", + "admissionListValid": true, + "illustrativeOnly": true + }, + { + "name": "repeated-last-four-illustrative-invalid", + "txids": [ + "000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f", + "ffeeddccbbaa99887766554433221100deadbeefcafebabe0123456789abcdef", + "112233445566778899aabbccddeeff00123456789abcdef00fedcba987654321", + "112233445566778899aabbccddeeff00123456789abcdef00fedcba987654321" + ], + "root": "82853ce87598bf885ef5fb2e288d92581a1addc09bc651ea5e3bc2834867f7a0", + "admissionListValid": false, + "illustrativeOnly": true + } + ], + "tac": [ + { + "name": "genesis-height-zero-includes-empty-heights", + "genesisHeight": 0, + "anchors": [ + { + "blockHeight": 0, + "blockHash": "0000000000000000000000000000000000000000000000000000000000000000", + "basmRoot": "0000000000000000000000000000000000000000000000000000000000000000", + "admittedCount": 0, + "rootSource": "empty", + "expectedTac": "3a464e1e43410c7add1dd81c3f10486f41eb473bb43e8d64feca3c7f0c8028d3" + }, + { + "blockHeight": 1, + "blockHash": "1111111111111111111111111111111111111111111111111111111111111111", + "basmRoot": "668066643818b3695ed31c7cfd09dcc8aaeba2f6eb46e6360b041fb3254ccffc", + "admittedCount": 2, + "rootSource": "even-two-asymmetric", + "expectedTac": "3771c04594ce4cf996178d68c3e4d640647b85d008d86a7613ead721c7c64875" + }, + { + "blockHeight": 2, + "blockHash": "2222222222222222222222222222222222222222222222222222222222222222", + "basmRoot": "0000000000000000000000000000000000000000000000000000000000000000", + "admittedCount": 0, + "rootSource": "empty", + "expectedTac": "f5652a4a6f978eac1278b61b7873740ee7109a5f6f05df27032b453eb2687232" + } + ] + }, + { + "name": "nonzero-genesis-mixed-anchors", + "genesisHeight": 100, + "anchors": [ + { + "blockHeight": 100, + "blockHash": "13579bdf2468ace00112233445566778899aabbccddeeff00fedcba987654321", + "basmRoot": "4a5e1e4baab89f3a32518a88c31bc87f618f76673e2cc77ab2127b7afdeda33b", + "admittedCount": 1, + "rootSource": "singleton-genesis-bitcoin-txid", + "expectedTac": "f5f755f70e8aea69a4f7f5c2d49b782e29de844e2aff24f0089d43d6697655ba" + }, + { + "blockHeight": 101, + "blockHash": "89abcdef0123456789abcdef0123456789abcdef0123456789abcdef01234567", + "basmRoot": "bd162ecb4feb15a6489b8e7a6112dadfb1a699e691077b53dabf4cf79462173c", + "admittedCount": 3, + "rootSource": "tac-three-asymmetric", + "expectedTac": "da1d66659e44f5eb7abb280cf7a323d05ba72b96f78d74de03694bebf3d7af66" + }, + { + "blockHeight": 102, + "blockHash": "abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789", + "basmRoot": "0000000000000000000000000000000000000000000000000000000000000000", + "admittedCount": 0, + "rootSource": "empty", + "expectedTac": "89f4bd216e66f821ba9cf8da487109efb595f0461a1788dc878040c58a5f2688" + } + ] + } + ] +} diff --git a/packages/overlays/topics/CHANGELOG.md b/packages/overlays/topics/CHANGELOG.md index 527acbf56..eb0540454 100644 --- a/packages/overlays/topics/CHANGELOG.md +++ b/packages/overlays/topics/CHANGELOG.md @@ -10,6 +10,8 @@ All notable changes to this project will be documented in this file. The format ## [Unreleased] - Updates the packed workspace dependency candidate for the additive overlay persistence contract. Runtime behavior and defaults are unchanged; no consumer migration is required. +- Advances the packed overlay dependency candidate for BASM validation hardening. + Package runtime behavior is unchanged; no consumer migration is required. ### Added diff --git a/packages/overlays/topics/package.json b/packages/overlays/topics/package.json index 4500673e1..bf49900ff 100644 --- a/packages/overlays/topics/package.json +++ b/packages/overlays/topics/package.json @@ -1,6 +1,6 @@ { "name": "@bsv/overlay-topics", - "version": "1.8.1", + "version": "1.8.2", "sideEffects": false, "engines": { "node": ">=22" diff --git a/packages/sdk/README.md b/packages/sdk/README.md index b356ee002..8b277ff23 100644 --- a/packages/sdk/README.md +++ b/packages/sdk/README.md @@ -100,6 +100,11 @@ For a more detailed tutorial and advanced examples, check our [Documentation](#d - **Merkle Proof Verification**: Tools for representing and verifying merkle proofs, adhering to various serialization standards. + BUMP transaction offsets retain their exact nonnegative safe-integer domain, + including positions above 32 bits. Root calculation, proof extraction, + combination, and trimming use the same full-width arithmetic. Offsets outside + that domain are rejected; no new wire format or application migration is needed. + - **Serializable SPV Structures**: Structures and interfaces for full SPV verification. - **Secure Encryption and Signed Messages**: Enhanced mechanisms for encryption and digital signatures, replacing outdated methods. diff --git a/packages/sdk/src/transaction/MerklePath.ts b/packages/sdk/src/transaction/MerklePath.ts index ee3fd9a4e..1eb80ddf0 100644 --- a/packages/sdk/src/transaction/MerklePath.ts +++ b/packages/sdk/src/transaction/MerklePath.ts @@ -10,6 +10,23 @@ export interface MerklePathLeaf { duplicate?: boolean } +function assertOffset(offset: number): void { + if (!Number.isSafeInteger(offset) || offset < 0) { + throw new Error('Invalid offset') + } +} + +// JavaScript bitwise operators truncate offsets to signed 32 bits. Division by +// a power of two is exact over the supported safe-integer offset domain. +const offsetAtHeight = (offset: number, height: number): number => Math.floor(offset / 2 ** height) + +const siblingOf = (offset: number): number => offset % 2 === 0 ? offset + 1 : offset - 1 + +const sameNodeAtHeight = (index: number, maxOffset: number, height: number): boolean => + offsetAtHeight(index, height) === offsetAtHeight(maxOffset, height) + +const offsetTreeHeight = (offset: number): number => offset === 0 ? 0 : offset.toString(2).length + function hashPair(left: string | undefined, right: string | undefined): string { return toHex(hash256(toArray((left ?? '') + (right ?? ''), 'hex').reverse()).reverse()) } @@ -191,6 +208,7 @@ export default class MerklePath { } const offsetsAtThisHeight = new Set() leaves.forEach(leaf => { + assertOffset(leaf.offset) if (offsetsAtThisHeight.has(leaf.offset)) { throw new Error(`Duplicate offset: ${leaf.offset}, at height: ${height}`) } @@ -198,7 +216,7 @@ export default class MerklePath { if (height === 0) { if (leaf.duplicate !== true) { for (let h = 1; h < this.path.length; h++) { - legalOffsets[h].add((leaf.offset >> h) ^ 1) + legalOffsets[h].add(siblingOf(offsetAtHeight(leaf.offset, h))) } } } else if (legalOffsetsOnly && !legalOffsets[height].has(leaf.offset)) { @@ -317,14 +335,14 @@ export default class MerklePath { if (typeof txid !== 'string') throw new TypeError('Transaction ID is undefined') const index = this.indexOf(txid) if (this.path.length === 1 && this.path[0].length === 1) return txid - const treeHeight = Math.max(this.path.length, 32 - Math.clz32(maxOffset)) + const treeHeight = Math.max(this.path.length, offsetTreeHeight(maxOffset)) let workingHash = txid for (let height = 0; height < treeHeight; height++) { - const nodeKey = `${height}:${index >> height}` + const nodeKey = `${height}:${offsetAtHeight(index, height)}` const cachedRoot = cachedMerkleRoot(nodeKey, workingHash, treeHeight, nodeHashCache) if (cachedRoot != null) return cachedRoot nodeHashCache.set(nodeKey, workingHash) - const offset = (index >> height) ^ 1 + const offset = siblingOf(offsetAtHeight(index, height)) const leaf = this.cachedFindLeaf(height, offset, sourceIndex, hashCache, maxOffset) workingHash = nextCachedHash( workingHash, @@ -332,32 +350,13 @@ export default class MerklePath { offset, index, height, - this.path.length === 1 && index >> height === maxOffset >> height + this.path.length === 1 && sameNodeAtHeight(index, maxOffset, height) ) } nodeHashCache.set(`${treeHeight}:0`, workingHash) return workingHash } - private nextRootHash( - workingHash: string, - index: number, - height: number, - maxOffset: number - ): string { - const offset = (index >> height) ^ 1 - const leaf = this.findOrComputeLeaf(height, offset) - if (leaf == null) { - const isLastOddNode = this.path.length === 1 && index >> height === maxOffset >> height - if (isLastOddNode) return hashPair(workingHash, workingHash) - throw new Error(`Missing hash for index ${index} at height ${height}`) - } - if (leaf.duplicate === true) return hashPair(workingHash, workingHash) - return offset % 2 === 1 - ? hashPair(leaf.hash, workingHash) - : hashPair(workingHash, leaf.hash) - } - /** * Computes the Merkle root from the provided transaction ID. * @@ -387,10 +386,18 @@ export default class MerklePath { // (path.length === 1 or intermediate levels are empty/trimmed), we need to compute up // to the height implied by the highest offset present in path[0]. const maxOffset = this.path[0].reduce((max, l) => Math.max(max, l.offset), 0) - const treeHeight = Math.max(this.path.length, 32 - Math.clz32(maxOffset)) + const treeHeight = Math.max(this.path.length, offsetTreeHeight(maxOffset)) for (let height = 0; height < treeHeight; height++) { - workingHash = this.nextRootHash(workingHash, index, height, maxOffset) + const offset = siblingOf(offsetAtHeight(index, height)) + workingHash = nextCachedHash( + workingHash, + this.findOrComputeLeaf(height, offset), + offset, + index, + height, + this.path.length === 1 && sameNodeAtHeight(index, maxOffset, height) + ) } return workingHash } @@ -404,7 +411,7 @@ export default class MerklePath { * @param offset */ findOrComputeLeaf(height: number, offset: number): MerklePathLeaf | undefined { - const hash = (m: string): string => toHex(hash256(toArray(m, 'hex').reverse()).reverse()) + assertOffset(offset) let leaf: MerklePathLeaf | undefined = height < this.path.length ? this.path[height].find(l => l.offset === offset) : undefined @@ -414,7 +421,9 @@ export default class MerklePath { if (height === 0) return undefined const h = height - 1 - const l = offset << 1 + const l = offset * 2 + // No descendant of this node is representable by a supported leaf offset. + if (!Number.isSafeInteger(l)) return undefined const leaf0 = this.findOrComputeLeaf(h, l) if (leaf0?.hash == null || leaf0.hash === '') return undefined @@ -423,13 +432,13 @@ export default class MerklePath { if (leaf1?.hash == null) { // Explicit duplicate marker — duplicate leaf0 regardless of path depth. if (leaf1?.duplicate === true) { - return { offset, hash: hash(leaf0.hash + leaf0.hash) } + return { offset, hash: hashPair(leaf0.hash, leaf0.hash) } } // For single-level paths, leaf0 may be the last odd node at height h — duplicate it. if (this.path.length === 1) { const maxOffset0 = this.path[0].reduce((max, lf) => Math.max(max, lf.offset), 0) - if (l === maxOffset0 >> h) { - return { offset, hash: hash(leaf0.hash + leaf0.hash) } + if (l === offsetAtHeight(maxOffset0, h)) { + return { offset, hash: hashPair(leaf0.hash, leaf0.hash) } } } return undefined @@ -437,9 +446,9 @@ export default class MerklePath { let workinghash: string if (leaf1.duplicate === true) { - workinghash = hash(leaf0.hash + leaf0.hash) + workinghash = hashPair(leaf0.hash, leaf0.hash) } else { - workinghash = hash((leaf1.hash ?? '') + (leaf0.hash ?? '')) + workinghash = hashPair(leaf1.hash, leaf0.hash) } leaf = { offset, @@ -530,7 +539,7 @@ export default class MerklePath { const nextComputedOffsets = (cos: number[]): number[] => { const ncos: number[] = [] for (const o of cos) { - pushIfNew(o >> 1, ncos) + pushIfNew(offsetAtHeight(o, 1), ncos) } return ncos } @@ -545,7 +554,7 @@ export default class MerklePath { const n = this.path[0][l] if (n.txid === true) { // level 0 must enable computing level 1 for txid nodes - pushIfNew(n.offset >> 1, computedOffsets) + pushIfNew(offsetAtHeight(n.offset, 1), computedOffsets) } else { const isOdd = n.offset % 2 === 1 const peer = this.path[0][l + (isOdd ? -1 : 1)] @@ -577,7 +586,6 @@ export default class MerklePath { const key = `${height}:${offset}` if (hashCache.has(key)) return hashCache.get(key) - const doHash = (m: string): string => toHex(hash256(toArray(m, 'hex').reverse()).reverse()) let leaf: MerklePathLeaf | undefined = height < sourceIndex.length ? sourceIndex[height].get(offset) : undefined @@ -593,7 +601,8 @@ export default class MerklePath { } const h = height - 1 - const l = offset << 1 + const l = offset * 2 + if (!Number.isSafeInteger(l)) return undefined const leaf0 = this.cachedFindLeaf(h, l, sourceIndex, hashCache, maxOffset) if (leaf0?.hash == null || leaf0.hash === '') { hashCache.set(key, undefined) @@ -602,8 +611,8 @@ export default class MerklePath { const leaf1 = this.cachedFindLeaf(h, l + 1, sourceIndex, hashCache, maxOffset) if (leaf1?.hash == null) { - if (leaf1?.duplicate === true || (this.path.length === 1 && l === maxOffset >> h)) { - leaf = { offset, hash: doHash(leaf0.hash + leaf0.hash) } + if (leaf1?.duplicate === true || (this.path.length === 1 && l === offsetAtHeight(maxOffset, h))) { + leaf = { offset, hash: hashPair(leaf0.hash, leaf0.hash) } hashCache.set(key, leaf) return leaf } @@ -613,8 +622,8 @@ export default class MerklePath { const workinghash = leaf1.duplicate === true - ? doHash(leaf0.hash + leaf0.hash) - : doHash((leaf1.hash ?? '') + (leaf0.hash ?? '')) + ? hashPair(leaf0.hash, leaf0.hash) + : hashPair(leaf1.hash, leaf0.hash) leaf = { offset, hash: workinghash } hashCache.set(key, leaf) return leaf @@ -648,7 +657,7 @@ export default class MerklePath { const originalRoot = this.computeRoot() const maxOffset = this.path[0].reduce((max, l) => Math.max(max, l.offset), 0) - const treeHeight = Math.max(this.path.length, 32 - Math.clz32(maxOffset)) + const treeHeight = Math.max(this.path.length, offsetTreeHeight(maxOffset)) const sourceIndex = this.createSourceLeafIndex() const hashCache = new Map() @@ -720,7 +729,7 @@ export default class MerklePath { // Level 0: the txid leaf + its sibling. neededPerLevel[0].set(txOffset, { offset: txOffset, txid: true, hash: txid }) - const levelZeroSiblingOffset = txOffset ^ 1 + const levelZeroSiblingOffset = siblingOf(txOffset) if (!neededPerLevel[0].has(levelZeroSiblingOffset)) { const sibling = this.cachedFindLeaf(0, levelZeroSiblingOffset, sourceIndex, hashCache, maxOffset) if (sibling != null) neededPerLevel[0].set(levelZeroSiblingOffset, sibling) @@ -728,12 +737,12 @@ export default class MerklePath { // Higher levels need only the sibling at each height. for (let h = 1; h < treeHeight; h++) { - const siblingOffset = (txOffset >> h) ^ 1 + const siblingOffset = siblingOf(offsetAtHeight(txOffset, h)) if (neededPerLevel[h].has(siblingOffset)) continue const sibling = this.cachedFindLeaf(h, siblingOffset, sourceIndex, hashCache, maxOffset) if (sibling != null) { neededPerLevel[h].set(siblingOffset, sibling) - } else if (txOffset >> h === maxOffset >> h) { + } else if (sameNodeAtHeight(txOffset, maxOffset, h)) { neededPerLevel[h].set(siblingOffset, { offset: siblingOffset, duplicate: true }) } } diff --git a/packages/sdk/src/transaction/__tests/MerklePath.safeOffsets.test.ts b/packages/sdk/src/transaction/__tests/MerklePath.safeOffsets.test.ts new file mode 100644 index 000000000..e0224db77 --- /dev/null +++ b/packages/sdk/src/transaction/__tests/MerklePath.safeOffsets.test.ts @@ -0,0 +1,234 @@ +import { createHash } from 'node:crypto' +import MerklePath, { type MerklePathLeaf } from '../MerklePath' + +const TARGET_PREFIX = 'safe-offset-target' + +function sha256d(bytes: Uint8Array): Buffer { + const first = createHash('sha256').update(bytes).digest() + return createHash('sha256').update(first).digest() +} + +function displayToInternal(hash: string): Buffer { + return Buffer.from(hash, 'hex').reverse() +} + +function internalToDisplay(bytes: Uint8Array): string { + return Buffer.from(bytes).reverse().toString('hex') +} + +function hashPair(left: string, right: string): string { + return internalToDisplay( + sha256d(Buffer.concat([displayToInternal(left), displayToInternal(right)])) + ) +} + +function labelledHash(label: string): string { + return createHash('sha256').update(label).digest('hex') +} + +function bitLength(value: bigint): number { + let remaining = value + let length = 0 + while (remaining > 0n) { + remaining >>= 1n + length++ + } + return length +} + +function compact(value: bigint): number[] { + if (value < 253n) return [Number(value)] + if (value < 0x10000n) return [0xfd, Number(value & 0xffn), Number((value >> 8n) & 0xffn)] + if (value < 0x100000000n) { + return [ + 0xfe, + ...Array.from({ length: 4 }, (_, index) => Number((value >> BigInt(index * 8)) & 0xffn)) + ] + } + return [ + 0xff, + ...Array.from({ length: 8 }, (_, index) => Number((value >> BigInt(index * 8)) & 0xffn)) + ] +} + +interface SyntheticPath { + readonly target: string + readonly sibling: string + readonly index: number + readonly depth: number + readonly path: MerklePathLeaf[][] + readonly root: string +} + +/** + * Builds a bounded mathematical proof: only the target, its paired level-zero + * sibling, and one sibling per higher level are represented. It does not claim + * to model a mined block with 2^n transactions. + */ +function pairedPath(originalIndex: bigint): SyntheticPath { + const index = Number(originalIndex) + const depth = bitLength(originalIndex) + 1 + const target = labelledHash(`${TARGET_PREFIX}:${originalIndex}:target`) + const sibling = labelledHash(`${TARGET_PREFIX}:${originalIndex}:sibling:0`) + const path: MerklePathLeaf[][] = [ + [ + { offset: index, hash: target, txid: true }, + { offset: Number(originalIndex ^ 1n), hash: sibling, txid: true } + ].sort((left, right) => left.offset - right.offset) + ] + let root = originalIndex % 2n === 0n ? hashPair(target, sibling) : hashPair(sibling, target) + + for (let height = 1; height < depth; height++) { + const node = originalIndex >> BigInt(height) + const siblingHash = labelledHash(`${TARGET_PREFIX}:${originalIndex}:sibling:${height}`) + path.push([{ offset: Number(node ^ 1n), hash: siblingHash }]) + root = node % 2n === 0n ? hashPair(root, siblingHash) : hashPair(siblingHash, root) + } + + return { target, sibling, index, depth, path, root } +} + +function canonicalOddWidthPath(originalIndex: bigint): SyntheticPath { + const index = Number(originalIndex) + const target = labelledHash(`${TARGET_PREFIX}:${originalIndex}:odd-target`) + const path: MerklePathLeaf[][] = [] + let node = originalIndex + let width = originalIndex + 1n + let root = target + + // The target is the final node in each odd-width level. A canonical BUMP + // duplicates it at the next offset until the width reduces to two nodes. + while (width > 2n) { + if (width % 2n !== 1n || node !== width - 1n) + throw new Error('Invalid synthetic odd-width fixture') + const duplicate = { offset: Number(node ^ 1n), duplicate: true } + if (path.length === 0) { + path.push( + [{ offset: index, hash: target, txid: true }, duplicate].sort( + (left, right) => left.offset - right.offset + ) + ) + } else { + path.push([duplicate]) + } + root = hashPair(root, root) + node >>= 1n + width = (width + 1n) >> 1n + } + const leftRoot = labelledHash(`${TARGET_PREFIX}:${originalIndex}:odd-left-root`) + path.push([{ offset: Number(node ^ 1n), hash: leftRoot }]) + root = node % 2n === 0n ? hashPair(root, leftRoot) : hashPair(leftRoot, root) + + return { target, sibling: target, index, depth: path.length, path, root } +} + +describe('MerklePath safe offsets', () => { + it('keeps a low-offset control proof compatible', () => { + const fixture = pairedPath(5n) + const merklePath = new MerklePath(777, fixture.path) + + expect(merklePath.computeRoot(fixture.target)).toBe(fixture.root) + expect(MerklePath.fromHex(merklePath.toHex()).computeRoot(fixture.target)).toBe(fixture.root) + }) + + it.each([ + (1n << 31n) - 1n, + 1n << 31n, + (1n << 32n) - 1n, + 1n << 32n, + (1n << 53n) - 2n, + (1n << 53n) - 1n + ])('preserves the full synthetic proof at original offset %s', originalIndex => { + const fixture = pairedPath(originalIndex) + const merklePath = new MerklePath(777, fixture.path) + + expect(merklePath.path).toHaveLength(fixture.depth) + expect(merklePath.path[0].map(leaf => leaf.offset)).toEqual( + fixture.path[0].map(leaf => leaf.offset) + ) + expect(merklePath.computeRoot(fixture.target)).toBe(fixture.root) + expect(merklePath.computeRoot(fixture.sibling)).toBe(fixture.root) + + const roundTripped = MerklePath.fromHex(merklePath.toHex()) + expect(roundTripped.path[0].map(leaf => leaf.offset)).toEqual( + fixture.path[0].map(leaf => leaf.offset) + ) + expect(roundTripped.computeRoot(fixture.target)).toBe(fixture.root) + expect(roundTripped.computeRoot(fixture.sibling)).toBe(fixture.root) + }) + + it.each([31, 32, 52])('computes a canonical odd-width proof at 2^%i', power => { + const fixture = canonicalOddWidthPath(1n << BigInt(power)) + const merklePath = new MerklePath(777, fixture.path) + + expect(fixture.depth).toBe(power + 1) + expect(merklePath.computeRoot(fixture.target)).toBe(fixture.root) + }) + + it.each([31, 32, 52])('round-trips a canonical odd-width proof at 2^%i', power => { + const fixture = canonicalOddWidthPath(1n << BigInt(power)) + const roundTripped = MerklePath.fromHex(new MerklePath(777, fixture.path).toHex()) + + expect(roundTripped.path).toEqual(fixture.path) + expect(roundTripped.computeRoot(fixture.target)).toBe(fixture.root) + }) + + it('computes recursive leaves, extracts, combines, and trims a pruned high-offset proof', () => { + const fixture = pairedPath((1n << 53n) - 1n) + const full = new MerklePath(777, fixture.path) + const parentOffset = Math.floor(fixture.index / 2) + const expectedParent = hashPair(fixture.sibling, fixture.target) + + expect(full.findOrComputeLeaf(1, parentOffset)?.hash).toBe(expectedParent) + + const first = full.extract([fixture.target]) + const second = full.extract([fixture.sibling]) + expect(first.computeRoot(fixture.target)).toBe(fixture.root) + expect(second.computeRoot(fixture.sibling)).toBe(fixture.root) + + first.combine(second) + first.trim() + expect(first.computeRoot(fixture.target)).toBe(fixture.root) + expect(first.computeRoot(fixture.sibling)).toBe(fixture.root) + + const roundTripped = MerklePath.fromBinary(first.toBinaryUint8Array()) + expect(roundTripped.computeRoot(fixture.target)).toBe(fixture.root) + expect(roundTripped.computeRoot(fixture.sibling)).toBe(fixture.root) + }) + + it('computes safe high offsets when constructor root validation is deferred', () => { + const fixture = pairedPath(1n << 32n) + const merklePath = new MerklePath(777, fixture.path, true, false) + + expect(merklePath.computeRoot(fixture.target)).toBe(fixture.root) + }) + + it.each([Number.MAX_SAFE_INTEGER + 1, -1, 1.5, Number.NaN, Number.POSITIVE_INFINITY])( + 'rejects unsafe offset %p even when root validation is disabled', + offset => { + expect( + () => + new MerklePath( + 777, + [[{ offset, hash: labelledHash(`invalid:${offset}`), txid: true }]], + true, + false + ) + ).toThrow() + } + ) + + it('rejects an unsafe offset encoded directly as a BUMP varint', () => { + const unsafeOffset = 1n << 53n + const bytes = Uint8Array.from([ + ...compact(777n), + 1, + ...compact(1n), + ...compact(unsafeOffset), + 2, + ...displayToInternal(labelledHash('unsafe-wire')) + ]) + + expect(() => MerklePath.fromBinary(bytes, true, false)).toThrow() + }) +})