diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index 6241305c..061bdbb5 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -1,5 +1,9 @@ +## Base branch + +- [ ] This pull request targets `develop`. `master` only takes release merges, and a pull request into it from a fork cannot be merged. + ## What this changes diff --git a/CHANGELOG.md b/CHANGELOG.md index 49376e92..c47e9a73 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,27 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +## [0.24.0] - 2026-10-17 + +### Added +- Added the mainnet launch gate section, its catalogue row and the prelaunch seed sources, with the flag-day page regenerated. +- Added the order-match price-cross, nested execute gas sum, settlement bounds and slash gates to the reference registry and catalogue. +- Added the wallet UI style guide, device layout checklist and layout, network label and shell sizing rules. + +### Changed +- Corrected the wallet README, architecture, keys, signing, security and threat model pages against current source. +- Documented the new indexer, explorer, encoder, hub and sync settings, including the indexer balance delta mode. +- Described the oracle aggregation rule the hub applies and the websocket feed reorg, replay and resubscribe pacing. +- Armed ORDER_MATCH_PRICE_CROSS_ACTIVATION on BTC:testnet at 156908. +- Armed ORDER_MATCH_PRICE_CROSS_ACTIVATION on LTC:testnet at 4927400. +- Armed ORDER_MATCH_PRICE_CROSS_ACTIVATION on DOGE:testnet at 68005165. +- Armed REMOTE_TOKEN_ADMISSION_ACTIVATION on BTC:testnet at 156908. +- Armed REMOTE_TOKEN_ADMISSION_ACTIVATION on LTC:testnet at 4927400. +- Armed REMOTE_TOKEN_ADMISSION_ACTIVATION on DOGE:testnet at 68005165. + +### Fixed +- Recorded the six reclaimed chain codes as reserved outright and registered the new encoder refusal reasons. + ## [0.23.0] - 2026-10-11 ### Added diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 4c923ee1..3d752876 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -137,7 +137,7 @@ All byte-level encoding examples and on-chain data samples in the spec must matc Match the existing log style: a concise subject line, then a short body explaining what changed and why. -- Branch off `develop` and keep history linear (rebase, don't merge). `develop` is where work lands; `master` only ever receives release merges (see [`operations/release-process.md`](./operations/release-process.md)). +- Branch off `develop` and keep history linear (rebase, don't merge). `develop` is where work lands; `master` only ever receives release merges, and a pull request into `master` from a fork or any other branch fails the `base-branch guard` check (see [`operations/release-process.md`](./operations/release-process.md)). - One logical change per commit; don't batch unrelated edits. - **No `Co-Authored-By` trailers.** This is a project policy. - **Never `--no-verify`.** If a hook fails, fix the cause; don't bypass it. diff --git a/README.md b/README.md index f41ac959..a84686f1 100644 --- a/README.md +++ b/README.md @@ -38,7 +38,7 @@ A blockchain-agnostic token protocol currently running on Bitcoin, Litecoin, and | [**xchain-vm**](https://github.com/XChain-Platform/xchain-vm/) | Sandboxed JavaScript virtual machine for on-chain smart contracts with gas metering, deterministic execution, and reorg-safe state | | [**xchain-contracts**](https://github.com/XChain-Platform/xchain-contracts/) | MIT-licensed template library: audited example smart contracts, reusable patterns, a no-code policy generator, and a CLI, deployed via the ordinary `DEPLOY` action | | [**xchain-sdk**](https://github.com/XChain-Platform/xchain-sdk/) | Developer SDK: builders for all 32 developer-invocable actions, 100+ explorer query methods, smart contract support, live WebSocket events, batch builder, PSBT generation | -| [**xchain-wallet**](https://github.com/XChain-Platform/xchain-wallet/) | Reference self-custodial multi-chain wallet: browser, Chrome extension, Electron desktop, and Capacitor mobile (Android shipped, iOS later) from a single codebase; software + Trezor + Ledger + remote signers (multisig runs as flows over these); full DEX, messaging, contracts, staking, and `window.xchain` dApp bridge | +| [**xchain-wallet**](https://github.com/XChain-Platform/xchain-wallet/) | Reference self-custodial multi-chain wallet: browser, Chrome extension, Electron desktop, and Capacitor mobile (Android shipped, iOS pending App Store release) from a single codebase; software + Trezor + Ledger + remote signers (multisig runs as flows over these); full DEX, messaging, contracts, staking, and `window.xchain` dApp bridge | | [**xchain-regtest-miner**](https://github.com/XChain-Platform/xchain-regtest-miner/) | Auto-mines blocks for regtest development environments | | [**xchain-e2e-test**](https://github.com/XChain-Platform/xchain-e2e-test/) | Full-stack Mocha test suite running against a live regtest deployment | ## Legal diff --git a/ai-agents/agent-wallets.md b/ai-agents/agent-wallets.md index f19842e1..538dd8fb 100644 --- a/ai-agents/agent-wallets.md +++ b/ai-agents/agent-wallets.md @@ -7,86 +7,105 @@ A commercial license is available, contact legal@dankest.llc. # Giving an AI Agent a Wallet, Safely -If an AI agent is going to move real value, it should only be able to move the value you decided it can move: on the actions you chose, to the places you allow, at a rate you set. The SDK's **agent session** exists for exactly that. +If an AI agent is going to move real value, it should only be able to move the value you decided it can move: through the actions you chose, to the places you allow, and at a rate you set. The SDK's **agent session** provides those guardrails. ## The idea -A normal wallet session can do anything the key can do. An agent session wraps the same key with a policy that is checked before every action leaves the SDK: +A normal wallet session can do anything the key can do. An agent session wraps the same key with a policy that is checked before each submission is signed or broadcast: ```js -const sdk = new XChainSDK({ network: 'dogecoin-testnet' }); +const sdk = new XChainSDK({ network: 'bitcoin-testnet' }); const agent = sdk.agentSession(wif, { // Nothing is allowed unless you list it. - allowedActions: ['SEND', 'EXECUTE', 'COINPAY'], + allowedActions: ['SEND'], - // Where funds may go (optional; omit to allow any destination). - allowedDestinations: ['DTqQ...storefront', 'DBgH...treasury'], + // Where token SEND actions may go (optional; omit to allow any destination). + allowedDestinations: ['tb1q...storefront', 'tb1q...treasury'], // The most a single action may move. - maxPerAction: { SEND: { MYTOKEN: '100', '*': '10' } }, // '*' = any other token + maxPerAction: { SEND: { MYTOKEN: '100', '*': '10' } }, // '*' = any other token // The most the agent may move in total, per rolling window. maxPerWindow: { hours: 24, perTick: { MYTOKEN: '500' }, maxActions: 50 }, - // Above this, a human (or supervising process) must say yes. + // Above this, a human or supervising process must say yes. confirmAbove: { perTick: { '*': '50' }, - handler: async (ctx) => await askTheOperator(ctx), // ctx: action, tick, amount, destinations, address, windowUsage + handler: async (ctx) => await askTheOperator(ctx), + // ctx: action, tick, amount, destinations, address, windowUsage }, - // See every denial (alerting, logs). + // Observe policy denials raised through the session's denial path. onPolicyViolation: (v) => console.error('agent blocked:', v.code, v.message), // Override the default usage-state path (~/.xchain/agent-usage-
.json). - // The file persists window usage across restarts so a crash-loop cannot reset caps. + // The file persists window and idempotency records across restarts. stateFile: '/var/lib/myapp/agent-usage.json', }); -await agent.send({ tick: 'MYTOKEN', amount: '5', destination: 'DTqQ...storefront' }); +await agent.send( + { tick: 'MYTOKEN', amount: '5', destination: 'tb1q...storefront' }, + {}, + { idempotencyKey: 'store-order-1842' }, +); ``` -Every successful action's result carries `result.policy` (what was checked and how much of the window is used) so the agent can reason about its own remaining budget instead of discovering limits by hitting them. +The idempotency key identifies one intended submission. Reuse the same key when retrying that submission. Use a new key only when you intend to make another payment. `AgentSession` refuses an unkeyed submission by default; `allowUnkeyedSubmits: true` is an explicit opt-out. + +Every successful action's result carries `result.policy`, including what was checked and how much of the window is used. The agent can inspect its remaining budget instead of discovering limits by hitting them. ## What happens on a violation -The action is refused **before anything is signed or broadcast**, with a typed `SDKPolicyError` whose `code` says exactly why: +The action is refused before signing or broadcast. Construction and submission failures use a typed `SDKPolicyError` whose `code` identifies the reason: | Code | Meaning | |------|---------| +| `POLICY_INVALID` | The policy is malformed, has no allowed actions, or has neither an enforceable hard cap nor `confirmAbove` without `allowUnbounded: true` | +| `POLICY_HALTED` | The session is paused, or its kill-switch file is present or unreadable | | `POLICY_ACTION_DENIED` | Action type not in `allowedActions` | +| `POLICY_AMOUNT_INVALID` | Amount is not a canonical non-negative decimal | | `POLICY_DESTINATION_DENIED` | Destination not in `allowedDestinations` | -| `POLICY_UNRESOLVED_DESTINATION` | A MINT, MESSAGE or SWEEP destination arrived as an address index reference (`^`), the SDK's default compaction of an already-indexed address, which `allowedDestinations` cannot match. It is refused rather than resolved; list the destination by address and submit with `compactAddresses` disabled. | +| `POLICY_DESTINATION_UNENFORCEABLE` | The decoded action version has no destination field that `allowedDestinations` can constrain | +| `POLICY_UNRESOLVED_DESTINATION` | Destination is an address index reference (`^`) that the allowlist cannot match | | `POLICY_AMOUNT_EXCEEDED` | Single-action amount exceeds the per-action cap | | `POLICY_WINDOW_AMOUNT_EXCEEDED` | Rolling-window token total would breach the cap | -| `POLICY_WINDOW_COUNT_EXCEEDED` | Rolling window already holds `maxActions` actions | -| `POLICY_CONFIRMATION_DENIED` | Amount was above `confirmAbove` threshold and the handler returned false | -| `POLICY_UNBOUNDED_ACTION` | The action moves an amount the policy cannot measure from the action alone (SWEEP drains whole balances; AIRDROP/DIVIDEND multiply by an off-chain recipient set) while an amount cap is set. Remove the cap or disallow the action. | -| `POLICY_UNRESOLVED_TICK` | The action references a token by its `^` wire form and the policy has tick-scoped limits it cannot bind to an id. Declare the token in `tickIds` (see the co-signer section) or submit with `compactTickers` disabled. | -| `POLICY_STATE_CORRUPT` | Usage-state file is unreadable or structurally invalid: indicates a corrupt state file, not a policy denial. Do not retry; inspect and remove or repair the file deliberately to recover. | +| `POLICY_WINDOW_COUNT_EXCEEDED` | This action would exceed `maxActions` in the rolling window | +| `POLICY_CONFIRMATION_DENIED` | Amount is above the `confirmAbove` threshold and the handler returned false | +| `POLICY_UNBOUNDED_ACTION` | The action moves value that the configured amount cap cannot measure from the action alone | +| `POLICY_UNRESOLVED_TICK` | A token uses its `^` wire form and the policy cannot bind its tick-scoped limits to a name | +| `POLICY_TICK_CASE_COLLISION` | A cap table names the same case-insensitive token more than once | +| `POLICY_IDEMPOTENCY_REQUIRED` | A stable `submitOpts.idempotencyKey` was not supplied | +| `POLICY_DUPLICATE_SUBMIT` | That idempotency key was already recorded; the error includes the earlier transaction id when known | +| `POLICY_STATE_CORRUPT` | The usage-state file is unreadable or structurally invalid | ```mermaid flowchart TD - A["Action proposed"] --> B{"Action type in allowedActions?"} + A["Action proposed"] --> Z{"Session halted?"} + Z -- "Yes" --> D0["POLICY_HALTED"] + Z -- "No" --> B{"Action in allowedActions?"} B -- "No" --> D1["POLICY_ACTION_DENIED"] - B -- "Yes" --> C{"Destination in allowedDestinations?"} - C -- "No" --> D2["POLICY_DESTINATION_DENIED"] - C -- "Yes" --> E{"Within per-action limit
(maxPerAction)?"} - E -- "No" --> D3["POLICY_AMOUNT_EXCEEDED"] - E -- "Yes" --> F{"Within rolling window limits
(maxPerWindow)?"} - F -- "No" --> D4["POLICY_WINDOW_AMOUNT_EXCEEDED,
POLICY_WINDOW_COUNT_EXCEEDED"] - F -- "Yes" --> G{"Amount above confirmAbove threshold?"} - G -- "Yes, handler declines" --> D5["POLICY_CONFIRMATION_DENIED"] - G -- "No, or handler approves" --> H["Allowed: signed and broadcast"] + B -- "Yes" --> C{"Destination allowed and enforceable?"} + C -- "No" --> D2["Destination policy refusal"] + C -- "Yes" --> E{"Within amount and window limits?"} + E -- "No" --> D3["Amount or window refusal"] + E -- "Yes" --> F{"Confirmation required?"} + F -- "Yes, handler declines" --> D4["POLICY_CONFIRMATION_DENIED"] + F -- "No, or handler approves" --> G{"New idempotency key?"} + G -- "No" --> D5["POLICY_DUPLICATE_SUBMIT"] + G -- "Yes" --> H["Record authorization, then sign and broadcast"] ``` -Agents should treat all `SDKPolicyError` codes as final answers, not errors to retry. +Do not blindly retry a policy error. Change a denied request only when the operator's policy permits it. For `POLICY_DUPLICATE_SUBMIT`, resume tracking the original transaction instead of submitting again. Configuration, halt, and state-file errors require deliberate operator action. ## Designed to fail closed -- No `allowedActions` list → the session refuses to construct. Nothing is allowed by default. -- Window usage is persisted to disk (default `~/.xchain/agent-usage-
.json`), so restarting the agent (or crash-looping it) does not reset the spending window. -- If that usage file is corrupted, the session **blocks** rather than silently starting a fresh window. Delete it deliberately to reset. +- A missing or empty `allowedActions` list makes the session refuse to construct. +- A policy also needs at least one enforceable `maxPerAction` or `maxPerWindow.perTick` entry, or a `confirmAbove` object. Running with only an action allowlist requires the explicit `allowUnbounded: true` opt-in. +- Window usage and idempotency records persist to disk. Restarting or crash-looping the agent does not reset a spending window or immediately forget a submitted payment. +- The session records budget when it authorizes a submission, before the irreversible broadcast. A later failure can therefore consume budget even when no transaction id is returned. This conservative behavior prevents a lost acknowledgement from reopening the cap. +- If the usage file is corrupted, the session blocks instead of silently starting a fresh window. Inspect, repair, or remove it deliberately to recover. +- `pause()` stops the current session object. The default kill-switch file, `~/.xchain/agent-halt-
`, can halt a running session that the operator cannot call directly. - Amount checks use exact decimal arithmetic. There is no floating-point edge to slip through. ## What this does: and does not: protect against @@ -96,49 +115,76 @@ This is a guardrail around the **agent**, not around the **key**. It protects yo Two practices close most of that gap: 1. **Fund the agent's address like a spending account, not a vault.** The window cap only has to be wrong by one top-up. -2. For hard enforcement, use the **MuSig2 co-signer** (next section): the agent holds one key, a policy daemon holds another, and the network sees a single aggregate signature that simply cannot be produced outside policy. +2. For hard enforcement on a SegWit network, use the **MuSig2 co-signer** described next. The agent holds one key, a policy service holds another, and neither can complete the ordinary spend path alone. ## Hard enforcement: the MuSig2 co-signer -The agent session is a guardrail inside the SDK; whoever holds the raw key can go around it. The co-signer removes that path. The agent's address becomes a MuSig2 (BIP-327) aggregate of two keys: the agent holds one, a policy daemon you run holds the other. Neither key can spend alone. Before adding its half of any signature, the daemon decodes the action **from the PSBT itself** (not from anything the agent claims), evaluates the same policy rules described above against its own persisted spending window, and checks the transaction outputs against value drains. The chain sees one ordinary Taproot signature, so there is no script an attacker can satisfy some other way. +The agent session is a guardrail inside the SDK; whoever holds the raw key can go around it. A MuSig2 account removes that path. In the plain 2-of-2 form, the spending address aggregates the agent key and a co-signer key. Before adding its signature share, the co-signer decodes the action from the PSBT, evaluates its own policy and persisted window, checks that the inputs belong to the configured account, restricts transaction outputs, and reconciles the miner fee. The agent combines both shares into one Taproot signature. + +This path requires a network that supports SegWit and P2TR. The SDK refuses to derive it on networks, including Dogecoin networks, where a witness-v1 output would not be safe. The co-signer also fails closed on action formats it cannot decode. In particular, an action being present in `allowedActions` does not guarantee that every protocol version of that action is decodable or that an amount or destination cap can constrain it. ```mermaid sequenceDiagram participant Agent - participant Daemon as Co-signer Daemon - - Agent->>Daemon: Propose transaction (PSBT) - Daemon->>Daemon: Decode action from the PSBT itself - Daemon->>Daemon: Evaluate policy against its own persisted window, check outputs for value drains - alt Within policy - Daemon-->>Agent: Add daemon signature share (co-sign) - else Policy violation - Daemon-->>Agent: Refuse (no signature) - Note over Agent: Surfaces as SDKPolicyError + participant Daemon as Co-signer Service + + Agent->>Daemon: Propose transaction PSBT + Daemon->>Daemon: Decode action from the PSBT + Daemon->>Daemon: Evaluate policy and persisted window + Daemon->>Daemon: Check inputs, outputs, and fee + alt Within every gate + Daemon-->>Agent: Return daemon signature share + else Refused + Daemon-->>Agent: Return approved false and a reason + Note over Agent: CoSignerClient throws SDKPolicyError end ``` ```js -// Daemon side, in its own process (ideally its own host): -const { CoSigner } = require('@dankest-llc/xchain-sdk').coSigner; -const { createCoSignerApp } = require('@dankest-llc/xchain-sdk/src/cosigner/server.js'); +const { coSigner } = require('@dankest-llc/xchain-sdk'); + +const transport = coSigner.httpTransport('http://127.0.0.1:8787/cosign', { + token: coSignerToken, +}); -// Agent side: same session API, hard-enforced. -const session = sdk.musig2AgentSession(agentWif, policy, { transport }); +const session = sdk.musig2AgentSession(agentWif, policy, { + coSigner: { + transport, + publicKeys: [agentPublicKey, daemonPublicKey], + // Optional: add a separately stored recovery public key. + recovery: recoveryPublicKey, + }, +}); ``` -Policy denials surface to the agent as the same `SDKPolicyError` codes listed above; the difference is that the daemon's refusal is a missing signature, not a raised error the agent could patch out. +The co-signer service is built with `coSigner.CoSigner`. A Node daemon can expose it with `createCoSignerApp` from `@dankest-llc/xchain-sdk/src/cosigner/server.js`; that HTTP sidecar requires a bearer token unless the operator deliberately enables its development-only unauthenticated mode. + +Co-signer refusals reach the agent as `SDKPolicyError`, but their codes are not limited to the `POLICY_*` values in the agent-session table. Decode failures, unauthorized outputs, fee checks, malformed requests, and a confirmation-required action at a headless daemon all withhold the signature and return their own refusal reasons. A standalone headless co-signer automatically signs requests that pass every configured gate. It denies `confirmAbove` requests by default because it cannot prompt a human. -One wrinkle to know about: the SDK normally shrinks on-chain payloads by writing an indexed token's tick as its numeric id (`^123` instead of `MYTOKEN`), and the daemon judges the transaction exactly as it will appear on chain. If your policy has tick-scoped limits, tell the daemon which ids those tokens have (`tickIds: { MYTOKEN: 123 }` in the policy); an id the policy cannot resolve is refused (`POLICY_UNRESOLVED_TICK`) rather than allowed to slip past a named cap. +One wrinkle to know about: the SDK can shrink on-chain payloads by writing an indexed token's tick as its numeric id (`^123` instead of `MYTOKEN`), and the co-signer judges the transaction exactly as it appears on chain. If your policy has tick-scoped limits, tell the co-signer which ids those tokens have with `tickIds: { MYTOKEN: 123 }`. An id the policy cannot resolve is refused rather than allowed to slip past a named cap. + +`allowedDestinations` constrains destination fields in supported action formats. It is not an allowlist for arbitrary native-coin transaction outputs. The co-signer separately permits its OP_RETURN carrier, change back to the account, and outputs explicitly configured in its `allowedOutputs`; `maxFeeSats` can add an absolute miner-fee ceiling. + +### Signing through XChain Wallet + +XChain Wallet's dApp bridge is an interactive co-signer, not the unattended policy daemon described above. Every `coSign` request opens an approval prompt and requires the wallet password. The wallet decodes the PSBT and evaluates account policy as a safety check, but the human remains the final gate. An automated agent cannot use the wallet's `coSign` bridge method unattended. + +The wallet's saved site permissions do not change that rule. For ordinary `signAction` requests, `never` is an immediate refusal, while both the default `ask` state and a saved `always` state still reach the signing approval flow. `canSignMessage: true` does not avoid the password prompt by itself. `signPsbt`, `signIn`, and `coSign` remain interactive. + +The only shipped bridge prompt bypass is a separate developer feature for localhost. When Developer Mode and a positive localhost auto-sign timeout are both enabled, a successful real approval can cache the wallet password in the extension service worker's memory. Later `signMessage` and single `signAction` requests from localhost can reuse it until expiry without another prompt. The feature is off by default, applies only to loopback origins, does not survive a service-worker or browser restart, and still enforces site, chain, account, action-denial, and throttle checks. It does not apply to `coSign`. ### The 2-of-3 recovery account, and where that key must live -A 2-of-2 address has an obvious failure mode: lose either key (or the daemon's host) and the funds are stuck forever. `deriveMuSig2P2TR2of3` adds a third **recovery key** so any two of the three can move funds. Normal operation still runs agent + daemon through policy; `buildRecoverySpend` is the operator escape hatch for when one party is lost. +A 2-of-2 address has an obvious failure mode: lose either key or the co-signer host and the funds are stuck forever. `deriveMuSig2P2TR2of3` adds a third **recovery key** so any two of the three can move funds. Normal operation still runs agent plus co-signer through the policy-controlled key path. `buildRecoverySpend` is the operator escape hatch for when one party is lost. That escape hatch is deliberate, and it comes with one hard operational rule: -**Never store the recovery key where the agent or the daemon runs.** A recovery-path spend involves no policy check at all; that is its purpose. If the recovery key sits on the same machine as the agent key, a compromised agent holds two of the three keys and can drain the account while the policy daemon watches. Keep the recovery key cold (hardware wallet, paper, or an offline machine), and treat any use of it as an incident to investigate, not a routine workflow. +**Never store the recovery key where the agent or the co-signer runs.** A recovery-path spend involves no policy check; that is its purpose. If the recovery key sits on the same machine as the agent key, a compromised agent holds two of the three keys and can drain the account without the policy service. Keep the recovery key cold, such as in a hardware wallet or on an offline machine, and treat any use of it as an incident to investigate, not a routine workflow. ## For MCP users -The `xchain-mcp` server's read tools never touch keys. Its write tool (`submit_action`) stays off until the operator configures a key and policy, and when on it routes every submission through an agent session, there is deliberately no unpoliced write path for agents. See the [MCP Quickstart](mcp-quickstart.md). +Without wallet configuration, `xchain-mcp` has no key material. Its query tools are read-only, and `compose_action` creates an unsigned PSBT for external signing. + +The operator can add a WIF and policy outside the model conversation. That configuration adds `get_agent_wallet` and `submit_action`; every submission routes through an agent session. The MCP server rejects `confirmAbove` because there is no human in its loop, and it requires a binding `maxPerAction` or `maxPerWindow.perTick` amount cap. A request that passes the policy is signed and broadcast **without per-action user approval**. This is why the configured key should control only a deliberately funded spending account. + +`submit_action` derives a stable idempotency key from the coin, action, and parameters unless the caller supplies one. An identical retry is refused as a duplicate and returns the earlier transaction id when available. There is no unpoliced MCP write path. See the [MCP Quickstart](mcp-quickstart.md). diff --git a/ai-agents/charging-agents.md b/ai-agents/charging-agents.md index 7c192a62..753e9090 100644 --- a/ai-agents/charging-agents.md +++ b/ai-agents/charging-agents.md @@ -7,7 +7,7 @@ A commercial license is available, contact legal@dankest.llc. # Charging Agents for Data and APIs -You have something an AI agent wants; a data feed, an API, a file. XChain lets the agent pay for it in your token, inside the request itself: the agent asks, your server answers "402, here's the price," the agent pays on-chain, retries, and gets the goods. No accounts, no API keys, no card forms. This guide is for the **seller** side; agents paying are covered by the SDK's `X402Client` (one call: `client.fetchUrl(url)`). +You have something an AI agent wants; a data feed, an API, a file. XChain lets the agent pay for it in your token, inside the request itself: the agent asks, your server answers "402, here's the price," the agent pays on-chain, retries, and gets the goods. No accounts, no API keys, no card forms. This guide is for the **seller** side; agents paying are covered by the SDK's `X402Client` (one call: `client.fetchUrl(url)`, on a client built with a per-tick ceiling such as `maxAmount: { YOURTICK: '5' }`; without one it refuses to pay). ```mermaid sequenceDiagram diff --git a/architecture/component-map.md b/architecture/component-map.md index b2701c17..bb950ef6 100644 --- a/architecture/component-map.md +++ b/architecture/component-map.md @@ -85,8 +85,8 @@ See [`../components/indexer/`](../components/indexer/) for full documentation. Key technical details: -- 305 REST endpoint patterns across the `/api` and `/explorer` namespaces, covering tokens, balances, holders, orders, dispensers, transactions, events, market data, contracts, staking, attestations, cross-chain calls, betting feeds and bets, governance polls and ballots, contract emissions, vote delegations, validator capabilities and slashing, chain reorgs, anchor reward attestations, per-block commitments, and more. The breakdown, re-derived from `xchain-explorer/src/XChainExplorer.js` on 2026-08-28: - - 175 `/{COIN}/api/...` and 112 `/{COIN}/explorer/...` patterns in the dispatch table built by `setupUrls()`, matched by the catch-all handler rather than registered with Express individually. +- 320 REST endpoint patterns across the `/api` and `/explorer` namespaces, covering tokens, balances, holders, orders, dispensers, transactions, events, market data, contracts, staking, attestations, cross-chain calls, betting feeds and bets, governance polls and ballots, contract emissions, vote delegations, validator capabilities and slashing, chain reorgs, anchor reward attestations, per-block commitments, and more. The breakdown, re-derived from `xchain-explorer/src/XChainExplorer.js` on 2026-08-28: + - 190 `/{COIN}/api/...` and 112 `/{COIN}/explorer/...` patterns in the dispatch table built by `setupUrls()`, matched by the catch-all handler rather than registered with Express individually. - 18 hand-registered `/{COIN}/api/...` routes that bypass the dispatch table: raw file download, fee quote, oracle fee quote, preflight (registered twice, GET and POST, because the largest legal action does not fit a query string), the two POST batch reads (balances, COINPay obligations), fee schedule, checkpoint list, checkpoint range, checkpoint verify, hub-mirror status, the five Merkle proof endpoints (balance, locked balance, action, validator set, contract state), and the POST contract-call query endpoint. - Outside those two namespaces the same server also registers 117 HTML page routes plus `/openapi.json`, `/icon`, `/relay`, and the static asset mounts. - JSON-RPC 2.0 interface compatible with Counterparty-style tooling. @@ -166,7 +166,7 @@ See [`../components/encoder/`](../components/encoder/) for full documentation. Key technical details: -- LevelDB key schema uses single-character prefixes: `B`=block, `T`=transaction, `I`=input, `O`=output, `H`/`J`=address hints. +- LevelDB key schema uses single-character prefixes: `B`=block, `T`=transaction, `I`=spent input, `O`=unspent output keyed by scriptHash, `H`=outpoint-to-scriptHash hint, `J`=input hint, and `S`/`Z`/`K`/`M`/`N`/`W` for script tracking and reorg undo (full layouts in the [utxo-tracker key schema](../components/utxo-tracker/architecture.md#leveldb-key-schema)). - Processes blocks in batches of up to 200 (flush may trigger earlier under heap pressure), writing each batch atomically to LevelDB. - Maintains a per-chain, per-network undo window (mainnet and regtest: BTC 12 / LTC 120 / DOGE 120 blocks; testnet: BTC 120 / LTC 5000 / DOGE 120 blocks; overridable via XCHAIN_UNDO_BLOCKS_) to support chain reorganization rollback. - Tracks the mempool for real-time unconfirmed UTXO state. @@ -327,17 +327,17 @@ See [`../components/vm/`](../components/vm/) for full documentation. | | | |---|---| -| **Purpose** | Self-custodial multi-chain reference wallet; browser SPA, Chrome MV3 extension, Electron desktop app, and Capacitor mobile app (Android shipped, iOS later) | +| **Purpose** | Self-custodial multi-chain reference wallet; browser SPA, Chrome MV3 extension, Electron desktop app, and Capacitor mobile app (Android shipped, iOS pending App Store release) | | **Inputs** | User interaction; xchain-sdk for action construction; xchain-explorer for balance and history queries; xchain-hub for config and fee data | | **Outputs** | Signed transactions broadcast to coin nodes via the encoder; read-only views of balances, tokens, actions, and markets | -| **Storage** | Client-side only (browser localStorage / extension storage / Electron local store); no server-side state | +| **Storage** | Client-side only; no server-side state. The AES-256-GCM-encrypted vault lives in IndexedDB (web), `chrome.storage.local` (extension), an encrypted file under Electron `userData` (desktop), or an app-private file via the native `XChainVault` plugin under an OS-keystore key (mobile). The vault never sits in `localStorage`. Browser-based shells do keep small side records there beside UI state, such as the web shell's KDF parameters and, with biometric unlock on, the authenticator-wrapped copy of the password. See [wallet architecture](../components/wallet/architecture.md) and [wallet security](../components/wallet/security.md) | | **Communication** | Outbound JSON-RPC and REST to xchain-encoder, xchain-explorer, and xchain-hub; no inbound API | Key technical details: - Built on xchain-sdk; all action construction goes through the SDK's 32 developer-invocable ACTION methods. - Supports every chain the platform runs on, today Bitcoin, Litecoin, and Dogecoin (mainnet, testnet, regtest), from the same codebase. -- Deployed as a web SPA (served from a static docroot), a Chrome MV3 extension (packaged from the same source), an Electron desktop application, and a Capacitor mobile app wrapping the same web build (Android shipped, iOS later). +- Deployed as a web SPA (served from a static docroot), a Chrome MV3 extension (packaged from the same source), an Electron desktop application, and a Capacitor mobile app wrapping the same web build (Android shipped, iOS pending App Store release). - Private keys never leave the client; signing happens locally before broadcast. - Targets non-technical end users; UI language is intentionally plain (e.g., "About" not "Token Spec"). diff --git a/architecture/data-pipeline.md b/architecture/data-pipeline.md index d19b7a5d..2c2d49ef 100644 --- a/architecture/data-pipeline.md +++ b/architecture/data-pipeline.md @@ -67,7 +67,7 @@ From this point, the transaction is a normal blockchain transaction. Miners pick The decoder polls the coin node every few seconds via JSON-RPC (`getblockcount`, `getblockhash`, `getblock`, `getrawtransaction`). When it finds a new block: 1. **Parses each transaction** using bitcoinjs-lib. Coin-specific pre-processing is applied first: - - Litecoin: strip the HogEx flag from the block header before parsing + - Litecoin: strip the MWEB/HogEx marker and flag from each raw transaction before parsing - Dogecoin: strip AuxPoW merge-mining headers before parsing 2. **Scans outputs** for the `XCHN` magic prefix after deobfuscation. Deobfuscation uses the first input's txid, so the decoder must retrieve the full transaction (including inputs) to reconstruct the key and IV. diff --git a/architecture/database-design.md b/architecture/database-design.md index 439ad4a4..8910126b 100644 --- a/architecture/database-design.md +++ b/architecture/database-design.md @@ -145,15 +145,7 @@ LevelDB is used by the UTXO tracker for fast key-value access without relational ### xchain-utxo-tracker -Stores the full UTXO set of the monitored coin node. Key schema uses single-character prefixes: - -| Prefix | Contents | -|---|---| -| `B` | Block records (height → hash) | -| `T` | Transaction records (txid → metadata) | -| `I` | Input records (txid:vout → spending txid) | -| `O` | Output records (txid:vout → value, scriptPubKey) | -| `H` / `J` | Address hints (scriptPubKey hash → txids) | +Stores the full UTXO set of the monitored coin node in a single LevelDB instance, using single-byte key prefixes. Unspent outputs (`O`) are keyed by scriptHash first, so one range scan answers "what UTXOs does this address have?". `B`, `T` and `I` hold block metadata keyed by block hash, the transaction-to-block mapping and spent inputs. The hint keys `H` (outpoint to scriptHash) and `J` (input hint) let a spend find and delete its output without a scan, `S` records each script's first appearance, and `Z`, `K`, `M`, `N` and `W` hold the indexes and deleted-record archives that reorg rollback uses. The [utxo-tracker key schema](../components/utxo-tracker/architecture.md#leveldb-key-schema) is the single reference for every key and value layout. Writes are batched in groups of up to 200 blocks (flush may trigger earlier under heap pressure). A per-chain, per-network undo window (mainnet and regtest: BTC 12 / LTC 120 / DOGE 120 blocks; testnet: BTC 120 / LTC 5000 / DOGE 120 blocks) is retained to support reorg rollback. diff --git a/architecture/platform-map-app.html b/architecture/platform-map-app.html index c1c17515..ad841052 100644 --- a/architecture/platform-map-app.html +++ b/architecture/platform-map-app.html @@ -245,14 +245,14 @@

Flows

{ "id": "indexer", "label": "xchain-indexer", "type": "service", "group": "index", "tech": "Node.js, MariaDB", "description": "Reads the decoder DB and applies ACTION semantics: double-entry ledger, XCHAIN gas fees, orders/swaps/dispensers, staking capabilities, attestation validation, system-injected expiries. Executes contracts through xchain-vm and follows decoder reorgs." }, { "id": "indexerdb", "label": "Indexer DB", "type": "database", "group": "index", "tech": "MariaDB", "description": "Canonical protocol state: balances, tokens, orders, contracts, contract_state (append-only), staking, attestation and SPV tables." }, { "id": "vm", "label": "xchain-vm", "type": "library", "group": "index", "tech": "Node.js, isolated-vm", "description": "Deterministic smart-contract engine: sandboxed V8 isolates, host-side gas metering, 30s CPU / 8MB memory / call depth 4 limits, hardened sandbox (constructor neutering, RegExp removal). Emits actions (XCALL, ATTEST) back to the indexer." }, - { "id": "hub", "label": "xchain-hub", "type": "service", "group": "hub", "tech": "Node.js, MariaDB", "description": "Config oracle and cross-chain coordinator. PBFT federation (stake-weighted source-deduped quorum at/above STAKE_WEIGHTED_QUORUM_ACTIVATION, majority-floored count max(2f+1, ceil((N+1)/2)) below it), five stake-qualified capabilities, price-oracle rounds, attestation engine (http_get / llm providers), reorg-retraction co-signing, governance/slash, and StateAnchorPublisher (ANCHOR v0 bundle on DOGE)." }, + { "id": "hub", "label": "xchain-hub", "type": "service", "group": "hub", "tech": "Node.js, MariaDB", "description": "Config oracle and cross-chain coordinator. PBFT federation (stake-weighted source-deduped quorum at/above STAKE_WEIGHTED_QUORUM_ACTIVATION, majority-floored count max(2f+1, ceil((N+1)/2)) below it), five stake-qualified capabilities, price-oracle rounds, attestation engine (http_get / llm providers), reorg-retraction co-signing, governance/slash, and StateAnchorPublisher (ANCHOR v0/v3 bundles on DOGE)." }, { "id": "hubdb", "label": "Hub DB", "type": "database", "group": "hub", "tech": "MariaDB", "description": "Hub state tables, one per file in src/sql/: configs, validators, consensus state, cross-chain calls, price_snapshots, oracle_prices, attestation stats." }, { "id": "explorer", "label": "xchain-explorer", "type": "service", "group": "serve", "tech": "Node.js, Express, WS", "description": "Read-only REST + JSON-RPC + WebSocket API and web UI over indexer state (60+ endpoints, /{COIN} prefixed). Runs a hub-mirror sync for consensus tables, ABI introspection, and an optional sandboxed contract-simulation endpoint." }, { "id": "sync", "label": "xchain-sync", "type": "service", "group": "serve", "tech": "Node.js, WS", "description": "Replicates indexer + decoder DBs to validators: REST snapshots plus a WebSocket block feed, transactional apply with rollback, merkle transparency log (sync_meta, merkle_epochs) and pinned-validator checkpoint quorum verification." }, { "id": "replicas", "label": "Validator / Replica Nodes", "type": "external", "group": "serve", "description": "Downstream validator stacks fed by xchain-sync. Run their own indexer/explorer against replicated state and re-verify state commitments." }, { "id": "miner", "label": "xchain-regtest-miner", "type": "service", "group": "access", "tech": "Node.js", "description": "Regtest-only auto-miner: dual-timer mempool mining, generate_blocks, send_funds, fill_mempool stress tool, set_mock_time for deterministic expiries. Refused on mainnet." }, { "id": "sdk", "label": "xchain-sdk", "type": "library", "group": "clients", "tech": "Node.js (npm: library + JSON-RPC server)", "description": "Developer SDK generating all 31 user ACTION types with ticker compaction and a MuSig2 co-signer toolkit. Discovers endpoints via hub getallconfigs with an https downgrade guard; defaults to public .xchain.io hosts off-regtest." }, - { "id": "wallet", "label": "xchain-wallet", "type": "service", "group": "clients", "tech": "JS monorepo: web SPA, MV3 extension, Electron, Capacitor mobile", "description": "Self-custodial multi-chain wallet built on the SDK. Packages: core (shell-free, enforced by CI), web, extension, desktop, mobile (Capacitor; Android shipped, iOS later), plus Trezor/Ledger signer packages. Signs PSBTs locally; keys never leave the device." }, + { "id": "wallet", "label": "xchain-wallet", "type": "service", "group": "clients", "tech": "JS monorepo: web SPA, MV3 extension, Electron, Capacitor mobile", "description": "Self-custodial multi-chain wallet built on the SDK. Packages: core (shell-free, enforced by CI), web, extension, desktop, mobile (Capacitor; Android shipped, iOS pending App Store release), plus Trezor/Ledger signer packages. Signs PSBTs locally; keys never leave the device." }, { "id": "node", "label": "xchain-node", "type": "service", "group": "clients", "tech": "Node.js, Docker, Commander CLI", "description": "Installer/operator CLI: provisions coin nodes (SHA-256 pinned binaries) and every platform service as Docker containers. Ed25519-signed bootstraps, go-live and skew guards, telemetry (opt-out), interactive TUI." }, { "id": "e2e", "label": "xchain-e2e-test", "type": "service", "group": "clients", "tech": "Mocha", "description": "Full-stack end-to-end suite: broadcasts real transactions on regtest through service connectors (Blockchain, Encoder, Indexer, Hub, UtxoTracker, RegtestMiner), 30+ waitFor* DB pollers, BTC/LTC/DOGE parity corpus, perf gates." }, { "id": "contracts", "label": "xchain-contracts", "type": "library", "group": "clients", "tech": "Node.js, mocha (tooling only)", "description": "Smart-contract template library (amm, escrow, vesting, crowdsale, treasury, priceBet, stableVault, urlOracle, cardDispenser) plus a policy-driven controller-guard generator. Deployed as base64 DEPLOY actions; tested against xchain-vm." }, @@ -275,7 +275,7 @@

Flows

{ "id": "r13", "source": "hub", "target": "hubdb", "label": "config / consensus / oracle state", "kind": "write" }, { "id": "r14", "source": "hub", "target": "encoder", "label": "publishes ANCHOR / ATTEST / PRICE actions", "kind": "rpc" }, { "id": "r15", "source": "sync", "target": "indexerdb", "label": "snapshots + follows state", "kind": "read" }, - { "id": "r16", "source": "sync", "target": "decoderdb", "label": "snapshots decoded txs (8 of 9 tables)", "kind": "read" }, + { "id": "r16", "source": "sync", "target": "decoderdb", "label": "snapshots decoded txs (8 of 10 tables)", "kind": "read" }, { "id": "r17", "source": "sync", "target": "replicas", "label": "REST snapshots + WebSocket block feed", "kind": "async" }, { "id": "r18", "source": "miner", "target": "coinnode", "label": "auto-mines mempool (regtest)", "kind": "rpc" }, { "id": "r19", "source": "sdk", "target": "encoder", "label": "create_tx / broadcast", "kind": "rpc" }, @@ -391,7 +391,7 @@

Flows

"description": "How a validator or replica node acquires and verifies platform state without trusting the source.", "steps": [ { "n": 1, "node": "sync", "edge": "r15", "action": "Sync server snapshots the indexer DB and tails new blocks." }, - { "n": 2, "node": "sync", "edge": "r16", "action": "Decoder DB is snapshotted too (8 of 9 tables; transparency/mempool excluded by design)." }, + { "n": 2, "node": "sync", "edge": "r16", "action": "Decoder DB is snapshotted too (8 of 10 tables; mempool_transactions and dispenser_extension_undo excluded by design)." }, { "n": 3, "node": "sync", "edge": "r17", "action": "Replicas bootstrap over REST snapshots then follow the WebSocket block feed." }, { "n": 4, "node": "replicas", "action": "ClientApplier applies transactionally (batches of 100, rollback support); merkle epochs and pinned-validator checkpoint quorum verify the stream." } ] @@ -399,10 +399,10 @@

Flows

{ "id": "f8", "name": "State anchoring (ANCHOR on DOGE)", - "description": "Quorum-signed per-chain state checkpoints are anchored on Dogecoin as one ANCHOR v0 bundle per network, for chain-parse recoverability.", + "description": "Quorum-signed per-chain state checkpoints are anchored on Dogecoin in ANCHOR v0 checkpoint bundles or activation-gated v3 folded bundles, for chain-parse recoverability.", "steps": [ { "n": 1, "node": "hub", "edge": "r13", "action": "StateAnchorPublisher runs one publisher election per bundle by hash ordering and assembles every checkpointed chain's quorum-signed checkpoint as its own bundle section, plus the compressed cross-chain match archive." }, - { "n": 2, "node": "hub", "edge": "r14", "action": "ONE ANCHOR v0 checkpoint bundle per network per publishing cycle is built and submitted through the DOGE encoder from the elected publisher's own DOGE wallet." }, + { "n": 2, "node": "hub", "edge": "r14", "action": "The elected publisher builds and submits ANCHOR v0 checkpoint bundles, or v3 folded bundles at or above ANCHOR_FOLD_ACTIVATION, through the DOGE encoder from its own DOGE wallet." }, { "n": 3, "node": "encoder", "edge": "r4", "action": "Anchor transaction is broadcast on Dogecoin." }, { "n": 4, "node": "decoder", "edge": "r6", "action": "DOGE decoder picks the ANCHOR out of the block." }, { "n": 5, "node": "indexer", "edge": "r8", "action": "Indexers verify and record the checkpoint; RewardTracker records publisher rewards." } diff --git a/architecture/platform-map.json b/architecture/platform-map.json index 177767cb..346aaf6b 100644 --- a/architecture/platform-map.json +++ b/architecture/platform-map.json @@ -98,7 +98,7 @@ "type": "service", "group": "hub", "tech": "Node.js, MariaDB", - "description": "Config oracle and cross-chain coordinator. PBFT federation (stake-weighted source-deduped quorum at/above STAKE_WEIGHTED_QUORUM_ACTIVATION, majority-floored count max(2f+1, ceil((N+1)/2)) below it), five stake-qualified capabilities, price-oracle rounds, attestation engine (http_get / llm providers), reorg-retraction co-signing, governance/slash, and StateAnchorPublisher (ANCHOR v0 bundle on DOGE)." + "description": "Config oracle and cross-chain coordinator. PBFT federation (stake-weighted source-deduped quorum at/above STAKE_WEIGHTED_QUORUM_ACTIVATION, majority-floored count max(2f+1, ceil((N+1)/2)) below it), five stake-qualified capabilities, price-oracle rounds, attestation engine (http_get / llm providers), reorg-retraction co-signing, governance/slash, and StateAnchorPublisher (ANCHOR v0/v3 bundles on DOGE)." }, { "id": "hubdb", @@ -153,7 +153,7 @@ "type": "service", "group": "clients", "tech": "JS monorepo: web SPA, MV3 extension, Electron, Capacitor mobile", - "description": "Self-custodial multi-chain wallet built on the SDK. Packages: core (shell-free, enforced by CI), web, extension, desktop, mobile (Capacitor; Android shipped, iOS later), plus Trezor/Ledger signer packages. Signs PSBTs locally; keys never leave the device." + "description": "Self-custodial multi-chain wallet built on the SDK. Packages: core (shell-free, enforced by CI), web, extension, desktop, mobile (Capacitor; Android shipped, iOS pending App Store release), plus Trezor/Ledger signer packages. Signs PSBTs locally; keys never leave the device." }, { "id": "node", @@ -306,7 +306,7 @@ "id": "r16", "source": "sync", "target": "decoderdb", - "label": "snapshots decoded txs (8 of 9 tables)", + "label": "snapshots decoded txs (8 of 10 tables)", "kind": "read" }, { @@ -776,7 +776,7 @@ "n": 2, "node": "sync", "edge": "r16", - "action": "Decoder DB is snapshotted too (8 of 9 tables; transparency/mempool excluded by design)." + "action": "Decoder DB is snapshotted too (8 of 10 tables; mempool_transactions and dispenser_extension_undo excluded by design)." }, { "n": 3, @@ -794,7 +794,7 @@ { "id": "f8", "name": "State anchoring (ANCHOR on DOGE)", - "description": "Quorum-signed per-chain state checkpoints are anchored on Dogecoin as one ANCHOR v0 bundle per network, for chain-parse recoverability.", + "description": "Quorum-signed per-chain state checkpoints are anchored on Dogecoin in ANCHOR v0 checkpoint bundles or activation-gated v3 folded bundles, for chain-parse recoverability.", "steps": [ { "n": 1, @@ -806,7 +806,7 @@ "n": 2, "node": "hub", "edge": "r14", - "action": "ONE ANCHOR v0 checkpoint bundle per network per publishing cycle is built and submitted through the DOGE encoder from the elected publisher's own DOGE wallet." + "action": "The elected publisher builds and submits ANCHOR v0 checkpoint bundles, or v3 folded bundles at or above ANCHOR_FOLD_ACTIVATION, through the DOGE encoder from its own DOGE wallet." }, { "n": 3, diff --git a/bin/generate-flag-days.js b/bin/generate-flag-days.js index 4dd8a6ce..6d3b5740 100644 --- a/bin/generate-flag-days.js +++ b/bin/generate-flag-days.js @@ -274,6 +274,95 @@ function collectMapSlots(body, gate, source, label, add, unreadable) { /** An `addGate('.', '', {` row with an object-literal table. */ const GATE_ROW = /addGate\(\s*'([A-Za-z0-9_/]+)\.([A-Z][A-Z0-9_]*)'\s*,\s*'([a-z]+)'\s*,\s*\{/g; +/** A whole string literal in any quoting, with no escapes or nested quotes. */ +const STRING_LITERAL = /^(['"`])([^'"`\\]*)\1$/; + +/** A bare `mainnet:` slot inside a table body. */ +const BARE_MAINNET_SLOT = /(?:^|[{,\s])mainnet\s*:\s*([^,}\n]*)/; + +/** + * The first `count` top-level arguments of the call whose `(` sits at `open`, as + * `{ text, index }` with `index` the offset of the argument's first character. + * String-aware, so a comma or bracket inside a quoted key or regex body ends + * nothing. + */ +function callArgs(text, open, count) { + const args = []; + let depth = 0; + let start = open + 1; + let quote = null; + const push = (end) => { + const raw = text.slice(start, end); + const lead = raw.length - raw.trimStart().length; + args.push({ text: raw.trim(), index: start + lead }); + }; + for (let i = start; i < text.length && args.length < count; i++) { + const ch = text[i]; + if (quote !== null) { + if (ch === '\\') i++; + else if (ch === quote) quote = null; + continue; + } + if (ch === "'" || ch === '"' || ch === '`') quote = ch; + else if (ch === '(' || ch === '[' || ch === '{') depth++; + else if (ch === ')' || ch === ']' || ch === '}') { + if (depth === 0) { push(i); break; } + depth--; + } else if (ch === ',' && depth === 0) { push(i); start = i + 1; } + } + return args; +} + +/** + * Refuses a registry `addGate` call that could carry a time-keyed gate but that + * GATE_ROW cannot read. + * + * WHY THIS IS LOUD. GATE_ROW is the only way the mainnet table, both UNARMED lists + * and the testnet arms find an addGate row, and it demands a single-quoted key, a + * single-quoted unit and an inline table. A row written any other way (a + * double-quoted key, a key or table bound to a const first) left every list in + * silence, the anchor count fell by one, and the currency test stayed green + * because it compares the page with the same parse. + * + * WHY THE UNIT LITERAL FIRES ON SYNTAX while the addChange arm fires on value. + * An addGate call states its unit, so `'time'` in any quoting already says the + * row is this page's subject, and a mainnet-only value test would miss the rows + * that park mainnet on UNARMED, which the UNARMED lists publish. Only a call whose + * unit is not a literal falls back to the value: loud when its inline table holds + * a mainnet time literal or UNARMED. A literal non-time unit and the registry's + * own plumbing (`addGate(key, unit, table)`) stay quiet. + */ +function assertEveryGateRowParsed(sources) { + const text = sources.scannable; + const matched = new Set([...text.matchAll(GATE_ROW)].map((m) => m.index)); + const unread = []; + for (const call of text.matchAll(/\baddGate\s*\(/g)) { + if (matched.has(call.index)) continue; + const args = callArgs(text, call.index + call[0].length - 1, 3); + if (args.length < 2) continue; + const unit = STRING_LITERAL.exec(args[1].text); + let loud = unit !== null && unit[2] === 'time'; + if (unit === null && args[2] && args[2].text.startsWith('{')) { + const body = objectBody(text, args[2].index); + const slot = body === null ? null : BARE_MAINNET_SLOT.exec(body); + const value = slot === null ? '' : slot[1].trim(); + loud = value === 'UNARMED' + || (TIME_LITERAL.test(value) && Number(value.replace(/_/g, '')) >= TIMESTAMP_FLOOR); + } + if (loud) unread.push(`${sources.where(call.index)}: ${text.slice(call.index, call.index + 80).replace(/\s+/g, ' ')}`); + } + if (unread.length > 0) { + throw new Error( + 'the protocol_changes registry declares a time-keyed addGate row this generator cannot read, so ' + + 'protocol/flag-days.md would publish an inventory that calls itself complete and is not:\n ' + + unread.join('\n ') + + "\n\nThe gate-row pass reads addGate('.', 'time', { ... }) with single quotes and an " + + 'inline table. Either write the row in that shape or widen GATE_ROW in ' + + 'bin/generate-flag-days.js deliberately.', + ); + } +} + /** * Every time-keyed gate the registry's `addGate(key, unit, table)` rows declare. * @@ -288,6 +377,7 @@ const GATE_ROW = /addGate\(\s*'([A-Za-z0-9_/]+)\.([A-Z][A-Z0-9_]*)'\s*,\s*'([a-z * sibling scan's reader, so the same shapes are quiet and the same are loud. */ function collectGateRows(sources, add) { + assertEveryGateRowParsed(sources); const unreadable = []; for (const row of sources.scannable.matchAll(GATE_ROW)) { if (row[3] !== 'time') continue; @@ -307,7 +397,14 @@ function collectGateRows(sources, add) { } } -function collectUnarmedGateRows(sources, network, add) { +/** + * One network's slot of every registry `addGate(..., 'time', { ... })` row, into + * `add` as `(gate, value)`: a decimal literal, or `UNARMED` resolved through + * core.js. The caller's `add` decides the range, so the one reader serves the + * testnet arms (a real instant) and both UNARMED lists (the sentinel). + */ +function collectGateRowSlots(sources, network, add) { + assertEveryGateRowParsed(sources); const slot = new RegExp(`(?:^|[,\\s])${network}\\s*:\\s*([^,}\\n]*)`); for (const row of sources.scannable.matchAll(GATE_ROW)) { if (row[3] !== 'time') continue; @@ -646,9 +743,12 @@ function collectGates(indexerSrc = INDEXER_SRC) { * every time-keyed gate is genesis-active off mainnet, and the page states * that as an invariant, so a gate that arms testnet at an instant of its own * must be surfaced as the exception rather than left silently contradicting - * the prose. Read from the same two declaration shapes as the mainnet parse: - * a `const NAME_TESTNET_TIME = ;` line, or a literal nonzero - * testnet_time slot (the fourth argument) in an addChange call. + * the prose. Read from the same three declaration shapes as the mainnet parse: + * a `const NAME_TESTNET_TIME = ;` line, a literal nonzero testnet_time + * slot (the fourth argument) in an addChange call or part-file row, and the bare + * `testnet:` slot of an `addGate('.', 'time', { ... })` row. The + * gate-row pass was once missing, so three gates that arm testnet on an instant + * of their own were published as genesis-active there. */ function collectTestnetArms(indexerSrc = INDEXER_SRC) { const sources = registrySources(indexerSrc); @@ -662,6 +762,7 @@ function collectTestnetArms(indexerSrc = INDEXER_SRC) { for (const d of declaredTimeConstants(sources)) { if (d.network === 'TESTNET' && !consumed.has(d.name)) add(d.prefix, d.value); } + collectGateRowSlots(sources, 'testnet', add); return [...found.values()].sort((a, b) => (a.time - b.time) || a.gate.localeCompare(b.gate)); } @@ -690,7 +791,7 @@ function collectTestnetUnarmed(indexerSrc = INDEXER_SRC) { for (const d of declaredTimeConstants(sources)) { if (d.network === 'TESTNET' && !consumed.has(d.name)) add(d.prefix, d.value); } - collectUnarmedGateRows(sources, 'testnet', add); + collectGateRowSlots(sources, 'testnet', add); return [...found.values()].sort((a, b) => a.gate.localeCompare(b.gate)); } @@ -721,7 +822,7 @@ function collectMainnetUnarmed(indexerSrc = INDEXER_SRC) { for (const d of declaredTimeConstants(sources)) { if (d.network === 'MAINNET' && !consumed.has(d.name)) add(d.prefix, d.value); } - collectUnarmedGateRows(sources, 'mainnet', add); + collectGateRowSlots(sources, 'mainnet', add); return [...found.values()].sort((a, b) => a.gate.localeCompare(b.gate)); } diff --git a/blockchains.md b/blockchains.md index 565517ba..202d9f22 100644 --- a/blockchains.md +++ b/blockchains.md @@ -21,8 +21,6 @@ Each chain supports three network types: | **Testnet** | Public test network: free test coins, mirrors mainnet behavior | | **Regtest** | Local regression testing network: instant block generation, fully controlled environment | -> **BTC testnet4 contract limitation:** Contract deploys on BTC testnet4 cannot be mined at current miner block sizes. Use LTC or DOGE testnet for contract deployment and testing. - ## Chain-Specific Differences The same protocol specification applies across all chains. Chain-specific differences are limited to: diff --git a/components/README.md b/components/README.md index 147dd9c5..d04bb981 100644 --- a/components/README.md +++ b/components/README.md @@ -17,7 +17,7 @@ This section contains documentation for each of the 14 XChain Platform component | [utxo-tracker](./utxo-tracker/) | Indexes UTXOs from coin nodes and serves address and balance queries | | [sdk](./sdk/) | Developer SDK for constructing and submitting XChain actions | | [contracts](./contracts/) | MIT-licensed smart contract template library, patterns, linter CLI, and no-code policy generator | -| [wallet](./wallet/) | Reference self-custodial multi-chain wallet: browser, Chrome extension, Electron desktop, and Capacitor mobile (Android shipped, iOS later) | +| [wallet](./wallet/) | Reference self-custodial multi-chain wallet: browser, Chrome extension, Electron desktop, and Capacitor mobile (Android shipped, iOS pending App Store release) | | [node](./node/) | CLI tool for installing and managing all platform services as Docker containers | | [e2e-test](./e2e-test/) | End-to-end Mocha test suite that exercises the full platform stack | | [regtest-miner](./regtest-miner/) | Auto-mines mempool transactions for regtest development environments | diff --git a/components/decoder/README.md b/components/decoder/README.md index 2d796116..af7bec0a 100644 --- a/components/decoder/README.md +++ b/components/decoder/README.md @@ -30,7 +30,7 @@ The decoder's job is extraction only; it does not interpret action semantics. It |---|---| | [Architecture](architecture.md) | Data pipeline, internal components, polling loop, transaction parsing, deobfuscation | | [Configuration](configuration.md) | Environment variables, internal constants, network-specific settings | -| [Database](database.md) | Full schema reference: 9 tables covering blocks, transactions, dispensers, pubkeys, indexes, and events | +| [Database](database.md) | Full schema reference: 10 tables covering blocks, transactions, mempool, dispensers, pubkeys, indexes, events, and reorg bookkeeping | | [Operations](operations.md) | Running, Docker, API endpoints, reorg handling, mempool, troubleshooting | ## Installation diff --git a/components/decoder/architecture.md b/components/decoder/architecture.md index c25e1ead..1a76cdf2 100644 --- a/components/decoder/architecture.md +++ b/components/decoder/architecture.md @@ -58,7 +58,7 @@ flowchart TD | `src/chain/XChainBlockDecoder.js` | `XChainBlockDecoder` | Block and transaction parsing via bitcoinjs-lib with coin-specific fixes (Litecoin MWEB, Dogecoin AuxPoW) | | `src/chain/crypto_networks.js` | `CryptoNetworks` | Network configuration: bitcoinjs-lib network objects and start block indexes for all 9 chain/network combinations | | `src/util.js` | None | Utility functions: sleep, SHA256, hex conversion, timer | -| `src/sql/*.sql` | None | Table creation SQL for all 9 database tables | +| `src/sql/*.sql` | None | Table creation SQL for all 10 database tables | ## Block Polling Loop @@ -71,7 +71,7 @@ flowchart TD S3["3. Fetch and parse next block
- getBlockHash → getBlock (hex)
- For Dogecoin: strip AuxPoW first
- Parse block with bitcoinjs-lib"] S4["4. Reorg detection
- Compare previous_block_hash
- If mismatch: delete bad block, log reorg event, retry"] S5["5. Process each transaction
- Parse outputs for XChain data
- Deobfuscate (AES-128-CTR)
- Validate XCHN prefix
- Resolve source address
- Check for dispenser payments"] - S6["6. Write to database
- Begin transaction
- Insert block, transactions, dispensers, outputs
- Delete expired dispensers
- Commit transaction"] + S6["6. Write to database
- Begin transaction
- Insert block, transactions, dispensers, outputs
- Soft-expire expired dispensers (stamp expired_block_index)
- Commit transaction
- After commit: hard-purge rows soft-expired at reorg-safe depth"] S7["7. Mempool updates (when synced)
- Every 60 seconds
- Fetch mempool txids
- Diff against DB, add new, remove stale
- Batch fetch in 1000-tx chunks"] S1 --> S2 --> S3 --> S4 --> S5 --> S6 --> S7 diff --git a/components/decoder/configuration.md b/components/decoder/configuration.md index 124c1f38..7a9e1bf3 100644 --- a/components/decoder/configuration.md +++ b/components/decoder/configuration.md @@ -27,7 +27,7 @@ Configuration is loaded from a `.env` file via `dotenv`. All variables are read | Variable | Description | Default | |---|---|---| -| `AUX_POW` | Enable AuxPoW header stripping (Dogecoin) | _(unset)_ | +| `AUX_POW` | Legacy; read but ignored. AuxPoW header stripping is chosen from `NETWORK` (Dogecoin only) | _(unset)_ | | `DECODER_RATE_LIMIT_RPM` | API requests per minute per IP | `100` | | `GETMEMPOOL_CACHE_MS` | How long the `getmempool` JSON-RPC response is cached, in milliseconds. That method reads the database, so without a cache a burst of unauthenticated requests contends with the block loop; the window is read once and sliced per request, and the underlying snapshot only changes every 60 seconds. | `5000` | | `FEE_DESTINATION` | Native-coin protocol fee destination override for this coin+network. The decoder persists outputs paying the resolved address to `transaction_outputs` so the indexer can validate native-coin fee payments. By default the address comes from the bundled coin registry (`src/coins`, pinned per coin/network), so capture is on for a stock install. This variable overrides the default on testnet/regtest only; on mainnet it is ignored with a warning, because fee acceptance is consensus and must not depend on operator environment. | _(coin registry)_ | @@ -45,7 +45,7 @@ Configuration is loaded from a `.env` file via `dotenv`. All variables are read | `DECODER_POLL_SILENT_MS` | How long the block loop may go without completing a single iteration before `/live` reports unhealthy (503) and the container restart policy recycles the process. Measures the loop, not the chain: the stall window `DECODER_STALL_ALERT_MS` (default `900000` ms, documented under Operations) asks whether the chain is advancing, and a caught-up decoder advances nothing for hours while being perfectly healthy, so only an iteration count separates "idle" from "the loop is gone". Defaults to twice the stall window, because every normal path through the loop, the node-outage retry included, returns to the loop top far inside it. | `1800000` (30 minutes) | | `XCHAIN_INDEXER_DIR` | Path override for the sibling `xchain-indexer` checkout that `bin/sync-batch-limits.js` reads to regenerate the vendored BATCH limit tables (`src/protocol/indexer_batch_limits.js`) from the indexer's own `src/actions/batch.js` and `src/protocol_changes.js`. A maintenance-tool setting only; the decoder service itself never reads it. | `../../xchain-indexer` relative to `bin/` (the sibling checkout layout) | -The `AUX_POW` variable should be set to any truthy value when running against Dogecoin nodes. It enables the `getBlockWithoutAuxPow()` code path that strips merge-mining headers before parsing. +The `AUX_POW` variable is legacy: the decoder still parses it but no longer consults it. The `getBlockWithoutAuxPow()` code path that strips merge-mining headers before parsing is selected from the coin's wire format, so it is always on for a `dogecoin-*` `NETWORK` and never on for Bitcoin or Litecoin, whatever `AUX_POW` says. ## Network Identifier Format diff --git a/components/decoder/database.md b/components/decoder/database.md index 00a5f1b7..f012b0e3 100644 --- a/components/decoder/database.md +++ b/components/decoder/database.md @@ -190,7 +190,7 @@ Notable applied migrations: | File | Mode | Effect | |---|---|---| | `2026-05-28-unique-index-tables.sql` | manual | De-duplicates `index_addresses`/`index_transactions` rows, then adds UNIQUE indexes | -| `2026-06-02-widen-ids-to-bigint.sql` | manual | Widens all `INTEGER UNSIGNED` id/key columns to `BIGINT UNSIGNED` across seven tables (`index_addresses`, `index_transactions`, `pubkeys`, `blocks`, `transactions`, `dispensers`, `transaction_outputs`). `mempool_transactions` is not included; the `2026-06-15-mempool-raw-strings.sql` migration later drops and recreates that table entirely with the correct types. | +| `2026-06-02-widen-ids-to-bigint.sql` | manual | Widens all `INTEGER UNSIGNED` id/key columns to `BIGINT UNSIGNED` across seven tables (`index_addresses`, `index_transactions`, `pubkeys`, `blocks`, `transactions`, `dispensers`, `transaction_outputs`). `mempool_transactions` is not included; the `2026-06-15-mempool-raw-strings.sql` migration later drops and recreates that table entirely with the correct types. Recorded as applied without running where every one of those columns is already `BIGINT UNSIGNED` and the `pubkeys_ibfk_1` foreign key exists, which is the shape a fresh install builds. | | `2026-06-13-dispensers-expiration-bigint.sql` | manual | Changes `dispensers.expiration` from `DATETIME` to `BIGINT UNSIGNED` | | `2026-06-15-events-data-mediumtext.sql` | auto | Widens `events.data` from `TEXT` to `MEDIUMTEXT` | | `2026-06-15-mempool-raw-strings.sql` | manual | Drops and recreates `mempool_transactions` with raw `VARCHAR` columns instead of FK integer IDs | diff --git a/components/decoder/operations.md b/components/decoder/operations.md index 87236f2f..8d9429ee 100644 --- a/components/decoder/operations.md +++ b/components/decoder/operations.md @@ -22,7 +22,7 @@ On startup, the decoder: 2. Starts the Express JSON-RPC API server on `DECODER_API_PORT` 3. Validates the database name (alphanumeric + underscores only) 4. Creates the database if it doesn't exist -5. Creates all 9 tables if they don't exist +5. Creates all 10 tables if they don't exist 6. Waits for the coin node to reach 99% verification progress 7. Begins parsing from the configured start block (or last parsed block + 1) @@ -103,6 +103,9 @@ Detailed health status including decoder state. "lag_blocks": 1, "rpc_errors": 0, "parse_errors": 0, + "quarantined_txs": 0, + "last_quarantined_block": null, + "last_quarantined_tx_hash": null, "dispenser_purge_failures": 0, "error": null }, @@ -132,7 +135,10 @@ Detailed health status including decoder state. | `reorg_halt_parked` | `boolean` | `true` once the parse loop has stopped on the halt and is waiting for the clear. `false` on a decoder that carries the marker but is still parsing forward, which is the only thing that separates the two | | `reorg_halt_parked_at` | `string\|null` | When the parse loop parked (ISO 8601); `null` when it is not parked | | `rpc_errors` | `integer` | Combined RPC error count from the decoder and its `BlockchainConnector` | -| `parse_errors` | `integer` | Number of transactions quarantined due to parse failures | +| `parse_errors` | `integer` | Transactions the decoder rejected or failed to parse since process start. It includes routine rejections such as other protocols' data on the chain, so it rises on a healthy decoder; `quarantined_txs` is the permanent-skip count | +| `quarantined_txs` | `integer` | Transactions this decoder skipped for good since process start, after `TX_PARSE_MAX_RETRIES` whole-block retries, each recorded as a `PARSE_ERROR` event. Also exported as `xchain_decoder_quarantined_tx_total`. Reported only: it never changes `status` or the HTTP code, since a restart cannot undo a skip. A rise means a transaction may be missing from this instance's ledger and from everything downstream | +| `last_quarantined_block` | `integer\|null` | Block height of the most recent quarantine; `null` until the first one | +| `last_quarantined_tx_hash` | `string\|null` | Hash of the most recently quarantined transaction; `null` until the first one or when the id itself could not be read | | `dispenser_purge_failures` | `integer` | Post-commit expired-dispenser purges that failed since process start, also exported as `xchain_decoder_dispenser_purge_failures_total`. A failed purge keeps soft-expired rows longer and retries at the next commit; a count that keeps rising means the dispensers table is no longer being trimmed | | `error` | `string\|null` | Error message if the decoder crashed, otherwise `null` | @@ -142,7 +148,7 @@ Returns HTTP 200 with `{status: "healthy", db, running}` when the decoder is run ### `GET /live` (REST) -The liveness probe, and the route the Docker HEALTHCHECK runs. Everything `/status` reports, plus `stalled`, `poll_silent`, `last_poll_at`, `last_processed_block`, `node_height`, `lag`, `parse_errors`, and `rpc_errors`. Returns HTTP 503 when the decoder is not running, MariaDB is unreachable, the block loop is stalled, **or** the block loop has stopped iterating. +The liveness probe, and the route the Docker HEALTHCHECK runs. Everything `/status` reports, plus `stalled`, `poll_silent`, `last_poll_at`, `last_processed_block`, `node_height`, `lag`, `parse_errors`, `quarantined_txs`, `last_quarantined_block`, `last_quarantined_tx_hash`, `last_quarantined_at`, and `rpc_errors`. Returns HTTP 503 when the decoder is not running, MariaDB is unreachable, the block loop is stalled, **or** the block loop has stopped iterating. Stalled means the loop is alive and retrying but no longer making progress the chain is waiting on: either one height has failed to fetch on many consecutive attempts, or nothing advanced for `DECODER_STALL_ALERT_MS` (default `900000` ms, 15 minutes) while the node tip was fresh and visibly ahead. The block loop never skips a block on a fetch or parse fault, because skipping would corrupt the index, so a deterministic fault at one height retries forever with the process alive and the DB answering. That case is invisible to `/status`, which is why autoheal probes `/live` instead. @@ -262,7 +268,7 @@ Mempool tracking pauses if the decoder falls more than 3 blocks behind the tip, - Verify the coin node is accessible at `NODE_URL:NODE_PORT` - Check that `verificationprogress` is >= 0.99 (run `bitcoin-cli getblockchaininfo`) -- If using Dogecoin, ensure `AUX_POW` is set in the environment +- If using Dogecoin, ensure `NETWORK` names the dogecoin chain (for example `dogecoin-mainnet`); AuxPoW stripping follows it, and `AUX_POW` has no effect ### Decoder is stuck / not advancing @@ -284,7 +290,7 @@ SELECT id, time, code, data FROM events ORDER BY id DESC LIMIT 5; ``` -The newest row decides: a `REORG_HALT` newer than every `REORG_HALT_CLEARED` is live. +The newest row decides: a `REORG_HALT` newer than every `REORG_HALT_CLEARED` is live. A newest `REORG_HALT_CLEARED` row lifts the halt only when its `data` parses to an object whose `reason` is at least 8 characters, which every audited clear writes; any other clear row (unparseable, empty, or with no or a short reason) reads as a live halt under its own id, and the audited clear below supersedes it. Two recoveries: diff --git a/components/encoder/README.md b/components/encoder/README.md index c8cd6c67..1937ac4a 100644 --- a/components/encoder/README.md +++ b/components/encoder/README.md @@ -56,7 +56,7 @@ The obfuscated payload is embedded in an `OP_RETURN` output. This is a single tr Maximum payload: **8,192 bytes** (decoder-enforced ceiling), packed in 476-byte chunks across as many outputs as needed -The payload is embedded in a redeem script, which is hashed and locked to a P2SH output in a funding transaction. Payloads larger than a single 476-byte chunk are split across multiple P2SH outputs (one fund-then-spend pair per chunk), up to the shared 8,192-byte compiled-ACTION ceiling. A second spending transaction then reveals the full redeem script in the scriptSig, making the payload visible on-chain. Two transactions must be signed and broadcast in order: +The payload is embedded in a redeem script, which is hashed and locked to a P2SH output in a funding transaction. Payloads larger than a single 476-byte chunk are split across multiple P2SH outputs (one fund tx creates them all, one spend tx reveals them all), up to the shared 8,192-byte compiled-ACTION ceiling. A second spending transaction then reveals the full redeem script in the scriptSig, making the payload visible on-chain. Two transactions must be signed and broadcast in order: 1. **Fund**: locks coin to the P2SH output containing the hashed script 2. **Spend**: spends from the P2SH output, revealing the full script (and therefore the payload) in the scriptSig @@ -139,10 +139,11 @@ npm run api | `SUGGESTED_FEE_MAX_PER_VBYTE` | No | `20` on test chains, uncapped on mainnet | Ceiling on the fee rate the encoder picks for a caller that supplies none, in base units per vByte. A quiet test chain has no fee market, so `estimatesmartfee` returns a large fallback at every confirmation target, which prices an ordinary action above the balance funding it. Mainnet is unclamped by default, where the estimate is real and a ceiling would underpay a genuine spike. Set a number to override on any chain, or `0` to disable. Bounds only the rate chosen on the caller's behalf, never a caller-supplied rate and never the anchor the fee-drain caps derive from | | `MAX_CPFP_UPLIFT_SAT` | No | `10000000` | Ceiling, in base units, on the extra fee a transaction spending unconfirmed inputs may add so that its whole mempool package reaches the target rate. Miners select by ancestor fee rate, so a transaction behind cheap unconfirmed parents stays unmined however much it pays on its own bytes. `0` turns package-aware sizing off entirely | | `FEE_NO_ESTIMATE_RELAY_MULTIPLIER` | No | `10` | Multiple of the node's own relay floor charged on a non-mainnet chain when `estimatesmartfee` has no data to answer with. Ten is the documented recommended rate and some quiet chains' miners do not accept it: on Dogecoin testnet, measured inclusion starts around 0.03 DOGE per kB while 0.0102 sits unmined. Which rate a chain accepts is a property of that chain, so measure and raise it where needed (`100` gives 0.1 DOGE per kB). Mainnet is unaffected: a missing estimate there raises an error rather than a guess | -| `FEE_ESTIMATE_SANITY_CEILING` | No | Per chain: `0.01` BTC/kB, `0.01` LTC/kB, `10` DOGE/kB | Absolute per-chain sanity ceiling, in COIN/kB, on the raw `estimatesmartfee` result before anything else sees it, including the caller-facing `MAX_FEE_RATE_KB`/`MAX_FEE_RATE_MULTIPLIER` caps, which derive their own relative ceiling from this anchor rather than bounding it directly. A non-regtest chain whose node returns a feerate above the ceiling is clamped to the ceiling and logged rather than propagated. Overrides the coin-scale default (parsed with `parseFloat`) for every chain at once; a non-positive or unparseable value keeps the per-chain default, and an unrecognized or missing `NETWORK` coin prefix falls back to the bitcoin-scale default | +| `FEE_ESTIMATE_SANITY_CEILING` | No | Per chain: `0.01` BTC/kB, `0.01` LTC/kB, `10` DOGE/kB | Absolute per-chain sanity ceiling, in COIN/kB, on the raw `estimatesmartfee` result before anything else sees it, including the caller-facing `MAX_FEE_RATE_KB`/`MAX_FEE_RATE_MULTIPLIER` caps, which derive their own relative ceiling from this anchor rather than bounding it directly. On mainnet, a node feerate above the ceiling is clamped to the ceiling and logged rather than propagated. On a test chain, the same implausible result takes the no estimate path, which uses the node's relay floor multiplied by `FEE_NO_ESTIMATE_RELAY_MULTIPLIER`. Overrides the coin-scale default (parsed with `parseFloat`) for every chain at once; a non-positive or unparseable value keeps the per-chain default, and an unrecognized or missing `NETWORK` coin prefix falls back to the bitcoin-scale default | | `API_KEY` | No | Disabled | API key for `x-api-key` header authentication | | `XCHAIN_COMPRESSION_DEFAULT` | No | On | Set to `0`/`false`/`off` to stop compressing FILE payloads by default. A deploy-time lever, not a feature switch: compressed FILEs are unreadable to a reader that predates compression support, so this exists to let the code release and the behaviour change land separately (readers everywhere first, then the encoder). Callers can always override per request with the `compress` parameter | | `ENCODER_RATE_LIMIT_RPM` | No | `60` | Maximum requests per minute per IP | +| `ENCODER_MAINNET_PRELAUNCH_OVERRIDE` | No | Unset | On a mainnet encoder before the 2027-01-01 launch instant, `create_tx` refuses every action except the validator set (ANCHOR, PRICE format 0, NODEPROOF, ROLLCALL and quorum ATTEST formats). Set to `1` only on the foundation's bootstrap encoder; the encoder logs a warning at start when it is set | | `ENCODER_MAX_RPC_BATCH` | No | `20` | Maximum calls permitted in one inbound JSON-RPC batch. The router runs `Promise.all` over a batch while the rate limiter counts the batch as a single request, so without this bound one ~1MB array fans out into thousands of concurrent `estimate_fee` / `create_tx` handlers | | `ENCODER_TRUST_PROXY` | No | `loopback, uniquelocal` | Express `trust proxy` setting. The default recovers the real client IP behind a host proxy (loopback) or a Docker bridge (uniquelocal), and ignores a forged `X-Forwarded-For` when the encoder is exposed directly. Override with `false`, a hop count (e.g. `1`), or an address/CIDR list. Mirrors the hub's `HUB_TRUST_PROXY` | | `UTXO_TRACKER_MAX_LAG_BLOCKS` | No | `2` | Maximum blocks the utxo-tracker may report as lag (its per-response `sync` field) before `create_tx` refuses to select UTXOs from it and fails with `UTXO_TRACKER_STALE` | diff --git a/components/encoder/api.md b/components/encoder/api.md index 75ba9e5f..78a91d4a 100644 --- a/components/encoder/api.md +++ b/components/encoder/api.md @@ -108,7 +108,7 @@ Probes the encoder's hard dependencies (the UTXO tracker) and reports their stat ### `estimate_fee` -Suggested fee tiers in base units per vByte (satoshis on Bitcoin, litoshis on Litecoin, koinu on Dogecoin), sourced from the coin node's `estimatesmartfee`. Takes no parameters. +Suggested fee tiers in base units per vByte (satoshis on Bitcoin, litoshis on Litecoin, koinu on Dogecoin), sourced from the coin node's `estimatesmartfee`. Tiers are normalized so a faster tier is never quoted below a slower tier: `medium` is at least `low`, and `high` is at least `medium`. Takes no parameters. **Request:** ```json diff --git a/components/explorer/api.md b/components/explorer/api.md index 1e672f60..7a711345 100644 --- a/components/explorer/api.md +++ b/components/explorer/api.md @@ -165,7 +165,7 @@ The example above shows the gate in action: `RBTC` has a frozen tip, so it is | `replica_halted` | Per-coin durable consensus-divergence halt verdict, read from the sync client's `sync_halt` table on the same replica this instance reads (`true` when an active, uncleared halt row exists). A halted replica keeps reporting a small lag until its source mints past it, so this catches what neither `stale` nor `tip_age_seconds` can see; it composes with `stale` rather than replacing it (this field flags the halt immediately, `stale`/`available` only drop the coin once its tip actually ages out). `true`/`false` only once the table was read successfully; `null` when the signal could not be determined (no live pool, the table doesn't exist on this replica, or the read failed), and `null` is never coerced to `false`. Only coins this instance measures appear here | | `chain_tip` | Per-coin chain tip as reported by the decoder's own health endpoint (what the coin node sees) | | `chain_lag_blocks` | `chain_tip − decoder_tip` (how far the decoder trails the chain) | -| `decoder_health` | Per-coin decoder health string: `"healthy"`, `"unhealthy"` (decoder up but reporting problems), `"node-stale"` (the decoder's cached coin-node height is frozen, so `chain_tip` and `chain_lag_blocks` are nulled), `"unreachable"` (decoder not responding), or `"unconfigured"` (no decoder endpoint resolves for this coin, from either the explorer's configuration or `DECODER_API_URL[__]`; see [Configuration](configuration.md)) | +| `decoder_health` | Per-coin decoder health string: `"healthy"`, `"unhealthy"` (decoder up but reporting problems), `"reorg-halt-parked"` (the decoder has stopped on a durable REORG_HALT marker that only an operator clear removes), `"reorg-halted"` (the decoder carries that marker but is still parsing; both halt values outrank every value after them), `"node-stale"` (the decoder's cached coin-node height is frozen, so `chain_tip` and `chain_lag_blocks` are nulled), `"unreachable"` (decoder not responding), or `"unconfigured"` (no decoder endpoint resolves for this coin, from either the explorer's configuration or `DECODER_API_URL[__]`; see [Configuration](configuration.md)) | | `hub_consensus_hash_mismatch` | Explorer-to-hub coin-configuration integrity verdict: `true`, `false`, or `null`. `true` means at least one consensus hash served by the hub differs from the explorer's bundled coin registry, `false` means every hash served for a bundled coin matches, and `null` means no verdict is available because the explorer has not received the hash map or the hub is too old to send it | | `hub_consensus_hash_mismatch_details` | One string per mismatch in `TICK/network: hub HASH vs bundled HASH` form. Empty when hashes match or no verdict is available | diff --git a/components/explorer/configuration.md b/components/explorer/configuration.md index f9d4d352..a745e3d2 100644 --- a/components/explorer/configuration.md +++ b/components/explorer/configuration.md @@ -22,6 +22,7 @@ Hub-sourced configuration takes precedence for database connection details, allo |---|---|---|---| | `EXPLORER_API_PORT_HTTP` | No | `8080` | HTTP server port | | `EXPLORER_API_PORT_HTTPS` | No | `8081` | HTTPS server port | +| `EXPLORER_CONTENT_VIEWER_ORIGIN` | No | None | HTTPS origin (scheme, host and optional port only) that serves the custom token content viewer page; token pages frame custom content from it. Anything else, or unset, leaves the viewer off. | | `EXPLORER_DEFAULT_THEME` | No | `classic` | Theme served to a visitor who has not picked one: a directory name under `src/content/themes` (`classic`, `skin-demo`). An unknown name falls back to `classic`. | | `API_HOST` | No | `127.0.0.1` | Bind address for the API server | | `DEBUG` | No | None | Enable debug output when set to any truthy value | @@ -30,6 +31,10 @@ Hub-sourced configuration takes precedence for database connection details, allo | `EXPLORER_FORCE_HTTPS` | No | None | Explicitly enable (`1`) or disable (`0`) the HTTPS-hardening headers (HSTS and `upgrade-insecure-requests`). By default they are active only when `NODE_ENV=production`, so plain-HTTP dev/regtest deploys are not broken. Set to `1` when running behind a TLS-terminating proxy without `NODE_ENV=production`. | | `EXPLORER_HOLDERS_CACHE_MS` | No | `15000` | TTL (ms) of the per-tick holders-query result cache. `getHolders` requires an unindexable full-table filesort, so results are cached briefly to bound repeated-query cost. | | `EXPLORER_HOLDERS_CACHE_MAX` | No | `500` | Maximum number of distinct holders-query results kept in the cache. | +| `EXPLORER_TWO_PARTY_DENSE_ROWS` | No | `10000` | Row count at or over which an address of a two-party action family is paged through the ordered-walk query instead of the id-set query. 10,000 balances a family of about 1.1M rows; a non-numeric or non-positive value falls back to the default. | +| `EXPLORER_BIND_HOST` | No | `127.0.0.1` | Interface both API listeners bind. Loopback by default so the one trusted proxy hop is the only way in; the container image sets `0.0.0.0` so its published port reaches it. An empty value counts as unset. | +| `EXPLORER_FEESCHEDULE_CACHE_MS` | No | `15000` | Lifetime (ms) of a cached per-indexer fee schedule before it is refetched. Read per call through the config env view. | +| `EXPLORER_FEESCHEDULE_STALE_MAX_MS` | No | `120000` | Longest age (ms) at which a cached fee schedule is still served while the indexer is failing, bridging a short indexer outage. Read per call through the config env view. | | `EXPLORER_TOTALS_CACHE_MS` | No | `60000` | TTL (ms) of the platform-totals query cache. | | `MEMPOOL_COUNT_CACHE_MS` | No | `15000` | TTL (ms) of the mempool-count cache. | | `PRICE_CACHE_MS` | No | `60000` | TTL (ms) of the oracle-price cache. | @@ -186,6 +191,17 @@ The explorer carries a byte-identical copy of the shared activation rows the ind | `XC_AMOUNTS_PRICE_REGTEST_TIME` | No | unset (inert) | **Regtest only.** Arms the `regtest` entries of `AMOUNT_REPRESENTABILITY_ACTIVATION`, a time-keyed gate in the explorer's registry copy. `armed`, `genesis` or `on` arms at `0`; a non-negative integer arms at that value (a unix time in seconds). Unset or `off` leaves the gate inert; anything else is refused with a warning and leaves the gate inert. | | `XC_LISTS_MARKET_REGTEST_ACTIVATION` | No | unset (inert) | **Regtest only.** Arms the `regtest` entries of the eleven height-keyed LISTS MARKET gates (the `XC_LISTS_MARKET` rows) in the explorer's registry copy. `armed`, `genesis` or `on` arms at `0`; a non-negative integer arms at that height. Unset or `off` leaves the gates inert; anything else is refused with a warning and leaves them inert. | | `XC_LISTS_MARKET_REGTEST_TIME` | No | unset (inert) | **Regtest only.** Arms the `regtest` entries of the three time-keyed LISTS MARKET gates in the explorer's registry copy. `armed`, `genesis` or `on` arms at `0`; a non-negative integer arms at that value (a unix time in seconds). Unset or `off` leaves the gates inert; anything else is refused with a warning and leaves them inert. | +| `XC_REMOTE_TOKEN_ADMISSION_REGTEST_ACTIVATION` | No | unset (inert) | **Regtest only.** Arms the `regtest` entry of `REMOTE_TOKEN_ADMISSION_ACTIVATION` in the explorer's registry copy. `armed`, `genesis` or `on` arms at `0`; a non-negative integer arms at that height. Unset or `off` leaves the gate inert; anything else is refused with a warning and leaves the gate inert. Set identically on the hub and every indexer of the venue. | +| `XC_PASS_PROGRESS_RETIREMENT_REGTEST_ACTIVATION` | No | unset (inert) | **Regtest only.** Arms the `regtest` entry of `PASS_PROGRESS_RETIREMENT_ACTIVATION` in the explorer's registry copy. `armed`, `genesis` or `on` arms at `0`; a non-negative integer arms at that height. Unset or `off` leaves the gate inert; anything else is refused with a warning and leaves the gate inert. Set identically on the hub and every indexer of the venue. | +| `XC_SLASH_HISTORICAL_SET_REGTEST_ACTIVATION` | No | unset (inert) | **Regtest only.** Arms the `regtest` entry of `SLASH_HISTORICAL_SET_ACTIVATION` in the explorer's registry copy. `armed`, `genesis` or `on` arms at `0`; a non-negative integer arms at that height. Unset or `off` leaves the gate inert; anything else is refused with a warning and leaves the gate inert. Set identically on the hub and every indexer of the venue. | +| `XC_ANCHOR_REWARD_COPY_SELECTION_REGTEST_ACTIVATION` | No | unset (inert) | **Regtest only.** Arms the `regtest` entry of `ANCHOR_REWARD_COPY_SELECTION_ACTIVATION` in the explorer's registry copy. `armed`, `genesis` or `on` arms at `0`; a non-negative integer arms at that height. Unset or `off` leaves the gate inert; anything else is refused with a warning and leaves the gate inert. Set identically on the hub and every indexer of the venue. | +| `XC_TOKEN_SUPPLY_REPRESENTATION_REGTEST_ACTIVATION` | No | unset (inert) | **Regtest only.** Arms the `regtest` entry of `TOKEN_SUPPLY_REPRESENTATION_ACTIVATION` in the explorer's registry copy. `armed`, `genesis` or `on` arms at `0`; a non-negative integer arms at that height. Unset or `off` leaves the gate inert; anything else is refused with a warning and leaves the gate inert. Set identically on the hub and every indexer of the venue. | +| `XC_DISPENSER_FIAT_PRICING_REGTEST_ACTIVATION` | No | unset (inert) | **Regtest only.** Arms the `regtest` entry of `DISPENSER_FIAT_PRICING_ACTIVATION` in the explorer's registry copy. `armed`, `genesis` or `on` arms at `0`; a non-negative integer arms at that height. Unset or `off` leaves the gate inert; anything else is refused with a warning and leaves the gate inert. Set identically on the hub and every indexer of the venue. | +| `XC_BRIDGE_POLICY_APPLY_ORDER_REGTEST_ACTIVATION` | No | unset (inert) | **Regtest only.** Arms the `regtest` entry of `BRIDGE_POLICY_APPLY_ORDER_ACTIVATION` in the explorer's registry copy. `armed`, `genesis` or `on` arms at `0`; a non-negative integer arms at that height. Unset or `off` leaves the gate inert; anything else is refused with a warning and leaves the gate inert. Set identically on the hub and every indexer of the venue. | +| `XC_VM_EMISSION_AMOUNT_FORMAT_REGTEST_ACTIVATION` | No | unset (inert) | **Regtest only.** Arms the `regtest` entry of `VM_EMISSION_AMOUNT_FORMAT_ACTIVATION` in the explorer's registry copy. `armed`, `genesis` or `on` arms at `0`; a non-negative integer arms at that height. Unset or `off` leaves the gate inert; anything else is refused with a warning and leaves the gate inert. Set identically on the hub and every indexer of the venue. | +| `XC_EXECUTE_FEE_CAP_REGTEST_ACTIVATION` | No | unset (inert) | **Regtest only.** Arms the `regtest` entry of `EXECUTE_FEE_CAP_ACTIVATION` in the explorer's registry copy. `armed`, `genesis` or `on` arms at `0`; a non-negative integer arms at that height. Unset or `off` leaves the gate inert; anything else is refused with a warning and leaves the gate inert. Set identically on the hub and every indexer of the venue. | +| `XC_VOTE_V3_TICK_EXISTENCE_REGTEST_ACTIVATION` | No | unset (inert) | **Regtest only.** Arms the `regtest` entry of `VOTE_V3_TICK_EXISTENCE_ACTIVATION` in the explorer's registry copy. `armed`, `genesis` or `on` arms at `0`; a non-negative integer arms at that height. Unset or `off` leaves the gate inert; anything else is refused with a warning and leaves the gate inert. Set identically on the hub and every indexer of the venue. | +| `XC_MATCH_ORIGIN_OPEN_REGTEST_ACTIVATION` | No | unset (inert) | **Regtest only.** Arms the `regtest` entry of `MATCH_ORIGIN_OPEN_ACTIVATION` in the explorer's registry copy. `armed`, `genesis` or `on` arms at `0`; a non-negative integer arms at that height. Unset or `off` leaves the gate inert; anything else is refused with a warning and leaves the gate inert. Set identically on the hub and every indexer of the venue. | | `XC_CONTRACTS_REGTEST_ACTIVATION` | No | unset (inert) | **Regtest only.** Arms the `regtest` entries of `VOTE_CALLBACK_BINDING_REQUIRES_USABLE_METHOD` in the explorer's registry copy. `armed`, `genesis` or `on` arms at `0`; a non-negative integer arms at that height. Unset or `off` leaves the gate inert; anything else is refused with a warning and leaves the gate inert. | | `OWNER_WITHDRAW_OPT_IN_REGTEST_TIME` | No | `0` (activate at genesis) | **Regtest only.** Not one of the four shared-grammar variables above: `src/contract/owner_withdraw.js` reads it directly with a plain `parseInt` (unset or unparseable falls back to `0`) to override the `OWNER_WITHDRAW_OPT_IN` activation time the explorer's owner-withdraw presentation judges against, mirroring the indexer's own regtest override of the same name. Deliberately regtest-scoped: two nodes with different values would disagree on consensus. | @@ -392,6 +408,7 @@ The explorer uses `express-rate-limit` middleware: | `EXPLORER_FEE_QUOTE_RATE_LIMIT_RPM` | No | `120` | Separate limit for the fee lookups `/{COIN}/api/feequote`, `/{COIN}/api/oraclefeequote` and `/{COIN}/api/feeschedule`. One tier looser than the proof routes because a quote is a lookup rather than a cryptographic recompute. | | `EXPLORER_CHECKPOINT_LIST_RATE_LIMIT_RPM` | No | `120` | Separate limit for `/{COIN}/api/checkpoints`, which lists stored checkpoints. | | `EXPLORER_CHECKPOINT_VERIFY_RATE_LIMIT_RPM` | No | `90` | Separate, tighter limit for `/{COIN}/api/checkpoint/{blockIndex}/verify`, which recomputes a checkpoint rather than reading one, derived from the measured wallet profile's one verify per proof job with retries and 3x headroom. | +| `EXPLORER_STATIC_RATE_LIMIT_RPM` | No | `1200` | Separate limit for static files (scripts, stylesheets, components, themes and fonts). The default is ten cold page loads of about 110 files each; these are disk reads, so the bucket only has to stop a flood, and a warm visitor sends almost none. | | `WS_TRUST_PROXY_HOPS` | No | `1` | Proxy hop count used to resolve the real client address for the WebSocket per-IP cap. The upgrade is handled on the raw HTTP server, where Express's `trust proxy` does not apply, so the hop count must be passed explicitly or the cap keys on a spoofable `X-Forwarded-For`. Keep it aligned with the HTTP side. | Rate limiting applies to all non-image endpoints (API, Explorer, and HTML). diff --git a/components/explorer/websocket.md b/components/explorer/websocket.md index 2b4c44d6..0e949432 100644 --- a/components/explorer/websocket.md +++ b/components/explorer/websocket.md @@ -65,6 +65,7 @@ Sent automatically on connection. Provides server info, current state, limits, a "limits": { "max_subscriptions": 25, "max_message_rate": 10, + "max_message_burst": 27, "max_message_size": 1024, "max_connections_per_ip": 5 }, @@ -75,7 +76,9 @@ Sent automatically on connection. Provides server info, current state, limits, a } ``` -Use `latest_block_index` and `latest_action_index` to seed your local state for catch-up on reconnect when you hold no cursor yet; on a reconnect they report the current tip, so they must not move a cursor you already have (see [Reconnection and Catch-Up](#reconnection-and-catch-up)). +Use `latest_action_index` to seed your catch-up cursor when you hold no cursor yet; on a reconnect it reports the current live position, so it must not move a cursor you already have (see [Reconnection and Catch-Up](#reconnection-and-catch-up)). + +`latest_action_index` is the live feed's position for the coin: the highest `action_index` the live poll has already sent, at or above which every later row reaches a subscriber live. It is not the database tip, which can sit a poll ahead of it; a cursor seeded from the tip would skip the rows between the two if the connection dropped before the next poll. When the coin's first poll has not run yet (the server polls coins in turn), the server seeds that coin before it answers, so `WELCOME` can arrive a database read later than usual. A server running without a live detector for the coin reports the database tip. `latest_block_index` is the database's highest block. --- @@ -83,6 +86,8 @@ Use `latest_block_index` and `latest_action_index` to seed your local state for All client messages are JSON with an `action` field. An optional `id` field enables request-response correlation; the server echoes it back in the corresponding response. +Each connection's messages are measured as a leaky bucket that holds `limits.max_message_burst` messages and drains at `limits.max_message_rate` per second (`WELCOME`). The burst is the larger of the rate and `max_subscriptions + 2` (27 with the defaults of 10 per second and 25 subscriptions), so a client can resubscribe every subscription it may hold, plus one catch-up and a ping, in one tick after a reconnect; past the burst, one more message is accepted per `1000 / max_message_rate` ms. A message over the bucket is dropped and answered with a `RATE_LIMITED` error that carries the dropped message's `id` (no `id` when the message had none or was not JSON). Send a dropped request again once the bucket has drained. Both bundled clients pace every send one slot under the advertised rate and retry a subscribe or catch-up dropped with `RATE_LIMITED`; a client written before the `id` was carried reads that refusal as final, which the burst keeps it from meeting on a reconnect. + ### subscribe ```json @@ -132,6 +137,8 @@ All client messages are JSON with an `action` field. An optional `id` field enab An `xcall` call id must be 64 hex characters; it is lower-cased at subscribe time, and a missing or malformed id is refused with `INVALID_CHANNEL`. +An `address` (and every entry of `addresses`) must be a non-empty string; any other value is refused with `INVALID_CHANNEL` and nothing in the request is subscribed. + ### unsubscribe ```json @@ -145,6 +152,8 @@ An `xcall` call id must be 64 hex characters; it is lower-cased at subscribe tim The server confirms with one `UNSUBSCRIBED` frame (`reason: "client_request"`) per targeted entity; see [UNSUBSCRIBED](#unsubscribed). +A channel name the server does not know, or an entity channel whose params name no entity (or name one that is not a valid key, such as a non-string `address`), is answered with an `INVALID_CHANNEL` error echoing the request's `id`, one per such channel. The other channels in the same request are still unsubscribed and confirmed. + ### list_subscriptions ```json @@ -189,10 +198,19 @@ renamed, retyped, or restructured in a way an existing consumer could mis-parse (additive optional fields do not bump it). The official SDK warns when the server reports a newer schema than it was built against. +**`stale: true`** sits at the top level of the envelope on `WELCOME`, +replayed catch-up frames, `CATCH_UP_COMPLETE`, `SNAPSHOT` and live frames +while the coin's indexed tip is older than its maximum age. The rows are +still real, but they end at a tip that is behind the chain. `WELCOME` also +repeats it as `data.stale`, beside the tip indices it qualifies. The field +is absent, never `false`, while the tip is fresh, so one `if (msg.stale)` +check covers every frame. + **Wire types under schema v2:** BigInt-backed fields (`action_index`, `block_index`, `block_time`, `latest_block_index`, `latest_action_index`, -the `NEW_BLOCK` counts `tx_count` and `action_count`, and other -chain-index/amount fields) serialize as **decimal strings**, not +the `NEW_BLOCK` counts `tx_count` and `action_count`, the `TOKEN_UPDATE` +holder count `holders`, and other chain-index/amount fields) serialize as +**decimal strings**, not JSON numbers, matching the REST API. Compare them as strings or parse with `BigInt(...)`; a numeric `===` against them will silently fail. `timestamp` and rate-limit metadata remain JSON numbers. The payload examples below show @@ -291,7 +309,7 @@ Channel: `token` "data": { "tick": "PEPE", "supply": "100000000.00000000", - "holders": 1234, + "holders": "1234", "last_action_index": "45680" } } @@ -425,7 +443,10 @@ the `BET_CLOSED` cursor is. ### Order Lifecycle Events -These fire on both the `actions` channel and the `address` channel for involved addresses. +These fire on the `actions` channel and on the `address` channel of each address the frame's +`data` names: its `source`, or on `COINPAY_REQUIRED` its `payer_address` and `payee_address`. +A lifecycle frame does not carry the `destinations` a `NEW_ACTION` lists, and it is not sent to +their `address` channels. #### ORDER_MATCH @@ -607,7 +628,7 @@ Emitted when the mempool row is promoted to a confirmed transaction or evicted. Channel: `attestation` -Emitted when an ATTEST action lands in a confirmed block. +Emitted when an ATTEST request, response or expiry lands in a confirmed block. Relay legs (ATTEST v3/v4) arrive as the request or response they record. Response-batch actions (ATTEST v5 head, v6 continuation) send no event on this channel; they still arrive as `NEW_ACTION` on `actions`. #### ATTESTATION_REQUEST @@ -624,7 +645,7 @@ Emitted for ATTEST v0 (request) rows. #### ATTESTATION_RESPONSE -Emitted for ATTEST v1 (response) and ATTEST v2 (system expiry) rows. +Emitted for ATTEST v1 (response) rows and for ATTEST v2 (system expiry) actions. ```json { @@ -635,6 +656,14 @@ Emitted for ATTEST v1 (response) and ATTEST v2 (system expiry) rows. } ``` +An expiry writes no attestation row of its own; it marks its request expired. Its frame therefore names the expiry action: + +- `action_index` is the expiry action, and `version` is `2`. +- `request_id`, `provider_id` and `contract_index` come from the request it retired. +- `request_status` is `expired` and `response_status` is `null`. + +When the explorer cannot tell which request an expiry retired, it sends no frame for that expiry; the request's `expired` status is still readable over REST. + --- ### System Events @@ -697,6 +726,35 @@ Sent when subscribing with `snapshot: true`. Contains current state of the subsc An `xcall` snapshot is `{ "channel": "xcall", "call_id": "...", ...the call's current lifecycle fields }`; for a `call_id` this chain has no row for, it carries only the two identity fields. +#### REORG + +Sent to every connection on the coin that holds at least one subscription when the live feed detects that the chain was rolled back below rows or blocks it already announced. It arrives before any replacement frame. + +```json +{ + "type": "REORG", + "chain": "BTC", + "network": "mainnet", + "timestamp": 1743638400000, + "schema_version": 2, + "data": { + "fork_block_index": "890120", + "first_invalid_action_index": "45601" + } +} +``` + +| Field | Meaning | +|---|---| +| `fork_block_index` | The lowest height whose announced block is no longer on the chain. Every `NEW_BLOCK` you hold at or above it is void. | +| `first_invalid_action_index` | The lowest `action_index` whose announced row may be gone or replaced. Every `NEW_ACTION` (and frame derived from it) you hold at or above it is void. | + +After the frame, the server sends the new chain's blocks from `fork_block_index` and its rows from `first_invalid_action_index` as ordinary live frames. The indexer reassigns `action_index` on a reorg, so a replacement row can carry the same index as the void row it replaces with different contents: drop what you hold at or above `first_invalid_action_index` first, then apply the frames that follow. A rollback whose replacement chain is shorter leaves the void indexes above its tip empty until new rows are committed there. + +Move your catch-up cursor to `first_invalid_action_index - 1` when it is higher, so a reconnect before the replacements arrive replays them. Both bundled clients do this. `REORG` is an additive frame type under schema v2; a client that ignores unknown types keeps working but goes on holding the void rows. A connection too backed up to take the frame is closed with `4008` instead of skipping it, and its reconnect catch-up is refused with `CATCH_UP_REORGED` (see [Reconnection and Catch-Up](#reconnection-and-catch-up)). + +A rollback is detected either by the database layer's tip check or by a poll whose fetch comes back short of the tip it read (rows were rolled back between the two reads); in the second case the next poll compares the blocks it announced against the chain and sends `REORG` only if one of them was replaced. Rows the server never announced need no `REORG`: the live cursor stops at the last row it fetched, so their replacements arrive as ordinary new rows. + #### CATCH_UP_COMPLETE Sent after all catch-up events have been replayed. @@ -708,11 +766,13 @@ Sent after all catch-up events have been replayed. "events_replayed": 5, "latest_action_index": "45690", "truncated": false, - "not_replayed": ["BET_CLOSED", "XCALL_COMPLETED", "XCALL_EXPIRED"] + "not_replayed": ["BET_CLOSED", "XCALL_COMPLETED", "XCALL_EXPIRED", "NEW_BLOCK"] } } ``` +`not_replayed` names `NEW_BLOCK` when the request names the `blocks` channel, as above; without `blocks` it lists only the three lifecycle types. + Catch-up events have `"catch_up": true` in the envelope to distinguish them from live events. `events_replayed` counts every frame sent; `truncated` and `latest_action_index` count action rows. @@ -726,6 +786,20 @@ even when they are committed before the query runs. As a result, a reconnecting each matching `NEW_ACTION` exactly once per subscription key rather than once from replay and again from the live poll. +Every ceiling is also bounded by the coin's current live mark, the position `WELCOME` reports, so a +replay never reads a row the live poll has yet to send. A catch-up on a coin whose first poll has not +run yet seeds that coin first and replays up to the seed; when the coin cannot be seeded (its +database read fails), the catch-up is refused with the retryable `CATCH_UP_NOT_READY`. + +The replay honours `WS_MAX_BACKPRESSURE` at each row boundary: before each replayed row the server +checks the connection's buffered bytes, and above the limit it waits for the socket to drain below +it before sending the row, so the server never buffers more than about one row past the limit and a +slow link still receives the whole replay. A connection that does not drain within 30 seconds is +closed with code `4008` (`backpressure`) at that row boundary, with no `CATCH_UP_COMPLETE`. Its +cursor still sits before the first row it did not receive, so its reconnect replays from there. While +a replay waits, the connection's catch-up latch stays held, so another `since_action_index` +subscribe on it is refused with `CATCH_UP_IN_PROGRESS`. + The boundary does not make the entire row atomic. Lifecycle frames are derived after `NEW_ACTION` and that work can yield. A subscription that starts while the live detector is still deriving the boundary row can receive one of that row's lifecycle frames both from replay and from the in-flight @@ -740,6 +814,8 @@ lifecycle event the live feed derives from that row (`ORDER_MATCH`, `COINPAY_REQ | Not replayed | Why | Recover with | |---|---|---| | The types in `not_replayed` | Emitted by a status cursor of their own, not derived from an action row | `GET /{COIN}/api/bet_feed/{action_index}` and `GET /{COIN}/api/xcall/{callId}` ([REST API](api.md)) | +| `NEW_BLOCK` | A replay is keyed on `action_index`, and a block has none; `not_replayed` names it when the request names `blocks` | The [REST blocks list](api.md) from your last seen `block_index` | +| `REORG` | A live notice; a catch-up across a logged rollback is refused with `CATCH_UP_REORGED` instead | Resync over REST from the refusal's `fork_block_index` | | `ADDRESS_UPDATE` and the other entity-update frames | State, not events | Subscribe with `snapshot: true` | | `MEMPOOL_ACTION` / `MEMPOOL_REMOVED` | Unconfirmed and transient | The [REST mempool read](api.md#mempool) | @@ -810,14 +886,16 @@ Response to client `ping`. | Code | Meaning | |---|---| | `INVALID_ACTION` | Unrecognized client action or malformed JSON | -| `INVALID_CHANNEL` | Unknown channel name or missing entity params | +| `INVALID_CHANNEL` | Unknown channel name, missing entity params, or an invalid entity key (such as a non-string `address`), on `subscribe` and on `unsubscribe` | | `INVALID_PARAMS` | A malformed param: `fields` is not an array of strings, or `since_action_index` is not a non-negative integer | | `INVALID_TYPE` | Unknown action type in `types` filter | | `SUBSCRIPTION_LIMIT` | Exceeded max 25 subscriptions per connection | -| `RATE_LIMITED` | Client sending more than 10 messages/sec | +| `RATE_LIMITED` | Client sent more than `max_message_burst` messages at once (27 by default) or more than `max_message_rate` (10 by default) per second after it; the dropped message was not processed, and the frame echoes its `id` so the client can send it again | | `CATCH_UP_TOO_OLD` | `since_action_index` is more than 1,000 actions behind current | | `CATCH_UP_AHEAD_OF_TIP` | `since_action_index` is above the server's latest `action_index` (a reorg lowered the tip, or this server is behind the one that issued your cursor), so nothing can be replayed from it; resync over REST | | `CATCH_UP_IN_PROGRESS` | A catch-up request is already running for this client | +| `CATCH_UP_REORGED` | A rollback this server logged replaced rows at or below `since_action_index`, so a replay from it would skip their replacements. The frame adds `fork_block_index` and `first_invalid_action_index`; resync over REST from the fork. See [Reconnection and Catch-Up](#reconnection-and-catch-up) | +| `CATCH_UP_NOT_READY` | The coin's live feed has not recorded its first position and could not be seeded now; retry the catch-up shortly | | `COIN_DATA_STALE` | The coin's indexed tip is older than its maximum age and the server is set to fail closed, so the snapshot or catch-up is refused rather than served as current | | `SNAPSHOT_QUEUE_FULL` | A snapshot fan-out is already running and this client's pending-snapshot queue (capped at its subscription limit) is full; retry the subscribe with `snapshot: true` once it completes | | `INTERNAL_ERROR` | The server threw while handling the message; the request `id` is echoed | @@ -876,9 +954,9 @@ with `tx_hash` and `source` null and `destinations: []`. ## Reconnection and Catch-Up -1. Track the highest `action_index` you have processed. Seed it from `WELCOME`'s `latest_action_index` only when you hold no cursor yet: on a reconnect, `WELCOME` reports the current tip, which is past the actions you missed, so letting it move your cursor skips them. -2. On disconnect, reconnect with exponential backoff. For action-derived frames, backpressure admission is decided once per action row, when the first frame that passes this client's subscriptions and filters is ready: at or below `WS_MAX_BACKPRESSURE` the server sends every wanted frame from that row, even if those sends take the buffer above the limit; above the limit it sends no frame from that row and closes with code `4008` (`backpressure`) after the row ends. Handle that close like any other disconnect: your cursor still sits before the rejected row, so catch-up replays the whole row rather than repairing a partial one. -3. Resubscribe with `since_action_index` set to your last known value, on the `actions` and `address` subscriptions. Their replay carries each missed action's `NEW_ACTION` and its lifecycle events. A `since_action_index` subscribe on any other channel alone closes at once with nothing replayed: the `actions` channel already carries every lifecycle frame the `dispenser`, `bet_feed`, `xcall` and `attestation` channels do, and entity state comes back with `snapshot: true`. Backfill the `not_replayed` types over REST (see [CATCH_UP_COMPLETE](#catch_up_complete)). +1. Track the highest `action_index` you have processed. Seed it from `WELCOME`'s `latest_action_index` only when you hold no cursor yet: on a reconnect, `WELCOME` reports the current live position, which is past the actions you missed, so letting it move your cursor skips them. On a `REORG` frame, lower the cursor to `first_invalid_action_index - 1` when it is higher (see [REORG](#reorg)). +2. On disconnect, reconnect with exponential backoff. For action-derived frames, backpressure admission is decided once per action row, when the first frame that passes this client's subscriptions and filters is ready: at or below `WS_MAX_BACKPRESSURE` the server sends every wanted frame from that row, even if those sends take the buffer above the limit; above the limit it sends no frame from that row and closes with code `4008` (`backpressure`) after the row ends. Handle that close like any other disconnect: your cursor still sits before the rejected row, so catch-up replays the whole row rather than repairing a partial one. The replay itself checks the same limit at each row boundary and waits for the socket to drain rather than closing it (see [CATCH_UP_COMPLETE](#catch_up_complete)). +3. Resubscribe with `since_action_index` set to your last known value, on the `actions` and `address` subscriptions. Their replay carries each missed action's `NEW_ACTION` and its lifecycle events. A `since_action_index` subscribe on any other channel alone closes at once with nothing replayed: the `actions` channel already carries every lifecycle frame the `dispenser`, `bet_feed`, `xcall` and `attestation` channels do, and entity state comes back with `snapshot: true`. Backfill the `not_replayed` types over REST (see [CATCH_UP_COMPLETE](#catch_up_complete)). `NEW_BLOCK` is never replayed: send `since_action_index` on your `blocks` subscription too and its `CATCH_UP_COMPLETE` names `NEW_BLOCK` in `not_replayed`, then backfill blocks over REST. Send all of these subscribes under the message rate (see [Client-to-Server Messages](#client-to-server-messages)), and send again any that a `RATE_LIMITED` frame names by `id`. 4. Send one `since_action_index` subscribe at a time, each with its own `id`, and wait for the `CATCH_UP_COMPLETE` or `error` frame that echoes that `id` before sending the next. The server runs one catch-up per connection and refuses an overlapping one with `CATCH_UP_IN_PROGRESS`; the subscription is still registered, but its missed actions are not replayed. 5. Process events with `catch_up: true` as replayed frames. The server uses ordering option B: it does not gate live delivery while replay is running. Replay preserves action-index order within @@ -888,9 +966,11 @@ with `tx_hash` and `source` null and `destinations: []`. from them until every catch-up has closed. Also de-duplicate lifecycle frames for the replay's boundary row as described under [CATCH_UP_COMPLETE](#catch_up_complete). 6. If `CATCH_UP_COMPLETE` has `truncated: true`, the replay stopped at its row cap: send the same subscribe again with `since_action_index` set to that frame's `latest_action_index`, and repeat until a frame arrives with `truncated: false` -7. If the catch-up is refused (`CATCH_UP_TOO_OLD`, `CATCH_UP_AHEAD_OF_TIP`, or any other `error` carrying its `id`), use the REST API to backfill +7. If the catch-up is refused (`CATCH_UP_TOO_OLD`, `CATCH_UP_AHEAD_OF_TIP`, `CATCH_UP_REORGED`, or any other `error` carrying its `id` except `RATE_LIMITED` and `CATCH_UP_NOT_READY`, which you retry), use the REST API to backfill. `CATCH_UP_REORGED` means a rollback while you were away replaced rows at or below your cursor: drop what you hold at or above its `first_invalid_action_index` and reload from its `fork_block_index`. + +The server keeps the last 64 rollbacks per coin, in memory, for the `CATCH_UP_REORGED` check. A rollback logs the range from its `first_invalid_action_index` to the highest index the live feed had sent before it; a cursor inside that range is refused, because a client that saw the old row and one that saw its replacement hold the same number. A cursor below the range, or above it (only the new chain reached it), replays normally. A client that applied the `REORG` live and then dropped with its cursor still inside the range is refused too, and reloads over REST. Rollbacks older than the last 64, or from before a server restart, are not in the log. -The SDK's WebSocket client and the explorer's bundled browser client both follow these steps, and each reports a refused or unanswered catch-up as a `resync_required` event. +The SDK's WebSocket client and the explorer's bundled browser client both follow these steps, and each reports a refused or unanswered catch-up as a `resync_required` event; for `CATCH_UP_REORGED` the event carries `fork_block_index` and `first_invalid_action_index`. Each sends its `blocks` subscription through the same one-at-a-time catch-up queue, so it reports the `NEW_BLOCK` note as a `not_replayed` event. ```mermaid sequenceDiagram @@ -905,7 +985,7 @@ sequenceDiagram Note over C,S: disconnect C->>S: reconnect - S-->>C: WELCOME (tip, does not move the cursor) + S-->>C: WELCOME (live position, does not move the cursor) C->>S: subscribe actions, since_action_index, id c1 S-->>C: replayed events S-->>C: higher live event may interleave @@ -931,4 +1011,4 @@ See [CONFIGURATION.md](configuration.md) for the `WS_*` environment variables th | `WS_IDLE_TIMEOUT` | `300000` | Idle timeout for zero-subscription clients (ms) | | `WS_MAX_CONNECTIONS_PER_IP` | `5` | Max concurrent connections per IP | | `WS_MAX_SUBSCRIPTIONS` | `25` | Max subscriptions per connection | -| `WS_MAX_BACKPRESSURE` | `65536` | Max buffered bytes per client. Backpressure admission is decided once per action row at its first wanted frame: at or below the limit the whole wanted row is sent, while above it no frame from that row is sent and the connection closes with code `4008` (`backpressure`) after the row ends. Reconnect and catch up from the last processed action index. | +| `WS_MAX_BACKPRESSURE` | `65536` | Max buffered bytes per client. Backpressure admission is decided once per action row at its first wanted frame: at or below the limit the whole wanted row is sent, while above it no frame from that row is sent and the connection closes with code `4008` (`backpressure`) after the row ends. A catch-up replay checks the same limit before each replayed row and waits up to 30 seconds for the socket to drain below it before closing, and a `REORG` frame that finds a client above it closes that client instead of skipping the frame. Reconnect and catch up from the last processed action index. | diff --git a/components/hub/architecture.md b/components/hub/architecture.md index 3765964c..36f735cb 100644 --- a/components/hub/architecture.md +++ b/components/hub/architecture.md @@ -414,12 +414,14 @@ Indexers bootstrap by fetching the REST snapshots (paginated by `since_id`) then ### Trimmed Median -Given N validator submissions for a coin pair: -1. Sort all submissions by price. -2. Discard the top 15% and bottom 15%. -3. Compute the median of the remaining values. +Given the price-snapshot members' submissions for a coin pair (one value per submitter, each strictly between 0 and `PRICE_MAX`): +1. Sort the values by exact decimal value. +2. Remove `ceil(15% of N)` values from each end, when at least one value remains. The trim is symmetric, so it never moves the middle value or values: the result is the plain median of all the submissions. +3. Take the median. An odd count publishes the middle value; an even count publishes the mean of the two middle values, rounded once to 8 decimals. +4. Even-split gate: for an even count, if either middle value lies more than `ORACLE_DEVIATION_THRESHOLD` (5%) from that mean, the pair is omitted from the round and consumers keep its last finalized price. A median that rounds to zero is omitted the same way. +5. Per-round clamp: the result is bounded to `ORACLE_MAX_CHANGE_PER_ROUND` (25%) either side of the pair's last finalized price, or `XCHAIN_PRICE_MAX_CHANGE_PER_ROUND` (10%) for XCHAIN/USD, and a larger move publishes at the bound. A pair with no finalized price yet is not clamped. -This resists manipulation: an attacker would need to control >30% of validators to significantly shift the price. +Moving the median outside the range of the honest submissions requires controlling at least half of a pair's submissions, and even a controlling majority moves the published price at most 25% per round (10% for XCHAIN/USD). A follower that priced the pair itself co-signs the leader's value only when it lies within 5% of the follower's own aggregate over the other validators' submissions; when that set has an even count and the leader itself submitted the pair, within 5% of either of its two middle values, since the leader's median of the full set is one of them or a value between them. ## Cross-Chain Attestation @@ -497,15 +499,16 @@ On each finalized oracle round, `RewardTracker` splits `ORACLE_REWARD_PER_ROUND` ### Slash Detection -Three offense types are monitored: +Four offense types are monitored: | Offense | Trigger | Description | |---|---|---| | `price_deviation` | >5% from consensus | Submission deviates more than `SLASH_DEVIATION_THRESHOLD` from the finalized price | | `repeated_deviation` | 3+ in 24 hours | Three or more deviations within a rolling 24-hour window | | `non_participation` | 30+ missed rounds | `SLASH_MISSED_ROUNDS_THRESHOLD` consecutive rounds without a submission | +| `attestation_divergence` | Diverged body on a `byte_equality` provider | A validator's `ok` PROPOSE body hash differs from the quorum-agreed winner; only `byte_equality` providers are checked, and a `provider_error` report never counts | -Detection is recorded in the `slash_proposals` table for governance review. All three offenses are hub-local: the strongest outcome of a governance vote is `validators.status='suspended'`, which excludes the validator from PBFT rounds and leaves on-chain stake untouched. Stake is burned only when the indexer processes a permissionless SLASH proof of equivocation, which no offense in this table produces. See [Decentralization](decentralization.md) for the three penalty lanes. +Detection is recorded in the `slash_proposals` table for governance review. All four offenses are hub-local: the strongest outcome of a governance vote is `validators.status='suspended'`, which excludes the validator from PBFT rounds and leaves on-chain stake untouched. Stake is burned only when the indexer processes a permissionless SLASH proof of equivocation, which no offense in this table produces. See [Decentralization](decentralization.md) for the three penalty lanes. --- diff --git a/components/hub/configuration.md b/components/hub/configuration.md index c795341b..e713673b 100644 --- a/components/hub/configuration.md +++ b/components/hub/configuration.md @@ -373,6 +373,7 @@ Controls `StateAnchorPublisher` (commits checkpoints and the cross-chain match a | `ANCHOR_RATELIMIT_MAX_WAITS` | No | `3` | Caps how many rate-limit waits one broadcast may take before the anchor defers to a later flush instead of stalling the current one. | | `ANCHOR_AMBIGUOUS_POLL_ATTEMPTS` | No | `3` | Re-polls before an ambiguous publish result (broadcast may or may not have landed) is resolved. | | `ANCHOR_AMBIGUOUS_POLL_MS` | No | `5000` | Delay between those re-polls. | +| `ANCHOR_DEADLETTER_PATH` | No | `./data/anchor-publish.deadletter.jsonl` | Append-only JSONL record of an ANCHOR send that failed after the signer hook committed funds, holding the signed reveal an operator rebroadcasts instead of paying for the payload again; an unset or empty value uses the default. | | `ANCHOR_ANNOUNCE_RETRY_MS` | No | `300000` | Delay between retries of the anchor announcement (5 minutes). | | `ANCHOR_ANNOUNCE_RETRY_TTL_MS` | No | `21600000` | How long announcement retries continue before the entry is dropped (6 hours, roughly six times the 60-confirmation DOGE window). | | `ANCHOR_ANNOUNCE_QUEUE_MAX` | No | `500` | Maximum queued anchor announcements, bounding memory. | @@ -533,7 +534,7 @@ Controls `FullNodeChallengeRound`, the periodic possession challenge proving a v | `FULLNODE_PROOF_WINDOW_BLOCKS` | No | per-coin | Blocks a challenged node has to submit a proof. | | `FULLNODE_REWARD_PASS_WINDOW_BLOCKS` | No | per-coin | Blocks in the reward-pass window. | | `FULLNODE_MIN_PASS_RATE_BPS` | No | per-coin | Minimum pass rate, in basis points, for a node to be treated as passing. | -| `FULLNODE_REWARD_SHARE` | No | per-coin | Reward share for passing full nodes. | +| `FULLNODE_REWARD_SHARE` | No | per-coin | Full-node fraction of the designed oracle-round reward split; a consensus parameter that must match the coin default. Nothing pays the full-node tranche today (see [COLLECT](../../protocol/actions/collect.md#reward-population-path)). | | `FULLNODE_GENESIS_VERIFIERS` | No | per-coin | Comma-separated genesis verifier pubkeys (lowercased). | | `XCHAIN_HUB_SKIP_FULLNODE_ASSERT` | No | _(unset, assertion active)_ | Set to `1` to skip the canonical-FULLNODE assertion at startup and warn instead. Intended as a loud one-off bypass for a venue running its own challenge cadence and verifier set: divergent `NODEPROOF` knobs fork the challenge schedule, so this is not a setting to leave on. | @@ -676,6 +677,9 @@ Read-only operator tools; neither broadcasts nor writes anything and neither is | `HUB_RPC_URLS` | No | _(empty; `--hubs` required instead)_ | Comma-separated hub JSON-RPC URLs `bin/oracle-round-presence.js` polls (`getoracleroundpresence`) when no `--hubs` flag is given. Asks every named hub about the same round range and reports whether the federation agrees on which rounds happened, so a round that finalized on some validators and not others shows up as a named divergence instead of looking like ordinary absence. At least two URLs are required; comparing one hub to itself is refused. | | `XCHAIN_HUB_DIR` | No | `../xchain-hub` | Sibling-checkout override `bin/lib/carrier_logic_pin.js`'s `siblingDir()` resolves for cross-repo carrier-logic comparison; the `repo_guards` twin test points it at a second checkout with `XCHAIN_REQUIRE_SIBLINGS=1`. Never read by the running hub process. | | `XCHAIN_INDEXER_DIR` | No | `../xchain-indexer` | Sibling-checkout override for the `xchain-indexer` tree the same `siblingDir()` resolves when the pin tool compares against the indexer's canonical copy; read by literal name so the coverage gate can see it. Never read by the running hub process. | +| `NODE_OPTIONS` | No | _(unset)_ | Node's own option string. `bin/suite-title-map.js` appends `--require=` to the caller's value while it collects suite titles, so every mocha child it spawns loads the sibling mask, and restores the caller's value afterwards. Never read by the running hub process. | +| `XCHAIN_SUITE_TITLE_REPO_ROOT` | No | _(unset)_ | Set only by `bin/suite-title-map.js` for the mocha children it spawns: the checkout being collected. When present, the child hides every adjacent `xchain-*` checkout from file and module reads, so a pin collected beside a platform checkout matches one collected in standalone CI. Not an operator setting; never read by the running hub process. | +| `XCHAIN_DOCS_DIR` | No | `../xchain-documentation` | Sibling-checkout override for the documentation tree the hub's conformance tests read their vectors from. `bin/suite-title-map.js` unsets it, together with `XCHAIN_INDEXER_DIR`, for the mocha children it spawns and restores both afterwards, so an override cannot point a child back at a sibling the mask hides. Never read by the running hub process. | | `XCHAIN_SYNC_DIR` | No | `../xchain-sync` | Sibling-checkout override for the `xchain-sync` tree the same `siblingDir()` resolves when the pin tool compares against sync's copy; read by literal name so the coverage gate can see it. Never read by the running hub process. | ## Database Schema diff --git a/components/hub/database.md b/components/hub/database.md index 2b76da1b..692ae543 100644 --- a/components/hub/database.md +++ b/components/hub/database.md @@ -210,7 +210,7 @@ The durable at-most-once marker for PRICE v0 round broadcasts, written by `Oracl | `bridge_transfers` | PBFT-finalized bridge transfer records: one signed row per confirmed lock or burn, carrying `tick`, `decimals` and `amount`, from which the destination indexer injects the XBRIDGE settle leg (mirror to indexers) | | `policy_snapshots` | PBFT-finalized per-token policy snapshots (allow list, block list, tick sleep) a destination chain materializes onto a bridged copy. Append-only, latest-wins by `policy_seq`, on the `state_checkpoints` terms (mirror to indexers) | | `list_snapshots` | PBFT-finalized shared-list snapshot records: one signed row per finalized list version, carrying the three per-chain admission columns (mirror to indexers once the shared-list consumer gate arms) | -| `remote_token_snapshots` | PBFT-finalized remote token facts (tick, decimals, owner) pinned at a `snapshot_block`, one signed row per finalized fact; mirrored to indexers, which read the pinned remote token for cross-chain orders | +| `remote_token_snapshots` | PBFT-finalized remote token facts (tick, decimals, owner) pinned at a `snapshot_block`, one signed row per finalized fact; mirrored to indexers, which read the pinned remote token for cross-chain orders. From `remote_token_admission_activation` (keyed on the row's BTC `snapshot_block`) a row also carries a signed `admit_block_btc` / `admit_block_ltc` / `admit_block_doge` admission map stamped from the hub's admission tips | ### `attestations` @@ -480,14 +480,14 @@ Tracks XCHAIN rewards earned by validators for participating in oracle rounds. R ### `slash_proposals` -Records detected validator misbehavior for governance review. The offenses recorded here (`price_deviation`, `repeated_deviation`, `non_participation`, attestation divergence) are hub-local: a governance vote can suspend the validator, and on-chain stake is untouched. Stake is burned only when the indexer processes a permissionless SLASH proof of equivocation, which is a separate on-chain path this table does not feed. +Records detected validator misbehavior for governance review. The offenses recorded here (`price_deviation`, `repeated_deviation`, `non_participation`, `attestation_divergence`) are hub-local: a governance vote can suspend the validator, and on-chain stake is untouched. Stake is burned only when the indexer processes a permissionless SLASH proof of equivocation, which is a separate on-chain path this table does not feed. | Column | Type | Description | |---|---|---| | `id` | `BIGINT AUTO_INCREMENT` | Primary key | | `validator_pubkey` | `CHAR(64) NOT NULL` | Offending validator's pubkey | -| `offense_type` | `VARCHAR(30) NOT NULL` | Type: `price_deviation`, `repeated_deviation`, `non_participation` | -| `round_number` | `BIGINT` | Round where offense occurred | +| `offense_type` | `VARCHAR(30) NOT NULL` | Type: `price_deviation`, `repeated_deviation`, `non_participation`, `attestation_divergence` | +| `round_number` | `BIGINT` | Round where offense occurred (for `attestation_divergence`, a pseudo-round taken from the request id) | | `evidence` | `TEXT` | Serialized evidence details | | `status` | `ENUM('pending','approved','rejected','expired')` | Proposal status (default: `pending`) | | `created_at` | `DATETIME` | Detection time | diff --git a/components/hub/decentralization.md b/components/hub/decentralization.md index 092dd68b..71658a58 100644 --- a/components/hub/decentralization.md +++ b/components/hub/decentralization.md @@ -31,7 +31,7 @@ The validator network uses a **capability model** rather than fixed tiers. A val | `cross_chain` | Attest to cross-chain swap actions per chain-pair | CrossChainEngine | | `oracle_publish` | Broadcast finalized PRICE v0 rounds on-chain (typically DOGE for low fees) | PRICE v0 publisher, formerly "Tier 3" | | `attestation` | Fetch external attestation requests via registered providers (`http_get`, `llm`) and PBFT-finalize the response | AttestationRound, AttestationConsensus | -| `full_node` | Claim the full-node reward tier by passing the chain-derived possession challenges run by `FullNodeChallengeRound` (answer: the `scriptPubKey` of a seed-selected output in a buried block) | FullNodeChallengeRound | +| `full_node` | Pass the chain-derived possession challenges run by `FullNodeChallengeRound` (answer: the `scriptPubKey` of a seed-selected output in a buried block); passing qualifies a validator for the full-node reward tier, which ships inert and is not minted today | FullNodeChallengeRound | A single pubkey can hold any combination of capabilities, there is no overlap restriction. `min_stake[capability]` is governance-configurable; defaults are set by hub config. The historical model where the cross-chain tier required 5× the oracle stake is preserved by appropriate `min_stake` defaults but the protocol does not enforce a hierarchy. @@ -41,7 +41,7 @@ Validators with the `price` capability independently fetch cryptocurrency prices ### `cross_chain`: Cross-Chain Validators -Validators with the `cross_chain` capability attest to cross-chain swap actions. Rather than running full decoder and indexer stacks for every chain, they use **xchain-sync** to replicate indexer + decoder databases, keeping them lightweight. Consensus is calculated per chain-pair; only validators supporting both chains in a swap participate in attestation, using a PBFT-derived consensus requiring quorum agreement over that pair's validator set (stake-weighted and source-deduped at/above `STAKE_WEIGHTED_QUORUM_ACTIVATION`, otherwise the majority-floored count; see [Quorum](#quorum) below). +Validators with the `cross_chain` capability attest to cross-chain swap actions. Matching reads each chain's indexer, so a validator need not run full decoder and indexer stacks for every chain and can use **xchain-sync** to replicate the indexer + decoder databases. It still needs a coin node: the hub's `cross_chain` self-test requires an `rpc` entry, the chain's coin-node RPC, for every chain configured under `cross_chain.chains` (for BTC, `cross_chain.chains.BTC.rpc`; see [Capability Configuration](configuration.md#capability-configuration)). A validator running only a sync replica lists `cross_chain` under `DISABLED_CAPABILITIES` (see [Running a validator](../../getting-started/running-a-validator.md)). Consensus is calculated per chain-pair; only validators supporting both chains in a swap participate in attestation, using a PBFT-derived consensus requiring quorum agreement over that pair's validator set (stake-weighted and source-deduped at/above `STAKE_WEIGHTED_QUORUM_ACTIVATION`, otherwise the majority-floored count; see [Quorum](#quorum) below). ### `oracle_publish`: PRICE v0 Broadcasters @@ -68,7 +68,7 @@ All staking operations (STAKE, UNSTAKE, DELEGATE, COLLECT) are standard XChain a | Deactivation delay | 6 BTC blocks on UNSTAKE (same reorg safety) | | Cooldown | 1000 BTC blocks before staked XCHAIN returns to source (configurable via `STAKING.COOLDOWN_BLOCKS`) | | Delegation | DELEGATE manages signing keys, never stake: v0 adds a signing key that may sign for the broadcaster's own capability stake and v2 revokes one (contract stakes use v1 / v3). A delegated key is backed by the delegating address's active stake; staked amounts never move or change, and there is no third-party stake delegation. See [DELEGATE](../../protocol/actions/delegate.md). | -| Rewards | Not weighted by stake: stake decides which capabilities a pubkey qualifies for, not the size of its share. The per-round oracle budget is split equally across the qualified signers of each finalized round (see [COLLECT reward sources](../../protocol/actions/collect.md#reward-sources)). A pubkey holding both `price` and `oracle_publish` earns both per-round (consensus + broadcast). | +| Rewards | Not weighted by stake: stake decides which capabilities a pubkey qualifies for, not the size of its share. The per-round oracle budget is designed to split equally across the qualified signers of each finalized round, but no consensus rail mints an oracle round reward today, so none is claimable with `COLLECT`; the hub keeps a local per-round ledger for visibility only (see [COLLECT reward sources](../../protocol/actions/collect.md#reward-sources) and [Reward Population Path](../../protocol/actions/collect.md#reward-population-path)). A pubkey may hold both `price` and `oracle_publish`; the claimable rewards tied to `oracle_publish` are the ANCHOR publisher rewards (`anchor_bundle`). | > **Note:** Capability staking above is BTC-only, so the activation/deactivation delays are stated in BTC blocks (~1 hour). Contract-targeted staking (STAKE v3 / UNSTAKE v1 / DELEGATE v1/v3) runs on every chain and calibrates this delay per chain for equivalent ~60-min reorg protection (**6 blocks on BTC, 24 on LTC, 60 on DOGE**) since a flat 6 blocks would give DOGE only ~6 minutes. See `protocol/Contract_Staking.md`. @@ -81,7 +81,7 @@ All staking operations (STAKE, UNSTAKE, DELEGATE, COLLECT) are standard XChain a | **Price data** | `price`-capable validators with trimmed median consensus; published on-chain by `oracle_publish`-capable validators | | **Cross-chain coordination** | `cross_chain`-capable validator attestation with per-chain-pair PBFT | | **External attestation** | `attestation`-capable validators fetch from registered providers (`http_get`, `llm`) and PBFT-finalize; result submitted on-chain as `ATTEST` v1 (response) | -| **Full-node verification** | `full_node`-capable validators answer chain-derived possession challenges (the `scriptPubKey` of a seed-selected output in a buried block) run by `FullNodeChallengeRound`; passing validators earn the full-node reward tranche (NODEPROOF) | +| **Full-node verification** | `full_node`-capable validators answer chain-derived possession challenges (the `scriptPubKey` of a seed-selected output in a buried block) run by `FullNodeChallengeRound`; passing validators get a NODEPROOF verdict, and the full-node reward tranche it qualifies them for ships inert and is not minted today (see [Rewards, and what is not live yet](../../getting-started/running-a-validator.md#rewards-and-what-is-not-live-yet)) | | **Governance** | Off-chain PBFT voting with 7-day period, 2/3+ approval | ### Transport auth follows on-chain key rotation @@ -100,7 +100,7 @@ A validator's P2P signing key is authorized by the **union** of the hub's local | 5 | **Open validator set + governance**; Off-chain PBFT voting for parameter changes, version signaling in heartbeats | v2.0.0 | Complete | | 6 | **Capability model + external attestation framework**; Replace Tier 1/2 with four independent capabilities (`price`, `cross_chain`, `oracle_publish`, `attestation`) auto-qualified by stake amount; block-boundary federation snapshots; external attestation framework (`http_get` byte_equality + `llm` judge_model providers); STAKE rewritten as `VERSION|AMOUNT|SIGNING_PUBKEY` with v1=new / v2=top-up; UNSTAKE rewritten as pubkey-based | 2026-05 | Complete | | 7 | **Federation key rotation + quorum hardening**: addr-keyed `rotatevalidator`/`deregistervalidator` RPC; stake-key revocation via `DELEGATE` v2; Option A union-membership transport auth (on-chain effective signer set + local registry, never fail-open); anchor reward archive (shipped as the `anchor_` reward_type, since superseded by `anchor_bundle`, see [COLLECT](../../protocol/actions/collect.md)) + hub-local oracle/attest_fee reward separation; simple-majority quorum floor (`max(2f+1, ceil((N+1)/2))`) across all consensus engines | 2026-06 | Complete | -| 8 | **Full-node verified tier**: `full_node` capability and `FullNodeChallengeRound`; derived possession challenge (no broadcast of the question); answer = scriptPubKey of a seed-selected UTXO in a buried block, provably absent from a sync mirror; verdict collected as NODEPROOF; full-node reward tranche paid via indexer price.js; `failed_full_node_challenge` slash detection (later removed; tier is reward-only, no slashing) | 2026-06 (bcbe4b8) | Complete | +| 8 | **Full-node verified tier**: `full_node` capability and `FullNodeChallengeRound`; derived possession challenge (no broadcast of the question); answer = scriptPubKey of a seed-selected UTXO in a buried block, provably absent from a sync mirror; verdict collected as NODEPROOF; full-node reward tranche (`oracle_full_node`) defined but shipped inert (`FULLNODE.REWARD_SHARE` 0, no genesis verifiers), so no consensus rail mints it (see [COLLECT](../../protocol/actions/collect.md#reward-population-path)); `failed_full_node_challenge` slash detection (later removed; tier is reward-only, no slashing) | 2026-06 (bcbe4b8) | Complete | ## Architecture Summary diff --git a/components/hub/operations.md b/components/hub/operations.md index e48e31d5..62d41668 100644 --- a/components/hub/operations.md +++ b/components/hub/operations.md @@ -364,13 +364,48 @@ These series are served on the metrics endpoint only when [`METRICS_ENABLED`](./ | `xchain_oracle_current_round` | Gauge | None | Oracle round number the round timer last opened; flat means the round loop itself is wedged. | | `xchain_oracle_consecutive_skipped_rounds` | Gauge | None | Trailing streak of oracle rounds that did not finalize. | | `xchain_oracle_round_timeouts_total` | Counter | None | Oracle rounds evicted before reaching commit quorum, leader and follower seats alike. | +| `xchain_oracle_abandoned_rounds_total` | Counter | None | Oracle rounds this hub watched open and then recorded as abandoned, counted apart from other finalize-phase losses. | | `xchain_oracle_single_source_rounds_total` | Counter | None | Oracle rounds finalized with one uncorrelated price source on a normally-multi-source pair. | | `xchain_oracle_price_source_live` | Gauge | `source` | `1` when that upstream returned at least one usable price on this hub's last fetch and `0` when it did not; absent before the first fetch. | | `xchain_oracle_price_source_fetch_attempts_total` | Counter | `source` | Cumulative fetch dispatches per upstream this hub attempted this round, live or dead; the dispatch evidence behind the liveness gauge. | | `xchain_oracle_price_source_bound_rejects_total` | Counter | `source` | Upstream values dropped on the ingest bound. | +| `xchain_oracle_publisher_published_total` | Counter | None | Oracle price publishes this hub broadcast. | +| `xchain_oracle_publisher_abandoned_total` | Counter | None | Oracle rounds this hub gave up publishing and moved to the dead-letter file. | +| `xchain_oracle_publisher_takeover_attempts_total` | Counter | None | Windows this hub re-assembled to cover a publish leader that went silent. | +| `xchain_oracle_publisher_takeover_published_total` | Counter | None | Takeover windows this hub went on to publish; flat while attempts climb means takeovers are not reaching chain. | +| `xchain_oracle_publisher_takeover_declined_unproven_total` | Counter | None | Takeovers this hub declined because it has never observed an on-chain PRICE batch (a deaf feed, not a healthy leader). | +| `xchain_oracle_publisher_takeover_declined_read_error_total` | Counter | None | Takeovers this hub declined because its own DB could not say whether the window was already on chain. | +| `xchain_oracle_publisher_takeover_deferred_total` | Counter | None | Takeover attempts deferred because the leader may already have a batch in flight. | +| `xchain_oracle_publisher_takeover_unlanded_windows` | Gauge | None | Takeover windows past the election that have not yet reached the send queue. | +| `xchain_oracle_publisher_takeover_armed` | Gauge | None | 1 when a failover window is configured and this hub has proven it would see a leader batch land; 0 means takeover cannot fire. | +| `xchain_oracle_publisher_batch_sign_timeouts_total` | Counter | None | PRICE batch signing rounds that timed out waiting for co-signatures. | +| `xchain_oracle_publisher_last_published_round` | Gauge | None | Last oracle round this hub published; absent until it has published one. | +| `xchain_oracle_publisher_unconfirmed_publishes` | Gauge | None | Publishes this hub broadcast that have not yet reached a block. | Single-source warning lines now name the live and dead upstreams (for example, `live: coingecko; dead: kraken, coinbase`) and the pairs that went single-source with their submitters, so an operator can tell which source dropped. +## Attestation and Checkpoint Metrics + +These series are served under the same [`METRICS_ENABLED`](./configuration.md#metrics-and-log-shipping) switch. Each engine's series stay absent on a hub that does not run that engine. + +| Series | Type | Labels | What it tells you | +|---|---|---|---| +| `xchain_attestation_consensus_timeouts_total` | Counter | None | Attestation rounds torn down by the consensus timeout before reaching quorum. | +| `xchain_attestation_poll_errors_total` | Counter | `kind` | Attestation request polls that failed, by cause: `rpc` rejection, `transport` failure or `auth` refusal. | +| `xchain_attestation_last_successful_poll_timestamp_seconds` | Gauge | None | Unix time of the last usable attestation request poll; absent until one succeeds. | +| `xchain_attestation_poll_unsuccessful_seconds` | Gauge | None | Seconds since the last usable attestation poll, or since the first attempt when none has succeeded. | +| `xchain_attestation_failed_rounds` | Gauge | None | Attestation rounds this hub failed to fetch locally that are still inside the round retention window. | +| `xchain_checkpoint_round_timeouts_total` | Counter | None | Checkpoint rounds this hub led that timed out below quorum. | +| `xchain_checkpoint_sub_quorum_finalized_total` | Counter | None | FINALIZED checkpoint broadcasts this hub dropped as a follower for carrying too few signatures. | +| `xchain_checkpoint_malformed_finalized_total` | Counter | None | FINALIZED checkpoint broadcasts this hub dropped as malformed. | +| `xchain_checkpoint_seq_conflicts_total` | Counter | None | Quorum-signed checkpoints refused because a different payload already holds that sequence; any rise is an equivocation report. | +| `xchain_checkpoint_seq_double_sign_refusals_total` | Counter | None | Signature requests refused because this hub already signed a different payload at that sequence. | +| `xchain_checkpoint_cadence_stalls_total` | Counter | None | Checkpoint cadence ticks that were due but this hub could not lead. | +| `xchain_checkpoint_cadence_stalled` | Gauge | None | `1` while the checkpoint cadence is stalled with a named reason, `0` once a round clears it. | +| `xchain_checkpoint_observer_idle` | Gauge | None | `1` when this hub sits outside the checkpoint signer set and deliberately opens no round. | +| `xchain_checkpoint_frozen_tip_ticks` | Gauge | None | Consecutive not-my-slot checkpoint ticks that saw the same BTC block. | +| `xchain_checkpoint_frozen_tip_stall_ticks` | Gauge | None | The tick count at which a frozen tip is metered as a cadence stall; alert when `xchain_checkpoint_frozen_tip_ticks` reaches it. | + ## Resilience and Recovery ### Database Connection Recovery diff --git a/components/indexer/README.md b/components/indexer/README.md index 000c3558..11087a3d 100644 --- a/components/indexer/README.md +++ b/components/indexer/README.md @@ -7,11 +7,11 @@ xchain-indexer is the state-processing engine of the XChain Platform. It reads decoded blockchain transactions from the Decoder database, validates and executes each ACTION according to protocol rules, maintains authoritative token state (balances, supplies, ownership, DEX orders, dispensers) in a separate MariaDB database, and makes that indexed data available for querying by xchain-explorer. The indexer runs as a long-lived Node.js process with an embedded Express JSON-RPC API server. -Every XChain ACTION; SEND, ISSUE, MINT, ORDER, DISPENSER, SWAP, ANCHOR, XCALL, SLASH, BET, and 38 more, passes through the indexer. The indexer determines whether the action is valid (checking balances, permissions, token rules, allow/block lists, sleep states), records the action with its status, updates the ledger (credits, debits, escrows), recalculates balances, and runs a sanity check after every block to guarantee consistency between token supplies and ledger totals. +Every XChain ACTION; SEND, ISSUE, MINT, ORDER, DISPENSER, SWAP, ANCHOR, XCALL, SLASH, BET, and 28 more (38 named ACTIONs in all), passes through the indexer. The indexer determines whether the action is valid (checking balances, permissions, token rules, allow/block lists, sleep states), records the action with its status, updates the ledger (credits, debits, escrows), recalculates balances, and runs a sanity check after every block to guarantee consistency between token supplies and ledger totals. ## Features -- **48 record types processed by the indexer** (of which 35 are wire-decoded ACTIONs; the rest are derived/system rows such as ORDER_MATCH, DISPENSE, COLLECT, `*_EXPIRE`, UNKNOWN, XEXEC, and XCALL - see `concepts/ACTIONS.md` for the canonical wire-decoded count): ADDRESS, AIRDROP, ANCHOR, ATTEST, BATCH, BET, BET_EXPIRE, BROADCAST, CALLBACK, COINPAY, COINPAY_EXPIRE, COLLECT, CROSS_SETTLE, DELEGATE, DEPLOY, DEPOSIT, DESTROY, DISPENSE, DISPENSER, DISPENSER_CLOSE, DISPENSER_EXPIRE, DIVIDEND, EXECUTE, FILE, ISSUE, LINK, LIST, MESSAGE, MINT, NODEPROOF, ORDER, ORDER_EXPIRE, ORDER_MATCH, PRICE, SEND, SLASH, SLEEP, STAKE, SWAP, SWAP_EXPIRE, SWAP_MATCH, SWEEP, UNKNOWN, UNSTAKE, VOTE, WITHDRAW, XCALL, XEXEC +- **50 record types processed by the indexer** (37 wire-decoded ACTIONs, plus XCALL, which is mirror-injected rather than decoded, and 12 derived/system rows such as ORDER_MATCH, DISPENSE, CROSS_SETTLE, `*_EXPIRE`, XEXEC and UNKNOWN; see [concepts/actions.md](../../concepts/actions.md) for the canonical counts): ADDRESS, AIRDROP, ANCHOR, ATTEST, BATCH, BET, BET_EXPIRE, BROADCAST, CALLBACK, COINPAY, COINPAY_EXPIRE, COLLECT, CROSS_SETTLE, DELEGATE, DEPLOY, DEPOSIT, DESTROY, DISPENSE, DISPENSER, DISPENSER_CLOSE, DISPENSER_EXPIRE, DIVIDEND, EXECUTE, FILE, ISSUE, LINK, LIST, MESSAGE, MINT, NODEPROOF, ORDER, ORDER_EXPIRE, ORDER_MATCH, PRICE, ROLLCALL, SEND, SLASH, SLEEP, STAKE, SWAP, SWAP_EXPIRE, SWAP_MATCH, SWEEP, UNKNOWN, UNSTAKE, VOTE, WITHDRAW, XBRIDGE, XCALL, XEXEC - **Multi-chain support**: Bitcoin, Litecoin, and Dogecoin today, on mainnet, testnet, and regtest - **Atomic block processing**: every block is wrapped in a database transaction; failures roll back cleanly - **Block reorganization handling**: detects reorgs from the Decoder DB, rolls back affected data, and re-indexes @@ -21,7 +21,7 @@ Every XChain ACTION; SEND, ISSUE, MINT, ORDER, DISPENSER, SWAP, ANCHOR, XCALL, S - **Staking**: capability staking (STAKE v1/v2, UNSTAKE v0, DELEGATE v0/v2, COLLECT) is BTC-only; contract-targeted staking (STAKE v3, UNSTAKE v1, DELEGATE v1/v3) works on any chain - **Virtual Machine**: DEPLOY, EXECUTE, DEPOSIT, WITHDRAW with isolated-vm sandbox and gas metering - **Unified Gas Fee Schedule**: gas-based fee system for VM and staking actions, replaces per-action flat fees post-activation -- **Protocol versioning**: actions are gated by the indexer's own protocol version (21 at `0.1.0`, 16 at `0.2.0`, all with zero activation heights); block-height and timestamp flag-days gate behaviour changes to already-live actions, per network +- **Protocol versioning**: actions are gated by the indexer's own protocol version (21 at `0.1.0`, 17 at `0.2.0`, all with zero activation heights); block-height and timestamp flag-days gate behaviour changes to already-live actions, per network - **Action mapping**: creates address↔ticker↔action_index cross-references for fast lookups - **Circuit-breaker DB connections**: automatic failure detection and recovery for database connectivity - **Watchdog timeout**: configurable per-block processing timeout detects deadlocks @@ -33,7 +33,7 @@ Every XChain ACTION; SEND, ISSUE, MINT, ORDER, DISPENSER, SWAP, ANCHOR, XCALL, S |---|---| | [Architecture](architecture.md) | Data pipeline, internal components, action handlers, block processing pipeline | | [Configuration](configuration.md) | Environment variables, coin-specific config, indexer constants | -| [Actions](actions.md) | All 48 record types, categories, format versions, protocol versioning | +| [Actions](actions.md) | All 50 record types, categories, format versions, protocol versioning | | [Database](database.md) | Full schema reference: core, ledger, action, state, index, and mapping tables | | [Ledger](ledger.md) | Double-entry ledger, balance calculation, sanity checks, gas token fees | | [Operations](operations.md) | Running, Docker, API endpoints, resilience, troubleshooting | @@ -70,7 +70,7 @@ INDEXER_COIN=BTC INDEXER_NETWORK=mainnet ``` -> **Optional: Hub database variables.** If `HUB_DB_HOST` and `HUB_DB_NAME` are not set, the indexer logs a `WARNING: HUB_DB_HOST / HUB_DB_NAME not set` message at startup and reads price/oracle tables from its own local database instead. This is correct for single-host deployments. On a distributed node where the hub runs on a separate host, set these variables (along with `HUB_DB_PORT`, `HUB_DB_USER`, `HUB_DB_PASS`) to avoid using stale or absent fee/price data. See [Configuration](configuration.md) for the full variable list. +> **Hub database variables.** `HUB_DB_HOST` and `HUB_DB_NAME` are required for a BTC testnet or regtest indexer while `ATTEST_RESPONSE_MIRROR_ACTIVATION` is armed. Finalized responses in `attestation_responses` are off-chain consensus input, and AnchorRecovery cannot reconstruct it, so `INDEXER_ALLOW_LOCAL_PRICE_SOURCE` does not bypass this startup check. This requirement currently applies to BTC testnet and regtest. On currently unpinned BTC mainnet, a single-host indexer may still read price/oracle tables from its local database when `INDEXER_ALLOW_LOCAL_PRICE_SOURCE=true`; other testnet/regtest chains retain the local-database fallback. A distributed node should set the hub variables (along with `HUB_DB_PORT`, `HUB_DB_USER`, `HUB_DB_PASS`) in every case. See [Configuration](configuration.md) for the full topology and variable list. Start the indexer: diff --git a/components/indexer/actions.md b/components/indexer/actions.md index 3a5832f0..3a7a01e9 100644 --- a/components/indexer/actions.md +++ b/components/indexer/actions.md @@ -95,6 +95,7 @@ stateDiagram-v2 | **SWAP_EXPIRE** | Automatic: expires swaps past their expiration time | Block time check, escrow release | | **CROSS_SETTLE** | System-injected: settles this chain's leg of a hub-matched cross-chain trade (covers both SWAP and ORDER legs). There is no on-chain transaction; the indexer injects one per unsettled match from the hub-mirrored `cross_chain_matches` table, verifies the `cross_chain` quorum of validator signatures (stake-weighted and source-deduped at/above `STAKE_WEIGHTED_QUORUM_ACTIVATION`, otherwise the legacy `max(2f+1, ceil((N+1)/2))` signer count), and releases the local escrow to the counterparty's payout address. Recorded in `cross_chain_settlements` for idempotency and rollback. Not registered in `protocol_changes.js` (never decoded from a transaction); dispatched from `utility.js processCrossChainSettlements` at each block. | Network scope matches this indexer's network; quorum signatures verify against the locked capability snapshot at `snapshot_block`; local offer must still be open | | [**XCALL**](../../protocol/actions/xcall.md) | Mirror-injected: dispatches a cross-chain contract call from the hub-mirrored capability snapshot onto the target chain, applied in `(snapshot_block, call_id)` order. There is no on-chain transaction on the source chain; not registered in `protocol_changes.js`. | Quorum signatures verify against the locked capability snapshot; per-block dispatch cap enforced (overflow carries forward, never dropped) | +| [**XBRIDGE**](../../protocol/actions/xbridge.md) | Moves a token between chains by lock-and-mint / burn-and-release against a protocol-owned escrow. Users broadcast v0/v3 (lock) and v1/v4 (burn); the indexer system-injects the v2/v5 settle leg from a finalized `bridge_transfers` row, and a broadcast v2/v5 is refused. | Below `XCHAIN_BRIDGE_ACTIVATION` (v0 to v2) or `TOKEN_BRIDGE_ACTIVATION` (v3 to v5) every version is invalid; per-version chain rules (v0 BTC only, v1 non-BTC only), destination address checks, amount within balance, and the source's SLEEP and allow/block lists on locks and burns. See [`protocol/actions/xbridge.md`](../../protocol/actions/xbridge.md). | | [**XEXEC**](../../protocol/actions/xcall.md) | System-injected: executes the mirrored contract call produced by XCALL on the target chain and records the outcome. Recorded in `cross_chain_call_executions` (see [DATABASE.md](database.md)). | Target contract exists and is active; execution outcome recorded for idempotency | ```mermaid @@ -125,6 +126,7 @@ sequenceDiagram | [**BATCH**](../../protocol/actions/batch.md) | Execute multiple actions in a single transaction | Each sub-action validated independently | | [**LINK**](../../protocol/actions/link.md) | Link two action_indexes (e.g., FILE to ISSUE) | Both action_indexes exist, valid link type | | [**LIST**](../../protocol/actions/list.md) | Create or update a list of addresses/items | Valid list format | +| **UNKNOWN** | Fallback handler for any ACTION string that has no handler of its own | Always recorded invalid with the dispatcher's error; records the source address and its action mappings, and changes no balances | ## Staking Actions @@ -135,11 +137,11 @@ Two staking systems share the same four action names. **Capability staking** (ST | **STAKE** | Lock tokens against a signing pubkey. v1 = new capability stake (XCHAIN), v2 = top-up of existing capability stake (XCHAIN), v3 = contract-targeted stake (any token, targets a stakeable contract: see DEPLOY v1) | VERSION valid (1/2/3), AMOUNT positive, SIGNING_PUBKEY is 64-char hex Ed25519. v1/v2: aggregate per-pubkey active stake auto-qualifies the pubkey for each of five capabilities (`price`, `cross_chain`, `oracle_publish`, `attestation`, `full_node`) based on governance `min_stake[capability]`. v3: target contract must be stakeable; row keyed by `(target, pubkey, tick, source)`. | | **UNSTAKE** | Release staked tokens. v0 = capability unstake keyed by pubkey. v1 = contract-targeted unstake keyed by `(target, pubkey, tick)`. Both take an optional trailing `AMOUNT` (gated by `PARTIAL_UNSTAKE_COLLECT`): absent means a full sweep, present moves only `AMOUNT` into cooldown and the residual stays staked ([Partial Unstake](../../protocol/actions/unstake.md#partial-unstake-optional-amount)). | Pubkey has active stake of matching type; sets `deactivation_block`. v1 cooldown is per-contract (set at DEPLOY v1 time); v0 uses the global `STAKING.COOLDOWN_BLOCKS`. | | **DELEGATE** | Manage the signing key for a stake. v0 = capability rotate, v1 = contract rotate, v2 = capability revoke, v3 = contract revoke. | Active stake/delegation of matching type exists. For rotates, new pubkey valid and unused. Takes effect after the activation delay: 6 blocks for capability rotate/revoke (v0/v2); the per-chain delay (6 blocks on BTC, 24 on LTC, 60 on DOGE) for contract rotate/revoke (v1/v3). | -| **COLLECT** | Collect accumulated rewards | Address has unclaimed rewards > 0. `attest_fee` rewards are derived by the indexer during block processing; the oracle labels (`oracle_round` / `oracle_base` / `oracle_full_node`) are not minted by any consensus rail today (see [COLLECT](../../protocol/actions/collect.md#reward-population-path)); `anchor_bundle` and `anchor_archive` rewards are derived by the indexer from the on-chain ANCHOR bytes at or above their own flag-days (`ANCHOR_REWARD_ACTIVATION` / `ARCHIVE_REWARD_ACTIVATION`); below those heights they were pushed from `xchain-hub` via `pushvalidatorrewards`, a method since removed (a call answers JSON-RPC `-32601`), and those rows are restored by ANCHOR full-parse recovery. | +| **COLLECT** | Collect accumulated rewards | Address has unclaimed rewards > 0. `attest_fee` rewards are derived by the indexer during block processing; `rollcall_publish` rewards are derived at the BTC-side ROLLCALL epoch close and pay the frozen `ROLLCALL_REWARD_AMOUNT` to that epoch's elected leader (see [ROLLCALL](../../protocol/actions/rollcall.md)); the oracle labels (`oracle_round` / `oracle_base` / `oracle_full_node`) are not minted by any consensus rail today (see [COLLECT](../../protocol/actions/collect.md#reward-population-path)); `anchor_bundle` and `anchor_archive` rewards are derived by the indexer from the on-chain ANCHOR bytes at or above their own flag-days (`ANCHOR_REWARD_ACTIVATION` / `ARCHIVE_REWARD_ACTIVATION`); below those heights they were pushed from `xchain-hub` via `pushvalidatorrewards`, a method since removed (a call answers JSON-RPC `-32601`), and those rows are restored by ANCHOR full-parse recovery. | ### `oracle_publish` capability (formerly "Tier 3") -The publisher role for broadcasting finalized PRICE v0 transactions to a chain (DOGE recommended for low fees). Auto-qualifies when a pubkey's aggregate active stake ≥ `min_stake[oracle_publish]`. A pubkey may hold the `oracle_publish` and `price` capabilities simultaneously (and earn both rewards in the same round). +The publisher role for broadcasting finalized PRICE v0 transactions to a chain (DOGE recommended for low fees). Auto-qualifies when a pubkey's aggregate active stake ≥ `min_stake[oracle_publish]`. A pubkey may hold the `oracle_publish` and `price` capabilities simultaneously. No oracle round reward is minted today (see [COLLECT → Reward Population Path](../../protocol/actions/collect.md#reward-population-path)); the claimable reward tied to `oracle_publish` is the ANCHOR publisher reward (`anchor_bundle`). ## Oracle Actions @@ -165,7 +167,7 @@ Virtual Machine actions are available on **all chains** (BTC, LTC, DOGE). DEPLOY | Action | Purpose | Key Validations | |---|---|---| -| [**ANCHOR**](../../protocol/actions/anchor.md) | Commit quorum-signed state checkpoints and a compressed archive of cross-chain match rows on-chain. DOGE-only, validator-broadcast action. | Valid only on the DOGE chain (all networks). No protocol fee. On parse, the indexer writes to `anchor_actions` and records checkpoint hashes. The archived data makes all platform state recoverable from a full chain re-parse. See `src/actions/anchor/index.js` and `protocol/actions/ANCHOR.md`. | +| [**ANCHOR**](../../protocol/actions/anchor.md) | Commit quorum-signed state checkpoints and a compressed archive of cross-chain match rows on-chain. DOGE-only, validator-broadcast action. | Valid only on the DOGE chain (all networks). No protocol fee. On parse, the indexer writes to `anchor_actions` and records checkpoint hashes. The archived data makes all platform state recoverable from a full chain re-parse. See `src/actions/anchor/index.js` and [`protocol/actions/anchor.md`](../../protocol/actions/anchor.md). | ## Attestation & Validator Actions @@ -173,10 +175,11 @@ These are the Attestation and Validator categories named under Protocol Versioni | Action | Purpose | Key Validations | |---|---|---| -| [**ATTEST**](../../protocol/actions/attest.md) | External-data attestation lifecycle in three version-discriminated phases: v0 (VM-emitted request), v1 (validator-broadcast response), v2 (system-synthesized expiry). | Governance-registered `PROVIDER_ID`; v1 responses are quorum-signed by the attestation-capable validator set; deterministic `REQUEST_ID` derivation. See `protocol/actions/ATTEST.md`. | -| [**NODEPROOF**](../../protocol/actions/nodeproof.md) | Records an on-chain, quorum-signed verdict of which validators answered a periodic possession challenge, proving they run a real coin full node rather than mirroring DBs via `xchain-sync`. | `EPOCH_HEIGHT` a multiple of `CHALLENGE_INTERVAL_BLOCKS`; passing pubkeys quorum-signed; challenge derived from the epoch. See `protocol/actions/NODEPROOF.md`. | -| [**SLASH**](../../protocol/actions/slash.md) | Permissionless equivocation proof that burns a capability validator's entire bond when they signed two conflicting values for the same consensus slot. | `CAPABILITY` matches the engine the `EQUIV_KEY` names (derived, not trusted); both signatures verify against `OFFENDER_PUBKEY`; writes a `capability_slash_events` audit row with the bounty/treasury split. See `protocol/actions/SLASH.md`. | -| [**VOTE**](../../protocol/actions/vote.md) | Token-weighted governance polls in four version-discriminated phases: v0 (create), v1 (ballot), v2 (system finalization), v3 (delegate). | Weight measured from on-chain holdings of `TICK` at the poll's effective close (never read from payload), so finalization needs no validator consensus round; `QUORUM` / `MIN_VOTERS` thresholds gate a winner. See `protocol/actions/VOTE.md`. | +| [**ATTEST**](../../protocol/actions/attest.md) | External-data attestation lifecycle in three version-discriminated phases: v0 (VM-emitted request), v1 (validator-broadcast response), v2 (system-synthesized expiry). | Governance-registered `PROVIDER_ID`; v1 responses are quorum-signed by the attestation-capable validator set; deterministic `REQUEST_ID` derivation. See [`protocol/actions/attest.md`](../../protocol/actions/attest.md). | +| [**NODEPROOF**](../../protocol/actions/nodeproof.md) | Records an on-chain, quorum-signed verdict of which validators answered a periodic possession challenge, proving they run a real coin full node rather than mirroring DBs via `xchain-sync`. | `EPOCH_HEIGHT` a multiple of `CHALLENGE_INTERVAL_BLOCKS`; passing pubkeys quorum-signed; challenge derived from the epoch. See [`protocol/actions/nodeproof.md`](../../protocol/actions/nodeproof.md). | +| [**ROLLCALL**](../../protocol/actions/rollcall.md) | Lands one epoch's validator presence proofs on DOGE. The BTC indexer closes each epoch by recording who was absent and evicting a source absent for `ROLLCALL_EVICT_MISSES` consecutive rolled epochs; eviction deactivates stake and refunds it after the cooldown, never burns it. | DOGE only, validator-broadcast, not batchable; `EPOCH_HEIGHT` at or above `ROLLCALL_ACTIVATION` and a multiple of `ROLLCALL_INTERVAL_BLOCKS`; every signature verifies over the canonical bound to the BTC epoch block's `LEDGER_HASH`, with no duplicate pubkey; `SIG_COUNT` equals the pair count. See [`protocol/actions/rollcall.md`](../../protocol/actions/rollcall.md). | +| [**SLASH**](../../protocol/actions/slash.md) | Permissionless equivocation proof that burns a capability validator's entire bond when they signed two conflicting values for the same consensus slot. | `CAPABILITY` matches the engine the `EQUIV_KEY` names (derived, not trusted); both signatures verify against `OFFENDER_PUBKEY`; writes a `capability_slash_events` audit row with the bounty/treasury split. See [`protocol/actions/slash.md`](../../protocol/actions/slash.md). | +| [**VOTE**](../../protocol/actions/vote.md) | Token-weighted governance polls in four version-discriminated phases: v0 (create), v1 (ballot), v2 (system finalization), v3 (delegate). | Weight measured from on-chain holdings of `TICK` at the poll's effective close (never read from payload), so finalization needs no validator consensus round; `QUORUM` / `MIN_VOTERS` thresholds gate a winner. See [`protocol/actions/vote.md`](../../protocol/actions/vote.md). | --- diff --git a/components/indexer/architecture.md b/components/indexer/architecture.md index 7abbcefb..6b321c67 100644 --- a/components/indexer/architecture.md +++ b/components/indexer/architecture.md @@ -40,7 +40,7 @@ After processing each block, the indexer pushes its chain tip to xchain-hub (so flowchart TD API["api.js
Express + JSON-RPC server
Validates env vars, starts indexer"] XI["XChainIndexer
Main orchestrator class
Block polling loop (5s interval)
Reorg detection → block processing → sanity check"] - ACTIONS["Actions
48 handlers + aliases
inc. PRICE"] + ACTIONS["Actions
50 handlers + aliases
inc. PRICE"] DB["Database
3 pool conns
(decoder/idx/hub)"] ROLLBACK["Rollback
Atomic undo by block"] UTIL["Utility
Math, timer
Expirations
Ledger ops"] @@ -133,7 +133,7 @@ The source-layout consolidation replaced the removed top-level `stateHash.js` mo |---|---|---| | `src/api.js` | None | Entry point: Express server + JSON-RPC, env var validation, indexer startup | | `src/XChainIndexer.js` | `XChainIndexer` | Main orchestrator: block polling loop, reorg detection, block processing pipeline | -| `src/actions/index.js` | `Actions` | Loads all 48 action handler classes (one per routable ACTION string, including the `UNKNOWN` fallback), routes transactions to the correct handler. The internal `deploy_chunk` sub-handler is loaded by `deploy/index.js`, not here | +| `src/actions/index.js` | `Actions` | Loads all 50 action handler classes (one per routable ACTION string, including the `UNKNOWN` fallback; built by `wireCoreHandlers` / `wireProtocolHandlers` in `src/actions/actions_class/handler_wiring.js`), routes transactions to the correct handler. The internal `deploy_chunk` sub-handler is loaded by `deploy/index.js`, not here | | `src/db/index.js` | `Database` | MariaDB connection pool management, table creation, sanity checks; the SQL queries live in the per-table modules under `src/db/` | | `src/config.js` | None | Merges environment variables with coin-specific config into a single config object | | `src/coins/BTC.js` | None | Bitcoin-specific: fee schedules, BURN/GAS/DONATE addresses per network | diff --git a/components/indexer/configuration.md b/components/indexer/configuration.md index 92915e1a..ee6092d3 100644 --- a/components/indexer/configuration.md +++ b/components/indexer/configuration.md @@ -31,6 +31,7 @@ Configuration is loaded from a `.env` file and environment variables. Copy the ` | `INDEXER_API_PORT` | API server listening port | `3004` | | `CORS_ORIGIN` | Allowed CORS origin for API requests | `http://localhost` | | `INDEXER_RATE_LIMIT_RPM` | API requests per minute per IP | `600` | +| `BALANCE_DELTA_MODE` | How address balances are maintained after each block. `off` re-sums each touched address's credit and debit history; `verify` re-sums too while comparing the collected per-block delta and logging any difference; `on` applies the collected delta instead of re-summing. A rollback always re-sums. Read case-insensitively; any other value reads as `off`. | `off` | | `HUB_API_URL` | Hub JSON-RPC base URL used by the indexer's hub client. Falls back to the URL passed in code when unset. | _(unset)_ | | `HUB_API_KEY` | API key sent with hub calls. Required whenever the hub runs keyed, which is always in validator mode. Treat as a credential. | _(unset)_ | | `HUB_CONFIG_URL` | Hub API base URL for the config-oracle poll (`getallconfigs`). The hub keeps that method off its public feed port because the answer carries every service's DB credentials, so an indexer pointed at a validator's feed port pushes and mirrors correctly but fails its config poll once a minute forever, silently freezing the hub-supplied params at their startup values. Point this at a private hub API port to separate the two roles. Unset falls back to `HUB_API_URL`, so a single-hub deployment is unchanged. | _(unset)_ | @@ -69,6 +70,9 @@ Configuration is loaded from a `.env` file and environment variables. Copy the ` | `HUB_PRICE_SYNC_TIMEOUT_MS` | Price-sync barrier timeout. Before processing a block the indexer waits for its local price mirror to reach that block height, so native-coin fee validation is deterministic across operators. On timeout the block is deferred and retried rather than validated against a stale price copy. | `60000` | | `INDEXER_HEALTH_STALL_GRACE_MS` | How long with no committed block before the container reports unhealthy (`503`). Defaults to comfortably more than one barrier cycle so a single legitimate defer never flaps the healthcheck. Operational only, **not** a consensus parameter. | `max(2 × HUB_PRICE_SYNC_TIMEOUT_MS, 120000)` | | `INDEXER_POLL_SILENT_MS` | How long the block-poll loop may go without completing an iteration before the health payload reports it silent and the monitor crits. Deliberately **not** part of the `/status` 503 gate: one block can hold a single iteration across several sequential barrier waits, and restarting the container is the wrong answer to a slow block, so the default doubles the stall grace again. Measures loop **liveness**, never chain progress, so unlike the stall signal it has nothing to do with commits. Operational only, **not** a consensus parameter. | `2 × INDEXER_HEALTH_STALL_GRACE_MS` | +| `INDEXER_READY_NO_COMMIT_MS` | How long the indexer may sit behind a known tip without committing a block before `GET /ready`, and the `ready` / `notReadyReasons` fields on `/status`, report it not ready (`behind_tip_no_commit`). Readiness only: the `/status` 503 liveness gate never reads it. A value that is not a positive integer keeps the default. Operational only, **not** a consensus parameter. | `600000` (10 min) | +| `INDEXER_CHAIN_TIP_TTL_MS` | How long the coin node tip read from the utxo tracker is cached for the `chainLag` field on `/status`. Health reporting only. | `15000` | +| `INDEXER_CHAIN_TIP_TIMEOUT_MS` | Timeout for one coin node tip read from the utxo tracker. A failed or slow refresh keeps the last known tip, and `chainTipAt` says how old it is. Health reporting only. | `1500` | | `XCALL_DIRECT_PRESENCE_TIMEOUT_MS` | Call-presence barrier timeout in direct-hub-DB mode. With no HubDbSync mirror the cross-chain-call sync barrier is skipped, but reading the hub's MariaDB directly does not guarantee an in-flight relay row has landed, so the indexer waits this long for it before the cross-chain-call pass. | `10000` | | `CHAIN_TIP_PUSH_MAX_LAG` | Skip pushing the chain tip to the hub while the indexer is more than this many blocks behind the decoder tip. During a bulk re-index, pushing a tip per historical block floods the hub's rate limiter with `429`s for no value: the hub only cares about the live tip. | `100` | | `HUB_SYNC_ANCHOR_ATTEST_GRACE_S` | Grace margin on the anchor-reward attestation mirror barrier. The BTC indexer only derives an anchor/archive reward once the hub mirror is certified to hold everything produced up to the block being processed; a node that cannot certify that defers the block rather than deriving a partial reward set. Honoured on regtest only: elsewhere it is a consensus input and the frozen value wins. **A fast chain must lower this or it cannot make progress.** The barrier waits for the mirror watermark to reach `block_time + this value`, which costs nothing where blocks are minutes apart, because the watermark is long past by the time a block is processed. Where blocks are stamped at about wall clock, and the watermark tracks wall clock too, a freshly mined block can never be this far behind it: the wait cannot be satisfied and every affected block burns the full barrier timeout before proceeding anyway. Measured on a regtest e2e venue at the default: 367 blocks in six hours against 2013 in under two once lowered, with 160 deferrals that were waiting on a condition that could not arrive. Set it to a small fraction of the block interval on any venue that mines quickly. | `120` | @@ -161,6 +165,7 @@ carries the credit. Arming the leg is an edit to that bundle plus a `xchain-hub/bin/sync-coins.sh` re-vendoring wave, never a per-node export. | `CROSS_CHAIN_ROYALTY_REGTEST_TIME` | **Regtest only.** Override the cross-chain royalty activation time so the OFF/deny path stays drillable on a single-node stack. Deliberately regtest-scoped: two nodes with different values would disagree on consensus. | `0` (activate at genesis) | | `CONTROLLER_CUSTODY_GUARD_REGTEST_TIME` | **Regtest only.** Override the controller custody guard activation time so the pre-activation bypass and the guarded path stay drillable on a single-node stack. Deliberately regtest-scoped: two nodes with different values would disagree on consensus. | `0` (activate at genesis) | +| `EXACT_INTEGER_WIRE_FIELDS_REGTEST_TIME` | **Regtest only.** Arm the exact-integer wire fields activation at this time so a regtest venue can drive the strict wire parsing; a set non-negative integer arms it (0 means active from genesis). Deliberately regtest-scoped: testnet and mainnet read their pinned activation times. | unset (unarmed) | ### A7 replay-equivalence harness @@ -299,6 +304,14 @@ Native-coin fee validation and FIAT settlement read the oracle tables (`price_sn - **Single-host:** the indexer's own database already holds the synced hub copy, so no separate hub DB connection is needed and the oracle tables are read locally. +That price-source choice does not cover the finalized ATTEST response mirror. A **BTC** indexer on a +network where `ATTEST_RESPONSE_MIRROR_ACTIVATION` is armed must set `HUB_DB_HOST` and `HUB_DB_NAME`. +The `attestation_responses` table is an off-chain consensus input that settles request fees, and +AnchorRecovery cannot reconstruct it. The indexer therefore refuses to boot without a hub database +source on BTC testnet and regtest, where the gate is currently armed. The gate remains unpinned on +BTC mainnet. `INDEXER_ALLOW_LOCAL_PRICE_SOURCE=true` acknowledges only the local oracle-price +topology and cannot bypass this response-mirror requirement. + The two are indistinguishable from config alone: a node with no hub DB looks the same whether that is intentional (single-host) or an operator forgot `HUB_DB_HOST` / `HUB_DB_NAME` on a distributed node. In the latter case the indexer would silently value native-coin fees against stale or empty local price @@ -306,12 +319,12 @@ data, which on mainnet can diverge from the canonical fleet and fork the ledger. To make the intent explicit, a **mainnet** indexer fails closed at startup when no hub DB is configured, unless `INDEXER_ALLOW_LOCAL_PRICE_SOURCE=true` is set to confirm an intentional single-host node. On -testnet and regtest, single-host is the norm and there is no canonical fleet to diverge from, so the -missing hub DB only logs a warning. +testnet and regtest the local-price acknowledgement is ignored. A missing hub DB only logs a warning +for a chain that is not subject to the armed BTC response-mirror refusal above. | Variable | Description | Default | |---|---|---| -| `INDEXER_ALLOW_LOCAL_PRICE_SOURCE` | Acknowledge an intentional single-host setup (read oracle tables from the local indexer DB). Required to boot a **mainnet** node that has no `HUB_DB_HOST` / `HUB_DB_NAME`; ignored on testnet/regtest. | unset (mainnet fails closed) | +| `INDEXER_ALLOW_LOCAL_PRICE_SOURCE` | Acknowledge an intentional single-host setup (read oracle tables from the local indexer DB). Required to boot a **mainnet** node that has no `HUB_DB_HOST` / `HUB_DB_NAME`; ignored on testnet/regtest. This does not bypass the armed BTC response-mirror hub database requirement above. | unset (mainnet fails closed) | ## Coin-Specific Configuration diff --git a/components/indexer/data-retention.md b/components/indexer/data-retention.md index 5f5df3d3..8da25242 100644 --- a/components/indexer/data-retention.md +++ b/components/indexer/data-retention.md @@ -95,7 +95,14 @@ reference. | `STATE_ROOT_RETENTION_BLOCKS` | unset | A positive integer turns retention on and sets the phase-1 window. Unset or `0` means off (keep everything, the historical default behavior). | | `STATE_NODE_RECLAIM` | off | `1` or `true` additionally enables phase-2 orphan-node reclaim. Ignored unless retention is on. | | `STATE_RETENTION_INTERVAL_MS` | `21600000` (6h) | Sweep cadence. | -| `STATE_TREE_METRIC_MAX_NODES` | `2000000` | Shared with the orphan-growth metric: above this node count, the in-memory mark (and thus phase-2 reclaim) is skipped to bound memory. Phase-1 root prune still runs. | +| `STATE_TREE_METRIC_MAX_NODES` | `2000000` | Shared with the orphan-growth metric: above this node count, the in-memory mark (and thus phase-2 reclaim) is skipped to bound memory. Phase-1 root prune still runs. The table only grows, so once a chain passes the cap, reclaim stays off until the cap is raised. | + +With `STATE_NODE_RECLAIM` on, the indexer logs a warning the first time a sweep +skips reclaim at the cap, naming the node count and the cap, and one info line +when reclaim runs again. Every such sweep also emits a +`state_retention_node_reclaim` `[METRIC]` line carrying `skipped`, `reason`, +`total_nodes`, `reachable_nodes`, `max_nodes`, `deleted`, `sweep_duration_ms` +and `consecutive_skips`. Operational guidance: a retention window must be wider than the deepest reorg a chain will ever serve, and wide enough for the SPV proof horizon the diff --git a/components/indexer/database.md b/components/indexer/database.md index 31893bdd..73ef0b7d 100644 --- a/components/indexer/database.md +++ b/components/indexer/database.md @@ -104,7 +104,7 @@ The indexer creates and manages all tables in this database. SQL schema files li | `cross_chain_call_callbacks` | Records the system-injected callback EXECUTE delivered to the source contract after a cross-chain call result is processed. One row per `call_id` (idempotency) | | `bridge_transfers` | Hub-mirrored bridge transfer rows. Populated by `hub_db_sync`; carries the source leg (chain, action_index, address), the destination (chain, address), the signed `tick`, `decimals` and `amount`, and the `validator_signatures` the `cross_chain` quorum signed. The XBRIDGE pass injects the settle leg from a finalized row | | `bridge_settlements` | Settlement records for applied bridge legs on this chain. One row per `(transfer_id, kind)`, where `kind` separates a transfer settle from an applied policy snapshot; used for idempotency so a leg is never applied twice. Rolled back by `action_index` | -| `remote_token_snapshots` | Hub-mirrored federation-finalized token facts from remote chains. Versions are keyed by signed `snapshot_id`; consumers select the newest finalized row by `snapshot_block`, `source_action_index` and `snapshot_id`. Source-chain retractions remove affected versions by `coin` and `source_action_index` | +| `remote_token_snapshots` | Hub-mirrored federation-finalized token facts from remote chains. Versions are keyed by signed `snapshot_id`; consumers select the newest finalized row by `snapshot_block`, `source_action_index` and `snapshot_id`. From `remote_token_admission_activation` a consumer on chain C at block B selects only among rows whose signed `admit_block_` is at or below B, and waits for the hub's `remote_token_snapshots` height watermark to cover B before it reads. Source-chain retractions remove affected versions by `coin` and `source_action_index` | | `list_share_mirrors` | Maps a shared list's home chain and home root index to this chain's bridge-owned mirror list. Rolled back by `action_index` | | `policy_snapshots` | Hub-mirrored per-token policy snapshots (allow list, block list, tick sleep) for a bridged copy. Populated by `hub_db_sync`; append-only, latest-wins by `policy_seq`, on the same terms as `state_checkpoints`. Membership arrays are transport and are verified against the signed `policy_hash` on apply | | `list_snapshots` | Hub-mirrored quorum-signed shared-list versions, append-only by home chain, home list root index and `seq`. Version 1 carries the full membership; later versions carry only the members added and removed. Membership is verified against the signed `members_hash` on apply | @@ -146,7 +146,7 @@ The indexer creates and manages all tables in this database. SQL schema files li | `stakes` | Active and historical capability-staking STAKE records (`version` 1=new / 2=top-up): `signing_pubkey_id`, `amount`, `activation_block` (`block_index + 6`), `deactivation_block` (set on UNSTAKE), `status_id`, `source_id`. Capabilities (`price`, `cross_chain`, `oracle_publish`, `attestation`, `full_node`) are derived from a pubkey's aggregate active `amount` against the governance-configured minimums: there is no `tier` column. | | `unstakes` | Capability UNSTAKE v0 records: `signing_pubkey_id`, `amount`, `cooldown_end_block`, `status_id`; links back to the originating stake by pubkey. The cooldown end is `block_index + STAKING.COOLDOWN_BLOCKS`; the cooldown length is governance-configurable via the `STAKING.COOLDOWN_BLOCKS` parameter (default 1000 blocks), not a hardcoded constant. Contract-targeted UNSTAKE v1 records do **not** appear here; they are written to `contract_unstakes` with a per-contract cooldown (see below). | | `delegations` | Active and historical DELEGATE records: `signing_pubkey_id`, `activation_block`, `deactivation_block` (set on DELEGATE v2 revoke), `status_id` | -| `validator_rewards` | Per-validator accumulated rewards: `source_id`, `signing_pubkey_id`, `reward_type` (`oracle_round`, `oracle_base`, `oracle_full_node`, `attest_fee`, `anchor_bundle`, or `anchor_archive`), `round_reference`, `amount`, `block_index`. `attest_fee` rows are derived by the indexer during block processing; the oracle labels (`oracle_round` / `oracle_base` / `oracle_full_node`) are reserved by the schema, but no consensus rail mints them today (which one a rail would use depends on whether the full-node reward tier is active: see [COLLECT](../../protocol/actions/collect.md#reward-population-path)); `anchor_bundle` and `anchor_archive` rows are derived by the indexer from the on-chain ANCHOR bytes at or above their own flag-days (`ANCHOR_REWARD_ACTIVATION` / `ARCHIVE_REWARD_ACTIVATION`); rows below those heights were pushed from the hub via `pushvalidatorrewards` and are restored by ANCHOR full-parse recovery rather than re-derived. One `anchor_bundle` row is written per published bundle, keyed on the bundle's snapshot block, not one per chain. | +| `validator_rewards` | Per-validator accumulated rewards: `source_id`, `signing_pubkey_id`, `reward_type` (`oracle_round`, `oracle_base`, `oracle_full_node`, `attest_fee`, `rollcall_publish`, `anchor_bundle`, or `anchor_archive`), `round_reference`, `amount`, `block_index`. `attest_fee` rows are derived by the indexer during block processing; a `rollcall_publish` row is derived at the BTC-side ROLLCALL epoch close, paying the frozen `ROLLCALL_REWARD_AMOUNT` to the elected leader of a rolled epoch (an unrolled epoch mints none), and a rollback deletes it with its derive block (see [ROLLCALL](../../protocol/actions/rollcall.md)); the oracle labels (`oracle_round` / `oracle_base` / `oracle_full_node`) are reserved by the schema, but no consensus rail mints them today (which one a rail would use depends on whether the full-node reward tier is active: see [COLLECT](../../protocol/actions/collect.md#reward-population-path)); `anchor_bundle` and `anchor_archive` rows are derived by the indexer from the on-chain ANCHOR bytes at or above their own flag-days (`ANCHOR_REWARD_ACTIVATION` / `ARCHIVE_REWARD_ACTIVATION`); rows below those heights were pushed from the hub via `pushvalidatorrewards` and are restored by ANCHOR full-parse recovery rather than re-derived. One `anchor_bundle` row is written per published bundle, keyed on the bundle's snapshot block, not one per chain. | | `stake_key_revocations` | Records DELEGATE v2 revocations of the original stake signing key: `source_id`, `signing_pubkey_id`, `deactivation_block`, `action_index`, `block_index`, `status_id`. A later STAKE v2 (higher `action_index`) by the same `(source, pubkey)` clears the revocation. Queried via `createStakeKeyRevocation` / `getStakeKeyRevocation` in `src/db/stakes.js`. | | `reward_claims` | COLLECT records: `source_id`, `amount`, `status_id`, `block_index` | @@ -213,6 +213,7 @@ Two slashing systems produce distinct table families. | `oracle_prices` | Local mirror of the hub's `oracle_prices` table (PRICE v1 user oracle rows). Populated by `hub_db_sync`. Rolled back on reorg by `(source_chain, action_index)` | | `price_snapshots` | Local mirror of the hub's `price_snapshots` table (PRICE v0 consensus rounds). Populated by `hub_db_sync`. Rolled back on reorg by `reference_block` | | `anchor_reward_attestations` | Local mirror of the hub's table of the same name: quorum-attested ANCHOR publisher rewards. Populated by `hub_db_sync`, INSERT-IGNORE, never retracted | +| `anchor_reward_attestation_copies` | Every distinct copy of a mirrored `anchor_reward_attestations` tuple, keyed by the tuple plus a digest of `doge_anchor_txid` and `publisher_attestations`. Written by `hub_db_sync` beside the kept row, INSERT-IGNORE, local and append-only, never retracted or rolled back. From `ANCHOR_REWARD_COPY_SELECTION_ACTIVATION` the BTC derive pass reads it so a tuple pays when any copy meets quorum and proves on DOGE; below it nothing reads it | | `attestation_responses` | Local mirror of the hub's table of the same name: the finalized ATTEST response, carried here instead of as a validator-paid on-chain transaction. Populated by `hub_db_sync`, INSERT-IGNORE, never retracted, and re-paged from the start on every bootstrap because the hub's row id is stripped on apply. TRANSPORT ONLY: the applier re-verifies each row's signatures against the responsible set it resolves from its OWN local request row before the response takes effect, so a row that fails verification is inert rather than a fork. Applied state lives in `attests`, not here | | `anchor_reward_reconcile_log` | Pre-image log of validator-reward rows deleted when an ANCHOR reconciles a contested publisher. Stores each deleted row's exact `amount` string and its original `reward_block_index`, so a reorg restore re-inserts it byte-identically, and only when the earn-block itself survives | | `cross_chain_call_rejections` | Refused XCALL dispatch injections, one row per `call_id`, with the refusal `reason` family, human-readable `detail`, and the attempt count and first/last block. Makes a call that never lands diagnosable instead of silently absent | diff --git a/components/indexer/ledger.md b/components/indexer/ledger.md index 9a48bfa4..40c3f536 100644 --- a/components/indexer/ledger.md +++ b/components/indexer/ledger.md @@ -77,7 +77,14 @@ The GAS address (defined per-chain, per-network in `src/coins/.js`) is the ## Fee Distribution -Fee payments are recorded in the `fees` table as debits from the source address with credits split between the `DONATE1` (protocol development) and `DONATE2` (community development) addresses. +Per-action protocol fees, such as the issuance and listing fees above, are recorded in the `fees` table. A native-coin fee (BTC/LTC/DOGE) is paid by an on-chain output to the fee destination address, so the ledger records no XCHAIN debit or credit for it. An XCHAIN-balance fee is debited from the paying source address and then routed by the payer's `FEE_PREFERENCE`, set with the [ADDRESS](../../protocol/actions/address.md) action: + +- `1` (burn): nothing is credited, so the fee leaves supply permanently. +- `2`, `0`, or no preference set (protocol development, the default): the full fee is credited to the per-chain `DONATE1` address. + +Fees are never split between addresses. The accepted preference set is `{0, 1, 2}`, so no community-development bucket exists in consensus; `DONATE2` is a configured per-chain address role that receives no fee credits. See [Gas](../../concepts/gas.md#fee-collection-and-distribution) for the full routing rules. + +VM gas billed from an XCHAIN balance for DEPLOY and EXECUTE takes a different path: it is debited from the source with no matching credit, so it leaves supply whatever the payer's preference. --- diff --git a/components/node/configuration.md b/components/node/configuration.md index 02ee453e..62d3135c 100644 --- a/components/node/configuration.md +++ b/components/node/configuration.md @@ -275,7 +275,7 @@ These env vars override where xchain-node stores its filesystem state on the hos > | LTC testnet | `testnet4/blocks/` | > | DOGE / LTC regtest | `regtest/blocks/` | > -> The `testnet4/` path above is Litecoin's testnet directory. BTC testnet4 is a separate network, where contract deploys cannot be mined at current miner block sizes. Use LTC or DOGE testnet for contract deployment and testing. +> The `testnet4/` path above is Litecoin's testnet directory. BTC testnet4 is a separate network. > > This matters if you ever try to free up disk by hand-mounting *only* the bare `blocks/` path, e.g. `-v /misc/dogecoin/testnet/blocks:/root/.dogecoin/blocks`. On testnet/regtest the daemon writes to `testnet3/blocks/` (etc.), which that bind does **not** cover, so the mount silently catches nothing and blocks keep accumulating on the default disk. No error is raised. > diff --git a/components/regtest-miner/operations.md b/components/regtest-miner/operations.md index 8ce36cdd..8e3861c0 100644 --- a/components/regtest-miner/operations.md +++ b/components/regtest-miner/operations.md @@ -96,11 +96,17 @@ Return the current mining loop state for operator diagnostics and CI health chec "jsonrpc": "2.0", "result": { "wallet_ready": true, + "wallet_balance": 50.0, + "wallet_funded": true, + "wallet_balance_at": 1718900003000, "mempool_size": 0, + "idle_mine_interval_ms": 0, "blocks_mined": 42, "last_mine_at": 1718900000000, "consecutive_errors": 0, - "mining_paused": false + "mine_failures": 0, + "mining_paused": false, + "mining_started": true }, "id": 2 } @@ -108,12 +114,20 @@ Return the current mining loop state for operator diagnostics and CI health chec | Field | Type | Description | |---|---|---| -| `wallet_ready` | boolean | `true` once `prepareWallet` has completed; gates on mine-readiness, not just server availability | +| `wallet_ready` | boolean | `true` once `prepareWallet` has completed; gates on mine-readiness, not just server availability. It marks startup completion only and stays `true` across a reorg, even one that leaves the wallet unable to fund a send; read `wallet_funded` for that. | +| `wallet_balance` | number or null | The wallet's spendable balance in coin units from the last `getbalance` read. `null` when no read has happened or the last read failed; neither `null` nor `0` counts as funded. | +| `wallet_funded` | boolean | `true` only when `wallet_balance` is a number above 0. Use this, not `wallet_ready`, to learn whether the wallet can still fund a send (for example after a deep `invalidate_block`). | +| `wallet_balance_at` | number or null | Unix timestamp in milliseconds of the last balance read attempt, successful or not; `null` before the first read | | `mining_paused` | boolean | `true` while the auto-mine loop is paused (`pause_mining`, or a `fill_mempool`/`invalidate_block` that never resumed). If mining seems stalled, check this first. | +| `mining_started` | boolean | `true` once wallet preparation finished and the auto-mine loop started; a pause never resets it. A miner still starting reads `mining_paused: true` with `mining_started: false`. | | `mempool_size` | number | Number of transactions currently in the mempool as of the last polling cycle | +| `idle_mine_interval_ms` | number | Current mine-empty heartbeat interval in milliseconds; `0` means mining is driven only by the mempool (see `set_idle_mine_interval`) | | `blocks_mined` | number | Cumulative count of blocks mined by this miner instance since startup | | `last_mine_at` | number or null | Unix timestamp in milliseconds (`Date.now()`) of the most recent successful block mine; `null` if no block has been mined yet in this session | | `consecutive_errors` | number | Running count of consecutive RPC failures (either `getrawmempool` or `generatetoaddress`). Resets to 0 on the next successful RPC call. Useful for alerting: a non-zero value means the miner is retrying with exponential backoff. | +| `mine_failures` | number | Consecutive failed block mines, on a streak of its own: a successful mempool read zeroes `consecutive_errors`, so that counter cannot show a miner that answers RPC but cannot generate a block. Resets on the next mined block; `health` reports it as a stall reason. | + +**Balance freshness.** The balance is read at startup, re-read before `invalidate_block` and `reconsider_block` return, and refreshed by the auto-mine loop at most every 5 seconds, including while mining is paused or `fill_mempool` is running. A single failed read, a node restart included, shows as `wallet_balance: null` and `wallet_funded: false` for one interval and then recovers, so re-poll once before concluding the wallet is drained. A `wallet_balance_at` well over 5 seconds old while `mining_started` is `true` means the refresh has stalled. ### `health` @@ -375,7 +389,21 @@ When the coin node is unreachable (ECONNREFUSED, timeout, DNS failure), the mine ### Pinned Wallet Fee Rate -On a matured regtest chain, the node's `estimatesmartfee` can inflate to absurd rates, pushing funding-send fees above `-maxtxfee` and failing `send_funds`/`fill_mempool` with RPC error -6 (easily misread as UTXO exhaustion). `prepareWallet` therefore pins a fixed wallet fee rate via `settxfee` (best-effort; a daemon that rejects `settxfee` falls back to the estimate path, and startup logs say which happened). +On a matured regtest chain, the node's `estimatesmartfee` can inflate to absurd rates, pushing funding-send fees above `-maxtxfee` and failing `send_funds`/`fill_mempool` with RPC error -6 (easily misread as UTXO exhaustion). `prepareWallet` therefore pins a fixed funding fee ceiling, by whichever mechanism the coin's daemon implements. The ceiling is 0.001 coins/kB, which is the same rate as 100 sat/vB. + +The coin is the part of `NETWORK` before the first `-` (`bitcoin`, `litecoin`, `dogecoin`); a bare `regtest` has no coin. + +| Mechanism | Used by | What `sendtoaddress` sends | +|---|---|---| +| `fee_rate` | Bitcoin, always: Core 31 deleted `settxfee`, so the miner never calls it on BTC. Also Litecoin or a bare `regtest` whose `settxfee` answers method-not-found (-32601), and Litecoin whose `settxfee` stays unreachable through its retries. | Named params `{ address, amount, fee_rate: 100 }` on every call | +| `settxfee` | Litecoin, Dogecoin and a bare `regtest` whose daemon accepts `settxfee 0.001` (the normal path for LTC v0.21 and DOGE v1.14) | Positional `[address, amount]`; the wallet-wide pin caps the fee | +| none (fee estimate) | A daemon that refuses the rate (any other RPC error, or a reply that is not `true`); Dogecoin without `settxfee`, since DOGE v1.14 refuses `fee_rate`; Dogecoin or a bare `regtest` whose `settxfee` stays unreachable | Positional `[address, amount]`; `estimatesmartfee` sets the fee | + +Bitcoin has no fallback to the estimate: falling back would give the ceiling up on exactly the chain that needs it. A `settxfee` that times out, loses its transport or answers warmup (-28) is tried up to three times before it counts as unreachable. + +`settxfee` is in-memory wallet state and is lost when the node restarts, so the miner pins again after it reloads a lost wallet, both on the `send_funds` retry and from the background balance read. A re-pin whose `settxfee` cannot be reached keeps a per-call `fee_rate` already in force. + +Startup logs name the mechanism in force: `via the per-call fee_rate argument`, `via settxfee`, or `funding sends use the fee estimate`. On Bitcoin a funding -6 never comes with a `settxfee` call or log line, so look for the per-call `fee_rate` line instead. ### Error Sanitization diff --git a/components/sdk/actions.md b/components/sdk/actions.md index 655426e0..17376abe 100644 --- a/components/sdk/actions.md +++ b/components/sdk/actions.md @@ -1457,19 +1457,30 @@ await sdk.session(wif).stakeToContract({ Begin unstaking cooldown for a previously staked pubkey. Two flavors with different chain reach: -- **v0:** capability unstake.** BTC-only. Returns the full aggregate capability stake for the pubkey (v1 original + any v2 top-ups). -- **v1:** contract-targeted unstake.** Works on any chain (BTC, LTC, DOGE). Releases the single `(targetContractIndex, signingPubkey, tick)` stake row. +- **v0:** capability unstake.** BTC-only. Without `amount`, returns the full aggregate capability stake for the pubkey (v1 original + any v2 top-ups); with `amount`, only that much enters cooldown and the residual stays staked. +- **v1:** contract-targeted unstake.** Works on any chain (BTC, LTC, DOGE). Releases the single `(targetContractIndex, signingPubkey, tick)` stake row, or only `amount` of it when `amount` is given. ```js -// Capability unstake +// Capability unstake (full sweep) await sdk.unstake({ signingPubkey: 'aabb...' }); +// Partial capability unstake: 250 enters cooldown, the rest stays staked +await sdk.unstake({ signingPubkey: 'aabb...', amount: '250' }); + // Contract-targeted unstake (convenience wrapper forces version: 1) await sdk.session(wif).unstakeFromContract({ signingPubkey: 'aabb...', targetContractIndex: 500, tick: 'MYTOKEN' }); + +// Partial contract-targeted unstake +await sdk.session(wif).unstakeFromContract({ + signingPubkey: 'aabb...', + targetContractIndex: 500, + tick: 'MYTOKEN', + amount: '40' +}); ``` | Field | Required | Description | @@ -1478,13 +1489,16 @@ await sdk.session(wif).unstakeFromContract({ | `signingPubkey` | Yes | Ed25519 pubkey of the stake to release | | `targetContractIndex` | v1 only | `action_index` of the stakeable contract | | `tick` | v1 only | Ticker of the stake row to release | +| `amount` | Optional | Partial-unstake amount; omit for a full sweep. Must be greater than 0, no finer than the token's decimals, and no more than the active staked balance (an over-ask is rejected, never clamped). See [Partial Unstake](../../protocol/actions/unstake.md#partial-unstake-optional-amount) | ### Formats | Version | Format | |---|---| -| 0 | `VERSION\|SIGNING_PUBKEY` (capability) | -| 1 | `VERSION\|SIGNING_PUBKEY\|TARGET_CONTRACT_INDEX\|TICK` (contract-targeted) | +| 0 | `VERSION\|SIGNING_PUBKEY[\|AMOUNT]` (capability) | +| 1 | `VERSION\|SIGNING_PUBKEY\|TARGET_CONTRACT_INDEX\|TICK[\|AMOUNT]` (contract-targeted) | + +The trailing `AMOUNT` is gated by the `PARTIAL_UNSTAKE_COLLECT` protocol change; below it a present `AMOUNT` is ignored and the unstake is a full sweep. --- @@ -1540,17 +1554,27 @@ await sdk.delegate({ version: 3, signingPubkey: 'aabb...', targetContractIndex: ## COLLECT -Collect all accrued validator rewards (BTC chain only). No additional parameters required. +Collect accrued validator rewards (BTC chain only). With no parameters it claims the full unclaimed total; an optional `amount` claims only that much and leaves the remainder pending (see [Partial Claim](../../protocol/actions/collect.md#partial-claim-optional-amount)). ```js +// Claim everything await sdk.collect({}); + +// Claim 25 and leave the remainder pending +await sdk.collect({ amount: '25' }); ``` +| Field | Required | Description | +|---|---|---| +| `amount` | Optional | Partial-claim amount; omit to claim the full unclaimed total | + ### Formats | Version | Format | |---|---| -| 0 | `VERSION` | +| 0 | `VERSION[\|AMOUNT]` | + +The trailing `AMOUNT` is gated by the `PARTIAL_UNSTAKE_COLLECT` protocol change; below it a present `AMOUNT` is ignored and the full total is claimed. --- diff --git a/components/sdk/configuration.md b/components/sdk/configuration.md index db87cc20..a7b79141 100644 --- a/components/sdk/configuration.md +++ b/components/sdk/configuration.md @@ -111,6 +111,17 @@ The SDK reads these environment variables at construction time. A `.env` file in | `XC_CONTRACTS_REGTEST_ACTIVATION` | **Regtest only.** Arms the `regtest` entries of `VOTE_CALLBACK_BINDING_REQUIRES_USABLE_METHOD` in the SDK's registry copy. `armed`, `genesis` or `on` arms at `0`; a non-negative integer arms at that height. Unset or `off` leaves the gate inert; anything else is refused with a warning and leaves the gate inert. | Activation registry copy (`src/consensus/gate_registry.js`) | | `XC_LISTS_MARKET_REGTEST_ACTIVATION` | **Regtest only.** Arms the `regtest` entries of the height-keyed LISTS MARKET gates, among them `LIST_REFERENCE_REQUIRES_VALID_LIST`, so pre-flight judges an invalid LIST reference against the venue's armed height rather than genesis. `armed`, `genesis` or `on` arms at `0`; a non-negative integer arms at that height. Unset or `off` leaves the gates inert; anything else is refused with a warning and leaves them inert. Set identically on every process in the venue. | Activation registry copy (`src/consensus/gate_registry.js`) and pre-flight mirrors (`src/preflight/activation.js`) | | `XC_LISTS_MARKET_REGTEST_TIME` | **Regtest only.** Arms the `regtest` entries of the time-keyed LISTS MARKET gates, among them `DISPENSER_SETTLEMENT_PRICE_ACTIVATION`, which pre-flight words into its disclosures. `armed`, `genesis` or `on` arms at `0`; a non-negative integer arms at that value (a unix time in seconds). Unset or `off` leaves the gates inert; anything else is refused with a warning and leaves them inert. | Activation registry copy (`src/consensus/gate_registry.js`) and pre-flight mirrors (`src/preflight/activation.js`) | +| `XC_REMOTE_TOKEN_ADMISSION_REGTEST_ACTIVATION` | **Regtest only.** Arms the `regtest` entry of `REMOTE_TOKEN_ADMISSION_ACTIVATION` in the SDK's registry copy. `armed`, `genesis` or `on` arms at `0`; a non-negative integer arms at that height. Unset or `off` leaves the gate inert; anything else is refused with a warning and leaves the gate inert. Set identically on every process in the venue. | Activation registry copy (`src/consensus/gate_registry.js`) | +| `XC_PASS_PROGRESS_RETIREMENT_REGTEST_ACTIVATION` | **Regtest only.** Arms the `regtest` entry of `PASS_PROGRESS_RETIREMENT_ACTIVATION` in the SDK's registry copy. `armed`, `genesis` or `on` arms at `0`; a non-negative integer arms at that height. Unset or `off` leaves the gate inert; anything else is refused with a warning and leaves the gate inert. Set identically on every process in the venue. | Activation registry copy (`src/consensus/gate_registry.js`) | +| `XC_SLASH_HISTORICAL_SET_REGTEST_ACTIVATION` | **Regtest only.** Arms the `regtest` entry of `SLASH_HISTORICAL_SET_ACTIVATION` in the SDK's registry copy. `armed`, `genesis` or `on` arms at `0`; a non-negative integer arms at that height. Unset or `off` leaves the gate inert; anything else is refused with a warning and leaves the gate inert. Set identically on every process in the venue. | Activation registry copy (`src/consensus/gate_registry.js`) | +| `XC_ANCHOR_REWARD_COPY_SELECTION_REGTEST_ACTIVATION` | **Regtest only.** Arms the `regtest` entry of `ANCHOR_REWARD_COPY_SELECTION_ACTIVATION` in the SDK's registry copy. `armed`, `genesis` or `on` arms at `0`; a non-negative integer arms at that height. Unset or `off` leaves the gate inert; anything else is refused with a warning and leaves the gate inert. Set identically on every process in the venue. | Activation registry copy (`src/consensus/gate_registry.js`) | +| `XC_TOKEN_SUPPLY_REPRESENTATION_REGTEST_ACTIVATION` | **Regtest only.** Arms the `regtest` entry of `TOKEN_SUPPLY_REPRESENTATION_ACTIVATION` in the SDK's registry copy. `armed`, `genesis` or `on` arms at `0`; a non-negative integer arms at that height. Unset or `off` leaves the gate inert; anything else is refused with a warning and leaves the gate inert. Set identically on every process in the venue. | Activation registry copy (`src/consensus/gate_registry.js`) | +| `XC_DISPENSER_FIAT_PRICING_REGTEST_ACTIVATION` | **Regtest only.** Arms the `regtest` entry of `DISPENSER_FIAT_PRICING_ACTIVATION` in the SDK's registry copy. `armed`, `genesis` or `on` arms at `0`; a non-negative integer arms at that height. Unset or `off` leaves the gate inert; anything else is refused with a warning and leaves the gate inert. Set identically on every process in the venue. | Activation registry copy (`src/consensus/gate_registry.js`) | +| `XC_BRIDGE_POLICY_APPLY_ORDER_REGTEST_ACTIVATION` | **Regtest only.** Arms the `regtest` entry of `BRIDGE_POLICY_APPLY_ORDER_ACTIVATION` in the SDK's registry copy. `armed`, `genesis` or `on` arms at `0`; a non-negative integer arms at that height. Unset or `off` leaves the gate inert; anything else is refused with a warning and leaves the gate inert. Set identically on every process in the venue. | Activation registry copy (`src/consensus/gate_registry.js`) | +| `XC_VM_EMISSION_AMOUNT_FORMAT_REGTEST_ACTIVATION` | **Regtest only.** Arms the `regtest` entry of `VM_EMISSION_AMOUNT_FORMAT_ACTIVATION` in the SDK's registry copy. `armed`, `genesis` or `on` arms at `0`; a non-negative integer arms at that height. Unset or `off` leaves the gate inert; anything else is refused with a warning and leaves the gate inert. Set identically on every process in the venue. | Activation registry copy (`src/consensus/gate_registry.js`) | +| `XC_EXECUTE_FEE_CAP_REGTEST_ACTIVATION` | **Regtest only.** Arms the `regtest` entry of `EXECUTE_FEE_CAP_ACTIVATION` in the SDK's registry copy. `armed`, `genesis` or `on` arms at `0`; a non-negative integer arms at that height. Unset or `off` leaves the gate inert; anything else is refused with a warning and leaves the gate inert. Set identically on every process in the venue. | Activation registry copy (`src/consensus/gate_registry.js`) | +| `XC_VOTE_V3_TICK_EXISTENCE_REGTEST_ACTIVATION` | **Regtest only.** Arms the `regtest` entry of `VOTE_V3_TICK_EXISTENCE_ACTIVATION` in the SDK's registry copy. `armed`, `genesis` or `on` arms at `0`; a non-negative integer arms at that height. Unset or `off` leaves the gate inert; anything else is refused with a warning and leaves the gate inert. Set identically on every process in the venue. | Activation registry copy (`src/consensus/gate_registry.js`) | +| `XC_MATCH_ORIGIN_OPEN_REGTEST_ACTIVATION` | **Regtest only.** Arms the `regtest` entry of `MATCH_ORIGIN_OPEN_ACTIVATION` in the SDK's registry copy. `armed`, `genesis` or `on` arms at `0`; a non-negative integer arms at that height. Unset or `off` leaves the gate inert; anything else is refused with a warning and leaves the gate inert. Set identically on every process in the venue. | Activation registry copy (`src/consensus/gate_registry.js`) | The `XC_*` arming variables exist because the SDK carries a byte-identical copy of the shared activation rows the indexer, hub, sync and explorer carry, and that copy arms its regtest entries from the same variables by the same grammar (`src/consensus/gate_registry/regtest_env.js`). The SDK drives no roll call and admits no mirror row itself; honouring the levers keeps its reading of a row identical to the venue's, so a consensus-identity comparison across the venue's processes does not diverge on the SDK. Regtest ships inert on purpose, and a venue sets each lever **identically** on every process it arms. diff --git a/components/sdk/errors.md b/components/sdk/errors.md index 2c8889d8..93ef6448 100644 --- a/components/sdk/errors.md +++ b/components/sdk/errors.md @@ -272,7 +272,10 @@ Thrown by the `X402Gateway` and `X402Client` during HTTP 402 payment flows. See | `X402_DEPOSIT_EXHAUSTED` | Deposit-scheme: payer's deposited balance is insufficient for one more call | | `X402_STATE_CORRUPT` | An invoice file or deposit ledger file is unreadable | | `X402_NO_USABLE_SCHEME` | Client-side: the gateway's challenge contains no `xchain-send` offer | -| `X402_PRICE_TOO_HIGH` | Client-side: the gateway's offered amount exceeds the client's `maxAmount` | +| `X402_PRICE_TOO_HIGH` | Client-side: the gateway's offered amount exceeds the client's `maxAmount` ceiling for the offered tick | +| `X402_TICK_NOT_ALLOWED` | Client-side: the gateway's offer names a tick the client's `maxAmount` sets no ceiling for (every tick, when `maxAmount` is omitted) | +| `X402_BAD_OFFER` | Client-side: the gateway's `xchain-send` offer has a malformed amount, or no tick or `payTo` | +| `X402_CONFIG` | Construction-time: the gateway or client was built with a missing or malformed option (on the client, a missing session or fetch, or a malformed `maxAmount`) | | `X402_PAYMENT_NOT_ACCEPTED` | Client-side: payment was broadcast but the gateway did not accept it after all retries | --- diff --git a/components/sdk/light-client.md b/components/sdk/light-client.md index 6e9b784a..90eb47b4 100644 --- a/components/sdk/light-client.md +++ b/components/sdk/light-client.md @@ -197,8 +197,10 @@ sdk.light.followForward({ ``` It returns the new rolling trust root (`trusted`), the chain of `adopted` -checkpoints, and stops at the first step that fails (`VALIDATOR_SET_UNVERIFIED@h` -or `QUORUM_FAILED@h`). +checkpoints, and stops at the first step that fails (`ROOTS_NOT_SIGNED@h`, +`VALIDATOR_SET_UNVERIFIED@h` or `QUORUM_FAILED@h`). `ROOTS_NOT_SIGNED@h` is a +step below the checkpoint-commitment flag day, whose roots are outside the +signed bytes and so cannot carry the trust root forward. --- diff --git a/components/sdk/nft-and-registry.md b/components/sdk/nft-and-registry.md index af81d129..ad8ec544 100644 --- a/components/sdk/nft-and-registry.md +++ b/components/sdk/nft-and-registry.md @@ -88,7 +88,7 @@ const params = sdk.nft.collectionItem({ ### `sdk.nft.attachContentParams(params)` → LINK params -Build LINK params that officially attach a FILE to a token (the NFT content-attachment pattern). The LINK source must own the token; the indexer validates ownership. +Build LINK params that officially attach a FILE to a token (the NFT content-attachment pattern). The builder puts the FILE in `COIN1` and the token's ISSUE in `COIN2`, the only order the indexer owner-validates. The LINK source must own the token; the indexer validates ownership. ```js const params = sdk.nft.attachContentParams({ diff --git a/components/sdk/wallet.md b/components/sdk/wallet.md index 8d9ef566..25564aec 100644 --- a/components/sdk/wallet.md +++ b/components/sdk/wallet.md @@ -3,7 +3,7 @@ # XChain Platform SDK: Wallet & Auth Reference -This document covers the SDK's wallet and authentication modules: key management, address validation, message signing, challenge-response wallet verification, PSBT signing, transaction broadcasting, and UTXO queries. +This document covers the core SDK wallet and authentication APIs: key management, address validation, message signing, challenge-response wallet verification, PSBT signing, transaction broadcasting, and UTXO queries. --- @@ -14,7 +14,7 @@ The SDK provides two modules for wallet integration: - **`sdk.wallet`** (`WalletUtils`): Key management, address derivation and validation, PSBT signing, transaction broadcasting, and UTXO queries. - **`sdk.auth`** (`AuthUtils`): Challenge-response wallet ownership verification and message signing. -Both modules are available on the SDK instance and as top-level convenience methods: +Common methods are available through the module namespaces and as top-level convenience methods: ```js // Zero-config: for mainnet/testnet, encoderUrl resolves automatically to the @@ -25,7 +25,7 @@ const sdk = new XChainSDK({ network: 'bitcoin-mainnet' }); sdk.wallet.generateKeyPair(); sdk.auth.generateChallenge('bc1q...'); -// Convenience access (same thing) +// Top-level convenience access for the same calls sdk.generateKeyPair(); sdk.generateChallenge('bc1q...'); ``` @@ -108,7 +108,7 @@ Validate any coin address against the configured network (or a specific override ```js const result = sdk.validateAddress('bc1qexample...'); // result.valid: boolean -// result.type: 'p2pkh', 'p2wpkh', 'p2sh', 'p2wsh', or null +// result.type: 'p2pkh', 'p2wpkh', 'p2sh', 'p2wsh', 'p2tr', or null // result.network: 'bitcoin-mainnet', etc., or null // result.error: error message string, or null @@ -144,7 +144,9 @@ const challenge = sdk.generateChallenge('bc1q...', { // challenge.challenge === the custom message, used as-is ``` -When `opts.message` is provided, the SDK uses it verbatim. The nonce and timestamp metadata are still returned for the caller's bookkeeping but are not embedded in the message. This means sites can generate their own messages and verify independently of the SDK. +When `opts.message` is truthy, the SDK uses it verbatim. The nonce and timestamp metadata are still returned for the caller's bookkeeping but are not embedded in the message. This means sites can generate their own messages and verify independently of the SDK. + +Challenge state is caller-managed. Store the nonce and enforce expiry and single use on the server; `verifyOwnership` verifies only the address, message, and signature supplied to it. ### Step 2: Sign the Challenge (Wallet/Client) @@ -164,22 +166,26 @@ sdk.signMessage(message, wif, { segwitNative: true }); sdk.signMessage(message, wif, { segwitRedeemScript: true }); ``` +`signMessage` supports P2PKH, P2WPKH, and P2SH-P2WPKH addresses. It does not provide a Taproot message-signing mode. + ### Step 3: Verify Ownership (Server) ```js const result = sdk.verifyOwnership('bc1quseraddress...', challenge.challenge, signed.signature); // result.valid: boolean // result.address: the address that was verified -// result.error: null on success, error message on failure +// result.error: input/verification error message, or null ``` -`verifyOwnership` never throws; it returns `{ valid: false, error: '...' }` on failure. This makes it safe to use in request handlers without try/catch: +`verifyOwnership` never throws. Invalid input or a verification exception returns `{ valid: false, error: '...' }`. A well-formed signature that does not match returns `valid: false` and can still have `error: null`, so always make the authorization decision from `valid`: ```js app.post('/auth/verify', (req, res) => { const { address, message, signature } = req.body; const result = sdk.verifyOwnership(address, message, signature); - if (!result.valid) return res.status(401).json({ error: result.error }); + if (!result.valid) { + return res.status(401).json({ error: result.error || 'Signature does not match.' }); + } // Grant access... }); ``` @@ -206,7 +212,7 @@ sequenceDiagram ### Verify Any Message -`verifyMessage` is a simpler alias that doesn't return the address: +`verifyMessage` is a simpler wrapper that doesn't return the address: ```js const result = sdk.verifyMessage(address, message, signature); @@ -220,10 +226,20 @@ Sites that don't want to depend on the SDK for server-side verification can use ```js const bitcoinMessage = require('bitcoinjs-message'); -const valid = bitcoinMessage.verify(message, address, signature); +const messagePrefixes = { + bitcoin: '\x18Bitcoin Signed Message:\n', + litecoin: '\x19Litecoin Signed Message:\n', + dogecoin: '\x19Dogecoin Signed Message:\n' +}; +const valid = bitcoinMessage.verify( + message, + address, + signature, + messagePrefixes.bitcoin +); ``` -Or use the coin node's `verifymessage` JSON-RPC method. The signatures produced by `sdk.signMessage` are standard Bitcoin message signatures compatible with all of these. +The message prefix is chain-specific: Bitcoin, Litecoin, and Dogecoin use different prefixes. Signatures from `sdk.signMessage` can be verified with `bitcoinjs-message` when the verifier uses the same network prefix and address type. --- @@ -238,7 +254,17 @@ const result = sdk.signPsbt(unsignedPsbtHex, wif); // result.psbtHex: signed PSBT hex ``` -This method finalizes all inputs after signing and extracts the raw transaction. The `txHex` is what you pass to `broadcastTx`. +By default, this method attempts to sign and finalize every input, then extracts the raw transaction. The `txHex` is what you pass to `broadcastTx`. + +For an input-scoped signature, use the namespaced method and pass the exact approved input indices: + +```js +const partial = sdk.wallet.signPsbt(unsignedPsbtHex, wif, { + inputIndices: [0, 2] +}); +// partial.psbtHex contains the signed and finalized selected inputs. +// partial.txHex and partial.txid are null if any other input remains unfinalized. +``` ### Full Workflow: Create → Sign → Broadcast @@ -287,16 +313,18 @@ Requires the encoder to be configured. The encoder proxies the request to the xc ## Error Handling -The wallet and auth modules throw two error classes: +The wallet and auth modules use two domain error classes: | Class | When | |---|---| -| `SDKWalletError` | Key management, address derivation, PSBT signing, broadcasting, UTXO queries | -| `SDKAuthError` | Challenge generation, message signing, signature verification failures | +| `SDKWalletError` | Local wallet validation, address derivation, PSBT signing, and wrapped broadcasting or UTXO failures | +| `SDKAuthError` | Challenge generation and message signing | See [Errors](errors.md) for the complete list of error codes. -Exception: `verifyOwnership` and `verifyMessage` **do not throw**; they return `{ valid: false, error: '...' }` on failure. This is intentional so that verification can be used in request handlers without try/catch boilerplate. +The top-level `sdk.broadcastTx` and `sdk.getUTXOs` methods throw `SDKConfigError` when no encoder is configured. Encoder errors that already use an SDK error class are rethrown unchanged. + +Exception: `verifyOwnership` and `verifyMessage` **do not throw**. They return `valid: false`; `error` is populated for invalid input or a verification exception and may be `null` for a non-matching signature. --- diff --git a/components/sdk/websocket.md b/components/sdk/websocket.md index 0e00a442..d476f3a9 100644 --- a/components/sdk/websocket.md +++ b/components/sdk/websocket.md @@ -306,7 +306,11 @@ market receives stakes placed by strangers on it and nothing from any other. ## Reconnection and Catch-Up -The client automatically reconnects on disconnect with exponential backoff. On reconnect, it replays every subscription. Those that carry actions (`actions` and `address`) carry `since_action_index` set to the last action index received before the disconnect; the `WELCOME` of the new connection does not move that cursor. The explorer runs one catch-up per connection, so the client sends these one at a time, each after the previous one's `CATCH_UP_COMPLETE`, and continues a replay the server truncated at its row cap from where it stopped. +The client automatically reconnects on disconnect with exponential backoff. On reconnect, it replays every subscription. Those that carry actions (`actions` and `address`), and those naming `blocks`, carry `since_action_index` set to the last action index received before the disconnect; the `WELCOME` of the new connection does not move that cursor. The explorer runs one catch-up per connection, so the client sends these one at a time, each after the previous one's `CATCH_UP_COMPLETE`, and continues a replay the server truncated at its row cap from where it stopped. A `blocks` subscription replays nothing (the explorer never replays `NEW_BLOCK`); its `CATCH_UP_COMPLETE` names `NEW_BLOCK` in `not_replayed`, which the client emits as a `not_replayed` event. + +Every frame the client sends is paced under the explorer's per-connection message rate (`limits.max_message_rate` from `WELCOME`, 10 per second before it arrives), so a reconnect with many subscriptions goes out over a few hundred milliseconds instead of in one burst. A subscribe or catch-up the explorer still drops with `RATE_LIMITED` is sent again after one drain interval, up to five times. The `messageRate` constructor option sets the rate assumed before `WELCOME`. + +A `REORG` frame (the chain was rolled back below rows already delivered) lowers the cursor to just below its `first_invalid_action_index`, and is dispatched to `ws.on('REORG', ...)` handlers like any frame: drop what you hold at or above that index, then apply the replacement frames that follow. See [REORG](../explorer/websocket.md#reorg). Catch-up events have `catch_up: true` in the envelope. After each replay, a `CATCH_UP_COMPLETE` event is emitted. @@ -316,15 +320,25 @@ ws.on('CATCH_UP_COMPLETE', (msg) => { }); ``` -When a catch-up is refused (for example `CATCH_UP_TOO_OLD`, when the cursor is further behind than the explorer's catch-up depth limit) or goes unanswered for 30 seconds, the client emits `resync_required` and moves on to the next subscription. The missed actions were not replayed, so backfill them over REST: +When a catch-up is refused (for example `CATCH_UP_TOO_OLD`, when the cursor is further behind than the explorer's catch-up depth limit, or `CATCH_UP_REORGED`, when a rollback while the client was away replaced rows at or below its cursor) or goes unanswered for 30 seconds, the client emits `resync_required` and moves on to the next subscription. The missed actions were not replayed, so backfill them over REST: ```javascript ws.on('resync_required', (msg) => { - // msg.data: { code, message, channels, since_action_index } + // msg.data: { code, message, channels, since_action_index }, plus + // fork_block_index and first_invalid_action_index for CATCH_UP_REORGED reloadFromRest(msg.data.channels, msg.data.since_action_index); }); ``` +When the replay comes from an explorer whose indexed tip is behind the chain, its `CATCH_UP_COMPLETE` carries `stale: true` and the client emits `stale_replay`. Every row it replayed is real, and the cursor still moves to the last one, but the replay ends at that tip rather than at the chain's. Later rows arrive live once the explorer catches up, so treat the event as a "you are behind" signal, not a gap to backfill: + +```javascript +ws.on('stale_replay', (msg) => { + // msg.data: { channels, since_action_index, latest_action_index } + showBehindBanner(msg.data.latest_action_index); +}); +``` + ```mermaid sequenceDiagram participant Client @@ -409,7 +423,7 @@ try { | `WS_TIMEOUT` | No response for a request within timeout | | `WEBSOCKET_NOT_CONFIGURED` | Called a WS method without configuring WebSocket URL | -Server-side errors (e.g., `SUBSCRIPTION_LIMIT`, `RATE_LIMITED`) are returned as rejected Promises from `subscribe()`. +Server-side errors (e.g., `SUBSCRIPTION_LIMIT`) are returned as rejected Promises from `subscribe()`. A `RATE_LIMITED` refusal is sent again under the same id up to five times first, and rejects the Promise only after that. ## Related diff --git a/components/sync/README.md b/components/sync/README.md index 96bf1652..f669e449 100644 --- a/components/sync/README.md +++ b/components/sync/README.md @@ -7,7 +7,7 @@ xchain-sync is the database replication layer of the XChain Platform. It enables lightweight validators and other consumers to obtain and stay current with both the indexer database and the decoder database without running their own decoder+indexer stacks. The service runs as a single long-lived Node.js process with an embedded Express REST API and a WebSocket server, both served on the same port. -All REST and WebSocket endpoints include a `/:dbType/` path segment. The `dbType` parameter is either `indexer` (full table set, with transparency log) or `decoder` (8 of the 9 decoder DB tables; `mempool_transactions` is excluded because it is non-deterministic across nodes, and the transparency log is not applicable to decoder data). +All REST and WebSocket endpoints include a `/:dbType/` path segment. The `dbType` parameter is either `indexer` (full table set, with transparency log) or `decoder` (8 of the 10 decoder DB tables; `mempool_transactions` is excluded because it is non-deterministic across nodes, `dispenser_extension_undo` is excluded because it is decoder-local reorg bookkeeping, and the transparency log is not applicable to decoder data). The service operates in two modes. In **server mode**, it runs alongside authoritative indexers and decoders on an xchain-node, polls each database for new blocks, and serves the data to remote clients via REST snapshots and real-time WebSocket streaming. In **client mode**, it connects to one or more remote sync servers, downloads a full database snapshot for initial bootstrap, then subscribes to a WebSocket stream for ongoing block-by-block replication into a local MariaDB instance. diff --git a/components/sync/configuration.md b/components/sync/configuration.md index 86213819..a455098b 100644 --- a/components/sync/configuration.md +++ b/components/sync/configuration.md @@ -82,6 +82,7 @@ In client mode, the service connects to remote sync servers and replicates their | `VERIFY_RECOMPUTE` | No | `true` | **Security-critical. Default ON. Disabling is DECLARED UNSAFE for consensus-relevant replicas.** When `true`, each block's consensus hashes (ledger/actions/contract) are independently recomputed from the replicated rows and checked against the committed hash. This is the only mechanism that verifies the catch-up join block: with it disabled, a reorg that occurs while the client is disconnected is silently stitched onto the orphaned pre-reorg tip, permanently forking the replica. | | `VERIFY_STATE_HASH` | No | `true` | **Security-critical. Default ON.** When `true`, the per-block state hash (covering in-place mutations and backdated refund credits not captured by the three consensus hashes) is recomputed on apply and checked. A mismatch triggers a durable halt. A NULL state hash (block from a source without the feature) is skipped, so enabling this can never false-halt against a back-level source. | | `VERIFY_STATE_COMMITMENT` | No | `true` | **Security-critical. Default ON.** When `true`, the per-block SPV state-commitment roots (balances, block Merkle root) are recomputed over the replica and compared to the source's committed values. A mismatch triggers a durable halt. NULL roots (blocks before the flag-day) are skipped. **Set to `false` on truncated replicas** seeded via `SYNC_BOOTSTRAP_DEPTH_*`, because their incomplete balances history would produce wrong roots. | +| `VERIFY_RANGE_BATCH` | No | `500` | Blocks recomputed per window when a replica verifies the committed blocks above its durable last-verified block (a catch-up range, or the unverified tail at startup or after a halt clear). The verified marker is written after each window, so memory stays at one block and a restart resumes from the last finished window. Minimum 1. | | `INDEX_MAP_PARITY_CHECK` | No | `false` | **Advisory only; never halts.** When `true`, periodically checks that the replica's `index_addresses` id-to-address mapping matches the source's deterministic-subset checksum. A mismatch is logged and counted but never halted on. Off by default because computing it scans `index_addresses` (an index on `block_index` is advisable before enabling on a high-volume chain). | | `TABLE_CONTENT_PARITY_CHECK` | No | `false` | **Advisory only; never halts.** When `true`, compares per-table content checksums between source and replica, not just the row counts published beside them: an equal-count content substitution in a table that no consensus hash reads passes every other check. Read on both sides (the server publishes the checksums on `/status`, a client at the same height recomputes and compares). A mismatch is logged and durably counted. Off by default because it reads a window of about 93 indexer tables per status poll. | | `TOKEN_FOLD_PARITY_CHECK` | No | `false` | **Advisory only; never halts.** When `true`, compares a digest of the `tokens` metadata columns that an `ISSUE` edit folds in place. No consensus hash covers these columns, and the content-parity windows leave them out as in-place state, so this is the only check that sees a replica missing an edit or its reorg reversal. Read on both sides (the server publishes the digest on `/status`, a client at the same height recomputes and compares). A mismatch is logged and durably counted. Off by default because it scans `tokens` on the source once per status poll. | @@ -202,7 +203,7 @@ XChain_{TICKER}_{Network}_{Component} In **server mode**, the service reads from the authoritative indexer and decoder databases; the same ones the indexer/decoder write to. -In **client mode**, the service creates replica databases with the same names and schema. Indexer replicas contain an exact copy of the indexer data; decoder replicas contain 8 of the 9 decoder tables (`mempool_transactions` excluded). +In **client mode**, the service creates replica databases with the same names and schema. Indexer replicas contain an exact copy of the indexer data; decoder replicas contain 8 of the 10 decoder tables (`mempool_transactions` and `dispenser_extension_undo` excluded). ## Database Schema @@ -227,14 +228,14 @@ One additional table is used by the sync service itself for indexer replicas onl ### Decoder replicas -Decoder replica databases use a smaller schema derived from the decoder DB. Of the 9 tables in the decoder DB, xchain-sync replicates 8 (`mempool_transactions` is excluded as it is non-deterministic across nodes): +Decoder replica databases use a smaller schema derived from the decoder DB. Of the 10 tables in the decoder DB, xchain-sync replicates 8 (`mempool_transactions` is excluded as it is non-deterministic across nodes, and `dispenser_extension_undo` is decoder-local reorg bookkeeping that stays local): - `blocks`, `transactions`, `transaction_outputs`, `dispensers` - `index_addresses`, `index_transactions`, `pubkeys`, `events` The transparency log table (`sync_meta`) is not created for decoder replicas. -**Note on `dispensers` replication:** `dispensers` is present in the replica but does NOT ride the per-block WebSocket stream. The decoder soft-expires dispenser rows each block (updating `expired_block_index`) and defers hard-purges to a separate job; neither mutation is block-bound. Streaming only inserted rows would cause a follower's count to grow monotonically above the source. Instead, `dispensers` converges to the current source state via full snapshot only. Do not use the per-block row count for `dispensers` as a completeness signal. +**Note on `dispensers` replication:** `dispensers` is present in the replica but does NOT ride the per-block WebSocket stream. The decoder soft-expires dispenser rows each block (updating `expired_block_index`) and hard-purges them later, once that height is reorg-safe-deep, in a step that runs after the block commits; neither change is an insert, so the per-block stream cannot carry it. Streaming only inserted rows would cause a follower's count to grow monotonically above the source. Instead, `dispensers` converges to the current source state via full snapshot only. Do not use the per-block row count for `dispensers` as a completeness signal. ## Connection Pool Configuration diff --git a/components/sync/operations.md b/components/sync/operations.md index cbadc87a..25c29bee 100644 --- a/components/sync/operations.md +++ b/components/sync/operations.md @@ -734,19 +734,20 @@ Decoder data is fully deterministic from the coin node (there are no synthetic c ### Tables replicated -The decoder DB contains 9 tables. xchain-sync replicates 8 of them: +The decoder DB contains 10 tables. xchain-sync replicates 8 of them: | Table | Replicated | Notes | |---|---|---| | `blocks` | Yes | Block-scoped | | `transactions` | Yes | Block-scoped | | `transaction_outputs` | Yes | TX-scoped | -| `dispensers` | Yes (snapshot only) | Excluded from the per-block stream; the decoder prunes expired dispensers each block with a count-reducing DELETE that streaming inserts can't represent, so it converges via the full snapshot instead | +| `dispensers` | Yes (snapshot only) | Excluded from the per-block stream; the decoder soft-expires dispensers inside the block transaction (an UPDATE that stamps `expired_block_index`) and, after the block commits, hard-purges rows soft-expired at a reorg-safe depth with a DELETE. Neither change is an insert the stream can carry, so the replica converges via the initial snapshot and then a periodic full-table reconcile (`DISPENSERS_RECONCILE_EVERY`, `DISPENSERS_RECONCILE_MAX_INTERVAL_MS`) | | `index_addresses` | Yes | Append-only index | | `index_transactions` | Yes | Append-only index | | `pubkeys` | Yes | Append-only index | | `events` | Yes | Operational/logging | | `mempool_transactions` | **No** | Excluded: non-deterministic across nodes | +| `dispenser_extension_undo` | **No** | Excluded: decoder-local reorg bookkeeping (each dispenser row's values before a block's first expiration extend) | Additionally, the transparency log table (`sync_meta`) is **not** created for decoder replicas; it is indexer-only. diff --git a/components/utxo-tracker/README.md b/components/utxo-tracker/README.md index f8d7e1a5..fa267d2d 100644 --- a/components/utxo-tracker/README.md +++ b/components/utxo-tracker/README.md @@ -24,7 +24,7 @@ In addition to confirmed block data, the tracker maintains a separate in-memory - **Batch writes**: LevelDB writes batched in groups of 200 blocks for throughput efficiency with atomic commit - **Two-pass transaction processing**: outputs inserted before inputs within each block, correctly handling intra-block spends - **Multi-chain support**: Bitcoin, Litecoin, and Dogecoin today, on mainnet, testnet, and regtest -- **AuxPoW block parsing**: Dogecoin and Litecoin HogEx block header stripping for correct decoding +- **AuxPoW block parsing**: Dogecoin AuxPoW block header stripping for correct decoding; Litecoin MWEB/HogEx transactions are handled separately by a per-transaction flag strip - **Bootstrap support**: compressed tar archive backup and restore for fast initial sync without re-scanning the full chain - **REST + JSON-RPC API**: dual interface for UTXO queries, balance lookups, address info, and bootstrap operations - **Rolling ETA**: 1000-block rolling window with day/hour/minute display for sync progress estimation diff --git a/components/utxo-tracker/architecture.md b/components/utxo-tracker/architecture.md index e261b517..cc37b1be 100644 --- a/components/utxo-tracker/architecture.md +++ b/components/utxo-tracker/architecture.md @@ -53,7 +53,7 @@ flowchart TD | `src/store/level_up_db.js` | `LevelUpStore` | LevelDB abstraction: binary key encoding/decoding, batch transactions, range scans, all 12 prefix type operations | | `src/chain/blockchain_connector.js` | `BlockchainConnector` | HTTP JSON-RPC facade: wires transport, batch fetching, block queries, mempool tracking, AuxPoW codec, RPC helpers, and connection pooling | | `src/chain/blockchain_connector/` | (multiple) | Part modules: auxpow_codec (AuxPoW header encoding and decoding), batch_fetch (parallel block and hash fetching), block_queries (blockchain info and block data retrieval), rpc_helpers (RPC error handling and node health tracking), transport_and_mempool (HTTP client and mempool synchronization), constants (logging configuration) | -| `src/chain/XChainBlockDecoder.js` | `XChainBlockDecoder` | Block and transaction parser: standard Bitcoin blocks, AuxPoW header stripping for Dogecoin/Litecoin HogEx | +| `src/chain/XChainBlockDecoder.js` | `XChainBlockDecoder` | Block and transaction parser: standard Bitcoin blocks, AuxPoW header stripping for Dogecoin, per-transaction MWEB/HogEx flag stripping for Litecoin | | `src/chain/crypto_networks.js` | `CryptoNetworks` | Network parameter lookup: maps network names to bitcoinjs-lib network objects for 9 network variants | | `src/common/util.js` | None | Utility functions: timing, hex/uint8 conversion, formatting | | `src/chain/bufferutils.js` | `BufferReader`, `BufferWriter` | Binary buffer reading/writing: UInt8/16/32/64LE, VarInt, slices | @@ -139,11 +139,11 @@ Within each block, transactions are processed in two passes: ### Concurrent Block Prefetch -To minimize RPC idle time, the tracker maintains a prefetch queue of up to `PREFETCH_SIZE` (10) blocks. For non-AuxPoW chains, block hashes and hex data are fetched in two batch HTTP requests. For AuxPoW chains (Dogecoin/Litecoin with HogEx), blocks are fetched individually because AuxPoW headers must be stripped before parsing. +To minimize RPC idle time, the tracker maintains a prefetch queue of up to `PREFETCH_SIZE` (10) blocks. For non-AuxPoW chains (Bitcoin, Litecoin), block hashes and hex data are fetched in two batch HTTP requests. For the AuxPoW chain (Dogecoin), the batch also fetches each block header, and the AuxPoW section is stripped from each block before parsing. ### Batch Writes -LevelDB writes are accumulated in an in-memory transaction (Map of put/del operations) and flushed atomically via `db.batch()` every 200 blocks or when the tracker reaches the chain tip (or when heap usage exceeds 2 GB). This minimizes write amplification and ensures all-or-nothing semantics; a crash mid-batch loses at most 200 blocks of progress, which are re-indexed on restart. +LevelDB writes are accumulated in an in-memory transaction (Map of put/del operations) and flushed atomically via `db.batch()` every 200 blocks or when the tracker reaches the chain tip (or earlier, when the V8 heap exceeds `HEAP_FLUSH_THRESHOLD_MB`: by default an eighth of the tracker's memory budget, kept between 256 MB and 2048 MB; see [Memory Budget](configuration.md#memory-budget)). This minimizes write amplification and ensures all-or-nothing semantics; a crash mid-batch loses at most 200 blocks of progress, which are re-indexed on restart. ## Reorg Handling diff --git a/components/utxo-tracker/configuration.md b/components/utxo-tracker/configuration.md index 6f5e5455..21feb04f 100644 --- a/components/utxo-tracker/configuration.md +++ b/components/utxo-tracker/configuration.md @@ -30,7 +30,7 @@ BULK_SYNC_RAM_BUDGET=768 | Variable | Description | Default | |---|---|---| -| `AUX_POW` | Enable AuxPoW block header stripping (required for Dogecoin and Litecoin HogEx blocks) | `undefined` (falsy) | +| `AUX_POW` | Legacy; read but ignored. AuxPoW block header stripping is chosen from `NETWORK` (Dogecoin only); Litecoin MWEB/HogEx handling is a separate per-transaction flag strip that no variable controls | `undefined` (falsy) | | `UTXO_TRACKER_API_KEY` | Bearer token required on all admin JSON-RPC methods (`getbootstrap`, `getbootstrapstatus`, `restorebootstrap`, `getbootstraprestorestatus`, `get_input_from_key_pattern`). When unset these methods fail closed (HTTP 401). Read-only UTXO/balance queries are unaffected. | `""` (disabled) | | `UTXO_MAX_PAGE_LIMIT` | Maximum page size a caller may request via `?limit=`. Caps a single request so a caller cannot trigger an OOM by requesting one giant page. Independent of `UTXO_MAX_ADDRESS_OUTPUTS`. | `10000` | | `UTXO_MAX_ADDRESS_OUTPUTS` | Hard ceiling on outputs materialized for a single-address unbounded query; above this limit `/utxos` and `get_balance` return HTTP 413: callers must page via `?limit=&after=` | `500000` | @@ -39,6 +39,7 @@ BULK_SYNC_RAM_BUDGET=768 | `XCHAIN_UNDO_BLOCKS_LTC` | Override the LTC reorg recovery window (blocks). The default is per network: 120 on mainnet and regtest, 5000 on testnet (litecoin testnet outran the previous 48 on 2026-09-01, and its deep public-testnet forks need the 5000-block class). | `120` (testnet `5000`) | | `XCHAIN_UNDO_BLOCKS_DOGE` | Override the DOGE reorg recovery window (blocks). 120 on every network. | `120` | | `UTXO_TRACKER_RATE_LIMIT_RPM` | API requests per minute per IP | `500` | +| `UTXO_TRACKER_TRUST_PROXY_HOPS` | Number of reverse proxies in front of the tracker, handed to Express `trust proxy` so the per-IP limiter keys on the right `X-Forwarded-For` entry. Above 10 logs an error and falls back to `0`. | `0` (trust no `X-Forwarded-For`) | | `UTXO_TRACKER_NODE_RPC_STALE_MS` | Staleness window for the tracker`s last usable node-tip read, after which `health` reports the node RPC stale. Five times the loop`s `BLOCKCHAIN_INFO_REFRESH_MS` (30s), so a slow or skipped poll never trips it and only a sustained outage does. | `150000` | | `UTXO_MAX_RPC_BATCH` | Maximum calls accepted in one inbound JSON-RPC batch (array body). The router runs `Promise.all` over every element, so without this cap a single unauthenticated ~100kb POST fans out into thousands of concurrent read scans and node RPCs. Mirrors the decoder and encoder batch guards. | `20` | | `XCHAIN_COINBASE_MATURITY` | Confirmations a coinbase output needs before `getUtxosAddress` will serve it as spendable. The consensus rule is **per chain**, and the tracker resolves it per network rather than assuming one number: 100 on BTC and LTC on every net; 240 on DOGE mainnet and testnet (Dogecoin Core sets `nCoinbaseMaturity` to 240 from the Digishield fork at height 145000, which every live DOGE chain is long past); 60 on DOGE regtest. Serving an immature coinbase hands the caller an input every node rejects. Set this only to override the resolved value for a test harness that mines short chains. | _(resolved per chain, see left)_ | diff --git a/components/vm/architecture.md b/components/vm/architecture.md index 9ebc79c3..84e937dd 100644 --- a/components/vm/architecture.md +++ b/components/vm/architecture.md @@ -54,8 +54,8 @@ flowchart TD | `state.js` | StateManager: reads from initial snapshot, tracks writes/deletes in dirty map, enforces key count, key size, and value size limits, provides `getChanges()` for result collection | | `collector.js` | EmissionCollector: queues emitted actions (with emission cap), collects debug logs (100 entries, 1 KB UTF-8 each, with byte-aware truncation) | | `validator.js` | ActionValidator: pre-validates emitted actions against the 21 allowed action types (`SEND`, `DESTROY`, `ISSUE`, `MINT`, `ORDER`, `DISPENSER`, `DIVIDEND`, `AIRDROP`, `CALLBACK`, `FILE`, `LIST`, `COINPAY`, `SWEEP`, `LINK`, `BROADCAST`, `MESSAGE`, `ATTEST`, `SLASH`, `EXECUTE`, `XCALL`, `VOTE`) and checks params shape | -| `lint-core.js` | Dependency-light (no isolated-vm) acorn-only consensus validation, exporting `lintSource()`, `analyzeContract()`, and per-rule detectors: `findBannedMathCalls`, `findBannedLiterals`, `findBannedAsync`, `findBannedGenerator` (`function*`/`yield`), `findBannedWasm` (global `WebAssembly` reference), `findBannedRest` (unmeterable rest positions), `findBannedWith` (the `with` statement), `findBannedExponentiation` (`**`/`**=`, hardened-only), `findBannedProtoMethods` (advisory, name-only match, never consensus), `findBannedStrippedGlobals` (advisory, reads of sandbox-deleted globals, never consensus), `findReservedControlBinding` (`CONTRACT_WRAPPER` bindings, hardened-only), and `codeSizeBytes`/`MAX_CODE_SIZE` for the 64KiB deploy cap. Under `VM_LINT_HARDENING`, the Math ban widens from the five named transcendentals to the complement of the `SAFE_MATH_MEMBERS` whitelist. Under `LINT_GLOBAL_ALIAS` (the `globalAlias` option), the banned-math, banned-async and banned-wasm detectors also treat sloppy-mode `this` and the `globalThis.globalThis…` self-reference chain as the global object. Under `LINT_OPTIONAL_CHAIN` (the `optionalChain` option), the same three detectors also look through a parenthesized optional chain, so `(globalThis?.Math).pow(2, 3)` reads as a Math call. Holds `CONSENSUS_RULES` (the 10-member set of deploy-blocking rule names: `invalid-type`, `unsupported-syntax`, `reserved-identifier`, `banned-math`, `banned-literal`, `banned-async`, `banned-generator`, `banned-rest`, `banned-wasm`, `banned-with`) and `lintSource()`. Its detectors and constants live in helper modules under `lint-core/` and are re-exported here. Vendored byte-for-byte into `xchain-sdk/src/contract/lint-core.js`; a CI SHA-256 parity guard fails on drift. | -| `syntax.js` | Deploy-time validation: V8 syntax check (throwaway isolate, the one step requiring isolated-vm; blocks via a separate early return and is not itself a `CONSENSUS_RULES` member), then delegates all acorn-coverable consensus rules to `lint-core.lintSource()`. Re-exports `findBannedMathCalls`, `findBannedLiterals`, `findBannedAsync`, `findBannedGenerator`, `findBannedWasm`, and `findBannedRest` for backward compatibility. `validateSyntax(code, opts)` threads ten independent consensus-gate flags, and a caller that leaves one out enforces that rule at every height: `enforceBannedAsync` (the `VM_BANNED_ASYNC` flag-day), `enforceLintHardening` (the `VM_LINT_HARDENING` flag-day, widening the Math ban and adding the exponentiation/reserved-binding/dynamic-import rules), `enforceLintGlobalAlias` (the per-coin [`LINT_GLOBAL_ALIAS_ACTIVATION`](../../protocol/protocol-activation.md#vm-gates-service-carried) height gate, armed at genesis on every network, making banned-math, banned-async and banned-wasm resolve sloppy-mode `this` and the `globalThis` self-reference chain as the global object), `enforceLintOptionalChain` (the per-coin [`LINT_OPTIONAL_CHAIN_ACTIVATION`](../../protocol/protocol-activation.md#vm-gates-service-carried) height gate, unarmed on mainnet, forwarded to lint-core as `optionalChain` so the banned-math, banned-async and banned-wasm rules look through a parenthesized optional chain such as `(globalThis?.Math).pow(2, 3)`), `enforceBannedGenerator`/`enforceBannedWasm` (two flags on the Package 3 per-coin sandbox height gate, active from genesis on testnet/regtest), `enforceBannedRest` (the separate [`REST_PATTERN_METER`](../../protocol/flag-days.md) gate, likewise active from genesis on testnet/regtest), `enforceBannedWith` (the `banned-with` rule on its own per-coin [`LINT_BANNED_WITH_ACTIVATION`](../../protocol/protocol-activation.md#where-the-values-live) height gate), `enforceLintNestingDepth` (the pre-parse nesting-depth rule on the per-coin `VM_LINT_NESTING_DEPTH_ACTIVATION` height gate, unarmed on public networks and active from genesis on regtest), and `enforceLintDestructure` (the per-coin [`LINT_DESTRUCTURE_ACTIVATION`](../../protocol/protocol-activation.md#where-the-values-live) height gate). Each defaults to `true` for author-facing callers so a from-genesis replay reproduces the historical deploy verdict at every activation. | +| `lint-core.js` | Dependency-light (no isolated-vm) acorn-only consensus validation, exporting `lintSource()`, `analyzeContract()`, and per-rule detectors: `findBannedMathCalls`, `findBannedLiterals`, `findBannedAsync`, `findBannedGenerator` (`function*`/`yield`), `findBannedWasm` (global `WebAssembly` reference), `findBannedRest` (unmeterable rest positions), `findBannedWith` (the `with` statement), `findBannedExponentiation` (`**`/`**=`, hardened-only), `findBannedProtoMethods` (advisory, name-only match, never consensus), `findBannedStrippedGlobals` (advisory, reads of sandbox-deleted globals, never consensus), `findReservedControlBinding` (`CONTRACT_WRAPPER` bindings, hardened-only), and `codeSizeBytes`/`MAX_CODE_SIZE` for the 64KiB deploy cap. Under `VM_LINT_HARDENING`, the Math ban widens from the five named transcendentals to the complement of the `SAFE_MATH_MEMBERS` whitelist. Under `LINT_GLOBAL_ALIAS` (the `globalAlias` option), the banned-math, banned-async and banned-wasm detectors also treat sloppy-mode `this` and the `globalThis.globalThis…` self-reference chain as the global object. Under `LINT_OPTIONAL_CHAIN` (the `optionalChain` option), the same three detectors also look through a parenthesized optional chain, so `(globalThis?.Math).pow(2, 3)` reads as a Math call. Under `LINT_DESTRUCTURE` (the `destructure` option), the same three detectors also catch a banned global read through an object destructuring pattern, such as `const { Promise } = globalThis` or `const { pow } = Math`. Holds `CONSENSUS_RULES` (the 11-member set of deploy-blocking rule names: `invalid-type`, `nesting-depth`, `unsupported-syntax`, `reserved-identifier`, `banned-math`, `banned-literal`, `banned-async`, `banned-generator`, `banned-rest`, `banned-wasm`, `banned-with`) and `lintSource()`. Its detectors and constants live in helper modules under `lint-core/` and are re-exported here. Vendored byte-for-byte into `xchain-sdk/src/contract/lint-core.js`; a CI SHA-256 parity guard fails on drift. | +| `syntax.js` | Deploy-time validation: first the pre-parse nesting-depth check (`nestingDepthFinding`, more than 64 nested delimiters is rejected) and syntax-depth check (`syntaxDepthFinding`, a syntactic depth over 200 is rejected, delimiter-free chains included), then the V8 syntax check (throwaway isolate, the one step requiring isolated-vm; blocks via a separate early return and is not itself a `CONSENSUS_RULES` member), then delegates all acorn-coverable consensus rules to `lint-core.lintSource()`. Re-exports `findBannedMathCalls`, `findBannedLiterals`, `findBannedAsync`, `findBannedGenerator`, `findBannedWasm`, and `findBannedRest` for backward compatibility. `validateSyntax(code, opts)` threads eleven independent consensus-gate flags, and a caller that leaves one out enforces that rule at every height: `enforceBannedAsync` (the `VM_BANNED_ASYNC` flag-day), `enforceLintHardening` (the `VM_LINT_HARDENING` flag-day, widening the Math ban and adding the exponentiation/reserved-binding/dynamic-import rules), `enforceLintGlobalAlias` (the per-coin [`LINT_GLOBAL_ALIAS_ACTIVATION`](../../protocol/protocol-activation.md#vm-gates-service-carried) height gate, armed at genesis on every network, making banned-math, banned-async and banned-wasm resolve sloppy-mode `this` and the `globalThis` self-reference chain as the global object), `enforceLintOptionalChain` (the per-coin [`LINT_OPTIONAL_CHAIN_ACTIVATION`](../../protocol/protocol-activation.md#vm-gates-service-carried) height gate, unarmed on mainnet, forwarded to lint-core as `optionalChain` so the banned-math, banned-async and banned-wasm rules look through a parenthesized optional chain such as `(globalThis?.Math).pow(2, 3)`), `enforceBannedGenerator`/`enforceBannedWasm` (two flags on the Package 3 per-coin sandbox height gate, active from genesis on testnet/regtest), `enforceBannedRest` (the separate [`REST_PATTERN_METER`](../../protocol/flag-days.md) gate, likewise active from genesis on testnet/regtest), `enforceBannedWith` (the `banned-with` rule on its own per-coin [`LINT_BANNED_WITH_ACTIVATION`](../../protocol/protocol-activation.md#where-the-values-live) height gate, unarmed on mainnet, armed on testnet at `BTC:testnet` 156230, `LTC:testnet` 4912984 and `DOGE:testnet` 67989993, and active from genesis on regtest), `enforceLintNestingDepth` (the pre-parse nesting-depth rule on the per-coin `VM_LINT_NESTING_DEPTH_ACTIVATION` height gate, unarmed on mainnet, armed on testnet at `BTC:testnet` 156230, `LTC:testnet` 4912984 and `DOGE:testnet` 67989993, and active from genesis on regtest), and `enforceLintDestructure` (the per-coin [`LINT_DESTRUCTURE_ACTIVATION`](../../protocol/protocol-activation.md#where-the-values-live) height gate, unarmed on mainnet, armed on testnet at `BTC:testnet` 156230, `LTC:testnet` 4912984 and `DOGE:testnet` 67989993, and active from genesis on regtest, forwarded to lint-core as `destructure`), and `enforceLintSyntaxDepth` (the 200-unit syntax-depth bound on the VM's per-coin `LINT_SYNTAX_DEPTH_ACTIVATION` map, map, unarmed on mainnet, armed on testnet at `BTC:testnet` 156908, `LTC:testnet` 4927400 and `DOGE:testnet` 68005165, and always active on regtest; when on, it also forces the delimiter nesting-depth pre-check on). Each defaults to `true` for author-facing callers so a from-genesis replay reproduces the historical deploy verdict at every activation. | | `errors.js` | ContractRevertError (thrown by `revert()`/`require()`) and GasExhaustedError (thrown when gas ceiling exceeded) | ## JSON Bridge Protocol @@ -158,6 +158,7 @@ The V8 isolate provides hardware-level isolation (separate heap, no shared objec - Concurrency: `SharedArrayBuffer`, `Atomics` - Big integers: `BigInt` (super-linear native cost, unmeterable by the AST meter; use `xchain.math.*` instead; `BigInt` literals also rejected at deploy time) - WebAssembly: `WebAssembly` (consensus-gated: stripped at/after the Package 3 per-coin sandbox height gate, active from genesis on testnet/regtest; present on earlier blocks for replay fidelity. A wasm body carries no `__gas` instrumentation, so it would run unmetered native code; the `banned-wasm` deploy-lint rule also rejects references to the global) +- BigInt typed arrays and accessors: `BigInt64Array`, `BigUint64Array`, and the `DataView.prototype` methods `getBigInt64`, `getBigUint64`, `setBigInt64` and `setBigUint64` (set to `undefined`), with `BigInt.prototype.constructor` neutered alongside them (consensus-gated: stripped wherever the Package 3 per-coin sandbox height gate above is active, so from genesis on testnet/regtest and at/after the Package 3 heights on mainnet; the separate `BIGINT_SURFACE_STRIP_ACTIVATION` map also switches the same strip on, and has no mainnet height set. Present on earlier blocks for replay fidelity. These constructors and accessors keep the native BigInt surface after the `BigInt` global is removed, so they would reopen the unmetered-CPU path; use `xchain.math.*` instead) - Regex: `RegExp` (set to `undefined`; prevents catastrophic backtracking / ReDoS) - Locale/time: `Intl`, `Temporal`, `structuredClone`, `performance` - Eval/constructors: `eval`, `Function` (global reference set to `undefined`) diff --git a/components/vm/configuration.md b/components/vm/configuration.md index 932022dd..3a76d64d 100644 --- a/components/vm/configuration.md +++ b/components/vm/configuration.md @@ -101,9 +101,9 @@ These limits are hardcoded in the VM and not configurable: |---|---|---| | Log entries per execution | 100 | `collector.js` | | Log entry size | 1,024 bytes UTF-8 (truncated with `...(truncated)` marker) | `collector.js` | -| Return value size | 65,536 UTF-16 code units (truncated) | `index.js` | +| Return value size | 65,536 UTF-16 code units (truncated) | `index/runtime/contract_io.js` | | Throwaway isolate memory | 8 MB | `isolate.js`, `syntax.js` | -| Binary expression metering depth | 10 | `metering.js` | +| Binary expression metering depth | 10 | `metering/visitor_instrumentation.js` | The return-value cap is measured in UTF-16 code units, with `String.length` over the serialized value, not in UTF-8 bytes: a non-ASCII return at the cap can occupy up to roughly 196,608 UTF-8 bytes, so size a return against code units rather than the encoded size. Truncation is a UTF-16 `substring`, so a value cut in the middle of a surrogate pair ends with an unpaired surrogate. The log-entry limit directly above is genuinely byte-measured and is the contrasting case. diff --git a/components/vm/operations.md b/components/vm/operations.md index 830e4b5f..ccf75a48 100644 --- a/components/vm/operations.md +++ b/components/vm/operations.md @@ -144,7 +144,11 @@ The indexer uses database savepoints to ensure these guarantees extend to the pe ## Syntax Validation (Deploy-Time) -Before a contract is deployed, `vm.validateSyntax(code, opts)` runs the following checks in order. The indexer passes the per-block activation flags that `resolveLintFlags` in `deploy/lint.js` resolves for the DEPLOY's block. Any flag omitted from `opts` defaults on, which is the author-facing preflight behavior rather than the replay-safe deploy verdict, so a consensus caller must pass each gate it resolves. The V8 syntax check blocks a deploy via a separate early return and is not itself a `lint-core.CONSENSUS_RULES` member; checks 2-10 below are all deploy-blocking consensus rules (`invalid-type`, `unsupported-syntax`, `reserved-identifier`, `banned-math`, `banned-literal`, `banned-async`, `banned-generator`, `banned-rest`, `banned-wasm`, `banned-with`), and all must pass or the DEPLOY action is rejected: +Before a contract is deployed, `vm.validateSyntax(code, opts)` runs the following checks in order. The indexer passes the per-block activation flags that `resolveLintFlags` in `deploy/lint.js` resolves for the DEPLOY's block. Any flag omitted from `opts` defaults on, which is the author-facing preflight behavior rather than the replay-safe deploy verdict, so a consensus caller must pass each gate it resolves. The V8 syntax check blocks a deploy via a separate early return and is not itself a `lint-core.CONSENSUS_RULES` member. The nesting-depth pre-check and checks 2-10 below are the 11 deploy-blocking consensus rules (`invalid-type`, `nesting-depth`, `unsupported-syntax`, `reserved-identifier`, `banned-math`, `banned-literal`, `banned-async`, `banned-generator`, `banned-rest`, `banned-wasm`, `banned-with`), and all must pass or the DEPLOY action is rejected. + +Before check 1, a **nesting-depth pre-check** (consensus-gated, its own per-coin `VM_LINT_NESTING_DEPTH_ACTIVATION` height gate, `enforceLintNestingDepth`) tokenizes the source and rejects it once its `(`, `[`, `{` and `${` delimiters nest more than 64 deep (`MAX_NESTING_DEPTH`), with `nesting depth exceeds limit (64) at line N`. Delimiters inside comments, strings, regular expressions and template text do not count. The gate is unarmed on BTC, LTC and DOGE mainnet, armed on testnet at a per-chain local block height (`BTC:testnet` 156230, `LTC:testnet` 4912984, `DOGE:testnet` 67989993), and genesis-active on regtest; the `xchain-lint` CLI and the SDK linter enforce it by default. + +After it, a **syntax-depth pre-check** (`enforceLintSyntaxDepth`) rejects source whose syntactic depth exceeds 200 (`MAX_SYNTAX_DEPTH`), with `syntax depth exceeds limit (200) at line N`, reported under the same `nesting-depth` rule, so the deploy-blocking rule count does not change. The depth counts every construct the acorn parsers and the metering pass recurse on, including delimiter-free shapes (unary and binary operator chains, ternary and `else if` chains, chained assignment) that would otherwise drive those passes into a host stack overflow. When `enforceLintSyntaxDepth` is on, the delimiter pre-check above also runs whatever `enforceLintNestingDepth` says. The VM's per-coin gate map for this bound is `LINT_SYNTAX_DEPTH_ACTIVATION` (resolver `isLintSyntaxDepthActive`), which also gates the execute-time lint: it is unarmed on mainnet, armed on testnet at `BTC:testnet` 156908, `LTC:testnet` 4927400 and `DOGE:testnet` 68005165, and always active on regtest. Like every option here, `enforceLintSyntaxDepth` defaults on when omitted, and the `xchain-lint` CLI and the SDK linter enforce it by default. 1. **V8 syntax check**: compiles the code in a throwaway 8 MB isolate to catch syntax errors (the only step requiring `isolated-vm`) 2. **Acorn metering pass**: runs `meterCode()` to ensure acorn can parse the source (effective ES2020 ceiling) @@ -155,11 +159,12 @@ Before a contract is deployed, `vm.validateSyntax(code, opts)` runs the followin 7. **Banned generator check** (consensus-gated, Pkg 3 sandbox): rejects `function*`, generator methods, and any `yield`; live from genesis on testnet/regtest. 8. **Banned rest-pattern check** (consensus-gated, its own [`REST_PATTERN_METER`](../../protocol/flag-days.md) gate, not the Pkg 3 one): rejects a rest pattern in the four positions the metering transform cannot charge, because it charges a rest destructure by wrapping the source expression and these have none: a rest parameter in a function parameter list, a nested rest inside a destructuring pattern, a catch-clause rest, and a rest in a `for-of`/`for-in` loop head. Live from genesis on testnet/regtest, and on mainnet at/after that gate's block time; the `xchain-lint` CLI and the SDK linter enforce it today by default. 9. **Banned WebAssembly check** (consensus-gated, Pkg 3 sandbox): rejects any reference to the global `WebAssembly`, bare or [global-object-qualified](#global-object-spellings); live from genesis on testnet/regtest. -10. **Banned `with` statement check** (consensus-gated, its own per-coin [`LINT_BANNED_WITH_ACTIVATION`](../../protocol/protocol-activation.md#where-the-values-live) height gate, `enforceBannedWith`, not the Pkg 3 one): rejects any `with` statement. A `with` block resolves free identifiers against an arbitrary object at runtime, which bypasses the identifier-based deploy bans and the metering rewrite; read the properties through the object instead. Live from genesis on regtest and unarmed on mainnet and testnet until an operator arms a height; the `xchain-lint` CLI and the SDK linter enforce it today by default. +10. **Banned `with` statement check** (consensus-gated, its own per-coin [`LINT_BANNED_WITH_ACTIVATION`](../../protocol/protocol-activation.md#where-the-values-live) height gate, `enforceBannedWith`, not the Pkg 3 one): rejects any `with` statement. A `with` block resolves free identifiers against an arbitrary object at runtime, which bypasses the identifier-based deploy bans and the metering rewrite; read the properties through the object instead. Unarmed on BTC, LTC and DOGE mainnet, armed on testnet at a per-chain local block height (`BTC:testnet` 156230, `LTC:testnet` 4912984, `DOGE:testnet` 67989993), and live from genesis on regtest; the `xchain-lint` CLI and the SDK linter enforce it today by default. ```mermaid flowchart TD START["DEPLOY action: code"] + S0{"Nesting-depth and syntax-depth pre-checks (VM_LINT_NESTING_DEPTH_ACTIVATION, LINT_SYNTAX_DEPTH_ACTIVATION)"} S1{"1. V8 syntax check (throwaway isolate)"} S2{"2. Acorn metering pass"} S3{"3. Reserved identifier check"} @@ -173,7 +178,9 @@ flowchart TD ACCEPT["DEPLOY accepted"] REJECT["DEPLOY rejected"] - START --> S1 + START --> S0 + S0 -->|"nesting deeper than 64, gate active"| REJECT + S0 -->|"within the limit, or gate not active"| S1 S1 -->|"syntax error"| REJECT S1 -->|"compiles"| S2 S2 -->|"parse failure"| REJECT @@ -204,6 +211,8 @@ Checks 4, 6 and 9 match a banned global read through the global object as well a Under the [`LINT_OPTIONAL_CHAIN_ACTIVATION`](../../protocol/protocol-activation.md#vm-gates-service-carried) gate (`enforceLintOptionalChain`), checks 4, 6 and 9 also see through a parenthesized optional chain, so `(globalThis?.globalThis).Promise`, `(this?.globalThis)?.WebAssembly` and `(globalThis?.Math).pow(2, 3)` are rejected exactly like their plain spellings. Below the gate, those three spellings lint clean as they always have: on mainnet, and on testnet below that chain's height. An unparenthesized `globalThis?.Promise` is already rejected on every network and does not depend on the gate. The gate is unarmed on BTC, LTC and DOGE mainnet, armed on testnet at each chain's own local block height (`BTC:testnet` 155001, `LTC:testnet` 4906040, `DOGE:testnet` 67962387), and genesis-active on regtest. The contract simulator's deploy gate and the execute-time lint read the same per-coin arming as the indexer's deploy verdict, so they agree with it and never enforce the rule below a chain's height. +Under the [`LINT_DESTRUCTURE_ACTIVATION`](../../protocol/protocol-activation.md#where-the-values-live) gate (`enforceLintDestructure`), checks 4, 6 and 9 also catch a banned global read through an object destructuring pattern, in a declaration or an assignment. `const { pow } = Math`, `const { Math: { pow } } = globalThis`, `const { Promise } = globalThis`, `({ Promise: P } = this)` and `const { WebAssembly } = globalThis` are rejected exactly like `Math.pow`, `Promise` and `WebAssembly`, while a destructured SafeMath member such as `const { floor } = Math` stays legal. The gate is unarmed on BTC, LTC and DOGE mainnet, armed on testnet at each chain's own local block height (`BTC:testnet` 156230, `LTC:testnet` 4912984, `DOGE:testnet` 67989993), and genesis-active on regtest. Below it, the deploy verdict accepts those spellings, while the `xchain-lint` CLI and the SDK linter reject them by default. + ## Troubleshooting ### isolated-vm won't compile diff --git a/components/wallet/README.md b/components/wallet/README.md index 89e6077e..9b513d2f 100644 --- a/components/wallet/README.md +++ b/components/wallet/README.md @@ -5,42 +5,43 @@ ## What is xchain-wallet -xchain-wallet is the reference self-custodial wallet for the XChain Platform. It runs as a browser web app, a Chrome MV3 extension (popup + full-screen + side panel), a desktop application (Windows / macOS / Linux), and a Capacitor mobile app (Android now, iOS later), all from a single React codebase published as a pnpm workspace. The wallet supports Bitcoin, Litecoin, and Dogecoin at launch, with additional chains added as the platform adds them. It consumes [xchain-sdk](../sdk/) as its only data and signing layer and never duplicates SDK functionality. +xchain-wallet is the reference self-custodial wallet for the XChain Platform. It runs as a browser web app, a Chrome MV3 extension (popup + full-screen + side panel), a desktop application (Windows / macOS / Linux), and a Capacitor mobile app with Android and iOS projects, all from a single React codebase organized as a pnpm workspace. The wallet supports Bitcoin, Litecoin, and Dogecoin, with additional chains loaded from the hub's signed chain registry. It uses [xchain-sdk](../sdk/) for protocol actions, transaction construction, and chain services, while wallet-owned signer implementations keep signing keys either encrypted in the local vault or on a hardware device. -The wallet implements every XChain feature exposed by the platform: all 32 user-encodable ACTION types, a built-in DEX surface, encrypted messaging (ECIES / ECDH / AES), smart contracts, BTC staking + delegation, classical n-of-m + MuSig2 multisig, cross-chain flows, dispensers, parimutuel betting markets, on-chain governance voting, a `window.xchain` dApp bridge, and air-gapped PSBT signing via animated QR transport. +The wallet exposes all 32 current user-encodable ACTION types, a built-in DEX surface, ECIES and ECDH encrypted messaging, smart contracts, staking and delegation, classical n-of-m and MuSig2 multisig coordination, cross-chain flows, dispensers, parimutuel betting markets, on-chain governance voting, an extension-provided `window.xchain` dApp bridge, and air-gapped PSBT signing via animated QR transport. ## Features -- **Four shells, one codebase**: `@xchain-wallet/web` (Vite SPA, mobile-responsive), `@xchain-wallet/extension` (Chrome MV3 popup + full-screen + side panel + service worker), `@xchain-wallet/desktop` (Electron, main-process signing isolation), `@xchain-wallet/mobile` (Capacitor wrapper of the built web SPA; Android shipped, iOS later); all share `@xchain-wallet/core` for routes, components, flows, and signers +- **Four shells, one codebase**: `@xchain-wallet/web` (Vite SPA, mobile-responsive), `@xchain-wallet/extension` (Chrome MV3 popup + full-screen + side panel + service worker), `@xchain-wallet/desktop` (Electron with main-process vault and signing-host isolation), `@xchain-wallet/mobile` (Capacitor wrapper of the built web SPA with Android and iOS native projects); all use `@xchain-wallet/core` for routes, components, flows, and signer contracts - **All 32 user-encodable ACTIONs** (every action has an authoring form; `PROTOCOL_ONLY_ACTIONS` is empty): ADDRESS, AIRDROP, BATCH, BET, BROADCAST, CALLBACK, COINPAY, COLLECT, DELEGATE, DEPLOY, DEPOSIT, DESTROY, DISPENSER, DIVIDEND, EXECUTE, FILE, ISSUE, LINK, LIST, MESSAGE, MINT, ORDER, PRICE, SEND, SLEEP, STAKE, SWAP, SWEEP, UNSTAKE, VOTE, WITHDRAW, XBRIDGE. COLLECT is BTC-only; the other 31 are offered on all three chains - **Self-custodial key management**: BIP39 mnemonic + optional 25th-word passphrase (captured once at setup and stored encrypted alongside the mnemonic), BIP32 HD derivation per chain, AES-256-GCM vault encrypted with an Argon2id-derived master key (calibrated per device), Counterwallet legacy mnemonic import -- **Pluggable signer interface**: four concrete signers, `SoftwareSigner` (in-vault keys), `TrezorSigner` (Trezor Connect), `LedgerSigner` (WebHID), and `RemoteSigner` (cross-shell pairing). Multisig (classical n-of-m + MuSig2) is orchestrated by flows over these signers rather than by a signer of its own; a dedicated `MultisigSigner` is planned but not implemented +- **Pluggable signer interface**: four concrete signers, `SoftwareSigner` (in-vault keys), `TrezorSigner` (the hosted Trezor Connect build in the web and desktop shells), `LedgerSigner` (WebHID), and `RemoteSigner` (an in-shell bridge that lets background flows reach a Trezor or Ledger signer running in the UI; it does not pair separate devices or shells). The MV3 extension does not load Trezor's remote script, and store-profile mobile builds expose no hardware-signer path. Multisig (classical n-of-m + MuSig2) is coordinated by flows over the available signers; a dedicated `MultisigSigner` class is not implemented - **Token issuance suite**: issue / mint / destroy / distribute / dividend / dispenser / broadcast / airdrop / sweep, with parsed-recipient previews and dry-run review - **DEX surface**: markets list, market view with lightweight-charts price chart, place-order panel, orderbook, recent trades, open orders, trade history -- **Encrypted messaging inbox**: ECIES (default, multi-device), ECDH (session), AES (pre-shared); compose flow handles pubkey lookup + encryption + sign + broadcast in one step -- **Smart contracts**: deploy from source, execute methods, deposit and withdraw, contracts-list / contract-detail explorer views, gas estimation, ContractClient bindings -- **BTC staking + delegation**: stake / unstake / delegate (rotate or revoke) / collect rewards; staking dashboard, delegation form, operator dashboard -- **Cross-chain flows**: cross-chain swap form, cross-chain templates, parallel composer for atomic multi-chain submission, per-chain SDK registry -- **Multisig coordinator**: create n-of-m configs, paste-inbox for partial PSBTs, AnimatedQrFrames PSBT-QR transport, session state machine, MuSig2 round labels, per-address multi-config support, camera scanner -- **dApp bridge (`window.xchain`)**: typed bridge spec at `@xchain-wallet/bridge-spec`: connect, getAccounts, getBalances, signMessage, signPsbt, signAction, signIn (Sign-In with XChain), event subscriptions; per-origin permission grants enforced in the extension service worker and desktop main process +- **Encrypted messaging inbox**: ECIES (default, multi-device) and ECDH (shared-session) compose and decrypt flows; the inbox recognizes AES pre-shared-key records but has no AES key-entry or send surface. The compose flow handles pubkey lookup, encryption, signing, and broadcast +- **Smart contracts**: deploy from source, execute methods through the manual form or ABI-derived named and typed inputs, deposit and withdraw, contracts-list / contract-detail explorer views, and gas-limit validation and suggestions +- **Staking + delegation**: STAKE / UNSTAKE / DELEGATE contract versions are offered across supported chains; validator and capability versions plus COLLECT rewards are Bitcoin-only. The wallet includes the staking dashboard, delegation form, and operator dashboard +- **Cross-chain flows**: single-fill cross-chain swaps, federation-matched cross-chain orders, cross-chain templates, a parallel composer for multi-chain submission, a per-chain SDK registry, and link / swap coordinators +- **Multisig coordinator**: create n-of-m configs, paste-inbox for partial PSBTs, AnimatedQrFrames PSBT-QR transport, session state machine, MuSig2 round labels, schema-v2 per-address multi-config support, camera scanner, and cosigner account provisioning +- **dApp bridge (`window.xchain`)**: the extension injects a provider typed by `@xchain-wallet/bridge-spec`: connect, getAccounts, getBalances, signMessage, signPsbt, signAction, signIn (Sign-In with XChain), and event subscriptions; the extension service worker enforces per-origin permission grants - **Air-gapped PSBT signing**: BIP21 / multisig PSBT envelope / chunked PSBT-QR; QR scanner + AnimatedQrFrames for offline cosigner round-trips; `prefers-reduced-motion` honored with manual frame stepping - **Sign-screen safety rails**: plain-English action decoder shows `to` / `amount` / `asset` as the user typed them, even if the encoder fabricates output; per-action expectation summaries; explicit user confirmation - **Onboarding and recovery**: create / import / Counterwallet-migrate / dry-run-restore / discover-used-addresses gap-limit scan; view-private-key + export-WIF gated by a "Before you continue" warning in an unlocked session (no password re-entry; a locked wallet must be unlocked first) -- **Lock / unlock / auto-lock**: Argon2id-derived session key cached in `chrome.storage.session` (extension) or in-memory (web/desktop); foreground auto-lock; manual lock action; OS keychain auto-unlock on desktop -- **i18n + a11y**: string registry under `core/src/i18n`; static a11y audit gate (button label / img alt / input label / textarea label / div-onclick role+tabIndex) blocks regressions in CI; WCAG 2.2 AA target for the external audit -- **Reproducible builds**: Level-2 reproducibility of the pre-signing Linux desktop bundle: digest-pinned base image, frozen lockfile, `SOURCE_DATE_EPOCH` from `git log`, `RELEASE_HASHES.txt` SHA-256 manifest, 18-rule static repro-build audit -- **URI scheme handling**: registered handlers for `bitcoin:`, `dogecoin:`, `litecoin:`, and `xchain:` URIs across all three shells +- **Lock / unlock / auto-lock**: Argon2id-derived session key cached in `chrome.storage.session` together with a signing-capable copy of the password (extension), an OS-keychain-encrypted file plus an in-memory copy (desktop when a keychain backend is available), or memory only (web and desktop without a keychain); foreground auto-lock, background auto-lock backstops in the extension and desktop, a manual lock action, and OS keychain auto-unlock on desktop +- **i18n + a11y**: string registry under `packages/core/src/i18n`; static a11y audit gate (button label / img alt / input label / textarea label / div-onclick role+tabIndex) blocks regressions in CI; WCAG 2.2 AA target for the external audit +- **Reproducible builds**: Level-2 reproducibility of the pre-signing Linux desktop bundle: digest-pinned base image, frozen lockfile, `SOURCE_DATE_EPOCH` from `git log`, `RELEASE_HASHES.txt` SHA-256 manifest, and a 19-rule static repro-build audit +- **URI scheme handling**: desktop and mobile register handlers for `bitcoin:`, `dogecoin:`, `litecoin:`, and `xchain:` URIs; the web shell registers `web+xchain:` and attempts `xchain:` where the browser allows it; the extension registers `web+xchain:` - **Automatic Donation System (ADS)**: opt-out micro-donation appended to each submitted action; consent captured at onboarding; configurable per chain in Settings; invisible in normal use with no extra line on sign screens -- **Connected sites + permissions**: per-origin grant store, revocable from Settings, surfaced in the approval popup before every privileged action +- **Connected sites + permissions**: the extension keeps a per-origin grant store, revocable from Settings, and surfaces approval UI for privileged dApp requests ## Documentation | Document | Description | |---|---| -| [Architecture](architecture.md) | Repo layout, package boundaries, state flow, three-shell model, core/web/extension/desktop seams | -| [Keys & Signing](keys-signing.md) | BIP39 + passphrase, HD derivation, vault encryption, signer interface, software / Trezor / Ledger / Remote / Multisig | +| [Architecture](architecture.md) | Repo layout, package boundaries, state flow, four-shell model, core/web/extension/desktop/mobile seams | +| [Keys & Signing](keys-signing.md) | BIP39 + passphrase, HD derivation, vault encryption, software / Trezor / Ledger / Remote signers, multisig coordination | | [Security & Threat Model](security.md) | Protected assets, in-scope and out-of-scope threats, sign-screen safety rails, audit posture | | [UX Surfaces](ux.md) | Onboarding, lock/unlock, balances, history, send/receive, sign screens, contacts, QR scanner, command palette, settings | +| [UI Style Guide](ui-style-guide.md) | Contributor rules for page layout, responsive control sizing, vertical rhythm, templates, and shared UI components | | [Features](features.md) | Token issuance, DEX, messaging, dispensers, contracts, staking, multisig, cross-chain: surface-by-surface | | [Bridge](bridge.md) | `window.xchain` dApp bridge: connect, signMessage, signPsbt, signAction, signIn, events, error codes | | [URI Schemes](uri-schemes.md) | BIP21 + chain URIs + multisig PSBT envelope + chunked PSBT-QR transport | @@ -60,7 +61,7 @@ The wallet implements every XChain feature exposed by the platform: all 32 user- ## Installation -The wallet is a pnpm workspace. Clone the repo and install workspace dependencies: +The wallet is a pnpm 11 workspace and requires Node.js 22, the version the repo's `.nvmrc` pins. Clone the repo and install workspace dependencies: ```bash git clone https://github.com/XChain-Platform/xchain-wallet.git @@ -68,7 +69,7 @@ cd xchain-wallet pnpm install ``` -The repository depends on a sibling `xchain-sdk` checkout, both `packages/web/package.json` and `packages/extension/package.json` link `xchain-sdk` from `../../../xchain-sdk`. Clone [xchain-sdk](https://github.com/XChain-Platform/xchain-sdk) next to `xchain-wallet` before installing. +No sibling SDK checkout is required. The web, extension, and desktop packages install `npm:@dankest-llc/xchain-sdk@0.22.4` under the local package name `xchain-sdk`, and the lockfile pins the release dependency. For simultaneous SDK development, `pnpm run sdk:link` links a checkout at `../xchain-sdk` by default, `pnpm run sdk:status` reports what each shell resolves, and `pnpm run sdk:unlink` restores the published package. ## Quick Start @@ -78,7 +79,7 @@ The repository depends on a sibling `xchain-sdk` checkout, both `packages/web/pa pnpm --filter @xchain-wallet/web dev ``` -Vite serves the wallet at `http://localhost:5173`. The web shell is mobile-responsive, open it on a phone for the mobile UX. A native Android app, the Capacitor wrapper of this web build, has shipped (`v0.336.0`); a native iOS app is still on the post-launch roadmap. +Vite serves the wallet at `http://localhost:5173`. The web shell is mobile-responsive; open it on a phone for the mobile UX. The `@xchain-wallet/mobile` package wraps the same production web build and contains the native Android and iOS projects. ### Build the Chrome extension @@ -100,28 +101,28 @@ Builds the renderer and launches Electron locally. For packaged releases, see [B ### Browser web app -The web SPA is the lowest-friction entry point: visit a hosted instance, create or import a wallet, and sign locally in the browser. Master keys never leave the browser tab; vault ciphertext is persisted in IndexedDB. The web shell is mobile-responsive. +The web SPA is the lowest-friction entry point: visit a hosted instance, create or import a wallet, and sign locally in the browser. Master keys never leave the browser page; vault ciphertext is persisted in IndexedDB. The web shell is mobile-responsive. -Web shell key isolation is fundamentally weaker than the extension's, same-origin scripts on `wallet.xchain.io` could in principle access in-memory key material. Mitigations: short session lifetime (in-memory only; refresh = re-locked), no master key in `sessionStorage`, no third-party `