From b6110eee7a7b86f75c15bca07c0562fe06e3bd16 Mon Sep 17 00:00:00 2001 From: Benjamen Keroack Date: Thu, 15 Oct 2026 21:58:25 -0600 Subject: [PATCH] electrum: hash the protocol's txid:height: status preimage Confirmed rows hashed txid:height:blockhash:, so the status no longer matched the one the Electrum protocol defines. Electrum desktop recomputes the status from get_history (synchronizer.py history_status). On a mismatch it does not store the history, and after the network timeout it raises SynchronizerFailure and disconnects. Every address with a confirmed transaction never synced. The status is now SHA-256 of tx_hash:height: for each get_history row. A same-height replace that keeps every (txid, height) keeps the status and pushes nothing; chain_tip on responses still marks it. The asof journey pins that instead of a changed status, and the row-order pin recomputes the protocol preimage. Fixes #1028 Co-Authored-By: Claude Opus 5.5 --- COMPAT.md | 4 +- TESTING.md | 2 +- changelog.d/electrum-protocol-status.md | 3 + crates/rbitcoin-electrum/src/server.rs | 51 +++------------ crates/rbitcoin-electrum/src/server_tests.rs | 64 +++++-------------- .../rbitcoin-test/tests/electrum_protocol.rs | 49 +++++++------- 6 files changed, 57 insertions(+), 116 deletions(-) create mode 100644 changelog.d/electrum-protocol-status.md diff --git a/COMPAT.md b/COMPAT.md index 004f952b8..f48ac73f3 100644 --- a/COMPAT.md +++ b/COMPAT.md @@ -134,7 +134,7 @@ Per-method notes, auth, and the shindex matrix live in | blockchain.tweaks.subscribe | done | **Stream**, not a one-shot result: JSON-RPC `result` is the **first** height only; remaining heights are unsolicited notifications; `{"message":"done"}` ends the **chunk** (60s wall at a wave boundary, or the requested `count` if sooner) — Cake electrs clamps `count` to 1000 instead. Cake Wallet resubscribes from `syncHeight+1` ([cake_wallet#3574](https://github.com/cake-tech/cake_wallet/issues/3574) persist still lags one event). [kiss-bdk](https://github.com/kkdao/kiss-bdk) today treats `done` as the asked range finished ([kiss-bdk#10](https://github.com/kkdao/kiss-bdk/issues/10)). Each height carries tweak + txid + taproot `output_pubkeys` (client scans locally; no block fetch). Naive walk, or `--sp-tweaks` thin index (`len:tweak` only; one `txout` span per wave; the first height shares that span when indexed). Pre-taproot: **one** notify with ≤1024 empty height keys (Cake `fromJson` last key is progress); probe `[0,1,false]` stays `{"0": {}}`. Param `[2]` (Cake `historicalMode`): `false` **cut-through** (omit confirmed-spent P2TR outs / txs); `true` keeps spent outs. Serve-time `--sp-tweaks-dust` (default **1000**) omits P2TR outs with `value <=` the floor (`0` = all; **546** matches Cake electrs `sp_min_dust`). Sparrow Silent Payments uses Frigate `blockchain.silentpayments.subscribe` (server-side scan) — **not** this method. Cake isolate may still hardcode `electrs.cakewallet.com`. | | headers / block headers | done | Tip push on subscribe | | scripthash history / balance / listunspent | done | Unconf when mempool attached; `get_history` optional BCH-style `from_height` / exclusive `to_height` (`-1` = tip + mempool); 1-arg = full history; unconfirmed `get_history` rows include `fee` (same as `get_mempool`); `listunspent` mempool height is `0` or `-1` (unconfirmed parent); **subscribe status always full**; `blockchain.scripthash.unsubscribe` returns whether the connection was watching (frees a slot under `--electrum-max-subs`, default 10000 per connection). `listunspent` loads `txid.body` only for unspent creates; one TCP connection reuses the last SH outs+spent join until SH-view **hash** changes. Confirmed methods stamp live tip: a RAM SH head (pending write-behind) joins with durable SH so mempool can drop confirmed txs without a hole. Durable Class B seed waits until after tip announce. `get_history` skips mempool rows already in the confirmed list. `server.features.chain_tip = true`. Trailing **`asof:`** after the official args (`server.features.asof` / `asof_protocol = 1.4.2-asof`): confirmed rows as of that still-live ancestor **at or behind visible SH** (durable + pending), **no** mempool; stamp is the asof block; unknown hash or ahead of visible SH → `asof not on chain`. Prefix keeps it off the future positional-string landmine. Requires negotiated `1.4.2-asof` (first `server.version` only). Electrum `protocol_max` is **1.6**. | -| scripthash.get_mempool / subscribe | done | Status on mempool announce **and** when SH applies a height that creates or spends the hash (posting-list probe; no Class A expand on a miss). Headers subscribe still live tip. Reorg (`TipNotify.reorg_from_height`) restatuses every watch even if the new block misses the script. Status preimage is `txid:height:blockhash:` for confirmed rows (mempool rows stay `txid:height:`). Row **order** is confirmed height-asc then mempool tail, same as `get_history`. A script with no history has status `null` (subscribe reply and pushes), never `""`: wallets read any non-null status as a used address. `unsubscribe` is implemented. | +| scripthash.get_mempool / subscribe | done | Status on mempool announce **and** when SH applies a height that creates or spends the hash (posting-list probe; no Class A expand on a miss). Headers subscribe still live tip. Reorg (`TipNotify.reorg_from_height`) restatuses every watch even if the new block misses the script. Status preimage is the protocol's `txid:height:` for every row: Electrum desktop recomputes it from `get_history` and drops the history on a mismatch. A same-height replace that keeps every `(txid, height)` keeps the status; the `chain_tip` stamp carries that A-B-A. Row **order** is confirmed height-asc then mempool tail, same as `get_history`. A script with no history has status `null` (subscribe reply and pushes), never `""`: wallets read any non-null status as a used address. `unsubscribe` is implemented. | | transaction.get / get_merkle | done | get falls back to mempool unless `asof:`; confirmed responses stamp `chain_tip`. Verbose get is electrs-shaped: `time`/`blocktime`/`confirmations`/`blockhash` from the confirming header (mempool verbose is `confirmations: 0` with no block stamp), plus `vin`/`vout`/`size`/`version`/`locktime`/`hash` (wtxid) decoded from the hex already loaded. Trailing `asof:` (same dialect as scripthash): get returns the tx only if confirmed at or behind that ancestor (**no** mempool); get_merkle rejects a `height` above the asof pin (`asof not on chain`). `id_from_pos` is a txid string; third arg `merkle=true` returns `{tx_hash, merkle}` (same walk as `get_merkle`). | | transaction.broadcast | done | Mempool accept + P2P inv. `broadcast_package` wraps `accept_package` (Electrum **1.6**; P2P BIP331 still Q-48). | | relayfee / estimatefee / histogram / `mempool.get_info` | done | `minrelaytxfee` is the configured minimum. `relayfee` and `mempoolminfee` are the enforced floor. `incrementalrelayfee` stays 100 sat/kvB. Estimates are the 10-minute flow/history product ([`docs/mempool-fee-estimation.md`](./docs/mempool-fee-estimation.md)). `mempool.get_info` is 1.6 (`minrelaytxfee` replaces `relayfee` for 1.6 clients; `relayfee` stays for 1.4). | @@ -193,7 +193,7 @@ the published tip and retry if it disconnects | Surface | Token | Body | |---------|--------|------| | Esplora HTTP | `X-Bitcoin-Chain-Tip` + `X-Bitcoin-Chain-Tip-Height` | Unchanged JSON. Client: if two sequential fetches disagree on the hash, drop the batch and restart. | -| Electrum TCP | JSON-RPC extra members `chain_tip` / `chain_tip_height` next to `result` (ping/version omit). `server.features.chain_tip`. | `result` shape unchanged. Status preimage includes confirming `blockhash` so subscribe clients refetch on same-height replace. Notification `params` stay `[scripthash, status]`. | +| Electrum TCP | JSON-RPC extra members `chain_tip` / `chain_tip_height` next to `result` (ping/version omit). `server.features.chain_tip`. | `result` shape unchanged. Status preimage stays the protocol's `txid:height:`; a same-height replace shows in `chain_tip`, not in the status. Notification `params` stay `[scripthash, status]`. | We stamp **tip**, not only the last relevant history tx hash (empty history and list envelopes still need a token). diff --git a/TESTING.md b/TESTING.md index c42fe25da..482774d23 100644 --- a/TESTING.md +++ b/TESTING.md @@ -370,7 +370,7 @@ A cell that names a crate-local test describes the witness that exists today. It | `electrum_server_version_history_balance` | Electrum | One mature pad: version (`CARGO_PKG_VERSION`)/history/balance/headers, ping/features (`protocol_min` / `asof_protocol` / `server_version`)/tx/errors, confirmed history omits `fee`, scripthash subscribe notify, skip restatus when a new block misses the SH. TCP `get_history` `from_height` at create / exclusive `to_height` hides spend / `to_height=-1` open; subscribe status stays full; invalid `from_height` type. `get_merkle` wrong height for a known txid. Verbose `transaction.get` stamps `time`/`blocktime`/`confirmations`/`blockhash` and coinbase `vin`/`vout`. `id_from_pos` string vs `merkle=true` `{tx_hash, merkle}`; pos OOB errors. Line at `max_request_bytes` ignored; one past is JSON-RPC `-32600` `request line too long`. Electrum **1.6**: `mempool.get_info`, `blockchain.outpoint.*` (unsubscribe miss is `false`; tip notify while subscribed; missing vout errors), Frigate `silentpayments.subscribe` / unsubscribe / second unsubscribe / start past tip clamps / bad-params error, `block.headers` as a list, `broadcast_package` without a hub errors. With no hub, `estimatefee` is `-1.0`, the histogram is empty, and Cake `tweaks.subscribe [0,1,false]` answers | | `electrum_scripthash_sub_cap_unsubscribe_frees_slot` | Electrum | Per-connection scripthash cap + unsubscribe frees a slot. Outpoint subscriptions use that same numeric cap on their own set (resubscribe does not consume another slot; a third distinct outpoint is `max 2` until unsubscribe). | | `electrum_leftover_mempool_does_not_double_count` | Electrum | Relay-off leftover is confirmed, not a second mempool UTXO. With hub attached, `transaction.broadcast` of non-hex and consensus-invalid rejects (not hang / not admit). Same pad: confirmed `outpoint.get_status` (tip height, no `spending_txid`); `broadcast_package` invalid hex / reject / verbose success / non-verbose `"success"`; mempool-spent outpoint is `height=0` with `spending_txid`; mempool verbose `transaction.get` is `confirmations=0` with no `time`/`blocktime`/`blockhash` | -| `electrum_and_esplora_asof_hides_later_spend` | Electrum + Esplora | One pad: TCP `1.4.2-asof` plus HTTP `?asof=` hide a later spend; unknown asof errors; `GET /tx` `v0_p2wpkh` vout. SH-off tip+1: asof at visible SH watermark accepts, asof of the confirmed hash ahead of SH is `asof not on chain` / HTTP 404. Same-height A-B-A restatuses subscribe (confirming blockhash); `asof:` of the loser hash does not retry onto the sibling. Disconnect spend: history/utxo show the create again; live Esplora `/utxo` stamps the create hash (not the fork); loser `?asof=` 404 with no tip header. HTTP `503` `chain view moved` (no tip header) is `http_503_chain_view_moved_omits_tip_header` | +| `electrum_and_esplora_asof_hides_later_spend` | Electrum + Esplora | One pad: TCP `1.4.2-asof` plus HTTP `?asof=` hide a later spend; unknown asof errors; `GET /tx` `v0_p2wpkh` vout. SH-off tip+1: asof at visible SH watermark accepts, asof of the confirmed hash ahead of SH is `asof not on chain` / HTTP 404. Same-height A-B-A keeps the protocol status (no push; `chain_tip` restamps); `asof:` of the loser hash does not retry onto the sibling. Disconnect spend: history/utxo show the create again; live Esplora `/utxo` stamps the create hash (not the fork); loser `?asof=` 404 with no tip header. HTTP `503` `chain view moved` (no tip header) is `http_503_chain_view_moved_omits_tip_header` | | `electrum_empty_chain_headers_subscribe_and_empty_scripthash` | Electrum | Empty store: `headers.subscribe` errors; scripthash history/balance/unspent/mempool empty | | `electrum_tweaks_subscribe_streams_then_done` | Electrum | Cake `tweaks.subscribe`: one-height result, per-height notifies, `done`, and BIP352 hash-bind on a P2WPKH→P2TR spend. A zero-chunk subscribe returns wave 0 and then `done` while heights remain, and a resubscribe continues. Pre-taproot empty heights collapse into one notify | | `electrum_max_connections_rejects_extra_client` | Electrum | TCP cap drops the extra client | diff --git a/changelog.d/electrum-protocol-status.md b/changelog.d/electrum-protocol-status.md new file mode 100644 index 000000000..35a13937c --- /dev/null +++ b/changelog.d/electrum-protocol-status.md @@ -0,0 +1,3 @@ +Fixed + +- **Electrum scripthash status is the protocol's `txid:height:` again.** Confirmed rows also hashed the confirming block hash, so Electrum desktop, which recomputes the status from `get_history`, never stored the history of an address with a confirmed transaction and disconnected after its timeout. A same-height replace now keeps the status; `chain_tip` still marks it (#1028). diff --git a/crates/rbitcoin-electrum/src/server.rs b/crates/rbitcoin-electrum/src/server.rs index 1546dafde..32a267c47 100644 --- a/crates/rbitcoin-electrum/src/server.rs +++ b/crates/rbitcoin-electrum/src/server.rs @@ -1312,7 +1312,7 @@ fn restatus_notes( query .scripthash_history(sh) .ok() - .and_then(|h| scripthash_status(Some(query), &h).ok()) + .map(|h| scripthash_status(&h)) }; if let Some(status) = status { out.push((*sh, status)); @@ -1923,7 +1923,7 @@ fn dispatch_pinned( let hist = query .scripthash_history_slot(&sh, sh_join) .map_err(|e| e.to_string())?; - scripthash_status(Some(query), &hist)? + scripthash_status(&hist) }; Ok(status_json(status)) } @@ -2530,51 +2530,20 @@ fn append_mempool_history( } } -fn cached_confirming_hash( - height: u32, - cache: &mut HashMap, - load: impl FnOnce(u32) -> Result<[u8; 32], String>, -) -> Result<[u8; 32], String> { - if let Some(h) = cache.get(&height) { - return Ok(*h); - } - let h = load(height)?; - cache.insert(height, h); - Ok(h) -} - -fn scripthash_status( - query: Option<&Query>, - hist: &[rbitcoin_query::ScriptHashHistoryItem], -) -> Result { +/// Electrum protocol status: SHA-256 of `tx_hash:height:` for each row in +/// `get_history` order (confirmed by height, then the mempool tail). Clients +/// recompute it from `get_history` and drop the history when it differs. +fn scripthash_status(hist: &[rbitcoin_query::ScriptHashHistoryItem]) -> String { if hist.is_empty() { - return Ok(String::new()); + return String::new(); } use bitcoin::hashes::{sha256, Hash as _}; let mut s = String::new(); - let mut by_height = HashMap::new(); for i in hist { - if i.height > 0 { - let q = query.ok_or_else(|| "status preimage needs a chain query".to_string())?; - let hash = cached_confirming_hash(i.height as u32, &mut by_height, |h| { - let (_, rec) = q - .header_at_height(Height(h)) - .map_err(|e| e.to_string())? - .ok_or_else(|| "header missing for confirmed history row".to_string())?; - Ok(rec.hash) - })?; - s.push_str(&format!( - "{}:{}:{}:", - txid_hex(&i.txid), - i.height, - hash_hex_rev(&hash) - )); - } else { - s.push_str(&format!("{}:{}:", txid_hex(&i.txid), i.height)); - } + s.push_str(&format!("{}:{}:", txid_hex(&i.txid), i.height)); } let hash = sha256::Hash::hash(s.as_bytes()); - Ok(rbitcoin_primitives::hex_encode(hash.to_byte_array())) + rbitcoin_primitives::hex_encode(hash.to_byte_array()) } fn scripthash_status_full(query: &Query, mp: &MempoolHub, sh: &[u8; 32]) -> Result { @@ -2592,7 +2561,7 @@ fn scripthash_status_full_slot( .scripthash_history_slot(sh, slot) .map_err(|e| e.to_string())?; append_mempool_history(&mut hist, mp, sh); - scripthash_status(Some(query), &hist) + Ok(scripthash_status(&hist)) } /// Helper to compute electrum scripthash hex (reversed) from script bytes. diff --git a/crates/rbitcoin-electrum/src/server_tests.rs b/crates/rbitcoin-electrum/src/server_tests.rs index 25ee00717..b68f8abbd 100644 --- a/crates/rbitcoin-electrum/src/server_tests.rs +++ b/crates/rbitcoin-electrum/src/server_tests.rs @@ -199,38 +199,21 @@ fn config_helpers_and_param_parsers() { .unwrap_err() .contains("to_height")); - let empty_status = scripthash_status(None, &[]).unwrap(); - assert!(empty_status.is_empty()); - let missing = scripthash_status( - None, - &[rbitcoin_query::ScriptHashHistoryItem { - height: 1, - txid: [1u8; 32], - tx_fk: Fk::NULL, - fee: None, - }], - ) - .unwrap_err(); - assert!( - missing.contains("query"), - "confirmed preimage without query: {missing}" - ); - let (dir, q) = tmp_store(); - let missing_hdr = scripthash_status( - Some(&q), - &[rbitcoin_query::ScriptHashHistoryItem { - height: 1, - txid: [1u8; 32], - tx_fk: Fk::NULL, - fee: None, - }], - ) - .unwrap_err(); - assert!( - missing_hdr.contains("header missing"), - "confirmed row without header: {missing_hdr}" + assert!(scripthash_status(&[]).is_empty()); + let one = scripthash_status(&[rbitcoin_query::ScriptHashHistoryItem { + height: 1, + txid: [1u8; 32], + tx_fk: Fk::NULL, + fee: None, + }]); + let preimage = format!("{}:1:", txid_hex(&[1u8; 32])); + assert_eq!( + one, + rbitcoin_primitives::hex_encode( + bitcoin::hashes::sha256::Hash::hash(preimage.as_bytes()).to_byte_array() + ), + "confirmed row status is tx_hash:height: with no block hash" ); - let _ = std::fs::remove_dir_all(&dir); use bitcoin::hashes::Hash; let hdr = bitcoin::block::Header { @@ -267,23 +250,6 @@ fn config_helpers_and_param_parsers() { ); } -#[test] -fn cached_confirming_hash_loads_once_per_height() { - let mut cache = HashMap::new(); - let mut loads = 0u32; - let mut load = |h: u32| { - loads += 1; - Ok::<_, String>([h as u8; 32]) - }; - let a = cached_confirming_hash(7, &mut cache, &mut load).unwrap(); - let b = cached_confirming_hash(7, &mut cache, &mut load).unwrap(); - let c = cached_confirming_hash(8, &mut cache, &mut load).unwrap(); - assert_eq!(a, [7u8; 32]); - assert_eq!(b, [7u8; 32]); - assert_eq!(c, [8u8; 32]); - assert_eq!(loads, 2, "same height must not load twice"); -} - #[test] fn drop_unsubscribed_status_clears_idle_hashes() { let mut last = HashMap::new(); @@ -838,7 +804,7 @@ fn get_history_height_window_and_status_full() { }; let full_hist = q.scripthash_history(&sh_bytes).unwrap(); assert_eq!(full_hist.len(), 4); - assert_eq!(scripthash_status(Some(&q), &full_hist).unwrap(), status_s); + assert_eq!(scripthash_status(&full_hist), status_s); let _ = prev; let _ = std::fs::remove_dir_all(&dir); diff --git a/crates/rbitcoin-test/tests/electrum_protocol.rs b/crates/rbitcoin-test/tests/electrum_protocol.rs index d98ac53d9..83fa14d75 100644 --- a/crates/rbitcoin-test/tests/electrum_protocol.rs +++ b/crates/rbitcoin-test/tests/electrum_protocol.rs @@ -377,13 +377,11 @@ async fn assert_esplora_asof_dead(addr: SocketAddr, path: &str) { async fn mempool_status_matches_history_row_order( stream: &mut TcpStream, - q: &Query, hub: &rbitcoin_net::MempoolHub, chain: &rbitcoin_test::MatureRegtestChain, ) { use bitcoin::hashes::{sha256, Hash as _}; use bitcoin::{Amount, OutPoint, ScriptBuf, Sequence, Transaction, TxIn, TxOut, Witness}; - use rbitcoin_primitives::Height; let coinbase = &chain.blocks[2].txdata[0]; let spk = ScriptBuf::from_bytes(vec![0x51]); @@ -428,13 +426,10 @@ async fn mempool_status_matches_history_row_order( for row in rows { let height = row["height"].as_i64().unwrap(); let tx_hash = row["tx_hash"].as_str().unwrap(); - let piece = if height > 0 { - let (_, rec) = q.header_at_height(Height(height as u32)).unwrap().unwrap(); - let block = rbitcoin_primitives::display_hash_hex(&rec.hash); - format!("{tx_hash}:{height}:{block}:") - } else { - format!("{tx_hash}:{height}:") - }; + // Electrum protocol status: `tx_hash:height:` for every row. Electrum + // desktop recomputes it from get_history and drops the history on a + // mismatch (`synchronizer.py` `history_status`). + let piece = format!("{tx_hash}:{height}:"); preimage.push_str(&piece); ordered.push((height, piece)); } @@ -660,7 +655,7 @@ async fn electrum_server_version_history_balance() { assert!(v["result"]["count"].as_u64().unwrap() >= 1); assert!(!v["result"]["hex"].as_str().unwrap().is_empty()); - mempool_status_matches_history_row_order(&mut stream, &q, &hub, &chain).await; + mempool_status_matches_history_row_order(&mut stream, &hub, &chain).await; pin_wallet_protocol_1_6(&mut stream).await; @@ -2012,15 +2007,24 @@ async fn electrum_and_esplora_asof_hides_later_spend() { reorg_from_height: Some(103), }) .expect("same-height replace notify"); - let push_b = read_notify(&mut stream, "same-height status B").await; - assert_eq!( - push_b["method"].as_str(), - Some("blockchain.scripthash.subscribe") + // Same txids at the same heights: the protocol status is unchanged, so + // the restatus pushes nothing and a fresh subscribe answers status A. + let quiet_b = drain_lines(&mut stream, Duration::from_millis(500)).await; + assert!( + quiet_b.is_empty(), + "same-height replace keeps tx_hash:height:, so no status push: {quiet_b:?}" ); - let status_b = push_b["params"][1].as_str().expect("status B").to_string(); - assert_ne!( - status_a, status_b, - "same-height replace must change status via confirming blockhash" + let resub_b = rpc( + &mut stream, + 22, + "blockchain.scripthash.subscribe", + json!([sh.clone()]), + ) + .await; + assert_eq!( + resub_b["result"].as_str(), + Some(status_a.as_str()), + "status hashes tx_hash:height: only: {resub_b}" ); let live_b = rpc( &mut stream, @@ -2070,11 +2074,10 @@ async fn electrum_and_esplora_asof_hides_later_spend() { reorg_from_height: Some(103), }) .expect("A-B-A restore notify"); - let push_a2 = read_notify(&mut stream, "same-height status A again").await; - assert_eq!( - push_a2["params"][1].as_str(), - Some(status_a.as_str()), - "A-B-A must restore the original status string: {push_a2}" + let quiet_a2 = drain_lines(&mut stream, Duration::from_millis(500)).await; + assert!( + quiet_a2.is_empty(), + "A-B-A leaves the status unchanged, so no push: {quiet_a2:?}" ); q.disconnect_tip().unwrap();