wallet-toolbox: a stale merkle proof (orphaned block) in `proven_txs` makes every `createAction` that selects that coin fail permanently with `WERR_INTERNAL: merged Beef failed validation.`

[Calgooon]

Summary

After a small reorg on mainnet, an operator-hosted StorageServer kept merkle proofs that point at the orphaned block for 14 of a wallet's default-basket outputs. From then on, any createAction whose change selection picks one of those coins is signed successfully and then rejected by storage in processAction with:

{"name":"WERR_INTERNAL","message":"merged Beef failed validation.","isError":true,"code":"WERR_INTERNAL"}

The failed action is later marked failed, its inputs go back to spendable=1, selection is deterministic, so the next attempt picks the same coin and fails again. From the client there is no way to see which input is bad and no way to repair it. The wallet is effectively unable to spend until the operator re-proves the rows by hand.

Chain facts (public data)

• Orphaned block at height 965771: hash 0000000000000000153e10f465dba9697e4bde364fdf3a3224a736b019ffbfb1, merkle root 709df5698ce021edfdadd7eb153a97847e9eda65ed9c522e7accbe8d3f95e219, time 2026-09-07 22:39:32Z. WhatsOnChain still serves it with confirmations: 1.
• Canonical block at height 965771: hash 00000000000000001de5aa96baa3566ce66e4941f8295cc44cc85fc75949db4d, merkle root a785b0537cc490b792a9303660fbe5d6f5aaa96af29191c619cf71072100d6ae.
• The transactions from the orphan were re-mined at height 965773, hash 0000000000000000146bc084ec137a3c9608a07159128c66302051b6fe176e33, merkle root 9d9d67e15e74d554138531ef858f29ecdee471e7333ccb17af43215e13bebdfc.
• Chaintracks (mainnet-chaintracks.babbage.systems) agrees with WhatsOnChain on all three heights and returns isValidRootForHeight(709df569…, 965771) === false.

I can share the affected txids privately with a maintainer; I'd rather not put a wallet's outputs in a public issue.

Where it fails (origin/main, wallet-toolbox 2.13.0)

1. shareReqsWithWorld → getReqsAndBeefToShareWithWorld → mergeReqToBeefToShareExternally pulls each input's ancestry with getValidBeefForKnownTxid(txid, beef, undefined, knownTxids, trx) (storage/StorageProvider.ts:943). It never passes chainTracker / skipInvalidProofs, even though getValidBeefForTxid supports both (storage/StorageProvider.ts:1221-1231), so a stored proof is merged as-is.
2. verifyMergedBeef then runs beef.verify(chainTracker), gets false, and throws the generic WERR_INTERNAL (storage/methods/processAction.ts:238-241). The log line has the BEEF, the error returned to the app has nothing.
3. The TaskReorg header comment already names this gap: "Proper reorg handling also requires repairing invalid beefs for new transactions when createAction fails to verify a generated beef against the chaintracker." (monitor/tasks/TaskReorg.ts:25-26).

How I confirmed it from the client side

listOutputs({ basket: 'default', include: 'entire transactions' }) builds each coin's BEEF with the same getValidBeefForKnownTxid, so I walked every spendable coin's ancestry locally and checked each bump root against the chaintracks tracker. 982 of 996 coins verify; 14 carry a bump at height 965771 with root 709df569…. I then fetched the proven_tx_reqs rows of the last 25 signed-then-failed actions and parsed their rawTx: all 25 spend at least one of those 14 outputs. Nothing else in the wallet is wrong.

Why the client can't recover

• Explicit inputs: the coins are change, so createAction rejects them with "an unmanaged input. Change outputs are managed by your wallet."
• options.knownTxids only affects the BEEF returned at createAction; the send path calls getReqsAndBeefToShareWithWorld(txids, []).
• WalletStorageManager.reproveProven / verifyAndRepairBeef need a local StorageProvider; they do nothing for a StorageClient against a remote store.
• updateProvenTxReqWithNewProvenTx is exposed to clients but findOrInsertProvenTx returns the existing row, so a correct proof can't replace a stale one.
• The operator's Monitor apparently did not re-prove the rows (TaskReviewProvenTxs with minBlockAge = 100 should have; the reorg is ~1,400 blocks old). A client can't tell which toolbox version or which tasks the server runs.

Side effects observed

• Each failure leaves a transactions row in unprocessed and a proven_tx_reqs row in unprocessed until TaskFailAbandoned marks the transaction failed (~8 minutes). The req row stays unprocessed forever: 23 such rows accumulated. TaskSendWaiting only picks unsent/sending, so they are inert, but they are clutter.
• Because inputs are released on failed, the same stale coin is re-selected on the next attempt. Small spends hit the small stale coins, large spends hit the large ones. From the user's point of view the wallet is simply broken.

Suggestions

1. On the send path, pass the chain tracker and skipInvalidProofs: true into getValidBeefForKnownTxid (or call reproveProven for the offending txid when verifyMergedBeef fails), so one stale proof degrades to "include the raw tx and its inputs' proofs" instead of a hard failure. The machinery already exists.
2. Make the error actionable: throw WERR_INVALID_MERKLE_ROOT with txid/height/root (or include the rejected roots in the message) instead of WERR_INTERNAL.
3. Validate the allocated-change BEEF against the tracker at createAction time, before the client signs, or quarantine the offending coin (e.g. drop spendable, or tag the output) so selection doesn't loop on it.
4. Give StorageClient users a user-scoped way to ask the store to re-prove a txid (a specOp would fit the existing pattern), since operator-hosted stores can't be repaired from the client.

Repro sketch

Insert (or mutate) a proven_txs row so its merkle path has a root that the chain tracker rejects for its height, keep the corresponding output spendable=1 in default, then run createAction with an output that forces that coin to be selected. test/storage/processAction.test.ts:128 already asserts the throw; the missing piece is recovery.

Environment

• Client: MetaNet Client 0.7.3 (Tauri) on macOS, remote storage.
• Storage: operator-hosted StorageServer + StorageKnex (MySQL) with a Monitor; version unknown to me.
• Diagnosis scripts: @bsv/wallet-toolbox 1.8.3 client + @bsv/sdk 1.10.4 for the read-only walk. Code references above are against origin/main at 82ab4d3 (2.13.0).