policy: detect and count BPUB data

[kwsantiago]

BPUB stores files as fake public keys in P2WSH 1-of-N multisig scripts. Its readers join the keys of every such input that shares a control key, in input order, and read the result as a stream that opens with a header, so a transaction whose inputs form that header (the v4/v5 length and version, or the legacy BPUB magic) now has those witness scripts counted as non-standard datacarrier, as OLGA is. It follows both published readers, including keys of any size and headers split across inputs, and unlike #422 it does not depend on the number of keys, so avoiding it means changing every BPUB reader. -rejectbpub=0 turns it off.

Over every block from 961640 to 977624 (3,800,289 transactions), it counts the 2,113 transactions that both readers decode into a file whose hash checks out, and no other. Submitted to the node on regtest with their real prevouts recreated, those 2,113 are rejected, and the 4,670 other transactions checked get the same result as with -rejectbpub=0.

[jasonsopko]

ACK 1f0834d, code review and tested.

Read BPUBBytes against both readers. python-bitcointx iterates OP_0 as the int 0 and OP_1 to OP_16 as N, so the indexer skips a script with an OP_0 key and takes N in either form, as the code comment says; the header offsets match stream.ts and decode_stream.

Ran it over the 3,785,714 transactions in blocks 961640 to 977635. In your range it counts 869. A decoder written from the two readers' rules decodes the same 869 with the hash matching and none of the other 2,605 there whose last witness item starts OP_1 and ends OP_CHECKMULTISIG. My 725 from 976024 to 976740 are all among them. From 976741 to 977635 it counts 1,260 more, every one decodable the same way: 352,423 inputs, 351,192 of them still 1-of-15, 48.9% of all block weight in those 895 blocks, a reveal in 681 of them, 0.774 BTC in fees.

Cost: 0.02 to 0.41 ms per call in process on the largest forms Core's limits allow (one 3,597-byte script of 1-byte pushes, which Knots' default -maxscriptsize of 1650 refuses anyway; 1,200 1-of-2 inputs with distinct control keys or one; 520 1-of-15 inputs). Over P2P from one inbound peer, your commit on the 29.x-knots tip 58398ba against the tip alone: a real 159-input reveal 0.91 ms per transaction against 1.02, refused here as txn-datacarrier-nonstandard and there at the fee check; 1,000 1-of-2 inputs, refused at the fee check on both, 3.41 against 2.81 ms with distinct control keys and 3.31 against 2.81 with one; a 1,563-byte script of 1-byte pushes 0.19 against 0.16. RPC median 0.3 to 0.6 ms in every run, peer still connected.

Three things:

• It runs twice per transaction, in DatacarrierBytes and again in CalculateExtraTxWeight, the point you made on policy: Count data in output hashes and keys #389. Cheap here, but one call would do.
• No switch of its own. Turning only this off takes -acceptnonstddatacarrier=1 together with -datacarrierfullcount=0, since -datacarriersize counts both bytes, and that turns off the non-standard datacarrier check for every other form as well. policy: Count data in output hashes and keys #389 and policy: Count data in a dead conditional branch #400 each have one (-rejectfakeoutputs, -rejectdeadbranches).
• With the k422 head in the same tree a reveal is counted twice toward -datacarriercost: reveal 6024536974180b43 (159 inputs, 81,295 bytes of witness script) gets 20,856 from k422's unproven keys in first and 81,295 here in second, 102,151 for 81,295 bytes. 2,127 of the 2,129 reveals do. Under the defaults the transaction is refused either way; the double count shows in the -datacarriersize comparison (102,151 against 81,295 bytes) and, on a node that admits it, in the -datacarriercost vsize it is ranked by.

For the rc1 spec: git merge-tree gives a conflict in policy.cpp with the k400 head 847d6a2, in policy.cpp and policy.h with the k389 head 40e0e6f, and a clean merge with the k422 head eea2467.

Scripts, logs and the per-transaction data: https://gist.github.com/jasonsopko/b3ba966ac84ce4cd50f35d7734ae5afb

[kwsantiago]

Added -rejectbpub, default on. Kept the second call to stay clear of #389's signature change; #422 overlap only overcounts.

[kwsantiago]

Capped the head copy at 10 bytes per input; no change in what is counted.

[delcin-raj]

Encountered compile time warnings

[kwsantiago]

Fixed both as suggested; the test file now compiles clean on GCC 16 at -O2.

[jasonsopko]

re-ACK ad24234

Since 1f0834d: -rejectbpub, the head copy capped at 10 bytes, MAX_BLOCK_SERIALIZED_SIZE for the body bound (same value), the opcode == keys case dropped (no valid P2WSH spend reaches it), plus cleanups and test changes.

• Over the same 4,788 candidate transactions from 961640 to 977635, the counts are byte for byte those of 1f0834d.
• Cherry-picked onto 58398ba: ctest 131/131 and the functional suite (wallet off) pass.
• Over P2P the real 159-input reveal is refused as txn-datacarrier-nonstandard, and at the fee check with -rejectbpub=0. Cost per transaction matches 1f0834d on the same tip: 0.89 ms against 0.94 for the reveal, 3.42 against 3.41 for 1,000 1-of-2 inputs.
• For the rc2 spec: onto the tip, policy: count bare data envelopes (OP_2DROP/pushnum runs) as datacarrier #319, policy: count pubkeys no signature can prove as datacarrier bytes #422 with the fix I posted there, policy: detect and count BPUB data #463 and policy: Count data in a dead conditional branch #400 merge in that order without conflicts.