Repository navigation
policy: Count data in output hashes and keys - #389
jasonsopko wants to merge 17 commits into
Conversation
| for (size_t i{0}; i < n; ++i) { | ||
| if (!program_size[i]) continue; | ||
| const bool sibling{data_groups.count({types[i], tx.vout[i].nValue}) > 0}; | ||
| const bool dust_run{at_dust[i] && dust_runs[types[i]] >= DATA_OUTPUT_DUST_RUN}; |
There was a problem hiding this comment.
Ran this over 150 mainnet blocks below the fork (961481-961630, 691,479 txs). 215 txs are newly rejected; 213 of them by this rule alone. Of those 213: zero carry an OLGA stamp, 22 carry a runestone, and 202 repeat an identical scriptPubKey across the dust outputs, which cannot be a spread payload. The ones I pulled up are padding/consolidation to a single repeated address.
Zero true positives, 213 false positives. Requiring the programs to be distinct drops 202 of them for one std::set and costs the detector nothing, since payload chunks differ by construction. That still leaves the runestone-carrying ones.
There was a problem hiding this comment.
Done: the run counts distinct scripts now. I re-ran the mirror over the same 73 blocks (332,878 transactions) and got your numbers exactly: 83 by the dust run alone, 80 of them one repeated script, 8 with a runestone. With distinct scripts the 80 drop out.
Of the 85 newly rejected, five remain: an ACME payload at 961509 (344a3729…), a P2PKH paying 20,300 sat to a hash with nine 0xff bytes (988f28cb…), an eight-output 330-sat taproot batch with seven distinct keys (c3974e48…), the taproot case the PR body calls out, and two runestone transactions with distinct 330-sat outputs, which -rejecttokens (on by default as of 29.4.1) rejects anyway.
| data[i] = true; | ||
| } | ||
| if (program.size() == WITNESS_V0_SCRIPTHASH_SIZE && txout.nValue <= GetDustThreshold(txout, dust_rate)) { | ||
| at_dust[i] = true; |
There was a problem hiding this comment.
IsDust uses <, this uses <=, so the rule fires at exactly 330 sat, the standard spendable value wallets already use for anchors and padding. All 1,655 dust hits in that corpus were at exactly 330.
Verified on a regtest node at Knots defaults:
REJECTED whitened 6-chunk P2WSH payload at 330 sat
ACCEPTED whitened 6-chunk P2WSH payload at 331 sat
ACCEPTED whitened payload over 10 P2WPKH chunks at 294 sat
The program.size() == 32 gate means 20-byte chunks never enter the run count at all. Adaptation is +1 sat/output and an XOR; Counterparty already ships RC4 obfuscation.
There was a problem hiding this comment.
Deliberate. Below the threshold is rejected as dust before this code runs at the defaults, so the rule is about the floor itself, which is where every encoder sits because it is the cheapest value allowed. A payload at 331 sat gets past the run, yes, at one sat per output. The byte tells and the curve check apply at any value, and once one chunk trips them the sibling rule takes the rest at that value, so the encoder has to whiten every chunk, not only the header. A fully whitened P2WSH payload at 331 sat gets through, and I do not have an output-inspection rule that stops it without stopping payments.
20-byte hashes get the byte tells but not the run. 546 sat is the floor many wallets apply to every output type, so a run of 546-sat P2PKH outputs is not the tell a run of 330-sat P2WSH outputs is. No transaction in your 73 blocks had three distinct 20-byte outputs at their floor, so adding them would cost nothing today; I left them out.
| for (size_t i{tx.vout.size()}; i; ) { | ||
| const CTxOut& txout = tx.vout[--i]; | ||
| if (data_outputs[i]) { | ||
| ret.second += data_outputs[i]; |
There was a problem hiding this comment.
continue skips IsOLGA, which attributes the whole payload to the header output. If the header trips a byte tell and the chunks sit at a different value, the count collapses:
pre-PR {0,164} post-PR {0,41}
No occurrences in 691,479 txs, so latent. std::max of the two instead of replacing.
There was a problem hiding this comment.
Fixed. Outputs the OLGA detector already covers, header and chunks, return 0 from DataOutputBytes; for the rest, both counts take the larger of the OLGA figure and this one. A stamped payload counts once either way: 164 for the four-output test case, where the pushed version gave 41 when the header tripped a tell, and a plain max would give 287 once the chunks tripped the dust run. Unit tests for both shapes.
| program_size[i] = program.size(); | ||
| if (ProgramLooksLikeData(program)) { | ||
| data[i] = true; | ||
| } else if (types[i] == TxoutType::WITNESS_V1_TAPROOT && !XOnlyPubKey{program}.IsFullyValid()) { |
There was a problem hiding this comment.
secp parse per taproot output, in PreChecks before script validation. Max-standard-size tx, 2,300 outputs: 52.4 ms, vs 1.77 ms for the same shape as P2WSH. The tx can then be rejected for free. Runs twice when -datacarriercost > 1.0, since CalculateExtraTxWeight calls DataOutputBytes again.
There was a problem hiding this comment.
Two changes. The curve check stops after the first 64 taproot outputs of a transaction. A whitened chunk is off the curve half the time, so a payload trips it within its first few outputs and the sibling rule marks the rest; getting past the cap costs 64 real keys per transaction. On this box (Release build) a 2,300-output taproot transaction went from 10.1 ms to 0.67 ms in DataOutputBytes, against 0.37 ms for the same shape as P2WSH.
And it runs once: PreChecks calls DataOutputBytes and hands the vector to both functions, so the second pass at -datacarriercost above 1 is gone.
d5cd1f7 to
3c1d175
Compare
|
Pushed a new version addressing @kwsantiago's review:
Corpus, blocks 961481 to 961553 (332,878 transactions): of the 85 transactions newly rejected under Knots defaults, 5 remain: an ACME payload, a P2PKH to a hash with nine |
3c1d175 to
da0c3b8
Compare
|
Rebased onto |
jasonsopko
changed the title
da0c3b8 to
34ce3a5
Compare
|
Rebased on 29.4.2 and squashed. The detector is now behind |
The detector counts the OP_FALSE OP_IF envelope but not its mirror, OP_1 OP_NOTIF, or any other branch constants make unreachable. 23e8f946... at 969961 carried a 37 KB JPEG XL image that way, and nothing on the fence saw it. Follow the script as far as constants decide it, through stack moves, arithmetic, comparisons, no-ops and the altstack, and count every conditional part that can never run, including the dead part of an OP_ELSE. Anything read from the witness or a signature check is unknown and never counted, which is how spendable scripts branch. The rule sits behind -rejectdeadbranches, default on and reset by -corepolicy; with it off the detector is the old one byte for byte. Checked on Ubuntu 24.04 against 29.4.2: a gcc 13.3 -Werror build with wallet and ZMQ has no warnings, and all 147 ctest suites and the full functional suite pass, alone and merged with bitcoinknots#389, bitcoinknots#319 and bitcoinknots#422. The clang 18 ASan/UBSan build passes the unit suite and the policy functional tests. A fuzz target checked that with the switch off the result matches the original function, and that every part marked dead leaves the interpreter's verdict unchanged when an OP_RETURN is put in it, over 4.9 million inputs in 45 minutes. Over all 3,018,541 transactions since the fork it counts one: 23e8f946..., at 37,854 bytes.
|
Concept ACK |
|
Funding side of the same 717 blocks as my #422 comment, 976024 to 976740. The 725 bpub reveals there spend 193,117 outputs, all P2WSH at 546 sat, from 725 funding transactions, and the run rule (3 or more distinct P2WSH outputs sharing one value at or under 546 sat) matches all 725. It matches 47 other transactions in the window: one run of 6 outputs at 330 sat and 46 batches of 202 to 412 outputs at 546 sat, and none of their run outputs is spent in the window by anything. Scripts and data: https://gist.github.com/jasonsopko/d07b6b8044f2b02a3ebea87711dab9e7 |
|
Pushed a fix for a false positive. Tx An output whose script matches a coin the transaction spends is now left uncounted. The spender can spend that coin, and filling a hash or key with chosen bytes takes at least 2^128 work (a taproot key's discrete log), so the most such an output can carry is a vanity prefix. It is not marked as a sibling of a data output either. A dust P2WSH output to a spent script still counts toward a run, so paying one back cannot break a run up. Bare scripts are never looked at here, so nothing changes for bare multisig. My first push of this, 6d929d7, made the lookup quadratic. With every input spending one script and every output paying it, 1,197 inputs by 1,162 outputs near the size limit, From 961640 to 976951, 2,163 transactions have a non-OP_RETURN output Plumb counts as data, and this is the only one where such an output pays its own input's address. I reran every non-coinbase transaction in that range through Plumb's policy code, which carries this PR, with and without the fix: that verdict changes and nothing else does. Tested on Ubuntu 24.04.5, gcc 13.3.0, |
… bytes The rule only recognised a key in a 33 or 65 byte push, so the same payload written as tapscript went free: a 1-of-15 costs 132 datacarrier bytes under P2WSH and nothing at all under a taproot leaf, for 32 bytes of publishable space per key rather than 31. A 32 byte push is not a key on sight, since nothing separates an x-only key from a hash until something spends it. Count it where a checksig-family opcode claims it instead, which leaves the preimage of a hashlock alone. OP_CHECKSIGADD is how tapscript writes an m-of-n, so credit its canonical shape as OP_CHECKMULTISIG's is already credited: a run of <pubkey> OP_CHECKSIG (<pubkey> OP_CHECKSIGADD)* <k> OP_NUMEQUAL credits k, whatever size its keys are. Without it every honest tapscript multisig past eleven keys would be charged for the keys it does not spend, which is the opposite of the point: a 20-of-20 would cost 297 bytes. The OP_CHECKSIG that opens the run has credited one of the k already, so only the remaining k-1 are added there. Padding the run, reordering it or naming a k it cannot satisfy forfeits the credit rather than earning one, so working around the rule raises the charge instead of lowering it. An opcode that leaves the stack as it found it does not hide the key beneath it from the opcode that spends it, so it does not buy the count off either.
Checks the taproot leaf against the P2WSH script that says the same thing, shape by shape, so the two are charged alike rather than only plausibly, including at the tolerance boundary: 1-of-11 is free and 1-of-12, 1-of-13 and 5-of-16 pay the same either way. Covers the hashlock shapes a 32 byte push must not be mistaken for a key in, the padded and inflated runs that forfeit the credit, the OP_CHECKSIG that opens a run crediting one of the k and not two, a k above 16 written as a minimal push, and the key stranded on the stack that neither rule reaches.
…rrier' into policy-dead-branch-data bitcoinknots#422 at eea2467 changes DatacarrierBytes too. This branch builds on it so the 29.4.3 assembly can merge it after k422.
The detector counts the OP_FALSE OP_IF envelope but not its mirror, OP_1 OP_NOTIF, or any other branch constants make unreachable. 23e8f946... at 969961 carried a 37 KB JPEG XL image that way, and nothing on the fence saw it. Follow the script as far as constants decide it, through stack moves, arithmetic, comparisons, no-ops and the altstack, and count every conditional part that can never run, including the dead part of an OP_ELSE. Anything read from the witness or a signature check is unknown and never counted, which is how spendable scripts branch. The rule sits behind -rejectdeadbranches, default on and reset by -corepolicy; with it off the detector is the old one byte for byte. Checked on Ubuntu 24.04 against 29.4.2: a gcc 13.3 -Werror build with wallet and ZMQ has no warnings, and all 147 ctest suites and the full functional suite pass, alone and merged with bitcoinknots#389, bitcoinknots#319 and bitcoinknots#422. The clang 18 ASan/UBSan build passes the unit suite and the policy functional tests. A fuzz target checked that with the switch off the result matches the original function, and that every part marked dead leaves the interpreter's verdict unchanged when an OP_RETURN is put in it, over 4.9 million inputs in 45 minutes. Over all 3,018,541 transactions since the fork it counts one: 23e8f946..., at 37,854 bytes.
FindDeadParts ran on every input before IsWitnessStandard checks the witness size, and OP_ROLL with a known depth moved up to 1,000 stack entries. A 390 KB script repeating <998> OP_ROLL took 506 ms to analyze against 1.7 ms for the plain scan, on a transaction the node then rejects at no cost to the sender. AuthintegX raised it in the Knots Discord. Skip inputs whose witness is over -maxscriptsize, which fail IsWitnessStandard right after, and charge each opcode one step and each OP_ROLL its depth, stopping at four steps per byte. Stopping marks nothing more, so it can only undercount, the same as a condition read from the witness. The roll case drops to 92 us at 1,650 bytes (was 882) and 9.7 ms at 390 KB. The worst left is an arithmetic chain at about 85 ns per byte, the same order as the interpreter on that script. Counts over all 3,103,397 transactions from 961640 to 974573 are unchanged.
gcc 11 at -O2 cannot see that OP_WITHIN reads lo only after Num() has set it, and the previous-releases CI job (DEBUG, -Werror) fails with "'lo' may be used uninitialized". Start every int64_t the walk fills through Num() at zero so no compiler has to prove it. No behavior change: each one is still read only where Num() returned true.
OLGA hides a payload in P2WSH outputs at the dust threshold, and the detector keys on its "stamp:" magic. The same layout with "ACME" has put over 100 transactions on chain since 957497, and changing the magic is free. Judge the bytes instead. A hash or x-only key is uniformly random, so one byte value eight times in a program, a run of six, or a taproot key off the curve marks it as data, along with its same-type same-value siblings. Three or more distinct P2WSH outputs sharing one value of 546 sat or less are data on their own: encoders pay every chunk the same, batch payouts on chain start at 600 sat, Lightning anchors are a pair and HTLC amounts differ. Small taproot runs are left to the curve check, since the ones on chain are ordinals postage with no data in the outputs. The all-zero burn hash is left alone. The detector sits behind -rejectfakeoutputs, default on and reset by -corepolicy. Over the 3,018,541 transactions since the fork it newly rejects 45: 14 funding transactions of the fake-key and JXL encoders, 14 swap memos continued as ASCII in a P2WPKH hash, 11 ACME-style P2WSH payloads and 6 taproot outputs tagged DECODE WITNESS. Checked on Ubuntu 24.04 against 29.4.2: a gcc 13.3 -Werror build with wallet and ZMQ has no warnings, and all 147 ctest suites and the full functional suite pass, alone and merged with bitcoinknots#400, bitcoinknots#319 and bitcoinknots#422. The clang 18 ASan/UBSan build passes the unit suite and the policy functional tests.
Tx e2031ff1 at 976048 spends from the vanity address 11111119536NnU2vPAvoQhvkheQ8ejYJ, whose hash starts with six zero bytes, and pays its change back to it. The run tell read the zeros as data and -rejectfakeoutputs refused a plain payment. Of the 2,167 transactions since the fork with a counted non-OP_RETURN output, it is the only one where a counted output pays an address one of its own inputs spends from. An output whose script is the script of a coin the transaction spends is real: the spender holds that coin, and fitting chosen bytes into a hash or key takes about 2^160 work, so the most it can carry is a ground prefix. DataOutputBytes takes an optional view and leaves such outputs uncounted. They are not marked as siblings of a data output either, but a dust P2WSH output to a spent script still counts toward a run, so paying one back cannot break a run up. Mempool acceptance and the PSBT size estimate pass the view; callers without one behave as before. The lookup sorts the output indexes by script and searches each spent coin's script, copying nothing; at the standard size limit it adds at most about 0.3 ms. Only the hash and key outputs DataOutputBytes already reads (P2PKH, P2SH, P2WPKH, P2WSH, P2TR) can be exempt. A bare script that repeats a spent coin's, such as bare multisig with data keys, is never looked at here and keeps every other count and rule: Knots' own data count, -permitbaremultisig and -permitbarepubkey, and BIP110's 34-byte output cap while it is active. A unit test pins that. Checked on Ubuntu 24.04, gcc 13.3 with -Werror, wallet off: all 131 ctest suites and the full functional suite pass, alone (217 run, 139 skipped) and merged into Plumb (219 run, 139 skipped). The new mempool_data_outputs.py case fails on plumb6. Rerunning all 3,643,515 transactions from 961640 to 976951 through the policy code before and after changes the verdict of e2031ff1 and of nothing else.
I made the spent-script lookup in DataOutputBytes quadratic. Every input walked every output paying its script, so a transaction whose inputs all spend one script and whose outputs all pay it cost inputs times outputs: at 1,197 by 1,162, near the standard size limit, the lookup took 34.5 ms where the function takes 0.2 ms without it. It runs before the fee check, so zero-fee copies make a node pay that cost without the sender paying anything. The outputs paying one script sit together in the sorted index and are marked together, so an input whose script is already marked skips the walk, and each output is visited at most once. The same transaction now takes 0.68 ms with the lookup. A unit test covers two inputs spending one script beside an input spending another, and fails if the skip ends the loop instead of moving to the next input.
The DataOutputBytes comment said a chosen hash or key costs about 2^160 work. That holds for the 20-byte hashes, but a taproot output key someone can spend needs its discrete log, about 2^128 work, so 2^128 is the bound across the five output types. The conclusion does not change.
e4545aa to
40e0e6f
Compare
|
Pushed 40e0e6f: the same four commits, rebased so the release assembler can merge this after #400. The branch now sits on the #400 head (847d6a2), which is the k354 merge of 29.4.2 with the #422 head merged in. #389 and #400 both add a helper above Tested on the new head: -Werror build, ctest 128 of 128, and mempool_data_outputs.py, mempool_dead_branch.py and mempool_datacarrier.py. tool_cli_completion.py fails on the branch alone, as on #400: the root predates the regenerated completions, and #400's fork CI run on that root is red in every job that runs the functional suite for that one test; expect the same here. Merged into the 29.4.3rc1 candidate (chrisguida/bitcoin a7d7d2e) after #400: no conflict, the result differs by exactly this PR's diff, ctest 131 of 131, functional suite 219 passed, 141 skipped, 0 failed, tool_cli_completion.py among the passes. Spec line proposed on luke-jr/tmp#5. If #400's head moves, say so and I will re-root. |
4 participants
What
Counts outputs whose hash or key is data as data carrier bytes, behind a new
-rejectfakeoutputs(default on, reset by-corepolicy).Why
OLGA-style payloads with a different magic (
ACME) get past thestamp:detector; over 100 are on chain. The tells: one byte value eight times or a run of six in a hash, a taproot key off the curve, or three or more distinct P2WSH outputs sharing one value of 546 sat or less. Encoders pay every chunk the same; batch payouts on chain start at 600 sat, Lightning anchors are a pair and HTLC amounts differ. Small taproot runs are left to the curve check, because the ones on chain are ordinals postage with no data in the outputs. The all-zero burn hash is exempt.How I tested
The -Werror build and the full unit and functional suites pass on this branch merged into the 29.4.3rc1 candidate; the branch carries #319, #422 and #400. ASan/UBSan was clean on the earlier head. The functional tests are in
mempool_data_outputs.py. Over the 3,018,541 transactions since the fork it newly rejects 45: 14 funding transactions of the fake-key and JXL encoders, 14 swap memos continued as ASCII in a P2WPKH hash, 11 ACME-style P2WSH payloads and 6 taproot outputs taggedDECODE WITNESS.Risk and rollback
Policy only. The 14 swap memos are the case a user would notice: the service fills an 80-byte OP_RETURN and puts the rest of the memo in an address, and those swaps would stop relaying. An encoder paying more than 546 sat per chunk is not caught by the run rule.
-rejectfakeoutputs=0turns it off without accepting other carriers.Edit: 2026-10-15 removed the rebase note and updated How I tested for the head rebased for 29.4.3; the rule is unchanged.