Skip to content

perf: zero-extend big-endian bit fields with load_be - #666

Merged
wcampbell0x2a merged 1 commit into
sharksforarms:masterfrom
Simon-Calbert-Aerospacelab:perf/be-bits-load-be
Aug 8, 2026
Merged

perf: zero-extend big-endian bit fields with load_be#666
wcampbell0x2a merged 1 commit into
sharksforarms:masterfrom
Simon-Calbert-Aerospacelab:perf/be-bits-load-be

Conversation

@Simon-Calbert-Aerospacelab

@Simon-Calbert-Aerospacelab Simon-Calbert-Aerospacelab commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Closes #665.

Reading a #[deku(bits = N)] field zero-extends it to the container width with:

for _ in 0..(MAX_TYPE_BITS - bits.len()) {
    bits.insert(0, false);
}

BoundedBitVec::insert shifts the whole backing array right by one bit per call,
bit by bit, because the split_at_mut alias defeats bitvec's word-at-a-time
shift_right. That makes a field O(container_bits * (container_bits - N)) bit
operations: a 1-bit field in a u64 pays 63 whole-array shifts, about 2 us.

BitField::load_be does the same zero-extension in one step, and this file
already uses it in the Lsb0 branch below.

Change

Two hunks in src/impls/primitive.rs, one per DekuRead::read impl
((Endian, BitSize, Order) and (Endian, BitSize)):

if !input_is_le && order == Order::Msb0 && bit_size > 0 {
    let value = bit_slice.load_be::<$inner>();
    return Ok((bit_size, <$typ>::from_be_bytes(value.to_be_bytes())));
}

f32 and f64 are not funty::Integral, so the value goes through the unsigned
$inner and from_be_bytes, as the existing padded-array path does.
bit_size > 0 leaves bits = 0 panicking in insert as before.

Adds benches/bebits.rs, since the existing DekuBits bench declares no endian
and never reaches this path.

Scope

Explicit endian = "big" with the default Msb0 only. Untouched: no endian
attribute (which defaults to target endianness, so little-endian on x86),
endian = "little", and any bit_order = "lsb".

I left the little-endian side alone on purpose. #658 is an open correctness bug in
the Lsb0 + Endian::Little branch, so mixing a performance change into it seemed
unwise. It is fixable the same way, building ceil(N/8) bytes with the final
partial byte right-aligned and combining little-endian, once the intended
semantics are settled. Happy to follow up.

Testing

The full CI feature matrix passes and all nine examples run. tests/bit_order.rs
and tests/test_lsb_le.rs stay green.

A differential sweep of the old conversion against the new one, over u8, u16,
u32 and u64, every valid width and 40k random inputs: 4.8M cases, 0
mismatches.

Two jobs are already red on unpatched master, so they are not from this PR:

  • test_compile: the trybuild expected stderr does not match newer rustc.
    Verified identical by reverting only primitive.rs.
  • cargo clippy -- -D warnings: src/lib.rs:983 calls shift_right, deprecated
    in bitvec 1.1.1, which the "1.0.1" requirement now resolves to. Can fix
    separately if you want.

Performance

cargo bench --all-features --bench bebits, against a master baseline:

Bench master patched change
be_tm_primary_header_11_fields 2827 ns 712 ns -74.7%
be_one_bit_in_u64 2050 ns 79.1 ns -96.1%
be_six_bytes_aligned (byte-aligned control) 15.9 ns 15.8 ns none, p = 0.19

--bench deku is unchanged: deku_read_bits 740 to 729 ns (p = 0.34). It did
report +2.2% on deku_read_byte, but the same binary run twice moves that one
-3.0%, so run-to-run drift on an 8 ns benchmark is larger than the effect.

rustc 1.96.0, opt-level 3, no LTO, AMD EPYC 9374F.

@wcampbell0x2a

Copy link
Copy Markdown
Collaborator

I bet we can do something like this for LE+LSB.

@Simon-Calbert-Aerospacelab

Copy link
Copy Markdown
Contributor Author

Good instinct, and I tried it before replying. It does not carry over, and the
reason turned out to be worth reporting.

This PR fixes a padding problem. With big-endian + Msb0 the bits arrive in
the order the number wants them, so the only work left is putting zeros in
front, and load_be is exactly that operation. That is why the swap is
provably equivalent rather than just close.

Lsb0 is not a padding problem. It needs a byte reordering (the existing code
walks bytes backwards with rchunks_exact(8) while reading the bits inside each
one forwards), and neither load_be nor load_le can express that.

I also benchmarked where the pad loop actually bites, by reading a 1-bit field
into containers of increasing width:

container BE+Msb0 (this PR) BE+Lsb0 LE+Msb0 LE+Lsb0
u8 77.6 ns 323 ns 300 ns 316 ns
u64 79.7 ns 2123 ns 373 ns 373 ns

Little-endian is flat, which matches the source: the pad loop appends with
push when the input is little-endian and only prepends with insert(0, ..)
when it is big-endian. So LE never had the #665 behaviour, which is probably
why the dual felt forced. LE is still ~4.7x slower than this PR's path, but that
is a flat overhead in the generic assembly path, a different problem.

The quadrant that does still have the #665 defect is BE+Lsb0: 27x above, and
70x for a byte-aligned 16-bit field in a u64 (22.5 ns vs 1572 ns). Dropping
order == Order::Msb0 from the guard breaks test_bit_order_more_first_be and
test_idempotency_multi_byte, for the reordering reason above.

I would hold off on that quadrant anyway: #657's skip correction is
big-endian specific and it widens the branch to all Lsb0, so it changes BE+Lsb0
output too. Optimizing there today means validating against behaviour that is
about to change.

Happy to take BE+Lsb0 as a follow-up once #657 lands, with the same
bit-identical sweep I used here. Suggest keeping this PR scoped to BE+Msb0, the
one quadrant where the equivalence is provable today.

@Simon-Calbert-Aerospacelab

Copy link
Copy Markdown
Contributor Author

Follow-up worth flagging: I applied #657 on top of this branch, and it removes
the #665 defect from the Lsb0 column by itself.

The reason is that the expensive loop lives in the generic branch, the one
that pads all the way up to the container width with insert(0, false). Today
Lsb0 only reaches the cheaper Lsb0-specific branch when
len > 8 && pad != 0, so everything else falls through to it. #657 widens that
condition to plain order == Order::Lsb0, and the fall-through stops.

Measured on top of this PR:

BE+Lsb0 case before #657 with #657
1-bit field in a u64 2123 ns 374 ns
byte-aligned 16-bit field in a u64 1572 ns 62.9 ns

The two patches apply cleanly together and pass bit_order 21/21 and
test_lsb_le 9/9.

So BE+Lsb0 might need no performance patch of its own, and these two can merge
independently. Once both are in, the #665 defect (quadratic complexity over the container size) is gone from every quadrant,
and what remains is a uniform ~4.7x from the BoundedBitVec assembly itself.

@Simon-Calbert-Aerospacelab

Copy link
Copy Markdown
Contributor Author

@wcampbell0x2a FYI I have two follow-up PRs sitting on my fork that take the
same BE + Msb0 path further. Happy to open them here once this one lands.

  1. Read and write bit fields as integers. This PR made the zero-extension
    cheap. That one goes a level up and skips the BitSlice entirely: the field
    is read straight into a u64 and written straight out of one.
  2. Batch a run of adjacent fields in the derive. Neighbouring big-endian
    Msb0 fields that add up to 64 bits or less become a single
    read/write call plus a shift and mask per field. The 11-field CCSDS TM
    header becomes one 48-bit read.

Numbers below are per-item cost, measured on benches that push 128 frames
through a single reader or writer so nothing overlaps or gets optimised away.
Each column is the gain over the column to its left; the last is against
current master.

#666 alone +1 over #666 +2 over +1 all three
TM header, read 3.7x 29x 4.1x 441x
TM header, write 1.0x 2.8x 10x 29x
1-bit field in a u64, read 31x 41x 1.0x 1239x
6 plain byte fields, read 1.0x 1.0x 2.7x 2.7x
6 plain byte fields, write 1.0x 1.0x 1.9x 1.9x

Worth noting this PR is the only one of the three that fixes an asymptotic
problem rather than a constant factor, which is the 31x on the 1-bit case. It
also does nothing for writes, since it is a read-side fix; all the write gains
come from the follow-ups.

Is there anything you would like changed here before it can go in? And do you
have a rough idea of the timing? I would rather rebase the follow-ups onto a
merged master than maintain them as a stack.

@wcampbell0x2a

Copy link
Copy Markdown
Collaborator

@Simon-Calbert-Aerospacelab I can't give you a time-table of merging. I am only the co-maintainer.

@wcampbell0x2a
wcampbell0x2a merged commit 15358e5 into sharksforarms:master Aug 8, 2026
5 of 8 checks passed
@Simon-Calbert-Aerospacelab

Simon-Calbert-Aerospacelab commented Aug 10, 2026

Copy link
Copy Markdown
Contributor Author

Follow-up to this is now open: #673. Same quadrant, one level up, it skips
the BitSlice entirely rather than just making the zero-extension cheap.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Reading a bits = N field is quadratic in the container width: the insert(0, false) pad loop

2 participants