Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,11 @@ part was built.
- **The image hash and the PNG**: `sha256:` and the lowercase hex SHA-256 of the raw RGBA8 bytes
(ADR 0009), with Lumen's own SHA-256; a PNG encoder with stored deflate blocks, whose bytes are
a fixed function of the pixels (`write_png`, `png_len`). (flyingrobots/lumen#2.)
- **Image decoding, `lumen-image`** (ADR 0012): a workspace crate with no dependency and no
float, inside the deterministic core, that `lumen` depends on. It decompresses a zlib stream
(RFC 1950 and 1951: stored, fixed- and dynamic-Huffman blocks, the Adler-32 check) into a
caller-limited output, refusing every malformed stream with a typed error and its byte
offset. (flyingrobots/lumen#195.)
- **Tests**: hand-derived radiance, shadows against exact geometry, every ray and every sample
against brute force and an independent composition, order independence, counts against
oracles other than the counting code, golden image hashes and content digests (one value for
Expand Down
15 changes: 15 additions & 0 deletions crates/lumen-image/Cargo.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
[package]
name = "lumen-image"
description = "Lumen's image decoders: integer-only, with no dependency (ADR 0012)."
authors.workspace = true
edition.workspace = true
license.workspace = true
repository.workspace = true
rust-version.workspace = true
version.workspace = true
keywords = ["zlib", "inflate", "deterministic"]
categories = ["compression"]
publish = false

[lints]
workspace = true
144 changes: 144 additions & 0 deletions crates/lumen-image/src/inflate.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,144 @@
//! This module owns zlib decompression: the RFC 1950 frame (a two-byte header, deflate blocks,
//! an Adler-32 trailer) and the entry points.
//!
//! The rules the decoder holds to, beyond the RFCs' own:
//!
//! - **A preset dictionary is refused** (`FDICT` set), by name: no caller supplies one.
//! - **Distances are checked twice**: against the bytes decoded so far, and against the window
//! the header declares (`CINFO`, 256 to 32 768 bytes), as zlib's own decoder does.
//! - **Incomplete codes**: a literal/length or distance code whose lengths do not fill the code
//! space is refused, except a code of exactly one symbol of length 1 (RFC 1951 section 3.2.7
//! allows one distance code of one bit; zlib accepts the same shape in either alphabet) and a
//! distance code with no symbols at all (a block of literals only). Reading the unused bit
//! pattern of such a code is refused. The code-length code must be complete.
//! - **Trailing bytes are an error**: the stream must end at its Adler-32 trailer.
//!
//! Byte offsets in errors count from the start of the stream (for [`inflate_zlib_chunks`], of
//! the chunks' concatenation). A fault is reported at the byte holding the last bit read when
//! it was found; a truncation at the stream's length, the first byte that is missing.

mod adler32;
mod bits;
mod blocks;
mod error;
mod huffman;
mod output;
#[cfg(test)]
mod tests;

pub use error::{Alphabet, InflateError, InflateErrorKind};

use bits::BitReader;
use output::Output;

/// The identity of the decoder's contract: which zlib streams it accepts (the rules above and
/// in [`InflateErrorKind`]) and what it returns for them. An accepted stream's output is fixed
/// by RFC 1951; a change to what is refused makes a new identity.
pub const INFLATE_ID: &str = "lumen-zlib-inflate/1";

/// The most bytes a decode may produce, from the caller.
///
/// A stream whose output would pass it is refused with [`InflateErrorKind::OutputLimit`] as soon
/// as a block declares or decodes the first byte beyond it, and the output's buffer never
/// requests capacity beyond it.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct OutputLimit {
bytes: usize,
}

impl OutputLimit {
/// A limit of `bytes` decoded bytes; zero admits only an empty stream.
#[must_use]
pub const fn new(bytes: usize) -> Self {
Self { bytes }
}

/// The limit in bytes.
#[must_use]
pub const fn bytes(self) -> usize {
self.bytes
}
}

/// Decompresses the zlib stream `stream` into at most `limit` bytes.
///
/// Allocates the returned vector, at most `limit` bytes of capacity; reads nothing outside
/// `stream`.
///
/// # Errors
///
/// An [`InflateError`] for a malformed or truncated stream, a bad Adler-32 check, bytes after
/// the trailer, or an output beyond `limit`; its [`kind`](InflateError::kind) names the fault.
///
/// # Panics
///
/// Does not panic.
pub fn inflate_zlib(stream: &[u8], limit: OutputLimit) -> Result<Vec<u8>, InflateError> {
inflate_zlib_chunks(&[stream], limit)
}

/// Decompresses one zlib stream stored as the concatenation of `chunks`, in order, into at most
/// `limit` bytes: a PNG file's `IDAT` chunk data, passed without copying them together. Empty
/// chunks are allowed, and a chunk boundary may fall anywhere, even inside a code. Error offsets
/// count from the start of the first chunk.
///
/// # Errors
///
/// As [`inflate_zlib`].
///
/// # Panics
///
/// Does not panic.
pub fn inflate_zlib_chunks(
chunks: &[&[u8]],
limit: OutputLimit,
) -> Result<Vec<u8>, InflateError> {
let mut input = BitReader::new(chunks);
let window = header(&mut input)?;
let mut output = Output::new(limit.bytes, window);
blocks::inflate(&mut input, &mut output)?;
trailer(&mut input, output.bytes())?;
Ok(output.into_bytes())
}

/// Reads the two header bytes (RFC 1950 section 2.2) and returns the window size they declare.
fn header(input: &mut BitReader<'_>) -> Result<usize, InflateError> {
let cmf = input.byte()?;
let flg = input.byte()?;
let at = |kind| InflateError::new(kind, input.last_offset());
if (u16::from(cmf) << 8 | u16::from(flg)) % 31 != 0 {
return Err(at(InflateErrorKind::HeaderCheck { cmf, flg }));
}
if cmf & 0x0f != 8 {
return Err(at(InflateErrorKind::CompressionMethod { method: cmf & 0x0f }));
}
let cinfo = cmf >> 4;
if cinfo > 7 {
return Err(at(InflateErrorKind::WindowSize { cinfo }));
}
if flg & 0x20 != 0 {
return Err(at(InflateErrorKind::PresetDictionary));
}
Ok(1 << (cinfo + 8))
}

/// Reads the Adler-32 trailer after the last block (RFC 1950 section 2.2), checks it against
/// `decoded` and refuses anything after it.
fn trailer(input: &mut BitReader<'_>, decoded: &[u8]) -> Result<(), InflateError> {
input.align();
let mut stored = 0_u32;
for _ in 0..4 {
stored = stored << 8 | u32::from(input.byte()?);
}
let computed = adler32::adler32(decoded);
if stored != computed {
let kind = InflateErrorKind::Adler32Mismatch { stored, computed };
return Err(InflateError::new(kind, input.last_offset()));
}
let trailing = input.remaining();
if trailing != 0 {
let kind = InflateErrorKind::TrailingBytes { count: trailing };
return Err(InflateError::new(kind, input.next_offset()));
}
Ok(())
}
24 changes: 24 additions & 0 deletions crates/lumen-image/src/inflate/adler32.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
//! This module owns the Adler-32 checksum of RFC 1950 section 8.

/// The largest prime below 2^16.
const MODULUS: u32 = 65_521;

/// The most bytes summed before reducing: the largest `n` with
/// `255 n (n + 1) / 2 + (n + 1) (MODULUS - 1) < 2^32`, so neither sum can overflow (zlib's
/// `NMAX`).
const RUN: usize = 5_552;

/// The Adler-32 of `bytes`: the high half is the sum of the running sums, the low half one
/// plus the sum of the bytes, both modulo 65 521.
pub(super) fn adler32(bytes: &[u8]) -> u32 {
let (mut low, mut high) = (1_u32, 0_u32);
for run in bytes.chunks(RUN) {
for &byte in run {
low += u32::from(byte);
high += low;
}
low %= MODULUS;
high %= MODULUS;
}
high << 16 | low
}
99 changes: 99 additions & 0 deletions crates/lumen-image/src/inflate/bits.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
//! This module owns reading the input: bytes from a sequence of slices, and bits from those
//! bytes least significant first (RFC 1951 section 3.1.1).

use super::error::{InflateError, InflateErrorKind};

/// A reader of bits and bytes over the concatenation of its chunks.
///
/// Invariant: `held < 8` between calls, so the last byte taken holds every buffered bit, and
/// the bit most recently read is in byte `taken - 1`.
pub(super) struct BitReader<'a> {
chunks: &'a [&'a [u8]],
/// The chunk being read, and the position in it.
chunk: usize,
at: usize,
/// Bytes taken from the input so far.
taken: usize,
/// Bits taken but not yet read, the next one lowest, and how many.
buffer: u32,
held: u32,
}

impl<'a> BitReader<'a> {
pub(super) fn new(chunks: &'a [&'a [u8]]) -> Self {
Self { chunks, chunk: 0, at: 0, taken: 0, buffer: 0, held: 0 }
}

/// The offset of the byte that holds the last bit read (zero before any).
pub(super) fn last_offset(&self) -> usize {
self.taken.saturating_sub(1)
}

/// The offset of the next byte to be taken.
pub(super) fn next_offset(&self) -> usize {
self.taken
}

/// The next whole byte, after the bits already read are discarded by [`Self::align`].
pub(super) fn byte(&mut self) -> Result<u8, InflateError> {
while let Some(chunk) = self.chunks.get(self.chunk) {
if let Some(&byte) = chunk.get(self.at) {
self.at += 1;
self.taken = self.taken.saturating_add(1);
return Ok(byte);
}
self.chunk += 1;
self.at = 0;
}
Err(InflateError::new(InflateErrorKind::Truncated, self.taken))
}

/// The next `count` bits (at most 16) as an integer, the first read lowest.
pub(super) fn bits(&mut self, count: u32) -> Result<u32, InflateError> {
debug_assert!(count <= 16, "at most 16 bits at a time");
while self.held < count {
// `held < count <= 16`, so the byte lands below bit 24.
self.buffer |= u32::from(self.byte()?) << self.held;
self.held += 8;
}
let value = self.buffer & ((1 << count) - 1);
self.buffer >>= count;
self.held -= count;
Ok(value)
}

/// Discards the bits left in the current byte, so the next read starts on a byte boundary.
pub(super) fn align(&mut self) {
self.buffer = 0;
self.held = 0;
}

/// Copies the next `count` whole bytes into `out`, as a stored block's data. Call after
/// [`Self::align`].
pub(super) fn copy(&mut self, count: usize, out: &mut Vec<u8>) -> Result<(), InflateError> {
let mut left = count;
while left > 0 {
let Some(chunk) = self.chunks.get(self.chunk) else {
return Err(InflateError::new(InflateErrorKind::Truncated, self.taken));
};
let rest = chunk.get(self.at..).unwrap_or_default();
let take = rest.len().min(left);
out.extend_from_slice(rest.get(..take).unwrap_or_default());
self.at += take;
self.taken = self.taken.saturating_add(take);
left -= take;
if left > 0 {
self.chunk += 1;
self.at = 0;
}
}
Ok(())
}

/// The bytes not yet taken, in every chunk.
pub(super) fn remaining(&self) -> usize {
let current = self.chunks.get(self.chunk).map_or(0, |c| c.len().saturating_sub(self.at));
let later = self.chunks.get(self.chunk + 1..).unwrap_or_default();
later.iter().fold(current, |sum, c| sum.saturating_add(c.len()))
}
}
Loading
Loading