Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,13 @@
- **The image hash and the PNG**: `sha256:` and the lowercase hex SHA-256 of the raw RGBA8 bytes
(ADR 0009), with Lumen's own SHA-256; a PNG encoder with stored deflate blocks, whose bytes are
a fixed function of the pixels (`write_png`, `png_len`). (flyingrobots/lumen#2.)
- **A JSON reader that keeps numbers as text** (`parse_json`, `JsonValue`, `JsonLimits`,
`JsonError`, `JSON_READER_ID`; [reference](docs/reference/library.md#the-json-reader)): bytes

Check failure on line 104 in CHANGELOG.md

View workflow job for this annotation

GitHub Actions / Gates / doc links

docs/reference/library.md#the-json-reader: no such heading
in, a value tree out, with no float anywhere. A number is its source text, checked against the
JSON grammar and returned as written, so a later layer converts a decimal exactly. Duplicate
keys, invalid UTF-8, lone surrogates, trailing content, nesting past a depth limit and input
past a size limit are refused, each with a line and a character column. Nothing uses it yet.
(flyingrobots/lumen#211.)
- **Tests**: hand-derived radiance, shadows against exact geometry, every ray and every sample
against brute force and an independent composition, order independence, counts against
oracles other than the counting code, golden image hashes and content digests (one value for
Expand Down
57 changes: 57 additions & 0 deletions crates/lumen/src/json.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
//! A JSON reader that keeps numbers as text (`lumen-json-reader/1`, flyingrobots/lumen#211).
//!
//! [`parse_json`] reads a document from bytes into a [`JsonValue`] tree. It exists so that job
//! files (ADR 0015) and, later, scene files are read by Lumen's own code (R-DEP-1: no third-party
//! crate), and it contains no arithmetic and no float. A number is not converted: the tree holds
//! the token as written, after it has been checked against the grammar of RFC 8259, so a later
//! layer converts a decimal exactly (Bunny's `bunny-decimal-q32-32/1`) and never through a float.
//! The reader reads no file and asks the host nothing; the caller passes the bytes.
//!
//! What it decides, for a document that is valid JSON by RFC 8259:
//!
//! - The input is bytes and must be UTF-8. A byte order mark, a UTF-16 document and any invalid
//! sequence are refused. `\u` escapes are decoded, a surrogate pair into one character, and a
//! lone surrogate is refused.
//! - A duplicate key in one object is refused (RFC 8259 leaves it open). Keys are compared after
//! their escapes are decoded. An object keeps its members in document order.
//! - A number is `-? (0 | [1-9][0-9]*) (. [0-9]+)? ([eE] [+-]? [0-9]+)?` and nothing else. Its
//! size and magnitude are not limited beyond the document's size: `1E400` and a 1 000-digit
//! integer are tokens like any other, and `-0` stays `-0`.
//! - The document is one value with optional white space (space, tab, line feed, carriage
//! return) around it; anything after it is refused.
//! - Limits are parameters ([`JsonLimits`]). A document longer than `max_bytes` is refused
//! before it is read, and a container that would nest deeper than `max_depth` is refused
//! before anything is stored for it. Depth counts open containers, so `max_depth` 2 admits
//! `[[]]` and refuses `[[[]]]`; a scalar has depth 0. The reader does not recurse, and the
//! depth limit has a ceiling ([`JSON_MAX_DEPTH_CEILING`]) because dropping or comparing a deep
//! tree does recurse.
//! - Every error carries a byte offset, a 1-based line and a 1-based column ([`JsonError`]). A
//! column counts characters (Unicode scalar values), not bytes; a line ends at each line feed
//! (a carriage return is a character of its line).
//! - No input panics: every index is checked and every arithmetic step is bounded by the
//! document's length.

mod error;
mod limits;
mod parser;
mod scan;
mod value;

#[cfg(test)]
mod conformance;
#[cfg(test)]
mod generated;
#[cfg(test)]
mod tests;

pub use error::{JsonError, JsonErrorKind};
pub use limits::{
JsonLimitError, JsonLimits, JSON_DEFAULT_MAX_BYTES, JSON_DEFAULT_MAX_DEPTH,
JSON_MAX_DEPTH_CEILING,
};
pub use parser::parse_json;
pub use value::JsonValue;

/// The identity of this reader's contract: the grammar, the UTF-8 rule, the refusals above and
/// the position rule. A change to any of them is a new identity.
pub const JSON_READER_ID: &str = "lumen-json-reader/1";
1 change: 1 addition & 0 deletions crates/lumen/src/json/conformance.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
//! placeholder
139 changes: 139 additions & 0 deletions crates/lumen/src/json/error.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,139 @@
//! Why a document was refused, and where: [`JsonError`].

use core::fmt;

/// What was wrong with a document.
///
/// Non-exhaustive: a later version may report a new failure, so a `match` outside this crate
/// needs a wildcard arm.
#[derive(Clone, Debug, PartialEq, Eq)]
#[non_exhaustive]
pub enum JsonErrorKind {
/// The input is longer than the size limit. The position is that of the first byte past it;
/// nothing was read.
TooLarge {
/// The input's length in bytes.
length: usize,
/// The limit in bytes.
limit: usize,
},
/// A container opens deeper than the depth limit. The position is its opening bracket.
TooDeep {
/// The limit.
limit: usize,
},
/// The input ended where more was needed.
UnexpectedEnd {
/// What was needed.
expected: &'static str,
},
/// A byte that cannot come here.
UnexpectedByte {
/// The byte.
byte: u8,
/// What could come instead.
expected: &'static str,
},
/// A control character (below U+0020) inside a string, which must be escaped.
ControlCharacter {
/// The byte.
byte: u8,
},
/// Bytes inside a string that are not UTF-8. The position is the first bad byte.
InvalidUtf8,
/// A `\u` escape of a surrogate that is not half of a pair. The position is the escape's
/// backslash (the first one, for a high surrogate not followed by a low one).
LoneSurrogate {
/// The escape's code unit.
unit: u16,
},
/// A key that an object already has. The position is the second key's opening quote.
DuplicateKey {
/// The key, decoded.
key: String,
},
}

/// A refused document: what was wrong ([`JsonError::kind`]) and where.
///
/// The position is a byte offset, a line and a column, all pointing at the same place. Lines
/// and columns start at 1; a line ends at each line feed; a column counts characters (Unicode
/// scalar values) from the start of its line, so a character of several bytes is one column.
/// The offset of an error at the end of the input is the input's length.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct JsonError {
kind: JsonErrorKind,
offset: usize,
line: usize,
column: usize,
}

impl JsonError {
/// An error of `kind` at byte `offset` of `bytes`, with its line and column worked out.
pub(super) fn at(bytes: &[u8], offset: usize, kind: JsonErrorKind) -> Self {
let before = bytes.get(..offset).unwrap_or(bytes);
let line = 1 + before.iter().filter(|&&b| b == b'\n').count();
let line_start = before.iter().rposition(|&b| b == b'\n').map_or(0, |i| i + 1);
let column = 1 + before
.get(line_start..)
.unwrap_or_default()
.iter()
.filter(|&&b| b & 0xC0 != 0x80)
.count();
Self { kind, offset, line, column }
}

/// What was wrong.
#[must_use]
pub const fn kind(&self) -> &JsonErrorKind {
&self.kind
}

/// The byte offset, from 0.
#[must_use]
pub const fn offset(&self) -> usize {
self.offset
}

/// The line, from 1.
#[must_use]
pub const fn line(&self) -> usize {
self.line
}

/// The column in characters, from 1.
#[must_use]
pub const fn column(&self) -> usize {
self.column
}
}

impl fmt::Display for JsonError {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
write!(f, "line {}, column {}: ", self.line, self.column)?;
match &self.kind {
JsonErrorKind::TooLarge { length, limit } => {
write!(f, "the input is {length} bytes, over the limit of {limit}")
}
JsonErrorKind::TooDeep { limit } => {
write!(f, "nesting deeper than the limit of {limit}")
}
JsonErrorKind::UnexpectedEnd { expected } => {
write!(f, "the input ends where {expected} was expected")
}
JsonErrorKind::UnexpectedByte { byte, expected } => {
write!(f, "byte 0x{byte:02X} where {expected} was expected")
}
JsonErrorKind::ControlCharacter { byte } => {
write!(f, "control character 0x{byte:02X} in a string; escape it")
}
JsonErrorKind::InvalidUtf8 => write!(f, "bytes that are not UTF-8"),
JsonErrorKind::LoneSurrogate { unit } => {
write!(f, "the escape of surrogate U+{unit:04X} is not half of a pair")
}
JsonErrorKind::DuplicateKey { key } => write!(f, "the key {key:?} appears twice"),
}
}
}

impl std::error::Error for JsonError {}
1 change: 1 addition & 0 deletions crates/lumen/src/json/generated.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
//! placeholder
94 changes: 94 additions & 0 deletions crates/lumen/src/json/limits.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
//! The reader's limits: [`JsonLimits`].

use core::fmt;

/// The default depth limit: 128 open containers, far past any job or scene file and well inside
/// [`JSON_MAX_DEPTH_CEILING`].
pub const JSON_DEFAULT_MAX_DEPTH: usize = 128;

/// The default size limit: 16 MiB (`2^24` bytes) of input. A scene file with its meshes inline
/// would need more; its reader passes its own limit.
pub const JSON_DEFAULT_MAX_BYTES: usize = 1 << 24;

/// The largest depth limit [`JsonLimits::new`] accepts. The reader itself does not recurse, but
/// the tree's `Drop`, `Clone`, `Debug` and `PartialEq` do, one stack frame per level, so the
/// limit keeps them inside a test thread's 2 MiB stack.
pub const JSON_MAX_DEPTH_CEILING: usize = 1024;

/// The limits of one read: how deep containers may nest and how long the input may be.
///
/// # Examples
///
/// ```
/// use lumen::{parse_json, JsonLimits};
///
/// let limits = JsonLimits::new(2, 64)?;
/// assert!(parse_json(b"[[]]", limits).is_ok());
/// assert!(parse_json(b"[[[]]]", limits).is_err());
/// # Ok::<(), lumen::JsonLimitError>(())
/// ```
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct JsonLimits {
max_depth: usize,
max_bytes: usize,
}

impl JsonLimits {
/// Limits of `max_depth` open containers (0 admits scalars only) and `max_bytes` bytes of
/// input (the input may be exactly that long).
///
/// # Errors
///
/// [`JsonLimitError`] if `max_depth` is above [`JSON_MAX_DEPTH_CEILING`].
pub const fn new(max_depth: usize, max_bytes: usize) -> Result<Self, JsonLimitError> {
if max_depth > JSON_MAX_DEPTH_CEILING {
return Err(JsonLimitError { max_depth });
}
Ok(Self { max_depth, max_bytes })
}

/// The deepest nesting of containers admitted.
#[must_use]
pub const fn max_depth(self) -> usize {
self.max_depth
}

/// The longest input admitted, in bytes.
#[must_use]
pub const fn max_bytes(self) -> usize {
self.max_bytes
}
}

impl Default for JsonLimits {
/// [`JSON_DEFAULT_MAX_DEPTH`] and [`JSON_DEFAULT_MAX_BYTES`].
fn default() -> Self {
Self { max_depth: JSON_DEFAULT_MAX_DEPTH, max_bytes: JSON_DEFAULT_MAX_BYTES }
}
}

/// A depth limit above [`JSON_MAX_DEPTH_CEILING`] was asked for.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct JsonLimitError {
max_depth: usize,
}

impl JsonLimitError {
/// The depth limit that was refused.
#[must_use]
pub const fn max_depth(self) -> usize {
self.max_depth
}
}

impl fmt::Display for JsonLimitError {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
write!(
f,
"a JSON depth limit of {} is above the ceiling of {JSON_MAX_DEPTH_CEILING}",
self.max_depth
)
}
}

impl std::error::Error for JsonLimitError {}
Loading
Loading