feat: align encoder with TOON spec v4.1 (v0.7.0) - #32
Merged
Conversation
Upstream moved the spec into its own repo (toon-format/spec) and shipped v4.0
(breaking) and v4.1 since etoon's v3.1 alignment. This implements the
encoder-side changes.
New forms:
- Nested field groups (§9.3): a column of uniform objects collapses into
`orders[2]{id,customer{name,country},total}:` with depth-first row cells.
Cuts encoded size 76.6% on the benchmark payload.
- Keyed tabular (§9.5): an object of uniform objects becomes
`servers[2:]{host,port}:` with one entry row per key. Cuts encoded size
31.9%. Keyless `[N:]{...}:` at the root; never for array elements (§10).
Fixes:
- Tabular form is mandatory when detection holds (§9.3), except in list-item
position where a keyless fields-bearing header is invalid (§9.4). That case
previously emitted an illegal `- [2]{x}:` header.
- Quote leading-plus numeric-like strings (§7.2): `"+1"` decoded back as a
number.
- Quote brackets and braces anywhere in a value, not only at position 0 (§7.2).
Fixtures now come from the official toon-format/spec suite (MIT, 179 encode
cases); 178 pass and the one skip needs a non-default indentSize.
key-folding.json stays local: v4.0 removed key folding from the spec, so
fold_keys is an etoon extension now.
Output is byte-identical to @toon-format/toon 4.1 across all six benchmark
payloads. Flat tabular encoding is unchanged in speed; detection costs ~4% on
payloads with small non-qualifying object fields, held down by a first-row
column probe that rejects in O(columns).
clippy 1.97 flags the `Some/None` match as `question_mark`; the local toolchain (1.95) has no such lint, so CI caught it. Behaviour is unchanged: an unterminated string still ends the scan.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Upstream moved the spec into its own repo (toon-format/spec) and shipped v4.0 (breaking) and v4.1 since etoon's v3.1 alignment. This PR implements the encoder-side changes and swaps our fixtures for the official suite.
New forms
Nested field groups (§9.3) — a column of uniform objects collapses into the header:
Keyed tabular (§9.5) — an object of uniform objects becomes a table whose rows carry their own keys:
Keyless
[N:]{…}:at the document root; never for array elements, which stay anonymous (§10).Encoded-size reduction against the previous nested output: 76.6% (nested field groups) and 31.9% (keyed tabular) on the benchmark payloads.
Fixes
- [2]{x}:header"+1"was emitted unquoted and decoded back as a numberx[1]went out unquotedFixtures
tests/fixtures/encode/now tracks the officialtoon-format/specsuite (MIT, 179 encode cases). 178 pass; the single skip needs a non-defaultindentSize, which etoon hardcodes to 2.key-folding.jsonstays etoon-local — v4.0 removed key folding from the spec, sofold_keysis an etoon extension now. See ATTRIBUTION.md.Upstream's rationale for removing key folding is in
.out-of-scope/key-folding.md: 0.00% token savings on the reference benchmarks, wire ambiguity against literal dotted keys, and incompatibility with streaming decode.Verification
cargo test --no-default-features— 32 passpytest tests/test_spec_fixtures.py— 197 passcargo clippy --features python --all-targets -- -D warnings— cleanbenches/compare.py— output byte-identical to@toon-format/toon4.1 on all six payloads (toons and py-rtoon now differ on the two v4.1 shapes; they are still on spec v3.x)Performance (Rust core, A/B against the previous encoder): flat tabular unchanged (155.9 µs → 156.8 µs), deep nested unchanged. Detection costs ~4% on payloads with small object fields that do not qualify (mixed_500: 130.8 µs → 135.9 µs); a first-row column probe rejects those in O(columns) instead of walking every row, which brought that number down from +13%.
Breaking output change, so this is a minor bump to 0.7.0.