Skip to content

feat: align encoder with TOON spec v4.1 (v0.7.0) - #32

Merged
coseto6125 merged 2 commits into
mainfrom
feat/spec-v4.1
Aug 1, 2026
Merged

feat: align encoder with TOON spec v4.1 (v0.7.0)#32
coseto6125 merged 2 commits into
mainfrom
feat/spec-v4.1

Conversation

@coseto6125

Copy link
Copy Markdown
Owner

Upstream moved the spec into its own repo (toon-format/spec) and shipped v4.0 (breaking) and v4.1 since etoon's v3.1 alignment. This PR implements the encoder-side changes and swaps our fixtures for the official suite.

New forms

Nested field groups (§9.3) — a column of uniform objects collapses into the header:

orders[2]{id,customer{name,country},total}:
  1,Ada,DK,99
  2,Bob,UK,149

Keyed tabular (§9.5) — an object of uniform objects becomes a table whose rows carry their own keys:

servers[2:]{host,port}:
  alpha: a.example.com,8080
  beta: b.example.com,9090

Keyless [N:]{…}: at the document root; never for array elements, which stay anonymous (§10).

Encoded-size reduction against the previous nested output: 76.6% (nested field groups) and 31.9% (keyed tabular) on the benchmark payloads.

Fixes

Issue Spec
Tabular form is mandatory when detection holds, except in list-item position where a keyless fields-bearing header is invalid — that case emitted an illegal - [2]{x}: header §9.3, §9.4, §6
"+1" was emitted unquoted and decoded back as a number §7.2
Brackets and braces were only checked at position 0, so x[1] went out unquoted §7.2

Fixtures

tests/fixtures/encode/ now tracks the official toon-format/spec suite (MIT, 179 encode cases). 178 pass; the single skip needs a non-default indentSize, which etoon hardcodes to 2. key-folding.json stays etoon-local — v4.0 removed key folding from the spec, so fold_keys is an etoon extension now. See ATTRIBUTION.md.

Upstream's rationale for removing key folding is in .out-of-scope/key-folding.md: 0.00% token savings on the reference benchmarks, wire ambiguity against literal dotted keys, and incompatibility with streaming decode.

Verification

  • cargo test --no-default-features — 32 pass
  • pytest tests/test_spec_fixtures.py — 197 pass
  • cargo clippy --features python --all-targets -- -D warnings — clean
  • benches/compare.py — output byte-identical to @toon-format/toon 4.1 on all six payloads (toons and py-rtoon now differ on the two v4.1 shapes; they are still on spec v3.x)

Performance (Rust core, A/B against the previous encoder): flat tabular unchanged (155.9 µs → 156.8 µs), deep nested unchanged. Detection costs ~4% on payloads with small object fields that do not qualify (mixed_500: 130.8 µs → 135.9 µs); a first-row column probe rejects those in O(columns) instead of walking every row, which brought that number down from +13%.

Breaking output change, so this is a minor bump to 0.7.0.

Upstream moved the spec into its own repo (toon-format/spec) and shipped v4.0
(breaking) and v4.1 since etoon's v3.1 alignment. This implements the
encoder-side changes.

New forms:
- Nested field groups (§9.3): a column of uniform objects collapses into
  `orders[2]{id,customer{name,country},total}:` with depth-first row cells.
  Cuts encoded size 76.6% on the benchmark payload.
- Keyed tabular (§9.5): an object of uniform objects becomes
  `servers[2:]{host,port}:` with one entry row per key. Cuts encoded size
  31.9%. Keyless `[N:]{...}:` at the root; never for array elements (§10).

Fixes:
- Tabular form is mandatory when detection holds (§9.3), except in list-item
  position where a keyless fields-bearing header is invalid (§9.4). That case
  previously emitted an illegal `- [2]{x}:` header.
- Quote leading-plus numeric-like strings (§7.2): `"+1"` decoded back as a
  number.
- Quote brackets and braces anywhere in a value, not only at position 0 (§7.2).

Fixtures now come from the official toon-format/spec suite (MIT, 179 encode
cases); 178 pass and the one skip needs a non-default indentSize.
key-folding.json stays local: v4.0 removed key folding from the spec, so
fold_keys is an etoon extension now.

Output is byte-identical to @toon-format/toon 4.1 across all six benchmark
payloads. Flat tabular encoding is unchanged in speed; detection costs ~4% on
payloads with small non-qualifying object fields, held down by a first-row
column probe that rejects in O(columns).
clippy 1.97 flags the `Some/None` match as `question_mark`; the local
toolchain (1.95) has no such lint, so CI caught it. Behaviour is unchanged: an
unterminated string still ends the scan.
@coseto6125
coseto6125 merged commit dc036e8 into main Aug 1, 2026
5 checks passed
@coseto6125
coseto6125 deleted the feat/spec-v4.1 branch August 1, 2026 01:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant