feat: align encoder with TOON spec v4.1 (v0.7.0)

ATTRIBUTION.md

@@ -2,19 +2,27 @@
 
 ## Test Fixtures
 
-`tests/fixtures/encode/*.json` are copied from the [toons](https://github.com/alesanfra/toons)
-project by Alessandro Sanfratello, licensed under the Apache License 2.0.
+`tests/fixtures/encode/*.json` (except `key-folding.json`) are copied from the
+[toon-format/spec](https://github.com/toon-format/spec) repository, licensed
+under the MIT License. They are the official language-agnostic encode fixture
+suite for TOON spec v4.1. No modifications have been made to the fixture
+contents.
 
-These fixtures implement the TOON v1.4 encoding specification test suite.
-No modifications have been made to the fixture contents.
+`key-folding.json` is etoon-local. It started from the
+[toons](https://github.com/alesanfra/toons) project by Alessandro Sanfratello
+(Apache License 2.0) and carries etoon's own `@`/`$`/`#` sigil-prefix cases. The
+spec removed key folding in v4.0, so this file guards etoon's `fold_keys`
+extension rather than a spec requirement.
 
 ## TOON Specification
 
-The TOON (Token-Oriented Object Notation) format is defined by the
-[toon-format/toon](https://github.com/toon-format/toon) project (MIT License).
+The TOON (Token-Oriented Object Notation) format is defined by
+[toon-format/spec](https://github.com/toon-format/spec) (MIT License); the
+reference TypeScript implementation lives in
+[toon-format/toon](https://github.com/toon-format/toon) (MIT License).
 
 ## License
 
-This project is licensed under the Apache License 2.0, consistent with the
-upstream `toons` project to facilitate fixture reuse and potential contributions
-back to the TOON ecosystem.
+This project is licensed under the Apache License 2.0, which is compatible with
+both the MIT-licensed spec fixtures and the Apache 2.0 `toons` fixtures it
+reuses.

Cargo.toml

@@ -1,8 +1,8 @@
 [package]
 name = "etoon"
-version = "0.6.0"
+version = "0.7.0"
 edition = "2021"
-description = "Fast TOON (Token-Oriented Object Notation) encoder. 8x faster than toons, 2.7x faster than the official TS SDK."
+description = "Fast TOON (Token-Oriented Object Notation) encoder tracking spec v4.1. Up to 7.6x faster than toons, 3.0-8.3x faster than the official TS SDK."
 license = "Apache-2.0"
 repository = "https://github.com/coseto6125/etoon"
 keywords = ["toon", "serialization", "llm", "json", "encoder"]

README.md

@@ -7,25 +7,30 @@
 
 Fast [TOON](https://github.com/toon-format/toon) (Token-Oriented Object Notation) encoder for Python, Rust, and CLI.
 
-**8× faster than `toons`**, **2.7× faster than the official TS SDK**, byte-identical output.
+**Up to 7.6× faster than `toons`**, **3.0–8.3× faster than the official TS SDK**, byte-identical output — tracking **TOON spec v4.1**.
 
 [中文說明](https://github.com/coseto6125/etoon/blob/main/README.zh-TW.md)
 
 ## Performance
 
 Per-call encode time across representative payloads (`etoon` = Python/PyO3,
-best-of-7 × 2000 calls). `✓` = output byte-identical to etoon; `✗` = the
-encoder deviates from the TOON spec (e.g. py-rtoon emits `0.0` where the spec
-requires `0`).
-
-| Payload (encode)        | etoon   | toons          | py-rtoon       | @toon-format/toon (TS) |
-|-------------------------|---------|----------------|----------------|------------------------|
-| 1000 uniform objects    | 169 µs  | 888 µs (5.2×✓) | 868 µs (5.1×✗) | 455 µs (2.7×✓)         |
-| deep nested             | 123 µs  | 291 µs (2.4×✓) | 737 µs (6.0×✗) | 602 µs (4.9×✓)         |
-| 1000 string records     | 93 µs   | 747 µs (8.0×✓) | 596 µs (6.4×✓) | 640 µs (6.9×✓)         |
-| 500 mixed objects       | 136 µs  | 755 µs (5.5×✓) | 599 µs (4.4×✗) | 1165 µs (8.6×✓)        |
-
-**2.4–8.6× faster** than every other encoder, with **byte-identical, spec-canonical** output (toons and the TS SDK match byte-for-byte; py-rtoon does not).
+best-of-7 × 400 calls). `✓` = output byte-identical to etoon; `✗` = the encoder
+differs — either it deviates from the spec (py-rtoon emits `0.0` where the spec
+requires `0`) or it still implements spec v3.x and expands the v4.1 collapsing
+forms into nested blocks.
+
+| Payload (encode)         | etoon   | toons           | py-rtoon        | @toon-format/toon 4.1 (TS) |
+|--------------------------|---------|-----------------|-----------------|----------------------------|
+| 1000 uniform objects     | 171 µs  | 848 µs (5.0×✓)  | 772 µs (4.5×✗)  | 519 µs (3.0×✓)             |
+| deep nested              | 110 µs  | 272 µs (2.5×✓)  | 662 µs (6.0×✗)  | 611 µs (5.6×✓)             |
+| 1000 string records      | 89 µs   | 674 µs (7.6×✓)  | 540 µs (6.1×✓)  | 640 µs (7.2×✓)             |
+| 500 mixed objects        | 140 µs  | 680 µs (4.8×✓)  | 541 µs (3.9×✗)  | 1166 µs (8.3×✓)            |
+| 1000 nested field groups | 190 µs  | 1103 µs (5.8×✗) | 953 µs (5.0×✗)  | 586 µs (3.1×✓)             |
+| 1000 keyed-tabular rows  | 105 µs  | 679 µs (6.5×✗)  | 562 µs (5.4×✗)  | 603 µs (5.8×✓)             |
+
+**2.5–8.3× faster** than every other encoder, with **byte-identical,
+spec-canonical** output: only the official TS SDK matches etoon byte-for-byte on
+all six payloads.
 
 The CLI (`… | etoon`) adds process-spawn + pipe I/O on top — fine for shell
 pipelines / LLM logs, but for in-process use prefer the PyO3 `dumps` (no spawn,
@@ -257,14 +262,45 @@ Key optimizations:
 - **sonic-rs SIMD JSON parser** (~7× faster than serde_json)
 - **orjson bridge** — single boundary crossing (vs PyO3-based alternatives)
 - **uniform-order table fast path** — skips 300 key lookups per 50-row table
+- **first-row column probe** — an array or empty-object value rules out tabular
+  form from one element alone, so mixed arrays reach list form in O(columns)
 - **itoa specialized integer formatting**
 
 ## Compatibility
 
-Output is byte-identical to the `toons` Python package (Apache 2.0) and the
-official `toon-format/toon` TypeScript SDK. Passes **111/111** TOON spec
-fixtures covering primitives, objects, arrays (primitive/tabular/nested/bulleted),
-and whitespace.
+Tracks **TOON spec v4.1**. Output is byte-identical to the official
+`toon-format/toon` TypeScript SDK 4.1, and passes **178/179** cases of the
+official [`toon-format/spec`](https://github.com/toon-format/spec) encode
+fixture suite — every case except one requiring a non-default `indentSize`
+(etoon hardcodes 2 spaces).
+
+### v4 collapsing forms
+
+Spec v4.0 added two forms that cut nesting out of common shapes, both
+implemented here:
+
+```bash
+# Nested field groups (§9.3) — uniform nested objects become header columns
+echo '[{"id":1,"customer":{"name":"Ada","country":"DK"},"total":99}]' | etoon
+# orders[1]{id,customer{name,country},total}:
+#   1,Ada,DK,99
+
+# Keyed tabular (§9.5) — an object of uniform objects becomes a keyed table
+echo '{"alpha":{"host":"a.example.com","port":8080},"beta":{"host":"b.example.com","port":9090}}' | etoon
+# [2:]{host,port}:
+#   alpha: a.example.com,8080
+#   beta: b.example.com,9090
+```
+
+On the benchmark payloads these cut encoded size by **76.6%** (nested field
+groups) and **31.9%** (keyed tabular) against the v3.x nested output.
+
+Spec v4.0 also **removed** key folding and path expansion — folded output is
+still valid TOON (dotted keys are literal keys), but no decoder re-nests it, so
+`fold_keys` is now an etoon extension rather than a spec option. The upstream
+rationale is in [`.out-of-scope/key-folding.md`](https://github.com/toon-format/spec/blob/main/.out-of-scope/key-folding.md):
+0.00% token savings on the reference benchmarks, wire ambiguity against literal
+dotted keys, and incompatibility with streaming decode.
 
 ## Sigil-prefixed keys (`@`, `$`, `#`)
 
@@ -307,13 +343,14 @@ Savings increase with volume — 50 entries reach **35%+** (tiktoken) as the tab
 
 ## Advanced options
 
-> These are [TOON spec](https://github.com/toon-format/toon) optional parameters, intended for **programmatic use in your codebase** (Python / Rust library calls). The CLI `| etoon` pipe for LLM workflows uses defaults and does not need these.
+> Intended for **programmatic use in your codebase** (Python / Rust library calls). The CLI `| etoon` pipe for LLM workflows uses defaults and does not need these.
 
 ```python
-# Custom delimiter (when values contain commas)
+# Custom delimiter (when values contain commas) — TOON spec §11
 etoon.dumps(data, delimiter="|")   # or "\t"
 
 # Key folding: collapse {a:{b:{c:1}}} → "a.b.c: 1"
+# etoon extension — removed from the spec in v4.0, so nothing re-nests it.
 etoon.dumps(data, fold_keys=True)
 etoon.dumps(data, fold_keys=True, flatten_depth=2)  # partial fold
 ```
@@ -322,9 +359,13 @@ etoon.dumps(data, fold_keys=True, flatten_depth=2)  # partial fold
 
 - Integers > 2⁶³ are lossily coerced via f64 (works for most common big integers
   that happen to be representable; arbitrary-precision is not supported).
-- Custom `indent` is hardcoded to 2 spaces (TOON spec default).
+- `indentSize` is hardcoded to 2 spaces (TOON spec default).
+- Encoder only — etoon does not decode TOON back to JSON.
 
 ## License
 
-Apache 2.0. Test fixtures in `tests/fixtures/` are sourced from the
-[toons](https://github.com/alesanfra/toons) project (Apache 2.0).
+Apache 2.0. Test fixtures in `tests/fixtures/` come from the
+[toon-format/spec](https://github.com/toon-format/spec) suite (MIT), except the
+etoon-local `key-folding.json` which derives from
+[toons](https://github.com/alesanfra/toons) (Apache 2.0). See
+[ATTRIBUTION.md](ATTRIBUTION.md).

README.zh-TW.md

@@ -7,25 +7,28 @@
 
 快速的 [TOON](https://github.com/toon-format/toon) (Token-Oriented Object Notation) 編碼器，支援 Python、Rust、CLI。
 
-**比 `toons` 快 8 倍**、**比官方 TS SDK 快 2.7 倍**，輸出 byte-identical。
+**比 `toons` 最多快 7.6 倍**、**比官方 TS SDK 快 3.0–8.3 倍**，輸出 byte-identical，對齊 **TOON spec v4.1**。
 
 [English](https://github.com/coseto6125/etoon/blob/main/README.md)
 
 ## 效能
 
-各種代表性 payload 的單次編碼時間（`etoon` = Python/PyO3，best-of-7 × 2000 次）。
-`✓` = 輸出與 etoon byte-identical；`✗` = 該編碼器偏離 TOON spec（例如 py-rtoon
-對整數值浮點輸出 `0.0`，spec 要求 `0`）。
+各種代表性 payload 的單次編碼時間（`etoon` = Python/PyO3，best-of-7 × 400 次）。
+`✓` = 輸出與 etoon byte-identical；`✗` = 該編碼器輸出不同 — 或偏離 spec（py-rtoon
+對整數值浮點輸出 `0.0`，spec 要求 `0`），或仍停在 spec v3.x、把 v4.1 的收合形式
+展開成巢狀區塊。
 
-| Payload（編碼）         | etoon   | toons          | py-rtoon       | @toon-format/toon (TS) |
-|-------------------------|---------|----------------|----------------|------------------------|
-| 1000 筆同構物件         | 169 µs  | 888 µs (5.2×✓) | 868 µs (5.1×✗) | 455 µs (2.7×✓)         |
-| 深層巢狀                | 123 µs  | 291 µs (2.4×✓) | 737 µs (6.0×✗) | 602 µs (4.9×✓)         |
-| 1000 筆字串記錄         | 93 µs   | 747 µs (8.0×✓) | 596 µs (6.4×✓) | 640 µs (6.9×✓)         |
-| 500 筆混合物件          | 136 µs  | 755 µs (5.5×✓) | 599 µs (4.4×✗) | 1165 µs (8.6×✓)        |
+| Payload（編碼）          | etoon   | toons           | py-rtoon        | @toon-format/toon 4.1 (TS) |
+|--------------------------|---------|-----------------|-----------------|----------------------------|
+| 1000 筆同構物件          | 171 µs  | 848 µs (5.0×✓)  | 772 µs (4.5×✗)  | 519 µs (3.0×✓)             |
+| 深層巢狀                 | 110 µs  | 272 µs (2.5×✓)  | 662 µs (6.0×✗)  | 611 µs (5.6×✓)             |
+| 1000 筆字串記錄          | 89 µs   | 674 µs (7.6×✓)  | 540 µs (6.1×✓)  | 640 µs (7.2×✓)             |
+| 500 筆混合物件           | 140 µs  | 680 µs (4.8×✓)  | 541 µs (3.9×✗)  | 1166 µs (8.3×✓)            |
+| 1000 筆巢狀欄位群組      | 190 µs  | 1103 µs (5.8×✗) | 953 µs (5.0×✗)  | 586 µs (3.1×✓)             |
+| 1000 筆 keyed 表格列     | 105 µs  | 679 µs (6.5×✗)  | 562 µs (5.4×✗)  | 603 µs (5.8×✓)             |
 
-**比所有其他編碼器快 2.4–8.6 倍**，且輸出 **byte-identical、符合 spec canonical**
-（toons 與 TS SDK 逐位元組一致；py-rtoon 不符）。
+**比所有其他編碼器快 2.5–8.3 倍**，且輸出 **byte-identical、符合 spec canonical**：
+六個 payload 全部逐位元組一致的只有官方 TS SDK。
 
 CLI（`… | etoon`）在此之上多了進程啟動 + pipe I/O — 適合 shell pipeline / LLM log，
 但程式內呼叫請優先用 PyO3 的 `dumps`（無進程啟動、無 pipe）。Auto-detect 模式
@@ -256,14 +259,43 @@ Python dict → orjson.dumps → JSON bytes → sonic-rs (SIMD parse) → walk 
 - **sonic-rs SIMD JSON parser**（比 serde_json 快 ~7×）
 - **orjson bridge** — 只跨一次邊界（vs PyO3-based 方案需多次）
 - **uniform-order table fast path** — 50 筆 row 省 300 次 key lookup
+- **first-row column probe** — 只看第一個元素就能排除 tabular（含 array 或空物件
+  的欄位），混合陣列以 O(欄位數) 落到 list form
 - **itoa 特化整數格式化**
 
 ## 相容性
 
-輸出與 Python 套件 `toons`（Apache 2.0）和官方 `toon-format/toon`
-TypeScript SDK **byte-identical**。通過 **111/111** TOON spec fixtures
-涵蓋 primitives、objects、arrays（primitive/tabular/nested/bulleted）、
-whitespace。
+對齊 **TOON spec v4.1**。輸出與官方 `toon-format/toon` TypeScript SDK 4.1
+**byte-identical**，並通過官方
+[`toon-format/spec`](https://github.com/toon-format/spec) encode fixture 套件
+**178/179** — 唯一未過的是需要非預設 `indentSize` 的案例（etoon 固定 2 spaces）。
+
+### v4 收合形式
+
+spec v4.0 新增兩種把巢狀壓平的形式，兩者都已實作：
+
+```bash
+# 巢狀欄位群組（§9.3）— 同構的巢狀物件變成表頭欄位
+echo '[{"id":1,"customer":{"name":"Ada","country":"DK"},"total":99}]' | etoon
+# orders[1]{id,customer{name,country},total}:
+#   1,Ada,DK,99
+
+# Keyed tabular（§9.5）— 值為同構物件的物件變成帶 key 的表格
+echo '{"alpha":{"host":"a.example.com","port":8080},"beta":{"host":"b.example.com","port":9090}}' | etoon
+# [2:]{host,port}:
+#   alpha: a.example.com,8080
+#   beta: b.example.com,9090
+```
+
+在 benchmark payload 上，相較 v3.x 的巢狀輸出分別縮小 **76.6%**（巢狀欄位群組）
+與 **31.9%**（keyed tabular）。
+
+spec v4.0 同時**移除**了 key folding 與 path expansion — 折疊後的輸出仍是合法
+TOON（點號 key 就是字面 key），但沒有 decoder 會把它還原成巢狀，所以 `fold_keys`
+現在是 etoon 的擴充而非 spec 選項。上游理由見
+[`.out-of-scope/key-folding.md`](https://github.com/toon-format/spec/blob/main/.out-of-scope/key-folding.md)：
+在參考 benchmark 上 token 節省 0.00%、與字面點號 key 產生線上歧義、且與 streaming
+decode 不相容。
 
 ## Sigil 前綴 key（`@`、`$`、`#`）
 
@@ -306,13 +338,14 @@ echo '[{"@timestamp":"2026-04-06T12:00:01Z","@message":"POST /api/v1/users 504",
 
 ## 進階選項
 
-> 這些是 [TOON spec](https://github.com/toon-format/toon) 提供的可選參數，適用於 **codebase 內的程式呼叫**（Python / Rust library）。CLI 的 `| etoon` pipe 使用預設值，不需要設定這些。
+> 適用於 **codebase 內的程式呼叫**（Python / Rust library）。CLI 的 `| etoon` pipe 使用預設值，不需要設定這些。
 
 ```python
-# 自訂分隔符（資料含逗號時使用）
+# 自訂分隔符（資料含逗號時使用）— TOON spec §11
 etoon.dumps(data, delimiter="|")   # 或 "\t"
 
 # Key folding：壓扁 {a:{b:{c:1}}} → "a.b.c: 1"
+# etoon 擴充 — spec v4.0 已移除，沒有 decoder 會還原它。
 etoon.dumps(data, fold_keys=True)
 etoon.dumps(data, fold_keys=True, flatten_depth=2)  # 部分 fold
 ```
@@ -321,9 +354,13 @@ etoon.dumps(data, fold_keys=True, flatten_depth=2)  # 部分 fold
 
 - 超過 2⁶³ 的整數會被降為 f64（多數能整數表示的 1e20 等仍可來回，
   但任意精度不支援）。
-- `indent` 固定 2 spaces（TOON spec 預設）。
+- `indentSize` 固定 2 spaces（TOON spec 預設）。
+- 只做編碼 — etoon 不提供 TOON → JSON 的解碼。
 
 ## 授權
 
 Apache 2.0。`tests/fixtures/` 測試檔案來自
-[toons](https://github.com/alesanfra/toons) 專案（Apache 2.0）。
+[toon-format/spec](https://github.com/toon-format/spec) 官方套件（MIT），
+唯獨 etoon 自有的 `key-folding.json` 衍生自
+[toons](https://github.com/alesanfra/toons)（Apache 2.0）。詳見
+[ATTRIBUTION.md](ATTRIBUTION.md)。

benches/compare.py

@@ -42,6 +42,16 @@ def make_payloads() -> dict[str, object]:
             {"id": i, "meta": {"ok": True, "score": i * 0.1}, "tags": ["a", "b"]}
             for i in range(500)
         ],
+        # v4.1 collapsing forms: nested field groups (§9.3) and keyed tabular
+        # (§9.5). Encoders still on spec v3.x expand these into nested blocks.
+        "groups_1000": [
+            {"id": i, "customer": {"name": f"n{i}", "country": "DK"}, "total": i}
+            for i in range(1000)
+        ],
+        "keyed_1000": {
+            f"e{i}": {"host": f"h{i}.example.com", "port": 8000 + i}
+            for i in range(1000)
+        },
     }
 
 

package.json

@@ -1,5 +1,5 @@
 {
   "dependencies": {
-    "@toon-format/toon": "^2.3.0"
+    "@toon-format/toon": "^4.1.0"
   }
 }

pyproject.toml

@@ -5,7 +5,7 @@ build-backend = "maturin"
 [project]
 name = "etoon"
 dynamic = ["version"]
-description = "Fast TOON (Token-Oriented Object Notation) encoder. Byte-identical to the TOON spec, 8x faster than toons and 2.7x faster than the official TS SDK."
+description = "Fast TOON (Token-Oriented Object Notation) encoder tracking spec v4.1. Byte-identical to the official TS SDK, up to 7.6x faster than toons."
 requires-python = ">=3.10"
 license = "Apache-2.0"
 readme = "README.md"

python/etoon/__init__.py

@@ -10,7 +10,7 @@
 
 from etoon._etoon import dumps_bytes as _dumps_bytes
 
-__version__ = "0.6.0"
+__version__ = "0.7.0"
 __all__ = ["dumps"]
 
 Delimiter = Literal[",", "\t", "|"]
@@ -40,14 +40,19 @@ def dumps(
         fold_keys: If True, collapse single-key object chains into dot-notation
             keys (``{"a": {"b": 1}}`` → ``"a.b: 1"``). Safe mode: skips folding
             when a segment needs quoting, and avoids collisions with sibling keys.
+            An etoon extension: the spec dropped key folding in v4.0, so folded
+            output is still valid TOON (dotted keys are literal keys) but no
+            decoder re-nests it.
         flatten_depth: Maximum chain length when ``fold_keys=True``. ``None``
             means unlimited; ``0`` disables folding.
-        empty_array_bare: If True (default, TOON spec v3.1), emit empty arrays as
-            canonical ``[]`` / ``key: []`` instead of the legacy ``[0]:`` form.
-            A bare array *element* that is itself empty (e.g. ``[[], []]``) always
+        empty_array_bare: If True (default), emit empty arrays as canonical
+            ``[]`` / ``key: []`` instead of the legacy ``[0]:`` form. A bare
+            array *element* that is itself empty (e.g. ``[[], []]``) always
             keeps ``- [0]:`` per spec §9.2; object fields use ``key: []``.
-        escape_controls: If True (default, TOON spec v3.1), escape control chars
-            U+0000–U+001F (except ``\n`` ``\r`` ``\t``) as ``\uXXXX`` with lowercase hex.
+            Setting it False produces output the spec forbids since v3.1.
+        escape_controls: If True (default), escape control chars U+0000–U+001F
+            (except ``\n`` ``\r`` ``\t``) as ``\uXXXX`` with lowercase hex.
+            Setting it False produces output the spec forbids since v3.1.
         max_depth: Maximum JSON nesting depth for **raw bytes/bytearray input**
             (default ``1000``). Input nested deeper is rejected with
             ``ValueError`` before parsing, guarding against a stack overflow