-
Notifications
You must be signed in to change notification settings - Fork 0
ARX4 ideation: Discord packing bets (explore #2 implemented) #99
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
Changes from all commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
9f9278a
Explore ARX4 ideas for packing more data into Discord links.
cursoragent 158f5dc
Revise ARX4 bets after Discord length research.
cursoragent 621bd00
Implement and bench ARX4 bet #2 (content-first + real Brotli dict).
cursoragent 4911c29
Probe ARX4 silly cuts: priors, IR, mosaic, and fence math.
cursoragent b9f6c10
Deprioritize mosaic; probe kind dicts; teach skill semantic splits.
cursoragent File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,108 @@ | ||
| # ARX4 bet #2 — content-first / CBOR + real Brotli shared dictionary | ||
|
|
||
| _Experimental bench from `scripts/bench-arx4-bet2.mjs`. Not a shipped codec._ | ||
|
|
||
| ## What was implemented | ||
|
|
||
| 1. **Content-first binary envelope** — `src/lib/payload/arx4-content-first.ts` | ||
| - Wire: `A4 | version | kind | id | content | meta` | ||
| - Round-trip tested; stamps rebuilt envelopes as `codec: "plain"` (ARX4 is not shipped). | ||
| 2. **CBOR-ish tuple encoder** — same module; encodes the ARX2/3 tuple without JSON quotes. | ||
| 3. **Real Brotli shared dictionary** — measured via system `brotli -D` (LZ77 raw dictionary). | ||
| - Node `zlib.brotliCompressSync({ dictionary })` **silently ignores** the option on Node 22. | ||
| - Product `brotli-wasm@3` has **no** custom-dictionary API. | ||
| - Residual `brotli(dict‖data)−brotli(dict)` is kept only as a calibration column. | ||
|
|
||
| ## Environment | ||
|
|
||
| - Node: `v22.14.0` | ||
| - Brotli CLI available: **yes** | ||
| - Shared dictionary size: **1143** bytes (ARX slot text + scaffolding) | ||
| - Discord framing budget (current host): **1962** payload chars | ||
|
|
||
| ## Pre-compress sizes (envelope bytes before Brotli) | ||
|
|
||
| | Fixture | ARX3 substituted | CBOR | CBOR+text-sub | Content-first | CF+text-sub | | ||
| | --- | ---: | ---: | ---: | ---: | ---: | | ||
| | markdown-agents | 6871 | 8056 | 7006 | 8050 | 7000 | | ||
| | code-bench-report | 8203 | 8373 | 8360 | 8366 | 8353 | | ||
| | code-fragment | 7278 | 8065 | 7037 | 8063 | 7070 | | ||
| | json-package | 346 | 308 | 294 | 307 | 294 | | ||
| | small-markdown | 183 | 217 | 186 | 225 | 204 | | ||
|
|
||
| ## Brotli bytes (q11) vs ARX3 | ||
|
|
||
| | Fixture | raw chars | ARX3 (baseline) | ARX3 + residual dict est. | ARX3 + real Brotli −D | ARX3 deflate+dict (non-goal) | CBOR tuple + Brotli | CBOR + text-sub + Brotli | CBOR + real Brotli −D | CBOR + text-sub + real −D | Content-first + Brotli | Content-first + text-sub + Brotli | Content-first + real −D | Content-first + text-sub + real −D | Best real bet #2 candidate | | ||
| | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | | ||
| | markdown-agents | 8,000 | 401 (0.0%) | 351 (−12.5%) | 380 (−5.2%) | 471 (+17.5%) | 392 (−2.2%) | 397 (−1.0%) | 385 (−4.0%) | 379 (−5.5%) | 399 (−0.5%) | 403 (+0.5%) | 390 (−2.7%) | 387 (−3.5%) | 379 (−5.5%) | | ||
| | code-bench-report | 8,314 | 2219 (0.0%) | 2188 (−1.4%) | 2219 (0.0%) | 2624 (+18.3%) | 2208 (−0.5%) | 2209 (−0.5%) | 2202 (−0.8%) | 2206 (−0.6%) | 2217 (−0.1%) | 2222 (+0.1%) | 2217 (−0.1%) | 2219 (0.0%) | 2202 (−0.8%) | | ||
| | code-fragment | 8,000 | 308 (0.0%) | 272 (−11.7%) | 307 (−0.3%) | 360 (+16.9%) | 300 (−2.6%) | 296 (−3.9%) | 314 (+1.9%) | 290 (−5.8%) | 311 (+1.0%) | 310 (+0.6%) | 317 (+2.9%) | 299 (−2.9%) | 290 (−5.8%) | | ||
| | json-package | 259 | 182 (0.0%) | 141 (−22.5%) | 188 (+3.3%) | 184 (+1.1%) | 173 (−4.9%) | 173 (−4.9%) | 177 (−2.7%) | 177 (−2.7%) | 190 (+4.4%) | 182 (0.0%) | 181 (−0.5%) | 183 (+0.5%) | 177 (−2.7%) | | ||
| | small-markdown | 189 | 141 (0.0%) | 110 (−22.0%) | 143 (+1.4%) | 160 (+13.5%) | 160 (+13.5%) | 140 (−0.7%) | 144 (+2.1%) | 142 (+0.7%) | 184 (+30.5%) | 155 (+9.9%) | 177 (+25.5%) | 154 (+9.2%) | 142 (+0.7%) | | ||
|
|
||
| ## Visible chars @ baseBMP vs ARX3 | ||
|
|
||
| | Fixture | ARX3 (baseline) | ARX3 + residual dict est. | ARX3 + real Brotli −D | ARX3 deflate+dict (non-goal) | CBOR tuple + Brotli | CBOR + text-sub + Brotli | CBOR + real Brotli −D | CBOR + text-sub + real −D | Content-first + Brotli | Content-first + text-sub + Brotli | Content-first + real −D | Content-first + text-sub + real −D | Best real bet #2 candidate | fits Discord? | | ||
| | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | :---: | | ||
| | markdown-agents | 205 (0.0%) | 180 (−12.2%) | 194 (−5.4%) | 240 (+17.1%) | 200 (−2.4%) | 203 (−1.0%) | 197 (−3.9%) | 194 (−5.4%) | 204 (−0.5%) | 206 (+0.5%) | 199 (−2.9%) | 198 (−3.4%) | 194 (−5.4%) | yes | | ||
| | code-bench-report | 1119 (0.0%) | 1103 (−1.4%) | 1119 (0.0%) | 1322 (+18.1%) | 1113 (−0.5%) | 1114 (−0.4%) | 1110 (−0.8%) | 1112 (−0.6%) | 1118 (−0.1%) | 1120 (+0.1%) | 1118 (−0.1%) | 1119 (0.0%) | 1110 (−0.8%) | yes | | ||
| | code-fragment | 158 (0.0%) | 140 (−11.4%) | 158 (0.0%) | 184 (+16.5%) | 154 (−2.5%) | 152 (−3.8%) | 161 (+1.9%) | 149 (−5.7%) | 160 (+1.3%) | 159 (+0.6%) | 163 (+3.2%) | 154 (−2.5%) | 149 (−5.7%) | yes | | ||
| | json-package | 95 (0.0%) | 74 (−22.1%) | 98 (+3.2%) | 96 (+1.1%) | 90 (−5.3%) | 90 (−5.3%) | 92 (−3.2%) | 92 (−3.2%) | 99 (+4.2%) | 95 (0.0%) | 94 (−1.1%) | 95 (0.0%) | 92 (−3.2%) | yes | | ||
| | small-markdown | 74 (0.0%) | 59 (−20.3%) | 75 (+1.4%) | 84 (+13.5%) | 84 (+13.5%) | 74 (0.0%) | 76 (+2.7%) | 75 (+1.4%) | 96 (+29.7%) | 81 (+9.5%) | 92 (+24.3%) | 81 (+9.5%) | 75 (+1.4%) | yes | | ||
|
|
||
| ## Totals (all fixtures) | ||
|
|
||
| | Variant | Σ brotli | vs ARX3 | Σ BMP chars | vs ARX3 BMP | | ||
| | --- | ---: | ---: | ---: | ---: | | ||
| | ARX3 (baseline) | 3251 | 0.0% | 1651 | 0.0% | | ||
| | ARX3 + residual dict est. | 3062 | −5.8% | 1556 | −5.8% | | ||
| | ARX3 + real Brotli −D | 3237 | −0.4% | 1644 | −0.4% | | ||
| | ARX3 deflate+dict (non-goal) | 3799 | +16.9% | 1926 | +16.7% | | ||
| | CBOR tuple + Brotli | 3233 | −0.6% | 1641 | −0.6% | | ||
| | CBOR + text-sub + Brotli | 3215 | −1.1% | 1633 | −1.1% | | ||
| | CBOR + real Brotli −D | 3222 | −0.9% | 1636 | −0.9% | | ||
| | CBOR + text-sub + real −D | 3194 | −1.8% | 1622 | −1.8% | | ||
| | Content-first + Brotli | 3301 | +1.5% | 1677 | +1.6% | | ||
| | Content-first + text-sub + Brotli | 3272 | +0.6% | 1661 | +0.6% | | ||
| | Content-first + real −D | 3282 | +1.0% | 1666 | +0.9% | | ||
| | Content-first + text-sub + real −D | 3242 | −0.3% | 1647 | −0.2% | | ||
| | Best real bet #2 candidate | 3190 | −1.9% | 1620 | −1.9% | | ||
|
|
||
| ## Findings | ||
|
|
||
| ### Content-first / CBOR alone | ||
|
|
||
| - Dropping JSON (CBOR) or skipping the tuple (content-first) changes pre-Brotli size, but | ||
| **after Brotli q11 the win vs ARX3 is small** — often within ~1%, and content-first alone | ||
| can *lose* on small fixtures because it skips ARX text substitution. | ||
| - Re-applying v1 text substitution *inside* content-first / CBOR fields recovers most of | ||
| that gap; see the `+ text-sub` columns. | ||
|
|
||
| ### Real Brotli shared dictionary (`brotli -D`) | ||
|
|
||
| - Unlike the residual estimate (often −10% to −30%), **real LZ77 shared dictionaries are | ||
| modest** on this corpus (typically under ~1% total, fixture-dependent). | ||
| - Residual estimates systematically **overstate** the win; do not use them as a ship gate. | ||
| - Deflate+dictionary remains a **non-goal**: larger than plain Brotli on this corpus. | ||
|
|
||
| ### Product implications | ||
|
|
||
| 1. **Browser path blocked for real shared dicts today** — `brotli-wasm` has no dictionary | ||
| API; Node zlib ignores `dictionary`. Shipping ARX4 shared-dict needs a wasm fork or | ||
| alternate compressor with custom-dict support. | ||
| 2. **Binary envelopes are still useful plumbing** (no JSON escaping, cleaner wire) but are | ||
| not a Discord capacity unlock by themselves on this corpus. | ||
| 3. Prefer measuring with **real `brotli -D`** (or a dict-capable wasm) over residual proxies | ||
| before committing to a shared-dictionary protocol. | ||
| 4. Next exploration should focus on **dict contents matched to the post-substitution byte | ||
| stream** (or a wasm dict path), not more residual optimism. | ||
|
|
||
| ## How to re-run | ||
|
|
||
| ```bash | ||
| # requires system `brotli` CLI for real −D columns (apt install brotli) | ||
| npm run bench:arx4-bet2 | ||
| # or: node scripts/bench-arx4-bet2.mjs | ||
| ``` | ||
|
|
||
| _Generated in 292.4ms._ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,222 @@ | ||
| # ARX4 ideation — squeezing more into a Discord message | ||
|
|
||
| _Experimental notes from `scripts/arx4-ideation-probe.mjs`. Not a shipped codec._ | ||
|
|
||
| ## Goal | ||
|
|
||
| ARX3 already optimizes **visible fragment characters** via baseBMP (~15.92 bits/char). | ||
| Discord's hard limit is the full markdown link: | ||
|
|
||
| ```text | ||
| [label](https://host/path#<tag><payload>) ≤ 2000 characters | ||
| ``` | ||
|
|
||
| So ARX4 should optimize **Discord markdown-link length**, not just fragment length. | ||
|
|
||
| ## Discord budget math | ||
|
|
||
| | Framing | Framing overhead | Payload budget | Max brotli bytes @ baseBMP | Max @ baseAstral (code-point count) | | ||
| | --- | ---: | ---: | ---: | ---: | | ||
| | currentHost (`[Artifact](https://agent-render.com#c…)`) | 38 | 1962 | 3898 | 4897 | | ||
| | shortHost (`[a](https://arx.page#d…)`) | 23 | 1977 | 3928 | 4934 | | ||
| | bareHost (`[x](https://r.page#d…)`) | 21 | 1979 | 3932 | 4939 | | ||
|
|
||
| Takeaway: host + label overhead is only ~20–50 chars (and short labels are already the | ||
| agent skill default). The real ceiling is ~3.8–3.9 KB of compressed bytes under baseBMP. | ||
| Astral density (~4.9 KB) is not a Discord win under UTF-16 counting — see research note. | ||
|
|
||
| ## Wire density ceiling | ||
|
|
||
| | Encoding | Bits / JS `length` unit | Notes | | ||
| | --- | ---: | --- | | ||
| | base64url | 6.00 | ASCII-safe; ARX2 default on hostile surfaces | | ||
| | base76 | 6.27 | ASCII fragment-safe | | ||
| | base1k | 10.79 | BMP subset | | ||
| | baseBMP (ARX3) | 15.92 | Current best visible density | | ||
| | baseAstral (code points) | 20.00 | ~26% denser **only if** Discord counts code points | | ||
| | baseAstral (UTF-16 units) | 10.00 | **Worse** than baseBMP (~10 vs ~15.92 bits/unit) | | ||
|
|
||
| ### Research note — Discord length counting (2026-07 web pass) | ||
|
|
||
| Public sources disagree; the **client-facing** signal matters most for paste-to-send: | ||
|
|
||
| | Source | Claim | Weight | | ||
| | --- | --- | --- | | ||
| | TypeCount / Discord character-counter guides | Standard emoji count as **2** toward the 2000 limit ("Unicode encoding") | High for UX — matches JS/Electron `.length` | | ||
| | Discord desktop stack | Electron → JS strings → UTF-16 code units | High — composer counter almost certainly uses this | | ||
| | Our product (`markdown-link.ts`) | Already gates on `markdownLink.length` (UTF-16 units) | Aligns with client-side folk wisdom | | ||
| | twilight-interactions #41 | Slash-command option `min/max_length` uses Unicode **code points** (Python `len()`), not UTF-8 bytes | Medium — different API surface than message `content` | | ||
| | Secondary blogs (go-tools, discord-webhook) | "Code points, emoji = 1" | Low — contradicted by emoji=2 guides; some validators still use JS `.length` | | ||
|
|
||
| **Working conclusion:** treat Discord message limits as **UTF-16-unit** (JS `.length`) until a live paste test proves otherwise. | ||
| That **kills baseAstral as a Discord win** — astral scalars cost 2 units each, so density drops below baseBMP. | ||
| Keep a one-shot paste test on the backlog (1999 BMP chars vs 1000 astral + framing) only to close the API-vs-client gap; do not prototype baseAstral for Discord first. | ||
|
|
||
| ## Ideas probed | ||
|
|
||
| 1. **baseAstral wire** — pack into supplementary-plane scalars (~20 bits/code point). **Deprioritized for Discord** after UTF-16 research. | ||
| 2. **CBOR-ish binary tuple** — drop JSON quotes/escapes around the ARX2/3 tuple. **Worth exploring.** | ||
| 3. **Brotli shared static dictionary** — seed Brotli with domain patterns already in `/arx-dictionary.json`. | ||
| Node zlib cannot set a Brotli custom dictionary today, so the probe reports (a) a residual | ||
| `brotli(dict‖data)−brotli(dict)` estimate and (b) a real `deflateRaw+dictionary` proxy. **Worth exploring** via wasm. | ||
| 4. **Content-first binary envelope** — compress raw artifact bytes + tiny binary header, skip JSON entirely. **Worth exploring.** | ||
| 5. **Mined overlay growth** — corpus-mined n-grams as an extra substitution layer. | ||
| 6. **Combined ARX4 stack** — content-first + mined + shared-dict estimate. | ||
| 7. **Discord framing** — short host + 1-char label + compact tag. **Already practiced** (skill uses short labels; host shortening is DNS, not codec). | ||
|
|
||
| ## Corpus results | ||
|
|
||
| Visible char counts assume the ARX3-style compact tag + dense Unicode wire (marker + 2-char length + digits). | ||
| Percentages are vs ARX3 baseBMP visible chars (negative = larger / worse). | ||
|
|
||
| ### Brotli bytes | ||
|
|
||
| | Fixture | raw chars | ARX3 (baseline) | ARX3 + Brotli dict est. | ARX3 deflate+dict proxy | CBOR tuple + Brotli | CBOR + Brotli dict est. | Content-first binary | Content-first + dict est. | ARX3 + mined overlay | Content-first + mined | ARX4 stack (cf+mined+dict) | | ||
| | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | | ||
| | markdown-agents | 8,000 | 401 (0.0%) | 351 (−12.5%) | 471 (+17.5%) | 392 (−2.2%) | 337 (−16.0%) | 395 (−1.5%) | 342 (−14.7%) | 405 (+1.0%) | 398 (−0.7%) | 363 (−9.5%) | | ||
| | code-bench-report | 8,314 | 2219 (0.0%) | 2188 (−1.4%) | 2624 (+18.3%) | 2208 (−0.5%) | 2169 (−2.3%) | 2219 (0.0%) | 2171 (−2.2%) | 2296 (+3.5%) | 2285 (+3.0%) | 2241 (+1.0%) | | ||
| | code-fragment | 8,000 | 308 (0.0%) | 272 (−11.7%) | 360 (+16.9%) | 300 (−2.6%) | 255 (−17.2%) | 308 (0.0%) | 261 (−15.3%) | 322 (+4.5%) | 320 (+3.9%) | 286 (−7.1%) | | ||
| | json-package | 259 | 182 (0.0%) | 141 (−22.5%) | 184 (+1.1%) | 173 (−4.9%) | 129 (−29.1%) | 184 (+1.1%) | 131 (−28.0%) | 190 (+4.4%) | 192 (+5.5%) | 150 (−17.6%) | | ||
| | small-markdown | 189 | 141 (0.0%) | 110 (−22.0%) | 160 (+13.5%) | 160 (+13.5%) | 103 (−27.0%) | 163 (+15.6%) | 108 (−23.4%) | 154 (+9.2%) | 178 (+26.2%) | 118 (−16.3%) | | ||
|
|
||
| ### Visible chars @ baseBMP | ||
|
|
||
| | Fixture | ARX3 BMP | best idea BMP | win | fits Discord (current host) | fits (short host) | | ||
| | --- | ---: | ---: | ---: | :---: | :---: | | ||
| | markdown-agents | 205 | 173 (CBOR + Brotli dict est.) | −15.6% | yes → yes | yes | | ||
| | code-bench-report | 1119 | 1093 (CBOR + Brotli dict est.) | −2.3% | yes → yes | yes | | ||
| | code-fragment | 158 | 132 (CBOR + Brotli dict est.) | −16.5% | yes → yes | yes | | ||
| | json-package | 95 | 68 (CBOR + Brotli dict est.) | −28.4% | yes → yes | yes | | ||
| | small-markdown | 74 | 55 (CBOR + Brotli dict est.) | −25.7% | yes → yes | yes | | ||
|
|
||
| ### Visible chars @ baseAstral (optimistic code-point counting) | ||
|
|
||
| | Fixture | ARX3 BMP | ARX3 Astral | best idea Astral | astral win vs ARX3 BMP | | ||
| | --- | ---: | ---: | ---: | ---: | | ||
| | markdown-agents | 205 | 164 | 138 (CBOR + Brotli dict est.) | −32.7% | | ||
| | code-bench-report | 1119 | 891 | 871 (CBOR + Brotli dict est.) | −22.2% | | ||
| | code-fragment | 158 | 127 | 106 (CBOR + Brotli dict est.) | −32.9% | | ||
| | json-package | 95 | 76 | 55 (CBOR + Brotli dict est.) | −42.1% | | ||
| | small-markdown | 74 | 60 | 45 (CBOR + Brotli dict est.) | −39.2% | | ||
|
|
||
| ### Totals (fixtures with a value for that variant) | ||
|
|
||
| | Variant | Σ brotli | Σ BMP chars | vs ARX3 BMP | Σ Astral chars | vs ARX3 BMP | | ||
| | --- | ---: | ---: | ---: | ---: | ---: | | ||
| | ARX3 (baseline) | 3251 | 1651 | 0.0% | 1318 | −20.2% | | ||
| | ARX3 + Brotli dict est. | 3062 | 1556 | −5.8% | 1243 | −24.7% | | ||
| | ARX3 deflate+dict proxy | 3799 | 1926 | +16.7% | 1538 | −6.8% | | ||
| | CBOR tuple + Brotli | 3233 | 1641 | −0.6% | 1312 | −20.5% | | ||
| | CBOR + Brotli dict est. | 2993 | 1521 | −7.9% | 1215 | −26.4% | | ||
| | Content-first binary | 3269 | 1660 | +0.5% | 1326 | −19.7% | | ||
| | Content-first + dict est. | 3013 | 1531 | −7.3% | 1223 | −25.9% | | ||
| | ARX3 + mined overlay | 3367 | 1709 | +3.5% | 1365 | −17.3% | | ||
| | Content-first + mined | 3373 | 1713 | +3.8% | 1368 | −17.1% | | ||
| | ARX4 stack (cf+mined+dict) | 3158 | 1605 | −2.8% | 1282 | −22.4% | | ||
|
|
||
| ## Discord capacity (how much raw text fits) | ||
|
|
||
| Binary-searching the largest source string whose encoded markdown link stays ≤ 2000 chars | ||
| (current-host framing, payload budget 1962) shows the practical ceiling: | ||
|
|
||
| | Content shape | ARX3 @ baseBMP | ARX3 @ baseAstral | Content-first @ BMP | Notes | | ||
| | --- | ---: | ---: | ---: | --- | | ||
| | Tiled real report + unique headers | ≥120k (search cap) | ≥150k (~+25%) | ≥120k | Highly compressible; Discord is not the bottleneck | | ||
| | Generated TS helpers | ≥120k (search cap) | ≥150k (~+25%) | ≥120k | Same | | ||
| | Quote/newline-heavy prose | ~88k | ~112k (~+27%) | ~87k | JSON escaping barely matters once Brotli runs | | ||
| | Current `code-bench-report.md` (8.3k) | ~1117 BMP chars | ~891 astral | ~1117 | Uses ~57% of Discord budget today | | ||
|
|
||
| **Reading:** for Discord, ARX3 already leaves a lot of headroom on typical artifacts. With | ||
| baseAstral deprioritized (UTF-16), the ways to *raise the ceiling* are fewer compressed | ||
| bytes via shared dictionaries / better envelopes — not denser Unicode wire or more text substitutions. | ||
|
|
||
| ## Interpretation | ||
|
|
||
| ### What already works in ARX3 | ||
|
|
||
| - For typical single artifacts under ~8–12 KB of source, ARX3 baseBMP already fits Discord | ||
| with room to spare (see `small-markdown`, `json-package`, `code-fragment`, and the | ||
| 8.3k code-bench report at ~57% of budget). | ||
| - The painful case is large unique prose/reports where dictionary substitution helps less | ||
| and Brotli carries most of the work — chase bytes there, not astral wire. | ||
|
|
||
| ### Ranked bets for ARX4 | ||
|
|
||
| 1. **Agent-side semantic splits (skill) — preferred over mosaic** | ||
| - When Discord budget blows, agents should cut by meaning (sections/files) into | ||
| independent links — see `skills/agent-render-linking/SKILL.md`. | ||
| - **Mosaic/`1/N` reassembly is out** — not worth the protocol/UX cost. | ||
|
|
||
| 2. **Kind-specific dictionaries — measured, not yet (see `docs/arx4-kind-dicts.md`)** | ||
| - Free kind tags (`m`/`k`/`j`/…) cost **0** extra chars vs today’s `c`; a +1 selector is optional. | ||
| - On this LOO corpus, kind overlays **do not beat** shared ARX3 (Σ BMP **+0.4%** free-tag, | ||
| **+0.8%** with +1 selector). Wins were tiny/rare (2/7 fixtures free-tag). | ||
| - **Do not spend a char hoping to “save thousands”** — Discord wins look like tens of BMP | ||
| chars when they exist; agents should split oversized source instead. | ||
| - Revisit only with a larger per-kind held-out corpus. | ||
|
|
||
| 3. **Content-first / CBOR + real Brotli shared dict — explored (see `docs/arx4-bet2-bench.md`)** | ||
| - Residual dict estimates overstated wins (~6–8% corpus); **real `brotli -D` is ~0–1%**. | ||
| - Binary envelopes alone are not a Discord unlock; keep as plumbing. | ||
|
|
||
| 4. **Versioned shared prior (chunk / word) — only with LOO gate** | ||
| - Warm priors are fake (−70%+); LOO chunk priors **regressed** (~+20% BMP). See silly-cuts. | ||
|
|
||
| 5. **Implied envelope + kind IR + lossy — small honest cuts** | ||
| - Implied / kind IR ~3–4% corpus; lossy ~8% when stripping emphasis/comments. | ||
|
|
||
| 6. **Hybrid fence / label bitstream — wash** | ||
|
|
||
| 7. **Curated overlay growth (cautious)** — mined n-grams alone regressed (+3–4%). | ||
|
|
||
| 8. **baseAstral — deprioritized for Discord** (UTF-16 counting). | ||
|
|
||
| 9. **Discord framing — already practiced, not an ARX4 lever** | ||
|
|
||
| ### Suggested ARX4 shape (if pursued) | ||
|
|
||
| ```text | ||
| # Single-link path (marginal): | ||
| artifact bytes | ||
| → implied / content-first envelope + optional kind IR | ||
| → optional kind-tagged dict ONLY if held-out gate clears (prefer free tags m/k/j/…) | ||
| → Brotli q11 (+ real shared dict only if browser wasm gains dict support) | ||
| → baseBMP → compact tag | ||
|
|
||
| # Over-budget path (product, not codec): | ||
| agent semantically splits → N independent ARX3 links (skill guidance) | ||
| ``` | ||
|
|
||
| Selection policy: optimize `markdownLink.length` (JS/UTF-16 units) for a declared surface | ||
| (`discord` | `visible` | `transport`). | ||
|
|
||
| Takeaway: **stop mosaic; stop paying selector chars without a LOO win; prefer skill splits | ||
| and only revisit kind dicts with a bigger corpus.** | ||
|
|
||
| ## Non-goals / traps | ||
|
|
||
| - Do not weaken the 8192 fragment budget or 200k decoded budget for Discord wins. | ||
| - Do not put artifact bodies in query params. | ||
| - Do not chase baseAstral for Discord until a live paste test overturns UTF-16 counting. | ||
| - Do not treat short-host framing as an ARX4 deliverable. | ||
| - Do not replace UUID mode: hostile link scanners still want short opaque URLs. | ||
| - Do not grow substitution dictionaries without a corpus gate — mining can regress. | ||
|
|
||
| ## How to re-run | ||
|
|
||
| ```bash | ||
| npm run bench:arx4-ideation | ||
| # or: node scripts/arx4-ideation-probe.mjs | ||
|
|
||
| # Bet #2 follow-up (content-first/CBOR + real brotli -D): | ||
| npm run bench:arx4-bet2 | ||
|
|
||
| # Silly / deep cuts (priors, IR, mosaic math — mosaic deprioritized): | ||
| npm run bench:arx4-silly | ||
|
|
||
| # Kind-specific dictionaries (+ free tag vs +1 selector): | ||
| npm run bench:arx4-kind-dicts | ||
| ``` | ||
|
|
||
| _Generated in 9.6ms._ | ||
| See also `docs/arx4-bet2-bench.md`, `docs/arx4-silly-cuts.md`, and `docs/arx4-kind-dicts.md`. | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The tables immediately below use negative deltas for smaller/winning outputs (for example
351vs the401ARX3 baseline is shown as−12.5%, and thewincolumn also reports wins as negative), so this sentence tells readers to interpret the signs backwards and can invert the benchmark conclusions.Useful? React with 👍 / 👎.