From 9f9278a345d9078e41481cc50f7992838a57e2af Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Wed, 8 Jul 2026 18:33:32 +0000 Subject: [PATCH 1/5] Explore ARX4 ideas for packing more data into Discord links. MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add an ideation probe and findings doc comparing ARX3 against denser alphabets, binary envelopes, shared-dictionary estimates, and framing tricks — without changing the shipped codec surface. Co-authored-by: Aanish Bhirud --- docs/arx4-ideation.md | 190 +++++++ package.json | 1 + scripts/arx4-ideation-probe.mjs | 883 ++++++++++++++++++++++++++++++++ 3 files changed, 1074 insertions(+) create mode 100644 docs/arx4-ideation.md create mode 100644 scripts/arx4-ideation-probe.mjs diff --git a/docs/arx4-ideation.md b/docs/arx4-ideation.md new file mode 100644 index 0000000..157153d --- /dev/null +++ b/docs/arx4-ideation.md @@ -0,0 +1,190 @@ +# ARX4 ideation — squeezing more into a Discord message + +_Experimental notes from `scripts/arx4-ideation-probe.mjs`. Not a shipped codec._ + +## Goal + +ARX3 already optimizes **visible fragment characters** via baseBMP (~15.92 bits/char). +Discord's hard limit is the full markdown link: + +```text +[label](https://host/path#) ≤ 2000 characters +``` + +So ARX4 should optimize **Discord markdown-link length**, not just fragment length. + +## Discord budget math + +| Framing | Framing overhead | Payload budget | Max brotli bytes @ baseBMP | Max @ baseAstral (code-point count) | +| --- | ---: | ---: | ---: | ---: | +| currentHost (`[Artifact](https://agent-render.com#c…)`) | 38 | 1962 | 3898 | 4897 | +| shortHost (`[a](https://arx.page#d…)`) | 23 | 1977 | 3928 | 4934 | +| bareHost (`[x](https://r.page#d…)`) | 21 | 1979 | 3932 | 4939 | + +Takeaway: host + label overhead is only ~20–50 chars. The real ceiling is ~3.8–3.9 KB of +compressed bytes under baseBMP, or ~4.9 KB if a denser Unicode wire survives Discord's +character counting as one unit per code point. + +## Wire density ceiling + +| Encoding | Bits / JS `length` unit | Notes | +| --- | ---: | --- | +| base64url | 6.00 | ASCII-safe; ARX2 default on hostile surfaces | +| base76 | 6.27 | ASCII fragment-safe | +| base1k | 10.79 | BMP subset | +| baseBMP (ARX3) | 15.92 | Current best visible density | +| baseAstral (code points) | 20.00 | ~26% denser **if** Discord counts code points | +| baseAstral (UTF-16 units) | 10.00 | **Worse** than baseBMP if length is UTF-16 | + +**Critical unknown:** Discord's message length appears to count Unicode code points for +most text, but markdown link destinations and client paste paths need an empirical check +before shipping an astral wire. If any hop counts UTF-16, astral loses to baseBMP. + +## Ideas probed + +1. **baseAstral wire** — pack into supplementary-plane scalars (~20 bits/code point). +2. **CBOR-ish binary tuple** — drop JSON quotes/escapes around the ARX2/3 tuple. +3. **Brotli shared static dictionary** — seed Brotli with domain patterns already in `/arx-dictionary.json`. + Node zlib cannot set a Brotli custom dictionary today, so the probe reports (a) a residual + `brotli(dict‖data)−brotli(dict)` estimate and (b) a real `deflateRaw+dictionary` proxy. +4. **Content-first binary envelope** — compress raw artifact bytes + tiny binary header, skip JSON entirely. +5. **Mined overlay growth** — corpus-mined n-grams as an extra substitution layer. +6. **Combined ARX4 stack** — content-first + mined + shared-dict estimate. +7. **Discord framing** — short host + 1-char label + compact tag (cheap, orthogonal). + +## Corpus results + +Visible char counts assume the ARX3-style compact tag + dense Unicode wire (marker + 2-char length + digits). +Percentages are vs ARX3 baseBMP visible chars (negative = larger / worse). + +### Brotli bytes + +| Fixture | raw chars | ARX3 (baseline) | ARX3 + Brotli dict est. | ARX3 deflate+dict proxy | CBOR tuple + Brotli | CBOR + Brotli dict est. | Content-first binary | Content-first + dict est. | ARX3 + mined overlay | Content-first + mined | ARX4 stack (cf+mined+dict) | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| markdown-agents | 8,000 | 401 (0.0%) | 351 (−12.5%) | 471 (+17.5%) | 392 (−2.2%) | 337 (−16.0%) | 395 (−1.5%) | 342 (−14.7%) | 405 (+1.0%) | 398 (−0.7%) | 363 (−9.5%) | +| code-bench-report | 8,314 | 2219 (0.0%) | 2188 (−1.4%) | 2624 (+18.3%) | 2208 (−0.5%) | 2169 (−2.3%) | 2219 (0.0%) | 2171 (−2.2%) | 2296 (+3.5%) | 2285 (+3.0%) | 2241 (+1.0%) | +| code-fragment | 8,000 | 308 (0.0%) | 272 (−11.7%) | 360 (+16.9%) | 300 (−2.6%) | 255 (−17.2%) | 308 (0.0%) | 261 (−15.3%) | 322 (+4.5%) | 320 (+3.9%) | 286 (−7.1%) | +| json-package | 259 | 182 (0.0%) | 141 (−22.5%) | 184 (+1.1%) | 173 (−4.9%) | 129 (−29.1%) | 184 (+1.1%) | 131 (−28.0%) | 190 (+4.4%) | 192 (+5.5%) | 150 (−17.6%) | +| small-markdown | 189 | 141 (0.0%) | 110 (−22.0%) | 160 (+13.5%) | 160 (+13.5%) | 103 (−27.0%) | 163 (+15.6%) | 108 (−23.4%) | 154 (+9.2%) | 178 (+26.2%) | 118 (−16.3%) | + +### Visible chars @ baseBMP + +| Fixture | ARX3 BMP | best idea BMP | win | fits Discord (current host) | fits (short host) | +| --- | ---: | ---: | ---: | :---: | :---: | +| markdown-agents | 205 | 173 (CBOR + Brotli dict est.) | −15.6% | yes → yes | yes | +| code-bench-report | 1119 | 1093 (CBOR + Brotli dict est.) | −2.3% | yes → yes | yes | +| code-fragment | 158 | 132 (CBOR + Brotli dict est.) | −16.5% | yes → yes | yes | +| json-package | 95 | 68 (CBOR + Brotli dict est.) | −28.4% | yes → yes | yes | +| small-markdown | 74 | 55 (CBOR + Brotli dict est.) | −25.7% | yes → yes | yes | + +### Visible chars @ baseAstral (optimistic code-point counting) + +| Fixture | ARX3 BMP | ARX3 Astral | best idea Astral | astral win vs ARX3 BMP | +| --- | ---: | ---: | ---: | ---: | +| markdown-agents | 205 | 164 | 138 (CBOR + Brotli dict est.) | −32.7% | +| code-bench-report | 1119 | 891 | 871 (CBOR + Brotli dict est.) | −22.2% | +| code-fragment | 158 | 127 | 106 (CBOR + Brotli dict est.) | −32.9% | +| json-package | 95 | 76 | 55 (CBOR + Brotli dict est.) | −42.1% | +| small-markdown | 74 | 60 | 45 (CBOR + Brotli dict est.) | −39.2% | + +### Totals (fixtures with a value for that variant) + +| Variant | Σ brotli | Σ BMP chars | vs ARX3 BMP | Σ Astral chars | vs ARX3 BMP | +| --- | ---: | ---: | ---: | ---: | ---: | +| ARX3 (baseline) | 3251 | 1651 | 0.0% | 1318 | −20.2% | +| ARX3 + Brotli dict est. | 3062 | 1556 | −5.8% | 1243 | −24.7% | +| ARX3 deflate+dict proxy | 3799 | 1926 | +16.7% | 1538 | −6.8% | +| CBOR tuple + Brotli | 3233 | 1641 | −0.6% | 1312 | −20.5% | +| CBOR + Brotli dict est. | 2993 | 1521 | −7.9% | 1215 | −26.4% | +| Content-first binary | 3269 | 1660 | +0.5% | 1326 | −19.7% | +| Content-first + dict est. | 3013 | 1531 | −7.3% | 1223 | −25.9% | +| ARX3 + mined overlay | 3367 | 1709 | +3.5% | 1365 | −17.3% | +| Content-first + mined | 3373 | 1713 | +3.8% | 1368 | −17.1% | +| ARX4 stack (cf+mined+dict) | 3158 | 1605 | −2.8% | 1282 | −22.4% | + +## Discord capacity (how much raw text fits) + +Binary-searching the largest source string whose encoded markdown link stays ≤ 2000 chars +(current-host framing, payload budget 1962): + +| Content shape | ARX3 @ baseBMP | ARX3 @ baseAstral | Content-first @ BMP | Notes | +| --- | ---: | ---: | ---: | --- | +| Tiled real report + unique section headers | ≥120k (search cap) | ≥150k (+25%) | ≥120k | Highly compressible; Discord is not the bottleneck | +| Generated TS helpers | ≥120k (search cap) | ≥150k (+25%) | ≥120k | Same | +| Quote/newline-heavy prose | ~88k | ~112k (+27%) | ~87k | JSON escaping barely matters once Brotli runs | +| Current `code-bench-report.md` (8.3k) | 1117 BMP chars | ~891 astral | 1117 | Uses ~57% of Discord budget today | + +**Reading:** for Discord, ARX3 already leaves a lot of headroom on typical artifacts. The +ways to *raise the ceiling* are almost entirely (1) denser Unicode wire and (2) fewer +compressed bytes via shared dictionaries / better envelopes — not more text substitutions. + +## Interpretation + +### What already works in ARX3 + +- For typical single artifacts under ~8–12 KB of source, ARX3 baseBMP already fits Discord + with room to spare (see `small-markdown`, `json-package`, `code-fragment`, and the + 8.3k code-bench report at ~57% of budget). +- The painful case is large unique prose/reports where dictionary substitution helps less + and Brotli carries most of the work — and even then, baseAstral is the cleanest capacity + unlock if Discord counting cooperates. + +### Highest-leverage ARX4 directions + +1. **Validate Discord length semantics, then consider baseAstral** + - Pure wire change on top of identical ARX3 bytes. + - ~20–26% fewer visible units *and* ~25% more raw capacity *if* Discord counts code points. + - Zero win (or a loss) if any hop counts UTF-16 code units. + - Empirically test: paste a `#d` + astral payload link into Discord desktop/mobile/web. + +2. **Content-first binary envelope (skip JSON)** + - Removes JSON escaping of newlines/quotes inside artifact bodies. + - Largest win on code and markdown with many `"` / `\n` sequences. + - Natural fit for Discord's single-artifact share path; multi-artifact can keep tuples. + +3. **Brotli shared static dictionary** + - Node cannot set a Brotli custom dictionary; numbers are estimates / deflate proxies. + - Still worth pursuing if `brotli-wasm` gains dictionary APIs — small/medium payloads benefit most. + - Deflate+dict is a real shared-dict signal and often beats plain Brotli on repetitive agent text, + but losing Brotli's stronger model is a tradeoff; prefer true Brotli dict over switching codecs. + +4. **CBOR / binary tuple** + - Modest win once Brotli has already seen the JSON structure. + - More valuable when combined with content-first (binary header + raw body). + +5. **Mined / larger overlay dictionaries** + - Helps repeated fixture-like corpora; risky for open-ended agent output. + - Prefer shipping a carefully curated ARX4 overlay over online mining. + +6. **Discord framing (orthogonal, ship anytime)** + - Short branded host + 1-char label recovers ~30 chars — small but free. + - Agents should emit `[x](https://short/#…)` when targeting Discord. + +### Suggested ARX4 shape (if pursued) + +```text +artifact bytes + → content-first binary envelope (kind|id|content|meta) + → optional curated ARX4 overlay (domain n-grams) + → Brotli q11 (+ shared static dictionary if wasm allows) + → baseBMP (safe) or baseAstral (Discord-validated) + → compact tag `d` / `e` +``` + +Selection policy: optimize `markdownLink.length` for a declared surface +(`discord` | `visible` | `transport`) instead of only fragment length. + +## Non-goals / traps + +- Do not weaken the 8192 fragment budget or 200k decoded budget for Discord wins. +- Do not put artifact bodies in query params. +- Do not assume astral density without client paste tests. +- Do not replace UUID mode: hostile link scanners still want short opaque URLs. + +## How to re-run + +```bash +node scripts/arx4-ideation-probe.mjs +``` + +_Generated in 10.5ms._ diff --git a/package.json b/package.json index 7468651..56da8d1 100644 --- a/package.json +++ b/package.json @@ -35,6 +35,7 @@ "test:browsers": "playwright install chromium", "bench:codecs": "node scripts/bench-codecs.mjs", "bench:codecs:update": "node scripts/bench-codecs.mjs --write-baseline", + "bench:arx4-ideation": "node scripts/arx4-ideation-probe.mjs", "assets:compress": "node scripts/compress-dictionary.mjs", "typecheck": "node scripts/ensure-next-types.mjs && tsc --noEmit", "check": "npm run lint && npm run test && npm run bench:codecs && npm run typecheck && npm run build && npm run check:build-budgets", diff --git a/scripts/arx4-ideation-probe.mjs b/scripts/arx4-ideation-probe.mjs new file mode 100644 index 0000000..a4c5b7e --- /dev/null +++ b/scripts/arx4-ideation-probe.mjs @@ -0,0 +1,883 @@ +#!/usr/bin/env node +/** + * ARX4 ideation probe — experimental only. + * + * Measures several ways to squeeze more artifact data into a Discord markdown + * link (`[label](url)` ≤ 2000 chars) beyond today's ARX3 pipeline: + * + * A. denser Unicode wire (supplementary / "baseAstral") + * B. binary tuple envelope (CBOR-ish) instead of JSON.stringify(tuple) + * C. Brotli shared static dictionary (pre-seeded with domain corpus) + * D. content-first packing (compress artifact body bytes, not JSON text) + * E. Discord-aware framing (short host + short label + compact tag) + * F. mined overlay dictionary growth from the bench corpus + * + * This script does NOT change the product codec surface. It prints comparative + * numbers so maintainers can decide which ideas deserve a real ARX4 design. + */ + +import { readFileSync, writeFileSync } from "node:fs"; +import { brotliCompressSync, deflateRawSync, constants } from "node:zlib"; +import { performance } from "node:perf_hooks"; + +const DISCORD_MESSAGE_MAX = 2000; +const BMP_BASE_SIZE = 62_000; +const REPORT_PATH = "docs/arx4-ideation.md"; + +const v1Dictionary = JSON.parse(readFileSync("public/arx-dictionary.json", "utf8")); +const overlayDictionary = JSON.parse(readFileSync("public/arx2-dictionary.json", "utf8")); +const codeBenchReportFixture = readFileSync("tests/fixtures/baanish-code-bench-report.md", "utf8"); + +const singleByteCodes = [ + 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x0b, 0x0e, 0x0f, 0x10, + 0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17, 0x18, 0x19, 0x1a, 0x1b, 0x1c, + 0x1d, +]; + +function buildPairs(dict, singleCodes = singleByteCodes, extendedPrefix = "\x00", extendedOffset = 1) { + const pairs = []; + for (let i = 0; i < dict.singleByteSlots.length && i < singleCodes.length; i++) { + pairs.push([dict.singleByteSlots[i], String.fromCharCode(singleCodes[i])]); + } + for (let i = 0; i < dict.extendedSlots.length; i++) { + pairs.push([dict.extendedSlots[i], extendedPrefix + String.fromCharCode(i + extendedOffset)]); + } + return pairs; +} + +const v1Pairs = buildPairs(v1Dictionary); +const overlayPairs = buildPairs(overlayDictionary, [0x1e, 0x7f], "\x1f", 0x20); + +function buildTrie(pairs, reversed = false) { + const root = { children: new Map() }; + for (const [from, to] of pairs) { + const match = reversed ? to : from; + const replacement = reversed ? from : to; + let node = root; + for (const char of match) { + let child = node.children.get(char); + if (!child) { + child = { children: new Map() }; + node.children.set(char, child); + } + node = child; + } + node.replacement ??= replacement; + } + return root; +} + +function applyTrie(text, trie) { + const out = []; + let index = 0; + while (index < text.length) { + let node = trie; + let cursor = index; + let replacement; + let replacementLength = 0; + while (cursor < text.length) { + node = node.children.get(text[cursor]); + if (!node) break; + cursor++; + if (node.replacement !== undefined) { + replacement = node.replacement; + replacementLength = cursor - index; + } + } + if (replacement !== undefined) { + out.push(replacement); + index += replacementLength; + } else { + out.push(text[index]); + index++; + } + } + return out.join(""); +} + +const v1EncodeTrie = buildTrie(v1Pairs); +const overlayEncodeTrie = buildTrie(overlayPairs); + +function brotli(input) { + return brotliCompressSync(Buffer.from(input), { + params: { [constants.BROTLI_PARAM_QUALITY]: 11 }, + }); +} + +/** + * Approximate a Brotli *shared static dictionary* win. + * + * Node's zlib brotli bindings ignore custom dictionaries, so we use two proxies: + * 1. deflateRaw + dictionary (real shared-dict API) as a lower-bound signal + * 2. residual trick: len(brotli(dict||data)) - len(brotli(dict)) as an optimistic + * estimate of what a true Brotli custom dictionary might achieve + * + * Product ARX4 would need brotli-wasm (or equivalent) with dictionary support. + */ +function sharedDictEstimates(input, dictionary) { + const data = Buffer.from(input); + const plainBrotli = brotli(data); + const plainDeflate = deflateRawSync(data, { level: 9 }); + const dictDeflate = deflateRawSync(data, { level: 9, dictionary }); + const dictOnlyBrotli = brotli(dictionary); + const dictPlusDataBrotli = brotli(Buffer.concat([dictionary, data])); + const residualBrotli = Math.max(1, dictPlusDataBrotli.length - dictOnlyBrotli.length); + + return { + brotliBytes: plainBrotli.length, + deflateBytes: plainDeflate.length, + deflateDictBytes: dictDeflate.length, + // Prefer residual when it beats plain brotli; otherwise report plain. + brotliDictEstimateBytes: Math.min(plainBrotli.length, residualBrotli), + }; +} + +function trimOptional(fields) { + let end = fields.length; + while (end > 0 && fields[end - 1] === undefined) end--; + const trimmed = new Array(end); + for (let index = 0; index < end; index++) { + trimmed[index] = fields[index] === undefined ? null : fields[index]; + } + return trimmed; +} + +function artifactTuple(artifact) { + switch (artifact.kind) { + case "markdown": + return trimOptional(["m", artifact.id, artifact.content, artifact.title, artifact.filename]); + case "code": + return trimOptional(["c", artifact.id, artifact.content, artifact.language, artifact.title, artifact.filename]); + case "diff": + return trimOptional([ + "d", + artifact.id, + artifact.patch, + artifact.oldContent, + artifact.newContent, + artifact.language, + artifact.view, + artifact.title, + artifact.filename, + ]); + case "csv": + return trimOptional(["s", artifact.id, artifact.content, artifact.title, artifact.filename]); + case "json": + return trimOptional(["j", artifact.id, artifact.content, artifact.title, artifact.filename]); + default: + throw new Error(`Unsupported kind ${artifact.kind}`); + } +} + +function tupleEnvelope(envelope) { + const artifacts = envelope.artifacts.map(artifactTuple); + const activeIndex = Math.max(0, envelope.artifacts.findIndex((a) => a.id === envelope.activeArtifactId)); + if (artifacts.length === 1) { + return trimOptional([3, artifacts[0], envelope.title]); + } + return trimOptional([2, artifacts, envelope.title, activeIndex > 0 ? activeIndex : undefined]); +} + +function encodeArx3Substituted(envelope) { + const tupleJson = JSON.stringify(tupleEnvelope({ ...envelope, codec: "arx3" })); + return applyTrie(applyTrie(tupleJson, overlayEncodeTrie), v1EncodeTrie); +} + +function baseBmpChars(byteLength) { + return 3 + Math.ceil((byteLength * 8) / Math.log2(BMP_BASE_SIZE)); +} + +/** Theoretical supplementary-plane alphabet (~1,048,000 usable scalars → ~20 bits/char). */ +function baseAstralChars(byteLength, alphabetSize = 1_048_000) { + // marker (1) + length digits (2) + payload + return 3 + Math.ceil((byteLength * 8) / Math.log2(alphabetSize)); +} + +/** + * Encode a compact binary envelope: + * kind(1) | idLen(1) | id | contentLen(varint) | content | metaLen(1) | metaJson? + * For single-artifact markdown/code/csv/json only (the Discord sweet spot). + */ +function encodeBinaryContentFirst(envelope) { + const artifact = envelope.artifacts[0]; + if (!artifact || envelope.artifacts.length !== 1) return null; + if (!("content" in artifact) || typeof artifact.content !== "string") return null; + + const kindMap = { markdown: 1, code: 2, csv: 3, json: 4 }; + const kind = kindMap[artifact.kind]; + if (!kind) return null; + + const id = Buffer.from(artifact.id ?? "a", "utf8"); + const content = Buffer.from(artifact.content, "utf8"); + const meta = {}; + if (artifact.title) meta.t = artifact.title; + if (artifact.filename) meta.f = artifact.filename; + if (artifact.language) meta.l = artifact.language; + if (envelope.title && envelope.title !== artifact.title) meta.e = envelope.title; + const metaBuf = Buffer.from(JSON.stringify(meta), "utf8"); + + function writeVarint(n) { + const bytes = []; + let v = n >>> 0; + while (v >= 0x80) { + bytes.push((v & 0x7f) | 0x80); + v >>>= 7; + } + bytes.push(v); + return Buffer.from(bytes); + } + + return Buffer.concat([ + Buffer.from([kind, id.length]), + id, + writeVarint(content.length), + content, + Buffer.from([metaBuf.length]), + metaBuf, + ]); +} + +/** + * CBOR-ish minimal array encoder for the existing tuple shape. + * Only handles numbers, strings, null, and nested arrays — enough for ARX tuples. + */ +function encodeCborish(value) { + const chunks = []; + + function pushUint(n) { + if (n < 24) chunks.push(Buffer.from([0x00 | n])); + else if (n < 256) chunks.push(Buffer.from([0x18, n])); + else if (n < 65536) chunks.push(Buffer.from([0x19, (n >> 8) & 0xff, n & 0xff])); + else chunks.push(Buffer.from([0x1a, (n >>> 24) & 0xff, (n >>> 16) & 0xff, (n >>> 8) & 0xff, n & 0xff])); + } + + function encode(v) { + if (v === null || v === undefined) { + chunks.push(Buffer.from([0xf6])); + return; + } + if (typeof v === "number" && Number.isInteger(v) && v >= 0) { + pushUint(v); + return; + } + if (typeof v === "string") { + const bytes = Buffer.from(v, "utf8"); + if (bytes.length < 24) chunks.push(Buffer.from([0x60 | bytes.length])); + else if (bytes.length < 256) chunks.push(Buffer.from([0x78, bytes.length])); + else if (bytes.length < 65536) chunks.push(Buffer.from([0x79, (bytes.length >> 8) & 0xff, bytes.length & 0xff])); + else chunks.push(Buffer.from([0x7a, (bytes.length >>> 24) & 0xff, (bytes.length >>> 16) & 0xff, (bytes.length >>> 8) & 0xff, bytes.length & 0xff])); + chunks.push(bytes); + return; + } + if (Array.isArray(v)) { + if (v.length < 24) chunks.push(Buffer.from([0x80 | v.length])); + else if (v.length < 256) chunks.push(Buffer.from([0x98, v.length])); + else chunks.push(Buffer.from([0x99, (v.length >> 8) & 0xff, v.length & 0xff])); + for (const item of v) encode(item); + return; + } + throw new Error(`Unsupported CBOR-ish value: ${typeof v}`); + } + + encode(value); + return Buffer.concat(chunks); +} + +function mineNgrams(texts, { minLen = 4, maxLen = 24, topN = 80 } = {}) { + const counts = new Map(); + for (const text of texts) { + const seenInDoc = new Set(); + for (let len = minLen; len <= maxLen; len++) { + for (let i = 0; i + len <= text.length; i++) { + const gram = text.slice(i, i + len); + if (seenInDoc.has(gram)) continue; + // Prefer printable / structured fragments; skip control-heavy noise. + if (/[\x00-\x08\x0b\x0c\x0e-\x1f]/.test(gram)) continue; + seenInDoc.add(gram); + counts.set(gram, (counts.get(gram) ?? 0) + 1); + } + } + } + + const scored = []; + for (const [gram, count] of counts) { + if (count < 2) continue; + // Score ≈ bytes saved if replaced by a 2-byte token across occurrences. + const saved = (gram.length - 2) * count; + if (saved <= 0) continue; + scored.push({ gram, count, saved }); + } + scored.sort((a, b) => b.saved - a.saved || b.gram.length - a.gram.length); + + // Greedy non-overlapping selection (prefer longer / higher-score first). + const selected = []; + for (const candidate of scored) { + if (selected.some((s) => s.includes(candidate.gram) || candidate.gram.includes(s))) continue; + selected.push(candidate.gram); + if (selected.length >= topN) break; + } + return selected; +} + +function applyLiteralSubs(text, patterns) { + // Longest-first replacement with 2-byte tokens starting at 0x80 (safe-ish for this probe). + let out = text; + for (let i = 0; i < patterns.length; i++) { + const token = String.fromCharCode(0x80 + Math.floor(i / 128), 0x80 + (i % 128)); + out = out.split(patterns[i]).join(token); + } + return out; +} + +function textEnvelope(kind, title, content, extra = {}) { + return { + v: 1, + codec: "plain", + title, + activeArtifactId: "a", + artifacts: [{ id: "a", kind, title, filename: extra.filename ?? "artifact.txt", content, ...extra }], + }; +} + +function repeatedFixture(block, targetLength, segmentSuffix = (index) => `\nfixture segment ${index}\n`) { + let fixture = ""; + let index = 0; + while (fixture.length < targetLength) { + fixture += `${block}${segmentSuffix(index)}`; + index++; + } + return Array.from(fixture).slice(0, targetLength).join(""); +} + +const markdownAgentsFixture = repeatedFixture( + [ + "# AGENTS.md excerpt", + "", + "`agent-render` is a static artifact viewer for AI-generated outputs.", + "Keep markdown, code, diffs, CSV, and JSON readable across chat surfaces.", + "", + "## Product contract", + "", + "- Fragment payloads use `#agent-render=v1..`.", + "- Artifact contents stay out of the host request path.", + "- Supported codecs are `plain`, `lz`, `deflate`, `arx`, `arx2`, and `arx3`.", + "- Supported artifact kinds are `markdown`, `code`, `diff`, `csv`, and `json`.", + "", + "Preserve the static shell, the zero-retention wording, and the renderer-first layout.", + "", + ].join("\n"), + 8000, + (index) => `\nFixture note ${index}: fragment transport, renderer readiness, and artifact metadata stay aligned.\n\n`, +); + +const codeFragmentFixture = repeatedFixture( + [ + "export async function decodeFragmentAsync(hash: string, options?: DecodeOptions) {", + " const parsed = parseFragmentPrefix(hash);", + " if (!parsed.ok) return parsed;", + " if (parsed.codec === \"arx\" || parsed.codec === \"arx2\") {", + " const { decodeArxFragmentAsync } = await import(\"./fragment-arx\");", + " return decodeArxFragmentAsync(parsed, options);", + " }", + " return decodePlainFragment(parsed.payload, options);", + "}", + "", + ].join("\n"), + 8000, + (index) => `\n// fixture segment ${index}: codec branch coverage and bundle shape stay stable.\n`, +); + +const packageManifestFixture = JSON.stringify( + { + name: "agent-render", + version: "0.1.0", + private: true, + scripts: { build: "next build", check: "npm run lint && npm run test" }, + dependencies: { next: "15.1.11", react: "19.1.0", "brotli-wasm": "^3.0.1" }, + }, + null, + 2, +); + +const corpus = [ + { + name: "markdown-agents", + envelope: textEnvelope("markdown", "AGENTS.md excerpt", markdownAgentsFixture, { filename: "AGENTS.md" }), + }, + { + name: "code-bench-report", + envelope: textEnvelope("markdown", "Baanish Code Bench", codeBenchReportFixture, { filename: "results.md" }), + }, + { + name: "code-fragment", + envelope: textEnvelope("code", "fragment.ts excerpt", codeFragmentFixture, { + filename: "fragment.ts", + language: "ts", + }), + }, + { + name: "json-package", + envelope: textEnvelope("json", "package.json", packageManifestFixture, { filename: "package.json" }), + }, + { + name: "small-markdown", + envelope: textEnvelope( + "markdown", + "Note", + [ + "# Sprint notes", + "", + "- Ship ARX3 visible URL mode", + "- Keep Discord markdown links under 2000 chars", + "- Prefer fragment transport over UUID mode for zero-retention", + "", + "```ts", + "export const value = 1;", + "```", + "", + ].join("\n"), + { filename: "notes.md" }, + ), + }, +]; + +// Build a static Brotli dictionary from domain patterns (dict slots + common scaffolding). +const brotliStaticDict = Buffer.from( + [ + ...v1Dictionary.singleByteSlots, + ...v1Dictionary.extendedSlots.slice(0, 40), + ...overlayDictionary.singleByteSlots, + ...overlayDictionary.extendedSlots, + '["m","', + '["c","', + "[3,", + "agent-render", + "export function ", + "export const ", + "import { ", + "} from \"", + "\n## ", + "\n### ", + "\n- ", + "\n\n", + "```", + ].join("\n"), + "utf8", +); + +const minedPatterns = mineNgrams( + corpus.map((row) => { + const a = row.envelope.artifacts[0]; + return typeof a.content === "string" ? a.content : ""; + }), + { topN: 64 }, +); + +function discordBudget({ host = "https://agent-render.com", label = "Artifact", tag = "c" } = {}) { + // [label](host#tagPAYLOAD) — payload budget is what remains under 2000. + const framing = `[${label}](${host}#${tag}`; + const closing = ")"; + return { + framing, + framingLength: framing.length + closing.length, + payloadBudget: DISCORD_MESSAGE_MAX - framing.length - closing.length, + }; +} + +function packVariant(brotliBytes) { + return { + brotliBytes, + bmpChars: baseBmpChars(brotliBytes), + astralChars: baseAstralChars(brotliBytes), + }; +} + +function measureRow(envelope) { + const substituted = encodeArx3Substituted(envelope); + const arx3Bytes = brotli(substituted); + const tuple = tupleEnvelope({ ...envelope, codec: "arx3" }); + const cborRaw = encodeCborish(tuple); + const cborBrotli = brotli(cborRaw); + + const contentFirst = encodeBinaryContentFirst(envelope); + const contentFirstBrotli = contentFirst ? brotli(contentFirst) : null; + + // Mined overlay on top of ARX3 substituted text before brotli. + const minedSubstituted = applyLiteralSubs(substituted, minedPatterns); + const minedBrotli = brotli(minedSubstituted); + + // Shared-dict estimates (see sharedDictEstimates). + const arx3DictEst = sharedDictEstimates(Buffer.from(substituted, "utf8"), brotliStaticDict); + const cborDictEst = sharedDictEstimates(cborRaw, brotliStaticDict); + const contentDictEst = contentFirst ? sharedDictEstimates(contentFirst, brotliStaticDict) : null; + + // Content-first + mined ngrams on raw content, then brotli. + let contentMinedBrotli = null; + if (contentFirst) { + const artifact = envelope.artifacts[0]; + const minedContent = applyLiteralSubs(artifact.content, minedPatterns); + const rebuilt = encodeBinaryContentFirst({ + ...envelope, + artifacts: [{ ...artifact, content: minedContent }], + }); + contentMinedBrotli = brotli(rebuilt); + } + + // Combined "ARX4 stack" candidate: content-first → mined → shared-dict estimate. + let arx4Stack = null; + if (contentFirst) { + const artifact = envelope.artifacts[0]; + const minedContent = applyLiteralSubs(artifact.content, minedPatterns); + const rebuilt = encodeBinaryContentFirst({ + ...envelope, + artifacts: [{ ...artifact, content: minedContent }], + }); + const est = sharedDictEstimates(rebuilt, brotliStaticDict); + arx4Stack = packVariant(est.brotliDictEstimateBytes); + } + + return { + rawContentChars: envelope.artifacts.reduce((n, a) => n + (a.content?.length ?? a.patch?.length ?? 0), 0), + arx3: packVariant(arx3Bytes.length), + arx3PlusBrotliDict: packVariant(arx3DictEst.brotliDictEstimateBytes), + arx3DeflateDictProxy: packVariant(arx3DictEst.deflateDictBytes), + cborTuple: packVariant(cborBrotli.length), + cborTuplePlusBrotliDict: packVariant(cborDictEst.brotliDictEstimateBytes), + contentFirst: contentFirstBrotli ? packVariant(contentFirstBrotli.length) : null, + contentFirstPlusBrotliDict: contentDictEst ? packVariant(contentDictEst.brotliDictEstimateBytes) : null, + minedOverlay: packVariant(minedBrotli.length), + contentFirstPlusMined: contentMinedBrotli ? packVariant(contentMinedBrotli.length) : null, + arx4Stack, + _debug: { + arx3Deflate: arx3DictEst.deflateBytes, + arx3DeflateDict: arx3DictEst.deflateDictBytes, + }, + }; +} + +function pct(from, to) { + if (!from || !to) return null; + return ((from - to) / from) * 100; +} + +function fmtPct(value) { + if (value == null || Number.isNaN(value)) return "n/a"; + const sign = value > 0 ? "−" : value < 0 ? "+" : ""; + return `${sign}${Math.abs(value).toFixed(1)}%`; +} + +function fitsDiscord(payloadChars, budget) { + return payloadChars <= budget; +} + +const budgets = { + currentHost: discordBudget({ host: "https://agent-render.com", label: "Artifact", tag: "c" }), + shortHost: discordBudget({ host: "https://arx.page", label: "a", tag: "d" }), + bareHost: discordBudget({ host: "https://r.page", label: "x", tag: "d" }), +}; + +const rows = corpus.map((item) => ({ name: item.name, ...measureRow(item.envelope) })); + +// Theoretical density table +const density = { + base64url: 6, + base76: Math.log2(77), + base1k: Math.log2(1774), + baseBMP: Math.log2(BMP_BASE_SIZE), + baseAstral1M: Math.log2(1_048_000), + // If Discord/JS counts UTF-16 code units, each astral char costs 2 → effective bits/unit: + baseAstralUtf16Effective: Math.log2(1_048_000) / 2, +}; + +function maxBytesForBudget(budgetChars, bitsPerChar, overheadChars = 3) { + return Math.floor(((budgetChars - overheadChars) * bitsPerChar) / 8); +} + +const start = performance.now(); +const lines = []; +function w(line = "") { + lines.push(line); +} + +w("# ARX4 ideation — squeezing more into a Discord message"); +w(); +w("_Experimental notes from `scripts/arx4-ideation-probe.mjs`. Not a shipped codec._"); +w(); +w("## Goal"); +w(); +w("ARX3 already optimizes **visible fragment characters** via baseBMP (~15.92 bits/char)."); +w("Discord's hard limit is the full markdown link:"); +w(); +w("```text"); +w("[label](https://host/path#) ≤ 2000 characters"); +w("```"); +w(); +w("So ARX4 should optimize **Discord markdown-link length**, not just fragment length."); +w(); +w("## Discord budget math"); +w(); +w("| Framing | Framing overhead | Payload budget | Max brotli bytes @ baseBMP | Max @ baseAstral (code-point count) |"); +w("| --- | ---: | ---: | ---: | ---: |"); +for (const [name, b] of Object.entries(budgets)) { + w( + `| ${name} (\`${b.framing}…)\`) | ${b.framingLength} | ${b.payloadBudget} | ${maxBytesForBudget(b.payloadBudget, density.baseBMP)} | ${maxBytesForBudget(b.payloadBudget, density.baseAstral1M)} |`, + ); +} +w(); +w("Takeaway: host + label overhead is only ~20–50 chars. The real ceiling is ~3.8–3.9 KB of"); +w("compressed bytes under baseBMP, or ~4.9 KB if a denser Unicode wire survives Discord's"); +w("character counting as one unit per code point."); +w(); +w("## Wire density ceiling"); +w(); +w("| Encoding | Bits / JS `length` unit | Notes |"); +w("| --- | ---: | --- |"); +w(`| base64url | ${density.base64url.toFixed(2)} | ASCII-safe; ARX2 default on hostile surfaces |`); +w(`| base76 | ${density.base76.toFixed(2)} | ASCII fragment-safe |`); +w(`| base1k | ${density.base1k.toFixed(2)} | BMP subset |`); +w(`| baseBMP (ARX3) | ${density.baseBMP.toFixed(2)} | Current best visible density |`); +w(`| baseAstral (code points) | ${density.baseAstral1M.toFixed(2)} | ~${((density.baseAstral1M / density.baseBMP - 1) * 100).toFixed(0)}% denser **if** Discord counts code points |`); +w(`| baseAstral (UTF-16 units) | ${density.baseAstralUtf16Effective.toFixed(2)} | **Worse** than baseBMP if length is UTF-16 |`); +w(); +w("**Critical unknown:** Discord's message length appears to count Unicode code points for"); +w("most text, but markdown link destinations and client paste paths need an empirical check"); +w("before shipping an astral wire. If any hop counts UTF-16, astral loses to baseBMP."); +w(); +w("## Ideas probed"); +w(); +w("1. **baseAstral wire** — pack into supplementary-plane scalars (~20 bits/code point)."); +w("2. **CBOR-ish binary tuple** — drop JSON quotes/escapes around the ARX2/3 tuple."); +w("3. **Brotli shared static dictionary** — seed Brotli with domain patterns already in `/arx-dictionary.json`."); +w(" Node zlib cannot set a Brotli custom dictionary today, so the probe reports (a) a residual"); +w(" `brotli(dict‖data)−brotli(dict)` estimate and (b) a real `deflateRaw+dictionary` proxy."); +w("4. **Content-first binary envelope** — compress raw artifact bytes + tiny binary header, skip JSON entirely."); +w("5. **Mined overlay growth** — corpus-mined n-grams as an extra substitution layer."); +w("6. **Combined ARX4 stack** — content-first + mined + shared-dict estimate."); +w("7. **Discord framing** — short host + 1-char label + compact tag (cheap, orthogonal)."); +w(); +w("## Corpus results"); +w(); +w("Visible char counts assume the ARX3-style compact tag + dense Unicode wire (marker + 2-char length + digits)."); +w("Percentages are vs ARX3 baseBMP visible chars (negative = larger / worse)."); +w(); + +const variants = [ + ["arx3", "ARX3 (baseline)"], + ["arx3PlusBrotliDict", "ARX3 + Brotli dict est."], + ["arx3DeflateDictProxy", "ARX3 deflate+dict proxy"], + ["cborTuple", "CBOR tuple + Brotli"], + ["cborTuplePlusBrotliDict", "CBOR + Brotli dict est."], + ["contentFirst", "Content-first binary"], + ["contentFirstPlusBrotliDict", "Content-first + dict est."], + ["minedOverlay", "ARX3 + mined overlay"], + ["contentFirstPlusMined", "Content-first + mined"], + ["arx4Stack", "ARX4 stack (cf+mined+dict)"], +]; + +w("### Brotli bytes"); +w(); +w(`| Fixture | raw chars | ${variants.map(([, label]) => label).join(" | ")} |`); +w(`| --- | ---: | ${variants.map(() => "---:").join(" | ")} |`); +for (const row of rows) { + const cells = variants.map(([key]) => { + const v = row[key]; + if (!v) return "—"; + const delta = pct(row.arx3.brotliBytes, v.brotliBytes); + return `${v.brotliBytes} (${fmtPct(delta)})`; + }); + w(`| ${row.name} | ${row.rawContentChars.toLocaleString("en-US")} | ${cells.join(" | ")} |`); +} +w(); + +w("### Visible chars @ baseBMP"); +w(); +w(`| Fixture | ARX3 BMP | best idea BMP | win | fits Discord (current host) | fits (short host) |`); +w(`| --- | ---: | ---: | ---: | :---: | :---: |`); +for (const row of rows) { + let bestKey = "arx3"; + let best = row.arx3.bmpChars; + for (const [key] of variants) { + const v = row[key]; + if (v && v.bmpChars < best) { + best = v.bmpChars; + bestKey = key; + } + } + const win = pct(row.arx3.bmpChars, best); + const label = variants.find(([k]) => k === bestKey)?.[1] ?? bestKey; + w( + `| ${row.name} | ${row.arx3.bmpChars} | ${best} (${label}) | ${fmtPct(win)} | ${fitsDiscord(row.arx3.bmpChars, budgets.currentHost.payloadBudget) ? "yes" : "no"} → ${fitsDiscord(best, budgets.currentHost.payloadBudget) ? "yes" : "no"} | ${fitsDiscord(best, budgets.shortHost.payloadBudget) ? "yes" : "no"} |`, + ); +} +w(); + +w("### Visible chars @ baseAstral (optimistic code-point counting)"); +w(); +w(`| Fixture | ARX3 BMP | ARX3 Astral | best idea Astral | astral win vs ARX3 BMP |`); +w(`| --- | ---: | ---: | ---: | ---: |`); +for (const row of rows) { + let best = row.arx3.astralChars; + let bestKey = "arx3"; + for (const [key] of variants) { + const v = row[key]; + if (v && v.astralChars < best) { + best = v.astralChars; + bestKey = key; + } + } + const label = variants.find(([k]) => k === bestKey)?.[1] ?? bestKey; + w( + `| ${row.name} | ${row.arx3.bmpChars} | ${row.arx3.astralChars} | ${best} (${label}) | ${fmtPct(pct(row.arx3.bmpChars, best))} |`, + ); +} +w(); + +// Aggregate +const totals = Object.fromEntries( + variants.map(([key]) => { + let brotliBytes = 0; + let bmpChars = 0; + let astralChars = 0; + let n = 0; + for (const row of rows) { + const v = row[key]; + if (!v) continue; + brotliBytes += v.brotliBytes; + bmpChars += v.bmpChars; + astralChars += v.astralChars; + n++; + } + return [key, { brotliBytes, bmpChars, astralChars, n }]; + }), +); + +w("### Totals (fixtures with a value for that variant)"); +w(); +w("| Variant | Σ brotli | Σ BMP chars | vs ARX3 BMP | Σ Astral chars | vs ARX3 BMP |"); +w("| --- | ---: | ---: | ---: | ---: | ---: |"); +for (const [key, label] of variants) { + const t = totals[key]; + // Compare only over rows where both exist — for sparse variants, compare against arx3 subset. + let arx3Bmp = 0; + for (const row of rows) { + if (!row[key]) continue; + arx3Bmp += row.arx3.bmpChars; + } + w( + `| ${label} | ${t.brotliBytes} | ${t.bmpChars} | ${fmtPct(pct(arx3Bmp, t.bmpChars))} | ${t.astralChars} | ${fmtPct(pct(arx3Bmp, t.astralChars))} |`, + ); +} +w(); + +w("## Discord capacity (how much raw text fits)"); +w(); +w("Binary-searching the largest source string whose encoded markdown link stays ≤ 2000 chars"); +w("(current-host framing, payload budget 1962) shows the practical ceiling:"); +w(); +w("| Content shape | ARX3 @ baseBMP | ARX3 @ baseAstral | Content-first @ BMP | Notes |"); +w("| --- | ---: | ---: | ---: | --- |"); +w("| Tiled real report + unique headers | ≥120k (search cap) | ≥150k (~+25%) | ≥120k | Highly compressible; Discord is not the bottleneck |"); +w("| Generated TS helpers | ≥120k (search cap) | ≥150k (~+25%) | ≥120k | Same |"); +w("| Quote/newline-heavy prose | ~88k | ~112k (~+27%) | ~87k | JSON escaping barely matters once Brotli runs |"); +w("| Current `code-bench-report.md` (8.3k) | ~1117 BMP chars | ~891 astral | ~1117 | Uses ~57% of Discord budget today |"); +w(); +w("**Reading:** for Discord, ARX3 already leaves a lot of headroom on typical artifacts. The"); +w("ways to *raise the ceiling* are almost entirely (1) denser Unicode wire and (2) fewer"); +w("compressed bytes via shared dictionaries / better envelopes — not more text substitutions."); +w(); + +w("## Interpretation"); +w(); +w("### What already works in ARX3"); +w(); +w("- For typical single artifacts under ~8–12 KB of source, ARX3 baseBMP already fits Discord"); +w(" with room to spare (see `small-markdown`, `json-package`, `code-fragment`, and the"); +w(" 8.3k code-bench report at ~57% of budget)."); +w("- The painful case is large unique prose/reports where dictionary substitution helps less"); +w(" and Brotli carries most of the work — and even then, baseAstral is the cleanest capacity"); +w(" unlock if Discord counting cooperates."); +w(); +w("### Highest-leverage ARX4 directions"); +w(); +w("1. **Validate Discord length semantics, then consider baseAstral**"); +w(" - Pure wire change on top of identical ARX3 bytes."); +w(" - ~20–26% fewer visible units *and* ~25% more raw capacity *if* Discord counts code points."); +w(" - Zero win (or a loss) if any hop counts UTF-16 code units."); +w(" - Empirically test: paste a `#d` + astral payload link into Discord desktop/mobile/web."); +w(); +w("2. **Content-first / CBOR binary envelope (skip JSON)**"); +w(" - Alone: small (~0–5%) on already-substituted ARX3 text."); +w(" - Combined with a shared-dict estimate: best corpus win here (~8% fewer BMP chars)."); +w(" - Removes JSON escaping of newlines/quotes; more important for pathological quote-heavy bodies."); +w(" - Natural fit for Discord's single-artifact share path; multi-artifact can keep tuples."); +w(); +w("3. **Brotli shared static dictionary**"); +w(" - Node cannot set a Brotli custom dictionary; numbers are estimates / deflate proxies."); +w(" - Still worth pursuing if `brotli-wasm` gains dictionary APIs — small/medium payloads benefit most."); +w(" - Deflate+dict is a real shared-dict signal but often *loses* to plain Brotli on this corpus"); +w(" (+17% bytes) — do not switch away from Brotli; only add a true Brotli dictionary."); +w(); +w("4. **Mined / larger overlay dictionaries**"); +w(" - Alone, mined n-grams *regressed* this corpus (+3–4%)."); +w(" - Prefer a carefully curated ARX4 overlay over online mining; measure before shipping."); +w(); +w("5. **Discord framing (orthogonal, ship anytime)**"); +w(" - Short branded host + 1-char label recovers ~15–30 chars — small but free."); +w(" - Agents should emit `[x](https://short/#…)` when targeting Discord."); +w(); +w("### Suggested ARX4 shape (if pursued)"); +w(); +w("```text"); +w("artifact bytes"); +w(" → content-first binary envelope (kind|id|content|meta) // or CBOR tuple for bundles"); +w(" → optional curated ARX4 overlay (domain n-grams, measured)"); +w(" → Brotli q11 (+ shared static dictionary if wasm allows)"); +w(" → baseBMP (safe) or baseAstral (Discord-validated)"); +w(" → compact tag `d` / `e`"); +w("```"); +w(); +w("Selection policy: optimize `markdownLink.length` for a declared surface"); +w("(`discord` | `visible` | `transport`) instead of only fragment length."); +w(); +w("## Non-goals / traps"); +w(); +w("- Do not weaken the 8192 fragment budget or 200k decoded budget for Discord wins."); +w("- Do not put artifact bodies in query params."); +w("- Do not assume astral density without client paste tests."); +w("- Do not replace UUID mode: hostile link scanners still want short opaque URLs."); +w("- Do not grow substitution dictionaries without a corpus gate — mining can regress."); +w(); +w("## How to re-run"); +w(); +w("```bash"); +w("npm run bench:arx4-ideation"); +w("# or: node scripts/arx4-ideation-probe.mjs"); +w("```"); +w(); +w(`_Generated in ${(performance.now() - start).toFixed(1)}ms._`); + +const report = lines.join("\n") + "\n"; +writeFileSync(REPORT_PATH, report); + +// Also print a compact console summary. +console.log("ARX4 ideation probe\n"); +console.log("Discord payload budgets:"); +for (const [name, b] of Object.entries(budgets)) { + console.log(` ${name}: framing=${b.framingLength} payload=${b.payloadBudget} maxBMP=${maxBytesForBudget(b.payloadBudget, density.baseBMP)}B`); +} +console.log("\nPer-fixture BMP chars (ARX3 → best):"); +for (const row of rows) { + let best = row.arx3.bmpChars; + let bestKey = "arx3"; + for (const [key] of variants) { + const v = row[key]; + if (v && v.bmpChars < best) { + best = v.bmpChars; + bestKey = key; + } + } + console.log( + ` ${row.name.padEnd(20)} arx3=${String(row.arx3.bmpChars).padStart(5)} best=${String(best).padStart(5)} (${bestKey}) astral(arx3)=${row.arx3.astralChars}`, + ); +} +console.log(`\nWrote ${REPORT_PATH}`); From 158f5dcef3df39dffc62ebbf27adc7f6e10f235e Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Wed, 8 Jul 2026 18:38:15 +0000 Subject: [PATCH 2/5] Revise ARX4 bets after Discord length research. Web evidence favors UTF-16 client counting (emoji=2), so deprioritize baseAstral for Discord; keep content-first + real Brotli dict as the main exploration path and drop framing from ARX4 scope. Co-authored-by: Aanish Bhirud --- docs/arx4-ideation.md | 119 +++++++++++++++++--------------- scripts/arx4-ideation-probe.mjs | 92 +++++++++++++----------- 2 files changed, 113 insertions(+), 98 deletions(-) diff --git a/docs/arx4-ideation.md b/docs/arx4-ideation.md index 157153d..f177f0f 100644 --- a/docs/arx4-ideation.md +++ b/docs/arx4-ideation.md @@ -21,9 +21,9 @@ So ARX4 should optimize **Discord markdown-link length**, not just fragment leng | shortHost (`[a](https://arx.page#d…)`) | 23 | 1977 | 3928 | 4934 | | bareHost (`[x](https://r.page#d…)`) | 21 | 1979 | 3932 | 4939 | -Takeaway: host + label overhead is only ~20–50 chars. The real ceiling is ~3.8–3.9 KB of -compressed bytes under baseBMP, or ~4.9 KB if a denser Unicode wire survives Discord's -character counting as one unit per code point. +Takeaway: host + label overhead is only ~20–50 chars (and short labels are already the +agent skill default). The real ceiling is ~3.8–3.9 KB of compressed bytes under baseBMP. +Astral density (~4.9 KB) is not a Discord win under UTF-16 counting — see research note. ## Wire density ceiling @@ -33,24 +33,36 @@ character counting as one unit per code point. | base76 | 6.27 | ASCII fragment-safe | | base1k | 10.79 | BMP subset | | baseBMP (ARX3) | 15.92 | Current best visible density | -| baseAstral (code points) | 20.00 | ~26% denser **if** Discord counts code points | -| baseAstral (UTF-16 units) | 10.00 | **Worse** than baseBMP if length is UTF-16 | +| baseAstral (code points) | 20.00 | ~26% denser **only if** Discord counts code points | +| baseAstral (UTF-16 units) | 10.00 | **Worse** than baseBMP (~10 vs ~15.92 bits/unit) | -**Critical unknown:** Discord's message length appears to count Unicode code points for -most text, but markdown link destinations and client paste paths need an empirical check -before shipping an astral wire. If any hop counts UTF-16, astral loses to baseBMP. +### Research note — Discord length counting (2026-07 web pass) + +Public sources disagree; the **client-facing** signal matters most for paste-to-send: + +| Source | Claim | Weight | +| --- | --- | --- | +| TypeCount / Discord character-counter guides | Standard emoji count as **2** toward the 2000 limit ("Unicode encoding") | High for UX — matches JS/Electron `.length` | +| Discord desktop stack | Electron → JS strings → UTF-16 code units | High — composer counter almost certainly uses this | +| Our product (`markdown-link.ts`) | Already gates on `markdownLink.length` (UTF-16 units) | Aligns with client-side folk wisdom | +| twilight-interactions #41 | Slash-command option `min/max_length` uses Unicode **code points** (Python `len()`), not UTF-8 bytes | Medium — different API surface than message `content` | +| Secondary blogs (go-tools, discord-webhook) | "Code points, emoji = 1" | Low — contradicted by emoji=2 guides; some validators still use JS `.length` | + +**Working conclusion:** treat Discord message limits as **UTF-16-unit** (JS `.length`) until a live paste test proves otherwise. +That **kills baseAstral as a Discord win** — astral scalars cost 2 units each, so density drops below baseBMP. +Keep a one-shot paste test on the backlog (1999 BMP chars vs 1000 astral + framing) only to close the API-vs-client gap; do not prototype baseAstral for Discord first. ## Ideas probed -1. **baseAstral wire** — pack into supplementary-plane scalars (~20 bits/code point). -2. **CBOR-ish binary tuple** — drop JSON quotes/escapes around the ARX2/3 tuple. +1. **baseAstral wire** — pack into supplementary-plane scalars (~20 bits/code point). **Deprioritized for Discord** after UTF-16 research. +2. **CBOR-ish binary tuple** — drop JSON quotes/escapes around the ARX2/3 tuple. **Worth exploring.** 3. **Brotli shared static dictionary** — seed Brotli with domain patterns already in `/arx-dictionary.json`. Node zlib cannot set a Brotli custom dictionary today, so the probe reports (a) a residual - `brotli(dict‖data)−brotli(dict)` estimate and (b) a real `deflateRaw+dictionary` proxy. -4. **Content-first binary envelope** — compress raw artifact bytes + tiny binary header, skip JSON entirely. + `brotli(dict‖data)−brotli(dict)` estimate and (b) a real `deflateRaw+dictionary` proxy. **Worth exploring** via wasm. +4. **Content-first binary envelope** — compress raw artifact bytes + tiny binary header, skip JSON entirely. **Worth exploring.** 5. **Mined overlay growth** — corpus-mined n-grams as an extra substitution layer. 6. **Combined ARX4 stack** — content-first + mined + shared-dict estimate. -7. **Discord framing** — short host + 1-char label + compact tag (cheap, orthogonal). +7. **Discord framing** — short host + 1-char label + compact tag. **Already practiced** (skill uses short labels; host shortening is DNS, not codec). ## Corpus results @@ -105,18 +117,18 @@ Percentages are vs ARX3 baseBMP visible chars (negative = larger / worse). ## Discord capacity (how much raw text fits) Binary-searching the largest source string whose encoded markdown link stays ≤ 2000 chars -(current-host framing, payload budget 1962): +(current-host framing, payload budget 1962) shows the practical ceiling: | Content shape | ARX3 @ baseBMP | ARX3 @ baseAstral | Content-first @ BMP | Notes | | --- | ---: | ---: | ---: | --- | -| Tiled real report + unique section headers | ≥120k (search cap) | ≥150k (+25%) | ≥120k | Highly compressible; Discord is not the bottleneck | -| Generated TS helpers | ≥120k (search cap) | ≥150k (+25%) | ≥120k | Same | -| Quote/newline-heavy prose | ~88k | ~112k (+27%) | ~87k | JSON escaping barely matters once Brotli runs | -| Current `code-bench-report.md` (8.3k) | 1117 BMP chars | ~891 astral | 1117 | Uses ~57% of Discord budget today | +| Tiled real report + unique headers | ≥120k (search cap) | ≥150k (~+25%) | ≥120k | Highly compressible; Discord is not the bottleneck | +| Generated TS helpers | ≥120k (search cap) | ≥150k (~+25%) | ≥120k | Same | +| Quote/newline-heavy prose | ~88k | ~112k (~+27%) | ~87k | JSON escaping barely matters once Brotli runs | +| Current `code-bench-report.md` (8.3k) | ~1117 BMP chars | ~891 astral | ~1117 | Uses ~57% of Discord budget today | -**Reading:** for Discord, ARX3 already leaves a lot of headroom on typical artifacts. The -ways to *raise the ceiling* are almost entirely (1) denser Unicode wire and (2) fewer -compressed bytes via shared dictionaries / better envelopes — not more text substitutions. +**Reading:** for Discord, ARX3 already leaves a lot of headroom on typical artifacts. With +baseAstral deprioritized (UTF-16), the ways to *raise the ceiling* are fewer compressed +bytes via shared dictionaries / better envelopes — not denser Unicode wire or more text substitutions. ## Interpretation @@ -126,65 +138,60 @@ compressed bytes via shared dictionaries / better envelopes — not more text su with room to spare (see `small-markdown`, `json-package`, `code-fragment`, and the 8.3k code-bench report at ~57% of budget). - The painful case is large unique prose/reports where dictionary substitution helps less - and Brotli carries most of the work — and even then, baseAstral is the cleanest capacity - unlock if Discord counting cooperates. - -### Highest-leverage ARX4 directions + and Brotli carries most of the work — chase bytes there, not astral wire. -1. **Validate Discord length semantics, then consider baseAstral** - - Pure wire change on top of identical ARX3 bytes. - - ~20–26% fewer visible units *and* ~25% more raw capacity *if* Discord counts code points. - - Zero win (or a loss) if any hop counts UTF-16 code units. - - Empirically test: paste a `#d` + astral payload link into Discord desktop/mobile/web. +### Ranked bets for ARX4 -2. **Content-first binary envelope (skip JSON)** - - Removes JSON escaping of newlines/quotes inside artifact bodies. - - Largest win on code and markdown with many `"` / `\n` sequences. - - Natural fit for Discord's single-artifact share path; multi-artifact can keep tuples. +1. **Content-first binary envelope + CBOR/binary tuple (worth exploring)** + - Alone: small (~0–5%) on already-substituted ARX3 text. + - Combined with a shared-dict estimate: best corpus BMP win here (~2–28% depending on fixture). + - Removes JSON escaping of newlines/quotes; natural fit for Discord's single-artifact share path. -3. **Brotli shared static dictionary** - - Node cannot set a Brotli custom dictionary; numbers are estimates / deflate proxies. - - Still worth pursuing if `brotli-wasm` gains dictionary APIs — small/medium payloads benefit most. - - Deflate+dict is a real shared-dict signal and often beats plain Brotli on repetitive agent text, - but losing Brotli's stronger model is a tradeoff; prefer true Brotli dict over switching codecs. +2. **Real Brotli shared static dictionary (worth exploring)** + - Node cannot set a Brotli custom dictionary; residual / deflate+dict are proxies only. + - Next step: check whether `brotli-wasm` (or another browser-safe Brotli) accepts a custom dict. + - Do **not** switch the pipeline to deflate+dict — that proxy often *regressed* vs plain Brotli (~+17%). -4. **CBOR / binary tuple** - - Modest win once Brotli has already seen the JSON structure. - - More valuable when combined with content-first (binary header + raw body). +3. **Curated overlay growth (cautious)** + - Alone, mined n-grams *regressed* this corpus (+3–4%). + - Prefer a carefully curated ARX4 overlay over online mining; measure before shipping. -5. **Mined / larger overlay dictionaries** - - Helps repeated fixture-like corpora; risky for open-ended agent output. - - Prefer shipping a carefully curated ARX4 overlay over online mining. +4. **baseAstral — deprioritized for Discord** + - Web evidence favors UTF-16 client counting → astral loses to baseBMP (~10 vs ~15.92 bits/unit). + - Optional one-shot paste test only; not a primary ARX4 bet. -6. **Discord framing (orthogonal, ship anytime)** - - Short branded host + 1-char label recovers ~30 chars — small but free. - - Agents should emit `[x](https://short/#…)` when targeting Discord. +5. **Discord framing — already practiced, not an ARX4 lever** + - Skill/agents already use short labels (`[Short summary](…)`); product warns on full `markdownLink.length`. + - Host shortening (`arx.page`) is deployment/DNS, not a codec change — drop from ARX4 scope. ### Suggested ARX4 shape (if pursued) ```text artifact bytes - → content-first binary envelope (kind|id|content|meta) - → optional curated ARX4 overlay (domain n-grams) + → content-first binary envelope (kind|id|content|meta) // or CBOR tuple for bundles + → optional curated ARX4 overlay (domain n-grams, measured) → Brotli q11 (+ shared static dictionary if wasm allows) - → baseBMP (safe) or baseAstral (Discord-validated) + → baseBMP (Discord-safe; skip astral unless paste tests overturn UTF-16 finding) → compact tag `d` / `e` ``` -Selection policy: optimize `markdownLink.length` for a declared surface -(`discord` | `visible` | `transport`) instead of only fragment length. +Selection policy: optimize `markdownLink.length` (JS/UTF-16 units, matching Discord client +folk counting) for a declared surface (`discord` | `visible` | `transport`). ## Non-goals / traps - Do not weaken the 8192 fragment budget or 200k decoded budget for Discord wins. - Do not put artifact bodies in query params. -- Do not assume astral density without client paste tests. +- Do not chase baseAstral for Discord until a live paste test overturns UTF-16 counting. +- Do not treat short-host framing as an ARX4 deliverable. - Do not replace UUID mode: hostile link scanners still want short opaque URLs. +- Do not grow substitution dictionaries without a corpus gate — mining can regress. ## How to re-run ```bash -node scripts/arx4-ideation-probe.mjs +npm run bench:arx4-ideation +# or: node scripts/arx4-ideation-probe.mjs ``` -_Generated in 10.5ms._ +_Generated in 9.6ms._ diff --git a/scripts/arx4-ideation-probe.mjs b/scripts/arx4-ideation-probe.mjs index a4c5b7e..eb0ff3e 100644 --- a/scripts/arx4-ideation-probe.mjs +++ b/scripts/arx4-ideation-probe.mjs @@ -624,9 +624,9 @@ for (const [name, b] of Object.entries(budgets)) { ); } w(); -w("Takeaway: host + label overhead is only ~20–50 chars. The real ceiling is ~3.8–3.9 KB of"); -w("compressed bytes under baseBMP, or ~4.9 KB if a denser Unicode wire survives Discord's"); -w("character counting as one unit per code point."); +w("Takeaway: host + label overhead is only ~20–50 chars (and short labels are already the"); +w("agent skill default). The real ceiling is ~3.8–3.9 KB of compressed bytes under baseBMP."); +w("Astral density (~4.9 KB) is not a Discord win under UTF-16 counting — see research note."); w(); w("## Wire density ceiling"); w(); @@ -636,24 +636,36 @@ w(`| base64url | ${density.base64url.toFixed(2)} | ASCII-safe; ARX2 default on h w(`| base76 | ${density.base76.toFixed(2)} | ASCII fragment-safe |`); w(`| base1k | ${density.base1k.toFixed(2)} | BMP subset |`); w(`| baseBMP (ARX3) | ${density.baseBMP.toFixed(2)} | Current best visible density |`); -w(`| baseAstral (code points) | ${density.baseAstral1M.toFixed(2)} | ~${((density.baseAstral1M / density.baseBMP - 1) * 100).toFixed(0)}% denser **if** Discord counts code points |`); -w(`| baseAstral (UTF-16 units) | ${density.baseAstralUtf16Effective.toFixed(2)} | **Worse** than baseBMP if length is UTF-16 |`); +w(`| baseAstral (code points) | ${density.baseAstral1M.toFixed(2)} | ~${((density.baseAstral1M / density.baseBMP - 1) * 100).toFixed(0)}% denser **only if** Discord counts code points |`); +w(`| baseAstral (UTF-16 units) | ${density.baseAstralUtf16Effective.toFixed(2)} | **Worse** than baseBMP (~${density.baseAstralUtf16Effective.toFixed(0)} vs ~${density.baseBMP.toFixed(2)} bits/unit) |`); w(); -w("**Critical unknown:** Discord's message length appears to count Unicode code points for"); -w("most text, but markdown link destinations and client paste paths need an empirical check"); -w("before shipping an astral wire. If any hop counts UTF-16, astral loses to baseBMP."); +w("### Research note — Discord length counting (2026-07 web pass)"); +w(); +w("Public sources disagree; the **client-facing** signal matters most for paste-to-send:"); +w(); +w("| Source | Claim | Weight |"); +w("| --- | --- | --- |"); +w("| TypeCount / Discord character-counter guides | Standard emoji count as **2** toward the 2000 limit (\"Unicode encoding\") | High for UX — matches JS/Electron `.length` |"); +w("| Discord desktop stack | Electron → JS strings → UTF-16 code units | High — composer counter almost certainly uses this |"); +w("| Our product (`markdown-link.ts`) | Already gates on `markdownLink.length` (UTF-16 units) | Aligns with client-side folk wisdom |"); +w("| twilight-interactions #41 | Slash-command option `min/max_length` uses Unicode **code points** (Python `len()`), not UTF-8 bytes | Medium — different API surface than message `content` |"); +w("| Secondary blogs (go-tools, discord-webhook) | \"Code points, emoji = 1\" | Low — contradicted by emoji=2 guides; some validators still use JS `.length` |"); +w(); +w("**Working conclusion:** treat Discord message limits as **UTF-16-unit** (JS `.length`) until a live paste test proves otherwise."); +w("That **kills baseAstral as a Discord win** — astral scalars cost 2 units each, so density drops below baseBMP."); +w("Keep a one-shot paste test on the backlog (1999 BMP chars vs 1000 astral + framing) only to close the API-vs-client gap; do not prototype baseAstral for Discord first."); w(); w("## Ideas probed"); w(); -w("1. **baseAstral wire** — pack into supplementary-plane scalars (~20 bits/code point)."); -w("2. **CBOR-ish binary tuple** — drop JSON quotes/escapes around the ARX2/3 tuple."); +w("1. **baseAstral wire** — pack into supplementary-plane scalars (~20 bits/code point). **Deprioritized for Discord** after UTF-16 research."); +w("2. **CBOR-ish binary tuple** — drop JSON quotes/escapes around the ARX2/3 tuple. **Worth exploring.**"); w("3. **Brotli shared static dictionary** — seed Brotli with domain patterns already in `/arx-dictionary.json`."); w(" Node zlib cannot set a Brotli custom dictionary today, so the probe reports (a) a residual"); -w(" `brotli(dict‖data)−brotli(dict)` estimate and (b) a real `deflateRaw+dictionary` proxy."); -w("4. **Content-first binary envelope** — compress raw artifact bytes + tiny binary header, skip JSON entirely."); +w(" `brotli(dict‖data)−brotli(dict)` estimate and (b) a real `deflateRaw+dictionary` proxy. **Worth exploring** via wasm."); +w("4. **Content-first binary envelope** — compress raw artifact bytes + tiny binary header, skip JSON entirely. **Worth exploring.**"); w("5. **Mined overlay growth** — corpus-mined n-grams as an extra substitution layer."); w("6. **Combined ARX4 stack** — content-first + mined + shared-dict estimate."); -w("7. **Discord framing** — short host + 1-char label + compact tag (cheap, orthogonal)."); +w("7. **Discord framing** — short host + 1-char label + compact tag. **Already practiced** (skill uses short labels; host shortening is DNS, not codec)."); w(); w("## Corpus results"); w(); @@ -781,9 +793,9 @@ w("| Generated TS helpers | ≥120k (search cap) | ≥150k (~+25%) | ≥120k | S w("| Quote/newline-heavy prose | ~88k | ~112k (~+27%) | ~87k | JSON escaping barely matters once Brotli runs |"); w("| Current `code-bench-report.md` (8.3k) | ~1117 BMP chars | ~891 astral | ~1117 | Uses ~57% of Discord budget today |"); w(); -w("**Reading:** for Discord, ARX3 already leaves a lot of headroom on typical artifacts. The"); -w("ways to *raise the ceiling* are almost entirely (1) denser Unicode wire and (2) fewer"); -w("compressed bytes via shared dictionaries / better envelopes — not more text substitutions."); +w("**Reading:** for Discord, ARX3 already leaves a lot of headroom on typical artifacts. With"); +w("baseAstral deprioritized (UTF-16), the ways to *raise the ceiling* are fewer compressed"); +w("bytes via shared dictionaries / better envelopes — not denser Unicode wire or more text substitutions."); w(); w("## Interpretation"); @@ -794,36 +806,31 @@ w("- For typical single artifacts under ~8–12 KB of source, ARX3 baseBMP alrea w(" with room to spare (see `small-markdown`, `json-package`, `code-fragment`, and the"); w(" 8.3k code-bench report at ~57% of budget)."); w("- The painful case is large unique prose/reports where dictionary substitution helps less"); -w(" and Brotli carries most of the work — and even then, baseAstral is the cleanest capacity"); -w(" unlock if Discord counting cooperates."); +w(" and Brotli carries most of the work — chase bytes there, not astral wire."); w(); -w("### Highest-leverage ARX4 directions"); +w("### Ranked bets for ARX4"); w(); -w("1. **Validate Discord length semantics, then consider baseAstral**"); -w(" - Pure wire change on top of identical ARX3 bytes."); -w(" - ~20–26% fewer visible units *and* ~25% more raw capacity *if* Discord counts code points."); -w(" - Zero win (or a loss) if any hop counts UTF-16 code units."); -w(" - Empirically test: paste a `#d` + astral payload link into Discord desktop/mobile/web."); -w(); -w("2. **Content-first / CBOR binary envelope (skip JSON)**"); +w("1. **Content-first binary envelope + CBOR/binary tuple (worth exploring)**"); w(" - Alone: small (~0–5%) on already-substituted ARX3 text."); -w(" - Combined with a shared-dict estimate: best corpus win here (~8% fewer BMP chars)."); -w(" - Removes JSON escaping of newlines/quotes; more important for pathological quote-heavy bodies."); -w(" - Natural fit for Discord's single-artifact share path; multi-artifact can keep tuples."); +w(" - Combined with a shared-dict estimate: best corpus BMP win here (~2–28% depending on fixture)."); +w(" - Removes JSON escaping of newlines/quotes; natural fit for Discord's single-artifact share path."); w(); -w("3. **Brotli shared static dictionary**"); -w(" - Node cannot set a Brotli custom dictionary; numbers are estimates / deflate proxies."); -w(" - Still worth pursuing if `brotli-wasm` gains dictionary APIs — small/medium payloads benefit most."); -w(" - Deflate+dict is a real shared-dict signal but often *loses* to plain Brotli on this corpus"); -w(" (+17% bytes) — do not switch away from Brotli; only add a true Brotli dictionary."); +w("2. **Real Brotli shared static dictionary (worth exploring)**"); +w(" - Node cannot set a Brotli custom dictionary; residual / deflate+dict are proxies only."); +w(" - Next step: check whether `brotli-wasm` (or another browser-safe Brotli) accepts a custom dict."); +w(" - Do **not** switch the pipeline to deflate+dict — that proxy often *regressed* vs plain Brotli (~+17%)."); w(); -w("4. **Mined / larger overlay dictionaries**"); +w("3. **Curated overlay growth (cautious)**"); w(" - Alone, mined n-grams *regressed* this corpus (+3–4%)."); w(" - Prefer a carefully curated ARX4 overlay over online mining; measure before shipping."); w(); -w("5. **Discord framing (orthogonal, ship anytime)**"); -w(" - Short branded host + 1-char label recovers ~15–30 chars — small but free."); -w(" - Agents should emit `[x](https://short/#…)` when targeting Discord."); +w("4. **baseAstral — deprioritized for Discord**"); +w(" - Web evidence favors UTF-16 client counting → astral loses to baseBMP (~10 vs ~15.92 bits/unit)."); +w(" - Optional one-shot paste test only; not a primary ARX4 bet."); +w(); +w("5. **Discord framing — already practiced, not an ARX4 lever**"); +w(" - Skill/agents already use short labels (`[Short summary](…)`); product warns on full `markdownLink.length`."); +w(" - Host shortening (`arx.page`) is deployment/DNS, not a codec change — drop from ARX4 scope."); w(); w("### Suggested ARX4 shape (if pursued)"); w(); @@ -832,18 +839,19 @@ w("artifact bytes"); w(" → content-first binary envelope (kind|id|content|meta) // or CBOR tuple for bundles"); w(" → optional curated ARX4 overlay (domain n-grams, measured)"); w(" → Brotli q11 (+ shared static dictionary if wasm allows)"); -w(" → baseBMP (safe) or baseAstral (Discord-validated)"); +w(" → baseBMP (Discord-safe; skip astral unless paste tests overturn UTF-16 finding)"); w(" → compact tag `d` / `e`"); w("```"); w(); -w("Selection policy: optimize `markdownLink.length` for a declared surface"); -w("(`discord` | `visible` | `transport`) instead of only fragment length."); +w("Selection policy: optimize `markdownLink.length` (JS/UTF-16 units, matching Discord client"); +w("folk counting) for a declared surface (`discord` | `visible` | `transport`)."); w(); w("## Non-goals / traps"); w(); w("- Do not weaken the 8192 fragment budget or 200k decoded budget for Discord wins."); w("- Do not put artifact bodies in query params."); -w("- Do not assume astral density without client paste tests."); +w("- Do not chase baseAstral for Discord until a live paste test overturns UTF-16 counting."); +w("- Do not treat short-host framing as an ARX4 deliverable."); w("- Do not replace UUID mode: hostile link scanners still want short opaque URLs."); w("- Do not grow substitution dictionaries without a corpus gate — mining can regress."); w(); From 621bd0062f5292635877ef45f14fa08e33c3c49e Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Wed, 8 Jul 2026 18:50:41 +0000 Subject: [PATCH 3/5] Implement and bench ARX4 bet #2 (content-first + real Brotli dict). MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add an experimental content-first/CBOR envelope module with round-trip tests, and a bench that measures real `brotli -D` shared dictionaries against ARX3. Residual dict estimates overstated wins; real shared-dict gains are ~0–2%. Co-authored-by: Aanish Bhirud --- docs/arx4-bet2-bench.md | 108 ++++ docs/arx4-ideation.md | 29 +- package.json | 1 + scripts/arx4-ideation-probe.mjs | 6 +- scripts/bench-arx4-bet2.mjs | 741 ++++++++++++++++++++++++++ src/lib/payload/arx4-content-first.ts | 460 ++++++++++++++++ tests/arx4-content-first.test.ts | 143 +++++ 7 files changed, 1473 insertions(+), 15 deletions(-) create mode 100644 docs/arx4-bet2-bench.md create mode 100644 scripts/bench-arx4-bet2.mjs create mode 100644 src/lib/payload/arx4-content-first.ts create mode 100644 tests/arx4-content-first.test.ts diff --git a/docs/arx4-bet2-bench.md b/docs/arx4-bet2-bench.md new file mode 100644 index 0000000..70699b9 --- /dev/null +++ b/docs/arx4-bet2-bench.md @@ -0,0 +1,108 @@ +# ARX4 bet #2 — content-first / CBOR + real Brotli shared dictionary + +_Experimental bench from `scripts/bench-arx4-bet2.mjs`. Not a shipped codec._ + +## What was implemented + +1. **Content-first binary envelope** — `src/lib/payload/arx4-content-first.ts` + - Wire: `A4 | version | kind | id | content | meta` + - Round-trip tested; stamps rebuilt envelopes as `codec: "plain"` (ARX4 is not shipped). +2. **CBOR-ish tuple encoder** — same module; encodes the ARX2/3 tuple without JSON quotes. +3. **Real Brotli shared dictionary** — measured via system `brotli -D` (LZ77 raw dictionary). + - Node `zlib.brotliCompressSync({ dictionary })` **silently ignores** the option on Node 22. + - Product `brotli-wasm@3` has **no** custom-dictionary API. + - Residual `brotli(dict‖data)−brotli(dict)` is kept only as a calibration column. + +## Environment + +- Node: `v22.14.0` +- Brotli CLI available: **yes** +- Shared dictionary size: **1143** bytes (ARX slot text + scaffolding) +- Discord framing budget (current host): **1962** payload chars + +## Pre-compress sizes (envelope bytes before Brotli) + +| Fixture | ARX3 substituted | CBOR | CBOR+text-sub | Content-first | CF+text-sub | +| --- | ---: | ---: | ---: | ---: | ---: | +| markdown-agents | 6871 | 8056 | 7006 | 8050 | 7000 | +| code-bench-report | 8203 | 8373 | 8360 | 8366 | 8353 | +| code-fragment | 7278 | 8065 | 7037 | 8063 | 7070 | +| json-package | 346 | 308 | 294 | 307 | 294 | +| small-markdown | 183 | 217 | 186 | 225 | 204 | + +## Brotli bytes (q11) vs ARX3 + +| Fixture | raw chars | ARX3 (baseline) | ARX3 + residual dict est. | ARX3 + real Brotli −D | ARX3 deflate+dict (non-goal) | CBOR tuple + Brotli | CBOR + text-sub + Brotli | CBOR + real Brotli −D | CBOR + text-sub + real −D | Content-first + Brotli | Content-first + text-sub + Brotli | Content-first + real −D | Content-first + text-sub + real −D | Best real bet #2 candidate | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| markdown-agents | 8,000 | 401 (0.0%) | 351 (−12.5%) | 380 (−5.2%) | 471 (+17.5%) | 392 (−2.2%) | 397 (−1.0%) | 385 (−4.0%) | 379 (−5.5%) | 399 (−0.5%) | 403 (+0.5%) | 390 (−2.7%) | 387 (−3.5%) | 379 (−5.5%) | +| code-bench-report | 8,314 | 2219 (0.0%) | 2188 (−1.4%) | 2219 (0.0%) | 2624 (+18.3%) | 2208 (−0.5%) | 2209 (−0.5%) | 2202 (−0.8%) | 2206 (−0.6%) | 2217 (−0.1%) | 2222 (+0.1%) | 2217 (−0.1%) | 2219 (0.0%) | 2202 (−0.8%) | +| code-fragment | 8,000 | 308 (0.0%) | 272 (−11.7%) | 307 (−0.3%) | 360 (+16.9%) | 300 (−2.6%) | 296 (−3.9%) | 314 (+1.9%) | 290 (−5.8%) | 311 (+1.0%) | 310 (+0.6%) | 317 (+2.9%) | 299 (−2.9%) | 290 (−5.8%) | +| json-package | 259 | 182 (0.0%) | 141 (−22.5%) | 188 (+3.3%) | 184 (+1.1%) | 173 (−4.9%) | 173 (−4.9%) | 177 (−2.7%) | 177 (−2.7%) | 190 (+4.4%) | 182 (0.0%) | 181 (−0.5%) | 183 (+0.5%) | 177 (−2.7%) | +| small-markdown | 189 | 141 (0.0%) | 110 (−22.0%) | 143 (+1.4%) | 160 (+13.5%) | 160 (+13.5%) | 140 (−0.7%) | 144 (+2.1%) | 142 (+0.7%) | 184 (+30.5%) | 155 (+9.9%) | 177 (+25.5%) | 154 (+9.2%) | 142 (+0.7%) | + +## Visible chars @ baseBMP vs ARX3 + +| Fixture | ARX3 (baseline) | ARX3 + residual dict est. | ARX3 + real Brotli −D | ARX3 deflate+dict (non-goal) | CBOR tuple + Brotli | CBOR + text-sub + Brotli | CBOR + real Brotli −D | CBOR + text-sub + real −D | Content-first + Brotli | Content-first + text-sub + Brotli | Content-first + real −D | Content-first + text-sub + real −D | Best real bet #2 candidate | fits Discord? | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | :---: | +| markdown-agents | 205 (0.0%) | 180 (−12.2%) | 194 (−5.4%) | 240 (+17.1%) | 200 (−2.4%) | 203 (−1.0%) | 197 (−3.9%) | 194 (−5.4%) | 204 (−0.5%) | 206 (+0.5%) | 199 (−2.9%) | 198 (−3.4%) | 194 (−5.4%) | yes | +| code-bench-report | 1119 (0.0%) | 1103 (−1.4%) | 1119 (0.0%) | 1322 (+18.1%) | 1113 (−0.5%) | 1114 (−0.4%) | 1110 (−0.8%) | 1112 (−0.6%) | 1118 (−0.1%) | 1120 (+0.1%) | 1118 (−0.1%) | 1119 (0.0%) | 1110 (−0.8%) | yes | +| code-fragment | 158 (0.0%) | 140 (−11.4%) | 158 (0.0%) | 184 (+16.5%) | 154 (−2.5%) | 152 (−3.8%) | 161 (+1.9%) | 149 (−5.7%) | 160 (+1.3%) | 159 (+0.6%) | 163 (+3.2%) | 154 (−2.5%) | 149 (−5.7%) | yes | +| json-package | 95 (0.0%) | 74 (−22.1%) | 98 (+3.2%) | 96 (+1.1%) | 90 (−5.3%) | 90 (−5.3%) | 92 (−3.2%) | 92 (−3.2%) | 99 (+4.2%) | 95 (0.0%) | 94 (−1.1%) | 95 (0.0%) | 92 (−3.2%) | yes | +| small-markdown | 74 (0.0%) | 59 (−20.3%) | 75 (+1.4%) | 84 (+13.5%) | 84 (+13.5%) | 74 (0.0%) | 76 (+2.7%) | 75 (+1.4%) | 96 (+29.7%) | 81 (+9.5%) | 92 (+24.3%) | 81 (+9.5%) | 75 (+1.4%) | yes | + +## Totals (all fixtures) + +| Variant | Σ brotli | vs ARX3 | Σ BMP chars | vs ARX3 BMP | +| --- | ---: | ---: | ---: | ---: | +| ARX3 (baseline) | 3251 | 0.0% | 1651 | 0.0% | +| ARX3 + residual dict est. | 3062 | −5.8% | 1556 | −5.8% | +| ARX3 + real Brotli −D | 3237 | −0.4% | 1644 | −0.4% | +| ARX3 deflate+dict (non-goal) | 3799 | +16.9% | 1926 | +16.7% | +| CBOR tuple + Brotli | 3233 | −0.6% | 1641 | −0.6% | +| CBOR + text-sub + Brotli | 3215 | −1.1% | 1633 | −1.1% | +| CBOR + real Brotli −D | 3222 | −0.9% | 1636 | −0.9% | +| CBOR + text-sub + real −D | 3194 | −1.8% | 1622 | −1.8% | +| Content-first + Brotli | 3301 | +1.5% | 1677 | +1.6% | +| Content-first + text-sub + Brotli | 3272 | +0.6% | 1661 | +0.6% | +| Content-first + real −D | 3282 | +1.0% | 1666 | +0.9% | +| Content-first + text-sub + real −D | 3242 | −0.3% | 1647 | −0.2% | +| Best real bet #2 candidate | 3190 | −1.9% | 1620 | −1.9% | + +## Findings + +### Content-first / CBOR alone + +- Dropping JSON (CBOR) or skipping the tuple (content-first) changes pre-Brotli size, but + **after Brotli q11 the win vs ARX3 is small** — often within ~1%, and content-first alone + can *lose* on small fixtures because it skips ARX text substitution. +- Re-applying v1 text substitution *inside* content-first / CBOR fields recovers most of + that gap; see the `+ text-sub` columns. + +### Real Brotli shared dictionary (`brotli -D`) + +- Unlike the residual estimate (often −10% to −30%), **real LZ77 shared dictionaries are + modest** on this corpus (typically under ~1% total, fixture-dependent). +- Residual estimates systematically **overstate** the win; do not use them as a ship gate. +- Deflate+dictionary remains a **non-goal**: larger than plain Brotli on this corpus. + +### Product implications + +1. **Browser path blocked for real shared dicts today** — `brotli-wasm` has no dictionary + API; Node zlib ignores `dictionary`. Shipping ARX4 shared-dict needs a wasm fork or + alternate compressor with custom-dict support. +2. **Binary envelopes are still useful plumbing** (no JSON escaping, cleaner wire) but are + not a Discord capacity unlock by themselves on this corpus. +3. Prefer measuring with **real `brotli -D`** (or a dict-capable wasm) over residual proxies + before committing to a shared-dictionary protocol. +4. Next exploration should focus on **dict contents matched to the post-substitution byte + stream** (or a wasm dict path), not more residual optimism. + +## How to re-run + +```bash +# requires system `brotli` CLI for real −D columns (apt install brotli) +npm run bench:arx4-bet2 +# or: node scripts/bench-arx4-bet2.mjs +``` + +_Generated in 292.4ms._ diff --git a/docs/arx4-ideation.md b/docs/arx4-ideation.md index f177f0f..cbd84b6 100644 --- a/docs/arx4-ideation.md +++ b/docs/arx4-ideation.md @@ -142,25 +142,21 @@ bytes via shared dictionaries / better envelopes — not denser Unicode wire or ### Ranked bets for ARX4 -1. **Content-first binary envelope + CBOR/binary tuple (worth exploring)** - - Alone: small (~0–5%) on already-substituted ARX3 text. - - Combined with a shared-dict estimate: best corpus BMP win here (~2–28% depending on fixture). - - Removes JSON escaping of newlines/quotes; natural fit for Discord's single-artifact share path. +1. **Content-first / CBOR + real Brotli shared dict — explored (see `docs/arx4-bet2-bench.md`)** + - Implemented: `src/lib/payload/arx4-content-first.ts` + `npm run bench:arx4-bet2`. + - Residual dict estimates overstated wins (~6–8% corpus); **real `brotli -D` is ~0–1%**. + - Binary envelopes alone are not a Discord unlock; keep as plumbing, not the next capacity bet. + - Browser shared-dict still blocked (`brotli-wasm` has no dict API; Node zlib ignores it). -2. **Real Brotli shared static dictionary (worth exploring)** - - Node cannot set a Brotli custom dictionary; residual / deflate+dict are proxies only. - - Next step: check whether `brotli-wasm` (or another browser-safe Brotli) accepts a custom dict. - - Do **not** switch the pipeline to deflate+dict — that proxy often *regressed* vs plain Brotli (~+17%). - -3. **Curated overlay growth (cautious)** +2. **Curated overlay growth (cautious)** - Alone, mined n-grams *regressed* this corpus (+3–4%). - Prefer a carefully curated ARX4 overlay over online mining; measure before shipping. -4. **baseAstral — deprioritized for Discord** +3. **baseAstral — deprioritized for Discord** - Web evidence favors UTF-16 client counting → astral loses to baseBMP (~10 vs ~15.92 bits/unit). - Optional one-shot paste test only; not a primary ARX4 bet. -5. **Discord framing — already practiced, not an ARX4 lever** +4. **Discord framing — already practiced, not an ARX4 lever** - Skill/agents already use short labels (`[Short summary](…)`); product warns on full `markdownLink.length`. - Host shortening (`arx.page`) is deployment/DNS, not a codec change — drop from ARX4 scope. @@ -170,7 +166,7 @@ bytes via shared dictionaries / better envelopes — not denser Unicode wire or artifact bytes → content-first binary envelope (kind|id|content|meta) // or CBOR tuple for bundles → optional curated ARX4 overlay (domain n-grams, measured) - → Brotli q11 (+ shared static dictionary if wasm allows) + → Brotli q11 (+ real shared dict only if browser wasm gains dict support) → baseBMP (Discord-safe; skip astral unless paste tests overturn UTF-16 finding) → compact tag `d` / `e` ``` @@ -178,6 +174,9 @@ artifact bytes Selection policy: optimize `markdownLink.length` (JS/UTF-16 units, matching Discord client folk counting) for a declared surface (`discord` | `visible` | `transport`). +Bet #2 takeaway: chase **dict-capable wasm** or a better post-substitution dictionary before +expecting Discord capacity gains from binary envelopes. + ## Non-goals / traps - Do not weaken the 8192 fragment budget or 200k decoded budget for Discord wins. @@ -192,6 +191,10 @@ folk counting) for a declared surface (`discord` | `visible` | `transport`). ```bash npm run bench:arx4-ideation # or: node scripts/arx4-ideation-probe.mjs + +# Bet #2 follow-up (content-first/CBOR + real brotli -D): +npm run bench:arx4-bet2 ``` _Generated in 9.6ms._ +See also `docs/arx4-bet2-bench.md` for the real shared-dictionary follow-up. diff --git a/package.json b/package.json index 56da8d1..ecae8dd 100644 --- a/package.json +++ b/package.json @@ -36,6 +36,7 @@ "bench:codecs": "node scripts/bench-codecs.mjs", "bench:codecs:update": "node scripts/bench-codecs.mjs --write-baseline", "bench:arx4-ideation": "node scripts/arx4-ideation-probe.mjs", + "bench:arx4-bet2": "node scripts/bench-arx4-bet2.mjs", "assets:compress": "node scripts/compress-dictionary.mjs", "typecheck": "node scripts/ensure-next-types.mjs && tsc --noEmit", "check": "npm run lint && npm run test && npm run bench:codecs && npm run typecheck && npm run build && npm run check:build-budgets", diff --git a/scripts/arx4-ideation-probe.mjs b/scripts/arx4-ideation-probe.mjs index eb0ff3e..ba30f20 100644 --- a/scripts/arx4-ideation-probe.mjs +++ b/scripts/arx4-ideation-probe.mjs @@ -107,10 +107,12 @@ function brotli(input) { /** * Approximate a Brotli *shared static dictionary* win. * - * Node's zlib brotli bindings ignore custom dictionaries, so we use two proxies: + * Node 22's zlib brotli bindings silently ignore `{ dictionary }`, and product + * `brotli-wasm` has no dict API. This probe therefore reports: * 1. deflateRaw + dictionary (real shared-dict API) as a lower-bound signal * 2. residual trick: len(brotli(dict||data)) - len(brotli(dict)) as an optimistic - * estimate of what a true Brotli custom dictionary might achieve + * estimate (systematically overstates — see `npm run bench:arx4-bet2` for real + * `brotli -D` numbers) * * Product ARX4 would need brotli-wasm (or equivalent) with dictionary support. */ diff --git a/scripts/bench-arx4-bet2.mjs b/scripts/bench-arx4-bet2.mjs new file mode 100644 index 0000000..ce74588 --- /dev/null +++ b/scripts/bench-arx4-bet2.mjs @@ -0,0 +1,741 @@ +#!/usr/bin/env node +/** + * ARX4 bet #2 bench — content-first / CBOR + real Brotli shared dictionary. + * + * Experimental only. Does not change the shipped codec surface. + * + * Measures against ARX3 (tuple JSON → overlay → v1 dict → Brotli q11 → baseBMP): + * 1. Content-first binary envelope + Brotli q11 + * 2. CBOR-ish ARX tuple + Brotli q11 + * 3. Each of the above + a *real* Brotli LZ77 shared dictionary via the + * system `brotli -D` CLI (Node zlib / brotli-wasm ignore custom dicts) + * 4. Residual `brotli(dict‖data)−brotli(dict)` estimate (shown for calibration) + * 5. DeflateRaw+dictionary proxy (shown as a non-goal — usually worse) + * + * Writes `docs/arx4-bet2-bench.md`. + */ + +import { spawnSync } from "node:child_process"; +import { mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { brotliCompressSync, deflateRawSync, constants } from "node:zlib"; +import { performance } from "node:perf_hooks"; + +// Encode logic here mirrors src/lib/payload/arx4-content-first.ts so this bench +// stays runnable with plain `node` like the other scripts. The TypeScript module +// is the canonical round-tripable API. + +const REPORT_PATH = "docs/arx4-bet2-bench.md"; +const DISCORD_MESSAGE_MAX = 2000; +const BMP_BASE_SIZE = 62_000; + +const v1Dictionary = JSON.parse(readFileSync("public/arx-dictionary.json", "utf8")); +const overlayDictionary = JSON.parse(readFileSync("public/arx2-dictionary.json", "utf8")); +const codeBenchReportFixture = readFileSync("tests/fixtures/baanish-code-bench-report.md", "utf8"); + +const singleByteCodes = [ + 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x0b, 0x0e, 0x0f, 0x10, 0x11, 0x12, 0x13, 0x14, 0x15, + 0x16, 0x17, 0x18, 0x19, 0x1a, 0x1b, 0x1c, 0x1d, +]; + +function buildPairs(dict, singleCodes = singleByteCodes, extendedPrefix = "\x00", extendedOffset = 1) { + const pairs = []; + for (let i = 0; i < dict.singleByteSlots.length && i < singleCodes.length; i++) { + pairs.push([dict.singleByteSlots[i], String.fromCharCode(singleCodes[i])]); + } + for (let i = 0; i < dict.extendedSlots.length; i++) { + pairs.push([dict.extendedSlots[i], extendedPrefix + String.fromCharCode(i + extendedOffset)]); + } + return pairs; +} + +const v1Pairs = buildPairs(v1Dictionary); +const overlayPairs = buildPairs(overlayDictionary, [0x1e, 0x7f], "\x1f", 0x20); + +function buildTrie(pairs, reversed = false) { + const root = { children: new Map() }; + for (const [from, to] of pairs) { + const match = reversed ? to : from; + const replacement = reversed ? from : to; + let node = root; + for (const char of match) { + let child = node.children.get(char); + if (!child) { + child = { children: new Map() }; + node.children.set(char, child); + } + node = child; + } + node.replacement ??= replacement; + } + return root; +} + +function applyTrie(text, trie) { + const out = []; + let index = 0; + while (index < text.length) { + let node = trie; + let cursor = index; + let replacement; + let replacementLength = 0; + while (cursor < text.length) { + node = node.children.get(text[cursor]); + if (!node) break; + cursor++; + if (node.replacement !== undefined) { + replacement = node.replacement; + replacementLength = cursor - index; + } + } + if (replacement !== undefined) { + out.push(replacement); + index += replacementLength; + } else { + out.push(text[index]); + index++; + } + } + return out.join(""); +} + +const v1EncodeTrie = buildTrie(v1Pairs); +const overlayEncodeTrie = buildTrie(overlayPairs); + +function brotliZlib(input) { + return brotliCompressSync(Buffer.from(input), { + params: { [constants.BROTLI_PARAM_QUALITY]: 11 }, + }); +} + +function hasBrotliCli() { + const result = spawnSync("brotli", ["--version"], { encoding: "utf8" }); + return result.status === 0; +} + +const BROTLI_CLI = hasBrotliCli(); +const TMP_ROOT = mkdtempSync(join(tmpdir(), "arx4-bet2-")); +const DICT_PATH = join(TMP_ROOT, "shared.dict"); + +function writeVarint(n) { + const bytes = []; + let v = n >>> 0; + while (v >= 0x80) { + bytes.push((v & 0x7f) | 0x80); + v >>>= 7; + } + bytes.push(v); + return Buffer.from(bytes); +} + +/** Mirrors src/lib/payload/arx4-content-first.ts wire format. */ +function encodeContentFirst(envelope) { + const artifact = envelope.artifacts[0]; + if (!artifact || envelope.artifacts.length !== 1) return null; + if (!("content" in artifact) || typeof artifact.content !== "string") return null; + const kindMap = { markdown: 1, code: 2, csv: 3, json: 4 }; + const kind = kindMap[artifact.kind]; + if (!kind) return null; + + const id = Buffer.from(artifact.id ?? "a", "utf8"); + const content = Buffer.from(artifact.content, "utf8"); + const meta = {}; + if (artifact.title) meta.t = artifact.title; + if (artifact.filename) meta.f = artifact.filename; + if (artifact.language) meta.l = artifact.language; + if (envelope.title && envelope.title !== artifact.title) meta.e = envelope.title; + const metaBuf = Buffer.from(Object.keys(meta).length ? JSON.stringify(meta) : "", "utf8"); + + return Buffer.concat([ + Buffer.from("A4"), + Buffer.from([1, kind, id.length]), + id, + writeVarint(content.length), + content, + writeVarint(metaBuf.length), + metaBuf, + ]); +} + +function trimOptional(fields) { + let end = fields.length; + while (end > 0 && (fields[end - 1] === undefined || fields[end - 1] === null)) end -= 1; + return fields.slice(0, end); +} + +function artifactTuple(artifact) { + switch (artifact.kind) { + case "markdown": + return trimOptional(["m", artifact.id, artifact.content, artifact.title, artifact.filename]); + case "code": + return trimOptional(["c", artifact.id, artifact.content, artifact.language, artifact.title, artifact.filename]); + case "csv": + return trimOptional(["s", artifact.id, artifact.content, artifact.title, artifact.filename]); + case "json": + return trimOptional(["j", artifact.id, artifact.content, artifact.title, artifact.filename]); + case "diff": + return trimOptional([ + "d", + artifact.id, + artifact.patch, + artifact.oldContent, + artifact.newContent, + artifact.language, + artifact.view, + artifact.title, + artifact.filename, + ]); + default: + throw new Error(`unsupported kind ${artifact.kind}`); + } +} + +function tupleEnvelope(envelope) { + const artifacts = envelope.artifacts.map(artifactTuple); + const activeIndex = Math.max( + 0, + envelope.artifacts.findIndex((a) => a.id === envelope.activeArtifactId), + ); + if (artifacts.length === 1) return trimOptional([3, artifacts[0], envelope.title]); + return trimOptional([2, artifacts, envelope.title, activeIndex > 0 ? activeIndex : undefined]); +} + +function encodeCborish(value) { + const chunks = []; + function pushUint(major, n) { + if (n < 24) chunks.push(Buffer.from([(major << 5) | n])); + else if (n < 256) chunks.push(Buffer.from([(major << 5) | 24, n])); + else if (n < 65536) chunks.push(Buffer.from([(major << 5) | 25, (n >> 8) & 0xff, n & 0xff])); + else + chunks.push( + Buffer.from([ + (major << 5) | 26, + (n >>> 24) & 0xff, + (n >>> 16) & 0xff, + (n >>> 8) & 0xff, + n & 0xff, + ]), + ); + } + function encode(v) { + if (v === null || v === undefined) { + chunks.push(Buffer.from([0xf6])); + return; + } + if (typeof v === "number" && Number.isInteger(v) && v >= 0) { + pushUint(0, v); + return; + } + if (typeof v === "string") { + const bytes = Buffer.from(v, "utf8"); + pushUint(3, bytes.length); + chunks.push(bytes); + return; + } + if (Array.isArray(v)) { + pushUint(4, v.length); + for (const item of v) encode(item); + return; + } + throw new Error(`Unsupported CBOR-ish value: ${typeof v}`); + } + encode(value); + return Buffer.concat(chunks); +} + +function encodeArx3Substituted(envelope) { + const tupleJson = JSON.stringify(tupleEnvelope({ ...envelope, codec: "arx3" })); + return applyTrie(applyTrie(tupleJson, overlayEncodeTrie), v1EncodeTrie); +} + +function baseBmpChars(byteLength) { + return 3 + Math.ceil((byteLength * 8) / Math.log2(BMP_BASE_SIZE)); +} + +function discordBudget({ host = "https://agent-render.com", label = "Artifact", tag = "c" } = {}) { + const framing = `[${label}](${host}#${tag}`; + const closing = ")"; + return { + framing, + framingLength: framing.length + closing.length, + payloadBudget: DISCORD_MESSAGE_MAX - framing.length - closing.length, + }; +} + +function textEnvelope(kind, title, content, extra = {}) { + return { + v: 1, + codec: "plain", + title, + activeArtifactId: "a", + artifacts: [{ id: "a", kind, title, filename: extra.filename ?? "artifact.txt", content, ...extra }], + }; +} + +function repeatedFixture(block, targetLength, segmentSuffix = (index) => `\nfixture segment ${index}\n`) { + let fixture = ""; + let index = 0; + while (fixture.length < targetLength) { + fixture += `${block}${segmentSuffix(index)}`; + index++; + } + return Array.from(fixture).slice(0, targetLength).join(""); +} + +const markdownAgentsFixture = repeatedFixture( + [ + "# AGENTS.md excerpt", + "", + "`agent-render` is a static artifact viewer for AI-generated outputs.", + "Keep markdown, code, diffs, CSV, and JSON readable across chat surfaces.", + "", + "## Product contract", + "", + "- Fragment payloads use `#agent-render=v1..`.", + "- Artifact contents stay out of the host request path.", + "- Supported codecs are `plain`, `lz`, `deflate`, `arx`, `arx2`, and `arx3`.", + "- Supported artifact kinds are `markdown`, `code`, `diff`, `csv`, and `json`.", + "", + "Preserve the static shell, the zero-retention wording, and the renderer-first layout.", + "", + ].join("\n"), + 8000, + (index) => + `\nFixture note ${index}: fragment transport, renderer readiness, and artifact metadata stay aligned.\n\n`, +); + +const codeFragmentFixture = repeatedFixture( + [ + "export async function decodeFragmentAsync(hash: string, options?: DecodeOptions) {", + " const parsed = parseFragmentPrefix(hash);", + " if (!parsed.ok) return parsed;", + ' if (parsed.codec === "arx" || parsed.codec === "arx2") {', + ' const { decodeArxFragmentAsync } = await import("./fragment-arx");', + " return decodeArxFragmentAsync(parsed, options);", + " }", + " return decodePlainFragment(parsed.payload, options);", + "}", + "", + ].join("\n"), + 8000, + (index) => `\n// fixture segment ${index}: codec branch coverage and bundle shape stay stable.\n`, +); + +const packageManifestFixture = JSON.stringify( + { + name: "agent-render", + version: "0.1.0", + private: true, + scripts: { build: "next build", check: "npm run lint && npm run test" }, + dependencies: { next: "15.1.11", react: "19.1.0", "brotli-wasm": "^3.0.1" }, + }, + null, + 2, +); + +const corpus = [ + { + name: "markdown-agents", + envelope: textEnvelope("markdown", "AGENTS.md excerpt", markdownAgentsFixture, { filename: "AGENTS.md" }), + }, + { + name: "code-bench-report", + envelope: textEnvelope("markdown", "Baanish Code Bench", codeBenchReportFixture, { filename: "results.md" }), + }, + { + name: "code-fragment", + envelope: textEnvelope("code", "fragment.ts excerpt", codeFragmentFixture, { + filename: "fragment.ts", + language: "ts", + }), + }, + { + name: "json-package", + envelope: textEnvelope("json", "package.json", packageManifestFixture, { filename: "package.json" }), + }, + { + name: "small-markdown", + envelope: textEnvelope( + "markdown", + "Note", + [ + "# Sprint notes", + "", + "- Ship ARX3 visible URL mode", + "- Keep Discord markdown links under 2000 chars", + "- Prefer fragment transport over UUID mode for zero-retention", + "", + "```ts", + "export const value = 1;", + "```", + "", + ].join("\n"), + { filename: "notes.md" }, + ), + }, +]; + +/** Domain shared dictionary: ARX substitution slots + common scaffolding. */ +const sharedDictBuffer = Buffer.from( + [ + ...v1Dictionary.singleByteSlots, + ...v1Dictionary.extendedSlots.slice(0, 40), + ...overlayDictionary.singleByteSlots, + ...overlayDictionary.extendedSlots, + '["m","', + '["c","', + "[3,", + "agent-render", + "export function ", + "export const ", + "import { ", + '} from "', + "\n## ", + "\n### ", + "\n- ", + "\n\n", + "```", + ].join("\n"), + "utf8", +); +writeFileSync(DICT_PATH, sharedDictBuffer); + +function brotliCli(data, { dictionaryPath = null } = {}) { + const inputPath = join(TMP_ROOT, `in-${process.hrtime.bigint()}.bin`); + writeFileSync(inputPath, data); + const args = ["-q", "11", "-c"]; + if (dictionaryPath) args.push("-D", dictionaryPath); + args.push(inputPath); + const result = spawnSync("brotli", args, { maxBuffer: 32 * 1024 * 1024 }); + try { + rmSync(inputPath, { force: true }); + } catch { + /* ignore */ + } + if (result.status !== 0) { + throw new Error(`brotli CLI failed: ${result.stderr?.toString() || result.error}`); + } + return Buffer.from(result.stdout); +} + +function residualEstimate(data, dictionary) { + const dictOnly = brotliZlib(dictionary).length; + const combo = brotliZlib(Buffer.concat([Buffer.from(dictionary), Buffer.from(data)])).length; + return Math.max(0, combo - dictOnly); +} + +function packVariant(brotliBytes) { + return { + brotliBytes, + bmpChars: baseBmpChars(brotliBytes), + }; +} + +function measureRow(envelope) { + const substituted = encodeArx3Substituted(envelope); + const arx3Bytes = brotliZlib(substituted); + const tuple = tupleEnvelope({ ...envelope, codec: "arx3" }); + const cborRaw = encodeCborish(tuple); + const contentFirst = encodeContentFirst(envelope); + + const cborBrotli = brotliZlib(cborRaw); + const contentBrotli = contentFirst ? brotliZlib(contentFirst) : null; + + // Content-first with ARX v1 substitution applied to the artifact body first + // (keeps the binary envelope but restores the text-dict win ARX3 already has). + let contentSubRaw = null; + let contentSubBrotli = null; + if (contentFirst) { + const artifact = envelope.artifacts[0]; + const subContent = applyTrie(artifact.content, v1EncodeTrie); + contentSubRaw = encodeContentFirst({ + ...envelope, + artifacts: [{ ...artifact, content: subContent }], + }); + contentSubBrotli = brotliZlib(contentSubRaw); + } + + // CBOR of the tuple *after* the same overlay+v1 substitution ARX3 uses on JSON — + // approximate by CBOR-encoding the substituted JSON string as a single text item is + // wrong; instead substitute inside tuple string fields then CBOR-encode. + const substitutedTuple = (() => { + const tuple = tupleEnvelope({ ...envelope, codec: "arx3" }); + function subDeep(value) { + if (typeof value === "string") return applyTrie(applyTrie(value, overlayEncodeTrie), v1EncodeTrie); + if (Array.isArray(value)) return value.map(subDeep); + return value; + } + return subDeep(tuple); + })(); + const cborSubRaw = encodeCborish(substitutedTuple); + const cborSubBrotli = brotliZlib(cborSubRaw); + + const arx3Residual = residualEstimate(Buffer.from(substituted, "utf8"), sharedDictBuffer); + const cborResidual = residualEstimate(cborRaw, sharedDictBuffer); + const contentResidual = contentFirst ? residualEstimate(contentFirst, sharedDictBuffer) : null; + + const deflateDict = deflateRawSync(Buffer.from(substituted, "utf8"), { + level: 9, + dictionary: sharedDictBuffer, + }).length; + + let arx3RealDict = null; + let cborRealDict = null; + let contentRealDict = null; + let contentSubRealDict = null; + let cborSubRealDict = null; + let bet2Best = null; + + if (BROTLI_CLI) { + arx3RealDict = brotliCli(Buffer.from(substituted, "utf8"), { dictionaryPath: DICT_PATH }).length; + cborRealDict = brotliCli(cborRaw, { dictionaryPath: DICT_PATH }).length; + cborSubRealDict = brotliCli(cborSubRaw, { dictionaryPath: DICT_PATH }).length; + if (contentFirst) { + contentRealDict = brotliCli(contentFirst, { dictionaryPath: DICT_PATH }).length; + contentSubRealDict = brotliCli(contentSubRaw, { dictionaryPath: DICT_PATH }).length; + } + // Best real bet-#2 candidate among binary envelopes + real −D. + bet2Best = Math.min( + ...[cborRealDict, cborSubRealDict, contentRealDict, contentSubRealDict].filter((n) => n != null), + ); + } + + return { + rawContentChars: envelope.artifacts.reduce((n, a) => n + (a.content?.length ?? a.patch?.length ?? 0), 0), + rawEnvelopeBytes: { + arx3Substituted: Buffer.byteLength(substituted, "utf8"), + cbor: cborRaw.length, + cborSubstituted: cborSubRaw.length, + contentFirst: contentFirst?.length ?? null, + contentFirstSubstituted: contentSubRaw?.length ?? null, + }, + arx3: packVariant(arx3Bytes.length), + arx3ResidualDict: packVariant(arx3Residual), + arx3RealDict: arx3RealDict != null ? packVariant(arx3RealDict) : null, + arx3DeflateDict: packVariant(deflateDict), + cbor: packVariant(cborBrotli.length), + cborResidualDict: packVariant(cborResidual), + cborRealDict: cborRealDict != null ? packVariant(cborRealDict) : null, + cborSub: packVariant(cborSubBrotli.length), + cborSubRealDict: cborSubRealDict != null ? packVariant(cborSubRealDict) : null, + contentFirst: contentBrotli ? packVariant(contentBrotli.length) : null, + contentFirstResidualDict: contentResidual != null ? packVariant(contentResidual) : null, + contentFirstRealDict: contentRealDict != null ? packVariant(contentRealDict) : null, + contentFirstSub: contentSubBrotli ? packVariant(contentSubBrotli.length) : null, + contentFirstSubRealDict: contentSubRealDict != null ? packVariant(contentSubRealDict) : null, + bet2Stack: bet2Best != null ? packVariant(bet2Best) : null, + }; +} + +function pct(from, to) { + if (from == null || to == null) return null; + return ((to - from) / from) * 100; +} + +function fmtDelta(from, to) { + const p = pct(from, to); + if (p == null || Number.isNaN(p)) return "n/a"; + const sign = p > 0 ? "+" : p < 0 ? "−" : ""; + // Display negative (smaller) as win with −, positive (larger) as + + // pct = (to-from)/from*100; smaller to → negative → win shown as − + if (p === 0) return "0.0%"; + return `${p < 0 ? "−" : "+"}${Math.abs(p).toFixed(1)}%`; +} + +function cell(bytes, baseline) { + if (bytes == null) return "—"; + return `${bytes} (${fmtDelta(baseline, bytes)})`; +} + +const start = performance.now(); +const rows = corpus.map((item) => ({ name: item.name, ...measureRow(item.envelope) })); +const budget = discordBudget(); + +const variants = [ + ["arx3", "ARX3 (baseline)"], + ["arx3ResidualDict", "ARX3 + residual dict est."], + ["arx3RealDict", "ARX3 + real Brotli −D"], + ["arx3DeflateDict", "ARX3 deflate+dict (non-goal)"], + ["cbor", "CBOR tuple + Brotli"], + ["cborSub", "CBOR + text-sub + Brotli"], + ["cborRealDict", "CBOR + real Brotli −D"], + ["cborSubRealDict", "CBOR + text-sub + real −D"], + ["contentFirst", "Content-first + Brotli"], + ["contentFirstSub", "Content-first + text-sub + Brotli"], + ["contentFirstRealDict", "Content-first + real −D"], + ["contentFirstSubRealDict", "Content-first + text-sub + real −D"], + ["bet2Stack", "Best real bet #2 candidate"], +]; + +const lines = []; +function w(line = "") { + lines.push(line); +} + +w("# ARX4 bet #2 — content-first / CBOR + real Brotli shared dictionary"); +w(); +w("_Experimental bench from `scripts/bench-arx4-bet2.mjs`. Not a shipped codec._"); +w(); +w("## What was implemented"); +w(); +w("1. **Content-first binary envelope** — `src/lib/payload/arx4-content-first.ts`"); +w(" - Wire: `A4 | version | kind | id | content | meta`"); +w(" - Round-trip tested; stamps rebuilt envelopes as `codec: \"plain\"` (ARX4 is not shipped)."); +w("2. **CBOR-ish tuple encoder** — same module; encodes the ARX2/3 tuple without JSON quotes."); +w("3. **Real Brotli shared dictionary** — measured via system `brotli -D` (LZ77 raw dictionary)."); +w(" - Node `zlib.brotliCompressSync({ dictionary })` **silently ignores** the option on Node 22."); +w(" - Product `brotli-wasm@3` has **no** custom-dictionary API."); +w(" - Residual `brotli(dict‖data)−brotli(dict)` is kept only as a calibration column."); +w(); +w("## Environment"); +w(); +w(`- Node: \`${process.version}\``); +w(`- Brotli CLI available: **${BROTLI_CLI ? "yes" : "no — real −D columns are empty"}**`); +w(`- Shared dictionary size: **${sharedDictBuffer.length}** bytes (ARX slot text + scaffolding)`); +w(`- Discord framing budget (current host): **${budget.payloadBudget}** payload chars`); +w(); +w("## Pre-compress sizes (envelope bytes before Brotli)"); +w(); +w("| Fixture | ARX3 substituted | CBOR | CBOR+text-sub | Content-first | CF+text-sub |"); +w("| --- | ---: | ---: | ---: | ---: | ---: |"); +for (const row of rows) { + w( + `| ${row.name} | ${row.rawEnvelopeBytes.arx3Substituted} | ${row.rawEnvelopeBytes.cbor} | ${row.rawEnvelopeBytes.cborSubstituted} | ${row.rawEnvelopeBytes.contentFirst ?? "—"} | ${row.rawEnvelopeBytes.contentFirstSubstituted ?? "—"} |`, + ); +} +w(); +w("## Brotli bytes (q11) vs ARX3"); +w(); +w( + `| Fixture | raw chars | ${variants.map(([, label]) => label).join(" | ")} |`, +); +w(`| --- | ---: | ${variants.map(() => "---:").join(" | ")} |`); +for (const row of rows) { + const baseline = row.arx3.brotliBytes; + const cells = variants.map(([key]) => { + const pack = row[key]; + return cell(pack?.brotliBytes ?? null, baseline); + }); + w(`| ${row.name} | ${row.rawContentChars.toLocaleString("en-US")} | ${cells.join(" | ")} |`); +} +w(); +w("## Visible chars @ baseBMP vs ARX3"); +w(); +w( + `| Fixture | ${variants.map(([, label]) => label).join(" | ")} | fits Discord? |`, +); +w(`| --- | ${variants.map(() => "---:").join(" | ")} | :---: |`); +for (const row of rows) { + const baseline = row.arx3.bmpChars; + const cells = variants.map(([key]) => { + const pack = row[key]; + if (!pack) return "—"; + return `${pack.bmpChars} (${fmtDelta(baseline, pack.bmpChars)})`; + }); + const bestReal = [ + row.bet2Stack, + row.contentFirstSubRealDict, + row.cborSubRealDict, + row.contentFirstSub, + row.cborSub, + row.cbor, + row.arx3RealDict, + ] + .filter(Boolean) + .sort((a, b) => a.bmpChars - b.bmpChars)[0]; + const fits = bestReal && bestReal.bmpChars <= budget.payloadBudget ? "yes" : "check"; + w(`| ${row.name} | ${cells.join(" | ")} | ${fits} |`); +} +w(); + +function sumKey(key) { + let total = 0; + let count = 0; + for (const row of rows) { + const pack = row[key]; + if (pack?.brotliBytes != null) { + total += pack.brotliBytes; + count += 1; + } + } + return count === rows.length ? total : null; +} + +function sumBmp(key) { + let total = 0; + let count = 0; + for (const row of rows) { + const pack = row[key]; + if (pack?.bmpChars != null) { + total += pack.bmpChars; + count += 1; + } + } + return count === rows.length ? total : null; +} + +const arx3Sum = sumKey("arx3"); +const arx3Bmp = sumBmp("arx3"); + +w("## Totals (all fixtures)"); +w(); +w("| Variant | Σ brotli | vs ARX3 | Σ BMP chars | vs ARX3 BMP |"); +w("| --- | ---: | ---: | ---: | ---: |"); +for (const [key, label] of variants) { + const bytes = sumKey(key); + const bmp = sumBmp(key); + if (bytes == null) { + w(`| ${label} | — | n/a | — | n/a |`); + continue; + } + w(`| ${label} | ${bytes} | ${fmtDelta(arx3Sum, bytes)} | ${bmp} | ${fmtDelta(arx3Bmp, bmp)} |`); +} +w(); +w("## Findings"); +w(); +w("### Content-first / CBOR alone"); +w(); +w("- Dropping JSON (CBOR) or skipping the tuple (content-first) changes pre-Brotli size, but"); +w(" **after Brotli q11 the win vs ARX3 is small** — often within ~1%, and content-first alone"); +w(" can *lose* on small fixtures because it skips ARX text substitution."); +w("- Re-applying v1 text substitution *inside* content-first / CBOR fields recovers most of"); +w(" that gap; see the `+ text-sub` columns."); +w(); +w("### Real Brotli shared dictionary (`brotli -D`)"); +w(); +w("- Unlike the residual estimate (often −10% to −30%), **real LZ77 shared dictionaries are"); +w(" modest** on this corpus (typically under ~1% total, fixture-dependent)."); +w("- Residual estimates systematically **overstate** the win; do not use them as a ship gate."); +w("- Deflate+dictionary remains a **non-goal**: larger than plain Brotli on this corpus."); +w(); +w("### Product implications"); +w(); +w("1. **Browser path blocked for real shared dicts today** — `brotli-wasm` has no dictionary"); +w(" API; Node zlib ignores `dictionary`. Shipping ARX4 shared-dict needs a wasm fork or"); +w(" alternate compressor with custom-dict support."); +w("2. **Binary envelopes are still useful plumbing** (no JSON escaping, cleaner wire) but are"); +w(" not a Discord capacity unlock by themselves on this corpus."); +w("3. Prefer measuring with **real `brotli -D`** (or a dict-capable wasm) over residual proxies"); +w(" before committing to a shared-dictionary protocol."); +w("4. Next exploration should focus on **dict contents matched to the post-substitution byte"); +w(" stream** (or a wasm dict path), not more residual optimism."); +w(); +w("## How to re-run"); +w(); +w("```bash"); +w("# requires system `brotli` CLI for real −D columns (apt install brotli)"); +w("npm run bench:arx4-bet2"); +w("# or: node scripts/bench-arx4-bet2.mjs"); +w("```"); +w(); +w(`_Generated in ${(performance.now() - start).toFixed(1)}ms._`); + +writeFileSync(REPORT_PATH, `${lines.join("\n")}\n`); +console.log(lines.join("\n")); + +try { + rmSync(TMP_ROOT, { recursive: true, force: true }); +} catch { + /* ignore */ +} diff --git a/src/lib/payload/arx4-content-first.ts b/src/lib/payload/arx4-content-first.ts new file mode 100644 index 0000000..c8236cc --- /dev/null +++ b/src/lib/payload/arx4-content-first.ts @@ -0,0 +1,460 @@ +/** + * ARX4 experimental content-first / CBOR envelope helpers. + * + * Not a shipped codec. These packers exist so ideation benches and unit tests can + * measure binary envelopes against today's ARX3 JSON-tuple path without changing + * the fragment protocol surface (`plain|lz|deflate|arx|arx2|arx3`). + * + * Content-first (single text artifact): + * magic "A4" | version(1) | kind(1) | idLen(1) | id | contentLen(varint) | content + * | metaLen(varint) | metaJson(utf8) + * + * CBOR-ish: minimal CBOR major types 0/2/3/4/7 for the existing ARX2/3 tuple shape + * (non-negative ints, byte/text strings, arrays, null). + */ + +import type { + Arx2ArtifactTuple, + Arx2EnvelopeTuple, + ArtifactPayload, + CodeArtifact, + CsvArtifact, + JsonArtifact, + MarkdownArtifact, + PayloadEnvelope, +} from "@/lib/payload/schema"; + +/** Magic bytes identifying an ARX4 content-first binary envelope. */ +export const ARX4_CONTENT_FIRST_MAGIC = "A4"; + +/** Current content-first binary format version. */ +export const ARX4_CONTENT_FIRST_VERSION = 1; + +const KIND_TO_CODE = { + markdown: 1, + code: 2, + csv: 3, + json: 4, +} as const; + +const CODE_TO_KIND = { + 1: "markdown", + 2: "code", + 3: "csv", + 4: "json", +} as const; + +type ContentFirstKind = keyof typeof KIND_TO_CODE; +type ContentFirstArtifact = MarkdownArtifact | CodeArtifact | CsvArtifact | JsonArtifact; + +type ContentFirstMeta = { + t?: string; + f?: string; + l?: string; + e?: string; +}; + +function isContentFirstArtifact(artifact: ArtifactPayload): artifact is ContentFirstArtifact { + return artifact.kind === "markdown" || artifact.kind === "code" || artifact.kind === "csv" || artifact.kind === "json"; +} + +function writeVarint(n: number): Uint8Array { + if (!Number.isInteger(n) || n < 0 || n > 0xffff_ffff) { + throw new Error("ARX4 content-first varint out of range."); + } + const bytes: number[] = []; + let v = n >>> 0; + while (v >= 0x80) { + bytes.push((v & 0x7f) | 0x80); + v >>>= 7; + } + bytes.push(v); + return Uint8Array.from(bytes); +} + +function readVarint(bytes: Uint8Array, offset: number): { value: number; next: number } { + let value = 0; + let shift = 0; + let cursor = offset; + while (cursor < bytes.length) { + const byte = bytes[cursor]!; + cursor += 1; + value |= (byte & 0x7f) << shift; + if ((byte & 0x80) === 0) { + return { value: value >>> 0, next: cursor }; + } + shift += 7; + if (shift > 28) { + throw new Error("ARX4 content-first varint too long."); + } + } + throw new Error("ARX4 content-first truncated varint."); +} + +function concatBytes(chunks: Uint8Array[]): Uint8Array { + let total = 0; + for (const chunk of chunks) total += chunk.length; + const out = new Uint8Array(total); + let offset = 0; + for (const chunk of chunks) { + out.set(chunk, offset); + offset += chunk.length; + } + return out; +} + +function textEncoder(): TextEncoder { + return new TextEncoder(); +} + +function textDecoder(): TextDecoder { + return new TextDecoder("utf-8", { fatal: true }); +} + +function trimOptionalTuple(fields: T): T { + let end = fields.length; + while (end > 0) { + const value = fields[end - 1]; + if (value !== undefined && value !== null) break; + end -= 1; + } + if (end === fields.length) return fields; + return fields.slice(0, end) as T; +} + +/** + * Returns true when the envelope is a single markdown/code/csv/json artifact + * (the Discord share sweet spot for content-first packing). + */ +export function canEncodeArx4ContentFirst(envelope: PayloadEnvelope): boolean { + if (!Array.isArray(envelope.artifacts) || envelope.artifacts.length !== 1) return false; + const artifact = envelope.artifacts[0]; + return artifact !== undefined && isContentFirstArtifact(artifact); +} + +/** + * Encodes a single text artifact as a content-first binary envelope. + * Throws when the envelope is not a supported single-artifact shape. + */ +export function encodeArx4ContentFirst(envelope: PayloadEnvelope): Uint8Array { + if (!canEncodeArx4ContentFirst(envelope)) { + throw new Error("ARX4 content-first requires a single markdown/code/csv/json artifact."); + } + + const artifact = envelope.artifacts[0] as ContentFirstArtifact; + const kindCode = KIND_TO_CODE[artifact.kind as ContentFirstKind]; + const encoder = textEncoder(); + const idBytes = encoder.encode(artifact.id); + if (idBytes.length > 255) { + throw new Error("ARX4 content-first id exceeds 255 bytes."); + } + + const contentBytes = encoder.encode(artifact.content); + const meta: ContentFirstMeta = {}; + if (artifact.title) meta.t = artifact.title; + if (artifact.filename) meta.f = artifact.filename; + if (artifact.kind === "code" && artifact.language) meta.l = artifact.language; + if (envelope.title && envelope.title !== artifact.title) meta.e = envelope.title; + const metaBytes = encoder.encode(Object.keys(meta).length > 0 ? JSON.stringify(meta) : ""); + + return concatBytes([ + encoder.encode(ARX4_CONTENT_FIRST_MAGIC), + Uint8Array.of(ARX4_CONTENT_FIRST_VERSION, kindCode, idBytes.length), + idBytes, + writeVarint(contentBytes.length), + contentBytes, + writeVarint(metaBytes.length), + metaBytes, + ]); +} + +/** + * Decodes a content-first binary envelope back into a standard payload envelope. + * The rebuilt envelope stamps `codec: "plain"` because ARX4 is not a shipped codec. + */ +export function decodeArx4ContentFirst(bytes: Uint8Array): PayloadEnvelope { + if (!(bytes instanceof Uint8Array)) { + throw new Error("ARX4 content-first decode expects Uint8Array."); + } + if (bytes.length < 5) { + throw new Error("ARX4 content-first payload too short."); + } + + const decoder = textDecoder(); + const magic = decoder.decode(bytes.subarray(0, 2)); + if (magic !== ARX4_CONTENT_FIRST_MAGIC) { + throw new Error("ARX4 content-first magic mismatch."); + } + + const version = bytes[2]!; + if (version !== ARX4_CONTENT_FIRST_VERSION) { + throw new Error(`Unsupported ARX4 content-first version: ${version}.`); + } + + const kindCode = bytes[3]! as keyof typeof CODE_TO_KIND; + const kind = CODE_TO_KIND[kindCode]; + if (!kind) { + throw new Error(`Unknown ARX4 content-first kind code: ${kindCode}.`); + } + + const idLen = bytes[4]!; + let cursor = 5; + if (cursor + idLen > bytes.length) { + throw new Error("ARX4 content-first truncated id."); + } + const id = decoder.decode(bytes.subarray(cursor, cursor + idLen)); + cursor += idLen; + + const contentLen = readVarint(bytes, cursor); + cursor = contentLen.next; + if (cursor + contentLen.value > bytes.length) { + throw new Error("ARX4 content-first truncated content."); + } + const content = decoder.decode(bytes.subarray(cursor, cursor + contentLen.value)); + cursor += contentLen.value; + + const metaLen = readVarint(bytes, cursor); + cursor = metaLen.next; + if (cursor + metaLen.value > bytes.length) { + throw new Error("ARX4 content-first truncated meta."); + } + if (cursor + metaLen.value !== bytes.length) { + throw new Error("ARX4 content-first trailing bytes."); + } + + let meta: ContentFirstMeta = {}; + if (metaLen.value > 0) { + const parsed: unknown = JSON.parse(decoder.decode(bytes.subarray(cursor, cursor + metaLen.value))); + if (typeof parsed !== "object" || parsed === null || Array.isArray(parsed)) { + throw new Error("ARX4 content-first meta must be a JSON object."); + } + meta = parsed as ContentFirstMeta; + } + + const base = { + id, + title: typeof meta.t === "string" ? meta.t : undefined, + filename: typeof meta.f === "string" ? meta.f : undefined, + }; + + let artifact: ContentFirstArtifact; + switch (kind) { + case "markdown": + artifact = { ...base, kind: "markdown", content }; + break; + case "code": + artifact = { + ...base, + kind: "code", + content, + language: typeof meta.l === "string" ? meta.l : undefined, + }; + break; + case "csv": + artifact = { ...base, kind: "csv", content }; + break; + case "json": + artifact = { ...base, kind: "json", content }; + break; + default: { + const _exhaustive: never = kind; + throw new Error(`Unhandled ARX4 content-first kind: ${_exhaustive}`); + } + } + + return { + v: 1, + codec: "plain", + title: typeof meta.e === "string" ? meta.e : artifact.title, + activeArtifactId: artifact.id, + artifacts: [artifact], + }; +} + +function artifactToArx2Tuple(artifact: ArtifactPayload): Arx2ArtifactTuple { + switch (artifact.kind) { + case "markdown": + return trimOptionalTuple(["m", artifact.id, artifact.content, artifact.title, artifact.filename]); + case "code": + return trimOptionalTuple(["c", artifact.id, artifact.content, artifact.language, artifact.title, artifact.filename]); + case "diff": + return trimOptionalTuple([ + "d", + artifact.id, + artifact.patch, + artifact.oldContent, + artifact.newContent, + artifact.language, + artifact.view, + artifact.title, + artifact.filename, + ]); + case "csv": + return trimOptionalTuple(["s", artifact.id, artifact.content, artifact.title, artifact.filename]); + case "json": + return trimOptionalTuple(["j", artifact.id, artifact.content, artifact.title, artifact.filename]); + default: { + const _exhaustive: never = artifact; + throw new Error(`Unhandled artifact kind: ${JSON.stringify(_exhaustive)}`); + } + } +} + +/** + * Builds the ARX2/ARX3 tuple shape used by the CBOR-ish encoder (mirrors product tuple packing). + */ +export function envelopeToArx4Tuple(envelope: PayloadEnvelope): Arx2EnvelopeTuple { + if (!Array.isArray(envelope.artifacts) || envelope.artifacts.length < 1) { + throw new Error("ARX4 tuple encode requires at least one artifact."); + } + + const artifacts: Arx2ArtifactTuple[] = new Array(envelope.artifacts.length); + let activeIndex = -1; + + for (let index = 0; index < envelope.artifacts.length; index += 1) { + const artifact = envelope.artifacts[index]!; + artifacts[index] = artifactToArx2Tuple(artifact); + if (artifact.id === envelope.activeArtifactId) { + activeIndex = index; + } + } + + if (artifacts.length === 1) { + return trimOptionalTuple([3, artifacts[0]!, envelope.title]); + } + + return trimOptionalTuple([2, artifacts, envelope.title, activeIndex > 0 ? activeIndex : undefined]); +} + +type CborishValue = null | number | string | CborishValue[]; + +function pushUintHeader(chunks: number[], major: number, n: number): void { + if (n < 24) { + chunks.push((major << 5) | n); + } else if (n < 256) { + chunks.push((major << 5) | 24, n); + } else if (n < 65536) { + chunks.push((major << 5) | 25, (n >> 8) & 0xff, n & 0xff); + } else { + chunks.push( + (major << 5) | 26, + (n >>> 24) & 0xff, + (n >>> 16) & 0xff, + (n >>> 8) & 0xff, + n & 0xff, + ); + } +} + +function encodeCborishValue(value: CborishValue, chunks: number[], encoder: TextEncoder): void { + if (value === null) { + chunks.push(0xf6); + return; + } + if (typeof value === "number") { + if (!Number.isInteger(value) || value < 0) { + throw new Error("ARX4 CBOR-ish only supports non-negative integers."); + } + pushUintHeader(chunks, 0, value); + return; + } + if (typeof value === "string") { + const bytes = encoder.encode(value); + pushUintHeader(chunks, 3, bytes.length); + for (const byte of bytes) chunks.push(byte); + return; + } + if (Array.isArray(value)) { + pushUintHeader(chunks, 4, value.length); + for (const item of value) encodeCborishValue(item as CborishValue, chunks, encoder); + return; + } + throw new Error(`Unsupported ARX4 CBOR-ish value: ${typeof value}`); +} + +/** + * Encodes an ARX2/ARX3-compatible tuple as minimal CBOR (ints/strings/arrays/null). + */ +export function encodeArx4CborishTuple(tuple: Arx2EnvelopeTuple): Uint8Array { + const chunks: number[] = []; + encodeCborishValue(tuple as CborishValue, chunks, textEncoder()); + return Uint8Array.from(chunks); +} + +/** + * Encodes a payload envelope as CBOR-ish bytes of its ARX2/ARX3 tuple. + */ +export function encodeArx4CborishEnvelope(envelope: PayloadEnvelope): Uint8Array { + return encodeArx4CborishTuple(envelopeToArx4Tuple(envelope)); +} + +function readCborHeader(bytes: Uint8Array, offset: number): { major: number; value: number; next: number } { + if (offset >= bytes.length) { + throw new Error("ARX4 CBOR-ish truncated header."); + } + const initial = bytes[offset]!; + const major = initial >> 5; + const additional = initial & 0x1f; + let next = offset + 1; + let value = additional; + if (additional === 24) { + if (next >= bytes.length) throw new Error("ARX4 CBOR-ish truncated uint8."); + value = bytes[next]!; + next += 1; + } else if (additional === 25) { + if (next + 1 >= bytes.length) throw new Error("ARX4 CBOR-ish truncated uint16."); + value = (bytes[next]! << 8) | bytes[next + 1]!; + next += 2; + } else if (additional === 26) { + if (next + 3 >= bytes.length) throw new Error("ARX4 CBOR-ish truncated uint32."); + value = + ((bytes[next]! << 24) | (bytes[next + 1]! << 16) | (bytes[next + 2]! << 8) | bytes[next + 3]!) >>> 0; + next += 4; + } else if (additional >= 28) { + throw new Error(`Unsupported ARX4 CBOR-ish additional info: ${additional}.`); + } + return { major, value, next }; +} + +function decodeCborishValue(bytes: Uint8Array, offset: number): { value: CborishValue; next: number } { + const header = readCborHeader(bytes, offset); + if (header.major === 7 && header.value === 22) { + return { value: null, next: header.next }; + } + if (header.major === 0) { + return { value: header.value, next: header.next }; + } + if (header.major === 3) { + const end = header.next + header.value; + if (end > bytes.length) throw new Error("ARX4 CBOR-ish truncated string."); + return { value: textDecoder().decode(bytes.subarray(header.next, end)), next: end }; + } + if (header.major === 4) { + const items: CborishValue[] = []; + let cursor = header.next; + for (let i = 0; i < header.value; i += 1) { + const decoded = decodeCborishValue(bytes, cursor); + items.push(decoded.value); + cursor = decoded.next; + } + return { value: items, next: cursor }; + } + throw new Error(`Unsupported ARX4 CBOR-ish major type: ${header.major}.`); +} + +/** + * Decodes CBOR-ish bytes produced by {@link encodeArx4CborishTuple}. + */ +export function decodeArx4CborishTuple(bytes: Uint8Array): Arx2EnvelopeTuple { + if (!(bytes instanceof Uint8Array)) { + throw new Error("ARX4 CBOR-ish decode expects Uint8Array."); + } + const decoded = decodeCborishValue(bytes, 0); + if (decoded.next !== bytes.length) { + throw new Error("ARX4 CBOR-ish trailing bytes."); + } + if (!Array.isArray(decoded.value)) { + throw new Error("ARX4 CBOR-ish root must be an array."); + } + return decoded.value as Arx2EnvelopeTuple; +} diff --git a/tests/arx4-content-first.test.ts b/tests/arx4-content-first.test.ts new file mode 100644 index 0000000..d9f91c9 --- /dev/null +++ b/tests/arx4-content-first.test.ts @@ -0,0 +1,143 @@ +import { describe, expect, it } from "vitest"; +import { + ARX4_CONTENT_FIRST_MAGIC, + ARX4_CONTENT_FIRST_VERSION, + canEncodeArx4ContentFirst, + decodeArx4CborishTuple, + decodeArx4ContentFirst, + encodeArx4CborishEnvelope, + encodeArx4CborishTuple, + encodeArx4ContentFirst, + envelopeToArx4Tuple, +} from "@/lib/payload/arx4-content-first"; +import type { PayloadEnvelope } from "@/lib/payload/schema"; + +function markdownEnvelope(content: string, extra: Partial = {}): PayloadEnvelope { + return { + v: 1, + codec: "plain", + title: "Note", + activeArtifactId: "a", + artifacts: [ + { + id: "a", + kind: "markdown", + title: "Note", + filename: "notes.md", + content, + ...extra, + }, + ], + }; +} + +describe("arx4 content-first (experimental)", () => { + it("reports support only for single text artifacts", () => { + expect(canEncodeArx4ContentFirst(markdownEnvelope("hi"))).toBe(true); + expect( + canEncodeArx4ContentFirst({ + v: 1, + codec: "plain", + artifacts: [ + { id: "a", kind: "diff", patch: "--- a\n+++ b\n" }, + ], + }), + ).toBe(false); + expect( + canEncodeArx4ContentFirst({ + v: 1, + codec: "plain", + artifacts: [ + { id: "a", kind: "markdown", content: "one" }, + { id: "b", kind: "markdown", content: "two" }, + ], + }), + ).toBe(false); + }); + + it("round-trips markdown with meta", () => { + const envelope = markdownEnvelope("# Hello\n\n- item\n"); + const encoded = encodeArx4ContentFirst(envelope); + expect(encoded[0]).toBe(ARX4_CONTENT_FIRST_MAGIC.charCodeAt(0)); + expect(encoded[1]).toBe(ARX4_CONTENT_FIRST_MAGIC.charCodeAt(1)); + expect(encoded[2]).toBe(ARX4_CONTENT_FIRST_VERSION); + + const decoded = decodeArx4ContentFirst(encoded); + expect(decoded.codec).toBe("plain"); + expect(decoded.artifacts).toHaveLength(1); + expect(decoded.artifacts[0]).toMatchObject({ + id: "a", + kind: "markdown", + title: "Note", + filename: "notes.md", + content: "# Hello\n\n- item\n", + }); + expect(decoded.title).toBe("Note"); + }); + + it("round-trips code language and envelope title override", () => { + const envelope: PayloadEnvelope = { + v: 1, + codec: "plain", + title: "Bundle", + activeArtifactId: "src", + artifacts: [ + { + id: "src", + kind: "code", + title: "fragment.ts", + filename: "fragment.ts", + language: "ts", + content: "export const value = 1;\n", + }, + ], + }; + const decoded = decodeArx4ContentFirst(encodeArx4ContentFirst(envelope)); + expect(decoded.artifacts[0]).toMatchObject({ + kind: "code", + language: "ts", + content: "export const value = 1;\n", + }); + expect(decoded.title).toBe("Bundle"); + }); + + it("rejects truncated payloads", () => { + const encoded = encodeArx4ContentFirst(markdownEnvelope("x")); + expect(() => decodeArx4ContentFirst(encoded.subarray(0, 4))).toThrow(/too short|truncated/i); + }); +}); + +describe("arx4 CBOR-ish tuple (experimental)", () => { + it("round-trips a single-artifact tuple", () => { + const envelope = markdownEnvelope("body"); + const tuple = envelopeToArx4Tuple(envelope); + const encoded = encodeArx4CborishTuple(tuple); + const decoded = decodeArx4CborishTuple(encoded); + expect(decoded).toEqual(tuple); + }); + + it("encodes envelopes and stays smaller than JSON for quote-heavy content", () => { + const envelope = markdownEnvelope('He said "hello"\n\n```ts\nconst x = "y";\n```\n'); + const tuple = envelopeToArx4Tuple(envelope); + const json = JSON.stringify(tuple); + const cbor = encodeArx4CborishEnvelope(envelope); + expect(cbor.length).toBeLessThan(Buffer.byteLength(json, "utf8")); + expect(decodeArx4CborishTuple(cbor)).toEqual(tuple); + }); + + it("round-trips a multi-artifact bundle tuple", () => { + const envelope: PayloadEnvelope = { + v: 1, + codec: "plain", + title: "Bundle", + activeArtifactId: "b", + artifacts: [ + { id: "a", kind: "markdown", content: "one" }, + { id: "b", kind: "json", content: "{\"ok\":true}", title: "data" }, + ], + }; + const tuple = envelopeToArx4Tuple(envelope); + expect(tuple[0]).toBe(2); + expect(decodeArx4CborishTuple(encodeArx4CborishTuple(tuple))).toEqual(tuple); + }); +}); From 4911c299dfb10c8b013bad1307e5b6463471776d Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Wed, 8 Jul 2026 19:10:07 +0000 Subject: [PATCH 4/5] Probe ARX4 silly cuts: priors, IR, mosaic, and fence math. Measure unconventional Discord packing levers beyond envelope/Brotli tweaks. Leave-one-out priors regress vs ARX3; warm priors are fake. Mosaic across messages is the only real capacity multiplier. Co-authored-by: Aanish Bhirud --- docs/arx4-ideation.md | 56 +- docs/arx4-silly-cuts.md | 201 ++++++ package.json | 1 + scripts/bench-arx4-silly.mjs | 1161 ++++++++++++++++++++++++++++++++++ 4 files changed, 1401 insertions(+), 18 deletions(-) create mode 100644 docs/arx4-silly-cuts.md create mode 100644 scripts/bench-arx4-silly.mjs diff --git a/docs/arx4-ideation.md b/docs/arx4-ideation.md index cbd84b6..964720f 100644 --- a/docs/arx4-ideation.md +++ b/docs/arx4-ideation.md @@ -142,40 +142,57 @@ bytes via shared dictionaries / better envelopes — not denser Unicode wire or ### Ranked bets for ARX4 -1. **Content-first / CBOR + real Brotli shared dict — explored (see `docs/arx4-bet2-bench.md`)** +1. **Mosaic / multi-message assembler — top “break the box” bet (see `docs/arx4-silly-cuts.md`)** + - Only lever that *multiplies* Discord’s 2000 budget (N messages ≈ N× capacity). + - N links in *one* message do not help — they share the same 2000 chars. + - Protocol + UX cost (click/`1/3` assembly); not a denser alphabet. + +2. **Content-first / CBOR + real Brotli shared dict — explored (see `docs/arx4-bet2-bench.md`)** - Implemented: `src/lib/payload/arx4-content-first.ts` + `npm run bench:arx4-bet2`. - Residual dict estimates overstated wins (~6–8% corpus); **real `brotli -D` is ~0–1%**. - Binary envelopes alone are not a Discord unlock; keep as plumbing, not the next capacity bet. - Browser shared-dict still blocked (`brotli-wasm` has no dict API; Node zlib ignores it). -2. **Curated overlay growth (cautious)** +3. **Versioned shared prior (chunk / word) — only with LOO gate** + - Warm/contaminated priors look magical (−70%+) and are fake (content-addressed cache). + - Leave-one-out chunk priors **regressed** this corpus (~+20% BMP) — literals + ID framing lose to ARX3+Brotli. + - Revisit only for a *domain* prior that LOO-beats ARX3 on held-out agent chat; never trust warm numbers. + +4. **Implied envelope + kind IR + lossy — small honest cuts** + - Implied / kind IR ~3–4% corpus; lossy ~8% when stripping emphasis/comments. + - Plumbing / opt-in preview — not a Discord unlock alone. + +5. **Hybrid fence / label bitstream — wash** + - Fence stub steals budget; label-as-title costs framing. Skip as density bets; fence may still be a paste UX. + +6. **Curated overlay growth (cautious)** - Alone, mined n-grams *regressed* this corpus (+3–4%). - - Prefer a carefully curated ARX4 overlay over online mining; measure before shipping. -3. **baseAstral — deprioritized for Discord** - - Web evidence favors UTF-16 client counting → astral loses to baseBMP (~10 vs ~15.92 bits/unit). - - Optional one-shot paste test only; not a primary ARX4 bet. +7. **baseAstral — deprioritized for Discord** + - UTF-16 client counting → astral loses to baseBMP (~10 vs ~15.92 bits/unit). -4. **Discord framing — already practiced, not an ARX4 lever** - - Skill/agents already use short labels (`[Short summary](…)`); product warns on full `markdownLink.length`. - - Host shortening (`arx.page`) is deployment/DNS, not a codec change — drop from ARX4 scope. +8. **Discord framing — already practiced, not an ARX4 lever** + - Short labels are skill default; host shortening is DNS, not codec. ### Suggested ARX4 shape (if pursued) ```text +# Single-link path (marginal): artifact bytes - → content-first binary envelope (kind|id|content|meta) // or CBOR tuple for bundles - → optional curated ARX4 overlay (domain n-grams, measured) + → implied / content-first envelope + optional kind IR → Brotli q11 (+ real shared dict only if browser wasm gains dict support) - → baseBMP (Discord-safe; skip astral unless paste tests overturn UTF-16 finding) - → compact tag `d` / `e` + → baseBMP → compact tag + +# Break-the-box path (capacity): +artifact bytes → ARX3/ARX4 pack → split into N Discord messages (mosaic 1/N) + OR stub link + ```arx fence (paste UX, not denser wire) ``` -Selection policy: optimize `markdownLink.length` (JS/UTF-16 units, matching Discord client -folk counting) for a declared surface (`discord` | `visible` | `transport`). +Selection policy: optimize `markdownLink.length` (JS/UTF-16 units) for a declared surface +(`discord` | `discord-mosaic` | `discord-fence` | `visible` | `transport`). -Bet #2 takeaway: chase **dict-capable wasm** or a better post-substitution dictionary before -expecting Discord capacity gains from binary envelopes. +Silly-cuts takeaway: **stop chasing alphabet / residual-dict / contaminated priors.** +The real unlock is either a proven LOO prior or accepting multi-message / fence UX. ## Non-goals / traps @@ -194,7 +211,10 @@ npm run bench:arx4-ideation # Bet #2 follow-up (content-first/CBOR + real brotli -D): npm run bench:arx4-bet2 + +# Silly / deep cuts (priors, IR, mosaic, fence): +npm run bench:arx4-silly ``` _Generated in 9.6ms._ -See also `docs/arx4-bet2-bench.md` for the real shared-dictionary follow-up. +See also `docs/arx4-bet2-bench.md` and `docs/arx4-silly-cuts.md`. diff --git a/docs/arx4-silly-cuts.md b/docs/arx4-silly-cuts.md new file mode 100644 index 0000000..61f515a --- /dev/null +++ b/docs/arx4-silly-cuts.md @@ -0,0 +1,201 @@ +# ARX4 silly cuts — unconventional Discord packing + +_Experimental notes from `scripts/bench-arx4-silly.mjs`. Not a shipped codec._ + +## Why this pass + +Bet #2 (binary envelopes + real Brotli `-D`) topped out around **~0–2%** vs ARX3. +Wire density is already ~99.5% of the 16-bit UTF-16 ceiling. Alphabet and residual-dict +tweaks are exhausted. This pass looks for **deeper / sillier** levers: + +- shared priors the *viewer already knows* (chunks, word tables, templates) +- kind-specific IR and lossy readable-enough transforms +- Discord UX bends (mosaic links, fence hybrid, label-as-bitstream) +- implied envelopes that drop metadata from the fragment + +## Ideas measured + +| Cut | Idea | Lossless? | Product tension | +| --- | --- | :---: | --- | +| A | **Implied envelope** — 1-byte kind + raw content | yes | drops id/title/filename from fragment | +| B | **Kind IR** — md normalize / JSON key-dict / CSV columnar | mostly | kind-specific decoders | +| C | **Chunk prior** — CDC-ish windows → 2-byte IDs vs shared prior | yes* | prior must be pinned/versioned | +| D | **Template delta** — strip known skeleton, store residual+marks | yes* | skeleton catalog | +| E | **Lossy readable** — collapse ws, strip emphasis/comments | no | quality trade | +| F | **Label bitstream** — put title in Discord `[label]` | yes | label UX / length | +| G | **Mosaic** — N markdown links / N messages | yes | multi-click UX | +| H | **Hybrid fence** — stub URL + ` ```arx ` payload in same message | yes | not pure-link; scanners differ | +| I | **Word pack** — corpus BPE-ish token table → id stream | yes* | shared vocab | + +\* Lossless only if encoder and decoder share the same prior/vocab/skeleton version. + +Prior sizes this run: cold chunks **77**, warm chunks **7061**, cold words **15**, warm words **312**. +Leave-one-out (LOO) priors exclude the measured fixture — that is the honest “pinned mother dict” estimate. + +## Per-fixture Brotli bytes (vs ARX3) + +| Fixture | raw | ARX3 | implied | kind IR | chunk cold | chunk LOO | chunk warm† | word LOO | lossy | silly stack (IR+LOO) | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| markdown-agents | 8000 | 401 | 370 (−7.7%) | 371 (−7.5%) | 654 (+63.1%) | 654 (+63.1%) | 274 (−31.7%) | 461 (+15.0%) | 371 (−7.5%) | 657 (+63.8%) | +| code-bench-report | 8314 | 2219 | 2180 (−1.8%) | 2181 (−1.7%) | 2484 (+11.9%) | 2484 (+11.9%) | 319 (−85.6%) | 2667 (+20.2%) | 2143 (−3.4%) | 2474 (+11.5%) | +| code-fragment | 8000 | 308 | 292 (−5.2%) | 292 (−5.2%) | 516 (+67.5%) | 516 (+67.5%) | 284 (−7.8%) | 305 (−1.0%) | 195 (−36.7%) | 516 (+67.5%) | +| json-package | 259 | 182 | 162 (−11.0%) | 172 (−5.5%) | 181 (−0.5%) | 181 (−0.5%) | 25 (−86.3%) | 186 (+2.2%) | 162 (−11.0%) | 187 (+2.7%) | +| csv-leaderboard | 218 | 191 | 156 (−18.3%) | 174 (−8.9%) | 172 (−9.9%) | 149 (−22.0%) | 29 (−84.8%) | 173 (−9.4%) | 156 (−18.3%) | 151 (−20.9%) | +| small-markdown | 189 | 141 | 135 (−4.3%) | 135 (−4.3%) | 133 (−5.7%) | 136 (−3.5%) | 32 (−77.3%) | 153 (+8.5%) | 135 (−4.3%) | 136 (−3.5%) | + +† **chunk warm** includes the target fixture in the prior — contamination ceiling, not a real win. + +## Per-fixture visible BMP chars (vs ARX3) + +| Fixture | ARX3 BMP | implied | kind IR | chunk LOO | word LOO | lossy | best honest | Discord fit | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | :---: | +| markdown-agents | 205 | 189 (−7.8%) | 190 (−7.3%) | 332 (+62.0%) | 235 (+14.6%) | 190 (−7.3%) | 189 (−7.8%) | yes | +| code-bench-report | 1119 | 1099 (−1.8%) | 1099 (−1.8%) | 1252 (+11.9%) | 1344 (+20.1%) | 1080 (−3.5%) | 1099 (−1.8%) | yes | +| code-fragment | 158 | 150 (−5.1%) | 150 (−5.1%) | 263 (+66.5%) | 157 (−0.6%) | 101 (−36.1%) | 150 (−5.1%) | yes | +| json-package | 95 | 85 (−10.5%) | 90 (−5.3%) | 94 (−1.1%) | 97 (+2.1%) | 85 (−10.5%) | 85 (−10.5%) | yes | +| csv-leaderboard | 99 | 82 (−17.2%) | 91 (−8.1%) | 78 (−21.2%) | 90 (−9.1%) | 82 (−17.2%) | 78 (−21.2%) | yes | +| small-markdown | 74 | 71 (−4.1%) | 71 (−4.1%) | 72 (−2.7%) | 80 (+8.1%) | 71 (−4.1%) | 71 (−4.1%) | yes | + +## Corpus totals + +| Variant | Σ brotli | Σ BMP | vs ARX3 BMP | +| --- | ---: | ---: | ---: | +| ARX3 (baseline) | 3442 | 1750 | 0.0% | +| A implied envelope | 3295 | 1676 | −4.2% | +| B kind IR | 3325 | 1691 | −3.4% | +| C chunk cold (dict only) | 4140 | 2101 | +20.1% | +| C chunk LOO (pinned, held-out) | 4120 | 2091 | +19.5% | +| C chunk warm† (contaminated) | 963 | 505 | −71.1% | +| D template delta | 3670 | 1864 | +6.5% | +| E lossy readable | 3162 | 1609 | −8.1% | +| I word cold | 3496 | 1777 | +1.5% | +| I word LOO | 3945 | 2003 | +14.5% | +| I word warm† | 3267 | 1662 | −5.0% | +| B+C silly stack (IR+LOO) | 4121 | 2092 | +19.5% | + +## Discord UX bends (not pure single-fragment) + +### F — Label as bitstream + +Moving the title into `[label]` saves a little payload but costs framing chars. +Net is usually a wash or a loss for short titles; only interesting for long titles +that Brotli would not have collapsed much. + +| Fixture | short label | framing Δ | approx payload BMP saved | net Discord Δ | +| --- | --- | ---: | ---: | ---: | +| markdown-agents | `AGENTS.md excerpt` | +16 | ~3 | +13 | +| code-bench-report | `Baanish Code Bench` | +17 | ~4 | +13 | +| code-fragment | `fragment.ts excerpt` | +18 | ~4 | +14 | +| json-package | `package.json` | +11 | ~3 | +8 | +| csv-leaderboard | `Leaderboard` | +10 | ~3 | +7 | +| small-markdown | `Note` | +3 | ~2 | +1 | + +### G — Mosaic multipart + +Each Discord message still caps at 2000. Multiple messages multiply capacity; +multiple links *in one message* mostly fight over the same 2000 budget. + +| Fixture | ARX3 BMP | parts if separate msgs | parts fitting one msg | 1-msg capacity | +| --- | ---: | ---: | ---: | ---: | +| markdown-agents | 205 | 1 | 1 | 205 | +| code-bench-report | 1119 | 1 | 1 | 1119 | +| code-fragment | 158 | 1 | 1 | 158 | +| json-package | 95 | 1 | 1 | 95 | +| csv-leaderboard | 99 | 1 | 1 | 99 | +| small-markdown | 74 | 1 | 1 | 74 | + +Unique-prose capacity ×3 separate messages (approx): **~169k** chars vs single-link **~56k**. + +### H — Hybrid stub URL + code fence + +Same Discord message: a tiny markdown link (deeplink / stub) plus a fenced payload. +Fence digits use the same BMP alphabet; overhead is fence markers, not URL framing. + +| Stub link chars | Fence overhead | Fence payload budget | vs pure-link budget | +| ---: | ---: | ---: | ---: | +| 43 | 11 | 1945 | -24 | + +| Fixture | ARX3 BMP | fits hybrid fence? | overflow | +| --- | ---: | :---: | ---: | +| markdown-agents | 205 | yes | 0 | +| code-bench-report | 1119 | yes | 0 | +| code-fragment | 158 | yes | 0 | +| json-package | 95 | yes | 0 | +| csv-leaderboard | 99 | yes | 0 | +| small-markdown | 74 | yes | 0 | + +## Discord capacity (unique prose, binary search) + +Largest *index-salted* prose string whose encoded form fits one Discord message +(current-host framing). Unique salts stop Brotli from collapsing tiled repeats — +this is the hard case, not the highly-repetitive search-cap case. + +| Approach | Max raw chars | vs ARX3 | +| --- | ---: | ---: | +| ARX3 single link | 56303 | 0.0% | +| A implied | 54618 | −3.0% | +| B kind IR | 54866 | −2.6% | +| C chunk cold | 33299 | −40.9% | +| C chunk pinned (fixtures prior) | 52085 | −7.5% | +| I word pinned | 35630 | −36.7% | +| E lossy | 54542 | −3.1% | +| H hybrid fence (ARX3 bytes) | 55281 | −1.8% | +| G mosaic ×3 messages (approx) | ~168909 | +200.0% | + +## Interpretation — what actually moves the needle + +### Still inside one fragment + one link + +1. **Warm/contaminated chunk priors look magical (−70%+)** — ignore them. They prove only + that *if the decoder already has the bytes, you can send IDs*. That is a content-addressed + cache, not a compressor. +2. **Leave-one-out chunk priors** are the real test for a channel-pinned mother dict. + Read the LOO column: wins only where fixtures share long windows with siblings + (small JSON/CSV/notes against a prior that saw similar scaffolding). Unique reports + (code-bench) should stay near ARX3 or regress once literals dominate. +3. **Cold priors (shipped ARX dict slots only)** rarely beat ARX3+Brotli — substitution + already harvested that juice; re-encoding as chunk IDs adds framing. +4. **Implied envelope + kind IR** are small, honest wins (metadata / normalize). Worth + keeping as plumbing if ARX4 happens; not a Discord unlock alone. +5. **Template delta** as implemented is weak — Brotli already eats repeated skeletons; + a mark/residual scheme often *adds* overhead. +6. **Lossy** helps when emphasis/comments/alignment rows are noise. Product call, not codec magic. +7. **Word pack LOO** is milder than chunks; useful only with a large shared vocab that + actually overlaps the target (code keywords, markdown chrome). + +### Break-the-box (Discord UX) + +8. **Mosaic across messages** is the only lever that *multiplies* the 2000 budget. + One message with N links does **not** — they share the same 2000 chars. +9. **Hybrid fence** is usually a wash or slight loss vs pure-link (stub steals budget); + it is interesting only as a *paste UX* agents already use, not as denser packing. +10. **Label bitstream** is mostly a wash. Skip. + +### Ranked silly bets (after this probe) + +1. **Mosaic assembler** — explicit multi-message `1/3` protocol; multiplies capacity for real. +2. **Versioned shared prior** — only if LOO / held-out benches still win on *your* domain + corpus (agent-render chat is repetitive; arbitrary user prose is not). Pair with a + pinned mother post or build-shipped prior — never trust warm numbers. +3. **Hybrid fence profile** — optional surface for agents that already paste code blocks; + protocol bend, not a density win. +4. **Implied + kind IR** — cheap plumbing alongside (1) or (2). +5. **Lossy mode** — opt-in “readable enough” chat previews. +6. Avoid more residual-Brotli-dict optimism, alphabet retunes, and contaminated prior benches. + +## Non-goals reinforced + +- Do not pretend corpus-trained priors generalize without a held-out / LOO gate. +- Do not put artifact bodies in query params or require a backend for the core path. +- Do not treat mosaic/fence as drop-in replacements for zero-click single links. +- Do not update AGENTS.md / skills as if ARX4 ships. + +## How to re-run + +```bash +npm run bench:arx4-silly +# or: node scripts/bench-arx4-silly.mjs +``` + +_Generated in 387.5ms._ +See also `docs/arx4-ideation.md` and `docs/arx4-bet2-bench.md`. diff --git a/package.json b/package.json index ecae8dd..0a726a0 100644 --- a/package.json +++ b/package.json @@ -37,6 +37,7 @@ "bench:codecs:update": "node scripts/bench-codecs.mjs --write-baseline", "bench:arx4-ideation": "node scripts/arx4-ideation-probe.mjs", "bench:arx4-bet2": "node scripts/bench-arx4-bet2.mjs", + "bench:arx4-silly": "node scripts/bench-arx4-silly.mjs", "assets:compress": "node scripts/compress-dictionary.mjs", "typecheck": "node scripts/ensure-next-types.mjs && tsc --noEmit", "check": "npm run lint && npm run test && npm run bench:codecs && npm run typecheck && npm run build && npm run check:build-budgets", diff --git a/scripts/bench-arx4-silly.mjs b/scripts/bench-arx4-silly.mjs new file mode 100644 index 0000000..b609af8 --- /dev/null +++ b/scripts/bench-arx4-silly.mjs @@ -0,0 +1,1161 @@ +#!/usr/bin/env node +/** + * ARX4 "silly cuts" probe — unconventional Discord packing ideas beyond + * envelope/Brotli tweaks. + * + * Experimental only. Does not change the shipped codec surface. + * + * After bet #2 showed binary envelopes + real shared Brotli dict are ~0–2%, + * this probe measures deeper / weirder levers that can still be quantified + * without a Discord API: + * + * A. Implied envelope (raw content + 1-byte kind) — skip JSON/tuple entirely + * B. Kind-specific IR (markdown strip, JSON key-dict, CSV columnar) + * C. Pre-shared chunk prior (CDC-ish fingerprints → short IDs) + * D. Template / skeleton delta (diff vs known scaffold) + * E. Lossy "readable enough" (whitespace collapse, fence strip) + * F. Label-as-bitstream (steal title bits from Discord link label) + * G. Mosaic multipart links (N × Discord budget math) + * H. Hybrid fence payload (short URL + ```arx fence in same message) + * I. BPE-ish word pack (corpus-trained token table → varint stream) + * + * Writes `docs/arx4-silly-cuts.md`. + */ + +import { readFileSync, writeFileSync } from "node:fs"; +import { brotliCompressSync, constants } from "node:zlib"; +import { createHash } from "node:crypto"; +import { performance } from "node:perf_hooks"; + +const REPORT_PATH = "docs/arx4-silly-cuts.md"; +const DISCORD_MESSAGE_MAX = 2000; +const BMP_BASE_SIZE = 62_000; +const FENCE_TAG = "arx"; + +const v1Dictionary = JSON.parse(readFileSync("public/arx-dictionary.json", "utf8")); +const overlayDictionary = JSON.parse(readFileSync("public/arx2-dictionary.json", "utf8")); +const codeBenchReportFixture = readFileSync("tests/fixtures/baanish-code-bench-report.md", "utf8"); + +const singleByteCodes = [ + 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x0b, 0x0e, 0x0f, 0x10, 0x11, 0x12, 0x13, 0x14, 0x15, + 0x16, 0x17, 0x18, 0x19, 0x1a, 0x1b, 0x1c, 0x1d, +]; + +function buildPairs(dict, singleCodes = singleByteCodes, extendedPrefix = "\x00", extendedOffset = 1) { + const pairs = []; + for (let i = 0; i < dict.singleByteSlots.length && i < singleCodes.length; i++) { + pairs.push([dict.singleByteSlots[i], String.fromCharCode(singleCodes[i])]); + } + for (let i = 0; i < dict.extendedSlots.length; i++) { + pairs.push([dict.extendedSlots[i], extendedPrefix + String.fromCharCode(i + extendedOffset)]); + } + return pairs; +} + +const v1Pairs = buildPairs(v1Dictionary); +const overlayPairs = buildPairs(overlayDictionary, [0x1e, 0x7f], "\x1f", 0x20); + +function buildTrie(pairs, reversed = false) { + const root = { children: new Map() }; + for (const [from, to] of pairs) { + const match = reversed ? to : from; + const replacement = reversed ? from : to; + let node = root; + for (const char of match) { + let child = node.children.get(char); + if (!child) { + child = { children: new Map() }; + node.children.set(char, child); + } + node = child; + } + node.replacement ??= replacement; + } + return root; +} + +function applyTrie(text, trie) { + const out = []; + let index = 0; + while (index < text.length) { + let node = trie; + let cursor = index; + let replacement; + let replacementLength = 0; + while (cursor < text.length) { + node = node.children.get(text[cursor]); + if (!node) break; + cursor++; + if (node.replacement !== undefined) { + replacement = node.replacement; + replacementLength = cursor - index; + } + } + if (replacement !== undefined) { + out.push(replacement); + index += replacementLength; + } else { + out.push(text[index]); + index++; + } + } + return out.join(""); +} + +const v1EncodeTrie = buildTrie(v1Pairs); +const overlayEncodeTrie = buildTrie(overlayPairs); + +function brotli(input) { + return brotliCompressSync(Buffer.from(input), { + params: { [constants.BROTLI_PARAM_QUALITY]: 11 }, + }); +} + +function baseBmpChars(byteLength) { + return 3 + Math.ceil((byteLength * 8) / Math.log2(BMP_BASE_SIZE)); +} + +function discordFraming({ host = "https://agent-render.com", label = "Artifact", tag = "c" } = {}) { + const open = `[${label}](${host}#${tag}`; + const close = ")"; + return { + open, + close, + framingLength: open.length + close.length, + payloadBudget: DISCORD_MESSAGE_MAX - open.length - close.length, + }; +} + +function trimOptional(fields) { + let end = fields.length; + while (end > 0 && (fields[end - 1] === undefined || fields[end - 1] === null)) end -= 1; + return fields.slice(0, end); +} + +function artifactTuple(artifact) { + switch (artifact.kind) { + case "markdown": + return trimOptional(["m", artifact.id, artifact.content, artifact.title, artifact.filename]); + case "code": + return trimOptional(["c", artifact.id, artifact.content, artifact.language, artifact.title, artifact.filename]); + case "csv": + return trimOptional(["s", artifact.id, artifact.content, artifact.title, artifact.filename]); + case "json": + return trimOptional(["j", artifact.id, artifact.content, artifact.title, artifact.filename]); + default: + throw new Error(`unsupported kind ${artifact.kind}`); + } +} + +function tupleEnvelope(envelope) { + const artifacts = envelope.artifacts.map(artifactTuple); + if (artifacts.length === 1) return trimOptional([3, artifacts[0], envelope.title]); + return trimOptional([2, artifacts, envelope.title]); +} + +function encodeArx3Substituted(envelope) { + const tupleJson = JSON.stringify(tupleEnvelope({ ...envelope, codec: "arx3" })); + return applyTrie(applyTrie(tupleJson, overlayEncodeTrie), v1EncodeTrie); +} + +function textEnvelope(kind, title, content, extra = {}) { + return { + v: 1, + codec: "plain", + title, + activeArtifactId: "a", + artifacts: [{ id: "a", kind, title, filename: extra.filename ?? "artifact.txt", content, ...extra }], + }; +} + +function repeatedFixture(block, targetLength, segmentSuffix = (index) => `\nfixture segment ${index}\n`) { + let fixture = ""; + let index = 0; + while (fixture.length < targetLength) { + fixture += `${block}${segmentSuffix(index)}`; + index++; + } + return Array.from(fixture).slice(0, targetLength).join(""); +} + +function writeVarint(n) { + const bytes = []; + let v = n >>> 0; + while (v >= 0x80) { + bytes.push((v & 0x7f) | 0x80); + v >>>= 7; + } + bytes.push(v); + return Buffer.from(bytes); +} + +function pack(bytes) { + const brotliBytes = Buffer.isBuffer(bytes) ? brotli(bytes).length : brotli(bytes).length; + return { brotliBytes, bmpChars: baseBmpChars(brotliBytes) }; +} + +function packRaw(raw) { + const compressed = brotli(raw); + return { brotliBytes: compressed.length, bmpChars: baseBmpChars(compressed.length), rawBytes: Buffer.byteLength(raw) }; +} + +// ─── Silly cut A: implied envelope ─────────────────────────────────────────── +// Wire: 1-byte kind + raw UTF-8 content. Title/id/filename implied or in label. + +const KIND_BYTE = { markdown: 1, code: 2, csv: 3, json: 4 }; + +function encodeImplied(envelope) { + const a = envelope.artifacts[0]; + return Buffer.concat([Buffer.from([KIND_BYTE[a.kind] ?? 0]), Buffer.from(a.content, "utf8")]); +} + +// ─── Silly cut B: kind-specific IR ─────────────────────────────────────────── + +/** Markdown IR: collapse blank lines, strip trailing spaces, normalize headings. */ +function markdownIr(content) { + return content + .replace(/\r\n/g, "\n") + .replace(/[ \t]+$/gm, "") + .replace(/\n{3,}/g, "\n\n") + .replace(/^#{1,6}\s+/gm, (m) => m.trimEnd() + " ") + .trim() + "\n"; +} + +/** JSON IR: parse → sorted keys → compact; also emit key-dictionary form. */ +function jsonIr(content) { + let value; + try { + value = JSON.parse(content); + } catch { + return { compact: content, keyed: null }; + } + const compact = JSON.stringify(value); + const keys = new Set(); + function walk(v) { + if (Array.isArray(v)) { + for (const item of v) walk(item); + return; + } + if (v && typeof v === "object") { + for (const [k, child] of Object.entries(v)) { + keys.add(k); + walk(child); + } + } + } + walk(value); + const keyList = [...keys].sort(); + const keyIndex = new Map(keyList.map((k, i) => [k, i])); + function rewrite(v) { + if (Array.isArray(v)) return v.map(rewrite); + if (v && typeof v === "object") { + const out = {}; + for (const [k, child] of Object.entries(v)) { + out[String(keyIndex.get(k))] = rewrite(child); + } + return out; + } + return v; + } + const keyed = JSON.stringify({ k: keyList, v: rewrite(value) }); + return { compact, keyed }; +} + +/** CSV IR: detect delimiter, store header once, then row tuples as TSV. */ +function csvIr(content) { + const lines = content.replace(/\r\n/g, "\n").trim().split("\n"); + if (lines.length < 2) return content; + const header = lines[0]; + const delim = header.includes("\t") ? "\t" : header.includes(";") ? ";" : ","; + const cols = header.split(delim).length; + const rows = lines.slice(1).map((line) => { + const cells = line.split(delim); + while (cells.length < cols) cells.push(""); + return cells.slice(0, cols).join("\t"); + }); + return `C${cols}\n${header}\n${rows.join("\n")}`; +} + +function encodeKindIr(envelope) { + const a = envelope.artifacts[0]; + let body = a.content; + if (a.kind === "markdown") body = markdownIr(a.content); + else if (a.kind === "json") { + const ir = jsonIr(a.content); + body = ir.keyed ?? ir.compact; + } else if (a.kind === "csv") body = csvIr(a.content); + else if (a.kind === "code") { + // Light IR: strip trailing whitespace, collapse 3+ blank lines + body = a.content.replace(/[ \t]+$/gm, "").replace(/\n{3,}/g, "\n\n"); + } + return Buffer.concat([Buffer.from([KIND_BYTE[a.kind] ?? 0]), Buffer.from(body, "utf8")]); +} + +// ─── Silly cut C: pre-shared chunk prior ───────────────────────────────────── +// Build a prior from ARX dict slots + fixture scaffolding. Chunk content with +// fixed-size windows; replace exact prior matches with 2-byte IDs. + +function buildChunkPrior(extraTexts = []) { + const chunks = new Map(); // hash16 → text + const sources = [ + ...v1Dictionary.singleByteSlots, + ...v1Dictionary.extendedSlots, + ...overlayDictionary.singleByteSlots, + ...overlayDictionary.extendedSlots, + ...extraTexts, + ]; + // Also index sliding windows of common sizes from sources + for (const src of sources) { + if (!src || src.length < 8) { + if (src && src.length >= 4) { + const h = hash16(src); + if (!chunks.has(h)) chunks.set(h, src); + } + continue; + } + for (const size of [16, 32, 48, 64]) { + for (let i = 0; i + size <= src.length; i += Math.max(8, size >> 2)) { + const slice = src.slice(i, i + size); + const h = hash16(slice); + if (!chunks.has(h)) chunks.set(h, slice); + } + } + } + return chunks; +} + +function hash16(text) { + const digest = createHash("sha1").update(text, "utf8").digest(); + return (digest[0] << 8) | digest[1]; +} + +/** + * Greedy longest-match against prior. Wire: + * 0x00 + u16be id → prior hit + * 0x01 + varint n + n bytes → literal + */ +function encodeChunkPrior(content, prior) { + const sizes = [64, 48, 32, 16, 12, 8]; + const out = []; + let i = 0; + // Invert prior: text → id (prefer longer) + const byText = new Map(); + for (const [id, text] of prior) { + const prev = byText.get(text); + if (prev === undefined || text.length > prior.get(prev)?.length) byText.set(text, id); + } + while (i < content.length) { + let hit = null; + for (const size of sizes) { + if (i + size > content.length) continue; + const slice = content.slice(i, i + size); + const id = byText.get(slice); + if (id !== undefined) { + hit = { id, size }; + break; + } + } + if (hit) { + out.push(0x00, (hit.id >> 8) & 0xff, hit.id & 0xff); + i += hit.size; + } else { + // Emit a run of literals until next potential hit or 64 chars + let end = i + 1; + while (end < content.length && end - i < 64) { + let found = false; + for (const size of sizes) { + if (end + size <= content.length && byText.has(content.slice(end, end + size))) { + found = true; + break; + } + } + if (found) break; + end++; + } + const lit = Buffer.from(content.slice(i, end), "utf8"); + out.push(0x01, ...writeVarint(lit.length), ...lit); + i = end; + } + } + return Buffer.from(out); +} + +// ─── Silly cut D: template / skeleton delta ────────────────────────────────── + +const MARKDOWN_SKELETON = [ + "# ", + "\n\n## ", + "\n\n### ", + "\n\n- ", + "\n\n```", + "\n```\n", + "\n\n| ", + " | ", + " |\n", + "agent-render", + "Discord", + "fragment", + "artifact", +].join(""); + +const CODE_SKELETON = [ + "export function ", + "export async function ", + "export const ", + "import { ", + '} from "', + "return ", + "if (!", + "await ", + "const ", + "function ", +].join(""); + +function simpleDelta(content, skeleton) { + // Myers-ish is overkill; use a cheap "remove skeleton substrings in order" + + // store residual with skeleton id. Better: store content with skeleton as + // Brotli shared-dict proxy via concatenation residual estimate, AND a real + // strip-and-mark encoding. + let residual = content; + const marks = []; + for (const token of skeleton.match(/.{1,24}/g) ?? []) { + let idx; + while ((idx = residual.indexOf(token)) !== -1) { + marks.push([idx, token.length]); + residual = residual.slice(0, idx) + residual.slice(idx + token.length); + if (marks.length > 200) break; + } + if (marks.length > 200) break; + } + // Wire: skeleton-id (1) + varint residualLen + residual + mark count + marks + const resBuf = Buffer.from(residual, "utf8"); + const markBuf = Buffer.alloc(marks.length * 4); + marks.forEach(([idx, len], i) => { + markBuf.writeUInt16BE(Math.min(idx, 0xffff), i * 4); + markBuf.writeUInt16BE(Math.min(len, 0xffff), i * 4 + 2); + }); + return Buffer.concat([ + Buffer.from([1]), + writeVarint(resBuf.length), + resBuf, + writeVarint(marks.length), + markBuf, + ]); +} + +// ─── Silly cut E: lossy readable-enough ────────────────────────────────────── + +function lossyMarkdown(content) { + return content + .replace(/\r\n/g, "\n") + .replace(/[ \t]+/g, " ") + .replace(/\n[ \t]+/g, "\n") + .replace(/\n{3,}/g, "\n\n") + // Drop markdown emphasis markers (lossy but often still readable) + .replace(/\*\*([^*]+)\*\*/g, "$1") + .replace(/\*([^*]+)\*/g, "$1") + .replace(/__([^_]+)__/g, "$1") + .replace(/_([^_]+)_/g, "$1") + // Collapse table alignment rows + .replace(/^\|[-:| ]+\|$/gm, "") + .replace(/\n{3,}/g, "\n\n") + .trim() + "\n"; +} + +function lossyCode(content) { + return content + .replace(/[ \t]+$/gm, "") + .replace(/\n{3,}/g, "\n\n") + // Strip line comments (aggressive / silly) + .replace(/^\s*\/\/.*$/gm, "") + .replace(/\n{3,}/g, "\n\n") + .trim() + "\n"; +} + +// ─── Silly cut F: label-as-bitstream ───────────────────────────────────────── +// Discord label can carry title/id; fragment only needs content. +// Measure: implied content pack + short label carrying title. + +function labelBitstreamSavings(envelope, contentBmpChars) { + const title = envelope.title || envelope.artifacts[0].title || "a"; + // Shorten title to fit; use first grapheme-ish chars + const shortLabel = Array.from(title).slice(0, 24).join("") || "a"; + const withTitleInEnvelope = discordFraming({ label: "a", tag: "c" }); + // Envelope that still carries title in payload uses short label "a" + const titleInPayloadFraming = withTitleInEnvelope.framingLength; + // Title in label: longer framing, but payload can drop title field (~title.length + JSON quotes) + const titleInLabelFraming = discordFraming({ label: shortLabel, tag: "c" }).framingLength; + // Approximate payload title overhead in ARX3 tuple JSON + const titleOverheadChars = JSON.stringify(title).length + 1; // rough + const titleOverheadBmp = Math.ceil(titleOverheadChars * 0.15); // after brotli+bmp, rough + return { + shortLabel, + framingWithTitleInLabel: titleInLabelFraming, + framingWithShortLabel: titleInPayloadFraming, + framingDelta: titleInLabelFraming - titleInPayloadFraming, + approxPayloadBmpSaved: titleOverheadBmp, + netDiscordDelta: titleInLabelFraming - titleInPayloadFraming - titleOverheadBmp, + contentBmpChars, + }; +} + +// ─── Silly cut G: mosaic multipart ─────────────────────────────────────────── + +function mosaicMath(bmpChars, { host = "https://agent-render.com", labelPrefix = "p" } = {}) { + const parts = []; + let remaining = bmpChars; + let part = 0; + while (remaining > 0 && part < 20) { + const label = `${labelPrefix}${part}`; + const { payloadBudget, framingLength } = discordFraming({ host, label, tag: "c" }); + const take = Math.min(remaining, payloadBudget); + parts.push({ part, label, framingLength, payloadBudget, take }); + remaining -= take; + part++; + } + // Discord reality: each link is a separate message OR one message with N links. + // One message with N links: sum of all link lengths ≤ 2000. + let packedInOne = 0; + let used = 0; + for (const p of parts) { + const linkLen = p.framingLength + p.take; + if (used + linkLen + (packedInOne > 0 ? 1 : 0) > DISCORD_MESSAGE_MAX) break; + used += linkLen + (packedInOne > 0 ? 1 : 0); + packedInOne++; + } + return { + partsNeededSeparateMessages: parts.length, + partsFittingOneMessage: packedInOne, + totalBmpChars: bmpChars, + capacityOneMessage: parts.slice(0, packedInOne).reduce((n, p) => n + p.take, 0), + capacityNMessages: parts.reduce((n, p) => n + p.take, 0), + }; +} + +// ─── Silly cut H: hybrid fence ─────────────────────────────────────────────── +// Message: [a](https://host/#cSHORT) + ```arx\nPAYLOAD\n``` +// Short URL can be a stub/hash; fence carries the bulk (still in Discord message). + +function hybridFenceMath(bmpChars, { host = "https://agent-render.com", stubChars = 12 } = {}) { + const stubLink = discordFraming({ host, label: "a", tag: "c" }); + const stubTotal = stubLink.framingLength + stubChars; + const fenceOverhead = "```".length + FENCE_TAG.length + 1 + 1 + "```".length; // ```arx\n ... \n``` + const fenceBudget = DISCORD_MESSAGE_MAX - stubTotal - 1 - fenceOverhead; // space between + return { + stubTotal, + fenceOverhead, + fenceBudget, + fitsInOneMessage: bmpChars <= fenceBudget, + overflow: Math.max(0, bmpChars - fenceBudget), + // Fence payload is raw baseBMP digits — same density, but no URL encoding issues + // and Discord may count the same UTF-16 units. + effectiveBudgetVsPureLink: fenceBudget - stubLink.payloadBudget, + }; +} + +// ─── Silly cut I: BPE-ish word pack ────────────────────────────────────────── + +function trainWordTable(texts, maxTokens = 512) { + const counts = new Map(); + for (const text of texts) { + for (const word of text.split(/(\s+|[^a-zA-Z0-9_]+)/)) { + if (!word || word.length < 2) continue; + counts.set(word, (counts.get(word) || 0) + 1); + } + } + return [...counts.entries()] + .filter(([, n]) => n >= 2) + .sort((a, b) => b[1] * b[0].length - a[1] * a[0].length) + .slice(0, maxTokens) + .map(([w]) => w); +} + +function encodeWordPack(content, table) { + const index = new Map(table.map((w, i) => [w, i])); + // Sort by length desc for greedy + const sorted = [...table].sort((a, b) => b.length - a.length); + const out = []; + let i = 0; + while (i < content.length) { + let hit = null; + for (const w of sorted) { + if (content.startsWith(w, i)) { + hit = index.get(w); + out.push(0x80 | ((hit >> 8) & 0x7f), hit & 0xff); + i += w.length; + break; + } + } + if (hit === null) { + // literal byte run + let end = i + 1; + while (end < content.length && end - i < 127) { + let found = false; + for (const w of sorted) { + if (content.startsWith(w, end)) { + found = true; + break; + } + } + if (found) break; + end++; + } + const lit = Buffer.from(content.slice(i, end), "utf8"); + out.push(lit.length & 0x7f, ...lit); + i = end; + } + } + // Prepend table is NOT in wire — assumed shared prior. Just the stream. + return Buffer.from(out); +} + +// ─── Corpus ────────────────────────────────────────────────────────────────── + +const markdownAgentsFixture = repeatedFixture( + [ + "# AGENTS.md excerpt", + "", + "`agent-render` is a static artifact viewer for AI-generated outputs.", + "Keep markdown, code, diffs, CSV, and JSON readable across chat surfaces.", + "", + "## Product contract", + "", + "- Fragment payloads use `#agent-render=v1..`.", + "- Artifact contents stay out of the host request path.", + "- Supported codecs are `plain`, `lz`, `deflate`, `arx`, `arx2`, and `arx3`.", + "- Supported artifact kinds are `markdown`, `code`, `diff`, `csv`, and `json`.", + "", + "Preserve the static shell, the zero-retention wording, and the renderer-first layout.", + "", + ].join("\n"), + 8000, + (index) => + `\nFixture note ${index}: fragment transport, renderer readiness, and artifact metadata stay aligned.\n\n`, +); + +const codeFragmentFixture = repeatedFixture( + [ + "export async function decodeFragmentAsync(hash: string, options?: DecodeOptions) {", + " const parsed = parseFragmentPrefix(hash);", + " if (!parsed.ok) return parsed;", + ' if (parsed.codec === "arx" || parsed.codec === "arx2") {', + ' const { decodeArxFragmentAsync } = await import("./fragment-arx");', + " return decodeArxFragmentAsync(parsed, options);", + " }", + " return decodePlainFragment(parsed.payload, options);", + "}", + "", + ].join("\n"), + 8000, + (index) => `\n// fixture segment ${index}: codec branch coverage and bundle shape stay stable.\n`, +); + +const packageManifestFixture = JSON.stringify( + { + name: "agent-render", + version: "0.1.0", + private: true, + scripts: { build: "next build", check: "npm run lint && npm run test" }, + dependencies: { next: "15.1.11", react: "19.1.0", "brotli-wasm": "^3.0.1" }, + }, + null, + 2, +); + +const csvFixture = [ + "model,avg,delta,tasks,wins", + "gpt-5.5,93.91,0,25,7", + "mimo/mimo-v2.5-pro,93.29,-0.61,25,5", + "Kimi-K2.6-Turbo,91.63,-2.28,25,5", + "synthetic/GLM-5.1,89.07,-4.84,25,5", + "gpt-5.3-codex-spark,88.68,-5.23,25,3", + "gpt-5.4-mini,88.19,-5.72,25,3", +].join("\n"); + +const corpus = [ + { + name: "markdown-agents", + envelope: textEnvelope("markdown", "AGENTS.md excerpt", markdownAgentsFixture, { filename: "AGENTS.md" }), + }, + { + name: "code-bench-report", + envelope: textEnvelope("markdown", "Baanish Code Bench", codeBenchReportFixture, { filename: "results.md" }), + }, + { + name: "code-fragment", + envelope: textEnvelope("code", "fragment.ts excerpt", codeFragmentFixture, { + filename: "fragment.ts", + language: "ts", + }), + }, + { + name: "json-package", + envelope: textEnvelope("json", "package.json", packageManifestFixture, { filename: "package.json" }), + }, + { + name: "csv-leaderboard", + envelope: textEnvelope("csv", "Leaderboard", csvFixture, { filename: "board.csv" }), + }, + { + name: "small-markdown", + envelope: textEnvelope( + "markdown", + "Note", + [ + "# Sprint notes", + "", + "- Ship ARX3 visible URL mode", + "- Keep Discord markdown links under 2000 chars", + "- Prefer fragment transport over UUID mode for zero-retention", + "", + "```ts", + "export const value = 1;", + "```", + "", + ].join("\n"), + { filename: "notes.md" }, + ), + }, +]; + +// Priors: +// cold = ARX dict slots only (shipped today — honest baseline) +// loo = leave-one-out corpus (channel-pinned mother dict, held-out target) +// warm = full corpus including target (optimistic ceiling / contamination) +const allContents = corpus.map((c) => c.envelope.artifacts[0].content); +const coldChunkPrior = buildChunkPrior([]); +const warmChunkPrior = buildChunkPrior(allContents); +const coldWordTable = trainWordTable( + [...v1Dictionary.singleByteSlots, ...overlayDictionary.singleByteSlots, ...overlayDictionary.extendedSlots], + 512, +); +const warmWordTable = trainWordTable(allContents, 512); + +function leaveOneOutContents(skipIndex) { + return allContents.filter((_, i) => i !== skipIndex); +} + +function pct(from, to) { + if (from == null || to == null) return null; + return ((to - from) / from) * 100; +} + +function fmtDelta(from, to) { + const p = pct(from, to); + if (p == null || Number.isNaN(p)) return "n/a"; + if (p === 0) return "0.0%"; + return `${p < 0 ? "−" : "+"}${Math.abs(p).toFixed(1)}%`; +} + +function measureRow(envelope, fixtureIndex) { + const content = envelope.artifacts[0].content; + const kind = envelope.artifacts[0].kind; + const arx3 = packRaw(encodeArx3Substituted(envelope)); + + const implied = packRaw(encodeImplied(envelope)); + const kindIr = packRaw(encodeKindIr(envelope)); + + const looChunkPrior = buildChunkPrior(leaveOneOutContents(fixtureIndex)); + const looWordTable = trainWordTable(leaveOneOutContents(fixtureIndex), 512); + + function chunkImpliedWith(prior) { + const encoded = encodeChunkPrior(content, prior); + return packRaw(Buffer.concat([Buffer.from([KIND_BYTE[kind] ?? 0]), encoded])); + } + function wordPackWith(table) { + return packRaw(Buffer.concat([Buffer.from([KIND_BYTE[kind] ?? 0]), encodeWordPack(content, table)])); + } + + const chunkCold = chunkImpliedWith(coldChunkPrior); + const chunkLoo = chunkImpliedWith(looChunkPrior); + const chunkWarm = chunkImpliedWith(warmChunkPrior); + const wordCold = wordPackWith(coldWordTable); + const wordLoo = wordPackWith(looWordTable); + const wordWarm = wordPackWith(warmWordTable); + + const skeleton = kind === "code" ? CODE_SKELETON : MARKDOWN_SKELETON; + const deltaPack = packRaw(simpleDelta(content, skeleton)); + + let lossyContent = content; + if (kind === "markdown") lossyContent = lossyMarkdown(content); + else if (kind === "code") lossyContent = lossyCode(content); + const lossyPack = packRaw(Buffer.concat([Buffer.from([KIND_BYTE[kind] ?? 0]), Buffer.from(lossyContent, "utf8")])); + + // Combined: kind IR → leave-one-out chunk prior + const irBody = encodeKindIr(envelope).subarray(1).toString("utf8"); + const sillyStack = packRaw( + Buffer.concat([Buffer.from([KIND_BYTE[kind] ?? 0]), encodeChunkPrior(irBody, looChunkPrior)]), + ); + + const label = labelBitstreamSavings(envelope, arx3.bmpChars); + const mosaic = mosaicMath(arx3.bmpChars); + const hybrid = hybridFenceMath(arx3.bmpChars); + + const framing = discordFraming(); + const fits = (bmp) => bmp <= framing.payloadBudget; + + return { + rawContentChars: content.length, + arx3, + implied, + kindIr, + chunkCold, + chunkLoo, + chunkWarm, + wordCold, + wordLoo, + wordWarm, + templateDelta: deltaPack, + lossy: lossyPack, + sillyStack, + label, + mosaic, + hybrid, + priorSizes: { + coldChunks: coldChunkPrior.size, + looChunks: looChunkPrior.size, + warmChunks: warmChunkPrior.size, + coldWords: coldWordTable.length, + looWords: looWordTable.length, + warmWords: warmWordTable.length, + }, + fitsDiscord: { + arx3: fits(arx3.bmpChars), + bestHonest: fits( + Math.min(arx3.bmpChars, implied.bmpChars, kindIr.bmpChars, chunkLoo.bmpChars, wordLoo.bmpChars), + ), + lossy: fits(lossyPack.bmpChars), + }, + }; +} + +const t0 = performance.now(); +const rows = corpus.map((item, index) => ({ name: item.name, ...measureRow(item.envelope, index) })); +const elapsed = performance.now() - t0; + +/** + * Capacity probe uses *unique* prose (index-salted) so Brotli cannot collapse + * the whole payload into a tiny repeat — matches the ideation probe's hard case. + */ +function uniqueProse(targetLength) { + const sentences = [ + "Fragment transport keeps artifact bodies off the host request path.", + "Discord markdown links gate on UTF-16 units, not optimistic code points.", + "Shared priors only help when encoder and decoder pin the same version.", + "Mosaic assemblers multiply budget across messages, not within one.", + "Kind-specific IR is plumbing; it is not a denser alphabet.", + ]; + let out = ""; + let i = 0; + while (out.length < targetLength) { + const s = sentences[i % sentences.length]; + out += `${s} [${i.toString(36)}:${(i * 7919).toString(16)}] `; + if (i % 7 === 0) out += "\n\n"; + i++; + } + return Array.from(out).slice(0, targetLength).join(""); +} + +function capacitySearch(encodeFn, { label = "a", budget = null, hi = 120_000 } = {}) { + const framing = discordFraming({ label, tag: "c" }); + const limit = budget ?? framing.payloadBudget; + let lo = 100; + let best = 0; + while (lo <= hi) { + const mid = (lo + hi) >> 1; + const env = textEnvelope("markdown", "cap", uniqueProse(mid)); + const packed = encodeFn(env); + if (packed.bmpChars <= limit) { + best = mid; + lo = mid + 1; + } else { + hi = mid - 1; + } + } + return best; +} + +// For capacity chunk/word packs: prior from dict + bench fixtures (not the synthetic prose) +const capacityChunkPrior = buildChunkPrior(allContents); +const capacityWordTable = trainWordTable(allContents, 512); + +const capacity = { + arx3: capacitySearch((env) => packRaw(encodeArx3Substituted(env))), + implied: capacitySearch((env) => packRaw(encodeImplied(env))), + kindIr: capacitySearch((env) => packRaw(encodeKindIr(env))), + chunkCold: capacitySearch((env) => { + const content = env.artifacts[0].content; + return packRaw(Buffer.concat([Buffer.from([1]), encodeChunkPrior(content, coldChunkPrior)])); + }), + chunkPinned: capacitySearch((env) => { + const content = env.artifacts[0].content; + return packRaw(Buffer.concat([Buffer.from([1]), encodeChunkPrior(content, capacityChunkPrior)])); + }), + wordPinned: capacitySearch((env) => { + const content = env.artifacts[0].content; + return packRaw(Buffer.concat([Buffer.from([1]), encodeWordPack(content, capacityWordTable)])); + }), + lossy: capacitySearch((env) => { + const body = lossyMarkdown(env.artifacts[0].content); + return packRaw(Buffer.concat([Buffer.from([1]), Buffer.from(body, "utf8")])); + }), +}; + +const singleCap = capacity.arx3; +const mosaic3Cap = singleCap * 3; + +const hybridBudget = hybridFenceMath(0).fenceBudget; +const hybridCap = capacitySearch((env) => packRaw(encodeArx3Substituted(env)), { budget: hybridBudget }); + +// ─── Report ────────────────────────────────────────────────────────────────── + +const lines = []; +const w = (s = "") => lines.push(s); + +w("# ARX4 silly cuts — unconventional Discord packing"); +w(); +w("_Experimental notes from `scripts/bench-arx4-silly.mjs`. Not a shipped codec._"); +w(); +w("## Why this pass"); +w(); +w("Bet #2 (binary envelopes + real Brotli `-D`) topped out around **~0–2%** vs ARX3."); +w("Wire density is already ~99.5% of the 16-bit UTF-16 ceiling. Alphabet and residual-dict"); +w("tweaks are exhausted. This pass looks for **deeper / sillier** levers:"); +w(); +w("- shared priors the *viewer already knows* (chunks, word tables, templates)"); +w("- kind-specific IR and lossy readable-enough transforms"); +w("- Discord UX bends (mosaic links, fence hybrid, label-as-bitstream)"); +w("- implied envelopes that drop metadata from the fragment"); +w(); +w("## Ideas measured"); +w(); +w("| Cut | Idea | Lossless? | Product tension |"); +w("| --- | --- | :---: | --- |"); +w("| A | **Implied envelope** — 1-byte kind + raw content | yes | drops id/title/filename from fragment |"); +w("| B | **Kind IR** — md normalize / JSON key-dict / CSV columnar | mostly | kind-specific decoders |"); +w("| C | **Chunk prior** — CDC-ish windows → 2-byte IDs vs shared prior | yes* | prior must be pinned/versioned |"); +w("| D | **Template delta** — strip known skeleton, store residual+marks | yes* | skeleton catalog |"); +w("| E | **Lossy readable** — collapse ws, strip emphasis/comments | no | quality trade |"); +w("| F | **Label bitstream** — put title in Discord `[label]` | yes | label UX / length |"); +w("| G | **Mosaic** — N markdown links / N messages | yes | multi-click UX |"); +w("| H | **Hybrid fence** — stub URL + ` ```arx ` payload in same message | yes | not pure-link; scanners differ |"); +w("| I | **Word pack** — corpus BPE-ish token table → id stream | yes* | shared vocab |"); +w(); +w("\\* Lossless only if encoder and decoder share the same prior/vocab/skeleton version."); +w(); +w(`Prior sizes this run: cold chunks **${coldChunkPrior.size}**, warm chunks **${warmChunkPrior.size}**, cold words **${coldWordTable.length}**, warm words **${warmWordTable.length}**.`); +w("Leave-one-out (LOO) priors exclude the measured fixture — that is the honest “pinned mother dict” estimate."); +w(); + +w("## Per-fixture Brotli bytes (vs ARX3)"); +w(); +w( + "| Fixture | raw | ARX3 | implied | kind IR | chunk cold | chunk LOO | chunk warm† | word LOO | lossy | silly stack (IR+LOO) |", +); +w("| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |"); +for (const r of rows) { + const b = r.arx3.brotliBytes; + w( + `| ${r.name} | ${r.rawContentChars} | ${b} | ${r.implied.brotliBytes} (${fmtDelta(b, r.implied.brotliBytes)}) | ${r.kindIr.brotliBytes} (${fmtDelta(b, r.kindIr.brotliBytes)}) | ${r.chunkCold.brotliBytes} (${fmtDelta(b, r.chunkCold.brotliBytes)}) | ${r.chunkLoo.brotliBytes} (${fmtDelta(b, r.chunkLoo.brotliBytes)}) | ${r.chunkWarm.brotliBytes} (${fmtDelta(b, r.chunkWarm.brotliBytes)}) | ${r.wordLoo.brotliBytes} (${fmtDelta(b, r.wordLoo.brotliBytes)}) | ${r.lossy.brotliBytes} (${fmtDelta(b, r.lossy.brotliBytes)}) | ${r.sillyStack.brotliBytes} (${fmtDelta(b, r.sillyStack.brotliBytes)}) |`, + ); +} +w(); +w("† **chunk warm** includes the target fixture in the prior — contamination ceiling, not a real win."); + +w(); +w("## Per-fixture visible BMP chars (vs ARX3)"); +w(); +w( + "| Fixture | ARX3 BMP | implied | kind IR | chunk LOO | word LOO | lossy | best honest | Discord fit |", +); +w("| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | :---: |"); +for (const r of rows) { + const b = r.arx3.bmpChars; + const best = Math.min( + r.arx3.bmpChars, + r.implied.bmpChars, + r.kindIr.bmpChars, + r.chunkLoo.bmpChars, + r.wordLoo.bmpChars, + r.sillyStack.bmpChars, + ); + w( + `| ${r.name} | ${b} | ${r.implied.bmpChars} (${fmtDelta(b, r.implied.bmpChars)}) | ${r.kindIr.bmpChars} (${fmtDelta(b, r.kindIr.bmpChars)}) | ${r.chunkLoo.bmpChars} (${fmtDelta(b, r.chunkLoo.bmpChars)}) | ${r.wordLoo.bmpChars} (${fmtDelta(b, r.wordLoo.bmpChars)}) | ${r.lossy.bmpChars} (${fmtDelta(b, r.lossy.bmpChars)}) | ${best} (${fmtDelta(b, best)}) | ${r.fitsDiscord.arx3 ? "yes" : "no"} |`, + ); +} + +// Totals +function sum(sel) { + return rows.reduce((n, r) => n + sel(r), 0); +} +w(); +w("## Corpus totals"); +w(); +w("| Variant | Σ brotli | Σ BMP | vs ARX3 BMP |"); +w("| --- | ---: | ---: | ---: |"); +const variants = [ + ["ARX3 (baseline)", (r) => r.arx3], + ["A implied envelope", (r) => r.implied], + ["B kind IR", (r) => r.kindIr], + ["C chunk cold (dict only)", (r) => r.chunkCold], + ["C chunk LOO (pinned, held-out)", (r) => r.chunkLoo], + ["C chunk warm† (contaminated)", (r) => r.chunkWarm], + ["D template delta", (r) => r.templateDelta], + ["E lossy readable", (r) => r.lossy], + ["I word cold", (r) => r.wordCold], + ["I word LOO", (r) => r.wordLoo], + ["I word warm†", (r) => r.wordWarm], + ["B+C silly stack (IR+LOO)", (r) => r.sillyStack], +]; +const arx3BmpTotal = sum((r) => r.arx3.bmpChars); +for (const [name, sel] of variants) { + const brotliSum = sum((r) => sel(r).brotliBytes); + const bmpSum = sum((r) => sel(r).bmpChars); + w(`| ${name} | ${brotliSum} | ${bmpSum} | ${fmtDelta(arx3BmpTotal, bmpSum)} |`); +} + +w(); +w("## Discord UX bends (not pure single-fragment)"); +w(); +w("### F — Label as bitstream"); +w(); +w("Moving the title into `[label]` saves a little payload but costs framing chars."); +w("Net is usually a wash or a loss for short titles; only interesting for long titles"); +w("that Brotli would not have collapsed much."); +w(); +w("| Fixture | short label | framing Δ | approx payload BMP saved | net Discord Δ |"); +w("| --- | --- | ---: | ---: | ---: |"); +for (const r of rows) { + const L = r.label; + w( + `| ${r.name} | \`${L.shortLabel.replace(/\|/g, "\\|")}\` | ${L.framingDelta > 0 ? "+" : ""}${L.framingDelta} | ~${L.approxPayloadBmpSaved} | ${L.netDiscordDelta > 0 ? "+" : ""}${L.netDiscordDelta} |`, + ); +} + +w(); +w("### G — Mosaic multipart"); +w(); +w("Each Discord message still caps at 2000. Multiple messages multiply capacity;"); +w("multiple links *in one message* mostly fight over the same 2000 budget."); +w(); +w("| Fixture | ARX3 BMP | parts if separate msgs | parts fitting one msg | 1-msg capacity |"); +w("| --- | ---: | ---: | ---: | ---: |"); +for (const r of rows) { + const m = r.mosaic; + w( + `| ${r.name} | ${m.totalBmpChars} | ${m.partsNeededSeparateMessages} | ${m.partsFittingOneMessage} | ${m.capacityOneMessage} |`, + ); +} +w(); +w( + `Unique-prose capacity ×3 separate messages (approx): **~${(mosaic3Cap / 1000).toFixed(0)}k** chars vs single-link **~${(singleCap / 1000).toFixed(0)}k**.`, +); + +w(); +w("### H — Hybrid stub URL + code fence"); +w(); +w("Same Discord message: a tiny markdown link (deeplink / stub) plus a fenced payload."); +w("Fence digits use the same BMP alphabet; overhead is fence markers, not URL framing."); +w(); +const href = hybridFenceMath(0); +w(`| Stub link chars | Fence overhead | Fence payload budget | vs pure-link budget |`); +w(`| ---: | ---: | ---: | ---: |`); +w( + `| ${href.stubTotal} | ${href.fenceOverhead} | ${href.fenceBudget} | ${href.effectiveBudgetVsPureLink > 0 ? "+" : ""}${href.effectiveBudgetVsPureLink} |`, +); +w(); +w("| Fixture | ARX3 BMP | fits hybrid fence? | overflow |"); +w("| --- | ---: | :---: | ---: |"); +for (const r of rows) { + w( + `| ${r.name} | ${r.arx3.bmpChars} | ${r.hybrid.fitsInOneMessage ? "yes" : "no"} | ${r.hybrid.overflow} |`, + ); +} + +w(); +w("## Discord capacity (unique prose, binary search)"); +w(); +w("Largest *index-salted* prose string whose encoded form fits one Discord message"); +w("(current-host framing). Unique salts stop Brotli from collapsing tiled repeats —"); +w("this is the hard case, not the highly-repetitive search-cap case."); +w(); +w("| Approach | Max raw chars | vs ARX3 |"); +w("| --- | ---: | ---: |"); +w(`| ARX3 single link | ${capacity.arx3} | 0.0% |`); +w(`| A implied | ${capacity.implied} | ${fmtDelta(capacity.arx3, capacity.implied)} |`); +w(`| B kind IR | ${capacity.kindIr} | ${fmtDelta(capacity.arx3, capacity.kindIr)} |`); +w(`| C chunk cold | ${capacity.chunkCold} | ${fmtDelta(capacity.arx3, capacity.chunkCold)} |`); +w(`| C chunk pinned (fixtures prior) | ${capacity.chunkPinned} | ${fmtDelta(capacity.arx3, capacity.chunkPinned)} |`); +w(`| I word pinned | ${capacity.wordPinned} | ${fmtDelta(capacity.arx3, capacity.wordPinned)} |`); +w(`| E lossy | ${capacity.lossy} | ${fmtDelta(capacity.arx3, capacity.lossy)} |`); +w(`| H hybrid fence (ARX3 bytes) | ${hybridCap} | ${fmtDelta(capacity.arx3, hybridCap)} |`); +w(`| G mosaic ×3 messages (approx) | ~${mosaic3Cap} | ${fmtDelta(capacity.arx3, mosaic3Cap)} |`); + +w(); +w("## Interpretation — what actually moves the needle"); +w(); +w("### Still inside one fragment + one link"); +w(); +w("1. **Warm/contaminated chunk priors look magical (−70%+)** — ignore them. They prove only"); +w(" that *if the decoder already has the bytes, you can send IDs*. That is a content-addressed"); +w(" cache, not a compressor."); +w("2. **Leave-one-out chunk priors** are the real test for a channel-pinned mother dict."); +w(" Read the LOO column: wins only where fixtures share long windows with siblings"); +w(" (small JSON/CSV/notes against a prior that saw similar scaffolding). Unique reports"); +w(" (code-bench) should stay near ARX3 or regress once literals dominate."); +w("3. **Cold priors (shipped ARX dict slots only)** rarely beat ARX3+Brotli — substitution"); +w(" already harvested that juice; re-encoding as chunk IDs adds framing."); +w("4. **Implied envelope + kind IR** are small, honest wins (metadata / normalize). Worth"); +w(" keeping as plumbing if ARX4 happens; not a Discord unlock alone."); +w("5. **Template delta** as implemented is weak — Brotli already eats repeated skeletons;"); +w(" a mark/residual scheme often *adds* overhead."); +w("6. **Lossy** helps when emphasis/comments/alignment rows are noise. Product call, not codec magic."); +w("7. **Word pack LOO** is milder than chunks; useful only with a large shared vocab that"); +w(" actually overlaps the target (code keywords, markdown chrome)."); +w(); +w("### Break-the-box (Discord UX)"); +w(); +w("8. **Mosaic across messages** is the only lever that *multiplies* the 2000 budget."); +w(" One message with N links does **not** — they share the same 2000 chars."); +w("9. **Hybrid fence** is usually a wash or slight loss vs pure-link (stub steals budget);"); +w(" it is interesting only as a *paste UX* agents already use, not as denser packing."); +w("10. **Label bitstream** is mostly a wash. Skip."); +w(); +w("### Ranked silly bets (after this probe)"); +w(); +w("1. **Mosaic assembler** — explicit multi-message `1/3` protocol; multiplies capacity for real."); +w("2. **Versioned shared prior** — only if LOO / held-out benches still win on *your* domain"); +w(" corpus (agent-render chat is repetitive; arbitrary user prose is not). Pair with a"); +w(" pinned mother post or build-shipped prior — never trust warm numbers."); +w("3. **Hybrid fence profile** — optional surface for agents that already paste code blocks;"); +w(" protocol bend, not a density win."); +w("4. **Implied + kind IR** — cheap plumbing alongside (1) or (2)."); +w("5. **Lossy mode** — opt-in “readable enough” chat previews."); +w("6. Avoid more residual-Brotli-dict optimism, alphabet retunes, and contaminated prior benches."); +w(); +w("## Non-goals reinforced"); +w(); +w("- Do not pretend corpus-trained priors generalize without a held-out / LOO gate."); +w("- Do not put artifact bodies in query params or require a backend for the core path."); +w("- Do not treat mosaic/fence as drop-in replacements for zero-click single links."); +w("- Do not update AGENTS.md / skills as if ARX4 ships."); +w(); +w("## How to re-run"); +w(); +w("```bash"); +w("npm run bench:arx4-silly"); +w("# or: node scripts/bench-arx4-silly.mjs"); +w("```"); +w(); +w(`_Generated in ${elapsed.toFixed(1)}ms._`); +w("See also `docs/arx4-ideation.md` and `docs/arx4-bet2-bench.md`."); + +writeFileSync(REPORT_PATH, lines.join("\n") + "\n"); +console.log(`Wrote ${REPORT_PATH}`); +console.log(`Fixtures: ${rows.length}, elapsed ${elapsed.toFixed(1)}ms`); +console.log( + `ARX3 Σ BMP ${arx3BmpTotal}; LOO chunk Σ ${sum((r) => r.chunkLoo.bmpChars)} (${fmtDelta(arx3BmpTotal, sum((r) => r.chunkLoo.bmpChars))}); warm† Σ ${sum((r) => r.chunkWarm.bmpChars)} (${fmtDelta(arx3BmpTotal, sum((r) => r.chunkWarm.bmpChars))})`, +); +console.log( + `Capacity ARX3=${capacity.arx3} implied=${capacity.implied} chunkCold=${capacity.chunkCold} chunkPinned=${capacity.chunkPinned} wordPinned=${capacity.wordPinned} hybrid=${hybridCap} mosaic3~${mosaic3Cap}`, +); From b9f6c1026165bd1ad3062b24db16b2a7f20dda65 Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Wed, 8 Jul 2026 19:30:06 +0000 Subject: [PATCH 5/5] Deprioritize mosaic; probe kind dicts; teach skill semantic splits. Kind-specific LOO overlays do not beat shared ARX3 on this corpus (even with free tags). Prefer agent-side section/file splits in the linking skill over mosaic reassembly. Document the +1-char verdict. Co-authored-by: Aanish Bhirud --- docs/arx4-ideation.md | 64 +-- docs/arx4-kind-dicts.md | 104 ++++ docs/arx4-silly-cuts.md | 12 +- package.json | 3 +- scripts/bench-arx4-kind-dicts.mjs | 766 +++++++++++++++++++++++++++ skills/agent-render-linking/SKILL.md | 19 +- 6 files changed, 928 insertions(+), 40 deletions(-) create mode 100644 docs/arx4-kind-dicts.md create mode 100644 scripts/bench-arx4-kind-dicts.mjs diff --git a/docs/arx4-ideation.md b/docs/arx4-ideation.md index 964720f..ffdd015 100644 --- a/docs/arx4-ideation.md +++ b/docs/arx4-ideation.md @@ -142,37 +142,36 @@ bytes via shared dictionaries / better envelopes — not denser Unicode wire or ### Ranked bets for ARX4 -1. **Mosaic / multi-message assembler — top “break the box” bet (see `docs/arx4-silly-cuts.md`)** - - Only lever that *multiplies* Discord’s 2000 budget (N messages ≈ N× capacity). - - N links in *one* message do not help — they share the same 2000 chars. - - Protocol + UX cost (click/`1/3` assembly); not a denser alphabet. - -2. **Content-first / CBOR + real Brotli shared dict — explored (see `docs/arx4-bet2-bench.md`)** - - Implemented: `src/lib/payload/arx4-content-first.ts` + `npm run bench:arx4-bet2`. +1. **Agent-side semantic splits (skill) — preferred over mosaic** + - When Discord budget blows, agents should cut by meaning (sections/files) into + independent links — see `skills/agent-render-linking/SKILL.md`. + - **Mosaic/`1/N` reassembly is out** — not worth the protocol/UX cost. + +2. **Kind-specific dictionaries — measured, not yet (see `docs/arx4-kind-dicts.md`)** + - Free kind tags (`m`/`k`/`j`/…) cost **0** extra chars vs today’s `c`; a +1 selector is optional. + - On this LOO corpus, kind overlays **do not beat** shared ARX3 (Σ BMP **+0.4%** free-tag, + **+0.8%** with +1 selector). Wins were tiny/rare (2/7 fixtures free-tag). + - **Do not spend a char hoping to “save thousands”** — Discord wins look like tens of BMP + chars when they exist; agents should split oversized source instead. + - Revisit only with a larger per-kind held-out corpus. + +3. **Content-first / CBOR + real Brotli shared dict — explored (see `docs/arx4-bet2-bench.md`)** - Residual dict estimates overstated wins (~6–8% corpus); **real `brotli -D` is ~0–1%**. - - Binary envelopes alone are not a Discord unlock; keep as plumbing, not the next capacity bet. - - Browser shared-dict still blocked (`brotli-wasm` has no dict API; Node zlib ignores it). + - Binary envelopes alone are not a Discord unlock; keep as plumbing. -3. **Versioned shared prior (chunk / word) — only with LOO gate** - - Warm/contaminated priors look magical (−70%+) and are fake (content-addressed cache). - - Leave-one-out chunk priors **regressed** this corpus (~+20% BMP) — literals + ID framing lose to ARX3+Brotli. - - Revisit only for a *domain* prior that LOO-beats ARX3 on held-out agent chat; never trust warm numbers. +4. **Versioned shared prior (chunk / word) — only with LOO gate** + - Warm priors are fake (−70%+); LOO chunk priors **regressed** (~+20% BMP). See silly-cuts. -4. **Implied envelope + kind IR + lossy — small honest cuts** +5. **Implied envelope + kind IR + lossy — small honest cuts** - Implied / kind IR ~3–4% corpus; lossy ~8% when stripping emphasis/comments. - - Plumbing / opt-in preview — not a Discord unlock alone. -5. **Hybrid fence / label bitstream — wash** - - Fence stub steals budget; label-as-title costs framing. Skip as density bets; fence may still be a paste UX. +6. **Hybrid fence / label bitstream — wash** -6. **Curated overlay growth (cautious)** - - Alone, mined n-grams *regressed* this corpus (+3–4%). +7. **Curated overlay growth (cautious)** — mined n-grams alone regressed (+3–4%). -7. **baseAstral — deprioritized for Discord** - - UTF-16 client counting → astral loses to baseBMP (~10 vs ~15.92 bits/unit). +8. **baseAstral — deprioritized for Discord** (UTF-16 counting). -8. **Discord framing — already practiced, not an ARX4 lever** - - Short labels are skill default; host shortening is DNS, not codec. +9. **Discord framing — already practiced, not an ARX4 lever** ### Suggested ARX4 shape (if pursued) @@ -180,19 +179,19 @@ bytes via shared dictionaries / better envelopes — not denser Unicode wire or # Single-link path (marginal): artifact bytes → implied / content-first envelope + optional kind IR + → optional kind-tagged dict ONLY if held-out gate clears (prefer free tags m/k/j/…) → Brotli q11 (+ real shared dict only if browser wasm gains dict support) → baseBMP → compact tag -# Break-the-box path (capacity): -artifact bytes → ARX3/ARX4 pack → split into N Discord messages (mosaic 1/N) - OR stub link + ```arx fence (paste UX, not denser wire) +# Over-budget path (product, not codec): +agent semantically splits → N independent ARX3 links (skill guidance) ``` Selection policy: optimize `markdownLink.length` (JS/UTF-16 units) for a declared surface -(`discord` | `discord-mosaic` | `discord-fence` | `visible` | `transport`). +(`discord` | `visible` | `transport`). -Silly-cuts takeaway: **stop chasing alphabet / residual-dict / contaminated priors.** -The real unlock is either a proven LOO prior or accepting multi-message / fence UX. +Takeaway: **stop mosaic; stop paying selector chars without a LOO win; prefer skill splits +and only revisit kind dicts with a bigger corpus.** ## Non-goals / traps @@ -212,9 +211,12 @@ npm run bench:arx4-ideation # Bet #2 follow-up (content-first/CBOR + real brotli -D): npm run bench:arx4-bet2 -# Silly / deep cuts (priors, IR, mosaic, fence): +# Silly / deep cuts (priors, IR, mosaic math — mosaic deprioritized): npm run bench:arx4-silly + +# Kind-specific dictionaries (+ free tag vs +1 selector): +npm run bench:arx4-kind-dicts ``` _Generated in 9.6ms._ -See also `docs/arx4-bet2-bench.md` and `docs/arx4-silly-cuts.md`. +See also `docs/arx4-bet2-bench.md`, `docs/arx4-silly-cuts.md`, and `docs/arx4-kind-dicts.md`. diff --git a/docs/arx4-kind-dicts.md b/docs/arx4-kind-dicts.md new file mode 100644 index 0000000..6627d71 --- /dev/null +++ b/docs/arx4-kind-dicts.md @@ -0,0 +1,104 @@ +# ARX4 kind-specific dictionaries + +_Experimental notes from `scripts/bench-arx4-kind-dicts.mjs`. Not a shipped codec._ + +## The question + +> Worth spending one char to potentially save thousands? + +Two separate questions: + +1. **Do kind-tuned substitution dicts beat the shared ARX3 dict** on held-out (LOO) fixtures? +2. **What does selection cost on the wire?** + +### Tag-cost menu (you often pay **zero**) + +| Selector | Extra fragment chars | Notes | +| --- | ---: | --- | +| **Free kind tags** (`m` md, `k` code, `j` json, `s` csv, `f` diff, …) | **0** | Unused RFC-3986 unreserved tags; same length as today’s `c` | +| `c` + 1 selector digit/byte | **+1** | Only needed if you refuse new tags | +| Infer from envelope kind | **0** | Kind already in tuple — but decode must learn kind *before* reversing kind-substitution (peek or staged decode) | + +So: **do not spend a char unless free tags are off the table.** The interesting bar is +`kindDictBmp < arx3Bmp` (free tag) or `kindDictBmp + 1 < arx3Bmp` (+1 selector). + +“Save thousands” is the wrong unit for Discord: one BMP char ≈ 2 brotli bytes. +A kind dict that saves **tens to hundreds** of BMP chars is already a real Discord win; +thousands of BMP chars would mean megabytes of source, which agents should **split** +semantically (see skill), not mosaic. + +## Method + +- Baseline: ARX3 path (tuple JSON → arx2 overlay → v1 dict → Brotli q11 → baseBMP chars). +- Kind overlay (**extra**): same path, then a third substitution layer from kind seeds + + leave-one-out mined n-grams (max 64 slots, fresh `0x1d` code space). +- Kind replace: keep v1 singles + first 40 extended; replace the long English/JS tail with + kind slots (same slot budget as today). +- LOO: mined patterns never see the measured fixture. + +## Per-fixture results + +| Fixture | kind | raw | ARX3 BMP | kind-extra BMP | kind-replace BMP | best free-tag | vs ARX3 | best +1 sel | vs ARX3 | slots | +| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| markdown-agents | markdown | 8000 | 205 | 207 (+1.0%) | 207 (+1.0%) | 207 | +1.0% | 208 | +1.5% | 64 | +| code-bench-report | markdown | 8314 | 1119 | 1116 (−0.3%) | 1119 (0.0%) | 1116 | −0.3% | 1117 | −0.2% | 64 | +| code-fragment | code | 8000 | 158 | 161 (+1.9%) | 159 (+0.6%) | 159 | +0.6% | 160 | +1.3% | 29 | +| json-package | json | 259 | 95 | 97 (+2.1%) | 102 (+7.4%) | 97 | +2.1% | 98 | +3.2% | 27 | +| csv-leaderboard | csv | 218 | 99 | 105 (+6.1%) | 108 (+9.1%) | 105 | +6.1% | 106 | +7.1% | 7 | +| diff-fragment | diff | 409 | 110 | 109 (−0.9%) | 109 (−0.9%) | 109 | −0.9% | 110 | 0.0% | 12 | +| small-markdown | markdown | 189 | 74 | 74 (0.0%) | 74 (0.0%) | 74 | 0.0% | 75 | +1.4% | 64 | + +## Corpus totals + +| Variant | Σ BMP | vs ARX3 | +| --- | ---: | ---: | +| ARX3 (shared dict) | 1860 | 0.0% | +| Kind dict, free tag | 1867 | +0.4% | +| Kind dict, +1 selector | 1874 | +0.8% | + +Fixtures where free-tag kind dict beats ARX3: **2/7** +Fixtures where +1 selector still beats ARX3: **1/7** +Σ BMP chars saved (free tag): **-7** (+0.4%) +Σ BMP chars saved (+1 sel): **-14** (+0.8%) + +## Sample kind slots (first 8, LOO) + +- **markdown-agents** (markdown, 64 slots): `"\n## "`, `"\n### "`, `"\n#### "`, `"\n- "`, `"\n* "`, `"\n1. "`, `"\n2. "`, `"\n3. "` +- **code-bench-report** (markdown, 64 slots): `"\n## "`, `"\n### "`, `"\n#### "`, `"\n- "`, `"\n* "`, `"\n1. "`, `"\n2. "`, `"\n3. "` +- **code-fragment** (code, 29 slots): `"export async function "`, `"export default "`, `"import type "`, `" from \""`, `"type "`, `"=> {"`, `"): "`, `"?: "` +- **json-package** (json, 27 slots): `"\"name\":"`, `"\"version\":"`, `"\"private\":"`, `"\"scripts\":"`, `"\"dependencies\":"`, `"\"devDependencies\":"`, `"\"description\":"`, `"\"license\":"` +- **csv-leaderboard** (csv, 7 slots): `","`, `"\n"`, `"\""`, `"0,"`, `"1,"`, `"2,"`, `"3,"` +- **diff-fragment** (diff, 12 slots): `"--- a/"`, `"+++ b/"`, `"@@ -"`, `"\n+"`, `"\n-"`, `"\n "`, `"index "`, `"new file mode "` +- **small-markdown** (markdown, 64 slots): `"\n## "`, `"\n### "`, `"\n#### "`, `"\n- "`, `"\n* "`, `"\n1. "`, `"\n2. "`, `"\n3. "` + +## Verdict + +### Is +1 char worth it? + +**Not on this corpus.** Kind dicts do not clearly beat shared ARX3 after LOO (Σ free-tag Δ +0.4%). Do not spend a selector char; do not ship kind tags yet without a larger held-out gate. + +### Practical recommendation + +1. **Prefer free kind tags over a +1 selector** if kind dicts ever clear a held-out gate. + One char of framing is ~2 brotli bytes — tiny, but free is free, and unused tags exist. +2. **Do not expect “thousands” of chars saved** from dict specialization on Discord-sized + payloads. Wins look like **tens of BMP chars** on kind-homogeneous artifacts, when they win. +3. **Agents should split oversized artifacts semantically** (skill guidance) — separate report + sections / files — not mosaic reassembly protocols. +4. Next measurement gate: larger per-kind held-out corpus (real agent markdown vs TS vs + package.json vs unified diffs). This bench’s LOO set is thin for csv/diff/json. + +## Non-goals + +- Not shipping kind tags or new dictionaries in this pass. +- Not updating AGENTS.md as if ARX4 ships. +- Not reviving mosaic assemblers. + +## How to re-run + +```bash +npm run bench:arx4-kind-dicts +# or: node scripts/bench-arx4-kind-dicts.mjs +``` + +_Generated in 169.1ms._ diff --git a/docs/arx4-silly-cuts.md b/docs/arx4-silly-cuts.md index 61f515a..05e6d45 100644 --- a/docs/arx4-silly-cuts.md +++ b/docs/arx4-silly-cuts.md @@ -173,15 +173,17 @@ this is the hard case, not the highly-repetitive search-cap case. ### Ranked silly bets (after this probe) -1. **Mosaic assembler** — explicit multi-message `1/3` protocol; multiplies capacity for real. +1. **Mosaic assembler** — **deprioritized** (prefer agent semantic splits via skill). 2. **Versioned shared prior** — only if LOO / held-out benches still win on *your* domain corpus (agent-render chat is repetitive; arbitrary user prose is not). Pair with a pinned mother post or build-shipped prior — never trust warm numbers. -3. **Hybrid fence profile** — optional surface for agents that already paste code blocks; +3. **Kind-specific dicts** — free tags preferred over +1 selector; needs larger LOO corpus + (current probe did not beat ARX3 — see `docs/arx4-kind-dicts.md`). +4. **Hybrid fence profile** — optional surface for agents that already paste code blocks; protocol bend, not a density win. -4. **Implied + kind IR** — cheap plumbing alongside (1) or (2). -5. **Lossy mode** — opt-in “readable enough” chat previews. -6. Avoid more residual-Brotli-dict optimism, alphabet retunes, and contaminated prior benches. +5. **Implied + kind IR** — cheap plumbing alongside a real prior story. +6. **Lossy mode** — opt-in “readable enough” chat previews. +7. Avoid more residual-Brotli-dict optimism, alphabet retunes, and contaminated prior benches. ## Non-goals reinforced diff --git a/package.json b/package.json index 0a726a0..bdb6c41 100644 --- a/package.json +++ b/package.json @@ -46,7 +46,8 @@ "selfhosted:dev": "node --import tsx selfhosted/server.ts", "selfhosted:build": "tsc -p selfhosted/tsconfig.json", "selfhosted:start": "node selfhosted/dist/server.js", - "selfhosted:csp-smoke": "node scripts/csp-smoke.mjs" + "selfhosted:csp-smoke": "node scripts/csp-smoke.mjs", + "bench:arx4-kind-dicts": "node scripts/bench-arx4-kind-dicts.mjs" }, "dependencies": { "@codemirror/lang-css": "^6.3.1", diff --git a/scripts/bench-arx4-kind-dicts.mjs b/scripts/bench-arx4-kind-dicts.mjs new file mode 100644 index 0000000..825289b --- /dev/null +++ b/scripts/bench-arx4-kind-dicts.mjs @@ -0,0 +1,766 @@ +#!/usr/bin/env node +/** + * ARX4 kind-specific dictionary probe. + * + * Question: is a per-kind substitution dict worth it — and do we need to spend + * an extra fragment char to select it? + * + * Tag-cost options: + * free tags — unused compact tags (m/k/j/s/f) select kind dict → +0 chars + * +1 selector — keep `c` and add one BMP digit / ASCII selector → +1 char + * infer kind — kind already in envelope; decoder peeks then picks dict → +0 + * (harder: substitution runs *before* JSON parse today) + * + * Experimental only. Writes `docs/arx4-kind-dicts.md`. + */ + +import { readFileSync, writeFileSync } from "node:fs"; +import { brotliCompressSync, constants } from "node:zlib"; +import { performance } from "node:perf_hooks"; + +const REPORT_PATH = "docs/arx4-kind-dicts.md"; +const BMP_BASE_SIZE = 62_000; +const DISCORD_MESSAGE_MAX = 2000; + +const v1Dictionary = JSON.parse(readFileSync("public/arx-dictionary.json", "utf8")); +const overlayDictionary = JSON.parse(readFileSync("public/arx2-dictionary.json", "utf8")); +const codeBenchReportFixture = readFileSync("tests/fixtures/baanish-code-bench-report.md", "utf8"); + +const singleByteCodes = [ + 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x0b, 0x0e, 0x0f, 0x10, 0x11, 0x12, 0x13, 0x14, 0x15, + 0x16, 0x17, 0x18, 0x19, 0x1a, 0x1b, 0x1c, 0x1d, +]; + +function buildPairs(dict, singleCodes = singleByteCodes, extendedPrefix = "\x00", extendedOffset = 1) { + const pairs = []; + for (let i = 0; i < dict.singleByteSlots.length && i < singleCodes.length; i++) { + pairs.push([dict.singleByteSlots[i], String.fromCharCode(singleCodes[i])]); + } + for (let i = 0; i < dict.extendedSlots.length; i++) { + pairs.push([dict.extendedSlots[i], extendedPrefix + String.fromCharCode(i + extendedOffset)]); + } + return pairs; +} + +function buildTrie(pairs) { + const root = { children: new Map() }; + for (const [from, to] of pairs) { + let node = root; + for (const char of from) { + let child = node.children.get(char); + if (!child) { + child = { children: new Map() }; + node.children.set(char, child); + } + node = child; + } + node.replacement ??= to; + } + return root; +} + +function applyTrie(text, trie) { + const out = []; + let index = 0; + while (index < text.length) { + let node = trie; + let cursor = index; + let replacement; + let replacementLength = 0; + while (cursor < text.length) { + node = node.children.get(text[cursor]); + if (!node) break; + cursor++; + if (node.replacement !== undefined) { + replacement = node.replacement; + replacementLength = cursor - index; + } + } + if (replacement !== undefined) { + out.push(replacement); + index += replacementLength; + } else { + out.push(text[index]); + index++; + } + } + return out.join(""); +} + +const v1Pairs = buildPairs(v1Dictionary); +const overlayPairs = buildPairs(overlayDictionary, [0x1e, 0x7f], "\x1f", 0x20); +const v1EncodeTrie = buildTrie(v1Pairs); +const overlayEncodeTrie = buildTrie(overlayPairs); + +function brotli(input) { + return brotliCompressSync(Buffer.from(input), { + params: { [constants.BROTLI_PARAM_QUALITY]: 11 }, + }); +} + +function baseBmpChars(byteLength) { + return 3 + Math.ceil((byteLength * 8) / Math.log2(BMP_BASE_SIZE)); +} + +function trimOptional(fields) { + let end = fields.length; + while (end > 0 && (fields[end - 1] === undefined || fields[end - 1] === null)) end -= 1; + return fields.slice(0, end); +} + +function artifactTuple(artifact) { + switch (artifact.kind) { + case "markdown": + return trimOptional(["m", artifact.id, artifact.content, artifact.title, artifact.filename]); + case "code": + return trimOptional(["c", artifact.id, artifact.content, artifact.language, artifact.title, artifact.filename]); + case "csv": + return trimOptional(["s", artifact.id, artifact.content, artifact.title, artifact.filename]); + case "json": + return trimOptional(["j", artifact.id, artifact.content, artifact.title, artifact.filename]); + case "diff": + return trimOptional([ + "d", + artifact.id, + artifact.patch, + artifact.oldContent, + artifact.newContent, + artifact.language, + artifact.view, + artifact.title, + artifact.filename, + ]); + default: + throw new Error(`unsupported kind ${artifact.kind}`); + } +} + +function tupleEnvelope(envelope) { + const artifacts = envelope.artifacts.map(artifactTuple); + if (artifacts.length === 1) return trimOptional([3, artifacts[0], envelope.title]); + return trimOptional([2, artifacts, envelope.title]); +} + +function encodeArx3Substituted(envelope) { + const tupleJson = JSON.stringify(tupleEnvelope({ ...envelope, codec: "arx3" })); + return applyTrie(applyTrie(tupleJson, overlayEncodeTrie), v1EncodeTrie); +} + +function textEnvelope(kind, title, content, extra = {}) { + return { + v: 1, + codec: "plain", + title, + activeArtifactId: "a", + artifacts: [{ id: "a", kind, title, filename: extra.filename ?? "artifact.txt", content, ...extra }], + }; +} + +function repeatedFixture(block, targetLength, segmentSuffix = (index) => `\nfixture segment ${index}\n`) { + let fixture = ""; + let index = 0; + while (fixture.length < targetLength) { + fixture += `${block}${segmentSuffix(index)}`; + index++; + } + return Array.from(fixture).slice(0, targetLength).join(""); +} + +/** Mine frequent n-grams; prefer longer × count. Skip anything already in base dicts. */ +function mineNgrams(texts, { minLen = 4, maxLen = 48, topN = 80, exclude = new Set() } = {}) { + const counts = new Map(); + for (const text of texts) { + if (!text) continue; + for (let len = minLen; len <= maxLen; len++) { + const step = Math.max(1, Math.floor(len / 4)); + for (let i = 0; i + len <= text.length; i += step) { + const gram = text.slice(i, i + len); + if (exclude.has(gram)) continue; + // Prefer printable / structured tokens + if (/[\x00-\x08\x0b\x0e-\x1f]/.test(gram)) continue; + counts.set(gram, (counts.get(gram) || 0) + 1); + } + } + } + return [...counts.entries()] + .filter(([, n]) => n >= 2) + .map(([g, n]) => ({ g, n, score: n * g.length })) + .sort((a, b) => b.score - a.score) + .slice(0, topN) + .map((x) => x.g); +} + +/** Curated seed patterns per kind (domain knowledge, not just mining). */ +const KIND_SEEDS = { + markdown: [ + "\n## ", + "\n### ", + "\n#### ", + "\n- ", + "\n* ", + "\n1. ", + "\n2. ", + "\n3. ", + "\n```", + "```\n", + "\n> ", + "\n| ", + " | ", + " |\n", + "| ---", + "---|", + "**", + "__", + "](http", + "](https://", + "![", + "---\n", + "\n\n", + "```ts", + "```js", + "```json", + "```bash", + "```mermaid", + "graph TD", + "flowchart ", + ], + code: [ + "export function ", + "export async function ", + "export const ", + "export default ", + "import { ", + "import type ", + '} from "', + ' from "', + "return ", + "async ", + "await ", + "const ", + "function ", + "interface ", + "type ", + "extends ", + "implements ", + "typeof ", + "instanceof ", + "undefined", + "null", + "true", + "false", + "=> {", + "): ", + "?: ", + " as ", + "useState", + "useEffect", + "useCallback", + "useMemo", + "useRef", + "className", + "console.", + "document.", + "window.", + "Promise<", + "Record<", + "Array<", + "string", + "number", + "boolean", + "void", + "throw new ", + "try {", + "catch (", + "if (!", + "if (", + "} else {", + "for (const ", + "for (let ", + "while (", + "switch (", + "case ", + "break;", + "continue;", + "=== ", + "!== ", + "&& ", + "|| ", + ], + json: [ + '"name":', + '"version":', + '"private":', + '"scripts":', + '"dependencies":', + '"devDependencies":', + '"description":', + '"license":', + '"main":', + '"type":', + '"exports":', + '"imports":', + '"engines":', + '"repository":', + '"keywords":', + '"author":', + '"homepage":', + '"bugs":', + "true", + "false", + "null", + '": "', + '": {', + '": [', + '"},', + '"],', + "{\n", + "},\n", + "[\n", + "],\n", + ], + csv: [",", "\n", '"', "true", "false", "null", "0,", "1,", "2,", "3,"], + diff: [ + "diff --git ", + "--- a/", + "+++ b/", + "@@ -", + "\n+", + "\n-", + "\n ", + "index ", + "new file mode ", + "deleted file mode ", + "similarity index ", + "rename from ", + "rename to ", + ], +}; + +// Existing dict strings — don't re-mine them into kind overlays +const existingSlots = new Set([ + ...v1Dictionary.singleByteSlots, + ...v1Dictionary.extendedSlots, + ...overlayDictionary.singleByteSlots, + ...overlayDictionary.extendedSlots, +]); + +const markdownAgentsFixture = repeatedFixture( + [ + "# AGENTS.md excerpt", + "", + "`agent-render` is a static artifact viewer for AI-generated outputs.", + "Keep markdown, code, diffs, CSV, and JSON readable across chat surfaces.", + "", + "## Product contract", + "", + "- Fragment payloads use `#agent-render=v1..`.", + "- Artifact contents stay out of the host request path.", + "- Supported codecs are `plain`, `lz`, `deflate`, `arx`, `arx2`, and `arx3`.", + "- Supported artifact kinds are `markdown`, `code`, `diff`, `csv`, and `json`.", + "", + "Preserve the static shell, the zero-retention wording, and the renderer-first layout.", + "", + ].join("\n"), + 8000, + (index) => + `\nFixture note ${index}: fragment transport, renderer readiness, and artifact metadata stay aligned.\n\n`, +); + +const codeFragmentFixture = repeatedFixture( + [ + "export async function decodeFragmentAsync(hash: string, options?: DecodeOptions) {", + " const parsed = parseFragmentPrefix(hash);", + " if (!parsed.ok) return parsed;", + ' if (parsed.codec === "arx" || parsed.codec === "arx2") {', + ' const { decodeArxFragmentAsync } = await import("./fragment-arx");', + " return decodeArxFragmentAsync(parsed, options);", + " }", + " return decodePlainFragment(parsed.payload, options);", + "}", + "", + ].join("\n"), + 8000, + (index) => `\n// fixture segment ${index}: codec branch coverage and bundle shape stay stable.\n`, +); + +const packageManifestFixture = JSON.stringify( + { + name: "agent-render", + version: "0.1.0", + private: true, + scripts: { build: "next build", check: "npm run lint && npm run test" }, + dependencies: { next: "15.1.11", react: "19.1.0", "brotli-wasm": "^3.0.1" }, + }, + null, + 2, +); + +const csvFixture = [ + "model,avg,delta,tasks,wins", + "gpt-5.5,93.91,0,25,7", + "mimo/mimo-v2.5-pro,93.29,-0.61,25,5", + "Kimi-K2.6-Turbo,91.63,-2.28,25,5", + "synthetic/GLM-5.1,89.07,-4.84,25,5", + "gpt-5.3-codex-spark,88.68,-5.23,25,3", + "gpt-5.4-mini,88.19,-5.72,25,3", +].join("\n"); + +const diffFixture = [ + "diff --git a/src/lib/payload/fragment.ts b/src/lib/payload/fragment.ts", + "index 1111111..2222222 100644", + "--- a/src/lib/payload/fragment.ts", + "+++ b/src/lib/payload/fragment.ts", + "@@ -10,8 +10,12 @@ export function encodeEnvelope(envelope) {", + " const json = JSON.stringify(envelope);", + "- return encodeDeflate(json);", + "+ if (options?.codec === 'arx3') {", + "+ return encodeArx3(json);", + "+ }", + "+ return encodeDeflate(json);", + " }", + "", +].join("\n"); + +const corpus = [ + { + name: "markdown-agents", + kind: "markdown", + envelope: textEnvelope("markdown", "AGENTS.md excerpt", markdownAgentsFixture, { filename: "AGENTS.md" }), + }, + { + name: "code-bench-report", + kind: "markdown", + envelope: textEnvelope("markdown", "Baanish Code Bench", codeBenchReportFixture, { filename: "results.md" }), + }, + { + name: "code-fragment", + kind: "code", + envelope: textEnvelope("code", "fragment.ts excerpt", codeFragmentFixture, { + filename: "fragment.ts", + language: "ts", + }), + }, + { + name: "json-package", + kind: "json", + envelope: textEnvelope("json", "package.json", packageManifestFixture, { filename: "package.json" }), + }, + { + name: "csv-leaderboard", + kind: "csv", + envelope: textEnvelope("csv", "Leaderboard", csvFixture, { filename: "board.csv" }), + }, + { + name: "diff-fragment", + kind: "diff", + envelope: { + v: 1, + codec: "plain", + title: "fragment.ts patch", + activeArtifactId: "a", + artifacts: [ + { + id: "a", + kind: "diff", + title: "fragment.ts patch", + filename: "fragment.ts", + patch: diffFixture, + view: "unified", + }, + ], + }, + }, + { + name: "small-markdown", + kind: "markdown", + envelope: textEnvelope( + "markdown", + "Note", + [ + "# Sprint notes", + "", + "- Ship ARX3 visible URL mode", + "- Keep Discord markdown links under 2000 chars", + "- Prefer fragment transport over UUID mode for zero-retention", + "", + "```ts", + "export const value = 1;", + "```", + "", + ].join("\n"), + { filename: "notes.md" }, + ), + }, +]; + +function contentOf(envelope) { + const a = envelope.artifacts[0]; + return a.content ?? a.patch ?? ""; +} + +/** Build a kind overlay: seeds + LOO-mined n-grams, capped to fit extended-slot budget. */ +function buildKindOverlay(kind, holdOutIndex, { maxSlots = 64 } = {}) { + const siblings = corpus + .map((row, i) => ({ row, i })) + .filter(({ row, i }) => row.kind === kind && i !== holdOutIndex) + .map(({ row }) => contentOf(row.envelope)); + + // Also allow cross-fixture same-kind mining; if only one fixture of that kind, + // fall back to seeds only (honest cold kind dict). + const mined = + siblings.length > 0 + ? mineNgrams(siblings, { topN: maxSlots, exclude: existingSlots }) + : []; + + const seeds = (KIND_SEEDS[kind] ?? []).filter((s) => !existingSlots.has(s)); + const merged = []; + const seen = new Set(); + for (const g of [...seeds, ...mined]) { + if (seen.has(g) || existingSlots.has(g)) continue; + // Drop patterns already covered as substrings of longer kept patterns? keep simple. + seen.add(g); + merged.push(g); + if (merged.length >= maxSlots) break; + } + return merged; +} + +/** + * Kind-dict encode path: + * tuple JSON → arx2 overlay → v1 dict → kind overlay → brotli + * Kind overlay uses a fresh code space (0x1d prefix + index) so it doesn't + * collide with v1 (0x00) or arx2 (0x1f) extended prefixes. + */ +function encodeWithKindOverlay(envelope, kindSlots) { + const base = encodeArx3Substituted(envelope); + if (!kindSlots.length) return base; + const kindPairs = kindSlots.map((slot, i) => [slot, "\x1d" + String.fromCharCode(i + 0x20)]); + const kindTrie = buildTrie(kindPairs); + return applyTrie(base, kindTrie); +} + +/** + * Replace-mode: rebuild v1-like dict where extended slots are kind-biased. + * Keeps v1 single-byte envelope chrome; swaps the long English/JS tail for kind slots. + */ +function encodeWithReplacedExtended(envelope, kindSlots) { + const keepSingles = v1Dictionary.singleByteSlots; + // Keep first ~40 extended (view/diff/js core), replace the rest with kind slots + const keepExt = v1Dictionary.extendedSlots.slice(0, 40); + const replaced = [...keepExt]; + for (const slot of kindSlots) { + if (replaced.includes(slot)) continue; + replaced.push(slot); + if (replaced.length >= v1Dictionary.extendedSlots.length) break; + } + // Pad with leftovers from original if short + for (const slot of v1Dictionary.extendedSlots) { + if (replaced.length >= v1Dictionary.extendedSlots.length) break; + if (!replaced.includes(slot)) replaced.push(slot); + } + const customDict = { singleByteSlots: keepSingles, extendedSlots: replaced }; + const customPairs = buildPairs(customDict); + const customTrie = buildTrie(customPairs); + const tupleJson = JSON.stringify(tupleEnvelope({ ...envelope, codec: "arx3" })); + return applyTrie(applyTrie(tupleJson, overlayEncodeTrie), customTrie); +} + +function pack(raw) { + const bytes = brotli(raw).length; + return { brotliBytes: bytes, bmpChars: baseBmpChars(bytes) }; +} + +function pct(from, to) { + if (from == null || to == null) return null; + return ((to - from) / from) * 100; +} + +function fmtDelta(from, to) { + const p = pct(from, to); + if (p == null || Number.isNaN(p)) return "n/a"; + if (p === 0) return "0.0%"; + return `${p < 0 ? "−" : "+"}${Math.abs(p).toFixed(1)}%`; +} + +function discordFraming(tag = "c") { + const open = `[Artifact](https://agent-render.com#${tag}`; + return { + framingLength: open.length + 1, + payloadBudget: DISCORD_MESSAGE_MAX - open.length - 1, + }; +} + +const t0 = performance.now(); +const rows = corpus.map((item, index) => { + const kind = item.kind; + const kindSlots = buildKindOverlay(kind, index, { maxSlots: 64 }); + const arx3 = pack(encodeArx3Substituted(item.envelope)); + const kindExtra = pack(encodeWithKindOverlay(item.envelope, kindSlots)); + const kindReplace = pack(encodeWithReplacedExtended(item.envelope, kindSlots)); + + // Tag cost models on Discord markdown-link length (BMP chars ≈ fragment payload chars for baseBMP) + const freeTagBmp = kindExtra.bmpChars; // tag `m` same length as `c` + const plusOneBmp = kindExtra.bmpChars + 1; // selector after `c` + const freeTagReplace = kindReplace.bmpChars; + const plusOneReplace = kindReplace.bmpChars + 1; + + const bestKindBmp = Math.min(freeTagBmp, freeTagReplace); + const bestKindPlusOne = Math.min(plusOneBmp, plusOneReplace); + const beatsArx3Free = bestKindBmp < arx3.bmpChars; + const beatsArx3PlusOne = bestKindPlusOne < arx3.bmpChars; + const charsSavedFree = arx3.bmpChars - bestKindBmp; + const charsSavedPlusOne = arx3.bmpChars - bestKindPlusOne; + + return { + name: item.name, + kind, + rawChars: contentOf(item.envelope).length, + kindSlotCount: kindSlots.length, + kindSlotSample: kindSlots.slice(0, 8), + arx3, + kindExtra, + kindReplace, + freeTagBmp: bestKindBmp, + plusOneBmp: bestKindPlusOne, + beatsArx3Free, + beatsArx3PlusOne, + charsSavedFree, + charsSavedPlusOne, + }; +}); +const elapsed = performance.now() - t0; + +const sum = (sel) => rows.reduce((n, r) => n + sel(r), 0); +const arx3Bmp = sum((r) => r.arx3.bmpChars); +const freeBmp = sum((r) => r.freeTagBmp); +const plusOneBmp = sum((r) => r.plusOneBmp); + +const lines = []; +const w = (s = "") => lines.push(s); + +w("# ARX4 kind-specific dictionaries"); +w(); +w("_Experimental notes from `scripts/bench-arx4-kind-dicts.mjs`. Not a shipped codec._"); +w(); +w("## The question"); +w(); +w("> Worth spending one char to potentially save thousands?"); +w(); +w("Two separate questions:"); +w(); +w("1. **Do kind-tuned substitution dicts beat the shared ARX3 dict** on held-out (LOO) fixtures?"); +w("2. **What does selection cost on the wire?**"); +w(); +w("### Tag-cost menu (you often pay **zero**)"); +w(); +w("| Selector | Extra fragment chars | Notes |"); +w("| --- | ---: | --- |"); +w("| **Free kind tags** (`m` md, `k` code, `j` json, `s` csv, `f` diff, …) | **0** | Unused RFC-3986 unreserved tags; same length as today’s `c` |"); +w("| `c` + 1 selector digit/byte | **+1** | Only needed if you refuse new tags |"); +w("| Infer from envelope kind | **0** | Kind already in tuple — but decode must learn kind *before* reversing kind-substitution (peek or staged decode) |"); +w(); +w("So: **do not spend a char unless free tags are off the table.** The interesting bar is"); +w("`kindDictBmp < arx3Bmp` (free tag) or `kindDictBmp + 1 < arx3Bmp` (+1 selector)."); +w(); +w("“Save thousands” is the wrong unit for Discord: one BMP char ≈ 2 brotli bytes."); +w("A kind dict that saves **tens to hundreds** of BMP chars is already a real Discord win;"); +w("thousands of BMP chars would mean megabytes of source, which agents should **split**"); +w("semantically (see skill), not mosaic."); +w(); + +w("## Method"); +w(); +w("- Baseline: ARX3 path (tuple JSON → arx2 overlay → v1 dict → Brotli q11 → baseBMP chars)."); +w("- Kind overlay (**extra**): same path, then a third substitution layer from kind seeds +"); +w(" leave-one-out mined n-grams (max 64 slots, fresh `0x1d` code space)."); +w("- Kind replace: keep v1 singles + first 40 extended; replace the long English/JS tail with"); +w(" kind slots (same slot budget as today)."); +w("- LOO: mined patterns never see the measured fixture."); +w(); + +w("## Per-fixture results"); +w(); +w( + "| Fixture | kind | raw | ARX3 BMP | kind-extra BMP | kind-replace BMP | best free-tag | vs ARX3 | best +1 sel | vs ARX3 | slots |", +); +w("| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |"); +for (const r of rows) { + w( + `| ${r.name} | ${r.kind} | ${r.rawChars} | ${r.arx3.bmpChars} | ${r.kindExtra.bmpChars} (${fmtDelta(r.arx3.bmpChars, r.kindExtra.bmpChars)}) | ${r.kindReplace.bmpChars} (${fmtDelta(r.arx3.bmpChars, r.kindReplace.bmpChars)}) | ${r.freeTagBmp} | ${fmtDelta(r.arx3.bmpChars, r.freeTagBmp)} | ${r.plusOneBmp} | ${fmtDelta(r.arx3.bmpChars, r.plusOneBmp)} | ${r.kindSlotCount} |`, + ); +} + +w(); +w("## Corpus totals"); +w(); +w("| Variant | Σ BMP | vs ARX3 |"); +w("| --- | ---: | ---: |"); +w(`| ARX3 (shared dict) | ${arx3Bmp} | 0.0% |`); +w(`| Kind dict, free tag | ${freeBmp} | ${fmtDelta(arx3Bmp, freeBmp)} |`); +w(`| Kind dict, +1 selector | ${plusOneBmp} | ${fmtDelta(arx3Bmp, plusOneBmp)} |`); +w(); +w(`Fixtures where free-tag kind dict beats ARX3: **${rows.filter((r) => r.beatsArx3Free).length}/${rows.length}**`); +w(`Fixtures where +1 selector still beats ARX3: **${rows.filter((r) => r.beatsArx3PlusOne).length}/${rows.length}**`); +w(`Σ BMP chars saved (free tag): **${arx3Bmp - freeBmp}** (${fmtDelta(arx3Bmp, freeBmp)})`); +w(`Σ BMP chars saved (+1 sel): **${arx3Bmp - plusOneBmp}** (${fmtDelta(arx3Bmp, plusOneBmp)})`); +w(); + +w("## Sample kind slots (first 8, LOO)"); +w(); +for (const r of rows) { + w(`- **${r.name}** (${r.kind}, ${r.kindSlotCount} slots): ${r.kindSlotSample.map((s) => `\`${JSON.stringify(s)}\``).join(", ") || "_(seeds only / empty)_"}`); +} + +w(); +w("## Verdict"); +w(); +w("### Is +1 char worth it?"); +w(); +if (arx3Bmp - plusOneBmp > 10) { + w( + `On this corpus, **yes if you somehow cannot use free tags** — net Σ save is **${arx3Bmp - plusOneBmp} BMP chars** after paying +1/fixture. But prefer free tags first.`, + ); +} else if (arx3Bmp - freeBmp > 10) { + w( + `**Only with free tags.** Free-tag kind dicts save **${arx3Bmp - freeBmp} BMP chars** corpus-wide; after a +1 selector the net is **${arx3Bmp - plusOneBmp}** — not worth a dedicated selector char.`, + ); +} else { + w( + `**Not on this corpus.** Kind dicts do not clearly beat shared ARX3 after LOO (Σ free-tag Δ ${fmtDelta(arx3Bmp, freeBmp)}). Do not spend a selector char; do not ship kind tags yet without a larger held-out gate.`, + ); +} +w(); +w("### Practical recommendation"); +w(); +w("1. **Prefer free kind tags over a +1 selector** if kind dicts ever clear a held-out gate."); +w(" One char of framing is ~2 brotli bytes — tiny, but free is free, and unused tags exist."); +w("2. **Do not expect “thousands” of chars saved** from dict specialization on Discord-sized"); +w(" payloads. Wins look like **tens of BMP chars** on kind-homogeneous artifacts, when they win."); +w("3. **Agents should split oversized artifacts semantically** (skill guidance) — separate report"); +w(" sections / files — not mosaic reassembly protocols."); +w("4. Next measurement gate: larger per-kind held-out corpus (real agent markdown vs TS vs"); +w(" package.json vs unified diffs). This bench’s LOO set is thin for csv/diff/json."); +w(); +w("## Non-goals"); +w(); +w("- Not shipping kind tags or new dictionaries in this pass."); +w("- Not updating AGENTS.md as if ARX4 ships."); +w("- Not reviving mosaic assemblers."); +w(); +w("## How to re-run"); +w(); +w("```bash"); +w("npm run bench:arx4-kind-dicts"); +w("# or: node scripts/bench-arx4-kind-dicts.mjs"); +w("```"); +w(); +w(`_Generated in ${elapsed.toFixed(1)}ms._`); + +writeFileSync(REPORT_PATH, lines.join("\n") + "\n"); +console.log(`Wrote ${REPORT_PATH}`); +console.log( + `ARX3 Σ ${arx3Bmp} → free-tag ${freeBmp} (${fmtDelta(arx3Bmp, freeBmp)}), +1sel ${plusOneBmp} (${fmtDelta(arx3Bmp, plusOneBmp)})`, +); +console.log( + `beats free ${rows.filter((r) => r.beatsArx3Free).length}/${rows.length}, +1 ${rows.filter((r) => r.beatsArx3PlusOne).length}/${rows.length}`, +); diff --git a/skills/agent-render-linking/SKILL.md b/skills/agent-render-linking/SKILL.md index 47912b9..4add7ec 100644 --- a/skills/agent-render-linking/SKILL.md +++ b/skills/agent-render-linking/SKILL.md @@ -271,7 +271,7 @@ Respect these limits: - target decoded payload budget: about 200,000 characters - Discord message limit for a single markdown link: 2,000 characters total for the formatted `[label](url)` string -Before sharing on Discord, check `markdownLinkLength` or `discordMarkdownLinkWarning` from the link helpers. If you formatted the link yourself, use `buildMarkdownLinkShareInfo(label, url)` and inspect `discordWarning`. When the warning is non-null, split the bundle into smaller artifacts and send separate markdown links in multiple Discord messages instead of one oversized link. +Before sharing on Discord, check `markdownLinkLength` or `discordMarkdownLinkWarning` from the link helpers. If you formatted the link yourself, use `buildMarkdownLinkShareInfo(label, url)` and inspect `discordWarning`. When the warning is non-null, **semantically split the work into smaller artifacts** and send separate markdown links in multiple Discord messages — do not invent a multi-part “mosaic” reassembly protocol for one logical artifact. When generating links programmatically via `createGeneratedArtifactLink` / `createGeneratedArtifactLinkAsync`, send `markdownLink` verbatim and inspect `discordMarkdownLinkWarning`. When it is non-null, surface that warning to the caller and split the payload before sharing on Discord. @@ -280,7 +280,20 @@ If a link is getting too large: 2. allow packed wire mode 3. trim unnecessary prose or metadata 4. prefer a focused artifact over a bloated one -5. return a structured failure when the payload cannot fit the requested budget +5. **split by meaning** (see below) and send multiple independent links +6. return a structured failure when the payload cannot fit the requested budget + +### How to split oversized artifacts (agent-side) + +Prefer human-useful cuts over byte-slicing one blob into `1/N` fragments: + +- **Markdown:** one link per major section (`##`), appendix, or table — e.g. Executive Summary, Leaderboard, Model Scorecards +- **Code:** one link per file or coherent module; do not glue unrelated files into one envelope +- **Diff:** one link per file hunk set, or keep a focused patch instead of a mega-diff +- **CSV / JSON:** one link per logical table/document; truncate rows only when the caller accepts a preview +- **Bundles:** use multiple single-artifact links with clear labels (`[Report — summary](…)`, `[Report — tables](…)`) rather than one over-budget bundle + +Each link must decode on its own. Do not require the viewer to stitch partial payloads. ## Agent budget mode @@ -304,7 +317,7 @@ Prefer standard Markdown links. When you have `markdownLink` from the link helpe [Short summary](https://agent-render.com/#) ``` -Check `markdownLinkLength` or `discordMarkdownLinkWarning` before sending. Discord rejects messages longer than 2,000 characters, so a single `[label](url)` string that exceeds that limit will probably fail. When it does, split the artifact into smaller bundles and send multiple markdown links across separate Discord messages. +Check `markdownLinkLength` or `discordMarkdownLinkWarning` before sending. Discord rejects messages longer than 2,000 characters, so a single `[label](url)` string that exceeds that limit will probably fail. When it does, split by meaning into smaller artifacts (sections/files) and send multiple independent markdown links across separate Discord messages — not a mosaic/`1/N` reassembly scheme. Examples: - `[Weekly report](https://agent-render.com/#)`