research: stage frozen ARX6 codec, conformance tests, and production-baseline harness

docs/arx6-research.md

@@ -0,0 +1,103 @@
+# ARX6 research import and deployment gates
+
+Base revision: `72fb152e0cf92a11ff658c3b0dd3916b387f3e98`.
+
+## Scope of this draft
+
+This PR imports the executable, frozen general ARX6 codec, tests, and benchmark
+glue under `experiments/arx6/`. **It does not implement viewer integration.**
+The application protocol, default auto pool, UI, existing assets, dependencies,
+and legacy decoders are unchanged. Experimental links are not yet shareable
+through the deployed viewer. There are no server uploads or per-artifact fetches.
+
+## Mechanism
+
+The existing tuple pipeline JSON-escapes body newlines before the context mixer
+sees them; substitutions can also create control bytes unrelated to source rows.
+ARX6 codes a native tuple with raw WTF-8 strings instead. That preserves exact JS
+strings, including lone surrogates, while exposing body rows to the existing
+column expert. It adds causal previous-word / character-class contexts, a
+syntax-conditioned residual mixer, and four-byte rather than seven-byte match
+discovery. Direct low-order tables offset part of the additional allocation.
+
+The wire is `#g1L<prior-id><base64url>`. All coding decisions use deterministic
+integer arithmetic. It reuses the existing three 16 KiB curated priors, verifies
+their SHA-256 identities, and copies caller-owned buffers at installation. A
+CRC32 over the native frame is included in measured link lengths. No target
+artifact is added to the priors.
+
+## Historical measurements, not new production-baseline results
+
+The original development set had 40 artifacts from 18 source groups. The model
+was then frozen before evaluation on 93 artifacts from 47 disjoint source groups.
+The source groups and artifact kinds are software-text proxies, not a measured
+sample of actual agent traffic; samples within groups are correlated.
+
+The historical baseline was the shorter of exact ARX5 entropy/base64url and
+ARX2 with **Node Brotli q11**, not the complete production auto pool and not a
+byte-identical validation of brotli-wasm. The new benchmark runner is designed to
+close that gap; it has not yet supplied replacement numbers.
+
+| Kind | Samples | Aggregate full-link saving with fallback |
+|---|---:|---:|
+| Markdown | 20 | 5.76% |
+| Code | 20 | 6.03% |
+| Diff | 20 | 5.91% |
+| CSV | 13 | 12.36% |
+| JSON | 20 | 8.52% |
+| Overall | 93 | 7.41% |
+
+Standalone ARX6 saved 7.36% aggregate, with 3 regressions and a worst regression
+of 2.77%. The portfolio won 90/93 comparisons and kept the legacy wire on the
+other three. Entire artifacts fitting 2,000 characters went from 75 to 76.
+These scores count `[View](https://agent-render.com/#...)`, not visible glyphs.
+
+The existing report fixture was a **separate diagnostic, not holdout evidence**.
+Its full link fell from 2,616 to 2,301 characters, still over the limit. Exhaustive
+complete-line prefix testing increased the largest fitting prefix from 5,867 to
+7,158 source characters at the same 1,996-character link size. That 22% capacity
+gain applies to this fixture, not arbitrary inputs.
+
+The historical summaries and freeze record are retained under
+`experiments/arx6/results/`. Their filenames and timings refer to the original
+laboratory archive. Third-party corpus text and its large license bundle are not
+silently republished as MIT application fixtures in this PR. Supply a permitted,
+independent corpus to the new runner, and report its results separately.
+
+The original environment measured median encode times of 105 ms for ARX6 versus
+56 ms for ARX5 and approximately 37.87 MiB versus 33.51 MiB of typed arrays. These
+are neither peak-RSS measurements nor production/browser latency guarantees.
+Async installation does not make the compression loop non-blocking.
+
+## Second-pass research disposition
+
+Dependency-aware numeric reconstruction programs and compression-chosen decode
+order produced large synthetic specialist wins, but only 0.24% aggregate saving
+on a fresh general-software set over ARX6. A separate 12-input natural CSV set
+improved 4.21% with fallback, while the table method alone regressed 3.84% on the
+ten inputs where it activated. A later root-only numeric ablation beat the graph
+on some natural inputs. These results do not justify including that machinery in
+this general codec PR. No universal breakthrough or scientific-first claim is
+made here.
+
+## Required before viewer integration / default enablement
+
+- [ ] Reuse the canonical tuple helpers; validate reconstructed envelopes through
+  the normal schema/normalization path and reconcile decoded/fragment budgets.
+- [ ] Introduce versioned async encode/decode routing and a compact tag; keep all
+  old tags and decoder bytes unchanged. Missing/skewed assets must be explicit,
+  retryable decode failures, never guessed priors.
+- [ ] Run CPU-bound encoding/decoding in a bounded Worker with cancellation and
+  deliberate error handling; measure maximum-payload latency and peak memory.
+- [ ] Compare the new candidate against the **complete** existing auto result,
+  retaining legacy bytes on ties, losses, and candidate unavailability. Keep
+  emission opt-in until independent real-agent data supports default enablement.
+- [ ] Complete application/UI, asset-loading, TypeScript, lint, build, Chromium,
+  WebKit, and live Discord/WhatsApp paste/click verification. Update the app docs,
+  examples, codec pickers, and agent skill together when the protocol is wired in.
+
+Current local evidence: the original prototype suite passed 20/20 groups; the
+new repository-asset conformance runner passed 16/16 checks. The full application
+suite, the new production-baseline benchmark, and browser integration were not
+run in the authoring environment. A draft PR is intentional, not a claim of
+production readiness.

experiments/arx6/README.md

@@ -0,0 +1,78 @@
+# ARX6: frozen native-frame codec experiment
+
+**Draft research import, not viewer support.** Nothing here registers `arx6` in the
+application schema, adds a picker option, changes automatic selection, or changes
+any historical decoder. The laboratory `#g1L...` wire does **not** open in the
+current viewer. Do not distribute it as a supported share link.
+
+This is the general ARX6 candidate from the first research pass. The second-pass
+reconstruction-program / dependency-order experiments are deliberately excluded:
+they did not earn a general-purpose default.
+
+## Run the conformance checks
+
+From the repository root, with Node 22:
+
+```sh
+node experiments/arx6/check.mjs
+```
+
+This command needs no installed packages and makes no network requests. It reads
+the existing three dictionary/prior assets from `public/`; no new corpus is
+trained or downloaded. The same 16 cases are registered with the normal Vitest
+suite in `tests/arx6-core.test.ts`:
+
+```sh
+npx vitest run tests/arx6-core.test.ts
+```
+
+The tests cover implementation hashes, CRC32, all UTF-16 code units, lone
+surrogates, exact whitespace/control preservation, bounded framing, all three
+prior identities, immutable prior copies, bundles/diff metadata, seeded random
+round trips, truncation/corruption, decoded limits, and byte-preserving fallback.
+
+## Compare against the actual production baseline
+
+After the usual `npm ci`, supply a JSON array of `{ "id": "...", "envelope": ... }`
+samples. Each envelope must satisfy the existing application schema.
+
+```sh
+node --import tsx experiments/arx6/bench.mts corpus.json > arx6-comparison.json
+```
+
+The benchmark loads the existing pinned assets, normalizes each envelope, calls
+**the complete production `encodeEnvelopeAsync` auto pool** with transport
+budgeting, and uses the real Markdown-link formatter. It verifies both the
+baseline envelope and experimental tuple round trips. Every header, checksum,
+base URL, and Markdown character is counted. Losses and oversized candidates
+retain the original baseline wire, byte for byte. It emits measurements and
+hashes, not unsupported viewer links.
+
+This production-baseline runner was added for this PR but has **not been executed
+in the authoring environment**, which could not clone the repository or install
+its npm dependencies. Do not substitute the historical research scores for its
+results. The dependency-free conformance command was executed: **16/16 passed**.
+The original, larger prototype suite was also rerun separately: **20/20 passed**.
+
+## What is frozen
+
+`src/cm6.mjs`, `src/native-frame.mjs`, `src/arx6-core.mjs`, and its `.d.mts` are copied
+byte-for-byte from the evaluated prototype. Keep the wire-affecting code frozen;
+a model/representation/prior change needs a new version and compatibility plan,
+not a casual golden update. The source-hash test intentionally detects even
+formatting changes. If a later cleanup changes formatting only, document it and
+also prove unchanged wire vectors before replacing these source pins.
+
+`runtime.mjs` is Node-only laboratory glue. It reproduces the ARX2 tuple mapping
+without modifying or importing private production helpers. Application
+integration should expose/reuse the canonical production tuple helpers instead
+of maintaining this duplicate mapping indefinitely.
+
+The browser-safe core returns **unknown tuples**, not validated application
+envelopes. It is not an alternative to schema validation. CRC32 detects accidental
+corruption, not malicious tampering or secret disclosure. Its 8,192-character
+budget includes `#`; the current application budgets fragment bodies, so the
+one-character boundary must be reconciled during integration.
+
+See [research notes](../../docs/arx6-research.md) for the qualified measurements
+and the uncompleted deployment gates.

experiments/arx6/bench.mts

@@ -0,0 +1,80 @@
+/** Compare ARX6 with the complete production auto pool; run with node --import tsx. */
+import assert from "node:assert/strict";
+import { createHash } from "node:crypto";
+import { readFileSync } from "node:fs";
+import { performance } from "node:perf_hooks";
+import { createLabCodec, envelopeToTuple, priorForEnvelope } from "./runtime.mjs";
+import { formatMarkdownLink } from "@/lib/markdown-link";
+import { normalizeEnvelope } from "@/lib/payload/envelope";
+import { encodeEnvelopeAsync, decodeFragmentAsync } from "@/lib/payload/fragment";
+import { isPayloadEnvelope, MAX_DECODED_PAYLOAD_LENGTH } from "@/lib/payload/schema";
+import { loadArxDictionarySync, loadArx2OverlayDictionarySync } from "@/lib/payload/arx-codec";
+import { loadArx4PriorsSync } from "@/lib/payload/arx4-codec";
+import base from "../../public/arx-dictionary.json";
+import overlay from "../../public/arx2-dictionary.json";
+import priors from "../../public/arx4-priors.json";
+
+const inputPath = process.argv[2];
+if (!inputPath) throw new Error("Usage: node --import tsx experiments/arx6/bench.mts corpus.json");
+const corpus: unknown = JSON.parse(readFileSync(inputPath, "utf8"));
+if (!Array.isArray(corpus) || corpus.length === 0) throw new Error("Expected a nonempty array of {id, envelope} samples.");
+loadArxDictionarySync(base);
+loadArx2OverlayDictionarySync(overlay);
+if (loadArx4PriorsSync(priors) !== 1) throw new Error("Production priors did not match their pins.");
+const { codec } = await createLabCodec();
+
+function link(fragmentBody: string): string {
+  const url = new URL("https://agent-render.com/");
+  url.hash = fragmentBody;
+  return formatMarkdownLink("View", url.href);
+}
+
+const rows = [];
+for (const [index, sample] of corpus.entries()) {
+  if (sample === null || typeof sample !== "object" || !isPayloadEnvelope(sample.envelope)) {
+    throw new Error(`Invalid sample envelope at index ${index}.`);
+  }
+  const normalized = normalizeEnvelope(sample.envelope);
+  if (!normalized.ok) throw new Error(`Sample ${index}: ${normalized.message}`);
+  const envelope = normalized.envelope;
+  if (JSON.stringify(envelope).length > MAX_DECODED_PAYLOAD_LENGTH) throw new Error(`Sample ${index} exceeds the decoded budget.`);
+  const baselineStart = performance.now();
+  const legacy = await encodeEnvelopeAsync(envelope, { budgetByTransport: true });
+  const baselineMs = performance.now() - baselineStart;
+  const decoded = await decodeFragmentAsync(`#${legacy}`, { skipFragmentBudget: true });
+  assert.ok(decoded.ok, `Production baseline failed to decode sample ${index}.`);
+  assert.deepEqual({ ...decoded.envelope, codec: "plain" }, { ...envelope, codec: "plain" });
+  const value = envelopeToTuple(envelope);
+  const candidateStart = performance.now();
+  const candidate = codec.encode(value, priorForEnvelope(envelope));
+  const candidateMs = performance.now() - candidateStart;
+  if (candidate !== null) assert.deepEqual(codec.decode(candidate), value);
+  const baselineChars = link(legacy).length;
+  const candidateChars = candidate === null ? null : link(candidate.slice(1)).length;
+  const wins = candidateChars !== null && candidateChars < baselineChars;
+  const selected = wins ? candidate!.slice(1) : legacy;
+  // A tie/loss preserves the previous wire exactly; never wrap it in an ARX6 header.
+  if (!wins) assert.equal(selected, legacy);
+  rows.push({
+    id: typeof sample.id === "string" ? sample.id : `sample-${index}`,
+    kind: envelope.artifacts[0].kind,
+    baselineTag: legacy.charAt(0), baselineChars, candidateChars,
+    selectedChars: wins ? candidateChars : baselineChars, wins,
+    baselineMs, candidateMs,
+    candidateSha256: candidate === null ? null : createHash("sha256").update(candidate).digest("hex"),
+  });
+}
+const baselineTotal = rows.reduce((n, row) => n + row.baselineChars, 0);
+const selectedTotal = rows.reduce((n, row) => n + (row.selectedChars ?? row.baselineChars), 0);
+console.log(JSON.stringify({
+  baseline: "complete production encodeEnvelopeAsync auto pool, budgetByTransport=true",
+  framing: "[View](https://agent-render.com/#...); includes every header and checksum character",
+  note: "Experimental ARX6 wires are not registered in the viewer; no shareable links are emitted.",
+  summary: {
+    samples: rows.length, wins: rows.filter(row => row.wins).length,
+    unavailableCandidates: rows.filter(row => row.candidateChars === null).length,
+    baselineTotal, selectedTotal, savedPercent: 100 * (1 - selectedTotal / baselineTotal),
+    baselineFits2000: rows.filter(row => row.baselineChars <= 2000).length,
+    selectedFits2000: rows.filter(row => (row.selectedChars ?? row.baselineChars) <= 2000).length,
+  }, rows,
+}, null, 2));

experiments/arx6/check.d.mts

@@ -0,0 +1,4 @@
+/** A dependency-free assertion case shared with the repository's Vitest runner. */
+export interface Arx6Check { name: string; run(): void | Promise<void>; }
+/** Install the pinned corpus and return the frozen-core conformance cases. */
+export function buildChecks(): Promise<Arx6Check[]>;