[0xvigilant]

Stacked on #53. This branch includes #53's commit (0abc27a), so please review from 1857a92 onward. Once #53 merges, I'll rebase this branch.

What changed

Adds JSON source support. application/json responses become json-leaf-text/v1 evidence windows:

• strict parse;
• one <path>: <value> line per leaf, in document order;
• number literals kept as written (ASCII only);
• string escapes decoded;
• line breaks normalised to one space.

The extractor identity (charset, max_characters, max_depth: 64, max_values: 1000000) is recorded on evidence-window v2.

Fail-closed checks run in this order: the value bound and a linear, string-aware nesting scan (both before parsing), then strict parsing.

• Invalid JSON, duplicate keys, a UTF-8 byte-order mark, NaN/Infinity, lone surrogates, non-ASCII digits or nesting beyond 64 give source_media_unsupported.
• More than 1,000,000 values gives source_too_large.
• Each failure carries a fixed detail.

Rendering memory is bounded by the character budget, not by depth × key length.

The v1 window rules are frozen and re-applied on replay:

• empty: an empty root;
• hollow: 4,096 characters or fewer, matching v1's own frozen block patterns;
• sparse: under 256 characters;
• otherwise complete.

Capture declarations admit application/json. The declaration-to-response media match stays strict.

Behaviour change: up to 0.2.0a3, the resolver stored application/json as raw http-text/v2. Now:

• new captures hash differently;
• the bodies listed above now fail;
• small envelopes are no longer complete.

Existing raw-JSON snapshots replay unchanged; fixtures archived by 0.2.0a3 pin this. Comparing a new capture with an old raw-JSON snapshot reports different. See docs/MIGRATION.md (Unreleased).

Evidence

The ClinicalTrials.gov API v2, the bioRxiv/medRxiv API and Europe PMC REST publish primary records only as JSON. Their HTML pages are script shells (the ClinicalTrials.gov study page captures as a 175-character sparse window) or return 403. In the IC pipeline that leaves current trial status and preprint abstracts uncapturable; see research-pipelin3/claude-research-pipeline#105. Synthetic fixtures with the same shape are in tests/test_json_sources.py.

Acceptance:

• JSON quotes anchor byte-exactly through the IC ledger (e.g. COMPLETED, 47079);
• captures replay byte-for-byte;
• bad, deep or oversized JSON fails closed;
• the media match stays strict.

Review hardening, measured through the default HttpResolver:

• A 6.9 KB gzip body took 1,769 MB extra memory before and 86 MB after.
• A 30 MB body of deep emoji keys took 4,069 MB before and 179 MB after.
• 10 KB bracket bodies crashed 512 KiB and 1 MiB threads before. They now fail closed at stacks from 64 KiB to 1 MiB.
• The nesting scan is linear: worst case 1.2 s on 30 MiB.

Owner decision needed

The full NCT04368728 record renders to 8.88M characters, over the 8,388,608-character cap (about 4x growth from repeated paths), so it captures as truncated. ?fields=protocolSection, with fields retained for clinicaltrials.gov, renders to 206,529 characters and captures complete. The docs use that form.

Options:

• raise the cap for JSON only;
• adopt a more compact format as a later version (v1 is frozen once snapshots exist, so this is worth deciding before release);
• keep citing filtered URLs.

Lane

• IC product path
• Canonical core
• Active experiment
• Legacy compatibility

Contract impact

• No schema, policy, gate, metric, holdout, or public API changed.
• Any intentional contract change is versioned and documented. This covers the new extractor json-leaf-text/v1, application/json admitted in the existing declaration schemas, and fixed failure details. The change from raw JSON capture is disclosed in EVIDENCE-WINDOWS and MIGRATION.
• No quality claim is borrowed from a different domain or evaluation.
• No metric or threshold was selected after viewing protected results. The thresholds (256 / 4,096 / 64 / 1M) were set from synthetic shapes and resource measurements, and are frozen in v1.
• Experimental behaviour is not described as a current product stage.

Determinism and disclosure

• Deterministic tests do not access the network. They use injected openers, and the subprocess tests make no requests.
• New model-backed behaviour is replaceable and benchmark-gated. None was added.
• Outputs retain source, policy, component, and engine identity.
• The diff contains no credentials, private source text, protected holdout data, or local filesystem paths.

Validation

python3.12 -m pytest -q   # 584 passed (main: 479; with #53: 483)

No version bump or release; that's your call. The MIGRATION entry is under "Unreleased".

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

• Configuration used: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: 59610580-5228-400e-ae44-4e5d765bf288

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[b1rdmania]

Owner decision on the size cap: keep the current 8,388,608-character extracted-text cap. (This is a character cap, not an 8 MiB response-byte cap.) Do not raise it for the large ClinicalTrials.gov example and do not introduce another extractor format in this PR.

Use explicit filtered URLs where their selected fields cover the claim being cited. ?fields=protocolSection is acceptable for protocol evidence, including status fields present there. “Complete” refers only to that filtered response, not to the whole study record or omitted results. Results claims need a separate appropriately scoped capture containing the results; absent or truncated evidence remains a limitation. Keep fields in the host-specific retained query policy so filtered and unfiltered records cannot share an identity.

Fork CI approved after inspecting the code/workflow changes. Independent review and local tests are running; this comment records the scope decision, not yet a merge approval.

[b1rdmania]

Review update: no blocking production-code finding from the code review or independent JSON review. The original head passed 584 tests locally, but CI on Python 3.12.14 exposed four failures in the test that expected the stdlib’s permissive Unicode-digit behaviour.

I pushed the narrow repair in 57208cf: test the installed Python fallback, then explicitly emulate the older NUMBER_RE to verify Groundnut’s own ASCII guard on every patch release. This keeps the regression meaningful rather than skipping it. No production parser change. All 584 tests pass locally after the fix.

The same commit records the owner decision in READ-TIME-CAPTURE: retain the character cap; “complete” applies to the explicitly selected protocol response, never to omitted results/full-record coverage.

#53 is now reviewed and merged, with 483 local tests and green CI. It was merged with ancestry preserved, so this stack does not need a history rewrite just to remove #53’s diff. Waiting for the updated #54 CI before merging; no pipeline pin or existing run configuration has been changed.

[b1rdmania]

Reviewed the JSON extraction/capture/replay changes and exact final head 57208cf, including an independent review. 584 local tests pass and CI is green on Python 3.12.14. The scanner regression repair preserves the ASCII-guard check; an independent mutation check confirms it catches removal of that guard. Keep the existing character cap and the documented filtered-response evidence boundary. No blocking findings; no release/pipeline adoption implied by this merge.

The merge-base changed after approval.