Proposal: recursive summarization for transcripts larger than the context window

[nnunley]

Draft / proposal. This is offered, not assumed. It is a subsystem, not a patch: new config keys, three new tables, two new CLI verbs, and a ~1,200-line orchestrator. If the maintainers do not want to own this surface, say so and it stays downstream. If parts of it are wanted and parts are not, it can be cut down.

Problem

Some days produce transcripts larger than any single context window. The existing path builds one prompt per (date, project) group and sends it to the model. When the group is too large, the options today are truncation or a skip — both silently lose engineering history, and a skip is indistinguishable from a day with no work.

A naive map-reduce over chunks does not solve it either, because nothing guarantees the model actually looked at every chunk. A summarizer that quietly covers 60% of a day and returns a confident summary is worse than one that fails loudly.

Approach

summarize picks a strategy by transcript size:

• One-shot fast path (src/oneshot-summarize.ts) when the transcript fits under the provider threshold. Sandwich prompt with the instructions repeated before and after the transcript, plus explicit anti-continuation guidance so the model summarizes instead of extending the transcript.
• Recursive fallback (src/recursive-summarize.ts) otherwise.

The recursive summarizer gives the model a scope — a byte range of the transcript — and three tools: search, read_range, and spawn. spawn creates a child invocation over a sub-range, which recurses with the same rules. Each invocation returns a structured extraction that is folded into its parent.

The 100% coverage guarantee

An invocation may only declare done when the union of its read_range spans, plus the scopes of its children, covers its entire scope. Coverage is tracked as span algebra:

• mergeSpans normalizes and unions overlapping and adjacent spans.
• residualGaps computes the uncovered intervals of [0, scopeLen).
• A done with non-empty residual gaps is rejected, and the gaps are handed back to the model as the remaining work.

This is why it matters: the failure mode of every LLM summarizer over long inputs is confident partial coverage. Here, partial coverage cannot be reported as success. The summarizer can fail — it cannot silently under-read.

Termination comes from validateSpawnRange: a child scope must be in bounds, non-empty, and at most 50% of its parent's length. Scope therefore at least halves per level, so recursion depth is bounded by log2 of transcript size. Each invocation is additionally capped at MAX_ACTIONS_PER_INVOCATION (20) and read_range returns at most READ_WINDOW (2000) bytes per call, so a single invocation cannot brute-force a large scope without spawning.

Local models return dirty JSON. parseLlmJsonResponse strips ```json and bare fences, strips <think>...</think> leaks including a dangling closing tag with no opener, and falls back to `extractFirstBalancedObject`, a brace matcher that respects string literals and escapes. Top-level arrays and scalars are rejected rather than coerced.

Auditability

The invocation tree is persisted (journal_invocations, journal_fragments) with per-invocation scope, actions, and outcome, and rendered by renderInvocationTree. summarize --why explains why a group produced the entry it did, and a new skipped verb audits skip reasons across the corpus. Without this the recursion is a black box; with it, an under-covered or misbehaving run is inspectable after the fact.

Supporting pieces in this branch

• OpenAI-compatible provider alongside the existing Claude path (summary_provider, summary_base_url, summary_model, summary_extras), with extras passthrough for provider-specific knobs. Required request keys override extras so extras cannot break the request shape. Config validation rejects an unknown provider with a clear error. Settings are exposed in the web UI.
• Weekly per-project rollups (src/rollup.ts, journal_rollups, rollup verb) aggregating daily entries into a Monday-anchored week, keeping skipped days visible in a separate section.

Scope note

This branch also contains the two ingest/grouping fixes submitted separately as #27. If #27 merges first, this branch rebases onto it and those commits drop out. Reviewing #27 first is the easier path.

How to test

bun test
bun run typecheck

166 passing, 0 failing, typecheck clean (baseline on main is 92 passing).

Unit coverage is on the parts where correctness is checkable without a model: mergeSpans, residualGaps, validateSpawnRange (including the 51%-of-parent rejection), extractFirstBalancedObject, parseLlmJsonResponse (including a real Ollama think-leak case), and invocation-tree round-trip, idempotent rewrite, and rendering.

Known limitations

• The 100% guarantee is over byte coverage, not comprehension. It proves every region was read by some invocation; it does not prove the extraction from that region was good.
• Recursion costs many model calls on large transcripts. It is a fallback, not the normal path — the one-shot path handles typical days.
• Tuning constants (50% spawn cap, 20 actions, 2000-byte read window, provider size thresholds) are hardcoded, chosen empirically against local models, and not yet configurable.
• End-to-end recursion is not covered by automated tests because it requires a live model; the deterministic components are unit-tested and the orchestration loop is not.
• New config keys default to the existing Claude behavior, so an existing install is unaffected until it opts in.
• Schema migrations are additive with IF NOT EXISTS and guarded ALTER TABLE, but journal_fragments is rebuilt via table-rename to add a foreign key, since SQLite cannot add a column with REFERENCES. That path deserves attention in review.