TTSR paragraph-repeat detector misses near-repeat drift (exact hash match); needs normalized match gated by progress

[code-yeongyu]

Summary

The TTSR paragraph-repeat detector landed in #1326 (packages/coding-agent/src/core/extensions/builtin/ttsr/detectors/collapse-paragraphs.ts) matches paragraphs by exact content: two hashes (FNV-1a style + djb2) plus the UTF-16 length (isSameParagraph), counted over a 64-entry ring, threshold 3, paragraphs >= 64 chars / 24 word chars.

Because the match is exact, a near-repeat loop escapes it: a model that re-narrates the same paragraph while only a counter, timestamp, or clause order drifts ("Now writing step 3...", "Now writing step 4...") produces distinct hashes, so the repeat count never reaches 3 and the stream runs until the user aborts. Interleaved repeats (A B A B A B) are already covered because the ring is windowed, not consecutive.

Constraint on the fix

Loosening the match alone (whitespace/number masking before hashing) re-creates the false positive this detector was tuned to avoid: a legitimate list traversal that prints "processing item N" for N = 1..k collapses to one hash and would be interrupted as a loop. So a normalized match needs a progress gate as a precondition, e.g. treat a candidate repeat as a real loop only when no tool call was emitted between the repeats (the incident stream had 0 tool calls over 38K chars) or when tool arguments/checkpoints did not advance. Both halves belong in one change.

Evidence

• Incident and replay analysis: sionicai session 01a06648 (2026-09-03), 38,193-char single text part, 7-paragraph cycle repeated ~9x, 0 tool calls; the exact-cycle case is what fix(ttsr): interrupt within-message paragraph repetition #1326 now catches.
• Source: collapse-paragraphs.ts isSameParagraph (hashA/hashB/utf16Length equality), PARAGRAPH_REPEAT_THRESHOLD = 3, PARAGRAPH_RING_CAPACITY = 64.

Surfaced while answering a community question in the Ultraworkers Discord (#chat-ko, 2026-09-03).

[code-yeongyu]

A second incident hit this gap on 2026-09-16, and it also shows a shape the ring size hides.

claude-fable-5-1 streamed narration for 12 minutes with zero tool calls. It began as paraphrase - the same action restated in different words each time ("I'm assembling the final delivery ... - task complete." / "Both batches sent with photos and reaction cleared - done.") - and then settled into an exact cycle of 114 paragraphs, about 14.4 KB per turn of the cycle. Replaying the captured text through the shipped collapseDetector returns no match: the paraphrased region never produces three identical hashes, and once the cycle does repeat verbatim its period is longer than PARAGRAPH_RING_CAPACITY = 64, so the third occurrence is never in the ring at the same time as the first two. Measured on the tail: 315 paragraphs, 112 distinct, max occurrences inside the ring = 2.

The generation ended only because the session was killed. It was never persisted either, since an in-flight message is written at message_end, so the evidence came from the terminal scrollback.

On the constraint in the issue body - that a normalized match alone would interrupt a legitimate "processing item N" traversal - a frequency gate answers it without a separate progress signal. Treat a paragraph as an echo at 0.5 Jaccard over normalized word sets against the last 32 paragraphs, then fire only when at least 8 of the last 12 paragraphs are echoes. One similar pair, a callback to an earlier point, or a closing summary never reaches that density, and the rule does not care how long the cycle is.

Calibrated against the local session store: over 12,499 real assistant text and thinking parts (28.9 MB), that rule fires twice, and both are already-known runaway generations. Everything else is silent.

#1763 implements it.

[code-yeongyu]

Closed by #1763, merged as 76dfe1760a on main.

What shipped - detectors/collapse-near-duplicates.ts (near-duplicate-paragraphs), wired last in the collapse chain for text and thinking streams. A paragraph is an echo at 0.5 Jaccard over normalized word sets against any of the last 32 eligible paragraphs, and the detector fires only once at least 8 of the last 12 eligible paragraphs are echoes. The density gate is what answers the constraint in the issue body: a normalized match alone would interrupt a legitimate "processing item N" traversal, but one similar pair, a callback, or a closing summary never reaches 8-of-12. Because it counts rate rather than identity, it is also independent of cycle length, which the 64-entry exact ring is not.

Commits

sha	subject
d775155704	test(ttsr): RED contract for paraphrased paragraph loops
c68a979863	test(ttsr): give the synthetic prose fixtures real vocabulary
599b1f779a	fix(ttsr): interrupt paragraph loops by near-duplicate frequency
291770a465	docs(ttsr): record the near-duplicate paragraph mechanism
bb370ca240	test(senpi-qa): drive a paraphrased paragraph loop through the real CLI

Evidence

• Calibration: replaying the shipped detector over 12,499 real assistant text and thinking parts (28.9 MB) fires twice, and both are known runaway generations from one 2026-09-03 session. No other match.
• Incident replay: the captured 2026-09-16 tail fires after 1,935 characters, against 12 minutes of unguarded streaming before.
• Unit: test/ttsr/detector-collapse-near-duplicates.test.ts; full test/ttsr suite 18 files / 321 tests green on a separate macOS machine (bun 1.4.2), bun run check exit 0.
• Mutation: removing the single wiring line in collapse.ts fails exactly the three positive tests; restoring returns 9/9.
• Real CLI: senpi-qa mock-loop scenario ttsr-near-duplicate-loop 7/7 - abort, persisted message truncated 2,446 -> 527 chars, collapse-repetition interrupt in the recovery request, recovered answer. ttsr-paragraph-loop 6/6 and ttsr-collapse 4/4 unchanged.
• CI on the merged head: 22 checks green, 0 failures.

One fixture note - two synthetic generators (buildHealthyPrefix, narration()) varied only by a counter, so after digit normalization every paragraph carried the same tokens: healthy input for a byte-exact rule, a narration loop for a normalized one. Both gained real vocabulary and their assertions are untouched, so the exact-repeat contract still tests exact repetition. The digit-only-drift shape they used to have is the shape this issue asks to catch, and it is now pinned by the new positive test.