Defect
An appended trajectory record larger than the bounded tail-read window permanently blocks usage ingestion at that byte offset, while coverage is still reported complete. The oversized-row sentinel logic is reachable only after a newline has already been found in the bounded read, so a single oversized complete line whose terminating newline lies beyond the first read can never be skipped.
Exact locations
src/usage/rollup.py:588 reads at most _BACKFILL_BYTES + 1 bytes from the current tail cursor.
src/usage/rollup.py:589-592 returns without advancing high_offset when that bounded chunk contains no newline.
src/usage/rollup.py:594-597 has oversized-row handling, but it cannot run in this case because the preceding no-newline return has already exited.
src/usage/rollup.py:701-711 determines completion only from each cursor's historical initial_complete flag, without checking whether high_offset caught up to the current complete tail.
Concrete reproduction sequence
- Create a trajectory JSONL file with one normal settled turn and run
_one_backfill_pass() until its cursor has initial_complete=1 and the turn is indexed.
- Append one valid JSON trajectory record whose serialized line is slightly larger than
_BACKFILL_BYTES, followed by its newline.
- Run
_one_backfill_pass() again. _consume_tail() reads only _BACKFILL_BYTES + 1 bytes from the old high_offset, sees no newline within that prefix, and returns without changing the cursor.
- Run additional passes and request
summary("all").
- The cursor remains at the pre-append offset, the appended turn is never indexed, but
_one_backfill_pass() returns True and coverage.backfill_complete is true. Any later normal rows behind the oversized row are blocked as well.
Observed in a focused executable reproduction at master 67dafd8ae82b643f0a37743c91ae17351abfe29e: the source grew beyond the tail-read bound, repeated passes left high_offset at the old file size and the indexed turn count unchanged, while both pass completion and reported coverage remained true.
Expected
A complete oversized appended row should be skipped as malformed with its cursor advanced past the terminating newline, and coverage must not report complete while the tail cursor is behind complete source data.
Defect
An appended trajectory record larger than the bounded tail-read window permanently blocks usage ingestion at that byte offset, while coverage is still reported complete. The oversized-row sentinel logic is reachable only after a newline has already been found in the bounded read, so a single oversized complete line whose terminating newline lies beyond the first read can never be skipped.
Exact locations
src/usage/rollup.py:588reads at most_BACKFILL_BYTES + 1bytes from the current tail cursor.src/usage/rollup.py:589-592returns without advancinghigh_offsetwhen that bounded chunk contains no newline.src/usage/rollup.py:594-597has oversized-row handling, but it cannot run in this case because the preceding no-newline return has already exited.src/usage/rollup.py:701-711determines completion only from each cursor's historicalinitial_completeflag, without checking whetherhigh_offsetcaught up to the current complete tail.Concrete reproduction sequence
_one_backfill_pass()until its cursor hasinitial_complete=1and the turn is indexed._BACKFILL_BYTES, followed by its newline._one_backfill_pass()again._consume_tail()reads only_BACKFILL_BYTES + 1bytes from the oldhigh_offset, sees no newline within that prefix, and returns without changing the cursor.summary("all")._one_backfill_pass()returnsTrueandcoverage.backfill_completeistrue. Any later normal rows behind the oversized row are blocked as well.Observed in a focused executable reproduction at master
67dafd8ae82b643f0a37743c91ae17351abfe29e: the source grew beyond the tail-read bound, repeated passes lefthigh_offsetat the old file size and the indexed turn count unchanged, while both pass completion and reported coverage remained true.Expected
A complete oversized appended row should be skipped as malformed with its cursor advanced past the terminating newline, and coverage must not report complete while the tail cursor is behind complete source data.