Skip to content

Usage coverage never recovers from a transient source-scan error #309

Description

@Calmingstorm

Defect

UsageRollup accumulates source-scan failures in a process-lifetime counter and never clears it after a later successful scan. A one-off source read failure therefore leaves otherwise complete usage coverage reported as incomplete until the process restarts.

Exact locations

  • src/usage/rollup.py:620 and src/usage/rollup.py:627 increment _source_scan_errors for trajectory discovery/open failures.
  • src/usage/rollup.py:645 increments the same counter for an audit snapshot failure.
  • src/usage/rollup.py:664 uses the accumulated counter to decide whether an empty-source backfill pass completed.
  • src/usage/rollup.py:941 and src/usage/rollup.py:954 use that same process-lifetime value in every subsequent coverage response.

Concrete reproduction sequence

  1. Construct an available UsageRollup with no trajectory rows and an audit source whose first open_read_snapshot() call raises OSError.
  2. Run _one_backfill_pass(). _audit_snapshots() increments _source_scan_errors to 1, and the pass reports incomplete coverage.
  3. Make the same audit source recover so its next open_read_snapshot() succeeds and returns an empty snapshot list.
  4. Run _one_backfill_pass() again, then request summary("all").
  5. The second pass still returns False; coverage.backfill_complete remains false and coverage.scan_errors remains 1 even though every source in that pass succeeded. With no source files, the reconciler also keeps selecting the incomplete-pass retry interval rather than the completed polling interval.

Observed in a focused executable reproduction at master 67dafd8ae82b643f0a37743c91ae17351abfe29e: first_complete=False, second_complete_after_recovery=False, _source_scan_errors=1, and reported backfill_complete=False.

Expected

Source-scan health should reflect the current/recent pass (while durable cursor error totals can retain historical diagnostics). A successful complete pass should restore backfill_complete=true without requiring a process restart.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions