Skip to content

Cancelled knowledge worker can overwrite a newer successfully stored document #397

Description

@Calmingstorm

Confirmed stale write after cancelled ingest

Reviewed master at 886c36d8ebe861aa987059a1744d45b78797baae (v4.7.0). Suggested priority: P1, conditional data-integrity failure.

Knowledge ingest holds an asyncio lock while awaiting asyncio.to_thread. Cancelling that await releases the lock but does not stop the worker. A later ingest can finish and report stored, then the cancelled earlier worker publishes stale content over it.

Source: lock and worker calls. Async delete/merge wrappers use the same ownership pattern and should be audited, though this reproduction covers ingest.

Deterministic isolated reproduction

Use a temporary KnowledgeStore and threading events, not timing guesses:

  1. Pause the first ingest worker after dispatch, before its first store access.
  2. Cancel its waiting coroutine.
  3. Complete a second ingest to the same source.
  4. Release the first worker and read the final rows.
after cancellation: lock free=True, worker finished=False
second ingest: stored 1
rows before release: ['second current content']
rows after cancelled worker: ['first stale content']
acknowledged second source durable: False

Independently reproduced, including three consecutive confirmation runs and a parent rerun. The script uses dedup=False to isolate writing; normal ingest reaches the same write block. No live stores or services were touched.

Impact / acceptance criteria

  • A successfully acknowledged newer document must not be overwritten by an earlier cancelled worker.
  • Keep exclusive write ownership until physical worker completion, including cancellation cleanup and repeated cancellation, or use an equivalent serialized publication protocol.
  • Include version/snapshot publication in the consistency review.
  • Test actual task cancellation, later same-source ingest and final source verification.
  • Do not claim this fixture proves physical SQLite corruption or FTS corruption; it proves logical stale replacement.

Behavior change: cancellation may wait for a running write to settle. Normal successful writes retain their semantics. Frequency in production is unknown. No source changes were made.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions