Skip to content

fix: deduplicate concurrent upload parse dispatch - #351

Open
Frankie-Xu wants to merge 1 commit into
Ontos-AI:mainfrom
Frankie-Xu:fix/Frankie-Xu/dedupe-parse-dispatch
Open

Frankie-Xu wants to merge 1 commit into
Ontos-AI:mainfrom
Frankie-Xu:fix/Frankie-Xu/dedupe-parse-dispatch

Conversation

@Frankie-Xu

@Frankie-Xu Frankie-Xu commented Aug 28, 2026

Copy link
Copy Markdown

Summary

  • Dispatch parsing only when this handoff owns the successful waiting-file -> pending transition.
  • Treat a CAS retry that succeeds as pending -> pending as an idempotent no-op.
  • Recreated on current main (85d5b2ba): main already no-ops a rejected CAS from pending, but a successful same-state retry still dispatched.
  • Add direct and HTTP-interleaving regression coverage for the S3 notification / confirm-upload race.
  • confirm-upload remains the fallback when the storage event is absent. RedisJobLock and the worker terminal-state gate are unchanged.

Closes #286

Test plan

  • make lint
  • uv run pytest apps/api/tests/contract/test_s3_event_contract.py -k "pending_to_pending or cas_winner or localstack" -q — 3 passed (no FastAPI import)
  • HTTP interleaving cases are in the same file; they import apps/api/main.py and currently fail on Windows because fcntl is missing (project P3.2). Upstream CI is Ubuntu.

@Frankie-Xu

Copy link
Copy Markdown
Author

This is a first-time fork workflow and GitHub currently reports no checks. The focused regression, concurrent HTTP cases, affected contract file, lint, and typecheck pass locally; the full API contract result and its unrelated environment-specific failure are documented in the PR body. Please approve the workflow run when convenient.

@Frankie-Xu
Frankie-Xu force-pushed the fix/Frankie-Xu/dedupe-parse-dispatch branch from ad0e247 to 6912ba3 Compare September 11, 2026 16:29
@Frankie-Xu

Copy link
Copy Markdown
Author

Rebased/recreated on current main (85d5b2ba). The remaining hole was a successful pending -> pending CAS retry still calling start_uploaded_file_parse. Only waiting-file -> pending now dispatches.

@Frankie-Xu

Copy link
Copy Markdown
Author

Follow-up: #412 lands a Windows-safe telemetry file lock, so test_s3_event_contract.py can be collected on Windows instead of dying on import fcntl. Ubuntu CI for this PR is unchanged.

A successful pending-to-pending CAS retry was still enqueueing a second parse_task when the S3 notification raced confirm-upload.
@cursor
cursor Bot force-pushed the fix/Frankie-Xu/dedupe-parse-dispatch branch from 6912ba3 to a4cfb82 Compare September 14, 2026 18:45
@Frankie-Xu

Copy link
Copy Markdown
Author

Rebased onto current main (9489fa2c, includes #414/#415). No overlap with those parser/retrieval changes; still waiting on first-time fork Actions workflow approval.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Avoid duplicate parse dispatch when S3 notification races confirm-upload

1 participant