Skip to content

fix(scheduler): persist last_run_at per-tenant schema to stop duplicate schedule fires - #44

Merged
paktusov merged 1 commit into
mainfrom
fix/scheduler-per-schema-sync-duplicate-fire
Jul 8, 2026
Merged

fix(scheduler): persist last_run_at per-tenant schema to stop duplicate schedule fires#44
paktusov merged 1 commit into
mainfrom
fix/scheduler-per-schema-sync-duplicate-fire

Conversation

@paktusov

@paktusov paktusov commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator

Problem

The custom multitenant DatabaseScheduler (workflow/schedulers.py) loads schedules across all tenant schemas and keys entries "<schema>:<name>", but does not override sync(). The base django-celery-beat sync() saves each dirty entry's last_run_at / total_run_count under whatever DB search_path is active at sync time — not the entry's schema. So the bookkeeping lands in the wrong schema (or nowhere) and last_run_at never advances for the tenant.

Consequence: the crontab stays due on every beat cycle until the workflow task itself finally runs and stamps last_run_at. When the workflow launch is delayed (busy workers / ordering gate), beat re-sends the same scheduled task several times.

Observed (realm04pdn / space0uph9, Mars Capital)

portfolio_history (cron 30 1) fired 3–4×/night; register / price_history fired once. Beat log:

01:30:00  Sending due task portfolio_history
01:30:30  Sending due task portfolio_history
01:35:32  Sending due task portfolio_history
01:40:34  Sending due task portfolio_history

DB tell-tale: total_run_count = 0 on all schedules despite nightly runs. Result: 3–4× duplicate heavy calculations in parallel, duplicate PortfolioHistory rows, ~9h runtime instead of ~2h.

Fix

Override sync() to set_schema_from_context({"space_code": <entry schema>}) per dirty entry (schema parsed from the "<schema>:<name>" key) before save(). Entries that can't be saved are re-queued for the next sync, matching base-class semantics.

Verification

  • After deploy, total_run_count should become > 0 (regression signal).
  • A schedule whose launch is delayed should fire exactly once.

Notes

Space-level stop-gaps are already live for space0uph9 while this lands: a duplicate-run guard in the helper-calculate-portfolio-history workflow module + retimed cron. This PR is the platform root fix and covers all schedules / tenants.

🤖 Generated with Claude Code

…te fires

The multitenant DatabaseScheduler keys entries "<schema>:<name>" but the base
sync() saved last_run_at/total_run_count under whatever search_path was active,
so the update never landed in the tenant schema. last_run_at never advanced,
the crontab stayed "due", and beat re-sent the same scheduled task every cycle
while its due window was open -- observed as portfolio_history firing 3-4x/night
in space0uph9 (register/price fire once). total_run_count stuck at 0 in the DB
was the tell-tale.

Override sync() to set_schema_from_context per dirty entry before save().

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@paktusov
paktusov merged commit d210e50 into main Jul 8, 2026
2 checks passed
@paktusov
paktusov deleted the fix/scheduler-per-schema-sync-duplicate-fire branch July 8, 2026 10:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant