fix(scheduler): persist last_run_at per-tenant schema to stop duplicate schedule fires - #44
Merged
Conversation
…te fires The multitenant DatabaseScheduler keys entries "<schema>:<name>" but the base sync() saved last_run_at/total_run_count under whatever search_path was active, so the update never landed in the tenant schema. last_run_at never advanced, the crontab stayed "due", and beat re-sent the same scheduled task every cycle while its due window was open -- observed as portfolio_history firing 3-4x/night in space0uph9 (register/price fire once). total_run_count stuck at 0 in the DB was the tell-tale. Override sync() to set_schema_from_context per dirty entry before save(). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The custom multitenant
DatabaseScheduler(workflow/schedulers.py) loads schedules across all tenant schemas and keys entries"<schema>:<name>", but does not overridesync(). The base django-celery-beatsync()saves each dirty entry'slast_run_at/total_run_countunder whatever DBsearch_pathis active at sync time — not the entry's schema. So the bookkeeping lands in the wrong schema (or nowhere) andlast_run_atnever advances for the tenant.Consequence: the crontab stays due on every beat cycle until the workflow task itself finally runs and stamps
last_run_at. When the workflow launch is delayed (busy workers / ordering gate), beat re-sends the same scheduled task several times.Observed (realm04pdn / space0uph9, Mars Capital)
portfolio_history(cron30 1) fired 3–4×/night;register/price_historyfired once. Beat log:DB tell-tale:
total_run_count = 0on all schedules despite nightly runs. Result: 3–4× duplicate heavy calculations in parallel, duplicatePortfolioHistoryrows, ~9h runtime instead of ~2h.Fix
Override
sync()toset_schema_from_context({"space_code": <entry schema>})per dirty entry (schema parsed from the"<schema>:<name>"key) beforesave(). Entries that can't be saved are re-queued for the next sync, matching base-class semantics.Verification
total_run_countshould become> 0(regression signal).Notes
Space-level stop-gaps are already live for space0uph9 while this lands: a duplicate-run guard in the
helper-calculate-portfolio-historyworkflow module + retimed cron. This PR is the platform root fix and covers all schedules / tenants.🤖 Generated with Claude Code