fix(scheduler): swallow per-entry sync errors so beat stays up - #45
Merged
Conversation
The multitenant sync() override caught only KeyError/ObjectDoesNotExist inside the per-entry try block, so any other exception raised while switching schema or saving an entry (e.g. a stale schema entry whose name split gives a space_code no longer in get_all_tenant_schemas, which makes set_schema_from_context raise plain Exception) propagates out of sync(), out of DatabaseScheduler.schedule, out of beat.tick(), and kills the beat process. Observed on realm04pdn/space0uph9 with the workflow-scheduler pod: 254 restarts in the month after the previous scheduler fix was deployed (2026-07-13). On 2026-08-11 the restart coincided with a user cancelling four in-progress price-history workflows; after each restart beat treated portfolio_history as due, dispatched it, crashed again on the same bad entry, restarted, dispatched again -- 316 portfolio_history workflows fired in 89 minutes with identical payload before the loop broke. Fix: log the offending entry and re-queue it via _failed, matching the existing (KeyError, ObjectDoesNotExist) behaviour, instead of letting the exception escape sync(). One healthy schedule can no longer be taken down by an unrelated broken entry.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The multitenant
sync()override inDatabaseSchedulercatches only(KeyError, ObjectDoesNotExist)inside the per-entry try block. Any other exception — for exampleset_schema_from_context({"space_code": schema})raising a plainExceptionwhen a stale entry’s schema is no longer inget_all_tenant_schemas()— escapessync(), propagates throughDatabaseScheduler.scheduleandbeat.tick(), and kills the beat process.Impact (observed)
Realm
realm04pdn, spacespace0uph9:workflow-schedulerpod restarted 254 times in the month after the previous scheduler fix was deployed (2026-07-13 → 2026-08-13).helper-calculate-portfolio-price-historyworkflows. After each restart beat treatedportfolio_historyas due, dispatchedworkflow.tasks.workflows.execute, crashed again on the same bad entry, restarted, dispatched again.portfolio_historyworkflows fired in 89 minutes, all with the same payload (T-22 × 30 portfolios), before the loop finally settled. To the user this looked like an uncontrolled cascade.Fix
Add an outer
except Exceptioninside the per-entrytryblock that mirrors the existing(KeyError, ObjectDoesNotExist)branch: log the entry name and error, add it to_failedso it gets re-queued for the next sync, and continue with the rest of_dirty. One healthy schedule can no longer be taken down by an unrelated broken entry.Diff is 15 lines (comment +
except), no behavior change on the happy path.Rollout notes
workflow-scheduler-realm04pdn. Once beat is stable,last_run_atgets persisted normally on each tick and the "keep re-firing because we crashed before saving" pattern goes away.