Fix job timeout - #351
Open
HeDo88TH wants to merge 3 commits into
Open
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR addresses long-running Hangfire jobs being re-dequeued (and effectively “timing out”) by making the MySQL storage invisibility timeout configurable, and adjusts DroneDB build wrappers to treat “build already in progress” as a benign condition.
Changes:
- Add
HangfireInvisibilityTimeoutHourstoAppSettings+ default configuration, and surface it in the configuration UI. - Apply the configured invisibility timeout to Hangfire MySQL storage options.
- Change
HangfireUtilsbehavior onDdbBuildInProgressExceptionto log and skip instead of sleeping/retrying inline.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| Registry.Web/Utilities/StartupExtenders.cs | Configures Hangfire MySQL storage InvisibilityTimeout from settings. |
| Registry.Web/Utilities/HangfireUtils.cs | Updates build wrappers’ handling of DdbBuildInProgressException. |
| Registry.Web/Services/Adapters/ConfigurationDataBuilder.cs | Exposes the new invisibility-timeout setting in the Cron Jobs configuration section. |
| Registry.Web/Models/Configuration/AppSettings.cs | Adds the new HangfireInvisibilityTimeoutHours setting with XML documentation and default. |
| Registry.Web/appsettings-default.json | Adds the default HangfireInvisibilityTimeoutHours value under AppSettings. |
Suppressed comments (1)
Registry.Web/Utilities/HangfireUtils.cs:92
- Similar to BuildWrapper: when BuildPendingWrapper hits DdbBuildInProgressException it currently logs "skipping" but still continues to log "Done build pending", which can make the job look like it successfully processed pending builds even though it did no work.
catch (DdbBuildInProgressException ex)
{
// Benign: the lock holder will process the pending builds. See BuildWrapper.
writeLine($"Build lock currently held by another process ({ex.Message}); skipping");
}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
As title says