Proposal: Separate Security vs. Feature Releases to Reduce Upgrade Downtime

[sktnetwork]

Hello everyone,

I know a bit crazy opinion, but please hear me out.

AI Summary / TL;DR: The author is transitioning to XyOps after years of using Cronicle, but is concerned about frequent updates (2–4 times per week) causing roughly 5 minutes of downtime per restart due to server reboot requirements. To avoid disrupting time-sensitive cron jobs and uptime services, they propose creating separate release channels or tag-marking releases specifically for security fixes versus general feature updates. This would allow production environments to stay on stable builds unless a critical security update is required.

Now, well, we're using Cronicle in production for past 5 years or so. Works great, and had been waiting for Cronicle v2 / Orchestra / XyOps. Great that it's here now for last few months.

One thing I really notice is that while XyOps gets a lot of features get added frequently, there are like 2-4 releases every week. Cronicle had like 2-4 releases a month. Everytime, I still update Cronicle, the service has to be restarted, which it almost always botches up (because it can't stop the server in time), and I have to usually reboot the server. There goes about 3-5 minutes of downtime of upgrading Cronicle.

While 5 minutes of downtime may not seem much, it often is when you're running crons that have to run every-minute, or run services that do uptime-management, and so on.

Looking at XyOps releases, I believe it's pretty stable, but also believe that this would mean that I have to kind of update it multiple times a week, and have that 5 minutes of downtime.

Possible solution: Maybe, have two release-branches of XyOps (security and general) or have at-least proper tags on whether the release also fixes a security vulnerability (on itself and/or dependencies) or just a feature release.

e.g. looking at the release notes, I see Version v1.0.76 is the last one that mentions anything about security. Right now, latest release is v1.0.85. So, would it be safe to assume that if nothing is broken, and we need an LTS/Stable kind of environment, we shouldn't update from v1.0.76 unless there's a release that also fixes security stuff.

Any other idea or feedback is also welcome.

[jhuckaby]

The author is transitioning to XyOps after years of using Cronicle, but is concerned about frequent updates (2–4 times per week) causing roughly 5 minutes of downtime per restart due to server reboot requirements.

Hmmm... xyOps should have effectively zero downtime for upgrades. The upgrade process takes mere seconds, and no jobs are interrupted while it happens. All job updates are automatically queued up while the service restarts. I don't understand where this "5 minutes" of downtime comes from. Can you elaborate?

xyOps Upgrade Behavior

It is generally recommended to schedule upgrades during a time window when no jobs are running. However, if you have jobs running constantly 24x7, here is what you can expect:

Conductor Upgrade Behavior

If you have a multi-conductor setup:

• The primary conductor runs upgrades of all peer conductors first, with a "stagger" delay (default 60 seconds).
• After those are complete, it upgrades itself.
• Primary conductor downtime is usually around 5 seconds.
• Jobs are not paused or aborted for any upgrade operations.
  • xySat handles this downtime seamlessly by "holding" all job updates and completions until it reconnects to a primary.
• If the primary upgrade takes longer than 10 seconds, an election is held and a backup conductor takes over.
  • xySat handles this as well, and reconnects to the new primary and resumes job updates and completions.
• If any jobs were scheduled to run during the downtime, behavior depends on the event settings:
  • If the event has the Catch-Up modifier, it will run any missed jobs after the upgrade completes.
  • Otherwise, the job will be skipped until the next scheduled run.

If you have a single-conductor setup:

• Primary conductor downtime is usually around 5 seconds.
• Jobs are not paused or aborted for any upgrade operations.
  • xySat handles this downtime seamlessly by "holding" all job updates and completions until it reconnects.
• If any jobs were scheduled to run during the downtime, behavior depends on the event settings:
  • If the event has the Catch-Up modifier, it will run any missed jobs after the upgrade completes.
  • Otherwise, the job will be skipped until the next scheduled run.

Server Upgrade Behavior

For upgrading xySat on worker servers:

• Servers are upgraded in sequence, with a "stagger" delay (default 60 seconds).
• For each server, the upgrade request is "queued" until all jobs are complete on the server.
• No jobs are aborted for upgrades.
• Only an idle server will execute an upgrade request.
• Downtime for the server is usually around 5 seconds.
• If jobs are scheduled to run on the server during downtime:
  • If the event is targeted at a server group (recommended), xyOps automatically picks a different server to run the job.
  • If the event is targeted directly at the server, or the group only has one server, and Queuing is enabled, the job will be queued until the server comes back online.
  • If the event has Retries configured (recommended), those kick in, and the job will keep trying until the server comes back online, or the retry limit is hit.
  • Otherwise, the job will fail and show as "aborted" with the error message: "No available servers matching targets."

[sktnetwork]

Thank you @jhuckaby for explanation.

Regarding 5 minutes of downtime, I was talking about Cronicle.

Everytime, I still update Cronicle, the service has to be restarted, which it almost always botches up (because it can't stop the server in time), and I have to usually reboot the server. There goes about 3-5 minutes of downtime of upgrading Cronicle.

I was hesitant to switch to XyOps because I thought I would have to undergo same routine as I have to do for Cronicle. As I understand now, Orchestra XyOps is built from the ground-up and uses a pretty advanced background update stuff, I am pretty happy to note that, and I guess it's time to actually try it out on a staging server.

Thank you again for taking the time to reply, and quickly resolving my apprehension. Much appreciated.

[jhuckaby]

All that being said, xyOps release cadence will eventually slow down to monthly. They are frequent now because the project is still quite young, and under highly active development.