why is my cronicle event starting late

[mcastil00]

lately, i've noticed some delays in the execution of my events - ranging from several seconds to even a minute. when that happens, the start date appears in red. i'm curious why that is. i checked the documentation and saw that this could happen if the Catch Up option is enabled, but it's not enabled for any of my events.

[jhuckaby]

Oh wow, this is the first time I've heard of this happening in the wild. That date/time field turning red is basically Cronicle telling you that something is very, very wrong on the server. It means that the job scheduled run time, and the actual start time of the job, were more than 60 seconds off from each other (which is catastrophic).

So something is horribly wrong somewhere. Is your server running low on memory or overloaded on CPU or disk I/O? This situation basically means that Cronicle is getting "stuck" somehow (the Node.js process is freezing up) and is unable to actually launch a job for 60+ seconds after it started trying to do so. Usually this means the OS kernel is intervening because it is overloaded dealing with other things (memory swap, CPU load, I/O wait, etc. etc.).

We may have to dig into the logs to see what is happening here.

[mcastil00]

i only noticed this happening when running 80+ events simultaneously. there’s a noticeable spike in cpu usage on the primary node — going above 20% — which i expected given the number of concurrent processes. still, there’s no i/o wait or memory exhaustion, and the two backup nodes remain unaffected — all three nodes have 8 cpus and 32 gb of memory. nothing stands out in the logs either, so i figured it might be tied to some tunable parameter.

i’ve just increased the log verbosity to 10 (it was previously set to the default), so i’ll monitor it and see if anything new shows up.

since you mentioned this might be an issue, i’m planning to add two more backup nodes — each with 4 cpus and 32 gb of memory. it might simply be due to the volume of events trying to run at once.

i’ll share an update once i’ve set this up.