4.22.0-okd-scos.5 - chrony-wait.service fails during node boot

[ensar-bht]

Describe the bug

On OKD nodes, chrony-wait.service consistently enters failed state during boot because it reaches its 180 seconds timeout before chronyd selects a valid NTP source.

The node eventually synchronizes time successfully a few seconds after chrony-wait.service fails.

This results in a persistent failed systemd unit after every node reboot:

chrony-wait.service

The failure does not currently impact cluster functionality, but it creates false-positive failed systemd state on all nodes.

Environment

• Node role: workers + masters
• Runtime: systemd + chrony
• NTP source: internal NTP server

Version

4.22.0-okd-scos.5

Reproducibility

100%

Log Bundle

Current behavior

During boot:

Jun 29 09:06:49 chronyd.service started
Jun 29 09:06:49 chrony-wait.service started

chrony initially cannot select the NTP source:

Jun 29 09:06:50 chronyd[1532]: Source 195.222.32.129 offline
Jun 29 09:07:00 chronyd[1532]: Source 195.222.32.129 offline

After 180 seconds:

Jun 29 09:09:50 chrony-wait.service:
start operation timed out

However, immediately afterwards chronyd successfully synchronizes:

Jun 29 09:10:03 chronyd[1532]:
Selected source 195.222.32.129

Current synchronization status:

chronyc tracking

Reference ID : 195.222.32.129
Stratum : 3
Leap status : Normal

and:

chronyc sources -v

^* 195.222.32.129

Additional information

chrony.conf contains:

server 195.222.32.129
sourcedir /run/chrony-dhcp

so the DHCP NTP sourcedir issue does not apply.

NetworkManager is already online before the timeout:

NetworkManager-wait-online.service:
Active: active (exited)

Systemd blame:

3min 81ms chrony-wait.service

Expected behavior

chrony-wait.service should complete successfully if chronyd is able to synchronize shortly after boot.

Alternatively, the service ordering or timeout should allow enough time for chronyd to select an available source.

Possible root cause

This looks like a boot ordering/race condition:

system boot
|
chronyd starts
|
chrony-wait starts
|
NTP source temporarily unavailable
|
180s timeout reached
|
chrony-wait fails
|
chronyd synchronizes shortly afterwards

Impact

No functional cluster impact observed.

After boot completion:

• node time is synchronized
• chronyd is healthy
• NTP source is reachable

However, every reboot leaves nodes with failed chrony-wait.service, causing misleading systemd health state.

Question

Should chrony-wait.service be adjusted in OKD nodes, for example:

• different ordering dependency
• increased timeout
• better integration with NetworkManager-online state
• or should this be handled differently in OKD node bootstrap?

[JaimeMagiera]

Sorry for the delay. We have seen this previously. I'll gather more details to see what options can realistically be implemented short and long term.