OVS fails to start after OKD 4.14 -> 4.20 upgrade due to `openvswitch` UID mismatch

[emmanuelreyes97]

Describe the bug

Summary

After upgrading an OKD cluster from 4.14 to 4.20, some nodes may fail to start ovsdb-server.service.

The affected node has inconsistent openvswitch account information:

• /etc/passwd resolves openvswitch to UID 977
• NSS altfiles resolves openvswitch to UID 800
• /etc/openvswitch is owned by UID 800
• ovsdb-server resolves the openvswitch service account to UID 977

As a result, ovsdb-server cannot create /etc/openvswitch/conf.db or its lock file.

Environment

• OKD upgrade: 4.14 → 4.20
• Open vSwitch: openvswitch3.5-3.5.0-6.el10s.x86_64

The issue was observed following the OKD 4.14 -> 4.20 upgrade.

Symptoms

ovsdb-server.service fails with:

/etc/openvswitch/conf.db does not exist
/etc/openvswitch/.conf.db.~lock~: failed to open lock file: Permission denied
ovsdb-tool: I/O error: /etc/openvswitch/conf.db: failed to lock lockfile (Resource temporarily unavailable)
Creating empty database /etc/openvswitch/conf.db ... failed!

The service also reports:

chown: cannot access '/run/openvswitch': No such file or directory

The missing /run/openvswitch directory does not appear to be the primary failure, because the working node also initially reports the directory as missing during startup and subsequently starts successfully.

Account mismatch

On the affected node:

$ cat /etc/passwd
root:x:0:0:Super User:/root:/bin/bash
core:x:1000:1000:CoreOS Admin:/var/home/core:/bin/bash
openvswitch:x:977:977:Open vSwitch Daemons:/:/usr/sbin/nologin
unbound:x:976:976:Unbound DNS resolver:/var/lib/unbound:/usr/sbin/nologin

However:

$ getent passwd 800
openvswitch:x:800:800::/:/sbin/nologin

The NSS configuration is:

$ grep '^passwd:' /etc/nsswitch.conf
passwd:     files altfiles sss systemd

The UID 800 entry comes specifically from altfiles:

$ getent -s files passwd 800

$ getent -s altfiles passwd 800
openvswitch:x:800:800::/:/sbin/nologin

Meanwhile:

$ getent passwd openvswitch
openvswitch:x:977:977:Open vSwitch Daemons:/:/usr/sbin/nologin

Therefore the node effectively has two different openvswitch identities depending on the NSS lookup:

/etc/passwd:
    openvswitch -> UID 977

NSS altfiles:
    openvswitch -> UID 800

Filesystem ownership

/etc/openvswitch is owned by UID 800:

$ sudo stat -c '%n uid=%u gid=%g mode=%a type=%F' /etc/openvswitch
/etc/openvswitch uid=800 gid=800 mode=755 type=directory

$ sudo stat -c '%n uid=%u gid=%g mode=%a type=%F' /etc/openvswitch/default.conf
/etc/openvswitch/default.conf uid=800 gid=800 mode=644 type=regular file

But:

$ id openvswitch
uid=977(openvswitch) gid=977(openvswitch) groups=977(openvswitch),978(hugetlbfs),801(hugetlbfs)

Consequently:

$ sudo -u openvswitch touch /etc/openvswitch/testfile
touch: cannot touch '/etc/openvswitch/testfile': Permission denied

This prevents ovsdb-server from creating its database and lock file.

Working node comparison

A working node has the expected UID 800 identity:

$ id openvswitch
uid=800(openvswitch) gid=800(openvswitch) groups=800(openvswitch),801(hugetlbfs)

$ getent passwd openvswitch
openvswitch:x:800:800::/:/sbin/nologin

Both nodes have the same OVS package:

openvswitch3.5-3.5.0-6.el10s.x86_64

SELinux

SELinux is enforcing:

$ getenforce
Enforcing

The OVS directory has the expected context:

$ matchpathcon /etc/openvswitch
/etc/openvswitch system_u:object_r:openvswitch_rw_t:s0

Running:

$ sudo restorecon -Rv /etc/openvswitch

does not resolve the problem.

No relevant OVS SELinux AVC denial was found.

OVS configuration

The service configuration specifies:

OVS_USER_ID="openvswitch:hugetlbfs"

The expected OVS configuration therefore assumes the openvswitch account resolves to the same UID that owns the OVS configuration/database directory.

Expected behavior

After upgrading from OKD 4.14 to 4.20, the openvswitch service account should remain consistent across:

• /etc/passwd
• NSS altfiles
• filesystem ownership
• OVS service configuration

The openvswitch daemon should be able to create:

/etc/openvswitch/conf.db
/etc/openvswitch/.conf.db.~lock~

Actual behavior

Following the 4.14 → 4.20 upgrade, the affected node has:

/etc/passwd              -> openvswitch UID 977
NSS altfiles             -> openvswitch UID 800
/etc/openvswitch         -> UID 800
ovsdb-server             -> UID 977

This results in:

ovsdb-server
    |
    | runs as UID 977
    v
/etc/openvswitch
    |
    | owned by UID 800
    X
Permission denied

Questions

1. What changed between OKD 4.14 and 4.20 that can result in the openvswitch account being assigned UID 977 while an altfiles entry still provides UID 800?
2. Is there a known migration issue involving libnss-altfiles, system users, or OVS package transitions during the 4.14 → 4.20 upgrade?
3. Which component is responsible for the UID 800 openvswitch entry in altfiles?
4. What is the supported remediation for nodes that already have this inconsistent state?
5. Should the upgrade process reconcile the openvswitch UID and existing /etc/openvswitch file ownership?
6. Could this affect only nodes that had an older OVS/system-account configuration before the upgrade?

Reproduction / investigation

The key difference can be demonstrated with:

$ getent -s files passwd 800

$ getent -s altfiles passwd 800
openvswitch:x:800:800::/:/sbin/nologin

$ getent passwd openvswitch
openvswitch:x:977:977:Open vSwitch Daemons:/:/usr/sbin/nologin

This suggests that the upgrade leaves behind an openvswitch identity in altfiles with UID 800 while the current /etc/passwd definition uses UID 977.

Impact

Nodes affected by this condition cannot start ovsdb-server, which prevents the Open vSwitch stack from initializing correctly on those nodes.

The issue is particularly problematic because the error presented by ovsdb-tool appears as a lock/resource error:

failed to lock lockfile (Resource temporarily unavailable)

while the underlying cause is that the daemon cannot create the lock file due to the UID mismatch.

Version

4.20.0-okd-scos.17

Reproducibility

100% when following exact upgrade path

Log Bundle

No response