ACM-39327: Fix non-admin SSE OOM under large inventory (release-2.13)

[Ginxo]

Summary

• Manual backport of the RBAC/SelfSubjectRulesReview fix from ACM-44885: Fix non-admin SSE OOM under large inventory #6638 to release-2.13 (fast-forwards to backplane-2.8).
• After cluster-scoped list is denied, use one SelfSubjectRulesReview per token/kind instead of O(namespaces) namespaced list + O(N) per-object SSARs, which OOMs/hangs restricted users under large inventory (MOCK_CLUSTERS / real fleets).
• Harden the access cache: SHA-256 token keys, verb in cache keys, TTL cleanup, per-token entry cap.
• SSRR HTTP failures are marked unavailable and fall back to SSAR (not cached as deny-all).

Not ported (not present on 2.13): event-body compression, meta on cached events, and filter-before-inflate in server-side-events.ts. Those half of #6638 do not apply here; the O(N) SSAR path does.

Related to #6638

Test plan

• backend unit tests for access cache + SSRR short-circuit (npm test -- --testPathPattern=test/routes/events.test.ts)
• npm run check in backend
• Local: MOCK_CLUSTERS=1000, kubeadmin vs restricted/none on Inventory; confirm backend RSS/CPU stay bounded
• Confirm admin Inventory still populates clusters normally
• Confirm none completes SSE load without backend hang

Test support

I normally use

watch -n 1 "ps -p $(fuser 4000/tcp 2>/dev/null) -o pid,%cpu,%mem,rss,cmd"

for checking the CPU/MEM consumption on the local environment

Test results

I can confirm the situation is even worse for 2.13 and the memory grows up out of control for non-admin users. This PR is totally mandatory for 2.13

Test commands

Image generation

git checkout bug/ACM-39327-release-2.13

USERNAME=YOUR_USER_NAME
TAG="acm39327-6648-$(date +%Y%m%d-%H%M%S)"
IMAGE="quay.io/${USERNAME}/console-mce:${TAG}"

podman login registry.redhat.io
podman login quay.io

podman build --arch amd64 -f Containerfile.mce.konflux -t "${IMAGE}" .
podman push "${IMAGE}"

Deployment

./scripts/patch-deployment.sh "${TAG}" quay.io/${USERNAME}/console-mce
oc -n multicluster-engine rollout status deployment/console-mce-console
# Verify                                                     
oc get deployment console-mce-console -n multicluster-engine \
  -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
oc get pods -n multicluster-engine -l app=console-mce

# Additional verification
POD=$(oc get pods -n multicluster-engine -l app=console-mce \
  -o jsonpath='{.items[0].metadata.name}')

oc get pod "$POD" -n multicluster-engine \
  -o jsonpath='{.spec.containers[0].image}{"\n"}'

and the output should be something like quay.io/emingora/console-mce:acm39327-6648-20260818-105951

Test execution

• uncompress attached file acm-39327-repro.zip to /tmp

cd /tmp/acm-39327-repro
source ./env.sh
./gate.sh

we should see something like

./gate.sh 
GATE_A: MC=25 MCI=25 MCA=200 POL=3 NS=25  → need 25/25/200/3/25
GATE_B: CD=25 MP=50 CP=25 SEC=150           → need 25/50/25/≥100
GATE_C: IE=25 AG=250 BMH=250 NMC=250 ACI=25 AM=0 → need 25/250/250/250/25/0
GATE_D: PR=25 CSR=25                          → need 25/25
LABELED_TOTAL=1378 (target ≥1329 or ≥1079 without AgentMachine)
GATE: PASS

otherwise execute

source ./env.sh 
./apply-layer-d.sh
./gate.sh 

Execute the resource watcher in a terminal

WORKDIR=/tmp/acm-39327-repro
START_TS=$(date +%Y%m%dT%H%M%S)
LOG="$WORKDIR/acm39327-console-mce-${START_TS}"
MCE_NS=$(oc get pods -A -l app=console-mce -o jsonpath='{.items[0].metadata.namespace}')

printf 'time\tsource\tpod\tcpu\tmemory\n' > "$LOG"

( while true; do
    TS=$(date -Is)
    while read -r name cpu mem; do
      [[ "$name" == "NAME" || -z "$name" ]] && continue
      printf '%s\ttop\t%s\t%s\t%s\n' "$TS" "$name" "$cpu" "$mem" >> "$LOG"
    done < <(oc adm top pod -n "$MCE_NS" -l app=console-mce --no-headers 2>/dev/null)
    sleep 2
  done ) & TOP_PID=$!

( while true; do
    TS=$(date -Is)
    POD=$(oc get pods -n "$MCE_NS" -l app=console-mce -o jsonpath='{.items[0].metadata.name}' 2>/dev/null)
    if [[ -n "$POD" ]]; then
      BYTES=$(oc exec -n "$MCE_NS" "$POD" -- sh -c \
        'cat /sys/fs/cgroup/memory.current 2>/dev/null || cat /sys/fs/cgroup/memory/memory.usage_in_bytes' 2>/dev/null)
      [[ -n "$BYTES" ]] && printf '%s\tcgroup\t%s\t-\t%.1fMi\n' "$TS" "$POD" "$(awk -v b="$BYTES" 'BEGIN{print b/1024/1024}')" >> "$LOG"
    fi
    sleep 2
  done ) & CGROUP_PID=$!

echo "Watchers: top=$TOP_PID cgroup=$CGROUP_PID  log=$LOG"
tail -f "$LOG"

Note: kill it once it is over by executing kill $TOP_PID $CGROUP_PID

and then start the test execution on another terminal

export CONSOLE_URL=$(oc get route -A -o json | jq -r '.items[] | select(.metadata.name|test("console")) | "https://\(.spec.host)"' | head -1)
export ACM_NONE_PASS='Acm39327!'
export ACM_IDP_NAME='acm39327-htpasswd'
export ACM_NONE_USERS="$(seq -w 1 20 | tr '\n' ' ')"

node ./acm39327-host-inventory.mjs

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository: stolostron/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Advanced

Run ID: b8565f94-e16a-492b-8b7b-46496186d101

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

_{Comment @coderabbitai help to get the list of available commands.}

[Ginxo]

/hold

[Ginxo]

/unhold

[KevinFCormier]

There are a couple issues with the logic, so fixing them an maintaining the performance gain may be tricky.

• Checking permissions in the default namespace is used a proxy for checking cluster-scoped resource access, but many resources are not cluster-scoped, and there is no code guaranteeing that this check is used only for cluster-scoped resources. I granted user user1 permission to get secrets in the default namespace, but then the user is able to see all Credentials. This is major security hole.
• I granted user1 cluster-admin access for the acm39327-mc-01 cluster. With the old code, it shows up in the cluster list. With your new code, it does not.

Other areas for improvement - caching keys include the API version, but permissions checking is only done against groups. I think you could drop the version part earlier.

[Ginxo]

There are a couple issues with the logic, so fixing them an maintaining the performance gain may be tricky.

• Checking permissions in the default namespace is used a proxy for checking cluster-scoped resource access, but many resources are not cluster-scoped, and there is no code guaranteeing that this check is used only for cluster-scoped resources. I granted user user1 permission to get secrets in the default namespace, but then the user is able to see all Credentials. This is major security hole.
• I granted user1 cluster-admin access for the acm39327-mc-01 cluster. With the old code, it shows up in the cluster list. With your new code, it does not.

Other areas for improvement - caching keys include the API version, but permissions checking is only done against groups. I think you could drop the version part earlier.

What was wrong

SelfSubjectRulesReview is namespaced. The old code always queried default and then applied allow-all to every resource of that kind. Kubernetes does not tell you whether a rule came from a ClusterRoleBinding or a RoleBinding, so:

• get secrets in default made every Credential visible
• cluster-admin in acm39327-mc-01 never showed up in that review, so namespaced cluster objects (ManagedClusterInfo, ClusterDeployment, …) were denied, and the cluster list is the union of those, not only ManagedCluster

What changed

• Namespaced resources: SelfSubjectRulesReview in that resource’s namespace, cached per token + namespace
• Cluster-scoped resources: default is only a probe. Empty rules still deny with no SelfSubjectAccessReview (none-user fast path). Unrestricted grants are confirmed with SelfSubjectAccessReview so a RoleBinding in default cannot impersonate cluster access
• Cache keys use API group + plural, not apiVersion (Placement v1alpha1 / v1beta1 share one entry)
• eventFilter now calls canGetResource, which answers @KevinFCormier "this is only used in tests" comment

[Ginxo]

/test Red Hat Konflux / console-acm-213-on-pull-request

[Ginxo]

/test images

[KevinFCormier]

/hold

[KevinFCormier]

I discovered a new issue while reviewing the main PR. Please see inline comments.

[Ginxo]

@KevinFCormier your cluster scope feedback is in on both PRs.

eventsCache.ts / eventsAccess.ts are the same on main and release-2.13. Only events.ts differs by branch (main keeps compression/meta/SSE optimizations).

Changes:

• clusterScoped on watch definitions (no drifting hardcoded list)
• namespaced kinds no longer treated as cluster-scoped
• cluster-scoped grants still SSAR-confirmed
• SSAR keys include API group
• failed SSRR no longer cached as incomplete.

[KevinFCormier]

This seems overly complicated. I don't follow why in eventsAccess.ts you have the clusterScopedKinds defined as a let, and then you are depending on the side effect of loading this file to call the configureClusterScopedKinds function. And in the tests, you are testing with a subset of cluster-scoped kinds. Wouldn't it be more straightforward to do something like this?

Suggested change

	export const CLUSTER_SCOPED_KINDS = new Set(definitions.filter((definition) => definition.clusterScoped).map((definition) => definition.kind))
	)

If you're trying to avoid a circular dependency between events.ts and eventsAccess.ts, maybe we just need to move the definitions to their own file?

[Ginxo]

what do you think about 0a66bdb
as soon as we agree on this I will forward/backport it to the rest of versions (2.14-5.1)

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
86.7% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Ginxo, KevinFCormier

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [KevinFCormier]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment