fix(metrics): resolve pods to their Deployment, not their ReplicaSet

[mayankpande88]

Problem

Relationship metrics publish two identities for the same workload — the Deployment name, and the ReplicaSet name with its pod-template-hash:

name	namespace	kind	series
llm-server	nudgebee	Deployment	2428
llm-server-779f867dc9	nudgebee	ReplicaSet	181
llm-server-78c94f7dfd	nudgebee	ReplicaSet	122

Measured on dev-gke: 62% of container_http_requests_duration_seconds_total_bucket series carry kind=ReplicaSet (13,620 via src_workload_kind, 1,236 via destination_workload_kind), across 77 distinct ReplicaSet names.

Two consequences:

• Every rollout mints a fresh set of series under the new hash that never collapse back — a real contributor to TSDB cardinality.
• Alert rules aggregating by destination_workload_name fire once per variant. This surfaced as duplicate High P95 latency for llm-server / ...for llm-server-779f867dc9 events at the same second.

Affects container_net_latency_seconds, all container_net_tcp_*, and all 15 L7 protocol families (src / destination / actual_destination).

Root cause

resolvePodDescriptor intends to climb Pod -> ReplicaSet -> Deployment, but:

1. The climb is skipped on any error. getControllerOfOwner errors when the owner is absent from the informer snapshot — informers not yet synced, or the ReplicaSet pruned by revisionHistoryLimit while its pods still run. name then stays as the ReplicaSet.

2. The wrong answer was cached forever.

if owner, err := resolver.getControllerOfOwner(&owner); err == nil && owner != nil {

:= declares a new err scoped to the if. The outer err was never assigned, so the trailing if err == nil { PodDescriptors.Store(...) } was always true. A transient informer miss got memoized under the pod UID and returned for the rest of the pod's life — which is why this never self-healed and why both names coexist steadily.

Changes

• Replace the shadowed err with an explicit cacheable flag; only memoize a fully-resolved chain.
• Distinguish errOwnerNotCached (transient -> don't cache, retry next scrape) from errUnsupportedOwnerKind (terminal -> still cache). This matters: Elasticsearch, OpenTelemetryCollector, VMAgent, VMAlertmanager and Runner all appear as owner kinds in this cluster and would otherwise re-walk the chain on every scrape.
• stripPodTemplateHash fallback for a genuinely unresolvable ReplicaSet. Conservative by design — only strips a final segment matching Kubernetes' vowel-free hash alphabet [bcdfghjklmnpqrstvwxz2456789]{6,11}, so a bare ReplicaSet (replicaset1, my-app-frontend) is never renamed into a Deployment that doesn't exist.
• NewDestinationKey no longer overwrites a k8s-resolved workload's Name with the DNS FQDN. That produced mixed-provenance labels (FQDN name + real namespace + real kind), 173 series. Gated on Kind rather than name so genuinely external destinations still get the FQDN — which the 11 external-service discovery queries in api-server/services/application/discovery.go depend on.
• ResolvePodOwner's API-server fallback returned Kind: "Pod" while resolvePodDescriptor returns "pod". Aligned to lowercase.

Deliberately not changed

kind casing is inconsistent (pod/node/external lowercase vs Deployment/ReplicaSet PascalCase), but "external" is pinned by 11 production queries. Normalising it would silently break external-service discovery for postgres, mysql, mongo, clickhouse, cassandra, zookeeper, redis, memcached, rabbitmq, kafka and nats. The lowercase values are sentinels meaning "not a k8s owner kind" — a real distinction, not an accident.

Tests

New common/ip_resolver_workload_identity_test.go and common/net_workload_identity_test.go — 11 test functions, 19 table cases:

• full chain resolves to Deployment and is cached
• transient miss is not cached (the regression)
• recovers to Deployment on the next call once informers sync
• unsupported owner kind is terminal and still cached
• bare ReplicaSet keeps its own name
• external / unresolved destinations keep the FQDN; resolved ones keep their workload name

Full ./common/ suite passes, including pre-existing TestDestinationKey and "pod controlled by replicaset resolve to replicaset". go vet, gofmt, and GOOS=linux GOARCH=amd64 build all clean.

Rollout note

This fixes new series only. Pods already running keep their cached identity until replaced — natural churn on spot nodes, or kubectl rollout restart to clear immediately. Historical series age out with retention.

[gemini-code-assist]

Code Review

This pull request improves Kubernetes workload identity resolution by ensuring that transient owner-resolution failures (e.g., when an owner is missing from the informer cache) are not cached, preventing pods from being permanently pinned to intermediate identities like ReplicaSets. It also standardizes pod kinds to lowercase, introduces a fallback to strip pod-template hashes from ReplicaSet names, and ensures that FQDNs do not overwrite fully-resolved Kubernetes workload names in destination keys. The reviewer suggested adding a depth limit to the ownership climb loop to prevent potential infinite loops from circular owner references.