Default OTLP endpoint points at an addon namespace that usually does not exist

[git286]

Default OTLP endpoint points at an addon namespace that usually does not exist

manifests/ate-install/ate-otel-config.yaml defaults to http://opentelemetry-collector.gke-managed-otel.svc.cluster.local:4317. On any cluster without GKE's managed OTel addon enabled - the default gke-managed-otel namespace does not exist, so every component fails to resolve the collector once a minute, forever, and telemetry is silently dead. It presents as a networking fault rather than a missing dependency.

The installer rewrites the ConfigMap, so patching it after install is silently reverted. The override has to go in at install time via ATE_OTLP_ENDPOINT, which hack/install-ate.sh already honours.

Ask: default to no exporter, or make the install refuse when the endpoint's namespace is absent.

[JeffLuoo]

Ask: default to no exporter, or make the install refuse when the endpoint's namespace is absent.

We should make Observability a must-have, despite which OTel collector users use, managed one or customized. I prefer "make the install refuse when the endpoint's namespace is absent", WDYT? Da Huang (@git286)

[git286]

yeah, I think we should have some preflight check in the install script and explicitly tell users what's wrong and what they should do, instead of silently failure

[BenTheElder]

We should make Observability a must-have,

really? If the user doesn't want it, why is this a hard requirement? I could imagine not wanting to collect otel because storing it will add up with a lot of actors and you may already be doing something else at the actor level.

Our components shouldn't even assume that the install scripts are in use.

Ask: default to no exporter, or make the install refuse when the endpoint's namespace is absent.

We need to split out installing for GKE versus installing generally. install-ate.sh is currently primarily for GKE but quasi-pretending to be generic. Instead what we have here must become generic. vendor neutrality is actually a requirement for e.g. CNCF sandbox were we to go that route.

Installing for GKE can expect the GKE namespace, the components need to be well behaved without it though.

[JeffLuoo]

I could imagine not wanting to collect otel because storing it will add up with a lot of actors and you may already be doing something else at the actor level.

We don't have actor level metrics as we also discussed in other threads, but cardinality will grow with ate namespace or template name, which the system metric use as attributes.

We need to split out installing for GKE versus installing generally. install-ate.sh is currently primarily for GKE but quasi-pretending to be generic. Instead what we have here must become generic. vendor neutrality is actually a requirement for e.g. CNCF sandbox were we to go that route.

Thanks for your feedback. It makes sense that we should make monitoring stack optional but recommended. Hence, we can add a flag in the script to make it configurable, and if it's configured, we ask users to provide an OTLP endpoint. If the endpoint is not provided, the script warns or return error.