Can't find labels of Envoy Metrics when use otel as sink

[nealwei2008]

I enable Open Telemetry sink in Evovy Proxy. But I can't find labels of Envoy Metrics when showing the metrics with curl pod-otel-colletion:19001/metrics.
Enable otel just like:

• https://gateway.envoyproxy.io/docs/tasks/observability/proxy-metric/#opentelemetry-metrics

I can find Gateway Metrics with curl pod-envoy-default:19001/metrics:

# curl -s http://10.244.166.159:19001/stats/prometheus | grep envoy_cluster_upstream_cx_connect_ms_sum
envoy_cluster_upstream_cx_connect_ms_sum{envoy_cluster_name="accesslog_otel_0_0"} 0
envoy_cluster_upstream_cx_connect_ms_sum{envoy_cluster_name="default/eg"} 0
envoy_cluster_upstream_cx_connect_ms_sum{envoy_cluster_name="httproute/default/gateway-bases/rule/0"} 0
envoy_cluster_upstream_cx_connect_ms_sum{envoy_cluster_name="httproute/default/nacos/rule/0"} 0
envoy_cluster_upstream_cx_connect_ms_sum{envoy_cluster_name="httproute/demo/demo-hub/rule/0"} 0
...

And find nothing with curl pod-otel-colletion:19001/metrics:

# curl -s http://10.244.166.190:19001/metrics | grep envoy_cluster_upstream_cx_connect_ms_sum

What should I do?

[zirain]

is there any traffic happen? does these metrics changed?

[nealwei2008]

Maybe there's no traffic. I try to get some traffics, and actually found Envoy metrics. But there's also some differences:

1. There's also not have envoy_cluster_upstream_cx_connect_ms_sum in otel metrics
2. The metrics from Envoy pod's prometheuse metrics interface are different from those of Otel-collector pod's metrics interface.

Otel pod

# curl -s http://10.244.166.190:19001/metrics | grep "cluster_upstream_rq_total"
# HELP cluster_upstream_rq_total 
# TYPE cluster_upstream_rq_total counter
cluster_upstream_rq_total{envoy_cluster_name="httproute/default/gateway-bases/rule/0"} 1
cluster_upstream_rq_total{envoy_cluster_name="httproute/monitoring/grafana/rule/0"} 67
cluster_upstream_rq_total{envoy_cluster_name="otel_metric_sink_0"} 71
cluster_upstream_rq_total{envoy_cluster_name="prometheus_stats"} 31
cluster_upstream_rq_total{envoy_cluster_name="xds_cluster"} 1
cluster_upstream_rq_total{envoy_cluster_name="httproute/default/gateway-bases/rule/0",envoy_response_code="200"} 1
cluster_upstream_rq_total{envoy_cluster_name="otel_metric_sink_0",envoy_response_code="200"} 71
cluster_upstream_rq_total{envoy_cluster_name="prometheus_stats",envoy_response_code="200"} 31
cluster_upstream_rq_total{envoy_cluster_name="xds_cluster",envoy_response_code="200"} 1

Envoy pod

# curl -s http://10.244.166.161:19001/stats/prometheus | grep "cluster_upstream_rq_total"
# TYPE envoy_cluster_upstream_rq_total counter
envoy_cluster_upstream_rq_total{envoy_cluster_name="default/eg"} 0
envoy_cluster_upstream_rq_total{envoy_cluster_name="httproute/default/gateway-bases/rule/0"} 1
envoy_cluster_upstream_rq_total{envoy_cluster_name="httproute/default/nacos/rule/0"} 0
envoy_cluster_upstream_rq_total{envoy_cluster_name="httproute/demo/demo-hub/rule/0"} 0
envoy_cluster_upstream_rq_total{envoy_cluster_name="httproute/demo/hostnames/rule/0"} 0
envoy_cluster_upstream_rq_total{envoy_cluster_name="httproute/demo/quickstart-consumer/rule/0"} 0
envoy_cluster_upstream_rq_total{envoy_cluster_name="httproute/monitoring/grafana/rule/0"} 69
envoy_cluster_upstream_rq_total{envoy_cluster_name="httproute/monitoring/prometheus/rule/0"} 0
envoy_cluster_upstream_rq_total{envoy_cluster_name="otel_metric_sink_0"} 74
envoy_cluster_upstream_rq_total{envoy_cluster_name="prometheus_stats"} 33
envoy_cluster_upstream_rq_total{envoy_cluster_name="tcproute/default/redis-tcproute/rule/-1"} 0
envoy_cluster_upstream_rq_total{envoy_cluster_name="tlsroute/kubernetes-dashboard/dashboard/rule/-1"} 0
envoy_cluster_upstream_rq_total{envoy_cluster_name="xds_cluster"} 1

I'm confused that:

• Does Envoy output different metrics from Prometheus and Otel? Where can I find the corresponding document?
• Does Envoy only output metrics to Otel when there is traffic?

[zirain]

there's a different between Push and Pull mode. that's the main cause.

[zirain]

we could add support for prefix in OpenTelemetry stats sink to make the stats output same or you can try to add it in OTel collector.

[nealwei2008]

we could add support for prefix in OpenTelemetry stats sink to make the stats output same or you can try to add it in OTel collector.

it does't seems like the prefix is the only different between them. there's envoy_response_code in otel interface but not in prometheus'.

cluster_upstream_rq_total{envoy_cluster_name="otel_metric_sink_0",envoy_response_code="200"} 71

[zirain]

Maybe there's something I didn't follow recently.

[nealwei2008]

Got it, 3Q