Skip to content

feat(DPE-10536): COS support for K8s and VM - #131

Open
skourta wants to merge 33 commits into
9/edgefrom
dpe-10536-cos-integration
Open

skourta wants to merge 33 commits into
9/edgefrom
dpe-10536-cos-integration

Conversation

@skourta

@skourta skourta commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Integrate the Canonical Observability Stack (COS) for Valkey on both Kubernetes and VM substrates.

  • Kubernetes telemetry: Adds metrics-endpoint, grafana-dashboard, and logging relation endpoints using MetricsEndpointProvider, GrafanaDashboardProvider, and LogProxyConsumer.
  • VM telemetry: Adds cos-agent subordinate relation using COSAgentProvider (vendoring charms.grafana_agent.v0.cos_agent v0.27) to configure Prometheus metrics scrape endpoints, Loki log targets, and dashboard files.
  • Log collection: On Kubernetes, runs Pebble log tailing services for Valkey and Sentinel logs. On VM, targets snap log paths via cos-agent.
  • Dashboards and alerts: Ships a curated Valkey Grafana dashboard with Juju topology filters, 9 Prometheus alert rules, and 4 Loki alert rules.
  • Testing: Adds comprehensive unit tests for observability providers on both substrates, substrate-separated integration suites (tests/integration/observability/), and Spread task definitions (tests/spread/{k8s,vm}/test_telemetry.py).

Verification

  • tox run -e lint passed.
  • tox run -e static passed with 0 errors.
  • tox run -e unit passed (100% coverage on src/events/observability.py).
  • Integration tests verified live against LXD and Canonical K8s.

K8s Dashboard

valkey-k8s #### VM Dashboard valkey-vm

@skourta
skourta marked this pull request as ready for review September 14, 2026 10:35

@reneradoi reneradoi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor comments/questions. Looks good so far, very nice integration tests!

dns_ip = subprocess.check_output(cmd, shell=True).decode().strip().strip("'\"")
except Exception:
dns_ip = ""
dns_ip = dns_ip or "10.152.183.10"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What magic ip address is this? Can you please add a comment?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

logger.info("Waiting for applications to settle after K8s COS integration")
juju.wait(
lambda s: (
s.apps[APP_NAME].app_status.current == "active"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not are_apps_active_and_agents_idle?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OTELCOL stays blocked until it is fully integrated into a backend. That is why I only used active.

valkey_unit = f"{APP_NAME}/0"
logger.info("Verifying cos-agent relation data for %s", valkey_unit)
raw_config = None
for attempt in Retrying(stop=stop_after_delay(180), wait=wait_fixed(5)):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why 3 minutes here and only 2 in the K8s test?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On VM it raised a couple of times for me. I will unify the tests to 180s.

Comment thread src/cos/grafana_dashboards/valkey.json Outdated
}
]
},
"description": "Redis Dashboard for Prometheus Redis Exporter 1.x",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I know we can't change the metrics themselves, but let's change at least the description from Redis to Valkey.

Comment thread src/managers/metrics.py Outdated
Comment on lines +57 to +62
password = self.state.cluster.internal_users_credentials.get(
CharmUsers.VALKEY_MONITORING.value
)
if not password:
return

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I believe this is duplicate, as one line below we get env = self.exporter_env() and we could do if not env.get(CharmUsers.VALKEY_MONITORING.value): return

Comment thread src/workload_k8s.py Outdated
return False

self.container.add_layer(CHARM, self.pebble_layer, combine=True)
self.container.replan()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm confused why we replan here (which should restart changed services) and restart the service in the caller method in the manager class - I believe these are duplicate restarts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants