feat(claude-code): OTLP governance receiver + Loki store#404
Merged
Conversation
Deploy an OpenTelemetry Collector that receives Claude Code telemetry over OTLP at cc.i.shion1305.com (internal Gateway, WireGuard-only) and persists conversation/usage data for governance. The endpoint is protected by a bearer token validated by the collector's bearertokenauth extension; the token is sourced from Vault (dedicated claude-code/ KV mount) via ESO. - Collector (opentelemetry-collector chart, contrib image): OTLP/HTTP only, logs -> Loki, metrics -> Prometheus via ServiceMonitor. - Loki (single-binary, filesystem): 90-day retention on longhorn-hdd (100Gi), queryable in Grafana through a new Loki datasource. - eso-claude-code policy/role added to setup-eso-policies.sh. No custom NetworkPolicy needed: existing allow-from-infra + the Kyverno cross-namespace isolation generator already cover Gateway->collector, Prometheus scrape, and same-namespace collector<->Loki. Signed-off-by: Shion Ichikawa <shion1305@gmail.com>
Roll the vault CLI tool back from v2.0.1 to v2.0.0. Signed-off-by: Shion Ichikawa <shion1305@gmail.com>
Cross-namespace ingress is default-denied (Kyverno generator); egress is open. The only cluster-wide ingress allows are node/apiserver, envoy-gateway-system, and the grafana namespace. Spell out the "who needs to reach this app on ingress?" rule so new apps don't silently rely on, or miss, a cross-namespace allow. Signed-off-by: Shion Ichikawa <shion1305@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a Claude Code OTLP receiver for governance: an OpenTelemetry Collector
published on the internal Gateway at
cc.i.shion1305.com, persisting Claude Codetelemetry to a new single-binary Loki store.
Changes
apps/claude-code-app.yaml+claude-code/— opentelemetry-collector chart(contrib image; core lacks
bearertokenauth). OTLP/HTTP only (HTTPRoute isHTTP-only; gRPC would need a GRPCRoute).
logs → Loki,metrics → Prometheus.HTTPRoute + ReferenceGrant on the internal Gateway, ESO SecretStore/ExternalSecret
for the bearer token, ServiceMonitor, README.
apps/loki-app.yaml+loki/values.yaml— Loki 7.0.0, SingleBinary,filesystem, 90-day retention on
longhorn-hdd(100Gi).grafana/datasource-loki.yaml— cross-namespace Loki datasource (stableuid: loki).vault/scripts/setup-eso-policies.sh—eso-claude-codepolicy + role(dedicated
claude-code/KV mount).aqua.yaml— pinhashicorp/vaultCLI to v2.0.0 (separate, unrelated change).Networking
No custom NetworkPolicy required — the cluster-wide
allow-from-infrapoliciesalready permit Envoy Gateway → collector and Prometheus scrapes, and the Kyverno
cross-namespace isolation generator allows same-namespace collector ↔ Loki.
What it captures
Claude Code's OTel event stream (
user_prompt,tool_result,tool_decision,api_request,api_error) + metrics (token/cost/session/active-time). Not fullassistant responses. Prompt content only when the client sets
OTEL_LOG_USER_PROMPTS=1.Manual steps before it goes Healthy
claude-code-otlp-tokenSecret syncs.cc.i.shion1305.comresolves to the internal Gateway (10.130.5.21;the
*.i.shion1305.comwildcard + cert should cover it).Validation
scripts/render-validate.shpasses for all apps including the newclaude-codeand
loki.