Normalize the Kubernetes log schema before retiring Loki

[kelchm]

Summary

The VictoriaLogs capture path is complete and healthy, but the post-soak UX audit found that the shared Alloy feed does not expose the canonical fields expected by native VMUI or Grafana. This makes the VictoriaLogs presentation materially less coherent than Loki even though both backends receive the same entries.

This issue is a convergence gate for #470. Retain the current Loki dual-write until the normalization rollout is measured and the VictoriaLogs browsing experience is accepted; do not interpret it as approval for indefinite dual storage.

Evidence from the live audit

• Alloy sent the same 9,239 entries to Loki and VictoriaLogs over the measured ten-minute window, with zero drops and no retry increase.
• A 10,000-row VictoriaLogs sample had no top-level level, service_name, or service_instance fields. 943 rows contained structured severity only as the collision-safe msg.level field, which VMUI does not recognize.
• Over a one-hour query, 2,630 rows had msg.level: 2,621 info and 9 warn. Valid structured severity is therefore present but hidden from VMUI.
• VictoriaLogs v1.52.0 VMUI recognizes canonical top-level fields such as level, severity_text, and detected_level; msg.level is outside that list, so the affected entries render as OTHER.
• Kubernetes application identity is available for 173 of 184 current containers through app.kubernetes.io/name or legacy app; every generic container="app" workload has a usable application name. Container fallback covers the remainder.
• A recent 9,306-row sample contained 253 ANSI-colored rows. Global decolorization is safe after CRI reconstruction and improves both retained sinks.
• Apparent continuation-line candidates were dominated by intentional iperf separator output. There is no evidence supporting a global multiline rule.
• Loki improves presentation by synthesizing service_name and detected_level, but its plaintext detector uses broad substring heuristics. Copying that inference into stored VictoriaLogs fields would create confidently wrong severity.
• Grafana Logs Drilldown remains Loki-query-API-specific. VictoriaLogs can improve standard Explore and native VMUI, but cannot become a Drilldown datasource through configuration alone.

Schema decision

Use a small, stable envelope while preserving application payload fields under msg.*:

• Stream identity: cluster, namespace, service_name, pod, container, node.
• Event fields: _time, _msg, stream, filename, normalized level, plus application fields under msg.*.
• Derive service_name from app.kubernetes.io/name, then legacy app, then container.
• Preserve app.kubernetes.io/instance separately as app_instance when present. It is commonly a Helm/release identity and must not be mislabeled as the OpenTelemetry service.instance.id.
• Normalize only explicit JSON or credible logfmt severity values into Grafana's canonical set trace, debug, info, warning, error, and critical. Fold fatal/panic aliases into critical while retaining the original application payload.
• Keep level, request IDs, trace IDs, users, and other per-event values out of VictoriaLogs stream identity.
• Do not add global plaintext severity inference or global multiline assembly.

Phase 1: platform baseline

• Decode CRI records, then strip ANSI escape sequences in Alloy.
• Derive service_name and optional app_instance from Kubernetes discovery metadata with the documented fallbacks.
• Extract JSON and credible logfmt severity, normalize only recognized aliases, and expose canonical top-level level without removing the original msg.level or raw message.
• Add service_name to the explicit VictoriaLogs stream-field set; leave level and app_instance as ordinary fields.
• Provision exact Grafana compatibility rules for historical msg.level rows retained before the schema change.
• Set the external vmuiUrl so Grafana's Run in VMUI action carries the current query and time range to https://vlogs.home.kelch.io/select/vmui/ instead of an unreachable cluster DNS URL.
• Explicitly provision the 1,000-line query default and pin the VictoriaLogs datasource plugin to the reviewed 0.31.0 release.
• Switch Grafana console logs and all three Traefik instances to their documented JSON formats.
• Update the logging runbook with the canonical fields, query examples, UI handoff, and explicit non-goals.

Rollout acceptance

• Flux and all affected Alloy, Grafana, and Traefik workloads are Ready after rollout.
• Loki and VictoriaLogs sent-entry deltas remain equal during the validation window; drops remain zero and retries do not increase.
• Current entries from all three nodes arrive in VictoriaLogs.
• VMUI displays recognized severities for structured/logfmt emitters and offers service_name as a stream facet.
• Grafana Explore severity colors and filters work for both new level fields and retained historical msg.level data.
• Grafana's Run in VMUI action opens the external authenticated VMUI with the query and time range preserved.
• Representative structured JSON, logfmt, and plain-text workloads retain readable _msg values and application fields.
• No ANSI escapes remain in newly ingested entries.
• VictoriaLogs stream creation stays within the existing alert envelope and there is no material series/log duplication beyond the intentional dual-write soak.
• Loki removal remains blocked until the normalized VictoriaLogs experience is explicitly accepted.

Phase 2: measured source inventory

After the platform baseline, inventory the remaining high-volume/plain-text emitters and enable native structured output only where the application documents it. Track noisy-log reduction and secret/PII redaction separately rather than adding generic collector regexes. A small Git-managed Logs Overview dashboard is optional follow-up; do not turn this issue into broad dashboard curation tracked by #50.

Related

• Execute the observability convergence or formally supersede the bake-off decision #470 — observability convergence and single-backend acceptance
• Pre-baked Grafana dashboards: UID collisions and post-convergence gaps #50 — final post-convergence dashboard inventory; remains on hold
• Wire actionable alert delivery and restore control-plane metric coverage #218 — independent dead-man monitoring and stale-backup delivery proof; unchanged