Skip to content

Coroot returns no logs or traces older than a few hours despite a 7-day retention #6143

Open

Description

@devantler

🤖 Generated by the Agentic Engineer

Evidence

Read-only checks through Coroot's API on 12 November at about 19:40 UTC:

Signal Window Result
Logs, whole project last 3 hours Present, newest-first as expected
Logs, whole project 30 days ago to 1 day ago 0 entries
Logs, whole project 24 hours ago to 6 hours ago 0 entries
Logs, Kyverno background controller oldest entry returned 16:21:58 UTC today; nothing before it in any 7-day sub-window
Traces, whole project 2 days ago to 1 day ago No data
Traces, whole project last hour 62 endpoints, normal
Metrics, count(up) 2 days ago to 1 day ago Present, 13 targets in every hour

The production Coroot instance keeps logs, traces and profiles for 7 days (logsTTL, tracesTTL and profilesTTL in the Hetzner HA patch). The ClickHouse shard and keeper Pods have run without restarts since 29 October and 3 November. The Coroot Pods restarted at 16:07 UTC today, after #6131 deployed the alert-retirement build. Coroot's own warning and error logs since then show no ClickHouse error.

So the metric history survived, but logs and traces from before about 16:21 UTC are not returned. That is consistent with the ClickHouse-backed history having been dropped, or with the new Coroot build reading logs and traces from a different place than the old one wrote them.

Problem

Seven days of logs and traces should be available for incident work, and are not. Any investigation that needs them, such as the Coroot window #5527 needs, currently sees only a few hours. If the data was dropped rather than hidden, the next Coroot image change may drop it again without anyone noticing, because nothing alerts on a shrinking log or trace history.

Hypothesis and success signal

The quickest discriminator is a read of the stored data itself: the oldest timestamp and the row count per day in the ClickHouse log and trace tables, through an authorized read-only path.

  • If old rows are there, Coroot is querying the wrong tables or database. Fix the configuration or the build so the history is visible again.
  • If they are gone, find what dropped them (a table migration in the new build, a TTL or partition change, a volume event) and stop it happening on the next image change.

Success: Coroot returns logs and traces across the configured 7-day window, and a check fails when the oldest returned entry is much younger than the configured retention.

Acceptance criteria

  • Establish from the stored data whether the history before 16:21 UTC on 12 November exists, and record which hypothesis held.
  • Restore visibility, or record the cause of the loss with a fix that prevents it on the next Coroot image or configuration change.
  • Add a read-only check, in the deploy receipt or an alert, that compares the oldest returned log and trace entry with the configured retention.
  • After the fix, a query for logs from 3 days ago returns entries again, read back through Coroot's API.

Related: #6131 (the build deployed just before the gap), #4869 (ClickHouse backups), #5527 (needs a Coroot window).

Activity

  1. added theissue type on Nov 12, 2026
  2. moved this to 📥 Backlog in 🌊 Project Boardon Nov 12, 2026
  3. devantler commented on Nov 12, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    One narrowing fact. #6131 changed only the Coroot image, from v1.28.1-checkpoints.2 to v1.28.1-checkpoints.3: the same upstream release with the repository's patch set, and no storage, TTL or ClickHouse setting touched. A table migration in that restart is therefore less likely than the first hypothesis suggests. The stored-data read should also cover both ClickHouse replicas, since the second replica's Pod started on 3 November and the first's on 29 October.

  4. devantler commented on Nov 12, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    Root cause found. The history was deleted, not hidden, and it happens every day, not just after #6131. Coroot's own disk cleanup deletes the current day's logs, traces and profiles once the ClickHouse disk passes 70%. That is an upstream bug. Behind it is a sizing problem: one day of data nearly fills each 15 Gi ClickHouse volume, so 7 days was never possible.

    What the evidence shows

    1. The disk was emptied, and it happens daily. ClickHouse data-volume usage from Coroot's metrics (container_resources_disk_used_bytes, /var/lib/clickhouse, shard replica 0; replica 1 moves the same way):

    Time (UTC) Before After What was dropped
    9 Nov ~21:54 11.2 GB 1.3 GB that day's data
    10 Nov ~18:54 11.6 GB 1.4 GB that day's data
    11 Nov ~11:54 11.5 GB 8.3 GB the previous day's remainder
    11 Nov ~16:54 11.0 GB 1.3 GB that day's data
    12 Nov ~09:35 11.5 GB 7.4 GB the previous day's remainder
    12 Nov 16:05–16:08 11.0 GB about 0.7 GB that day's data

    The volumes are 15 Gi each. Between drops, usage grows about 0.5 GB an hour, so one day of logs, traces and profiles is about 11–12 GB, roughly 73% of a volume. Coroot's last startup reported disk "default" usage: 4.57% and 0.56% for the two replicas at 16:08:42, less than a minute after the 16:07 restart. That supports this narrowing: #6131 only made the newest drop line up with the restart. The image change did not cause it.

    2. Why the current day goes too. Coroot's space manager runs every hour. Above 70% disk use it deletes the oldest daily partition of each telemetry table, but is meant to keep at least min_partitions (default 1). Its documentation says it "will never delete the last remaining partition". The code does not do that: getPartitionsFromDiskOnServer reads system.parts, which has one row per data part, and a daily partition normally holds several parts. So len(tablePartitionList) <= MinPartitions counts parts, not partitions, and once only today's partition remains the guard still lets it drop ALTER TABLE … DROP PARTITION ID '<today>'. The same code is unchanged on Coroot main and in v1.29.2.

    3. The 7-day setting has no effect. Coroot sets a table's TTL only when it creates the table (CREATE TABLE IF NOT EXISTS). Coroot's documentation says a later TTL change needs a manual ALTER TABLE … MODIFY TTL. So the logsTTL: 7d in the Hetzner HA patch has not reached tables created earlier. Either way, 7 days of data (about 80 GB) cannot fit in 15 Gi.

    So hypothesis 2 of this issue holds: the rows are gone. The cause is the space manager, not a table migration, a TTL change or a volume event. The next drop of the current day is due once usage passes 70% again, about 22 hours after the last one.

    Plan

    1. Fix the guard (this is mine, under the claim): our patched Coroot build counts distinct partitions, so the documented "never delete the last remaining partition" holds. Regression tests cover the multi-part case. As with fix(coroot): retire check alerts whose report no longer applies #6129 and fix(coroot): deploy the inapplicable-report alert retirement #6131, one PR publishes the build and a second pins it in production.
    2. What the fix alone gives. With one day at about 73% of a volume, the guard can only ever keep the current day. Retention becomes "since 00:00 UTC", with yesterday dropped shortly after midnight, instead of today's data disappearing every afternoon. Peak use stays near 80%, so the disk does not fill. It is a real improvement, but not the 7 days this issue asks for.
    3. Capacity for the retention we actually want is a maintainer decision. It is recurring capacity, which the temporary-cost grant excludes. As engineering sizing (no prices): keeping at least one full previous day needs about 35 Gi per ClickHouse replica; a real 7 days needs about 130 Gi per replica. The alternative is collecting less data. I am not changing volume sizes or what Coroot collects.
    4. Detection (acceptance item 3): a read-only check comparing the oldest returned log and trace entry with the configured retention, kept as a separate slice so it is not waiting on the build.

    Board: moved to In Progress as rung-0 live breakage. Incident work currently has at most a few hours of logs and traces. This is a recorded WIP exception, because platform's In Progress column is already over its ceiling. First started: 2026-11-12.

  5. self-assigned this
    on Nov 12, 2026
  6. moved this from 📥 Backlog to 🏃🏻‍♂️ In Progress in 🌊 Project Boardon Nov 12, 2026
  7. devantler commented on Nov 14, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    Post-merge check of #6189 (merged 13:21Z on 13 November), read 11:10Z on 14 November. I read the ClickHouse data-volume usage (container_resources_disk_used_bytes, /var/lib/clickhouse, both replicas) over the last 24 hours in 30-minute steps:

    Time (UTC) Replica 0 Replica 1 What happened
    13 Nov ~13:40 10.5 → 1.5 GB 9.8 → 0.8 GB the current day was dropped, about 20 minutes after the merge, before the rollout
    13 Nov 14:00 → 14 Nov 03:10 grew to 11.5 GB grew to 11.0 GB no afternoon drop: the current day survived past 70% for the first time
    14 Nov ~03:40 11.5 → 3.9 GB 11.0 → 3.3 GB the previous day's partition dropped after midnight, and today's partition was kept
    14 Nov 11:10 8.9 GB 8.3 GB growing again

    This matches plan item 2: retention is now "since 00:00 UTC", and the current day is no longer deleted in the afternoon. Acceptance item 2 holds for one full cycle.

    Still open on this issue:

    • Item 3, the read-only retention check.
    • Item 4, logs from 3 days ago. That cannot pass at 15 Gi per replica, because it needs the capacity decision in plan item 3 (about 35 Gi per replica for one full previous day, about 130 Gi for 7 days).

    The capacity decision is asked in devantler-tech/monorepo#5034.

    | asked digest devantler-tech/monorepo#5034 2026-11-14

  8. added 2 commits that reference this issue on Nov 15, 2026
  9. devantler commented on Nov 15, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    The maintainer selected option 2:A in the 15 November digest reply: one full previous day of history, using the proposed 35 Gi ClickHouse volume size per replica and two-day telemetry retention. This replaces the original seven-day/three-days-ago target on this issue.

    I am preparing the declarative capacity and retention change. Acceptance remains open until the existing volumes have expanded, effective table retention is verified, and Coroot returns yesterday’s logs and traces after a complete day boundary. Existing history already lost is not recoverable from this configuration change. The separate read-only retention check remains required. No live mutation is performed by this record.

  10. added 3 commits that reference this issue on Nov 15, 2026
  11. devantler commented on Nov 15, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    Post-merge readback of #6336 at evaluated head 56370a4ac6595904c21a69be99774a1eab8aece4, merged as cf0cf532bf9094ec9e7c9951392a071b49a00185 on 15 November at 14:16 UTC:

    • Both existing ClickHouse data claims now request 35 Gi and report 35 Gi in bound capacity. Neither reports an outstanding resize condition. The volume-expansion acceptance step is verified.
    • The merged source sets log, trace and profile retention to 2 days, matching the decision recorded above. Source configuration is verified; effective table TTLs are not. This lane cannot read the Coroot custom resource, and the current fixed runtime-health observer does not measure table retention or historical log/trace coverage.
    • Yesterday's complete history remains unverified. A larger volume cannot recover previously deleted data, and merge time does not establish a full retained day.

    Keep #6143 open. The remaining delivery work is a reviewed read-only retention check that observes effective table retention and oldest log/trace timestamps, followed by the complete-day-boundary readback through Coroot. No production mutation or new workflow dispatch was performed for this observation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Fields

Priority

None yet

Effort

None yet

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions