Repository navigation
Description
🤖 Generated by the Agentic Engineer
Evidence
Read-only checks through Coroot's API on 12 November at about 19:40 UTC:
| Signal | Window | Result |
|---|---|---|
| Logs, whole project | last 3 hours | Present, newest-first as expected |
| Logs, whole project | 30 days ago to 1 day ago | 0 entries |
| Logs, whole project | 24 hours ago to 6 hours ago | 0 entries |
| Logs, Kyverno background controller | oldest entry returned | 16:21:58 UTC today; nothing before it in any 7-day sub-window |
| Traces, whole project | 2 days ago to 1 day ago | No data |
| Traces, whole project | last hour | 62 endpoints, normal |
Metrics, count(up) |
2 days ago to 1 day ago | Present, 13 targets in every hour |
The production Coroot instance keeps logs, traces and profiles for 7 days (logsTTL, tracesTTL and profilesTTL in the Hetzner HA patch). The ClickHouse shard and keeper Pods have run without restarts since 29 October and 3 November. The Coroot Pods restarted at 16:07 UTC today, after #6131 deployed the alert-retirement build. Coroot's own warning and error logs since then show no ClickHouse error.
So the metric history survived, but logs and traces from before about 16:21 UTC are not returned. That is consistent with the ClickHouse-backed history having been dropped, or with the new Coroot build reading logs and traces from a different place than the old one wrote them.
Problem
Seven days of logs and traces should be available for incident work, and are not. Any investigation that needs them, such as the Coroot window #5527 needs, currently sees only a few hours. If the data was dropped rather than hidden, the next Coroot image change may drop it again without anyone noticing, because nothing alerts on a shrinking log or trace history.
Hypothesis and success signal
The quickest discriminator is a read of the stored data itself: the oldest timestamp and the row count per day in the ClickHouse log and trace tables, through an authorized read-only path.
- If old rows are there, Coroot is querying the wrong tables or database. Fix the configuration or the build so the history is visible again.
- If they are gone, find what dropped them (a table migration in the new build, a TTL or partition change, a volume event) and stop it happening on the next image change.
Success: Coroot returns logs and traces across the configured 7-day window, and a check fails when the oldest returned entry is much younger than the configured retention.
Acceptance criteria
- Establish from the stored data whether the history before 16:21 UTC on 12 November exists, and record which hypothesis held.
- Restore visibility, or record the cause of the loss with a fix that prevents it on the next Coroot image or configuration change.
- Add a read-only check, in the deploy receipt or an alert, that compares the oldest returned log and trace entry with the configured retention.
- After the fix, a query for logs from 3 days ago returns entries again, read back through Coroot's API.
Related: #6131 (the build deployed just before the gap), #4869 (ClickHouse backups), #5527 (needs a Coroot window).
Metadata
Metadata
Assignees
Labels
Type
Fields
Priority
Effort
Projects
- StatusShow more project fields🏃🏻♂️ In Progress
Activity
devantler commentedon Nov 12, 2026
One narrowing fact. #6131 changed only the Coroot image, from
v1.28.1-checkpoints.2tov1.28.1-checkpoints.3: the same upstream release with the repository's patch set, and no storage, TTL or ClickHouse setting touched. A table migration in that restart is therefore less likely than the first hypothesis suggests. The stored-data read should also cover both ClickHouse replicas, since the second replica's Pod started on 3 November and the first's on 29 October.devantler commentedon Nov 12, 2026
Root cause found. The history was deleted, not hidden, and it happens every day, not just after #6131. Coroot's own disk cleanup deletes the current day's logs, traces and profiles once the ClickHouse disk passes 70%. That is an upstream bug. Behind it is a sizing problem: one day of data nearly fills each 15 Gi ClickHouse volume, so 7 days was never possible.
What the evidence shows
1. The disk was emptied, and it happens daily. ClickHouse data-volume usage from Coroot's metrics (
container_resources_disk_used_bytes,/var/lib/clickhouse, shard replica 0; replica 1 moves the same way):The volumes are 15 Gi each. Between drops, usage grows about 0.5 GB an hour, so one day of logs, traces and profiles is about 11–12 GB, roughly 73% of a volume. Coroot's last startup reported
disk "default" usage: 4.57%and0.56%for the two replicas at 16:08:42, less than a minute after the 16:07 restart. That supports this narrowing: #6131 only made the newest drop line up with the restart. The image change did not cause it.2. Why the current day goes too. Coroot's space manager runs every hour. Above 70% disk use it deletes the oldest daily partition of each telemetry table, but is meant to keep at least
min_partitions(default 1). Its documentation says it "will never delete the last remaining partition". The code does not do that:getPartitionsFromDiskOnServerreadssystem.parts, which has one row per data part, and a daily partition normally holds several parts. Solen(tablePartitionList) <= MinPartitionscounts parts, not partitions, and once only today's partition remains the guard still lets it dropALTER TABLE … DROP PARTITION ID '<today>'. The same code is unchanged on Corootmainand in v1.29.2.3. The 7-day setting has no effect. Coroot sets a table's TTL only when it creates the table (
CREATE TABLE IF NOT EXISTS). Coroot's documentation says a later TTL change needs a manualALTER TABLE … MODIFY TTL. So thelogsTTL: 7din the Hetzner HA patch has not reached tables created earlier. Either way, 7 days of data (about 80 GB) cannot fit in 15 Gi.So hypothesis 2 of this issue holds: the rows are gone. The cause is the space manager, not a table migration, a TTL change or a volume event. The next drop of the current day is due once usage passes 70% again, about 22 hours after the last one.
Plan
Board: moved to In Progress as rung-0 live breakage. Incident work currently has at most a few hours of logs and traces. This is a recorded WIP exception, because platform's In Progress column is already over its ceiling. First started: 2026-11-12.
chore: agent-claim #6143 nonce=7c0fb43e47457d4a0668cafd40bcdeed
fix(coroot): free the last partition only when the disk is nearly full
fix(coroot): carry each disk's usage into its cleanup decision
test(coroot): run the space manager against a real ClickHouse before …
test(coroot): fail the integration job unless its test actually ran
chore: agent-claim #6143 nonce=e3c963cc9e0f6209f06e3342c024add4
devantler commentedon Nov 14, 2026
Post-merge check of #6189 (merged 13:21Z on 13 November), read 11:10Z on 14 November. I read the ClickHouse data-volume usage (
container_resources_disk_used_bytes,/var/lib/clickhouse, both replicas) over the last 24 hours in 30-minute steps:This matches plan item 2: retention is now "since 00:00 UTC", and the current day is no longer deleted in the afternoon. Acceptance item 2 holds for one full cycle.
Still open on this issue:
The capacity decision is asked in devantler-tech/monorepo#5034.
| asked digest devantler-tech/monorepo#5034 2026-11-14
chore: agent-claim #6143 nonce=793d40cf8fe717e60c6812d133a2b075
chore: agent-claim #6143 nonce=5bb7c0a8bdcff2397db0ed7ef7e19388
devantler commentedon Nov 15, 2026
The maintainer selected option 2:A in the 15 November digest reply: one full previous day of history, using the proposed 35 Gi ClickHouse volume size per replica and two-day telemetry retention. This replaces the original seven-day/three-days-ago target on this issue.
I am preparing the declarative capacity and retention change. Acceptance remains open until the existing volumes have expanded, effective table retention is verified, and Coroot returns yesterday’s logs and traces after a complete day boundary. Existing history already lost is not recoverable from this configuration change. The separate read-only retention check remains required. No live mutation is performed by this record.
chore: agent-claim #6143 nonce=75a77c6a2aa41863a554fac8742bf120
chore: agent-claim #6143 nonce=df0c99296920edaf51337a665cd9af5f
chore: agent-claim #6143 nonce=ba16f1c7af19c3f85f38230a56c48c51
devantler commentedon Nov 15, 2026
Post-merge readback of #6336 at evaluated head
56370a4ac6595904c21a69be99774a1eab8aece4, merged ascf0cf532bf9094ec9e7c9951392a071b49a00185on 15 November at 14:16 UTC:Keep #6143 open. The remaining delivery work is a reviewed read-only retention check that observes effective table retention and oldest log/trace timestamps, followed by the complete-day-boundary readback through Coroot. No production mutation or new workflow dispatch was performed for this observation.