Start monitoring prometheus WAL and give alerts if corrupted

[gabrielcocenza]

Bug Description

We have a cloud that for some reason one of the wal files got corrupted. With this, the number and size of segments skyrocket and now it's taking more than one hour to load.

du -sh /var/lib/prometheus/wal
101G	/var/lib/prometheus/wal

 find /var/lib/prometheus/wal -type f | wc -l
1706

For reference, in a health prometheus unit in another cloud there are just 27 files and 3GB.

The consequence of that is that the dead man snitch triggers every time when the pod for some reason restart complaining that AlertManager is down.

Would be good if the charm ships alert rules or loki rules to check if the wal is corrupted.

To Reproduce

I think you can easily corrupt a file using dd

Environment

cos-lite

Relevant log output

2026-08-21T17:04:06.942Z [prometheus] ts=2026-08-21T17:04:06.937Z caller=db.go:1014 level=error component=tsdb msg="compaction failed" err="WAL truncation in Compact: create checkpoint: read segments: corruption in segment /var/lib/prometheus/wal/00007095 at 4497: unexpected full record"

2026-08-21T17:03:58.509Z [prometheus] ts=2026-08-21T17:03:58.508Z caller=head.go:820 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=380.960908ms wal_replay_duration=1h9m58.548110792s wbl_replay_duration=171ns chunk_snapshot_load_duration=0s mmap_chunk_replay_duration=236.544485ms total_replay_duration=1h9m59.166367009s

Additional context

For future travellers:

I've tried to remove the corrupted segment as this article says, but prometheus was failing on start saying that segments need to be sequential.

To have the service back running again I had to do this, which means I started from scratch:

root@prometheus-0:/# pebble stop prometheus
root@prometheus-0:/# mv /var/lib/prometheus/wal /var/lib/prometheus/wal.bak
root@prometheus-0:/# mkdir /var/lib/prometheus/wal
root@prometheus-0:/# pebble start prometheus