ETCD: Detect and handle DB size quota exhaustion

[yulianedyalkova]

Why is this important?

When etcd's database reaches its quota limit, the cluster hard stops accepting writes. No new resources can be created in Kubernetes, existing workloads are unaffected but the control plane is effectively frozen.

Outcome

We can see DB size growth trending toward quota before it becomes a problem, have a runbook that covers the response options, and have validated both through a real exercise.

Acceptance criteria:

Monitoring

• There is sufficient monitoring in place. Some metrics to consider:
  • etcd_mvcc_db_total_size_in_use_in_bytes
  • etcd_server_quota_backend_bytes
  • node_filesystem_size_bytes{mountpoint="/var/lib/etcd"}

Runbook

• Document how to confirm DB size is the issue (vs. other write failures)
• Document the response options and when to use each:
  • Compact revision history
  • Defrag the database
  • Increase quota / expand disk
  • Remove/replace member
• Document how to validate writes are accepted again post-intervention
• Runbook is linked from alert annotations

Chaos exercise

• Simulate disk exhaustion on a cluster node:
  • fallocate -l 10G /var/lib/etcd/space_eater
• Confirm alerting fires on the correct signals
• Confirm Kubernetes API write failures are observable
• Follow the runbook to resolve
• Document findings and any gaps as follow-up issues