registry: bound zot storage growth and lift the 20Gi lab cap off the estate registry - #1091
registry: bound zot storage growth and lift the 20Gi lab cap off the estate registry#1091mdheller wants to merge 1 commit into
Conversation
…estate registry
Every image build that targets registry.socioprophet.ai has been failing on main —
socioprophet-web, workspace-caldav, workspace-smtp, workspace-mail — with:
failed to push registry.socioprophet.ai/<image>:latest:
unknown: blob upload unknown to registry
The images build fine; only the push breaks. The real error is server-side. zot's log
during those builds:
XMinioStorageFull: Storage backend has reached its minimum free drive threshold
(507) ... unexpected error, removing .uploads/ files
MinIO's volume is 100% full (20Gi, 193M free), so blob commits are refused, zot tears
down the in-flight upload session, and the next chunk from buildx lands on a session
that no longer exists — which the client reports as BLOB_UPLOAD_UNKNOWN.
Two things put it there, and both are fixed here.
1. Nothing ever expired. storage.gc was true, but GC only reclaims untagged/dangling
blobs, and with no retention policy no tag ever becomes untagged — so GC had nothing
to collect. CI pushes latest + sha-<commit> + a cosign .sig on every commit to main,
and dedupe is false (mandatory for the S3 driver), so nothing is shared between tags.
Four repos reached ~410 tags / ~3,250 blobs each and consumed 14.9G of the 20G.
config.json now carries a storage.retention policy: keep latest, keep anything pulled
within 90d (this is what protects the sha-* tags the cluster is running), keep the 100
most recently pushed tags per repo, delete untagged manifests and their referrers.
gcInterval: 6h so reclamation actually runs — it was unset, which leaves periodic GC
disabled entirely.
2. The p0-lab overlay pinned the shared MinIO PVC to 20Gi, overriding the base's 100Gi.
That cap dates from when this MinIO served only the workspace; zot now uses the same
MinIO as the estate registry's blob store. Dropped, so base's 100Gi applies.
The config is also moved from a hand-written ConfigMap to a configMapGenerator over a
real config.json. zot reads its config once at startup and does not hot-reload, so as a
plain ConfigMap an edit changed the mounted file but left the running process on its old
in-memory policy until someone remembered to restart it by hand — the retention policy
added here would have been declared and never read. The generator's name hash rolls the
pod whenever the config changes.
Both the source and the kustomize-rendered config were validated against the pinned
binary (zot v2.1.2 verify → "config file is valid", exit 0). Checked the negative too: a
typo'd retention key exits 1 with "Storage.Retention.Policies[0].KeepTags[2] has invalid
keys", so the block is genuinely parsed rather than accepted and ignored.
There was a problem hiding this comment.
Pull request overview
Note
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
This PR addresses failing pushes to registry.socioprophet.ai by preventing unbounded zot/MinIO storage growth and removing the lab overlay’s 20Gi PVC cap that became the estate registry’s effective ceiling.
Changes:
- Add zot retention + periodic GC configuration (
storage.retention,gcInterval) to bound registry growth. - Switch zot config delivery from a static ConfigMap manifest to a
configMapGeneratorso config updates trigger a rollout. - Remove the
p0-labpatch that shrankworkspace-minio-pvcto 20Gi; rely on the base 100Gi claim.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| infra/k8s/zot/base/kustomization.yaml | Removes hand-written ConfigMap resource and generates it from config.json to force rollouts on config change. |
| infra/k8s/zot/base/configmap.yaml | Deletes the previously embedded JSON ConfigMap manifest. |
| infra/k8s/zot/base/config.json | Introduces retention policies and scheduled GC to stop storage growth and reclaim space. |
| infra/k8s/zot/README.md | Documents why space filled, how retention/GC work, and how to validate config changes. |
| infra/k8s/workspace-minio/overlays/p0-lab/kustomization.yaml | Drops the overlay patch that capped MinIO PVC at 20Gi, restoring base sizing. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| "storageDriver": { | ||
| "name": "s3", | ||
| "rootdirectory": "/zot", | ||
| "region": "us-east-1", | ||
| "regionendpoint": "workspace-minio.socioprophet.svc.cluster.local:9000", | ||
| "bucket": "zot", | ||
| "secure": false, | ||
| "skipverify": true | ||
| } |
| # kubectl -n socioprophet create configmap zot-cfg-verify --from-file=config.json=config.json | ||
| # kubectl -n socioprophet run zot-verify --restart=Never --image=ghcr.io/project-zot/zot-linux-amd64:v2.1.2 \ | ||
| # --overrides='{"spec":{"containers":[{"name":"zot","image":"ghcr.io/project-zot/zot-linux-amd64:v2.1.2","args":["verify","/etc/zot/config.json"],"volumeMounts":[{"name":"c","mountPath":"/etc/zot"}]}],"volumes":[{"name":"c","configMap":{"name":"zot-cfg-verify"}}]}}' |
| - CI pushes `latest` + `sha-<commit>` + a cosign `.sig` on **every** commit to `main`. | ||
|
|
||
| `gc: true` alone does **not** bound this: GC only reclaims *untagged/dangling* blobs, and without a | ||
| retention policy no tag is ever untagged, so GC finds nothing to do. On 2026-07-30 that reached its |
|
Live-cluster state for whoever merges this — merging it resolves an active ArgoCD sync failure. The volume could not wait for CI, so It expanded online in ~25s, no pod restart: As predicted in the PR description, Git still declaring 20Gi leaves Harmless (a shrink is rejected by the API server, nothing is destroyed) but it will keep failing every sync until this lands, and a permanently-failing app masks real sync errors. No action needed on this PR — it is correct as written; it just needs to merge. Currently blocked on ~49 queued checks. Two things verified on the live cluster that bear on the config here:
Capacity side is handled separately in #1105 (bounded PVC autoscaling) — it does not touch |
What is red
Four image builds have been failing on
main—socioprophet-web,workspace-caldav,workspace-smtp,workspace-mail. Every one of them is a matrix leg that pushes toregistry.socioprophet.ai; every leg that pushes to GAR is green. They fail identically, atdocker/build-push-action@v6:The image builds fine every time — layers export, manifest exports, manifest list exports. Only the push breaks, which is why the Dockerfiles look innocent.
Root cause
Client-side
BLOB_UPLOAD_UNKNOWNis a symptom. zot's own log during those builds has the cause:workspace-minio-pvcis 20Gi, 100% used, 193M free. MinIO refuses blob commits below its minimum-free-drive threshold, zot tears down the in-flight.uploads/staging session, and buildx's next chunk arrives at a session that no longer exists.Two things filled it, and both are fixed here.
1. Nothing ever expired.
storage.gcwastrue, but GC only reclaims untagged/dangling blobs — and with noretentionpolicy, no tag ever becomes untagged, so GC had nothing to collect. Meanwhile CI pusheslatest+sha-<commit>+ a cosign.sigon every commit tomain, anddedupe: falseis mandatory for the S3 driver, so nothing is shared between tags. Measured on the live volume:workspace-smtpworkspace-mailworkspace-caldavsocioprophet-webollama,searxng,library,gitea)14.9G of first-party images, none of it ever expired, on a 20G volume.
gcIntervalwas also unset, which leaves periodic GC disabled outright.2. A lab cap became the estate ceiling.
infra/k8s/workspace-minio/overlays/p0-lab/kustomization.yamlpatched the PVC from the base's100Gidown to20Gi. That cap is from when this MinIO served only the workspace. zot now uses the same MinIO as the estate registry's blob store, so every image push in the estate lands on it.The fix
infra/k8s/zot/base/config.json— addsstorage.retention: keeplatest, keep anything pulled within 90d, keep the 100 most recently pushed tags per repo, delete untagged manifests and their referrers. PlusgcInterval: 6hso reclamation runs without a human. This is the load-bearing change; headroom alone would just postpone the same outage.infra/k8s/workspace-minio/overlays/p0-lab/kustomization.yaml— drops the20Gishrink patch so base's100Giapplies.infra/k8s/zot/base/kustomization.yaml— config moves from a hand-written ConfigMap to aconfigMapGeneratorover a realconfig.json.That last one matters more than it looks. zot reads its config once at startup and does not hot-reload. As a plain ConfigMap, editing the config changed the mounted file but left the running process on its old in-memory policy until someone remembered
kubectl rollout restart deploy/zot— this repo has been bitten by exactly that before. Without the generator, the retention policy added here would have been declared in Git and never actually read. The generator's name hash (zot-config-<hash>) changes the pod spec on every config edit, so Kubernetes rolls the pod and the config is genuinely loaded.Verification
Validated against the pinned binary, not a local approximation:
And the negative, so "valid" means something — a typo'd retention key:
The retention block is genuinely parsed, not accepted and dropped.
kubectl kustomizerenders clean for bothzot/overlays/p0-labandzot/overlays/p1-single-site, and the Deployment'sconfigMapreference is rewritten to the hashed name automatically.What merging does, and the one ordering note
Merge = deploy (ArgoCD auto-syncs). On sync:
workspace-minio-pvcexpands 20Gi -> 100Gi (online, non-destructive;standard-rwohasallowVolumeExpansion: true). This is what unblocks the red builds — re-runimagesafter the PVC shows 100Gi.zot-config-<hash>is created, the oldzot-configis pruned, and the Deployment rolls. zot is a single replica, so expect a few seconds where pulls through the registry fail. First GC pass then reclaims the expired tags.Reviewer's call, please: retention will expire tags the cluster is not pulling. The
pulledWithin: 2160h(90d) rule is there to protect thesha-*tags currently deployed, and zot is only ~14 days old so everything it has ever served is inside that window today. The residual risk is an image that runs >90d without a single pull and has fallen outside the 100 most recent pushes — that would be expired and wouldImagePullBackOffon a later reschedule. If that is too tight for anything you know of, widenpulledWithinor pin the image by digest.Not fixed here
main—memoryd,liberty-stack-readout,tritfabric-consumption-api— are a different failure:cosign signfails withfetching ambient OIDC credentials: invalid character 'u' looking for beginning of value. That'u'is GitHub'sUnicorn!5xx page where the Actions OIDC token endpoint's JSON should be. Transient GitHub, same signature as the 2026-07-16 incident. Retry, no code change.:latestis still pushed to zot on every commit, which is both the moving-tag trap this repo has flagged before and a direct contributor to the growth above. Out of scope here.