LitestreamRestore: no way to restore when target DB file already exists on PVC (missing spec.force)

[jlaska]

Summary

litestream restore refuses to overwrite an existing output file (cannot restore, output path already exists and is not empty: ... Use -force to overwrite). The LitestreamRestore CR has no spec.force field to pass -force to the underlying litestream restore command, and the operator performs no pre-flight deletion of an existing file before running the restore Job. When the target file exists, the Job fails immediately and the only recovery is manually exec-ing into the PVC to delete the file.

Additionally, when the Job fails due to this error, status.message only surfaces the Kubernetes Job backoff reason ("Job has reached the specified backoff limit"), not the underlying litestream error — making it hard to diagnose.

Environment

• litestream-operator: v0.4.2
• litestream: v0.5.14
• Kubernetes: v1.36.2+k3s1

Most Common Trigger

This is most easily hit in the following sequence (which is the natural recovery workflow):

1. Scale Deployment to 0, delete DB from PVC (to prepare for restore).
2. Scale Deployment back to 1 to verify/observe the archive-check behavior.
3. The application initializes a fresh DB on the PVC during pod startup.
4. Apply LitestreamRestore CR to trigger operator-managed restore.
5. The operator scales the Deployment to 0 and runs the restore Job.
6. The restore Job immediately fails because step 3 left a DB file on the PVC.

There is no way to make the operator clear the existing file — the user must manually intervene.

Observed Behavior

status:
  phase: Failed
  originalReplicas: 1
  message: "Job has reached the specified backoff limit"
  jobName: myapp-restore-restore

The actual error (cannot restore, output path already exists) is only visible in the Job pod logs, which are often garbage-collected before the user can inspect them.

Expected Behavior

Either:

Option A: A spec.force: true field on LitestreamRestoreSpec that:

• Adds a pre-restore init container to delete targetPath (and targetPath-wal, targetPath-shm) before the restore container runs.
• Does NOT pass -force to litestream restore directly (litestream v0.5.x may not support it).

Option B: The operator surfaces the underlying litestream error in status.message so users can self-diagnose without inspecting Job pod logs.

Minimal Reproducer

Direct litestream (no k8s needed)

# Create an existing file at the target path
touch /tmp/app.db

# Attempt restore — fails immediately
docker run --rm \
  -v /tmp:/data \
  -v ./litestream.yml:/etc/litestream/litestream.yml \
  litestream/litestream:0.5.14 \
  restore -o /data/app.db /data/app.db
# Error: cannot restore, output path already exists and is not empty: /data/app.db.
#        Use -force to overwrite

litestream-operator scenario

# 1. Scale to 0, clear PVC
kubectl scale deployment myapp -n myapp --replicas=0
kubectl run cleanup --image=cgr.dev/chainguard/wolfi-base -n myapp --restart=Never \
  --overrides='{"spec":{"containers":[{"name":"c","image":"cgr.dev/chainguard/wolfi-base",
    "command":["rm","-f","/data/app.db"],"volumeMounts":[{"name":"d","mountPath":"/data"}]}],
    "volumes":[{"name":"d","persistentVolumeClaim":{"claimName":"pvc-myapp-data"}}]}}'
kubectl delete pod cleanup -n myapp

# 2. Scale back to 1 — app creates fresh app.db on PVC
kubectl scale deployment myapp -n myapp --replicas=1
kubectl rollout status deployment/myapp -n myapp

# 3. Apply LitestreamRestore CR — fails because fresh DB now exists
kubectl apply -f - <<EOF
apiVersion: litestream.io/v1
kind: LitestreamRestore
metadata:
  name: myapp-restore
  namespace: myapp
spec:
  sourceRef: myapp-db
  targetPVC: pvc-myapp-data
  targetPath: /data/app.db
EOF

# 4. Observe: Job fails, status shows only "backoff limit exceeded"
kubectl get litestreamrestore myapp-restore -n myapp -o jsonpath='{.status}'
# {"phase":"Failed","message":"Job has reached the specified backoff limit","originalReplicas":1}
# No mention of "output path already exists"

Source Location

• api/v1/litestreamrestore_types.go — LitestreamRestoreSpec: no Force field.
• internal/controller/litestreamrestore_controller.go — buildRestoreJob() (lines 512–611): no pre-flight file deletion, no -force flag.
• internal/controller/litestreamrestore_controller.go — error surfacing: status.message is set from the Job condition message, not from the Job pod logs.

Suggested Fix

1. Add spec.force to LitestreamRestoreSpec

// Force, if true, deletes the target database file before running the restore Job.
// Use when a DB file already exists on the PVC from a previous (failed) restore
// or from a fresh application initialization.
// +optional
Force bool `json:"force,omitempty"`

2. When force=true, add a pre-restore init container

// In buildRestoreJob, when spec.Force is true:
initContainers := []corev1.Container{{
    Name:  "remove-existing-db",
    Image: "cgr.dev/chainguard/wolfi-base",
    Command: []string{"sh", "-c",
        fmt.Sprintf("rm -f %s %s-wal %s-shm && echo 'pre-restore cleanup done'",
            spec.TargetPath, spec.TargetPath, spec.TargetPath)},
    VolumeMounts: []corev1.VolumeMount{{Name: "data", MountPath: filepath.Dir(spec.TargetPath)}},
}}

3. Surface the underlying litestream error in status.message

Read the failed Job pod's logs during reconciliation and include the litestream error text in status.message so users can diagnose without manual log inspection.