Skip to content

[Bug]: Plakar backup fails when cache drive is full #2254

Description

@toxn

Describe the bug

A backup task failing while running leaves the cache full
A backup task fails when the cache is full from the start
Nowhere, plakar seems to be able to clean its own cache, except at the end of a successful backup?!

In my case, I have 1 2T backup and one 1.2T backup plus various others.

Steps to reproduce

Run a task with massive cache usage, then run any following task

Expected behavior

Whenever a backup starts, or in the middle of a backup task, if the cache drive is full, plakar should initiate a cleanup of the cache, according to multiple running backup/restore tasks, maybe privileging quick-to-end tasks so that their cache cleaned leaving space for others.

Actual behavior

The current backup fails, leaving the cache full.
All the following backup tasks fail, also leaving cache full

Plakar version

$ plakar version                                        
plakar/v1.1.3

Plugin versions

$ plakar pkg show
mysql@v1.1.0-beta.4
postgresql@v1.1.0-beta.8
s3@v1.1.2
sftp@v1.1.3

OS, version and architecture

$ cat /etc/os-release 
PRETTY_NAME="Debian GNU/Linux 13 (trixie)"
NAME="Debian GNU/Linux"
VERSION_ID="13"
VERSION="13 (trixie)"
VERSION_CODENAME=trixie
DEBIAN_VERSION_FULL=13.5
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

Store location

$ df -h /home/save/.cache/plakar
Filesystem                     Size  Used Avail Use% Mounted on
/dev/mapper/vg01-plakar_cache   49G  488M   46G   2% /home/save/.cache/plakar

Backup configuration

# /etc/systemd/system/backup-[redacted]-db.service
[Unit]
Description=Sauvegarde quotidienne de la base de données [redacted] sur [redacted]
After=network.target
Before=plakar_prune-[redacted].service

[Service]
Type=oneshot
LoadCredentialEncrypted=save_id_ed25519
LoadCredentialEncrypted=[redacted].kloset.passphrase
User=save
Group=save
ExecStart=/usr/bin/flock -w 7200 /run/lock/plakar-[redacted] bash -c 'ln -sf %d/save_id_ed25519 /home/save/.ssh/id_ed25519.[redacted] && ssh -n [redacted] sudo mysqldump --single-transaction --routines --triggers --events [redacted] | /usr/bin/plakar -keyfile %d/[redacted].kloset.passphrase at @[redacted] backup stdin:backup.sql'
StandardOutput=journal
StandardError=journal

Backup destination

through sftp

Debug output

Jun 30 00:22:38 [redacted] systemd[1]: Starting backup-[redacted]-db.service - Sauvegarde quotidienne de la base de données [redacted] sur [redacted]...
Jun 30 00:22:43 [redacted] plakar[1995380]: stdoutinfo: cached rebuild request for 27588ef5-546f-468b-b8d5-66099225cc67
Jun 30 00:22:44 [redacted] flock[1995218]: error: error opening database at "/home/save/.cache/plakar/2.0.0/scan/775541f65eee216b683bf6df03bcb3fa06aea308fc61b4741e3a5315ac28f570": mkdir /home/save/.cache/plakar/2.0.0/scan/775541f65eee216b683bf6df03bcb3fa06aea308fc61b4741e3a5315ac28f570: no space left on device
Jun 30 00:22:44 [redacted] flock[1995218]: /usr/bin/plakar: error opening database at "/home/save/.cache/plakar/2.0.0/scan/775541f65eee216b683bf6df03bcb3fa06aea308fc61b4741e3a5315ac28f570": mkdir /home/save/.cache/plakar/2.0.0/scan/775541f65eee216b683bf6df03bcb3fa06aea308fc61b4741e3a5315ac28f570: no space left on device
Jun 30 00:22:44 [redacted] systemd[1]: backup-[redacted]-db.service: Main process exited, code=exited, status=1/FAILURE
Jun 30 00:22:44 [redacted] plakar[1995380]: stderr/usr/bin/plakar: received interrupt signal, stopping gracefully...
Jun 30 00:22:44 [redacted] plakar[1995380]: stdout/usr/bin/plakar: interrupted
Jun 30 00:22:44 [redacted] systemd[1]: backup-[redacted]-db.service: Failed with result 'exit-code'.
Jun 30 00:22:44 [redacted] systemd[1]: Failed to start backup-[redacted]-db.service - Sauvegarde quotidienne de la base de données [redacted] sur [redacted].
Jun 30 00:22:44 [redacted] systemd[1]: backup-[redacted]-db.service: Consumed 979ms CPU time, 359.3M memory peak.
Jul 01 00:35:18 [redacted] systemd[1]: Starting backup-[redacted]-db.service - Sauvegarde quotidienne de la base de données [redacted] sur [redacted]...
Jul 01 00:35:21 [redacted] plakar[247105]: stdoutinfo: cached rebuild request for 27588ef5-546f-468b-b8d5-66099225cc67
Jul 01 00:35:21 [redacted] flock[247065]: error: error opening database at "/home/save/.cache/plakar/2.0.0/scan/4890724bfb995eea7866b935433d47045ddd9497450f91ce3494878ba8d93ad5": mkdir /home/save/.cache/plakar/2.0.0/scan/4890724bfb995eea7866b935433d47045ddd9497450f91ce3494878ba8d93ad5: no space left on device
Jul 01 00:35:21 [redacted] flock[247065]: /usr/bin/plakar: error opening database at "/home/save/.cache/plakar/2.0.0/scan/4890724bfb995eea7866b935433d47045ddd9497450f91ce3494878ba8d93ad5": mkdir /home/save/.cache/plakar/2.0.0/scan/4890724bfb995eea7866b935433d47045ddd9497450f91ce3494878ba8d93ad5: no space left on device
Jul 01 00:35:21 [redacted] systemd[1]: backup-[redacted]-db.service: Main process exited, code=exited, status=1/FAILURE
Jul 01 00:35:21 [redacted] plakar[247105]: stderr/usr/bin/plakar: received interrupt signal, stopping gracefully...
Jul 01 00:35:21 [redacted] plakar[247105]: stdout/usr/bin/plakar: interrupted
Jul 01 00:35:21 [redacted] systemd[1]: backup-[redacted]-db.service: Failed with result 'exit-code'.
Jul 01 00:35:21 [redacted] systemd[1]: Failed to start backup-[redacted]-db.service - Sauvegarde quotidienne de la base de données [redacted] sur [redacted].
Jul 01 00:35:21 [redacted] systemd[1]: backup-[redacted]-db.service: Consumed 917ms CPU time, 304.5M memory peak.
Jul 01 14:28:31 [redacted] systemd[1]: Starting backup-[redacted]-db.service - Sauvegarde quotidienne de la base de données [redacted] sur [redacted]...

Additional context

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions