Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

### Added

- `ox_prune --queue` restricts pruning to one queue's task rows, matching
`ox_health --queue`, so queues with different retention needs can each
be pruned with their own `--older-than`. Old schedule ticks are still
pruned for every schedule.

### Changed

- `ox_health --max-age` and `--worker-timeout` accept the duration forms
Expand Down
4 changes: 3 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -201,6 +201,7 @@ python manage.py ox_prune --older-than 7d

| Flag | Default | Meaning |
| --- | --- | --- |
| `--queue` | all queues | Delete only this queue's task rows, so queues with different retention needs can be pruned separately. |
| `--older-than` | `7d` | Minimum time since the task finished. Accepts `7d`, `24h`, `90m`, `45s`, or a plain number of seconds. |
| `--include-failed` | off | Also delete FAILED and LOST rows. By default they are kept: they hold the per-attempt tracebacks and can be retried. |
| `--batch-size` | `1000` | Rows per DELETE statement, so pruning a large table never takes a long lock or builds a giant IN clause. |
Expand All @@ -209,7 +210,8 @@ python manage.py ox_prune --older-than 7d
Only SUCCESSFUL and DISCARDED rows (and, with `--include-failed`, FAILED and
LOST rows) past the cutoff are deleted. READY and RUNNING rows are never touched, whatever their
age. Old rows from the recurring-schedule tick log are cleared with the same
cutoff, always keeping each schedule's most recent tick.
cutoff, always keeping each schedule's most recent tick, and `--queue` does
not narrow that.

## Health and monitoring

Expand Down
6 changes: 6 additions & 0 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -161,6 +161,7 @@ python manage.py ox_prune --older-than 7d

| Flag | Default | Meaning |
| --- | --- | --- |
| `--queue` | all queues | Delete only this queue's task rows, so queues with different retention needs can be pruned separately. Same name and meaning as `ox_health --queue`. Old schedule ticks are still pruned for every schedule. |
| `--older-than` | `7d` | Minimum time since the task finished. Accepts `7d`, `24h`, `90m`, `45s`, or a plain number of seconds. |
| `--include-failed` | off | Also delete FAILED and LOST rows. By default they are kept, because they hold the per-attempt tracebacks and can be retried. |
| `--batch-size` | `1000` | Rows per DELETE statement, so pruning a large table never takes a long lock or builds a giant IN clause. Must be at least 1. |
Expand All @@ -176,6 +177,11 @@ been removed from settings is kept by the same rule; such rows are
harmless and can be deleted by hand if unwanted. See
[Recurring tasks](recurring-tasks.md#missed-ticks).

`--queue` narrows the task rows, not the tick log. A run for one queue
prunes every schedule's old ticks at its own cutoff, so when queues are
pruned separately, the shortest `--older-than` decides how much tick
history stays.

## ox_health

A health check for cron alerting and container probes: exits 0 when
Expand Down
13 changes: 13 additions & 0 deletions docs/llms-full.txt
Original file line number Diff line number Diff line change
Expand Up @@ -370,6 +370,7 @@ python manage.py ox_prune --older-than 7d

| Flag | Default | Meaning |
| --- | --- | --- |
| `--queue` | all queues | Delete only this queue's task rows, so queues with different retention needs can be pruned separately. Same name and meaning as `ox_health --queue`. Old schedule ticks are still pruned for every schedule. |
| `--older-than` | `7d` | Minimum time since the task finished. Accepts `7d`, `24h`, `90m`, `45s`, or a plain number of seconds. |
| `--include-failed` | off | Also delete FAILED and LOST rows. By default they are kept, because they hold the per-attempt tracebacks and can be retried. |
| `--batch-size` | `1000` | Rows per DELETE statement, so pruning a large table never takes a long lock or builds a giant IN clause. Must be at least 1. |
Expand All @@ -385,6 +386,11 @@ been removed from settings is kept by the same rule; such rows are
harmless and can be deleted by hand if unwanted. See
[Recurring tasks](recurring-tasks.md#missed-ticks).

`--queue` narrows the task rows, not the tick log. A run for one queue
prunes every schedule's old ticks at its own cutoff, so when queues are
pruned separately, the shortest `--older-than` decides how much tick
history stays.

## ox_health

A health check for cron alerting and container probes: exits 0 when
Expand Down Expand Up @@ -3528,6 +3534,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

### Added

- `ox_prune --queue` restricts pruning to one queue's task rows, matching
`ox_health --queue`, so queues with different retention needs can each
be pruned with their own `--older-than`. Old schedule ticks are still
pruned for every schedule.

### Changed

- `ox_health --max-age` and `--worker-timeout` accept the duration forms
Expand Down
13 changes: 13 additions & 0 deletions src/django_ox/management/commands/ox_prune.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,11 @@ class Command(BaseCommand):
help = "Delete finished task rows older than a cutoff."

def add_arguments(self, parser: CommandParser) -> None:
parser.add_argument(
"--queue",
default=None,
help="Restrict pruning to one queue's task rows (default: all queues).",
)
parser.add_argument(
"--older-than",
default="7d",
Expand Down Expand Up @@ -59,6 +64,14 @@ def handle(self, *args: Any, **options: Any) -> None:
statuses += [OxTask.Status.FAILED, OxTask.Status.LOST]
prunable = OxTask.objects.filter(status__in=statuses, finished_at__lt=cutoff)
label = "/".join(statuses)
queue: str | None = options["queue"]
if queue is not None:
# Task rows only. The tick log below is pruned for every schedule
# whatever --queue names: deleting a task clears its ticks' link
# to it, and an anchor never had one, so a tick's queue cannot be
# read reliably.
prunable = prunable.filter(queue_name=queue)
label = f"{label} (queue {queue})"

# Old schedule ticks are dispatch-log bookkeeping, but each
# schedule's latest tick is the anchor the dispatcher measures
Expand Down
61 changes: 60 additions & 1 deletion tests/test_prune.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,11 +10,12 @@
from django_ox.models import OxScheduleTick, OxTask


def make_task(status, *, finished_days_ago=None):
def make_task(status, *, finished_days_ago=None, queue_name="default"):
now = timezone.now()
return OxTask.objects.create(
task_path="tests.tasks.add",
backend_name="default",
queue_name=queue_name,
enqueued_at=now - timedelta(days=400),
status=status,
finished_at=(
Expand Down Expand Up @@ -73,6 +74,64 @@ def test_prunes_old_successful_only_by_default(self):
assert OxTask.objects.filter(pk=young_ok.pk).exists()
assert "Deleted 1 SUCCESSFUL/DISCARDED task row(s)" in out

def test_queue_restricts_pruning_to_that_queue(self):
emails = make_task(
OxTask.Status.SUCCESSFUL, finished_days_ago=8, queue_name="emails"
)
reports = make_task(
OxTask.Status.SUCCESSFUL, finished_days_ago=8, queue_name="reports"
)

out = prune("--queue", "emails")

assert not OxTask.objects.filter(pk=emails.pk).exists()
assert OxTask.objects.filter(pk=reports.pk).exists()
assert "Deleted 1 SUCCESSFUL/DISCARDED (queue emails) task row(s)" in out

def test_queue_dry_run_counts_only_that_queue(self):
make_task(OxTask.Status.SUCCESSFUL, finished_days_ago=8, queue_name="emails")
make_task(OxTask.Status.SUCCESSFUL, finished_days_ago=8, queue_name="reports")
make_task(OxTask.Status.SUCCESSFUL, finished_days_ago=8, queue_name="reports")

out = prune("--queue", "reports", "--dry-run")

assert "Would delete 2 SUCCESSFUL/DISCARDED (queue reports) task row(s)" in out
assert OxTask.objects.count() == 3

def test_queue_keeps_batching(self, django_assert_num_queries):
for _ in range(5):
make_task(
OxTask.Status.SUCCESSFUL, finished_days_ago=8, queue_name="emails"
)
kept = make_task(
OxTask.Status.SUCCESSFUL, finished_days_ago=8, queue_name="reports"
)

# The same fifteen queries as test_deletes_in_batches: the queue is
# one more condition on each statement, not one more statement.
with django_assert_num_queries(15):
out = prune("--queue", "emails", "--batch-size", "2")

assert "Deleted 5 SUCCESSFUL/DISCARDED (queue emails) task row(s)" in out
assert list(OxTask.objects.values_list("pk", flat=True)) == [kept.pk]

def test_queue_still_prunes_every_schedules_old_ticks(self):
make_tick("a", scheduled_days_ago=30)
latest_a = make_tick("a", scheduled_days_ago=10)
make_tick("b", scheduled_days_ago=20)
latest_b = make_tick("b", scheduled_days_ago=9)

# No task row is in this queue. The tick log is pruned all the same:
# a tick's queue cannot be read reliably, so --queue does not narrow it.
out = prune("--queue", "emails")

assert set(OxScheduleTick.objects.values_list("pk", flat=True)) == {
latest_a.pk,
latest_b.pk,
}
assert "Deleted 0 SUCCESSFUL/DISCARDED (queue emails) task row(s)" in out
assert "Deleted 2 schedule tick row(s)" in out

def test_include_failed_prunes_failed_and_lost_too(self):
make_task(OxTask.Status.SUCCESSFUL, finished_days_ago=8)
make_task(OxTask.Status.FAILED, finished_days_ago=8)
Expand Down