spored: short task can signal completion before on-complete tag is visible → no prompt self-terminate (TTL backstop)

[scttfrdmn]

Summary

A very short-lived task can write the completion file (/tmp/SPAWN_COMPLETE) before the spawn:on-complete / spawn:completion-file EC2 tags become readable, so spored never starts watching for completion → the instance doesn't self-terminate and falls back to the TTL backstop. Surfaced while validating miniwdl-spawn (#395), and applies equally to nf-spawn and any short workflow-engine task.

Mechanism (code refs)

• spored loads lifecycle config from spawn:* EC2 tags. Tags set at launch are subject to EC2 tag eventual-consistency and may not be readable in the seconds after boot; spored picks them up on its RefreshConfig cadence — tick 1 (~60s after boot), then every 5 ticks (~5 min) (pkg/agent/agent.go ~361-373; the code comments already acknowledge this).
• The monitor loop interval is 60s (pkg/agent/agent.go:297); checkCompletion os.Stats the file each tick (~1299).
• If spawn:on-complete/spawn:completion-file aren't in the live config yet when the task finishes, spored has nothing to act on. A task that completes in < ~60s (before the first successful config refresh) can miss the window.

Normal (non-bug) latency, for the record

Even when everything is visible, self-terminate lags the completion file by ~90s–3min: up to 60s to the next monitor tick + 30s completion-delay + 5s warn + up to 60s region-vacated settle when it's the only instance. Observed live: a task returning at T had its instance self-terminate ~60s later — normal. (An earlier observation of "still running after a few minutes" was a too-early check, not a defect.)

Why it matters

Workflow-engine backends (miniwdl-spawn, nf-spawn) launch one ephemeral instance per task; many tasks are short. A missed on-complete means the instance lives until its TTL (billable) instead of terminating promptly. TTL prevents a true leak but wastes cost.

Fix directions (pick during triage)

1. spored: resolve on-complete eagerly. Until on-complete is populated, refresh tags every tick (not every 5th), and/or read the completion-relevant tags with a short retry loop at boot, so a short task's signal is honored.
2. Bootstrap: pass on-complete via user-data, not only tags, so it's known at boot without a tag round-trip (avoids the eventual-consistency race entirely).
3. Backend-side mitigation (miniwdl-spawn/nf-spawn): the backend already knows when the task is done (it polls .exitcode in S3) — it could spawn terminate <name> itself after pulling results, rather than relying solely on spored's on-complete. (miniwdl-spawn already builds this teardown argv for the terminating() path.)

Evidence

• miniwdl-spawn Phase-4 runs (feat(array): spawn array command group (#389) #395): first run's instance hadn't self-terminated when checked ~minutes in (too-early check); confirming run self-terminated ~60s post-completion. Both tasks ran ~75s, i.e. right at the tag-visibility boundary.

Refs #395.