Heartbeat from a child process, and commit only a key still held (0.6.8)

[sotashimozono]

Draft until DataVault 0.8.9 is registered (QAtlasHub/DataVault.jl#75). This PR raises the compat floor to 0.8.9, so CI cannot resolve its dependencies before that. Supersedes #60.

What was wrong (measured)

The key heartbeat was a Threads.@spawn task inside the master. While work_fn computed without yielding, it never beat, with -t 1, -t 2 and -t 2,1 alike. A control run whose work_fn yielded (sleep) beat normally.

A sibling therefore took the live master's key after stale_after. The master's lost flag could only be raised by that starved task, so the master then saved and committed the key as well.

Change

• The heartbeat is DataVault.start_heartbeat, a child sh process. It stops when the master dies, or when the lock's name no longer points at the file it opened (the key was released or taken).
• Ownership is checked at commit time:
  • running_owner == tok before save!
  • the commit uses mark_done!(vault, key, tok), which refuses a key a sibling took in the meantime
• The release (clear_running!(vault, key, tok)) is owner-checked, so it now runs unconditionally.

Tests

• new test_run_busy_holder.jl
  • A master whose work_fn never yields for 4 s, run in its own process under -t 1 and -t 2. A sibling in this process tries to take the key every 0.1 s (stale_after = 1). It must never get the key before the master commits, and the master must commit. The test also checks the master really computed for ≥ 3.5 s and that there were ≥ 25 attempts, so a master that dies at once cannot pass it.
  • A master whose key is taken during work_fn commits nothing and leaves the sibling's lock in place.
• test_run_loop_busy.jl: back to elapsed >= stale_after. It is now timed from before the stale lock is written, so no allowance is needed. The 1 s allowance in Two run tests assumed timings a hosted runner does not give (0.6.8) #60 was for DataVault's whole-second heartbeat, which 0.8.9 replaces.
• test_run_deadline.jl: the first key runs until a deadline 5 s ahead has passed, so the test no longer depends on how fast run! starts (from Two run tests assumed timings a hosted runner does not give (0.6.8) #60).
• test_double_run_safety.jl / test_run_adversarial.jl: a lost key is simulated with a sibling's lock, and a crashed holder by making both heartbeat lines old.

Run locally against DataVault#75 (both branches loaded in a throwaway env), the touched and related files pass:

• busy_holder, double_run_safety, loop_busy, deadline, adversarial, kill_recovery, liveness, minimal, loop, result_digest, stop_flag
• keylock and artifacts, after two DataVault-side fixes that they uncovered:
  • keylock computed 2 of its keys twice, because of the order in which DataVault's owner-checked commit wrote the marker; fixed in DataVault b4fecc8.
  • artifacts called a DataVault internal I had renamed, which showed up as a 600 s wait; the old name is kept.

🤖 Generated with Claude Code

[github-actions]

📚 Docs preview: https://qatlashub.github.io/SweepRunner.jl/previews/PR61/

(updates on each push to this PR)

[codecov]

Codecov Report

❌ Patch coverage is 77.77778% with 2 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
src/Run.jl	77.77%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!