Skip to content

Stop overlapping kagent runs from overwriting each other's memory in the lane store #4910

Closed

Description

@devantler

🤖 Generated by the Agentic Engineer

Evidence

The kagent lane keeps its memory in the in-cluster lane-store (devantler-tech/platform#6014): each run downloads memory/, works on a local copy, and uploads it back. Nothing stops two runs from overlapping. The lane has no per-run time limit, and kagent's scheduled runs have no overlap guard (devantler-tech/platform#5999). A CodeRabbit review on #4905 raised this.

Problem

If two runs overlap, both download the same memory and the later upload overwrites everything the earlier run wrote. Memory has no version history, so those notes, cursors and claims are lost without a trace.

Hypothesis

If a run has to hold a single-writer lease on the store before it downloads, and releases the lease only after its upload, then overlapping runs cannot lose each other's memory. A second run sees the lease and stops before acting on memory.

Success signal

  • An overlap test against the real rclone serve webdav listener: two runs start, exactly one gets the lease, and the other stops with UNKNOWN and uploads nothing.
  • A lease left behind by a crashed run is reported, never silently taken over.

Smallest useful change

Pick a lease primitive the rclone WebDAV listener actually enforces atomically (for example, a MKCOL that must fail when the collection already exists), and prove it against the pinned image. Then add the lease steps to the lane's run message in agent.yaml, and to the durable-memory guide paragraph that #4905 adds.

Acceptance criteria

  • The lease is acquired before the download and released after the upload, on every exit path the run controls.
  • A run that cannot get the lease, or cannot tell whether it holds it, stops before acting on memory and uploads nothing.
  • The durable-memory guide names the lease and links this issue's fix.

Size: S–M.

Activity

  1. added theissue type on Nov 10, 2026
  2. moved this to 📥 Backlog in 🌊 Project Boardon Nov 10, 2026
  3. devantler commented on Nov 11, 2026

    @devantler
    ContributorAuthor

    Done and proven on a real run. The kagent lane's second full run (2026-11-11, started 22:05 UTC) used the single-writer lease end to end:

    • Taking the lease: PROPFIND memory/ returned 404 (first run), so the run created it with MKCOL (201). LOCK memory/ returned 200 with a numbered Lock-Token, and the reply named the run's owner (start time plus a random nonce). The token was about 34,000 s below the reply's Date, so the run treated it as safe straight away.
    • Uploading: the refresh returned 200 and named the same owner. Three memory files were uploaded one at a time, each PUT with If: (<token>) (201 each).
    • Releasing: UNLOCK returned 204, after every write under memory/ had succeeded.

    The design is in devantler-tech/platform#6070 and the guide in #4938. The first full run found that Substrate's egress gateway drops a connection that opens with a WebDAV method. devantler-tech/platform#6089 fixed that: the run message now sends every WebDAV method on a connection an OPTIONS opened. CI proves the lease semantics against the store's pinned rclone, and the transport against the gateway image Substrate deploys.

    🤖 Generated with Claude Code

  4. moved this from 📥 Backlog to ✅ Done in 🌊 Project Boardon Nov 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Fields

    Priority

    None yet

    Effort

    None yet

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions