Skip to content

feat: share a 60-second fetch window and wait for busy caches - #30

Merged
yuezengwu merged 12 commits into
mainfrom
feat/sync-freshness
Nov 4, 2026
Merged

yuezengwu merged 12 commits into
mainfrom
feat/sync-freshness

Conversation

@liuchao-001

Copy link
Copy Markdown
Contributor

Part of #29. Stacked on #28 (which includes #27); retarget to main after those merge.

Summary

  • Fetch freshness window. sync skips the network when the same branch of the repository cache was fetched within the last 60 seconds by any process on the machine, and resolves from the local ref. The fetch time is recorded per remote-tracking ref in repositories/<id>.git.fetched.json under the cache lock. sync --max-age <seconds> changes the window; --max-age 0 always fetches.
  • connect and write prepare always fetch, so a write starts from the remote head.
  • Wait instead of BUSY. Repository-cache and snapshot locks wait up to two minutes for a live owner (withLockWait), so concurrent agents share one clone or fetch instead of failing. Dead owners are still reclaimed; snapshot eviction keeps non-waiting locks and skips busy snapshots.
  • Spec updated.

Tests

  • New tests/fetch-freshness.test.ts: a recent fetch is reused, --max-age 0 fetches, an expired record fetches, and write prepare fetches despite a recent sync.
  • tests/state.test.ts: a sync waits for a live lock owner and proceeds once it exits; withLockWait reports busy after its timeout.
  • Tests that push and immediately expect new content now pass maxAge 0.
  • pnpm check, pnpm typecheck, pnpm test (125 tests), pnpm check:package pass. Under heavy local load some unrelated tests exceeded the default 5 s timeout once and passed on rerun.

Parallel fetch across sources is not in this PR: syncTrees is synchronous. OpenTag's command gateway runs one CLI call per source and can run them concurrently.

🤖 Generated with Claude Code

liuchao-001 and others added 8 commits November 4, 2026 17:38
Use a blobless partial clone (--filter=blob:none) for the bare repository
cache and an anchored, literal non-cone sparse pattern for subtree
snapshots and write worktrees, so only the selected subtree's file
contents are downloaded and materialized.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every fetching sync that lands on a new commit creates a snapshot, and
nothing removed the old ones, so the snapshot directory grew without
bound. A fetching sync now removes snapshots of the same repository cache
that are superseded and unused for 24 hours. Each resolution records a
use on its snapshot. Modified, locked, or busy snapshots and write-session
worktrees are kept, and eviction never fails the sync.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The blobless cache still downloaded every commit's full directory
structure (12 MB for a repository whose docs subtree is 1 MB). A
treeless clone stores commits only and downloads trees and contents
when a snapshot or write worktree checks them out. Covers commit,
publish, and rejected-push recovery through a treeless cache.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A sync of a branch fetched by any process within the last 60 seconds
reuses that fetch instead of contacting the remote; --max-age adjusts
the window and 0 always fetches. connect and write prepare always
fetch. Repository-cache and snapshot locks wait up to two minutes for a
live owner instead of failing as busy, so concurrent agents share one
fetch or clone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
liuchao-001 and others added 3 commits November 4, 2026 22:12
…s through removal

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Base automatically changed from feat/snapshot-gc to main November 4, 2026 14:42

@yuezengwu yuezengwu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 49873d4, including the corrected parent eviction behavior. No additional defect was found in fetch freshness, forced connect/write refresh, per-ref metadata, or bounded cache-lock waiting. All 126 tests passed at this exact head with HTTPS Git transport prohibited, along with repository checks, typecheck and real package smoke.

The PR has now been retargeted to main. Actual main 997a68c still conflicts in docs/specification.md and tests/snapshot-eviction.test.ts; my local integration attempt was aborted. This is a code approval, not a successful integration result. Resolve those conflicts on the author branch and validate the resulting head/main combination before merge.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@liuchao-001

Copy link
Copy Markdown
Contributor Author

Merged current main (with squashed #28) in a882727. main's tree equals the final feat/snapshot-gc tree, so the conflicts in docs/specification.md and tests/snapshot-eviction.test.ts resolve to this branch's side; the diff against main is unchanged from the diff you approved (10 files, +147/−31). pnpm check, typecheck, test, check:package pass locally (three tests hit the 5s timeout under host load and pass at a 30s timeout).

@liuchao-001
liuchao-001 requested a review from yuezengwu November 4, 2026 15:07

@yuezengwu yuezengwu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed a882727 against actual main 997a68c. This successor merges main, resolves the integration conflict, and is tree-identical to the independently verified 49873d4. Actual main is now an ancestor of this head; no source, tests, package manifests or lockfile changed in the merge. The prior 126 passing tests (HTTPS Git transport prohibited), repository check, typecheck and real package smoke therefore apply to this exact tree; repeating them would not add coverage. No remaining code finding in the shared 60-second fetch window, per-ref freshness metadata, forced refresh or bounded cache-lock waiting.

@yuezengwu
yuezengwu merged commit 6f09a30 into main Nov 4, 2026
3 checks passed
@yuezengwu
yuezengwu deleted the feat/sync-freshness branch November 4, 2026 15:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants