Skip to content

fix: write workspace-state and package-info paths relative to scratch dir#37

Open
pepicrft wants to merge 2 commits into
mainfrom
fix/relative-paths-for-cacheable-build-state
Open

fix: write workspace-state and package-info paths relative to scratch dir#37
pepicrft wants to merge 2 commits into
mainfrom
fix/relative-paths-for-cacheable-build-state

Conversation

@pepicrft

@pepicrft pepicrft commented Jun 12, 2026

Copy link
Copy Markdown
Contributor

Summary

Make .build/ portable by writing paths in workspace-state.json and swifterpm/package-info/index.json relative to the scratch directory instead of with the host's absolute prefix. After this, a cached .build/ restored on a different host (or under a different checkout location) still resolves to real files without rewriting either JSON.

Pairs with tuist/tuist#11262, which teaches Tuist's SwiftPackageManagerGraphLoader to anchor the relative paths back against the scratch directory. That loader change is permissive (AbsolutePath(validating:relativeTo:) accepts both absolute and relative input), so the two PRs can land in either order: today's absolute output keeps working after the Tuist loader change, and tomorrow's relative output Just Works once the swifterpm pin is bumped.

Motivation

Right after the recent tuist/tuist swifterpm bump, someone asked whether .build/ could be cached across CI runs. Walking through what leaks even on CI:

  • replaceWithCachedDirectory already copies the global SwifterPM cache into .build/ on CI (Sources/swifterpm/FileSystemSupport.swift:122), so symlink targets in ~/.cache/swifterpm/... are not a blocker there.
  • But the metadata files still embed the host's absolute scratch-dir prefix in Artifact.path, fileSystem Dependency.state.path / packageRef.location, and every entry of the package-info index. That means caching only works if the checkout path is identical across runs, which is fragile.

Anchoring those paths to scratchDir removes the last leak. The on-disk content (checkouts/, registry/downloads/, swifterpm/artifacts/) is unchanged and content-addressed already.

Changes

Sources/swifterpm/FileSystemSupport.swift

  • New URL.relativePathString(to:) helper. Uses tuist/Path's relative(to:) so paths that escape the base get .. components instead of failing.

Sources/swifterpm/Restore.swift

  • writeWorkspaceState: Artifact.path and the fileSystem dep's state.path are written relative to scratchDir.
  • rootPackageRef, fileSystemPackageRef, and packageRef(_:scratchDir:) (the source-control variant): location is rewritten relative to scratchDir for kinds where it carries a filesystem path (root, fileSystem, localSourceControl). Remote URLs and registry identities stay verbatim.
  • Existing prebuilt entries are still forwarded verbatim through existingWorkspacePrebuilts. Those originate from SwiftPM's prebuilt manager, not us, so we don't touch their form.

Sources/swifterpm/PackageInfoCache.swift

  • PackageInfoCacheWriter.write: root + per-package package_path, package_info_path, and location (for localSourceControl / fileSystem) are all written relative to scratchDir.
  • Bumps schema_version from 1 to 2 so downstream consumers can detect the new encoding.

Migration story

Running new swifterpm against an old .build/ is idempotent. Both JSON files are overwritten on every resolve, so a single tuist install (or swift package resolve via swifterpm) is enough to flip the state to relative encoding. On-disk content is reused, no re-downloads. Going the other way (downgrading swifterpm against a .build/ already in relative form) would re-resolve normally and rewrite both files back to absolute.

Fields converted from absolute to scratch-relative

Each field below is relativized to scratchDir only when the target path resolves inside the consuming project (either under packageDir or under scratchDir). Paths that escape both — typically external fileSystem / localSourceControl deps living on another part of the disk — stay absolute, matching SwiftPM's output for entries that are inherently not portable across hosts. The /private/var vs /var symlink layer on macOS is collapsed with a string substitution rather than realpath, since following every symlink would resolve <scratch>/swifterpm/artifacts/<id>/<target> out into the global cache and make a logically-in-scratch path look like it escapes.

For reference, every field this PR touches in those two metadata files, and what each one points at on disk:

.build/workspace-state.json

Field What the path targets on disk
object.artifacts[].path The xcframework or artifactbundle binary itself, at <scratch>/swifterpm/artifacts/<id>/<target>/<target>.xcframework for remote / local-zip sources, or at the in-repo location for local non-zip xcframeworks
object.dependencies[].packageRef.location (kinds root, fileSystem, localSourceControl) A source package directory on disk: the consuming project for root, an external local package for fileSystem / localSourceControl
object.dependencies[].state.path (kind fileSystem) Same target as the matching packageRef.location
object.prebuilts[].path / checkoutPath / includePath Extracted prebuilt libs (SwiftSyntax-style). Forwarded as-is — these entries are written by SwiftPM's prebuilt manager, not by swifterpm, so this PR doesn't rewrite them. Downstream loaders can still anchor them via AbsolutePath(validating:relativeTo:)

Remote URLs (packageRef.location for remoteSourceControl) and registry identities (location for registry) are not paths and stay verbatim.

.build/swifterpm/package-info/index.json

Field What the path targets on disk
root.package_path, packages[].package_path The source package directory: under <scratch>/checkouts/... for SCM, <scratch>/registry/downloads/... for registry, or an external local path for fileSystem
root.package_info_path, packages[].package_info_path The dumped per-package manifest JSON inside the cache directory (default <scratch>/swifterpm/package-info/...)
root.location, packages[].location (kinds root, localSourceControl, fileSystem) Same target as the matching package_path

How binary artifacts are scoped

For both the cross-host caching story and reviewer sanity, this is where binaries actually live:

  1. cache.binaryArtifactDirectory(identity:, targetName:, checksum:) (swifterpm/Sources/swifterpm/Restore.swift:130) returns a checksum-namespaced path under the global SwifterPM cache: ~/.cache/swifterpm/artifacts/<identity>/<target>-<checksum-prefix>/. Two checksums of the same identity / target cannot collide.
  2. downloadBinaryArtifact (Restore.swift:209) downloads the archive into cache.binaryArtifactArchivePath (also checksum-keyed), verifies the SHA-256, and extracts into that cached directory. Re-download is gated by validCachedBinaryArtifactArchive so checksum-good archives skip re-hashing.
  3. artifactDirectory(scratchDir:, packageIdentity:, targetName:) (Restore.swift:468) is the per-project landing spot: <scratch>/swifterpm/artifacts/<id>/<target>/.
  4. replaceScratchArtifact (Restore.swift:197) materialises the cached dir at that scratch path through fileSystem.replaceWithSymlinkedDirectory. Inside that helper (FileSystemSupport.swift:114) the branch is Environment.isCIcopy on CI, symlink on dev.
  5. workspaceArtifact (Restore.swift:972) calls binaryArtifact(in: directory) against that scratch-side path and writes its location — which after this PR is swifterpm/artifacts/<id>/<target>/<target>.xcframework relative to scratchDir.

So artifact.path always resolves to a file inside <scratch>/swifterpm/artifacts/<id>/<target>/. The dev-vs-CI distinction is what makes the relative-path encoding actually useful for cross-host caching:

  • On CI: the scratch-side directory is a real copy of the xcframework / artifactbundle. .build/ is self-contained: the relative artifact.path plus the on-disk file resolves end-to-end without referencing the host's home directory at all.
  • On dev: the scratch-side entry is a symlink into ~/.cache/swifterpm/artifacts/.... Restoring .build/ from another machine works at the metadata layer (the relative path still anchors against scratch), but the symlink target won't exist on the new host. Solution is either to cache ~/.cache/swifterpm/ alongside .build/, or to run one tuist install to re-create the symlinks (cheap — checksums match, no re-download).

Considered alternatives

  • Keep workspace-state.json absolute, only fix the swifterpm-owned package-info/index.json. Smaller blast radius, no schema divergence from SwiftPM's spec file. But workspace-state.json is the file that carries Artifact.path and fileSystem-dep paths, and in the Tuist pipeline only SwiftPackageManagerGraphLoader reads it (SwiftPM itself is no longer in that path) — so the cost of diverging is low and the benefit is full .build/ portability.
  • Use placeholder substitution (${SCRATCH_DIR}/...) instead of plain relative paths. More explicit about anchoring intent, but invents a syntax SwiftPM never used. Plain relative paths plus AbsolutePath(validating:relativeTo:) on read handle the same cases without a new format.

Test plan

  • Existing assertions updated to expect the relative encoding (artifact paths, fileSystem dep locations, schema_version, etc.)
  • New writeWorkspaceStateEmitsScratchRelativeArtifactAndDependencyPaths regression: asserts no emitted path starts with / and that re-anchoring against the scratch directory still resolves to the on-disk artifact
  • Full suite: 105/105 passing locally (`swift test`)
  • CI green
  • Bump the pin in fix(cli): anchor SwiftPM workspace-state paths to scratch directory tuist#11262 once this merges + tags, and confirm end-to-end resolve + generate against the new revision

🤖 Generated with Claude Code

@pepicrft
pepicrft requested a review from a team as a code owner June 12, 2026 15:38
@pepicrft
pepicrft requested review from esnunes and fortmarek and removed request for a team June 12, 2026 15:38
pepicrft and others added 2 commits June 15, 2026 13:45
… dir

Emit every path swifterpm writes into `.build/workspace-state.json` and
`.build/swifterpm/package-info/index.json` as relative to the scratch
directory instead of the host's absolute prefix, so a cached `.build/`
can be restored on a different host (or under a different checkout
location on the same host) and the metadata still resolves to real files.

What changes
- `Restore.writeWorkspaceState`: `Artifact.path`, fileSystem
  `Dependency.state.path`, and `packageRef.location` for root /
  fileSystem / localSourceControl kinds are now anchored at `scratchDir`
  via a new `URL.relativePathString(to:)` helper (uses `tuist/Path`'s
  `relative(to:)` so we get `..` components when needed). Remote URLs
  and registry identities pass through unchanged.
- `PackageInfoCacheWriter.write`: same treatment for `package_path`,
  `package_info_path`, and `location` on every entry. Bumps
  `schema_version` from 1 to 2 so downstream consumers can detect the
  new encoding.
- Existing prebuilt entries are still forwarded verbatim through
  `existingWorkspacePrebuilts`; they originate from SwiftPM's prebuilt
  manager and are out of scope here, and downstream readers can handle
  either form by anchoring with `AbsolutePath(validating:relativeTo:)`.

The on-disk content (`checkouts/`, `registry/downloads/`,
`swifterpm/artifacts/`) is unchanged: only the metadata files are
rewritten. Running new swifterpm against an old `.build/` is idempotent
— both JSON files are overwritten on every resolve, so one
`tuist install` (or `swift package resolve` via swifterpm) flips the
state.

Pairs with tuist/tuist#11262 which teaches Tuist's
`SwiftPackageManagerGraphLoader` to anchor relative paths back against
the scratch directory; that PR is permissive (still accepts absolute
paths), so the two changes can land in either order.

Tests
- Existing assertions updated to expect the relative encoding (artifact
  paths, fileSystem dep locations, `schema_version`, etc.).
- New `writeWorkspaceStateEmitsScratchRelativeArtifactAndDependencyPaths`
  regression asserts no emitted path starts with `/` and that
  re-anchoring against the scratch directory still resolves to the
  on-disk artifact.
- Full suite: 105/105 passing.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The first pass relativized everything unconditionally, which broke the
e2e differential suite: external `localSourceControl` and `fileSystem`
deps live outside the consuming project, and rewriting them as
seven-`..` chains diverges from SwiftPM's output for paths that
inherently cannot be made portable across hosts.

Scope the relative-path encoding to URLs that resolve inside either
`packageDir` (the consuming project) or `scratchDir` (the build dir).
Paths that escape both stay absolute, matching SwiftPM byte-for-byte for
the e2e checks. The portability story for `.build/` is unchanged: every
file that lives inside the cacheable unit (artifacts, in-tree binary
xcframeworks, in-tree fileSystem deps, package-info entries) is still
emitted relative to scratch.

Use a string-level `/private/var/` -> `/var/` collapse for the
"inside packageDir or scratchDir" check rather than POSIX `realpath`.
`realpath` resolves every symlink, and on dev machines
`<scratch>/swifterpm/artifacts/<id>/<target>` is a symlink into the
global cache; following it would make a path that's logically inside
scratch look like it escapes.

Tests now assert the clean relative form directly (e.g.
`../Package/LocalOne`) instead of constructing it from
`PathCanonicalizer.realpath(...).relativePathString(to: scratch)`, which
mixed canonicalized self with non-canonicalized base.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@pepicrft
pepicrft force-pushed the fix/relative-paths-for-cacheable-build-state branch from cb4ae7e to 4204020 Compare June 15, 2026 11:55
@github-actions

Copy link
Copy Markdown

Blick review didn't run

The blick review step failed before producing a manifest, so there's no review to post on this PR. This usually means the agent (opencode) couldn't start — common causes are an expired or suspended model API key, a missing secret, or the workflow timing out.

See the workflow run for details: https://github.com/tuist/swifterpm/actions/runs/27544499325

Commit: 4204020d04dd908df95199e19a7be48cf895956d

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant