fix: write workspace-state and package-info paths relative to scratch dir#37
Open
pepicrft wants to merge 2 commits into
Open
fix: write workspace-state and package-info paths relative to scratch dir#37pepicrft wants to merge 2 commits into
pepicrft wants to merge 2 commits into
Conversation
pepicrft
requested review from
esnunes and
fortmarek
and removed request for
a team
June 12, 2026 15:38
3 tasks
… dir Emit every path swifterpm writes into `.build/workspace-state.json` and `.build/swifterpm/package-info/index.json` as relative to the scratch directory instead of the host's absolute prefix, so a cached `.build/` can be restored on a different host (or under a different checkout location on the same host) and the metadata still resolves to real files. What changes - `Restore.writeWorkspaceState`: `Artifact.path`, fileSystem `Dependency.state.path`, and `packageRef.location` for root / fileSystem / localSourceControl kinds are now anchored at `scratchDir` via a new `URL.relativePathString(to:)` helper (uses `tuist/Path`'s `relative(to:)` so we get `..` components when needed). Remote URLs and registry identities pass through unchanged. - `PackageInfoCacheWriter.write`: same treatment for `package_path`, `package_info_path`, and `location` on every entry. Bumps `schema_version` from 1 to 2 so downstream consumers can detect the new encoding. - Existing prebuilt entries are still forwarded verbatim through `existingWorkspacePrebuilts`; they originate from SwiftPM's prebuilt manager and are out of scope here, and downstream readers can handle either form by anchoring with `AbsolutePath(validating:relativeTo:)`. The on-disk content (`checkouts/`, `registry/downloads/`, `swifterpm/artifacts/`) is unchanged: only the metadata files are rewritten. Running new swifterpm against an old `.build/` is idempotent — both JSON files are overwritten on every resolve, so one `tuist install` (or `swift package resolve` via swifterpm) flips the state. Pairs with tuist/tuist#11262 which teaches Tuist's `SwiftPackageManagerGraphLoader` to anchor relative paths back against the scratch directory; that PR is permissive (still accepts absolute paths), so the two changes can land in either order. Tests - Existing assertions updated to expect the relative encoding (artifact paths, fileSystem dep locations, `schema_version`, etc.). - New `writeWorkspaceStateEmitsScratchRelativeArtifactAndDependencyPaths` regression asserts no emitted path starts with `/` and that re-anchoring against the scratch directory still resolves to the on-disk artifact. - Full suite: 105/105 passing. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The first pass relativized everything unconditionally, which broke the e2e differential suite: external `localSourceControl` and `fileSystem` deps live outside the consuming project, and rewriting them as seven-`..` chains diverges from SwiftPM's output for paths that inherently cannot be made portable across hosts. Scope the relative-path encoding to URLs that resolve inside either `packageDir` (the consuming project) or `scratchDir` (the build dir). Paths that escape both stay absolute, matching SwiftPM byte-for-byte for the e2e checks. The portability story for `.build/` is unchanged: every file that lives inside the cacheable unit (artifacts, in-tree binary xcframeworks, in-tree fileSystem deps, package-info entries) is still emitted relative to scratch. Use a string-level `/private/var/` -> `/var/` collapse for the "inside packageDir or scratchDir" check rather than POSIX `realpath`. `realpath` resolves every symlink, and on dev machines `<scratch>/swifterpm/artifacts/<id>/<target>` is a symlink into the global cache; following it would make a path that's logically inside scratch look like it escapes. Tests now assert the clean relative form directly (e.g. `../Package/LocalOne`) instead of constructing it from `PathCanonicalizer.realpath(...).relativePathString(to: scratch)`, which mixed canonicalized self with non-canonicalized base. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
pepicrft
force-pushed
the
fix/relative-paths-for-cacheable-build-state
branch
from
June 15, 2026 11:55
cb4ae7e to
4204020
Compare
Blick review didn't runThe See the workflow run for details: https://github.com/tuist/swifterpm/actions/runs/27544499325 Commit: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Make
.build/portable by writing paths inworkspace-state.jsonandswifterpm/package-info/index.jsonrelative to the scratch directory instead of with the host's absolute prefix. After this, a cached.build/restored on a different host (or under a different checkout location) still resolves to real files without rewriting either JSON.Pairs with tuist/tuist#11262, which teaches Tuist's
SwiftPackageManagerGraphLoaderto anchor the relative paths back against the scratch directory. That loader change is permissive (AbsolutePath(validating:relativeTo:)accepts both absolute and relative input), so the two PRs can land in either order: today's absolute output keeps working after the Tuist loader change, and tomorrow's relative output Just Works once the swifterpm pin is bumped.Motivation
Right after the recent
tuist/tuistswifterpm bump, someone asked whether.build/could be cached across CI runs. Walking through what leaks even on CI:replaceWithCachedDirectoryalready copies the global SwifterPM cache into.build/on CI (Sources/swifterpm/FileSystemSupport.swift:122), so symlink targets in~/.cache/swifterpm/...are not a blocker there.Artifact.path, fileSystemDependency.state.path/packageRef.location, and every entry of the package-info index. That means caching only works if the checkout path is identical across runs, which is fragile.Anchoring those paths to
scratchDirremoves the last leak. The on-disk content (checkouts/,registry/downloads/,swifterpm/artifacts/) is unchanged and content-addressed already.Changes
Sources/swifterpm/FileSystemSupport.swiftURL.relativePathString(to:)helper. Usestuist/Path'srelative(to:)so paths that escape the base get..components instead of failing.Sources/swifterpm/Restore.swiftwriteWorkspaceState:Artifact.pathand the fileSystem dep'sstate.pathare written relative toscratchDir.rootPackageRef,fileSystemPackageRef, andpackageRef(_:scratchDir:)(the source-control variant):locationis rewritten relative toscratchDirfor kinds where it carries a filesystem path (root,fileSystem,localSourceControl). Remote URLs and registry identities stay verbatim.existingWorkspacePrebuilts. Those originate from SwiftPM's prebuilt manager, not us, so we don't touch their form.Sources/swifterpm/PackageInfoCache.swiftPackageInfoCacheWriter.write: root + per-packagepackage_path,package_info_path, andlocation(forlocalSourceControl/fileSystem) are all written relative toscratchDir.schema_versionfrom1to2so downstream consumers can detect the new encoding.Migration story
Running new swifterpm against an old
.build/is idempotent. Both JSON files are overwritten on every resolve, so a singletuist install(orswift package resolvevia swifterpm) is enough to flip the state to relative encoding. On-disk content is reused, no re-downloads. Going the other way (downgrading swifterpm against a.build/already in relative form) would re-resolve normally and rewrite both files back to absolute.Fields converted from absolute to scratch-relative
Each field below is relativized to
scratchDironly when the target path resolves inside the consuming project (either underpackageDiror underscratchDir). Paths that escape both — typically externalfileSystem/localSourceControldeps living on another part of the disk — stay absolute, matching SwiftPM's output for entries that are inherently not portable across hosts. The/private/varvs/varsymlink layer on macOS is collapsed with a string substitution rather thanrealpath, since following every symlink would resolve<scratch>/swifterpm/artifacts/<id>/<target>out into the global cache and make a logically-in-scratch path look like it escapes.For reference, every field this PR touches in those two metadata files, and what each one points at on disk:
.build/workspace-state.jsonobject.artifacts[].path<scratch>/swifterpm/artifacts/<id>/<target>/<target>.xcframeworkfor remote / local-zip sources, or at the in-repo location for local non-zip xcframeworksobject.dependencies[].packageRef.location(kindsroot,fileSystem,localSourceControl)root, an external local package forfileSystem/localSourceControlobject.dependencies[].state.path(kindfileSystem)packageRef.locationobject.prebuilts[].path/checkoutPath/includePathAbsolutePath(validating:relativeTo:)Remote URLs (
packageRef.locationforremoteSourceControl) and registry identities (locationforregistry) are not paths and stay verbatim..build/swifterpm/package-info/index.jsonroot.package_path,packages[].package_path<scratch>/checkouts/...for SCM,<scratch>/registry/downloads/...for registry, or an external local path forfileSystemroot.package_info_path,packages[].package_info_path<scratch>/swifterpm/package-info/...)root.location,packages[].location(kindsroot,localSourceControl,fileSystem)package_pathHow binary artifacts are scoped
For both the cross-host caching story and reviewer sanity, this is where binaries actually live:
cache.binaryArtifactDirectory(identity:, targetName:, checksum:)(swifterpm/Sources/swifterpm/Restore.swift:130) returns a checksum-namespaced path under the global SwifterPM cache:~/.cache/swifterpm/artifacts/<identity>/<target>-<checksum-prefix>/. Two checksums of the same identity / target cannot collide.downloadBinaryArtifact(Restore.swift:209) downloads the archive intocache.binaryArtifactArchivePath(also checksum-keyed), verifies the SHA-256, and extracts into that cached directory. Re-download is gated byvalidCachedBinaryArtifactArchiveso checksum-good archives skip re-hashing.artifactDirectory(scratchDir:, packageIdentity:, targetName:)(Restore.swift:468) is the per-project landing spot:<scratch>/swifterpm/artifacts/<id>/<target>/.replaceScratchArtifact(Restore.swift:197) materialises the cached dir at that scratch path throughfileSystem.replaceWithSymlinkedDirectory. Inside that helper (FileSystemSupport.swift:114) the branch isEnvironment.isCI→ copy on CI, symlink on dev.workspaceArtifact(Restore.swift:972) callsbinaryArtifact(in: directory)against that scratch-side path and writes its location — which after this PR isswifterpm/artifacts/<id>/<target>/<target>.xcframeworkrelative toscratchDir.So
artifact.pathalways resolves to a file inside<scratch>/swifterpm/artifacts/<id>/<target>/. The dev-vs-CI distinction is what makes the relative-path encoding actually useful for cross-host caching:.build/is self-contained: the relativeartifact.pathplus the on-disk file resolves end-to-end without referencing the host's home directory at all.~/.cache/swifterpm/artifacts/.... Restoring.build/from another machine works at the metadata layer (the relative path still anchors against scratch), but the symlink target won't exist on the new host. Solution is either to cache~/.cache/swifterpm/alongside.build/, or to run onetuist installto re-create the symlinks (cheap — checksums match, no re-download).Considered alternatives
package-info/index.json. Smaller blast radius, no schema divergence from SwiftPM's spec file. Butworkspace-state.jsonis the file that carriesArtifact.pathand fileSystem-dep paths, and in the Tuist pipeline onlySwiftPackageManagerGraphLoaderreads it (SwiftPM itself is no longer in that path) — so the cost of diverging is low and the benefit is full.build/portability.${SCRATCH_DIR}/...) instead of plain relative paths. More explicit about anchoring intent, but invents a syntax SwiftPM never used. Plain relative paths plusAbsolutePath(validating:relativeTo:)on read handle the same cases without a new format.Test plan
schema_version, etc.)writeWorkspaceStateEmitsScratchRelativeArtifactAndDependencyPathsregression: asserts no emitted path starts with/and that re-anchoring against the scratch directory still resolves to the on-disk artifact🤖 Generated with Claude Code