Add buildcache channel for Build Caching Service runtime resolution#878
Add buildcache channel for Build Caching Service runtime resolution#878LoopedBard3 wants to merge 2 commits intodotnet:mainfrom
Conversation
There was a problem hiding this comment.
Pull request overview
This PR adds a new buildcache channel to Crank’s agent/runtime resolution pipeline so the runtime binaries can be sourced per-commit from the Build Caching Service (BCS), enabling finer-grained performance regression bisection than VMR feed cadence allows.
Changes:
- Added build-cache-related properties to
Jobfor selecting commit/branch/config. - Added
BuildCacheClientfor resolving latest builds, downloading/extracting artifacts, and overlaying runtime binaries. - Updated agent startup/version resolution logic and docs to support the new
buildcachechannel and new agent CLI options.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
| src/Microsoft.Crank.Models/Job.cs | Adds job-level inputs for BCS commit/branch/config selection. |
| src/Microsoft.Crank.Agent/Startup.cs | Adds CLI options and integrates buildcache into runtime version resolution + post-publish overlay. |
| src/Microsoft.Crank.Agent/BuildCacheClient.cs | Implements BCS querying/downloading/extraction and overlay helpers. |
| docs/dotnet_versions.md | Documents buildcache channel usage and configuration knobs. |
| docs/build_cache_requirements.md | Describes required BCS-side blob layout/access for the integration. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| var overlayCount = BuildCacheClient.OverlayPublishedOutput(buildCacheExtractDir, outputFolder); | ||
| Log.Info($"Build Cache: Overlaid {overlayCount} runtime files from commit {buildCacheCommitSha.Substring(0, 8)} into published output"); | ||
|
|
||
| // Update the reported runtime version to reflect the BCS commit | ||
| runtimeVersion = $"{runtimeVersion}+buildcache.{buildCacheCommitSha.Substring(0, 8)}"; | ||
| } | ||
| catch (Exception ex) |
There was a problem hiding this comment.
runtimeVersion is being mutated to include +buildcache.<sha> and then passed into PatchRuntimeConfig. For framework-dependent publishes (i.e., when runtimeOptions does not contain includedFrameworks), this will write a runtime version that is not actually installed, which can prevent the app from starting. Keep the feed-resolved runtime version for runtimeconfig patching and store the BCS commit separately for reporting/metadata, or gate buildcache usage to self-contained publishes only.
| // Build Cache: overlay BCS runtime binaries into the published output | ||
| // This replaces the NuGet-sourced runtime DLLs with BCS-built ones from the specific commit | ||
| if (useBuildCache && buildCacheExtractDir != null) | ||
| { | ||
| try | ||
| { | ||
| var overlayCount = BuildCacheClient.OverlayPublishedOutput(buildCacheExtractDir, outputFolder); | ||
| Log.Info($"Build Cache: Overlaid {overlayCount} runtime files from commit {buildCacheCommitSha.Substring(0, 8)} into published output"); | ||
|
|
||
| // Update the reported runtime version to reflect the BCS commit | ||
| runtimeVersion = $"{runtimeVersion}+buildcache.{buildCacheCommitSha.Substring(0, 8)}"; | ||
| } | ||
| catch (Exception ex) | ||
| { | ||
| Log.Info($"Build Cache: Warning - overlay failed: {ex.Message}. Published app will use feed-sourced runtime."); | ||
| } | ||
| } | ||
|
|
||
| PatchRuntimeConfig(job, outputFolder, aspNetCoreVersion, runtimeVersion); |
There was a problem hiding this comment.
BuildCache overlay is applied to the published output, but for non-self-contained publishes the output typically won’t contain runtime binaries to replace (and PatchRuntimeConfig will still run). If buildcache is intended to work only with job.SelfContained, enforce that with a clear error message; otherwise, consider using the existing BuildCacheClient.InstallRuntimeFromBuildCacheAsync approach to overlay into dotnetHome for framework-dependent runs.
| private static readonly HttpClient _httpClient = new HttpClient { Timeout = TimeSpan.FromMinutes(10) }; | ||
|
|
||
| // Cache latestBuilds.json responses to avoid repeated downloads | ||
| private static readonly Dictionary<string, (DateTimeOffset fetchedAt, LatestBuildsResponse data)> _latestBuildsCache = new(); | ||
| private static readonly TimeSpan _latestBuildsCacheDuration = TimeSpan.FromHours(1); | ||
|
|
||
| // Cache of already-installed BCS commit SHAs to avoid re-extracting | ||
| private static readonly HashSet<string> _installedBuildCacheRuntimes = new(StringComparer.OrdinalIgnoreCase); | ||
|
|
There was a problem hiding this comment.
_latestBuildsCache and _installedBuildCacheRuntimes are static non-thread-safe collections. The agent can process multiple jobs concurrently, so concurrent accesses can corrupt these collections or throw. Use ConcurrentDictionary/ConcurrentHashSet (or locks) around these caches, and also include baseUrl in the latestBuilds cache key to avoid cross-contaminating entries when the base URL is overridden.
| ``` | ||
| GET https://pvscmdupload.blob.core.windows.net/$web/builds/{repoName}/latest/{branch}/latestBuilds.json | ||
| GET https://pvscmdupload.blob.core.windows.net/$web/builds/{repoName}/buildArtifacts/{commitSha}/{configKey}/{artifactFile} | ||
| ``` |
There was a problem hiding this comment.
This doc’s example URLs use the blob endpoint with /$web/..., but the agent default base URL (and client URL construction) uses the static website endpoint (https://pvscmdupload.z22.web.core.windows.net) without /$web. Please align the documented URLs with what the agent actually calls (or clarify which base URL should be configured via --build-cache-base-url).
Adds a new 'buildcache' channel that resolves .NET runtime binaries from the Build Caching Service (BCS) instead of VMR feeds. This provides per-commit granularity for performance regression bisection. Key changes: - New Job.cs properties: BuildCacheCommitSha, BuildCacheBranch, BuildCacheConfig - New BuildCacheClient.cs: HTTP client for BCS latestBuilds.json and artifact download/extraction with post-build overlay into published output - Startup.cs: 'buildcache' channel in version resolution that builds with real NuGet packages then overlays BCS runtime binaries (194 files) after publish - Agent CLI options: --build-cache-base-url, --build-cache-repo-name, --build-cache-disabled - Documentation in docs/dotnet_versions.md and docs/build_cache_requirements.md Usage: --application.channel buildcache [--application.buildCacheCommitSha <sha>] Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
LoopedBard3
force-pushed
the
AddBuildCacheSystemSupportForCrank
branch
from
99d0195 to
41394e4
Compare
Changes:
* BuildCacheClient: rewrite to drop ~250 LOC of dead code from an earlier
abandoned design (synthesizing a new shared-framework dir with a synthetic
version). The FDD overlay path is restored as a new public API.
* New OverlayDotnetHome(extractDir, dotnetHome, runtimeVersion): overlays BCS
bits into dotnetHome/shared/Microsoft.NETCore.App/{runtimeVersion}/ and
host/fxr/{runtimeVersion}/ + the dotnet host. Wired into Startup.cs after
publish so framework-dependent jobs actually run against BCS bits instead of
silently using the feed runtime.
* OverlayPublishedOutput: copy ALL managed/native runtime files
unconditionally (previously skipped any file not already in destination,
which would silently drop new DLLs introduced by the BCS commit). Also
copies hostfxr/hostpolicy/dotnet for self-contained.
* Hardening:
- URL-encode repoName/commitSha/buildCacheConfig/branch with
Uri.EscapeDataString.
- Atomic download (.partial -> rename) and Content-Length validation so
truncated archives are not reused after a failed run.
- Retry transient HTTP failures via ProcessUtil.RetryOnExceptionAsync.
- Replace shelling out to tar with System.Formats.Tar.TarFile.
- Wrap synchronous archive extraction in Task.Run.
- Per-(commit,config) SemaphoreSlim + per-call unique extract dir to avoid
races between concurrent jobs.
- Drop unused targetFramework parameter from DownloadAndExtractAsync.
- All commit-SHA Substring uses go through ShortSha (Math.Min length guard).
- ParseLatestBuilds: case-insensitive branch_name/BranchName handling and
skip non-object metadata properties safely.
* Startup.cs:
- Validate user-supplied BuildCacheCommitSha length (>= 8 chars) up front
instead of crashing later.
- Stop mutating runtimeVersion before PatchRuntimeConfig - use feed-resolved
version so runtimeconfig.json points to a really-installed shared
framework dir. Suffix +buildcache.<sha> is now applied to
job.RuntimeVersion only, after PatchRuntimeConfig has run.
- Treat 0-file overlay as fatal (job.Error + return null) so silent failures
do not produce wrong-runtime benchmarks.
- Promote overlay-failure log from Info to fatal job error.
* docs/dotnet_versions.md: add trailing newline.
* test/Microsoft.Crank.UnitTests/BuildCacheClientTests.cs: 17 new tests
covering ParseLatestBuilds (PascalCase / snake_case / mixed / missing
fields / case-insensitive lookup / non-object values), GetPlatformMoniker,
ShortSha, GetNativeLibName, OverlayPublishedOutput unconditional copy,
pdb/dbg skip, OverlayDotnetHome shared-framework precondition, and full
shared-framework + hostfxr overlay.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds a new 'buildcache' channel that resolves .NET runtime binaries from the Build Caching Service (BCS) instead of VMR feeds. This provides per-commit granularity for performance regression bisection.
Key changes:
Usage: --application.channel buildcache [--application.buildCacheCommitSha ]