Repository navigation
🐳 ci: Publish Code API Images to GHCR #307
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,35 @@ | ||
| #!/usr/bin/env bash | ||
| # Prints the org.opencontainers.image.revision label of an image reference. | ||
| # Used by .github/workflows/images.yml, which must tell a missing tag apart | ||
| # from a registry that cannot answer. | ||
| # | ||
| # Exit 0 the revision was printed. | ||
| # Exit 3 the tag does not exist, or this token may not read it (a token that | ||
| # may not read a repository may not write to it either). | ||
| # Exit 1 the registry did not answer after retries, or the image has no | ||
| # revision label. | ||
| set -uo pipefail | ||
|
|
||
| ref=${1:?usage: image-revision.sh REF} | ||
| attempts=${IMAGE_REVISION_ATTEMPTS:-3} | ||
| err=$(mktemp) | ||
| trap 'rm -f "$err"' EXIT | ||
|
|
||
| for attempt in $(seq 1 "$attempts"); do | ||
| if json=$(docker buildx imagetools inspect "$ref" --format '{{ json .Image }}' 2>"$err"); then | ||
| if jq -er 'if has("config") then . else .["linux/amd64"] end | ||
| | .config.Labels["org.opencontainers.image.revision"]' <<<"$json"; then | ||
| exit 0 | ||
| fi | ||
| echo "$ref has no revision label" >&2 | ||
| exit 1 | ||
| fi | ||
| if grep -qiE ': not found$|denied|\b40[134]\b' "$err"; then | ||
| exit 3 | ||
| fi | ||
| cat "$err" >&2 | ||
| if [ "$attempt" -lt "$attempts" ]; then | ||
| sleep $(( attempt * 5 )) | ||
| fi | ||
| done | ||
| exit 1 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,350 @@ | ||
| # Builds the Code API container images and publishes them to GHCR, so a host | ||
| # can pull a tested image instead of compiling one in place. Like ci.yml this | ||
| # file is inert inside the monorepo — GitHub only runs workflows from the repo | ||
| # root — and becomes a root workflow in the published repo. | ||
| # | ||
| # Entry points: | ||
| # | ||
| # * push to main — build every image, push `sha-<commit>` tags, and refresh | ||
| # each image's registry build cache. The promote job then points the | ||
| # `main` tags at the newest main commit that has all images. | ||
| # * pull_request — build every image without pushing, reading the cache that | ||
| # main wrote, so Dockerfile breakage surfaces before merge. | ||
| # * workflow_dispatch — on main, the same as a push; on any other branch, a | ||
| # build-only run. | ||
| # | ||
| # The image list lives in the plan job and feeds both the build matrix and the | ||
| # promotion, so the two cannot drift apart. README.md ("Prebuilt images") | ||
| # documents the tags and how a Compose host switches to them. | ||
| name: Images | ||
|
|
||
| on: | ||
| push: | ||
| branches: [main] | ||
| # Everything the Dockerfiles below COPY from the build context, plus this | ||
| # workflow. Every run builds every image, so a commit that has tags has | ||
| # them for all images; unchanged images are cache hits. | ||
| paths: &image-inputs | ||
| - .dockerignore | ||
| - .github/scripts/image-revision.sh | ||
| - .github/workflows/images.yml | ||
| - api/** | ||
| - docker/** | ||
| - javascript-packages.txt | ||
| - launcher/** | ||
| - packages/code/src/** | ||
| - service/** | ||
| - shared/** | ||
| pull_request: | ||
| paths: *image-inputs | ||
| workflow_dispatch: | ||
|
|
||
| permissions: | ||
| contents: read | ||
|
|
||
| concurrency: | ||
| # Each main commit gets its own group and is never cancelled, so no commit's | ||
| # build can be dropped by a later queued run (GitHub keeps only one pending | ||
| # run per group). Other refs share one group per ref and cancel superseded | ||
| # runs. | ||
| group: images-${{ github.ref == 'refs/heads/main' && github.sha || github.ref }} | ||
| cancel-in-progress: ${{ github.ref != 'refs/heads/main' }} | ||
|
|
||
| env: | ||
| NAMESPACE: ghcr.io/librechat-ai | ||
|
|
||
| jobs: | ||
| plan: | ||
| name: Plan | ||
| runs-on: ubuntu-24.04 | ||
| outputs: | ||
| images: ${{ steps.images.outputs.images }} | ||
| publish: ${{ github.repository == 'LibreChat-AI/code-interpreter' && github.ref == 'refs/heads/main' }} | ||
| steps: | ||
| - name: List images | ||
| id: images | ||
| # One image per distinct Compose build in docker-compose.yaml. The | ||
| # sandbox runner's two Compose targets (KVM_ENABLED=true/false) are | ||
| # published as two images. The Lambda MicroVM targets are not built | ||
| # here; they are not used by Compose hosts. | ||
| run: | | ||
| images=$(jq -c . <<'JSON' | ||
| [ | ||
| {"image": "code-interpreter-api", "dockerfile": "service/Dockerfile", "target": "api", | ||
| "description": "Code API HTTP server (Compose service api)", "timeout": 30}, | ||
| {"image": "code-interpreter-worker", "dockerfile": "service/Dockerfile", "target": "worker", | ||
| "description": "Code API job worker (Compose service service-worker)", "timeout": 30}, | ||
| {"image": "code-interpreter-file-server", "dockerfile": "service/Dockerfile", "target": "production", | ||
| "description": "Code API file server (Compose service file_server)", "timeout": 30}, | ||
| {"image": "code-interpreter-egress-gateway", "dockerfile": "service/Dockerfile.egress-gateway", "target": "production", | ||
| "description": "Sandbox egress gateway (Compose service egress_gateway)", "timeout": 30}, | ||
| {"image": "code-interpreter-tool-call-server", "dockerfile": "service/Dockerfile.tool-call-server", "target": "production", | ||
| "description": "Tool call server (Compose service tool_call_server)", "timeout": 30}, | ||
| {"image": "code-interpreter-sandbox-runner", "dockerfile": "api/Dockerfile", "target": "sandbox-runner-true", | ||
| "description": "libkrun microVM sandbox runner with the guest rootfs and runtime packages baked in (Compose service sandbox-runner, KVM_ENABLED=true)", | ||
| "timeout": 60, "free_disk": true}, | ||
| {"image": "code-interpreter-sandbox-runner-direct", "dockerfile": "api/Dockerfile", "target": "sandbox-runner-false", | ||
| "description": "Directory-root sandbox runner; runtime packages are mounted from the host (Compose service sandbox-runner, KVM_ENABLED=false)", | ||
| "timeout": 30} | ||
| ] | ||
| JSON | ||
| ) | ||
| echo "images=$images" >> "$GITHUB_OUTPUT" | ||
|
|
||
| build: | ||
| name: ${{ matrix.image }} | ||
| needs: plan | ||
| runs-on: ubuntu-24.04 | ||
| timeout-minutes: ${{ matrix.timeout }} | ||
| permissions: | ||
| contents: read | ||
| # Pushes happen only when needs.plan.outputs.publish is true (main). | ||
| # Pull request runs read the build cache and never push; GitHub caps a | ||
| # fork's token at read access regardless. | ||
| packages: write | ||
| strategy: | ||
| fail-fast: false | ||
| matrix: | ||
| include: ${{ fromJSON(needs.plan.outputs.images) }} | ||
| steps: | ||
| - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6 | ||
| with: | ||
| persist-credentials: false | ||
|
|
||
| - name: Free runner disk | ||
| # A cold KVM runner build peaks at roughly 17 GB of builder state. The | ||
| # hosted runner had about 87 GB free when measured, but GitHub only | ||
| # guarantees 14 GB, so this leg removes unused preinstalled SDKs | ||
| # (about 28 GB, 1.5 minutes) rather than depend on the surplus. | ||
| if: matrix.free_disk | ||
| run: | | ||
| sudo rm -rf \ | ||
| /opt/ghc \ | ||
| /opt/hostedtoolcache/CodeQL \ | ||
| /usr/local/.ghcup \ | ||
| /usr/local/lib/android \ | ||
| /usr/local/share/powershell \ | ||
| /usr/share/dotnet \ | ||
| /usr/share/swift | ||
| docker image prune --all --force >/dev/null | ||
| df -h / | ||
|
|
||
| - name: Sample free disk | ||
| run: | | ||
| nohup bash -c 'while :; do df -BM --output=avail / | tail -n1 | tr -dc 0-9; echo; sleep 10; done' \ | ||
| >"$RUNNER_TEMP/disk-avail-mib.log" 2>/dev/null & | ||
|
|
||
| - uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1 | ||
|
|
||
| - name: Log in to GHCR | ||
| # Pull request runs log in only to read the cache, and only from this | ||
| # repository; fork runs read public packages anonymously. | ||
| if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository | ||
| uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0 | ||
| with: | ||
| registry: ghcr.io | ||
| username: ${{ github.actor }} | ||
| password: ${{ github.token }} | ||
|
|
||
| - name: Check for an existing tag | ||
| # `sha-` tags are written once. A re-run (or a dispatch on the same | ||
| # commit) builds only the images whose tag is missing, so a failed | ||
| # re-run cannot leave the commit's set mixed across two builds. | ||
| id: existing | ||
| if: needs.plan.outputs.publish == 'true' | ||
| env: | ||
| REF: ${{ env.NAMESPACE }}/${{ matrix.image }}:sha-${{ github.sha }} | ||
| # A registry that cannot answer fails the job rather than risk | ||
| # rebuilding over a tag that may exist. | ||
| run: | | ||
| status=0 | ||
| revision=$(.github/scripts/image-revision.sh "$REF") || status=$? | ||
| case $status in | ||
| 0) | ||
| if [ "$revision" != "$GITHUB_SHA" ]; then | ||
| echo "::error::$REF records revision $revision" | ||
| exit 1 | ||
| fi | ||
| echo "$REF already exists; not rebuilding it." | ||
| echo "exists=true" >> "$GITHUB_OUTPUT" | ||
| ;; | ||
| 3) | ||
| echo "exists=false" >> "$GITHUB_OUTPUT" | ||
| ;; | ||
| *) | ||
| echo "::error::Could not read $REF from the registry." | ||
| exit 1 | ||
| ;; | ||
| esac | ||
|
|
||
| - name: Image metadata | ||
| id: meta | ||
| if: steps.existing.outputs.exists != 'true' | ||
| uses: docker/metadata-action@dc802804100637a589fabce1cb79ff13a1411302 # v6.2.0 | ||
| env: | ||
| DOCKER_METADATA_ANNOTATIONS_LEVELS: manifest,index | ||
| with: | ||
| images: ${{ env.NAMESPACE }}/${{ matrix.image }} | ||
| tags: type=sha,format=long | ||
| labels: | | ||
| org.opencontainers.image.title=${{ matrix.image }} | ||
| org.opencontainers.image.description=${{ matrix.description }} | ||
| annotations: | | ||
| org.opencontainers.image.title=${{ matrix.image }} | ||
| org.opencontainers.image.description=${{ matrix.description }} | ||
|
|
||
| - name: Build${{ needs.plan.outputs.publish == 'true' && ' and push' || '' }} | ||
| id: build | ||
| if: steps.existing.outputs.exists != 'true' | ||
| uses: docker/build-push-action@c3c9e263c25d99ce0380d002d59b67737d91b0dc # v7.4.0 | ||
| with: | ||
| context: . | ||
| file: ${{ matrix.dockerfile }} | ||
| target: ${{ matrix.target }} | ||
| platforms: linux/amd64 | ||
| push: ${{ needs.plan.outputs.publish == 'true' }} | ||
| # Without a push, keep the result in the builder only: nothing is | ||
| # loaded into the runner's Docker engine, which would need a second | ||
| # copy of the multi-gigabyte sandbox image on disk. | ||
| outputs: ${{ needs.plan.outputs.publish != 'true' && 'type=cacheonly' || '' }} | ||
| tags: ${{ steps.meta.outputs.tags }} | ||
| labels: ${{ steps.meta.outputs.labels }} | ||
| annotations: ${{ steps.meta.outputs.annotations }} | ||
| # A registry cache next to each image, written only from main. | ||
| # mode=max keeps the intermediate stages (the KVM runner's package | ||
| # stage above all), which the GitHub Actions cache could not hold: | ||
| # its 10 GB repository limit would evict them between pushes. | ||
| cache-from: type=registry,ref=${{ env.NAMESPACE }}/${{ matrix.image }}:buildcache | ||
| cache-to: ${{ needs.plan.outputs.publish == 'true' && format('type=registry,ref={0}/{1}:buildcache,mode=max,ignore-error=true', env.NAMESPACE, matrix.image) || '' }} | ||
|
Comment on lines
+217
to
+218
There was a problem hiding this comment.
Because main runs now use one concurrency group per SHA, builds for different commits can export concurrently to this same per-image Useful? React with 👍 / 👎.
Collaborator
Author
There was a problem hiding this comment. Not changing this one. The outcome is only a cache miss: correctness and tags are unaffected, because every image is still built from its own commit. It needs two image-input merges whose KVM builds overlap (about 9 minutes), with the older finishing last. Even then, both cache manifests usually share the expensive package-stage layers, since that stage is keyed on |
||
|
|
||
| - name: Summarize | ||
| if: always() | ||
| env: | ||
| IMAGE_NAME: ${{ matrix.image }} | ||
| PUSHED: ${{ needs.plan.outputs.publish == 'true' && steps.build.outcome == 'success' }} | ||
| EXISTED: ${{ steps.existing.outputs.exists == 'true' }} | ||
| DIGEST: ${{ steps.build.outputs.digest }} | ||
| run: | | ||
| lowest=$(sort -n "$RUNNER_TEMP/disk-avail-mib.log" 2>/dev/null | head -n1) | ||
| { | ||
| echo "### $IMAGE_NAME" | ||
| if [ "$EXISTED" = true ]; then | ||
| echo "\`$NAMESPACE/$IMAGE_NAME:sha-$GITHUB_SHA\` was already published; left unchanged." | ||
| echo | ||
| fi | ||
| if [ "$PUSHED" = true ]; then | ||
| echo "Pushed \`$NAMESPACE/$IMAGE_NAME:sha-$GITHUB_SHA\`, pinnable as \`$NAMESPACE/$IMAGE_NAME@$DIGEST\`." | ||
| echo | ||
| fi | ||
| echo "Lowest free space on / while building: ${lowest:-unknown} MiB" | ||
| echo '```' | ||
| df -h / | ||
| docker buildx du 2>/dev/null | tail -n1 || true | ||
| echo '```' | ||
| } | tee -a "$GITHUB_STEP_SUMMARY" | ||
|
|
||
| promote: | ||
| name: Move main tags | ||
| needs: [plan, build] | ||
| if: needs.plan.outputs.publish == 'true' | ||
| runs-on: ubuntu-24.04 | ||
| timeout-minutes: 15 | ||
| # Promotions run one at a time. Each resolves its target when it runs | ||
| # instead of trusting its own commit, so a stale re-run, or a queued | ||
| # promotion that GitHub replaces, cannot leave `main` behind or move it | ||
| # backwards. | ||
| concurrency: | ||
| group: images-promote | ||
| cancel-in-progress: false | ||
| permissions: | ||
| contents: read | ||
| packages: write | ||
| steps: | ||
| - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6 | ||
| with: | ||
| persist-credentials: false | ||
| sparse-checkout: .github/scripts | ||
|
|
||
| - uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0 | ||
| with: | ||
| registry: ghcr.io | ||
| username: ${{ github.actor }} | ||
| password: ${{ github.token }} | ||
|
|
||
| - name: Point main at the newest complete commit | ||
| # Target: the newest of the last 100 commits on main for which every | ||
| # image has a `sha-` tag. No image's `main` moves to an older commit | ||
| # than the one it names, and a registry that cannot answer fails the | ||
| # promotion instead of being read as a missing tag, so an error can | ||
| # delay a promotion but never reverse one. A registry cannot move | ||
| # several repositories' tags in one transaction, so a failure part-way | ||
| # leaves some `main` tags behind; the next promotion, or a re-run of | ||
| # this job, finishes the move. Deployments should pin `sha-` tags | ||
| # rather than `main`. | ||
| env: | ||
| GH_TOKEN: ${{ github.token }} | ||
| IMAGES: ${{ needs.plan.outputs.images }} | ||
| run: | | ||
| set -euo pipefail | ||
|
|
||
| # Prints the revision a tag records, or nothing if the tag does not | ||
| # exist; exits the step if the registry cannot answer. | ||
| revision_of() { | ||
| local revision status=0 | ||
| revision=$(.github/scripts/image-revision.sh "$1") || status=$? | ||
| case $status in | ||
| 0) echo "$revision" ;; | ||
| 3) ;; | ||
| *) echo "::error::Could not read $1 from the registry." >&2; exit 1 ;; | ||
| esac | ||
| } | ||
|
|
||
| complete() { | ||
| local image revision | ||
| for image in "${images[@]}"; do | ||
| revision=$(revision_of "$NAMESPACE/$image:sha-$1") || exit 1 | ||
| [ "$revision" = "$1" ] || return 1 | ||
| done | ||
| } | ||
|
|
||
| retag() { | ||
| local attempt | ||
| for attempt in 1 2 3; do | ||
| docker buildx imagetools create --tag "$1:main" "$1:sha-$2" && return 0 | ||
| sleep $(( attempt * 10 )) | ||
| done | ||
| return 1 | ||
| } | ||
|
|
||
| mapfile -t images < <(jq -r '.[].image' <<<"$IMAGES") | ||
| mapfile -t commits < <(gh api "repos/$GITHUB_REPOSITORY/commits?sha=main&per_page=100" --jq '.[].sha') | ||
| declare -A position | ||
| for i in "${!commits[@]}"; do position[${commits[$i]}]=$i; done | ||
|
|
||
| target='' | ||
| for sha in "${commits[@]}"; do | ||
| if complete "$sha"; then | ||
| target=$sha | ||
| break | ||
| fi | ||
| done | ||
| if [ -z "$target" ]; then | ||
| echo "::error::None of the last ${#commits[@]} commits on main has all images." | ||
| exit 1 | ||
| fi | ||
|
|
||
| for image in "${images[@]}"; do | ||
| ref="$NAMESPACE/$image" | ||
| current=$(revision_of "$ref:main") || exit 1 | ||
| if [ "$current" = "$target" ]; then | ||
| echo "$ref:main already at sha-$target" | tee -a "$GITHUB_STEP_SUMMARY" | ||
| continue | ||
| fi | ||
| if [ -n "$current" ] && [ -n "${position[$current]:-}" ] && | ||
| [ "${position[$current]}" -lt "${position[$target]}" ]; then | ||
| echo "::warning::$ref:main names $current, newer than $target; leaving it." | ||
| continue | ||
| fi | ||
| retag "$ref" "$target" | ||
| echo "$ref:main -> sha-$target" | tee -a "$GITHUB_STEP_SUMMARY" | ||
| done | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When a previously successful workflow is rerun for the same SHA, each successful matrix job immediately overwrites its existing
sha-<commit>tag; if another matrix job then fails, its old tag remains, leaving the recommended shared SHA deployment with images from two different builds. The latercomplete()check cannot detect this because both old and new images carry the same revision label, so a subsequent promotion can also copy this mixed set tomain; publish run-scoped tags first or skip existing SHA tags so a failed rerun cannot partially replace the set.Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agreed. Fixed in 0dcb68a by making
sha-tags write-once. In publish mode, a new step reads<image>:sha-<commit>first. If it exists and records this revision, the job skips the build and push and says so in the summary. If it records another revision, the job fails. So a re-run, or a dispatch on the same commit, only builds the images whose tag is missing, and a set can never mix two builds. Simulated against a fake registry: exists, skip; missing, build; wrong revision, fail. The README now says the tags are written once.