Skip to content

ci: rework the delivery pipeline around the build matrix and source packages - #110

Open
Christopher Obbard (obbardc) wants to merge 41 commits into
mainfrom
wip/obbardc/delivery-matrix
Open

ci: rework the delivery pipeline around the build matrix and source packages#110
Christopher Obbard (obbardc) wants to merge 41 commits into
mainfrom
wip/obbardc/delivery-matrix

Conversation

@obbardc

@obbardc Christopher Obbard (obbardc) commented Sep 15, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

The kernel delivery pipeline reworked end to end: the build matrix, the workflows that consume it, how a package is built, and how a release reaches the archive.

Best reviewed commit by commit. The series is ordered so every commit is green, and each message explains why that step exists.

Based on #109 - review or merge that first.

Fixes #97
Fixes #98

This was referenced Sep 15, 2026
Closed
Draft
@obbardc Christopher Obbard (obbardc) changed the title ci: make one matrix entry one generated package ci: rework the delivery pipeline around the build matrix and source packages Sep 15, 2026
@obbardc
Christopher Obbard (obbardc) force-pushed the wip/obbardc/delivery-matrix branch 2 times, most recently from eac31c9 to 5d6f6b4 Compare September 15, 2026 04:14
@obbardc
ci: hand Debusine the source package prepare built
a2b82f4 
The build job downloaded the prepared tree, unpacked it, and had
debusine-action's generate-source-package assemble an orig tarball from it
with tar czf -- a different tarball on every run. It now downloads the
.changes set the prepare job built, reads the version from the .dsc, and
gives lib/build that .dsc: the source package Debusine builds is the one
prepare-kernel-source wrote and reported, with the reproducible orig.

DEBUSINE_ASSEMBLE_ORIG in debusine-action stays for its other callers;
nothing here uses it any more.

Signed-off-by: Christopher Obbard <chris.obbard@oss.qualcomm.com>
@obbardc
ci: build the Ubuntu package from the source package
eb7c95f 
The build job downloaded the prepared tree and ran build-kernel.sh
--local-source --skip-prepare, which had docker_deb_build.py make an orig
tarball of its own with tar czf before handing sbuild a .dsc. It now
downloads the .changes set prepare built and passes its .dsc to
build-kernel.sh --dsc, which runs sbuild on it in the suite's pkg-builder
image. The Ubuntu family therefore builds the same kind of thing Debusine
builds for the Debian family: the source package the run reported, with the
reproducible orig, and nothing is prepared twice.

Signed-off-by: Christopher Obbard <chris.obbard@oss.qualcomm.com>
@obbardc
ci: stop uploading the prepared tree
bf49e26 
Both build paths now start from the source-package artifact, so the
tarball of the prepared tree has no reader. It was the larger of the two
uploads, and the reason the README explained why an artifact had to be a
tar.gz: the tree carried execute bits that upload-artifact's zip would
strip. A .changes set is plain files, so that note goes with it, and the
pipeline diagrams now show the source package as the handoff.

Signed-off-by: Christopher Obbard <chris.obbard@oss.qualcomm.com>
@obbardc
ci: fail when the archive holds a different orig for this version
d0914ea 
The orig tarball is now a function of the kernel commit, so the archive can
hold a different one for the same upstream version only if something about
how it was written differs -- and that is what this check is for. The
already-published step, which already reads the target workspace's source
index, now also finds every stanza naming this run's orig and compares its
SHA-256 with the one prepare reported. A mismatch fails the run with both
checksums, before anything is built, rather than surfacing as a file
conflict at promotion or, worse, not at all.

The case that matters is the packaging rebuild: 0qli1 to 0qli2 of one
snapshot, where the archive requires the two revisions to share one file.
The check reads this suite's index only, since apt indexes are per suite;
across suites the reproducibility of the tarball itself is the guarantee.

Signed-off-by: Christopher Obbard <chris.obbard@oss.qualcomm.com>
@obbardc
ci: build the Ubuntu leg in one job
202d9e9 
The prepare job and the build job ran on the same runner label, so the
only thing the artifact between them did was cross a job boundary that
had no other reason to exist: uploaded by one ephemeral runner to be
downloaded by the next. One job now clones, prepares, builds the source
package and builds the binaries from it in place. The action still
uploads the .changes set, as the record of what was built, but nothing
downloads it.

The build job's own image-fetch step goes with it. It repeated the
action's pull-or-build decision for one suite; the action makes that
decision for every suite before prepare-source.sh runs, and leaves the
docker-pkg-build checkout where build-kernel.sh looks for it when it had
to build the image.

Signed-off-by: Christopher Obbard <chris.obbard@oss.qualcomm.com>
@obbardc
ci: publish the source package to S3 beside the binaries
b472673 
The .deb files reach S3 under one run's directory; the source package they
were built from now sits in source/ beneath it, on both paths. The
.changes records every checksum, the .orig.tar.gz is the exact source, and
the .dsc is enough to rebuild the packages anywhere. The Actions artifact
holding the same files expires after a week; this is the copy that stays
with the binaries.

The Debian publish job downloads the artifact for it, since the source
package was built on another runner; the Ubuntu job uploads it from the
workspace it was built in.

Signed-off-by: Christopher Obbard <chris.obbard@oss.qualcomm.com>
@obbardc
ci: test for a published version before comparing the orig
da2c771 
The orig comparison ran first, so a night on which the tracked tag had not
moved failed instead of skipping: the version was already published, and
the orig in the archive predates git archive, so the checksums differ. That
is a run with nothing to do reported as a conflict.

A run that publishes nothing uploads no files, so nothing it built can
conflict with anything. The already-published test therefore comes first
and exits, and the orig comparison guards only a version that is actually
going to be promoted. That leaves it doing the job it was added for: a
packaging rebuild publishing a second Debian revision of one upstream
version, where the archive requires both revisions to name one file.

Signed-off-by: Christopher Obbard <chris.obbard@oss.qualcomm.com>
@obbardc
ci: inline the debusine-build action into its one caller
ce5dbe2 
The action had a single caller and paid for being one. The vars and secrets
contexts do not reach a composite action, so the Debusine host, scope, user
and token were all declared as inputs whose only purpose was to carry values
the calling workflow could already see. Its steps now sit in the build job
and read vars and secrets directly, and six inputs and their plumbing go.

Two more things fall out. The job checked out pkg-linux-qcom solely to
reach the action, so that checkout goes too: the package comes from the
artifact and the tooling from debusine-action, and nothing of this
repository is needed in that job at all. And workspace_url was an action
output that only ever fed a step summary, so it is now a local variable in
the step that writes it.

Signed-off-by: Christopher Obbard <chris.obbard@oss.qualcomm.com>
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Implement suite-based dkms enablement list, driven / configured from matrix. Ubuntu releases hard-coded to always include kgsl DKMS

1 participant

@obbardc