From 218c4f382f379edb393c49d98d97c53980ee5aa6 Mon Sep 17 00:00:00 2001 From: Christian Ehrhardt Date: Thu, 4 Nov 2021 14:36:18 +0100 Subject: [PATCH 01/20] Generalize golang->vendored statements Some golang statements are actually meant to apply to any kind of vendored code, generalize those statements. Signed-off-by: Christian Ehrhardt --- README.md | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index e8d94bdf7d..f0fa73d0f3 100644 --- a/README.md +++ b/README.md @@ -403,13 +403,12 @@ RULE: not a recommended but a common case with golang packages. RULE: - the security team will track CVEs for all golang packages in main RULE: - the security team will provide updates to main for all `golang-*-dev` RULE: packages -RULE: - the security team will provide updates to main for golang applications -RULE: whose non-vendored source code is affected as per normal procedures -RULE: (including e.g., sponsoring/coordinating uploads from teams/upstream -RULE: projects, etc) +RULE: - the security team will provide updates to main for non-vendored +RULE: dependencies as per normal procedures (including e.g., +RULE: sponsoring/coordinating uploads from teams/upstream projects, etc) RULE: - the security team will perform no-change-rebuilds for all packages -RULE: Built-Using `golang-*-dev` packages it has provided, and coordinate -RULE: testing with the owning teams responsible for the rebuilt packages +RULE: listing an CVE-fixed package as Built-Using and coordinate testing +RULE: with the owning teams responsible for the rebuilt packages RULE: - for packages that build using any `golang-*-dev` packages: RULE: - the owning team must state their commitment to test RULE: no-change-rebuilds triggered by a dependent library/compiler and to @@ -542,7 +541,7 @@ RULE: which should be discouraged (except golang, see below) OK: TODO: - no embedded source present TODO: - no static linking -TODO: - does not have odd Built-Using entries +TODO: - does not have unexpected Built-Using entries RULE: Golang RULE: - golang 1.4 packages and earlier could only statically compile their @@ -558,8 +557,8 @@ RULE: libraries during this period, the MIR team decided for 17.10 and later RULE: to allow static builds of golang packages in main, so long as the number RULE: of these packages remains low and they follow the guidelines below: RULE: - golang applications in main are expected: -RULE: 1. to build using `golang-*-dev` packages from the Ubuntu archive -RULE: with `Built-Using` in debian/control. This requirement ensures +RULE: 1. to build golang use `golang-*-dev` packages creating +RULE: `Built-Using` in debian/control. This requirement ensures RULE: that the security team is able to track security issues for all RULE: affected static binary packages RULE: 2. not to build any vendored (i.e. embedded) code in the source From de643611e29bd61e638f2a2df31fec3649a56b2c Mon Sep 17 00:00:00 2001 From: Christian Ehrhardt Date: Thu, 4 Nov 2021 14:37:21 +0100 Subject: [PATCH 02/20] Derive Rust from Go rules Some Golang specific rules apply to rust as well, list go+rust in those cases to make it clear that it applies to both. Signed-off-by: Christian Ehrhardt --- README.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index f0fa73d0f3..766eabf542 100644 --- a/README.md +++ b/README.md @@ -399,8 +399,8 @@ TODO-A: - Team is already subscribed to the package TODO-B: - Team is not yet, but will subscribe to the package before promotion RULE: - Responsibilities implied by static builds promoted to main, which is -RULE: not a recommended but a common case with golang packages. -RULE: - the security team will track CVEs for all golang packages in main +RULE: not a recommended but a common case with golang and rust packages. +RULE: - the security team will track CVEs for all vendored packages in main RULE: - the security team will provide updates to main for all `golang-*-dev` RULE: packages RULE: - the security team will provide updates to main for non-vendored @@ -536,7 +536,7 @@ RULE: maintenance burden. For this reason, static linking in archive builds RULE: is discouraged unless static linking is required for the package in RULE: question to function correctly (e.g. an integrity scanner). RULE: - Does debian/control use `Built-Using`? This may indicate static linking -RULE: which should be discouraged (except golang, see below) +RULE: which should be discouraged (except golang/rust, see below) OK: TODO: - no embedded source present @@ -579,8 +579,8 @@ RULE: adjusting their packaging as necessary, all teams responsible for RULE: golang packages coordinating on transitions and the requesting team RULE: occasionally creating new `golang-*-dev` packages as agreed to in the RULE: MIR bug (upstreaming to Debian whenever possible). -RULE: - As a practical matter, golang source packages in main are not required -RULE: to remove unused embedded code copies. +RULE: - As a practical matter, golang/rust source packages in main are not +RULE required to remove unused embedded code copies. RULE: - If based on the above options it's a statically compiled golang package: RULE: - Does the package use dh-golang (if not, suggest dh-make-golang to RULE: create the package)? From 4a2859cc010baf2a526928a8370163008f255d86 Mon Sep 17 00:00:00 2001 From: Christian Ehrhardt Date: Thu, 4 Nov 2021 14:43:59 +0100 Subject: [PATCH 03/20] Add Rust specific rules v2 I was discussing with mclemencau/mdhudson/slangasek about what we should do looking forward to rust binaries (like firefox, but less single special snowflakes) in main. We excercised a bit based on my current example of mdevctl [1] and out of this we have efforts to bring the rust toolchain to main as well as these efforts to define the MIR rules for those. But the discussion is more wide-reaching as I want some commitment to get the toolchain itself to be considered supported as well as some guidance how to best deal with rust in its current state. At first it seems to be similar to golang, but it is worse. On one hand lib transitions are often quite hard in rust, see [2]. But more than that I was told that often transitions get stuck for quite a long time and are way too broken (this might improve and then we want to make rust less special). Finally while the concept of dynamic libraries [3] exists, it isn't usable as effectively only .so files from the very same build [4] are usable reliably and therefore are like static builds but in multiple files. In the discussions it was suggested: > " > Regarding the overall rust-in-main question: I'm fine with rustc being in > main provided there's resourcing. I think packages /using/ rust should only > be allowed in main via vendoring of their dependencies, because the > maintenance of rust libraries in Debian is bad right now. > ... > When we did the exception for golang, Jamie Strandboge drafted it for the > Security Team and I think there was some consultation with Foundations / > Archive Admins. > ... > ultimately it's the Security Team that bears the maintenance burden of > decisions we make about what to put in main, so I would defer to them and > have them consult whoever they think is relevant. > " In addition the extra burden will also be on the owning team and sustained engineering, so we want to get all their consultation before we switch to this. And while this is shockingly different from how everything else works today it really seems the only doable way *right now*. All the best practise we are used to (lib only once for re-use and less maintenance) does no more apply here until rust and the rust ecosystem evolves further. But while it is shocking at first, compared to all other approaches that came up so far that at least is realizable in a not too far future. And we also have to admit that all golang MIRs of the recent years have eventually went the vendoring-route as well, so that - for now - just seems to be the more applicable and doable way. Updates to v1: - The new rule/template layout has made this much smaller. - From the first iteration of this we realized that many golang rules will just apply to rust the same way - that again made it less noisy - We also have identified that vendored code in general (not just rust) should have an update plan, so we add a rule for that as well. [1]: https://bugs.launchpad.net/ubuntu/+source/mdevctl/+bug/1942394 [2]: https://github.com/dtolnay/semver-trick [3]: https://rust-lang.github.io/rfcs/0404-change-prefer-dynamic.html [4]: https://doc.rust-lang.org/reference/linkage.html Signed-off-by: Christian Ehrhardt --- README.md | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/README.md b/README.md index 766eabf542..ed8b0697a6 100644 --- a/README.md +++ b/README.md @@ -426,16 +426,40 @@ RULE: - the owning team will provide timely, high quality updates for the RULE: security team to sponsor to fix issues in the affected vendored code RULE: - if subsequent uploads add new vendored components or dependencies RULE: these have to be reviewed and agreed by the security team. +RULE: - The rust ecosystem currently isn't yet considered stable enough for +RULE: classic lib dependencies and transitions in main; therefore the +RULE: expectation for those packages is to vendor (and own/test) all +RULE: dependencies (except those provided by the rust runtime itself). +RULE: This implies that all the rules for vendored builds always +RULE: apply to them. In addition +RULE: - Rust builds using librust-*-dev packges will populate the attribute +RULE: `X-Cargo-Built-Using`, right now this is expected not to be present. +RULE: - It is expected rust builds will use dh-cargo so that a later switch +RULE: to proper package dependencies isn't too hard (e.g. it is likely +RULE: that over time some more common/stable libs shall come from proper +RULE: archive packages). +RULE: - All vendored dependencies (no matter what language) shall have a +RULE: way to be refreshed (recommended is a documentation README.source and +RULE: scripts in debian/ TODO-A: - This does not use static builds TODO-B: - The team TBD is aware of the implications by a static build and TODO-B: commits to test no-change-rebuilds and to fix any issues found for the TODO-B: lifetime of the release (including ESM) + TODO-A: - This does not use vendored code TODO-B: - The team TBD is aware of the implications of vendored code and (as TODO-B: alerted by the security team) commits to provide updates to the security TODO-B: team for any affected vendored code for the lifetime of the release TODO-B: (including ESM). +TODO-A: - This package is not rust based +TODO-B: - This package is rust based and vendors all non language-runtime +TODO-B: dependencies + +TODO-A: - This package is not rust based +TODO-B: - This package is rust based, refreshing vendored code is outlined +TODO-B: in + RULE: - if there has been an archive test rebuild that has occurred more recently RULE: than the last upload, the package must have rebuilt successfully TODO-A: - The package has been built in the archive more recently than the last @@ -604,6 +628,15 @@ TODO-A: - golang: shared builds TODO-B: - golang: static builds are used, the team confirmed their commitment TODO-B: to the additional responsibilities implied by static builds. +TODO-A: - not a rust package, no extra constraints to consider in that regard +TODO-B: - rust package that has all dependencies vendored +TODO-B: - rust package does not have X-Cargo-Built-Using (after build) +TODO-B: - rust package does use dh_cargo (dh ... --buildsystem cargo) + +TODO-A: - Includes vendored code, the package has documented how to refresh this +TODO-A: code at +TODO-B: - Does not include vendored code + TODO-A: Problems: TODO-A: - TBD TODO-B: Problems: None From e597b7f60a5a51c94461e0e8aeda43970de40745 Mon Sep 17 00:00:00 2001 From: Christian Ehrhardt Date: Mon, 8 Nov 2021 16:24:46 +0100 Subject: [PATCH 04/20] suggestion to clarify vendoring MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Lukas Märdian --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index ed8b0697a6..ca38e71912 100644 --- a/README.md +++ b/README.md @@ -400,7 +400,7 @@ TODO-B: - Team is not yet, but will subscribe to the package before promotion RULE: - Responsibilities implied by static builds promoted to main, which is RULE: not a recommended but a common case with golang and rust packages. -RULE: - the security team will track CVEs for all vendored packages in main +RULE: - the security team will track CVEs for all vendored/embedded sources in main RULE: - the security team will provide updates to main for all `golang-*-dev` RULE: packages RULE: - the security team will provide updates to main for non-vendored From 80ebc94e945f4f6473d140255ae1de11e0ed9caf Mon Sep 17 00:00:00 2001 From: Christian Ehrhardt Date: Tue, 9 Nov 2021 12:51:58 +0100 Subject: [PATCH 05/20] fix golang reference link Signed-off-by: Christian Ehrhardt --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index ca38e71912..5c4ce00f8f 100644 --- a/README.md +++ b/README.md @@ -612,7 +612,7 @@ RULE: - Does debian/control use `Built-Using: ${misc:Built-Using}` for each RULE: non'-dev' binary package (importantly, golang-*-dev packages only RULE: ship source files so don't need Built-Using)? RULE: - Does the package follow -RULE: [[http://pkg-go.alioth.debian.org/packaging.html|Debian Go packaging]] +RULE: [[https://go-team.pages.debian.net/packaging.html|Debian Go packaging]] RULE: guidelines? RULE: - When it is infeasible to comply with this policy, the justification, RULE: discussion and approval should all be clearly represented in the bug. From 76fa37e44bf69fa43e641e9825382668e18127a0 Mon Sep 17 00:00:00 2001 From: Christian Ehrhardt Date: Tue, 9 Nov 2021 15:00:32 +0100 Subject: [PATCH 06/20] Extend team commitment for backports of vendored code Signed-off-by: Christian Ehrhardt --- README.md | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 5c4ce00f8f..e92f2e3b50 100644 --- a/README.md +++ b/README.md @@ -426,6 +426,18 @@ RULE: - the owning team will provide timely, high quality updates for the RULE: security team to sponsor to fix issues in the affected vendored code RULE: - if subsequent uploads add new vendored components or dependencies RULE: these have to be reviewed and agreed by the security team. +RULE: - Such updates in the project might be trivial, but imply that a +RULE: dependency for e.g. a CVE fix will be moved to a new major version. +RULE: Being vendored that does gladly at least not imply incompatibility +RULE: issues with other packages or the SRU policy. But it might happen +RULE: that this triggers either: +RULE: a) The need to adapt the current version of the main package and/or +RULE: other vendored dependencies to work with the new dependency +RULE: b) The need to backport the fix in the dependency as the main +RULE: package will functionally only work well with the older version +RULE: c) The need to backport the fix in the dependency, as it would imply +RULE: requiring a newer toolchain to be buildable that isn't available +RULE: in the target release. RULE: - The rust ecosystem currently isn't yet considered stable enough for RULE: classic lib dependencies and transitions in main; therefore the RULE: expectation for those packages is to vendor (and own/test) all @@ -448,9 +460,9 @@ TODO-B: lifetime of the release (including ESM) TODO-A: - This does not use vendored code TODO-B: - The team TBD is aware of the implications of vendored code and (as -TODO-B: alerted by the security team) commits to provide updates to the security -TODO-B: team for any affected vendored code for the lifetime of the release -TODO-B: (including ESM). +TODO-B: alerted by the security team) commits to provide updates and backports +TODO-B: to the security team for any affected vendored code for the lifetime +TODO-B: of the release (including ESM). TODO-A: - This package is not rust based TODO-B: - This package is rust based and vendors all non language-runtime From 7e62cc0ce6dfb5f8e41c1426bd39e28b684aa60e Mon Sep 17 00:00:00 2001 From: Christian Ehrhardt Date: Tue, 16 Nov 2021 14:34:09 +0100 Subject: [PATCH 07/20] avoid calling non classic deps not proper Signed-off-by: Christian Ehrhardt --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index e92f2e3b50..9a8bcecee3 100644 --- a/README.md +++ b/README.md @@ -447,9 +447,9 @@ RULE: apply to them. In addition RULE: - Rust builds using librust-*-dev packges will populate the attribute RULE: `X-Cargo-Built-Using`, right now this is expected not to be present. RULE: - It is expected rust builds will use dh-cargo so that a later switch -RULE: to proper package dependencies isn't too hard (e.g. it is likely -RULE: that over time some more common/stable libs shall come from proper -RULE: archive packages). +RULE: to non vendored dependencies isn't too complex (e.g. it is likely +RULE: that over time more common libs shall become stable and then archive +RULE: packages will be used to build). RULE: - All vendored dependencies (no matter what language) shall have a RULE: way to be refreshed (recommended is a documentation README.source and RULE: scripts in debian/ From cfd2672a71d56588ec9951e18f37e2a378399719 Mon Sep 17 00:00:00 2001 From: Christian Ehrhardt Date: Tue, 16 Nov 2021 14:34:40 +0100 Subject: [PATCH 08/20] update refreshing of code This is less duplication (and was not rust specific) and makes it more clear that if there are common ways to refresh vendored sources in some languages using those is appreciated. Signed-off-by: Christian Ehrhardt --- README.md | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 9a8bcecee3..547d87781d 100644 --- a/README.md +++ b/README.md @@ -451,8 +451,7 @@ RULE: to non vendored dependencies isn't too complex (e.g. it is likely RULE: that over time more common libs shall become stable and then archive RULE: packages will be used to build). RULE: - All vendored dependencies (no matter what language) shall have a -RULE: way to be refreshed (recommended is a documentation README.source and -RULE: scripts in debian/ +RULE: way to be refreshed TODO-A: - This does not use static builds TODO-B: - The team TBD is aware of the implications by a static build and TODO-B: commits to test no-change-rebuilds and to fix any issues found for the @@ -464,14 +463,18 @@ TODO-B: alerted by the security team) commits to provide updates and backports TODO-B: to the security team for any affected vendored code for the lifetime TODO-B: of the release (including ESM). +TODO-A: - This does not use vendored code +TODO-B: - This package uses vendored go code tracked in go.mod as shiped in the +TODO-B: package, refreshing that code works via `go mod ...`. +TODO-C: - This package uses vendored rust code tracked in Cargo.lock as shipped, +TODO-C: in the package, refreshing that code works via `cargo update ...`. +TODO-D: - This package uses vendored code, refreshing that code is outlined +TODO-D: in debian/README.source + TODO-A: - This package is not rust based TODO-B: - This package is rust based and vendors all non language-runtime TODO-B: dependencies -TODO-A: - This package is not rust based -TODO-B: - This package is rust based, refreshing vendored code is outlined -TODO-B: in - RULE: - if there has been an archive test rebuild that has occurred more recently RULE: than the last upload, the package must have rebuilt successfully TODO-A: - The package has been built in the archive more recently than the last From 6c4a3d9393a95286de1c569a1cfb91c8a47b6228 Mon Sep 17 00:00:00 2001 From: Christian Ehrhardt Date: Tue, 14 Dec 2021 11:06:52 +0100 Subject: [PATCH 09/20] Clarify preliminary state and usage of Built-Using Signed-off-by: Christian Ehrhardt --- README.md | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 547d87781d..c500fc8cc7 100644 --- a/README.md +++ b/README.md @@ -443,9 +443,13 @@ RULE: classic lib dependencies and transitions in main; therefore the RULE: expectation for those packages is to vendor (and own/test) all RULE: dependencies (except those provided by the rust runtime itself). RULE: This implies that all the rules for vendored builds always -RULE: apply to them. In addition +RULE: apply to them. In addition: +RULE: - The rules and checks for rust based packges are preliminary and might +RULE: change while finalizing the rust toolchain in main and while +RULE: processing the first few rust based packages. RULE: - Rust builds using librust-*-dev packges will populate the attribute -RULE: `X-Cargo-Built-Using`, right now this is expected not to be present. +RULE: `Built-Using`, right now this is expected not to be present as +RULE: - for now - all rust dependencies shall be vendored. RULE: - It is expected rust builds will use dh-cargo so that a later switch RULE: to non vendored dependencies isn't too complex (e.g. it is likely RULE: that over time more common libs shall become stable and then archive @@ -576,6 +580,10 @@ RULE: is discouraged unless static linking is required for the package in RULE: question to function correctly (e.g. an integrity scanner). RULE: - Does debian/control use `Built-Using`? This may indicate static linking RULE: which should be discouraged (except golang/rust, see below) +RULE: - Especially for rust - where toolchain and dh tools are new and still +RULE: changing a lot - please double check to not only have no Built-Using +RULE: entry, but also that there is no code of librust-*-dev used that just +RULE: happened to be missed while generating Built-Using. OK: TODO: - no embedded source present @@ -644,9 +652,11 @@ TODO-B: - golang: static builds are used, the team confirmed their commitment TODO-B: to the additional responsibilities implied by static builds. TODO-A: - not a rust package, no extra constraints to consider in that regard -TODO-B: - rust package that has all dependencies vendored -TODO-B: - rust package does not have X-Cargo-Built-Using (after build) -TODO-B: - rust package does use dh_cargo (dh ... --buildsystem cargo) +TODO-B: - Rust package that has all dependencies vendored. It does neither +TODO-B: have *Built-Using (after build). Nor does the biuld log indicate +TODO-B: built-in sources that are missed to be reported as Built-Using. + +TODO: - rust package using dh_cargo (dh ... --buildsystem cargo) TODO-A: - Includes vendored code, the package has documented how to refresh this TODO-A: code at From c689b581c2c400b95466409504a976f854efac61 Mon Sep 17 00:00:00 2001 From: Christian Ehrhardt Date: Fri, 1 Jul 2022 09:13:40 +0200 Subject: [PATCH 10/20] rust: the toolchain is in main now Signed-off-by: Christian Ehrhardt --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index c500fc8cc7..a591114125 100644 --- a/README.md +++ b/README.md @@ -444,8 +444,8 @@ RULE: expectation for those packages is to vendor (and own/test) all RULE: dependencies (except those provided by the rust runtime itself). RULE: This implies that all the rules for vendored builds always RULE: apply to them. In addition: -RULE: - The rules and checks for rust based packges are preliminary and might -RULE: change while finalizing the rust toolchain in main and while +RULE: - The rules and checks for rust based packages are preliminary and might +RULE: change over time as the ecosytem matures and while RULE: processing the first few rust based packages. RULE: - Rust builds using librust-*-dev packges will populate the attribute RULE: `Built-Using`, right now this is expected not to be present as From 5330b791f4673b76b88d722d94bb0733c797b635 Mon Sep 17 00:00:00 2001 From: Christian Ehrhardt Date: Fri, 1 Jul 2022 09:14:40 +0200 Subject: [PATCH 11/20] rust: built-using no more used this way Following Debians dh tooling built-using no more tracks all dependencies anway. Adapt the wording to reflect that. Furthermore the same applies to the rule statments which need to reflect the go-mod/cargo.lock that we have come to in discussions for other sections of the document. Signed-off-by: Christian Ehrhardt --- README.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index a591114125..b02d342d73 100644 --- a/README.md +++ b/README.md @@ -447,9 +447,6 @@ RULE: apply to them. In addition: RULE: - The rules and checks for rust based packages are preliminary and might RULE: change over time as the ecosytem matures and while RULE: processing the first few rust based packages. -RULE: - Rust builds using librust-*-dev packges will populate the attribute -RULE: `Built-Using`, right now this is expected not to be present as -RULE: - for now - all rust dependencies shall be vendored. RULE: - It is expected rust builds will use dh-cargo so that a later switch RULE: to non vendored dependencies isn't too complex (e.g. it is likely RULE: that over time more common libs shall become stable and then archive @@ -578,12 +575,15 @@ RULE: with the updated libraries to receive the fix, which increases the RULE: maintenance burden. For this reason, static linking in archive builds RULE: is discouraged unless static linking is required for the package in RULE: question to function correctly (e.g. an integrity scanner). -RULE: - Does debian/control use `Built-Using`? This may indicate static linking +RULE: - If debian/control uses `Built-Using` it may indicate static linking RULE: which should be discouraged (except golang/rust, see below) -RULE: - Especially for rust - where toolchain and dh tools are new and still -RULE: changing a lot - please double check to not only have no Built-Using -RULE: entry, but also that there is no code of librust-*-dev used that just -RULE: happened to be missed while generating Built-Using. +RULE: - Rust - toolchain and dh tools are still changing a lot. Currently it +RULE: is expected to only list the rust toolchain in `Built-Using`. +RULE: the remaining (currently vendored) dependencies shall be tracked +RULE: in a cargo.lock file +RULE: - Go - here `Built-Using` is expected to only contain the go +RULE: toolchain used to build it. Additional dependencies shall be +RULE: tracked in a go.mod file OK: TODO: - no embedded source present From d27df7cba9675b6260b34871e9431eeeba644c5e Mon Sep 17 00:00:00 2001 From: Christian Ehrhardt Date: Fri, 1 Jul 2022 09:16:39 +0200 Subject: [PATCH 12/20] rust/go: flexibility for refreshing vendored dependencies Until the ecosystems settled and all projects adapted. Ass well as the dh_* toolchain tolerates and produces reliable go.mod/cargo.lock tracking we have to be slightly more lenient on how exactly vendored dependencies are updated. That shall now - for all types - be outlined in README.source. That can for some projects still be `go mod ...` or `cargo update ...` but in other cases might need other steps and README.source provides the flexibility to do so. Signed-off-by: Christian Ehrhardt --- README.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index b02d342d73..40f450cb62 100644 --- a/README.md +++ b/README.md @@ -466,9 +466,10 @@ TODO-B: of the release (including ESM). TODO-A: - This does not use vendored code TODO-B: - This package uses vendored go code tracked in go.mod as shiped in the -TODO-B: package, refreshing that code works via `go mod ...`. +TODO-B: package, refreshing that code is outlined in debian/README.source TODO-C: - This package uses vendored rust code tracked in Cargo.lock as shipped, -TODO-C: in the package, refreshing that code works via `cargo update ...`. +TODO-C: in the package, refreshing that code is outlined in +TODO-C: debian/README.source TODO-D: - This package uses vendored code, refreshing that code is outlined TODO-D: in debian/README.source From e4e48e09a0b417a86a2bf391405f43a067ccd38e Mon Sep 17 00:00:00 2001 From: Christian Ehrhardt Date: Fri, 1 Jul 2022 09:26:25 +0200 Subject: [PATCH 13/20] rust: acknowledge that atm dh-cargo can not provide cargo.lock Signed-off-by: Christian Ehrhardt --- README.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/README.md b/README.md index 40f450cb62..2e8b8c1b98 100644 --- a/README.md +++ b/README.md @@ -451,6 +451,10 @@ RULE: - It is expected rust builds will use dh-cargo so that a later switch RULE: to non vendored dependencies isn't too complex (e.g. it is likely RULE: that over time more common libs shall become stable and then archive RULE: packages will be used to build). +RULE: - the rust tooling e.g. dh-cargo can not yet automatically provide +RULE: all that we require - for example Cargo.lock - until it does the +RULE: package build shall be adapted in a way to provide that without +RULE: dh* tooling support. RULE: - All vendored dependencies (no matter what language) shall have a RULE: way to be refreshed TODO-A: - This does not use static builds From 665ca0b3f09fccd38cc6909953ead33536668e8d Mon Sep 17 00:00:00 2001 From: Christian Ehrhardt Date: Fri, 1 Jul 2022 09:28:18 +0200 Subject: [PATCH 14/20] rust: define expected placement of Cargo.lock Signed-off-by: Christian Ehrhardt --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 2e8b8c1b98..827ed284fd 100644 --- a/README.md +++ b/README.md @@ -472,8 +472,8 @@ TODO-A: - This does not use vendored code TODO-B: - This package uses vendored go code tracked in go.mod as shiped in the TODO-B: package, refreshing that code is outlined in debian/README.source TODO-C: - This package uses vendored rust code tracked in Cargo.lock as shipped, -TODO-C: in the package, refreshing that code is outlined in -TODO-C: debian/README.source +TODO-C: in the package (at /usr/share/doc//Cargo.lock - might be +TODO-C: compressed), refreshing that code is outlined in debian/README.source TODO-D: - This package uses vendored code, refreshing that code is outlined TODO-D: in debian/README.source From d7966f6fe1722d72f42062f6dd82384264205a73 Mon Sep 17 00:00:00 2001 From: Christian Ehrhardt Date: Tue, 5 Jul 2022 14:19:01 +0200 Subject: [PATCH 15/20] Update README.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Lukas Märdian --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 827ed284fd..d11c48d1a1 100644 --- a/README.md +++ b/README.md @@ -658,7 +658,7 @@ TODO-B: to the additional responsibilities implied by static builds. TODO-A: - not a rust package, no extra constraints to consider in that regard TODO-B: - Rust package that has all dependencies vendored. It does neither -TODO-B: have *Built-Using (after build). Nor does the biuld log indicate +TODO-B: have *Built-Using (after build). Nor does the build log indicate TODO-B: built-in sources that are missed to be reported as Built-Using. TODO: - rust package using dh_cargo (dh ... --buildsystem cargo) From bcc6aac4213c7072a52db580c6432a369730063b Mon Sep 17 00:00:00 2001 From: Christian Ehrhardt Date: Tue, 5 Jul 2022 14:20:21 +0200 Subject: [PATCH 16/20] Update README.md - golang wording MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Lukas Märdian --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index d11c48d1a1..938d096515 100644 --- a/README.md +++ b/README.md @@ -609,8 +609,8 @@ RULE: libraries during this period, the MIR team decided for 17.10 and later RULE: to allow static builds of golang packages in main, so long as the number RULE: of these packages remains low and they follow the guidelines below: RULE: - golang applications in main are expected: -RULE: 1. to build golang use `golang-*-dev` packages creating -RULE: `Built-Using` in debian/control. This requirement ensures +RULE: 1. to build using `golang-*-dev` packages from the Ubuntu archive +RULE: creating `Built-Using` in debian/control. This requirement ensures RULE: that the security team is able to track security issues for all RULE: affected static binary packages RULE: 2. not to build any vendored (i.e. embedded) code in the source From 7fb99d44910e64e3c3b6b79381f2f2c1858cc20f Mon Sep 17 00:00:00 2001 From: Christian Ehrhardt Date: Tue, 5 Jul 2022 17:56:24 +0200 Subject: [PATCH 17/20] go.mod -> go.sum as suggested by didrocks Signed-off-by: Christian Ehrhardt --- README.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 938d096515..bfb6b01fe7 100644 --- a/README.md +++ b/README.md @@ -469,7 +469,7 @@ TODO-B: to the security team for any affected vendored code for the lifetime TODO-B: of the release (including ESM). TODO-A: - This does not use vendored code -TODO-B: - This package uses vendored go code tracked in go.mod as shiped in the +TODO-B: - This package uses vendored go code tracked in go.sum as shiped in the TODO-B: package, refreshing that code is outlined in debian/README.source TODO-C: - This package uses vendored rust code tracked in Cargo.lock as shipped, TODO-C: in the package (at /usr/share/doc//Cargo.lock - might be @@ -588,7 +588,10 @@ RULE: the remaining (currently vendored) dependencies shall be tracked RULE: in a cargo.lock file RULE: - Go - here `Built-Using` is expected to only contain the go RULE: toolchain used to build it. Additional dependencies shall be -RULE: tracked in a go.mod file +RULE: tracked in a go.sum file (go.mod are direct dependencies, go.sum +RULE: covers checksum content for direct and indirect dependencies. This +RULE: should be present for reproducible builds already which involve +RULE: having a go.sum. OK: TODO: - no embedded source present From fb00c391a9142940fc2c692f1caf46157069666a Mon Sep 17 00:00:00 2001 From: Christian Ehrhardt Date: Tue, 12 Jul 2022 10:01:27 +0200 Subject: [PATCH 18/20] Fix RULE: format Signed-off-by: Christian Ehrhardt --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index bfb6b01fe7..27b61b5f2f 100644 --- a/README.md +++ b/README.md @@ -342,7 +342,7 @@ TODO-A: - The package will be installed by default, but does not ask debconf TODO-A: questions higher than medium TODO-B: - The package will not be installed by default -RULE - The source packaging (in debian/) should be reasonably easy to +RULE: - The source packaging (in debian/) should be reasonably easy to RULE: understand and maintain. TODO-A: - Packaging and build is easy, link to d/rules TBD TODO-B: - Packaging is complex, but that is ok because TBD @@ -381,7 +381,7 @@ RULE: to its complexity: RULE: - All packages must have a designated "owning" team, regardless of RULE: complexity, which is set as a package bug contact. This is not a RULE: requirement for the MIR team ACK, but for the package to be promoted -RULE by an archive admin. Still, it is strongly suggested to subscribe, +RULE: by an archive admin. Still, it is strongly suggested to subscribe, RULE: as the owning team will get a preview of the to-be-expected incoming RULE: bugs later on. RULE: - Simple packages (e.g. language bindings, simple Perl modules, small @@ -635,7 +635,7 @@ RULE: golang packages coordinating on transitions and the requesting team RULE: occasionally creating new `golang-*-dev` packages as agreed to in the RULE: MIR bug (upstreaming to Debian whenever possible). RULE: - As a practical matter, golang/rust source packages in main are not -RULE required to remove unused embedded code copies. +RULE: required to remove unused embedded code copies. RULE: - If based on the above options it's a statically compiled golang package: RULE: - Does the package use dh-golang (if not, suggest dh-make-golang to RULE: create the package)? From 259a852cb6374d0b85f43136d19ce09dc572e2c7 Mon Sep 17 00:00:00 2001 From: Christian Ehrhardt Date: Wed, 13 Jul 2022 11:46:12 +0200 Subject: [PATCH 19/20] Rust/Go: update rules to insist on tracking files and consider Static-Built-Using As discussed in #1 this will add awareness for Static-Built-Using but even more important make the requirement for a generated cargo.lock/go.sum a requirement for now. Signed-off-by: Christian Ehrhardt --- README.md | 21 ++++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 27b61b5f2f..9fd7998e35 100644 --- a/README.md +++ b/README.md @@ -452,9 +452,9 @@ RULE: to non vendored dependencies isn't too complex (e.g. it is likely RULE: that over time more common libs shall become stable and then archive RULE: packages will be used to build). RULE: - the rust tooling e.g. dh-cargo can not yet automatically provide -RULE: all that we require - for example Cargo.lock - until it does the -RULE: package build shall be adapted in a way to provide that without -RULE: dh* tooling support. +RULE: all that we require - for example Cargo.lock - but that will be +RULE: required to track dependencies. Until available in e.g. dh-cargo +RULE: a package can not yet get fully compliant. RULE: - All vendored dependencies (no matter what language) shall have a RULE: way to be refreshed TODO-A: - This does not use static builds @@ -580,18 +580,29 @@ RULE: with the updated libraries to receive the fix, which increases the RULE: maintenance burden. For this reason, static linking in archive builds RULE: is discouraged unless static linking is required for the package in RULE: question to function correctly (e.g. an integrity scanner). -RULE: - If debian/control uses `Built-Using` it may indicate static linking +RULE: - If debian/control uses `Built-Using` or `Static-Built-Using:` it may +RULE: indicate static linking RULE: which should be discouraged (except golang/rust, see below) RULE: - Rust - toolchain and dh tools are still changing a lot. Currently it RULE: is expected to only list the rust toolchain in `Built-Using`. RULE: the remaining (currently vendored) dependencies shall be tracked RULE: in a cargo.lock file +RULE: Right now that tooling to get a cargo.lock that will include internal +RULE: packaged (if any) and used vendored dependencies isn't in place. +RULE: So right now due to the lack of that for example in dh-cargo one +RULE: can not yet get a package fully compliant. RULE: - Go - here `Built-Using` is expected to only contain the go -RULE: toolchain used to build it. Additional dependencies shall be +RULE: toolchain used to build it. Additional packaged dependencies +RULE: will be tracked in `Static-Built-Using:` automatically. +RULE: The superset of packaged and vendored (if used) dependencies shall be RULE: tracked in a go.sum file (go.mod are direct dependencies, go.sum RULE: covers checksum content for direct and indirect dependencies. This RULE: should be present for reproducible builds already which involve RULE: having a go.sum. +RULE: We have let go packages into main before this existed, so we have +RULE: sub-optimal prior-art. But down the road - if vendoring is used - we +RULE: want to switch to require that once the toolchain is ready to +RULE: create it accordingly. OK: TODO: - no embedded source present From 8625876c3fe4a93a1058b44e2f7390931a2a7721 Mon Sep 17 00:00:00 2001 From: Christian Ehrhardt Date: Thu, 14 Jul 2022 07:51:45 +0200 Subject: [PATCH 20/20] Rust: provide guidance on manually managed Cargo.lock as fallback Signed-off-by: Christian Ehrhardt --- README.md | 32 ++++++++++++++++++++++++-------- 1 file changed, 24 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index 9fd7998e35..a338f932f9 100644 --- a/README.md +++ b/README.md @@ -451,10 +451,26 @@ RULE: - It is expected rust builds will use dh-cargo so that a later switch RULE: to non vendored dependencies isn't too complex (e.g. it is likely RULE: that over time more common libs shall become stable and then archive RULE: packages will be used to build). -RULE: - the rust tooling e.g. dh-cargo can not yet automatically provide -RULE: all that we require - for example Cargo.lock - but that will be -RULE: required to track dependencies. Until available in e.g. dh-cargo -RULE: a package can not yet get fully compliant. +RULE: - Right now that tooling to get a Cargo.lock that will include internal +RULE: vendored dependencies isn't in place yet (expect a dh-cargo change +RULE: later). Until it is available, as a fallback one can scan the +RULE: directory at build time and let it be generated in debian/rules. +RULE: An example might look like: +RULE: d/rules: +RULE: override_dh_auto_test: +RULE: CARGO_HOME=debian /usr/share/cargo/bin/cargo test --offline +RULE: d/.install: +RULE: Cargo.lock /usr/share/doc/ +RULE: d/config.toml +RULE: # Use the vendorized sources to produce the Cargo.lock file. This +RULE: # can be performed by pointing $CARGO_HOME to the path containing +RULE: # this file. +RULE: [source] +RULE: [source.my-vendor-source] +RULE: directory = "vendor" +RULE: [source.crates-io] +RULE: replace-with = "my-vendor-source" + RULE: - All vendored dependencies (no matter what language) shall have a RULE: way to be refreshed TODO-A: - This does not use static builds @@ -587,10 +603,10 @@ RULE: - Rust - toolchain and dh tools are still changing a lot. Currently it RULE: is expected to only list the rust toolchain in `Built-Using`. RULE: the remaining (currently vendored) dependencies shall be tracked RULE: in a cargo.lock file -RULE: Right now that tooling to get a cargo.lock that will include internal -RULE: packaged (if any) and used vendored dependencies isn't in place. -RULE: So right now due to the lack of that for example in dh-cargo one -RULE: can not yet get a package fully compliant. +RULE: - The rust tooling can not yet automatically provide all we require. +RULE: For example Cargo.lock - until available a package is at least +RULE: expected to generate this file itself at build time - an example +RULE: how to do so is shown above in the template for the reporter. RULE: - Go - here `Built-Using` is expected to only contain the go RULE: toolchain used to build it. Additional packaged dependencies RULE: will be tracked in `Static-Built-Using:` automatically.