[Devops] Branching Strategy - Phase 3

[lowlydba]

Goal

Build and publish packages to the right destinations using version strings computed by Phase 2. Each workflow has one trigger and one destination. No version logic lives here.

Trigger × destination matrix

Trigger	Workflow	Packages	Version	Destination
Push to vnext	p3-dev-builds-ca	affected pkgs only	<last-published>+dev.N	CA dev repo
Push to main (no <major>/<minor> bump)	p3-main-publish	affected pkgs only	<major>.<minor>.<next-patch>	⚠️ TBD — see below
<major>/<minor> bump on main	p3-release-publish	affected pkgs only	<major>.<minor>.0	Public PyPI

CodeArtifact: domain overture-pypi / account 505071440022 / region us-west-2.

Warning

Decision required before Phase 3 can start: Where do patch releases go?

• Option A — Patches → Public PyPI: every main merge is a public release. Consumers always get latest. PyPI gets noisy; every merged PR is a public event.
• Option B — Patches → CA only, PyPI on major/minor bumps only: cleaner public release signal. Internal (CDP) and external (PyPI) consumers diverge — external consumers only see curated major/minor releases.

This decision also affects whether p2-release-trigger (Phase 2.B) is needed as a PyPI gate, or whether a direct main push trigger is sufficient.

Tasks

p3-dev-builds-ca

• Trigger: push to vnext
• Detect affected packages (changed since last dev build).
• Use compute-version action to get <last-published>+dev.<run_number> per affected pkg.
• Build + publish to CA dev repo.
• CDP consumers pin >=<last-published>+dev.0 per package to track latest.

p3-main-publish

• Trigger: push to main with no <major>/<minor> changes.
• Detect affected packages.
• Use compute-version action to get <major>.<minor>.<next-patch> per affected pkg.
• Build + publish to TBD destination (CA or PyPI — see decision above).

p3-release-publish

• Trigger: <major>/<minor> bump on main (any package).
• Build affected packages at <major>.<minor>.0.
• Publish to public PyPI via Trusted Publishing (OIDC, no long-lived tokens) + PyPI attestations.
• Pattern follows overturemaps-py publish workflow.
• Prereq: PyPI Trusted Publisher OIDC must be configured per-package in PyPI project settings before this workflow lands.

p3-docs

Update docs/versioning.md with:

• Trigger × destination matrix (above)
• CDP dev build consumption pattern (>=<last-published>+dev.0)
• PyPI Trusted Publisher setup notes

Contributor-facing changes

• Push to vnext → affected packages auto-publish to CA dev repo as <version>+dev.N.
• Push to main (no version bump) → affected packages publish as <major>.<minor>.<next-patch> to TBD destination.
• Bump any package's <major>/<minor> on main → public PyPI publish fires automatically.
• Internal consumers (CDP): pin >=<last-published>+dev.0 to track vnext dev builds.

Definition of done

• Patch destination decision made (CA vs PyPI)
• PyPI Trusted Publisher OIDC configured for all packages
• p3-dev-builds-ca workflow live
• p3-main-publish workflow live
• p3-release-publish workflow live, tested against a real release
• docs/versioning.md updated

[vcschapp]

Arrived here from #534, via a #wg-schema Slack thread.

What's the best place to get an overall glossary and picture of the process along with reasons for key decisions?

Some questions I have are:

1/ What is a supplemental package?

Bump a supplemental package's <major>/<minor> on main → CA release repo only, never PyPI.

2/ Why do we need, and what is the utility of, having two separate CA repos, one dev and one release?

3/ What is the intention behind publishing every commit to main to the CA release repo?

4/ The only row that indicates Public PyPI, the third one, p3-pypi-publish, is speaking only about overture-schema, but overture-schema depends on a bunch of other packages that need to be in public PyPI, e.g., overture-schema-transportation-theme, overture-schema-common, overture-schema-system, etc. How are these other packages getting in to public PyPI?

5/ It will be common for changes to cut across packages, e.g. a change may be made to the common package and then used in multiple theme packages with the effect then manifesting also in overture-schema. All of these packages will need version changes to stay in sync. How will this work?

6/ The last two rows in the table that indicate "version bump on main" with a version <major>.<minor>.0 suggest to me that a human-generated PR can only produce .0 patch versions, and that only .0 patch versions will go to public PyPI. This isn't likely to be true for typical development: there may be trivial changes, such as bug fixes or docstring updates, that we want to make public via via patch version. How will this work?

[vcschapp]

One thing I meant to mention in the meeting this morning but didn't get a chance to: when doing the auto-publishes to dev CodeArtifact, we'll need to be aware of the dependency graph, which may require cascading version changes.

As an example, one branch of the dependency graph looks like:

overture-schema ← overture-schema-places-theme ← overture-schema-common ← overture-schema-system

If a change is made in overture-schema-common such that it gets a new auto-version, you need to ensure that the version of overture-schema-places-theme depends on the new published version, which necessitates a version bump for overture-schema-places-theme; with a cascading impact down the line to overture-schema.

This may be a non-trivial problem to solve, because the version-controlled pyproject.toml's are referring to the public versioning.

[lowlydba]

Victor Schappert (@vcschapp) I'm going to do some more cleanup, but this issue is updated post today's meeting and hopefully catalogs some of the thinking/decisions.

1/ What is a supplemental package?

Good callout - not a word we should use here. It was being used to refer to essentially non-core packages, but I think after the Schema WG meeting today (5/27) we established we can/should treat all packages the same in the context of versions/releases broadly.

2/ Why do we need, and what is the utility of, having two separate CA repos, one dev and one release?

This was more an artifact of the fact that those CAs will (soon) exist as our general approach for package management, but as we mentioned in todays WG call - not necessarily pertinent for Schema packages. The chart has been updated to remove that target - now we'll only hit Dev CA and PyPI, just need to decide which target receives main patches (i.e. do we release patches often publicly, or only publish major/minor updates publicly that contain lots of patches)

3/ What is the intention behind publishing every commit to main to the CA release repo?

As teased in the previous reply - the initial thought was to align us with true Continuous Deployments. But if there are negative outcomes from this (i.e. we don't want release churn) we could alternatively only make the patches internal, and publish big major/minor changes (see updated top comment)

4/ The only row that indicates Public PyPI, the third one, p3-pypi-publish, is speaking only about overture-schema, but overture-schema depends on a bunch of other packages that need to be in public PyPI, e.g., overture-schema-transportation-theme, overture-schema-common, overture-schema-system, etc. How are these other packages getting in to public PyPI?

Fixed - removed that row, and made the others more generic - we'll talk about packages all together going forward for release/versioning.

5/ It will be common for changes to cut across packages, e.g. a change may be made to the common package and then used in multiple theme packages with the effect then manifesting also in overture-schema. All of these packages will need version changes to stay in sync. How will this work?

See my last comment reply below - we're not pinning dep versions, so this should flow naturally.

6/ The last two rows in the table that indicate "version bump on main" with a version ..0 suggest to me that a human-generated PR can only produce .0 patch versions, and that only .0 patch versions will go to public PyPI. This isn't likely to be true for typical development: there may be trivial changes, such as bug fixes or docstring updates, that we want to make public via via patch version. How will this work?

Depends on the above answers - but we can do whatever mix/match we choose pretty easily.

One thing I meant to mention in the meeting this morning but didn't get a chance to: when doing the auto-publishes to dev CodeArtifact, we'll need to be aware of the dependency graph, which may require cascading version changes.

I actually am not super worried about this for us - primarily because all the non-core package dependencies aren't pinned to versions.

To walk through an example: say someone merges a fix to overture-schema-common on main. CI auto-publishes overture-schema-common 0.1.2. Now a consumer installs overture-schema-places-theme 0.1.1 -- because places-theme declares "overture-schema-common" with no pin, they get 0.1.2 automatically. Same logic flows all the way up to overture-schema.

The only time you'd need to manually bump a downstream is if you're intentionally encoding a floor, like "this release of places-theme requires at least common 0.2.0 because we depend on a new API". But that's a deliberate / bump decision to begin with, not something that should cascade automatically. At that point it needs to be a human choice, and the assumption is the code change author/reviewers are best to set floors as-needed for deps.