Pulumi CI Management

Purpose

This repository contains code to manage CI/CD for the many Pulumi providers in a consistent and (mostly) automated manner. The repo's intended audience are Pulumi Corp engineers, but its contents may also serve as a helpful example for Pulumi community members looking to maintain their own providers with a similar CI/CD process to Pulumi Corp.

Pulumi providers use GitHub Actions for CI/CD. Because we maintain a long list of providers, we use this repository to:

• Generate GitHub Actions Workflow files for any provider. These can be deployed to all providers or a single provider respectively by the GitHub Actions workflows in this repository.
• Keep an inventory of existing Pulumi providers.
• Maintain logic for branch protection across provider repositories.

Usage

This repository has the following components:

• The provider-ci directory contains code to generate GitHub Actions workflow files for Pulumi providers, as well as the generated output for each provider (retained for the purpose of convenient output diffing).

• The infra/providers/ directory contains a Pulumi program which uses the Pulumi GitHub provider to ensure consistent branch protections across our provider repositories.

  For an overview of how Pulumi programs work, see the Pulumi docs.

• GitHub Actions workflows to automate common operations across all providers or a single provider.

Prerequisites

The following tools are required for generating and deploying GitHub Actions workflows:

• Mise

Dependencies required are modeled on mise.toml. Run mise install and mise settings experimental=true (required for Go binaries such as golangci-lint and pulumictl) to fetch and install them. Finally, run mise env to check if env variables are being set correctly.

Building

After checking out the code, run the following command:

cd provider-ci && make clean && make -j

Common commands:

• make: Generate all code and check the output.
• make provider NAME=aws: Generate code for single provider with debug information
• make lint-providers: Check the generated code for all providers.
• make lint-providers/aws/repo: Check the generated code for a specific provider.

Adding generated workflows to your local provider repository

Sometimes, you want those changes NOW rather than having to wait for a GitHub PR. This example command will generate workflows for pulumi-datadog, and place them in the specified --out directory. Adjust for your provider and filesystem.

./bin/provider-ci generate --name pulumi/pulumi-datadog --template bridged-provider --config ./providers/datadog/config.yaml --out ../../pulumi-dtadog

Adding a New Bridged Provider

To add a new provider:

1. A new provider needs a top-level .ci-mgmt.yaml in its own repository with the following basic configuration:

   # Required values:
   provider: foo # substitute the name of your provider, without the pulumi- prefix
   env: # A map of required configuration for any integration tests, etc.
     AN_OPTION: value
     ANOTHER_OPTION: true
     # etc.
   lint: true # Linting should be true in most cases, unless failing rules in the upstream provider makes this impractical.
   
   # Optional values:
   docker: true # Whether the provider's tests use Docker to run. If set to true, a file `testing/docker-compose.yml` must be present in the provider repository.
   setup-script: testing/setup.sh # Path to a script that's used for testing bootstraps

   ci-mgmt will read your provider's .ci-mgmt.yaml and generate the standard set of CI files from templates. You can override every one of the default values in your .ci-mgmt.yaml file.

2. Add your provider to provider-ci/providers.json in alphabetical order. This ensures your provider receives regular updates and maintenance.

3. Add your provider to .github/ISSUE_TEMPLATE/0-ecosystem-providers.md to ensure manual rollouts track your provider.

4. Commit the changes and open a pull request.

5. To receive a pull request with your new config files, you can run the Update Workflows, Single Bridged Provider workflow run, using your provider name as the input. Another option is to wait for the nightly cronjob to send this pull request automatically.

6. If you would like to manually generate the configuration to get started right away, you can do so in your provider repository root:

   go run github.com/pulumi/ci-mgmt/provider-ci@master generate \
      --name pulumi/pulumi-$(PROVIDER_NAME) \
      --out . \
      --template bridged-provider \
      --config .ci-mgmt.yaml

   The generated files will be writen to your current directory.

Customizing SDK Generation (sdk-hooks.mk)

Providers occasionally need to fine-tune pulumi package gen-sdk (or legacy codegen) output, for example to delete a stray generated file or patch a type stub after each regeneration. Because ci-mgmt regenerates and overwrites the Makefile, editing the generated recipes directly is not regen-safe. The generated Makefile therefore exposes a provider-owned escape hatch: an optional sdk-hooks.mk.

Contract

The generated Makefile ends with -include sdk-hooks.mk (the leading - means no error if the file is absent), and invokes two slots around each per-language SDK codegen recipe, in both the gen-sdk and the legacy codegen branches:

• PRE_GEN_SDK_<LANG> runs immediately before the codegen step.
• POST_GEN_SDK_<LANG> runs immediately after codegen, before the recipe's completion sentinel is touched.

<LANG> is the uppercased SDK language, giving ten slots in total: PRE_GEN_SDK_GO / POST_GEN_SDK_GO, and likewise for NODEJS, PYTHON, DOTNET, and JAVA. Each slot is a plain make variable expanded as a recipe line, so an unset slot expands to nothing and make skips it entirely. With no sdk-hooks.mk present, every slot is unset and behavior is identical to today.

A slot's value is a shell command run from the provider repo root (the same working directory as the codegen recipe). For anything beyond a single command, point the slot at a checked-in script so the logic stays readable and testable.

Opt-in and regen-safe

ci-mgmt never creates sdk-hooks.mk; it is entirely opt-in, and regenerating CI does not touch it. To adopt it, copy provider-ci/sdk-hooks.mk.example into your provider repo root as sdk-hooks.mk, keep only the slots you need, and commit it.

Worked example

pulumi-pulumiservice needs to drop a generated file from the Go SDK and patch a Python stub after each regeneration. With this sdk-hooks.mk in the repo root:

POST_GEN_SDK_GO     = rm -f sdk/go/pulumiservice/internal/unused.go
POST_GEN_SDK_PYTHON = ./scripts/patch-python-stubs.sh

running make generate_go deletes the stray file right after gen-sdk emits it, and make generate_python runs the stub patcher. No other language is affected, and removing sdk-hooks.mk restores stock behavior.

Updating All Bridged Providers

The Update GH Workflows, ecosystem providers Workflow runs on a nightly schedule. You may trigger this Workflow manually; however be aware that this causes a lot of GitHub Actions to run at the same time, which may cause rate limiting across the org. Plan ahead and do this at a low-traffic time.

Updating Agentic Workflows

The root .github/workflows/gh-aw-*.md files are the source of truth for provider agentic workflows. After editing or upgrading those files, sync them into the provider templates:

make sync-gh-aw-templates

To check that the templates are already synced:

make check-gh-aw-templates

The Upgrade Agentic Workflows GitHub Actions workflow runs on a schedule and uses gh-aw upgrade to update the root agentic workflow sources and lock files, then syncs them into provider-ci/internal/pkg/templates/internal.

Provider repositories can opt out of generated agentic workflows by setting this in their .ci-mgmt.yaml:

disableAgenticWorkflows: true

Updating GitHub workflow schema

Fetch the latest JSON Schema then re-generate type definitions:

make discovery

Automatically editing source code across provider repositories

You can apply ad-hoc source edits across provider repositories even on files that are not managed or generated by ci-mgmt. For example, you might need to automatically update example code to use a newer Pulumi SDK major version dependency or a newer version of the underlying infrastructure such as .NET Framework. This can be done with migrations:

• describe your desired edits as a SourceMigration in provider-ci/internal/pkg/migrations

• test your changes locally by running ci-mgmt:

  go run . generate -c ../../pulumi-azure/.ci-mgmt.yaml -o ../../pulumi-azure/

• stand up a PR to ci-mgmt

• trigger an action such as update-workflows-bridged-providers.yml from the PR; this will create PRs that synchronize the selected repositories with ci-mgmt and apply the source migration as part of the change

• merge the PR to ci-mgmt

• merge the PR to the desired provider repositories (not needed if these PRs are set to automerge)