Skip to content

feat(firestartr-bootstrap): add azure support for "bring your own cluster" #493

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jalvarezit wants to merge 2 commits into
base: main
Choose a base branch
from
Open

feat(firestartr-bootstrap): add azure support for "bring your own cluster" #493

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f3873fd
add new implementation according to new ADR
jalvarezit Aug 7, 2026
1d42e10
Merge branch 'main' into fix/GH-486
jalvarezit Aug 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
134 changes: 134 additions & 0 deletions docs/research/azure-bearer-token-auth-support.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,134 @@
# Research: Azure Bearer Access Token Auth Support

**Question:** Can an Azure Bearer access token (from `az account get-access-token`) be used
directly as authentication in ESO Azure Key Vault SecretStore, ESO PushSecret, and the
Terraform `azurerm` provider?

**Date:** 2026-07-24
**Sources:** Official ESO docs, ESO Go types (source), azurerm provider source, Terraform docs.

---

## ESO Version in this codebase

`firestartr-bootstrap/operator.go` installs ESO with:

```go
helm upgrade --install external-secrets external-secrets/external-secrets \
-n external-secrets --create-namespace
```

**No `--version` flag is set.** ESO is always installed at the latest available Helm chart
version. The `v1beta1` API types analysed below are current as of ESO v0.9+.

---

## Scenario 1: ESO SecretStore — Azure Key Vault (read path)

### Sources
- Docs: https://external-secrets.io/latest/provider/azure-key-vault/
- Source: `apis/externalsecrets/v1beta1/secretstore_azurekv_types.go` (ESO main branch)

### Supported `authType` values (enum, exhaustive)

| Value | Description |
|---|---|
| `ServicePrincipal` (default) | Needs `tenantId` + `clientId` + (`clientSecret` OR `clientCertificate`) |
| `ManagedIdentity` | AAD Pod Identity (deprecated upstream). No secret ref needed. |
| `WorkloadIdentity` | OIDC-based federated identity. Uses a Kubernetes `ServiceAccount`. |

### `AzureKVAuth` struct fields (the `authSecretRef` object)

```go
type AzureKVAuth struct {
ClientID *smmeta.SecretKeySelector `json:"clientId,omitempty"`
TenantID *smmeta.SecretKeySelector `json:"tenantId,omitempty"`
ClientSecret *smmeta.SecretKeySelector `json:"clientSecret,omitempty"`
ClientCertificate *smmeta.SecretKeySelector `json:"clientCertificate,omitempty"`
}
```

### Does ESO support a raw Bearer access token?

**No.** There is no `accessToken` field in `AzureKVAuth` and no `AccessToken` auth type in the
`AzureAuthType` enum. The struct is exhaustive — only `clientSecret` and `clientCertificate` are
accepted alongside `clientId`.

A token from `az account get-access-token` is a short-lived ARM Bearer token (valid ~1 hour,
audience `https://vault.azure.net/`). ESO has no mechanism to inject a pre-obtained Bearer token
directly into Key Vault HTTP requests.

**Workaround:** The closest option without a permanent SP is `WorkloadIdentity`, which uses OIDC
federation and requires cluster-level setup (`azure.workload.identity/client-id` annotation on a
`ServiceAccount`).

---

## Scenario 2: ESO PushSecret — Azure Key Vault (write path)

### Does PushSecret support different auth methods?

**No difference from the read path.** A `PushSecret` references a `SecretStore` (or
`ClusterSecretStore`) object via `spec.secretStoreRefs`. Authentication is entirely delegated to
the store — the `PushSecret` resource itself has no auth fields.

The same `AzureKVProvider` / `AzureKVAuth` types govern both read (`ExternalSecret`) and write
(`PushSecret`) operations. A raw Bearer access token is equally unsupported on the write path.

The only write-specific requirement is an elevated RBAC role:
- For secrets: `Key Vault Secrets Officer` (or Access Policy `Set`/`Delete`)
- For keys: `Key Vault Crypto Officer`
- For certificates: `Key Vault Certificates Officer`

---

## Scenario 3: Terraform `azurerm` provider

### Sources
- Docs (source): `website/docs/index.html.markdown` (hashicorp/terraform-provider-azurerm main)
- Provider schema: `internal/provider/provider.go` (hashicorp/terraform-provider-azurerm main)

### Supported authentication methods (exhaustive from provider schema)

| Method | Key fields / env vars |
|---|---|
| Azure CLI | `use_cli` / `ARM_USE_CLI` (default `true`) |
| Service Principal + Client Secret | `client_secret` / `ARM_CLIENT_SECRET` |
| Service Principal + Client Certificate | `client_certificate` / `ARM_CLIENT_CERTIFICATE` |
| Managed Service Identity | `use_msi` / `ARM_USE_MSI` |
| AKS Workload Identity | `use_aks_workload_identity` / `ARM_USE_AKS_WORKLOAD_IDENTITY` |
| OIDC | `use_oidc` / `ARM_USE_OIDC` + `oidc_token` / `ARM_OIDC_TOKEN` |

### Does `azurerm` support `access_token` or `ARM_ACCESS_TOKEN`?

**No.** Neither `access_token` nor `ARM_ACCESS_TOKEN` appears anywhere in the provider schema or
documentation.

The `oidc_token` / `ARM_OIDC_TOKEN` field looks superficially similar but is fundamentally
different: it accepts an **OIDC ID token** (a JWT issued by an OIDC provider to be exchanged for
an Azure access token), not a pre-obtained ARM Bearer access token from
`az account get-access-token`.

### `use_cli = true` is the implicit path

When `ARM_USE_CLI` is not overridden (default `true`), the provider delegates to the Azure CLI
credential chain, which calls `az account get-access-token` internally. This is the only way a
CLI-obtained token is consumed — indirectly, with the provider managing token refresh. You cannot
supply the raw token string yourself.

---

## Summary table

| Scenario | Bearer token supported? | Config field/env var | Notes |
|---|---|---|---|
| ESO SecretStore (Azure KV, read) | **No** | — | Only `ServicePrincipal`, `ManagedIdentity`, `WorkloadIdentity` auth types exist in the API |
| ESO PushSecret (Azure KV, write) | **No** | — | Delegates to same SecretStore; no separate auth path |
| Terraform `azurerm` provider | **No** | — | No `access_token`/`ARM_ACCESS_TOKEN` field; CLI token used implicitly via `use_cli=true` |

### Token characteristics (for context)

- `az account get-access-token` returns a token valid for **~1 hour**.
- The default audience is `https://management.azure.com/` (ARM). For Key Vault operations you
need `--resource https://vault.azure.net/` — a different token.
- Neither ESO nor `azurerm` expose a field to inject either token directly.
175 changes: 152 additions & 23 deletions firestartr-bootstrap/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,13 +49,24 @@ The following AWS Parameter Store parameters are required:

### 2. Bootstrap File

There are two deployment modes, selected via `deploymentMode`:

| Field | SaaS (default) | Dedicated (Azure) |
|---|---|---|
| `deploymentMode` | `saas` (or omit) | `dedicated` |
| `env` | **required** — `"pre"` or `"pro"` | **omit** — no environment concept |
| `domain` | not used | **required** — base domain (e.g. `"azure-pre.firestartr.dev"`) |

#### 2.1 SaaS Bootstrap File (AWS)

```yaml
# BootstrapFile.yaml
# BootstrapFile.yaml (SaaS / AWS)
---
deploymentMode: saas # optional — saas is the default
org: <github org name> # github org name
customer: <customer name> # customer name used for Firestartr internally, for example to find secrets within the parameter store
env: <environment> # set either "pre" for firestartr-pre or "pro" for firestartr-pro
defaultOrgPermissions: <none | view | contribute> # default permissions for the <org>-all group, can be none, view or contribute
customer: <customer name> # customer name used for Firestartr internally
env: <environment> # required for SaaS: "pre" (firestartr-pre) or "pro" (firestartr-pro)
defaultOrgPermissions: <none | view | contribute> # default permissions for the <org>-all group, can be none, view or contribute
defaultBranch: main
defaultBranchStrategy: none
defaultDomainName: <default domain name> # ask customer for a default domain name to be used in the claims, for example "myproduct"
Expand All @@ -65,8 +76,8 @@ defaultFirestartrGroup: firestartr # default group for firestartr users and rela

firestartr:
# Check latest available release at github.com/prefapp/gitops-k8s
operator: <operator_version> # Ex. v1.56.1
cli: <cli_version> # Ex. v1.56.1
operator: <operator_version> # Ex. v1.56.1
cli: <cli_version> # Ex. v1.56.1

pushFiles:
claims:
Expand Down Expand Up @@ -147,10 +158,55 @@ components:
version: latest # Check available versions at github.com/prefapp/features
```

#### 2.2 Dedicated Bootstrap File (Azure)

For dedicated (non-SaaS) deployments, replace `env` with `deploymentMode: dedicated` and `domain`:

```yaml
# BootstrapFile.yaml (dedicated / Azure)
---
deploymentMode: dedicated
domain: "azure-pre.firestartr.dev" # fully-qualified base domain; no env suffix needed
org: <github org name>
customer: <customer name>
# NOTE: 'env' is absent — dedicated deployments have no environment concept
defaultOrgPermissions: <none | view | contribute>
defaultBranch: main
defaultBranchStrategy: none
defaultDomainName: <default domain name>
defaultSystemName: <default system name>
defaultGroup: <default group>
defaultFirestartrGroup: firestartr

firestartr:
operator: <operator_version>
cli: <cli_version>

pushFiles:
claims:
push: true
repo: "claims"
dotFirestartr:
push: true
repo: ".firestartr"
crs:
providers:
github:
push: true
repo: "state-github"

components:
# state-sys-services and state-argocd are auto-injected; no need to declare them here
- name: "dot-firestartr"
...
```

All the parameters must be filled. When copy pasting this file, `<placeholders>` must be replaced, but any other values can be treated as defaults and changed if needed:

- `org`: name of the GitHub organization where Firestartr will be installed.
- `env`: environment where the deployment and ArgoCD application will be created. Can be either `pre` or `pro`, and will result in commits being created to the necessary repositories in the `firestartr-<env>` organization.
- `deploymentMode`: deployment topology. `saas` (default, can be omitted) targets the shared `firestartr-<env>` org on AWS. `dedicated` targets the customer's own GitHub org on Azure.
- `env`: **SaaS only.** Environment where the deployment and ArgoCD application will be created. Can be either `pre` or `pro`, resulting in commits to the `firestartr-<env>` organization. **Omit for dedicated deployments.**
- `domain`: **Dedicated only.** Fully-qualified base domain for the deployment (e.g. `azure-pre.firestartr.dev`). The webhook URL will be `https://<customer>.events.<domain>`. **Omit for SaaS deployments.**
- `customer`: name used for the org internally, to compose the parameter store paths (e.g. `/firestartr/fs-<customer>-admin/app-id`). Must be set even if it matches the org name.
- `defaultBranch`: default branch name to set in the `defaults` config file, `claims_defaults.yaml`. Usually `main` or `master`.
- `defaultSystemName`: the name of the system that will be created by the bootstrapping process and set in the `claims_defaults.yaml` configuration file. Though any name can be used, it's recommended the bootstrap operator asks the client which system name they want to use as default.
Expand All @@ -162,10 +218,10 @@ All the parameters must be filled. When copy pasting this file, `<placeholders>`
- `firestartr.operator`: Firestartr version to be used by the operator. Must be the name of an image tag, without the flavor (i.e., `v1.53.0` instead of `v1.53.0_full-aws` or `v1.53.0_slim`). You can check the latest available image version [here](https://github.com/prefapp/gitops-k8s/pkgs/container/gitops-k8s).
- `firestartr.cli`: Firestartr CLI version to be used in the importation process. You can check the latest available CLI version [here](https://github.com/prefapp/gitops-k8s/blob/main/.release-please-manifest.json#L2). Note that this CLI version **won't** be the version set as the `FIRESTARTR_CLI_VERSION` organization variable, which is set from the parameter store instead (`/firestartr/<customer-name>/firestartr-cli-version`).
- `pushFiles`: whether or not to push the files create to their respective repositories once the bootstrap process finishes. Each section has two parameters: `push`, which if `true` will push those files to `repo`, whose value should be the name of the repository where those files will be pushed to.
- `components`: list of repositories to create during the bootstrap process. The values of each component will be explained in section 2.1. For a default bootstrap installation, it's recommended to leave them as is and update only the `<feature-version>` placeholders. This section should only be updated on special cases (e.g., the client already has a `claims` repository created).
- `components`: list of repositories to create during the bootstrap process. The values of each component will be explained in section 2.3. For a default bootstrap installation, it's recommended to leave them as is and update only the `<feature-version>` placeholders. This section should only be updated on special cases (e.g., the client already has a `claims` repository created). **For dedicated deployments, `state-sys-services` and `state-argocd` are auto-injected and do not need to be listed here.**


#### 2.1 Components
#### 2.3 Components

Each component represents a repository that will be created in the organization. All fields are mandatory. The parameters are:

Expand Down Expand Up @@ -212,28 +268,83 @@ The rest of the parameters of the `cloudProvider` section are the AWS S3 bucket
- `github.prefappBotPat`: Personal Access Token for the Prefapp Bot user, used to download the features from the features repository.
- `github.operatorPat`: Personal Access Token for the Operator user, used to commit the deployment and ArgoCD application PRs to the `firestartr-<env>` organization.

#### 3.2 Azure terraform backend provider configuration (currently not supported)
#### 3.2 Azure dedicated deployment configuration

For dedicated (non-SaaS) deployments on Azure, set `deploymentMode: dedicated` in your `Bootstrapfile.yaml` and use the following `Credentialsfile.yaml` format:

```yaml
# Credentialsfile.yaml
---
cloudProvider:
providerConfigName: backend-provider-config-name
name: azurerm
name: azure
config:
use_azuread_auth: true
tenant_id: "00000000-0000-0000-0000-000000000000"
client_id: "00000000-0000-0000-0000-000000000000"
client_secret: "************************************"
storage_account_name: "abcd1234"
tenant_id: "<azure-tenant-id>"
subscription_id: "<azure-subscription-id>"
# Runtime identity: firestartr-mi User-Assigned Managed Identity.
# Used in the deployed AKS cluster via Workload Identity (no client secret).
client_id: "<firestartr-mi-client-id>"
# Bootstrap identity: dedicated App Registration (Service Principal).
# Used only during bootstrap in the local kind cluster by ESO and Terraform.
# Delete the entire App Registration after bootstrap completes.
bootstrap_client_id: "<bootstrap-sp-application-id>"
bootstrap_client_secret: "<bootstrap-sp-client-secret>"
storage_account_name: "tfstate<customer>" # Azure Storage Account name for Terraform state
container_name: "tfstate"
source: hashicorp/aws
type: aws
version: ~> 4.0
resource_group_name: "rg-firestartr" # Also used for DNS zone resource group
key_vault_name: "firestartr-kv" # Azure Key Vault name (no slashes in secret names)
aks_cluster_name: "<aks-cluster-name>" # Name of the target AKS cluster
location: "<azure-region>" # Azure region, e.g. "westeurope" — must match resource group
source: hashicorp/azurerm
type: azurerm
version: "~> 3.0"
github:
providerConfigName: github-app-provider-config-name
prefappBotPat: "<prefapp-bot-pat>"
operatorPat: "<operator-pat>"
```

And your `Bootstrapfile.yaml` must include `deploymentMode` and `domain`:

```yaml
# Bootstrapfile.yaml (dedicated Azure example)
---
deploymentMode: dedicated
domain: "azure-pre.firestartr.dev" # Fully-qualified base domain; encodes env if needed
org: "<your-github-org>"
customer: "<customer-name>"
# (no 'env' field for dedicated deployments)
...
```

**Pre-flight requirements for dedicated Azure deployments:**

1. A `firestartr-mi` User-Assigned Managed Identity with OIDC federation configured on the AKS cluster.
2. A bootstrap App Registration (Service Principal) in Entra ID with:
- `Key Vault Secrets Officer` on the Azure Key Vault
- `Storage Blob Data Contributor` on the Terraform state storage account
- A client secret (populate `bootstrap_client_id` and `bootstrap_client_secret` in the credentials file)
- **Delete the entire App Registration after bootstrap completes.**
3. Azure Key Vault populated with all required secrets using the `[a-zA-Z0-9-]` naming convention:
- `fs-pem`, `fs-app-id`, `fs-<org>-installation-id`
- `fs-admin-pem`, `fs-admin-app-id`, `fs-admin-<org>-installation-id`
- `fs-argocd-pem`, `fs-argocd-app-id`, `fs-argocd-<org>-installation-id`
- `fs-state-pem`, `fs-state-app-id`, `fs-state-<org>-installation-id`
- `fs-checks-pem`, `fs-checks-app-id`, `fs-checks-<org>-installation-id`
- `fs-import-pem`, `fs-import-app-id`, `fs-import-<org>-installation-id`
- `github-webhook-secret`, `prefapp-bot-pat`, `firestartr-cli-version`
4. Delegated DNS zone matching `domain` in the Azure resource group.

**What the dedicated bootstrap does differently from SaaS:**

| Concern | SaaS (AWS) | Dedicated (Azure) |
|---|---|---|
| Repo targeting | `firestartr-<env>/<repo>` | `<bootstrap.Org>/<repo>` |
| Cluster services | ESO only (pre-provisioned) | Installs ESO + nginx + cert-manager + external-dns + ArgoCD |
| Webhook URL | `<customer>.events[.<env>].firestartr.dev` | `<customer>.events.<domain>` |
| Secret refs | AWS Parameter Store paths | Azure Key Vault dash-delimited names |
| Deployment values | `values.tmpl` (AWS/ALB/IAM) | `azure_values.tmpl` (Azure/nginx/MI) |

> **Important:** The `bootstrap_client_id` and `bootstrap_client_secret` belong to a dedicated App Registration created solely for this bootstrap run. The deployed state uses `firestartr-mi` via Workload Identity and never needs a client secret. **Delete the entire App Registration after bootstrap completes** — not just the secret.

### 4. How to launch the bootstrap

`<your-kind-port>`: Replace with the port that kind is using to expose the Kubernetes API server (noted in step 1.1).
Expand Down Expand Up @@ -406,8 +517,25 @@ dagger --bootstrap-file="./Bootstrapfile.yaml" \
call cmd-push-deployment
```

Creates a deployment PR in `firestartr-<env>/app-firestartr`.
- **SaaS:** Creates a deployment PR in `firestartr-<env>/app-firestartr`.
- **Dedicated:** Creates a deployment PR in `<org>/state-sys-services` with all sys-service release descriptors and values.

Apply sys-services with values (**dedicated only**):

```shell
dagger --bootstrap-file="./Bootstrapfile.yaml" \
--credentials-secret="file:./Credentialsfile.yaml" \
call cmd-apply-sys-services \
--docker-socket=/var/run/docker.sock \
--kind-svc=tcp://localhost:<your-kind-port> \
--kind-cluster-name=<your-kind-cluster-name>
```

This step is **only needed for dedicated deployments**. It renders the Azure-specific Helm values and applies them directly to the AKS cluster via `helm upgrade --install --values`, ensuring all cluster services (nginx, cert-manager, external-dns, ArgoCD, argo-events, argo-workflows) are correctly configured from the start.

The AKS cluster name is read from `aks_cluster_name` in the credentials file. If the external-dns Managed Identity client ID cannot be resolved automatically, the command prompts for the client ID on the terminal.

> Merge the `state-sys-services` PR **before** running this step so the desired state is recorded in git first.

Create ArgoCD application PR:

Expand All @@ -417,7 +545,8 @@ dagger --bootstrap-file="./Bootstrapfile.yaml" \
call cmd-push-argo
```

Creates an application PR in `firestartr-<env>/state-argocd`.
- **SaaS:** Creates an application PR in `firestartr-<env>/state-argocd`.
- **Dedicated:** Creates an application PR in `<org>/state-argocd` and patches `<org>/state-sys-services` with the ArgoCD secrets entry.

## 7. Troubleshooting

Expand Down
Loading
Loading