Variable Expansion in Migration Files

This feature lets one migration file carry per-deployment values — namespace prefixes, TTLs, replication factors, role names — instead of one hand-copied file per environment.

It is the complement to profiles. Profiles pick which file runs; expansion substitutes values within a file. Use profiles when the variance is structural and the set of targets is closed (mongo vs documentdb). Use variables when the value space is open — a tenant namespace, a region, a retention window — where you would otherwise need one file per value.

Expansion is off unless you add a manifest. A repository without one behaves exactly as it does today.

Quick start

Three things are needed: a manifest, {{ .VARIABLE }} references in your bodies, and one call.

1. Declare the variables in migrate.vars.json, beside the migrations:

{
  "version": 1,
  "vars": [
    {
      "name": "NAMESPACE",
      "pattern": "^[a-z][a-z0-9_]{0,62}$",
      "description": "Tenant namespace prefix for collection names"
    },
    {
      "name": "TTL_SECONDS",
      "default": "604800",
      "pattern": "^[0-9]{1,10}$",
      "description": "Document retention window"
    }
  ]
}

2. Reference them in migration bodies:

[
  { "create": "{{ .NAMESPACE }}_sessions" },
  { "collMod": "{{ .NAMESPACE }}_sessions",
    "expireAfterSeconds": {{ .TTL_SECONDS }} }
]

3. Supply the values. In Go:

err := m.SetSourceVars(source.VarConfig{
    Vars: map[string]string{"NAMESPACE": "acme"},
})

or on the CLI:

migrate -path ./migrations -database "$URL" -var NAMESPACE=acme up

TTL_SECONDS is not supplied, so it takes its declared default.

The manifest

Every variable must be declared in migrate.vars.json in the migrations directory. The scanner ignores the file — it does not match the migration filename pattern, so it is never treated as a migration.

Field	Meaning
`version`	Required, must be exactly `1`.
`name`	Required, `^[A-Z][A-Z0-9_]*$`. Uppercase makes substitution sites obvious in review.
`default`	Used when no value is supplied. A declaration with no `default` is required — there is no separate `required` flag.
`pattern`	Optional RE2 the value must match. Strongly recommended — it is the primary defense against a value that changes what a statement means.
`description`	Free text for whoever reads the manifest. Nothing prints it.

A default is validated exactly like a supplied value — same pattern, same forbidden characters. A manifest whose default contradicts its own pattern is rejected, and a default containing ; is rejected even though no one passed it on a command line.

Notes:

Patterns are anchored for you. [a-z]+ behaves as ^[a-z]+$; a pattern that matched only a prefix would look like a control in review and not be one. Writing ^…$ yourself still works.
Unknown keys are rejected. A typo'd "patern" is an error rather than a silently ignored key that leaves the real pattern empty.
Names must be unique. vars is an array, so duplicates are possible to write; whichever won would silently decide the default.
JSON, not YAML, so no new dependency is pulled in for it.

Syntax: values only

Accepted: {{ .VARIABLE }} — a single name, nothing else. Whitespace inside the braces is optional.

Rejected, with an error naming the file and the offending text:

{{ if .SHARDED }}   {{ range .LIST }}   {{ .A.B }}
{{ printf .X }}     {{ .X | upper }}    {{ template "x" }}
{{ define "x" }}    {{ block "x" . }}

This restriction is deliberate. schema_migrations records only a version number, so "prod and staging are both at version 7" has to mean the same schema. Conditionals would make a file's structure a function of the environment and break that. If you need structural variance, use a profile.

Where values come from

Precedence, lowest to highest: manifest default → -vars-file → -var.

Neither the library nor the CLI reads the environment. Values must be passed in explicitly.

Programmatic (Go)

The typical service integration — a Migrate helper called at startup against a *mongo.Client the service already holds:

func Migrate(
    mongoClient *mongo.Client,
    dbName string,
    migrationFilePath string,
    profile string,           // "" | "mongo" | "documentdb" | …
    vars map[string]string,   // from the service config, not os.Getenv here
) error {
    dbDriver, err := mongoMigrate.WithInstance(mongoClient, &mongoMigrate.Config{
        DatabaseName: dbName,
    })
    if err != nil {
        return err
    }

    m, err := migrate.NewWithDatabaseInstance("file://"+migrationFilePath, dbName, dbDriver)
    if err != nil {
        return err
    }

    // Order matters: the profile selects the file set that variables are
    // validated against.
    if profile != "" && !m.SetSourceProfile(profile) {
        return errors.New("source driver does not support profiles")
    }

    // Vars is the only field most callers set. Note this does NOT ignore
    // source.ErrNoManifest — this service ships a manifest, so its absence
    // means it failed to deploy.
    if err := m.SetSourceVars(source.VarConfig{Vars: vars}); err != nil {
        return fmt.Errorf("migration variables: %w", err)
    }

    err = m.Up()
    if err != nil && !errors.Is(err, migrate.ErrNoChange) {
        return err
    }
    return nil
}

Any constructor works — New, NewWithDatabaseInstance, NewWithSourceInstance — because both SetSourceProfile and SetSourceVars are mutators on the returned *Migrate.

Two things worth calling out:

Take vars as a parameter, not os.Getenv inside the function. That keeps configuration arriving through explicit parameters and leaves the function testable.
Do not add defer m.Close() if the *mongo.Client is shared. Migrate.Close closes both drivers, and the MongoDB driver's Close is client.Disconnect — it would tear down the caller's client.

CLI

  -var NAME=VALUE  Supply a value for a declared migration variable.
                   Repeatable.
  -vars-file PATH  Load values from a flat JSON object of name to string.
                   -var overrides -vars-file.
  -vars-check      Validate the manifest and all migration bodies, then exit
                   without migrating. Honors -profile and needs no -database.
                   Reports variable names only, never values.

migrate -path ./migrations -database "$URL" \
        -profile documentdb-sharded \
        -var NAMESPACE=acme \
        -var TTL_SECONDS=2592000 \
        up

-vars-file values must be JSON strings. Substitution is textual, so there is nothing to convert a number into — and an unquoted large integer would be decoded as a float and silently changed (9007199254740993 → 9007199254740992).

{ "NAMESPACE": "acme", "TTL_SECONDS": "604800" }

create, force, drop, and version skip variable validation. They never read a migration body, and blocking them would block recovery: if a deploy fails on an unresolvable variable and leaves the database dirty, the operator running migrate … force 7 still does not have the value that caused the outage.

Validation: everything fails before anything runs

SetSourceVars reads and parses every migration body visible under the active profile — up and down, including versions applied years ago — and validates them all. A nil return means every migration that will run is guaranteed to render: reads execute the template that was validated, not a body re-read later.

Errors, all raised before Lock() and before a single statement executes:

Situation	Result
A referenced variable is not declared in the manifest	error naming file and variable
A referenced variable has no supplied value and no `default`	error naming file and variable
A supplied name is not declared (a typo)	error naming the variable
A value does not match its `pattern`	error naming the variable
A value contains `;`	error — see Multi-statement bodies
A body uses `{{ if }}`, `{{ range }}`, a function call, …	error naming file and the offending text
The manifest is malformed, has duplicate names, or an unknown key	error naming the manifest

Findings aggregate: twelve files with undeclared references produce one error listing all twelve, not twelve runs.

A declared variable that no visible migration references is fine — not an error, not a warning. Under profile Y, every variable belonging to profile X is unreferenced. That is the normal case, which is what lets one manifest serve every profile.

SetSourceVars either returns an error or succeeds silently. Expansion never logs anything.

Escaping

The escaping mode is chosen per migration from the file extension. You do not configure it.

Extension	Mode	Behavior
`.json`	JSON	The value is escaped as a JSON string body, so a `"` or `\` cannot break out of a string literal.
anything else	none	The value is spliced verbatim. The author owns the surrounding quotes.

A directory mixing .sql and .json bodies works; each gets the right mode.

pattern still matters under both modes. Escaping is position-blind: the same value is spliced into "name": "{{ .X }}" and into "ttl": {{ .X }}, and in the number position it works only because escaping is a no-op on digits. A value containing " there produces invalid JSON, failing at execution time instead of in validation. "pattern": "^[0-9]{1,10}$" moves that failure forward.

There is no "SQL string" mode. It would be correct only inside a quoted literal, the renderer cannot tell where it landed, and applied to CREATE TABLE {{ .NAMESPACE }}_users it would silently corrupt an identifier. For SQL: verbatim splicing plus a pattern.

Using variables with profiles

Both features work together, but SetProfile must come first:

m.SetSourceProfile("documentdb-sharded")   // 1. selects which files are visible
m.SetSourceVars(source.VarConfig{...})     // 2. validates exactly those files

Validation only covers files visible under the active profile. Set the profile afterwards and the compiled state is discarded — reads then fail with:

migrate: variable expansion: profile changed after SetVars; call SetVars again

SetProfile cannot return an error (its signature is fixed by the interface), so the failure surfaces at the next read. It is loud, and it states the fix.

This ordering is also what lets one manifest serve every profile: a variable used only by .documentdb-sharded. files is not required when a plain mongo deployment runs.

Multi-statement bodies

Several database drivers split a body on ; when multi-statement mode is on (postgres, pgx, pgx/v5, clickhouse, cassandra, neo4j). Expansion runs in the source layer, before that split, so a value containing ; would silently turn one statement into two.

The source layer cannot know the driver's delimiter, so instead any supplied value containing ; is rejected during validation.

If you genuinely need a ; in a value, set ForbidChars explicitly and accept the consequence:

source.VarConfig{
    Vars:        vars,
    ForbidChars: "\x00", // check against a byte no config can carry
}

Source driver support

Source driver	Expansion?
`file`	✅
`iofs`	✅
everything else	❌

Same matrix as profiles, and for the same reason: both are implemented on iofs.PartialDriver, and only file embeds it.

Unlike -profile, Migrate.SetSourceVars treats an unsupported driver as a hard error (migrate.ErrSourceVarsUnsupported) rather than a warning. Silently skipping expansion would ship literal {{ … }} to the database, so in-process callers should not swallow it.

The CLI is one notch softer, because every registered URL scheme except file:// is in the unsupported group and failing on all of them would break s3://, github://, gitlab://, and bitbucket:// runs that work today. It tolerates the error only when no values were supplied. Pass any -var or -vars-file against an unsupported driver and it is fatal.

CI gate

Validate per environment without a database:

migrate -source=file://./db/migrations -vars-check \
        -profile=documentdb-sharded \
        -vars-file=/tmp/prod.vars.json

Exits non-zero if the manifest is missing or invalid, if a referenced variable does not resolve, or if a value fails its pattern. It reports variable names only — never values, and never rendered bodies. This gate is built to run in CI against production values, so printing them would put any templated secret into a log that outlives the job.

Pass the same -profile the deploy uses, once per profile/environment pair. Checking without a profile proves nothing about profile-tagged files.

Generate the values file from whatever the deploy actually reads. A hand-maintained copy drifts, and then the gate validates a fiction — worse than no gate, because it reports green.

Services on the in-process path can skip the CLI entirely and assert the same thing from a test, using the same config the service loads:

func TestMigrationsRenderForAllEnvironments(t *testing.T) {
    for _, env := range []string{"dev", "staging", "prod"} {
        vars, profile := loadVarsFromConfig(t, env)

        src, err := source.Open("file://" + migrationFilePath)
        require.NoError(t, err)
        t.Cleanup(func() { _ = src.Close() })

        if profile != "" {
            src.(source.ProfileAware).SetProfile(profile)
        }
        err = src.(source.VarAware).SetVars(source.VarConfig{Vars: vars})
        require.NoError(t, err, "env %s", env)
    }
}

The type assertions are the point: if someone switches the service to an unsupported source driver, this fails here rather than in production.

Before you adopt: check for existing `{{`

Enabling expansion is not per-migration. The moment one migration uses a variable, every visible up and down body is parsed — including versions applied years ago. A pre-existing literal {{ anywhere in that corpus is a hard failure that blocks all migration.

Check before committing the first variable:

grep -rn '{{' ./db/migrations/

Nothing returned means adoption is safe. {{ cannot appear in a structural position in any supported dialect, but it does appear in two ordinary places:

Inside a string literal — {"k": "{{literal}}"} is valid JSON today.
In a Postgres nested array literal: '{{1,2},{3,4}}' is standard multidimensional-array syntax, and '{{}}' a plausible column default. This is the likeliest real hit in a SQL corpus.

One case is not a hard failure and so is worth grepping for separately: {{/* … */}} is a template comment, which the parser removes from the body rather than rejecting. If any migration contains that sequence today, the text disappears instead of erroring.

If there are hits, move the delimiters rather than editing the bodies:

source.VarConfig{Vars: vars, LeftDelim: "<<", RightDelim: ">>"}

Editing an already-applied migration changes nothing in the database while changing what a from-scratch rebuild produces. That is the worse trade.

Operational notes

Ship the manifest with the migrations. COPY db/migrations/ /migrations/ is safe; COPY db/migrations/*.up.json … silently omits the manifest. With embed.FS they are inseparable and this cannot happen.
Do not template the migrations directory through Helm. Helm uses {{ }} too, so a chart-rendered ConfigMap consumes {{ .NAMESPACE }} before migrate ever sees it. Bake migrations into the image, mount them from a source Helm does not render, or move the delimiters.
Prefer not to template secrets. Grant to a role name and set the password out of band. Expansion never logs values, but a validation error can name one, and a failed migration leaves the rendered statement wherever the database logs it. If a secret must be templated on the CLI path, mount it and point -vars-file at the mount so only the path appears in the pod spec and in ps.
Nothing is recorded. schema_migrations still holds (version, dirty); there is no checksum column and no render log. The values are recoverable from the deploy config they came from, which is versioned alongside the deploy.
In-process callers should not swallow source.ErrNoManifest. It means "expansion is not configured", which for a service that ships a manifest means the manifest failed to deploy. Only the CLI ignores it, because it serves repositories that may legitimately have none.

The one silent failure mode

A body containing {{ .NAMESPACE }} reaches the database verbatim when expansion never runs. That is usually loud — Postgres rejects CREATE TABLE {{ .NAMESPACE }}_users as a syntax error, MongoDB fails to unmarshal — but it is silent when the reference sits inside a string literal:

INSERT INTO config (key, value) VALUES ('prefix', '{{ .NAMESPACE }}');

That commits the literal text and reports success. It is reachable in exactly two situations:

The manifest failed to deploy and the CLI was invoked with no values at all. Supplying any value is a hard error, and in-process callers that do not ignore ErrNoManifest are caught regardless.
A library caller never calls SetSourceVars. Nothing can help here; not calling it is indistinguishable from not wanting it.

Ship the manifest with the migrations, run the -vars-check gate, and do not swallow the sentinel in service code.

Full example

db/migrations/
  migrate.vars.json
  001_create_sessions.up.json
  001_create_sessions.down.json
  002_add_ttl.up.json
  002_add_ttl.documentdb-sharded.up.json     # profile variant, also templated
  002_add_ttl.documentdb-sharded.down.json

migrate.vars.json:

{
  "version": 1,
  "vars": [
    {"name": "NAMESPACE", "pattern": "^[a-z][a-z0-9_]*$"},
    {"name": "TTL_SECONDS", "default": "604800", "pattern": "^[0-9]+$"},
    {"name": "SHARD_KEY", "pattern": "^[a-z_]+$"}
  ]
}

SHARD_KEY has no default and is referenced only by the .documentdb-sharded. file, so:

# Plain deployment: SHARD_KEY is not referenced, so it is not required.
migrate -path db/migrations -database "$URL" -var NAMESPACE=acme up

# Sharded deployment: now it is.
migrate -path db/migrations -database "$URL" \
        -profile documentdb-sharded \
        -var NAMESPACE=acme -var SHARD_KEY=tenant_id up

# Without SHARD_KEY, this fails before touching the database:
#   migrate: variable expansion: 002_add_ttl.documentdb-sharded.up.json:
#   no value supplied for variable "SHARD_KEY" and it declares no default

Navigation Menu

Uh oh!

Variable Expansion in Migration Files

Quick start

The manifest

Syntax: values only

Where values come from

Programmatic (Go)

CLI

Validation: everything fails before anything runs

Escaping

Using variables with profiles

Multi-statement bodies

Source driver support

CI gate

Before you adopt: check for existing `{{`

Operational notes

The one silent failure mode

Full example

Uh oh!

FilesExpand file tree

variable-expansion.md

Latest commit

History

variable-expansion.md

File metadata and controls

Variable Expansion in Migration Files

Quick start

The manifest

Syntax: values only

Where values come from

Programmatic (Go)

CLI

Validation: everything fails before anything runs

Escaping

Using variables with profiles

Multi-statement bodies

Source driver support

CI gate

Before you adopt: check for existing {{

Operational notes

The one silent failure mode

Full example

Before you adopt: check for existing `{{`