diff --git a/.github/actions/harden-runner/README.md b/.github/actions/harden-runner/README.md index 6140573b980..5445237ff15 100644 --- a/.github/actions/harden-runner/README.md +++ b/.github/actions/harden-runner/README.md @@ -58,9 +58,11 @@ proxy binds or its eBPF block policy finishes initializing, especially while upstream API requests retry. The helper does not widen firewall permissions or change Docker DNS. Other runner types retain StepSecurity's normal behavior. -Set `dns-repair: 'false'` for jobs that need no DNS, such as the echo-only autofix -workflow, or to observe the original behavior in diagnostics. This disables only -the resolver repair; StepSecurity still runs with the selected policy. +Keep DNS repair enabled even for echo-only jobs: the runner still needs DNS +for cleanup and final job reporting. The helper detects eBPF enforcement and +retains host DNS when no proxy is required. Set `dns-repair: 'false'` only to +observe the original behavior in diagnostics. This disables only the resolver +repair; StepSecurity still runs with the selected policy. The diagnostic workflow invokes the internal helper later to measure before and after on the same runner. Normal workflows need only the composite above. diff --git a/.github/workflows/autofix.yml b/.github/workflows/autofix.yml index 0222383a2f3..81cb4381574 100644 --- a/.github/workflows/autofix.yml +++ b/.github/workflows/autofix.yml @@ -13,9 +13,8 @@ jobs: - name: Harden Runner uses: $/.github/actions/harden-runner # nosemgrep: yaml.github-actions.security.github-actions-mutable-action-tag.github-actions-mutable-action-tag with: - # This job makes no network calls at all (just echo); block all egress. + # No workload endpoints are needed; retain StepSecurity implicit endpoints. policies: >- none - # The echo-only job needs no DNS; eBPF agents do not expose the legacy resolver. - dns-repair: 'false' + # Keep default DNS repair: the runner still needs final job reporting. - run: echo "See Atmos Pro" diff --git a/docs/prd/plugins/platform/protocol.md b/docs/prd/plugins/platform/protocol.md index 3a1c8784a76..424995d2b49 100644 --- a/docs/prd/plugins/platform/protocol.md +++ b/docs/prd/plugins/platform/protocol.md @@ -256,15 +256,17 @@ The runtime writes events to caller-provided stdout/stderr writers; callers supp ### Component configuration services -The component configuration protocol, generated CLI and SDK bindings, and component-specific host descriptors, package loading, and adapters are planned. CLI stack and execution integration are also unavailable. These proposed independently negotiated version-1 services replace the earlier proposed monolithic `Component` service: +The component configuration protocol is now defined in `proto/atmos/plugin/v1/plugin.proto` and generated for the CLI and SDK. The host has static descriptors, explicit package loading, and service adapters. Stack processing and Terraform execution call the plugin configuration runtime for components that declare a `configuration` settings section, and that call reaches the provider `Prepare` RPC. These independently negotiated services replace the earlier proposed monolithic `Component` service. Discovery, validation, and state use version 1. Preparation uses version 2: | Capability | RPC | Result | |---|---|---| | `component.discovery` | `ComponentDiscovery.Discover` | Logical instances, source watches, and dependency edges for an explicit root/revision. | | `component.validation` | `ComponentValidation.Validate` | Unsupported mode/command/options diagnostics before effects. | -| `component.preparation` | `ComponentPreparation.Prepare` | Fresh inputs, module, backend identity, and literal generated files. | +| `component.preparation` | `ComponentPreparation.Prepare` | Fresh inputs, module, backend identity, literal generated files, backend ownership, and execution metadata. | | `component.state` | `ComponentState.Resolve` | Backend/workspace identity without preparation. | +Preparation version 2 adds `terraform_arguments` to the request, and `backend_generated` and `execution` to the response. A host that only speaks version 1 would ignore these fields, drop hooks and extra arguments, and omit the arguments that `prevent_destroy` checks. The version change makes such a host reject the plugin during the handshake, before preparation. The manifest declares the same requirement as `component.preparation: ">=2 <3"`. + `ComponentRequest` carries explicit invocation, provider, coordinates, root/source, operation/options, effects, resolved component configuration, and separate provider settings. It does not carry host Go structs or callbacks. `ComponentValue` uses cty type JSON plus MessagePack and sensitivity paths, preserving exact numbers, unknowns, nulls, collection types, and sensitive values. Do not use `protobuf.Struct` for these values. The SDK rejects unsupported marks and sensitivity paths that cannot be applied. Configuration providers do not replace execution engines. Native Terragrunt units remain Terraform components executed by Atmos. Optional external execution and component CI services remain follow-on contracts: they must stream output, diagnostics/logs, lifecycle events, then exactly one authoritative outcome. Preserve detailed plan exit 2 as changes. Saved-plan identity must bind the exact applied artifact and effective configuration. diff --git a/errors/plugin.go b/errors/plugin.go index abd53c586de..abeb0df834e 100644 --- a/errors/plugin.go +++ b/errors/plugin.go @@ -89,6 +89,14 @@ var ( ErrComponentVarFileMissing = errors.New("required variable file does not exist") // ErrComponentHookWorkingDirectory identifies a hook whose working directory does not exist. ErrComponentHookWorkingDirectory = errors.New("hook working directory does not exist") + // ErrComponentMaxAttempts identifies a command that kept failing with a retryable error until its retry policy ran out of attempts. + ErrComponentMaxAttempts = errors.New("maximum retry attempts reached") + // ErrComponentBinaryChanged identifies a unit whose terraform_binary differs from the one it was discovered with. + ErrComponentBinaryChanged = errors.New("terraform_binary changed after discovery") + // ErrComponentRunIgnored ends a component run successfully after error handling ignored a failure in a preparatory step. + ErrComponentRunIgnored = errors.New("component run ended by an ignored error") + // ErrComponentExcluded ends a component run successfully because the provider excluded the operation. + ErrComponentExcluded = errors.New("component excluded from this operation") // ErrComponentOutputCapture identifies command output that could not be recorded for error hook matching. ErrComponentOutputCapture = errors.New("failed to capture command output for error hooks") diff --git a/go.mod b/go.mod index a4c07236e42..98cffa2a14c 100644 --- a/go.mod +++ b/go.mod @@ -185,7 +185,7 @@ require ( github.com/arsham/rainbow v1.2.1 // indirect github.com/avast/retry-go v3.0.0+incompatible // indirect github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.20 // indirect - github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.19.1 // indirect + github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.19.1 github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.22.34 // indirect github.com/aws/aws-sdk-go-v2/internal/configsources v1.5.6 // indirect github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.8.6 // indirect @@ -195,7 +195,7 @@ require ( github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.14.1 // indirect github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.20.1 // indirect github.com/aws/aws-sdk-go-v2/service/secretsmanager v1.47.0 - github.com/aws/aws-sdk-go-v2/service/signin v1.8.0 // indirect + github.com/aws/aws-sdk-go-v2/service/signin v1.8.0 github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect github.com/aymerick/douceur v0.2.0 // indirect github.com/bahlo/generic-list-go v0.2.0 // indirect diff --git a/internal/exec/configuration_imports_test.go b/internal/exec/configuration_imports_test.go index 9885fdb22e4..1c9ab0a52b3 100644 --- a/internal/exec/configuration_imports_test.go +++ b/internal/exec/configuration_imports_test.go @@ -214,7 +214,8 @@ func TestConfigurationWatchExclusionsDoNotChangeSharedIndex(t *testing.T) { filtered, err := configurationWatchIndex(config, settings, index) require.NoError(t, err) require.Equal(t, []string{filepath.Join(root, "live", "new.yaml")}, filtered.getAllFiles()) - require.Len(t, index.getAllFiles(), 2) + require.ElementsMatch(t, []string{filepath.Join(root, "live", ".terragrunt-cache", "generated.tf"), filepath.Join(root, "live", "new.yaml")}, index.getAllFiles(), + "the shared index keeps both changed files, each once") again, err := configurationWatchIndex(config, settings, index) require.NoError(t, err) require.Same(t, filtered, again, "components sharing exclusions must reuse one filtered index") @@ -253,6 +254,17 @@ func TestConfigurationWatchExclusionsFailClosedOnPathsOutsideTheProject(t *testi filtered, err := configurationWatchIndex(config, settings, index) require.ErrorIs(t, err, errUtils.ErrPluginInvalidConfig) require.Nil(t, filtered) + + // With an absolute project root, a changed file in a sibling directory is outside the project: + // project-relative exclusions cannot describe it, so it stays watched instead of disappearing. + parent := t.TempDir() + root := filepath.Join(parent, "project") + sibling := filepath.Join(parent, "sibling", "main.tf") + config = &schema.AtmosConfiguration{BasePathAbsolute: root} + index = newChangedFilesIndex(config, []string{sibling, filepath.Join(root, "live", "main.tf")}, "") + filtered, err = configurationWatchIndex(config, settings, index) + require.NoError(t, err) + require.Equal(t, []string{sibling}, filtered.getAllFiles()) } func TestDependencyChangeCheckReportsInvalidWatchExclusions(t *testing.T) { diff --git a/internal/exec/describe_affected_utils.go b/internal/exec/describe_affected_utils.go index fd0fffd42a1..a49aa709699 100644 --- a/internal/exec/describe_affected_utils.go +++ b/internal/exec/describe_affected_utils.go @@ -12,6 +12,7 @@ import ( errUtils "github.com/cloudposse/atmos/errors" "github.com/cloudposse/atmos/pkg/auth" cfg "github.com/cloudposse/atmos/pkg/config" + "github.com/cloudposse/atmos/pkg/filesystem" log "github.com/cloudposse/atmos/pkg/logger" "github.com/cloudposse/atmos/pkg/schema" u "github.com/cloudposse/atmos/pkg/utils" @@ -22,21 +23,7 @@ import ( // the non-existent remainder. The plain filepath.EvalSymlinks errors on // missing paths, and unused helmfile/packer default dirs routinely do not exist. func evalSymlinksBestEffort(path string) string { - remainder := "" - current := path - for { - resolved, err := filepath.EvalSymlinks(current) - if err == nil { - return filepath.Join(resolved, remainder) - } - parent := filepath.Dir(current) - if parent == current { - // Nothing along the path resolves — return the input unchanged. - return path - } - remainder = filepath.Join(filepath.Base(current), remainder) - current = parent - } + return filesystem.EvalSymlinksBestEffort(path) } // rebaseOnePathOntoWorktree maps one absolute config path from the local repo diff --git a/internal/exec/describe_affected_watch_cache_test.go b/internal/exec/describe_affected_watch_cache_test.go new file mode 100644 index 00000000000..ac6eb9d05a3 --- /dev/null +++ b/internal/exec/describe_affected_watch_cache_test.go @@ -0,0 +1,145 @@ +package exec + +import ( + "encoding/json" + "fmt" + "path/filepath" + "testing" + "time" + + "github.com/stretchr/testify/require" + + "github.com/cloudposse/atmos/pkg/schema" +) + +// BenchmarkConfigurationWatchCacheHitDuringBuild measures cached lookup latency +// while another exclusion set is built from a large shared change set. +func BenchmarkConfigurationWatchCacheHitDuringBuild(b *testing.B) { + root := b.TempDir() + config := &schema.AtmosConfiguration{BasePathAbsolute: root, BasePath: root} + files := make([]string, 9000) + for i := range files { + files[i] = filepath.Join(root, "modules", fmt.Sprintf("unit-%d/main.tf", i)) + } + settings := func(pattern string) map[string]any { + return map[string]any{"configuration": map[string]any{"watch_excludes": []any{pattern}}} + } + var worstTotal time.Duration + for i := 0; i < b.N; i++ { + index := newChangedFilesIndex(config, files, root) + if _, err := configurationWatchIndex(config, settings("**/.terragrunt-cache/**"), index); err != nil { + b.Fatal(err) + } + done := make(chan error, 1) + go func() { _, err := configurationWatchIndex(config, settings("**/*.md"), index); done <- err }() + var worst time.Duration + loop: + for { + select { + case err := <-done: + if err != nil { + b.Fatal(err) + } + break loop + default: + } + start := time.Now() + if _, err := configurationWatchIndex(config, settings("**/.terragrunt-cache/**"), index); err != nil { + b.Fatal(err) + } + if elapsed := time.Since(start); elapsed > worst { + worst = elapsed + } + } + worstTotal += worst + } + b.ReportMetric(float64(worstTotal.Nanoseconds())/float64(b.N), "max-hit-ns/build") +} + +// TestConfigurationWatchIndexConcurrentViews checks publication to concurrent +// callers, independent keys, and preservation of the original change set. +func TestConfigurationWatchIndexConcurrentViews(t *testing.T) { + root := t.TempDir() + config := &schema.AtmosConfiguration{BasePathAbsolute: root, BasePath: root} + files := make([]string, 9000) + for i := range files { + files[i] = filepath.Join("modules", fmt.Sprintf("unit-%d/main.tf", i)) + } + files = append(files, "README.md") + index := newChangedFilesIndex(config, files, root) + type result struct { + key int + view *changedFilesIndex + err error + } + results := make(chan result, 32) + start := make(chan struct{}) + for i := 0; i < cap(results); i++ { + go func(key int) { + <-start + patterns := []string{"**/*.tf", "**/*.md"} + settings := map[string]any{"configuration": map[string]any{"watch_excludes": []any{patterns[key]}}} + view, err := configurationWatchIndex(config, settings, index) + results <- result{key, view, err} + }(i % 2) + } + close(start) + views := [2]*changedFilesIndex{} + for i := 0; i < cap(results); i++ { + got := <-results + require.NoError(t, got.err) + if views[got.key] == nil { + views[got.key] = got.view + } else { + require.Same(t, views[got.key], got.view) + } + } + require.NotSame(t, views[0], views[1]) + require.Equal(t, []string{filepath.Join(root, "README.md")}, views[0].getAllFiles()) + require.Len(t, views[1].getAllFiles(), 9000) + require.Len(t, index.getAllFiles(), 9001) +} + +// TestConfigurationWatchIndexBuildsAreIsolated verifies that an in-flight key +// does not prevent cached lookups or a different key from being built. +func TestConfigurationWatchIndexBuildsAreIsolated(t *testing.T) { + root := t.TempDir() + config := &schema.AtmosConfiguration{BasePathAbsolute: root, BasePath: root} + index := newChangedFilesIndex(config, []string{"main.tf", "README.md"}, root) + settings := func(pattern string) map[string]any { + return map[string]any{"configuration": map[string]any{"watch_excludes": []any{pattern}}} + } + cached, err := configurationWatchIndex(config, settings("**/*.md"), index) + require.NoError(t, err) + key, err := json.Marshal([]string{root, "**/*.tf"}) + require.NoError(t, err) + entered, release := make(chan struct{}), make(chan struct{}) + blocked := index.watchBuilds.DoChan(string(key), func() (any, error) { + close(entered) + <-release + return cached, nil + }) + defer func() { close(release); <-blocked }() + <-entered + type lookup struct { + sameView bool + err error + } + done := make(chan lookup, 1) + go func() { + hit, err := configurationWatchIndex(config, settings("**/*.md"), index) + if err != nil { + done <- lookup{err: err} + return + } + _, err = configurationWatchIndex(config, settings("**/*.hcl"), index) + done <- lookup{sameView: hit == cached, err: err} + }() + select { + case result := <-done: + require.NoError(t, result.err) + require.True(t, result.sameView, "the cached view must be returned unchanged") + case <-time.After(5 * time.Second): + t.Fatal("an unrelated cache build blocked other keys") + } +} diff --git a/pkg/auth/identities/aws/inline_role.go b/pkg/auth/identities/aws/inline_role.go new file mode 100644 index 00000000000..09238c65075 --- /dev/null +++ b/pkg/auth/identities/aws/inline_role.go @@ -0,0 +1,109 @@ +package aws + +import ( + "cmp" + "context" + "fmt" + "strings" + "time" + + errUtils "github.com/cloudposse/atmos/errors" + "github.com/cloudposse/atmos/pkg/auth/types" + atmosio "github.com/cloudposse/atmos/pkg/io" + "github.com/cloudposse/atmos/pkg/perf" + "github.com/cloudposse/atmos/pkg/schema" +) + +// inlineRoleIdentityName names the transient identity behind a role that a component asks for. +const inlineRoleIdentityName = "component-iam-role" + +// InlineRole is an IAM role that a component, rather than the Atmos configuration, asks to assume. It +// reuses the aws/assume-role identity, so STS calls, endpoints, and error reporting behave the same way. +type InlineRole struct { + // Role is the role ARN. + Role string + // SessionName is the STS role session name. It defaults to a generated name. + SessionName string + // Duration is the session length. Zero selects the STS default. + Duration time.Duration + // WebIdentityToken selects AssumeRoleWithWebIdentity, which needs no starting credentials. + WebIdentityToken string +} + +// AssumeInlineRole assumes the role on top of the AWS credentials found in environ, a list of KEY=VALUE +// entries in which the last entry of a key wins. The starting credentials come from environ alone, in the +// order of the SDK default chain: static keys, a web identity token file, the profile and files that environ +// names, container credentials, then instance metadata. The process environment is neither read nor changed. It returns the +// environment variables that make the new session the credentials of a child process, and registers the +// secrets with the output masker. +func AssumeInlineRole(ctx context.Context, role InlineRole, environ []string) (map[string]string, error) { + defer perf.Track(nil, "aws.AssumeInlineRole")() + + env := environMap(environ) + identity, err := NewAssumeRoleIdentity(inlineRoleIdentityName, inlineRoleConfig(role, env)) + if err != nil { + return nil, err + } + var base types.ICredentials + if role.WebIdentityToken != "" { + base = &types.OIDCCredentials{Token: role.WebIdentityToken} + } else if base, err = baseCredentials(ctx, env); err != nil { + return nil, err + } + assumed, err := identity.Authenticate(ctx, base) + if err != nil { + return nil, err + } + session, ok := assumed.(*types.AWSCredentials) + if !ok || session.AccessKeyID == "" { + return nil, fmt.Errorf("%w: STS returned no credentials for role %s", errUtils.ErrAuthenticationFailed, role.Role) + } + atmosio.RegisterSecretValue(session.AccessKeyID) + atmosio.RegisterSecretValue(session.SecretAccessKey) + atmosio.RegisterSecretValue(session.SessionToken) + result := map[string]string{ + "AWS_ACCESS_KEY_ID": session.AccessKeyID, + "AWS_SECRET_ACCESS_KEY": session.SecretAccessKey, + "AWS_SESSION_TOKEN": session.SessionToken, + "AWS_SECURITY_TOKEN": session.SessionToken, + // Emptied so that a profile never competes with the session for the child process. + "AWS_PROFILE": "", + } + // The region is left to the invocation's own settings: the session does not choose one. + return result, nil +} + +// inlineRoleConfig describes the transient aws/assume-role identity. An STS endpoint in the environment, +// as emulators export, directs the call. +func inlineRoleConfig(role InlineRole, env map[string]string) *schema.Identity { + principal := map[string]any{"assume_role": role.Role} + if role.SessionName != "" { + principal[principalSessionNameKey] = role.SessionName + } + if role.Duration > 0 { + principal[principalDurationKey] = fmt.Sprintf("%ds", int64(role.Duration.Seconds())) + } + if region := cmp.Or(env["AWS_REGION"], env["AWS_DEFAULT_REGION"]); region != "" { + principal["region"] = region + } + identity := &schema.Identity{Kind: "aws/assume-role", Principal: principal} + if endpoint := cmp.Or(env["AWS_ENDPOINT_URL_STS"], env["AWS_ENDPOINT_URL"]); endpoint != "" { + identity.Spec = map[string]any{"endpoint_url": endpoint} + } + return identity +} + +func environMap(environ []string) map[string]string { + env := make(map[string]string, len(environ)) + for _, entry := range environ { + if key, value, ok := strings.Cut(entry, "="); ok { + env[key] = value + } + } + return env +} + +// invocationEnvironmentKeys applies SDK alias precedence without changing the invocation map. +func invocationEnvironmentKeys(env map[string]string) (string, string) { + return cmp.Or(env["AWS_ACCESS_KEY_ID"], env["AWS_ACCESS_KEY"]), cmp.Or(env["AWS_SECRET_ACCESS_KEY"], env["AWS_SECRET_KEY"]) +} diff --git a/pkg/auth/identities/aws/inline_role_container_login_test.go b/pkg/auth/identities/aws/inline_role_container_login_test.go new file mode 100644 index 00000000000..5bab3ee3c9c --- /dev/null +++ b/pkg/auth/identities/aws/inline_role_container_login_test.go @@ -0,0 +1,220 @@ +package aws + +import ( + "context" + "encoding/json" + "errors" + "net" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "testing" + "time" + + "github.com/aws/aws-sdk-go-v2/config" + "github.com/aws/aws-sdk-go-v2/credentials/logincreds" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + errUtils "github.com/cloudposse/atmos/errors" +) + +func TestValidatedContainerDialerDialsOnlyValidatedAddresses(t *testing.T) { + for _, test := range []struct { + name string + address string + resolved []string + lookupErr error + wantDial []string + wantErr error + }{ + {name: "host resolving to loopback", address: "credentials.internal:8080", resolved: []string{"127.0.0.1"}, wantDial: []string{"127.0.0.1:8080"}}, + {name: "host resolving to the ECS address", address: "ecs.internal:80", resolved: []string{"169.254.170.2"}, wantDial: []string{"169.254.170.2:80"}}, + {name: "IP literal skips the lookup", address: "127.0.0.1:9000", wantDial: []string{"127.0.0.1:9000"}}, + {name: "host resolving outside the allowlist", address: "evil.example:80", resolved: []string{"203.0.113.9"}, wantErr: errUtils.ErrInvalidAuthConfig}, + {name: "one rebound address among allowed ones", address: "rebind.example:80", resolved: []string{"127.0.0.1", "203.0.113.9"}, wantErr: errUtils.ErrInvalidAuthConfig}, + {name: "lookup failure", address: "missing.example:80", lookupErr: errors.New("no such host"), wantErr: errUtils.ErrInvalidAuthConfig}, + {name: "malformed address", address: "no-port", wantErr: errUtils.ErrInvalidAuthConfig}, + } { + t.Run(test.name, func(t *testing.T) { + var dialed []string + lookup := func(context.Context, string) ([]string, error) { return test.resolved, test.lookupErr } + dial := func(_ context.Context, _, address string) (net.Conn, error) { + dialed = append(dialed, address) + client, server := net.Pipe() + t.Cleanup(func() { _ = client.Close(); _ = server.Close() }) + return client, nil + } + + conn, err := validatedContainerDialer(lookup, dial)(t.Context(), "tcp", test.address) + + if test.wantErr != nil { + require.ErrorIs(t, err, test.wantErr) + assert.Empty(t, dialed, "nothing is dialed for an address that failed validation") + return + } + require.NoError(t, err) + require.NotNil(t, conn) + assert.Equal(t, test.wantDial, dialed, "the dial uses the validated address, not the host name") + }) + } +} + +func TestValidatedContainerDialerTriesNextValidatedAddress(t *testing.T) { + var dialed []string + lookup := func(context.Context, string) ([]string, error) { return []string{"127.0.0.1", "::1"}, nil } + dial := func(_ context.Context, _, address string) (net.Conn, error) { + dialed = append(dialed, address) + if len(dialed) == 1 { + return nil, errors.New("connection refused") + } + client, server := net.Pipe() + t.Cleanup(func() { _ = client.Close(); _ = server.Close() }) + return client, nil + } + + _, err := validatedContainerDialer(lookup, dial)(t.Context(), "tcp", "credentials.internal:80") + + require.NoError(t, err) + assert.Equal(t, []string{"127.0.0.1:80", "[::1]:80"}, dialed) +} + +// writeLoginToken caches an unexpired login token for the session, so no refresh request is needed. +func writeLoginToken(t *testing.T, session, dir, accessKeyID string) { + t.Helper() + path, err := logincreds.StandardCachedTokenFilepath(session, dir) + require.NoError(t, err) + token := map[string]any{ + "accessToken": map[string]any{ + "accessKeyId": accessKeyID, "secretAccessKey": "secret", "sessionToken": "session", "accountId": "123456789012", + "expiresAt": time.Now().Add(time.Hour).UTC().Format(time.RFC3339), + }, + "clientId": "client", "refreshToken": "refresh", "dpopKey": "key", + } + data, err := json.Marshal(token) + require.NoError(t, err) + require.NoError(t, os.MkdirAll(filepath.Dir(path), 0o700)) + require.NoError(t, os.WriteFile(path, data, 0o600)) +} + +func TestLoginSessionIsResolvedFromTheInvocationEnvironment(t *testing.T) { + session := "arn:aws:iam::123456789012:user/example" + root := &config.SharedConfig{LoginSession: session, Region: "us-east-1"} + assert.True(t, profileSourceNeedsEnvironment(root), "the SDK would read the login cache from the process environment") + + // The process environment names a different cache that holds other credentials. + processCache := t.TempDir() + writeLoginToken(t, session, processCache, "AKIAPROCESSCACHE") + t.Setenv("AWS_LOGIN_CACHE_DIRECTORY", processCache) + + t.Run("explicit cache directory", func(t *testing.T) { + dir := t.TempDir() + writeLoginToken(t, session, dir, "AKIAINVOCATION") + provider, err := loginSessionCredentials(map[string]string{"AWS_LOGIN_CACHE_DIRECTORY": dir}, root, "") + require.NoError(t, err) + + retrieved, err := provider.Retrieve(t.Context()) + + require.NoError(t, err) + assert.Equal(t, "AKIAINVOCATION", retrieved.AccessKeyID) + }) + + t.Run("invocation home", func(t *testing.T) { + home := t.TempDir() + writeLoginToken(t, session, filepath.Join(home, ".aws", "login", "cache"), "AKIAHOME") + provider, err := loginSessionCredentials(map[string]string{"HOME": home, "USERPROFILE": home}, root, "") + require.NoError(t, err) + + retrieved, err := provider.Retrieve(t.Context()) + + require.NoError(t, err) + assert.Equal(t, "AKIAHOME", retrieved.AccessKeyID, "the process's AWS_LOGIN_CACHE_DIRECTORY is not consulted") + }) + + t.Run("no cached token", func(t *testing.T) { + provider, err := loginSessionCredentials(map[string]string{"AWS_LOGIN_CACHE_DIRECTORY": t.TempDir()}, root, "") + require.NoError(t, err) + + _, err = provider.Retrieve(t.Context()) + + require.Error(t, err) + }) +} + +func TestContainerEndpointClientBoundsRequests(t *testing.T) { + release := make(chan struct{}) + hanging := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + select { + case <-release: + case <-time.After(30 * time.Second): + } + })) + // The handler is released before Close, which waits for outstanding requests. + t.Cleanup(hanging.Close) + t.Cleanup(func() { close(release) }) + + answering := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + _, _ = w.Write([]byte("ok")) + })) + t.Cleanup(answering.Close) + + t.Run("an endpoint that never answers is abandoned after the timeout", func(t *testing.T) { + client := containerEndpointClient(200 * time.Millisecond) + start := time.Now() + resp, err := client.Get(hanging.URL) + if resp != nil { + _ = resp.Body.Close() + } + require.Error(t, err) + assert.Less(t, time.Since(start), 10*time.Second) + }) + + t.Run("an endpoint that answers within the timeout still works", func(t *testing.T) { + client := containerEndpointClient(5 * time.Second) + resp, err := client.Get(answering.URL) + require.NoError(t, err) + require.NoError(t, resp.Body.Close()) + assert.Equal(t, http.StatusOK, resp.StatusCode) + }) +} + +func TestLoginSessionWithoutInvocationHomeDoesNotReadProcessCache(t *testing.T) { + const session = "arn:aws:iam::123456789012:user/example" + processHome := t.TempDir() + processCache := filepath.Join(processHome, ".aws", "login", "cache") + writeLoginToken(t, session, processCache, "AKIAPROCESSCACHE") + t.Setenv("HOME", processHome) + t.Setenv("USERPROFILE", processHome) + t.Setenv("AWS_LOGIN_CACHE_DIRECTORY", processCache) + home := homeWithProfiles(t, "", "[default]\nlogin_session = "+session+"\nregion = us-east-1\n") + env := map[string]string{"AWS_CONFIG_FILE": filepath.Join(home, ".aws", "config")} + + value, err := baseCredentials(t.Context(), env) + require.ErrorIs(t, err, errUtils.ErrAuthenticationFailed) + require.ErrorIs(t, err, errUtils.ErrInvalidAuthConfig) + require.Nil(t, value, "credentials from the process cache must never escape") + + // An explicit invocation cache remains sufficient even without an invocation home. + invocationCache := t.TempDir() + writeLoginToken(t, session, invocationCache, "AKIAINVOCATION") + env["AWS_LOGIN_CACHE_DIRECTORY"] = invocationCache + value, err = baseCredentials(t.Context(), env) + require.NoError(t, err) + require.Equal(t, "AKIAINVOCATION", value.AccessKeyID) + require.Equal(t, processHome, os.Getenv("HOME")) + require.Equal(t, processHome, os.Getenv("USERPROFILE")) + require.Equal(t, processCache, os.Getenv("AWS_LOGIN_CACHE_DIRECTORY")) +} + +func TestExplicitStaticProfileDoesNotRequireInvocationHome(t *testing.T) { + poisoned := poisonProcessEnvironment(t) + home := homeWithProfiles(t, "[default]\naws_access_key_id = invocation-key\naws_secret_access_key = invocation-secret\n", "") + value, err := baseCredentials(t.Context(), map[string]string{ + "AWS_SHARED_CREDENTIALS_FILE": filepath.Join(home, ".aws", "credentials"), + }) + require.NoError(t, err) + require.Equal(t, "invocation-key", value.AccessKeyID) + require.Equal(t, "invocation-secret", value.SecretAccessKey) + assertProcessEnvironmentUnchanged(t, poisoned) +} diff --git a/pkg/auth/identities/aws/inline_role_credentials.go b/pkg/auth/identities/aws/inline_role_credentials.go new file mode 100644 index 00000000000..483166f9223 --- /dev/null +++ b/pkg/auth/identities/aws/inline_role_credentials.go @@ -0,0 +1,598 @@ +package aws + +import ( + "cmp" + "context" + "crypto/sha1" //nolint:gosec // The SDK and the AWS CLI name SSO token cache files after the SHA-1 of the cache key. + "encoding/hex" + "errors" + "fmt" + "maps" + "net" + "net/http" + "net/url" + "os" + "os/exec" + "path/filepath" + "runtime" + "slices" + "strconv" + "strings" + "time" + + "github.com/aws/aws-sdk-go-v2/aws" + "github.com/aws/aws-sdk-go-v2/config" + "github.com/aws/aws-sdk-go-v2/credentials" + "github.com/aws/aws-sdk-go-v2/credentials/ec2rolecreds" + "github.com/aws/aws-sdk-go-v2/credentials/endpointcreds" + "github.com/aws/aws-sdk-go-v2/credentials/logincreds" + "github.com/aws/aws-sdk-go-v2/credentials/processcreds" + "github.com/aws/aws-sdk-go-v2/credentials/ssocreds" + "github.com/aws/aws-sdk-go-v2/credentials/stscreds" + "github.com/aws/aws-sdk-go-v2/feature/ec2/imds" + "github.com/aws/aws-sdk-go-v2/service/signin" + "github.com/aws/aws-sdk-go-v2/service/sts" + + errUtils "github.com/cloudposse/atmos/errors" + "github.com/cloudposse/atmos/pkg/auth/types" +) + +// The pinned AWS SDK (config v1.33) reads its credential chain inputs with os.Getenv and offers no way to +// pass another environment. The base credentials of an inline role must come from the invocation +// environment, and changing the process environment would leak into concurrent invocations. So the chain is +// assembled here, in the SDK's order, from the invocation environment alone: static keys, a web identity +// token file, the shared profile, container credentials, then instance metadata. Only the profile sources +// that the shared files fully describe (keys, web identity, SSO, and login) are left to the SDK, with every +// input that selects them passed as an explicit option, including the SSO token cache of the invocation home. +// A credential_process runs from here too, because the SDK starts it with the environment of the process. + +const ( + credentialSourceEnvironment = "Environment" + credentialSourceContainer = "EcsContainer" + credentialSourceIMDS = "Ec2InstanceMetadata" + + defaultProfileName = "default" + ecsContainerEndpoint = "http://169.254.170.2" + imdsDefaultIPv4Endpoint = "http://169.254.169.254" + imdsDefaultIPv6Endpoint = "http://[fd00:ec2::254]" + containerCredentialExpiry = 5 * time.Minute + // Bounds on a container credential request, so an endpoint that accepts a connection and never answers + // cannot block the caller after its own deadline has passed. + containerCredentialDialTimeout = 3 * time.Second + containerCredentialRequestTimeout = 10 * time.Second + // The SDK passes duration_seconds on only when it exceeds 15 whole minutes; shorter values use the STS default. + minimumProfileRoleDuration = 16 * time.Minute +) + +// awsConfigDirName is the directory of the AWS shared files and caches below a home directory. +const awsConfigDirName = ".aws" + +// containerHosts are the ECS and EKS container credential hosts that plain HTTP may reach. +var containerHosts = []net.IP{net.ParseIP("169.254.170.2"), net.ParseIP("169.254.170.23"), net.ParseIP("fd00:ec2::23")} + +// errCredentialProcess hides broker output from authentication errors. +var errCredentialProcess = errors.New("credential_process failed to supply valid credentials") + +// errNoInvocationCredentials reports an invocation environment that selects a source with nothing in it. +var errNoInvocationCredentials = errors.New("the invocation environment provides no credentials for the selected source") + +// baseCredentials resolves the credentials that sign the AssumeRole request. +func baseCredentials(ctx context.Context, env map[string]string) (*types.AWSCredentials, error) { + region := cmp.Or(env["AWS_REGION"], env["AWS_DEFAULT_REGION"]) + provider, profileRegion, err := baseCredentialsProvider(ctx, env, region) + if err != nil { + return nil, fmt.Errorf("%w: no AWS credentials to assume a role with: %w", errUtils.ErrAuthenticationFailed, err) + } + retrieved, err := aws.NewCredentialsCache(provider).Retrieve(ctx) + if err != nil { + return nil, fmt.Errorf("%w: no AWS credentials to assume a role with: %w", errUtils.ErrAuthenticationFailed, err) + } + return &types.AWSCredentials{ + AccessKeyID: retrieved.AccessKeyID, + SecretAccessKey: retrieved.SecretAccessKey, + SessionToken: retrieved.SessionToken, + Region: cmp.Or(region, profileRegion), + }, nil +} + +// baseCredentialsProvider selects the first source of the SDK default chain that the invocation +// environment configures. It also returns the region of the profile it read, if any. +func baseCredentialsProvider(ctx context.Context, env map[string]string, region string) (aws.CredentialsProvider, string, error) { + if key, secret := invocationEnvironmentKeys(env); key != "" && secret != "" { + return staticEnvCredentials(env), "", nil + } + if tokenFile := env["AWS_WEB_IDENTITY_TOKEN_FILE"]; tokenFile != "" { + provider, err := envWebIdentityCredentials(env, region, tokenFile) + return provider, "", err + } + profile, err := loadInvocationProfile(ctx, env) + if err != nil { + return nil, "", err + } + if profile == nil { + provider, err := containerOrInstanceCredentials(env, nil) + return provider, "", err + } + region = cmp.Or(region, profile.Region) + chain := profileChain(profile) + if root := chain[len(chain)-1]; !profileSourceNeedsEnvironment(root) { + provider, err := sdkProfileCredentials(ctx, env, profile.Profile, region, root) + return provider, profile.Region, err + } + provider, err := envProfileCredentials(env, chain, region) + return provider, profile.Region, err +} + +func staticEnvCredentials(env map[string]string) aws.CredentialsProvider { + key, secret := invocationEnvironmentKeys(env) + return credentials.NewStaticCredentialsProvider(key, secret, env["AWS_SESSION_TOKEN"]) +} + +// envWebIdentityCredentials exchanges the web identity token file that the environment names. +func envWebIdentityCredentials(env map[string]string, region, tokenFile string) (aws.CredentialsProvider, error) { + roleARN := env["AWS_ROLE_ARN"] + if roleARN == "" { + return nil, fmt.Errorf("%w: AWS_WEB_IDENTITY_TOKEN_FILE is set without AWS_ROLE_ARN", errNoInvocationCredentials) + } + sessionName := env["AWS_ROLE_SESSION_NAME"] + return stscreds.NewWebIdentityRoleProvider(invocationSTSClient(env, region, nil), roleARN, stscreds.IdentityTokenFile(tokenFile), + func(o *stscreds.WebIdentityRoleOptions) { o.RoleSessionName = sessionName }), nil +} + +// loadInvocationProfile reads the profile that the environment selects from the files that it names. A +// missing default profile is not an error and returns nil, as in the SDK; a missing named profile is. +func loadInvocationProfile(ctx context.Context, env map[string]string) (*config.SharedConfig, error) { + selected := cmp.Or(env["AWS_PROFILE"], env["AWS_DEFAULT_PROFILE"]) + name := cmp.Or(selected, defaultProfileName) + configFile, credentialsFile := invocationSharedFiles(env) + profile, err := config.LoadSharedConfigProfile(ctx, name, func(o *config.LoadSharedConfigOptions) { + o.ConfigFiles = []string{configFile} + o.CredentialsFiles = []string{credentialsFile} + }) + var notExist config.SharedConfigProfileNotExistError + switch { + case err == nil: + return &profile, nil + case errors.As(err, ¬Exist) && selected == "": + return nil, nil + default: + return nil, err + } +} + +// invocationSharedFiles returns the shared config and credentials files of the environment, defaulting to +// the files in the home directory that the environment names. +func invocationSharedFiles(env map[string]string) (configFile, credentialsFile string) { + configFile, credentialsFile = env["AWS_CONFIG_FILE"], env["AWS_SHARED_CREDENTIALS_FILE"] + home := invocationHomeDir(env) + if configFile == "" { + configFile = os.DevNull + if home != "" { + configFile = filepath.Join(home, awsConfigDirName, "config") + } + } + if credentialsFile == "" { + credentialsFile = os.DevNull + if home != "" { + credentialsFile = filepath.Join(home, awsConfigDirName, "credentials") + } + } + return configFile, credentialsFile +} + +// invocationHomeDir returns the home directory that the environment names, or "". +func invocationHomeDir(env map[string]string) string { + if runtime.GOOS == "windows" { + return cmp.Or(env["USERPROFILE"], env["HOME"]) + } + return env["HOME"] +} + +// profileChain lists a profile and its source profiles, ending with the profile that holds the source. +func profileChain(profile *config.SharedConfig) []*config.SharedConfig { + chain := []*config.SharedConfig{profile} + for current := profile; current.Source != nil; current = current.Source { + chain = append(chain, current.Source) + } + return chain +} + +// profileSourceNeedsEnvironment reports whether the profile's starting credentials come from the invocation +// environment rather than from the shared files alone: a credential_source, a credential_process that runs +// with that environment, or no source at all so that container and instance credentials apply. +func profileSourceNeedsEnvironment(root *config.SharedConfig) bool { + switch { + case root.Credentials.HasKeys(), root.WebIdentityTokenFile != "": + return false + case root.LoginSession != "": + // The SDK reads the login cache directory from the process environment, so the invocation's own + // AWS_LOGIN_CACHE_DIRECTORY and home would be ignored. + return true + case root.SSOSessionName != "", root.SSORegion != "", root.SSOAccountID != "", root.SSOStartURL != "", root.SSORoleName != "": + return false + default: + return true + } +} + +// profileSourceUsesSTSOnly reports whether the SDK builds only STS clients from the configuration that +// resolves a profile chain with this root, so that the STS endpoint can be the configuration's endpoint. +func profileSourceUsesSTSOnly(root *config.SharedConfig) bool { + return root.SSOSessionName == "" && root.SSORegion == "" && root.SSOAccountID == "" && root.SSOStartURL == "" && + root.SSORoleName == "" && root.LoginSession == "" +} + +// sdkProfileCredentials lets the SDK resolve a profile whose source the shared files describe. The profile, +// files, region, and SSO token cache are explicit, so the SDK does not consult the process environment to +// choose them. The root is the profile that holds the source of the chain. +func sdkProfileCredentials(ctx context.Context, env map[string]string, profile, region string, root *config.SharedConfig) (aws.CredentialsProvider, error) { + // Keys and web identity need no token cache; the remaining SDK-managed source is SSO. + if !root.Credentials.HasKeys() && root.WebIdentityTokenFile == "" && invocationHomeDir(env) == "" { + return nil, fmt.Errorf("%w: SSO token cache requires an invocation home directory", errUtils.ErrInvalidAuthConfig) + } + configFile, credentialsFile := invocationSharedFiles(env) + options := []func(*config.LoadOptions) error{ + config.WithSharedConfigProfile(profile), + config.WithSharedConfigFiles([]string{configFile}), + config.WithSharedCredentialsFiles([]string{credentialsFile}), + config.WithWebIdentityRoleCredentialOptions(func(o *stscreds.WebIdentityRoleOptions) { + o.Client = invocationSTSClient(env, region, nil) + }), + } + if region != "" { + options = append(options, config.WithRegion(region)) + } + options = append(options, ssoTokenCacheOptions(env, root)...) + // The SDK builds the STS client of a role_arn and source_profile chain itself, so the invocation STS + // endpoint is set as the endpoint of the loaded configuration. Only STS clients are built from it when + // the chain starts from keys, a web identity, or a credential process; SSO and login chains also build + // clients of other services, which this endpoint must not reach. + if endpoint := invocationSTSEndpoint(env); endpoint != "" && profileSourceUsesSTSOnly(root) { + options = append(options, config.WithBaseEndpoint(endpoint)) + } + cfg, err := config.LoadDefaultConfig(ctx, options...) + if err != nil { + return nil, fmt.Errorf("%w: %w", errUtils.ErrLoadAWSConfig, err) + } + return cfg.Credentials, nil +} + +// ssoTokenCacheOptions points the SDK at the SSO token cache of the invocation home. The SDK derives the +// cache path from the home directory of the process, and a session profile and a legacy profile each have +// an option for it: a session's token provider is keyed by the session name, a legacy profile by its start +// URL, as in the AWS CLI. Anything else leaves the SDK's own resolution untouched. +func ssoTokenCacheOptions(env map[string]string, root *config.SharedConfig) []func(*config.LoadOptions) error { + home := invocationHomeDir(env) + switch { + case home == "": + return nil + case root.SSOSessionName != "": + path := ssoCachedTokenPath(home, root.SSOSessionName) + return []func(*config.LoadOptions) error{config.WithSSOTokenProviderOptions(func(o *ssocreds.SSOTokenProviderOptions) { + o.CachedTokenFilepath = path + })} + case root.SSOStartURL != "": + path := ssoCachedTokenPath(home, root.SSOStartURL) + return []func(*config.LoadOptions) error{config.WithSSOProviderOptions(func(o *ssocreds.Options) { + o.CachedTokenFilepath = path + })} + default: + return nil + } +} + +// ssoCachedTokenPath is the cache file of an SSO login: ~/.aws/sso/cache/.json. +func ssoCachedTokenPath(home, key string) string { + sum := sha1.Sum([]byte(key)) //nolint:gosec // The cache file name is defined by the AWS CLI, not chosen for security. + return filepath.Join(home, awsConfigDirName, "sso", "cache", hex.EncodeToString(sum[:])+".json") +} + +// envProfileCredentials resolves the source of a profile from the environment, then assumes each role of +// the chain from the source profile outward, as the SDK does. +func envProfileCredentials(env map[string]string, chain []*config.SharedConfig, region string) (aws.CredentialsProvider, error) { + provider, err := profileSourceCredentials(env, chain, region) + if err != nil { + return nil, err + } + for i := len(chain) - 1; i >= 0; i-- { + hop := chain[i] + if hop.RoleARN == "" { + continue + } + if hop.MFASerial != "" { + return nil, config.AssumeRoleTokenProviderNotSetError{} + } + provider = aws.NewCredentialsCache(stscreds.NewAssumeRoleProvider(invocationSTSClient(env, region, provider), hop.RoleARN, + func(o *stscreds.AssumeRoleOptions) { applyProfileRoleOptions(o, hop) })) + } + return provider, nil +} + +func applyProfileRoleOptions(options *stscreds.AssumeRoleOptions, profile *config.SharedConfig) { + options.RoleSessionName = profile.RoleSessionName + if profile.RoleDurationSeconds != nil && *profile.RoleDurationSeconds >= minimumProfileRoleDuration { + options.Duration = *profile.RoleDurationSeconds + } + if profile.ExternalID != "" { + options.ExternalID = aws.String(profile.ExternalID) + } +} + +// profileSourceCredentials resolves a credential_source, a credential_process, or the container and instance +// fallback, for the source at the end of the profile chain. The first profile of the chain is the one that +// the invocation selected, and it supplies the instance metadata settings, as in the SDK. +func profileSourceCredentials(env map[string]string, chain []*config.SharedConfig, region string) (aws.CredentialsProvider, error) { + root, selected := chain[len(chain)-1], chain[0] + switch { + case root.LoginSession != "": + return loginSessionCredentials(env, root, region) + case root.CredentialSource == "": + return unnamedSourceCredentials(env, root, selected) + default: + return namedSourceCredentials(env, root.CredentialSource, selected) + } +} + +// namedSourceCredentials resolves the credential_source of a profile source: the invocation environment, a +// container endpoint, or the instance metadata service. +func namedSourceCredentials(env map[string]string, source string, selected *config.SharedConfig) (aws.CredentialsProvider, error) { + switch source { + case credentialSourceEnvironment: + if key, secret := invocationEnvironmentKeys(env); key == "" || secret == "" { + return nil, fmt.Errorf("%w: credential_source %s", errNoInvocationCredentials, credentialSourceEnvironment) + } + return staticEnvCredentials(env), nil + case credentialSourceContainer: + provider, ok, err := containerCredentials(env) + if err == nil && !ok { + err = fmt.Errorf("%w: credential_source %s needs AWS_CONTAINER_CREDENTIALS_RELATIVE_URI or AWS_CONTAINER_CREDENTIALS_FULL_URI", + errNoInvocationCredentials, credentialSourceContainer) + } + return provider, err + case credentialSourceIMDS: + return instanceCredentials(env, selected), nil + default: + return nil, fmt.Errorf("%w: credential_source values must be %s, %s, or %s", errUtils.ErrInvalidAuthConfig, + credentialSourceContainer, credentialSourceIMDS, credentialSourceEnvironment) + } +} + +// loginSessionCredentials resolves an `aws login` session from the invocation environment: the token cache is +// the directory that AWS_LOGIN_CACHE_DIRECTORY names, or the login cache below the invocation home. +func loginSessionCredentials(env map[string]string, root *config.SharedConfig, region string) (aws.CredentialsProvider, error) { + dir := env["AWS_LOGIN_CACHE_DIRECTORY"] + if home := invocationHomeDir(env); dir == "" && home != "" { + dir = filepath.Join(home, awsConfigDirName, "login", "cache") + } + if dir == "" { + return nil, fmt.Errorf("%w: login_session requires an invocation home or AWS_LOGIN_CACHE_DIRECTORY", errUtils.ErrInvalidAuthConfig) + } + path, err := logincreds.StandardCachedTokenFilepath(root.LoginSession, dir) + if err != nil { + return nil, fmt.Errorf("%w: %w", errUtils.ErrLoadAWSConfig, err) + } + client := signin.New(signin.Options{Region: cmp.Or(region, root.Region)}) + return aws.NewCredentialsCache(logincreds.New(client, path)), nil +} + +// unnamedSourceCredentials resolves a profile source that names no credential_source: its credential_process +// when it has one, and the container and instance credentials otherwise. +func unnamedSourceCredentials(env map[string]string, root, selected *config.SharedConfig) (aws.CredentialsProvider, error) { + if root.CredentialProcess != "" { + return processCredentials(env, root.CredentialProcess), nil + } + return containerOrInstanceCredentials(env, selected) +} + +func containerOrInstanceCredentials(env map[string]string, profile *config.SharedConfig) (aws.CredentialsProvider, error) { + provider, ok, err := containerCredentials(env) + if err != nil || ok { + return provider, err + } + return instanceCredentials(env, profile), nil +} + +// processCredentials runs a profile's credential_process with the invocation environment. The SDK starts it +// with the environment of the process, so the command is built here; the SDK's builder still supplies the +// shell, stderr, and stdin, and its provider the output parsing, timeout, and expiry. +func processCredentials(env map[string]string, command string) aws.CredentialsProvider { + environ := make([]string, 0, len(env)) + for _, key := range slices.Sorted(maps.Keys(env)) { + environ = append(environ, key+"="+env[key]) + } + builder := processcreds.NewCommandBuilderFunc(func(ctx context.Context) (*exec.Cmd, error) { + cmd, err := processcreds.DefaultNewCommandBuilder{Args: []string{command}}.NewCommand(ctx) + if err != nil { + return nil, err + } + cmd.Env = environ + return cmd, nil + }) + provider := processcreds.NewProviderCommand(builder) + return aws.NewCredentialsCache(aws.CredentialsProviderFunc(func(ctx context.Context) (aws.Credentials, error) { + value, err := provider.Retrieve(ctx) + if err != nil { + // SDK parse errors embed broker stdout, which can contain unregistered secrets. + // Never retain that error in the wrapping chain. + if ctx.Err() != nil { + return aws.Credentials{}, ctx.Err() + } + return aws.Credentials{}, errCredentialProcess + } + return value, nil + })) +} + +// containerCredentials returns the container credential endpoint that the environment names, if any. Like +// the SDK, it accepts a plain HTTP full URI only on a loopback or container metadata host. +func containerCredentials(env map[string]string) (aws.CredentialsProvider, bool, error) { + endpoint := env["AWS_CONTAINER_CREDENTIALS_FULL_URI"] + if relative := env["AWS_CONTAINER_CREDENTIALS_RELATIVE_URI"]; relative != "" { + endpoint = ecsContainerEndpoint + relative + } else if endpoint == "" { + return nil, false, nil + } else if err := checkContainerEndpoint(endpoint); err != nil { + return nil, false, err + } + token, tokenFile := env["AWS_CONTAINER_AUTHORIZATION_TOKEN"], env["AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE"] + provider := endpointcreds.New(endpoint, func(o *endpointcreds.Options) { + if strings.HasPrefix(endpoint, "http://") { + o.HTTPClient = containerEndpointClient(containerCredentialRequestTimeout) + } + o.AuthorizationToken = token + if tokenFile != "" { + o.AuthorizationTokenProvider = endpointcreds.TokenProviderFunc(func() (string, error) { + contents, err := os.ReadFile(tokenFile) + if err != nil { + return "", fmt.Errorf("read the container authorization token file %s: %w", tokenFile, err) + } + return string(contents), nil + }) + } + }) + return aws.NewCredentialsCache(provider, func(o *aws.CredentialsCacheOptions) { o.ExpiryWindow = containerCredentialExpiry }), true, nil +} + +// containerEndpointClient is the HTTP client for a plain-HTTP container credential endpoint. It dials only an +// address that passed the same allowlist as the endpoint URL, and it never uses a proxy, so a host name that +// resolves differently after the early check cannot redirect the request. The request as a whole is bounded by +// timeout, and establishing the connection by containerCredentialDialTimeout. +func containerEndpointClient(timeout time.Duration) *http.Client { + dialer := &net.Dialer{Timeout: containerCredentialDialTimeout} + return &http.Client{ + Timeout: timeout, + Transport: &http.Transport{ + Proxy: nil, + DisableKeepAlives: true, // Providers are short-lived; retain no idle connections. + DialContext: validatedContainerDialer(net.DefaultResolver.LookupHost, dialer.DialContext), + TLSHandshakeTimeout: containerCredentialDialTimeout, + ResponseHeaderTimeout: timeout, + }, + } +} + +// dialFunc opens a network connection. +type dialFunc func(ctx context.Context, network, address string) (net.Conn, error) + +// validatedContainerDialer resolves the host when it connects and dials one of the validated addresses itself, +// rather than handing the name to the network stack to resolve again. +func validatedContainerDialer(lookup func(ctx context.Context, host string) ([]string, error), dial dialFunc) dialFunc { + return func(ctx context.Context, network, address string) (net.Conn, error) { + host, port, err := net.SplitHostPort(address) + if err != nil { + return nil, fmt.Errorf("%w: container credentials address %q: %w", errUtils.ErrInvalidAuthConfig, address, err) + } + addresses := []string{host} + if net.ParseIP(host) == nil { + if addresses, err = lookup(ctx, host); err != nil { + return nil, fmt.Errorf("%w: resolve container credentials host %q: %w", errUtils.ErrInvalidAuthConfig, host, err) + } + } + for _, candidate := range addresses { + if ip := net.ParseIP(candidate); ip == nil || !containerHostAllowed(ip) { + return nil, fmt.Errorf("%w: container credentials host %q is not a loopback, ECS, or EKS host", errUtils.ErrInvalidAuthConfig, host) + } + } + var dialErr error + for _, candidate := range addresses { + conn, err := dial(ctx, network, net.JoinHostPort(candidate, port)) + if err == nil { + return conn, nil + } + dialErr = err + } + return nil, dialErr + } +} + +func checkContainerEndpoint(endpoint string) error { + parsed, err := url.Parse(endpoint) + if err != nil { + return fmt.Errorf("%w: invalid container credentials URL: %w", errUtils.ErrInvalidAuthConfig, err) + } + host := parsed.Hostname() + if host == "" { + return fmt.Errorf("%w: no host in the container credentials URL", errUtils.ErrInvalidAuthConfig) + } + if parsed.Scheme != "http" { + return nil + } + addresses := []string{host} + if net.ParseIP(host) == nil { + if addresses, err = net.LookupHost(host); err != nil { + return fmt.Errorf("%w: resolve container credentials host %q: %w", errUtils.ErrInvalidAuthConfig, host, err) + } + } + for _, address := range addresses { + if ip := net.ParseIP(address); ip == nil || !containerHostAllowed(ip) { + return fmt.Errorf("%w: container credentials host %q is not a loopback, ECS, or EKS host", errUtils.ErrInvalidAuthConfig, host) + } + } + return nil +} + +func containerHostAllowed(ip net.IP) bool { + return ip.IsLoopback() || slices.ContainsFunc(containerHosts, ip.Equal) +} + +// instanceCredentials reads the instance role credentials. Every client setting is explicit, because the +// metadata client otherwise falls back to the process environment. The invocation environment decides over +// the selected profile, which may be nil, as in the SDK. +func instanceCredentials(env map[string]string, profile *config.SharedConfig) aws.CredentialsProvider { + options := imds.Options{ClientEnableState: imds.ClientEnabled, Endpoint: imdsEndpoint(env, profile)} + if strings.EqualFold(env["AWS_EC2_METADATA_DISABLED"], "true") { + options.ClientEnableState = imds.ClientDisabled + } + if imdsV1Disabled(env, profile) { + options.EnableFallback = aws.FalseTernary + } + return aws.NewCredentialsCache(ec2rolecreds.New(func(o *ec2rolecreds.Options) { o.Client = imds.New(options) })) +} + +// imdsEndpoint returns the metadata endpoint: an explicit endpoint wins over an endpoint mode, and the +// invocation environment wins over the profile at each step. +func imdsEndpoint(env map[string]string, profile *config.SharedConfig) string { + profileEndpoint, profileMode := "", imds.EndpointModeStateUnset + if profile != nil { + profileEndpoint, profileMode = profile.EC2IMDSEndpoint, profile.EC2IMDSEndpointMode + } + if endpoint := cmp.Or(env["AWS_EC2_METADATA_SERVICE_ENDPOINT"], profileEndpoint); endpoint != "" { + return endpoint + } + ipv6 := profileMode == imds.EndpointModeStateIPv6 + if mode := env["AWS_EC2_METADATA_SERVICE_ENDPOINT_MODE"]; mode != "" { + ipv6 = strings.EqualFold(mode, "IPv6") + } + if ipv6 { + return imdsDefaultIPv6Endpoint + } + return imdsDefaultIPv4Endpoint +} + +// imdsV1Disabled reports whether the fallback to IMDSv1 is restricted: a boolean in the invocation +// environment decides, and the selected profile's ec2_metadata_v1_disabled applies otherwise. +func imdsV1Disabled(env map[string]string, profile *config.SharedConfig) bool { + if disabled, err := strconv.ParseBool(env["AWS_EC2_METADATA_V1_DISABLED"]); err == nil { + return disabled + } + return profile != nil && profile.EC2IMDSv1Disabled != nil && *profile.EC2IMDSv1Disabled +} + +// invocationSTSClient builds an STS client from the environment's region and STS endpoint only. +func invocationSTSClient(env map[string]string, region string, credentials aws.CredentialsProvider) *sts.Client { + options := sts.Options{Region: region, Credentials: credentials} + applyInvocationSTSEndpoint(env, &options) + return sts.New(options) +} + +// applyInvocationSTSEndpoint sets the STS endpoint that the invocation environment names, if any. +func applyInvocationSTSEndpoint(env map[string]string, options *sts.Options) { + if endpoint := invocationSTSEndpoint(env); endpoint != "" { + options.BaseEndpoint = aws.String(endpoint) + } +} + +// invocationSTSEndpoint returns the STS endpoint that the invocation environment names, or "". +func invocationSTSEndpoint(env map[string]string) string { + return cmp.Or(env["AWS_ENDPOINT_URL_STS"], env["AWS_ENDPOINT_URL"]) +} diff --git a/pkg/auth/identities/aws/inline_role_credentials_test.go b/pkg/auth/identities/aws/inline_role_credentials_test.go new file mode 100644 index 00000000000..d44c1252b22 --- /dev/null +++ b/pkg/auth/identities/aws/inline_role_credentials_test.go @@ -0,0 +1,575 @@ +package aws + +import ( + "context" + "crypto/sha1" + "encoding/hex" + "encoding/json" + "errors" + "fmt" + "io" + "net" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "runtime" + "strings" + "sync" + "sync/atomic" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + errUtils "github.com/cloudposse/atmos/errors" +) + +// poisonProcessEnvironment fills the process environment with a credential source of every kind. None may +// reach an invocation that does not name it, and none may be changed by resolving one. +func poisonProcessEnvironment(t *testing.T) map[string]string { + t.Helper() + dir := t.TempDir() + tokenFile := filepath.Join(dir, "process-token") + require.NoError(t, os.WriteFile(tokenFile, []byte("process-web-identity-token"), 0o600)) + credentialsFile := filepath.Join(dir, "credentials") + require.NoError(t, os.WriteFile(credentialsFile, + []byte("[default]\naws_access_key_id = AKIAPROCESSDEFAULT\naws_secret_access_key = s\n[process]\naws_access_key_id = AKIAPROCESSPROFILE\naws_secret_access_key = s\n"), 0o600)) + poisoned := map[string]string{ + "AWS_ACCESS_KEY_ID": "AKIAPROCESSKEY", + "AWS_SECRET_ACCESS_KEY": "process-secret", + "AWS_PROFILE": "process", + "AWS_SHARED_CREDENTIALS_FILE": credentialsFile, + "AWS_CONFIG_FILE": filepath.Join(dir, "config"), + "AWS_WEB_IDENTITY_TOKEN_FILE": tokenFile, + "AWS_ROLE_ARN": "arn:aws:iam::9:role/process", + "AWS_CONTAINER_CREDENTIALS_FULL_URI": "http://127.0.0.1:1/process", + "AWS_EC2_METADATA_DISABLED": "true", + "AWS_REGION": "eu-north-1", + } + for key, value := range poisoned { + t.Setenv(key, value) + } + return poisoned +} + +func assertProcessEnvironmentUnchanged(t *testing.T, poisoned map[string]string) { + t.Helper() + for key, value := range poisoned { + assert.Equal(t, value, os.Getenv(key), "the process environment is not changed: %s", key) + } +} + +// homeWithProfiles makes a home directory whose default shared files hold the given contents. +func homeWithProfiles(t *testing.T, credentials, config string) string { + t.Helper() + home := t.TempDir() + require.NoError(t, os.MkdirAll(filepath.Join(home, ".aws"), 0o700)) + require.NoError(t, os.WriteFile(filepath.Join(home, ".aws", "credentials"), []byte(credentials), 0o600)) + require.NoError(t, os.WriteFile(filepath.Join(home, ".aws", "config"), []byte(config), 0o600)) + return home +} + +func invocationHome(home string) []string { + return []string{"HOME=" + home, "USERPROFILE=" + home} +} + +// credentialServer serves container credentials, recording the authorization header of each request. +func credentialServer(t *testing.T, key string) (*httptest.Server, *[]string) { + t.Helper() + var mu sync.Mutex + var authorizations []string + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + mu.Lock() + authorizations = append(authorizations, r.Header.Get("Authorization")) + mu.Unlock() + require.NoError(t, json.NewEncoder(w).Encode(map[string]string{ + "AccessKeyId": key, "SecretAccessKey": "container-secret", "Token": "container-token", "Expiration": "2099-01-01T00:00:00Z", + })) + })) + t.Cleanup(server.Close) + return server, &authorizations +} + +// metadataServer emulates the instance metadata role credentials API. +func metadataServer(t *testing.T, key string) (*httptest.Server, *atomic.Int32) { + t.Helper() + var requests atomic.Int32 + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + requests.Add(1) + switch { + case r.Method == http.MethodPut && r.URL.Path == "/latest/api/token": + w.Header().Set("X-Aws-Ec2-Metadata-Token-Ttl-Seconds", "21600") + fmt.Fprint(w, "imds-session-token") + case r.URL.Path == "/latest/meta-data/iam/security-credentials/": + fmt.Fprint(w, "instance-role") + case r.URL.Path == "/latest/meta-data/iam/security-credentials/instance-role": + require.NoError(t, json.NewEncoder(w).Encode(map[string]string{ + "Code": "Success", "Type": "AWS-HMAC", "AccessKeyId": key, "SecretAccessKey": "instance-secret", + "Token": "instance-token", "Expiration": "2099-01-01T00:00:00Z", "LastUpdated": "2026-01-01T00:00:00Z", + })) + default: + http.NotFound(w, r) + } + })) + t.Cleanup(server.Close) + return server, &requests +} + +func resolveBase(t *testing.T, environ ...string) (string, string, error) { + t.Helper() + credentials, err := baseCredentials(context.Background(), environMap(environ)) + if err != nil { + return "", "", err + } + return credentials.AccessKeyID, credentials.Region, nil +} + +func TestBaseCredentialsComeFromTheInvocationProfileNotTheProcess(t *testing.T) { + poisoned := poisonProcessEnvironment(t) + home := homeWithProfiles(t, "[default]\naws_access_key_id = AKIAINVOCATIONDEFAULT\naws_secret_access_key = s\n", "[default]\nregion = us-west-2\n") + + key, region, err := resolveBase(t, invocationHome(home)...) + require.NoError(t, err) + assert.Equal(t, "AKIAINVOCATIONDEFAULT", key, "the invocation's default profile is used, not the process keys, profile, or files") + assert.Equal(t, "us-west-2", region, "the profile region is used, not the process region") + assertProcessEnvironmentUnchanged(t, poisoned) +} + +func TestBaseCredentialsFromTheInvocationWebIdentity(t *testing.T) { + poisoned := poisonProcessEnvironment(t) + sts := newSTSRecorder(t) + tokenFile := filepath.Join(t.TempDir(), "token") + require.NoError(t, os.WriteFile(tokenFile, []byte("invocation-web-identity-token"), 0o600)) + + key, _, err := resolveBase(t, "AWS_WEB_IDENTITY_TOKEN_FILE="+tokenFile, "AWS_ROLE_ARN=arn:aws:iam::1:role/irsa", + "AWS_ROLE_SESSION_NAME=pod", "AWS_REGION=us-east-1", "AWS_ENDPOINT_URL_STS="+sts.server.URL, "HOME="+t.TempDir()) + require.NoError(t, err) + assert.Equal(t, assumedKey, key) + form, _ := sts.last(t) + assert.Equal(t, "AssumeRoleWithWebIdentity", form.Get("Action")) + assert.Equal(t, "invocation-web-identity-token", form.Get("WebIdentityToken")) + assert.Equal(t, "arn:aws:iam::1:role/irsa", form.Get("RoleArn")) + assert.Equal(t, "pod", form.Get("RoleSessionName")) + assertProcessEnvironmentUnchanged(t, poisoned) + + _, _, err = resolveBase(t, "AWS_WEB_IDENTITY_TOKEN_FILE="+tokenFile) + require.ErrorIs(t, err, errNoInvocationCredentials, "a token file without a role is rejected") +} + +func TestBaseCredentialsFromTheInvocationContainer(t *testing.T) { + poisoned := poisonProcessEnvironment(t) + server, authorizations := credentialServer(t, "AKIACONTAINER") + empty := t.TempDir() + + key, _, err := resolveBase(t, "HOME="+empty, "AWS_CONTAINER_CREDENTIALS_FULL_URI="+server.URL+"/creds", + "AWS_CONTAINER_AUTHORIZATION_TOKEN=inline-token") + require.NoError(t, err) + assert.Equal(t, "AKIACONTAINER", key) + + tokenFile := filepath.Join(empty, "token") + require.NoError(t, os.WriteFile(tokenFile, []byte("file-token"), 0o600)) + _, _, err = resolveBase(t, "HOME="+empty, "AWS_CONTAINER_CREDENTIALS_FULL_URI="+server.URL+"/creds", + "AWS_CONTAINER_AUTHORIZATION_TOKEN=inline-token", "AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE="+tokenFile) + require.NoError(t, err) + assert.Equal(t, []string{"inline-token", "file-token"}, *authorizations, "the token file named by the invocation overrides the inline token") + assertProcessEnvironmentUnchanged(t, poisoned) + + _, _, err = resolveBase(t, "HOME="+empty, "AWS_CONTAINER_CREDENTIALS_FULL_URI=http://192.0.2.10/creds") + require.ErrorIs(t, err, errUtils.ErrInvalidAuthConfig, "plain HTTP is accepted only on loopback and container hosts") +} + +func TestBaseCredentialsFromTheInvocationInstanceMetadata(t *testing.T) { + poisoned := poisonProcessEnvironment(t) + server, requests := metadataServer(t, "AKIAINSTANCE") + empty := t.TempDir() + + key, _, err := resolveBase(t, "HOME="+empty, "AWS_EC2_METADATA_SERVICE_ENDPOINT="+server.URL) + require.NoError(t, err, "the process AWS_EC2_METADATA_DISABLED does not disable the invocation's metadata client") + assert.Equal(t, "AKIAINSTANCE", key) + + before := requests.Load() + _, _, err = resolveBase(t, "HOME="+empty, "AWS_EC2_METADATA_SERVICE_ENDPOINT="+server.URL, "AWS_EC2_METADATA_DISABLED=true") + require.ErrorIs(t, err, errUtils.ErrAuthenticationFailed, "the invocation disables the metadata client, and the process keys are not a fallback") + assert.Equal(t, before, requests.Load(), "a disabled metadata client sends no request") + assertProcessEnvironmentUnchanged(t, poisoned) +} + +func TestBaseCredentialsFromAProfileCredentialSource(t *testing.T) { + poisonProcessEnvironment(t) + container, _ := credentialServer(t, "AKIACONTAINERSOURCE") + sts := newSTSRecorder(t) + home := homeWithProfiles(t, "", "[profile ecs]\nrole_arn = arn:aws:iam::1:role/from-ecs\ncredential_source = EcsContainer\nrole_session_name = ecs-hop\nexternal_id = ext\nduration_seconds = 3600\n"+ + "[profile env]\nrole_arn = arn:aws:iam::1:role/from-env\ncredential_source = Environment\n"+ + "[profile bogus]\nrole_arn = arn:aws:iam::1:role/r\ncredential_source = Nowhere\n") + + key, _, err := resolveBase(t, "HOME="+home, "AWS_PROFILE=ecs", "AWS_REGION=us-east-1", "AWS_ENDPOINT_URL_STS="+sts.server.URL, + "AWS_CONTAINER_CREDENTIALS_FULL_URI="+container.URL) + require.NoError(t, err) + assert.Equal(t, assumedKey, key) + form, authorization := sts.last(t) + assert.Equal(t, "AssumeRole", form.Get("Action")) + assert.Equal(t, "arn:aws:iam::1:role/from-ecs", form.Get("RoleArn")) + assert.Equal(t, "ecs-hop", form.Get("RoleSessionName")) + assert.Equal(t, "ext", form.Get("ExternalId")) + assert.Equal(t, "3600", form.Get("DurationSeconds")) + assert.Contains(t, authorization, "Credential=AKIACONTAINERSOURCE/", "the role is assumed with the invocation's container credentials") + + _, _, err = resolveBase(t, "HOME="+home, "AWS_PROFILE=ecs") + require.ErrorIs(t, err, errNoInvocationCredentials, "the process container URI is not used") + _, _, err = resolveBase(t, "HOME="+home, "AWS_PROFILE=env") + require.ErrorIs(t, err, errNoInvocationCredentials, "the process keys are not the Environment source") + _, _, err = resolveBase(t, "HOME="+home, "AWS_PROFILE=bogus") + require.ErrorIs(t, err, errUtils.ErrInvalidAuthConfig) +} + +func TestBaseCredentialsStayWithTheirInvocationUnderConcurrency(t *testing.T) { + poisoned := poisonProcessEnvironment(t) + const invocations = 16 + homes := make([]string, invocations) + for i := range homes { + homes[i] = homeWithProfiles(t, fmt.Sprintf("[default]\naws_access_key_id = AKIAINVOCATION%02d\naws_secret_access_key = s\n", i), "") + } + server, _ := metadataServer(t, "AKIAINSTANCE") + + var wg sync.WaitGroup + keys := make([]string, invocations) + errs := make([]error, invocations) + for i := range invocations { + wg.Add(1) + go func() { + defer wg.Done() + // Odd invocations name a metadata endpoint too; the profile still comes first. + environ := invocationHome(homes[i]) + if i%2 == 1 { + environ = append(environ, "AWS_EC2_METADATA_SERVICE_ENDPOINT="+server.URL) + } + credentials, err := baseCredentials(context.Background(), environMap(environ)) + if err == nil { + keys[i] = credentials.AccessKeyID + } + errs[i] = err + }() + } + wg.Wait() + for i := range invocations { + require.NoError(t, errs[i]) + assert.Equal(t, fmt.Sprintf("AKIAINVOCATION%02d", i), keys[i]) + } + assertProcessEnvironmentUnchanged(t, poisoned) +} + +func TestInvocationSharedFiles(t *testing.T) { + configFile, credentialsFile := invocationSharedFiles(map[string]string{"AWS_CONFIG_FILE": "c", "AWS_SHARED_CREDENTIALS_FILE": "k"}) + assert.Equal(t, "c", configFile) + assert.Equal(t, "k", credentialsFile) + home := t.TempDir() + configFile, credentialsFile = invocationSharedFiles(map[string]string{"HOME": home, "USERPROFILE": home}) + assert.Equal(t, filepath.Join(home, ".aws", "config"), configFile) + assert.Equal(t, filepath.Join(home, ".aws", "credentials"), credentialsFile) +} + +func TestBaseCredentialsFromAProfileWebIdentityUseTheInvocationSTSEndpoint(t *testing.T) { + poisonProcessEnvironment(t) + processSTS := newSTSRecorder(t) + t.Setenv("AWS_ENDPOINT_URL_STS", processSTS.server.URL) + sts := newSTSRecorder(t) + home := t.TempDir() + tokenFile := filepath.Join(home, "token") + require.NoError(t, os.WriteFile(tokenFile, []byte("profile-web-identity-token"), 0o600)) + home = homeWithProfiles(t, "", fmt.Sprintf("[profile oidc]\nrole_arn = arn:aws:iam::1:role/oidc\nweb_identity_token_file = %s\nregion = us-east-2\n", tokenFile)) + + key, region, err := resolveBase(t, "HOME="+home, "USERPROFILE="+home, "AWS_PROFILE=oidc", "AWS_ENDPOINT_URL_STS="+sts.server.URL) + require.NoError(t, err) + assert.Equal(t, assumedKey, key) + assert.Equal(t, "us-east-2", region) + form, _ := sts.last(t) + assert.Equal(t, "profile-web-identity-token", form.Get("WebIdentityToken")) + processSTS.mu.Lock() + defer processSTS.mu.Unlock() + assert.Empty(t, processSTS.forms, "the process STS endpoint receives nothing") +} + +func TestBaseCredentialsFromAStaticSourceProfileChainUseTheInvocationSTSEndpoint(t *testing.T) { + poisonProcessEnvironment(t) + processSTS := newSTSRecorder(t) + t.Setenv("AWS_ENDPOINT_URL_STS", processSTS.server.URL) + sts := newSTSRecorder(t) + home := homeWithProfiles(t, + "[base]\naws_access_key_id = AKIACHAINBASE\naws_secret_access_key = s\n", + "[profile chain]\nrole_arn = arn:aws:iam::1:role/chain\nsource_profile = base\nregion = us-east-2\n") + + key, region, err := resolveBase(t, "HOME="+home, "USERPROFILE="+home, "AWS_PROFILE=chain", "AWS_ENDPOINT_URL_STS="+sts.server.URL) + require.NoError(t, err) + assert.Equal(t, assumedKey, key) + assert.Equal(t, "us-east-2", region) + form, _ := sts.last(t) + assert.Equal(t, "arn:aws:iam::1:role/chain", form.Get("RoleArn"), "the profile role assumption reaches the invocation STS endpoint") + processSTS.mu.Lock() + defer processSTS.mu.Unlock() + assert.Empty(t, processSTS.forms, "the process STS endpoint receives nothing") +} + +func TestBaseCredentialsHonorAWSDefaultProfile(t *testing.T) { + poisonProcessEnvironment(t) + home := homeWithProfiles(t, + "[default]\naws_access_key_id = AKIAPLAINDEFAULT\naws_secret_access_key = s\n[production]\naws_access_key_id = AKIAPRODUCTION\naws_secret_access_key = s\n", "") + + key, _, err := resolveBase(t, append(invocationHome(home), "AWS_DEFAULT_PROFILE=production")...) + require.NoError(t, err) + assert.Equal(t, "AKIAPRODUCTION", key, "AWS_DEFAULT_PROFILE selects the profile when AWS_PROFILE is unset") + + key, _, err = resolveBase(t, append(invocationHome(home), "AWS_DEFAULT_PROFILE=production", "AWS_PROFILE=default")...) + require.NoError(t, err) + assert.Equal(t, "AKIAPLAINDEFAULT", key, "AWS_PROFILE takes precedence over AWS_DEFAULT_PROFILE") + + _, _, err = resolveBase(t, append(invocationHome(home), "AWS_DEFAULT_PROFILE=missing")...) + require.Error(t, err, "a missing profile named by AWS_DEFAULT_PROFILE is an error, not a fallback") +} + +// brokerCommand returns a credential_process value that runs the test binary as the fake broker. +func brokerCommand(t *testing.T) string { + t.Helper() + exe, err := os.Executable() + require.NoError(t, err) + if runtime.GOOS == "windows" { + if strings.ContainsAny(exe, " \t") { + return `"` + exe + `"` + } + return exe + } + return `'` + strings.ReplaceAll(exe, `'`, `'\''`) + `'` +} + +// A credential_process broker runs with the invocation environment, so a selector such as BROKER_PROFILE +// comes from the invocation and never from the process. +func TestBaseCredentialsFromAProfileCredentialProcessRunWithTheInvocationEnvironment(t *testing.T) { + poisonProcessEnvironment(t) + t.Setenv("BROKER_PROFILE", "development") + // The process also acts as a broker, so a broker that wrongly inherits the process environment answers + // with the process value instead of re-running the whole test binary. + t.Setenv(testCredentialBrokerEnv, "1") + home := homeWithProfiles(t, "", "[profile brokered]\ncredential_process = "+brokerCommand(t)+"\n") + invocation := append(invocationHome(home), "AWS_PROFILE=brokered", testCredentialBrokerEnv+"=1", "BROKER_PROFILE=production") + if root := os.Getenv("SystemRoot"); root != "" { + invocation = append(invocation, "SystemRoot="+root) + } + + key, _, err := resolveBase(t, invocation...) + require.NoError(t, err) + assert.Equal(t, "AKIABROKERproduction", key, "the broker sees the invocation's BROKER_PROFILE, not the process value") + + sts := newSTSRecorder(t) + chained := homeWithProfiles(t, "", "[profile brokered]\ncredential_process = "+brokerCommand(t)+"\n[profile deploy]\nrole_arn = arn:aws:iam::1:role/deploy\nsource_profile = brokered\n") + _, _, err = resolveBase(t, append(invocationHome(chained), "AWS_PROFILE=deploy", "AWS_REGION=us-east-1", "AWS_ENDPOINT_URL_STS="+sts.server.URL, + testCredentialBrokerEnv+"=1", "BROKER_PROFILE=staging")...) + require.NoError(t, err) + _, authorization := sts.last(t) + assert.Contains(t, authorization, "Credential=AKIABROKERstaging/", "a role assumed from a broker profile is signed with the invocation's broker credentials") +} + +// ssoTokenCache writes a cached SSO token for the key under the home directory. +func ssoTokenCache(t *testing.T, home, key, token string) { + t.Helper() + sum := sha1.Sum([]byte(key)) + dir := filepath.Join(home, ".aws", "sso", "cache") + require.NoError(t, os.MkdirAll(dir, 0o700)) + contents, err := json.Marshal(map[string]string{"accessToken": token, "expiresAt": "2099-01-01T00:00:00Z"}) + require.NoError(t, err) + require.NoError(t, os.WriteFile(filepath.Join(dir, hex.EncodeToString(sum[:])+".json"), contents, 0o600)) +} + +// ssoPortal serves the SSO role credentials API and records the bearer token of each request. +func ssoPortal(t *testing.T) *[]string { + t.Helper() + var mu sync.Mutex + var tokens []string + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + mu.Lock() + tokens = append(tokens, r.Header.Get("X-Amz-Sso_bearer_token")) + mu.Unlock() + require.NoError(t, json.NewEncoder(w).Encode(map[string]any{"roleCredentials": map[string]any{ + "accessKeyId": "ASIASSOROLE", "secretAccessKey": "sso-secret", "sessionToken": "sso-token", "expiration": int64(4070908800000), + }})) + })) + t.Cleanup(server.Close) + t.Setenv("AWS_ENDPOINT_URL_SSO", server.URL) + return &tokens +} + +// An SSO profile reads its cached login from the invocation home, in both profile forms, and never from the +// home of the process. +func TestBaseCredentialsFromAnSSOProfileReadTheInvocationTokenCache(t *testing.T) { + const startURL = "https://corp.awsapps.com/start" + forms := map[string]struct{ config, cacheKey string }{ + "legacy": { + config: "[profile sso]\nsso_start_url = " + startURL + "\nsso_region = us-east-1\nsso_account_id = 111122223333\nsso_role_name = Admin\n", + cacheKey: startURL, + }, + "session": { + config: "[profile sso]\nsso_session = corp\nsso_account_id = 111122223333\nsso_role_name = Admin\n" + + "[sso-session corp]\nsso_start_url = " + startURL + "\nsso_region = us-east-1\n", + cacheKey: "corp", + }, + } + for name, form := range forms { + t.Run(name, func(t *testing.T) { + tokens := ssoPortal(t) + processHome := t.TempDir() + t.Setenv("HOME", processHome) + t.Setenv("USERPROFILE", processHome) + ssoTokenCache(t, processHome, form.cacheKey, "process-login") + home := homeWithProfiles(t, "", form.config) + invocation := append(invocationHome(home), "AWS_PROFILE=sso") + + _, _, err := resolveBase(t, "AWS_PROFILE=sso", "AWS_CONFIG_FILE="+filepath.Join(home, ".aws", "config")) + require.ErrorIs(t, err, errUtils.ErrInvalidAuthConfig, "an explicit profile without an invocation home cannot select the process cache") + assert.Empty(t, *tokens, "the process login is never sent when the invocation home is absent") + + _, _, err = resolveBase(t, invocation...) + require.Error(t, err, "an invocation without a cached login fails instead of using the process login") + assert.Empty(t, *tokens, "the process login is never sent") + + ssoTokenCache(t, home, form.cacheKey, "invocation-login") + key, _, err := resolveBase(t, invocation...) + require.NoError(t, err) + assert.Equal(t, "ASIASSOROLE", key) + assert.Equal(t, []string{"invocation-login"}, *tokens, "the invocation's cached login is used") + }) + } +} + +// metadataServerWithoutTokens emulates an instance whose token endpoint refuses requests, which is when +// the IMDSv1 fallback applies, and counts the credential reads that arrive without a session token. +func metadataServerWithoutTokens(t *testing.T) (*httptest.Server, *atomic.Int32) { + t.Helper() + var v1Reads atomic.Int32 + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + switch { + case r.Method == http.MethodPut && r.URL.Path == "/latest/api/token": + http.Error(w, "forbidden", http.StatusForbidden) + case r.URL.Path == "/latest/meta-data/iam/security-credentials/": + v1Reads.Add(1) + fmt.Fprint(w, "instance-role") + case r.URL.Path == "/latest/meta-data/iam/security-credentials/instance-role": + v1Reads.Add(1) + require.NoError(t, json.NewEncoder(w).Encode(map[string]string{ + "Code": "Success", "Type": "AWS-HMAC", "AccessKeyId": "AKIAIMDSV1", "SecretAccessKey": "instance-secret", + "Token": "instance-token", "Expiration": "2099-01-01T00:00:00Z", "LastUpdated": "2026-01-01T00:00:00Z", + })) + default: + http.NotFound(w, r) + } + })) + t.Cleanup(server.Close) + return server, &v1Reads +} + +// The restriction of a profile reaches the metadata client, and an invocation environment variable decides +// over it, as it does in the SDK. +func TestBaseCredentialsFromAProfileCredentialSourceHonorTheIMDSv1Restriction(t *testing.T) { + poisonProcessEnvironment(t) + sts := newSTSRecorder(t) + metadata, v1Reads := metadataServerWithoutTokens(t) + profile := func(restriction string) string { + return "[profile imds]\nrole_arn = arn:aws:iam::1:role/from-imds\ncredential_source = Ec2InstanceMetadata\n" + restriction + } + invocation := func(home string, extra ...string) []string { + return append(append(invocationHome(home), "AWS_PROFILE=imds", "AWS_REGION=us-east-1", + "AWS_ENDPOINT_URL_STS="+sts.server.URL, "AWS_EC2_METADATA_SERVICE_ENDPOINT="+metadata.URL), extra...) + } + + restricted := homeWithProfiles(t, "", profile("ec2_metadata_v1_disabled = true\n")) + _, _, err := resolveBase(t, invocation(restricted)...) + require.Error(t, err, "a failed token request does not fall back to IMDSv1 when only the profile restricts it") + assert.Zero(t, v1Reads.Load(), "no credentials are read without a session token") + + _, _, err = resolveBase(t, invocation(restricted, "AWS_EC2_METADATA_V1_DISABLED=false")...) + require.NoError(t, err, "the environment variable overrides the profile") + assert.Positive(t, v1Reads.Load()) + + unrestricted := homeWithProfiles(t, "", profile("")) + before := v1Reads.Load() + _, _, err = resolveBase(t, invocation(unrestricted)...) + require.NoError(t, err, "a profile without the restriction keeps the fallback") + assert.Greater(t, v1Reads.Load(), before) + + before = v1Reads.Load() + _, _, err = resolveBase(t, invocation(unrestricted, "AWS_EC2_METADATA_V1_DISABLED=true")...) + require.Error(t, err, "the environment variable restricts a profile that does not") + assert.Equal(t, before, v1Reads.Load()) +} + +func TestInvocationWithoutHomeDoesNotReadProcessSharedFiles(t *testing.T) { + home := t.TempDir() + t.Setenv("HOME", home) + t.Setenv("USERPROFILE", home) + require.NoError(t, os.MkdirAll(filepath.Join(home, ".aws"), 0o700)) + require.NoError(t, os.WriteFile(filepath.Join(home, ".aws", "credentials"), []byte("[default]\naws_access_key_id = process-key\naws_secret_access_key = process-secret\n"), 0o600)) + profile, err := loadInvocationProfile(t.Context(), map[string]string{}) + require.NoError(t, err) + require.Nil(t, profile) + profile, err = loadInvocationProfile(t.Context(), map[string]string{"AWS_SHARED_CREDENTIALS_FILE": filepath.Join(home, ".aws", "credentials")}) + require.NoError(t, err) + require.Equal(t, "process-key", profile.Credentials.AccessKeyID) +} + +func TestContainerClientClosesCompletedConnection(t *testing.T) { + closed := make(chan struct{}, 1) + server := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { _, _ = w.Write([]byte("ok")) })) + server.Config.ConnState = func(_ net.Conn, state http.ConnState) { + if state == http.StateClosed { + closed <- struct{}{} + } + } + server.Start() + defer server.Close() + response, err := containerEndpointClient(time.Second).Get(server.URL) + require.NoError(t, err) + _, err = io.Copy(io.Discard, response.Body) + require.NoError(t, err) + require.NoError(t, response.Body.Close()) + select { + case <-closed: + case <-time.After(5 * time.Second): + t.Fatal("completed container connection retained") + } +} + +func TestCredentialProcessErrorsDoNotExposeOutput(t *testing.T) { + if runtime.GOOS == "windows" { + t.Skip("POSIX shell command") + } + file := filepath.Join(t.TempDir(), "config") + require.NoError(t, os.WriteFile(file, []byte("[default]\ncredential_process = printf secret-broker-output\n"), 0o600)) + _, err := baseCredentials(t.Context(), map[string]string{"AWS_CONFIG_FILE": file}) + require.ErrorIs(t, err, errUtils.ErrAuthenticationFailed) + require.Contains(t, err.Error(), "credential_process") + for current := err; current != nil; current = errors.Unwrap(current) { + require.NotContains(t, fmt.Sprintf("%+v", current), "secret-broker-output") + } +} + +func TestEnvironmentCredentialAliasesOverrideProfile(t *testing.T) { + file := filepath.Join(t.TempDir(), "credentials") + require.NoError(t, os.WriteFile(file, []byte("[default]\naws_access_key_id = profile-key\naws_secret_access_key = profile-secret\n"), 0o600)) + env := map[string]string{"AWS_SHARED_CREDENTIALS_FILE": file, "AWS_ACCESS_KEY": "alias-key", "AWS_SECRET_KEY": "alias-secret"} + for _, canonical := range []bool{false, true} { + wantKey, wantSecret := "alias-key", "alias-secret" + if canonical { + env["AWS_ACCESS_KEY_ID"], env["AWS_SECRET_ACCESS_KEY"] = "canonical-key", "canonical-secret" + wantKey, wantSecret = "canonical-key", "canonical-secret" + } + result, err := baseCredentials(t.Context(), env) + require.NoError(t, err) + require.Equal(t, wantKey, result.AccessKeyID) + require.Equal(t, wantSecret, result.SecretAccessKey) + provider, err := namedSourceCredentials(env, credentialSourceEnvironment, nil) + require.NoError(t, err) + selected, err := provider.Retrieve(t.Context()) + require.NoError(t, err) + require.Equal(t, wantKey, selected.AccessKeyID) + require.Equal(t, wantSecret, selected.SecretAccessKey) + } +} diff --git a/pkg/auth/identities/aws/inline_role_test.go b/pkg/auth/identities/aws/inline_role_test.go new file mode 100644 index 00000000000..40927b05d61 --- /dev/null +++ b/pkg/auth/identities/aws/inline_role_test.go @@ -0,0 +1,149 @@ +package aws + +import ( + "context" + "fmt" + "net/http" + "net/http/httptest" + "net/url" + "os" + "path/filepath" + "strings" + "sync" + "testing" + "time" + + "github.com/charmbracelet/x/ansi" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + errUtils "github.com/cloudposse/atmos/errors" +) + +// stsRecorder is an STS endpoint that accepts AssumeRole and AssumeRoleWithWebIdentity and records them. +type stsRecorder struct { + mu sync.Mutex + server *httptest.Server + forms []url.Values + authorization []string +} + +const ( + assumedKey = "ASIAASSUMEDKEY" + assumedSecret = "assumed-secret-value" + assumedSession = "assumed-session-token" +) + +func newSTSRecorder(t *testing.T) *stsRecorder { + t.Helper() + recorder := &stsRecorder{} + recorder.server = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + require.NoError(t, r.ParseForm()) + recorder.mu.Lock() + recorder.forms = append(recorder.forms, r.PostForm) + recorder.authorization = append(recorder.authorization, r.Header.Get("Authorization")) + recorder.mu.Unlock() + action := r.PostForm.Get("Action") + w.Header().Set("Content-Type", "text/xml") + fmt.Fprintf(w, ` +<%[1]sResponse xmlns="https://sts.amazonaws.com/doc/2011-06-15/"><%[1]sResult> +%[2]s%[3]s%[4]s +2099-01-01T00:00:00Zr`, + action, assumedKey, assumedSecret, assumedSession) + })) + t.Cleanup(recorder.server.Close) + return recorder +} + +func (r *stsRecorder) last(t *testing.T) (url.Values, string) { + t.Helper() + r.mu.Lock() + defer r.mu.Unlock() + require.NotEmpty(t, r.forms, "STS received no request") + return r.forms[len(r.forms)-1], r.authorization[len(r.authorization)-1] +} + +func TestAssumeInlineRoleLayersOnStaticCredentials(t *testing.T) { + sts := newSTSRecorder(t) + env, err := AssumeInlineRole(context.Background(), + InlineRole{Role: "arn:aws:iam::123456789012:role/deploy", SessionName: "terragrunt-test", Duration: 2 * time.Hour}, + []string{"AWS_ACCESS_KEY_ID=AKIABASEKEY", "AWS_SECRET_ACCESS_KEY=base-secret", "AWS_REGION=eu-west-1", "AWS_ENDPOINT_URL_STS=" + sts.server.URL, "AWS_PROFILE=ambient"}) + require.NoError(t, err) + + form, authorization := sts.last(t) + assert.Equal(t, "AssumeRole", form.Get("Action")) + assert.Equal(t, "arn:aws:iam::123456789012:role/deploy", form.Get("RoleArn")) + assert.Equal(t, "terragrunt-test", form.Get("RoleSessionName")) + assert.Equal(t, "7200", form.Get("DurationSeconds")) + assert.Contains(t, authorization, "Credential=AKIABASEKEY/", "the request is signed by the invocation credentials") + + assert.Equal(t, map[string]string{ + "AWS_ACCESS_KEY_ID": assumedKey, "AWS_SECRET_ACCESS_KEY": assumedSecret, "AWS_SESSION_TOKEN": assumedSession, + "AWS_SECURITY_TOKEN": assumedSession, "AWS_PROFILE": "", + }, env) +} + +func TestAssumeInlineRoleUsesProfileFilesFromTheEnvironment(t *testing.T) { + sts := newSTSRecorder(t) + dir := t.TempDir() + credentialsFile := filepath.Join(dir, "credentials") + require.NoError(t, os.WriteFile(credentialsFile, []byte("[atmos]\naws_access_key_id = AKIAFROMFILE\naws_secret_access_key = file-secret\n"), 0o600)) + t.Setenv("AWS_ACCESS_KEY_ID", "AKIAPROCESSKEY") + t.Setenv("AWS_SECRET_ACCESS_KEY", "process-secret") + + _, err := AssumeInlineRole(context.Background(), InlineRole{Role: "arn:aws:iam::1:role/r"}, + []string{"AWS_PROFILE=atmos", "AWS_SHARED_CREDENTIALS_FILE=" + credentialsFile, "AWS_CONFIG_FILE=" + filepath.Join(dir, "config"), "AWS_ENDPOINT_URL=" + sts.server.URL, "AWS_DEFAULT_REGION=us-east-2"}) + require.NoError(t, err) + form, authorization := sts.last(t) + assert.Contains(t, authorization, "Credential=AKIAFROMFILE/", "the profile named by the environment signs the request, not the process environment") + assert.Empty(t, form.Get("DurationSeconds"), "no duration is sent unless one is requested") + assert.Contains(t, form.Get("RoleSessionName"), "atmos-", "a generated session name is used") +} + +func TestAssumeInlineRoleWithWebIdentityNeedsNoCredentials(t *testing.T) { + sts := newSTSRecorder(t) + env, err := AssumeInlineRole(context.Background(), + InlineRole{Role: "arn:aws:iam::1:role/oidc", WebIdentityToken: "oidc-token-value", SessionName: "ci"}, + []string{"AWS_ENDPOINT_URL_STS=" + sts.server.URL}) + require.NoError(t, err) + form, _ := sts.last(t) + assert.Equal(t, "AssumeRoleWithWebIdentity", form.Get("Action")) + assert.Equal(t, "oidc-token-value", form.Get("WebIdentityToken")) + assert.Equal(t, "ci", form.Get("RoleSessionName")) + assert.Equal(t, assumedKey, env["AWS_ACCESS_KEY_ID"]) + assert.NotContains(t, env, "oidc-token-value") +} + +func TestAssumeInlineRoleFailures(t *testing.T) { + t.Run("STS rejects the role", func(t *testing.T) { + server := stsErrorServer(t, http.StatusForbidden, "AccessDenied", "not authorized to assume the role") + _, err := AssumeInlineRole(context.Background(), InlineRole{Role: "arn:aws:iam::1:role/denied"}, + []string{"AWS_ACCESS_KEY_ID=AKIAX", "AWS_SECRET_ACCESS_KEY=secret-never-shown", "AWS_ENDPOINT_URL_STS=" + server.URL}) + require.ErrorIs(t, err, errUtils.ErrAuthenticationFailed) + // CI renders color and wraps long lines, so compare the text without escapes and whitespace. + rendered := strings.Join(strings.Fields(ansi.Strip(errUtils.Format(err, errUtils.FormatterConfig{}))), "") + assert.Contains(t, rendered, "arn:aws:iam::1:role/denied") + assert.NotContains(t, err.Error(), "secret-never-shown") + }) + t.Run("no starting credentials", func(t *testing.T) { + empty := t.TempDir() + // The process keys are not a fallback for an invocation that names a missing profile. + t.Setenv("AWS_ACCESS_KEY_ID", "AKIAPROCESSKEY") + t.Setenv("AWS_SECRET_ACCESS_KEY", "process-secret") + _, err := AssumeInlineRole(context.Background(), InlineRole{Role: "arn:aws:iam::1:role/r"}, + []string{"AWS_SHARED_CREDENTIALS_FILE=" + filepath.Join(empty, "none"), "AWS_CONFIG_FILE=" + filepath.Join(empty, "none"), "AWS_PROFILE=missing"}) + require.ErrorIs(t, err, errUtils.ErrAuthenticationFailed) + // Nor for one whose every source is empty. + _, err = AssumeInlineRole(context.Background(), InlineRole{Role: "arn:aws:iam::1:role/r"}, + []string{"HOME=" + empty, "USERPROFILE=" + empty, "AWS_EC2_METADATA_DISABLED=true"}) + require.ErrorIs(t, err, errUtils.ErrAuthenticationFailed) + }) +} + +func TestInlineRoleConfigAndEnvironment(t *testing.T) { + env := environMap([]string{"A=1", "A=2", "B=x=y", "malformed"}) + assert.Equal(t, map[string]string{"A": "2", "B": "x=y"}, env) + identity := inlineRoleConfig(InlineRole{Role: "r"}, map[string]string{}) + assert.Nil(t, identity.Spec) + assert.NotContains(t, identity.Principal, "duration") +} diff --git a/pkg/auth/identities/aws/testmain_test.go b/pkg/auth/identities/aws/testmain_test.go index b8b87032bcd..6249f4de88c 100644 --- a/pkg/auth/identities/aws/testmain_test.go +++ b/pkg/auth/identities/aws/testmain_test.go @@ -34,7 +34,18 @@ import ( // the test binary itself (via os.Executable) so they work on every platform without Unix-only tools. const testCredentialProcessJSONEnv = "_ATMOS_TEST_CREDENTIAL_PROCESS_JSON" +// testCredentialBrokerEnv, when set to "1", turns the test binary into a fake credential_process broker +// that returns the access key "AKIABROKER" followed by the value of BROKER_PROFILE in its own environment, +// so a test can tell which environment the broker ran with. +const testCredentialBrokerEnv = "_ATMOS_TEST_CREDENTIAL_BROKER" + func TestMain(m *testing.M) { + // Act as the environment-sensitive credential_process broker when requested. + if os.Getenv(testCredentialBrokerEnv) == "1" { + fmt.Fprintf(os.Stdout, `{"Version":1,"AccessKeyId":"AKIABROKER%s","SecretAccessKey":"broker-secret"}`, os.Getenv("BROKER_PROFILE")) + os.Exit(0) + } + // Act as the fake credential_process helper when requested. if out, ok := os.LookupEnv(testCredentialProcessJSONEnv); ok { fmt.Fprint(os.Stdout, out) diff --git a/pkg/ci/plugins/terraform/aggregate_provider.go b/pkg/ci/plugins/terraform/aggregate_provider.go index 7cf5fc0111c..6f9b58f0761 100644 --- a/pkg/ci/plugins/terraform/aggregate_provider.go +++ b/pkg/ci/plugins/terraform/aggregate_provider.go @@ -107,7 +107,7 @@ func (p *Plugin) updateAggregateCheckRuns(ctx *plugin.HookContext, aggregate *te for i := range aggregate.Components { component := &aggregate.Components[i] componentCtx := p.aggregateComponentContext(ctx, component) - if component.Skipped { + if component.Skipped && component.Result.Status != "succeeded" { componentCtx.CommandError = aggregateCommandError(component.Summary) componentCtx.ExitCode = 1 } diff --git a/pkg/ci/plugins/terraform/aggregate_test.go b/pkg/ci/plugins/terraform/aggregate_test.go index 4380ed85e08..7295584a073 100644 --- a/pkg/ci/plugins/terraform/aggregate_test.go +++ b/pkg/ci/plugins/terraform/aggregate_test.go @@ -845,3 +845,23 @@ func TestAggregateCommentKeyDoesNotCollideWithComponentNamedAggregate(t *testing assert.NotEqual(t, buildCommentKey(command, "aggregate", "dev"), buildAggregateCommentKey(command, "dev")) } } + +func TestAggregateChecksDistinguishIntentionalAndFailureSkips(t *testing.T) { + p := &Plugin{} + ctx := newAggregateHookContext() + ctx.Config.CI.Checks.Enabled = boolPtr(true) + mp := ctx.Provider.(*mockProvider) + results := schema.TerraformPlanCIResultSet{Results: []schema.TerraformPlanCIResult{ + {Stack: "dev", Component: "excluded", Status: "succeeded", Processed: false}, + {Stack: "dev", Component: "blocked", Status: "skipped", Processed: false}, + }} + aggregate := p.buildPlanAggregate(ctx, results) + require.Len(t, aggregate.Components, 2) + for _, component := range aggregate.Components { + assert.True(t, component.Skipped) + } + p.updateAggregateCheckRuns(ctx, &aggregate) + require.Len(t, mp.updateRunCalls, 2) + assert.Equal(t, provider.CheckRunStateFailure, mp.updateRunCalls[0].Status) // blocked + assert.Equal(t, provider.CheckRunStateSuccess, mp.updateRunCalls[1].Status) // excluded +} diff --git a/pkg/component/configuration/imports.go b/pkg/component/configuration/imports.go index 90b8a60b67b..edf2898ca14 100644 --- a/pkg/component/configuration/imports.go +++ b/pkg/component/configuration/imports.go @@ -9,6 +9,7 @@ import ( "github.com/bmatcuk/doublestar/v4" errUtils "github.com/cloudposse/atmos/errors" + "github.com/cloudposse/atmos/pkg/filesystem" "github.com/cloudposse/atmos/pkg/perf" "github.com/cloudposse/atmos/pkg/schema" ) @@ -79,24 +80,52 @@ func WatchExcludes(settings map[string]any) ([]string, error) { return patterns, nil } -// UnexcludedWatchFiles returns the files under root that match none of the patterns. +// UnexcludedWatchFiles returns the files that match none of the project-relative patterns. +// +// The project root and a changed file can spell one directory differently, for example through a +// symlinked temporary directory, so a path that appears outside the root is compared again with +// symlinks resolved on both sides. A file that is still outside the project, including one on +// another volume, cannot match a project-relative exclusion and stays watched rather than letting +// a pattern such as `**` swallow a `../` path. Comparing a relative root with an absolute file is +// rejected, because it would silently match nothing. func UnexcludedWatchFiles(root string, all []string, patterns []string) ([]string, error) { defer perf.Track(nil, "configuration.UnexcludedWatchFiles")() files := make([]string, 0, len(all)) + resolvedRoot := "" for _, path := range all { - relative, err := filepath.Rel(root, path) - if err != nil { - return nil, fmt.Errorf("%w: watch path %q is not under %q: %w", errUtils.ErrPluginInvalidConfig, path, root, err) + if filepath.IsAbs(root) != filepath.IsAbs(path) { + return nil, fmt.Errorf("%w: watch path %q is not under %q", errUtils.ErrPluginInvalidConfig, path, root) } - excluded := false - for _, pattern := range patterns { - matched, _ := doublestar.Match(pattern, filepath.ToSlash(relative)) - excluded = excluded || matched + relative, inside := projectRelative(root, path) + if !inside { + if resolvedRoot == "" { + resolvedRoot = filesystem.EvalSymlinksBestEffort(root) + } + relative, inside = projectRelative(resolvedRoot, filesystem.EvalSymlinksBestEffort(path)) } - if !excluded { + if !inside || !matchesAny(patterns, relative) { files = append(files, path) } } return files, nil } + +// projectRelative returns path relative to root, and whether it lies inside root. A path that +// cannot be made relative, such as one on another volume, is outside. +func projectRelative(root, path string) (string, bool) { + relative, err := filepath.Rel(root, path) + if err != nil || relative == ".." || strings.HasPrefix(relative, ".."+string(filepath.Separator)) { + return "", false + } + return relative, true +} + +func matchesAny(patterns []string, relative string) bool { + for _, pattern := range patterns { + if matched, _ := doublestar.Match(pattern, filepath.ToSlash(relative)); matched { + return true + } + } + return false +} diff --git a/pkg/component/configuration/manifest.go b/pkg/component/configuration/manifest.go index b111231b26e..31f73a74a10 100644 --- a/pkg/component/configuration/manifest.go +++ b/pkg/component/configuration/manifest.go @@ -55,7 +55,16 @@ func readManifest(root *os.Root) map[string]string { return files } +// writeManifest records the generated files. A failure keeps its filesystem cause and carries the +// configuration sentinel, so every caller reports it the same way as the other generated-file errors. func writeManifest(root *os.Root, files map[string]string) error { + if err := storeManifest(root, files); err != nil { + return fmt.Errorf("%w: write %s: %w", errUtils.ErrPluginInvalidConfig, manifestName, err) + } + return nil +} + +func storeManifest(root *os.Root, files map[string]string) error { if len(files) == 0 { err := root.Remove(manifestName) if errors.Is(err, os.ErrNotExist) { diff --git a/pkg/component/configuration/manifest_test.go b/pkg/component/configuration/manifest_test.go index 042450e45cf..bcb5003af00 100644 --- a/pkg/component/configuration/manifest_test.go +++ b/pkg/component/configuration/manifest_test.go @@ -276,6 +276,21 @@ func TestExplicitLockGenerationReplacesAdjacentCopy(t *testing.T) { require.Equal(t, "# pinned\n", read(t, root, lockFileName)) } +// A manifest that cannot be written is reported with the configuration sentinel and keeps its +// filesystem cause; the generated file written before it is rolled back, because nothing records it. +func TestManifestWriteFailureCarriesSentinel(t *testing.T) { + root := t.TempDir() + // A regular file where the bookkeeping directory belongs makes the manifest write fail. + require.NoError(t, os.WriteFile(filepath.Join(root, ".atmos"), []byte("blocker\n"), 0o600)) + + err := WriteFiles(root, []schema.ComponentConfigurationFile{generated("main.tf", "locals {}\n", "overwrite")}) + require.ErrorIs(t, err, errUtils.ErrPluginInvalidConfig) + require.Contains(t, err.Error(), manifestName) + var pathErr *os.PathError + require.ErrorAs(t, err, &pathErr, "the filesystem cause is preserved") + require.NoFileExists(t, filepath.Join(root, "main.tf"), "a run whose manifest cannot be written leaves no unowned file") +} + // requireEffectiveDirectoryPermissions skips a test when the process can still create files in a // read-only directory, as root with CAP_DAC_OVERRIDE can, or on Windows. func requireEffectiveDirectoryPermissions(t *testing.T, directory string) { diff --git a/pkg/component/configuration/watch_test.go b/pkg/component/configuration/watch_test.go new file mode 100644 index 00000000000..f5f38ff599d --- /dev/null +++ b/pkg/component/configuration/watch_test.go @@ -0,0 +1,65 @@ +package configuration + +import ( + "os" + "path/filepath" + "runtime" + "testing" + + "github.com/stretchr/testify/require" + + errUtils "github.com/cloudposse/atmos/errors" +) + +// A project root reached through a symlink still applies its exclusions to the resolved files. +func TestUnexcludedWatchFilesResolveRootAliases(t *testing.T) { + parent := t.TempDir() + root := filepath.Join(parent, "project") + require.NoError(t, os.Mkdir(root, 0o755)) + alias := filepath.Join(parent, "alias") + if err := os.Symlink(root, alias); err != nil { + t.Skipf("symlinks unavailable: %v", err) + } + cache := filepath.Join(root, "live", ".terragrunt-cache", "deleted.tf") + ordinary := filepath.Join(root, "live", "main.tf") + files, err := UnexcludedWatchFiles(alias, []string{cache, ordinary}, []string{"**/.terragrunt-cache/**"}) + require.NoError(t, err) + require.Equal(t, []string{ordinary}, files) +} + +// A changed file outside the project cannot match a project-relative exclusion, even `**`. +func TestUnexcludedWatchFilesKeepFilesOutsideTheProject(t *testing.T) { + parent := t.TempDir() + root := filepath.Join(parent, "project") + sibling := filepath.Join(parent, "sibling", "main.tf") + inside := filepath.Join(root, "live", "main.tf") + files, err := UnexcludedWatchFiles(root, []string{sibling, inside}, []string{"**"}) + require.NoError(t, err) + require.Equal(t, []string{sibling}, files) + + _, err = UnexcludedWatchFiles("relative", []string{inside}, []string{"**"}) + require.ErrorIs(t, err, errUtils.ErrPluginInvalidConfig, "a relative root would silently match nothing") +} + +func TestProjectRelativeTreatsUnrelatablePathsAsOutside(t *testing.T) { + root := t.TempDir() + relative, inside := projectRelative(root, filepath.Join(root, "a", "b.tf")) + require.True(t, inside) + require.Equal(t, filepath.Join("a", "b.tf"), relative) + _, inside = projectRelative(root, filepath.Dir(root)) + require.False(t, inside) + // filepath.Rel fails when it cannot relate the two paths; the file is outside, not an error. + _, inside = projectRelative(root, "relative.tf") + require.False(t, inside) + if runtime.GOOS == "windows" { + other := `Z:\elsewhere\main.tf` + if filepath.VolumeName(root) == "Z:" { + other = `Y:\elsewhere\main.tf` + } + _, inside = projectRelative(root, other) + require.False(t, inside, "a file on another volume is outside the project") + files, err := UnexcludedWatchFiles(root, []string{other}, []string{"**"}) + require.NoError(t, err) + require.Equal(t, []string{other}, files) + } +} diff --git a/pkg/component/execution/execution.go b/pkg/component/execution/execution.go index 9475786d4b7..d1a424413cd 100644 --- a/pkg/component/execution/execution.go +++ b/pkg/component/execution/execution.go @@ -11,6 +11,7 @@ package execution import ( "fmt" "regexp" + "slices" "strconv" "strings" @@ -19,6 +20,17 @@ import ( "github.com/cloudposse/atmos/pkg/schema" ) +// Lifecycle commands name stages of a provider's own pipeline instead of Terraform commands. A hook +// lists them in `commands` like any command, and they run in the unit's configuration directory. +const ( + // CommandReadConfig runs after_hooks once the unit's configuration is read. + CommandReadConfig = "terragrunt-read-config" + // CommandReadConfigAlias is the documented spelling of CommandReadConfig. + CommandReadConfigAlias = "read-config" + // CommandInitFromModule runs hooks around the module source being fetched. + CommandInitFromModule = "init-from-module" +) + const ( commandApply = "apply" commandDestroy = "destroy" @@ -116,3 +128,53 @@ func hooksFor(hooks []schema.ComponentHook, command string) []*schema.ComponentH } return selected } + +// ListsCommand reports whether any hook lists one of the commands. +func ListsCommand(hooks []schema.ComponentHook, commands ...string) bool { + defer perf.Track(nil, "execution.ListsCommand")() + + for i := range hooks { + for _, listed := range hooks[i].Commands { + if slices.Contains(commands, listed) { + return true + } + } + } + return false +} + +// HooksFor returns copies of the hooks that list any of the commands, each rewritten to list only +// the canonical command, so a hook written with an alias still matches the phase that runs it. +func HooksFor(hooks []schema.ComponentHook, canonical string, aliases ...string) []schema.ComponentHook { + defer perf.Track(nil, "execution.HooksFor")() + + var selected []schema.ComponentHook + for i := range hooks { + if !ListsCommand(hooks[i:i+1], append([]string{canonical}, aliases...)...) { + continue + } + hook := hooks[i] + hook.Commands = []string{canonical} + selected = append(selected, hook) + } + return selected +} + +// ExcludeMatches reports whether an exclude block's actions name the command. "all" matches every +// command and "all_except_output" every command but output; other entries match case-insensitively. +func ExcludeMatches(actions []string, command string) bool { + defer perf.Track(nil, "execution.ExcludeMatches")() + + command = strings.ToLower(command) + for _, action := range actions { + switch { + case action == "all": + return true + case action == "all_except_output" && command != "output": + return true + case strings.ToLower(action) == command: + return true + } + } + return false +} diff --git a/pkg/component/execution/lifecycle_test.go b/pkg/component/execution/lifecycle_test.go index 52379687b0b..dd0da53b836 100644 --- a/pkg/component/execution/lifecycle_test.go +++ b/pkg/component/execution/lifecycle_test.go @@ -1,17 +1,27 @@ package execution +//go:generate go run go.uber.org/mock/mockgen@v0.6.0 -destination=mock_reporter_test.go -package=execution github.com/cloudposse/atmos/pkg/component/execution Reporter + import ( + "bytes" "context" + "encoding/json" "errors" + "fmt" "io" + "os" + "path/filepath" "slices" "strings" + "sync" "testing" + "time" "github.com/stretchr/testify/require" "go.uber.org/mock/gomock" errUtils "github.com/cloudposse/atmos/errors" + log "github.com/cloudposse/atmos/pkg/logger" "github.com/cloudposse/atmos/pkg/schema" ) @@ -297,6 +307,113 @@ func TestValidate(t *testing.T) { } } +// recoveryRun drives a Recoverer with a scripted sequence of failures and records the pauses. +type recoveryRun struct { + failures []error + texts []string + attempts int + pauses []time.Duration + mu sync.Mutex +} + +// count reports the attempts made so far, for readers that run beside the recoverer. +func (r *recoveryRun) count() int { + r.mu.Lock() + defer r.mu.Unlock() + return r.attempts +} + +func (r *recoveryRun) attempt(context.Context) (string, error) { + r.mu.Lock() + index := r.attempts + r.attempts++ + r.mu.Unlock() + var text string + if index < len(r.texts) { + text = r.texts[index] + } + if index < len(r.failures) { + return text, r.failures[index] + } + return text, nil +} + +func (r *recoveryRun) sleep(_ context.Context, d time.Duration) error { + r.pauses = append(r.pauses, d) + return nil +} + +var errExit = errors.New("exit status 1") + +func TestRecovererWithoutPolicyReturnsTheFailure(t *testing.T) { + var nilRecoverer *Recoverer + for name, recoverer := range map[string]*Recoverer{"nil recoverer": nilRecoverer, "nil policy": NewRecoverer(nil), "empty policy": NewRecoverer(&schema.ComponentErrors{})} { + t.Run(name, func(t *testing.T) { + run := &recoveryRun{failures: []error{errExit}, texts: []string{"timeout"}} + ignored, err := recoverer.Run(t.Context(), run.attempt) + require.ErrorIs(t, err, errExit) + require.False(t, ignored) + require.Equal(t, 1, run.attempts) + }) + } + ignored, err := NewRecoverer(&schema.ComponentErrors{}).Run(t.Context(), (&recoveryRun{}).attempt) + require.NoError(t, err) + require.False(t, ignored) +} + +func TestRecovererRetriesUntilSuccess(t *testing.T) { + reporter := NewMockReporter(gomock.NewController(t)) + gomock.InOrder(reporter.EXPECT().Retrying("net", 1, 3, 7*time.Second), reporter.EXPECT().Retrying("net", 2, 3, 7*time.Second)) + run := &recoveryRun{failures: []error{errExit, errExit}, texts: []string{"Error: connection timeout", "Error: connection timeout"}} + policy := &schema.ComponentErrors{Retry: []schema.ComponentRetryPolicy{{Name: "net", Patterns: []string{"timeout"}, MaxAttempts: 3, SleepInterval: 7 * time.Second}}} + ignored, err := NewRecoverer(policy, WithReporter(reporter), WithSleep(run.sleep)).Run(t.Context(), run.attempt) + require.NoError(t, err) + require.False(t, ignored) + require.Equal(t, 3, run.attempts) + require.Equal(t, []time.Duration{7 * time.Second, 7 * time.Second}, run.pauses) +} + +func TestRecovererStopsAtMaxAttempts(t *testing.T) { + run := &recoveryRun{failures: []error{errExit, errExit, errExit, errExit}, texts: []string{"timeout", "timeout", "timeout", "timeout"}} + policy := &schema.ComponentErrors{Retry: []schema.ComponentRetryPolicy{{Name: "net", Patterns: []string{"timeout"}, MaxAttempts: 3}}} + _, err := NewRecoverer(policy, WithSleep(run.sleep)).Run(t.Context(), run.attempt) + require.ErrorIs(t, err, errUtils.ErrComponentMaxAttempts) + require.ErrorIs(t, err, errExit, "the last failure stays in the chain") + require.Equal(t, 3, run.attempts, "max_attempts counts the first attempt") + require.Len(t, run.pauses, 2) +} + +func TestRecovererDoesNotRetryUnmatchedErrors(t *testing.T) { + run := &recoveryRun{failures: []error{errExit}, texts: []string{"permission denied"}} + policy := &schema.ComponentErrors{Retry: []schema.ComponentRetryPolicy{{Name: "net", Patterns: []string{"timeout"}, MaxAttempts: 3}}} + _, err := NewRecoverer(policy, WithSleep(run.sleep)).Run(t.Context(), run.attempt) + require.ErrorIs(t, err, errExit) + require.Equal(t, 1, run.attempts) + require.Empty(t, run.pauses) +} + +func TestRecovererMatchesStderrOrFallsBackToError(t *testing.T) { + policy := &schema.ComponentErrors{Ignore: []schema.ComponentIgnorePolicy{{Name: "exit", Patterns: []string{"exit status"}}}} + for _, tc := range []struct { + name, stderr string + ignored bool + }{ + {name: "stderr takes precedence", stderr: "permission denied", ignored: false}, + {name: "empty stderr falls back", stderr: "", ignored: true}, + {name: "blank stderr falls back", stderr: " \n", ignored: true}, + } { + t.Run(tc.name, func(t *testing.T) { + ignored, err := NewRecoverer(policy).Run(t.Context(), (&recoveryRun{failures: []error{errExit}, texts: []string{tc.stderr}}).attempt) + require.Equal(t, tc.ignored, ignored) + if tc.ignored { + require.NoError(t, err) + } else { + require.ErrorIs(t, err, errExit) + } + }) + } +} + // streamOutput is captured output that records how it was read. type streamOutput struct { text string @@ -370,6 +487,194 @@ func TestErrorHooksStreamCapturedOutput(t *testing.T) { } } +func TestRecovererIgnorePolicyWinsOverRetryAndWritesSignals(t *testing.T) { + signalsFile := filepath.Join(t.TempDir(), SignalsFileName) + reporter := NewMockReporter(gomock.NewController(t)) + reporter.EXPECT().Ignored("exists", "fine to reuse") + reporter.EXPECT().SignalsWritten(signalsFile) + run := &recoveryRun{failures: []error{errExit}, texts: []string{"Error: bucket already exists"}} + policy := &schema.ComponentErrors{ + Retry: []schema.ComponentRetryPolicy{{Name: "any", Patterns: []string{".*"}, MaxAttempts: 5}}, + Ignore: []schema.ComponentIgnorePolicy{ + {Name: "unrelated", Patterns: []string{"never"}}, + {Name: "exists", Patterns: []string{"already exists"}, Message: "fine to reuse", Signals: map[string]any{"alert": false, "note": "kept"}}, + }, + } + ignored, err := NewRecoverer(policy, WithReporter(reporter), WithSignalsFile(signalsFile), WithSleep(run.sleep)).Run(t.Context(), run.attempt) + require.NoError(t, err) + require.True(t, ignored) + require.Equal(t, 1, run.attempts) + data, err := os.ReadFile(signalsFile) + require.NoError(t, err) + var written map[string]any + require.NoError(t, json.Unmarshal(data, &written)) + require.Equal(t, map[string]any{"alert": false, "note": "kept"}, written) +} + +func TestRecovererSignalsDoNotFollowSymlink(t *testing.T) { + directory := t.TempDir() + target := filepath.Join(directory, "target") + require.NoError(t, os.WriteFile(target, []byte("keep"), 0o600)) + signals := filepath.Join(directory, SignalsFileName) + if err := os.Symlink(target, signals); err != nil { + t.Skipf("symlinks unavailable: %v", err) + } + policy := &schema.ComponentErrors{Ignore: []schema.ComponentIgnorePolicy{{Name: "ignore", Patterns: []string{"alpha"}, Signals: map[string]any{"x": 1}}}} + reporter := NewMockReporter(gomock.NewController(t)) + reporter.EXPECT().Ignored("ignore", "") + reporter.EXPECT().SignalsWritten(signals) + ignored, err := NewRecoverer(policy, WithReporter(reporter), WithSignalsFile(signals)).Run(t.Context(), (&recoveryRun{failures: []error{errExit}, texts: []string{"alpha"}}).attempt) + require.NoError(t, err) + require.True(t, ignored) + data, err := os.ReadFile(target) + require.NoError(t, err) + require.Equal(t, "keep", string(data)) +} + +func TestRecovererIgnoreWithoutSignalsOrFileWritesNothing(t *testing.T) { + directory := t.TempDir() + policy := &schema.ComponentErrors{Ignore: []schema.ComponentIgnorePolicy{{Name: "a", Patterns: []string{"alpha"}, Signals: map[string]any{"x": 1}}, {Name: "b", Patterns: []string{"beta"}}}} + reporter := NewMockReporter(gomock.NewController(t)) + reporter.EXPECT().Ignored("a", "") + ignored, err := NewRecoverer(policy, WithReporter(reporter)).Run(t.Context(), (&recoveryRun{failures: []error{errExit}, texts: []string{"alpha"}}).attempt) + require.NoError(t, err) + require.True(t, ignored) + + // A signals file that cannot be written emits a warning and never undoes the ignore. + var output bytes.Buffer + log.Default().SetOutput(&output) + t.Cleanup(func() { log.Default().SetOutput(os.Stderr) }) + blocked := filepath.Join(directory, "missing", SignalsFileName) + reporter = NewMockReporter(gomock.NewController(t)) + reporter.EXPECT().Ignored("a", "") + ignored, err = NewRecoverer(policy, WithReporter(reporter), WithSignalsFile(blocked)).Run(t.Context(), (&recoveryRun{failures: []error{errExit}, texts: []string{"alpha"}}).attempt) + require.NoError(t, err) + require.True(t, ignored) + require.NoFileExists(t, blocked) + require.Contains(t, output.String(), "Failed to write error signals") + require.Contains(t, output.String(), blocked) + require.Contains(t, output.String(), "error") + + reporter = NewMockReporter(gomock.NewController(t)) + reporter.EXPECT().Ignored("b", "") + ignored, err = NewRecoverer(policy, WithReporter(reporter), WithSignalsFile(filepath.Join(directory, SignalsFileName))).Run(t.Context(), (&recoveryRun{failures: []error{errExit}, texts: []string{"beta"}}).attempt) + require.NoError(t, err) + require.True(t, ignored) + require.NoFileExists(t, filepath.Join(directory, SignalsFileName), "a policy without signals writes no file") +} + +func TestRecovererNegativePatternVetoesAPolicy(t *testing.T) { + policy := &schema.ComponentErrors{Retry: []schema.ComponentRetryPolicy{{Name: "net", Patterns: []string{"timeout", "!fatal"}, MaxAttempts: 2}}} + run := &recoveryRun{failures: []error{errExit}, texts: []string{"timeout and fatal"}} + _, err := NewRecoverer(policy, WithSleep(run.sleep)).Run(t.Context(), run.attempt) + require.ErrorIs(t, err, errExit) + require.Equal(t, 1, run.attempts) + + run = &recoveryRun{failures: []error{errExit}, texts: []string{"timeout only"}} + _, err = NewRecoverer(policy, WithSleep(run.sleep)).Run(t.Context(), run.attempt) + require.NoError(t, err) + require.Equal(t, 2, run.attempts) +} + +func TestRecovererMatchesCleanedText(t *testing.T) { + require.Equal(t, `Error: Failed to load state: dial tcp: i/o timeout`, cleanErrorText("\x1b[31mError:\x1b[0m Failed to load state:\n\tdial tcp: i/o timeout\n")) + require.Equal(t, "a b", cleanErrorText("a\n\n\t")) + policy := &schema.ComponentErrors{Retry: []schema.ComponentRetryPolicy{{Name: "state", Patterns: []string{`Failed to load state: dial tcp: i/o timeout`}, MaxAttempts: 2}}} + run := &recoveryRun{failures: []error{errExit}, texts: []string{"\x1b[31m│ Error:\x1b[0m Failed to load state:\n│ dial tcp: i/o timeout"}} + _, err := NewRecoverer(policy, WithSleep(run.sleep)).Run(t.Context(), run.attempt) + require.NoError(t, err) + require.Equal(t, 2, run.attempts) +} + +func TestRecovererMatchesTheErrorMessageToo(t *testing.T) { + policy := &schema.ComponentErrors{Ignore: []schema.ComponentIgnorePolicy{{Name: "hook", Patterns: []string{"hook failed"}}}} + run := &recoveryRun{failures: []error{fmt.Errorf("%w: hook failed", errUtils.ErrComponentHookFailed)}} + ignored, err := NewRecoverer(policy).Run(t.Context(), run.attempt) + require.NoError(t, err) + require.True(t, ignored) +} + +func TestRecovererStopsWhenCanceledDuringTheSleep(t *testing.T) { + ctx, cancel := context.WithCancel(t.Context()) + defer cancel() + policy := &schema.ComponentErrors{Retry: []schema.ComponentRetryPolicy{{Name: "net", Patterns: []string{"timeout"}, MaxAttempts: 5, SleepInterval: time.Hour}}} + run := &recoveryRun{failures: []error{errExit, errExit}, texts: []string{"timeout", "timeout"}} + sleeping := make(chan struct{}) + sleep := func(ctx context.Context, d time.Duration) error { + close(sleeping) + return sleepContext(ctx, d) + } + done := make(chan error, 1) + go func() { + _, err := NewRecoverer(policy, WithSleep(sleep)).Run(ctx, run.attempt) + done <- err + }() + // The first attempt fails and the run sleeps for an hour; cancellation must end it at once. + select { + case <-sleeping: + case <-time.After(5 * time.Second): + t.Fatal("the recoverer never started sleeping") + } + cancel() + require.ErrorIs(t, <-done, context.Canceled) + require.Equal(t, 1, run.count()) +} + +func TestRecovererDoesNotRecoverAfterCancellation(t *testing.T) { + ctx, cancel := context.WithCancel(t.Context()) + policy := &schema.ComponentErrors{Ignore: []schema.ComponentIgnorePolicy{{Name: "all", Patterns: []string{".*"}}}} + ignored, err := NewRecoverer(policy).Run(ctx, func(context.Context) (string, error) { + cancel() + return "boom", errExit + }) + require.ErrorIs(t, err, errExit) + require.False(t, ignored) +} + +func TestSleepContext(t *testing.T) { + require.NoError(t, sleepContext(t.Context(), 0)) + require.NoError(t, sleepContext(t.Context(), time.Millisecond)) + ctx, cancel := context.WithCancel(t.Context()) + cancel() + require.ErrorIs(t, sleepContext(ctx, 0), context.Canceled) + require.ErrorIs(t, sleepContext(ctx, time.Hour), context.Canceled) +} + +func TestExcludeMatches(t *testing.T) { + for _, test := range []struct { + actions []string + command string + want bool + }{ + {[]string{"plan"}, "plan", true}, + {[]string{"PLAN"}, "plan", true}, + {[]string{"plan"}, "apply", false}, + {[]string{"all"}, "output", true}, + {[]string{"all_except_output"}, "output", false}, + {[]string{"all_except_output"}, "destroy", true}, + {nil, "plan", false}, + } { + require.Equal(t, test.want, ExcludeMatches(test.actions, test.command), "%v %s", test.actions, test.command) + } +} + +func TestLifecycleHookSelection(t *testing.T) { + hooks := []schema.ComponentHook{ + {Name: "alias", Commands: []string{"plan", CommandReadConfigAlias}}, + {Name: "canonical", Commands: []string{CommandReadConfig}}, + {Name: "module", Commands: []string{CommandInitFromModule}}, + } + require.True(t, ListsCommand(hooks, CommandInitFromModule)) + require.False(t, ListsCommand(hooks[:2], CommandInitFromModule)) + require.False(t, ListsCommand(nil, CommandReadConfig)) + selected := HooksFor(hooks, CommandReadConfig, CommandReadConfigAlias) + require.Len(t, selected, 2) + require.Equal(t, "alias", selected[0].Name) + require.Equal(t, []string{CommandReadConfig}, selected[0].Commands, "the alias is rewritten to the canonical command") + require.Equal(t, []string{"plan", CommandReadConfigAlias}, hooks[0].Commands, "the input is not modified") + require.Empty(t, HooksFor(hooks, "other")) +} + func errorHook(name string) schema.ComponentHook { return schema.ComponentHook{Name: name, Commands: []string{"plan"}, OnErrors: []string{".*"}, Execute: []string{"tool", name}} } @@ -447,3 +752,24 @@ func TestJoinFailuresKeepsLoneErrorIntact(t *testing.T) { require.ErrorIs(t, joined, lone) require.ErrorIs(t, joined, other) } + +// The signals file can depend on what the failed attempt left behind, so it is resolved when written. +func TestRecovererResolvesTheSignalsFileWhenIgnoring(t *testing.T) { + directory := t.TempDir() + signalsFile := filepath.Join(directory, SignalsFileName) + policy := &schema.ComponentErrors{Ignore: []schema.ComponentIgnorePolicy{{Name: "ignore", Patterns: []string{"alpha"}, Signals: map[string]any{"x": 1}}}} + failing := func() *recoveryRun { return &recoveryRun{failures: []error{errExit}, texts: []string{"alpha"}} } + + resolved := 0 + ignored, err := NewRecoverer(policy, WithSignalsPath(func() string { resolved++; return signalsFile })).Run(t.Context(), failing().attempt) + require.NoError(t, err) + require.True(t, ignored) + require.Equal(t, 1, resolved) + require.FileExists(t, signalsFile) + + require.NoError(t, os.Remove(signalsFile)) + ignored, err = NewRecoverer(policy, WithSignalsPath(func() string { return "" })).Run(t.Context(), failing().attempt) + require.NoError(t, err) + require.True(t, ignored) + require.NoFileExists(t, signalsFile, "an empty path writes no signals") +} diff --git a/pkg/component/execution/mock_reporter_test.go b/pkg/component/execution/mock_reporter_test.go new file mode 100644 index 00000000000..dc9a441f1df --- /dev/null +++ b/pkg/component/execution/mock_reporter_test.go @@ -0,0 +1,77 @@ +// Code generated by MockGen. DO NOT EDIT. +// Source: github.com/cloudposse/atmos/pkg/component/execution (interfaces: Reporter) +// +// Generated by this command: +// +// mockgen -destination=mock_reporter_test.go -package=execution github.com/cloudposse/atmos/pkg/component/execution Reporter +// + +// Package execution is a generated GoMock package. +package execution + +import ( + reflect "reflect" + time "time" + + gomock "go.uber.org/mock/gomock" +) + +// MockReporter is a mock of Reporter interface. +type MockReporter struct { + ctrl *gomock.Controller + recorder *MockReporterMockRecorder + isgomock struct{} +} + +// MockReporterMockRecorder is the mock recorder for MockReporter. +type MockReporterMockRecorder struct { + mock *MockReporter +} + +// NewMockReporter creates a new mock instance. +func NewMockReporter(ctrl *gomock.Controller) *MockReporter { + mock := &MockReporter{ctrl: ctrl} + mock.recorder = &MockReporterMockRecorder{mock} + return mock +} + +// EXPECT returns an object that allows the caller to indicate expected use. +func (m *MockReporter) EXPECT() *MockReporterMockRecorder { + return m.recorder +} + +// Ignored mocks base method. +func (m *MockReporter) Ignored(policy, message string) { + m.ctrl.T.Helper() + m.ctrl.Call(m, "Ignored", policy, message) +} + +// Ignored indicates an expected call of Ignored. +func (mr *MockReporterMockRecorder) Ignored(policy, message any) *gomock.Call { + mr.mock.ctrl.T.Helper() + return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "Ignored", reflect.TypeOf((*MockReporter)(nil).Ignored), policy, message) +} + +// Retrying mocks base method. +func (m *MockReporter) Retrying(policy string, attempt, maxAttempts int, sleep time.Duration) { + m.ctrl.T.Helper() + m.ctrl.Call(m, "Retrying", policy, attempt, maxAttempts, sleep) +} + +// Retrying indicates an expected call of Retrying. +func (mr *MockReporterMockRecorder) Retrying(policy, attempt, maxAttempts, sleep any) *gomock.Call { + mr.mock.ctrl.T.Helper() + return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "Retrying", reflect.TypeOf((*MockReporter)(nil).Retrying), policy, attempt, maxAttempts, sleep) +} + +// SignalsWritten mocks base method. +func (m *MockReporter) SignalsWritten(path string) { + m.ctrl.T.Helper() + m.ctrl.Call(m, "SignalsWritten", path) +} + +// SignalsWritten indicates an expected call of SignalsWritten. +func (mr *MockReporterMockRecorder) SignalsWritten(path any) *gomock.Call { + mr.mock.ctrl.T.Helper() + return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "SignalsWritten", reflect.TypeOf((*MockReporter)(nil).SignalsWritten), path) +} diff --git a/pkg/component/execution/recovery.go b/pkg/component/execution/recovery.go new file mode 100644 index 00000000000..0c247410592 --- /dev/null +++ b/pkg/component/execution/recovery.go @@ -0,0 +1,231 @@ +package execution + +import ( + "context" + "encoding/json" + "fmt" + "regexp" + "strings" + "time" + + errUtils "github.com/cloudposse/atmos/errors" + "github.com/cloudposse/atmos/pkg/ansi" + "github.com/cloudposse/atmos/pkg/filesystem" + log "github.com/cloudposse/atmos/pkg/logger" + "github.com/cloudposse/atmos/pkg/perf" + "github.com/cloudposse/atmos/pkg/schema" +) + +// signalsFilePermissions is the mode of the ignore signals file. +const signalsFilePermissions = 0o644 + +// SignalsFileName is the file that an ignore policy writes its signals to, in the module directory. +const SignalsFileName = "error-signals.json" + +// patternNoise matches everything that is not part of a readable error message. Terragrunt cleans +// error text this way before matching, so a pattern written for it matches here too. +var patternNoise = regexp.MustCompile(`[^a-zA-Z0-9./'"():=\- ]+`) + +// Reporter tells the user what error handling decided. Implementations print on the UI channel. +type Reporter interface { + // Retrying reports that the command failed with a retryable error and will run again. + Retrying(policy string, attempt, maxAttempts int, sleep time.Duration) + // Ignored reports that a failure was turned into a success. + Ignored(policy, message string) + // SignalsWritten reports where an ignore policy wrote its signals. + SignalsWritten(path string) +} + +// Recoverer applies a provider's errors policy to a failing command: an ignore policy turns the +// failure into a success, and a retry policy runs the command again after a pause. Ignore +// policies are checked before retry policies, each in declaration order, and the first match +// decides, like Terragrunt's errors block. +type Recoverer struct { + policy *schema.ComponentErrors + reporter Reporter + signalsFile string + // signalsPath, when set, supplies the signals file at the moment an ignore policy writes it. + signalsPath func() string + sleep func(ctx context.Context, duration time.Duration) error +} + +// RecovererOption configures a Recoverer. +type RecovererOption func(*Recoverer) + +// WithReporter sets where decisions are reported. +func WithReporter(reporter Reporter) RecovererOption { + defer perf.Track(nil, "execution.WithReporter")() + + return func(r *Recoverer) { r.reporter = reporter } +} + +// WithSignalsPath sets the file an ignore policy writes its signals to, resolved when the signals are +// written rather than when the recoverer is built. An empty path writes no signals. +func WithSignalsPath(resolve func() string) RecovererOption { + defer perf.Track(nil, "execution.WithSignalsPath")() + + return func(r *Recoverer) { r.signalsPath = resolve } +} + +// WithSignalsFile sets the file an ignore policy writes its signals to. Without it no signals are written. +func WithSignalsFile(path string) RecovererOption { + defer perf.Track(nil, "execution.WithSignalsFile")() + + return func(r *Recoverer) { r.signalsFile = path } +} + +// WithSleep replaces the pause between attempts, so tests do not wait. +func WithSleep(sleep func(ctx context.Context, duration time.Duration) error) RecovererOption { + defer perf.Track(nil, "execution.WithSleep")() + + return func(r *Recoverer) { r.sleep = sleep } +} + +// NewRecoverer builds a Recoverer for a policy. A nil or empty policy never recovers. +func NewRecoverer(policy *schema.ComponentErrors, options ...RecovererOption) *Recoverer { + defer perf.Track(nil, "execution.NewRecoverer")() + + recoverer := &Recoverer{policy: policy, sleep: sleepContext} + for _, option := range options { + option(recoverer) + } + return recoverer +} + +// Attempt runs the command once. It returns the error text the policies match against, usually the +// command's standard error, and the command's error. +type Attempt func(ctx context.Context) (text string, err error) + +// Run runs the attempt until it succeeds, fails with an error no policy covers, or exhausts a +// retry policy. It returns ignored = true when an ignore policy swallowed the failure. +func (r *Recoverer) Run(ctx context.Context, attempt Attempt) (ignored bool, err error) { + defer perf.Track(nil, "execution.Recoverer.Run")() + + for current := 1; ; current++ { + text, runErr := attempt(ctx) + if runErr == nil { + return false, nil + } + if r == nil || r.policy == nil || ctx.Err() != nil { + return false, runErr + } + if strings.TrimSpace(text) == "" { + text = runErr.Error() + } + retry, ignored, err := r.decide(ctx, cleanErrorText(text), runErr, current) + if !retry { + return ignored, err + } + } +} + +// decide applies the policies to one failure. It reports whether the command should run again; +// when it should not, ignored and err describe the final outcome. +func (r *Recoverer) decide(ctx context.Context, text string, runErr error, current int) (retry, ignored bool, err error) { + if policy := r.matchingIgnore(text); policy != nil { + r.ignore(policy) + return false, true, nil + } + policy := r.matchingRetry(text) + if policy == nil { + return false, false, runErr + } + if current >= policy.MaxAttempts { + return false, false, fmt.Errorf("%w (%d) for policy %q: %w", errUtils.ErrComponentMaxAttempts, policy.MaxAttempts, policy.Name, runErr) + } + if r.reporter != nil { + r.reporter.Retrying(policy.Name, current, policy.MaxAttempts, policy.SleepInterval) + } + if err := r.sleep(ctx, policy.SleepInterval); err != nil { + return false, false, err + } + return true, false, nil +} + +func (r *Recoverer) matchingIgnore(text string) *schema.ComponentIgnorePolicy { + for i := range r.policy.Ignore { + if matchesPatterns(text, r.policy.Ignore[i].Patterns) { + return &r.policy.Ignore[i] + } + } + return nil +} + +func (r *Recoverer) matchingRetry(text string) *schema.ComponentRetryPolicy { + for i := range r.policy.Retry { + if matchesPatterns(text, r.policy.Retry[i].Patterns) { + return &r.policy.Retry[i] + } + } + return nil +} + +// ignore reports the decision and writes the signals. A signals file that cannot be written is +// reported but never turns the ignored failure back into an error. +func (r *Recoverer) ignore(policy *schema.ComponentIgnorePolicy) { + if r.reporter != nil { + r.reporter.Ignored(policy.Name, policy.Message) + } + r.WriteSignals(policy) +} + +// WriteSignals writes signals for an already selected policy, retaining non-fatal write handling. +func (r *Recoverer) WriteSignals(policy *schema.ComponentIgnorePolicy) { + defer perf.Track(nil, "execution.Recoverer.WriteSignals")() + file := r.signalsFile + if r.signalsPath != nil { + file = r.signalsPath() + } + if len(policy.Signals) == 0 || file == "" { + return + } + data, err := json.MarshalIndent(policy.Signals, "", " ") + if err == nil { + err = filesystem.NewOSFileSystem().WriteFileAtomic(file, data, signalsFilePermissions) + } + if err != nil { + log.Warn("Failed to write error signals", "path", file, "error", err) + return + } + if r.reporter != nil { + r.reporter.SignalsWritten(file) + } +} + +// cleanErrorText removes terminal sequences and punctuation noise, then collapses whitespace. +func cleanErrorText(text string) string { + return strings.Join(strings.Fields(patternNoise.ReplaceAllString(ansi.Strip(text), " ")), " ") +} + +// matchesPatterns reports whether the text matches a positive pattern and no negative one. A +// pattern that starts with "!" is negative. +func matchesPatterns(text string, patterns []string) bool { + matched := false + for _, pattern := range patterns { + negative := strings.HasPrefix(pattern, "!") + expression, err := regexp.Compile(strings.TrimPrefix(pattern, "!")) + if err != nil || !expression.MatchString(text) { + continue + } + if negative { + return false + } + matched = true + } + return matched +} + +// sleepContext waits for the duration or until the context ends. +func sleepContext(ctx context.Context, duration time.Duration) error { + if duration <= 0 { + return ctx.Err() + } + timer := time.NewTimer(duration) + defer timer.Stop() + select { + case <-timer.C: + return nil + case <-ctx.Done(): + return ctx.Err() + } +} diff --git a/pkg/component/local_source_subdir_test.go b/pkg/component/local_source_subdir_test.go new file mode 100644 index 00000000000..e428b4847c0 --- /dev/null +++ b/pkg/component/local_source_subdir_test.go @@ -0,0 +1,86 @@ +package component + +import ( + "context" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/stretchr/testify/require" + + cfg "github.com/cloudposse/atmos/pkg/config" + "github.com/cloudposse/atmos/pkg/provisioner" + "github.com/cloudposse/atmos/pkg/provisioner/workdir" + "github.com/cloudposse/atmos/pkg/schema" +) + +func TestConfiguredLocalSourceSubdirSelectsModule(t *testing.T) { + root := t.TempDir() + module := filepath.Join(root, "modules", "app") + require.NoError(t, os.MkdirAll(module, 0o755)) + info := &schema.ConfigAndStacksInfo{BaseComponentPath: "identity-only", ComponentSection: map[string]any{ + "metadata": map[string]any{"source_subdir": "modules/app"}, + }} + selected, err := ResolveConfiguredSubpath(info, root) + require.NoError(t, err) + require.Equal(t, module, selected) + info.ComponentSection[workdir.WorkdirPathKey] = root + selected, err = ApplyWorkdirSubpathToSection(info) + require.NoError(t, err) + require.Equal(t, module, selected) + require.NoError(t, os.RemoveAll(module)) + _, err = ResolveConfiguredSubpath(info, root) + require.Error(t, err, "an explicit module path cannot fall back to the root") +} + +func TestPendingSourceSubdirResolvesBeforeTheSourceIsProvisioned(t *testing.T) { + root := t.TempDir() + info := &schema.ConfigAndStacksInfo{ComponentSection: map[string]any{"metadata": map[string]any{"source_subdir": "modules/app"}}} + expected := filepath.Join(root, "modules", "app") + + selected, present, err := ResolvePendingSubpath(info, root) + require.NoError(t, err) + require.False(t, present) + require.Equal(t, expected, selected, "a missing subdirectory resolves to the path it will have") + _, err = ResolveConfiguredSubpath(info, root) + require.Error(t, err, "the strict resolution after provisioning still requires the directory") + + require.NoError(t, os.MkdirAll(expected, 0o755)) + selected, present, err = ResolvePendingSubpath(info, root) + require.NoError(t, err) + require.True(t, present) + require.Equal(t, expected, selected) + + invalid := &schema.ConfigAndStacksInfo{ComponentSection: map[string]any{"metadata": map[string]any{"source_subdir": "../escape"}}} + _, _, err = ResolvePendingSubpath(invalid, root) + require.Error(t, err, "an invalid subdirectory is rejected whether or not the source is provisioned") + + require.NoError(t, os.WriteFile(filepath.Join(root, "file"), nil, 0o644)) + notDir := &schema.ConfigAndStacksInfo{ComponentSection: map[string]any{"metadata": map[string]any{"source_subdir": "file"}}} + _, _, err = ResolvePendingSubpath(notDir, root) + require.Error(t, err, "only a missing subdirectory is tolerated") +} + +func TestDryRunReportsAMissingSourceSubdirAsAbsent(t *testing.T) { + base := t.TempDir() + section := map[string]any{ + "component": "vpc", "atmos_component": "vpc", "atmos_stack": "dev", + "provision": map[string]any{"workdir": map[string]any{"enabled": true}}, + "source": map[string]any{"uri": "./module"}, + "metadata": map[string]any{"source_subdir": "modules/app"}, + } + info := &schema.ConfigAndStacksInfo{DryRun: true, FinalComponent: "vpc", Stack: "dev", ComponentSection: section} + + path, exists, err := ProvisionAndResolveComponentPath(context.Background(), provisioner.OutputWriters{}, &schema.AtmosConfiguration{BasePath: base}, info, cfg.TerraformComponentType, filepath.Join(base, "components", "vpc")) + + require.NoError(t, err) + require.False(t, exists) + suffix := string(filepath.Separator) + filepath.Join("modules", "app") + require.True(t, strings.HasSuffix(path, suffix), "the path names the missing module directory: %s", path) + require.Equal(t, strings.TrimSuffix(path, suffix), section[workdir.WorkdirPathKey], "the section keeps the workdir root for a later resolution") + require.NotContains(t, section, workdirSubpathAppliedKey) + entries, err := os.ReadDir(base) + require.NoError(t, err) + require.Empty(t, entries, "a dry run creates nothing") +} diff --git a/pkg/component/terraform/configuration.go b/pkg/component/terraform/configuration.go index ca111639bf3..2e9d6cea784 100644 --- a/pkg/component/terraform/configuration.go +++ b/pkg/component/terraform/configuration.go @@ -11,11 +11,8 @@ import ( "os" "os/exec" "path/filepath" - "runtime" - "slices" "strings" - "github.com/google/shlex" "github.com/hashicorp/hcl/v2" "github.com/hashicorp/hcl/v2/hclparse" @@ -24,7 +21,6 @@ import ( "github.com/cloudposse/atmos/pkg/component/configuration" cfg "github.com/cloudposse/atmos/pkg/config" "github.com/cloudposse/atmos/pkg/dependencies" - envpkg "github.com/cloudposse/atmos/pkg/env" "github.com/cloudposse/atmos/pkg/perf" "github.com/cloudposse/atmos/pkg/provisioner" "github.com/cloudposse/atmos/pkg/provisioner/workdir" @@ -50,28 +46,54 @@ func prepareConfiguration(ctx context.Context, config *schema.AtmosConfiguration if info.DryRun { return nil, nil } - if err := resolveHostExecutable(info); err != nil { + if err := resolveSelectedExecutable(info); err != nil { return nil, err } - prepared, execution, err := prepareOrRestore(ctx, config, info) + prepared, stagedIAM, err := restoreOrPrepareConfiguration(ctx, config, info) if err != nil || prepared == nil { return prepared, err } - if err = applyStackOverrides(config, info, prepared); err != nil { + if err = finalizePreparedConfiguration(ctx, config, info, prepared, stagedIAM); err != nil { return nil, err } - applyPreparedConfiguration(config, info, prepared, execution) - warnOnCacheLocalState(info, prepared) + applyPreparedConfiguration(config, info, prepared) return prepared, nil } -// applyPreparedConfiguration projects the prepared configuration onto the invocation. -func applyPreparedConfiguration(config *schema.AtmosConfiguration, info *schema.ConfigAndStacksInfo, prepared *schema.PreparedComponentConfiguration, execution *schema.ComponentExecution) { +func finalizePreparedConfiguration(ctx context.Context, config *schema.AtmosConfiguration, info *schema.ConfigAndStacksInfo, prepared *schema.PreparedComponentConfiguration, stagedIAM *schema.ComponentIAM) error { + if err := applyStackOverrides(config, info, prepared); err != nil { + return err + } + return finishIAM(ctx, info, prepared, stagedIAM) +} + +// Plugins do not inherit PATH. Resolve the selected host executable before +// Terragrunt evaluates version constraints or dependency outputs. +func resolveSelectedExecutable(info *schema.ConfigAndStacksInfo) error { + binary, err := exec.LookPath(info.Command) + if err != nil { + return errUtils.Build(fmt.Errorf("%w: Terraform executable of component %q in stack %q: %w", errUtils.ErrPluginInvalidConfig, info.ComponentFromArg, info.Stack, err)). + WithContext("component", info.ComponentFromArg). + WithContext("stack", info.Stack). + WithContext("executable", info.Command). + WithHint("The unit's `terraform_binary` or the component's `command` must name an installed executable, a toolchain tool from `dependencies.tools`, or an absolute path"). + Err() + } + binary, err = filepath.Abs(binary) + if err != nil { + return fmt.Errorf("%w: %w", errUtils.ErrPluginInvalidConfig, err) + } + info.Command = binary + return nil +} + +func applyPreparedConfiguration(_ *schema.AtmosConfiguration, info *schema.ConfigAndStacksInfo, prepared *schema.PreparedComponentConfiguration) { info.ComponentVarsSection = prepared.Inputs // The stack's env section wins over provider environment in the metadata this invocation runs, // including the apply-time metadata of a restored saved plan. - info.ComponentExecution = withoutStackEnvironment(execution, info.ComponentEnvSection) + info.ComponentExecution = withoutStackEnvironment(invocationExecution(info, prepared), info.ComponentEnvSection) info.ConfigurationSensitiveInputKeys = maps.Clone(prepared.SensitiveInputKeys) + info.SourceAuthorization = prepared.SourceAuthorization info.ComponentBackendType, info.ComponentBackendSection = prepared.BackendType, prepared.Backend info.TerraformWorkspace = prepared.Workspace info.ComponentSection["vars"] = prepared.Inputs @@ -86,7 +108,24 @@ func applyPreparedConfiguration(config *schema.AtmosConfiguration, info *schema. info.ComponentSection["source"] = source info.ComponentSection["backend_type"] = prepared.BackendType info.ComponentSection["backend"] = prepared.Backend - config.Components.Terraform.AutoGenerateBackendFile = !prepared.BackendGenerated + info.SetBackendFileGeneration(!prepared.BackendGenerated) + warnOnCacheLocalState(info, prepared) +} + +func restoreOrPrepareConfiguration(ctx context.Context, config *schema.AtmosConfiguration, info *schema.ConfigAndStacksInfo) (*schema.PreparedComponentConfiguration, *schema.ComponentIAM, error) { + if usesConfigurationPlan(info) { + if err := useRetainedWorkdir(config, info); err != nil { + return nil, nil, err + } + prepared, err := restoreConfigurationPlan(ctx, config, info) + return prepared, nil, err + } + staged, err := probeAndApplyIAM(ctx, config.ComponentConfigurations, info) + if err != nil { + return nil, nil, err + } + prepared, err := config.ComponentConfigurations.Prepare(ctx, info) + return prepared, staged, err } // terragruntCacheDirectory is where Terragrunt keeps its working copies of a unit. @@ -121,41 +160,13 @@ func warnOnCacheLocalState(info *schema.ConfigAndStacksInfo, prepared *schema.Pr "`path = \"${get_terragrunt_dir()}/terraform.tfstate\"`, and move the existing state file there.") } -// resolveHostExecutable makes info.Command the absolute path of the selected executable. -func resolveHostExecutable(info *schema.ConfigAndStacksInfo) error { - // Plugins do not inherit PATH. Resolve the selected host executable before - // Terragrunt evaluates version constraints or dependency outputs. - binary, err := exec.LookPath(info.Command) - if err != nil { - return fmt.Errorf("%w: Terraform executable: %w", errUtils.ErrPluginInvalidConfig, err) - } - binary, err = filepath.Abs(binary) - if err != nil { - return fmt.Errorf("%w: %w", errUtils.ErrPluginInvalidConfig, err) - } - info.Command = binary - return nil -} - -// prepareOrRestore prepares the component, or restores the reviewed preparation of a saved -// plan. It also returns the hooks and extra arguments the invocation runs: a saved plan was -// prepared for plan, so it runs the apply-time metadata recorded with it. -func prepareOrRestore(ctx context.Context, config *schema.AtmosConfiguration, info *schema.ConfigAndStacksInfo) (*schema.PreparedComponentConfiguration, *schema.ComponentExecution, error) { - if !usesConfigurationPlan(info) { - prepared, err := config.ComponentConfigurations.Prepare(ctx, info) - if err != nil || prepared == nil { - return prepared, nil, err - } - return prepared, prepared.Execution, nil - } - if err := useRetainedWorkdir(config, info); err != nil { - return nil, nil, err - } - prepared, err := restoreConfigurationPlan(ctx, config, info) - if err != nil || prepared == nil { - return prepared, nil, err +// invocationExecution returns the hooks and extra arguments the invocation runs. A saved plan was +// prepared for plan, so applying it runs the apply-time metadata recorded with it. +func invocationExecution(info *schema.ConfigAndStacksInfo, prepared *schema.PreparedComponentConfiguration) *schema.ComponentExecution { + if usesConfigurationPlan(info) { + return prepared.Execution.ForSavedPlanApply() } - return prepared, prepared.Execution.ForSavedPlanApply(), nil + return prepared.Execution } // useRetainedWorkdir points the invocation at the directory the reviewed plan was created in. @@ -302,7 +313,8 @@ func retainedBackendError(path string) error { func terraformConfigurationPath(path string) bool { lower := strings.ToLower(path) - return strings.HasSuffix(lower, ".tf") || strings.HasSuffix(lower, ".tf.json") + return strings.HasSuffix(lower, ".tf") || strings.HasSuffix(lower, ".tf.json") || + strings.HasSuffix(lower, ".tofu") || strings.HasSuffix(lower, ".tofu.json") } // configurationHasBackend also returns true for unparseable configuration: absence @@ -387,119 +399,6 @@ func isRuntimeArtifact(path string) bool { return false } -// validatePlanOutputFlags rejects ambiguous plan destinations before hooks or Terraform run. -func validatePlanOutputFlags(command string, args []string) error { - if command != "plan" { - return nil - } - count := 0 - for i := 0; i < len(args); i++ { - arg := args[i] - if arg == "-out" || arg == "--out" { - count++ - i++ - } else if strings.HasPrefix(arg, "-out=") || strings.HasPrefix(arg, "--out=") { - count++ - } - if count > 1 { - return fmt.Errorf("%w: multiple plan output flags", errUtils.ErrPlanVerificationFailed) - } - } - return nil -} - -// validateEnvironmentPlanOutput rejects plan output flags in TF_CLI_ARGS or -// TF_CLI_ARGS_plan for a configuration-owned plan. Terraform reads these -// variables itself, so a flag there could choose a plan file that Atmos does not -// see and therefore does not bind. The check inspects the effective value of each -// variable in the subprocess environment: the sanitized process environment (or -// the inherited one when auth did not sanitize it), the global atmos.yaml -// environment, the component environment, and provider extra-argument -// environment, with later layers overriding earlier ones. Hooks run as child -// processes and cannot change it. -func validateEnvironmentPlanOutput(atmosConfig *schema.AtmosConfiguration, info *schema.ConfigAndStacksInfo, execCtx *componentExecContext, opts ...component.ExecutionOption) error { - if execCtx.preparedConfiguration == nil || info.DryRun || info.SubCommand != subcommandPlan { - return nil - } - return rejectEnvironmentPlanOutput(planEnvironment(atmosConfig, info, opts), runtime.GOOS) -} - -// rejectEnvironmentPlanOutput rejects a plan output flag in the effective TF_CLI_ARGS or -// TF_CLI_ARGS_plan of an environment listed in precedence order. Variable names are -// case-insensitive on Windows only, where the last matching entry is the effective value. -func rejectEnvironmentPlanOutput(entries []string, goos string) error { - for _, name := range []string{tfCliArgsEnvVar, tfCliArgsEnvVar + "_" + subcommandPlan} { - value, _ := lastEnvListValue(entries, name, goos) - args, err := tokenizeEnvironmentArgs(value) - if err != nil { - return fmt.Errorf("%w: %s cannot be parsed: %w", errUtils.ErrPlanVerificationFailed, name, err) - } - if containsOutputFlag(args) { - return fmt.Errorf("%w: %s sets a plan output flag; pass -out as a command argument instead", errUtils.ErrPlanVerificationFailed, name) - } - } - return nil -} - -// tokenizeEnvironmentArgs splits a TF_CLI_ARGS value the way the engine does: -// spaces, tabs, and newlines separate tokens outside quotes, and a malformed -// value (for example an unterminated quote) is an error because the engine -// refuses to run with it. The shared parseArgs only splits on spaces, so a tab -// before -out would hide the flag from the check. -func tokenizeEnvironmentArgs(value string) ([]string, error) { - return shlex.Split(value) -} - -// planEnvironment lists the environment layers a plan subprocess receives, in precedence order. -func planEnvironment(atmosConfig *schema.AtmosConfiguration, info *schema.ConfigAndStacksInfo, opts []component.ExecutionOption) []string { - entries := slices.Clone(planBaseEnvironment(info, opts)) - if atmosConfig != nil { - entries = envpkg.MergeGlobalEnv(entries, atmosConfig.Env) - } - entries = append(entries, info.ComponentEnvList...) - if info.ComponentExecution == nil { - return entries - } - for i := range info.ComponentExecution.ExtraArguments { - item := &info.ComponentExecution.ExtraArguments[i] - if !slices.Contains(item.Commands, subcommandPlan) { - continue - } - for name, value := range item.Environment { - entries = append(entries, name+"="+value) - } - } - return entries -} - -// containsOutputFlag reports any plan output flag, with or without a value. -func containsOutputFlag(args []string) bool { - return slices.ContainsFunc(args, func(arg string) bool { - return arg == "-out" || arg == "--out" || strings.HasPrefix(arg, "-out=") || strings.HasPrefix(arg, "--out=") - }) -} - -func plannedOutputPath(directory string, args []string) string { - for i, arg := range args { - path := "" - switch { - case (arg == "-out" || arg == "--out") && i+1 < len(args): - path = args[i+1] - case strings.HasPrefix(arg, "-out="): - path = strings.TrimPrefix(arg, "-out=") - case strings.HasPrefix(arg, "--out="): - path = strings.TrimPrefix(arg, "--out=") - } - if path != "" { - if filepath.IsAbs(path) { - return path - } - return filepath.Join(directory, path) - } - } - return "" -} - // planSnapshot is the configuration a saved plan is bound to: the provider identity and the // digest of the retained working directory, excluding the plan and the state Terraform writes. type planSnapshot struct { diff --git a/pkg/component/terraform/configuration_backend_test.go b/pkg/component/terraform/configuration_backend_test.go index 5554c017728..d40b7003e45 100644 --- a/pkg/component/terraform/configuration_backend_test.go +++ b/pkg/component/terraform/configuration_backend_test.go @@ -65,6 +65,12 @@ func TestPreparedConfigurationRetainedBackend(t *testing.T) { {"non backend", "provider.tf", `locals { enabled = true }`, `locals { enabled = false }`, "skip", false}, {"auxiliary", "data.json", `{"value":true}`, `{"value":false}`, "skip", false}, {"overwrite", "remote.tf", expected, `locals { enabled = true }`, "overwrite", false}, + {"OpenTofu missing backend", "remote.tofu", expected, `locals { enabled = true }`, "skip", true}, + {"OpenTofu json backend", "remote.tofu.json", `{"terraform":{"backend":{"local":{"path":"expected"}}}}`, `{"terraform":{"backend":{"local":{"path":"other"}}}}`, "skip", true}, + {"OpenTofu invalid retained", "remote.tofu", expected, `terraform {`, "skip", true}, + {"OpenTofu non backend", "provider.tofu", `locals { enabled = true }`, `locals { enabled = false }`, "skip", false}, + {"OpenTofu json non backend", "provider.tofu.json", `{"locals":{"enabled":true}}`, `{"locals":{"enabled":false}}`, "skip", false}, + {"OpenTofu overwrite", "remote.tofu", expected, `locals { enabled = true }`, "overwrite", false}, } { t.Run(test.name, func(t *testing.T) { directory := t.TempDir() @@ -88,6 +94,56 @@ func TestPreparedConfigurationRetainedBackend(t *testing.T) { } } +// Terragrunt writes `remote_state.encryption` only into the file `remote_state.generate` produces, +// together with the backend block. A different file kept by if_exists = "skip" would drop the +// encryption and let OpenTofu write plain state, so it is refused before anything is written. +func TestPreparedConfigurationRetainedBackendKeepsEncryption(t *testing.T) { + const generated = `terraform { + backend "local" { + path = "expected.tfstate" + } + encryption { + key_provider "pbkdf2" "default" { + passphrase = "correct-horse-battery-staple" + } + } +}` + const generatedJSON = `{"terraform":{"backend":{"local":{"path":"expected.tfstate"}},` + + `"encryption":{"key_provider":{"pbkdf2":{"default":{"passphrase":"correct-horse-battery-staple"}}}}}}` + const unencrypted = "terraform {\n backend \"local\" {\n path = \"expected.tfstate\"\n }\n}" + for _, test := range []struct { + name, path, generated, retained string + blocked bool + }{ + {"same backend without encryption", "backend.tf", generated, unencrypted, true}, + {"no backend and no encryption", "backend.tf", generated, "locals { enabled = true }", true}, + {"identical encrypted backend", "backend.tf", generated, generated, false}, + {"OpenTofu same backend without encryption", "backend.tofu", generated, unencrypted, true}, + {"OpenTofu no backend and no encryption", "backend.tofu", generated, "locals { enabled = true }", true}, + {"OpenTofu identical encrypted backend", "backend.tofu", generated, generated, false}, + {"OpenTofu json same backend without encryption", "backend.tofu.json", generatedJSON, `{"terraform":{"backend":{"local":{"path":"expected.tfstate"}}}}`, true}, + {"OpenTofu json no backend and no encryption", "backend.tofu.json", generatedJSON, `{"locals":{"enabled":true}}`, true}, + {"OpenTofu json identical encrypted backend", "backend.tofu.json", generatedJSON, generatedJSON, false}, + } { + t.Run(test.name, func(t *testing.T) { + directory := t.TempDir() + path := filepath.Join(directory, test.path) + require.NoError(t, os.WriteFile(path, []byte(test.retained), 0o600)) + prepared := &schema.PreparedComponentConfiguration{BackendGenerated: true, Files: []schema.ComponentConfigurationFile{{Path: test.path, Content: []byte(test.generated), IfExists: "skip"}}} + err := writePreparedConfiguration(directory, prepared) + if !test.blocked { + require.NoError(t, err) + return + } + require.ErrorIs(t, err, errUtils.ErrPluginInvalidConfig) + assert.NotContains(t, err.Error(), "correct-horse-battery-staple", "the error never shows the generated secrets") + content, readErr := os.ReadFile(path) + require.NoError(t, readErr) + assert.Equal(t, test.retained, string(content), "the retained file is left alone") + }) + } +} + func TestRetainedBackendFailurePreservesHostBackend(t *testing.T) { directory := t.TempDir() config := &schema.AtmosConfiguration{} diff --git a/pkg/component/terraform/configuration_iam.go b/pkg/component/terraform/configuration_iam.go new file mode 100644 index 00000000000..761becef2b8 --- /dev/null +++ b/pkg/component/terraform/configuration_iam.go @@ -0,0 +1,135 @@ +package terraform + +import ( + "context" + "fmt" + "maps" + "os" + "reflect" + "slices" + "strings" + "time" + + errUtils "github.com/cloudposse/atmos/errors" + awsidentity "github.com/cloudposse/atmos/pkg/auth/identities/aws" + cfg "github.com/cloudposse/atmos/pkg/config" + "github.com/cloudposse/atmos/pkg/schema" + "github.com/cloudposse/atmos/pkg/ui" +) + +func probeAndApplyIAM(ctx context.Context, runtime schema.ComponentConfigurationRuntime, info *schema.ConfigAndStacksInfo) (*schema.ComponentIAM, error) { + if requestedIdentity(info) != "" { + return nil, nil + } + probe, ok := runtime.(schema.ComponentConfigurationIAMProbe) + if !ok { + return nil, nil + } + iam, err := probe.ProbeIAM(ctx, info) + if err != nil || iam == nil { + return iam, err + } + if err := applyIAMRole(ctx, info, iam); err != nil { + return nil, err + } + return iam, nil +} + +func finishIAM(ctx context.Context, info *schema.ConfigAndStacksInfo, prepared *schema.PreparedComponentConfiguration, staged *schema.ComponentIAM) error { + if staged != nil { + if !reflect.DeepEqual(staged, prepared.IAM) { + return fmt.Errorf("%w: Terragrunt IAM settings changed between role and unit evaluation", errUtils.ErrPluginInvalidConfig) + } + return nil + } + // An operation the provider excludes, or ends because a failure was ignored, never runs, so it must + // not need the unit's role. + if excludedFromOperation(info, prepared) || prepared.IgnoredRun != nil { + return nil + } + return applyIAMRole(ctx, info, prepared.IAM) +} + +// assumeInlineRole is replaced by tests that must not call STS. +var assumeInlineRole = awsidentity.AssumeInlineRole + +// applyIAMRole assumes the role a component asks for, on top of the credentials of the invocation, and +// exports the session to every command the component runs, and to in-process source downloads, as Terragrunt +// fetches a unit's source with the assumed role. An Atmos identity that the user selected +// explicitly decides who the component runs as, so it replaces the role instead of being layered under it. +// The session lives only in the environment of this invocation: it is neither written to a file nor to a +// saved plan, and it is registered with the output masker. +func applyIAMRole(ctx context.Context, info *schema.ConfigAndStacksInfo, iam *schema.ComponentIAM) error { + if iam == nil { + return nil + } + if requested := requestedIdentity(info); requested != "" { + ui.Infof("Component `%s` in stack `%s` runs as Atmos identity `%s`; its Terragrunt `iam_role` `%s` is not assumed.", info.ComponentFromArg, info.Stack, info.Identity, iam.Role) + return nil + } + token, err := webIdentityToken(iam) + if err != nil { + return iamRoleError(info, iam, err) + } + role := awsidentity.InlineRole{Role: iam.Role, SessionName: iam.SessionName, Duration: time.Duration(iam.DurationSeconds) * time.Second, WebIdentityToken: token} + base := info.SanitizedEnv + if base == nil { + base = os.Environ() + } + session, err := assumeInlineRole(ctx, role, append(slices.Clone(base), info.ComponentEnvList...)) + if err != nil { + return iamRoleError(info, iam, err) + } + for _, key := range slices.Sorted(maps.Keys(session)) { + info.ComponentEnvList = append(info.ComponentEnvList, key+"="+session[key]) + } + info.IAMSession = session + return nil +} + +// webIdentityToken returns the token to assume the role with: read from its file when one is set, +// otherwise the literal value. A literal token is not stored with a saved plan, so applying such a +// plan fails here instead of assuming the role with the base credentials. +func webIdentityToken(iam *schema.ComponentIAM) (string, error) { + token := iam.WebIdentityToken + if iam.WebIdentityTokenFile != "" { + content, err := os.ReadFile(iam.WebIdentityTokenFile) + if err != nil { + return "", fmt.Errorf("%w: web identity token file: %w", errUtils.ErrAuthenticationFailed, err) + } + token = strings.TrimSpace(string(content)) + if token == "" { + return "", fmt.Errorf("%w: web identity token file %s is empty", errUtils.ErrAuthenticationFailed, iam.WebIdentityTokenFile) + } + } + if iam.WebIdentityTokenLiteral && token == "" { + return "", fmt.Errorf("%w: the web identity token is not stored with a saved plan", errUtils.ErrAuthenticationFailed) + } + return token, nil +} + +// requestedIdentity returns what the user asked for with --identity or ATMOS_IDENTITY, or empty when no +// identity was requested. A default identity that authentication selected on its own is not a request, so +// the role is assumed on top of it. Disabling authentication requests nothing either. +func requestedIdentity(info *schema.ConfigAndStacksInfo) string { + requested := info.Identity + if info.RequestedIdentity != nil { + requested = *info.RequestedIdentity + } + if requested == cfg.IdentityFlagDisabledValue { + return "" + } + return requested +} + +func iamRoleError(info *schema.ConfigAndStacksInfo, iam *schema.ComponentIAM, cause error) error { + return errUtils.Build(errUtils.ErrAuthenticationFailed). + WithCause(cause). + WithExplanationf("Component `%s` in stack `%s` sets the Terragrunt `iam_role` `%s`, and Atmos could not assume it", info.ComponentFromArg, info.Stack, iam.Role). + WithHint("Check that the credentials in use may assume the role, and that its trust policy allows them"). + WithHint("To run as an Atmos identity instead of assuming `iam_role`, select it with `--identity `"). + WithHint("A web identity token must be a file path when a plan is saved and applied later"). + WithContext("component", info.ComponentFromArg). + WithContext("stack", info.Stack). + Err() +} diff --git a/pkg/component/terraform/configuration_overrides_test.go b/pkg/component/terraform/configuration_overrides_test.go index 8a7d1e963ec..386172fbb0e 100644 --- a/pkg/component/terraform/configuration_overrides_test.go +++ b/pkg/component/terraform/configuration_overrides_test.go @@ -179,6 +179,29 @@ func TestApplyStackOverridesReportsMergeFailures(t *testing.T) { require.Error(t, err) } +func TestPrepareConfigurationDoesNotAssumeTheRoleOfAnExcludedOperation(t *testing.T) { + recorded := fakeAssume(t, nil) + runtime := NewMockComponentConfigurationRuntime(gomock.NewController(t)) + config := &schema.AtmosConfiguration{ComponentConfigurations: runtime} + info := overrideInfo(t) + prepared := &schema.PreparedComponentConfiguration{ + Engine: "terraform", Source: "./module", BackendType: "local", Backend: map[string]any{}, Workspace: "default", Inputs: map[string]any{}, + IAM: &schema.ComponentIAM{Role: "arn:aws:iam::1:role/x"}, + Exclusion: &schema.ComponentExclusion{Actions: []string{"plan"}}, + } + runtime.EXPECT().Prepare(gomock.Any(), info).Return(prepared, nil) + _, err := prepareConfiguration(t.Context(), config, info) + require.NoError(t, err) + require.Zero(t, recorded.calls, "an excluded operation must not assume the unit's role") + + info = overrideInfo(t) + prepared.Exclusion = &schema.ComponentExclusion{Actions: []string{"apply"}} + runtime.EXPECT().Prepare(gomock.Any(), info).Return(prepared, nil) + _, err = prepareConfiguration(t.Context(), config, info) + require.NoError(t, err) + require.Equal(t, 1, recorded.calls, "an operation the block does not exclude still assumes the role") +} + // A component's own settings.list_merge_strategy governs how its stack vars combine with the // provider's inputs, whatever the global strategy is, and the shared configuration is not modified. func TestApplyStackOverridesHonorsTheComponentListMergeStrategy(t *testing.T) { diff --git a/pkg/component/terraform/configuration_plan_test.go b/pkg/component/terraform/configuration_plan_test.go index b3e8494ce31..b7b1e1c6543 100644 --- a/pkg/component/terraform/configuration_plan_test.go +++ b/pkg/component/terraform/configuration_plan_test.go @@ -650,6 +650,44 @@ func TestDetailedExitCodePlanRecordsReviewedPlan(t *testing.T) { } } +// An ignored failure of the main command ends the run successfully, without a plan to bind. A plan +// that completed is still bound when an errors policy is present but does not match. +func TestIgnoredPlanFailureDoesNotRecordAPlanBinding(t *testing.T) { + for _, test := range []struct { + name string + ignore string + }{ + {name: "ignored failure", ignore: "backend unavailable"}, + {name: "unmatched policy", ignore: "never matches"}, + } { + t.Run(test.name, func(t *testing.T) { + config, info, execCtx, _, _ := planFixture(t) + info.ComponentExecution = &schema.ComponentExecution{Errors: &schema.ComponentErrors{ + Ignore: []schema.ComponentIgnorePolicy{{Name: "ignored", Patterns: []string{test.ignore}}}, + }} + // The failed plan writes no artifact. + shell := newShellFixture(t, func(call *shellCall) error { + if call.options.StderrCapture != nil { + _, _ = io.WriteString(call.options.StderrCapture, "Error: backend unavailable") + } + return errUtils.ExitCodeError{Code: 1} + }) + executor := testExecutor(t) + executor.services.ExecuteShellCommand = shell.mock.Execute + path := filepath.Join(execCtx.componentPath, "failed.plan") + + err := executor.runMainCommandPhase(config, info, execCtx, []string{"plan", "-out=" + path}, execCtx.componentPath, false) + + if test.ignore == "backend unavailable" { + require.NoError(t, err, "the ignored failure completes the run") + } else { + require.Error(t, err, "a failure that no policy ignores still fails") + } + require.NoFileExists(t, path+configurationPlanSuffix) + }) + } +} + // newPlanShowRunner returns a generated runner whose plan show prints an empty JSON plan. func newPlanShowRunner(t *testing.T) process.Runner { t.Helper() diff --git a/pkg/component/terraform/configuration_source_pipeline_test.go b/pkg/component/terraform/configuration_source_pipeline_test.go index f667df0a70d..e2647d829f0 100644 --- a/pkg/component/terraform/configuration_source_pipeline_test.go +++ b/pkg/component/terraform/configuration_source_pipeline_test.go @@ -93,7 +93,7 @@ func TestConfigurationSourceSymlinkRejectedBeforeInitAndMain(t *testing.T) { if err == nil { err = executor.executeCommandPipeline(project.config, info, execution) } - require.Equal(t, 1, provider.prepares, "must reach the real session's preparation boundary") + require.Equal(t, 2, provider.prepares, "must reach the IAM probe and full preparation boundaries") if escaping { require.ErrorIs(t, err, errUtils.ErrPluginInvalidConfig) require.Nil(t, execution) diff --git a/pkg/component/terraform/configuration_test.go b/pkg/component/terraform/configuration_test.go index 93727312294..642165862cb 100644 --- a/pkg/component/terraform/configuration_test.go +++ b/pkg/component/terraform/configuration_test.go @@ -1,17 +1,22 @@ package terraform import ( + "context" + "encoding/json" "errors" "os" "path/filepath" "testing" + "time" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" "go.uber.org/mock/gomock" errUtils "github.com/cloudposse/atmos/errors" + awsidentity "github.com/cloudposse/atmos/pkg/auth/identities/aws" componentpkg "github.com/cloudposse/atmos/pkg/component" + cfg "github.com/cloudposse/atmos/pkg/config" atmosio "github.com/cloudposse/atmos/pkg/io" "github.com/cloudposse/atmos/pkg/schema" ) @@ -35,7 +40,32 @@ func TestPrepareConfiguration(t *testing.T) { require.Equal(t, map[string]any{"uri": expected.Source}, info.ComponentSection["source"]) require.Equal(t, expected.Backend, info.ComponentBackendSection) require.Equal(t, "default", info.TerraformWorkspace) + require.False(t, config.Components.Terraform.AutoGenerateBackendFile, "preparation leaves the shared project default unchanged") + require.True(t, schema.GenerateBackendFile(config, info)) +} + +func TestPreparedBackendChoiceDoesNotLeakBetweenInvocations(t *testing.T) { + config := &schema.AtmosConfiguration{} + config.Components.Terraform.AutoGenerateBackendFile = true + newInfo := func() *schema.ConfigAndStacksInfo { + return &schema.ConfigAndStacksInfo{SubCommand: "plan", ComponentSection: map[string]any{}, ComponentBackendType: "local", ComponentBackendSection: map[string]any{}} + } + providerOwned := newInfo() + applyPreparedConfiguration(config, providerOwned, &schema.PreparedComponentConfiguration{BackendGenerated: true, BackendType: "local", Backend: map[string]any{}}) + hostOwned := newInfo() + applyPreparedConfiguration(config, hostOwned, &schema.PreparedComponentConfiguration{BackendGenerated: false, BackendType: "local", Backend: map[string]any{}}) require.True(t, config.Components.Terraform.AutoGenerateBackendFile) + require.False(t, schema.GenerateBackendFile(config, providerOwned)) + require.True(t, schema.GenerateBackendFile(config, hostOwned)) + + providerDir := t.TempDir() + hostDir := t.TempDir() + require.NoError(t, GenerateBackendConfig(config, providerOwned, providerDir)) + require.NoFileExists(t, filepath.Join(providerDir, "backend.tf.json")) + require.NoError(t, GenerateBackendConfig(config, hostOwned, hostDir)) + require.FileExists(t, filepath.Join(hostDir, "backend.tf.json")) + require.NoError(t, GenerateBackendConfig(config, providerOwned, providerDir)) + require.NoFileExists(t, filepath.Join(providerDir, "backend.tf.json")) } func TestPrepareConfigurationNeverForcesSourceRefresh(t *testing.T) { @@ -84,12 +114,15 @@ func TestPrepareConfigurationGuards(t *testing.T) { } { t.Run(test.name, func(t *testing.T) { config := &schema.AtmosConfiguration{} + config.Components.Terraform.AutoGenerateBackendFile = true if test.runtime { config.ComponentConfigurations = NewMockComponentConfigurationRuntime(gomock.NewController(t)) } result, err := prepareConfiguration(t.Context(), config, &test.info) require.ErrorIs(t, err, test.want) require.Nil(t, result) + require.Nil(t, test.info.BackendFileGeneration) + require.True(t, config.Components.Terraform.AutoGenerateBackendFile) }) } } @@ -192,6 +225,175 @@ func TestPrepareConfigurationKeepsCacheLocalStateLocation(t *testing.T) { require.Equal(t, cache, info.ComponentBackendSection["path"]) } +// fakeAssume replaces STS and records what it was asked to assume and with which environment. +func fakeAssume(t *testing.T, err error) *struct { + role awsidentity.InlineRole + environ []string + calls int +} { + t.Helper() + recorded := &struct { + role awsidentity.InlineRole + environ []string + calls int + }{} + previous := assumeInlineRole + assumeInlineRole = func(_ context.Context, role awsidentity.InlineRole, environ []string) (map[string]string, error) { + recorded.role, recorded.environ, recorded.calls = role, environ, recorded.calls+1 + if err != nil { + return nil, err + } + return map[string]string{"AWS_ACCESS_KEY_ID": "ASIASESSION", "AWS_SESSION_TOKEN": "session-token", "AWS_PROFILE": ""}, nil + } + t.Cleanup(func() { assumeInlineRole = previous }) + return recorded +} + +// iamProbingRuntime is a configuration runtime whose provider also probes IAM before preparation. +type iamProbingRuntime interface { + schema.ComponentConfigurationRuntime + schema.ComponentConfigurationIAMProbe +} + +var _ iamProbingRuntime = (*MockIAMProbingRuntime)(nil) + +//go:generate go run go.uber.org/mock/mockgen@v0.6.0 -source=$GOFILE -destination=mock_iam_probing_runtime_test.go -package=$GOPACKAGE -mock_names=iamProbingRuntime=MockIAMProbingRuntime + +func TestPrepareConfigurationStagesIAMForEvaluation(t *testing.T) { + recorded := fakeAssume(t, nil) + binary, err := os.Executable() + require.NoError(t, err) + iam := &schema.ComponentIAM{Role: "arn:aws:iam::1:role/x"} + runtime := NewMockIAMProbingRuntime(gomock.NewController(t)) + runtime.EXPECT().ProbeIAM(gomock.Any(), gomock.Any()).Return(iam, nil).Times(1) + runtime.EXPECT().Prepare(gomock.Any(), gomock.Any()).DoAndReturn(func(_ context.Context, info *schema.ConfigAndStacksInfo) (*schema.PreparedComponentConfiguration, error) { + require.Contains(t, info.ComponentEnvList, "AWS_ACCESS_KEY_ID=ASIASESSION", "the probed role is assumed before the unit is evaluated") + return &schema.PreparedComponentConfiguration{Engine: "terraform", Source: "./module", IAM: iam}, nil + }) + config := &schema.AtmosConfiguration{ComponentConfigurations: runtime} + info := &schema.ConfigAndStacksInfo{Command: binary, SubCommand: "plan", ComponentSection: map[string]any{}, ComponentSettingsSection: map[string]any{"configuration": true}, SanitizedEnv: []string{"AWS_ACCESS_KEY_ID=BASE", "AWS_SECRET_ACCESS_KEY=base-secret"}} + _, err = prepareConfiguration(t.Context(), config, info) + require.NoError(t, err) + require.Equal(t, 1, recorded.calls, "the staged session is reused for execution") + require.Contains(t, recorded.environ, "AWS_ACCESS_KEY_ID=BASE") +} + +func TestPrepareConfigurationRejectsChangedIAMAfterStaging(t *testing.T) { + fakeAssume(t, nil) + binary, err := os.Executable() + require.NoError(t, err) + runtime := NewMockIAMProbingRuntime(gomock.NewController(t)) + runtime.EXPECT().ProbeIAM(gomock.Any(), gomock.Any()).Return(&schema.ComponentIAM{Role: "role-a"}, nil) + runtime.EXPECT().Prepare(gomock.Any(), gomock.Any()).Return(&schema.PreparedComponentConfiguration{Engine: "terraform", Source: "./module", IAM: &schema.ComponentIAM{Role: "role-b"}}, nil) + info := &schema.ConfigAndStacksInfo{Command: binary, SubCommand: "plan", ComponentSection: map[string]any{}, ComponentSettingsSection: map[string]any{"configuration": true}} + _, err = prepareConfiguration(t.Context(), &schema.AtmosConfiguration{ComponentConfigurations: runtime}, info) + require.ErrorContains(t, err, "changed between role and unit evaluation") +} + +func TestApplyIAMRoleExportsTheSessionToTheComponent(t *testing.T) { + recorded := fakeAssume(t, nil) + info := &schema.ConfigAndStacksInfo{Stack: "live/dev", ComponentFromArg: "app", SanitizedEnv: []string{"AWS_PROFILE=atmos", "PATH=/bin"}, ComponentEnvList: []string{"TF_IN_AUTOMATION=true"}} + iam := &schema.ComponentIAM{Role: "arn:aws:iam::1:role/x", SessionName: "s", DurationSeconds: 1800} + require.NoError(t, applyIAMRole(t.Context(), info, iam)) + require.Equal(t, 1, recorded.calls) + require.Equal(t, awsidentity.InlineRole{Role: "arn:aws:iam::1:role/x", SessionName: "s", Duration: 30 * time.Minute}, recorded.role) + require.Equal(t, []string{"AWS_PROFILE=atmos", "PATH=/bin", "TF_IN_AUTOMATION=true"}, recorded.environ, "the role is assumed on top of the invocation's own credentials") + require.Equal(t, []string{"TF_IN_AUTOMATION=true", "AWS_ACCESS_KEY_ID=ASIASESSION", "AWS_PROFILE=", "AWS_SESSION_TOKEN=session-token"}, info.ComponentEnvList) + require.Equal(t, map[string]string{"AWS_ACCESS_KEY_ID": "ASIASESSION", "AWS_SESSION_TOKEN": "session-token", "AWS_PROFILE": ""}, info.IAMSession, "source downloads use the same session") +} + +func TestApplyIAMRoleIsSkippedForAnExplicitIdentity(t *testing.T) { + recorded := fakeAssume(t, nil) + asked := func(identity string) *string { return &identity } + iam := &schema.ComponentIAM{Role: "arn:aws:iam::1:role/x"} + + for name, info := range map[string]*schema.ConfigAndStacksInfo{ + "requested with --identity": {Identity: "deployer", RequestedIdentity: asked("deployer")}, + "requested interactively": {Identity: "picked", RequestedIdentity: asked(cfg.IdentityFlagSelectValue)}, + "identity without a recorder": {Identity: "deployer"}, + } { + t.Run(name, func(t *testing.T) { + info.Stack, info.ComponentFromArg = "live/dev", "app" + require.NoError(t, applyIAMRole(t.Context(), info, iam)) + require.Empty(t, info.ComponentEnvList, "the selected identity decides who the component runs as") + require.Nil(t, info.IAMSession, "source downloads keep the selected identity") + }) + } + require.Zero(t, recorded.calls) + + // A default identity that authentication chose, and disabled authentication, request nothing: the + // role is assumed on top of whatever credentials the invocation has. + for name, info := range map[string]*schema.ConfigAndStacksInfo{ + "default identity chosen by authentication": {Identity: "mock-identity", RequestedIdentity: asked("")}, + "authentication disabled": {Identity: cfg.IdentityFlagDisabledValue, RequestedIdentity: asked(cfg.IdentityFlagDisabledValue)}, + "no identity": {}, + } { + t.Run(name, func(t *testing.T) { + before := recorded.calls + info.Stack, info.ComponentFromArg = "live/dev", "app" + require.NoError(t, applyIAMRole(t.Context(), info, iam)) + require.Equal(t, before+1, recorded.calls) + }) + } +} + +func TestApplyIAMRoleWithoutARoleDoesNothing(t *testing.T) { + recorded := fakeAssume(t, nil) + info := &schema.ConfigAndStacksInfo{} + require.NoError(t, applyIAMRole(t.Context(), info, nil)) + require.Zero(t, recorded.calls) + require.Empty(t, info.ComponentEnvList) +} + +func TestApplyIAMRoleReadsTheTokenFileAtAssumptionTime(t *testing.T) { + recorded := fakeAssume(t, nil) + file := filepath.Join(t.TempDir(), "token") + require.NoError(t, os.WriteFile(file, []byte(" file-token \n"), 0o600)) + info := &schema.ConfigAndStacksInfo{Stack: "s", ComponentFromArg: "app"} + require.NoError(t, applyIAMRole(t.Context(), info, &schema.ComponentIAM{Role: "r", WebIdentityTokenFile: file})) + require.Equal(t, "file-token", recorded.role.WebIdentityToken) + + require.NoError(t, applyIAMRole(t.Context(), info, &schema.ComponentIAM{Role: "r", WebIdentityToken: "literal", WebIdentityTokenLiteral: true})) + require.Equal(t, "literal", recorded.role.WebIdentityToken) +} + +func TestApplyIAMRoleFailuresNameTheRoleAndLeakNothing(t *testing.T) { + info := &schema.ConfigAndStacksInfo{Stack: "live/dev", ComponentFromArg: "app"} + for name, test := range map[string]struct { + iam *schema.ComponentIAM + assume error + }{ + "STS rejects the role": {&schema.ComponentIAM{Role: "arn:aws:iam::1:role/x"}, errUtils.ErrAuthenticationFailed}, + "missing token file": {&schema.ComponentIAM{Role: "arn:aws:iam::1:role/x", WebIdentityTokenFile: filepath.Join(t.TempDir(), "absent")}, nil}, + "literal token lost with a plan": {&schema.ComponentIAM{Role: "arn:aws:iam::1:role/x", WebIdentityTokenLiteral: true}, nil}, + "empty token file": {&schema.ComponentIAM{Role: "arn:aws:iam::1:role/x", WebIdentityTokenFile: emptyTokenFile(t)}, nil}, + } { + t.Run(name, func(t *testing.T) { + fakeAssume(t, test.assume) + err := applyIAMRole(t.Context(), info, test.iam) + require.ErrorIs(t, err, errUtils.ErrAuthenticationFailed) + rendered := errUtils.Format(err, errUtils.FormatterConfig{}) + require.Contains(t, rendered, "arn:aws:iam::1:role/x") + require.Contains(t, rendered, "--identity") + }) + } +} + +func TestComponentIAMNeverSerializesALiteralToken(t *testing.T) { + prepared := &schema.PreparedComponentConfiguration{IAM: &schema.ComponentIAM{Role: "r", WebIdentityToken: "super-secret-token", WebIdentityTokenLiteral: true}} + record, err := json.Marshal(prepared) + require.NoError(t, err) + require.NotContains(t, string(record), "super-secret-token") + require.Contains(t, string(record), "WebIdentityTokenLiteral") +} + +func emptyTokenFile(t *testing.T) string { + t.Helper() + file := filepath.Join(t.TempDir(), "token") + require.NoError(t, os.WriteFile(file, []byte(" \n"), 0o600)) + return file +} + func TestRejectRuntimeArtifacts(t *testing.T) { require.NoError(t, rejectRuntimeArtifacts([]schema.ComponentConfigurationFile{{Path: "backend.tf", IfExists: "overwrite"}})) for _, path := range []string{"examples/terraform.tfstate", "nested/provider.tf"} { diff --git a/pkg/component/terraform/configuration_workdir_test.go b/pkg/component/terraform/configuration_workdir_test.go index e9bdd1fa718..6d0fdf792f8 100644 --- a/pkg/component/terraform/configuration_workdir_test.go +++ b/pkg/component/terraform/configuration_workdir_test.go @@ -3,6 +3,7 @@ package terraform import ( "context" "encoding/json" + "errors" "os" "path/filepath" "testing" @@ -115,3 +116,36 @@ func TestConfigurationExplicitWorkdirRejectsAnotherOwner(t *testing.T) { require.Equal(t, "existing state", string(state)) } } + +// A provider that excludes the operation ends preparation before any source is fetched; a failing +// read-config hook stops it the same way, and the unit's other hooks stay out of preparation. +func TestPrepareComponentExecutionHonorsExclusionAndReadConfigHooks(t *testing.T) { + root := t.TempDir() + config := &schema.AtmosConfiguration{BasePath: root, BasePathAbsolute: root, TerraformDirAbsolutePath: filepath.Join(root, "components")} + config.Components.Terraform.BasePath = "components" + provider := NewMockComponentConfigurationRuntime(gomock.NewController(t)) + config.ComponentConfigurations = provider + binary, err := os.Executable() + require.NoError(t, err) + newInfo := func() *schema.ConfigAndStacksInfo { + return &schema.ConfigAndStacksInfo{Stack: "dev", ComponentFromArg: "app", Component: "app", FinalComponent: "app", Command: binary, SubCommand: "plan", ComponentSettingsSection: map[string]any{"configuration": true}, ComponentEnvSection: map[string]any{}, ComponentSection: map[string]any{"component": "app", "atmos_component": "app", "atmos_stack": "dev", "source": map[string]any{"uri": "./module"}}} + } + + provider.EXPECT().Prepare(gomock.Any(), gomock.Any()).Return(&schema.PreparedComponentConfiguration{ + Engine: "terraform", Source: "./module", Inputs: map[string]any{}, BackendType: "local", Backend: map[string]any{}, Workspace: "default", + Exclusion: &schema.ComponentExclusion{Actions: []string{"plan"}}, + }, nil) + _, err = testExecutor(t).prepareComponentExecution(t.Context(), provisioner.OutputWriters{}, config, newInfo(), false) + require.ErrorIs(t, err, errUtils.ErrComponentExcluded) + require.NoDirExists(t, filepath.Join(root, ".workdir"), "an excluded operation must not provision a source") + + hooks := &schema.ComponentExecution{AfterHooks: []schema.ComponentHook{{Name: "read", Commands: []string{"terragrunt-read-config"}, Execute: []string{"failing-hook"}}}} + provider.EXPECT().Prepare(gomock.Any(), gomock.Any()).Return(&schema.PreparedComponentConfiguration{ + Engine: "terraform", Source: "./module", Inputs: map[string]any{}, BackendType: "local", Backend: map[string]any{}, Workspace: "default", Execution: hooks, + }, nil) + shell := newShellFixture(t, func(*shellCall) error { return errors.New("hook failed") }) + executor := NewExecutor(Services{ExecuteShellCommand: shell.mock.Execute}) + _, err = executor.prepareComponentExecution(t.Context(), provisioner.OutputWriters{}, config, newInfo(), false) + require.ErrorIs(t, err, errUtils.ErrComponentHookFailed) + require.Equal(t, []string{"failing-hook"}, shell.names()) +} diff --git a/pkg/component/terraform/execute.go b/pkg/component/terraform/execute.go index e18093d9b2c..e503a9ab589 100644 --- a/pkg/component/terraform/execute.go +++ b/pkg/component/terraform/execute.go @@ -2,6 +2,7 @@ package terraform import ( "context" + "errors" "fmt" "maps" "os" @@ -211,6 +212,9 @@ func (e *Executor) execute(info schema.ConfigAndStacksInfo, opts ...component.Ex // Resolve paths, install toolchain, write varfiles, validate, run hooks, and build env. execCtx, err := e.prepareComponentExecution(component.ContextFromOptions(opts...), executionOutputWriters(opts...), &atmosConfig, &info, shouldProcess) + if endedWithoutFailure(err) { + return nil + } if err != nil { return err } @@ -231,7 +235,7 @@ func (e *Executor) execute(info schema.ConfigAndStacksInfo, opts ...component.Ex // Run the full command pipeline: init, arg build, workspace, execute, cleanup. // Forward caller-provided options (e.g. CI stdout/stderr capture) alongside the environment option. opts = append(opts, component.WithEnvironment(info.SanitizedEnv)) - err = e.executeCommandPipeline(&atmosConfig, &info, execCtx, opts...) + err = e.runPreparedPipeline(&atmosConfig, &info, execCtx, opts...) // A Terraform command can create, change, or remove state, and a failed apply can // still persist partial changes. Drop any preflight snapshot either way so a later // graph node (including one in a keep-going run) reads the current outputs. @@ -246,6 +250,19 @@ func (e *Executor) execute(info schema.ConfigAndStacksInfo, opts ...component.Ex return err } +// endedWithoutFailure reports whether preparation or the run ended the invocation without failing it: +// the component is excluded from the operation, or the errors policy ignored the failure. +func endedWithoutFailure(err error) bool { + return errors.Is(err, errUtils.ErrComponentExcluded) || onlyRunIgnored(err) +} + +// onlyRunIgnored reports whether err is an ignored failure and nothing else. A lifecycle hook that +// failed beside it, such as an init-from-module after hook, is an independent failure that still +// fails the command. +func onlyRunIgnored(err error) bool { + return errors.Is(err, errUtils.ErrComponentRunIgnored) && !errors.Is(err, errUtils.ErrComponentHookFailed) +} + // invalidateTerraformExecutionCaches clears snapshots under both execution identities. func invalidateTerraformExecutionCaches(info *schema.ConfigAndStacksInfo) { invalidateTerraformStateCache(info.Stack, info.ComponentFromArg) diff --git a/pkg/component/terraform/execute_hook_arguments.go b/pkg/component/terraform/execute_hook_arguments.go new file mode 100644 index 00000000000..a2a51bdce21 --- /dev/null +++ b/pkg/component/terraform/execute_hook_arguments.go @@ -0,0 +1,115 @@ +package terraform + +import ( + "encoding/json" + "fmt" + "maps" + "os" + "path/filepath" + "slices" + "strings" + + errUtils "github.com/cloudposse/atmos/errors" +) + +// This marker is emitted by the Terragrunt provider for optional TFLint variable +// files. Only the host knows the provisioned module directory at hook execution. +const tflintOptionalVarFilePrefix = "--atmos-optional-var-file=" + +// This marker stands for one --var per input of the final prepared configuration. The provider cannot +// know those inputs: the host merges the stack's variables over them after preparation. +const tflintInputVariablesMarker = "--atmos-tflint-input-variables" + +const ( + tflintConfigSearchPrefix = "--atmos-tflint-config-search=" + tflintConfigName = ".tflint.hcl" + tflintMaxParents = 100 +) + +func resolveTflintHookArguments(arguments []string, moduleDir string, inputs map[string]any) ([]string, error) { + arguments, err := resolveTflintInputVariables(arguments, inputs) + if err != nil { + return nil, err + } + result := make([]string, 0, len(arguments)) + for _, argument := range resolveOptionalTflintVarFiles(arguments, moduleDir) { + unitDir, search := strings.CutPrefix(argument, tflintConfigSearchPrefix) + if !search { + result = append(result, argument) + continue + } + config, err := findTflintConfig(unitDir) + if err != nil { + return nil, fmt.Errorf("%w: no .tflint.hcl found beside the unit or in a parent directory; pass --config to select one: %w", errUtils.ErrComponentHookFailed, err) + } + result = append(result, config) + } + return result, nil +} + +// resolveTflintInputVariables replaces the input marker with a --var for every input, sorted so that +// repeated runs build the same command. +func resolveTflintInputVariables(arguments []string, inputs map[string]any) ([]string, error) { + result := make([]string, 0, len(arguments)+len(inputs)) + for _, argument := range arguments { + if argument != tflintInputVariablesMarker { + result = append(result, argument) + continue + } + for _, name := range slices.Sorted(maps.Keys(inputs)) { + value, err := tflintVariableValue(inputs[name]) + if err != nil { + return nil, fmt.Errorf("%w: input %q cannot be passed to tflint: %w", errUtils.ErrComponentHookFailed, name, err) + } + result = append(result, "--var="+name+"="+value) + } + } + return result, nil +} + +// tflintVariableValue renders a string as itself and any other value as JSON, escaping HCL templates. +func tflintVariableValue(value any) (string, error) { + if text, ok := value.(string); ok { + return text, nil + } + data, err := json.Marshal(value) + if err != nil { + return "", err + } + return strings.NewReplacer("${", "$${", "%{", "%%{").Replace(string(data)), nil +} + +func findTflintConfig(unitDir string) (string, error) { + directory := unitDir + for range tflintMaxParents { + candidate := filepath.Join(directory, tflintConfigName) + if info, err := os.Stat(candidate); err == nil && !info.IsDir() { + return candidate, nil + } + parent := filepath.Dir(directory) + if parent == directory { + break + } + directory = parent + } + return "", os.ErrNotExist +} + +func resolveOptionalTflintVarFiles(arguments []string, moduleDir string) []string { + result := make([]string, 0, len(arguments)) + for _, argument := range arguments { + file, optional := strings.CutPrefix(argument, tflintOptionalVarFilePrefix) + if !optional { + result = append(result, argument) + continue + } + candidate := file + if !filepath.IsAbs(candidate) { + candidate = filepath.Join(moduleDir, candidate) + } + if info, err := os.Stat(candidate); err == nil && !info.IsDir() { + result = append(result, "--var-file="+file) + } + } + return result +} diff --git a/pkg/component/terraform/execute_hook_arguments_test.go b/pkg/component/terraform/execute_hook_arguments_test.go new file mode 100644 index 00000000000..ffbd0a7c36f --- /dev/null +++ b/pkg/component/terraform/execute_hook_arguments_test.go @@ -0,0 +1,80 @@ +package terraform + +import ( + "os" + "path/filepath" + "testing" + + "github.com/hashicorp/hcl/v2" + "github.com/hashicorp/hcl/v2/hclsyntax" + "github.com/stretchr/testify/require" + + errUtils "github.com/cloudposse/atmos/errors" +) + +func TestOptionalTflintVarFilesResolveAtHookExecution(t *testing.T) { + moduleDir := t.TempDir() + require.NoError(t, os.WriteFile(filepath.Join(moduleDir, "module.tfvars"), nil, 0o644)) + unitDir := t.TempDir() + require.NoError(t, os.WriteFile(filepath.Join(unitDir, "unit.tfvars"), nil, 0o644)) + absolute := filepath.Join(unitDir, "unit.tfvars") + arguments := []string{"--config", ".tflint.hcl", tflintOptionalVarFilePrefix + "module.tfvars", tflintOptionalVarFilePrefix + "missing.tfvars", tflintOptionalVarFilePrefix + absolute} + require.Equal(t, []string{"--config", ".tflint.hcl", "--var-file=module.tfvars", "--var-file=" + absolute}, + resolveOptionalTflintVarFiles(arguments, moduleDir)) +} + +func TestTflintConfigLookupWaitsForHookExecution(t *testing.T) { + unitDir := t.TempDir() + arguments := []string{"--config", tflintConfigSearchPrefix + unitDir, "--chdir", "."} + // Preparation can finish before a previous hook generates the config. + config := filepath.Join(unitDir, ".tflint.hcl") + require.NoError(t, os.WriteFile(config, []byte("plugin \"aws\" {}"), 0o644)) + resolved, err := resolveTflintHookArguments(arguments, t.TempDir(), nil) + require.NoError(t, err) + require.Equal(t, []string{"--config", config, "--chdir", "."}, resolved) +} + +func TestTflintInputVariablesBindToTheFinalInputsAtHookExecution(t *testing.T) { + // The provider's own value of count is overridden by the stack's, so the hook must see the stack's. + inputs := map[string]any{ + "count": 2, + "name": "web", + "tags": map[string]any{"env": "${prod}"}, + "zone": "a", + "nested": []any{"%{x}"}, + } + arguments := []string{"--config", ".tflint.hcl", tflintInputVariablesMarker, "--var-file=extra.tfvars", "--minimum-failure-severity=error"} + + resolved, err := resolveTflintHookArguments(arguments, t.TempDir(), inputs) + + require.NoError(t, err) + require.Equal(t, []string{ + "--config", ".tflint.hcl", + "--var=count=2", "--var=name=web", "--var=nested=[\"%%{x}\"]", "--var=tags={\"env\":\"$${prod}\"}", "--var=zone=a", + "--var-file=extra.tfvars", "--minimum-failure-severity=error", + }, resolved) + none, err := resolveTflintInputVariables([]string{tflintInputVariablesMarker}, nil) + require.NoError(t, err) + require.Empty(t, none, "no inputs bind no variables") + _, err = resolveTflintInputVariables([]string{tflintInputVariablesMarker}, map[string]any{"bad": make(chan int)}) + require.ErrorIs(t, err, errUtils.ErrComponentHookFailed) + require.ErrorContains(t, err, `"bad"`) +} + +func TestTflintVariableValueRendering(t *testing.T) { + text, err := tflintVariableValue("plain") + require.NoError(t, err) + require.Equal(t, "plain", text) + text, err = tflintVariableValue(map[string]any{"a": []any{1, "${x}"}}) + require.NoError(t, err) + require.Equal(t, `{"a":[1,"$${x}"]}`, text) + text, err = tflintVariableValue(map[string]any{"literal": "%{ if true } ${x}"}) + require.NoError(t, err) + expr, diags := hclsyntax.ParseExpression([]byte(text), "input", hcl.Pos{Line: 1, Column: 1}) + require.False(t, diags.HasErrors(), diags.Error()) + value, diags := expr.Value(nil) + require.False(t, diags.HasErrors(), diags.Error()) + require.Equal(t, "%{ if true } ${x}", value.GetAttr("literal").AsString()) + _, err = tflintVariableValue(make(chan int)) + require.Error(t, err) +} diff --git a/pkg/component/terraform/execute_hooks.go b/pkg/component/terraform/execute_hooks.go index 389dbb51b4b..1618eb1b92d 100644 --- a/pkg/component/terraform/execute_hooks.go +++ b/pkg/component/terraform/execute_hooks.go @@ -5,13 +5,11 @@ import ( "errors" "fmt" "io" - "io/fs" "os" "path/filepath" "slices" "strconv" "strings" - "sync" errUtils "github.com/cloudposse/atmos/errors" "github.com/cloudposse/atmos/pkg/component" @@ -47,11 +45,24 @@ type executionPhase struct { planFileArgument bool // componentPath is the provisioned module directory. componentPath string + // main marks the command the user asked for, as opposed to a step that prepares for it. + main bool } // mainCommandArgsKey carries the effective main command arguments to plan binding. type mainCommandArgsKey struct{} +// mainCommandIgnoredKey carries a flag that the errors policy sets when it ignores a failure of the +// main command, so the caller can tell that run from one that succeeded. +type mainCommandIgnoredKey struct{} + +// markMainCommandIgnored records that the errors policy ignored the main command's failure. +func markMainCommandIgnored(ctx context.Context) { + if ignored, ok := ctx.Value(mainCommandIgnoredKey{}).(*bool); ok { + *ignored = true + } +} + // mainCommandRecoveryKey carries smart init's recovery into the main command's hook lifecycle. type mainCommandRecoveryKey struct{} @@ -82,6 +93,7 @@ func (e *Executor) runMainShellCommand(atmosConfig *schema.AtmosConfiguration, i insertAt: position, planFileArgument: info.SubCommand == subcommandApply && info.UseTerraformPlan, componentPath: componentPath, + main: true, } return e.runExecutionPhase(atmosConfig, info, phase, opts, func(phaseInfo *schema.ConfigAndStacksInfo, phaseArgs []string, phaseOpts []component.ExecutionOption) error { ctx := component.ContextFromOptions(phaseOpts...) @@ -194,20 +206,28 @@ func (e *Executor) runExecutionPhase(atmosConfig *schema.AtmosConfiguration, inf runner := &hookRunner{executor: e, atmosConfig: atmosConfig, info: &phaseInfo, phase: phase, opts: hookOptions(opts), capture: capture} // With -detailed-exitcode, exit status 2 reports a plan with changes. Hooks must not see that // as a failure, but the status still has to reach the caller once the hooks are done. - var changesPresent error env := slices.Concat(envpkg.ConvertMapToSlice(atmosConfig.Env), phaseInfo.ComponentEnvList) - err = execution.Run(component.ContextFromOptions(opts...), config, phase.command, runner, func() (execution.Output, error) { - output, actionErr := runPhaseAction(&phaseInfo, applied.Args, opts, capture, action) - if isDetailedExitCodeChanges(phase.command, applied.Args, env, actionErr, opts...) { - changesPresent = actionErr - return output, nil - } - return output, actionErr - }) + attempt := func(attemptOpts []component.ExecutionOption, hookStderr io.Writer) (changesPresent, runErr error) { + runner.recoveryStderr = hookStderr + runErr = execution.Run(component.ContextFromOptions(attemptOpts...), config, phase.command, runner, func() (execution.Output, error) { + output, actionErr := runPhaseAction(&phaseInfo, applied.Args, attemptOpts, capture, action) + if isDetailedExitCodeChanges(phase.command, applied.Args, env, actionErr, attemptOpts...) { + // Mark the completed plan here, so the recovery path reports it the same way. + changesPresent = &successfulPlanChanges{error: actionErr} + return output, nil + } + return output, actionErr + }) + return changesPresent, runErr + } + if config.Errors != nil { + return e.recoverPhase(&phaseInfo, phase, config.Errors, opts, attempt) + } + changesPresent, err := attempt(opts, nil) if err != nil { return err } - return successfulPlanResult(phase.command, applied.Args, env, changesPresent, opts...) + return changesPresent } // successfulPlanChanges distinguishes a completed plan (including its hooks) from a @@ -361,6 +381,8 @@ type hookRunner struct { opts []component.ExecutionOption // capture records hook output for error hook matching; nil means no error hooks exist. capture *phaseCapture + // recoveryStderr is per attempt and separate from the caller's command capture. + recoveryStderr io.Writer } // RunHook runs one hook and returns its captured output for error hook matching. @@ -388,11 +410,15 @@ func (r *hookRunner) RunHook(_ context.Context, kind execution.HookKind, hook *s file = r.capture.create() captured = file } + stderrCapture := captured + if r.recoveryStderr != nil { + stderrCapture = io.MultiWriter(captured, &syncWriter{w: r.recoveryStderr}) + } opts := append( slices.Clone(r.opts), component.WithStdoutOverride(stdout), component.WithStdoutCapture(captured), - component.WithStderrCapture(captured), + component.WithStderrCapture(stderrCapture), ) env := append( slices.Clone(r.info.ComponentEnvList), @@ -404,13 +430,31 @@ func (r *hookRunner) RunHook(_ context.Context, kind execution.HookKind, hook *s if err != nil { return nil, err } - err = r.executor.services.ExecuteShellCommand(*r.atmosConfig, hook.Execute[0], hook.Execute[1:], directory, env, false, "", opts...) + arguments, err := r.hookArguments(hook, directory) + if err != nil { + return nil, err + } + err = r.executor.services.ExecuteShellCommand(*r.atmosConfig, hook.Execute[0], arguments, directory, env, false, "", opts...) if file == nil { return nil, err } return file.finish(), err } +// hookArguments returns the arguments a hook runs with. A tflint hook gets the stack's final inputs +// bound to its variables. +func (r *hookRunner) hookArguments(hook *schema.ComponentHook, directory string) ([]string, error) { + arguments := hook.Execute[1:] + if hook.Execute[0] != "tflint" { + return arguments, nil + } + arguments, err := resolveTflintHookArguments(arguments, directory, r.info.ComponentVarsSection) + if err != nil { + return nil, fmt.Errorf("tflint hook %q: %w", hook.Name, err) + } + return arguments, nil +} + // hookUnitLabel names the component and stack a hook runs for, so output of a bulk run can be attributed. func hookUnitLabel(info *schema.ConfigAndStacksInfo) string { if info.ComponentFromArg == "" { @@ -505,101 +549,3 @@ func extraArgumentsPosition(info *schema.ConfigAndStacksInfo, args []string) int } return position } - -// phaseCapture owns the temporary files that record phase and hook output for error hook -// matching. Cleanup removes every file once the lifecycle, including matching, has finished. -type phaseCapture struct { - mu sync.Mutex - files []*captureFile -} - -// create starts a new capture file. If the file cannot be created, the command still runs and -// the capture reports the failure when error hooks read it. -func (c *phaseCapture) create() *captureFile { - file, err := os.CreateTemp("", "atmos-phase-output-*") - capture := &captureFile{file: file} - if err != nil { - capture.err = captureError(err) - return capture - } - capture.path = file.Name() - c.mu.Lock() - c.files = append(c.files, capture) - c.mu.Unlock() - return capture -} - -// cleanup closes and removes every capture file. -func (c *phaseCapture) cleanup() { - c.mu.Lock() - defer c.mu.Unlock() - for _, file := range c.files { - file.close() - if err := os.Remove(file.path); err != nil && !errors.Is(err, fs.ErrNotExist) { - log.Debug("Failed to remove captured command output", "path", file.path, "error", err) - } - } - c.files = nil -} - -// captureFile records output in a temporary file. A write failure is kept and reported when -// the output is read. The command keeps running, so a full disk cannot fail it. -type captureFile struct { - mu sync.Mutex - file *os.File - path string - err error -} - -// Write appends to the capture and always reports success to the writer chain. -// -//nolint:lintroller // Called for every output chunk; tracking would dominate the copy. -func (c *captureFile) Write(p []byte) (int, error) { - c.mu.Lock() - defer c.mu.Unlock() - if c.err == nil && c.file != nil { - if _, err := c.file.Write(p); err != nil { - c.err = captureError(err) - } - } - return len(p), nil -} - -// finish closes the file for writing and returns it as lifecycle output. -func (c *captureFile) finish() execution.Output { - c.close() - return c -} - -func (c *captureFile) close() { - c.mu.Lock() - defer c.mu.Unlock() - if c.file == nil { - return - } - if err := c.file.Close(); err != nil && c.err == nil { - c.err = captureError(err) - } - c.file = nil -} - -// Open returns a reader over the captured output, or the capture failure. -func (c *captureFile) Open() (io.Reader, error) { - defer perf.Track(nil, "terraform.captureFile.Open")() - - c.mu.Lock() - defer c.mu.Unlock() - if c.err != nil { - return nil, c.err - } - file, err := os.Open(c.path) - if err != nil { - return nil, captureError(err) - } - return file, nil -} - -// captureError marks a failure to record or read captured command output. -func captureError(err error) error { - return fmt.Errorf("%w: %w", errUtils.ErrComponentOutputCapture, err) -} diff --git a/pkg/component/terraform/execute_hooks_capture.go b/pkg/component/terraform/execute_hooks_capture.go new file mode 100644 index 00000000000..db131b9a74c --- /dev/null +++ b/pkg/component/terraform/execute_hooks_capture.go @@ -0,0 +1,113 @@ +package terraform + +import ( + "errors" + "fmt" + "io" + "io/fs" + "os" + "sync" + + errUtils "github.com/cloudposse/atmos/errors" + "github.com/cloudposse/atmos/pkg/component/execution" + log "github.com/cloudposse/atmos/pkg/logger" + "github.com/cloudposse/atmos/pkg/perf" +) + +// phaseCapture owns the temporary files that record phase and hook output for error hook +// matching. Cleanup removes every file once the lifecycle, including matching, has finished. +type phaseCapture struct { + mu sync.Mutex + files []*captureFile +} + +// create starts a new capture file. If the file cannot be created, the command still runs and +// the capture reports the failure when error hooks read it. +func (c *phaseCapture) create() *captureFile { + file, err := os.CreateTemp("", "atmos-phase-output-*") + capture := &captureFile{file: file} + if err != nil { + capture.err = captureError(err) + return capture + } + capture.path = file.Name() + c.mu.Lock() + c.files = append(c.files, capture) + c.mu.Unlock() + return capture +} + +// cleanup closes and removes every capture file. +func (c *phaseCapture) cleanup() { + c.mu.Lock() + defer c.mu.Unlock() + for _, file := range c.files { + file.close() + if err := os.Remove(file.path); err != nil && !errors.Is(err, fs.ErrNotExist) { + log.Debug("Failed to remove captured command output", "path", file.path, "error", err) + } + } + c.files = nil +} + +// captureFile records output in a temporary file. A write failure is kept and reported when +// the output is read. The command keeps running, so a full disk cannot fail it. +type captureFile struct { + mu sync.Mutex + file *os.File + path string + err error +} + +// Write appends to the capture and always reports success to the writer chain. +// +//nolint:lintroller // Called for every output chunk; tracking would dominate the copy. +func (c *captureFile) Write(p []byte) (int, error) { + c.mu.Lock() + defer c.mu.Unlock() + if c.err == nil && c.file != nil { + if _, err := c.file.Write(p); err != nil { + c.err = captureError(err) + } + } + return len(p), nil +} + +// finish closes the file for writing and returns it as lifecycle output. +func (c *captureFile) finish() execution.Output { + c.close() + return c +} + +func (c *captureFile) close() { + c.mu.Lock() + defer c.mu.Unlock() + if c.file == nil { + return + } + if err := c.file.Close(); err != nil && c.err == nil { + c.err = captureError(err) + } + c.file = nil +} + +// Open returns a reader over the captured output, or the capture failure. +func (c *captureFile) Open() (io.Reader, error) { + defer perf.Track(nil, "terraform.captureFile.Open")() + + c.mu.Lock() + defer c.mu.Unlock() + if c.err != nil { + return nil, c.err + } + file, err := os.Open(c.path) + if err != nil { + return nil, captureError(err) + } + return file, nil +} + +// captureError marks a failure to record or read captured command output. +func captureError(err error) error { + return fmt.Errorf("%w: %w", errUtils.ErrComponentOutputCapture, err) +} diff --git a/pkg/component/terraform/execute_hooks_process_test.go b/pkg/component/terraform/execute_hooks_process_test.go index 4b86f85ad75..61213fbfd28 100644 --- a/pkg/component/terraform/execute_hooks_process_test.go +++ b/pkg/component/terraform/execute_hooks_process_test.go @@ -233,6 +233,17 @@ func TestPipelineRunsHooksAroundEachPhaseSeparately(t *testing.T) { require.NotContains(t, pipeline.stdout.String(), "hook says hello", "hook stdout must not mix into data stdout") } +func TestIgnoredInitStopsPreparedPipelineSuccessfully(t *testing.T) { + pipeline := newProcessPipeline(t, &schema.ComponentExecution{ + ExtraArguments: []schema.ComponentExtraArguments{{Commands: []string{"init"}, Arguments: []string{"stderr=already exists", "exit=1"}}}, + Errors: &schema.ComponentErrors{Ignore: []schema.ComponentIgnorePolicy{{Name: "existing", Patterns: []string{"already exists"}}}}, + }) + streams := process.Streams{Stdout: pipeline.stdout, Stderr: pipeline.stderr} + err := testExecutor(t).runPreparedPipeline(&schema.AtmosConfiguration{}, pipeline.info, pipeline.execCtx, component.WithProcessStreams(streams)) + require.NoError(t, err, "an ignored preparatory failure ends the unit successfully") + require.Equal(t, []string{"init"}, timeline(readHookLog(t, pipeline.log)), "plan and workspace setup must not run after ignored init") +} + func physicalPath(t *testing.T, path string) string { t.Helper() resolved, err := filepath.EvalSymlinks(path) @@ -504,3 +515,53 @@ func TestFinishMainTerraformCommandKeepsHookFailuresUnderPermissiveCIMapping(t * err = executor.finishMainTerraformCommand(config, pipeline.info, hookErr, false) require.ErrorIs(t, err, errUtils.ErrComponentHookFailed) } + +// TestLifecycleCapturesBeforeHookOutputForErrorHooks verifies that a lifecycle stage with error hooks records +// the output of its before hooks, so an error hook can match a before hook's failure, and removes the +// capture files once the lifecycle has finished. +func TestLifecycleCapturesBeforeHookOutputForErrorHooks(t *testing.T) { + for _, test := range []struct { + name string + pattern string + wantHook bool + }{ + {"matching before hook output", "BeforeNeedle", true}, + {"non matching before hook output", "NotInAnyOutput", false}, + } { + t.Run(test.name, func(t *testing.T) { + temp := t.TempDir() + for _, name := range []string{"TMPDIR", "TMP", "TEMP"} { + t.Setenv(name, temp) + } + shell := newShellFixture(t, func(call *shellCall) error { + if call.command == "hook-before" { + _, _ = call.options.StdoutCapture.Write([]byte("BeforeNeedle")) + return errUtils.ExitCodeError{Code: 2} + } + return nil + }) + config := &schema.ComponentExecution{ + BeforeHooks: []schema.ComponentHook{execHook("before", "init")}, + ErrorHooks: []schema.ComponentHook{{Name: "recover", Commands: []string{"init"}, Execute: []string{"hook-recover"}, OnErrors: []string{test.pattern}}}, + } + actionRan := false + + err := hookExecutor(shell).runLifecycle(t.Context(), &schema.AtmosConfiguration{}, hookInfo("init", config), + lifecycleStage{command: "init", config: config}, func(context.Context) error { + actionRan = true + return nil + }) + + require.Error(t, err) + require.False(t, actionRan, "a failing before hook keeps the action from starting") + if test.wantHook { + require.Equal(t, []string{"hook-before arg", "hook-recover"}, shell.names()) + } else { + require.Equal(t, []string{"hook-before arg"}, shell.names()) + } + entries, readErr := os.ReadDir(temp) + require.NoError(t, readErr) + require.Empty(t, entries, "every capture file is removed") + }) + } +} diff --git a/pkg/component/terraform/execute_hooks_test.go b/pkg/component/terraform/execute_hooks_test.go index 8496596ecc7..dbfa8e3485f 100644 --- a/pkg/component/terraform/execute_hooks_test.go +++ b/pkg/component/terraform/execute_hooks_test.go @@ -1,9 +1,11 @@ package terraform import ( + "bytes" "context" "encoding/json" "errors" + "fmt" "io" "os" "path/filepath" @@ -15,11 +17,16 @@ import ( "go.uber.org/mock/gomock" errUtils "github.com/cloudposse/atmos/errors" + "github.com/cloudposse/atmos/pkg/ansi" "github.com/cloudposse/atmos/pkg/component" + iolib "github.com/cloudposse/atmos/pkg/io" "github.com/cloudposse/atmos/pkg/policy" "github.com/cloudposse/atmos/pkg/policy/planfile" "github.com/cloudposse/atmos/pkg/process" + "github.com/cloudposse/atmos/pkg/provisioner" + "github.com/cloudposse/atmos/pkg/provisioner/source" "github.com/cloudposse/atmos/pkg/schema" + "github.com/cloudposse/atmos/pkg/ui" ) // shellCall is one invocation of the host's process facility. @@ -617,6 +624,251 @@ func TestHooksMatchTheTerraformCommandNotItsSubcommand(t *testing.T) { require.Equal(t, []string{"hook-state-hook arg", "terraform state"}, shell.names()) } +// shellScript scripts a sequence of results for one executable; other executables succeed. +func scriptedFailures(stderr []string, failures ...error) func(call *shellCall) error { + attempt := 0 + return func(call *shellCall) error { + if call.command != "terraform" { + return nil + } + index := attempt + attempt++ + if index < len(stderr) && call.options.StderrCapture != nil { + _, _ = io.WriteString(call.options.StderrCapture, stderr[index]) + } + if index < len(failures) { + return failures[index] + } + return nil + } +} + +func errorsInfo(subCommand string, policy *schema.ComponentErrors, hooks ...schema.ComponentHook) *schema.ConfigAndStacksInfo { + return hookInfo(subCommand, &schema.ComponentExecution{Errors: policy, BeforeHooks: hooks}) +} + +func TestErrorsPolicyRetriesTheWholeHookedCommand(t *testing.T) { + failure := errUtils.ExitCodeError{Code: 1} + shell := newShellFixture(t, scriptedFailures([]string{"Error: connection timeout", "Error: connection timeout", ""}, failure, failure)) + policy := &schema.ComponentErrors{Retry: []schema.ComponentRetryPolicy{{Name: "net", Patterns: []string{"timeout"}, MaxAttempts: 3}}} + info := errorsInfo("apply", policy, execHook("lint", "apply")) + var callerCapture strings.Builder + err := hookExecutor(shell).runMainTerraformCommand(&schema.AtmosConfiguration{}, info, []string{"apply"}, t.TempDir(), component.WithStderrCapture(&callerCapture)) + require.NoError(t, err) + require.Equal(t, []string{"hook-lint arg", "terraform apply", "hook-lint arg", "terraform apply", "hook-lint arg", "terraform apply"}, shell.names(), + "every attempt runs the before hook again, as Terragrunt retries the whole unit run") + require.Contains(t, callerCapture.String(), "connection timeout", "the caller's own capture keeps receiving the output") +} + +func TestErrorsPolicyStopsAtMaxAttemptsAndKeepsTheExitCode(t *testing.T) { + failure := errUtils.ExitCodeError{Code: 3} + shell := newShellFixture(t, scriptedFailures([]string{"timeout", "timeout"}, failure, failure, failure)) + policy := &schema.ComponentErrors{Retry: []schema.ComponentRetryPolicy{{Name: "net", Patterns: []string{"timeout"}, MaxAttempts: 2}}} + err := hookExecutor(shell).runMainTerraformCommand(&schema.AtmosConfiguration{}, errorsInfo("plan", policy), []string{"plan"}, t.TempDir()) + require.ErrorIs(t, err, errUtils.ErrComponentMaxAttempts) + require.Equal(t, 3, resolveExitCode(err)) + require.Len(t, shell.calls, 2) +} + +func TestErrorsPolicyMatchesStandardErrorOnly(t *testing.T) { + shell := newShellFixture(t, func(call *shellCall) error { + if call.command != "terraform" { + return nil + } + // A plan can print the word "timeout" on stdout without having failed because of it. + _, _ = io.WriteString(call.options.StdoutCapture, "resource with timeout = 5") + _, _ = io.WriteString(call.options.StderrCapture, "Error: invalid argument") + return errUtils.ExitCodeError{Code: 1} + }) + policy := &schema.ComponentErrors{Retry: []schema.ComponentRetryPolicy{{Name: "net", Patterns: []string{"timeout"}, MaxAttempts: 3}}} + info := errorsInfo("plan", policy) + err := hookExecutor(shell).runMainTerraformCommand(&schema.AtmosConfiguration{}, info, []string{"plan"}, t.TempDir(), + component.WithStdoutCapture(io.Discard)) + require.Equal(t, 1, resolveExitCode(err)) + require.Len(t, shell.calls, 1) +} + +func TestErrorsPolicyMatchesHookStderrWithoutCapturingItAsCommandOutput(t *testing.T) { + attempts := 0 + shell := newShellFixture(t, func(call *shellCall) error { + if call.command != "hook-lint" { + return nil + } + attempts++ + if attempts == 1 { + _, _ = io.WriteString(call.options.StderrCapture, "connection timeout") + return errors.New("hook failed") + } + return nil + }) + policy := &schema.ComponentErrors{Retry: []schema.ComponentRetryPolicy{{Name: "net", Patterns: []string{"timeout"}, MaxAttempts: 2}}} + info := errorsInfo("plan", policy, execHook("lint", "plan")) + var callerCapture strings.Builder + err := hookExecutor(shell).runMainTerraformCommand(&schema.AtmosConfiguration{}, info, []string{"plan"}, t.TempDir(), component.WithStderrCapture(&callerCapture)) + require.NoError(t, err) + require.Equal(t, 2, attempts) + require.NotContains(t, callerCapture.String(), "connection timeout", "hook diagnostics stay out of command capture") +} + +func TestErrorsPolicyDoesNotMatchHookStdout(t *testing.T) { + shell := newShellFixture(t, func(call *shellCall) error { + if call.command != "hook-lint" { + return nil + } + _, _ = io.WriteString(call.options.StdoutCapture, "connection timeout") + return errors.New("hook failed") + }) + policy := &schema.ComponentErrors{Retry: []schema.ComponentRetryPolicy{{Name: "net", Patterns: []string{"timeout"}, MaxAttempts: 2}}} + info := errorsInfo("plan", policy, execHook("lint", "plan")) + err := hookExecutor(shell).runMainTerraformCommand(&schema.AtmosConfiguration{}, info, []string{"plan"}, t.TempDir()) + require.Error(t, err) + require.Len(t, shell.calls, 1, "hook stdout alone must not match the retry policy") +} + +func TestErrorsPolicyIgnoresAFailureAndWritesSignals(t *testing.T) { + dir := t.TempDir() + shell := newShellFixture(t, scriptedFailures([]string{"Error: already exists"}, errUtils.ExitCodeError{Code: 1})) + policy := &schema.ComponentErrors{Ignore: []schema.ComponentIgnorePolicy{{Name: "exists", Patterns: []string{"already exists"}, Message: "reuse it", Signals: map[string]any{"alert": true}}}} + require.NoError(t, hookExecutor(shell).runMainTerraformCommand(&schema.AtmosConfiguration{}, errorsInfo("apply", policy), []string{"apply"}, dir)) + data, err := os.ReadFile(filepath.Join(dir, "error-signals.json")) + require.NoError(t, err) + require.JSONEq(t, `{"alert": true}`, string(data)) +} + +func TestErrorsPolicyIgnoreInAPreparatoryStepEndsTheRun(t *testing.T) { + shell := newShellFixture(t, scriptedFailures([]string{"Error: already exists"}, errUtils.ExitCodeError{Code: 1})) + policy := &schema.ComponentErrors{Ignore: []schema.ComponentIgnorePolicy{{Name: "exists", Patterns: []string{"already exists"}}}} + e := hookExecutor(shell) + info := errorsInfo("plan", policy) + phase := &executionPhase{command: "init", args: []string{"init"}, insertAt: 1, componentPath: t.TempDir()} + err := e.runExecutionPhase(&schema.AtmosConfiguration{}, info, phase, nil, func(phaseInfo *schema.ConfigAndStacksInfo, args []string, opts []component.ExecutionOption) error { + return e.services.ExecuteShellCommand(schema.AtmosConfiguration{}, phaseInfo.Command, args, phase.componentPath, phaseInfo.ComponentEnvList, false, "", opts...) + }) + require.ErrorIs(t, err, errUtils.ErrComponentRunIgnored, "the command that follows must not run after an ignored init failure") +} + +func TestErrorsPolicyDoesNotChangeDetailedExitCodeResults(t *testing.T) { + shell := newShellFixture(t, scriptedFailures(nil, errUtils.ExitCodeError{Code: 2})) + policy := &schema.ComponentErrors{Ignore: []schema.ComponentIgnorePolicy{{Name: "all", Patterns: []string{".*"}}}} + err := hookExecutor(shell).runMainTerraformCommand(&schema.AtmosConfiguration{}, errorsInfo("plan", policy), []string{"plan", "-detailed-exitcode"}, t.TempDir()) + require.Equal(t, 2, resolveExitCode(err), "exit status 2 is a plan result and not a failure to recover from") +} + +func TestExcludedFromOperation(t *testing.T) { + prepared := func(actions ...string) *schema.PreparedComponentConfiguration { + return &schema.PreparedComponentConfiguration{Exclusion: &schema.ComponentExclusion{Actions: actions}} + } + for _, test := range []struct { + name string + info schema.ConfigAndStacksInfo + prepared *schema.PreparedComponentConfiguration + want bool + }{ + {"no provider data", schema.ConfigAndStacksInfo{SubCommand: "plan"}, nil, false}, + {"no exclusion", schema.ConfigAndStacksInfo{SubCommand: "plan"}, &schema.PreparedComponentConfiguration{}, false}, + {"listed", schema.ConfigAndStacksInfo{SubCommand: "plan"}, prepared("plan"), true}, + {"other command", schema.ConfigAndStacksInfo{SubCommand: "apply"}, prepared("plan"), false}, + {"deploy is apply", schema.ConfigAndStacksInfo{SubCommand: "deploy"}, prepared("apply"), true}, + {"apply destroy is destroy", schema.ConfigAndStacksInfo{SubCommand: "apply", AdditionalArgsAndFlags: []string{"-destroy"}}, prepared("destroy"), true}, + } { + t.Run(test.name, func(t *testing.T) { + require.Equal(t, test.want, excludedFromOperation(&test.info, test.prepared)) + }) + } +} + +type warningTestStreams struct{ stderr *bytes.Buffer } + +func (s warningTestStreams) Input() io.Reader { return strings.NewReader("") } +func (s warningTestStreams) Output() io.Writer { return io.Discard } +func (s warningTestStreams) Error() io.Writer { return s.stderr } +func (s warningTestStreams) RawOutput() io.Writer { return io.Discard } +func (s warningTestStreams) RawError() io.Writer { return s.stderr } + +// A provider that reports an ignored failure, such as a registry resolution failure with no resolved +// source, ends the unit's run successfully with the message of an ignored source fetch. Nothing runs: +// no source is provisioned, no read-config hook starts, and no terraform command is executed. +func TestPrepareComponentExecutionEndsAnIgnoredRunWithoutExecuting(t *testing.T) { + var output bytes.Buffer + ioCtx, err := iolib.NewContext(iolib.WithStreams(warningTestStreams{stderr: &output})) + require.NoError(t, err) + ui.InitFormatter(ioCtx) + t.Cleanup(ui.ReinitFormatter) + + root := t.TempDir() + config := &schema.AtmosConfiguration{BasePath: root, BasePathAbsolute: root, TerraformDirAbsolutePath: filepath.Join(root, "components")} + config.Components.Terraform.BasePath = "components" + provider := NewMockComponentConfigurationRuntime(gomock.NewController(t)) + config.ComponentConfigurations = provider + binary, err := os.Executable() + require.NoError(t, err) + info := &schema.ConfigAndStacksInfo{Stack: "dev", ComponentFromArg: "app", Component: "app", FinalComponent: "app", Command: binary, SubCommand: "plan", ComponentSettingsSection: map[string]any{"configuration": true}, ComponentEnvSection: map[string]any{}, ComponentSection: map[string]any{"component": "app", "atmos_component": "app", "atmos_stack": "dev", "source": map[string]any{"uri": "tfr://registry.example/acme/vpc/aws"}}} + hooks := &schema.ComponentExecution{AfterHooks: []schema.ComponentHook{{Name: "read", Commands: []string{"terragrunt-read-config"}, Execute: []string{"must-not-run"}}}} + provider.EXPECT().Prepare(gomock.Any(), gomock.Any()).Return(&schema.PreparedComponentConfiguration{ + Engine: "terraform", Source: "tfr://registry.example/acme/vpc/aws", Inputs: map[string]any{}, BackendType: "local", Backend: map[string]any{}, Workspace: "default", Execution: hooks, + Files: []schema.ComponentConfigurationFile{{Path: "generated.tf", Content: []byte("# generated"), IfExists: "overwrite"}}, + IgnoredRun: &schema.ComponentIgnoredRun{Policy: "registry_down", Message: "registry is down"}, + }, nil) + shell := newShellFixture(t, nil) + executor := NewExecutor(Services{ExecuteShellCommand: shell.mock.Execute}) + + _, err = executor.prepareComponentExecution(t.Context(), provisioner.OutputWriters{}, config, info, false) + + require.ErrorIs(t, err, errUtils.ErrComponentRunIgnored) + require.True(t, endedWithoutFailure(err), "an ignored run ends the invocation successfully") + require.Empty(t, shell.names(), "nothing may execute") + require.NoDirExists(t, filepath.Join(root, ".workdir"), "an ignored run must not provision a source") + require.NoFileExists(t, filepath.Join(root, "components", "terraform", "app", "generated.tf"), "an ignored run writes no generated files") + // CI runners advertise color, so strip the ANSI styling before matching the text. + require.Contains(t, ansi.Strip(output.String()), "Ignoring an error in app in dev because it matches policy registry_down: registry is down") +} + +func TestEndIgnoredRunReportsLikeAnIgnoredFetch(t *testing.T) { + var output bytes.Buffer + ioCtx, err := iolib.NewContext(iolib.WithStreams(warningTestStreams{stderr: &output})) + require.NoError(t, err) + ui.InitFormatter(ioCtx) + t.Cleanup(ui.ReinitFormatter) + info := &schema.ConfigAndStacksInfo{Stack: "dev", ComponentFromArg: "app"} + + require.ErrorIs(t, endIgnoredRun(info, &schema.ComponentIgnoredRun{Policy: "quiet"}), errUtils.ErrComponentRunIgnored) + // CI runners advertise color, so strip the ANSI styling before matching the text. + plain := ansi.Strip(output.String()) + require.Contains(t, plain, "Ignoring an error in app in dev because it matches policy quiet\n") + require.NotContains(t, plain, "quiet:", "no message means no trailing colon") +} + +func TestWarnOverlappingRetryOnlyForBothMechanisms(t *testing.T) { + retry := &schema.PreparedComponentConfiguration{Execution: &schema.ComponentExecution{Errors: &schema.ComponentErrors{Retry: []schema.ComponentRetryPolicy{{Name: "n"}}}}} + withSettings := &schema.ConfigAndStacksInfo{ComponentRetrySection: &schema.RetryConfig{Conditions: []string{"x"}}} + var output bytes.Buffer + ioCtx, err := iolib.NewContext(iolib.WithStreams(warningTestStreams{stderr: &output})) + require.NoError(t, err) + ui.InitFormatter(ioCtx) + t.Cleanup(ui.ReinitFormatter) + for _, tc := range []struct { + name string + info *schema.ConfigAndStacksInfo + prepared *schema.PreparedComponentConfiguration + wantWarning bool + }{ + {"both", withSettings, retry, true}, + {"no settings", &schema.ConfigAndStacksInfo{}, retry, false}, + {"no configuration", withSettings, nil, false}, + {"no execution", withSettings, &schema.PreparedComponentConfiguration{}, false}, + } { + t.Run(tc.name, func(t *testing.T) { + output.Reset() + require.NotPanics(t, func() { warnOverlappingRetry(tc.info, tc.prepared) }) + if tc.wantWarning { + require.Contains(t, output.String(), "has both") + } else { + require.Empty(t, output.String()) + } + }) + } +} + // The flag can come from TF_CLI_ARGS_plan or TF_CLI_ARGS, and a command-line setting takes precedence. func TestDetailedExitCodeEnabledFollowsTerraformPrecedence(t *testing.T) { t.Setenv("TF_CLI_ARGS_plan", "") @@ -717,6 +969,157 @@ func TestMainCommandLegacyRetryDetailedPlan(t *testing.T) { } } +func TestReadConfigHooksRunOncePerPrepareInTheUnitDirectory(t *testing.T) { + unit := t.TempDir() + hook := func(name string, commands ...string) schema.ComponentHook { + return schema.ComponentHook{Name: name, Commands: commands, Execute: []string{"hook-" + name}, WorkingDirectory: unit} + } + prepared := &schema.PreparedComponentConfiguration{Execution: &schema.ComponentExecution{ + AfterHooks: []schema.ComponentHook{hook("alias", "read-config"), hook("canonical", "terragrunt-read-config"), hook("plain", "plan")}, + BeforeHooks: []schema.ComponentHook{hook("before", "terragrunt-read-config")}, + }} + shell := newShellFixture(t, nil) + info := hookInfo("plan", nil) + require.NoError(t, hookExecutor(shell).runReadConfigHooks(t.Context(), &schema.AtmosConfiguration{}, info, prepared)) + require.Equal(t, []string{"hook-alias", "hook-canonical"}, shell.names(), "only after hooks run, and only those listing the lifecycle") + require.Equal(t, unit, shell.calls[0].dir) + require.Contains(t, shell.calls[0].env, "TG_CTX_COMMAND=terragrunt-read-config") + + // A dry run, a unit without hooks, and a failing hook. + dry := hookInfo("plan", nil) + dry.DryRun = true + quiet := newShellFixture(t, nil) + require.NoError(t, hookExecutor(quiet).runReadConfigHooks(t.Context(), &schema.AtmosConfiguration{}, dry, prepared)) + require.NoError(t, hookExecutor(quiet).runReadConfigHooks(t.Context(), &schema.AtmosConfiguration{}, info, nil)) + require.NoError(t, hookExecutor(quiet).runReadConfigHooks(t.Context(), &schema.AtmosConfiguration{}, info, &schema.PreparedComponentConfiguration{Execution: &schema.ComponentExecution{AfterHooks: []schema.ComponentHook{hook("plain", "plan")}}})) + require.Empty(t, quiet.calls) + + failing := newShellFixture(t, func(*shellCall) error { return errors.New("boom") }) + err := hookExecutor(failing).runReadConfigHooks(t.Context(), &schema.AtmosConfiguration{}, info, prepared) + require.ErrorIs(t, err, errUtils.ErrComponentHookFailed) +} + +func TestInitFromModuleHooksWrapTheSourceFetch(t *testing.T) { + unit := t.TempDir() + execution := &schema.ComponentExecution{ + BeforeHooks: []schema.ComponentHook{{Name: "before", Commands: []string{"init-from-module"}, Execute: []string{"hook-before"}, WorkingDirectory: unit}}, + AfterHooks: []schema.ComponentHook{{Name: "after", Commands: []string{"init-from-module", "plan"}, Execute: []string{"hook-after"}, WorkingDirectory: unit}}, + ErrorHooks: []schema.ComponentHook{{Name: "report", Commands: []string{"init-from-module"}, Execute: []string{"hook-report"}, OnErrors: []string{"download failed"}}}, + } + shell := newShellFixture(t, nil) + e := hookExecutor(shell) + info := hookInfo("plan", execution) + + wrapper := e.initFromModuleWrapper(&schema.AtmosConfiguration{}, info) + require.NotNil(t, wrapper) + var timeline []string + require.NoError(t, wrapper(t.Context(), &source.Target{Dir: unit}, func(context.Context) error { + timeline = append(timeline, strings.Join(shell.names(), ",")) + return nil + })) + require.Equal(t, []string{"hook-before"}, strings.Split(timeline[0], ","), "the before hook ran before the fetch") + require.Equal(t, []string{"hook-before", "hook-after"}, shell.names()) + + shell.calls = nil + err := wrapper(t.Context(), &source.Target{Dir: unit}, func(context.Context) error { return errors.New("download failed: 404") }) + require.Error(t, err) + require.Equal(t, []string{"hook-before", "hook-report"}, shell.names(), "a failed fetch skips the after hook and runs the matching error hook") + + require.Nil(t, e.initFromModuleWrapper(&schema.AtmosConfiguration{}, hookInfo("plan", nil))) + require.Nil(t, e.initFromModuleWrapper(&schema.AtmosConfiguration{}, hookInfo("plan", &schema.ComponentExecution{BeforeHooks: []schema.ComponentHook{execHook("x", "plan")}}))) + ctx := e.withInitFromModule(t.Context(), &schema.AtmosConfiguration{}, hookInfo("plan", nil)) + require.Equal(t, t.Context(), ctx, "a unit without lifecycle hooks leaves the context alone") + require.NotEqual(t, t.Context(), e.withInitFromModule(t.Context(), &schema.AtmosConfiguration{}, info)) +} + +func TestSourceFetchFollowsTheErrorsPolicyInsideTheHooks(t *testing.T) { + unit := t.TempDir() + hook := schema.ComponentHook{Name: "fetch", Commands: []string{"init-from-module"}, Execute: []string{"hook-fetch"}, WorkingDirectory: unit} + policy := &schema.ComponentErrors{ + Retry: []schema.ComponentRetryPolicy{{Name: "net", Patterns: []string{"flaky download"}, MaxAttempts: 3}}, + Ignore: []schema.ComponentIgnorePolicy{{Name: "gone", Patterns: []string{"permanently gone"}}}, + } + shell := newShellFixture(t, nil) + e := hookExecutor(shell) + info := hookInfo("plan", &schema.ComponentExecution{BeforeHooks: []schema.ComponentHook{hook}, Errors: policy}) + + wrapper := e.initFromModuleWrapper(&schema.AtmosConfiguration{}, info) + require.NotNil(t, wrapper) + attempts := 0 + require.NoError(t, wrapper(t.Context(), &source.Target{Dir: unit}, func(context.Context) error { + attempts++ + if attempts < 3 { + return errors.New("flaky download") + } + return nil + })) + require.Equal(t, 3, attempts) + require.Equal(t, []string{"hook-fetch"}, shell.names(), "the hooks wrap the retries, they do not repeat with them") + + err := wrapper(t.Context(), &source.Target{Dir: unit}, func(context.Context) error { return errors.New("permanently gone") }) + require.ErrorIs(t, err, errUtils.ErrComponentRunIgnored) + err = wrapper(t.Context(), &source.Target{Dir: unit}, func(context.Context) error { return errors.New("other failure") }) + require.EqualError(t, err, "other failure") + + // An errors policy alone is enough to wrap the fetch. + require.NotNil(t, e.initFromModuleWrapper(&schema.AtmosConfiguration{}, hookInfo("plan", &schema.ComponentExecution{Errors: policy}))) +} + +func TestIgnoredSourceRefreshWritesSignalsInModuleDirectory(t *testing.T) { + workdir := t.TempDir() + moduleDir := filepath.Join(workdir, "module") + require.NoError(t, os.Mkdir(moduleDir, 0o755)) + policy := &schema.ComponentErrors{Ignore: []schema.ComponentIgnorePolicy{{ + Name: "missing", Patterns: []string{"download failed"}, Signals: map[string]any{"fetch_failed": true}, + }}} + info := hookInfo("plan", &schema.ComponentExecution{Errors: policy}) + info.BaseComponentPath = "module" + wrapper := hookExecutor(newShellFixture(t, nil)).initFromModuleWrapper(&schema.AtmosConfiguration{}, info) + err := wrapper(t.Context(), &source.Target{Dir: workdir, IsWorkdir: true}, + func(context.Context) error { return errors.New("download failed") }) + require.ErrorIs(t, err, errUtils.ErrComponentRunIgnored) + data, err := os.ReadFile(filepath.Join(moduleDir, "error-signals.json")) + require.NoError(t, err) + require.JSONEq(t, `{"fetch_failed": true}`, string(data)) + require.NoFileExists(t, filepath.Join(workdir, "error-signals.json")) +} + +func TestSourceFetchResolvesAnExplicitSubdirectoryAfterProvisioning(t *testing.T) { + workdir := t.TempDir() + policy := &schema.ComponentErrors{Ignore: []schema.ComponentIgnorePolicy{{ + Name: "gone", Patterns: []string{"download failed"}, Signals: map[string]any{"fetch_failed": true}, + }}} + info := hookInfo("plan", &schema.ComponentExecution{Errors: policy}) + info.ComponentSection = map[string]any{"metadata": map[string]any{"source_subdir": "modules/app"}} + wrapper := hookExecutor(newShellFixture(t, nil)).initFromModuleWrapper(&schema.AtmosConfiguration{}, info) + target := &source.Target{Dir: workdir, IsWorkdir: true} + + // The module directory is created by the fetch, so it does not exist when the fetch starts. + require.NoError(t, wrapper(t.Context(), target, func(context.Context) error { + return os.MkdirAll(filepath.Join(workdir, "modules", "app"), 0o755) + })) + + // A failed fetch that provisioned nothing is still ignored, and has no module directory for signals. + empty := t.TempDir() + err := wrapper(t.Context(), &source.Target{Dir: empty, IsWorkdir: true}, func(context.Context) error { return errors.New("download failed") }) + require.ErrorIs(t, err, errUtils.ErrComponentRunIgnored) + require.NoDirExists(t, filepath.Join(empty, "modules")) + + // An invalid subdirectory is not a pending one. + info.ComponentSection = map[string]any{"metadata": map[string]any{"source_subdir": "../escape"}} + err = wrapper(t.Context(), &source.Target{Dir: empty, IsWorkdir: true}, func(context.Context) error { return errors.New("download failed") }) + require.ErrorIs(t, err, errUtils.ErrComponentRunIgnored, "the ignore decision does not depend on where signals would go") +} + +func TestOnlyRunIgnoredKeepsIndependentHookFailures(t *testing.T) { + require.True(t, onlyRunIgnored(errUtils.ErrComponentRunIgnored)) + require.True(t, onlyRunIgnored(fmt.Errorf("fetch: %w", errUtils.ErrComponentRunIgnored))) + require.False(t, onlyRunIgnored(nil)) + hookFailure := fmt.Errorf("%w: after_hook %q", errUtils.ErrComponentHookFailed, "fetch-after") + require.False(t, onlyRunIgnored(errors.Join(errUtils.ErrComponentRunIgnored, hookFailure)), + "an init-from-module hook that failed beside the ignored fetch still fails the command") +} + // TestPolicyPlanUsesProviderOutputArguments verifies that a plan policy selects and binds // the output path Terraform receives, including one supplied only by provider extra arguments. func TestPolicyPlanUsesProviderOutputArguments(t *testing.T) { diff --git a/pkg/component/terraform/execute_init.go b/pkg/component/terraform/execute_init.go index 3cf7befed5b..f5b653d0a69 100644 --- a/pkg/component/terraform/execute_init.go +++ b/pkg/component/terraform/execute_init.go @@ -223,7 +223,7 @@ func prepareInitExecution(ctx context.Context, writers provisioner.OutputWriters // invocation via prepareInitExecution. These two code paths must never both execute // in the same command invocation or provisioners will run twice. func (e *Executor) executeTerraformInitPhase(atmosConfig *schema.AtmosConfiguration, info *schema.ConfigAndStacksInfo, componentPath, varFile string, opts ...component.ExecutionOption) (string, error) { - newPath, err := prepareInitExecution(component.ContextFromOptions(opts...), executionOutputWriters(opts...), atmosConfig, info, componentPath) + newPath, err := prepareInitExecution(e.withInitFromModule(component.ContextFromOptions(opts...), atmosConfig, info), executionOutputWriters(opts...), atmosConfig, info, componentPath) if err != nil { return componentPath, err } diff --git a/pkg/component/terraform/execute_pipeline.go b/pkg/component/terraform/execute_pipeline.go index dc753f760e2..72c321a29be 100644 --- a/pkg/component/terraform/execute_pipeline.go +++ b/pkg/component/terraform/execute_pipeline.go @@ -3,13 +3,8 @@ package terraform import ( "bytes" "context" - "crypto/sha256" - "encoding/hex" "errors" "fmt" - "io/fs" - "maps" - "os" "path/filepath" "slices" "strings" @@ -65,9 +60,12 @@ func (e *Executor) prepareComponentExecution( if err != nil { return nil, err } + if err = e.beginPreparedConfiguration(ctx, atmosConfig, info, prepared); err != nil { + return nil, err + } // A restored plugin plan was validated when it was created. Its binding omits // variable values, so validating the reconstructed component would test empty vars. - restoredConfigurationPlan := prepared != nil && isRestoredConfigurationPlan(info) + restoredConfigurationPlan := isRestoredConfigurationPlan(info, prepared) componentPath, err := e.resolveConfiguredComponentPath(ctx, writers, atmosConfig, info, prepared) if err != nil { @@ -100,6 +98,32 @@ func (e *Executor) prepareComponentExecution( }, nil } +// beginPreparedConfiguration stops an excluded component, or one whose preparation an ignore policy +// ended, before anything runs, then warns about +// overlapping retries, clears stale signal files and runs the provider's read-configuration hooks. +func (e *Executor) beginPreparedConfiguration( + ctx context.Context, + atmosConfig *schema.AtmosConfiguration, + info *schema.ConfigAndStacksInfo, + prepared *schema.PreparedComponentConfiguration, +) error { + if excludedFromOperation(info, prepared) { + skipExcluded(info) + return errUtils.ErrComponentExcluded + } + if prepared != nil && prepared.IgnoredRun != nil { + if err := refreshIgnoredSignals(atmosConfig, info, prepared); err != nil { + return err + } + return endIgnoredRun(info, prepared.IgnoredRun) + } + warnOverlappingRetry(info, prepared) + if err := clearStaleSignals(atmosConfig, info); err != nil { + return err + } + return e.runReadConfigHooks(ctx, atmosConfig, info, prepared) +} + // terraformTestVarfileName returns the test var file name, or "" when the component has no test vars. func terraformTestVarfileName(info *schema.ConfigAndStacksInfo) string { if !hasTerraformTestVars(info) { @@ -109,8 +133,8 @@ func terraformTestVarfileName(info *schema.ConfigAndStacksInfo) string { } // isRestoredConfigurationPlan reports whether the invocation applies a provider-owned saved plan. -func isRestoredConfigurationPlan(info *schema.ConfigAndStacksInfo) bool { - return ownsConfigurationPlan(info) && usesConfigurationPlan(info) && restoresConfigurationPlan(info) +func isRestoredConfigurationPlan(info *schema.ConfigAndStacksInfo, prepared *schema.PreparedComponentConfiguration) bool { + return prepared != nil && ownsConfigurationPlan(info) && usesConfigurationPlan(info) && restoresConfigurationPlan(info) } // installToolchainForCommand installs the toolchain dependencies and resolves the command against them. @@ -326,6 +350,17 @@ func (e *Executor) executeCommandPipeline( return nil } +// runPreparedPipeline translates an ignored preparatory failure into the +// successful end of this unit invocation, after the pipeline has stopped. +func (e *Executor) runPreparedPipeline(atmosConfig *schema.AtmosConfiguration, info *schema.ConfigAndStacksInfo, execCtx *componentExecContext, opts ...component.ExecutionOption) error { + err := e.executeCommandPipeline(atmosConfig, info, execCtx, opts...) + if onlyRunIgnored(err) { + cleanupTerraformFiles(atmosConfig, info) + return nil + } + return err +} + // runMainCommandPhase runs the main plan/apply/destroy/… subcommand under its own CI phase // group, capturing scoped exec-metadata output (FR-006f), and — on failure — gives smart init // exactly one chance to recover via recoverFromInitRequired before propagating the error. @@ -356,7 +391,8 @@ func (e *Executor) runMainCommandPhase( //nolint:revive // argument-limit: every defer restore() var execMetadataStdoutBuf, execMetadataStderrBuf bytes.Buffer var effectiveArgs []string - captureCtx := context.WithValue(component.ContextFromOptions(opts...), mainCommandArgsKey{}, &effectiveArgs) + var ignored bool + captureCtx := context.WithValue(context.WithValue(component.ContextFromOptions(opts...), mainCommandArgsKey{}, &effectiveArgs), mainCommandIgnoredKey{}, &ignored) captureCtx = execCtx.observePlanSnapshots(captureCtx, atmosConfig, info, allArgsAndFlags) captureArgs := component.WithProcessContext(captureCtx) // A component with provider-supplied hooks recovers inside its main-command lifecycle, so @@ -385,6 +421,10 @@ func (e *Executor) runMainCommandPhase( //nolint:revive // argument-limit: every if mainErr != nil && !completedPlanWithChanges(mainErr) { return mainErr } + // An ignored failure produced no plan to bind, so the run ends without recording one. + if ignored { + return nil + } // Policy plans run against a private artifact that is removed after the guard // publishes it to the original output path. The effective arguments were // validated before execution; bind the published destination instead. @@ -568,97 +608,3 @@ func addTerraformTestVarfileArg(info *schema.ConfigAndStacksInfo, testVarFile st } info.AdditionalArgsAndFlags = append([]string{varFileFlag, testVarFile}, info.AdditionalArgsAndFlags...) } - -// withoutSensitiveContent copies files, dropping the bytes of every sensitive one. -func withoutSensitiveContent(files []schema.ComponentConfigurationFile) []schema.ComponentConfigurationFile { - stored := slices.Clone(files) - for i := range stored { - if stored[i].Sensitive { - stored[i].Content = nil - } - } - return stored -} - -// restoreSensitiveContent reads the bytes of each sensitive file back from the working directory and -// restores them only when they are the reviewed ones. The working directory digest excludes some -// locations, so the reviewed bytes are verified here against the digest recorded with the plan. -func restoreSensitiveContent(workdir string, files []schema.ComponentConfigurationFile, reviewed string) error { - contents, err := readSensitiveFiles(workdir, files) - if err != nil { - return err - } - if contents == nil { - return nil - } - for i := range files { - // A file the entry removes is absent by design, and the digest still binds that absence. - if files[i].Sensitive && contents[files[i].Path] == nil && !slices.Contains([]string{"remove", "remove_terragrunt"}, files[i].IfExists) { - return fmt.Errorf("%w: reviewed generated file %s is missing: %w", errUtils.ErrPlanVerificationFailed, files[i].Path, fs.ErrNotExist) - } - } - if digestSensitiveContents(contents) != reviewed { - return fmt.Errorf("%w: reviewed generated file changed", errUtils.ErrPlanVerificationFailed) - } - for i := range files { - if files[i].Sensitive { - files[i].Content = contents[files[i].Path] - } - } - return nil -} - -// sensitiveDigest fingerprints the sensitive generated files as they are in the working directory. -// It is empty when no file is sensitive. -func sensitiveDigest(workdir string, files []schema.ComponentConfigurationFile) (string, error) { - contents, err := readSensitiveFiles(workdir, files) - if err != nil || contents == nil { - return "", err - } - return digestSensitiveContents(contents), nil -} - -// readSensitiveFiles reads every sensitive file below workdir, keyed by path. A missing file maps -// to nil content. It returns nil when no file is sensitive. -func readSensitiveFiles(workdir string, files []schema.ComponentConfigurationFile) (map[string][]byte, error) { - if !slices.ContainsFunc(files, func(file schema.ComponentConfigurationFile) bool { return file.Sensitive }) { - return nil, nil - } - root, err := os.OpenRoot(workdir) - if err != nil { - return nil, planIOError(err) - } - defer root.Close() - contents := make(map[string][]byte) - for i := range files { - if !files[i].Sensitive { - continue - } - content, err := root.ReadFile(files[i].Path) - if errors.Is(err, fs.ErrNotExist) { - contents[files[i].Path] = nil - continue - } - if err != nil { - return nil, fmt.Errorf("%w: reviewed generated file %s cannot be read: %w", errUtils.ErrPlanVerificationFailed, files[i].Path, err) - } - if content == nil { - content = []byte{} - } - contents[files[i].Path] = content - } - return contents, nil -} - -// digestSensitiveContents hashes the paths and bytes of the sensitive files in a stable order. -func digestSensitiveContents(contents map[string][]byte) string { - hash := sha256.New() - for _, path := range slices.Sorted(maps.Keys(contents)) { - if contents[path] == nil { - fmt.Fprintf(hash, "%s\x00absent\n", path) - continue - } - fmt.Fprintf(hash, "%s\x00%x\n", path, sha256.Sum256(contents[path])) - } - return hex.EncodeToString(hash.Sum(nil)) -} diff --git a/pkg/component/terraform/execute_prepare.go b/pkg/component/terraform/execute_prepare.go index c024b59afe4..35bbe05dc30 100644 --- a/pkg/component/terraform/execute_prepare.go +++ b/pkg/component/terraform/execute_prepare.go @@ -14,6 +14,7 @@ import ( component "github.com/cloudposse/atmos/pkg/component" cfg "github.com/cloudposse/atmos/pkg/config" degradation "github.com/cloudposse/atmos/pkg/degradation" + downloader "github.com/cloudposse/atmos/pkg/downloader" atmosio "github.com/cloudposse/atmos/pkg/io" log "github.com/cloudposse/atmos/pkg/logger" perf "github.com/cloudposse/atmos/pkg/perf" @@ -51,6 +52,9 @@ func (e *Executor) resolveAndProvisionComponentPath(ctx context.Context, writers } ctx, cancel := context.WithTimeout(ctx, 5*time.Minute) defer cancel() + ctx = e.withInitFromModule(ctx, atmosConfig, info) + ctx = downloader.WithSourceAuthorization(ctx, info.SourceAuthorization) + ctx = downloader.WithAWSSession(ctx, info.IAMSession) componentPath, componentPathExists, err := provisionAndResolveTerraformComponentPath( ctx, writers, atmosConfig, info, cfg.TerraformComponentType, componentPath, ) diff --git a/pkg/component/terraform/execute_recovery.go b/pkg/component/terraform/execute_recovery.go new file mode 100644 index 00000000000..84b793852ab --- /dev/null +++ b/pkg/component/terraform/execute_recovery.go @@ -0,0 +1,263 @@ +package terraform + +import ( + "bytes" + "context" + "fmt" + "io" + "path/filepath" + "slices" + "strings" + "time" + + errUtils "github.com/cloudposse/atmos/errors" + "github.com/cloudposse/atmos/pkg/component" + "github.com/cloudposse/atmos/pkg/component/execution" + "github.com/cloudposse/atmos/pkg/perf" + "github.com/cloudposse/atmos/pkg/provisioner/source" + "github.com/cloudposse/atmos/pkg/schema" + "github.com/cloudposse/atmos/pkg/ui" +) + +// recoveryReporter prints what error handling decided for one component. +type recoveryReporter struct { + info *schema.ConfigAndStacksInfo +} + +func (r *recoveryReporter) label() string { + return fmt.Sprintf("`%s` in `%s`", r.info.ComponentFromArg, r.info.Stack) +} + +// Retrying implements execution.Reporter. +func (r *recoveryReporter) Retrying(policy string, attempt, maxAttempts int, sleep time.Duration) { + defer perf.Track(nil, "terraform.recoveryReporter.Retrying")() + + ui.Warningf("Retryable error matched policy `%s` for %s (attempt %d of %d). Retrying in %s.", policy, r.label(), attempt, maxAttempts, sleep) +} + +// Ignored implements execution.Reporter. +func (r *recoveryReporter) Ignored(policy, message string) { + defer perf.Track(nil, "terraform.recoveryReporter.Ignored")() + + text := fmt.Sprintf("Ignoring an error in %s because it matches policy `%s`", r.label(), policy) + if message != "" { + text += ": " + message + } + ui.Warning(text) +} + +// SignalsWritten implements execution.Reporter. +func (r *recoveryReporter) SignalsWritten(path string) { + defer perf.Track(nil, "terraform.recoveryReporter.SignalsWritten")() + + ui.Warningf("Wrote error signals to `%s`.", path) +} + +// attemptFunc runs the hooks and the command once. It returns the exit status 2 of +// `plan -detailed-exitcode` separately, because that is a result and not a failure. +type attemptFunc func(opts []component.ExecutionOption, hookStderr io.Writer) (changes, err error) + +// recoverPhase runs an attempt under the component's errors policy. Error text for matching is +// the command's standard error, as it is for Terragrunt, so output on stdout never triggers a policy. +func (e *Executor) recoverPhase(info *schema.ConfigAndStacksInfo, phase *executionPhase, policy *schema.ComponentErrors, opts []component.ExecutionOption, attempt attemptFunc) error { + defer perf.Track(nil, "terraform.Executor.recoverPhase")() + + var changes error + attempts := 0 + recoverer := execution.NewRecoverer( + policy, + execution.WithReporter(&recoveryReporter{info: info}), + execution.WithSignalsFile(filepath.Join(phase.componentPath, execution.SignalsFileName)), + ) + stdoutMetadata, stderrMetadata := component.MetadataOutputCaptureFromOptions(opts...) + ignored, err := recoverer.Run(component.ContextFromOptions(opts...), func(context.Context) (string, error) { + attempts++ + if attempts > 1 { + resetExecMetadataBufs(stdoutMetadata, stderrMetadata) + } + var stderr bytes.Buffer + var runErr error + changes, runErr = attempt(withStderrCapture(opts, &stderr), &stderr) + return stderr.String(), runErr + }) + switch { + case err != nil: + return err + case ignored && !phase.main: + // A preparatory step such as the implicit init failed with an ignorable error. Terragrunt ends + // the whole unit run there, so the command that follows must not run against a missing init. + return errUtils.ErrComponentRunIgnored + case ignored: + markMainCommandIgnored(component.ContextFromOptions(opts...)) + return nil + } + return changes +} + +// withStderrCapture adds a standard error capture beside any capture the caller already has. +func withStderrCapture(opts []component.ExecutionOption, buffer *bytes.Buffer) []component.ExecutionOption { + var probe component.ExecutionOptions + for _, opt := range opts { + opt(&probe) + } + var writer io.Writer = &syncWriter{w: buffer} + if probe.StderrCapture != nil { + writer = io.MultiWriter(probe.StderrCapture, writer) + } + return append(slices.Clone(opts), component.WithStderrCapture(writer)) +} + +// warnOverlappingRetry tells the user that two retry mechanisms apply to the same component. +// `settings.retry` wraps each Terraform subprocess and runs first; an errors block retries what is +// left after it is exhausted, so one failure can be attempted more often than either alone. +func warnOverlappingRetry(info *schema.ConfigAndStacksInfo, prepared *schema.PreparedComponentConfiguration) { + if prepared == nil || prepared.Execution == nil || prepared.Execution.Errors == nil || len(prepared.Execution.Errors.Retry) == 0 { + return + } + if info.ComponentRetrySection == nil || len(info.ComponentRetrySection.Conditions) == 0 { + return + } + ui.Warningf("Component `%s` in `%s` has both `settings.retry` and a Terragrunt `errors.retry` policy. `settings.retry` retries each Terraform command first; the `errors.retry` policy then retries what is left, so a failure can be attempted more often than either setting alone allows.", info.ComponentFromArg, info.Stack) +} + +// excludedFromOperation reports whether the provider says this operation must not run the unit. +func excludedFromOperation(info *schema.ConfigAndStacksInfo, prepared *schema.PreparedComponentConfiguration) bool { + if prepared == nil || prepared.Exclusion == nil { + return false + } + command := info.SubCommand + if command == subcommandDeploy { + command = subcommandApply + } + return slices.Contains(prepared.Exclusion.Actions, execution.CommandName(command, info.AdditionalArgsAndFlags)) +} + +// skipExcluded announces why a unit does not run. +func skipExcluded(info *schema.ConfigAndStacksInfo) { + ui.Infof("Skipping `%s` for `%s` in `%s`: its Terragrunt `exclude` block sets `no_run`.", info.SubCommand, info.ComponentFromArg, info.Stack) +} + +// endIgnoredRun ends the unit's run successfully after an ignore policy matched a failure that left it +// without a resolved source. It announces the decision as an ignored source fetch does. +func endIgnoredRun(info *schema.ConfigAndStacksInfo, ignored *schema.ComponentIgnoredRun) error { + (&recoveryReporter{info: info}).Ignored(ignored.Policy, ignored.Message) + return errUtils.ErrComponentRunIgnored +} + +// runReadConfigHooks runs the unit's read-config after hooks once its configuration is read. They +// run on every command that prepares the unit, in the unit's configuration directory, before any +// source is fetched. Hooks written for `read-config` and `terragrunt-read-config` both run. +func (e *Executor) runReadConfigHooks(ctx context.Context, atmosConfig *schema.AtmosConfiguration, info *schema.ConfigAndStacksInfo, prepared *schema.PreparedComponentConfiguration) error { + defer perf.Track(atmosConfig, "terraform.Executor.runReadConfigHooks")() + + if prepared == nil || prepared.Execution == nil || info.DryRun { + return nil + } + hooks := execution.HooksFor(prepared.Execution.AfterHooks, execution.CommandReadConfig, execution.CommandReadConfigAlias) + if len(hooks) == 0 { + return nil + } + stage := lifecycleStage{command: execution.CommandReadConfig, config: &schema.ComponentExecution{AfterHooks: hooks}} + return e.runLifecycle(ctx, atmosConfig, info, stage, func(context.Context) error { return nil }) +} + +// initFromModuleWrapper runs the unit's init-from-module hooks around a source fetch. It returns +// nil when no hook lists the stage, so units without such hooks pay nothing. +func (e *Executor) initFromModuleWrapper(atmosConfig *schema.AtmosConfiguration, info *schema.ConfigAndStacksInfo) source.ProvisionWrapper { + config := info.ComponentExecution + recovered := config != nil && config.Errors != nil + if !recovered && !hasLifecycleHooks(config, execution.CommandInitFromModule) { + return nil + } + stage := lifecycleStage{command: execution.CommandInitFromModule, config: config} + return func(ctx context.Context, target *source.Target, provision func(context.Context) error) error { + fetch := provision + if recovered { + fetch = recoveringFetch(info, target, config.Errors, provision) + } + return e.runLifecycle(ctx, atmosConfig, info, stage, fetch) + } +} + +// recoveringFetch applies the errors policy to the source fetch, inside the init-from-module +// hooks, as Terragrunt does for its download. An ignored failure ends the unit's run, because +// there is no source to run the command against. +func recoveringFetch(info *schema.ConfigAndStacksInfo, target *source.Target, policy *schema.ComponentErrors, provision func(context.Context) error) func(context.Context) error { + return func(ctx context.Context) error { + // An explicit source subdirectory exists only once the fetch has provisioned the source, so the + // directory that receives ignore signals is resolved when they are written. + recoverer := execution.NewRecoverer(policy, execution.WithReporter(&recoveryReporter{info: info}), + execution.WithSignalsPath(func() string { return fetchedSignalsFile(info, target) })) + ignored, err := recoverer.Run(ctx, func(ctx context.Context) (string, error) { + if err := provision(ctx); err != nil { + return err.Error(), err + } + return "", nil + }) + if err != nil { + return err + } + if ignored { + return errUtils.ErrComponentRunIgnored + } + return nil + } +} + +// fetchedSignalsFile returns the signals file in the module directory of a fetched source, or "" when +// that directory does not exist, which is the case for an ignored fetch that provisioned nothing. +func fetchedSignalsFile(info *schema.ConfigAndStacksInfo, target *source.Target) string { + moduleDir := target.Dir + if target.IsWorkdir { + var present bool + var err error + moduleDir, present, err = component.ResolvePendingSubpath(info, target.Dir) + if err != nil || !present { + return "" + } + } + return filepath.Join(moduleDir, execution.SignalsFileName) +} + +// hasLifecycleHooks reports whether any before, after, or error hook lists the command. +func hasLifecycleHooks(config *schema.ComponentExecution, command string) bool { + if config == nil { + return false + } + return execution.ListsCommand(config.BeforeHooks, command) || execution.ListsCommand(config.AfterHooks, command) || execution.ListsCommand(config.ErrorHooks, command) +} + +// lifecycleStage is a provider stage and the hooks that wrap it. +type lifecycleStage struct { + command string + config *schema.ComponentExecution +} + +// withInitFromModule makes source provisioning run the init-from-module hooks, when the unit has any. +func (e *Executor) withInitFromModule(ctx context.Context, atmosConfig *schema.AtmosConfiguration, info *schema.ConfigAndStacksInfo) context.Context { + if wrapper := e.initFromModuleWrapper(atmosConfig, info); wrapper != nil { + return source.WithProvisionWrapper(ctx, wrapper) + } + return ctx +} + +// runLifecycle runs action wrapped by the hooks of one lifecycle stage. +func (e *Executor) runLifecycle(ctx context.Context, atmosConfig *schema.AtmosConfiguration, info *schema.ConfigAndStacksInfo, stage lifecycleStage, action func(context.Context) error) error { + phase := &executionPhase{command: stage.command} + hookOpts := hookOptions([]component.ExecutionOption{component.WithProcessContext(ctx), component.WithEnvironment(info.SanitizedEnv)}) + // Error hooks match on the output of the hooks that ran before them, so their output is recorded for as + // long as the lifecycle runs, including the matching. + var capture *phaseCapture + if stage.config != nil && len(stage.config.ErrorHooks) > 0 { + capture = &phaseCapture{} + defer capture.cleanup() + } + runner := &hookRunner{executor: e, atmosConfig: atmosConfig, info: info, phase: phase, opts: hookOpts, capture: capture} + return execution.Run(ctx, stage.config, stage.command, runner, func() (execution.Output, error) { + err := action(ctx) + if err == nil { + return nil, nil + } + return execution.TextOutput(strings.TrimSpace(err.Error())), err + }) +} diff --git a/pkg/component/terraform/execute_signals.go b/pkg/component/terraform/execute_signals.go new file mode 100644 index 00000000000..3a039beffa4 --- /dev/null +++ b/pkg/component/terraform/execute_signals.go @@ -0,0 +1,114 @@ +package terraform + +import ( + "errors" + "fmt" + "os" + "path/filepath" + + errUtils "github.com/cloudposse/atmos/errors" + "github.com/cloudposse/atmos/pkg/component" + "github.com/cloudposse/atmos/pkg/component/execution" + cfg "github.com/cloudposse/atmos/pkg/config" + "github.com/cloudposse/atmos/pkg/provisioner/source" + "github.com/cloudposse/atmos/pkg/provisioner/workdir" + "github.com/cloudposse/atmos/pkg/schema" +) + +// clearStaleSignals runs once during unit preparation, before read-config hooks +// or a source refresh can produce new signals for this invocation. +func clearStaleSignals(atmosConfig *schema.AtmosConfiguration, info *schema.ConfigAndStacksInfo) error { + if info.DryRun { + return nil + } + moduleDir, err := signalModuleDirectory(atmosConfig, info) + if err != nil { + return err + } + // A module directory that the source fetch has not created yet holds no stale signals. + if moduleDir == "" { + return nil + } + path := filepath.Join(moduleDir, execution.SignalsFileName) + if err := os.Remove(path); err != nil && !errors.Is(err, os.ErrNotExist) { + return fmt.Errorf("%w: clear stale error signals from %s: %w", errUtils.ErrGeneratorRemoveFailed, path, err) + } + return nil +} + +// signalModuleDirectory returns the module directory the unit runs in, or "" when it does not exist yet. +func signalModuleDirectory(atmosConfig *schema.AtmosConfiguration, info *schema.ConfigAndStacksInfo) (string, error) { + section := info.ComponentSection + if source.HasSource(section) { + target, err := source.ResolveTarget(atmosConfig, cfg.TerraformComponentType, info.FinalComponent, section) + if err != nil { + return "", err + } + if !target.IsWorkdir { + return target.Dir, nil + } + return existingSubpath(info, target.Dir) + } + if workdir.IsWorkdirEnabled(section) { + root, err := workdir.BuildPath(atmosConfig.BasePath, cfg.TerraformComponentType, info.FinalComponent, info.Stack, section) + if err != nil { + return "", err + } + return existingSubpath(info, root) + } + return ConstructTerraformComponentWorkingDir(atmosConfig, info), nil +} + +// existingSubpath resolves the module directory under root, or "" when an explicit source subdirectory +// has not been fetched yet. +func existingSubpath(info *schema.ConfigAndStacksInfo, root string) (string, error) { + moduleDir, present, err := component.ResolvePendingSubpath(info, root) + if err != nil || !present { + return "", err + } + return moduleDir, nil +} + +// refreshIgnoredSignals replaces stale signals in a retained module after preparation is ignored. +func refreshIgnoredSignals(atmosConfig *schema.AtmosConfiguration, info *schema.ConfigAndStacksInfo, prepared *schema.PreparedComponentConfiguration) error { + if info.DryRun { + return nil + } + if err := clearStaleSignals(atmosConfig, info); err != nil { + return err + } + dir, err := signalModuleDirectory(atmosConfig, info) + if err != nil || dir == "" { + return err + } + if present, err := signalDirectoryExists(dir); err != nil || !present { + return err + } + writeIgnoredPolicySignals(info, prepared, dir) + return nil +} + +// writeIgnoredPolicySignals preserves the signals of the policy selected during preparation. +func writeIgnoredPolicySignals(info *schema.ConfigAndStacksInfo, prepared *schema.PreparedComponentConfiguration, dir string) { + if prepared.Execution == nil || prepared.Execution.Errors == nil { + return + } + for _, policy := range prepared.Execution.Errors.Ignore { + if policy.Name == prepared.IgnoredRun.Policy { + execution.NewRecoverer(nil, execution.WithReporter(&recoveryReporter{info: info}), execution.WithSignalsFile(filepath.Join(dir, execution.SignalsFileName))).WriteSignals(&policy) + return + } + } +} + +// signalDirectoryExists avoids creating a module directory for an ignored preparation. +func signalDirectoryExists(dir string) (bool, error) { + stat, err := os.Stat(dir) + if errors.Is(err, os.ErrNotExist) { + return false, nil + } + if err != nil { + return false, err + } + return stat.IsDir(), nil +} diff --git a/pkg/component/terraform/execute_signals_test.go b/pkg/component/terraform/execute_signals_test.go new file mode 100644 index 00000000000..4d75647416d --- /dev/null +++ b/pkg/component/terraform/execute_signals_test.go @@ -0,0 +1,112 @@ +package terraform + +import ( + "errors" + "fmt" + "os" + "path/filepath" + "testing" + + "github.com/stretchr/testify/require" + + errUtils "github.com/cloudposse/atmos/errors" + "github.com/cloudposse/atmos/pkg/component/execution" + + "github.com/cloudposse/atmos/pkg/provisioner/source" + "github.com/cloudposse/atmos/pkg/schema" +) + +func TestClearStaleSignalsUsesProvisionedModuleDirectory(t *testing.T) { + atmosConfig := &schema.AtmosConfiguration{BasePath: t.TempDir()} + section := map[string]any{ + "component": "example", "atmos_stack": "dev", + "source": map[string]any{"uri": "git::https://example.test/repo.git"}, + "provision": map[string]any{"workdir": map[string]any{"enabled": true}}, + } + info := &schema.ConfigAndStacksInfo{ComponentSection: section, FinalComponent: "example", Stack: "dev", BaseComponentPath: "module"} + target, err := source.ResolveTarget(atmosConfig, "terraform", "example", section) + require.NoError(t, err) + moduleDir := filepath.Join(target.Dir, "module") + require.NoError(t, os.MkdirAll(moduleDir, 0o755)) + stale := filepath.Join(moduleDir, "error-signals.json") + require.NoError(t, os.WriteFile(stale, []byte(`{"old":true}`), 0o644)) + require.NoError(t, clearStaleSignals(atmosConfig, info)) + require.NoFileExists(t, stale) + + info.DryRun = true + require.NoError(t, os.WriteFile(stale, []byte(`{"old":true}`), 0o644)) + require.NoError(t, clearStaleSignals(atmosConfig, info)) + require.FileExists(t, stale, "dry runs leave the module untouched") +} + +// TestClearStaleSignalsWrapsRemovalFailures checks that a signals file that cannot be removed fails +// with the static Atmos error and keeps the operating-system error as its cause. +func TestClearStaleSignalsWrapsRemovalFailures(t *testing.T) { + atmosConfig := &schema.AtmosConfiguration{BasePath: t.TempDir()} + section := map[string]any{ + "component": "example", "atmos_stack": "dev", + "source": map[string]any{"uri": "git::https://example.test/repo.git"}, + "provision": map[string]any{"workdir": map[string]any{"enabled": true}}, + } + info := &schema.ConfigAndStacksInfo{ComponentSection: section, FinalComponent: "example", Stack: "dev", BaseComponentPath: "module"} + target, err := source.ResolveTarget(atmosConfig, "terraform", "example", section) + require.NoError(t, err) + // A non-empty directory in place of the signals file cannot be removed on any platform. + blocked := filepath.Join(target.Dir, "module", "error-signals.json") + require.NoError(t, os.MkdirAll(filepath.Join(blocked, "child"), 0o755)) + + err = clearStaleSignals(atmosConfig, info) + require.ErrorIs(t, err, errUtils.ErrGeneratorRemoveFailed) + var pathErr *os.PathError + require.True(t, errors.As(err, &pathErr), "the operating-system error is preserved: %v", err) + require.DirExists(t, blocked) +} + +// A source subdirectory that the fetch has not created yet holds no stale signals, so preparation proceeds +// to the fetch instead of failing on the missing directory. +func TestClearStaleSignalsToleratesAnUnfetchedSourceSubdirectory(t *testing.T) { + atmosConfig := &schema.AtmosConfiguration{BasePath: t.TempDir()} + section := map[string]any{ + "component": "example", "atmos_stack": "dev", + "source": map[string]any{"uri": "git::https://example.test/repo.git"}, + "provision": map[string]any{"workdir": map[string]any{"enabled": true}}, + "metadata": map[string]any{"source_subdir": "modules/app"}, + } + info := &schema.ConfigAndStacksInfo{ComponentSection: section, FinalComponent: "example", Stack: "dev"} + require.NoError(t, clearStaleSignals(atmosConfig, info)) + + target, err := source.ResolveTarget(atmosConfig, "terraform", "example", section) + require.NoError(t, err) + moduleDir := filepath.Join(target.Dir, "modules", "app") + require.NoError(t, os.MkdirAll(moduleDir, 0o755)) + stale := filepath.Join(moduleDir, "error-signals.json") + require.NoError(t, os.WriteFile(stale, []byte(`{"old":true}`), 0o644)) + require.NoError(t, clearStaleSignals(atmosConfig, info)) + require.NoFileExists(t, stale, "a fetched module directory is still cleared") + + section["metadata"] = map[string]any{"source_subdir": "../escape"} + require.ErrorIs(t, clearStaleSignals(atmosConfig, info), errUtils.ErrWorkdirProvision, "an invalid subdirectory still fails") +} + +func TestIgnoredPreparationRefreshesRetainedSignals(t *testing.T) { + for _, signals := range []map[string]any{nil, {"current": true}} { + t.Run(fmt.Sprint(signals), func(t *testing.T) { + root := t.TempDir() + cfg := &schema.AtmosConfiguration{TerraformDirAbsolutePath: root} + info := &schema.ConfigAndStacksInfo{FinalComponent: "app"} + dir := ConstructTerraformComponentWorkingDir(cfg, info) + require.NoError(t, os.MkdirAll(dir, 0o755)) + path := filepath.Join(dir, execution.SignalsFileName) + require.NoError(t, os.WriteFile(path, []byte(`{"stale":true}`), 0o600)) + prepared := &schema.PreparedComponentConfiguration{IgnoredRun: &schema.ComponentIgnoredRun{Policy: "registry"}, Execution: &schema.ComponentExecution{Errors: &schema.ComponentErrors{Ignore: []schema.ComponentIgnorePolicy{{Name: "registry", Signals: signals}}}}} + require.NoError(t, refreshIgnoredSignals(cfg, info, prepared)) + if signals == nil { + require.NoFileExists(t, path) + } else { + data, err := os.ReadFile(path) + require.NoError(t, err) + require.JSONEq(t, `{"current":true}`, string(data)) + } + }) + } +} diff --git a/pkg/component/terraform/generate_planfile.go b/pkg/component/terraform/generate_planfile.go index a0f3ffb8341..04a078e5dba 100644 --- a/pkg/component/terraform/generate_planfile.go +++ b/pkg/component/terraform/generate_planfile.go @@ -5,10 +5,17 @@ import ( "fmt" "os" "path/filepath" + "runtime" + "slices" + "strings" + "github.com/google/shlex" "github.com/spf13/cobra" + errUtils "github.com/cloudposse/atmos/errors" + "github.com/cloudposse/atmos/pkg/component" cfg "github.com/cloudposse/atmos/pkg/config" + envpkg "github.com/cloudposse/atmos/pkg/env" log "github.com/cloudposse/atmos/pkg/logger" "github.com/cloudposse/atmos/pkg/perf" "github.com/cloudposse/atmos/pkg/schema" @@ -257,3 +264,116 @@ func writePlanfile(planFilePath, format string, planJSON string) error { return err } + +// validatePlanOutputFlags rejects ambiguous plan destinations before hooks or Terraform run. +func validatePlanOutputFlags(command string, args []string) error { + if command != "plan" { + return nil + } + count := 0 + for i := 0; i < len(args); i++ { + arg := args[i] + if arg == "-out" || arg == "--out" { + count++ + i++ + } else if strings.HasPrefix(arg, "-out=") || strings.HasPrefix(arg, "--out=") { + count++ + } + if count > 1 { + return fmt.Errorf("%w: multiple plan output flags", errUtils.ErrPlanVerificationFailed) + } + } + return nil +} + +// validateEnvironmentPlanOutput rejects plan output flags in TF_CLI_ARGS or +// TF_CLI_ARGS_plan for a configuration-owned plan. Terraform reads these +// variables itself, so a flag there could choose a plan file that Atmos does not +// see and therefore does not bind. The check inspects the effective value of each +// variable in the subprocess environment: the sanitized process environment (or +// the inherited one when auth did not sanitize it), the global atmos.yaml +// environment, the component environment, and provider extra-argument +// environment, with later layers overriding earlier ones. Hooks run as child +// processes and cannot change it. +func validateEnvironmentPlanOutput(atmosConfig *schema.AtmosConfiguration, info *schema.ConfigAndStacksInfo, execCtx *componentExecContext, opts ...component.ExecutionOption) error { + if execCtx.preparedConfiguration == nil || info.DryRun || info.SubCommand != subcommandPlan { + return nil + } + return rejectEnvironmentPlanOutput(planEnvironment(atmosConfig, info, opts), runtime.GOOS) +} + +// rejectEnvironmentPlanOutput rejects a plan output flag in the effective TF_CLI_ARGS or +// TF_CLI_ARGS_plan of an environment listed in precedence order. Variable names are +// case-insensitive on Windows only, where the last matching entry is the effective value. +func rejectEnvironmentPlanOutput(entries []string, goos string) error { + for _, name := range []string{tfCliArgsEnvVar, tfCliArgsEnvVar + "_" + subcommandPlan} { + value, _ := lastEnvListValue(entries, name, goos) + args, err := tokenizeEnvironmentArgs(value) + if err != nil { + return fmt.Errorf("%w: %s cannot be parsed: %w", errUtils.ErrPlanVerificationFailed, name, err) + } + if containsOutputFlag(args) { + return fmt.Errorf("%w: %s sets a plan output flag; pass -out as a command argument instead", errUtils.ErrPlanVerificationFailed, name) + } + } + return nil +} + +// tokenizeEnvironmentArgs splits a TF_CLI_ARGS value the way the engine does: +// spaces, tabs, and newlines separate tokens outside quotes, and a malformed +// value (for example an unterminated quote) is an error because the engine +// refuses to run with it. The shared parseArgs only splits on spaces, so a tab +// before -out would hide the flag from the check. +func tokenizeEnvironmentArgs(value string) ([]string, error) { + return shlex.Split(value) +} + +// planEnvironment lists the environment layers a plan subprocess receives, in precedence order. +func planEnvironment(atmosConfig *schema.AtmosConfiguration, info *schema.ConfigAndStacksInfo, opts []component.ExecutionOption) []string { + entries := slices.Clone(planBaseEnvironment(info, opts)) + if atmosConfig != nil { + entries = envpkg.MergeGlobalEnv(entries, atmosConfig.Env) + } + entries = append(entries, info.ComponentEnvList...) + if info.ComponentExecution == nil { + return entries + } + for i := range info.ComponentExecution.ExtraArguments { + item := &info.ComponentExecution.ExtraArguments[i] + if !slices.Contains(item.Commands, subcommandPlan) { + continue + } + for name, value := range item.Environment { + entries = append(entries, name+"="+value) + } + } + return entries +} + +// containsOutputFlag reports any plan output flag, with or without a value. +func containsOutputFlag(args []string) bool { + return slices.ContainsFunc(args, func(arg string) bool { + return arg == "-out" || arg == "--out" || strings.HasPrefix(arg, "-out=") || strings.HasPrefix(arg, "--out=") + }) +} + +func plannedOutputPath(directory string, args []string) string { + for i, arg := range args { + path := "" + switch { + case (arg == "-out" || arg == "--out") && i+1 < len(args): + path = args[i+1] + case strings.HasPrefix(arg, "-out="): + path = strings.TrimPrefix(arg, "-out=") + case strings.HasPrefix(arg, "--out="): + path = strings.TrimPrefix(arg, "--out=") + } + if path != "" { + if filepath.IsAbs(path) { + return path + } + return filepath.Join(directory, path) + } + } + return "" +} diff --git a/pkg/component/terraform/mock_iam_probing_runtime_test.go b/pkg/component/terraform/mock_iam_probing_runtime_test.go new file mode 100644 index 00000000000..2db7a9019e5 --- /dev/null +++ b/pkg/component/terraform/mock_iam_probing_runtime_test.go @@ -0,0 +1,87 @@ +// Code generated by MockGen. DO NOT EDIT. +// Source: configuration_test.go +// +// Generated by this command: +// +// mockgen -source=configuration_test.go -destination=mock_iam_probing_runtime_test.go -package=terraform -mock_names=iamProbingRuntime=MockIAMProbingRuntime +// + +// Package terraform is a generated GoMock package. +package terraform + +import ( + context "context" + reflect "reflect" + + schema "github.com/cloudposse/atmos/pkg/schema" + gomock "go.uber.org/mock/gomock" +) + +// MockIAMProbingRuntime is a mock of iamProbingRuntime interface. +type MockIAMProbingRuntime struct { + ctrl *gomock.Controller + recorder *MockIAMProbingRuntimeMockRecorder + isgomock struct{} +} + +// MockIAMProbingRuntimeMockRecorder is the mock recorder for MockIAMProbingRuntime. +type MockIAMProbingRuntimeMockRecorder struct { + mock *MockIAMProbingRuntime +} + +// NewMockIAMProbingRuntime creates a new mock instance. +func NewMockIAMProbingRuntime(ctrl *gomock.Controller) *MockIAMProbingRuntime { + mock := &MockIAMProbingRuntime{ctrl: ctrl} + mock.recorder = &MockIAMProbingRuntimeMockRecorder{mock} + return mock +} + +// EXPECT returns an object that allows the caller to indicate expected use. +func (m *MockIAMProbingRuntime) EXPECT() *MockIAMProbingRuntimeMockRecorder { + return m.recorder +} + +// Discover mocks base method. +func (m *MockIAMProbingRuntime) Discover(arg0 context.Context) (map[string]map[string]any, error) { + m.ctrl.T.Helper() + ret := m.ctrl.Call(m, "Discover", arg0) + ret0, _ := ret[0].(map[string]map[string]any) + ret1, _ := ret[1].(error) + return ret0, ret1 +} + +// Discover indicates an expected call of Discover. +func (mr *MockIAMProbingRuntimeMockRecorder) Discover(arg0 any) *gomock.Call { + mr.mock.ctrl.T.Helper() + return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "Discover", reflect.TypeOf((*MockIAMProbingRuntime)(nil).Discover), arg0) +} + +// Prepare mocks base method. +func (m *MockIAMProbingRuntime) Prepare(arg0 context.Context, arg1 *schema.ConfigAndStacksInfo) (*schema.PreparedComponentConfiguration, error) { + m.ctrl.T.Helper() + ret := m.ctrl.Call(m, "Prepare", arg0, arg1) + ret0, _ := ret[0].(*schema.PreparedComponentConfiguration) + ret1, _ := ret[1].(error) + return ret0, ret1 +} + +// Prepare indicates an expected call of Prepare. +func (mr *MockIAMProbingRuntimeMockRecorder) Prepare(arg0, arg1 any) *gomock.Call { + mr.mock.ctrl.T.Helper() + return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "Prepare", reflect.TypeOf((*MockIAMProbingRuntime)(nil).Prepare), arg0, arg1) +} + +// ProbeIAM mocks base method. +func (m *MockIAMProbingRuntime) ProbeIAM(arg0 context.Context, arg1 *schema.ConfigAndStacksInfo) (*schema.ComponentIAM, error) { + m.ctrl.T.Helper() + ret := m.ctrl.Call(m, "ProbeIAM", arg0, arg1) + ret0, _ := ret[0].(*schema.ComponentIAM) + ret1, _ := ret[1].(error) + return ret0, ret1 +} + +// ProbeIAM indicates an expected call of ProbeIAM. +func (mr *MockIAMProbingRuntimeMockRecorder) ProbeIAM(arg0, arg1 any) *gomock.Call { + mr.mock.ctrl.T.Helper() + return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "ProbeIAM", reflect.TypeOf((*MockIAMProbingRuntime)(nil).ProbeIAM), arg0, arg1) +} diff --git a/pkg/component/terraform/verify_plan.go b/pkg/component/terraform/verify_plan.go index b6c66f9feb4..a9f149a2fbf 100644 --- a/pkg/component/terraform/verify_plan.go +++ b/pkg/component/terraform/verify_plan.go @@ -1,10 +1,16 @@ package terraform import ( + "crypto/sha256" + "encoding/hex" "encoding/json" + "errors" "fmt" + "io/fs" + "maps" "os" "path/filepath" + "slices" errUtils "github.com/cloudposse/atmos/errors" "github.com/cloudposse/atmos/pkg/component" @@ -166,3 +172,97 @@ func finalizeVerification(info *schema.ConfigAndStacksInfo, freshPlanFile, diff } return nil } + +// withoutSensitiveContent copies files, dropping the bytes of every sensitive one. +func withoutSensitiveContent(files []schema.ComponentConfigurationFile) []schema.ComponentConfigurationFile { + stored := slices.Clone(files) + for i := range stored { + if stored[i].Sensitive { + stored[i].Content = nil + } + } + return stored +} + +// restoreSensitiveContent reads the bytes of each sensitive file back from the working directory and +// restores them only when they are the reviewed ones. The working directory digest excludes some +// locations, so the reviewed bytes are verified here against the digest recorded with the plan. +func restoreSensitiveContent(workdir string, files []schema.ComponentConfigurationFile, reviewed string) error { + contents, err := readSensitiveFiles(workdir, files) + if err != nil { + return err + } + if contents == nil { + return nil + } + for i := range files { + // A file the entry removes is absent by design, and the digest still binds that absence. + if files[i].Sensitive && contents[files[i].Path] == nil && !slices.Contains([]string{"remove", "remove_terragrunt"}, files[i].IfExists) { + return fmt.Errorf("%w: reviewed generated file %s is missing: %w", errUtils.ErrPlanVerificationFailed, files[i].Path, fs.ErrNotExist) + } + } + if digestSensitiveContents(contents) != reviewed { + return fmt.Errorf("%w: reviewed generated file changed", errUtils.ErrPlanVerificationFailed) + } + for i := range files { + if files[i].Sensitive { + files[i].Content = contents[files[i].Path] + } + } + return nil +} + +// sensitiveDigest fingerprints the sensitive generated files as they are in the working directory. +// It is empty when no file is sensitive. +func sensitiveDigest(workdir string, files []schema.ComponentConfigurationFile) (string, error) { + contents, err := readSensitiveFiles(workdir, files) + if err != nil || contents == nil { + return "", err + } + return digestSensitiveContents(contents), nil +} + +// readSensitiveFiles reads every sensitive file below workdir, keyed by path. A missing file maps +// to nil content. It returns nil when no file is sensitive. +func readSensitiveFiles(workdir string, files []schema.ComponentConfigurationFile) (map[string][]byte, error) { + if !slices.ContainsFunc(files, func(file schema.ComponentConfigurationFile) bool { return file.Sensitive }) { + return nil, nil + } + root, err := os.OpenRoot(workdir) + if err != nil { + return nil, planIOError(err) + } + defer root.Close() + contents := make(map[string][]byte) + for i := range files { + if !files[i].Sensitive { + continue + } + content, err := root.ReadFile(files[i].Path) + if errors.Is(err, fs.ErrNotExist) { + contents[files[i].Path] = nil + continue + } + if err != nil { + return nil, fmt.Errorf("%w: reviewed generated file %s cannot be read: %w", errUtils.ErrPlanVerificationFailed, files[i].Path, err) + } + if content == nil { + content = []byte{} + } + contents[files[i].Path] = content + } + return contents, nil +} + +// digestSensitiveContents hashes the paths and bytes of the sensitive files in a stable order. +func digestSensitiveContents(contents map[string][]byte) string { + hash := sha256.New() + for _, path := range slices.Sorted(maps.Keys(contents)) { + if contents[path] == nil { + fmt.Fprintf(hash, "%s\x00absent\n", path) + continue + } + fmt.Fprintf(hash, "%s\x00%x\n", path, sha256.Sum256(contents[path])) + } + return hex.EncodeToString(hash.Sum(nil)) +} diff --git a/pkg/component/workdir_path.go b/pkg/component/workdir_path.go index 2164d6acf56..c92ba18f189 100644 --- a/pkg/component/workdir_path.go +++ b/pkg/component/workdir_path.go @@ -6,6 +6,7 @@ import ( "fmt" "os" "path/filepath" + "strings" errUtils "github.com/cloudposse/atmos/errors" cfg "github.com/cloudposse/atmos/pkg/config" @@ -85,6 +86,82 @@ func ResolveWorkdirSubpath(metadataSubpath, workdirRoot string) (string, error) return workdirRoot, nil } +// ResolveConfiguredSubpath selects the explicit module under a copied local +// source root. Unlike the legacy metadata.component hint, an explicit source +// subdirectory must exist and cannot silently fall back to the root. Use it for paths that are +// resolved after the source is provisioned; callers that run before it use ResolvePendingSubpath. +func ResolveConfiguredSubpath(info *schema.ConfigAndStacksInfo, workdirRoot string) (string, error) { + defer perf.Track(nil, "component.ResolveConfiguredSubpath")() + + selected, _, err := resolveConfiguredSubpath(info, workdirRoot, false) + return selected, err +} + +// ResolvePendingSubpath is ResolveConfiguredSubpath for callers that run before the source is +// provisioned, when an explicit source subdirectory may not exist yet. It returns the path the module +// will have and whether it exists now; every other failure, including an invalid subdirectory, is still +// an error. +func ResolvePendingSubpath(info *schema.ConfigAndStacksInfo, workdirRoot string) (string, bool, error) { + defer perf.Track(nil, "component.ResolvePendingSubpath")() + + return resolveConfiguredSubpath(info, workdirRoot, true) +} + +func resolveConfiguredSubpath(info *schema.ConfigAndStacksInfo, workdirRoot string, allowMissing bool) (string, bool, error) { + metadata, _ := info.ComponentSection["metadata"].(map[string]any) + subdir, _ := metadata["source_subdir"].(string) + if subdir == "" { + selected, err := ResolveWorkdirSubpath(info.BaseComponentPath, workdirRoot) + return selected, err == nil, err + } + if err := validateSourceSubdir(subdir); err != nil { + return "", false, err + } + selected := filepath.Join(workdirRoot, filepath.FromSlash(subdir)) + selectedInfo, err := os.Stat(selected) + if allowMissing && errors.Is(err, os.ErrNotExist) { + return selected, false, nil + } + if err != nil { + return "", false, fmt.Errorf("%w: inspect source subdirectory: %w", errUtils.ErrWorkdirProvision, err) + } + if !selectedInfo.IsDir() { + return "", false, fmt.Errorf("%w: source subdirectory is not a directory", errUtils.ErrWorkdirProvision) + } + if err := checkSourceSubdirBoundary(workdirRoot, selected); err != nil { + return "", false, err + } + return selected, true, nil +} + +func validateSourceSubdir(subdir string) error { + if filepath.IsAbs(subdir) || filepath.VolumeName(subdir) != "" || strings.HasPrefix(subdir, "/") || strings.HasPrefix(subdir, `\`) { + return fmt.Errorf("%w: invalid source subdirectory", errUtils.ErrWorkdirProvision) + } + for _, part := range strings.Split(filepath.ToSlash(subdir), "/") { + if part == "" || part == "." || part == ".." { + return fmt.Errorf("%w: invalid source subdirectory", errUtils.ErrWorkdirProvision) + } + } + return nil +} + +func checkSourceSubdirBoundary(workdirRoot, selected string) error { + root, err := filepath.EvalSymlinks(workdirRoot) + if err != nil { + return fmt.Errorf("%w: resolve source root: %w", errUtils.ErrWorkdirProvision, err) + } + module, err := filepath.EvalSymlinks(selected) + if err != nil { + return fmt.Errorf("%w: resolve source module: %w", errUtils.ErrWorkdirProvision, err) + } + rel, err := filepath.Rel(root, module) + if err != nil || rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) { + return fmt.Errorf("%w: source subdirectory escapes its root", errUtils.ErrWorkdirProvision) + } + return nil +} + // ApplyWorkdirSubpathToSection resolves provWorkdir.WorkdirPathKey in // info.ComponentSection against info.BaseComponentPath (joining the subpath // only when it exists on disk; see ResolveWorkdirSubpath) and returns the @@ -110,7 +187,7 @@ func ApplyWorkdirSubpathToSection(info *schema.ConfigAndStacksInfo) (string, err if _, applied := info.ComponentSection[workdirSubpathAppliedKey].(subpathAppliedMarker); applied { return workdirPath, nil } - resolved, err := ResolveWorkdirSubpath(info.BaseComponentPath, workdirPath) + resolved, err := ResolveConfiguredSubpath(info, workdirPath) if err != nil { return "", err } @@ -154,7 +231,7 @@ func BuildAndResolveWorkdirPath( } workdirRoot = target.Dir } - resolved, err := ResolveWorkdirSubpath(info.BaseComponentPath, workdirRoot) + resolved, err := ResolveConfiguredSubpath(info, workdirRoot) if err != nil { return "", false, err } @@ -306,6 +383,31 @@ func DryRunDefersMissingPath(info *schema.ConfigAndStacksInfo) bool { return info != nil && info.DryRun && provSource.HasSource(info.ComponentSection) } +// applyDryRunWorkdirSubpath resolves the workdir subpath of a dry run, which never provisions the +// source, so an explicit source subdirectory that does not exist yet resolves to the path it will have. +// The section is then left untouched, so a later resolution after provisioning starts from the root. +// +// Locating a retained workdir is not a dry run and keeps the strict resolution. +func applyDryRunWorkdirSubpath(info *schema.ConfigAndStacksInfo) (string, error) { + workdirPath, _ := info.ComponentSection[provWorkdir.WorkdirPathKey].(string) + if !info.DryRun || workdirPath == "" { + return ApplyWorkdirSubpathToSection(info) + } + if _, applied := info.ComponentSection[workdirSubpathAppliedKey].(subpathAppliedMarker); applied { + return workdirPath, nil + } + selected, present, err := ResolvePendingSubpath(info, workdirPath) + if err != nil { + return "", err + } + if !present { + return selected, nil + } + info.ComponentSection[provWorkdir.WorkdirPathKey] = selected + info.ComponentSection[workdirSubpathAppliedKey] = subpathAppliedMarker{} + return selected, nil +} + // resolveDryRunComponentPath validates placement without downloading source or copying a workdir. // The existence result describes the current filesystem, not a simulated successful provision. // A sourced component's directory follows source target rules, which honor an explicit @@ -323,9 +425,12 @@ func resolveDryRunComponentPath(atmosConfig *schema.AtmosConfiguration, info *sc return "", false, err } if isWorkdir { + // The section now holds the root again, so a subpath applied by an earlier resolution no longer + // describes it; clearing the marker lets this resolution select source_subdir once more. + delete(info.ComponentSection, workdirSubpathAppliedKey) info.ComponentSection[provWorkdir.WorkdirPathKey] = fallback } - path, err := ApplyWorkdirSubpathToSection(info) + path, err := applyDryRunWorkdirSubpath(info) if err != nil { return "", false, err } diff --git a/pkg/component/workdir_path_test.go b/pkg/component/workdir_path_test.go index 69a1a1902a6..0fd866041cd 100644 --- a/pkg/component/workdir_path_test.go +++ b/pkg/component/workdir_path_test.go @@ -646,6 +646,33 @@ func TestResolveRetainedWorkdirRecordsPathWithoutProvisioning(t *testing.T) { assert.NotContains(t, plain.ComponentSection, provWorkdir.WorkdirPathKey) } +// A retained workdir is located again for each phase of a command. Replacing the selected path with the +// workdir root also clears the applied marker, so the next resolution selects source_subdir again +// instead of returning the root. +func TestResolveRetainedWorkdirSelectsSourceSubdirOnEveryCall(t *testing.T) { + base := t.TempDir() + fallback := filepath.Join(base, "components", "vpc") + config := &schema.AtmosConfiguration{BasePath: base} + section := map[string]any{ + "component": "vpc", "atmos_component": "vpc", "atmos_stack": "dev", + "provision": map[string]any{"workdir": map[string]any{"enabled": true}}, + "source": map[string]any{"uri": "./module"}, + "metadata": map[string]any{"source_subdir": "modules/app"}, + } + info := &schema.ConfigAndStacksInfo{FinalComponent: "vpc", Stack: "dev", ComponentSection: section} + root, err := provWorkdir.BuildPath(base, cfg.TerraformComponentType, "vpc", "dev", section) + require.NoError(t, err) + module := filepath.Join(root, "modules", "app") + require.NoError(t, os.MkdirAll(module, 0o755)) + + for call := 1; call <= 2; call++ { + path, err := ResolveRetainedWorkdir(config, info, cfg.TerraformComponentType, fallback) + require.NoError(t, err) + assert.Equal(t, module, path, "call %d selects the source subdirectory", call) + assert.Equal(t, module, section[provWorkdir.WorkdirPathKey], "call %d records the selected path", call) + } +} + // TestProvisionAndResolveComponentPath_NoSourceReturnsFallback verifies the // short-circuit when no source is configured: the orchestrator must skip // AutoProvisionSource entirely and report whether the fallback exists. diff --git a/pkg/datafetcher/schema/atmos/manifest/1.0.json b/pkg/datafetcher/schema/atmos/manifest/1.0.json index 387afae680f..a2fef9fb714 100644 --- a/pkg/datafetcher/schema/atmos/manifest/1.0.json +++ b/pkg/datafetcher/schema/atmos/manifest/1.0.json @@ -2714,6 +2714,10 @@ } ] }, + "source_subdir": { + "type": "string", + "description": "Module subdirectory within the copied source root. Sibling modules remain available in the workdir." + }, "terraform_workspace": { "type": "string", "description": "Terraform workspace" diff --git a/pkg/datafetcher/schema/stacks/stack-config/1.0.json b/pkg/datafetcher/schema/stacks/stack-config/1.0.json index ce7469c0359..f0cbda138ee 100644 --- a/pkg/datafetcher/schema/stacks/stack-config/1.0.json +++ b/pkg/datafetcher/schema/stacks/stack-config/1.0.json @@ -1887,6 +1887,10 @@ } ] }, + "source_subdir": { + "type": "string", + "description": "Module subdirectory within the copied source root. Sibling modules remain available in the workdir." + }, "terraform_workspace": { "type": "string", "description": "Terraform workspace" diff --git a/pkg/datafetcher/schema_global_metadata_test.go b/pkg/datafetcher/schema_global_metadata_test.go index 8f22b960710..9306a80181f 100644 --- a/pkg/datafetcher/schema_global_metadata_test.go +++ b/pkg/datafetcher/schema_global_metadata_test.go @@ -75,3 +75,36 @@ func TestGlobalMetadataSchema_ValidatesAllowlist(t *testing.T) { }) } } + +// TestSourceSubdirMetadataSchema keeps discovered local modules valid without +// allowing component-only source metadata at global scope in the manifest schema. +// The legacy stack-config schema permits additional root properties. +func TestSourceSubdirMetadataSchema(t *testing.T) { + for _, source := range []string{embeddedSchemaSource, "atmos://schema/stacks/stack-config/1.0"} { + t.Run(source, func(t *testing.T) { + data, err := (&atmosFetcher{}).FetchData(source) + require.NoError(t, err) + tests := []struct { + name string + value any + global bool + valid bool + }{ + {name: "module subdirectory", value: "modules/app", valid: true}, + {name: "wrong type", value: 42}, + {name: "global metadata", value: "modules/app", global: true, valid: source != embeddedSchemaSource}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + doc := map[string]any{"metadata": map[string]any{"source_subdir": tc.value}} + if !tc.global { + doc = map[string]any{"components": map[string]any{"terraform": map[string]any{"app": doc}}} + } + result, err := gojsonschema.Validate(gojsonschema.NewBytesLoader(data), gojsonschema.NewGoLoader(doc)) + require.NoError(t, err) + require.Equal(t, tc.valid, result.Valid(), "%v", result.Errors()) + }) + } + }) + } +} diff --git a/pkg/datafetcher/schema_workflow_coverage_test.go b/pkg/datafetcher/schema_workflow_coverage_test.go index 1aa2bb65d10..68c3ce26b9c 100644 --- a/pkg/datafetcher/schema_workflow_coverage_test.go +++ b/pkg/datafetcher/schema_workflow_coverage_test.go @@ -85,7 +85,7 @@ func workflowDefinitionProps(t *testing.T, defName string) map[string]struct{} { // authoredYAMLKeys returns the set of YAML keys a struct authors, read from `yaml` struct tags // (skipping `-` and empty tags), plus any extra keys decoded by a custom UnmarshalYAML. func authoredYAMLKeys(rt reflect.Type, extra ...string) map[string]struct{} { - if rt.Kind() == reflect.Ptr { + if rt.Kind() == reflect.Pointer { rt = rt.Elem() } out := make(map[string]struct{}, rt.NumField()) diff --git a/pkg/downloader/gogetter_downloader.go b/pkg/downloader/gogetter_downloader.go index d6c7e102e4b..c8399b85367 100644 --- a/pkg/downloader/gogetter_downloader.go +++ b/pkg/downloader/gogetter_downloader.go @@ -74,20 +74,7 @@ func (f *goGetterClientFactory) NewClient(ctx context.Context, src, dest string, return nil, err } - // Lazily provision ambient credential brokers (e.g., Atmos Pro github/sts) before the first - // remote read so private repositories are reachable. Brokers export GIT_CONFIG_*/tokens into - // the process env, which the git subprocess this client spawns inherits. Process-once and - // gated (CI + configured), so local-only fetches and non-Pro repos pay nothing. - remote := f.atmosConfig != nil && isRemoteSource(src) - if remote { - broker.EnsureCredentials(ctx, f.atmosConfig) - } - - // When Atmos brokered a fresh token this process, a freshly minted GitHub token can - // briefly 401 before GitHub propagates it. Let the git getter retry such auth failures - // within a bounded window — only for remote, brokered reads, so static credentials still - // fail fast. - retryAuthErrors := remote && broker.HasBrokeredCredentials() + retryAuthErrors := brokeredAuthRetry(ctx, f.atmosConfig, src) // Create HTTP getter with optional custom client, always wrapped in a metadataCapturingTransport // so a caller can retrieve ETag/Last-Modified after Get() runs (see goGetterClient.Metadata). @@ -130,6 +117,16 @@ func (f *goGetterClientFactory) NewClient(ctx context.Context, src, dest string, return &goGetterClient{client: client, metadata: capture}, nil } +// brokeredAuthRetry provisions ambient credential brokers only for remote sources. +// Fresh brokered GitHub tokens can briefly 401 before propagation. +func brokeredAuthRetry(ctx context.Context, config *schema.AtmosConfiguration, src string) bool { + if config == nil || !isRemoteSource(src) { + return false + } + broker.EnsureCredentials(ctx, config) + return broker.HasBrokeredCredentials() +} + // customDetectors creates a private detector list for each client. Global detectors are never mutated. func customDetectors(ctx context.Context, atmosConfig *schema.AtmosConfiguration, src string) []getter.Detector { return append([]getter.Detector{&CustomGitDetector{ctx: ctx, atmosConfig: atmosConfig, source: src}}, getter.Detectors...) @@ -295,5 +292,6 @@ func (f *goGetterClientFactory) metadataHTTPGetter(ctx context.Context) (*getter } } + httpGetter.Client = authorizedHTTPClient(ctx, httpGetter.Client) return httpGetter, capture } diff --git a/pkg/downloader/s3/getter.go b/pkg/downloader/s3/getter.go index 05114b6a190..a77d7886472 100644 --- a/pkg/downloader/s3/getter.go +++ b/pkg/downloader/s3/getter.go @@ -13,6 +13,7 @@ import ( "sync" "github.com/aws/aws-sdk-go-v2/aws" + "github.com/aws/aws-sdk-go-v2/credentials" "github.com/aws/aws-sdk-go-v2/service/s3" "github.com/aws/smithy-go" "github.com/hashicorp/go-getter" @@ -32,6 +33,8 @@ type sourceAWSAuthKey struct{} type sourceAWSAuthResolverKey struct{} +type sourceAWSSessionKey struct{} + // WithAuthResolver defers authentication until an S3 source actually needs downloading. func WithAuthResolver(ctx context.Context, resolver func(context.Context) (*schema.AWSAuthContext, error)) context.Context { defer perf.Track(nil, "s3.WithAuthResolver")() @@ -48,6 +51,23 @@ func WithAuthContext(ctx context.Context, auth *schema.AWSAuthContext) context.C return context.WithValue(ctx, sourceAWSAuthKey{}, &snapshot) } +// WithSession scopes an assumed-role session (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and +// AWS_SESSION_TOKEN) to source downloads. It replaces the credentials of the auth context, which +// only still selects the region and endpoint, because the role was assumed on top of them. +func WithSession(ctx context.Context, session map[string]string) context.Context { + defer perf.Track(nil, "s3.WithSession")() + if session["AWS_ACCESS_KEY_ID"] == "" { + return ctx + } + value := aws.Credentials{ + AccessKeyID: session["AWS_ACCESS_KEY_ID"], + SecretAccessKey: session["AWS_SECRET_ACCESS_KEY"], + SessionToken: session["AWS_SESSION_TOKEN"], + Source: "AtmosComponentIAMRole", + } + return context.WithValue(ctx, sourceAWSSessionKey{}, value) +} + type sourceClient interface { HeadObject(context.Context, *s3.HeadObjectInput, ...func(*s3.Options)) (*s3.HeadObjectOutput, error) GetObject(context.Context, *s3.GetObjectInput, ...func(*s3.Options)) (*s3.GetObjectOutput, error) @@ -83,6 +103,9 @@ func (g *Getter) buildClient(ctx context.Context, loc *sourceLocation) (sourceCl if err != nil { return nil, err } + if session, ok := ctx.Value(sourceAWSSessionKey{}).(aws.Credentials); ok { + config.Credentials = credentials.StaticCredentialsProvider{Value: session} + } endpoint := loc.endpoint if g.auth != nil && g.auth.EndpointURL != "" { endpoint = g.auth.EndpointURL diff --git a/pkg/downloader/s3/getter_test.go b/pkg/downloader/s3/getter_test.go index 8f73af70741..0268ceb4118 100644 --- a/pkg/downloader/s3/getter_test.go +++ b/pkg/downloader/s3/getter_test.go @@ -393,3 +393,46 @@ func TestS3DownloadRemovesPartialFile(t *testing.T) { assert.NoFileExists(t, dst) assert.Equal(t, 1, client.gets) } + +// TestS3SourceUsesAssumedRoleSession checks that an assumed-role session replaces the auth context's +// credentials for a source download, and that a context without one keeps the auth context's credentials. +func TestS3SourceUsesAssumedRoleSession(t *testing.T) { + root := t.TempDir() + var mu sync.Mutex + var seen []string + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + mu.Lock() + seen = append(seen, r.Header.Get("Authorization")) + mu.Unlock() + _, _ = fmt.Fprint(w, "Resources: {}\n") + })) + defer server.Close() + credentialFile := filepath.Join(root, "base.credentials") + require.NoError(t, os.WriteFile(credentialFile, []byte("[source]\naws_access_key_id = BASEKEY\naws_secret_access_key = synthetic\n"), 0o600)) + auth := &schema.AWSAuthContext{Profile: "source", CredentialsFile: credentialFile, ConfigFile: filepath.Join(root, "absent-config"), Region: "us-east-2"} + download := func(ctx context.Context, name string) { + t.Helper() + ctx, cancel := context.WithTimeout(ctx, 10*time.Second) + defer cancel() + client := &getter.Client{Ctx: ctx, Src: "s3::" + server.URL + "/bucket/template.yaml?region=us-east-2", Dst: filepath.Join(root, name), Mode: getter.ClientModeAny, Getters: map[string]getter.Getter{"s3": NewGetter(ctx)}} + require.NoError(t, client.Get()) + } + + base := WithAuthContext(context.Background(), auth) + download(WithSession(base, map[string]string{"AWS_ACCESS_KEY_ID": "ASIAROLE", "AWS_SECRET_ACCESS_KEY": "role-secret", "AWS_SESSION_TOKEN": "role-token"}), "role") + mu.Lock() + require.NotEmpty(t, seen) + for _, header := range seen { + assert.Contains(t, header, "Credential=ASIAROLE/") + } + seen = nil + mu.Unlock() + + download(WithSession(base, map[string]string{"AWS_PROFILE": ""}), "base") + mu.Lock() + defer mu.Unlock() + require.NotEmpty(t, seen) + for _, header := range seen { + assert.Contains(t, header, "Credential=BASEKEY/") + } +} diff --git a/pkg/downloader/s3_getter.go b/pkg/downloader/s3_getter.go index 0206bc3027d..05140f80324 100644 --- a/pkg/downloader/s3_getter.go +++ b/pkg/downloader/s3_getter.go @@ -19,3 +19,10 @@ func WithAWSAuthContext(ctx context.Context, auth *schema.AWSAuthContext) contex defer perf.Track(nil, "downloader.WithAWSAuthContext")() return s3source.WithAuthContext(ctx, auth) } + +// WithAWSSession scopes a component's assumed AWS session to its source downloads. The session takes +// precedence over an Atmos auth context, because the role is assumed on top of it. +func WithAWSSession(ctx context.Context, session map[string]string) context.Context { + defer perf.Track(nil, "downloader.WithAWSSession")() + return s3source.WithSession(ctx, session) +} diff --git a/pkg/downloader/source_authorization.go b/pkg/downloader/source_authorization.go new file mode 100644 index 00000000000..93d1a3f0f5e --- /dev/null +++ b/pkg/downloader/source_authorization.go @@ -0,0 +1,53 @@ +package downloader + +import ( + "context" + "net/http" + "strings" + + "github.com/cloudposse/atmos/pkg/schema" +) + +type sourceAuthorizationKey struct{} + +// WithSourceAuthorization binds a registry archive header to one invocation. +func WithSourceAuthorization(ctx context.Context, auth *schema.SourceAuthorization) context.Context { + if auth == nil { + return ctx + } + copy := *auth + return context.WithValue(ctx, sourceAuthorizationKey{}, ©) +} + +func authorizedHTTPClient(ctx context.Context, client *http.Client) *http.Client { + auth, ok := ctx.Value(sourceAuthorizationKey{}).(*schema.SourceAuthorization) + if !ok || auth == nil { + return client + } + cloned := *client + cloned.Transport = &sourceAuthorizationTransport{base: cloned.Transport, origin: auth.Origin, header: auth.Header} + return &cloned +} + +type sourceAuthorizationTransport struct { + base http.RoundTripper + origin string + header string +} + +func (t *sourceAuthorizationTransport) RoundTrip(request *http.Request) (*http.Response, error) { + copy := request.Clone(request.Context()) + origin := copy.URL.Scheme + "://" + copy.URL.Host + if strings.EqualFold(origin, t.origin) { + copy.Header.Set("Authorization", t.header) + } else if copy.Header.Get("Authorization") == t.header { + // http.Client may copy Authorization onto a redirect request. Never let + // the registry header cross the exact origin boundary. + copy.Header.Del("Authorization") + } + base := t.base + if base == nil { + base = http.DefaultTransport + } + return base.RoundTrip(copy) +} diff --git a/pkg/downloader/source_authorization_test.go b/pkg/downloader/source_authorization_test.go new file mode 100644 index 00000000000..8d58cc724be --- /dev/null +++ b/pkg/downloader/source_authorization_test.go @@ -0,0 +1,37 @@ +package downloader + +import ( + "context" + "io" + "net/http" + "net/http/httptest" + "testing" + + "github.com/stretchr/testify/require" + + "github.com/cloudposse/atmos/pkg/schema" +) + +func TestSourceAuthorizationStaysOnTheExactOrigin(t *testing.T) { + var externalAuthorization string + external := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + externalAuthorization = r.Header.Get("Authorization") + w.WriteHeader(http.StatusOK) + })) + t.Cleanup(external.Close) + var firstAuthorization string + registry := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + firstAuthorization = r.Header.Get("Authorization") + http.Redirect(w, r, external.URL+"/archive", http.StatusFound) + })) + t.Cleanup(registry.Close) + ctx := WithSourceAuthorization(context.Background(), &schema.SourceAuthorization{Origin: registry.URL, Header: "Bearer registry-secret"}) + client := authorizedHTTPClient(ctx, &http.Client{}) + response, err := client.Get(registry.URL + "/archive") + require.NoError(t, err) + defer response.Body.Close() + _, err = io.Copy(io.Discard, response.Body) + require.NoError(t, err) + require.Equal(t, "Bearer registry-secret", firstAuthorization) + require.Empty(t, externalAuthorization) +} diff --git a/pkg/filesystem/symlinks.go b/pkg/filesystem/symlinks.go new file mode 100644 index 00000000000..77a81351a16 --- /dev/null +++ b/pkg/filesystem/symlinks.go @@ -0,0 +1,32 @@ +package filesystem + +import ( + "path/filepath" + + "github.com/cloudposse/atmos/pkg/perf" +) + +// EvalSymlinksBestEffort resolves symlinks in path even when the full path +// does not exist, by resolving the deepest existing ancestor and re-appending +// the non-existent remainder. The plain filepath.EvalSymlinks errors on +// missing paths, and changed or deleted files routinely do not exist. +// A path with no resolvable ancestor is returned unchanged. +func EvalSymlinksBestEffort(path string) string { + defer perf.Track(nil, "filesystem.EvalSymlinksBestEffort")() + + remainder := "" + current := path + for { + resolved, err := filepath.EvalSymlinks(current) + if err == nil { + return filepath.Join(resolved, remainder) + } + parent := filepath.Dir(current) + if parent == current { + // Nothing along the path resolves — return the input unchanged. + return path + } + remainder = filepath.Join(filepath.Base(current), remainder) + current = parent + } +} diff --git a/pkg/git/worktree_signal_test.go b/pkg/git/worktree_signal_test.go index 63d51dc54be..a6ba704f70b 100644 --- a/pkg/git/worktree_signal_test.go +++ b/pkg/git/worktree_signal_test.go @@ -1,13 +1,181 @@ package git import ( + "os" + "sync" + "sync/atomic" "syscall" "testing" + "time" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" ) +func TestRemoveOnSignalWaitsForAllWorktrees(t *testing.T) { + for _, received := range []os.Signal{syscall.SIGINT, syscall.SIGTERM} { + t.Run(received.String(), func(t *testing.T) { + exited := make(chan int, 1) + cleanups := NewWorktreeSignalCleanups(func(code int) { exited <- code }) + started := make(chan struct{}) + release := make(chan struct{}) + unblock := sync.OnceFunc(func() { close(release) }) + var cleaned atomic.Int32 + stopFirst := cleanups.Register(func() { cleaned.Add(1) }) + stopSecond := cleanups.Register(func() { + close(started) + <-release + cleaned.Add(1) + }) + // Unblock before stopping even when an assertion fails. + defer stopFirst() + defer stopSecond() + defer unblock() + cleanups.mu.Lock() + signals := cleanups.watcher.signals + cleanups.mu.Unlock() + signals <- received + select { + case <-started: + case <-time.After(10 * time.Second): + t.Fatal("cleanup did not start") + } + select { + case code := <-exited: + t.Fatalf("exited with %d while another worktree cleanup was blocked", code) + case <-time.After(20 * time.Millisecond): + } + unblock() + select { + case code := <-exited: + require.Equal(t, 128+int(received.(syscall.Signal)), code) + require.Equal(t, int32(2), cleaned.Load()) + case <-time.After(10 * time.Second): + t.Fatal("cleanup did not finish") + } + }) + } +} + +func TestRemoveOnSignalWaitsForNormalCleanup(t *testing.T) { + exited := make(chan int, 1) + cleanups := NewWorktreeSignalCleanups(func(code int) { exited <- code }) + started := make(chan struct{}) + release := make(chan struct{}) + finished := make(chan struct{}) + unblock := sync.OnceFunc(func() { close(release) }) + var cleaned atomic.Int32 + cleanup := sync.OnceFunc(func() { + close(started) + <-release + cleaned.Add(1) + }) + stop := cleanups.Register(cleanup) + defer stop() + defer unblock() + go func() { + cleanup() + stop() + close(finished) + }() + <-started + cleanups.mu.Lock() + signals := cleanups.watcher.signals + cleanups.mu.Unlock() + signals <- syscall.SIGTERM + select { + case code := <-exited: + t.Fatalf("exited with %d before normal cleanup finished", code) + case <-time.After(20 * time.Millisecond): + } + unblock() + select { + case <-finished: + case <-time.After(10 * time.Second): + t.Fatal("normal cleanup did not finish") + } + require.Equal(t, int32(1), cleaned.Load()) + // Normal release may retire the watcher before it handles the queued signal; the received + // signal must still end the process with its status. + select { + case code := <-exited: + require.Equal(t, 128+int(syscall.SIGTERM), code) + case <-time.After(10 * time.Second): + t.Fatal("a received SIGTERM must terminate the process after the final release") + } +} + +// A signal that races the final release still terminates the process, whichever wins the lock, +// and the cleanup never runs twice. +func TestRemoveOnSignalAfterFinalReleaseStillExits(t *testing.T) { + exited := make(chan int, 1) + cleanups := NewWorktreeSignalCleanups(func(code int) { exited <- code }) + var cleaned atomic.Int32 + stop := cleanups.Register(func() { cleaned.Add(1) }) + cleanups.mu.Lock() + watcher := cleanups.watcher + // Queue the signal while holding the lock, so the watcher and the release below race for it. + watcher.signals <- syscall.SIGTERM + cleanups.mu.Unlock() + stop() + select { + case code := <-exited: + require.Equal(t, 128+int(syscall.SIGTERM), code) + case <-time.After(10 * time.Second): + t.Fatal("a signal received before the final release must still terminate the process") + } + require.LessOrEqual(t, cleaned.Load(), int32(1), "a cleanup runs at most once") +} + +func TestRemoveOnSignalReleaseKeepsOtherWorktreesRegistered(t *testing.T) { + cleanups := NewWorktreeSignalCleanups(func(code int) { t.Errorf("unexpected exit %d", code) }) + var cleaned atomic.Int32 + stopFirst := cleanups.Register(func() { cleaned.Add(1) }) + stopSecond := cleanups.Register(func() { cleaned.Add(1) }) + stopFirst() + stopFirst() // Repeated release must not close a watcher twice. + cleanups.mu.Lock() + remaining := len(cleanups.watcher.cleanups) + cleanups.mu.Unlock() + require.Equal(t, 1, remaining) + stopSecond() + cleanups.mu.Lock() + watcher := cleanups.watcher + cleanups.mu.Unlock() + require.Nil(t, watcher) + require.Zero(t, cleaned.Load()) +} + +func TestRemoveOnSignalCleansUpBeforeExit(t *testing.T) { + exited := make(chan int, 1) + cleanups := NewWorktreeSignalCleanups(func(code int) { exited <- code }) + + var cleaned atomic.Int32 + stop := cleanups.Register(func() { cleaned.Add(1) }) + defer stop() + process, err := os.FindProcess(os.Getpid()) + require.NoError(t, err) + if err := process.Signal(syscall.SIGTERM); err != nil { + t.Skipf("signals cannot be delivered on this platform: %v", err) + } + select { + case code := <-exited: + require.Equal(t, 128+int(syscall.SIGTERM), code) + require.Equal(t, int32(1), cleaned.Load(), "the worktree is removed before the process exits") + case <-time.After(10 * time.Second): + t.Fatal("signal was not handled") + } +} + +func TestRemoveOnSignalStopReleasesWatcher(t *testing.T) { + cleanups := NewWorktreeSignalCleanups(func(code int) { t.Errorf("unexpected exit %d", code) }) + var cleaned atomic.Int32 + cleanups.Register(func() { cleaned.Add(1) })() + // After a normal return the watcher is gone: it must never run cleanup on its own. + time.Sleep(20 * time.Millisecond) + require.Equal(t, int32(0), cleaned.Load()) +} + // A signal queued for a watcher that its final release retired still cleans up the registrations // that another caller made afterwards, instead of exiting and leaking their worktrees. func TestRetiredWatcherSignalCleansActiveWorktrees(t *testing.T) { diff --git a/pkg/perf/perf.go b/pkg/perf/perf.go index 17e8ecc8880..0663605d39b 100644 --- a/pkg/perf/perf.go +++ b/pkg/perf/perf.go @@ -158,6 +158,34 @@ func Track(atmosConfig *schema.AtmosConfiguration, name string) func() { return trackWithGoroutineLocalStack(name, start) } +// ObserveExternal records a child duration measured in another process. It +// counts as child time of the active host span so the heatmap does not count +// plugin work twice. +func ObserveExternal(name string, duration time.Duration) { + if !isTrackingEnabled() || duration <= 0 { + return + } + if parent := activeFrame(); parent != nil { + parent.childTime.Add(int64(duration)) + } + recordMetrics(name, duration, duration) +} + +// activeFrame returns the innermost tracked span of the calling goroutine, or nil when it has none. +func activeFrame() *StackFrame { + gid := getGoroutineID() + if gid == 0 { + return nil + } + if useSimpleStack.Load() && simpleOwnerGID.Load() == gid { + return simpleStack.peek() + } + if value, ok := callStacks.Load(gid); ok { + return value.(*CallStack).peek() + } + return nil +} + // trackWithSimpleStack uses the global simple stack for single-goroutine scenarios. // If a different goroutine is detected, it automatically falls back to goroutine-local tracking // and disables simple mode globally to avoid future expensive checks. diff --git a/pkg/perf/perf_test.go b/pkg/perf/perf_test.go index 73c17fd43b4..0d20bddafb1 100644 --- a/pkg/perf/perf_test.go +++ b/pkg/perf/perf_test.go @@ -6,6 +6,31 @@ import ( "time" ) +func TestObserveExternalCountsChildTime(t *testing.T) { + EnableTracking(true) + UseSimpleTracking(true) + ResetForTesting() + t.Cleanup(func() { + EnableTracking(false) + ResetForTesting() + }) + finish := Track(nil, "host-parent") + time.Sleep(2 * time.Millisecond) + ObserveExternal("plugin-registry", time.Millisecond) + time.Sleep(2 * time.Millisecond) + finish() + reg.mu.Lock() + parent := *reg.data["host-parent"] + external := *reg.data["plugin-registry"] + reg.mu.Unlock() + if external.Count != 1 || external.SelfTime != time.Millisecond { + t.Fatalf("external duration was not recorded: %+v", external) + } + if parent.Total-parent.SelfTime < time.Millisecond { + t.Fatalf("external duration was counted in the parent self time: %+v", parent) + } +} + func TestTrack(t *testing.T) { // Reset registry for clean test. reg = ®istry{ diff --git a/pkg/plugin/adapter/component/calls.go b/pkg/plugin/adapter/component/calls.go index 6de1487c165..e6dc63f0f73 100644 --- a/pkg/plugin/adapter/component/calls.go +++ b/pkg/plugin/adapter/component/calls.go @@ -2,7 +2,9 @@ package component import ( "context" + "math" "path/filepath" + "time" "google.golang.org/protobuf/proto" @@ -91,6 +93,10 @@ func (p *Provider) Prepare(ctx context.Context, input *pluginv1.ComponentRequest if err != nil { return nil, err } + // The registry time is observed while this call's span is open, so it is a child of that span. + if nanos := result.GetRegistryResolutionNanos(); nanos > 0 && nanos <= uint64(math.MaxInt64) { + perf.ObserveExternal("atmosbridge.ResolveRegistrySource", time.Duration(nanos)) + } return result, nil } diff --git a/pkg/plugin/adapter/component/calls_test.go b/pkg/plugin/adapter/component/calls_test.go index f68ea0e1c61..d368409e989 100644 --- a/pkg/plugin/adapter/component/calls_test.go +++ b/pkg/plugin/adapter/component/calls_test.go @@ -20,7 +20,9 @@ import ( "google.golang.org/protobuf/proto" errUtils "github.com/cloudposse/atmos/errors" + "github.com/cloudposse/atmos/pkg/perf" "github.com/cloudposse/atmos/pkg/plugin/extension" + "github.com/cloudposse/atmos/pkg/plugin/manifest" pluginv1 "github.com/cloudposse/atmos/pkg/plugin/proto/atmos/plugin/v1" ) @@ -33,7 +35,8 @@ func fixtureActivations(t *testing.T) []*extension.Activation { expected := &pluginv1.DescribeResponse{PluginName: "fixture/components", PluginVersion: "0.1.0", ManifestDigest: "sha256:fixture"} var result []*extension.Activation for _, capability := range []string{extension.ComponentDiscovery, extension.ComponentValidation, extension.ComponentPreparation, extension.ComponentState} { - expected.Services = append(expected.Services, &pluginv1.ServiceVersionRange{Service: capability, Min: 1, Max: 1}) + version := manifest.HostContractVersion(capability) + expected.Services = append(expected.Services, &pluginv1.ServiceVersionRange{Service: capability, Min: version, Max: version}) expected.Exports = append(expected.Exports, &pluginv1.Export{Service: capability, Kind: "fixture/hcl"}) result = append(result, &extension.Activation{Binding: extension.Binding{Capability: capability, Name: "fixture/hcl", Directory: directory, Metadata: map[string]any{"component_type": "terraform", "schema": "schema.json"}}, Package: expected.PluginName, Executable: executable, Description: expected}) } @@ -356,6 +359,24 @@ func TestRegistryRejectsPartialAndConflictingBatches(t *testing.T) { require.Empty(t, empty.Names()) } +// TestRegistryRejectsEarlierPreparationContract keeps a plugin built for the +// first preparation contract from losing execution metadata on this host. +func TestRegistryRejectsEarlierPreparationContract(t *testing.T) { + t.Parallel() + require.Equal(t, uint32(2), manifest.HostContractVersion(extension.ComponentPreparation)) + for _, versions := range [][2]uint32{{1, 1}, {3, 3}} { + items := fixtureActivations(t) + for _, service := range items[0].Description.Services { + if service.Service == extension.ComponentPreparation { + service.Min, service.Max = versions[0], versions[1] + } + } + registry, err := NewRegistry(items) + require.Error(t, err, "preparation versions %v must be rejected", versions) + require.Nil(t, registry) + } +} + func TestUnsupportedAndInvalidCallsDoNotStartExecutable(t *testing.T) { t.Parallel() items := fixtureActivations(t)[:1] @@ -426,3 +447,35 @@ func TestNonSchemaSettingsErrorStillNamesTheSection(t *testing.T) { require.ErrorIs(t, err, errUtils.ErrPluginInvalidConfig) assert.Contains(t, strings.Join(errUtils.AllDetails(err), "\n"), "components.terraform.terragrunt") } + +func TestRegistryResolutionDurationIsAttributedInsideTheProviderSpan(t *testing.T) { + perf.EnableTracking(true) + perf.UseSimpleTracking(true) + perf.ResetForTesting() + t.Cleanup(func() { + perf.EnableTracking(false) + perf.ResetForTesting() + }) + p := fixtureProvider(t) + request := &pluginv1.ComponentRequest{ + RootDirectory: t.TempDir(), Address: fixtureAddress(), + Invocation: &pluginv1.Invocation{}, + Operation: &pluginv1.ComponentOperation{Command: "registry-prepare", Mode: "native"}, + Configuration: wireValue(t, cty.EmptyObjectVal), + } + + finish := perf.Track(nil, "host-parent") + time.Sleep(5 * time.Millisecond) + _, err := p.Prepare(t.Context(), request) + finish() + require.NoError(t, err) + + self := map[string]time.Duration{} + for _, row := range perf.SnapshotTop("total", 100).Rows { + self[row.Name] = row.SelfTime + } + require.Contains(t, self, "atmosbridge.ResolveRegistrySource", "the registry duration is recorded by the provider call") + require.Equal(t, time.Hour, self["atmosbridge.ResolveRegistrySource"]) + require.Zero(t, self["componentadapter.Prepare"], "the registry duration is a child of the provider span, so the span's own time excludes it") + require.GreaterOrEqual(t, self["host-parent"], 5*time.Millisecond, "the registry duration is not counted again in the enclosing span") +} diff --git a/pkg/plugin/adapter/component/main_test.go b/pkg/plugin/adapter/component/main_test.go index b75fe05ede2..1ec20d8f1d7 100644 --- a/pkg/plugin/adapter/component/main_test.go +++ b/pkg/plugin/adapter/component/main_test.go @@ -5,6 +5,7 @@ import ( "os" "path/filepath" "testing" + "time" "github.com/hashicorp/go-plugin" "google.golang.org/grpc" @@ -124,7 +125,11 @@ func (*componentFixture) Prepare(ctx context.Context, request *pluginv1.Componen <-ctx.Done() return nil, status.FromContextError(ctx.Err()).Err() } - return &pluginv1.PrepareComponentResponse{Engine: "terraform", WorkingDirectory: request.RootDirectory, ModuleSource: "./module", Inputs: request.Configuration, State: fixtureState(request.RootDirectory), Files: []*pluginv1.ComponentGeneratedFile{{Path: "backend.tf", Content: []byte("$${literal}\n%{ literal }\n"), IfExists: "error"}}}, nil + response := &pluginv1.PrepareComponentResponse{Engine: "terraform", WorkingDirectory: request.RootDirectory, ModuleSource: "./module", Inputs: request.Configuration, State: fixtureState(request.RootDirectory), Files: []*pluginv1.ComponentGeneratedFile{{Path: "backend.tf", Content: []byte("$${literal}\n%{ literal }\n"), IfExists: "error"}}} + if request.GetOperation().GetCommand() == "registry-prepare" { + response.RegistryResolutionNanos = uint64(time.Hour) + } + return response, nil } func (*componentFixture) Resolve(_ context.Context, request *pluginv1.ComponentRequest) (*pluginv1.ResolveComponentStateResponse, error) { diff --git a/pkg/plugin/adapter/component/objects.go b/pkg/plugin/adapter/component/objects.go index 971f4737491..01c83c4a3fd 100644 --- a/pkg/plugin/adapter/component/objects.go +++ b/pkg/plugin/adapter/component/objects.go @@ -75,6 +75,51 @@ func DecodeObject(input *pluginv1.ComponentValue) (map[string]any, error) { return object, nil } +// DecodeValue adapts one fully evaluated value of any type, such as an entry of a value map. +// A null value decodes to nil. +func DecodeValue(input *pluginv1.ComponentValue) (any, error) { + defer perf.Track(nil, "componentadapter.DecodeValue")() + + value, err := decodePlain(input) + if err != nil { + return nil, err + } + if !value.IsWhollyKnown() { + return nil, errUtils.ErrPluginUnsupportedExtension + } + if value.IsNull() { + return nil, nil + } + if err := registerSensitivePaths(value, input.SensitivePaths); err != nil { + return nil, err + } + return objectJSON(value) +} + +// DecodeString reads a fully evaluated string value. Sensitivity paths are registered with masked I/O +// before the text escapes, so a secret carried this way never reaches output unmasked. +func DecodeString(input *pluginv1.ComponentValue) (string, error) { + defer perf.Track(nil, "componentadapter.DecodeString")() + + if input == nil { + return "", nil + } + value, err := decodePlain(input) + if err != nil { + return "", err + } + if !value.IsWhollyKnown() { + return "", errUtils.ErrPluginUnsupportedExtension + } + if value.IsNull() || value.Type() != cty.String { + return "", errUtils.ErrPluginProtocol + } + if err := registerSensitivePaths(value, input.SensitivePaths); err != nil { + return "", err + } + return value.AsString(), nil +} + // DecodeObjectWithSensitiveKeys retains validated sensitivity paths as top-level // input keys. Terraform partitions whole variables, so a sensitive descendant // makes its containing input sensitive even when it has no non-empty leaves. diff --git a/pkg/plugin/adapter/component/objects_test.go b/pkg/plugin/adapter/component/objects_test.go index cfc84c683ff..0088dfe6cab 100644 --- a/pkg/plugin/adapter/component/objects_test.go +++ b/pkg/plugin/adapter/component/objects_test.go @@ -84,6 +84,50 @@ func TestObjectSensitivePaths(t *testing.T) { } } +func TestDecodeValueHandlesAnyKnownValue(t *testing.T) { + text, err := DecodeValue(objectWireValue(t, cty.StringVal("plain"))) + require.NoError(t, err) + require.Equal(t, "plain", text) + + object, err := DecodeValue(objectWireValue(t, cty.ObjectVal(map[string]cty.Value{"a": cty.True}))) + require.NoError(t, err) + require.Equal(t, map[string]any{"a": true}, object) + + null, err := DecodeValue(objectWireValue(t, cty.NullVal(cty.String))) + require.NoError(t, err) + require.Nil(t, null) + + _, err = DecodeValue(objectWireValue(t, cty.UnknownVal(cty.String))) + require.ErrorIs(t, err, errUtils.ErrPluginUnsupportedExtension) + _, err = DecodeValue(nil) + require.ErrorIs(t, err, errUtils.ErrPluginProtocol) +} + +func TestDecodeStringReadsStringsAndRejectsEverythingElse(t *testing.T) { + t.Cleanup(atmosio.Reset) + atmosio.Reset() + require.NoError(t, atmosio.Initialize()) + text, err := DecodeString(nil) + require.NoError(t, err) + require.Empty(t, text) + + secret := objectWireValue(t, cty.StringVal("token-value")) + secret.SensitivePaths = []*pluginv1.ComponentValuePath{{}} + text, err = DecodeString(secret) + require.NoError(t, err) + require.Equal(t, "token-value", text) + require.NotContains(t, atmosio.GetContext().Masker().Mask("token-value"), "token-value") + + _, err = DecodeString(objectWireValue(t, cty.NumberIntVal(1))) + require.ErrorIs(t, err, errUtils.ErrPluginProtocol) + _, err = DecodeString(objectWireValue(t, cty.NullVal(cty.String))) + require.ErrorIs(t, err, errUtils.ErrPluginProtocol) + _, err = DecodeString(objectWireValue(t, cty.UnknownVal(cty.String))) + require.ErrorIs(t, err, errUtils.ErrPluginUnsupportedExtension) + _, err = DecodeString(&pluginv1.ComponentValue{TypeJson: []byte("{")}) + require.Error(t, err) +} + func TestObjectStructuredSensitivity(t *testing.T) { atmosio.Reset() t.Cleanup(atmosio.Reset) diff --git a/pkg/plugin/adapter/component/registry.go b/pkg/plugin/adapter/component/registry.go index dd3261a7703..88416608d75 100644 --- a/pkg/plugin/adapter/component/registry.go +++ b/pkg/plugin/adapter/component/registry.go @@ -17,6 +17,7 @@ import ( errUtils "github.com/cloudposse/atmos/errors" "github.com/cloudposse/atmos/pkg/perf" "github.com/cloudposse/atmos/pkg/plugin/extension" + "github.com/cloudposse/atmos/pkg/plugin/manifest" pluginv1 "github.com/cloudposse/atmos/pkg/plugin/proto/atmos/plugin/v1" ) @@ -95,8 +96,9 @@ func declaresCapability(activation *extension.Activation) bool { exported = true } } + version := manifest.HostContractVersion(activation.Capability) for _, service := range activation.Description.Services { - if service.GetService() == activation.Capability && service.GetMin() == 1 && service.GetMax() == 1 { + if service.GetService() == activation.Capability && service.GetMin() == version && service.GetMax() == version { versioned = true } } diff --git a/pkg/plugin/adapter/configuration/execution.go b/pkg/plugin/adapter/configuration/execution.go index a2a0d6d9e52..9233c0bfe79 100644 --- a/pkg/plugin/adapter/configuration/execution.go +++ b/pkg/plugin/adapter/configuration/execution.go @@ -3,10 +3,12 @@ package configuration import ( "fmt" "maps" + "os" "slices" errUtils "github.com/cloudposse/atmos/errors" "github.com/cloudposse/atmos/pkg/component/execution" + adapter "github.com/cloudposse/atmos/pkg/plugin/adapter/component" pluginv1 "github.com/cloudposse/atmos/pkg/plugin/proto/atmos/plugin/v1" "github.com/cloudposse/atmos/pkg/schema" ) @@ -86,3 +88,23 @@ func preparedHooks(hooks []*pluginv1.ComponentHook) ([]schema.ComponentHook, err } return result, nil } + +// preparedIAM copies the role the provider asks the host to assume. A web identity token names a file +// when one exists at that path, as it does for Terragrunt, and is the token itself otherwise; only the +// file path may be kept with a saved plan, so a literal token stays in memory. +func preparedIAM(source *pluginv1.ComponentIam) (*schema.ComponentIAM, error) { + if source == nil || source.GetRole() == "" { + return nil, nil + } + token, err := adapter.DecodeString(source.GetWebIdentityToken()) + if err != nil { + return nil, err + } + result := &schema.ComponentIAM{Role: source.GetRole(), SessionName: source.GetSessionName(), DurationSeconds: source.GetDurationSeconds()} + if info, statErr := os.Stat(token); token != "" && statErr == nil && info.Mode().IsRegular() { + result.WebIdentityTokenFile = token + } else { + result.WebIdentityToken, result.WebIdentityTokenLiteral = token, token != "" + } + return result, nil +} diff --git a/pkg/plugin/adapter/configuration/parity.go b/pkg/plugin/adapter/configuration/parity.go new file mode 100644 index 00000000000..a7b2dadb5fc --- /dev/null +++ b/pkg/plugin/adapter/configuration/parity.go @@ -0,0 +1,135 @@ +package configuration + +import ( + "fmt" + "regexp" + "slices" + "strings" + "time" + + errUtils "github.com/cloudposse/atmos/errors" + adapter "github.com/cloudposse/atmos/pkg/plugin/adapter/component" + pluginv1 "github.com/cloudposse/atmos/pkg/plugin/proto/atmos/plugin/v1" + "github.com/cloudposse/atmos/pkg/schema" +) + +// preparedErrors copies the provider's retry and ignore policies. Anything the host cannot honor, +// such as a pattern that does not compile, is a protocol error and never a silently skipped policy. +func preparedErrors(source *pluginv1.ComponentErrorsConfiguration) (*schema.ComponentErrors, error) { + if source == nil || len(source.GetRetry())+len(source.GetIgnore()) == 0 { + return nil, nil + } + result := &schema.ComponentErrors{} + for _, retry := range source.GetRetry() { + if retry == nil || retry.GetMaxAttempts() < 1 || validatePatterns(retry.GetRetryableErrors()) != nil { + return nil, fmt.Errorf("%w: invalid retry policy", errUtils.ErrPluginProtocol) + } + result.Retry = append(result.Retry, schema.ComponentRetryPolicy{ + Name: retry.GetName(), Patterns: slices.Clone(retry.GetRetryableErrors()), + MaxAttempts: int(retry.GetMaxAttempts()), SleepInterval: time.Duration(retry.GetSleepIntervalSeconds()) * time.Second, + }) + } + for _, ignore := range source.GetIgnore() { + policy, err := preparedIgnore(ignore) + if err != nil { + return nil, err + } + result.Ignore = append(result.Ignore, policy) + } + return result, nil +} + +// preparedIgnore copies one ignore policy and decodes its signals to plain values. +func preparedIgnore(ignore *pluginv1.ComponentIgnorePolicy) (schema.ComponentIgnorePolicy, error) { + if ignore == nil || validatePatterns(ignore.GetIgnorableErrors()) != nil { + return schema.ComponentIgnorePolicy{}, fmt.Errorf("%w: invalid ignore policy", errUtils.ErrPluginProtocol) + } + signals := make(map[string]any, len(ignore.GetSignals())) + for name, value := range ignore.GetSignals() { + decoded, err := adapter.DecodeValue(value) + if err != nil { + return schema.ComponentIgnorePolicy{}, err + } + signals[name] = decoded + } + return schema.ComponentIgnorePolicy{Name: ignore.GetName(), Patterns: slices.Clone(ignore.GetIgnorableErrors()), Message: ignore.GetMessage(), Signals: signals}, nil +} + +// validatePatterns checks that every error pattern compiles, ignoring the "!" veto prefix. +func validatePatterns(patterns []string) error { + for _, pattern := range patterns { + if _, err := regexp.Compile(strings.TrimPrefix(pattern, "!")); err != nil { + return err + } + } + return nil +} + +// preparedExclusion records that the provider wants this operation skipped. +func preparedExclusion(source *pluginv1.ComponentExclude) *schema.ComponentExclusion { + if source == nil || !source.GetCondition() { + return nil + } + return &schema.ComponentExclusion{Actions: slices.Clone(source.GetActions())} +} + +// preparedIgnoredRun records that an ignore policy ended preparation of the unit. +func preparedIgnoredRun(result *pluginv1.PrepareComponentResponse) *schema.ComponentIgnoredRun { + if !result.GetRunIgnored() { + return nil + } + return &schema.ComponentIgnoredRun{Policy: result.GetIgnoredPolicy(), Message: result.GetIgnoredMessage()} +} + +// checkTerraformBinary verifies that the executable the unit selects now is the one it selected at +// discovery, because discovery handed it to the host as the component's `command`. A difference +// means the unit's configuration changed between the two evaluations. +func checkTerraformBinary(info *schema.ConfigAndStacksInfo, instance *pluginv1.DiscoveredComponent, binary string) error { + discovered, err := adapter.DecodeObject(instance.Configuration) + if err != nil { + return err + } + // A terraform_binary removed after discovery is a change too: the host would otherwise keep + // running the executable discovery selected. + command, _ := discovered["command"].(string) + if command == binary { + return nil + } + return errUtils.Build(fmt.Errorf("%w: component %q in stack %q now selects %q, but was discovered with %q", errUtils.ErrComponentBinaryChanged, info.ComponentFromArg, info.Stack, binary, command)). + WithContext("component", info.ComponentFromArg). + WithContext("stack", info.Stack). + WithHint("Run the command again; `terraform_binary` must evaluate to the same value every time, so avoid values that differ between invocations"). + Err() +} + +// warnDiagnostics prints the provider's warnings about a unit and never fails the command. +func warnDiagnostics(diagnostics []*pluginv1.Diagnostic) { + for _, diagnostic := range diagnostics { + adapter.WarnDiagnostic(diagnostic) + } +} + +// applyParity copies the settings that select how a unit runs into the prepared configuration: +// the executable, error handling, exclusion, and feature values. Error handling lives with the +// execution metadata because the host applies it around the same subprocesses as hooks. +func applyParity(prepared *schema.PreparedComponentConfiguration, result *pluginv1.PrepareComponentResponse) error { + policies, err := preparedErrors(result.GetErrors()) + if err != nil { + return err + } + if policies != nil { + if prepared.Execution == nil { + prepared.Execution = &schema.ComponentExecution{} + } + prepared.Execution.Errors = policies + // The unit's errors block is evaluated once, so a saved plan applied with the + // apply-time metadata keeps the same retry and ignore policies. + if prepared.Execution.Apply != nil { + prepared.Execution.Apply.Errors = policies + } + } + prepared.TerraformBinary = result.GetTerraformBinary() + prepared.Exclusion = preparedExclusion(result.GetExclude()) + prepared.FeatureFlags = result.GetFeatureFlags() + return nil +} diff --git a/pkg/plugin/adapter/configuration/preparation.go b/pkg/plugin/adapter/configuration/preparation.go index d6604cd9fc7..27523a05b38 100644 --- a/pkg/plugin/adapter/configuration/preparation.go +++ b/pkg/plugin/adapter/configuration/preparation.go @@ -4,6 +4,7 @@ import ( "context" "fmt" "maps" + "net/url" "os" "path/filepath" "slices" @@ -27,8 +28,8 @@ func (s *Session) Prepare(ctx context.Context, info *schema.ConfigAndStacksInfo) if !schema.ConfigurationOwned(info.ComponentSettingsSection) { return nil, nil } - if info.UseTerraformPlan || info.PlanFile != "" || info.StoredPlanFile != "" { - return nil, fmt.Errorf("%w: plugin saved-plan binding is not implemented", errUtils.ErrPluginUnsupportedExtension) + if err := rejectSavedPlan(info); err != nil { + return nil, err } selected, err := s.componentRequest(ctx, info) if err != nil { @@ -42,12 +43,50 @@ func (s *Session) Prepare(ctx context.Context, info *schema.ConfigAndStacksInfo) if err != nil { return nil, err } + if err = checkTerraformBinary(info, selected.instance, prepared.TerraformBinary); err != nil { + return nil, err + } + warnDiagnostics(result.Diagnostics) if err := checkSourcesInRoot(ctx, s.root, prepared.Sources); err != nil { return nil, err } return prepared, nil } +// ProbeIAM reads only the Terragrunt flags before full evaluation. It must not +// run infrastructure commands or return a module configuration. +func (s *Session) ProbeIAM(ctx context.Context, info *schema.ConfigAndStacksInfo) (*schema.ComponentIAM, error) { + defer perf.Track(nil, "configuration.Session.ProbeIAM")() + + selected, err := s.componentRequest(ctx, info) + if err != nil { + return nil, err + } + if selected.instance.Provider != "terragrunt" { + return nil, nil + } + selected.request.Operation.Mode = "iam-probe" + selected.request.Effects = &pluginv1.ComponentEffects{Commands: true, Network: true, Secrets: true, Writes: true, StateReads: true} + result, err := selected.provider.Prepare(ctx, selected.request) + if err != nil { + return nil, err + } + for _, diagnostic := range result.Diagnostics { + if err := adapter.DiagnosticError(diagnostic); err != nil { + return nil, err + } + } + return preparedIAM(result.Iam) +} + +// rejectSavedPlan refuses a saved plan, which a plugin-provided configuration cannot bind to yet. +func rejectSavedPlan(info *schema.ConfigAndStacksInfo) error { + if info.UseTerraformPlan || info.PlanFile != "" || info.StoredPlanFile != "" { + return fmt.Errorf("%w: plugin saved-plan binding is not implemented", errUtils.ErrPluginUnsupportedExtension) + } + return nil +} + // runProviderPrepare asks the owning provider to reevaluate the selected unit with every effect // granted, after checking module ownership and engine selection. Only an explicit --command // flag may switch the engine discovered from the global configuration. The dependencies the @@ -67,6 +106,13 @@ func (s *Session) runProviderPrepare(ctx context.Context, info *schema.ConfigAnd if err != nil { return nil, err } + // An error diagnostic fails the preparation; warnings are printed by the caller. + for _, diagnostic := range result.Diagnostics { + if err = adapter.DiagnosticError(diagnostic); err != nil { + return nil, err + } + } + // The provider prepares for the binary it is given, so any other engine is a protocol violation. if result.Engine != expectedEngine(info, selected.instance) { return nil, errUtils.ErrPluginProtocol } @@ -433,11 +479,21 @@ func preparedConfiguration(result *pluginv1.PrepareComponentResponse) (*schema.P resolveLocalBackend(backend, result.State.WorkingDirectory) } prepared := &schema.PreparedComponentConfiguration{Engine: result.Engine, Source: result.ModuleSource, Inputs: inputs, SensitiveInputKeys: sensitiveKeys, BackendType: result.State.BackendType, Backend: backend, Workspace: result.State.Workspace, BackendGenerated: result.BackendGenerated} + if prepared.SourceAuthorization, err = preparedSourceAuthorization(result); err != nil { + return nil, err + } execution, err := preparedExecution(result.Execution) if err != nil { return nil, err } prepared.Execution = execution + prepared.IgnoredRun = preparedIgnoredRun(result) + if prepared.IAM, err = preparedIAM(result.Iam); err != nil { + return nil, err + } + if err := applyParity(prepared, result); err != nil { + return nil, err + } for _, file := range result.Files { prepared.Files = append(prepared.Files, schema.ComponentConfigurationFile{Path: file.Path, Content: file.Content, IfExists: file.IfExists, Sensitive: file.Sensitive, Mode: file.Mode, UnitFileCopy: file.UnitFileCopy}) } @@ -448,6 +504,42 @@ func preparedConfiguration(result *pluginv1.PrepareComponentResponse) (*schema.P return prepared, nil } +func preparedSourceAuthorization(result *pluginv1.PrepareComponentResponse) (*schema.SourceAuthorization, error) { + encoded := result.GetModuleSourceAuthorization() + origin := result.GetModuleSourceAuthOrigin() + if encoded == nil && origin == "" { + return nil, nil + } + if !validSensitiveAuthorization(encoded, origin) { + return nil, errUtils.ErrPluginProtocol + } + authorization, err := adapter.DecodeString(encoded) + if err != nil || authorization == "" { + return nil, errUtils.ErrPluginProtocol + } + if !sameSourceOrigin(result.GetModuleSource(), origin) { + return nil, errUtils.ErrPluginProtocol + } + return &schema.SourceAuthorization{Origin: origin, Header: authorization}, nil +} + +func validSensitiveAuthorization(encoded *pluginv1.ComponentValue, origin string) bool { + return encoded != nil && origin != "" && len(encoded.GetSensitivePaths()) == 1 && len(encoded.GetSensitivePaths()[0].GetSteps()) == 0 +} + +func sameSourceOrigin(source, origin string) bool { + archive, archiveErr := url.Parse(source) + registry, registryErr := url.Parse(origin) + if archiveErr != nil || registryErr != nil || !validRegistryOrigin(registry) { + return false + } + return archive.Scheme == registry.Scheme && strings.EqualFold(archive.Host, registry.Host) +} + +func validRegistryOrigin(registry *url.URL) bool { + return registry.Host != "" && registry.User == nil && registry.Path == "" && registry.RawQuery == "" && registry.Fragment == "" && (registry.Scheme == "https" || registry.Scheme == "http") +} + func resolveLocalBackend(backend map[string]any, root string) { for key, fallback := range map[string]string{"path": "terraform.tfstate", "workspace_dir": "terraform.tfstate.d"} { path, _ := backend[key].(string) diff --git a/pkg/plugin/adapter/configuration/preparation_execution_test.go b/pkg/plugin/adapter/configuration/preparation_execution_test.go index 3da72d46fcb..0c411ce6c5d 100644 --- a/pkg/plugin/adapter/configuration/preparation_execution_test.go +++ b/pkg/plugin/adapter/configuration/preparation_execution_test.go @@ -1,9 +1,15 @@ package configuration import ( + "encoding/json" + "os" + "path/filepath" "testing" "github.com/stretchr/testify/require" + "github.com/zclconf/go-cty/cty" + ctyjson "github.com/zclconf/go-cty/cty/json" + "github.com/zclconf/go-cty/cty/msgpack" "google.golang.org/protobuf/proto" errUtils "github.com/cloudposse/atmos/errors" @@ -102,6 +108,88 @@ func TestPreparedConfigurationRejectsUnrunnableHooks(t *testing.T) { } } +// sensitiveString encodes a string the way a provider sends a secret: the whole value is sensitive. +func sensitiveString(t *testing.T, text string) *pluginv1.ComponentValue { + t.Helper() + typeJSON, err := ctyjson.MarshalType(cty.String) + require.NoError(t, err) + payload, err := msgpack.Marshal(cty.StringVal(text), cty.String) + require.NoError(t, err) + return &pluginv1.ComponentValue{TypeJson: typeJSON, Msgpack: payload, SensitivePaths: []*pluginv1.ComponentValuePath{{}}} +} + +func TestPreparedConfigurationCarriesIAM(t *testing.T) { + response := executionResponse(t, nil) + response.Iam = &pluginv1.ComponentIam{Role: "arn:aws:iam::1:role/x", SessionName: "s", DurationSeconds: 900} + got, err := preparedConfiguration(roundTrip(t, response)) + require.NoError(t, err) + require.Equal(t, &schema.ComponentIAM{Role: "arn:aws:iam::1:role/x", SessionName: "s", DurationSeconds: 900}, got.IAM) + + response.Iam = nil + got, err = preparedConfiguration(roundTrip(t, response)) + require.NoError(t, err) + require.Nil(t, got.IAM) + + response.Iam = &pluginv1.ComponentIam{} + got, err = preparedConfiguration(roundTrip(t, response)) + require.NoError(t, err) + require.Nil(t, got.IAM, "an IAM message without a role asks for nothing") +} + +func TestPreparedRegistryAuthorizationRequiresSameOriginAndSensitivity(t *testing.T) { + response := executionResponse(t, nil) + response.ModuleSource = "https://registry.example/module.tar.gz//subdir" + response.ModuleSourceAuthOrigin = "https://registry.example" + response.ModuleSourceAuthorization = sensitiveString(t, "Bearer registry-secret") + prepared, err := preparedConfiguration(roundTrip(t, response)) + require.NoError(t, err) + require.Equal(t, &schema.SourceAuthorization{Origin: "https://registry.example", Header: "Bearer registry-secret"}, prepared.SourceAuthorization) + + response.ModuleSourceAuthOrigin = "https://other.example" + _, err = preparedConfiguration(roundTrip(t, response)) + require.ErrorIs(t, err, errUtils.ErrPluginProtocol) + response.ModuleSourceAuthOrigin = "https://registry.example" + response.ModuleSourceAuthorization.SensitivePaths = nil + _, err = preparedConfiguration(roundTrip(t, response)) + require.ErrorIs(t, err, errUtils.ErrPluginProtocol) +} + +func TestPreparedIAMDistinguishesTokenFilesFromLiteralTokens(t *testing.T) { + file := filepath.Join(t.TempDir(), "token") + require.NoError(t, os.WriteFile(file, []byte("file-token\n"), 0o600)) + + fromFile, err := preparedIAM(&pluginv1.ComponentIam{Role: "r", WebIdentityToken: sensitiveString(t, file)}) + require.NoError(t, err) + require.Equal(t, &schema.ComponentIAM{Role: "r", WebIdentityTokenFile: file}, fromFile) + + literal, err := preparedIAM(&pluginv1.ComponentIam{Role: "r", WebIdentityToken: sensitiveString(t, "literal-token")}) + require.NoError(t, err) + require.Equal(t, &schema.ComponentIAM{Role: "r", WebIdentityToken: "literal-token", WebIdentityTokenLiteral: true}, literal) + + // A literal token never reaches a serialized record. + record, err := json.Marshal(literal) + require.NoError(t, err) + require.NotContains(t, string(record), "literal-token") + + // A directory is not a token file. + directory, err := preparedIAM(&pluginv1.ComponentIam{Role: "r", WebIdentityToken: sensitiveString(t, t.TempDir())}) + require.NoError(t, err) + require.Empty(t, directory.WebIdentityTokenFile) + + _, err = preparedIAM(&pluginv1.ComponentIam{Role: "r", WebIdentityToken: &pluginv1.ComponentValue{TypeJson: []byte("{")}}) + require.Error(t, err) +} + +func TestDecodeStringRejectsNonStrings(t *testing.T) { + text, err := adapter.DecodeString(nil) + require.NoError(t, err) + require.Empty(t, text) + number, err := adapter.EncodeObject(map[string]any{"a": 1}) + require.NoError(t, err) + _, err = adapter.DecodeString(number) + require.ErrorIs(t, err, errUtils.ErrPluginProtocol) +} + // A plan preparation carries the apply-time evaluation separately. An empty one is kept, // because it means apply runs nothing rather than falling back to the plan-time hooks. func TestPreparedConfigurationCarriesApplyTimeExecution(t *testing.T) { @@ -132,3 +220,30 @@ func TestPreparedConfigurationCarriesApplyTimeExecution(t *testing.T) { require.NoError(t, err) require.Same(t, got.Execution, got.Execution.ForSavedPlanApply()) } + +// The unit's errors block is not command specific, so a saved plan applied with the +// apply-time metadata keeps the retry and ignore policies of the reviewed preparation. +func TestPreparedApplyTimeExecutionKeepsErrorPolicies(t *testing.T) { + response := executionResponse(t, &pluginv1.ComponentExecutionConfiguration{ + Apply: &pluginv1.ComponentExecutionConfiguration{ + AfterHooks: []*pluginv1.ComponentHook{{Name: "notify", Commands: []string{"apply"}, Execute: []string{"tool", "notify"}}}, + }, + }) + response.Errors = &pluginv1.ComponentErrorsConfiguration{ + Retry: []*pluginv1.ComponentRetryPolicy{{Name: "net", RetryableErrors: []string{"timeout"}, MaxAttempts: 3, SleepIntervalSeconds: 1}}, + } + got, err := preparedConfiguration(roundTrip(t, response)) + require.NoError(t, err) + require.NotNil(t, got.Execution.Errors) + applied := got.Execution.ForSavedPlanApply() + require.Same(t, got.Execution.Apply, applied) + require.Equal(t, got.Execution.Errors, applied.Errors) + require.Equal(t, "net", applied.Errors.Retry[0].Name) + + // Without errors policies, the apply-time metadata has none either. + response.Errors = nil + got, err = preparedConfiguration(roundTrip(t, response)) + require.NoError(t, err) + require.Nil(t, got.Execution.Errors) + require.Nil(t, got.Execution.ForSavedPlanApply().Errors) +} diff --git a/pkg/plugin/adapter/configuration/preparation_test.go b/pkg/plugin/adapter/configuration/preparation_test.go index c4ef54491fe..163eb7c6829 100644 --- a/pkg/plugin/adapter/configuration/preparation_test.go +++ b/pkg/plugin/adapter/configuration/preparation_test.go @@ -5,6 +5,7 @@ import ( "errors" "path/filepath" "testing" + "time" "github.com/stretchr/testify/require" "go.uber.org/mock/gomock" @@ -68,6 +69,18 @@ func TestPrepareTranslatesGeneratedArtifactsAndWatchedSources(t *testing.T) { require.Equal(t, "-var=a=1", sent.Operation.TerraformArguments[0], "the request owns a copy of the arguments") } +func TestProbeIAMUsesStagedProviderMode(t *testing.T) { + session, provider, info := preparationSession(t) + provider.EXPECT().Prepare(gomock.Any(), gomock.Any()).DoAndReturn(func(_ context.Context, request *pluginv1.ComponentRequest) (*pluginv1.PrepareComponentResponse, error) { + require.Equal(t, "iam-probe", request.Operation.Mode) + require.True(t, request.Effects.Secrets) + return &pluginv1.PrepareComponentResponse{Iam: &pluginv1.ComponentIam{Role: "arn:aws:iam::1:role/x"}}, nil + }) + iam, err := session.ProbeIAM(t.Context(), info) + require.NoError(t, err) + require.Equal(t, "arn:aws:iam::1:role/x", iam.Role) +} + func TestEngineMismatchNamesTheCommandAndBothEngines(t *testing.T) { session, _, info := preparationSession(t) info.Command = "terraform" @@ -210,6 +223,105 @@ func TestPrepareRejectsAnotherModuleSourceButAllowsOtherSourceSettings(t *testin } } +func TestPrepareTranslatesErrorsExclusionAndFeatures(t *testing.T) { + session, provider, info := preparationSession(t) + signal, err := adapter.EncodeObject(map[string]any{"flag": true}) + require.NoError(t, err) + provider.EXPECT().Prepare(gomock.Any(), gomock.Any()).Return(&pluginv1.PrepareComponentResponse{ + Engine: "opentofu", State: &pluginv1.ComponentStateIdentity{Backend: signal}, + Errors: &pluginv1.ComponentErrorsConfiguration{ + Retry: []*pluginv1.ComponentRetryPolicy{{Name: "net", RetryableErrors: []string{"timeout", "!fatal"}, MaxAttempts: 3, SleepIntervalSeconds: 4}}, + Ignore: []*pluginv1.ComponentIgnorePolicy{{Name: "known", IgnorableErrors: []string{"exists"}, Message: "fine", Signals: map[string]*pluginv1.ComponentValue{"alert": signal}}}, + }, + Exclude: &pluginv1.ComponentExclude{Condition: true, Actions: []string{"plan"}}, + FeatureFlags: map[string]string{"flag": "true"}, + Diagnostics: []*pluginv1.Diagnostic{{Detail: &pluginv1.ErrorDetail{Code: "read_config_before_hook_ignored", Summary: "never runs", Severity: pluginv1.Severity_SEVERITY_WARNING}}}, + }, nil) + prepared, err := session.Prepare(t.Context(), info) + require.NoError(t, err) + + require.NotNil(t, prepared.Execution, "error handling alone still produces execution metadata") + require.Equal(t, []schema.ComponentRetryPolicy{{Name: "net", Patterns: []string{"timeout", "!fatal"}, MaxAttempts: 3, SleepInterval: 4 * time.Second}}, prepared.Execution.Errors.Retry) + require.Equal(t, "known", prepared.Execution.Errors.Ignore[0].Name) + require.Equal(t, "fine", prepared.Execution.Errors.Ignore[0].Message) + require.Equal(t, map[string]any{"flag": true}, prepared.Execution.Errors.Ignore[0].Signals["alert"]) + require.Equal(t, &schema.ComponentExclusion{Actions: []string{"plan"}}, prepared.Exclusion) + require.Equal(t, map[string]string{"flag": "true"}, prepared.FeatureFlags) +} + +func TestPrepareTranslatesAnIgnoredRun(t *testing.T) { + for name, test := range map[string]struct { + response *pluginv1.PrepareComponentResponse + want *schema.ComponentIgnoredRun + }{ + "ignored": {&pluginv1.PrepareComponentResponse{RunIgnored: true, IgnoredPolicy: "registry", IgnoredMessage: "down"}, &schema.ComponentIgnoredRun{Policy: "registry", Message: "down"}}, + "ignored, no text": {&pluginv1.PrepareComponentResponse{RunIgnored: true}, &schema.ComponentIgnoredRun{}}, + "text without flag": {&pluginv1.PrepareComponentResponse{IgnoredPolicy: "registry", IgnoredMessage: "down"}, nil}, + "not ignored": {&pluginv1.PrepareComponentResponse{}, nil}, + } { + t.Run(name, func(t *testing.T) { + session, provider, info := preparationSession(t) + response := test.response + response.Engine, response.State = "opentofu", &pluginv1.ComponentStateIdentity{} + provider.EXPECT().Prepare(gomock.Any(), gomock.Any()).Return(response, nil) + prepared, err := session.Prepare(t.Context(), info) + require.NoError(t, err) + require.Equal(t, test.want, prepared.IgnoredRun) + }) + } +} + +func TestPreparedErrorsRejectsPoliciesTheHostCannotHonor(t *testing.T) { + empty, err := preparedErrors(&pluginv1.ComponentErrorsConfiguration{}) + require.NoError(t, err) + require.Nil(t, empty) + for name, source := range map[string]*pluginv1.ComponentErrorsConfiguration{ + "no attempts": {Retry: []*pluginv1.ComponentRetryPolicy{{Name: "x", RetryableErrors: []string{"a"}}}}, + "nil retry": {Retry: []*pluginv1.ComponentRetryPolicy{nil}}, + "bad retry pattern": {Retry: []*pluginv1.ComponentRetryPolicy{{Name: "x", RetryableErrors: []string{"("}, MaxAttempts: 1}}}, + "bad ignore pattern": {Ignore: []*pluginv1.ComponentIgnorePolicy{{Name: "x", IgnorableErrors: []string{"!("}}}}, + "nil ignore": {Ignore: []*pluginv1.ComponentIgnorePolicy{nil}}, + "undecodable signals": {Ignore: []*pluginv1.ComponentIgnorePolicy{{Name: "x", Signals: map[string]*pluginv1.ComponentValue{"s": {TypeJson: []byte("nope")}}}}}, + } { + t.Run(name, func(t *testing.T) { + _, err := preparedErrors(source) + require.Error(t, err) + }) + } +} + +func TestPreparedExclusionNeedsItsCondition(t *testing.T) { + require.Nil(t, preparedExclusion(nil)) + require.Nil(t, preparedExclusion(&pluginv1.ComponentExclude{Actions: []string{"plan"}})) + require.Equal(t, &schema.ComponentExclusion{Actions: []string{"plan"}}, preparedExclusion(&pluginv1.ComponentExclude{Condition: true, Actions: []string{"plan"}})) +} + +func TestTerraformBinaryMustMatchDiscovery(t *testing.T) { + info := &schema.ConfigAndStacksInfo{ComponentFromArg: "app", Stack: "live"} + configuration, err := adapter.EncodeObject(map[string]any{"command": "tofu"}) + require.NoError(t, err) + discovered := &pluginv1.DiscoveredComponent{Configuration: configuration} + + require.NoError(t, checkTerraformBinary(info, discovered, "tofu")) + plain, err := adapter.EncodeObject(map[string]any{}) + require.NoError(t, err) + require.NoError(t, checkTerraformBinary(info, &pluginv1.DiscoveredComponent{Configuration: plain}, ""), "a unit without terraform_binary has nothing to compare") + removed := checkTerraformBinary(info, discovered, "") + require.ErrorIs(t, removed, errUtils.ErrComponentBinaryChanged, "removing terraform_binary after discovery is a change") + require.ErrorContains(t, removed, `discovered with "tofu"`) + err = checkTerraformBinary(info, discovered, "terraform") + require.ErrorIs(t, err, errUtils.ErrComponentBinaryChanged) + require.ErrorContains(t, err, `now selects "terraform"`) + require.ErrorContains(t, err, `discovered with "tofu"`) +} + +func TestPrepareRejectsAnExecutableChangeSinceDiscovery(t *testing.T) { + session, provider, info := preparationSession(t) + provider.EXPECT().Prepare(gomock.Any(), gomock.Any()).Return(&pluginv1.PrepareComponentResponse{Engine: "opentofu", TerraformBinary: "tofu", State: &pluginv1.ComponentStateIdentity{}}, nil) + _, err := session.Prepare(t.Context(), info) + require.ErrorIs(t, err, errUtils.ErrComponentBinaryChanged) +} + func TestSessionResolvesStateWithoutPreparation(t *testing.T) { config := enabledConfig(t) mock := NewMockProvider(gomock.NewController(t)) diff --git a/pkg/plugin/bootstrap/description.go b/pkg/plugin/bootstrap/description.go index 9aa8c63e51b..ca249c1a8a6 100644 --- a/pkg/plugin/bootstrap/description.go +++ b/pkg/plugin/bootstrap/description.go @@ -32,7 +32,8 @@ func DescribePackage(directory string, definition *manifest.Config, bindings []e } sort.Strings(names) for _, name := range names { - result.Services = append(result.Services, &pluginv1.ServiceVersionRange{Service: name, Min: 1, Max: 1}) + version := manifest.HostContractVersion(name) + result.Services = append(result.Services, &pluginv1.ServiceVersionRange{Service: name, Min: version, Max: version}) } return result, nil } diff --git a/pkg/plugin/bootstrap/description_test.go b/pkg/plugin/bootstrap/description_test.go new file mode 100644 index 00000000000..035f11767e8 --- /dev/null +++ b/pkg/plugin/bootstrap/description_test.go @@ -0,0 +1,41 @@ +package bootstrap + +import ( + "os" + "path/filepath" + "testing" + + "github.com/stretchr/testify/require" + + "github.com/cloudposse/atmos/pkg/plugin/extension" + "github.com/cloudposse/atmos/pkg/plugin/manifest" +) + +// TestDescribePackageAdvertisesHostContractVersions verifies the handshake +// requires the current preparation contract. A plugin answering with the first +// preparation version fails negotiation before Prepare runs. +func TestDescribePackageAdvertisesHostContractVersions(t *testing.T) { + t.Parallel() + directory := t.TempDir() + require.NoError(t, os.WriteFile(filepath.Join(directory, "plugin.yaml"), []byte("fixture"), 0o600)) + definition := &manifest.Config{} + definition.Metadata.Name = "fixture/components" + definition.Metadata.Version = "0.1.0" + bindings := []extension.Binding{ + {Capability: extension.ComponentPreparation, Name: "fixture/hcl"}, + {Capability: extension.ComponentDiscovery, Name: "fixture/hcl"}, + } + result, err := DescribePackage(directory, definition, bindings) + require.NoError(t, err) + versions := make(map[string][2]uint32) + for _, service := range result.Services { + versions[service.Service] = [2]uint32{service.Min, service.Max} + } + require.Equal(t, map[string][2]uint32{extension.ComponentDiscovery: {1, 1}, extension.ComponentPreparation: {2, 2}}, versions) + + registry, err := Descriptors() + require.NoError(t, err) + descriptor, ok := registry.Get(extension.ComponentPreparation) + require.True(t, ok) + require.Equal(t, uint32(2), descriptor.Version) +} diff --git a/pkg/plugin/bootstrap/descriptors.go b/pkg/plugin/bootstrap/descriptors.go index fe290168e4f..ab10453be35 100644 --- a/pkg/plugin/bootstrap/descriptors.go +++ b/pkg/plugin/bootstrap/descriptors.go @@ -11,6 +11,7 @@ import ( adapter "github.com/cloudposse/atmos/pkg/plugin/adapter/auth" "github.com/cloudposse/atmos/pkg/plugin/credential" "github.com/cloudposse/atmos/pkg/plugin/extension" + "github.com/cloudposse/atmos/pkg/plugin/manifest" ) // Descriptors declares supported extension types. Lifecycle code receives this @@ -30,7 +31,7 @@ func descriptors(constructors *ConstructorSet) (*extension.Registry, error) { {Capability: "auth.identity", Path: []string{"auth", "identities"}, Version: 1, Validate: extension.ValidateAuth, Scaffold: "plugin-oidc"}, {Capability: extension.ComponentDiscovery, Path: []string{componentsPath, "discovery"}, Version: 1, Validate: extension.ValidateComponent}, {Capability: extension.ComponentValidation, Path: []string{componentsPath, "validation"}, Version: 1, Validate: extension.ValidateComponent}, - {Capability: extension.ComponentPreparation, Path: []string{componentsPath, "preparation"}, Version: 1, Validate: extension.ValidateComponent}, + {Capability: extension.ComponentPreparation, Path: []string{componentsPath, "preparation"}, Version: manifest.HostContractVersion(extension.ComponentPreparation), Validate: extension.ValidateComponent}, {Capability: extension.ComponentState, Path: []string{componentsPath, "state"}, Version: 1, Validate: extension.ValidateComponent}, } { if constructors != nil { diff --git a/pkg/plugin/manifest/manifest.go b/pkg/plugin/manifest/manifest.go index f34ac4d29ff..dfd48c1d4e5 100644 --- a/pkg/plugin/manifest/manifest.go +++ b/pkg/plugin/manifest/manifest.go @@ -24,6 +24,22 @@ const maxManifestBytes = 8 << 20 var packageName = regexp.MustCompile(`^[a-z][a-z0-9-]*/[a-z][a-z0-9-]*$`) +// hostContractVersions lists contracts whose host version is newer than 1. +// Version 2 of component.preparation adds terraform_arguments, backend_generated, +// and execution metadata. An earlier host cannot honor those fields, so the +// version change makes that host reject the plugin before preparation. +var hostContractVersions = map[string]uint32{"component.preparation": 2} + +// HostContractVersion returns the single version of a contract this host speaks. +// +//nolint:lintroller // Pure lookup; no I/O. +func HostContractVersion(contract string) uint32 { + if version, ok := hostContractVersions[contract]; ok { + return version + } + return 1 +} + // Config is a versioned Atmos manifest; loading it never executes plugin code. type Config = manifest.Manifest[Spec] @@ -146,8 +162,8 @@ func checkCompatibility(compatibility Compatibility) error { if err := supportsV1(compatibility.Protocol); err != nil { return err } - for _, expression := range compatibility.Contracts { - if err := supportsV1(expression); err != nil { + for contract, expression := range compatibility.Contracts { + if err := supports(expression, HostContractVersion(contract)); err != nil { return err } } @@ -165,8 +181,12 @@ func checkCompatibility(compatibility Compatibility) error { } func supportsV1(expression string) error { + return supports(expression, 1) +} + +func supports(expression string, major uint32) error { constraint, err := semver.NewConstraint(expression) - if err != nil || expression == "" || !constraint.Check(semver.MustParse("1.0.0")) { + if err != nil || expression == "" || !constraint.Check(semver.New(uint64(major), 0, 0, "", "")) { return errUtils.ErrPluginUnsupportedExtension } return nil diff --git a/pkg/plugin/manifest/manifest_test.go b/pkg/plugin/manifest/manifest_test.go index 5a6d61d30f6..f055d40ab47 100644 --- a/pkg/plugin/manifest/manifest_test.go +++ b/pkg/plugin/manifest/manifest_test.go @@ -37,6 +37,8 @@ func TestParseValidatesPackageAndRuntimeContracts(t *testing.T) { {"protocol", "protocol: 1,", "protocol: 2,", errUtils.ErrPluginInvalidConfig}, {"executable", "bin/plugin", "../escape", errUtils.ErrPluginInvalidConfig}, {"unsupported contract", `auth.provider: ">=1 <2"`, `auth.provider: ">=2"`, errUtils.ErrPluginUnsupportedExtension}, + {"earlier preparation contract", `auth.provider: ">=1 <2"`, `component.preparation: ">=1 <2"`, errUtils.ErrPluginUnsupportedExtension}, + {"later preparation contract", `auth.provider: ">=1 <2"`, `component.preparation: ">=3"`, errUtils.ErrPluginUnsupportedExtension}, {"unsupported protocol", `protocol: ">=1 <2"`, `protocol: ">=2"`, errUtils.ErrPluginUnsupportedExtension}, {"empty protocol", `protocol: ">=1 <2"`, `protocol: ""`, errUtils.ErrPluginUnsupportedExtension}, } @@ -48,6 +50,11 @@ func TestParseValidatesPackageAndRuntimeContracts(t *testing.T) { require.Nil(t, value) }) } + current, err := Parse([]byte(strings.Replace(validManifest, `auth.provider: ">=1 <2"`, `component.preparation: ">=2 <3"`, 1))) + require.NoError(t, err, "the current preparation contract must be accepted") + require.NotNil(t, current) + require.Equal(t, uint32(1), HostContractVersion("component.discovery")) + require.Equal(t, uint32(2), HostContractVersion("component.preparation")) _, err = Parse([]byte("{invalid yaml")) require.Error(t, err) } diff --git a/pkg/plugin/proto/atmos/plugin/v1/plugin.pb.go b/pkg/plugin/proto/atmos/plugin/v1/plugin.pb.go index 57f75fe9c17..14d6af372ae 100644 --- a/pkg/plugin/proto/atmos/plugin/v1/plugin.pb.go +++ b/pkg/plugin/proto/atmos/plugin/v1/plugin.pb.go @@ -3380,23 +3380,43 @@ type PrepareComponentResponse struct { // Maps each input name to the authored file, line, and column that produced // its value, for provenance reporting. Missing entries mean unknown origin. InputOrigins map[string]*SourceLocation `protobuf:"bytes,12,rep,name=input_origins,json=inputOrigins,proto3" json:"input_origins,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"` - // Fields 13-18 are reserved for Terragrunt parity. The host does not read - // them yet; a plugin that cannot honor a setting must reject it with an - // ErrorDetail instead of returning it, and must leave these fields unset. - // Engine executable selected by terraform_binary. + // Fields 13-18 carry Terragrunt settings that decide how the host runs the unit. The host reads + // terraform_binary, errors, exclude, iam, and feature_flags (13-16 and 18). It does not read + // encryption (17) yet. A plugin that cannot honor a setting must reject it with an ErrorDetail + // instead of returning a value the host would silently ignore. + // Engine executable selected by terraform_binary. Empty means the engine of the component. TerraformBinary string `protobuf:"bytes,13,opt,name=terraform_binary,json=terraformBinary,proto3" json:"terraform_binary,omitempty"` - // errors block: retry and ignore policies. + // errors block: retry and ignore policies. The host applies them around the engine subprocess. Errors *ComponentErrorsConfiguration `protobuf:"bytes,14,opt,name=errors,proto3" json:"errors,omitempty"` - // exclude block: conditional unit exclusion. + // exclude block, set only when its condition holds and it applies to the current operation. + // The host skips the operation for the component and runs nothing. See ComponentExclude. Exclude *ComponentExclude `protobuf:"bytes,15,opt,name=exclude,proto3" json:"exclude,omitempty"` // iam_role, iam_assume_role_session_name, iam_assume_role_duration, iam_web_identity_token. + // The host assumes the role, never the plugin. Iam *ComponentIam `protobuf:"bytes,16,opt,name=iam,proto3" json:"iam,omitempty"` - // encryption block as opaque, provider-defined data. + // encryption block as opaque, provider-defined data. The host ignores it. Encryption *ComponentValue `protobuf:"bytes,17,opt,name=encryption,proto3" json:"encryption,omitempty"` - // feature flag name to its evaluated value rendered as a string. - FeatureFlags map[string]string `protobuf:"bytes,18,rep,name=feature_flags,json=featureFlags,proto3" json:"feature_flags,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"` - unknownFields protoimpl.UnknownFields - sizeCache protoimpl.SizeCache + // Feature flag name to its evaluated value rendered as a string. The plugin has already applied + // the flags while evaluating the unit, so they are reflected in every other field. The host + // keeps them with the prepared configuration. + FeatureFlags map[string]string `protobuf:"bytes,18,rep,name=feature_flags,json=featureFlags,proto3" json:"feature_flags,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"` + // Authorization for an archive returned by a module registry. This value is + // sensitive and may be sent only to module_source_auth_origin. + ModuleSourceAuthorization *ComponentValue `protobuf:"bytes,19,opt,name=module_source_authorization,json=moduleSourceAuthorization,proto3" json:"module_source_authorization,omitempty"` + ModuleSourceAuthOrigin string `protobuf:"bytes,20,opt,name=module_source_auth_origin,json=moduleSourceAuthOrigin,proto3" json:"module_source_auth_origin,omitempty"` + // Time spent resolving a tfr:// source, including its bounded retries. + RegistryResolutionNanos uint64 `protobuf:"varint,21,opt,name=registry_resolution_nanos,json=registryResolutionNanos,proto3" json:"registry_resolution_nanos,omitempty"` + // True when an errors-block ignore policy matched a failure the plugin hit while preparing the unit, + // for example a registry resolution failure, so no source could be resolved. The host ends the unit's + // run successfully without executing it, as it does for an ignored source fetch. The remaining fields + // describe the unit as written. + RunIgnored bool `protobuf:"varint,23,opt,name=run_ignored,json=runIgnored,proto3" json:"run_ignored,omitempty"` + // Name of the ignore policy that matched. Set with run_ignored. + IgnoredPolicy string `protobuf:"bytes,24,opt,name=ignored_policy,json=ignoredPolicy,proto3" json:"ignored_policy,omitempty"` + // Message of the ignore policy that matched. Set with run_ignored. + IgnoredMessage string `protobuf:"bytes,25,opt,name=ignored_message,json=ignoredMessage,proto3" json:"ignored_message,omitempty"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache } func (x *PrepareComponentResponse) Reset() { @@ -3555,7 +3575,49 @@ func (x *PrepareComponentResponse) GetFeatureFlags() map[string]string { return nil } -// ComponentErrorsConfiguration mirrors the Terragrunt errors block. Reserved; not implemented. +func (x *PrepareComponentResponse) GetModuleSourceAuthorization() *ComponentValue { + if x != nil { + return x.ModuleSourceAuthorization + } + return nil +} + +func (x *PrepareComponentResponse) GetModuleSourceAuthOrigin() string { + if x != nil { + return x.ModuleSourceAuthOrigin + } + return "" +} + +func (x *PrepareComponentResponse) GetRegistryResolutionNanos() uint64 { + if x != nil { + return x.RegistryResolutionNanos + } + return 0 +} + +func (x *PrepareComponentResponse) GetRunIgnored() bool { + if x != nil { + return x.RunIgnored + } + return false +} + +func (x *PrepareComponentResponse) GetIgnoredPolicy() string { + if x != nil { + return x.IgnoredPolicy + } + return "" +} + +func (x *PrepareComponentResponse) GetIgnoredMessage() string { + if x != nil { + return x.IgnoredMessage + } + return "" +} + +// ComponentErrorsConfiguration mirrors the Terragrunt errors block. type ComponentErrorsConfiguration struct { state protoimpl.MessageState `protogen:"open.v1"` Retry []*ComponentRetryPolicy `protobuf:"bytes,1,rep,name=retry,proto3" json:"retry,omitempty"` @@ -3747,7 +3809,7 @@ func (x *ComponentIgnorePolicy) GetSignals() map[string]*ComponentValue { return nil } -// ComponentExclude mirrors the Terragrunt exclude block. Reserved; not implemented. +// ComponentExclude mirrors the Terragrunt exclude block. type ComponentExclude struct { state protoimpl.MessageState `protogen:"open.v1"` // Evaluated "if" condition. @@ -3810,7 +3872,7 @@ func (x *ComponentExclude) GetExcludeDependencies() bool { return false } -// ComponentIam mirrors the Terragrunt IAM settings. Reserved; not implemented. +// ComponentIam mirrors the Terragrunt IAM settings. // The web identity token is a secret and must be carried as a sensitive value. type ComponentIam struct { state protoimpl.MessageState `protogen:"open.v1"` @@ -4223,8 +4285,7 @@ const file_atmos_plugin_v1_plugin_proto_rawDesc = "" + "\frun_on_error\x18\x05 \x01(\bR\n" + "runOnError\x12'\n" + "\x0fsuppress_stdout\x18\x06 \x01(\bR\x0esuppressStdout\x12\x1b\n" + - "\ton_errors\x18\a \x03(\tR\bonErrors\"\x85\n" + - "\n" + + "\ton_errors\x18\a \x03(\tR\bonErrors\"\xce\f\n" + "\x18PrepareComponentResponse\x12\x16\n" + "\x06engine\x18\x01 \x01(\tR\x06engine\x12+\n" + "\x11working_directory\x18\x02 \x01(\tR\x10workingDirectory\x12#\n" + @@ -4246,7 +4307,14 @@ const file_atmos_plugin_v1_plugin_proto_rawDesc = "" + "\n" + "encryption\x18\x11 \x01(\v2\x1f.atmos.plugin.v1.ComponentValueR\n" + "encryption\x12`\n" + - "\rfeature_flags\x18\x12 \x03(\v2;.atmos.plugin.v1.PrepareComponentResponse.FeatureFlagsEntryR\ffeatureFlags\x1a`\n" + + "\rfeature_flags\x18\x12 \x03(\v2;.atmos.plugin.v1.PrepareComponentResponse.FeatureFlagsEntryR\ffeatureFlags\x12_\n" + + "\x1bmodule_source_authorization\x18\x13 \x01(\v2\x1f.atmos.plugin.v1.ComponentValueR\x19moduleSourceAuthorization\x129\n" + + "\x19module_source_auth_origin\x18\x14 \x01(\tR\x16moduleSourceAuthOrigin\x12:\n" + + "\x19registry_resolution_nanos\x18\x15 \x01(\x04R\x17registryResolutionNanos\x12\x1f\n" + + "\vrun_ignored\x18\x17 \x01(\bR\n" + + "runIgnored\x12%\n" + + "\x0eignored_policy\x18\x18 \x01(\tR\rignoredPolicy\x12'\n" + + "\x0fignored_message\x18\x19 \x01(\tR\x0eignoredMessage\x1a`\n" + "\x11InputOriginsEntry\x12\x10\n" + "\x03key\x18\x01 \x01(\tR\x03key\x125\n" + "\x05value\x18\x02 \x01(\v2\x1f.atmos.plugin.v1.SourceLocationR\x05value:\x028\x01\x1a?\n" + @@ -4484,61 +4552,62 @@ var file_atmos_plugin_v1_plugin_proto_depIdxs = []int32{ 53, // 77: atmos.plugin.v1.PrepareComponentResponse.iam:type_name -> atmos.plugin.v1.ComponentIam 31, // 78: atmos.plugin.v1.PrepareComponentResponse.encryption:type_name -> atmos.plugin.v1.ComponentValue 64, // 79: atmos.plugin.v1.PrepareComponentResponse.feature_flags:type_name -> atmos.plugin.v1.PrepareComponentResponse.FeatureFlagsEntry - 50, // 80: atmos.plugin.v1.ComponentErrorsConfiguration.retry:type_name -> atmos.plugin.v1.ComponentRetryPolicy - 51, // 81: atmos.plugin.v1.ComponentErrorsConfiguration.ignore:type_name -> atmos.plugin.v1.ComponentIgnorePolicy - 65, // 82: atmos.plugin.v1.ComponentIgnorePolicy.signals:type_name -> atmos.plugin.v1.ComponentIgnorePolicy.SignalsEntry - 31, // 83: atmos.plugin.v1.ComponentIam.web_identity_token:type_name -> atmos.plugin.v1.ComponentValue - 43, // 84: atmos.plugin.v1.ResolveComponentStateResponse.state:type_name -> atmos.plugin.v1.ComponentStateIdentity - 35, // 85: atmos.plugin.v1.ResolveComponentStateResponse.sources:type_name -> atmos.plugin.v1.ComponentSource - 17, // 86: atmos.plugin.v1.ResolveComponentStateResponse.diagnostics:type_name -> atmos.plugin.v1.Diagnostic - 31, // 87: atmos.plugin.v1.ComponentOperation.OptionsEntry.value:type_name -> atmos.plugin.v1.ComponentValue - 15, // 88: atmos.plugin.v1.PrepareComponentResponse.InputOriginsEntry.value:type_name -> atmos.plugin.v1.SourceLocation - 31, // 89: atmos.plugin.v1.ComponentIgnorePolicy.SignalsEntry.value:type_name -> atmos.plugin.v1.ComponentValue - 7, // 90: atmos.plugin.v1.Info.Describe:input_type -> atmos.plugin.v1.DescribeRequest - 9, // 91: atmos.plugin.v1.Info.Configure:input_type -> atmos.plugin.v1.ConfigureRequest - 10, // 92: atmos.plugin.v1.AuthProvider.Validate:input_type -> atmos.plugin.v1.ProviderRequest - 10, // 93: atmos.plugin.v1.AuthProvider.Authenticate:input_type -> atmos.plugin.v1.ProviderRequest - 20, // 94: atmos.plugin.v1.AuthProvider.Refresh:input_type -> atmos.plugin.v1.RefreshRequest - 10, // 95: atmos.plugin.v1.AuthProvider.Logout:input_type -> atmos.plugin.v1.ProviderRequest - 10, // 96: atmos.plugin.v1.AuthProvider.Environment:input_type -> atmos.plugin.v1.ProviderRequest - 10, // 97: atmos.plugin.v1.AuthProvider.Paths:input_type -> atmos.plugin.v1.ProviderRequest - 23, // 98: atmos.plugin.v1.AuthProvider.PrepareEnvironment:input_type -> atmos.plugin.v1.PrepareEnvironmentRequest - 11, // 99: atmos.plugin.v1.AuthIdentity.Validate:input_type -> atmos.plugin.v1.IdentityRequest - 26, // 100: atmos.plugin.v1.AuthIdentity.Authenticate:input_type -> atmos.plugin.v1.IdentityAuthenticateRequest - 11, // 101: atmos.plugin.v1.AuthIdentity.Logout:input_type -> atmos.plugin.v1.IdentityRequest - 11, // 102: atmos.plugin.v1.AuthIdentity.Environment:input_type -> atmos.plugin.v1.IdentityRequest - 11, // 103: atmos.plugin.v1.AuthIdentity.Paths:input_type -> atmos.plugin.v1.IdentityRequest - 27, // 104: atmos.plugin.v1.AuthIdentity.PrepareEnvironment:input_type -> atmos.plugin.v1.IdentityPrepareRequest - 28, // 105: atmos.plugin.v1.Commands.Run:input_type -> atmos.plugin.v1.CommandRequest - 40, // 106: atmos.plugin.v1.ComponentDiscovery.Discover:input_type -> atmos.plugin.v1.DiscoverComponentsRequest - 42, // 107: atmos.plugin.v1.ComponentValidation.Validate:input_type -> atmos.plugin.v1.ComponentRequest - 42, // 108: atmos.plugin.v1.ComponentPreparation.Prepare:input_type -> atmos.plugin.v1.ComponentRequest - 42, // 109: atmos.plugin.v1.ComponentState.Resolve:input_type -> atmos.plugin.v1.ComponentRequest - 8, // 110: atmos.plugin.v1.Info.Describe:output_type -> atmos.plugin.v1.DescribeResponse - 68, // 111: atmos.plugin.v1.Info.Configure:output_type -> google.protobuf.Empty - 18, // 112: atmos.plugin.v1.AuthProvider.Validate:output_type -> atmos.plugin.v1.ValidateResponse - 14, // 113: atmos.plugin.v1.AuthProvider.Authenticate:output_type -> atmos.plugin.v1.AuthenticateEvent - 21, // 114: atmos.plugin.v1.AuthProvider.Refresh:output_type -> atmos.plugin.v1.RefreshResponse - 68, // 115: atmos.plugin.v1.AuthProvider.Logout:output_type -> google.protobuf.Empty - 22, // 116: atmos.plugin.v1.AuthProvider.Environment:output_type -> atmos.plugin.v1.EnvironmentResponse - 25, // 117: atmos.plugin.v1.AuthProvider.Paths:output_type -> atmos.plugin.v1.PathsResponse - 22, // 118: atmos.plugin.v1.AuthProvider.PrepareEnvironment:output_type -> atmos.plugin.v1.EnvironmentResponse - 18, // 119: atmos.plugin.v1.AuthIdentity.Validate:output_type -> atmos.plugin.v1.ValidateResponse - 14, // 120: atmos.plugin.v1.AuthIdentity.Authenticate:output_type -> atmos.plugin.v1.AuthenticateEvent - 68, // 121: atmos.plugin.v1.AuthIdentity.Logout:output_type -> google.protobuf.Empty - 22, // 122: atmos.plugin.v1.AuthIdentity.Environment:output_type -> atmos.plugin.v1.EnvironmentResponse - 25, // 123: atmos.plugin.v1.AuthIdentity.Paths:output_type -> atmos.plugin.v1.PathsResponse - 22, // 124: atmos.plugin.v1.AuthIdentity.PrepareEnvironment:output_type -> atmos.plugin.v1.EnvironmentResponse - 30, // 125: atmos.plugin.v1.Commands.Run:output_type -> atmos.plugin.v1.CommandEvent - 41, // 126: atmos.plugin.v1.ComponentDiscovery.Discover:output_type -> atmos.plugin.v1.DiscoverComponentsResponse - 18, // 127: atmos.plugin.v1.ComponentValidation.Validate:output_type -> atmos.plugin.v1.ValidateResponse - 48, // 128: atmos.plugin.v1.ComponentPreparation.Prepare:output_type -> atmos.plugin.v1.PrepareComponentResponse - 54, // 129: atmos.plugin.v1.ComponentState.Resolve:output_type -> atmos.plugin.v1.ResolveComponentStateResponse - 110, // [110:130] is the sub-list for method output_type - 90, // [90:110] is the sub-list for method input_type - 90, // [90:90] is the sub-list for extension type_name - 90, // [90:90] is the sub-list for extension extendee - 0, // [0:90] is the sub-list for field type_name + 31, // 80: atmos.plugin.v1.PrepareComponentResponse.module_source_authorization:type_name -> atmos.plugin.v1.ComponentValue + 50, // 81: atmos.plugin.v1.ComponentErrorsConfiguration.retry:type_name -> atmos.plugin.v1.ComponentRetryPolicy + 51, // 82: atmos.plugin.v1.ComponentErrorsConfiguration.ignore:type_name -> atmos.plugin.v1.ComponentIgnorePolicy + 65, // 83: atmos.plugin.v1.ComponentIgnorePolicy.signals:type_name -> atmos.plugin.v1.ComponentIgnorePolicy.SignalsEntry + 31, // 84: atmos.plugin.v1.ComponentIam.web_identity_token:type_name -> atmos.plugin.v1.ComponentValue + 43, // 85: atmos.plugin.v1.ResolveComponentStateResponse.state:type_name -> atmos.plugin.v1.ComponentStateIdentity + 35, // 86: atmos.plugin.v1.ResolveComponentStateResponse.sources:type_name -> atmos.plugin.v1.ComponentSource + 17, // 87: atmos.plugin.v1.ResolveComponentStateResponse.diagnostics:type_name -> atmos.plugin.v1.Diagnostic + 31, // 88: atmos.plugin.v1.ComponentOperation.OptionsEntry.value:type_name -> atmos.plugin.v1.ComponentValue + 15, // 89: atmos.plugin.v1.PrepareComponentResponse.InputOriginsEntry.value:type_name -> atmos.plugin.v1.SourceLocation + 31, // 90: atmos.plugin.v1.ComponentIgnorePolicy.SignalsEntry.value:type_name -> atmos.plugin.v1.ComponentValue + 7, // 91: atmos.plugin.v1.Info.Describe:input_type -> atmos.plugin.v1.DescribeRequest + 9, // 92: atmos.plugin.v1.Info.Configure:input_type -> atmos.plugin.v1.ConfigureRequest + 10, // 93: atmos.plugin.v1.AuthProvider.Validate:input_type -> atmos.plugin.v1.ProviderRequest + 10, // 94: atmos.plugin.v1.AuthProvider.Authenticate:input_type -> atmos.plugin.v1.ProviderRequest + 20, // 95: atmos.plugin.v1.AuthProvider.Refresh:input_type -> atmos.plugin.v1.RefreshRequest + 10, // 96: atmos.plugin.v1.AuthProvider.Logout:input_type -> atmos.plugin.v1.ProviderRequest + 10, // 97: atmos.plugin.v1.AuthProvider.Environment:input_type -> atmos.plugin.v1.ProviderRequest + 10, // 98: atmos.plugin.v1.AuthProvider.Paths:input_type -> atmos.plugin.v1.ProviderRequest + 23, // 99: atmos.plugin.v1.AuthProvider.PrepareEnvironment:input_type -> atmos.plugin.v1.PrepareEnvironmentRequest + 11, // 100: atmos.plugin.v1.AuthIdentity.Validate:input_type -> atmos.plugin.v1.IdentityRequest + 26, // 101: atmos.plugin.v1.AuthIdentity.Authenticate:input_type -> atmos.plugin.v1.IdentityAuthenticateRequest + 11, // 102: atmos.plugin.v1.AuthIdentity.Logout:input_type -> atmos.plugin.v1.IdentityRequest + 11, // 103: atmos.plugin.v1.AuthIdentity.Environment:input_type -> atmos.plugin.v1.IdentityRequest + 11, // 104: atmos.plugin.v1.AuthIdentity.Paths:input_type -> atmos.plugin.v1.IdentityRequest + 27, // 105: atmos.plugin.v1.AuthIdentity.PrepareEnvironment:input_type -> atmos.plugin.v1.IdentityPrepareRequest + 28, // 106: atmos.plugin.v1.Commands.Run:input_type -> atmos.plugin.v1.CommandRequest + 40, // 107: atmos.plugin.v1.ComponentDiscovery.Discover:input_type -> atmos.plugin.v1.DiscoverComponentsRequest + 42, // 108: atmos.plugin.v1.ComponentValidation.Validate:input_type -> atmos.plugin.v1.ComponentRequest + 42, // 109: atmos.plugin.v1.ComponentPreparation.Prepare:input_type -> atmos.plugin.v1.ComponentRequest + 42, // 110: atmos.plugin.v1.ComponentState.Resolve:input_type -> atmos.plugin.v1.ComponentRequest + 8, // 111: atmos.plugin.v1.Info.Describe:output_type -> atmos.plugin.v1.DescribeResponse + 68, // 112: atmos.plugin.v1.Info.Configure:output_type -> google.protobuf.Empty + 18, // 113: atmos.plugin.v1.AuthProvider.Validate:output_type -> atmos.plugin.v1.ValidateResponse + 14, // 114: atmos.plugin.v1.AuthProvider.Authenticate:output_type -> atmos.plugin.v1.AuthenticateEvent + 21, // 115: atmos.plugin.v1.AuthProvider.Refresh:output_type -> atmos.plugin.v1.RefreshResponse + 68, // 116: atmos.plugin.v1.AuthProvider.Logout:output_type -> google.protobuf.Empty + 22, // 117: atmos.plugin.v1.AuthProvider.Environment:output_type -> atmos.plugin.v1.EnvironmentResponse + 25, // 118: atmos.plugin.v1.AuthProvider.Paths:output_type -> atmos.plugin.v1.PathsResponse + 22, // 119: atmos.plugin.v1.AuthProvider.PrepareEnvironment:output_type -> atmos.plugin.v1.EnvironmentResponse + 18, // 120: atmos.plugin.v1.AuthIdentity.Validate:output_type -> atmos.plugin.v1.ValidateResponse + 14, // 121: atmos.plugin.v1.AuthIdentity.Authenticate:output_type -> atmos.plugin.v1.AuthenticateEvent + 68, // 122: atmos.plugin.v1.AuthIdentity.Logout:output_type -> google.protobuf.Empty + 22, // 123: atmos.plugin.v1.AuthIdentity.Environment:output_type -> atmos.plugin.v1.EnvironmentResponse + 25, // 124: atmos.plugin.v1.AuthIdentity.Paths:output_type -> atmos.plugin.v1.PathsResponse + 22, // 125: atmos.plugin.v1.AuthIdentity.PrepareEnvironment:output_type -> atmos.plugin.v1.EnvironmentResponse + 30, // 126: atmos.plugin.v1.Commands.Run:output_type -> atmos.plugin.v1.CommandEvent + 41, // 127: atmos.plugin.v1.ComponentDiscovery.Discover:output_type -> atmos.plugin.v1.DiscoverComponentsResponse + 18, // 128: atmos.plugin.v1.ComponentValidation.Validate:output_type -> atmos.plugin.v1.ValidateResponse + 48, // 129: atmos.plugin.v1.ComponentPreparation.Prepare:output_type -> atmos.plugin.v1.PrepareComponentResponse + 54, // 130: atmos.plugin.v1.ComponentState.Resolve:output_type -> atmos.plugin.v1.ResolveComponentStateResponse + 111, // [111:131] is the sub-list for method output_type + 91, // [91:111] is the sub-list for method input_type + 91, // [91:91] is the sub-list for extension type_name + 91, // [91:91] is the sub-list for extension extendee + 0, // [0:91] is the sub-list for field type_name } func init() { file_atmos_plugin_v1_plugin_proto_init() } diff --git a/pkg/plugin/proto/parity_test.go b/pkg/plugin/proto/parity_test.go index 70f9af1f828..e91ff0c24d1 100644 --- a/pkg/plugin/proto/parity_test.go +++ b/pkg/plugin/proto/parity_test.go @@ -7,6 +7,10 @@ import ( "testing" "github.com/stretchr/testify/require" + "google.golang.org/protobuf/proto" + "google.golang.org/protobuf/reflect/protoreflect" + + pluginv1 "github.com/cloudposse/atmos/pkg/plugin/proto/atmos/plugin/v1" ) func TestGeneratedProtocolParity(t *testing.T) { @@ -29,3 +33,31 @@ func TestGeneratedProtocolParity(t *testing.T) { require.Equal(t, string(cli), string(author), "regenerate both bindings using atmos dev generate proto") } } + +// The ignored-run fields are additive: they use numbers above every field of the message, and an +// ignored run survives the wire while a response that never sets them reads as not ignored. +func TestPrepareComponentResponseIgnoredRunRoundTrip(t *testing.T) { + t.Parallel() + fields := (&pluginv1.PrepareComponentResponse{}).ProtoReflect().Descriptor().Fields() + for name, number := range map[string]int32{"run_ignored": 23, "ignored_policy": 24, "ignored_message": 25} { + field := fields.ByName(protoreflect.Name(name)) + require.NotNil(t, field, name) + require.EqualValues(t, number, field.Number(), name) + } + + data, err := proto.Marshal(&pluginv1.PrepareComponentResponse{Engine: "terraform", RunIgnored: true, IgnoredPolicy: "registry", IgnoredMessage: "down"}) + require.NoError(t, err) + var got pluginv1.PrepareComponentResponse + require.NoError(t, proto.Unmarshal(data, &got)) + require.True(t, got.GetRunIgnored()) + require.Equal(t, "registry", got.GetIgnoredPolicy()) + require.Equal(t, "down", got.GetIgnoredMessage()) + require.Equal(t, "terraform", got.GetEngine()) + + data, err = proto.Marshal(&pluginv1.PrepareComponentResponse{Engine: "terraform"}) + require.NoError(t, err) + var plain pluginv1.PrepareComponentResponse + require.NoError(t, proto.Unmarshal(data, &plain)) + require.False(t, plain.GetRunIgnored()) + require.Empty(t, plain.GetIgnoredPolicy()) +} diff --git a/pkg/plugin/runtime/native/native.go b/pkg/plugin/runtime/native/native.go index ae54b6483fe..ace5ed3a10d 100644 --- a/pkg/plugin/runtime/native/native.go +++ b/pkg/plugin/runtime/native/native.go @@ -26,6 +26,8 @@ const ( maxMessage = 16 << 20 pluginName = "atmos" brokerService = "atmos.api.v1" + // A first launch can include OS assessment of a newly installed binary under load. + defaultStartTimeout = 45 * time.Second ) var handshake = plugin.HandshakeConfig{ProtocolVersion: 1, MagicCookieKey: "ATMOS_PLUGIN_MAGIC_COOKIE", MagicCookieValue: "atmos-native-plugin-v1"} @@ -34,11 +36,12 @@ var handshake = plugin.HandshakeConfig{ProtocolVersion: 1, MagicCookieKey: "ATMO type ( Option func(*options) options struct { - directory string - environment []string - api *bridge.Server - expected *pluginv1.DescribeResponse - callTimeout time.Duration + directory string + environment []string + api *bridge.Server + expected *pluginv1.DescribeResponse + callTimeout time.Duration + startTimeout time.Duration } ) @@ -59,6 +62,17 @@ func WithEnvironment(environment []string) Option { //nolint:lintroller // Trivial accessor, pure lookup, factory option, or delegation; exempt under repository tracking policy. func WithAPI(server *bridge.Server) Option { return func(o *options) { o.api = server } } +// WithStartTimeout bounds the plugin handshake and initial configuration RPCs. +// +//nolint:lintroller // Factory option; exempt under repository tracking policy. +func WithStartTimeout(timeout time.Duration) Option { + return func(o *options) { + if timeout > 0 { + o.startTimeout = timeout + } + } +} + // Handle owns a native process and its versioned service clients. type Handle struct { Info pluginv1.InfoClient @@ -88,7 +102,7 @@ func Start(ctx context.Context, executable string, opts ...Option) (*Handle, err if err := ctx.Err(); err != nil { return nil, err } - settings := options{api: bridge.New()} + settings := options{api: bridge.New(), startTimeout: defaultStartTimeout} for _, apply := range opts { apply(&settings) } @@ -98,7 +112,7 @@ func Start(ctx context.Context, executable string, opts ...Option) (*Handle, err command := exec.CommandContext(stopped.ctx, executable) configureProcess(command) command.Dir, command.Env = settings.directory, settings.environment - client := newClient(command) + client := newClient(command, settings.startTimeout) // The process-level signal handler exits without unwinding, so it runs this first. forget := signals.RegisterExitCleanup(func() { client.Kill() @@ -115,7 +129,7 @@ func Start(ctx context.Context, executable string, opts ...Option) (*Handle, err } handle.process, handle.api, handle.command = client, settings.api, command handle.interrupt, handle.forget, handle.callTimeout = stopped, forget, settings.callTimeout - startup, cancel := context.WithTimeout(stopped.ctx, 15*time.Second) + startup, cancel := context.WithTimeout(stopped.ctx, settings.startTimeout) defer cancel() if err := handle.describe(startup, settings.expected); err != nil { handle.Close() @@ -128,11 +142,11 @@ func Start(ctx context.Context, executable string, opts ...Option) (*Handle, err return handle, nil } -func newClient(command *exec.Cmd) *plugin.Client { +func newClient(command *exec.Cmd, startTimeout time.Duration) *plugin.Client { return plugin.NewClient(&plugin.ClientConfig{ HandshakeConfig: handshake, Plugins: map[string]plugin.Plugin{pluginName: &protocol{}}, Cmd: command, AllowedProtocols: []plugin.Protocol{plugin.ProtocolGRPC}, - AutoMTLS: true, SkipHostEnv: true, StartTimeout: 15 * time.Second, + AutoMTLS: true, SkipHostEnv: true, StartTimeout: startTimeout, Logger: hclog.NewNullLogger(), Stderr: io.Discard, SyncStderr: io.Discard, SyncStdout: io.Discard, GRPCDialOptions: []grpc.DialOption{grpc.WithDefaultCallOptions(grpc.MaxCallRecvMsgSize(maxMessage), grpc.MaxCallSendMsgSize(maxMessage))}, }) diff --git a/pkg/provisioner/source/floating_ref_test.go b/pkg/provisioner/source/floating_ref_test.go new file mode 100644 index 00000000000..3ddaa7642d6 --- /dev/null +++ b/pkg/provisioner/source/floating_ref_test.go @@ -0,0 +1,68 @@ +package source + +import ( + "testing" + + "github.com/stretchr/testify/assert" + + "github.com/cloudposse/atmos/pkg/schema" +) + +func TestFloatingRef(t *testing.T) { + tests := []struct { + name string + spec schema.VendorComponentSource + wantRef string + floating bool + }{ + {name: "branch in query", spec: schema.VendorComponentSource{Uri: "git::https://example.invalid/m.git?ref=main"}, wantRef: "main", floating: true}, + {name: "branch with subdirectory", spec: schema.VendorComponentSource{Uri: "git::file:///repo//sub?ref=feature/x"}, wantRef: "feature/x", floating: true}, + {name: "branch via version", spec: schema.VendorComponentSource{Uri: "github.com/org/repo//m", Version: "develop"}, wantRef: "develop", floating: true}, + {name: "heads ref", spec: schema.VendorComponentSource{Uri: "git::https://example.invalid/m.git?ref=refs/heads/main"}, wantRef: "refs/heads/main", floating: true}, + {name: "semver tag", spec: schema.VendorComponentSource{Uri: "git::https://example.invalid/m.git?ref=v1.2.3"}}, + {name: "bare version", spec: schema.VendorComponentSource{Uri: "git::https://example.invalid/m.git?ref=1.2"}}, + {name: "prerelease tag", spec: schema.VendorComponentSource{Uri: "git::https://example.invalid/m.git?ref=v1.2.3-rc.1"}}, + {name: "tag ref", spec: schema.VendorComponentSource{Uri: "git::https://example.invalid/m.git?ref=refs/tags/v2"}}, + {name: "commit", spec: schema.VendorComponentSource{Uri: "git::https://example.invalid/m.git?ref=0123456789abcdef0123456789abcdef01234567"}}, + {name: "short commit", spec: schema.VendorComponentSource{Uri: "git::https://example.invalid/m.git?ref=abc1234"}}, + {name: "query wins over version", spec: schema.VendorComponentSource{Uri: "git::https://example.invalid/m.git?ref=v1", Version: "main"}}, + {name: "git source without a ref follows its default branch", spec: schema.VendorComponentSource{Uri: "git::https://example.invalid/m.git"}, floating: true}, + {name: "github path without a ref", spec: schema.VendorComponentSource{Uri: "github.com/org/repo//m"}, floating: true}, + {name: "git repository with subdirectory", spec: schema.VendorComponentSource{Uri: "https://example.invalid/m.git//sub"}, floating: true}, + {name: "archive without a ref", spec: schema.VendorComponentSource{Uri: "https://example.invalid/m.tar.gz"}}, + {name: "object store without a ref", spec: schema.VendorComponentSource{Uri: "s3::https://s3.amazonaws.com/bucket/m.zip"}}, + {name: "local path", spec: schema.VendorComponentSource{Uri: "./modules/m", Version: "main"}}, + {name: "empty", spec: schema.VendorComponentSource{}}, + } + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + ref, floating := floatingRef(&test.spec) + assert.Equal(t, test.floating, floating) + assert.Equal(t, test.wantRef, ref) + }) + } + _, floating := floatingRef(nil) + assert.False(t, floating) +} + +func TestFloatingRefNotice(t *testing.T) { + spec := &schema.VendorComponentSource{Uri: "git::https://example.invalid/m.git?ref=main"} + notice := floatingRefNotice(spec, "terraform", "vpc", "live/prod") + assert.Contains(t, notice, `floating ref "main"`) + assert.Contains(t, notice, "atmos terraform source pull vpc -s live/prod --force") + + assert.Contains(t, floatingRefNotice(spec, "", "vpc", ""), "atmos terraform source pull vpc --force") + + // The command matches the component's own type. + assert.Contains(t, floatingRefNotice(spec, "helmfile", "ingress", "live/prod"), "atmos helmfile source pull ingress -s live/prod --force") + assert.Contains(t, floatingRefNotice(spec, "aws/cloudformation", "stack", ""), "atmos aws cloudformation source pull stack --force") + + // A Git source without a ref follows its default branch. + assert.Contains(t, floatingRefNotice(&schema.VendorComponentSource{Uri: "git::https://example.invalid/m.git"}, "terraform", "vpc", ""), "follows its default branch") + + // A TTL already decides freshness, and immutable refs cannot drift. + withTTL := *spec + withTTL.TTL = "1h" + assert.Empty(t, floatingRefNotice(&withTTL, "terraform", "vpc", "live/prod")) + assert.Empty(t, floatingRefNotice(&schema.VendorComponentSource{Uri: "git::https://example.invalid/m.git?ref=v1"}, "terraform", "vpc", "live/prod")) +} diff --git a/pkg/provisioner/source/local_stamp.go b/pkg/provisioner/source/local_stamp.go index b36c90f1eb7..65a9f21fa37 100644 --- a/pkg/provisioner/source/local_stamp.go +++ b/pkg/provisioner/source/local_stamp.go @@ -24,7 +24,13 @@ var localStampSkippedDirs = map[string]struct{}{".git": {}, ".terraform": {}, ". // way as vendoring resolves it. It reports false for remote sources and for missing directories. func localSourceRoot(spec *schema.VendorComponentSource) (string, bool) { uri := vendor.NormalizeURI(resolveSourceURI(spec)) - if vendor.IsLocalPath(uri) && !filepath.IsAbs(uri) { + if root, subdir, ok := vendor.LocalRootSubdir(uri); ok && !filepath.IsAbs(root) { + absolute, err := resolveLocalSourcePath(root, "") + if err != nil { + return "", false + } + uri = absolute + "//" + subdir + } else if vendor.IsLocalPath(uri) && !filepath.IsAbs(uri) { absolute, err := resolveLocalSourcePath(uri, "") if err != nil { return "", false @@ -249,6 +255,9 @@ func localSourceChanged(metadata *workdir.WorkdirMetadata, spec *schema.VendorCo // be read. It reports false for remote sources. func localSourceMissing(spec *schema.VendorComponentSource) bool { uri := vendor.NormalizeURI(resolveSourceURI(spec)) + if root, subdir, ok := vendor.LocalRootSubdir(uri); ok { + return localRootSubdirMissing(root, subdir, uri) + } switch { case vendor.IsFileURI(uri): path, err := fileURIPath(uri) @@ -259,14 +268,32 @@ func localSourceMissing(spec *schema.VendorComponentSource) bool { case !vendor.IsLocalPath(uri): return false } - if !filepath.IsAbs(uri) { - absolute, err := resolveLocalSourcePath(uri, "") + return localPathMissing(uri) +} + +// localRootSubdirMissing reports whether a root//subdir local source cannot be found or read. +func localRootSubdirMissing(root, subdir, uri string) bool { + if !filepath.IsAbs(root) { + absolute, err := resolveLocalSourcePath(root, "") if err != nil { return true } - uri = absolute + uri = absolute + "//" + subdir } - _, err := os.Stat(uri) + _, present, err := localDirectorySource(uri) + return err != nil || !present +} + +// localPathMissing reports whether a plain local path cannot be found or read. +func localPathMissing(path string) bool { + if !filepath.IsAbs(path) { + absolute, err := resolveLocalSourcePath(path, "") + if err != nil { + return true + } + path = absolute + } + _, err := os.Stat(path) return err != nil } @@ -294,7 +321,15 @@ func sameLocalSource(left, right string) bool { if !isLocalSource(left) || !isLocalSource(right) { return false } - absoluteLeft, errLeft := filepath.Abs(left) - absoluteRight, errRight := filepath.Abs(right) + leftRoot, leftSubdir, leftHasSubdir := vendor.LocalRootSubdir(left) + rightRoot, rightSubdir, rightHasSubdir := vendor.LocalRootSubdir(right) + if leftHasSubdir != rightHasSubdir || leftSubdir != rightSubdir { + return false + } + if !leftHasSubdir { + leftRoot, rightRoot = left, right + } + absoluteLeft, errLeft := filepath.Abs(leftRoot) + absoluteRight, errRight := filepath.Abs(rightRoot) return errLeft == nil && errRight == nil && absoluteLeft == absoluteRight } diff --git a/pkg/provisioner/source/local_stamp_test.go b/pkg/provisioner/source/local_stamp_test.go index 55dbd5d5cb8..df1b3d1d3ab 100644 --- a/pkg/provisioner/source/local_stamp_test.go +++ b/pkg/provisioner/source/local_stamp_test.go @@ -1,8 +1,10 @@ package source import ( + "context" "crypto/sha256" "encoding/hex" + "errors" "io" "os" "path/filepath" @@ -211,6 +213,48 @@ func TestLocalSourceStampFollowsASymlinkedRoot(t *testing.T) { assert.NotEqual(t, before, localSourceStamp(spec)) } +// The provisioning wrapper runs only when the source is actually copied, never when the +// workdir is already up to date, and its error is the provisioning error. +func TestAutoProvisionSourceRunsTheWrapperOnlyWhenProvisioning(t *testing.T) { + sourceDir := t.TempDir() + writeStampFile(t, filepath.Join(sourceDir, "main.tf"), "# v1") + config := &schema.AtmosConfiguration{BasePath: t.TempDir()} + calls := 0 + var wrapperErr error + ctx := WithProvisionWrapper(t.Context(), func(ctx context.Context, target *Target, provision func(context.Context) error) error { + calls++ + assert.NotEmpty(t, target.Dir) + if wrapperErr != nil { + return wrapperErr + } + return provision(ctx) + }) + provision := func() error { + component := map[string]any{"component": "example", "atmos_stack": "dev", "source": map[string]any{"uri": sourceDir}, "provision": map[string]any{"workdir": map[string]any{"enabled": true}}} + return AutoProvisionSource(ctx, config, "terraform", component, nil, provisioner.OutputWriters{}) + } + + require.NoError(t, provision()) + assert.Equal(t, 1, calls, "the first fetch is wrapped") + require.NoError(t, provision()) + assert.Equal(t, 1, calls, "an up-to-date workdir is not provisioned, so nothing is wrapped") + + writeStampFile(t, filepath.Join(sourceDir, "main.tf"), "# v2") + require.NoError(t, provision()) + assert.Equal(t, 2, calls, "an edited local source is provisioned again") + + writeStampFile(t, filepath.Join(sourceDir, "main.tf"), "# v3") + wrapperErr = errors.New("hook failed") + assert.ErrorIs(t, provision(), wrapperErr) + assert.Equal(t, 3, calls) +} + +func TestProvisionWithoutWrapperRunsDirectly(t *testing.T) { + ran := false + require.NoError(t, provisionWithWrapper(t.Context(), nil, func(context.Context) error { ran = true; return nil })) + assert.True(t, ran) +} + // An unstamped workdir of a local source that is gone is refreshed, so provisioning names the // missing source instead of reusing stale configuration. A remote source is unaffected. func TestLocalSourceChangedReportsAMissingUnstampedSource(t *testing.T) { diff --git a/pkg/provisioner/source/local_subdir_test.go b/pkg/provisioner/source/local_subdir_test.go new file mode 100644 index 00000000000..717cded750e --- /dev/null +++ b/pkg/provisioner/source/local_subdir_test.go @@ -0,0 +1,82 @@ +package source + +import ( + "os" + "path/filepath" + "testing" + + "github.com/stretchr/testify/require" + + errUtils "github.com/cloudposse/atmos/errors" + "github.com/cloudposse/atmos/pkg/schema" +) + +func TestLocalSubdirCopiesRepositoryRootAndTracksSibling(t *testing.T) { + root := filepath.Join(t.TempDir(), "repository") + app := filepath.Join(root, "modules", "app") + shared := filepath.Join(root, "modules", "shared") + require.NoError(t, os.MkdirAll(app, 0o755)) + require.NoError(t, os.MkdirAll(shared, 0o755)) + require.NoError(t, os.WriteFile(filepath.Join(app, "main.tf"), []byte(`module "shared" { source = "../shared" }`), 0o644)) + require.NoError(t, os.WriteFile(filepath.Join(shared, "main.tf"), []byte("output \"id\" {}"), 0o644)) + spec := &schema.VendorComponentSource{Uri: root + "//modules/app"} + target := filepath.Join(t.TempDir(), "checkout") + require.NoError(t, VendorSource(t.Context(), nil, spec, target)) + require.FileExists(t, filepath.Join(target, "modules", "app", "main.tf")) + require.FileExists(t, filepath.Join(target, "modules", "shared", "main.tf")) + first := localSourceStamp(spec) + require.NotEmpty(t, first) + require.NoError(t, os.WriteFile(filepath.Join(shared, "main.tf"), []byte("output \"changed\" {}"), 0o644)) + require.NotEqual(t, first, localSourceStamp(spec)) +} + +func TestLocalSubdirRejectsTraversalAndOutsideSymlink(t *testing.T) { + root := filepath.Join(t.TempDir(), "repository") + require.NoError(t, os.MkdirAll(root, 0o755)) + outside := t.TempDir() + require.NoError(t, os.Symlink(outside, filepath.Join(root, "outside"))) + for _, suffix := range []string{"../outside", "modules/../outside", "/outside", "outside"} { + err := VendorSource(t.Context(), nil, &schema.VendorComponentSource{Uri: root + "//" + suffix}, filepath.Join(t.TempDir(), "target")) + require.ErrorIs(t, err, errUtils.ErrSourceInvalidSpec) + } +} + +func TestLocalSubdirRetainsCacheIdentity(t *testing.T) { + root := filepath.Join(t.TempDir(), "repository") + require.True(t, sameLocalSource(root+"//modules/app", root+"//modules/app")) + require.False(t, sameLocalSource(root+"//modules/app", root+"//modules/shared")) + require.False(t, sameLocalSource(root+"//modules/app", filepath.Join(root, "modules", "app"))) +} + +func TestLocalSubdirProtectsGeneratedBackend(t *testing.T) { + stage := t.TempDir() + target := filepath.Join(t.TempDir(), "checkout") + module := filepath.Join(stage, "modules", "app") + require.NoError(t, os.MkdirAll(module, 0o755)) + require.NoError(t, os.WriteFile(filepath.Join(module, generatedBackendFile), []byte("{}"), 0o644)) + config := &schema.AtmosConfiguration{} + config.Components.Terraform.AutoGenerateBackendFile = true + section := map[string]any{"metadata": map[string]any{"source_subdir": "modules/app"}} + require.ErrorIs(t, rejectGeneratedBackendCollision(config, stage, target, section), errUtils.ErrSourceCopyFailed) + info := &schema.ConfigAndStacksInfo{ComponentSection: section} + info.SetBackendFileGeneration(false) + require.NoError(t, rejectGeneratedBackendCollision(config, stage, target, section)) +} + +func TestLocalSubdirRefreshPreservesModuleState(t *testing.T) { + root := filepath.Join(t.TempDir(), "repository") + module := filepath.Join(root, "modules", "app") + require.NoError(t, os.MkdirAll(module, 0o755)) + require.NoError(t, os.WriteFile(filepath.Join(module, "main.tf"), []byte("terraform {}"), 0o644)) + target := filepath.Join(t.TempDir(), "checkout") + section := map[string]any{"metadata": map[string]any{"source_subdir": "modules/app"}} + spec := &schema.VendorComponentSource{Uri: root + "//modules/app"} + require.NoError(t, VendorSource(t.Context(), nil, spec, target, WithRuntimePreservation(section))) + state := filepath.Join(target, "modules", "app", "terraform.tfstate") + require.NoError(t, os.WriteFile(state, []byte("retained state"), 0o600)) + require.NoError(t, os.WriteFile(filepath.Join(module, "main.tf"), []byte("terraform { required_version = \">= 1.0\" }"), 0o644)) + require.NoError(t, VendorSource(t.Context(), nil, spec, target, WithRuntimePreservation(section))) + content, err := os.ReadFile(state) + require.NoError(t, err) + require.Equal(t, "retained state", string(content)) +} diff --git a/pkg/provisioner/source/provision_hook.go b/pkg/provisioner/source/provision_hook.go index abeafd0443d..3a1ea5564d6 100644 --- a/pkg/provisioner/source/provision_hook.go +++ b/pkg/provisioner/source/provision_hook.go @@ -183,7 +183,10 @@ func AutoProvisionSource( if componentType == cfg.TerraformComponentType { options = append(options, WithRuntimePreservation(componentConfig), withProvenance(target, stack, sourceSpec, &receipt)) } - if err := vendorToTarget(ctx, atmosConfig, sourceSpec, &vendorTarget{path: targetDir, component: component, writers: writers, options: options, preserveRuntime: componentType == cfg.TerraformComponentType}); err != nil { + vendor := func(ctx context.Context) error { + return vendorToTarget(ctx, atmosConfig, sourceSpec, &vendorTarget{path: targetDir, component: component, writers: writers, options: options, preserveRuntime: componentType == cfg.TerraformComponentType}) + } + if err := provisionWithWrapper(ctx, target, vendor); err != nil { return err } @@ -691,3 +694,25 @@ func prepareAutoVendorTarget(path string, preserveRuntime bool) (bool, error) { } return os.IsNotExist(statErr), nil } + +// ProvisionWrapper runs a source provisioning action. A wrapper may run work before and after the +// action and decide what its error means; it is called only when the source is actually provisioned +// (fetched or refreshed), never when the existing copy is up to date. +type ProvisionWrapper func(ctx context.Context, target *Target, provision func(context.Context) error) error + +type provisionWrapperKey struct{} + +// WithProvisionWrapper returns a context whose source provisioning runs through wrapper. +func WithProvisionWrapper(ctx context.Context, wrapper ProvisionWrapper) context.Context { + defer perf.Track(nil, "source.WithProvisionWrapper")() + + return context.WithValue(ctx, provisionWrapperKey{}, wrapper) +} + +// provisionWithWrapper runs provision directly, or through the wrapper the context carries. +func provisionWithWrapper(ctx context.Context, target *Target, provision func(context.Context) error) error { + if wrapper, ok := ctx.Value(provisionWrapperKey{}).(ProvisionWrapper); ok && wrapper != nil { + return wrapper(ctx, target, provision) + } + return provision(ctx) +} diff --git a/pkg/provisioner/source/refresh.go b/pkg/provisioner/source/refresh.go index 20515163de6..6694f2e7bd0 100644 --- a/pkg/provisioner/source/refresh.go +++ b/pkg/provisioner/source/refresh.go @@ -120,7 +120,7 @@ const generatedBackendFile = "backend.tf.json" // Installing it would replace the generated backend, and the next generation would overwrite the // source's copy, so neither owner's version could be trusted. func rejectGeneratedBackendCollision(config *schema.AtmosConfiguration, stage, target string, componentConfig map[string]any) error { - if config == nil || !config.Components.Terraform.AutoGenerateBackendFile { + if config == nil || !schema.GenerateBackendFileForSection(config, componentConfig) { return nil } for _, root := range moduleRoots(target, componentConfig) { @@ -465,6 +465,8 @@ func moduleRoots(target string, config map[string]any) []string { if metadata, ok := config["metadata"].(map[string]any); ok { subpath, _ := metadata["component"].(string) candidates = append(candidates, subpath) + moduleSubdir, _ := metadata["source_subdir"].(string) + candidates = append(candidates, moduleSubdir) } for _, candidate := range candidates { if candidate != "" && filepath.IsLocal(candidate) { diff --git a/pkg/provisioner/source/vendor.go b/pkg/provisioner/source/vendor.go index d85a1a943d6..94b318cc57e 100644 --- a/pkg/provisioner/source/vendor.go +++ b/pkg/provisioner/source/vendor.go @@ -78,23 +78,16 @@ func VendorSource( } func vendorSource(ctx context.Context, atmosConfig *schema.AtmosConfiguration, sourceSpec *schema.VendorComponentSource, targetDir string, vendorOpts vendorSourceOptions) error { - if sourceSpec == nil { - return errUtils.Build(errUtils.ErrNilParam). - WithExplanation("source specification cannot be nil"). - Err() - } - - if sourceSpec.Uri == "" { - return errUtils.Build(errUtils.ErrSourceInvalidSpec). - WithExplanation("source URI is required"). - Err() - } - if err := validateVendorTargetDir(targetDir); err != nil { + if err := validateVendorSource(sourceSpec, targetDir); err != nil { return err } // Resolve version into URI if specified separately. uri := resolveSourceURI(sourceSpec) + uri, err := resolveLocalSubdirURI(uri, vendorOpts.baseDir) + if err != nil { + return err + } // Normalize the URI for go-getter using the same logic as regular vendoring. uri = vendor.NormalizeURI(uri) @@ -122,6 +115,40 @@ func vendorSource(ctx context.Context, atmosConfig *schema.AtmosConfiguration, s return vendorRemoteSource(ctx, atmosConfig, sourceSpec, targetDir, vendorResolvedOptions{vendorSourceOptions: vendorOpts, uri: uri}) } +// validateVendorSource checks the source specification and the target directory before vendoring. +func validateVendorSource(sourceSpec *schema.VendorComponentSource, targetDir string) error { + if sourceSpec == nil { + return errUtils.Build(errUtils.ErrNilParam). + WithExplanation("source specification cannot be nil"). + Err() + } + + if sourceSpec.Uri == "" { + return errUtils.Build(errUtils.ErrSourceInvalidSpec). + WithExplanation("source URI is required"). + Err() + } + return validateVendorTargetDir(targetDir) +} + +func resolveLocalSubdirURI(uri, baseDir string) (string, error) { + root, subdir, ok := vendor.LocalRootSubdir(uri) + if !ok { + return uri, nil + } + if err := validateLocalSubdir(subdir); err != nil { + return "", err + } + if !filepath.IsAbs(root) { + absolute, err := resolveLocalSourcePath(root, baseDir) + if err != nil { + return "", err + } + root = absolute + } + return root + "//" + subdir, nil +} + // downloadSourceWithMetadata populates tempDir from uri through the shared go-getter downloader // and returns its fetch receipt. OCI sources (oci://) need no special handling: the downloader // registers an OCI getter alongside the other protocols. @@ -184,6 +211,23 @@ func effectiveRetryConfig(sourceSpec *schema.VendorComponentSource) *schema.Retr // localDirectorySource recognizes existing local directories while leaving remote URIs to the downloader. func localDirectorySource(uri string) (string, bool, error) { + if root, subdir, ok := vendor.LocalRootSubdir(uri); ok { + if err := validateLocalSubdir(subdir); err != nil { + return "", false, err + } + if _, present, err := existingDirectory(filepath.Join(root, filepath.FromSlash(subdir))); err != nil || !present { + return "", false, errUtils.Build(errUtils.ErrSourceInvalidSpec). + WithCause(err). + WithExplanation("Local module subdirectory is not a directory").Err() + } + canonicalRoot, rootErr := filepath.EvalSymlinks(root) + canonicalModule, moduleErr := filepath.EvalSymlinks(filepath.Join(root, filepath.FromSlash(subdir))) + if rootErr != nil || moduleErr != nil || !localSubdirWithin(canonicalModule, canonicalRoot) { + return "", false, errUtils.Build(errUtils.ErrSourceInvalidSpec). + WithExplanation("Local module subdirectory escapes its source root").Err() + } + return existingDirectory(root) + } switch { case vendor.IsFileURI(uri): path, err := fileURIPath(uri) @@ -198,6 +242,23 @@ func localDirectorySource(uri string) (string, bool, error) { } } +func validateLocalSubdir(subdir string) error { + if subdir == "" || filepath.IsAbs(subdir) || filepath.VolumeName(subdir) != "" || strings.HasPrefix(subdir, "/") || strings.HasPrefix(subdir, `\`) { + return errUtils.Build(errUtils.ErrSourceInvalidSpec).WithExplanation("Local module subdirectory must stay inside its source root").Err() + } + for _, part := range strings.Split(filepath.ToSlash(subdir), "/") { + if part == "" || part == "." || part == ".." { + return errUtils.Build(errUtils.ErrSourceInvalidSpec).WithExplanation("Local module subdirectory must stay inside its source root").Err() + } + } + return nil +} + +func localSubdirWithin(path, root string) bool { + rel, err := filepath.Rel(root, path) + return err == nil && rel != ".." && !strings.HasPrefix(rel, ".."+string(filepath.Separator)) +} + // validateVendorTargetDir rejects empty, parent-relative, and filesystem-root provisioning destinations. func validateVendorTargetDir(targetDir string) error { trimmed := strings.TrimSpace(targetDir) diff --git a/pkg/scheduler/adapters/terraform.go b/pkg/scheduler/adapters/terraform.go index ee049c67715..629c2400d3d 100644 --- a/pkg/scheduler/adapters/terraform.go +++ b/pkg/scheduler/adapters/terraform.go @@ -163,17 +163,28 @@ func ExecuteTerraform(ctx context.Context, opts TerraformOptions) error { if err != nil { return err } - graph, err = buildScopedTerraformGraph(opts.Stacks, leftDelim, terraformGraphNodeIDs(filtered)) + selected := terraformGraphNodeIDs(filtered) + // Discovery retains direct edges, unlike the filtered graph's contracted ordering edges. + direct := graph.Clone() + for id := range direct.Nodes { + if !selected[id] { + delete(direct.Nodes, id) + } + } + excluded := terraformExclusions(direct, opts.Info) + graph, err = buildTerraformGraphMode(opts.Stacks, leftDelim, terraformGraphMode{ + selectedNodeIDs: selected, validationSources: selected, ignoredSources: excluded, warn: log.Warn, + }) if err != nil { return fmt.Errorf("%w: %w", errUtils.ErrBuildDepGraph, err) } - addContractedTerraformEdges(graph, filtered) + addContractedTerraformEdges(graph, filtered, excluded) if graph.Size() == 0 { ui.Success("No components matched") return nil } - if err := preflightTerraformGraph(ctx, opts.AtmosConfig, graph); err != nil { + if err := preflightTerraformGraph(ctx, opts.AtmosConfig, graph, excluded); err != nil { return err } @@ -220,6 +231,7 @@ func ExecuteTerraform(ctx context.Context, opts TerraformOptions) error { disablePluginCache: disableTerraformPluginCacheForConcurrentRun(opts.Info), lockDeploySourceDir: deployVerificationUsesSourceDir(opts.AtmosConfig, opts.Info), queryAppliedAtSelection: terraformClosureRequested(opts.Info, opts.Selection), + excluded: excluded, } startTerraformCIBefore(opts.Info, graph) @@ -241,13 +253,13 @@ func ExecuteTerraform(ctx context.Context, opts TerraformOptions) error { return err } if result.Err != nil { - if skipped := skippedResultCount(result); skipped > 0 { - ui.Warningf("%d component(s) skipped after an earlier failure", skipped) + if names := skippedResultNames(result, graph); len(names) > 0 { + ui.Warningf("%d component(s) skipped after an earlier failure: %s", len(names), strings.Join(names, ", ")) } return result.Err } - if processedCount(result) == 0 { + if processedCount(result) == 0 && !hasExcludedResult(result, excluded) { ui.Success("No components matched") } if opts.Info.SubCommand == terraformSubCommandPlan && terraformPlanChanged(result) { @@ -411,6 +423,7 @@ func buildTerraformGraphWithWarnings(stacks map[string]any, leftDelim string, se type terraformGraphMode struct { selectedNodeIDs map[string]bool validationSources map[string]bool + ignoredSources map[string]string warn func(any, ...any) // collect keeps going after a dependency error and reports every one of them. collect bool @@ -471,6 +484,9 @@ func addTerraformEdges(builder *dependency.GraphBuilder, stacks map[string]any, if shouldSkipComponent(componentSection) || !mode.selects(terraformNodeID(componentName, stackName)) { return nil } + if _, excluded := mode.ignoredSources[terraformNodeID(componentName, stackName)]; excluded { + return nil + } edges := &terraformEdges{builder, targets, mode.validationSources, mode.selectedNodeIDs, leftDelim, stackName, componentName, mode.warn} errs := edges.addAll(componentSection, !mode.collect) if mode.collect { @@ -664,11 +680,14 @@ func terraformSelectionSeedNodeIDs(graph *dependency.Graph, info *schema.ConfigA // only re-creates edges that are declared directly between selected nodes. // Every edge in filtered whose endpoints both exist in scoped but is missing // there is copied over with the same optional flag. -func addContractedTerraformEdges(scoped, filtered *dependency.Graph) { +func addContractedTerraformEdges(scoped, filtered *dependency.Graph, excluded map[string]string) { if scoped == nil || filtered == nil { return } for _, fromID := range sortedGraphNodeIDs(filtered) { + if _, skip := excluded[fromID]; skip { + continue + } fromFiltered := filtered.Nodes[fromID] if fromFiltered == nil || scoped.Nodes[fromID] == nil { continue @@ -959,6 +978,8 @@ type TerraformDispatcher struct { // FilterTerraformGraph, and re-applying it per node would silently drop // closure-added prerequisites the query never selected. queryAppliedAtSelection bool + // excluded maps the nodes a provider's exclude block removes from this run to the reason. + excluded map[string]string } // Dispatch executes one Terraform scheduler node. @@ -970,6 +991,10 @@ func (d *TerraformDispatcher) Dispatch(ctx context.Context, node *dependency.Nod if err := ctx.Err(); err != nil { return scheduler.Result{NodeID: node.ID, Status: scheduler.StatusFailed}, err } + if reason, excluded := d.excluded[node.ID]; excluded { + ui.Info(describeExclusion(node, reason)) + return scheduler.Result{NodeID: node.ID, Status: scheduler.StatusSucceeded, Value: false}, nil + } if d.shouldSkipByQuery(node) { return scheduler.Result{NodeID: node.ID, Status: scheduler.StatusSucceeded, Value: false}, nil } @@ -1584,6 +1609,20 @@ func containsString(values []string, value string) bool { return false } +// hasExcludedResult reports an exclusion that was actually dispatched. +func hasExcludedResult(result *scheduler.AggregateResult, excluded map[string]string) bool { + if result == nil { + return false + } + for i := range result.Results { + nodeResult := &result.Results[i] + if _, ok := excluded[nodeResult.NodeID]; ok && nodeResult.Status == scheduler.StatusSucceeded { + return true + } + } + return false +} + // processedCount counts nodes that actually reached Terraform execution. func processedCount(result *scheduler.AggregateResult) int { count := 0 diff --git a/pkg/scheduler/adapters/terraform_exclude.go b/pkg/scheduler/adapters/terraform_exclude.go new file mode 100644 index 00000000000..36d472ffa01 --- /dev/null +++ b/pkg/scheduler/adapters/terraform_exclude.go @@ -0,0 +1,125 @@ +package adapters + +import ( + "fmt" + "slices" + "strings" + + "github.com/cloudposse/atmos/pkg/component/execution" + "github.com/cloudposse/atmos/pkg/dependency" + "github.com/cloudposse/atmos/pkg/perf" + "github.com/cloudposse/atmos/pkg/scheduler" + "github.com/cloudposse/atmos/pkg/schema" +) + +// terraformExcludeSetting is where a configuration provider records the exclude block of a unit: +// `settings.terragrunt.exclude` with `actions` and `exclude_dependencies`. +const ( + terraformExcludeProvider = "terragrunt" + terraformExcludeKey = "exclude" +) + +// terraformExclusions decides which nodes of a bulk run a provider's exclude block removes, with +// the reason for each. A node is excluded when its block lists the command being run, and so are +// its direct dependencies when the block sets exclude_dependencies. A component the user named +// explicitly is never excluded: bulk exclusion only narrows what a selection pulls in. +func terraformExclusions(graph *dependency.Graph, info *schema.ConfigAndStacksInfo) map[string]string { + defer perf.Track(nil, "scheduler.adapters.terraformExclusions")() + + excluded := map[string]string{} + if graph == nil || info == nil { + return excluded + } + command := terraformBulkCommand(info) + for _, id := range sortedGraphNodeIDs(graph) { + node := graph.Nodes[id] + block := terraformExcludeBlock(node) + if block == nil || !execution.ExcludeMatches(stringList(block["actions"]), command) { + continue + } + excluded[id] = fmt.Sprintf("it lists `%s`", command) + if withDependencies, _ := block["exclude_dependencies"].(bool); withDependencies { + excludeDependencies(graph, node, excluded) + } + } + for id := range excluded { + if node := graph.Nodes[id]; node != nil && slices.Contains(info.Components, node.Component) { + delete(excluded, id) + } + } + return excluded +} + +// terraformBulkCommand is the command name exclude blocks list: deploy is apply, and apply with +// -destroy is destroy. +func terraformBulkCommand(info *schema.ConfigAndStacksInfo) string { + command := info.SubCommand + if command == terraformSubCommandDeploy { + command = terraformSubCommandApply + } + return execution.CommandName(command, info.AdditionalArgsAndFlags) +} + +// excludeDependencies adds the direct dependencies of a node whose block sets exclude_dependencies. +func excludeDependencies(graph *dependency.Graph, node *dependency.Node, excluded map[string]string) { + for _, dependencyID := range sortedCopy(node.Dependencies) { + if graph.Nodes[dependencyID] == nil { + continue + } + if _, already := excluded[dependencyID]; !already { + excluded[dependencyID] = fmt.Sprintf("a dependency of `%s` in `%s`, whose block sets exclude_dependencies", node.Component, node.Stack) + } + } +} + +// terraformExcludeBlock returns the provider's exclude section of a node, or nil. +func terraformExcludeBlock(node *dependency.Node) map[string]any { + if node == nil { + return nil + } + settings, _ := node.Metadata["settings"].(map[string]any) + provider, _ := settings[terraformExcludeProvider].(map[string]any) + block, _ := provider[terraformExcludeKey].(map[string]any) + return block +} + +func stringList(value any) []string { + switch typed := value.(type) { + case []string: + return typed + case []any: + list := make([]string, 0, len(typed)) + for _, item := range typed { + if text, ok := item.(string); ok { + list = append(list, text) + } + } + return list + } + return nil +} + +// skippedResultNames names the nodes the scheduler skipped, sorted, as "`component` in `stack`". +func skippedResultNames(result *scheduler.AggregateResult, graph *dependency.Graph) []string { + if result == nil { + return nil + } + var names []string + for i := range result.Results { + if result.Results[i].Status != scheduler.StatusSkipped { + continue + } + id := result.Results[i].NodeID + if node := graph.Nodes[id]; node != nil { + id = fmt.Sprintf("`%s` in `%s`", node.Component, node.Stack) + } + names = append(names, id) + } + slices.Sort(names) + return names +} + +// describeExclusion is the line printed for a node a bulk run leaves out. +func describeExclusion(node *dependency.Node, reason string) string { + return fmt.Sprintf("Skipped `%s` in `%s`: excluded by the Terragrunt `exclude` block (%s)", node.Component, node.Stack, strings.TrimSpace(reason)) +} diff --git a/pkg/scheduler/adapters/terraform_preflight.go b/pkg/scheduler/adapters/terraform_preflight.go index 6fc6c0f18c8..88f345e026e 100644 --- a/pkg/scheduler/adapters/terraform_preflight.go +++ b/pkg/scheduler/adapters/terraform_preflight.go @@ -11,13 +11,16 @@ import ( // preflightTerraformGraph rejects selected components that their configuration provider could not // evaluate, before any component runs. Without it a bulk run would apply the healthy components // that sort first and only then fail on the broken one. Every broken component is reported. -func preflightTerraformGraph(ctx context.Context, atmosConfig *schema.AtmosConfiguration, graph *dependency.Graph) error { +func preflightTerraformGraph(ctx context.Context, atmosConfig *schema.AtmosConfiguration, graph *dependency.Graph, excluded map[string]string) error { checker, ok := atmosConfig.ComponentConfigurations.(schema.ComponentConfigurationFailureRuntime) if !ok { return nil } var failures []error for _, id := range sortedGraphNodeIDs(graph) { + if _, skip := excluded[id]; skip { + continue + } node := graph.Nodes[id] if node == nil { continue diff --git a/pkg/scheduler/adapters/terraform_preflight_test.go b/pkg/scheduler/adapters/terraform_preflight_test.go index 462a546ad8d..7000f6e18d3 100644 --- a/pkg/scheduler/adapters/terraform_preflight_test.go +++ b/pkg/scheduler/adapters/terraform_preflight_test.go @@ -61,7 +61,7 @@ func TestPreflightRejectsEveryBrokenComponentBeforeAnyRuns(t *testing.T) { config := &schema.AtmosConfiguration{ComponentConfigurations: runtime.mock(t)} graph := graphOf(t, [2]string{"live", "a"}, [2]string{"live", "b"}, [2]string{"live", "c"}) - err := preflightTerraformGraph(t.Context(), config, graph) + err := preflightTerraformGraph(t.Context(), config, graph, nil) require.ErrorIs(t, err, errUtils.ErrPluginComponentsFailed) assert.ErrorContains(t, err, "first") assert.ErrorContains(t, err, "second") @@ -70,8 +70,18 @@ func TestPreflightRejectsEveryBrokenComponentBeforeAnyRuns(t *testing.T) { func TestPreflightIgnoresHealthyGraphsAndProvidersWithoutFailures(t *testing.T) { graph := graphOf(t, [2]string{"live", "a"}) - require.NoError(t, preflightTerraformGraph(t.Context(), &schema.AtmosConfiguration{}, graph)) - require.NoError(t, preflightTerraformGraph(t.Context(), &schema.AtmosConfiguration{ComponentConfigurations: (&failureRuntime{}).mock(t)}, graph)) + require.NoError(t, preflightTerraformGraph(t.Context(), &schema.AtmosConfiguration{}, graph, nil)) + require.NoError(t, preflightTerraformGraph(t.Context(), &schema.AtmosConfiguration{ComponentConfigurations: (&failureRuntime{}).mock(t)}, graph, nil)) +} + +func TestPreflightSkipsExcludedButStillRejectsIncludedFailures(t *testing.T) { + runtime := &failureRuntime{failures: map[string]error{"live/a": errors.New("excluded"), "live/b": errors.New("included")}} + config := &schema.AtmosConfiguration{ComponentConfigurations: runtime.mock(t)} + graph := graphOf(t, [2]string{"live", "a"}, [2]string{"live", "b"}) + err := preflightTerraformGraph(t.Context(), config, graph, map[string]string{terraformNodeID("a", "live"): "excluded"}) + require.ErrorContains(t, err, "included") + require.NotContains(t, runtime.asked, "live/a") + require.Contains(t, runtime.asked, "live/b") } func TestExecuteTerraformPreflightRejectsFailuresBeforeDispatch(t *testing.T) { diff --git a/pkg/scheduler/adapters/terraform_test.go b/pkg/scheduler/adapters/terraform_test.go index 73f19739329..8f2077fdef8 100644 --- a/pkg/scheduler/adapters/terraform_test.go +++ b/pkg/scheduler/adapters/terraform_test.go @@ -20,10 +20,12 @@ import ( errUtils "github.com/cloudposse/atmos/errors" cfg "github.com/cloudposse/atmos/pkg/config" "github.com/cloudposse/atmos/pkg/dependency" + iolib "github.com/cloudposse/atmos/pkg/io" provWorkdir "github.com/cloudposse/atmos/pkg/provisioner/workdir" "github.com/cloudposse/atmos/pkg/scheduler" "github.com/cloudposse/atmos/pkg/schema" tfcache "github.com/cloudposse/atmos/pkg/terraform/cache" + "github.com/cloudposse/atmos/pkg/ui" ) func TestExecuteTerraformSharesRegistryCacheAcrossBulkRun(t *testing.T) { @@ -3651,3 +3653,151 @@ func TestTerraformOutputGroupsNonPlanSubcommands(t *testing.T) { }) } } + +// excludeStacks marks components of the dev test stack with a provider exclude block. +func excludeStacks(blocks map[string]map[string]any) map[string]any { + stacks := terraformAdapterTestStacks() + components := stacks["dev"].(map[string]any)[cfg.ComponentsSectionName].(map[string]any)[cfg.TerraformSectionName].(map[string]any) + for name, block := range blocks { + components[name].(map[string]any)[cfg.SettingsSectionName] = map[string]any{"terragrunt": map[string]any{"exclude": block}} + } + return stacks +} + +func runExcluded(t *testing.T, stacks map[string]any, info *schema.ConfigAndStacksInfo) []string { + t.Helper() + var executed []string + require.NoError(t, ExecuteTerraform(context.Background(), TerraformOptions{ + AtmosConfig: &schema.AtmosConfiguration{}, Info: info, Stacks: stacks, + Executor: func(execution TerraformExecution) (TerraformExecutionResult, error) { + executed = append(executed, execution.Info.Component) + return TerraformExecutionResult{}, nil + }, + })) + return executed +} + +func TestExecuteTerraformSkipsComponentsExcludedForTheCommand(t *testing.T) { + stacks := excludeStacks(map[string]map[string]any{"vpc": {"actions": []any{"plan"}}}) + require.Equal(t, []string{"database", "app"}, runExcluded(t, stacks, &schema.ConfigAndStacksInfo{All: true, SubCommand: "plan"}), + "the excluded component is skipped and the rest keep their order") + require.Equal(t, []string{"vpc", "database", "app"}, runExcluded(t, stacks, &schema.ConfigAndStacksInfo{All: true, SubCommand: "apply"}), + "a command the block does not list runs everything") +} + +func TestExecuteTerraformExcludeDependenciesExcludesDirectDependencies(t *testing.T) { + stacks := excludeStacks(map[string]map[string]any{"app": {"actions": []string{"all"}, "exclude_dependencies": true}}) + require.Equal(t, []string{"vpc"}, runExcluded(t, stacks, &schema.ConfigAndStacksInfo{All: true, SubCommand: "destroy"}), + "app and its direct dependency database are excluded; vpc is two levels away") +} + +func TestExecuteTerraformNamedComponentsAreNeverExcluded(t *testing.T) { + stacks := excludeStacks(map[string]map[string]any{"vpc": {"actions": []any{"all"}}, "database": {"actions": []any{"all_except_output"}}}) + require.Equal(t, []string{"vpc", "database"}, runExcluded(t, stacks, &schema.ConfigAndStacksInfo{Components: []string{"vpc", "database"}, SubCommand: "plan"})) +} + +func TestExecuteTerraformDeployMatchesApplyExclusions(t *testing.T) { + stacks := excludeStacks(map[string]map[string]any{"app": {"actions": []any{"apply"}}}) + require.Equal(t, []string{"vpc", "database"}, runExcluded(t, stacks, &schema.ConfigAndStacksInfo{All: true, SubCommand: "deploy"})) +} + +func TestTerraformExclusionsReasonsAndEdges(t *testing.T) { + graph, err := BuildTerraformGraph(excludeStacks(map[string]map[string]any{ + "app": {"actions": []any{"plan"}, "exclude_dependencies": true}, + })) + require.NoError(t, err) + excluded := terraformExclusions(graph, &schema.ConfigAndStacksInfo{SubCommand: "plan"}) + require.Len(t, excluded, 2) + require.Contains(t, excluded[terraformNodeID("app", "dev")], "`plan`") + require.Contains(t, excluded[terraformNodeID("database", "dev")], "dependency of `app` in `dev`") + require.Empty(t, terraformExclusions(nil, &schema.ConfigAndStacksInfo{})) + require.Empty(t, terraformExclusions(graph, nil)) + require.Empty(t, terraformExclusions(graph, &schema.ConfigAndStacksInfo{SubCommand: "apply"})) + require.Equal(t, "Skipped `app` in `dev`: excluded by the Terragrunt `exclude` block (it lists `plan`)", describeExclusion(graph.Nodes[terraformNodeID("app", "dev")], " it lists `plan` ")) + require.Equal(t, []string{"a"}, stringList([]any{"a", 1})) + require.Equal(t, []string{"a"}, stringList([]string{"a"})) + require.Nil(t, stringList("a")) +} + +func TestSkippedResultNamesNameTheComponents(t *testing.T) { + graph, err := BuildTerraformGraph(terraformAdapterTestStacks()) + require.NoError(t, err) + result := &scheduler.AggregateResult{Results: []scheduler.Result{ + {NodeID: terraformNodeID("vpc", "dev"), Status: scheduler.StatusSucceeded}, + {NodeID: terraformNodeID("database", "dev"), Status: scheduler.StatusSkipped}, + {NodeID: "unknown-node", Status: scheduler.StatusSkipped}, + {NodeID: terraformNodeID("app", "dev"), Status: scheduler.StatusSkipped}, + }} + require.Equal(t, []string{"`app` in `dev`", "`database` in `dev`", "unknown-node"}, skippedResultNames(result, graph)) + require.Nil(t, skippedResultNames(nil, graph)) +} + +func TestExecuteTerraformExcludeDependenciesIgnoresContractedEdges(t *testing.T) { + stacks := excludeStacks(map[string]map[string]any{"app": {"actions": []any{"plan"}, "exclude_dependencies": true}}) + components := stacks["dev"].(map[string]any)[cfg.ComponentsSectionName].(map[string]any)[cfg.TerraformSectionName].(map[string]any) + for _, name := range []string{"app", "vpc"} { + components[name].(map[string]any)[cfg.MetadataSectionName].(map[string]any)["tags"] = []any{"edge"} + } + // The tag filter drops database, so app's ordering edge is contracted onto vpc. vpc is still + // two levels away from app and must not be excluded as if it were a direct dependency. + var executed []string + require.NoError(t, ExecuteTerraform(context.Background(), TerraformOptions{ + AtmosConfig: &schema.AtmosConfiguration{}, Info: &schema.ConfigAndStacksInfo{Tags: []string{"edge"}, SubCommand: "plan"}, Stacks: stacks, + Selection: &TerraformSelection{NodeIDs: []string{"vpc-dev"}, IncludeDependents: true}, + Executor: func(execution TerraformExecution) (TerraformExecutionResult, error) { + executed = append(executed, execution.Info.Component) + return TerraformExecutionResult{}, nil + }, + })) + require.Equal(t, []string{"vpc"}, executed) +} + +func TestExecuteTerraformAllExcludedCIResults(t *testing.T) { + t.Setenv("NO_COLOR", "1") + ioCtx, err := iolib.NewContext() + require.NoError(t, err) + ui.InitFormatter(ioCtx) + t.Cleanup(ui.Reset) + var output bytes.Buffer + t.Cleanup(iolib.PushUIWriter(&output)) + handler := &capturingTerraformPlanCIResultHandler{} + stacks := excludeStacks(map[string]map[string]any{ + "vpc": {"actions": []any{"plan"}}, "database": {"actions": []any{"plan"}}, "app": {"actions": []any{"plan"}}, + }) + executed := runExcluded(t, stacks, &schema.ConfigAndStacksInfo{All: true, SubCommand: "plan", TerraformPlanCIResultHandler: handler}) + require.Empty(t, executed) + require.Contains(t, output.String(), "Skipped") + require.NotContains(t, output.String(), "No components matched") + require.Len(t, handler.resultSet.Results, 3) + for _, result := range handler.resultSet.Results { + require.Equal(t, "succeeded", result.Status) + require.False(t, result.Processed) + } +} + +func TestExecuteTerraformExclusionPrecedesValidationAndPreflight(t *testing.T) { + stacks := excludeStacks(map[string]map[string]any{"app": {"actions": []any{"plan"}}}) + components := stacks["dev"].(map[string]any)[cfg.ComponentsSectionName].(map[string]any)[cfg.TerraformSectionName].(map[string]any) + components["app"].(map[string]any)[cfg.DependenciesSectionName] = map[string]any{"components": []any{map[string]any{"name": "missing", "required": true}}} + runtime := &failureRuntime{failures: map[string]error{"dev/app": errors.New("excluded failure")}} + var executed []string + err := ExecuteTerraform(context.Background(), TerraformOptions{ + AtmosConfig: &schema.AtmosConfiguration{ComponentConfigurations: runtime.mock(t)}, + Info: &schema.ConfigAndStacksInfo{All: true, SubCommand: "plan"}, Stacks: stacks, + Executor: func(execution TerraformExecution) (TerraformExecutionResult, error) { + executed = append(executed, execution.Info.Component) + return TerraformExecutionResult{}, nil + }, + }) + require.NoError(t, err) + require.Equal(t, []string{"vpc", "database"}, executed) + require.NotContains(t, runtime.asked, "dev/app") + + // The same invalid dependency must still fail when its source is included. + info := &schema.ConfigAndStacksInfo{All: true, SubCommand: "apply"} + err = ExecuteTerraform(context.Background(), TerraformOptions{ + AtmosConfig: &schema.AtmosConfiguration{}, Info: info, Stacks: stacks, + Executor: func(TerraformExecution) (TerraformExecutionResult, error) { return TerraformExecutionResult{}, nil }, + }) + require.ErrorContains(t, err, "missing") +} diff --git a/pkg/schema/backend_generation.go b/pkg/schema/backend_generation.go new file mode 100644 index 00000000000..bf0c27cf0b0 --- /dev/null +++ b/pkg/schema/backend_generation.go @@ -0,0 +1,31 @@ +package schema + +// The typed value cannot be forged by a YAML component section. +const backendFileGenerationKey = "_backend_file_generation" + +type backendGenerationChoice struct{ enabled bool } + +// SetBackendFileGeneration stores the prepared choice on this invocation and +// its section for provisioners that receive only the section. +func (info *ConfigAndStacksInfo) SetBackendFileGeneration(enabled bool) { + info.BackendFileGeneration = &enabled + info.ComponentSection[backendFileGenerationKey] = backendGenerationChoice{enabled: enabled} +} + +// GenerateBackendFile returns the prepared choice when present, otherwise the +// project default. A nil info covers read-only and early-return paths. +func GenerateBackendFile(config *AtmosConfiguration, info *ConfigAndStacksInfo) bool { + if info != nil && info.BackendFileGeneration != nil { + return *info.BackendFileGeneration + } + return config.Components.Terraform.AutoGenerateBackendFile +} + +// GenerateBackendFileForSection returns a prepared choice carried on a local +// section or the project default for fresh stack descriptions. +func GenerateBackendFileForSection(config *AtmosConfiguration, section map[string]any) bool { + if choice, ok := section[backendFileGenerationKey].(backendGenerationChoice); ok { + return choice.enabled + } + return config.Components.Terraform.AutoGenerateBackendFile +} diff --git a/pkg/schema/component_configuration.go b/pkg/schema/component_configuration.go index 232751fa3d8..e767dcb93e1 100644 --- a/pkg/schema/component_configuration.go +++ b/pkg/schema/component_configuration.go @@ -36,6 +36,12 @@ type ComponentConfigurationRuntime interface { Prepare(context.Context, *ConfigAndStacksInfo) (*PreparedComponentConfiguration, error) } +// ComponentConfigurationIAMProbe is implemented by providers whose configuration +// needs role credentials while it is being evaluated. +type ComponentConfigurationIAMProbe interface { + ProbeIAM(context.Context, *ConfigAndStacksInfo) (*ComponentIAM, error) +} + // ComponentConfigurationFailureRuntime reports components that a provider discovered but could not // evaluate. Bulk commands use it to reject every selected broken component before they run any // infrastructure command. A nil error means the component is healthy or unknown to the provider. @@ -68,10 +74,58 @@ type PreparedComponentConfiguration struct { BackendGenerated bool // Execution is data the host runs; it is nil when the provider adds no hooks or arguments. Execution *ComponentExecution + // TerraformBinary is the executable the unit selects, verified against the one it was discovered with. + TerraformBinary string + // Exclusion is set when the provider says this operation must not run the unit. + Exclusion *ComponentExclusion + // IgnoredRun is set when an ignore policy matched a failure that left the unit without a resolved + // source, such as a registry resolution failure. The host ends the run without executing the unit. + IgnoredRun *ComponentIgnoredRun + // FeatureFlags are the evaluated feature values of the unit, rendered as strings. + FeatureFlags map[string]string + // IAM is a role the host assumes for the component's commands; it is nil when the provider asks for none. + IAM *ComponentIAM + // SourceAuthorization is invocation-only registry auth for a same-origin archive. + SourceAuthorization *SourceAuthorization `json:"-"` // SensitiveInputKeys projects provider sensitivity paths onto whole input variables. SensitiveInputKeys map[string]bool } +// SourceAuthorization is never serialized with a source URL or saved plan. +type SourceAuthorization struct { + Origin string + Header string +} + +// ComponentIAM is an AWS role a configuration provider asks the host to assume. The host assumes it +// on top of the invocation's credentials and exports the resulting session to the component's commands. +type ComponentIAM struct { + Role string + SessionName string + DurationSeconds uint32 + // WebIdentityTokenFile holds the token. Saved-plan bindings keep only this path. + WebIdentityTokenFile string + // WebIdentityToken is a literal token. It is a secret and is never serialized. + WebIdentityToken string `json:"-"` + // WebIdentityTokenLiteral records that the token was a literal value, so a restored record that + // has lost it can say why instead of assuming the role without a token. + WebIdentityTokenLiteral bool +} + +// ComponentExclusion records that a unit's exclude block prevents the operation from running. +type ComponentExclusion struct { + // Actions lists the commands the exclusion applies to. + Actions []string +} + +// ComponentIgnoredRun names the ignore policy that ended preparation of a unit. +type ComponentIgnoredRun struct { + // Policy is the name of the matching ignore policy. + Policy string + // Message is the message of the matching ignore policy. + Message string +} + // ComponentConfigurationFile must bypass template rendering. type ComponentConfigurationFile struct { Path string diff --git a/pkg/schema/component_configuration_test.go b/pkg/schema/component_configuration_test.go index 0881ae039e6..6d2b4fd977a 100644 --- a/pkg/schema/component_configuration_test.go +++ b/pkg/schema/component_configuration_test.go @@ -50,6 +50,19 @@ func TestComponentDiscoveryRequiresBothOptIns(t *testing.T) { require.False(t, (&AtmosConfiguration{}).HasComponentDiscovery()) } +func TestComponentExecutionIsEmptyCountsErrorHandling(t *testing.T) { + var none *ComponentExecution + if !none.IsEmpty() || !(&ComponentExecution{}).IsEmpty() { + t.Fatal("an absent or zero execution adds nothing") + } + if (&ComponentExecution{Errors: &ComponentErrors{}}).IsEmpty() { + t.Fatal("error handling alone is execution metadata") + } + if (&ComponentExecution{BeforeHooks: []ComponentHook{{Name: "h"}}}).IsEmpty() { + t.Fatal("a hook is execution metadata") + } +} + func TestConfigurationOwned(t *testing.T) { for _, test := range []struct { name string diff --git a/pkg/schema/component_execution.go b/pkg/schema/component_execution.go index 65716eed1ee..1e046407111 100644 --- a/pkg/schema/component_execution.go +++ b/pkg/schema/component_execution.go @@ -1,5 +1,7 @@ package schema +import "time" + // ComponentExecution is evaluated command metadata returned by a configuration provider. // The host executes it with the invocation's environment, toolchain, working directory, // cancellation, and I/O; the provider never runs these commands. Lists keep declaration order. @@ -8,12 +10,40 @@ type ComponentExecution struct { BeforeHooks []ComponentHook AfterHooks []ComponentHook ErrorHooks []ComponentHook + // Errors holds the retry and ignore policies applied to the Terraform subprocesses of a component. + Errors *ComponentErrors // Apply is the metadata the provider evaluated for the apply command while preparing a // plan. A saved plan applies it instead of the plan-time metadata, as Terragrunt would // evaluate the configuration again for apply. Nil means the plan-time metadata applies. Apply *ComponentExecution } +// ComponentErrors is the provider's error handling policy: ignore policies are checked first, then +// retry policies, each in declaration order. +type ComponentErrors struct { + Retry []ComponentRetryPolicy + Ignore []ComponentIgnorePolicy +} + +// ComponentRetryPolicy retries a failed command whose output matches a pattern. A pattern that +// starts with "!" vetoes the policy when it matches. +type ComponentRetryPolicy struct { + Name string + Patterns []string + // MaxAttempts counts the first attempt. + MaxAttempts int + SleepInterval time.Duration +} + +// ComponentIgnorePolicy turns a failed command whose output matches a pattern into a success. +type ComponentIgnorePolicy struct { + Name string + Patterns []string + Message string + // Signals are written as JSON to a file next to the component's module when the policy applies. + Signals map[string]any +} + // ComponentExtraArguments adds arguments, variable files, and environment to the listed commands. type ComponentExtraArguments struct { Commands []string @@ -48,5 +78,5 @@ func (e *ComponentExecution) ForSavedPlanApply() *ComponentExecution { // IsEmpty reports whether the execution adds nothing to a command. func (e *ComponentExecution) IsEmpty() bool { - return e == nil || len(e.ExtraArguments)+len(e.BeforeHooks)+len(e.AfterHooks)+len(e.ErrorHooks) == 0 + return e == nil || (len(e.ExtraArguments)+len(e.BeforeHooks)+len(e.AfterHooks)+len(e.ErrorHooks) == 0 && e.Errors == nil) } diff --git a/pkg/schema/schema.go b/pkg/schema/schema.go index 58dfb9166ee..52ce564d17c 100644 --- a/pkg/schema/schema.go +++ b/pkg/schema/schema.go @@ -2146,6 +2146,11 @@ type ConfigAndStacksInfo struct { ComponentEnvSection AtmosSectionMapType ComponentAuthSection AtmosSectionMapType ComponentEnvList []string + // SourceAuthorization is an invocation-only header for a resolved registry archive. + SourceAuthorization *SourceAuthorization `json:"-" yaml:"-" mapstructure:"-"` + // IAMSession is the invocation-only AWS session of a role the component configuration assumed. + // In-process source downloads use it so they run as the same principal as the component's commands. + IAMSession map[string]string `json:"-" yaml:"-" mapstructure:"-"` // TerraformSecretVarKeys holds the set of top-level keys in ComponentVarsSection // whose value contains a resolved secret (detected via the masker). These variables // are excluded from the on-disk Terraform varfile and injected at runtime as @@ -2161,6 +2166,9 @@ type ConfigAndStacksInfo struct { // which would reintroduce problematic vars (e.g., IRSA credentials on EKS pods). SanitizedEnv []string ComponentBackendSection AtmosSectionMapType + // BackendFileGeneration is the prepared component's invocation-local choice. + // Nil retains the project-wide auto_generate_backend_file default. + BackendFileGeneration *bool `json:"-" yaml:"-" mapstructure:"-"` // ComponentRetrySection holds the per-component `retry:` block parsed from stack // manifests. When non-nil and `Conditions` is populated, each subprocess invocation // in the terraform execution pipeline (init, workspace, main) is retried independently diff --git a/pkg/terraform/backend/file.go b/pkg/terraform/backend/file.go index ae58396ed79..a30f7d96c49 100644 --- a/pkg/terraform/backend/file.go +++ b/pkg/terraform/backend/file.go @@ -17,7 +17,7 @@ func GenerateFile(atmosConfig *schema.AtmosConfiguration, info *schema.ConfigAnd defer perf.Track(atmosConfig, "backend.GenerateFile")() // Terraform tests use their own state and do not need a generated backend. - if info.SubCommand == "test" || !atmosConfig.Components.Terraform.AutoGenerateBackendFile { + if info.SubCommand == "test" || !schema.GenerateBackendFile(atmosConfig, info) { return nil } // Auto-generate backend file. diff --git a/pkg/terraform/clean/clean.go b/pkg/terraform/clean/clean.go index 46cfaa595f2..4ec52283e7e 100644 --- a/pkg/terraform/clean/clean.go +++ b/pkg/terraform/clean/clean.go @@ -310,7 +310,7 @@ func (s *Service) initializeFilesToClear(info schema.ConfigAndStacksInfo, atmosC files = append(files, TerraformLockFile) } - if atmosConfig.Components.Terraform.AutoGenerateBackendFile { + if schema.GenerateBackendFile(atmosConfig, &info) { files = append(files, BackendConfigFile) } diff --git a/pkg/terraform/clean/clean_test.go b/pkg/terraform/clean/clean_test.go index a66d582d5e9..7c10ff0a6ed 100644 --- a/pkg/terraform/clean/clean_test.go +++ b/pkg/terraform/clean/clean_test.go @@ -266,6 +266,7 @@ func TestBuildRelativePath(t *testing.T) { // TestInitializeFilesToClear tests the initializeFilesToClear method. func TestInitializeFilesToClear(t *testing.T) { + providerOwned := false mock := &mockStackProcessor{ constructVarfileNameFn: func(info *schema.ConfigAndStacksInfo) string { return "test-stack.tfvars.json" @@ -326,6 +327,15 @@ func TestInitializeFilesToClear(t *testing.T) { }, expectedContains: []string{".terraform", "backend.tf.json"}, }, + { + name: "Prepared provider backend overrides project default", + info: schema.ConfigAndStacksInfo{ + ComponentFromArg: "vpc", + BackendFileGeneration: &providerOwned, + }, + atmosConfig: &schema.AtmosConfiguration{Components: schema.Components{Terraform: schema.Terraform{AutoGenerateBackendFile: true}}}, + expectedNotContains: []string{"backend.tf.json"}, + }, { name: "With auto generate files enabled", info: schema.ConfigAndStacksInfo{ diff --git a/pkg/terraform/output/config.go b/pkg/terraform/output/config.go index d51e48ee858..c85cb0b7c43 100644 --- a/pkg/terraform/output/config.go +++ b/pkg/terraform/output/config.go @@ -260,7 +260,7 @@ func extractOptionalFields(sections map[string]any, config *ComponentConfig) { func ExtractComponentConfig(atmosConfig *schema.AtmosConfiguration, sections map[string]any, component, stack string) (*ComponentConfig, error) { defer perf.Track(atmosConfig, "output.ExtractComponentConfig")() - config := newComponentConfig(atmosConfig) + config := newComponentConfig(atmosConfig, sections) if err := extractRequiredFields(atmosConfig, sections, component, stack, config); err != nil { return nil, err } @@ -274,7 +274,7 @@ func ExtractComponentConfig(atmosConfig *schema.AtmosConfiguration, sections map // in a scratch or the unit's own initialized directory, never in the component's execution // workdir, so the workdir path is not resolved and its configuration cannot fail the read. func extractConfigurationStateConfig(atmosConfig *schema.AtmosConfiguration, sections map[string]any, component, stack string) (*ComponentConfig, error) { - config := newComponentConfig(atmosConfig) + config := newComponentConfig(atmosConfig, sections) if err := extractCommandFields(sections, component, stack, config); err != nil { return nil, err } @@ -283,9 +283,9 @@ func extractConfigurationStateConfig(atmosConfig *schema.AtmosConfiguration, sec } // newComponentConfig applies the project-wide output defaults. -func newComponentConfig(atmosConfig *schema.AtmosConfiguration) *ComponentConfig { +func newComponentConfig(atmosConfig *schema.AtmosConfiguration, sections map[string]any) *ComponentConfig { return &ComponentConfig{ - AutoGenerateBackend: atmosConfig.Components.Terraform.AutoGenerateBackendFile, + AutoGenerateBackend: schema.GenerateBackendFileForSection(atmosConfig, sections), InitMode: atmosConfig.Components.Terraform.EffectiveInitMode(), InitReconfigure: atmosConfig.Components.Terraform.EffectiveInitReconfigure(), InitUpgrade: atmosConfig.Components.Terraform.EffectiveInitUpgrade(), diff --git a/pkg/terraform/output/config_test.go b/pkg/terraform/output/config_test.go index a988dd35fc2..c3ac4881533 100644 --- a/pkg/terraform/output/config_test.go +++ b/pkg/terraform/output/config_test.go @@ -771,3 +771,25 @@ func TestExtractComponentConfig_InitPolicy(t *testing.T) { assert.Equal(t, schema.TerraformInitReconfigureAlways, config.InitReconfigure) assert.Equal(t, schema.TerraformInitUpgradeAlways, config.InitUpgrade) } + +func TestExtractComponentConfigUsesInvocationBackendChoice(t *testing.T) { + config := validAtmosConfig(t) + config.Components.Terraform.AutoGenerateBackendFile = true + sections := validSections() + info := &schema.ConfigAndStacksInfo{ComponentSection: sections} + info.SetBackendFileGeneration(false) + selected, err := ExtractComponentConfig(config, sections, "app", "dev") + require.NoError(t, err) + require.False(t, selected.AutoGenerateBackend) + state, err := extractConfigurationStateConfig(config, sections, "app", "dev") + require.NoError(t, err) + require.False(t, state.AutoGenerateBackend) + defaulted, err := ExtractComponentConfig(config, validSections(), "app", "dev") + require.NoError(t, err) + require.True(t, defaulted.AutoGenerateBackend) + forged := validSections() + forged["_backend_file_generation"] = false + defaulted, err = ExtractComponentConfig(config, forged, "app", "dev") + require.NoError(t, err) + require.True(t, defaulted.AutoGenerateBackend) +} diff --git a/pkg/vendor/uri.go b/pkg/vendor/uri.go index 9012af8bc86..bc3ac46b557 100644 --- a/pkg/vendor/uri.go +++ b/pkg/vendor/uri.go @@ -263,6 +263,26 @@ func IsLocalPath(uri string) bool { return true } +// LocalRootSubdir recognizes a local root//module source before path cleaning +// erases the go-getter delimiter. It leaves validation of the selected module +// to the caller. +func LocalRootSubdir(uri string) (root, subdir string, ok bool) { + volume := filepath.VolumeName(uri) + rest := uri[len(volume):] + if strings.ContainsAny(rest, "?:") { + return "", "", false + } + index := strings.Index(filepath.ToSlash(rest), "//") + if index <= 0 { + return "", "", false + } + root = volume + rest[:index] + if !IsLocalPath(root) { + return "", "", false + } + return root, rest[index+2:], true +} + // IsDomainLikeURI checks if the URI has a domain-like structure (hostname.domain/path). func IsDomainLikeURI(uri string) bool { dotPos := strings.Index(uri, ".") diff --git a/plugins/terragrunt/README.md b/plugins/terragrunt/README.md index a33b12e805c..33507922530 100644 --- a/plugins/terragrunt/README.md +++ b/plugins/terragrunt/README.md @@ -76,7 +76,7 @@ The host executes `extra_arguments` and `before_hook`, `after_hook`, and `error_ A saved plan is kept until it is applied. `output`, `validate`, `init` and `state list` leave it in place. A successful `apply`, `deploy` or `destroy` removes the plan and its Atmos record, so applying the same plan twice reports that no saved plan exists. Apply reuses the execution data recorded at plan time and requires the original working directory. The record holds no input values and no backend settings; Atmos compares the backend by digest at apply time. -Remote module sources (`git::https`, `git::http`, `git::ssh`, `git::file`, `https`, `http`) keep their getter URI for the Atmos source provisioner. A local module is copied into the working directory again only when its files change (path, size, or modification time); an explicit `source.ttl` takes precedence. A source that follows a branch (`?ref=main`) is fetched once and reused, and Atmos prints a notice when it reuses such a copy: run `atmos terraform source pull -s --force` to refresh it, or set `source.ttl`. Release tags and commit ids are never reported. Registry sources and remote stack catalogs are not supported. Backend configuration is adopted, including cloud backends, but Atmos does not provision the backend resources. +Remote module sources (`git::https`, `git::http`, `git::ssh`, `git::file`, `https`, `http`) keep their getter URI for the Atmos source provisioner. A local module is copied into the working directory again only when its files change (path, size, or modification time); an explicit `source.ttl` takes precedence. A source that follows a branch (`?ref=main`) is fetched once and reused, and Atmos prints a notice when it reuses such a copy: run `atmos terraform source pull -s --force` to refresh it, or set `source.ttl`. Release tags and commit ids are never reported. `tfr://` registry sources are resolved to their getter URL when the unit is prepared for a command; discovery never contacts the registry. Remote stack catalogs are not supported. Backend configuration is adopted, including cloud backends, but Atmos does not provision the backend resources. Atmos removes a generated file when its `generate` block is removed, but only while the file is unchanged or still carries the Terragrunt signature. `if_exists = "error"` accepts the file Atmos generated on an earlier run, and all generated files are mode 0600. A `generate` block whose contents declare a `backend` or `cloud` block is rejected because Atmos cannot verify its state-reader identity. Declare the backend with `remote_state` and use `remote_state.generate` instead. Backend-only state reads inspect generation without evaluating inputs or commands: Terraform generate blocks must have literal paths and contents that can be proven not to declare a backend; dynamic or attribute-form generation is rejected in this phase, including in parent configurations. Automatic includes are also rejected for state reads because the upstream partial parser does not merge them; use explicit includes for backend configuration. The `.terraform.lock.hcl` beside `terragrunt.hcl` is copied into the working directory when it changes; Atmos does not copy changes back, so copy `.workdir/.../.terraform.lock.hcl` yourself after `init -upgrade`. @@ -84,11 +84,13 @@ A unit without a stable `remote_state` path keeps local state under `.terragrunt `dependencies.tools` is read from stack manifests and component sections, not from `atmos.yaml`. An imported unit takes `dependencies.tools` from the importing stack's configuration, and those pins take precedence over `.tool-versions`. A discovered unit has no stack manifest, so it takes its tool pins from `.tool-versions`. -Unsupported semantics fail before any infrastructure command and never switch executors. Currently rejected: `terraform_binary`; `errors`, `exclude`, feature flags, and IAM role or web-identity settings; `remote_state` with `disable_init` or encryption; inline modules without `terraform.source`; output dependencies with a non-default workspace during discovery; the `read-config`, `terragrunt-read-config`, and `init-from-module` hook lifecycles; hooks that run a built-in integration such as `tflint`; and an invalid `on_errors` expression. Executable mode, typed provider flags and help, exact cty types and sensitivity, scoped Terragrunt logging, migration and setup generation, and Atmos Pro integration remain in the PRDs. +Unsupported semantics fail before any infrastructure command and never switch executors. Currently rejected: inline modules without `terraform.source`; output dependencies with a non-default workspace during discovery; stack-level overrides of a unit's backend, `metadata.component`, or `source.uri`; and an invalid `on_errors` expression. Not implemented: Terragrunt's built-in default retry list, `--no-auto-retry`, and `errors` handling around `read-config` hooks. Executable mode, typed provider flags and help, exact cty types and sensitivity, scoped Terragrunt logging, migration and setup generation, and Atmos Pro integration remain in the PRDs. + +`terraform_binary` is published as the unit's `command`, so precedence is stack `command`, then the unit's `terraform_binary`, then `components.terraform.command`; the host errors if Prepare returns a different binary. `errors` policies wrap each Terraform subprocess phase and the source fetch in the host (`pkg/component/execution/recovery.go`). `exclude` is read from `settings.terragrunt.exclude` by the scheduler (`pkg/scheduler/adapters/terraform_exclude.go`); `no_run` is applied by the plugin for the current operation because the protocol carries no `no_run` field. A built-in `tflint` hook expands into `tflint --init` and a `tflint` run. IAM roles are assumed in the host through a transient `aws/assume-role` identity (`pkg/auth/identities/aws/inline_role.go`). AWS SDK calls evaluated inside the plugin read only the plugin's process environment and static keys from the invocation, so file-based Atmos identities are not yet visible to `get_aws_account_id()` and similar functions. `tfr://` sources resolve through the upstream registry client (`bridge/registry.go`). Unsupported settings fail with an error that names the unit, the file, and the setting, plus the line and column when the evaluator reports a position (settings rejected after decoding, such as `terraform_binary`, name only the file). A broken unit does not affect other units or YAML components: it stays listed, and only commands that target it fail. `apply`, `plan`, and `destroy` with `--all` validate every selected unit first and fail with "no infrastructure command was run". Evaluation errors can contain secrets, so Atmos shows the file, line, unit, and a hint but not the evaluator's text; run with `ATMOS_LOGS_LEVEL=Debug` to log the full message with secrets masked. -Plugin settings are validated when first used, not when `atmos.yaml` loads, so `atmos config set` can always repair them. Unknown keys under `components.terraform.terragrunt` and invalid `mode` values fail with the key name. With `discovery.enabled: true` and no `included_paths`, discovery covers every unit under the project root. Only enabled/mode/provider/discovery settings are accepted by this package. Additional plugin settings and typed CLI option handling remain planned. Discovery that needs command execution, decryption, network access, or identity-dependent effects fails. The native plugin is trusted code, not an OS sandbox. +Plugin settings are validated when first used, not when `atmos.yaml` loads, so `atmos config set` can always repair them. Unknown keys under `components.terraform.terragrunt` and invalid `mode` values fail with the key name. With `discovery.enabled: true` and no `included_paths`, discovery covers every unit under the project root. Only the enabled, mode, provider, discovery, and features settings are accepted by this package. Additional plugin settings and typed CLI option handling remain planned. Discovery that needs command execution, decryption, network access, or identity-dependent effects fails. The native plugin is trusted code, not an OS sandbox. ## Maintained upstream facade diff --git a/plugins/terragrunt/bridge/go.mod b/plugins/terragrunt/bridge/go.mod index 8ffaf34646a..d8f0da01c41 100644 --- a/plugins/terragrunt/bridge/go.mod +++ b/plugins/terragrunt/bridge/go.mod @@ -5,6 +5,7 @@ go 1.27.3 require ( github.com/gruntwork-io/terragrunt v1.2.1 github.com/spf13/afero v1.15.0 + github.com/stretchr/testify v1.12.1 ) require ( diff --git a/plugins/terragrunt/bridge/registry.go b/plugins/terragrunt/bridge/registry.go new file mode 100644 index 00000000000..987013162c5 --- /dev/null +++ b/plugins/terragrunt/bridge/registry.go @@ -0,0 +1,195 @@ +package atmosbridge + +import ( + "context" + "errors" + "fmt" + "io" + "net/http" + "net/url" + "path" + "strings" + "sync" + "time" + + "github.com/gruntwork-io/terragrunt/internal/getter" + "github.com/gruntwork-io/terragrunt/internal/tfimpl" + "github.com/gruntwork-io/terragrunt/pkg/log" +) + +// registryTimeout bounds each registry request when the caller supplies no client. +const registryTimeout = 30 * time.Second + +var ( + // ErrInvalidRegistrySource identifies a source that cannot use the tfr getter. + ErrInvalidRegistrySource = errors.New("invalid Terraform registry source") + // ErrMalformedRegistryVersion identifies an invalid exact version query. + ErrMalformedRegistryVersion = errors.New("malformed Terraform registry version") + // ErrRegistryVersionConstraint identifies a constraint in an exact-version query. + ErrRegistryVersionConstraint = errors.New("Terraform registry version constraint requires the terraform block") + // ErrRegistryLookup identifies a failure of the registry protocol: service discovery, version + // lookup, the X-Terraform-Get header, or the download URL it names. The upstream error is kept. + ErrRegistryLookup = errors.New("Terraform registry lookup failed") +) + +// RegistryRequest describes one `tfr://` module source to resolve. +type RegistryRequest struct { + // Client issues the registry requests. A nil client uses the default transport with a timeout. + Client *http.Client + // Environment supplies TG_TF_DEFAULT_REGISTRY_HOST, TG_TF_REGISTRY_TOKEN, and the variables that locate + // the user's CLI configuration, such as TF_TOKEN_ and TF_CLI_CONFIG_FILE. + Environment map[string]string + Source string + // Constraint is the terraform block's `version` constraint. It applies when the source has no `?version=`. + Constraint string + // OpenTofu selects the default registry host and the CLI configuration files that supply credentials. + OpenTofu bool +} + +// ResolveRegistrySource turns a `tfr://` source into the getter URL the registry's download endpoint names, using +// Terragrunt's own implementation of the Terraform module registry protocol: service discovery, version +// selection, and the X-Terraform-Get header. A `//subdir` on the module path is appended to the result. +func ResolveRegistrySource(ctx context.Context, logger log.Logger, request RegistryRequest) (string, error) { + resolution, err := ResolveRegistrySourceWithAuth(ctx, logger, request) + return resolution.Source, err +} + +// RegistryResolution carries same-origin archive authorization separately from the source URL. +// Authorization must remain secret and be sent only to requests for AuthOrigin. +type RegistryResolution struct { + Source string + Authorization string + AuthOrigin string +} + +// ResolveRegistrySourceWithAuth also returns the registry authorization needed by a +// same-origin archive. It never embeds credentials in the getter URL. +func ResolveRegistrySourceWithAuth(ctx context.Context, logger log.Logger, request RegistryRequest) (RegistryResolution, error) { + source, err := url.Parse(request.Source) + if err != nil { + return RegistryResolution{}, fmt.Errorf("%w %q: %w", ErrInvalidRegistrySource, request.Source, err) + } + if source.Scheme != getter.SchemeTFR { + return RegistryResolution{}, fmt.Errorf("%w: %q is not a tfr:// source", ErrInvalidRegistrySource, request.Source) + } + implementation := tfimpl.Terraform + if request.OpenTofu { + implementation = tfimpl.OpenTofu + } + client := request.Client + if client == nil { + client = &http.Client{Timeout: registryTimeout} + } + // Capture the exact header the registry lookup used. This honors Terraform's + // CLI credential precedence without making a second, potentially different lookup. + captured := ®istryAuthCapture{base: client.Transport} + copyClient := *client + copyClient.Transport = captured + client = ©Client + env := Environment(request.Environment, io.Discard, io.Discard).WithHTTP(client) + auth := getter.NewRegistryAuth(env) + auth.Impl = implementation + domain := source.Host + if domain == "" { + domain = tfimpl.DefaultRegistryDomain(request.Environment, implementation) + } + base, err := getter.GetModuleRegistryURLBasePath(ctx, logger, client, auth, domain) + if err != nil { + return RegistryResolution{}, fmt.Errorf("%w: service discovery for %s: %w", ErrRegistryLookup, domain, err) + } + modulePath, moduleSubdir := getter.SourceDirSubdir(source.Path) + version, err := registryVersion(ctx, logger, client, auth, request, source, domain, base, modulePath) + if err != nil { + return RegistryResolution{}, err + } + moduleURL, err := getter.BuildRequestURL(domain, base, modulePath, version) + if err != nil { + return RegistryResolution{}, fmt.Errorf("%w: module URL for %s: %w", ErrRegistryLookup, modulePath, err) + } + header, err := getter.GetTerraformGetHeader(ctx, logger, client, auth, moduleURL) + if err != nil { + return RegistryResolution{}, fmt.Errorf("%w: download header for %s: %w", ErrRegistryLookup, modulePath, err) + } + download, err := getter.GetDownloadURLFromHeader(moduleURL, header) + if err != nil { + return RegistryResolution{}, fmt.Errorf("%w: download URL for %s: %w", ErrRegistryLookup, modulePath, err) + } + archive, subdir := getter.SourceDirSubdir(download) + resolution := RegistryResolution{Source: withSubdir(archive, path.Join(subdir, moduleSubdir))} + archiveURL, archiveErr := url.Parse(archive) + if archiveErr == nil && (archiveURL.Scheme == "https" || archiveURL.Scheme == "http") && + archiveURL.Scheme == moduleURL.Scheme && strings.EqualFold(archiveURL.Host, moduleURL.Host) { + resolution.Authorization = captured.authorization(moduleURL) + if resolution.Authorization != "" { + resolution.AuthOrigin = moduleURL.Scheme + "://" + moduleURL.Host + } + } + return resolution, nil +} + +type registryAuthCapture struct { + base http.RoundTripper + mu sync.Mutex + byOrigin map[string]string +} + +func (c *registryAuthCapture) RoundTrip(request *http.Request) (*http.Response, error) { + if value := request.Header.Get("Authorization"); value != "" { + c.mu.Lock() + if c.byOrigin == nil { + c.byOrigin = make(map[string]string) + } + c.byOrigin[request.URL.Scheme+"://"+request.URL.Host] = value + c.mu.Unlock() + } + base := c.base + if base == nil { + base = http.DefaultTransport + } + return base.RoundTrip(request) +} + +func (c *registryAuthCapture) authorization(registry *url.URL) string { + c.mu.Lock() + defer c.mu.Unlock() + return c.byOrigin[registry.Scheme+"://"+registry.Host] +} + +// registryVersion applies Terragrunt's precedence: an exact `?version=`, else the terraform block's constraint, +// else the latest stable release. +func registryVersion(ctx context.Context, logger log.Logger, client *http.Client, auth getter.RegistryAuth, request RegistryRequest, source *url.URL, domain, base, modulePath string) (string, error) { + versions, present := source.Query()["version"] + if present { + if len(versions) != 1 || versions[0] == "" { + return "", fmt.Errorf("%w: the version query must appear once and not be empty", ErrMalformedRegistryVersion) + } + if getter.SourceHasVersionConstraint(request.Source) { + return "", fmt.Errorf("%w: the source %q sets a version constraint in its ?version= query, which accepts an exact version only; express a constraint with the terraform block's version attribute instead", ErrRegistryVersionConstraint, request.Source) + } + return versions[0], nil + } + var version string + var err error + if request.Constraint != "" { + version, err = getter.GetMatchingModuleVersion(ctx, logger, client, auth, domain, base, modulePath, request.Constraint) + } else { + version, err = getter.GetLatestModuleVersion(ctx, logger, client, auth, domain, base, modulePath) + } + if err != nil { + return "", fmt.Errorf("%w: version of %s: %w", ErrRegistryLookup, modulePath, err) + } + return version, nil +} + +// withSubdir appends `//subdir` to the path of a getter URL, ahead of its query, which is where go-getter reads it. +func withSubdir(source, subdir string) string { + if subdir == "" || subdir == "." { + return source + } + location, query, hasQuery := strings.Cut(source, "?") + location = strings.TrimSuffix(location, "/") + "//" + strings.TrimPrefix(subdir, "/") + if hasQuery { + return location + "?" + query + } + return location +} diff --git a/plugins/terragrunt/bridge/registry_test.go b/plugins/terragrunt/bridge/registry_test.go new file mode 100644 index 00000000000..5e5a7003d2b --- /dev/null +++ b/plugins/terragrunt/bridge/registry_test.go @@ -0,0 +1,224 @@ +package atmosbridge + +import ( + "context" + "errors" + "fmt" + "io" + "net/http" + "net/http/httptest" + "net/url" + "os" + "path/filepath" + "testing" + + "github.com/gruntwork-io/terragrunt/pkg/log" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +// fakeRegistry serves the Terraform module registry protocol over TLS. +type fakeRegistry struct { + server *httptest.Server + authHeaders map[string]string + download string + requests []string +} + +func newFakeRegistry(t *testing.T, versions ...string) *fakeRegistry { + t.Helper() + registry := &fakeRegistry{download: "git::https://example.test/acme/vpc.git?ref=v%s", authHeaders: make(map[string]string)} + mux := http.NewServeMux() + mux.HandleFunc("/.well-known/terraform.json", func(w http.ResponseWriter, r *http.Request) { + registry.record(r) + fmt.Fprint(w, `{"modules.v1":"/v1/modules/"}`) + }) + mux.HandleFunc("/v1/modules/acme/vpc/aws/versions", func(w http.ResponseWriter, r *http.Request) { + registry.record(r) + body := `{"modules":[{"versions":[` + for i, version := range versions { + if i > 0 { + body += "," + } + body += fmt.Sprintf(`{"version":%q}`, version) + } + fmt.Fprint(w, body+`]}]}`) + }) + mux.HandleFunc("/v1/modules/acme/vpc/aws/", func(w http.ResponseWriter, r *http.Request) { + registry.record(r) + version := filepath.Base(filepath.Dir(r.URL.Path)) + w.Header().Set("X-Terraform-Get", fmt.Sprintf(registry.download, version)) + w.WriteHeader(http.StatusNoContent) + }) + registry.server = httptest.NewTLSServer(mux) + t.Cleanup(registry.server.Close) + return registry +} + +func (r *fakeRegistry) record(req *http.Request) { + r.requests = append(r.requests, req.URL.Path) + r.authHeaders[req.URL.Path] = req.Header.Get("Authorization") +} + +func (r *fakeRegistry) source(path, query string) string { + address, _ := url.Parse(r.server.URL) + return "tfr://" + address.Host + path + query +} + +func resolveTestSource(t *testing.T, registry *fakeRegistry, source, constraint string, env map[string]string) (string, error) { + t.Helper() + logger := log.New(log.WithOutput(io.Discard)) + return ResolveRegistrySource(context.Background(), logger, RegistryRequest{ + Client: registry.server.Client(), Source: source, Constraint: constraint, Environment: env, OpenTofu: true, + }) +} + +func TestResolveRegistrySourceVersionSelection(t *testing.T) { + tests := []struct { + name, query, constraint, want string + }{ + {"exact version", "?version=1.2.0", "", "git::https://example.test/acme/vpc.git?ref=v1.2.0"}, + {"latest stable skips prereleases", "", "", "git::https://example.test/acme/vpc.git?ref=v1.2.0"}, + {"terraform block constraint", "", "~> 1.0, < 1.2", "git::https://example.test/acme/vpc.git?ref=v1.0.0"}, + } + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + registry := newFakeRegistry(t, "1.0.0", "1.2.0", "2.0.0-beta.1") + got, err := resolveTestSource(t, registry, registry.source("/acme/vpc/aws", test.query), test.constraint, map[string]string{}) + require.NoError(t, err) + assert.Equal(t, test.want, got) + }) + } +} + +func TestResolveRegistrySourceSubdirectories(t *testing.T) { + registry := newFakeRegistry(t, "1.0.0") + got, err := resolveTestSource(t, registry, registry.source("/acme/vpc/aws//modules/nat", "?version=1.0.0"), "", map[string]string{}) + require.NoError(t, err) + assert.Equal(t, "git::https://example.test/acme/vpc.git//modules/nat?ref=v1.0.0", got) + + // A subdirectory on both the registry's answer and the source joins them. + registry.download = "git::https://example.test/acme/vpc.git//root?ref=v%s" + got, err = resolveTestSource(t, registry, registry.source("/acme/vpc/aws//modules/nat", "?version=1.0.0"), "", map[string]string{}) + require.NoError(t, err) + assert.Equal(t, "git::https://example.test/acme/vpc.git//root/modules/nat?ref=v1.0.0", got) +} + +func TestResolveRegistrySourceRelativeDownload(t *testing.T) { + registry := newFakeRegistry(t, "1.0.0") + registry.download = "/archives/%s.tar.gz" + got, err := resolveTestSource(t, registry, registry.source("/acme/vpc/aws", "?version=1.0.0"), "", map[string]string{}) + require.NoError(t, err) + assert.Equal(t, registry.server.URL+"/archives/1.0.0.tar.gz", got) +} + +func TestResolveRegistrySourceCredentials(t *testing.T) { + t.Run("TG_TF_REGISTRY_TOKEN is sent as a bearer token", func(t *testing.T) { + registry := newFakeRegistry(t, "1.0.0") + _, err := resolveTestSource(t, registry, registry.source("/acme/vpc/aws", "?version=1.0.0"), "", map[string]string{"TG_TF_REGISTRY_TOKEN": "bearer-secret"}) + require.NoError(t, err) + assert.Equal(t, map[string]string{"/.well-known/terraform.json": "Bearer bearer-secret", "/v1/modules/acme/vpc/aws/1.0.0/download": "Bearer bearer-secret"}, registry.authHeaders) + }) + t.Run("TF_TOKEN_host wins over the CLI configuration", func(t *testing.T) { + registry := newFakeRegistry(t, "1.0.0") + _, err := resolveTestSource(t, registry, registry.source("/acme/vpc/aws", "?version=1.0.0"), "", map[string]string{"TF_TOKEN_127_0_0_1": "env-secret", "TG_TF_REGISTRY_TOKEN": "ignored", "HOME": t.TempDir()}) + require.NoError(t, err) + assert.Equal(t, map[string]string{"/.well-known/terraform.json": "Bearer env-secret", "/v1/modules/acme/vpc/aws/1.0.0/download": "Bearer env-secret"}, registry.authHeaders) + }) + t.Run("the CLI configuration credentials block is used", func(t *testing.T) { + registry := newFakeRegistry(t, "1.0.0") + address, _ := url.Parse(registry.server.URL) + rc := filepath.Join(t.TempDir(), "terraform.rc") + require.NoError(t, os.WriteFile(rc, []byte(fmt.Sprintf("credentials %q {\n token = \"rc-secret\"\n}\n", address.Hostname())), 0o600)) + _, err := resolveTestSource(t, registry, registry.source("/acme/vpc/aws", "?version=1.0.0"), "", map[string]string{"TF_CLI_CONFIG_FILE": rc, "TOFU_CLI_CONFIG_FILE": rc, "HOME": t.TempDir()}) + require.NoError(t, err) + assert.Equal(t, map[string]string{"/.well-known/terraform.json": "Bearer rc-secret", "/v1/modules/acme/vpc/aws/1.0.0/download": "Bearer rc-secret"}, registry.authHeaders) + }) + t.Run("versions request uses credentials", func(t *testing.T) { + registry := newFakeRegistry(t, "1.0.0") + _, err := resolveTestSource(t, registry, registry.source("/acme/vpc/aws", ""), "", map[string]string{"TG_TF_REGISTRY_TOKEN": "bearer-secret"}) + require.NoError(t, err) + assert.Equal(t, "Bearer bearer-secret", registry.authHeaders["/.well-known/terraform.json"]) + assert.Equal(t, "Bearer bearer-secret", registry.authHeaders["/v1/modules/acme/vpc/aws/versions"]) + assert.Equal(t, "Bearer bearer-secret", registry.authHeaders["/v1/modules/acme/vpc/aws/1.0.0/download"]) + }) + t.Run("anonymous requests carry no authorization", func(t *testing.T) { + registry := newFakeRegistry(t, "1.0.0") + _, err := resolveTestSource(t, registry, registry.source("/acme/vpc/aws", "?version=1.0.0"), "", map[string]string{"HOME": t.TempDir()}) + require.NoError(t, err) + assert.Equal(t, map[string]string{"/.well-known/terraform.json": "", "/v1/modules/acme/vpc/aws/1.0.0/download": ""}, registry.authHeaders) + }) +} + +func TestRegistryArchiveAuthorizationIsScopedToTheRegistryOrigin(t *testing.T) { + registry := newFakeRegistry(t, "1.0.0") + registry.download = "/archives/%s.tar.gz" + logger := log.New(log.WithOutput(io.Discard)) + request := RegistryRequest{Client: registry.server.Client(), Source: registry.source("/acme/vpc/aws", "?version=1.0.0"), Environment: map[string]string{"TG_TF_REGISTRY_TOKEN": "secret-token"}, OpenTofu: true} + resolution, err := ResolveRegistrySourceWithAuth(t.Context(), logger, request) + require.NoError(t, err) + assert.Equal(t, registry.server.URL+"/archives/1.0.0.tar.gz", resolution.Source) + assert.Equal(t, registry.server.URL, resolution.AuthOrigin) + assert.Equal(t, "Bearer secret-token", resolution.Authorization) + assert.NotContains(t, resolution.Source, "secret-token") + + registry.download = "https://unrelated.example/archive-%s.tar.gz" + resolution, err = ResolveRegistrySourceWithAuth(t.Context(), logger, request) + require.NoError(t, err) + assert.Empty(t, resolution.Authorization) + assert.Empty(t, resolution.AuthOrigin) +} + +func TestResolveRegistrySourceErrors(t *testing.T) { + registry := newFakeRegistry(t, "1.0.0") + tests := []struct{ name, source, constraint, wantErr string }{ + {"not a registry source", "https://example.test/module.zip", "", "not a tfr://"}, + {"constraint in the query", registry.source("/acme/vpc/aws", "?version=~%3E1.0"), "", "exact version only"}, + {"empty version", registry.source("/acme/vpc/aws", "?version="), "", "must appear once"}, + {"constraint with no match", registry.source("/acme/vpc/aws", ""), ">= 9.0", "satisfies constraint"}, + {"unknown module", registry.source("/acme/other/aws", "?version=1.0.0"), "", "Error downloading module"}, + } + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + _, err := resolveTestSource(t, registry, test.source, test.constraint, map[string]string{}) + require.Error(t, err) + assert.Contains(t, err.Error(), test.wantErr) + }) + } + for _, test := range []struct { + source string + want error + }{ + {"https://example.test/module.zip", ErrInvalidRegistrySource}, + {registry.source("/acme/vpc/aws", "?version="), ErrMalformedRegistryVersion}, + {registry.source("/acme/vpc/aws", "?version=~%3E1.0"), ErrRegistryVersionConstraint}, + } { + _, err := resolveTestSource(t, registry, test.source, "", map[string]string{}) + require.True(t, errors.Is(err, test.want), "expected %v in %v", test.want, err) + require.False(t, errors.Is(err, ErrRegistryLookup), "a source rejected before any lookup is not a lookup failure: %v", err) + } + // Upstream protocol failures keep their cause and gain the bridge sentinel. + for _, test := range []struct{ name, source, constraint, upstream string }{ + {"version lookup", registry.source("/acme/vpc/aws", ""), ">= 9.0", "satisfies constraint"}, + {"download header", registry.source("/acme/other/aws", "?version=1.0.0"), "", "Error downloading module"}, + } { + t.Run(test.name, func(t *testing.T) { + _, err := resolveTestSource(t, registry, test.source, test.constraint, map[string]string{}) + require.ErrorIs(t, err, ErrRegistryLookup) + wrapped, ok := err.(interface{ Unwrap() []error }) + require.True(t, ok, "the upstream error is wrapped beside the sentinel") + require.Len(t, wrapped.Unwrap(), 2) + require.ErrorContains(t, wrapped.Unwrap()[1], test.upstream) + }) + } + unreachable := RegistryRequest{Client: registry.server.Client(), Source: "tfr://127.0.0.1:1/acme/vpc/aws?version=1.0.0", Environment: map[string]string{}} + _, err := ResolveRegistrySource(t.Context(), log.New(log.WithOutput(io.Discard)), unreachable) + require.ErrorIs(t, err, ErrRegistryLookup, "service discovery failure") +} + +func TestWithSubdir(t *testing.T) { + assert.Equal(t, "https://x/y.zip", withSubdir("https://x/y.zip", "")) + assert.Equal(t, "https://x/y.zip", withSubdir("https://x/y.zip", ".")) + assert.Equal(t, "https://x/y.zip//a/b", withSubdir("https://x/y.zip", "/a/b")) + assert.Equal(t, "git::https://x/y.git//a?ref=v1", withSubdir("git::https://x/y.git/?ref=v1", "a")) +} diff --git a/plugins/terragrunt/internal/engine/backend.go b/plugins/terragrunt/internal/engine/backend.go index b7954a50fd9..bbe60c49bc8 100644 --- a/plugins/terragrunt/internal/engine/backend.go +++ b/plugins/terragrunt/internal/engine/backend.go @@ -1,26 +1,24 @@ package engine import ( - "errors" - "github.com/gruntwork-io/terragrunt/pkg/atmosbridge" "github.com/gruntwork-io/terragrunt/pkg/config" "github.com/cloudposse/atmos/plugins/terragrunt/internal/model" ) -// ErrUnsupportedRemoteState identifies remote_state settings that the native adapter cannot reproduce. -var ErrUnsupportedRemoteState = errors.New("native adaptation of remote_state disable_init or encryption is not implemented") +// Warning codes for backend settings that have no effect. +const warningEncryptionNotGenerated = "remote_state_encryption_not_generated" -func resolveBackend(cfg *config.TerragruntConfig, result *model.Resolution) error { +func resolveBackend(cfg *config.TerragruntConfig, result *model.Resolution, request *model.Request) error { if cfg.RemoteState == nil { return nil } - if cfg.RemoteState.DisableInit || len(cfg.RemoteState.Encryption) > 0 { - return &model.Problem{ - Code: model.CodeUnsupportedSetting, Summary: "Unsupported Terragrunt setting in `remote_state`", Detail: "The native adaptation does not implement `disable_init` or `remote_state.encryption`.", - Hints: []string{"Remove `disable_init` and `encryption` from `remote_state`. To manage the backend with Atmos instead, migrate the unit to a regular Atmos component with its own `backend` settings."}, File: result.Path, Key: "remote_state", Safe: true, Err: errors.Join(ErrUnsupportedExecution, ErrUnsupportedRemoteState), - } + // `disable_init` only stops Terragrunt from creating backend resources, which it also does not do unless + // asked with --backend-bootstrap. Atmos never creates backend resources, and OpenTofu initializes the + // backend normally in both cases, so the setting needs no handling here. + if err := checkEncryption(len(cfg.RemoteState.Encryption) > 0, cfg.RemoteState.Generate != nil, result, request); err != nil { + return err } result.BackendType = cfg.RemoteState.BackendName result.Backend = atmosbridge.BackendConfig(cfg.RemoteState.Config) @@ -41,3 +39,36 @@ func resolveBackend(cfg *config.TerragruntConfig, result *model.Resolution) erro } return nil } + +// checkEncryption validates the state encryption settings. Terragrunt writes `encryption` into the +// `terraform { encryption }` block of the file that `remote_state.generate` produces and nowhere else, so it +// has no effect without `generate`; that is reported as a warning rather than silently ignored. The block is +// OpenTofu syntax, so it is rejected before Terraform fails on it at init. Whether the generated file is +// written is decided later, by the host in the provisioned working directory: when `if_exists = "skip"` keeps +// an existing file and nothing else configures encryption, the host warns that the state is not encrypted. +func checkEncryption(encrypted, generated bool, result *model.Resolution, request *model.Request) error { + if !encrypted || request.Phase == "backend" { + return nil + } + if !generated { + result.Warnings = append(result.Warnings, model.Warning{ + Code: warningEncryptionNotGenerated, Path: mustRel(request.BasePath, result.Path), + Message: "`remote_state.encryption` has no effect without `remote_state.generate`, so the state is not encrypted. Terragrunt behaves the same way.", + }) + return nil + } + // The unit's terraform_binary, resolved before the backend, is the executable that runs it. + binary := result.TerraformBinary + if binary == "" { + binary = request.Tofu + } + if IsOpenTofu(binary) { + return nil + } + return &model.Problem{ + Code: model.CodeUnsupportedSetting, Summary: "State encryption needs OpenTofu", + Detail: "`remote_state.encryption` writes an `encryption` block that only OpenTofu understands, and the selected executable is not OpenTofu.", + Hints: []string{"Set `components.terraform.command: tofu` in atmos.yaml, or remove `encryption` from `remote_state`."}, + File: result.Path, Key: "remote_state.encryption", Safe: true, Err: ErrUnsupportedExecution, + } +} diff --git a/plugins/terragrunt/internal/engine/backend_test.go b/plugins/terragrunt/internal/engine/backend_test.go index 05189ce1e38..c2308536a03 100644 --- a/plugins/terragrunt/internal/engine/backend_test.go +++ b/plugins/terragrunt/internal/engine/backend_test.go @@ -2,13 +2,14 @@ package engine import ( "cmp" - "errors" "fmt" "os" "path/filepath" "testing" "github.com/stretchr/testify/require" + + "github.com/cloudposse/atmos/plugins/terragrunt/internal/model" ) func TestCloudBackendIdentityWithoutNetwork(t *testing.T) { @@ -75,16 +76,6 @@ func TestRemoteBackendGenerationRetainsPathAndPolicy(t *testing.T) { func TestUnsupportedBackendSemanticsAndPaths(t *testing.T) { for _, test := range []struct{ name, hcl, want string }{ - {"disable_init", `remote_state { - backend = "s3" - disable_init = true - config = { bucket = "existing" } -}`, "disable_init"}, - {"encryption", `remote_state { - backend = "s3" - encryption = { key_provider = "pbkdf2", passphrase = "0123456789abcdef" } - config = { bucket = "existing" } -}`, "disable_init"}, {"backend path escapes", `remote_state { backend = "local" generate = { path = "../escape.tf", if_exists = "overwrite" } @@ -119,12 +110,80 @@ generate "duplicate" { t.Run(test.name, func(t *testing.T) { result, err := Resolve(t.Context(), nativeUnit(t, test.hcl)) require.ErrorContains(t, err, test.want) - require.Equal(t, test.name == "disable_init" || test.name == "encryption", errors.Is(err, ErrUnsupportedRemoteState)) require.Nil(t, result) }) } } +const pbkdf2Passphrase = "correct-horse-battery-staple" + +func encryptedBackend(generate string) string { + return `remote_state { + backend = "local" + ` + generate + ` + encryption = { key_provider = "pbkdf2", passphrase = "` + pbkdf2Passphrase + `" } + config = { path = "state.tfstate" } +}` +} + +func TestRemoteStateEncryptionIsWrittenToGeneratedBackend(t *testing.T) { + request := nativeUnit(t, encryptedBackend(`generate = { path = "backend.tf", if_exists = "overwrite_terragrunt" }`)) + request.Tofu = "/opt/bin/tofu" + result, err := Resolve(t.Context(), request) + require.NoError(t, err) + require.True(t, result.BackendGenerated) + generated := result.Generated["backend.tf"] + require.Contains(t, generated, `backend "local"`) + require.Contains(t, generated, "encryption {") + require.Contains(t, generated, `key_provider "pbkdf2" "default"`) + require.Contains(t, generated, "passphrase = \""+pbkdf2Passphrase+"\"") + require.Empty(t, result.Warnings) + require.NoFileExists(t, filepath.Join(filepath.Dir(request.Path), "backend.tf")) +} + +func TestRemoteStateEncryptionRejectsTerraform(t *testing.T) { + request := nativeUnit(t, encryptedBackend(`generate = { path = "backend.tf", if_exists = "overwrite_terragrunt" }`)) + request.Tofu = "/opt/bin/terraform" + _, err := Resolve(t.Context(), request) + var problem *model.Problem + require.ErrorAs(t, err, &problem) + require.Equal(t, "remote_state.encryption", problem.Key) + require.Contains(t, problem.Summary, "OpenTofu") + require.NotEmpty(t, problem.Hints) + require.NotContains(t, problem.Error(), pbkdf2Passphrase) + + // State identity does not write the block, so it is not blocked. + request.Phase = "backend" + _, err = Resolve(t.Context(), request) + require.NoError(t, err) +} + +// TestRemoteStateEncryptionChecksTheUnitExecutable checks that a unit's terraform_binary, not the host's +// default executable, decides whether the encryption block is supported. +func TestRemoteStateEncryptionChecksTheUnitExecutable(t *testing.T) { + for _, test := range []struct { + name, unit, host string + rejected bool + }{ + {name: "unit OpenTofu over host Terraform", unit: "/opt/bin/tofu", host: "/opt/bin/terraform"}, + {name: "unit Terraform over host OpenTofu", unit: "/opt/bin/terraform", host: "/opt/bin/tofu", rejected: true}, + {name: "host executable without a unit binary", host: "/opt/bin/tofu"}, + {name: "host Terraform without a unit binary", host: "/opt/bin/terraform", rejected: true}, + } { + t.Run(test.name, func(t *testing.T) { + result := &model.Resolution{Path: "unit/terragrunt.hcl", TerraformBinary: test.unit} + err := checkEncryption(true, true, result, &model.Request{Tofu: test.host}) + if !test.rejected { + require.NoError(t, err) + return + } + var problem *model.Problem + require.ErrorAs(t, err, &problem) + require.Equal(t, "remote_state.encryption", problem.Key) + }) + } +} + func TestGeneratedBackendIdentityRejectedAcrossPhases(t *testing.T) { for _, phase := range []string{"", "backend"} { for backendIndex, backend := range []string{ @@ -172,6 +231,35 @@ func TestBackendGenerationPartialParseFailsClosed(t *testing.T) { } } +func TestRemoteStateEncryptionWithoutGenerateWarns(t *testing.T) { + request := nativeUnit(t, encryptedBackend("")) + request.Tofu = "/opt/bin/terraform" + result, err := Resolve(t.Context(), request) + require.NoError(t, err) + require.False(t, result.BackendGenerated) + require.Len(t, result.Warnings, 1) + require.Equal(t, warningEncryptionNotGenerated, result.Warnings[0].Code) + require.NotContains(t, result.Warnings[0].Message, pbkdf2Passphrase) +} + +func TestRemoteStateDisableInitDoesNotChangeTheBackend(t *testing.T) { + const backend = `remote_state { + backend = "s3" + %s + generate = { path = "backend.tf", if_exists = "overwrite_terragrunt" } + config = { bucket = "existing-bucket", key = "env/state", region = "us-east-1", skip_bucket_versioning = true } +}` + plain, err := Resolve(t.Context(), nativeUnit(t, fmt.Sprintf(backend, ""))) + require.NoError(t, err) + disabled, err := Resolve(t.Context(), nativeUnit(t, fmt.Sprintf(backend, "disable_init = true"))) + require.NoError(t, err) + require.Equal(t, plain.BackendType, disabled.BackendType) + require.Equal(t, plain.Backend, disabled.Backend) + require.Contains(t, disabled.Generated["backend.tf"], `backend "s3"`) + require.Equal(t, plain.Generated["backend.tf"], disabled.Generated["backend.tf"]) + require.True(t, disabled.BackendGenerated) +} + func TestBackendGenerationPartialParseInspectsIncludes(t *testing.T) { request := nativeUnit(t, "include \"parent\" { path = \"parent.hcl\" }") request.Phase = "backend" diff --git a/plugins/terragrunt/internal/engine/discovery.go b/plugins/terragrunt/internal/engine/discovery.go index 9fe16311fdd..a1ebdc476c4 100644 --- a/plugins/terragrunt/internal/engine/discovery.go +++ b/plugins/terragrunt/internal/engine/discovery.go @@ -115,6 +115,7 @@ func discoverProjectWithResolver(ctx context.Context, request *model.Request, ev return nil, err } slices.SortFunc(d.result.Inventory, func(a, b model.DiscoveredUnit) int { return strings.Compare(a.Stack+"/"+a.Name, b.Stack+"/"+b.Name) }) + d.result.Warnings = append(d.result.Warnings, featureOverrideWarnings(request.Features, d.result.Inventory)...) if err := checkIdentityCollisions(d.root, d.result.Inventory); err != nil { return nil, err } @@ -433,22 +434,17 @@ func inventorySource(environment *atmosbridge.Venv, cfg *config.TerragruntConfig return nil } result.Source = *cfg.Terraform.Source - if !plainLocalSource(result.Source) { - // A local `dir//subdir` source must exist, like a plain local source. - if localSubdirSource(result.Source) { - if problem := localSubdirSourceProblem(environment.FS, path, result.Source); problem != nil { - return problem - } - } - // An unsupported remote transport fails discovery too, before any unit runs. - return validateRemoteSource(result.Source) - } - result.Source = localPath(filepath.Dir(path), result.Source) // The active filesystem also sees modules that a generated stack unit names in the overlay. - if problem := localSourceProblem(environment.FS, path, *cfg.Terraform.Source, result.Source); problem != nil { + local, problem := resolveLocalModuleSource(environment.FS, path, result.Source) + if problem != nil { return problem } - result.Folders = []string{result.Source} + if local == nil { + // An unsupported remote transport fails discovery too, before any unit runs. + return validateRemoteSource(result.Source) + } + result.Source = local.source + result.Folders = []string{local.root} return nil } @@ -468,26 +464,38 @@ func (e *unitEvaluator) resolveUnitInventory(ctx context.Context) (*model.Resolu return nil, err } result := &model.Resolution{Path: e.path, Files: append(e.pctx.FilesRead.Paths(), e.path)} + if err := applyParity(cfg, e.request, e.path, result); err != nil { + return nil, err + } if err := inventoryWorkspaceProblem(e.request, cfg, e.path); err != nil { return nil, err } if err := inventorySource(e.environment, cfg, e.path, result); err != nil { return nil, err } - if result.OutputDependencies, err = outputDependencies(e.environment, cfg, e.path); err != nil { + if err := e.populateInventoryDependencies(cfg, result); err != nil { return nil, err } + return result, nil +} + +// populateInventoryDependencies keeps output reads in the scheduling dependency set. +func (e *unitEvaluator) populateInventoryDependencies(cfg *config.TerragruntConfig, result *model.Resolution) error { + var err error + if result.OutputDependencies, err = outputDependencies(e.environment, cfg, e.path); err != nil { + return err + } // Output reads also require a scheduling edge before preparation. order, err := orderDependencies(e.environment, cfg, e.path) if err != nil { - return nil, err + return err } for _, dependency := range append(slices.Clone(result.OutputDependencies), order...) { if !slices.Contains(result.Dependencies, dependency) { result.Dependencies = append(result.Dependencies, dependency) } } - return result, nil + return nil } func outputDependencies(environment *atmosbridge.Venv, cfg *config.TerragruntConfig, path string) ([]string, error) { diff --git a/plugins/terragrunt/internal/engine/discovery_paths_test.go b/plugins/terragrunt/internal/engine/discovery_paths_test.go index 652d31ea08f..bbbf557fefe 100644 --- a/plugins/terragrunt/internal/engine/discovery_paths_test.go +++ b/plugins/terragrunt/internal/engine/discovery_paths_test.go @@ -272,7 +272,7 @@ func TestDiscoveryKeepsSiblingsOfAFailedGeneratedUnit(t *testing.T) { } write("modules/unit/main.tf", "") write("catalog/good/terragrunt.hcl", "terraform { source = \"${get_repo_root()}/modules/unit\" }\n") - write("catalog/broken/terragrunt.hcl", "terraform { source = \"${get_repo_root()}/modules/unit\" }\nterraform_binary = \"tofu\"\n") + write("catalog/broken/terragrunt.hcl", "terraform { source = \"${get_repo_root()}/modules/unit\" }\nerrors {\n retry \"x\" {\n retryable_errors = [\".*\"]\n max_attempts = 0\n sleep_interval_sec = 1\n }\n}\n") write(".git/HEAD", "ref: refs/heads/main\n") write("live/terragrunt.stack.hcl", ` unit "a_broken" { @@ -292,7 +292,7 @@ unit "z_good" { require.Equal(t, "live", result.Failures[0].Stack) require.Equal(t, "a_broken", result.Failures[0].Name) problem := asProblem(t, result.Failures[0].Err) - require.Equal(t, "terraform_binary", problem.Key) + require.Equal(t, `errors.retry "x"`, problem.Key) } // stackProject writes a module, a catalog unit pointing at it, and the given files. diff --git a/plugins/terragrunt/internal/engine/execution.go b/plugins/terragrunt/internal/engine/execution.go index 09cc8656af0..177d997c119 100644 --- a/plugins/terragrunt/internal/engine/execution.go +++ b/plugins/terragrunt/internal/engine/execution.go @@ -26,18 +26,6 @@ import ( // command names, never the argv or evaluated values, so they are safe to show to the user. var ErrUnsupportedExecution = errors.New("unsupported native Terragrunt execution configuration") -// unsupportedLifecycles are hook commands that name Terragrunt's own lifecycle rather than a -// Terraform command. The host has no equivalent lifecycle yet, so accepting them would silently -// never run the hook. -var unsupportedLifecycles = map[string]string{ - "terragrunt-read-config": "runs after Terragrunt reads configuration", - "read-config": "runs after Terragrunt reads configuration", - "init-from-module": "runs while Terragrunt downloads the module source", -} - -// builtinHookExecutables are executable names Terragrunt intercepts instead of running them. -var builtinHookExecutables = map[string]struct{}{"tflint": {}} - // executionFromConfig converts the evaluated terraform block into data the host executes. // It returns nil when the unit declares no hooks or extra arguments. Evaluation never runs a hook. func executionFromConfig(terraform *config.TerraformConfig) (*model.Execution, error) { @@ -81,7 +69,7 @@ func (e *unitEvaluator) applyTimeExecution(planned *model.Execution, parse func( return nil, nil, err } // Apply is evaluated with the request's own download directory, so the working directory - // matches the plan-time evaluation. + // matches the plan-time evaluation. requestParsingContext also carries the feature flags. applyRequest := *e.request applyRequest.Command, applyRequest.Arguments = "apply", nil apply := requestParsingContext(&applyRequest, e.path) @@ -100,6 +88,8 @@ func (e *unitEvaluator) applyTimeExecution(planned *model.Execution, parse func( if err != nil { return nil, nil, err } + // The plan-time execution is already finalized, so both are compared in the same form. + finalizeExecution(applied, e.path) if applied == nil { applied = &model.Execution{} } @@ -206,22 +196,10 @@ func errorHooksFromConfig(source []config.ErrorHook) ([]model.Hook, error) { } func validateHook(kind string, hook *model.Hook) error { - for _, command := range hook.Commands { - if reason, unsupported := unsupportedLifecycles[command]; unsupported { - return unsupportedHook(fmt.Sprintf("%s %q", kind, hook.Name), fmt.Sprintf("Unsupported hook command `%s` in %s %q", command, kind, hook.Name), - fmt.Sprintf("The `%s` lifecycle command %s, which Atmos does not implement, so the hook would never run.", command, reason), - "Remove the lifecycle command from `commands`. Hooks can name Terraform commands such as `plan`, `apply`, and `init`.") - } - } if len(hook.Execute) == 0 || hook.Execute[0] == "" { return unsupportedHook(fmt.Sprintf("%s %q", kind, hook.Name), fmt.Sprintf("%s %q has no command to run", kind, hook.Name), "The hook sets no `execute` list or its first entry is empty.", "Set `execute = [\"program\", \"argument\"]` on the hook.") } - if _, builtin := builtinHookExecutables[hook.Execute[0]]; builtin { - return unsupportedHook(fmt.Sprintf("%s %q", kind, hook.Name), fmt.Sprintf("Unsupported built-in `%s` hook in %s %q", hook.Execute[0], kind, hook.Name), - fmt.Sprintf("Terragrunt intercepts a hook that executes `%s` and runs its own integration, which Atmos does not implement.", hook.Execute[0]), - "Run the tool through a wrapper script, for example `execute = [\"./run-tflint.sh\"]`, or run it as a separate workflow step before the Terraform command.") - } return nil } diff --git a/plugins/terragrunt/internal/engine/execution_test.go b/plugins/terragrunt/internal/engine/execution_test.go index a347288f1f0..745e9417571 100644 --- a/plugins/terragrunt/internal/engine/execution_test.go +++ b/plugins/terragrunt/internal/engine/execution_test.go @@ -4,10 +4,14 @@ import ( "encoding/json" "os" "path/filepath" + "slices" "testing" + "github.com/gruntwork-io/terragrunt/pkg/config" "github.com/spf13/afero" + "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" + "github.com/zclconf/go-cty/cty" "github.com/cloudposse/atmos/plugins/terragrunt/internal/model" ) @@ -110,23 +114,6 @@ func TestExecutionSkipsHooksWhoseConditionIsFalse(t *testing.T) { func TestExecutionRejectsUnimplementedHookSemantics(t *testing.T) { for _, test := range []struct{ name, body, message string }{ - {"read-config before", `before_hook "x" { - commands = ["terragrunt-read-config"] - execute = ["tool"] - }`, "terragrunt-read-config"}, - {"read-config alias after", `after_hook "x" { - commands = ["plan", "read-config"] - execute = ["tool"] - }`, "read-config"}, - {"init-from-module error", `error_hook "x" { - commands = ["init-from-module"] - execute = ["tool"] - on_errors = [".*"] - }`, "init-from-module"}, - {"tflint built in", `before_hook "lint" { - commands = ["plan"] - execute = ["tflint", "--init"] - }`, "tflint"}, {"invalid on_errors", `error_hook "bad" { commands = ["plan"] execute = ["tool"] @@ -200,6 +187,569 @@ func TestValidateHookRequiresRunnableExecuteList(t *testing.T) { require.NoError(t, validateHook("before_hook", &model.Hook{Name: "h", Commands: []string{"plan"}, Execute: []string{"tool"}})) } +// fixtureEngine returns a request whose executable is a fake OpenTofu that answers `version`. +func fixtureEngine(t *testing.T, original *model.Request) model.Request { + request := *original + t.Helper() + request.Tofu = installFixtureExecutable(t, t.TempDir(), toolchainTofuName) + request.Environment = map[string]string{toolchainChildEnv: "1"} + return request +} + +func TestTerraformBinaryAnchorsRelativePaths(t *testing.T) { + unit := filepath.Join("repo", "live", "terragrunt.hcl") + assert.Equal(t, "tofu", terraformBinary(" tofu ", unit), "a bare name is left to the toolchain") + assert.Empty(t, terraformBinary("", unit)) + assert.Equal(t, filepath.Join("repo", "live", "bin", "tofu"), terraformBinary("./bin/tofu", unit)) + absolute := filepath.Join(t.TempDir(), "tofu") + assert.Equal(t, absolute, terraformBinary(absolute, unit)) +} + +func TestResolveReturnsTerraformBinaryAndValidatesTheEngine(t *testing.T) { + // The probe runs the executable the unit selected, not the host's command. + tofu := filepath.ToSlash(installFixtureExecutable(t, t.TempDir(), toolchainTofuName)) + request := nativeUnit(t, `terraform_binary = "`+tofu+`"`) + request.Environment = map[string]string{toolchainChildEnv: "1"} + result, err := Resolve(t.Context(), request) + require.NoError(t, err) + require.Equal(t, tofu, result.TerraformBinary, "an absolute path is returned as the unit wrote it") + + // A unit without terraform_binary never probes the executable. + plain := nativeUnit(t, "") + result, err = Resolve(t.Context(), plain) + require.NoError(t, err) + require.Empty(t, result.TerraformBinary) + + // An executable that is not Terraform or OpenTofu is rejected with the setting named. + tool := filepath.ToSlash(installFixtureExecutable(t, t.TempDir(), toolchainToolName)) + wrong := nativeUnit(t, `terraform_binary = "`+tool+`"`) + wrong.Environment = map[string]string{toolchainChildEnv: "1"} + _, err = Resolve(t.Context(), wrong) + problem := asProblem(t, err) + require.Equal(t, "terraform_binary", problem.Key) + require.Contains(t, problem.Summary, "not a Terraform or OpenTofu executable") +} + +func TestBareTerraformBinaryIsResolvedOnTheInvocationPath(t *testing.T) { + bin := t.TempDir() + installFixtureExecutable(t, bin, toolchainTofuName) + empty := t.TempDir() + + t.Run("a name found on the invocation PATH is probed", func(t *testing.T) { + request := nativeUnit(t, `terraform_binary = "`+toolchainTofuName+`"`) + request.Environment = map[string]string{toolchainChildEnv: "1", "PATH": bin} + result, err := Resolve(t.Context(), request) + require.NoError(t, err) + require.Equal(t, toolchainTofuName, result.TerraformBinary, "the unit's name is returned as written") + }) + + t.Run("a name missing from the invocation PATH fails even when this process could find it", func(t *testing.T) { + t.Setenv("PATH", bin+string(os.PathListSeparator)+os.Getenv("PATH")) + request := nativeUnit(t, `terraform_binary = "`+toolchainTofuName+`"`) + request.Environment = map[string]string{toolchainChildEnv: "1", "PATH": empty} + _, err := Resolve(t.Context(), request) + problem := asProblem(t, err) + require.Equal(t, "terraform_binary", problem.Key) + require.Contains(t, problem.Summary, "not a Terraform or OpenTofu executable") + }) + + t.Run("an invocation without PATH leaves the name to the executor", func(t *testing.T) { + executable, err := resolveOnInvocationPath("tofu", map[string]string{}) + require.NoError(t, err) + require.Equal(t, "tofu", executable) + }) + + t.Run("an explicit path is not searched", func(t *testing.T) { + explicit := filepath.Join(empty, "bin", "tofu") + executable, err := resolveOnInvocationPath(explicit, map[string]string{"PATH": bin}) + require.NoError(t, err) + require.Equal(t, explicit, executable) + }) +} + +func TestDiscoveryReadsTerraformBinaryWithoutRunningIt(t *testing.T) { + request := nativeUnit(t, `terraform_binary = "./bin/tofu"`) + request.Phase = "discover" + result, err := Resolve(t.Context(), request) + require.NoError(t, err) + require.Equal(t, filepath.Join(physicalPath(t, filepath.Dir(request.Path)), "bin", "tofu"), result.TerraformBinary) +} + +func TestErrorsBlockIsReturnedInDeclarationOrder(t *testing.T) { + request := nativeUnit(t, `errors { + retry "network" { + retryable_errors = ["(?s).*timeout.*", "!ignore me"] + max_attempts = 4 + sleep_interval_sec = 2 + } + retry "quota" { + retryable_errors = ["quota"] + max_attempts = 2 + sleep_interval_sec = 0 + } + ignore "known" { + ignorable_errors = ["already exists"] + message = "resource existed" + signals = { + note = "kept" + other = "also" + } + } +}`) + request.Phase = "discover" + result, err := Resolve(t.Context(), request) + require.NoError(t, err) + require.Equal(t, []model.RetryPolicy{ + {Name: "network", Patterns: []string{"(?s).*timeout.*", "!ignore me"}, MaxAttempts: 4, SleepIntervalSec: 2}, + {Name: "quota", Patterns: []string{"quota"}, MaxAttempts: 2}, + }, result.Errors.Retry) + require.Len(t, result.Errors.Ignore, 1) + ignore := result.Errors.Ignore[0] + require.Equal(t, "known", ignore.Name) + require.Equal(t, "resource existed", ignore.Message) + require.Equal(t, "kept", ignore.Signals["note"]) + require.Equal(t, "also", ignore.Signals["other"]) +} + +func TestErrorsBlockRejectsInvalidPolicies(t *testing.T) { + for name, test := range map[string]struct{ body, key, message string }{ + "no attempts": {`retry "x" { + retryable_errors = ["a"] + max_attempts = 0 + sleep_interval_sec = 1 + }`, `errors.retry "x"`, "max_attempts"}, + "negative sleep": {`retry "x" { + retryable_errors = ["a"] + max_attempts = 1 + sleep_interval_sec = -1 + }`, `errors.retry "x"`, "sleep_interval_sec"}, + "attempts out of range": {`retry "x" { + retryable_errors = ["a"] + max_attempts = 4294967296 + sleep_interval_sec = 1 + }`, `errors.retry "x"`, "too large"}, + "sleep out of range": {`retry "x" { + retryable_errors = ["a"] + max_attempts = 1 + sleep_interval_sec = 4294967296 + }`, `errors.retry "x"`, "too large"}, + "bad retry pattern": {`retry "x" { + retryable_errors = ["("] + max_attempts = 1 + sleep_interval_sec = 1 + }`, `errors.retry "x"`, "Invalid error pattern"}, + "bad ignore pattern": {`ignore "y" { + ignorable_errors = ["!("] + }`, `errors.ignore "y"`, "Invalid error pattern"}, + } { + t.Run(name, func(t *testing.T) { + request := nativeUnit(t, "errors {\n "+test.body+"\n}") + request.Phase = "discover" + _, err := Resolve(t.Context(), request) + problem := asProblem(t, err) + require.Equal(t, test.key, problem.Key) + require.Contains(t, problem.Summary+problem.Detail, test.message) + require.True(t, problem.Safe) + require.NotEmpty(t, problem.Hints) + }) + } +} + +func TestExcludeBlockOnlyAppliesWhileItsConditionHolds(t *testing.T) { + enabled := nativeUnit(t, "exclude {\n if = true\n actions = [\"plan\", \"destroy\"]\n exclude_dependencies = true\n no_run = true\n}") + enabled.Phase = "discover" + result, err := Resolve(t.Context(), enabled) + require.NoError(t, err) + noRun := true + require.Equal(t, &model.Exclude{Actions: []string{"plan", "destroy"}, ExcludeDependencies: true, NoRun: &noRun}, result.Exclude) + + disabled := nativeUnit(t, "exclude {\n if = false\n actions = [\"all\"]\n}") + disabled.Phase = "discover" + result, err = Resolve(t.Context(), disabled) + require.NoError(t, err) + require.Nil(t, result.Exclude) + + absent := nativeUnit(t, "") + absent.Phase = "discover" + result, err = Resolve(t.Context(), absent) + require.NoError(t, err) + require.Nil(t, result.Exclude) +} + +func TestExcludeActionMatching(t *testing.T) { + for _, test := range []struct { + actions []string + command string + want bool + }{ + {[]string{"plan"}, "plan", true}, + {[]string{"PLAN"}, "plan", true}, + {[]string{"plan"}, "apply", false}, + {[]string{"all"}, "output", true}, + {[]string{"all_except_output"}, "output", false}, + {[]string{"all_except_output"}, "plan", true}, + {nil, "plan", false}, + } { + assert.Equal(t, test.want, ExcludeListsAction(test.actions, test.command), "%v %s", test.actions, test.command) + } +} + +func TestPreventsRunFollowsNoRun(t *testing.T) { + yes, no := true, false + all := []string{"all"} + for name, test := range map[string]struct { + exclude *model.Exclude + command string + want bool + }{ + "no block": {nil, "plan", false}, + "unset, exact action": {&model.Exclude{Actions: []string{"plan"}}, "plan", true}, + "unset, all does not apply": {&model.Exclude{Actions: all}, "plan", false}, + "unset, other action": {&model.Exclude{Actions: []string{"apply"}}, "plan", false}, + "true honors all": {&model.Exclude{Actions: all, NoRun: &yes}, "plan", true}, + "true honors all_except": {&model.Exclude{Actions: []string{"all_except_output"}, NoRun: &yes}, "output", false}, + "false never prevents": {&model.Exclude{Actions: []string{"plan"}, NoRun: &no}, "plan", false}, + "true, command not listed": {&model.Exclude{Actions: []string{"apply"}, NoRun: &yes}, "plan", false}, + "true, case insensitive set": {&model.Exclude{Actions: []string{"PLAN"}, NoRun: &yes}, "plan", true}, + } { + t.Run(name, func(t *testing.T) { + assert.Equal(t, test.want, PreventsRun(test.exclude, test.command)) + }) + } +} + +const featureUnit = ` +feature "enabled" { + default = false +} +feature "size" { + default = 3 +} +feature "tier" { + default = "basic" +} +inputs = { + enabled = feature.enabled.value + size = feature.size.value + tier = feature.tier.value +} +` + +func TestFeatureDefaultsAreRenderedAsStrings(t *testing.T) { + result, err := Resolve(t.Context(), nativeUnit(t, featureUnit)) + require.NoError(t, err) + require.Equal(t, map[string]string{"enabled": "false", "size": "3", "tier": "basic"}, result.FeatureFlags) + require.Equal(t, false, result.Inputs["enabled"]) + require.Equal(t, json.Number("3"), result.Inputs["size"]) + require.Equal(t, "basic", result.Inputs["tier"]) +} + +func TestFeatureOverridesReplaceDefaults(t *testing.T) { + request := nativeUnit(t, featureUnit) + request.Features = map[string]string{"enabled": "true", "tier": "premium", "size": "8"} + result, err := Resolve(t.Context(), request) + require.NoError(t, err) + require.Equal(t, map[string]string{"enabled": "true", "size": "8", "tier": "premium"}, result.FeatureFlags) + require.Equal(t, true, result.Inputs["enabled"], "a boolean feature is typed from its override") + require.Equal(t, "8", result.Inputs["size"], "Terragrunt keeps a numeric feature override as its CLI string") + require.Equal(t, "premium", result.Inputs["tier"]) +} + +func TestFeatureOverrideOfWrongTypeNamesTheFeature(t *testing.T) { + request := nativeUnit(t, featureUnit) + request.Features = map[string]string{"enabled": "sometimes"} + _, err := Resolve(t.Context(), request) + require.Error(t, err) + require.ErrorContains(t, err, "enabled") +} + +func TestFeatureCanChooseADependencyOrSource(t *testing.T) { + request := nativeUnit(t, ` +feature "wide" { + default = false +} +exclude { + if = feature.wide.value + actions = ["plan"] +} +`) + request.Phase = "discover" + result, err := Resolve(t.Context(), request) + require.NoError(t, err) + require.Nil(t, result.Exclude) + + request.Features = map[string]string{"wide": "true"} + result, err = Resolve(t.Context(), request) + require.NoError(t, err) + require.NotNil(t, result.Exclude, "the override must reach the discovery evaluation too") +} + +func TestParseFeatureEnvironment(t *testing.T) { + parsed, err := ParseFeatureEnvironment(" a=1, b = text ,,c=") + require.NoError(t, err) + require.Equal(t, map[string]string{"a": "1", "b": "text", "c": ""}, parsed) + + for _, bad := range []string{"novalue", "=x"} { + _, err = ParseFeatureEnvironment(bad) + problem := asProblem(t, err) + require.Equal(t, "TG_FEATURE", problem.Key) + } +} + +func TestFeatureOverrideWarningsNameUnusedKeys(t *testing.T) { + inventory := []model.DiscoveredUnit{{Config: model.Resolution{FeatureFlags: map[string]string{"known": "x"}}}} + assert.Nil(t, featureOverrideWarnings(nil, inventory)) + assert.Empty(t, featureOverrideWarnings(map[string]string{"known": "1"}, inventory)) + warnings := featureOverrideWarnings(map[string]string{"typo": "1", "known": "1", "also": "2"}, inventory) + require.Len(t, warnings, 2) + require.Equal(t, WarningFeatureOverrideUnused, warnings[0].Code) + require.Contains(t, warnings[0].Message, "`also`") + require.Contains(t, warnings[1].Message, "`typo`") +} + +func TestLifecycleHooksRunInTheUnitDirectory(t *testing.T) { + request := executionUnit(t, ` + after_hook "read" { + commands = ["terragrunt-read-config", "read-config"] + execute = ["tool", "read"] + } + before_hook "fetch" { + commands = ["init-from-module"] + execute = ["tool", "fetch"] + working_dir = "elsewhere" + } + error_hook "download_failed" { + commands = ["init-from-module"] + execute = ["tool", "report"] + on_errors = [".*"] + } + after_hook "mixed" { + commands = ["init-from-module", "apply"] + execute = ["tool", "mixed"] + } +`) + result, err := Resolve(t.Context(), request) + require.NoError(t, err) + unit := physicalPath(t, filepath.Dir(request.Path)) + + require.Equal(t, "read", result.Execution.AfterHooks[0].Name) + require.Equal(t, unit, result.Execution.AfterHooks[0].WorkingDirectory) + require.Equal(t, "elsewhere", result.Execution.BeforeHooks[0].WorkingDirectory, "an explicit working_dir is kept") + require.Equal(t, unit, result.Execution.ErrorHooks[0].WorkingDirectory) + + // A hook listing both stages is split so each keeps its own default directory. + require.Len(t, result.Execution.AfterHooks, 3) + require.Equal(t, []string{"init-from-module"}, result.Execution.AfterHooks[1].Commands) + require.Equal(t, unit, result.Execution.AfterHooks[1].WorkingDirectory) + require.Equal(t, []string{"apply"}, result.Execution.AfterHooks[2].Commands) + require.Empty(t, result.Execution.AfterHooks[2].WorkingDirectory) +} + +func TestReadConfigBeforeHookIsReportedAsNeverRunning(t *testing.T) { + result, err := Resolve(t.Context(), executionUnit(t, ` + before_hook "early" { + commands = ["terragrunt-read-config"] + execute = ["tool"] + } + before_hook "disabled" { + commands = ["read-config"] + execute = ["tool"] + if = false + } +`)) + require.NoError(t, err) + require.Len(t, result.Warnings, 1) + require.Equal(t, WarningReadConfigBeforeHook, result.Warnings[0].Code) + require.Contains(t, result.Warnings[0].Message, `"early"`) +} + +func TestTflintHookBecomesInitAndLintCommands(t *testing.T) { + request := executionUnit(t, ` + extra_arguments "vars" { + commands = ["plan"] + arguments = ["-var=region=eu"] + required_var_files = ["required.tfvars"] + env_vars = { TF_VAR_zone = "a" } + } + before_hook "lint" { + commands = ["plan"] + execute = ["tflint", "--terragrunt-external-tflint", "--minimum-failure-severity=error"] + } +`) + request.Phase = "" + root := physicalPath(t, filepath.Dir(request.Path)) + config := filepath.Join(root, ".tflint.hcl") + require.NoError(t, os.WriteFile(config, []byte("plugin \"x\" {}\n"), 0o644)) + body, err := os.ReadFile(request.Path) + require.NoError(t, err) + require.NoError(t, os.WriteFile(request.Path, append(body, []byte("inputs = { name = \"web\", count = 2 }\n")...), 0o644)) + request = fixtureEngine(t, &request) + + result, err := Resolve(t.Context(), request) + require.NoError(t, err) + require.Len(t, result.Execution.BeforeHooks, 2) + initHook, lintHook := result.Execution.BeforeHooks[0], result.Execution.BeforeHooks[1] + require.Equal(t, []string{"tflint", "--init", "--config", tflintConfigSearchPrefix + root, "--chdir", "."}, initHook.Execute) + require.Equal(t, []string{ + "tflint", "--config", tflintConfigSearchPrefix + root, "--chdir", ".", + tflintInputVariablesMarker, + "--var=region=eu", "--var-file=required.tfvars", + "--minimum-failure-severity=error", + }, lintHook.Execute) + require.Equal(t, "lint", initHook.Name) + require.Equal(t, []string{"plan"}, lintHook.Commands) +} + +func TestTflintHookDefersConfigurationLookup(t *testing.T) { + root := t.TempDir() + unit := filepath.Join(root, "a", "b") + require.NoError(t, os.MkdirAll(unit, 0o755)) + hooks := expandTflint(&model.Hook{Name: "lint", Commands: []string{"plan"}, Execute: []string{"tflint"}}, &model.Execution{}, filepath.Join(unit, "terragrunt.hcl")) + require.Len(t, hooks, 2) + require.Contains(t, hooks[0].Execute, tflintConfigSearchPrefix+unit) +} + +func TestTflintConfigFlagForms(t *testing.T) { + dir := t.TempDir() + absolute := filepath.Join(dir, "abs.hcl") + for _, arguments := range [][]string{ + {"tflint", "--config", absolute}, + {"tflint", "-c", absolute}, + {"tflint", "--config=" + absolute}, + {"tflint", "-c=" + absolute}, + } { + got, err := tflintConfig(arguments, dir) + require.NoError(t, err) + require.Equal(t, absolute, got) + } + got, err := tflintConfig([]string{"tflint", "--config", "rel.hcl"}, dir) + require.NoError(t, err) + require.Equal(t, filepath.Join(dir, "rel.hcl"), got, "a relative config is anchored at the unit") + require.Equal(t, []string{"--format", "json"}, withoutConfigFlag([]string{"--config", "x", "--format", "json", "-c=y", "--config=z"})) +} + +func TestCtyStringRendersEveryValueKind(t *testing.T) { + assert.Equal(t, "text", ctyString(cty.StringVal("text"))) + assert.Equal(t, "true", ctyString(cty.True)) + assert.Equal(t, "2.5", ctyString(cty.NumberFloatVal(2.5))) + assert.Equal(t, "7", ctyString(cty.NumberIntVal(7))) + assert.Equal(t, `["a","b"]`, ctyString(cty.ListVal([]cty.Value{cty.StringVal("a"), cty.StringVal("b")}))) + assert.Equal(t, "", ctyString(cty.NullVal(cty.String))) + assert.Equal(t, "", ctyString(cty.UnknownVal(cty.String))) + assert.Equal(t, "secret", ctyString(cty.StringVal("secret").Mark("sensitive"))) +} + +func TestSignalValuesConvertToPlainJSON(t *testing.T) { + assert.Nil(t, mustSignals(t, nil)) + signals := mustSignals(t, map[string]cty.Value{ + "text": cty.StringVal("x"), + "number": cty.NumberIntVal(3), + "large": cty.NumberIntVal(9007199254740993), + "list": cty.TupleVal([]cty.Value{cty.True, cty.StringVal("y")}), + "nothing": cty.NullVal(cty.String), + "unknown": cty.UnknownVal(cty.String), + }) + assert.Equal(t, "x", signals["text"]) + assert.Equal(t, json.Number("3"), signals["number"]) + data, err := json.Marshal(signals) + require.NoError(t, err) + assert.Contains(t, string(data), `"large":9007199254740993`) + assert.Equal(t, []any{true, "y"}, signals["list"]) + assert.Nil(t, signals["nothing"]) + assert.Nil(t, signals["unknown"]) +} + +func mustSignals(t *testing.T, signals map[string]cty.Value) map[string]any { + t.Helper() + result, err := signalValues(signals) + require.NoError(t, err) + return result +} + +func TestFinalizeExecutionHandlesAbsentAndEveryHookKind(t *testing.T) { + finalizeExecution(nil, "terragrunt.hcl") + unit := filepath.Join(t.TempDir(), "terragrunt.hcl") + lint := []model.Hook{{Name: "lint", Commands: []string{"plan"}, Execute: []string{"tflint"}}} + execution := &model.Execution{BeforeHooks: slices.Clone(lint), AfterHooks: slices.Clone(lint), ErrorHooks: slices.Clone(lint)} + finalizeExecution(execution, unit) + for name, hooks := range map[string][]model.Hook{"before": execution.BeforeHooks, "after": execution.AfterHooks, "error": execution.ErrorHooks} { + require.Len(t, hooks, 2, name) + require.Contains(t, hooks[1].Execute, tflintInputVariablesMarker, name) + } +} + +func TestTflintVariablesFromExtraArguments(t *testing.T) { + existing := filepath.Join(t.TempDir(), "optional.tfvars") + require.NoError(t, os.WriteFile(existing, nil, 0o644)) + execution := &model.Execution{ExtraArguments: []model.ExtraArguments{ + {Commands: []string{"apply"}, Arguments: []string{"-var=ignored=1"}}, + {Commands: []string{"plan"}, Arguments: []string{"-var-file=a.tfvars", "-lock=false", "-var", "c=3", "-var-file", "d.tfvars"}, OptionalVarFiles: []string{existing, "absent.tfvars"}, Environment: map[string]string{"OTHER": "x", "TF_VAR_b": "2"}}, + }} + require.Equal(t, []string{"--var-file=a.tfvars", "--var=c=3", "--var-file=d.tfvars", tflintOptionalVarFilePrefix + existing, tflintOptionalVarFilePrefix + "absent.tfvars"}, tflintExtraVariables("plan", execution)) + require.Equal(t, []string{"--var=ignored=1"}, tflintExtraVariables("apply", execution)) + require.Empty(t, tflintExtraVariables("plan", &model.Execution{ExtraArguments: []model.ExtraArguments{ + {Commands: []string{"plan"}, Environment: map[string]string{"TF_VAR_b": "2"}}, + }}), "TFLint reads TF_VAR_ from the hook environment, below terraform.tfvars") +} + +func TestTflintHookScopesExtraArgumentsToEachCommand(t *testing.T) { + unit := filepath.Join(t.TempDir(), "terragrunt.hcl") + require.NoError(t, os.WriteFile(filepath.Join(filepath.Dir(unit), ".tflint.hcl"), nil, 0o644)) + execution := &model.Execution{ExtraArguments: []model.ExtraArguments{ + {Commands: []string{"plan"}, Arguments: []string{"-var=only=plan"}}, + {Commands: []string{"apply"}, Arguments: []string{"-var=only=apply"}}, + }} + hooks := expandTflint(&model.Hook{Name: "lint", Commands: []string{"plan", "apply"}, Execute: []string{"tflint"}}, execution, unit) + require.Len(t, hooks, 4) + require.Equal(t, []string{"plan"}, hooks[1].Commands) + require.Contains(t, hooks[1].Execute, "--var=only=plan") + require.NotContains(t, hooks[1].Execute, "--var=only=apply") + require.Equal(t, []string{"apply"}, hooks[3].Commands) + require.Contains(t, hooks[3].Execute, "--var=only=apply") + require.NotContains(t, hooks[3].Execute, "--var=only=plan") +} + +func TestExcludeFromConfigWithoutBlock(t *testing.T) { + assert.Nil(t, excludeFromConfig(nil)) + assert.Nil(t, excludeFromConfig(&config.ExcludeConfig{If: false})) + assert.Equal(t, &model.Exclude{Actions: []string{"plan"}}, excludeFromConfig(&config.ExcludeConfig{If: true, Actions: []string{"plan"}})) + assert.Nil(t, mustErrors(t, nil)) + assert.Nil(t, mustErrors(t, &config.ErrorsConfig{})) +} + +func mustErrors(t *testing.T, block *config.ErrorsConfig) *model.Errors { + t.Helper() + result, err := errorsFromConfig(block, "terragrunt.hcl") + require.NoError(t, err) + return result +} + +func TestFeatureValuesIgnoreUndeclaredAndNilFlags(t *testing.T) { + assert.Nil(t, mustFeatures(t, nil, map[string]string{"x": "1"})) + tier := cty.StringVal("basic") + values := mustFeatures(t, []*config.FeatureFlag{nil, {Name: "tier", Default: &tier}, {Name: "nodefault"}}, nil) + assert.Equal(t, map[string]string{"tier": "basic"}, values) +} + +func mustFeatures(t *testing.T, flags []*config.FeatureFlag, overrides map[string]string) map[string]string { + t.Helper() + result, err := featureFlagValues(flags, overrides, "terragrunt.hcl") + require.NoError(t, err) + return result +} + +func TestCheckFeatureOverridesLeavesParseErrorsToTheEvaluator(t *testing.T) { + failing := func() (*config.TerragruntConfig, error) { return nil, os.ErrInvalid } + require.NoError(t, checkFeatureOverrides("terragrunt.hcl", map[string]string{"a": "1"}, failing)) + require.NoError(t, checkFeatureOverrides("terragrunt.hcl", nil, failing)) + flag := cty.True + declared := func() (*config.TerragruntConfig, error) { + return &config.TerragruntConfig{FeatureFlags: []*config.FeatureFlag{{Name: "on", Default: &flag}}}, nil + } + require.NoError(t, checkFeatureOverrides("terragrunt.hcl", map[string]string{"on": "false"}, declared)) + require.Error(t, checkFeatureOverrides("terragrunt.hcl", map[string]string{"on": "maybe"}, declared)) +} + // effectChildEnv makes the test binary act as a cross-platform effect command (see TestMain). const effectChildEnv = "ATMOS_TERRAGRUNT_EFFECT_CHILD" @@ -265,14 +815,6 @@ func TestDiscoveryValidatesExecutionSyntaxWithoutEffects(t *testing.T) { for _, test := range []struct { name, body, want string }{ - {"lifecycle", `before_hook "hook" { - commands = ["read-config"] - execute = [run_cmd("must-not-execute")] -}`, "Unsupported hook command"}, - {"builtin", `before_hook "hook" { - commands = [run_cmd("must-not-execute")] - execute = ["tflint"] -}`, "Unsupported built-in"}, {"missing execute", `before_hook "hook" { commands = ["plan"] }`, "execute"}, @@ -444,17 +986,18 @@ func TestDiscoveryDefersInheritedExecutionValues(t *testing.T) { } func TestPreparationValidatesDynamicHookSettings(t *testing.T) { - request := executionUnit(t, `before_hook "dynamic" { - commands = [get_env("HOOK_COMMAND")] + request := executionUnit(t, `error_hook "dynamic" { + commands = ["plan"] execute = ["echo"] + on_errors = [get_env("HOOK_PATTERN")] }`) - request.Environment["HOOK_COMMAND"] = "read-config" + request.Environment["HOOK_PATTERN"] = "[" request.Phase = "discover" _, err := Resolve(t.Context(), request) require.NoError(t, err) request.Phase = "" _, err = Resolve(t.Context(), request) - require.ErrorContains(t, err, "Unsupported hook command") + require.ErrorContains(t, err, "Invalid `on_errors`") } func TestObservesCommandAssumesUnreadableFilesDo(t *testing.T) { @@ -469,8 +1012,53 @@ func hookNames(hooks []model.Hook) []string { return names } -// A parent hook that a unit replaces by name never runs, so its unsupported built-in executable -// does not block the unit. The same parent hook is still rejected when nothing replaces it. +// The apply-time evaluation of a saved plan sees the same feature overrides as the plan, and its +// hooks are prepared for the host the same way: a lifecycle hook runs in the unit directory. +func TestApplyTimeExecutionUsesFeatureOverridesAndFinalizesHooks(t *testing.T) { + root := t.TempDir() + module := filepath.Join(root, "module") + require.NoError(t, os.MkdirAll(module, 0o755)) + source, err := json.Marshal(filepath.ToSlash(module)) + require.NoError(t, err) + path := filepath.Join(root, "terragrunt.hcl") + require.NoError(t, os.WriteFile(path, []byte(`feature "notify" { + default = false +} +terraform { + source = `+string(source)+` + after_hook "notify" { + commands = ["apply"] + execute = ["tool", "notify"] + if = feature.notify.value && get_terraform_command() == "apply" + } + before_hook "fetch" { + commands = ["init-from-module"] + execute = ["tool", "fetch"] + if = get_terraform_command() == "apply" + } +} +`), 0o644)) + request := model.Request{Path: path, BasePath: root, Command: "plan", Tofu: "must-not-execute", Environment: map[string]string{}} + + planned, err := Resolve(t.Context(), request) + require.NoError(t, err) + require.NotNil(t, planned.Execution) + require.NotNil(t, planned.Execution.Apply) + require.Empty(t, planned.Execution.Apply.AfterHooks, "the feature defaults to off for apply too") + require.Equal(t, []string{"fetch"}, hookNames(planned.Execution.Apply.BeforeHooks)) + unitDir, err := logicalPath(root) + require.NoError(t, err) + require.Equal(t, unitDir, planned.Execution.Apply.BeforeHooks[0].WorkingDirectory, "apply-time lifecycle hooks are finalized") + + request.Features = map[string]string{"notify": "true"} + overridden, err := Resolve(t.Context(), request) + require.NoError(t, err) + require.NotNil(t, overridden.Execution.Apply) + require.Equal(t, []string{"notify"}, hookNames(overridden.Execution.Apply.AfterHooks), "feature overrides reach the apply-time evaluation") +} + +// A parent hook that a unit replaces by name never runs, so its invalid execute list does not +// block the unit. The same parent hook is still rejected when nothing replaces it. func TestDiscoveryValidatesEffectiveHooksNotOverriddenParents(t *testing.T) { for _, test := range []struct { name, child, want string @@ -479,7 +1067,7 @@ func TestDiscoveryValidatesEffectiveHooksNotOverriddenParents(t *testing.T) { commands = ["plan"] execute = ["./wrapper.sh"] }`, ""}, - {"inherited unchanged", "", "Unsupported built-in"}, + {"inherited unchanged", "", "has no command to run"}, } { t.Run(test.name, func(t *testing.T) { request := executionUnit(t, test.child) @@ -487,7 +1075,7 @@ func TestDiscoveryValidatesEffectiveHooksNotOverriddenParents(t *testing.T) { require.NoError(t, os.WriteFile(parent, []byte(`terraform { before_hook "lint" { commands = ["plan"] - execute = ["tflint"] + execute = [""] } } `), 0o644)) @@ -506,21 +1094,22 @@ func TestDiscoveryValidatesEffectiveHooksNotOverriddenParents(t *testing.T) { } // Includes merge in declaration order, so a later declaration replaces an earlier one whatever the -// include names sort to. Here the unsupported built-in hook is replaced only when its include is first. +// include names sort to. Here the invalid hook, which has no command to run, is replaced only when its +// include is first. func TestDiscoveryUsesIncludeDeclarationOrderForHookOverrides(t *testing.T) { for _, test := range []struct { name, first, second, want string }{ {"later declaration replaces an earlier one", "zeta", "alpha", ""}, - {"earlier declaration is replaced by the later one", "alpha", "zeta", "Unsupported built-in"}, + {"earlier declaration is replaced by the later one", "alpha", "zeta", "has no command to run"}, } { t.Run(test.name, func(t *testing.T) { request := executionUnit(t, "") hook := func(execute string) []byte { return []byte("terraform {\n before_hook \"lint\" {\n commands = [\"plan\"]\n execute = [" + execute + "]\n }\n}\n") } - // The built-in executable is declared by the "zeta" include, the wrapper by "alpha". - require.NoError(t, os.WriteFile(filepath.Join(request.BasePath, "zeta.hcl"), hook(`"tflint"`), 0o644)) + // The invalid hook is declared by the "zeta" include, the wrapper by "alpha". + require.NoError(t, os.WriteFile(filepath.Join(request.BasePath, "zeta.hcl"), hook(`""`), 0o644)) require.NoError(t, os.WriteFile(filepath.Join(request.BasePath, "alpha.hcl"), hook(`"./wrapper.sh"`), 0o644)) body, err := os.ReadFile(request.Path) require.NoError(t, err) @@ -589,3 +1178,62 @@ inputs = { value = dependency.upstream.outputs.value } require.Equal(t, "planned", result.Inputs["value"]) require.NotNil(t, result.Execution.Apply) } + +func TestUnitBinaryOverriddenByTheStackIsNotProbed(t *testing.T) { + bin := t.TempDir() + unitBinary := installFixtureExecutable(t, bin, toolchainTofuName) + // A copy, not a hard link: hard links of the fixture are the same file. + other := filepath.Join(t.TempDir(), "other-"+filepath.Base(unitBinary)) + content, err := os.ReadFile(unitBinary) + require.NoError(t, err) + require.NoError(t, os.WriteFile(other, content, 0o755)) + missing := filepath.Join(t.TempDir(), "absent", "tofu") + notEngine := installFixtureExecutable(t, t.TempDir(), toolchainToolName) + + link := filepath.Join(t.TempDir(), "link-"+filepath.Base(unitBinary)) + linked := os.Symlink(unitBinary, link) == nil + + tests := []struct { + name string + unitBinary string + effective string + wantProblem bool + wantWarnings int + needsLink bool + }{ + {name: "same path probes without a warning", unitBinary: unitBinary, effective: unitBinary}, + {name: "same file through a symlink probes without a warning", unitBinary: unitBinary, effective: link, needsLink: true}, + {name: "unset effective binary probes as before", unitBinary: unitBinary}, + {name: "unset effective binary still fails a bad unit binary", unitBinary: missing, wantProblem: true}, + {name: "bad unit binary without an override fails", unitBinary: notEngine, wantProblem: true, effective: notEngine}, + {name: "another executable skips the probe and warns", unitBinary: notEngine, effective: other, wantWarnings: 1}, + {name: "missing unit binary with another effective one warns", unitBinary: missing, effective: other, wantWarnings: 1}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + if tt.needsLink && !linked { + t.Skip("symlinks are not available on this platform") + } + request := nativeUnit(t, `terraform_binary = "`+filepath.ToSlash(tt.unitBinary)+`"`) + request.Environment = map[string]string{toolchainChildEnv: "1", "PATH": bin} + if tt.effective != "" { + request.Environment["ATMOS_TERRAFORM_BINARY"] = tt.effective + } + result, err := Resolve(t.Context(), request) + if tt.wantProblem { + problem := asProblem(t, err) + require.Equal(t, "terraform_binary", problem.Key) + return + } + require.NoError(t, err) + require.Len(t, result.Warnings, tt.wantWarnings) + if tt.wantWarnings == 0 { + return + } + warning := result.Warnings[0] + require.Equal(t, WarningTerraformBinaryOverridden, warning.Code) + require.Contains(t, warning.Message, tt.effective) + require.Contains(t, warning.Message, filepath.ToSlash(tt.unitBinary)) + }) + } +} diff --git a/plugins/terragrunt/internal/engine/local_subdir_test.go b/plugins/terragrunt/internal/engine/local_subdir_test.go new file mode 100644 index 00000000000..e40ba53a770 --- /dev/null +++ b/plugins/terragrunt/internal/engine/local_subdir_test.go @@ -0,0 +1,76 @@ +package engine + +import ( + "encoding/json" + "os" + "path/filepath" + "testing" + + "github.com/stretchr/testify/require" +) + +func TestLocalModuleSubdirPreservesRootAndIdentity(t *testing.T) { + request := nativeUnit(t, "") + base, err := filepath.EvalSymlinks(request.BasePath) + require.NoError(t, err) + root := filepath.Join(base, "repository") + module := filepath.Join(root, "modules", "app") + require.NoError(t, os.MkdirAll(module, 0o755)) + require.NoError(t, os.MkdirAll(filepath.Join(root, "modules", "shared"), 0o755)) + source := "./repository//modules/app" + encoded, err := json.Marshal(source) + require.NoError(t, err) + require.NoError(t, os.WriteFile(request.Path, []byte("terraform { source = "+string(encoded)+" }"), 0o644)) + + result, err := Resolve(t.Context(), request) + require.NoError(t, err) + require.Equal(t, root+"//modules/app", result.Source) + require.Contains(t, result.Folders, root) + require.NotEqual(t, root, result.WorkingDir) + + request.Phase = "inventory" + inventory, err := Resolve(t.Context(), request) + require.NoError(t, err) + require.Len(t, inventory.Inventory, 1) + require.Equal(t, root+"//modules/app", inventory.Inventory[0].Config.Source) +} + +func TestLocalModuleSubdirRejectsTraversal(t *testing.T) { + request := nativeUnit(t, "") + root := filepath.Join(request.BasePath, "repository") + require.NoError(t, os.MkdirAll(root, 0o755)) + for _, source := range []string{"./repository//../outside", "./repository//modules/../outside", "./repository///absolute"} { + encoded, err := json.Marshal(source) + require.NoError(t, err) + require.NoError(t, os.WriteFile(request.Path, []byte("terraform { source = "+string(encoded)+" }"), 0o644)) + _, err = Resolve(t.Context(), request) + require.ErrorContains(t, err, "subdirectory") + } +} + +func TestLocalModuleSubdirAcceptsExistingRootWithDotInName(t *testing.T) { + for _, test := range []struct { + name string + create bool + valid bool + }{ + {name: "existing directory", create: true, valid: true}, + {name: "missing directory", create: false, valid: false}, + } { + t.Run(test.name, func(t *testing.T) { + request := nativeUnit(t, "") + if test.create { + require.NoError(t, os.MkdirAll(filepath.Join(filepath.Dir(request.Path), "acme.modules", "app"), 0o755)) + } + require.NoError(t, os.WriteFile(request.Path, []byte(`terraform { source = "acme.modules//app" }`), 0o644)) + + _, err := Resolve(t.Context(), request) + + if test.valid { + require.NoError(t, err) + return + } + require.Error(t, err, "a dotted bare path that names no local directory stays a remote shorthand") + }) + } +} diff --git a/plugins/terragrunt/internal/engine/parity.go b/plugins/terragrunt/internal/engine/parity.go new file mode 100644 index 00000000000..c7987182805 --- /dev/null +++ b/plugins/terragrunt/internal/engine/parity.go @@ -0,0 +1,520 @@ +package engine + +import ( + "bytes" + "cmp" + "context" + "encoding/json" + "fmt" + "maps" + "math" + "os" + "os/exec" + "path/filepath" + "regexp" + "runtime" + "slices" + "strconv" + "strings" + + "github.com/gruntwork-io/terragrunt/pkg/config" + "github.com/zclconf/go-cty/cty" + ctyjson "github.com/zclconf/go-cty/cty/json" + + "github.com/cloudposse/atmos/plugins/terragrunt/internal/model" +) + +// applyParity copies the settings that select how a unit runs from the evaluated configuration: +// terraform_binary, the errors block, the exclude block, and feature flags. Everything it returns +// is data; the host owns executable selection, retries, and unit exclusion. +func applyParity(cfg *config.TerragruntConfig, request *model.Request, path string, result *model.Resolution) error { + result.TerraformBinary = terraformBinary(cfg.TerraformBinary, path) + policies, err := errorsFromConfig(cfg.Errors, path) + if err != nil { + return err + } + result.Errors = policies + result.Exclude = excludeFromConfig(cfg.Exclude) + flags, err := featureFlagValues(cfg.FeatureFlags, request.Features, path) + if err != nil { + return err + } + result.FeatureFlags = flags + return nil +} + +// terraformBinary keeps a bare name for the toolchain or PATH to resolve, and anchors a relative +// path at the unit's directory so the result does not depend on where Atmos runs. +func terraformBinary(binary, path string) string { + binary = strings.TrimSpace(binary) + if binary == "" || filepath.IsAbs(binary) || !strings.ContainsAny(binary, `/\`) { + return binary + } + return filepath.Join(filepath.Dir(path), filepath.FromSlash(binary)) +} + +// errorsFromConfig validates and converts the errors block. A negative pattern ("!" prefix) vetoes +// a policy, exactly as it does in Terragrunt. +func errorsFromConfig(block *config.ErrorsConfig, path string) (*model.Errors, error) { + if block == nil || len(block.Retry)+len(block.Ignore) == 0 { + return nil, nil + } + result := &model.Errors{} + for _, retry := range block.Retry { + if retry == nil { + continue + } + if err := validateRetryBlock(retry, path); err != nil { + return nil, err + } + result.Retry = append(result.Retry, model.RetryPolicy{ + Name: retry.Label, Patterns: slices.Clone(retry.RetryableErrors), MaxAttempts: retry.MaxAttempts, SleepIntervalSec: retry.SleepIntervalSec, + }) + } + for _, ignore := range block.Ignore { + if ignore == nil { + continue + } + key := fmt.Sprintf("errors.ignore %q", ignore.Label) + if err := checkPatterns(key, path, ignore.IgnorableErrors); err != nil { + return nil, err + } + signals, err := signalValues(ignore.Signals) + if err != nil { + return nil, errorsProblem(key, path, fmt.Sprintf("`signals` cannot be represented in %s", key), "Signals must be plain values: strings, numbers, booleans, lists, or maps.", "Remove unsupported values from `signals`.") + } + result.Ignore = append(result.Ignore, model.IgnorePolicy{Name: ignore.Label, Patterns: slices.Clone(ignore.IgnorableErrors), Message: ignore.Message, Signals: signals}) + } + return result, nil +} + +func checkPatterns(key, path string, patterns []string) error { + for _, pattern := range patterns { + if _, err := regexp.Compile(strings.TrimPrefix(pattern, "!")); err != nil { + return errorsProblem(key, path, fmt.Sprintf("Invalid error pattern in %s", key), "An entry of the error list is not a valid regular expression.", "Fix the pattern; Atmos matches these entries as Go regular expressions. A leading `!` makes a pattern veto the policy.") + } + } + return nil +} + +func errorsProblem(key, file, summary, detail, hint string) *model.Problem { + return evaluationSetting(key, file, summary, detail, hint) +} + +// signalValues converts the cty values of an ignore block to plain JSON values. +func signalValues(signals map[string]cty.Value) (map[string]any, error) { + if len(signals) == 0 { + return nil, nil + } + result := make(map[string]any, len(signals)) + for _, name := range slices.Sorted(maps.Keys(signals)) { + value, _ := signals[name].UnmarkDeep() + if value.IsNull() || !value.IsWhollyKnown() { + result[name] = nil + continue + } + data, err := ctyjson.Marshal(value, value.Type()) + if err != nil { + return nil, err + } + var plain any + decoder := json.NewDecoder(bytes.NewReader(data)) + decoder.UseNumber() + if err := decoder.Decode(&plain); err != nil { + return nil, err + } + result[name] = plain + } + return result, nil +} + +// excludeFromConfig returns the exclude block only while its condition holds. +func excludeFromConfig(block *config.ExcludeConfig) *model.Exclude { + if block == nil || !block.If { + return nil + } + result := &model.Exclude{Actions: slices.Clone(block.Actions), NoRun: block.NoRun} + if block.ExcludeDependencies != nil { + result.ExcludeDependencies = *block.ExcludeDependencies + } + return result +} + +// ExcludeListsAction reports whether the actions name the command. "all" matches every command and +// "all_except_output" every command but output; other entries match case-insensitively. +// +//nolint:lintroller // Standalone SDK-only plugin has no host perf runtime dependency. +func ExcludeListsAction(actions []string, command string) bool { + command = strings.ToLower(command) + for _, action := range actions { + switch { + case action == "all": + return true + case action == "all_except_output" && command != "output": + return true + case strings.ToLower(action) == command: + return true + } + } + return false +} + +// PreventsRun reports whether the exclude block makes a direct run of the command a no-op. Without +// no_run only an exact action match prevents the run; no_run = true also honors "all" and +// "all_except_output"; no_run = false never prevents it. +// +//nolint:lintroller // Standalone SDK-only plugin has no host perf runtime dependency. +func PreventsRun(exclude *model.Exclude, command string) bool { + if exclude == nil { + return false + } + switch { + case exclude.NoRun == nil: + return slices.Contains(exclude.Actions, command) + case !*exclude.NoRun: + return false + default: + return ExcludeListsAction(exclude.Actions, command) + } +} + +// OperationCommand is the command name Terragrunt matches exclude actions against: destroy for +// `apply -destroy`, apply for deploy, otherwise the command itself. +// +//nolint:lintroller // Standalone SDK-only plugin has no host perf runtime dependency. +func OperationCommand(command string, arguments []string) string { + if command == "deploy" { + command = "apply" + } + if command == "destroy" { + return command + } + if command == "apply" { + for _, argument := range slices.Backward(arguments) { + name, value, hasValue := strings.Cut(argument, "=") + if name != "-destroy" && name != "--destroy" { + continue + } + if !hasValue { + return "destroy" + } + if parsed, err := strconv.ParseBool(value); err == nil && parsed { + return "destroy" + } + return command + } + } + return command +} + +// featureFlagValues returns the effective value of every declared feature, as a string. An override +// replaces the declared default; a boolean feature rejects an override that is not a boolean. +func featureFlagValues(flags []*config.FeatureFlag, overrides map[string]string, path string) (map[string]string, error) { + if len(flags) == 0 { + return nil, nil + } + result := make(map[string]string, len(flags)) + for _, flag := range flags { + if flag == nil { + continue + } + override, overridden := overrides[flag.Name] + switch { + case overridden: + if flag.Default != nil && flag.Default.Type() == cty.Bool { + if _, err := strconv.ParseBool(override); err != nil { + return nil, evaluationSetting("feature "+flag.Name, path, + fmt.Sprintf("Invalid value for the `%s` feature", flag.Name), + fmt.Sprintf("The `%s` feature is a boolean, but its override is not `true` or `false`.", flag.Name), + "Set the override to `true` or `false`, in `components.terraform.terragrunt.features` or in `TG_FEATURE`.") + } + } + result[flag.Name] = override + case flag.Default != nil: + result[flag.Name] = ctyString(*flag.Default) + } + } + return result, nil +} + +func ctyString(value cty.Value) string { + value, _ = value.UnmarkDeep() + switch { + case value.IsNull() || !value.IsKnown(): + return "" + case value.Type() == cty.String: + return value.AsString() + case value.Type() == cty.Bool: + return strconv.FormatBool(value.True()) + case value.Type() == cty.Number: + return value.AsBigFloat().Text('f', -1) + } + data, err := ctyjson.Marshal(value, value.Type()) + if err != nil { + return "" + } + return string(data) +} + +// ParseFeatureEnvironment reads the comma-separated name=value list that Terragrunt accepts in +// TG_FEATURE. +// +//nolint:lintroller // Standalone SDK-only plugin has no host perf runtime dependency. +func ParseFeatureEnvironment(value string) (map[string]string, error) { + result := map[string]string{} + for _, entry := range strings.Split(value, ",") { + entry = strings.TrimSpace(entry) + if entry == "" { + continue + } + name, raw, found := strings.Cut(entry, "=") + name = strings.TrimSpace(name) + if !found || name == "" { + return nil, evaluationSetting("TG_FEATURE", "", + fmt.Sprintf("Invalid feature override %q in TG_FEATURE", name), + "Each entry of TG_FEATURE must look like `name=value`.", + "Set `TG_FEATURE=name=value[,name2=value2]`.") + } + result[name] = strings.TrimSpace(raw) + } + return result, nil +} + +// unusedFeatureOverrides names the overrides that no discovered unit declares, in sorted order. +func unusedFeatureOverrides(overrides map[string]string, declared map[string]bool) []string { + var unused []string + for name := range overrides { + if !declared[name] { + unused = append(unused, name) + } + } + slices.Sort(unused) + return unused +} + +// WarningFeatureOverrideUnused is reported when an override names a feature no discovered unit declares. +const WarningFeatureOverrideUnused = "feature_override_unused" + +// WarningReadConfigBeforeHook is reported for a before_hook that lists the read-config lifecycle. +const WarningReadConfigBeforeHook = "read_config_before_hook_ignored" + +// featureOverrideWarnings names every override that matches no declared feature, so a typo is +// visible instead of silently leaving the default in place. It is a warning, not an error: an +// override can legitimately name a feature that only some revisions of the project declare. +func featureOverrideWarnings(overrides map[string]string, inventory []model.DiscoveredUnit) []model.Warning { + if len(overrides) == 0 { + return nil + } + declared := map[string]bool{} + for i := range inventory { + for name := range inventory[i].Config.FeatureFlags { + declared[name] = true + } + } + var warnings []model.Warning + for _, name := range unusedFeatureOverrides(overrides, declared) { + warnings = append(warnings, model.Warning{ + Code: WarningFeatureOverrideUnused, + Message: fmt.Sprintf("The feature override `%s` does not match any `feature` block of the discovered units; check the spelling in `components.terraform.terragrunt.features` or `TG_FEATURE`", name), + }) + } + return warnings +} + +// lifecycleWarnings reports hooks that can never run: Terragrunt runs read-config hooks as after +// hooks only, and ignores a before_hook that lists it. +func lifecycleWarnings(terraform *config.TerraformConfig, path string) []model.Warning { + if terraform == nil { + return nil + } + var warnings []model.Warning + for i := range terraform.BeforeHooks { + hook := &terraform.BeforeHooks[i] + if hook.If != nil && !*hook.If { + continue + } + for _, command := range hook.Commands { + if command == model.LifecycleReadConfig || command == model.LifecycleReadConfigAlias { + warnings = append(warnings, model.Warning{ + Code: WarningReadConfigBeforeHook, Path: path, + Message: fmt.Sprintf("before_hook %q lists `%s`, which only runs from an after_hook, so it never runs; Terragrunt ignores it as well", hook.Name, command), + }) + } + } + } + return warnings +} + +// validateEngine checks that the executable a unit selected with terraform_binary answers +// `version -json` the way Terraform and OpenTofu do, so a wrong path fails with the unit's name +// instead of as an obscure error from the first command that runs. +func validateEngine(ctx context.Context, request *model.Request, binary, path string) error { + var version struct { + Version string `json:"terraform_version"` + } + executable, err := resolveOnInvocationPath(binary, request.Environment) + if err == nil { + command := exec.CommandContext(ctx, executable, "version", "-json") + command.Env = make([]string, 0, len(request.Environment)) + for key, value := range request.Environment { + command.Env = append(command.Env, key+"="+value) + } + var data []byte + if data, err = command.Output(); err == nil { + err = json.Unmarshal(data, &version) + } + } + if err == nil && version.Version != "" { + return nil + } + return evaluationSetting("terraform_binary", path, + "`terraform_binary` is not a Terraform or OpenTofu executable", + "The selected executable does not answer `version -json` the way Terraform and OpenTofu do.", + "Point `terraform_binary` at `tofu`, `terraform`, or a wrapper that forwards to one of them, or remove it to use `components.terraform.command`.") +} + +// resolveOnInvocationPath resolves a bare executable name against the PATH of the invocation, which +// is not the PATH of this process. A name with a path separator, and any name when the invocation +// has no PATH, is returned as it is. On Windows the extensions of the invocation's PATHEXT apply. +func resolveOnInvocationPath(binary string, environment map[string]string) (string, error) { + search := environmentValue(environment, "PATH") + if search == "" || filepath.IsAbs(binary) || strings.ContainsAny(binary, `/\`) { + return binary, nil + } + extensions := []string{""} + if runtime.GOOS == "windows" { + pathext := cmp.Or(environmentValue(environment, "PATHEXT"), ".COM;.EXE;.BAT;.CMD") + extensions = strings.Split(strings.ToLower(pathext), ";") + if slices.Contains(extensions, strings.ToLower(filepath.Ext(binary))) { + extensions = []string{""} + } + } + for _, directory := range filepath.SplitList(search) { + for _, extension := range extensions { + candidate := filepath.Join(cmp.Or(directory, "."), binary+extension) + if isInvocationExecutable(candidate) { + return candidate, nil + } + } + } + return "", fmt.Errorf("%w: %q is not on the invocation PATH", exec.ErrNotFound, binary) +} + +// environmentValue reads a variable of the invocation, ignoring case on Windows where names have none. +func environmentValue(environment map[string]string, name string) string { + if value, found := environment[name]; found || runtime.GOOS != "windows" { + return value + } + for key, value := range environment { + if strings.EqualFold(key, name) { + return value + } + } + return "" +} + +// checkFeatureOverrides rejects an override the unit's feature declarations cannot accept, naming +// the feature. The evaluator reports a bad override as a null feature far from its cause, so the +// declarations are read first. Anything else is left to the real evaluation to report. +func checkFeatureOverrides(path string, overrides map[string]string, declared func() (*config.TerragruntConfig, error)) error { + if len(overrides) == 0 { + return nil + } + cfg, err := declared() + if err != nil { + return nil //nolint:nilerr // The evaluation that follows reports the same error with full context. + } + _, err = featureFlagValues(cfg.FeatureFlags, overrides, path) + return err +} + +// applyUnitSettings applies the parity settings of a fully evaluated unit and checks the +// executable it selected. +func applyUnitSettings(ctx context.Context, cfg *config.TerragruntConfig, request *model.Request, path string, result *model.Resolution) error { + if err := applyParity(cfg, request, path, result); err != nil { + return err + } + result.Warnings = append(result.Warnings, lifecycleWarnings(cfg.Terraform, path)...) + if result.TerraformBinary == "" { + return nil + } + if warning, overridden := binaryOverrideWarning(request, result.TerraformBinary, path); overridden { + result.Warnings = append(result.Warnings, warning) + return nil + } + return validateEngine(ctx, request, result.TerraformBinary, path) +} + +// executablePermissionBits selects any executable permission on POSIX. +const executablePermissionBits = 0o111 + +func isInvocationExecutable(path string) bool { + info, err := os.Stat(path) + return err == nil && !info.IsDir() && (runtime.GOOS == "windows" || info.Mode()&executablePermissionBits != 0) +} + +func validateRetryBlock(retry *config.RetryBlock, path string) error { + key := fmt.Sprintf("errors.retry %q", retry.Label) + if retry.MaxAttempts < 1 { + return errorsProblem(key, path, fmt.Sprintf("`max_attempts` is %d in %s", retry.MaxAttempts, key), "A retry policy must allow at least one attempt.", "Set `max_attempts` to 1 or more.") + } + if retry.SleepIntervalSec < 0 { + return errorsProblem(key, path, fmt.Sprintf("`sleep_interval_sec` is %d in %s", retry.SleepIntervalSec, key), "The sleep between attempts cannot be negative.", "Set `sleep_interval_sec` to 0 or more.") + } + if int64(retry.MaxAttempts) > math.MaxUint32 || int64(retry.SleepIntervalSec) > math.MaxUint32 { + return errorsProblem(key, path, fmt.Sprintf("A value is too large in %s", key), "`max_attempts` and `sleep_interval_sec` must fit in 32 bits.", fmt.Sprintf("Use values no larger than %d.", uint32(math.MaxUint32))) + } + if err := checkPatterns(key, path, retry.RetryableErrors); err != nil { + return err + } + return nil +} + +// resolveIAM resolves the role before the full parse evaluates AWS functions. The host +// assumes it and supplies the resulting session on the second call. +func (e *unitEvaluator) resolveIAM(ctx context.Context) (*model.Resolution, error) { + flags, err := config.PartialParseConfigFile(ctx, e.logger, e.environment, + e.pctx.WithDecodeList(config.TerragruntFlags, config.FeatureFlagsBlock, config.ExcludeBlock), e.path, nil) + if err != nil { + return nil, err + } + if PreventsRun(excludeFromConfig(flags.Exclude), OperationCommand(e.request.Command, e.request.Arguments)) { + return &model.Resolution{Path: e.path}, nil + } + iam, warnings, err := iamSettings(flags, e.request, e.path) + if err != nil { + return nil, err + } + return &model.Resolution{Path: e.path, IAM: iam, Warnings: warnings}, nil +} + +// checkRequestFeatureOverrides rejects feature overrides the unit does not declare. Backend +// reads do not evaluate feature flags. +func (e *unitEvaluator) checkRequestFeatureOverrides(ctx context.Context) error { + if e.request.Phase == "backend" { + return nil + } + return checkFeatureOverrides(e.path, e.request.Features, func() (*config.TerragruntConfig, error) { + // Read the declarations without the overrides, which are what is being checked. + declarations := e.pctx.WithDecodeList(config.FeatureFlagsBlock) + declarations.FeatureFlags = nil + return config.PartialParseConfigFile(ctx, e.logger, e.environment, declarations, e.path, nil) + }) +} + +// unitSettings applies the parity and IAM settings of a fully evaluated unit. Backend reads +// evaluate only the backend identity, so they skip them. +func unitSettings(ctx context.Context, cfg *config.TerragruntConfig, request *model.Request, path string, result *model.Resolution) error { + if request.Phase == "backend" { + return nil + } + if err := applyUnitSettings(ctx, cfg, request, path, result); err != nil { + return err + } + iam, warnings, err := iamSettings(cfg, request, path) + if err != nil { + return err + } + result.IAM, result.Warnings = iam, append(result.Warnings, warnings...) + return nil +} diff --git a/plugins/terragrunt/internal/engine/parity_hooks.go b/plugins/terragrunt/internal/engine/parity_hooks.go new file mode 100644 index 00000000000..a6b1ff0e2ac --- /dev/null +++ b/plugins/terragrunt/internal/engine/parity_hooks.go @@ -0,0 +1,208 @@ +package engine + +import ( + "os" + "path/filepath" + "slices" + "strings" + + "github.com/cloudposse/atmos/plugins/terragrunt/internal/model" +) + +// Names and limits of the built-in tflint hook. The executable is the hook program that Terragrunt +// replaces with its own tflint integration. +const ( + tflintExecutable = "tflint" + tflintExternalFlag = "--terragrunt-external-tflint" + tflintConfigName = ".tflint.hcl" + // The host resolves this marker against the provisioned module directory when the hook runs. + tflintOptionalVarFilePrefix = "--atmos-optional-var-file=" + // The host replaces this marker with one --var per input of the final prepared configuration, after it + // has merged the stack's variables, so that TFLint and Terraform see the same values. + tflintInputVariablesMarker = "--atmos-tflint-input-variables" + // The host searches for this config only when the selected hook executes. + tflintConfigSearchPrefix = "--atmos-tflint-config-search=" + // TFLint searches at most tflintMaxParents ancestors; this bounds the search for .tflint.hcl, like Terragrunt's default folder limit. + tflintMaxParents = 100 +) + +// isLifecycleCommand reports whether a hook command names a Terragrunt stage that runs in the +// unit's configuration directory instead of the module directory. +func isLifecycleCommand(command string) bool { + return command == model.LifecycleReadConfig || command == model.LifecycleReadConfigAlias || command == model.LifecycleInitFromModule +} + +// finalizeExecution prepares evaluated hooks for the host: lifecycle hooks default to the unit's +// configuration directory, as in Terragrunt, and the built-in tflint hook becomes the two tflint +// invocations Terragrunt runs. The unit directory is where terragrunt.hcl lives. +func finalizeExecution(execution *model.Execution, path string) { + if execution == nil { + return + } + execution.BeforeHooks = finalizeHooks(execution.BeforeHooks, execution, path) + execution.AfterHooks = finalizeHooks(execution.AfterHooks, execution, path) + execution.ErrorHooks = finalizeHooks(execution.ErrorHooks, execution, path) +} + +func finalizeHooks(hooks []model.Hook, execution *model.Execution, path string) []model.Hook { + var result []model.Hook + for i := range hooks { + for _, hook := range splitLifecycle(&hooks[i], filepath.Dir(path)) { + if len(hook.Execute) > 0 && hook.Execute[0] == tflintExecutable { + result = append(result, expandTflint(&hook, execution, path)...) + continue + } + result = append(result, hook) + } + } + return result +} + +// splitLifecycle gives lifecycle commands the unit directory as working directory. A hook that +// lists both lifecycle and Terraform commands, with no working_dir of its own, becomes two hooks so +// each command keeps its own default directory. +func splitLifecycle(hook *model.Hook, unitDir string) []model.Hook { + var lifecycle, other []string + for _, command := range hook.Commands { + if isLifecycleCommand(command) { + lifecycle = append(lifecycle, command) + } else { + other = append(other, command) + } + } + if len(lifecycle) == 0 || hook.WorkingDirectory != "" { + return []model.Hook{*hook} + } + first := *hook + first.Commands, first.WorkingDirectory = lifecycle, unitDir + if len(other) == 0 { + return []model.Hook{first} + } + second := *hook + second.Commands = other + return []model.Hook{first, second} +} + +// expandTflint replaces a hook that executes tflint with the two commands Terragrunt runs for it: +// `tflint --init` and the lint itself. Both run in the module directory against the .tflint.hcl +// that Terragrunt would find, with the unit's inputs and variable files passed as variables. A hook +// that lists several commands expands once per command, so each lint sees only the extra +// arguments of the command that is running. +func expandTflint(hook *model.Hook, execution *model.Execution, path string) []model.Hook { + arguments := slices.DeleteFunc(slices.Clone(hook.Execute), func(arg string) bool { return arg == tflintExternalFlag }) + configFile := tflintConfigSearchPrefix + filepath.Dir(path) + if explicit, found := explicitTflintConfig(arguments, filepath.Dir(path)); found { + configFile = explicit + } + passthrough := withoutConfigFlag(arguments[1:]) + var result []model.Hook + for _, command := range hook.Commands { + variables := append([]string{tflintInputVariablesMarker}, tflintExtraVariables(command, execution)...) + initHook := *hook + initHook.Commands = []string{command} + initHook.Execute = []string{tflintExecutable, "--init", "--config", configFile, "--chdir", "."} + runHook := initHook + runHook.Execute = append(append([]string{tflintExecutable, "--config", configFile, "--chdir", "."}, variables...), passthrough...) + // The module directory is where tflint runs; a working_dir would only change the lock key upstream. + initHook.WorkingDirectory, runHook.WorkingDirectory = "", "" + result = append(result, initHook, runHook) + } + return result +} + +// tflintConfig returns the configuration named by --config/-c, or the nearest .tflint.hcl. +func tflintConfig(arguments []string, unitDir string) (string, error) { + if explicit, found := explicitTflintConfig(arguments, unitDir); found { + return explicit, nil + } + dir := unitDir + for range tflintMaxParents { + candidate := filepath.Join(dir, tflintConfigName) + if _, err := os.Stat(candidate); err == nil { + return candidate, nil + } + parent := filepath.Dir(dir) + if parent == dir { + break + } + dir = parent + } + return "", os.ErrNotExist +} + +func explicitTflintConfig(arguments []string, unitDir string) (string, bool) { + for i, arg := range arguments { + for _, flag := range []string{"--config", "-c"} { + if arg == flag && len(arguments) > i+1 { + return anchored(arguments[i+1], unitDir), true + } + if value, found := strings.CutPrefix(arg, flag+"="); found { + return anchored(value, unitDir), true + } + } + } + return "", false +} + +func anchored(path, dir string) string { + if filepath.IsAbs(path) { + return path + } + return filepath.Join(dir, path) +} + +// withoutConfigFlag drops the --config/-c pair that tflintConfig already consumed. +func withoutConfigFlag(arguments []string) []string { + var result []string + for i := 0; i < len(arguments); i++ { + arg := arguments[i] + switch { + case arg == "--config" || arg == "-c": + i++ + case strings.HasPrefix(arg, "--config=") || strings.HasPrefix(arg, "-c="): + default: + result = append(result, arg) + } + } + return result +} + +// tflintExtraVariables mirrors the variables Terragrunt takes from extra_arguments that apply to +// the command the hook runs for. +func tflintExtraVariables(command string, execution *model.Execution) []string { + var variables []string + for i := range execution.ExtraArguments { + extra := &execution.ExtraArguments[i] + if slices.Contains(extra.Commands, command) { + variables = append(variables, extraArgumentVariables(extra)...) + } + } + return variables +} + +// extraArgumentVariables lists the --var and --var-file flags one extra_arguments block implies. +// Both `-var=x` and `-var x` forms are recognized, as Terraform accepts both. +// TF_VAR_ entries of env_vars are not turned into --var flags, unlike Terragrunt: a flag would +// override terraform.tfvars, while Terraform ranks the environment below it. The hook inherits the +// phase environment, and TFLint reads TF_VAR_ variables with the same precedence as Terraform. +func extraArgumentVariables(extra *model.ExtraArguments) []string { + var variables []string + for i := 0; i < len(extra.Arguments); i++ { + argument := extra.Arguments[i] + for _, flag := range []string{"var", "var-file"} { + if value, found := strings.CutPrefix(argument, "-"+flag+"="); found { + variables = append(variables, "--"+flag+"="+value) + } else if argument == "-"+flag && i+1 < len(extra.Arguments) { + i++ + variables = append(variables, "--"+flag+"="+extra.Arguments[i]) + } + } + } + for _, file := range extra.RequiredVarFiles { + variables = append(variables, "--var-file="+file) + } + for _, file := range extra.OptionalVarFiles { + variables = append(variables, tflintOptionalVarFilePrefix+file) + } + return variables +} diff --git a/plugins/terragrunt/internal/engine/policies.go b/plugins/terragrunt/internal/engine/policies.go index 2dc25a5c876..9625c805c07 100644 --- a/plugins/terragrunt/internal/engine/policies.go +++ b/plugins/terragrunt/internal/engine/policies.go @@ -6,9 +6,11 @@ import ( "errors" "fmt" "maps" + "os" "os/exec" "path/filepath" "slices" + "strconv" "strings" "github.com/gruntwork-io/terragrunt/pkg/atmosbridge" @@ -19,6 +21,8 @@ import ( "github.com/cloudposse/atmos/plugins/terragrunt/internal/model" ) +const currentDirectory = "." + // ErrVersionRejected reports a terraform or terragrunt version that violates a // declared version constraint. var ErrVersionRejected = errors.New("version constraint rejected") @@ -37,6 +41,40 @@ func plainLocalSource(source string) bool { return !strings.ContainsAny(path, "?:") && !strings.Contains(filepath.ToSlash(path), "//") } +// localModuleSubdir separates Terragrunt's local root//module spelling without +// cleaning away the delimiter. A subdirectory may descend within the root, +// but must never select a parent or an absolute path. A bare relative root whose first segment looks like +// a hostname is a remote shorthand unless existingRoot reports that it names a local directory. +func localModuleSubdir(source string, existingRoot func(root string) bool) (root, subdir string, ok bool) { + volume := filepath.VolumeName(source) + rest := source[len(volume):] + if strings.ContainsAny(rest, "?:") { + return "", "", false + } + index := strings.Index(filepath.ToSlash(rest), "//") + if index <= 0 { + return "", "", false + } + root, subdir = volume+rest[:index], rest[index+2:] + first := strings.SplitN(strings.TrimPrefix(root, "./"), "/", 2)[0] + if strings.Contains(first, ".") && !strings.HasPrefix(root, ".") && !filepath.IsAbs(root) && (existingRoot == nil || !existingRoot(root)) { + return "", "", false + } + return root, subdir, true +} + +func safeModuleSubdir(subdir string) bool { + if subdir == "" || filepath.IsAbs(subdir) || filepath.VolumeName(subdir) != "" || strings.HasPrefix(subdir, "/") || strings.HasPrefix(subdir, `\`) { + return false + } + for _, part := range strings.Split(filepath.ToSlash(subdir), "/") { + if part == "" || part == ".." || part == currentDirectory { + return false + } + } + return true +} + func checkVersions(ctx context.Context, request *model.Request, terraform, terragrunt string) error { check := func(name, constraint, actual string) error { if constraint == "" { @@ -124,17 +162,16 @@ func localSubdirSource(source string) bool { if _, ok := copiedSourceRoot("", source); !ok { return false } - return filepath.IsAbs(source) || strings.HasPrefix(source, ".") + return filepath.IsAbs(source) || strings.HasPrefix(source, currentDirectory) } -// validateRemoteSource preserves the getter URI for Atmos's source provisioner. -// Registry sources need Terragrunt version selection and are not interchangeable -// with direct archive or Git sources. +// validateRemoteSource preserves the getter URI for Atmos's source provisioner. A `tfr://` registry +// source is resolved to one of these URIs when the unit is prepared. func validateRemoteSource(source string) error { if localSubdirSource(source) { return nil } - for _, prefix := range []string{"git::https://", "git::http://", "git::ssh://", "git::file://", "https://", "http://"} { + for _, prefix := range []string{"git::https://", "git::http://", "git::ssh://", "git::file://", "https://", "http://", registryScheme} { if strings.HasPrefix(source, prefix) { return nil } @@ -142,12 +179,80 @@ func validateRemoteSource(source string) error { scheme, _, found := strings.Cut(source, "://") return &model.Problem{ Code: model.CodeInvalidSource, Summary: "Unsupported module source transport", - Detail: fmt.Sprintf("The module source scheme %q is not supported. Registry sources such as `tfr://` need Terragrunt's version selection and are not interchangeable with Git or archive sources.", scheme), - Hints: []string{"Use a local path, a `git::https://`, `git::ssh://`, or `https://` source, or a Git source pinned with `?ref=`."}, + Detail: fmt.Sprintf("The module source scheme %q is not supported.", scheme), + Hints: []string{"Use a local path, a `git::https://`, `git::ssh://`, or `https://` source, a `tfr://` registry source, or a Git source pinned with `?ref=`."}, Key: "terraform.source", Safe: found, Err: ErrUnsupportedExecution, } } +// decimalBase is the base of the IAM session duration, a count of seconds. +const decimalBase = 10 + +// iamDurationBits is the width of the signed IAM duration field. +const iamDurationBits = 64 + +// Warning code for IAM settings that have no effect. +const warningIAMTokenWithoutRole = "iam_web_identity_token_without_role" + +// iamEnvironment names the variables that Terragrunt reads its IAM options from, newest name first. +// Like its --iam-assume-role flags, they take precedence over the same settings in the unit. +var iamEnvironment = map[string][]string{ + "role": {"TG_IAM_ASSUME_ROLE", "TERRAGRUNT_IAM_ROLE"}, + "duration": {"TG_IAM_ASSUME_ROLE_DURATION", "TERRAGRUNT_IAM_ASSUME_ROLE_DURATION"}, + "session": {"TG_IAM_ASSUME_ROLE_SESSION_NAME", "TERRAGRUNT_IAM_ASSUME_ROLE_SESSION_NAME"}, + "token": {"TG_IAM_ASSUME_ROLE_WEB_IDENTITY_TOKEN", "TERRAGRUNT_IAM_WEB_IDENTITY_TOKEN", "TERRAGRUNT_IAM_ASSUME_ROLE_WEB_IDENTITY_TOKEN"}, +} + +// iamSettings returns the role the unit asks for, or nil. Terragrunt assumes it only when a role ARN is +// present, so a token alone is reported and ignored. The host assumes the role, not the plugin. +func iamSettings(cfg *config.TerragruntConfig, request *model.Request, path string) (*model.IAM, []model.Warning, error) { + pick := func(key, configured string) string { + for _, name := range iamEnvironment[key] { + if value := request.Environment[name]; value != "" { + return value + } + } + return configured + } + var configuredDuration int64 + if cfg.IamAssumeRoleDuration != nil { + configuredDuration = *cfg.IamAssumeRoleDuration + } + iam := &model.IAM{ + Role: pick("role", cfg.IamRole), SessionName: pick("session", cfg.IamAssumeRoleSessionName), + DurationSeconds: configuredDuration, WebIdentityToken: pick("token", cfg.IamWebIdentityToken), + } + if configured := pick("duration", ""); configured != "" { + seconds, err := strconv.ParseInt(configured, decimalBase, iamDurationBits) + if err != nil || seconds < 0 { + return nil, nil, evaluationSetting("iam_assume_role_duration", path, "Invalid IAM session duration", + "The IAM session duration from the environment is not a number of seconds.", "Set `TG_IAM_ASSUME_ROLE_DURATION` to a number of seconds, for example `3600`.") + } + iam.DurationSeconds = seconds + } + if iam.Role != "" { + iam.WebIdentityToken = unitTokenFile(iam.WebIdentityToken, filepath.Dir(path)) + return iam, nil, nil + } + if iam.WebIdentityToken == "" { + return nil, nil, nil + } + return nil, []model.Warning{{Code: warningIAMTokenWithoutRole, Path: mustRel(request.BasePath, path), Message: "`iam_web_identity_token` has no effect without `iam_role`, so no role is assumed. Terragrunt behaves the same way."}}, nil +} + +// unitTokenFile anchors a relative web identity token file at the unit directory, so the host finds it +// whatever directory Atmos runs from. Absolute paths and token values that name no file are unchanged. +func unitTokenFile(token, unitDir string) string { + if token == "" || filepath.IsAbs(token) { + return token + } + candidate := filepath.Join(unitDir, token) + if info, err := os.Stat(candidate); err == nil && info.Mode().IsRegular() { + return candidate + } + return token +} + // effectivePolicyArguments matches engine insertion order: command environment, // global environment, extra arguments, then explicit arguments. Matching extra // argument blocks override environment values in declaration order. diff --git a/plugins/terragrunt/internal/engine/problems.go b/plugins/terragrunt/internal/engine/problems.go index 0ba29f487a3..66fe2d63a62 100644 --- a/plugins/terragrunt/internal/engine/problems.go +++ b/plugins/terragrunt/internal/engine/problems.go @@ -19,30 +19,6 @@ import ( // or using the network. const readOnlyDenial = "read-only discovery cannot" -// unsupportedSettingHints maps each setting the native adaptation rejects to its remedy. -var unsupportedSettingHints = map[string]string{ - "terraform_binary": "Select the executable with `components.terraform.command` in atmos.yaml, for example `tofu`, and remove `terraform_binary` from the unit.", - "errors": "Remove the `errors` block. Retry failing steps with a workflow step `retry:` setting instead.", - "exclude": "Remove the `exclude` block and leave the unit out with `discovery.excluded_paths` in atmos.yaml.", - "iam_role": "Remove the IAM setting and authenticate with an Atmos identity, for example `atmos terraform plan --identity `.", - "iam_web_identity_token": "Remove `iam_web_identity_token` and authenticate with an Atmos identity, for example `atmos terraform plan --identity `.", - "feature": "Remove the `feature` blocks and pass the values as Atmos stack variables instead.", -} - -// unsupportedSetting reports a Terragrunt setting the native adaptation does not implement. -func unsupportedSetting(key, file string) *model.Problem { - hint := unsupportedSettingHints[key] - problem := &model.Problem{ - Code: model.CodeUnsupportedSetting, Summary: fmt.Sprintf("Unsupported Terragrunt setting `%s`", key), - Detail: fmt.Sprintf("The native Terragrunt adaptation does not implement `%s`, so this unit cannot be planned or applied.", key), - File: file, Key: key, Safe: true, Err: ErrUnsupportedExecution, - } - if hint != "" { - problem.Hints = []string{hint} - } - return problem -} - // problemFor classifies an evaluation error. A Problem passes through with its file filled in. // An HCL diagnostic becomes a syntax or evaluation problem at the diagnostic's position. func problemFor(err error, file string) error { @@ -120,20 +96,6 @@ func unsupportedHook(key, summary, detail string, hints ...string) *model.Proble // checks everything a partial or full parse decodes, in a fixed order, so the first rejection is // stable. Hook and extra_arguments semantics are checked by executionFromConfig. func rejectUnsupported(cfg *config.TerragruntConfig, path string) error { - switch { - case cfg.TerraformBinary != "": - return unsupportedSetting("terraform_binary", path) - case cfg.Errors != nil: - return unsupportedSetting("errors", path) - case cfg.Exclude != nil: - return unsupportedSetting("exclude", path) - case len(cfg.FeatureFlags) > 0: - return unsupportedSetting("feature", path) - case cfg.IamRole != "": - return unsupportedSetting("iam_role", path) - case cfg.IamWebIdentityToken != "": - return unsupportedSetting("iam_web_identity_token", path) - } if cfg.Terraform != nil { if _, err := executionFromConfig(cfg.Terraform); err != nil { return hookProblem(err, path) @@ -156,7 +118,7 @@ func missingSourceProblem(path string) *model.Problem { Code: model.CodeMissingSource, Summary: "The unit has no `terraform.source`", Detail: "Native Terragrunt support runs units that name a module with `terraform { source = ... }`. A unit with inline Terraform files is not supported.", Hints: []string{"Add `terraform { source = \"...\" }` to the unit, or move the inline Terraform files into a module directory and reference it."}, - File: path, Key: "terraform.source", Safe: true, Err: ErrUnsupportedExecution, + File: path, Key: terraformSourceKey, Safe: true, Err: ErrUnsupportedExecution, } } @@ -167,7 +129,7 @@ func sourceDirectoryProblem(path, source string) *model.Problem { Code: model.CodeInvalidSource, Summary: "The module `source` is not an existing local directory", Detail: "`terraform.source` resolves to a local path that does not exist or is not a directory.", Hints: []string{"Point `terraform.source` at a module directory relative to the unit, or use a Git or HTTPS module source."}, - File: path, Key: "terraform.source", Safe: true, Err: fmt.Errorf("%w: local source %q is not a directory", ErrUnsupportedExecution, source), + File: path, Key: terraformSourceKey, Safe: true, Err: fmt.Errorf("%w: local source %q is not a directory", ErrUnsupportedExecution, source), } } @@ -203,12 +165,47 @@ func localSourceProblem(filesystem afero.Fs, path, source, resolved string) *mod Code: model.CodeInvalidSource, Summary: "The module `source` directory cannot be read", Detail: "Checking the local `terraform.source` directory failed with a filesystem error other than a missing path.", Hints: []string{"Check the permissions of the module directory and its parent directories."}, - File: path, Key: "terraform.source", Safe: true, Err: err, + File: path, Key: terraformSourceKey, Safe: true, Err: err, } } return nil } +// resolveLocalModuleSource validates both the repository root and selected +// module while retaining // in the source identity passed to the provisioner. +func resolveLocalModuleSource(filesystem afero.Fs, unit, source string) (*localModuleSource, *model.Problem) { + base, subdir, hasSubdir := localModuleSubdir(source, func(candidate string) bool { + exists, err := afero.DirExists(filesystem, localPath(filepath.Dir(unit), candidate)) + return err == nil && exists + }) + if !hasSubdir && !plainLocalSource(source) { + return nil, nil + } + if hasSubdir && !safeModuleSubdir(subdir) { + return nil, &model.Problem{ + Code: model.CodeInvalidSource, Summary: "The local module subdirectory is invalid", + Detail: "A local `terraform.source` subdirectory must stay inside its source root.", + Hints: []string{"Use `root//modules/name` without parent (`..`) or absolute subdirectory segments."}, + File: unit, Key: terraformSourceKey, Safe: true, Err: ErrUnsupportedExecution, + } + } + if !hasSubdir { + base = source + } + root := localPath(filepath.Dir(unit), base) + module := root + if hasSubdir { + module = filepath.Join(root, filepath.FromSlash(subdir)) + } + if problem := validateLocalModuleDirectories(filesystem, unit, source, root, module); problem != nil { + return nil, problem + } + if hasSubdir { + return &localModuleSource{source: root + "//" + filepath.ToSlash(filepath.FromSlash(subdir)), root: root}, nil + } + return &localModuleSource{source: root, root: root}, nil +} + // dependencyProblem keeps evaluated target paths out of safe details; the cause remains available // for classification and local debugging. func dependencyProblem(file, _ string, err error) *model.Problem { @@ -221,20 +218,90 @@ func dependencyProblem(file, _ string, err error) *model.Problem { return &model.Problem{Code: model.CodeDependency, Summary: summary, Detail: "The dependency target could not be read.", Hints: []string{hint}, File: file, Key: "dependency", Safe: true, Err: err} } -// localSubdirSourceProblem checks a local `dir//subdir` module source. Terragrunt copies the tree -// before `//` and runs the subdirectory, so both must exist; each is reported like a plain local -// source. A source without a local copied tree has nothing to check. -func localSubdirSourceProblem(filesystem afero.Fs, path, source string) *model.Problem { - root, ok := copiedSourceRoot(filepath.Dir(path), source) - if !ok { - return nil +// WarningTerraformBinaryOverridden is reported for a unit whose terraform_binary is not the +// executable Atmos will run because a stack `command` or the command line selected another one. +const WarningTerraformBinaryOverridden = "terraform_binary_overridden" + +// effectiveBinaryEnvironment carries the executable the host resolved for the component, after +// applying the stack `command` and command-line overrides. +const effectiveBinaryEnvironment = "ATMOS_TERRAFORM_BINARY" + +// binaryOverrideWarning reports whether the stack or the command line replaced the unit's +// terraform_binary with another executable of the same engine. The unit's binary never runs then, +// so probing it would fail a unit on an executable that is not used; the caller skips the probe and +// reports the returned warning instead. +// +// A string comparison cannot tell an override from the default, so the resolved files are compared: +// the unit's binary on the invocation PATH against the host's effective binary, both with symlinks +// evaluated. A unit binary that cannot be resolved while the effective one differs counts as an +// override, because the host already refused to run when the unit's binary was the effective one and +// was missing. The host and this plugin may see different PATHs; a false "override" only skips an +// advisory probe and adds a warning, so the check fails open. +func binaryOverrideWarning(request *model.Request, binary, path string) (model.Warning, bool) { + effective := environmentValue(request.Environment, effectiveBinaryEnvironment) + if effective == "" { + return model.Warning{}, false + } + if resolved, err := resolveOnInvocationPath(binary, request.Environment); err == nil && sameExecutable(resolved, effective) { + return model.Warning{}, false } - if problem := localSourceProblem(filesystem, path, source, root); problem != nil { + return model.Warning{ + Code: WarningTerraformBinaryOverridden, Path: path, + Message: fmt.Sprintf("The stack `command` %s overrides this unit's `terraform_binary` %s; the unit's binary is not used. "+ + "Remove `terraform_binary` from the unit or the `command` override from the stack to make them agree.", effective, binary), + }, true +} + +// sameExecutable reports whether two paths name the same file, following symlinks. A path that +// cannot be inspected is compared by its cleaned spelling. +func sameExecutable(a, b string) bool { + a, b = evalSymlinksOrClean(a), evalSymlinksOrClean(b) + infoA, errA := os.Stat(a) + infoB, errB := os.Stat(b) + if errA == nil && errB == nil { + return os.SameFile(infoA, infoB) + } + return a == b +} + +// evalSymlinksOrClean resolves symlinks where the path exists and cleans it otherwise. +func evalSymlinksOrClean(path string) string { + if resolved, err := filepath.EvalSymlinks(path); err == nil { + return filepath.Clean(resolved) + } + return filepath.Clean(path) +} + +// evaluationSetting describes an invalid native setting without exposing evaluated values. +func evaluationSetting(key, file, summary, detail string, hints ...string) *model.Problem { + return &model.Problem{Code: model.CodeEvaluationError, Key: key, File: file, Summary: summary, Detail: detail, Hints: hints, Safe: true, Err: ErrUnsupportedExecution} +} + +// terraformSourceKey identifies module-source diagnostics. +const terraformSourceKey = "terraform.source" + +func validateLocalModuleDirectories(filesystem afero.Fs, unit, source, root, module string) *model.Problem { + if problem := localSourceProblem(filesystem, unit, source, root); problem != nil { return problem } - _, subdirectory, _ := strings.Cut(filepath.ToSlash(source), "//") - if subdirectory == "" { - return nil + if problem := localSourceProblem(filesystem, unit, source, module); problem != nil { + return problem } - return localSourceProblem(filesystem, path, source, filepath.Join(root, filepath.FromSlash(subdirectory))) + if root != module { + canonicalRoot, rootErr := canonicalPath(root) + canonicalModule, moduleErr := canonicalPath(module) + if rootErr == nil && moduleErr == nil && !within(canonicalModule, canonicalRoot) { + return &model.Problem{ + Code: model.CodeInvalidSource, Summary: "The local module subdirectory escapes its source root", + Detail: "The selected module resolves outside the local source root.", + Hints: []string{"Select a module directory inside the source root."}, + File: unit, Key: terraformSourceKey, Safe: true, Err: ErrUnsupportedExecution, + } + } + } + + return nil } + +// localModuleSource retains the selected source identity and its watched repository root. +type localModuleSource struct{ source, root string } diff --git a/plugins/terragrunt/internal/engine/problems_test.go b/plugins/terragrunt/internal/engine/problems_test.go index f12cbd741ab..013b9d35497 100644 --- a/plugins/terragrunt/internal/engine/problems_test.go +++ b/plugins/terragrunt/internal/engine/problems_test.go @@ -7,7 +7,6 @@ import ( "io/fs" "os" "path/filepath" - "strings" "testing" "github.com/hashicorp/hcl/v2" @@ -24,46 +23,6 @@ func asProblem(t *testing.T, err error) *model.Problem { return problem } -// Every unsupported setting is rejected during discovery, so a bulk run can validate each selected -// unit before any infrastructure command runs. -func TestDiscoveryRejectsUnsupportedSettingsNamingTheKey(t *testing.T) { - for name, test := range map[string]struct{ extra, key string }{ - "terraform_binary": {`terraform_binary = "tofu"`, "terraform_binary"}, - "errors": {"errors {\n retry \"x\" {\n retryable_errors = [\".*\"]\n max_attempts = 2\n sleep_interval_sec = 1\n }\n}", "errors"}, - "exclude": {"exclude {\n if = true\n actions = [\"plan\"]\n}", "exclude"}, - "iam": {`iam_role = "arn:aws:iam::1:role/x"`, "iam_role"}, - "web identity": {`iam_web_identity_token = "token"`, "iam_web_identity_token"}, - "feature": {"feature \"flag\" {\n default = true\n}", "feature"}, - } { - t.Run(name, func(t *testing.T) { - request := nativeUnit(t, test.extra) - request.Phase = "discover" - _, err := Resolve(t.Context(), request) - problem := asProblem(t, err) - require.Equal(t, model.CodeUnsupportedSetting, problem.Code) - require.Equal(t, test.key, problem.Key) - require.Equal(t, "terragrunt.hcl", filepath.Base(problem.File)) - require.True(t, problem.Safe) - require.NotEmpty(t, problem.Hints) - }) - } -} - -func TestDiscoveryRejectsUnsupportedHooks(t *testing.T) { - request := nativeUnit(t, "") - body, err := os.ReadFile(request.Path) - require.NoError(t, err) - hooked := strings.Replace(string(body), "{ source", "{\n source", 1) - hooked = strings.Replace(hooked, " }\n", "\n before_hook \"lint\" {\n commands = [\"plan\"]\n execute = [\"tflint\"]\n }\n}\n", 1) - require.NoError(t, os.WriteFile(request.Path, []byte(hooked), 0o644)) - request.Phase = "discover" - _, err = Resolve(t.Context(), request) - problem := asProblem(t, err) - require.Equal(t, model.CodeUnsupportedHook, problem.Code) - require.Contains(t, problem.Key, "lint") - require.Equal(t, "terragrunt.hcl", filepath.Base(problem.File)) -} - func TestSyntaxErrorNamesFileLineAndColumnAndStaysSafe(t *testing.T) { request := nativeUnit(t, "inputs = {\n a = find_in_parent_folders(\"x\"\n}\n") request.Phase = "discover" diff --git a/plugins/terragrunt/internal/engine/registry.go b/plugins/terragrunt/internal/engine/registry.go new file mode 100644 index 00000000000..a1c39d5c2c1 --- /dev/null +++ b/plugins/terragrunt/internal/engine/registry.go @@ -0,0 +1,228 @@ +package engine + +import ( + "context" + "errors" + "fmt" + "net/http" + "regexp" + "strings" + "time" + + "github.com/gruntwork-io/terragrunt/pkg/atmosbridge" + "github.com/gruntwork-io/terragrunt/pkg/config" + "github.com/gruntwork-io/terragrunt/pkg/log" + + "github.com/cloudposse/atmos/plugins/terragrunt/internal/model" +) + +// registryScheme prefixes Terraform module registry sources. +const registryScheme = "tfr://" + +// registryClient is the HTTP client for registry requests. It is nil in production, which selects the +// default transport; tests substitute a client that trusts their registry. +var registryClient *http.Client + +// prepareRemoteSource validates a remote module source and resolves a registry source to the getter URL +// that the Atmos source provisioner downloads. Resolution talks to the registry, so it only happens when +// the unit is prepared for a command: discovery and state identity keep the source as written. +func prepareRemoteSource(ctx context.Context, logger log.Logger, request *model.Request, cfg *config.TerragruntConfig, result *model.Resolution) error { + if err := validateRemoteSource(result.Source); err != nil { + return err + } + if !strings.HasPrefix(result.Source, registryScheme) || request.Phase == "backend" { + return nil + } + constraint := registryVersionConstraint(cfg) + var auth *model.RegistryAuth + started := time.Now() + resolved, ignored, err := retryRegistryResolution(ctx, result.Errors, func() (string, error) { + resolution, err := atmosbridge.ResolveRegistrySourceWithAuth(ctx, logger, atmosbridge.RegistryRequest{ + Client: registryClient, Source: result.Source, Constraint: constraint, Environment: request.Environment, OpenTofu: IsOpenTofu(request.Tofu), + }) + if err == nil && resolution.Authorization != "" { + auth = &model.RegistryAuth{Origin: resolution.AuthOrigin, Authorization: resolution.Authorization} + } + return resolution.Source, err + }) + if ignored != nil { + // No source was resolved, so the unit keeps the source it was written with and the host ends its + // run without executing it, as it does for an ignored source fetch. + result.RunIgnored = &model.IgnoredRun{Policy: ignored.Name, Message: ignored.Message} + return nil + } + if err != nil { + return registryProblem(result.Source, err) + } + if elapsed := time.Since(started); elapsed > 0 { + result.RegistryResolutionNanos = uint64(elapsed) + } + original := result.Source + resolved = normalizeGetterURL(resolved) + if err := validateRemoteSource(resolved); err != nil { + return rejectedResolvedSource(original, resolved, err) + } + result.Source = resolved + result.RegistryAuth = auth + return nil +} + +// retryRegistryResolution applies the unit's source-fetch policies to registry discovery and version +// lookup, which happen before the host fetch. As for a source fetch, ignore policies are read before +// retry policies: a failure an ignore policy matches is not retried, and the matching policy is +// returned with the failure so the caller can end the unit's run. +func retryRegistryResolution(ctx context.Context, policies *model.Errors, resolve func() (string, error)) (string, *model.IgnorePolicy, error) { + for attempt := 1; ; attempt++ { + resolved, err := resolve() + if err == nil || policies == nil || ctx.Err() != nil { + return resolved, nil, err + } + text := cleanRegistryError(err.Error()) + if ignore := matchingRegistryIgnore(policies.Ignore, text); ignore != nil { + return "", ignore, err + } + retry := matchingRegistryRetry(policies.Retry, text) + if retry == nil || attempt >= retry.MaxAttempts { + return "", nil, err + } + timer := time.NewTimer(time.Duration(retry.SleepIntervalSec) * time.Second) + select { + case <-ctx.Done(): + timer.Stop() + return "", nil, ctx.Err() + case <-timer.C: + } + } +} + +// registryErrorNoise matches everything that is not part of a readable error message. The host cleans +// the text of a failed source fetch the same way before matching, so one pattern behaves alike in both. +var registryErrorNoise = regexp.MustCompile(`[^a-zA-Z0-9./'"():=\- ]+`) + +// cleanRegistryError reduces a failure to the readable text that ignore and retry patterns match. The +// cleaning is idempotent, so text that is already clean passes through unchanged. +func cleanRegistryError(message string) string { + return strings.Join(strings.Fields(registryErrorNoise.ReplaceAllString(message, " ")), " ") +} + +// matchingRegistryIgnore returns the first ignore policy, in declaration order, that matches the failure. +func matchingRegistryIgnore(policies []model.IgnorePolicy, message string) *model.IgnorePolicy { + text := cleanRegistryError(message) + for i := range policies { + if registryErrorMatches(text, policies[i].Patterns) { + return &policies[i] + } + } + return nil +} + +func registryErrorMatches(message string, patterns []string) bool { + matched := false + for _, pattern := range patterns { + negative := strings.HasPrefix(pattern, "!") + expression, err := regexp.Compile(strings.TrimPrefix(pattern, "!")) + if err == nil && expression.MatchString(message) { + if negative { + return false + } + matched = true + } + } + return matched +} + +// IsOpenTofu reports whether the selected executable is OpenTofu. It is the one rule of the host and the +// plugin: any executable whose name contains "tofu", in any case, is OpenTofu, whichever separator its +// path uses. Everything else is Terraform. +// +//nolint:lintroller // Standalone SDK-only plugin has no host perf runtime dependency. +func IsOpenTofu(binary string) bool { + name := binary + if index := strings.LastIndexAny(name, `/\`); index >= 0 { + name = name[index+1:] + } + return strings.Contains(strings.ToLower(name), "tofu") +} + +// normalizeGetterURL spells the host shorthands a registry may return as the explicit Git URLs that +// validation and the source provisioner accept, for example github.com/org/repo?ref=v1. +func normalizeGetterURL(source string) string { + for _, host := range []string{"github.com/", "bitbucket.org/"} { + if !strings.HasPrefix(source, host) { + continue + } + location, query, hasQuery := strings.Cut(source, "?") + repository, subdir, hasSubdir := strings.Cut(location, "//") + // Like go-getter, a shorthand names the repository with its first three path segments + // (host/owner/name); anything after them is a subdirectory of that repository. + if segments := strings.SplitN(repository, "/", 4); !hasSubdir && len(segments) == 4 { + repository, subdir, hasSubdir = strings.Join(segments[:3], "/"), segments[3], true + } + if !strings.HasSuffix(repository, ".git") { + repository += ".git" + } + normalized := "git::https://" + repository + if hasSubdir { + normalized += "//" + subdir + } + if hasQuery { + normalized += "?" + query + } + return normalized + } + return source +} + +// registryProblem describes a registry source that could not be resolved. The error text names the +// registry URL and status but never a credential, though a registry body can echo arbitrary text, so +// it is not marked safe. +func registryProblem(source string, err error) *model.Problem { + return &model.Problem{ + Code: model.CodeInvalidSource, Summary: "The module registry could not resolve the `tfr://` source", + Detail: fmt.Sprintf("Resolving %q failed: %s", source, err.Error()), + Hints: []string{ + "Check the module path (`namespace/name/provider`), the `version`, and that the registry host is reachable.", + "For a private registry, set `TF_TOKEN_` (dots as underscores), add a `credentials` block to the CLI configuration, or set `TG_TF_REGISTRY_TOKEN`.", + }, + Key: "terraform.source", Err: err, + } +} + +// urlUserinfo matches the credentials in front of the host of a URL, also inside a getter-prefixed URL such as +// git::https://user:token@host/repo. +var urlUserinfo = regexp.MustCompile(`(://)[^/@\s]*@`) + +// redactURLCredentials removes the userinfo from every URL in text, so a credential a registry embedded in a +// getter URL cannot reach a diagnostic that is shown to the user. +func redactURLCredentials(text string) string { + return urlUserinfo.ReplaceAllString(text, "$1") +} + +// rejectedResolvedSource names both the registry source a unit was written with and the getter URL the +// registry returned for it, so a rejected result can be traced to the source that produced it. The detail may be +// shown as safe, so credentials in the resolved URL are removed first. +func rejectedResolvedSource(original, resolved string, err error) error { + var problem *model.Problem + if !errors.As(err, &problem) { + return err + } + annotated := *problem + annotated.Detail = fmt.Sprintf("%s The registry source %q resolved to %q.", problem.Detail, original, redactURLCredentials(resolved)) + return &annotated +} + +func registryVersionConstraint(cfg *config.TerragruntConfig) string { + if cfg.Terraform != nil && cfg.Terraform.Version != nil { + return *cfg.Terraform.Version + } + return "" +} + +func matchingRegistryRetry(policies []model.RetryPolicy, text string) *model.RetryPolicy { + for i := range policies { + if registryErrorMatches(text, policies[i].Patterns) { + return &policies[i] + } + } + return nil +} diff --git a/plugins/terragrunt/internal/engine/registry_test.go b/plugins/terragrunt/internal/engine/registry_test.go new file mode 100644 index 00000000000..b80980d53f1 --- /dev/null +++ b/plugins/terragrunt/internal/engine/registry_test.go @@ -0,0 +1,522 @@ +package engine + +import ( + "encoding/json" + "errors" + "fmt" + "net/http" + "net/http/httptest" + "net/url" + "os" + "path/filepath" + "sync/atomic" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "github.com/cloudposse/atmos/plugins/terragrunt/internal/model" +) + +// registryServer serves the Terraform module registry protocol for acme/vpc/aws over TLS and counts requests. +func registryServer(t *testing.T) (host string, requests *atomic.Int32) { + t.Helper() + requests = &atomic.Int32{} + mux := http.NewServeMux() + mux.HandleFunc("/.well-known/terraform.json", func(w http.ResponseWriter, _ *http.Request) { + requests.Add(1) + fmt.Fprint(w, `{"modules.v1":"/v1/modules/"}`) + }) + mux.HandleFunc("/v1/modules/acme/vpc/aws/versions", func(w http.ResponseWriter, _ *http.Request) { + requests.Add(1) + fmt.Fprint(w, `{"modules":[{"versions":[{"version":"1.0.0"},{"version":"1.4.0"},{"version":"2.0.0"}]}]}`) + }) + mux.HandleFunc("/v1/modules/acme/vpc/aws/", func(w http.ResponseWriter, r *http.Request) { + requests.Add(1) + w.Header().Set("X-Terraform-Get", "github.com/acme/terraform-aws-vpc?ref=v"+filepath.Base(filepath.Dir(r.URL.Path))) + w.WriteHeader(http.StatusNoContent) + }) + server := httptest.NewTLSServer(mux) + t.Cleanup(server.Close) + previous := registryClient + registryClient = server.Client() + t.Cleanup(func() { registryClient = previous }) + address, err := url.Parse(server.URL) + require.NoError(t, err) + return address.Host, requests +} + +func registryUnit(t *testing.T, terraformBody string) model.Request { + t.Helper() + root := t.TempDir() + path := filepath.Join(root, "terragrunt.hcl") + require.NoError(t, os.WriteFile(path, []byte("terraform {\n"+terraformBody+"\n}\n"), 0o644)) + return model.Request{Path: path, BasePath: root, Command: "plan", Tofu: "/bin/tofu", Environment: map[string]string{"HOME": root}} +} + +func TestRegistryDiscoveryRetriesTransientFailure(t *testing.T) { + var discoveryCalls atomic.Int32 + mux := http.NewServeMux() + mux.HandleFunc("/.well-known/terraform.json", func(w http.ResponseWriter, _ *http.Request) { + if discoveryCalls.Add(1) == 1 { + http.Error(w, "temporary", http.StatusServiceUnavailable) + return + } + fmt.Fprint(w, `{"modules.v1":"/v1/modules/"}`) + }) + mux.HandleFunc("/v1/modules/acme/vpc/aws/", func(w http.ResponseWriter, _ *http.Request) { + w.Header().Set("X-Terraform-Get", "https://example.test/module.zip") + w.WriteHeader(http.StatusNoContent) + }) + server := httptest.NewTLSServer(mux) + defer server.Close() + previous := registryClient + registryClient = server.Client() + defer func() { registryClient = previous }() + address, err := url.Parse(server.URL) + require.NoError(t, err) + request := registryUnit(t, `source = "tfr://`+address.Host+`/acme/vpc/aws?version=1.0.0"`) + file := `terraform { + source = "tfr://` + address.Host + `/acme/vpc/aws?version=1.0.0" +} +errors { + retry "registry" { + retryable_errors = ["status code 503"] + max_attempts = 2 + sleep_interval_sec = 0 + } +}` + require.NoError(t, os.WriteFile(request.Path, []byte(file), 0o644)) + result, err := Resolve(t.Context(), request) + require.NoError(t, err) + require.Equal(t, "https://example.test/module.zip", result.Source) + require.EqualValues(t, 2, discoveryCalls.Load()) +} + +// failingRegistryUnit writes a unit whose registry always answers 503 and counts the discovery requests. +func failingRegistryUnit(t *testing.T, policies string) (request model.Request, discoveryCalls *atomic.Int32) { + t.Helper() + discoveryCalls = &atomic.Int32{} + mux := http.NewServeMux() + mux.HandleFunc("/.well-known/terraform.json", func(w http.ResponseWriter, _ *http.Request) { + discoveryCalls.Add(1) + http.Error(w, "temporary", http.StatusServiceUnavailable) + }) + server := httptest.NewTLSServer(mux) + t.Cleanup(server.Close) + previous := registryClient + registryClient = server.Client() + t.Cleanup(func() { registryClient = previous }) + address, err := url.Parse(server.URL) + require.NoError(t, err) + source := "tfr://" + address.Host + "/acme/vpc/aws?version=1.0.0" + request = registryUnit(t, `source = "`+source+`"`) + file := "terraform {\n source = \"" + source + "\"\n}\nerrors {\n" + policies + "\n}\n" + require.NoError(t, os.WriteFile(request.Path, []byte(file), 0o644)) + return request, discoveryCalls +} + +const registryRetryPolicy = ` retry "registry" { + retryable_errors = ["status code 503"] + max_attempts = 3 + sleep_interval_sec = 0 + }` + +// A failure that matches an ignore policy and a retry policy is ignored on the first attempt: ignore +// policies are read before retry policies, and the unit keeps the source it was written with. +func TestRegistryResolutionFailureMatchingIgnoreAndRetryIsIgnoredWithoutRetrying(t *testing.T) { + request, discoveryCalls := failingRegistryUnit(t, registryRetryPolicy+` + ignore "registry_down" { + ignorable_errors = ["status code 503"] + message = "registry is down" + }`) + result, err := Resolve(t.Context(), request) + require.NoError(t, err) + require.EqualValues(t, 1, discoveryCalls.Load(), "an ignored failure must not be retried") + require.NotNil(t, result.RunIgnored) + assert.Equal(t, model.IgnoredRun{Policy: "registry_down", Message: "registry is down"}, *result.RunIgnored) + assert.Contains(t, result.Source, "tfr://", "no source was resolved, so the unit keeps the source it was written with") + assert.Nil(t, result.RegistryAuth) + assert.NotEmpty(t, result.WorkingDir) +} + +// A failure no ignore policy matches is retried exactly as before and then fails. +func TestRegistryResolutionFailureNotIgnoredIsStillRetried(t *testing.T) { + request, discoveryCalls := failingRegistryUnit(t, registryRetryPolicy+` + ignore "unrelated" { + ignorable_errors = ["certificate has expired"] + }`) + _, err := Resolve(t.Context(), request) + var problem *model.Problem + require.ErrorAs(t, err, &problem) + assert.Equal(t, model.CodeInvalidSource, problem.Code) + assert.EqualValues(t, 3, discoveryCalls.Load(), "a failure that no ignore policy matches is retried up to max_attempts") +} + +// Retry patterns match the same cleaned text as ignore patterns, so one pattern behaves alike in both +// lists, like a source fetch: a message split across lines, or with characters outside the readable +// set, still matches a pattern written for its readable form. +func TestRetryRegistryResolutionMatchesRetryPatternsAgainstCleanedText(t *testing.T) { + policies := &model.Errors{Retry: []model.RetryPolicy{{ + Name: "registry", Patterns: []string{"status code 503"}, MaxAttempts: 3, + }}} + attempts := 0 + _, ignore, err := retryRegistryResolution(t.Context(), policies, func() (string, error) { + attempts++ + return "", errors.New("GET https://reg/acme: [status code\n503]") + }) + require.Error(t, err) + assert.Nil(t, ignore) + assert.Equal(t, 3, attempts, "a pattern that matches the cleaned text retries up to max_attempts") +} + +func TestMatchingRegistryIgnore(t *testing.T) { + policies := []model.IgnorePolicy{ + {Name: "vetoed", Patterns: []string{"status code 503", "!acme"}}, + {Name: "first", Patterns: []string{"status code 503"}}, + {Name: "second", Patterns: []string{"status code"}}, + } + got := matchingRegistryIgnore(policies, "GET https://reg/acme: \x1b[31mstatus code\n 503\x1b[0m") + require.NotNil(t, got) + assert.Equal(t, "first", got.Name, "text is cleaned like a source fetch and the first match decides") + assert.Nil(t, matchingRegistryIgnore(policies, "connection refused")) + assert.Nil(t, matchingRegistryIgnore(nil, "status code 503")) +} + +func TestRegistrySourceResolvesAtPreparation(t *testing.T) { + host, requests := registryServer(t) + request := registryUnit(t, `source = "tfr://`+host+`/acme/vpc/aws//modules/nat?version=1.4.0"`) + result, err := Resolve(t.Context(), request) + require.NoError(t, err) + assert.Equal(t, "git::https://github.com/acme/terraform-aws-vpc.git//modules/nat?ref=v1.4.0", result.Source) + assert.Empty(t, result.Folders, "a registry module is remote and has no local watch") + assert.NotEmpty(t, result.WorkingDir) + assert.Positive(t, requests.Load()) +} + +func TestRegistrySourceVersionSelection(t *testing.T) { + host, _ := registryServer(t) + latest, err := Resolve(t.Context(), registryUnit(t, `source = "tfr://`+host+`/acme/vpc/aws"`)) + require.NoError(t, err) + assert.Equal(t, "git::https://github.com/acme/terraform-aws-vpc.git?ref=v2.0.0", latest.Source) + + constrained, err := Resolve(t.Context(), registryUnit(t, "source = \"tfr://"+host+"/acme/vpc/aws\"\nversion = \"~> 1.0\"")) + require.NoError(t, err) + assert.Equal(t, "git::https://github.com/acme/terraform-aws-vpc.git?ref=v1.4.0", constrained.Source) +} + +func TestRegistrySourceStaysUnresolvedWithoutNetwork(t *testing.T) { + host, requests := registryServer(t) + source := "tfr://" + host + "/acme/vpc/aws?version=1.0.0" + for _, phase := range []string{"discover", "backend"} { + request := registryUnit(t, `source = "`+source+`"`) + request.Phase = phase + if phase == "discover" { + request.Command = "discover" + } + result, err := Resolve(t.Context(), request) + require.NoError(t, err, phase) + assert.Equal(t, int32(0), requests.Load(), phase) + if phase == "discover" { + assert.Equal(t, source, result.Source) + assert.Empty(t, result.Folders) + } + } +} + +func TestRegistrySourceIsNotResolvedForAnExcludedOperation(t *testing.T) { + host, requests := registryServer(t) + source := "tfr://" + host + "/acme/vpc/aws?version=1.0.0" + request := registryUnit(t, `source = "`+source+`" +} +exclude { + if = true + actions = ["plan"] + no_run = true`) + result, err := Resolve(t.Context(), request) + require.NoError(t, err) + assert.Equal(t, int32(0), requests.Load(), "a prevented plan must not contact the registry") + assert.Equal(t, source, result.Source) + + request.Command = "apply" + result, err = Resolve(t.Context(), request) + require.NoError(t, err) + assert.Positive(t, requests.Load(), "an operation the block does not prevent resolves the source") + assert.NotEqual(t, source, result.Source) +} + +func TestRegistrySourceFailureIsAnActionableProblem(t *testing.T) { + host, _ := registryServer(t) + _, err := Resolve(t.Context(), registryUnit(t, `source = "tfr://`+host+`/acme/missing/aws?version=1.0.0"`)) + var problem *model.Problem + require.ErrorAs(t, err, &problem) + assert.Equal(t, model.CodeInvalidSource, problem.Code) + assert.Equal(t, "terraform.source", problem.Key) + assert.NotEmpty(t, problem.Hints) +} + +func TestNormalizeGetterURL(t *testing.T) { + for source, want := range map[string]string{ + "github.com/acme/repo?ref=v1": "git::https://github.com/acme/repo.git?ref=v1", + "github.com/acme/repo.git//sub?ref=v1": "git::https://github.com/acme/repo.git//sub?ref=v1", + "bitbucket.org/acme/repo": "git::https://bitbucket.org/acme/repo.git", + "github.com/acme/repo/modules/vpc?ref=v1": "git::https://github.com/acme/repo.git//modules/vpc?ref=v1", + "git::https://example.test/repo.git": "git::https://example.test/repo.git", + "https://example.test/module.tar.gz": "https://example.test/module.tar.gz", + } { + assert.Equal(t, want, normalizeGetterURL(source)) + } +} + +func TestIsOpenTofu(t *testing.T) { + assert.True(t, IsOpenTofu(filepath.Join("bin", "tofu"))) + assert.True(t, IsOpenTofu(`C:\tools\tofu.exe`)) + assert.False(t, IsOpenTofu("/usr/bin/terraform")) + assert.False(t, IsOpenTofu("")) +} + +// The host and the provider treat any executable whose name contains "tofu", in any case, as OpenTofu. +func TestIsOpenTofuUsesTheHostDetectionRule(t *testing.T) { + for _, binary := range []string{"opentofu", "tofu-wrapper", "TOFU.exe", `C:\tools\OpenTofu.EXE`, "/opt/bin/tofu1.8"} { + assert.True(t, IsOpenTofu(binary), binary) + } + for _, binary := range []string{"terraform", "/opt/tofu/bin/terraform", "terraform-wrapper.exe"} { + assert.False(t, IsOpenTofu(binary), binary, "only the executable name decides, not its directories") + } +} + +func TestUnsupportedSourceTransportsStillFail(t *testing.T) { + for _, source := range []string{"s3::https://bucket/module.zip", "ftp://example.test/module"} { + _, err := Resolve(t.Context(), registryUnit(t, `source = "`+source+`"`)) + var problem *model.Problem + require.ErrorAs(t, err, &problem, source) + assert.Equal(t, model.CodeInvalidSource, problem.Code) + } +} + +func iamUnit(t *testing.T, iam string) model.Request { + t.Helper() + request := nativeUnit(t, iam) + request.Tofu = "/bin/tofu" + return request +} + +func TestIAMSettingsAreReturnedAsData(t *testing.T) { + request := iamUnit(t, `iam_role = "arn:aws:iam::123456789012:role/deploy" +iam_assume_role_session_name = "session-${get_env("SESSION", "local")}" +iam_assume_role_duration = 7200 +iam_web_identity_token = "token-value" +inputs = { account = "kept" }`) + request.Environment = map[string]string{"SESSION": "ci"} + result, err := Resolve(t.Context(), request) + require.NoError(t, err) + require.Equal(t, &model.IAM{Role: "arn:aws:iam::123456789012:role/deploy", SessionName: "session-ci", DurationSeconds: 7200, WebIdentityToken: "token-value"}, result.IAM) + require.Equal(t, "kept", result.Inputs["account"], "iam settings do not disturb the rest of the unit") +} + +func TestIAMWebIdentityTokenFileIsAnchoredAtTheUnit(t *testing.T) { + request := iamUnit(t, `iam_role = "arn:aws:iam::123456789012:role/deploy" +iam_web_identity_token = "token.jwt"`) + unitDir := filepath.Dir(request.Path) + require.NoError(t, os.WriteFile(filepath.Join(unitDir, "token.jwt"), []byte("jwt"), 0o600)) + result, err := Resolve(t.Context(), request) + require.NoError(t, err) + resolvedUnitDir, err := filepath.EvalSymlinks(unitDir) + require.NoError(t, err) + require.Equal(t, filepath.Join(resolvedUnitDir, "token.jwt"), result.IAM.WebIdentityToken, "a relative token file resolves against the unit directory") + + require.Equal(t, "literal-token", unitTokenFile("literal-token", unitDir), "a token that names no file is kept") + absolute := filepath.Join(unitDir, "token.jwt") + require.Equal(t, absolute, unitTokenFile(absolute, t.TempDir())) + require.Empty(t, unitTokenFile("", unitDir)) +} + +func TestIAMRoleIsEvaluatedBeforeInputs(t *testing.T) { + // A role named by a local that a function computes is a plain string once the unit is read. + request := iamUnit(t, "locals { account = \"123456789012\" }\niam_role = \"arn:aws:iam::${local.account}:role/x\"") + result, err := Resolve(t.Context(), request) + require.NoError(t, err) + require.Equal(t, "arn:aws:iam::123456789012:role/x", result.IAM.Role) +} + +func TestIAMProbeResolvesRoleWithoutEvaluatingInputs(t *testing.T) { + request := iamUnit(t, `locals { account = "123456789012" } +iam_role = "arn:aws:iam::${local.account}:role/x" +inputs = { account = get_aws_account_id() }`) + request.Phase = "iam" + result, err := Resolve(t.Context(), request) + require.NoError(t, err) + require.Equal(t, "arn:aws:iam::123456789012:role/x", result.IAM.Role) + require.Nil(t, result.Inputs) +} + +func TestIAMProbeSkipsRoleForExcludedOperation(t *testing.T) { + request := iamUnit(t, `iam_role = "arn:aws:iam::123456789012:role/x" +exclude { + if = true + actions = ["plan"] +}`) + request.Phase = "iam" + result, err := Resolve(t.Context(), request) + require.NoError(t, err) + require.Nil(t, result.IAM) +} + +// An exclude block that reads a feature must see the unit's feature declarations during the IAM probe, with +// the declared default and with an override, or the probe fails for a unit that has no IAM problem. +func TestIAMProbeEvaluatesFeatureDependentExclude(t *testing.T) { + unit := `feature "skip" { + default = false +} +iam_role = "arn:aws:iam::123456789012:role/x" +exclude { + if = feature.skip.value + actions = ["plan"] +}` + t.Run("default applies without an override", func(t *testing.T) { + request := iamUnit(t, unit) + request.Phase = "iam" + result, err := Resolve(t.Context(), request) + require.NoError(t, err) + require.NotNil(t, result.IAM) + assert.Equal(t, "arn:aws:iam::123456789012:role/x", result.IAM.Role) + }) + t.Run("override excludes the operation", func(t *testing.T) { + request := iamUnit(t, unit) + request.Phase = "iam" + request.Features = map[string]string{"skip": "true"} + result, err := Resolve(t.Context(), request) + require.NoError(t, err) + assert.Nil(t, result.IAM, "an excluded operation resolves no role") + }) +} + +func TestAWSFunctionUsesStagedSessionCredentials(t *testing.T) { + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + require.Contains(t, r.Header.Get("Authorization"), "Credential=ASIASESSION/") + w.Header().Set("Content-Type", "text/xml") + fmt.Fprint(w, `222222222222arn:aws:sts::222222222222:assumed-role/x/sessionsession`) + })) + t.Cleanup(server.Close) + t.Setenv("AWS_ENDPOINT_URL_STS", server.URL) + request := iamUnit(t, `iam_role = "arn:aws:iam::222222222222:role/x" +inputs = { account = get_aws_account_id() }`) + request.Environment = map[string]string{"AWS_ACCESS_KEY_ID": "ASIASESSION", "AWS_SECRET_ACCESS_KEY": "session-secret", "AWS_SESSION_TOKEN": "session-token", "AWS_REGION": "us-east-1"} + result, err := Resolve(t.Context(), request) + require.NoError(t, err) + require.Equal(t, "222222222222", result.Inputs["account"]) +} + +func TestIAMEnvironmentOverridesTheUnit(t *testing.T) { + request := iamUnit(t, `iam_role = "arn:aws:iam::1:role/from-unit" +iam_assume_role_duration = 900`) + request.Environment = map[string]string{"TG_IAM_ASSUME_ROLE": "arn:aws:iam::2:role/from-env", "TG_IAM_ASSUME_ROLE_DURATION": "1800", "TERRAGRUNT_IAM_ASSUME_ROLE_SESSION_NAME": "legacy"} + result, err := Resolve(t.Context(), request) + require.NoError(t, err) + require.Equal(t, "arn:aws:iam::2:role/from-env", result.IAM.Role) + require.Equal(t, int64(1800), result.IAM.DurationSeconds) + require.Equal(t, "legacy", result.IAM.SessionName) + + request.Environment = map[string]string{"TG_IAM_ASSUME_ROLE_DURATION": "soon"} + _, err = Resolve(t.Context(), request) + var problem *model.Problem + require.ErrorAs(t, err, &problem) + require.Equal(t, "iam_assume_role_duration", problem.Key) +} + +func TestIAMTokenWithoutRoleIsIgnoredWithAWarning(t *testing.T) { + result, err := Resolve(t.Context(), iamUnit(t, `iam_web_identity_token = "orphan-token"`)) + require.NoError(t, err) + require.Nil(t, result.IAM) + require.Len(t, result.Warnings, 1) + require.Equal(t, warningIAMTokenWithoutRole, result.Warnings[0].Code) + require.NotContains(t, result.Warnings[0].Message, "orphan-token") + + none, err := Resolve(t.Context(), iamUnit(t, "")) + require.NoError(t, err) + require.Nil(t, none.IAM) + require.Empty(t, none.Warnings) +} + +func TestIAMSettingsDoNotRunCommandsDuringDiscovery(t *testing.T) { + request := iamUnit(t, `iam_role = "arn:aws:iam::1:role/x"`) + request.Phase, request.Command = "discover", "discover" + result, err := Resolve(t.Context(), request) + require.NoError(t, err) + require.Nil(t, result.IAM, "discovery does not resolve credentials or roles") +} + +// A unit that a stack generates exists only in the generation overlay on a clean checkout, and the host +// probes IAM before the provider generates the stack. +func TestIAMProbeSeesUnitsThatTheirStackGenerates(t *testing.T) { + root := stackProject(t, map[string]string{ + "live/terragrunt.stack.hcl": "unit \"app\" {\n source = \"../catalog/units/app\"\n path = \"app\"\n no_dot_terragrunt_stack = true\n}\n", + }) + module, err := json.Marshal(filepath.ToSlash(filepath.Join(root, "modules", "unit"))) + require.NoError(t, err) + require.NoError(t, os.WriteFile(filepath.Join(root, "catalog", "units", "app", "terragrunt.hcl"), + []byte("iam_role = \"arn:aws:iam::123456789012:role/generated\"\nterraform { source = "+string(module)+" }\n"), 0o644)) + request := model.Request{ + Phase: "iam", Command: "plan", BasePath: root, Environment: map[string]string{}, Tofu: "/bin/tofu", + Path: filepath.Join(root, "live", "app", "terragrunt.hcl"), + GenerationRoot: filepath.Join(root, "live", "terragrunt.stack.hcl"), + } + + result, err := Resolve(t.Context(), request) + + require.NoError(t, err) + require.NotNil(t, result.IAM) + require.Equal(t, "arn:aws:iam::123456789012:role/generated", result.IAM.Role) + require.NoFileExists(t, request.Path, "the generated unit exists only in the invocation's overlay") +} + +func TestIAMProbeOfAnUnknownUnitStillFails(t *testing.T) { + request := iamUnit(t, `iam_role = "arn:aws:iam::123456789012:role/x"`) + request.Phase = "iam" + request.Path = filepath.Join(filepath.Dir(request.Path), "missing", "terragrunt.hcl") + _, err := Resolve(t.Context(), request) + require.Error(t, err, "without a generation root the unit must exist") +} + +func TestRejectedResolvedSourceNamesBothSources(t *testing.T) { + problem := &model.Problem{Code: model.CodeInvalidSource, Summary: "Unsupported module source transport", Detail: "The module source scheme \"ftp\" is not supported."} + + err := rejectedResolvedSource("tfr:///terraform-aws-modules/vpc/aws?version=5.0.0", "ftp://example.invalid/vpc.zip", problem) + + var annotated *model.Problem + require.ErrorAs(t, err, &annotated) + assert.Equal(t, model.CodeInvalidSource, annotated.Code) + assert.Contains(t, annotated.Detail, "ftp\" is not supported") + assert.Contains(t, annotated.Detail, "tfr:///terraform-aws-modules/vpc/aws?version=5.0.0") + assert.Contains(t, annotated.Detail, "ftp://example.invalid/vpc.zip") + assert.NotContains(t, problem.Detail, "resolved to", "the original problem is not modified") + + plain := errors.New("not a problem") + assert.Same(t, plain, rejectedResolvedSource("tfr:///a/b/c", "x", plain)) +} + +func TestRejectedResolvedSourceOmitsURLCredentials(t *testing.T) { + tests := []struct { + name string + resolved string + want string + }{ + {"user and token", "ftp://alice:s3cret@example.invalid/vpc.zip", "ftp://example.invalid/vpc.zip"}, + {"token only", "git::https://ghp_token@github.com/org/repo.git?ref=v1", "git::https://github.com/org/repo.git?ref=v1"}, + {"no credentials", "ftp://example.invalid/vpc.zip", "ftp://example.invalid/vpc.zip"}, + {"at sign in the path", "ftp://example.invalid/a@b.zip", "ftp://example.invalid/a@b.zip"}, + } + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + problem := &model.Problem{Code: model.CodeInvalidSource, Summary: "Unsupported", Detail: "Not supported.", Safe: true} + err := rejectedResolvedSource("tfr:///a/b/c", test.resolved, problem) + + var annotated *model.Problem + require.ErrorAs(t, err, &annotated) + assert.Contains(t, annotated.Detail, test.want) + assert.NotContains(t, annotated.Detail, "s3cret") + assert.NotContains(t, annotated.Detail, "ghp_token") + assert.NotContains(t, annotated.Detail, "alice") + }) + } +} diff --git a/plugins/terragrunt/internal/engine/resolve.go b/plugins/terragrunt/internal/engine/resolve.go index 09adf866763..def2b17373f 100644 --- a/plugins/terragrunt/internal/engine/resolve.go +++ b/plugins/terragrunt/internal/engine/resolve.go @@ -81,9 +81,15 @@ type unitEvaluator struct { } func (e *unitEvaluator) resolve(ctx context.Context) (*model.Resolution, error) { + if e.request.Phase == "iam" { + return e.resolveIAM(ctx) + } if filepath.Base(e.path) == stackFile { return e.resolveStack(ctx) } + if err := e.checkRequestFeatureOverrides(ctx); err != nil { + return nil, err + } if discovering(e.request) { return e.resolveUnitInventory(ctx) } @@ -105,13 +111,13 @@ func (e *unitEvaluator) resolveUnit(ctx context.Context) (*model.Resolution, err if err != nil { return nil, err } - result, err := e.unitResolution(cfg) + result, err := e.unitResolution(ctx, cfg) if err != nil { return nil, err } result.Execution = execution result.Files = append(result.Files, applyFiles...) - if err := resolveBackend(cfg, result); err != nil { + if err := resolveBackend(cfg, result, e.request); err != nil { return nil, err } if e.request.Phase != phaseBackend { @@ -138,10 +144,11 @@ const ( ) // backendGenerationOverlay generates the owning stack into an invocation-local -// copy-on-write overlay before a backend read of a stack-generated unit. The -// returned filesystem is the one the unit is then parsed from. +// copy-on-write overlay before a backend read or IAM probe of a stack-generated unit. The +// host probes IAM before the provider generates the stack, so on a clean checkout the unit +// exists only in this overlay. The returned filesystem is the one the unit is then parsed from. func backendGenerationOverlay(ctx context.Context, request *model.Request, filesystem afero.Fs) (afero.Fs, error) { - if request.Phase != phaseBackend || request.GenerationRoot == "" { + if (request.Phase != phaseBackend && request.Phase != "iam") || request.GenerationRoot == "" { return filesystem, nil } filesystem = afero.NewCopyOnWriteFs(afero.NewReadOnlyFs(afero.NewOsFs()), afero.NewMemMapFs()) @@ -212,6 +219,8 @@ func requestParsingContext(request *model.Request, path string) *config.ParsingC if configured := request.Environment[downloadDirEnv]; configured != "" { pctx.DownloadDir = localPath(filepath.Dir(path), configured) } + // Explicit feature overrides apply to every unit, as Terragrunt's --feature flag does. + pctx.FeatureFlags = request.Features return pctx } @@ -308,6 +317,7 @@ func (e *unitEvaluator) unitExecution(ctx context.Context, cfg *config.Terragrun if err != nil { return nil, nil, err } + finalizeExecution(execution, e.path) reparse := func(apply *config.ParsingContext) (*config.TerragruntConfig, error) { return config.ParseConfigFile(ctx, e.logger, e.environment, apply, e.path, nil) } @@ -564,18 +574,27 @@ func traversalName(step hcl.Traverser) (string, bool) { return "", false } -// unitResolution preserves local directory identities and remote getter URIs. -func (e *unitEvaluator) unitResolution(cfg *config.TerragruntConfig) (*model.Resolution, error) { +// unitResolution converts the parsed unit into its module identity. Local sources +// must be existing directories; remote sources keep the identity they were written with. +func (e *unitEvaluator) unitResolution(ctx context.Context, cfg *config.TerragruntConfig) (*model.Resolution, error) { result := &model.Resolution{Path: e.path, Inputs: cfg.Inputs, Source: *cfg.Terraform.Source, Generated: map[string]string{}, Files: append(e.pctx.FilesRead.Paths(), e.path)} - if plainLocalSource(result.Source) { - result.Source = localPath(filepath.Dir(e.path), result.Source) - if problem := localSourceProblem(e.environment.FS, e.path, *cfg.Terraform.Source, result.Source); problem != nil { - return nil, problem - } - result.Folders = []string{result.Source} - } else if err := validateRemoteSource(result.Source); err != nil { + if err := unitSettings(ctx, cfg, e.request, e.path, result); err != nil { return nil, err } + // A remote source keeps the identity it was written with, exactly as Terragrunt names its working + // copy; resolving a registry source below only changes where the module is fetched from. + identity := result.Source + if local, problem := resolveLocalModuleSource(e.environment.FS, e.path, result.Source); problem != nil { + return nil, problem + } else if local != nil { + result.Source = local.source + identity, result.Folders = result.Source, []string{local.root} + } else if !PreventsRun(result.Exclude, OperationCommand(e.request.Command, e.request.Arguments)) { + // An operation the exclude block prevents never runs, so its registry source is not resolved. + if err := prepareRemoteSource(ctx, e.logger, e.request, cfg, result); err != nil { + return nil, err + } + } download := e.pctx.DownloadDir if cfg.DownloadDir != "" && e.request.Environment[downloadDirEnv] == "" { download = localPath(filepath.Dir(e.path), cfg.DownloadDir) @@ -583,7 +602,7 @@ func (e *unitEvaluator) unitResolution(cfg *config.TerragruntConfig) (*model.Res if configured := e.request.Environment[downloadDirEnv]; configured != "" { download = localPath(filepath.Dir(e.path), configured) } - workingDir, err := atmosbridge.WorkingDir(e.logger, e.environment, result.Source, download, filepath.Dir(e.path)) + workingDir, err := atmosbridge.WorkingDir(e.logger, e.environment, identity, download, filepath.Dir(e.path)) if err != nil { return nil, err } diff --git a/plugins/terragrunt/internal/engine/resolve_test.go b/plugins/terragrunt/internal/engine/resolve_test.go index 25ff65c84af..fe1a2ce391f 100644 --- a/plugins/terragrunt/internal/engine/resolve_test.go +++ b/plugins/terragrunt/internal/engine/resolve_test.go @@ -57,8 +57,6 @@ inputs = { large = 9007199254740993, nested = { nullable = null, enabled = false func TestNativeRejectsUnsupportedSemantics(t *testing.T) { for _, test := range []struct{ name, extra, message string }{ - {"binary override", `terraform_binary = "another-tool"`, "terraform_binary"}, - {"IAM role", `iam_role = "unsupported"`, "iam_role"}, {"generation escape", `generate "x" { path = "../x.tf" if_exists = "overwrite" @@ -170,7 +168,7 @@ generate "disabled" { require.NoError(t, os.WriteFile(request.Path, []byte("inputs = {}"), 0o644)) _, err = Resolve(t.Context(), request) require.ErrorContains(t, err, "terraform.source") - for _, source := range []string{"tfr://example.invalid/repo", "./absent"} { + for _, source := range []string{"ftp://example.invalid/repo", "./absent"} { encoded, err := json.Marshal(source) require.NoError(t, err) require.NoError(t, os.WriteFile(request.Path, []byte("terraform { source = "+string(encoded)+" }"), 0o644)) @@ -187,7 +185,10 @@ func TestDiscoveryValidatesRemoteModuleSources(t *testing.T) { source string invalid bool }{ - {source: "tfr://example.invalid/repo", invalid: true}, + {source: "ftp://example.invalid/repo", invalid: true}, + {source: "s3://example-bucket/module.zip", invalid: true}, + // A registry source is valid at discovery; it is resolved only when the unit is prepared. + {source: "tfr://example.invalid/modules/vpc/aws"}, {source: "git::https://example.invalid/modules.git//vpc?ref=v1.0.0"}, {source: "https://example.invalid/module.zip"}, // A local module with a `//` subdirectory is not a remote source: Terragrunt copies the tree before it. @@ -206,6 +207,10 @@ func TestDiscoveryValidatesRemoteModuleSources(t *testing.T) { encoded, err := json.Marshal(test.source) require.NoError(t, err) require.NoError(t, os.WriteFile(request.Path, []byte("terraform { source = "+string(encoded)+" }"), 0o644)) + // A local source must name an existing directory, so create the one the case selects. + if strings.HasPrefix(test.source, ".") && !test.invalid { + require.NoError(t, os.MkdirAll(filepath.Join(filepath.Dir(request.Path), filepath.FromSlash(strings.ReplaceAll(test.source, "//", "/"))), 0o755)) + } request.Phase = "discover" _, err = Resolve(t.Context(), request) if !test.invalid { @@ -260,16 +265,6 @@ func TestResolveUnitInDirectoryWithGlobMetacharacters(t *testing.T) { require.Equal(t, []byte("# lint"), result.Overlay[".tflint.hcl"]) } -func TestRejectRemoteStateEncryption(t *testing.T) { - request := nativeUnit(t, `remote_state { - backend = "local" - config = { path = "state.tfstate" } - encryption = { key_provider = "pbkdf2" } -}`) - _, err := Resolve(t.Context(), request) - require.ErrorContains(t, err, "remote_state.encryption") -} - func TestBackendResolutionHonorsConfiguredDownloadDirectory(t *testing.T) { request := nativeUnit(t, `download_dir = "custom-cache" remote_state { @@ -412,6 +407,16 @@ func TestOverlayRejectsLockFilesAndCarriesModes(t *testing.T) { }) } +func TestValidateEngineProbesTheUnitsExecutable(t *testing.T) { + request := nativeUnit(t, "") + // request.Tofu is the host's command and must not be the executable that the probe runs. + err := validateEngine(t.Context(), &request, filepath.Join(t.TempDir(), "no-such-terraform-binary"), request.Path) + + var problem *model.Problem + require.ErrorAs(t, err, &problem) + require.Equal(t, "terraform_binary", problem.Key) +} + func TestBackendDownloadDirectoryFromReadConfigInputs(t *testing.T) { for name, expression := range map[string]string{ "direct": `download_dir = try(read_terragrunt_config("settings.hcl").inputs.cache, "fallback-cache")`, diff --git a/plugins/terragrunt/internal/model/model.go b/plugins/terragrunt/internal/model/model.go index dd2393206a6..19b91eed9b8 100644 --- a/plugins/terragrunt/internal/model/model.go +++ b/plugins/terragrunt/internal/model/model.go @@ -16,6 +16,8 @@ type Request struct { Arguments []string `json:"arguments"` Tofu string `json:"tofu"` Environment map[string]string `json:"environment"` + // Features are explicit feature flag overrides (name to raw value), applied to every unit. + Features map[string]string `json:"features,omitempty"` } type Resolution struct { @@ -35,11 +37,15 @@ type Resolution struct { StackConfigs map[string]string `json:"stack_configs,omitempty"` Inputs map[string]any `json:"inputs"` Source string `json:"source"` - BackendGenerated bool `json:"backend_generated,omitempty"` - BackendType string `json:"backend_type"` - Backend map[string]any `json:"backend"` - Generated map[string]string `json:"generated"` - GeneratedPolicies map[string]string `json:"generated_policies,omitempty"` + // RegistryAuth is an invocation-only bearer header for a same-origin archive. + // It must never enter saved configuration or source provenance. + RegistryAuth *RegistryAuth `json:"-"` + RegistryResolutionNanos uint64 `json:"registry_resolution_nanos,omitempty"` + BackendGenerated bool `json:"backend_generated,omitempty"` + BackendType string `json:"backend_type"` + Backend map[string]any `json:"backend"` + Generated map[string]string `json:"generated"` + GeneratedPolicies map[string]string `json:"generated_policies,omitempty"` // UnitFileCopies identifies literal files copied from beside the unit. UnitFileCopies map[string]bool `json:"unit_file_copies,omitempty"` // SensitiveGenerated names generated files whose contents can hold @@ -51,8 +57,83 @@ type Resolution struct { Failures []UnitFailure `json:"failures,omitempty"` // Warnings are non-fatal facts about the scan that the user should see. Warnings []Warning `json:"warnings,omitempty"` + // TerraformBinary is the unit's terraform_binary: a name for the toolchain or PATH, or an absolute path. + TerraformBinary string `json:"terraform_binary,omitempty"` + // Errors holds the retry and ignore policies of the unit's errors block. + Errors *Errors `json:"errors,omitempty"` + // Exclude is the evaluated exclude block, set only when the block's condition holds. + Exclude *Exclude `json:"exclude,omitempty"` + // FeatureFlags maps each declared feature to its effective value rendered as a string. + FeatureFlags map[string]string `json:"feature_flags,omitempty"` + // IAM is the role the host assumes for the unit's commands; nil when the unit names none. + IAM *IAM `json:"iam,omitempty"` + // RunIgnored is set when an ignore policy matched a failure that left the source unresolved. The unit + // keeps the source it was written with, and the host ends its run without executing it. + RunIgnored *IgnoredRun `json:"run_ignored,omitempty"` } +type RegistryAuth struct { + Origin string + Authorization string +} + +// IAM is the evaluated iam_role setting and its companions. The plugin never assumes the role: the host +// does, on top of the credentials of the invocation. +type IAM struct { + Role string `json:"role"` + SessionName string `json:"session_name,omitempty"` + DurationSeconds int64 `json:"duration_seconds,omitempty"` + // WebIdentityToken is a token, or the path of a file that holds one. It is a secret. + WebIdentityToken string `json:"-"` +} + +// Errors is the evaluated errors block, in declaration order. +type Errors struct { + Retry []RetryPolicy `json:"retry,omitempty"` + Ignore []IgnorePolicy `json:"ignore,omitempty"` +} + +// RetryPolicy retries a failed command whose output matches a pattern. A pattern that starts with +// "!" vetoes the policy when it matches. +type RetryPolicy struct { + Name string `json:"name"` + Patterns []string `json:"patterns"` + MaxAttempts int `json:"max_attempts"` + SleepIntervalSec int `json:"sleep_interval_sec"` +} + +// IgnoredRun names the ignore policy that ended a unit's run before it could execute. +type IgnoredRun struct { + Policy string `json:"policy"` + Message string `json:"message,omitempty"` +} + +// IgnorePolicy turns a failed command whose output matches a pattern into a success. +type IgnorePolicy struct { + Name string `json:"name"` + Patterns []string `json:"patterns"` + Message string `json:"message,omitempty"` + Signals map[string]any `json:"signals,omitempty"` +} + +// Exclude is an exclude block whose condition holds. +type Exclude struct { + Actions []string `json:"actions"` + ExcludeDependencies bool `json:"exclude_dependencies,omitempty"` + // NoRun is nil when the block does not set it, which keeps Terragrunt's exact-action default. + NoRun *bool `json:"no_run,omitempty"` +} + +// Hook lifecycle commands that name Terragrunt's own stages rather than a Terraform command. +const ( + // LifecycleReadConfig runs after the unit's configuration is read (after_hook only). + LifecycleReadConfig = "terragrunt-read-config" + // LifecycleReadConfigAlias is the documented spelling of LifecycleReadConfig. + LifecycleReadConfigAlias = "read-config" + // LifecycleInitFromModule runs around the source download. + LifecycleInitFromModule = "init-from-module" +) + // UnitFailure is a unit that discovery could not evaluate. Stack and Name are the Atmos // coordinates the unit would have had; Err is a Problem or a plain error. type UnitFailure struct { diff --git a/plugins/terragrunt/internal/provider/discovery.go b/plugins/terragrunt/internal/provider/discovery.go index 01d41aa4105..3c95883c77c 100644 --- a/plugins/terragrunt/internal/provider/discovery.go +++ b/plugins/terragrunt/internal/provider/discovery.go @@ -11,6 +11,8 @@ import ( pluginv1 "github.com/cloudposse/atmos/sdk/proto/atmos/plugin/v1" ) +const currentDirectory = "." + // Discover expands local stacks in an isolated overlay without preparing units. // //nolint:lintroller // Standalone plugin code cannot import the host performance tracker. @@ -22,6 +24,9 @@ func (p *Provider) Discover(ctx context.Context, request *pluginv1.DiscoverCompo if err := settings(request.GetProviderConfiguration()); err != nil { return nil, rpcError(err) } + if err := withFeatures(&req, request.GetProviderConfiguration()); err != nil { + return nil, rpcError(err) + } paths, err := importPaths(request) if err != nil { return nil, rpcError(err) @@ -59,7 +64,7 @@ func discoveredUnit(req *model.Request, unit *model.DiscoveredUnit, addresses ma if err != nil { return nil, err } - module, err := discoveredModule(req.BasePath, &unit.Config) + module, moduleSubdir, err := discoveredModule(req.BasePath, &unit.Config) if err != nil { return nil, err } @@ -74,7 +79,7 @@ func discoveredUnit(req *model.Request, unit *model.DiscoveredUnit, addresses ma if workspace == "" { workspace = "default" } - config, err := encode(map[string]any{ + component := map[string]any{ "source": module, "vars": map[string]any{}, "provision": map[string]any{"workdir": map[string]any{"enabled": true}}, "backend_type": "local", "backend": map[string]any{"local": map[string]any{}}, @@ -83,7 +88,12 @@ func discoveredUnit(req *model.Request, unit *model.DiscoveredUnit, addresses ma "terragrunt": map[string]any{"generation_root": generation}, "configuration": map[string]any{"watch_excludes": []string{"**/.terragrunt-cache/**", "**/.terragrunt-stack/**", "**/.terraform/**", "**/.atmos/**"}}, }, - }) + } + if moduleSubdir != "" { + component["metadata"].(map[string]any)["source_subdir"] = moduleSubdir + } + discoveryParity(component, &unit.Config) + config, err := encode(component) if err != nil { return nil, err } @@ -91,7 +101,7 @@ func discoveredUnit(req *model.Request, unit *model.DiscoveredUnit, addresses ma if err != nil { return nil, err } - instance := &pluginv1.DiscoveredComponent{Address: addresses[unit.Config.Path], Provider: "terragrunt", Engine: engineName(req.Tofu), Source: source, Configuration: config, Sources: watched} + instance := &pluginv1.DiscoveredComponent{Address: addresses[unit.Config.Path], Provider: "terragrunt", Engine: unitEngine(req, &unit.Config), Source: source, Configuration: config, Sources: watched} instance.Dependencies, err = discoveredDependencies(req.BasePath, &unit.Config, addresses) if err != nil { return nil, err @@ -100,8 +110,12 @@ func discoveredUnit(req *model.Request, unit *model.DiscoveredUnit, addresses ma } // discoveredModule resolves local module identities relative to the project. -func discoveredModule(root string, cfg *model.Resolution) (string, error) { +func discoveredModule(root string, cfg *model.Resolution) (string, string, error) { module := cfg.Source + if localRoot, subdir, ok := localModuleSubdir(module); ok { + local, err := relative(root, localRoot) + return "./" + local + "//" + subdir, subdir, err + } if len(cfg.Folders) > 0 && (module == "" || filepath.IsAbs(module)) { path := module if path == "" { @@ -109,14 +123,14 @@ func discoveredModule(root string, cfg *model.Resolution) (string, error) { } local, err := relative(root, path) if err != nil { - return "", err + return "", "", err } module = "./" + local } if module == "" { - return "", ErrUnsupported + return "", "", ErrUnsupported } - return module, nil + return module, "", nil } // discoveredDependencies binds evaluated dependency paths to discovered addresses. @@ -141,6 +155,24 @@ func discoveredDependencies(root string, cfg *model.Resolution, addresses map[st return dependencies, nil } +func localModuleSubdir(source string) (root, subdir string, ok bool) { + volume := filepath.VolumeName(source) + rest := source[len(volume):] + if strings.ContainsAny(rest, "?:") { + return "", "", false + } + index := strings.Index(filepath.ToSlash(rest), "//") + if index <= 0 { + return "", "", false + } + root = volume + rest[:index] + first := strings.SplitN(strings.TrimPrefix(root, "./"), "/", 2)[0] + if strings.Contains(first, ".") && !strings.HasPrefix(root, ".") && !filepath.IsAbs(root) { + return "", "", false + } + return root, rest[index+2:], true +} + // importPaths adds explicit files or directory roots to the same scoped inventory. // No HCL evaluation occurs while determining which roots were requested. func importPaths(request *pluginv1.DiscoverComponentsRequest) ([]string, error) { @@ -174,7 +206,7 @@ func importPaths(request *pluginv1.DiscoverComponentsRequest) ([]string, error) // project-relative unit paths discovery compares against. func directoryPattern(path string) string { clean := filepath.ToSlash(filepath.Clean(filepath.FromSlash(path))) - if clean == "." { + if clean == currentDirectory { return "**" } return clean + "/**" @@ -188,9 +220,9 @@ func derivedAddress(base, path string) (*pluginv1.ComponentAddress, error) { return nil, err } stack, name := "default", "root" - if dir != "." { + if dir != currentDirectory { name = filepath.Base(dir) - if parent := filepath.ToSlash(filepath.Dir(dir)); parent != "." { + if parent := filepath.ToSlash(filepath.Dir(dir)); parent != currentDirectory { stack = parent } } diff --git a/plugins/terragrunt/internal/provider/errors.go b/plugins/terragrunt/internal/provider/errors.go index d1e7d44ceb6..245e3b3aef2 100644 --- a/plugins/terragrunt/internal/provider/errors.go +++ b/plugins/terragrunt/internal/provider/errors.go @@ -163,7 +163,7 @@ func problemFor(err error, file string) error { } } -// warningDiagnostics turns discovery findings into diagnostics the host prints without +// warningDiagnostics turns discovery and preparation findings into diagnostics the host prints without // failing the command. The messages never contain evaluated values: only paths and patterns. func warningDiagnostics(warnings []model.Warning) []*pluginv1.Diagnostic { diagnostics := make([]*pluginv1.Diagnostic, 0, len(warnings)) diff --git a/plugins/terragrunt/internal/provider/fixtures_test.go b/plugins/terragrunt/internal/provider/fixtures_test.go index 7e1a798787f..b48ce602aa9 100644 --- a/plugins/terragrunt/internal/provider/fixtures_test.go +++ b/plugins/terragrunt/internal/provider/fixtures_test.go @@ -174,6 +174,22 @@ func TestRemoteModuleSourceIsNotAFileWatch(t *testing.T) { require.Equal(t, "live/app/terragrunt.hcl", instance.Sources[0].Path) } +func TestDiscoveredLocalModuleSubdirKeepsDelimiter(t *testing.T) { + root, err := filepath.EvalSymlinks(t.TempDir()) + require.NoError(t, err) + unitPath := filepath.Join(root, "live", "app", "terragrunt.hcl") + repository := filepath.Join(root, "repository") + unit := model.DiscoveredUnit{Config: model.Resolution{Path: unitPath, Source: repository + "//modules/app", Folders: []string{repository}, Files: []string{unitPath}}} + addresses := map[string]*pluginv1.ComponentAddress{unitPath: {Stack: "live", Component: "app", ComponentType: "terraform"}} + instance, err := discoveredUnit(&model.Request{BasePath: root}, &unit, addresses) + require.NoError(t, err) + var configuration map[string]any + require.NoError(t, decode(instance.Configuration, &configuration)) + require.Equal(t, "./repository//modules/app", configuration["source"]) + metadata := configuration["metadata"].(map[string]any) + require.Equal(t, "modules/app", metadata["source_subdir"]) +} + func TestDerivedAddressNamesAnUnscopedUnit(t *testing.T) { root, err := filepath.EvalSymlinks(t.TempDir()) require.NoError(t, err) diff --git a/plugins/terragrunt/internal/provider/preparation.go b/plugins/terragrunt/internal/provider/preparation.go index 550585a2c7f..e7df2d7874e 100644 --- a/plugins/terragrunt/internal/provider/preparation.go +++ b/plugins/terragrunt/internal/provider/preparation.go @@ -3,16 +3,22 @@ package provider import ( "context" "crypto/sha256" + "encoding/json" "fmt" "maps" + "math" "os" "path/filepath" "slices" + "strconv" "time" "github.com/gofrs/flock" + "github.com/zclconf/go-cty/cty" + "github.com/cloudposse/atmos/plugins/terragrunt/internal/engine" "github.com/cloudposse/atmos/plugins/terragrunt/internal/model" + "github.com/cloudposse/atmos/sdk/component" pluginv1 "github.com/cloudposse/atmos/sdk/proto/atmos/plugin/v1" ) @@ -28,6 +34,9 @@ func (p *Provider) componentRequest(ctx context.Context, request *pluginv1.Compo return req, ErrUnsupported } req.Path = filepath.Join(request.RootDirectory, filepath.FromSlash(request.Source)) + if err := withFeatures(&req, request.ProviderConfiguration); err != nil { + return req, err + } var config struct { Settings struct { Terragrunt struct { @@ -64,6 +73,11 @@ func (p *Provider) prepare(ctx context.Context, request *pluginv1.ComponentReque if !preparationEffects(request.GetEffects()) { return nil, rpcError(ErrUnsupported) } + // The role is resolved before any evaluation that can call AWS functions, so the + // probe runs before stack generation. + if req.Phase == "iam" { + return p.probeIAM(ctx, &req) + } // All provider processes generating the same stack must retain the lock // until the subsequent unit parse has finished. unlock, err := p.generateLocked(ctx, &req) @@ -149,7 +163,20 @@ func preparedResponse(req *model.Request, result *model.Resolution) (*pluginv1.P if err != nil { return nil, err } - response := &pluginv1.PrepareComponentResponse{Engine: engineName(req.Tofu), ModuleSource: result.Source, WorkingDirectory: result.WorkingDir, Inputs: inputs, State: state, Sources: watched, BackendGenerated: result.BackendGenerated, Execution: executionResponse(result.Execution)} + response := &pluginv1.PrepareComponentResponse{Engine: engineName(req.Tofu), ModuleSource: result.Source, WorkingDirectory: result.WorkingDir, Inputs: inputs, State: state, Sources: watched, BackendGenerated: result.BackendGenerated, Execution: executionResponse(result.Execution), RegistryResolutionNanos: result.RegistryResolutionNanos} + if result.RunIgnored != nil { + response.RunIgnored, response.IgnoredPolicy, response.IgnoredMessage = true, result.RunIgnored.Policy, result.RunIgnored.Message + } + if err := parityResponse(response, req, result); err != nil { + return nil, rpcError(err) + } + if response.Iam, err = iamResponse(result.IAM); err != nil { + return nil, rpcError(err) + } + if err = encodeRegistryAuth(response, result.RegistryAuth); err != nil { + return nil, rpcError(err) + } + response.Diagnostics = unitWarnings(req.BasePath, result.Warnings) response.Files = preparedFiles(result) return response, nil @@ -180,6 +207,31 @@ func preparedFiles(result *model.Resolution) []*pluginv1.ComponentGeneratedFile return filesOut } +func encodeRegistryAuth(response *pluginv1.PrepareComponentResponse, auth *model.RegistryAuth) error { + if auth == nil { + return nil + } + value, err := component.Encode(cty.StringVal(auth.Authorization).Mark(component.Sensitive)) + if err != nil { + return err + } + response.ModuleSourceAuthorization = value + response.ModuleSourceAuthOrigin = auth.Origin + return nil +} + +func (p *Provider) probeIAM(ctx context.Context, req *model.Request) (*pluginv1.PrepareComponentResponse, error) { + result, err := p.evaluate(ctx, *req) + if err != nil { + return nil, rpcError(err) + } + iam, err := iamResponse(result.IAM) + if err != nil { + return nil, rpcError(err) + } + return &pluginv1.PrepareComponentResponse{Iam: iam, Diagnostics: unitWarnings(req.BasePath, result.Warnings)}, nil +} + func stateIdentity(req *model.Request, result *model.Resolution) (*pluginv1.ComponentStateIdentity, error) { backend := result.Backend if backend == nil { @@ -270,3 +322,171 @@ func hooksResponse(hooks []model.Hook) []*pluginv1.ComponentHook { } return result } + +// iamResponse hands the role to the host, which assumes it. The web identity token travels as a +// sensitive value. +func iamResponse(iam *model.IAM) (*pluginv1.ComponentIam, error) { + if iam == nil { + return nil, nil + } + response := &pluginv1.ComponentIam{Role: iam.Role, SessionName: iam.SessionName, DurationSeconds: uint32(max(0, min(iam.DurationSeconds, math.MaxUint32)))} + if iam.WebIdentityToken != "" { + token, err := component.Encode(cty.StringVal(iam.WebIdentityToken).Mark(component.Sensitive)) + if err != nil { + return nil, err + } + response.WebIdentityToken = token + } + return response, nil +} + +// featureEnvironment is the variable Terragrunt reads feature overrides from. +const featureEnvironment = "TG_FEATURE" + +// featureSettings renders the `features` setting as strings. Values must be booleans, numbers, or +// strings, because a feature is a scalar. +func featureSettings(value any) (map[string]string, error) { + if value == nil { + return nil, nil + } + entries, ok := value.(map[string]any) + if !ok { + return nil, settingProblem("features", "Invalid value for `components.terraform.terragrunt.features`", "Set `features` to a map of feature names to booleans, numbers, or strings.") + } + result := make(map[string]string, len(entries)) + for _, name := range slices.Sorted(maps.Keys(entries)) { + switch typed := entries[name].(type) { + case bool: + result[name] = strconv.FormatBool(typed) + case string: + result[name] = typed + case json.Number: + result[name] = typed.String() + default: + return nil, settingProblem("features."+name, fmt.Sprintf("Invalid value for the `%s` feature override", name), "Use a boolean, a number, or a string.") + } + } + return result, nil +} + +// withFeatures sets the request's feature overrides. Overrides from the `features` setting apply +// first; TG_FEATURE in the invocation environment replaces them per name, as environment +// variables outrank configuration files. +func withFeatures(req *model.Request, configuration *pluginv1.ComponentValue) error { + var config map[string]any + if configuration != nil { + if err := decode(configuration, &config); err != nil { + return err + } + } + features, err := featureSettings(config["features"]) + if err != nil { + return err + } + if raw := req.Environment[featureEnvironment]; raw != "" { + fromEnvironment, err := engine.ParseFeatureEnvironment(raw) + if err != nil { + return err + } + if features == nil { + features = map[string]string{} + } + maps.Copy(features, fromEnvironment) + } + req.Features = features + return nil +} + +// unitEngine is the engine a unit runs with: its terraform_binary when it sets one, otherwise the +// host's selection. +func unitEngine(req *model.Request, resolved *model.Resolution) string { + if resolved != nil && resolved.TerraformBinary != "" { + return engineName(resolved.TerraformBinary) + } + return engineName(req.Tofu) +} + +// discoveryParity adds what the host needs before a unit is prepared: the executable the unit +// selects, as the component's `command` so a stack-level `command` still overrides it, and the +// exclude block that bulk commands consult. +func discoveryParity(config map[string]any, resolved *model.Resolution) { + if resolved.TerraformBinary != "" { + config["command"] = resolved.TerraformBinary + } + if resolved.Exclude == nil { + return + } + settings, _ := config["settings"].(map[string]any) + terragrunt, _ := settings["terragrunt"].(map[string]any) + terragrunt["exclude"] = map[string]any{ + "actions": slices.Clone(resolved.Exclude.Actions), + "exclude_dependencies": resolved.Exclude.ExcludeDependencies, + } +} + +// operationCommand is the command name Terragrunt matches exclude actions against. +func operationCommand(command string, arguments []string) string { + return engine.OperationCommand(command, arguments) +} + +// parityResponse adds the evaluated terraform_binary, errors, exclude, and feature flags to a +// Prepare response. Exclude carries only the outcome for this operation: the block's no_run +// semantics are applied here, so it is set exactly when the host must not run the command. +func parityResponse(response *pluginv1.PrepareComponentResponse, req *model.Request, resolved *model.Resolution) error { + response.TerraformBinary = resolved.TerraformBinary + response.FeatureFlags = maps.Clone(resolved.FeatureFlags) + errorsResponse, err := errorsResponse(resolved.Errors) + if err != nil { + return err + } + response.Errors = errorsResponse + command := operationCommand(req.Command, req.Arguments) + if engine.PreventsRun(resolved.Exclude, command) { + response.Exclude = &pluginv1.ComponentExclude{Condition: true, Actions: []string{command}, ExcludeDependencies: resolved.Exclude.ExcludeDependencies} + } + return nil +} + +func errorsResponse(policies *model.Errors) (*pluginv1.ComponentErrorsConfiguration, error) { + if policies == nil { + return nil, nil + } + result := &pluginv1.ComponentErrorsConfiguration{} + for i := range policies.Retry { + retry := &policies.Retry[i] + result.Retry = append(result.Retry, &pluginv1.ComponentRetryPolicy{ + Name: retry.Name, RetryableErrors: slices.Clone(retry.Patterns), MaxAttempts: uint32(retry.MaxAttempts), SleepIntervalSeconds: uint32(retry.SleepIntervalSec), //nolint:gosec // Validated as positive when the unit is evaluated. + }) + } + for i := range policies.Ignore { + ignore := &policies.Ignore[i] + signals := make(map[string]*pluginv1.ComponentValue, len(ignore.Signals)) + for _, name := range slices.Sorted(maps.Keys(ignore.Signals)) { + value, err := encode(ignore.Signals[name]) + if err != nil { + return nil, err + } + signals[name] = value + } + result.Ignore = append(result.Ignore, &pluginv1.ComponentIgnorePolicy{Name: ignore.Name, IgnorableErrors: slices.Clone(ignore.Patterns), Message: ignore.Message, Signals: signals}) + } + return result, nil +} + +// unitWarnings reports findings about one unit as warnings, with paths relative to the project. +func unitWarnings(root string, warnings []model.Warning) []*pluginv1.Diagnostic { + relativeWarnings := slices.Clone(warnings) + for i := range relativeWarnings { + if relativeWarnings[i].Path == "" { + continue + } + if path, err := relative(root, relativeWarnings[i].Path); err == nil { + relativeWarnings[i].Path = path + } + } + diagnostics := warningDiagnostics(relativeWarnings) + for _, diagnostic := range diagnostics { + diagnostic.Detail.Operation = "prepare" + } + return diagnostics +} diff --git a/plugins/terragrunt/internal/provider/provider.go b/plugins/terragrunt/internal/provider/provider.go index 07478c501a0..9e409f7b182 100644 --- a/plugins/terragrunt/internal/provider/provider.go +++ b/plugins/terragrunt/internal/provider/provider.go @@ -84,9 +84,9 @@ func settings(value *pluginv1.ComponentValue) error { } for _, key := range slices.Sorted(maps.Keys(config)) { switch key { - case "enabled", "provider", "mode", "discovery": + case "enabled", "provider", "mode", "discovery", "features": default: - return settingProblem(key, fmt.Sprintf("Unknown setting `components.terraform.terragrunt.%s`", key), "Valid keys are `enabled`, `provider`, `mode`, and `discovery`. Check the spelling.") + return settingProblem(key, fmt.Sprintf("Unknown setting `components.terraform.terragrunt.%s`", key), "Valid keys are `enabled`, `provider`, `mode`, `discovery`, and `features`. Check the spelling.") } } if mode, _ := config["mode"].(string); mode != "" && mode != "native" { @@ -96,7 +96,7 @@ func settings(value *pluginv1.ComponentValue) error { } func operation(root, name string, op *pluginv1.ComponentOperation, invocation *pluginv1.Invocation) (model.Request, error) { - if name != "terragrunt" || !filepath.IsAbs(root) || op == nil || op.Mode != "native" || len(op.Options) != 0 { + if name != "terragrunt" || !filepath.IsAbs(root) || op == nil || (op.Mode != "native" && op.Mode != "iam-probe") || len(op.Options) != 0 { return model.Request{}, ErrUnsupported } env := maps.Clone(invocation.GetEnvironment()) @@ -107,13 +107,16 @@ func operation(root, name string, op *pluginv1.ComponentOperation, invocation *p if binary == "" { binary = "terraform" } - return model.Request{BasePath: root, Mode: "native", Command: op.Command, Arguments: slices.Clone(op.TerraformArguments), Tofu: binary, Environment: env}, nil + request := model.Request{BasePath: root, Mode: "native", Command: op.Command, Arguments: slices.Clone(op.TerraformArguments), Tofu: binary, Environment: env} + if op.Mode == "iam-probe" { + request.Phase = "iam" + } + return request, nil } +// engineName names the engine of an executable with the rule the engine package shares with the host. func engineName(binary string) string { - // Match the host's executable-name detection, which treats any basename containing - // "tofu", case-insensitively, as OpenTofu. - if strings.Contains(strings.ToLower(filepath.Base(binary)), "tofu") { + if engine.IsOpenTofu(binary) { return "opentofu" } return "terraform" diff --git a/plugins/terragrunt/internal/provider/provider_test.go b/plugins/terragrunt/internal/provider/provider_test.go index 1777dc05d31..3b7471c7b1b 100644 --- a/plugins/terragrunt/internal/provider/provider_test.go +++ b/plugins/terragrunt/internal/provider/provider_test.go @@ -61,6 +61,33 @@ func TestPreparationAndStateUseSeparatePhases(t *testing.T) { require.Equal(t, result.State.WorkingDirectory, state.State.WorkingDirectory) } +func TestIAMProbeReturnsOnlyRoleSettings(t *testing.T) { + request := componentRequest(t) + request.Operation.Mode = "iam-probe" + p := New(func(_ context.Context, req model.Request) (*model.Resolution, error) { + require.Equal(t, "iam", req.Phase) + return &model.Resolution{IAM: &model.IAM{Role: "arn:aws:iam::1:role/x"}}, nil + }) + result, err := p.Prepare(t.Context(), request) + require.NoError(t, err) + require.Equal(t, "arn:aws:iam::1:role/x", result.Iam.Role) + require.Empty(t, result.ModuleSource) + require.Nil(t, result.State) +} + +func TestPreparationCarriesSensitiveRegistryAuthorization(t *testing.T) { + request := componentRequest(t) + p := New(func(_ context.Context, _ model.Request) (*model.Resolution, error) { + return &model.Resolution{Source: "https://registry.example/archive.tar.gz", WorkingDir: request.RootDirectory, RegistryAuth: &model.RegistryAuth{Origin: "https://registry.example", Authorization: "Bearer secret"}, Inputs: map[string]any{}, BackendType: "local", Backend: map[string]any{}}, nil + }) + response, err := p.Prepare(t.Context(), request) + require.NoError(t, err) + require.Equal(t, "https://registry.example", response.ModuleSourceAuthOrigin) + require.Len(t, response.ModuleSourceAuthorization.SensitivePaths, 1) + require.Empty(t, response.ModuleSourceAuthorization.SensitivePaths[0].Steps) + require.NotContains(t, response.ModuleSource, "secret") +} + func TestRequestsRejectUnsupportedBeforeEvaluation(t *testing.T) { for _, test := range []struct { name string @@ -218,6 +245,189 @@ func TestUnsupportedExecutionIsReportedSpecifically(t *testing.T) { require.NotEmpty(t, detail.Unit) } +func TestPrepareReturnsIAMAsDataWithASensitiveToken(t *testing.T) { + request := componentRequest(t) + p := New(func(_ context.Context, req model.Request) (*model.Resolution, error) { + if req.Phase == "discover" { + return &model.Resolution{}, nil + } + return &model.Resolution{Source: "git::https://example.test/m.git", WorkingDir: filepath.Join(request.RootDirectory, "cache"), IAM: &model.IAM{Role: "arn:aws:iam::1:role/x", SessionName: "s", DurationSeconds: 7200, WebIdentityToken: "token-secret"}}, nil + }) + result, err := p.Prepare(t.Context(), request) + require.NoError(t, err) + require.Equal(t, "arn:aws:iam::1:role/x", result.Iam.Role) + require.Equal(t, "s", result.Iam.SessionName) + require.Equal(t, uint32(7200), result.Iam.DurationSeconds) + token, err := component.Decode(result.Iam.WebIdentityToken) + require.NoError(t, err) + require.True(t, token.ContainsMarked(), "the token is a sensitive value") + require.Len(t, result.Iam.WebIdentityToken.SensitivePaths, 1) + require.Empty(t, result.Iam.WebIdentityToken.SensitivePaths[0].Steps, "the whole value is sensitive") +} + +func TestIAMResponseOmitsWhatIsNotSet(t *testing.T) { + none, err := iamResponse(nil) + require.NoError(t, err) + require.Nil(t, none) + bare, err := iamResponse(&model.IAM{Role: "r", DurationSeconds: -5}) + require.NoError(t, err) + require.Nil(t, bare.WebIdentityToken) + require.Zero(t, bare.DurationSeconds) + huge, err := iamResponse(&model.IAM{Role: "r", DurationSeconds: 1 << 40}) + require.NoError(t, err) + require.Equal(t, uint32(1<<32-1), huge.DurationSeconds) +} + +func TestFeatureSettingsAndEnvironmentMerge(t *testing.T) { + config, err := encode(map[string]any{"enabled": true, "features": map[string]any{"flag": true, "size": 3, "tier": "gold", "env": "config"}}) + require.NoError(t, err) + req := model.Request{Environment: map[string]string{"TG_FEATURE": "env=environment, extra=1"}} + require.NoError(t, withFeatures(&req, config)) + require.Equal(t, map[string]string{"flag": "true", "size": "3", "tier": "gold", "env": "environment", "extra": "1"}, req.Features, + "TG_FEATURE replaces the configured value per name") + + none := model.Request{} + require.NoError(t, withFeatures(&none, nil)) + require.Nil(t, none.Features) + + for name, bad := range map[string]any{ + "not a map": map[string]any{"features": "x"}, + "nested value": map[string]any{"features": map[string]any{"a": map[string]any{"b": 1}}}, + "list value": map[string]any{"features": map[string]any{"a": []any{1}}}, + } { + t.Run(name, func(t *testing.T) { + value, err := encode(bad) + require.NoError(t, err) + var request model.Request + require.Error(t, withFeatures(&request, value)) + }) + } + unknown, err := encode(map[string]any{"features": map[string]any{}, "typo": 1}) + require.NoError(t, err) + require.Error(t, settings(unknown)) + invalid := model.Request{Environment: map[string]string{"TG_FEATURE": "novalue"}} + require.Error(t, withFeatures(&invalid, nil)) +} + +func TestOperationCommandNamesTheTerragruntAction(t *testing.T) { + for _, test := range []struct { + command string + arguments []string + want string + }{ + {"plan", nil, "plan"}, + {"deploy", nil, "apply"}, + {"destroy", nil, "destroy"}, + {"apply", []string{"-destroy"}, "destroy"}, + {"apply", []string{"--destroy=true"}, "destroy"}, + {"apply", []string{"-destroy=false"}, "apply"}, + {"apply", []string{"-destroy", "-destroy=false"}, "apply"}, + {"plan", []string{"-destroy"}, "plan"}, + } { + require.Equal(t, test.want, operationCommand(test.command, test.arguments), "%s %v", test.command, test.arguments) + } +} + +func TestPrepareReturnsParityData(t *testing.T) { + request := componentRequest(t) + yes := true + p := New(func(_ context.Context, req model.Request) (*model.Resolution, error) { + if req.Phase == "discover" { + return &model.Resolution{}, nil + } + return &model.Resolution{ + Source: filepath.Join(request.RootDirectory, "module"), WorkingDir: filepath.Join(request.RootDirectory, "cache"), + Inputs: map[string]any{}, BackendType: "local", Backend: map[string]any{}, + TerraformBinary: "tofu", FeatureFlags: map[string]string{"flag": "true"}, + Exclude: &model.Exclude{Actions: []string{"all"}, NoRun: &yes, ExcludeDependencies: true}, + Errors: &model.Errors{ + Retry: []model.RetryPolicy{{Name: "net", Patterns: []string{"timeout"}, MaxAttempts: 3, SleepIntervalSec: 5}}, + Ignore: []model.IgnorePolicy{{Name: "known", Patterns: []string{"exists"}, Message: "fine", Signals: map[string]any{"note": "kept", "nothing": nil}}}, + }, + Warnings: []model.Warning{{Code: "read_config_before_hook_ignored", Path: filepath.Join(request.RootDirectory, "live", "terragrunt.hcl"), Message: "never runs"}}, + }, nil + }) + result, err := p.Prepare(t.Context(), request) + require.NoError(t, err) + require.Equal(t, "tofu", result.TerraformBinary) + require.Equal(t, map[string]string{"flag": "true"}, result.FeatureFlags) + require.Equal(t, []string{"apply"}, result.Exclude.Actions, "the exclusion carries this operation, with no_run applied") + require.True(t, result.Exclude.Condition) + require.True(t, result.Exclude.ExcludeDependencies) + require.Equal(t, "net", result.Errors.Retry[0].Name) + require.EqualValues(t, 3, result.Errors.Retry[0].MaxAttempts) + require.EqualValues(t, 5, result.Errors.Retry[0].SleepIntervalSeconds) + require.Equal(t, []string{"exists"}, result.Errors.Ignore[0].IgnorableErrors) + require.Contains(t, result.Errors.Ignore[0].Signals, "note") + require.Contains(t, result.Errors.Ignore[0].Signals, "nothing") + require.Len(t, result.Diagnostics, 1) + require.Equal(t, "live/terragrunt.hcl", result.Diagnostics[0].Field) + require.Equal(t, pluginv1.Severity_SEVERITY_WARNING, result.Diagnostics[0].Detail.Severity) + require.Equal(t, "prepare", result.Diagnostics[0].Detail.Operation) +} + +// An ignored registry resolution failure crosses the boundary as a successful preparation that names the +// policy, and an ordinary preparation leaves the fields unset so the host runs the unit. +func TestPrepareReturnsAnIgnoredRun(t *testing.T) { + request := componentRequest(t) + for name, ignored := range map[string]*model.IgnoredRun{ + "ignored": {Policy: "registry", Message: "registry is down"}, + "not ignored": nil, + } { + t.Run(name, func(t *testing.T) { + p := New(func(_ context.Context, req model.Request) (*model.Resolution, error) { + return &model.Resolution{Inputs: map[string]any{}, Backend: map[string]any{}, WorkingDir: req.BasePath, RunIgnored: ignored}, nil + }) + result, err := p.Prepare(t.Context(), request) + require.NoError(t, err) + require.Equal(t, ignored != nil, result.GetRunIgnored()) + if ignored == nil { + require.Empty(t, result.GetIgnoredPolicy()) + require.Empty(t, result.GetIgnoredMessage()) + return + } + require.Equal(t, "registry", result.GetIgnoredPolicy()) + require.Equal(t, "registry is down", result.GetIgnoredMessage()) + }) + } +} + +func TestPrepareOmitsExclusionWhenTheRunIsNotPrevented(t *testing.T) { + request := componentRequest(t) + no := false + for name, exclude := range map[string]*model.Exclude{ + "no block": nil, + "no_run false": {Actions: []string{"apply"}, NoRun: &no}, + "other action": {Actions: []string{"plan"}}, + "all, no_run unset": {Actions: []string{"all"}}, + } { + t.Run(name, func(t *testing.T) { + p := New(func(_ context.Context, req model.Request) (*model.Resolution, error) { + return &model.Resolution{Inputs: map[string]any{}, Backend: map[string]any{}, Exclude: exclude, WorkingDir: req.BasePath}, nil + }) + result, err := p.Prepare(t.Context(), request) + require.NoError(t, err) + require.Nil(t, result.Exclude) + require.Nil(t, result.Errors) + }) + } +} + +func TestDiscoveryPublishesCommandAndExclude(t *testing.T) { + config := map[string]any{"settings": map[string]any{"terragrunt": map[string]any{}}} + discoveryParity(config, &model.Resolution{}) + require.NotContains(t, config, "command") + require.NotContains(t, config["settings"].(map[string]any)["terragrunt"], "exclude") + + discoveryParity(config, &model.Resolution{TerraformBinary: "tofu", Exclude: &model.Exclude{Actions: []string{"plan"}, ExcludeDependencies: true}}) + require.Equal(t, "tofu", config["command"]) + require.Equal(t, map[string]any{"actions": []string{"plan"}, "exclude_dependencies": true}, config["settings"].(map[string]any)["terragrunt"].(map[string]any)["exclude"]) + + require.Equal(t, "opentofu", unitEngine(&model.Request{Tofu: "terraform"}, &model.Resolution{TerraformBinary: "/opt/bin/tofu"})) + require.Equal(t, "terraform", unitEngine(&model.Request{Tofu: "terraform"}, &model.Resolution{})) + require.Equal(t, "opentofu", unitEngine(&model.Request{Tofu: "tofu"}, nil)) +} + func TestPreparationTransmitsUnitFileCopyProvenance(t *testing.T) { for _, copied := range []bool{false, true} { t.Run(map[bool]string{false: "explicit generation", true: "adjacent copy"}[copied], func(t *testing.T) { @@ -363,6 +573,19 @@ dependencies { paths = ["../network", "../network"] }`, []pluginv1.ComponentDepe } } +// The engine package and the provider classify executables with one rule. +func TestEngineNameMatchesTheSharedOpenTofuRule(t *testing.T) { + for _, binary := range []string{"tofu", "opentofu", "tofu-wrapper", "TOFU.exe", "terraform", "", "/opt/tofu/bin/terraform"} { + want := "terraform" + if engine.IsOpenTofu(binary) { + want = "opentofu" + } + require.Equal(t, want, engineName(binary), binary) + } + require.Equal(t, "opentofu", engineName("TOFU.exe")) + require.Equal(t, "terraform", engineName("/opt/tofu/bin/terraform")) +} + func TestRPCErrorPreservesDestroyProtection(t *testing.T) { err := rpcError(fmt.Errorf("private evaluator context: %w", engine.ErrPreventDestroy)) require.Equal(t, codes.InvalidArgument, status.Code(err)) diff --git a/plugins/terragrunt/main.go b/plugins/terragrunt/main.go index 62eea063af2..2e6927ca2e8 100644 --- a/plugins/terragrunt/main.go +++ b/plugins/terragrunt/main.go @@ -32,7 +32,7 @@ func description() *pluginv1.DescribeResponse { return &pluginv1.DescribeResponse{ PluginName: manifest.Metadata.Name, PluginVersion: manifest.Metadata.Version, ManifestDigest: "sha256:" + hex.EncodeToString(digest[:]), - Services: []*pluginv1.ServiceVersionRange{{Service: "component.discovery", Min: 1, Max: 1}, {Service: "component.validation", Min: 1, Max: 1}, {Service: "component.preparation", Min: 1, Max: 1}, {Service: "component.state", Min: 1, Max: 1}}, + Services: []*pluginv1.ServiceVersionRange{{Service: "component.discovery", Min: 1, Max: 1}, {Service: "component.validation", Min: 1, Max: 1}, {Service: "component.preparation", Min: 2, Max: 2}, {Service: "component.state", Min: 1, Max: 1}}, Exports: []*pluginv1.Export{{Service: "component.discovery", Kind: providerKind}, {Service: "component.validation", Kind: providerKind}, {Service: "component.preparation", Kind: providerKind}, {Service: "component.state", Kind: providerKind}}, } } diff --git a/plugins/terragrunt/main_test.go b/plugins/terragrunt/main_test.go index 8288113afc1..4e77ab283ff 100644 --- a/plugins/terragrunt/main_test.go +++ b/plugins/terragrunt/main_test.go @@ -13,10 +13,13 @@ func TestDescriptionMatchesManifestAndIndependentServices(t *testing.T) { digest := sha256.Sum256(manifestData) require.Equal(t, "sha256:"+hex.EncodeToString(digest[:]), got.ManifestDigest) require.Equal(t, "cloudposse/terragrunt", got.PluginName) + // Preparation version 2 carries execution metadata, Terraform arguments, and + // backend ownership. An earlier host only speaks version 1 and rejects it. + versions := map[string]uint32{"component.discovery": 1, "component.validation": 1, "component.preparation": 2, "component.state": 1} services := []string{} for _, service := range got.Services { - require.EqualValues(t, 1, service.Min) - require.EqualValues(t, 1, service.Max) + require.Equal(t, versions[service.Service], service.Min, service.Service) + require.Equal(t, versions[service.Service], service.Max, service.Service) services = append(services, service.Service) } require.Equal(t, []string{"component.discovery", "component.validation", "component.preparation", "component.state"}, services) diff --git a/plugins/terragrunt/plugin.tests.yaml b/plugins/terragrunt/plugin.tests.yaml index 99f79bf95ad..c0bbb71d48c 100644 --- a/plugins/terragrunt/plugin.tests.yaml +++ b/plugins/terragrunt/plugin.tests.yaml @@ -118,3 +118,63 @@ integration: script: | hardening = module.load("scripts/hardening.star") hardening.pinned_tools() + - name: state-encryption + type: script + interpreter: starlark + script: | + identity = module.load("scripts/identity_sources.star") + identity.state_encryption() + - name: disable-init + type: script + interpreter: starlark + script: | + identity = module.load("scripts/identity_sources.star") + identity.disable_init() + - name: registry-sources + type: script + interpreter: starlark + script: | + identity = module.load("scripts/identity_sources.star") + identity.registry_sources() + - name: iam-role + type: script + interpreter: starlark + script: | + identity = module.load("scripts/identity_sources.star") + identity.iam_role() + - name: semantics-features + type: script + interpreter: starlark + script: | + semantics = module.load("scripts/semantics.star") + semantics.features() + - name: semantics-terraform-binary + type: script + interpreter: starlark + script: | + semantics = module.load("scripts/semantics.star") + semantics.terraform_binary() + - name: semantics-errors-block + type: script + interpreter: starlark + script: | + semantics = module.load("scripts/semantics.star") + semantics.errors_block() + - name: semantics-exclude-block + type: script + interpreter: starlark + script: | + semantics = module.load("scripts/semantics.star") + semantics.exclude_block() + - name: semantics-lifecycle-hooks + type: script + interpreter: starlark + script: | + semantics = module.load("scripts/semantics.star") + semantics.lifecycle_hooks() + - name: semantics-tflint-hook + type: script + interpreter: starlark + script: | + semantics = module.load("scripts/semantics.star") + semantics.tflint_hook() diff --git a/plugins/terragrunt/plugin.yaml b/plugins/terragrunt/plugin.yaml index 4cb32a20527..e10d15805aa 100644 --- a/plugins/terragrunt/plugin.yaml +++ b/plugins/terragrunt/plugin.yaml @@ -30,7 +30,7 @@ spec: contracts: component.discovery: '>=1' component.validation: '>=1' - component.preparation: '>=1' + component.preparation: '>=2 <3' component.state: '>=1' capabilities: components: diff --git a/plugins/terragrunt/schemas/config.json b/plugins/terragrunt/schemas/config.json index bc02cf66490..8ad46f931a1 100644 --- a/plugins/terragrunt/schemas/config.json +++ b/plugins/terragrunt/schemas/config.json @@ -11,6 +11,17 @@ "mode": { "const": "native" }, + "features": { + "type": "object", + "additionalProperties": { + "type": [ + "boolean", + "number", + "string" + ] + }, + "description": "Feature flag overrides applied to every unit, by feature name. TG_FEATURE overrides these per name." + }, "discovery": { "type": "object", "properties": { diff --git a/plugins/terragrunt/scripts/diagnostics.star b/plugins/terragrunt/scripts/diagnostics.star index a6b3a56b064..fb602cdb439 100644 --- a/plugins/terragrunt/scripts/diagnostics.star +++ b/plugins/terragrunt/scripts/diagnostics.star @@ -65,13 +65,7 @@ def verify_rejections_name_file_unit_and_setting(root): expect(syntax, ["Missing argument separator", "live/dev/syntax/terragrunt.hcl", "Component syntax in stack live/dev"], "syntax error") fixtures.check(syntax.count("live/dev/syntax/terragrunt.hcl:") > 0, "syntax error has no line and column:\n" + syntax) for name, setting, hint in [ - ("binary", "terraform_binary", "components.terraform.command"), - ("errors", "errors", "retry"), - ("exclude", "exclude", "excluded_paths"), - ("iam", "iam_role", "identity"), - ("feature", "feature", "stack variables"), - ("lint", "tflint", "wrapper script"), - ("lifecycle", "init-from-module", "Remove the lifecycle command"), + ("errors", "errors.retry", "max_attempts"), ("escape", "generate", "relative"), ]: output = plan(root, name) @@ -114,7 +108,7 @@ def verify_bulk_runs_validate_before_running(root, invocations): for operation in ["plan", "apply", "destroy"]: arguments = ["terraform", operation, "--all", "-s", STACK, "--"] + (["-auto-approve"] if operation != "plan" else []) + ["-no-color"] output = output_of(fixtures.run_failing(root, arguments)) - expect(output, ["no infrastructure command was run", "Component syntax", "Component binary"], operation + " --all") + expect(output, ["no infrastructure command was run", "Component syntax", "Component errors"], operation + " --all") ran = fs.read_file(invocations) if fs.exists(invocations) else "" fixtures.check(ran.strip() == "", operation + " --all ran the engine before rejecting the broken units:\n" + ran) fixtures.check(not fs.exists(root + "/live/dev/good/state/terraform.tfstate"), operation + " --all ran a healthy unit before rejecting the broken ones") @@ -126,13 +120,7 @@ def diagnostics(): units = { "good": GOOD, "syntax": 'terraform {\n source = "../../../modules/unit"\n}\ninputs = {\n a = find_in_parent_folders("x"\n}\n', - "binary": unit("") + 'terraform_binary = "terraform"\n', - "errors": unit("") + 'errors {\n retry "x" {\n retryable_errors = [".*"]\n max_attempts = 2\n sleep_interval_sec = 1\n }\n}\n', - "exclude": unit("") + 'exclude {\n if = true\n actions = ["plan"]\n}\n', - "iam": unit("") + 'iam_role = "arn:aws:iam::123456789012:role/x"\n', - "feature": unit("") + 'feature "flag" {\n default = true\n}\n', - "lint": unit(' before_hook "lint" {\n commands = ["plan"]\n execute = ["tflint"]\n }\n'), - "lifecycle": unit(' before_hook "setup" {\n commands = ["init-from-module"]\n execute = ["echo", "x"]\n }\n'), + "errors": unit("") + 'errors {\n retry "x" {\n retryable_errors = [".*"]\n max_attempts = 0\n sleep_interval_sec = 1\n }\n}\n', "escape": unit("") + 'generate "x" {\n path = "../escape.tf"\n if_exists = "overwrite"\n contents = ""\n}\n', "secret": unit("") + 'inputs = {\n a = tonumber("hunter2")\n}\n', } diff --git a/plugins/terragrunt/scripts/identity_sources.star b/plugins/terragrunt/scripts/identity_sources.star new file mode 100644 index 00000000000..7309e57ac7f --- /dev/null +++ b/plugins/terragrunt/scripts/identity_sources.star @@ -0,0 +1,227 @@ +# Opt-in CLI acceptance tests for the IAM, state encryption, and registry source features of native +# Terragrunt execution. Every fixture is generated here and uses local state and built-in terraform_data +# resources. State encryption needs OpenTofu; the other scenarios also run with Terraform. Nothing reaches a cloud: the IAM scenarios end at an unreachable STS endpoint, +# and the successful STS exchange is covered by Go tests against an STS test server. +# +# Importing this module has no effects; each integration step calls one scenario function. +fixtures = module.load("fixtures.star") +lifecycle = module.load("lifecycle_extended.star") + +STACK = lifecycle.STACK +PASSPHRASE = "correct-horse-battery-staple" +ROLE = "arn:aws:iam::123456789012:role/terragrunt-deploy" +WEB_TOKEN = "web-identity-token-must-stay-secret" +UNREACHABLE_STS = "http://127.0.0.1:1" + +MODULE_UNIT = """ +terraform { + source = "../../../modules/unit" +} +""" + + +def tofu(): + binary = exec.which("tofu") + fixtures.check(binary != None, "Install OpenTofu for integration tests") + return binary + + +def engine(): + binary = exec.which("tofu") or exec.which("terraform") + fixtures.check(binary != None, "Install OpenTofu or Terraform for integration tests") + return binary + + +def new_project(prefix, auth=False, binary=None): + binary = binary or engine() + work = fixtures.workspace(prefix) + root = work + "/project" + config = fixtures.configuration(fs.resolve("."), binary=binary) + if auth: + config["auth"] = { + "providers": {"mock-provider": {"kind": "mock/aws"}}, + "identities": { + "mock-identity": {"kind": "mock/aws", "default": True, "via": {"provider": "mock-provider"}, "principal": {"account": {"id": "123456789012"}}}, + }, + } + fixtures.write(root, "atmos.yaml", json.encode(config)) + fixtures.write(root, "modules/unit/main.tf", lifecycle.UNIT_MODULE) + return root, work + + +def write_hcl(root, name, content): + fixtures.write(root, "live/dev/" + name + "/terragrunt.hcl", content) + + +def invoke(root, work, arguments, extra={}): + env = { + "NO_COLOR": "1", + "ATMOS_TELEMETRY_ENABLED": "false", + "TF_IN_AUTOMATION": "true", + "ATMOS_KEYRING_TYPE": "memory", + "XDG_CONFIG_HOME": work + "/xdg", + "AWS_EC2_METADATA_DISABLED": "true", + } + env.update(extra) + return atmos.run(arguments, working_directory=root, env=env, output="capture", check=False) + + +def output_of(result): + return result.stdout + result.stderr + + +def terraform(root, work, operation, unit, extra={}, flags=[], failing=False): + arguments = ["terraform", operation, unit, "-s", STACK] + flags + ["--"] + (["-auto-approve"] if operation == "apply" else []) + ["-no-color"] + result = invoke(root, work, arguments, extra) + if failing: + fixtures.check(result.exit_code != 0, " ".join(arguments) + " unexpectedly succeeded:\n" + output_of(result)) + else: + fixtures.check(result.exit_code == 0, " ".join(arguments) + ":\n" + output_of(result)) + return result + + +def expect(result, texts, context): + for text in texts: + fixtures.check(text in output_of(result), "Expected '" + text + "' in " + context + ":\n" + output_of(result)) + + +def forbid(result, texts, context): + for text in texts: + fixtures.check(text not in output_of(result), "Did not expect '" + text + "' in " + context + ":\n" + output_of(result)) + + +ENCRYPTED_BACKEND = """ +remote_state { + backend = "local" + generate = { path = "backend.tf", if_exists = "overwrite_terragrunt" } + config = { path = "${get_terragrunt_dir()}/state/terraform.tfstate" } + encryption = { + key_provider = "pbkdf2" + passphrase = "@PASSPHRASE@" + } +} +""" + + +def state_encryption(): + if exec.which("tofu") == None: + ui.warning("State encryption scenario needs OpenTofu; skipped") + return + root, work = new_project("atmos-terragrunt-encryption-", binary=tofu()) + write_hcl(root, "secure", MODULE_UNIT + ENCRYPTED_BACKEND.replace("@PASSPHRASE@", PASSPHRASE)) + applied = terraform(root, work, "apply", "secure") + expect(applied, ["Apply complete! Resources: 1 added"], "apply with state encryption") + forbid(applied, [PASSPHRASE], "apply with state encryption") + + # The state OpenTofu wrote is an encrypted envelope, not readable JSON state. + state = fs.read_file(lifecycle.state_path(root, "secure")) + fixtures.check("encrypted_data" in state, "The state is not encrypted:\n" + state) + fixtures.check("terraform_data" not in state and "\"resources\"" not in state, "The state holds plaintext resources:\n" + state) + + # The key configuration is stable: a later plan decrypts the state and finds nothing to do. + planned = terraform(root, work, "plan", "secure") + expect(planned, ["No changes."], "plan after an encrypted apply") + forbid(planned, [PASSPHRASE], "plan after an encrypted apply") + + # Encryption without generate has no effect in Terragrunt, and the state stays plain. + write_hcl(root, "plain", MODULE_UNIT + """ +remote_state { + backend = "local" + config = { path = "${get_terragrunt_dir()}/state/terraform.tfstate" } + encryption = { key_provider = "pbkdf2", passphrase = "@PASSPHRASE@" } +} +""".replace("@PASSPHRASE@", PASSPHRASE)) + ignored = terraform(root, work, "apply", "plain") + expect(ignored, ["no effect without remote_state.generate"], "encryption without generate") + forbid(ignored, [PASSPHRASE], "encryption without generate") + fixtures.check("encrypted_data" not in fs.read_file(lifecycle.state_path(root, "plain")), "State was encrypted without generate") + + # A module that already ships the backend file would keep it under if_exists = "skip", so the + # generated encryption block would never be written and the state would stay plain. Atmos refuses + # the run before writing anything or running the engine, and leaves the module's file alone. + fixtures.write(root, "modules/kept/main.tf", lifecycle.UNIT_MODULE) + fixtures.write(root, "modules/kept/backend.tf", "terraform {\n backend \"local\" {}\n}\n") + write_hcl(root, "kept", """ +terraform { + source = "../../../modules/kept" +} +""" + ENCRYPTED_BACKEND.replace("overwrite_terragrunt", "skip").replace("@PASSPHRASE@", PASSPHRASE)) + kept = terraform(root, work, "plan", "kept", failing=True) + expect(kept, ["cannot verify backend identity for retained generated file backend.tf"], "skipped encrypted backend") + forbid(kept, [PASSPHRASE, "Plan:"], "skipped encrypted backend") + ui.success("State encryption: pbkdf2 encrypts the OpenTofu state, plans decrypt it, and the passphrase stays out of output") + + +def disable_init(): + root, work = new_project("atmos-terragrunt-disableinit-") + write_hcl(root, "no-bootstrap", MODULE_UNIT + """ +remote_state { + backend = "local" + disable_init = true + config = { path = "${get_terragrunt_dir()}/state/terraform.tfstate" } +} +""") + # Atmos never creates backend resources, and OpenTofu initializes the backend as usual. + applied = terraform(root, work, "apply", "no-bootstrap") + expect(applied, ["Apply complete! Resources: 1 added"], "apply with disable_init") + fixtures.check(fs.exists(lifecycle.state_path(root, "no-bootstrap")), "disable_init prevented the backend from being used") + ui.success("disable_init: accepted, and the backend initializes and stores state normally") + + +def registry_sources(): + root, work = new_project("atmos-terragrunt-registry-") + unit = """ +terraform { + source = "tfr://127.0.0.1:1/acme/vpc/aws//modules/nat?version=1.0.0" +} +""" + write_hcl(root, "registry", unit) + write_hcl(root, "plain", MODULE_UNIT) + + # Discovery never contacts the registry: the unit is listed and the plain unit still plans. + instances = invoke(root, work, ["list", "instances", "--format=json"]) + fixtures.check(instances.exit_code == 0, output_of(instances)) + expect(instances, ["registry", "plain"], "list instances with a registry source") + terraform(root, work, "plan", "plain") + + # Preparing the unit resolves the source through the registry protocol; an unreachable registry + # is reported against the setting, with the credentials it would use. + failed = terraform(root, work, "plan", "registry", failing=True) + expect(failed, ["could not resolve the tfr:// source", "terraform.source", "TF_TOKEN_"], "unreachable registry") + forbid(failed, ["unsupported plugin extension", "Unsupported module source transport"], "unreachable registry") + ui.success("Registry sources: discovery is offline, preparation resolves through the registry and reports failures against terraform.source") + + +IAM_UNIT = """ +terraform { + source = "../../../modules/unit" +} +iam_role = "@ROLE@" +iam_assume_role_session_name = "atmos-test" +iam_assume_role_duration = 1800 +iam_web_identity_token = "@TOKEN@" +""" + + +def iam_role(): + root, work = new_project("atmos-terragrunt-iam-", auth=True) + write_hcl(root, "assumed", IAM_UNIT.replace("@ROLE@", ROLE).replace("@TOKEN@", WEB_TOKEN)) + sts = {"AWS_ENDPOINT_URL_STS": UNREACHABLE_STS} + + # The settings are accepted at discovery: nothing is rejected before a command runs. + instances = invoke(root, work, ["list", "instances", "--format=json"], sts) + fixtures.check(instances.exit_code == 0, output_of(instances)) + expect(instances, ["assumed"], "list instances with iam_role") + + # With no identity selected, Atmos assumes the role itself, in process. The STS endpoint is + # unreachable, so the failure names the role and the way out, and never prints the token. + failed = terraform(root, work, "plan", "assumed", sts, failing=True) + expect(failed, [ROLE, "--identity"], "iam_role assumption failure") + forbid(failed, [WEB_TOKEN, "unsupported plugin extension", "Unsupported Terragrunt setting"], "iam_role assumption failure") + + # An Atmos identity selected explicitly decides who the component runs as: iam_role is not + # assumed, no STS call is made, and the command succeeds even with STS unreachable. + selected = terraform(root, work, "plan", "assumed", sts, flags=["--identity=mock-identity"]) + expect(selected, ["runs as Atmos identity mock-identity", ROLE], "explicit identity") + forbid(selected, [WEB_TOKEN], "explicit identity") + ui.success("iam_role: accepted at discovery, assumed in process otherwise, replaced by an explicit identity, token never printed") diff --git a/plugins/terragrunt/scripts/semantics.star b/plugins/terragrunt/scripts/semantics.star new file mode 100644 index 00000000000..6242e302d97 --- /dev/null +++ b/plugins/terragrunt/scripts/semantics.star @@ -0,0 +1,535 @@ +# Opt-in CLI acceptance tests for the Terragrunt settings that choose how a unit runs: the +# `terraform_binary` executable, the `errors` block, the `exclude` block, `feature` flags, and the +# `read-config` / `init-from-module` hook stages with the built-in `tflint` hook. Every fixture is +# generated here, uses only built-in terraform_data resources and real OpenTofu or Terraform with +# local state, and never touches a cloud. Importing this module has no effects. +fixtures = module.load("fixtures.star") +lifecycle = module.load("lifecycle_extended.star") +hardening = module.load("hardening.star") + +STACK = lifecycle.STACK + +ALT_MODULE = lifecycle.UNIT_MODULE.replace('input = "${var.greeting}|${var.suffix}|${var.tag}|${var.envvar}"', 'input = "alt-module"') + + +def engine(): + binary = exec.which("tofu") or exec.which("terraform") + fixtures.check(binary != None, "Install OpenTofu or Terraform for integration tests") + return binary + + +def is_opentofu(binary): + return "tofu" in binary.replace("\\", "/").rsplit("/", 1)[-1] + + +def new_project(prefix, binary=None): + root = lifecycle.project(fixtures.workspace(prefix), "project", fs.resolve("."), binary or engine()) + fixtures.write(root, "modules/alt/main.tf", ALT_MODULE) + return root + + +def output_of(result): + return result.stdout + result.stderr + + +def expect(result, text, context): + fixtures.check(text in output_of(result), "Expected '" + text + "' in " + context + ":\n" + output_of(result)) + + +def expect_absent(result, text, context): + fixtures.check(text not in output_of(result), "Did not expect '" + text + "' in " + context + ":\n" + output_of(result)) + + +def invoke(root, arguments, env={}, path=None): + # Extra environment variables reach Atmos exactly like a shell would pass them. + variables = {"NO_COLOR": "1", "ATMOS_TELEMETRY_ENABLED": "false", "TF_IN_AUTOMATION": "true"} + variables.update(env) + if path != None: + variables["PATH"] = path + return atmos.run(arguments, working_directory=root, env=variables, output="capture", check=False) + + +def run(root, arguments, env={}, path=None, failing=False): + result = invoke(root, arguments, env, path) + if failing: + fixtures.check(result.exit_code != 0, " ".join(arguments) + " unexpectedly succeeded:\n" + output_of(result)) + else: + fixtures.check(result.exit_code == 0, " ".join(arguments) + ":\n" + output_of(result)) + return result + + +def terraform(root, operation, unit, arguments=[], flags=[], env={}, path=None, failing=False, stack=STACK): + command = ["terraform", operation, unit, "-s", stack] + flags + ["--"] + arguments + ["-no-color"] + return run(root, command, env, path, failing) + + +def configure(root, binary, features=None, extra=None): + config = fixtures.configuration(fs.resolve("."), binary=binary) + if features != None: + config["components"]["terraform"]["terragrunt"]["features"] = features + if extra != None: + config.update(extra) + fixtures.write(root, "atmos.yaml", json.encode(config)) + + +def value_of(root, unit): + return lifecycle.state_output(root, unit) + + +# --------------------------------------------------------------------------------------------- +# Feature flags +# --------------------------------------------------------------------------------------------- + +FEATURE_UNIT = """ +terraform { + source = feature.alt.value ? "../../../modules/alt" : "../../../modules/unit" +} + +feature "alt" { + default = false +} +feature "wide" { + default = false +} +feature "size" { + default = 2 +} +feature "tier" { + default = "basic" +} + +inputs = { + greeting = feature.tier.value + suffix = tostring(feature.size.value) + tag = feature.wide.value ? "wide" : "narrow" +} +""" + +FEATURE_DEPENDENCY_UNIT = """ +terraform { + source = "../../../modules/unit" +} + +feature "link" { + default = false +} + +dependency "base" { + config_path = "../base" + skip_outputs = !feature.link.value + mock_outputs = { value = "no-link" } +} + +inputs = { + tag = dependency.base.outputs.value +} +""" + + +def features(): + binary = engine() + root = new_project("atmos-terragrunt-features-", binary) + lifecycle.write_unit(root, "featured", FEATURE_UNIT) + lifecycle.write_unit(root, "base", lifecycle.PLAN_UNIT.replace("@GREETING@", "base-value")) + lifecycle.write_unit(root, "linked", FEATURE_DEPENDENCY_UNIT) + + # Declared defaults apply when nothing overrides them: a string, a number, and a boolean. + configure(root, binary) + terraform(root, "apply", "featured", ["-auto-approve"]) + fixtures.check(value_of(root, "featured") == "basic|2|narrow|", "Defaults were not used: " + value_of(root, "featured")) + + # components.terraform.terragrunt.features overrides defaults, typed from the declaration. + configure(root, binary, features={"wide": True, "size": 5, "tier": "gold"}) + terraform(root, "apply", "featured", ["-auto-approve"]) + fixtures.check(value_of(root, "featured") == "gold|5|wide|", "Configured features were not used: " + value_of(root, "featured")) + + # TG_FEATURE outranks the configuration, per name; other configured names stay. + env = {"TG_FEATURE": "tier=premium,wide=false"} + terraform(root, "apply", "featured", ["-auto-approve"], env=env) + fixtures.check(value_of(root, "featured") == "premium|5|narrow|", "TG_FEATURE did not take precedence: " + value_of(root, "featured")) + + # A feature can choose the module source... + configure(root, binary, features={"alt": True}) + terraform(root, "apply", "featured", ["-auto-approve"]) + fixtures.check(value_of(root, "featured") == "alt-module", "The source feature was ignored: " + value_of(root, "featured")) + + # ...and a dependency: off, the mock output is used and the dependency is not required. + configure(root, binary) + terraform(root, "apply", "linked", ["-auto-approve"]) + fixtures.check("no-link" in value_of(root, "linked"), "The disabled dependency was read: " + value_of(root, "linked")) + terraform(root, "apply", "base", ["-auto-approve"]) + configure(root, binary, features={"link": True}) + terraform(root, "apply", "linked", ["-auto-approve"]) + fixtures.check("base-value" in value_of(root, "linked"), "The enabled dependency output was not used: " + value_of(root, "linked")) + + + # A bad boolean override names the feature and the way to fix it. + bad = terraform(root, "plan", "featured", env={"TG_FEATURE": "wide=sometimes"}, failing=True) + expect(bad, "wide", "boolean feature with a non boolean override") + expect(bad, "true", "boolean feature with a non boolean override") + malformed = terraform(root, "plan", "featured", env={"TG_FEATURE": "novalue"}, failing=True) + expect(malformed, "TG_FEATURE", "malformed TG_FEATURE") + + # An override no unit declares is reported by name and does not fail the command. + configure(root, binary, features={"wdie": True}) + typo = run(root, ["list", "instances"]) + expect(typo, "wdie", "listing with an unknown feature override") + ui.success("Feature flags: defaults, atmos.yaml and TG_FEATURE overrides, typed values, source and dependency toggles, clear errors") + + +# --------------------------------------------------------------------------------------------- +# terraform_binary +# --------------------------------------------------------------------------------------------- + +def invocations(directory): + log = directory + "/invocations.log" + return [line.split(" ")[0] for line in fs.read_file(log).splitlines()] if fs.exists(log) else [] + + +def terraform_binary(): + real = engine() + if not is_opentofu(real): + ui.warning("terraform_binary scenario installs OpenTofu probes; skipped without OpenTofu") + return + work = fixtures.workspace("atmos-terragrunt-binary-") + suffix = exec.run(["go", "env", "GOEXE"], output="capture").stdout.strip() + tools = work + "/tools" + pinned = hardening.install_probe(tools, "9.9.9", real, suffix) + absolute = hardening.install_probe(work + "/absolute-tools", "1.0.0", real, suffix) + root = work + "/project" + config = fixtures.configuration(fs.resolve("."), binary=real) + config["toolchain"] = {"install": "never", "install_path": tools} + config["stacks"] = {"base_path": "stacks", "included_paths": ["*.yaml"]} + fixtures.write(root, "atmos.yaml", json.encode(config)) + fixtures.write(root, ".tool-versions", "opentofu 9.9.9\n") + fixtures.write(root, "modules/unit/main.tf", lifecycle.UNIT_MODULE) + ambient = work + "/empty-path" + fs.mkdir(ambient, parents=True) + + def clear(directory): + if fs.exists(directory + "/invocations.log"): + fs.remove(directory + "/invocations.log") + + plain = lifecycle.PLAN_UNIT.replace("@GREETING@", "plain") + named = 'terraform_binary = "tofu"\n' + lifecycle.PLAN_UNIT.replace("@GREETING@", "named") + pathed = 'terraform_binary = "' + lifecycle.hcl_path(absolute) + '/tofu' + suffix + '"\n' + lifecycle.PLAN_UNIT.replace("@GREETING@", "pathed") + consumer = 'terraform_binary = "tofu"\ndependency "named" {\n config_path = "../named"\n}\ninputs = {\n tag = dependency.named.outputs.value\n}\n' + lifecycle.PLAN_UNIT.replace("@GREETING@", "consumer") + lifecycle.write_unit(root, "plain", plain) + lifecycle.write_unit(root, "named", named) + lifecycle.write_unit(root, "pathed", pathed) + lifecycle.write_unit(root, "consumer", consumer) + + # The real executable is the default; a unit that names an executable runs through the toolchain. + terraform(root, "apply", "plain", ["-auto-approve"], path=ambient) + fixtures.check(invocations(pinned) == [] and invocations(absolute) == [], "A unit without terraform_binary used a probe") + terraform(root, "apply", "named", ["-auto-approve"], path=ambient) + commands = invocations(pinned) + for command in ["init", "apply"]: + fixtures.check(command in commands, "The toolchain 'tofu' never ran '" + command + "' for terraform_binary: " + str(commands)) + fixtures.check(invocations(absolute) == [], "The wrong executable ran: " + str(invocations(absolute))) + + # An absolute path selects that exact executable. + terraform(root, "apply", "pathed", ["-auto-approve"], path=ambient) + fixtures.check("apply" in invocations(absolute), "The absolute terraform_binary never ran apply: " + str(invocations(absolute))) + + # Reading a dependency's outputs and the implicit init use the unit's executable. + clear(pinned) + terraform(root, "plan", "consumer", path=ambient) + commands = invocations(pinned) + for command in ["init", "output", "plan"]: + fixtures.check(command in commands, "The consumer did not use the selected executable for '" + command + "': " + str(commands)) + + # A stack's explicit `command` is the stronger override: the unit's executable is not used. + fixtures.write(root, "stacks/override.yaml", "import:\n - terragrunt://live/dev/named\ncomponents:\n terraform:\n named:\n command: " + real + "\n") + clear(pinned) + terraform(root, "plan", "named", stack="override", path=ambient) + fixtures.check(invocations(pinned) == [], "A stack command did not override terraform_binary: " + str(invocations(pinned))) + + # A missing executable names the component, the stack, and the executable. + lifecycle.write_unit(root, "missing", 'terraform_binary = "nxtfengine"\n' + plain) + missing = terraform(root, "plan", "missing", path=ambient, failing=True) + for text in ["missing", STACK, "nxtfengine", "not found"]: + expect(missing, text, "plan with a missing terraform_binary") + + # An executable that is not Terraform or OpenTofu is rejected before any command runs. + other = exec.which("go") + lifecycle.write_unit(root, "wrong", 'terraform_binary = "' + lifecycle.hcl_path(other) + '"\n' + plain) + wrong = terraform(root, "plan", "wrong", failing=True) + expect(wrong, "terraform_binary", "plan with a non engine executable") + expect(wrong, "not a Terraform or OpenTofu executable", "plan with a non engine executable") + ui.success("terraform_binary: toolchain name, absolute path, dependency reads and init, stack precedence, clear errors") + + +# --------------------------------------------------------------------------------------------- +# errors block +# --------------------------------------------------------------------------------------------- + +# A resource whose provisioner fails the first time and succeeds the second, so a retry has +# something to recover from. The marker lives beside the module, outside Terraform state. +FLAKY_MODULE = """ +variable "marker" { + type = string + default = "" +} +resource "terraform_data" "flaky" { + provisioner "local-exec" { + interpreter = ["sh", "-c"] + command = "test -f '${var.marker}' || { touch '${var.marker}'; echo transient-failure >&2; exit 1; }" + } +} +output "value" { + value = terraform_data.flaky.id != "" ? "recovered" : "" +} +""" + +FAILING_MODULE = """ +variable "marker" { + type = string + default = "" +} +resource "terraform_data" "broken" { + provisioner "local-exec" { + interpreter = ["sh", "-c"] + command = "test -z '${var.marker}' || echo attempt >> '${var.marker}'; echo known-problem >&2; exit 1" + } +} +output "value" { + value = "unreachable" +} +""" + + +def errors_unit(module, block, marker=None): + inputs = "" + if marker != None: + inputs = 'inputs = { marker = "' + lifecycle.hcl_path(marker) + '" }\n' + return 'terraform {\n source = "../../../modules/' + module + '"\n}\n' + inputs + block + + +def errors_block(): + binary = engine() + if exec.which("sh") == None: + ui.warning("errors scenario needs a POSIX shell for its failing provisioners; skipped") + return + root = new_project("atmos-terragrunt-errors-", binary) + fixtures.write(root, "modules/flaky/main.tf", FLAKY_MODULE) + fixtures.write(root, "modules/failing/main.tf", FAILING_MODULE) + marker = root + "/flaky.marker" + + retry = 'errors {\n retry "transient" {\n retryable_errors = [".*transient-failure.*"]\n max_attempts = 3\n sleep_interval_sec = 0\n }\n}\n' + lifecycle.write_unit(root, "retried", errors_unit("flaky", retry, marker)) + result = terraform(root, "apply", "retried", ["-auto-approve"]) + expect(result, "Retryable error matched policy transient", "apply with a retry policy") + expect(result, "attempt 1 of 3", "apply with a retry policy") + expect_absent(result, "attempt 2 of 3", "apply that recovered on the second attempt") + fixtures.check(lifecycle.state_output(root, "retried") == "recovered", "The retried apply did not complete") + + # A failure the policy does not cover is not retried, and a policy that runs out of attempts says so. + other = 'errors {\n retry "other" {\n retryable_errors = ["something-else"]\n max_attempts = 3\n sleep_interval_sec = 0\n }\n}\n' + lifecycle.write_unit(root, "unmatched", errors_unit("failing", other)) + unmatched = terraform(root, "apply", "unmatched", ["-auto-approve"], failing=True) + expect_absent(unmatched, "Retryable error matched", "apply of an unmatched failure") + always = 'errors {\n retry "always" {\n retryable_errors = [".*known-problem.*"]\n max_attempts = 2\n sleep_interval_sec = 0\n }\n}\n' + exhausted_marker = root + "/exhausted.attempts" + lifecycle.write_unit(root, "exhausted", errors_unit("failing", always, exhausted_marker)) + exhausted = terraform(root, "apply", "exhausted", ["-auto-approve"], failing=True) + expect(exhausted, "attempt 1 of 2", "apply with a retry policy that never succeeds") + expect(exhausted, "maximum retry attempts reached", "apply with a retry policy that never succeeds") + lifecycle.expect_count(output_of(exhausted), "attempt 1 of 2", 1, "exhausted retry policy") + lifecycle.expect_count(fs.read_file(exhausted_marker), "attempt", 2, "failing provisioner invocations") + + # A negative pattern vetoes the policy. + veto = 'errors {\n retry "vetoed" {\n retryable_errors = [".*known-problem.*", "!.*known-problem.*"]\n max_attempts = 3\n sleep_interval_sec = 0\n }\n}\n' + lifecycle.write_unit(root, "vetoed", errors_unit("failing", veto)) + vetoed = terraform(root, "apply", "vetoed", ["-auto-approve"], failing=True) + expect_absent(vetoed, "Retryable error matched", "apply with a vetoed retry policy") + + # An ignore policy turns the failure into success, prints its message, and writes its signals. + ignore = 'errors {\n ignore "known" {\n ignorable_errors = [".*known-problem.*"]\n message = "this failure is expected"\n signals = {\n alert = "none"\n scope = "test"\n }\n }\n retry "never" {\n retryable_errors = [".*"]\n max_attempts = 5\n sleep_interval_sec = 0\n }\n}\n' + lifecycle.write_unit(root, "ignored", errors_unit("failing", ignore)) + ignored = terraform(root, "apply", "ignored", ["-auto-approve"]) + expect(ignored, "this failure is expected", "apply with an ignore policy") + expect_absent(ignored, "Retryable error matched", "ignore wins over retry") + signals = fs.glob(root + "/.workdir/terraform/" + STACK + "-ignored-*/error-signals.json") + fixtures.check(len(signals) == 1, "Expected one signals file: " + str(signals)) + fixtures.check(json.decode(fs.read_file(signals[0])) == {"alert": "none", "scope": "test"}, "Unexpected signals: " + fs.read_file(signals[0])) + + # Invalid policies are rejected with the block named. + invalid = 'errors {\n retry "bad" {\n retryable_errors = ["x"]\n max_attempts = 0\n sleep_interval_sec = 0\n }\n}\n' + lifecycle.write_unit(root, "invalid", errors_unit("failing", invalid)) + rejected = terraform(root, "plan", "invalid", failing=True) + expect(rejected, "max_attempts", "plan with an invalid retry policy") + ui.success("errors block: retry until success, max attempts, veto patterns, ignore with message and signals, validation") + + +# --------------------------------------------------------------------------------------------- +# exclude block +# --------------------------------------------------------------------------------------------- + +def exclude_unit(block, dependency=None): + text = 'terraform {\n source = "../../../modules/unit"\n}\n' + block + if dependency != None: + text += 'dependency "' + dependency + '" {\n config_path = "../' + dependency + '"\n}\ninputs = {\n tag = dependency.' + dependency + '.outputs.value\n}\n' + return text + + + +def exclude_block(): + binary = engine() + root = new_project("atmos-terragrunt-exclude-", binary) + fixtures.write(root, "modules/broken/main.tf", 'resource "terraform_data" "broken" {\n lifecycle {\n precondition {\n condition = false\n error_message = "this unit always fails"\n }\n }\n}\n') + # core is app's dependency; app excludes its dependencies for apply, but can still be run by name. + lifecycle.write_unit(root, "core", exclude_unit("")) + lifecycle.write_unit(root, "app", exclude_unit('exclude {\n if = true\n actions = ["apply"]\n exclude_dependencies = true\n no_run = false\n}\n', "core")) + # With no_run unset only an exact action prevents even a direct run. + lifecycle.write_unit(root, "stopped", exclude_unit('exclude {\n if = true\n actions = ["apply"]\n}\n')) + # no_run = true also honors the "all" vocabulary. + lifecycle.write_unit(root, "frozen", exclude_unit('exclude {\n if = true\n actions = ["all"]\n no_run = true\n}\n')) + # A false condition excludes nothing. + lifecycle.write_unit(root, "off", exclude_unit('exclude {\n if = false\n actions = ["all"]\n}\n')) + + listed = output_of(run(root, ["list", "instances"])) + for unit in ["core", "app", "stopped", "frozen", "off"]: + fixtures.check(unit in listed, "list instances must show every unit including " + unit + ":\n" + listed) + + # A bulk apply leaves out every unit whose exclude block lists apply (here also through + # "all"), and app's dependency because app sets exclude_dependencies. + bulk = run(root, ["terraform", "apply", "--all", "-s", STACK, "--", "-auto-approve", "-no-color"]) + for unit in ["app", "stopped", "core", "frozen"]: + expect(bulk, "Skipped " + unit + " in " + STACK + ": excluded by the Terragrunt exclude block", "bulk apply") + fixtures.check(not fs.exists(lifecycle.state_path(root, unit)), unit + " was applied by a bulk run that excludes it") + expect_absent(bulk, "Skipped off", "bulk apply") + fixtures.check(fs.exists(lifecycle.state_path(root, "off")), "A false exclude condition still excluded the unit") + + # Naming a unit runs it unless no_run prevents the command. app reads core's outputs. + terraform(root, "apply", "core", ["-auto-approve"]) + terraform(root, "apply", "app", ["-auto-approve"]) + fixtures.check(fs.exists(lifecycle.state_path(root, "app")), "no_run = false must let an explicit apply run") + stopped = terraform(root, "apply", "stopped", ["-auto-approve"]) + expect(stopped, "no_run", "explicit apply of a unit that prevents it") + fixtures.check(not fs.exists(lifecycle.state_path(root, "stopped")), "A prevented apply still ran") + frozen = terraform(root, "plan", "frozen") + expect(frozen, "no_run", "explicit plan of a frozen unit") + + # Bulk commands the block does not list still include the unit. + planned = run(root, ["terraform", "plan", "--all", "-s", STACK, "--", "-no-color"]) + expect_absent(planned, "Skipped stopped", "bulk plan") + + # Components skipped after a failure are named. + lifecycle.write_unit(root, "broken", 'terraform {\n source = "../../../modules/broken"\n}\n') + lifecycle.write_unit(root, "after", exclude_unit("", "broken")) + failed = run(root, ["terraform", "plan", "--all", "-s", STACK, "--", "-no-color"], failing=True) + expect(failed, "skipped after an earlier failure", "bulk plan with a failing unit") + expect(failed, "after in " + STACK, "bulk plan with a failing unit") + ui.success("exclude block: bulk skip with named lines, dependencies, no_run semantics, explicit runs, unaffected listing") + + +# --------------------------------------------------------------------------------------------- +# read-config and init-from-module hooks, and the built-in tflint hook +# --------------------------------------------------------------------------------------------- + +LIFECYCLE_UNIT = """ +terraform { + source = "../../../modules/unit" + + # The relative path only exists in the unit directory, which is where these hooks run. + after_hook "read" { + commands = ["terragrunt-read-config"] + execute = ["@TOOL@", "fmt", "-check", "marker.tf"] + } + after_hook "read-alias" { + commands = ["read-config"] + execute = ["@TOOL@", "version"] + } + before_hook "never-runs" { + commands = ["terragrunt-read-config"] + execute = ["@TOOL@", "version"] + } + before_hook "fetch-before" { + commands = ["init-from-module"] + execute = ["@TOOL@", "fmt", "-check", "marker.tf"] + } + after_hook "fetch-after" { + commands = ["init-from-module"] + execute = ["@TOOL@", "version"] + } + error_hook "fetch-error" { + commands = ["init-from-module"] + execute = ["@TOOL@", "version"] + on_errors = [".*"] + } +} +""" + + +def lifecycle_hooks(): + binary = engine() + root = new_project("atmos-terragrunt-lifecycle-", binary) + tool = lifecycle.tool_of(binary) + fixtures.write(root, "modules/local/main.tf", lifecycle.versioned_module("v1")) + fixtures.write(root, "live/dev/staged/marker.tf", 'variable "marker" {}\n') + unit = LIFECYCLE_UNIT.replace("@TOOL@", tool).replace("../../../modules/unit", "../../../modules/local") + lifecycle.write_unit(root, "staged", unit) + + # The first plan fetches the module: read-config runs, then the fetch hooks, then the plan. + first = terraform(root, "plan", "staged") + err = first.stderr + lifecycle.expect_count(err, "Executing after hook: read (", 1, "first plan") + lifecycle.expect_count(err, "Executing after hook: read-alias", 1, "first plan") + lifecycle.expect_count(err, "Executing before hook: fetch-before", 1, "first plan") + lifecycle.expect_count(err, "Executing after hook: fetch-after", 1, "first plan") + lifecycle.expect_count(err, "Executing error hook", 0, "first plan") + lifecycle.expect_order(err, ["Executing after hook: read", "Executing before hook: fetch-before", "Executing after hook: fetch-after"], "first plan") + # A before hook that lists read-config can never run; the user is told once. + expect(first, "never-runs", "plan of a unit with a read-config before hook") + lifecycle.expect_count(err, "Executing before hook: never-runs", 0, "first plan") + + # An unchanged module is not fetched again, but read-config runs on every command. + second = terraform(root, "plan", "staged") + lifecycle.expect_count(second.stderr, "Executing after hook: read (", 1, "second plan") + lifecycle.expect_count(second.stderr, "Executing before hook: fetch-before", 0, "second plan") + lifecycle.expect_count(second.stderr, "Executing after hook: fetch-after", 0, "second plan") + + # An edited module is fetched again, so the fetch hooks run again. + fixtures.write(root, "modules/local/main.tf", lifecycle.versioned_module("v2")) + third = terraform(root, "plan", "staged") + lifecycle.expect_count(third.stderr, "Executing before hook: fetch-before", 1, "plan after an edit") + lifecycle.expect_count(third.stderr, "Executing after hook: fetch-after", 1, "plan after an edit") + ui.success("Lifecycle hooks: read-config on every command in the unit directory, init-from-module only when the source is fetched") + + +def tflint_hook(): + binary = engine() + echo = exec.which("echo") + if echo == None: + ui.warning("tflint scenario needs an echo executable to stand in for tflint; skipped") + return + work = fixtures.workspace("atmos-terragrunt-tflint-") + suffix = exec.run(["go", "env", "GOEXE"], output="capture").stdout.strip() + path_dir = work + "/path" + fs.mkdir(path_dir, parents=True) + exec.run(["go", "build", "-o", path_dir + "/tflint" + suffix, "./fixtures/toolprobe"], working_directory=fs.resolve("."), output="capture") + fs.write_file(path_dir + "/real-tool.txt", echo) + root = new_project("atmos-terragrunt-tflint-", binary) + fixtures.write(root, ".tflint.hcl", "# test configuration\n") + unit = 'terraform {\n source = "../../../modules/unit"\n\n before_hook "lint" {\n commands = ["plan"]\n execute = ["tflint", "--terragrunt-external-tflint", "--minimum-failure-severity=error"]\n }\n}\ninputs = { greeting = "hi" }\n' + lifecycle.write_unit(root, "linted", unit) + windows = exec.run(["go", "env", "GOOS"], output="capture").stdout.strip() == "windows" + system = [echo.replace("\\", "/").rsplit("/", 1)[0]] if windows else ["/usr/bin", "/bin"] + result = terraform(root, "plan", "linted", path=(";" if windows else ":").join([path_dir] + system)) + lifecycle.expect_count(result.stderr, "Executing before hook: lint", 2, "plan with a tflint hook") + lines = fs.read_file(path_dir + "/invocations.log").splitlines() + fixtures.check(len(lines) == 2, "tflint should run twice (init and lint): " + str(lines)) + fixtures.check(lines[0].startswith("--init --config ") and lines[0].endswith("--chdir ."), "Unexpected tflint init: " + lines[0]) + fixtures.check(".tflint.hcl" in lines[1] and "--var=greeting=hi" in lines[1] and "--minimum-failure-severity=error" in lines[1], "Unexpected tflint run: " + lines[1]) + fixtures.check("--terragrunt-external-tflint" not in lines[1], "The external flag must not reach tflint: " + lines[1]) + ui.success("tflint hook: runs tflint --init and tflint with the unit's configuration and inputs") + + +def main(): + features() + terraform_binary() + errors_block() + exclude_block() + lifecycle_hooks() + tflint_hook() diff --git a/proto/atmos/plugin/v1/plugin.proto b/proto/atmos/plugin/v1/plugin.proto index 0b261ef8e5a..3038c60f528 100644 --- a/proto/atmos/plugin/v1/plugin.proto +++ b/proto/atmos/plugin/v1/plugin.proto @@ -184,6 +184,8 @@ service Commands { // implement an executor or replace the built-in Terraform component provider. // Negotiate each service through Info using component.discovery, // component.validation, component.preparation, and component.state respectively. +// component.preparation is version 2: it adds terraform_arguments, +// backend_generated, and execution. A version-1 host must reject the plugin. // ComponentValue uses cty's JSON type encoding and MessagePack value encoding. // Unlike protobuf.Struct, it preserves exact numbers, nulls, unknowns, and @@ -370,24 +372,44 @@ message PrepareComponentResponse { // its value, for provenance reporting. Missing entries mean unknown origin. map input_origins = 12; - // Fields 13-18 are reserved for Terragrunt parity. The host does not read - // them yet; a plugin that cannot honor a setting must reject it with an - // ErrorDetail instead of returning it, and must leave these fields unset. - // Engine executable selected by terraform_binary. + // Fields 13-18 carry Terragrunt settings that decide how the host runs the unit. The host reads + // terraform_binary, errors, exclude, iam, and feature_flags (13-16 and 18). It does not read + // encryption (17) yet. A plugin that cannot honor a setting must reject it with an ErrorDetail + // instead of returning a value the host would silently ignore. + // Engine executable selected by terraform_binary. Empty means the engine of the component. string terraform_binary = 13; - // errors block: retry and ignore policies. + // errors block: retry and ignore policies. The host applies them around the engine subprocess. ComponentErrorsConfiguration errors = 14; - // exclude block: conditional unit exclusion. + // exclude block, set only when its condition holds and it applies to the current operation. + // The host skips the operation for the component and runs nothing. See ComponentExclude. ComponentExclude exclude = 15; // iam_role, iam_assume_role_session_name, iam_assume_role_duration, iam_web_identity_token. + // The host assumes the role, never the plugin. ComponentIam iam = 16; - // encryption block as opaque, provider-defined data. + // encryption block as opaque, provider-defined data. The host ignores it. ComponentValue encryption = 17; - // feature flag name to its evaluated value rendered as a string. + // Feature flag name to its evaluated value rendered as a string. The plugin has already applied + // the flags while evaluating the unit, so they are reflected in every other field. The host + // keeps them with the prepared configuration. map feature_flags = 18; + // Authorization for an archive returned by a module registry. This value is + // sensitive and may be sent only to module_source_auth_origin. + ComponentValue module_source_authorization = 19; + string module_source_auth_origin = 20; + // Time spent resolving a tfr:// source, including its bounded retries. + uint64 registry_resolution_nanos = 21; + // True when an errors-block ignore policy matched a failure the plugin hit while preparing the unit, + // for example a registry resolution failure, so no source could be resolved. The host ends the unit's + // run successfully without executing it, as it does for an ignored source fetch. The remaining fields + // describe the unit as written. + bool run_ignored = 23; + // Name of the ignore policy that matched. Set with run_ignored. + string ignored_policy = 24; + // Message of the ignore policy that matched. Set with run_ignored. + string ignored_message = 25; } -// ComponentErrorsConfiguration mirrors the Terragrunt errors block. Reserved; not implemented. +// ComponentErrorsConfiguration mirrors the Terragrunt errors block. message ComponentErrorsConfiguration { repeated ComponentRetryPolicy retry = 1; repeated ComponentIgnorePolicy ignore = 2; @@ -407,7 +429,7 @@ message ComponentIgnorePolicy { // Extra signals written for the caller, keyed by name. map signals = 4; } -// ComponentExclude mirrors the Terragrunt exclude block. Reserved; not implemented. +// ComponentExclude mirrors the Terragrunt exclude block. message ComponentExclude { // Evaluated "if" condition. bool condition = 1; @@ -415,7 +437,7 @@ message ComponentExclude { repeated string actions = 2; bool exclude_dependencies = 3; } -// ComponentIam mirrors the Terragrunt IAM settings. Reserved; not implemented. +// ComponentIam mirrors the Terragrunt IAM settings. // The web identity token is a secret and must be carried as a sensitive value. message ComponentIam { string role = 1; diff --git a/sdk/README.md b/sdk/README.md index 8ec26da0adb..29f6fa1f95b 100644 --- a/sdk/README.md +++ b/sdk/README.md @@ -18,7 +18,7 @@ Official monorepo packages under `../plugins` are an exception to external SDK v ## Component configuration contracts (in development) -The planned `plugin.Services` extension will accept optional `Discovery`, `Validation`, `Preparation`, and `State` handlers. Negotiate these separately as `component.discovery`, `component.validation`, `component.preparation`, and `component.state`, each at version 1. A configuration provider does not need to supply an executor. Matching host clients and an explicit package loader with private configuration-provider registries are planned. CLI stack and execution integration are not available yet. +`plugin.Services` accepts optional `Discovery`, `Validation`, `Preparation`, and `State` handlers. Negotiate these separately as `component.discovery`, `component.validation`, `component.preparation`, and `component.state`. Preparation uses version 2, which carries execution metadata, Terraform arguments, and backend ownership. The other services use version 1. A configuration provider does not need to supply an executor. The host runtime has matching clients and an explicit package loader with private configuration-provider registries. Stack processing and Terraform execution call the plugin configuration runtime for components that declare a `configuration` settings section, and that call reaches the provider `Prepare` RPC. The planned `component.Encode` and `component.Decode` helpers will handle infrastructure values; the component package is not yet available. They retain exact cty numbers, types, nulls, unknowns, and nested sensitivity. Translate evaluator-specific sensitivity to `component.Sensitive` before encoding. Other marks are rejected. The payload and its sensitivity paths can contain secrets; never log them. Invalid wire values produce a payload-free `component.ErrInvalidValue`. diff --git a/sdk/proto/atmos/plugin/v1/plugin.pb.go b/sdk/proto/atmos/plugin/v1/plugin.pb.go index 57f75fe9c17..14d6af372ae 100644 --- a/sdk/proto/atmos/plugin/v1/plugin.pb.go +++ b/sdk/proto/atmos/plugin/v1/plugin.pb.go @@ -3380,23 +3380,43 @@ type PrepareComponentResponse struct { // Maps each input name to the authored file, line, and column that produced // its value, for provenance reporting. Missing entries mean unknown origin. InputOrigins map[string]*SourceLocation `protobuf:"bytes,12,rep,name=input_origins,json=inputOrigins,proto3" json:"input_origins,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"` - // Fields 13-18 are reserved for Terragrunt parity. The host does not read - // them yet; a plugin that cannot honor a setting must reject it with an - // ErrorDetail instead of returning it, and must leave these fields unset. - // Engine executable selected by terraform_binary. + // Fields 13-18 carry Terragrunt settings that decide how the host runs the unit. The host reads + // terraform_binary, errors, exclude, iam, and feature_flags (13-16 and 18). It does not read + // encryption (17) yet. A plugin that cannot honor a setting must reject it with an ErrorDetail + // instead of returning a value the host would silently ignore. + // Engine executable selected by terraform_binary. Empty means the engine of the component. TerraformBinary string `protobuf:"bytes,13,opt,name=terraform_binary,json=terraformBinary,proto3" json:"terraform_binary,omitempty"` - // errors block: retry and ignore policies. + // errors block: retry and ignore policies. The host applies them around the engine subprocess. Errors *ComponentErrorsConfiguration `protobuf:"bytes,14,opt,name=errors,proto3" json:"errors,omitempty"` - // exclude block: conditional unit exclusion. + // exclude block, set only when its condition holds and it applies to the current operation. + // The host skips the operation for the component and runs nothing. See ComponentExclude. Exclude *ComponentExclude `protobuf:"bytes,15,opt,name=exclude,proto3" json:"exclude,omitempty"` // iam_role, iam_assume_role_session_name, iam_assume_role_duration, iam_web_identity_token. + // The host assumes the role, never the plugin. Iam *ComponentIam `protobuf:"bytes,16,opt,name=iam,proto3" json:"iam,omitempty"` - // encryption block as opaque, provider-defined data. + // encryption block as opaque, provider-defined data. The host ignores it. Encryption *ComponentValue `protobuf:"bytes,17,opt,name=encryption,proto3" json:"encryption,omitempty"` - // feature flag name to its evaluated value rendered as a string. - FeatureFlags map[string]string `protobuf:"bytes,18,rep,name=feature_flags,json=featureFlags,proto3" json:"feature_flags,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"` - unknownFields protoimpl.UnknownFields - sizeCache protoimpl.SizeCache + // Feature flag name to its evaluated value rendered as a string. The plugin has already applied + // the flags while evaluating the unit, so they are reflected in every other field. The host + // keeps them with the prepared configuration. + FeatureFlags map[string]string `protobuf:"bytes,18,rep,name=feature_flags,json=featureFlags,proto3" json:"feature_flags,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"` + // Authorization for an archive returned by a module registry. This value is + // sensitive and may be sent only to module_source_auth_origin. + ModuleSourceAuthorization *ComponentValue `protobuf:"bytes,19,opt,name=module_source_authorization,json=moduleSourceAuthorization,proto3" json:"module_source_authorization,omitempty"` + ModuleSourceAuthOrigin string `protobuf:"bytes,20,opt,name=module_source_auth_origin,json=moduleSourceAuthOrigin,proto3" json:"module_source_auth_origin,omitempty"` + // Time spent resolving a tfr:// source, including its bounded retries. + RegistryResolutionNanos uint64 `protobuf:"varint,21,opt,name=registry_resolution_nanos,json=registryResolutionNanos,proto3" json:"registry_resolution_nanos,omitempty"` + // True when an errors-block ignore policy matched a failure the plugin hit while preparing the unit, + // for example a registry resolution failure, so no source could be resolved. The host ends the unit's + // run successfully without executing it, as it does for an ignored source fetch. The remaining fields + // describe the unit as written. + RunIgnored bool `protobuf:"varint,23,opt,name=run_ignored,json=runIgnored,proto3" json:"run_ignored,omitempty"` + // Name of the ignore policy that matched. Set with run_ignored. + IgnoredPolicy string `protobuf:"bytes,24,opt,name=ignored_policy,json=ignoredPolicy,proto3" json:"ignored_policy,omitempty"` + // Message of the ignore policy that matched. Set with run_ignored. + IgnoredMessage string `protobuf:"bytes,25,opt,name=ignored_message,json=ignoredMessage,proto3" json:"ignored_message,omitempty"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache } func (x *PrepareComponentResponse) Reset() { @@ -3555,7 +3575,49 @@ func (x *PrepareComponentResponse) GetFeatureFlags() map[string]string { return nil } -// ComponentErrorsConfiguration mirrors the Terragrunt errors block. Reserved; not implemented. +func (x *PrepareComponentResponse) GetModuleSourceAuthorization() *ComponentValue { + if x != nil { + return x.ModuleSourceAuthorization + } + return nil +} + +func (x *PrepareComponentResponse) GetModuleSourceAuthOrigin() string { + if x != nil { + return x.ModuleSourceAuthOrigin + } + return "" +} + +func (x *PrepareComponentResponse) GetRegistryResolutionNanos() uint64 { + if x != nil { + return x.RegistryResolutionNanos + } + return 0 +} + +func (x *PrepareComponentResponse) GetRunIgnored() bool { + if x != nil { + return x.RunIgnored + } + return false +} + +func (x *PrepareComponentResponse) GetIgnoredPolicy() string { + if x != nil { + return x.IgnoredPolicy + } + return "" +} + +func (x *PrepareComponentResponse) GetIgnoredMessage() string { + if x != nil { + return x.IgnoredMessage + } + return "" +} + +// ComponentErrorsConfiguration mirrors the Terragrunt errors block. type ComponentErrorsConfiguration struct { state protoimpl.MessageState `protogen:"open.v1"` Retry []*ComponentRetryPolicy `protobuf:"bytes,1,rep,name=retry,proto3" json:"retry,omitempty"` @@ -3747,7 +3809,7 @@ func (x *ComponentIgnorePolicy) GetSignals() map[string]*ComponentValue { return nil } -// ComponentExclude mirrors the Terragrunt exclude block. Reserved; not implemented. +// ComponentExclude mirrors the Terragrunt exclude block. type ComponentExclude struct { state protoimpl.MessageState `protogen:"open.v1"` // Evaluated "if" condition. @@ -3810,7 +3872,7 @@ func (x *ComponentExclude) GetExcludeDependencies() bool { return false } -// ComponentIam mirrors the Terragrunt IAM settings. Reserved; not implemented. +// ComponentIam mirrors the Terragrunt IAM settings. // The web identity token is a secret and must be carried as a sensitive value. type ComponentIam struct { state protoimpl.MessageState `protogen:"open.v1"` @@ -4223,8 +4285,7 @@ const file_atmos_plugin_v1_plugin_proto_rawDesc = "" + "\frun_on_error\x18\x05 \x01(\bR\n" + "runOnError\x12'\n" + "\x0fsuppress_stdout\x18\x06 \x01(\bR\x0esuppressStdout\x12\x1b\n" + - "\ton_errors\x18\a \x03(\tR\bonErrors\"\x85\n" + - "\n" + + "\ton_errors\x18\a \x03(\tR\bonErrors\"\xce\f\n" + "\x18PrepareComponentResponse\x12\x16\n" + "\x06engine\x18\x01 \x01(\tR\x06engine\x12+\n" + "\x11working_directory\x18\x02 \x01(\tR\x10workingDirectory\x12#\n" + @@ -4246,7 +4307,14 @@ const file_atmos_plugin_v1_plugin_proto_rawDesc = "" + "\n" + "encryption\x18\x11 \x01(\v2\x1f.atmos.plugin.v1.ComponentValueR\n" + "encryption\x12`\n" + - "\rfeature_flags\x18\x12 \x03(\v2;.atmos.plugin.v1.PrepareComponentResponse.FeatureFlagsEntryR\ffeatureFlags\x1a`\n" + + "\rfeature_flags\x18\x12 \x03(\v2;.atmos.plugin.v1.PrepareComponentResponse.FeatureFlagsEntryR\ffeatureFlags\x12_\n" + + "\x1bmodule_source_authorization\x18\x13 \x01(\v2\x1f.atmos.plugin.v1.ComponentValueR\x19moduleSourceAuthorization\x129\n" + + "\x19module_source_auth_origin\x18\x14 \x01(\tR\x16moduleSourceAuthOrigin\x12:\n" + + "\x19registry_resolution_nanos\x18\x15 \x01(\x04R\x17registryResolutionNanos\x12\x1f\n" + + "\vrun_ignored\x18\x17 \x01(\bR\n" + + "runIgnored\x12%\n" + + "\x0eignored_policy\x18\x18 \x01(\tR\rignoredPolicy\x12'\n" + + "\x0fignored_message\x18\x19 \x01(\tR\x0eignoredMessage\x1a`\n" + "\x11InputOriginsEntry\x12\x10\n" + "\x03key\x18\x01 \x01(\tR\x03key\x125\n" + "\x05value\x18\x02 \x01(\v2\x1f.atmos.plugin.v1.SourceLocationR\x05value:\x028\x01\x1a?\n" + @@ -4484,61 +4552,62 @@ var file_atmos_plugin_v1_plugin_proto_depIdxs = []int32{ 53, // 77: atmos.plugin.v1.PrepareComponentResponse.iam:type_name -> atmos.plugin.v1.ComponentIam 31, // 78: atmos.plugin.v1.PrepareComponentResponse.encryption:type_name -> atmos.plugin.v1.ComponentValue 64, // 79: atmos.plugin.v1.PrepareComponentResponse.feature_flags:type_name -> atmos.plugin.v1.PrepareComponentResponse.FeatureFlagsEntry - 50, // 80: atmos.plugin.v1.ComponentErrorsConfiguration.retry:type_name -> atmos.plugin.v1.ComponentRetryPolicy - 51, // 81: atmos.plugin.v1.ComponentErrorsConfiguration.ignore:type_name -> atmos.plugin.v1.ComponentIgnorePolicy - 65, // 82: atmos.plugin.v1.ComponentIgnorePolicy.signals:type_name -> atmos.plugin.v1.ComponentIgnorePolicy.SignalsEntry - 31, // 83: atmos.plugin.v1.ComponentIam.web_identity_token:type_name -> atmos.plugin.v1.ComponentValue - 43, // 84: atmos.plugin.v1.ResolveComponentStateResponse.state:type_name -> atmos.plugin.v1.ComponentStateIdentity - 35, // 85: atmos.plugin.v1.ResolveComponentStateResponse.sources:type_name -> atmos.plugin.v1.ComponentSource - 17, // 86: atmos.plugin.v1.ResolveComponentStateResponse.diagnostics:type_name -> atmos.plugin.v1.Diagnostic - 31, // 87: atmos.plugin.v1.ComponentOperation.OptionsEntry.value:type_name -> atmos.plugin.v1.ComponentValue - 15, // 88: atmos.plugin.v1.PrepareComponentResponse.InputOriginsEntry.value:type_name -> atmos.plugin.v1.SourceLocation - 31, // 89: atmos.plugin.v1.ComponentIgnorePolicy.SignalsEntry.value:type_name -> atmos.plugin.v1.ComponentValue - 7, // 90: atmos.plugin.v1.Info.Describe:input_type -> atmos.plugin.v1.DescribeRequest - 9, // 91: atmos.plugin.v1.Info.Configure:input_type -> atmos.plugin.v1.ConfigureRequest - 10, // 92: atmos.plugin.v1.AuthProvider.Validate:input_type -> atmos.plugin.v1.ProviderRequest - 10, // 93: atmos.plugin.v1.AuthProvider.Authenticate:input_type -> atmos.plugin.v1.ProviderRequest - 20, // 94: atmos.plugin.v1.AuthProvider.Refresh:input_type -> atmos.plugin.v1.RefreshRequest - 10, // 95: atmos.plugin.v1.AuthProvider.Logout:input_type -> atmos.plugin.v1.ProviderRequest - 10, // 96: atmos.plugin.v1.AuthProvider.Environment:input_type -> atmos.plugin.v1.ProviderRequest - 10, // 97: atmos.plugin.v1.AuthProvider.Paths:input_type -> atmos.plugin.v1.ProviderRequest - 23, // 98: atmos.plugin.v1.AuthProvider.PrepareEnvironment:input_type -> atmos.plugin.v1.PrepareEnvironmentRequest - 11, // 99: atmos.plugin.v1.AuthIdentity.Validate:input_type -> atmos.plugin.v1.IdentityRequest - 26, // 100: atmos.plugin.v1.AuthIdentity.Authenticate:input_type -> atmos.plugin.v1.IdentityAuthenticateRequest - 11, // 101: atmos.plugin.v1.AuthIdentity.Logout:input_type -> atmos.plugin.v1.IdentityRequest - 11, // 102: atmos.plugin.v1.AuthIdentity.Environment:input_type -> atmos.plugin.v1.IdentityRequest - 11, // 103: atmos.plugin.v1.AuthIdentity.Paths:input_type -> atmos.plugin.v1.IdentityRequest - 27, // 104: atmos.plugin.v1.AuthIdentity.PrepareEnvironment:input_type -> atmos.plugin.v1.IdentityPrepareRequest - 28, // 105: atmos.plugin.v1.Commands.Run:input_type -> atmos.plugin.v1.CommandRequest - 40, // 106: atmos.plugin.v1.ComponentDiscovery.Discover:input_type -> atmos.plugin.v1.DiscoverComponentsRequest - 42, // 107: atmos.plugin.v1.ComponentValidation.Validate:input_type -> atmos.plugin.v1.ComponentRequest - 42, // 108: atmos.plugin.v1.ComponentPreparation.Prepare:input_type -> atmos.plugin.v1.ComponentRequest - 42, // 109: atmos.plugin.v1.ComponentState.Resolve:input_type -> atmos.plugin.v1.ComponentRequest - 8, // 110: atmos.plugin.v1.Info.Describe:output_type -> atmos.plugin.v1.DescribeResponse - 68, // 111: atmos.plugin.v1.Info.Configure:output_type -> google.protobuf.Empty - 18, // 112: atmos.plugin.v1.AuthProvider.Validate:output_type -> atmos.plugin.v1.ValidateResponse - 14, // 113: atmos.plugin.v1.AuthProvider.Authenticate:output_type -> atmos.plugin.v1.AuthenticateEvent - 21, // 114: atmos.plugin.v1.AuthProvider.Refresh:output_type -> atmos.plugin.v1.RefreshResponse - 68, // 115: atmos.plugin.v1.AuthProvider.Logout:output_type -> google.protobuf.Empty - 22, // 116: atmos.plugin.v1.AuthProvider.Environment:output_type -> atmos.plugin.v1.EnvironmentResponse - 25, // 117: atmos.plugin.v1.AuthProvider.Paths:output_type -> atmos.plugin.v1.PathsResponse - 22, // 118: atmos.plugin.v1.AuthProvider.PrepareEnvironment:output_type -> atmos.plugin.v1.EnvironmentResponse - 18, // 119: atmos.plugin.v1.AuthIdentity.Validate:output_type -> atmos.plugin.v1.ValidateResponse - 14, // 120: atmos.plugin.v1.AuthIdentity.Authenticate:output_type -> atmos.plugin.v1.AuthenticateEvent - 68, // 121: atmos.plugin.v1.AuthIdentity.Logout:output_type -> google.protobuf.Empty - 22, // 122: atmos.plugin.v1.AuthIdentity.Environment:output_type -> atmos.plugin.v1.EnvironmentResponse - 25, // 123: atmos.plugin.v1.AuthIdentity.Paths:output_type -> atmos.plugin.v1.PathsResponse - 22, // 124: atmos.plugin.v1.AuthIdentity.PrepareEnvironment:output_type -> atmos.plugin.v1.EnvironmentResponse - 30, // 125: atmos.plugin.v1.Commands.Run:output_type -> atmos.plugin.v1.CommandEvent - 41, // 126: atmos.plugin.v1.ComponentDiscovery.Discover:output_type -> atmos.plugin.v1.DiscoverComponentsResponse - 18, // 127: atmos.plugin.v1.ComponentValidation.Validate:output_type -> atmos.plugin.v1.ValidateResponse - 48, // 128: atmos.plugin.v1.ComponentPreparation.Prepare:output_type -> atmos.plugin.v1.PrepareComponentResponse - 54, // 129: atmos.plugin.v1.ComponentState.Resolve:output_type -> atmos.plugin.v1.ResolveComponentStateResponse - 110, // [110:130] is the sub-list for method output_type - 90, // [90:110] is the sub-list for method input_type - 90, // [90:90] is the sub-list for extension type_name - 90, // [90:90] is the sub-list for extension extendee - 0, // [0:90] is the sub-list for field type_name + 31, // 80: atmos.plugin.v1.PrepareComponentResponse.module_source_authorization:type_name -> atmos.plugin.v1.ComponentValue + 50, // 81: atmos.plugin.v1.ComponentErrorsConfiguration.retry:type_name -> atmos.plugin.v1.ComponentRetryPolicy + 51, // 82: atmos.plugin.v1.ComponentErrorsConfiguration.ignore:type_name -> atmos.plugin.v1.ComponentIgnorePolicy + 65, // 83: atmos.plugin.v1.ComponentIgnorePolicy.signals:type_name -> atmos.plugin.v1.ComponentIgnorePolicy.SignalsEntry + 31, // 84: atmos.plugin.v1.ComponentIam.web_identity_token:type_name -> atmos.plugin.v1.ComponentValue + 43, // 85: atmos.plugin.v1.ResolveComponentStateResponse.state:type_name -> atmos.plugin.v1.ComponentStateIdentity + 35, // 86: atmos.plugin.v1.ResolveComponentStateResponse.sources:type_name -> atmos.plugin.v1.ComponentSource + 17, // 87: atmos.plugin.v1.ResolveComponentStateResponse.diagnostics:type_name -> atmos.plugin.v1.Diagnostic + 31, // 88: atmos.plugin.v1.ComponentOperation.OptionsEntry.value:type_name -> atmos.plugin.v1.ComponentValue + 15, // 89: atmos.plugin.v1.PrepareComponentResponse.InputOriginsEntry.value:type_name -> atmos.plugin.v1.SourceLocation + 31, // 90: atmos.plugin.v1.ComponentIgnorePolicy.SignalsEntry.value:type_name -> atmos.plugin.v1.ComponentValue + 7, // 91: atmos.plugin.v1.Info.Describe:input_type -> atmos.plugin.v1.DescribeRequest + 9, // 92: atmos.plugin.v1.Info.Configure:input_type -> atmos.plugin.v1.ConfigureRequest + 10, // 93: atmos.plugin.v1.AuthProvider.Validate:input_type -> atmos.plugin.v1.ProviderRequest + 10, // 94: atmos.plugin.v1.AuthProvider.Authenticate:input_type -> atmos.plugin.v1.ProviderRequest + 20, // 95: atmos.plugin.v1.AuthProvider.Refresh:input_type -> atmos.plugin.v1.RefreshRequest + 10, // 96: atmos.plugin.v1.AuthProvider.Logout:input_type -> atmos.plugin.v1.ProviderRequest + 10, // 97: atmos.plugin.v1.AuthProvider.Environment:input_type -> atmos.plugin.v1.ProviderRequest + 10, // 98: atmos.plugin.v1.AuthProvider.Paths:input_type -> atmos.plugin.v1.ProviderRequest + 23, // 99: atmos.plugin.v1.AuthProvider.PrepareEnvironment:input_type -> atmos.plugin.v1.PrepareEnvironmentRequest + 11, // 100: atmos.plugin.v1.AuthIdentity.Validate:input_type -> atmos.plugin.v1.IdentityRequest + 26, // 101: atmos.plugin.v1.AuthIdentity.Authenticate:input_type -> atmos.plugin.v1.IdentityAuthenticateRequest + 11, // 102: atmos.plugin.v1.AuthIdentity.Logout:input_type -> atmos.plugin.v1.IdentityRequest + 11, // 103: atmos.plugin.v1.AuthIdentity.Environment:input_type -> atmos.plugin.v1.IdentityRequest + 11, // 104: atmos.plugin.v1.AuthIdentity.Paths:input_type -> atmos.plugin.v1.IdentityRequest + 27, // 105: atmos.plugin.v1.AuthIdentity.PrepareEnvironment:input_type -> atmos.plugin.v1.IdentityPrepareRequest + 28, // 106: atmos.plugin.v1.Commands.Run:input_type -> atmos.plugin.v1.CommandRequest + 40, // 107: atmos.plugin.v1.ComponentDiscovery.Discover:input_type -> atmos.plugin.v1.DiscoverComponentsRequest + 42, // 108: atmos.plugin.v1.ComponentValidation.Validate:input_type -> atmos.plugin.v1.ComponentRequest + 42, // 109: atmos.plugin.v1.ComponentPreparation.Prepare:input_type -> atmos.plugin.v1.ComponentRequest + 42, // 110: atmos.plugin.v1.ComponentState.Resolve:input_type -> atmos.plugin.v1.ComponentRequest + 8, // 111: atmos.plugin.v1.Info.Describe:output_type -> atmos.plugin.v1.DescribeResponse + 68, // 112: atmos.plugin.v1.Info.Configure:output_type -> google.protobuf.Empty + 18, // 113: atmos.plugin.v1.AuthProvider.Validate:output_type -> atmos.plugin.v1.ValidateResponse + 14, // 114: atmos.plugin.v1.AuthProvider.Authenticate:output_type -> atmos.plugin.v1.AuthenticateEvent + 21, // 115: atmos.plugin.v1.AuthProvider.Refresh:output_type -> atmos.plugin.v1.RefreshResponse + 68, // 116: atmos.plugin.v1.AuthProvider.Logout:output_type -> google.protobuf.Empty + 22, // 117: atmos.plugin.v1.AuthProvider.Environment:output_type -> atmos.plugin.v1.EnvironmentResponse + 25, // 118: atmos.plugin.v1.AuthProvider.Paths:output_type -> atmos.plugin.v1.PathsResponse + 22, // 119: atmos.plugin.v1.AuthProvider.PrepareEnvironment:output_type -> atmos.plugin.v1.EnvironmentResponse + 18, // 120: atmos.plugin.v1.AuthIdentity.Validate:output_type -> atmos.plugin.v1.ValidateResponse + 14, // 121: atmos.plugin.v1.AuthIdentity.Authenticate:output_type -> atmos.plugin.v1.AuthenticateEvent + 68, // 122: atmos.plugin.v1.AuthIdentity.Logout:output_type -> google.protobuf.Empty + 22, // 123: atmos.plugin.v1.AuthIdentity.Environment:output_type -> atmos.plugin.v1.EnvironmentResponse + 25, // 124: atmos.plugin.v1.AuthIdentity.Paths:output_type -> atmos.plugin.v1.PathsResponse + 22, // 125: atmos.plugin.v1.AuthIdentity.PrepareEnvironment:output_type -> atmos.plugin.v1.EnvironmentResponse + 30, // 126: atmos.plugin.v1.Commands.Run:output_type -> atmos.plugin.v1.CommandEvent + 41, // 127: atmos.plugin.v1.ComponentDiscovery.Discover:output_type -> atmos.plugin.v1.DiscoverComponentsResponse + 18, // 128: atmos.plugin.v1.ComponentValidation.Validate:output_type -> atmos.plugin.v1.ValidateResponse + 48, // 129: atmos.plugin.v1.ComponentPreparation.Prepare:output_type -> atmos.plugin.v1.PrepareComponentResponse + 54, // 130: atmos.plugin.v1.ComponentState.Resolve:output_type -> atmos.plugin.v1.ResolveComponentStateResponse + 111, // [111:131] is the sub-list for method output_type + 91, // [91:111] is the sub-list for method input_type + 91, // [91:91] is the sub-list for extension type_name + 91, // [91:91] is the sub-list for extension extendee + 0, // [0:91] is the sub-list for field type_name } func init() { file_atmos_plugin_v1_plugin_proto_init() } diff --git a/tests/fixtures/schemas/atmos/atmos-manifest/1.0/atmos-manifest.json b/tests/fixtures/schemas/atmos/atmos-manifest/1.0/atmos-manifest.json index d7c334b44c9..05d25b95c05 100644 --- a/tests/fixtures/schemas/atmos/atmos-manifest/1.0/atmos-manifest.json +++ b/tests/fixtures/schemas/atmos/atmos-manifest/1.0/atmos-manifest.json @@ -1575,6 +1575,10 @@ } ] }, + "source_subdir": { + "type": "string", + "description": "Module subdirectory within the copied source root. Sibling modules remain available in the workdir." + }, "terraform_workspace": { "type": "string", "description": "Terraform workspace" diff --git a/website/docs/migration/terragrunt.mdx b/website/docs/migration/terragrunt.mdx index 2b387288127..569e8ec915c 100644 --- a/website/docs/migration/terragrunt.mdx +++ b/website/docs/migration/terragrunt.mdx @@ -1052,17 +1052,33 @@ Atmos follows the same rules as Terragrunt v1.2.1, whose symlink support is an o - A unit without a stable `remote_state` path keeps local state under `.terragrunt-cache`, which moves with the checkout path and disappears when the cache is deleted. Atmos keeps that location so existing state stays addressable, and warns when it applies. - `!terraform.output` and `!terraform.state` can read a Terragrunt unit like any other component. +### Executables, errors, exclusions, and features + +- `terraform_binary` in a unit selects the executable for that unit. Atmos resolves a name such as `tofu` through the toolchain and uses a path as given, resolved from the unit directory when relative. A stack-level `command` overrides it, and it overrides `components.terraform.command`. The executable must be Terraform or OpenTofu and cannot switch the unit between the two engines. +- A unit's `errors` block applies to its `init` and main Terraform commands and to fetching its source. Ignore policies are checked before retry policies, and the first matching policy decides. `max_attempts` includes the first attempt. An ignored failure succeeds and writes its `signals` to `error-signals.json` in the module directory. When both `settings.retry` and an `errors` retry policy apply, `settings.retry` retries first and the policy retries what is left, so a failure can be attempted more often than either setting alone. +- A unit whose `exclude` block lists the running command (`all` and `all_except_output` included) is skipped in `--all`, `--affected`, and `--stack` runs and reported by name. `exclude_dependencies = true` also skips its direct dependencies. With `no_run` unset, a direct run is skipped only for an exact action. `no_run = true` also honors `all` and `all_except_output`, and `no_run = false` never skips a direct run. +- Set `components.terraform.terragrunt.features` to override `feature` blocks for every unit. `TG_FEATURE=name=value,name2=value2` overrides those values per name. An override that no unit declares prints a warning naming it. +- `terragrunt-read-config` (also `read-config`) after-hooks run after every command prepares the unit, in the unit directory. `init-from-module` hooks run around fetching the module source, only when it is fetched or refreshed. +- A hook that executes `tflint` runs `tflint --init` and then `tflint` with the unit's `.tflint.hcl`, inputs, and variable files. `TF_VAR_` variables from `env_vars` reach TFLint through the environment, so `terraform.tfvars` takes precedence over them, as it does for Terraform. Install `tflint` through the toolchain or `PATH`. + +### Identity, state encryption, and registry sources + +- `iam_role`, `iam_assume_role_duration`, `iam_assume_role_session_name`, and `iam_web_identity_token` are supported. Atmos assumes the role in process through STS, on top of the credentials in use: the default Atmos identity or the ambient credential chain. It downloads an `s3::` source with the temporary session, exports the session to Terraform, hooks, and workspace commands, and never writes it to a file or a saved plan. `iam_web_identity_token` is a token or the path of a file that contains one. Use a file if you apply saved plans, because a literal token is never stored with a plan. +- If you select an identity with `--identity` or `ATMOS_IDENTITY`, that identity decides who the component runs as, and `iam_role` is not assumed. `TG_IAM_ASSUME_ROLE`, `TG_IAM_ASSUME_ROLE_DURATION`, `TG_IAM_ASSUME_ROLE_SESSION_NAME`, and `TG_IAM_ASSUME_ROLE_WEB_IDENTITY_TOKEN` override the unit's settings, as they do in Terragrunt. `run_cmd` and `get_env` evaluate with the invocation's own environment, before the role is assumed, as in Terragrunt. +- `remote_state.encryption` is written to the file that `remote_state.generate` creates, as an OpenTofu `encryption` block, so it requires OpenTofu. Any OpenTofu key provider works, for example `pbkdf2`, `aws_kms`, or `gcp_kms`. Without `generate` it has no effect, as in Terragrunt, and Atmos warns. `remote_state.disable_init` is accepted: Terragrunt uses it only to skip creating backend resources, Atmos never creates them, and OpenTofu initializes the backend as usual. +- `tfr://` module sources are resolved when a unit is prepared, using the Terraform module registry protocol. `?version=` pins an exact version, the `terraform { version }` constraint selects among published versions, and with neither the latest stable release is used. Private registries authenticate with `TF_TOKEN_`, a `credentials` block in the CLI configuration, or `TG_TF_REGISTRY_TOKEN`. A `//subdir` selects a directory inside the module. + +### What it does not support yet + ### What it does not support yet Unsupported settings fail before any infrastructure command runs, with an error that names the unit, the file with line and column, and the setting. With automatic discovery, a broken unit does not affect other units or YAML components: it stays listed, and only commands that target it fail. An explicit import is stricter: it fails when any unit it selects reports an error, so that import contributes no units until the error is fixed. `apply`, `plan`, and `destroy` with `--all` validate every selected unit first and report "no infrastructure command was run". Evaluation errors can contain secrets, so Atmos shows the location and a hint but not the evaluator's text; run with `ATMOS_LOGS_LEVEL=Debug` to log the full message with secrets masked. Atmos does not fall back to the Terragrunt executable. Currently rejected: -- `terraform_binary`, `errors`, `exclude`, feature flags, and the `iam_role` and web-identity settings. -- `remote_state` with `disable_init` or encryption. Atmos does not create backend resources. -- Inline modules without `terraform.source`, and registry module sources. -- Hooks for the `read-config`, `terragrunt-read-config`, and `init-from-module` lifecycles, and hooks that run a built-in integration such as `tflint`. +- Inline modules without `terraform.source`. - Output dependencies in a non-default workspace during discovery. +- Stack-level overrides of a unit's `backend`, `backend_type`, `metadata.component`, or `source.uri`, because they would change state identity. Edit the unit's `remote_state` instead. -Discovery runs without side effects, so a unit whose `locals` call `run_cmd` cannot be discovered. When a unit uses one of these features, keep it on Terragrunt for now or convert it with the steps below. +Terragrunt's built-in default retry list for transient errors, the `--no-auto-retry` flag, and `errors` handling around `read-config` hooks are not implemented; use `settings.retry` or an `errors` block. Discovery runs without side effects, so a unit whose `locals` call `run_cmd` cannot be discovered. ## Migration Steps