Repository navigation
feat(plugin): add native Terragrunt discovery and execution #3571
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
Erik Osterman (Cloud Posse) (osterman)
wants to merge
66
commits into
test/terraform-lifecycle-field-fixtures
from
feat/component-plugin-contracts
+17,826
−789
Open
Changes from all commits
Commits
Show all changes
66 commits
Select commit
Hold shift + click to select a range
f7b1515
feat(plugin): add component configuration protocol and SDK
osterman dcd46e5
feat(plugin): load and adapt component configuration providers
osterman b26110a
[autocommit] formatting fixes
atmos-pro[bot] 98c54e2
feat(plugin): wire native Terragrunt discovery and execution
osterman 1ddc34e
feat(terragrunt): add revision-aware affected selection and Starlark …
osterman f5ab663
test(describe): refresh describe snapshots for the merged Terraform c…
osterman b0bbf3a
fix(terragrunt): repair CI for the plugin contracts layer
osterman 2fc4697
chore(deps): tidy go.sum after rebasing onto main
osterman 4238aad
fix(validate): validate discovered inventories without a YAML stack d…
osterman 673fd60
ci(plugins): fail when the stacked release base cannot be resolved
osterman 4dcd0f0
fix(plugin): keep declared dependency settings on discovered components
osterman df30cc4
fix(schema): reserve empty built-in configuration keys
osterman 100554e
test(terragrunt): name each affected scenario and keep its debug stderr
osterman 51acb96
fix(affected): match provider watches across root spellings
osterman 6aa32fd
fix(plugin): keep discovered numbers numeric in describe output
osterman 9cc202d
fix(plugin): keep one caller's cancellation from failing shared disco…
osterman 2907ae0
fix(terragrunt): keep validation status codes from Prepare and Resolve
osterman 73ddc3a
fix(terraform): keep provider artifact names distinct without a CLI c…
osterman 4f6b3b7
test(plugin): assert an explicit empty environment override stays set
osterman 35ff969
fix(affected): pass the command context to configuration initialization
osterman c4b070e
fix(plugin): treat settings.configuration false as an ordinary component
osterman d9fffd6
fix(terraform): reject a TF_WORKSPACE that conflicts with the prepare…
osterman f767975
fix(auth): include discovered units in scoped auth validation
osterman 8d95fc8
fix(terraform): keep generated files away from state and workdir runt…
osterman 66e627a
fix(validate): check discovered definitions against the manifest schema
osterman 16b9f22
Revert "fix(auth): include discovered units in scoped auth validation"
osterman 81f5dce
Merge remote-tracking branch 'origin/test/terraform-lifecycle-field-f…
osterman b23e0e2
fix(auth): include discovered units in scoped auth validation
osterman 851ed78
test(terraform): verify query initializer cancellation
osterman 596408f
fix(terragrunt): keep backend resolution read-only
osterman 52023ef
test(plugin): cover preparation deadline
osterman f630689
test(terragrunt): assert discovered fixture dependencies
osterman 7fd032b
fix(affected): scope resolved paths to changed file index
osterman c0923d3
test(plugin): assert masking for sensitive paths
osterman 7d65e46
fix(terragrunt): serialize stack generation and preparation
osterman 5dc5f8c
fix(terraform): retain structured input sensitivity
osterman e5a39f1
fix(terragrunt): preserve secret-safe discovery diagnostics
osterman 7830d96
fix: check published plugin versions before merge
osterman 614a50e
fix: revalidate preparation dependencies and restore link checks
osterman fa5dd7d
fix(terraform): reject state lookup for configuration-owned components
osterman 2957a9e
fix(configuration): own discovery validation and check dependency state
osterman 2a4a787
fix(terragrunt): scope stack generation and preserve unit identities
osterman ca3c8ba
test(plugin): fold component registry tests into calls tests
osterman 23be87e
Merge published L3 head a485f402bd into feat/component-plugin-contracts
osterman 8853787
fix(terragrunt): preserve native unit overlays and download settings
osterman 34a9168
Merge updated lifecycle fixtures into component plugin contracts
osterman 810430a
Merge remote-tracking branch 'origin/test/terraform-lifecycle-field-f…
osterman f31f1d3
fix(terragrunt): keep download_dir for backend reads and reject overl…
osterman 1db0b51
feat(plugin): carry file modes through the component contract
osterman 581cfd7
Merge remote-tracking branch 'origin/test/terraform-lifecycle-field-f…
osterman 22b3766
test(plugin): compare the default file mode against a umask-aware ref…
osterman 0121205
fix(terragrunt): keep copied private unit files private
osterman a21fee6
docs(terragrunt): qualify file-mode behavior on Windows
osterman 2fd707f
Merge branch 'test/terraform-lifecycle-field-fixtures' into feat/comp…
osterman 6bd287b
Merge branch 'test/terraform-lifecycle-field-fixtures' into feat/comp…
osterman 3e64413
Merge branch 'test/terraform-lifecycle-field-fixtures' into feat/comp…
osterman 41dc10f
Merge branch 'test/terraform-lifecycle-field-fixtures' into feat/comp…
osterman 3e150f3
fix(terragrunt): read download_dir without evaluating unrelated inputs
osterman 995e635
fix(terragrunt): preserve directory restrictions on copied credentials
osterman 7862d46
Merge remote-tracking branch 'origin/test/terraform-lifecycle-field-f…
osterman 69b0b1c
Merge remote-tracking branch 'origin/test/terraform-lifecycle-field-f…
osterman 2efba5f
fix(terragrunt): preserve backend download directory input references
osterman e9a2f04
Merge L3 field-fixture checkpoint into L4 plugin contracts
osterman b57bdcc
refactor(terragrunt): colocate backend input projection helpers
osterman 672ca77
Merge L3 prerequisite fixes and preserve native Stat errors in L4
osterman 3088f56
refactor(terragrunt): reconcile L4 plugin lint and integrate L3
osterman File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,76 @@ | ||
| package plugin | ||
|
|
||
| import ( | ||
| "net/http" | ||
| "os" | ||
| "path/filepath" | ||
| "testing" | ||
|
|
||
| "github.com/google/go-containerregistry/pkg/name" | ||
| v1 "github.com/google/go-containerregistry/pkg/v1" | ||
| "github.com/google/go-containerregistry/pkg/v1/remote/transport" | ||
| "github.com/spf13/viper" | ||
| "github.com/stretchr/testify/require" | ||
| "go.uber.org/mock/gomock" | ||
|
|
||
| errUtils "github.com/cloudposse/atmos/errors" | ||
| atmosoci "github.com/cloudposse/atmos/pkg/oci" | ||
| "github.com/cloudposse/atmos/pkg/plugin/extension" | ||
| "github.com/cloudposse/atmos/pkg/plugin/install" | ||
| "github.com/cloudposse/atmos/pkg/plugin/manage" | ||
| pluginoci "github.com/cloudposse/atmos/pkg/plugin/oci" | ||
| ) | ||
|
|
||
| func TestReleaseCheckPublishedVersion(t *testing.T) { | ||
| descriptors := extension.New() | ||
| require.NoError(t, descriptors.Register(extension.Descriptor{Capability: "fixture", Path: []string{"fixtures"}, Version: 1, Validate: func(extension.Binding) error { return nil }})) | ||
| parentLayout := publicReleaseLayout(t, descriptors) | ||
| index, err := atmosoci.LoadIndex(parentLayout, "") | ||
| require.NoError(t, err) | ||
| parent, err := pluginoci.Inspect(index, "") | ||
| require.NoError(t, err) | ||
| reference, err := name.NewTag("registry.example.com/example/fixture:1.0.0") | ||
| require.NoError(t, err) | ||
| missing := &transport.Error{StatusCode: http.StatusNotFound} | ||
| for _, test := range []struct { | ||
| name string | ||
| changed bool | ||
| check bool | ||
| destination string | ||
| registryErr error | ||
| wantErr error | ||
| }{ | ||
| {name: "unpublished version", changed: true, check: true, destination: "oci://registry.example.com/example/fixture", registryErr: missing}, | ||
| {name: "identical published parent", check: true, destination: "oci://registry.example.com/example/fixture"}, | ||
| {name: "child reuses published parent version", changed: true, check: true, destination: "oci://registry.example.com/example/fixture", wantErr: errUtils.ErrPluginImmutableRelease}, | ||
| {name: "registry access denied", check: true, destination: "oci://registry.example.com/example/fixture", registryErr: os.ErrPermission, wantErr: os.ErrPermission}, | ||
| {name: "registry timeout", check: true, destination: "oci://registry.example.com/example/fixture", registryErr: os.ErrDeadlineExceeded, wantErr: os.ErrDeadlineExceeded}, | ||
| {name: "offline directory", check: true}, | ||
| {name: "publication checks later", destination: "oci://registry.example.com/example/fixture"}, | ||
| } { | ||
| t.Run(test.name, func(t *testing.T) { | ||
| directory := t.TempDir() | ||
| require.NoError(t, parent.Extract("linux/amd64", directory)) | ||
| if test.changed { | ||
| binary := filepath.Join(directory, "bin", "plugin") | ||
| raw, err := os.ReadFile(binary) | ||
| require.NoError(t, err) | ||
| require.NoError(t, os.WriteFile(binary, append(raw, []byte("child change")...), 0o755)) | ||
| } | ||
| client := NewMockRegistry(gomock.NewController(t)) | ||
| if test.check && test.destination != "" { | ||
| client.EXPECT().Head(gomock.Any(), reference).Return(&v1.Descriptor{Digest: hashOf(t, parent.Digest)}, test.registryErr) | ||
| } | ||
| values := viper.New() | ||
| values.Set(releaseCheckFlag, test.check) | ||
| values.Set(publishDestinationFlag, test.destination) | ||
| backend := &releaseBackend{values: values, manager: &manage.Manager{Installer: &install.Service{Registry: client, Descriptors: descriptors}}} | ||
| err := backend.Pack(t.Context(), directory, filepath.Join(t.TempDir(), "layout")) | ||
| if test.wantErr != nil { | ||
| require.ErrorIs(t, err, test.wantErr) | ||
| } else { | ||
| require.NoError(t, err) | ||
| } | ||
| }) | ||
| } | ||
| } |
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,73 @@ | ||
| package exec | ||
|
|
||
| import ( | ||
| "context" | ||
| "fmt" | ||
| "path/filepath" | ||
|
|
||
| "github.com/santhosh-tekuri/jsonschema/v5" | ||
|
|
||
| errUtils "github.com/cloudposse/atmos/errors" | ||
| "github.com/cloudposse/atmos/pkg/component/configuration" | ||
| "github.com/cloudposse/atmos/pkg/schema" | ||
| ) | ||
|
|
||
| // Discovered definitions are literal provider inventory data, so static inspection | ||
| // (for example scoped `atmos auth validate`) includes them as well. | ||
| func init() { | ||
| mergeStaticStacks = mergeConfigurationStacks | ||
| } | ||
|
|
||
| // mergeConfigurationStacks feeds provider definitions through the ordinary stack compiler. | ||
| // The invocation-owned provider retains the caller's cancellation scope. | ||
| func mergeConfigurationStacks(config *schema.AtmosConfiguration, stacks map[string]any, raw map[string]map[string]any, contexts AllStacksDeferredContexts) error { | ||
| if config.ComponentConfigurations == nil { | ||
| return nil | ||
| } | ||
| discovered, err := config.ComponentConfigurations.Discover(context.Background()) | ||
| if err != nil { | ||
| return err | ||
| } | ||
| for name, definition := range discovered { | ||
| for file, value := range stacks { | ||
| existing, _ := value.(map[string]any) | ||
| if file == name || existing["name"] == name { | ||
| return fmt.Errorf("%w: stack %s", errUtils.ErrDuplicatePluginExtension, name) | ||
| } | ||
| } | ||
| processed, deferred, err := ProcessStackConfig(config, config.StacksBaseAbsolutePath, config.TerraformDirAbsolutePath, config.HelmfileDirAbsolutePath, config.PackerDirAbsolutePath, config.AnsibleDirAbsolutePath, filepath.Join(config.StacksBaseAbsolutePath, name+".yaml"), definition, false, true, "", map[string]map[string][]string{}, map[string]map[string]any{}, false) | ||
| if err != nil { | ||
| return err | ||
| } | ||
| stacks[name], raw[name], contexts[name] = processed, definition, deferred | ||
| } | ||
| return nil | ||
| } | ||
|
|
||
| // validateDiscoveredManifests adapts the configuration package's manifest validation | ||
| // to the stack processor's compiled-schema cache. | ||
| func validateDiscoveredManifests(config *schema.AtmosConfiguration, schemaPath string) ([]string, error) { | ||
|
coderabbitai[bot] marked this conversation as resolved.
Resolved
|
||
| if config.ComponentConfigurations == nil { | ||
| return nil, nil | ||
| } | ||
| discovered, err := config.ComponentConfigurations.Discover(context.Background()) | ||
| if err != nil { | ||
| return nil, err | ||
| } | ||
| return configuration.ValidateDiscoveredManifests(discovered, func() (*jsonschema.Schema, error) { | ||
| return compiledManifestSchema(schemaPath) | ||
| }) | ||
| } | ||
|
|
||
| // compiledManifestSchema shares the stack processor's compiled-schema cache. | ||
| func compiledManifestSchema(path string) (*jsonschema.Schema, error) { | ||
| if compiled, found := getCachedCompiledSchema(path); found { | ||
| return compiled, nil | ||
| } | ||
| compiled, err := configuration.CompileManifestSchema(path) | ||
| if err != nil { | ||
| return nil, err | ||
| } | ||
| cacheCompiledSchema(path, compiled) | ||
| return compiled, nil | ||
| } | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.