diff --git a/docs/prd/artifacts.md b/docs/prd/artifacts.md new file mode 100644 index 00000000000..d1757d22ff7 --- /dev/null +++ b/docs/prd/artifacts.md @@ -0,0 +1,599 @@ +# Atmos Artifacts: Repositories, Publishing, Discovery, and Automation + +**Status:** Proposed; configuration and command examples describe future capabilities. Documentation only. + +**Implementation:** [Work packages](./artifacts/implementation.md), [contracts](./artifacts/contracts.md), and [validation](./artifacts/validation.md). + +## Problem and outcome + +Atmos needs one way to publish, retrieve, inspect, and discover the assets its users produce and +consume: components, scaffolds, backups, verified tool packages, CI outputs, and OCI content. +Users should configure one destination and publish many named assets into it. Switching between +OCI, S3, directories, and archives should change the selected repository, not the publishing workflow. + +The same operations must work interactively, in CI, in workflows, and as lifecycle hooks. CI runs +must leave useful job summaries rather than requiring users to reconstruct publication results +from logs. Package README and CHANGELOG documents must be available without downloading a package. + +## Use-case inventory + +Artifacts are the distribution and restoration layer for the following use cases. Offline and +air-gapped operation is a cross-cutting goal: content must remain usable after its upstream is unavailable. + +| Use case | Content to store | Intended consumption or restoration | +| --- | --- | --- | +| Scaffolds and templates | Template trees, scaffold definitions, package README/CHANGELOG, and supporting files | Generate or update projects from a pinned template release | +| Components | Terraform, Helmfile, and other component source trees, scripts, and associated files | Provision components from a controlled repository | +| Terraform modules | Root or child module source, origin/version metadata, and checksums | Resolve pinned module dependencies without the original Git/registry source | +| Stack configuration | Stack manifests, catalogs, mixins, imports, and their dependencies | Load and compose stacks from published configuration | +| Atmos configuration | `atmos.yaml`, imported configuration, custom commands, workflows, policies, and supporting scripts | Bootstrap a consistent Atmos environment and reuse organizational automation | +| Toolchain packages | Verified original tool releases, platform information, lock metadata, and verification evidence | Install the required tools without contacting upstream release servers | +| Toolchain caches | Selected cached downloads and, eventually, installed tool trees with platform/version metadata | Warm or restore a compatible toolchain cache for offline execution | +| Terraform provider cache | Provider packages, origin addresses, versions, platforms, checksums, and mirror index metadata | Export Atmos-managed provider caches and restore a usable offline provider mirror | +| Terraform module cache | Cached module sources, registry resolution metadata, versions, and integrity records | Export Atmos-managed module caches and restore offline module resolution | +| OCI images and artifacts | Complete manifests, indexes, blobs, platforms, and required related artifacts | Run or consume pinned OCI content from an internal registry or local layout | +| Vendored dependencies | Git/HTTP source snapshots and dependency provenance | Hydrate the same source trees without contacting upstream sources | +| CI outputs | Planfiles, build outputs, generated configuration, reports, and run provenance | Exchange verified results between jobs and retain inspectable outputs | +| Backups | Selected files/directories and restoration metadata | Restore a copied directory or a versioned archive from local storage or S3 | +| Atmos plugins | Existing plugin release packages and associated verification material | Distribute/install existing plugin formats while preserving immutable-release policy | +| SBOMs and attestations | Inventory documents, provenance, signatures, and subject relationships | Inspect and verify artifacts and their dependencies in disconnected environments | +| Air-gap distribution | A selected closure of configuration, components, modules, providers, tools, images, and metadata | Transfer into a disconnected environment and restore everything needed for a defined workflow | + +This inventory defines the product vision; delivery phases are specified below. Cache restoration must +preserve indexes, locks, checksums, and dependency relationships and prove consumption with upstream disabled. + +## Model and V1 boundaries + +- **Repository:** A configured collection root holding many named artifacts, possibly of different + kinds. It is a destination and discovery source, not a declaration of one package. +- **Artifact:** Versioned content with portable metadata, labels, document descriptors, and digests. +- **Artifact kind:** Meaning of the content, such as `component/terraform`, `scaffold`, + `toolchain/package`, or generic `files`. It is independent of the repository kind. +- **Packaging:** Representation of a file payload: individual files or an archive. +- **Reference:** Repository alias plus artifact name and version, or name and immutable digest. +- **Catalog:** An optional maintained discovery snapshot. It is not necessary for direct reads. + +V1 includes publishing, pulling, copying, inspection, listing, search, deletion, package documents, +label filtering, catalogs, signatures/locks, hooks, and CI reporting. OCI manifests/indexes are transferred +as content graphs, not interpreted as ordinary directory archives. + +Artifacts are not components. Existing top-level `stores` remain key-value and secret stores. +An archive in this proposal packages one artifact; it is not a snapshot of an entire repository. + +## Public configuration + +`artifacts.repositories` is the single list of destinations to use and discover. Do not add another +`discovery.repositories` list or require one configuration entry per scaffold or component. + +```yaml +artifacts: + repositories: + shared: + kind: oci/registry + identity: registry-publisher + options: + registry: ghcr.io + prefix: acme/atmos + defaults: + metadata: + labels: + team: platform + + backups: + kind: aws/s3 + identity: backup-writer + packaging: tgz + options: + bucket: acme-artifacts + prefix: atmos + region: us-east-2 + tags: + purpose: backup + + local: + kind: local/dir + options: + path: .atmos/artifacts + + archives: + kind: local/archive + packaging: tgz + permissions: "0600" + options: + path: .atmos/backups + key_pattern: '{{ .CreatedAt | date "2006/01/02" }}/{{ .Name }}/{{ .Version }}.tgz' + + offline: + kind: oci/layout + options: + path: .atmos/oci +``` + +`identity` references Atmos auth identities; credentials do not belong in repository options. +`prefix` is an Atmos collection-root convention, not an ORAS namespace field. ORAS addresses a +registry, an OCI repository, and a tag or digest. Atmos maps a collection and artifact name onto +that OCI repository. Local backends use `path` for their destination root. + +### Uniform addressing + +```shell +atmos artifacts publish ./components/terraform/vpc \ + --repository shared \ + --name components/terraform/vpc \ + --version 1.2.0 \ + --kind component/terraform +``` + +Changing only `--repository` selects another destination: + +| Repository | Resolved payload location with the configuration above | +| --- | --- | +| `shared` | `ghcr.io/acme/atmos/components/terraform/vpc:1.2.0` | +| `backups` | `s3://acme-artifacts/atmos/components/terraform/vpc/1.2.0.tgz` | +| `local` | `.atmos/artifacts/components/terraform/vpc/1.2.0/` | +| `archives` | `.atmos/backups/2026/10/30/components/terraform/vpc/1.2.0.tgz` for that creation date | +| `offline` | A named reference in the OCI layout at `.atmos/oci` | + +Metadata and document sidecars accompany payloads where required; the table shows payload locations. +Names may contain relative slash-separated segments. Reject absolute paths, traversal, empty +segments, and backend-invalid names or versions before uploading; do not silently rename them. +Versions are explicit labels, including CI build identifiers, rather than universally requiring +SemVer. Kind-specific rules, including plugin release rules, still apply. + +Publishing takes a local source argument. Explicit `--name` wins; otherwise use available package +metadata, then the source basename. Explicit `--version` wins over package metadata; fail when +neither supplies a version. Do not invent `latest`. Reads use `name:version` or `name@sha256:...`. +Require `--repository` for single-repository operations; list/search may span configured repositories. +Resolve mutable references once per operation and pin subsequent reads to that digest. Digest +references select OCI manifest/index digests on OCI backends and immutable record digests on +local/S3 backends; portable content digests are reported separately. + +### Packaging, destinations, and permissions + +| Repository kind | File payload packaging | Default | +| --- | --- | --- | +| `local/dir` | `none` | `none` | +| `local/archive` | `tar`, `tgz`, `zip` | `tgz` | +| `aws/s3` | `none`, `tar`, `tgz`, `zip` | `tgz` | +| `oci/registry`, `oci/layout` | `tar`, `tgz`, `zip` for newly packaged files | `tgz` | + +With S3 `packaging: none`, publish separate objects below `///`. +With archive packaging, publish one payload object plus metadata/documents. Existing OCI content +keeps its original media types and graph; repository packaging settings do not repackage it. + +`options.path` is always the repository destination, never the publish source. Relative configured +paths resolve against the configuration file declaring them; CLI source/output paths resolve +against the invocation working directory. Workflow/hook source/output paths follow their existing +working-directory rules. Config loading must retain path provenance through imports. + +Local/S3 `key_pattern` overrides the path below the stable root. Default paths follow the addressing +table. Expose `.Name`, `.Version`, `.Kind`, and `.CreatedAt`; use the existing Go template functions. +Capture `.CreatedAt` once in UTC for a publication and reuse it across retries. Reject root escapes +and conflicting resolved paths; validate archive extensions against packaging. There is no implicit +strftime interpretation of `%Y` or `%M`. + +Persist the resolved location with the artifact's name/version record. Reads, deletes, and retries +use that record, not today's date. Keep discoverable records under the stable root even when +payloads use dated paths. An interrupted publication must not expose a complete-looking record. + +`permissions: "0600"` controls the resulting local archive file, not its members, S3 access, or OCI +access. Accept quoted octal permission bits; reject `preserve` and unsupported backend usage. +Default local archive permissions are `"0600"`. Preserve ordinary payload permission bits, +including executable bits, where the target filesystem supports them. Full owner, ACL, xattr, +and privileged-bit backup fidelity is outside V1; report unsupported permission requirements. + +## Command surface and mutation semantics + +The proposed public command space is plural: `atmos artifacts`. + +```shell +atmos artifacts publish ./dist --repository backups --name builds/app --version build-123 +atmos artifacts publish ./dist --repository backups --name builds/app --version build-123 --force +atmos artifacts publish ./dist --repository backups --name builds/app --version build-124 --dry-run +atmos artifacts pull components/terraform/vpc:1.2.0 --repository shared --output ./vendor/vpc +atmos artifacts copy components/terraform/vpc:1.2.0 --from shared --to backups +atmos artifacts inspect components/terraform/vpc:1.2.0 --repository shared --format json +atmos artifacts sign 'components/terraform/vpc@sha256:' --repository shared +atmos artifacts verify components/terraform/vpc:1.2.0 --repository shared +atmos artifacts list --repository shared +atmos artifacts list components/terraform/vpc --repository shared +atmos artifacts search vpc --labels 'team=platform,purpose=backup' +atmos artifacts show readme components/terraform/vpc:1.2.0 --repository shared +atmos artifacts show changelog components/terraform/vpc:1.2.0 --repository shared +atmos artifacts delete builds/app:build-123 --repository backups +``` + +- `publish` creates a versioned artifact from a file or directory. Update means republishing that + reference with `--force`; there is no separate in-place editing command or `--replace` flag. +- `pull` verifies and materializes content at `--output`. Existing different output requires + `--force`; validate/extract into staging before replacing it. Do not overlay stale files. +- `copy` preserves artifact identity, metadata, documents, and verified content. Destination + packaging may change physical representation; destination label defaults do not restamp copies. + Use backend-native transfer where available and a bounded streaming/staging fallback elsewhere. +- `inspect` reads metadata; `show` reads a selected document. Neither downloads the main payload. +- `list` enumerates artifacts/versions; `search` filters name/description and labels. Both identify + the source alias, support structured output, and accept `--refresh` for discovery-cache refresh. +- `delete` removes the selected published reference and its exclusively owned content where the + backend supports it. It never means delete the collection, purge all versions, or remove a + catalog entry. Shared OCI blobs are subject to registry garbage collection. + +Compare content, documents, and effective user metadata before publishing. An identical publication +is a no-op; generated timestamps must not turn an identical retry into different content. Reuse +the existing record on a no-op. Different content at the same name/version fails unless `--force` +is supplied. Forced publication creates a replacement record and updates the reference; it does +not mutate bytes addressed by an old digest. Old digests remain readable only while retained. + +Archive assembly must be deterministic for the same input bytes, paths, modes, and declared +metadata. Normalize incidental archive timestamps and ordering. Separate content identity from +generated operation metadata and backend representation digests. `inspect` and machine output +must clearly label content/record digests versus OCI manifest or archive checksums. + +OCI tag updates are not portable compare-and-swap. Require external serialization for writers to +the same mutable reference and reconcile interrupted writes before retrying. Use conditional +writes/locks where supported; never claim universal atomic publication across multiple objects. +Publish completion metadata last. Unsupported operations fail explicitly before mutation. + +`--force` does not bypass authentication, verification, or immutable-release policy. In particular, +plugin releases remain immutable. Generic artifact operations must not provide an overwrite path +around plugin-format immutability when they encounter plugin releases. + +Dry runs display selected files, exclusions, metadata, and resolved destination without mutation. +Commands are noninteractive in CI, return nonzero for operation failures, and keep structured +results on stdout and human progress on stderr. Authentication uses existing Atmos auth/profile +flags and identities; no new artifact login/logout subsystem is introduced. + +## Metadata and package documentation + +The portable artifact record contains name, version, kind, description, creation/provenance data, +labels, content inventory, document descriptors, and integrity information. Inventory describes +relative paths, sizes, digests, and file modes. CI provenance may include commit, run, and source +repository identifiers; generic artifacts do not require a Terraform stack/component/SHA tuple. + +Repository `defaults.metadata.labels` supplies defaults for newly published artifacts. Explicit +publication metadata overrides matching default keys. These labels belong to each published +artifact version, not to the repository itself. Metadata changes affect artifact identity and +follow the same conflict/force rules as content changes. +Use `publish --labels 'team=platform,purpose=backup'` for CLI overrides and `metadata.labels` in +artifact steps. Inspection and machine output return the effective labels after defaults merge. + +| Backend | Authoritative artifact metadata | Efficient document access | +| --- | --- | --- | +| OCI | Versioned Atmos config blob referenced by the manifest; standard annotations for standard fields | Separate document blobs referenced by descriptors | +| S3 | JSON sidecar record, not only object headers or tags | Separate document objects | +| Local directory | Managed metadata alongside payload content | Direct document files | +| Local archive | Adjacent record plus embedded portable metadata | Adjacent document sidecars; embedded copies for transport | + +For archives, embed portable metadata and documents under `.atmos/artifact/`; keep operation and +storage-location data in the adjacent record to avoid self-referential archive checksums. Readers +verify document descriptors and the payload checksum from that record. If only a standalone archive +is available, recover embedded metadata/documents by scanning it; do not promise random access into +gzip. Sidecars are part of the managed publication and must be consistent with its record. + +S3 `options.tags` specifies backend object tags for lifecycle/access integration. Apply them to +payload, metadata, and document objects of the publication. They do not replace portable labels, +and labels are not automatically copied into tags. Validate the S3 limit of ten tags before writes; +tag changes outside Atmos do not change portable artifact labels. +See [S3 object tagging](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-tagging.html). + +### README and CHANGELOG conventions + +```text +components/terraform/vpc/ + main.tf + README.md + .atmos/artifact/ + README.md + CHANGELOG.md +``` + +Default package documents are `.atmos/artifact/README.md` and `.atmos/artifact/CHANGELOG.md`. +Both are optional. `--readme` and `--changelog` override their source paths, resolved relative to +the publish source directory (or the parent for a single-file source); explicit missing files fail. +Step equivalents are `readme` and `changelog`. No changelog format/parser is required in V1. + +Root README/CHANGELOG files remain payload unless explicitly selected as package documents. +Existing scaffold root README handling, generated files, and post-generation instructions retain +their behavior. Reserve `.atmos/artifact/` for distribution metadata and exclude it from scaffold +generated output. Explicit document paths do not implicitly remove an ordinary payload file. + +Store document media types, sizes, and digests in metadata. `show readme` and `show changelog` fetch +only the selected blob/object/file. Missing optional documents produce a clear not-found result. +Catalogs carry descriptors, not copies of full Markdown documents. + +### OCI media types and Helm lessons + +Use OCI image manifests with artifact metadata, not fabricated runnable image configurations. +ORAS is the transport/assembly library; it does not dictate one media type for all payloads. + +| Role | Proposed type | +| --- | --- | +| OCI manifest | `application/vnd.oci.image.manifest.v1+json` | +| Generic artifact type | `application/vnd.atmos.artifact.v1` | +| Scaffold artifact type | `application/vnd.atmos.scaffold.v1` | +| Component artifact type | `application/vnd.atmos.component.v1` | +| Toolchain package artifact type | `application/vnd.atmos.toolchain.package.v1` | +| Metadata config | `application/vnd.atmos.artifact.config.v1+json` | +| Tar payload | `application/vnd.atmos.artifact.content.v1.tar` | +| Gzip-compressed tar payload | `application/vnd.atmos.artifact.content.v1.tar+gzip` | +| Zip payload | `application/vnd.atmos.artifact.content.v1.zip` | +| README/CHANGELOG blob | `text/markdown` | + +Record component subtype in portable metadata. Preserve upstream media types for tool release +assets and existing OCI graphs. Existing plugin media types, manifest bytes, signatures, and +lockfiles must remain compatible; this proposal does not rename plugin packages. + +Follow the [OCI artifact manifest guidance](https://github.com/opencontainers/image-spec/blob/main/manifest.md#guidelines-for-artifact-usage). +Adopt Helm's useful conventions: package name/version drive publication beneath a collection root, +and metadata and payload have distinct media types. See [Helm OCI registries](https://helm.sh/docs/topics/registries/). +Atmos does not reuse Helm media types for non-chart content. + +## Discovery, labels, and optional catalogs + +`list` and `search` use `artifacts.repositories` directly. Without `--repository`, search all +configured sources. `--labels 'team=platform,purpose=backup'` matches both exact key/value pairs +on the published version. Filter portable metadata, not S3 tags or a registry-specific label API. +Bound concurrent reads and reuse metadata by immutable digest; do not fetch payloads to search. + +Directory, archive-root, S3, and OCI-layout repositories enumerate their managed records locally +or by object listing. An OCI collection prefix is not itself a portable repository-enumeration API. +Use supported registry enumeration scoped to the prefix when available. Otherwise collection-wide +discovery requires a catalog; fail with an actionable capability diagnostic rather than silently +returning an empty or supposedly complete result. Explicit-name operations, including +`list components/terraform/vpc`, can list that OCI repository's tags without a catalog. + +An optional `catalog` belongs directly to the configured repository: + +```yaml +artifacts: + repositories: + shared: + kind: oci/registry + identity: registry-reader + options: + registry: ghcr.io + prefix: acme/atmos + catalog: oci://ghcr.io/acme/atmos/catalog:stable +``` + +Adapt the [plugin discovery and catalog design](./plugins/discovery.md): separate catalog +build/publish/add/remove/sync operations, version/digest-level labels, bounded metadata reads, +and a private local cache scoped by effective source/auth configuration. Catalog references +never grant credentials to another host or bypass current source/verification policy. + +`publish` never updates a catalog implicitly. Catalog maintainers use one externally serialized +writer spanning read-latest, build, and publish. Revision/digest checks detect stale updates but +are not atomic registry locks. Removing an entry unlists it without deleting its artifact. + +Use a 15-minute discovery-cache TTL initially. `--refresh` revalidates sources and reuses unchanged +digest metadata. Offline operation uses cached discovery only; missing/corrupt caches and offline +refresh requests fail. Online refresh failures must not silently return stale or incomplete search +results. Cached discovery does not prove payload availability or authorize installation. + +## Producers, workflows, and hooks + +### Components and scaffolds + +```shell +atmos artifacts publish ./components/terraform/vpc \ + --repository shared --name components/terraform/vpc --version 1.2.0 --kind component/terraform +atmos scaffold publish ./scaffolds/landing-zone \ + --repository shared --name scaffolds/landing-zone --version 2.0.0 +``` + +`atmos scaffold publish` delegates to the generic publisher with scaffold validation and kind. +Keep scaffold source resolution and component source provisioning as consumers of shared OCI +operations. Publishing a component does not run Terraform or require a stack selection. +The component producer excludes Terraform working directories, state, and generated plan files; +dry-run output makes those exclusions visible. Generic backup/file publishing does not inherit +component-specific exclusions. Preserve executable scripts and package documents. + +### Verified toolchain packages + +Publish original downloaded release archives/executables plus platform, upstream URL, checksum, +size, and verification metadata from toolchain lock data. Verification precedes publication. +Do not treat repackaged installed binaries as equivalent to verified upstream release bytes. +Toolchain remains responsible for resolution, verification, extraction, installation, and PATH; +artifacts provide storage/distribution and an explicit repository-backed package source. +Prove installation from published packages with upstream access disabled. Installed-tool snapshots, +automatic dependency crawling, and changing native CI cache semantics are deferred. + +### Shared step handler + +Add `type: artifact` to the existing step registry. Its `action` selects the same service operation +as the CLI. `artifact_name` and `artifact_kind` distinguish artifact identity from the step's own +`name` and hook `kind`; the CLI equivalents are `--name` and `--kind`. + +```yaml +workflows: + publish-vpc: + steps: + - name: publish-vpc + type: artifact + action: publish + source: ./components/terraform/vpc + repository: shared + artifact_name: components/terraform/vpc + artifact_kind: component/terraform + version: "1.2.0" +``` + +```yaml +hooks: + backup-output: + events: [after-terraform-apply] + kind: step + type: artifact + on_failure: fail + with: + action: publish + source: ./dist + repository: backups + artifact_name: backups/app + version: build-123 + force: false +``` + +Read/delete step actions use `ref`; pull uses `destination` for CLI `--output`; copy uses `from` +and `to`. Preserve existing templating, working-directory, retry, conditional, and failure behavior +through the [step hook bridge](./hooks-step-types.md). Return typed operation results and named +outputs, including reference, resolved location, digests, size, and outcome. Retry context must +retain publication identity and the captured timestamp. + +Do not introduce a separate artifact hook implementation. Existing `type: publish` delivery behavior +and [archive steps](./archive-step.md) keep their semantics; reuse archive primitives internally. + +## CI compatibility and job summaries + +Artifact operations must produce polished Markdown job summaries when CI reporting is enabled. +Use the existing `ci.Reporter`, summary feature gates, provider routing, masking, and template +loader, following the [container reporting integration](./native-ci/providers/container.md). +Honor `ci.enabled`, `ci.summary.enabled`, and existing explicit CI-mode overrides. Do not write +directly to provider-specific environment files from artifact code. + +```yaml +ci: + enabled: true + summary: + enabled: true + templates: + base_path: .atmos/ci/templates + artifacts: + publish: artifacts/publish.md + copy: artifacts/copy.md +``` + +`ci.templates.artifacts` is a proposed operation-to-template map. Resolution follows explicit +override, `/artifacts/.md`, then embedded default. An invalid explicit +override produces a reporting diagnostic rather than silently choosing a different template. +Reporting remains best-effort and cannot turn a successful transfer into a failed operation. + +The default presentation includes a heading and compact results table with operation, outcome, +artifact name/version, repository, destination, digest, human-readable size, and duration. +Use short display digests with full values in expandable details; identify their digest type. +Include useful links only where a backend can supply a valid viewer/download location; otherwise +show a copyable reference. Do not emit credentials or signed URLs into summaries. + +Illustrative result rows (additional destination and integrity details appear below the table): + +| Operation | Outcome | Artifact | Repository | Size | Duration | +| --- | --- | --- | --- | --- | --- | +| Publish | Published | `components/terraform/vpc:1.2.0` | `shared` | 48 KiB | 1.2 s | +| Publish | Unchanged | `scaffolds/landing-zone:2.0.0` | `shared` | 96 KiB | 0.3 s | +| Publish | Failed | `backups/app:build-123` | `backups` | — | 2.1 s | + +Distinguish published, unchanged, replaced, downloaded, copied, deleted, signed, verified, and failed outcomes. +Read operations report inspected/listed results without claiming a payload transfer. Collapsible +sections show metadata, selected labels, file counts, document availability, and useful failure +diagnostics. Do not embed entire package files or changelogs by default. + +One shared renderer consumes operation results for CLI, workflow, and hook entry points. +Assign reporting ownership at the outer execution boundary so hook bridging and nested steps +do not emit duplicate summaries. Aggregate concurrent work in deterministic order and serialize +writes; retain completed operations if a later operation fails. Retries produce one final outcome +per operation, with retry information in details. + +Escape Markdown content, mask secrets, and truncate to provider limits with a visible omission +notice. Summary rendering/delivery failures log a diagnostic and preserve the operation's exit +status. Disabled summaries perform no additional metadata fetches. Preserve unrelated existing +job-summary content. Structured stdout remains clean; local presentation uses existing stderr/UI +behavior. PR comments and status checks are separate features and are not implicitly enabled. + +## Shared implementation and compatibility + +Build a common artifact service over registered repository adapters with explicit capabilities: +resolve/inspect, publish, pull, list, delete, document reads, copy, signing, verification, pinning, +and write locking. Each adapter implements these contracts through its native mechanisms or shared +providers: OCI referrers, S3 evidence objects/conditional claims, and local detached files/locks. +Capabilities state supported signature formats and lock guarantees; unsupported requirements fail. +Verify the pinned signature subject against current trust policy before required consumption/copy, +preserve evidence across transports, and support offline verification with independently trusted roots. +Distinguish immutable dependency pins from write locks and signatures from checksums; `--force` +cannot weaken verification or coordination requirements. Keep backend authentication separate from +signing identity and producer-specific validation. See the implementation contracts for policy and APIs. + +Refactor [shared OCI operations](./plugins/platform/oci-reuse.md) around +[`oras-go/v2`](https://github.com/oras-project/oras-go). The plugin implementation selected +`go-containerregistry` to reuse the existing stack; this migration is deliberate convergence, not +a second independent OCI client in artifact commands. Replace library-specific public types with +shared descriptors/references and migrate callers behind compatibility entry points. + +Preserve raw manifests/indexes, blobs, digests, media types, platforms, signature/referrer transport, +verification policy, and plugin immutability. Retain credential precedence, identity failure +behavior, private CA handling, cancellation, and independent source/destination credentials. +Existing plugin packages, catalogs, signatures, and locks must remain readable without rewriting. +Do not remove `go-containerregistry` until remaining image-specific consumers have been audited. +No Skopeo subprocess or new `containers/image/v5` transport is required by this proposal. + +For generic artifacts copied across packaging backends, preserve verified logical content and +metadata while reporting any changed representation checksum. For existing OCI graphs transferred +through S3/local storage, preserve a standard OCI-layout representation, including referenced blobs; +restore it without rewriting manifests. Do not flatten container layers into an ordinary file tree. + +Adapt [native CI artifact storage](./native-ci-artifact-storage.md) to the common substrate instead +of duplicating its S3/local clients. Preserve existing planfile store config, key patterns, identity +resolution, metadata, selection rules, and archive compatibility. General metadata must not require +CI-only fields. Fix mode loss when adapting file bundles so executable tools/scripts survive. +Existing GitHub artifact storage remains CI-scoped; it is not a durable general repository promise. + +## Delivery and acceptance + +Deliver in stages: shared OCI migration with conformance coverage; repository service and local/S3 +adapters; command/step/hook surfaces; metadata/documents and discovery/catalogs; producer integrations +and CI reporting. Each stage must preserve existing consumers. Runtime implementation must add +schema definitions, CLI help, and website documentation; this PRD change adds no runtime behavior. + +Acceptance scenarios: + +1. Configure one repository, then publish multiple components, scaffolds, and generic files without + adding repository entries. Repeat against OCI, S3, directories, and archives using the same CLI. +2. Round-trip payloads and documents across backends; compare content digests and file modes. + Transfer an OCI graph through a layout and a second registry with identical manifest digests. +3. Read metadata and README/CHANGELOG with payload downloads forbidden by the test transport. + Recover metadata from a standalone archive and explicitly account for the required scan. +4. Verify unchanged publication is a no-op, different bytes require force, and plugin releases + reject replacement. Exercise interrupted writes, conflicting names, staging, and retries. +5. Read dated publications on a later day; confirm retry stability, source/destination anchoring, + path containment, archive permission bits, and executable payload fidelity. +6. Filter version-specific labels across repositories. Test catalog-free discovery where supported, + explicit-name OCI lookup, unsupported collection enumeration, and catalog-based discovery. + Publication must leave catalogs unchanged. Verify offline cache and auth-scope behavior. +7. Publish a scaffold without changing generated README behavior; consume a component through + existing provisioning; install a verified tool package with its upstream unavailable. +8. Run equivalent CLI, workflow, and hook operations with matching results, retry semantics, and + exactly one CI report. Verify reporting gates, template overrides, secret masking, Markdown + escaping, size limits, concurrent aggregation, no-op results, and partial failures. +9. Preserve existing CI planfile fixtures, plugin release/catalog/lock compatibility, signature + verification, and source/destination auth isolation through the ORAS migration. + +Use unit tests with injected transports and generated mocks, temporary local repositories, the +existing in-process OCI registry harness, and deterministic S3 fakes. Opt-in emulator round trips +must seed a source, publish to a destination, disable the source, and restore into fresh state. +Never substitute successful metadata lookup for proof that offline payload restoration works. + +## Deferred requirements preserved from the earlier draft + +These remain intended follow-up capabilities, not additional V1 command/configuration promises: + +- Bulk image mirror/sync: standalone and component-driven selection, explicit host/all/selected + platform policies, source-host-preserving names, tag preservation, promotion, and collision rules. +- Bundle integration: select an exact dependency closure, mirror locally, export/import for air-gap + use, and verify restoration without upstream access. A bundle is distinct from one archive artifact. +- Vendoring convergence: compile existing vendor sources into source-artifact operations while + preserving hydration targets, include/exclude filters, version resolution, and provenance. +- Additional Git-backed, HTTP archive, GitHub release, and CI-scoped repository adapters; retain + native Git delivery semantics and document each backend's capability and retention boundaries. +- Terraform registry caches: export/restore individual provider/module records and required indexes + by origin, version, platform, and digest; verify restored mirrors and module resolution offline. +- Toolchain installed-tree snapshots and resolver/index distribution for complete offline tool + discovery. V1 verified upstream package storage must remain independently useful. +- SBOMs and attestations: subject-digest associations, preserve/require/ignore transfer policy, + CycloneDX/SPDX documents, pluggable generation, and aggregate dependency-closure inventories. + Existing plugin signature requirements are compatibility obligations now, not deferred work. +- Repository statistics, retention/prune/garbage-collection policies, additional signing schemes/encryption, + and daemon/container-storage transports. V1 explicit reference deletion remains in scope. +- Structured changelog parsing, version-range release-note aggregation, and template-update PR + enrichment using the package documents introduced here. + +No historical singular `artifact.repositories`, per-image publish-target schema, or Skopeo-specific +flags in the old draft are introduced as compatibility aliases by this proposal. Future extensions +must build on the collection-root model and plural `atmos artifacts` command space defined here. diff --git a/docs/prd/artifacts/contracts.md b/docs/prd/artifacts/contracts.md new file mode 100644 index 00000000000..73469940b60 --- /dev/null +++ b/docs/prd/artifacts/contracts.md @@ -0,0 +1,465 @@ +# Atmos Artifacts implementation contracts + +**Status:** Proposed implementation specification, not an implemented API. +Read with the [implementation plan](./implementation.md) and [product requirements](../artifacts.md). +The names below are implementation defaults for V1. Change an externally visible contract only by +updating its specification, examples, schema, tests, and consuming adapters together. + +## Package and dependency boundaries + +```mermaid +flowchart TD + A[CLI, steps, hooks, CI adapters] --> B[Artifact service] + B --> C[Repository contracts] + B --> P[Producer interfaces] + C --> L[Local adapter] + C --> S[S3 adapter] + C --> O[OCI adapter] + L --> IO[Shared blob I/O] + S --> IO + O --> OCI[pkg/oci and ORAS] + C --> V[Shared signer and verifier interfaces] + C --> K[Lock provider interfaces] + CI[Legacy CI adapters] --> IO +``` + +The artifact service imports no CLI, runner, hook, plugin, or CI packages. Backends and producers +register through composition roots; plugin-specific policy stays above generic OCI transport. +`pkg/artifact/report` may depend on CI reporting; `pkg/artifact` itself must not. Share cached +discovery infrastructure below both plugin and artifact discovery, with separate schema adapters. + +## Domain records and integrity + +Use these types in `pkg/artifact`, with explicit JSON tags and versioned serialization: + +| Type | Required data | +| --- | --- | +| `Ref` | Name and exactly one version or digest; repository alias is a separate request field | +| `Record` | Schema version 1, name, version, kind, created_at, metadata, sorted inventory, documents, content_digest, optional typed kind data | +| `Metadata` | Description, string labels, optional CI/source provenance; no required stack/component/SHA | +| `Entry` | Relative path, type (file/directory/symlink), size, ordinary mode bits, file digest or link target | +| `Document` | Role (readme/changelog), original source path, media type, size, digest | +| `Representation` | Packaging, payload/blob descriptors, backend locator, record digest, optional original OCI root descriptor | +| `PreparedArtifact` | Record, intent digest, staged blob readers, preparation timestamp, cleanup ownership | +| `OperationResult` | Operation ID, action, outcome, requested/resolved refs, aliases/locations, digest fields, size, duration, diagnostics | +| `Capabilities` | Read/write, enumeration, delete semantics, conditional update, packaging/modes, evidence formats, lock scopes | +| `SignatureSubject` | Subject kind, immutable digest, media type, size; portable Record or native OCI graph root | +| `SignatureEvidence` | Subject, format, evidence descriptor, storage locator; untrusted until verified | +| `VerificationResult` | Subject, policy fingerprint, verified signer claims, evidence digests, status and diagnostics | +| `PinnedReference` | Repository locator, resolved digest, record/content digests, evidence descriptors, optional selected platform | +| `LockRequest` / `LockHandle` | Resource scope, owner/operation ID, required guarantee, expiry where leased, ownership token, renew/release methods | + +Keep record bytes independent of repository alias, credentials, resolved storage paths, upload +duration, and archive/OCI envelope digests. Those belong to Representation or OperationResult. +`Record` never includes its own hash. OCI config stores the same canonical Record bytes used by +local/S3 records; the manifest references the backend representation separately. + +Canonical serialization uses a typed JSON schema, sorted inventory/document arrays and map keys, +UTF-8 strings, integer sizes/modes, UTC RFC3339 timestamps, no whitespace, and no trailing newline. +Freeze byte-level test vectors; do not rely on arbitrary `map[string]any` or float serialization. + +- `content_digest`: SHA-256 of the canonical inventory and document descriptors, including paths, + modes and link targets. Exclude storage locators and generated operation data. +- `intent_digest`: SHA-256 of name/version/kind, content_digest, effective user metadata and kind + data. Exclude created_at and generated execution provenance; they cannot defeat retry/no-op. +- `record_digest`: SHA-256 of the full canonical Record bytes, including the original created_at. +- `representation_digest`: Checksum of an archive or other transferred payload representation. +- `resolved_digest`: OCI manifest/index digest for OCI references; record_digest for local/S3 refs. + +A copy preserves Record bytes and content identity. Repacking may change representation_digest; +an OCI envelope newly assembled from another backend may have a different resolved_digest. +Existing OCI graphs preserve every original descriptor and blob digest instead. + +Record and catalog JSON reads are bounded to 8 MiB, each document to 4 MiB, OCI manifest/index +reads to the existing bounded transport policy. Reject oversize metadata before publishing; +stream payloads to disk/network rather than loading them into memory. Use a default transfer and +discovery concurrency of four, injected through service options for tests; retain existing plugin limits. + +## Service and repository APIs + +Define separate request structs for Publish, Pull, Copy, Inspect, List, Search, Delete, and +ShowDocument. Each takes context first and returns typed results plus error. Do not use maps for +request fields or one method with backend-specific optional arguments. +Add Sign, Verify, and Pin requests; publication/copy/delete use the same Lock contract internally. + +The service owns name/version selection, producer preparation, effective metadata, digest pinning, +conflict policy, retries, and result aggregation. Repositories own storage mechanics and capabilities. + +Repository interfaces must provide: + +- `Capabilities()` and `Resolve(ctx, Ref)` returning a pinned descriptor/record handle. +- `Inspect(ctx, pinned)` and `OpenDocument(ctx, pinned, role)` without payload download. +- `Read(ctx, pinned)` returning closeable verified content streams or an OCI graph source. +- `Publish(ctx, prepared, options)` with expected prior reference and force policy already checked. +- `List(ctx, query)` returning a page and cursor; pagination is consumed by the service. +- `Delete(ctx, pinned, options)` with expected current reference and explicit deletion scope. +- `Pin(ctx, Ref)` returning an immutable PinnedReference suitable for consumer lockfiles. +- `Sign(ctx, subject, signer, options)` returning stored SignatureEvidence descriptors. +- `Verify(ctx, subject, policy)` returning authenticated VerificationResult, never metadata claims. +- `Lock(ctx, LockRequest)` returning an ownership-checked LockHandle or unsupported-capability error. +- Optional `CopyFrom` for a compatible native transfer; absence uses the shared staging fallback. + +Use OCI-standard descriptors at the OCI seam. Keep `remote.Repository`, go-containerregistry image +objects, AWS clients, and SDK errors out of the generic service API. Wrap errors in common sentinels +while retaining their causes for diagnostics. Generate mocks from these interfaces. + +## Public command and step mapping + +Signatures, verification, pinning, and coordination are repository contracts, not OCI-only service +branches. The following operations use them identically for every backend. + +The following options supplement normal Atmos auth/profile/CI/output flags. A missing required +repository is an error; only collection list/search intentionally default to all configured repositories. + +| Operation | CLI input | Artifact step fields | +| --- | --- | --- | +| Publish | `publish --repository R --name N --version V --kind K` | `action: publish`, `source`, `repository`, `artifact_name`, `version`, `artifact_kind` | +| Publish controls | `--force`, `--dry-run`, `--labels`, `--readme`, `--changelog` | `force`, `dry_run`, `metadata.labels`, `readme`, `changelog` | +| Pull | `pull --repository R --output DIR [--force]` | `action: pull`, `ref`, `repository`, `destination`, `force` | +| Copy | `copy --from A --to B [--force]` | `action: copy`, `ref`, `from`, `to`, `force` | +| Inspect | `inspect --repository R` | `action: inspect`, `ref`, `repository` | +| Sign | `sign --repository R` | `action: sign`, digest-only `ref`, `repository` | +| Verify | `verify --repository R` | `action: verify`, `ref`, `repository` | +| List | `list [name] [--repository R] [--labels L] [--refresh]` | `action: list`, optional `artifact_name`, `repository`, `labels`, `refresh` | +| Search | `search [text] [--repository R] [--labels L] [--refresh]` | `action: search`, `query`, `repository`, `labels`, `refresh` | +| Delete | `delete --repository R` | `action: delete`, `ref`, `repository` | +| Documents | `show readme\|changelog --repository R` | `action: show`, `document`, `ref`, `repository` | + +Publishing defaults to kind `files`; specialized convenience commands supply their kind. Name +precedence is explicit flag, producer metadata, source basename. Version precedence is explicit +flag then producer metadata; otherwise fail. No generated version, implicit latest, or package manifest +file is required for generic files. CLI publish labels use the existing exact key/value parser. + +Explicit document paths resolve relative to the publish source directory or file's parent. Generic +source files can have any basename; name validation applies to artifact addressing, not payload names. +Validate portable relative name segments and OCI tag syntax for versions; reject backend-specific +invalid references before any writes rather than silently normalizing case or characters. + +Step `name` remains the runner's name; `output` remains its display mode; `Artifacts` remains the +runner's expected-output declaration. Add only missing fields to WorkflowStep/Task and implement +the conversions explicitly. `source` must be a string for this step, despite other steps accepting maps. + +Store the typed operation result under StepResult.Metadata[`artifact`]. Named Outputs are `ref`, +`location`, `resolved_digest`, `record_digest`, `content_digest`, `representation_digest`, `size`, +and `outcome`; optional values are omitted, never filled with misleading empty-success strings. +List/search use an `artifacts` results array. Unknown action/field combinations fail validation. +Structured CLI JSON/YAML uses the same named result fields; display format never changes semantics. + +## Signatures, verification, and locks + +Each repository implements Sign, Verify, Pin, and Lock, optionally by composing shared providers. +Capabilities describe supported evidence formats, immutable subjects, lock scope, and conditional +publication guarantees. An unavailable capability returns a typed error; it is never a successful +no-op. The orchestration layer asks for guarantees, not `if backend == OCI` behavior. + +| Repository | Signature evidence | Pinning | Default write coordination | +| --- | --- | --- | --- | +| OCI registry | Subject-bound OCI referrers and supported fallback attachments | Original manifest/index digest, or portable Record digest bound through config | Host-local guard plus documented external serialization across hosts; configured lock repository for stronger scope | +| OCI layout | Referrer descriptors/blobs in layout plus subject index | Immutable layout descriptors | Filesystem guard for the layout/ref | +| S3 | Digest-keyed signature objects and subject records | Canonical Record digest, original OCI root when carrying a graph | Conditional owner claims and conditional reference updates | +| Local directory/archive | Detached signature files; portable evidence included when exporting archives | Canonical Record digest, original OCI root for graph payloads | Filesystem locks and atomic reference updates | + +### Signature subjects and trust + +For Atmos file packages, sign the canonical portable Record. It binds name/version, metadata, +inventory, content_digest, and document digests, so evidence survives representation changes. +For foreign OCI images and existing plugins, sign/verify the original manifest or index digest; +never substitute an archive checksum or rewrite an existing signature subject on copy. +An OCI attachment carrying a Record signature must bind that Record to the addressed manifest's +config descriptor. Verification distinguishes a signed Record from a signed native image envelope. + +Store evidence by subject kind/digest and evidence digest. Local/S3 locations are beneath +`.atmos-artifacts/v1/signatures///.json`. +Signing adds evidence without changing the signed Record or native graph root. A local archive +signed after publication receives detached evidence; subsequent export/copy can embed that evidence +without changing the portable Record. Do not claim the original archive bytes were signed. + +Extract reusable cryptography from `pkg/oci/signature` into `pkg/signature`, retaining compatibility +wrappers and existing plugin behavior. Repository adapters fetch/store evidence and invoke the +shared verifier; they may add native formats through registered providers with the same result contract. +Reuse the existing managed Cosign authoring path behind a generic Signer interface for V1 signing; +registry authentication remains separate from signing identity. Do not implement custom cryptography. + +V1 supports Sigstore bundles with trusted certificate identities, and an explicitly configured +public-key bundle verification path. Keep that key verifier separate: a public-key-only signature +must never satisfy an OIDC issuer/identity requirement or weaken existing plugin SCT/log checks. +Legacy Cosign tag payloads and other formats require dedicated tested providers; do not claim support +merely because a signature-shaped object is present. Unsupported required evidence fails closed. +Follow [Sigstore verification](https://docs.sigstore.dev/cosign/verifying/verify/) and +[bundle semantics](https://docs.sigstore.dev/about/bundle/) through their supported Go libraries. + +Proposed policy configuration: + +```yaml +artifacts: + verification: + require_signature: true + trusted_identities: + - identity: https://github.com/acme/images/.github/workflows/release.yml@refs/heads/main + issuer: https://token.actions.githubusercontent.com + trusted_keys: [] + repositories: + shared: + kind: oci/registry + options: + registry: ghcr.io + prefix: acme/images + locking: + required_scope: repository + repository: coordination + coordination: + kind: aws/s3 + identity: lock-writer + options: + bucket: acme-artifact-coordination +``` + +`verification` is allowed globally and per repository. Unset require_signature defaults to false +for new general artifacts; existing plugin enforcement remains unchanged. Global requirements +cannot be weakened by repository overrides or CLI flags. Within one trust set, any configured signer +can satisfy it; if global and repository trust restrictions both exist, satisfy both sets rather +than unioning them into a weaker policy. Trusted keys are operator-managed public-key file paths. +Protected governing policies remain authoritative over ordinary imported configuration. +This enforcement covers artifact repository operations and their adapters; it does not imply +new admission controls on unrelated container-runtime pulls that bypass these operations. + +`verify` always requires a trusted valid signature; `sign` accepts digest refs only and adds evidence. +Automatic consumption/copy verifies when policy requires it or supplies trust constraints. Without +trust configuration, report unchecked rather than verified. Required verification runs before +destination publication or local materialization; a normal checksum is not signature verification. +Copy evaluates both source and destination requirements and retains the evidence needed by each. +`--force` never bypasses verification. Catalog entries, labels, prior result files, and cached +"verified" booleans never establish trust. + +At least one valid permitted signature must satisfy each required trust set. Invalid unrelated +attachments cannot override valid matching evidence, but malformed/unsupported evidence is bounded +and reported. If no permitted signature verifies, return a nonzero policy error before consumption. +Verify signature subject binding, cryptographic validity, signer/key trust, and applicable log, +certificate, and signing-time proofs. Preserve original index signatures when selecting platforms; +platform selection must not claim a newly generated index retains the original signature. + +Offline verification uses preserved evidence and independently provisioned trust material. Reuse +unexpired TUF-authenticated roots or explicitly configured operator trust, never roots supplied by +the artifact itself. Missing/expired trust or missing required proofs fail without network access. +Expose verification subject, evidence digests, policy fingerprint, verified claims, and failure reason +in inspect/verify output and CI summaries. Explicit `verify` is the metadata/evidence-only check; +payload-consuming operations additionally verify every consumed content descriptor. +Extraction matches the signed Record's inventory exactly: reject unlisted user payload members and +duplicate paths, and match each path, type, mode, link target, and file digest. Only generated +`.atmos/artifact/` distribution members excluded from the inventory are permitted, and they are +verified against their own descriptors. Tests must prove an archive with an extra member is rejected. + +### Mutation locks versus dependency locks + +`Pin` creates the immutable reference data that existing consumer lockfiles persist. It does not +acquire a write mutex, and a pinned reference is not a substitute for signature verification. +Keep existing plugin/toolchain lock schemas compatible; add adapters rather than a new mandatory +project-wide artifact lockfile. Reevaluate current trust policy when consuming a persisted pin. + +`Lock` coordinates mutations. A LockHandle identifies owner, resource, guarantee/scope, and current +ownership token; release checks the token. Lock scopes are `host` and `repository`. Host scope is +the default for backward-compatible single-host use; cross-host publication still requires external +serialization unless repository scope is available/requested. Report the actual scope in results. +Acquire locks in canonical repository/resource order for multi-resource work to avoid deadlocks. + +Local adapters use OS-backed file locks. S3 uses create-if-absent owner claims with conditional +release/update; V1 claims do not expire automatically, so a paused writer cannot become an unfenced +stale owner. A crash leaves an explicit recovery diagnostic; an operator may remove the owner claim +only after confirming that writer has stopped. Do not automatically steal another operation's lock. +Future expiring leases must add fencing before advertising equivalent guarantees. + +An OCI adapter provides host locking but cannot implement distributed exclusion from mutable tags. +`locking.repository` delegates Lock to another configured repository that advertises repository +scope; its identity and storage root stay independent. Reject cycles and insufficient scope before +mutating. A catalog writer may use this coordinator or an externally serialized CI workflow. +No S3 Object Lock retention setting, OCI tag, or local-cache lock is advertised as a distributed lease. +External writers that ignore Atmos coordination remain outside these guarantees. + +## Preparation and document conventions + +Relative local repository roots use their declaring config directory. For remote imported config, +require a locally declared or absolute destination override rather than using ephemeral download paths. + +Capture a single UTC preparation time and logical operation ID before the first attempt. Explicit +user metadata participates in intent identity; automatically attached run IDs/durations do not. +On a no-op return the existing Record unchanged and report the current execution separately. + +Walk source paths using lstat. Preserve directories, file bytes, ordinary permissions and contained +relative symlinks; never follow a link outside the source. Reject absolute/escaping link targets, +devices, sockets, and unsupported entry types. Represent hard links as independent regular files. +Fail materialization when a target cannot implement a requested link instead of silently changing it. +Clear privileged mode bits; do not restore owner/group IDs or ACLs in V1. + +Use epoch archive timestamps, deterministic gzip headers, stable ordering, and normalized owners +for newly assembled generic archives. Preserve executable bits. Keep verified upstream tool assets +as opaque files; any outer archive does not change the original package bytes or its checksum. + +The source `.atmos/artifact/` directory is reserved for README/CHANGELOG distribution documents. +Reject conflicting unknown reserved files during preparation. Write generated `manifest.json` +there only inside the packaged representation; do not change the user's source tree. Its Record +excludes generated member entries to prevent self-reference. Root README remains normal payload. +Verify embedded document bytes against descriptors and expose documents separately for efficient reads. + +For local archive permissions, retain whether the value was explicit. Parse quoted octal permission +bits, default to 0600, and fail explicitly when an explicit request cannot be enforced on the host. +Tests must distinguish archive-file mode from member mode and from filesystem ACL guarantees. + +## Local/S3 record layout and publication lifecycle + +Below the configured root reserve `.atmos-artifacts/v1/` for managed records. Reject payload paths +and key patterns that overlap this namespace in either direction, including ancestors such as +`.atmos-artifacts`. Reject a payload path that contains a `.atmos-artifacts` segment at any depth, +in any letter case, so a root's payload can never enter the control namespace of another +configured root nested below it, and a write or deletion through one root can never change another +root's records or references. Use SHA-256 of canonical name/version for private +reference keys so names containing slashes cannot collide with control paths. + +```text +/.atmos-artifacts/v1/refs/.json +/.atmos-artifacts/v1/records/.json +/.atmos-artifacts/v1/representations/.json +/.atmos-artifacts/v1/operations/.json + +.metadata.json # archive sidecar +.docs/README.md # archive document sidecar +.docs/CHANGELOG.md +``` + +For directory/per-file payloads, metadata/documents live at `/.atmos/artifact/` instead +of suffix paths. Managed readers omit this distribution directory when materializing user content. +Reference records bind name/version to record digest and Representation; immutable records never +contain a mutable storage locator. Enumerate reference records, not arbitrary user files. +Search ignores staging/journal entries. Direct digest lookup checks the Record's embedded name. + +The operation journal records captured time, intent, expected prior reference, resolved location, +and prepared representation checksums. It is an implementation recovery record, not a public catalog. +Use local reference/location locks or S3 conditional ownership records. Reserve a location before +overwriting it; another reference cannot claim it even when its key_pattern collides. Identical, +ancestor, and descendant locations conflict. Claims atomically reserve the payload and its sidecar +paths together, rejecting payload-to-sidecar and sidecar-to-sidecar overlaps as well as payload +overlaps. Thus a force replacement or deletion cannot remove another reference's payload or sidecars. +Claims cover the references of one configured root. Payload claims are not checked across roots, so +configured roots must not nest or overlap where both publish payloads; such a configuration is +unsupported. The control-namespace rule above still protects every root's records and references. + +Publication sequence: + +1. Validate capability, policy, paths, metadata sizes, and identity; prepare a stable source snapshot. +2. Read current reference under the writer guard; compare intent and apply conflict/immutability rules. +3. Record the pending operation before changing payload paths; pending references are unavailable + to managed readers and absent from completed search results. +4. Upload/stage payloads and separate documents, verify their descriptors, and write immutable Record. +5. Write Representation/adjacent sidecars, then atomically publish the completed reference last. +6. Reconcile an ambiguous response by reading the completed reference and verifying expected digests; + mark success only after reconciliation. Clean only temporary resources owned by this operation. + +Use [S3 conditional writes](https://docs.aws.amazon.com/AmazonS3/latest/userguide/conditional-writes.html) +for reference ownership and publication, treating precondition failures as conflicts. ETags are +concurrency tokens, not payload SHA-256 checksums. External serialization is still required for +writers to a mutable OCI reference and for catalog updates; no universal distributed lock is promised. + +Readers check pending/completed state before accessing content, verify every consumed descriptor, +and recheck the generation after reads from mutable payload locations. A failed/interrupted force +replacement may leave the reference unavailable until its same-intent retry repairs it. Do not +claim rollback or serve changed bytes under an old digest. Do not automatically steal a pending +operation owned by another active writer; surface its operation ID for recovery diagnostics. + +A retry in the same invocation reuses operation ID and preparation. A subsequent invocation may +resume a matching pending intent only under exclusive writer ownership and after checking staged +checksums; a different intent fails with a recovery diagnostic. Keep this conservative behavior in V1. +Old digest reads succeed only while their payload representation remains available and verifies; +otherwise return unavailable/not-found, never the current version's bytes. + +## OCI-specific behavior + +New Atmos packages use the PRD artifact/config/layer media types. Record bytes are the config blob; +README/CHANGELOG are separate Markdown blobs with explicit roles/titles. Transferred foreign OCI +graphs bypass new-package assembly. Copy required referrers through the existing policy-aware +transport; do not silently omit signature material on an unsupported registry. + +Use [ORAS graph APIs](https://pkg.go.dev/oras.land/oras-go/v2@v2.6.2) through `pkg/oci` adapters. +Preserve original raw manifest/index bytes on graph transfer. Do not use the simple file store's +automatic filename-based extraction for arbitrary untrusted graphs. + +Resolve names before reading. Name-scoped list can enumerate OCI tags without a catalog; +collection list/search uses supported registry repository enumeration or the optional catalog. +No public fallback repository list is added. A provider without enumeration/catalog returns a +capability error explaining that direct named operations still work. + +For tag deletion, inspect other tags referencing the same digest and require a tag-safe operation; +if only manifest deletion is available and it would remove sibling references, reject. Digest +deletion is explicitly manifest-scoped and reports affected references. Never delete shared blobs +directly. Protect plugin immutability on both destination format and incoming format. + +## Catalog commands and cache + +Use a separate versioned `AtmosArtifactCatalog` manifest, not the plugin catalog schema. Each entry +contains logical name, kind, source, description, and versions with resolved/record/content digests, +labels, and document descriptors. Version labels are opaque; do not reuse plugin SemVer sorting. +Sort names/versions lexically and bind a catalog entry to its actual repository path, not credentials. +Publish catalogs as generic Records of kind `catalog`, with artifact type +`application/vnd.atmos.artifact.catalog.v1` and a direct JSON payload layer of type +`application/vnd.atmos.artifact.catalog.v1+json`; retain the shared Record config media type. + +```shell +atmos artifacts catalog build --repository shared --input inventory.yaml --output catalog.json +atmos artifacts catalog publish catalog.json --repository shared +atmos artifacts catalog add components/terraform/vpc:1.2.0 --repository shared +atmos artifacts catalog remove components/terraform/vpc:1.2.0 --repository shared +atmos artifacts catalog sync --repository shared --input inventory.yaml +``` + +`inventory.yaml` is an `AtmosArtifactInventory` manifest whose spec contains `refs`, a list of +explicit names/versions or digests relative to the selected configured repository. Build resolves +and verifies metadata only and writes a snapshot; it never publishes. Publish writes to the +repository's configured catalog URI. Add/remove read and republish the latest catalog; sync merges +verified inventory releases while retaining entries absent from the inventory. Remove only unlists. +Missing catalog configuration fails these mutating catalog commands before writes. Catalog writes +require externally serialized read/build/write, and digest/revision rechecks reject already-stale input. + +Initially support OCI catalog URIs as in the PRD and plugins; local/S3/layout discovery works +directly without a catalog. Store artifact caches under XDG `atmos/artifacts/discovery`, keeping +plugin cache paths and serialization unchanged. Cache scopes include effective config/auth realm +and host bindings without credential values; refreshed snapshots replace old snapshots atomically. + +## Producer selection and CI compatibility + +Toolchain convenience interface (new, explicit; no automatic repository fallback): + +```shell +atmos toolchain publish hashicorp/terraform@1.13.4 --repository shared --platform linux/amd64 +atmos toolchain install hashicorp/terraform@1.13.4 --artifact-repository shared +``` + +Publish accepts repeated platform selectors, defaulting to the host, and requires an explicit +version plus pinned checksum data in the configured lockfile. Name packages as +`toolchain////:`. Bundle the selected lock entry, resolved install +definition, original asset, and verification evidence. Resolve aliases using existing config. +Artifact-selected installation is authoritative: fetch definition and package from that source, +reapply verification, then use the existing installer. Missing metadata, platform, or evidence fails +without contacting upstream. Default installations retain their current behavior. + +Native CI reuses existing store selection through a new explicit adapter type: + +```yaml +planfiles: + default: shared + stores: + shared: + type: artifact/repository + options: + repository: shared +``` + +Keep the existing `--store`, default, priority, and environment detection behavior. The adapter +references `artifacts.repositories.shared`; it does not duplicate its backend options. Map legacy +keys to the fixed reference `ci/planfiles/:current`, so Download, Delete, Exists, and +GetMetadata resolve by key alone and a new upload for the same key replaces the current version under +the writer guard. Retain the original key, commit SHA, and full CI metadata in typed kind data. +Implement existing query predicates from that data, not approximate label matches. +Preserve the existing planfile overwrite contract only inside this explicit adapter; it cannot +override plugin releases. Old store configurations and serialized archives are unchanged. + +Do not weaken `pkg/ci/artifact.Metadata.Validate`. The generic Record is a new type; convert at +the boundary. Add optional modes to CI FileEntry/FileResult with legacy 0644 encoding when unset. +Keep backend constructors/adapters compatible with existing tests and public package callers. diff --git a/docs/prd/artifacts/implementation.md b/docs/prd/artifacts/implementation.md new file mode 100644 index 00000000000..7010b6f42bc --- /dev/null +++ b/docs/prd/artifacts/implementation.md @@ -0,0 +1,377 @@ +# Atmos Artifacts implementation plan + +**Status:** Ready for implementation; all work packages below are unstarted. +**Product contract:** [Atmos Artifacts PRD](../artifacts.md). +**Companion specifications:** [Contracts and storage](./contracts.md), [validation and handoff](./validation.md). + +## Instructions for the implementing agent + +Implement the work packages in dependency order. Finish the acceptance checks for one package +before marking it complete. A package is not complete because its interfaces compile or its mocks +pass: demonstrate the described behavior through its real adapter or deterministic test harness. +Keep the public names and compatibility guarantees in the PRD. This plan fills in implementation +choices; it does not authorize implementing the PRD's deferred features as part of V1. + +Start by reading this plan and both companions, then inspect the indicated production code. +Recheck current repository instructions and branch state. Preserve concurrent changes. Build the +workspace binary with `atmos build` before using repository hooks; run subsequent commands with +the workspace `build` directory first on PATH. Do not disable hooks, signing, or verification. + +Use small changes that compile independently. Use registered backends, CommandProvider commands, +standard flag parsing, functional options, static error sentinels, context cancellation, and +generated `go.uber.org/mock` mocks. Keep configuration/dependencies in explicit structs, not context +values. Keep package files below the repository's size limit. Add no artifact business logic to +the CLI or a second implementation in hooks. + +Keep a handoff record listing completed package IDs, commits, exact commands/results, remaining +failures, and the next package. Do not mark all of V1 complete while a producer, backend, document +read path, or CI surface listed here is missing. + +## Current integration map + +These are existing entry points to inspect, not a proposed file-by-file rewrite. + +| Area | Current entry points | Constraint | +| --- | --- | --- | +| OCI transport | `pkg/oci/client.go`, `identity.go`, `credentials.go`, `referrers.go` | Registry methods expose go-containerregistry types; auth and referrers are already shared | +| OCI layouts/materialization | `pkg/oci/layout.go`, `pull.go`, `selection.go`, `verified.go` | Existing layout helpers expect one root release index; collections need an additional API | +| OCI consumers | `pkg/plugin/oci`, `pkg/plugin/catalog`, `pkg/plugin/discovery`, `pkg/oci/signature`, `pkg/version/resolver/oci` | Preserve release bytes, locks, policies, and immutable plugin publication | +| File consumers | `pkg/generator/source/resolver.go`, `pkg/vendoring/install/fetch.go`, `pkg/provisioner/source/vendor.go` | Preserve existing source resolution and extraction | +| CI artifacts | `pkg/ci/artifact/{backend,store,bundled_store,metadata,tar}.go` | Existing metadata validation requires stack/component/SHA; file entries omit modes | +| CI backends | `pkg/ci/artifact/local`, `s3`, `github` | Existing local/dir stores a bundled stream, not an ordinary copied directory | +| Planfiles | `pkg/ci/plugins/terraform/planfile/adapter/store.go` | Existing store selection, key patterns, metadata, and integrity remain compatible | +| Configuration | `pkg/schema/schema.go`, `pkg/config/imports.go`, `pkg/config/load.go` | Preserve declaring-file origins for destination paths through config merges | +| Schema assets | `pkg/datafetcher/schema/atmos/config/1.0.json`, `stacks/stack-config/1.0.json`, `scaffold/scaffold-config/1.0.json` | Update the surfaces that actually admit the new configuration/steps | +| Steps/hooks | `pkg/schema/workflow.go`, `pkg/schema/task.go`, `pkg/runner/step/{registry,types,variables}.go`, `pkg/hooks/step_engine.go` | Step name, existing Artifacts declarations, and output display settings already have meanings | +| Packaging | `pkg/archive/{archive,walk,tar,zip,mtime}.go` | Reuse archive primitives; do not change existing archive-step defaults | +| Scaffold | `cmd/scaffold`, `pkg/generator`, `pkg/generator/source` | Preserve root README and generated-file behavior | +| Toolchain | `cmd/toolchain/install.go`, `pkg/toolchain/installer/{installer,download,asset,extract}.go`, `pkg/toolchain/lockfile` | Cached filename alone is not identity; URL sidecars and verification matter | +| Reporting | `pkg/ci/reporter.go`, `mode.go`, `templates/loader.go`, `pkg/component/container/imagesummary` | Existing gates, providers, template conventions, and masking are authoritative | + +## Dependency order and delivery units + +| ID | Deliverable | Depends on | Completion evidence | +| --- | --- | --- | --- | +| A00 | Compatibility fixtures and contract types | None | Baseline fixtures and digest vectors | +| A01 | Config/schema/registry foundation | A00 | Import-origin and validation tests | +| A02 | Shared ORAS transport and legacy adapters | A00 | Existing consumers pass conformance | +| A03 | Preparation, packaging, operation service | A00, A01 | Determinism, streaming, and conflict tests | +| A04 | Local directory/archive repositories | A03 | Real filesystem round trips | +| A05 | Shared blob I/O and S3 repository | A03, A04 | S3 and legacy CI adapter suites | +| A06 | OCI repositories and signature/lock parity | A02, A03, A04, A05 | Signed graph round trips and shared contract tests | +| A07 | Public artifact commands | A04, A05, A06 | CLI contract tests and documentation | +| A08 | Discovery/cache/catalogs | A02, A04, A05, A06, A07 | Offline and catalog independence tests | +| A09 | Step and hook integration | A03, A07 | CLI/step/hook parity | +| A10 | CI summaries and outputs | A07, A09 | One report per operation, including failures | +| A11 | Scaffold/component producers | A03, A06, A07 | Existing consumer integration tests | +| A12 | Verified toolchain producer/consumer | A03, A05, A06, A07 | Cold offline installation with verification | +| A13 | Native CI repository bridge and release gates | A05, A09, A10, A11, A12 | Planfile compatibility and full acceptance matrix | + +A02 and A01 can be developed independently after A00. Backend work can overlap after A03's +contracts are stable. No parallelism is required. Keep review units under the repository's file +limit; a work package can take several PRs. Behavior-preserving foundations use `no-release`; +each PR exposing new public behavior follows the repository's `minor` release documentation rules. + +## A00 — Freeze compatibility and introduce domain contracts + +- [ ] Inventory all direct go-containerregistry imports and network operations in current consumers. + Record which are transport, image construction, extraction, or verification; do not remove the + dependency while image-specific callers still need it. +- [ ] Add `pkg/artifact` for generic domain models, typed requests/results, repository registration, + and service orchestration. Do not reuse `schema.Artifacts`, which describes step output files. +- [ ] Establish the record, reference, operation, and capability contracts in [contracts](./contracts.md). + Add canonical serialization vectors before implementing any backend writer. +- [ ] Define shared Sign, Verify, Pin, and Lock contracts, evidence subjects, policy results, and + lock guarantees. Signatures and locks must not be implemented only as OCI-specific options. +- [ ] Capture deterministic fixtures for a plugin release index, its manifests/blobs, a plugin + catalog, signature material, signed native images, legacy CI metadata/tar, and single-root OCI layouts. + Preserve captured bytes as testdata; do not regenerate them to hide incompatible changes. +- [ ] Add shared sentinels for invalid config/reference, conflict, not found, unavailable publication, + unsupported capability, integrity failure, missing document, invalid packaging, and unsafe path. + Preserve existing sentinel wrapping in compatibility adapters. + +**Done when:** Every changed public contract has a test fixture, and a compatibility test can detect +a changed plugin manifest digest or planfile archive even before the implementation is replaced. + +## A01 — Configuration and backend registry + +- [ ] Add `ArtifactConfig` and `ArtifactRepositoryConfig` to `pkg/schema`, attached as + `AtmosConfiguration.Artifacts`. Include repositories, kind, identity, options, packaging, + permissions, optional catalog, and defaults.metadata; keep backend option decoding typed. +- [ ] Add global/repository verification policy and repository locking configuration from contracts. + Validate trust restrictions, delegated lock identities, cycles, and required lock scopes. +- [ ] Register repository factories by kind in `pkg/artifact`. Return capability descriptors without + network access. Reject duplicate kinds and unknown kinds; avoid a central backend switch. +- [ ] Normalize local destination paths while loading the declaring config, before imported values + lose their source directory. Track the origin of `options.path` independently of unrelated merged + repository fields. Do not expand key-pattern templates during general config rendering. +- [ ] Validate root/prefix containment, packaging matrix, octal strings, label values, catalog + references, and required backend options. Reject `permissions` for non-local archives. +- [ ] Resolve identity bindings lazily when a real operation requires credentials; no credentials + or destination directories should be created merely by describing config or running dry-run. +- [ ] Update embedded config JSON Schema and relevant stack/scaffold step schemas together with + Go types. Verify YAML, JSON, mapstructure, imports, and config describe round trips. + +**Done when:** The PRD's repository examples load unchanged, two same-named components can share one +collection under different logical paths, and imported local roots resolve to their declaring files. + +## A02 — ORAS transport behind shared OCI interfaces + +- [ ] Promote the existing pinned `oras.land/oras-go/v2` dependency to direct use; do not combine + this migration with an unrelated version upgrade. Consult documentation for that pinned version. +- [ ] Add OCI-standard descriptor/reference/blob interfaces alongside the existing Registry API. + Use `opencontainers/image-spec` descriptors and `go-digest`, not ORAS remote.Repository or + go-containerregistry images in the new public transport boundary. +- [ ] Implement resolve, fetch, exists, push, tag, list tags, delete manifest, graph copy, layout + storage, and referrer discovery using ORAS. Keep bounded reads/digest verification at the boundary. +- [ ] Adapt existing authentication resolution to ORAS credentials. Preserve explicit identity + failure behavior, ambient precedence, host/realm scoping, token refresh, and TLS/proxy options. + Source and destination clients must not share a credential cache across different identity scopes. +- [ ] Bridge old Registry methods onto the new transport while legacy callers still construct + go-containerregistry images/indexes. Serialize their existing raw bytes; never rebuild an old + release with ORAS PackManifest just to transfer it. +- [ ] Migrate plugin release/catalog reads and writes, discovery, version resolution, signatures, + and shared source pulls in small groups. Preserve `ResolveImage`, `PinDigest`, and `ProcessImage`. +- [ ] Add multi-reference layout operations without weakening `LoadIndex`'s existing single-release + validation. Preserve OCI 1.1 referrers and the existing fallback behavior required by signatures. +- [ ] Reconcile writes by resolving the destination after ambiguous failures; do not blindly retry + mutations or downgrade to anonymous authentication. Cancellation must stop graph traversal. +- [ ] Extract reusable verifier/signer primitives into `pkg/signature`, retaining compatibility + wrappers and existing plugin trust requirements. Keep attachment transport in the adapters. + +**Done when:** All A00 OCI fixtures retain their digest and verification result through registry → +layout → second registry movement with distinct credentials and the source switched off. + +## A03 — Artifact preparation and shared operation service + +- [ ] Implement a producer registry. Start with generic files; kind-specific producers add validation + and exclusions later. Preparation resolves name/version, labels, documents, inventory, and digests. +- [ ] Snapshot input into bounded disk staging before committing publication. Detect files changing + during capture and fail instead of publishing an inventory that describes different bytes. + Close readers and remove owned temporary data on success, error, or cancellation. +- [ ] Reuse archive traversal/writers with explicit deterministic options: sorted entries, normalized + incidental timestamps and ownership, preserved ordinary modes, and safe symlink handling. + Existing `type: archive` behavior remains unchanged. +- [ ] Recognize the reserved document paths and explicit overrides. Validate explicitly requested + files, compute document descriptors, and attach generated metadata without checksum recursion. +- [ ] Build publish/pull/copy/inspect/list/search/delete/show orchestration over Repository interfaces. + Pin mutable references once, check capabilities before mutation, and return partial operation + results alongside errors so reporting can describe completed work. +- [ ] Implement intent comparison/no-op, force conflicts, plugin immutability admission, operation + timestamps, publication journals, and retry reconciliation from the contracts companion. +- [ ] Call repository Sign/Verify/Pin/Lock methods through common orchestration. Enforce source and + destination trust requirements before copy/consumption; preserve evidence for the actual subject. + Reject insufficient lock guarantees and retain explicit ownership across retries. +- [ ] Implement extraction staging and replacement. Reject traversal and escaping symlinks; never + merge a newly pulled tree over old files or delete files outside the explicit output destination. +- [ ] Expose dry-run preparation results without calling repository mutation/auth operations. + +**Done when:** One service test suite drives all entry points; unchanged input yields the same +content identity, unchanged publication is a no-op, and failed reads/writes never report success. + +## A04 — Local directory and archive repositories + +- [ ] Implement separate `local/dir` and `local/archive` factories in `pkg/artifact/local`. + Do not redirect the existing CI `local/dir` name to the new copied-directory behavior. +- [ ] Use stable-root reference records, immutable metadata records, and adjacent document files + as specified in contracts. Directory payloads remain ordinary copied files; archives contain + one artifact and have adjacent metadata/documents plus embedded portable copies. +- [ ] Hold per-reference and resolved-location locks over comparison, preparation, and commit. + Publish completion last; use same-filesystem staging and atomic file replacement where possible. +- [ ] Persist dated resolved locations. A subsequent read/delete must not rerender the key pattern + with a new timestamp. Detect two references attempting to own the same resolved path. +- [ ] Apply quoted octal archive-file permissions separately from payload modes. Verify behavior + on Unix and document the platform capability error for explicitly unsupported requirements. +- [ ] Implement metadata-only inspect/show, record enumeration, verified digest reads, and scoped + deletion. Keep private managed records out of the user's pulled payload. +- [ ] Implement detached signature storage, shared signer/verifier delegation, immutable pinning, + and filesystem LockHandles. Signature additions must not mutate their signed Record. + +**Done when:** Real temporary-directory tests cover every allowed packaging format, dated roots, +ordinary executable files, safe links, interrupted writes, output replacement, and missing sidecars. + +## A05 — Share blob transport with CI and add S3 + +- [ ] Extract transport-only local/S3 object I/O from `pkg/ci/artifact` into `pkg/artifact/blob`. + The low-level API handles streams, size, headers/tags, conditional writes, listing, and deletion; + it must not require CI Metadata. Keep legacy metadata encoding in CI adapters. +- [ ] Preserve existing CI factories and wire formats using adapters over that transport. Keep + GitHub artifact storage CI-specific and avoid a reverse dependency from artifact core to CI. +- [ ] Implement `aws/s3` with archive payloads or per-file objects, stable-root records, document + objects, configured prefix/key_pattern, and explicit object tags. Copy the configured tags to + all objects in the publication; portable labels remain independent. +- [ ] Use streaming/multipart transfers and bounded worker concurrency. Abort multipart uploads + owned by a canceled operation. Handle pagination, zero-length files, and endpoint overrides. +- [ ] Use S3 conditional writes for completion/reference updates where supported. An endpoint that + cannot enforce a needed condition must fail that operation clearly, not silently overwrite. +- [ ] Implement digest-addressed evidence objects, Pin, and persistent owner-checked S3 lock claims. + Keep crash recovery explicit; no automatic lease expiry without fencing and no Object Lock confusion. +- [ ] Preserve lazy identity resolution and endpoint/TLS behavior. New generic repository identity + failures must fail closed; do not copy the legacy missing-resolver ambient fallback into it. + +**Done when:** An S3 fake proves request ordering, tags, pagination, and conditional conflicts; +an opt-in S3-compatible emulator proves archive/files round trips and existing planfile compatibility. + +## A06 — OCI collection repositories and foreign graphs + +- [ ] Implement registry and layout factories over A02. Map root/prefix/name/version consistently; + expose capability differences without requiring users to configure each artifact name separately. +- [ ] Assemble new Atmos manifests with the PRD media types, config metadata, deterministic payload + descriptors, and separate README/CHANGELOG blobs. Supply the captured creation time explicitly. +- [ ] Support selective config/document fetches and multi-reference layout enumeration. Never fetch + payload layers merely to inspect labels or show a document. +- [ ] Implement OCI content-graph import/export and cross-backend transport through a standard OCI + layout payload. Keep original manifests and related required signature material byte-identical. +- [ ] Distinguish generic metadata from foreign OCI image/plugin schemas. Inspect foreign objects + without pretending they have an Atmos config blob; do not repackage them on copy. +- [ ] Implement the same Sign/Verify/Pin/Lock contract as local/S3, using OCI attachments and the + shared verifier. Bind image signatures to the original manifest/index; bind portable Record + signatures through the addressed config descriptor. Support delegated repository-scope locks. +- [ ] Add genuine keyless and configured-public-key verification tests. Missing, tampered, + wrong-subject, wrong-issuer, untrusted-key, and expired/missing offline trust must fail policy. +- [ ] Enforce plugin-format immutability for generic publication/copy before a tag can be overwritten. + Preserve caller policy/verification checks on every read and write path. +- [ ] Implement safe deletion semantics: no silent deletion of sibling tags that share a manifest. + Report unsupported tag-only deletion when the registry cannot provide the required semantics. + +**Done when:** Generic artifacts and legacy plugins coexist beneath one collection prefix, a +multi-platform image survives S3/layout transfer, and metadata/document-only request budgets pass. + +## A07 — Command space + +- [ ] Add a `cmd/artifacts` CommandProvider with publish, pull, copy, inspect, list, search, delete, + sign, verify, and show readme/changelog subcommands through the existing command registry. +- [ ] Implement the exact flag/step mapping in contracts using StandardParser and normal Atmos + configuration precedence. Reuse list/table formatting and the existing label parser. +- [ ] Keep data on stdout, progress on stderr, and errors in the established structured error path. + Emit result envelopes on failure when structured output is requested; retain a nonzero exit code. +- [ ] Embed command usage Markdown and add website command/configuration pages. Label all new + functionality accurately; the PRD is not a substitute for shipped command help. +- [ ] Test with `cmd.NewTestKit(t)` and injected service dependencies; add real temporary-local CLI + tests so command validation, serialization, and service wiring are exercised together. + +**Done when:** Every PRD command example has a tested counterpart and a documented error path; +no command invents a default repository, `latest`, catalog mutation, or interactive CI prompt. + +## A08 — Discovery cache and optional catalogs + +- [ ] Extract reusable source-scoped snapshot/cache mechanics from `pkg/plugin/discovery` into a + neutral `pkg/discoverycache` package; retain plugin-specific schemas, keys, defaults, and policy. + Prove plugin cache compatibility before switching its consumer to shared mechanics. +- [ ] Enumerate managed local/S3 records and OCI-layout refs. For OCI registries use supported + repository enumeration scoped to prefix, or the configured catalog; explicit-name tag listing + works without either. An unsupported collection scan fails, not an empty successful result. +- [ ] Apply exact-match AND label filtering to published-version metadata and substring matching + to name/description. Sort results deterministically and retain repository alias and pinned digest. +- [ ] Implement the 15-minute cache, explicit refresh, offline semantics, credential-scope keys, + policy reevaluation, bounded reads, and atomic local snapshot replacement from the PRD. +- [ ] Add a separate `AtmosArtifactCatalog` manifest using `pkg/manifest`. Carry artifact kind, + source, versions, digests, labels, and document descriptors; do not impose plugin SemVer rules. +- [ ] Add catalog build/publish/add/remove/sync commands specified in contracts. Catalog writes + require one externally serialized publisher; release publication never invokes these commands. +- [ ] Treat catalog payloads as discoverable data, never trust grants. Validate every referenced + location against the destination repository and current auth/source policy before access. + +**Done when:** Multiple configured repositories search together, cache hits fetch no payloads, +catalog changes never rewrite artifacts, and incomplete/unauthorized results are never presented as complete. + +## A09 — Workflow and hook integration + +- [ ] Add `ArtifactHandler` to `pkg/runner/step`; implement Validate, Execute, KnownFields, and + RequiresTTY=false. Use flat step fields and the existing kind: step hook bridge. +- [ ] Add shared schema fields without repurposing `name`, `kind`, `output`, or `Artifacts`. + Cover WorkflowStep, Task conversion, custom-command decoding, script step dispatch, and hook `with`. +- [ ] Resolve source/destination paths through the runner's existing working-directory helpers; + resolve repository roots independently from their config origin. +- [ ] Return typed operation results in a documented StepResult.Metadata entry and strings in + named Outputs; use the same payload as CLI JSON. Return partial results even on service error. +- [ ] Retain a preparation/operation identity across wrapper-level retries so dates, versions, and + reports do not change on every attempt. Do not multiply the runner and transport retry budgets. +- [ ] Exercise before/after Terraform hooks, workflow execution, custom commands, and direct script + dispatch. Preserve conditions, continue/on_failure, and cancellation semantics. + +**Done when:** The same publication produces the same metadata and result across CLI, workflow, +and hook calls, including retry/failure cases; no shelling out to `atmos artifacts` is required. + +## A10 — CI summaries and machine outputs + +- [ ] Add `pkg/artifact/report` as a renderer over typed results, with no repository mutations or + extra inspection requests. Add embedded templates and CITemplatesConfig.Artifacts overrides. +- [ ] Use the existing template loader's explicit/convention/default order, with an error for a + missing explicit override. Route completed Markdown through `ci.Reporter.Summary`. +- [ ] Collect at execution boundaries: CLI owns standalone reports; the runner owns step reports + after retries; hooks reuse that runner collector and retain an empty duplicate hook body. + Pass collectors explicitly through runner services/options, never through context values. +- [ ] Deduplicate by logical operation ID and retain the final attempt. Buffer concurrent results, + order by scheduled step/operation position, and flush from a deferred finalizer on error too. +- [ ] Render outcomes, destinations, digest types, sizes, duration, and collapsible details. Escape + Markdown, mask secrets, avoid signed URLs, and enforce provider byte limits with visible truncation. +- [ ] Include signature subject, verified signer/key claims, verification status, and actual lock + guarantee. Distinguish signed from verified and a content checksum from publisher authentication. +- [ ] Wire `ref`, `location`, digest fields, size, and outcome to existing CI output mechanisms when + enabled. Keep JSON/YAML stdout valid and leave PR comments/checks opt-in through their own features. + +**Done when:** Success, no-op, force replacement, partial failure, canceled siblings, disabled +reporting, template failure, and provider failure all have deterministic tests and correct exit status. + +## A11 — Scaffold and component publishing + +- [ ] Add a scaffold producer and `atmos scaffold publish` provider delegating to the generic service. + Validate scaffold configuration without generating output or executing scaffold hooks. +- [ ] Add a component/terraform producer with visible default exclusions for `.terraform/**`, + `*.tfstate`, `*.tfstate.*`, `*.tfplan`, and `*.tfplan.*`; retain `.terraform.lock.hcl` and scripts. + Preserve explicit dry-run inventory and do not apply these exclusions to generic backups. +- [ ] Exclude reserved package metadata from generated scaffold output; preserve root README + rendering and user-selected ordinary payload files. Cover existing generator source strategies. +- [ ] Teach existing OCI materializers to select the Atmos payload descriptor rather than trying + to extract every layer, while keeping legacy component/scaffold layer heuristics intact. +- [ ] Prove published components work through existing source provisioning and scaffolds through + existing generation/update flows, including documentation without pulling the payload. + +**Done when:** A component with an executable helper and a scaffold with both README roles round-trip +without leaked distribution files or altered generated output. Other configuration/module trees +remain publishable as generic files; native cache/export adapters are not silently added here. + +## A12 — Verified toolchain package integration + +- [ ] Add `atmos toolchain publish` and explicit artifact-repository installation selection as + specified in contracts. Reuse tool resolution and lockfile validation; do not introduce another + tool registry format or automatic upstream fallback. +- [ ] Package the original upstream asset with its resolved install definition, platform identity, + relevant lock entry, and available checksum/signature evidence. Retain original asset bytes. +- [ ] Verification metadata is evidence to recheck, not a boolean trust grant. Require pinned + checksums and rerun applicable verification policy before publication and installation. +- [ ] Inject an asset source into the existing installer before it consults network registries. + Feed the verified original package to normal extraction/install/PATH logic. Update URL cache + sidecars transactionally and retain the upstream identity separately from artifact location. +- [ ] Keep adapter wiring outside artifact core and installer core to avoid cyclic imports. + Fail explicitly when an offline verification dependency or install definition is absent. + +**Done when:** With empty toolchain caches, blocked upstream HTTP, and a pinned local/OCI/S3 +artifact source, install the fixture tool and execute it; a tampered asset or incompatible platform fails. + +## A13 — CI compatibility bridge and release completion + +- [ ] Add explicit repository selection to native CI planfile storage without changing legacy + `planfiles.stores` selection. Follow the selector contract; never guess a repository from a + missing legacy store or silently fall back after download-not-found. +- [ ] Adapt CI bundle metadata to generic records while preserving old metadata validation and + serialized sidecars for existing callers. Add file-mode fields with a legacy-compatible default; + preserve existing planfile tar bytes when no mode is supplied. +- [ ] Complete all backend/entry-point scenarios in [validation](./validation.md), including restore + with upstream disabled, negative auth tests, and CI result parity. +- [ ] Update CLI/config/hook docs, examples, schema tests, and generated mocks. Record the exact + semantics of unsupported capabilities and offline data requirements. +- [ ] Run repository checks, targeted suites, full regression checks, and website build; inspect + generated changes. Follow the release skill for labels, blog, roadmap, and milestone requirements. + +**Done when:** The validation matrix is green for new behavior and compatibility, required docs are +published with the feature, and no deferred capability is presented as implemented. + +## Follow-on work queue + +Keep these outside V1 and link future designs back to the use-case inventory: Terraform provider +and module cache export/restore with mirror indexes; native stack/Atmos-config dependency closure; +installed toolchain snapshots; bulk image mirroring/platform selection; complete air-gap bundles; +additional transports; SBOM generation and aggregate inventories; retention/prune; structured +changelog aggregation. Generic file upload alone is not completion evidence for these integrations. diff --git a/docs/prd/artifacts/validation.md b/docs/prd/artifacts/validation.md new file mode 100644 index 00000000000..2028fc2e81b --- /dev/null +++ b/docs/prd/artifacts/validation.md @@ -0,0 +1,204 @@ +# Atmos Artifacts validation and agent handoff + +**Status:** Required future implementation checks; no box is marked complete by writing this plan. +Read the [work packages](./implementation.md), [contracts](./contracts.md), and [PRD](../artifacts.md). + +## Test strategy and fixtures + +Use small real files, temporary repositories, deterministic clocks, injected failures, and bounded +fake transports. Generate interface mocks with the repository's mockgen workflow. Favor observable +behavior over tests that repeat implementation details. Do not manually edit existing golden files. + +Create reusable fixtures in the owning package testdata, not under unrelated existing golden trees: + +- File package: nested files, an executable helper, empty file/directory, confined symlink, Unicode + filename, root README, and distinct package README/CHANGELOG. +- Invalid package: traversal, symlink escape, duplicate normalized paths, conflicting reserved + metadata, input changing during capture, and unsupported special-file entries. +- Generic Record: canonical byte vectors and expected content/intent/record digests, including + alternate map insertion order and fixed creation times. +- OCI graph: tiny single-platform image and two-platform index with distinguishable child bytes. +- Legacy plugin: existing format manifests/index, executable fixture, catalog, lock, and genuine + Sigstore evidence. Keep expected digest values stable through transport migration. +- Signed generic artifact/native image: distinct subject types, genuine trusted and untrusted + signer fixtures, public-key-only evidence, incorrect issuer, and mismatched subject digest. +- CI planfile: pre-change tar and metadata plus old store/default/priority configurations. +- Toolchain: tiny original release archive/raw executable, two platform definitions, lock entries, + URL-cache sidecars, checksum/signature evidence, and an executable version-reporting fixture. + +Reuse `pkg/oci/ocitest` and `pkg/oci/signature/signaturetest` rather than live public registries or +identity providers. Fake S3 must model pagination, conditional state, multipart cancellation, object +tags, and response ambiguity; a map that always succeeds does not test publication correctness. +Opt-in emulators supplement these tests and must not be required for ordinary unit suites. + +## Required conformance matrix + +Run the shared repository conformance suite against each registered backend, with capability-specific +assertions for supported operations and explicit errors for unsupported guarantees. + +| Behavior | Local directory | Local archive | S3 | OCI layout | OCI registry | +| --- | --- | --- | --- | --- | --- | +| Multiple names/versions per configured root | Required | Required | Required | Required | Required | +| Publish, pull, inspect, show, delete | Required | Required | Required | Required | Required within declared delete capability | +| Packaging | Copied files | tar/tgz/zip | Files and tar/tgz/zip | Packaged files and native graphs | Packaged files and native graphs | +| Metadata/document reads without main payload | Required | With sidecars | Required | Required | Required | +| Collection enumeration | Managed refs | Managed refs | Paginated managed refs | Layout refs | Native enumeration or catalog; error otherwise | +| Generic Sign/Verify/Pin | Detached evidence | Detached/portable evidence | Evidence objects | Layout evidence | Referrer evidence | +| Lock | Host filesystem | Host filesystem | Repository owner claim | Host filesystem | Host guard; repository scope through configured delegate | +| Missing required lock/evidence capability | Error | Error | Error | Error | Error | + +Test local directory ↔ archive, local ↔ S3 in both packaging modes, local ↔ OCI, registry ↔ layout, +registry ↔ second registry, and native OCI graph ↔ S3/layout carrier. Byte-identical native graphs +retain resolved digests; generic repackaging retains Record/content identity but may change the +representation/OCI envelope digest. Assert both rather than assuming every digest is interchangeable. + +## Acceptance scenarios + +### Configuration and interface parity + +- [ ] Load every PRD YAML example; validate identical values across YAML/JSON/mapstructure. +- [ ] Load two imported configs with different source directories and merged repository fields; + each destination path uses the declaration that supplied that path. +- [ ] A remote config must not accidentally make an ephemeral download directory a write root; + until local anchoring is explicit, require a locally declared/absolute destination override. +- [ ] Reject unknown kinds, invalid option combinations, traversal, catalog/identity mismatch, + unquoted numeric permissions, malformed labels, and delegated lock cycles before mutation. +- [ ] Configuration inspection and dry-run perform no credential requests, destination directory + creation, signature publication, or remote object writes. +- [ ] Reuse one alias for components, scaffolds, configuration trees, and generic backups. + +### Publication, integrity, and recovery + +- [ ] Repeated preparation produces canonical records/inventory and deterministic payloads. +- [ ] An identical republish is unchanged even when invocation time/run metadata differ. +- [ ] Changed content, labels, documents, or modes conflict without force; plugin releases remain + immutable with force. Existing/incoming plugin formats both trigger the guard. +- [ ] Resolve a tag, then move it during a read/copy; consume only the originally pinned subject. +- [ ] Inject interruption before payload, during payload, after metadata, and after server-side + commit before its response; retry reconciles correctly and never advertises partial success. +- [ ] Inject two writers for one ref and two refs with a colliding key_pattern. Verify ownership + and conditional guards; never silently bind one reference to another artifact's payload. +- [ ] Date rollover and wrapper-level retries reuse captured time and resolved locations. +- [ ] Pull replacement removes stale destination files only inside the requested output root. +- [ ] Tampered metadata, payload, document, or OCI blob fails its declared integrity check. +- [ ] Direct digest reads never return replacement bytes under an old digest; unavailable retained + records report failure. Scope deletion to the selected artifact and its exclusively owned files. +- [ ] Archive/file member permissions remain distinct; executable bits survive. Oversize records, + unsafe links, unsupported target modes, and extraction errors are explicit. + +### Signatures and coordination + +- [ ] Every backend's Sign/Verify/Pin/Lock implementation passes the same interface tests; backend + differences are represented by capabilities, not omitted operations that claim success. +- [ ] Sign a portable Record, copy it through local/archive/S3/OCI packaging, and verify the same + signed subject after restoring it. Attachments bind to the correct record/config. +- [ ] Copy a signed native image/index through an offline carrier and a second registry, disable + the source, and verify the original root digest and preserved evidence at the destination. +- [ ] Reject wrong digest, wrong issuer, untrusted identity/key, corrupt signature, absent required + evidence, unsupported format, expired trust, and missing required transparency/signing-time proof. +- [ ] Public-key verification cannot satisfy an OIDC identity policy. Existing plugin rejection of + public-key-only bundles under identity policy remains unchanged. +- [ ] Signing alone reports signed, not policy-verified; plain hashing reports integrity, not signer + authenticity. `verify` without a usable trust configuration does not return a green result. +- [ ] Independent source/destination policies both apply to copy; repository overrides and force + cannot weaken governing requirements. Cached verification results are reevaluated under new policy. +- [ ] Preserve a signed multi-platform root when selecting a child; a reconstructed root must not + inherit an original signature claim. Required evidence is fetched before materialization. +- [ ] Offline verification uses only cached/provisioned trust and evidence, with a transport that + fails any network request. Missing trusted roots cannot be filled from the artifact itself. +- [ ] S3 owner claims reject competing owners and wrong-token release; they never expire into an + unfenced takeover. Local guards exclude competing processes and release when their owner exits. +- [ ] OCI host guards are not reported as repository-scope locks; a required distributed guard + fails without a capable delegate. Delegated lock access uses its own identity. +- [ ] Canceled lock acquisition does not leak ownership; pending publication recovery requires + valid ownership and matching intent. Document operator recovery of abandoned S3 claims. +- [ ] Pins persist immutable digests/evidence descriptors without changing existing consumer lock + schemas or granting trust merely because the lockfile says a prior run succeeded. + +### Discovery and catalogs + +- [ ] Label pairs use AND semantics on artifact-version labels, not S3 tags or repository defaults + retroactively applied to older artifacts. Keep aliases visible and sorting deterministic. +- [ ] Named OCI tag listing works without a catalog; unsupported collection discovery returns a + capability error. Add no second repository list in configuration. +- [ ] Fresh-cache search performs zero registry requests; refresh reuses metadata by digest and + removes disappeared refs only after a complete refresh succeeds. +- [ ] Offline cache works without payload downloads, rejects corrupt/missing snapshots and refresh, + and does not cross auth realms/host bindings or bypass current policy. +- [ ] Catalog build reads metadata only. Publish/sign/delete of an artifact never implicitly edit + its catalog. Catalog removal unlists without deleting content or rewriting locks. +- [ ] Concurrent catalog updates expose stale revision/digest conflicts; serialized workflows + read latest inside the guard. Do not claim a preflight check alone prevents races. +- [ ] Plugin discovery/cache behavior stays compatible after common mechanics are extracted. + +### Producers and existing consumers + +- [ ] Scaffold generation/update preserves root README behavior and excludes reserved distribution + metadata. `scaffold publish` validates without executing generation hooks. +- [ ] Component source provisioning reads new Atmos payload descriptors and old formats; helpers + remain executable, Terraform transient data is excluded, and generic backups retain requested files. +- [ ] Toolchain publication verifies the original package against lock data. Offline installation + succeeds with empty caches and blocked upstreams, using the normal extractor/installer. +- [ ] Reject toolchain tampering, wrong platform, missing install metadata/evidence, and cache URL + collisions. Recheck configured verification; a recorded "verified" string is not sufficient. +- [ ] Existing CI stores retain metadata validation, tar defaults, key patterns, selection order, + and not-found behavior. Only explicit `artifact/repository` opts into the new repository bridge. +- [ ] The planfile adapter preserves original key/query information and does not overwrite unrelated + generic or plugin artifacts. Existing CI fixtures continue reading through their old adapters. + +### CLI, steps, hooks, and summaries + +- [ ] Run matching operations through CLI, workflow, custom command, script dispatch, and kind: step + hooks; verify identical typed results, named outputs, source paths, and repository roots. +- [ ] Retry, on_failure/continue, cancellation, and failure after some completed operations preserve + operation identity and report the correct final outcome exactly once. +- [ ] Summary rendering covers published, unchanged, replaced, downloaded, copied, deleted, signed, + verified, and failed outcomes; includes actual signature subjects and lock guarantees. +- [ ] Sequential/parallel/nested steps produce one deterministically ordered report per operation; + the hook bridge does not render a second copy. Existing unrelated job summary content remains. +- [ ] CI disabled, summaries disabled, output variables disabled, and explicit CI-mode overrides + honor existing gates. Disabled summaries perform no extra metadata/evidence fetches. +- [ ] Custom template precedence, missing override errors, secret masking, Markdown escaping, + provider size limits, and safe links work without corrupting JSON/YAML stdout. +- [ ] Reporting errors preserve artifact exit status; operation/verification errors remain failures. + +## Commands and validation gates for implementation + +Run package-scoped checks after each change; the following package paths include proposed packages +and are executable once those packages exist. Do not run unimplemented paths and count them as skipped. + +```shell +go test -short ./pkg/artifact/... ./pkg/signature/... +go test -short ./pkg/oci/... ./pkg/plugin/oci/... ./pkg/plugin/catalog/... ./pkg/plugin/discovery/... +go test -short ./pkg/ci/artifact/... ./pkg/ci/plugins/terraform/planfile/... +go test -short ./cmd/artifacts/... ./cmd/scaffold/... ./pkg/generator/... +go test -short ./pkg/runner/step/... ./pkg/hooks/... ./pkg/ci/... +go test -short ./pkg/toolchain/... ./cmd/toolchain/... +go test -race -short ./pkg/artifact/... ./pkg/discoverycache/... +atmos build +atmos lint --changed +atmos test +``` + +Also run the website build for shipped command/config docs and the opt-in OCI/S3 emulator round +trips. Use current repository instructions for regeneration and CI checks. Target the required +patch coverage with behavioral tests; do not add redundant tests solely to inflate coverage. + +Known prior workspace observations, not acceptance waivers: a documentation-only baseline build +passed, while a broad test invocation encountered missing Packer, plugin/formatting failures and +a package timeout. Recheck the current baseline before implementation, record exact failures, and +distinguish environment/baseline failures from regressions. New artifact tests must still pass. + +## Completion record for the next agent + +For each A00–A13 package, record: + +- [ ] Implementation commit(s), changed interfaces, and migration/compatibility notes. +- [ ] Exact successful test/check commands, relevant fixtures, and skipped emulator prerequisites. +- [ ] Failure reproduction, baseline comparison, and owner for any unresolved check. +- [ ] Public docs/schema/help changes and release-label implications where applicable. +- [ ] Remaining work package IDs and the next concrete action. + +V1 is complete only when every required backend and public surface passes this matrix and all +claimed offline flows work with fresh local state and unavailable upstreams. An upload success, +metadata search result, detached signature file, or apparent lock object alone is not that proof.