From 408a92069a6f7218d079d9ae84806c2ed5c4b0fd Mon Sep 17 00:00:00 2001 From: Erik Osterman Date: Fri, 30 Oct 2026 12:16:52 -0500 Subject: [PATCH 1/7] docs: define artifacts repositories and offline workflows --- docs/prd/artifacts.md | 599 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 599 insertions(+) create mode 100644 docs/prd/artifacts.md diff --git a/docs/prd/artifacts.md b/docs/prd/artifacts.md new file mode 100644 index 00000000000..0d8670e5cba --- /dev/null +++ b/docs/prd/artifacts.md @@ -0,0 +1,599 @@ +# Atmos Artifacts: Repositories, Publishing, Discovery, and Automation + +**Status:** Proposed. This document specifies future capabilities; its configuration and command +examples are not a claim that those capabilities are implemented. This change is documentation only. + +## Problem and outcome + +Atmos needs one way to publish, retrieve, inspect, and discover the assets its users produce and +consume: components, scaffolds, backups, verified tool packages, CI outputs, and OCI content. +Users should configure one destination and publish many named assets into it. Switching between +OCI, S3, directories, and archives should change the selected repository, not the publishing workflow. + +The same operations must work interactively, in CI, in workflows, and as lifecycle hooks. CI runs +must leave useful job summaries rather than requiring users to reconstruct publication results +from logs. Package README and CHANGELOG documents must be available without downloading a package. + +This restores and revises the earlier `docs/prd/artifacts.md` draft preserved at commit +`2f81e11338d28a74c2a7fdf18a52d3cd5136ac4a`. That draft focused on container images and +`containers/image/v5`. This proposal broadens V1 to general artifacts and selects `oras-go/v2` +for shared OCI transport. Deferred requirements from that draft are retained below. + +The scaffold documentation requirements also reflect the Erik/Judith conversation supplied during +planning: package documentation is distinct from generated repository documentation, and release +notes should eventually support template-update PRs. Changelog aggregation is deferred. + +## Use-case inventory + +Artifacts are the distribution and restoration layer for the following use cases. Offline and +air-gapped operation is a cross-cutting goal: content must remain usable after its upstream is unavailable. + +| Use case | Content to store | Intended consumption or restoration | +| --- | --- | --- | +| Scaffolds and templates | Template trees, scaffold definitions, package README/CHANGELOG, and supporting files | Generate or update projects from a pinned template release | +| Components | Terraform, Helmfile, and other component source trees, scripts, and associated files | Provision components from a controlled repository | +| Terraform modules | Root or child module source, origin/version metadata, and checksums | Resolve pinned module dependencies without the original Git/registry source | +| Stack configuration | Stack manifests, catalogs, mixins, imports, and their dependencies | Load and compose stacks from published configuration | +| Atmos configuration | `atmos.yaml`, imported configuration, custom commands, workflows, policies, and supporting scripts | Bootstrap a consistent Atmos environment and reuse organizational automation | +| Toolchain packages | Verified original tool releases, platform information, lock metadata, and verification evidence | Install the required tools without contacting upstream release servers | +| Toolchain caches | Selected cached downloads and, eventually, installed tool trees with platform/version metadata | Warm or restore a compatible toolchain cache for offline execution | +| Terraform provider cache | Provider packages, origin addresses, versions, platforms, checksums, and mirror index metadata | Export Atmos-managed provider caches and restore a usable offline provider mirror | +| Terraform module cache | Cached module sources, registry resolution metadata, versions, and integrity records | Export Atmos-managed module caches and restore offline module resolution | +| OCI images and artifacts | Complete manifests, indexes, blobs, platforms, and required related artifacts | Run or consume pinned OCI content from an internal registry or local layout | +| Vendored dependencies | Git/HTTP source snapshots and dependency provenance | Hydrate the same source trees without contacting upstream sources | +| CI outputs | Planfiles, build outputs, generated configuration, reports, and run provenance | Exchange verified results between jobs and retain inspectable outputs | +| Backups | Selected files/directories and restoration metadata | Restore a copied directory or a versioned archive from local storage or S3 | +| Atmos plugins | Existing plugin release packages and associated verification material | Distribute/install existing plugin formats while preserving immutable-release policy | +| SBOMs and attestations | Inventory documents, provenance, signatures, and subject relationships | Inspect and verify artifacts and their dependencies in disconnected environments | +| Air-gap distribution | A selected closure of configuration, components, modules, providers, tools, images, and metadata | Transfer into a disconnected environment and restore everything needed for a defined workflow | + +This inventory defines the product vision; delivery phases are specified below. Cache restoration must +preserve indexes, locks, checksums, and dependency relationships and prove consumption with upstream disabled. + +## Model and V1 boundaries + +- **Repository:** A configured collection root holding many named artifacts, possibly of different + kinds. It is a destination and discovery source, not a declaration of one package. +- **Artifact:** Versioned content with portable metadata, labels, document descriptors, and digests. +- **Artifact kind:** Meaning of the content, such as `component/terraform`, `scaffold`, + `toolchain/package`, or generic `files`. It is independent of the repository kind. +- **Packaging:** Representation of a file payload: individual files or an archive. +- **Reference:** Repository alias plus artifact name and version, or name and immutable digest. +- **Catalog:** An optional maintained discovery snapshot. It is not necessary for direct reads. + +V1 includes publishing, pulling, copying, inspection, listing, search, deletion, package documents, +label filtering, optional catalogs, hooks, and CI reporting. OCI manifests/indexes are transferred +as content graphs, not interpreted as ordinary directory archives. + +Artifacts are not components. Existing top-level `stores` remain key-value and secret stores. +An archive in this proposal packages one artifact; it is not a snapshot of an entire repository. + +## Public configuration + +`artifacts.repositories` is the single list of destinations to use and discover. Do not add another +`discovery.repositories` list or require one configuration entry per scaffold or component. + +```yaml +artifacts: + repositories: + shared: + kind: oci/registry + identity: registry-publisher + options: + registry: ghcr.io + prefix: acme/atmos + defaults: + metadata: + labels: + team: platform + + backups: + kind: aws/s3 + identity: backup-writer + packaging: tgz + options: + bucket: acme-artifacts + prefix: atmos + region: us-east-2 + tags: + purpose: backup + + local: + kind: local/dir + options: + path: .atmos/artifacts + + archives: + kind: local/archive + packaging: tgz + permissions: "0600" + options: + path: .atmos/backups + key_pattern: '{{ .CreatedAt | date "2006/01/02" }}/{{ .Name }}/{{ .Version }}.tgz' + + offline: + kind: oci/layout + options: + path: .atmos/oci +``` + +`identity` references Atmos auth identities; credentials do not belong in repository options. +`prefix` is an Atmos collection-root convention, not an ORAS namespace field. ORAS addresses a +registry, an OCI repository, and a tag or digest. Atmos maps a collection and artifact name onto +that OCI repository. Local backends use `path` for their destination root. + +### Uniform addressing + +```shell +atmos artifacts publish ./components/terraform/vpc \ + --repository shared \ + --name components/terraform/vpc \ + --version 1.2.0 \ + --kind component/terraform +``` + +Changing only `--repository` selects another destination: + +| Repository | Resolved payload location with the configuration above | +| --- | --- | +| `shared` | `ghcr.io/acme/atmos/components/terraform/vpc:1.2.0` | +| `backups` | `s3://acme-artifacts/atmos/components/terraform/vpc/1.2.0.tgz` | +| `local` | `.atmos/artifacts/components/terraform/vpc/1.2.0/` | +| `archives` | `.atmos/backups/2026/10/30/components/terraform/vpc/1.2.0.tgz` for that creation date | +| `offline` | A named reference in the OCI layout at `.atmos/oci` | + +Metadata and document sidecars accompany payloads where required; the table shows payload locations. +Names may contain relative slash-separated segments. Reject absolute paths, traversal, empty +segments, and backend-invalid names or versions before uploading; do not silently rename them. +Versions are explicit labels, including CI build identifiers, rather than universally requiring +SemVer. Kind-specific rules, including plugin release rules, still apply. + +Publishing takes a local source argument. Explicit `--name` wins; otherwise use available package +metadata, then the source basename. Explicit `--version` wins over package metadata; fail when +neither supplies a version. Do not invent `latest`. Reads use `name:version` or `name@sha256:...`. +Require `--repository` for single-repository operations; list/search may span configured repositories. +Resolve mutable references once per operation and pin subsequent reads to that digest. Digest +references select OCI manifest/index digests on OCI backends and immutable record digests on +local/S3 backends; portable content digests are reported separately. + +### Packaging, destinations, and permissions + +| Repository kind | File payload packaging | Default | +| --- | --- | --- | +| `local/dir` | `none` | `none` | +| `local/archive` | `tar`, `tgz`, `zip` | `tgz` | +| `aws/s3` | `none`, `tar`, `tgz`, `zip` | `tgz` | +| `oci/registry`, `oci/layout` | `tar`, `tgz`, `zip` for newly packaged files | `tgz` | + +With S3 `packaging: none`, publish separate objects below `///`. +With archive packaging, publish one payload object plus metadata/documents. Existing OCI content +keeps its original media types and graph; repository packaging settings do not repackage it. + +`options.path` is always the repository destination, never the publish source. Relative configured +paths resolve against the configuration file declaring them; CLI source/output paths resolve +against the invocation working directory. Workflow/hook source/output paths follow their existing +working-directory rules. Config loading must retain path provenance through imports. + +Local/S3 `key_pattern` overrides the path below the stable root. Default paths follow the addressing +table. Expose `.Name`, `.Version`, `.Kind`, and `.CreatedAt`; use the existing Go template functions. +Capture `.CreatedAt` once in UTC for a publication and reuse it across retries. Reject root escapes +and conflicting resolved paths; validate archive extensions against packaging. There is no implicit +strftime interpretation of `%Y` or `%M`. + +Persist the resolved location with the artifact's name/version record. Reads, deletes, and retries +use that record, not today's date. Keep discoverable records under the stable root even when +payloads use dated paths. An interrupted publication must not expose a complete-looking record. + +`permissions: "0600"` controls the resulting local archive file, not its members, S3 access, or OCI +access. Accept quoted octal permission bits; reject `preserve` and unsupported backend usage. +Default local archive permissions are `"0600"`. Preserve ordinary payload permission bits, +including executable bits, where the target filesystem supports them. Full owner, ACL, xattr, +and privileged-bit backup fidelity is outside V1; report unsupported permission requirements. + +## Command surface and mutation semantics + +All examples in this section are proposed interfaces. The public command space is plural: +`atmos artifacts`. + +```shell +atmos artifacts publish ./dist --repository backups --name builds/app --version build-123 +atmos artifacts publish ./dist --repository backups --name builds/app --version build-123 --force +atmos artifacts publish ./dist --repository backups --name builds/app --version build-124 --dry-run +atmos artifacts pull components/terraform/vpc:1.2.0 --repository shared --output ./vendor/vpc +atmos artifacts copy components/terraform/vpc:1.2.0 --from shared --to backups +atmos artifacts inspect components/terraform/vpc:1.2.0 --repository shared --format json +atmos artifacts list --repository shared +atmos artifacts list components/terraform/vpc --repository shared +atmos artifacts search vpc --labels 'team=platform,purpose=backup' +atmos artifacts show readme components/terraform/vpc:1.2.0 --repository shared +atmos artifacts show changelog components/terraform/vpc:1.2.0 --repository shared +atmos artifacts delete builds/app:build-123 --repository backups +``` + +- `publish` creates a versioned artifact from a file or directory. Update means republishing that + reference with `--force`; there is no separate in-place editing command or `--replace` flag. +- `pull` verifies and materializes content at `--output`. Existing different output requires + `--force`; validate/extract into staging before replacing it. Do not overlay stale files. +- `copy` preserves artifact identity, metadata, documents, and verified content. Destination + packaging may change physical representation; destination label defaults do not restamp copies. + Use backend-native transfer where available and a bounded streaming/staging fallback elsewhere. +- `inspect` reads metadata; `show` reads a selected document. Neither downloads the main payload. +- `list` enumerates artifacts/versions; `search` filters name/description and labels. Both identify + the source alias, support structured output, and accept `--refresh` for discovery-cache refresh. +- `delete` removes the selected published reference and its exclusively owned content where the + backend supports it. It never means delete the collection, purge all versions, or remove a + catalog entry. Shared OCI blobs are subject to registry garbage collection. + +Compare content, documents, and effective user metadata before publishing. An identical publication +is a no-op; generated timestamps must not turn an identical retry into different content. Reuse +the existing record on a no-op. Different content at the same name/version fails unless `--force` +is supplied. Forced publication creates a replacement record and updates the reference; it does +not mutate bytes addressed by an old digest. Old digests remain readable only while retained. + +Archive assembly must be deterministic for the same input bytes, paths, modes, and declared +metadata. Normalize incidental archive timestamps and ordering. Separate content identity from +generated operation metadata and backend representation digests. `inspect` and machine output +must clearly label content/record digests versus OCI manifest or archive checksums. + +OCI tag updates are not portable compare-and-swap. Require external serialization for writers to +the same mutable reference and reconcile interrupted writes before retrying. Use conditional +writes/locks where supported; never claim universal atomic publication across multiple objects. +Publish completion metadata last. Unsupported operations fail explicitly before mutation. + +`--force` does not bypass authentication, verification, or immutable-release policy. In particular, +plugin releases remain immutable. Generic artifact operations must not provide an overwrite path +around plugin-format immutability when they encounter plugin releases. + +Dry runs display selected files, exclusions, metadata, and resolved destination without mutation. +Commands are noninteractive in CI, return nonzero for operation failures, and keep structured +results on stdout and human progress on stderr. Authentication uses existing Atmos auth/profile +flags and identities; no new artifact login/logout subsystem is introduced. + +## Metadata and package documentation + +The portable artifact record contains name, version, kind, description, creation/provenance data, +labels, content inventory, document descriptors, and integrity information. Inventory describes +relative paths, sizes, digests, and file modes. CI provenance may include commit, run, and source +repository identifiers; generic artifacts do not require a Terraform stack/component/SHA tuple. + +Repository `defaults.metadata.labels` supplies defaults for newly published artifacts. Explicit +publication metadata overrides matching default keys. These labels belong to each published +artifact version, not to the repository itself. Metadata changes affect artifact identity and +follow the same conflict/force rules as content changes. +Use `publish --labels 'team=platform,purpose=backup'` for CLI overrides and `metadata.labels` in +artifact steps. Inspection and machine output return the effective labels after defaults merge. + +| Backend | Authoritative artifact metadata | Efficient document access | +| --- | --- | --- | +| OCI | Versioned Atmos config blob referenced by the manifest; standard annotations for standard fields | Separate document blobs referenced by descriptors | +| S3 | JSON sidecar record, not only object headers or tags | Separate document objects | +| Local directory | Managed metadata alongside payload content | Direct document files | +| Local archive | Adjacent record plus embedded portable metadata | Adjacent document sidecars; embedded copies for transport | + +For archives, embed portable metadata and documents under `.atmos/artifact/`; keep operation and +storage-location data in the adjacent record to avoid self-referential archive checksums. Readers +verify document descriptors and the payload checksum from that record. If only a standalone archive +is available, recover embedded metadata/documents by scanning it; do not promise random access into +gzip. Sidecars are part of the managed publication and must be consistent with its record. + +S3 `options.tags` specifies backend object tags for lifecycle/access integration. Apply them to +payload, metadata, and document objects of the publication. They do not replace portable labels, +and labels are not automatically copied into tags. Validate the S3 limit of ten tags before writes; +tag changes outside Atmos do not change portable artifact labels. +See [S3 object tagging](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-tagging.html). + +### README and CHANGELOG conventions + +```text +components/terraform/vpc/ + main.tf + README.md + .atmos/artifact/ + README.md + CHANGELOG.md +``` + +Default package documents are `.atmos/artifact/README.md` and `.atmos/artifact/CHANGELOG.md`. +Both are optional. `--readme` and `--changelog` override their source paths, resolved relative to +the publish source directory (or the parent for a single-file source); explicit missing files fail. +Step equivalents are `readme` and `changelog`. No changelog format/parser is required in V1. + +Root README/CHANGELOG files remain payload unless explicitly selected as package documents. +Existing scaffold root README handling, generated files, and post-generation instructions retain +their behavior. Reserve `.atmos/artifact/` for distribution metadata and exclude it from scaffold +generated output. Explicit document paths do not implicitly remove an ordinary payload file. + +Store document media types, sizes, and digests in metadata. `show readme` and `show changelog` fetch +only the selected blob/object/file. Missing optional documents produce a clear not-found result. +Catalogs carry descriptors, not copies of full Markdown documents. + +### OCI media types and Helm lessons + +Use OCI image manifests with artifact metadata, not fabricated runnable image configurations. +ORAS is the transport/assembly library; it does not dictate one media type for all payloads. + +| Role | Proposed type | +| --- | --- | +| OCI manifest | `application/vnd.oci.image.manifest.v1+json` | +| Generic artifact type | `application/vnd.atmos.artifact.v1` | +| Scaffold artifact type | `application/vnd.atmos.scaffold.v1` | +| Component artifact type | `application/vnd.atmos.component.v1` | +| Toolchain package artifact type | `application/vnd.atmos.toolchain.package.v1` | +| Metadata config | `application/vnd.atmos.artifact.config.v1+json` | +| Tar payload | `application/vnd.atmos.artifact.content.v1.tar` | +| Gzip-compressed tar payload | `application/vnd.atmos.artifact.content.v1.tar+gzip` | +| Zip payload | `application/vnd.atmos.artifact.content.v1.zip` | +| README/CHANGELOG blob | `text/markdown` | + +Record component subtype in portable metadata. Preserve upstream media types for tool release +assets and existing OCI graphs. Existing plugin media types, manifest bytes, signatures, and +lockfiles must remain compatible; this proposal does not rename plugin packages. + +Follow the [OCI artifact manifest guidance](https://github.com/opencontainers/image-spec/blob/main/manifest.md#guidelines-for-artifact-usage). +Adopt Helm's useful conventions: package name/version drive publication beneath a collection root, +and metadata and payload have distinct media types. See [Helm OCI registries](https://helm.sh/docs/topics/registries/). +Atmos does not reuse Helm media types for non-chart content. + +## Discovery, labels, and optional catalogs + +`list` and `search` use `artifacts.repositories` directly. Without `--repository`, search all +configured sources. `--labels 'team=platform,purpose=backup'` matches both exact key/value pairs +on the published version. Filter portable metadata, not S3 tags or a registry-specific label API. +Bound concurrent reads and reuse metadata by immutable digest; do not fetch payloads to search. + +Directory, archive-root, S3, and OCI-layout repositories enumerate their managed records locally +or by object listing. An OCI collection prefix is not itself a portable repository-enumeration API. +Use supported registry enumeration scoped to the prefix when available. Otherwise collection-wide +discovery requires a catalog; fail with an actionable capability diagnostic rather than silently +returning an empty or supposedly complete result. Explicit-name operations, including +`list components/terraform/vpc`, can list that OCI repository's tags without a catalog. + +An optional `catalog` belongs directly to the configured repository: + +```yaml +artifacts: + repositories: + shared: + kind: oci/registry + identity: registry-reader + options: + registry: ghcr.io + prefix: acme/atmos + catalog: oci://ghcr.io/acme/atmos/catalog:stable +``` + +Adapt the [plugin discovery and catalog design](./plugins/discovery.md): separate catalog +build/publish/add/remove/sync operations, version/digest-level labels, bounded metadata reads, +and a private local cache scoped by effective source/auth configuration. Catalog references +never grant credentials to another host or bypass current source/verification policy. + +`publish` never updates a catalog implicitly. Catalog maintainers use one externally serialized +writer spanning read-latest, build, and publish. Revision/digest checks detect stale updates but +are not atomic registry locks. Removing an entry unlists it without deleting its artifact. + +Use a 15-minute discovery-cache TTL initially. `--refresh` revalidates sources and reuses unchanged +digest metadata. Offline operation uses cached discovery only; missing/corrupt caches and offline +refresh requests fail. Online refresh failures must not silently return stale or incomplete search +results. Cached discovery does not prove payload availability or authorize installation. + +## Producers, workflows, and hooks + +### Components and scaffolds + +```shell +atmos artifacts publish ./components/terraform/vpc \ + --repository shared --name components/terraform/vpc --version 1.2.0 --kind component/terraform +atmos scaffold publish ./scaffolds/landing-zone \ + --repository shared --name scaffolds/landing-zone --version 2.0.0 +``` + +`atmos scaffold publish` delegates to the generic publisher with scaffold validation and kind. +Keep scaffold source resolution and component source provisioning as consumers of shared OCI +operations. Publishing a component does not run Terraform or require a stack selection. +The component producer excludes Terraform working directories, state, and generated plan files; +dry-run output makes those exclusions visible. Generic backup/file publishing does not inherit +component-specific exclusions. Preserve executable scripts and package documents. + +### Verified toolchain packages + +Publish original downloaded release archives/executables plus platform, upstream URL, checksum, +size, and verification metadata from toolchain lock data. Verification precedes publication. +Do not treat repackaged installed binaries as equivalent to verified upstream release bytes. +Toolchain remains responsible for resolution, verification, extraction, installation, and PATH; +artifacts provide storage/distribution and an explicit repository-backed package source. +Prove installation from published packages with upstream access disabled. Installed-tool snapshots, +automatic dependency crawling, and changing native CI cache semantics are deferred. + +### Shared step handler + +Add `type: artifact` to the existing step registry. Its `action` selects the same service operation +as the CLI. `artifact_name` and `artifact_kind` distinguish artifact identity from the step's own +`name` and hook `kind`; the CLI equivalents are `--name` and `--kind`. + +```yaml +workflows: + publish-vpc: + steps: + - name: publish-vpc + type: artifact + action: publish + source: ./components/terraform/vpc + repository: shared + artifact_name: components/terraform/vpc + artifact_kind: component/terraform + version: "1.2.0" +``` + +```yaml +hooks: + backup-output: + events: [after-terraform-apply] + kind: step + type: artifact + on_failure: fail + with: + action: publish + source: ./dist + repository: backups + artifact_name: backups/app + version: build-123 + force: false +``` + +Read/delete step actions use `ref`; pull uses `destination` for CLI `--output`; copy uses `from` +and `to`. Preserve existing templating, working-directory, retry, conditional, and failure behavior +through the [step hook bridge](./hooks-step-types.md). Return typed operation results and named +outputs, including reference, resolved location, digests, size, and outcome. Retry context must +retain publication identity and the captured timestamp. + +Do not introduce a separate artifact hook implementation. Existing `type: publish` delivery behavior +and [archive steps](./archive-step.md) keep their semantics; reuse archive primitives internally. + +## CI compatibility and job summaries + +Artifact operations must produce polished Markdown job summaries when CI reporting is enabled. +Use the existing `ci.Reporter`, summary feature gates, provider routing, masking, and template +loader, following the [container reporting integration](./native-ci/providers/container.md). +Honor `ci.enabled`, `ci.summary.enabled`, and existing explicit CI-mode overrides. Do not write +directly to provider-specific environment files from artifact code. + +```yaml +ci: + enabled: true + summary: + enabled: true + templates: + base_path: .atmos/ci/templates + artifacts: + publish: artifacts/publish.md + copy: artifacts/copy.md +``` + +`ci.templates.artifacts` is a proposed operation-to-template map. Resolution follows explicit +override, `/artifacts/.md`, then embedded default. An invalid explicit +override produces a reporting diagnostic rather than silently choosing a different template. +Reporting remains best-effort and cannot turn a successful transfer into a failed operation. + +The default presentation includes a heading and compact results table with operation, outcome, +artifact name/version, repository, destination, digest, human-readable size, and duration. +Use short display digests with full values in expandable details; identify their digest type. +Include useful links only where a backend can supply a valid viewer/download location; otherwise +show a copyable reference. Do not emit credentials or signed URLs into summaries. + +Illustrative result rows (additional destination and integrity details appear below the table): + +| Operation | Outcome | Artifact | Repository | Size | Duration | +| --- | --- | --- | --- | --- | --- | +| Publish | Published | `components/terraform/vpc:1.2.0` | `shared` | 48 KiB | 1.2 s | +| Publish | Unchanged | `scaffolds/landing-zone:2.0.0` | `shared` | 96 KiB | 0.3 s | +| Publish | Failed | `backups/app:build-123` | `backups` | — | 2.1 s | + +Distinguish published, unchanged, replaced, downloaded, copied, deleted, and failed outcomes. +Read operations report inspected/listed results without claiming a payload transfer. Collapsible +sections show metadata, selected labels, file counts, document availability, and useful failure +diagnostics. Do not embed entire package files or changelogs by default. + +One shared renderer consumes operation results for CLI, workflow, and hook entry points. +Assign reporting ownership at the outer execution boundary so hook bridging and nested steps +do not emit duplicate summaries. Aggregate concurrent work in deterministic order and serialize +writes; retain completed operations if a later operation fails. Retries produce one final outcome +per operation, with retry information in details. + +Escape Markdown content, mask secrets, and truncate to provider limits with a visible omission +notice. Summary rendering/delivery failures log a diagnostic and preserve the operation's exit +status. Disabled summaries perform no additional metadata fetches. Preserve unrelated existing +job-summary content. Structured stdout remains clean; local presentation uses existing stderr/UI +behavior. PR comments and status checks are separate features and are not implicitly enabled. + +## Shared implementation and compatibility + +Build a common artifact service over registered repository adapters with explicit capabilities: +resolve/inspect, publish, pull, list, delete, document reads, and optional native copy. Keep backend +options and authentication separate from producer-specific validation and packaging. + +Refactor [shared OCI operations](./plugins/platform/oci-reuse.md) around +[`oras-go/v2`](https://github.com/oras-project/oras-go). The plugin implementation selected +`go-containerregistry` to reuse the existing stack; this migration is deliberate convergence, not +a second independent OCI client in artifact commands. Replace library-specific public types with +shared descriptors/references and migrate callers behind compatibility entry points. + +Preserve raw manifests/indexes, blobs, digests, media types, platforms, signature/referrer transport, +verification policy, and plugin immutability. Retain credential precedence, identity failure +behavior, private CA handling, cancellation, and independent source/destination credentials. +Existing plugin packages, catalogs, signatures, and locks must remain readable without rewriting. +Do not remove `go-containerregistry` until remaining image-specific consumers have been audited. +No Skopeo subprocess or new `containers/image/v5` transport is required by this proposal. + +For generic artifacts copied across packaging backends, preserve verified logical content and +metadata while reporting any changed representation checksum. For existing OCI graphs transferred +through S3/local storage, preserve a standard OCI-layout representation, including referenced blobs; +restore it without rewriting manifests. Do not flatten container layers into an ordinary file tree. + +Adapt [native CI artifact storage](./native-ci-artifact-storage.md) to the common substrate instead +of duplicating its S3/local clients. Preserve existing planfile store config, key patterns, identity +resolution, metadata, selection rules, and archive compatibility. General metadata must not require +CI-only fields. Fix mode loss when adapting file bundles so executable tools/scripts survive. +Existing GitHub artifact storage remains CI-scoped; it is not a durable general repository promise. + +## Delivery and acceptance + +Deliver in stages: shared OCI migration with conformance coverage; repository service and local/S3 +adapters; command/step/hook surfaces; metadata/documents and discovery/catalogs; producer integrations +and CI reporting. Each stage must preserve existing consumers. Runtime implementation must add +schema definitions, CLI help, and website documentation; this PRD change adds no runtime behavior. + +Acceptance scenarios: + +1. Configure one repository, then publish multiple components, scaffolds, and generic files without + adding repository entries. Repeat against OCI, S3, directories, and archives using the same CLI. +2. Round-trip payloads and documents across backends; compare content digests and file modes. + Transfer an OCI graph through a layout and a second registry with identical manifest digests. +3. Read metadata and README/CHANGELOG with payload downloads forbidden by the test transport. + Recover metadata from a standalone archive and explicitly account for the required scan. +4. Verify unchanged publication is a no-op, different bytes require force, and plugin releases + reject replacement. Exercise interrupted writes, conflicting names, staging, and retries. +5. Read dated publications on a later day; confirm retry stability, source/destination anchoring, + path containment, archive permission bits, and executable payload fidelity. +6. Filter version-specific labels across repositories. Test catalog-free discovery where supported, + explicit-name OCI lookup, unsupported collection enumeration, and catalog-based discovery. + Publication must leave catalogs unchanged. Verify offline cache and auth-scope behavior. +7. Publish a scaffold without changing generated README behavior; consume a component through + existing provisioning; install a verified tool package with its upstream unavailable. +8. Run equivalent CLI, workflow, and hook operations with matching results, retry semantics, and + exactly one CI report. Verify reporting gates, template overrides, secret masking, Markdown + escaping, size limits, concurrent aggregation, no-op results, and partial failures. +9. Preserve existing CI planfile fixtures, plugin release/catalog/lock compatibility, signature + verification, and source/destination auth isolation through the ORAS migration. + +Use unit tests with injected transports and generated mocks, temporary local repositories, the +existing in-process OCI registry harness, and deterministic S3 fakes. Opt-in emulator round trips +must seed a source, publish to a destination, disable the source, and restore into fresh state. +Never substitute successful metadata lookup for proof that offline payload restoration works. + +## Deferred requirements preserved from the earlier draft + +These remain intended follow-up capabilities, not additional V1 command/configuration promises: + +- Bulk image mirror/sync: standalone and component-driven selection, explicit host/all/selected + platform policies, source-host-preserving names, tag preservation, promotion, and collision rules. +- Bundle integration: select an exact dependency closure, mirror locally, export/import for air-gap + use, and verify restoration without upstream access. A bundle is distinct from one archive artifact. +- Vendoring convergence: compile existing vendor sources into source-artifact operations while + preserving hydration targets, include/exclude filters, version resolution, and provenance. +- Additional Git-backed, HTTP archive, GitHub release, and CI-scoped repository adapters; retain + native Git delivery semantics and document each backend's capability and retention boundaries. +- Terraform registry caches: export/restore individual provider/module records and required indexes + by origin, version, platform, and digest; verify restored mirrors and module resolution offline. +- Toolchain installed-tree snapshots and resolver/index distribution for complete offline tool + discovery. V1 verified upstream package storage must remain independently useful. +- SBOMs and attestations: subject-digest associations, preserve/require/ignore transfer policy, + CycloneDX/SPDX documents, pluggable generation, and aggregate dependency-closure inventories. + Existing plugin signature requirements are compatibility obligations now, not deferred work. +- Repository statistics, retention/prune/garbage-collection policies, advanced signing/encryption, + and daemon/container-storage transports. V1 explicit reference deletion remains in scope. +- Structured changelog parsing, version-range release-note aggregation, and template-update PR + enrichment using the package documents introduced here. + +No historical singular `artifact.repositories`, per-image publish-target schema, or Skopeo-specific +flags in the old draft are introduced as compatibility aliases by this proposal. Future extensions +must build on the collection-root model and plural `atmos artifacts` command space defined here. From 9a9ebdeb9f5bfc58ec118b22918542f9175d3bac Mon Sep 17 00:00:00 2001 From: Erik Osterman Date: Fri, 30 Oct 2026 13:04:09 -0500 Subject: [PATCH 2/7] docs: remove artifacts planning provenance --- docs/prd/artifacts.md | 9 --------- 1 file changed, 9 deletions(-) diff --git a/docs/prd/artifacts.md b/docs/prd/artifacts.md index 0d8670e5cba..cdcbbaad935 100644 --- a/docs/prd/artifacts.md +++ b/docs/prd/artifacts.md @@ -14,15 +14,6 @@ The same operations must work interactively, in CI, in workflows, and as lifecyc must leave useful job summaries rather than requiring users to reconstruct publication results from logs. Package README and CHANGELOG documents must be available without downloading a package. -This restores and revises the earlier `docs/prd/artifacts.md` draft preserved at commit -`2f81e11338d28a74c2a7fdf18a52d3cd5136ac4a`. That draft focused on container images and -`containers/image/v5`. This proposal broadens V1 to general artifacts and selects `oras-go/v2` -for shared OCI transport. Deferred requirements from that draft are retained below. - -The scaffold documentation requirements also reflect the Erik/Judith conversation supplied during -planning: package documentation is distinct from generated repository documentation, and release -notes should eventually support template-update PRs. Changelog aggregation is deferred. - ## Use-case inventory Artifacts are the distribution and restoration layer for the following use cases. Offline and From 021e2201a162032f939be5b29e4f50728cc0ef36 Mon Sep 17 00:00:00 2001 From: Erik Osterman Date: Fri, 30 Oct 2026 17:53:28 -0500 Subject: [PATCH 3/7] docs: detail artifact implementation and repository trust contracts --- docs/prd/artifacts.md | 27 +- docs/prd/artifacts/contracts.md | 449 +++++++++++++++++++++++++++ docs/prd/artifacts/implementation.md | 377 ++++++++++++++++++++++ docs/prd/artifacts/validation.md | 204 ++++++++++++ 4 files changed, 1048 insertions(+), 9 deletions(-) create mode 100644 docs/prd/artifacts/contracts.md create mode 100644 docs/prd/artifacts/implementation.md create mode 100644 docs/prd/artifacts/validation.md diff --git a/docs/prd/artifacts.md b/docs/prd/artifacts.md index cdcbbaad935..d1757d22ff7 100644 --- a/docs/prd/artifacts.md +++ b/docs/prd/artifacts.md @@ -1,7 +1,8 @@ # Atmos Artifacts: Repositories, Publishing, Discovery, and Automation -**Status:** Proposed. This document specifies future capabilities; its configuration and command -examples are not a claim that those capabilities are implemented. This change is documentation only. +**Status:** Proposed; configuration and command examples describe future capabilities. Documentation only. + +**Implementation:** [Work packages](./artifacts/implementation.md), [contracts](./artifacts/contracts.md), and [validation](./artifacts/validation.md). ## Problem and outcome @@ -53,7 +54,7 @@ preserve indexes, locks, checksums, and dependency relationships and prove consu - **Catalog:** An optional maintained discovery snapshot. It is not necessary for direct reads. V1 includes publishing, pulling, copying, inspection, listing, search, deletion, package documents, -label filtering, optional catalogs, hooks, and CI reporting. OCI manifests/indexes are transferred +label filtering, catalogs, signatures/locks, hooks, and CI reporting. OCI manifests/indexes are transferred as content graphs, not interpreted as ordinary directory archives. Artifacts are not components. Existing top-level `stores` remain key-value and secret stores. @@ -183,8 +184,7 @@ and privileged-bit backup fidelity is outside V1; report unsupported permission ## Command surface and mutation semantics -All examples in this section are proposed interfaces. The public command space is plural: -`atmos artifacts`. +The proposed public command space is plural: `atmos artifacts`. ```shell atmos artifacts publish ./dist --repository backups --name builds/app --version build-123 @@ -193,6 +193,8 @@ atmos artifacts publish ./dist --repository backups --name builds/app --version atmos artifacts pull components/terraform/vpc:1.2.0 --repository shared --output ./vendor/vpc atmos artifacts copy components/terraform/vpc:1.2.0 --from shared --to backups atmos artifacts inspect components/terraform/vpc:1.2.0 --repository shared --format json +atmos artifacts sign 'components/terraform/vpc@sha256:' --repository shared +atmos artifacts verify components/terraform/vpc:1.2.0 --repository shared atmos artifacts list --repository shared atmos artifacts list components/terraform/vpc --repository shared atmos artifacts search vpc --labels 'team=platform,purpose=backup' @@ -479,7 +481,7 @@ Illustrative result rows (additional destination and integrity details appear be | Publish | Unchanged | `scaffolds/landing-zone:2.0.0` | `shared` | 96 KiB | 0.3 s | | Publish | Failed | `backups/app:build-123` | `backups` | — | 2.1 s | -Distinguish published, unchanged, replaced, downloaded, copied, deleted, and failed outcomes. +Distinguish published, unchanged, replaced, downloaded, copied, deleted, signed, verified, and failed outcomes. Read operations report inspected/listed results without claiming a payload transfer. Collapsible sections show metadata, selected labels, file counts, document availability, and useful failure diagnostics. Do not embed entire package files or changelogs by default. @@ -499,8 +501,15 @@ behavior. PR comments and status checks are separate features and are not implic ## Shared implementation and compatibility Build a common artifact service over registered repository adapters with explicit capabilities: -resolve/inspect, publish, pull, list, delete, document reads, and optional native copy. Keep backend -options and authentication separate from producer-specific validation and packaging. +resolve/inspect, publish, pull, list, delete, document reads, copy, signing, verification, pinning, +and write locking. Each adapter implements these contracts through its native mechanisms or shared +providers: OCI referrers, S3 evidence objects/conditional claims, and local detached files/locks. +Capabilities state supported signature formats and lock guarantees; unsupported requirements fail. +Verify the pinned signature subject against current trust policy before required consumption/copy, +preserve evidence across transports, and support offline verification with independently trusted roots. +Distinguish immutable dependency pins from write locks and signatures from checksums; `--force` +cannot weaken verification or coordination requirements. Keep backend authentication separate from +signing identity and producer-specific validation. See the implementation contracts for policy and APIs. Refactor [shared OCI operations](./plugins/platform/oci-reuse.md) around [`oras-go/v2`](https://github.com/oras-project/oras-go). The plugin implementation selected @@ -580,7 +589,7 @@ These remain intended follow-up capabilities, not additional V1 command/configur - SBOMs and attestations: subject-digest associations, preserve/require/ignore transfer policy, CycloneDX/SPDX documents, pluggable generation, and aggregate dependency-closure inventories. Existing plugin signature requirements are compatibility obligations now, not deferred work. -- Repository statistics, retention/prune/garbage-collection policies, advanced signing/encryption, +- Repository statistics, retention/prune/garbage-collection policies, additional signing schemes/encryption, and daemon/container-storage transports. V1 explicit reference deletion remains in scope. - Structured changelog parsing, version-range release-note aggregation, and template-update PR enrichment using the package documents introduced here. diff --git a/docs/prd/artifacts/contracts.md b/docs/prd/artifacts/contracts.md new file mode 100644 index 00000000000..cc2b24adac9 --- /dev/null +++ b/docs/prd/artifacts/contracts.md @@ -0,0 +1,449 @@ +# Atmos Artifacts implementation contracts + +**Status:** Proposed implementation specification, not an implemented API. +Read with the [implementation plan](./implementation.md) and [product requirements](../artifacts.md). +The names below are implementation defaults for V1. Change an externally visible contract only by +updating its specification, examples, schema, tests, and consuming adapters together. + +## Package and dependency boundaries + +```mermaid +flowchart TD + A[CLI, steps, hooks, CI adapters] --> B[Artifact service] + B --> C[Repository contracts] + B --> P[Producer interfaces] + C --> L[Local adapter] + C --> S[S3 adapter] + C --> O[OCI adapter] + L --> IO[Shared blob I/O] + S --> IO + O --> OCI[pkg/oci and ORAS] + C --> V[Shared signer and verifier interfaces] + C --> K[Lock provider interfaces] + CI[Legacy CI adapters] --> IO +``` + +The artifact service imports no CLI, runner, hook, plugin, or CI packages. Backends and producers +register through composition roots; plugin-specific policy stays above generic OCI transport. +`pkg/artifact/report` may depend on CI reporting; `pkg/artifact` itself must not. Share cached +discovery infrastructure below both plugin and artifact discovery, with separate schema adapters. + +## Domain records and integrity + +Use these types in `pkg/artifact`, with explicit JSON tags and versioned serialization: + +| Type | Required data | +| --- | --- | +| `Ref` | Name and exactly one version or digest; repository alias is a separate request field | +| `Record` | Schema version 1, name, version, kind, created_at, metadata, sorted inventory, documents, content_digest, optional typed kind data | +| `Metadata` | Description, string labels, optional CI/source provenance; no required stack/component/SHA | +| `Entry` | Relative path, type (file/directory/symlink), size, ordinary mode bits, file digest or link target | +| `Document` | Role (readme/changelog), original source path, media type, size, digest | +| `Representation` | Packaging, payload/blob descriptors, backend locator, record digest, optional original OCI root descriptor | +| `PreparedArtifact` | Record, intent digest, staged blob readers, preparation timestamp, cleanup ownership | +| `OperationResult` | Operation ID, action, outcome, requested/resolved refs, aliases/locations, digest fields, size, duration, diagnostics | +| `Capabilities` | Read/write, enumeration, delete semantics, conditional update, packaging/modes, evidence formats, lock scopes | +| `SignatureSubject` | Subject kind, immutable digest, media type, size; portable Record or native OCI graph root | +| `SignatureEvidence` | Subject, format, evidence descriptor, storage locator; untrusted until verified | +| `VerificationResult` | Subject, policy fingerprint, verified signer claims, evidence digests, status and diagnostics | +| `PinnedReference` | Repository locator, resolved digest, record/content digests, evidence descriptors, optional selected platform | +| `LockRequest` / `LockHandle` | Resource scope, owner/operation ID, required guarantee, expiry where leased, ownership token, renew/release methods | + +Keep record bytes independent of repository alias, credentials, resolved storage paths, upload +duration, and archive/OCI envelope digests. Those belong to Representation or OperationResult. +`Record` never includes its own hash. OCI config stores the same canonical Record bytes used by +local/S3 records; the manifest references the backend representation separately. + +Canonical serialization uses a typed JSON schema, sorted inventory/document arrays and map keys, +UTF-8 strings, integer sizes/modes, UTC RFC3339 timestamps, no whitespace, and no trailing newline. +Freeze byte-level test vectors; do not rely on arbitrary `map[string]any` or float serialization. + +- `content_digest`: SHA-256 of the canonical inventory and document descriptors, including paths, + modes and link targets. Exclude storage locators and generated operation data. +- `intent_digest`: SHA-256 of name/version/kind, content_digest, effective user metadata and kind + data. Exclude created_at and generated execution provenance; they cannot defeat retry/no-op. +- `record_digest`: SHA-256 of the full canonical Record bytes, including the original created_at. +- `representation_digest`: Checksum of an archive or other transferred payload representation. +- `resolved_digest`: OCI manifest/index digest for OCI references; record_digest for local/S3 refs. + +A copy preserves Record bytes and content identity. Repacking may change representation_digest; +an OCI envelope newly assembled from another backend may have a different resolved_digest. +Existing OCI graphs preserve every original descriptor and blob digest instead. + +Record and catalog JSON reads are bounded to 8 MiB, each document to 4 MiB, OCI manifest/index +reads to the existing bounded transport policy. Reject oversize metadata before publishing; +stream payloads to disk/network rather than loading them into memory. Use a default transfer and +discovery concurrency of four, injected through service options for tests; retain existing plugin limits. + +## Service and repository APIs + +Define separate request structs for Publish, Pull, Copy, Inspect, List, Search, Delete, and +ShowDocument. Each takes context first and returns typed results plus error. Do not use maps for +request fields or one method with backend-specific optional arguments. +Add Sign, Verify, and Pin requests; publication/copy/delete use the same Lock contract internally. + +The service owns name/version selection, producer preparation, effective metadata, digest pinning, +conflict policy, retries, and result aggregation. Repositories own storage mechanics and capabilities. + +Repository interfaces must provide: + +- `Capabilities()` and `Resolve(ctx, Ref)` returning a pinned descriptor/record handle. +- `Inspect(ctx, pinned)` and `OpenDocument(ctx, pinned, role)` without payload download. +- `Read(ctx, pinned)` returning closeable verified content streams or an OCI graph source. +- `Publish(ctx, prepared, options)` with expected prior reference and force policy already checked. +- `List(ctx, query)` returning a page and cursor; pagination is consumed by the service. +- `Delete(ctx, pinned, options)` with expected current reference and explicit deletion scope. +- `Pin(ctx, Ref)` returning an immutable PinnedReference suitable for consumer lockfiles. +- `Sign(ctx, subject, signer, options)` returning stored SignatureEvidence descriptors. +- `Verify(ctx, subject, policy)` returning authenticated VerificationResult, never metadata claims. +- `Lock(ctx, LockRequest)` returning an ownership-checked LockHandle or unsupported-capability error. +- Optional `CopyFrom` for a compatible native transfer; absence uses the shared staging fallback. + +Use OCI-standard descriptors at the OCI seam. Keep `remote.Repository`, go-containerregistry image +objects, AWS clients, and SDK errors out of the generic service API. Wrap errors in common sentinels +while retaining their causes for diagnostics. Generate mocks from these interfaces. + +## Public command and step mapping + +Signatures, verification, pinning, and coordination are repository contracts, not OCI-only service +branches. The following operations use them identically for every backend. + +The following options supplement normal Atmos auth/profile/CI/output flags. A missing required +repository is an error; only collection list/search intentionally default to all configured repositories. + +| Operation | CLI input | Artifact step fields | +| --- | --- | --- | +| Publish | `publish --repository R --name N --version V --kind K` | `action: publish`, `source`, `repository`, `artifact_name`, `version`, `artifact_kind` | +| Publish controls | `--force`, `--dry-run`, `--labels`, `--readme`, `--changelog` | `force`, `dry_run`, `metadata.labels`, `readme`, `changelog` | +| Pull | `pull --repository R --output DIR [--force]` | `action: pull`, `ref`, `repository`, `destination`, `force` | +| Copy | `copy --from A --to B [--force]` | `action: copy`, `ref`, `from`, `to`, `force` | +| Inspect | `inspect --repository R` | `action: inspect`, `ref`, `repository` | +| Sign | `sign --repository R` | `action: sign`, digest-only `ref`, `repository` | +| Verify | `verify --repository R` | `action: verify`, `ref`, `repository` | +| List | `list [name] [--repository R] [--labels L] [--refresh]` | `action: list`, optional `artifact_name`, `repository`, `labels`, `refresh` | +| Search | `search [text] [--repository R] [--labels L] [--refresh]` | `action: search`, `query`, `repository`, `labels`, `refresh` | +| Delete | `delete --repository R` | `action: delete`, `ref`, `repository` | +| Documents | `show readme|changelog --repository R` | `action: show`, `document`, `ref`, `repository` | + +Publishing defaults to kind `files`; specialized convenience commands supply their kind. Name +precedence is explicit flag, producer metadata, source basename. Version precedence is explicit +flag then producer metadata; otherwise fail. No generated version, implicit latest, or package manifest +file is required for generic files. CLI publish labels use the existing exact key/value parser. + +Explicit document paths resolve relative to the publish source directory or file's parent. Generic +source files can have any basename; name validation applies to artifact addressing, not payload names. +Validate portable relative name segments and OCI tag syntax for versions; reject backend-specific +invalid references before any writes rather than silently normalizing case or characters. + +Step `name` remains the runner's name; `output` remains its display mode; `Artifacts` remains the +runner's expected-output declaration. Add only missing fields to WorkflowStep/Task and implement +the conversions explicitly. `source` must be a string for this step, despite other steps accepting maps. + +Store the typed operation result under StepResult.Metadata[`artifact`]. Named Outputs are `ref`, +`location`, `resolved_digest`, `record_digest`, `content_digest`, `representation_digest`, `size`, +and `outcome`; optional values are omitted, never filled with misleading empty-success strings. +List/search use an `artifacts` results array. Unknown action/field combinations fail validation. +Structured CLI JSON/YAML uses the same named result fields; display format never changes semantics. + +## Signatures, verification, and locks + +Each repository implements Sign, Verify, Pin, and Lock, optionally by composing shared providers. +Capabilities describe supported evidence formats, immutable subjects, lock scope, and conditional +publication guarantees. An unavailable capability returns a typed error; it is never a successful +no-op. The orchestration layer asks for guarantees, not `if backend == OCI` behavior. + +| Repository | Signature evidence | Pinning | Default write coordination | +| --- | --- | --- | --- | +| OCI registry | Subject-bound OCI referrers and supported fallback attachments | Original manifest/index digest, or portable Record digest bound through config | Host-local guard plus documented external serialization across hosts; configured lock repository for stronger scope | +| OCI layout | Referrer descriptors/blobs in layout plus subject index | Immutable layout descriptors | Filesystem guard for the layout/ref | +| S3 | Digest-keyed signature objects and subject records | Canonical Record digest, original OCI root when carrying a graph | Conditional owner claims and conditional reference updates | +| Local directory/archive | Detached signature files; portable evidence included when exporting archives | Canonical Record digest, original OCI root for graph payloads | Filesystem locks and atomic reference updates | + +### Signature subjects and trust + +For Atmos file packages, sign the canonical portable Record. It binds name/version, metadata, +inventory, content_digest, and document digests, so evidence survives representation changes. +For foreign OCI images and existing plugins, sign/verify the original manifest or index digest; +never substitute an archive checksum or rewrite an existing signature subject on copy. +An OCI attachment carrying a Record signature must bind that Record to the addressed manifest's +config descriptor. Verification distinguishes a signed Record from a signed native image envelope. + +Store evidence by subject kind/digest and evidence digest. Local/S3 locations are beneath +`.atmos-artifacts/v1/signatures///.json`. +Signing adds evidence without changing the signed Record or native graph root. A local archive +signed after publication receives detached evidence; subsequent export/copy can embed that evidence +without changing the portable Record. Do not claim the original archive bytes were signed. + +Extract reusable cryptography from `pkg/oci/signature` into `pkg/signature`, retaining compatibility +wrappers and existing plugin behavior. Repository adapters fetch/store evidence and invoke the +shared verifier; they may add native formats through registered providers with the same result contract. +Reuse the existing managed Cosign authoring path behind a generic Signer interface for V1 signing; +registry authentication remains separate from signing identity. Do not implement custom cryptography. + +V1 supports Sigstore bundles with trusted certificate identities, and an explicitly configured +public-key bundle verification path. Keep that key verifier separate: a public-key-only signature +must never satisfy an OIDC issuer/identity requirement or weaken existing plugin SCT/log checks. +Legacy Cosign tag payloads and other formats require dedicated tested providers; do not claim support +merely because a signature-shaped object is present. Unsupported required evidence fails closed. +Follow [Sigstore verification](https://docs.sigstore.dev/cosign/verifying/verify/) and +[bundle semantics](https://docs.sigstore.dev/about/bundle/) through their supported Go libraries. + +Proposed policy configuration: + +```yaml +artifacts: + verification: + require_signature: true + trusted_identities: + - identity: https://github.com/acme/images/.github/workflows/release.yml@refs/heads/main + issuer: https://token.actions.githubusercontent.com + trusted_keys: [] + repositories: + shared: + kind: oci/registry + options: + registry: ghcr.io + prefix: acme/images + locking: + required_scope: repository + repository: coordination + coordination: + kind: aws/s3 + identity: lock-writer + options: + bucket: acme-artifact-coordination +``` + +`verification` is allowed globally and per repository. Unset require_signature defaults to false +for new general artifacts; existing plugin enforcement remains unchanged. Global requirements +cannot be weakened by repository overrides or CLI flags. Within one trust set, any configured signer +can satisfy it; if global and repository trust restrictions both exist, satisfy both sets rather +than unioning them into a weaker policy. Trusted keys are operator-managed public-key file paths. +Protected governing policies remain authoritative over ordinary imported configuration. +This enforcement covers artifact repository operations and their adapters; it does not imply +new admission controls on unrelated container-runtime pulls that bypass these operations. + +`verify` always requires a trusted valid signature; `sign` accepts digest refs only and adds evidence. +Automatic consumption/copy verifies when policy requires it or supplies trust constraints. Without +trust configuration, report unchecked rather than verified. Required verification runs before +destination publication or local materialization; a normal checksum is not signature verification. +Copy evaluates both source and destination requirements and retains the evidence needed by each. +`--force` never bypasses verification. Catalog entries, labels, prior result files, and cached +"verified" booleans never establish trust. + +At least one valid permitted signature must satisfy each required trust set. Invalid unrelated +attachments cannot override valid matching evidence, but malformed/unsupported evidence is bounded +and reported. If no permitted signature verifies, return a nonzero policy error before consumption. +Verify signature subject binding, cryptographic validity, signer/key trust, and applicable log, +certificate, and signing-time proofs. Preserve original index signatures when selecting platforms; +platform selection must not claim a newly generated index retains the original signature. + +Offline verification uses preserved evidence and independently provisioned trust material. Reuse +unexpired TUF-authenticated roots or explicitly configured operator trust, never roots supplied by +the artifact itself. Missing/expired trust or missing required proofs fail without network access. +Expose verification subject, evidence digests, policy fingerprint, verified claims, and failure reason +in inspect/verify output and CI summaries. Explicit `verify` is the metadata/evidence-only check; +payload-consuming operations additionally verify every consumed content descriptor. + +### Mutation locks versus dependency locks + +`Pin` creates the immutable reference data that existing consumer lockfiles persist. It does not +acquire a write mutex, and a pinned reference is not a substitute for signature verification. +Keep existing plugin/toolchain lock schemas compatible; add adapters rather than a new mandatory +project-wide artifact lockfile. Reevaluate current trust policy when consuming a persisted pin. + +`Lock` coordinates mutations. A LockHandle identifies owner, resource, guarantee/scope, and current +ownership token; release checks the token. Lock scopes are `host` and `repository`. Host scope is +the default for backward-compatible single-host use; cross-host publication still requires external +serialization unless repository scope is available/requested. Report the actual scope in results. +Acquire locks in canonical repository/resource order for multi-resource work to avoid deadlocks. + +Local adapters use OS-backed file locks. S3 uses create-if-absent owner claims with conditional +release/update; V1 claims do not expire automatically, so a paused writer cannot become an unfenced +stale owner. A crash leaves an explicit recovery diagnostic; an operator may remove the owner claim +only after confirming that writer has stopped. Do not automatically steal another operation's lock. +Future expiring leases must add fencing before advertising equivalent guarantees. + +An OCI adapter provides host locking but cannot implement distributed exclusion from mutable tags. +`locking.repository` delegates Lock to another configured repository that advertises repository +scope; its identity and storage root stay independent. Reject cycles and insufficient scope before +mutating. A catalog writer may use this coordinator or an externally serialized CI workflow. +No S3 Object Lock retention setting, OCI tag, or local-cache lock is advertised as a distributed lease. +External writers that ignore Atmos coordination remain outside these guarantees. + +## Preparation and document conventions + +Relative local repository roots use their declaring config directory. For remote imported config, +require a locally declared or absolute destination override rather than using ephemeral download paths. + +Capture a single UTC preparation time and logical operation ID before the first attempt. Explicit +user metadata participates in intent identity; automatically attached run IDs/durations do not. +On a no-op return the existing Record unchanged and report the current execution separately. + +Walk source paths using lstat. Preserve directories, file bytes, ordinary permissions and contained +relative symlinks; never follow a link outside the source. Reject absolute/escaping link targets, +devices, sockets, and unsupported entry types. Represent hard links as independent regular files. +Fail materialization when a target cannot implement a requested link instead of silently changing it. +Clear privileged mode bits; do not restore owner/group IDs or ACLs in V1. + +Use epoch archive timestamps, deterministic gzip headers, stable ordering, and normalized owners +for newly assembled generic archives. Preserve executable bits. Keep verified upstream tool assets +as opaque files; any outer archive does not change the original package bytes or its checksum. + +The source `.atmos/artifact/` directory is reserved for README/CHANGELOG distribution documents. +Reject conflicting unknown reserved files during preparation. Write generated `manifest.json` +there only inside the packaged representation; do not change the user's source tree. Its Record +excludes generated member entries to prevent self-reference. Root README remains normal payload. +Verify embedded document bytes against descriptors and expose documents separately for efficient reads. + +For local archive permissions, retain whether the value was explicit. Parse quoted octal permission +bits, default to 0600, and fail explicitly when an explicit request cannot be enforced on the host. +Tests must distinguish archive-file mode from member mode and from filesystem ACL guarantees. + +## Local/S3 record layout and publication lifecycle + +Below the configured root reserve `.atmos-artifacts/v1/` for managed records. Reject payload names +and key patterns that enter this namespace. Use SHA-256 of canonical name/version for private +reference keys so names containing slashes cannot collide with control paths. + +```text +/.atmos-artifacts/v1/refs/.json +/.atmos-artifacts/v1/records/.json +/.atmos-artifacts/v1/representations/.json +/.atmos-artifacts/v1/operations/.json + +.metadata.json # archive sidecar +.docs/README.md # archive document sidecar +.docs/CHANGELOG.md +``` + +For directory/per-file payloads, metadata/documents live at `/.atmos/artifact/` instead +of suffix paths. Managed readers omit this distribution directory when materializing user content. +Reference records bind name/version to record digest and Representation; immutable records never +contain a mutable storage locator. Enumerate reference records, not arbitrary user files. +Search ignores staging/journal entries. Direct digest lookup checks the Record's embedded name. + +The operation journal records captured time, intent, expected prior reference, resolved location, +and prepared representation checksums. It is an implementation recovery record, not a public catalog. +Use local reference/location locks or S3 conditional ownership records. Reserve a location before +overwriting it; another reference cannot claim it even when its key_pattern collides. + +Publication sequence: + +1. Validate capability, policy, paths, metadata sizes, and identity; prepare a stable source snapshot. +2. Read current reference under the writer guard; compare intent and apply conflict/immutability rules. +3. Record the pending operation before changing payload paths; pending references are unavailable + to managed readers and absent from completed search results. +4. Upload/stage payloads and separate documents, verify their descriptors, and write immutable Record. +5. Write Representation/adjacent sidecars, then atomically publish the completed reference last. +6. Reconcile an ambiguous response by reading the completed reference and verifying expected digests; + mark success only after reconciliation. Clean only temporary resources owned by this operation. + +Use [S3 conditional writes](https://docs.aws.amazon.com/AmazonS3/latest/userguide/conditional-writes.html) +for reference ownership and publication, treating precondition failures as conflicts. ETags are +concurrency tokens, not payload SHA-256 checksums. External serialization is still required for +writers to a mutable OCI reference and for catalog updates; no universal distributed lock is promised. + +Readers check pending/completed state before accessing content, verify every consumed descriptor, +and recheck the generation after reads from mutable payload locations. A failed/interrupted force +replacement may leave the reference unavailable until its same-intent retry repairs it. Do not +claim rollback or serve changed bytes under an old digest. Do not automatically steal a pending +operation owned by another active writer; surface its operation ID for recovery diagnostics. + +A retry in the same invocation reuses operation ID and preparation. A subsequent invocation may +resume a matching pending intent only under exclusive writer ownership and after checking staged +checksums; a different intent fails with a recovery diagnostic. Keep this conservative behavior in V1. +Old digest reads succeed only while their payload representation remains available and verifies; +otherwise return unavailable/not-found, never the current version's bytes. + +## OCI-specific behavior + +New Atmos packages use the PRD artifact/config/layer media types. Record bytes are the config blob; +README/CHANGELOG are separate Markdown blobs with explicit roles/titles. Transferred foreign OCI +graphs bypass new-package assembly. Copy required referrers through the existing policy-aware +transport; do not silently omit signature material on an unsupported registry. + +Use [ORAS graph APIs](https://pkg.go.dev/oras.land/oras-go/v2@v2.6.2) through `pkg/oci` adapters. +Preserve original raw manifest/index bytes on graph transfer. Do not use the simple file store's +automatic filename-based extraction for arbitrary untrusted graphs. + +Resolve names before reading. Name-scoped list can enumerate OCI tags without a catalog; +collection list/search uses supported registry repository enumeration or the optional catalog. +No public fallback repository list is added. A provider without enumeration/catalog returns a +capability error explaining that direct named operations still work. + +For tag deletion, inspect other tags referencing the same digest and require a tag-safe operation; +if only manifest deletion is available and it would remove sibling references, reject. Digest +deletion is explicitly manifest-scoped and reports affected references. Never delete shared blobs +directly. Protect plugin immutability on both destination format and incoming format. + +## Catalog commands and cache + +Use a separate versioned `AtmosArtifactCatalog` manifest, not the plugin catalog schema. Each entry +contains logical name, kind, source, description, and versions with resolved/record/content digests, +labels, and document descriptors. Version labels are opaque; do not reuse plugin SemVer sorting. +Sort names/versions lexically and bind a catalog entry to its actual repository path, not credentials. +Publish catalogs as generic Records of kind `catalog`, with artifact type +`application/vnd.atmos.artifact.catalog.v1` and a direct JSON payload layer of type +`application/vnd.atmos.artifact.catalog.v1+json`; retain the shared Record config media type. + +```shell +atmos artifacts catalog build --repository shared --input inventory.yaml --output catalog.json +atmos artifacts catalog publish catalog.json --repository shared +atmos artifacts catalog add components/terraform/vpc:1.2.0 --repository shared +atmos artifacts catalog remove components/terraform/vpc:1.2.0 --repository shared +atmos artifacts catalog sync --repository shared --input inventory.yaml +``` + +`inventory.yaml` is an `AtmosArtifactInventory` manifest whose spec contains `refs`, a list of +explicit names/versions or digests relative to the selected configured repository. Build resolves +and verifies metadata only and writes a snapshot; it never publishes. Publish writes to the +repository's configured catalog URI. Add/remove read and republish the latest catalog; sync merges +verified inventory releases while retaining entries absent from the inventory. Remove only unlists. +Missing catalog configuration fails these mutating catalog commands before writes. Catalog writes +require externally serialized read/build/write, and digest/revision rechecks reject already-stale input. + +Initially support OCI catalog URIs as in the PRD and plugins; local/S3/layout discovery works +directly without a catalog. Store artifact caches under XDG `atmos/artifacts/discovery`, keeping +plugin cache paths and serialization unchanged. Cache scopes include effective config/auth realm +and host bindings without credential values; refreshed snapshots replace old snapshots atomically. + +## Producer selection and CI compatibility + +Toolchain convenience interface (new, explicit; no automatic repository fallback): + +```shell +atmos toolchain publish hashicorp/terraform@1.13.4 --repository shared --platform linux/amd64 +atmos toolchain install hashicorp/terraform@1.13.4 --artifact-repository shared +``` + +Publish accepts repeated platform selectors, defaulting to the host, and requires an explicit +version plus pinned checksum data in the configured lockfile. Name packages as +`toolchain////:`. Bundle the selected lock entry, resolved install +definition, original asset, and verification evidence. Resolve aliases using existing config. +Artifact-selected installation is authoritative: fetch definition and package from that source, +reapply verification, then use the existing installer. Missing metadata, platform, or evidence fails +without contacting upstream. Default installations retain their current behavior. + +Native CI reuses existing store selection through a new explicit adapter type: + +```yaml +planfiles: + default: shared + stores: + shared: + type: artifact/repository + options: + repository: shared +``` + +Keep the existing `--store`, default, priority, and environment detection behavior. The adapter +references `artifacts.repositories.shared`; it does not duplicate its backend options. Map legacy +keys to `ci/planfiles/:` and retain the original key plus full CI metadata +in typed kind data. Implement existing query predicates from that data, not approximate label matches. +Preserve the existing planfile overwrite contract only inside this explicit adapter; it cannot +override plugin releases. Old store configurations and serialized archives are unchanged. + +Do not weaken `pkg/ci/artifact.Metadata.Validate`. The generic Record is a new type; convert at +the boundary. Add optional modes to CI FileEntry/FileResult with legacy 0644 encoding when unset. +Keep backend constructors/adapters compatible with existing tests and public package callers. diff --git a/docs/prd/artifacts/implementation.md b/docs/prd/artifacts/implementation.md new file mode 100644 index 00000000000..7010b6f42bc --- /dev/null +++ b/docs/prd/artifacts/implementation.md @@ -0,0 +1,377 @@ +# Atmos Artifacts implementation plan + +**Status:** Ready for implementation; all work packages below are unstarted. +**Product contract:** [Atmos Artifacts PRD](../artifacts.md). +**Companion specifications:** [Contracts and storage](./contracts.md), [validation and handoff](./validation.md). + +## Instructions for the implementing agent + +Implement the work packages in dependency order. Finish the acceptance checks for one package +before marking it complete. A package is not complete because its interfaces compile or its mocks +pass: demonstrate the described behavior through its real adapter or deterministic test harness. +Keep the public names and compatibility guarantees in the PRD. This plan fills in implementation +choices; it does not authorize implementing the PRD's deferred features as part of V1. + +Start by reading this plan and both companions, then inspect the indicated production code. +Recheck current repository instructions and branch state. Preserve concurrent changes. Build the +workspace binary with `atmos build` before using repository hooks; run subsequent commands with +the workspace `build` directory first on PATH. Do not disable hooks, signing, or verification. + +Use small changes that compile independently. Use registered backends, CommandProvider commands, +standard flag parsing, functional options, static error sentinels, context cancellation, and +generated `go.uber.org/mock` mocks. Keep configuration/dependencies in explicit structs, not context +values. Keep package files below the repository's size limit. Add no artifact business logic to +the CLI or a second implementation in hooks. + +Keep a handoff record listing completed package IDs, commits, exact commands/results, remaining +failures, and the next package. Do not mark all of V1 complete while a producer, backend, document +read path, or CI surface listed here is missing. + +## Current integration map + +These are existing entry points to inspect, not a proposed file-by-file rewrite. + +| Area | Current entry points | Constraint | +| --- | --- | --- | +| OCI transport | `pkg/oci/client.go`, `identity.go`, `credentials.go`, `referrers.go` | Registry methods expose go-containerregistry types; auth and referrers are already shared | +| OCI layouts/materialization | `pkg/oci/layout.go`, `pull.go`, `selection.go`, `verified.go` | Existing layout helpers expect one root release index; collections need an additional API | +| OCI consumers | `pkg/plugin/oci`, `pkg/plugin/catalog`, `pkg/plugin/discovery`, `pkg/oci/signature`, `pkg/version/resolver/oci` | Preserve release bytes, locks, policies, and immutable plugin publication | +| File consumers | `pkg/generator/source/resolver.go`, `pkg/vendoring/install/fetch.go`, `pkg/provisioner/source/vendor.go` | Preserve existing source resolution and extraction | +| CI artifacts | `pkg/ci/artifact/{backend,store,bundled_store,metadata,tar}.go` | Existing metadata validation requires stack/component/SHA; file entries omit modes | +| CI backends | `pkg/ci/artifact/local`, `s3`, `github` | Existing local/dir stores a bundled stream, not an ordinary copied directory | +| Planfiles | `pkg/ci/plugins/terraform/planfile/adapter/store.go` | Existing store selection, key patterns, metadata, and integrity remain compatible | +| Configuration | `pkg/schema/schema.go`, `pkg/config/imports.go`, `pkg/config/load.go` | Preserve declaring-file origins for destination paths through config merges | +| Schema assets | `pkg/datafetcher/schema/atmos/config/1.0.json`, `stacks/stack-config/1.0.json`, `scaffold/scaffold-config/1.0.json` | Update the surfaces that actually admit the new configuration/steps | +| Steps/hooks | `pkg/schema/workflow.go`, `pkg/schema/task.go`, `pkg/runner/step/{registry,types,variables}.go`, `pkg/hooks/step_engine.go` | Step name, existing Artifacts declarations, and output display settings already have meanings | +| Packaging | `pkg/archive/{archive,walk,tar,zip,mtime}.go` | Reuse archive primitives; do not change existing archive-step defaults | +| Scaffold | `cmd/scaffold`, `pkg/generator`, `pkg/generator/source` | Preserve root README and generated-file behavior | +| Toolchain | `cmd/toolchain/install.go`, `pkg/toolchain/installer/{installer,download,asset,extract}.go`, `pkg/toolchain/lockfile` | Cached filename alone is not identity; URL sidecars and verification matter | +| Reporting | `pkg/ci/reporter.go`, `mode.go`, `templates/loader.go`, `pkg/component/container/imagesummary` | Existing gates, providers, template conventions, and masking are authoritative | + +## Dependency order and delivery units + +| ID | Deliverable | Depends on | Completion evidence | +| --- | --- | --- | --- | +| A00 | Compatibility fixtures and contract types | None | Baseline fixtures and digest vectors | +| A01 | Config/schema/registry foundation | A00 | Import-origin and validation tests | +| A02 | Shared ORAS transport and legacy adapters | A00 | Existing consumers pass conformance | +| A03 | Preparation, packaging, operation service | A00, A01 | Determinism, streaming, and conflict tests | +| A04 | Local directory/archive repositories | A03 | Real filesystem round trips | +| A05 | Shared blob I/O and S3 repository | A03, A04 | S3 and legacy CI adapter suites | +| A06 | OCI repositories and signature/lock parity | A02, A03, A04, A05 | Signed graph round trips and shared contract tests | +| A07 | Public artifact commands | A04, A05, A06 | CLI contract tests and documentation | +| A08 | Discovery/cache/catalogs | A02, A04, A05, A06, A07 | Offline and catalog independence tests | +| A09 | Step and hook integration | A03, A07 | CLI/step/hook parity | +| A10 | CI summaries and outputs | A07, A09 | One report per operation, including failures | +| A11 | Scaffold/component producers | A03, A06, A07 | Existing consumer integration tests | +| A12 | Verified toolchain producer/consumer | A03, A05, A06, A07 | Cold offline installation with verification | +| A13 | Native CI repository bridge and release gates | A05, A09, A10, A11, A12 | Planfile compatibility and full acceptance matrix | + +A02 and A01 can be developed independently after A00. Backend work can overlap after A03's +contracts are stable. No parallelism is required. Keep review units under the repository's file +limit; a work package can take several PRs. Behavior-preserving foundations use `no-release`; +each PR exposing new public behavior follows the repository's `minor` release documentation rules. + +## A00 — Freeze compatibility and introduce domain contracts + +- [ ] Inventory all direct go-containerregistry imports and network operations in current consumers. + Record which are transport, image construction, extraction, or verification; do not remove the + dependency while image-specific callers still need it. +- [ ] Add `pkg/artifact` for generic domain models, typed requests/results, repository registration, + and service orchestration. Do not reuse `schema.Artifacts`, which describes step output files. +- [ ] Establish the record, reference, operation, and capability contracts in [contracts](./contracts.md). + Add canonical serialization vectors before implementing any backend writer. +- [ ] Define shared Sign, Verify, Pin, and Lock contracts, evidence subjects, policy results, and + lock guarantees. Signatures and locks must not be implemented only as OCI-specific options. +- [ ] Capture deterministic fixtures for a plugin release index, its manifests/blobs, a plugin + catalog, signature material, signed native images, legacy CI metadata/tar, and single-root OCI layouts. + Preserve captured bytes as testdata; do not regenerate them to hide incompatible changes. +- [ ] Add shared sentinels for invalid config/reference, conflict, not found, unavailable publication, + unsupported capability, integrity failure, missing document, invalid packaging, and unsafe path. + Preserve existing sentinel wrapping in compatibility adapters. + +**Done when:** Every changed public contract has a test fixture, and a compatibility test can detect +a changed plugin manifest digest or planfile archive even before the implementation is replaced. + +## A01 — Configuration and backend registry + +- [ ] Add `ArtifactConfig` and `ArtifactRepositoryConfig` to `pkg/schema`, attached as + `AtmosConfiguration.Artifacts`. Include repositories, kind, identity, options, packaging, + permissions, optional catalog, and defaults.metadata; keep backend option decoding typed. +- [ ] Add global/repository verification policy and repository locking configuration from contracts. + Validate trust restrictions, delegated lock identities, cycles, and required lock scopes. +- [ ] Register repository factories by kind in `pkg/artifact`. Return capability descriptors without + network access. Reject duplicate kinds and unknown kinds; avoid a central backend switch. +- [ ] Normalize local destination paths while loading the declaring config, before imported values + lose their source directory. Track the origin of `options.path` independently of unrelated merged + repository fields. Do not expand key-pattern templates during general config rendering. +- [ ] Validate root/prefix containment, packaging matrix, octal strings, label values, catalog + references, and required backend options. Reject `permissions` for non-local archives. +- [ ] Resolve identity bindings lazily when a real operation requires credentials; no credentials + or destination directories should be created merely by describing config or running dry-run. +- [ ] Update embedded config JSON Schema and relevant stack/scaffold step schemas together with + Go types. Verify YAML, JSON, mapstructure, imports, and config describe round trips. + +**Done when:** The PRD's repository examples load unchanged, two same-named components can share one +collection under different logical paths, and imported local roots resolve to their declaring files. + +## A02 — ORAS transport behind shared OCI interfaces + +- [ ] Promote the existing pinned `oras.land/oras-go/v2` dependency to direct use; do not combine + this migration with an unrelated version upgrade. Consult documentation for that pinned version. +- [ ] Add OCI-standard descriptor/reference/blob interfaces alongside the existing Registry API. + Use `opencontainers/image-spec` descriptors and `go-digest`, not ORAS remote.Repository or + go-containerregistry images in the new public transport boundary. +- [ ] Implement resolve, fetch, exists, push, tag, list tags, delete manifest, graph copy, layout + storage, and referrer discovery using ORAS. Keep bounded reads/digest verification at the boundary. +- [ ] Adapt existing authentication resolution to ORAS credentials. Preserve explicit identity + failure behavior, ambient precedence, host/realm scoping, token refresh, and TLS/proxy options. + Source and destination clients must not share a credential cache across different identity scopes. +- [ ] Bridge old Registry methods onto the new transport while legacy callers still construct + go-containerregistry images/indexes. Serialize their existing raw bytes; never rebuild an old + release with ORAS PackManifest just to transfer it. +- [ ] Migrate plugin release/catalog reads and writes, discovery, version resolution, signatures, + and shared source pulls in small groups. Preserve `ResolveImage`, `PinDigest`, and `ProcessImage`. +- [ ] Add multi-reference layout operations without weakening `LoadIndex`'s existing single-release + validation. Preserve OCI 1.1 referrers and the existing fallback behavior required by signatures. +- [ ] Reconcile writes by resolving the destination after ambiguous failures; do not blindly retry + mutations or downgrade to anonymous authentication. Cancellation must stop graph traversal. +- [ ] Extract reusable verifier/signer primitives into `pkg/signature`, retaining compatibility + wrappers and existing plugin trust requirements. Keep attachment transport in the adapters. + +**Done when:** All A00 OCI fixtures retain their digest and verification result through registry → +layout → second registry movement with distinct credentials and the source switched off. + +## A03 — Artifact preparation and shared operation service + +- [ ] Implement a producer registry. Start with generic files; kind-specific producers add validation + and exclusions later. Preparation resolves name/version, labels, documents, inventory, and digests. +- [ ] Snapshot input into bounded disk staging before committing publication. Detect files changing + during capture and fail instead of publishing an inventory that describes different bytes. + Close readers and remove owned temporary data on success, error, or cancellation. +- [ ] Reuse archive traversal/writers with explicit deterministic options: sorted entries, normalized + incidental timestamps and ownership, preserved ordinary modes, and safe symlink handling. + Existing `type: archive` behavior remains unchanged. +- [ ] Recognize the reserved document paths and explicit overrides. Validate explicitly requested + files, compute document descriptors, and attach generated metadata without checksum recursion. +- [ ] Build publish/pull/copy/inspect/list/search/delete/show orchestration over Repository interfaces. + Pin mutable references once, check capabilities before mutation, and return partial operation + results alongside errors so reporting can describe completed work. +- [ ] Implement intent comparison/no-op, force conflicts, plugin immutability admission, operation + timestamps, publication journals, and retry reconciliation from the contracts companion. +- [ ] Call repository Sign/Verify/Pin/Lock methods through common orchestration. Enforce source and + destination trust requirements before copy/consumption; preserve evidence for the actual subject. + Reject insufficient lock guarantees and retain explicit ownership across retries. +- [ ] Implement extraction staging and replacement. Reject traversal and escaping symlinks; never + merge a newly pulled tree over old files or delete files outside the explicit output destination. +- [ ] Expose dry-run preparation results without calling repository mutation/auth operations. + +**Done when:** One service test suite drives all entry points; unchanged input yields the same +content identity, unchanged publication is a no-op, and failed reads/writes never report success. + +## A04 — Local directory and archive repositories + +- [ ] Implement separate `local/dir` and `local/archive` factories in `pkg/artifact/local`. + Do not redirect the existing CI `local/dir` name to the new copied-directory behavior. +- [ ] Use stable-root reference records, immutable metadata records, and adjacent document files + as specified in contracts. Directory payloads remain ordinary copied files; archives contain + one artifact and have adjacent metadata/documents plus embedded portable copies. +- [ ] Hold per-reference and resolved-location locks over comparison, preparation, and commit. + Publish completion last; use same-filesystem staging and atomic file replacement where possible. +- [ ] Persist dated resolved locations. A subsequent read/delete must not rerender the key pattern + with a new timestamp. Detect two references attempting to own the same resolved path. +- [ ] Apply quoted octal archive-file permissions separately from payload modes. Verify behavior + on Unix and document the platform capability error for explicitly unsupported requirements. +- [ ] Implement metadata-only inspect/show, record enumeration, verified digest reads, and scoped + deletion. Keep private managed records out of the user's pulled payload. +- [ ] Implement detached signature storage, shared signer/verifier delegation, immutable pinning, + and filesystem LockHandles. Signature additions must not mutate their signed Record. + +**Done when:** Real temporary-directory tests cover every allowed packaging format, dated roots, +ordinary executable files, safe links, interrupted writes, output replacement, and missing sidecars. + +## A05 — Share blob transport with CI and add S3 + +- [ ] Extract transport-only local/S3 object I/O from `pkg/ci/artifact` into `pkg/artifact/blob`. + The low-level API handles streams, size, headers/tags, conditional writes, listing, and deletion; + it must not require CI Metadata. Keep legacy metadata encoding in CI adapters. +- [ ] Preserve existing CI factories and wire formats using adapters over that transport. Keep + GitHub artifact storage CI-specific and avoid a reverse dependency from artifact core to CI. +- [ ] Implement `aws/s3` with archive payloads or per-file objects, stable-root records, document + objects, configured prefix/key_pattern, and explicit object tags. Copy the configured tags to + all objects in the publication; portable labels remain independent. +- [ ] Use streaming/multipart transfers and bounded worker concurrency. Abort multipart uploads + owned by a canceled operation. Handle pagination, zero-length files, and endpoint overrides. +- [ ] Use S3 conditional writes for completion/reference updates where supported. An endpoint that + cannot enforce a needed condition must fail that operation clearly, not silently overwrite. +- [ ] Implement digest-addressed evidence objects, Pin, and persistent owner-checked S3 lock claims. + Keep crash recovery explicit; no automatic lease expiry without fencing and no Object Lock confusion. +- [ ] Preserve lazy identity resolution and endpoint/TLS behavior. New generic repository identity + failures must fail closed; do not copy the legacy missing-resolver ambient fallback into it. + +**Done when:** An S3 fake proves request ordering, tags, pagination, and conditional conflicts; +an opt-in S3-compatible emulator proves archive/files round trips and existing planfile compatibility. + +## A06 — OCI collection repositories and foreign graphs + +- [ ] Implement registry and layout factories over A02. Map root/prefix/name/version consistently; + expose capability differences without requiring users to configure each artifact name separately. +- [ ] Assemble new Atmos manifests with the PRD media types, config metadata, deterministic payload + descriptors, and separate README/CHANGELOG blobs. Supply the captured creation time explicitly. +- [ ] Support selective config/document fetches and multi-reference layout enumeration. Never fetch + payload layers merely to inspect labels or show a document. +- [ ] Implement OCI content-graph import/export and cross-backend transport through a standard OCI + layout payload. Keep original manifests and related required signature material byte-identical. +- [ ] Distinguish generic metadata from foreign OCI image/plugin schemas. Inspect foreign objects + without pretending they have an Atmos config blob; do not repackage them on copy. +- [ ] Implement the same Sign/Verify/Pin/Lock contract as local/S3, using OCI attachments and the + shared verifier. Bind image signatures to the original manifest/index; bind portable Record + signatures through the addressed config descriptor. Support delegated repository-scope locks. +- [ ] Add genuine keyless and configured-public-key verification tests. Missing, tampered, + wrong-subject, wrong-issuer, untrusted-key, and expired/missing offline trust must fail policy. +- [ ] Enforce plugin-format immutability for generic publication/copy before a tag can be overwritten. + Preserve caller policy/verification checks on every read and write path. +- [ ] Implement safe deletion semantics: no silent deletion of sibling tags that share a manifest. + Report unsupported tag-only deletion when the registry cannot provide the required semantics. + +**Done when:** Generic artifacts and legacy plugins coexist beneath one collection prefix, a +multi-platform image survives S3/layout transfer, and metadata/document-only request budgets pass. + +## A07 — Command space + +- [ ] Add a `cmd/artifacts` CommandProvider with publish, pull, copy, inspect, list, search, delete, + sign, verify, and show readme/changelog subcommands through the existing command registry. +- [ ] Implement the exact flag/step mapping in contracts using StandardParser and normal Atmos + configuration precedence. Reuse list/table formatting and the existing label parser. +- [ ] Keep data on stdout, progress on stderr, and errors in the established structured error path. + Emit result envelopes on failure when structured output is requested; retain a nonzero exit code. +- [ ] Embed command usage Markdown and add website command/configuration pages. Label all new + functionality accurately; the PRD is not a substitute for shipped command help. +- [ ] Test with `cmd.NewTestKit(t)` and injected service dependencies; add real temporary-local CLI + tests so command validation, serialization, and service wiring are exercised together. + +**Done when:** Every PRD command example has a tested counterpart and a documented error path; +no command invents a default repository, `latest`, catalog mutation, or interactive CI prompt. + +## A08 — Discovery cache and optional catalogs + +- [ ] Extract reusable source-scoped snapshot/cache mechanics from `pkg/plugin/discovery` into a + neutral `pkg/discoverycache` package; retain plugin-specific schemas, keys, defaults, and policy. + Prove plugin cache compatibility before switching its consumer to shared mechanics. +- [ ] Enumerate managed local/S3 records and OCI-layout refs. For OCI registries use supported + repository enumeration scoped to prefix, or the configured catalog; explicit-name tag listing + works without either. An unsupported collection scan fails, not an empty successful result. +- [ ] Apply exact-match AND label filtering to published-version metadata and substring matching + to name/description. Sort results deterministically and retain repository alias and pinned digest. +- [ ] Implement the 15-minute cache, explicit refresh, offline semantics, credential-scope keys, + policy reevaluation, bounded reads, and atomic local snapshot replacement from the PRD. +- [ ] Add a separate `AtmosArtifactCatalog` manifest using `pkg/manifest`. Carry artifact kind, + source, versions, digests, labels, and document descriptors; do not impose plugin SemVer rules. +- [ ] Add catalog build/publish/add/remove/sync commands specified in contracts. Catalog writes + require one externally serialized publisher; release publication never invokes these commands. +- [ ] Treat catalog payloads as discoverable data, never trust grants. Validate every referenced + location against the destination repository and current auth/source policy before access. + +**Done when:** Multiple configured repositories search together, cache hits fetch no payloads, +catalog changes never rewrite artifacts, and incomplete/unauthorized results are never presented as complete. + +## A09 — Workflow and hook integration + +- [ ] Add `ArtifactHandler` to `pkg/runner/step`; implement Validate, Execute, KnownFields, and + RequiresTTY=false. Use flat step fields and the existing kind: step hook bridge. +- [ ] Add shared schema fields without repurposing `name`, `kind`, `output`, or `Artifacts`. + Cover WorkflowStep, Task conversion, custom-command decoding, script step dispatch, and hook `with`. +- [ ] Resolve source/destination paths through the runner's existing working-directory helpers; + resolve repository roots independently from their config origin. +- [ ] Return typed operation results in a documented StepResult.Metadata entry and strings in + named Outputs; use the same payload as CLI JSON. Return partial results even on service error. +- [ ] Retain a preparation/operation identity across wrapper-level retries so dates, versions, and + reports do not change on every attempt. Do not multiply the runner and transport retry budgets. +- [ ] Exercise before/after Terraform hooks, workflow execution, custom commands, and direct script + dispatch. Preserve conditions, continue/on_failure, and cancellation semantics. + +**Done when:** The same publication produces the same metadata and result across CLI, workflow, +and hook calls, including retry/failure cases; no shelling out to `atmos artifacts` is required. + +## A10 — CI summaries and machine outputs + +- [ ] Add `pkg/artifact/report` as a renderer over typed results, with no repository mutations or + extra inspection requests. Add embedded templates and CITemplatesConfig.Artifacts overrides. +- [ ] Use the existing template loader's explicit/convention/default order, with an error for a + missing explicit override. Route completed Markdown through `ci.Reporter.Summary`. +- [ ] Collect at execution boundaries: CLI owns standalone reports; the runner owns step reports + after retries; hooks reuse that runner collector and retain an empty duplicate hook body. + Pass collectors explicitly through runner services/options, never through context values. +- [ ] Deduplicate by logical operation ID and retain the final attempt. Buffer concurrent results, + order by scheduled step/operation position, and flush from a deferred finalizer on error too. +- [ ] Render outcomes, destinations, digest types, sizes, duration, and collapsible details. Escape + Markdown, mask secrets, avoid signed URLs, and enforce provider byte limits with visible truncation. +- [ ] Include signature subject, verified signer/key claims, verification status, and actual lock + guarantee. Distinguish signed from verified and a content checksum from publisher authentication. +- [ ] Wire `ref`, `location`, digest fields, size, and outcome to existing CI output mechanisms when + enabled. Keep JSON/YAML stdout valid and leave PR comments/checks opt-in through their own features. + +**Done when:** Success, no-op, force replacement, partial failure, canceled siblings, disabled +reporting, template failure, and provider failure all have deterministic tests and correct exit status. + +## A11 — Scaffold and component publishing + +- [ ] Add a scaffold producer and `atmos scaffold publish` provider delegating to the generic service. + Validate scaffold configuration without generating output or executing scaffold hooks. +- [ ] Add a component/terraform producer with visible default exclusions for `.terraform/**`, + `*.tfstate`, `*.tfstate.*`, `*.tfplan`, and `*.tfplan.*`; retain `.terraform.lock.hcl` and scripts. + Preserve explicit dry-run inventory and do not apply these exclusions to generic backups. +- [ ] Exclude reserved package metadata from generated scaffold output; preserve root README + rendering and user-selected ordinary payload files. Cover existing generator source strategies. +- [ ] Teach existing OCI materializers to select the Atmos payload descriptor rather than trying + to extract every layer, while keeping legacy component/scaffold layer heuristics intact. +- [ ] Prove published components work through existing source provisioning and scaffolds through + existing generation/update flows, including documentation without pulling the payload. + +**Done when:** A component with an executable helper and a scaffold with both README roles round-trip +without leaked distribution files or altered generated output. Other configuration/module trees +remain publishable as generic files; native cache/export adapters are not silently added here. + +## A12 — Verified toolchain package integration + +- [ ] Add `atmos toolchain publish` and explicit artifact-repository installation selection as + specified in contracts. Reuse tool resolution and lockfile validation; do not introduce another + tool registry format or automatic upstream fallback. +- [ ] Package the original upstream asset with its resolved install definition, platform identity, + relevant lock entry, and available checksum/signature evidence. Retain original asset bytes. +- [ ] Verification metadata is evidence to recheck, not a boolean trust grant. Require pinned + checksums and rerun applicable verification policy before publication and installation. +- [ ] Inject an asset source into the existing installer before it consults network registries. + Feed the verified original package to normal extraction/install/PATH logic. Update URL cache + sidecars transactionally and retain the upstream identity separately from artifact location. +- [ ] Keep adapter wiring outside artifact core and installer core to avoid cyclic imports. + Fail explicitly when an offline verification dependency or install definition is absent. + +**Done when:** With empty toolchain caches, blocked upstream HTTP, and a pinned local/OCI/S3 +artifact source, install the fixture tool and execute it; a tampered asset or incompatible platform fails. + +## A13 — CI compatibility bridge and release completion + +- [ ] Add explicit repository selection to native CI planfile storage without changing legacy + `planfiles.stores` selection. Follow the selector contract; never guess a repository from a + missing legacy store or silently fall back after download-not-found. +- [ ] Adapt CI bundle metadata to generic records while preserving old metadata validation and + serialized sidecars for existing callers. Add file-mode fields with a legacy-compatible default; + preserve existing planfile tar bytes when no mode is supplied. +- [ ] Complete all backend/entry-point scenarios in [validation](./validation.md), including restore + with upstream disabled, negative auth tests, and CI result parity. +- [ ] Update CLI/config/hook docs, examples, schema tests, and generated mocks. Record the exact + semantics of unsupported capabilities and offline data requirements. +- [ ] Run repository checks, targeted suites, full regression checks, and website build; inspect + generated changes. Follow the release skill for labels, blog, roadmap, and milestone requirements. + +**Done when:** The validation matrix is green for new behavior and compatibility, required docs are +published with the feature, and no deferred capability is presented as implemented. + +## Follow-on work queue + +Keep these outside V1 and link future designs back to the use-case inventory: Terraform provider +and module cache export/restore with mirror indexes; native stack/Atmos-config dependency closure; +installed toolchain snapshots; bulk image mirroring/platform selection; complete air-gap bundles; +additional transports; SBOM generation and aggregate inventories; retention/prune; structured +changelog aggregation. Generic file upload alone is not completion evidence for these integrations. diff --git a/docs/prd/artifacts/validation.md b/docs/prd/artifacts/validation.md new file mode 100644 index 00000000000..2028fc2e81b --- /dev/null +++ b/docs/prd/artifacts/validation.md @@ -0,0 +1,204 @@ +# Atmos Artifacts validation and agent handoff + +**Status:** Required future implementation checks; no box is marked complete by writing this plan. +Read the [work packages](./implementation.md), [contracts](./contracts.md), and [PRD](../artifacts.md). + +## Test strategy and fixtures + +Use small real files, temporary repositories, deterministic clocks, injected failures, and bounded +fake transports. Generate interface mocks with the repository's mockgen workflow. Favor observable +behavior over tests that repeat implementation details. Do not manually edit existing golden files. + +Create reusable fixtures in the owning package testdata, not under unrelated existing golden trees: + +- File package: nested files, an executable helper, empty file/directory, confined symlink, Unicode + filename, root README, and distinct package README/CHANGELOG. +- Invalid package: traversal, symlink escape, duplicate normalized paths, conflicting reserved + metadata, input changing during capture, and unsupported special-file entries. +- Generic Record: canonical byte vectors and expected content/intent/record digests, including + alternate map insertion order and fixed creation times. +- OCI graph: tiny single-platform image and two-platform index with distinguishable child bytes. +- Legacy plugin: existing format manifests/index, executable fixture, catalog, lock, and genuine + Sigstore evidence. Keep expected digest values stable through transport migration. +- Signed generic artifact/native image: distinct subject types, genuine trusted and untrusted + signer fixtures, public-key-only evidence, incorrect issuer, and mismatched subject digest. +- CI planfile: pre-change tar and metadata plus old store/default/priority configurations. +- Toolchain: tiny original release archive/raw executable, two platform definitions, lock entries, + URL-cache sidecars, checksum/signature evidence, and an executable version-reporting fixture. + +Reuse `pkg/oci/ocitest` and `pkg/oci/signature/signaturetest` rather than live public registries or +identity providers. Fake S3 must model pagination, conditional state, multipart cancellation, object +tags, and response ambiguity; a map that always succeeds does not test publication correctness. +Opt-in emulators supplement these tests and must not be required for ordinary unit suites. + +## Required conformance matrix + +Run the shared repository conformance suite against each registered backend, with capability-specific +assertions for supported operations and explicit errors for unsupported guarantees. + +| Behavior | Local directory | Local archive | S3 | OCI layout | OCI registry | +| --- | --- | --- | --- | --- | --- | +| Multiple names/versions per configured root | Required | Required | Required | Required | Required | +| Publish, pull, inspect, show, delete | Required | Required | Required | Required | Required within declared delete capability | +| Packaging | Copied files | tar/tgz/zip | Files and tar/tgz/zip | Packaged files and native graphs | Packaged files and native graphs | +| Metadata/document reads without main payload | Required | With sidecars | Required | Required | Required | +| Collection enumeration | Managed refs | Managed refs | Paginated managed refs | Layout refs | Native enumeration or catalog; error otherwise | +| Generic Sign/Verify/Pin | Detached evidence | Detached/portable evidence | Evidence objects | Layout evidence | Referrer evidence | +| Lock | Host filesystem | Host filesystem | Repository owner claim | Host filesystem | Host guard; repository scope through configured delegate | +| Missing required lock/evidence capability | Error | Error | Error | Error | Error | + +Test local directory ↔ archive, local ↔ S3 in both packaging modes, local ↔ OCI, registry ↔ layout, +registry ↔ second registry, and native OCI graph ↔ S3/layout carrier. Byte-identical native graphs +retain resolved digests; generic repackaging retains Record/content identity but may change the +representation/OCI envelope digest. Assert both rather than assuming every digest is interchangeable. + +## Acceptance scenarios + +### Configuration and interface parity + +- [ ] Load every PRD YAML example; validate identical values across YAML/JSON/mapstructure. +- [ ] Load two imported configs with different source directories and merged repository fields; + each destination path uses the declaration that supplied that path. +- [ ] A remote config must not accidentally make an ephemeral download directory a write root; + until local anchoring is explicit, require a locally declared/absolute destination override. +- [ ] Reject unknown kinds, invalid option combinations, traversal, catalog/identity mismatch, + unquoted numeric permissions, malformed labels, and delegated lock cycles before mutation. +- [ ] Configuration inspection and dry-run perform no credential requests, destination directory + creation, signature publication, or remote object writes. +- [ ] Reuse one alias for components, scaffolds, configuration trees, and generic backups. + +### Publication, integrity, and recovery + +- [ ] Repeated preparation produces canonical records/inventory and deterministic payloads. +- [ ] An identical republish is unchanged even when invocation time/run metadata differ. +- [ ] Changed content, labels, documents, or modes conflict without force; plugin releases remain + immutable with force. Existing/incoming plugin formats both trigger the guard. +- [ ] Resolve a tag, then move it during a read/copy; consume only the originally pinned subject. +- [ ] Inject interruption before payload, during payload, after metadata, and after server-side + commit before its response; retry reconciles correctly and never advertises partial success. +- [ ] Inject two writers for one ref and two refs with a colliding key_pattern. Verify ownership + and conditional guards; never silently bind one reference to another artifact's payload. +- [ ] Date rollover and wrapper-level retries reuse captured time and resolved locations. +- [ ] Pull replacement removes stale destination files only inside the requested output root. +- [ ] Tampered metadata, payload, document, or OCI blob fails its declared integrity check. +- [ ] Direct digest reads never return replacement bytes under an old digest; unavailable retained + records report failure. Scope deletion to the selected artifact and its exclusively owned files. +- [ ] Archive/file member permissions remain distinct; executable bits survive. Oversize records, + unsafe links, unsupported target modes, and extraction errors are explicit. + +### Signatures and coordination + +- [ ] Every backend's Sign/Verify/Pin/Lock implementation passes the same interface tests; backend + differences are represented by capabilities, not omitted operations that claim success. +- [ ] Sign a portable Record, copy it through local/archive/S3/OCI packaging, and verify the same + signed subject after restoring it. Attachments bind to the correct record/config. +- [ ] Copy a signed native image/index through an offline carrier and a second registry, disable + the source, and verify the original root digest and preserved evidence at the destination. +- [ ] Reject wrong digest, wrong issuer, untrusted identity/key, corrupt signature, absent required + evidence, unsupported format, expired trust, and missing required transparency/signing-time proof. +- [ ] Public-key verification cannot satisfy an OIDC identity policy. Existing plugin rejection of + public-key-only bundles under identity policy remains unchanged. +- [ ] Signing alone reports signed, not policy-verified; plain hashing reports integrity, not signer + authenticity. `verify` without a usable trust configuration does not return a green result. +- [ ] Independent source/destination policies both apply to copy; repository overrides and force + cannot weaken governing requirements. Cached verification results are reevaluated under new policy. +- [ ] Preserve a signed multi-platform root when selecting a child; a reconstructed root must not + inherit an original signature claim. Required evidence is fetched before materialization. +- [ ] Offline verification uses only cached/provisioned trust and evidence, with a transport that + fails any network request. Missing trusted roots cannot be filled from the artifact itself. +- [ ] S3 owner claims reject competing owners and wrong-token release; they never expire into an + unfenced takeover. Local guards exclude competing processes and release when their owner exits. +- [ ] OCI host guards are not reported as repository-scope locks; a required distributed guard + fails without a capable delegate. Delegated lock access uses its own identity. +- [ ] Canceled lock acquisition does not leak ownership; pending publication recovery requires + valid ownership and matching intent. Document operator recovery of abandoned S3 claims. +- [ ] Pins persist immutable digests/evidence descriptors without changing existing consumer lock + schemas or granting trust merely because the lockfile says a prior run succeeded. + +### Discovery and catalogs + +- [ ] Label pairs use AND semantics on artifact-version labels, not S3 tags or repository defaults + retroactively applied to older artifacts. Keep aliases visible and sorting deterministic. +- [ ] Named OCI tag listing works without a catalog; unsupported collection discovery returns a + capability error. Add no second repository list in configuration. +- [ ] Fresh-cache search performs zero registry requests; refresh reuses metadata by digest and + removes disappeared refs only after a complete refresh succeeds. +- [ ] Offline cache works without payload downloads, rejects corrupt/missing snapshots and refresh, + and does not cross auth realms/host bindings or bypass current policy. +- [ ] Catalog build reads metadata only. Publish/sign/delete of an artifact never implicitly edit + its catalog. Catalog removal unlists without deleting content or rewriting locks. +- [ ] Concurrent catalog updates expose stale revision/digest conflicts; serialized workflows + read latest inside the guard. Do not claim a preflight check alone prevents races. +- [ ] Plugin discovery/cache behavior stays compatible after common mechanics are extracted. + +### Producers and existing consumers + +- [ ] Scaffold generation/update preserves root README behavior and excludes reserved distribution + metadata. `scaffold publish` validates without executing generation hooks. +- [ ] Component source provisioning reads new Atmos payload descriptors and old formats; helpers + remain executable, Terraform transient data is excluded, and generic backups retain requested files. +- [ ] Toolchain publication verifies the original package against lock data. Offline installation + succeeds with empty caches and blocked upstreams, using the normal extractor/installer. +- [ ] Reject toolchain tampering, wrong platform, missing install metadata/evidence, and cache URL + collisions. Recheck configured verification; a recorded "verified" string is not sufficient. +- [ ] Existing CI stores retain metadata validation, tar defaults, key patterns, selection order, + and not-found behavior. Only explicit `artifact/repository` opts into the new repository bridge. +- [ ] The planfile adapter preserves original key/query information and does not overwrite unrelated + generic or plugin artifacts. Existing CI fixtures continue reading through their old adapters. + +### CLI, steps, hooks, and summaries + +- [ ] Run matching operations through CLI, workflow, custom command, script dispatch, and kind: step + hooks; verify identical typed results, named outputs, source paths, and repository roots. +- [ ] Retry, on_failure/continue, cancellation, and failure after some completed operations preserve + operation identity and report the correct final outcome exactly once. +- [ ] Summary rendering covers published, unchanged, replaced, downloaded, copied, deleted, signed, + verified, and failed outcomes; includes actual signature subjects and lock guarantees. +- [ ] Sequential/parallel/nested steps produce one deterministically ordered report per operation; + the hook bridge does not render a second copy. Existing unrelated job summary content remains. +- [ ] CI disabled, summaries disabled, output variables disabled, and explicit CI-mode overrides + honor existing gates. Disabled summaries perform no extra metadata/evidence fetches. +- [ ] Custom template precedence, missing override errors, secret masking, Markdown escaping, + provider size limits, and safe links work without corrupting JSON/YAML stdout. +- [ ] Reporting errors preserve artifact exit status; operation/verification errors remain failures. + +## Commands and validation gates for implementation + +Run package-scoped checks after each change; the following package paths include proposed packages +and are executable once those packages exist. Do not run unimplemented paths and count them as skipped. + +```shell +go test -short ./pkg/artifact/... ./pkg/signature/... +go test -short ./pkg/oci/... ./pkg/plugin/oci/... ./pkg/plugin/catalog/... ./pkg/plugin/discovery/... +go test -short ./pkg/ci/artifact/... ./pkg/ci/plugins/terraform/planfile/... +go test -short ./cmd/artifacts/... ./cmd/scaffold/... ./pkg/generator/... +go test -short ./pkg/runner/step/... ./pkg/hooks/... ./pkg/ci/... +go test -short ./pkg/toolchain/... ./cmd/toolchain/... +go test -race -short ./pkg/artifact/... ./pkg/discoverycache/... +atmos build +atmos lint --changed +atmos test +``` + +Also run the website build for shipped command/config docs and the opt-in OCI/S3 emulator round +trips. Use current repository instructions for regeneration and CI checks. Target the required +patch coverage with behavioral tests; do not add redundant tests solely to inflate coverage. + +Known prior workspace observations, not acceptance waivers: a documentation-only baseline build +passed, while a broad test invocation encountered missing Packer, plugin/formatting failures and +a package timeout. Recheck the current baseline before implementation, record exact failures, and +distinguish environment/baseline failures from regressions. New artifact tests must still pass. + +## Completion record for the next agent + +For each A00–A13 package, record: + +- [ ] Implementation commit(s), changed interfaces, and migration/compatibility notes. +- [ ] Exact successful test/check commands, relevant fixtures, and skipped emulator prerequisites. +- [ ] Failure reproduction, baseline comparison, and owner for any unresolved check. +- [ ] Public docs/schema/help changes and release-label implications where applicable. +- [ ] Remaining work package IDs and the next concrete action. + +V1 is complete only when every required backend and public surface passes this matrix and all +claimed offline flows work with fresh local state and unavailable upstreams. An upload success, +metadata search result, detached signature file, or apparent lock object alone is not that proof. From 98188ea99a57cef74927a630bb8b9260f9291c2c Mon Sep 17 00:00:00 2001 From: Erik Osterman Date: Sun, 1 Nov 2026 00:18:51 -0500 Subject: [PATCH 4/7] docs(artifacts): escape the pipe in the contracts Documents row Co-Authored-By: Claude Sonnet 5.5 --- docs/prd/artifacts/contracts.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/prd/artifacts/contracts.md b/docs/prd/artifacts/contracts.md index cc2b24adac9..6330a888530 100644 --- a/docs/prd/artifacts/contracts.md +++ b/docs/prd/artifacts/contracts.md @@ -123,7 +123,7 @@ repository is an error; only collection list/search intentionally default to all | List | `list [name] [--repository R] [--labels L] [--refresh]` | `action: list`, optional `artifact_name`, `repository`, `labels`, `refresh` | | Search | `search [text] [--repository R] [--labels L] [--refresh]` | `action: search`, `query`, `repository`, `labels`, `refresh` | | Delete | `delete --repository R` | `action: delete`, `ref`, `repository` | -| Documents | `show readme|changelog --repository R` | `action: show`, `document`, `ref`, `repository` | +| Documents | `show readme\|changelog --repository R` | `action: show`, `document`, `ref`, `repository` | Publishing defaults to kind `files`; specialized convenience commands supply their kind. Name precedence is explicit flag, producer metadata, source basename. Version precedence is explicit From 2e3285c837f0293eb20ec988d660612d34d57ec6 Mon Sep 17 00:00:00 2001 From: Erik Osterman Date: Tue, 10 Nov 2026 16:28:54 -0600 Subject: [PATCH 5/7] docs(artifacts): tighten extraction, location overlap, and planfile key contracts Co-Authored-By: Claude Opus 5.5 --- docs/prd/artifacts/contracts.md | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/docs/prd/artifacts/contracts.md b/docs/prd/artifacts/contracts.md index 6330a888530..b23f492ef0d 100644 --- a/docs/prd/artifacts/contracts.md +++ b/docs/prd/artifacts/contracts.md @@ -244,6 +244,10 @@ the artifact itself. Missing/expired trust or missing required proofs fail witho Expose verification subject, evidence digests, policy fingerprint, verified claims, and failure reason in inspect/verify output and CI summaries. Explicit `verify` is the metadata/evidence-only check; payload-consuming operations additionally verify every consumed content descriptor. +Extraction matches the signed Record's inventory exactly: reject unlisted user payload members and +duplicate paths, and match each path, type, mode, link target, and file digest. Only generated +`.atmos/artifact/` distribution members excluded from the inventory are permitted, and they are +verified against their own descriptors. Tests must prove an archive with an extra member is rejected. ### Mutation locks versus dependency locks @@ -326,7 +330,9 @@ Search ignores staging/journal entries. Direct digest lookup checks the Record's The operation journal records captured time, intent, expected prior reference, resolved location, and prepared representation checksums. It is an implementation recovery record, not a public catalog. Use local reference/location locks or S3 conditional ownership records. Reserve a location before -overwriting it; another reference cannot claim it even when its key_pattern collides. +overwriting it; another reference cannot claim it even when its key_pattern collides. Identical, +ancestor, and descendant payload paths conflict: claims check and reserve overlapping locations +atomically, so a force replacement or deletion of a parent cannot remove a child another reference owns. Publication sequence: @@ -439,8 +445,10 @@ planfiles: Keep the existing `--store`, default, priority, and environment detection behavior. The adapter references `artifacts.repositories.shared`; it does not duplicate its backend options. Map legacy -keys to `ci/planfiles/:` and retain the original key plus full CI metadata -in typed kind data. Implement existing query predicates from that data, not approximate label matches. +keys to the fixed reference `ci/planfiles/:current`, so Download, Delete, Exists, and +GetMetadata resolve by key alone and a new upload for the same key replaces the current version under +the writer guard. Retain the original key, commit SHA, and full CI metadata in typed kind data. +Implement existing query predicates from that data, not approximate label matches. Preserve the existing planfile overwrite contract only inside this explicit adapter; it cannot override plugin releases. Old store configurations and serialized archives are unchanged. From b363a31ef691a0ffefea6efc36f41bbc0b792c13 Mon Sep 17 00:00:00 2001 From: Erik Osterman Date: Wed, 11 Nov 2026 15:35:28 -0600 Subject: [PATCH 6/7] docs(artifacts): protect control paths and sidecar claims --- docs/prd/artifacts/contracts.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/docs/prd/artifacts/contracts.md b/docs/prd/artifacts/contracts.md index b23f492ef0d..3b4d01fbec5 100644 --- a/docs/prd/artifacts/contracts.md +++ b/docs/prd/artifacts/contracts.md @@ -306,8 +306,9 @@ Tests must distinguish archive-file mode from member mode and from filesystem AC ## Local/S3 record layout and publication lifecycle -Below the configured root reserve `.atmos-artifacts/v1/` for managed records. Reject payload names -and key patterns that enter this namespace. Use SHA-256 of canonical name/version for private +Below the configured root reserve `.atmos-artifacts/v1/` for managed records. Reject payload paths +and key patterns that overlap this namespace in either direction, including ancestors such as +`.atmos-artifacts`. Use SHA-256 of canonical name/version for private reference keys so names containing slashes cannot collide with control paths. ```text @@ -331,8 +332,9 @@ The operation journal records captured time, intent, expected prior reference, r and prepared representation checksums. It is an implementation recovery record, not a public catalog. Use local reference/location locks or S3 conditional ownership records. Reserve a location before overwriting it; another reference cannot claim it even when its key_pattern collides. Identical, -ancestor, and descendant payload paths conflict: claims check and reserve overlapping locations -atomically, so a force replacement or deletion of a parent cannot remove a child another reference owns. +ancestor, and descendant locations conflict. Claims atomically reserve the payload and its sidecar +paths together, rejecting payload-to-sidecar and sidecar-to-sidecar overlaps as well as payload +overlaps. Thus a force replacement or deletion cannot remove another reference's payload or sidecars. Publication sequence: From 83fa41eb81654f1923545c48ef4029c88212956e Mon Sep 17 00:00:00 2001 From: Erik Osterman Date: Fri, 13 Nov 2026 06:36:53 -0600 Subject: [PATCH 7/7] docs(prd): state how configured artifact roots stay isolated A payload path may not contain a .atmos-artifacts segment at any depth, so a write or deletion through one configured root can never reach the records or references of a root nested below it. Payload claims cover one root and are not checked across roots, so configured roots must not nest or overlap where both publish payloads. Co-Authored-By: Claude Opus 5.5 --- docs/prd/artifacts/contracts.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/docs/prd/artifacts/contracts.md b/docs/prd/artifacts/contracts.md index 3b4d01fbec5..73469940b60 100644 --- a/docs/prd/artifacts/contracts.md +++ b/docs/prd/artifacts/contracts.md @@ -308,7 +308,10 @@ Tests must distinguish archive-file mode from member mode and from filesystem AC Below the configured root reserve `.atmos-artifacts/v1/` for managed records. Reject payload paths and key patterns that overlap this namespace in either direction, including ancestors such as -`.atmos-artifacts`. Use SHA-256 of canonical name/version for private +`.atmos-artifacts`. Reject a payload path that contains a `.atmos-artifacts` segment at any depth, +in any letter case, so a root's payload can never enter the control namespace of another +configured root nested below it, and a write or deletion through one root can never change another +root's records or references. Use SHA-256 of canonical name/version for private reference keys so names containing slashes cannot collide with control paths. ```text @@ -335,6 +338,9 @@ overwriting it; another reference cannot claim it even when its key_pattern coll ancestor, and descendant locations conflict. Claims atomically reserve the payload and its sidecar paths together, rejecting payload-to-sidecar and sidecar-to-sidecar overlaps as well as payload overlaps. Thus a force replacement or deletion cannot remove another reference's payload or sidecars. +Claims cover the references of one configured root. Payload claims are not checked across roots, so +configured roots must not nest or overlap where both publish payloads; such a configuration is +unsupported. The control-namespace rule above still protects every root's records and references. Publication sequence: