Composition: services: plural + remote source: per entry, build pipeline resolves source per service

[krisrowe]

Tier 3: Composition — services: plural with optional remote source: per entry

Summary

Add an opt-in plural form services: to gapp.yaml, mutually exclusive with the existing service: singular. Each entry can optionally declare a remote source: block (git/github + required ref:) that pulls code from an external repo at deploy time. The build pipeline resolves source per service into a temp build dir; local-source services git archive from the deployment repo, remote-source services shallow-clone at the specified ref.

Background

Problem

A "solution" today is forced to be one repo. To compose a deployment from multiple cooperating services, the operator must:

• Fork each upstream service to add gapp.yaml. Forks rot, drift from upstream, and require periodic re-merges.
• Vendor source into a monorepo. Loses upstream identity, conflates ownership, scales poorly when upstream services have independent maintainers.
• Maintain every upstream service themselves. Even when the service is better-maintained externally.

This is a known pain point with no clean workaround in the current model. The deployment of a multi-service system shouldn't force every component into one repo or require forking.

Established pattern

"Declarative composition" / "GitOps composition" is the industry-standard answer:

• Terraform modules support source = "git::https://github.com/.../...?ref=v1.2.3" — modules are reusable, the consumer composes.
• Helm umbrella charts reference subcharts by chart-name + version pulled from external repos.
• Nix flakes declare inputs = { upstream-tool.url = "github:owner/repo/ref"; } and compose from there.
• Bazel WORKSPACE uses git_repository(...) for external code refs.
• Crossplane Compositions layer external resource definitions.

Common shape: a deployment manifest references upstream content by URL + pinned ref, configures per-service env/secrets/scaling, and deploys as one unit. The "deployment repo" becomes the integration/configuration layer; service code lives wherever it lives.

Benefits

Without composition	With composition
Fork upstream to add gapp.yaml → fork rots, drifts	Upstream stays clean; the deployment repo pins a ref
Vendor source into a monorepo → loses upstream identity	Upstream identity preserved as a citation
Every operator must maintain every service	Operator maintains the deployment; upstream maintainers maintain their code
Upgrade = manual git merge from upstream	Upgrade = bump ref: in gapp.yaml

Proposed Changes

1. Schema additions (strict superset)

• New top-level key services: (list, optional).
• Mutually exclusive with service: (singular). Validation enforces.
• Each services: entry has all the fields a service: block has, plus optional source:.

# Simple case (95%): unchanged from v3.0
service:
  entrypoint: my_package.server:app

# Composed case: explicit opt-in
services:
  - name: api
    entrypoint: api.server:app          # local code, this repo

  - name: worker
    source:
      github: someone/worker
      ref: v1.2.3                       # tag, branch, or SHA
    env:
      - name: WORKER_TOKEN
        secret: { name: worker-token }

2. source: block

• Optional within a services: entry. Default is local code (this deployment repo).
• Source identifier: one of git: <https url> or github: <owner/name> (shorthand).
• ref: is required when source: is given. No silent default to main or HEAD. To "track default branch," write ref: main explicitly.

Pinning recommendation:

Ref kind	Reproducible?	Notes
Pin to a SHA (ref: 7f3c1a9)	Fully	Same commit = same image. No drift.
Pin to a tag (ref: v1.2.3)	If tag is immutable (convention)	Trust the upstream's tag policy
Track a branch (ref: main)	No	Re-pulls every deploy; surprise-deploy risk

3. Build pipeline source resolution

For each service in the deploy:

• Local source (no source: block): git archive <ref> of the deployment repo's HEAD or --ref override (per the deploy-flag issue). The working tree is never copied directly.
• Remote source: shallow clone (git clone --depth=1 --branch=<ref> <url>) into a temp build dir, then docker build from that dir. Cleanup on success or failure.

Image tag policy:

• Local source: {deployment-repo-HEAD-sha} or {--ref} value
• Remote source: {source.ref}, resolved to a SHA when ref is a branch, for cache-key stability

Cache: skip rebuild if image already exists in Artifact Registry for the resolved tag.

Auth for private remotes: lean on gh CLI auth for v1 (anyone running gapp likely already has gh auth status configured). HTTPS + GitHub token. SSH-key / agent forwarding deferred.

4. Status remains cloud-only

Per the tier 2 issue, status reads service_urls map from TF outputs and probes /health per entry. The deployed services are visible regardless of where their code came from — composition is invisible to status. Status has no manifest dependency.

5. Min-barrier preservation

The single-service case stays identical:

• No new required fields
• service: (singular) syntax unchanged
• New optional fields (services:, source:, per-service name:) are absent unless the operator opts in
• Same progressive-disclosure principle as owner, env, --force, paths:

v-3 Contract Preservation

The schema additions are a strict superset — existing v-3 manifests with service: (singular) validate and behave identically. No label/bucket change.

The build pipeline gains new code paths for remote source resolution; existing local-source builds are unaffected.

The TF template refactor is the only place where v-4 risk lives. The new template must support N services as a single root module while preserving byte-identical resource addresses for an equivalent single-service deploy. Use terraform moved {} blocks if addresses must change.

v-4 is not on the table without a separate design discussion. Mitigation priority: preserve addresses → moved {} blocks → targeted state mv → template carve-out → defer/scope-cut. Only after exhausting these does a contract bump enter the conversation, and only with explicit owner sign-off. See the tier 2 issue for the full escalation rules.

Out of Scope

• Per-service ref override via CLI (--ref worker=v2.0.0). Bumping a remote pin = edit gapp.yaml and commit.
• A gapp services update <name> --ref=... subcommand to edit pins programmatically. Future enhancement; not v1.
• SSH-key / agent forwarding for private remote sources. v1 leans on gh CLI HTTPS auth.
• Caching cloned remote sources between deploys. Initial implementation re-clones each time. Optimization later.

Work Breakdown

• Schema: add services: (list), mutual-exclusion validation with service:, source: block (one of git/github + required ref)
• Manifest helpers: get_services(manifest) normalizing both singular and plural forms to a list of service dicts
• Build pipeline: source resolver dispatching local vs remote; temp build dir lifecycle (creation, build, cleanup on success/failure)
• Local source path: git archive <ref> of deployment repo (existing logic; unchanged)
• Remote source path: shallow clone at ref; integrate gh CLI auth for HTTPS pulls
• TF templates: refactor to emit N services from a single root module; preserve resource addresses for single-service equivalence (or use moved {} blocks)
• Status: relies on tier 2's outputs-map cloud-only path; no additional changes here
• CLI: gapp deploy accepts composed manifests; --ref semantics per the tier 2 issue
• Tests: schema mutual-exclusion, schema with source: variants (git/github + required ref), manifest normalization, build pipeline source resolution (local + remote), multi-service smoke deploy, status across composed solutions
• Document composition in CONTRIBUTING.md ("composition pattern, when to use, ref-pinning policy, auth model")
• Document composition in the deploy skill (lead with service: singular as canonical simple shape; show services: plural as opt-in for composition)
• Capture proven gh CLI auth pattern in code comments at the source-resolver call site, including what worked vs alternatives considered (SSH agent forwarding, embedded tokens) so a future session does not redo the discovery
• Capture the terraform moved {} mitigation pattern in CONTRIBUTING.md as the standard approach when a template needs to rename resource addresses

Acceptance Criteria

• An operator can deploy a solution composed from one or more remote-sourced services without forking those repos.
• Existing v-3 single-service deployments continue to work without any gapp.yaml change.
• gapp deploy for a composed solution produces a single TF state covering all services.
• gapp status for a composed solution probes every service's /health, with no manifest dependency.
• Schema validation rejects service: and services: declared together; rejects source: without ref:.
• Pinning a remote source to a SHA produces a fully reproducible build (same SHA in → same image out).