[doublewhy]

Summary

This PR completes #965 on its own. It now carries #1262's five commits (the generic half), which #1262 opened separately, so #1262 is closed in favor of this PR. On dev, a declared variable can be filled by name from APTL's own process environment or the project .env, and a same-named shell variable can override an authored fixture. A matching name is not authority to read a credential, so each declared variable now comes from exactly one source that names its consumer:

• generated: an artifact output, only for a variable declared with value_from, and only from an artifact bound to this node.
• authored: the scenario's value, delivered exactly. A declaration without a value is delivered empty.
• grant:<kind>:<variable>: for a value-less operator_secret, redacted or secret_fixture, a grant in aptl.json (deployment.environment_grants) for this pack, node and variable. The grant names one process-environment variable or one .env key.
• startup-adapter:project-env-file:<variable>: failing a grant, a name that the admitted pack's own startup adapter writes to .env.

A missing or empty source stops aptl lab start in the backend preflight, before realization creates or changes any scenario network, volume or container. Logs, errors, argv and run evidence name each source, never a value or a digest of one.

Generic path: base-container nodes (1130ec8..f04ed25, formerly #1262)

• _environment_bindings.py (new) does the binding. _realization_preflight in realize() runs it first. It reads each grant source once, and warns about a grant for the admitted pack that matches no value-less secret.
• _credential_sources.py (new) holds EnvironmentGrant and the source models. ProcessEnvironmentCredentialSource moves there from config.py, which keeps that file under Sonar's length limit, and aptl.core.config still exports it. The config provenance projection is an allowlist, so grants stay out of run provenance.
• get_backend() passes the grants to both Compose backends. BaseContainerSpec gains environment_sourced and environment_generated.
• _compose_base_substrate.py writes the bindings through fix: write generated environment files safely #1256's writer. When a node binds at least one variable, its container gets an aptl.environment.names label. A container is reused only when that label lists exactly the bound names and its Config.Env holds each bound value.

Compose path: generated services (57a1c02..d9feadf)

A generated Compose service took each value-less operator_secret or secret_fixture through a ${NAME} reference, which Compose resolves, like bare environment and build.args entries, from its own process environment before the --env-file that APTL binds (#874). APTL ran every docker compose command with its whole environment, and one invocation shares that environment across every service and Compose file of the project.

• _compose_environment.py (new) holds the Compose side, including the generic preflight mixin. The preflight binds every Compose service that APTL generates, before any mutation, with the same rules. A value that a single-quoted Compose env-file line (fix: write generated environment files safely #1256) cannot carry, one that contains a single quote or ends in an odd number of backslashes, stops the start like a missing source.
  • Each service gets its own owner-only env file at .aptl/realization/sourced-environment/<sha256(service)[:20]>.env, written through fix: write generated environment files safely #1256's no-follow writer, and a generated compose.override.yml attaches it to that service alone. A value reaches only the service that declares it. Files that a request does not use are removed. A Compose file set assembled without the preflight refuses a service that needs a grant.
  • Logs name each service's sources, for example Compose service db environment: DB_PASSWORD from grant:process-environment:LAB_DB_PASSWORD.
• docker_compose.py: _run, _run_streaming and _run_with_input give docker compose a bounded environment: the Docker client's own variables (toolchain, locale, TLS trust, temporary, XDG_*, credential-helper paths, Windows profile and system directories, DOCKER_*, Compose client options, proxies), the APTL_* settings including APTL_HP_*, and BUILDKIT_* and BUILDX_*. COMPOSE_FILE, COMPOSE_PROFILES and COMPOSE_PROJECT_NAME stay out. No bound value is added, and a scenario variable named HTTPS_PROXY leaves the client's proxy alone. The SSH backend keeps its DOCKER_HOST.
• Scenario and pack text is a Compose literal. Every Compose file that APTL writes and that can carry a $ from that text goes through compose_literal() in _compose_runtime_config.py, which writes each $ as $$: the generated base model, its port, stateful, content and image overrides, and the Kali capture model. A command, entrypoint, environment value, mount path, content destination, pack file name, image reference or host address therefore reaches the container as written. The pre-start model check reads config --no-interpolate back through compose_readback().
• _compose_runtime_config.py: _environment_config derives from declared_environment() and holds authored values only. _compose_observability.py: the Grafana credential check reads .env only.
• scenario_bundle.py: env_pack_bundle() refuses with EnvPackError an env-pack whose root holds docker-compose.yml, which APTL used unchanged as the base model, so its services skipped the preflight; the released TechVault packs ship none. A project-tree scenario's own docker-compose.yml, the development path, is used as written, and docs/deployment.md says to run only a trusted scenario from the project tree.

Evidence: no value or digest of a granted or adapter value (75fa8fb)

After start, _raes_runtime_environment_observation.py read each value-less secret_fixture off the container, and RAES disclosed it as an unsalted SHA-256 commitment over the variable name and the value. That commitment reached the runtime snapshot and the REP-001 run record, where a guessed value could be checked against it offline. For the released TechVault pack, these were the six Wazuh INDEXER_*, API_* and DASHBOARD_* credentials on wazuh-manager and wazuh-dashboard. observe_environment now never reads a value-less secret_fixture whose value comes from a grant or the adapter (the names that declared_environment() marks as sourced) and leaves it out of the disclosed environment. A value-less operator_secret or redacted variable is still disclosed by presence only, and a value that the scenario authors keeps RAES's commitment. RAES's _PROTECTED set is unchanged.

TechVault's environment authority is open, so its readback gate passes as before. Under exact authority, RAES expects a value-less secret_fixture to be realized empty, so the gate rejects a grant or adapter value after the start. It already did so when the readback disclosed the commitment. #1332 tracks that case.

docs

docs/deployment.md gains a "Scenario Environment Grants" section: which variables take a grant, the grant format, the per-service env files and the values they cannot carry, the bounded Compose process environment, literal scenario and pack text, the project-tree path, and what logs and evidence record. The three new modules and the new test module have tracked-file inventory rows.

How each done-when item of #965 is met

• Explicit bindings on both paths with no inherited fallback: _environment_bindings.py for base containers, _compose_environment.py and the bounded docker compose environment for Compose services.
• Authored values survive same-named ambient variables: authored values are delivered exactly on both paths, and Compose no longer interpolates scenario or pack text.
• Generated outputs and grant or adapter values reach only their consumers, and a missing binding fails before realization mutates anything: the preflight on both paths, the per-node value_from filter, and per-service env files.
• Canary tests for wrong consumer, pack and run identity, missing binding and changed source on both paths: see Verification.
• Logs, argv and evidence name sources only: describe_bindings() logs, values reach Docker only through owner-only env files and never through argv, and the readback above.

Breaking change for operators

A shell variable or same-named .env key no longer reaches a scenario node or Compose.

• A value-less operator_secret, redacted or secret_fixture needs a grant or a value from the pack's own startup adapter. Otherwise aptl lab start stops in the backend preflight, before realization creates or changes any scenario network, volume or container. Steps that run before realization, such as Suricata volume seeding, may already have run. The error names the node and the variable, and either the pack or the source that has no value.
• Any other value-less variable on a base-container node is delivered empty instead of omitted, even when the image sets a default.
• If you overrode a .env value such as INDEXER_PASSWORD or GRAFANA_ADMIN_PASSWORD by exporting it before aptl lab start, set it in .env instead.
• A ${NAME} in scenario or pack text of a generated Compose service now reaches the container as written. A pack that relied on Compose filling it must declare the variable and bind it.
• A project-tree scenario has no pack identity, so no grant applies to it. APTL_* settings still work from the environment.

The released TechVault pack (0.1.1) and its participant-study variant declare no base-container environment. They bind all seven of their value-less Compose variables through their own adapter, so their start needs no grant. The only $ in their generated base model and its port and stateful overrides is the container-shell "$@" in misp-redis's entrypoint, which Compose passed through before and still does. The title's ! and the BREAKING CHANGE: footers of 1130ec8 and 57a1c02 mark this break. The squash message keeps the commit messages, so drop them if you don't want release-please to cut a major release.

Verification

The branch is rebased onto dev 1abe440. git range-diff shows all eleven earlier patches unchanged and no conflicts: 04c5133, 33037ef, bf15b7c, 8d8185c and 717cef6 (#1262's head) are now 1130ec8, a7db21a, d4343e4, 370d803 and f04ed25. 331782b, 06762d9, 389b9dc, a7ac44d, 6d18d92 and 6fc6706 are now 57a1c02, 5078bb4, 063190c, 0b9706c, 8f986a4 and d9feadf. 75fa8fb is new. Run from the worktree with Python 3.12 and requirements/dev.txt, as CI installs it.

• New evidence tests, at 75fa8fb:
  • tests/test_raes_runtime_observation.py::test_environment_sourced_secret_fixture_leaves_no_value_or_digest (open and exact). A container holds a canary in a value-less secret_fixture beside an authored plain variable. The disclosure lists only the authored variable, and neither the canary nor the hex of RAES's commitment of it appears in runtime_snapshot_record_payload() of the snapshot. The open gate passes, and the exact gate rejects, as before.
  • tests/test_compose_environment_bindings.py::test_the_released_techvault_readback_records_no_adapter_value_or_digest. The released TechVault realization, read back with adapter values in place, discloses none of the six Wazuh names, discloses MISP's ADMIN_KEY by presence with no commitment, and its remaining commitments are exactly the ones that RAES projects from the scenario's own values.
  • Regression proof: with _raes_runtime_environment_observation.py reset to d9feadf, both tests fail. The disclosure lists ['FIXTURE', 'FOO'], and the TechVault readback carries a commitment for API_PASSWORD and six more. With the name assertion disabled, the commitment's hex is found in the run record payload.
• 22 test modules with -k "not integration" and -n 8: 829 passed. They are the test modules this PR changes and every module that imports the observation, binding, Compose-environment, provisioner or repro code or the TechVault realization helpers. -m integration on tests/test_raes_runtime_observation.py, tests/test_compose_environment_bindings.py and tests/test_techvault_static_gate.py (the RAES scenario validation gate's tests): 2 passed. The first two have no integration-marked tests, and none of the three has fuzz tests. The full suite was not run locally; CI's Python lanes run it.
• ruff check src/ (the C901 gate) passed, and so did pre-commit run --files with both .pre-commit-config.yaml and .pre-commit-ci.yaml on the four files 75fa8fb changes. Vale 3.15.2 on docs/deployment.md: no alerts. mkdocs build --strict was not run locally; CI's Docs job runs it.
• Canary coverage from the earlier commits, unchanged by the rebase:
  • Base containers (tests/test_raes_runtime_environment.py, tests/test_compose_base_substrate.py, tests/test_config.py): a grant binds nothing for another node, another pack or a project-tree run; planted process and .env values do not override authored ones; a missing binding is refused by realize() after zero Docker commands; a changed source recreates the container; a source that has gone missing fails before any inspect or removal; no secret value reaches process argv.
  • Compose (tests/test_compose_environment_bindings.py, tests/test_capture_apparatus.py, tests/test_scenario_bundle.py): same-named GITHUB_TOKEN and INDEXER_PASSWORD canaries never reach a docker compose process; a grant for db lands only in db's 0600 env file; missing binding, wrong consumer, pack identity, an empty adapter value and values an env file cannot carry each stop realize() after zero Docker commands with no value in the error; consecutive starts write the grant's current value; a reused backend never carries an earlier request's value; every generated file has only escaped $, checked through docker compose config with .env and process canaries; a pack with a root docker-compose.yml is refused.
• Regression proofs for those tests, run before the rebases on patch-identical commits (earlier SHA in brackets):
  • 5078bb4 [bff9465]: with its writer and readback files reset to 57a1c02 [c13e856], three of its new tests fail. docker compose config rendered the canary command with the two .env values, the client proxy and an APTL_* value.
  • 063190c [3560b95]: its tests on 5078bb4's source fail four tests. The inline text and the pack file were bound at paths holding .env values, and the file-set and capture tests found a live $.
  • 0b9706c [526d19d]: with scenario_bundle.py from before it, the root Compose file test fails (DID NOT RAISE EnvPackError).
  • With _realization_primitives.py from before fix: write generated environment files safely #1256's fa630b7, the value that ends in a backslash passes the preflight and the start runs Docker commands.
  • f04ed25 [7128aa7]: the generic value_from cases fail when the preflight's consumer filter is removed, or when a missing generated output is delivered empty.
• CI run 38213179533 on 75fa8fb passes all 38 jobs, among them Python tests + coverage (6541 passed, 72 skipped, 241 deselected, 51 passed in the fuzz run, and 2 passed in the env-file readback step), DEP-008, the RAES scenario validation gate, Docs, the dependency and Trivy scans, and the SonarCloud quality gate with its new-issue step. CodeQL run 38213176754, PR title lint and GitGuardian pass as well; all 45 checks on the PR pass.

Requirement UIDs: none (bug/maintenance).

Closes #965