Repository navigation
Fresh aptl lab init + aptl lab start fails at the bind-mount preflight for every scenario (5.2.0 and dev) #951
Copy link
Copy link
Closed
Labels
bugSomething isn't workingSomething isn't workingscope:aptl-coreAPTL platform/range capability independent of one scenario.APTL platform/range capability independent of one scenario.user-facingDirectly changes a user journey or user-visible outcomeDirectly changes a user journey or user-visible outcome
Milestone
Description
Finding
The documented quick start fails on a clean directory in about nine seconds, for the default scenario and for all four curated catalog scenarios, from the released 5.2.0 wheel on PyPI and from dev source (verified 2026-09-04):
pipx install aptl-labs && aptl lab init my-lab && cd my-lab && aptl lab start
[lab start] Checking bind-mount sources.
Bind-mount issue: Service 'misp': bind-mount source './config/soc_certs/misp/server.pem' does not exist. ...
Lab start failed.
error: Bind-mount pre-flight failed: (8 errors: misp, thehive, shuffle-frontend, cortex)Every developer checkout that has ever booted a lab carries the gitignored config/soc_certs/ tree, so the failure is invisible there. CI runs aptl lab init but never aptl lab start. The released 5.2.0 wheel ships no scenarios/techvault-operational.sdl.yaml and carries the same code paths.
Root cause
Two defects, both introduced by the env-pack cutover (#908 / #875):
_load_stateful_artifact_ownershipinsrc/aptl/core/lab.pysubstitutesDEFAULT_RAES_SCENARIO(scenarios/techvault-operational.sdl.yaml, deleted in feat(scenario): declare TechVault content in the SDL via the env-pack (#875) #908) when no scenario is selected and returns silently when that file is missing, soctx.stateful_artifact_ownershipstays empty._step_generate_soc_certsthen skips host generation becausescenario.source == "env-pack", and_step_check_bind_mountsfails.tests/test_lab.py::test_missing_scenario_leaves_legacy_ownershipcodifies the silent return. Callingadmitted_stateful_artifact_ownership(..., scenario_path=None)in the same directory resolves 15 ownership entries, including the four SOC certificate consumers.- Even with ownership loaded the preflight still fails. It validates
docker-compose.ymlbind destinations (/etc/nginx/certs/cert.pem,/etc/thehive/keystore.p12,/etc/lab-ca/lab-ca.pem) while the pack realizes those certificates at/opt/aptl/soc-certs/..., and_stateful_realization_owns_mountrequires both source and destination to match (Remove silent sudo + root-owned cert bind mounts (generate certs user-owned) #677). The compose file no longer starts those services; the preflight checks a file the realization does not use. Verified by running_check_bind_mountswith the pack-declared ownership applied: still 8 errors.
Required work
- Derive the bind-mount preflight from the admitted realization (the same plan the backend applies), or exempt by admitted generated source path while keeping the service match. No TechVault, service-name, or path special case.
- Stop substituting a deleted default path. When
scenario.source == "env-pack"and no scenario is selected, use the same pack resolution the start path uses. - Replace the test that codifies the silent return with one that runs the step sequence from a fresh
aptl lab initdirectory with the env-pack default and no pre-existing generated state. - Add a CI job that builds the wheel, installs it into a clean venv, runs
aptl lab init, bootstechvault-observability-core, runsaptl lab stop -v --yes, and asserts zero project containers, networks, and volumes remain. Three containers; it fitsubuntu-latest. - Publish 5.2.1 once green.
Acceptance criteria
-
pipx install aptl-labs && aptl lab init d && cd d && aptl lab startpasses the preflight for the default scenario and every catalog scenario from a clean directory. - No lab-start step validates
docker-compose.ymlmounts for services the admitted plan realizes from the pack, or the check is derived from the admitted plan. - CI boots and tears down the smallest scenario from the built wheel on every PR.
- 5.2.1 is published with the fix.
Relationships
- Introduced by feat(scenario): declare TechVault content in the SDL via the env-pack (#875) #908 (Declare scenario content in the SDL instead of pointing at repository paths #875). Blocks Verify full TechVault from a clean package install #870.
- Related: lab teardown/boot robustness: a partially-up or lingering range causes confusing re-boot failures #905 (closed), OpenRAE/hub#3 (clean-machine rehearsal against released artifacts).
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't workingscope:aptl-coreAPTL platform/range capability independent of one scenario.APTL platform/range capability independent of one scenario.user-facingDirectly changes a user journey or user-visible outcomeDirectly changes a user journey or user-visible outcome
Activity
Brad-Edwards commentedon Sep 4, 2026
🛠️ Picked up by /implement - driver claude, branch
951-bind-mount-preflight, 2026-09-04T04:48:13.249Z.Brad-Edwards commentedon Sep 4, 2026
gc workflow phase recorded:
preflight(issue #951). Posted by the MCP server to enforce ordering between workflow steps (issue #794 MVP-2). Do not edit or delete — used by downstream tools to gate phase prerequisites.Brad-Edwards commentedon Sep 4, 2026
gc workflow phase recorded:
plan(issue #951). Posted by the MCP server to enforce ordering between workflow steps (issue #794 MVP-2). Do not edit or delete — used by downstream tools to gate phase prerequisites.Plan — fix the fresh-start bind-mount preflight (#951)
Verified root cause
Reproduced locally from
devsource in a freshaptl lab initdirectory (defaultscenario and
--scenario techvault-observability-coreand--scenario bounded-participant-agency-techvault): all three fail the preflight with thesame 8 errors.
Three defects, one shared cause: lab-start steps guess scenario facts from the config
ceiling or a hardcoded path instead of reading the admitted realization.
_load_stateful_artifact_ownership(src/aptl/core/lab.py:1163) substitutesDEFAULT_RAES_SCENARIO(scenarios/techvault-operational.sdl.yaml, deleted in feat(scenario): declare TechVault content in the SDL via the env-pack (#875) #908)when
scenario_path is None, then returns silently because that file is missing. So onthe default env-pack path nothing is ever admitted and
ctx.stateful_artifact_ownershipstays empty. Callingadmitted_stateful_artifact_ownership(..., scenario_path=None)in the same directoryresolves 15 entries.
_step_check_bind_mountsvalidatesproject_dir/docker-compose.yml. Measured: for anenv-pack bundle the pack ships no
docker-compose.yml, sobase_compose_file()generates the base model from the realization (24 services, zero relative binds);
the stateful/content overrides bind absolute paths under
DeploymentBackend.realization_root. The in-treedocker-compose.ymlis not an inputto the run at all. The preflight is parsing an unrelated file.
_scenario_is_env_pack(lab.py:1203) readsconfig.scenario.source, so an explicit--scenario <catalog id>run — whichresolve_scenario_bundle()resolves asproject-tree — still skips host-side SOC cert generation. That is why the catalog
scenarios fail too: certs are skipped and the static compose is checked.
A fourth, compounding fact: the preflight filters services by
config.containers.enabled_profiles()(the operator ceiling,soc: trueby default)rather than the profiles the admitted plan selects.
techvault-observability-coreselectsonly observability, yet the preflight validates MISP/TheHive/Shuffle/Cortex.
Design
Binding guardrails are the Step 2.5 note
docs/architecture/issue-951-fresh-env-pack-start-preflight.md. The forecast for this workalready exists in
docs/architecture/issue-550-soc-selected-profile-preflight.md:and
raes_realization._load_profile_indexalready states the rule this fix restores:"the Compose model is a scenario-declared input, so it is read from the bundle, never the
engine checkout."
Admit once, then read the admitted facts.
_step_load_configresolves the bundle,parses, plans, and interprets exactly once;
_step_start_containersapplies that sametarget/plan instead of re-planning. Today the bundle is staged and planned twice per lab
start (I observed 5 staged pack copies accumulate under
.aptl/staged-packs/, one perplan) at a measured 92 s each.
src/aptl/backends/raes.py— expose anadmit_raes_scenario(...)seam returning theexisting
(bundle, target, execution_plan, realization)produced by_plan_scenario;start_raes_scenario(..., admitted=...)skips re-planning and goes straight to_apply_with_backend_retry. No new DTO for artifacts, no second parser, no new exceptiontype — the guardrails forbid all three.
src/aptl/backends/_raes_scenario_queries.py— deriveselected_profiles_for_scenarioand
admitted_stateful_artifact_ownershipfrom one shared projection over analready-admitted realization, so query and apply cannot disagree.
src/aptl/core/lab.py_load_stateful_artifact_ownership→_load_admitted_start_surface: passesctx.scenario_paththrough unchanged (noDEFAULT_RAES_SCENARIO, no silentreturn), caches bundle root,
ScenarioSourceKind, admitted selected profiles andartifact ownership on
_LabStartContext, and fails closed onEnvPackError/OSError/ValueError/TypeErrorwith a redactedLabResult._scenario_is_env_packreads the resolved bundle'ssource_kind, not the config flag._step_check_bind_mountspasses the admitted bundle root and the admittedselected profiles to
_check_bind_mounts. For an env-pack bundle the pack has nodocker-compose.yml, so the existingcompose_path.exists()guard makes it a loggedno-op; source existence stays where it already belongs — the backend's post-realization
effective_stateful_model_errors()/ provider output checks, beforecompose up._stateful_realization_owns_mountis deliberately left alone. Once the check reads themodel that actually governs the run, the destination mismatch never arises, so #677's
both-sides match stays intact. Weakening it is explicitly named as an anti-pattern.
Security. No new endpoint, authz decision, or config key. Pack ingress keeps
validate_pack()/validate_pack_content_manifest(); a missing or rejected pack failsclosed as
EnvPackErrorrather than degrading to a deleted path. Immutable input staysunder
ScenarioBundle.root, generated output underrealization_root, operator secrets inproject_dir/.env. Planning errors surface throughredact()/LabResult— no stagingroots, YAML context, or tracebacks. Failing open (skipping the guard on the env-pack
path) is safe only because realization fails closed before
compose up; the guard moves,it does not disappear.
Maintainability. Reuses
resolve_scenario_bundle,ScenarioSourceKind,select_backend_profiles,_plan_scenario,AptlProvisioner.realize_plan,base_compose_file,observe_project_runtime,project_scoped_volume_names. No scenarioname, service allowlist, TechVault path, or certificate filename anywhere in the diff.
Extensibility. The seam is the request-scoped
ScenarioBundleplus one admittedExecutionPlan. A new pack identity or acquisition mechanism changes the resolver inputonly; lab steps gain no branches.
Whole-repo view. Touches lab orchestration, the RAES query/apply seam, CI, and tests.
make policy(Vale) and mkdocs nav cover the new architecture note. Release Please ownsthe version —
CHANGELOG.md/pyproject.tomlare not edited.Work items and TDD paths
raes.py+ shared projection in_raes_scenario_queries.py_load_admitted_start_surface(no deleted-default substitution, fail closed)_scenario_is_env_packfrom the resolved bundle source kind_step_check_bind_mountsuses admitted bundle root + admitted profilestest_missing_scenario_leaves_legacy_ownership; add fresh-init step-sequence testlab init→ boot →lab stop -v --yes→ zero-residue CI jobPer the repo plan rules every Python change lands with
pytesttests intests/. No MCP,web,
docker-compose.yml, orconfig/change, so those rules do not bind.Tests
Fast unit tests (default suite) pin the contract without paying the 92 s admission cost:
_load_admitted_start_surfacecalls the resolver withscenario_pathunchanged andnever references
DEFAULT_RAES_SCENARIO; a resolver failure returns a failed, redactedLabResultinstead of empty ownership._step_check_bind_mountsreads the compose model from the admitted bundle root(a bundle root without a compose file makes it a no-op even when
project_dirhas one),and filters by admitted selected profiles, not
containers.enabled_profiles()._scenario_is_env_packis False for an explicit project-tree selection even whenconfig.scenario.source == "env-pack".TestCheckBindMountscases stay green — including the Remove silent sudo + root-owned cert bind mounts (generate certs user-owned) #677 both-sides cases.Integration (
@pytest.mark.integration, named..._integrationso the repo's-k "not integration"fast run excludes it): from a freshaptl lab initdirectory withthe env-pack default and no pre-existing generated state, run the real step sequence
through
_step_check_bind_mountsand assert it passes. It assertsconfig/soc_certs/and.aptl/are absent at the start, per the guardrail against treating a developer's ignoredgenerated state as a fixture.
CI gate (DEP-008)
New
clean-install-lab-bootjob in.github/workflows/checks.yml, ubuntu-latest,read-only permissions, SHA-pinned actions:
--no-deps. Nopip install -e ., no source-checkout CLI, no local integration plugin.aptl lab init ci-lab;aptl lab start --scenario techvault-observability-core(three containers).
aptl lab stop -v --yesin analways()step, then assert zero project containers(including stopped), networks, and volumes, scoped to the validated Compose project
identity using the same label/namespace rules as
observe_project_runtime()and_compose_volume_cleanup— never anaptl-*name prefix, never a daemon-wide prune.A failed start or a failed cleanup fails the job.
This is the executable gate for DEP-008's own final clause — "
pipx install aptl-labsfollowed by
aptl lab initandaptl lab startshall produce a working lab without a gitclone" — which #951 shows is currently false. Step 16 adds the job and the integration
test to DEP-008's
## Traceability. DEP-008 is already ACTIVE, so no status transition.Local validation
CI cannot be the only proof. I will boot and tear down a real lab on this machine from a
fresh
aptl lab initdirectory for the default env-pack scenario and fortechvault-observability-core, then re-run the two catalog scenarios that reproduce today.Docs
docs/architecture/issue-951-fresh-env-pack-start-preflight.mdplus itsindex.md/mkdocs.ymlentries (added by Step 2.5) ship with the diff. No ADR change: the note recordsthat ADR-046/#874/#875/#677 already settle the boundary.
Out of scope, flagged for your call
With ownership now populated on the env-pack path,
_step_generate_certsand_step_sync_credentialsstill run their legacy host-side generators, because the_WAZUH_*_OWNERSHIPconstants use the static compose's dotted service keys(
wazuh.manager) while an env-pack realization names services from its generated base(
wazuh-manager). Behaviour is unchanged by this fix (ownership is empty today, sothose generators already run), and the lab boots correctly — it is redundant work, not a
failure. Retiring it is the "migrate all legacy project-tree producers" non-goal the
preflight note excludes. Say the word and I will fix it here instead.
Brad-Edwards commentedon Sep 4, 2026
gc_codex_review — cycle 1 of 1 (pre-push) on issue #951 (branch
951-bind-mount-preflight)Diff mode: inline — the complete diff was supplied in one prompt
Core review
Verdict:
shipThis is shaped correctly. Architecture-fit: it establishes a single admitted-execution value at the existing RAES boundary and derives bundle location, resolved source kind, profiles, and artifact ownership from that value, keeping preflight and apply aligned while reusing the canonical resolution, diagnostic, profile-selection, and runtime seams. Code-quality: the tests pin resolution precedence, fail-closed admission, propagation without replanning, bind-mount scoping, and the narrow Docker DNS baseline; the wheel-based boot-and-teardown job validates the released artifact. The seam can accommodate additional pre-start projections without introducing another planning pass or foreclosing scenario parameterization.
No blocking findings.
Security review
Verdict:
ship-with-fixesThe change is broadly shaped correctly: it creates a single scenario-admission seam and threads its resolved bundle, profiles, artifact ownership, and execution plan through preflight and apply, preventing divergent re-planning across the startup lifecycle. The cross-cutting boundary spans scenario resolution, bind-mount validation, runtime observation, and clean-install CI without foreclosing alternate scenario sources. The exception is the runtime-listener baseline, where an address-wide exemption weakens the security gate beyond the Docker-owned socket it intends to recognize.
Blocking findings (1):
src/aptl/backends/_runtime_concern_excess.py:209Attacker model: code executing in a monitored container that wants to conceal an unauthorized backdoor or control listener from the SEM-218 runtime gate. Linux permits a workload to bind an unused TCP or UDP port on 127.0.0.11; Docker owns particular daemon-selected resolver sockets, not every socket on that address. Because this predicate discards every observed listener whose address equals 127.0.0.11 before comparing it with declarations, an attacker can run the declared service normally and add an arbitrary listener on that address without triggering excess detection. Subtract only sockets positively identified as Docker's embedded resolver—for example, the exact TCP/UDP sockets corroborated by the namespace's DNS redirect/runtime metadata—and continue evaluating any additional sockets on the address.
Brad-Edwards commentedon Sep 4, 2026
gc_codex_review pre-push cycle 1 of 1 complete for issue #951 on branch '951-bind-mount-preflight'. Posted by the MCP server to enforce the pre-push hard-cap-1 contract (issues #796, #804, #906). Do not edit or delete — used by the next
gc_codex_review(uncommitted) invocation to count cycles.Brad-Edwards commentedon Sep 4, 2026
Review decision record — codex cycle 1 (issue #951)
Reviewer: codex
Cycle: 1
Architectural read:
Blocking findings: 1
Finding 1 —
one-offF1src/aptl/backends/_runtime_concern_excess.py:209Brad-Edwards commentedon Sep 4, 2026
gc_test_quality_review cycle 1 of 1 — issue #951
Reviewer: test-quality (claude-sonnet-5 via gc_test_quality_review)
Branch:
951-bind-mount-preflightCycle: 1 / 1
Findings: 0 (clean run)
Brad-Edwards commentedon Sep 4, 2026
gc_test_quality_review cycle 1 of 1 complete for issue #951 on branch '951-bind-mount-preflight'. Posted by the MCP server to enforce the gc_test_quality_review hard-cap-1 contract (issue #884 follow-up, default lowered in #906). Do not edit or delete — used by the next
gc_test_quality_reviewinvocation to count cycles.Brad-Edwards commentedon Sep 4, 2026
Review decision record — test-quality cycle 1 (issue #951)
Reviewer: test-quality
Cycle: 1
Architectural read:
Blocking findings: 0 (clean run)
Brad-Edwards commentedon Sep 4, 2026
Pre-PR base synchronization
refs/remotes/origin/devatb6928e52fdf6f88d02a4422b3592f4f003f1e3c6already_currentaffaf570a6fd14023a875f2b9726d8953a2f6b1c496a86829d97927059ce034054e3b2e6ae63d3eaBrad-Edwards commentedon Sep 4, 2026
Pre-PR base synchronization
refs/remotes/origin/devatb6928e52fdf6f88d02a4422b3592f4f003f1e3c6already_current848423a55c59890244c0cfc49cdb7a486ec857f0db4c2e29648f3a41d06fa31a4c0309f47462d481Brad-Edwards commentedon Sep 4, 2026
Pre-PR base synchronization
refs/remotes/origin/devatb6928e52fdf6f88d02a4422b3592f4f003f1e3c6already_current1b283d693b8aa2fa1d9126f0669578ba0c892ce8eca26d1bf69deb9c0da03aca966e86a8e1381317Brad-Edwards commentedon Sep 4, 2026
Ready for review — issue #951
PR: #960
Plan: #951 (comment)
Outcome
Installing APTL from PyPI and following the documented quick start now works. Before this,
aptl lab startfailed after about nine seconds with eight bind-mount errors, for the default scenario and every catalog scenario; it stayed hidden because developer machines carry leftover generated files and CI had never booted a lab. Lab start now reads what the scenario it is about to run actually declares. CI now builds the wheel, installs it clean, boots a real lab, tears it down, and proves nothing is left behind.Lab start admits the selected scenario exactly once and derives the bundle root, resolved source kind, selected profiles, and generated-artifact ownership from that single admission, which apply also consumes. That replaces three separate guesses: a deleted default scenario path, the operator's container ceiling, and the configured scenario source. The bind-mount pre-flight now validates the Compose model the run actually applies. Also fixes the SEM-218 excess-listener check, which counted Docker's embedded DNS resolver sockets as undeclared exposure and failed every node declaring a service listener.
In-scope requirements
Proposed state in this PR — authoritative only after merge.
DEP-008(Self-Contained Distribution of Lab Assets from Installed Package) — ACTIVE (proposed) — Already ACTIVE; this PR adds the executable gate for its clean-install clause.DSL-008(APTL Realization of ACES Infrastructure Topology) — ACTIVE (proposed) — Already ACTIVE; adds traceability for the admitted-scenario seam and the realization-observation excess check.Files changed
Added:
docs/architecture/issue-951-fresh-env-pack-start-preflight.mdscripts/ci/assert_project_teardown.pytests/test_lab_fresh_start.pyModified:
.github/workflows/checks.ymldocs/architecture/index.mddocs/requirements/DEP-008/requirement.mddocs/requirements/DSL-008/requirement.mdmkdocs.ymlsrc/aptl/backends/_raes_scenario_queries.pysrc/aptl/backends/_runtime_concern_excess.pysrc/aptl/backends/raes.pysrc/aptl/backends/raes_start_model.pysrc/aptl/core/lab.pysrc/aptl/validation/_live_gate_probes.pysrc/aptl/validation/participant_agency_qualification.pysrc/aptl/validation/participant_agency_readiness.pysrc/aptl/validation/participant_qualification_boundary_environment.pytests/test_bounded_participant_agency_scenario.pytests/test_bounded_participant_runtime.pytests/test_lab.pytests/test_raes_backend.pytests/test_raes_runtime_observation.pytests/test_scenario_bundle_wiring.pyReviews
Traceability reconciliation
Status
Brad-Edwards commentedon Sep 4, 2026
Final report — issue #951 complete
PR: #960
Plan: #951 (comment)
Outcome
Installing APTL from PyPI and following the documented quick start now works. Before this,
aptl lab startfailed after about nine seconds with eight bind-mount errors, for the default scenario and every catalog scenario; it stayed hidden because developer machines carry leftover generated files and CI had never booted a lab. Lab start now reads what the scenario it is about to run actually declares. CI now builds the wheel, installs it clean, boots a real lab, tears it down, and proves nothing is left behind.Lab start admits the selected scenario exactly once and derives the bundle root, resolved source kind, selected profiles, and generated-artifact ownership from that single admission, which apply also consumes. That replaces three separate guesses: a deleted default scenario path, the operator's container ceiling, and the configured scenario source. Also fixes the SEM-218 excess-listener check, which counted Docker's embedded DNS resolver sockets as undeclared exposure and failed every node declaring a service listener. Merged as 170593a; 5.2.1 follows from Release Please on the
fix:title.Files changed
Added:
docs/architecture/issue-951-fresh-env-pack-start-preflight.mdscripts/ci/assert_project_teardown.pytests/test_lab_fresh_start.pyModified:
.github/workflows/checks.ymldocs/architecture/index.mddocs/requirements/DEP-008/requirement.mddocs/requirements/DSL-008/requirement.mdmkdocs.ymlsrc/aptl/backends/_raes_scenario_queries.pysrc/aptl/backends/_runtime_concern_excess.pysrc/aptl/backends/raes.pysrc/aptl/backends/raes_start_model.pysrc/aptl/core/lab.pysrc/aptl/validation/_live_gate_probes.pysrc/aptl/validation/participant_agency_qualification.pysrc/aptl/validation/participant_agency_readiness.pysrc/aptl/validation/participant_qualification_boundary_environment.pytests/test_bounded_participant_agency_scenario.pytests/test_bounded_participant_runtime.pytests/test_lab.pytests/test_raes_backend.pytests/test_raes_runtime_observation.pytests/test_scenario_bundle_wiring.pyReviews
Traceability reconciliation
Requirement-free issue: no UIDs are in scope, so no status transition applied. The two requirements above are traceability anchors for the new CI gate and the realization-observation change; both were already ACTIVE and verified so at merge revision 170593a.
Status