Repository navigation
feat(continuous-ops): fail loudly on what's required: one preflight, a lean config (XWING-2440) - #312
Conversation
…es, with a mirror cache (XWING-2440)
…bility ref is required (an order always names its commit); history is none, a depth or an anchor sha, and writable asks for a worktree. SourceProvider loses checkout/release: they move to the checkout service's providers.
A worktree per order under one cache root, removed on release; the host's forge token rides git's configuration environment, never argv. The git hardening flags move next to GitCheckout, shared.
…XWING-2440) Kept files, no history, read-only go to ArchiveCheckout; everything else to MirrorCheckout. refresh(repo, head) fetches a moved head, evict (idle_days) drops idle mirrors; wired once in Context.services(). ExposureService calls the service. The cache root is locations .mirror_cache, defaulting under the data home.
…fs and objects Review fixes for the mirror cache: - evict skips a mirror whose worktrees/ still holds an entry, so a checkout held longer than idle_days keeps its mirror - a mirror counts as healthy only when refs/ and objects/ exist beside HEAD; a crash mid-rmtree re-initializes instead of failing every fetch - one flock per mirror serializes concurrent fetches (git's ref locks) - the fetch refspec narrows to refs/heads/* and refs/tags/*, dropping refs/pull/* and refs/merge-requests/* - GIT_ALLOW_PROTOCOL=https joins the fetch env (defense in depth against an insteadOf rewrite to ext::) - RepoCheckoutService refuses keep=() (an empty keep set would yield an empty workspace) the same way it refuses a zero history depth - CheckoutRequest and Workspace are re-exported by the context facade for consumers fenced from providers
PR Summary by QodoAdd a shared preflight and fail on missing required inputs
AI Description
Diagram
High-Level Assessment
Files changed (28)
|
Code Review by QodoResolved findings 1.
|
… contracts#102 (726d273)
CheckoutRequest.repo carries a LocalRepoUrl beside HttpsRepoUrl (a file:// URL or an absolute path to a git repository), and RepoCheckoutService routes a local origin to LocalCheckout: a git worktree straight off the repository the caller materialized — no mirror, no fetch, no cache — released by removing and pruning the worktree so the materialized repository re-runs clean. An unhealthy path (no HEAD/refs/objects, bare root or .git/) is refused with a typed error; the https path is unchanged.
…nfig (XWING-2440)
…umers pin one commit
e8ab152 to
09ad226
Compare
CheckoutRequest.repo carries a LocalRepoUrl beside HttpsRepoUrl (a file:// URL or an absolute path to a git repository), and RepoCheckoutService routes a local origin to LocalCheckout: a git worktree straight off the repository the caller materialized — no mirror, no fetch, no cache — released by removing and pruning the worktree so the materialized repository re-runs clean. An unhealthy path (no HEAD/refs/objects, bare root or .git/) is refused with a typed error; the https path is unchanged.
…o feat/fail-loudly-on-what-s-required-one-preflight-a-l
… evict Qodo findings on #311/#313/#318: HttpsRepoUrl validated path segments but not the host, so https://../owner/repo escaped the mirror cache root; GitLab's git smart-HTTP endpoints take HTTP Basic, not the REST PRIVATE-TOKEN header, so private GitLab mirrors fetched unauthenticated (ForgeApi.git_headers shapes auth per transport); the last-checkout touch ran after the worktree's cleanup handler, so a failed touch stranded the worktree; a negative idle_days cleared the whole cache.
CheckoutRequest.repo carries a LocalRepoUrl beside HttpsRepoUrl (a file:// URL or an absolute path to a git repository), and RepoCheckoutService routes a local origin to LocalCheckout: a git worktree straight off the repository the caller materialized — no mirror, no fetch, no cache — released by removing and pruning the worktree so the materialized repository re-runs clean. An unhealthy path (no HEAD/refs/objects, bare root or .git/) is refused with a typed error; the https path is unchanged.
…o feat/fail-loudly-on-what-s-required-one-preflight-a-l
…nd disabled feeds
Historical code-review findingsThis PR has already merged (2026-10-31T00:33:51Z). The findings below apply to reviewed commit [P2] Preserve the deployment process policy in checkout runnersLocation: src/traust_core/v1/context/context.py:771–775 The new checkout service creates an unconditional git allowlist and a 600-second runner instead of using the deployment's process policy; the local provider at line 781 does the same. A deployment that deliberately excludes git from process.allowed_binaries can therefore still execute it through services().checkouts, and a smaller process.timeout_seconds is discarded. This bypasses the existing execution boundary, rather than merely reporting incorrect readiness. Preserve the configured binary allowlist and timeout ceiling when wiring both checkout providers. Evidence: At the pinned head, Config(process=ProcessConfig(allowed_binaries=('wc',), timeout_seconds=0.1)) makes ctx.process.allows('git') false, but Context._wire constructs ProcessRunner(('git',), timeout_seconds=FETCH_TIMEOUT_SECONDS) and the local runner similarly. ProcessRunner._validate enforces its own allowlist, so these fresh runners replace rather than inherit the configured restriction. This entire wiring is added relative to PR312's pinned base. Verification: Executed RequiredPreflight-checkout-repro.py against this exact core head with real git and a disposable local repository: configured runner allows git: False; local checkout succeeded despite denial: True; mirror and local runner timeouts were both 600.0 rather than configured 0.1. CoreCheckout-smoke.py 311 independently produced configured_git_allowed False / checkout_git_allowed True. No application source was changed. [P1] Use HTTP Basic authentication for GitHub git fetchesLocation: src/traust_core/v1/providers/forge_api.py:55–58 The GitHub branch returns the REST API Bearer header for the git smart-HTTP transport. GitHub rejects that header on /info/refs even when the same configured token has repository access, so every mirror checkout with a configured GitHub token fails authentication, including public repositories where the supplied invalid auth prevents anonymous access. Encode the GitHub token as HTTP Basic (for example x-access-token:), as required by the git transport, rather than reusing REST authentication. Evidence: MirrorCheckout._fetch_env obtains ForgeApi.git_headers(repo.host) and installs its Authorization value in git's http..extraheader. At this head GitLab uses Basic but the GitHub return at line 58 still uses Bearer. The new mirror fetch path and this header method are added in PR312's base-to-head diff. Verification: Read-only real GitHub smart-HTTP probes with one valid gh credential returned HTTP 401 text/plain for Bearer and HTTP 200 application/x-git-upload-pack-advertisement for Basic using the same token, on both traust-core and traust-internal. Executed CoreCheckout-auth-probe.py through bash with Homebrew Python; no token was printed or written. An initial Framework-Python HTTPS attempt failed certificate validation before HTTP and was replaced by the functioning Homebrew interpreter. [P2] Serialize mirror eviction with in-flight checkout acquisitionLocation: src/traust_core/v1/providers/mirror.py:96–100 An idle mirror remains eligible for eviction while checkout is fetching it: _fetch holds the sibling .lock, but evict ignores that lock, and checkout updates last-use / registers its worktree only after the fetch. Concurrent cache maintenance can therefore delete a mirror during an active checkout, causing the real git fetch or subsequent worktree creation to fail. Use the same per-mirror synchronization for eviction and hold acquisition protection until the worktree or equivalent live-use marker is established, then recheck idleness/live worktrees before deleting. Evidence: _fetch takes flock at lines 109-110; evict only checks the old traust-last-checkout timestamp and worktrees directory before shutil.rmtree. checkout calls _touch and worktree add after _fetch returns, leaving the old-idle mirror unprotected from eviction. These methods are new in PR312. Verification: Executed CoreCheckout-smoke.py 311 with PYTHONPATH pointing to this pinned head. A real disposable git mirror was marked 40 days idle, and eviction was deterministically interleaved immediately before the actual fetch while _fetch held its lock. evict(30) removed https://github.com/o/r; checkout then raised ProcessError, git exit 128, cannot change to the deleted mirror directory. The repro substitutes a local git origin only to avoid external network dependency. [P2] Accept valid SHA-256 commits in the new local checkout pathLocation: src/traust_core/v1/providers/local.py:37–40 The new local provider accepts GitSha values containing 64 hexadecimal characters but checks them using GitCheckout.reachable, whose private regex accepts only 7-40 characters. Consequently a healthy local repository created with git init --object-format=sha256 can never be checked out: an existing commit is reported as absent before git is even asked. Extend the reachability validator to the same supported SHA shapes as GitSha so the local seam honors its accepted input. Evidence: models.values.GitSha permits SHA-1 or SHA-256 values; providers.checkout._SHA remains [0-9a-f]{7,40}, and reachable short-circuits to false on a 64-character SHA. LocalCheckout and its call to reachable are new relative to PR312's pinned base. Verification: Executed CoreCheckout-smoke.py 318 against this exact core head. Created and committed a real disposable SHA-256 git repository; git cat-file confirmed the 64-character commit exists. LocalCheckout.checkout raised NotFoundError for that same commit. [P2] Fetch requested commits retained only under PR or MR refsLocation: src/traust_core/v1/providers/mirror.py:124–126 The mirror fetches only branch and tag refs, without fetching the requested commit. A valid requested SHA retained solely by refs/pull//head or refs/merge-requests//head is therefore absent from a fresh mirror even though the forge still serves it. Whole-tree checkout then reports NotFoundError, preventing analysis or verification of such PR/MR heads; the same problem affects refresh for a retained non-branch head. Fetch the requested SHA (or its forge change ref) when it is not reachable after the normal branch/tag update, and only then decide the commit is unavailable. Evidence: At this exact head, checkout passes only request.repo to _fetch and checks request.ref afterward. _fetch's only refspecs are +refs/heads/:refs/heads/ and +refs/tags/:refs/tags/ at lines 125-126; neither checkout nor refresh requests the missing SHA. These methods are added in PR312 relative to its own pinned base. Verification: Executed CoreCheckout-pr-ref-smoke.py against pinned core312 be35227. A real isolated git origin contained the requested commit exclusively under refs/pull/1/head, confirmed by git cat-file and git rev-parse. The production MirrorCheckout with a fresh mirror fetched real git branch/tag refs and raised NotFoundError for that existing SHA. The local origin substitution only removes external-network dependence. |
Plan:
plans/traust-standardization/0-continuous-ops/in-progress/22-shaun-fail-loud-preflight.mdStarted by shaun. Draft until ready for review. This PR is the plan's step 3, the traust-core part only: one preflight plus the core rows of the step-1 inventory. The engine gate (step 4) and traust's doctor (step 5) are later PRs pinned to this one.
Contracts pin: 55b69a9a (contracts#102, the requirement vocabulary). Note: core#311 pins contracts#103 (1436d659…) — two open core PRs currently carry different contracts pins; whoever merges second rebases and re-pins; the final pin lands after both contracts PRs merge.
What
One preflight door, built on what core already has (
Toolchain.preflight/preflight_failures,config_completeness_audit) — no second readiness path:Requirement(models.operations, the pack schema'srequires:vocabulary from contracts#102: tools, config keys, recorded facts, feed sources, secrets), validated against the closed vocabularies (traust.yaml's schema keys +model-registry; the recorded-fact artifact names inmodels.artifacts.RECORDED_FACTS).Context.preflight(requirements) -> None(context.preflight): merges the enabled assets' requirements, checks each kind and raisesPreflightError(ConfigError)listing every unmet requirement with its fix in one message:preflight_failures, now carrying the pin so the fix names it);Config.provided, a private attr so the contracts schema keeps owning the public field set),model-registryagainst the registered model catalog;artifacts.latest_imported, read-only, no new query), the fix naming the record door;config_completeness_audit's errors and warnings become findings (the confirmed decision for that row)._secret_hits): a secret scan that cannot run fails the observation instead of warning and recordingNone; a patch fetch inside the quota reserve still defers it (the explicitQUOTA_DEFERREDoutcome, no document kept).Context.deps_devstops being| None: it raises naming the feeds source to enable.ReportingService.lint_threat_modelsrequires the ATT&CK and ATLAS catalogs (no more shape-only fallback); it raises naming the register doors.NotFoundErrorso a batch refresh records the failure per view), never silently absent.Contextalways wires one) instead of warning and excluding them.ledger/identityabsent, the grype DB dir default, the bicep resolution chain, feed sources without a probe (reported unchecked).Deviations to look at:
| Nonefrom the constructor — the constructor-required shape would force every core test fixture that callsservices()(~50 files) to register pack tables, for behaviour traust already guarantees (it always registers both). Same user-visible outcome: no shape-only degradation.ledger.public_key_path unset, which sits next to the keep-soft "ledger/identity absent" row (whose doors still raise on use). If the ledger warning should stay soft, that is a one-line filter.compliance_sources): the door already raises per product (RedHatDocSource.fetch); recording a product → docs slug/version map in core needs a contracts schema, so theNEED(config: product -> docs slug/version)marker stays.enabled: falseexempts an asset at the pack layer (contracts#102); preflight only ever sees the enabled assets' requirements — the engine (step 4) does the filtering.No repository queries were added or changed (the facts check reuses
artifacts.latest_imported), so notest-repositories/perfrun;make testshows the same single Postgres-marked skip as before the branch.Qodo findings addressed
Requirementnow refuses an unknownrequires:kind with aValidationErrornaming the kind and the valid ones, instead ofDto's extra-ignore dropping it. The contracts pack schema already rejects unknown kinds (additionalProperties: false, the requirement vocabulary), so this closes the core-side parsing door.Configrecords the home it was loaded from (Config.config_home, a private attr likeprovided), and preflight's completeness audit examines that home, not whatever$TRAUST_CONFIG_HOMEhappens to point at.load_feedsnow opts outaccess: disabledcached sources the way the source loader always has, so a required feed whose source is disabled fails preflight naming the feeds registry to enable.tools: no toolchain config: Context.register_toolchain()), not aConfigErrorabort before the other checks; every gap still lands in one message.Acquisition, the provenance the adapter door records.except SourceErroraround_secret_hitsis this plan's explicit decision, not a silent drop: a patch-fetch refusal returns the observation carrying the refusal's status (QUOTA_DEFERRED,ERROR, …) and error string, surfaced as the order's outcome; the code comment and_secret_hits' docstring state the contract (a source refusal defers the observation, a missing or failing scanner fails it — partial data must not look complete). Withholding the churn numbers from a refused observation is that fail-loud outcome; the old warn-and-record-Nonepath was the silent one.Reviewer checks (from the plan)
except: return defaultorlog.warningfollowed by carrying on, for anything required.Quality checklist
Legacy files
Each file this plan replaces: reuse (core already does it), retire (the model made it obsolete) or port.
Changed tests
Every existing test edited, with why (each is an inventory row's confirmed decision; new tests are not listed):
tests/v1/context/test_feeds.py::test_no_deps_dev_without_an_enabled_sourcetest_deps_dev_without_an_enabled_source_raises; assertedctx.deps_dev is None→ now expectsConfigErrornaming the feeds sourcedeps_devdoor stops being `tests/v1/context/test_tool_reports.py::test_osv_api_runs_through_the_registry_lookup_and_deps_devresolver=None→ now registers adeps_devsource and asserts the resolver is passed throughtests/v1/providers/tools/test_toolchain.py::test_check_uses_the_pinToolCheckequality now includesexpected="1.6.0"tests/v1/services/findings/test_precedents.py(World.precedentshelper)FindingPrecedents(uow)→ wired with aValidationRecordsreadertests/v1/services/reporting/conftest.py(reportingfixture)tests/v1/services/reporting/test_threat_model.py::test_attack_refs_are_checked_against_the_catalog_when_one_is_registered…_against_the_pinned_catalog; registered only ATT&CK → registers bothtests/v1/services/reporting/test_threat_model.py::test_atlas_refs_are_checked_against_the_catalog_when_one_is_registered…_against_the_pinned_catalog; registered only ATLAS → registers bothtests/v1/services/test_insight_spend.py::test_the_spend_view_is_built_only_on_a_model_registrytest_the_spend_view_without_a_model_registry_names_the_file;ValidationError "'spend' is not built"→NotFoundError "no model registry: …"tests/v1/services/test_insight_attack_coverage.py::test_the_view_renders_through_contextValidationError "not built"before registering → expectsNotFoundError "no ATT&CK catalog: …"tests/v1/services/test_portfolio_observation.py::test_batch_quota_decision_does_not_query_live_quota[info, head, compare]→[info, head, compare, compare_patch]tests/v1/services/test_portfolio_observation.py(compared()helper)compared.patchtests/v1/services/test_portfolio_observation_secrets.py::test_an_unscanned_patch_is_none_and_says_why(7 cases)test_a_scan_that_cannot_run_fails_the_observation(6 cases: the observation now raises, nothing recorded) andtest_a_patch_fetch_inside_the_quota_reserve_defers_the_observation(quota case →QUOTA_DEFERRED, no document)tests/v1/services/test_portfolio_observation_secrets.py::test_scanner_failure_does_not_log_exception_text_or_repository_urlServiceErrorpropagates and nothing is loggedtests/v1/services/test_portfolio_observation_store.py(Observedhelper)default_tools()(a clean gitleaks)tests/v1/observation_support.py(support, not a test)observer()defaulted to no tools and an empty patch at quota 100 → defaults to a registered clean gitleaks, a non-empty patch and a quota above the tripwire reservetools=Noneexplicitlytests/v1/context/test_traust_yaml.py::test_an_empty_traust_yaml_is_the_defaultsConfig.from_home(empty traust.yaml) == Config()→ the twomodel_dump()s are comparedConfignow records the home it was loaded from (a private attr, likeprovided) so preflight audits the loaded home; an empty file's values are still the defaultsNew tests:
tests/v1/context/test_preflight.py(17: one per requirement kind, every gap at once, the audit as findings, dedupe across assets, unknown/duplicate/empty and typo'd-kind requirements rejected, a complete home passes, the audit examines the loaded home, a missing toolchain is one more gap, a disabled feed source fails),test_lint_without_the_catalogs_names_the_register_doors,test_validation_precedents_without_an_evidence_reader_refuse,test_osv_api_without_deps_dev_is_refused,test_the_default_scanner_carries_the_adapter_doors_provenance.Cross-repo coordination
Track:
development. Use these explicit related-repository targets rather than assuming the connected repository'smainis the matching overhaul baseline.Review targets
Integration and rollout
The requirement vocabulary is already merged into Contracts development. The linked Engine and application PRs are the corresponding preflight callers; reconcile their Core pins before rollout.
Branch URLs are compatibility baselines; PR URLs are proposed companion code. Neither proves that a manifest/lockfile pin already contains the changes or that the target is deployed. Honor the prerequisites above, reconcile compatible dependency pins before rollout, and do not dismiss a real compatibility gap merely because another PR is planned.