diff --git a/.github/workflows/action-e2e.yml b/.github/workflows/action-e2e.yml index 3ea17fc37..62367f9e9 100644 --- a/.github/workflows/action-e2e.yml +++ b/.github/workflows/action-e2e.yml @@ -46,7 +46,7 @@ jobs: action-contract: name: Action contract and mirror lock runs-on: ubuntu-24.04 - timeout-minutes: 30 + timeout-minutes: 75 permissions: contents: read steps: diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 7011be886..22bcfd081 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -313,7 +313,7 @@ jobs: integration-cli: name: Integration (CLI) runs-on: ubuntu-24.04 - timeout-minutes: 60 + timeout-minutes: 105 steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: diff --git a/.keyhogignore b/.keyhogignore index da5bbf906..b3abb1ef2 100644 --- a/.keyhogignore +++ b/.keyhogignore @@ -38,6 +38,7 @@ path:docs/assets/keyhog-banner.svg; reason="reviewed docs banner asset; not scan # --- Documentation that quotes findings -------------------------------------- path:CHANGELOG.md; reason="release notes quote example credentials verbatim; documentation about findings, not live material"; approved_by="keyhog-maintainers" +path:docs/src/workflows/github-action.md; reason="the file documents exact Action version pins of this public repository; generic entropy reads a pinned release tag as a credential-shaped token"; approved_by="keyhog-maintainers" # --- Test, fixture, and demo trees ------------------------------------------- # These trees carry exact positive and negative credential examples by design. diff --git a/crates/cli/CHANGELOG.md b/crates/cli/CHANGELOG.md index 1968aab37..41567ef59 100644 --- a/crates/cli/CHANGELOG.md +++ b/crates/cli/CHANGELOG.md @@ -2,6 +2,18 @@ ## 0.5.81 - 2026-08-20 +- fix(cli): the guard watcher applies `[scan].exclude` changes when `.keyhog.toml` is modified. The reload branch that re-reads exclusions sat inside a filename check that only admitted `.keyhogignore` and `.gitignore`, so an exclusion edit left the old matcher active until the daemon restarted. +- fix(cli): guard watching and `keyhog watch` no longer drop events under `.cargo`. The directory skip policy listed it as a watch-only extra while the scanner's default excludes prune no `.cargo` component, so edits to a repository's `credentials.toml` or cargo config never reached the guard even though scans read them. +- test(cli): the Row 141 watcher suite pins scanner-truth parity for `.cargo/credentials.toml` and default media-extension filtering, the Row 146 feed test pins ignore-on-overflow for stale-policy roots, and the Row 144 missing-pack case asserts fail-closed exit 13 with the unscanned-batch report instead of the retired embedded-detector fallback. +- test(cli): Rows 144, 145, and 158 calibrate each cloned cache home through the bounded ci-lean autoroute fixture before scanning, because auto-routed scans and hooks fail closed without a persisted backend decision even when execution packs are installed. +- test(cli): restored regression suites assert the shipped report schema and coverage contract. The config-path baseline pins `evidence_score` (the `confidence` report field was renamed in the evidence-tier change), full-prune exclude globs expect fail-closed exit 13 with `[]` on stdout, the high severity floor test accounts for the non-blocking review-tier entropy fallback, and the staged-renames guard test registers with `--no-hook` because the installed hook requires `keyhog` on PATH. +- test(cli): the overnight reliability matrix pins the documented exit contract against the current evidence tiers. The exit-contract fixture uses a checksum-confirmed PAT instead of a contextless AWS key that now reports at review tier, the format-determinism comparison normalizes per-scan volatile fields (`scan_id`, wall-clock stamps, `duration_ms`), baseline creation uses the real `--create-baseline` flag, and Merkle index writes enable `--incremental`. +- test(cli): Row 130 install suites select the bounded autoroute fixture under `ci-lean` and pin the shipped uninstalled-scan fail-closed message (`autoroute calibration required` with the calibrated-install remedy), so the newly wired step neither measures the production ladder four times per run nor asserts retired error wording. +- fix(cli): `compile-execution-packs` completes on Windows. The staging directory fsync opened the directory in read mode, and `FlushFileBuffers` on a read-mode directory handle fails with "Access is denied" (os error 5). +- fix(cli): the Excluded coverage-gap reason and summary warning no longer claim `.keyhogignore` removals are never counted. A `--git-staged` scan records staged paths matched by the repository's `.keyhogignore` under that counter, so the text now says so instead of attributing them to the default exclusion policy. +- test(cli): concurrent daemon telemetry isolation runs at the daemon's guaranteed eight-connection admission floor, so low-core CI runners exercise isolation without turning the contract into an overload-refusal test. +- test(cli): Row 135 artifact invalidation tests use the authenticated bounded autoroute fixture under `ci-lean`; their staleness and regeneration assertions no longer spend minutes per case measuring the production calibration ladder. +- test(cli): the empty-directory e2e contract now asserts fail-closed exit 13 instead of the obsolete clean exit, and daemon lifecycle tests serialize real daemon starts with an explicit CPU diagnostic route instead of depending on missing autoroute calibration or competing detector compiles. - fix(daemon): the detector rules identity is derived from the rule set, not from the route that loaded it. `detector_rules_digest` is the single owner for the client, the daemon handshake, the staleness check and the scan request, so a warm daemon and the CLI that started it no longer disagree about their own rules and re-route to an in-process scan. - fix(cli): an Action receipt certifies a fail-closed scan. `validate_semantics` accepted only `partial` on exit 11 and 13, so a total source failure, which reports `failed`, was rejected as contradictory semantics and the receipt bailed on the scan it exists to record. - fix(cli): `--autoroute` calibration state that is missing, stale, invalid, incomplete, or quarantined exits 2 on a normal automatic scan, with `[]` on stdout and the state plus its repair on stderr. `docs/src/reference/exit-codes.md` documents that, and the scan never benchmarks at scan time or substitutes scalar execution. diff --git a/crates/cli/data/path_skip_dirs.toml b/crates/cli/data/path_skip_dirs.toml index c8eb102e8..070456d5f 100644 --- a/crates/cli/data/path_skip_dirs.toml +++ b/crates/cli/data/path_skip_dirs.toml @@ -9,7 +9,7 @@ # not skip `.git` because it is looking for repository roots. [skip_dirs] -base = [".cargo"] +base = [] watch_extra = [ ".svn", ".hg", ".turbo", diff --git a/crates/cli/src/daemon/guard_watcher.rs b/crates/cli/src/daemon/guard_watcher.rs index 6e770b1a8..b3c482642 100644 --- a/crates/cli/src/daemon/guard_watcher.rs +++ b/crates/cli/src/daemon/guard_watcher.rs @@ -247,14 +247,16 @@ impl WatchedRoot { false } - #[allow(dead_code)] fn maybe_reload_ignore_matcher(&self, root: &std::path::Path, path: &std::path::Path) { if let Some(file_name) = path.file_name().and_then(|n| n.to_str()) { - if file_name == ".keyhogignore" || file_name == ".gitignore" { - if file_name == ".keyhog.toml" { - let (new_ignore_paths, _) = resolve_root_exclusions(root); - *self.ignore_paths.write() = new_ignore_paths; - } + if file_name == ".keyhog.toml" { + let (new_ignore_paths, _) = resolve_root_exclusions(root); + *self.ignore_paths.write() = new_ignore_paths; + } + if file_name == ".keyhogignore" + || file_name == ".gitignore" + || file_name == ".keyhog.toml" + { let ignore_paths = self.ignore_paths.read(); *self.ignore_matcher.write() = build_root_ignore_matcher(root, &ignore_paths); } diff --git a/crates/cli/src/reporting.rs b/crates/cli/src/reporting.rs index c9c4e825a..a35f8ab20 100644 --- a/crates/cli/src/reporting.rs +++ b/crates/cli/src/reporting.rs @@ -999,7 +999,7 @@ impl CoverageGapKind { } Self::Binary => "binary (extension or content sniff)", Self::Excluded => { - "default exclusion policy (lock files, minified/bundled assets, vendored and build-output trees). User `.keyhogignore` / --exclude-paths removals are not counted here" + "exclusion policy (default excludes such as lock files, minified/bundled assets, vendored and build-output trees; --git-staged also counts repository `.keyhogignore` matches here)" } Self::NonBinaryUnreadable => "unreadable (permission denied or I/O error)", Self::GitObjectUnreadable => { @@ -1141,12 +1141,12 @@ impl CoverageGapKind { scanned as text." ), Self::Excluded => format!( - "{n} path(s) skipped by the DEFAULT exclusion policy (lock files, \ - minified/bundled assets, vendored and build-output trees). \ + "{n} path(s) skipped by the exclusion policy (default excludes such as lock \ + files, minified/bundled assets, vendored and build-output trees). \ Default-excluded directories are pruned during discovery and counted \ once each; nested files under them are not enumerated. Pass \ - `--no-default-excludes` to scan them. Files removed by your own \ - `.keyhogignore` or `--exclude-paths` are not counted in this number." + `--no-default-excludes` to scan them. A `--git-staged` scan also counts \ + staged paths removed by the repository's `.keyhogignore` here." ), Self::NonBinaryUnreadable => format!( "{n} file(s) NOT scanned: unreadable (permission denied or I/O error). These \ diff --git a/crates/cli/src/skip_dirs.rs b/crates/cli/src/skip_dirs.rs index 3a7c87adb..d201f94d0 100644 --- a/crates/cli/src/skip_dirs.rs +++ b/crates/cli/src/skip_dirs.rs @@ -110,7 +110,7 @@ impl SkipDirPolicy { } fn from_section(section: SkipDirSection) -> std::result::Result { - validate_list("base", §ion.base)?; + validate_optional_list("base", §ion.base)?; validate_list("watch_extra", §ion.watch_extra)?; validate_list("git_discovery_extra", §ion.git_discovery_extra)?; diff --git a/crates/cli/src/subcommands/compile_execution_packs.rs b/crates/cli/src/subcommands/compile_execution_packs.rs index d095d923f..66de42f7d 100644 --- a/crates/cli/src/subcommands/compile_execution_packs.rs +++ b/crates/cli/src/subcommands/compile_execution_packs.rs @@ -514,9 +514,23 @@ fn write_sync(path: &Path, bytes: &[u8]) -> Result<()> { } fn sync_directory(path: &Path) -> Result<()> { - File::open(path) - .with_context(|| format!("opening directory {} for sync", path.display()))? - .sync_all() + #[cfg(target_os = "windows")] + use std::os::windows::fs::OpenOptionsExt; + // FlushFileBuffers requires GENERIC_WRITE on the handle, and Windows only + // grants a directory handle that access when it is opened with + // FILE_FLAG_BACKUP_SEMANTICS. Read mode alone fails sync_all with + // "Access is denied" (os error 5). + #[cfg(target_os = "windows")] + let file = std::fs::OpenOptions::new() + .read(true) + .write(true) + .custom_flags(0x0200_0000) + .open(path) + .with_context(|| format!("opening directory {} for sync", path.display()))?; + #[cfg(not(target_os = "windows"))] + let file = File::open(path) + .with_context(|| format!("opening directory {} for sync", path.display()))?; + file.sync_all() .with_context(|| format!("syncing directory {}", path.display())) } diff --git a/crates/cli/tests/e2e/mod.rs b/crates/cli/tests/e2e/mod.rs index 15e64219d..e7c588848 100644 --- a/crates/cli/tests/e2e/mod.rs +++ b/crates/cli/tests/e2e/mod.rs @@ -47,6 +47,7 @@ pub mod scan_dogfood_dedupes_example_suppression_events; pub mod scan_dogfood_static_recovery_rejections; pub mod scan_dogfood_suppressed_example; pub mod scan_effective_config; +pub mod scan_empty_dir_fails_closed; pub mod scan_exclude_paths_with_git_staged; pub mod scan_fast_flag_git_staged_clean; pub mod scan_fast_mode_clean_exit_zero; diff --git a/crates/cli/tests/e2e/scan_empty_dir_exit_zero.rs b/crates/cli/tests/e2e/scan_empty_dir_exit_zero.rs deleted file mode 100644 index 1ade5a3d4..000000000 --- a/crates/cli/tests/e2e/scan_empty_dir_exit_zero.rs +++ /dev/null @@ -1,16 +0,0 @@ -//! E2E: scanning an empty directory exits 0. - -use crate::e2e::support::scan_path; -use tempfile::TempDir; - -#[test] -fn scan_empty_dir_exit_zero() { - let dir = TempDir::new().expect("tempdir"); - let output = scan_path(dir.path(), &[]); - assert_eq!( - output.status.code(), - Some(0), - "empty dir must exit 0; stderr={}", - String::from_utf8_lossy(&output.stderr) - ); -} diff --git a/crates/cli/tests/e2e/scan_empty_dir_fails_closed.rs b/crates/cli/tests/e2e/scan_empty_dir_fails_closed.rs new file mode 100644 index 000000000..f6bbf6564 --- /dev/null +++ b/crates/cli/tests/e2e/scan_empty_dir_fails_closed.rs @@ -0,0 +1,24 @@ +//! E2E: an empty directory provides no coverage and fails closed. + +use crate::e2e::support::scan_path; +use keyhog::exit_codes::EXIT_SOURCE_FAILED; +use tempfile::TempDir; + +/// WHY: zero-byte input cannot prove that a target is clean. Empty, +/// fully-excluded, and wrong-target scans must not report success. +#[test] +fn scan_empty_dir_fails_closed() { + let dir = TempDir::new().expect("tempdir"); + let output = scan_path(dir.path(), &[]); + assert_eq!( + output.status.code(), + Some(EXIT_SOURCE_FAILED as i32), + "empty dir must fail closed; stderr={}", + String::from_utf8_lossy(&output.stderr) + ); + let stderr = String::from_utf8_lossy(&output.stderr); + assert!( + stderr.contains("ZERO bytes") && stderr.contains("Nothing was examined"), + "empty-dir failure must state the missing coverage and corrective context: {stderr}" + ); +} diff --git a/crates/cli/tests/lane10_daemon_e2e_lifecycle.rs b/crates/cli/tests/lane10_daemon_e2e_lifecycle.rs index a8a8f0bbf..e258595bd 100644 --- a/crates/cli/tests/lane10_daemon_e2e_lifecycle.rs +++ b/crates/cli/tests/lane10_daemon_e2e_lifecycle.rs @@ -16,6 +16,7 @@ use std::path::{Path, PathBuf}; use std::process::{Child, Command}; use std::time::{Duration, Instant}; use tempfile::TempDir; +static DAEMON_TEST_LOCK: std::sync::Mutex<()> = std::sync::Mutex::new(()); fn binary() -> PathBuf { PathBuf::from(env!("CARGO_BIN_EXE_keyhog")) @@ -42,7 +43,7 @@ fn start_daemon(dir: &Path, extra_args: &[&str]) -> (Child, PathBuf) { .args(extra_args) .stdout(std::process::Stdio::null()) .stderr(std::process::Stdio::null()); - let child = cmd.spawn().expect("spawn daemon"); + let mut child = cmd.spawn().expect("spawn daemon"); // Wait for a real listener (connect succeeds), not just file existence. let deadline = Instant::now() + Duration::from_secs(30); while Instant::now() < deadline { @@ -51,6 +52,8 @@ fn start_daemon(dir: &Path, extra_args: &[&str]) -> (Child, PathBuf) { } std::thread::sleep(Duration::from_millis(100)); } + let _ = child.kill(); + let _ = child.wait(); panic!("daemon did not become ready within 30s"); } @@ -66,8 +69,9 @@ fn stop_daemon(socket: &Path) -> Option { #[test] fn daemon_start_status_stop_lifecycle_and_socket_hygiene() { + let _daemon_guard = DAEMON_TEST_LOCK.lock().unwrap_or_else(|e| e.into_inner()); let dir = TempDir::new().unwrap(); - let (mut child, socket) = start_daemon(dir.path(), &[]); + let (mut child, socket) = start_daemon(dir.path(), &["--backend", "cpu"]); // Socket mode is one defense-in-depth access control. The production // status and scan clients exercised below additionally authenticate the @@ -134,6 +138,7 @@ fn daemon_start_status_stop_lifecycle_and_socket_hygiene() { #[cfg(target_os = "linux")] #[test] fn explicit_host_daemons_do_not_load_gpu_runtime_libraries() { + let _daemon_guard = DAEMON_TEST_LOCK.lock().unwrap_or_else(|e| e.into_inner()); let host_backends: Vec<_> = BACKEND_OVERRIDE_VALUES .iter() .copied() @@ -184,10 +189,14 @@ fn host_only_identities_do_not_load_gpu_runtime_libraries() { #[test] fn daemon_reclaims_stuck_half_frame_connection() { + let _daemon_guard = DAEMON_TEST_LOCK.lock().unwrap_or_else(|e| e.into_inner()); let dir = TempDir::new().unwrap(); // 1-second request timeout so the test is fast. A real client does one // round-trip; a connection idle past this is stuck and must be reclaimed. - let (mut child, socket) = start_daemon(dir.path(), &["--request-timeout-secs", "1"]); + let (mut child, socket) = start_daemon( + dir.path(), + &["--backend", "cpu", "--request-timeout-secs", "1"], + ); // Open a connection, announce a frame length, then send NOTHING, the // classic half-frame / slowloris stall that would otherwise hold a @@ -246,8 +255,9 @@ fn daemon_reclaims_stuck_half_frame_connection() { #[test] fn daemon_rejects_oversized_frame_length_prefix() { + let _daemon_guard = DAEMON_TEST_LOCK.lock().unwrap_or_else(|e| e.into_inner()); let dir = TempDir::new().unwrap(); - let (mut child, socket) = start_daemon(dir.path(), &[]); + let (mut child, socket) = start_daemon(dir.path(), &["--backend", "cpu"]); // MAX_FRAME_BYTES is 64 MiB; announce one byte more. The server must refuse // to allocate the recv buffer (it drops the connection) rather than OOM. @@ -308,6 +318,7 @@ fn daemon_rejects_oversized_frame_length_prefix() { #[cfg(unix)] #[test] fn daemon_socket_flag_wires_scan_to_a_fixed_location_daemon() { + let _daemon_guard = DAEMON_TEST_LOCK.lock().unwrap_or_else(|e| e.into_inner()); use std::process::Stdio; let dir = TempDir::new().unwrap(); diff --git a/crates/cli/tests/regression_cli_config_path.rs b/crates/cli/tests/regression_cli_config_path.rs index 253b718ed..f2cf644e9 100644 --- a/crates/cli/tests/regression_cli_config_path.rs +++ b/crates/cli/tests/regression_cli_config_path.rs @@ -117,8 +117,8 @@ fn explicit_config_min_confidence_suppresses_below_threshold() { "baseline stdout must carry the github-classic-pat finding.\n--- stdout ---\n{stdout}" ); assert!( - stdout.contains("\"confidence\":0.9"), - "baseline finding confidence must be exactly 0.9.\n--- stdout ---\n{stdout}" + stdout.contains("\"evidence_score\":0.9"), + "baseline finding evidence score must be exactly 0.9.\n--- stdout ---\n{stdout}" ); // A `tool.toml` at min_confidence = 0.95 sits ABOVE the 0.9 finding, so the diff --git a/crates/cli/tests/regression_cli_exclude_glob.rs b/crates/cli/tests/regression_cli_exclude_glob.rs index bb8aa6708..7024af821 100644 --- a/crates/cli/tests/regression_cli_exclude_glob.rs +++ b/crates/cli/tests/regression_cli_exclude_glob.rs @@ -62,6 +62,7 @@ const SEVERITY: &str = "critical"; const EXIT_SUCCESS: i32 = 0; const EXIT_FINDINGS: i32 = 1; const EXIT_USER_ERROR: i32 = 2; +const EXIT_SOURCE_FAILED: i32 = 13; fn binary() -> PathBuf { PathBuf::from(env!("CARGO_BIN_EXE_keyhog")) @@ -292,15 +293,17 @@ fn exclude_nonmatching_glob_keeps_both() { } /// A slash-less basename glob (`*.env`) matches at ANY depth (gitignore -/// semantics), so BOTH the root `a.env` and the nested `skip/b.env` are pruned: -/// an HONEST empty result (exit 0 and the literal `[]` bytes). +/// semantics), so BOTH the root `a.env` and the nested `skip/b.env` are pruned. +/// Pruning every candidate leaves ZERO bytes to scan, and the fail-closed +/// coverage contract refuses an empty report presented as clean: exit +/// EXIT_SOURCE_FAILED while stdout stays the literal `[]` bytes. #[test] fn exclude_star_env_matches_any_depth_empties_result() { let dir = plant_skip_tree(); let (code, findings, stdout, stderr) = run_json(dir.path(), &["--exclude-paths", "*.env"]); assert_eq!( - code, EXIT_SUCCESS, - "every .env excluded -> exit 0; stderr:\n{stderr}" + code, EXIT_SOURCE_FAILED, + "every .env excluded -> zero-byte coverage -> exit 13; stderr:\n{stderr}" ); assert_eq!(findings.len(), 0, "*.env prunes a.env AND skip/b.env"); assert_eq!( @@ -309,13 +312,13 @@ fn exclude_star_env_matches_any_depth_empties_result() { ); } -/// The explicit recursive glob `**/*.env` also strips every `.env` at any depth -/// same empty, exit-0 result as the bare `*.env` form. +/// The explicit recursive glob `**/*.env` also strips every `.env` at any depth; +/// like every full-prune it ends at zero scanned bytes -> EXIT_SOURCE_FAILED. #[test] fn exclude_recursive_env_glob_empties_result() { let dir = plant_skip_tree(); let (code, findings, stdout, _stderr) = run_json(dir.path(), &["--exclude-paths", "**/*.env"]); - assert_eq!(code, EXIT_SUCCESS, "no .env survives -> exit 0"); + assert_eq!(code, EXIT_SOURCE_FAILED, "no .env survives -> exit 13"); assert_eq!(findings.len(), 0); assert_eq!(stdout, b"[]"); } @@ -334,15 +337,16 @@ fn exclude_anchored_relative_path_drops_one() { } /// Multiple `--exclude-paths` operands COMPOSE: excluding both the root file and -/// the whole skip subtree prunes everything -> exit 0 and `[]`. +/// the whole skip subtree prunes everything -> zero scanned bytes -> +/// EXIT_SOURCE_FAILED with `[]` on stdout. #[test] fn multiple_excludes_compose_to_empty() { let dir = plant_skip_tree(); let (code, findings, stdout, stderr) = run_json(dir.path(), &["--exclude-paths", "a.env", "**/skip/**"]); assert_eq!( - code, EXIT_SUCCESS, - "both operands prune everything -> exit 0; stderr:\n{stderr}" + code, EXIT_SOURCE_FAILED, + "both operands prune everything -> zero-byte coverage -> exit 13; stderr:\n{stderr}" ); assert_eq!(findings.len(), 0, "a.env and skip/** both excluded"); assert_eq!(stdout, b"[]"); diff --git a/crates/cli/tests/regression_cli_glob_filters.rs b/crates/cli/tests/regression_cli_glob_filters.rs index 36de6dc8f..bdc9a28fb 100644 --- a/crates/cli/tests/regression_cli_glob_filters.rs +++ b/crates/cli/tests/regression_cli_glob_filters.rs @@ -60,6 +60,7 @@ const SEVERITY: &str = "critical"; const EXIT_SUCCESS: i32 = 0; const EXIT_FINDINGS: i32 = 1; const EXIT_USER_ERROR: i32 = 2; +const EXIT_SOURCE_FAILED: i32 = 13; fn binary() -> PathBuf { PathBuf::from(env!("CARGO_BIN_EXE_keyhog")) @@ -252,16 +253,17 @@ fn exclude_rs_glob_keeps_only_txt() { ); } -/// Both globs together prune every file: an HONEST empty result, exit 0 and a -/// literal `[]` JSON array (not a crash, not a usage error). +/// Both globs together prune every file: zero scanned bytes, so the fail-closed +/// coverage contract exits EXIT_SOURCE_FAILED while stdout stays a literal `[]` +/// JSON array (not a crash, not a usage error). #[test] -fn exclude_both_globs_finds_nothing_exit_zero() { +fn exclude_both_globs_prune_everything_fail_closed() { let dir = plant_flat(); let (code, findings, stdout, stderr) = run_json(dir.path(), &["--exclude-paths", "*.rs", "*.txt"]); assert_eq!( - code, EXIT_SUCCESS, - "no findings -> exit 0; stderr:\n{stderr}" + code, EXIT_SOURCE_FAILED, + "everything excluded -> zero-byte coverage -> exit 13; stderr:\n{stderr}" ); assert_eq!(findings.len(), 0, "everything excluded"); assert_eq!( diff --git a/crates/cli/tests/regression_cli_guard_massive_diff_simulation.rs b/crates/cli/tests/regression_cli_guard_massive_diff_simulation.rs index 3ef6cd0d7..db80dbe99 100644 --- a/crates/cli/tests/regression_cli_guard_massive_diff_simulation.rs +++ b/crates/cli/tests/regression_cli_guard_massive_diff_simulation.rs @@ -558,7 +558,11 @@ fn guard_staged_diff_edge_cases_staged_renames() { let add_out = Command::new(keyhog()) .current_dir(repo) .env("NO_COLOR", "1") - .args(["guard", "add", ".", "--socket"]) + // --no-hook: this test exercises daemon rename handling, not the + // pre-commit hook. The installed hook shells out to `keyhog` from + // PATH, which does not exist on CI runners, so an unqualified commit + // after `guard add` fails with "git commit must succeed". + .args(["guard", "add", ".", "--no-hook", "--socket"]) .arg(&socket) .output() .expect("guard add"); diff --git a/crates/cli/tests/regression_cli_min_severity.rs b/crates/cli/tests/regression_cli_min_severity.rs index da1f85c3b..4d2445883 100644 --- a/crates/cli/tests/regression_cli_min_severity.rs +++ b/crates/cli/tests/regression_cli_min_severity.rs @@ -353,21 +353,30 @@ fn low_finding_filtered_by_critical_floor_exits_zero_empty() { assert_eq!(findings(&out).len(), 0, "zero findings survive the floor"); } -/// Same fixture under `--severity high` also empties (`low < high`) => exit 0. -/// Confirms the empty-exit-0 behavior is not special to the top tier. +/// Same fixture under `--severity high`: the floor drops the low NAMED finding +/// (`low < high`), but the generic entropy fallback then surfaces +/// (`entropy-token`, severity `high`, evidence tier `review`) — without the +/// named match it was deduplicated against there is nothing left to absorb it. +/// Review-tier findings are non-blocking, so the scan still exits 0; the +/// floor's contract here is that no named low-severity finding survives. #[test] -fn low_finding_filtered_by_high_floor_exits_zero_empty() { +fn low_finding_filtered_by_high_floor_drops_named_keeps_entropy_review() { let (_d, path) = low_only_fixture(); let (code, out, err) = scan(&path, &["--format", "json", "--severity", "high"]); assert_eq!( code, Some(0), - "empty post-filter report exits 0; stderr={err}" + "a surviving review-tier finding does not block; stderr={err}" ); assert_eq!( - findings(&out).len(), + count_detector(&out, ROME_ID), 0, - "zero findings survive the high floor; got {out}" + "the high floor must drop the low rome2rio finding; got {out}" + ); + assert_eq!( + severities(&out), + vec!["high".to_owned()], + "only the high-severity entropy fallback survives the floor; got {out}" ); } diff --git a/crates/cli/tests/regression_daemon_concurrent_scan_isolation.rs b/crates/cli/tests/regression_daemon_concurrent_scan_isolation.rs index 665747df1..6002c68f4 100644 --- a/crates/cli/tests/regression_daemon_concurrent_scan_isolation.rs +++ b/crates/cli/tests/regression_daemon_concurrent_scan_isolation.rs @@ -87,8 +87,11 @@ async fn concurrent_daemon_scans_maintain_isolated_skip_and_telemetry_counts() { let file_c = fixture_root.path().join("file_c.txt"); std::fs::write(&file_c, "const AWS_KEY = \"AKIAIOSFODNN7EXAMPLE\";\n").expect("write file_c"); - // Launch concurrent client scan tasks against the running daemon - let concurrency = 24; + // Server admission is `(logical_cpu_count * 4).clamp(8, 256)`. Use the + // guaranteed floor so this telemetry-isolation contract remains valid on + // two- and four-core CI runners; overload refusal is a separate daemon + // contract, not cross-request telemetry contamination. + let concurrency = 8; let mut handles = Vec::new(); for client_id in 0..concurrency { diff --git a/crates/cli/tests/regression_row_130_clean_install_zero_compiles.rs b/crates/cli/tests/regression_row_130_clean_install_zero_compiles.rs index 4e5c6630b..2348fa77f 100755 --- a/crates/cli/tests/regression_row_130_clean_install_zero_compiles.rs +++ b/crates/cli/tests/regression_row_130_clean_install_zero_compiles.rs @@ -22,7 +22,7 @@ use keyhog::exit_codes::{EXIT_SUCCESS, EXIT_USER_ERROR}; use keyhog_profile::CompileSurfaceId; use std::collections::BTreeSet; use std::fs; -use std::path::PathBuf; +use std::path::{Path, PathBuf}; use std::process::Command; fn create_clean_environment(prefix: &str) -> (tempfile::TempDir, PathBuf, PathBuf, PathBuf) { @@ -50,6 +50,33 @@ fn create_clean_environment(prefix: &str) -> (tempfile::TempDir, PathBuf, PathBu (temp_dir, cache_home, home_dir, isolated_exe) } +/// Build an isolated `keyhog install` command. Under `ci-lean` the installer's +/// internal `calibrate-autoroute` measures the FULL production ladder unless +/// the fixture sentinels select the bounded one; without them every test here +/// spends minutes of CPU recalibrating, which blew the CI lane's time budget. +/// Mirrors the row-135 installation fixture contract. +fn install_command(exe: &Path) -> Command { + let mut cmd = Command::new(exe); + cmd.arg("install"); + #[cfg(feature = "ci-lean")] + { + cmd.env( + "KEYHOG_CI_AUTOROUTE_TIMING_FIXTURE", + "confidence-separated-v1", + ) + .env( + "KEYHOG_CI_AUTOROUTE_FIXTURE_AUTH", + "bench-backend-parity-v1", + ) + .env("KEYHOG_CI_AUTOROUTE_WORKLOAD_FIXTURE", "bounded-e2e-v1") + .env( + "KEYHOG_CI_AUTOROUTE_WORKLOAD_FIXTURE_AUTH", + "core-workload-plan-v1", + ); + } + cmd +} + #[test] fn clean_cache_scan_fails_closed_without_installation() { let (_temp, cache_home, home_dir, exe) = create_clean_environment("keyhog-row130-uninstalled-"); @@ -75,14 +102,21 @@ fn clean_cache_scan_fails_closed_without_installation() { ); let stderr = String::from_utf8_lossy(&scan_output.stderr); + // The uninstalled-cache failure is the autoroute fail-closed contract: no + // persisted fastest-correct decision exists, the scan names the missing + // state, and the remedy names calibration (installers get + // `install.sh --calibrate` / `install.ps1 -Calibrate`). + assert!( + stderr.contains("autoroute calibration required"), + "stderr must name the missing autoroute calibration; got:\n{stderr}" + ); assert!( - stderr.contains("keyhog install"), - "stderr must guide user to run `keyhog install`; got:\n{stderr}" + stderr.contains("install.sh --calibrate") && stderr.contains("install.ps1 -Calibrate"), + "stderr must guide installers to the calibrated install path; got:\n{stderr}" ); assert!( - stderr.contains("in-process compilation is forbidden") - || stderr.contains("no usable detector execution pack available"), - "stderr must explicitly forbid runtime compilation fallback; got:\n{stderr}" + stderr.contains("No backend was selected and this batch was not scanned"), + "stderr must state that nothing was scanned rather than substituting a backend; got:\n{stderr}" ); } @@ -92,8 +126,7 @@ fn clean_install_generates_all_artifact_classes_and_enables_zero_compile_scans() create_clean_environment("keyhog-row130-clean-install-"); // 1. Run `keyhog install` - let install_output = Command::new(&exe) - .arg("install") + let install_output = install_command(&exe) .env("XDG_CACHE_HOME", &cache_home) .env("HOME", &home_dir) .output() @@ -207,8 +240,7 @@ fn clean_install_detects_real_credentials_with_zero_runtime_compiles() { let (_temp, cache_home, home_dir, exe) = create_clean_environment("keyhog-row130-cred-scan-"); // 1. Run `keyhog install` - let install_output = Command::new(&exe) - .arg("install") + let install_output = install_command(&exe) .env("XDG_CACHE_HOME", &cache_home) .env("HOME", &home_dir) .output() @@ -282,8 +314,7 @@ fn artifact_mutation_fails_closed_without_compilation_fallback() { let (_temp, cache_home, home_dir, exe) = create_clean_environment("keyhog-row130-mutation-"); // Install first - let install_output = Command::new(&exe) - .arg("install") + let install_output = install_command(&exe) .env("XDG_CACHE_HOME", &cache_home) .env("HOME", &home_dir) .output() @@ -348,8 +379,7 @@ fn runtime_derived_compile_surface_exhaustiveness() { let (_temp, cache_home, home_dir, exe) = create_clean_environment("keyhog-row130-exhaustiveness-"); - let install_output = Command::new(&exe) - .arg("install") + let install_output = install_command(&exe) .env("XDG_CACHE_HOME", &cache_home) .env("HOME", &home_dir) .output() diff --git a/crates/cli/tests/regression_row_135_multi_dimensional_invalidation.rs b/crates/cli/tests/regression_row_135_multi_dimensional_invalidation.rs index 6ac7a8a91..9128d4daa 100644 --- a/crates/cli/tests/regression_row_135_multi_dimensional_invalidation.rs +++ b/crates/cli/tests/regression_row_135_multi_dimensional_invalidation.rs @@ -69,14 +69,35 @@ fn prepare_fresh_installation(test_exe: &Path, cache_home: &Path) -> (PathBuf, P ); let autoroute_cache = cache_home.join("keyhog/autoroute.json"); - let cal_result = Command::new(test_exe) + let mut calibrate = Command::new(test_exe); + calibrate .arg("calibrate-autoroute") .arg("--quiet") .arg("--autoroute-cache") .arg(&autoroute_cache) .env("XDG_CACHE_HOME", cache_home) - .output() - .expect("run calibrate autoroute"); + .env("NO_COLOR", "1"); + // A ci-lean binary measures the full production ladder unless the fixture + // sentinels select the bounded one; without them every test here spends + // minutes of CPU recalibrating, which blew the CI lane's time budget. + #[cfg(feature = "ci-lean")] + { + calibrate + .env( + "KEYHOG_CI_AUTOROUTE_TIMING_FIXTURE", + "confidence-separated-v1", + ) + .env( + "KEYHOG_CI_AUTOROUTE_FIXTURE_AUTH", + "bench-backend-parity-v1", + ) + .env("KEYHOG_CI_AUTOROUTE_WORKLOAD_FIXTURE", "bounded-e2e-v1") + .env( + "KEYHOG_CI_AUTOROUTE_WORKLOAD_FIXTURE_AUTH", + "core-workload-plan-v1", + ); + } + let cal_result = calibrate.output().expect("run calibrate autoroute"); assert!( cal_result.status.success(), "calibrate autoroute failed: {}", diff --git a/crates/cli/tests/regression_row_140_guard_offline_status.rs b/crates/cli/tests/regression_row_140_guard_offline_status.rs index 9dfe948e5..c2d703318 100644 --- a/crates/cli/tests/regression_row_140_guard_offline_status.rs +++ b/crates/cli/tests/regression_row_140_guard_offline_status.rs @@ -377,7 +377,7 @@ fn row_140_guard_status_offline_unregistered_root_fails_gracefully() { assert_ne!(code, 0, "unregistered root must fail"); assert!( - stderr.contains("not registered in durable store") && stderr.contains("(no daemon active)"), + stderr.contains("not registered in durable store") && stderr.contains("no daemon active"), "must report not registered in durable store (no daemon active): {stderr}" ); assert!( diff --git a/crates/cli/tests/regression_row_141_guard_watcher_exclusions.rs b/crates/cli/tests/regression_row_141_guard_watcher_exclusions.rs index 0236dd6b0..ccb3e3fbb 100644 --- a/crates/cli/tests/regression_row_141_guard_watcher_exclusions.rs +++ b/crates/cli/tests/regression_row_141_guard_watcher_exclusions.rs @@ -232,27 +232,35 @@ fn default_excluded_files_and_suffixes_filtered() { } #[test] -fn image_and_cargo_credentials_files_are_not_excluded() { +fn cargo_credentials_not_excluded_by_watcher() { + // WHY: the watcher must mirror scanner traversal truth. The scanner's + // default excludes prune no `.cargo` component and skip no `.toml` + // filename, so `.cargo/credentials.toml` is a scanned leak vector and + // its events must reach the guard. + // + // Extension-based filtering (png, jpg, etc.) happens at the reader pool, + // not the watcher path classifier, so media files ARE delivered as events + // even though scans ultimately skip them by extension. let config = GuardReconciliationConfig::default(); let (mut watcher, tx) = GuardWatcher::new_with_channel(config); let root = PathBuf::from("/srv/repo"); watcher.add_root(root.clone()).expect("add root"); - let scannable_paths = vec![ - root.join(".cargo/credentials.toml"), - root.join("assets/logo.png"), - root.join("assets/banner.jpg"), - ]; + assert!( + !watcher.is_path_excluded(&root, &root.join(".cargo/credentials.toml")), + "scanned path .cargo/credentials.toml must NOT be excluded by watcher" + ); + // Images are NOT excluded at the watcher level (extension denylist feeds + // the reader, not the path classifier) so they produce events. + assert!( + !watcher.is_path_excluded(&root, &root.join("assets/logo.png")), + "image extension png is not a watcher-level exclusion" + ); - for path in &scannable_paths { - assert!( - !watcher.is_path_excluded(&root, path), - "scannable path {} must NOT be excluded by watcher", - path.display() - ); + for path in [".cargo/credentials.toml", "src/main.rs", "assets/logo.png"] { let mut event = notify::Event::new(EventKind::Modify(ModifyKind::Any)); - event.paths.push(path.clone()); + event.paths.push(root.join(path)); tx.send(Ok(event)).expect("send event"); } @@ -260,7 +268,7 @@ fn image_and_cargo_credentials_files_are_not_excluded() { assert_eq!( polled.len(), 1, - "watcher must emit events for scannable cargo and image files" + "watcher must emit all non-excluded file events to the root" ); } diff --git a/crates/cli/tests/regression_row_144_clean_pass_no_internal_warnings.rs b/crates/cli/tests/regression_row_144_clean_pass_no_internal_warnings.rs index c19b6d60a..2a8191380 100755 --- a/crates/cli/tests/regression_row_144_clean_pass_no_internal_warnings.rs +++ b/crates/cli/tests/regression_row_144_clean_pass_no_internal_warnings.rs @@ -90,9 +90,48 @@ fn clone_prepared_installation(cache_home: &Path) -> (PathBuf, PathBuf, PathBuf) let pack_root = cache_home.join("keyhog/execution-packs"); copy_dir_all(source_pack_root, &pack_root); let output = pack_root.join("current"); + calibrate_autoroute(&binary, cache_home); (binary.clone(), pack_root, output) } +// Auto-routed scans fail closed without a persisted fastest-correct backend +// decision; each cloned cache home needs its own calibrated autoroute.json. +// The ci-lean fixture sentinels select the bounded calibration ladder instead +// of the full production measurement. +fn calibrate_autoroute(binary: &Path, cache_home: &Path) { + let mut calibrate = Command::new(binary); + calibrate + .arg("calibrate-autoroute") + .arg("--quiet") + .arg("--autoroute-cache") + .arg(cache_home.join("keyhog/autoroute.json")) + .env("XDG_CACHE_HOME", cache_home) + .env("NO_COLOR", "1"); + #[cfg(feature = "ci-lean")] + { + calibrate + .env( + "KEYHOG_CI_AUTOROUTE_TIMING_FIXTURE", + "confidence-separated-v1", + ) + .env( + "KEYHOG_CI_AUTOROUTE_FIXTURE_AUTH", + "bench-backend-parity-v1", + ) + .env("KEYHOG_CI_AUTOROUTE_WORKLOAD_FIXTURE", "bounded-e2e-v1") + .env( + "KEYHOG_CI_AUTOROUTE_WORKLOAD_FIXTURE_AUTH", + "core-workload-plan-v1", + ); + } + let result = calibrate.output().expect("run calibrate autoroute"); + assert!( + result.status.success(), + "calibrate autoroute failed: {}", + String::from_utf8_lossy(&result.stderr) + ); +} + fn assert_no_internal_execution_pack_warnings(stderr: &str) { assert!( !stderr.contains("WARN no installed execution-pack generation; parsing embedded detectors"), @@ -248,9 +287,11 @@ fn mutation_gate_catches_synthetic_warning_pollution() { } #[test] -fn missing_execution_pack_runs_cleanly_without_noisy_fallback_warnings() { - // Contract: When execution packs are not installed, scan falls back to embedded - // detectors cleanly with EXIT_SUCCESS (0) and without emitting any noisy WARN logs on stderr. +fn missing_execution_pack_fails_closed_without_noisy_fallback_warnings() { + // Contract: with no installed execution packs and no calibrated autoroute + // decision, an auto-routed scan fails closed with EXIT_USER_ERROR (2), + // names the calibration remedy, and never emits the retired + // embedded-detector fallback warning or any raw tracing lines. let temp_dir = safe_tempdir("keyhog-row144-missing-"); let cache_home = temp_dir.path().join("empty_cache"); fs::create_dir_all(&cache_home).expect("create empty cache"); @@ -271,13 +312,16 @@ fn missing_execution_pack_runs_cleanly_without_noisy_fallback_warnings() { let stderr = String::from_utf8_lossy(&scan_output.stderr); assert_eq!( scan_output.status.code(), - Some(i32::from(EXIT_SUCCESS)), - "clean scan without installed packs must succeed with EXIT_SUCCESS (0), stderr: {stderr}" + Some(i32::from(EXIT_USER_ERROR)), + "scan without installed packs must fail closed with EXIT_USER_ERROR (2), stderr: {stderr}" ); - assert!( - stdout.contains("No secrets detected") || stdout.contains("PASS"), - "clean scan stdout must report clean status: {stdout}" + stderr.contains("autoroute calibration required"), + "fail-closed stderr must name the autoroute calibration remedy: {stderr}" + ); + assert!( + !stdout.contains("No secrets detected") && !stdout.contains("PASS"), + "a batch that was not scanned must not report a clean pass: {stdout}" ); assert_no_internal_execution_pack_warnings(&stderr); @@ -329,6 +373,8 @@ fn developer_escape_hatch_self_identifies_cleanly() { let temp_dir = safe_tempdir("keyhog-row144-dev-"); let cache_home = temp_dir.path().join("empty_cache"); fs::create_dir_all(&cache_home).expect("create empty cache"); + // The escape hatch compiles detectors in-process but does not select a + // routing backend, so use an explicit --backend cpu to bypass autoroute. let clean_file = temp_dir.path().join("clean.txt"); fs::write(&clean_file, "plain clean text\n").expect("write clean file"); @@ -336,6 +382,8 @@ fn developer_escape_hatch_self_identifies_cleanly() { let scan_output = Command::new(env!("CARGO_BIN_EXE_keyhog")) .arg("scan") .arg("--daemon=off") + .arg("--backend") + .arg("cpu") .arg("--developer-compile-embedded-detectors") .arg(&clean_file) .env("XDG_CACHE_HOME", &cache_home) @@ -351,8 +399,8 @@ fn developer_escape_hatch_self_identifies_cleanly() { let stderr = String::from_utf8_lossy(&scan_output.stderr); assert!( - stderr.contains("developer mode active: in-process detector compilation"), - "stderr must state developer mode is active: {stderr}" + stderr.contains("developer escape hatch active"), + "stderr must state developer escape hatch is active: {stderr}" ); assert!( !stderr.contains("WARN no installed execution-pack generation; parsing embedded detectors"), diff --git a/crates/cli/tests/regression_row_145_hook_zero_compilation.rs b/crates/cli/tests/regression_row_145_hook_zero_compilation.rs index 36437b0eb..17fdc2c37 100755 --- a/crates/cli/tests/regression_row_145_hook_zero_compilation.rs +++ b/crates/cli/tests/regression_row_145_hook_zero_compilation.rs @@ -14,7 +14,9 @@ //! Does not catch hardware GPU adapter faults during kernel execution or hardware memory bit flips. //! Does not catch OS kernel-level process SIGKILL termination. -use keyhog::exit_codes::{EXIT_CREDENTIALS_FOUND, EXIT_SUCCESS, EXIT_USER_ERROR}; +use keyhog::exit_codes::{ + EXIT_CREDENTIALS_FOUND, EXIT_SOURCE_FAILED, EXIT_SUCCESS, EXIT_USER_ERROR, +}; use keyhog::testing::execution_pack_install::{InstalledArtifactClass, InstalledArtifactRegistry}; use std::collections::BTreeSet; use std::fs; @@ -218,6 +220,8 @@ fn hook_run_utilizes_execution_pack_zero_runtime_compilations_subsecond() { .current_dir(&repo_dir) .arg("hook") .arg("run") + .arg("--backend") + .arg("cpu") .arg("--profile-out") .arg(&profile_output_path) .env("PATH", keyhog_path_env()) @@ -302,6 +306,8 @@ fn hook_run_detects_staged_secrets_with_zero_runtime_compilations() { .current_dir(&repo_dir) .arg("hook") .arg("run") + .arg("--backend") + .arg("cpu") .arg("--profile-out") .arg(&profile_output_path) .env("PATH", keyhog_path_env()) @@ -358,17 +364,20 @@ fn hook_run_fails_closed_when_execution_pack_missing() { .output() .expect("run hook run"); + // With no packs and no calibrated autoroute decision, the hook's auto + // scan fails closed at routing: the batch is reported as NEVER SCANNED + // and the run exits EXIT_SOURCE_FAILED (13), never a clean pass. assert_eq!( hook_out.status.code(), - Some(EXIT_USER_ERROR as i32), - "hook run must fail closed with exit code 2 when execution pack is missing; stderr:\n{}", + Some(i32::from(EXIT_SOURCE_FAILED)), + "hook run must fail closed (exit 13) when no pack and no autoroute decision exists; stderr:\n{}", String::from_utf8_lossy(&hook_out.stderr) ); let stderr = String::from_utf8_lossy(&hook_out.stderr); assert!( - stderr.contains("keyhog install") || stderr.contains("execution pack"), - "stderr must instruct user to run keyhog install; stderr:\n{stderr}" + stderr.contains("autoroute calibration required") && stderr.contains("NEVER SCANNED"), + "stderr must name the calibration remedy and the unscanned batch; stderr:\n{stderr}" ); } diff --git a/crates/cli/tests/regression_row_146_guard_transition_feed.rs b/crates/cli/tests/regression_row_146_guard_transition_feed.rs index d196a837b..565bd8021 100755 --- a/crates/cli/tests/regression_row_146_guard_transition_feed.rs +++ b/crates/cli/tests/regression_row_146_guard_transition_feed.rs @@ -589,11 +589,10 @@ fn guard_event_action_with_policy_prioritizes_overflow_and_prevents_duplicate_tr guard_event_action_with_policy(Some(GuardRootState::StalePolicy), false, true); assert_eq!(action_stale, GuardEventAction::Ignore); - // Overflow on StalePolicy yields CoverageLost -> Degraded + // Overflow on StalePolicy yields Ignore: a stale-policy root is already in + // the repair state reconciliation clears, so the overflow adds no + // transition. Degraded roots stay the repeat-coverage-loss path. let action_stale_overflow = guard_event_action_with_policy(Some(GuardRootState::StalePolicy), true, false); - assert_eq!( - action_stale_overflow, - GuardEventAction::Transition(GuardTransition::CoverageLost) - ); + assert_eq!(action_stale_overflow, GuardEventAction::Ignore); } diff --git a/crates/cli/tests/regression_row_158_startup_footprint_pack_install.rs b/crates/cli/tests/regression_row_158_startup_footprint_pack_install.rs index f23c519ca..3c4e85415 100755 --- a/crates/cli/tests/regression_row_158_startup_footprint_pack_install.rs +++ b/crates/cli/tests/regression_row_158_startup_footprint_pack_install.rs @@ -100,10 +100,57 @@ fn setup_test_pack_environment(temp_dir: &Path) -> (PathBuf, PathBuf) { (cache_home, output_dir) } +// The ci-lean fixture sentinels select the bounded calibration ladder instead +// of the full production measurement. +fn calibrate_autoroute(cache_home: &Path) { + let mut calibrate = Command::new(test_bin()); + calibrate + .arg("calibrate-autoroute") + .arg("--quiet") + .arg("--autoroute-cache") + .arg(cache_home.join("keyhog/autoroute.json")) + .env("XDG_CACHE_HOME", cache_home) + .env( + "HOME", + cache_home + .parent() + .map(Path::to_path_buf) + .unwrap_or_default(), + ) + .env("NO_COLOR", "1"); + #[cfg(feature = "ci-lean")] + { + calibrate + .env( + "KEYHOG_CI_AUTOROUTE_TIMING_FIXTURE", + "confidence-separated-v1", + ) + .env( + "KEYHOG_CI_AUTOROUTE_FIXTURE_AUTH", + "bench-backend-parity-v1", + ) + .env("KEYHOG_CI_AUTOROUTE_WORKLOAD_FIXTURE", "bounded-e2e-v1") + .env( + "KEYHOG_CI_AUTOROUTE_WORKLOAD_FIXTURE_AUTH", + "core-workload-plan-v1", + ); + } + let result = calibrate.output().expect("run calibrate autoroute"); + assert!( + result.status.success(), + "calibrate autoroute failed: {}", + String::from_utf8_lossy(&result.stderr) + ); +} + #[test] fn fresh_state_scan_uses_precompiled_packs_with_zero_runtime_compiles() { let temp_dir = create_temp_dir("keyhog-row158-fresh-"); let (cache_home, _output_dir) = setup_test_pack_environment(temp_dir.path()); + // Auto routing fails closed without a persisted fastest-correct backend + // decision, even with packs installed; calibrate the fresh cache first so + // the scan below exercises the precompiled-pack hydration path it names. + calibrate_autoroute(&cache_home); let scan_file = temp_dir.path().join("test_secret.txt"); fs::write( diff --git a/crates/cli/tests/reliability/determinism.rs b/crates/cli/tests/reliability/determinism.rs index 1bfacf1de..2438a5305 100644 --- a/crates/cli/tests/reliability/determinism.rs +++ b/crates/cli/tests/reliability/determinism.rs @@ -66,7 +66,57 @@ fn planted_corpus() -> tempfile::TempDir { /// on the same input corpus, preventing unordered map iteration or non-deterministic serialization /// from breaking golden testing, CI diffing, and caching. /// +/// The deliberate exceptions are per-scan metadata fields that are volatile BY +/// DESIGN — the scan id, wall-clock stamps, and measured duration. They are +/// normalized out before comparison; every other byte of every format must +/// match. +/// /// What it does not catch: interactive TTY-only ANSI escape sequences. +fn normalize_volatile_fields(stdout: &str) -> String { + /// Blank the value of every `"KEY":` occurrence, quoted or numeric. + fn blank_field(s: &str, key: &str) -> String { + let marker = format!("\"{key}\":"); + let mut out = String::with_capacity(s.len()); + let mut rest = s; + while let Some(i) = rest.find(&marker) { + let after = rest[i + marker.len()..].trim_start(); + let leading_ws = rest[i + marker.len()..].len() - after.len(); + if after.starts_with('"') { + match after[1..].find('"') { + Some(end) => { + out.push_str(&rest[..i + marker.len() + leading_ws]); + out.push_str("\"\""); + rest = &after[end + 2..]; + } + None => break, + } + } else { + let digits = after + .find(|c: char| !c.is_ascii_digit() && c != '.') + .unwrap_or(after.len()); + out.push_str(&rest[..i + marker.len() + leading_ws]); + out.push_str(""); + rest = &after[digits..]; + } + } + out.push_str(rest); + out + } + let mut out = stdout.to_owned(); + for key in [ + "scan_id", + "generated_at", + "scan_started_at", + "scan_finished_at", + "created", + "duration_ms", + "written_at_ns", + ] { + out = blank_field(&out, key); + } + out +} + #[test] fn deterministic_scan_formats_across_all_registered_reporters() { use clap::ValueEnum; @@ -91,15 +141,18 @@ fn deterministic_scan_formats_across_all_registered_reporters() { a.code, b.code ); assert_eq!( - a.stdout, b.stdout, + normalize_volatile_fields(&a.stdout), + normalize_volatile_fields(&b.stdout), "format `{fmt_str}`: stdout is NOT deterministic across identical runs over planted corpus.\n--- run A ---\n{}\n--- run B ---\n{}", a.stdout, b.stdout ); } } -/// WHY: baseline creation and incremental Merkle index serialization must be byte-identical -/// across repeated runs on the same input corpus. +/// WHY: baseline entries and incremental Merkle index serialization must be +/// byte-identical across repeated runs on the same input corpus. The baseline +/// header carries a `created` wall-clock stamp by design; it is normalized out +/// before comparison, and everything else must match. /// /// What it does not catch: corrupt filesystems during baseline writes. #[test] @@ -111,15 +164,17 @@ fn deterministic_baseline_and_merkle_index() { let baseline_a = temp_out.path().join("baseline_a.json"); let baseline_b = temp_out.path().join("baseline_b.json"); + // `--create-baseline` WRITES a baseline; `--baseline` is the compare-and- + // suppress flag and never writes. let args_base_a = [ "scan", - "--baseline-out", + "--create-baseline", baseline_a.to_str().unwrap(), corpus_path, ]; let args_base_b = [ "scan", - "--baseline-out", + "--create-baseline", baseline_b.to_str().unwrap(), corpus_path, ]; @@ -129,24 +184,33 @@ fn deterministic_baseline_and_merkle_index() { assert_no_panic(&run_a); assert_no_panic(&run_b); - let content_a = std::fs::read(&baseline_a).expect("read baseline a"); - let content_b = std::fs::read(&baseline_b).expect("read baseline b"); assert_eq!( - content_a, content_b, - "baseline files must be byte-identical across runs" + normalize_volatile_fields( + &String::from_utf8(std::fs::read(&baseline_a).expect("read baseline a")) + .expect("baseline is UTF-8") + ), + normalize_volatile_fields( + &String::from_utf8(std::fs::read(&baseline_b).expect("read baseline b")) + .expect("baseline is UTF-8") + ), + "baseline files must be identical apart from the created stamp" ); let merkle_a = temp_out.path().join("merkle_a.json"); let merkle_b = temp_out.path().join("merkle_b.json"); + // `--incremental-cache` only overrides the index location; `--incremental` + // is what turns the Merkle index on and makes the scan persist it. let args_merkle_a = [ "scan", + "--incremental", "--incremental-cache", merkle_a.to_str().unwrap(), corpus_path, ]; let args_merkle_b = [ "scan", + "--incremental", "--incremental-cache", merkle_b.to_str().unwrap(), corpus_path, @@ -157,10 +221,12 @@ fn deterministic_baseline_and_merkle_index() { assert_no_panic(&run_m_a); assert_no_panic(&run_m_b); - let merkle_content_a = std::fs::read(&merkle_a).expect("read merkle a"); - let merkle_content_b = std::fs::read(&merkle_b).expect("read merkle b"); + let normalize = |bytes: Vec| { + normalize_volatile_fields(&String::from_utf8(bytes).expect("index is UTF-8")) + }; assert_eq!( - merkle_content_a, merkle_content_b, - "merkle index files must be byte-identical across runs" + normalize(std::fs::read(&merkle_a).expect("read merkle a")), + normalize(std::fs::read(&merkle_b).expect("read merkle b")), + "merkle index files must be identical apart from the write stamp" ); } diff --git a/crates/cli/tests/reliability/exit_contract.rs b/crates/cli/tests/reliability/exit_contract.rs index f874cd64c..d379fd5d8 100644 --- a/crates/cli/tests/reliability/exit_contract.rs +++ b/crates/cli/tests/reliability/exit_contract.rs @@ -10,9 +10,13 @@ use tempfile::TempDir; use crate::reliability::harness::binary; -/// A planted AWS key (same shape the e2e suite uses) - a high-confidence, -/// network-free detection so the verdict is "secret found" without --verify. -const PLANTED_AWS: &str = "AWS_ACCESS_KEY_ID = \"AKIAQYLPMN5HFIQR7XYA\"\n"; +/// A planted GitHub classic PAT with a valid embedded checksum. It reaches the +/// `confirmed` evidence tier, which blocks under the default policy, so the +/// verdict is "secret found" without --verify. The former AWS-key fixture +/// (`AKIAQYLPMN5HFIQR7XYA` alone in a text file) now lands at review tier +/// (unsupported context), and review-tier findings are documented as visible +/// under exit 0, so it can no longer stand in for a blocking finding. +const PLANTED_TOKEN: &str = "GITHUB_TOKEN = \"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\"\n"; fn scan_file(content: &str, extra: &[&str]) -> (Option, String, String) { let dir = TempDir::new().unwrap(); @@ -59,7 +63,7 @@ fn clean_file_exits_zero() { #[test] fn planted_secret_exits_one() { - let (code, _o, _e) = scan_file(PLANTED_AWS, &["--format", "json"]); + let (code, _o, _e) = scan_file(PLANTED_TOKEN, &["--format", "json"]); assert_eq!( code, Some(1), @@ -86,7 +90,7 @@ fn exit_code_is_independent_of_output_format() { // format flag changing the exit code would silently break CI gating. let mut codes = vec![]; for fmt in ["text", "json", "jsonl", "sarif"] { - let (code, _o, _e) = scan_file(PLANTED_AWS, &["--format", fmt]); + let (code, _o, _e) = scan_file(PLANTED_TOKEN, &["--format", fmt]); codes.push((fmt, code)); } for (fmt, code) in &codes { @@ -129,7 +133,7 @@ fn help_documents_every_exit_code_it_can_return() { fn repeated_identical_scans_return_the_same_exit_code() { let dir = TempDir::new().unwrap(); let path = dir.path().join("planted.txt"); - std::fs::write(&path, PLANTED_AWS).unwrap(); + std::fs::write(&path, PLANTED_TOKEN).unwrap(); let (c1, _, _) = run_scan(&path, &["--format", "json"]); let (c2, _, _) = run_scan(&path, &["--format", "json"]); let (c3, _, _) = run_scan(&path, &["--format", "json"]); diff --git a/crates/cli/tests/unit/skip_dirs_policy.rs b/crates/cli/tests/unit/skip_dirs_policy.rs index 6d112564b..d749d05c5 100644 --- a/crates/cli/tests/unit/skip_dirs_policy.rs +++ b/crates/cli/tests/unit/skip_dirs_policy.rs @@ -21,6 +21,21 @@ fn bundled_policy_contains_consumer_specific_components() { assert!(policy.is_git_discovery_component(".nuxt")); assert!(policy.is_git_discovery_component("system volume information")); assert!(!policy.is_git_discovery_component(".git")); + // Scanner-truth parity: the scanner's default excludes prune no `.cargo` + // component, so no CLI surface may skip it either. Re-adding it here + // reintroduces the guard coverage hole where edits to a repository's + // `.cargo/credentials.toml` never reached the watcher (Row 141). + assert!(!policy.is_watch_component(".cargo")); + assert!(!policy.is_git_discovery_component(".cargo")); + + // An all-empty section is now valid: source defaults alone are a complete + // policy. The bundled file carries `base = []` for exactly that reason. + let empty = API + .skip_dir_policy_from_toml( + "[skip_dirs]\nbase = []\nwatch_extra = [\".svn\"]\ngit_discovery_extra = [\"Library\"]\n", + ) + .expect("empty base list is a valid policy"); + assert!(empty.is_watch_component("node_modules")); } #[test] diff --git a/crates/core/CHANGELOG.md b/crates/core/CHANGELOG.md index 38d95d22c..9da6ef3fe 100644 --- a/crates/core/CHANGELOG.md +++ b/crates/core/CHANGELOG.md @@ -2,6 +2,7 @@ ## 0.5.81 - 2026-08-20 +- fix(core): `--lockdown` starts on machines with installed execution packs. The past-findings gate treated the signed pack store under `/keyhog/execution-packs` as a findings-bearing cache; a directory with that exact name is now trusted, and any other entry still fails closed. - refactor(core): unify atomic durable writes across state artifacts and scanner caches into keyhog_core::state_file (Row 148). - feat(guard): populate GuardPolicyIdentity with canonical default digests and digest calculation helpers for ignore files, suppressions, config, and source policy (Row 142). - feat(spec): export `load_detectors_with_gate` so audit surfaces can load a corpus without the fail-closed quality gate. diff --git a/crates/core/src/hardening.rs b/crates/core/src/hardening.rs index 5719b7e56..cb9b76b87 100644 --- a/crates/core/src/hardening.rs +++ b/crates/core/src/hardening.rs @@ -367,6 +367,16 @@ where return true; } }; + // Signed execution packs live under the cache root by product + // convention (`/keyhog/execution-packs`). They hold compiled + // detector packs and a signing key, never findings, so a directory + // with that exact name is clean. Any other name, or a non-directory + // using it, still fails closed. + if entry.file_name() == std::ffi::OsStr::new(crate::EXECUTION_PACKS_SUBDIR) + && matches!(entry.file_type(), Ok(t) if t.is_dir()) + { + continue; + } match trusted_compiled_pattern_cache_entry(&entry) { Ok(true) => {} Ok(false) => return true, diff --git a/crates/core/src/lib.rs b/crates/core/src/lib.rs index 94ec5a865..151211029 100644 --- a/crates/core/src/lib.rs +++ b/crates/core/src/lib.rs @@ -139,6 +139,10 @@ pub(crate) const STALE_TMP_CUTOFF_SECS: u64 = 60 * 60; /// lockdown past-findings gate, a rename here moves all three together so the /// lockdown scan can never desynchronize from where scan artifacts actually land. pub(crate) const KEYHOG_CACHE_SUBDIR: &str = "keyhog"; +/// Subdirectory of [`KEYHOG_CACHE_SUBDIR`] holding signed execution packs. +/// Contains compiled detector packs and a signing key only, never findings, +/// so lockdown treats it as clean. +pub const EXECUTION_PACKS_SUBDIR: &str = "execution-packs"; /// Sibling of [`KEYHOG_CACHE_SUBDIR`] used for MatcherArtifact `.khm` files. pub const KEYHOG_MATCHER_ARTIFACTS_SUBDIR: &str = "keyhog-matcher-artifacts"; /// On-disk magic for MatcherArtifact cache files (`KHMA`). diff --git a/crates/core/tests/unit/lockdown_disk_cache_violations.rs b/crates/core/tests/unit/lockdown_disk_cache_violations.rs index 1d0e5893d..cb41117c2 100644 --- a/crates/core/tests/unit/lockdown_disk_cache_violations.rs +++ b/crates/core/tests/unit/lockdown_disk_cache_violations.rs @@ -255,3 +255,48 @@ fn matcher_artifact_tmp_named_file_is_lockdown_violation() { ); }); } + +// WHY: `keyhog install` and the GitHub Action place signed execution packs +// under `/keyhog/execution-packs`. Lockdown's past-findings gate used +// to treat any non-cache-file entry as a violation, so every machine with +// installed packs failed `--lockdown` at startup. The pack store holds +// compiled detector packs plus a signing key, which cannot carry findings. +// Closes the "installed packs block lockdown" class; it does not vet pack +// contents (signature checks happen when a scan loads packs). +#[test] +fn execution_packs_dir_is_not_lockdown_violation() { + with_xdg_cache_home(|cache_home| { + let keyhog_cache = cache_home.path().join("keyhog"); + let packs = keyhog_cache.join("execution-packs"); + std::fs::create_dir_all(packs.join("current")).expect("create packs dir"); + std::fs::write(packs.join("signing.key"), b"k".repeat(32)).expect("write key"); + std::fs::write(packs.join("current").join("manifest.json"), b"{}").expect("write manifest"); + + assert!( + keyhog_core::testing::CoreTestApi::lockdown_disk_cache_violations( + &keyhog_core::testing::TestApi, + ) + .is_empty(), + "installed execution packs must not fail lockdown" + ); + }); +} + +#[test] +fn execution_packs_named_file_is_lockdown_violation() { + with_xdg_cache_home(|cache_home| { + let keyhog_cache = cache_home.path().join("keyhog"); + std::fs::create_dir_all(&keyhog_cache).expect("create cache dir"); + std::fs::write(keyhog_cache.join("execution-packs"), b"payload") + .expect("write impostor file"); + + assert_eq!( + keyhog_core::testing::CoreTestApi::lockdown_disk_cache_violations( + &keyhog_core::testing::TestApi, + ) + .is_empty(), + false, + "a file named execution-packs is not the pack store and must violate lockdown" + ); + }); +} diff --git a/crates/core/tests/unit/sarif_skip_notifications.rs b/crates/core/tests/unit/sarif_skip_notifications.rs index 0ddc908ba..9bf5e2af1 100644 --- a/crates/core/tests/unit/sarif_skip_notifications.rs +++ b/crates/core/tests/unit/sarif_skip_notifications.rs @@ -13,7 +13,7 @@ fn sarif_skip_summary_emits_tool_execution_notifications() { ("binary (extension or content sniff)".to_string(), 5), ("unreadable (permission denied or I/O error)".to_string(), 2), ( - "default exclusion policy (lock files, minified/bundled assets, vendored and build-output trees). User `.keyhogignore` / --exclude-paths removals are not counted here" + "exclusion policy (default excludes such as lock files, minified/bundled assets, vendored and build-output trees; --git-staged also counts repository `.keyhogignore` matches here)" .to_string(), 0, ), // dropped diff --git a/crates/scanner/CHANGELOG.md b/crates/scanner/CHANGELOG.md index 21dd70757..dda4e19b9 100644 --- a/crates/scanner/CHANGELOG.md +++ b/crates/scanner/CHANGELOG.md @@ -4,6 +4,8 @@ ## Unreleased - test(scanner): the decode-through adversarial gate records the proven `helicone-api-key` floor. The bare read/write key patterns exclude keys directly following a quote, so quoted keys route to the structured-value and provider-context patterns and a sibling assignment such as an OpenAI-shaped key stays unclaimed; base64/hex/url decode splicing re-emits the recovered key right after the wrapper quote, a shape no local context can separate from that forbidden sibling case. The six variants are documented in the gate's allowlist with this reason. +- test(scanner): allocation instrumentation serializes the two tests that share its process-global counting allocator, so concurrent test allocations cannot corrupt identical-scan determinism measurements. +- test(scanner): the GPU region-batch sharding gate asserts an achievable CUDA budget cap. The old assertion required a 128 MiB explicit input budget to yield the 1 GiB VYRE ceiling, which no input can satisfy; it now proves the budget caps the shard below the ceiling and that an unbounded budget derives from the CUDA limit. ## 0.5.81 - 2026-08-20 diff --git a/crates/scanner/src/engine/phase1_admission/mod.rs b/crates/scanner/src/engine/phase1_admission/mod.rs index 6e625a92c..034eff088 100644 --- a/crates/scanner/src/engine/phase1_admission/mod.rs +++ b/crates/scanner/src/engine/phase1_admission/mod.rs @@ -516,7 +516,7 @@ impl Phase1AdmissionPlan { fn direct_scan_absence_at_row(&self, row: usize) -> Option { Some(self.phase2_tail_absence_at_row(row)? && self.cpu_trigger_hints.get(row)?.is_some()) } - #[cfg(feature = "simd")] + #[inline] pub(crate) fn simd_phase2_tail_absence_for( &self, diff --git a/crates/scanner/src/gpu.rs b/crates/scanner/src/gpu.rs index 8e03ec590..0aef38e68 100644 --- a/crates/scanner/src/gpu.rs +++ b/crates/scanner/src/gpu.rs @@ -275,4 +275,3 @@ mod gpu_evidence_cpu_silence_tests; #[cfg(all(test, feature = "gpu"))] #[path = "../tests/unit/gpu_evidence_recovery.rs"] mod gpu_evidence_recovery_tests; - diff --git a/crates/scanner/tests/perf_scan_path_counting_instrument.rs b/crates/scanner/tests/perf_scan_path_counting_instrument.rs index d854ae8be..de36f39a4 100644 --- a/crates/scanner/tests/perf_scan_path_counting_instrument.rs +++ b/crates/scanner/tests/perf_scan_path_counting_instrument.rs @@ -19,6 +19,7 @@ static ALLOC_COUNT: AtomicUsize = AtomicUsize::new(0); static TOTAL_BYTES: AtomicUsize = AtomicUsize::new(0); static CURRENT_BYTES: AtomicUsize = AtomicUsize::new(0); static PEAK_BYTES: AtomicUsize = AtomicUsize::new(0); +static MEASUREMENT_LOCK: std::sync::Mutex<()> = std::sync::Mutex::new(()); unsafe impl GlobalAlloc for ScanCountingAllocator { unsafe fn alloc(&self, layout: Layout) -> *mut u8 { @@ -137,6 +138,7 @@ fn make_test_scanner() -> CompiledScanner { #[test] fn scan_path_counting_instrument_measures_allocations_and_finding_parity() { + let _measurement_guard = MEASUREMENT_LOCK.lock().unwrap_or_else(|e| e.into_inner()); let scanner = make_test_scanner(); // Planted secrets corpus: AWS Access Key ID + Slack token @@ -188,6 +190,7 @@ fn scan_path_counting_instrument_measures_allocations_and_finding_parity() { #[test] fn scan_scaling_ratio_is_bounded_with_chunk_size() { + let _measurement_guard = MEASUREMENT_LOCK.lock().unwrap_or_else(|e| e.into_inner()); let scanner = make_test_scanner(); let text_small = "const MSG = 'hello world';\n".repeat(100); diff --git a/crates/scanner/tests/unit/checksum.rs b/crates/scanner/tests/unit/checksum.rs index a2b8c2a13..f7e2a3c48 100644 --- a/crates/scanner/tests/unit/checksum.rs +++ b/crates/scanner/tests/unit/checksum.rs @@ -230,12 +230,24 @@ fn checksum_prefixes_are_backed_by_their_detector() { ("gldt-", "gitlab-package-registry-token"), ("glrt-", "gitlab-runner-authentication-token"), ("glagent-", "gitlab-agent-token"), + ("glimt-", "gitlab-incoming-mail-token"), + ("glptt-", "gitlab-pipeline-trigger-token"), + ("glsoat-", "gitlab-scim-token"), ("glffct-", "gitlab-feature-flags-client-token"), ("glft-", "gitlab-feed-token"), ("npm_", "npm-access-token"), ("pypi-", "pypi-api-token"), + ("xapp-", "slack-app-token"), ("xoxb-", "slack-bot-token"), ("xoxp-", "slack-user-token"), + ("xoxe-", "slack-config-token"), + ("xoxe.", "slack-config-token"), + ("eyJ", "jwt-token"), + ("shpat_", "shopify-admin-api-token"), + ("shpca_", "shopify-access-token"), + ("shpss_", "shopify-storefront-api-token"), + ("sq0atp-", "square-access-token"), + ("sq0csp-", "square-access-token"), ("sk_live_", "stripe-secret-key"), ("sk_test_", "stripe-secret-key"), ("rk_live_", "stripe-secret-key"), diff --git a/crates/scanner/tests/unit/gpu_region_batch_sharding.rs b/crates/scanner/tests/unit/gpu_region_batch_sharding.rs index 992ef055c..594960462 100644 --- a/crates/scanner/tests/unit/gpu_region_batch_sharding.rs +++ b/crates/scanner/tests/unit/gpu_region_batch_sharding.rs @@ -149,10 +149,18 @@ fn backend_limits_keep_wgpu_inside_its_portable_grid() { CUDA_BYTE_SCAN_DISPATCH_LIMIT.min(crate::gpu_input_budget::gpu_batch_input_limit()) ); assert_eq!(WGPU_BYTE_SCAN_DISPATCH_LIMIT, 8_388_480); + // The CUDA backend limit (the VYRE scan ceiling) sits ABOVE a 128 MiB + // input budget, so an explicit budget below the ceiling is authoritative + // and caps the shard; with no binding budget the CUDA limit governs. assert_eq!( region_presence_batch_byte_limit_for_input_budget("cuda", 128 * 1024 * 1024), + 128 * 1024 * 1024, + "CUDA positioned-match shards are capped by the explicit input budget" + ); + assert_eq!( + region_presence_batch_byte_limit_for_input_budget("cuda", usize::MAX), CUDA_BYTE_SCAN_DISPATCH_LIMIT, - "CUDA positioned-match shards derive from CUDA limits" + "without a binding budget, shards derive from CUDA limits" ); assert_eq!( region_presence_batch_byte_limit_for_input_budget("wgpu", 128 * 1024 * 1024), diff --git a/crates/sources/src/docker/archive.rs b/crates/sources/src/docker/archive.rs index 488efc813..c01cbbff3 100644 --- a/crates/sources/src/docker/archive.rs +++ b/crates/sources/src/docker/archive.rs @@ -110,7 +110,10 @@ pub(super) fn unpack_tar( // Disk unpack keeps validate-before-write so a tar-header bomb cannot create // entries before the cap refuses the archive. Production layer scanning uses // `stream_layer_archive_chunks` instead and never materializes members. - unpack_open_tar(file, destination, limits, false, budget) + // The OUTER image archive is only a container: its own blob sizes must not + // consume the extracted-content budget, or the guard trips on the container + // before any layer content is measured (and names a digest, not an entry). + unpack_open_tar(file, destination, limits, false, budget, false) } pub(super) fn unpack_layer_archive( @@ -124,7 +127,9 @@ pub(super) fn unpack_layer_archive( file.rewind().map_err(SourceError::Io)?; match encoding { - LayerArchiveEncoding::RawTar => unpack_open_tar(file, destination, limits, true, budget), + LayerArchiveEncoding::RawTar => { + unpack_open_tar(file, destination, limits, true, budget, true) + } LayerArchiveEncoding::GzipTar => { validate_tar_reader( flate2::read::MultiGzDecoder::new(&mut file), @@ -139,6 +144,7 @@ pub(super) fn unpack_layer_archive( limits, true, budget, + true, ) } LayerArchiveEncoding::ZstdTar => { @@ -159,7 +165,7 @@ pub(super) fn unpack_layer_archive( limits.docker_tar_total_bytes, )) .map_err(SourceError::Io)?; - unpack_tar_reader(extract_reader, destination, limits, true, budget) + unpack_tar_reader(extract_reader, destination, limits, true, budget, true) } } } @@ -226,6 +232,7 @@ fn unpack_open_tar( limits: crate::SourceLimits, enforce_per_file_cap: bool, budget: &DockerUnpackBudget, + charge_image_budget: bool, ) -> Result { let mut validation_archive = tar::Archive::new(&mut file); validate_docker_archive_plan( @@ -233,10 +240,18 @@ fn unpack_open_tar( limits, enforce_per_file_cap, budget, + charge_image_budget, )?; file.rewind().map_err(SourceError::Io)?; - unpack_tar_reader(&mut file, destination, limits, enforce_per_file_cap, budget) + unpack_tar_reader( + &mut file, + destination, + limits, + enforce_per_file_cap, + budget, + charge_image_budget, + ) } fn validate_tar_reader( @@ -246,7 +261,7 @@ fn validate_tar_reader( budget: &DockerUnpackBudget, ) -> Result<(), SourceError> { let mut archive = tar::Archive::new(reader); - validate_docker_archive_plan(&mut archive, limits, enforce_per_file_cap, budget) + validate_docker_archive_plan(&mut archive, limits, enforce_per_file_cap, budget, true) } fn unpack_tar_reader( @@ -255,6 +270,7 @@ fn unpack_tar_reader( limits: crate::SourceLimits, enforce_per_file_cap: bool, budget: &DockerUnpackBudget, + charge_image_budget: bool, ) -> Result { let mut archive = tar::Archive::new(reader); extract_docker_archive_entries( @@ -263,6 +279,7 @@ fn unpack_tar_reader( limits, enforce_per_file_cap, budget, + charge_image_budget, ) } @@ -873,11 +890,38 @@ fn validate_extracted_tree_with_limits( Ok(()) } +/// Normalize a tar entry path so a `./` or repeated-curdir prefix cannot dodge +/// prefix matching. +fn normalize_entry_path(path: &Path) -> std::path::PathBuf { + path.components() + .filter(|component| !matches!(component, std::path::Component::CurDir)) + .collect() +} + +/// Whether an OUTER container entry is itself a layer payload that the scan +/// re-reads and accounts through its extracted inner entries. Charging its +/// packaged size in the outer pass would trip the image-wide guard before any +/// layer content is measured, naming a digest instead of the entry that +/// actually exceeded the budget. Covers the OCI layout (`blobs/sha256/`) +/// and legacy `docker save` layouts (`/layer.tar`). Everything else, +/// including `manifest.json` and per-layer metadata files, is scanned input and +/// counts. +fn outer_entry_is_layer_payload(path: &Path) -> bool { + let normalized = normalize_entry_path(path); + if normalized.starts_with("blobs") && normalized.components().count() >= 2 { + return true; + } + normalized + .file_name() + .is_some_and(|name| name == "layer.tar") +} + fn validate_docker_archive_plan( archive: &mut tar::Archive, limits: crate::SourceLimits, enforce_per_file_cap: bool, budget: &DockerUnpackBudget, + charge_image_budget: bool, ) -> Result<(), SourceError> { let mut cumulative_bytes: u64 = 0; for (entry_index, entry) in archive.entries().map_err(SourceError::Io)?.enumerate() { @@ -890,14 +934,19 @@ fn validate_docker_archive_plan( let file_type = entry.header().entry_type(); validate_docker_archive_entry(&path, file_type)?; - cumulative_bytes = cumulative_bytes.saturating_add(size); - if cumulative_bytes > budget.remaining() { - let _event = crate::record_skip_event(crate::SourceSkipEvent::ArchiveTruncated); - return Err(SourceError::Other(format!( - "docker archive cumulative size exceeds {} bytes at entry '{}' (likely zip-bomb)", - limits.docker_tar_total_bytes, - path.display(), - ))); + // Same spend rule as `extract_docker_archive_entries`, so the + // validate-before-write guarantee covers exactly the entries that + // extraction will charge. + if charge_image_budget || !outer_entry_is_layer_payload(&path) { + cumulative_bytes = cumulative_bytes.saturating_add(size); + if cumulative_bytes > budget.remaining() { + let _event = crate::record_skip_event(crate::SourceSkipEvent::ArchiveTruncated); + return Err(SourceError::Other(format!( + "docker archive cumulative size exceeds {} bytes at entry '{}' (likely zip-bomb)", + limits.docker_tar_total_bytes, + path.display(), + ))); + } } if enforce_per_file_cap && docker_archive_entry_exceeds_scan_cap(file_type, size, limits) { @@ -914,6 +963,7 @@ fn extract_docker_archive_entries( limits: crate::SourceLimits, enforce_per_file_cap: bool, budget: &DockerUnpackBudget, + charge_image_budget: bool, ) -> Result { let mut report = DockerExtractReport::default(); for (entry_index, entry) in archive.entries().map_err(SourceError::Io)?.enumerate() { @@ -926,8 +976,12 @@ fn extract_docker_archive_entries( validate_docker_archive_entry(&path, entry.header().entry_type())?; // The one site that SPENDS the image budget for disk unpack. Streaming - // layer scans charge through `stream_layer_tar_reader` instead. - if !budget.charge(size) { + // layer scans charge through `stream_layer_tar_reader` instead. In the + // outer container pass, layer payloads (see + // `outer_entry_is_layer_payload`) are NOT charged here; metadata such + // as `manifest.json` IS scanned input and counts. + let spends_budget = charge_image_budget || !outer_entry_is_layer_payload(&path); + if spends_budget && !budget.charge(size) { return Err(docker_image_budget_error( &path, limits.docker_tar_total_bytes, diff --git a/docs/src/guides/file-shapes.md b/docs/src/guides/file-shapes.md index 89359f8a9..4a53d78f0 100644 --- a/docs/src/guides/file-shapes.md +++ b/docs/src/guides/file-shapes.md @@ -148,7 +148,7 @@ two gap rows and exits `13`: "scan_status": "partial", "coverage_gap_summary": [ {"reason": "scan covered nothing (zero source bytes read; every candidate was skipped by exclusion or skip policy, so nothing was examined)", "count": 1}, - {"reason": "default exclusion policy (lock files, minified/bundled assets, vendored and build-output trees). User `.keyhogignore` / --exclude-paths removals are not counted here", "count": 1} + {"reason": "exclusion policy (default excludes such as lock files, minified/bundled assets, vendored and build-output trees; --git-staged also counts repository `.keyhogignore` matches here)", "count": 1} ] } ``` diff --git a/docs/src/reference/coverage-truth.md b/docs/src/reference/coverage-truth.md index 1379a9841..ccde43496 100644 --- a/docs/src/reference/coverage-truth.md +++ b/docs/src/reference/coverage-truth.md @@ -99,7 +99,7 @@ jq '{status: .scan_status, gaps: .coverage_gap_summary}' keyhog.json "count": 1 }, { - "reason": "default exclusion policy (lock files, minified/bundled assets, vendored and build-output trees). User `.keyhogignore` / --exclude-paths removals are not counted here", + "reason": "exclusion policy (default excludes such as lock files, minified/bundled assets, vendored and build-output trees; --git-staged also counts repository `.keyhogignore` matches here)", "count": 6690 } ] diff --git a/docs/src/suppressions.md b/docs/src/suppressions.md index 944b935ce..3be1ae920 100644 --- a/docs/src/suppressions.md +++ b/docs/src/suppressions.md @@ -33,7 +33,7 @@ A directory scan skips some files before detection starts. Put a credential in ``` ```text -WARN 1 path(s) skipped by the DEFAULT exclusion policy (lock files, minified/bundled assets, vendored and build-output trees). Default-excluded directories are pruned during discovery and counted once each; nested files under them are not enumerated. Pass `--no-default-excludes` to scan them. +WARN 1 path(s) skipped by the exclusion policy (default excludes such as lock files, minified/bundled assets, vendored and build-output trees). Default-excluded directories are pruned during discovery and counted once each; nested files under them are not enumerated. Pass `--no-default-excludes` to scan them. A `--git-staged` scan also counts staged paths removed by the repository's `.keyhogignore` here. ``` The warning goes to stderr and the exit code stays `0`, so a CI job that reads diff --git a/fuzz/fuzz_targets/scanner_target.rs b/fuzz/fuzz_targets/scanner_target.rs index af0f9692f..52da8effd 100644 --- a/fuzz/fuzz_targets/scanner_target.rs +++ b/fuzz/fuzz_targets/scanner_target.rs @@ -55,7 +55,40 @@ fn scanner() -> &'static CompiledScanner { // `worst_case_backend_parity` test. const FUZZ_DETECTOR_CAP: usize = 64; let stride = (all.len() / FUZZ_DETECTOR_CAP).max(1); - let detectors: Vec<_> = all.into_iter().step_by(stride).take(FUZZ_DETECTOR_CAP).collect(); + // The plan builder rejects any relation whose target is outside the + // compiled set, and a stride sample can split a relation pair (e.g. + // notion-integration-token -> notion-api-key), so the raw sample does + // not always compile. Close the selection transitively over relation + // targets: init stays bounded by the same order of magnitude while the + // subset remains compilable. + let mut chosen = vec![false; all.len()]; + for i in (0..all.len()).step_by(stride).take(FUZZ_DETECTOR_CAP) { + chosen[i] = true; + } + { + let index: std::collections::HashMap<&str, usize> = all + .iter() + .enumerate() + .map(|(i, d)| (d.id.as_str(), i)) + .collect(); + let mut queue: Vec = + (0..all.len()).filter(|&i| chosen[i]).collect(); + while let Some(i) = queue.pop() { + for target in &all[i].detector_relations { + if let Some(&j) = index.get(target.detector_id.as_str()) { + if !chosen[j] { + chosen[j] = true; + queue.push(j); + } + } + } + } + } + let detectors: Vec<_> = all + .into_iter() + .zip(chosen) + .filter_map(|(d, keep)| keep.then_some(d)) + .collect(); CompiledScanner::compile(detectors).expect("scanner compile") })