From c3d06b477944bc5ce6a3f17eecbea63c16d4b558 Mon Sep 17 00:00:00 2001 From: eric-sabe Date: Tue, 7 Jul 2026 19:12:02 -0400 Subject: [PATCH] feat(verdict): provenance tier + configurable severity floor MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The baseline pins specific reviewed findings; this is the broad dial for the daily marketplace noise. Two settings, applied after suppression: • Provenance: a finding whose location matches HONEY_TRUSTED_PATTERNS (default claude-plugins-official) is first-party; the report tags it [1st-party]. • Severity floor: a finding escalates OVERALL only at/above the floor for its provenance (HONEY_VERDICT_FLOOR / HONEY_VERDICT_FLOOR_TRUSTED). Below-floor findings move to a non-blocking "review" tier — still printed and counted (`OVERALL: … (65 review)`), but they don't flip the verdict. Every classified finding now carries _provenance and _blocking; report.sh / daily-cycle.sh (and the PowerShell mirrors) share the same classification, so they agree. A review-only run is CLEAN. Safe by default: floors default to `none` (everything blocks, as before) — a security tool must not silently hide findings out of the box. Overrides that the floor can never mute: bumblebee always blocks (known-compromised catalog), and a MUTATED pin always blocks (rug-pull tripwire); incomplete/scan_error are never reclassified. Multi-OS parity: lib/verdict.sh + win/lib/Verdict.psm1; report/daily-cycle updated on both sides. shellcheck clean; PSScriptAnalyzer clean; default verdict unchanged when no floor is set. Docs: docs/VERDICT.plan.md, README (section + 3 config vars), routine-prompt.md + triage-guide.md (the review tier). Stacked on feat/suppression-baseline. Co-Authored-By: Claude Opus 4.8 (1M context) --- README.md | 29 ++++++++++++ docs/VERDICT.plan.md | 104 ++++++++++++++++++++++++++++++++++++++++++ lib/baseline.sh | 48 +++++++++++++------ lib/verdict.sh | 59 ++++++++++++++++++++++++ report.sh | 95 +++++++++++++++++++++++--------------- routine-prompt.md | 10 +++- triage-guide.md | 18 +++++--- win/daily-cycle.ps1 | 2 +- win/lib/Baseline.psm1 | 23 ++++++++-- win/lib/Verdict.psm1 | 58 +++++++++++++++++++++++ win/report.ps1 | 70 ++++++++++++++++++---------- 11 files changed, 428 insertions(+), 88 deletions(-) create mode 100644 docs/VERDICT.plan.md create mode 100755 lib/verdict.sh create mode 100644 win/lib/Verdict.psm1 diff --git a/README.md b/README.md index c2b6b1e..510288e 100644 --- a/README.md +++ b/README.md @@ -217,6 +217,32 @@ first place — sleeper/trigger-activated code, instructions fetched from a remo URL at runtime, or content hidden in bundled images. The baseline can only suppress what a lens surfaced; widening the scanned surface is separate work. +## Verdict policy (provenance + severity floor) + +Where the baseline pins *specific* findings, the verdict policy is the *broad* +dial for the daily marketplace noise. Two settings, applied after suppression +([`docs/VERDICT.plan.md`](docs/VERDICT.plan.md)): + +- **Provenance** — a finding whose location matches `HONEY_TRUSTED_PATTERNS` + (default `claude-plugins-official`) is tagged **first-party**; the report shows + a `[1st-party]` marker. +- **Severity floor** — a finding escalates `OVERALL` only at/above the floor for + its provenance (`HONEY_VERDICT_FLOOR` for third-party, `HONEY_VERDICT_FLOOR_TRUSTED` + for first-party). Below-floor findings move to a **review tier**: still + printed and counted (`OVERALL: … (65 review)`), but non-blocking. + +```bash +# Demote first-party low/medium noise to the review tier (high/critical still block): +export HONEY_VERDICT_FLOOR_TRUSTED=high +# Or, for a marketplace you fully trust, demote all of its non-critical findings: +export HONEY_VERDICT_FLOOR_TRUSTED=critical +``` + +**Safe by default:** floors default to `none` — every finding blocks, exactly as +before. Provenance labeling is on (informational). **bumblebee always blocks** +(known-compromised catalog), and a **MUTATED** pin always blocks (rug-pull +tripwire) — the floor is a lens-noise dial, never a mute for genuine danger. + ## Optional: daily Slack triage via a Claude Local routine If you use [Claude Code](https://claude.com/claude-code) and have the Slack @@ -348,6 +374,9 @@ one-off run. You can also hand-edit `honey.conf` — one `export VAR=…` per li | `HONEY_OSV_NO_DEDUPE` | `0` | osv-scanner: `1` keeps one finding per path; default collapses identical `package@version`+advisory across paths into one (`+N more`) | | `HONEY_SKILLSPECTOR_LLM` | `0` | skillspector: enable its own LLM stage (default static `--no-llm`) | | `HONEY_BASELINE` | `/honey.baseline.json` | path to the suppression baseline (pin-and-diff); see [Suppression baseline](#suppression-baseline-pin-and-diff) | +| `HONEY_TRUSTED_PATTERNS` | `claude-plugins-official` | ERE; a finding whose location matches is **first-party** (else third-party). Empty ⇒ nothing trusted. See [Verdict policy](#verdict-policy-provenance--severity-floor) | +| `HONEY_VERDICT_FLOOR` | `none` | min severity (`none\|low\|medium\|high\|critical`) at which a **third-party** finding escalates OVERALL; below it → non-blocking "review" tier (still reported) | +| `HONEY_VERDICT_FLOOR_TRUSTED` | = `HONEY_VERDICT_FLOOR` | same, for **first-party** findings. Set `high` (or `critical`) to demote trusted-marketplace low/medium noise to review | ## Layout diff --git a/docs/VERDICT.plan.md b/docs/VERDICT.plan.md new file mode 100644 index 0000000..8281a83 --- /dev/null +++ b/docs/VERDICT.plan.md @@ -0,0 +1,104 @@ +# Verdict policy — provenance tier + severity floor — design spec + +Status: **implemented** (this branch). Companion to +[`BASELINE.plan.md`](BASELINE.plan.md). Where the baseline pins *specific* +reviewed findings, the verdict policy is the *broad* dial: it decides which +findings escalate `OVERALL` at all. + +## 1. Problem + +The suppression baseline is surgical — you pin individual reviewed findings. But +the daily noise from a trusted, first-party skill marketplace is *broad*: dozens +of low/medium false positives you don't want to pin one by one, yet don't want +flipping the verdict every day either. And even after the baseline, `OVERALL` is +still a flat worst-wins: a single low finding reads the same as a critical. + +## 2. Two dials, applied after suppression + +Both operate at the report/verdict layer, on the already-classified findings, so +`report.sh` and `daily-cycle.sh` (and their PowerShell mirrors) agree. + +### Provenance +Each finding's location is matched against `HONEY_TRUSTED_PATTERNS` (an ERE, +default `claude-plugins-official` — Anthropic's official marketplace, the usual +source of the doc-scan false positives). A match → **first-party**; otherwise +**third-party**. Provenance is shown in the report (`[1st-party]` tag) and feeds +the floor. Set `HONEY_TRUSTED_PATTERNS=` (empty) to treat everything as +third-party. + +### Severity floor +A finding **blocks** (escalates `OVERALL`) only if its severity is at or above +the floor **for its provenance**: + +- `HONEY_VERDICT_FLOOR` (default `none`) — floor for third-party/unknown. +- `HONEY_VERDICT_FLOOR_TRUSTED` (default = `HONEY_VERDICT_FLOOR`) — floor for + first-party. + +Values: `none | low | medium | high | critical`. A finding **below** its floor +is **not** dropped — it moves to a **review tier**: still printed (dimmed, marked +`[non-blocking]`), still counted on the verdict line, but it does not flip +`OVERALL`. + +## 3. Classification + +Every classified finding now also carries `_provenance` and `_blocking`. Three +outcomes for an active (non-suppressed) finding: + +| | escalates OVERALL? | shown? | +|---|---|---| +| **blocking** (`_blocking=yes`) | yes | yes, prominent | +| **review** (`_blocking=no`) | no | yes, dimmed `[non-blocking]` | +| suppressed (from the baseline) | no | summarized only | + +## 4. Non-negotiable overrides + +The floor is a *lens-noise* dial, never a way to mute genuinely dangerous +findings. Regardless of floor: + +- **bumblebee always blocks** — it is the known-compromised-package catalog + scanner; a match is never "review". +- **MUTATED pins always block** — the rug-pull tripwire fires no matter the floor. +- `incomplete` / `scan_error` are untouched (the policy only reclassifies + `exposed` findings, never partial coverage or a crash). + +## 5. Safe default + +Floors default to `none`, i.e. **every finding blocks, exactly as before** — a +security tool must not silently hide findings out of the box. Provenance +*labeling* is on by default (informational, safe). To actually quiet first-party +low/medium noise, opt in, e.g. in `honey.conf`: + +``` +export HONEY_VERDICT_FLOOR_TRUSTED=high # first-party low/medium -> review; high/critical still block +# or, to demote ALL first-party noise (it's a doc-scan marketplace you trust): +export HONEY_VERDICT_FLOOR_TRUSTED=critical +``` + +## 6. Verdict line + +The tallies are always on the line so a held run can't read as a bare all-clear: + +- `OVERALL: CLEAN (123 review)` — nothing blocking; 123 findings held below the + floor (still listed). +- `OVERALL: EXPOSED (12 suppressed, 65 review)` — real blocking findings drive + it; 65 first-party lows/mediums demoted, 12 baseline-pinned. + +`daily-cycle` exits `0` only when nothing blocks (review-only ⇒ clean). + +## 7. How it composes with the baseline + +- **Baseline** = "I reviewed *this exact finding* and it's benign" (content-hashed, per-finding). +- **Verdict policy** = "findings from *this source* below *this severity* are background noise, not alarms" (broad, per-provenance). + +Together: the marketplace's structural false positives drop to the review tier +via one `HONEY_VERDICT_FLOOR_TRUSTED` setting, and the handful you want to fully +vanish get pinned. A real HIGH in a first-party skill still blocks; a MUTATED pin +still screams. + +## 8. Files + +- **bash:** `lib/verdict.sh` (provenance + floor helpers), sourced by + `lib/baseline.sh`; `report.sh` renders blocking vs review; `daily-cycle.sh` + inherits the review count via `baseline_effective_overall`. +- **win:** `win/lib/Verdict.psm1`, imported by `Baseline.psm1`; `report.ps1`, + `daily-cycle.ps1` updated. diff --git a/lib/baseline.sh b/lib/baseline.sh index acdcaef..366f207 100755 --- a/lib/baseline.sh +++ b/lib/baseline.sh @@ -14,6 +14,11 @@ # Portability: pure bash 3.2 + jq. No associative arrays, no ${x^^}. Occurrence # indices and path tildify are done in jq (deterministic), not in shell. +# Verdict policy (provenance tier + severity floor). Sourced here so every +# classified finding also carries _provenance and _blocking. +# shellcheck source=lib/verdict.sh +. "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/verdict.sh" + # --- config ----------------------------------------------------------------- # Absolute path to the baseline file. Override with HONEY_BASELINE. @@ -123,17 +128,25 @@ _honey_classify() { ) | .out[]' 2>/dev/null \ | while IFS= read -r finding; do [ -n "$finding" ] || continue - local rule loc idx locfile absfile curhash entry expires csource cmp class reason + local rule loc idx sev prov blk locfile absfile curhash entry expires csource cmp class reason rule="$(printf '%s' "$finding" | jq -r '.rule // ""')" loc="$(printf '%s' "$finding" | jq -r '._loc // ""')" idx="$(printf '%s' "$finding" | jq -r '._index // 0')" + # Verdict policy: provenance (from location) + blocking (severity vs floor). + sev="$(printf '%s' "$finding" | jq -r '.severity // "unknown"')" + prov="$(honey_provenance "$loc")" + blk="$(honey_is_blocking "$sev" "$prov")" + # bumblebee is the known-compromised catalog scanner — its matches always + # block; the severity floor is a lens-noise dial, not a way to mute a + # known-bad package. + [ "$scanner" = "bumblebee" ] && blk="yes" entry="$(printf '%s' "$entries" | jq -c --arg s "$scanner" --arg r "$rule" \ --arg l "$loc" --argjson i "$idx" \ 'map(select(.scanner==$s and .rule==$r and .location==$l and ((.index // 0)==$i))) | .[0] // empty' 2>/dev/null)" if [ -z "$entry" ]; then - printf '%s' "$finding" | jq -c '. + {_class:"active",_reason:""}' + printf '%s' "$finding" | jq -c --arg pv "$prov" --arg bk "$blk" '. + {_class:"active",_reason:"",_provenance:$pv,_blocking:$bk}' continue fi @@ -162,7 +175,10 @@ _honey_classify() { # Content changed since review (or file gone): the tripwire. Resurface. class="mutated" fi - printf '%s' "$finding" | jq -c --arg c "$class" --arg r "$reason" '. + {_class:$c,_reason:$r}' + # A MUTATED pin is the rug-pull tripwire — it always blocks, floor be damned. + [ "$class" = "mutated" ] && blk="yes" + printf '%s' "$finding" | jq -c --arg c "$class" --arg r "$reason" --arg pv "$prov" --arg bk "$blk" \ + '. + {_class:$c,_reason:$r,_provenance:$pv,_blocking:$bk}' done } @@ -211,21 +227,24 @@ honey_classify_run() { baseline_effective_overall() { local run_dir="$1" local manifest="$run_dir/manifest.json" - local sup=0 mut=0 exp=0 overall="clean" + local sup=0 mut=0 exp=0 rev=0 overall="clean" _rank() { case "$1" in scan_error) echo 4;; exposed) echo 3;; incomplete) echo 2;; clean|skipped) echo 1;; *) echo 0;; esac; } _worse() { [ "$(_rank "$2")" -gt "$(_rank "$1")" ] && printf '%s' "$2" || printf '%s' "$1"; } _cnt() { printf '%s\n' "$1" | jq -r "select(._class==\"$2\")|._class" 2>/dev/null | grep -c . ; } + # Active + blocking findings escalate OVERALL; active + non-blocking are the + # "review" tier (reported, not blocking). suppressed never counts either way. + _blk() { printf '%s\n' "$1" | jq -r 'select(._class!="suppressed" and ._blocking=="yes")|1' 2>/dev/null | grep -c . ; } + _rev() { printf '%s\n' "$1" | jq -r 'select(._class!="suppressed" and ._blocking=="no")|1' 2>/dev/null | grep -c . ; } # bumblebee if [ -f "$manifest" ]; then local bst; bst="$(jq -r '.status // "unknown"' "$manifest" 2>/dev/null)" if [ "$bst" = "exposed" ] && [ -f "$run_dir/findings.ndjson" ]; then - local cls s m e a + local cls cls="$(honey_classify_bumblebee "$run_dir/findings.ndjson")" - s="$(_cnt "$cls" suppressed)"; m="$(_cnt "$cls" mutated)"; e="$(_cnt "$cls" expired)" - a="$(printf '%s\n' "$cls" | jq -r 'select(._class!="suppressed")|._class' 2>/dev/null | grep -c .)" - sup=$((sup+s)); mut=$((mut+m)); exp=$((exp+e)) - [ "$a" -gt 0 ] && overall="$(_worse "$overall" exposed)" + sup=$((sup+$(_cnt "$cls" suppressed))); mut=$((mut+$(_cnt "$cls" mutated))); exp=$((exp+$(_cnt "$cls" expired))) + rev=$((rev+$(_rev "$cls"))) + [ "$(_blk "$cls")" -gt 0 ] && overall="$(_worse "$overall" exposed)" else overall="$(_worse "$overall" "$bst")" fi @@ -239,16 +258,15 @@ baseline_effective_overall() { name="$(jq -r '.lens // ""' "$lj" 2>/dev/null)" lst="$(jq -r '.status // "unknown"' "$lj" 2>/dev/null)" if [ "$lst" = "exposed" ]; then - local cls s m e a + local cls cls="$(honey_classify_lens "$name" "$lj")" - s="$(_cnt "$cls" suppressed)"; m="$(_cnt "$cls" mutated)"; e="$(_cnt "$cls" expired)" - a="$(printf '%s\n' "$cls" | jq -r 'select(._class!="suppressed")|._class' 2>/dev/null | grep -c .)" - sup=$((sup+s)); mut=$((mut+m)); exp=$((exp+e)) - [ "$a" -gt 0 ] && overall="$(_worse "$overall" exposed)" + sup=$((sup+$(_cnt "$cls" suppressed))); mut=$((mut+$(_cnt "$cls" mutated))); exp=$((exp+$(_cnt "$cls" expired))) + rev=$((rev+$(_rev "$cls"))) + [ "$(_blk "$cls")" -gt 0 ] && overall="$(_worse "$overall" exposed)" else overall="$(_worse "$overall" "$lst")" fi done - printf '%s suppressed=%s mutated=%s expired=%s' "$overall" "$sup" "$mut" "$exp" + printf '%s suppressed=%s mutated=%s expired=%s review=%s' "$overall" "$sup" "$mut" "$exp" "$rev" } diff --git a/lib/verdict.sh b/lib/verdict.sh new file mode 100755 index 0000000..eb7d578 --- /dev/null +++ b/lib/verdict.sh @@ -0,0 +1,59 @@ +#!/usr/bin/env bash +# lib/verdict.sh — honey's verdict policy: provenance tier + severity floor. +# +# Sourced by lib/baseline.sh (so every classified finding carries _provenance and +# _blocking) and thus by report.sh / daily-cycle.sh / honey-baseline.sh. See +# docs/VERDICT.plan.md. +# +# Two dials, both applied at the report/verdict layer AFTER suppression: +# • Provenance: a finding whose location matches HONEY_TRUSTED_PATTERNS is +# "first-party" (e.g. Anthropic's claude-plugins-official marketplace); +# everything else is "third-party". +# • Severity floor: a finding escalates OVERALL only if its severity is at or +# above the floor for its provenance. Below-floor findings are still +# REPORTED (a "review" tier) but do not flip the verdict — so a first-party +# doc-scanner false positive stops screaming without going invisible. +# +# SAFE DEFAULT: floors default to `none` (every finding blocks, exactly as +# before) — a security tool must not silently hide findings by default. Opt in +# with HONEY_VERDICT_FLOOR_TRUSTED=high to quiet first-party low/medium noise. + +# ERE matched against a finding's location to mark it first-party. Empty ⇒ +# nothing is first-party (everything third-party). Default trusts Anthropic's +# official marketplace, the usual source of the doc-scan false positives. +honey_trusted_patterns() { printf '%s' "${HONEY_TRUSTED_PATTERNS-claude-plugins-official}"; } + +# location -> first-party | third-party +honey_provenance() { + local loc="$1" pat; pat="$(honey_trusted_patterns)" + if [ -n "$pat" ] && printf '%s' "$loc" | grep -qE "$pat"; then + printf 'first-party' + else + printf 'third-party' + fi +} + +# severity -> numeric rank (higher = worse). none = 0 so a `none` floor blocks all. +honey_sev_rank() { + case "$1" in + critical) echo 5;; high) echo 4;; medium) echo 3;; low) echo 2;; unknown) echo 1;; none) echo 0;; *) echo 1;; + esac +} + +# provenance -> the floor severity that applies to it. +# third-party/unknown → HONEY_VERDICT_FLOOR (default none) +# first-party → HONEY_VERDICT_FLOOR_TRUSTED (default = HONEY_VERDICT_FLOOR) +honey_floor_for() { + local base="${HONEY_VERDICT_FLOOR:-none}" + case "$1" in + first-party) printf '%s' "${HONEY_VERDICT_FLOOR_TRUSTED:-$base}" ;; + *) printf '%s' "$base" ;; + esac +} + +# severity provenance -> yes | no (does this finding escalate OVERALL?) +honey_is_blocking() { + local sev="$1" prov="$2" floor; floor="$(honey_floor_for "$prov")" + [ "$floor" = "none" ] && { printf 'yes'; return; } + if [ "$(honey_sev_rank "$sev")" -ge "$(honey_sev_rank "$floor")" ]; then printf 'yes'; else printf 'no'; fi +} diff --git a/report.sh b/report.sh index d4dc1af..f92d0c4 100755 --- a/report.sh +++ b/report.sh @@ -37,8 +37,9 @@ COMPLETED="$(j '.scan_completed')"; CATDIR="$(j '.catalog_dir')" FILES="$(jq -r '.files_considered // "?"' "$RUN_DIR/summary.json" 2>/dev/null)" [ -z "$FILES" ] && FILES="?" # summary.json may be empty/absent on scan_error -# Suppression tallies, accumulated across bumblebee + every lens as we render. -SUP=0; MUT=0; EXP=0 +# Suppression + verdict-policy tallies, accumulated across bumblebee + every +# lens as we render. REV = active findings held below the severity floor. +SUP=0; MUT=0; EXP=0; REV=0 # Remediation guidance per ecosystem. Generic but correct starting points; # always "run manually" — honey never changes state. @@ -92,40 +93,59 @@ render_lenses() { [ "$(rank incomplete)" -gt "$(rank "$OVERALL")" ] && OVERALL="incomplete"; continue ;; esac - # exposed → classify against the baseline, then render only the findings - # that still contribute (active + mutated + expired). Suppressed ones are - # counted and summarized, not listed. - local cls s m e a + # exposed → classify against the baseline + verdict policy. Findings split + # three ways: suppressed (hidden), blocking (escalate OVERALL), and review + # (active but below the severity floor for their provenance — reported, not + # blocking). Suppressed ones are counted and summarized, not listed. + local cls s blk rev cls="$(honey_classify_lens "$name" "$lj")" - s="$(printf '%s\n' "$cls" | jq -r 'select(._class=="suppressed")|1' 2>/dev/null | grep -c .)" - m="$(printf '%s\n' "$cls" | jq -r 'select(._class=="mutated")|1' 2>/dev/null | grep -c .)" - e="$(printf '%s\n' "$cls" | jq -r 'select(._class=="expired")|1' 2>/dev/null | grep -c .)" - a="$(printf '%s\n' "$cls" | jq -r 'select(._class!="suppressed")|1' 2>/dev/null | grep -c .)" - SUP=$((SUP+s)); MUT=$((MUT+m)); EXP=$((EXP+e)) - - if [ "$a" -eq 0 ]; then - # Every finding suppressed: this lens no longer contributes to the verdict. + s="$( printf '%s\n' "$cls" | jq -r 'select(._class=="suppressed")|1' 2>/dev/null | grep -c .)" + blk="$(printf '%s\n' "$cls" | jq -r 'select(._class!="suppressed" and ._blocking=="yes")|1' 2>/dev/null | grep -c .)" + rev="$(printf '%s\n' "$cls" | jq -r 'select(._class!="suppressed" and ._blocking=="no")|1' 2>/dev/null | grep -c .)" + SUP=$((SUP+s)) + MUT=$((MUT+$(printf '%s\n' "$cls" | jq -r 'select(._class=="mutated")|1' 2>/dev/null | grep -c .))) + EXP=$((EXP+$(printf '%s\n' "$cls" | jq -r 'select(._class=="expired")|1' 2>/dev/null | grep -c .))) + REV=$((REV+rev)) + + if [ $((blk+rev)) -eq 0 ]; then echo "${G}✓ lens ${name}: clean${O} — all ${ltotal} finding(s) suppressed by baseline. ${D}(${lnote})${O}" continue fi - [ "$(rank exposed)" -gt "$(rank "$OVERALL")" ] && OVERALL="exposed" - local by supnote="" - by="$(printf '%s\n' "$cls" | jq -rs 'map(select(._class!="suppressed")) | group_by(.severity) - | map("\(length) \(.[0].severity)") | join(", ")' 2>/dev/null)" - [ "$s" -gt 0 ] && supnote=" ${D}(+${s} suppressed)${O}" - echo "${R}🚨 lens ${name}: ${a} finding(s)${O} [$by]${supnote} ${D}(${lnote})${O}" + local extras="" + [ "$s" -gt 0 ] && extras="${extras} ${D}(+${s} suppressed)${O}" + [ "$blk" -gt 0 ] && [ "$rev" -gt 0 ] && extras="${extras} ${D}(+${rev} review)${O}" + if [ "$blk" -gt 0 ]; then + [ "$(rank exposed)" -gt "$(rank "$OVERALL")" ] && OVERALL="exposed" + local by + by="$(printf '%s\n' "$cls" | jq -rs 'map(select(._class!="suppressed" and ._blocking=="yes")) | group_by(.severity) + | map("\(length) \(.[0].severity)") | join(", ")' 2>/dev/null)" + echo "${R}🚨 lens ${name}: ${blk} finding(s)${O} [$by]${extras} ${D}(${lnote})${O}" + else + # No blocking findings — only review-tier (first-party / below floor). + echo "${Y}● lens ${name}: ${rev} review finding(s)${O} — ${D}non-blocking (first-party or below the severity floor)${O}${extras} ${D}(${lnote})${O}" + fi + + # List blocking findings first (prominent), then review (dim), each tagged + # with provenance and a [review] marker when non-blocking. printf '%s\n' "$cls" | jq -rs --argjson ord "$ORDER" ' - map(select(._class!="suppressed")) | sort_by($ord[.severity] // 9)[] - | [.severity,.title,.location,.detail,.ref,._class] | @tsv' 2>/dev/null \ - | while IFS=$'\t' read -r SEV TITLE LOC DET REF CLASS; do - case "$SEV" in critical|high) C="$R";; medium) C="$Y";; *) C="$D";; esac + map(select(._class!="suppressed")) + | sort_by([ (if ._blocking=="yes" then 0 else 1 end), ($ord[.severity] // 9) ])[] + | [.severity,.title,.location,.detail,.ref,._class,._blocking,._provenance] | @tsv' 2>/dev/null \ + | while IFS=$'\t' read -r SEV TITLE LOC DET REF CLASS BLK PROV; do SEV_UC="$(printf '%s' "$SEV" | tr '[:lower:]' '[:upper:]')" REFSUFFIX=""; [ -n "$REF" ] && REFSUFFIX=" ${D}(${REF})${O}" - echo " $(class_prefix "$CLASS")${C}● ${SEV_UC}${O} ${B}${TITLE}${O}${REFSUFFIX}" - echo " where : $LOC" - [ -n "$DET" ] && echo " detail: $DET" - [ "$CLASS" = "mutated" ] && echo " ${R}note : content changed since this was pinned reviewed-benign — re-review before re-pinning.${O}" + PTAG=""; [ "$PROV" = "first-party" ] && PTAG=" ${D}[1st-party]${O}" + if [ "$BLK" = "no" ]; then + echo " ${D}○ review ${SEV_UC} ${TITLE}${REFSUFFIX} [non-blocking]${O}" + echo " ${D}where : $LOC${O}" + else + case "$SEV" in critical|high) C="$R";; medium) C="$Y";; *) C="$D";; esac + echo " $(class_prefix "$CLASS")${C}● ${SEV_UC}${O} ${B}${TITLE}${O}${REFSUFFIX}${PTAG}" + echo " where : $LOC" + [ -n "$DET" ] && echo " detail: $DET" + [ "$CLASS" = "mutated" ] && echo " ${R}note : content changed since this was pinned reviewed-benign — re-review before re-pinning.${O}" + fi done done } @@ -187,21 +207,24 @@ esac # --- additional lenses ------------------------------------------------------ render_lenses -# --- suppression summary ---------------------------------------------------- -if [ $((SUP+MUT+EXP)) -gt 0 ]; then +# --- suppression / policy summary ------------------------------------------- +if [ $((SUP+MUT+EXP+REV)) -gt 0 ]; then echo echo "${D}baseline: ${SUP} suppressed, ${MUT} mutated, ${EXP} expired — honey.baseline.json. Review: ./honey-baseline.sh status${O}" + [ "$REV" -gt 0 ] && echo "${D}policy: ${REV} finding(s) held below the severity floor (review tier, non-blocking). Floor: ${HONEY_VERDICT_FLOOR:-none} / trusted ${HONEY_VERDICT_FLOOR_TRUSTED:-${HONEY_VERDICT_FLOOR:-none}}.${O}" [ "$MUT" -gt 0 ] && echo "${R}⚠ ${MUT} MUTATED: a file pinned as reviewed-benign has CHANGED. Treat as a possible rug pull and re-review.${O}" fi # --- overall verdict -------------------------------------------------------- -# Suffix tallies so a suppressed run can never read as a bare all-clear. +# Suffix tallies so a suppressed/held run can never read as a bare all-clear. SUFFIX="" -if [ $((SUP+MUT+EXP)) -gt 0 ]; then - SUFFIX=" (${SUP} suppressed" - [ "$MUT" -gt 0 ] && SUFFIX="${SUFFIX}, ${MUT} mutated" - [ "$EXP" -gt 0 ] && SUFFIX="${SUFFIX}, ${EXP} expired" - SUFFIX="${SUFFIX})" +if [ $((SUP+MUT+EXP+REV)) -gt 0 ]; then + parts="" + [ "$SUP" -gt 0 ] && parts="${parts}, ${SUP} suppressed" + [ "$REV" -gt 0 ] && parts="${parts}, ${REV} review" + [ "$MUT" -gt 0 ] && parts="${parts}, ${MUT} mutated" + [ "$EXP" -gt 0 ] && parts="${parts}, ${EXP} expired" + SUFFIX=" (${parts#, })" fi echo diff --git a/routine-prompt.md b/routine-prompt.md index efe09f5..3d3e274 100644 --- a/routine-prompt.md +++ b/routine-prompt.md @@ -62,8 +62,14 @@ SUPPRESSION BASELINE — the OVERALL line may carry a tally, e.g. fold it into "all clear". A run with any `mutated` is never CLEAN. • `expired` = a pin passed its expiry and resurfaced; treat as a normal active finding due for re-review. - • `CLEAN (N suppressed)` is still all-clear — but say "all clear (N previously - reviewed findings suppressed)", not a bare "all clear". + • `review` = active findings held BELOW the severity floor (verdict policy) — + typically first-party / trusted-marketplace low/medium noise. They are + reported but non-blocking. Summarize them briefly ("N low-severity first-party + findings, non-blocking"); do not treat them as urgent. A run whose only + findings are `review` is CLEAN. + • `CLEAN (N suppressed)` / `CLEAN (N review)` is still all-clear — but name the + counts ("all clear; N previously reviewed / N below-floor findings"), not a + bare "all clear". Treat report.sh's output as the factual baseline; do not contradict it or invent findings beyond it. For extra detail read HONEY_DIR/latest/lens-*.json diff --git a/triage-guide.md b/triage-guide.md index 8b9e850..d81b3c7 100644 --- a/triage-guide.md +++ b/triage-guide.md @@ -21,13 +21,17 @@ exactly matches a known-compromised entry in a threat-intelligence catalog. agent skills); the OVERALL verdict is the worst across all of them. Surface every scanner's findings, not just bumblebee's. - The OVERALL line may carry a suppression tally, e.g. `CLEAN (12 suppressed)` - or `EXPOSED — … (12 suppressed, 2 mutated)`. `suppressed` = findings the user - pinned as reviewed-benign in `honey.baseline.json` (dropped from the verdict — - don't re-alarm). `mutated` (🔁 in the report) = a pinned file CHANGED since - review — treat as high signal (possible rug pull), surface it, and never call - such a run clean. Manage pins with `./honey-baseline.sh` (see - [`docs/BASELINE.plan.md`](docs/BASELINE.plan.md)). + The OVERALL line may carry a tally, e.g. `CLEAN (12 suppressed)` or + `EXPOSED — … (12 suppressed, 65 review, 2 mutated)`. `suppressed` = findings + the user pinned as reviewed-benign in `honey.baseline.json` (dropped from the + verdict — don't re-alarm). `review` = active findings held below the severity + floor (verdict policy — usually first-party low/medium noise): reported but + non-blocking, so a `review`-only run is clean; summarize, don't alarm. + `mutated` (🔁 in the report) = a pinned file CHANGED since review — treat as + high signal (possible rug pull), surface it, and never call such a run clean. + Manage pins with `./honey-baseline.sh`; tune the floor with + `HONEY_VERDICT_FLOOR_TRUSTED` (see [`docs/BASELINE.plan.md`](docs/BASELINE.plan.md) + and [`docs/VERDICT.plan.md`](docs/VERDICT.plan.md)). 2. For detail beyond the report, read the run dir: - `manifest.json` — bumblebee verdict, counts, metadata. diff --git a/win/daily-cycle.ps1 b/win/daily-cycle.ps1 index 2bf5835..91b1a8a 100644 --- a/win/daily-cycle.ps1 +++ b/win/daily-cycle.ps1 @@ -82,5 +82,5 @@ if (Test-Path $lensDir) { $eff = Get-HoneyEffectiveOverall -RunDir $runDir $overall = Resolve-WorseStatus $eff.status $crash -Log "=== cycle end (bumblebee=$bbStatus overall=$overall suppressed=$($eff.suppressed) mutated=$($eff.mutated) expired=$($eff.expired)) ===" +Log "=== cycle end (bumblebee=$bbStatus overall=$overall suppressed=$($eff.suppressed) mutated=$($eff.mutated) expired=$($eff.expired) review=$($eff.review)) ===" if ($overall -eq 'clean') { exit 0 } else { exit 1 } diff --git a/win/lib/Baseline.psm1 b/win/lib/Baseline.psm1 index 48dbecc..305a26b 100644 --- a/win/lib/Baseline.psm1 +++ b/win/lib/Baseline.psm1 @@ -22,6 +22,7 @@ $ErrorActionPreference = 'Stop' # Import Honey WITHOUT -Force: a forced re-import here would evict Honey's # commands from a caller that already imported it (report.ps1/daily-cycle.ps1). Import-Module (Join-Path $PSScriptRoot 'Honey.psm1') +Import-Module (Join-Path $PSScriptRoot 'Verdict.psm1') # --- config ------------------------------------------------------------------ @@ -129,6 +130,14 @@ function Get-HoneyClassified { $f | Add-Member -NotePropertyName '_loc' -NotePropertyValue $loc -Force $f | Add-Member -NotePropertyName '_index' -NotePropertyValue $idx -Force + # Verdict policy: provenance (from location) + blocking (severity vs floor). + # bumblebee (known-compromised catalog) always blocks; the floor is a + # lens-noise dial, not a way to mute a known-bad package. + $sev = if ($f.PSObject.Properties.Name -contains 'severity' -and $f.severity) { [string]$f.severity } else { 'unknown' } + $prov = Get-HoneyProvenance $loc + $blk = if ($Scanner -eq 'bumblebee') { $true } else { (Test-HoneyBlocking $sev $prov) } + $f | Add-Member -NotePropertyName '_provenance' -NotePropertyValue $prov -Force + $entry = $entries | Where-Object { $_.scanner -eq $Scanner -and $_.rule -eq $rule -and $_.location -eq $loc -and (([int](Get-HoneyEntryIndex $_)) -eq $idx) @@ -137,6 +146,7 @@ function Get-HoneyClassified { if (-not $entry) { $f | Add-Member -NotePropertyName '_class' -NotePropertyValue 'active' -Force $f | Add-Member -NotePropertyName '_reason' -NotePropertyValue '' -Force + $f | Add-Member -NotePropertyName '_blocking' -NotePropertyValue $(if ($blk) { 'yes' } else { 'no' }) -Force [void]$out.Add($f); continue } @@ -155,8 +165,11 @@ function Get-HoneyClassified { if ($curhash -and $curhash -eq $cmp) { if ($expires -ne 'never' -and $expires -lt $today) { $class = 'expired' } else { $class = 'suppressed' } } + # A MUTATED pin is the rug-pull tripwire — it always blocks, floor be damned. + if ($class -eq 'mutated') { $blk = $true } $f | Add-Member -NotePropertyName '_class' -NotePropertyValue $class -Force $f | Add-Member -NotePropertyName '_reason' -NotePropertyValue $reason -Force + $f | Add-Member -NotePropertyName '_blocking' -NotePropertyValue $(if ($blk) { 'yes' } else { 'no' }) -Force [void]$out.Add($f) } return $out.ToArray() @@ -233,7 +246,7 @@ function Get-HoneyClassifiedRun { # through untouched. function Get-HoneyEffectiveOverall { param([string]$RunDir) - $sup = 0; $mut = 0; $exp = 0; $overall = 'clean' + $sup = 0; $mut = 0; $exp = 0; $rev = 0; $overall = 'clean' $manifestPath = Join-Path $RunDir 'manifest.json' if (Test-Path -LiteralPath $manifestPath) { @@ -246,7 +259,8 @@ function Get-HoneyEffectiveOverall { $sup += @($cls | Where-Object { $_._class -eq 'suppressed' }).Count $mut += @($cls | Where-Object { $_._class -eq 'mutated' }).Count $exp += @($cls | Where-Object { $_._class -eq 'expired' }).Count - if (@($cls | Where-Object { $_._class -ne 'suppressed' }).Count -gt 0) { $overall = Resolve-WorseStatus $overall 'exposed' } + $rev += @($cls | Where-Object { $_._class -ne 'suppressed' -and $_._blocking -eq 'no' }).Count + if (@($cls | Where-Object { $_._class -ne 'suppressed' -and $_._blocking -eq 'yes' }).Count -gt 0) { $overall = Resolve-WorseStatus $overall 'exposed' } } else { $overall = Resolve-WorseStatus $overall $bst } @@ -261,13 +275,14 @@ function Get-HoneyEffectiveOverall { $sup += @($cls | Where-Object { $_._class -eq 'suppressed' }).Count $mut += @($cls | Where-Object { $_._class -eq 'mutated' }).Count $exp += @($cls | Where-Object { $_._class -eq 'expired' }).Count - if (@($cls | Where-Object { $_._class -ne 'suppressed' }).Count -gt 0) { $overall = Resolve-WorseStatus $overall 'exposed' } + $rev += @($cls | Where-Object { $_._class -ne 'suppressed' -and $_._blocking -eq 'no' }).Count + if (@($cls | Where-Object { $_._class -ne 'suppressed' -and $_._blocking -eq 'yes' }).Count -gt 0) { $overall = Resolve-WorseStatus $overall 'exposed' } } else { $overall = Resolve-WorseStatus $overall $lst } } - return @{ status = $overall; suppressed = $sup; mutated = $mut; expired = $exp } + return @{ status = $overall; suppressed = $sup; mutated = $mut; expired = $exp; review = $rev } } Export-ModuleMember -Function Get-HoneyBaselineFile, Get-HoneyBaselineEntry, diff --git a/win/lib/Verdict.psm1 b/win/lib/Verdict.psm1 new file mode 100644 index 0000000..f878f00 --- /dev/null +++ b/win/lib/Verdict.psm1 @@ -0,0 +1,58 @@ +<# +.SYNOPSIS + honey (Windows): verdict policy (provenance tier + severity floor) — PowerShell + mirror of lib/verdict.sh. Imported by Baseline.psm1 so every classified finding + carries _provenance and _blocking. See docs/VERDICT.plan.md. + +.NOTES + Provenance: a finding whose location matches HONEY_TRUSTED_PATTERNS is + "first-party"; everything else "third-party". Severity floor: a finding + escalates OVERALL only if its severity is at/above the floor for its + provenance. Below-floor findings are still reported (a "review" tier) but do + not flip the verdict. SAFE DEFAULT: floors default to `none` (everything + blocks, as before). +#> + +Set-StrictMode -Version Latest +$ErrorActionPreference = 'Stop' +Import-Module (Join-Path $PSScriptRoot 'Honey.psm1') + +# ERE matched against a location to mark it first-party. Unset -> the Anthropic +# marketplace default; empty -> nothing is first-party. (mirrors ${VAR-default}) +function Get-HoneyTrustedPattern { + Get-HoneySettingRaw 'HONEY_TRUSTED_PATTERNS' 'claude-plugins-official' +} + +function Get-HoneyProvenance { + param([string]$Location) + $pat = Get-HoneyTrustedPattern + if ($pat -and ($Location -match $pat)) { return 'first-party' } + return 'third-party' +} + +# severity -> numeric rank (higher = worse); none = 0 so a `none` floor blocks all. +function Get-HoneySeverityRank { + param([string]$Severity) + switch ($Severity) { + 'critical' { 5 } 'high' { 4 } 'medium' { 3 } 'low' { 2 } 'unknown' { 1 } 'none' { 0 } default { 1 } + } +} + +# provenance -> the floor severity that applies to it. +function Get-HoneyFloorFor { + param([string]$Provenance) + $base = Get-HoneySetting 'HONEY_VERDICT_FLOOR' 'none' + if ($Provenance -eq 'first-party') { return (Get-HoneySetting 'HONEY_VERDICT_FLOOR_TRUSTED' $base) } + return $base +} + +# severity, provenance -> $true if this finding escalates OVERALL. +function Test-HoneyBlocking { + param([string]$Severity, [string]$Provenance) + $floor = Get-HoneyFloorFor $Provenance + if ($floor -eq 'none') { return $true } + return ((Get-HoneySeverityRank $Severity) -ge (Get-HoneySeverityRank $floor)) +} + +Export-ModuleMember -Function Get-HoneyTrustedPattern, Get-HoneyProvenance, + Get-HoneySeverityRank, Get-HoneyFloorFor, Test-HoneyBlocking diff --git a/win/report.ps1 b/win/report.ps1 index 3767928..821bb0c 100644 --- a/win/report.ps1 +++ b/win/report.ps1 @@ -52,8 +52,9 @@ if (Test-Path $sumPath) { $L = New-Object System.Collections.ArrayList function Emit { param($s) [void]$L.Add($s) } -# Suppression tallies, accumulated across bumblebee + every lens as we render. -$script:sup = 0; $script:mut = 0; $script:exp = 0 +# Suppression + verdict-policy tallies, accumulated across bumblebee + every +# lens. rev = active findings held below the severity floor (review tier). +$script:sup = 0; $script:mut = 0; $script:exp = 0; $script:rev = 0 # A class marker printed before a finding line (mutated/expired resurface loudly). function Get-ClassMarker { param($c) switch ($c) { 'mutated' { '[MUTATED] ' } 'expired' { '[EXPIRED-PIN] ' } default { '' } } } @@ -119,45 +120,68 @@ Get-ChildItem -Path $RunDir -Filter 'lens-*.json' -ErrorAction SilentlyContinue 'exposed' { $cls = Get-HoneyClassifiedLens -Scanner $name -LensObj $lj $s = @($cls | Where-Object { $_._class -eq 'suppressed' }).Count - $active = @($cls | Where-Object { $_._class -ne 'suppressed' }) + $blocking = @($cls | Where-Object { $_._class -ne 'suppressed' -and $_._blocking -eq 'yes' }) + $review = @($cls | Where-Object { $_._class -ne 'suppressed' -and $_._blocking -eq 'no' }) $script:sup += $s $script:mut += @($cls | Where-Object { $_._class -eq 'mutated' }).Count $script:exp += @($cls | Where-Object { $_._class -eq 'expired' }).Count - if ($active.Count -eq 0) { + $script:rev += $review.Count + if (($blocking.Count + $review.Count) -eq 0) { Emit "[OK] lens ${name}: clean - all $tot finding(s) suppressed by baseline. ($note)" } else { - $overall = Resolve-WorseStatus $overall 'exposed' - $bySev = @($active) | Group-Object severity | ForEach-Object { "$($_.Count) $($_.Name)" } - $supStr = if ($s -gt 0) { " (+$s suppressed)" } else { "" } - Emit "[!!] lens ${name}: $($active.Count) finding(s) [$(( $bySev ) -join ', ')]$supStr ($note)" - $rank = @{ critical=0; high=1; medium=2; low=3; unknown=4 } - @($active) | Sort-Object @{ Expression = { $r = $rank[[string](Get-Prop $_ 'severity' 'unknown')]; if ($null -eq $r) { 9 } else { $r } } } | ForEach-Object { - Emit (" - {0}{1} {2}" -f (Get-ClassMarker $_._class), (Get-Prop $_ 'severity' '?').ToUpper(), (Get-Prop $_ 'title' '?')) - $loc = Get-Prop $_ 'location' '' - if ($loc) { Emit " where : $loc" } - $det = Get-Prop $_ 'detail' '' - if ($det) { Emit " detail: $det" } - if ($_._class -eq 'mutated') { Emit " note : content changed since this was pinned reviewed-benign - re-review before re-pinning." } + $extras = "" + if ($s -gt 0) { $extras += " (+$s suppressed)" } + if ($blocking.Count -gt 0 -and $review.Count -gt 0) { $extras += " (+$($review.Count) review)" } + if ($blocking.Count -gt 0) { + $overall = Resolve-WorseStatus $overall 'exposed' + $bySev = @($blocking) | Group-Object severity | ForEach-Object { "$($_.Count) $($_.Name)" } + Emit "[!!] lens ${name}: $($blocking.Count) finding(s) [$(( $bySev ) -join ', ')]$extras ($note)" + } else { + Emit "[~] lens ${name}: $($review.Count) review finding(s) - non-blocking (first-party or below the severity floor)$extras ($note)" } + $rank = @{ critical=0; high=1; medium=2; low=3; unknown=4 } + @($cls | Where-Object { $_._class -ne 'suppressed' }) | + Sort-Object @{ Expression = { if ($_._blocking -eq 'yes') { 0 } else { 1 } } }, @{ Expression = { $r = $rank[[string](Get-Prop $_ 'severity' 'unknown')]; if ($null -eq $r) { 9 } else { $r } } } | + ForEach-Object { + $ptag = if ($_._provenance -eq 'first-party') { ' [1st-party]' } else { '' } + if ($_._blocking -eq 'no') { + Emit (" o review {0} {1} [non-blocking]" -f (Get-Prop $_ 'severity' '?').ToUpper(), (Get-Prop $_ 'title' '?')) + Emit " where : $(Get-Prop $_ 'location' '')" + } else { + Emit (" - {0}{1} {2}{3}" -f (Get-ClassMarker $_._class), (Get-Prop $_ 'severity' '?').ToUpper(), (Get-Prop $_ 'title' '?'), $ptag) + $loc = Get-Prop $_ 'location' '' + if ($loc) { Emit " where : $loc" } + $det = Get-Prop $_ 'detail' '' + if ($det) { Emit " detail: $det" } + if ($_._class -eq 'mutated') { Emit " note : content changed since this was pinned reviewed-benign - re-review before re-pinning." } + } + } } } } } -# --- suppression summary ---------------------------------------------------- -if (($script:sup + $script:mut + $script:exp) -gt 0) { +# --- suppression / policy summary ------------------------------------------- +if (($script:sup + $script:mut + $script:exp + $script:rev) -gt 0) { Emit "" Emit "baseline: $($script:sup) suppressed, $($script:mut) mutated, $($script:exp) expired - honey.baseline.json. Review: honey-baseline.ps1 status" + if ($script:rev -gt 0) { + $ft = Get-HoneySetting 'HONEY_VERDICT_FLOOR' 'none' + $ftt = Get-HoneySetting 'HONEY_VERDICT_FLOOR_TRUSTED' $ft + Emit "policy: $($script:rev) finding(s) held below the severity floor (review tier, non-blocking). Floor: $ft / trusted $ftt." + } if ($script:mut -gt 0) { Emit "[!!] $($script:mut) MUTATED: a file pinned as reviewed-benign has CHANGED. Treat as a possible rug pull and re-review." } } # --- overall verdict -------------------------------------------------------- $suffix = "" -if (($script:sup + $script:mut + $script:exp) -gt 0) { - $suffix = " ($($script:sup) suppressed" - if ($script:mut -gt 0) { $suffix += ", $($script:mut) mutated" } - if ($script:exp -gt 0) { $suffix += ", $($script:exp) expired" } - $suffix += ")" +if (($script:sup + $script:mut + $script:exp + $script:rev) -gt 0) { + $parts = @() + if ($script:sup -gt 0) { $parts += "$($script:sup) suppressed" } + if ($script:rev -gt 0) { $parts += "$($script:rev) review" } + if ($script:mut -gt 0) { $parts += "$($script:mut) mutated" } + if ($script:exp -gt 0) { $parts += "$($script:exp) expired" } + $suffix = " (" + ($parts -join ', ') + ")" } Emit ""