Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -217,6 +217,32 @@ first place — sleeper/trigger-activated code, instructions fetched from a remo
URL at runtime, or content hidden in bundled images. The baseline can only
suppress what a lens surfaced; widening the scanned surface is separate work.

## Verdict policy (provenance + severity floor)

Where the baseline pins *specific* findings, the verdict policy is the *broad*
dial for the daily marketplace noise. Two settings, applied after suppression
([`docs/VERDICT.plan.md`](docs/VERDICT.plan.md)):

- **Provenance** — a finding whose location matches `HONEY_TRUSTED_PATTERNS`
(default `claude-plugins-official`) is tagged **first-party**; the report shows
a `[1st-party]` marker.
- **Severity floor** — a finding escalates `OVERALL` only at/above the floor for
its provenance (`HONEY_VERDICT_FLOOR` for third-party, `HONEY_VERDICT_FLOOR_TRUSTED`
for first-party). Below-floor findings move to a **review tier**: still
printed and counted (`OVERALL: … (65 review)`), but non-blocking.

```bash
# Demote first-party low/medium noise to the review tier (high/critical still block):
export HONEY_VERDICT_FLOOR_TRUSTED=high
# Or, for a marketplace you fully trust, demote all of its non-critical findings:
export HONEY_VERDICT_FLOOR_TRUSTED=critical
```

**Safe by default:** floors default to `none` — every finding blocks, exactly as
before. Provenance labeling is on (informational). **bumblebee always blocks**
(known-compromised catalog), and a **MUTATED** pin always blocks (rug-pull
tripwire) — the floor is a lens-noise dial, never a mute for genuine danger.

## Optional: daily Slack triage via a Claude Local routine

If you use [Claude Code](https://claude.com/claude-code) and have the Slack
Expand Down Expand Up @@ -348,6 +374,9 @@ one-off run. You can also hand-edit `honey.conf` — one `export VAR=…` per li
| `HONEY_OSV_NO_DEDUPE` | `0` | osv-scanner: `1` keeps one finding per path; default collapses identical `package@version`+advisory across paths into one (`+N more`) |
| `HONEY_SKILLSPECTOR_LLM` | `0` | skillspector: enable its own LLM stage (default static `--no-llm`) |
| `HONEY_BASELINE` | `<repo>/honey.baseline.json` | path to the suppression baseline (pin-and-diff); see [Suppression baseline](#suppression-baseline-pin-and-diff) |
| `HONEY_TRUSTED_PATTERNS` | `claude-plugins-official` | ERE; a finding whose location matches is **first-party** (else third-party). Empty ⇒ nothing trusted. See [Verdict policy](#verdict-policy-provenance--severity-floor) |
| `HONEY_VERDICT_FLOOR` | `none` | min severity (`none\|low\|medium\|high\|critical`) at which a **third-party** finding escalates OVERALL; below it → non-blocking "review" tier (still reported) |
| `HONEY_VERDICT_FLOOR_TRUSTED` | = `HONEY_VERDICT_FLOOR` | same, for **first-party** findings. Set `high` (or `critical`) to demote trusted-marketplace low/medium noise to review |

## Layout

Expand Down
104 changes: 104 additions & 0 deletions docs/VERDICT.plan.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
# Verdict policy — provenance tier + severity floor — design spec

Status: **implemented** (this branch). Companion to
[`BASELINE.plan.md`](BASELINE.plan.md). Where the baseline pins *specific*
reviewed findings, the verdict policy is the *broad* dial: it decides which
findings escalate `OVERALL` at all.

## 1. Problem

The suppression baseline is surgical — you pin individual reviewed findings. But
the daily noise from a trusted, first-party skill marketplace is *broad*: dozens
of low/medium false positives you don't want to pin one by one, yet don't want
flipping the verdict every day either. And even after the baseline, `OVERALL` is
still a flat worst-wins: a single low finding reads the same as a critical.

## 2. Two dials, applied after suppression

Both operate at the report/verdict layer, on the already-classified findings, so
`report.sh` and `daily-cycle.sh` (and their PowerShell mirrors) agree.

### Provenance
Each finding's location is matched against `HONEY_TRUSTED_PATTERNS` (an ERE,
default `claude-plugins-official` — Anthropic's official marketplace, the usual
source of the doc-scan false positives). A match → **first-party**; otherwise
**third-party**. Provenance is shown in the report (`[1st-party]` tag) and feeds
the floor. Set `HONEY_TRUSTED_PATTERNS=` (empty) to treat everything as
third-party.

### Severity floor
A finding **blocks** (escalates `OVERALL`) only if its severity is at or above
the floor **for its provenance**:

- `HONEY_VERDICT_FLOOR` (default `none`) — floor for third-party/unknown.
- `HONEY_VERDICT_FLOOR_TRUSTED` (default = `HONEY_VERDICT_FLOOR`) — floor for
first-party.

Values: `none | low | medium | high | critical`. A finding **below** its floor
is **not** dropped — it moves to a **review tier**: still printed (dimmed, marked
`[non-blocking]`), still counted on the verdict line, but it does not flip
`OVERALL`.

## 3. Classification

Every classified finding now also carries `_provenance` and `_blocking`. Three
outcomes for an active (non-suppressed) finding:

| | escalates OVERALL? | shown? |
|---|---|---|
| **blocking** (`_blocking=yes`) | yes | yes, prominent |
| **review** (`_blocking=no`) | no | yes, dimmed `[non-blocking]` |
| suppressed (from the baseline) | no | summarized only |

## 4. Non-negotiable overrides

The floor is a *lens-noise* dial, never a way to mute genuinely dangerous
findings. Regardless of floor:

- **bumblebee always blocks** — it is the known-compromised-package catalog
scanner; a match is never "review".
- **MUTATED pins always block** — the rug-pull tripwire fires no matter the floor.
- `incomplete` / `scan_error` are untouched (the policy only reclassifies
`exposed` findings, never partial coverage or a crash).

## 5. Safe default

Floors default to `none`, i.e. **every finding blocks, exactly as before** — a
security tool must not silently hide findings out of the box. Provenance
*labeling* is on by default (informational, safe). To actually quiet first-party
low/medium noise, opt in, e.g. in `honey.conf`:

```
export HONEY_VERDICT_FLOOR_TRUSTED=high # first-party low/medium -> review; high/critical still block
# or, to demote ALL first-party noise (it's a doc-scan marketplace you trust):
export HONEY_VERDICT_FLOOR_TRUSTED=critical
```

## 6. Verdict line

The tallies are always on the line so a held run can't read as a bare all-clear:

- `OVERALL: CLEAN (123 review)` — nothing blocking; 123 findings held below the
floor (still listed).
- `OVERALL: EXPOSED (12 suppressed, 65 review)` — real blocking findings drive
it; 65 first-party lows/mediums demoted, 12 baseline-pinned.

`daily-cycle` exits `0` only when nothing blocks (review-only ⇒ clean).

## 7. How it composes with the baseline

- **Baseline** = "I reviewed *this exact finding* and it's benign" (content-hashed, per-finding).
- **Verdict policy** = "findings from *this source* below *this severity* are background noise, not alarms" (broad, per-provenance).

Together: the marketplace's structural false positives drop to the review tier
via one `HONEY_VERDICT_FLOOR_TRUSTED` setting, and the handful you want to fully
vanish get pinned. A real HIGH in a first-party skill still blocks; a MUTATED pin
still screams.

## 8. Files

- **bash:** `lib/verdict.sh` (provenance + floor helpers), sourced by
`lib/baseline.sh`; `report.sh` renders blocking vs review; `daily-cycle.sh`
inherits the review count via `baseline_effective_overall`.
- **win:** `win/lib/Verdict.psm1`, imported by `Baseline.psm1`; `report.ps1`,
`daily-cycle.ps1` updated.
48 changes: 33 additions & 15 deletions lib/baseline.sh
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,11 @@
# Portability: pure bash 3.2 + jq. No associative arrays, no ${x^^}. Occurrence
# indices and path tildify are done in jq (deterministic), not in shell.

# Verdict policy (provenance tier + severity floor). Sourced here so every
# classified finding also carries _provenance and _blocking.
# shellcheck source=lib/verdict.sh
. "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/verdict.sh"

# --- config -----------------------------------------------------------------

# Absolute path to the baseline file. Override with HONEY_BASELINE.
Expand Down Expand Up @@ -123,17 +128,25 @@ _honey_classify() {
) | .out[]' 2>/dev/null \
| while IFS= read -r finding; do
[ -n "$finding" ] || continue
local rule loc idx locfile absfile curhash entry expires csource cmp class reason
local rule loc idx sev prov blk locfile absfile curhash entry expires csource cmp class reason
rule="$(printf '%s' "$finding" | jq -r '.rule // ""')"
loc="$(printf '%s' "$finding" | jq -r '._loc // ""')"
idx="$(printf '%s' "$finding" | jq -r '._index // 0')"
# Verdict policy: provenance (from location) + blocking (severity vs floor).
sev="$(printf '%s' "$finding" | jq -r '.severity // "unknown"')"
prov="$(honey_provenance "$loc")"
blk="$(honey_is_blocking "$sev" "$prov")"
# bumblebee is the known-compromised catalog scanner — its matches always
# block; the severity floor is a lens-noise dial, not a way to mute a
# known-bad package.
[ "$scanner" = "bumblebee" ] && blk="yes"

entry="$(printf '%s' "$entries" | jq -c --arg s "$scanner" --arg r "$rule" \
--arg l "$loc" --argjson i "$idx" \
'map(select(.scanner==$s and .rule==$r and .location==$l and ((.index // 0)==$i))) | .[0] // empty' 2>/dev/null)"

if [ -z "$entry" ]; then
printf '%s' "$finding" | jq -c '. + {_class:"active",_reason:""}'
printf '%s' "$finding" | jq -c --arg pv "$prov" --arg bk "$blk" '. + {_class:"active",_reason:"",_provenance:$pv,_blocking:$bk}'
continue
fi

Expand Down Expand Up @@ -162,7 +175,10 @@ _honey_classify() {
# Content changed since review (or file gone): the tripwire. Resurface.
class="mutated"
fi
printf '%s' "$finding" | jq -c --arg c "$class" --arg r "$reason" '. + {_class:$c,_reason:$r}'
# A MUTATED pin is the rug-pull tripwire — it always blocks, floor be damned.
[ "$class" = "mutated" ] && blk="yes"
printf '%s' "$finding" | jq -c --arg c "$class" --arg r "$reason" --arg pv "$prov" --arg bk "$blk" \
'. + {_class:$c,_reason:$r,_provenance:$pv,_blocking:$bk}'
done
}

Expand Down Expand Up @@ -211,21 +227,24 @@ honey_classify_run() {
baseline_effective_overall() {
local run_dir="$1"
local manifest="$run_dir/manifest.json"
local sup=0 mut=0 exp=0 overall="clean"
local sup=0 mut=0 exp=0 rev=0 overall="clean"
_rank() { case "$1" in scan_error) echo 4;; exposed) echo 3;; incomplete) echo 2;; clean|skipped) echo 1;; *) echo 0;; esac; }
_worse() { [ "$(_rank "$2")" -gt "$(_rank "$1")" ] && printf '%s' "$2" || printf '%s' "$1"; }
_cnt() { printf '%s\n' "$1" | jq -r "select(._class==\"$2\")|._class" 2>/dev/null | grep -c . ; }
# Active + blocking findings escalate OVERALL; active + non-blocking are the
# "review" tier (reported, not blocking). suppressed never counts either way.
_blk() { printf '%s\n' "$1" | jq -r 'select(._class!="suppressed" and ._blocking=="yes")|1' 2>/dev/null | grep -c . ; }
_rev() { printf '%s\n' "$1" | jq -r 'select(._class!="suppressed" and ._blocking=="no")|1' 2>/dev/null | grep -c . ; }

# bumblebee
if [ -f "$manifest" ]; then
local bst; bst="$(jq -r '.status // "unknown"' "$manifest" 2>/dev/null)"
if [ "$bst" = "exposed" ] && [ -f "$run_dir/findings.ndjson" ]; then
local cls s m e a
local cls
cls="$(honey_classify_bumblebee "$run_dir/findings.ndjson")"
s="$(_cnt "$cls" suppressed)"; m="$(_cnt "$cls" mutated)"; e="$(_cnt "$cls" expired)"
a="$(printf '%s\n' "$cls" | jq -r 'select(._class!="suppressed")|._class' 2>/dev/null | grep -c .)"
sup=$((sup+s)); mut=$((mut+m)); exp=$((exp+e))
[ "$a" -gt 0 ] && overall="$(_worse "$overall" exposed)"
sup=$((sup+$(_cnt "$cls" suppressed))); mut=$((mut+$(_cnt "$cls" mutated))); exp=$((exp+$(_cnt "$cls" expired)))
rev=$((rev+$(_rev "$cls")))
[ "$(_blk "$cls")" -gt 0 ] && overall="$(_worse "$overall" exposed)"
else
overall="$(_worse "$overall" "$bst")"
fi
Expand All @@ -239,16 +258,15 @@ baseline_effective_overall() {
name="$(jq -r '.lens // ""' "$lj" 2>/dev/null)"
lst="$(jq -r '.status // "unknown"' "$lj" 2>/dev/null)"
if [ "$lst" = "exposed" ]; then
local cls s m e a
local cls
cls="$(honey_classify_lens "$name" "$lj")"
s="$(_cnt "$cls" suppressed)"; m="$(_cnt "$cls" mutated)"; e="$(_cnt "$cls" expired)"
a="$(printf '%s\n' "$cls" | jq -r 'select(._class!="suppressed")|._class' 2>/dev/null | grep -c .)"
sup=$((sup+s)); mut=$((mut+m)); exp=$((exp+e))
[ "$a" -gt 0 ] && overall="$(_worse "$overall" exposed)"
sup=$((sup+$(_cnt "$cls" suppressed))); mut=$((mut+$(_cnt "$cls" mutated))); exp=$((exp+$(_cnt "$cls" expired)))
rev=$((rev+$(_rev "$cls")))
[ "$(_blk "$cls")" -gt 0 ] && overall="$(_worse "$overall" exposed)"
else
overall="$(_worse "$overall" "$lst")"
fi
done

printf '%s suppressed=%s mutated=%s expired=%s' "$overall" "$sup" "$mut" "$exp"
printf '%s suppressed=%s mutated=%s expired=%s review=%s' "$overall" "$sup" "$mut" "$exp" "$rev"
}
59 changes: 59 additions & 0 deletions lib/verdict.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
#!/usr/bin/env bash
# lib/verdict.sh — honey's verdict policy: provenance tier + severity floor.
#
# Sourced by lib/baseline.sh (so every classified finding carries _provenance and
# _blocking) and thus by report.sh / daily-cycle.sh / honey-baseline.sh. See
# docs/VERDICT.plan.md.
#
# Two dials, both applied at the report/verdict layer AFTER suppression:
# • Provenance: a finding whose location matches HONEY_TRUSTED_PATTERNS is
# "first-party" (e.g. Anthropic's claude-plugins-official marketplace);
# everything else is "third-party".
# • Severity floor: a finding escalates OVERALL only if its severity is at or
# above the floor for its provenance. Below-floor findings are still
# REPORTED (a "review" tier) but do not flip the verdict — so a first-party
# doc-scanner false positive stops screaming without going invisible.
#
# SAFE DEFAULT: floors default to `none` (every finding blocks, exactly as
# before) — a security tool must not silently hide findings by default. Opt in
# with HONEY_VERDICT_FLOOR_TRUSTED=high to quiet first-party low/medium noise.

# ERE matched against a finding's location to mark it first-party. Empty ⇒
# nothing is first-party (everything third-party). Default trusts Anthropic's
# official marketplace, the usual source of the doc-scan false positives.
honey_trusted_patterns() { printf '%s' "${HONEY_TRUSTED_PATTERNS-claude-plugins-official}"; }

# location -> first-party | third-party
honey_provenance() {
local loc="$1" pat; pat="$(honey_trusted_patterns)"
if [ -n "$pat" ] && printf '%s' "$loc" | grep -qE "$pat"; then
printf 'first-party'
else
printf 'third-party'
fi
}

# severity -> numeric rank (higher = worse). none = 0 so a `none` floor blocks all.
honey_sev_rank() {
case "$1" in
critical) echo 5;; high) echo 4;; medium) echo 3;; low) echo 2;; unknown) echo 1;; none) echo 0;; *) echo 1;;
esac
}

# provenance -> the floor severity that applies to it.
# third-party/unknown → HONEY_VERDICT_FLOOR (default none)
# first-party → HONEY_VERDICT_FLOOR_TRUSTED (default = HONEY_VERDICT_FLOOR)
honey_floor_for() {
local base="${HONEY_VERDICT_FLOOR:-none}"
case "$1" in
first-party) printf '%s' "${HONEY_VERDICT_FLOOR_TRUSTED:-$base}" ;;
*) printf '%s' "$base" ;;
esac
}

# severity provenance -> yes | no (does this finding escalate OVERALL?)
honey_is_blocking() {
local sev="$1" prov="$2" floor; floor="$(honey_floor_for "$prov")"
[ "$floor" = "none" ] && { printf 'yes'; return; }
if [ "$(honey_sev_rank "$sev")" -ge "$(honey_sev_rank "$floor")" ]; then printf 'yes'; else printf 'no'; fi
}
Loading
Loading