Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,10 @@ honey.conf.ps1
# committed and reviewed in git — that auditability is an anti-gaming control
# (see docs/BASELINE.plan.md). Do not add it here.

# MCP-manifest lens state — auto-maintained hashes of MCP server definitions for
# rug-pull (drift) detection. Machine-specific; contains local config hashes.
.mcp-state.json

# Local secrets, if you ever add any (e.g. Slack webhook/token).
secrets.env
*.env
Expand Down
60 changes: 56 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,17 +107,34 @@ Every lens is **opt-in**: honey never installs a lens's tool for you, and an
uninstalled lens is fully inert. Install the tool, and the lens activates on
the next run. `./doctor.sh` shows which lenses are active.

**Offline vs. external.** The first six lenses run **offline** — nothing leaves
your machine (`bumblebee`, `smuggle`, `mcp`, and `ocr` are honey-native or use a
local tool; `osv-scanner`/`govulncheck` do static DB lookups). The last two are
**external and off by default**: `mcp-scan` calls a cloud API, and `garak`
probes a live model — so both stay inert unless you *explicitly* enable them
(`HONEY_ENABLE_MCP_SCAN=1` / `HONEY_ENABLE_GARAK=1`) and they never run in the
default cycle. honey's native `mcp` lens covers the MCP surface without the
cloud call.

| Lens | Tool | Covers | Install |
|---|---|---|---|
| `skillspector` | [NVIDIA SkillSpector](https://github.com/NVIDIA/skillspector) | AI agent skills (`SKILL.md` + scripts): prompt injection, data exfil, excessive agency, tool poisoning | per its README (Python 3.12+) |
| `smuggle` | honey-native | **Scanner-evasion** in agent-skill/instruction files: invisible-Unicode (tag chars), bidi overrides (Trojan Source), zero-width chars, and remote-include instructions — the tricks static pattern scanners miss | none (uses `perl`; PowerShell uses native .NET) |
| `mcp` | honey-native | **MCP servers** (`.mcp.json` + host configs — invisible to skillspector): inventories every server and **hash-diffs its definition across runs to catch rug pulls** (a server that changes after you approve it); flags fetch-and-exec launch commands | none (jq / native .NET) |
| `ocr` | honey-native | **Image-payload / SkillCamo**: OCRs images bundled in skills and flags hidden instructions — the multimodal blind spot text scanners can't see | [tesseract](https://github.com/tesseract-ocr/tesseract) (`brew install tesseract`) |
| `mcp-scan` | [Invariant / Snyk Agent Scan](https://github.com/snyk/agent-scan) | Hybrid rules+model analysis of MCP servers & skills (tool poisoning/shadowing, toxic flows) — **opt-in; cloud** | `pip install mcp-scan`; enable with `HONEY_ENABLE_MCP_SCAN=1` |
| `garak` | [NVIDIA garak](https://github.com/NVIDIA/garak) | Probes a **live model** for jailbreaks/injection/leakage — **opt-in; not offline** | `pip install garak`; enable with `HONEY_ENABLE_GARAK=1` + a target |
| `osv-scanner` | [Google/OSV osv-scanner](https://github.com/google/osv-scanner) | Known vulns in lockfiles/manifests across **all** ecosystems (npm, pypi, cargo, go, …) | `go install github.com/google/osv-scanner/cmd/osv-scanner@latest` |
| `govulncheck` | [Go vuln team govulncheck](https://golang.org/x/vuln) | Go vulns that your code **actually calls** (reachability-aware → far less noise) | `go install golang.org/x/vuln/cmd/govulncheck@latest` |

**What each answers.** bumblebee: *"do I have a known-compromised package?"*
(exact catalog match). skillspector: *"is this agent skill behaving
maliciously?"* (content analysis). osv-scanner: *"do my dependencies have known
CVEs?"* (broad). govulncheck: *"do I actually reach a vulnerable Go function?"*
(deep, low-noise). Together they cover reactive, proactive, breadth, and depth.
maliciously?"* (content analysis). smuggle: *"is something hidden from the
scanners — invisible Unicode, a bidi trick, a remote include?"* (evasion). mcp:
*"did an MCP server's definition change since I approved it?"* (rug-pull
diffing). osv-scanner: *"do my dependencies have known CVEs?"* (broad).
govulncheck: *"do I actually reach a vulnerable Go function?"* (deep, low-noise).
Together they cover reactive, proactive, breadth, depth, anti-evasion, and drift.

**Where the vuln lenses scan.** osv-scanner and govulncheck scan your *project*
directories, not all of `$HOME`. Set `HONEY_PROJECT_ROOTS` (colon-separated;
Expand Down Expand Up @@ -217,6 +234,32 @@ first place — sleeper/trigger-activated code, instructions fetched from a remo
URL at runtime, or content hidden in bundled images. The baseline can only
suppress what a lens surfaced; widening the scanned surface is separate work.

## Verdict policy (provenance + severity floor)

Where the baseline pins *specific* findings, the verdict policy is the *broad*
dial for the daily marketplace noise. Two settings, applied after suppression
([`docs/VERDICT.plan.md`](docs/VERDICT.plan.md)):

- **Provenance** — a finding whose location matches `HONEY_TRUSTED_PATTERNS`
(default `claude-plugins-official`) is tagged **first-party**; the report shows
a `[1st-party]` marker.
- **Severity floor** — a finding escalates `OVERALL` only at/above the floor for
its provenance (`HONEY_VERDICT_FLOOR` for third-party, `HONEY_VERDICT_FLOOR_TRUSTED`
for first-party). Below-floor findings move to a **review tier**: still
printed and counted (`OVERALL: … (65 review)`), but non-blocking.

```bash
# Demote first-party low/medium noise to the review tier (high/critical still block):
export HONEY_VERDICT_FLOOR_TRUSTED=high
# Or, for a marketplace you fully trust, demote all of its non-critical findings:
export HONEY_VERDICT_FLOOR_TRUSTED=critical
```

**Safe by default:** floors default to `none` — every finding blocks, exactly as
before. Provenance labeling is on (informational). **bumblebee always blocks**
(known-compromised catalog), and a **MUTATED** pin always blocks (rug-pull
tripwire) — the floor is a lens-noise dial, never a mute for genuine danger.

## Optional: daily Slack triage via a Claude Local routine

If you use [Claude Code](https://claude.com/claude-code) and have the Slack
Expand Down Expand Up @@ -340,14 +383,23 @@ one-off run. You can also hand-edit `honey.conf` — one `export VAR=…` per li
| `BUMBLEBEE_SCAN_ROOT` | `$HOME` | what bumblebee scans |
| `BUMBLEBEE_MAX_DURATION` | `30m` | bumblebee scan time cap |
| `HONEY_PROJECT_ROOTS` | `~/git:~/code:~/Developer:~/src` | where the vuln lenses (osv-scanner, govulncheck) look for projects |
| `HONEY_SKILL_ROOTS` | skill dirs under `~/.claude` | where the skillspector lens looks for agent skills |
| `HONEY_SKILL_ROOTS` | skill dirs under `~/.claude` | where the skillspector and smuggle lenses look for agent skills |
| `HONEY_SMUGGLE_EXCLUDE` | `node_modules`/`.git`/`vendor`/`.pnpm` | smuggle lens: skip files whose path matches (ERE) |
| `HONEY_MCP_CONFIGS` | Claude/Cursor/VS Code/Windsurf host configs | mcp lens: colon-separated MCP config files to inventory (plus every `.mcp.json` under the project roots) |
| `HONEY_MCP_STATE` | `<repo>/.mcp-state.json` | mcp lens: where the per-run manifest hashes are stored for rug-pull diffing (gitignored) |
| `HONEY_OCR_MAX` | `300` | ocr lens: cap on images OCR-scanned per run |
| `HONEY_ENABLE_MCP_SCAN` | `0` | set `1` to enable the opt-in `mcp-scan` lens (**cloud/phones home**; also needs `SNYK_TOKEN`/`OPENAI_API_KEY`) |
| `HONEY_ENABLE_GARAK` | `0` | set `1` to enable the opt-in `garak` lens (**probes a live model**; also needs `HONEY_GARAK_TARGET`) |
| `HONEY_UPDATE_LENSES` | `1` | `go install @latest` the Go lens binaries (osv-scanner, govulncheck) before scanning; `0` to skip |
| `HONEY_OSV_OFFLINE` | `0` | osv-scanner: use local vuln DBs instead of OSV.dev |
| `HONEY_OSV_INCLUDE_GO_STDLIB` | `0` | osv-scanner: keep Go stdlib advisories (default defers them to govulncheck) |
| `HONEY_OSV_EXCLUDE_PATHS` | `node_modules`/`vendor`/`.pnpm`/`testdata`/`fixtures`/`.claude/worktrees` (ERE) | osv-scanner: skip findings whose source path matches — keeps the scan on *declared* manifests, not installed/vendored copies. Set empty to scan everything. |
| `HONEY_OSV_NO_DEDUPE` | `0` | osv-scanner: `1` keeps one finding per path; default collapses identical `package@version`+advisory across paths into one (`+N more`) |
| `HONEY_SKILLSPECTOR_LLM` | `0` | skillspector: enable its own LLM stage (default static `--no-llm`) |
| `HONEY_BASELINE` | `<repo>/honey.baseline.json` | path to the suppression baseline (pin-and-diff); see [Suppression baseline](#suppression-baseline-pin-and-diff) |
| `HONEY_TRUSTED_PATTERNS` | `claude-plugins-official` | ERE; a finding whose location matches is **first-party** (else third-party). Empty ⇒ nothing trusted. See [Verdict policy](#verdict-policy-provenance--severity-floor) |
| `HONEY_VERDICT_FLOOR` | `none` | min severity (`none\|low\|medium\|high\|critical`) at which a **third-party** finding escalates OVERALL; below it → non-blocking "review" tier (still reported) |
| `HONEY_VERDICT_FLOOR_TRUSTED` | = `HONEY_VERDICT_FLOOR` | same, for **first-party** findings. Set `high` (or `critical`) to demote trusted-marketplace low/medium noise to review |

## Layout

Expand Down
104 changes: 104 additions & 0 deletions docs/VERDICT.plan.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
# Verdict policy — provenance tier + severity floor — design spec

Status: **implemented** (this branch). Companion to
[`BASELINE.plan.md`](BASELINE.plan.md). Where the baseline pins *specific*
reviewed findings, the verdict policy is the *broad* dial: it decides which
findings escalate `OVERALL` at all.

## 1. Problem

The suppression baseline is surgical — you pin individual reviewed findings. But
the daily noise from a trusted, first-party skill marketplace is *broad*: dozens
of low/medium false positives you don't want to pin one by one, yet don't want
flipping the verdict every day either. And even after the baseline, `OVERALL` is
still a flat worst-wins: a single low finding reads the same as a critical.

## 2. Two dials, applied after suppression

Both operate at the report/verdict layer, on the already-classified findings, so
`report.sh` and `daily-cycle.sh` (and their PowerShell mirrors) agree.

### Provenance
Each finding's location is matched against `HONEY_TRUSTED_PATTERNS` (an ERE,
default `claude-plugins-official` — Anthropic's official marketplace, the usual
source of the doc-scan false positives). A match → **first-party**; otherwise
**third-party**. Provenance is shown in the report (`[1st-party]` tag) and feeds
the floor. Set `HONEY_TRUSTED_PATTERNS=` (empty) to treat everything as
third-party.

### Severity floor
A finding **blocks** (escalates `OVERALL`) only if its severity is at or above
the floor **for its provenance**:

- `HONEY_VERDICT_FLOOR` (default `none`) — floor for third-party/unknown.
- `HONEY_VERDICT_FLOOR_TRUSTED` (default = `HONEY_VERDICT_FLOOR`) — floor for
first-party.

Values: `none | low | medium | high | critical`. A finding **below** its floor
is **not** dropped — it moves to a **review tier**: still printed (dimmed, marked
`[non-blocking]`), still counted on the verdict line, but it does not flip
`OVERALL`.

## 3. Classification

Every classified finding now also carries `_provenance` and `_blocking`. Three
outcomes for an active (non-suppressed) finding:

| | escalates OVERALL? | shown? |
|---|---|---|
| **blocking** (`_blocking=yes`) | yes | yes, prominent |
| **review** (`_blocking=no`) | no | yes, dimmed `[non-blocking]` |
| suppressed (from the baseline) | no | summarized only |

## 4. Non-negotiable overrides

The floor is a *lens-noise* dial, never a way to mute genuinely dangerous
findings. Regardless of floor:

- **bumblebee always blocks** — it is the known-compromised-package catalog
scanner; a match is never "review".
- **MUTATED pins always block** — the rug-pull tripwire fires no matter the floor.
- `incomplete` / `scan_error` are untouched (the policy only reclassifies
`exposed` findings, never partial coverage or a crash).

## 5. Safe default

Floors default to `none`, i.e. **every finding blocks, exactly as before** — a
security tool must not silently hide findings out of the box. Provenance
*labeling* is on by default (informational, safe). To actually quiet first-party
low/medium noise, opt in, e.g. in `honey.conf`:

```
export HONEY_VERDICT_FLOOR_TRUSTED=high # first-party low/medium -> review; high/critical still block
# or, to demote ALL first-party noise (it's a doc-scan marketplace you trust):
export HONEY_VERDICT_FLOOR_TRUSTED=critical
```

## 6. Verdict line

The tallies are always on the line so a held run can't read as a bare all-clear:

- `OVERALL: CLEAN (123 review)` — nothing blocking; 123 findings held below the
floor (still listed).
- `OVERALL: EXPOSED (12 suppressed, 65 review)` — real blocking findings drive
it; 65 first-party lows/mediums demoted, 12 baseline-pinned.

`daily-cycle` exits `0` only when nothing blocks (review-only ⇒ clean).

## 7. How it composes with the baseline

- **Baseline** = "I reviewed *this exact finding* and it's benign" (content-hashed, per-finding).
- **Verdict policy** = "findings from *this source* below *this severity* are background noise, not alarms" (broad, per-provenance).

Together: the marketplace's structural false positives drop to the review tier
via one `HONEY_VERDICT_FLOOR_TRUSTED` setting, and the handful you want to fully
vanish get pinned. A real HIGH in a first-party skill still blocks; a MUTATED pin
still screams.

## 8. Files

- **bash:** `lib/verdict.sh` (provenance + floor helpers), sourced by
`lib/baseline.sh`; `report.sh` renders blocking vs review; `daily-cycle.sh`
inherits the review count via `baseline_effective_overall`.
- **win:** `win/lib/Verdict.psm1`, imported by `Baseline.psm1`; `report.ps1`,
`daily-cycle.ps1` updated.
39 changes: 39 additions & 0 deletions doctor.sh
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,45 @@ if [ -d "$HONEY/lenses" ] && ls "$HONEY"/lenses/*.sh >/dev/null 2>&1; then
bad "lens $lname inactive — govulncheck not installed (optional)"
hint "Go reachability-aware vuln scanning; install: go install golang.org/x/vuln/cmd/govulncheck@latest"
fi ;;
smuggle)
# honey-native lens (no external tool); needs perl for Unicode scanning.
if command -v perl >/dev/null 2>&1; then
ok "lens $lname active (native; perl $(perl -e 'print substr($^V,1)' 2>/dev/null))"
else
bad "lens $lname inactive — perl not found (optional)"
hint "invisible-Unicode/bidi + remote-include detection; install perl to enable (ships on macOS + most Linux)"
fi ;;
mcp)
# honey-native lens (jq only) — MCP manifest inventory + rug-pull diffing.
ok "lens $lname active (native; MCP manifest hash-and-diff, jq only)" ;;
ocr)
# honey-native multimodal lens; needs tesseract for image OCR.
if command -v tesseract >/dev/null 2>&1; then
ok "lens $lname active (native; image-payload OCR via tesseract $(tesseract --version 2>&1 | head -1 | awk '{print $2}'))"
else
bad "lens $lname inactive — tesseract not installed (optional)"
hint "image-payload (SkillCamo) detection; macOS: brew install tesseract · Debian/Ubuntu: apt-get install tesseract-ocr"
fi ;;
mcp-scan)
# opt-in external lens (cloud/phones home).
if [ "${HONEY_ENABLE_MCP_SCAN:-0}" != "1" ]; then
ok "lens $lname opt-in (disabled) — set HONEY_ENABLE_MCP_SCAN=1 to enable (note: cloud/phones home)"
elif command -v mcp-scan >/dev/null 2>&1; then
ok "lens $lname active (enabled; mcp-scan present)"
else
bad "lens $lname enabled but mcp-scan not installed"
hint "Invariant/Snyk Agent Scan: pip install mcp-scan (https://github.com/snyk/agent-scan)"
fi ;;
garak)
# opt-in external lens (probes a live model).
if [ "${HONEY_ENABLE_GARAK:-0}" != "1" ]; then
ok "lens $lname opt-in (disabled) — set HONEY_ENABLE_GARAK=1 + HONEY_GARAK_TARGET (note: probes a live model)"
elif command -v garak >/dev/null 2>&1; then
ok "lens $lname active (enabled; garak present)"
else
bad "lens $lname enabled but garak not installed"
hint "NVIDIA garak: pip install garak (https://github.com/NVIDIA/garak)"
fi ;;
*)
if command -v "$lname" >/dev/null 2>&1; then ok "lens $lname active"; else bad "lens $lname inactive (optional tool '$lname' not installed)"; fi ;;
esac
Expand Down
Loading
Loading