Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,8 @@ cargo fmt --check
# Run CLI
cargo run -- scan tests/fixtures/mcp_servers/vuln_cmd_inject
cargo run -- scan . --ignore-tests --format html --output report.html
cargo run -- scan . --experimental-risk
cargo run -- scan . --format json --experimental-risk
cargo run -- scan . --write-baseline baseline.json
cargo run -- scan . --baseline baseline.json
cargo run -- discover --no-default-paths --root .
Expand Down
8 changes: 7 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -252,6 +252,7 @@ AgentShield runs all matching adapters in a repository instead of stopping at th
|---------|---------|
| `agentshield discover` | Read allowlisted local client configuration paths without execution, network access, or automatic scanning; use `--no-default-paths`, repeated `--root`, `--format console\|json`, and `--explain` to control consent and output. |
| `agentshield scan [path]` | Scan an agent extension directory and emit console, JSON, SARIF, or HTML output. |
| `agentshield scan [path] --experimental-risk` | Add an informational, versioned risk index to console or JSON output without changing policy or exit status. |
| `agentshield scan [path] --explain` | Print a console-only gate, coverage, confidence, grouped findings, next-actions, and limits summary. |
| `agentshield quickstart [path]` | Create first-run config, suggest CI setup, run the first scan, and explain the result. |
| `agentshield ci install` | Generate a GitHub Actions workflow for AgentShield. |
Expand All @@ -267,7 +268,7 @@ AgentShield runs all matching adapters in a repository instead of stopping at th
| `agentshield guard --stdin` | Evaluate one runtime event JSON document when built with the `runtime-guard` feature. |
| `agentshield guard --mcp-proxy [-- <server cmd...>]` | EXPERIMENTAL: evaluate line-delimited MCP JSON-RPC `tools/call` messages, block unsafe calls, and either emit forward markers or bridge stdio to a spawned downstream MCP server when built with the `runtime-guard` feature. |

Useful `scan` options include `--config`, `--format`, `--fail-on`, `--output`, `--ignore-tests`, `--explain`, `--baseline`, `--write-baseline`, and `--emit-egress-policy`.
Useful `scan` options include `--config`, `--format`, `--fail-on`, `--output`, `--ignore-tests`, `--explain`, `--experimental-risk`, `--baseline`, `--write-baseline`, and `--emit-egress-policy`.
Configured `[scan] include` and `[scan] exclude` filters scope source and
metadata-derived findings before detectors run.

Expand All @@ -284,6 +285,11 @@ agentshield ci install --baseline .agentshield-baseline.json
SARIF, or HTML output. Explain output includes the scan root, metadata root
when different, and hotspot summaries for concentrated blocking findings.

`--experimental-risk` is a separate console/JSON-only mode and cannot be
combined with `--explain`. Its versioned index is informational: it does not
change findings, PASS/FAIL, policy thresholds, baselines, suppressions, or the
process exit status. SARIF, HTML, and DSSE remain unchanged.

---

## Detection Rules
Expand Down
19 changes: 14 additions & 5 deletions docs/specs/risk-model-v1.md
Original file line number Diff line number Diff line change
@@ -1,14 +1,15 @@
# AgentShield risk model v1

Status: internal golden model
Status: experimental opt-in output

Model identifier: `agentshield-risk-v1`

This document freezes the E.1 model selected under
[`explainable-risk-score.md`](explainable-risk-score.md). The model is
crate-private and is not emitted by any default or opt-in output in E.1.
Findings remain the security facts; policy verdict and process exit status
remain the only enforcement contract.
emitted only when `scan --experimental-risk` is explicitly selected with
console or JSON output. Default output remains unchanged. Findings remain the
security facts; policy verdict and process exit status remain the only
enforcement contract.

## Inputs

Expand Down Expand Up @@ -74,4 +75,12 @@ contexts; E.1 does not expose a comparison UI.
Fingerprint deduplication removes exact duplicates but does not prove semantic
independence across different rules. Golden tests validate deterministic
mechanics, not breach probability or real-world calibration. E.2 output remains
blocked if representative correlated findings make the ranking misleading.
experimental and must be removed or revised if representative correlated
findings make the ranking misleading.

## Experimental output boundary

`--experimental-risk` supports console and JSON only. It is incompatible with
`--explain`; it does not modify SARIF, HTML or DSSE. At most 50 ordered
contributions are emitted, with an explicit omitted count. The flag never
changes findings, policy, verdict, baseline behavior or process exit status.
6 changes: 6 additions & 0 deletions src/bin/cli.rs
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,10 @@ enum Commands {
/// This is console-only so JSON, SARIF, and HTML output contracts stay stable.
#[arg(long)]
explain: bool,

/// Add an experimental informational risk index (console or JSON only)
#[arg(long)]
experimental_risk: bool,
},

/// First-run setup: create config, inspect coverage, and run an explained scan
Expand Down Expand Up @@ -323,6 +327,7 @@ fn main() {
write_baseline,
emit_egress_policy,
explain,
experimental_risk,
} => cmd_scan(ScanArgs {
path,
config,
Expand All @@ -334,6 +339,7 @@ fn main() {
write_baseline_path: write_baseline,
emit_egress_policy_path: emit_egress_policy,
explain,
experimental_risk,
}),
Commands::Quickstart {
path,
Expand Down
14 changes: 14 additions & 0 deletions src/bin/cli/scan.rs
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ pub(super) struct ScanArgs {
pub(super) write_baseline_path: Option<PathBuf>,
pub(super) emit_egress_policy_path: Option<PathBuf>,
pub(super) explain: bool,
pub(super) experimental_risk: bool,
}

pub(super) fn cmd_scan(args: ScanArgs) -> Result<i32, agentshield::error::ShieldError> {
Expand All @@ -33,6 +34,7 @@ pub(super) fn cmd_scan(args: ScanArgs) -> Result<i32, agentshield::error::Shield
write_baseline_path,
emit_egress_policy_path,
explain,
experimental_risk,
} = args;
let format = OutputFormat::from_str_lenient(&format_str).unwrap_or_else(|| {
eprintln!("Warning: unknown format '{}', using console", format_str);
Expand All @@ -44,6 +46,16 @@ pub(super) fn cmd_scan(args: ScanArgs) -> Result<i32, agentshield::error::Shield
"`scan --explain` is console-only; remove --format or use --format console".into(),
));
}
if explain && experimental_risk {
return Err(agentshield::error::ShieldError::Config(
"`scan --explain` and `--experimental-risk` are separate output modes".into(),
));
}
if experimental_risk && !matches!(format, OutputFormat::Console | OutputFormat::Json) {
return Err(agentshield::error::ShieldError::Config(
"`--experimental-risk` supports only console and JSON output".into(),
));
}

let config_path = config
.clone()
Expand Down Expand Up @@ -130,6 +142,8 @@ pub(super) fn cmd_scan(args: ScanArgs) -> Result<i32, agentshield::error::Shield
ignore_tests: effective_ignore_tests,
},
)
} else if experimental_risk {
agentshield::render_report_with_experimental_risk(&report, format)?
} else {
agentshield::render_report(&report, format)?
};
Expand Down
21 changes: 21 additions & 0 deletions src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -151,6 +151,27 @@ pub fn render_report(report: &ScanReport, format: OutputFormat) -> Result<String
)
}

/// Render a scan report with the opt-in experimental informational risk index.
///
/// This additive API supports console and JSON only. It does not change policy,
/// the report verdict, process exit status, findings, or default rendering.
pub fn render_report_with_experimental_risk(
report: &ScanReport,
format: OutputFormat,
) -> Result<String> {
if !matches!(format, OutputFormat::Console | OutputFormat::Json) {
return Err(error::ShieldError::Config(
"`--experimental-risk` supports only console and JSON output".to_owned(),
));
}

let base_report = render_report(report, format)?;
let coverage = risk::CoverageDescriptor::current();
let assessment = risk::assess(&report.findings, &report.scan_root, &coverage)
.map_err(|error| error::ShieldError::Internal(error.to_string()))?;
risk::render_experimental(&base_report, &assessment, format)
}

#[cfg(test)]
mod integration_tests {
use super::*;
Expand Down
127 changes: 123 additions & 4 deletions src/risk.rs
Original file line number Diff line number Diff line change
Expand Up @@ -3,22 +3,21 @@
//! Findings remain the security facts and policy remains the enforcement
//! boundary. This module is intentionally not part of the public API.
//!
//! E.1 deliberately has no production caller: integration is the separate E.2
//! review gate. Remove this allowance when that gate adds an explicit consumer.
#![cfg_attr(not(test), allow(dead_code))]

use std::collections::BTreeMap;
use std::path::Path;

use serde::{Deserialize, Serialize};
use sha2::{Digest, Sha256};

use crate::error::{Result as ShieldResult, ShieldError};
use crate::output::OutputFormat;
use crate::rules::{Confidence, Finding, RuleEngine, RuleMetadata, Severity};

pub(crate) const MODEL_VERSION: &str = "agentshield-risk-v1";
const COVERAGE_SCHEMA: &str = "agentshield-coverage-v1";
const SATURATION_CONSTANT: u64 = 30;
const MAX_EMITTED_SCORE: u8 = 99;
const MAX_OUTPUT_CONTRIBUTIONS: usize = 50;

#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub(crate) struct RiskAssessment {
Expand All @@ -31,6 +30,7 @@ pub(crate) struct RiskAssessment {
}

impl RiskAssessment {
#[cfg(test)]
pub(crate) fn is_comparable_to(&self, other: &Self) -> bool {
self.model_version == other.model_version && self.coverage_id == other.coverage_id
}
Expand All @@ -52,6 +52,19 @@ pub(crate) struct RiskSummary {
pub(crate) duplicate_findings: usize,
}

#[derive(Serialize)]
struct ExperimentalRiskOutput<'a> {
status: &'static str,
score: u8,
model_version: &'a str,
coverage_id: &'a str,
raw_points: u64,
contributions: &'a [RiskContribution],
contributions_truncated: usize,
summary: &'a RiskSummary,
interpretation: &'static str,
}

#[derive(Debug, Clone, PartialEq, Eq)]
pub(crate) struct CoverageDescriptor {
scanner_version: String,
Expand Down Expand Up @@ -182,6 +195,80 @@ pub(crate) fn assess(
})
}

pub(crate) fn render_experimental(
base_report: &str,
assessment: &RiskAssessment,
format: OutputFormat,
) -> ShieldResult<String> {
let displayed = assessment.contributions.len().min(MAX_OUTPUT_CONTRIBUTIONS);
let output = ExperimentalRiskOutput {
status: "informational",
score: assessment.score,
model_version: &assessment.model_version,
coverage_id: &assessment.coverage_id,
raw_points: assessment.raw_points,
contributions: &assessment.contributions[..displayed],
contributions_truncated: assessment.contributions.len() - displayed,
summary: &assessment.summary,
interpretation:
"Prioritization index only; not a probability, percentage, grade, or policy verdict.",
};

match format {
OutputFormat::Console => Ok(render_experimental_console(base_report, &output)),
OutputFormat::Json => render_experimental_json(base_report, &output),
OutputFormat::Sarif | OutputFormat::Html => Err(ShieldError::Config(
"`--experimental-risk` supports only console and JSON output".to_owned(),
)),
}
}

fn render_experimental_console(base_report: &str, output: &ExperimentalRiskOutput<'_>) -> String {
use std::fmt::Write;

let mut rendered = base_report.to_owned();
if !rendered.ends_with('\n') {
rendered.push('\n');
}
rendered.push('\n');
rendered.push_str("Experimental risk assessment (informational)\n");
let _ = writeln!(rendered, "Score: {}", output.score);
let _ = writeln!(rendered, "Model: {}", output.model_version);
let _ = writeln!(rendered, "Coverage: {}", output.coverage_id);
let _ = writeln!(rendered, "Raw points: {}", output.raw_points);
let _ = writeln!(
rendered,
"Contributions: {} shown, {} omitted",
output.contributions.len(),
output.contributions_truncated
);
for contribution in output.contributions {
let _ = writeln!(
rendered,
"- {} {} {} {}: {} point(s)",
contribution.fingerprint,
contribution.rule_id,
contribution.effective_severity,
contribution.confidence,
contribution.points
);
}
let _ = writeln!(rendered, "Interpretation: {}", output.interpretation);
rendered
}

fn render_experimental_json(
base_report: &str,
output: &ExperimentalRiskOutput<'_>,
) -> ShieldResult<String> {
let mut report: serde_json::Value = serde_json::from_str(base_report)?;
let object = report
.as_object_mut()
.ok_or_else(|| ShieldError::Output("default JSON report was not an object".to_owned()))?;
object.insert("risk_assessment".to_owned(), serde_json::to_value(output)?);
Ok(serde_json::to_string_pretty(&report)?)
}

fn contribution_points(severity: Severity, confidence: Confidence) -> u64 {
severity_weight(severity) * confidence_multiplier(confidence)
}
Expand Down Expand Up @@ -556,6 +643,38 @@ mod tests {
assert!(!base.is_comparable_to(&other));
}

#[test]
fn experimental_output_bounds_contributions_and_reports_omissions() {
let findings = (0..51)
.map(|index| {
finding(
&format!("SHIELD-GOLDEN-{index:02}"),
Severity::Low,
Confidence::Low,
&format!("evidence {index}"),
)
})
.collect::<Vec<_>>();
let assessment = assess(&findings, Path::new("/scan"), &coverage()).expect("assessment");
let rendered = render_experimental(
r#"{"findings":[],"verdict":{"pass":true}}"#,
&assessment,
OutputFormat::Json,
)
.expect("render experimental JSON");
let report: serde_json::Value =
serde_json::from_str(&rendered).expect("parse experimental JSON");

assert_eq!(
report["risk_assessment"]["contributions"]
.as_array()
.expect("contributions")
.len(),
50
);
assert_eq!(report["risk_assessment"]["contributions_truncated"], 1);
}

#[test]
fn arithmetic_overflow_is_explicit() {
assert_eq!(
Expand Down
Loading
Loading