Add inference-time threat evidence critic

.env.example

@@ -39,6 +39,10 @@ LOCAL_MODEL=local-model
 OLLAMA_BASE_URL=http://localhost:11434
 OLLAMA_MODEL=llama2
 LLM_CONFIG_FILE=./llm_config.json
+# Inference-time threat evidence verification (Design phase)
+EVIDENCE_CRITIC_ENABLED=true
+EVIDENCE_CRITIC_MAX_CANDIDATES=5
+EVIDENCE_CRITIC_MAX_THREATS=20
 
 # Governance policy packs and optional infrastructure
 POLICY_PACK_ID=generic-ssdlc

.github/workflows/ci.yml

@@ -41,6 +41,9 @@ jobs:
       - name: Run tests
         run: pytest --cov=app --cov-report=xml --cov-report=term
 
+      - name: Run evaluation harness tests
+        run: pytest evals/tests
+
       - name: Verify generated contracts
         run: python scripts/export_contracts.py --check
 

ARCHITECTURE.md

@@ -201,10 +201,15 @@ stateDiagram-v2
 ### 2. SSDLC Orchestrator (LangGraph) | SSDLC 编排器
 
 -   Built on **LangChain + LangGraph**: stateful, graph-based agent workflow with conditional edges.
--   **Graph Definition**: `StateGraph` with nodes for Router, 6 phase agents, and Reviewer. Graph nodes: Parser → SSDLC Router → Policy+History Agent ∥ Evidence Agent → Drafter Agent → Reviewer Agent.
+-   **Graph Definition**: `StateGraph` with nodes for Router, 6 phase agents, and Reviewer. Assessment graph nodes: Skill Loader → Document Context → Policy+History ∥ Evidence → Drafter → Reviewer → Report Parser → Threat Evidence Critic → Governance Persistence.
 -   **State Schema**: `SSDLCState` TypedDict containing parsed documents, phase findings, threat models, cross-phase references, and metadata.
 -   **Conditional Edges**: Route based on requested phase, project risk level, or full SSDLC mode. SSDLC Router node determines the lifecycle stage and injects stage-specific skill + checklist.
 -   **Parallel Execution**: Policy and Evidence nodes run **in parallel** (LangGraph fan-out/fan-in). Within phases, sub-tasks (e.g. KB lookup + document parsing) run concurrently via `asyncio.gather`.
+-   **Threat Evidence Critic**: For Design reports, an independent inference-time
+    verifier checks every normalized STRIDE threat against stable, line-addressed
+    passages from the current uploaded documents. Unknown citations and verifier
+    failures become `insufficient_evidence`; policy and history chunks cannot prove
+    current-design facts.
 -   **Checkpointing**: Persistent state via LangGraph `MemorySaver` or database-backed checkpointer.
 -   Assessment submission is **non-blocking** — returns task_id immediately, processes in background.
 -   Singleton `KnowledgeBaseService` and cached LLM client shared across requests.

CHANGELOG.md

@@ -9,6 +9,11 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 ## [Unreleased] — PallasGuard merge
 
 ### Added
+- Inference-time Threat Evidence Critic for Design assessments, with
+  supported/contradicted/insufficient-evidence verdicts, stable current-document
+  line citations, safe abstention, grounding metrics, and a reviewer-facing
+  evidence panel.
+- Public threat-model demo architecture document under `examples/`.
 - Governance domain model and Alembic migrations for projects, submissions,
   control instances, questionnaires, audit trails, prompt audit records, and
   sub-agent runs.
@@ -36,6 +41,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 - v5 product trust RFC and architecture, evidence-model, and frontend-design ADRs.
 
 ### Changed
+- Design Drafter and Reviewer prompts now preserve a normalized STRIDE/DREAD
+  threat model before the independent evidence-verification graph node runs.
 - Converged orchestration on LangGraph while keeping DocSentinel's assessment
   task lifecycle, report contracts, and existing API surface intact.
 - Converged LLM access on the DocSentinel provider abstraction with

README.md

@@ -249,6 +249,26 @@ project workflows:
 - **Human-in-the-loop**: Interrupt points for human review at phase boundaries
 - **Checkpointing**: Long-running assessments persist state and resume
 
+### Threat Evidence Critic
+Design-phase STRIDE threats are independently checked against the currently
+uploaded architecture documents in one inference-time pass:
+
+- Every threat receives a `supported`, `contradicted`, or
+  `insufficient_evidence` verdict with a support score and reviewer-facing
+  rationale.
+- Evidence references use content hashes plus stable line locators and include
+  the exact source excerpt shown in the Assessment Workbench.
+- Only current-project document passages are accepted. Invented, missing, policy,
+  or historical references cannot make a threat appear supported.
+- Verification failures safely abstain as `insufficient_evidence`; all verdicts
+  still require human review.
+- No training or fine-tuning pipeline is required. The critic uses the configured
+  OpenAI, Anthropic, Qwen, DeepSeek, compatible, or local runtime model.
+
+For a public-demo input and walkthrough, see
+[`examples/evidence-critic-architecture.md`](./examples/evidence-critic-architecture.md)
+and [`examples/README.md`](./examples/README.md).
+
 ### RAG-Powered Knowledge Base
 Upload your organization's security policies, standards, and past audits. Phase-specific collections ensure each agent retrieves the most relevant context:
 - Requirements: compliance frameworks, security policies
@@ -279,6 +299,9 @@ implemented path covers OWASP Benchmark v1.2 SAST triage:
   semantics while staying self-contained.
 - **Hard-key scoring**: M1 scores CWE-based binary triage with accuracy,
   precision, recall, F1, and false-positive rate, with no LLM judge.
+- **Threat grounding scoring**: the Evidence Critic scorer measures verdict
+  accuracy, supported precision/recall/F1, contradiction recall, abstention rate,
+  citation validity, and the full verdict confusion matrix.
 - **Scorecards**: every run writes machine-readable `scorecard.json` and a
   human-readable `scorecard.md` under `evals/reports/<run_id>/`.
 
@@ -563,6 +586,9 @@ DocSentinel/
 | `DEEPSEEK_API_KEY` / `DEEPSEEK_MODEL` | DeepSeek OpenAI-compatible API | -- / `deepseek-chat` |
 | `COMPAT_API_KEY` / `COMPAT_BASE_URL` / `COMPAT_MODEL` | Any OpenAI-compatible hosted API | -- |
 | `LOCAL_API_KEY` / `LOCAL_BASE_URL` / `LOCAL_MODEL` | Local OpenAI-compatible API | -- / `http://localhost:1234/v1` / `local-model` |
+| `EVIDENCE_CRITIC_ENABLED` | Verify Design-phase threats against current document evidence | `true` |
+| `EVIDENCE_CRITIC_MAX_CANDIDATES` | Candidate passages supplied per threat | `5` |
+| `EVIDENCE_CRITIC_MAX_THREATS` | Maximum threats verified in one inference pass | `20` |
 | `CHROMA_PERSIST_DIR` | Vector DB path | `./data/chroma` |
 | `PARSER_ENGINE` | Parser: `auto`, `docling`, or `legacy` | `auto` |
 | `ENABLE_GRAPH_RAG` | Enable LightRAG graph retrieval | `true` |

app/agent/graph/assessment_graph.py

@@ -221,6 +221,18 @@ async def _parse_report(state: AssessmentGraphState) -> AssessmentGraphState:
     return {"report": report}
 
 
+async def _verify_threat_evidence(
+    state: AssessmentGraphState,
+) -> AssessmentGraphState:
+    from app.services.evidence_critic import verify_threat_model_evidence
+
+    report = await verify_threat_model_evidence(
+        state["report"],
+        state["parsed_documents"],
+    )
+    return {"report": report}
+
+
 async def _persist_governance(state: AssessmentGraphState) -> AssessmentGraphState:
     try:
         count = persist_assessment_control_evidence(
@@ -241,14 +253,16 @@ def compile_assessment_graph():
     graph.add_node("draft_assessment", _draft)
     graph.add_node("review_assessment", _review)
     graph.add_node("parse_report", _parse_report)
+    graph.add_node("verify_threat_evidence", _verify_threat_evidence)
     graph.add_node("persist_gate3_control_evidence", _persist_governance)
     graph.add_edge(START, "load_skill")
     graph.add_edge("load_skill", "build_document_context")
     graph.add_edge("build_document_context", "gather_policy_history_and_evidence")
     graph.add_edge("gather_policy_history_and_evidence", "draft_assessment")
     graph.add_edge("draft_assessment", "review_assessment")
     graph.add_edge("review_assessment", "parse_report")
-    graph.add_edge("parse_report", "persist_gate3_control_evidence")
+    graph.add_edge("parse_report", "verify_threat_evidence")
+    graph.add_edge("verify_threat_evidence", "persist_gate3_control_evidence")
     graph.add_edge("persist_gate3_control_evidence", END)
     return graph.compile()
 

app/agent/orchestrator.py

@@ -257,7 +257,8 @@ async def _drafter_agent(
         "You are DrafterAgent in a multi-agent security workflow. "
         f"{UNTRUSTED_CONTENT_INSTRUCTION} "
         "Create an assessment draft in JSON only with keys: summary, risk_items, "
-        "compliance_gaps, remediations."
+        "compliance_gaps, threat_model, remediations. For design reviews, "
+        "threat_model must use STRIDE categories and identify affected components."
     )
 
     if skill:
@@ -267,7 +268,12 @@ async def _drafter_agent(
             f"Focus areas: {', '.join(skill.risk_focus)}.\n"
             f"{UNTRUSTED_CONTENT_INSTRUCTION}\n"
             "Output strictly JSON with keys: summary, risk_items, "
-            "compliance_gaps, remediations."
+            "compliance_gaps, threat_model, remediations. For design reviews, "
+            "threat_model must contain methodology and a threats array. Every threat "
+            "must contain id, category, description, affected_component, and "
+            "mitigations. Category must be exactly one of: Spoofing, Tampering, "
+            "Repudiation, InformationDisclosure, DenialOfService, "
+            "ElevationOfPrivilege."
         )
 
     user_prompt = (
@@ -318,8 +324,9 @@ async def _merge_drafts(drafts: list[str], skill: object | None = None) -> str:
         f"{UNTRUSTED_CONTENT_INSTRUCTION} "
         f"{skill_info}"
         "Deduplicate findings, keep the highest-severity version of duplicates, "
-        "and merge remediations. "
-        "Output JSON with keys: summary, risk_items, compliance_gaps, remediations."
+        "merge threat-model entries, and merge remediations. "
+        "Output JSON with keys: summary, risk_items, compliance_gaps, threat_model, "
+        "remediations."
     )
     user_prompt = (
         f"{_truncate_at_boundary(drafts_text, 14000)}\n\n"
@@ -352,7 +359,8 @@ async def _reviewer_agent(
         "hallucination resistance. "
         f"{UNTRUSTED_CONTENT_INSTRUCTION} "
         "Output JSON only with keys: summary, confidence, risk_items, "
-        "compliance_gaps, remediations, sources. "
+        "compliance_gaps, threat_model, remediations, sources. Preserve and improve "
+        "the design threat model when one is present. "
         f"Available chunk IDs: {chunk_ids}. "
         "In the `sources` array each entry MUST have: "
         '`"chunk_id"` (one of the available IDs above) and '
@@ -367,7 +375,8 @@ async def _reviewer_agent(
             "Ensure findings match the persona's focus. "
             f"{UNTRUSTED_CONTENT_INSTRUCTION} "
             "Output JSON only with keys: summary, confidence, risk_items, "
-            "compliance_gaps, remediations, sources. "
+            "compliance_gaps, threat_model, remediations, sources. Preserve and "
+            "improve the design threat model when one is present. "
             f"Available chunk IDs: {chunk_ids}. "
             "In the `sources` array each entry MUST have: "
             '`"chunk_id"` (one of the available IDs above) and '
@@ -467,6 +476,7 @@ def _resolve_citations_from_llm(
                 excerpt=src.get("quote", doc.page_content[:240]),
                 evidence_link=evidence_link,
                 score=float(metadata["score"]) if metadata.get("score") else None,
+                source_kind="history" if is_history else "policy",
             )
         )
     return citations
@@ -494,13 +504,101 @@ def _derive_sources_from_chunks(
                 excerpt=doc.page_content[:240],
                 evidence_link=f"{file}#chunk={paragraph_id}" if paragraph_id else None,
                 score=float(metadata.get("score")) if metadata.get("score") else None,
+                source_kind=origin,
             )
         )
         if origin == "history" and citations[-1].evidence_link:
             citations[-1].evidence_link = f"history://{citations[-1].evidence_link}"
     return citations
 
 
+_THREAT_CATEGORY_ALIASES = {
+    "spoofing": "Spoofing",
+    "tampering": "Tampering",
+    "repudiation": "Repudiation",
+    "informationdisclosure": "InformationDisclosure",
+    "informationleakage": "InformationDisclosure",
+    "denialofservice": "DenialOfService",
+    "dos": "DenialOfService",
+    "elevationofprivilege": "ElevationOfPrivilege",
+    "privilegeescalation": "ElevationOfPrivilege",
+}
+
+
+def _canonical_threat_category(value: object) -> str | None:
+    normalized = "".join(character for character in str(value) if character.isalnum())
+    return _THREAT_CATEGORY_ALIASES.get(normalized.casefold())
+
+
+def _normalized_dread_score(value: object) -> dict | None:
+    if not isinstance(value, dict):
+        return None
+    normalized: dict[str, int | float] = {}
+    for key in [
+        "damage",
+        "reproducibility",
+        "exploitability",
+        "affected_users",
+        "discoverability",
+    ]:
+        try:
+            normalized[key] = min(10, max(1, int(value[key])))
+        except (KeyError, TypeError, ValueError):
+            continue
+    try:
+        normalized["total"] = float(value["total"])
+    except (KeyError, TypeError, ValueError):
+        pass
+    return normalized or None
+
+
+def _normalize_threat_model(value: object) -> dict | None:
+    """Normalize common model variations into the strict report contract."""
+    if not isinstance(value, dict):
+        return None
+    raw_threats = value.get("threats", [])
+    if not isinstance(raw_threats, list):
+        return None
+
+    threats: list[dict] = []
+    seen_ids: set[str] = set()
+    for index, raw in enumerate(raw_threats, start=1):
+        if not isinstance(raw, dict):
+            continue
+        category = _canonical_threat_category(raw.get("category"))
+        description = str(raw.get("description") or "").strip()
+        if not category or not description:
+            continue
+        threat_id = str(raw.get("id") or f"T{index}").strip() or f"T{index}"
+        if threat_id in seen_ids:
+            threat_id = f"{threat_id}-{index}"
+        seen_ids.add(threat_id)
+        mitigations = raw.get("mitigations", [])
+        if not isinstance(mitigations, list):
+            mitigations = [mitigations] if mitigations else []
+        threats.append(
+            {
+                "id": threat_id,
+                "category": category,
+                "description": description,
+                "affected_component": raw.get("affected_component"),
+                "dread_score": _normalized_dread_score(raw.get("dread_score")),
+                "mitigations": [
+                    str(mitigation).strip()
+                    for mitigation in mitigations
+                    if str(mitigation).strip()
+                ],
+            }
+        )
+
+    if not threats:
+        return None
+    methodology = str(value.get("methodology") or "STRIDE_DREAD").upper()
+    if methodology not in {"STRIDE", "DREAD", "STRIDE_DREAD"}:
+        methodology = "STRIDE_DREAD"
+    return {"methodology": methodology, "threats": threats}
+
+
 def _parse_llm_output_to_report(
     raw: str,
     task_id: UUID,
@@ -565,7 +663,7 @@ def _parse_llm_output_to_report(
             )
             for gap in parsed.get("compliance_gaps", [])
         ],
-        threat_model=parsed.get("threat_model"),
+        threat_model=_normalize_threat_model(parsed.get("threat_model")),
         vulnerabilities=parsed.get("vulnerabilities", []),
         remediations=[
             Remediation(

app/core/config.py

@@ -73,6 +73,9 @@ class Settings(BaseSettings):
         "http://localhost:1234/v1,http://127.0.0.1:1234/v1,"
         "http://[::1]:1234/v1"
     )
+    EVIDENCE_CRITIC_ENABLED: bool = True
+    EVIDENCE_CRITIC_MAX_CANDIDATES: int = 5
+    EVIDENCE_CRITIC_MAX_THREATS: int = 20
 
     # Governance and optional infrastructure
     POLICY_PACK_ID: str = "generic-ssdlc"

app/models/assessment.py

@@ -55,6 +55,15 @@ class DreadScore(BaseModel):
     total: float | None = None
 
 
+class EvidenceVerification(BaseModel):
+    status: Literal["supported", "contradicted", "insufficient_evidence"]
+    support_score: float = Field(default=0.0, ge=0.0, le=1.0)
+    rationale: str
+    evidence_ids: list[str] = Field(default_factory=list)
+    counterevidence_ids: list[str] = Field(default_factory=list)
+    requires_human_review: Literal[True] = True
+
+
 class Threat(BaseModel):
     id: str
     category: Literal[
@@ -69,11 +78,24 @@ class Threat(BaseModel):
     affected_component: str | None = None
     dread_score: DreadScore | None = None
     mitigations: list[str] = Field(default_factory=list)
+    confidence: float | None = Field(default=None, ge=0.0, le=1.0)
+    citation_ids: list[str] = Field(default_factory=list)
+    verification: EvidenceVerification | None = None
+
+
+class EvidenceCriticSummary(BaseModel):
+    status: Literal["completed", "fallback"]
+    verifier: str
+    supported: int = 0
+    contradicted: int = 0
+    insufficient_evidence: int = 0
+    total: int = 0
 
 
 class ThreatModel(BaseModel):
     methodology: Literal["STRIDE", "DREAD", "STRIDE_DREAD"] | None = None
     threats: list[Threat] = Field(default_factory=list)
+    verification_summary: EvidenceCriticSummary | None = None
 
 
 class Vulnerability(BaseModel):
@@ -112,6 +134,13 @@ class SourceCitation(BaseModel):
     excerpt: str
     evidence_link: str | None = None
     score: float | None = None
+    document_hash: str | None = None
+    locator: str | None = None
+    source_kind: Literal[
+        "current_document",
+        "policy",
+        "history",
+    ] = "policy"
 
 
 class ReportMetadata(BaseModel):