Improve evidence provenance and citation fidelity

[arthurpanhku]

Why this matters

Security teams cannot approve or reject a finding just because an LLM said so. They need to see the exact evidence, document location, policy rule, retrieval context, and reviewer decision trail.

From first principles: trust in an automated assessment comes from provenance, not prose.

Community help wanted

We need stronger evidence provenance and citation fidelity across parsing, retrieval, assessment, and human review.

Suggested scope

• Preserve source document identifiers, page/section/table coordinates, and parser confidence where available.
• Attach evidence spans to findings and policy controls.
• Track whether a finding came from policy rules, retrieval context, LLM suggestion text, or human review.
• Add tests for citation integrity after document parsing and chunking.
• Improve exported reports so reviewers can jump from a finding to supporting evidence.

Acceptance criteria

• Findings include structured evidence references when source material is available.
• Citation references survive parser and RAG chunk boundaries.
• Reports distinguish rule-derived conclusions from LLM-generated recommendations.
• Tests cover at least one multi-page document and one table-heavy document.

Progress (2026-08-18)

• Add structured SourceCitation fields for source kind, document hash, locator, and evidence link.
• Generate stable line-based current-document locators and exact evidence excerpts.
• Distinguish current-document, policy, and historical retrieval sources.
• Add an independent evidence critic that rejects unsupported or invented citations.
• Add contract, API, evidence-critic, and orchestrator tests for structured provenance.
• Preserve page, section, and table coordinates plus parser confidence through parsing and chunking.
• Add end-to-end citation-integrity tests for a multi-page document and a table-heavy document.
• Add reviewer jump links and complete provenance to all exported finding types.
• Persist the human reviewer decision trail alongside source provenance.

[arthurpanhku]

Roadmap update: connect document citations to a general evidence contract

This issue remains focused on source-document citation fidelity. The broader software-artifact contract is now tracked in #43 (Evidence Envelope v1), with dedicated adapters for SARIF (#44), SBOM (#45), and SLSA provenance/signatures (#46).

The shared objective is that every conclusion records immutable source identity, digest, locator, validator version, derivation path, policy version, and human decision.