Provide source information for every answer in Assistant

[vanguard-hack]

[FEATURE TICKET] Implement Unified Source Citation & Attribution Pipeline

---

1. Overview & Business Context

1.1 Problem Statement

To ensure complete legal compliance, data provenance, and user trust in law enforcement and operational workflows, every factual answer returned by the assistant must provide transparent, standardized source citations.

Currently, responses stream raw text without unified references back to:

• Unstructured Documents: SOPs, legal reference manuals, and policies indexed in the Vector DB.
• Structured Records: FIR records, vehicle registries, and suspect master files queried via Text2ZCQL.
• Agentic/Tool Outputs: External web URLs, Zia OCR extractions, and Qwen2.5-VL vision model observations.

1.2 Objective

Build an attribution parsing and normalization pipeline inside the Response Aggregator & Formatter that maps source metadata into a unified JSON contract and renders interactive citations in the Chat UI Client.

---

2. Technical Architecture & Data Flow

[Vector DB / RAG]       ---> (Document Name, Page, File Type, URI)  ----+
[Catalyst DataStore]    ---> (Table Name, Filter Scope, Record ID)  ----+---> [Response Aggregator & Formatter] ---> [Unified Payload] ---> [Chat UI & Audit Layer]
[Agentic Tools (Groq)]  ---> (Tool Type, Source URL / OCR Target)   ----+

---

3. Detailed Component Specifications

3.1 RAG Pipeline (Unstructured Documents)

• Metadata Extraction: Extract chunk-level metadata upon retrieval from the Vector DB:
  • document_name: Filename (e.g., SOP_Arrest_Procedures_v2.pdf, IPC_Section_Reference.txt).
  • file_type: Format identifier (pdf | txt | docx).
  • location: Specific pointer (e.g., Page 14, Section 3.2 or chunk index).
  • document_uri: Catalyst FileStore URI or authenticated preview link.
• Clearance Check: Ensure the Pre-Retrieval Clearance Filter suppresses document names/IDs if the officer's badge_id lacks clearance.

3.2 Text2ZCQL Pipeline (Structured DataStore)

• Metadata Extraction: Intercept the execution layer of the AST-validated ZCQL query:
  • table_name: Target Catalyst DataStore table (e.g., FIR_Records, Vehicle_Registry).
  • query_filter_summary: Evaluated filter parameters (e.g., FIR_ID = 4029).
  • matched_record_ids: List of primary keys returned by the read-only execution role.
  • execution_type: Hardcoded to Catalyst_DataStore_ZCQL_ReadOnly.

3.3 Agentic Fallback Engine (ReAct Agent / Groq LLM)

• Tool Observation Normalization: Intercept tool execution payloads during ReAct iterations:
  • Web Search Tool: Capture page_title, external url, and target domain.
  • Fast Vision Pre-Parser & Vision Tool: Capture image_filename, model name (Zia_OCR | Qwen2.5-VL), and extracted_field (e.g., License Plate: KA01AB1234).
  • Page Context Tool: Capture screen_id, active_module, and metadata filters.

---

4. API Contract & Response Schema

4.1 JSON Response Payload

The Response Aggregator & Formatter must attach the unified sources array to the API response:

{
  "response_id": "resp_987421_a4b9",
  "badge_id": "OFFICER_4482",
  "text": "According to SOP Section 3.2 and FIR #4029 records, the vehicle was impounded at 14:30 hrs...",
  "sources": [
    {
      "source_id": "src_01",
      "source_type": "rag_document",
      "display_name": "SOP_Arrest_and_Impound_v3.pdf",
      "location": "Page 12, Paragraph 4",
      "uri": "catalyst://filestore/sops/SOP_Arrest_and_Impound_v3.pdf",
      "mime_type": "application/pdf"
    },
    {
      "source_id": "src_02",
      "source_type": "database_record",
      "display_name": "FIR_Records",
      "identifier": "FIR #4029",
      "scope": "Catalyst DataStore (ZCQL Read-Only)",
      "filter_applied": "fir_id = 4029 AND precinct_code = 'BLR_CENTRAL'"
    },
    {
      "source_id": "src_03",
      "source_type": "external_web",
      "display_name": "Ministry of Transport Official Guidelines",
      "uri": "https://mha.gov.in/guidelines/vehicle-seizure-protocol",
      "domain": "mha.gov.in"
    },
    {
      "source_id": "src_04",
      "source_type": "vision_extraction",
      "display_name": "Zia Vision OCR",
      "identifier": "Vehicle_Plate_Scan_img01.jpg",
      "extracted_field": "KA01AB1234"
    }
  ],
  "metrics": {
    "pipeline_route": "HYBRID_RAG_ZCQL",
    "latency_ms": 780
  }
}

---

5. Frontend & UI Implementation Requirements

5.1 Citation Chips

• Display numbered footnote references [1], [2] inline with the generated text.
• Render interactive citation cards/pills below the primary message bubble.

5.2 Interactive Behaviors

• Document Citations (rag_document): Clicking the chip triggers an in-app viewer modal jumping to the exact page and section.
• Database Records (database_record): Clicking triggers a slide-out drawer displaying sanitized record details.
• Web Sources (external_web): Opens the verified external URL in a secure new tab (rel="noopener noreferrer").

---

6. Security, Compliance & Audit Requirements

• Dual-Tier Redaction: Apply pre-retrieval clearance filtering so unauthorized document titles or restricted table columns are never exposed in source metadata.
• Immutable Audit Trail: The Immutable, Append-Only Audit & Logging Layer must log the complete sources array along with the badge_id, route, and execution timestamps.
• URL Allowlisting: External links returned by the web search tool must pass an allowlist check before being displayed to users.

---

7. Definition of Done & Acceptance Criteria

• Unified Ingestion: Backend successfully extracts and parses metadata from RAG chunks, ZCQL AST outputs, and ReAct agent tool observations.
• Hybrid Deduplication: Citations from parallel RAG + Text2ZCQL executions are merged without duplicate entries.
• Role-Based Clearance: Verified that unauthorized badge IDs receive zero metadata leaks from restricted documents or tables.
• Audit Store Sync: Every assistant response logs an immutable audit event containing the complete sources list.
• UI Verification: All citation types (rag_document, database_record, external_web, vision_extraction) render correctly in the chat UI with functional interactive triggers.