OMNISCIENT

Unified Enterprise & OSINT Search Platform

OMNISCIENT is a federated, multi-tenant search platform that empowers organizations to query public web sources, authorized internal systems, and licensed threat-intelligence feeds through a single natural-language query interface. It features real-time streaming results, hybrid keyword + semantic ranking, and standing alert monitoring.

Built on a legitimate, credential-scoped, rate-limit-respecting data-access model with no unauthorized crawling or CAPTCHA evasion.

---

🌟 Key Features & Technical Innovations

• Federated Search Engine: Single interface to query internal tools (Jira, Confluence, Slack, Google Drive), public web APIs (arXiv, GitHub, SEC EDGAR), and Threat Intel Feeds (CVE/NVD, HaveIBeenPwned, CISA KEV).
• Source-Aware Hybrid Ranking: Fuses traditional BM25 keyword search (Elasticsearch) with Semantic Vector search (Qdrant via BGE-small embeddings) using Reciprocal Rank Fusion (RRF), re-weighted by a learned source-trust/freshness prior.
• Smart Query Classification: A lightweight, local classifier (sub-20ms) predicts source categories before dispatch. E.g., "Q3 churn" only routes internally; "CVE-2025" routes to threat-intel feeds, saving API calls.
• OSINT Graph Pivot View: Entities (people, orgs, domains) extracted via NER are written to Neo4j, allowing analysts to pivot from a search result to an interactive graph of related entities.
• Real-Time Streaming (SSE): Results stream to the React client via Server-Sent Events (SSE) as each source connector responds, providing a per-source "searching… (N found)" indicator for immediate feedback.
• Unified Alerting Subsystem: Standing alerts re-run through the identical orchestration/ranking pipeline as interactive queries, ensuring alert quality never diverges from search quality.
• Deduplication: Near-duplicate text detection using SimHash and perceptual hashing to clean up federated noise.

---

🛡 Security, Privacy & Permission Boundaries

OMNISCIENT enforces strict isolation rules. The One Non-Negotiable Rule: Any ticket touching an internal connector MUST pass a permission-boundary test in CI.

• Zero-Trust Connectors: The system mathematically proves that a user restricted from a document in the native platform cannot retrieve it via OMNISCIENT. We never use global service accounts; all queries execute using the user's scoped OAuth tokens.
• WORM Audit Logging: Immutable audit logs natively enforced at the database level to track all mutations, queries, and connector auth events (crucial for SOC 2 compliance).
• Multi-Tenant & RBAC: Enforces strict isolation of data by org_id using Postgres/Elasticsearch/Qdrant Row-Level Security (RLS) and granular Role-Based Access Control (Viewer, Analyst, Org Admin).
• Secure Credential Vault: All OAuth tokens and API keys are stored in HashiCorp Vault, never in plaintext application databases.
• Good API Citizen: A shared request-hygiene layer enforces per-source rate limits, exponential backoff, and robots.txt/ToS compliance.

---

🏗 Architecture & Core Components

OMNISCIENT is built around a microservices-based monorepo architecture designed for Kubernetes/Helm deployment.

graph TD;
    User((User/Client)) --> API_Gateway
    API_Gateway[API Gateway] --> Classifier[Query Classifier]
    API_Gateway --> Auth[Auth & Vault]
    
    Classifier --> Orchestrator
    
    Orchestrator --> Connectors[Source Connectors]
    Connectors --> PublicWeb[Public Web APIs]
    Connectors --> ThreatIntel[Threat Intel Feeds]
    Connectors --> Internal[Internal SaaS]
    
    Orchestrator --> Indexing[Indexing Service]
    Indexing --> Elasticsearch[(Elasticsearch)]
    Indexing --> Qdrant[(Qdrant)]
    Indexing --> Neo4j[(Neo4j Graph)]
    
    Orchestrator --> Ranking[Ranking Service]
    Ranking --> RRF(RRF Fusion)
    
    Alerting[Alerting Service] --> Orchestrator

Loading

Services Directory

• api-gateway/: The entry point. Handles routing, multi-tenant JWT auth, RBAC, and Write-Once-Read-Many (WORM) audit logging.
• query-classifier/: Lightweight NLP model predicting source routing and semantic/keyword weighting per query.
• orchestrator/: Manages the SourceConnector contracts. Fans out queries through a robust rate-limited, exponential backoff client (Celery-backed).
• indexing-service/: Receives normalized results, deduplicates them (SimHash), and writes to Elasticsearch (BM25), Qdrant (Vectors), and Neo4j (Entities). Features a Kafka-driven CDC/Webhook pipeline for incremental updates.
• embedding-service/: Self-hosted bge-small-en-v1.5 served via ONNX Runtime for semantic embeddings (Zero external API costs).
• ranking-service/: Implements Reciprocal Rank Fusion (RRF) and source-trust weighting.
• alerting-service/: Manages saved queries and periodic incremental state diffing for real-time alerting.
• credential-vault-client/: Handles HashiCorp Vault interactions to securely manage tokens.
• frontend/: React + TypeScript + Vite SPA featuring natural-language search, live SSE streaming, and an admin connector dashboard.

---

🚀 Getting Started

Prerequisites

• Docker & Docker Compose
• Python 3.11+
• Poetry (Dependency Management)
• Make

Quickstart

1. Clone the repository

   git clone https://github.com/HarshkumarG007/OMNISCIENT.git
   cd OMNISCIENT

2. Environment Variables Copy the example config and populate required API keys (if testing specific licensed connectors).

   cp .env.example .env

3. Spin up Infrastructure Launch Postgres (15), Elasticsearch, Qdrant, Neo4j, Redis, Kafka, Zookeeper, and the Microservices.

   docker compose up -d
   # Check service health
   ./scripts/smoke_test.sh

4. Run the API & Tests

   poetry install
   make test   # Run full test suite (including permission-boundary tests)
   make lint   # Run ruff formatting & mypy type-checking

---

🛠 Tech Stack

• Frontend: React, TypeScript, Vite, Cytoscape.js (for Neo4j graph visualization)
• Backend: Python 3.11+, FastAPI, Pydantic v2, Celery
• Databases: PostgreSQL (Relational), Elasticsearch (Text/BM25), Qdrant (Vector/HNSW), Neo4j (Graph), Redis (Cache)
• Infrastructure: Docker, Kafka, Zookeeper, HashiCorp Vault
• Observability: Prometheus, Grafana, OpenTelemetry, Loki
• Testing & Quality: Pytest, pytest-asyncio, Ruff, MyPy, Bandit
• CI/CD: GitHub Actions (Unit Tests, Syft/Grype SBOM Scanning)

---

🤝 Contributing

OMNISCIENT uses Conventional Commits. For all new features, please branch from main using the pattern ticket/<ticket-id>-<slug>. Ensure all tests, linting, and Grype security scans pass before opening a Pull Request.

Note on SBOM Scanning: The CI pipeline enforces rigorous software supply-chain security using syft and grype. Any dependencies containing High/Critical CVEs will fail the build unless explicitly documented and safely ignored via .grype.yaml for false positives.