Lakehouse AI Copilot for Enterprise Decisioning

A GenAI-style enterprise incident decisioning system built around lakehouse architecture, retrieval-augmented context, hybrid risk scoring, explainable recommendations, and auditability.

The project is intentionally more than a chatbot: it separates data foundation, feature signals, retrieval, decisioning, API access, audit logs, and evaluation metrics.

Architecture

raw enterprise data
  -> lakehouse bronze/silver/gold tables
  -> feature and signal layer
  -> RAG retrieval over policies, playbooks, prior cases
  -> hybrid decision engine
  -> FastAPI service
  -> audit and monitoring tables

See docs/system-design.md for design tradeoffs and docs/architecture.md for the diagram.

Demo

Run the deterministic local demo:

python3 scripts/run_demo.py

Build the local lakehouse tables:

python3 scripts/build_lakehouse.py

Example API server:

python3 -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"
uvicorn lakehouse_copilot.api.main:app --reload

Then submit an incident:

curl -X POST http://127.0.0.1:8000/submit_incident \
  -H "Content-Type: application/json" \
  -d @examples/incident.json

What It Shows

• Lakehouse-style storage layout using raw, bronze, silver, and gold zones.
• Structured incident features alongside unstructured policy/playbook context.
• RAG retrieval with citations.
• Hybrid decisioning: ML-style risk score, rules, retrieved evidence, and recommendation generation.
• Audit records for every decision, including prompt version, policy evidence, and latency.
• Evaluation hooks for accuracy, retrieval relevance, consistency, latency, and estimated cost.

Test And CI

python3 -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"
pytest
ruff check .

GitHub Actions runs the same focused test/lint gate on pull requests and pushes to main.

Secrets Policy

Do not commit credentials, tokens, private data, customer incident exports, runtime lakehouse outputs, API keys, local databases, logs, or generated parquet/jsonl files. Keep local generated lakehouse files under data/lakehouse/; the repository tracks only data/lakehouse/.gitkeep.

Repository Layout

src/lakehouse_copilot/
  api/           FastAPI endpoints
  core/          shared schemas and settings
  data/          ingestion and sample data loading
  features/      risk and anomaly features
  rag/           document index and retrieval
  decisioning/   decision engine and audit event creation
  evaluation/    metrics calculation
  storage/       local lakehouse table abstraction
docs/            architecture and system design docs
examples/        sample incident payloads
scripts/         runnable demo and data build scripts
tests/           focused unit tests