Local AI (umbrella): court-defensible, on-machine AI — vision, architecture, roadmap

[markkerzner]

Local AI — court-defensible, on-machine AI (umbrella)

The single reference for FreeEed's local-AI story, architecture, and roadmap. Individual work is tracked in the linked issues; this is the map.

Why local

Confidential documents cannot go to ChatGPT/cloud AI. Courts now require it — Morgan v. V2X (no-training/no-disclosure plus written documentation of safeguards); law enforcement has CJIS; the EU has GDPR + the AI Act. For legal / forensic / government data, cloud AI is a non-starter. So the AI runs where the data already is — nothing leaves the machine. That is the differentiator cloud tools (Harvey et al.) structurally can't match for sensitive data.

Architecture (the "how", and a reusable pattern)

• OpenAI-compatible local serving. All AI calls speak the OpenAI API shape, so "go local" = re-point base_url + model name. Ollama on the desktop, vLLM on a server — one code path, config-scaled. Toggle, not a rewrite.
• Everything local, not just the LLM: local embeddings + vector DB (Chroma) for RAG, and local OCR (Tesseract) — otherwise the document leaks at the OCR step even with a local model.
• Prove it, don't just claim it: an egress monitor that records zero outbound connections during a run — a log an auditor/court can rely on. "Nothing leaves the machine, and we can prove it."
• Reproducibility bundle: pin the model (by hash) + runtime + temp=0/seed + prompts + retrieval → re-run a year later, same determinations. "You can put a local model in the case file; you can't put ChatGPT in a case file."
• Accuracy validation: measure precision/recall vs an incumbent before switching a model in — the switch is defensible because it's measured, not assumed.

Roadmap (linked work)

Local serving & models

• Scaia-ai/ai_advisor#53 — On-prem AI mode: local/Ollama, no egress, no-training guarantee (flagship)
• Use local LLM #506 — Use local LLM · LLM management: download/manage models the Ollama way + share Ollama's model folder #570 — model management the Ollama way · Multiple AI models - choice of the user #498 / Scaia-ai/ai_advisor#50 — multiple models, user's choice
• Local vector database #469 (done) — local vector database · Scaia-ai/ai_advisor#57 — local-model hardware sizing + EU AI Act posture

What the local AI does

• RAG implementation (including a reference corpus of law-texts) #571 — RAG implementation · Conversational + voice case interaction (citation-backed Q&A, à la Relativity aiR Assist) #558 — citation-backed Q&A · Case notebook: write AI answers to a running document + add verified findings to the case #575 — case notebook (answers → case file)
• Scaia-ai/ai_advisor#55 — defensible review + privilege log · Research: investigative AI experiments (contradictions, stalled-case leads, narratives) #578 / Scaia-ai/ai_advisor#56 — investigative lead-gen
• Scaia-ai/ai_advisor#54 — multilingual (EU languages) · FOIA — AI Summary and Document Analysis #540 — FOIA AI summary · Search: offer keyword vs AI search choice after ingestion #568/Search: run AI search on a subset (by directory and by keyword-created set) #569 — AI search choice/subsets
• Case statistics from metadata, not RAG (impl: ai_advisor#58) #596 / Scaia-ai/ai_advisor#58 — route statistics to metadata, not RAG

Compliance / defensibility / onboarding

• Compliance: auto-append a configurable legal disclaimer (EU AI Act) to every AI output #576 — EU AI Act disclaimer on every AI output · docs: LLM-paste setup guide (.md 'skills file') for non-coder guided install + first project #562 — non-coder LLM-paste setup guide

Design docs

local-ai-architecture.md (the how) · hardware-sizing.md (what hardware to run it on — workstation/server tiers) · local-ai-cjis-briefing.md (the why/compliance) · FreeEed-2027-Vision.md.

Status

Local vector DB done (#469); local-LLM path + model management + RAG in progress; egress attestation and the reproducibility bundle are the defensibility capstones. Implementation of the AI layer lives in Scaia-ai/ai_advisor (roadmap stubs here per policy).