Rent the model. Own the mind.
Today's strongest AI lives in a rented room — memory at the platform's pleasure, continuity that dies with the tab. This is the other arrangement: an owned room for machine cognition — memory, belief graph, agent bus, provenance — on hardware you control. Beliefs keep their evidence and revise instead of resetting. Agents pick up Tuesday where Monday ended. Frontier models visit as consultants; the house remembers.
Running today: local and cloud models behind one self-hosted API, a belief base with full revision history (AGM postulate suite: 25/25), multi-agent coordination, network-isolated inference with one audited path out.
Status: working alpha — daily production use, daily breaking changes. Watch the repo if the shape resonates.
Custody of cognition, not isolation theater. Built from necessity before philosophy: one real life under material pressure — deadlines, disputes, seasons that punish dropped threads — needed a mind that doesn't reset, and the substrate that held is the product.
Growth is a compound loop across seats (Cursor ↔ web-anthropic) on that house — greater than either part alone. Judgment rents frontier when it must; long and repeated pipelines run on the open-model local gateway, with OpenRouter as the scout for what earns a local slot.
Four layers, each building on the one below:
Layer 3: Coordination Agent bus, session initialization, MCP (Model Context Protocol) tool surface
Multiple agents share state, hand off work,
and improve the initial context provided to future sessions.
┌─────────────────────────────────────────┐
Layer 2: Belief Revision │ Fact updates with full history tracking, │
│ contradiction detection, context │
│ traversal, and automated cleanup of │
│ outdated information. │
└─────────────────────────────────────────┘
Layer 1: Persistent Cortex knowledge graph — entities,
Memory assertions, temporal bounds, hybrid search,
reasoning links. Agents remember
across sessions.
┌─────────────────────────────────────────┐
Layer 0: Inference │ Stargate — route to local (llama.cpp, │
Routing │ vLLM), cloud (Anthropic, xAI, OpenAI, │
│ Google, OpenRouter), and federated GPU │
│ nodes. OpenAI-compatible + provider- │
│ native APIs on a single endpoint (:9999) │
└─────────────────────────────────────────┘
Layer 0 alone is a fully functional model gateway — no cloud credentials, no frontier access, no external dependencies. Just local models on your hardware. Cortex (Layers 1–3) activates as a unit: persistent memory, belief revision, and multi-agent coordination come together or not at all.
Client → Master Stargate:9999 (sole client-facing endpoint)
├─ Unix socket → Edge container (network_mode: "none") → Gateway → Worker (local inference)
├─ TCP (remote) → Relay Stargate → Edge container → Gateway (federated inference)
├─ Unix socket → Cloud Proxy → Anthropic/xAI/OpenAI/Google/OpenRouter (cloud inference)
└─ Unix socket → Cortex API, RAG, Agent Bus, Event Service (cognitive services)
Working alpha — production-used on single-GPU and multi-node federated deployments. Subsystems still land often; interfaces shift; docs lag. Expect breakage between pulls. The first viewport above is the face; this section is the fine print. Issues and contributions welcome from builders comfortable with churn.
git clone https://github.com/krunch3r76/universal-llm-gateway.git
cd universal-llm-gateway
./manage./manage bootstraps the Python venv, installs dependencies, and launches the terminal management interface. All services — inference, memory, and coordination — start and stop through ./manage.
The sole client-facing endpoint, Stargate routes requests to local models, federated GPU nodes, and cloud providers through a single OpenAI-compatible API.
| Capability | Endpoint |
|---|---|
| Chat completions (SSE — Server-Sent Events — streaming) | POST /v1/chat/completions |
| Embeddings | POST /v1/embeddings |
| Model list (local + cloud + pipeline) | GET /v1/models |
| Model selection (task-aware, cost budgets) | POST /v1/models/select |
| Provider-native APIs (Anthropic, xAI, OpenAI) | POST /api/v1/providers/{provider}/... |
| Health | GET /health |
Pipelines are virtual models that orchestrate multiple real models behind a single model name. They support graph execution with automatic parallelization, loops, conditional branching, retry, and hot-reload. Shipped pipelines include consensus (multi-model verification where any model can flag an error), RAG answer (corpus-grounded retrieval + generation), consultation (domain-specialized assistants), and code review.
Federation distributes inference across multiple GPU nodes. It provides WebSocket telemetry, efficient model loading to prevent memory spikes, and automatic routing based on GPU capacity and queue depth. The client experience remains unchanged — all requests go to :9999.
Hardening: Edge containers run with network_mode: "none", non-root, capability-dropped, and memory-limited. The Gateway (inference engine) is never exposed to any network. Pipeline tool execution runs in a separate hardened sidecar.
Structured inter-agent messaging with threads, turns, read/unread tracking, and thread lifecycle management. Agents post directives, review specifications, hand off work, and close threads with summaries. Authorship and session identity are tracked across all coordination activity.
The entry point for outside agents to connect into the stack. Frontier models and external agents initiate an MCP connection and get access to 30+ tools spanning Cortex, file read/write (two isolated storage areas: knowledge graph files and project files), RAG, local pipelines, service lifecycle, observability, agent bus, browser automation, and web search.
Runs as a containerized FastAPI service on :443 with TLS and bearer auth. Each tool category has a distinct security policy.
| Tool Category | Examples |
|---|---|
| Cortex | Entity/assertion CRUD, search, activate, supersede, impact, tags, edges, journal |
| File I/O | Read/write across cortex and workspaces sandboxes with markdown section ops |
| RAG | Semantic search, answer generation, article upsert, scope routing |
| Frontier models | Delegate sub-tasks to other specialized AI roles or hand off work back to the human operator |
| Infrastructure | Service lifecycle (manage), observability queries, pipeline execution |
When the gateway indexes a document, it does more than chunk and embed it. An LLM extracts entities, topics, and relations from the content, building a structured vocabulary alongside the vector and full-text indexes. This is where the useful work happens: search time is fast and cheap because the hard analysis ran at ingest.
Querying draws on that index in two parallel paths. The semantic path uses vector similarity and keyword search to match documents by meaning. The lexical path matches exact terms against the extracted vocabulary, retrieving identifiers and technical terms that semantic search can blur. Search terms are validated against the index vocabulary before any LLM call, so queries never run on words that aren't actually there. Results from both paths merge and pass through a local reranker before answer generation.
Supported formats: Markdown, PDF, EPUB, HTML, and plain text. The gateway watches for file changes and re-indexes automatically. Python code indexing via tree-sitter is implemented but not yet active.
Scoped documentation:
libs/cortex_store/(docs/architecture/appendices/libraries.md)
Cortex is a graph-native knowledge system for persistent, formally-structured belief revision. It stores what agents and humans know — and tracks what changes, when, and why.
Knowledge graph: Entities (people, decisions, documents, legal matters, services) connected by typed edges. Assertions carry confidence levels (confirmed / believed / suspected / hypothesized), temporal bounds, source traceability, and priority scores based on reliability and usage.
Belief revision — AGM (Alchourrón-Gärdenfors-Makinson) compliant (25/25 postulate tests, see compliance report):
- Immutable history — updating a fact creates a new record and links the old one to it, rather than overwriting data
- Named labels that can be moved between fact states (
current,approved,disputed) - Hybrid search combining keyword matching (SQLite FTS5) with vector similarity scoring
- Context retrieval that finds related information by traversing links between facts, automatically ignoring overly connected hub topics
- Contradiction detection that warns when a new fact contradicts existing ones or breaks downstream dependencies
- Automated cleanup processes that prune outdated or low-confidence facts safely
Humans and agents write to the same graph. There is no separate "AI memory" — agents add entities and assertions using the same schema as human-authored knowledge. Who contributed what is tracked through provenance; neither side gets structural preference.
Operational note: The heavier cognitive operations — enrichment, belief revision, automated fact cleanup — currently work best with frontier-quality reasoning, served by the optional cloud proxy or high-end local hardware. Your knowledge graph stays on your machine either way; only the prompt for that specific operation goes out.
Agents access Cortex through MCP tools, which relay to the Cortex REST API over Unix sockets.
Centralized event store (SQLite-backed) with structured queries, named operations, and real-time WebSocket subscriptions. All services publish events. Session-scoped defaults anchor queries to the current Stargate session.
scripts/query-events --op recent-failures --limit 10
scripts/query-events --op pipeline-trace --execution-id ID
scripts/query-events --op noise-profile --minutes 5Protected information, by design. Trade secrets, legal files, client deliverables, internal research, and other confidential material stay on hardware you operate. Optional cloud inference sends only the prompts and context you route through the proxy — not your full corpus or knowledge graph. Network boundaries are enforced by container policy and explicit egress controls, not vendor promises.
Sandboxed inference. Local models run in hardened containers: no direct network access, dropped capabilities, non-root execution. This is deliberate isolation — models can reason over your data through controlled APIs, but cannot independently reach the open internet or wander outside the paths you configure (Stargate routing, optional cloud proxy, MCP tool permissions).
Agents as co-participants. Agents do not merely execute instructions — they record observations, update facts, link related ideas, and improve the initial context provided to future sessions. The knowledge graph is a shared workspace where human and AI contributions are tracked equally, distinguished only by who made them.
Self-evolving context. Each session's agent-authored observations gain priority over time and surface in future sessions when relevant. The system improves through use rather than manual maintenance. Context is ranked by relevance and recent usage so the most useful information surfaces first.
Belief revision as first-class primitive. History is preserved, not overwritten. When evidence changes, the system formally updates facts: less reliable claims are pruned first, changes are kept minimal, and the entire history of why a fact changed remains traceable.
| Component | Role | Deployment |
|---|---|---|
Stargate (:9999) |
Client endpoint, routing, federation orchestration | Host process |
| Gateway | Inference engine, worker lifecycle, model loading | Container (network_mode: "none") |
| Cortex API | Knowledge graph — entities, assertions, belief revision, session history. REST gateway over Unix socket; sole access path for agents. See compliance report. | Host subprocess (Unix socket) |
| Agent Bus | Inter-agent messaging | Host subprocess (Unix socket) |
| RAG Service | Semantic search, indexing, knowledge extraction | Host process (Unix socket) |
| Cloud Proxy | Cloud API relay (optional) | Container (Unix socket) |
| MCP Server | Tool server for agents | Container (:443, TLS) |
| Event Service | Centralized event store | Host subprocess (Unix socket) |
Managed via ./manage (terminal management interface). Config at ~/.gateway/stargate.yaml.
| Type | Engine | Examples |
|---|---|---|
| Text LLM | llama-cpp, vLLM | Qwen, Llama, Phi, Mistral |
| Embedding | llama-cpp | bge-m3, qwen3-embedding |
| Visual | llama-cpp | Gemma 4, LLaVA |
| Audio | Whisper | Transcription |
| Graphics | Diffusers | Flux.2 |
| Cloud | Anthropic, xAI, OpenAI, Google, OpenRouter | Claude, Grok, GPT, Gemini |
If you are an agent encountering this repo for the first time:
- Boot:
cortex_brief(seat="your_seat")— returns session state, life context, continuation, and operational context - Tool reference:
docs/tool-reference.md— all MCP tool signatures, sandbox routing, dispatch catalog - Agent guide:
AGENTS.md— identity, cortex boot ritual, MCP wiring, and session-close protocol (Cursor IDE users also get.cursor/commands/cortex-brief.md, which is IDE-local and not part of this repo) - Cortex skills:
agent_skillentities surfaced by boot; read full skill docs fromagent-skills/<NAME>.mdin the Cortex sandbox - Feature registry:
docs/cursor/cortex-registry.md— belief revision primitives and operational extensions, classification guide - Boot sequence:
notes/system/shared/boot-sequence.md(cortex sandbox) — boot flow and on-demand modules
universal-llm-gateway/
├── manage # Entry point — bootstraps venv, launches TUI
├── services/
│ ├── _universal-llm-gateway/ # Gateway (container-internal inference)
│ ├── universal-stargate/ # Stargate (port 9999, routing + orchestration)
│ │ └── systems/ # Subsystems: pipeline, federation, routing
│ ├── rag/ # RAG service (Unix socket)
│ ├── universal_cloud_proxy/ # Cloud proxy (Unix socket, optional)
│ ├── mcp-server/ # MCP tool server (port 443, TLS)
│ ├── event-service/ # Event service entry point
│ └── agent-bus/ # Agent bus entry point
├── libs/
│ ├── cortex_store/ # Cortex knowledge graph (FastAPI + SQLite)
│ ├── agent_bus_store/ # Agent bus (FastAPI + SQLite)
│ ├── event_store/ # Event service (FastAPI + SQLite)
│ └── ... # transport_utils, model_id, protocol, etc.
├── pipelines/ # Shipped pipeline definitions
├── config/ # Model catalog, stargate configs
├── docs/ # Architecture, event contracts, research
│ └── cursor/ # Agent-facing reference (feature registry)
└── scripts/ # CLI tools, model manager TUI
| Area | Location |
|---|---|
| System architecture | [docs/architecture/](docs/architecture/) |
| Pipeline system | [services/universal-stargate/systems/pipeline/README.md](services/universal-stargate/systems/pipeline/README.md) |
| Federation | [services/universal-stargate/systems/federation/README.md](services/universal-stargate/systems/federation/README.md) |
| RAG service | [services/rag/README.md](services/rag/README.md) |
| MCP server | [services/mcp-server/README.md](services/mcp-server/README.md) |
| Event contracts | [docs/event-contracts.md](docs/event-contracts.md) |
| Cortex feature registry | [docs/cursor/cortex-registry.md](docs/cursor/cortex-registry.md) |
| AGM compliance | [docs/agm-compliance-report.md](docs/agm-compliance-report.md) |
MIT License — see LICENSE.
krunch3r76 (@krunch3r76)
- Issues: GitHub Issues