Skip to content

STACK_OVERVIEW.md

Latest commit

 

History

History
84 lines (64 loc) · 4.07 KB

STACK_OVERVIEW.md

File metadata and controls

84 lines (64 loc) · 4.07 KB
Raw

AI Systems Reliability Stack

This repository is one layer in a three-project local evidence stack for reliable AI/agent systems.

System Map

Layer Repository Role Current evidence
Governed workflow gdev-agent Multi-tenant support-triage workflow with webhook intake, guardrails, approval, audit, cost, and observability controls. 310-test local baseline (2026-07-13), default-Compose FORCE RLS/demo proof, 180-case internal smoke eval, load and isolation evidence.
Quality layer Eval-Ground-Truth-Lab Deterministic regression evaluation framework for structured output, routing, unsafe auto-approval, cost, latency, and adapter behavior. 55-case live local gdev-agent conformance baseline; canonical 100-case challenge against exact revision 0e4c5f0fd50382bbf12ffd35cfca4632384fb0cc failed its stricter quality gate.
Runtime layer Agent-Runtime-Grid Queue-backed runtime for running many AI/agent jobs with retries, timeouts, idempotent finalization, artifacts, metrics, and cost controls. 100-job smoke, 500-job reliability proof, failure-injection reports, cross-project artifact proof, and 20-job operator-run live-local HTTP proof snapshot.

How They Connect

The simplest live local path is:

Eval Ground Truth Lab
  -> configured HTTP adapter
  -> gdev-agent /webhook
  -> deterministic validators
  -> baseline report and run artifact

The default Runtime Grid path is artifact-linked runtime proof:

Agent Runtime Grid
  -> selected Eval Lab / gdev case jobs
  -> Redis Streams workers
  -> Postgres lifecycle state
  -> runtime artifacts and reliability report
  -> links back to Eval Lab and gdev-agent evidence

That Runtime Grid mode does not call live gdev-agent over HTTP by default. A separate optional full-stack-live-local mode now runs Grid workers that call a local gdev-agent /webhook endpoint and write sanitized runtime artifacts. The latest committed Runtime Grid snapshot records 20/20 queued local HTTP jobs completed against a local demo-mode gdev-agent stack on 2026-06-15. That live-local mode is still local/operator-run evidence; it does not claim hosted operations, external users, or production traffic.

What An Agent Means Here

An agent is a bounded job type, not an open-ended autonomous persona. It has:

  • input schema
  • output schema
  • allowed tools or side effects
  • model/provider policy
  • budget and timeout
  • guardrail and approval rules
  • eval cases and validators

In this stack:

  • gdev-agent is the first real governed workflow under test.
  • Eval Lab checks whether that workflow still behaves correctly.
  • Runtime Grid runs many agent/eval jobs reliably and records runtime evidence.

Provider Strategy

Default local and CI mode should stay deterministic:

Mode Provider Use
demo / stub deterministic fixtures tests, CI, load/reliability proofs, zero-cost demos
live in gdev-agent Anthropic Claude implemented live support-triage provider path
optional judge in Eval Lab OpenAI provider contract bounded, budget-gated, non-authoritative judging
Runtime Grid full-stack-live-local local gdev-agent HTTP endpoint optional end-to-end local proof; provider behavior is still governed by gdev-agent LLM_MODE
future runtime live jobs model router over Anthropic/OpenAI/Gemini/Mistral/local planned only after explicit budget, egress, and eval gates
future local mode Ollama or vLLM offline/dev/privacy/inference-infra demonstrations

The important boundary is that runtime control decisions stay deterministic. Providers may supply task intelligence, but they do not own scheduling, budget, terminal state, tenant isolation, approval policy, or eval pass/fail authority.

What Is Not Claimed

This stack is v1 local evidence, not production adoption evidence. It does not claim external users, hosted SaaS operations, production SLOs, exactly-once execution, or a general autonomous swarm.