Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -242,6 +242,7 @@ Run with `./geval` (or ensure this repo’s binary is the one in your PATH):
| [**Extending**](geval/docs/extending.md) | How to add a combination rule or change behavior; process and conventions |
| [**GitHub Actions**](geval/docs/github-actions.md) | Use Geval in CI |
| [**Examples**](geval/examples/README.md) | Sample data and rules files |
| [**Customer demo (feature story)**](geval/docs/customer-demo-feature.md) | Signals, policies, rules, and PASS/BLOCK/approval narrative for demos |
| [**Installation**](geval/docs/installation.md) | Install, PATH, build from source |
| [**Developer workflow**](geval/docs/developer-workflow.md) | PRs, check, approve/reject |
| [**Auditing**](geval/docs/auditing.md) | How decisions are recorded |
Expand Down
235 changes: 235 additions & 0 deletions geval/docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,241 @@

Geval is **contract-centric**: a **contract** is a named, versioned set of **policies** evaluated together with a **combination rule**. Every decision is fully versioned and auditable.

## Customer-facing overview

**What Geval is (one sentence):** You describe **rules** in files; your pipeline feeds **signals** (measurements and flags). Geval applies those rules in a fixed order and gives you **one clear outcome**—go, needs human approval, or stop—plus a **written record** you can keep for audits. It runs **on your machine or in CI**; it does not call the cloud or “decide” with AI.

### How you use it (typical workflow)

1. **Author** a **contract** (which policy files count, and how their results combine) and **policies** (ordered rules: *when* this signal looks like *this*, *then* pass / block / require approval). You can write YAML by hand, use `geval init` for a template, or generate files at **[config.geval.io](https://config.geval.io)**.
2. **Produce** a **signals** JSON file from your eval pipeline, tests, or release process (metrics, scores, presence-only flags—mixed is OK).
3. **Run** `geval check` (locally or in GitHub Actions / your CI) pointing at your contract(s) and signals.
4. **Act** on the **outcome** (merge, hold for review, or fix) and optionally keep the **decision file** Geval writes under `.geval/decisions/` for accountability.

### Replacing informal “can we ship?” with a formal bar

Many teams **decide in Slack** (“any objections?”, “LGTM in thread”) or **in meetings** (“we’re good to go”). That is fast, but it is **hard to repeat**, **hard to audit**, and **easy to drift** (different people, different bars, no link to the actual metrics).

Geval does **not** remove humans when you need them—it **formalizes the bar**:

| Informal (today) | Formal (with Geval) |
|------------------|---------------------|
| Ship discussion scattered across Slack | **Rules live in Git** (reviewed like code, versioned) |
| “We looked at the dashboard” | **Same signals file** produced by CI for every PR or release |
| “I thought Alice approved” | **Require approval** is a named outcome; you can still use `geval approve` with a reason |
| Auditor asks “what was the policy?” | **Decision artifact** ties outcome to rule set and data hashes |

Geval is the **thin layer** that sits between **evidence** (signals) and **policy** (your YAML), in **CI**, so every change runs the **same** check—not a new meeting every time.

### Before vs after: from mixed signals to deploy

**Before:** All kinds of **non-uniform** evidence (scores, flags, presence-only items, business KPIs, per-component metrics) exist in different tools. **Slack** and **meetings** are where people **interpret** that mess, argue, and **eventually agree** on one decision—then you **deploy** (or not).

**After:** The same evidence is **collected into one signals file** per run. **Geval** applies **written rules** in CI, producing **one outcome** every time (pass / require approval / block) and a **record**—then you **deploy** when the bar is met.

```mermaid
flowchart LR
subgraph legacyBefore [Before Geval]
direction TB
mixed[Non-uniform signals — numbers flags presence labels components AB KPIs]
discuss[Slack and meetings — people interpret and debate]
converge[Converge on one ship or no-ship decision]
deploy1[Deployment]
mixed --> discuss
discuss --> converge
converge --> deploy1
end
subgraph withGevalAfter [With Geval]
direction TB
collected[Same evidence as one signals.json from CI]
rulesAndGeval[Rules and contracts in Git — Geval applies them]
outcome[One outcome — Pass require approval or Block plus artifact]
deploy2[Deployment when policy allows]
collected --> rulesAndGeval
rulesAndGeval --> outcome
outcome --> deploy2
end
```

**Same story, different middle:** the **middle** stops being “where did we discuss?” and becomes “what did we **encode** and what did Geval **say**?”

### Where Geval sits in your organization (big picture)

This is the stakeholder view: **no internals**, only how Geval fits next to Git, pipelines, and people.

```mermaid
flowchart TB
subgraph informalPatterns [What you can move off critical path]
slack[Ad-hoc Slack threads — is it OK to ship]
meetings[Standing meetings — verbal go or no-go]
tribal[Who to ping varies by release]
end
subgraph yourEngineeringWorld [Your engineering world]
gitRepo[Git repo — contracts and policies as code]
pipelines[CI pipelines — tests evals metrics flags]
gevalGate[Geval — the written release or merge bar]
outcomes[Clear result — go need approval or stop]
record[Saved decision record for audit]
gitRepo --> gevalGate
pipelines --> gevalGate
gevalGate --> outcomes
gevalGate --> record
end
```

**How to read it:** Slack and meetings can still exist for **design and context**; Geval replaces using them as the **authoritative** gate when you are ready. The **authoritative** bar becomes: *what is merged in Git + what CI measured + what Geval said*.

### One diagram: inputs, Geval as the gate, outputs (black box)

Same flow as the workflow above, but **Geval is a single step**—no engine internals. Use this when explaining mechanics without implementation detail.

```mermaid
flowchart TB
subgraph yourInputs [Your inputs]
contract[Contract — which policies and how they combine]
policies[Policies — your pass block and approval rules]
signals[Signals — this runs facts from your pipeline]
end
subgraph gevalBlackBox [Geval]
gevalStep[Applies your written rules to this runs data — deterministic]
end
subgraph yourOutputs [Your outputs]
exitCode[CI exit code — automate merge or block]
humanReport[Report for humans — what happened]
auditFile[Artifact — versioned record with hashes]
end
contract --> gevalStep
policies --> gevalStep
signals --> gevalStep
gevalStep --> exitCode
gevalStep --> humanReport
gevalStep --> auditFile
```

**How to read the outcome:** **Pass** → rules allow proceeding. **Require approval** → a rule says a person must sign off (you can formalize that step too). **Block** → do not proceed until the underlying signals or rules change.

---

## Architecture diagrams (technical)

These diagrams are for engineers contributing to or integrating Geval. They render on GitHub and in many Markdown viewers that support [Mermaid](https://mermaid.js.org/).

### High level: what Geval is in your stack

Geval is a **local, deterministic** step: files in → decision + artifact out. No network, no ML.

```mermaid
flowchart LR
subgraph human [Authoring]
Author[You or team]
CYAML[contract.yaml]
PYAML[policy YAML files]
SJSON[signals.json]
Author --> CYAML
Author --> PYAML
Author --> SJSON
end
subgraph geval [Geval CLI]
Bin[geval binary]
end
subgraph outputs [Outputs]
Exit[exit code 0/1/2]
Art[.geval/decisions/*.json]
Text[stdout report]
end
CYAML --> Bin
PYAML --> Bin
SJSON --> Bin
Bin --> Exit
Bin --> Art
Bin --> Text
```

**Typical placement:** CI (e.g. GitHub Actions) runs `geval check` on a PR; your pipeline produces `signals.json`; policies live in-repo.

### End-to-end: `geval check` (multi-contract)

One **signals** graph is shared. Each **contract file** is loaded, policies evaluated, then outcomes are merged twice: **within** each contract (`combine`), then **across** contracts (`--combine-contracts`).

```mermaid
flowchart TB
subgraph load [Load and prepare]
LC[contract/loader]
LS[signals/loader]
LG[signal_graph builder]
LC --> RC[run_contract per file]
LS --> LG
LG --> RC
end
subgraph perContract [Per contract]
RC --> EP[evaluator: each policy rules in priority order]
EP --> CR[contract/combine: policy outcomes]
CR --> Res[ContractResult]
end
subgraph multi [Multi-contract]
Res --> CC[apply_combine_rule across contracts]
CC --> Overall[overall Decision]
end
subgraph persist [Audit]
Overall --> Hash[hashing: contract/policy/signals/bundle]
Hash --> Art[artifact v3 JSON]
end
```

### Inside one policy: first matching rule wins

```mermaid
flowchart TD
P[Policy sorted rules by priority]
G[SignalGraph lookup]
P --> R1[Rule 1: when matches?]
R1 -->|yes| D1[Decision from then action]
R1 -->|no| R2[Rule 2: when matches?]
R2 -->|yes| D2[Decision from then action]
R2 -->|no| RN[...]
RN -->|no match| Pass[Default PASS]
```

### Internal module layers (dependency direction)

Upper layers call lower layers; there are **no** remote calls.

```mermaid
flowchart TB
subgraph entry [Entry]
CLI[cli]
end
subgraph orchestration [Orchestration and IO]
Contract[contract]
Artifact[artifact]
Explain[explanation]
Approve[approval]
end
subgraph core [Core evaluation]
Eval[evaluator]
Graph[signal_graph]
end
subgraph parse [Parsing and model]
Policy[policy]
Signals[signals]
end
subgraph util [Utilities]
Hash[hashing]
end
CLI --> Contract
CLI --> Artifact
CLI --> Explain
CLI --> Approve
Contract --> Eval
Contract --> Policy
Contract --> Hash
Eval --> Graph
Eval --> Policy
Graph --> Signals
Artifact --> Hash
Explain --> Eval
```

## Core concepts

| Concept | Description |
Expand Down
141 changes: 141 additions & 0 deletions geval/docs/customer-demo-feature.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,141 @@
# Customer demo: one feature, end-to-end

Use this as a **single story** you can walk through with a buyer: a real-ish **feature release** (“Support Copilot answer upgrade”), **why each signal exists**, **why policies are split**, **what rules actually look like**, and **how PASS / BLOCK / REQUIRE_APPROVAL** fall out.

---

## 1. The feature (one sentence)

> **You’re about to ship an upgrade to the answer-generation path** (new model + retrieval tweak). Before merge/deploy, you want **product quality**, **safety**, and **business impact** checked **the same way every time**—not re-argued in Slack.

Everything below supports explaining **that** gate.

---

## 2. Signals to include — and what to tell the customer

These are **inputs** (your pipeline or eval harness writes one `signals.json` per run). Mix **numbers** and **presence** so you can say: *“Geval handles non-uniform evidence in one file.”*

| Signal (metric) | Typical `value` | Why it resonates |
|-----------------|-------------------|------------------|
| **`context_relevance`** (component: `retrieval`) | e.g. `0.88` | “Are we pulling the **right** docs before we answer?” Everyone gets **bad answers from bad retrieval**. |
| **`hallucination_rate`** (component: `generator`) | e.g. `0.04` | “Is the model **making things up**?” Safety and trust; legal/comms care. |
| **`answer_correctness_score`** | e.g. `0.82` | “On our **gold** Q&A set, are answers **factually** right?” Classic product/ML quality bar. |
| **`engagement_drop`** | e.g. `0.01` or `0.03` | “Did A/B or holdout show **users engaging less**?” Ties ML to **revenue / product**—execs notice. |
| **`human_review_sampled`** | *presence only* (no value OK) | “Did we at least **run** a human spot-check this release?” Governance: **process** signal, not a score. |
| **`incident_severity_max`** | e.g. `0` | “Any **sev-1/2** linked to this build in the window?” Ops/incident language people already use. |

**Customer line:**
*“These aren’t Geval-specific—they’re the same facts you’d put in a deck or a meeting; Geval just reads one JSON so the **bar is explicit**.”*

---

## 3. Policies — meaningful split (and what to say)

Use **separate policy files** so **ownership** is clear (security vs product vs business). The **contract** lists them and sets **`combine: all_pass`** so: *every policy must pass; any BLOCK wins; REQUIRE_APPROVAL without BLOCK means “needs a human”.*

| Policy file | Owner (story) | Why separate |
|-------------|---------------|--------------|
| **`policies/safety.yaml`** | Safety / platform | “**Hard lines**—if we hurt trust or ship after a bad incident, we stop.” Easy to audit. |
| **`policies/product_quality.yaml`** | Product / ML | “**Quality** on retrieval, generator, and factual correctness.” Changes often; different reviewers than safety. |
| **`policies/business_risk.yaml`** | Product lead / GM | “**Business** and **process**—engagement, human review present.” Connects model metrics to outcomes. |

**Customer line:**
*“You don’t cram everything into one file. Each team owns a policy; the **contract** says how results combine—like having three reviewers, but **encoded**.”*

---

## 4. Rules customers actually write (examples + why)

Rules are **ordered**; **first match wins**. Priorities below are **intentional** (stop fast on catastrophes, then quality, then business).

### 4.1 `policies/safety.yaml`

| Priority | Name | Plain English | Why customers write it |
|----------|------|---------------|-------------------------|
| 1 | `block_after_severe_incident` | If `incident_severity_max` ≥ 1 → **BLOCK** | “No shipping on top of a **live fire**.” |
| 2 | `block_high_hallucination` | If `hallucination_rate` > 0.05 → **BLOCK** | Industry talks about **hallucination caps**; easy to justify. |

### 4.2 `policies/product_quality.yaml`

| Priority | Name | Plain English | Why customers write it |
|----------|------|---------------|-------------------------|
| 1 | `block_poor_retrieval` | If retrieval `context_relevance` < 0.80 → **BLOCK** | Below this, answers are **untrustworthy** even if the model is fancy. |
| 2 | `require_approval_marginal_retrieval` | If relevance between 0.80 and 0.85 → **REQUIRE_APPROVAL** | **Yellow zone**: ship only if someone **signs off**. |
| 3 | `block_low_correctness` | If `answer_correctness_score` < 0.78 → **BLOCK** | Tied to **labeled eval**—defensible with product. |

### 4.3 `policies/business_risk.yaml`

| Priority | Name | Plain English | Why customers write it |
|----------|------|---------------|-------------------------|
| 1 | `block_engagement_regression` | If `engagement_drop` > 0 → **BLOCK** | Same idea as your demo: **business guardrail**. |
| 2 | `require_approval_if_no_human_review` | If `human_review_sampled` **not present** → **REQUIRE_APPROVAL** | “We said we’d **spot-check**; prove it or get approval.” |

---

## 5. Contract (how it ties together)

```yaml
name: support-copilot-release-gate
version: "1.0.0"
combine: all_pass
policies:
- path: policies/safety.yaml
- path: policies/product_quality.yaml
- path: policies/business_risk.yaml
```

**Customer line:**
*`all_pass`* means: every policy must end in PASS for an overall PASS; any policy BLOCK → overall BLOCK; if no BLOCK but something needs approval → overall REQUIRE_APPROVAL.

---

## 6. End-to-end: what happens (no implementation jargon)

1. **CI (or a human) runs** your evals and **writes `signals.json`** with the metrics above (and optional `name` / `version` on the file for audit).
2. **Geval loads** the contract → loads the three policies → builds a small **lookup** from signals.
3. **Per policy**, rules run in **priority order**; the **first** rule whose `when` matches decides that policy’s outcome.
4. **Per policy outcomes** are merged with the contract’s **`combine`** rule → **one** outcome for the run.
5. **Geval exits** with 0 / 1 / 2 and can write a **decision artifact** (who/what/when + hashes).

**Customer line:**
*“Same inputs + same rules → same answer. The meeting isn’t where the bar is defined—the **repo** is.”*

---

## 7. Three demo scenarios (flip one signal, tell a story)

### Scenario A — **PASS** (green path)

- Relevance **0.88**, hallucination **0.04**, correctness **0.85**, engagement_drop **0**, incidents **0**, **`human_review_sampled`** present.
- **Story:** “We’re inside guardrails; spot-check done; no business regression.”

### Scenario B — **BLOCK** (hard stop)

- Set **`engagement_drop`** to **0.04** (or hallucination **0.08**, or relevance **0.75**—pick **one** for the demo).
- **Story:** “The bar fired **before** merge—this is exactly the Slack argument, **encoded**.”

### Scenario C — **REQUIRE_APPROVAL** (yellow path)

- Relevance **0.82** (between 0.80 and 0.85), everything else OK, human review present.
- **Story:** “Not automatically bad, not automatically good—**someone with authority** must say OK.”

---

## 8. One-liner recap for the customer

| Layer | One line |
|-------|-----------|
| **Signals** | “The facts from **this** run, in **one** place.” |
| **Policies** | “**Who owns** which bar (safety / quality / business).” |
| **Rules** | “**If** the data looks like **this**, **then** we pass, block, or ask a human.” |
| **Contract** | “**How** those team bars combine into **one** release decision.” |
| **Geval** | “Runs that logic **every time**, **deterministically**, with a **record**.” |

---

## 9. Optional: link to tooling

- Generate YAML from forms: **[config.geval.io](https://config.geval.io)**
- Deeper signal semantics: [signals-and-rules.md](signals-and-rules.md)
- Architecture / stakeholder diagrams: [architecture.md](architecture.md) (customer-facing section)
Loading