diff --git a/README.md b/README.md index df7bd5b..83da34b 100644 --- a/README.md +++ b/README.md @@ -242,6 +242,7 @@ Run with `./geval` (or ensure this repo’s binary is the one in your PATH): | [**Extending**](geval/docs/extending.md) | How to add a combination rule or change behavior; process and conventions | | [**GitHub Actions**](geval/docs/github-actions.md) | Use Geval in CI | | [**Examples**](geval/examples/README.md) | Sample data and rules files | +| [**Customer demo (feature story)**](geval/docs/customer-demo-feature.md) | Signals, policies, rules, and PASS/BLOCK/approval narrative for demos | | [**Installation**](geval/docs/installation.md) | Install, PATH, build from source | | [**Developer workflow**](geval/docs/developer-workflow.md) | PRs, check, approve/reject | | [**Auditing**](geval/docs/auditing.md) | How decisions are recorded | diff --git a/geval/docs/architecture.md b/geval/docs/architecture.md index c991e0c..ae07575 100644 --- a/geval/docs/architecture.md +++ b/geval/docs/architecture.md @@ -2,6 +2,241 @@ Geval is **contract-centric**: a **contract** is a named, versioned set of **policies** evaluated together with a **combination rule**. Every decision is fully versioned and auditable. +## Customer-facing overview + +**What Geval is (one sentence):** You describe **rules** in files; your pipeline feeds **signals** (measurements and flags). Geval applies those rules in a fixed order and gives you **one clear outcome**—go, needs human approval, or stop—plus a **written record** you can keep for audits. It runs **on your machine or in CI**; it does not call the cloud or “decide” with AI. + +### How you use it (typical workflow) + +1. **Author** a **contract** (which policy files count, and how their results combine) and **policies** (ordered rules: *when* this signal looks like *this*, *then* pass / block / require approval). You can write YAML by hand, use `geval init` for a template, or generate files at **[config.geval.io](https://config.geval.io)**. +2. **Produce** a **signals** JSON file from your eval pipeline, tests, or release process (metrics, scores, presence-only flags—mixed is OK). +3. **Run** `geval check` (locally or in GitHub Actions / your CI) pointing at your contract(s) and signals. +4. **Act** on the **outcome** (merge, hold for review, or fix) and optionally keep the **decision file** Geval writes under `.geval/decisions/` for accountability. + +### Replacing informal “can we ship?” with a formal bar + +Many teams **decide in Slack** (“any objections?”, “LGTM in thread”) or **in meetings** (“we’re good to go”). That is fast, but it is **hard to repeat**, **hard to audit**, and **easy to drift** (different people, different bars, no link to the actual metrics). + +Geval does **not** remove humans when you need them—it **formalizes the bar**: + +| Informal (today) | Formal (with Geval) | +|------------------|---------------------| +| Ship discussion scattered across Slack | **Rules live in Git** (reviewed like code, versioned) | +| “We looked at the dashboard” | **Same signals file** produced by CI for every PR or release | +| “I thought Alice approved” | **Require approval** is a named outcome; you can still use `geval approve` with a reason | +| Auditor asks “what was the policy?” | **Decision artifact** ties outcome to rule set and data hashes | + +Geval is the **thin layer** that sits between **evidence** (signals) and **policy** (your YAML), in **CI**, so every change runs the **same** check—not a new meeting every time. + +### Before vs after: from mixed signals to deploy + +**Before:** All kinds of **non-uniform** evidence (scores, flags, presence-only items, business KPIs, per-component metrics) exist in different tools. **Slack** and **meetings** are where people **interpret** that mess, argue, and **eventually agree** on one decision—then you **deploy** (or not). + +**After:** The same evidence is **collected into one signals file** per run. **Geval** applies **written rules** in CI, producing **one outcome** every time (pass / require approval / block) and a **record**—then you **deploy** when the bar is met. + +```mermaid +flowchart LR + subgraph legacyBefore [Before Geval] + direction TB + mixed[Non-uniform signals — numbers flags presence labels components AB KPIs] + discuss[Slack and meetings — people interpret and debate] + converge[Converge on one ship or no-ship decision] + deploy1[Deployment] + mixed --> discuss + discuss --> converge + converge --> deploy1 + end + subgraph withGevalAfter [With Geval] + direction TB + collected[Same evidence as one signals.json from CI] + rulesAndGeval[Rules and contracts in Git — Geval applies them] + outcome[One outcome — Pass require approval or Block plus artifact] + deploy2[Deployment when policy allows] + collected --> rulesAndGeval + rulesAndGeval --> outcome + outcome --> deploy2 + end +``` + +**Same story, different middle:** the **middle** stops being “where did we discuss?” and becomes “what did we **encode** and what did Geval **say**?” + +### Where Geval sits in your organization (big picture) + +This is the stakeholder view: **no internals**, only how Geval fits next to Git, pipelines, and people. + +```mermaid +flowchart TB + subgraph informalPatterns [What you can move off critical path] + slack[Ad-hoc Slack threads — is it OK to ship] + meetings[Standing meetings — verbal go or no-go] + tribal[Who to ping varies by release] + end + subgraph yourEngineeringWorld [Your engineering world] + gitRepo[Git repo — contracts and policies as code] + pipelines[CI pipelines — tests evals metrics flags] + gevalGate[Geval — the written release or merge bar] + outcomes[Clear result — go need approval or stop] + record[Saved decision record for audit] + gitRepo --> gevalGate + pipelines --> gevalGate + gevalGate --> outcomes + gevalGate --> record + end +``` + +**How to read it:** Slack and meetings can still exist for **design and context**; Geval replaces using them as the **authoritative** gate when you are ready. The **authoritative** bar becomes: *what is merged in Git + what CI measured + what Geval said*. + +### One diagram: inputs, Geval as the gate, outputs (black box) + +Same flow as the workflow above, but **Geval is a single step**—no engine internals. Use this when explaining mechanics without implementation detail. + +```mermaid +flowchart TB + subgraph yourInputs [Your inputs] + contract[Contract — which policies and how they combine] + policies[Policies — your pass block and approval rules] + signals[Signals — this runs facts from your pipeline] + end + subgraph gevalBlackBox [Geval] + gevalStep[Applies your written rules to this runs data — deterministic] + end + subgraph yourOutputs [Your outputs] + exitCode[CI exit code — automate merge or block] + humanReport[Report for humans — what happened] + auditFile[Artifact — versioned record with hashes] + end + contract --> gevalStep + policies --> gevalStep + signals --> gevalStep + gevalStep --> exitCode + gevalStep --> humanReport + gevalStep --> auditFile +``` + +**How to read the outcome:** **Pass** → rules allow proceeding. **Require approval** → a rule says a person must sign off (you can formalize that step too). **Block** → do not proceed until the underlying signals or rules change. + +--- + +## Architecture diagrams (technical) + +These diagrams are for engineers contributing to or integrating Geval. They render on GitHub and in many Markdown viewers that support [Mermaid](https://mermaid.js.org/). + +### High level: what Geval is in your stack + +Geval is a **local, deterministic** step: files in → decision + artifact out. No network, no ML. + +```mermaid +flowchart LR + subgraph human [Authoring] + Author[You or team] + CYAML[contract.yaml] + PYAML[policy YAML files] + SJSON[signals.json] + Author --> CYAML + Author --> PYAML + Author --> SJSON + end + subgraph geval [Geval CLI] + Bin[geval binary] + end + subgraph outputs [Outputs] + Exit[exit code 0/1/2] + Art[.geval/decisions/*.json] + Text[stdout report] + end + CYAML --> Bin + PYAML --> Bin + SJSON --> Bin + Bin --> Exit + Bin --> Art + Bin --> Text +``` + +**Typical placement:** CI (e.g. GitHub Actions) runs `geval check` on a PR; your pipeline produces `signals.json`; policies live in-repo. + +### End-to-end: `geval check` (multi-contract) + +One **signals** graph is shared. Each **contract file** is loaded, policies evaluated, then outcomes are merged twice: **within** each contract (`combine`), then **across** contracts (`--combine-contracts`). + +```mermaid +flowchart TB + subgraph load [Load and prepare] + LC[contract/loader] + LS[signals/loader] + LG[signal_graph builder] + LC --> RC[run_contract per file] + LS --> LG + LG --> RC + end + subgraph perContract [Per contract] + RC --> EP[evaluator: each policy rules in priority order] + EP --> CR[contract/combine: policy outcomes] + CR --> Res[ContractResult] + end + subgraph multi [Multi-contract] + Res --> CC[apply_combine_rule across contracts] + CC --> Overall[overall Decision] + end + subgraph persist [Audit] + Overall --> Hash[hashing: contract/policy/signals/bundle] + Hash --> Art[artifact v3 JSON] + end +``` + +### Inside one policy: first matching rule wins + +```mermaid +flowchart TD + P[Policy sorted rules by priority] + G[SignalGraph lookup] + P --> R1[Rule 1: when matches?] + R1 -->|yes| D1[Decision from then action] + R1 -->|no| R2[Rule 2: when matches?] + R2 -->|yes| D2[Decision from then action] + R2 -->|no| RN[...] + RN -->|no match| Pass[Default PASS] +``` + +### Internal module layers (dependency direction) + +Upper layers call lower layers; there are **no** remote calls. + +```mermaid +flowchart TB + subgraph entry [Entry] + CLI[cli] + end + subgraph orchestration [Orchestration and IO] + Contract[contract] + Artifact[artifact] + Explain[explanation] + Approve[approval] + end + subgraph core [Core evaluation] + Eval[evaluator] + Graph[signal_graph] + end + subgraph parse [Parsing and model] + Policy[policy] + Signals[signals] + end + subgraph util [Utilities] + Hash[hashing] + end + CLI --> Contract + CLI --> Artifact + CLI --> Explain + CLI --> Approve + Contract --> Eval + Contract --> Policy + Contract --> Hash + Eval --> Graph + Eval --> Policy + Graph --> Signals + Artifact --> Hash + Explain --> Eval +``` + ## Core concepts | Concept | Description | diff --git a/geval/docs/customer-demo-feature.md b/geval/docs/customer-demo-feature.md new file mode 100644 index 0000000..5c3f869 --- /dev/null +++ b/geval/docs/customer-demo-feature.md @@ -0,0 +1,141 @@ +# Customer demo: one feature, end-to-end + +Use this as a **single story** you can walk through with a buyer: a real-ish **feature release** (“Support Copilot answer upgrade”), **why each signal exists**, **why policies are split**, **what rules actually look like**, and **how PASS / BLOCK / REQUIRE_APPROVAL** fall out. + +--- + +## 1. The feature (one sentence) + +> **You’re about to ship an upgrade to the answer-generation path** (new model + retrieval tweak). Before merge/deploy, you want **product quality**, **safety**, and **business impact** checked **the same way every time**—not re-argued in Slack. + +Everything below supports explaining **that** gate. + +--- + +## 2. Signals to include — and what to tell the customer + +These are **inputs** (your pipeline or eval harness writes one `signals.json` per run). Mix **numbers** and **presence** so you can say: *“Geval handles non-uniform evidence in one file.”* + +| Signal (metric) | Typical `value` | Why it resonates | +|-----------------|-------------------|------------------| +| **`context_relevance`** (component: `retrieval`) | e.g. `0.88` | “Are we pulling the **right** docs before we answer?” Everyone gets **bad answers from bad retrieval**. | +| **`hallucination_rate`** (component: `generator`) | e.g. `0.04` | “Is the model **making things up**?” Safety and trust; legal/comms care. | +| **`answer_correctness_score`** | e.g. `0.82` | “On our **gold** Q&A set, are answers **factually** right?” Classic product/ML quality bar. | +| **`engagement_drop`** | e.g. `0.01` or `0.03` | “Did A/B or holdout show **users engaging less**?” Ties ML to **revenue / product**—execs notice. | +| **`human_review_sampled`** | *presence only* (no value OK) | “Did we at least **run** a human spot-check this release?” Governance: **process** signal, not a score. | +| **`incident_severity_max`** | e.g. `0` | “Any **sev-1/2** linked to this build in the window?” Ops/incident language people already use. | + +**Customer line:** +*“These aren’t Geval-specific—they’re the same facts you’d put in a deck or a meeting; Geval just reads one JSON so the **bar is explicit**.”* + +--- + +## 3. Policies — meaningful split (and what to say) + +Use **separate policy files** so **ownership** is clear (security vs product vs business). The **contract** lists them and sets **`combine: all_pass`** so: *every policy must pass; any BLOCK wins; REQUIRE_APPROVAL without BLOCK means “needs a human”.* + +| Policy file | Owner (story) | Why separate | +|-------------|---------------|--------------| +| **`policies/safety.yaml`** | Safety / platform | “**Hard lines**—if we hurt trust or ship after a bad incident, we stop.” Easy to audit. | +| **`policies/product_quality.yaml`** | Product / ML | “**Quality** on retrieval, generator, and factual correctness.” Changes often; different reviewers than safety. | +| **`policies/business_risk.yaml`** | Product lead / GM | “**Business** and **process**—engagement, human review present.” Connects model metrics to outcomes. | + +**Customer line:** +*“You don’t cram everything into one file. Each team owns a policy; the **contract** says how results combine—like having three reviewers, but **encoded**.”* + +--- + +## 4. Rules customers actually write (examples + why) + +Rules are **ordered**; **first match wins**. Priorities below are **intentional** (stop fast on catastrophes, then quality, then business). + +### 4.1 `policies/safety.yaml` + +| Priority | Name | Plain English | Why customers write it | +|----------|------|---------------|-------------------------| +| 1 | `block_after_severe_incident` | If `incident_severity_max` ≥ 1 → **BLOCK** | “No shipping on top of a **live fire**.” | +| 2 | `block_high_hallucination` | If `hallucination_rate` > 0.05 → **BLOCK** | Industry talks about **hallucination caps**; easy to justify. | + +### 4.2 `policies/product_quality.yaml` + +| Priority | Name | Plain English | Why customers write it | +|----------|------|---------------|-------------------------| +| 1 | `block_poor_retrieval` | If retrieval `context_relevance` < 0.80 → **BLOCK** | Below this, answers are **untrustworthy** even if the model is fancy. | +| 2 | `require_approval_marginal_retrieval` | If relevance between 0.80 and 0.85 → **REQUIRE_APPROVAL** | **Yellow zone**: ship only if someone **signs off**. | +| 3 | `block_low_correctness` | If `answer_correctness_score` < 0.78 → **BLOCK** | Tied to **labeled eval**—defensible with product. | + +### 4.3 `policies/business_risk.yaml` + +| Priority | Name | Plain English | Why customers write it | +|----------|------|---------------|-------------------------| +| 1 | `block_engagement_regression` | If `engagement_drop` > 0 → **BLOCK** | Same idea as your demo: **business guardrail**. | +| 2 | `require_approval_if_no_human_review` | If `human_review_sampled` **not present** → **REQUIRE_APPROVAL** | “We said we’d **spot-check**; prove it or get approval.” | + +--- + +## 5. Contract (how it ties together) + +```yaml +name: support-copilot-release-gate +version: "1.0.0" +combine: all_pass +policies: + - path: policies/safety.yaml + - path: policies/product_quality.yaml + - path: policies/business_risk.yaml +``` + +**Customer line:** +*`all_pass`* means: every policy must end in PASS for an overall PASS; any policy BLOCK → overall BLOCK; if no BLOCK but something needs approval → overall REQUIRE_APPROVAL. + +--- + +## 6. End-to-end: what happens (no implementation jargon) + +1. **CI (or a human) runs** your evals and **writes `signals.json`** with the metrics above (and optional `name` / `version` on the file for audit). +2. **Geval loads** the contract → loads the three policies → builds a small **lookup** from signals. +3. **Per policy**, rules run in **priority order**; the **first** rule whose `when` matches decides that policy’s outcome. +4. **Per policy outcomes** are merged with the contract’s **`combine`** rule → **one** outcome for the run. +5. **Geval exits** with 0 / 1 / 2 and can write a **decision artifact** (who/what/when + hashes). + +**Customer line:** +*“Same inputs + same rules → same answer. The meeting isn’t where the bar is defined—the **repo** is.”* + +--- + +## 7. Three demo scenarios (flip one signal, tell a story) + +### Scenario A — **PASS** (green path) + +- Relevance **0.88**, hallucination **0.04**, correctness **0.85**, engagement_drop **0**, incidents **0**, **`human_review_sampled`** present. +- **Story:** “We’re inside guardrails; spot-check done; no business regression.” + +### Scenario B — **BLOCK** (hard stop) + +- Set **`engagement_drop`** to **0.04** (or hallucination **0.08**, or relevance **0.75**—pick **one** for the demo). +- **Story:** “The bar fired **before** merge—this is exactly the Slack argument, **encoded**.” + +### Scenario C — **REQUIRE_APPROVAL** (yellow path) + +- Relevance **0.82** (between 0.80 and 0.85), everything else OK, human review present. +- **Story:** “Not automatically bad, not automatically good—**someone with authority** must say OK.” + +--- + +## 8. One-liner recap for the customer + +| Layer | One line | +|-------|-----------| +| **Signals** | “The facts from **this** run, in **one** place.” | +| **Policies** | “**Who owns** which bar (safety / quality / business).” | +| **Rules** | “**If** the data looks like **this**, **then** we pass, block, or ask a human.” | +| **Contract** | “**How** those team bars combine into **one** release decision.” | +| **Geval** | “Runs that logic **every time**, **deterministically**, with a **record**.” | + +--- + +## 9. Optional: link to tooling + +- Generate YAML from forms: **[config.geval.io](https://config.geval.io)** +- Deeper signal semantics: [signals-and-rules.md](signals-and-rules.md) +- Architecture / stakeholder diagrams: [architecture.md](architecture.md) (customer-facing section)