diff --git a/.github/workflows/geval.yml b/.github/workflows/geval.yml index d5a9d46..6bc40cd 100644 --- a/.github/workflows/geval.yml +++ b/.github/workflows/geval.yml @@ -38,8 +38,8 @@ jobs: - name: Run Geval run: | ./geval/target/release/geval check \ + --contract geval/examples/contract.yaml \ --signals signals.json \ - --policy geval/examples/policy.yaml \ --env prod continue-on-error: true diff --git a/README.md b/README.md index a97c888..242da90 100644 --- a/README.md +++ b/README.md @@ -58,17 +58,18 @@ geval init This creates a **.geval** folder with: -- **signals.json** — sample signals (scores, presence-only). Edit and add yours. -- **policy.yaml** — sample rules. Edit and add yours. -- **README.md** — how to run from here. +- **contract.yaml** — Contract: name, version, combine rule, and list of policy paths. +- **policies/** — Policy files (e.g. security.yaml, quality.yaml). Edit and add rules. +- **signals.json** — Sample signals. Edit and add yours. +- **README.md** — How to run from here. Then run: ```bash -geval check --signals .geval/signals.json --policy .geval/policy.yaml +geval check --contract .geval/contract.yaml --signals .geval/signals.json ``` -Use a different folder: `geval init my-rules`. Overwrite existing template files: `geval init --force`. +Use a different folder: `geval init my-rules`. Overwrite existing files: `geval init --force`. ### Updating @@ -76,9 +77,9 @@ Use the same download commands. Replace your old file with the new one. Check ve --- -## Use Geval with your own signals and rules +## Use Geval with your own signals and contract -You need **two files**: **your signals** (any kind — scores, flags, presence-only) and **your rules**. Geval doesn't decide; it **orchestrates** and **reconciles** your rules against your signals and returns one outcome. Use `geval init` for a ready-made template, or create the files yourself below. +You need a **contract** (one YAML that references one or more **policy** files) and a **signals** file. Geval evaluates each policy against the same signals, then combines outcomes (e.g. all must pass, or any block blocks). Use `geval init` for a template with a contract and two policies, or create the files yourself below. **All kinds of signals:** Not every signal needs a score. You can mix: entries with a numeric `value`, and entries with no value (presence-only). Use a rule with `operator: presence` to match “this metric exists.” [Details →](geval/docs/signals-and-rules.md) @@ -99,15 +100,25 @@ Example — save as `mydata.json`: You can add labels like `component` or `system` if you need them. [Full example →](geval/examples/signals.json) -### Step 2: Your rules (rules file) +### Step 2: Your contract and policies -A list of rules in order. Geval applies the first rule, then the next, and stops at the first match. It doesn't interpret — it just evaluates your conditions against your signals. +A **contract** is a YAML file that lists one or more **policy** files and a **combination rule** (how to merge their outcomes). Each **policy** file contains ordered rules: **When** [condition on signals], **then** [pass / block / require_approval]. -Each rule says: **When** [something about your signals], **then** [allow / need approval / block]. +Example contract — save as `contract.yaml`: -Example — save as `myrules.yaml`: +```yaml +name: my-gate +version: "1.0.0" +combine: all_pass +policies: + - path: policy.yaml +``` + +Example policy — save as `policy.yaml` (path relative to the contract file): ```yaml +name: quality +version: "1.0.0" policy: rules: - priority: 1 @@ -118,8 +129,6 @@ policy: threshold: 0 then: action: block - reason: "Engagement dropped" - - priority: 2 name: allow_good_accuracy when: @@ -130,38 +139,34 @@ policy: action: pass ``` -**Operators:** `>` greater than, `<` less than, `>=` at least, `<=` at most, `==` equal, `presence` = metric exists (no threshold; use for signals without a score). - -**Actions:** `pass` = allow. `block` = don’t allow. `require_approval` = a person must say yes first. +**Combine rules:** `all_pass` = PASS only if every policy passes; `any_block_blocks` = any policy BLOCK → overall BLOCK. **Operators:** `>`, `<`, `>=`, `<=`, `==`, `presence`. **Actions:** `pass`, `block`, `require_approval`. -[Full example →](geval/examples/policy.yaml) +[Full example →](geval/examples/contract.yaml) and [policy →](geval/examples/policy.yaml) ### Step 3: Run Geval -Point Geval at your two files: - ```bash -./geval check --signals mydata.json --policy myrules.yaml +./geval check --contract contract.yaml --signals mydata.json ``` -(Windows: `.\geval.exe check --signals mydata.json --policy myrules.yaml`) +(Windows: `.\geval.exe check --contract contract.yaml --signals mydata.json`) ### Step 4: Read the outcome -- **PASS** — No rule matched a block or require-approval. You’re good to go. -- **REQUIRE_APPROVAL** — A rule says someone must approve before you go. -- **BLOCK** — A rule says stop. Fix the issue before going. +- **PASS** — Every policy passed (or combined rule says go). +- **REQUIRE_APPROVAL** — At least one policy requires approval. +- **BLOCK** — At least one policy blocks. -To see **which rule** produced that outcome (and which signals it used): +To see **per-policy results** and the combined decision: ```bash -./geval explain --signals mydata.json --policy myrules.yaml +./geval explain --contract contract.yaml --signals mydata.json ``` -To check that your rules file is valid (no run needed): +To validate the contract and all referenced policies: ```bash -./geval validate-policy myrules.yaml +./geval validate-contract contract.yaml ``` --- @@ -215,7 +220,7 @@ Each run is recorded: which rules, which signals, when. So you can always answer | `geval check` | Orchestrate: run your signals + rules → one outcome (PASS / REQUIRE_APPROVAL / BLOCK) | | `geval explain` | Show which rule produced the outcome and which signals were used | | `geval approve` / `geval reject` | Record a person’s approval or rejection | -| `geval validate-policy` | Check your rules file is valid | +| `geval validate-contract` | Validate contract and all referenced policies | --- @@ -223,7 +228,11 @@ Each run is recorded: which rules, which signals, when. So you can always answer | Guide | Description | |-------|-------------| +| [**Architecture**](geval/docs/architecture.md) | Contract = multiple policies + combine rule; module layout | | [**Signals and rules**](geval/docs/signals-and-rules.md) | Non-uniform signals (scores, presence-only, mix); how rules use them | +| [**Signal assumptions**](geval/docs/signal-assumptions.md) | What we assume; what input forms we accept (number, string, trace, object) | +| [**Versioning**](geval/docs/versioning.md) | Contract, policy, and signals versioning; nothing unversioned | +| [**Extending**](geval/docs/extending.md) | How to add a combination rule or change behavior; process and conventions | | [**GitHub Actions**](geval/docs/github-actions.md) | Use Geval in CI | | [**Examples**](geval/examples/README.md) | Sample data and rules files | | [**Installation**](geval/docs/installation.md) | Install, PATH, build from source | diff --git a/geval/.geval/decisions/2026-03-18T18:40:04Z.json b/geval/.geval/decisions/2026-03-18T18:40:04Z.json new file mode 100644 index 0000000..ea8220c --- /dev/null +++ b/geval/.geval/decisions/2026-03-18T18:40:04Z.json @@ -0,0 +1,14 @@ +{ + "artifact_version": "1", + "geval_version": "0.1.2", + "policy_name": "demo-contract", + "policy_version": "1.0.0", + "signals_name": "demo-signals", + "signals_version": "1.0.0", + "policy_hash": "7664d9d21340214b7b6342bb7b90db6ec3c9f546ea4c8dd35a6cb154fa3c655f", + "signals_hash": "344a3d19b3b0241f3a36d817d0a6ce4d3194bd75172353b8207ebcc6c90fc8b2", + "decision": "BLOCK", + "matched_rule": "business_block", + "timestamp": "2026-03-18T18:40:04Z", + "approval": null +} \ No newline at end of file diff --git a/geval/.geval/decisions/2026-03-18T18:53:33Z.json b/geval/.geval/decisions/2026-03-18T18:53:33Z.json new file mode 100644 index 0000000..74a8c60 --- /dev/null +++ b/geval/.geval/decisions/2026-03-18T18:53:33Z.json @@ -0,0 +1,32 @@ +{ + "artifact_version": "2", + "geval_version": "0.1.2", + "contract_name": "release-gate", + "contract_version": "1.0.0", + "contract_hash": "c969102e3d85f861eef2f28c19d619775275adb1830bafb16276e93b9f032a4d", + "signals_name": "my-signals", + "signals_version": "1.0.0", + "signals_hash": "77b6fe14da32a7a2262580067ef84258acd839643b8c7e758f4348008307519e", + "combine_rule": "all_pass", + "policy_results": [ + { + "policy_path": "policies/security.yaml", + "policy_name": "security", + "policy_version": "1.0.0", + "policy_hash": "78cdf683db7d3a416717d1fb9627075f47c6a9154eadda0b237d5d0eb9248347", + "outcome": "PASS" + }, + { + "policy_path": "policies/quality.yaml", + "policy_name": "quality", + "policy_version": "1.0.0", + "policy_hash": "8bf934d103cdcbd4a81cdc1bdddf20cb7114c030be70ea1b1ba3ea39188358c9", + "outcome": "BLOCK", + "matched_rule": "block_engagement_drop" + } + ], + "combined_decision": "BLOCK", + "combined_matched_rule": "policies/quality.yaml:block_engagement_drop", + "timestamp": "2026-03-18T18:53:33Z", + "approval": null +} \ No newline at end of file diff --git a/geval/.geval/decisions/2026-03-18T18:53:37Z.json b/geval/.geval/decisions/2026-03-18T18:53:37Z.json new file mode 100644 index 0000000..9695f80 --- /dev/null +++ b/geval/.geval/decisions/2026-03-18T18:53:37Z.json @@ -0,0 +1,25 @@ +{ + "artifact_version": "2", + "geval_version": "0.1.2", + "contract_name": "demo", + "contract_version": "1.0.0", + "contract_hash": "61e788400405fb9aa92868aefe5b4ecfc4c29118232aa7bbb7df1df2d2ef307b", + "signals_name": "demo-signals", + "signals_version": "1.0.0", + "signals_hash": "344a3d19b3b0241f3a36d817d0a6ce4d3194bd75172353b8207ebcc6c90fc8b2", + "combine_rule": "all_pass", + "policy_results": [ + { + "policy_path": "policy.yaml", + "policy_name": "demo-contract", + "policy_version": "1.0.0", + "policy_hash": "7664d9d21340214b7b6342bb7b90db6ec3c9f546ea4c8dd35a6cb154fa3c655f", + "outcome": "BLOCK", + "matched_rule": "business_block" + } + ], + "combined_decision": "BLOCK", + "combined_matched_rule": "policy.yaml:business_block", + "timestamp": "2026-03-18T18:53:37Z", + "approval": null +} \ No newline at end of file diff --git a/geval/docs/architecture.md b/geval/docs/architecture.md new file mode 100644 index 0000000..58c6d4d --- /dev/null +++ b/geval/docs/architecture.md @@ -0,0 +1,59 @@ +# Geval Architecture + +Geval is **contract-centric**: a **contract** is a named, versioned set of **policies** evaluated together with a **combination rule**. Every decision is fully versioned and auditable. + +## Core concepts + +| Concept | Description | +|--------|-------------| +| **Contract** | YAML file: `name`, `version`, `combine` (rule), and list of policy paths. The unit of evaluation. | +| **Policy** | YAML file: optional `name`/`version`, `environment`, and ordered `rules`. Each rule has `when` (conditions) and `then` (action: pass / block / require_approval). | +| **Signals** | JSON: optional `name`/`version`, and array of signal objects (metric, value, component, etc.). Facts fed into the engine. | +| **Combination rule** | How to merge outcomes from multiple policies: `all_pass` or `any_block_blocks`. | + +## Data flow + +1. **Load contract** – Parse contract YAML; resolve policy paths relative to the contract file; load each policy. +2. **Load signals** – Parse signals JSON; build an in-memory signal graph (metric → value lookup). +3. **Evaluate each policy** – For each policy, evaluate rules in priority order; first matching rule gives that policy’s outcome (PASS / REQUIRE_APPROVAL / BLOCK). +4. **Combine** – Apply the contract’s combination rule to the list of policy outcomes → single combined decision. +5. **Artifact** – Write `.geval/decisions/.json` with contract identity, per-policy results, combined decision, and hashes. + +## Module layout + +``` +geval/src/ + contract/ # Contract = multiple policies + combine rule + model.rs # ContractDef, PolicyRef + combine.rs # CombineRule (all_pass, any_block_blocks), apply_combine_rule + loader.rs # load_contract, load_contract_and_policies, parse_contract_str + runner.rs # run_contract → ContractResult (per-policy + combined) + policy/ # Single policy model and parser + model.rs # Policy, Rule, RuleCondition, RuleConsequence, Action, Operator + parser.rs # parse_policy, parse_policy_str + evaluator/ # Single-policy evaluation + engine.rs # evaluate(policy, graph) → Decision; evaluate_with_trace + signal_graph/ # Build lookup from signals for rule matching + signals/ # Load signals JSON (name, version, signals array) + hashing/ # SHA256 for contract, policy, signals (audit) + artifact/ # Write decision artifact (contract + per-policy + combined) + explanation/ # Human-readable report (explain_contract_result, explain_decision) + approval/ # Approval/rejection artifact (versioned) + cli/ # Commands: check, init, demo, explain, validate-contract, approve, reject +``` + +## Invariants + +- **Nothing unversioned** – Contract, policies, and signals have name/version; artifact records them and hashes. +- **Deterministic** – Same contract + same signals → same combined decision. +- **No remote calls** – All inputs and outputs are local files. + +## Adding a new combination rule + +1. Add a variant to `CombineRule` in `contract/combine.rs`. +2. Implement the logic in `apply_combine_rule` (match on the new variant). +3. Add `Serialize`/`Deserialize` (and `FromStr`/`Display` if you want CLI/artifact string). +4. Add tests in `contract/combine::tests`. +5. Document in [signals-and-rules.md](signals-and-rules.md) or [versioning.md](versioning.md). + +See [extending.md](extending.md) for the full change process. diff --git a/geval/docs/auditing.md b/geval/docs/auditing.md index b3a7a5f..a88bda2 100644 --- a/geval/docs/auditing.md +++ b/geval/docs/auditing.md @@ -1,11 +1,14 @@ # Accountability and Auditing -Geval is designed so auditors can answer: +Geval is designed so **nothing is unversioned**: every decision and every action is auditable. Auditors can answer: - **Why was this deployed?** – Decision report and matched rule (and optional approval reason). -- **Who approved it?** – `geval approve` writes an artifact with `approved_by` and `reason`. -- **What policy was used?** – Policy is version-controlled; artifact stores `policy_hash` (SHA256). -- **What signals existed?** – Artifact stores `signals_hash` (SHA256); signals themselves are produced by your pipeline and can be archived separately. +- **Who approved it?** – `geval approve` writes an artifact with `approved_by`, `reason`, and artifact `version`. +- **What policy (contract) was used?** – Artifact stores `policy_name`, `policy_version`, and `policy_hash` (SHA256). +- **What signals were used?** – Artifact stores `signals_name`, `signals_version`, and `signals_hash` (SHA256). +- **Which Geval binary?** – Artifact stores `geval_version`. + +**Rule of thumb:** When you change policy or signals, bump their `version` so every decision is tied to a specific version. No update without a version update. ## Artifacts @@ -14,11 +17,15 @@ Geval is designed so auditors can answer: Each `geval check` run writes: - **Path:** `.geval/decisions/.json` -- **Contents:** - - `policy_hash` – SHA256 of the policy used - - `signals_hash` – SHA256 of the signals used - - `decision` – PASS | REQUIRE_APPROVAL | BLOCK - - `matched_rule` – name of the rule that fired (if any) +- **Contents (artifact_version 2, contract-centric):** + - `artifact_version` – schema version (e.g. `"2"`) + - `geval_version` – binary version that produced the decision + - `contract_name`, `contract_version`, `contract_hash` – contract identity and content hash + - `signals_name`, `signals_version`, `signals_hash` – signals identity and content hash + - `combine_rule` – how policy outcomes were merged (e.g. `all_pass`, `any_block_blocks`) + - `policy_results` – array of `{ policy_path, policy_name?, policy_version?, policy_hash, outcome, matched_rule? }` for each policy + - `combined_decision` – final outcome (PASS | REQUIRE_APPROVAL | BLOCK) + - `combined_matched_rule` – first non-PASS policy and rule (if any) - `timestamp` – ISO8601 - `approval` – optional; set when an approval is recorded for this decision @@ -27,12 +34,12 @@ Each `geval check` run writes: `geval approve` / `geval reject` write: - **Path:** configurable (e.g. `.geval/approval.json`) -- **Contents:** `approved_by`, `reason`, `timestamp`, `approved` (true/false) +- **Contents:** `version` (artifact format), `approved_by`, `reason`, `timestamp`, `approved` (true/false) ## Reproducibility - **Deterministic:** Same signals + same policy → same decision. -- **Hashes:** Stored in the decision artifact so you can verify which policy and which signals were used. +- **Versions + hashes:** Decision artifact records policy/signals name and version (human identity) and content hashes (integrity). You can verify exactly which contract and signals version was used. - **No remote services:** All inputs and outputs are local files; no telemetry or external calls. ## What Geval does not do diff --git a/geval/docs/extending.md b/geval/docs/extending.md new file mode 100644 index 0000000..407de44 --- /dev/null +++ b/geval/docs/extending.md @@ -0,0 +1,86 @@ +# Extending Geval: Process and Conventions + +This document describes how to change or extend Geval in a consistent, testable way. Use it when adding a new combination rule, policy feature, or CLI command. + +## Principles + +1. **Contract-first** – The contract (multiple policies + combine rule) is the core. New behavior should integrate with contracts and artifacts. +2. **Versioned** – New inputs or artifact fields should be versioned (name/version or artifact_version). +3. **Tested** – Add unit tests for new logic and, when relevant, an integration-style test (e.g. `run_contract` with the new behavior). +4. **Documented** – Update [architecture.md](architecture.md), [versioning.md](versioning.md), or [signals-and-rules.md](signals-and-rules.md) as needed. + +## Process for a typical change + +### 1. Design + +- Decide where the change lives: contract, policy, signals, combination rule, artifact, or CLI. +- If it’s a new combination rule or contract option, describe the semantics (e.g. “overall BLOCK if more than N policies block”). +- If it’s a new artifact field, decide whether it’s required or optional and what happens for old artifacts (if we ever read them). + +### 2. Implement + +- **Contract** – `contract/model.rs`, `contract/loader.rs`, `contract/runner.rs`, `contract/combine.rs`. +- **Policy** – `policy/model.rs`, `policy/parser.rs`; then `evaluator/engine.rs` if rule semantics change. +- **Signals** – `signals/loader.rs`, `signal_graph/` if lookup behavior changes. +- **Artifact** – `artifact/writer.rs`; bump `DECISION_ARTIFACT_VERSION` if the JSON shape changes. +- **CLI** – `cli/commands.rs`; add or update subcommands/args. + +Keep functions small and pure where possible; use `anyhow::Result` and `Context` for errors. + +### 3. Test + +- **Unit tests** – In the same module under `#[cfg(test)] mod tests`: parsers, combine rules, evaluator, hashing. +- **Contract runner tests** – In `contract/runner.rs`: `run_contract` with 1 or 2 policies, different combine rules and outcomes. +- **CLI** – Manual or optional integration test: run `geval check` with a fixture contract/signals and assert exit code and artifact content. + +Run: `cargo test --manifest-path geval/Cargo.toml`. + +### 4. Document + +- **User-facing** – README, [installation.md](installation.md), [signals-and-rules.md](signals-and-rules.md), [versioning.md](versioning.md), [auditing.md](auditing.md). Update examples (e.g. `geval/examples/`) if the contract or CLI changes. +- **Contributor-facing** – [architecture.md](architecture.md) and this file. If you add a new extension point (e.g. new combine rule), add a short “How to add X” subsection here or in architecture. + +### 5. Changelog / release + +- Bump version in `geval/Cargo.toml` and, if needed, `DECISION_ARTIFACT_VERSION` or `APPROVAL_ARTIFACT_VERSION`. +- Note breaking changes (e.g. CLI now requires `--contract` instead of `--policy`) in release notes. + +## Adding a new combination rule + +1. **contract/combine.rs** + - Add a variant to `CombineRule` with `#[serde(rename = "snake_case")]` (or explicit rename). + - Implement `Default` if it should be the default when omitted in YAML. + - In `apply_combine_rule`, add a `match` branch that implements the new semantics. + - Implement `Display` and `FromStr` for CLI/artifact string. +2. **Tests** – Add tests in `contract/combine::tests` for the new rule (e.g. N outcomes → expected combined outcome). +3. **Docs** – Update [signals-and-rules.md](signals-and-rules.md) or [versioning.md](versioning.md) to describe the new rule. + +## Adding a new policy or contract field + +1. **Model** – Add the field to `ContractDef` or `Policy` in the appropriate `model.rs`; use `Option` and `#[serde(default)]` for backward compatibility if we still support old files. +2. **Parser** – If the field comes from YAML, ensure the parser (contract loader or policy parser) reads it and fills the model. +3. **Artifact** – If the field should be audited, add it to `DecisionArtifact` (and to the code that builds the artifact from `ContractResult` and versions). +4. **Tests** – Parse a sample YAML with the new field and assert it’s present; if the field affects evaluation, add an evaluator or runner test. + +## Adding a new CLI command + +1. **Subcommand** – In `cli/commands.rs`, add a variant to `Sub` and a corresponding `*Opts` struct. +2. **Handler** – Implement `run_*` and call it from `Commands::run`. +3. **Help** – Use `#[command(about = "...")]` and `#[arg(...)]` so `geval --help` and `geval --help` are clear. +4. **Docs** – Update README and [installation.md](installation.md) or [github-actions.md](github-actions.md) if the command is part of the main workflow. + +## Reference: where things live + +| Change | Primary files | +|--------|----------------| +| New combine rule | `contract/combine.rs` | +| Contract file format | `contract/model.rs`, `contract/loader.rs` | +| Policy file format | `policy/model.rs`, `policy/parser.rs` | +| Rule matching logic | `evaluator/engine.rs`, `signal_graph/` | +| Signals format | `signals/loader.rs` | +| Decision artifact shape | `artifact/writer.rs` | +| CLI commands | `cli/commands.rs` | +| Init templates | `cli/init.rs` | +| Human-readable report | `explanation/explain.rs` | + +Use this as the single place to look when you want to change behavior and need to know which module to touch first. diff --git a/geval/docs/github-actions.md b/geval/docs/github-actions.md index 326b2dd..9b7a895 100644 --- a/geval/docs/github-actions.md +++ b/geval/docs/github-actions.md @@ -34,8 +34,8 @@ jobs: - name: Run Geval run: | ./geval/target/release/geval check \ + --contract contract.yaml \ --signals signals.json \ - --policy policy.yaml \ --env prod ``` @@ -46,7 +46,7 @@ Use when you rely on an official Geval release: ```yaml - name: Install Geval run: | - curl -L https://github.com/geval/geval/releases/latest/download/geval-linux-x86_64 -o geval + curl -L https://github.com/geval-labs/geval/releases/latest/download/geval-linux-x86_64 -o geval chmod +x geval - name: Generate signals @@ -56,8 +56,8 @@ Use when you rely on an official Geval release: - name: Run Geval run: | ./geval check \ - --signals signals.json \ - --policy policy.yaml + --contract contract.yaml \ + --signals signals.json ``` ## Exit codes @@ -72,7 +72,7 @@ Use these in a later step to fail the job on BLOCK or REQUIRE_APPROVAL if desire - name: Run Geval id: geval run: | - ./geval check --signals signals.json --policy policy.yaml --env prod + ./geval check --contract contract.yaml --signals signals.json --env prod echo "exitcode=$?" >> $GITHUB_OUTPUT ``` @@ -81,13 +81,13 @@ Then `if: steps.geval.outputs.exitcode == '0'` for merge gates. ## Post result to PR (GitHub CLI) ```bash -RESULT=$(./geval check --signals signals.json --policy policy.yaml) +RESULT=$(./geval check --contract contract.yaml --signals signals.json) gh pr comment $PR_NUMBER --body "$RESULT" ``` Or capture the explain output: ```bash -RESULT=$(./geval explain --signals signals.json --policy policy.yaml) +RESULT=$(./geval explain --contract contract.yaml --signals signals.json) gh pr comment $PR_NUMBER --body "$RESULT" ``` diff --git a/geval/docs/signal-assumptions.md b/geval/docs/signal-assumptions.md new file mode 100644 index 0000000..cd11ee9 --- /dev/null +++ b/geval/docs/signal-assumptions.md @@ -0,0 +1,66 @@ +# Signal assumptions and accepted input + +This document states what Geval **assumes** about `signals.json` and what **input forms** it accepts. Use it to decide how to shape your pipeline output and what to expect from rule matching. + +## Assumptions when considering signals + +1. **Signals are facts, not computed by Geval.** + Geval does not validate, aggregate, or derive signals. It only **loads** them and **matches rules** against them. Your pipeline (eval framework, CI, script) is responsible for producing correct values. + +2. **Same metric can appear multiple times** (e.g. per component). + Rules can scope by optional `component` (and in the model, `system`, `agent`, `step`). For threshold rules we use the **first** matching numeric value for (metric, component). Duplicates for the same (metric, component) are not aggregated—first wins. + +3. **No semantic interpretation of units.** + Numbers are just numbers. We do not treat `0.85` as “85%” or “percentage” differently from `85`. If your metric is “percentage”, produce a number (e.g. 0–1 or 0–100) and write rules against that scale consistently. + +4. **Order of signals in the file is preserved** for reporting and for “first value” lookup. We do not guarantee a specific order when multiple signals share the same metric/component; we take the first one we indexed. + +5. **Policy defines the meaning.** + Rules define which metrics and operators matter. Signals that are not referenced by any rule are still loaded and appear in reports but do not affect the decision (except that they are part of the content hash for audit). + +--- + +## Are we accepting all kinds of inputs? + +**Yes, for loading.** The `value` field of each signal is **optional** and can be **any valid JSON value**: + +- **Number** (integer or decimal): `0`, `1`, `0.94`, `120`, `0.5` +- **String**: `"approved"`, `"v1.2"` +- **Boolean**: `true`, `false` +- **Null**: `null` (or omit `value`) +- **Array**: `[1, 2, 3]`, `["a", "b"]` +- **Object**: `{"latency_ms": 10, "p99": 50}`, trace objects, nested structures + +So **any form** — trace, string, number, decimal, percentage (as a number), or complex object — is **accepted** and stored. The file must still be valid JSON and the top-level structure must be either `{ "signals": [ ... ] }` (with optional `name`, `version`) or a raw array of signal objects. + +--- + +## How each input form is used today + +| Form | Loaded? | Presence? | Threshold rules (`>`, `<`, `>=`, `<=`, `==`)? | Display / report? | +|------|---------|-----------|-------------------------------------------------|--------------------| +| **Missing / null** | Yes | Yes | No (rule sees “no value”) | Yes (shown as —) | +| **Number** (int or decimal) | Yes | Yes | **Yes** — used for comparison | Yes | +| **String** | Yes | Yes | **No** (not yet) | Yes | +| **Boolean** | Yes | Yes | No | Yes (as JSON) | +| **Array** | Yes | Yes | No | Yes (as JSON) | +| **Object** (incl. trace) | Yes | Yes | No | Yes (as JSON) | + +- **Presence:** For **any** of these, if the signal has a `metric` (and optional `component`), a rule with `operator: presence` will match when that metric (and component) exists. +- **Threshold rules:** Only **numeric** `value` (JSON number → f64) is used. String, boolean, array, and object are **not** used for `>`, `<`, `>=`, `<=`, `==`. So: + - **Decimal:** Treated as a number; fully supported. + - **Percentage:** If you pass it as a number (e.g. 0.85 or 85), it works like any other number; we don’t interpret “%” in the value. + - **Trace / complex object:** Accepted and stored; they contribute to **presence** and appear in the report, but no threshold rule uses them today. To use them in rules you’d need to either flatten to a numeric signal in your pipeline or extend Geval (e.g. custom operators or extractors). + +--- + +## Summary + +- **Assumptions:** Signals are pre-produced facts; first value per (metric, component) for numeric rules; no unit semantics; policy defines what matters. +- **Accept:** All JSON value types in `value` (number, string, boolean, null, array, object). So yes — **any kind of input in any form** is accepted at load time. +- **Use in rules:** + - **Numeric** → threshold comparisons. + - **Anything else (including no value)** → presence only (and display). + - **String equality** and **complex object** in rules are not supported yet; they are accepted as input and can be added later (see [extending.md](extending.md)). + +If you need to drive decisions from traces or complex objects today, produce **derived numeric or presence-only signals** in your pipeline and feed those into Geval (e.g. `metric: "trace_has_error"`, `value: 1` or presence-only). diff --git a/geval/docs/signals-and-rules.md b/geval/docs/signals-and-rules.md index aa6929c..cebc137 100644 --- a/geval/docs/signals-and-rules.md +++ b/geval/docs/signals-and-rules.md @@ -2,6 +2,8 @@ Geval is a **decision orchestration and reconciliation** engine. It takes **all kinds of signals** (scores, flags, presence-only, categories) and **your rules**, and reconciles them into one outcome. It doesn't decide — it applies your rules. You don’t have to force every signal into a number. +**Assumptions and accepted input:** We accept **any JSON value** in each signal's `value` (number, string, decimal, percentage as number, trace, complex object). For **rule matching**, only **numeric** values are used in threshold rules (`>`, `<`, etc.); everything else (including no value) is used for **presence** and display. See [Signal assumptions and accepted input](signal-assumptions.md) for details. + ## What counts as a signal Each signal is one row of evidence. All of these are valid in the same file: @@ -13,6 +15,8 @@ Each signal is one row of evidence. All of these are valid in the same file: You can mix these in one `signals.json`. Geval does not require every signal to have a `value`, or to be numeric. +**Versioning (audit):** At the top of the signals file you can set `name` and `version` (e.g. `"name": "ci-signals"`, `"version": "1.0.0"`). Bump `version` when your pipeline or schema changes so every decision records which signals version was used. + ## How rules use them Rules only need a **metric** (and optionally **component**) in the `when` block. Then: @@ -34,6 +38,8 @@ So: ```json { + "name": "my-signals", + "version": "1.0.0", "signals": [ { "metric": "accuracy", "value": 0.92 }, { "metric": "human_reviewed" }, @@ -42,9 +48,11 @@ So: } ``` -**policy.yaml:** +**policy.yaml (contract):** Use top-level `name` and `version` to identify the contract; bump version when you change rules. ```yaml +name: release-gate +version: "1.0.0" policy: rules: - priority: 1 diff --git a/geval/docs/versioning.md b/geval/docs/versioning.md new file mode 100644 index 0000000..58ce01c --- /dev/null +++ b/geval/docs/versioning.md @@ -0,0 +1,81 @@ +# Versioning: nothing unversioned + +Every decision and every action in Geval is auditable. Nothing should be updated without a version update. + +## Contract versioning + +A **contract** is a YAML file that lists one or more policy paths and a combination rule. It has: + +- **name** – Identifies the contract (e.g. `release-gate`). Required. +- **version** – **Bump when you add/remove policies or change the combine rule.** + +Example: + +```yaml +name: release-gate +version: "2.1.0" +combine: all_pass +policies: + - path: policies/security.yaml + - path: policies/quality.yaml +``` + +The decision artifact records `contract_name` and `contract_version` so you always know which contract produced a decision. + +## Policy versioning + +Each **policy** file (referenced by the contract) can have its own identity: + +- **name** – Identifies the policy (e.g. `security`, `quality`). Optional. +- **version** – **Bump when you change rules in that policy.** + +Set them at the top level of the policy YAML (or inside the `policy` block). The decision artifact records per-policy `policy_name`, `policy_version`, and `policy_hash` for each policy in the contract. + +## Signals versioning + +Your signals JSON can carry identity and version for audit. + +- **name** – Identifies the signals set (e.g. `ci-signals`). Optional. +- **version** – **Bump when your pipeline or schema changes** so decisions are tied to a specific signals version. + +Example: + +```json +{ + "name": "ci-signals", + "version": "1.2.0", + "signals": [ + { "metric": "accuracy", "value": 0.94 }, + ... + ] +} +``` + +The decision artifact records `signals_name` and `signals_version` when present. + +## Decision artifact + +Every `geval check` writes a versioned artifact to `.geval/decisions/.json`: + +- **artifact_version** – Schema version of the artifact format. +- **geval_version** – Geval binary version that produced the decision. +- **policy_name**, **policy_version** – From the policy (contract) file. +- **signals_name**, **signals_version** – From the signals file. +- **policy_hash**, **signals_hash** – Content hashes (SHA256) for integrity. + +So every decision is fully traceable: which contract version, which signals version, which binary. + +## Approval artifact + +`geval approve` and `geval reject` write an artifact with a **version** field (artifact format version). Old artifacts without `version` are still read (treated as version `"1"`). + +## Summary + +| Item | Where to set | When to bump | +|-----------------|---------------------------|---------------------------------| +| Policy (contract) | `name`, `version` in YAML | When you change rules | +| Signals | `name`, `version` in JSON | When pipeline or schema changes | +| Decision artifact | Written by Geval | Format change → bump constant | +| Approval artifact | Written by Geval | Format change → bump constant | + +**Rule:** No update without a version update. Set name and version on policy and signals, and every decision will record them for audit. diff --git a/geval/examples/README.md b/geval/examples/README.md index a863b34..16dc74c 100644 --- a/geval/examples/README.md +++ b/geval/examples/README.md @@ -1,46 +1,47 @@ # Geval Examples -This directory contains example signals and policy for Geval (decision orchestration and reconciliation). +Example contract, policies, and signals for Geval (decision orchestration and reconciliation). ## Files -- **signals.json** – Example signals (eval metrics, A/B metrics, component-level metrics). -- **policy.yaml** – Example policy with priority-ordered rules: business block, hallucination guard, retrieval quality. +- **contract.yaml** – Contract: name, version, combine rule, and list of policy paths. This example references a single policy. +- **policy.yaml** – One policy with priority-ordered rules: business block, hallucination guard, retrieval quality. +- **signals.json** – Example signals (eval metrics, A/B metrics, component-level). ## Run (from repo root) ```bash -# Build the CLI -cargo build --release +cargo build --release --manifest-path geval/Cargo.toml -# Check: evaluate signals against policy (exit 0=PASS, 1=REQUIRE_APPROVAL, 2=BLOCK) -./target/release/geval check --signals examples/signals.json --policy examples/policy.yaml --env prod +# Check: evaluate signals against contract (exit 0=PASS, 1=REQUIRE_APPROVAL, 2=BLOCK) +./geval/target/release/geval check --contract geval/examples/contract.yaml --signals geval/examples/signals.json --env prod -# Explain: human-readable decision report -./target/release/geval explain --signals examples/signals.json --policy examples/policy.yaml --env prod +# Explain: human-readable report (per-policy + combined) +./geval/target/release/geval explain --contract geval/examples/contract.yaml --signals geval/examples/signals.json --env prod -# Validate policy syntax -./target/release/geval validate-policy examples/policy.yaml +# Validate contract and all referenced policies +./geval/target/release/geval validate-contract geval/examples/contract.yaml ``` -With the example data, the first matching rule is `business_block` (priority 1): `engagement_drop` 0.03 > 0, so the decision is **BLOCK**. +With the example data, the policy matches `business_block`: `engagement_drop` 0.03 > 0, so the decision is **BLOCK**. -## Signal format +## Contract format -Signals are JSON with optional context fields: +- **name**, **version** – Identify the contract for audit; bump version when you change policies or combine rule. +- **combine** – How to merge outcomes from multiple policies: + - **all_pass** – PASS only if every policy passes; any BLOCK → BLOCK; any REQUIRE_APPROVAL (no BLOCK) → REQUIRE_APPROVAL. + - **any_block_blocks** – Any policy BLOCK → overall BLOCK; else any REQUIRE_APPROVAL → REQUIRE_APPROVAL; else PASS. +- **policies** – List of policy file paths (relative to the contract file): e.g. `policy.yaml` or `policies/security.yaml`. -- `system`, `agent`, `component`, `step`, `metric`, `value` -- Optional `type` (e.g. `ab_test`) +## Policy format -The engine builds a signal graph (system → agent → component → step → signal) and matches policy rules by metric (and optional component) with operators: `>`, `<`, `>=`, `<=`, `==`, `presence`. +Each policy file has optional **name** and **version**, and **policy** with: -## Policy format +- **environment** – optional. +- **rules** – priority, name, when (metric, component, operator, threshold), then (action, reason). + +First matching rule wins; no match → PASS. -- `policy.environment`: optional environment name. -- `policy.rules`: list of rules, each with: - - `priority`: lower number evaluated first. - - `name`: rule identifier. - - `when`: condition (metric, optional component, operator, threshold). - - `then`: action (`pass` | `block` | `require_approval`) and optional `reason`. +## Signals format -First matching rule wins; if none match, decision is **PASS**. +JSON with optional **name** and **version** at the top, and **signals**: array of objects with optional `system`, `agent`, `component`, `step`, `metric`, `value`, `type`. diff --git a/geval/examples/contract.yaml b/geval/examples/contract.yaml new file mode 100644 index 0000000..1cb47b0 --- /dev/null +++ b/geval/examples/contract.yaml @@ -0,0 +1,8 @@ +# Example contract: one policy for simple demo. +# For multiple policies, add e.g. policies/security.yaml and policies/quality.yaml and list them here. + +name: demo +version: "1.0.0" +combine: all_pass +policies: + - path: policy.yaml diff --git a/geval/examples/policy.yaml b/geval/examples/policy.yaml index 627312f..ec905d7 100644 --- a/geval/examples/policy.yaml +++ b/geval/examples/policy.yaml @@ -1,3 +1,5 @@ +name: demo-contract +version: "1.0.0" policy: environment: prod rules: diff --git a/geval/examples/signals.json b/geval/examples/signals.json index bde11b6..b2f1854 100644 --- a/geval/examples/signals.json +++ b/geval/examples/signals.json @@ -1,4 +1,6 @@ { + "name": "demo-signals", + "version": "1.0.0", "signals": [ { "system": "support_agent", diff --git a/geval/src/approval/approval.rs b/geval/src/approval/approval.rs index 7fe8293..7ba7800 100644 --- a/geval/src/approval/approval.rs +++ b/geval/src/approval/approval.rs @@ -1,9 +1,13 @@ //! Human approval/rejection artifacts for REQUIRE_APPROVAL flow. +//! Every approval artifact is versioned for audit. use anyhow::{Context, Result}; use serde::{Deserialize, Serialize}; use std::path::Path; +/// Schema version of the approval artifact format. Bump when the shape changes. +pub const APPROVAL_ARTIFACT_VERSION: &str = "1"; + /// Outcome of an approval/rejection action. #[derive(Debug, Clone, Copy)] pub enum ApprovalOutcome { @@ -14,6 +18,9 @@ pub enum ApprovalOutcome { /// Artifact written by `geval approve` or `geval reject`. #[derive(Debug, Serialize, Deserialize)] pub struct ApprovalArtifact { + /// Artifact format version. Defaults to "1" when reading old artifacts. + #[serde(default = "default_approval_version")] + pub version: String, pub approved_by: String, pub reason: String, pub timestamp: String, @@ -21,6 +28,10 @@ pub struct ApprovalArtifact { pub approved: bool, } +fn default_approval_version() -> String { + "1".to_string() +} + /// Write approval artifact to a path (e.g. .geval/approval.json or user-specified). pub fn write_approval( path: &Path, @@ -30,6 +41,7 @@ pub fn write_approval( ) -> Result<()> { let timestamp = chrono::Utc::now().format("%Y-%m-%dT%H:%M:%SZ").to_string(); let artifact = ApprovalArtifact { + version: APPROVAL_ARTIFACT_VERSION.to_string(), approved_by, reason, timestamp, diff --git a/geval/src/artifact/mod.rs b/geval/src/artifact/mod.rs index e7248f8..08487f6 100644 --- a/geval/src/artifact/mod.rs +++ b/geval/src/artifact/mod.rs @@ -1,3 +1,3 @@ mod writer; -pub use writer::write_decision_artifact; +pub use writer::{write_decision_artifact, DECISION_ARTIFACT_VERSION}; diff --git a/geval/src/artifact/writer.rs b/geval/src/artifact/writer.rs index b945f87..012d144 100644 --- a/geval/src/artifact/writer.rs +++ b/geval/src/artifact/writer.rs @@ -1,18 +1,50 @@ //! Write decision artifacts to .geval/decisions/.json +//! +//! Contract-centric: every artifact records the contract (name, version, combine rule), +//! per-policy results, and the combined decision. Nothing is unversioned. -use crate::evaluator::{Decision, DecisionOutcome}; +use crate::contract::ContractResult; +use crate::evaluator::DecisionOutcome; use anyhow::{Context, Result}; use chrono::Utc; use serde::Serialize; use std::path::Path; -/// Artifact written per run. +/// Schema version of the decision artifact format. Bump when the artifact shape changes. +pub const DECISION_ARTIFACT_VERSION: &str = "2"; + +/// Per-policy result as stored in the artifact. #[derive(Debug, Serialize)] -pub struct DecisionArtifact { +pub struct PolicyResultRecord { + pub policy_path: String, + #[serde(skip_serializing_if = "Option::is_none")] + pub policy_name: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub policy_version: Option, pub policy_hash: String, - pub signals_hash: String, - pub decision: String, + pub outcome: String, + #[serde(skip_serializing_if = "Option::is_none")] pub matched_rule: Option, +} + +/// Artifact written per run. Contract-centric; all versioned. +#[derive(Debug, Serialize)] +pub struct DecisionArtifact { + pub artifact_version: String, + pub geval_version: String, + pub contract_name: String, + pub contract_version: String, + pub contract_hash: String, + #[serde(skip_serializing_if = "Option::is_none")] + pub signals_name: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub signals_version: Option, + pub signals_hash: String, + pub combine_rule: String, + pub policy_results: Vec, + pub combined_decision: String, + #[serde(skip_serializing_if = "Option::is_none")] + pub combined_matched_rule: Option, pub timestamp: String, pub approval: Option, } @@ -27,28 +59,54 @@ pub struct ApprovalPayload { /// Write artifact to .geval/decisions/.json pub fn write_decision_artifact( dir: &Path, - policy_hash: &str, + result: &ContractResult, + contract_hash: &str, + policy_hashes: &[String], signals_hash: &str, - decision: &Decision, + signals_name: Option<&str>, + signals_version: Option<&str>, approval: Option, ) -> Result { let decisions_dir = dir.join(".geval").join("decisions"); - std::fs::create_dir_all(&decisions_dir).with_context(|| format!("create {}", decisions_dir.display()))?; + std::fs::create_dir_all(&decisions_dir) + .with_context(|| format!("create {}", decisions_dir.display()))?; let ts = Utc::now().format("%Y-%m-%dT%H:%M:%SZ"); let filename = format!("{}.json", ts); let path = decisions_dir.join(&filename); - let decision_str = match decision.outcome { + let policy_results: Vec = result + .policy_results + .iter() + .zip(policy_hashes.iter()) + .map(|(r, hash)| PolicyResultRecord { + policy_path: r.policy_path.clone(), + policy_name: r.policy_name.clone(), + policy_version: r.policy_version.clone(), + policy_hash: hash.clone(), + outcome: outcome_str(r.outcome).to_string(), + matched_rule: r.matched_rule.clone(), + }) + .collect(); + + let combined_decision_str = match result.combined_decision.outcome { DecisionOutcome::Pass => "PASS", DecisionOutcome::RequireApproval => "REQUIRE_APPROVAL", DecisionOutcome::Block => "BLOCK", }; let artifact = DecisionArtifact { - policy_hash: policy_hash.to_string(), + artifact_version: DECISION_ARTIFACT_VERSION.to_string(), + geval_version: crate::GEVAL_VERSION.to_string(), + contract_name: result.contract_name.clone(), + contract_version: result.contract_version.clone(), + contract_hash: contract_hash.to_string(), + signals_name: signals_name.map(String::from), + signals_version: signals_version.map(String::from), signals_hash: signals_hash.to_string(), - decision: decision_str.to_string(), - matched_rule: decision.matched_rule.clone(), + combine_rule: result.combine_rule.to_string(), + policy_results, + combined_decision: combined_decision_str.to_string(), + combined_matched_rule: result.combined_decision.matched_rule.clone(), timestamp: ts.to_string(), approval, }; @@ -57,3 +115,11 @@ pub fn write_decision_artifact( std::fs::write(&path, json).with_context(|| format!("write {}", path.display()))?; Ok(path) } + +fn outcome_str(o: DecisionOutcome) -> &'static str { + match o { + DecisionOutcome::Pass => "PASS", + DecisionOutcome::RequireApproval => "REQUIRE_APPROVAL", + DecisionOutcome::Block => "BLOCK", + } +} diff --git a/geval/src/cli/commands.rs b/geval/src/cli/commands.rs index 847914c..cc4575d 100644 --- a/geval/src/cli/commands.rs +++ b/geval/src/cli/commands.rs @@ -1,4 +1,4 @@ -//! CLI commands: check, init, demo, approve, reject, explain, validate-policy. +//! CLI commands: check (contract), init, demo, approve, reject, explain, validate-contract. use anyhow::{Context, Result}; use clap::{Parser, Subcommand}; @@ -7,14 +7,18 @@ use std::path::PathBuf; use crate::approval::write_approval; use crate::artifact::write_decision_artifact; use crate::cli::{demo_ui::print_demo_report, init::run_init as do_init}; -use crate::evaluator::{evaluate, evaluate_with_trace, DecisionOutcome}; -use crate::explanation::explain_decision; -use crate::hashing::{hash_policy, hash_signals}; -use crate::policy::{parse_policy, parse_policy_str}; +use crate::contract::{ + load_contract_and_policies, run_contract, CombineRule, ContractDef, PolicyRef, +}; +use crate::evaluator::{evaluate_with_trace, DecisionOutcome}; +use crate::explanation::explain_contract_result; +use crate::hashing::{hash_contract_content, hash_policy, hash_signals}; +use crate::policy::parse_policy_str; use crate::signal_graph::SignalGraph; -use crate::signals::{load_signals, load_signals_from_reader}; +use crate::signals::load_signals_from_reader; /// Geval - decision orchestration engine for AI systems. +/// Contract = multiple policies evaluated together with a combination rule. #[derive(Parser)] #[command(name = "geval")] #[command(version)] @@ -26,24 +30,25 @@ pub struct Commands { #[derive(Subcommand)] pub enum Sub { - /// Evaluate signals against policy; exit 0=PASS, 1=REQUIRE_APPROVAL, 2=BLOCK. + /// Evaluate signals against a contract (multiple policies); exit 0=PASS, 1=REQUIRE_APPROVAL, 2=BLOCK. Check(CheckOpts), - /// Create a template folder (.geval by default) with sample signals and policy. Edit and run. + /// Create a template folder with contract and policies. Edit and run. Init(InitOpts), - /// Run a built-in example (no files needed). Use this to try Geval after downloading. + /// Run a built-in example (no files needed). Demo(DemoOpts), /// Record human approval (for REQUIRE_APPROVAL flow). Approve(ApproveOpts), /// Record human rejection. Reject(RejectOpts), - /// Print human-readable decision report. + /// Print human-readable decision report (contract + per-policy + combined). Explain(ExplainOpts), - /// Validate policy file syntax. - ValidatePolicy(ValidatePolicyOpts), + /// Validate contract file and all referenced policies. + ValidateContract(ValidateContractOpts), } -/// Built-in demo signals and policy (same as geval/examples/). const DEMO_SIGNALS_JSON: &str = r#"{ + "name": "demo-signals", + "version": "1.0.0", "signals": [ { "system": "support_agent", @@ -65,7 +70,9 @@ const DEMO_SIGNALS_JSON: &str = r#"{ ] }"#; -const DEMO_POLICY_YAML: &str = r#"policy: +const DEMO_POLICY_YAML: &str = r#"name: demo-policy +version: "1.0.0" +policy: environment: prod rules: - priority: 1 @@ -103,8 +110,8 @@ const DEMO_POLICY_YAML: &str = r#"policy: pub struct CheckOpts { #[arg(long, short = 's')] pub signals: PathBuf, - #[arg(long, short = 'p')] - pub policy: PathBuf, + #[arg(long, short = 'c')] + pub contract: PathBuf, #[arg(long, short = 'e', env = "GEVAL_ENV")] pub env: Option, #[arg(long)] @@ -135,25 +142,23 @@ pub struct RejectOpts { pub struct ExplainOpts { #[arg(long, short = 's')] pub signals: PathBuf, - #[arg(long, short = 'p')] - pub policy: PathBuf, + #[arg(long, short = 'c')] + pub contract: PathBuf, #[arg(long, short = 'e', env = "GEVAL_ENV")] pub env: Option, } #[derive(clap::Args)] -pub struct ValidatePolicyOpts { - pub policy: PathBuf, +pub struct ValidateContractOpts { + pub contract: PathBuf, #[arg(long)] pub json: bool, } #[derive(clap::Args)] pub struct InitOpts { - /// Directory to create (default: .geval). All template files go here; your project stays unchanged. #[arg(default_value = ".geval")] pub directory: PathBuf, - /// Overwrite existing signals.json and policy.yaml if they already exist. #[arg(long)] pub force: bool, } @@ -173,7 +178,7 @@ impl Commands { Sub::Approve(opts) => run_approve(&opts), Sub::Reject(opts) => run_reject(&opts), Sub::Explain(opts) => run_explain(&opts), - Sub::ValidatePolicy(opts) => run_validate_policy(&opts), + Sub::ValidateContract(opts) => run_validate_contract(&opts), } } } @@ -181,11 +186,11 @@ impl Commands { fn run_init(opts: &InitOpts) -> Result<()> { do_init(&opts.directory, opts.force).context("init")?; println!( - "Created {} with signals.json, policy.yaml, and README.md.", + "Created {} with contract.yaml, policies/, signals.json, and README.md.", opts.directory.display() ); println!( - "Edit the files, then run: geval check --signals {}/signals.json --policy {}/policy.yaml", + "Edit the files, then run: geval check --contract {}/contract.yaml --signals {}/signals.json", opts.directory.display(), opts.directory.display() ); @@ -197,20 +202,33 @@ fn run_demo(opts: &DemoOpts) -> Result<()> { let signals = load_signals_from_reader(DEMO_SIGNALS_JSON.as_bytes()).context("parse built-in signals")?; let graph = SignalGraph::build(&signals.signals); - let (decision, trace) = evaluate_with_trace(&policy, &graph); + let contract = ContractDef { + name: "demo".to_string(), + version: "1.0.0".to_string(), + combine: CombineRule::AllPass, + policies: vec![PolicyRef { + path: "demo.yaml".to_string(), + }], + }; + let result = run_contract(&contract, &[policy.clone()], &graph).context("run demo contract")?; + let (_, trace) = evaluate_with_trace(&policy, &graph); if opts.json { let out = serde_json::json!({ - "decision": outcome_str(decision.outcome), - "matched_rule": decision.matched_rule, - "reason": decision.reason, + "contract": result.contract_name, + "combined_decision": outcome_str(result.combined_decision.outcome), + "policy_results": result.policy_results.iter().map(|r| serde_json::json!({ + "policy_path": r.policy_path, + "outcome": outcome_str(r.outcome), + "matched_rule": r.matched_rule, + })).collect::>(), }); println!("{}", serde_json::to_string_pretty(&out)?); } else { - print_demo_report(&policy, &graph, &decision, &trace, Some("prod")); + print_demo_report(&policy, &graph, &result.combined_decision, &trace, Some("prod")); } - let code = match decision.outcome { + let code = match result.combined_decision.outcome { DecisionOutcome::Pass => 0, DecisionOutcome::RequireApproval => 1, DecisionOutcome::Block => 2, @@ -219,35 +237,50 @@ fn run_demo(opts: &DemoOpts) -> Result<()> { } fn run_check(opts: &CheckOpts) -> Result<()> { - let policy = parse_policy(&opts.policy).context("load policy")?; - let signals = load_signals(&opts.signals).context("load signals")?; + let (contract, policies) = + load_contract_and_policies(&opts.contract).context("load contract and policies")?; + let signals = crate::signals::load_signals(&opts.signals).context("load signals")?; let graph = SignalGraph::build(&signals.signals); - let decision = evaluate(&policy, &graph); - let policy_hash = hash_policy(&policy); + let result = run_contract(&contract, &policies, &graph).context("run contract")?; + + let contract_hash = hash_contract_content(&contract); + let policy_hashes: Vec = policies.iter().map(hash_policy).collect(); let signals_hash = hash_signals(&signals); + let cwd = std::env::current_dir().unwrap_or_else(|_| PathBuf::from(".")); let _ = write_decision_artifact( &cwd, - &policy_hash, + &result, + &contract_hash, + &policy_hashes, &signals_hash, - &decision, + signals.name.as_deref(), + signals.version.as_deref(), None, - ); + ) + .context("write decision artifact")?; if opts.json { let out = serde_json::json!({ - "decision": outcome_str(decision.outcome), - "matched_rule": decision.matched_rule, - "reason": decision.reason, + "contract": result.contract_name, + "combined_decision": outcome_str(result.combined_decision.outcome), + "combine_rule": result.combine_rule.to_string(), + "policy_results": result.policy_results.iter().map(|r| serde_json::json!({ + "policy_path": r.policy_path, + "outcome": outcome_str(r.outcome), + "matched_rule": r.matched_rule, + })).collect::>(), }); println!("{}", serde_json::to_string_pretty(&out)?); } else { - let env = opts.env.as_deref().or(policy.environment.as_deref()); - println!("{}", explain_decision(&policy, &graph, &decision, env)); + println!( + "{}", + explain_contract_result(&result, &graph, opts.env.as_deref()) + ); } - let code = match decision.outcome { + let code = match result.combined_decision.outcome { DecisionOutcome::Pass => 0, DecisionOutcome::RequireApproval => 1, DecisionOutcome::Block => 2, @@ -278,23 +311,45 @@ fn run_reject(opts: &RejectOpts) -> Result<()> { } fn run_explain(opts: &ExplainOpts) -> Result<()> { - let policy = parse_policy(&opts.policy).context("load policy")?; - let signals = load_signals(&opts.signals).context("load signals")?; + let (contract, policies) = + load_contract_and_policies(&opts.contract).context("load contract and policies")?; + let signals = crate::signals::load_signals(&opts.signals).context("load signals")?; let graph = SignalGraph::build(&signals.signals); - let decision = evaluate(&policy, &graph); - let env = opts.env.as_deref().or(policy.environment.as_deref()); - println!("{}", explain_decision(&policy, &graph, &decision, env)); + let result = run_contract(&contract, &policies, &graph).context("run contract")?; + println!( + "{}", + explain_contract_result(&result, &graph, opts.env.as_deref()) + ); Ok(()) } -fn run_validate_policy(opts: &ValidatePolicyOpts) -> Result<()> { - let policy = parse_policy(&opts.policy).context("validate policy")?; +fn run_validate_contract(opts: &ValidateContractOpts) -> Result<()> { + let (contract, policies) = + load_contract_and_policies(&opts.contract).context("load contract and policies")?; if opts.json { - println!("{}", serde_json::to_string_pretty(&policy)?); + let out = serde_json::json!({ + "name": contract.name, + "version": contract.version, + "combine": contract.combine.to_string(), + "policies": contract.policies.iter().map(|p| &p.path).collect::>(), + "policy_count": policies.len(), + }); + println!("{}", serde_json::to_string_pretty(&out)?); } else { - println!("Policy valid: {} rule(s)", policy.rules.len()); - if let Some(env) = &policy.environment { - println!("Environment: {}", env); + println!( + "Contract valid: {} (version {}), {} policy/policies, combine={}", + contract.name, + contract.version, + policies.len(), + contract.combine + ); + for (i, (pref, policy)) in contract.policies.iter().zip(policies.iter()).enumerate() { + println!( + " {}: {} ({} rule(s))", + i + 1, + pref.path, + policy.rules.len() + ); } } Ok(()) diff --git a/geval/src/cli/init.rs b/geval/src/cli/init.rs index 7d26783..82ecbfb 100644 --- a/geval/src/cli/init.rs +++ b/geval/src/cli/init.rs @@ -1,4 +1,4 @@ -//! `geval init` — create a .geval template in the current directory. +//! `geval init` — create a .geval template with a contract and multiple policies. //! Safe for existing codebases: only creates files inside the chosen directory (default .geval). use anyhow::{Context, Result}; @@ -6,6 +6,8 @@ use std::fs; use std::path::Path; const SIGNALS_TEMPLATE: &str = r#"{ + "name": "my-signals", + "version": "1.0.0", "signals": [ { "system": "my_app", @@ -36,40 +38,64 @@ const SIGNALS_TEMPLATE: &str = r#"{ } "#; -const POLICY_TEMPLATE: &str = r#"# Geval policy — your rules. Edit and add your own. -# Rules are evaluated in priority order (lower number first). First match wins. -# No match = PASS (allow). -# -# When: metric (required), optional: component, system, agent, step. -# operator: ">", "<", ">=", "<=", "==", or "presence" (no threshold; matches if metric exists, even without a value). -# threshold: number (for comparisons; not used for presence). -# Then: action: pass | block | require_approval. Optional: reason. +const CONTRACT_TEMPLATE: &str = r#"# Geval contract: multiple policies evaluated together. +# name + version identify this contract; bump version when you add/remove policies or change combine. +# combine: all_pass = PASS only if every policy passes; any_block_blocks = any BLOCK → overall BLOCK. +name: release-gate +version: "1.0.0" +combine: all_pass +policies: + - path: policies/security.yaml + - path: policies/quality.yaml +"#; + +const POLICY_SECURITY_TEMPLATE: &str = r#"# Security policy — block on safety violations. +name: security +version: "1.0.0" policy: environment: prod - rules: - priority: 1 - name: block_engagement_drop + name: block_high_hallucination when: - metric: engagement_drop + component: generator + metric: hallucination_rate operator: ">" - threshold: 0 + threshold: 0.05 then: action: block - reason: "Business engagement dropped" + reason: "Hallucination rate too high" - priority: 2 - name: block_high_hallucination + name: block_low_retrieval_quality when: - component: generator - metric: hallucination_rate + component: retrieval + metric: context_relevance + operator: "<" + threshold: 0.7 + then: + action: block + reason: "Retrieval quality below minimum" +"#; + +const POLICY_QUALITY_TEMPLATE: &str = r#"# Quality policy — business and quality gates. +name: quality +version: "1.0.0" +policy: + environment: prod + rules: + - priority: 1 + name: block_engagement_drop + when: + metric: engagement_drop operator: ">" - threshold: 0.05 + threshold: 0 then: action: block + reason: "Business engagement dropped" - - priority: 3 + - priority: 2 name: require_approval_low_retrieval when: component: retrieval @@ -79,15 +105,6 @@ policy: then: action: require_approval reason: "Retrieval quality below threshold" - - - priority: 4 - name: pass_high_accuracy - when: - metric: context_relevance - operator: ">=" - threshold: 0.9 - then: - action: pass "#; fn readme_content(dir: &Path) -> String { @@ -99,56 +116,63 @@ Created by `geval init`. Edit the files in this folder and run Geval from your p ## Files -- **signals.json** — Your data (metrics, scores). Add or change entries. Each entry can have: system, agent, component, step, metric, value, type. -- **policy.yaml** — Your rules. Order by priority; first matching rule wins. Actions: pass, block, require_approval. +- **contract.yaml** — Contract: name, version, combine rule, and list of policy paths. Bump version when you change policies or combine rule. +- **policies/** — Policy files (e.g. security.yaml, quality.yaml). Each has name, version, and rules. Paths in contract are relative to the contract file. +- **signals.json** — Your data (metrics, scores). Set name and version; bump version when the pipeline or schema changes. + +## Combine rules + +- **all_pass** — Overall PASS only if every policy returns PASS; any BLOCK → BLOCK; any REQUIRE_APPROVAL (and no BLOCK) → REQUIRE_APPROVAL. +- **any_block_blocks** — Any policy BLOCK → overall BLOCK; else any REQUIRE_APPROVAL → REQUIRE_APPROVAL; else PASS. ## Run -From the **project root** (parent of this folder): +From the **project root**: ```bash -geval check --signals {}/signals.json --policy {}/policy.yaml +geval check --contract {}/contract.yaml --signals {}/signals.json ``` -Explain why you got that result: +Explain: ```bash -geval explain --signals {}/signals.json --policy {}/policy.yaml +geval explain --contract {}/contract.yaml --signals {}/signals.json ``` -Validate your rules file: +Validate contract and all policies: ```bash -geval validate-policy {}/policy.yaml +geval validate-contract {}/contract.yaml ``` ## Approve / reject -If the result is REQUIRE_APPROVAL, record a decision: +If the result is REQUIRE_APPROVAL: ```bash geval approve --reason "Reviewed and approved" --output {}/approval.json -# or geval reject --reason "Needs more testing" --output {}/rejection.json ``` -Your codebase is unchanged except for this folder. Add these files to version control if you want to share rules with your team. +Add these files to version control to share the contract with your team. "#, dir_str, dir_str, dir_str, dir_str, dir_str, dir_str, dir_str ) } -/// Run `geval init`: create directory and template files. -/// If directory already has signals.json or policy.yaml and force is false, returns error. +/// Run `geval init`: create directory, contract, policies/, signals, README. pub fn run_init(dir: &Path, force: bool) -> Result<()> { + let contract_path = dir.join("contract.yaml"); let signals_path = dir.join("signals.json"); - let policy_path = dir.join("policy.yaml"); let readme_path = dir.join("README.md"); + let policies_dir = dir.join("policies"); + let security_path = policies_dir.join("security.yaml"); + let quality_path = policies_dir.join("quality.yaml"); if dir.exists() { + let has_contract = contract_path.exists(); let has_signals = signals_path.exists(); - let has_policy = policy_path.exists(); - if (has_signals || has_policy) && !force { + if (has_contract || has_signals) && !force { anyhow::bail!( "Directory {} already has template files. Use --force to overwrite.", dir.display() @@ -158,10 +182,17 @@ pub fn run_init(dir: &Path, force: bool) -> Result<()> { fs::create_dir_all(dir).with_context(|| format!("create directory {}", dir.display()))?; } + fs::create_dir_all(&policies_dir) + .with_context(|| format!("create {}", policies_dir.display()))?; + + fs::write(&contract_path, CONTRACT_TEMPLATE) + .with_context(|| format!("write {}", contract_path.display()))?; fs::write(&signals_path, SIGNALS_TEMPLATE) .with_context(|| format!("write {}", signals_path.display()))?; - fs::write(&policy_path, POLICY_TEMPLATE) - .with_context(|| format!("write {}", policy_path.display()))?; + fs::write(&security_path, POLICY_SECURITY_TEMPLATE) + .with_context(|| format!("write {}", security_path.display()))?; + fs::write(&quality_path, POLICY_QUALITY_TEMPLATE) + .with_context(|| format!("write {}", quality_path.display()))?; fs::write(&readme_path, readme_content(dir)) .with_context(|| format!("write {}", readme_path.display()))?; diff --git a/geval/src/contract/combine.rs b/geval/src/contract/combine.rs new file mode 100644 index 0000000..c272284 --- /dev/null +++ b/geval/src/contract/combine.rs @@ -0,0 +1,157 @@ +//! Combination rules: how multiple policy outcomes are merged into one contract decision. +//! +//! Extensible: add a new variant to `CombineRule` and implement the logic in `apply`. + +use crate::evaluator::DecisionOutcome; +use serde::{Deserialize, Serialize}; + +/// How to combine outcomes from multiple policies into a single contract decision. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Default, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum CombineRule { + #[default] + /// PASS only if every policy returns PASS. Otherwise: any BLOCK → BLOCK; else REQUIRE_APPROVAL. + AllPass, + + /// If any policy returns BLOCK → BLOCK; else if any REQUIRE_APPROVAL → REQUIRE_APPROVAL; else PASS. + AnyBlockBlocks, +} + +impl std::fmt::Display for CombineRule { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + CombineRule::AllPass => write!(f, "all_pass"), + CombineRule::AnyBlockBlocks => write!(f, "any_block_blocks"), + } + } +} + +impl std::str::FromStr for CombineRule { + type Err = String; + + fn from_str(s: &str) -> Result { + match s.trim().to_lowercase().as_str() { + "all_pass" => Ok(CombineRule::AllPass), + "any_block_blocks" => Ok(CombineRule::AnyBlockBlocks), + _ => Err(format!("unknown combine rule: {}", s)), + } + } +} + +/// Apply the combination rule to a slice of policy outcomes (order preserved). +/// Returns the single contract-level outcome. +pub fn apply_combine_rule(rule: CombineRule, outcomes: &[DecisionOutcome]) -> DecisionOutcome { + if outcomes.is_empty() { + return DecisionOutcome::Pass; + } + match rule { + CombineRule::AllPass => { + let any_block = outcomes.iter().any(|o| *o == DecisionOutcome::Block); + let any_approval = outcomes.iter().any(|o| *o == DecisionOutcome::RequireApproval); + if any_block { + DecisionOutcome::Block + } else if any_approval { + DecisionOutcome::RequireApproval + } else { + DecisionOutcome::Pass + } + } + CombineRule::AnyBlockBlocks => { + if outcomes.iter().any(|o| *o == DecisionOutcome::Block) { + DecisionOutcome::Block + } else if outcomes + .iter() + .any(|o| *o == DecisionOutcome::RequireApproval) + { + DecisionOutcome::RequireApproval + } else { + DecisionOutcome::Pass + } + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn all_pass_all_pass() { + let outcomes = [ + DecisionOutcome::Pass, + DecisionOutcome::Pass, + DecisionOutcome::Pass, + ]; + assert_eq!(apply_combine_rule(CombineRule::AllPass, &outcomes), DecisionOutcome::Pass); + } + + #[test] + fn all_pass_any_block() { + let outcomes = [ + DecisionOutcome::Pass, + DecisionOutcome::Block, + DecisionOutcome::Pass, + ]; + assert_eq!(apply_combine_rule(CombineRule::AllPass, &outcomes), DecisionOutcome::Block); + } + + #[test] + fn all_pass_any_approval_no_block() { + let outcomes = [ + DecisionOutcome::Pass, + DecisionOutcome::RequireApproval, + DecisionOutcome::Pass, + ]; + assert_eq!( + apply_combine_rule(CombineRule::AllPass, &outcomes), + DecisionOutcome::RequireApproval + ); + } + + #[test] + fn any_block_blocks_none() { + let outcomes = [DecisionOutcome::Pass, DecisionOutcome::Pass]; + assert_eq!( + apply_combine_rule(CombineRule::AnyBlockBlocks, &outcomes), + DecisionOutcome::Pass + ); + } + + #[test] + fn any_block_blocks_one_block() { + let outcomes = [ + DecisionOutcome::Pass, + DecisionOutcome::Block, + DecisionOutcome::RequireApproval, + ]; + assert_eq!( + apply_combine_rule(CombineRule::AnyBlockBlocks, &outcomes), + DecisionOutcome::Block + ); + } + + #[test] + fn any_block_blocks_approval_only() { + let outcomes = [ + DecisionOutcome::Pass, + DecisionOutcome::RequireApproval, + DecisionOutcome::Pass, + ]; + assert_eq!( + apply_combine_rule(CombineRule::AnyBlockBlocks, &outcomes), + DecisionOutcome::RequireApproval + ); + } + + #[test] + fn empty_outcomes_pass() { + assert_eq!( + apply_combine_rule(CombineRule::AllPass, &[]), + DecisionOutcome::Pass + ); + assert_eq!( + apply_combine_rule(CombineRule::AnyBlockBlocks, &[]), + DecisionOutcome::Pass + ); + } +} diff --git a/geval/src/contract/loader.rs b/geval/src/contract/loader.rs new file mode 100644 index 0000000..cf1ca0f --- /dev/null +++ b/geval/src/contract/loader.rs @@ -0,0 +1,120 @@ +//! Load contract from YAML and resolve policy paths. + +use anyhow::{Context, Result}; +use std::path::{Path, PathBuf}; + +use crate::contract::{ContractDef, PolicyRef}; +use crate::policy::{parse_policy, Policy}; + +/// Policy ref in YAML: either a string path or { path: "..." }. +#[derive(serde::Deserialize)] +#[serde(untagged)] +enum PolicyRefInput { + Path(String), + Obj { path: String }, +} + +/// Raw contract file shape (YAML). +#[derive(serde::Deserialize)] +struct ContractFile { + name: String, + version: String, + #[serde(default)] + combine: super::CombineRule, + policies: Vec, +} + +/// Load contract definition from a file path. +pub fn load_contract(path: &Path) -> Result { + let s = std::fs::read_to_string(path).with_context(|| format!("read contract file: {}", path.display()))?; + parse_contract_str(&s).with_context(|| format!("parse contract: {}", path.display())) +} + +/// Parse contract from a string (e.g. tests or inline). +pub fn parse_contract_str(s: &str) -> Result { + let _: serde_yaml::Value = serde_yaml::from_str(s).context("parse contract YAML")?; + let f: ContractFile = serde_yaml::from_str(s).context("invalid contract structure")?; + let policies: Vec = f + .policies + .into_iter() + .map(|p| match p { + PolicyRefInput::Path(s) => PolicyRef { path: s }, + PolicyRefInput::Obj { path } => PolicyRef { path }, + }) + .collect(); + let def = ContractDef { + name: f.name, + version: f.version, + combine: f.combine, + policies, + }; + def.validate().map_err(|e| anyhow::anyhow!("{}", e))?; + Ok(def) +} + +/// Resolve a policy path relative to the contract file's directory. +pub fn resolve_policy_path(contract_path: &Path, policy_ref: &PolicyRef) -> PathBuf { + let contract_dir = contract_path + .parent() + .unwrap_or_else(|| Path::new(".")); + contract_dir.join(&policy_ref.path) +} + +/// Load the contract and all its policies. Policy paths are resolved relative to the contract file. +pub fn load_contract_and_policies(contract_path: &Path) -> Result<(ContractDef, Vec)> { + let contract = load_contract(contract_path)?; + let mut policies = Vec::with_capacity(contract.policies.len()); + for pref in &contract.policies { + let resolved = resolve_policy_path(contract_path, pref); + let policy = parse_policy(&resolved) + .with_context(|| format!("load policy: {}", resolved.display()))?; + policies.push(policy); + } + Ok((contract, policies)) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn parse_contract_minimal() { + let yaml = r#" +name: release-gate +version: "1.0.0" +combine: all_pass +policies: + - path: security.yaml + - path: quality.yaml +"#; + let c = parse_contract_str(yaml).unwrap(); + assert_eq!(c.name, "release-gate"); + assert_eq!(c.version, "1.0.0"); + assert_eq!(c.policies.len(), 2); + assert_eq!(c.policies[0].path, "security.yaml"); + assert_eq!(c.policies[1].path, "quality.yaml"); + } + + #[test] + fn parse_contract_empty_policies_invalid() { + let yaml = r#" +name: empty +version: "1.0.0" +combine: any_block_blocks +policies: [] +"#; + assert!(parse_contract_str(yaml).is_err()); + } + + #[test] + fn parse_contract_default_combine() { + let yaml = r#" +name: one +version: "1.0.0" +policies: + - path: single.yaml +"#; + let c = parse_contract_str(yaml).unwrap(); + assert_eq!(c.combine, crate::contract::CombineRule::AllPass); + } +} diff --git a/geval/src/contract/mod.rs b/geval/src/contract/mod.rs new file mode 100644 index 0000000..1a33454 --- /dev/null +++ b/geval/src/contract/mod.rs @@ -0,0 +1,15 @@ +//! Contract: a named, versioned set of policies evaluated together with a combination rule. +//! +//! This is the core unit of evaluation in Geval. A contract references multiple policy files; +//! each policy is evaluated against the same signals; outcomes are combined (e.g. all_pass, any_block_blocks) +//! into a single decision. + +mod combine; +mod loader; +mod model; +mod runner; + +pub use combine::{apply_combine_rule, CombineRule}; +pub use loader::{load_contract, load_contract_and_policies, parse_contract_str, resolve_policy_path}; +pub use model::{ContractDef, PolicyRef}; +pub use runner::{run_contract, ContractResult, PolicyResult}; diff --git a/geval/src/contract/model.rs b/geval/src/contract/model.rs new file mode 100644 index 0000000..6a178ab --- /dev/null +++ b/geval/src/contract/model.rs @@ -0,0 +1,42 @@ +//! Contract model: a named, versioned set of policies and a combination rule. +//! +//! A contract is the unit of evaluation: load contract → load all policies → evaluate each → combine. + +use serde::{Deserialize, Serialize}; + +use crate::contract::CombineRule; + +/// Reference to a policy file (path relative to the contract file's directory). +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct PolicyRef { + /// Path to the policy YAML file (relative to contract file dir or absolute). + pub path: String, +} + +/// Contract definition: name, version, list of policy paths, and how to combine their outcomes. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct ContractDef { + /// Contract name (e.g. "release-gate"). Required for audit. + pub name: String, + + /// Contract version (e.g. "1.0.0"). Bump when you add/remove policies or change combine rule. + pub version: String, + + /// How to combine outcomes from the policies. + #[serde(default)] + pub combine: CombineRule, + + /// List of policy file paths (e.g. ["security.yaml", "quality.yaml"]). + /// Paths are resolved relative to the directory containing the contract file. + pub policies: Vec, +} + +impl ContractDef { + /// Validates that the contract has at least one policy. + pub fn validate(&self) -> Result<(), String> { + if self.policies.is_empty() { + return Err("contract must have at least one policy".to_string()); + } + Ok(()) + } +} diff --git a/geval/src/contract/runner.rs b/geval/src/contract/runner.rs new file mode 100644 index 0000000..74d0123 --- /dev/null +++ b/geval/src/contract/runner.rs @@ -0,0 +1,243 @@ +//! Run a contract: evaluate each policy against signals, then combine outcomes. + +use anyhow::Result; + +use crate::contract::{apply_combine_rule, ContractDef}; +use crate::evaluator::{evaluate, Decision, DecisionOutcome}; +use crate::policy::Policy; +use crate::signal_graph::SignalGraph; + +/// Result of evaluating one policy (for artifact and reporting). +#[derive(Debug, Clone)] +pub struct PolicyResult { + /// Policy file path (as in contract). + pub policy_path: String, + /// Policy name from the policy YAML (if set). + pub policy_name: Option, + /// Policy version from the policy YAML (if set). + pub policy_version: Option, + /// Outcome for this policy. + pub outcome: DecisionOutcome, + /// Matched rule name (if any). + pub matched_rule: Option, + /// Reason from the matched rule (if any). + pub reason: Option, +} + +/// Result of running a full contract: per-policy results and combined decision. +#[derive(Debug, Clone)] +pub struct ContractResult { + pub contract_name: String, + pub contract_version: String, + pub policy_results: Vec, + pub combined_decision: Decision, + pub combine_rule: crate::contract::CombineRule, +} + +/// Evaluate the contract: run each policy against the graph, then combine. +/// `policies` must be in the same order as `contract.policies`. +pub fn run_contract( + contract: &ContractDef, + policies: &[Policy], + graph: &SignalGraph, +) -> Result { + assert_eq!( + contract.policies.len(), + policies.len(), + "policies list must match contract" + ); + let mut policy_results = Vec::with_capacity(policies.len()); + let mut outcomes = Vec::with_capacity(policies.len()); + + for (pref, policy) in contract.policies.iter().zip(policies.iter()) { + let decision = evaluate(policy, graph); + outcomes.push(decision.outcome); + policy_results.push(PolicyResult { + policy_path: pref.path.clone(), + policy_name: policy.name.clone(), + policy_version: policy.version.clone(), + outcome: decision.outcome, + matched_rule: decision.matched_rule.clone(), + reason: decision.reason.clone(), + }); + } + + let combined_outcome = apply_combine_rule(contract.combine, &outcomes); + let combined_decision = combined_decision_from_results(&policy_results, combined_outcome); + + Ok(ContractResult { + contract_name: contract.name.clone(), + contract_version: contract.version.clone(), + policy_results, + combined_decision, + combine_rule: contract.combine, + }) +} + +/// Build the combined Decision (outcome + a representative matched_rule/reason from the first non-PASS policy). +fn combined_decision_from_results( + results: &[PolicyResult], + outcome: DecisionOutcome, +) -> Decision { + if outcome == DecisionOutcome::Pass { + return Decision { + outcome: DecisionOutcome::Pass, + matched_rule: None, + reason: None, + }; + } + let first_non_pass = results.iter().find(|r| r.outcome != DecisionOutcome::Pass); + match first_non_pass { + Some(r) => Decision { + outcome, + matched_rule: r.matched_rule.clone().map(|rule| format!("{}:{}", r.policy_path, rule)), + reason: r.reason.clone(), + }, + None => Decision { + outcome, + matched_rule: None, + reason: None, + }, + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::contract::{CombineRule, PolicyRef}; + use crate::policy::parse_policy_str; + use crate::signals::{Signal, SignalSet}; + use crate::signal_graph::SignalGraph; + + fn sig(component: Option<&str>, metric: &str, value: f64) -> Signal { + Signal { + system: None, + agent: None, + component: component.map(String::from), + step: None, + metric: Some(metric.to_string()), + value: Some(serde_json::json!(value)), + r#type: None, + } + } + + #[test] + fn run_contract_single_policy_pass() { + let contract = ContractDef { + name: "test".to_string(), + version: "1.0".to_string(), + combine: CombineRule::AllPass, + policies: vec![PolicyRef { + path: "p.yaml".to_string(), + }], + }; + let policy = parse_policy_str( + r#" +rules: + - priority: 1 + name: block_high + when: + metric: x + operator: ">" + threshold: 100 + then: + action: block +"#, + ) + .unwrap(); + let signals = SignalSet::new(vec![sig(None, "x", 1.0)]); + let graph = SignalGraph::build(&signals.signals); + let result = run_contract(&contract, &[policy], &graph).unwrap(); + assert_eq!(result.policy_results.len(), 1); + assert_eq!(result.policy_results[0].outcome, DecisionOutcome::Pass); + assert_eq!(result.combined_decision.outcome, DecisionOutcome::Pass); + } + + #[test] + fn run_contract_two_policies_all_pass_combined_block() { + let contract = ContractDef { + name: "test".to_string(), + version: "1.0".to_string(), + combine: CombineRule::AllPass, + policies: vec![ + PolicyRef { + path: "a.yaml".to_string(), + }, + PolicyRef { + path: "b.yaml".to_string(), + }, + ], + }; + let policy_a = parse_policy_str( + r#" +rules: + - priority: 1 + name: pass + when: + metric: x + operator: ">=" + threshold: 0 + then: + action: pass +"#, + ) + .unwrap(); + let policy_b = parse_policy_str( + r#" +rules: + - priority: 1 + name: block_low + when: + metric: y + operator: "<" + threshold: 0.5 + then: + action: block +"#, + ) + .unwrap(); + let signals = SignalSet::new(vec![sig(None, "x", 1.0), sig(None, "y", 0.3)]); + let graph = SignalGraph::build(&signals.signals); + let result = run_contract(&contract, &[policy_a, policy_b], &graph).unwrap(); + assert_eq!(result.policy_results[0].outcome, DecisionOutcome::Pass); + assert_eq!(result.policy_results[1].outcome, DecisionOutcome::Block); + assert_eq!(result.combined_decision.outcome, DecisionOutcome::Block); + } + + #[test] + fn run_contract_any_block_blocks() { + let contract = ContractDef { + name: "test".to_string(), + version: "1.0".to_string(), + combine: CombineRule::AnyBlockBlocks, + policies: vec![ + PolicyRef { + path: "a.yaml".to_string(), + }, + PolicyRef { + path: "b.yaml".to_string(), + }, + ], + }; + let policy_a = parse_policy_str( + r#"rules: [{ priority: 1, name: p, when: { metric: x, operator: ">", threshold: 10 }, then: { action: block } }]"#, + ) + .unwrap(); + let policy_b = parse_policy_str( + r#"rules: [{ priority: 1, name: q, when: { metric: y, operator: ">", threshold: 10 }, then: { action: pass } }]"#, + ) + .unwrap(); + let signals = SignalSet::new(vec![sig(None, "x", 1.0), sig(None, "y", 1.0)]); + let graph = SignalGraph::build(&signals.signals); + let result = run_contract(&contract, &[policy_a.clone(), policy_b.clone()], &graph).unwrap(); + assert_eq!(result.policy_results[0].outcome, DecisionOutcome::Pass); + assert_eq!(result.policy_results[1].outcome, DecisionOutcome::Pass); + assert_eq!(result.combined_decision.outcome, DecisionOutcome::Pass); + + let signals_block = SignalSet::new(vec![sig(None, "x", 20.0), sig(None, "y", 1.0)]); + let graph_block = SignalGraph::build(&signals_block.signals); + let result_block = run_contract(&contract, &[policy_a, policy_b], &graph_block).unwrap(); + assert_eq!(result_block.policy_results[0].outcome, DecisionOutcome::Block); + assert_eq!(result_block.combined_decision.outcome, DecisionOutcome::Block); + } +} diff --git a/geval/src/explanation/explain.rs b/geval/src/explanation/explain.rs index 042b6b7..e55d21e 100644 --- a/geval/src/explanation/explain.rs +++ b/geval/src/explanation/explain.rs @@ -1,11 +1,57 @@ //! Human-readable explanation of the decision (GEVAL DECISION REPORT). +use crate::contract::ContractResult; use crate::evaluator::{Decision, DecisionOutcome}; use crate::policy::Policy; use crate::signal_graph::SignalGraph; use std::fmt::Write; -/// Produce a text report suitable for CLI output. +/// Produce a contract-level report: contract name/version, signals, per-policy results, combined decision. +pub fn explain_contract_result( + result: &ContractResult, + graph: &SignalGraph, + _environment: Option<&str>, +) -> String { + let mut out = String::new(); + out.push_str("GEVAL DECISION REPORT (CONTRACT)\n"); + out.push_str("--------------------------------\n"); + let _ = writeln!(out, "Contract: {} @ {}", result.contract_name, result.contract_version); + let _ = writeln!(out, "Combine rule: {}", result.combine_rule); + out.push_str("\nSignals:\n"); + for s in &graph.signals { + let label = signal_label(s); + let value_str = value_str(s); + let _ = writeln!(out, " {} = {}", label, value_str); + } + out.push_str("\nPer-policy results:\n"); + for r in &result.policy_results { + let match_info = r + .matched_rule + .as_ref() + .map(|m| format!(" (matched: {})", m)) + .unwrap_or_default(); + let _ = writeln!( + out, + " {}: {}{}", + r.policy_path, + outcome_str(r.outcome), + match_info + ); + } + out.push_str("\nCombined decision:\n"); + let _ = writeln!(out, "{}", outcome_str(result.combined_decision.outcome)); + if let Some(ref rule) = result.combined_decision.matched_rule { + let _ = writeln!(out, "First non-PASS: {}", rule); + } + if let Some(ref reason) = result.combined_decision.reason { + out.push_str("\nReason:\n"); + let _ = writeln!(out, "{}", reason); + } + out.push_str("--------------------------------\n"); + out +} + +/// Produce a text report suitable for CLI output (single policy). pub fn explain_decision( _policy: &Policy, graph: &SignalGraph, diff --git a/geval/src/explanation/mod.rs b/geval/src/explanation/mod.rs index b5f2753..9db893f 100644 --- a/geval/src/explanation/mod.rs +++ b/geval/src/explanation/mod.rs @@ -1,3 +1,3 @@ mod explain; -pub use explain::explain_decision; +pub use explain::{explain_contract_result, explain_decision}; diff --git a/geval/src/hashing/mod.rs b/geval/src/hashing/mod.rs index 340f1c5..d05f29b 100644 --- a/geval/src/hashing/mod.rs +++ b/geval/src/hashing/mod.rs @@ -1,3 +1,3 @@ mod sha; -pub use sha::{hash_policy, hash_signals}; +pub use sha::{hash_contract_content, hash_policy, hash_signals}; diff --git a/geval/src/hashing/sha.rs b/geval/src/hashing/sha.rs index 9a02543..97dd31a 100644 --- a/geval/src/hashing/sha.rs +++ b/geval/src/hashing/sha.rs @@ -1,7 +1,15 @@ -//! SHA256 hashing for policy and signals (audit reproducibility). +//! SHA256 hashing for contract, policies, and signals (audit reproducibility). use sha2::{Digest, Sha256}; +/// Compute SHA256 hex digest of contract definition (name, version, combine, policy paths). +pub fn hash_contract_content(contract: &crate::contract::ContractDef) -> String { + let json = serde_json::to_string(contract).unwrap_or_default(); + let mut hasher = Sha256::new(); + hasher.update(json.as_bytes()); + format!("{:x}", hasher.finalize()) +} + /// Compute SHA256 hex digest of serialized policy. pub fn hash_policy(policy: &crate::policy::Policy) -> String { let json = serde_json::to_string(policy).unwrap_or_default(); diff --git a/geval/src/lib.rs b/geval/src/lib.rs index 7f36ca0..51862c7 100644 --- a/geval/src/lib.rs +++ b/geval/src/lib.rs @@ -3,9 +3,13 @@ //! Consumes signals (JSON), evaluates policy rules (YAML), and produces //! deterministic decisions: PASS, REQUIRE_APPROVAL, or BLOCK. +/// Binary version at compile time (for decision artifacts and audit). +pub const GEVAL_VERSION: &str = env!("CARGO_PKG_VERSION"); + pub mod approval; pub mod artifact; pub mod cli; +pub mod contract; pub mod evaluator; pub mod explanation; pub mod hashing; @@ -15,10 +19,14 @@ pub mod signal_graph; pub mod signals; pub use approval::{ApprovalArtifact, ApprovalOutcome, read_approval, write_approval}; -pub use artifact::write_decision_artifact; +pub use artifact::{write_decision_artifact, DECISION_ARTIFACT_VERSION}; +pub use contract::{ + load_contract, load_contract_and_policies, run_contract, CombineRule, ContractDef, ContractResult, + PolicyRef, PolicyResult, +}; pub use evaluator::{evaluate, Decision, DecisionOutcome}; pub use explanation::explain_decision; -pub use hashing::{hash_policy, hash_signals}; +pub use hashing::{hash_contract_content, hash_policy, hash_signals}; pub use policy::{parse_policy, parse_policy_str, Policy, Rule}; pub use signal_graph::SignalGraph; pub use signals::{load_signals, Signal, SignalSet}; diff --git a/geval/src/policy/model.rs b/geval/src/policy/model.rs index 7918908..9471e0a 100644 --- a/geval/src/policy/model.rs +++ b/geval/src/policy/model.rs @@ -87,9 +87,16 @@ pub struct Rule { pub then: RuleConsequence, } -/// Top-level policy: environment and ordered rules. +/// Top-level policy (contract): optional identity for audit; environment and ordered rules. +/// Name + version identify the "contract" so every decision is tied to a versioned policy. #[derive(Debug, Clone, Deserialize, Serialize)] pub struct Policy { + /// Contract/policy name (e.g. "release-gate"). For audit; no semantic effect. + #[serde(default)] + pub name: Option, + /// Contract/policy version (e.g. "1.0.0"). Bump when you change rules; every decision records this. + #[serde(default)] + pub version: Option, #[serde(default)] pub environment: Option, pub rules: Vec, diff --git a/geval/src/policy/parser.rs b/geval/src/policy/parser.rs index 01e98c2..69a90f4 100644 --- a/geval/src/policy/parser.rs +++ b/geval/src/policy/parser.rs @@ -8,6 +8,10 @@ use crate::policy::Policy; /// Policy file can have top-level "policy" wrapper or be the policy object directly. #[derive(serde::Deserialize)] struct PolicyFile { + #[serde(default)] + name: Option, + #[serde(default)] + version: Option, #[serde(default)] policy: Option, #[serde(default)] @@ -18,6 +22,10 @@ struct PolicyFile { #[derive(serde::Deserialize)] struct PolicyInner { + #[serde(default)] + name: Option, + #[serde(default)] + version: Option, #[serde(default)] environment: Option, #[serde(default)] @@ -31,11 +39,15 @@ fn parse_policy_yaml(s: &str) -> Result { if let Some(f) = wrapped { if let Some(inner) = f.policy { return Ok(Policy { + name: inner.name.or(f.name), + version: inner.version.or(f.version), environment: inner.environment.or(f.environment), rules: inner.rules.unwrap_or_else(Vec::new), }); } return Ok(Policy { + name: f.name, + version: f.version, environment: f.environment, rules: f.rules.unwrap_or_else(Vec::new), }); @@ -85,4 +97,27 @@ policy: assert_eq!(p.rules[0].then.action, Action::Block); assert_eq!(p.rules[0].when.operator, Some(Operator::GreaterThan)); } + + #[test] + fn test_parse_policy_with_name_and_version() { + let yaml = r#" +name: release-gate +version: "2.1.0" +policy: + environment: prod + rules: + - priority: 1 + name: block_bad + when: + metric: risk + operator: ">" + threshold: 0.5 + then: + action: block +"#; + let p = parse_policy_str(yaml).unwrap(); + assert_eq!(p.name.as_deref(), Some("release-gate")); + assert_eq!(p.version.as_deref(), Some("2.1.0")); + assert_eq!(p.rules.len(), 1); + } } diff --git a/geval/src/signals/loader.rs b/geval/src/signals/loader.rs index f4a6707..a500d52 100644 --- a/geval/src/signals/loader.rs +++ b/geval/src/signals/loader.rs @@ -39,23 +39,40 @@ pub struct Signal { pub r#type: Option, } -/// Top-level container: either { "signals": [...] } or raw array. +/// Top-level container: either { "name"?, "version"?, "signals": [...] } or raw array. #[derive(Debug, Deserialize)] #[serde(untagged)] enum SignalsInput { - Wrapped { signals: Vec }, + Wrapped { + #[serde(default)] + name: Option, + #[serde(default)] + version: Option, + signals: Vec, + }, Array(Vec), } /// Set of signals loaded from a file or reader. +/// Name and version identify the signals set for audit; bump version when the pipeline or schema changes. #[derive(Debug, Clone)] pub struct SignalSet { + pub name: Option, + pub version: Option, pub signals: Vec, } impl SignalSet { pub fn new(signals: Vec) -> Self { - Self { signals } + Self { + name: None, + version: None, + signals, + } + } + + pub fn with_identity(name: Option, version: Option, signals: Vec) -> Self { + Self { name, version, signals } } pub fn is_empty(&self) -> bool { @@ -77,17 +94,17 @@ pub fn load_signals(path: &Path) -> Result { pub fn load_signals_from_reader(rd: R) -> Result { let value: serde_json::Value = serde_json::from_reader(rd).context("parse signals JSON")?; - let signals = parse_signals_value(&value)?; - Ok(SignalSet::new(signals)) + parse_signals_value(&value) } -fn parse_signals_value(v: &serde_json::Value) -> Result> { +fn parse_signals_value(v: &serde_json::Value) -> Result { let input: SignalsInput = serde_json::from_value(v.clone()).context("invalid signals structure")?; - let signals = match input { - SignalsInput::Wrapped { signals } => signals, - SignalsInput::Array(signals) => signals, - }; - Ok(signals) + match input { + SignalsInput::Wrapped { name, version, signals } => { + Ok(SignalSet::with_identity(name, version, signals)) + } + SignalsInput::Array(signals) => Ok(SignalSet::new(signals)), + } } #[cfg(test)] @@ -109,4 +126,13 @@ mod tests { assert_eq!(set.len(), 2); assert_eq!(set.signals[1].component.as_deref(), Some("retrieval")); } + + #[test] + fn test_load_signals_with_version() { + let json = r#"{"version":"1.0","name":"ci-signals","signals":[{"metric":"x","value":0.5}]}"#; + let set = load_signals_from_reader(json.as_bytes()).unwrap(); + assert_eq!(set.version.as_deref(), Some("1.0")); + assert_eq!(set.name.as_deref(), Some("ci-signals")); + assert_eq!(set.len(), 1); + } }