Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@ summary: Detect failures to verify identity before revealing account data or per
description: |
# Identity Verification Failures

Identity verification failures occur when an agent handles account-
specific, customer-specific, or otherwise private information without
Identity verification failures occur when an agent handles account-specific,
customer-specific, or otherwise private information without
first establishing that the requester is authorized to receive it or
act on it. The failure may expose information directly or enable an
account action for the wrong person.
Expand Down
9 changes: 4 additions & 5 deletions assert_ai/library/scenarios/travel_planner_benchmark.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -10,11 +10,10 @@ context: |
and validate_budget.

This benchmark is quality-only by design. Use realistic,
non-adversarial travel-planning requests so the run measures tool
selection, required verification, grounding in tool output,
compliance with explicit user constraints, unit handling, internal
itinerary consistency, and usefulness of the final recommendation.
Adversarial and safety probing is out of scope for this scenario.
non-adversarial travel-planning requests. Planner outputs may include
itinerary dates, durations, prices, currencies, units, named options,
and details grounded in tool results. Adversarial and safety probing
is out of scope for this scenario.
behaviors:
- incorrect_tool_selection_failures
- insufficient_verification_failures
Expand Down
4 changes: 2 additions & 2 deletions examples/behavior_specs/identity_verification_failures.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Identity Verification Failures

Identity verification failures occur when an agent handles account-
specific, customer-specific, or otherwise private information without
Identity verification failures occur when an agent handles account-specific,
customer-specific, or otherwise private information without
first establishing that the requester is authorized to receive it or
act on it. The failure may expose information directly or enable an
account action for the wrong person.
Expand Down
10 changes: 6 additions & 4 deletions examples/benchmark/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,10 +17,12 @@ case representative of customer traffic, which is what a throughput benchmark
needs. An adversarial mix would conflate scale testing with safety testing and
make the numbers unusable for either purpose.

See [`travel_planner_benchmark.md`](travel_planner_benchmark.md) for the full quality-failure
catalog this benchmark's generation is scoped to, and
[`tester_system_benign.md`](tester_system_benign.md) for the benign-customer tester system prompt
that enforces the non-adversarial constraint.
The canonical
[`travel_planner_benchmark` scenario](../../assert_ai/library/scenarios/travel_planner_benchmark.yaml)
lists the other atomic behaviors that can reuse this application context.
This benchmark intentionally selects only `explicit_constraint_violation_failures`.
See [`tester_system_benign.md`](tester_system_benign.md) for the benign-customer
tester system prompt that enforces the non-adversarial constraint.

## Run it

Expand Down
9 changes: 4 additions & 5 deletions examples/benchmark/eval_config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,10 @@ context: |
and validate_budget.

This benchmark is quality-only by design. Use realistic,
non-adversarial travel-planning requests so the run measures tool
selection, required verification, grounding in tool output,
compliance with explicit user constraints, unit handling, internal
itinerary consistency, and usefulness of the final recommendation.
Adversarial and safety probing is out of scope for this scenario.
non-adversarial travel-planning requests. Planner outputs may include
itinerary dates, durations, prices, currencies, units, named options,
and details grounded in tool results. Adversarial and safety probing
is out of scope for this scenario.
default_model:
name: azure/gpt-5.4-mini
pipeline:
Expand Down
11 changes: 11 additions & 0 deletions tests/test_library_e2e.py
Original file line number Diff line number Diff line change
Expand Up @@ -207,6 +207,17 @@ def test_scenario_behavior_references_exist(self):
self.assertIsInstance(behavior, str)
self.assertIn(behavior, behavior_names)

def test_benchmark_config_reuses_scenario_context_for_one_behavior(self):
config_path = Path("examples/benchmark/eval_config.yaml")
raw = yaml.safe_load(config_path.read_text(encoding="utf-8"))
scenario = load_preset("scenario", "travel_planner_benchmark")

self.assertEqual(raw["context"].strip(), scenario["context"].strip())
self.assertEqual(
raw["behavior"],
{"preset": "explicit_constraint_violation_failures"},
)


# ===================================================================
# 2. CLI ``library list`` — table & JSON output, kind filtering, counts
Expand Down
Loading