Skip to content

Commit ed19e4e

Browse files
committed
docs(examples): update composite strict-or example to bun run
Entire-Checkpoint: 0034f597b72d
1 parent 4c2510a commit ed19e4e

4 files changed

Lines changed: 92 additions & 9 deletions

File tree

‎apps/web/src/content/docs/docs/graders/composite.mdx‎

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -100,15 +100,18 @@ Because this is an average, the final score is the fraction of passing children
100100

101101
### OR Logic (Strict)
102102

103-
For a strict OR, add a custom code-grader aggregator and return `1.0` when any child score passes:
103+
For a strict OR, add a custom code-grader aggregator and return `1.0` when any child score passes.
104+
105+
Composite aggregator execution accepts either a direct script path or a shell command.
106+
The `bun run` form is the recommended pattern:
104107

105108
```yaml
106109
assertions:
107110
- name: strict_or
108111
type: composite
109112
aggregator:
110113
type: code-grader
111-
path: ./scripts/or-aggregator.js
114+
path: bun run ../scripts/or-aggregator.js
112115
assertions:
113116
- name: mentions-paris
114117
type: contains
@@ -132,8 +135,7 @@ console.log(
132135
verdict: anyPassed ? 'pass' : 'fail',
133136
assertions: [{ text: `Any-or gate: ${anyPassed ? 'passed' : 'failed'}`, passed: anyPassed }],
134137
}),
135-
);
136-
```
138+
);
137139
```
138140

139141
### Code Grader Aggregator
@@ -143,7 +145,7 @@ Run a custom command to decide the final score based on all grader results:
143145
```yaml
144146
aggregator:
145147
type: code-grader
146-
path: node ./scripts/safety-gate.js
148+
path: bun run ./scripts/safety-gate.js
147149
cwd: ./graders # optional working directory
148150
```
149151

‎examples/features/composite/README.md‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,10 +13,13 @@ Demonstrates composite grader patterns for combining multiple evaluation criteri
1313

1414
```bash
1515
# From repository root
16-
bun agentv eval examples/features/composite/evals/dataset.eval.yaml
16+
bun agentv eval run examples/features/composite/evals/dataset.eval.yaml
17+
# Run only the strict-or script path example in dry-run (no live LLM targets required)
18+
bun agentv eval run examples/features/composite/evals/dataset.eval.yaml --test-id strict-or-local --dry-run
1719
```
1820

1921
## Key Files
2022

2123
- `evals/dataset.eval.yaml` - Test cases with composite grader patterns
24+
- `scripts/or-aggregator.js` - Strict OR aggregator script used by `composite` examples
2225
- `apps/web/src/content/docs/docs/graders/composite.mdx` - Detailed AND/OR and strict-OR composition guidance

‎examples/features/composite/evals/dataset.eval.yaml‎

Lines changed: 28 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -56,9 +56,34 @@ tests:
5656
prompt: ../prompts/technical-accuracy.md
5757
aggregator:
5858
type: code-grader
59-
path: node ../scripts/safety-gate-aggregator.js
59+
path: bun run ../scripts/safety-gate-aggregator.js
6060

61-
# Example 3: LLM Grader Aggregator
61+
# Example 3: Strict OR with a local code-grader aggregator
62+
- id: strict-or-local
63+
input:
64+
- role: user
65+
content: "Where is Paris?"
66+
expected_output:
67+
- role: assistant
68+
content: |
69+
Paris is the capital city of France.
70+
criteria: |
71+
The response should include either Paris or the phrase "capital of France".
72+
assertions:
73+
- name: strict_or
74+
type: composite
75+
assertions:
76+
- name: mentions-paris
77+
type: contains
78+
value: Paris
79+
- name: mentions-capital
80+
type: contains
81+
value: capital
82+
aggregator:
83+
type: code-grader
84+
path: bun run ../scripts/or-aggregator.js
85+
86+
# Example 4: LLM Grader Aggregator
6287
- id: llm-grader-conflict-resolution
6388
# Baseline note: aggregator may report minor omissions (score ~0.9).
6489
input:
@@ -84,7 +109,7 @@ tests:
84109
type: llm-grader
85110
prompt: ../prompts/conflict-resolution.md
86111

87-
# Example 4: Nested Composite Graders
112+
# Example 5: Nested Composite Graders
88113
- id: nested-composite
89114
input:
90115
- role: user
Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
const fs = require('node:fs');
2+
3+
function getScore(result) {
4+
if (result === null || typeof result !== 'object') {
5+
return 0;
6+
}
7+
8+
if (result.verdict === 'pass') {
9+
return 1;
10+
}
11+
12+
if (typeof result.verdict === 'string' && result.verdict === 'skip') {
13+
return 0;
14+
}
15+
16+
if (typeof result.score === 'number') {
17+
return result.score >= 0.5 ? 1 : 0;
18+
}
19+
20+
return 0;
21+
}
22+
23+
try {
24+
const input = JSON.parse(fs.readFileSync(0, 'utf8'));
25+
const results = Object.values(input.results ?? {});
26+
const anyPassed = results.some(getScore);
27+
28+
console.log(
29+
JSON.stringify({
30+
score: anyPassed ? 1 : 0,
31+
verdict: anyPassed ? 'pass' : 'fail',
32+
assertions: [
33+
{
34+
text: `Strict OR passed if any child passed: ${anyPassed ? 'true' : 'false'}`,
35+
passed: anyPassed,
36+
},
37+
],
38+
}),
39+
);
40+
} catch (error) {
41+
console.log(
42+
JSON.stringify({
43+
score: 0,
44+
verdict: 'fail',
45+
assertions: [
46+
{
47+
text: `Failed to evaluate strict OR: ${error instanceof Error ? error.message : String(error)}`,
48+
passed: false,
49+
},
50+
],
51+
}),
52+
);
53+
}

0 commit comments

Comments
 (0)