Commit 3fc5fcf
- Hard churn confirmed OK (not yet in production) — clean break, no aliases.
- Simplify: collapse tests[].input into prompts + vars (one way to express
what's sent to the target); input_files stays as prompt-content.
- Runner: laptop-first, no DB store. Instance expansion + pass@k + infra-only
retry (margin ideas), but a simple worker pool; resumability via index.jsonl
(--rerun-failed); workspace = shared or reset-based pool, not per-instance
containers.
- nunjucks confirmed to meet inline text+var mixing; {{var}} for vars,
${ENV} for env.
- Redteam + unimplemented exotic assertions: treated as unrecognized fields
(future scope), not stubbed. `similar` in (needs embeddings provider).
- test_id: layered identity — content-hash (content), author tag/metadata
(governance/trend, Dashboard keys on this), description/vars (display).
- LLM judge type name = `llm-rubric`; default grading adopts skeptical
evidence-by-path judge (opt-out via explicit prompt); documented what changes.
- New §9: PR #1592 is the extensions/workspace implementation slice — reasonable,
mergeable after 4 amendments (chiefly vars.workspace + built-in agentv: scheme).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 351dd2d commit 3fc5fcf
1 file changed
Lines changed: 47 additions & 22 deletions
0 commit comments