Summary
Add a dataset management and versioning experience to AgentGuard for evaluation corpora and golden sets, with dataset diff/merge, trace promotion, deduplication, and labeling workflow.
Problem to solve
Current golden/test datasets are file-based and hard to maintain at scale. Operators need lifecycle management, version history, promotion paths, review pipeline, and ability to manage eval coverage over time.
Proposed solution
- Web or CLI interface for managing datasets
- Promote traces to candidate sets, with dedupe/resolution
- Dataset diff, review, and merge tools
- Labeling workflows (edge, high-risk, regression cases, etc.)
- Dataset versioning and easy rollback
Alternatives considered
- Maintain datasets manually outside the platform
- Store only in Git; no operator review UX
Who benefits?
Priority / impact
- Data quality is the foundation of eval and regression gate maturity
- Confidence in change safety and release gates
Additional context
Inspiration: annotation/data management tools in NLP (Prodigy, LIT, Snorkel, etc.)
Summary
Add a dataset management and versioning experience to AgentGuard for evaluation corpora and golden sets, with dataset diff/merge, trace promotion, deduplication, and labeling workflow.
Problem to solve
Current golden/test datasets are file-based and hard to maintain at scale. Operators need lifecycle management, version history, promotion paths, review pipeline, and ability to manage eval coverage over time.
Proposed solution
Alternatives considered
Who benefits?
Priority / impact
Additional context
Inspiration: annotation/data management tools in NLP (Prodigy, LIT, Snorkel, etc.)