feat: add EvaluationPlugin for agent invocation evaluation and retry#5
Open
feat: add EvaluationPlugin for agent invocation evaluation and retry#5
Conversation
Introduce an EvaluationPlugin that hooks into agent invocations to evaluate outputs against expected results and automatically retries with improved system prompts on failure. - Add EvaluationPlugin class that wraps agent __call__ to intercept invocations, run evaluators, and retry with LLM-suggested prompt improvements when evaluations fail - Add improvement suggestion prompt template for generating better system prompts based on evaluation feedback - Add comprehensive test suite covering plugin initialization, wrapping, evaluation execution, retry logic, and edge cases - Update ruff config to ignore line-length in plugin prompt templates
afarntrog
had a problem deploying
to
auto-approve
GitHub Actions
Failure
afarntrog
had a problem deploying
to
auto-approve
GitHub Actions
Failure
- Add comprehensive docstrings to all methods in EvaluationPlugin - Change _suggest_improvements to return ImprovementSuggestion instead of str - Add debug logging with reasoning when applying improved system prompt - Replace Union[X, Y] with modern X | Y syntax - Use dict default in kwargs.get() instead of
afarntrog
had a problem deploying
to
auto-approve
GitHub Actions
Failure
- Export new module from the package's public API - Bump minimum strands-agents dependency from 1.0.0 to 1.28.0 to support functionality required by the plugins module
afarntrog
had a problem deploying
to
auto-approve
GitHub Actions
Failure
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Introduce an EvaluationPlugin that hooks into agent invocations to evaluate outputs against expected results and automatically retries with improved system prompts on failure.
Description
Related Issues
Documentation PR
Type of Change
Bug fix
New feature
Breaking change
Documentation update
Other (please describe):
Testing
How have you tested the change? Verify that the changes do not break functionality or introduce warnings in consuming repositories: agents-docs, agents-tools, agents-cli
hatch run prepareChecklist
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.