feat: add pre/post call hooks to ToolSimulator (#167)#168
Open
kaghatim wants to merge 1 commit intostrands-agents:mainfrom
Open
feat: add pre/post call hooks to ToolSimulator (#167)#168kaghatim wants to merge 1 commit intostrands-agents:mainfrom
kaghatim wants to merge 1 commit intostrands-agents:mainfrom
Conversation
Add optional pre_call_hook and post_call_hook parameters to ToolSimulator for fault injection and response modification during tool simulation. - pre_call_hook: intercepts before LLM call, can short-circuit with a custom response (e.g., rate limit errors, timeouts) - post_call_hook: modifies LLM-generated response before caching (e.g., inject metadata, corrupt fields for robustness testing) Hooks receive typed dataclass events (PreCallHookEvent, PostCallHookEvent) following the Strands SDK hook pattern for extensibility. Closes strands-agents#167
cfdcc34 to
2314f7f
Compare
Contributor
|
Thanks for creating this PR @kaghatim! Have you considered taking the Plugin approach? I think that would be cleaner. You would pass in the plugin here: https://github.com/kaghatim/evals/blob/2314f7f66d6b33c0bb72b8301b65296a310a07fd/src/strands_evals/simulation/tool_simulator.py#L243-L250 and you would do: simulator = ToolSimulator(
model=model,
plugin=my_hook_plugin, <=== new
)See https://strandsagents.com/docs/user-guide/concepts/plugins/ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Add optional
pre_call_hookandpost_call_hookparameters toToolSimulator.__init__for fault injection and response modification during tool simulation.pre_call_hook: invoked before the LLM generates a response. Receives aPreCallHookEventdataclass. Returns a dict to short-circuit the LLM call, orNoneto proceed normally. Short-circuited responses are cached in the state registry.post_call_hook: invoked after the LLM generates a response but before caching. Receives aPostCallHookEventdataclass. Returns a (possibly modified) response dict.Hook events follow the Strands SDK pattern (
@dataclassevents, before/after separation) and live intypes/simulation/hook_events.py.Both hooks are optional and default to
None, making this fully backward compatible.Related Issues
Closes #167
Documentation PR
N/A - No new docs needed for this change. Hook usage is documented in the
__init__docstring.Type of Change
New feature
Testing
hatch test tests/strands_evals/simulation/test_tool_simulator.pyhatch run prepareChecklist
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.