fix(agentic): tolerate LLM call failures in attempt_answer and ensure trace completion - #62
Closed
suguanYang wants to merge 2 commits into
Closed
suguanYang wants to merge 2 commits into
suguanYang wants to merge 2 commits into
Conversation
… trace completion - Return NOT_FOUND instead of re-raising when text LLM fails in attempt_answer - Wrap VLM-to-text fallback in its own try/except to handle double-failure - Catch non-budget exceptions around attempt_answer in orchestrator, degrade to llm_error - Wrap Final Assembly in try/finally so trace.complete() always runs - Return degraded AgenticResult on unexpected exceptions instead of crashing
- Text LLM failure returns NOT_FOUND instead of crashing - VLM + text LLM fallback double-failure returns NOT_FOUND - BudgetExceeded in fallback still propagates Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
attempt_answerno longer crashes on LLM errors — both text LLM failures and VLM+fallback double-failures now returnNOT_FOUNDinstead of propagating exceptionsattempt_answerand degrades gracefully withstop_reason='llm_error'trace.complete()is now guaranteed viatry/finallyin Final Assembly, preventing orphaned DB rowsrun()lives up to its docstring — "Never raises" is now actually true; unexpected exceptions produce a degradedAgenticResultRoot cause
Trace
019e2432a3f546cba83fdfb88cab66d6showed a qwen-vl-plus 400 error (The image format is illegal and cannot be opened) cascading into INSERT failures and an unrecoverable pipeline crash. The pipeline already handles LLM failures gracefully inkg_document_select,navigate_step, anddiscovery_select_step—attempt_answerwas the inconsistent outlier.Test plan
test_agentic_answer_policy_contract.py019e2432a3f546cba83fdfb88cab66d6type errors no longer crash the pipelinestop_reason='llm_error'