fix(compiler): guardrail retry→raise escalation (live loop iteration + custom coercion) - #316
Closed
ling-senpeng13 wants to merge 3 commits into
Closed
fix(compiler): guardrail retry→raise escalation (live loop iteration + custom coercion)#316ling-senpeng13 wants to merge 3 commits into
ling-senpeng13 wants to merge 3 commits into
Conversation
… + custom coercion)
Two compiler bugs made custom OUTPUT guardrails with onFail=RETRY never
escalate to raise after maxRetries — the agent ran to COMPLETED instead
of FAILED/TERMINATED (Suite8/Suite8b escalation asserts).
(a) Wrong iteration reference path. The compiler read the live loop
counter as ${<loop>.iteration}, but Conductor exposes {input, output}
per task ref, so the counter lives under output. The bare reference
resolves to null mid-loop, silently disabling every iteration-based
escalation. Use ${<loop>.output.iteration} in GuardrailCompiler
(regex/llm/custom), TerminationCompiler, and the MultiAgentCompiler
round-robin selector. Verified by an engine probe: a loop-body task
reading ${loop.output.iteration} resolves to the live int 1,2,3;
the bare form resolves to null.
(b) Custom-guardrail normalize never escalated. customGuardrailNormalizeScript
returned on_fail unchanged and compileCustomGuardrail passed neither
iteration nor max_retries. Add the retry->raise (and fix->raise)
coercion the regex/llm scripts already have, and wire iteration
(= ${<loop>.output.iteration}) + max_retries (= guard.getMaxRetries())
into the normalize INLINE.
Validated end-to-end without an LLM: a DO_WHILE whose body is
[normalize INLINE (coercion), route SWITCH -> TERMINATE] escalates
retry->raise at max_retries and the workflow ends FAILED, not COMPLETED.
Full module suite green (703/703); added a regression test asserting the
normalize wiring.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The escalation coercion added for retry->raise also coerced fix->raise unconditionally, which wrongly terminated custom `fix` guardrails that provide a fixedOutput (they should apply the fix and complete, not raise). This broke the Suite17 custom-fix cells (aout_custom_fix, tout_custom_fix: expected COMPLETED with REDACTED output, got FAILED). Make fix->raise conditional on fixedOutput being null/undefined — matching the java worker's own rule (AgentRuntime: `if fix && fixedOutput == null -> raise`). retry->raise escalation is unchanged, so Suite8 max_retries escalation still works. Verified the script logic against fix-with-output, fix-without-output, retry-at-max, and retry-below-max cases. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Fix custom OUTPUT guardrails with
onFail=RETRYso they actually escalate toraiseoncemaxRetriesis exhausted — the agent now endsFAILED/TERMINATEDinstead of running toCOMPLETED. Two compiler bugs:Wrong iteration reference path. The compiler read the live loop counter as
${<loop>.iteration}, but Conductor exposes{input, output}per task ref, so the counter lives underoutput. The bare reference resolves to null mid-loop on engines that use the{input,output}document shape (e.g. orkes'oss-core), silently disabling every iteration-based escalation. Use${<loop>.output.iteration}inGuardrailCompiler(regex/llm/custom),TerminationCompiler, and theMultiAgentCompilerround-robin selector.Custom-guardrail normalize never escalated.
customGuardrailNormalizeScriptreturnedon_failunchanged andcompileCustomGuardrailpassed neitheriterationnormax_retriesto the normalize INLINE. Added theretry→raise(andfix→raise) coercion the regex/llm scripts already have, and wireiteration(=${<loop>.output.iteration}) +max_retries(=guard.getMaxRetries()) into the normalize step.Why
The bug was invisible in AgentSpan's own CI because it only runs against the standalone server (upstream conductor-core), where the bare
${<loop>.iteration}reference does resolve to the live value — so escalation fired and the suite was green. On engines that expose the per-ref document as{input, output}(orkesoss-core), the same reference isnull, so the custom guardrail returnedretryforever and the agent hitmaxTurns→COMPLETED.${<loop>.output.iteration}resolves correctly on both engines.Surfaced by the AgentSpan SDK e2e suites against orkes-embedded:
Suite8Guardrails.test_custom_guardrail_retry_escalationandSuite8bGuardrailsExtended.test_max_retries_escalation.Changes
GuardrailCompiler.java— iteration ref →${<loop>.output.iteration}; wireiteration+max_retriesinto the custom normalize INLINE.JavaScriptBuilder.java— add theretry→raise/fix→raisecoercion tocustomGuardrailNormalizeScript.TerminationCompiler.java,MultiAgentCompiler.java— same.output.iterationreference fix (they were silently reading null too).GuardrailCompilerTestcase asserting the normalize wiring;TerminationCompilerTestupdated for the corrected reference.Validation
DO_WHILEwhose body is[normalize INLINE (coercion), route SWITCH → TERMINATE]escalatesretry→raiseatmaxRetriesand the workflow endsFAILED.openai/gpt-4o-mini: both guardrail-escalation tests pass; the workflow showsLLM_CHAT_COMPLETECOMPLETED → normalizeon_fail=raise→TERMINATE→FAILED.Why
${<loop>.output.iteration}is safe on both enginesThe fix does not trade one engine-specific behavior for another — it moves from an undocumented resolution path to the deliberately published one:
DoWhilesystem task explicitly writes the live counter into its own output document on every iteration —DoWhile.java:doWhileTaskModel.addOutput("iteration", doWhileTaskModel.getIteration())(and seedsiteration: 0for the empty-list case).${<loop>.output.iteration}reads exactly what the engine intentionally publishes.oss-core: the per-task-ref document is{input, output}, sooutput.iterationis the only live location — verified by the engine probe in this PR (loop-body task observes 1, 2, 3).The old bare
${<loop>.iteration}relied on incidental root-level task-model spillover in upstream's reference document — behavior that is not part of the documented expression contract (${taskRef.input|output.*}— see Task Inputs, which enumerates only theinput/outputtask-level forms) and does not survive on orkes' engine. The DO_WHILE operator docs (and Orkes' equivalent, whose worked example uses${..._loop_ref.output.iteration}) documentoutput.iterationas the official way to read the loop counter. In other words: the previous form worked on OSS by accident; the new form works on both by contract.Validation on the OSS side specifically: full module suite green, and the LLM-free
DO_WHILEescalation probe in this PR runs against the standalone server (upstream engine) and escalatesretry→raiseatmaxRetriesas expected.Sources:
Task Inputs — Conductor OSS Documentation
Wiring Parameters — Orkes Conductor Documentation
Do While — Conductor OSS Documentation
Do While — Orkes Conductor Documentation