Skip to content

fix(compiler): guardrail retry→raise escalation (live loop iteration + custom coercion) - #316

Closed
ling-senpeng13 wants to merge 3 commits into
mainfrom
fix/guardrail-retry-escalation
Closed

fix(compiler): guardrail retry→raise escalation (live loop iteration + custom coercion)#316
ling-senpeng13 wants to merge 3 commits into
mainfrom
fix/guardrail-retry-escalation

Conversation

@ling-senpeng13

@ling-senpeng13 ling-senpeng13 commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

What

Fix custom OUTPUT guardrails with onFail=RETRY so they actually escalate to raise once maxRetries is exhausted — the agent now ends FAILED/TERMINATED instead of running to COMPLETED. Two compiler bugs:

  1. Wrong iteration reference path. The compiler read the live loop counter as ${<loop>.iteration}, but Conductor exposes {input, output} per task ref, so the counter lives under output. The bare reference resolves to null mid-loop on engines that use the {input,output} document shape (e.g. orkes' oss-core), silently disabling every iteration-based escalation. Use ${<loop>.output.iteration} in GuardrailCompiler (regex/llm/custom), TerminationCompiler, and the MultiAgentCompiler round-robin selector.

  2. Custom-guardrail normalize never escalated. customGuardrailNormalizeScript returned on_fail unchanged and compileCustomGuardrail passed neither iteration nor max_retries to the normalize INLINE. Added the retry→raise (and fix→raise) coercion the regex/llm scripts already have, and wire iteration (= ${<loop>.output.iteration}) + max_retries (= guard.getMaxRetries()) into the normalize step.

Why

The bug was invisible in AgentSpan's own CI because it only runs against the standalone server (upstream conductor-core), where the bare ${<loop>.iteration} reference does resolve to the live value — so escalation fired and the suite was green. On engines that expose the per-ref document as {input, output} (orkes oss-core), the same reference is null, so the custom guardrail returned retry forever and the agent hit maxTurnsCOMPLETED. ${<loop>.output.iteration} resolves correctly on both engines.

Surfaced by the AgentSpan SDK e2e suites against orkes-embedded: Suite8Guardrails.test_custom_guardrail_retry_escalation and Suite8bGuardrailsExtended.test_max_retries_escalation.

Changes

  • GuardrailCompiler.java — iteration ref → ${<loop>.output.iteration}; wire iteration + max_retries into the custom normalize INLINE.
  • JavaScriptBuilder.java — add the retry→raise/fix→raise coercion to customGuardrailNormalizeScript.
  • TerminationCompiler.java, MultiAgentCompiler.java — same .output.iteration reference fix (they were silently reading null too).
  • Tests: new GuardrailCompilerTest case asserting the normalize wiring; TerminationCompilerTest updated for the corrected reference.

Validation

  • Full module test suite green (671/0).
  • LLM-free engine probe: a DO_WHILE whose body is [normalize INLINE (coercion), route SWITCH → TERMINATE] escalates retry→raise at maxRetries and the workflow ends FAILED.
  • End-to-end against orkes-embedded + real openai/gpt-4o-mini: both guardrail-escalation tests pass; the workflow shows LLM_CHAT_COMPLETE COMPLETED → normalize on_fail=raiseTERMINATEFAILED.

Why ${<loop>.output.iteration} is safe on both engines

The fix does not trade one engine-specific behavior for another — it moves from an undocumented resolution path to the deliberately published one:

  • Upstream conductor-core (conductor-oss, standalone AgentSpan server): the DoWhile system task explicitly writes the live counter into its own output document on every iteration — DoWhile.java: doWhileTaskModel.addOutput("iteration", doWhileTaskModel.getIteration()) (and seeds iteration: 0 for the empty-list case). ${<loop>.output.iteration} reads exactly what the engine intentionally publishes.
  • Orkes oss-core: the per-task-ref document is {input, output}, so output.iteration is the only live location — verified by the engine probe in this PR (loop-body task observes 1, 2, 3).

The old bare ${<loop>.iteration} relied on incidental root-level task-model spillover in upstream's reference document — behavior that is not part of the documented expression contract (${taskRef.input|output.*} — see Task Inputs, which enumerates only the input/output task-level forms) and does not survive on orkes' engine. The DO_WHILE operator docs (and Orkes' equivalent, whose worked example uses ${..._loop_ref.output.iteration}) document output.iteration as the official way to read the loop counter. In other words: the previous form worked on OSS by accident; the new form works on both by contract.

Validation on the OSS side specifically: full module suite green, and the LLM-free DO_WHILE escalation probe in this PR runs against the standalone server (upstream engine) and escalates retry→raise at maxRetries as expected.

Sources:

Task Inputs — Conductor OSS Documentation
Wiring Parameters — Orkes Conductor Documentation
Do While — Conductor OSS Documentation
Do While — Orkes Conductor Documentation

ling-senpeng13 and others added 2 commits July 14, 2026 17:12
… + custom coercion)

Two compiler bugs made custom OUTPUT guardrails with onFail=RETRY never
escalate to raise after maxRetries — the agent ran to COMPLETED instead
of FAILED/TERMINATED (Suite8/Suite8b escalation asserts).

(a) Wrong iteration reference path. The compiler read the live loop
    counter as ${<loop>.iteration}, but Conductor exposes {input, output}
    per task ref, so the counter lives under output. The bare reference
    resolves to null mid-loop, silently disabling every iteration-based
    escalation. Use ${<loop>.output.iteration} in GuardrailCompiler
    (regex/llm/custom), TerminationCompiler, and the MultiAgentCompiler
    round-robin selector. Verified by an engine probe: a loop-body task
    reading ${loop.output.iteration} resolves to the live int 1,2,3;
    the bare form resolves to null.

(b) Custom-guardrail normalize never escalated. customGuardrailNormalizeScript
    returned on_fail unchanged and compileCustomGuardrail passed neither
    iteration nor max_retries. Add the retry->raise (and fix->raise)
    coercion the regex/llm scripts already have, and wire iteration
    (= ${<loop>.output.iteration}) + max_retries (= guard.getMaxRetries())
    into the normalize INLINE.

Validated end-to-end without an LLM: a DO_WHILE whose body is
[normalize INLINE (coercion), route SWITCH -> TERMINATE] escalates
retry->raise at max_retries and the workflow ends FAILED, not COMPLETED.
Full module suite green (703/703); added a regression test asserting the
normalize wiring.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The escalation coercion added for retry->raise also coerced fix->raise
unconditionally, which wrongly terminated custom `fix` guardrails that
provide a fixedOutput (they should apply the fix and complete, not raise).
This broke the Suite17 custom-fix cells (aout_custom_fix, tout_custom_fix:
expected COMPLETED with REDACTED output, got FAILED).

Make fix->raise conditional on fixedOutput being null/undefined — matching
the java worker's own rule (AgentRuntime: `if fix && fixedOutput == null ->
raise`). retry->raise escalation is unchanged, so Suite8 max_retries
escalation still works. Verified the script logic against fix-with-output,
fix-without-output, retry-at-max, and retry-below-max cases.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@ling-senpeng13
ling-senpeng13 requested a review from v1r3n July 16, 2026 20:46
@ling-senpeng13 ling-senpeng13 self-assigned this Jul 23, 2026
@ling-senpeng13
ling-senpeng13 deleted the fix/guardrail-retry-escalation branch July 28, 2026 16:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant