Repository navigation
Conversation
Add conformance/core/agent_function_routing.yaml, whose route_agent_function cases evaluate function calls (as bindings, check rules and action function calls) on a v1.0 surface created through a message processor, answer the callAgentFunction messages it sends, and check the outbound payloads, values and reported errors. The action is described in conformance_schema.json and the README, and implemented in the TypeScript, Python, Dart and Swift conformance runners. - dart-core: report an agent error or timeout as EXPRESSION_ERROR instead of EXECUTION_ERROR, as the other SDKs do. - web_core: apply the default of an optional argument of a function loaded with Catalog.fromJson, so schema-only functions are sent with their defaults. Known divergences: Dart cannot declare schema-only functions, and the TypeScript and Dart cores run a local function before an agent call in its arguments has settled. Signed-off-by: Nan Yu <nanyu@google.com>
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a new conformance test suite (route_agent_function) to verify the fallback routing of function calls to the agent with callAgentFunction. It implements the corresponding test runners across the Dart, Python, Swift, and TypeScript SDKs. Additionally, it updates the Dart SDK to report agent errors and timeouts as EXPRESSION_ERROR instead of EXECUTION_ERROR for consistency with other SDKs, and updates the TypeScript SDK to correctly apply default values for missing optional function arguments. A review comment identified an unused variable atLeastV10 in the Swift conformance test implementation that should be removed.
…yright year Remove the unused atLeastV10 in the Swift routing conformance runner, and use the copyright year required by the license check in the new files. Signed-off-by: Nan Yu <nanyu@google.com>
…nction-routing Bring in the web_core and Dart fixes that make a binding wait for a pending agent argument, and remove the known-divergence entries for test_agent_routing_agent_result_feeds_local_function from the web_core and Dart conformance runners and the README. Signed-off-by: Nan Yu <nanyu@google.com>
…-agent-function-routing
…-agent-function-routing
…routing' into test/conformance-agent-function-routing Signed-off-by: Nan Yu <nanyu@google.com> # Conflicts: # swift/core/Sources/A2UICore/Resolution/NodeResolver.swift # swift/core/Sources/A2UICore/State/DataContext.swift
…-agent-function-routing
…lls, and check results Signed-off-by: Nan Yu <nanyu@google.com>
…' into HEAD Signed-off-by: Nan Yu <nanyu@google.com>
…d pin failure eviction Signed-off-by: Nan Yu <nanyu@google.com>
…' into test/conformance-agent-function-routing
…' into test/conformance-agent-function-routing Signed-off-by: Nan Yu <nanyu@google.com>
gspencergoog
left a comment
gspencergoog
left a comment
There was a problem hiding this comment.
This change adds the shared core/agent_function_routing.yaml conformance suite (26 cases across v0.9 and v1.0), extends conformance/conformance_schema.json and conformance/README.md with action: route_agent_function, implements runners in Dart, Python, Swift, and TypeScript, and updates TypeScript Catalog.fromJson to apply optional argument defaults. Four high-severity issues need attention: Catalog.fromJson still drops defaults on nested object and array arguments (H1), the Dart runner crashes on catalogless check or action cases (H2), RoutingCatalog.agentFunctions omits returnType and allowedCallers (H3), and conformance_harness_implementation_guide.md is stale (H4).
🤖 Agent-generated
| for (const [name, field] of Object.entries(shape)) { | ||
| if (field instanceof z.ZodOptional && field.unwrap() instanceof z.ZodDefault) { | ||
| shape[name] = field.unwrap(); | ||
| } | ||
| } |
There was a problem hiding this comment.
convertFunctionArgsJsonSchemaToZod unwraps ZodOptional around ZodDefault only on top-level shape entries. Nested object and array schemas keep ZodOptional outside ZodDefault, returning undefined on missing keys. Omitted nested properties with JSON Schema default values resolve to undefined instead of materializing their defaults in outbound callAgentFunction payloads.
| for (const [name, field] of Object.entries(shape)) { | |
| if (field instanceof z.ZodOptional && field.unwrap() instanceof z.ZodDefault) { | |
| shape[name] = field.unwrap(); | |
| } | |
| } | |
| const unwrapOptionalDefaults = (schema: z.ZodTypeAny): z.ZodTypeAny => { | |
| if (schema instanceof z.ZodOptional) { | |
| const inner = unwrapOptionalDefaults(schema.unwrap()); | |
| return inner instanceof z.ZodDefault | |
| ? inner | |
| : new z.ZodOptional({...schema._def, innerType: inner}); | |
| } | |
| if (schema instanceof z.ZodDefault) { | |
| return new z.ZodDefault({ | |
| ...schema._def, | |
| innerType: unwrapOptionalDefaults(schema._def.innerType), | |
| }); | |
| } | |
| if (schema instanceof z.ZodObject) { | |
| const nextShape = Object.fromEntries( | |
| Object.entries(schema.shape).map(([k, v]) => [ | |
| k, | |
| unwrapOptionalDefaults(v as z.ZodTypeAny), | |
| ]), | |
| ); | |
| return new z.ZodObject({...schema._def, shape: () => nextShape}); | |
| } | |
| if (schema instanceof z.ZodArray) { | |
| return new z.ZodArray({...schema._def, type: unwrapOptionalDefaults(schema.element)}); | |
| } | |
| return schema; | |
| }; | |
| for (const [name, field] of Object.entries(shape)) { | |
| shape[name] = unwrapOptionalDefaults(field); | |
| } |
🤖 Agent-generated
| final component = ComponentModel(id, 'Probe', { | ||
| if (kind == 'check') | ||
| 'checks': [ | ||
| {'condition': call, 'message': 'Check failed'}, | ||
| ] | ||
| else | ||
| 'action': {'functionCall': call}, | ||
| }); |
There was a problem hiding this comment.
When surface.catalogId is omitted, ComponentModel(id, 'Probe', ...) leaves catalog null, causing GenericBinder to throw A2uiCatalogResolutionError before evaluating call. Swift avoids this by setting the first catalog ID on synthetic Probe components. Any check or action step on a catalogless v1.0 surface crashes the Dart runner instead of routing calls or reporting CATALOG_ERROR.
| final component = ComponentModel(id, 'Probe', { | |
| if (kind == 'check') | |
| 'checks': [ | |
| {'condition': call, 'message': 'Check failed'}, | |
| ] | |
| else | |
| 'action': {'functionCall': call}, | |
| }); | |
| final component = ComponentModel( | |
| id, | |
| 'Probe', | |
| { | |
| if (kind == 'check') | |
| 'checks': [ | |
| {'condition': call, 'message': 'Check failed'}, | |
| ] | |
| else | |
| 'action': {'functionCall': call}, | |
| }, | |
| catalog: surface.defaultCatalog == null | |
| ? processor.catalogs.first.id | |
| : null, | |
| ); |
🤖 Agent-generated
| "properties": { | ||
| "args": { | ||
| "type": "object", | ||
| "description": "JSON Schema of the function's args." | ||
| }, | ||
| "requiresUserActivation": { | ||
| "type": "boolean" | ||
| } | ||
| }, | ||
| "required": ["args"], | ||
| "additionalProperties": false |
There was a problem hiding this comment.
RoutingCatalog.agentFunctions sets additionalProperties: false while omitting returnType and allowedCallers from the v1.0 FunctionDefinition schema (raised by both reviewers). Cases therefore cannot declare schema-only functions with allowedCallers: "agentOnly" or explicit return types. The suite misses Python's live bug where DataContext._call_agent_function sends agentOnly calls to the agent instead of reporting EXPRESSION_ERROR.
| "properties": { | |
| "args": { | |
| "type": "object", | |
| "description": "JSON Schema of the function's args." | |
| }, | |
| "requiresUserActivation": { | |
| "type": "boolean" | |
| } | |
| }, | |
| "required": ["args"], | |
| "additionalProperties": false | |
| "properties": { | |
| "args": { | |
| "type": "object", | |
| "description": "JSON Schema of the function's args." | |
| }, | |
| "returnType": { | |
| "type": "string", | |
| "enum": [ | |
| "string", | |
| "number", | |
| "boolean", | |
| "array", | |
| "object", | |
| "validationResult", | |
| "any", | |
| "void" | |
| ], | |
| "default": "any" | |
| }, | |
| "allowedCallers": { | |
| "type": "string", | |
| "enum": ["rendererOnly", "agentOnly", "rendererOrAgent"] | |
| }, | |
| "requiresUserActivation": { | |
| "type": "boolean" | |
| } | |
| }, | |
| "required": ["args"], | |
| "additionalProperties": false |
🤖 Agent-generated
| - `core/functions.yaml`: Contains test cases for the basic catalog functions, for the catalog a function call runs in on a surface, and for a call on a surface to a function whose `allowedCallers` is `agentOnly`, which the renderer refuses. See [Writing cases for `evaluate_function`](#writing-cases-for-evaluate_function). | ||
| - `core/functions.yaml`: Contains test cases for the basic catalog functions, and for the catalog a function call runs in on a surface. See [Writing cases for `evaluate_function`](#writing-cases-for-evaluate_function). | ||
| - `core/message_processor_scaling.yaml`: Timing cases for `processMessages`, with a validation config and without one, which check that a surface arriving in many small messages takes time in proportion to its size. See [Timing cases](#timing-cases). | ||
| - `core/agent_function_routing.yaml`: Covers the v1.0 fallback routing of function calls to the agent with `callAgentFunction` ("Function Location Resolution & Fallback Routing" in the v1.0 protocol): which calls are sent and which run or fail locally, the `catalogId` an outbound call names, the value while a call is pending and once the agent answers, how an agent error is reported, surface-level caching and request sharing for reactive bindings (including canonical argument key ordering), and fresh execution without sharing or caching for imperative action calls. See [Writing cases for `route_agent_function`](#writing-cases-for-route_agent_function). |
There was a problem hiding this comment.
README.md adds core/agent_function_routing.yaml and route_agent_function while linking implementers to conformance_harness_implementation_guide.md. That guide still lists only the five older C4 suites and 14 actions. Engineers building a new SDK test runner from conformance_harness_implementation_guide.md will omit route_agent_function and skip C4 fallback routing cases.
Update C4 and Action reference by suite in conformance/conformance_harness_implementation_guide.md.
🤖 Agent-generated
| }, | ||
| "expectValues": { | ||
| "type": "array", | ||
| "description": "The current value of every binding and check evaluated so far, in order, with null for a pending value. A check's value is whether its rule passes." |
There was a problem hiding this comment.
Line 1009 specifies null for all pending values, contradicting README.md:139. In practice, Dart (_checkOutcome) maps pending checks to null while Python, TypeScript, and Swift return false. Future cases asserting expectValues while an earlier check remains pending will pass in Dart and fail in the other three SDKs.
| "description": "The current value of every binding and check evaluated so far, in order, with null for a pending value. A check's value is whether its rule passes." | |
| "description": "The current value of every binding and check evaluated so far, in order, with null for a pending binding. A check's value is whether its rule passes; cases read a check only after its response arrives." |
🤖 Agent-generated
| - Reactive bindings cache settled agent function answers per surface (bounded | ||
| to 256 entries, evicted on failure or surface disposal), while imperative | ||
| action calls always execute fresh without sharing or caching. |
There was a problem hiding this comment.
This PR modifies only the Dart conformance test file and zero files under dart/a2ui_core/lib/. Under update-changelog Rule 3, test-only package changes do not add runtime changelog bullets. Recording library behavior in a test-only PR leaves CHANGELOG.md inaccurate if this PR or its predecessor lands separately.
Move this sub-bullet to the pull request that modifies dart/a2ui_core/lib/.
🤖 Agent-generated
|
|
||
| - (v1_0) A function call whose function the renderer does not implement (undeclared in the catalog the call resolves to, or declared schema-only) is routed to the agent with `callAgentFunction`, as the specification's fallback routing defines, instead of failing with an `EXPRESSION_ERROR`. This applies to bindings, check rules, `action.functionCall`, and `action.event` `context`/`userMessage`. The value is pending (`undefined`) until the `agentFunctionResponse` arrives; an agent error or a timeout dispatches an `EXPRESSION_ERROR`. The outbound call always names the catalog the call resolved to, even when the payload relied on the surface default. Arguments of a schema-only function are validated against its schema before the call is sent. Reactive bindings share in-flight calls and cache settled answers per surface (bounded to 256 entries, keyed by canonically sorted argument keys, evicted on failure or surface deletion), while imperative action calls always execute fresh without sharing or caching. In a binding, check, or action, a function whose argument is still pending (such as an agent call) waits for it to settle instead of running with `undefined`, and stops without running or re-reporting if a nested argument fails. `MessageProcessor.dispose()` now disposes all surfaces in `model` in addition to `rpc`, and deleting a surface cancels its pending RPC calls. `MessageProcessor` enables fallback routing on v1.0 surfaces when it has an `outboundListener`; without one, such calls still fail locally. Adds `SurfaceModel.agentFunctionCaller`, `SurfaceModel.callAgentFunction`, `SurfaceModel.disposed`, `RpcHandler.cancelSurfaceCalls`, and the `AgentFunctionCall` and `AgentFunctionCaller` types. [#3415](https://github.com/a2ui-project/a2ui/issues/3415), [#3536](https://github.com/a2ui-project/a2ui/issues/3536) | ||
| - `DataContext.resolveSignal` seeds the result of a function call that returns a `Signal` with the signal's current value before it creates the forwarding `effect`. Under a reactivity backend whose effects run on the next flush rather than on creation (the Angular adapter), an enclosing call such as `not(areComponentsValid(...))` previously read `undefined` for that argument on its first evaluation and dispatched a spurious `EXPRESSION_ERROR`. | ||
| - The argument schema of a function loaded with `Catalog.fromJson` applies the `default` of an optional argument that a call leaves out, so a schema-only function routed to the agent is sent with its defaults, as one defined with Zod is. |
There was a problem hiding this comment.
Line 4 places a non-breaking Catalog.fromJson fix above the **BREAKING CHANGE**: entries on lines 5-7 and omits a PR link. update-changelog Rules 4 and 6 require **BREAKING CHANGE**: items at the top of ## Unreleased and PR links on every entry. Readers scanning the top of ## Unreleased for breaking migrations can miss breaking entries below non-breaking items.
Move line 4 below line 7 and append ([#3742](https://github.com/a2ui-project/a2ui/pull/3742)).
🤖 Agent-generated
| await waitUntil { | ||
| recorder.agentCalls.count >= outboundSeen + expectedOutbound.count | ||
| && genericErrors(recorder).count >= errorsSeen + expectedErrors.count | ||
| && (expectedValues == nil || currentValues() == expectedValues!.map(Self.canonical)) |
There was a problem hiding this comment.
Line 186 (expectedValues!) and line 350 (object[$0]!) use ! force unwraps in test code, violating swift/CODING_STANDARDS.md Section 3. Use optional map here and compactMap in canonical. Force unwraps in Swift test helpers crash the entire test runner process on unexpected nil instead of recording a test failure.
| && (expectedValues == nil || currentValues() == expectedValues!.map(Self.canonical)) | |
| && (expectedValues.map { currentValues() == $0.map(Self.canonical) } ?? true) |
🤖 Agent-generated
| from a2ui.core.state import DataModel, SurfaceModel | ||
| from a2ui.core.resolution import DataContext | ||
| from a2ui.core.state import ComponentModel, DataModel, SurfaceModel | ||
| from a2ui.core.resolution import ComponentContext, DataContext, GenericBinder |
There was a problem hiding this comment.
Line 49 places from a2ui.core.resolution after from a2ui.core.schema and from a2ui.core.state rather than in alphabetical module order per AGENTS.md Section 8. Unsorted imports in modified blocks create avoidable merge conflicts and violate the import ordering rules in AGENTS.md.
Move the a2ui.core.resolution import above line 45 in alphabetical module order.
🤖 Agent-generated
| if (testCase['dataModel'] case final Map<String, Object?> data) { | ||
| for (final MapEntry<String, Object?> entry in data.entries) { | ||
| surface.dataModel.set('/${entry.key}', entry.value); | ||
| } | ||
| } |
There was a problem hiding this comment.
Lines 135-139 (and AgentFunctionRoutingConformanceTests.swift:92-100) write each dataModel key to '/${key}' without RFC 6901 escaping, unlike Python and TypeScript's root set('/', ...). Any future case with / or ~ in a top-level dataModel key will diverge across runners.
| if (testCase['dataModel'] case final Map<String, Object?> data) { | |
| for (final MapEntry<String, Object?> entry in data.entries) { | |
| surface.dataModel.set('/${entry.key}', entry.value); | |
| } | |
| } | |
| if (testCase['dataModel'] case final Map<String, Object?> data) { | |
| surface.dataModel.set('/', data); | |
| } |
🤖 Agent-generated
Stacked on #3741. Review only the top commit (
test(conformance): ...).Part of #3536.
Description
#3735, #3739, #3740 and #3741 route v1.0 function calls the renderer can't run to the agent, in web_core, Dart, Python and Swift. No shared conformance case covered this:
test_rpc_call_agent_function_dispatchcalls the RPC handler directly and never goes through a binding. This PR adds a shared suite that every core runs, and fixes the two places where it found the SDKs differing.Changes
conformance/core/agent_function_routing.yaml(20 cases, new actionroute_agent_function). A case declares catalogs (localFunctionstaken from the SDK's basic catalog,agentFunctionsdeclared schema-only), a v1.0 surface with or without an outbound listener, and steps that either evaluate aFunctionCallas a binding, check or action, or answer the n-th outbound call. After each step it checks the outboundcallFunctionpayloads, the reported errors, and the current value of every binding and check. It uses no timers. The cases cover:catalogId(filled in when the call omits it), pending until answered;catalogIdcarried; an implemented function run locally; an agent result fed into a local function;EXPRESSION_ERRORwith a null value, for bindings and actions;catalogIdand a missing default catalog reported asCATALOG_ERROR, nothing sent;@system function: not routed;action.functionCallrouted;requiresUserActivationblocking a binding but not an action.RouteAgentFunctionTest,RoutingCatalogandRoutingStepinconformance_schema.json, and the suite plus a "Writing cases forroute_agent_function" section inconformance/README.md.typescript/core/web_core/tests/conformance/conformance_test.mjs,python/a2ui_core/tests/test_conformance.py, a new Dartagent_function_routing_conformance_test.dartand a new SwiftAgentFunctionRoutingConformanceTests.swift.EXECUTION_ERROR(or the RPC code). It now reportsEXPRESSION_ERROR, like the other cores, keeping thefunctionCallIdand the agent's message. A local function whose Future fails is stillEXECUTION_ERROR. This is marked breaking in the Dart CHANGELOG.defaults in the arguments of a function loaded withCatalog.fromJson, so a schema-only function sent its arguments without them.convertFunctionArgsJsonSchemaToZodnow applies them.Known divergences
agentFunctionsare skipped for Dart.Testing
yarn testpasses; the new suite 20/20.uv run pytest python/a2ui_core conformance/tests1677 passed, 15 skipped; the new suite 20/20.pyinkandmypyclean.dart testfor a2ui_core 1914 passed, 11 skipped; the new suite 17 passed and 3 skips.dart analyzeanddart formatclean.swift:6.1, Combine mapped to OpenCombine in a scratch copy): the new suite 20/20. The other failures in the full run fail the same way on the base commit.