diff --git a/conformance/README.md b/conformance/README.md index c336284e22..510c968feb 100644 --- a/conformance/README.md +++ b/conformance/README.md @@ -36,6 +36,7 @@ The suites that do not depend on an inference format sit directly under `agent/` - `agent/request_processor.yaml`: Building an inference format from its factory, and taking registered catalogs and renderer capabilities through to a request processor. - `agent/skill.yaml`: Generating skill documents and skill sets from a format and a catalog. - `agent/multi_catalog_formats.yaml`: Resolving components and function calls across active catalogs (omitting `createSurface.catalogId` when multiple catalogs are active, resolving un-annotated components and function calls by name when unique across active catalogs, and requiring an explicit `catalogId` override when ambiguous), across `direct_json` (and later `express`): compiling, decompiling (including omitting `catalogId` when name lookup is unambiguous), parsing a response, rendering a prompt snippet, and building a format. Component order inside a message carries no meaning, so a harness may compare the `components` of a compiled message by `id`. +- `agent/agent_functions.yaml`: Agent functions (v1.0) and the agent's side of the messages a renderer sends. It covers declaring agent functions on the agent's copy of a catalog (including building one from the catalog's own declaration), answering `callAgentFunction` (the response values and error codes, schema defaults, the handler's context, and which catalog a call runs in), validating model output that calls agent functions, and how `RendererToAgentMessageProcessor` routes `action`, `error` and `rendererFunctionResponse` messages to its callbacks. See [Writing cases for the agent function suite](#writing-cases-for-the-agent-function-suite). - `agent/streamed_answer.yaml`: Reading a streamed answer into a renderer's message processor, for an app that is both the agent and the renderer, and keeping the answer's text, surfaces and rejected blocks in the order the model wrote them. It runs on `direct_json`, the format that streams. The suite belongs to the optional `agent_renderer_app` feature, and only an SDK that implements it runs the suite. A case names its format with `args.format`, an `InferenceFormatName`: `direct_json` (or `express`). @@ -149,6 +150,29 @@ The case then runs its `steps` in order. An `evaluate` step evaluates a `Functio The cases use no timers, so they do not cover the timeout of an agent call. An SDK that cannot run a case lists it as a known divergence with the reason: the Dart core cannot declare a function without an implementation, so it skips the cases that use `agentFunctions`. Re-evaluating a binding whose cancelled call should be sent again has no single trigger that works in every core, so that rule has two cases, and each core runs the one that fits how it re-evaluates: `..._when_its_value_is_rewritten` (TypeScript and Dart) and `..._on_unrelated_change` (Python and Swift). An agent-initiated `callRendererFunction` whose argument calls an agent function is not covered here, because the suite has no step that sends one and the basic catalog has no function an agent may call; each SDK covers it in its unit tests. +### Writing cases for the agent function suite + +`agent/agent_functions.yaml` declares agent functions in `AgentFunctionCatalog` entries (`conformance_schema.json`): a catalog document in `catalog`, named as in the other agent suites, and `agent_functions`, the functions the agent adds to its copy of that catalog (`copy_with(agent_functions=...)`), by name. A declaration carries the JSON Schema of the function's `args`, its `returnType`, and optionally `description`; an agent function built this way is `rendererOnly`. Two optional fields change how it is built: + +- `from_api: true` builds it from the catalog's own declaration of the function (`AgentFunction.from_api`), which keeps that declaration's `allowedCallers`. If the catalog declares no such function, the harness builds the declaration from `returnType`, `args` and `description`, and the catalog must refuse it. +- `name` gives the function a name other than its key, so a case can declare two agent functions with one name. + +In place of a handler a declaration states a `behavior`, so every SDK runs the same function: + +- `returns`: returns that value, which may be `null`. +- `returns_args: true`: returns the arguments the handler received, after validation and with the schema's defaults applied. +- `returns_context: true`: returns the context of the call as `{catalogId, surfaceId, functionCallId}`. +- `raises`: fails with `message`. With `code`, it fails the way the SDK lets a handler choose the response's RPC error code; without one, it fails with an ordinary error. + +The suite has four actions: + +- `declare_agent_functions` builds one catalog with its agent functions. `expect` lists, without regard to order, the catalog's functions (agent functions included), its agent functions, and the functions its `toJson()` document lists. `expect_error` names a catalog the SDK must refuse to build: an agent function that takes the name of a function the catalog declares, two agent functions with one name, one built with `from_api` from a function the catalog doesn't declare, or one built from an `agentOnly` declaration. +- `handle_agent_function` builds a `RendererToAgentMessageProcessor` from the `catalogs` and processes `message`, a `callAgentFunction` message as received. The result must be handled, with exactly one reply. `expect` is the body of that `agentFunctionResponse`: `functionCallId` and either `value` or `error`. `error.code` is compared exactly and `error.message` is a substring of the message sent. `expect_error` means processing the message fails instead of answering it, which happens only when the message has no `functionCallId` to copy. +- `validate_payload` validates `messages` against the `catalogs`, the way an agent checks model output before sending it. A case states `expect_valid: true` or `expect_error`. +- `process_renderer_message` builds a `RendererToAgentMessageProcessor` from the agent functions of the optional `catalog` and the callbacks named in `callbacks` (`action`, `error` and `rendererFunctionResponse`, for `on_action`, `on_error` and `on_renderer_function_response`), each of which records the message it receives, then processes `message`. `expect.handled` is the result's `handled`. `expect.replies` lists the replies, serialized, in order; they are compared exactly, except that an expected `error.message` is a substring of the one sent and may be left out, and an absent `replies` means none. `expect.callback` is the one callback that ran, as `{kind, message}` with the message it received, serialized; absent, no callback ran. `expect_error` means processing fails, and then no callback runs either. + +The TypeScript and Dart agent SDKs do not implement agent functions yet, so their harnesses skip every case of this suite with that reason. + ### Writing cases for the agent SDK suites The suites under `agent/`, including the per-format folders, share one vocabulary, described in the `$defs` of `conformance_schema.json` and summarised here. `agent/builder/` is the exception, and uses the AST-plus-golden notation described in its own README. diff --git a/conformance/agent/agent_functions.yaml b/conformance/agent/agent_functions.yaml new file mode 100644 index 0000000000..bb6ebe35da --- /dev/null +++ b/conformance/agent/agent_functions.yaml @@ -0,0 +1,1002 @@ +# Copyright 2024 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Agent functions (v1.0): functions an agent declares on its copy of a catalog +# and runs when a renderer sends `callAgentFunction`, and how the agent's +# `RendererToAgentMessageProcessor` routes the messages a renderer sends. +# +# A case declares each agent function with the JSON Schema of its `args` (or +# `from_api: true` to wrap the catalog's own declaration) and a scripted +# behaviour in place of a handler, so every SDK runs the same function: it +# returns a fixed value, returns the arguments it received, returns the +# context of the call, or fails. +# See "Writing cases for the agent function suite" in conformance/README.md. + +# --- Declaring agent functions --- + +- name: test_declare_agent_function_lists_it_with_the_catalog_functions + description: >- + An agent function is one of the catalog's functions, so validation, prompts and compilers see it, and the catalog document lists it like any declared function. + action: declare_agent_functions + args: + catalog: + catalog: "test_data/catalogs/simplified_catalog_v1_0.json" + agent_functions: + fetchFxRate: + description: Current exchange rate from base to target. + returnType: number + args: + type: object + properties: + base: {type: string} + target: {type: string, default: USD} + required: [base] + additionalProperties: false + behavior: {returns: 1.08} + expect: + functions: [required, formatString, openUrl, fetchFxRate] + agent_functions: [fetchFxRate] + document_functions: [required, formatString, openUrl, fetchFxRate] + +- name: test_declare_agent_function_with_a_taken_name_is_a_catalog_error + description: >- + An agent function may not take the name of a function the catalog already declares, since a renderer would then run its own function and never send the call to the agent. + action: declare_agent_functions + args: + catalog: + catalog: "test_data/catalogs/simplified_catalog_v1_0.json" + agent_functions: + formatString: + returnType: string + args: + type: object + properties: + value: {type: string} + required: [value] + behavior: {returns: "formatted"} + expect_error: + category: CatalogError + message: "formatString" + +- name: test_declare_agent_function_from_the_catalog_declaration + description: >- + An agent function built from a function the catalog declares (from_api) replaces that declaration, keeps its allowedCallers (here rendererOrAgent), and leaves the catalog document as it was. + action: declare_agent_functions + args: + catalog: + catalog: "test_data/catalogs/agent_functions_catalog_v1_0.json" + agent_functions: + notify: + from_api: true + behavior: {returns: null} + expect: + functions: [notify] + agent_functions: [notify] + document_functions: [notify] + +- name: test_declare_agent_function_from_an_undeclared_function_is_a_catalog_error + description: >- + An agent function built from a declaration (from_api) that the catalog does not have is refused, so a foreign declaration can't leak into the catalog. + action: declare_agent_functions + args: + catalog: + catalog: "test_data/catalogs/simplified_catalog_v1_0.json" + agent_functions: + fetchFxRate: + from_api: true + returnType: number + args: + type: object + properties: + base: {type: string} + required: [base] + behavior: {returns: 1.08} + expect_error: + category: CatalogError + message: "fetchFxRate" + +- name: test_declare_agent_function_from_an_agent_only_declaration_is_a_catalog_error + description: >- + A function whose allowedCallers is agentOnly is never called by a renderer, so it can't become an agent function. + action: declare_agent_functions + args: + catalog: + catalog: + catalogId: conformance/agent_only + protocolVersion: "1.0" + components: {} + functions: + resetLedger: + type: object + description: Clears the ledger. + returnType: void + allowedCallers: agentOnly + properties: + "@call": {const: resetLedger} + args: {type: object, properties: {}} + required: ["@call", "args"] + agent_functions: + resetLedger: + from_api: true + behavior: {returns: null} + expect_error: + category: CatalogError + message: "resetLedger" + +- name: test_declare_two_agent_functions_with_one_name_is_a_catalog_error + description: >- + Two agent functions may not share a name. The second entry uses `name` to give itself the first one's name. + action: declare_agent_functions + args: + catalog: + catalog: "test_data/catalogs/simplified_catalog_v1_0.json" + agent_functions: + fetchFxRate: + returnType: number + args: {type: object, properties: {base: {type: string}}} + behavior: {returns: 1.08} + fetchFxRateAgain: + name: fetchFxRate + returnType: number + args: {type: object, properties: {base: {type: string}}} + behavior: {returns: 1.09} + expect_error: + category: CatalogError + message: "fetchFxRate" + +# --- Answering callAgentFunction --- + +- name: test_handle_agent_function_returns_the_value_and_copies_the_call_id + description: >- + The response carries the handler's value and copies the call's functionCallId verbatim, however it is spelled. + action: handle_agent_function + args: + catalogs: + - catalog: "test_data/catalogs/simplified_catalog_v1_0.json" + agent_functions: + fetchFxRate: + returnType: number + args: + type: object + properties: + base: {type: string} + target: {type: string, default: USD} + required: [base] + additionalProperties: false + behavior: {returns: 1.08} + message: + version: "1.0" + callAgentFunction: + surfaceId: s1 + functionCallId: "fx call/ü 42" + callFunction: + "@call": fetchFxRate + catalogId: conformance/simplified + args: {base: EUR} + expect: + functionCallId: "fx call/ü 42" + value: 1.08 + +- name: test_handle_agent_function_sends_a_null_value + description: >- + A handler that returns null still answers with a value, so the renderer can tell the answer from a missing one. + action: handle_agent_function + args: + catalogs: + - catalog: "test_data/catalogs/simplified_catalog_v1_0.json" + agent_functions: + lookupNickname: + returnType: string + args: + type: object + properties: + userId: {type: string} + required: [userId] + behavior: {returns: null} + message: + version: "1.0" + callAgentFunction: + surfaceId: s1 + functionCallId: nick-1 + callFunction: + "@call": lookupNickname + catalogId: conformance/simplified + args: {userId: u1} + expect: + functionCallId: nick-1 + value: null + +- name: test_handle_agent_function_applies_schema_defaults + description: >- + The arguments are validated against the function's schema before the handler runs, and the handler receives them with the schema's defaults applied. + action: handle_agent_function + args: + catalogs: + - catalog: "test_data/catalogs/simplified_catalog_v1_0.json" + agent_functions: + fetchFxRate: + returnType: object + args: + type: object + properties: + base: {type: string} + target: {type: string, default: USD} + required: [base] + additionalProperties: false + behavior: {returns_args: true} + message: + version: "1.0" + callAgentFunction: + surfaceId: s1 + functionCallId: fx-1 + callFunction: + "@call": fetchFxRate + catalogId: conformance/simplified + args: {base: EUR} + expect: + functionCallId: fx-1 + value: {base: EUR, target: USD} + +- name: test_handle_agent_function_passes_the_call_context + description: >- + The handler learns the catalog the call resolved to, the surface it was made on, and its functionCallId. + action: handle_agent_function + args: + catalogs: + - catalog: "test_data/catalogs/simplified_catalog_v1_0.json" + agent_functions: + whereAmI: + returnType: object + args: {type: object, properties: {}} + behavior: {returns_context: true} + message: + version: "1.0" + callAgentFunction: + surfaceId: checkout + functionCallId: ctx-1 + callFunction: + "@call": whereAmI + catalogId: conformance/simplified + args: {} + expect: + functionCallId: ctx-1 + value: + catalogId: conformance/simplified + surfaceId: checkout + functionCallId: ctx-1 + +- name: test_handle_agent_function_with_an_argument_of_the_wrong_type + description: Arguments that fail the function's schema get INVALID_FUNCTION_CALL, and the handler does not run. + action: handle_agent_function + args: + catalogs: + - catalog: "test_data/catalogs/simplified_catalog_v1_0.json" + agent_functions: + fetchFxRate: + returnType: number + args: + type: object + properties: + base: {type: string} + target: {type: string, default: USD} + required: [base] + additionalProperties: false + behavior: {returns: 1.08} + message: + version: "1.0" + callAgentFunction: + surfaceId: s1 + functionCallId: fx-2 + callFunction: + "@call": fetchFxRate + catalogId: conformance/simplified + args: {base: 3} + expect: + functionCallId: fx-2 + error: + code: INVALID_FUNCTION_CALL + message: fetchFxRate + +- name: test_handle_agent_function_with_a_missing_required_argument + description: A call that leaves out a required argument gets INVALID_FUNCTION_CALL. + action: handle_agent_function + args: + catalogs: + - catalog: "test_data/catalogs/simplified_catalog_v1_0.json" + agent_functions: + fetchFxRate: + returnType: number + args: + type: object + properties: + base: {type: string} + target: {type: string, default: USD} + required: [base] + additionalProperties: false + behavior: {returns: 1.08} + message: + version: "1.0" + callAgentFunction: + surfaceId: s1 + functionCallId: fx-3 + callFunction: + "@call": fetchFxRate + catalogId: conformance/simplified + args: {target: GBP} + expect: + functionCallId: fx-3 + error: + code: INVALID_FUNCTION_CALL + +- name: test_handle_agent_function_with_an_undeclared_argument + description: A call with an argument that a closed schema does not declare gets INVALID_FUNCTION_CALL. + action: handle_agent_function + args: + catalogs: + - catalog: "test_data/catalogs/simplified_catalog_v1_0.json" + agent_functions: + fetchFxRate: + returnType: number + args: + type: object + properties: + base: {type: string} + target: {type: string, default: USD} + required: [base] + additionalProperties: false + behavior: {returns: 1.08} + message: + version: "1.0" + callAgentFunction: + surfaceId: s1 + functionCallId: fx-4 + callFunction: + "@call": fetchFxRate + catalogId: conformance/simplified + args: {base: EUR, precision: 4} + expect: + functionCallId: fx-4 + error: + code: INVALID_FUNCTION_CALL + +- name: test_handle_agent_function_with_an_unknown_name + description: A call to a name no catalog registers as an agent function gets UNKNOWN_FUNCTION. + action: handle_agent_function + args: + catalogs: + - catalog: "test_data/catalogs/simplified_catalog_v1_0.json" + agent_functions: + fetchFxRate: + returnType: number + args: {type: object, properties: {base: {type: string}}} + behavior: {returns: 1.08} + message: + version: "1.0" + callAgentFunction: + surfaceId: s1 + functionCallId: u-1 + callFunction: + "@call": fetchStockPrice + catalogId: conformance/simplified + args: {} + expect: + functionCallId: u-1 + error: + code: UNKNOWN_FUNCTION + +- name: test_handle_agent_function_in_an_unknown_catalog + description: A call that names a catalog the agent does not have gets UNKNOWN_FUNCTION, even when another catalog registers the name. + action: handle_agent_function + args: + catalogs: + - catalog: "test_data/catalogs/simplified_catalog_v1_0.json" + agent_functions: + fetchFxRate: + returnType: number + args: {type: object, properties: {base: {type: string}}} + behavior: {returns: 1.08} + message: + version: "1.0" + callAgentFunction: + surfaceId: s1 + functionCallId: u-2 + callFunction: + "@call": fetchFxRate + catalogId: https://example.com/other/catalog.json + args: {base: EUR} + expect: + functionCallId: u-2 + error: + code: UNKNOWN_FUNCTION + +- name: test_handle_agent_function_for_a_renderer_function + description: >- + A function the catalog declares for the renderer to run is not an agent function, so a call to it gets UNKNOWN_FUNCTION. + action: handle_agent_function + args: + catalogs: + - catalog: "test_data/catalogs/simplified_catalog_v1_0.json" + agent_functions: + fetchFxRate: + returnType: number + args: {type: object, properties: {base: {type: string}}} + behavior: {returns: 1.08} + message: + version: "1.0" + callAgentFunction: + surfaceId: s1 + functionCallId: u-3 + callFunction: + "@call": formatString + catalogId: conformance/simplified + args: {value: Hello} + expect: + functionCallId: u-3 + error: + code: UNKNOWN_FUNCTION + +- name: test_handle_agent_function_whose_handler_fails + description: >- + A handler that fails gets EXECUTION_ERROR, with the failure's message. + action: handle_agent_function + args: + catalogs: + - catalog: "test_data/catalogs/simplified_catalog_v1_0.json" + agent_functions: + fetchFxRate: + returnType: number + args: {type: object, properties: {base: {type: string}}} + behavior: + raises: {message: "rate service unavailable"} + message: + version: "1.0" + callAgentFunction: + surfaceId: s1 + functionCallId: e-1 + callFunction: + "@call": fetchFxRate + catalogId: conformance/simplified + args: {base: EUR} + expect: + functionCallId: e-1 + error: + code: EXECUTION_ERROR + message: "rate service unavailable" + +- name: test_handle_agent_function_whose_handler_fails_with_its_own_code + description: >- + A handler that fails with an RPC error code of its choosing sends that code instead of EXECUTION_ERROR. + action: handle_agent_function + args: + catalogs: + - catalog: "test_data/catalogs/simplified_catalog_v1_0.json" + agent_functions: + fetchFxRate: + returnType: number + args: {type: object, properties: {base: {type: string}}} + behavior: + raises: {code: PROVIDER_NOT_FOUND, message: "no rate provider for EUR"} + message: + version: "1.0" + callAgentFunction: + surfaceId: s1 + functionCallId: e-2 + callFunction: + "@call": fetchFxRate + catalogId: conformance/simplified + args: {base: EUR} + expect: + functionCallId: e-2 + error: + code: PROVIDER_NOT_FOUND + message: "no rate provider for EUR" + +- name: test_handle_agent_function_picks_the_catalog_the_call_names + description: >- + Two catalogs may each register an agent function of the same name, and a call runs the one in the catalog it names. + action: handle_agent_function + args: + catalogs: + - catalog: "test_data/catalogs/simplified_catalog_v1_0.json" + agent_functions: + lookup: + returnType: string + args: {type: object, properties: {}} + behavior: {returns: "from simplified"} + - catalog: "test_data/catalogs/agent_functions_catalog_v1_0.json" + agent_functions: + lookup: + returnType: string + args: {type: object, properties: {}} + behavior: {returns: "from agent_functions"} + message: + version: "1.0" + callAgentFunction: + surfaceId: s1 + functionCallId: m-1 + callFunction: + "@call": lookup + catalogId: conformance/agent_functions + args: {} + expect: + functionCallId: m-1 + value: "from agent_functions" + +- name: test_handle_agent_function_without_catalog_id_runs_the_only_match + description: >- + Renderers name the catalog on every call. A call that names none runs the agent function of that name if exactly one catalog registers it. + action: handle_agent_function + args: + catalogs: + - catalog: "test_data/catalogs/simplified_catalog_v1_0.json" + agent_functions: + lookup: + returnType: string + args: {type: object, properties: {}} + behavior: {returns: "from simplified"} + - catalog: "test_data/catalogs/agent_functions_catalog_v1_0.json" + agent_functions: {} + message: + version: "1.0" + callAgentFunction: + surfaceId: s1 + functionCallId: m-2 + callFunction: + "@call": lookup + args: {} + expect: + functionCallId: m-2 + value: "from simplified" + +- name: test_handle_agent_function_without_catalog_id_and_no_match + description: A call that names no catalog, to a name no catalog registers, gets UNKNOWN_FUNCTION. + action: handle_agent_function + args: + catalogs: + - catalog: "test_data/catalogs/simplified_catalog_v1_0.json" + agent_functions: + lookup: + returnType: string + args: {type: object, properties: {}} + behavior: {returns: "from simplified"} + message: + version: "1.0" + callAgentFunction: + surfaceId: s1 + functionCallId: m-3 + callFunction: + "@call": search + args: {} + expect: + functionCallId: m-3 + error: + code: UNKNOWN_FUNCTION + +- name: test_handle_agent_function_without_catalog_id_and_several_matches + description: >- + A call that names no catalog, to a name several catalogs register, is ambiguous and gets INVALID_FUNCTION_CALL rather than running either function. + action: handle_agent_function + args: + catalogs: + - catalog: "test_data/catalogs/simplified_catalog_v1_0.json" + agent_functions: + lookup: + returnType: string + args: {type: object, properties: {}} + behavior: {returns: "from simplified"} + - catalog: "test_data/catalogs/agent_functions_catalog_v1_0.json" + agent_functions: + lookup: + returnType: string + args: {type: object, properties: {}} + behavior: {returns: "from agent_functions"} + message: + version: "1.0" + callAgentFunction: + surfaceId: s1 + functionCallId: m-4 + callFunction: + "@call": lookup + args: {} + expect: + functionCallId: m-4 + error: + code: INVALID_FUNCTION_CALL + +- name: test_handle_agent_function_with_a_malformed_message + description: >- + A message that carries a functionCallId but fails the callAgentFunction schema is answered with INVALID_FUNCTION_CALL. + action: handle_agent_function + args: + catalogs: + - catalog: "test_data/catalogs/simplified_catalog_v1_0.json" + agent_functions: + fetchFxRate: + returnType: number + args: {type: object, properties: {base: {type: string}}} + behavior: {returns: 1.08} + message: + version: "1.0" + callAgentFunction: + surfaceId: s1 + functionCallId: bad-1 + expect: + functionCallId: bad-1 + error: + code: INVALID_FUNCTION_CALL + +- name: test_handle_agent_function_with_a_message_without_call_id + description: >- + A message without a functionCallId cannot be answered, since a response must copy it, so handling it fails instead. + action: handle_agent_function + args: + catalogs: + - catalog: "test_data/catalogs/simplified_catalog_v1_0.json" + agent_functions: + fetchFxRate: + returnType: number + args: {type: object, properties: {base: {type: string}}} + behavior: {returns: 1.08} + message: + version: "1.0" + callAgentFunction: + surfaceId: s1 + callFunction: + "@call": fetchFxRate + catalogId: conformance/simplified + args: {base: EUR} + expect_error: + category: ValidationError + +- name: test_handle_agent_function_built_from_the_catalog_declaration + description: >- + An agent function built from the catalog's own declaration (from_api) validates the call against that declaration and runs. + action: handle_agent_function + args: + catalogs: + - catalog: "test_data/catalogs/agent_functions_catalog_v1_0.json" + agent_functions: + notify: + from_api: true + behavior: {returns_args: true} + message: + version: "1.0" + callAgentFunction: + surfaceId: s1 + functionCallId: n-1 + callFunction: + "@call": notify + catalogId: conformance/agent_functions + args: {message: Saved} + expect: + functionCallId: n-1 + value: {message: Saved} + +# --- Validating model output that calls agent functions --- + +- name: test_validate_payload_with_a_call_to_a_declared_agent_function + description: >- + Model output that calls an agent function the catalog declares validates like a call to any other declared function. + action: validate_payload + args: + catalogs: + - catalog: "test_data/catalogs/simplified_catalog_v1_0.json" + agent_functions: + fetchGreeting: + returnType: string + args: + type: object + properties: + name: {type: string} + required: [name] + additionalProperties: false + behavior: {returns: "Hello"} + messages: + - version: "1.0" + createSurface: + surfaceId: s1 + catalogId: conformance/simplified + components: + - id: root + component: Text + text: + "@call": fetchGreeting + args: {name: Ada} + expect_valid: true + +- name: test_validate_payload_with_a_call_to_an_undeclared_function + description: >- + Declaring agent functions does not open the catalog: a call to a function that is neither declared nor an agent function is still rejected. + action: validate_payload + args: + catalogs: + - catalog: "test_data/catalogs/simplified_catalog_v1_0.json" + agent_functions: + fetchGreeting: + returnType: string + args: + type: object + properties: + name: {type: string} + required: [name] + additionalProperties: false + behavior: {returns: "Hello"} + messages: + - version: "1.0" + createSurface: + surfaceId: s1 + catalogId: conformance/simplified + components: + - id: root + component: Text + text: + "@call": fetchFarewell + args: {name: Ada} + expect_error: + category: ValidationError + message: fetchFarewell + +- name: test_validate_payload_with_invalid_arguments_to_an_agent_function + description: Model output that calls an agent function with arguments its schema rejects is invalid. + action: validate_payload + args: + catalogs: + - catalog: "test_data/catalogs/simplified_catalog_v1_0.json" + agent_functions: + fetchGreeting: + returnType: string + args: + type: object + properties: + name: {type: string} + required: [name] + additionalProperties: false + behavior: {returns: "Hello"} + messages: + - version: "1.0" + createSurface: + surfaceId: s1 + catalogId: conformance/simplified + components: + - id: root + component: Text + text: + "@call": fetchGreeting + args: {nickname: Ada} + expect_error: + category: ValidationError + +# --- Routing renderer messages --- + +- name: test_process_action_with_its_callback + description: >- + With on_action set, an action is validated, handed to the callback, and handled with no replies. + action: process_renderer_message + args: + callbacks: [action] + message: + version: "1.0" + action: + name: submit + surfaceId: s1 + sourceComponentId: b1 + timestamp: "2026-01-01T00:00:00Z" + context: {} + expect: + handled: true + callback: + kind: action + message: + version: "1.0" + action: + name: submit + surfaceId: s1 + sourceComponentId: b1 + timestamp: "2026-01-01T00:00:00Z" + context: {} + +- name: test_process_action_without_its_callback + description: >- + Without on_action, an action is not handled, so a framework passes it on to the model's turn. + action: process_renderer_message + args: + callbacks: [error, rendererFunctionResponse] + message: + version: "1.0" + action: + name: submit + surfaceId: s1 + sourceComponentId: b1 + timestamp: "2026-01-01T00:00:00Z" + context: {} + expect: + handled: false + +- name: test_process_error_with_its_callback + description: With on_error set, an error message is validated, handed to the callback, and handled with no replies. + action: process_renderer_message + args: + callbacks: [error] + message: + version: "1.0" + error: + code: EXPRESSION_ERROR + message: formatDate failed + surfaceId: s1 + expect: + handled: true + callback: + kind: error + message: + version: "1.0" + error: + code: EXPRESSION_ERROR + message: formatDate failed + surfaceId: s1 + +- name: test_process_error_without_its_callback + description: Without on_error, an error message is not handled. + action: process_renderer_message + args: + message: + version: "1.0" + error: + code: EXPRESSION_ERROR + message: formatDate failed + surfaceId: s1 + expect: + handled: false + +- name: test_process_renderer_function_response_with_its_callback + description: >- + With on_renderer_function_response set, a rendererFunctionResponse is validated, handed to the callback, and handled with no replies. + action: process_renderer_message + args: + callbacks: [rendererFunctionResponse] + message: + version: "1.0" + rendererFunctionResponse: + functionCallId: r-1 + value: "12:00" + expect: + handled: true + callback: + kind: rendererFunctionResponse + message: + version: "1.0" + rendererFunctionResponse: + functionCallId: r-1 + value: "12:00" + +- name: test_process_renderer_function_response_without_its_callback + description: Without on_renderer_function_response, a rendererFunctionResponse is not handled. + action: process_renderer_message + args: + callbacks: [action] + message: + version: "1.0" + rendererFunctionResponse: + functionCallId: r-1 + value: "12:00" + expect: + handled: false + +- name: test_process_malformed_action_with_its_callback + description: >- + With on_action set, an action that fails its schema is a validation error, and the callback does not run. + action: process_renderer_message + args: + callbacks: [action] + message: + version: "1.0" + action: + name: submit + expect_error: + category: ValidationError + +- name: test_process_message_with_an_unknown_key + description: A message that is none of the renderer-to-agent messages is a validation error. + action: process_renderer_message + args: + callbacks: [action, error, rendererFunctionResponse] + message: + version: "1.0" + updateComponents: + surfaceId: s1 + components: [] + expect_error: + category: ValidationError + +- name: test_process_call_agent_function_is_always_handled + description: >- + A callAgentFunction is handled without any callback, with exactly one agentFunctionResponse. + action: process_renderer_message + args: + catalog: + catalog: "test_data/catalogs/simplified_catalog_v1_0.json" + agent_functions: + fetchFxRate: + returnType: number + args: {type: object, properties: {base: {type: string}}} + behavior: {returns: 1.08} + message: + version: "1.0" + callAgentFunction: + surfaceId: s1 + functionCallId: p-1 + callFunction: + "@call": fetchFxRate + catalogId: conformance/simplified + args: {base: EUR} + expect: + handled: true + replies: + - version: "1.0" + agentFunctionResponse: + functionCallId: p-1 + value: 1.08 + +- name: test_process_call_agent_function_with_an_empty_call_id + description: >- + A callAgentFunction with an empty functionCallId can't be answered, so processing it is a validation error. + action: process_renderer_message + args: + catalog: + catalog: "test_data/catalogs/simplified_catalog_v1_0.json" + agent_functions: + fetchFxRate: + returnType: number + args: {type: object, properties: {base: {type: string}}} + behavior: {returns: 1.08} + message: + version: "1.0" + callAgentFunction: + surfaceId: s1 + functionCallId: "" + callFunction: + "@call": fetchFxRate + catalogId: conformance/simplified + args: {base: EUR} + expect_error: + category: ValidationError + +- name: test_process_call_agent_function_with_an_empty_surface_id + description: >- + A callAgentFunction with an empty surfaceId is answered with INVALID_FUNCTION_CALL, and the handler does not run. + action: process_renderer_message + args: + catalog: + catalog: "test_data/catalogs/simplified_catalog_v1_0.json" + agent_functions: + fetchFxRate: + returnType: number + args: {type: object, properties: {base: {type: string}}} + behavior: {returns: 1.08} + message: + version: "1.0" + callAgentFunction: + surfaceId: "" + functionCallId: p-2 + callFunction: + "@call": fetchFxRate + catalogId: conformance/simplified + args: {base: EUR} + expect: + handled: true + replies: + - version: "1.0" + agentFunctionResponse: + functionCallId: p-2 + error: + code: INVALID_FUNCTION_CALL diff --git a/conformance/conformance_harness_implementation_guide.md b/conformance/conformance_harness_implementation_guide.md index 566e24ed49..817329994b 100644 --- a/conformance/conformance_harness_implementation_guide.md +++ b/conformance/conformance_harness_implementation_guide.md @@ -159,6 +159,7 @@ Implement the suites in stages so each layer is tested before higher-level compo | A4. Multi-catalog and session pipeline | `agent/multi_catalog_formats.yaml`, `agent/request_processor.yaml`, `agent/streamed_answer.yaml` | Cross-catalog lookup, `InferenceFormatFactory`, `A2uiGenerator`, `A2uiRequestProcessor`, and `StreamedAnswer` | | A5. Specification round-trips | `catalogs/basic/v1/examples/*.json` (and v0.9 examples if supported) | End-to-end `decompile` followed by `compile` across every published basic catalog example | | A6. Skills, macros, and scaling | `agent/skill.yaml`, `agent/builder/builder.yaml`, `agent/macros/macros.yaml`, `agent/*/scaling.yaml` | Skill generation, typesafe fluent builders, macros, and linear-time parsing | +| A7. Agent functions | `agent/agent_functions.yaml` | Agent functions on a catalog, answering `callAgentFunction`, and routing renderer messages through `RendererToAgentMessageProcessor` | | F1. Framework adapter | `core/adapter_conformance.yaml`, `core/accessibility.yaml` | Rendered view trees, repeaters, two-way bindings, action and user-activation gating, validation state, accessibility, and unmount cleanup | Notice that `agent/express/` has no `response_streaming.yaml` file. Express parses whole buffered blocks and does not implement `openStream`. @@ -289,6 +290,10 @@ All four actions in `response_parser.yaml` run against `format.createParser()`: 2. `create_processor` (`agent/request_processor.yaml`): Build `A2uiGenerator` with `args.catalogs`, `args.examples`, `args.accepts_inline_catalogs`, and optional `args.format` (omit the factory argument when `args.format` is absent to test the default Express factory). Snapshot the generator's catalogs before calling `generator.createProcessor(args.renderer_capabilities, formatOverride: args.format_override)`, then check `expect.active_catalog_ids`, positional `expect.catalogs`, prompt snippet checks, and `expect.generator_catalogs_unchanged`. If a sibling `then_parse` block is present, run `processor.parseResponse(then_parse.input, wrapped: then_parse.wrapped ?? true)` and check `then_parse.expect` or `then_parse.expect_error`. 3. `stream_answer` (`agent/streamed_answer.yaml`): Build an `A2uiRequestProcessor` (`DirectJsonFormatFactory` with `args.catalog`) and a Core `MessageProcessor` (`args.renderer_catalogs ?? [args.catalog]`), then open `StreamedAnswer(processor, renderer)`. Unlike `parse_chunk` deltas, each step's `expect_parts` checks the full `answer.parts` list accumulated so far (`{"text": ...}`, `{"surface": surface.id}`, or `{"error": {"category": ...}}`). Assert reference identity (`same` / `===` / `is`) between each `AnswerSurfacePart.surface` and `renderer.model.getSurface(surface.id)`, and check `step.expect_in_block` plus `expect_close` after `answer.close()`. +### Agent functions + +`agent/agent_functions.yaml` (`declare_agent_functions`, `handle_agent_function`, `validate_payload` and `process_renderer_message`) is described in [Writing cases for the agent function suite](README.md#writing-cases-for-the-agent-function-suite). Turn each declared `behavior` into a handler in the harness, build a declaration with `from_api: true` from the loaded catalog's own entry for it (or, when the catalog has none, from the declaration's `returnType`, `args` and `description`), and run every `callAgentFunction` through `RendererToAgentMessageProcessor`. For `process_renderer_message`, register a recording callback for each kind in `args.callbacks`. An SDK that does not implement agent functions yet registers every case as skipped with that reason. + ### Skills, builders, macros, round-trips, and timing suites 1. Skills (`agent/skill.yaml`): Dispatch five actions against `SkillGenerator` and `SkillSet`: `from_format` (`generateSkill`), `core_syntax` (`generateCoreSkill`), `from_catalog` (`generateCatalogSkill`), `skill_set` (`generateSkillSet`), and `export_to_directory` (`SkillSet.exportToDirectory`). Resolve all file paths in this suite from `REPO_ROOT`. Run `export_to_directory` in a temporary directory populated with `args.pre_existing_files` to check that exporting deletes only managed skill subdirectories and preserves sibling files (even on `expect_error`). @@ -356,8 +361,9 @@ Inspect these existing implementations for concrete code examples: 3. [python/a2ui_agent/tests/conformance/test_catalog_conformance.py](../python/a2ui_agent/tests/conformance/test_catalog_conformance.py) 4. [python/a2ui_agent/tests/conformance/test_skill_conformance.py](../python/a2ui_agent/tests/conformance/test_skill_conformance.py) 5. [python/a2ui_agent/tests/conformance/test_scaling_conformance.py](../python/a2ui_agent/tests/conformance/test_scaling_conformance.py) - 6. [python/a2ui_agent/tests/conformance/builder_suite.py](../python/a2ui_agent/tests/conformance/builder_suite.py) and [python/a2ui_agent/tests/conformance/macro_suite.py](../python/a2ui_agent/tests/conformance/macro_suite.py) - 7. [python/a2ui_agent/tests/test_specification_roundtrip.py](../python/a2ui_agent/tests/test_specification_roundtrip.py) + 6. [python/a2ui_agent/tests/conformance/test_agent_function_conformance.py](../python/a2ui_agent/tests/conformance/test_agent_function_conformance.py) + 7. [python/a2ui_agent/tests/conformance/builder_suite.py](../python/a2ui_agent/tests/conformance/builder_suite.py) and [python/a2ui_agent/tests/conformance/macro_suite.py](../python/a2ui_agent/tests/conformance/macro_suite.py) + 8. [python/a2ui_agent/tests/test_specification_roundtrip.py](../python/a2ui_agent/tests/test_specification_roundtrip.py) 3. Inference SDK, TypeScript (`vitest` suites with strict `KNOWN_FAILURES` vs `UNSUPPORTED` classification): 1. [typescript/agents/a2ui_agent/tests/conformance/loader.ts](../typescript/agents/a2ui_agent/tests/conformance/loader.ts), [typescript/agents/a2ui_agent/tests/conformance/fixtures.ts](../typescript/agents/a2ui_agent/tests/conformance/fixtures.ts), and [typescript/agents/a2ui_agent/tests/conformance/suite-helpers.ts](../typescript/agents/a2ui_agent/tests/conformance/suite-helpers.ts) 2. [typescript/agents/a2ui_agent/tests/conformance/direct-json-conformance.test.ts](../typescript/agents/a2ui_agent/tests/conformance/direct-json-conformance.test.ts) and [typescript/agents/a2ui_agent/tests/conformance/direct-json-response-streaming.test.ts](../typescript/agents/a2ui_agent/tests/conformance/direct-json-response-streaming.test.ts) diff --git a/conformance/conformance_schema.json b/conformance/conformance_schema.json index d59524653f..c6d4d1b8bd 100644 --- a/conformance/conformance_schema.json +++ b/conformance/conformance_schema.json @@ -32,6 +32,10 @@ {"$ref": "#/$defs/DispatchActionTest"}, {"$ref": "#/$defs/RouteAgentFunctionTest"}, {"$ref": "#/$defs/DeriveTest"}, + {"$ref": "#/$defs/DeclareAgentFunctionsTest"}, + {"$ref": "#/$defs/HandleAgentFunctionTest"}, + {"$ref": "#/$defs/ValidatePayloadTest"}, + {"$ref": "#/$defs/ProcessRendererMessageTest"}, {"$ref": "#/$defs/RenderSurfaceTest"} ] }, @@ -2249,6 +2253,257 @@ }, "required": ["action", "args"] }, + "AgentFunctionBehavior": { + "description": "What an agent function does when it runs, in place of a handler, so every SDK runs the same function. Exactly one key.", + "type": "object", + "properties": { + "returns": { + "description": "Returns this value, which may be null." + }, + "returns_args": { + "const": true, + "description": "Returns the arguments it received, after validation and with the schema's defaults applied." + }, + "returns_context": { + "const": true, + "description": "Returns the context of the call: {catalogId, surfaceId, functionCallId}." + }, + "raises": { + "type": "object", + "description": "Fails with `message`. With `code`, it fails the way the SDK lets a handler choose the RPC error code of the response; without one, it fails with an ordinary error.", + "properties": { + "code": {"type": "string"}, + "message": {"type": "string"} + }, + "required": ["message"], + "additionalProperties": false + } + }, + "minProperties": 1, + "maxProperties": 1, + "additionalProperties": false + }, + "AgentFunctionDeclaration": { + "type": "object", + "description": "An agent function: a catalog function that runs on the agent. Built directly from `args`, it is rendererOnly. With `from_api: true`, it is built from the catalog's own declaration of the function (keeping that declaration's allowedCallers), or, if the catalog declares none, from a declaration made of `returnType`, `args` and `description`, which the catalog must refuse.", + "properties": { + "name": { + "type": "string", + "description": "The function's name, when it differs from its key, so a case can give two entries one name." + }, + "from_api": { + "const": true, + "description": "Build the agent function from the catalog's declaration of it (AgentFunction.from_api)." + }, + "description": {"type": "string"}, + "returnType": {"type": "string"}, + "args": { + "type": "object", + "description": "JSON Schema of the function's args." + }, + "behavior": {"$ref": "#/$defs/AgentFunctionBehavior"} + }, + "required": ["behavior"], + "anyOf": [{"required": ["args"]}, {"required": ["from_api"]}], + "additionalProperties": false + }, + "AgentFunctionCatalog": { + "type": "object", + "description": "A catalog document and the agent functions the agent declares on its copy of it.", + "properties": { + "catalog": {"$ref": "#/$defs/AgentCatalogRef"}, + "agent_functions": { + "type": "object", + "description": "The agent functions, by name.", + "additionalProperties": {"$ref": "#/$defs/AgentFunctionDeclaration"} + } + }, + "required": ["catalog"], + "additionalProperties": false + }, + "DeclareAgentFunctionsTest": { + "type": "object", + "description": "Declaring agent functions on a catalog, and what the catalog then lists.", + "properties": { + "name": {"type": "string"}, + "description": {"type": "string"}, + "action": {"const": "declare_agent_functions"}, + "args": { + "type": "object", + "properties": { + "catalog": {"$ref": "#/$defs/AgentFunctionCatalog"} + }, + "required": ["catalog"], + "additionalProperties": false + }, + "expect": { + "type": "object", + "description": "Exhaustive name lists, compared without regard to order.", + "properties": { + "functions": { + "type": "array", + "items": {"type": "string"}, + "description": "The catalog's functions, agent functions included." + }, + "agent_functions": { + "type": "array", + "items": {"type": "string"}, + "description": "The catalog's agent functions." + }, + "document_functions": { + "type": "array", + "items": {"type": "string"}, + "description": "The functions the catalog's document lists, which is what a renderer would load." + } + }, + "required": ["functions", "agent_functions", "document_functions"], + "additionalProperties": false + }, + "expect_error": {"$ref": "#/$defs/ExpectError"} + }, + "required": ["name", "action", "args"], + "oneOf": [{"required": ["expect"]}, {"required": ["expect_error"]}], + "additionalProperties": false + }, + "HandleAgentFunctionTest": { + "type": "object", + "description": "Answering one callAgentFunction message with the agent functions of `catalogs`.", + "properties": { + "name": {"type": "string"}, + "description": {"type": "string"}, + "action": {"const": "handle_agent_function"}, + "args": { + "type": "object", + "properties": { + "catalogs": { + "type": "array", + "items": {"$ref": "#/$defs/AgentFunctionCatalog"}, + "minItems": 1 + }, + "message": { + "type": "object", + "description": "The callAgentFunction message, as received. It may be malformed." + } + }, + "required": ["catalogs", "message"], + "additionalProperties": false + }, + "expect": { + "type": "object", + "description": "The agentFunctionResponse body of the response. `error.message` is a substring of the message sent.", + "properties": { + "functionCallId": {"type": "string"}, + "value": {}, + "error": { + "type": "object", + "properties": { + "code": {"type": "string"}, + "message": {"type": "string"} + }, + "required": ["code"], + "additionalProperties": false + } + }, + "required": ["functionCallId"], + "oneOf": [{"required": ["value"]}, {"required": ["error"]}], + "additionalProperties": false + }, + "expect_error": { + "$ref": "#/$defs/ExpectError", + "description": "Handling the message fails instead of answering it." + } + }, + "required": ["name", "action", "args"], + "oneOf": [{"required": ["expect"]}, {"required": ["expect_error"]}], + "additionalProperties": false + }, + "ValidatePayloadTest": { + "type": "object", + "description": "Validating model output against catalogs that declare agent functions, as the agent does before sending it.", + "properties": { + "name": {"type": "string"}, + "description": {"type": "string"}, + "action": {"const": "validate_payload"}, + "args": { + "type": "object", + "properties": { + "catalogs": { + "type": "array", + "items": {"$ref": "#/$defs/AgentFunctionCatalog"}, + "minItems": 1 + }, + "messages": { + "type": "array", + "items": {"type": "object"}, + "description": "The A2UI messages to validate." + } + }, + "required": ["catalogs", "messages"], + "additionalProperties": false + }, + "expect_valid": {"const": true}, + "expect_error": {"$ref": "#/$defs/ExpectError"} + }, + "required": ["name", "action", "args"], + "oneOf": [{"required": ["expect_valid"]}, {"required": ["expect_error"]}], + "additionalProperties": false + }, + "ProcessRendererMessageTest": { + "type": "object", + "description": "Processing one renderer-to-agent message with RendererToAgentMessageProcessor, built from the agent functions of `catalog` (or no catalog) and the callbacks named in `callbacks`.", + "properties": { + "name": {"type": "string"}, + "description": {"type": "string"}, + "action": {"const": "process_renderer_message"}, + "args": { + "type": "object", + "properties": { + "catalog": {"$ref": "#/$defs/AgentFunctionCatalog"}, + "callbacks": { + "type": "array", + "description": "The callbacks the processor has: on_action, on_error and on_renderer_function_response. Each records the message it receives.", + "items": {"enum": ["action", "error", "rendererFunctionResponse"]}, + "uniqueItems": true + }, + "message": { + "description": "The message, as received. It may be malformed or not an object." + } + }, + "required": ["message"], + "additionalProperties": false + }, + "expect": { + "type": "object", + "properties": { + "handled": {"type": "boolean"}, + "replies": { + "type": "array", + "items": {"type": "object"}, + "description": "The messages sent back, in order. Compared exactly, except that an `error.message` in an expected reply is a substring of the one sent and may be left out. Absent means none." + }, + "callback": { + "type": "object", + "description": "The one callback that ran, and the message it received, serialized. Absent means no callback ran.", + "properties": { + "kind": {"enum": ["action", "error", "rendererFunctionResponse"]}, + "message": {"type": "object"} + }, + "required": ["kind", "message"], + "additionalProperties": false + } + }, + "required": ["handled"], + "additionalProperties": false + }, + "expect_error": { + "$ref": "#/$defs/ExpectError", + "description": "Processing the message fails, and no callback runs." + } + }, + "required": ["name", "action", "args"], + "oneOf": [{"required": ["expect"]}, {"required": ["expect_error"]}], + "additionalProperties": false + }, "ExpectError": { "description": "Expected error specification (error message string, regex, or structured error object).", "oneOf": [ diff --git a/conformance/core/message_processor_v1_0.yaml b/conformance/core/message_processor_v1_0.yaml index 3c12224447..a670408f8f 100644 --- a/conformance/core/message_processor_v1_0.yaml +++ b/conformance/core/message_processor_v1_0.yaml @@ -1303,6 +1303,62 @@ s: exists: true +- name: test_v10_strict_function_call_to_schema_only_function_of_surface_catalog + description: >- + Under strictMode: true, a function call that names no catalogId is checked against the surface default catalog, and a function that catalog declares without an implementation (schema-only, as for a function that runs on the agent) passes. Strict validation rejects only functions no catalog declares; a renderer that validates strictly can declare the agent's functions this way and still route calls to them via callAgentFunction. + action: process_messages + protocolVersion: "1.0" + strictMode: true + catalogs: + - catalogId: "agent-fns-v10" + protocolVersion: "1.0" + components: + Badge: + type: object + properties: + component: + const: Badge + label: + $ref: "common_types.json#/$defs/DynamicString" + required: ["component", "label"] + functions: + fetchGreeting: + type: object + returnType: string + properties: + "@call": + const: fetchGreeting + args: + type: object + properties: + name: + type: string + required: ["name"] + additionalProperties: false + required: ["@call", "args"] + messages: + - version: "1.0" + createSurface: + surfaceId: "s" + catalogId: "agent-fns-v10" + - version: "1.0" + updateComponents: + surfaceId: "s" + components: + - id: "root" + component: "Badge" + label: + "@call": "fetchGreeting" + args: + name: "Ada" + expect: + surfaces: + s: + exists: true + components: + - id: "root" + component: "Badge" + - name: test_v10_function_call_naming_unknown_catalog_errors description: >- A function call that names a catalogId the client does not have is rejected. diff --git a/dart/a2ui_agent/test/conformance/agent_functions_conformance_test.dart b/dart/a2ui_agent/test/conformance/agent_functions_conformance_test.dart new file mode 100644 index 0000000000..3639b61536 --- /dev/null +++ b/dart/a2ui_agent/test/conformance/agent_functions_conformance_test.dart @@ -0,0 +1,50 @@ +// Copyright 2024 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// https://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +import 'package:test/test.dart'; + +import 'suites.dart'; + +/// Registers the shared `conformance/agent/agent_functions.yaml` suite. +/// +/// The suite covers agent functions and the routing of renderer messages, +/// which this SDK does not implement yet, so every case is +/// skipped with the reason [_unimplementedActions] gives. A case whose action +/// has no reason fails instead of passing without running. +void main() { + const path = 'agent/agent_functions.yaml'; + group('conformance $path', () { + final List> cases = loadSuite(path); + test('suite is not empty', () => expect(cases, isNotEmpty)); + for (final testCase in cases) { + final action = testCase['action']! as String; + test( + '$action · ${testCase['name']}', + () => fail('Unhandled conformance action: $action'), + skip: _unimplementedActions[action], + ); + } + }); +} + +/// The suite's actions this SDK has no implementation for, with the reason. +const Map _unimplementedActions = { + 'declare_agent_functions': _notImplemented, + 'handle_agent_function': _notImplemented, + 'validate_payload': _notImplemented, + 'process_renderer_message': _notImplemented, +}; + +const String _notImplemented = + 'this SDK does not implement agent functions yet'; diff --git a/python/a2ui_agent/tests/conformance/test_agent_function_conformance.py b/python/a2ui_agent/tests/conformance/test_agent_function_conformance.py new file mode 100644 index 0000000000..3265b1ce1d --- /dev/null +++ b/python/a2ui_agent/tests/conformance/test_agent_function_conformance.py @@ -0,0 +1,303 @@ +# Copyright 2024 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Runs the shared `conformance/agent/agent_functions.yaml` suite. + +A case declares each agent function with a scripted behaviour, which +`_handler` turns into a handler. Every message goes through +`RendererToAgentMessageProcessor`. +""" + +from __future__ import annotations + +import asyncio +from collections.abc import Callable, Mapping +import os +import re +from typing import Any + +import pytest + +from a2ui.core import ( + A2uiCatalogError, + A2uiRpcError, + A2uiValidationError, + AgentFunction, + AgentFunctionContext, + CatalogApi, +) +from a2ui.core.catalog import FunctionApi +from a2ui.processor import InMemoryCatalogProvider +from a2ui.rpc import RendererToAgentMessageProcessor +from a2ui.utils import validate_payload + +from .conformance_helpers import load_conformance_json, load_conformance_yaml + +SUITE = "agent/agent_functions.yaml" + +CATEGORY_TO_EXCEPTION: dict[str, type[Exception]] = { + "CatalogError": A2uiCatalogError, + "ValidationError": A2uiValidationError, +} + +AgentFunctionHandler = Callable[[dict[str, Any], AgentFunctionContext], Any] + + +def _load_cases() -> list[Any]: + cases = load_conformance_yaml(SUITE) + return [pytest.param(case, id=case["name"]) for case in cases] + + +CASES = _load_cases() + + +def test_suite_is_not_empty() -> None: + assert CASES + + +def _document(ref: str | Mapping[str, Any]) -> tuple[dict[str, Any], str]: + """The catalog document a case names, and the id to use if it has none.""" + if isinstance(ref, str): + return load_conformance_json(ref), os.path.basename(ref).removesuffix(".json") + return dict(ref), "inline" + + +def _handler(behavior: Mapping[str, Any]) -> AgentFunctionHandler: + """The handler that carries out a case's scripted behaviour.""" + if "returns" in behavior: + value = behavior["returns"] + + async def returns(args: dict[str, Any], context: AgentFunctionContext) -> Any: + return value + + return returns + if behavior.get("returns_args"): + + async def returns_args( + args: dict[str, Any], context: AgentFunctionContext + ) -> Any: + return args + + return returns_args + if behavior.get("returns_context"): + + async def returns_context( + args: dict[str, Any], context: AgentFunctionContext + ) -> Any: + return { + "catalogId": context.catalog_id, + "surfaceId": context.surface_id, + "functionCallId": context.function_call_id, + } + + return returns_context + if "raises" in behavior: + failure = behavior["raises"] + + async def raises(args: dict[str, Any], context: AgentFunctionContext) -> Any: + if "code" in failure: + raise A2uiRpcError(failure["message"], code=failure["code"]) + raise RuntimeError(failure["message"]) + + return raises + raise ValueError(f"Unknown agent function behavior: {behavior}") + + +def _agent_function( + key: str, declaration: Mapping[str, Any], catalog: CatalogApi +) -> AgentFunction: + """The agent function a declaration describes, for `catalog`.""" + name = declaration.get("name", key) + handler = _handler(declaration["behavior"]) + if declaration.get("from_api"): + api = catalog.functions.get(name) or FunctionApi( + name, + return_type=declaration.get("returnType", "any"), + schema=declaration.get("args"), + description=declaration.get("description"), + ) + return AgentFunction.from_api(api, handler) + return AgentFunction( + name, + handler=handler, + return_type=declaration.get("returnType", "any"), + schema=declaration["args"], + description=declaration.get("description"), + ) + + +def _catalog(entry: Mapping[str, Any]) -> CatalogApi: + """The catalog a case's entry describes, with its agent functions.""" + document, fallback_id = _document(entry["catalog"]) + catalog = InMemoryCatalogProvider( + document, + protocol_version=None if "protocolVersion" in document else "1.0", + catalog_id=None if "catalogId" in document else fallback_id, + ).load() + return catalog.copy_with( + agent_functions=[ + _agent_function(key, declaration, catalog) + for key, declaration in entry.get("agent_functions", {}).items() + ] + ) + + +def _assert_error_matches(error: BaseException, expected: Mapping[str, Any]) -> None: + if "category" in expected: + assert isinstance(error, CATEGORY_TO_EXCEPTION[expected["category"]]), error + if "code" in expected: + assert getattr(error, "code", None) == expected["code"], error + if "message" in expected: + assert re.search(re.escape(expected["message"]), str(error)), error + + +def _run_declare(case: Mapping[str, Any]) -> None: + entry = case["args"]["catalog"] + if "expect_error" in case: + with pytest.raises(Exception) as excinfo: + _catalog(entry) + _assert_error_matches(excinfo.value, case["expect_error"]) + return + catalog = _catalog(entry) + expected = case["expect"] + assert sorted(catalog.functions) == sorted(expected["functions"]) + assert sorted(catalog.agent_functions) == sorted(expected["agent_functions"]) + document_functions = catalog.to_json().get("functions", {}) + assert sorted(document_functions) == sorted(expected["document_functions"]) + + +def _run_handle(case: Mapping[str, Any]) -> None: + args = case["args"] + processor = RendererToAgentMessageProcessor( + [_catalog(entry) for entry in args["catalogs"]] + ) + + if "expect_error" in case: + with pytest.raises(Exception) as excinfo: + asyncio.run(processor.process_message(args["message"])) + _assert_error_matches(excinfo.value, case["expect_error"]) + return + + result = asyncio.run(processor.process_message(args["message"])) + assert result.handled + assert len(result.replies) == 1, result.replies + dumped = result.replies[0].model_dump(by_alias=True, exclude_unset=True) + assert dumped["version"] == "1.0" + response = dumped["agentFunctionResponse"] + expected = case["expect"] + assert response["functionCallId"] == expected["functionCallId"] + if "error" in expected: + assert "value" not in response, response + error = response["error"] + assert error["code"] == expected["error"]["code"], error + if "message" in expected["error"]: + assert expected["error"]["message"] in error["message"], error + else: + assert "error" not in response, response + assert "value" in response, response + assert response["value"] == expected["value"] + + +def _run_validate(case: Mapping[str, Any]) -> None: + args = case["args"] + catalogs = [_catalog(entry) for entry in args["catalogs"]] + if "expect_error" in case: + with pytest.raises(Exception) as excinfo: + validate_payload(catalogs, args["messages"]) + _assert_error_matches(excinfo.value, case["expect_error"]) + return + assert case["expect_valid"] is True + validate_payload(catalogs, args["messages"]) + + +_CALLBACK_OPTIONS = { + "action": "on_action", + "error": "on_error", + "rendererFunctionResponse": "on_renderer_function_response", +} + + +def _assert_reply_matches(actual: Any, expected: Any) -> None: + """Compares a reply exactly, but an expected `error.message` as a substring.""" + if isinstance(expected, Mapping): + assert isinstance(actual, Mapping), actual + if "code" in expected and "message" not in expected: + actual = {k: v for k, v in actual.items() if k != "message"} + assert set(actual) == set(expected), (actual, expected) + for key, value in expected.items(): + if key == "message" and "code" in expected and isinstance(value, str): + assert value in actual[key], (actual, expected) + else: + _assert_reply_matches(actual[key], value) + else: + assert actual == expected, (actual, expected) + + +def _run_process(case: Mapping[str, Any]) -> None: + args = case["args"] + received: list[tuple[str, Any]] = [] + + def record(kind: str) -> Callable[[Any], None]: + return lambda message: received.append((kind, message)) + + catalogs = [_catalog(args["catalog"])] if "catalog" in args else [] + processor = RendererToAgentMessageProcessor( + catalogs, + **{_CALLBACK_OPTIONS[kind]: record(kind) for kind in args.get("callbacks", [])}, + ) + + if "expect_error" in case: + with pytest.raises(Exception) as excinfo: + asyncio.run(processor.process_message(args["message"])) + _assert_error_matches(excinfo.value, case["expect_error"]) + assert received == [] + return + + result = asyncio.run(processor.process_message(args["message"])) + expected = case["expect"] + assert result.handled is expected["handled"] + replies = [ + reply.model_dump(by_alias=True, exclude_unset=True) for reply in result.replies + ] + expected_replies = expected.get("replies", []) + assert len(replies) == len(expected_replies), replies + for reply, expected_reply in zip(replies, expected_replies): + _assert_reply_matches(reply, expected_reply) + if "callback" in expected: + assert len(received) == 1, received + kind, message = received[0] + assert kind == expected["callback"]["kind"] + assert ( + message.model_dump(by_alias=True, exclude_unset=True) + == expected["callback"]["message"] + ) + else: + assert received == [] + + +RUNNERS: dict[str, Callable[[Mapping[str, Any]], None]] = { + "declare_agent_functions": _run_declare, + "handle_agent_function": _run_handle, + "validate_payload": _run_validate, + "process_renderer_message": _run_process, +} + + +@pytest.mark.parametrize("case", CASES) +def test_agent_function_conformance(case: Mapping[str, Any]) -> None: + action = case["action"] + runner = RUNNERS.get(action) + if runner is None: + raise ValueError(f"Unhandled conformance action: {action}") + runner(case) diff --git a/typescript/agents/a2ui_agent/tests/conformance/agent-functions.test.ts b/typescript/agents/a2ui_agent/tests/conformance/agent-functions.test.ts new file mode 100644 index 0000000000..cfc3a4c5b6 --- /dev/null +++ b/typescript/agents/a2ui_agent/tests/conformance/agent-functions.test.ts @@ -0,0 +1,47 @@ +/* + * Copyright 2024 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import {describe, expect, test} from 'vitest'; + +import {classify, loadCases} from './loader.js'; +import {conformancePath} from './suite-helpers.js'; + +/** + * `agent_functions.yaml` covers agent functions and the routing of renderer messages, + * which this SDK does not implement yet. Every case is registered as skipped with the reason + * `classify` gives, so the gap shows in the test report, and a case with an action that has + * no reason fails instead of passing without running. + */ +describe('Conformance: agent_functions.yaml', () => { + const cases = loadCases([conformancePath('agent/agent_functions.yaml')]); + + test('suite is not empty', () => { + expect(cases.length).toBeGreaterThan(0); + }); + + for (const testCase of cases) { + const title = `${testCase.action} · ${testCase.name}`; + const verdict = classify(testCase); + const run = () => { + throw new Error(`Unhandled conformance action: ${testCase.action}`); + }; + if (verdict.runnable) { + test(title, run); + } else { + test.skip(`${title} (${verdict.reason})`, run); + } + } +}); diff --git a/typescript/agents/a2ui_agent/tests/conformance/loader.ts b/typescript/agents/a2ui_agent/tests/conformance/loader.ts index 9b959ac9f6..16f5d20cf2 100644 --- a/typescript/agents/a2ui_agent/tests/conformance/loader.ts +++ b/typescript/agents/a2ui_agent/tests/conformance/loader.ts @@ -116,7 +116,14 @@ const SUPPORTED_FORMATS = new Set(['direct_json', 'express']); * Conformance actions this SDK has no implementation for. Their cases are skipped with the * reason given, rather than falling through the harness and passing without running. */ -const UNIMPLEMENTED_ACTIONS = new Map([]); +const UNIMPLEMENTED_ACTIONS = new Map( + [ + 'declare_agent_functions', + 'handle_agent_function', + 'validate_payload', + 'process_renderer_message', + ].map((action): [string, string] => [action, 'this SDK does not implement agent functions yet']), +); export function classify(testCase: TestCase) { const superseded = SUPERSEDED.get(testCase.name) ?? SUPERSEDED_ACTIONS.get(testCase.action); diff --git a/typescript/core/web_core/tests/conformance/conformance_test.mjs b/typescript/core/web_core/tests/conformance/conformance_test.mjs index 9d458a3f7a..70c73b28e1 100644 --- a/typescript/core/web_core/tests/conformance/conformance_test.mjs +++ b/typescript/core/web_core/tests/conformance/conformance_test.mjs @@ -259,6 +259,10 @@ const UNIMPLEMENTED_ACTIONS = new Map([ ['transform_catalog', 'catalog transformers are agent-side only'], ['provide_catalog', 'catalog providers are agent-side only'], ['resolve_catalogs', 'catalog resolution is agent-side only'], + ['declare_agent_functions', 'agent functions are agent-side only'], + ['handle_agent_function', 'agent functions are agent-side only'], + ['validate_payload', 'validating model output is agent-side only'], + ['process_renderer_message', 'processing renderer messages is agent-side only'], ['common_types_schema', 'web_core does not generate the common types schema from its own models'], [ 'agent_to_renderer_schema',