feat(gemini-planner): support native Gemini tools (google_search, url_context, code_execution, google_maps)#2
Open
christopherwxyz wants to merge 1 commit into
Open
Conversation
The Gemini planner currently builds its Tools list exclusively from
registered AX subagents (via agentsToTools). This commit lets ax.yaml
opt into Gemini's native tool surfaces (`google_search`, `url_context`,
`code_execution`, `google_maps`) by listing them in
GeminiConfig.Tools []string, which already exists in config.go but
was only consumed by the standalone gemini agent type.
Three correctness points the implementation gets right (each driven
by an empirical Vertex error, with a test):
1. Single-Tool merge. Per Gemini's tool-combination docs, all
FunctionDeclarations + the native tool field must live on the
SAME *genai.Tool object. agentsToTools naturally produces one
Tool per registered agent, so we flatten + merge in process().
2. Empty-Tool guard. When both registry and config.Tools are empty,
don't send Tools: []*genai.Tool{ {} } — Vertex rejects with
400 INVALID_ARGUMENT ("Tool must contain at least one of
function_declarations, google_search, url_context, code_execution").
3. Native-field preservation from agentsToTools' nativeTools variadic.
The merge loop copies .GoogleSearch / .URLContext /
.CodeExecution / .GoogleMaps from each rawTool, not just
.FunctionDeclarations. First-non-nil wins with a stderr warning
on collision. Latent today (no in-tree caller exercises the
variadic) but the contract is what existing callers documented.
Tests:
TestProcess_AppendsNativeToolsFromConfig
TestProcess_MergesNativeToolsWithFunctionDeclarations
TestProcess_NoToolsWhenRegistryAndNativeBothEmpty
TestProcess_MergePreservesNativeFieldsFromRawTools
Vertex caveat documented in code: IncludeServerSideToolInvocations
(added in v1.51 of google.golang.org/genai) is documented and SDK-
enforced as Gemini-Developer-API-only. Vertex auto-execution of
native tools is gated by Google; our impl works around this with a
separate agent-as-tool pattern in downstream consumers, but that's
out of scope here.
christopherwxyz
added a commit
that referenced
this pull request
May 25, 2026
…earch_agent) These two examples were Slack-specific / Vertex-specific application code that has been extracted into the uno-infra monorepo where they belong (deployed alongside their K8s manifests). Keeping them in the AX fork's examples/ created merge conflicts on every upstream pull from google/ax. With them gone, the fork's divergence from upstream is bounded to: - the three PR branches (#1 #2 #3) of generic AX improvements - feat/agent-sandbox-backend's tracking branch examples/python_sandbox_agent/ stays — it's a generic reference implementation of the agent-sandbox backend pattern, useful upstream.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Plumb Gemini's native server-side tool surfaces —
google_search,url_context,code_execution,google_maps— throughgeminiPlannerAgent. Callers opt in by listing the tool name inGeminiConfig.Tools []string(an existing config field that until now was only honoured by the standalonegeminiagent type, not the planner).Motivation
The planner currently builds its
Toolslist exclusively from registered AX subagents viaagentsToTools. There was no way to give the planner web grounding, URL fetching, code execution, or maps without writing a dedicated subagent. With this change, an operator can drop a one-liner intoax.yamland get Gemini-side grounding for free:Three correctness points (each backed by a test)
Single-Tool merge. Per Gemini's tool-combination docs, all
FunctionDeclarations+ the native tool field must live on the same*genai.Toolobject.agentsToToolsnaturally produces one*genai.Toolper registered agent, soprocess()flattens them and merges native fields onto a singlemergedTool. Verified empirically againstgemini-3-flash-previewon Vertex — splitting across multiple*genai.Toolentries causes Gemini 3 to emit the native tool's name as a regular function call instead of auto-executing it server-side.Test:
TestProcess_MergesNativeToolsWithFunctionDeclarations.Empty-Tool guard. When the registry is empty AND no native tools are configured, the planner must send no tools at all — not
Tools: []*genai.Tool{ {} }. Vertex rejects the zero-valued Tool with400 INVALID_ARGUMENT: Tool must contain at least one of function_declarations, google_search, url_context, code_execution.Test:
TestProcess_NoToolsWhenRegistryAndNativeBothEmpty.Native-field preservation from
agentsToTools'nativeToolsvariadic. The merge loop copies.GoogleSearch / .URLContext / .CodeExecution / .GoogleMapsfrom each raw Tool — not justFunctionDeclarations. First-non-nil wins with a stderr warning on collision. No in-tree caller exercises the variadic yet, but the contract is what existing callers documented, and silently dropping native fields would be a confusing future bug.Test:
TestProcess_MergePreservesNativeFieldsFromRawTools.Plus a smoke test that ties (1) and the config plumbing together end-to-end:
TestProcess_AppendsNativeToolsFromConfig.Vertex caveat (documented in code, not implemented here)
IncludeServerSideToolInvocations(added ingoogle.golang.org/genaiv1.51) is documented and SDK-enforced as Gemini-Developer-API-only. Vertex auto-execution of native tools is gated by Google; downstream consumers work around this with a separate agent-as-tool pattern, but that's out of scope for this PR.Relationship to the structured-subagent-args PR
This PR is the sibling of
feat(proto): pass subagent prompt/history as structured AgentStart fields(#TBD). The two PRs touch different functions ininternal/gemini/gemini_planner.go(processhere,handleSubagentCallthere) and do not conflict at the line level. They can land in either order.Files touched
No proto changes, no new examples, no SDK bump required (existing
google.golang.org/genai v1.43.0already exposesGoogleSearch / URLContext / ToolCodeExecution / GoogleMapson*genai.Tool).Test plan
go build ./...cleango test -race ./internal/gemini/...passes (new tests + existing)go test -race ./...passes (full repo)gemini-3-flash-previewwithgoogle_searchenabled (covered by author in feature branch; behaviour confirmed before extracting this PR)