Repository navigation
feat(agent): add local decision-model UI synthesizer (a2ui.inference_formats.decision) - #3339
Draft
gspencergoog wants to merge 30 commits into
Draft
gspencergoog wants to merge 30 commits into
gspencergoog wants to merge 30 commits into
Conversation
…_formats.decision`) Adds a catalog-agnostic local decision-model synthesis pipeline for A2UI v0.9 and v1.0 surfaces in `a2ui-agent-sdk`. Agent servers can route template selection, leaf binding, and container assembly through a local classifier or logprob endpoint. Low-confidence predictions fall back to a foundation model.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Review feedback on PR a2ui-project#3339 identified an async-safety hazard when calibrating thresholds on a shared synthesizer instance, a zero logit margin on single-candidate step-0 logprob queries, unguarded float conversions during probability normalization, and sub-optimal pointer selection and camelCase formatting in check rules. - Add an optional `config` override parameter to `synthesize()` and pass the permissive configuration per call in `_collect_evaluated_outputs()` without mutating `synthesizer.config`. - Set `top_logit_margin` to `float("inf")` in `StepZeroLogprobBackend.predict` when only a single candidate is present. - Catch `(ValueError, TypeError)` when converting probability values in `_normalize_probabilities`. - Rank matched pointers by token overlap for optional `any` slots in `_select_optional_pointer` and select check function argument pointers via `_best_pointer_by_tokens` in `_default_arg_value`. - Preserve camelCase function names when formatting check failure messages in `_build_checks`. - Add unit tests covering each fix in `test_backends.py`, `test_calibration.py`, and `test_synthesizer.py`.
… backend Enrich DecisionQuery context with catalog property descriptions and slot metadata, enabling backends to score candidates without filesystem access. Add an in-process LocalCrossEncoderBackend powered by ONNX Runtime under the optional [decision-local] extra in a2ui-agent-sdk, allowing local CPU decision modeling without external FastAPI sidecars or PyTorch. Extend JevHttpBackend with native /v1/systemone format support, structured choice query building, and non-blocking cached Application Default Credentials (ADC) token resolution. Add create_decision_backend() and DecisionSynthesizer.from_env() for environment-driven backend initialization.
…rgonomics Plug the decision format into the existing a2ui_agent pipeline and remove the plumbing every adopter had to write by hand. - DecisionFormat / DecisionFormatFactory implement InferenceFormat and InferenceFormatFactory, so A2uiRequestProcessor(catalogs, format_factory=DecisionFormatFactory()) works like the Express and Direct JSON factories. DecisionPromptGenerator asks for <a2ui> decision blocks and DecisionParser renders them through DecisionSynthesizer. - DecisionRenderTool supplies a JSON Schema declaration and run_async / run_sync runners for MCP, ADK, and function-calling agents. A shared DecisionBlock keeps the prompt, parser, and tool on the same field names and decodes stringified payloads. - DecisionSynthesizer keeps every existing signature and adds an optional backend (defaulting to create_decision_backend()), CatalogDecisionGraph input, synthesize_sync() for callers inside or outside a running loop, explicit template_id, JSON-string and array payloads, and abstention fallbacks (fallback_dsl, fallback_fn, fallback_card) reported through SynthesisOutput.fallback_source. The format and tool default fallback_card to True because an empty render is an error there. - DecisionTask, DecisionWireFormat, and DecisionBackendType replace raw string literals. They override __str__/__format__ because on Python 3.11+ an f-string of a plain (str, Enum) member yields Class.MEMBER, which broke the mock backend's scripted lookups. - Backends support async with; HTTP backends re-create their httpx client per event loop and JevHttpBackend caches ADC tokens per instance instead of in a module global. - TemplateRegistry.promote_synthesized reads v1.0 createSurface envelopes with embedded components and v0.8 surfaceUpdate envelopes. - Decompose create_decision_backend (complexity 34 to 8) and _choose_child_component (30 to 4).
Two defects and several ergonomic refinements from the second API review of a2ui.inference_formats.decision. - DecisionBackendType.ONNX was a member that create_decision_backend rejected; it is now an alias of the local backend and appears in the error message's choices. - DecisionFormatFactory built a new backend for every create_format call, so A2uiGenerator would have loaded one ONNX model per renderer request. The factory now resolves the backend once and shares it; the catalog graph is still built per format because catalogs are negotiated per renderer. - Drop parse_json_strings from coerce_decision_payload, synthesize, and synthesize_sync: a string payload is never valid undecoded, so the flag only turned a usable input into an error. - SynthesisOutput.fallback_source is a FallbackSource enum (DSL, CALLBACK, CARD) instead of a three-valued str; members still compare equal to the previous strings. - DecisionWireFormat gains AUTO so JevHttpBackend.wire_format is no longer a Literal | Enum union. - DecisionQuery.task is annotated DecisionTask | str and FallbackCallback may return wire-format dicts, which to_message_models already accepted. - DECISION_RULES and the tool's payload schema state that a top-level array is stored under the items key; surface-id docstrings explain precedence. - DecisionRenderTool.from_env is renamed create: it takes an explicit backend, so the old name no longer described it. Its only caller is the a2ui-pane worker, updated on its branch.
… roundtrip validation - Manage owned httpx.AsyncClient instances per event loop in _HttpClientOwner and close temporary clients inside DecisionSynthesizer.synthesize_sync via aclose_current_loop() - Add aclose() and async context manager methods to DecisionBackend and DecisionFormatFactory - Offload LocalCrossEncoderBackend._ensure_loaded() inside asyncio.to_thread(_infer_sync) and serialize JevHttpBackend ADC refresh with an asyncio.Lock - Enforce allow_actions gating across leaf, child, and fallback synthesis and abstain cleanly when a container has no eligible child candidates - Validate rendered templates and fallback_fn messages against catalog schemas via validate_payload - Emit non-empty synthetic intent in decision_block_from_messages, validate prompt examples in DecisionPromptGenerator.generate(), and assign distinct default surface IDs across multi-block arrays in DecisionParser.compile() - Re-evaluate Stage 1 shadowing during threshold calibration, clamp recommended_mutable_threshold >= best.threshold, and normalize v0.8 component envelopes in TemplateRegistry.promote_synthesized
Land the data contracts and shared types that the archetype-driven layout stage builds on, so later tracks can work on disjoint modules. Contracts: catalogs/decision/v1 holds the catalog-agnostic archetype grammar (14 archetypes, 6 families, 19 blocks) with JSON Schemas for it and for per-catalog pattern files; catalogs/basic/v1/patterns.json says how the basic catalog draws 17 abstract patterns and lists both the v1.0 and v0.9 basic catalog ids. The Python SDK vendors copies under decision/assets/, kept identical by scripts/sync_decision_assets.py (--check) and a test. Shared types: QuestionKind, DecisionQuestion, DecisionAnswer, DecisionBatch, DecisionRecord, SemanticType, plan and binding types, PatternCapabilities, LayoutBrief.parse, new DecisionTask members, SynthesisStage.ARCHETYPE, and validated SynthesisConfig fields. DecisionBackend gains answer(batch); the _AnswerViaPredict mixin derives it from predict() for all four backends. bindings.py takes the wire helpers out of the synthesizer. PointerSummary gains defaulted semantic fields plus key and parent. archetypes.py loads, validates, and lints the grammar. Two deviations from the draft, both found by validation: the form block's range control required min and max fields that did not exist, so they were added; and the collection knob note moved to the pattern note so knobs stay a name-to-values map.
Resolves the round 3 API review findings so the layout grammar surface has one owner per concept and no hidden dependency on the default grammar. - Thresholds: the grammar's `thresholds` are the single source of truth. `SynthesisConfig` drops `action_gate_floor`, `action_gate_cap`, `expected_block_threshold`, `optional_block_threshold`, and `reask_below` in favor of `threshold_overrides`, and `LayoutPlanner.threshold(name)` resolves every threshold through one path. `ArchetypeGrammar.from_json` lints `actionGateFloor <= actionGateCap` and `expected <= optional`. - Custom grammars: `DecisionBlock.layout` carries raw `role:key,key` items and the synthesizer validates them against its own grammar; `DecisionPromptGenerator(archetypes=...)` and `decision_rules(grammar)` render the prompt roles per grammar instead of at import time, and `DECISION_RULES` is resolved lazily. - Backends: `PredictOnlyBackend` replaces the private `_AnswerViaPredict` mixin, `ensure_batch_backend` adapts predict-only objects, and `aclose_current_loop` is part of `DecisionBackend` so the synthesizer and `BatchingDecisionBackend` no longer duck-type it. - Eligibility: `eligible_archetypes` is pure with a single `pointers` parameter; the bounded per-synthesizer cache replaces the module-global dict keyed on `id(grammar)`. - Patterns: `MergedPatternLibrary.for_catalogs` replaces the classmethod on `PatternLibrary` that returned the other class, the `has()` alias is gone, `PatternRef` and `ResolvedPattern` replace tuple returns and string splitting, and `name:variant` is the only pattern-plus-variant spelling (`requires` entries now read `pattern:collection:grid`). The unimplemented `repeat-in-parent` and `first-item` fallback keywords are removed from the grammar and rejected by lint. - Types: `PointerSummary.semantic_type`, `element_semantic_types`, and `SlotBinding.semantic_type` are `SemanticType`; the deferred Place and composite names join the enum; `PlanReason`, `Requirement`, `LintCode`, and `LintSeverity` are string enums. `_WireEnum` hashes by its wire value so members interoperate with string sets and dict keys. - Signatures: `PatternExpander(protocol_version=)` replaces `is_v10`, `expand_page(page: PageBindings, ...)` replaces the loose slot parameters, `expand_block` takes only the nested `PlannedBlock.instances` shape, `build_batch` makes `eligible`, `brief`, and `payload` keyword-only, `BatchingDecisionBackend(max_pairs=None)` takes the budget from `SynthesisConfig.max_pairs` via the factory, and `BRIEF_ITEM_PATTERN` is shared by the parser and the tool schema. - Facade: exports `PlannerState`, `PredictOnlyBackend`, `ensure_batch_backend`, `PatternRef`, `ResolvedPattern`, `PlanReason`, `Requirement`, `LintCode`, `LintSeverity`, `classify_array_field`, `humanize_key`, `decision_rules`, `resolve_fallback_pattern`, `shape_signature`, `BRIEF_ITEM_PATTERN`, and `MAX_BRIEF_SECTIONS`.
A chat-style ADK/A2A sample (`samples/agent/adk/format_playground`, port 10010) for trying the three inference formats on the same request. Stage 1 is one model call that turns "show my stock portfolio" into a decision block (`intent`, `payload`, optional `layout`) using `decision_rules(grammar)`. Stage 2 renders it through the chosen mode: `decision` compiles the block with `DecisionSynthesizer` and makes no further model call; `express` and `direct` make a second call with that format's own prompt and the same payload, validated through the format's processor with one retry. Slash commands `/mode`, `/compare` (all three modes from one payload), `/trace` (the planner's questions, answers, and thresholds), and `/help`. Every turn ends with a footer naming the mode, decision backend, model calls, latency, tokens, and (for decision) the archetype and confidence. Because the Lit sample shell renders surfaces only and clears them per response, footers, help, trace, and errors are emitted as small A2UI status surfaces compiled from Express, so the agent stays client-agnostic. The executor accepts v0.9 `action` and `userAction` DataParts (fed to stage 1 as text so forms round-trip) and treats any other DataPart as a pasted payload, which is how the shell sends a bare JSON object. The decision backend comes from `A2UI_DECISION_BACKEND` via `create_decision_backend()` and is named in `/help` and every footer. `LlmCaller` abstracts the ADK runner so the 44 tests run with a scripted model and `DeterministicMockBackend` only. Registers the sample as a uv workspace member and pytest testpath, and adds a `playground` config to the Lit shell (`?app=playground`).
# Conflicts: # catalogs/README.md # python/a2ui_agent/CHANGELOG.md # python/a2ui_agent/README.md
…riety sampling with remix - Add thread tuning (defaulting to CPU core count), sequence length truncation, summary pointer limits, length-based pair sorting, and dynamic batch padding to LocalCrossEncoderBackend. - Support A2UI_DECISION_THREADS and A2UI_DECISION_MAX_SEQ environment configuration in create_decision_backend. - Implement VarietyMode.NEAR_TOP sampling across anatomy and instance choices with configurable variety_margin and seed control. - Support authored copy (title and summary) in layout planning and slot binding. - Add zero-decision-call remix and remix_sync methods to DecisionSynthesizer for instant layout re-sampling.
Introduce ADK tool mode, zero-model-latency tool rendering, /rate and /ratings commands, and ratings JSONL storage to the format playground.
…d measurements - Document tool-mode first across inference formats and playground READMEs. - Add /remix [seed] command to re-sample alternative layouts at zero model cost. - Wire decision strategy into eval runner and assemble format-agnostic prompts. - Record empirical benchmark measurements for local ONNX, Jev, and Inspect AI.
…e local batching Ensure every domain pointer emitted into updateDataModel is visibly bound in updateComponents and cut local/step-zero question batching overhead: - Add apply_no_orphaned_pointers in LayoutPlanner (after apply_urgency_terse) so top-level scalar pointers not bound by page/headline/prose/inputs/toggles are bound into facts even when gate:facts votes below threshold. - Widen collection and steps element slot preferences, add avoidKeys for id and currency, run three-pass element binding (required -> preferred optional -> fallback optional with redundant duplicate suppression), unroll primitive arrays into indexed sub-pointers, and walk shape-eligible family/archetype fallbacks when the top choice lacks required pointers. - Add build_stage1_batch and build_stage2_batch to LayoutPlanner and wire two-stage hierarchical batching into DecisionSynthesizer (enabled by default for LocalCrossEncoderBackend and StepZeroLogprobBackend). - Extend StepZeroLogprobBackend with /v1/chat/completions and auth support, and add --backend step0, --two-stage, and --single-stage to decision_bench.py.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
a2ui.inference_formats.decision, a catalog-agnostic local decision-model inference format and UI synthesizer fora2ui-agent-sdk. Agent servers can route component selection and data-binding decisions through an in-process ONNX cross-encoder or HTTP endpoint. Low-confidence queries fall back to Express DSL, a callback, or a summary card.Reviewer Focus
synthesizer.pyto check the three-stage routing pipeline (TEMPLATE_SELECT,FAST_1PASS_LEAF,CONTAINER_2PASS, andABSTAIN_FALLBACK). Check theallow_actionssafety gates and the fallback order (fallback_dsl,fallback_fn, andfallback_card).format.py,parser.py,prompt_generator.py, andtool.pyto check howDecisionFormatFactoryintegrates withA2uiRequestProcessorandA2uiGeneratorand howDecisionRenderToolexposes synthesis to tool-calling agents.backends.pyto checkcreate_decision_backend, off-thread ONNX inference inLocalCrossEncoderBackend,/v1/systemonewire support and ADC token caching inJevHttpBackend, and per-loophttpx.AsyncClientcleanup in_HttpClientOwner.catalog_graph.pyandcalibration.pyto check runtime schema inspection and the threshold sweep incalibrate_thresholds.python/a2ui_agent/tests/decision/contain self-contained domain catalog fixtures and can be skimmed after reviewing the core modules.Changes
DecisionFormat,DecisionFormatFactory,DecisionParser,DecisionPromptGenerator, andDecisionBlock(format.py,parser.py,prompt_generator.py,decision_block.py).A2uiRequestProcessor(catalogs, format_factory=DecisionFormatFactory())andA2uiGeneratorcompile and decompile<a2ui>decision blocks while sharing a single backend across format instances.DecisionRenderToolintool.pyto generate a JSON Schema tool declaration withrun_asyncandrun_syncexecution methods for MCP, ADK, and function-calling agents.CatalogDecisionGraphincatalog_graph.pyandinspect_payload_pointersinpayload_inspector.pyto classify component slots (SlotKind), checkable functions, and RFC 6901 JSON Pointers directly fromCatalogApischemas.SurfaceTemplateandTemplateRegistryintemplate_registry.pyfor Stage 1 template matching. Addedpromote_synthesizedto promote v0.8, v0.9, and v1.0 message envelopes.DecisionBackend,LocalCrossEncoderBackend,JevHttpBackend,StepZeroLogprobBackend,DeterministicMockBackend, andcreate_decision_backendinbackends.py, along with optional[decision-local]dependencies (onnxruntime,tokenizers,huggingface_hub,numpy) inpyproject.toml.DecisionSynthesizerinsynthesizer.pywithfrom_env(),synthesize(), andsynthesize_sync(). It supports directtemplate_idrendering, string and list payload coercion (coerce_decision_payload), and checked fallbacks (FallbackSource.DSL,FallbackSource.CALLBACK, andFallbackSource.CARD).calibrate_thresholdsandcompute_bayes_optimal_thresholdincalibration.pyfor threshold selection over labeled datasets.a2ui.inference_formats.decisionanda2ui.inference_formats, and documented the format andA2UI_DECISION_*environment variables inpython/a2ui_agent/README.md,src/a2ui/inference_formats/README.md, andpython/a2ui_agent/CHANGELOG.md.Impact & Risks
a2ui.inference_formats.decisionand optional[decision-local]dependencies, leaving existing Direct JSON and Express parsers unchanged.calibrated=Falseand requiretop_logit_margin >= uncalibrated_min_logit_marginbefore a local prediction is accepted.mutable_pathsorActionslots requireconfidence >= mutable_confidence_threshold(0.85by default) before action-bearing components or slots can be synthesized.upstream/main(c4e0e0697); if this PR is squash-merged, record ancestry withgit merge -s ours c4e0e0697before the next branch sync.Testing
a2ui_agenttest suite (including framework-agnostic isolation tests):a2ui_agent: