From 199b53a0941a6b54d98af313b42c69971986ad29 Mon Sep 17 00:00:00 2001 From: Joel Goldfoot Date: Tue, 28 Jul 2026 08:10:54 -0700 Subject: [PATCH 1/3] =?UTF-8?q?docs:=20weekly=20research=20scan=20[2026-07?= =?UTF-8?q?-28]=20=E2=80=94=20WeaveBench,=20MAG,=20WCAG-EM=202.0?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds three findings verified against primary sources: - WeaveBench (arXiv:2606.09426): first long-horizon hybrid-interface CUA benchmark; 114 tasks / 8 domains, best PassRate 41.2%; trajectory-aware judge shows outcome-only grading overestimates agent performance. - MAG (arXiv:2607.10079): screenshot-grounded web-agent benchmark; strongest model completes fewer than 40% of tasks. - WCAG-EM 2.0 (W3C Group Note, 23 Jul 2026): conformance evaluation methodology extended from websites to apps and other digital products. Also adds a caveat to whitepaper 5.2 noting the Hybrid Agents row is the least empirically anchored projection in the table, and that WeaveBench's judge result implies published benchmark scores are likely optimistic. Benchmark numbers remain directional projections, never presented as a BiModal A/B test. No changes to package.json, package-lock.json, or CI. --- README.md | 26 ++++++++++++++++++++++---- docs/whitepaper.md | 46 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 68 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 46fc621..dcb9513 100644 --- a/README.md +++ b/README.md @@ -4,9 +4,9 @@ > agent capability spectrum — from HTTP retrievers to protocol-native agents.** [![License: Apache 2.0](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](LICENSE) -[![Status: Research Framework](https://img.shields.io/badge/Status-Research%20Framework-yellow.svg)]() -[![Node.js](https://img.shields.io/badge/node-%3E%3D18.0.0-brightgreen.svg)]() -[![Version: 3.0](https://img.shields.io/badge/version-3.0-blue.svg)]() +[![Status: Research Framework](https://img.shields.io/badge/Status-Research%20Framework-yellow.svg)](<>) +[![Node.js](https://img.shields.io/badge/node-%3E%3D18.0.0-brightgreen.svg)](<>) +[![Version: 3.0](https://img.shields.io/badge/version-3.0-blue.svg)](<>) [![Contributions welcome](https://img.shields.io/badge/contributions-welcome-brightgreen.svg)](CONTRIBUTING.md) --- @@ -135,7 +135,17 @@ selector drift** (where brittle CSS classes break automations). A robust Layer 2 **~75-88%** with structured data. - **BrowseComp, VisualWebArena & MCP-Universe insights** indicate that pure visual reasoning is brittle; well-structured Layer 2 and Layer 3 correlate - with higher agent reliability and support reliable tool-use handoffs. + with higher agent reliability and support reliable tool-use handoffs. **MAG** + (2026) adds a fresh data point: evaluated on rendered screenshots rather than + DOM or accessibility trees, the strongest model completes fewer than 40% of + tasks. +- **Long-horizon orchestration remains unsolved, and reported scores may be + optimistic.** On **WeaveBench** (2026), which requires agents to combine GUI, + CLI, and code operations within a single trajectory, the best frontier + model-runtime pairing reaches a 41.2% PassRate. Its trajectory-aware judge + also found that outcome-only grading substantially overestimates agent + performance — a caution that applies to the benchmark figures the ranges above + are derived from. --- @@ -375,6 +385,14 @@ npm test long-horizon live-web tasks; strongest evaluated model 44.5%. - **VisualWebArena** — Evaluating Multimodal Agents on Realistic Visual Web Tasks +- **WeaveBench** — "A Long-Horizon, Real-World Benchmark for Computer-Use Agents + with Hybrid Interfaces" — 114 tasks across 8 work domains requiring GUI, CLI, + and code operations in one trajectory; best PassRate 41.2% + (arXiv:2606.09426, 2026) +- **MAG** — "A Web-Agent Benchmark and Harness for Multimodal Action and Guide + Generation" — screenshot-grounded evaluation via Set-of-Mark and pixel + coordinates; strongest model completes fewer than 40% of tasks + (arXiv:2607.10079, 2026) - **BrowseComp** — Benchmark for Agentic Browser Navigation & Task Execution - **OpAgent** — "Operator Agent for Web Navigation" achieving 71.6% on WebArena (arXiv:2602.13559) diff --git a/docs/whitepaper.md b/docs/whitepaper.md index 29b1d2e..202d385 100644 --- a/docs/whitepaper.md +++ b/docs/whitepaper.md @@ -526,6 +526,22 @@ it's measurable: - **WorkArena++ Benchmark**: Highlighted that enterprise knowledge work requires complex compositional reasoning across dense UIs, exposing the inadequacy of simple visual navigation for multi-step workflows. +- **WeaveBench (2026)**: The first benchmark to test long-horizon orchestration + _across_ interfaces rather than treating each interface as a separable + capability. Its 114 tasks across 8 real-world work domains each require an + agent to combine GUI observations and actions with CLI and code operations in + a single trajectory. Across frontier model-runtime pairings the best PassRate + reaches only 41.2%. Its companion trajectory-aware judge — which inspects + deliverables, files, screenshots, logs, and action traces — found that + outcome-only grading substantially overestimates agent performance + (arXiv:2606.09426). +- **MAG Benchmark (2026)**: Evaluates agents on rendered screenshots using + Set-of-Mark element selection and raw pixel coordinates, rather than the DOM + or accessibility-tree representations most prior benchmarks supply. Even the + strongest model completes fewer than 40% of tasks, and a GRPO method augmented + with expert trajectories nearly doubles a supervised 9B agent's success rate + from 6.9% to 13.2% — reinforcing that screenshot-only grounding remains a weak + substitute for structural signal (arXiv:2607.10079). ### **5.2 Performance Gaps by Agent Type** @@ -557,6 +573,21 @@ succeed**, and the **highest projected reliability comes from Hybrid Agents using API and protocol layers for execution** — reinforcing the defense-in-depth approach where UI structure allows discovery of robust API tools. +> **The Hybrid Agents row is the least anchored projection in this table.** The +> closest published measurement is WeaveBench (arXiv:2606.09426), where the best +> frontier model-runtime pairing reaches a 41.2% PassRate. That number is not a +> refutation of the 94-99% projection, because the two use "hybrid" differently: +> WeaveBench measures agents orchestrating GUI, CLI, and code operations across +> long-horizon tasks on a desktop, whereas this table projects agents that +> _discover_ a task through the UI and _execute_ it through a documented API or +> MCP tool. But no benchmark yet isolates that second pattern, so the row rests +> on inference rather than evidence and should be read as the framework's +> strongest claim in need of the weakest support. WeaveBench's second result is +> a caution that applies to every figure cited in this section: its +> trajectory-aware judge found that outcome-only grading substantially +> overestimates agent performance, which means the published success rates the +> columns above are mapped from are likely optimistic. + ### **5.3 Why BiModal Design Matters Now** #### **1. Agent Diversity is Increasing** @@ -1510,6 +1541,13 @@ explicit content policies: Design's semantic structure requirements - **WCAG 2.2**: BiModal Design's Layer 2 directly builds on and extends WCAG compliance +- **WCAG-EM 2.0**: Published as a W3C Group Note on 23 July 2026 by the + Accessibility Guidelines Working Group. Where WCAG-EM 1 covered websites and + web pages, WCAG-EM 2 extends its step-by-step conformance evaluation + methodology to apps and other digital products. Teams already running a + WCAG-EM audit can treat it as the procedural companion to a BiModal Design + Layer 2 assessment, though it evaluates human accessibility conformance and + does not assess Layers 1 or 3-5 ### **11.6 Human-in-the-Loop (HITL) for Tool-Agent-User Interaction (τ-bench)** @@ -2163,6 +2201,12 @@ resilient, semantic, structured, and protocol-aware. March 2025 17. **Agent Containment (Claude Cowork)**: Anthropic, "How we contain Claude" — https://www.anthropic.com/engineering/how-we-contain-claude (2026) +18. **WeaveBench**: "A Long-Horizon, Real-World Benchmark for Computer-Use + Agents with Hybrid Interfaces" — Li, Zhou, Yu, Xu, Yang, Li, Shan; + arXiv:2606.09426 (v1 8 Jun 2026, v3 6 Jul 2026) +19. **MAG**: "A Web-Agent Benchmark and Harness for Multimodal Action and Guide + Generation" — Gan, Wei, Liang, Cai, Zhang, Ni; arXiv:2607.10079 (11 + Jul 2026) ### **Agent Protocols** @@ -2199,6 +2243,8 @@ resilient, semantic, structured, and protocol-aware. 15. **WAI-ARIA 1.2**: https://www.w3.org/TR/wai-aria-1.2/ 16. **OpenAPI 3.0**: https://spec.openapis.org/oas/v3.0.0 17. **WCAG 2.2**: https://www.w3.org/TR/WCAG22/ +18. **WCAG-EM 2.0**: https://www.w3.org/TR/wcag-em-2/ — W3C Group Note, 23 July + 2026 ### **BiModal Design Resources** From d176d8c93c67625e3f5b0f958d71c46ecc679938 Mon Sep 17 00:00:00 2001 From: Joel Goldfoot Date: Tue, 28 Jul 2026 08:26:29 -0700 Subject: [PATCH 2/3] chore: reformat two files for prettier 3.9.6 Dependabot #144 bumped prettier 3.8.x -> 3.9.6 on 2026-07-27 without reformatting the repo. Prettier 3.9 changed how it wraps TypeScript union types inside embedded code blocks, leaving `docs/implementation-guide.md` and `examples/mcp-async-tasks.md` failing `npm run format:check` on main. This is pre-existing drift unrelated to the research scan, kept as its own commit so it can be dropped or split off. Content unchanged; wrapping only. --- docs/implementation-guide.md | 3 +-- examples/mcp-async-tasks.md | 3 +-- 2 files changed, 2 insertions(+), 4 deletions(-) diff --git a/docs/implementation-guide.md b/docs/implementation-guide.md index c3a2024..bcdf2da 100644 --- a/docs/implementation-guide.md +++ b/docs/implementation-guide.md @@ -633,8 +633,7 @@ server.setRequestHandler( async (request, ctx): Promise => { const { name, arguments: args } = request.params; const taskParams = (request.params._meta?.task ?? request.params.task) as - | { ttl?: number; pollInterval?: number } - | undefined; + { ttl?: number; pollInterval?: number } | undefined; if (!taskParams) throw new Error(`Tool ${name} requires task mode`); const options: CreateTaskOptions = { diff --git a/examples/mcp-async-tasks.md b/examples/mcp-async-tasks.md index 7cc3d9a..a11a4a2 100644 --- a/examples/mcp-async-tasks.md +++ b/examples/mcp-async-tasks.md @@ -116,8 +116,7 @@ function createServer(): Server { async (request, ctx): Promise => { const { name, arguments: args } = request.params; const taskParams = (request.params._meta?.task ?? request.params.task) as - | { ttl?: number; pollInterval?: number } - | undefined; + { ttl?: number; pollInterval?: number } | undefined; if (name !== 'generate_comprehensive_audit') { throw new Error(`Unknown tool: ${name}`); From eb21d164a6c0b73107345cac4b38c9899a36bbdd Mon Sep 17 00:00:00 2001 From: Joel Goldfoot Date: Wed, 29 Jul 2026 07:38:28 -0700 Subject: [PATCH 3/3] docs: soften Hybrid Agents caveat in 5.2 Drop the 'strongest claim in need of the weakest support' framing. The preceding sentence already states the row rests on inference rather than evidence, and the opening sentence already calls it the least anchored projection in the table, so the phrase restated the same point a third time as an aphorism. It also overstated: the Hybrid row is the highest number in one table, not the framework's strongest claim. Epistemic content unchanged - the row is still explicitly flagged as unanchored, and the WeaveBench overestimation caution is untouched. --- docs/whitepaper.md | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/docs/whitepaper.md b/docs/whitepaper.md index 202d385..133274d 100644 --- a/docs/whitepaper.md +++ b/docs/whitepaper.md @@ -581,12 +581,11 @@ approach where UI structure allows discovery of robust API tools. > long-horizon tasks on a desktop, whereas this table projects agents that > _discover_ a task through the UI and _execute_ it through a documented API or > MCP tool. But no benchmark yet isolates that second pattern, so the row rests -> on inference rather than evidence and should be read as the framework's -> strongest claim in need of the weakest support. WeaveBench's second result is -> a caution that applies to every figure cited in this section: its -> trajectory-aware judge found that outcome-only grading substantially -> overestimates agent performance, which means the published success rates the -> columns above are mapped from are likely optimistic. +> on inference rather than evidence. WeaveBench's second result is a caution +> that applies to every figure cited in this section: its trajectory-aware judge +> found that outcome-only grading substantially overestimates agent performance, +> which means the published success rates the columns above are mapped from are +> likely optimistic. ### **5.3 Why BiModal Design Matters Now**