Awesome Agent Experience (AX)

A curated public library for agent experience: the design of product, documentation, API, tool, protocol, evaluation, and recovery surfaces so AI agents can discover, understand, safely use, and repair workflows on behalf of people.

This list is intentionally narrow. It is about software agent experience: web agents, tool-using LLMs, MCP/API surfaces, evaluation harnesses, retrieval, human control, and operational recovery. It is not about call-center agent experience, real-estate agents, talent agents, or generic customer-experience copy.

The field framing includes the independent opinion essay Introducing AX: Why Agent Experience Matters. This repository is maintained by the same author as the Open Agent-Readiness Rubric (v0.3 draft, with a JSON version), and the same site publishes an AX evidence register of public findings (with a JSON download) and the ax-check rules behind that rubric; those links are provided as same-maintainer context, not as neutral endorsement.

Scope criteria

A source belongs here when it helps answer at least one of these questions:

• Can an agent discover the right surface, instruction, tool, or document?
• Can an agent select and call tools with clear schemas, examples, and error boundaries?
• Can humans inspect, approve, interrupt, or recover the agent's work?
• Can teams evaluate agent workflows in a reproducible way?
• Can the same surface stay useful as models, tools, and protocols change?

A source does not belong here if it is only a launch announcement, generic AI-agent marketing, SEO filler, private experiment, paid placement, or unlicensed copy of someone else's work.

How to read this list

Sections follow a reader's path: what AX is, how agents find and read a surface, how they call tools, which protocols carry identity, payments and approval, how to secure and evaluate the result, and where neighbouring catalogues take over. Each entry shows its source type and publication year (omitted when the source states none), a reason it matters, and a limitation. The limitation is part of the entry: read it before citing the source as evidence.

Vendor guides and scanners are included when they carry concrete, checkable guidance, and are labelled as vendor material in their limitations. Readiness scanners are listed as instruments, not as proof: none of those listed here has published evidence that a higher score predicts agent task success. GEO and AI search research is kept in its own adjacent section because it studies how answer engines cite content, which overlaps with, but is not the same as, whether an agent can complete a task.

Start here

• If you are defining the field: read Introducing AX and One Year of AX, then Principles of AX and the six DX-to-AX areas from Resend, and compare them with the scope criteria above.
• If you publish a website or documentation: read llms.txt, Markdown for Agents, Content Signals, WebMCP and ARD, then Designing Agent-Ready Websites for a controlled before-and-after study, and the Documentation for agents section (AGENTS.md, Agent Skills, Agent Plugins).
• If you build tools, APIs or MCP servers: read Anthropic's tool-writing guide, the MCP tools and client best-practice pages, OpenAPI, JSON Schema and Arazzo, then the tool-description and tool-selection research (smell-aware descriptions, BiasBusters, Schema First, Skill Shadowing).
• If agents act for your users: read the identity section (RFC 9421, Web Bot Auth, the IETF agent authentication draft, Auth.md), the commerce protocols if money moves, and Consent Integrity and Loopjacking before designing approval screens.
• If you own production risk: start with MCP security best practices, Exposed by Design and A First Look at MCP security, then The Framing Gap for why surface-level prompt-injection defences are not enough.
• If you run evaluations or readiness checks: read Benchmarking the Benchmarks, the Agentic Benchmark Checklist and Protocol Validity before trusting any score, then choose benchmarks and readiness instruments with their limitations in mind.
• If you need breadth rather than curation: use the Related lists section for MCP servers, Agent Skills, browser agents and computer-use tooling.

Contents

• Agent experience framing (9)
• Discovery and agent-readable web (19)
• Documentation for agents (9)
• Tool use, tool descriptions and MCP (35)
• Protocols and machine-readable surfaces (7)
• Agent identity and authorization (11)
• Agentic commerce and payments (5)
• Human control, approval and recovery (12)
• Security for agent-facing surfaces (14)
• Evaluation (25)
• Readiness instruments (7)
• GEO and AI search (adjacent) (7)
• Related lists (7)

Agent experience framing

• Introducing AX: Why Agent Experience Matters (essay, 2025): A public opinion frame for AX as a discipline, useful because it states that agent-facing surfaces deserve product-design attention rather than being treated as incidental automation glue. Limit: This is a framing essay, not a benchmark or standard; cite it as an opinionated field definition, not as proof of outcomes or ownership.
• One Year of AX (essay, 2026): A one-year follow-up to the founding AX essay that regroups the idea into four pillars (access, context, tools and orchestration), giving a compact vocabulary for auditing a product surface. Limit: Written by the same author as the original essay, so it is an update to one framing rather than independent validation that the four pillars are complete or correct.
• AX in Practice: Real-World Examples from Netlify and the Industry (essay, 2025): Turns the AX framing into named production patterns, including agent-first onboarding that lets an agent create a resource before a human claims it, and bidirectional links between agents and human work queues. Limit: Adopter examples are reported by a platform founder about his own and partner products; the article does not measure whether the patterns change outcomes.
• How Vercel ships for agents (essay, 2026): A case study that groups one platform's agent-facing changes into six patterns, including version-matched docs referenced from AGENTS.md, Markdown 404 pages that suggest nearby URLs, non-interactive CLI defaults when an agent is detected, and deploy-before-signup claim flows. Limit: Written by a company that sells agent testing, in collaboration with the platform it describes; its eval figures are the platform's own, and it does not measure whether the patterns transfer to other products.
• What is AX (Agent Experience) and how to improve it (essay, 2025): Maps familiar developer-experience concerns onto agents across six practical areas (onboarding, documentation, SDKs, audit logs, API keys and role-based access), useful as a checklist for platform teams. Limit: A vendor blog post drawn from one email platform's perspective; the six areas are the author's own grouping rather than the result of a study.
• Principles of AX (essay): States a short set of community AX principles, including treating agents as delegates of people and recording agent actions separately from human ones, which helps teams agree on design intent before choosing tools. Limit: The site is hosted and largely written by Netlify staff despite its community framing, and it says itself that the principles may change as practice matures.
• AI Agents as Users (essay, 2026): A UX research firm's argument that agents are users in a functional sense, contrasting screenshot, accessibility-tree and API interaction, and setting out when a product may reasonably choose not to serve agents. Limit: An opinion article with no new study behind it; its near-term advice is largely existing accessibility practice, and its remarks on token costs are general rather than measured.
• From Human Interfaces to Agent Interfaces: Rethinking Software Design in the Age of AI-Native Systems (paper, 2026): A position paper that names agent interfaces and invocable capabilities as a design discipline distinct from human-facing UX and conventional API design, giving the field an academic framing to cite. Limit: Conceptual and argumentative only; none of the proposed principles are measured against real systems in the paper.
• What is Agent Experience (AX)? (essay, 2026): A platform vendor's definition of AX that splits measurement into propensity (does the agent choose your technology) and efficacy (does it use it correctly), adds task cost, and names working terms such as bare baseline, lift and drag, and agent profile. Limit: Vendor framing illustrated almost entirely with coding agents on Microsoft's own developer products and small run counts, typically five per condition, so its figures hold only for the agent profiles it measured.

Discovery and agent-readable web

• llms.txt: The /llms.txt file, v2 (specification, 2024): Documents a proposed Markdown-based website discovery convention for helping language models and agents find the most relevant human-authored pages. Limit: It is a proposal rather than an enforced web standard, so teams should treat it as one discoverability surface among several.
• Agentic Resource Discovery (ARD) specification, v0.91 (specification, 2026): The dated, multi-vendor v0.91 proposal (authored across Google, Microsoft, and Hugging Face) distinguishes a plain catalogue entry from a stricter ARD entry built for search-based discovery beyond an LLM's context window. Limit: Status is explicitly Proposal, not final; it delegates authentication and distribution to the underlying agent or tool protocol rather than defining them itself.
• WebMCP (Draft Community Group Report) (specification, 2026): Defines the document.modelContext API for exposing client-side tools to agents and gives a detailed public security taxonomy for agent-callable web pages, covering tool poisoning, output injection, and over-parameterised data leakage. Limit: Published by a W3C Community Group; the spec itself states it is not a W3C Standard nor on the Standards Track, so treat it as an active proposal, not settled browser behaviour.
• Markdown for Agents (specification, 2026): Documents an HTTP content-negotiation pattern, an Accept: text/markdown request, for serving agents an edge-converted Markdown page, with a precise, checkable policy on which headers are preserved, recalculated, or dropped. Limit: The feature is limited to paid Cloudflare plans and to HTML-to-Markdown conversion of responses under 2 MB; it does not establish whether AI crawlers actually send the Accept header in practice.
• Designing Agent-Ready Websites for AI Web Agents (paper, 2026): Runs a controlled before/after e-commerce experiment across three browser-agent models and 300 task runs, finding an otherwise-identical agent-ready site raised strict task success from about 49 percent to about 89 percent. Limit: The experiment uses one e-commerce site design and three specific browser-agent models; results may not transfer directly to other site types or to agent models released after the study.
• Accessible Rich Internet Applications (WAI-ARIA) 1.2 (specification, 2023): Defines roles, states, and properties for accessible web interfaces; AX uses this as relevant machine-readable UI semantics, not as a shortcut to agent success. Limit: Correct ARIA use is necessary for accessibility in many interfaces, but it does not guarantee that an agent can complete a workflow.
• Web Content Accessibility Guidelines (WCAG) 2.2 (specification, 2023): Sets human accessibility requirements for perceivable, operable, understandable, and robust web content; AX relevance is an inference because agent tooling often benefits from the same explicit structure. Limit: WCAG conformance is not equivalent to agent readiness and should not be cited as evidence that agents will operate a product reliably.
• llms-txt (reference repository) (repository, 2024): The proposal authors' own repository, with the source of the specification site and Python tooling to parse and expand llms.txt files, which is the practical companion to the llms.txt entry above. Limit: Reflects the proposal authors' implementation; it does not test whether any given agent or answer engine actually requests or uses the file.
• Content Signals Policy (specification, 2025): Extends robots.txt with separate machine-readable preferences for search, AI input and AI training, so a site can distinguish being indexed from being fetched by a live agent from being used as training data. Limit: It expresses a preference rather than enforcing one, and it depends on crawler operators choosing to honour the signal.
• NLWeb (repository, 2025): Reference implementation for giving a website a natural-language query endpoint built on its existing Schema.org data, with every instance also exposed as an MCP server, a concrete pattern for making content queryable by agents. Limit: Useful results depend on structured data that is already present and accurate, and this pass found no independent evidence on deployment or outcomes.
• Build agent-friendly websites (documentation, 2026): Browser-vendor guidance that treats agents as a new kind of site visitor and translates that into concrete markup and interaction advice, including semantic structure, predictable forms and stable state changes. Limit: Framed around Chrome's own agentic browsing features, so some recommendations reflect one browser vendor's direction rather than a cross-browser standard.
• Make your site readable by AI agents (documentation, 2026): Combines an observational trace study of 1,033 agent runs across 25 sites with controlled fetch probes, then turns the results into specific advice on server rendering, Markdown responses and llms.txt. Limit: Vendor research run with a partner lab on a small set of sites; the study design and recommendations are shaped by one hosting platform and have not been independently replicated.
• Building the Web for Agents: A Declarative Framework for Agent-Web Interaction (paper, 2025): Proposes HTML elements that let a page declare agent-callable actions and relevant state directly in markup, so agents do not have to infer affordances from visual layout. Limit: A short feasibility study with a small developer group, not evidence of production adoption, robustness or behaviour under adversarial use.
• Web Agents Should Use Typed Actions Instead of Click-Based Browsing (paper, 2026): Argues for typed web actions with documented preconditions, postconditions and policy tags as a more reliable and auditable interface for agents than click-based page automation. Limit: A position paper without an implementation or benchmark; its value depends on site owners and standards bodies building the layer it describes.
• Affora: A Design System for Agent-Friendly Interfaces (paper, 2026): An evaluated design system bridging human UX and agent use, whose controlled studies indicate that agent performance follows preserved interaction meaning rather than visual style. Limit: Its benefit appears mainly where an interface had a matching deficit, and the authors describe the workflow case-study evidence as preliminary.
• terms.txt: A Consent and Compensation Protocol for Agentic Web Access (paper, 2026): Proposes a successor to robots.txt that combines per-path access terms with signed agent identity, delegation tokens and HTTP 402 payment negotiation, framing agent access as a consent and compensation question. Limit: A proposal with a reference implementation, not an adopted standard; it only works if both site operators and agent vendors implement it.
• AI Catalog (specification): A draft JSON format that lists MCP servers, A2A agents, skills, plugins and other AI artefacts by media type, with an optional signed trust manifest; the ARD specification above treats it as a plain catalogue, distinct from its own search-oriented entry. Limit: The page calls itself a draft that may change at any time and carries no date; its well-known path also differs from ARD v0.91, which asks publishers to move from ai-catalog.json to ard.json.
• AGENTS.TXT: Capability Declarations for Web Agents (specification, 2026): Proposes /.well-known/agents.txt and agents.json as a positive counterpart to robots.txt, in which a site declares the endpoints, protocols, authentication and rate limits agents may use, and explains how it relates to llms.txt and to MCP and A2A cards. Limit: An individual Internet-Draft in the independent stream with no IETF endorsement or formal standing, expiring in December 2026 unless revised; an earlier draft under the same name lapsed without a revision.
• Does anyone actually have an llms.txt? I checked millions of websites (essay, 2026): Measures llms.txt adoption from the public HTTP Archive crawl: about 5.6 per cent of the top 10,000 sites had a valid file in June 2026, with little change by rank, and some platforms, notably Shopify, now publish one by default. Limit: A single-author blog post that does not publish its queries; 'valid' means the file parses, not that it is useful, and presence says nothing about whether agents read the file.

Documentation for agents

• AGENTS.md (specification, 2026): A deliberately unstructured Markdown convention for repository-level agent instructions, now governed by the Agentic AI Foundation under the Linux Foundation rather than a single vendor, with adoption checkable via a linked GitHub code search. Limit: Has no required fields or schema, so file quality varies widely between projects; the specification itself does not address how a poorly written file affects agent task success.
• Agent Skills specification (specification): Defines the SKILL.md package format (a directory with YAML frontmatter, instructions and optional scripts and references) that many coding agents load on demand, now a common way for products to ship usage knowledge to agents. Limit: The format does not verify the quality or safety of a skill; installing a third-party skill can mean running its instructions and code.
• Agent Plugins Specification 1.0.0 (specification, 2026): A published, versioned specification with conformance requirements for packaging skills, MCP servers and client extensions into one portable plugin, useful for teams that ship agent-facing capabilities to several clients. Limit: Very new at retrieval time; there is no independent evidence yet on how consistently clients implement the conformance checklist.
• Agent Skills Discovery RFC (specification, 2026): Proposes a well-known URI index so an agent can discover the skills a site publishes, extending the RFC 8615 discovery pattern to agent skills. Limit: An RFC-stage proposal from one company; the index shape may change and it has no standards-body status.
• Open Knowledge Format (OKF) specification, v0.2 (specification, 2026): Specifies a directory of Markdown files with YAML frontmatter as a portable bundle of curated knowledge for agents, with index and log files for progressive disclosure and fields for provenance, trust tier and freshness. Limit: An early draft on a mutable main branch, published by Google Cloud with reference tooling centred on BigQuery; it targets internal data knowledge more than public product surfaces, and it reports no evidence on agent task outcomes.
• How to write a great agents.md: Lessons from over 2,500 repositories (essay, 2025): Draws authoring patterns from a large sample of real agent instruction files, such as naming exact commands, stating boundaries and giving concrete examples, which is practical advice the AGENTS.md convention itself leaves open. Limit: Written about GitHub Copilot custom agent files, which overlap with but are not identical to the cross-vendor AGENTS.md convention, and the analysis method is not published in detail.
• AGENTS.md outperforms skills in our agent evals (essay, 2026): Reports an internal evaluation in which a compressed documentation index placed in AGENTS.md outperformed on-demand skills for framework-specific coding tasks, a useful counterweight to skills-first advice. Limit: A single vendor, a single framework and the vendor's own evaluation set; the result is not shown to generalise to other stacks or agents.
• Effective context engineering for AI agents (essay, 2025): Explains how to treat an agent's context as a finite budget, covering just-in-time retrieval, compaction and structured notes, which informs how documentation and tool output should be shaped for agents. Limit: Written from one model provider's internal practice; specific patterns may not transfer unchanged to other models or agent harnesses.
• Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents? (paper, 2026): A controlled test on SWE-bench tasks and a new benchmark built from repositories with developer-written context files: the files did not generally raise task success and added over 20 per cent to inference cost, although agents did follow their instructions. Limit: Python repositories only, with success measured as issue resolution; the authors note that context files may matter more for less common languages and for qualities such as security, which they did not measure.

Tool use, tool descriptions and MCP

Tool design guides

• Writing effective tools for agents — with agents (essay, 2025): Lays out a concrete build-evaluate-refine loop for tool design, realistic multi-step evaluation tasks followed by using an agent to analyse transcripts, with five practical principles and a quantified example cutting one tool's response from 206 to 72 tokens. Limit: All worked examples are Anthropic's own internal Slack and Asana tools; it does not address cross-vendor tool catalogues, discovery, or MCP registries.
• Designing Agent-Friendly APIs (essay, 2026): Sets out six API design principles for agent callers, including treating errors as guidance, budgeting the reader's context and making the interface self-describing, with worked failure examples. Limit: A vendor engineering post without an accompanying controlled evaluation of the six principles.
• Steps toward great agent experience every API provider can take today (essay, 2026): Argues that specific, actionable error bodies and machine-readable API descriptions are part of an API's contract with agents, with concrete before-and-after examples an API team can apply. Limit: Written by an SDK-generation vendor and illustrated with its own examples rather than a controlled comparison.
• Agentic Experience Is the New Developer Experience (essay, 2026): A command-line perspective on AX that describes structured output, idempotent commands, managed context and informative errors as design goals for tools that agents call from a terminal. Limit: Describes one vendor's own product decisions; the principles are not tested against other command-line tools.
• AXI: Agent eXperience Interface (repository, 2026): Ten design principles for command-line tools used by agents, such as minimal default fields, definitive empty states, structured errors without interactive prompts and next-step hints after each output, with reference implementations to inspect. Limit: Its benchmark tables come from the maintainer's own runs with one model against its own wrappers, and its preferred compact output format (TOON) is a project choice rather than a shared standard.
• Tool search (documentation): OpenAI's version of letting the model search for tools instead of loading every definition: functions, namespaces or MCP servers are marked for deferred loading and found by a hosted or application-run search, with advice to group them in small, well-described namespaces. Limit: Vendor documentation for recent models on one API, with no published token or accuracy figures; for a single deferred function the name and description stay visible and only the parameter schema is deferred.
• Tool search tool (documentation): Anthropic's version of the same pattern, in which deferred tools stay out of the context entirely until a regex or BM25 search returns them; read alongside the OpenAI guide, it shows that vendors differ in what the model sees before it searches. Limit: Vendor documentation whose figures on token savings and tool-selection accuracy are the vendor's own and not linked to a published evaluation, and whose details apply only to the models it lists as supported.

Research on tool use and selection

• ReAct: Synergizing Reasoning and Acting in Language Models (paper, 2022): Connects reasoning traces with actions, making it a useful starting point for interfaces that expose why an agent chose a tool and what happened next. Limit: Primarily a prompting and evaluation paper; it does not by itself specify production observability, permissioning, or recovery contracts.
• Toolformer: Language Models Can Teach Themselves to Use Tools (paper, 2023): Frames tool use as something models can learn from demonstrations, which is central to designing tool descriptions and examples that agents can actually use. Limit: Uses a bounded set of tools and self-supervised data generation; production tool ecosystems add security, latency, and changing-schema problems.
• ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs (paper, 2023): Shows how large tool corpora and tool-use data shape agent behaviour, supporting AX work on catalogue structure and retrieval-friendly tool metadata. Limit: Large catalogues can hide quality variance; practical AX also needs clear scope boundaries, deprecation signals, and safe failure modes.
• The Devil Is in the Interface: Evaluating How Tool Architecture Shapes Coding Agent Behavior (paper, 2026): Runs 11,700 trajectories across three model actors and six tool architectures, from a bare bash tool to structured interfaces, showing that how tools are organised and exposed changes agent behaviour even when raw capability is held constant. Limit: Findings come from repository-level issue-fixing tasks with specific model actors; whether the same architecture ranking holds for other task domains is not tested here.
• DOCSCHISEL: Adaptive Tool Documentation Optimization Framework for LLM Agents (paper, 2026): Shows that no single fixed tool-documentation format generalises across tasks, models, and agent designs, then proposes a framework that analyses failed execution traces to iteratively add, remove, or refine documentation fields per tool. Limit: The proposed framework requires access to an agent's failed execution traces to iterate on, which is not available to every team publishing a tool or API.
• From Docs to Descriptions: Smell-Aware Evaluation of MCP Server Descriptions (paper, 2026): Defines 18 documentation 'smell' categories across four quality dimensions, applies them to 10,831 real MCP servers, and runs a mutation study showing that injecting or fixing smells measurably changes which tool an agent selects. Limit: The mutation study shows description smells move selection outcomes in the tested setup; it does not rank which individual smells matter most across every model.
• More Skills, Worse Agents? Skill Shadowing Degrades Performance When Expanding Skill Libraries (paper, 2026): Decomposes agent performance degradation from growing tool or skill libraries into a wrong-selection component, named skill shadowing, versus a context-overhead component, and estimates both empirically on a library scaling to 202 skills. Limit: Findings come from one scaling experiment up to 202 skills; whether the same shadowing-versus-overhead split holds at far larger catalogue sizes is not established.
• ToolChoiceConfusion: Causal Minimal Tool Filtering for Reliable LLM Agents (paper, 2026): Argues topical relevance alone is not enough to decide which tools to expose at each step, and tests a precondition-aware filtering method across 2,448 task-method-model runs, cutting token cost roughly 90 percent without hurting task success. Limit: Evaluated with 100 tools and four backend models in a controlled multi-step task set; real production catalogues and model mixes may behave differently.
• SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering (paper, 2024): An early demonstration that the interface an agent uses to read and edit files is itself a lever on task success, shown by varying interface design while holding the model fixed. Limit: Evaluated on repository-editing benchmarks with a hand-designed interface, so the size of the effect may be specific to that setting.
• AXIS: Efficient Human-Agent-Computer Interaction with API-First LLM-Based Agents (paper, 2024): Shows an agent that prefers application APIs over clicking through a graphical interface completing tasks faster with less user workload, supporting an API-first recommendation for agent-facing products. Limit: Evaluated on one desktop application against one baseline; its automatic API discovery is not shown to generalise to other applications.
• BiasBusters: Uncovering and Mitigating Tool Selection Bias in Large Language Models (paper, 2025): Finds that models choosing between functionally equivalent tools from different providers show systematic preferences that small wording changes can shift, which matters for anyone publishing into a shared tool marketplace. Limit: The sets of equivalent tools are constructed by the authors; real marketplaces rarely offer such clean alternatives.
• Tools are under-documented: Simple Document Expansion Boosts Tool Retrieval (paper, 2025): Uses a purpose-built benchmark to show that enriching sparse tool documentation with structured fields improves tool retrieval, evidence that under-documentation is a fixable cause of selection failures. Limit: The enriched documentation is itself generated by a model, so it can encode errors or drift from what a tool actually does.
• Learning to Rewrite Tool Descriptions for Reliable LLM-Agent Tool Use (paper, 2026): Trains a rewriter that improves tool descriptions without needing per-tool execution traces and generalises to unseen tools, relevant to teams maintaining large catalogues of tool metadata. Limit: Gains are reported on research benchmarks; behaviour on proprietary or highly specialised tool catalogues is not tested.
• Schema First Tool APIs for LLM Agents: A Controlled Study of Tool Misuse, Recovery, and Budgeted Performance (paper, 2026): A controlled pilot that isolates interface design and finds that JSON Schema contracts reduce malformed calls but not conceptually wrong ones, a useful check on the assumption that schemas alone fix tool misuse. Limit: A small pilot on one local model in which end-task success stayed at zero in every condition, which limits conclusions about downstream outcomes.
• Toollery: Scaling LLM Agents to Thousands of Skills and Tools (paper, 2026): A training-free method that keeps online tool selection bounded to a small candidate set as catalogues grow, evaluated on a benchmark of tens of thousands of capabilities, relevant to large MCP and skill catalogues. Limit: Benefits depend on how well generated intent queries cover the real workload and on provider-side prompt caching, both of which vary by deployment.
• The Bitter Lesson of Tool Calling (paper, 2026): Compares exposing tools as typed Python functions that the model calls from code with native JSON tool calls across 14 models on BFCL v4; calling from code matched or beat JSON for 11 of the 14 models, which bears on how tools are packaged for code-capable agents. Limit: Results come from one benchmark and the authors' own harness for running generated code, so they show the approach works there rather than that it improves any particular product's tools.
• The Scaffolding Matters More Than the Interface: A Controlled Comparison of MCP and CLI Tool Use Across Seven Agent Scaffoldings, Five Language Models, and One Software Task (paper, 2026): Runs one software task through MCP and through a plain command line across seven agent scaffoldings and five models, checking the repository rather than the agent's own report; the scaffolding moved cost far more than the interface, and agents often ignored the one assigned. Limit: One task of six git operations, as the title says; the paired MCP-to-CLI cost ratios were unstable, so they should not be read as general figures for either interface.
• MCP-GRANITE Benchmark: GRANularity Interface TEsting for MCP-Based LLM Agents (paper, 2026): Treats how finely functionality is split into tools as a controlled variable across 81 multi-step scenarios at four levels, from many primitives to one monolithic tool; at a well-matched granularity a 3.2B model beat a 20.9B model at a poor one. Limit: Uses nine small, locally run models in edge and IoT scenarios, so the best granularity found here may not hold for large hosted models or for other domains.
• SilentProbe: Measuring Silent Failure in Production APIs Used as Agent Tools (paper, 2026): Shows why an HTTP 200 is not enough for agents: constraints written only in prose let a misunderstood query return an empty success, while schema-encoded constraints produced honest errors, and models in the agent loop rarely noticed the silent failure. Limit: Several authors work for the aggregation service used to reach the 27 vendors' endpoints, a competing interest the paper declares, and live calls were limited to read-only endpoints reached through that one layer.
• Hybrid Semantic Tool Discovery for Enterprise MCP Gateway: Architecture and Implementation (paper, 2026): Describes a gateway pattern for large MCP estates: rather than loading over 2,000 tool schemas from more than 200 servers, the agent gets one meta-tool that searches the catalogue with combined keyword and vector retrieval and another that runs the chosen tool. Limit: An experience report on one company's gateway and catalogue; retrieval quality rests on 45 usable queries written for that catalogue, and poorly described tools may be missed with no way to detect it.
• Screenshots or Tools? Eliciting Tool Use and Managing Multimodal Context in Hybrid GUI-MCP Computer-Use Agents (paper, 2026): On OSWorld-MCP, adding the same MCP tools helped a reasoning model and hurt a non-reasoning one, and even the reasoning model called a tool on under a quarter of the tasks where one applied: offering a tool does not mean the agent will use it well. Limit: Compares two checkpoints of one 8-billion-parameter open model on one benchmark with five runs each, so the direction of the effect for other models and harnesses is not established.

Model Context Protocol

• Model Context Protocol architecture overview (specification, 2026): Gives the MCP mental model for hosts, clients, servers, tools, resources, prompts, and discovery, which is useful when designing agent-facing surfaces. Limit: It is high-level documentation; implementation work still needs the specific dated specification pages and SDK behaviour.
• Model Context Protocol tools specification (specification, 2026): Defines how MCP servers expose callable tools, including metadata and schemas that agents and clients use to understand possible actions. Limit: The spec defines the contract shape; product teams still need to write clear names, descriptions, input schemas, examples, and error responses.
• Model Context Protocol resources specification (specification, 2026): Defines the resource layer for exposing context such as files, data, and app-specific information to agent clients. Limit: It does not choose the semantic taxonomy or freshness policy for a particular product's resources.
• The MCP Registry (specification, 2026): Explains the official MCP Registry's architecture as metadata-only infrastructure meant for downstream aggregators rather than a store end-user MCP clients query directly, and documents its namespace-based publisher trust model. Limit: The registry was in preview status at retrieval time, so behaviour may change before general availability, and it explicitly delegates security scanning to underlying package registries instead of performing its own.
• MCP client best practices (documentation, 2026): Official guidance for MCP hosts on progressive tool discovery and on keeping large intermediate results out of the model context, the client-side counterpart to the server specifications above. Limit: Pinned to the 2026-07-28 documentation set; it should be rechecked against newer dated releases of the protocol.
• MCP Apps (ext-apps) (specification, 2025): The official specification and SDK for an MCP extension that lets a server return interactive user interface inside the host, extending agent-facing surfaces from text results to embedded UI. Limit: An extension rather than part of the core protocol, so support differs between MCP hosts.
• SEP-2127: MCP Server Cards - HTTP Server Discovery (specification, 2026): A Final Extensions Track proposal for a static, cacheable card that tells a client where a remote MCP server is, which transports and protocol versions it supports and who runs it, before connecting; tools are left out because they vary per user and session. Limit: The page is a historical record of the accepted design while the normative schema lives in a separate extension repository that can change, and a client still has to connect to learn what tools a server offers.

Protocols and machine-readable surfaces

• OpenAPI Specification v3.2.0 (specification, 2025): OpenAPI remains a practical bridge between HTTP APIs, developer docs, and agent-readable operation metadata. Limit: An OpenAPI file can be syntactically valid while still being poor AX if operation names, examples, errors, and auth flows are unclear.
• JSON Schema Draft 2020-12 (specification, 2020): JSON Schema is a foundation for tool input contracts, structured outputs, API validation, and agent-readable constraints. Limit: Schemas need examples, semantics, and versioning discipline; validation alone does not explain intent.
• Arazzo Specification v1.1.0 (specification, 2026): Describes sequences of API calls that achieve an outcome, including success criteria and failure actions, which is the workflow and recovery layer that per-endpoint OpenAPI descriptions leave an agent to infer. Limit: Less widely published than OpenAPI, and it describes workflows without executing them; an agent still needs a separate way to run the calls.
• Agent2Agent (A2A) Protocol specification (specification, 2025): Defines agent cards for capability discovery and a task-based exchange for delegating work between independently built agents, the agent-to-agent layer that MCP does not address. Limit: The latest URL tracks a moving version, and published adoption figures come from the project itself rather than independent measurement.
• AG-UI: Agent-User Interaction Protocol (specification, 2025): An open, event-based protocol for streaming agent state, UI intents and user input between an agent backend and a user-facing application, relevant to keeping people informed while an agent works. Limit: Governed by its originating project rather than a standards body, and support varies by agent framework.
• Agent Communication Protocols (agentproto) working group charter (specification, 2026): Tracks the IETF process for a working group on user-to-agent, agent-to-agent and agent-to-tool communication, a standards-track venue with more procedural weight than the many individual agent drafts. Limit: A proposed charter under review at retrieval time; its scope was contested and is expected to change before any working group is formed.
• A Survey of AI Agent Protocols (paper, 2025): Compares agent protocols along shared dimensions, giving one place to understand design choices across MCP, A2A and related proposals before reading each specification. Limit: A 2025 survey that predates several later protocols, including the commerce and payment protocols listed below.

Agent identity and authorization

• WorkOS FGA: The authorization layer for AI agents (essay, 2026): Distinguishes on-behalf-of agents, which need scope attenuation under the acting user's own permissions, from autonomous agents, which need fine-grained scope rather than broad client-credential grants, using worked confused-deputy and shared-context scenarios. Limit: A vendor architecture piece promoting WorkOS Fine-Grained Authorization; its one supporting statistic, an average 47-day stale-credential lifetime, is uncited in the retrieved text.
• Agent Registration with Auth.md (specification, 2026): Proposes an open protocol, a discoverable /auth.md document plus three HTTP endpoints extending RFC 9728 Protected Resource Metadata, for an agent to register a new user on their behalf, a gap OAuth's browser-consent flow does not cover. Limit: Adoption outside WorkOS's own ecosystem is unstated in the source; as an author-organisation announcement, independent implementation evidence is still needed.
• Understanding Authorization in MCP (specification, 2026): Explains consent, scopes, and OAuth-oriented authorisation patterns that matter when agents act through user-authorised tools. Limit: Remote and local transports differ, so the same authorisation pattern will not fit every deployment.
• HTTP Message Signatures (RFC 9421) (specification, 2024): The IETF standard for signing HTTP messages, on which verified-bot schemes and several agent identity proposals build so that a request can carry a verifiable signature. Limit: Defines the signing mechanism only; it says nothing about which signers to trust or what a verified agent may do.
• Web Bot Auth (documentation): Documents how bots and agents can prove operator identity on each request with HTTP message signatures instead of a spoofable user-agent string. Limit: Authenticates the operator of the agent, not the person it acts for, so it does not solve delegated or on-behalf-of authorisation.
• AI Agent Authentication and Authorization (Internet-Draft) (specification, 2026): A multi-author Internet-Draft describing how existing standards, the WIMSE workload identity architecture and the OAuth 2.0 family, can be applied to agent authentication and authorisation without a new protocol. Limit: An individual Internet-Draft, not an RFC; it may change or expire, and it composes existing standards rather than settling open questions such as multi-hop delegation.
• Agent Identity overview (Google Cloud IAM) (documentation): Shows one cloud provider giving each agent its own attested, SPIFFE-based identity distinct from human and service accounts, a concrete implementation to compare with vendor-neutral proposals. Limit: Describes one provider's implementation, which does not transfer directly to other clouds or self-hosted identity systems.
• Agent Name Service (ANS) for Secure AI Agent Discovery v1.0 (specification, 2025): A security-community design for agent naming and discovery that ties agent names to PKI-backed identity and adapts across A2A, MCP and other protocols. Limit: A working-group whitepaper rather than a deployed system, and independent adoption is not documented on the resource page.
• AI Agent Standards Initiative (NIST) (framework, 2026): A government standards programme focused on interoperability, security and identity for agents, useful for tracking where public-sector guidance on agent identity is heading. Limit: A programme page rather than a standard; concrete published outputs were limited at retrieval time.
• Claim Deployments (documentation): Documents a provision-first, claim-later flow: an agent deploys through the API, requests a transfer code valid for 24 hours and gives the person a claim URL, after which the project and supported linked databases move into their account. Limit: One vendor's implementation of the claim step; the page covers transfer mechanics but not how the agent is identified or what happens to deployments that are never claimed.
• Claimable Neon (documentation): The same claim-later pattern for a database: an agent provisions a Postgres project without an account, an unclaimed project expires after 72 hours with storage and transfer caps, a claim code lasts 15 minutes, and discovery runs through auth.md. Limit: Vendor documentation for one service; the expiry windows and caps are given as current values, and the page reports nothing on how often projects are claimed or abandoned in practice.

Agentic commerce and payments

• Agentic Commerce Protocol (ACP) (specification, 2025): The specification repository for an open checkout protocol in which an agent passes a scoped payment token while the merchant remains merchant of record, with schemas and a changelog. Limit: Maintained by two companies at retrieval time, and implementing it does not by itself list a merchant on any AI shopping surface.
• Agent Payments Protocol (AP2) (specification, 2025): Specifies signed mandates that record what a person authorised an agent to buy, aimed at verifiable authorisation for payments and offered as an extension to A2A and UCP. Limit: Partner counts are supplied by the sponsors, and independent evidence of production use was not found in this pass.
• Universal Commerce Protocol (UCP) (specification): An open specification for agentic commerce covering discovery, checkout and post-purchase steps, with a public repository, SDKs and reference implementations. Limit: Written from its sponsoring companies' perspective, and cross-protocol interoperability testing is not documented on the site.
• x402 (specification, 2025): An HTTP-native payment protocol built around the 402 Payment Required status code, so an agent can pay for a request in the same exchange, a different mechanism from checkout-based protocols. Limit: Early deployments centre on stablecoin settlement, which brings custody and regulatory considerations that card-based checkout does not.
• How Stripe is designing Checkout for AI agents (essay, 2026): Describes a payment page that exposes WebMCP tools progressively, so a submit tool appears only once the form is valid, and that derives form tools from the rendered HTML rather than maintaining a separate agent-only checkout. Limit: A vendor engineering post; its efficiency figures come from its own 60 test runs across six models on a fictional shop, and WebMCP is an experimental browser capability whose API may change.

Human control, approval and recovery

• What You Approve Is What Executes: Consent Integrity for Black-Box LLM Agents (paper, 2026): Names the missing 'Consent Integrity' property for agent approval dialogs, the action shown must be rendered by a trusted mediator from the real action rather than the agent's own narration, and tests a prototype against 1,330 trusted-tool abuse techniques. Limit: The prototype analyser, renderer, and execution binder is a research system tested on the GTFOBins and tldr corpora, not a deployed product feature available to adopt directly.
• Loopjacking: Hijacking Human-in-the-Loop Approval (paper, 2026): Defines Loopjacking as any case where the operation a human approves does not match what a system actually executes, and reproduces both representation-based and post-approval state-substitution variants in named, released agent products. Limit: The paper names a purposive set of tested products rather than a random or exhaustive sample, so prevalence across the wider agent-product ecosystem is not established.
• Oversight Has a Capacity: Calibrating Agent Guards to a Subjective, Fatiguing Human (paper, 2026): Measures inter-reviewer agreement on a hand-labelled set of 125 adversarially-weighted agent actions and models the human reviewer as fatiguing under escalation load, finding that more oversight can reduce realised safety past a point. Limit: The reviewer-fatigue model and the 125-action labelled set are the authors' own construction; whether the same capacity limit holds for other review populations or action types is not tested.
• Design Principles for Human-Agent Interaction (paper, 2026): Proposes principles across the stages of an agent interaction, including when things go wrong, and applies them to existing agent systems, separating graceful failure from raw task completion as a design goal. Limit: A position paper with an illustrative rather than statistically validated evaluation.
• Delegating or Doing? Understanding User Behavior in Hybrid Human-Agent Interfaces (paper, 2026): A 73-participant study of an interface with an MCP-connected assistant finding that assistance reduced interaction effort but not task time, and that the choice to delegate followed personal disposition more than task risk. Limit: One application and one participant pool, so findings about who delegates may not generalise to other domains.
• GoEX: Perspectives and Designs Towards a Runtime for Autonomous LLM Applications (paper, 2024): Proposes post-facto validation, undo and damage confinement for actions an agent has already taken, a reference design for recovery when review happens after execution. Limit: A research runtime, not a hardened dependency; teams still have to define undo semantics for their own actions.
• Human-in-the-loop (OpenAI Agents SDK) (documentation): Documents how an agent run pauses for approval of sensitive tool calls and resumes from saved state, a concrete framework-level pattern for approval gates. Limit: Specific to one SDK's run and approval model, so the mechanism does not transfer directly to other frameworks.
• Agent Approval Laundering: Transitive Effects Beyond the Approved Invocation (paper, 2026): Shows that approving one command or tool call can let through effects the approval never named, such as a package's install scripts or an MCP call's network access, and measures how often approval records in coding agents miss them. Limit: The benchmark (111 approval and trace pairs, 17 holdout workflows) is the authors' own and small, and the proposed fix is shown only as an integration with one coding agent's permission hook.
• When Consent Outlives Context: Residual Authority Replay in Long-Lived Agents (paper, 2026): Shows that approvals persisted across tasks or sessions can be replayed later without fresh consent, and that an attacker can build up such leftover authority through harmless requests first; it also compares how eight production coding agents store approvals. Limit: Attack gains are measured on AgentDojo and Terminal-Bench cases rather than observed incidents, and the per-agent findings describe the versions tested, whose approval storage can change between releases.
• Do User-Authored Permission Policies Improve Protection Against AI Agent Overreach? (paper, 2026): A randomised study of 113 people without software backgrounds: standing allow, ask and never rules cut runtime prompts but blocked less overreach than approving each action, because most rules were set to ask and unrequested actions were then approved. Limit: One scripted day of 18 simulated actions with online participants and no real consequences, so it does not show how rules are written, revised or trusted over months of real use.
• Let the Agent Do It? How Software Practitioners Understand and Make Permission Decisions in Agentic AI Assistants (paper, 2026): Interviews with 18 practitioners and a survey of 115 show how developers decide whether a coding assistant may act, and turn that into requirements for permission systems, such as keeping an agent's permissions separate from the user's intent. Limit: Relies on what practitioners report rather than logged approval behaviour, and its design implications are not built or tested in a working permission system.
• Idempotent requests (documentation): A concrete design for retry-safe writes in Stripe's API: the client sends a unique key with a create or update, and in API v1 a retry with that key returns the saved result of the first attempt instead of running it again, so a timed-out agent can retry. Limit: One vendor's API: key length, pruning after at least 24 hours, the v1 and v2 behaviours and errors on reused keys with different parameters are Stripe's own choices, and the page does not address agents.

Security for agent-facing surfaces

• Exposed by Design: A Dynamic Security Assessment of Internet-Facing MCP Servers at Scale (paper, 2026): Combines passive discovery across eleven data sources with active dynamic testing of 414 production MCP servers, providing scale evidence that most audited public servers currently ship without authentication. Limit: The audited sample, 640 confirmed and 414 tested servers found across four July 2026 measurement runs, is a snapshot of internet-facing MCP servers rather than a comprehensive census of the ecosystem.
• Breaking the Protocol: Security Analysis of the Model Context Protocol Specification (paper, 2026): Argues MCP's security gaps are architectural, naming missing capability attestation and unauthenticated bidirectional sampling as protocol-level issues, then tests 847 attack scenarios across five server implementations to support the claim. Limit: The proposed MCPSec mitigation is the authors' own backward-compatible extension, not an adopted part of the MCP specification, so its real-world deployment status is unverified here.
• Model Context Protocol security best practices (specification, 2026): Collects security guidance for tool exposure, trust boundaries, and defensive defaults in MCP-connected agent systems. Limit: Security guidance is implementation-sensitive; teams still need product-specific threat models and operational review.
• A First Look at the Security Issues in the Model Context Protocol Ecosystem (paper, 2025): Analyses tens of thousands of MCP servers across several registries and describes a two-stage risk: weak registry vetting admits bad servers, then attacker-controlled tool metadata steers the model. Limit: Registry practices change quickly, and the accompanying scanner's heuristics are not claimed to be complete.
• VIPER-MCP: Detecting and Exploiting Taint-Style Vulnerabilities in Model Context Protocol Servers (paper, 2026): An automated auditing method run over a large set of MCP server repositories that confirmed previously unknown exploitable vulnerabilities, evidence that server code, not only descriptions, needs review. Limit: The reported findings are a lower bound shaped by the method's heuristics, not an estimate of total ecosystem exposure.
• From Component Manipulation to System Compromise: Understanding and Detecting Malicious MCP Servers (paper, 2026): Builds a dataset of malicious MCP server proofs of concept and shows attacks spread across tools, resources and prompts, informing detection before and during execution. Limit: The dataset is researcher-authored rather than drawn from attacks observed in the wild, and real-world detection was demonstrated on few servers.
• Mitigating Taint-Style Vulnerabilities in MCP Servers via Security-Aware Tool Descriptions (paper, 2026): Tests adding behavioural risk guidance to tool descriptions as a low-cost mitigation, linking description quality to security outcomes as well as usability. Limit: A description-level mitigation reduces the chance of triggering a flaw but cannot fix the underlying code.
• Prompt Injection Attacks on Agentic Coding Assistants: A Systematic Analysis of Vulnerabilities in Skills, Tools, and Protocol Ecosystems (paper, 2026): Systematises published work on prompt injection against coding agents into a taxonomy covering skills, tools and protocol ecosystems, a map of the attack surface that agent-facing content creates. Limit: A synthesis of existing studies rather than new experiments, so its figures inherit the limits of the underlying work.
• The Framing Gap: Indirect Prompt-Injection Exfiltration Defeats Surface-Level Defenses in Tool-Using Agents (paper, 2026): Shows that reframing the same exfiltration request can move attack success from none to all attempts for one model, and that destination allow-lists and capability isolation close the gap where model-level fixes do not. Limit: Run in a synthetic laboratory setting with mock tools rather than a production deployment.
• Agent Browser Shield (tool, 2026): An open-source browser extension that filters pages before an agent reads them, removing likely prompt injection, masking personal data and stripping dark patterns, a practical mitigation for browsing agents. Limit: An alpha prototype whose rules may change; its published measurement is a small token benchmark, and blocking effectiveness is not independently evaluated.
• Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection (paper, 2023): Presents indirect prompt injection as a new class of attack: instructions planted in content an LLM-integrated application will later retrieve, rather than typed by the user, with a taxonomy of impacts from data theft to injections that spread between systems. Limit: The demonstrations target 2023 systems such as Bing Chat and code-completion engines, and the paper notes that effective mitigations were lacking, so it defines the threat rather than measuring current agents.
• Design Patterns for Securing LLM Agents against Prompt Injections (paper, 2025): Sets out six architectural patterns, including plan-then-execute, dual LLM and code-then-execute, that stop untrusted content from steering an agent's choice of actions, and applies them to ten application case studies with the utility each one gives up. Limit: The case studies are design analyses rather than measured deployments, and the authors say that securing general-purpose agents remains out of reach; the patterns suit narrow, application-specific agents.
• Indirect Prompt Injection in the Wild: An Empirical Study of Prevalence, Techniques, and Objectives (paper, 2026): Measures injected instructions already on the web: about 15,300 validated cases on some 11,700 pages from 1.2 billion URLs, mostly in parts of the response people never see, with controlled tests across 13 models in which only a small share were followed. Limit: Based mainly on Common Crawl, which under-represents logged-in content, and on an indicator list that can miss new phrasings; compliance was tested in controlled runs, not on deployed agents.
• We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs (paper, 2024): Across 16 code models and 576,000 generated samples in Python and JavaScript, models recommended packages that do not exist, names an attacker could register; relevant wherever agent instructions or skills tell an agent what to install. Limit: Measures code generation with 2024-era models rather than agents that install and run packages, so current rates, and how often a hallucinated name is actually installed, are not established.

Evaluation

Benchmarks

• WebArena: A Realistic Web Environment for Building Autonomous Agents (paper, 2023): Models web agents against realistic sites, showing why navigation, site state, and task context matter for agent-facing web design. Limit: Self-hosted benchmark environments are useful for repeatability, but live websites change and need ongoing monitoring.
• Mind2Web: Towards a Generalist Agent for the Web (paper, 2023): Focuses on mapping natural-language goals to web actions, which is a core AX concern for documentation, labels, and machine-readable task affordances. Limit: Dataset-driven results should be treated as evidence about task representation, not as a universal recipe for every website.
• τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (paper, 2024): Centres user-agent-tool interaction, making it especially relevant to AX work on handoffs, clarifications, and domain state. Limit: Domain simulations still simplify messy real systems; use it to guide evaluation design, not to claim production readiness.
• AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents (paper, 2024): Supports analysis of multi-turn agent behaviour rather than only recording a final success or failure, which helps AX reviews separate failure stages. Limit: Benchmark assumptions can age quickly as tools, models, and web environments change, so it should not be treated as a universal scorecard.
• StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Models (paper, 2024): Calls attention to benchmark stability for tool learning, a necessary concern when tracking whether AX changes move an evaluation in a repeatable way. Limit: Stability controls evaluation noise; it does not decide which product affordances matter to a given audience.
• AgentBench: Evaluating LLMs as Agents (paper, 2023): An early multi-environment agent benchmark spanning operating system, database, web and game settings, still used as a reference design for broad agent evaluation. Limit: Scores were measured on 2023 models and are dated, even though the environments are still reused.
• OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments (paper, 2024): Evaluates agents on open-ended tasks in full desktop environments using screenshots and accessibility trees, extending evaluation beyond the browser to whole applications. Limit: Desktop environments are costly to run at scale, and results may not transfer across operating systems or application versions.
• ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agents (paper, 2024): Adds policy compliance and safety to web-agent evaluation in enterprise-style tasks, so a run is judged on whether it stayed within rules as well as whether it finished. Limit: Built on one browser environment framework and a fixed policy set, which bounds what kinds of unsafe behaviour it can detect.
• τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment (paper, 2025): Extends τ-bench so that both the simulated user and the agent can change a shared environment through tools, surfacing coordination failures that single-control benchmarks miss. Limit: Two-party dual control is still a simplification of real systems where several actors change state at once.
• WebArena-Verified (repository, 2025): A verified release of WebArena in which tasks, reference answers and evaluators were reviewed, with deterministic scoring and offline evaluation from network traces for reproducible web-agent results. Limit: Verification is limited to the WebArena task set and its self-hosted sites; it does not add new site types.
• ClawBench (repository, 2026): Evaluates browser agents on everyday tasks across about 160 live websites, intercepting the final submission so agents use real sites without completing real transactions. Limit: Live sites change, so task validity drifts over time in a way a self-hosted benchmark avoids.
• LiveMCPBench: Can Agents Navigate an Ocean of MCP Tools? (paper, 2025): Tests agents against many real MCP servers and tools at once and finds retrieval of the right tool is a main failure point, which ties evaluation directly to catalogue and description design. Limit: A curated task set scored partly by a model judge; a later validity audit in this list examines this benchmark family.
• MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers (paper, 2026): Scores tasks on real MCP servers against required factual claims rather than one expected call path, with a failure taxonomy that separates tool-call errors from reasoning and stopping errors. Limit: A private test split is withheld, so the full results cannot be independently replicated.
• Berkeley Function Calling Leaderboard (BFCL) (benchmark): An academic leaderboard for function-calling accuracy covering single, parallel and multi-turn calls, with later versions adding agentic tasks, widely used as a shared reference for tool-calling ability. Limit: A periodically updated leaderboard subject to the reproducibility concerns raised in the validity audits below; treat single-run rankings with caution.

Evaluation tooling

• METR Task Standard (specification, 2024): A common format for packaging agent evaluation tasks (environment, instructions and optional scoring) so tasks written by one party can run in another party's harness, with adaptors for existing suites. Limit: Standardises packaging, not task quality, and the repository had not been updated for some time at retrieval.
• Inspect Evals (repository, 2024): A maintained collection of benchmark implementations for the Inspect framework from the UK AI Security Institute, letting teams run many agent evaluations through one harness instead of reimplementing each. Limit: Coverage depends on contributions, so newer benchmarks can lag behind their original release.
• Langfuse (tool, 2023): Open-source tracing, evaluation and prompt management for LLM and agent applications, supporting the practice of recording every agent run so failures can be reviewed and turned into test cases. Limit: Self-hosting at scale is an operational commitment, and tracing records what happened without judging whether it was correct.

Methodology and validity

• Benchmarking the Benchmarks: A Validity Audit of Tool-Calling Evaluation (paper, 2026): Audits four major tool-calling benchmark families against 496 expert-reviewed tasks and finds that repeating one benchmark's own setup 23 times produced an 18.9-point score spread, large enough to flip leaderboard conclusions. Limit: The audit covers four named benchmark families (BFCL v4, tau2-Bench, LiveMCPBench, MCP-Atlas); it does not establish that every tool-calling benchmark has comparable reproducibility problems.
• Establishing Best Practices for Building Rigorous Agentic Benchmarks (paper, 2025): Introduces a checklist for task validity, outcome validity and reporting in agent benchmarks, drawn from problems found in published benchmarks, giving teams a pre-publication review tool. Limit: A methodological checklist; it does not by itself re-score widely used benchmarks end to end.
• Evaluation and Benchmarking of LLM Agents: A Survey (paper, 2025): Organises agent evaluation work by what is evaluated and how, which helps a team find the existing benchmark or method that fits a specific AX question. Limit: A survey that summarises rather than resolves disagreements between the methods it covers.
• Do Agent Benchmarks Measure Capability? Protocol Validity in the Age of Agentic AI (paper, 2026): Audits agent traces across many benchmarks and finds reward hacking and protocol exploitation inflating scores on some, a failure mode distinct from ordinary evaluator noise. Limit: Covers a named set of benchmarks; prevalence elsewhere in the field is not estimated.
• The Double Measurement Confound in Agent Benchmarks: De-Scaffolding, Ground-Truth Scoring, and Reliability Beyond the Mean (paper, 2026): Names a confound in which a fixed scaffold makes execution-critical decisions and a scorer may not reflect correctness, and shows that repairing both changes what a leaderboard appears to show. Limit: The repair protocol is demonstrated mainly on one benchmark, and applying it elsewhere needs bespoke work.
• Beyond Pass@k: Measuring Reliability and Security of Agentic Code Generation (paper, 2026): Identifies a common miscalculation of pass@k that can greatly inflate reported agent scores, and proposes reliability and security measures to report alongside it. Limit: The live pilot is very small, and the authors describe the practical effect of their security-adjusted metric as unresolved.
• Demystifying evals for AI agents (essay, 2026): A practitioner walkthrough of building agent evaluations, covering task design, graders and reading transcripts, which complements the academic validity work in this section. Limit: Based on one model provider's internal practice rather than an independently validated method.
• Building AX evals that actually work (essay, 2026): Practical guidance for evaluating how agents use a product: realistic developer prompts, checklist criteria with pass, fail and skip conditions for an LLM judge, judge calibration, at least five runs per condition and pinned, representative environments. Limit: An experience report from one vendor's evaluation work on coding agents, not a controlled study; thresholds such as five runs per scenario are rules of thumb rather than statistically derived.

Readiness instruments

• Is Your Site Agent-Ready? (tool, 2026): A public scanner that checks a site for many emerging agent-facing conventions in one pass, from robots.txt rules and Markdown negotiation to OAuth discovery, MCP server cards, agent skills and commerce protocols. Limit: Measures adoption of conventions chosen by one vendor, and no published evidence yet shows that a higher score predicts agent task success.
• Lighthouse agentic browsing scoring (tool, 2026): An experimental Lighthouse category that audits WebMCP use, agent-relevant accessibility and page stability, reporting per-audit results instead of one weighted number. Limit: Experimental, page-level and subject to run-to-run variation, and no published evidence yet shows that a higher score predicts agent task success.
• Is Agentic (tool, 2026): Scores the public parts of a site that agents can discover, access and use, and documents its method, including known false positives and false negatives. Limit: A point-in-time vendor score whose weighting is the vendor's choice, and no published evidence yet shows that a higher score predicts agent task success.
• AXIS: Agent Experience Index Score (tool, 2026): Open-source tooling that runs a real agent through a scenario and grades the transcript on goal achievement, environment, service and agent dimensions, measuring an observed journey rather than static signals. Limit: Relies on a model judge, mixes product and model quality in one composite, and no published evidence yet shows that a higher score predicts agent task success beyond the scenarios run.
• Agent-Friendly Documentation Spec (specification): An open specification and checker for documentation sites that reports pass, warn or fail per check across discoverability, Markdown availability, page size, structure and URL stability. Limit: Grounded in the author's observations rather than a controlled study, covers web delivery only, and no published evidence yet shows that a higher score predicts agent task success.
• Agent Readiness Model (Factory) (framework): Scores a code repository, not a website, on whether it has the structure, validation and operational signals coding agents need, using gated levels that a repository unlocks in order. Limit: Tied to one vendor's coding agent, the level thresholds are the vendor's own, and no published evidence yet shows that a higher score predicts agent task success.
• AgentReady.market (tool): Scores an online store, not a website in general, on whether an AI shopping agent can find, understand and buy there, from crawler and firewall access to offer data, policies, transaction protocol endpoints and checkout reachability under an agent user agent, using deterministic checks with a published method and no model judge. Its vendor also publishes a pre-registered series relating the score to agent requests about each store. Limit: Vendor material whose checks and weights are the vendor's own. The published series measures agent attention on the vendor's own surfaces, not task success or purchases, and many agent requests in it follow a read of the vendor's own ranked index, so it is weak evidence that a higher score predicts agent task success.

GEO and AI search (adjacent)

• GEO: Generative Engine Optimization (paper, 2023): The paper that named generative engine optimisation and introduced a benchmark for measuring how content changes affect visibility in AI-generated answers, the starting point for most later GEO work. Limit: Visibility gains were measured on the paper's own benchmark and engines at the time, not on current production answer engines.
• Structural Feature Engineering for Generative Engine Optimization: How Content Structure Shapes Citation Behavior (paper, 2026): Separates page structure from wording and reports that structure alone changes citation rates across several generative engines, relevant to how agent-readable pages are organised. Limit: Engine behaviour changes as platforms update, and some quality judgements rely on a rating method the paper does not fully specify.
• From Citation Selection to Citation Absorption: A Measurement Framework for Generative Engine Optimization Across AI Search Platforms (paper, 2026): Distinguishes being cited from actually shaping an answer and shows the two diverge differently across platforms, a correction to measuring GEO by citation counts alone. Limit: Observational analysis of a public dataset rather than a controlled experiment, and platform behaviour shifts over time.
• What Gets Cited: Competitive GEO in AI Answer Engines (paper, 2026): A large factorial experiment across several models that isolates content factors and finds relevance and position matter far more than formatting-only edits for being cited. Limit: Uses a simplified two-document setting rather than realistic multi-source retrieval.
• Optimizing Visibility in Generative Engines: A Critical Survey of Generative Engine Optimization (2023-2026) (paper, 2026): A critical review of GEO research that concludes no reviewed technique yet shows a stable, cross-platform effect over time, the field's own check on optimisation claims. Limit: Its conclusions inherit the short time horizons of the studies it reviews, which the authors acknowledge.
• When Optimization Becomes Manipulation: Defending Generative Search against Malicious Generative Engine Optimization (paper, 2026): Describes automated, agent-driven content manipulation aimed at answer engines and proposes a defence that reduces attack success without penalising benign content. Limit: Evaluated against attack families the authors built, so novel manipulation strategies remain untested.
• Optimizing your website for generative AI features on Google Search (documentation): Primary guidance from one search engine on its AI features, which states that tactics such as content chunking or llms.txt files are unnecessary for its Search, a useful corrective to GEO marketing claims. Limit: Covers one search engine only and says nothing about how other answer engines or agents treat the same signals.

Related lists

• alexngai/awesome-agent-experience (list, 2026): A broad AX list with sections on standards, IETF discovery drafts, governance bodies, articles and case studies, useful for tracing the wider conversation around the sources curated here. Limit: Archived on GitHub at retrieval, entries carry no limitations, and some case studies repeat unverified vendor outcome figures.
• rolandfarkasCOM/awesome-agentic-web (list, 2026): A short list of open-source protocols, SDKs and specifications for the agentic web, including commerce protocols, that is quick to scan for implementations. Limit: Small and recently started, and its framework and orchestration sections sit outside agent experience as scoped here.
• VoltAgent/awesome-agent-skills (list, 2025): A large catalogue of Agent Skills published by product teams and the community, the place to look for existing skills rather than guidance on designing them. Limit: Its own notice says skills are curated rather than audited, so each skill should be reviewed before installation.
• punkpeye/awesome-mcp-servers (list, 2024): A large directory of MCP servers grouped by domain, useful when the question is whether an MCP server already exists for a given product or service. Limit: A directory rather than a review, so it does not assess description quality, security posture or maintenance.
• steel-dev/awesome-web-agents (list, 2025): A directory of browser-agent frameworks, automation tools and related benchmarks, covering the agent-building side of the web that this list deliberately leaves out. Limit: Mostly product and framework listings without annotation of limits, and the list is maintained by a browser-infrastructure vendor.
• pauldebdeep9/awesome-agentic-evaluation (list, 2026): A structured list of agent benchmarks, evaluation tooling and methodology papers, with a separation between benchmark catalogues and rigour concerns that complements the Evaluation section here. Limit: A single-maintainer project, and several domain sections such as cybersecurity sit outside agent experience as scoped here.
• hqhq1025/awesome-computer-use-ecosystem (list, 2026): Organises desktop and browser computer-use agent projects by capability, including accessibility and semantic grounding, for readers who need GUI-agent tooling. Limit: New at retrieval, and a notable share of entries are the maintainer's own projects.

Maintenance notes

• Retrieval date for the initial source pass: 2026-09-07.

• Retrieval date for the 2026-09-25 sources update (20 new entries plus one corrected URL, emphasising sources published or updated from 2026-06-25 onward): 2026-09-25.

• Retrieval date for the 2026-09-26 expansion (96 new entries after reconciling other public AX lists, a multi-engine web search sweep and the maintainer's own published research; sections reorganised into a reader's path, with subsections for tools and evaluation): 2026-09-26. Every URL was rechecked online on that date.

• Retrieval date for the 2026-10-07 update (five new entries on AX case studies, agent-facing CLI design, a knowledge format for agents and WebMCP checkout, plus the corrected Open Agent-Readiness Rubric name): 2026-10-07. Every URL was rechecked online on that date.

• Retrieval date for the 2026-10-12 update (26 new entries on approval and residual authority, prompt-injection and package-hallucination research, tool-calling and tool-granularity research, tool search, AGENTS.md evaluation, llms.txt evidence, agents.txt and AI Catalog drafts, claim-later onboarding and AX framing, plus the same-maintainer disclosure updated for rubric v0.3): 2026-10-12. Every URL was rechecked online on that date.

• Source metadata lives in data/sources.json. The section list and its order are enforced by scripts/check-links.mjs.

• Linked papers, specifications, frameworks, and essays keep their original copyrights and licenses. This repository licenses only the original curation text and repository scaffolding.

• Link checks are intentionally dependency-free. Run node scripts/check-links.mjs for offline structure checks, or node scripts/check-links.mjs --online for bounded URL verification.

• The same-maintainer disclosure above still applies: this list is maintained by the author of the Open Agent-Readiness Rubric, the AX evidence register and the ax-check rules, which are deliberately not listed as entries.

• Maintenance and source review explains the curation cadence and generated README workflow.