A multimodal web agent for studying why multi-step browser agents fail — and whether giving an agent more structure at plan time, rather than a bigger model, is what actually fixes it.
Decido started from AI2's MolmoWeb agent. I read the paper, wanted to build a web agent of my own from scratch, and seeded my planning agents with MolmoWeb's table of atomic web skills — the same taxonomy that shows up in core/skill.py, where the SkillPlan docstring cites it directly. Decido is not a reimplementation of MolmoWeb; it borrows the key idea (plan at the level of named web skills, not raw clicks) and builds a different system around it — where MolmoWeb navigates from screenshots alone, Decido pairs a vision planner with a DOM planner and ranks their plans against each other. The debt is real and I want it named up front.
The other thread running through this project is a research one. I was a research assistant at the UW Computer Systems Lab studying why personal multi-step agents fail. The lab's working hypothesis was that these agents need more deployment context, not more training — that the failures come from the agent not knowing what it has already done, what order things go in, or whether its last action landed, rather than from a weak underlying model. Decido's central experiment is direct evidence for that hypothesis, and the results section below is built around it.
Two planners propose high-level skill plans in parallel for a given task:
- a DOM planner (gpt-5-nano) reading a structured text observation of the page, and
- a vision planner (Qwen2.5-VL-7B, self-hosted on Modal) reading a screenshot.
A ranking layer scores the plans, a deterministic compiler grounds the winning plan into primitive browser actions against the live DOM, a Playwright executor runs each primitive with per-primitive verification, and a heuristic task checklist plus an independent LM evaluator decide when to stop. Every session ends tagged with a failure mode.
Benchmark: 34 tasks across 6 categories, each run 3 times = 102 sessions per configuration, on live demo sites (the-internet.herokuapp.com, httpbin.org, books/quotes.toscrape.com, saucedemo.com, demoqa.com, seleniumbase.io, Wikipedia). Most tasks carry independent verification: URL/text checks the harness runs against the final page, which override the agent's self-report in both directions (see the verification section).
| Configuration | Verified success | Notes |
|---|---|---|
| v2 atomic runner, May 2026 (before fixes) | 1/25 (4%) | old suite; 17 loop terminations |
| Atomic runner, both agents (with all shared fixes) | 70/102 (69%) | 40 loops, 7 false self-reports |
| Skill runner, DOM only | 93/102 (91%) | 6 loops, 6 premature terminations |
| Skill runner, DOM + vision | 99/102 (97%) | 8 loops, 1 false self-report |
The jump decomposes cleanly:
- 4% → 69%: shared bug fixes (see failure archaeology) applied to the old architecture. These were bugs, not architecture; fixing them is table stakes.
- 69% → 91% (+22 points): moving from the atomic per-action runner to the skill runner. Same model, same prompts, same everything else. This is the deployment-context result.
- 91% → 97% (+6 points): adding the vision planner as a second proposer.
Vision's contribution is concentrated in visually-ambiguous tasks, not smeared across the board:
books_cheapest: 3/3 with vision vs 1/3 DOM-only (comparing ~20 prices at once)herokuapp_inputs: 3/3 vs 0/3 (an unlabeled bare<input>)herokuapp_add_remove: 3/3 vs 1/3
Honest caveat: DOM-only actually beat both-agents on multi_page (12/12 vs 11/12). A second proposer adds noise as well as signal, and one lost run there is within run-to-run variance. Vision is not free.
This is the part I care about most, so I'll be explicit. Between the 69% atomic run and the 91% skill run, the models did not change — same gpt-5-nano, same Qwen2.5-VL-7B checkpoint, same temperatures (such as they are; see limitations). What changed was everything around the model:
- Checklist decomposition — the task is parsed into an explicit list of sub-goals ("check both boxes", "select this option", "submit"), so the agent knows what "done" means and what is still outstanding.
- Ordering context — the ranker knows which remaining checklist item comes first and rewards plans that match it, so the agent doesn't try to submit before it has filled the form.
- Repeat penalties — successful steps are remembered and re-proposing them is penalized, which is what kills the loop deaths that dominated the atomic runner (40 loops → 6-8).
- Execution feedback — every primitive is verified, and that verified evidence (not the mere presence of a word on the page) is what advances the checklist.
None of that is a stronger model. It is context the model didn't previously have about its own deployment: what it has done, what's left, and whether the last thing worked. Going from 69% to 91% on that alone is the concrete version of the lab's hypothesis. The atomic runner had the same intelligence available to it and used it worse because it was flying blind between steps.
POST /task {task, url, agents, runner}
│
┌───────────────────────────────▼───────────────────────────────┐
│ skill session loop │
│ │
│ observe ──► propose ──► rank ──► compile ──► execute ──► update
│ │ │ │ │ │ │ │
│ structured DOM + cross- LM-free, Playwright checklist
│ DOM obs: vision source text/token per-prim. advanced
│ roles, planners agree- grounding verify from
│ labels, (1-3 each, ment + of winning each VERIFIED
│ selectors, parallel); checklist plan to primitive evidence
│ bboxes, checklist order primitives │
│ below-fold recovery bonus, │
│ flag plans repeat ▼
│ prepended penalty evaluator (separate
│ LM) confirms any
│ success claim
└────────────────────────────────┬───────────────────────────────┘
▼
terminate with a tagged reason:
success | loop | cascade | premature_termination |
step_limit | action_failure | memory_miss | task_ambiguity
The loop, step by step:
- Observe —
core/observation.pyruns one JS pass over the page and returns a structured observation: every interactive element with its role, resolved label, a stable CSS selector, its bounding box, and a below-the-fold flag. This is what the DOM planner reads. - Propose — both planners emit 1–3
SkillPlans in parallel. Deterministic checklist-derived recovery plans are prepended so the ranker always has a known-good option to fall back on. - Rank —
core/skill_ranker.py, formula below. - Compile —
core/skill_compiler.pygrounds the winning plan into primitive actions (fill,click,check,select,press) by text/token matching against the observation. This step is deterministic and LM-free; no model is in the loop between "pick a plan" and "do it." - Execute —
core/skill_executor.pyruns each primitive through Playwright and verifies it individually (URL change, input value, checked state, etc.). - Update — the checklist advances only on verified evidence.
- Evaluate + terminate — an independent evaluator LM call (
core/evaluator.py) confirms any success claim before the session is allowed to end assuccess. Termination reasons are enumerated incore/session.py.
From core/skill_ranker.py:
score = 0.4·confidence
+ 0.4·cross-source agreement
+ 0.2·checklist-order bonus
− 0.3·repeat penalty
- Checklist recovery plans are pinned above scored plans (they're deterministic and safe).
- Plans that compile to zero primitives are skipped at selection time — a plan you can't ground is a loop waiting to happen, so the runner picks the best plan that actually compiles against the current page.
- Cross-source agreement is the project's original insight, lifted a level. In v1/v2 it was IoU overlap between the DOM and vision bounding boxes in pixel space. Here it's the same idea in plan space: agreement = skill-family match + target-text overlap. When both planners independently converge on the same skill against the same target, that convergence is stronger evidence than either one's self-reported confidence.
A deliberate echo of MolmoWeb's atomic-skills table. The WebSkill enum in core/skill.py:
| Skill | Meaning |
|---|---|
go_to |
wait for / confirm arrival at a URL |
search |
type a query into a search box and submit |
find |
locate and click a matching element |
find_and_open |
locate a link and follow it |
find_and_click |
locate a control and click it |
fill_form |
fill named fields |
fill_form_and_submit |
fill fields, then submit |
apply_filters |
set filter/selection controls |
apply_filters_and_search |
set filters, then submit |
add_to_cart |
click a real add-to-cart control (never the product link) |
navigate |
click a link that leads somewhere |
Success signals derived from the page — "the DOM mutated", "a URL changed" — produce false positives constantly. A button label that reads "Add to cart" contains the word "cart"; a confirmation area is present before anything is confirmed. If you trust the agent's self-report, you overcount.
So most tasks in tasks/suite.json carry a verify block — url_contains, url_excludes, text_contains, text_excludes — that tasks/runner.py checks against the final page state. Verification overrides self-report in both directions: a task the agent thought it failed but that verifies passing counts as a pass, and a task the agent proudly reported complete that fails verification counts as a fail.
The punchline: in the atomic arm, 7 of 60 self-reported successes were false. In the skill runner, that number was 1 in 102. The verification-first stance is why the headline numbers above are trustworthy rather than flattering.
Short war stories from getting here. Each of these was a real bug found on live sites, and several account for whole clusters of early failures.
- The evaluator never ran. The completion evaluator had never once executed successfully in the entire history of the project. It passed
max_tokensandtemperature, both of which gpt-5-nano rejects; the exception was swallowed and every call fell through to returning"incomplete". No session could ever be judged complete by the evaluator. This explains most of the early loop deaths. - The vision agent was structurally starved. Qwen2.5-VL answers in its processor's smart-resized coordinate space, and those coordinates were never rescaled back to page pixels. IoU agreement on a 4-pixel radio-button box rounds to zero. Lifetime v2 stats: 107 vision candidates logged, 2 ever selected, and both of those failed. The vision agent looked useless because it was being fed a broken coordinate frame, not because it was bad.
- Checklist false successes. "Click X" could terminate
successat step 0 because X was merely visible on the page. Fixed by requiring verified click evidence, not mere presence, to complete aCLICKEDitem. - Reshuffling element IDs. The enumeration IDs (
e0,e1, …) are re-assigned every observation, so post-action verification could end up checking a different element than the one that was acted on. Stable CSS selectors fixed it. - Real-web compilation lessons (each one a live-site surprise):
- Submit search with Enter, not by clicking the button — real keystrokes open suggestion overlays that re-render the header and invalidate the search button's pre-computed selector.
- "Add to cart" must target the button, never the product title link (navigates away), and never a different product's button — hence the same-card geometric guard in the compiler.
- Unlabeled password fields have to match on
input_typebecause there's no label to match on. - Icon-only SPA links (saucedemo's cart anchor has no
hrefand only a badge number as its text) need names derived fromhref/data-test/class attributes. - Sortable
<th>headers are interactive but invisible to standard interactive-element selectors, so the observer explicitly includes them. - Playwright's
fill()fires no key events, so pages that listen for keystrokes needpress_sequentially. - "Click Me" must not tie with "Double Click Me" — an exact-name equality bonus in the matcher breaks the tie.
- Counting tasks ("click the button three times") aren't representable in the checklist. The one such task in the suite passes only via weak verification.
- No temperature control on gpt-5-nano means run-to-run variance is inherent — which is exactly why every benchmark is 3 repeats.
- Rollback restores navigation state only. Server-side effects (form submissions, purchases, deletions) are unrecoverable. That's a limitation, but it's also a finding: not all failure states can be undone.
- The suite is demo sites, not WebArena-class. These numbers are for studying failure modes, not a SOTA claim — please don't read them as one.
- The compiler is text-matching based. A page with no matchable text anywhere would need vision grounding at the primitive level, which Decido doesn't do yet — vision currently only proposes plans, it doesn't ground clicks.
Requirements: Python 3.12, Playwright (Chromium), an OpenAI API key, and a Modal account for the vision server.
.env:
OPENAI_API_KEY=...
MODAL_TOKEN_ID=...
MODAL_TOKEN_SECRET=...
Install and deploy the vision server:
pip install -r requirements.txt
playwright install chromium
modal deploy modal_app/vision_server.pyServe the API:
uvicorn api.main:app --port 8000Run a single task:
curl -X POST http://localhost:8000/task \
-H 'Content-Type: application/json' \
-d '{"task": "Log in with username tomsmith and password SuperSecretPassword!",
"url": "https://the-internet.herokuapp.com/login",
"agents": "both", "runner": "skill"}'agents is both | dom | vision; runner is skill | atomic (the atomic runner is the v2-style per-action baseline, kept for comparison).
Run the benchmark:
python -m tasks.runner --agents both --runner skill --repeat 3
python -m tasks.runner --agents dom --runner skill --repeat 3 # DOM-only ablation
python -m tasks.runner --filter form --repeat 1 # just the form tasks--filter matches on name, category, or difficulty substrings; read tasks/runner.py for exact semantics. Results land in tasks/results_<runner>_<agents>_<timestamp>.json.
On macOS, wrap long runs in caffeinate -dims — the machine sleeping mid-run corrupts sessions. The runner retries infrastructure errors (network flaps, HTTP 500s) once automatically before scoring.
A trained scorer. The candidates and episodes tables now hold roughly 300 sessions of logged plan candidates, each with its features (confidence, agreement, order bonus, repeat penalty, latency, whether either planner silently failed) and its verified outcome. That is exactly the learning-to-rank dataset the v3 plan called for. The ranking weights (0.4 / 0.4 / 0.2 / 0.3) are hand-tuned right now; the obvious next step is to fit them, starting with logistic regression, and see whether a learned scorer beats the hand-tuned one on held-out tasks.
A vision-only ablation arm. The mode exists (--agents vision) but I haven't benchmarked it — a vision-only run would measure what the screenshot contributes in isolation, rather than only as a second voice.
A harder task suite. The demo sites were the right call for isolating failure modes cheaply, but they cap out. Moving to WebArena-class tasks would tell me whether the deployment-context result holds when the tasks are genuinely hard, or whether it was partly an artifact of easy pages.