fix(attachments): stop pasted screenshots going invisible to the agent - #915
wojciechszyjka wants to merge 1 commit into
Conversation
Cezar hands a pasted screenshot to the agent twice — as a base64 block it can
view, and as an on-disk path it can operate on. The block is the fragile half:
it rides the FIRST agent step only, `textOf()` drops it on the codex and
opencode runners, and a compacted session loses it. The path is the durable
half and was being described as if it were only for moving files around, so an
agent that could not see the image had no reason to believe reading the file
would show it one. The reported symptom is an agent answering "I can't see your
screenshot" with the PNG one Read away.
Three fixes, each independent:
- The prompt notes now say that reading a path is how you SEE an attachment,
not just how you save or upload it, and that the inline copy is the part that
goes missing. Both surfaces are updated: `pastedAttachmentsText` (per message)
and the `## Pasted attachments` handoff fragment (every agent step), whose old
closing clause — "the inline image is for viewing only" — actively taught the
opposite.
- `startAttachments` no longer resets alongside `startImages` after the first
agent step, and `runAgentStep` appends the path note ungated by `images`. The
blocks stay first-step-only (re-sending them is pure token cost), but a
workflow's second step and every post-check retry now still learn that the
screenshot exists and where to read it. Before, they got neither.
- The composer screens clipboard items on `kind === 'file'` instead of
`type.startsWith('image/')`, making `screenFiles` the single judge on both the
paste and drop paths, and `screenFiles` infers a media type from the extension
when the browser reports none — naming the file in a toast when it cannot.
A screenshot the OS handed over untyped used to vanish with no thumbnail and
no error, which reads exactly like a paste that never registered.
Every new assertion was confirmed red against the unfixed source.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
wojciechszyjka
added
bug
|
Labels:
The agent-facing half is prompt text, so its effect shows up as behavior rather than a screen: an agent that cannot see an inline image should now open the path instead of reporting the attachment missing. |
📦 npm preview published —
|
QA evidence — self-verifiedChecked the branch out and ran it, per the self-QA exception in
Confirmed the served bundle actually carries the change before testing: the new toast string is present in the candidate's built assets and absent from the baseline's, so nothing below is a stale-cache reading. 1. Agent-facing: a second workflow step keeps the attachmentA two-agent-step workflow, started through
On The persisted file is byte-identical to the source screenshot (643842 B, compared binary). 2. UI: the composer, driven in a real ChromeA headless Chromium dispatched genuine
On the baseline both middle cases vanished with no thumbnail and no toast, which is indistinguishable from a paste that never registered. Screenshots captured for all four. 3. Manual pass by a human, through the browserA maintainer tunnelled into the candidate cockpit and pasted a screenshot into a live task's composer by hand. Observed end to end:
Not covered
Applying |
wojciechszyjka
added
qa-approved
1 participant
The bug
A user pastes a screenshot into a task; the agent answers "I can't see your screenshot." The PNG is on disk, one
Readaway, and the agent was told about it — just not in a way that suggested opening it would show it the picture.Cezar hands a pasted attachment over twice: as a base64 image block (viewable) and as an absolute path (operable). The block is the fragile half and goes missing on three separate paths; the path is durable and was under-sold.
What was wrong
1. Both prompt surfaces described the path as move-only.
pastedAttachmentsTextsaid "When the task involves saving, uploading, attaching, or transforming the pasted content … operate on these files", and the## Pasted attachmentshandoff fragment closed with "the inline image is for viewing only" — which reads as no inline image, no attachment. Neither told an agent that couldn't see the image that reading the file was how to see it.2. A later workflow step got nothing at all.
execute()clearedstartAttachmentsalongsidestartImagesafter the first agent step, andrunAgentStepappended the path note only insideif (images?.length). So on a two-agent-step workflow — or any post-check retry — step two received neither the blocks nor the paths. The run said "1 screenshot attached to the task" and the step that did the work never heard about it.3. An untyped clipboard file vanished silently.
onPastescreened items ontype.startsWith('image/')beforescreenFilesever saw them — a narrower rule than the drop path used. AFilethe OS handed over withtype: ''(some Linux file managers and clipboard sources) produced no thumbnail and no toast: indistinguishable from the paste never registering.The fix
pastedAttachmentsText,HANDOFF_ONLY_INSTRUCTIONS) now state that reading a path is how you see an attachment, name the three cases where the inline copy is dropped, and say not to report a missing attachment without opening its path first.run.tskeepsstartAttachmentsalive across steps and appends the note ungated byimages. The base64 blocks stay deliberately first-step-only — re-sending megabytes to every retry buys nothing the path note doesn't — but the paths now ride the whole workflow.kind === 'file', makingscreenFilesthe single screening authority on both paste and drop, andscreenFilesinfers a media type from the extension when the browser reports none, naming the file in a toast when it can't. Extension inference is restricted to the four formats the backends actually decode (png/jpeg/gif/webp) — guessingimage/svg+xmlwould trade a silent drop for a rejected API request — and uses aMaprather than an object literal, since the key is a user-supplied filename fragment.Behavior deliberately unchanged: a typed non-image (
text/plain,application/pdf) is still ignored in silence — dropping a text file on the composer is not an error.Coverage
Nine new assertions across
pasted-attachments.test.ts,composer-images.test.tsandcomposer.test.tsx, including an end-to-end two-agent-step run throughCEZ_DRY_RUN=1that asserts step two seesimageCount: 0and the real on-disk path.Per
AGENTS.md§ Prove the regression test fails without the fix: with the four source files stashed and the tests kept, 13 assertions go red; restored, all pass.Validation
The full ordered gate from
.ai/agentic.config.json, run locally:npm run typechecknpm testnpm run test:unitnpm run buildcheck:pack475 filesnpm run test:packageNo contract surface touched:
BACKWARD_COMPATIBILITY.md§8 freezes theCEZ:*marker vocabulary inhandoff.ts, not the surrounding prose, and no marker changed.🤖 Generated with Claude Code