Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
b8be33a
fix(protocols): make unsupported image references explicit
invalid-email-address Sep 5, 2026
da1e5dd
test(protocols): cover external image wire roundtrips
invalid-email-address Sep 5, 2026
1c0735f
fix(vision): align tool image captions with parsed references
invalid-email-address Sep 5, 2026
5060ac8
test(state): isolate late-spill ordering from wall-clock reserve
invalid-email-address Sep 5, 2026
7484cc5
test(server): reserve loopback ports during public allocation
invalid-email-address Sep 5, 2026
75dc09e
fix(adapters): preserve orphan tool image carriers
invalid-email-address Sep 5, 2026
01e3cfb
fix(cursor): prepare active tool screenshot batches
invalid-email-address Sep 5, 2026
1775a4c
Merge remote-tracking branch 'origin/dev' into codex/image-stack-l2-38cb
invalid-email-address Sep 5, 2026
85c1e9a
Merge branch 'codex/image-stack-l2-38cb' into codex/image-stack-l3-38cb
invalid-email-address Sep 5, 2026
055cedb
Merge branch 'codex/image-stack-l3-38cb' into codex/image-stack-l4-38cb
invalid-email-address Sep 5, 2026
8d05059
Merge branch 'codex/image-stack-l4-38cb' into codex/image-stack-l5-38cb
invalid-email-address Sep 5, 2026
d6f63ac
Merge remote-tracking branch 'origin/dev' into codex/image-stack-l2-38cb
invalid-email-address Sep 5, 2026
94df1ef
Merge branch 'codex/image-stack-l2-38cb' into codex/image-stack-l3-38cb
invalid-email-address Sep 5, 2026
a244ba8
Merge branch 'codex/image-stack-l3-38cb' into codex/image-stack-l4-38cb
invalid-email-address Sep 5, 2026
083488e
Merge branch 'codex/image-stack-l4-38cb' into codex/image-stack-l5-38cb
invalid-email-address Sep 5, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 16 additions & 4 deletions devlog/_plan/260905_external_image_roundtrip/000_plan.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,12 +9,13 @@
release, image synthesis, or unrelated adapter refactor.
- Verifier: standalone converter/parser/adapter body inspection, TypeScript, exact-head
GitHub CI. ALL local test suites are forbidden by the user, including focused suites.
- Stop: reviewed two-layer stack merged bottom-up to dev with green CI and ancestry.
- Stop: reviewed image-repair stack merged bottom-up to dev with green CI and ancestry.
- Memory: this unit and the session-bound goalplan/ledger.
- Outcomes: DONE only with proof; external dependencies may be BLOCKED/NEEDS_HUMAN;
unsafe expansion is UNSAFE. No implementation-success claim from docs-only work.
- Scope: this managed checkout, read-only Aside official docs, GitHub stack/CI/admin
merge. Maximum four concurrent agents; reassess after 90 minutes; no token cap set.
merge. The user's follow-up permits unlimited useful parallel agents (subject to
actual tool capacity); reassess after 90 minutes; no token cap set.
- Escalation: reclaim a lane after two distinct failed agents; any delegated writes
must be planned with disjoint paths before B. No production credentials in artifacts.

Expand Down Expand Up @@ -45,8 +46,13 @@ serialization. Do not add a generic image helper or patch correct Responses code

1. wp0: docs-only roadmap and independent audit (this cycle).
2. wp1 / 010: preserve Chat image detail and structured tool output; lower PR to dev.
3. wp2 / 020: cross-protocol wire regressions and public contract; child PR to lower
branch, then CI/review/admin-merge bottom-up, retarget child and verify again.
3. wp2 / 020: cross-protocol wire regressions and public contract; child PR to lower.
4. User-expanded wp3 / 030: file references and explicit unsupported computer-output
boundary. 003 records full format coverage and rejected hypotheses.
5. wp4 / 040: orphan tool image carriers on Anthropic and Command Code.
6. wp5 / 050: active external Cursor tool screenshot attachments.
7. wp6 / 060: all-format finding disposition, CI/review/admin-merge bottom-up, retarget
children and verify exact heads/ancestry. Original completion criteria are unchanged.

Existing placement is reused: src/chat/, tests/responses/, public reference/proxy-formats,
structure/04_transports-and-sidecars.md. No new package, runtime module, or config.
Expand All @@ -55,6 +61,12 @@ converter and protects the integrated contract independently of unit-level asser

## Continuity

wp1 outcome: commit `1f1daa368` implements 010 with ten converter regression cases;
draft PR #3586 targets dev. Independent patch reviewer inspected both changed files
and returned PASS. Standalone request JSON changed from image/detail missing (exit 1)
to both retained (exit 0); node TypeScript and privacy scan passed. Suites are CI-only,
not claimed green yet. wp2 inherits this verified converter and adds wire/HTTP evidence.

Roadmap audit: independent gpt-6-astra high reviewer returned GO-WITH-FIXES,
two medium findings. Both folded: exact no-suite typecheck/push commands and actual
Claude converter export. Direct node tsc exits 0. Standalone reproduction at
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
# All-image-format audit, 2026-09-05

The user expanded the review from the reported OpenAI paths to every image format.
Two independent gpt-6-astra high reviewers inspected ingress and outgoing format
families. These are source-verified gaps, not claims that the first patch introduced
regressions. No unsupported provider-specific file resolver is being invented.

| Surface | Supported transport / current boundary | Disposition |
| --- | --- | --- |
| Chat user image_url object/data/HTTPS/detail | Existing URLs preserved; detail fixed in 010 | Covered by lower and wire PRs |
| Chat tool image arrays/string URLs | OpenCodex extension, not standard upstream tool-role support | Fixed in 010; native Chat keeps its contract |
| Responses message input_image | URL/data; file_id native or text marker translated | Existing supported behavior |
| Responses function/custom output file_id | Raw native retains reference; translated parser drops it | 030: use existing file marker convention |
| Responses computer_screenshot output | Raw native retains item; translated parser ignores screenshot | 030: explicit translated400, native unchanged |
| Claude user/tool base64 and URL | Dedicated mapper and nested tool outputs | Covered by wire matrix |
| Claude source:file | Native reference valid; translated mapper drops it | 030: explicit translated error, no cross-provider resolution |
| Responses/Azure | Native raw inputs and repairable orphan images retained | No additional loss found |
| Chat/Mimo | User-image carrier after pending tool batch | No additional loss found |
| Anthropic | Paired image result works; orphan baseline JSON-inlines image data | 040: native image sibling with provenance; baseline standalone exit1 confirmed |
| Command Code | Paired image carrier works; orphan baseline skips it | 040: reuse wireImagePart on orphan carrier; baseline standalone exit1 confirmed |
| Google/Vertex/Antigravity | Data -> inline_data, tool image siblings | Remote-URL marker remains existing limitation |
| Kiro | Data images on user carrier; orphan pairing rejected | Remote URL remains existing limitation |
| Ollama native | Data/raw base64 images; unsupported URL/pairing rejected | Existing explicit contract |
| Cursor native MCP | Image bytes carried with tool result | Preserve unchanged |
| Cursor external wire | Active user images work; trailing tool images not prepared | 050: active trailing run only, existing count/byte limits |

Key owners: src/responses/parser.ts:304, :732; src/claude/inbound.ts:134;
src/adapters/anthropic.ts:637, :753, :775; src/adapters/command-code.ts:110;
src/adapters/cursor/images.ts:647; cursor/live-transport.ts:621;
cursor/protobuf-request.ts:1383. Ordinary user images on OpenAI were already retained.

The live 10100 process is version2.43.0 from the maintainer's main checkout, not this
worktree. A safe configuration inspection found no text-only declaration for native
OpenAI. No model request or personal request inspection was done, so the reported
specific OCR failure remains unattributed. Do not infer loaded commit from version.

## Hypotheses and negative controls

- H1 adapter cannot carry images: falsified by paired/native image branches.
- H2 image-bearing representation is dropped on a branch: source evidence above;
confirm each modified owner with a standalone body/encoder probe before editing.
- H3 capability policy intentionally omits images: true for documented URL/history
limits and text-only sidecars, excluded from universal vision-support claims.

No new remote fetching, uploads, auth, provider metadata, tool execution, historical
image recall, or file-handle resolution. Full audit means every row has a disposition,
not that every upstream supports every representation. All new tests execute in CI only.
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,17 @@

Depends on wp1 and its corrected Chat converter. One full PABCD cycle.

Delegated B lanes (user reconfirmed unlimited useful parallelism): worker A exclusively
edits tests/responses/openai-responses-passthrough.test.ts; worker B exclusively adds
the HTTP regression in tests/responses/chat-completions-endpoint.test.ts. Main owns
public docs, structure, devlog and git/CI. C reviewers are read-only and independent.
All lanes prohibit local test suites, services, config/auth and git/FSM mutation.

User steering during B: all image representations must be audited before merge. This
cycle now publishes the wire-contract child; the original exact-head CI/merge/ancestry
criterion is unchanged and moves to appended wp3 after the expanded audit. No criterion
is dropped or marked met early. Only existing 020 implementation runs in this B.

## MODIFY tests/responses/openai-responses-passthrough.test.ts

Import real chatCompletionsToResponsesBody, anthropicToResponsesBody,
Expand Down Expand Up @@ -38,6 +49,10 @@ existing Chat-to-Responses HTTP regression (line 2834). POST a user image with h
detail and a paired tool screenshot to mock/grok-4.5; consume the stream and assert
one captured /responses body with unchanged ordered image parts. This is real HTTP
route proof in CI, not real-model OCR or canonical account authentication.
The manual HTTP probe showed that data-only mock Responses frames don't satisfy the
native event-name terminal observer. Add matching `event: response.output_text.delta`
and `event: response.completed` fields to mockDualWireUpstream's existing frames;
preserve all body assertions and require `[DONE]` on the new HTTP cases.

## MODIFY structure/04_transports-and-sidecars.md

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
# Accepted image representations without silent loss

Depends on 020; wp3. C3 parsing, with C4 care for the explicit translated-file error.
Same resource/credential bounds as000; writes only paths below, no live credentials,
network fetching, remote upload, auth changes or local suites. Stop on unresolvable
native-vs-routed ambiguity, not by weakening native preservation.

## MODIFY src/responses/parser.ts

In outputToToolResultContent, input_image must follow the already-owned precedence:

```diff
- else if (raw.type === "input_image" && typeof raw.image_url === "string") { ... }
+ else if (raw.type === "input_image") {
+ const imageUrl = nonEmptyString(raw.image_url);
+ const fileId = nonEmptyString(raw.file_id);
+ if (imageUrl) { /* existing image push and normalized detail; hasImage=true */ }
+ else if (fileId) parts.push({ type: "text", text: `[image: ${fileId}]` });
+ }
```

Do not lower computer screenshots in this shared parser. Independent audit found
that parser-only lowering shifts vision-caption alignment and breaks native/raw
consistency. No new computer execution or observation-message semantics are added.

## MODIFY src/server/responses/core.ts

Immediately after existing isPassthrough determination (before vision planning), inspect
raw input items. A non-passthrough adapter receiving computer_call_output returns fixed
400 invalid_request_error: `computer_call_output requires a Responses passthrough
route; send screenshots as user input_image content on translated routes.`
No payload, source URL or call ID in the error. Passthrough stays unchanged, including
native/keyed Responses and routed compaction using a passthrough adapter. This avoids
shared-parser rejection of valid native traffic and preserves vision-caption alignment.
No helper/export/import is needed; use the existing raw body and formatted error owner.

## MODIFY src/claude/inbound.ts

In imageBlockToInputImage, after validating source object and before base64/URL cases:

```diff
+ if (source.type === "file") throw new AnthropicRequestError(
+ "File-backed images require native Anthropic passthrough; use base64 or URL images on translated routes.");
```

The existing HTTP boundary catches AnthropicRequestError as400. Native Anthropic
passthrough never calls this converter. No file id, URL or payload echoed in errors.

## MODIFY existing tests

- tests/responses/responses-parser.test.ts: function/custom file-only output marker;
URL wins over file_id; malformed/empty refs never become images; original raw item and
caller object unchanged. No new computer tool declaration or toolCall emitted.
- tests/claude-integration/claude-inbound.test.ts: user/tool source:file throws the
existing error; base64/URL still preserve. Native negative control stays in existing
claude-native-passthrough.test.ts; add endpoint error coverage at its existing seam
only if the reviewer finds class-to400 mapping not covered.
- tests/responses/responses-compaction-routing.test.ts: beside the existing unpaired
output boundary, non-passthrough computer output returns400 with zero upstream fetch;
native/keyed Responses preserves exact raw screenshot and reaches its controlled
upstream; ordinary image message still works. Request includes another ordinary image
to prove there is no partial vision work or caption misassociation before rejection.
- docs-site/src/content/docs/reference/proxy-formats.md: file handles remain provider
scoped; native reference forwarding vs translated marker/error; hosted computer
outputs require Responses passthrough, screenshots can use ordinary input_image.

Verifier: one standalone direct parser/converter probe before/after (no test runner),
node TypeScript, static test bundling, privacy scan, exact-head CI. Independent security
review confirms no native rejection, payload logging, new fetch, or execution authority.
Reuse existing modules; defer broad splits in large files to avoid unrelated churn.
Publish third stacked PR against codex/external-image-wire-contract; no merge yet.

## C-review corrections

Accepted consumer mismatch: output parser's nonempty-URL predicate must match raw vision
caption indexing. MODIFY src/vision/index.ts syncRawBodyImageDescriptions to skip empty
URLs for both message/tool fields, preserve existing file marker when available, and
never consume a later image's caption. Remove the now-unneeded private boolean argument.
MODIFY tests/vision/vision-cache.test.ts with function/custom arrays containing empty
URLs before two real images; actual describeImagesInPlace must preserve caption order.
Standalone .tmp/vision-caption-alignment-probe.ts demonstrated the misalignment (exit1).

Accepted coverage gap: core guard test must activate vision in its ordinary-image
control. Explicit routed vision fixture, controlled description dependency, and no live
account resolution; control describes once, computer-output request describes zero.
These repairs preserve the original030scope and do not implement040or050.
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
# Orphan image carriers

Depends on030 observation/fallback contract; wp4. C3, same bounds as000.

## MODIFY src/adapters/anthropic.ts

Search found orphanToolResultText and toAnthropicContentPart as owners. Add one local
orphanToolResultContent helper returning string|unknown[]: image-free content returns
existing orphanToolResultText exactly; an
image-bearing array becomes an annotation text block followed by existing
toAnthropicContentPart mappings with empty text filtered as in toAnthropicToolResult.
Never JSON-stringify image bytes. Use helper at both sites:

```diff
- orphanBlocks.push({ type: "text", text: orphanToolResultText(tr) });
+ const orphan = orphanToolResultContent(tr);
+ orphanBlocks.push(...(typeof orphan === "string" ? [{ type: "text", text: orphan }] : orphan));
- messages.push({ role: "user", content: orphanToolResultText(msg) });
+ messages.push({ role: "user", content: orphanToolResultContent(msg) });
```

Declare orphanBlocks unknown[] to match the existing content mapper's unknown return;
do not add a cast/export. Keep valid tool_result blocks before orphan siblings.

## MODIFY src/adapters/command-code.ts

Hoist existing image extraction and wireImagePart mapping before paired/orphan split.
Append mapped images to orphan user carrier after its provenance text; leave
closePendingCalls before the carrier. Reuse mapped images in paired result buffer.
No new shared utility or change to parallel-result ordering.

## MODIFY tests/adapters/adapter-usage.test.ts and tests/providers/command-code-provider.test.ts

Extend existing orphan/paired-image tests: standalone, duplicate, unmatched adjacent,
user barrier, outstanding other call; data+HTTPS; mixed/empty text; native image blocks
and no base64 in text; no fabricated tool pairing; existing exact text-only behavior.

Main standalone adapter body probe must fail before and pass after. Workers have
disjoint adapter+test paths, no suites/services/git writes. Main owns verification,
docs note, fourth stacked PR and CI. No merge before050/060 acceptance.
Loading
Loading