test(eve): make model output image smoke deterministic by cmpadden · Pull Request #1342 · vercel/eve

cmpadden · 2026-07-29T18:59:49Z

Summary

Attempt to remove test flaky by not actually invoking image model, and instead confirming appropriate inputs.

move the exact toModelOutput file-projection contract into a deterministic unit test
keep the hosted image E2E as a smoke test for tool completion, without requiring a hosted model to classify randomized pixels

The previous E2E failed intermittently when the model omitted or confused a stripe color, even though the tool completed and the image content part was delivered. The unit test now verifies that raw tool output is given to the mapper while the model-facing result contains only the projected text and PNG file part.

Validation

pnpm --filter eve exec vitest run --config vitest.unit.config.ts src/harness/tools.test.ts
pnpm --filter eve exec tsc -p tsconfig.json --noEmit
pnpm exec oxlint packages/eve/src/harness/tools.test.ts e2e/fixtures/agent-tools/evals/static-tools/to-model-output-content-parts.eval.ts

Signed-off-by: Colton Padden <colton.padden@vercel.com>

vercel · 2026-07-29T18:59:55Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
eve-docs	Ready	Preview, Comment	Jul 29, 2026 7:01pm
eve-docs-4759	Ready	Preview, Comment, Open in v0	Jul 29, 2026 7:01pm

github-actions · 2026-07-29T19:03:14Z

Bundle + Package Summary: `apps/fixtures/weather-agent`

Key takeaways

No notable deltas vs main (5d25b16).

Delta vs `main (5d25b16)`

Area	Metric	Baseline	Current	Delta
Package	Packed tarball	7.61 MB	7.61 MB	-108 B ✅
Package	Unpacked publish size	28.75 MB	28.75 MB	0 B ➖
Package	Installed footprint	91.05 MB	91.05 MB	0 B ➖
Package	Published files	2855	2855	0
Package	Installed files	6642	6642	0
Runtime	Unique function payloads	2	2	0
Runtime	Total function bytes	16.72 MB	16.72 MB	-40 B ✅
Runtime	Public routes	11	11	0

Changed function payloads vs main (5d25b16) (2)

Function	Status	Baseline	Current	Delta	Route changes
`functions/__server.func`	changed	8.36 MB	8.36 MB	-20 B ✅	none
`functions/.well-known/workflow/v1/flow.func`	changed	8.36 MB	8.36 MB	-20 B ✅	none

`eve init` install

Metric	Baseline	Current	Delta
Installed footprint	129.45 MB	129.45 MB	0 B ➖
Installed packages	130	130	0
dependencies	4	4	0
devDependencies	2	2	0
Dependency package bytes	43.08 MB	43.08 MB	0 B ➖
devDependency package bytes	5.04 MB	5.04 MB	0 B ➖

Build Metadata

Preset: vercel
Nitro: nitro@3.0.260610-beta
Output directory: apps/fixtures/weather-agent/.vercel/output
Build metadata timestamp: 2026-07-29T19:02:26.530Z
Route aliases: 11 public, 1 internal (12 total aliases)
Vercel routes in config: 14
Severity legend: 🔴 dominant/large, 🟠 notable, 🟡 watch, ⚪ small

Package Drill-Down

Package Details

Package: eve@0.27.12
Package directory: packages/eve
Tarball: 7.61 MB (eve-0.27.12.tgz)
Unpacked payload: 28.75 MB across 2855 published files
Installed footprint: 91.05 MB across 6642 installed files
Installed root package: 27.39 MB
Installed dependencies: 63.66 MB
Runtime dependencies: 2
Peer dependencies: 5 (4 optional)

Installed footprint is measured from an isolated temporary npm install of the packed tarball.

Heavy installed dependencies

eve: 27.39 MB (30.1%)
@rolldown/binding-linux-x64-gnu: 19.28 MB (21.2%)
@rolldown/binding-wasm32-wasi: 10.66 MB (11.7%)
ai: 6.53 MB (7.2%)
@napi-rs/wasm-runtime: 6.39 MB (7.0%)

Publish payload breakdown

Published file size
🔴 dist/src/compiled/shadcn-registry/index.js       [########################] 13.15 MB 45.8%
🟠 dist/src/compiled/experimental-ai-sdk-code-mo... [###.....................] 1.51 MB 5.2%
🟡 dist/src/compiled/@vercel/sandbox/index.js       [#.......................] 632.5 kB 2.2%
🟡 dist/src/compiled/_chunks/workflow/undici-DWL... [#.......................] 502.4 kB 1.7%
🟡 dist/src/compiled/@chat-adapter/slack/index.js   [#.......................] 440.5 kB 1.5%
🔴 Other published files                            [#######################.] 12.51 MB 43.5%

Installed footprint breakdown

Installed package size
🔴 eve                             [########################] 27.39 MB 30.1%
🔴 @rolldown/binding-linux-x64-gnu [#################.......] 19.28 MB 21.2%
🔴 @rolldown/binding-wasm32-wasi   [#########...............] 10.66 MB 11.7%
🔴 ai                              [######..................] 6.53 MB 7.2%
🔴 @napi-rs/wasm-runtime           [######..................] 6.39 MB 7.0%
🔴 zod                             [####....................] 5.07 MB 5.6%
🔴 Other installed packages        [##############..........] 15.73 MB 17.3%

Runtime dependencies (2)

Package	Range	Notes
`nitro`	`3.0.260610-beta`
`undici`	`8.9.0`

Peer dependencies (5)

Package	Range	Notes
`@opentelemetry/api`	`^1.0.0`	optional peer
`ai`	`catalog:`
`braintrust`	`^3.0.0`	optional peer
`just-bash`	`^3.0.0`	optional peer
`microsandbox`	`^0.5.0`	optional peer

eve init install drill-down

`eve init` install details

Command: eve init my-agent
Package manager: npm
Installed footprint: 129.45 MB across 8510 installed files
Installed packages: 130 total (124 transitive-only)
dependencies: 4 direct packages totaling 43.08 MB
devDependencies: 2 direct packages totaling 5.04 MB
Other transitive package files: 81.33 MB

Installed footprint is measured from an isolated temporary eve init my-agent using the current packed eve tarball.

Heavy installed dependencies

@typescript/typescript-linux-x64: 27.95 MB (21.6%)
eve: 27.39 MB (21.2%)
@rolldown/binding-linux-x64-gnu: 19.28 MB (14.9%)
@rolldown/binding-wasm32-wasi: 10.66 MB (8.2%)
zod: 9.02 MB (7.0%)

Installed footprint breakdown

Installed package size
🔴 @typescript/typescript-linux-x64 [########################] 27.95 MB 21.6%
🔴 eve                              [########################] 27.39 MB 21.2%
🔴 @rolldown/binding-linux-x64-gnu  [#################.......] 19.28 MB 14.9%
🔴 @rolldown/binding-wasm32-wasi    [#########...............] 10.66 MB 8.2%
🔴 zod                              [########................] 9.02 MB 7.0%
🔴 ai                               [######..................] 6.53 MB 5.0%
🔴 @napi-rs/wasm-runtime            [#####...................] 6.39 MB 4.9%
🔴 Other installed packages         [###################.....] 22.23 MB 17.2%

dependencies (4)

Package	Range	Installed size	Share
`@vercel/connect`	`0.4.2`	135.8 kB	0.1%
`ai`	`^7.0.38`	6.53 MB	5.0%
`eve`	`file:eve-0.27.12.tgz`	27.39 MB	21.2%
`zod`	`4.4.3`	9.02 MB	7.0%

devDependencies (2)

Package	Range	Installed size	Share
`@types/node`	`24.x`	2.54 MB	2.0%
`typescript`	`7.0.2`	2.50 MB	1.9%

Function Drill-Down

Payload Size Graph

Unique function payload size and share of total
🔴 functions/.well-known/workflow/v1/flow.func     [########################] 8.36 MB 50.0%
🔴 functions/__server.func                         [########################] 8.36 MB 50.0%

Top Function Payloads

🟠 functions/.well-known/workflow/v1/flow.func • 1 public route • 8.36 MB

Metric	Value
Public routes	`/.well-known/workflow/v1/flow`
Runtime	`nodejs24.x`
Handler	`index.mjs`
Payload	8.36 MB
Function files	8.36 MB across 44 files
Traced dependencies	0 B
Signal	🟠 Bundled file `index.mjs` is 2.27 MB (27.1%)

🟠 🔎 Dependency Analysis

📦 Bundled files:

Bundled file size
🟠 index.mjs                       [#######################.] 2.27 MB 27.1%
🟠 _chunks/runtime-artifacts.mjs   [################........] 1.59 MB 19.0%
🟡 _libs/undici.mjs                [##########..............] 980.5 kB 11.7%
🟡 _chunks/sandbox.mjs             [########................] 768.8 kB 9.2%
🟡 _libs/@ai-sdk/gateway+[...].mjs [####....................] 432.8 kB 5.2%
🟠 Other bundled files             [########################] 2.33 MB 27.8%

🧾 Vercel Config

{
  "handler": "index.mjs",
  "launcherType": "Nodejs",
  "shouldAddHelpers": false,
  "supportsResponseStreaming": true,
  "runtime": "nodejs24.x",
  "maxDuration": "max",
  "experimentalTriggers": [
    {
      "type": "queue/v2beta",
      "topic": "__eve776561746865722d6167656e74_wkf_workflow_*",
      "consumer": "default",
      "retryAfterSeconds": 5,
      "initialDelaySeconds": 0
    }
  ],
  "environment": {
    "WORKFLOW_PRECONDITION_GUARD": "1"
  }
}

🟠 functions/__server.func • 10 public routes, 1 internal alias • 8.36 MB

Metric	Value
Public routes	`/` `/eve/v1/callback/[token]` `/eve/v1/connections/[name]/callback/[token]` `/eve/v1/health` `/eve/v1/info` `/eve/v1/session` `/eve/v1/session/[sessionId]` `/eve/v1/session/[sessionId]/cancel` `/eve/v1/session/[sessionId]/stream` `/eve/v1/session/reset`
Internal aliases	`/__server`
Runtime	`nodejs24.x`
Handler	`index.mjs`
Payload	8.36 MB
Function files	8.36 MB across 44 files
Traced dependencies	0 B
Signal	🟠 Bundled file `index.mjs` is 2.27 MB (27.1%)

🟠 🔎 Dependency Analysis

📦 Bundled files:

Bundled file size
🟠 index.mjs                       [#######################.] 2.27 MB 27.1%
🟠 _chunks/runtime-artifacts.mjs   [################........] 1.59 MB 19.0%
🟡 _libs/undici.mjs                [##########..............] 980.5 kB 11.7%
🟡 _chunks/sandbox.mjs             [########................] 768.8 kB 9.2%
🟡 _libs/@ai-sdk/gateway+[...].mjs [####....................] 432.8 kB 5.2%
🟠 Other bundled files             [########################] 2.32 MB 27.8%

🧾 Vercel Config

{
  "handler": "index.mjs",
  "launcherType": "Nodejs",
  "shouldAddHelpers": false,
  "supportsResponseStreaming": true,
  "runtime": "nodejs24.x"
}

Build Timing: `e2e/fixtures/agent-tools-sandbox`

This is an informational timing measurement inside eve build, from preflight through publication. Output-size measurement and profile writing are excluded.

Build mode: deployable Vercel build with sandbox template prewarm included.

Build pipeline: 2.17 s -> 2.27 s (+102.0 ms) vs main (5d25b16).
Timing is informational: shared GitHub runners are too variable for a hard timing budget.

Detailed phase timings vs `main (5d25b16)`

Phase	Baseline	Current	Delta
`extension.check`	1.1 ms	9.8 ms	+8.7 ms
`project.resolve`	0.6 ms	3.5 ms	+2.9 ms
`workspace.create`	0.8 ms	1.1 ms	+0.3 ms
`host.prepare`	419.2 ms	204.5 ms	-214.7 ms
`vercel.service-prefix.resolve`	2.1 ms	1.8 ms	-0.3 ms
`nitro.create`	202.8 ms	177.1 ms	-25.7 ms
`sandbox.prewarm`	279.3 ms	611.7 ms	+332.4 ms
`nitro.cache.prepare`	0.3 ms	0.3 ms	0.0 ms
`nitro.prepare`	0.8 ms	0.8 ms	0.0 ms
`nitro.public-assets`	0.8 ms	0.7 ms	-0.1 ms
`nitro.prerender`	0.4 ms	0.5 ms	+0.1 ms
`nitro.bundle`	1.23 s	1.23 s	-2.4 ms
`nitro.cache.write`	0.3 ms	0.3 ms	0.0 ms
`vercel.workflow-function.materialize`	21.8 ms	22.5 ms	+0.7 ms
`agent-summary.emit`	0.5 ms	0.5 ms	0.0 ms
`nitro.close`	0.1 ms	0.1 ms	0.0 ms
`output.publish`	3.3 ms	3.3 ms	0.0 ms
`workspace.remove`	1.9 ms	2.0 ms	+0.1 ms

ruiconti · 2026-07-29T20:12:09Z

-    t.eventsSatisfy("a reply names the rendered colors in order", (events) => {
-      const answer = assistantAnswers(events)[0];
-      return answer !== undefined && namesColorsInOrder(events, answer);
-    });
-
-    // The content part is baked into persisted history, so a follow-up turn
-    // must answer from replay without re-running the tool.
-    await t.send(
-      "Without calling any tool, repeat the stripe colors left to right, comma-separated.",
-    );
-
-    t.succeeded();
-    t.calledTool(TOOL_NAME, { count: 1 });
-    t.eventsSatisfy("the replayed image still answers the follow-up", (events) => {
-      const answer = assistantAnswers(events).at(-1);
-      return answer !== undefined && namesColorsInOrder(events, answer);
-    });


Removing this means that we do not have a stable signal that the model has seen it in context window. For instance, this test also ensures that AI SDK is doing the right thing.

But adding a model call may not be needed - I could see an equally expressible test being done strictly by looking at the context window before calling the model.

test(eve): make model output image smoke deterministic

51d2835

Signed-off-by: Colton Padden <colton.padden@vercel.com>

vercel Bot deployed to Preview – eve-docs-4759 July 29, 2026 19:01 View deployment

vercel Bot deployed to Preview – eve-docs July 29, 2026 19:01 View deployment

cmpadden requested review from prha and ruiconti July 29, 2026 19:06

ruiconti reviewed Jul 29, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

test(eve): make model output image smoke deterministic - #1342

test(eve): make model output image smoke deterministic#1342
cmpadden wants to merge 1 commit into
mainfrom
fix/model-output-vision-eval-flake

cmpadden commented Jul 29, 2026 •

edited

Loading

Uh oh!

vercel Bot commented Jul 29, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 29, 2026

Package Details

`eve init` install details

Payload Size Graph

Top Function Payloads

Uh oh!

ruiconti Jul 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

cmpadden commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Validation

Uh oh!

vercel Bot commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 29, 2026

Bundle + Package Summary: apps/fixtures/weather-agent

Delta vs main (5d25b16)

eve init install

Package Details

eve init install details

Payload Size Graph

Top Function Payloads

Build Timing: e2e/fixtures/agent-tools-sandbox

Uh oh!

ruiconti Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

cmpadden commented Jul 29, 2026 •

edited

Loading

vercel Bot commented Jul 29, 2026 •

edited

Loading

Bundle + Package Summary: `apps/fixtures/weather-agent`

Delta vs `main (5d25b16)`

`eve init` install

`eve init` install details

Build Timing: `e2e/fixtures/agent-tools-sandbox`