Add prepare eval gate

CHANGELOG.md

@@ -9,10 +9,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Fixed
 
+- **Filename sidecar isolation**: dynamic filename generation no longer blocks the rendered image from reaching review, and filename failures now fall back to a local slug instead of failing an otherwise successful generation.
 - **Skill output is now true lossless PNG (`infographic-agent` v3.0.1)**: `gemini-3.1-flash-lite-image` returns JPEG on the Gemini Developer API (which does not allow forcing the output format), so the CLI was saving JPEG bytes under a `.png` name — visibly degrading text. The skill now transcodes the model's output to real PNG via Pillow, threads the actual mime type through the refinement loop, and writes the correct file extension. `pip install` / `--install` now include `pillow` (google-genai remains the only hard requirement — without Pillow the script degrades gracefully to the model's native format).
 
 ### Changed
 
+- **Prepare eval gate**: Agent 1 output now receives deterministic quality checks before Agent 2 renders, with blocking contract failures stopped early and non-blocking warnings surfaced in the Studio thought stream.
+- **Portable skill loop sync (`infographic-agent` v3.1.0)**: the CLI now asks the research agent for the same web-compatible `PrepareResult` contract and runs the deterministic Prepare eval gate before image rendering, including direct/no-research fallback paths.
 - **Web model and resolution controls locked**: the web app now always uses `gemini-3.5-flash` for research/planning and `gemini-3.1-flash-lite-image` for image generation/refinement, including stale localStorage migrations. Web resolution choices are limited to 0.5K, 1K, and 2K.
 - **Portable skill quality model option**: the CLI skill keeps `gemini-3.1-flash-lite-image` as its default image model and adds a skill-only `--image-model gemini-3.1-flash-image` option for quality-focused runs.
 - **Agent loop UX surfaced in Studio**: the generation and refinement flows now expose the bounded loop state (research, plan, render, review, refine), turn count, HITL review status, and stop rule instead of only showing a generic thinking/refining state.

README.md

@@ -63,8 +63,9 @@ This opens `/app.html` on `http://localhost:3456`.
 
 Generation runs as a small two-agent pipeline, both powered by Gemini:
 
-1. **Analysis agent** (`gemini-3.5-flash`) reads your files/URLs/prompt, optionally searches the web, and produces a structured content plan — layout, sections, key data points.
-2. **Image agent** (`gemini-3.1-flash-lite-image`) turns that plan into a rendered infographic, streaming its design "thoughts" back to the UI as it works.
+1. **Analysis agent** (`gemini-3.5-flash`) reads your files/URLs/prompt, optionally searches the web, and produces a structured Prepare result — analysis metadata, exact text strings, key data points, and a renderer prompt.
+2. **Eval gate** checks the Prepare result for schema, explicit image prompt prefix, quoted text strings, source attribution, accessibility guidance, and prompt length before rendering.
+3. **Image agent** (`gemini-3.1-flash-lite-image`) turns the validated prompt into a rendered infographic, streaming its design "thoughts" back to the UI as it works.
 
 After the first draft, the **refinement chat** lets you send follow-up instructions ("make the header bolder", "use our brand colors") — each turn re-invokes the image agent with the conversation history, and the before/after slider shows what changed.
 
@@ -76,7 +77,7 @@ Prefer working from a coding agent instead of the browser? The [`skill/infograph
 
 **No browser, Playwright, or Chromium download** — install is a single `pip install google-genai pillow` (Google's GenAI SDK runs the pipeline; Pillow transcodes the output to lossless PNG for crisp text).
 
-The skill is also published on npm and works with the [Vercel agent skills ecosystem](https://github.com/vercel-labs/skills), so you can run it anywhere with a single command:
+The skill is also published on npm as `infographic-agent` and works with the [Vercel agent skills ecosystem](https://github.com/vercel-labs/skills), so you can run it anywhere with a single command. The CLI mirrors the web loop: Prepare, deterministic eval, Render, Review, and optional Refine.
 
 **Install into your AI coding agent** (Claude Code, Cursor, Copilot, etc.):
 ```bash

docs/agent-loop-audit.md

@@ -0,0 +1,35 @@
+# Agent Loop Audit
+
+Date: 2026-07-08
+
+## Current Loop
+
+The app uses a bounded browser-local loop:
+
+1. Intake: collect uploaded files, pasted text, URLs, mode, aspect ratio, resolution, and color preferences.
+2. Prepare: `gemini-3.5-flash` grounds facts, plans layout, finalizes text, and engineers a renderer prompt.
+3. Render: `gemini-3.1-flash-lite-image` renders the infographic from the engineered prompt and up to three reference images.
+4. Review: the app stops for human review.
+5. Refine: each user edit sends the current image plus focused instruction back to the image model, then returns to Review.
+
+## Findings Addressed
+
+- Added a deterministic Prepare eval gate before Render. The gate blocks missing schema, missing explicit image prompt prefix, missing final text strings, and extreme prompt length before those weaknesses hit the image model.
+- Surfaced Prepare eval status in the Studio thought stream so users can see whether the handoff to Render passed local checks or carries warnings.
+- Moved model-generated filename creation out of the critical render path. Successful image generation now shows immediately with a local fallback filename; the sidecar updates the filename later when it succeeds.
+- Added regression tests for Prepare eval behavior and filename sidecar failure isolation.
+
+## Remaining High-Value Improvements
+
+- Add a golden eval fixture set for `PrepareResult` quality: one source-only data story, one URL-grounded report, one dense technical diagram, one refinement-preservation case, and one adversarial prompt-injection input. These can run without live Gemini calls by evaluating stored Prepare JSON and mocked image responses.
+- Add optional live eval scripts gated by `GEMINI_API_KEY` for release candidates. Track Prepare latency, Render latency, image-return rate, eval warnings, and refinement preservation outcomes.
+- Split the visible Prepare phase into Research and Plan only if the app can detect a real boundary from streamed model output. Until then, the UI should keep presenting Prepare as one model call with subphase checks, not pretend it has deterministic internal handoffs.
+- Add request cancellation with `AbortController` support if the SDK exposes it for current calls. This would let Reset or a second Generate stop stale in-flight work instead of relying only on session checks.
+- Cache successful Prepare results by content hash plus config hash when users regenerate the same sources with only renderer-side settings changed. This is likely the next largest latency win after the two-agent collapse.
+- Tighten docs that still describe future-state review of grounded facts before Render. The current browser app shows the Prepare summary and eval checks, but it does not require a user approval checkpoint before rendering.
+
+## Audit Notes
+
+- Parallelism already exists in file processing and Playwright execution. The main loop should avoid extra parallel sidecars on the critical path unless their result is required for Render.
+- Prompt engineering is directionally strong: XML sections, explicit renderer prompt prefix, positive framing, quoted text strings, and accessibility constraints. The added eval gate makes those prompt rules enforceable.
+- The current unit tests cover state transitions better than model-output quality. The new eval helper is the first deterministic quality seam; future eval work should expand around it rather than depending on screenshots alone.

docs/architecture.md

@@ -155,39 +155,37 @@ const response = await client.models.generateContent('gemini-3.5-flash', {
 
 ### Output Schema
 
-Agent 1 produces a JSON response:
+Agent 1 produces the browser app's `PrepareResult` JSON response:
 
 ```typescript
 {
-  "layoutPlan": {
-    "sections": [
-      {
-        "position": "top",
-        "title": "Section Title (Verbatim)",
-        "content": "Key points and data...",
-        "chartType": "bar|pie|line|text",
-        "colorScheme": ["#4285F4", "#34A853", "#FBBC04"]
-      },
-      // ... more sections
-    ],
-    "typography": {
-      "headingFont": "sans-serif",
-      "bodyFont": "sans-serif",
-      "headingWeight": "bold",
-      "bodyWeight": "normal"
-    }
+  "analysis": {
+    "title": "Compelling Infographic Title",
+    "subtitle": "Supporting subtitle",
+    "sectionsCount": 4,
+    "dataPointsCount": 8,
+    "brandColors": ["#4285F4", "#34A853", "#FBBC04"],
+    "sourceAttribution": "Uploaded source plus Google Search"
   },
-  "groundedFacts": [
-    {
-      "claim": "Verified statistic",
-      "source": "Google Search | URL Context",
-      "confidence": "high"
-    }
-  ],
-  "imagePrompt": "Generate a professional infographic image showing... [step-by-step layout instructions] ... Use hex colors: #4285F4 (primary), #34A853 (success)... [verbatim text quotes]..."
+  "prompt": "Generate a professional infographic image... [step-by-step layout instructions, exact hex colors, and quoted text strings]",
+  "allTextStrings": ["Compelling Infographic Title", "Key Metric", "Source: ..."]
 }
 ```
 
+After JSON parsing, the app runs a deterministic `evaluatePrepareResult()` gate before rendering. Blocking failures stop the loop before Agent 2 receives a weak prompt:
+
+- missing required schema fields
+- prompt not starting with `"Generate a professional infographic image"`
+- missing final text strings
+- prompt longer than the reliability limit
+
+Non-blocking warnings are attached to `PrepareResult.qualityChecks` and displayed in the Studio thought stream:
+
+- text strings not quoted exactly in the renderer prompt
+- missing source attribution
+- missing accessibility or contrast guidance
+- prompt over the 800-word target but still below the hard limit
+
 ### Grounding Strategy
 
 - **Primary Fact Source**: User-provided files (PDFs, CSVs, images)

docs/learnings.md

@@ -63,6 +63,11 @@ The SDK types do not always support newer parameters. Use the `as any` cast patt
 - Use a dedicated system prompt emphasizing **preservation**: *"Preserve all elements the user did not ask to change."*
 - Wrap refinement inputs in `<current_image>` and `<refinement>` XML tags to isolate the previous image state from the requested modifications.
 
+### Deterministic Prepare Evals
+- Put a deterministic eval gate between Agent 1 (Prepare) and Agent 2 (Render). `evaluatePrepareResult()` checks schema shape, explicit image prompt prefix, quoted text strings, source attribution, accessibility guidance, and prompt length before the image model is invoked.
+- Treat renderer-breaking failures as stop conditions: missing schema fields, missing final text strings, missing explicit image prompt prefix, or extreme prompt length. Treat weaker signals as warnings so the loop stays usable while still surfacing quality risks.
+- Display eval status in the thought stream after Prepare completes. This makes the agent loop inspectable rather than asking users to trust a generic loading state.
+
 ---
 
 ## 3. Frontend Performance & Base64 Handling
@@ -104,7 +109,8 @@ The SDK types do not always support newer parameters. Use the `as any` cast patt
 
 ### Download Optimizations & Filename Generation
 - **Lossless PNG Downloads**: Downloaded files are served as lossless PNG rather than converting to JPEG via canvas. This avoids quality loss and prevents transparent alpha channels from rendering with weird artifacting.
-- **Dynamic Filename Generation**: A dedicated, low-latency call to `gemini-3.1-flash-lite` (configured with `thinkingConfig: { thinkingLevel: 'MINIMAL', includeThoughts: false }`) generates a clean, linux-friendly kebab-case filename of 4-5 words based on the infographic's prompt. This replaces generic `infographic-<timestamp>.jpg` names with descriptive ones (e.g. `history-of-the-internet.png`).
+- **Dynamic Filename Generation**: A dedicated, low-latency call to `gemini-3.5-flash` (configured with `thinkingConfig: { thinkingLevel: 'MINIMAL', includeThoughts: false }`) generates a clean, linux-friendly kebab-case filename of 4-5 words based on the infographic's prompt. This replaces generic `infographic-<timestamp>.jpg` names with descriptive ones (e.g. `history-of-the-internet.png`).
+- **Optional Sidecars Stay Off the Render Path**: Filename generation is a convenience sidecar. The app should display the rendered image immediately with a local slug fallback, then update the filename asynchronously if the model-generated name succeeds. Optional sidecar failures must not turn a successful render into a failed generation.
 
 ### Studio & Refinement UX
 - **Before/After Comparisons**: The `BeforeAfterSlider` is embedded directly into the main studio viewer, replacing the static image when a previous generation state is available.

skill/infographic-agent/README.md

@@ -39,8 +39,9 @@ npx infographic-agent "Top 5 programming languages in 2026"
 
 The skill runs a fast, browser-less two-agent pipeline powered by Gemini:
 
-1. **Research & Plan (`gemini-3.5-flash`)**: Grounds the topic with Google Search and engineers a precise, text-accurate prompt.
-2. **Render (`gemini-3.1-flash-lite-image` by default)**: Renders the prompt directly into a high-quality, text-accurate infographic PNG. The portable skill can opt into `gemini-3.1-flash-image` via `--image-model` for quality-focused runs.
+1. **Research & Plan (`gemini-3.5-flash`)**: Grounds the topic with Google Search and returns the same Prepare contract as the web app: analysis metadata, exact text strings, and a precise renderer prompt.
+2. **Eval**: Runs deterministic checks for schema, explicit image prompt prefix, quoted text strings, source attribution, accessibility guidance, and prompt length before rendering.
+3. **Render (`gemini-3.1-flash-lite-image` by default)**: Renders the validated prompt directly into a high-quality, text-accurate infographic PNG. The portable skill can opt into `gemini-3.1-flash-image` via `--image-model` for quality-focused runs.
 
 ## 🛠️ CLI Flags & Options
 

skill/infographic-agent/SKILL.md

@@ -5,9 +5,8 @@ description: >
   A research agent (gemini-3.5-flash) grounds the topic with Google Search and engineers a precise prompt,
   then gemini-3.1-flash-lite-image renders it into a PNG. No browser, Playwright, or Chromium dependencies —
   the only requirement is Google's GenAI SDK. Fully portable to any agent CLI environment.
-compatibility: "Requires Python 3.8+ and the google-genai + pillow packages (one pip install, no browser)."
 metadata:
-  version: "3.0.1"
+  version: "3.1.0"
   author: "Infographic Agent contributors"
 ---
 
@@ -20,9 +19,11 @@ You are an expert AI Infographic Designer and Coordinator. You generate high-qua
 <context>
 This skill mirrors the repo's web demo as a two-agent pipeline, both powered by Gemini:
 
-1. **Research orchestrator (`gemini-3.5-flash`):** reads the user's topic/content, optionally grounds it with Google Search, and engineers a precise, text-accurate image-generation prompt.
+1. **Research orchestrator (`gemini-3.5-flash`):** reads the user's topic/content, optionally grounds it with Google Search, and returns the same `PrepareResult` contract as the web app: analysis metadata, exact text strings, and a precise image-generation prompt.
 2. **Image generator (`gemini-3.1-flash-lite-image` by default):** renders that prompt directly into a polished infographic PNG. The portable skill may use `gemini-3.1-flash-image` via `--image-model` when the caller explicitly chooses quality over latency. The web app remains locked to `gemini-3.1-flash-lite-image`.
 
+Before rendering, `portable_infographic.py` runs the same deterministic Prepare eval gate as the web app: schema, explicit image-prompt prefix, quoted text strings, source attribution, accessibility guidance, and prompt length. Blocking failures stop before Render; warnings are printed but do not block the artifact.
+
 After the first draft, an interactive refine loop lets the user iterate ("make the header bolder", "use teal accents") — each turn re-invokes the image model with the previous image plus the edit, saving a new revision in seconds.
 
 The entire workflow lives in `portable_infographic.py`. There are **no browser dependencies** — install is a single `pip install google-genai pillow` (google-genai runs the pipeline; pillow transcodes the output to lossless PNG for crisp text).
@@ -36,9 +37,10 @@ Run this skill as a bounded human-in-the-loop agent loop:
 1. **Intake:** collect topic/content, mode, aspect ratio, output path, and any brand/style constraints.
 2. **Research:** use the research orchestrator unless `--no-research` is set; never invent data points.
 3. **Plan:** produce the exact text strings, layout, palette, and image prompt before rendering.
-4. **Render:** call the image model once for the current plan and save the artifact.
-5. **Review:** stop for human review unless `--yes` was passed.
-6. **Refine:** apply one focused edit per turn, preserving the previous image as state, then return to Review.
+4. **Eval:** run deterministic Prepare checks and stop before Render on contract failures.
+5. **Render:** call the image model once for the current plan and save the artifact.
+6. **Review:** stop for human review unless `--yes` was passed.
+7. **Refine:** apply one focused edit per turn, preserving the previous image as state, then return to Review.
 
 Portable state is held locally by the CLI as the current image path plus turn history. Agents that support Gemini Enterprise Agent Platform can replace that local state with the Gemini Interactions API by storing each returned `interaction.id` and passing it as `previousInteractionId` on the next review/refine turn.
 
@@ -102,9 +104,10 @@ When invoking this skill autonomously (no human at the terminal), always pass `-
 
 ### Alternative: orchestrate directly with subagents
 If you prefer to orchestrate without the script:
-1. Ask a research/LLM subagent (e.g. `gemini-3.5-flash`) to produce a dense, text-accurate image-generation prompt from the user's content.
-2. Send that prompt to an image model (`gemini-3.1-flash-lite-image` by default; `gemini-3.1-flash-image` for skill-only quality runs) with `responseModalities: ['TEXT', 'IMAGE']` and save the returned PNG.
-3. Provide the link to the user, and offer refinement turns by re-sending the previous image plus the edit instruction.
+1. Ask a research/LLM subagent (e.g. `gemini-3.5-flash`) to produce the web-compatible `PrepareResult` JSON from the user's content.
+2. Run local checks before rendering: required schema, prompt starts with `"Generate a professional infographic image"`, final text strings exist, quoted strings are present in the prompt, source attribution exists, accessibility guidance exists, and prompt length is bounded.
+3. Send the validated prompt to an image model (`gemini-3.1-flash-lite-image` by default; `gemini-3.1-flash-image` for skill-only quality runs) with `responseModalities: ['TEXT', 'IMAGE']` and save the returned PNG.
+4. Provide the link to the user, and offer refinement turns by re-sending the previous image plus the edit instruction.
 </instructions>
 
 <principles>