diff --git a/CHANGELOG.md b/CHANGELOG.md index a193b24..6f771e3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,10 +9,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed +- **Filename sidecar isolation**: dynamic filename generation no longer blocks the rendered image from reaching review, and filename failures now fall back to a local slug instead of failing an otherwise successful generation. - **Skill output is now true lossless PNG (`infographic-agent` v3.0.1)**: `gemini-3.1-flash-lite-image` returns JPEG on the Gemini Developer API (which does not allow forcing the output format), so the CLI was saving JPEG bytes under a `.png` name — visibly degrading text. The skill now transcodes the model's output to real PNG via Pillow, threads the actual mime type through the refinement loop, and writes the correct file extension. `pip install` / `--install` now include `pillow` (google-genai remains the only hard requirement — without Pillow the script degrades gracefully to the model's native format). ### Changed +- **Prepare eval gate**: Agent 1 output now receives deterministic quality checks before Agent 2 renders, with blocking contract failures stopped early and non-blocking warnings surfaced in the Studio thought stream. +- **Portable skill loop sync (`infographic-agent` v3.1.0)**: the CLI now asks the research agent for the same web-compatible `PrepareResult` contract and runs the deterministic Prepare eval gate before image rendering, including direct/no-research fallback paths. - **Web model and resolution controls locked**: the web app now always uses `gemini-3.5-flash` for research/planning and `gemini-3.1-flash-lite-image` for image generation/refinement, including stale localStorage migrations. Web resolution choices are limited to 0.5K, 1K, and 2K. - **Portable skill quality model option**: the CLI skill keeps `gemini-3.1-flash-lite-image` as its default image model and adds a skill-only `--image-model gemini-3.1-flash-image` option for quality-focused runs. - **Agent loop UX surfaced in Studio**: the generation and refinement flows now expose the bounded loop state (research, plan, render, review, refine), turn count, HITL review status, and stop rule instead of only showing a generic thinking/refining state. diff --git a/README.md b/README.md index 5de52da..af387f2 100644 --- a/README.md +++ b/README.md @@ -63,8 +63,9 @@ This opens `/app.html` on `http://localhost:3456`. Generation runs as a small two-agent pipeline, both powered by Gemini: -1. **Analysis agent** (`gemini-3.5-flash`) reads your files/URLs/prompt, optionally searches the web, and produces a structured content plan — layout, sections, key data points. -2. **Image agent** (`gemini-3.1-flash-lite-image`) turns that plan into a rendered infographic, streaming its design "thoughts" back to the UI as it works. +1. **Analysis agent** (`gemini-3.5-flash`) reads your files/URLs/prompt, optionally searches the web, and produces a structured Prepare result — analysis metadata, exact text strings, key data points, and a renderer prompt. +2. **Eval gate** checks the Prepare result for schema, explicit image prompt prefix, quoted text strings, source attribution, accessibility guidance, and prompt length before rendering. +3. **Image agent** (`gemini-3.1-flash-lite-image`) turns the validated prompt into a rendered infographic, streaming its design "thoughts" back to the UI as it works. After the first draft, the **refinement chat** lets you send follow-up instructions ("make the header bolder", "use our brand colors") — each turn re-invokes the image agent with the conversation history, and the before/after slider shows what changed. @@ -76,7 +77,7 @@ Prefer working from a coding agent instead of the browser? The [`skill/infograph **No browser, Playwright, or Chromium download** — install is a single `pip install google-genai pillow` (Google's GenAI SDK runs the pipeline; Pillow transcodes the output to lossless PNG for crisp text). -The skill is also published on npm and works with the [Vercel agent skills ecosystem](https://github.com/vercel-labs/skills), so you can run it anywhere with a single command: +The skill is also published on npm as `infographic-agent` and works with the [Vercel agent skills ecosystem](https://github.com/vercel-labs/skills), so you can run it anywhere with a single command. The CLI mirrors the web loop: Prepare, deterministic eval, Render, Review, and optional Refine. **Install into your AI coding agent** (Claude Code, Cursor, Copilot, etc.): ```bash diff --git a/docs/agent-loop-audit.md b/docs/agent-loop-audit.md new file mode 100644 index 0000000..eb3eb5a --- /dev/null +++ b/docs/agent-loop-audit.md @@ -0,0 +1,35 @@ +# Agent Loop Audit + +Date: 2026-07-08 + +## Current Loop + +The app uses a bounded browser-local loop: + +1. Intake: collect uploaded files, pasted text, URLs, mode, aspect ratio, resolution, and color preferences. +2. Prepare: `gemini-3.5-flash` grounds facts, plans layout, finalizes text, and engineers a renderer prompt. +3. Render: `gemini-3.1-flash-lite-image` renders the infographic from the engineered prompt and up to three reference images. +4. Review: the app stops for human review. +5. Refine: each user edit sends the current image plus focused instruction back to the image model, then returns to Review. + +## Findings Addressed + +- Added a deterministic Prepare eval gate before Render. The gate blocks missing schema, missing explicit image prompt prefix, missing final text strings, and extreme prompt length before those weaknesses hit the image model. +- Surfaced Prepare eval status in the Studio thought stream so users can see whether the handoff to Render passed local checks or carries warnings. +- Moved model-generated filename creation out of the critical render path. Successful image generation now shows immediately with a local fallback filename; the sidecar updates the filename later when it succeeds. +- Added regression tests for Prepare eval behavior and filename sidecar failure isolation. + +## Remaining High-Value Improvements + +- Add a golden eval fixture set for `PrepareResult` quality: one source-only data story, one URL-grounded report, one dense technical diagram, one refinement-preservation case, and one adversarial prompt-injection input. These can run without live Gemini calls by evaluating stored Prepare JSON and mocked image responses. +- Add optional live eval scripts gated by `GEMINI_API_KEY` for release candidates. Track Prepare latency, Render latency, image-return rate, eval warnings, and refinement preservation outcomes. +- Split the visible Prepare phase into Research and Plan only if the app can detect a real boundary from streamed model output. Until then, the UI should keep presenting Prepare as one model call with subphase checks, not pretend it has deterministic internal handoffs. +- Add request cancellation with `AbortController` support if the SDK exposes it for current calls. This would let Reset or a second Generate stop stale in-flight work instead of relying only on session checks. +- Cache successful Prepare results by content hash plus config hash when users regenerate the same sources with only renderer-side settings changed. This is likely the next largest latency win after the two-agent collapse. +- Tighten docs that still describe future-state review of grounded facts before Render. The current browser app shows the Prepare summary and eval checks, but it does not require a user approval checkpoint before rendering. + +## Audit Notes + +- Parallelism already exists in file processing and Playwright execution. The main loop should avoid extra parallel sidecars on the critical path unless their result is required for Render. +- Prompt engineering is directionally strong: XML sections, explicit renderer prompt prefix, positive framing, quoted text strings, and accessibility constraints. The added eval gate makes those prompt rules enforceable. +- The current unit tests cover state transitions better than model-output quality. The new eval helper is the first deterministic quality seam; future eval work should expand around it rather than depending on screenshots alone. diff --git a/docs/architecture.md b/docs/architecture.md index d2d041f..ad8e6f9 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -155,39 +155,37 @@ const response = await client.models.generateContent('gemini-3.5-flash', { ### Output Schema -Agent 1 produces a JSON response: +Agent 1 produces the browser app's `PrepareResult` JSON response: ```typescript { - "layoutPlan": { - "sections": [ - { - "position": "top", - "title": "Section Title (Verbatim)", - "content": "Key points and data...", - "chartType": "bar|pie|line|text", - "colorScheme": ["#4285F4", "#34A853", "#FBBC04"] - }, - // ... more sections - ], - "typography": { - "headingFont": "sans-serif", - "bodyFont": "sans-serif", - "headingWeight": "bold", - "bodyWeight": "normal" - } + "analysis": { + "title": "Compelling Infographic Title", + "subtitle": "Supporting subtitle", + "sectionsCount": 4, + "dataPointsCount": 8, + "brandColors": ["#4285F4", "#34A853", "#FBBC04"], + "sourceAttribution": "Uploaded source plus Google Search" }, - "groundedFacts": [ - { - "claim": "Verified statistic", - "source": "Google Search | URL Context", - "confidence": "high" - } - ], - "imagePrompt": "Generate a professional infographic image showing... [step-by-step layout instructions] ... Use hex colors: #4285F4 (primary), #34A853 (success)... [verbatim text quotes]..." + "prompt": "Generate a professional infographic image... [step-by-step layout instructions, exact hex colors, and quoted text strings]", + "allTextStrings": ["Compelling Infographic Title", "Key Metric", "Source: ..."] } ``` +After JSON parsing, the app runs a deterministic `evaluatePrepareResult()` gate before rendering. Blocking failures stop the loop before Agent 2 receives a weak prompt: + +- missing required schema fields +- prompt not starting with `"Generate a professional infographic image"` +- missing final text strings +- prompt longer than the reliability limit + +Non-blocking warnings are attached to `PrepareResult.qualityChecks` and displayed in the Studio thought stream: + +- text strings not quoted exactly in the renderer prompt +- missing source attribution +- missing accessibility or contrast guidance +- prompt over the 800-word target but still below the hard limit + ### Grounding Strategy - **Primary Fact Source**: User-provided files (PDFs, CSVs, images) diff --git a/docs/learnings.md b/docs/learnings.md index 97538e0..29babd2 100644 --- a/docs/learnings.md +++ b/docs/learnings.md @@ -63,6 +63,11 @@ The SDK types do not always support newer parameters. Use the `as any` cast patt - Use a dedicated system prompt emphasizing **preservation**: *"Preserve all elements the user did not ask to change."* - Wrap refinement inputs in `` and `` XML tags to isolate the previous image state from the requested modifications. +### Deterministic Prepare Evals +- Put a deterministic eval gate between Agent 1 (Prepare) and Agent 2 (Render). `evaluatePrepareResult()` checks schema shape, explicit image prompt prefix, quoted text strings, source attribution, accessibility guidance, and prompt length before the image model is invoked. +- Treat renderer-breaking failures as stop conditions: missing schema fields, missing final text strings, missing explicit image prompt prefix, or extreme prompt length. Treat weaker signals as warnings so the loop stays usable while still surfacing quality risks. +- Display eval status in the thought stream after Prepare completes. This makes the agent loop inspectable rather than asking users to trust a generic loading state. + --- ## 3. Frontend Performance & Base64 Handling @@ -104,7 +109,8 @@ The SDK types do not always support newer parameters. Use the `as any` cast patt ### Download Optimizations & Filename Generation - **Lossless PNG Downloads**: Downloaded files are served as lossless PNG rather than converting to JPEG via canvas. This avoids quality loss and prevents transparent alpha channels from rendering with weird artifacting. -- **Dynamic Filename Generation**: A dedicated, low-latency call to `gemini-3.1-flash-lite` (configured with `thinkingConfig: { thinkingLevel: 'MINIMAL', includeThoughts: false }`) generates a clean, linux-friendly kebab-case filename of 4-5 words based on the infographic's prompt. This replaces generic `infographic-.jpg` names with descriptive ones (e.g. `history-of-the-internet.png`). +- **Dynamic Filename Generation**: A dedicated, low-latency call to `gemini-3.5-flash` (configured with `thinkingConfig: { thinkingLevel: 'MINIMAL', includeThoughts: false }`) generates a clean, linux-friendly kebab-case filename of 4-5 words based on the infographic's prompt. This replaces generic `infographic-.jpg` names with descriptive ones (e.g. `history-of-the-internet.png`). +- **Optional Sidecars Stay Off the Render Path**: Filename generation is a convenience sidecar. The app should display the rendered image immediately with a local slug fallback, then update the filename asynchronously if the model-generated name succeeds. Optional sidecar failures must not turn a successful render into a failed generation. ### Studio & Refinement UX - **Before/After Comparisons**: The `BeforeAfterSlider` is embedded directly into the main studio viewer, replacing the static image when a previous generation state is available. diff --git a/skill/infographic-agent/README.md b/skill/infographic-agent/README.md index a44038b..2c1167c 100644 --- a/skill/infographic-agent/README.md +++ b/skill/infographic-agent/README.md @@ -39,8 +39,9 @@ npx infographic-agent "Top 5 programming languages in 2026" The skill runs a fast, browser-less two-agent pipeline powered by Gemini: -1. **Research & Plan (`gemini-3.5-flash`)**: Grounds the topic with Google Search and engineers a precise, text-accurate prompt. -2. **Render (`gemini-3.1-flash-lite-image` by default)**: Renders the prompt directly into a high-quality, text-accurate infographic PNG. The portable skill can opt into `gemini-3.1-flash-image` via `--image-model` for quality-focused runs. +1. **Research & Plan (`gemini-3.5-flash`)**: Grounds the topic with Google Search and returns the same Prepare contract as the web app: analysis metadata, exact text strings, and a precise renderer prompt. +2. **Eval**: Runs deterministic checks for schema, explicit image prompt prefix, quoted text strings, source attribution, accessibility guidance, and prompt length before rendering. +3. **Render (`gemini-3.1-flash-lite-image` by default)**: Renders the validated prompt directly into a high-quality, text-accurate infographic PNG. The portable skill can opt into `gemini-3.1-flash-image` via `--image-model` for quality-focused runs. ## 🛠️ CLI Flags & Options diff --git a/skill/infographic-agent/SKILL.md b/skill/infographic-agent/SKILL.md index 9b46192..0a98feb 100644 --- a/skill/infographic-agent/SKILL.md +++ b/skill/infographic-agent/SKILL.md @@ -5,9 +5,8 @@ description: > A research agent (gemini-3.5-flash) grounds the topic with Google Search and engineers a precise prompt, then gemini-3.1-flash-lite-image renders it into a PNG. No browser, Playwright, or Chromium dependencies — the only requirement is Google's GenAI SDK. Fully portable to any agent CLI environment. -compatibility: "Requires Python 3.8+ and the google-genai + pillow packages (one pip install, no browser)." metadata: - version: "3.0.1" + version: "3.1.0" author: "Infographic Agent contributors" --- @@ -20,9 +19,11 @@ You are an expert AI Infographic Designer and Coordinator. You generate high-qua This skill mirrors the repo's web demo as a two-agent pipeline, both powered by Gemini: -1. **Research orchestrator (`gemini-3.5-flash`):** reads the user's topic/content, optionally grounds it with Google Search, and engineers a precise, text-accurate image-generation prompt. +1. **Research orchestrator (`gemini-3.5-flash`):** reads the user's topic/content, optionally grounds it with Google Search, and returns the same `PrepareResult` contract as the web app: analysis metadata, exact text strings, and a precise image-generation prompt. 2. **Image generator (`gemini-3.1-flash-lite-image` by default):** renders that prompt directly into a polished infographic PNG. The portable skill may use `gemini-3.1-flash-image` via `--image-model` when the caller explicitly chooses quality over latency. The web app remains locked to `gemini-3.1-flash-lite-image`. +Before rendering, `portable_infographic.py` runs the same deterministic Prepare eval gate as the web app: schema, explicit image-prompt prefix, quoted text strings, source attribution, accessibility guidance, and prompt length. Blocking failures stop before Render; warnings are printed but do not block the artifact. + After the first draft, an interactive refine loop lets the user iterate ("make the header bolder", "use teal accents") — each turn re-invokes the image model with the previous image plus the edit, saving a new revision in seconds. The entire workflow lives in `portable_infographic.py`. There are **no browser dependencies** — install is a single `pip install google-genai pillow` (google-genai runs the pipeline; pillow transcodes the output to lossless PNG for crisp text). @@ -36,9 +37,10 @@ Run this skill as a bounded human-in-the-loop agent loop: 1. **Intake:** collect topic/content, mode, aspect ratio, output path, and any brand/style constraints. 2. **Research:** use the research orchestrator unless `--no-research` is set; never invent data points. 3. **Plan:** produce the exact text strings, layout, palette, and image prompt before rendering. -4. **Render:** call the image model once for the current plan and save the artifact. -5. **Review:** stop for human review unless `--yes` was passed. -6. **Refine:** apply one focused edit per turn, preserving the previous image as state, then return to Review. +4. **Eval:** run deterministic Prepare checks and stop before Render on contract failures. +5. **Render:** call the image model once for the current plan and save the artifact. +6. **Review:** stop for human review unless `--yes` was passed. +7. **Refine:** apply one focused edit per turn, preserving the previous image as state, then return to Review. Portable state is held locally by the CLI as the current image path plus turn history. Agents that support Gemini Enterprise Agent Platform can replace that local state with the Gemini Interactions API by storing each returned `interaction.id` and passing it as `previousInteractionId` on the next review/refine turn. @@ -102,9 +104,10 @@ When invoking this skill autonomously (no human at the terminal), always pass `- ### Alternative: orchestrate directly with subagents If you prefer to orchestrate without the script: -1. Ask a research/LLM subagent (e.g. `gemini-3.5-flash`) to produce a dense, text-accurate image-generation prompt from the user's content. -2. Send that prompt to an image model (`gemini-3.1-flash-lite-image` by default; `gemini-3.1-flash-image` for skill-only quality runs) with `responseModalities: ['TEXT', 'IMAGE']` and save the returned PNG. -3. Provide the link to the user, and offer refinement turns by re-sending the previous image plus the edit instruction. +1. Ask a research/LLM subagent (e.g. `gemini-3.5-flash`) to produce the web-compatible `PrepareResult` JSON from the user's content. +2. Run local checks before rendering: required schema, prompt starts with `"Generate a professional infographic image"`, final text strings exist, quoted strings are present in the prompt, source attribution exists, accessibility guidance exists, and prompt length is bounded. +3. Send the validated prompt to an image model (`gemini-3.1-flash-lite-image` by default; `gemini-3.1-flash-image` for skill-only quality runs) with `responseModalities: ['TEXT', 'IMAGE']` and save the returned PNG. +4. Provide the link to the user, and offer refinement turns by re-sending the previous image plus the edit instruction. diff --git a/skill/infographic-agent/bin/infographic-agent.js b/skill/infographic-agent/bin/infographic-agent.js index c7d9b7e..95bf0ce 100755 --- a/skill/infographic-agent/bin/infographic-agent.js +++ b/skill/infographic-agent/bin/infographic-agent.js @@ -14,7 +14,7 @@ * npx infographic-agent "Top 5 programming languages in 2026" * * First-time setup: - * npx infographic-agent --install # pip install google-genai + * npx infographic-agent --install # pip install google-genai pillow * # then just run it — the CLI walks you through getting a free API key. */ @@ -83,9 +83,9 @@ if (showHelp) { print(` infographic-agent v${pkg.version} -Generate a professional infographic PNG directly with Gemini — a research -agent (gemini-3.5-flash) grounds your topic, then the image model -(gemini-3.1-flash-lite-image) renders it. No browser or Playwright needed. +Generate a professional infographic PNG directly with Gemini: a research +agent (gemini-3.5-flash) prepares and validates the prompt, then the image +model (gemini-3.1-flash-lite-image) renders it. No browser or Playwright needed. Usage: npx infographic-agent "" [options] @@ -101,7 +101,7 @@ Options: --no-open Do not auto-open the result --yes, -y Non-interactive: generate once and exit (no refine loop) --setup (Re)configure your free Gemini API key and exit - --install Install the Python dependency (google-genai) + --install Install Python dependencies (google-genai, pillow) --help, -h Show this help message Getting started (a free key takes ~20 seconds): @@ -128,7 +128,7 @@ if (!python) { process.exit(1); } -// ─── --install: first-time dependency setup (just the GenAI SDK) ───────────── +// ─── --install: first-time dependency setup ───────────────────────────────── if (doInstall) { print("Installing Python dependencies (google-genai, pillow)..."); diff --git a/skill/infographic-agent/package.json b/skill/infographic-agent/package.json index 71d8569..067bd1a 100644 --- a/skill/infographic-agent/package.json +++ b/skill/infographic-agent/package.json @@ -1,7 +1,7 @@ { "name": "infographic-agent", - "version": "3.0.1", - "description": "Generate professional infographic PNGs from text with Gemini — a research agent (gemini-3.5-flash) grounds your topic, then gemini-3.1-flash-lite-image renders it. No browser/Playwright deps.", + "version": "3.1.0", + "description": "Generate professional infographic PNGs from text with Gemini — a research agent (gemini-3.5-flash) prepares and validates the prompt, then gemini-3.1-flash-lite-image renders it. No browser/Playwright deps.", "license": "MIT", "author": "Ryan Baumann", "repository": { diff --git a/skill/infographic-agent/portable_infographic.py b/skill/infographic-agent/portable_infographic.py index 458abe7..191651e 100755 --- a/skill/infographic-agent/portable_infographic.py +++ b/skill/infographic-agent/portable_infographic.py @@ -41,15 +41,26 @@ import time from pathlib import Path -try: - from google import genai - from google.genai import types -except ImportError: - sys.stderr.write( - "[Error] The Google GenAI SDK is not installed.\n" - " Install it with: pip install google-genai\n" - ) - sys.exit(1) +genai = None +types = None + + +def ensure_genai() -> None: + """Import the Gemini SDK only when an API call is about to run.""" + global genai, types + if genai is not None and types is not None: + return + try: + from google import genai as imported_genai + from google.genai import types as imported_types + except ImportError: + sys.stderr.write( + "[Error] The Google GenAI SDK is not installed.\n" + " Install it with: pip install google-genai\n" + ) + sys.exit(1) + genai = imported_genai + types = imported_types # --------------------------------------------------------------------------- # # Constants — keep the default model IDs in lockstep with the web demo (src/types.ts) @@ -59,6 +70,7 @@ IMAGE_MODEL = "gemini-3.1-flash-lite-image" # direct infographic rendering QUALITY_IMAGE_MODEL = "gemini-3.1-flash-image" # skill-only quality option SUPPORTED_IMAGE_MODELS = (IMAGE_MODEL, QUALITY_IMAGE_MODEL) +IMAGE_PROMPT_PREFIX = "Generate a professional infographic image" AISTUDIO_KEY_URL = "https://aistudio.google.com/apikey" CONFIG_DIR = Path(os.environ.get("XDG_CONFIG_HOME", Path.home() / ".config")) / "infographic-agent" @@ -114,8 +126,16 @@ Respond with valid JSON only. No markdown fences. No extra text. Schema: { - "title": "string — compelling infographic title", - "prompt": "string — the complete image-generation prompt following " + "analysis": { + "title": "string — compelling infographic title", + "subtitle": "string — supporting subtitle", + "sectionsCount": number, + "dataPointsCount": number, + "brandColors": ["#hex", "#hex", "..."], + "sourceAttribution": "string — source credits" + }, + "prompt": "string — the complete image-generation prompt following ", + "allTextStrings": ["every", "text", "string", "in", "the", "infographic"] } """ @@ -187,6 +207,186 @@ def _friendly_api_error(err: Exception) -> str: return _scrub(s) +def _word_count(text: str) -> int: + return len([w for w in text.strip().split() if w]) + + +def _as_string_list(value) -> list: + return [str(v).strip() for v in value if str(v).strip()] if isinstance(value, list) else [] + + +def _quoted_in_prompt(prompt: str, text: str) -> bool: + return f'"{text}"' in prompt + + +def evaluate_prepare_result(plan: dict) -> list: + """Deterministic handoff checks matching the web app's Prepare eval gate.""" + analysis = plan.get("analysis") if isinstance(plan.get("analysis"), dict) else {} + prompt = (plan.get("prompt") or "").strip() + all_text = _as_string_list(plan.get("allTextStrings")) + brand_colors = _as_string_list(analysis.get("brandColors")) + + schema_issues = [] + if not str(analysis.get("title") or "").strip(): + schema_issues.append("missing title") + if not isinstance(analysis.get("subtitle"), str): + schema_issues.append("missing subtitle") + if not isinstance(analysis.get("sectionsCount"), (int, float)): + schema_issues.append("missing sectionsCount") + if not isinstance(analysis.get("dataPointsCount"), (int, float)): + schema_issues.append("missing dataPointsCount") + if not isinstance(analysis.get("brandColors"), list): + schema_issues.append("missing brandColors") + if not isinstance(analysis.get("sourceAttribution"), str): + schema_issues.append("missing sourceAttribution") + if not prompt: + schema_issues.append("missing prompt") + if not isinstance(plan.get("allTextStrings"), list): + schema_issues.append("missing allTextStrings") + + invalid_colors = [c for c in brand_colors if not re.match(r"^#[0-9a-fA-F]{6}$", c)] + if invalid_colors: + schema_issues.append(f"invalid brand color {invalid_colors[0]}") + + missing_quoted = [text for text in all_text if not _quoted_in_prompt(prompt, text)] + prompt_words = _word_count(prompt) + + return [ + { + "id": "schema", + "label": "Structured schema", + "status": "pass" if not schema_issues else "fail", + "detail": "Prepare output includes the required analysis fields, prompt, and text list." + if not schema_issues else "; ".join(schema_issues), + }, + { + "id": "image-prompt", + "label": "Explicit image prompt", + "status": "pass" if prompt.startswith(IMAGE_PROMPT_PREFIX) else "fail", + "detail": "Prompt starts with the required image-generation request." + if prompt.startswith(IMAGE_PROMPT_PREFIX) else f'Prompt must start with "{IMAGE_PROMPT_PREFIX}".', + }, + { + "id": "text-fidelity", + "label": "Text fidelity", + "status": "fail" if not all_text else "pass" if not missing_quoted else "warn", + "detail": "No final text strings were returned for the renderer." + if not all_text else "All final text strings are quoted in the renderer prompt." + if not missing_quoted else f"{len(missing_quoted)} text string(s) are not quoted exactly in the renderer prompt.", + }, + { + "id": "grounding", + "label": "Grounding", + "status": "pass" if str(analysis.get("sourceAttribution") or "").strip() else "warn", + "detail": "Source attribution is present." + if str(analysis.get("sourceAttribution") or "").strip() else "Source attribution is empty; generated facts may be harder to audit.", + }, + { + "id": "accessibility", + "label": "Accessibility", + "status": "pass" if re.search(r"\b(WCAG|contrast|accessib)", prompt, re.I) else "warn", + "detail": "Prompt includes contrast or accessibility instructions." + if re.search(r"\b(WCAG|contrast|accessib)", prompt, re.I) else "Prompt does not explicitly mention contrast or accessibility.", + }, + { + "id": "prompt-length", + "label": "Prompt length", + "status": "pass" if prompt_words <= 800 else "warn" if prompt_words <= 1200 else "fail", + "detail": f"Prompt is {prompt_words} words, within the 800-word target." + if prompt_words <= 800 else f"Prompt is {prompt_words} words; target is 800 words for renderer reliability." + if prompt_words <= 1200 else f"Prompt is {prompt_words} words; shorten before rendering.", + }, + ] + + +def validate_prepare_result(plan: dict) -> dict: + checks = evaluate_prepare_result(plan) + failures = [c for c in checks if c["status"] == "fail"] + if failures: + details = " ".join(f'{c["label"]}: {c["detail"]}' for c in failures) + raise ValueError(f"Prepare result failed quality gates: {details}") + plan["qualityChecks"] = checks + return plan + + +def infer_title(content: str) -> str: + for line in content.splitlines(): + cleaned = re.sub(r"^[#*\-\s]+", "", line).strip() + if cleaned: + return cleaned[:80] + return "Infographic" + + +def normalize_prepare_plan(plan: dict, content: str, mode: str, extra: str) -> dict: + """Accept the current web schema and repair older two-field plans when possible.""" + if not isinstance(plan, dict): + plan = {} + + prompt = (plan.get("prompt") or "").strip() + title = "" + if isinstance(plan.get("analysis"), dict): + title = str(plan["analysis"].get("title") or "").strip() + title = title or str(plan.get("title") or "").strip() or infer_title(content) + + analysis = plan.get("analysis") if isinstance(plan.get("analysis"), dict) else {} + normalized = { + "analysis": { + "title": title, + "subtitle": str(analysis.get("subtitle") or MODES.get(mode, "") or "Visual summary").strip(), + "sectionsCount": analysis.get("sectionsCount") if isinstance(analysis.get("sectionsCount"), (int, float)) else 3, + "dataPointsCount": analysis.get("dataPointsCount") if isinstance(analysis.get("dataPointsCount"), (int, float)) else len(re.findall(r"\d+(?:\.\d+)?%?", content)), + "brandColors": analysis.get("brandColors") if isinstance(analysis.get("brandColors"), list) else ["#4285F4", "#34A853", "#FBBC04"], + "sourceAttribution": str(analysis.get("sourceAttribution") or "User-provided content").strip(), + }, + "prompt": prompt, + "allTextStrings": plan.get("allTextStrings") if isinstance(plan.get("allTextStrings"), list) else [title], + } + + if extra and extra not in normalized["prompt"]: + normalized["prompt"] = f'{normalized["prompt"]}\n\nAdditional instructions: {extra}' + + return normalized + + +def direct_prepare_plan(content: str, mode: str, extra: str) -> dict: + """Build a renderer-ready PrepareResult without the research agent.""" + title = infer_title(content) + mode_hint = MODES.get(mode, "") + prompt = ( + f'Generate a professional infographic image. At the top, place "{title}" as the main title. ' + "Clearly and accurately visualize the provided content using a clean modern layout with strong visual hierarchy, " + "legible typography, accessible color contrast (minimum 4.5:1), and polished spacing. " + "Use #4285F4 as the primary color, #34A853 as the secondary color, and #FBBC04 as an accent. " + "Render all quoted text exactly as written. " + + (f"Style: {mode_hint}. " if mode_hint else "") + + (f"Additional instructions: {extra}. " if extra else "") + + f"Source content to summarize visually: {content}" + ) + return validate_prepare_result({ + "analysis": { + "title": title, + "subtitle": mode_hint or "Visual summary", + "sectionsCount": 3, + "dataPointsCount": len(re.findall(r"\d+(?:\.\d+)?%?", content)), + "brandColors": ["#4285F4", "#34A853", "#FBBC04"], + "sourceAttribution": "User-provided content", + }, + "prompt": prompt, + "allTextStrings": [title], + }) + + +def print_prepare_eval(plan: dict) -> None: + checks = plan.get("qualityChecks") or [] + if not checks: + return + passed = sum(1 for c in checks if c.get("status") == "pass") + warnings = [c for c in checks if c.get("status") == "warn"] + info(f"✅ Prepare evals: {passed}/{len(checks)} passed" + (f", {len(warnings)} warning(s)" if warnings else "")) + for check in warnings: + warn(f'{check["label"]}: {check["detail"]}') + + # --------------------------------------------------------------------------- # # API key onboarding — the "one-click, free from AI Studio" flow # --------------------------------------------------------------------------- # @@ -291,6 +491,7 @@ def resolve_api_key(force_setup: bool = False) -> str: def build_client(api_key: str) -> "genai.Client": + ensure_genai() if api_key: return genai.Client(api_key=api_key) # Vertex AI fallback (GOOGLE_CLOUD_PROJECT is set). @@ -305,7 +506,7 @@ def build_client(api_key: str) -> "genai.Client": # Agent 1 — research orchestrator (gemini-3.5-flash + Google Search grounding) # --------------------------------------------------------------------------- # -def research_prompt(client, content: str, mode: str, aspect: str, extra: str) -> str: +def research_prompt(client, content: str, mode: str, aspect: str, extra: str) -> dict: """Ground the topic and engineer a precise image-generation prompt.""" mode_hint = MODES.get(mode, "") user_prompt = "\n".join( @@ -336,14 +537,14 @@ def research_prompt(client, content: str, mode: str, aspect: str, extra: str) -> response = client.models.generate_content( model=ORCHESTRATOR_MODEL, contents=user_prompt, config=config ) - plan = _parse_json(response.text or "") - prompt = (plan.get("prompt") or "").strip() - title = (plan.get("title") or "").strip() - if not prompt: + plan = validate_prepare_result(normalize_prepare_plan(_parse_json(response.text or ""), content, mode, extra)) + title = plan["analysis"]["title"] + if not plan["prompt"]: raise ValueError("research agent returned no prompt") if title: info(f' ↳ "{title}"') - return prompt + print_prepare_eval(plan) + return plan except Exception as e: # noqa: BLE001 last_error = e if _is_transient(e) and attempt < 2: @@ -356,21 +557,16 @@ def research_prompt(client, content: str, mode: str, aspect: str, extra: str) -> # Research is a best-effort enhancement: fall back to a direct prompt so the # user still gets an image instead of a hard failure. warn(f"Research step failed ({_friendly_api_error(last_error)}). Falling back to a direct prompt.") - return direct_prompt(content, mode, extra) + plan = direct_prompt(content, mode, extra) + print_prepare_eval(plan) + return plan -def direct_prompt(content: str, mode: str, extra: str) -> str: +def direct_prompt(content: str, mode: str, extra: str) -> dict: """Build an image prompt without the research agent (used by --no-research / fallback).""" - mode_hint = MODES.get(mode, "") - return ( - "Generate a professional infographic image that clearly and accurately visualizes " - "the following content. Use a clean modern layout with strong visual hierarchy, " - "legible typography, and accessible color contrast (minimum 4.5:1). " - "Render all key text exactly as written.\n\n" - + (f"Style: {mode_hint}\n\n" if mode_hint else "") - + (f"Additional instructions: {extra}\n\n" if extra else "") - + f"Content:\n{content}" - ) + plan = direct_prepare_plan(content, mode, extra) + print_prepare_eval(plan) + return plan def _parse_json(text: str) -> dict: @@ -594,10 +790,10 @@ def main() -> None: try: if args.no_research: - prompt = direct_prompt(content, args.mode, args.instructions) + plan = direct_prompt(content, args.mode, args.instructions) else: - prompt = research_prompt(client, content, args.mode, args.aspect, args.instructions) - image_bytes, mime = generate_image(client, prompt, args.aspect, args.image_model) + plan = research_prompt(client, content, args.mode, args.aspect, args.instructions) + image_bytes, mime = generate_image(client, plan["prompt"], args.aspect, args.image_model) except Exception as e: # noqa: BLE001 error(_friendly_api_error(e)) sys.exit(1) diff --git a/src/__tests__/hooks/useInfographicFlow.test.ts b/src/__tests__/hooks/useInfographicFlow.test.ts index a1c00f3..90b12e4 100644 --- a/src/__tests__/hooks/useInfographicFlow.test.ts +++ b/src/__tests__/hooks/useInfographicFlow.test.ts @@ -349,6 +349,38 @@ describe('useInfographicFlow', () => { expect(result.current.state.agentLoop.phases.find(phase => phase.id === 'render')?.status).toBe('complete'); }); + it('does not fail generation when filename generation fails', async () => { + vi.mocked(geminiService.generateFilename).mockRejectedValueOnce(new Error('filename failed')); + const { result } = renderHook(() => useInfographicFlow()); + + await act(async () => { + await result.current.handleGenerate(); + }); + + await waitFor(() => { + expect(result.current.state.generationPhase).toBe('complete'); + }); + + expect(result.current.state.error).toBeNull(); + expect(result.current.state.currentResult?.filename).toBe('loop-test'); + expect(result.current.state.history[0]?.filename).toBe('loop-test'); + }); + + it('updates the fallback filename when the filename sidecar completes', async () => { + vi.mocked(geminiService.generateFilename).mockResolvedValueOnce('generated-loop-filename'); + const { result } = renderHook(() => useInfographicFlow()); + + await act(async () => { + await result.current.handleGenerate(); + }); + + await waitFor(() => { + expect(result.current.state.currentResult?.filename).toBe('generated-loop-filename'); + }); + + expect(result.current.state.history[0]?.filename).toBe('generated-loop-filename'); + }); + it('advances the loop turn for a focused refinement', async () => { const { result } = renderHook(() => useInfographicFlow()); diff --git a/src/__tests__/services/geminiService.test.ts b/src/__tests__/services/geminiService.test.ts index b04a3ec..eb97f20 100644 --- a/src/__tests__/services/geminiService.test.ts +++ b/src/__tests__/services/geminiService.test.ts @@ -5,6 +5,7 @@ import { hasApiKey, getTrialTurnsCount, incrementTrialTurns, + evaluatePrepareResult, } from '../../services/geminiService'; import type { AdminConfig } from '../../types'; @@ -195,6 +196,61 @@ describe('geminiService', () => { }); }); + describe('prepare result evals', () => { + it('passes a renderer-ready prepare result', () => { + const checks = evaluatePrepareResult({ + analysis: { + title: 'Loop Test', + subtitle: 'Stateful generation', + sectionsCount: 2, + dataPointsCount: 3, + brandColors: ['#4285F4', '#34A853'], + sourceAttribution: 'Uploaded source', + }, + prompt: 'Generate a professional infographic image. At the top, place "Loop Test". Include WCAG AA contrast guidance.', + allTextStrings: ['Loop Test'], + }); + + expect(checks.every(check => check.status === 'pass')).toBe(true); + }); + + it('fails blocking renderer contract violations', () => { + const checks = evaluatePrepareResult({ + analysis: { + title: 'Loop Test', + subtitle: 'Stateful generation', + sectionsCount: 2, + dataPointsCount: 3, + brandColors: ['#4285F4'], + sourceAttribution: 'Uploaded source', + }, + prompt: 'Create a nice visualization with WCAG AA contrast guidance.', + allTextStrings: ['Loop Test'], + }); + + expect(checks.find(check => check.id === 'image-prompt')?.status).toBe('fail'); + expect(checks.find(check => check.id === 'text-fidelity')?.status).toBe('warn'); + }); + + it('warns when source attribution or accessibility guidance is missing', () => { + const checks = evaluatePrepareResult({ + analysis: { + title: 'Loop Test', + subtitle: 'Stateful generation', + sectionsCount: 2, + dataPointsCount: 3, + brandColors: ['#4285F4'], + sourceAttribution: '', + }, + prompt: 'Generate a professional infographic image. At the top, place "Loop Test".', + allTextStrings: ['Loop Test'], + }); + + expect(checks.find(check => check.id === 'grounding')?.status).toBe('warn'); + expect(checks.find(check => check.id === 'accessibility')?.status).toBe('warn'); + }); + }); + describe('trial turns', () => { it('should return 0 by default', () => { expect(getTrialTurnsCount()).toBe(0); diff --git a/src/components/ThoughtStream.tsx b/src/components/ThoughtStream.tsx index 60b28a4..da7859f 100644 --- a/src/components/ThoughtStream.tsx +++ b/src/components/ThoughtStream.tsx @@ -172,6 +172,9 @@ function LoopStatus({ agentLoop }: { agentLoop: AgentLoopState }) { export default function ThoughtStream({ thoughts, phase, elapsed, prepareResult, agentLoop }: ThoughtStreamProps) { const scrollRef = useRef(null); + const qualityChecks = prepareResult?.qualityChecks ?? []; + const passedQualityChecks = qualityChecks.filter(check => check.status === 'pass').length; + const warningQualityChecks = qualityChecks.filter(check => check.status === 'warn').length; useEffect(() => { if (scrollRef.current) { @@ -225,6 +228,20 @@ export default function ThoughtStream({ thoughts, phase, elapsed, prepareResult,

{prepareResult.analysis.sectionsCount} sections, {prepareResult.analysis.dataPointsCount} data points

+ {qualityChecks.length > 0 && ( +
+ + verified + {passedQualityChecks}/{qualityChecks.length} evals passed + + {warningQualityChecks > 0 && ( + + warning + {warningQualityChecks} warning{warningQualityChecks === 1 ? '' : 's'} + + )} +
+ )} {prepareResult.analysis.brandColors.length > 0 && (
Colors: diff --git a/src/hooks/useInfographicFlow.ts b/src/hooks/useInfographicFlow.ts index 841b093..071e086 100644 --- a/src/hooks/useInfographicFlow.ts +++ b/src/hooks/useInfographicFlow.ts @@ -90,6 +90,19 @@ function normalizeResolution(value: unknown): ImageResolution { : DEFAULT_INFOGRAPHIC_CONFIG.resolution; } +function buildFallbackFilename(title: string | undefined): string { + const normalized = (title || 'infographic') + .toLowerCase() + .replace(/[^a-z0-9]+/g, '-') + .replace(/^-+|-+$/g, '') + .split('-') + .filter(Boolean) + .slice(0, 5) + .join('-'); + + return normalized || `infographic-${Date.now()}`; +} + function loadHasVisited(): boolean { try { return localStorage.getItem(HAS_VISITED_KEY) === 'true'; @@ -317,11 +330,13 @@ export function useInfographicFlow() { // Phase 2: Generate image setState(s => ({ ...s, generationPhase: 'generating', streamingText: 'Generating infographic...' })); const referenceImages = state.files.filter(f => f.category === 'image').slice(0, 3); - const [result, filename] = await Promise.all([ - generateInfographic(prepResult.prompt, referenceImages, { ...state.config, resolution: state.adminConfig.imageResolution || state.config.resolution }, state.adminConfig), - generateFilename(prepResult.prompt, state.adminConfig) - ]); - result.filename = filename; + const result = await generateInfographic( + prepResult.prompt, + referenceImages, + { ...state.config, resolution: state.adminConfig.imageResolution || state.config.resolution }, + state.adminConfig + ); + result.filename = buildFallbackFilename(prepResult.analysis.title); const historyEntry = { id: `history_${Date.now()}`, @@ -361,6 +376,24 @@ export function useInfographicFlow() { hitlStatus: 'awaiting-input', }), })); + + void generateFilename(prepResult.prompt, state.adminConfig) + .then((filename) => { + if (!filename || filename === result.filename) return; + setState(s => { + const shouldUpdateCurrent = s.currentResult?.imageData === result.imageData && s.agentLoop.sessionId === sessionId; + return { + ...s, + currentResult: shouldUpdateCurrent && s.currentResult + ? { ...s.currentResult, filename } + : s.currentResult, + history: s.history.map(entry => entry.id === historyEntry.id ? { ...entry, filename } : entry), + }; + }); + }) + .catch(() => { + // Filename generation is a convenience sidecar, not part of the render loop. + }); } catch (err) { setState(s => ({ ...s, diff --git a/src/services/geminiService.ts b/src/services/geminiService.ts index 63292b5..be626a1 100644 --- a/src/services/geminiService.ts +++ b/src/services/geminiService.ts @@ -7,6 +7,7 @@ import { GoogleGenAI } from '@google/genai'; import type { UploadedFile, PrepareResult, + PrepareQualityCheck, GenerationResult, AdminConfig, InfographicConfig, @@ -292,6 +293,120 @@ function parseJsonResponse(text: string): T { return JSON.parse(cleaned) as T; } +// --------------------------------------------------------------------------- +// Prepare-result eval gate +// --------------------------------------------------------------------------- + +const IMAGE_PROMPT_PREFIX = 'Generate a professional infographic image'; + +function isRecord(value: unknown): value is Record { + return typeof value === 'object' && value !== null && !Array.isArray(value); +} + +function wordCount(text: string): number { + return text.trim().split(/\s+/).filter(Boolean).length; +} + +function asStringArray(value: unknown): string[] { + return Array.isArray(value) + ? value.filter((item): item is string => typeof item === 'string').map(item => item.trim()).filter(Boolean) + : []; +} + +function includesQuotedText(prompt: string, text: string): boolean { + return prompt.includes(`"${text}"`); +} + +export function evaluatePrepareResult(result: Partial | null | undefined): PrepareQualityCheck[] { + const analysis: Record = isRecord(result?.analysis) ? result.analysis : {}; + const prompt = typeof result?.prompt === 'string' ? result.prompt.trim() : ''; + const allTextStrings = asStringArray(result?.allTextStrings); + const brandColors = asStringArray(analysis.brandColors); + + const schemaIssues = [ + typeof analysis.title === 'string' && analysis.title.trim() ? null : 'missing title', + typeof analysis.subtitle === 'string' ? null : 'missing subtitle', + typeof analysis.sectionsCount === 'number' ? null : 'missing sectionsCount', + typeof analysis.dataPointsCount === 'number' ? null : 'missing dataPointsCount', + Array.isArray(analysis.brandColors) ? null : 'missing brandColors', + typeof analysis.sourceAttribution === 'string' ? null : 'missing sourceAttribution', + prompt ? null : 'missing prompt', + Array.isArray(result?.allTextStrings) ? null : 'missing allTextStrings', + ].filter((issue): issue is string => Boolean(issue)); + + const invalidColors = brandColors.filter(color => !/^#[0-9a-f]{6}$/i.test(color)); + if (invalidColors.length > 0) { + schemaIssues.push(`invalid brand color ${invalidColors[0]}`); + } + + const missingQuotedStrings = allTextStrings.filter(text => !includesQuotedText(prompt, text)); + const promptWords = wordCount(prompt); + + return [ + { + id: 'schema', + label: 'Structured schema', + status: schemaIssues.length === 0 ? 'pass' : 'fail', + detail: schemaIssues.length === 0 + ? 'Prepare output includes the required analysis fields, prompt, and text list.' + : schemaIssues.join('; '), + }, + { + id: 'image-prompt', + label: 'Explicit image prompt', + status: prompt.startsWith(IMAGE_PROMPT_PREFIX) ? 'pass' : 'fail', + detail: prompt.startsWith(IMAGE_PROMPT_PREFIX) + ? 'Prompt starts with the required image-generation request.' + : `Prompt must start with "${IMAGE_PROMPT_PREFIX}".`, + }, + { + id: 'text-fidelity', + label: 'Text fidelity', + status: allTextStrings.length === 0 ? 'fail' : missingQuotedStrings.length === 0 ? 'pass' : 'warn', + detail: allTextStrings.length === 0 + ? 'No final text strings were returned for the renderer.' + : missingQuotedStrings.length === 0 + ? 'All final text strings are quoted in the renderer prompt.' + : `${missingQuotedStrings.length} text string(s) are not quoted exactly in the renderer prompt.`, + }, + { + id: 'grounding', + label: 'Grounding', + status: typeof analysis.sourceAttribution === 'string' && analysis.sourceAttribution.trim() ? 'pass' : 'warn', + detail: typeof analysis.sourceAttribution === 'string' && analysis.sourceAttribution.trim() + ? 'Source attribution is present.' + : 'Source attribution is empty; generated facts may be harder to audit.', + }, + { + id: 'accessibility', + label: 'Accessibility', + status: /\b(WCAG|contrast|accessib)/i.test(prompt) ? 'pass' : 'warn', + detail: /\b(WCAG|contrast|accessib)/i.test(prompt) + ? 'Prompt includes contrast or accessibility instructions.' + : 'Prompt does not explicitly mention contrast or accessibility.', + }, + { + id: 'prompt-length', + label: 'Prompt length', + status: promptWords <= 800 ? 'pass' : promptWords <= 1200 ? 'warn' : 'fail', + detail: promptWords <= 800 + ? `Prompt is ${promptWords} words, within the 800-word target.` + : promptWords <= 1200 + ? `Prompt is ${promptWords} words; target is 800 words for renderer reliability.` + : `Prompt is ${promptWords} words; shorten before rendering.`, + }, + ]; +} + +function validatePrepareResult(result: PrepareResult): PrepareResult { + const qualityChecks = evaluatePrepareResult(result); + const failures = qualityChecks.filter(check => check.status === 'fail'); + if (failures.length > 0) { + throw new Error(`Prepare result failed quality gates: ${failures.map(check => `${check.label}: ${check.detail}`).join(' ')}`); + } + return { ...result, qualityChecks }; +} + // --------------------------------------------------------------------------- // Agent 1: Prepare Infographic (analyzes content, plans layout, engineers prompt) // --------------------------------------------------------------------------- @@ -452,7 +567,7 @@ export async function prepareInfographic( } accumulatedText += texts.join(''); } - return parseJsonResponse(accumulatedText); + return validatePrepareResult(parseJsonResponse(accumulatedText)); }); } @@ -688,6 +803,11 @@ export async function refineInfographic( } export async function generateFilename(prompt: string, adminConfig: AdminConfig): Promise { + if (!checkRateLimit()) { + const { resetSeconds } = getRateLimitStatus(); + throw new Error(`Rate limited: 10 requests per minute. Please wait ${resetSeconds}s and retry.`); + } + return retryWithBackoff(async () => { const ai = getAI(adminConfig); const response = await ai.models.generateContent({ diff --git a/src/types.ts b/src/types.ts index 0c08101..7160e7c 100644 --- a/src/types.ts +++ b/src/types.ts @@ -30,6 +30,22 @@ export interface UploadedFile { } // === Agent Output Types === +export type PrepareQualityCheckStatus = 'pass' | 'warn' | 'fail'; +export type PrepareQualityCheckId = + | 'schema' + | 'image-prompt' + | 'text-fidelity' + | 'grounding' + | 'accessibility' + | 'prompt-length'; + +export interface PrepareQualityCheck { + id: PrepareQualityCheckId; + label: string; + status: PrepareQualityCheckStatus; + detail: string; +} + export interface PrepareResult { analysis: { title: string; @@ -41,6 +57,7 @@ export interface PrepareResult { }; prompt: string; allTextStrings: string[]; + qualityChecks?: PrepareQualityCheck[]; } // === Generation Types ===