Skip to content

Cookbook: 'prompt-as-migration' recipe #244

@hidai25

Description

@hidai25

Why

A theme that's blowing up in May 2026 writeups: "your AI agent is one prompt update away from breaking three unrelated flows." Concrete examples: a one-line tone tweak that jumped refusal rate 14 points, a currency-format fix that broke vendor extraction days later.

The fix is conceptually simple — treat prompt changes like database migrations, gated by a paired before/after eval on the same examples. EvalView already does this. We just don't tell that story anywhere.

What

Add a cookbook entry under docs/ (or wherever cookbook docs live) walking a reader through:

  1. The pain (with one of the real examples cited above as a hook)
  2. The mental model: prompts are schema; changes need migrations
  3. The recipe with EvalView:
    • evalview snapshot before the change
    • make the prompt edit
    • evalview check after
    • what the diff tells you about which flows regressed
  4. Wiring it into CI so future prompt PRs auto-gate

Acceptance criteria

  • New cookbook page (e.g. docs/cookbook/prompt-as-migration.md)
  • One concrete worked example (can use the existing demo-agent)
  • Real command output (not pseudocode)
  • Cross-linked from README and ROADMAP
  • Title and meta-description tuned for "AI agent prompt regression" search intent

Hints

  • Reference posts for tone/context (don't lift content):
    • AscentCore: "Why Your AI Agents Are One Update Away from Breaking" (May 4, 2026)
    • The GitHub Blog post on validating non-deterministic agent behavior
  • Length target: 800-1200 words. Long enough to be useful, short enough to read in one sitting.

Size

~2-3 hours including the worked example.

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentationhelp wantedExtra attention is needed

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions