Skip to content

ci: run the examples/ cookbook end-to-end on every build - #173

Open
javimosch wants to merge 1 commit into
masterfrom
mago/task-172
Open

javimosch wants to merge 1 commit into
masterfrom
mago/task-172

Conversation

@javimosch

Copy link
Copy Markdown
Owner

Summary

Adds a CI job that runs the entire examples/ cookbook end-to-end on every push and PR, so published recipes are verified against every build — no API key or network required.

  • .github/workflows/examples.yml — new workflow: checkout → mlugg/setup-zig@v2 (0.16.0, matching the pinned toolchain) → zig build → scripts/smoke.sh --group=examples.
  • scripts/lib/fake-curl — a python3 curl shim. tau shells out to curl for all provider HTTP (src/llm/provider.zig), so placing this earlier on PATH stubs only the network leg while the real binary, agent loop, tools, sessions, and goal mode execute. It plans tool calls from the recipe prompt (verb → /abs/path pairs for read/write/edit), serves both streaming SSE and --no-stream JSON shapes, and emits <GOAL_MET> on its own line in goal mode so /goal recipes terminate.
  • scripts/smoke.sh — new examples group in the offline suite. It symlinks the stub as curl, runs every examples/[0-9]*.sh and examples/*.py under a temp HOME with throwaway provider keys, and also exercises examples/ci/run.sh when present (lands automatically once docs: examples/ cookbook — NDJSON stream consumer + non-interactive CI pattern #149 merges). Includes per-recipe output pins (write verification, session persistence, provider exercise).

Recipe bugs this caught and fixed

The first real end-to-end run surfaced two latent bugs in the published examples:

  • 03-provider-switching.sh — with several provider keys set, PROV="${OPENAI_API_KEY:+openai}${XIAOMI_API_KEY:+xiaomi}..." concatenated into an invalid provider name (openaixiaomideepseek) → exit 80. Now picks the first configured provider.
  • 01-file-editing.sh — jq '.content | fromjson' on streamed NDJSON errors on chunk lines (.content is null) under set -euo pipefail. Now selects the done envelope first; same fix applied to the content() helper.

Verification

scripts/smoke.sh --group=examples   →  6/6 checks pass
scripts/smoke.sh (full offline)     →  same 17 pre-existing keyless-env
                                        failures as baseline (verified by
                                        stash + re-run), +6 new examples
                                        checks all green

Verified real behavior under the stub: example 02's /goal run performs actual write + read tool calls, produces the JSON-array file, and ends with "status":"complete" in /goal status; example 01's edit step performs a real in-place replacement.

Docs: examples/README.md gains a "CI verification" section; CHANGELOG.md updated.

Closes #172 (mago task #172)

Adds an `examples` smoke group + GitHub Actions workflow that executes every
published recipe (examples/*.sh, *.py, ci/run.sh) against a freshly built
binary. Provider HTTP is stubbed by scripts/lib/fake-curl — a python3 `curl`
shim placed earlier on PATH — which plans tool calls from the recipe prompt
(read/write/edit + goal-mode <GOAL_MET>), so the real agent loop, tools,
sessions, and goal mode run with no API key and no network.

Two latent recipe bugs surfaced by the first end-to-end run are fixed:
- 03-provider-switching.sh concatenated provider names when several API keys
  were set ("openaixiaomideepseek" -> exit 80); now picks the first
  configured provider.
- 01-file-editing.sh piped NDJSON into `jq '.content | fromjson'`, which
  errors on stream-chunk lines where .content is null; now selects the
  "done" envelope first (same fix applied to the content() helper).

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@coderabbitai

coderabbitai Bot commented Sep 30, 2026

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 56b81987-96c2-4866-9cc6-359fc4e24c3c


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add a CI job that runs the examples/ cookbook end-to-end so published recipes are verified against every build

2 participants