ci: add a Live Tests job, starting with notes against Ollama Cloud by goosewobbler · Pull Request #697 · goosewobbler/releasekit

goosewobbler · 2026-10-22T22:25:39Z

Description

Second half of #589: CI now runs tests against live external services, starting with the notes suite against Ollama Cloud (ollama-live.spec.ts). #618 made that spec tolerant of provider outages; this adds the job.

A generic Live Tests job. What sets these tests apart is the same in any package: they need a secret, make paid calls, and skip on an outage. So the job is live-tests ("Live Tests", _ci-live.reusable.yml) and each suite is one step, today "Notes against live Ollama". Adding another suite means a step with its own env/secrets, plus its package paths in the live filter; AGENTS.md now says what a live test is and how CI runs it.

When it runs: a new run_live output from Detect Changes, covering notes' src/, package.json and the spec, releasekit.config.json, and the CI workflows that wire the job up. Docs edits don't make paid calls; force_all still sets it. The build also runs on run_live, so a live-only change gets a build without running the rest of the matrix. Every live step runs whenever any live path changes, which is simpler than per-step gating while there's one suite.
Skips: fork PRs and a fork's own CI (anything outside goosewobbler/releasekit, no secrets), the standing release PR (release/next, version bumps only), and pushes (the PR already ran it). Dependabot runs it, so notes' SDK bumps get a live check; that needs OLLAMA_API_KEY as a Dependabot secret.
Blocking: it feeds CI Status, so a failure blocks and a skip passes. The spec itself skips with a "notes LLM live test skipped" warning on an Ollama outage.
The notes step runs pnpm --filter @releasekit/notes test ollama-live (not turbo: its strict env mode strips the gate variable) with RELEASEKIT_NOTES_LIVE=1 and the OLLAMA_API_KEY secret. The model and base URL come from releasekit.config.json's release-notes settings, so it tests exactly what releases use, and a config change triggers it. A missing key fails the step. If a new model is slower than the spec's 45s per-call timeout, the spec skips with the warning instead of failing, so watch for it after a change. Permissions: contents: read.

Related Issue(s)

Closes #589

Type of Change

feat — New feature
fix — Bug fix
docs — Documentation change
refactor — Code refactoring
test — Adding or updating tests
chore — Maintenance, dependencies, tooling
Breaking change — Requires major version bump

Scope

Checklist

Lint passes (pnpm lint)
Tests pass (pnpm test)
Typecheck passes (pnpm typecheck)
Documentation updated (if applicable)
Breaking changes noted in description

Testing Notes

actionlint and Biome 2.6 are clean on the changed workflows.
This PR's runs passed against live Ollama Cloud with no skip: minimax-m2.7:cloud (chunk test 40.2s, text test 18.0s), then minimax-m3.1-flash:cloud (13.9s, 4.0s).
Against https://ollama.com with a bogus key, both tests fail on a non-retryable 401: the URL is right, the key is sent, and a bad secret fails rather than skips.

🤖 Generated with Claude Code

github-actions · 2026-10-22T22:26:15Z

Standing release PR: #614 · v0.42.0 queued (4 packages) · open 82d 8h 7m · ✅ ready to merge

Release Preview — v0.41.2 → v0.41.3

Note: Labels on this PR are advisory in standing-pr mode. Bumps come from conventional commits in the standing PR; override by editing labels on the standing PR itself. Add release:immediate to bypass the standing PR and release this PR directly.

These changes will be added to the release PR (#614) when merged:

Changelog

monorepo 0.41.2 → 0.41.3

Changed

test the release-notes model from the config, and widen who runs Live Tests

run tests when packages/forge or packages/git change (PR #705 · closes #704)

tighten the Live Tests job after review

say what live tests are and how CI runs them (agents)

make the live job generic: Live Tests, one step per suite

take the live test's Ollama model and base URL from repo variables

run the notes LLM live test per-PR against Ollama Cloud (#589)

All packages → 0.41.3 (sync versioning)

@releasekit/version

@releasekit/notes

@releasekit/publish

@releasekit/release

After merge — predicted release

No version escalation — this PR's changes will be included in the queued release without affecting the projected versions.

Package	Standing PR	This PR	After merge
`monorepo`	0.42.0	0.41.3	0.42.0

Updated automatically by ReleaseKit

Add a Notes LLM (live Ollama) job to CI that runs ollama-live.spec.ts against Ollama Cloud (minimax-m2.7:cloud, the model this repo's own release notes use) when packages/notes or the job's workflow changes. It feeds CI Status, so a failure blocks and a skip passes; the spec itself skips with a warning on an Ollama outage. Forks and Dependabot get no secrets, and the standing release PR only bumps versions, so the job skips for all three. Pushes skip too: the PR already ran it. Closes #589 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Tests that hit a live external service share what sets them apart (a secret, a paid call, skip-on-outage), whatever package they're in. Rename the job, its workflow and the detect-changes output to match, and name the notes run as one step. Another live suite adds a step and its package paths to the live filter. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

- Skip it outside goosewobbler/releasekit, so a fork's own CI doesn't run it without the secret. - Count live changes in has_changes, so a live suite in an otherwise unfiltered package still gets a build. - Narrow the notes paths to src, package.json and the spec, so docs edits don't make paid calls. - AGENTS.md: a new live suite's secret must be declared and passed in. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…Live Tests - Read the model and base URL from releasekit.config.json instead of repo variables, so the live test checks exactly what releases use, and a config change triggers it. - Run it for Dependabot too (needs OLLAMA_API_KEY as a Dependabot secret), so SDK bumps get a live run. - Add ci.yml and detect-changes to the live filter, so a broken guard or secret mapping shows up in the PR that breaks it. - Gate build on run_live as well, instead of counting live changes in has_changes, so a live-only change doesn't run the whole matrix. - OLLAMA_API_KEY isn't required, matching the other reusable workflows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

goosewobbler mentioned this pull request Oct 22, 2026

test(notes): make the LLM e2e infra-tolerant and close a false green #618

Merged

goosewobbler changed the title ~~ci: run the notes LLM e2e per-PR against Ollama Cloud~~ ci: run the notes LLM live test per-PR against Ollama Cloud Oct 23, 2026

goosewobbler force-pushed the ci/589-notes-llm-e2e branch from 885042a to 3007a48 Compare October 23, 2026 01:02

ci: take the live test's Ollama model and base URL from repo variables

ee34138

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

goosewobbler mentioned this pull request Oct 23, 2026

chore: generate release notes with the OLLAMA_MODEL repo variable's model #703

Merged

3 of 20 tasks

goosewobbler changed the title ~~ci: run the notes LLM live test per-PR against Ollama Cloud~~ ci: add a Live Tests job, starting with notes against Ollama Cloud Oct 23, 2026

github-actions Bot and others added 3 commits October 23, 2026 02:44

Merge remote-tracking branch 'origin/main' into ci/589-notes-llm-e2e

5192b35

docs(agents): say what live tests are and how CI runs them

5a661aa

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

goosewobbler mentioned this pull request Oct 23, 2026

ci: run tests when packages/forge or packages/git change #705

Merged

3 of 20 tasks

goosewobbler merged commit 2cd551e into main Oct 23, 2026
24 checks passed

goosewobbler deleted the ci/589-notes-llm-e2e branch October 23, 2026 02:30

Navigation Menu

Repository navigation

ci: add a Live Tests job, starting with notes against Ollama Cloud - #697

ci: add a Live Tests job, starting with notes against Ollama Cloud#697
goosewobbler merged 7 commits into
mainfrom
ci/589-notes-llm-e2e

goosewobbler commented Oct 22, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 22, 2026 •

edited

Loading

Changelog

Changed

After merge — predicted release

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

goosewobbler commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issue(s)

Type of Change

Scope

Checklist

Testing Notes

Uh oh!

github-actions Bot commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelog

Changed

After merge — predicted release

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

goosewobbler commented Oct 22, 2026 •

edited

Loading

github-actions Bot commented Oct 22, 2026 •

edited

Loading