fix(layer3): replace retired GitHub Models with any OpenAI-compatible endpoint by Ostico · Pull Request #5 · Ostico/test-guard

Ostico · 2026-07-31T21:32:06Z

Problem

Every Layer 3 run fails:

AI analysis failed (Error code: 410 - {'error': {'code': 'github_models_retirement_brownout',
'message': 'GitHub Models is temporarily unavailable as part of a scheduled retirement brownout.'}})
— shortcuts resolved; remaining files deferred to L1+L2.

Despite the "temporarily unavailable" wording, this is permanent. GitHub Models was fully retired on 30 July 2026 — playground, model catalog, inference API and BYOK, for all customers. The brownouts on 16 and 23 July were rehearsals.

src/layer3_ai.py and src/summary.py both hardcoded https://models.github.ai/inference and authenticated with GITHUB_TOKEN.

Why not stay on GitHub

Checked before changing anything:

models: read only ever gated GitHub Models. It is now inert.
api.githubcopilot.com is undocumented, expects editor-OAuth tokens rather than a PAT, and automated CI use risks Copilot suspension.
actions/ai-inference@v2 does have provider: copilot billed to GitHub AI Credits, but it drives the Copilot CLI binary — no OpenAI-compatible HTTP endpoint, no response_format=json_schema, and it wants a separate PAT (COPILOT_GITHUB_TOKEN), not github.token.

So Layer 3 needs a provider key. GitHub's own recommended destination, Azure AI Foundry, is OpenAI-compatible — as are OpenAI and OpenRouter — so one provider-agnostic base_url covers every migration path.

Changes

New ai-base-url (default https://api.openai.com/v1) and ai-api-key inputs, threaded through run_layer3 and generate_summary.
_call_github_models → _call_ai_provider; base_url is now a parameter.
Default model chain drops the publisher prefix (openai/gpt-4.1-mini → gpt-4.1-mini). Explicitly configured IDs pass through untouched, so OpenRouter-style namespacing still works.
ai-enabled: true with no key degrades to L1+L2 with a ::warning:: instead of failing every run on a 401.
Dropped the inert models: read from the dogfood workflow.
README: breaking-change note, provider table, migration guidance, troubleshooting rows for 410/401/404.

Breaking change

Layer 3 no longer works with only a GITHUB_TOKEN. Consumers must add ai-api-key. Without it the gate still runs on Layers 1–2 rather than erroring.

Verification

pytest — 425 passed (includes new cases for base-url plumbing, custom endpoint config, and the missing-key guard)
ruff check src tests — clean (7 pre-existing findings in benchmarks/ untouched)

Not exercised: a live call against a real provider endpoint, which needs a key in CI secrets.

🤖 Generated with Claude Code

Update: `ai-reasoning-effort` (default `none`)

Layer 3 caps max_tokens at 2048 and thinking models count thought tokens as output, so an unbounded thought trace can truncate the JSON verdict — which _parse_ai_response reads as SKIP at confidence 0.0. Sending reasoning_effort: none by default disables thinking on Gemini 2.5 and Groq's gpt-oss, making those models usable with no tuning.

Non-reasoning models reject the parameter (OpenAI: "Unsupported value"; Azure: "Unrecognized request argument"; others: "Unknown parameter"), and the default config is OpenAI + gpt-4.1-mini, so a blanket send would break the default path. Instead:

sent only when non-empty — empty string omits it entirely
a 400/404/422 naming reasoning_effort triggers one retry without it
the rejecting (base_url, model) pair is remembered, so the fallback costs one extra request per run, not per batch
unrelated 400s still propagate

Also corrected in this branch: Groq supports strict: true only on openai/gpt-oss-20b and openai/gpt-oss-120b — not kimi-k2-instruct-0905, which is best-effort JSON / tool-use and fails Layer 3's strict call. Free-tier guidance now notes Groq's 8K tokens/min cap against Layer 3's 8K input budget, and that Layer 3 has no 429 backoff.

430 tests pass; ruff clean on src and tests. Not exercised: a live call against a real provider.

… endpoint GitHub Models was fully retired on 2026-07-30 (playground, catalog, inference API, BYOK). Runs now fail with: 410 {'code': 'github_models_retirement_brownout'} The endpoint was hardcoded in two places and authenticated with GITHUB_TOKEN. There is no GITHUB_TOKEN-authenticated replacement: `models: read` only ever gated GitHub Models, and Copilot (now billed as AI Credits) exposes no OpenAI-compatible endpoint an Action can call. Layer 3 therefore takes a provider key: - new `ai-base-url` input (default https://api.openai.com/v1) and `ai-api-key` input, threaded through run_layer3 and generate_summary - `_call_github_models` -> `_call_ai_provider`, base_url is a parameter - default model IDs lose the GitHub Models publisher prefix (`openai/gpt-4.1-mini` -> `gpt-4.1-mini`); explicit IDs pass through untouched so OpenRouter-style namespacing still works - ai-enabled with no key degrades to L1+L2 with a ::warning:: instead of failing every run on a 401 - drop the now-inert `models: read` permission from dogfood - README: breaking-change note, provider table (OpenAI / Azure AI Foundry / OpenRouter), migration and troubleshooting rows 425 tests pass; ruff clean on src and tests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

GitHub offers no free inference after the GitHub Models retirement. Copilot Pro stays free for verified OSS maintainers, students and teachers, but it is an IDE/CLI entitlement with no OpenAI-compatible endpoint, so this action cannot reach it. Azure AI Foundry, the official migration target, requires an Azure subscription. Document the third-party free tiers that do work with ai-base-url (Groq, Gemini, OpenRouter) plus the two constraints that decide whether they are usable: Layer 3 requires response_format=json_schema with strict: true, and free tiers generally train on submitted prompts. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The previous note claimed Groq supported strict structured output on kimi-k2-instruct-0905. Groq's docs list strict: true support for openai/gpt-oss-20b and openai/gpt-oss-120b only; kimi-k2 is best-effort JSON / tool-use, so Layer 3's strict call fails on it. Also record the constraints that decide whether a free tier works here: - Groq free tier is 8K tokens/min while Layer 3 budgets up to 8K input tokens per request, so one full batch can exhaust the minute and the next 429s. Layer 3 retries only on 413 and falls back only on 403, so a 429 abandons the batch and defers those files to L1+L2. - gemini-2.5-flash has thinking on by default and thought tokens count as output; with max_tokens capped at 2048 the JSON can truncate, which parses as SKIP at confidence 0.0. Recommend gemini-2.5-flash-lite, which defaults to thinking off. - Gemini per-model free quotas are no longer published in the API docs; link AI Studio instead of quoting numbers. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Layer 3 caps max_tokens at 2048 and thinking models count thought tokens as output, so an unbounded thought trace can truncate the JSON verdict — which _parse_ai_response reads as SKIP at confidence 0.0. Sending reasoning_effort: none by default disables thinking on Gemini 2.5 and Groq gpt-oss, making those models usable without tuning. Non-reasoning models reject the parameter: OpenAI answers "Unsupported value", Azure "Unrecognized request argument", others "Unknown parameter". Since the default config is OpenAI + gpt-4.1-mini, a blanket send would break the default path, so: - the parameter is only sent when non-empty (empty string omits it) - a 400/404/422 naming reasoning_effort triggers one retry without it - the rejecting (base_url, model) pair is remembered, so the fallback costs one extra request per run rather than per batch - unrelated 400s still propagate untouched 430 tests pass, ruff clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

With ai-reasoning-effort defaulting to none, the reason to prefer flash-lite (thinking off by default) no longer applies, and the capability gap decides it instead. Layer 3 asks the model to map changed behaviours to test assertions across four dimensions and calibrate confidence to evidence quality. On Artificial Analysis, Gemini 2.5 Flash without reasoning scores 14 on the Intelligence Index against 15 for gpt-4.1-mini, the model this action was tuned on. Flash-Lite without reasoning scores 7. Record why a weak model is dangerous here specifically: missed problems fail open to L1+L2, but a confidently wrong FAIL blocks the PR, because ai-confidence-threshold only softens FAIL to WARNING when the model reports low confidence. Weak models also omit files more often, and _validate_batch_verdicts discards the entire batch when that happens. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Points the dogfood workflow at the Gemini OpenAI-compatible endpoint so the next PR exercises the new provider-agnostic path end to end. gemini-2.5-flash rather than flash-lite: without reasoning it scores 14 on the Artificial Analysis Intelligence Index against 15 for the gpt-4.1-mini baseline this prompt was tuned on, where flash-lite scores 7. reasoning-effort is set explicitly even though none is the default, since this workflow doubles as the reference example: thinking must stay off or a thought trace can consume the 2048-token output budget and truncate the JSON verdict. Requires a GEMINI_API_KEY repository secret. Without it Layer 3 is skipped with a warning and the gate falls back to Layer 1 + Layer 2, so fork PRs (which get no secrets) degrade rather than fail. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The dogfood run failed before executing anything: action.yml (Line: 36): Unrecognized named-value: 'secrets'. Located at position 1 within expression: secrets.OPENAI_API_KEY GitHub evaluates ${{ }} expressions anywhere in action.yml, including input descriptions, and `secrets` is not in scope for a composite action, so the manifest fails to load and every consumer of the action breaks. Reword the description in prose and leave a comment so the trap is not reintroduced. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

.dev/local_pr_test.py still passed token=config.github_token with no base_url, so after the GitHub Models retirement it would call OpenAI with a GitHub token and 401. It was missed in the migration sweep because it lives outside src/ and tests/. It now reads the provider key from AI_API_KEY / GEMINI_API_KEY / OPENAI_API_KEY and forwards base_url and reasoning_effort, defaulting to gemini-2.5-flash with thinking off. Without a key it warns and runs the shortcut gates only. This harness is how the AI path gets exercised at all: it runs with coverage_files=[], so files fall through to the AI gates (6-8) instead of being resolved by the coverage shortcuts that decide everything in the dogfood workflow at coverage-threshold 100. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Running `python .dev/local_pr_test.py` failed with ModuleNotFoundError: No module named 'src', because executing the file directly puts .dev/ on sys.path rather than the repo root. CI never hit this since action.yml sets PYTHONPATH. Insert the repo root explicitly so the documented invocation works from any working directory. Verified by running it from /tmp against #5: the pipeline completes and Layer 3 reaches the Gemini endpoint, returning "Missing or invalid Authorization header" locally where no key is set — which confirms base_url and request shape are correct. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Gemini 2.5 was shut down ahead of its documented Oct 16 2026 date, so gemini-2.5-flash now returns: 404 models/gemini-2.5-flash is not found for API version v1beta Two consequences beyond the ID change: - Gemini 3.x has no off switch for thinking. `minimal` is the floor and only "matches the no-thinking setting for most queries". flash-lite defaults to minimal, whereas Gemini 3 Flash defaults to high, which risks a thought trace eating the 2048-token output budget and truncating the JSON verdict. Default to gemini-3.1-flash-lite with reasoning-effort minimal. - Gemini 3.x documents temperature 1.0 as strongly recommended and warns that lowering it causes looping and degraded reasoning. Layer 3 hardcoded 0.1, so evaluating Gemini 3 with it would have produced a false negative on capability. Temperature is now the ai-temperature input, still defaulting to 0.1 for OpenAI-style models. Dogfood and the local harness both target gemini-3.1-flash-lite with minimal thinking and temperature 1.0. README documents how to list the models a key can actually reach, since a retired ID's 404 reads like a URL bug. 432 tests pass, ruff clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…inimal Thinking was pinned to minimal to protect a hardcoded 2048-token output cap. That is backwards: the published intelligence scores for Gemini 3.x are reasoning-enabled (gemini-3.1-flash-lite scores 25 on the Artificial Analysis Intelligence Index against 15 for the gpt-4.1-mini baseline this prompt was tuned on), and Google recommends thinking for code reasoning — which is exactly what Layer 3 does. Flooring the effort discards the capability the model is chosen for. Move the cap instead: - max_tokens becomes the ai-max-tokens input, default 4096. It is a cap rather than a reservation, so unused headroom costs nothing, and it keeps a thought trace from squeezing out the JSON verdict (truncated JSON parses as SKIP at confidence 0.0). - dogfood and the local harness move from minimal to low thinking. - Gemini 3.x permits up to 65536 output tokens, so the model is not the binding constraint; free-tier tokens/minute is. 434 tests pass, ruff clean on src and tests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

4096 was picked without checking how thought tokens interact with the output cap. They are charged against it: when thinking exhausts the budget the API returns finish_reason="length" with empty content, which _parse_ai_response reads as SKIP at confidence 0.0 — indistinguishable from a model that had nothing to say. 8192 is the reported floor that avoids this with thinking enabled. The cap stays well below the 65536 Gemini 3.x allows on purpose: thought tokens are billed as output and burn free-tier tokens/minute, so an unbounded cap turns a runaway thought trace into quota exhaustion. The cap is not a reservation, so headroom is free when unused — the trade is between bounded worst-case spend and silent truncation, and 8192 buys both. Also emit a ::warning:: naming the cap and the two levers when a request truncates before producing a verdict, so the degradation is visible. 436 tests pass, ruff clean on src and tests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Two runs in a row reported "All files resolved by deterministic shortcuts": at coverage-threshold 100 with a full coverage report, Gate 3 (>= threshold) or Gate 5 (below, tests exist) decides every file, so the AI phase is structurally unreachable and the dogfood cannot validate the provider migration it was pointed at. Withhold the coverage report when a PR carries the `dogfood-ai` label, so files fall through to the AI gates (6-8). `labeled` joins the trigger types so adding the label starts a run without a push. Unlabelled PRs keep the strict coverage gate unchanged. Also cover the config fallbacks that the gate itself flagged: config.py sat at 79% because the invalid-ai-temperature and invalid-ai-max-tokens except branches had no tests. 439 tests pass, ruff clean on src and tests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The conditional coverage-file could never withhold coverage: ${{ contains(...) && '' || 'coverage.xml' }} In GitHub expressions the empty string is falsy, so when the condition is true the && yields '' and the || short-circuits straight to 'coverage.xml'. A ternary cannot produce an empty value. The labeled run fired correctly; the expression was incapable of acting on it. Gate two whole steps on the label instead, so the AI-path step omits coverage-file entirely. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…limit _INPUT_TOKEN_LIMIT was not dead code — it derives the user-prompt budget that drives batching and decides how much diff evidence reaches the model — but its value of 8000 was sized for the retired GitHub Models 8K input cap. At that budget a large test diff is shed, and since the prompt tells the model to treat omitted code as untested, the result is a false warning: exactly what happened to src/layer3_ai.py, whose test diff is 22,952 chars against a 6,120-token (~18k char) budget. - new ai-input-token-limit input, default 8000, threaded through run_layer3 and _build_ai_prompt via _user_prompt_budget() - dogfood and the local harness raise it to 32000 for Gemini's 1M context - the default stays conservative because providers meter tokens per minute; Groq's free tier allows only 8K/min Raising the token budget alone was not enough. A second, independent truncation clips each diff at 10k characters first, so the per-diff cap is now derived from the budget (half of it, 10k floor) and the two scale together. An explicitly passed cap still wins, which the tests caught: without that, the 413 retry's deliberate 3000-char shrink would have been inflated back up and the retry defeated. 443 tests pass, ruff clean on src and tests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

ai-max-tokens capped output while ai-input-token-limit bounded the prompt, so the pair read as if one were the input and the other the maximum. That ambiguity had already cost real work: the output cap got raised twice while the input budget was the actual cause of the false warning on src/layer3_ai.py. ai-max-tokens -> ai-max-output-tokens ai-input-token-limit -> ai-max-input-tokens Also renamed through the config fields, module constants and function parameters. Both inputs are new on this branch and unreleased, so nothing depended on the old names. The OpenAI SDK parameter is still literally max_tokens, so the call site reads max_tokens=max_output_tokens and the tests that assert on the outgoing kwarg still check "max_tokens" — only our own names changed. 443 tests pass, ruff clean on src and tests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

github-actions · 2026-07-31T23:25:58Z

🧪 Test-Guard Report

✅ PASS — All changed source files have adequate test coverage.

Coverage Analysis: ⏭️ SKIP

No coverage files provided — skipping Layer 1.

Test File Matching: ✅ PASS

File matching: 4 pass

📋 4 files: 4 ✅ pass

File	Verdict	Reason
`src/config.py`	✅ pass	Test file modified in PR: tests/test_config.py
`src/layer3_ai.py`	✅ pass	Test file modified in PR: tests/test_layer3_ai.py
`src/main.py`	✅ pass	Test file modified in PR: tests/test_main.py
`src/summary.py`	✅ pass	Test file modified in PR: tests/test_summary.py

Per-File Evaluation: ✅ PASS

Evaluated 4 files: 4 via AI (3 batches), 0 via shortcuts.

📋 4 files: 4 ✅ pass

File	Verdict	Reason
`src/config.py`	✅ pass	New AI configuration parameters are exhaustively tested for valid input, fallback behavior, and disable logic.
`src/layer3_ai.py`	✅ pass	Comprehensive test suite validates AI parameter handling, retry logic, truncation warnings, and provider integration.
`src/main.py`	✅ pass	Updated configuration parameters and added essential guard clauses for API key presence.
`src/summary.py`	✅ pass	Successfully parameterizes the OpenAI client base URL to support custom providers.

Result: ✅ PASS

Ostico and others added 12 commits July 31, 2026 23:31

Repository owner deleted a comment from github-actions Bot Jul 31, 2026

Ostico added the dogfood-ai Withhold coverage so Layer 3's AI path runs label Jul 31, 2026

This comment was marked as outdated.

Sign in to view

Ostico marked this pull request as ready for review July 31, 2026 23:27

Ostico merged commit 6b63548 into main Jul 31, 2026
3 checks passed

Ostico deleted the worktree-fix-github-models-retirement branch July 31, 2026 23:27

Navigation Menu

Uh oh!

fix(layer3): replace retired GitHub Models with any OpenAI-compatible endpoint - #5

fix(layer3): replace retired GitHub Models with any OpenAI-compatible endpoint#5
Ostico merged 16 commits into
mainfrom
worktree-fix-github-models-retirement

Ostico commented Jul 31, 2026 •

edited

Loading

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

github-actions Bot commented Jul 31, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

Ostico commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Why not stay on GitHub

Changes

Breaking change

Verification

Update: ai-reasoning-effort (default none)

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

github-actions Bot commented Jul 31, 2026

🧪 Test-Guard Report

Coverage Analysis: ⏭️ SKIP

Test File Matching: ✅ PASS

Per-File Evaluation: ✅ PASS

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Ostico commented Jul 31, 2026 •

edited

Loading

Update: `ai-reasoning-effort` (default `none`)