Repository navigation
fix(chatgpt): improve model catalog discovery - #124
Merged
Merged
Conversation
Co-authored-by: agentty (gpt-6.1-terra) <noreply@agentty.org>
Contributor
Author
|
Stacked follow-up is now up: cloudbouz53#1 — |
1ay1
added a commit
that referenced
this pull request
Nov 12, 2026
- ci: a static-checks job (configure only, ~a minute) runs the source lints on every push and PR. they only ran nightly, so elm_purity and friends sat red while pushes looked green. paste_page_sync needs the binary, so it moved into the linux build's smoke step. - rag_eval_guard: agentty_code_heldout MRR drifted 0.666-0.680 across three trees with no retrieval change; its floor was 0.67, inside the drift, so any PR adding lines could trip it. floor 0.64 now. - docs: AGENTTY_CODEX_CLIENT_VERSION in environment.md. - a failed tool pin now says so (SetToolPinned had no failure toast), and two dead-code warnings are gone.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fix(chatgpt): improve model catalog discovery
What this changes and why: the ChatGPT (Codex) model catalog could come back
empty or badly incomplete — the picker falling back to a stale guess, then the first
turn 400ing on a slug the account no longer offers. Two independent causes: the
discovery request was gated out by the server's
client_versioncheck, and a singlemalformed field in any row aborted the whole payload parse. This fixes both.
Fixes # — no tracked issue; found while diagnosing a reported empty ChatGPT model
picker, and both failure mechanisms were then confirmed in code and, for the version
gate, against the live endpoint.
How you tested it (OS, and which tests or manual steps): macOS 15.8.1 (Apple
Silicon, arm64).
ctest -R "chatgpt catalog|chatgpt bundled|chatgpt codex version"→4/4; full
ctest -j8; plus a live OAuth run of the realfetch_models()through thenew manual-only
codex_catalog_probe. Everything is in Verification below,including one known
rag_eval_guardfloor break, disclosed in full.What breaks today
The ChatGPT (Codex) model picker can end up with no models, or with models that
the account no longer accepts, and then the first turn dies with
The '…' model is not supported when using Codex with a ChatGPT account.Three independent causes, all in
provider/chatgpt:version gets a filtered list — served as a successful 200. We were
advertising
0.145.0. Measured against the live endpoint on this account, thatversion returns three
gpt-5.6-*rows and none of the current models (thedocumented behaviour for an unknown version is HTTP 200 with
{"models":[]},which reads exactly like "this account has no models"). Either way the picker
silently under-reports the account, and whatever was last selected can be a
slug the endpoint no longer serves.
json::value()for every field inside a singletry { … } catch (…) { return {}; }. A row wherecontext_windowarrives as a string, oris_defaultasnull, threw and discarded every model — the failure isper-response, not per-row, and completely silent.
are ordered by a
priorityfield we ignored;hiddenwas filtered butsupported_in_api: falsewas not; slugs were not de-duplicated.Changes
include/agentty/provider/chatgpt/oauth.hppcodex_client_version→0.167.0(verified against the live endpoint and thecurrent Codex CLI), with the gating behaviour documented at the constant.
src/provider/chatgpt/responses.cppAGENTTY_CODEX_CLIENT_VERSIONso a future server-side gate can be worked around without a rebuild —
which matters because the failure mode is "the picker goes blank", not a
diagnostic the user can act on. The override is validated strictly: it must be
exactly
N.N.N, ≤32 chars, and anything else (empty,1.2,1..2.3, 33chars, or anything carrying
CR/LF/&) falls back to the built-in value,because the string is copied into a URL query parameter and a
user-agentheader.
between the query param and the header).
(
parse_models_payload_for_test): per-row type-checked accessors (str/boolean/integerwith range clamps) instead of one global catch, non-object rowsskipped, duplicate slugs dropped,
supported_in_api: falsedropped alongsidevisibility: hide|hidden, and rows sorted byprioritywith invalid or absentpriorities ordered after every valid one (
stable_sort, so server order stillbreaks ties). A malformed row now costs one row, not the catalog.
max_body_bytes1 MiB → 2 MiB: the raw/modelspayload carries a lot ofper-model metadata per row and the old ceiling left little headroom.
tests/chatgpt_bundled_models_test.cpporder. The 6 dropped rows cover every reject path in the parser:
visibility: "hide",visibility: "hidden",supported_in_api: false, a non-stringslug(
{"slug":42}), a barenullrow, and a duplicate slug (first occurrencewins). The survivors pin the fallbacks: a row carrying only
idgets both itsslug and its
display_namefromid, and its out-of-rangecontext_window: -1falls back to the struct default (272 000 on this PR —fix(chatgpt): honest context-window metadata + usage-corrected throughput cloudbouz53/agentty#1 replaces that fabrication with an honest
0) instead ofbeing trusted;
is_default: truestill marks the default. Degenerate payloads —not json,{"models":{}}, and a lone{"slug":"bad","priority":1e1000}— must parse toan empty list, not throw.
priorities (
1 … 1 000 000) sort before every absent, unparseable(
"priority": "1"), or out-of-range (1000001,18446744073709551615)priority, and the fallback group keeps server order — that is
stable_sortunder test, asserted by
firstandequalboth carrying priority1.1.2.3accepted,"",1.2,1..2.3,1.2.3\r\nX:bad,1.2.3&foo=barand a 33-char value all rejected.gpt-5.1-codexslug andcodex-auto-reviewmust never appear, and a model must always be selectable —from the live catalog when signed in, from the bundled fallback otherwise.
Verification
ctest -R "chatgpt catalog|chatgpt bundled|chatgpt codex version": 4/4 pass.Full suite (
ctest -j8, 1941 tests): every failure is reproduced on the basecommit
4ef0d8f1with identical messages (24 × sanitizer/static-gated binaries notproduced by a plain build;
ui_snapshots,palette_render_probe,elm_purity,command palette) — apart fromrag_eval_guard, documented below.What CI will and will not tell you here.
.github/workflows/ci.ymlbuilds--target agenttyon linux-gcc, macos-15-gcc, MSVC and msys2, and its "Smoke" step is./build/agentty --version. It does not buildagentty_testsand does not runctest. So a green run on this PR proves the shipped binary still compiles and startson all four platforms — it does not exercise the four catalog tests added here, and it
will never flag
rag_eval_guard. Those numbers came from local runs on macOS 15.8.1(arm64);
ctest -R "chatgpt catalog|chatgpt bundled|chatgpt codex version"andctest -R "^rag_eval_guard$are the two commands to re-run if you want the evidence.Base has moved since this was verified. This branch is based on
4ef0d8f1;upstream
masteris nowecc9fbe9(4 commits ahead: README trim,v0.9.54,changelog, and "ci green: four stale tests, a view env read, arch partial
upgrade"). None of those commits touch any file in this PR, so the branch applies
cleanly and the diff is exactly this PR's content. Two things follow, both in this
PR's favour:
ecc9fbe9already repairs several of the pre-existing failures above(including the
command palettekAlllist), and it advances therag-cpp/mayasubmodule pins — so the
rag_eval_guardfigure below is a4ef0d8f1measurementagainst those pins; re-running it on the current base is the open question. Submodule
pins were deliberately left untouched during verification.
Live A/B of the version gate, same account, same binary, seconds apart:
client_version0.145.0(pre-PR)gpt-5.6-sol,gpt-5.6-terra,gpt-5.6-luna0.167.0(this PR)gpt-6.1-sol,gpt-6-astra,gpt-6.1-terra,gpt-6-sol,gpt-6-lunaThe 8-model list is priority-ordered, carries no
gpt-5.1-codexand nocodex-auto-review.Injection control on the override:
AGENTTY_CODEX_CLIENT_VERSION='1.2&x=yy'is rejected by validation and the request goes out with the built-in version —
8 models returned, identical to the default. Same for
"",1.2,1..2.3andan over-length value, which are covered by unit assertions too.
A live turn on a catalog model completes normally
(
-m gpt-6.1-terra,--provider chatgpt).chatgpt bundled model catalogdoctest always exercises the offlinebranch, on
masteras on this branch:tests/test_main.cpprepointsAGENTTY_HOMEat a throwaway directory for the wholeagentty_testsbinary, socredentials are never visible under ctest and
responses_available()is false.The doctest therefore guards the offline floor; the live path has to be proven
out of harness. fix(chatgpt): honest context-window metadata + usage-corrected throughput cloudbouz53/agentty#1 adds
codex_catalog_probe, a manual-only (NO_TEST)tool that dumps the live catalog — its output above was produced with it, and it
runs this PR's discovery code unchanged.
Notes for reviewers
rag_eval_guardregression, disclosed in full — and it is not flaky, notenvironmental, and not a tuning accident. On this branch the
agentty_code_heldoutset scores MRR 0.666 against its required threshold of0.670, while hit@8 is unchanged at 0.950 (floor 0.90). Reproduced on every
run, including this one: the same binary was run against two pristine worktrees
differing only by this PR's tracked content — base
4ef0d8f1scores 0.670(clears the floor by exactly zero), this branch scores 0.666. The other three
eval sets are byte-identical between the two runs (0.917/0.707, 0.967/0.744,
1.000/0.887), so the one moved number is the whole effect.
The two misses are heldout queries whose gold file is
src/provider/selection.cpp,which this PR does not touch. The mechanism is corpus size, not retrieval quality:
the guard indexes whatever the checkout contains, and this PR adds ~100 lines (new
test cases, comments, the ordering path). Each added line is another BM25 chunk,
shifting document frequency and average document length corpus-wide, which is enough
to move one result one slot. The eval set
(
tests/rag_eval/agentty_code_heldout.json) is not touched by this PR. Anybranch that adds indexed lines trips this floor, because base sits exactly on it.
Nothing here changes retrieval code, so I have deliberately not lowered the floor
inside this PR — happy to, in a separate commit with its own rationale, if you'd
rather see CI green. If maintainers want the floor to stop firing on unrelated PRs,
the two obvious options are scoring the eval on a pinned corpus, or expressing the
floor relative to the base run.
provider/chatgpt; nothing user-visibleexcept a picker that stays populated.
Co-Authored-By: agentty noreply@agentty.org