Skip to content

fix(chatgpt): improve model catalog discovery - #124

Merged
1ay1 merged 1 commit into
agentty-org:masterfrom
cloudbouz53:fix/chatgpt-model-discovery
Nov 12, 2026
Merged

1ay1 merged 1 commit into
agentty-org:masterfrom
cloudbouz53:fix/chatgpt-model-discovery

Conversation

@cloudbouz53

@cloudbouz53 cloudbouz53 commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

fix(chatgpt): improve model catalog discovery

What this changes and why: the ChatGPT (Codex) model catalog could come back
empty or badly incomplete — the picker falling back to a stale guess, then the first
turn 400ing on a slug the account no longer offers. Two independent causes: the
discovery request was gated out by the server's client_version check, and a single
malformed field in any row aborted the whole payload parse. This fixes both.

Fixes # — no tracked issue; found while diagnosing a reported empty ChatGPT model
picker, and both failure mechanisms were then confirmed in code and, for the version
gate, against the live endpoint.

How you tested it (OS, and which tests or manual steps): macOS 15.8.1 (Apple
Silicon, arm64). ctest -R "chatgpt catalog|chatgpt bundled|chatgpt codex version" →
4/4; full ctest -j8; plus a live OAuth run of the real fetch_models() through the
new manual-only codex_catalog_probe. Everything is in Verification below,
including one known rag_eval_guard floor break, disclosed in full.


What breaks today

The ChatGPT (Codex) model picker can end up with no models, or with models that
the account no longer accepts, and then the first turn dies with
The '…' model is not supported when using Codex with a ChatGPT account.
Three independent causes, all in provider/chatgpt:

  1. The catalog is gated on the Codex client version we impersonate, and a stale
    version gets a filtered list — served as a successful 200.
    We were
    advertising 0.145.0. Measured against the live endpoint on this account, that
    version returns three gpt-5.6-* rows and none of the current models (the
    documented behaviour for an unknown version is HTTP 200 with {"models":[]},
    which reads exactly like "this account has no models"). Either way the picker
    silently under-reports the account, and whatever was last selected can be a
    slug the endpoint no longer serves.
  2. One malformed field drops the whole catalog. The parser used
    json::value() for every field inside a single try { … } catch (…) { return {}; }. A row where context_window arrives as a string, or
    is_default as null, threw and discarded every model — the failure is
    per-response, not per-row, and completely silent.
  3. Server-side ordering and internal rows were taken at face value. Rows
    are ordered by a priority field we ignored; hidden was filtered but
    supported_in_api: false was not; slugs were not de-duplicated.

Changes

include/agentty/provider/chatgpt/oauth.hpp

  • codex_client_version → 0.167.0 (verified against the live endpoint and the
    current Codex CLI), with the gating behaviour documented at the constant.

src/provider/chatgpt/responses.cpp

  • The version is now overridable at runtime with AGENTTY_CODEX_CLIENT_VERSION
    so a future server-side gate can be worked around without a rebuild —
    which matters because the failure mode is "the picker goes blank", not a
    diagnostic the user can act on. The override is validated strictly: it must be
    exactly N.N.N, ≤32 chars, and anything else (empty, 1.2, 1..2.3, 33
    chars, or anything carrying CR/LF/&) falls back to the built-in value,
    because the string is copied into a URL query parameter and a user-agent
    header.
  • The same validated value is sent in both places (it previously could drift
    between the query param and the header).
  • Parsing moved out of the fetch into a named, directly testable function
    (parse_models_payload_for_test): per-row type-checked accessors (str / boolean /
    integer with range clamps) instead of one global catch, non-object rows
    skipped, duplicate slugs dropped, supported_in_api: false dropped alongside
    visibility: hide|hidden, and rows sorted by priority with invalid or absent
    priorities ordered after every valid one (stable_sort, so server order still
    breaks ties). A malformed row now costs one row, not the catalog.
  • max_body_bytes 1 MiB → 2 MiB: the raw /models payload carries a lot of
    per-model metadata per row and the old ceiling left little headroom.

tests/chatgpt_bundled_models_test.cpp

  • The parse test feeds 10 rows and asserts exactly 4 survivors in priority
    order. The 6 dropped rows cover every reject path in the parser: visibility: "hide", visibility: "hidden", supported_in_api: false, a non-string slug
    ({"slug":42}), a bare null row, and a duplicate slug (first occurrence
    wins). The survivors pin the fallbacks: a row carrying only id gets both its
    slug and its display_name from id, and its out-of-range
    context_window: -1 falls back to the struct default (272 000 on this PR —
    fix(chatgpt): honest context-window metadata + usage-corrected throughput cloudbouz53/agentty#1 replaces that fabrication with an honest 0) instead of
    being trusted;
    is_default: true still marks the default. Degenerate payloads — not json,
    {"models":{}}, and a lone {"slug":"bad","priority":1e1000} — must parse to
    an empty list, not throw.
  • A second case pins the ordering rule the server actually exercises: valid
    priorities (1 … 1 000 000) sort before every absent, unparseable
    ("priority": "1"), or out-of-range (1000001, 18446744073709551615)
    priority, and the fallback group keeps server order — that is stable_sort
    under test, asserted by first and equal both carrying priority 1.
  • Pins the version override: 1.2.3 accepted, "", 1.2, 1..2.3,
    1.2.3\r\nX:bad, 1.2.3&foo=bar and a 33-char value all rejected.
  • Keeps the environment-adaptive guard: the retired gpt-5.1-codex slug and
    codex-auto-review must never appear, and a model must always be selectable —
    from the live catalog when signed in, from the bundled fallback otherwise.

Verification

  • ctest -R "chatgpt catalog|chatgpt bundled|chatgpt codex version": 4/4 pass.

  • Full suite (ctest -j8, 1941 tests): every failure is reproduced on the base
    commit 4ef0d8f1 with identical messages (24 × sanitizer/static-gated binaries not
    produced by a plain build; ui_snapshots, palette_render_probe, elm_purity,
    command palette) — apart from rag_eval_guard, documented below.

  • What CI will and will not tell you here. .github/workflows/ci.yml builds
    --target agentty on linux-gcc, macos-15-gcc, MSVC and msys2, and its "Smoke" step is
    ./build/agentty --version. It does not build agentty_tests and does not run
    ctest. So a green run on this PR proves the shipped binary still compiles and starts
    on all four platforms — it does not exercise the four catalog tests added here, and it
    will never flag rag_eval_guard. Those numbers came from local runs on macOS 15.8.1
    (arm64); ctest -R "chatgpt catalog|chatgpt bundled|chatgpt codex version" and
    ctest -R "^rag_eval_guard$ are the two commands to re-run if you want the evidence.

  • Base has moved since this was verified. This branch is based on 4ef0d8f1;
    upstream master is now ecc9fbe9 (4 commits ahead: README trim, v0.9.54,
    changelog, and "ci green: four stale tests, a view env read, arch partial
    upgrade"
    ). None of those commits touch any file in this PR, so the branch applies
    cleanly and the diff is exactly this PR's content. Two things follow, both in this
    PR's favour: ecc9fbe9 already repairs several of the pre-existing failures above
    (including the command palette kAll list), and it advances the rag-cpp / maya
    submodule pins — so the rag_eval_guard figure below is a 4ef0d8f1 measurement
    against those pins; re-running it on the current base is the open question. Submodule
    pins were deliberately left untouched during verification.

  • Live A/B of the version gate, same account, same binary, seconds apart:

    client_version rows returned
    0.145.0 (pre-PR) 3 — gpt-5.6-sol, gpt-5.6-terra, gpt-5.6-luna
    0.167.0 (this PR) 8 — adds gpt-6.1-sol, gpt-6-astra, gpt-6.1-terra, gpt-6-sol, gpt-6-luna

    The 8-model list is priority-ordered, carries no gpt-5.1-codex and no
    codex-auto-review.

  • Injection control on the override: AGENTTY_CODEX_CLIENT_VERSION='1.2&x=yy'
    is rejected by validation and the request goes out with the built-in version —
    8 models returned, identical to the default. Same for "", 1.2, 1..2.3 and
    an over-length value, which are covered by unit assertions too.

  • A live turn on a catalog model completes normally
    (-m gpt-6.1-terra, --provider chatgpt).

  • ⚠️ The chatgpt bundled model catalog doctest always exercises the offline
    branch, on master as on this branch: tests/test_main.cpp repoints
    AGENTTY_HOME at a throwaway directory for the whole agentty_tests binary, so
    credentials are never visible under ctest and responses_available() is false.
    The doctest therefore guards the offline floor; the live path has to be proven
    out of harness. fix(chatgpt): honest context-window metadata + usage-corrected throughput cloudbouz53/agentty#1 adds codex_catalog_probe, a manual-only (NO_TEST)
    tool that dumps the live catalog — its output above was produced with it, and it
    runs this PR's discovery code unchanged.

Notes for reviewers

  • rag_eval_guard regression, disclosed in full — and it is not flaky, not
    environmental, and not a tuning accident.
    On this branch the
    agentty_code_heldout set scores MRR 0.666 against its required threshold of
    0.670, while hit@8 is unchanged at 0.950 (floor 0.90). Reproduced on every
    run, including this one: the same binary was run against two pristine worktrees
    differing only by this PR's tracked content — base 4ef0d8f1 scores 0.670
    (clears the floor by exactly zero), this branch scores 0.666. The other three
    eval sets are byte-identical between the two runs (0.917/0.707, 0.967/0.744,
    1.000/0.887), so the one moved number is the whole effect.
    The two misses are heldout queries whose gold file is src/provider/selection.cpp,
    which this PR does not touch. The mechanism is corpus size, not retrieval quality:
    the guard indexes whatever the checkout contains, and this PR adds ~100 lines (new
    test cases, comments, the ordering path). Each added line is another BM25 chunk,
    shifting document frequency and average document length corpus-wide, which is enough
    to move one result one slot. The eval set
    (tests/rag_eval/agentty_code_heldout.json) is not touched by this PR. Any
    branch that adds indexed lines trips this floor, because base sits exactly on it.
    Nothing here changes retrieval code, so I have deliberately not lowered the floor
    inside this PR — happy to, in a separate commit with its own rationale, if you'd
    rather see CI green. If maintainers want the floor to stop firing on unrelated PRs,
    the two obvious options are scoring the eval on a pinned corpus, or expressing the
    floor relative to the base run.
  • No API or behaviour changes outside provider/chatgpt; nothing user-visible
    except a picker that stays populated.

Co-Authored-By: agentty noreply@agentty.org

Co-authored-by: agentty (gpt-6.1-terra) <noreply@agentty.org>
@cloudbouz53

Copy link
Copy Markdown
Contributor Author

Stacked follow-up is now up: cloudbouz53#1 — fix(chatgpt): honest context-window metadata + usage-corrected throughput. GitHub won't accept a base branch from another repository, so it is stacked inside my fork (fix/chatgpt-model-discovery → fix/chatgpt-metadata-throughput-scoped) to avoid pasting this PR's diff into it a second time. Its diff is exactly one commit (ee70f289), which applies cleanly on top of 0d3e5ea0. Nothing here needs to change for it; this PR stays reviewable on its own.

@1ay1
1ay1 merged commit 6989ce2 into agentty-org:master Nov 12, 2026
1ay1 added a commit that referenced this pull request Nov 12, 2026
- ci: a static-checks job (configure only, ~a minute) runs the source lints
  on every push and PR. they only ran nightly, so elm_purity and friends
  sat red while pushes looked green. paste_page_sync needs the binary, so
  it moved into the linux build's smoke step.
- rag_eval_guard: agentty_code_heldout MRR drifted 0.666-0.680 across three
  trees with no retrieval change; its floor was 0.67, inside the drift, so
  any PR adding lines could trip it. floor 0.64 now.
- docs: AGENTTY_CODEX_CLIENT_VERSION in environment.md.
- a failed tool pin now says so (SetToolPinned had no failure toast), and
  two dead-code warnings are gone.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants