Repository navigation
fix(chatgpt): honest context-window metadata + usage-corrected throughput - #1
Open
cloudbouz53 wants to merge 1 commit into
Open
cloudbouz53 wants to merge 1 commit into
cloudbouz53 wants to merge 1 commit into
Conversation
…hput Context windows: - CatalogModel::context_window defaulted to 272'000, so any slug whose /models row omitted or mis-typed the field displayed "272k" as if the endpoint had reported it. Default is now 0 (unknown) — the picker shows "auto" and the compaction ladder takes its conservative default; a value the endpoint DID report is used verbatim. - Verified against the live Codex catalog (via the new codex_catalog_probe) and codex-rs 0.167.0's embedded snapshot: gpt-6.1-sol is served at 500k per the live endpoint (the client snapshot still says 272k), gpt-6.1-terra 500k, everything else 272k. The endpoint also carries a separate max_context_window (872k) agentty does not consume — documented, not invented. Throughput: - The billed output_tokens figure includes reasoning tokens; the stats panel's single "Output rate" conflated billed with watched. It now records two series: "Generated rate" (incl. reasoning, labelled) and "Visible rate" (output − reasoning). - The status-bar sparkline samples streamed bytes / 4 mid-stream (usage only arrives at stream end); finalize_turn now rewrites the ring's last sample with real tokens over real generation time, so the frozen post-stream readout is authoritative. - Zero-duration, interrupted and unmeasured turns stay excluded, never zero-filled; a turn whose reasoning tokens are not a strict subset of output_tokens records no visible sample at all rather than a negative one. Tests: metadata_throughput_test (9 cases), codex_catalog_probe (manual, live dump), updated chatgpt parse expectations, and the stats golden re-pinned for the rate split.
1ay1
pushed a commit
to agentty-org/agentty
that referenced
this pull request
Nov 12, 2026
fix(chatgpt): improve model catalog discovery
**What this changes and why:** the ChatGPT (Codex) model catalog could
come back
empty or badly incomplete — the picker falling back to a stale guess,
then the first
turn 400ing on a slug the account no longer offers. Two independent
causes: the
discovery request was gated out by the server's `client_version` check,
and a single
malformed field in any row aborted the whole payload parse. This fixes
both.
**Fixes #** — no tracked issue; found while diagnosing a reported empty
ChatGPT model
picker, and both failure mechanisms were then confirmed in code and, for
the version
gate, against the live endpoint.
**How you tested it (OS, and which tests or manual steps):** macOS
15.8.1 (Apple
Silicon, arm64). `ctest -R "chatgpt catalog|chatgpt bundled|chatgpt
codex version"` →
4/4; full `ctest -j8`; plus a live OAuth run of the real
`fetch_models()` through the
new manual-only `codex_catalog_probe`. Everything is in **Verification**
below,
including one known `rag_eval_guard` floor break, disclosed in full.
---
## What breaks today
The ChatGPT (Codex) model picker can end up with no models, or with
models that
the account no longer accepts, and then the first turn dies with
`The '…' model is not supported when using Codex with a ChatGPT
account.`
Three independent causes, all in `provider/chatgpt`:
1. **The catalog is gated on the Codex client version we impersonate,
and a stale
version gets a filtered list — served as a successful 200.** We were
advertising `0.145.0`. Measured against the live endpoint on this
account, that
version returns three `gpt-5.6-*` rows and none of the current models
(the
documented behaviour for an *unknown* version is HTTP 200 with
`{"models":[]}`,
which reads exactly like "this account has no models"). Either way the
picker
silently under-reports the account, and whatever was last selected can
be a
slug the endpoint no longer serves.
2. **One malformed field drops the whole catalog.** The parser used
`json::value()` for every field inside a single `try { … } catch (…) {
return {}; }`. A row where `context_window` arrives as a string, or
`is_default` as `null`, threw and discarded *every* model — the failure
is
per-response, not per-row, and completely silent.
3. **Server-side ordering and internal rows were taken at face value.**
Rows
are ordered by a `priority` field we ignored; `hidden` was filtered but
`supported_in_api: false` was not; slugs were not de-duplicated.
## Changes
`include/agentty/provider/chatgpt/oauth.hpp`
- `codex_client_version` → `0.167.0` (verified against the live endpoint
and the
current Codex CLI), with the gating behaviour documented at the
constant.
`src/provider/chatgpt/responses.cpp`
- The version is now overridable at runtime with
`AGENTTY_CODEX_CLIENT_VERSION`
so a future server-side gate can be worked around **without a rebuild**
—
which matters because the failure mode is "the picker goes blank", not a
diagnostic the user can act on. The override is validated strictly: it
must be
exactly `N.N.N`, ≤32 chars, and anything else (empty, `1.2`, `1..2.3`,
33
chars, or anything carrying `CR`/`LF`/`&`) falls back to the built-in
value,
because the string is copied into a URL query parameter *and* a
`user-agent`
header.
- The same validated value is sent in both places (it previously could
drift
between the query param and the header).
- Parsing moved out of the fetch into a named, directly testable
function
(`parse_models_payload_for_test`): per-row type-checked accessors (`str`
/ `boolean` /
`integer` with range clamps) instead of one global catch, non-object
rows
skipped, duplicate slugs dropped, `supported_in_api: false` dropped
alongside
`visibility: hide|hidden`, and rows sorted by `priority` with invalid or
absent
priorities ordered *after* every valid one (`stable_sort`, so server
order still
breaks ties). A malformed row now costs one row, not the catalog.
- `max_body_bytes` 1 MiB → 2 MiB: the raw `/models` payload carries a
lot of
per-model metadata per row and the old ceiling left little headroom.
`tests/chatgpt_bundled_models_test.cpp`
- The parse test feeds 10 rows and asserts exactly **4 survivors** in
priority
order. The 6 dropped rows cover every reject path in the parser:
`visibility:
"hide"`, `visibility: "hidden"`, `supported_in_api: false`, a non-string
`slug`
(`{"slug":42}`), a bare `null` row, and a duplicate slug (first
occurrence
wins). The survivors pin the fallbacks: a row carrying only `id` gets
both its
slug and its `display_name` from `id`, and its out-of-range
`context_window: -1` falls back to the struct default (272 000 on this
PR —
cloudbouz53#1 replaces that fabrication with an honest `0`)
instead of
being trusted;
`is_default: true` still marks the default. Degenerate payloads — `not
json`,
`{"models":{}}`, and a lone `{"slug":"bad","priority":1e1000}` — must
parse to
an empty list, not throw.
- A second case pins the ordering rule the server actually exercises:
valid
priorities (`1 … 1 000 000`) sort before every absent, unparseable
(`"priority": "1"`), or out-of-range (`1000001`, `18446744073709551615`)
priority, and the fallback group keeps server order — that is
`stable_sort`
under test, asserted by `first` and `equal` both carrying priority `1`.
- Pins the version override: `1.2.3` accepted, `""`, `1.2`, `1..2.3`,
`1.2.3\r\nX:bad`, `1.2.3&foo=bar` and a 33-char value all rejected.
- Keeps the environment-adaptive guard: the retired `gpt-5.1-codex` slug
and
`codex-auto-review` must never appear, and a model must always be
selectable —
from the live catalog when signed in, from the bundled fallback
otherwise.
## Verification
- `ctest -R "chatgpt catalog|chatgpt bundled|chatgpt codex version"`:
4/4 pass.
- Full suite (`ctest -j8`, 1941 tests): every failure is reproduced on
the base
commit `4ef0d8f1` with identical messages (24 × sanitizer/static-gated
binaries not
produced by a plain build; `ui_snapshots`, `palette_render_probe`,
`elm_purity`,
`command palette`) — apart from `rag_eval_guard`, documented below.
- **What CI will and will not tell you here.**
`.github/workflows/ci.yml` builds
`--target agentty` on linux-gcc, macos-15-gcc, MSVC and msys2, and its
"Smoke" step is
`./build/agentty --version`. It does **not** build `agentty_tests` and
does **not** run
`ctest`. So a green run on this PR proves the shipped binary still
compiles and starts
on all four platforms — it does *not* exercise the four catalog tests
added here, and it
will never flag `rag_eval_guard`. Those numbers came from local runs on
macOS 15.8.1
(arm64); `ctest -R "chatgpt catalog|chatgpt bundled|chatgpt codex
version"` and
`ctest -R "^rag_eval_guard$` are the two commands to re-run if you want
the evidence.
- **Base has moved since this was verified.** This branch is based on
`4ef0d8f1`;
upstream `master` is now `ecc9fbe9` (4 commits ahead: README trim,
`v0.9.54`,
changelog, and *"ci green: four stale tests, a view env read, arch
partial
upgrade"*). None of those commits touch any file in this PR, so the
branch applies
cleanly and the diff is exactly this PR's content. Two things follow,
both in this
PR's favour: `ecc9fbe9` already repairs several of the pre-existing
failures above
(including the `command palette` `kAll` list), and it advances the
`rag-cpp` / `maya`
submodule pins — so the `rag_eval_guard` figure below is a `4ef0d8f1`
measurement
against those pins; re-running it on the current base is the open
question. Submodule
pins were deliberately left untouched during verification.
- **Live A/B of the version gate, same account, same binary, seconds
apart:**
| `client_version` | rows returned |
|---|---|
| `0.145.0` (pre-PR) | 3 — `gpt-5.6-sol`, `gpt-5.6-terra`,
`gpt-5.6-luna` |
| `0.167.0` (this PR) | 8 — adds `gpt-6.1-sol`, `gpt-6-astra`,
`gpt-6.1-terra`, `gpt-6-sol`, `gpt-6-luna` |
The 8-model list is priority-ordered, carries no `gpt-5.1-codex` and no
`codex-auto-review`.
- **Injection control on the override:**
`AGENTTY_CODEX_CLIENT_VERSION='1.2&x=yy'`
is rejected by validation and the request goes out with the built-in
version —
8 models returned, identical to the default. Same for `""`, `1.2`,
`1..2.3` and
an over-length value, which are covered by unit assertions too.
- A live turn on a catalog model completes normally
(`-m gpt-6.1-terra`, `--provider chatgpt`).
- ⚠️ The `chatgpt bundled model catalog` doctest always exercises the
**offline**
branch, on `master` as on this branch: `tests/test_main.cpp` repoints
`AGENTTY_HOME` at a throwaway directory for the whole `agentty_tests`
binary, so
credentials are never visible under ctest and `responses_available()` is
false.
The doctest therefore guards the offline floor; the live path has to be
proven
out of harness. cloudbouz53#1 adds `codex_catalog_probe`, a
manual-only (`NO_TEST`)
tool that dumps the live catalog — its output above was produced with
it, and it
runs this PR's discovery code unchanged.
## Notes for reviewers
- **`rag_eval_guard` regression, disclosed in full — and it is not
flaky, not
environmental, and not a tuning accident.** On this branch the
`agentty_code_heldout` set scores MRR **0.666** against its required
threshold of
**0.670**, while **hit@8 is unchanged at 0.950** (floor 0.90).
Reproduced on every
run, including this one: the **same binary** was run against two
pristine worktrees
differing only by this PR's tracked content — base `4ef0d8f1` scores
**0.670**
(clears the floor by exactly zero), this branch scores **0.666**. The
other three
eval sets are byte-identical between the two runs (0.917/0.707,
0.967/0.744,
1.000/0.887), so the one moved number is the whole effect.
The two misses are heldout queries whose gold file is
`src/provider/selection.cpp`,
which this PR does not touch. The mechanism is corpus size, not
retrieval quality:
the guard indexes whatever the checkout contains, and this PR adds ~100
lines (new
test cases, comments, the ordering path). Each added line is another
BM25 chunk,
shifting document frequency and average document length corpus-wide,
which is enough
to move one result one slot. The eval set
(`tests/rag_eval/agentty_code_heldout.json`) is **not** touched by this
PR. Any
branch that adds indexed lines trips this floor, because base sits
exactly on it.
Nothing here changes retrieval code, so I have deliberately **not**
lowered the floor
inside this PR — happy to, in a separate commit with its own rationale,
if you'd
rather see CI green. If maintainers want the floor to stop firing on
unrelated PRs,
the two obvious options are scoring the eval on a pinned corpus, or
expressing the
floor relative to the base run.
- No API or behaviour changes outside `provider/chatgpt`; nothing
user-visible
except a picker that stays populated.
Co-Authored-By: agentty <noreply@agentty.org>
Co-authored-by: Mac <mac@MacBook-Air.local>
Co-authored-by: agentty (gpt-6.1-terra) <noreply@agentty.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fix(chatgpt): honest context-window metadata + usage-corrected throughput
What this changes and why: two numbers agentty reported with more confidence
than it had. A context window the endpoint never reported was displayed as a precise
272k; and one "Output rate" mixed billed reasoning tokens with what the useractually watched. Context windows are now
unknownunless the server said otherwise,and throughput is split into Generated vs Visible with the frozen post-stream
sample rewritten from real usage.
Fixes # — no tracked issue; surfaced while verifying agentty-org#124's catalog fields
against the live endpoint, which is where the fabricated 272k became visible.
How you tested it (OS, and which tests or manual steps): macOS 15.8.1 (Apple
Silicon, arm64). Focused set 27/27; full
ctest -j8(1951 tests, 28 failures, alltriaged); live signed-in turns for the catalog, the
context_window: 500000metadata and a real mid-stream cancellation. See §4 Verification and the
disclosed
rag_eval_guardfloor break.Stacked on agentty-org#124 (this branch is that PR plus one commit;
src/provider/chatgpt/is untouched here, so agentty-org#124 can be reviewed on its own).
1. A fabricated context window
CatalogModel::context_windowdefaulted to 272 000 — the size of the oldbundled fallback — so any
/modelsrow that omitted the field, sent it as astring, sent
null, or sent a value outsideint32was displayed as 272k as ifthe endpoint had reported it, and drove the compaction ladder with that number.
0now means unknown: the picker rendersauto, the context ladder falls backconservatively, and a window the endpoint did report is used verbatim. Parsing
also clamps to a plausible range (
1 … 20 000 000) instead of trusting whateverarrives.
Measured against the live Codex endpoint (new
codex_catalog_probe, manual-only):context_windowservedgpt-6.1-solgpt-6.1-terragpt-6-astra,gpt-6-sol,gpt-6-luna,gpt-5.6-{sol,terra,luna}The 500k rows are the point: the client-side catalog snapshot still carries 272k
for the current Sol, so an hardcoded table would have been wrong in the opposite
direction. The endpoint also carries a separate
max_context_windowfield whichagentty deliberately does not consume — documented at the parse site, not
invented into the UI.
A live turn confirms the plumbing end to end:
-m gpt-6.1-terrareportsusage.context_window == 500000inagentty run --json, i.e. the gauge'sdenominator now comes from the server, not from a constant.
2. Throughput that conflated billed with watched
The billed
output_tokensfrom the Responses API includes reasoning tokens.The stats panel had one "Output rate", so a reasoning model's "tokens per second"
reported how fast we were billed, not how fast anything appeared on screen — and
the status-bar sparkline, which samples
streamed_bytes / 4mid-stream becauseusage only arrives at the end, kept a byte estimate as the permanent record.
Now:
TurnStatsrecords two series: Generated rate (billed output, reasoningincluded, and labelled as such) and Visible rate (
output − reasoning).visible_rate_seriesis the new field beside the existing billedrate_series; the two labelled rows are produced instats/extract.cpp.TurnFactscarries the new field through projection andmerge, so switching between thread and consolidated views keeps both.
agentty export's thread table is unchanged.finalize_turn()rewrites the sparkline's last sample withreal tokens ÷ real generation time— but only inside the Model's lock, and onlywhen the turn actually measured something. A zero-duration turn, a turn with no
tokens, an interrupted stream, or a stream that never even started
(
stream_ms == 0) leaves the estimate alone instead of being zero-filled into awindow that is already displaying a number. The rewrite is guarded on the
owned-Model accessor being available, so a second window's finalize can never
scribble on the first one's history.
output_tokens— thedouble-counting wire bug — cannot produce a negative visible rate: the visible
sample is only folded in when
reasoning_tokens < output_tokens, so that turnsimply records no visible sample. Tested explicitly
(
throughput: a provider double-counting reasoning cannot go negative).3. Changes
include/agentty/domain/stats/facts.hppvisible_rate_seriesbeside the billedrate_series, carried throughserialize/from_json/merge; comment now separates the usage-derived rate from the byte/4 sparkline estimatesrc/domain/stats/extract.cppsrc/domain/stats/fold.cppsrc/runtime/app/update/stream.cppfinalize_turn()authoritative last-sample rewrite, guardedsrc/runtime/view/status_bar/token_stream_sparkline.cppfinalize_turnlands the authoritative valueinclude/agentty/provider/chatgpt/responses.hppcontext_windowdefault → 0tests/metadata_throughput_test.cpptests/codex_catalog_probe.cppNO_TEST, never in CI)tests/stats_golden_test.cppTests added (9): unknown windows stay unknown on the wire; unknown windows render
as unknown rather than invented; the context ladder falls back conservatively for
an unknown ChatGPT window; generated vs visible rate split with reasoning; a
non-thinking turn yields exactly one rate; double-counted reasoning cannot go
negative; zero-duration and token-less turns fold nothing; unmeasured turns are
excluded, not zeroed; merge carries the visible series.
4. Verification
ctest -R "chatgpt catalog|chatgpt bundled|chatgpt codex version|throughput:|context ladder|stats_golden|stats panel|gauge: Responses"→27/27 pass in 21 s.
command paletteis deliberately not in this PR's scope and not in that set:on this branch
tests/command_palette_test.cppis byte-identical to the basecommit, so it fails with exactly the base failure
(
kCommands has exactly one row per Command enum value) — verified, not assumed.It is repaired upstream by
ecc9fbe9, so nothing here needs to carry it.ctest -j8, 1951 tests, 28 failures, 386 s): 27 of those 28 arereproduced on the base commit
4ef0d8f1with identical messages (24 ×sanitizer/static-gated binaries not produced by a plain build;
ui_snapshots,palette_render_probe,elm_purity—elm_purityisdiff_review.cpp:376readinggetenv("LINES"), untouched here — andcommand palette). The 28th israg_eval_guard, below. This branch is therefore not better than base: it isexactly base's failures plus the one corpus-perturbation note. (Earlier drafts of
this PR claimed 27 failures because they carried a drive-by palette-test repair;
that is out of scope here and upstream has since landed the same repair itself.)
-m gpt-6.1-terraturn reportingcontext_window: 500000; and a realmid-stream cancellation (
http.stream_hangup … stream reset (8)) that completednormally with usage intact and no zero-filled rate sample.
rests on an unrun test):
agentty_testsrepointsAGENTTY_HOMEat a throwawaydirectory for the whole binary, so every credential-gated doctest takes its offline
path in CI and in
ctest.metadata_throughput_testis deliberately credential-freeand deterministic, so it runs under plain
ctestwith no account present; but notethe separate point below that CI does not run
ctestat all, so its realexecution here was local. Everything about the wire — the reported windows and the
usage-corrected rates — was verified separately, live and signed in, via the
manual-only
codex_catalog_probeand real turns..github/workflows/ci.ymlbuilds--target agenttyon linux-gcc, macos-15-gcc, MSVC and msys2, and its "Smoke" step is./build/agentty --version. It never buildsagentty_testsand never runsctest, soCI cannot compile
metadata_throughput_test.cpporcodex_catalog_probe.cpp, cannotcheck the re-pinned stats golden, and will not flag
rag_eval_guard. A green run hereonly proves the shipped binary still compiles and starts on four platforms. The 27/27
focused result above is from a local build on macOS 15.8.1 (arm64); to reproduce: build
the
agentty_teststarget explicitly (cmake --build build --target agentty_tests— aplain
cmake --build buildleaves it stale) and runctest -R "throughput:|context ladder|chatgpt catalog|stats_golden".4ef0d8f1;upstream
masteris nowecc9fbe9(4 commits ahead, including "ci green: fourstale tests, a view env read, arch partial upgrade"). None of them touch a file in
this PR or in fix(chatgpt): improve model catalog discovery agentty-org/agentty#124, so both branches apply cleanly.
ecc9fbe9lands the samecommand_palette_test.cppkAllrepair that this branch deliberately drops (seeabove) — carrying it here would only have conflicted with it. It also advances the
rag-cpp/mayasubmodule pins, so therag_eval_guardfigure below is a4ef0d8f1measurement against those pins; re-running it on the current base is theopen question. Submodule pins were deliberately left untouched during verification.
rag_eval_guardregression, disclosed in full — deterministic, not flaky andnot environmental.
agentty_code_heldoutscores MRR 0.666 against its requiredthreshold of 0.670; hit@8 is unchanged at 0.950 (floor 0.90) and the other
three sets pass. It reproduces on this branch and on fix(chatgpt): improve model catalog discovery agentty-org/agentty#124 with identical numbers.
Measured as a controlled A/B: the same binary run against two pristine worktrees
differing only by tracked content — base
4ef0d8f1→ 0.670, this branch →0.666, with the other three sets byte-identical (0.917/0.707, 0.967/0.744,
1.000/0.887). (Run in place inside a working checkout with build artefacts present,
the branch reads 0.665 — same conclusion, still under the floor; the matched
worktree pair is the number to trust.) Master therefore clears the floor by exactly
zero, so any branch that adds indexed lines trips it. The mechanism is BM25 corpus
statistics (this PR adds ~330 indexed lines), not retrieval code: neither branch
touches RAG, and
tests/rag_eval/agentty_code_heldout.jsonis unmodified. The flooris not lowered anywhere in this PR — see fix(chatgpt): improve model catalog discovery agentty-org/agentty#124 for the suggested fixes.
Co-Authored-By: agentty noreply@agentty.org