Skip to content

fix(chatgpt): honest context-window metadata + usage-corrected throughput - #1

Open
cloudbouz53 wants to merge 1 commit into
fix/chatgpt-model-discoveryfrom
fix/chatgpt-metadata-throughput-scoped
Open

cloudbouz53 wants to merge 1 commit into
fix/chatgpt-model-discoveryfrom
fix/chatgpt-metadata-throughput-scoped

Conversation

@cloudbouz53

@cloudbouz53 cloudbouz53 commented Nov 12, 2026 •

Copy link
Copy Markdown
Owner

fix(chatgpt): honest context-window metadata + usage-corrected throughput

What this changes and why: two numbers agentty reported with more confidence
than it had. A context window the endpoint never reported was displayed as a precise
272k; and one "Output rate" mixed billed reasoning tokens with what the user
actually watched. Context windows are now unknown unless the server said otherwise,
and throughput is split into Generated vs Visible with the frozen post-stream
sample rewritten from real usage.

Fixes # — no tracked issue; surfaced while verifying agentty-org#124's catalog fields
against the live endpoint, which is where the fabricated 272k became visible.

How you tested it (OS, and which tests or manual steps): macOS 15.8.1 (Apple
Silicon, arm64). Focused set 27/27; full ctest -j8 (1951 tests, 28 failures, all
triaged); live signed-in turns for the catalog, the context_window: 500000
metadata and a real mid-stream cancellation. See §4 Verification and the
disclosed rag_eval_guard floor break.


Stacked on agentty-org#124 (this branch is that PR plus one commit; src/provider/chatgpt/
is untouched here, so agentty-org#124 can be reviewed on its own).

Why this pull request is opened inside the fork rather than against upstream.
GitHub does not accept a base branch from a different repository, so
fix/chatgpt-model-discovery (agentty-org#124's head) cannot be used as a base inside
agentty-org/agentty. Rather than open this against upstream master — which would
have pasted all of agentty-org#124's diff into it a second time — it is stacked inside my
fork
: base cloudbouz53:fix/chatgpt-model-discovery, head
cloudbouz53:fix/chatgpt-metadata-throughput-scoped. The review unit is therefore
exactly right: one commit, 11 files, no duplicated diff.
To land it: merge agentty-org#124, then either retarget this branch at agentty-org:master or
cherry-pick ee70f289 onto it — both give the same 11-file diff as a normal upstream
PR, and ee70f289 applies cleanly on top of agentty-org#124 by construction (its parent is
0d3e5ea0).

1. A fabricated context window

CatalogModel::context_window defaulted to 272 000 — the size of the old
bundled fallback — so any /models row that omitted the field, sent it as a
string, sent null, or sent a value outside int32 was displayed as 272k as if
the endpoint had reported it
, and drove the compaction ladder with that number.

0 now means unknown: the picker renders auto, the context ladder falls back
conservatively, and a window the endpoint did report is used verbatim. Parsing
also clamps to a plausible range (1 … 20 000 000) instead of trusting whatever
arrives.

Measured against the live Codex endpoint (new codex_catalog_probe, manual-only):

slug context_window served
gpt-6.1-sol 500 000
gpt-6.1-terra 500 000
gpt-6-astra, gpt-6-sol, gpt-6-luna, gpt-5.6-{sol,terra,luna} 272 000

The 500k rows are the point: the client-side catalog snapshot still carries 272k
for the current Sol, so an hardcoded table would have been wrong in the opposite
direction. The endpoint also carries a separate max_context_window field which
agentty deliberately does not consume — documented at the parse site, not
invented into the UI.

A live turn confirms the plumbing end to end: -m gpt-6.1-terra reports
usage.context_window == 500000 in agentty run --json, i.e. the gauge's
denominator now comes from the server, not from a constant.

2. Throughput that conflated billed with watched

The billed output_tokens from the Responses API includes reasoning tokens.
The stats panel had one "Output rate", so a reasoning model's "tokens per second"
reported how fast we were billed, not how fast anything appeared on screen — and
the status-bar sparkline, which samples streamed_bytes / 4 mid-stream because
usage only arrives at the end, kept a byte estimate as the permanent record.

Now:

  • TurnStats records two series: Generated rate (billed output, reasoning
    included, and labelled as such) and Visible rate (output − reasoning).
    visible_rate_series is the new field beside the existing billed
    rate_series; the two labelled rows are produced in stats/extract.cpp.
    TurnFacts carries the new field through projection and
    merge, so switching between thread and consolidated views keeps both.
    agentty export's thread table is unchanged.
  • finalize_turn() rewrites the sparkline's last sample with
    real tokens ÷ real generation time — but only inside the Model's lock, and only
    when the turn actually measured something. A zero-duration turn, a turn with no
    tokens, an interrupted stream, or a stream that never even started
    (stream_ms == 0) leaves the estimate alone instead of being zero-filled into a
    window that is already displaying a number. The rewrite is guarded on the
    owned-Model accessor being available, so a second window's finalize can never
    scribble on the first one's history.
  • A provider that reports reasoning tokens outside output_tokens — the
    double-counting wire bug — cannot produce a negative visible rate: the visible
    sample is only folded in when reasoning_tokens < output_tokens, so that turn
    simply records no visible sample. Tested explicitly
    (throughput: a provider double-counting reasoning cannot go negative).

3. Changes

File What
include/agentty/domain/stats/facts.hpp new visible_rate_series beside the billed rate_series, carried through serialize / from_json / merge; comment now separates the usage-derived rate from the byte/4 sparkline estimate
src/domain/stats/extract.cpp render the one billed rate as two labelled rows (Generated / Visible) from billed + reasoning tokens
src/domain/stats/fold.cpp compute and carry the visible series across projections/merges
src/runtime/app/update/stream.cpp finalize_turn() authoritative last-sample rewrite, guarded
src/runtime/view/status_bar/token_stream_sparkline.cpp comment-only: documents that the mid-stream chip is an estimate and that finalize_turn lands the authoritative value
include/agentty/provider/chatgpt/responses.hpp context_window default → 0
tests/metadata_throughput_test.cpp 9 new cases (below)
tests/codex_catalog_probe.cpp manual-only live catalog dumper (NO_TEST, never in CI)
tests/stats_golden_test.cpp re-pinned for the two-row rate split

Tests added (9): unknown windows stay unknown on the wire; unknown windows render
as unknown rather than invented; the context ladder falls back conservatively for
an unknown ChatGPT window; generated vs visible rate split with reasoning; a
non-thinking turn yields exactly one rate; double-counted reasoning cannot go
negative; zero-duration and token-less turns fold nothing; unmeasured turns are
excluded, not zeroed; merge carries the visible series.

4. Verification

  • Focused: ctest -R "chatgpt catalog|chatgpt bundled|chatgpt codex version|throughput:|context ladder|stats_golden|stats panel|gauge: Responses" →
    27/27 pass in 21 s.
  • command palette is deliberately not in this PR's scope and not in that set:
    on this branch tests/command_palette_test.cpp is byte-identical to the base
    commit, so it fails with exactly the base failure
    (kCommands has exactly one row per Command enum value) — verified, not assumed.
    It is repaired upstream by ecc9fbe9, so nothing here needs to carry it.
  • Full suite (ctest -j8, 1951 tests, 28 failures, 386 s): 27 of those 28 are
    reproduced on the base commit 4ef0d8f1 with identical messages (24 ×
    sanitizer/static-gated binaries not produced by a plain build; ui_snapshots,
    palette_render_probe, elm_purity — elm_purity is diff_review.cpp:376 reading
    getenv("LINES"), untouched here — and command palette). The 28th is
    rag_eval_guard, below. This branch is therefore not better than base: it is
    exactly base's failures plus the one corpus-perturbation note. (Earlier drafts of
    this PR claimed 27 failures because they carried a drive-by palette-test repair;
    that is out of scope here and upstream has since landed the same repair itself.)
  • Live, signed-in ChatGPT account: 8-model catalog with the windows above;
    -m gpt-6.1-terra turn reporting context_window: 500000; and a real
    mid-stream cancellation (http.stream_hangup … stream reset (8)) that completed
    normally with usage intact and no zero-filled rate sample.
  • Test-discovery limitation (same as fix(chatgpt): improve model catalog discovery agentty-org/agentty#124, stated here so nothing in this PR
    rests on an unrun test):
    agentty_tests repoints AGENTTY_HOME at a throwaway
    directory for the whole binary, so every credential-gated doctest takes its offline
    path in CI and in ctest. metadata_throughput_test is deliberately credential-free
    and deterministic, so it runs under plain ctest with no account present; but note
    the separate point below that CI does not run ctest at all, so its real
    execution here was local. Everything about the wire — the reported windows and the
    usage-corrected rates — was verified separately, live and signed in, via the
    manual-only codex_catalog_probe and real turns.
  • What CI will and will not tell you here. .github/workflows/ci.yml builds
    --target agentty on linux-gcc, macos-15-gcc, MSVC and msys2, and its "Smoke" step is
    ./build/agentty --version. It never builds agentty_tests and never runs ctest, so
    CI cannot compile metadata_throughput_test.cpp or codex_catalog_probe.cpp, cannot
    check the re-pinned stats golden, and will not flag rag_eval_guard. A green run here
    only proves the shipped binary still compiles and starts on four platforms. The 27/27
    focused result above is from a local build on macOS 15.8.1 (arm64); to reproduce: build
    the agentty_tests target explicitly (cmake --build build --target agentty_tests — a
    plain cmake --build build leaves it stale) and run
    ctest -R "throughput:|context ladder|chatgpt catalog|stats_golden".
  • Base has moved since this was verified. The stack is based on 4ef0d8f1;
    upstream master is now ecc9fbe9 (4 commits ahead, including "ci green: four
    stale tests, a view env read, arch partial upgrade"
    ). None of them touch a file in
    this PR or in fix(chatgpt): improve model catalog discovery agentty-org/agentty#124, so both branches apply cleanly. ecc9fbe9 lands the same
    command_palette_test.cpp kAll repair that this branch deliberately drops (see
    above) — carrying it here would only have conflicted with it. It also advances the
    rag-cpp / maya submodule pins, so the rag_eval_guard figure below is a
    4ef0d8f1 measurement against those pins; re-running it on the current base is the
    open question. Submodule pins were deliberately left untouched during verification.
  • ⚠️ rag_eval_guard regression, disclosed in full — deterministic, not flaky and
    not environmental.
    agentty_code_heldout scores MRR 0.666 against its required
    threshold of 0.670; hit@8 is unchanged at 0.950 (floor 0.90) and the other
    three sets pass. It reproduces on this branch and on fix(chatgpt): improve model catalog discovery agentty-org/agentty#124 with identical numbers.
    Measured as a controlled A/B: the same binary run against two pristine worktrees
    differing only by tracked content — base 4ef0d8f1 → 0.670, this branch →
    0.666, with the other three sets byte-identical (0.917/0.707, 0.967/0.744,
    1.000/0.887). (Run in place inside a working checkout with build artefacts present,
    the branch reads 0.665 — same conclusion, still under the floor; the matched
    worktree pair is the number to trust.) Master therefore clears the floor by exactly
    zero, so any branch that adds indexed lines trips it. The mechanism is BM25 corpus
    statistics (this PR adds ~330 indexed lines), not retrieval code: neither branch
    touches RAG, and tests/rag_eval/agentty_code_heldout.json is unmodified. The floor
    is not lowered anywhere in this PR — see fix(chatgpt): improve model catalog discovery agentty-org/agentty#124 for the suggested fixes.

Co-Authored-By: agentty noreply@agentty.org

…hput

Context windows:
- CatalogModel::context_window defaulted to 272'000, so any slug whose
  /models row omitted or mis-typed the field displayed "272k" as if the
  endpoint had reported it. Default is now 0 (unknown) — the picker shows
  "auto" and the compaction ladder takes its conservative default; a value
  the endpoint DID report is used verbatim.
- Verified against the live Codex catalog (via the new codex_catalog_probe)
  and codex-rs 0.167.0's embedded snapshot: gpt-6.1-sol is served at 500k
  per the live endpoint (the client snapshot still says 272k),
  gpt-6.1-terra 500k, everything else 272k. The endpoint also carries a
  separate max_context_window (872k) agentty does not consume — documented,
  not invented.

Throughput:
- The billed output_tokens figure includes reasoning tokens; the stats
  panel's single "Output rate" conflated billed with watched. It now
  records two series: "Generated rate" (incl. reasoning, labelled) and
  "Visible rate" (output − reasoning).
- The status-bar sparkline samples streamed bytes / 4 mid-stream (usage
  only arrives at stream end); finalize_turn now rewrites the ring's last
  sample with real tokens over real generation time, so the frozen
  post-stream readout is authoritative.
- Zero-duration, interrupted and unmeasured turns stay excluded, never
  zero-filled; a turn whose reasoning tokens are not a strict subset of
  output_tokens records no visible sample at all rather than a negative
  one.

Tests: metadata_throughput_test (9 cases), codex_catalog_probe (manual,
live dump), updated chatgpt parse expectations, and the stats golden
re-pinned for the rate split.
1ay1 pushed a commit to agentty-org/agentty that referenced this pull request Nov 12, 2026
fix(chatgpt): improve model catalog discovery

**What this changes and why:** the ChatGPT (Codex) model catalog could
come back
empty or badly incomplete — the picker falling back to a stale guess,
then the first
turn 400ing on a slug the account no longer offers. Two independent
causes: the
discovery request was gated out by the server's `client_version` check,
and a single
malformed field in any row aborted the whole payload parse. This fixes
both.

**Fixes #** — no tracked issue; found while diagnosing a reported empty
ChatGPT model
picker, and both failure mechanisms were then confirmed in code and, for
the version
gate, against the live endpoint.

**How you tested it (OS, and which tests or manual steps):** macOS
15.8.1 (Apple
Silicon, arm64). `ctest -R "chatgpt catalog|chatgpt bundled|chatgpt
codex version"` →
4/4; full `ctest -j8`; plus a live OAuth run of the real
`fetch_models()` through the
new manual-only `codex_catalog_probe`. Everything is in **Verification**
below,
including one known `rag_eval_guard` floor break, disclosed in full.

---

## What breaks today

The ChatGPT (Codex) model picker can end up with no models, or with
models that
the account no longer accepts, and then the first turn dies with
`The '…' model is not supported when using Codex with a ChatGPT
account.`
Three independent causes, all in `provider/chatgpt`:

1. **The catalog is gated on the Codex client version we impersonate,
and a stale
   version gets a filtered list — served as a successful 200.** We were
advertising `0.145.0`. Measured against the live endpoint on this
account, that
version returns three `gpt-5.6-*` rows and none of the current models
(the
documented behaviour for an *unknown* version is HTTP 200 with
`{"models":[]}`,
which reads exactly like "this account has no models"). Either way the
picker
silently under-reports the account, and whatever was last selected can
be a
   slug the endpoint no longer serves.
2. **One malformed field drops the whole catalog.** The parser used
`json::value()` for every field inside a single `try { … } catch (…) {
   return {}; }`. A row where `context_window` arrives as a string, or
`is_default` as `null`, threw and discarded *every* model — the failure
is
   per-response, not per-row, and completely silent.
3. **Server-side ordering and internal rows were taken at face value.**
Rows
are ordered by a `priority` field we ignored; `hidden` was filtered but
   `supported_in_api: false` was not; slugs were not de-duplicated.

## Changes

`include/agentty/provider/chatgpt/oauth.hpp`
- `codex_client_version` → `0.167.0` (verified against the live endpoint
and the
current Codex CLI), with the gating behaviour documented at the
constant.

`src/provider/chatgpt/responses.cpp`
- The version is now overridable at runtime with
`AGENTTY_CODEX_CLIENT_VERSION`
so a future server-side gate can be worked around **without a rebuild**
—
which matters because the failure mode is "the picker goes blank", not a
diagnostic the user can act on. The override is validated strictly: it
must be
exactly `N.N.N`, ≤32 chars, and anything else (empty, `1.2`, `1..2.3`,
33
chars, or anything carrying `CR`/`LF`/`&`) falls back to the built-in
value,
because the string is copied into a URL query parameter *and* a
`user-agent`
  header.
- The same validated value is sent in both places (it previously could
drift
  between the query param and the header).
- Parsing moved out of the fetch into a named, directly testable
function
(`parse_models_payload_for_test`): per-row type-checked accessors (`str`
/ `boolean` /
`integer` with range clamps) instead of one global catch, non-object
rows
skipped, duplicate slugs dropped, `supported_in_api: false` dropped
alongside
`visibility: hide|hidden`, and rows sorted by `priority` with invalid or
absent
priorities ordered *after* every valid one (`stable_sort`, so server
order still
  breaks ties). A malformed row now costs one row, not the catalog.
- `max_body_bytes` 1 MiB → 2 MiB: the raw `/models` payload carries a
lot of
  per-model metadata per row and the old ceiling left little headroom.

`tests/chatgpt_bundled_models_test.cpp`
- The parse test feeds 10 rows and asserts exactly **4 survivors** in
priority
order. The 6 dropped rows cover every reject path in the parser:
`visibility:
"hide"`, `visibility: "hidden"`, `supported_in_api: false`, a non-string
`slug`
(`{"slug":42}`), a bare `null` row, and a duplicate slug (first
occurrence
wins). The survivors pin the fallbacks: a row carrying only `id` gets
both its
  slug and its `display_name` from `id`, and its out-of-range
`context_window: -1` falls back to the struct default (272 000 on this
PR —
cloudbouz53#1 replaces that fabrication with an honest `0`)
instead of
  being trusted;
`is_default: true` still marks the default. Degenerate payloads — `not
json`,
`{"models":{}}`, and a lone `{"slug":"bad","priority":1e1000}` — must
parse to
  an empty list, not throw.
- A second case pins the ordering rule the server actually exercises:
valid
  priorities (`1 … 1 000 000`) sort before every absent, unparseable
(`"priority": "1"`), or out-of-range (`1000001`, `18446744073709551615`)
priority, and the fallback group keeps server order — that is
`stable_sort`
under test, asserted by `first` and `equal` both carrying priority `1`.
- Pins the version override: `1.2.3` accepted, `""`, `1.2`, `1..2.3`,
  `1.2.3\r\nX:bad`, `1.2.3&foo=bar` and a 33-char value all rejected.
- Keeps the environment-adaptive guard: the retired `gpt-5.1-codex` slug
and
`codex-auto-review` must never appear, and a model must always be
selectable —
from the live catalog when signed in, from the bundled fallback
otherwise.

## Verification

- `ctest -R "chatgpt catalog|chatgpt bundled|chatgpt codex version"`:
4/4 pass.
- Full suite (`ctest -j8`, 1941 tests): every failure is reproduced on
the base
commit `4ef0d8f1` with identical messages (24 × sanitizer/static-gated
binaries not
produced by a plain build; `ui_snapshots`, `palette_render_probe`,
`elm_purity`,
  `command palette`) — apart from `rag_eval_guard`, documented below.
- **What CI will and will not tell you here.**
`.github/workflows/ci.yml` builds
`--target agentty` on linux-gcc, macos-15-gcc, MSVC and msys2, and its
"Smoke" step is
`./build/agentty --version`. It does **not** build `agentty_tests` and
does **not** run
`ctest`. So a green run on this PR proves the shipped binary still
compiles and starts
on all four platforms — it does *not* exercise the four catalog tests
added here, and it
will never flag `rag_eval_guard`. Those numbers came from local runs on
macOS 15.8.1
(arm64); `ctest -R "chatgpt catalog|chatgpt bundled|chatgpt codex
version"` and
`ctest -R "^rag_eval_guard$` are the two commands to re-run if you want
the evidence.
- **Base has moved since this was verified.** This branch is based on
`4ef0d8f1`;
upstream `master` is now `ecc9fbe9` (4 commits ahead: README trim,
`v0.9.54`,
changelog, and *"ci green: four stale tests, a view env read, arch
partial
upgrade"*). None of those commits touch any file in this PR, so the
branch applies
cleanly and the diff is exactly this PR's content. Two things follow,
both in this
PR's favour: `ecc9fbe9` already repairs several of the pre-existing
failures above
(including the `command palette` `kAll` list), and it advances the
`rag-cpp` / `maya`
submodule pins — so the `rag_eval_guard` figure below is a `4ef0d8f1`
measurement
against those pins; re-running it on the current base is the open
question. Submodule
  pins were deliberately left untouched during verification.
- **Live A/B of the version gate, same account, same binary, seconds
apart:**

  | `client_version` | rows returned |
  |---|---|
| `0.145.0` (pre-PR) | 3 — `gpt-5.6-sol`, `gpt-5.6-terra`,
`gpt-5.6-luna` |
| `0.167.0` (this PR) | 8 — adds `gpt-6.1-sol`, `gpt-6-astra`,
`gpt-6.1-terra`, `gpt-6-sol`, `gpt-6-luna` |

The 8-model list is priority-ordered, carries no `gpt-5.1-codex` and no
  `codex-auto-review`.
- **Injection control on the override:**
`AGENTTY_CODEX_CLIENT_VERSION='1.2&x=yy'`
is rejected by validation and the request goes out with the built-in
version —
8 models returned, identical to the default. Same for `""`, `1.2`,
`1..2.3` and
  an over-length value, which are covered by unit assertions too.
- A live turn on a catalog model completes normally
  (`-m gpt-6.1-terra`, `--provider chatgpt`).
- ⚠️ The `chatgpt bundled model catalog` doctest always exercises the
**offline**
  branch, on `master` as on this branch: `tests/test_main.cpp` repoints
`AGENTTY_HOME` at a throwaway directory for the whole `agentty_tests`
binary, so
credentials are never visible under ctest and `responses_available()` is
false.
The doctest therefore guards the offline floor; the live path has to be
proven
out of harness. cloudbouz53#1 adds `codex_catalog_probe`, a
manual-only (`NO_TEST`)
tool that dumps the live catalog — its output above was produced with
it, and it
  runs this PR's discovery code unchanged.

## Notes for reviewers

- **`rag_eval_guard` regression, disclosed in full — and it is not
flaky, not
  environmental, and not a tuning accident.** On this branch the
`agentty_code_heldout` set scores MRR **0.666** against its required
threshold of
**0.670**, while **hit@8 is unchanged at 0.950** (floor 0.90).
Reproduced on every
run, including this one: the **same binary** was run against two
pristine worktrees
differing only by this PR's tracked content — base `4ef0d8f1` scores
**0.670**
(clears the floor by exactly zero), this branch scores **0.666**. The
other three
eval sets are byte-identical between the two runs (0.917/0.707,
0.967/0.744,
  1.000/0.887), so the one moved number is the whole effect.
The two misses are heldout queries whose gold file is
`src/provider/selection.cpp`,
which this PR does not touch. The mechanism is corpus size, not
retrieval quality:
the guard indexes whatever the checkout contains, and this PR adds ~100
lines (new
test cases, comments, the ordering path). Each added line is another
BM25 chunk,
shifting document frequency and average document length corpus-wide,
which is enough
  to move one result one slot. The eval set
(`tests/rag_eval/agentty_code_heldout.json`) is **not** touched by this
PR. Any
branch that adds indexed lines trips this floor, because base sits
exactly on it.
Nothing here changes retrieval code, so I have deliberately **not**
lowered the floor
inside this PR — happy to, in a separate commit with its own rationale,
if you'd
rather see CI green. If maintainers want the floor to stop firing on
unrelated PRs,
the two obvious options are scoring the eval on a pinned corpus, or
expressing the
  floor relative to the base run.
- No API or behaviour changes outside `provider/chatgpt`; nothing
user-visible
  except a picker that stays populated.

Co-Authored-By: agentty <noreply@agentty.org>

Co-authored-by: Mac <mac@MacBook-Air.local>
Co-authored-by: agentty (gpt-6.1-terra) <noreply@agentty.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant