Repository navigation
Add v4 quota cycle contribution sync - #13
Merged
Merged
Conversation
Emit attributed weekly Codex cycles as compact hosted contributions inside sync_batch.v4, with daily envelopes, exact boundary slices, and dirty tracking. Keep v1-v3 batches accepted without quota collections.
Empty batches now use sync_batch.v4, so ack validation tests expect sync_ack.v4 while still covering v1 and v3 pairing.
`cargo clippy --all-targets -D warnings` fails on this branch: sample_usage_event takes nine arguments against a limit of seven. It is a test fixture whose parameters are all distinct token counts, so a struct would obscure the call sites rather than clarify them. This branch targets sync-ready-quota-history rather than main, and ci.yml only runs on pull requests into main, so the failure has never surfaced in CI.
A Codex weekly cycle starts at the first request after the previous reset, not at the reset itself. Consecutive cycles therefore cannot overlap unless the earlier one was reset ahead of schedule, which happens whenever a banked or server-granted reset is used. Reconstruction already detects this and records it as has_schedule_overlap, but the contribution dropped the flag before sync, so the backend could only re-derive the geometry and had to treat every early reset as a conflict. On real local history that mislabelled 110 of 122 cycles, and 34 of 38 gaps across the three most-used accounts in the last 30 days are early resets rather than the nominal seven days. Carrying the flag lets the backend separate an overlap a single device observed from one that only appears after merging two devices, which is the case that genuinely indicates records that failed to cluster or two accounts resolving to one identity. inferred_start stays off the wire: it is exactly representative_reset minus the nominal duration, which the backend already computes. The field defaults to false so payloads written before it deserialize unchanged.
Every synced record is checked against a closed set of permitted keys, and an unrecognized key refuses the whole batch. That guard is deliberate: an unexpected key may carry a path, address, or message the contract excludes, and refusing guarantees none of it is stored. It has a second cause, though. A collector newer than the deployment it syncs to trips the same guard, and until now both looked identical from the client: an opaque 400 that stopped events and summaries as well as the affected collection. Adding has_schedule_overlap reproduced exactly this against an endpoint without the field. The endpoint now returns the offending field name, nested keys included as a dotted path, and collectors render it with the remedy instead of a truncated body. Only the name crosses the wire, never the value. Malformed values inside a known field stay a plain 400, since that is a client defect rather than version skew. The release order this implies is now written down: deploy the backend before releasing a collector that adds a record field. New top-level collections stay backward compatible and need no version bump, because per-collection ack counts already turn an endpoint that ignores one into a loud count mismatch.
A window cannot be observed once it has reset: from that moment the provider issues a fresh resets_at. Evidence that post-dates its own reset is therefore a replay of historical records stamped with the import time, and reconstruction was letting it back in — clustering it into the long-closed cycle by resets_at, then bucketing it into a daily envelope by observed_at. On one paired device that gave seven July cycles an envelope dated 2026-08-12, every one of them written inside the same ten seconds at a different percentage, up to 26 days past the cycle's end. 13 of that device's 106 synced envelopes fell outside their own window; the other device had none of 299. The bound is an hour rather than minutes because real recomputation lag exists: this machine has a window whose provider kept reporting an elapsed reset with a steady 77-79% for a continuous 47 minutes, and that evidence is genuine. Replays look nothing like lag — weeks late, in bursts of unrelated windows sharing one timestamp — so an hour separates them by more than two orders of magnitude.
Server merge now ignores daily envelopes observed more than an hour after the contribution reset, matching collector reconstruction. Already-stored rows are filtered on read; ingest strips them instead of rejecting the batch.
A window that reported zero usage for its whole life and was then replaced never functioned as a cycle. Nothing was ever attributed to it, so it cannot be told apart from a reschedule of the window that followed, and reconstruction drew each one as a reset that never happened. They arrive in runs. On 2026-07-10 one account produced four windows where two announced resets occurred, three of them minutes apart at zero percent. Across both paired devices 16 such windows exist, all superseded, 14 of them during the July tier migration. The rule is that a cycle must have spent something. Sitting at zero is not enough to be told apart from a reschedule, while three percent is: a window that peaked at 78 across a full week and another at three before a 51-day gap are both real, and both survive. The newest cluster is exempt, since a cycle that has only just begun legitimately reads zero. Announced provider resets are unaffected. Every one of the ten global resets publicly announced across July still reconstructs as its own cycle.
Consumption cannot fall inside a window, so a reading below one already seen is stale rather than a refund. Concurrent sessions produce them routinely: two poll at once and the slower one still holds the earlier figure, sometimes carrying the same timestamp to the second. When such a reading lands last on a day, that day closes below the one before it and the chart draws a decline that never happened. Three of 103 multi-day cycles fell by one to three points this way, and 5% of envelope days closed below their own observed maximum. Daily first and last now carry the running maximum for the cycle, which is what consumed-by-then means. The observed minimum and maximum are left alone, since they are the evidence that the jitter occurred.
…i/statsai into ai/quota-cycle-sync-f197
A reset is a handover: once a window resets, the provider reports the new schedule and never returns to the old one. A cluster that another cluster brackets in observation time - reported before it appeared and again after it vanished - therefore cannot be a cycle of its own, however many observations back it. During the July 2026 tier migration Codex answered scattered turns with a blank snapshot: near-zero usage against a fresh weekly reset, interleaved with the live schedule for as long as 34 hours. Each run became a cycle that took days away from the cycle actually running, which is what drew two rising lines over the same hours for a single account. Requiring the bracketing cluster to have passed the suspect's peak by the time it resumes preserves genuine early resets, which lead from the moment they start. Across the local store this drops 14 phantom cycles and keeps 108; every drop peaked at 4% or less while the live cycle stood between 8% and 100%, and all of them fall between 8 and 11 July 2026.
Two overlapping lines for one account over the same days, twice more. The tolerance was measured from the cluster's first point, as though a cycle's reported reset had to fit inside a five-minute window. It drifts a second or two at a time and has been seen wandering nine minutes over a cycle's life, so the cycle was cut in two the moment it passed five and the remainder drawn as a second cycle beside the first. Measured as a gap between neighbouring resets it stays whole; consecutive resets sit hours apart at the least, so chaining cannot reach across one. The phantom rule also asked the bracketing schedule to have outrun the bracketed one, which was over-cautious. Once a window resets its resets_at moves forward and can never be reported again, so a schedule cannot resume after another has taken over: bracketing alone settles it, and a genuine early reset is never bracketed because the schedule it replaced simply stops. This drops a cluster of 623 observations that ran to 99% over 19 hours while the schedule either side of it sat near 54% - not a cycle that reset early but a second counter reported under one slot for one account. Across the local store this takes 98 weekly cycles to 94, all four from starkovd and videochapter, and leaves every announced reset standing.
Every reconstruction rule throws away evidence that cannot describe a real cycle, and every one of them is silent. If a provider change made them fire on data they were never meant to judge, the only symptom would be fewer cycles, indistinguishable from a quieter week. `quota status` now counts the three: observations recorded after the window they describe had already reset, windows that never left zero before being superseded, and schedules another schedule was reported either side of. On this store that reads 730, 33 and 28. Status pays for one reconstruction it did not run before. That cost is inherent to reporting what reconstruction discards, and this is a diagnostic run by hand rather than anything on a hot path.
The comment justified the hour with a 47-minute event that turns out to be a replay burst: 134 records from one rollout file sharing a timestamp to the millisecond. The real evidence is 2026-06-01, where a live session polled ~180 times over 37 minutes and the provider returned the expired window at 96% every time.
The 5m and 1h counters break the aggregate down by lifetime rather than adding to it: Claude reports 248 beside 148 + 100. Summing all three doubled cache creation in every boundary slice and pushed the components past the slice's own total_tokens, which computed_total derives from the aggregate alone. Also narrows the two sync-eligibility checks to the weekly window exactly. Contributions already filter on equality, so a monthly window counted toward weekly sync coverage and appeared in the sync-windows export without ever being syncable.
A local store keeps quota observations and derives its own cycles from them; there is no table for another device's contributions. Acknowledging them as accepted told the sender to record them synced against this target and stop offering them, certifying a write that never happened. Refusing follows the authoritative-snapshot precedent: /v1/sync/batches is documented as loopback diagnostics, and /api/sync/batches is the contract that stores quota cycles. Reporting them rejected instead is not available — the sender treats any rejected record as a failed batch and separately requires accepted plus duplicates to equal what it sent.
A quota cycle can change without any summary changing: a reset moves, or an observation carries no tokens. The pending count summed rollups, passthrough summaries, code-change metrics and retirements, so those uploads were invisible and the menubar reported nothing to send while a sync would still have sent them. The contributions were already being read here for retirement detection; they now also go through pending_quota_cycle_contributions_for_sync and into the total. Pending days stay summary-shaped, and the menubar derives its day count from elapsed time whenever a sync has already succeeded.
ureq's request timeout covers reading the body, so the 30-second bound meant for JSON replies also capped every artifact download. An archive near the 256 MiB ceiling would have to sustain about 68 Mbit/s to finish inside it, which is why downloads failed intermittently on a home connection and succeeded on a well-connected machine. Downloads now get ten minutes, enough for the ceiling at roughly 3.5 Mbit/s, and STATSAI_DEV_DOWNLOAD_TIMEOUT_SECONDS overrides it, with zero meaning wait indefinitely. API calls keep the thirty seconds.
The wait loop retries the build lookup, not the transfer, so a read that died partway ended the whole install on the first try. The 'attempt 1' in the output is GitHub's own run attempt describing the CI build, not a retry of ours — there were none. A body cannot be resumed, so a retry re-requests it: two retries with a short backoff. A read failure surfaces from read_to_end rather than from call, so it is now marked retryable explicitly; without that the loop would classify it as permanent. Exceeding the size ceiling stays permanent, since re-requesting cannot make the artifact smaller.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds on the sync-ready quota-history reconstruction. This PR does not recreate parsing, attribution, or cycle reconstruction.
Sync contract
sync_batch.v4/sync_ack.v4while still accepting v1–v3.QuotaCycleContributionV1. Codex currently emits only attributed weekly cycles (nominal 10,080-minute window).resets_at, so import-time replay stamps cannot extend a closed cycle. Hosted merge applies the same bound on read and ingest for already-synced rows.Tests
Collector coverage for weekly-scope selection, daily envelopes, exact boundary slices, stable contribution IDs, and exclusion of unattributed / five-hour / monthly cycles.