Count the waypoints a late round drops, and how late they go out - #727
Count the waypoints a late round drops, and how late they go out#727v-positronic wants to merge 5 commits into
Conversation
The loop calls the policy and plays the trajectory in one round, so a round longer than the control period pops every waypoint that came due and emits only the newest. It discarded the rest silently: nothing counted them and nothing logged them. Each command channel now keeps its own account of the episode - the waypoints the schedule took, the rounds that sent one, the waypoints a round overtook, and a histogram of how far past its own due time each emitted waypoint went out. The episode's static meta carries it under 'eval.schedule.<channel>.*', beside the other eval facts a rollout report reads. The emit behaviour does not change. The arm goes to the newest setpoint, and this change measures that rather than redesigns it. Ticket: none - instrumentation asked for in chat; no ticket exists for it
The episode statics carry the per-channel account. A timing report reads spans, not statics, and that is where a reader already goes to ask where the loop's time went. The episode span now carries the same account totalled over its command channels, and 'positronic eval timing-report' prints what the pass scheduled, sent and dropped, the drop share of what came due, and the mean and worst lateness. Lateness rides on the span as a sum and a maximum, because those total across episodes exactly; a percentile does not, so the distribution stays per channel in the episode's statics. The two sinks carry one measurement. Telemetry is opt-in - 'telemetry.bind_from_env' is inert unless POSITRONIC_ENV_TELEMETRY_DIR is set - so a customer rollout writes no span at all, and the episode record is what makes the account unconditional. Ticket: none - instrumentation asked for in chat; no ticket exists for it
The sum-and-maximum rationale belongs with the span rendering that chooses it, and the eval and telemetry key comments carry only what a reader of the record needs. Ticket: none - a comment trim on the change above
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e61cdd50f5
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Ticket: none - a comment trim on the change above
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d28237171e
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
…covers A chunk arriving on a late round replaces waypoints that had already come due. The round then plays the new schedule, so those go out nowhere and were counted neither emitted nor dropped - understating the drop rate exactly when the loop runs slow. '_reschedule' now counts the due leading run of each schedule as dropped before it clears it. The reduce reads only the episodes whose span carries an account, and reports how many did. A directory holding passes from either side of the account's arrival had been counting an episode that measured nothing as one that dropped nothing. '_waypoint_report' moves beside the caller it has. Ticket: none - review findings on the change above
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 097902c0d7
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| for name, fidelity in self._fidelity.items(): | ||
| meta.update(fidelity.meta(f'{eval_keys.SCHEDULE}.{name}')) |
There was a problem hiding this comment.
Rule hidden-dependency violated:
Fresh evidence beyond the earlier _reschedule comment is _finalize_recording's separate clear path: when done, timeout, or shutdown is observed on a round with already-due waypoints, the STOP meta is built and the queues are cleared before _issue_due_commands. Those due waypoints went out on no round but never reach count_dropped, so both the static and span figures understate drops; classify each due prefix before building the meta and clearing the schedules.
AGENTS.md reference: AGENTS.md:L7-L8
Useful? React with 👍 / 👎.
| meta[keys.TASK] = self._task.instruction | ||
| return meta | ||
|
|
||
| def _episode_waypoints(self) -> _ScheduleFidelity: |
There was a problem hiding this comment.
Rule stranded-definition violated:
_episode_waypoints is first called by _finalize_recording, but the unrelated _ready and _pace definitions separate it from that use. Move the helper immediately above _finalize_recording; it has another caller in the exception path, so it should remain a method rather than be nested.
AGENTS.md reference: AGENTS.md:L7-L8
Useful? React with 👍 / 👎.
| covers. A directory holding passes from either side of this account's arrival reduces to fewer than the | ||
| pass's episodes, and the report says so rather than counting an episode that measured nothing as one | ||
| that dropped nothing. |
There was a problem hiding this comment.
Rule stale-doc violated:
WaypointReport, along with the matching test docstring, explains absent accounts as coming from “either side of this account's arrival,” tying the documentation to an unversioned past change. State the current behavior directly—that episodes without waypoint attributes are excluded and coverage is reported—or cite a durable version if the compatibility history is essential.
AGENTS.md reference: AGENTS.md:L7-L8
Useful? React with 👍 / 👎.
1 participant
The eval harness runs one loop that both calls the policy and plays the waypoints the policy
scheduled.
Harness._issue_due_commandspops every waypoint that has come due since the lastvisit and emits only the newest, so a round longer than the control period does not delay a
waypoint — it discards it. Nothing counted that and nothing logged it.
Each command channel now keeps an account of its episode: the waypoints its schedule took, the
rounds that sent one, the waypoints that went out on no round, and a millisecond histogram of how
far past its own due time each emitted waypoint went out. It lands in two places.
The episode's static meta, per channel, under
eval.schedule.<channel>.{scheduled, emitted, dropped, late_p50_ms, late_p90_ms, late_max_ms}— beside theeval.*andinference.policy.*facts a rollout report already reads.
The episode telemetry span, totalled over the channels, as
episode.waypoints.{scheduled, emitted, dropped, late_sum_ms, late_max_ms}, whichpositronic eval timing-reportreduces:One measurement, two sinks, because they answer cases the other cannot.
telemetry.bind_from_envis inert unless
POSITRONIC_ENV_TELEMETRY_DIRis set, so a customer rollout writes no span atall, and the episode record is what makes the account unconditional. A timing report reads spans
rather than statics, and is where a reader already goes to ask where the loop's time went.
Why
Measured over run
blind_20260908-135658, 36,548 rounds across 10 episodes: a policy call takes38 ms at the median (57% of it the temporal stack, 37% the image resize) against a 66.7 ms control
period at the model's 15 Hz action rate, so the loop achieves 23 Hz median and 9.7 Hz at p90.
34.5% of rounds swallow at least one waypoint — roughly one commanded waypoint in four never
reaches the arm — and what does arrive is up to a whole round late, 43 ms median, 103 ms at p90.
The arm executes a decimated version of the trajectory the model predicted, and no figure said so.
Shape
The emit behaviour is unchanged. Keeping only the newest due waypoint may well be right: the
arm should go to the newest setpoint, not replay stale ones. This measures; it does not redesign.
A drop is a due waypoint that went out on no round — one a later waypoint overtook inside a
round, and one a fresh chunk replaced after its time, the same loss on the same late round.
emitted + droppedis what came due;scheduledalso covers what a chunk replaced early.A histogram rather than a sample list. The loop is the thread that drives the arm, so the
account may not allocate or sort per round. Each channel holds 1001 integers — one bin per whole
millisecond to a one-second bound, plus a saturating last bin — fixed at construction.
The span carries a sum and a maximum, not percentiles. Counts sum across episodes and a
maximum maxes, so a pass figure reduced from them is exact, where averaging per-episode
percentiles is a percentile of nothing. The distribution stays per channel in the statics. The
reduce counts only episodes whose span carries an account, and reports how many did.
Verification
CI is the arbiter here: this box is shared and was at load 76+ on 8 cores, which fails
wall-clock-sensitive sim tests at random — a different set each run, each green in isolation.
pre-commitover the six files passes, with the basedpyright baseline untouched.Twelve new tests pin: a round with three due emits the newest and counts two dropped; a round with
one due counts none; a fresh chunk landing at 25 ms counts the waypoints due at 0, 10 and 20 as
drops and leaves the one due at 30 alone; ten rounds 0-9 ms late give p50 4, p90 8, max 9; and the
span totals equal the statics' per-channel figures added up, so the two sinks cannot drift.