Two gaps in the execution_time metric that make failures hard to measure, as opposed to hard to read about.
1. Only completed and error are ever recorded
ExecutionTimeMetric.update takes status as a tag and report() passes "cancelled", "preempted", "timeout" or "error" depending on how the run ended (modeling/base.py:589-643). In practice only two values appear. Over a 2h window on Test that included deliberate cancellations:
from(bucket:"metrics") |> range(start:-2h)
|> filter(fn:(r) => r._measurement=="execution_time" and r._field=="exec_ms")
|> group(columns:["status"]) |> count()
status count
completed 49
error 18
A request cancelled with ndif kill was recorded as status="completed". So cancellation and preemption rate cannot be measured at all — you cannot tell how often work is thrown away.
The proximate cause for cancelled is the companion issue: the actor never takes its cancellation branch, because kill only fails the dispatch. preempted and timeout are worth checking separately; timeout in particular cannot fire while NDIF_DEFAULT_EXECUTION_TIMEOUT_SECONDS is unset.
2. No error_type tag
Tags on execution_time are model_key, api_key, email, status. error_type is emitted on the log event but not the metric, so metrics say that something failed and never why. Answering "what is our most common failure?" requires a log query rather than a dashboard panel.
Verified the log side is fine — the same requests carried error_type=RunnerError (a block that raised) and error_type=ConnectionError (a block that killed its sandbox runner), both greppable and correct. It is only the metric that is blind.
Suggested
Add error_type as a tag on execution_time. It is bounded (exception class names) so cardinality is manageable, and it makes error breakdown chartable next to the existing status. Fixing #278 should make cancelled start appearing on its own.
Observed on the Test deployment, ndif 0.8 @ be38d78.
Two gaps in the
execution_timemetric that make failures hard to measure, as opposed to hard to read about.1. Only
completedanderrorare ever recordedExecutionTimeMetric.updatetakesstatusas a tag andreport()passes"cancelled","preempted","timeout"or"error"depending on how the run ended (modeling/base.py:589-643). In practice only two values appear. Over a 2h window on Test that included deliberate cancellations:A request cancelled with
ndif killwas recorded asstatus="completed". So cancellation and preemption rate cannot be measured at all — you cannot tell how often work is thrown away.The proximate cause for
cancelledis the companion issue: the actor never takes its cancellation branch, becausekillonly fails the dispatch.preemptedandtimeoutare worth checking separately;timeoutin particular cannot fire whileNDIF_DEFAULT_EXECUTION_TIMEOUT_SECONDSis unset.2. No
error_typetagTags on
execution_timearemodel_key,api_key,email,status.error_typeis emitted on the log event but not the metric, so metrics say that something failed and never why. Answering "what is our most common failure?" requires a log query rather than a dashboard panel.Verified the log side is fine — the same requests carried
error_type=RunnerError(a block that raised) anderror_type=ConnectionError(a block that killed its sandbox runner), both greppable and correct. It is only the metric that is blind.Suggested
Add
error_typeas a tag onexecution_time. It is bounded (exception class names) so cardinality is manageable, and it makes error breakdown chartable next to the existingstatus. Fixing #278 should makecancelledstart appearing on its own.Observed on the Test deployment, ndif 0.8 @ be38d78.