[P1][Telemetry] Emit Core download attempt, retry, and terminal-outcome telemetry

[sheetalarkadam]

Description

Current telemetry cannot distinguish a failed transport attempt from a logical download that later recovers, exhausts retries, is cancelled, or completes through another path. Raw failed-attempt counts are therefore not a valid final user-outcome metric.

PhonePe telemetry for September 1–8 recorded:

• 144 failed ModelDownload action attempts across 44 sessions;
• 476 successful attempts across 238 sessions;
• 19 of the failed-attempt sessions later recorded a successful attempt;
• 25 failed-attempt sessions had no success observed in the reporting window.

The last group is failure-only within the observed window, not automatically a proven terminal download failure. Android completion telemetry separately contained successful rows and no failed completion rows, but the existing surfaces cannot reconstruct Core retries or one final logical outcome.

Current Android origin/main passes only model and timeout into the Core download command. The callback exposes progress/cancellation, and Android infers completion after the native call returns. Android therefore cannot identify actual Core transport attempts, retry decisions, retry reasons, or retry exhaustion.

microsoft/Foundry-Local-for-Android#722 remains responsible only for terminal paths created by the Android IPC SDK, such as bind failure, service disconnect/death, pre-start cancellation, and AIDL RemoteException. This issue owns Core/download transport attempts, retries, and the canonical terminal result.

Required contract

For each logical download invocation, emit bounded typed telemetry that answers:

1. How many actual attempts occurred?
2. What was the normalized outcome of each attempt?
3. Why was another attempt scheduled or not scheduled?
4. What was the one terminal outcome delivered to the caller?
5. What were total and per-attempt duration, byte, resume, and cache semantics?

Required fields where available:

• attempt number/count and retry count;
• attempt outcome, retry decision, normalized retry reason, and failure domain;
• typed transport/HTTP status only when available as structured data;
• per-attempt and total elapsed time;
• bytes received, expected bytes, resumed bytes, and verified bytes;
• acquired, waiter, resume, cache-hit, repair, or fresh-download mode;
• exactly one final outcome: success, recovered-after-retry, retry-exhausted, cancelled, timed out, cache hit, integrity failure, or typed terminal failure;
• whether the terminal callback/result was delivered;
• model identity under existing telemetry privacy rules.

Preserve existing sessionId as the authoritative cross-table join key. If an invocation ordinal or identifier is required for exactly-once accounting, scope it to rows within the download table; it must not become a replacement cross-table OperationId, AttemptId, or correlation key.

Table semantics

• action: caller-visible operation outcome and duration; do not redefine it as the transport-attempt ledger.
• download: actual attempt/retry detail and exactly one terminal summary, including terminal failures.
• modelid: model/action/status context where applicable.
• error: exception diagnostics only when the producer contract emits them; a handled Core retry does not require one error row per failed attempt.
• deviceid: device/release cohort enrichment through sessionId.

Implementation direction

• Instrument the retry owner in shared Core/download transport; do not reconstruct retries from Android callback strings or row order.
• Extend the typed download result/callback so attempt state and the final outcome can cross host boundaries consistently.
• Use stable enums and explicit unknown/not_available states. Do not parse localized exception text to invent DNS, TLS, HTTP, certificate, or upstream classifications.
• Emit one row per actual attempt plus exactly one final summary, or a bounded attempt summary if ingestion volume requires aggregation.
• Make final-summary emission exactly-once across first-attempt success, fail-then-success, retry exhaustion, cancellation, timeout, cache hit, acquired/waiter completion, integrity repair, process interruption, and callback races.
• Map Android-owned terminal classifications from microsoft/Foundry-Local-for-Android#722 into the shared final-outcome vocabulary without moving their Android diagnostics into Core.
• Never record raw URLs, headers, tokens, certificates, filesystem paths, prompts, responses, model contents, or raw customer payloads.

Acceptance criteria

• First-attempt success reports one attempt and one terminal success.
• A deterministic transient failure followed by success reports both attempts and one recovered_after_retry terminal result.
• Retry exhaustion reports the complete attempt count, normalized final reason, and one terminal failure.
• Cancellation before and during transfer, timeout, cache hit, resume, acquired download, waiter completion, integrity repair, and process interruption each produce one defined terminal summary.
• Android bind failure, service disconnect/death, pre-start cancellation, and AIDL RemoteException remain owned and tested by microsoft/Foundry-Local-for-Android#722 while using compatible final-outcome values.
• Exactly one terminal summary and one caller terminal result are produced under completion/cancellation/death races.
• action, modelid, download, optional error, and deviceid remain joinable through sessionId.
• No new cross-table correlation identifier is introduced.
• A handled retry is not required to emit an error row, and failed attempt rows are never reported as final user failures.
• Telemetry opt-out and privacy behavior are preserved.

Validation

Add deterministic fault injection for first-attempt success, fail-then-success, retry exhaustion, timeout, cancellation, resume, cache hit, acquired/waiter completion, and one Android-owned terminal path. For each run, record the exact build, UTC window, model, expected rows, and authoritative sessionId; verify download, action, modelid, deviceid, and optional error independently.

Production/canary reporting must publish attempt failure rate, recovered-after-retry sessions, terminal success/failure rate, retry-count distribution, terminal-summary completeness, and acquired-download latency separately. Do not gate releases on terminal outcomes until completeness and exactly-once behavior are demonstrated.

Ownership

• Foundry Local Core/download transport: actual attempts, retry decisions, byte/timing data, and canonical terminal result.
• Android IPC SDK: local terminal paths tracked by microsoft/Foundry-Local-for-Android#722, callback exactly-once behavior, and forwarding existing session/model context.
• Telemetry schema/backend: typed fields, stable enums, completeness monitoring, and denominator-safe queries.