Skip to content

Smoke report header table shows unknown/NOT RUN despite a passing doctor check #1489

Description

@randlee

What's wrong

The header table at the top of every feature-smoke HTML report (and the CROSS-HOST TEST preflight rows) renders:

Advertised IP    unknown
ATM version      unknown
Doctor           NOT RUN

even when the run's own doctor-equivalent case genuinely ran and passed (e.g. AD11-CMD-DOCTOR-001 visible further down the same report as PASS).

Root cause

scripts/smoke/feature_smoke_report.py::host_preflight_facts (lines 52-73) only recognizes a case whose name is exactly "doctor" or ends with "doctor/version":

doctor_cases = [case for case in self_cases if case["name"] == "doctor" or case["name"].endswith("doctor/version")]

and only recognizes an IP-advertisement case whose name ends with "advertised host" (line 61). The thorough fixture's cases are named after their test IDs (e.g. AD11-CMD-DOCTOR-001), which match neither pattern, so host_preflight_facts silently falls through to "unknown" / "NOT RUN" even though the check ran and passed.

Repro

Open site/reports/smoke/macos/<host>/<timestamp>-thorough/thorough.html (or index.html) for any thorough smoke run and look at the top header table plus the "CROSS-HOST TEST" preflight row — both show unknown/NOT RUN despite AD11-CMD-DOCTOR-001 passing in the case ladder immediately below.

Impact

Cosmetic/report-accuracy only — the underlying smoke suite genuinely validated the daemon. But the report is misleading evidence: a reviewer skimming just the header table would wrongly conclude doctor/version was never checked.

Suggested fix

Widen the matcher in host_preflight_facts to also recognize the suite's actual doctor/version case-naming convention (e.g. match on case name containing DOCTOR or a dedicated kind/category tag on the case dict), rather than requiring the literal string "doctor".

Activity

  1. github-actions commented on Sep 13, 2026

    @github-actions

    🤖 Hi @randlee, I've received your request, and I'm working on it now! You can track my progress in the logs for more details.

  2. github-actions commented on Sep 13, 2026

    @github-actions

    🤖 I'm sorry @randlee, but I was unable to process your request. Please see the logs for more details.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions