Bound SqlServerHealthCheck with a per-probe timeout so failures publish before the publisher's outer timeout fires

[jnlycklama]

Summary

Fixes a production scenario where /health/check returns 200 OK even though the SQL database has been deleted, because the publisher pipeline never gets a chance to publish the failure.

Root cause

1. SqlServerHealthCheck issues a SQL query against the deleted DB.
2. SqlClient's connect-timeout (30s) plus retry policy (ConnectRetryCount + ConnectRetryInterval) can take 70+ seconds for a single failed connection attempt.
3. The framework's HealthCheckPublisherHostedService creates a linked CancellationTokenSource bounded by HealthCheckPublisherOptions.Timeout (default 30s).
4. The SQL retry exhaustion consumes the entire budget; the outer token cancels mid-probe.
5. The framework distinguishes "cancellation" from "exception": when cancellationToken.IsCancellationRequested == true, the OperationCanceledException is not converted into HealthStatus.Unhealthy. It propagates up to the hosted service, which treats the whole batch as cancelled — PublishAsync is never called.
6. The previously cached HealthReport (Healthy) stays in ValueCache<HealthReport> indefinitely, so CachedHealthCheckMiddleware keeps returning 200.

Fix

• New SqlServerDataStoreConfiguration.HealthCheckProbeTimeout (default 10s).
• SqlServerHealthCheck.CheckStorageHealthAsync now wraps the SQL call in a linked CancellationTokenSource with CancelAfter(HealthCheckProbeTimeout). Failing fast inside the publisher's budget guarantees the result reaches PublishAsync.
• Any SqlException that escapes the probe is converted to HealthCheckResult.Unhealthy(...) explicitly, with the SqlException attached so consumers see real diagnostics.
• A probe-timeout OperationCanceledException (where the caller did not cancel) is converted to Unhealthy with a clear description.
• Caller-initiated cancellation (the framework's outer token) is still allowed to propagate as OperationCanceledException, preserving the hosted service's "cancelled batch vs real failure" distinction.

Test plan

• 3 new unit tests in SqlServerHealthCheckTests:
  • login-failure (SqlException 18456) → Unhealthy with the exception attached.
  • probe exceeds configured timeout → Unhealthy with timeout description.
  • caller cancels → OperationCanceledException propagates (not swallowed).
• Existing tests still pass.
• Build clean (0 warnings, 0 errors).
• dotnet test src/Microsoft.Health.SqlServer.UnitTests → 100/100 (+2 skipped) on net8/9/10.

Related

Complements #1425 (ValueCache<HealthReport> expiry as a generic safety net for any publisher hang). This PR fixes the SQL-specific root cause; #1425 catches anything else that can wedge the publisher.

[jnlycklama]

Closing per design discussion. The per-probe timeout and explicit OCE conversion regress two important behaviors: (1) it hides the real SqlException from logs (the timer fires before SqlClient finishes retrying, so we never see 'Login failed for user'), and (2) it turns transient blips that today recover via SqlClient retries into published Unhealthy results. Going forward we'll rely on PR #1425 (ValueCache expiry as the safety net) plus a per-consumer bump of HealthCheckPublisherOptions.Timeout in dicom-server.