Skip to content

Add ACCELERATED_COMPUTE health check for the MPS control daemon - #5124

Merged
harishxr merged 1 commit into
aws:devfrom
harishxr:harishxr/mps-ec2-daemon-healthcheck
Sep 4, 2026
Merged

harishxr merged 1 commit into
aws:devfrom
harishxr:harishxr/mps-ec2-daemon-healthcheck

Conversation

@harishxr

@harishxr harishxr commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Summary

Add an ACCELERATED_COMPUTE doctor health check that reports the instance IMPAIRED when the NVIDIA MPS control daemon is persistently unavailable. Today the per-task MPS health gate refuses individual MPS tasks when the daemon is down, but nothing tells the control plane the instance itself is unhealthy, so it keeps routing MPS tasks to a box that rejects them one by one. This check surfaces a persistently broken daemon as an instance-level health signal on DescribeContainerInstances and the "ECS Container Instance Health Change" EventBridge event.

Implementation details

  • agent/doctor/mps_daemon_healthcheck.go - the health check. Runs the MPS control-daemon probe once per doctor tick and carries its own consecutive-failure counter: it reports IMPAIRED only after 3 consecutive failures and resets on a single success, so a brief RestartSec=1 systemd bounce does not flap. GetHealthcheckType() returns ACCELERATED_COMPUTE. Reuses the existing MPS probe (pipe-directory pre-check + one ProbeControlDaemon, 3s timeout); a nonzero exit or a timeout (wedged daemon) both count as a failure.
  • agent/app/agent_mps_healthcheck_unix.go - registers the check on MPS-capable instances via the shared gpu.ShouldAdvertiseMpsCapability predicate.
  • agent/app/agent_mps_healthcheck_unsupported.go - no-op so non-Linux builds are unaffected.
  • agent/app/agent.go - wiring in newDoctorWithHealthchecks.
  • agent/app/agent_capability_unix.go - extracted mpsCapabilityInputs() so the gpu-sharing-mps capability and the health check decide from the same host facts (single source of truth). No behavior change to the capability.

Testing

New tests cover the changes: yes

Live E2E on a EC2 GPU instance:

  • systemctl stop nvidia-mps.service: failures 1 and 2 stay Ok, failure 3 → IMPAIRED, after restart → OK.

Fault → IMPAIRED was ~75s (3 failures at the ~25s heartbeat tick); recovery is immediate on the first successful probe.

DescribeContainerInstances --include CONTAINER_INSTANCE_HEALTH 

"healthStatus": { "overallStatus": "IMPAIRED", "details": [
    { "type": "ACCELERATED_COMPUTE", "status": "IMPAIRED", "lastStatusChange": "..." },
    { "type": "CONTAINER_RUNTIME",   "status": "OK" } ] }

The "ECS Container Instance Health Change" EventBridge event 

(capacityProviderName: null, ec2InstanceId set, full healthChecks[]):
{ "detail-type": "ECS Container Instance Health Change", "source": "aws.ecs",
  "detail": { "capacityProviderName": null, "ec2InstanceId": "i-...",
    "overallStatus": "IMPAIRED",
    "healthChecks": [ { "type": "ACCELERATED_COMPUTE", "status": "IMPAIRED", ... },
                      { "type": "CONTAINER_RUNTIME", "status": "OK", ... } ] } }

Description for the changelog

Enhancement - Add ACCELERATED_COMPUTE health check for the MPS control daemon

Additional Information

Does this PR include breaking model changes? If so, Have you added transformation functions? No

Does this PR include the addition of new environment variables in the README? No

Licensing

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@harishxr
harishxr marked this pull request as ready for review September 2, 2026 23:15
@harishxr
harishxr requested a review from a team as a code owner September 2, 2026 23:15
Comment thread agent/doctor/mps_daemon_healthcheck.go Outdated
@harishxr
harishxr force-pushed the harishxr/mps-ec2-daemon-healthcheck branch from d993f51 to 099681c Compare September 3, 2026 16:57
Comment thread agent/doctor/mps_daemon_healthcheck.go Outdated
Comment thread agent/doctor/mps_daemon_healthcheck.go Outdated
Comment thread agent/doctor/mps_daemon_healthcheck.go Outdated
Comment thread agent/app/agent_mps_healthcheck_unix.go
assert.Equal(t, 2, hc.consecutiveFailures)
}

func TestMpsThirdConsecutiveFailureImpaired(t *testing.T) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Opportunity to use table driven tests?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in fe5d32c

@harishxr
harishxr force-pushed the harishxr/mps-ec2-daemon-healthcheck branch from 099681c to fe5d32c Compare September 3, 2026 20:55
Comment thread agent/doctor/mps_daemon_healthcheck.go Outdated
Comment thread agent/doctor/mps_daemon_healthcheck.go Outdated
@harishxr
harishxr force-pushed the harishxr/mps-ec2-daemon-healthcheck branch from fe5d32c to 4708d48 Compare September 3, 2026 22:36
@harishxr
harishxr force-pushed the harishxr/mps-ec2-daemon-healthcheck branch from 4708d48 to bbbb455 Compare September 3, 2026 22:45
@harishxr
harishxr enabled auto-merge (rebase) September 3, 2026 23:06
@harishxr
harishxr merged commit 0319ba8 into aws:dev Sep 4, 2026
45 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants