Fail-safe hardening: never leave the R730 uncooled#1
Merged
Conversation
The controller puts the iDRAC into manual fan mode and is then solely responsible for cooling. Previously, several failure paths abandoned that responsibility with fans potentially pinned low. Now every failure path fails safe: - Restore BMC auto mode on every abnormal exit: API server errors route through an error channel (no more log.Fatalf skipping cleanup), the control loop recovers panics, and a sync.Once restore choke point fires exactly once. Only SIGKILL/power loss can skip it. - Sensor loss fails UP: read errors hold speed instead of reading as 0C, empty parses (CPU and GPU) are errors, and 3 consecutive failures enter failsafe (BMC auto mode restored, retried every tick until confirmed, surfaced as restore_pending in /api/status). - Write failures get the same watchdog; write-failure failsafe is sticky (no manual/auto oscillation) until restart. - All ipmitool/nvidia-smi execs have timeouts: max(3s, min(10s, interval)). - Critical temps (critical_cpu_temp/critical_gpu_temp, validated at load) bypass step-ramping straight to max speed, overriding manual overrides. - Config that parses but fails safety validation refuses to start (BMC keeps auto control) instead of silently falling back to defaults and discarding iDRAC credentials; missing config still starts on defaults. - Truthful status: currentSpeed only updates after confirmed writes; failsafe_active, failsafe_reason, restore_pending, last_write_failed exposed; currentZone race fixed. Tests: table-driven coverage for calculateTarget, parsers, failsafe thresholds/stickiness/restore-retry, panic recovery, config resolution. go vet and go test -race clean. Task: dex o03q6gcj - Tier 1: Fail-safe hardening Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EQmqj1Hj8DL3GGAcBDNm8v
This was referenced Jul 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The controller switches the iDRAC into manual fan mode and is then solely responsible for cooling the server. This PR makes every failure path fail safe — restore BMC auto control or hold/raise speed — instead of leaving fans pinned low:
sync.Oncerestore choke point; the control loop recovers panics; only SIGKILL/power loss can skip the handback.restore_pendingin/api/status).max(3s, min(10s, interval)).critical_cpu_temp/critical_gpu_tempkeys), overriding even manual overrides.cpu_threshold ≥ 85must setcritical_cpu_tempexplicitly.currentSpeedupdates only after confirmed writes;failsafe_active,failsafe_reason,restore_pending,last_write_failedexposed.Review & verification
tl6vr184); reviewer empirically reproduced and re-verified each blocker.go vet,go build,go test -race -count=1 ./...clean (5 new test files).Task: dex o03q6gcj — Tier 1: Fail-safe hardening
🤖 Generated with Claude Code