fix(cuda): stop sending every RTX 40-series card to the CPU - #1289
Merged
Conversation
… not after Six reports are the same story: #1240, #1246, #1248, #1277, #1283, #1284 — 4 GB and 6 GB cards running an engine that wants 6 GB, each one waiting out the full 300s compute budget to be told the job "was too heavy". The routing layer knew the whole time. The error text even names the card and the figure. The caveat only ever surfaced on the engine-PICK toast, so it reached people who changed engines and nobody whose engine was already selected — the default, or one persisted from a previous session. That is most users. /generate does return X-OmniVoice-Routing, but a response header arrives when the job ends, five minutes too late to be a warning. So the check moves to the chokepoint every synth path shares (api/generate.ts, same argument as the in-flight count). Fire-and-forget: never awaited, so it cannot add latency to the request it warns about; never throws, so an unreachable backend costs a warning rather than a generate; once per engine+reason per session, so it informs instead of nagging. Advisory, not blocking — the driver can page to system RAM and short inputs fit where long ones don't. Extracts routingNotice() as the single frontend mirror of the backend's routing_notice(). Two callers now need "is this verdict worth interrupting for", and two inline copies would drift — invisibly, until someone on DirectML or an unavailable engine gets a hardware warning for a normal pick. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The SM-arch gate required the device's exact tag in get_arch_list(). NVIDIA's rules are not exact, and PyTorch depends on that: SASS is binary-compatible UPWARD within a major version, so the official wheels ship sm_80/sm_86 and deliberately no sm_89 — the 8.6 kernels already cover Ada. Exact matching therefore declared sm_89 unsupported, check_device_compatibility() returned False, and get_best_device() silently returned "cpu". That is every RTX 4060/4070/4080/4090, not just the reporter's card (#1285) — each one running TTS on the CPU on hardware that works fine, with a message telling them their GPU was unsupported. cuda_build_covers() now applies the real rules: sm_XY covers same-major devices with minor >= Y; compute_XY PTX JITs forward to anything newer; an a/f suffix (sm_90a) is architecture-specific and matches exactly. Unparseable entries are skipped, and an empty arch list still degrades to "compatible" — the pre-existing fail-open contract. The remediation text also pointed at a NIGHTLY index for what is a stable supported card; it now names the stable cu128 index. 12 tests: the Ada regression, Jetson Orin (8.7), downward-within-major and cross-major rejection, PTX forward-JIT, arch-specific suffixes, and a genuine sm_120-on-old-wheel mismatch so the gate is proven to still work. Closes #1285 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Contributor
|
| Filename | Overview |
|---|---|
| backend/core/device_caps.py | Replaces exact CUDA architecture matching with compatibility-aware SASS, PTX, and architecture-specific target handling. |
| backend/services/model_manager.py | Updates the incompatible-device remediation message to recommend the stable CUDA 12.8 PyTorch index. |
| tests/test_cuda_arch_compat.py | Adds focused coverage for CUDA architecture compatibility and CPU-fallback prevention. |
Reviews (3): Last reviewed commit: "fix: restore generate.ts and generatePre..." | Re-trigger Greptile
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthroughThe backend now evaluates CUDA builds using forward-compatible SASS/PTX rules. The frontend centralizes routing caveat handling and adds cached, deduplicated generation-time warnings with localized messages. ChangesCUDA architecture compatibility
Generation-time engine caveats
Estimated code review effort: 4 (Complex) | ~45 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 6 | ❌ 3❌ Failed checks (3 warnings)
✅ Passed checks (6 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@frontend/src/utils/generatePreflight.js`:
- Around line 37-42: The enginesCached function must evict a rejected
listEngines promise so subsequent calls can retry after recovery. Attach
rejection handling to the cached promise, clearing cache only when it still
references that failed promise, and add a regression test that verifies a failed
first call is followed by a successful retry.
In `@tests/test_cuda_arch_compat.py`:
- Around line 19-20: Move the application imports of device_caps,
arch_unsupported, and cuda_build_covers out of module scope in
tests/test_cuda_arch_compat.py and resolve them inside a fixture or helper
invoked by each test. Ensure every test receives the freshly resolved symbols at
runtime, avoiding stale sys.modules state.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: ca2dec3a-717c-401a-8901-19702c5eda69
📒 Files selected for processing (29)
backend/core/device_caps.pybackend/services/model_manager.pyfrontend/src/api/generate.tsfrontend/src/i18n/locales/ar.jsonfrontend/src/i18n/locales/de.jsonfrontend/src/i18n/locales/en.jsonfrontend/src/i18n/locales/es.jsonfrontend/src/i18n/locales/fr.jsonfrontend/src/i18n/locales/hi.jsonfrontend/src/i18n/locales/id.jsonfrontend/src/i18n/locales/it.jsonfrontend/src/i18n/locales/ja.jsonfrontend/src/i18n/locales/ko.jsonfrontend/src/i18n/locales/nl.jsonfrontend/src/i18n/locales/pl.jsonfrontend/src/i18n/locales/pt.jsonfrontend/src/i18n/locales/ru.jsonfrontend/src/i18n/locales/sv.jsonfrontend/src/i18n/locales/th.jsonfrontend/src/i18n/locales/tr.jsonfrontend/src/i18n/locales/uk.jsonfrontend/src/i18n/locales/vi.jsonfrontend/src/i18n/locales/zh-CN.jsonfrontend/src/i18n/locales/zh-TW.jsonfrontend/src/test/generatePreflight.test.jsfrontend/src/utils/engineSelectToast.jsfrontend/src/utils/generatePreflight.jsfrontend/src/utils/routingNotice.jstests/test_cuda_arch_compat.py
The tests/** review contract forbids module-level imports of app modules — they go stale under sys.modules pollution from other suites, which is the live cause of #1269's cross-suite failures. Binds core.device_caps per call. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
# Conflicts: # frontend/src/api/generate.ts # frontend/src/test/generatePreflight.test.js # frontend/src/utils/engineSelectToast.js # frontend/src/utils/generatePreflight.js
Conflict markers were committed in the previous merge — `git add` on the directory staged both files as resolved while the markers were still in them. Both belong to #1288 and are unchanged by this PR, so they take main's version verbatim. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Jul 29, 2026
This was referenced Aug 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1285.
Not one card — a whole generation
The SM-arch gate required the device's exact tag to appear in
torch.cuda.get_arch_list(). NVIDIA's compatibility rules are not exact, and PyTorch deliberately relies on that.SASS is binary-compatible upward within a major version: a cubin built for 8.6 runs on any 8.x device with minor ≥ 6. So the official wheels ship
sm_80andsm_86and nosm_89— the 8.6 kernels already cover Ada. The reporter's real cu128 arch list shows exactly this:sm_89is absent by design. Exact matching read that as "unsupported",check_device_compatibility()returned False, andget_best_device()silently returned"cpu".RTX 4060, 4070, 4080 and 4090 are all sm_89. Every one of them has been running TTS on the CPU on hardware that works fine — and being told their GPU was unsupported.
The fix
cuda_build_covers()applies the actual rules:sm_XYcompute_XYsm_90a/compute_100fUnparseable entries are skipped rather than guessed at, and an empty arch list still degrades to "compatible", preserving the pre-existing fail-open contract.
Also: the remediation text pointed users at a nightly index for a stable, long-supported card. It now names the stable cu128 index.
Tests
12 cases in
tests/test_cuda_arch_compat.py, built on the arch list verbatim from the report: the Ada regression (4060 + 4090), Jetson Orin (8.7), downward-within-major rejection (8.9 cubin does not run on 8.6), cross-major rejection, PTX forward-JIT, arch-specific suffixes, unparseable entries, and a genuine sm_120-on-an-old-wheel mismatch so the gate is proven to still catch real problems.Direct before/after on the reporter's arch list:
test_rocm_arch_gate.py+test_device_caps.py+ new suite: 50 passing, so #1228's ROCm branch is untouched.CUDA architecture detection now uses a new
cuda_build_covers()compatibility check (upward SASS within the same major, forward PTX JIT, exact matching forsm_*suffix targets, skipping malformed entries, and fail-open for empty/unknown lists) instead of requiring an exactsm_XX/compute_XXmatch, fixing cases like RTX 40-series (sm_89) being incorrectly marked unsupported; this is covered by expanded CUDA coverage tests including Ada/Jetson Orin, PTX forward-JIT, suffix/non-forwarding behavior, malformed-entry handling, and true mismatches. Generation-time UX also adds fire-and-forget preflight to warn users when the selected engine may run slowly, driven by shared routing-notice logic and new localized “generate caveat” strings. Main risk to review: the compatibility rule boundaries (especially suffix and PTX/SASS forward-compat interpretation) and the “warn without awaiting” behavior to ensure warnings don’t misfire or degrade error handling in edge cases.