fix(processing): scale TCD timeout for large orthos - #392
Conversation
|
Warning Review limit reached
More reviews will be available in 30 minutes and 4 seconds. Learn how PR review limits work. Your organization has run out of usage credits. Purchase more credits in the billing tab to continue. ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (4)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
would discard for CPU vs GPU fix see #430 |
Summary
Why
Dataset 10183 recovered from stale queue state but failed genuine treecover processing after the TCD container reached the fixed 4h wall-time limit. PR #389 made the wait timeout configurable and correctly classified wrapped Docker wait timeouts; this follow-up avoids applying the same fixed 4h limit to very large orthos.
Validation
Risk
This can let very large legacy treecover_v1 jobs occupy the processor longer, up to the configured cap. It does not requeue failed datasets or mutate production by itself; dataset 10183 still needs an explicitly approved production rerun after deployment.
Note
Medium Risk
Large treecover jobs can hold processor slots up to 12h per container; behavior change is limited to timeout/wait logic, not inference output.
Overview
Replaces the fixed TCD Docker wait timeout with a pixel-scaled policy so very large orthos can run longer without abandoning jobs at the previous 4h wall clock.
_compute_tcd_timeout_policykeeps the configured base timeout (default 4h) for inputs at or below 2B pixels, scales timeout proportionally for larger reprojected rasters, and caps at 12h (TCD_CONTAINER_TIMEOUT_MAX_SECONDS/TCD_CONTAINER_TIMEOUT_BASE_PIXELSin settings and.env.example).predict_treecoverreads width/height after EPSG:3395 reprojection and passes them into_run_tcd_pipeline_container, which logs the chosen policy and uses it forcontainer.wait(timeout=...).Timeout regression tests cover base vs scaled/capped policy and assert the wait timeout passed to Docker (including the 12h case for a ~9.5B-pixel ortho).
Reviewed by Cursor Bugbot for commit 69aaccd. Configure here.