Skip to content

fix(processing): scale TCD timeout for large orthos - #392

Open
JesJehle wants to merge 1 commit into
mainfrom
fix/adaptive-tcd-timeout
Open

fix(processing): scale TCD timeout for large orthos#392
JesJehle wants to merge 1 commit into
mainfrom
fix/adaptive-tcd-timeout

Conversation

@JesJehle

@JesJehle JesJehle commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

Summary

  • scale TCD container wait time from a 4h base by TCD input pixel count, capped at 12h
  • log the selected timeout policy for large treecover runs
  • document the new max/base-pixel env knobs and extend timeout regression coverage

Why

Dataset 10183 recovered from stale queue state but failed genuine treecover processing after the TCD container reached the fixed 4h wall-time limit. PR #389 made the wait timeout configurable and correctly classified wrapped Docker wait timeouts; this follow-up avoids applying the same fixed 4h limit to very large orthos.

Validation

  • PYTHONPATH=. uv run --with pytest --with requests --with docker --with pydantic-settings --with python-dotenv --with rasterio==1.4.2 --with pydantic-partial --with pydantic-geojson pytest --noconftest processor/tests/test_tcd_pipeline_container_timeout.py -q
  • ruff check processor/src/treecover_segmentation_oam_tcd/predict_treecover.py processor/tests/test_tcd_pipeline_container_timeout.py shared/settings.py
  • ruff format --check processor/src/treecover_segmentation_oam_tcd/predict_treecover.py processor/tests/test_tcd_pipeline_container_timeout.py shared/settings.py
  • git diff --check

Risk

This can let very large legacy treecover_v1 jobs occupy the processor longer, up to the configured cap. It does not requeue failed datasets or mutate production by itself; dataset 10183 still needs an explicitly approved production rerun after deployment.


Note

Medium Risk
Large treecover jobs can hold processor slots up to 12h per container; behavior change is limited to timeout/wait logic, not inference output.

Overview
Replaces the fixed TCD Docker wait timeout with a pixel-scaled policy so very large orthos can run longer without abandoning jobs at the previous 4h wall clock.

_compute_tcd_timeout_policy keeps the configured base timeout (default 4h) for inputs at or below 2B pixels, scales timeout proportionally for larger reprojected rasters, and caps at 12h (TCD_CONTAINER_TIMEOUT_MAX_SECONDS / TCD_CONTAINER_TIMEOUT_BASE_PIXELS in settings and .env.example). predict_treecover reads width/height after EPSG:3395 reprojection and passes them into _run_tcd_pipeline_container, which logs the chosen policy and uses it for container.wait(timeout=...).

Timeout regression tests cover base vs scaled/capped policy and assert the wait timeout passed to Docker (including the 12h case for a ~9.5B-pixel ortho).

Reviewed by Cursor Bugbot for commit 69aaccd. Configure here.

@coderabbitai

coderabbitai Bot commented Jun 11, 2026

Copy link
Copy Markdown

Warning

Review limit reached

@JesJehle, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 30 minutes and 4 seconds. Learn how PR review limits work.

Your organization has run out of usage credits. Purchase more credits in the billing tab to continue.

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 3056ce8b-7532-4f1d-b566-6bebb57018e5

📥 Commits

Reviewing files that changed from the base of the PR and between d74d84e and 69aaccd.

📒 Files selected for processing (4)
  • .env.example
  • processor/src/treecover_segmentation_oam_tcd/predict_treecover.py
  • processor/tests/test_tcd_pipeline_container_timeout.py
  • shared/settings.py
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/adaptive-tcd-timeout

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@cmosig

cmosig commented Jun 23, 2026

Copy link
Copy Markdown
Collaborator

would discard for CPU vs GPU fix see #430

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants