Skip to content

fix(proxies): keep the running copy when a redeploy's tunnel cannot start - #632

Closed
GeiserX wants to merge 6 commits into
mainfrom
fix/threads-desktop-staged-tunnel
Closed

GeiserX wants to merge 6 commits into
mainfrom
fix/threads-desktop-staged-tunnel

Conversation

@GeiserX

@GeiserX GeiserX commented Nov 7, 2026 •

Copy link
Copy Markdown
Owner

Redeploying a proxied copy could leave the proxy with no copy at all. deploy() removed the old earner and tunnel first, then started the new tunnel. If the new tunnel failed to start, for example because its image was not available locally or the proxy did not answer within the ready timeout, the old copy was already gone. Only changing the order of the calls does not work: the tunnel name is deterministic, so Docker refuses a second container with that name while the old one exists.

The fix:

  • The new tunnel starts under cashpilot-tunnel-<slug>-p<id>-next, with the final hostname, while the old pair keeps running. If it cannot prove an exit, it is removed and the running copy is untouched.
  • Once it is ready, the old pair is removed (earner first), the tunnel is renamed to its final name, and the earner is created in its namespace as before. If the rename fails, the staged tunnel is removed.
  • The swap and the heartbeat's status() share a lock, so a heartbeat cannot start an earner that is being removed or read a half-swapped pair. status() skips staged tunnels, so their probe traffic never counts toward a cap.
  • A staged tunnel left by a deploy that died midway is replaced by the next deploy, but only if it carries this copy's labels. Otherwise the deploy refuses. The worker's start-up sweep also removes staged tunnels, since they hold the proxy login and the heartbeat never reports them.

Tests in tests/test_proxy_tunnel.py (all five fail on main):

  • test_sidecar_is_ready_before_the_earner_is_created: the order is now staged ready, old removed, renamed, earner.
  • test_a_sidecar_that_cannot_start_keeps_the_running_copy
  • test_a_failed_promotion_removes_the_staged_sidecar
  • test_a_staged_sidecar_left_behind_is_replaced_but_never_a_stranger
  • test_a_staged_sidecar_of_a_deploy_that_died_is_swept
  • test_the_heartbeat_never_reports_a_staged_sidecar also covers the status() filter.

Follows up the review thread on #576 at app/proxy_tunnel.py:344. Desktop's proxied deploy had the same order and gets the same fix in GeiserX/CashPilot-Desktop#267.

Review follow-ups: the worker-start sweep removes only staged sidecars older than 300 s (STAGED_STALE_SECONDS), well past the ready timeout, so a deploy still in flight keeps its sidecar. Deploys of one copy run one at a time under a per-copy lock. Tests: test_a_young_staged_sidecar_is_left_to_its_deploy and test_two_deploys_of_one_copy_take_turns.

…tart

deploy() removed the old earner and tunnel before the new tunnel started, so a
tunnel that failed to start (an image missing locally, a proxy that does not
answer) left the proxy with no copy at all. The new tunnel now starts under a
staged name next to the running copy. Only once it has proved an exit does the
old pair go and the tunnel take the final name, under a lock the heartbeat's
status() shares. The heartbeat never reports a staged tunnel, and a worker
start sweeps one a dead deploy left behind.
@coderabbitai

coderabbitai Bot commented Nov 7, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 59 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 45ad696 and 5f36c10.

Files selected for processing: 2

  • app/proxy_tunnel.py
  • tests/test_proxy_tunnel.py
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@codecov

codecov Bot commented Nov 7, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.74468% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 93.98%. Comparing base (2d15dae) to head (5f36c10).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
app/proxy_tunnel.py 95.74% 2 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #632      +/-   ##
==========================================
+ Coverage   93.96%   93.98%   +0.01%     
==========================================
  Files          74       74              
  Lines       15687    15731      +44     
==========================================
+ Hits        14741    14784      +43     
- Misses        946      947       +1     
Files with missing lines Coverage Δ
app/proxy_tunnel.py 89.50% <95.74%> (+0.73%) ⬆️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

GeiserX added a commit to GeiserX/CashPilot-Desktop that referenced this pull request Nov 7, 2026
…tart (#267)

* fix(proxies): keep the running copy when a redeploy's tunnel cannot start

deployProxied removed the old pair before the new tunnel started, so a tunnel
that failed to start left no copy. The new tunnel now starts as <name>-next
beside the running copy; once it has an exit, the old pair goes and the tunnel
is renamed, under a lock the status read shares. The status read skips staged
tunnels and sweeps stale ones. Mirrors GeiserX/CashPilot#632.

* test(proxies): cover a failed promotion of the staged tunnel

* fix(proxies): run one deploy per proxied copy at a time

* fix(proxies): never remove a staged tunnel another deploy still owns

A deploy removed any staged tunnel of ours as a leftover, and the status read
swept any older than five minutes. A second app on the same Docker host, or a
deploy still waiting for the swap, could lose its tunnel after the old pair was
gone. A deploy now refuses a staged tunnel younger than the sweep age, and the
sweep skips one whose copy is being deployed in this app.
@GeiserX

GeiserX commented Nov 7, 2026

Copy link
Copy Markdown
Owner Author

Landed in #653 (f681889), which combined this fix with the others into one release.

@GeiserX GeiserX closed this Nov 7, 2026
@GeiserX
GeiserX deleted the fix/threads-desktop-staged-tunnel branch November 7, 2026 21:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant