Skip to content

fix(portal): bound the recovery-lock wait and log lock transitions and reopens - #255

Merged
VallierDev merged 1 commit into
VallierDev:mainfrom
hsxian:fix/portal-recovery-observability
Nov 20, 2026
Merged

VallierDev merged 1 commit into
VallierDev:mainfrom
hsxian:fix/portal-recovery-observability

Conversation

@hsxian

@hsxian hsxian commented Nov 20, 2026

Copy link
Copy Markdown
Contributor

Problem

A desktop connection that starts while the session is locked stays on the synthetic locked video until the auto unlock, and only the maintain loop reacts to that unlock. During a real incident this morning, a chain of such connections produced no maintain activity at all: no reopen log, no recovery-deferred log, no outcome — the viewer sat on black frames until its video timeout, and there was no way to tell where the pipeline stalled.

Three blameless suspects can all go silent the same way:

  1. an attempt cancelled by backoff or supersession,
  2. the opening gate held by a zombie worker,
  3. recover() spinning forever on a recovery lock another opener never released.

Suspect 3 is a genuine hazard on its own: the try_lock retry loop has no bound, so a wedged holder turns every future attempt into an attempt that never ends.

Fix

  • portal.rs recover(): bound the recovery.try_lock() wait to 15s and return code 3 (Retry), so a wedged holder surfaces as "recovery deferred" instead of an attempt that never ends.
  • desktop_session.rs maintain(): log each lock-state transition the loop reacts to (Portal desktop lock transition: state=... defer=...) and log when it stops because a newer connection superseded it.
  • desktop_session.rs portal_attempt(): log every real reopen (Portal attempt: reopening the capture session) so a hang after this line pins the stall to the blocking open/recovery worker.

These logs proved their worth the same day: the next "connect to a locked machine" incident walked through lock transition → reopening → silence, plus an xdg-desktop-portal-gnome dialog log — which immediately ruled out all three code suspects and identified the real cause (a consent dialog waiting for a click nobody was there to make).

Tested: desktop_session 22 passed, portal 27 passed, cargo check clean.

…d reopens

A desktop connection that starts while the session is locked stays on the
synthetic locked video until the auto unlock, and only the maintain loop
reacts to that unlock. This morning a chain of such connections produced
no maintain activity at all: no reopen log, no recovery-deferred log, no
outcome - the viewer sat on black frames until its video timeout. Three
blameless suspects can all go silent the same way: an attempt cancelled
by backoff or supersession, the opening gate held by a zombie worker, or
recover() spinning forever on a recovery lock another opener never
released.

- recover(): bound the try_lock wait to 15s and return code 3, so a wedged
  holder surfaces as 'recovery deferred' instead of an attempt that never
  ends.
- maintain(): log each lock-state transition the loop reacts to, and log
  when it stops because a newer connection superseded it.
- portal_attempt(): log every real reopen so a hang after this line pins
  the stall to the blocking open/recovery worker.
@VallierDev

Copy link
Copy Markdown
Owner

看过 9fd7ffa:recover() 的锁等待限 15s 后返回 code 3,经 classify() 归为 Retry、按退避重试,不会终止连接;新增三条日志不含用户数据,范围小而清楚。等 CI 跑完并请 Codex 审查,通过后合并。

@codex review

🤖 Generated with Claude Code

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-20T02:11:00.728479Z 9fd7ffa Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Keep them coming!

Reviewed commit: 9fd7ffa6f0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@VallierDev
VallierDev merged commit 1d939fc into VallierDev:main Nov 20, 2026
6 checks passed
@VallierDev

Copy link
Copy Markdown
Owner

Codex 复审无问题、CI 全绿,已 squash 合并,将随下个版本发布。感谢!

🤖 Generated with Claude Code

@hsxian
hsxian deleted the fix/portal-recovery-observability branch November 20, 2026 03:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants