post-mortem: silently-stale config sync reported as done + accidental tmux window kill

[ycpss91255]

Summary

During the tmux status-bar work, two related failures occurred, both of the same underlying class: a destructive or state-changing action was performed and reported as successful WITHOUT verifying at the point where it actually takes effect.

1. A config fix (window-status-separator "" + status-style backstop in ~/.config/tmux/status-length.sh) was repeatedly reported as "synced to remote and verified", but the deployed file on the remote machine never actually contained the fix. The user kept seeing the bug ("window tabs still separated") because the change had never really landed there.
2. While verifying that fix on the live remote session, an unrelated user window (rpi_realsense in the guotai session) was accidentally killed by a reckless index-based kill-window heuristic used to clean up a temporary test window.

Neither is a tmux bug. Both are process/verification-discipline failures on the assistant side. This issue records the determination and the corrective rules.

Failure 1: silently-stale sync reported as done

What was observed

• Local ~/.config/tmux/status-length.sh contained the backstop lines (tmux set -g status-style "fg=#c6d0f5,bg=#303446" and tmux set -g window-status-separator "").
• Remote ~/.config/tmux/status-length.sh did NOT contain them (grep returned empty; sha256sum differed between machines).
• The hooks that run the script WERE correctly set on the remote server (client-attached/session-created/client-resized all fire run-shell -b .../status-length.sh), so the script ran on every attach/resize -- but it was running the STALE file, which never set the separator, so window-status-separator stayed at tmux's built-in default " " (a single space), which is exactly the visible gap between window tabs.

Root cause (mechanical causes ruled out)

• Remote disk 66% used (not full), target dir writable, and a fresh scp re-sync landed correctly with matching sha256 -- so scp did not silently fail, and it was not a permissions/disk problem.
• No stale backup copy of the file existed that could have overwritten a good version.
• Therefore: after the FINAL local edit (adding the separator line), status-length.sh was simply never re-scp'd to the remote. The earlier "verified on remote" step was a MISATTRIBUTION: the verification command block ran the remote's (stale) script AND a manual tmux set -g window-status-separator "" in the same block; the resulting empty separator came from the manual set, not from the script/sync, and was wrongly credited to the fix having landed.
• Compounding failure: when the user reported the bug was still present, the assistant re-theorized (an architecture discussion about where the reset "should" live) INSTEAD OF first re-checking the actually-deployed file. A single sha256sum comparison at that moment would have caught the stale file immediately.

Blast radius (determined)

• A full sha256sum comparison of every file under ~/.config/tmux/ and ~/.config/tmux-powerline/ (excluding plugins/ and *.bak*/.pre-* backups) between the two machines was run AFTER the re-sync. Only status-length.sh had drifted; everything else matched. So the drift was limited to this one file, not a systemic sync collapse.

Failure 2: accidental window kill

What happened

• To confirm the separator fix held across a natural event, a temporary window was created on the live guotai session (tmux new-window -d -t guotai), then "cleaned up" by killing the highest-index window (last=$(tmux list-windows ... | tail -1); tmux kill-window -t guotai:$last), plus an earlier malformed tmux kill-window -t guotai:$ (literal $) whose error was suppressed with 2>/dev/null.
• Net result: the user's real rpi_realsense window was destroyed along with the test window. Its running contents (an SSH/realsense session) could not be recovered by the assistant.

Root cause

• A DESTRUCTIVE target was chosen by POSITION/INDEX heuristic ("the last window", "the highest index") on a live session whose window set was not exclusively owned by the test, instead of by the EXACT identity of the object that was created. new-window does not guarantee the created window is the highest index, and suppressing the malformed command's stderr hid a second unintended kill.

Corrective rules (to prevent recurrence)

1. Sync is not "done" until re-verified at the point of use. After the FINAL edit to any file that is meant to be deployed to another machine, re-scp and immediately sha256sum-compare local vs remote for THAT exact path. Never report "synced" from memory of an earlier sync; the edit that invalidates a prior sync is usually the last one.
2. Never attribute a verification result to a mechanism without isolating it. If a test block both deploys a change AND manually sets the same state, the observed state proves nothing about the deployment. Verify the deployed artifact directly (grep/sha of the file that will actually run), separately from any manual state-setting.
3. When a user says "still broken", re-check the deployed reality FIRST, before any re-theorizing. The cheapest first move is confirming the fix is actually present where it runs (sha256/grep on the live path), not designing a better fix.
4. Never choose a destructive target by position/index on shared/live state. Capture the exact id of anything created (wid=$(tmux new-window -d -P -F '#{window_id}' ...)) and destroy only that id. Do not suppress stderr on destructive commands; a malformed target must surface, not silently hit something else.
5. Prefer a throwaway server/socket for verification. Structural tests (does the hook reset the separator?) should run on an isolated tmux -L test server, never on the user's live sessions, so a cleanup mistake cannot touch real windows.

Status

• Failure 1: fixed. status-length.sh re-synced; sha256 now matches; full-tree comparison confirms no other drift; the live remote server's window-status-separator is now "" and holds across new-window events.
• Failure 2: not recoverable by the assistant. The rpi_realsense window's contents were lost; the user was informed immediately and offered to recreate it.