tests_v2: expand Linux socket-test coverage beyond the current stable baseline

[Jesssullivan]

Summary

Track expansion of Linux tests_v2 socket coverage beyond the current stable baseline.

Current state as of 2026-04-21

• scripts/run-socket-tests.sh now uses an explicit BASELINE allowlist instead of the old blacklist model
• the stable baseline currently contains 15 tests; failures here fail the self-hosted Linux job
• CMUX_TEST_PHASE1=1 now enables 14 additional candidate tests as non-fatal observations
• .github/workflows/test-socket.yml currently enables that candidate gate on the self-hosted Linux runner
• tests_v2/ currently has 123 test files on disk, but most remain outside the Linux baseline because they still depend on broader browser, debug-helper, CLI, or remote/fleet parity

What this issue now covers

1. keep the Linux baseline small, explicit, and trustworthy
2. observe candidate results on real CI and promote consistently green tests into BASELINE
3. land the next highest-yield Linux command coverage without pretending the entire suite is ready
4. keep runner-selection mechanics separate from larger parity families

Current promotion lane

The current phase-1 candidate set already includes:

• surface.action.* coverage slices
• workspace.action
• system_tree
• auth and notification helper coverage
• pane.resize
• the current sprint-B parity bundle in test_sprint_b_core_parity

The immediate next move is not another runner rewrite. It is recording candidate behavior and promoting the stable subset.

Separate follow-on lanes

These remain parallel or follow-on work and should not be hidden inside runner wording:

• #206 Linux WebAuthn bridge completion
• broader browser automation parity
• command-palette and debug-helper parity
• Linux CLI parity beyond the current socket baseline
• remote/fleet transport work under #201

Exit criteria

• at least one full candidate cohort has been observed on CI
• stable candidate tests have been promoted into BASELINE
• Linux socket-test coverage is broader than the current 15-test baseline without regressing the green self-hosted lane
• remaining unsupported families are explicitly tracked as parity follow-up, not mislabeled as “already covered by tests”

References

• scripts/run-socket-tests.sh
• .github/workflows/test-socket.yml
• docs/distro-testing-readiness-plan.md
• docs/linear-qa-shard-punchlist.md

[Jesssullivan]

Audit correction — surface.move/reorder is blocked by pane grouping, not stub-replacement

After looking at cmux-linux/src/socket.zig:1007-1032 (handleSurfaceCreate), the Linux data model is currently 1:1 — every new surface becomes its own pane via split_tree.splitPane. There is no concept of "multiple surfaces sharing a pane."

tests_v2/test_surface_move_reorder_api.py lines 33-50 set up the test by creating two surfaces in the same pane, then asserting that moving one to a different pane removes it from the source. With the 1:1 pane:surface model, the source pane disappears when the surface leaves, and there is no second surface to reorder against.

So Phase 2 item #1 ("real surface.move/reorder") is mis-scoped. Implementing the stubs as real handlers does not unblock test_surface_move_reorder_api — that test needs pane grouping first, which is a much larger architectural change (likely Phase 3 territory under "feature parity").

Updated Phase 2 priorities (most-tractable first):

1. surface.action rename subset → unblocks test_surface_list_custom_titles, contributes to 3 command-palette tests later. macOS reference: Sources/TerminalController.swift surface.action handler. Self-contained.
2. browser.eval + browser.get.title + browser.wait → unblocks test_browser_file_url_load. Clear contract, well-bounded. Foundational for further browser parity.
3. pane.resize → unblocks 2 tests but they assert PTY/scrollback content, so likely also need real surface (CMUX_NO_SURFACE=1 may not be enough).

Will pursue (1) next. Leaving (2) and (3) for follow-up PRs.

[Jesssullivan]

Phase 2 update — surface.action rename + metadata mutations: PR #218

Implements the trivial subset of v2TabAction on Linux:

• rename, clear_name — drop Panel.custom_title
• pin, unpin — toggle Panel.is_pinned
• mark_read, mark_unread — toggle Panel.is_manually_unread

All map directly to fields already present on Panel in cmux-linux/src/workspace.zig. Sidebar refresh wired the same way as handleWorkspaceRename. Both surface.action and tab.action route through the same handler (Linux is currently 1:1 panel:pane).

Tab-relative close_* and new_* actions are recognized but explicitly return "action not implemented on linux" so callers can detect the gap programmatically without guessing.

Test coverage:

• tests_v2/test_surface_action_rename.py — pure-socket, zero GUI/CLI deps; runs against any cmux daemon that exposes the v2 socket. Will be picked up by the existing socket runner automatically (no test_surface_action_* exclusion).

After #217 (Phase 1 allowlist) merges, follow-up commit will promote test_surface_action_rename into BASELINE.

Still outstanding from this issue:

• test_surface_move_reorder_api and test_surface_list_custom_titles are blocked by deeper gaps (Linux 1:1 panel:pane model + macOS-only CLI binary lookup). Phase 3 territory.

[Jesssullivan]

Status update — 2026-04-25

Coverage counts have moved since the older tracker text:

• tests_v2/ has 123 test_*.py files on disk
• scripts/run-socket-tests.sh currently has 18 Linux baseline tests
• phase-1 currently runs 11 candidate tests as non-fatal observations

Latest self-hosted socket run on codex/socket-ci-truthfulness made the contract useful but exposed one baseline failure: test_surface_action_close_variants. The next move should be fixing that baseline behavior, or explicitly reclassifying it, before promoting more candidate tests.

The 50%+ Linux socket-coverage goal remains valid, but the current floor should be described as 18 baseline + 11 candidates, not the older 14/111 or 15-test wording.

[Jesssullivan]

2026-04-25 reconciliation note:

Current checked-in socket-test allowlist state:

• tests_v2/ has 123 test_*.py files.
• scripts/run-socket-tests.sh has 18 baseline tests in BASELINE.
• CANDIDATES_PHASE1 has 11 non-fatal candidate tests.
• The latest test-socket.yml run on main from PR Linux CI, packaging, lmux, and Ghostty parent pin #244 is still queued on the self-hosted runner.

Next useful action is to wait for one fresh self-hosted run, then either promote consistently green phase-1 candidates or split the next failing candidate into a focused Linux parity issue. Do not expand by source-shape tests; only runtime socket behavior should count.

[Jesssullivan]

Socket-test artifact readout from the last completed self-hosted runs.

Evidence:

• Last green run: https://github.com/Jesssullivan/cmux/actions/runs/24760778180
• Follow-up failed run: https://github.com/Jesssullivan/cmux/actions/runs/24762421036
• Current post-merge socket run remains queued: https://github.com/Jesssullivan/cmux/actions/runs/24941793858
• Repo-visible runner state: only honey-cmux, currently offline. That runner has KVM/distro labels; test-socket.yml asks for cmux-nix, so runner-label visibility needs reconciliation before fresh proof is reliable.

Last green run:

• 18/18 baseline passed.
• 7/11 phase-1 candidates passed: test_workspace_create_background_starts_terminal, test_workspace_create_initial_env, test_workspace_action, test_auth_login, test_system_tree, test_app_simulate_active, test_sprint_b_core_parity.
• 4/11 candidates failed non-fatally: test_browser_open_split_reuse_policy, test_notification_create_for_target, test_surface_report_tty, test_pane_resize.

Follow-up failed run:

• Baseline failure: test_surface_action_close_variants failed with FAIL: source pane not found.
• Candidate improvements: test_surface_report_tty and test_pane_resize passed in that run.
• Candidate regressions/still-failing: browser split/reuse, notification target validation, sprint B remote-workspace case.

Next practical move: restore a matching self-hosted runner for cmux-nix or update workflow labels to the intended live runner authority, rerun test-socket.yml on current main, then promote only candidates that are green on current-head artifacts.