fix(devtools): authenticate sudo before minikube tunnel

[Shriprasad-P]

PR Type

• Bug fix
• New feature
• Core Runtime change (higher bar -- see CONTRIBUTING.md)
• Docs / tooling
• Refactoring

Summary

Fixes #2605.

metaflow-dev up backgrounds minikube tunnel before any required sudo authentication occurs. When Minikube needs an interactive sudo password, that prompt happens in a background job, so the password cannot be entered reliably.

This change runs sudo -v in the foreground first. Failed or cancelled authentication exits with a clear error and does not start the tunnel or Tilt. After a successful preflight, the existing background tunnel and foreground Tilt lifecycle are unchanged.

Issue

Fixes #2605

Reproduction

Runtime: local metaflow-dev / Minikube (macOS and Linux)

Commands to run:

# With no cached sudo timestamp:
metaflow-dev up
# or:
cd devtools && make up

Where evidence shows up: parent console while start.sh is running.

Before (error / log snippet)

minikube tunnel is started as a background job:
  minikube tunnel &

If sudo credentials are not cached, minikube later asks:
  [sudo] password for user:

Because the tunnel process is no longer in the terminal's foreground
process group, the password prompt cannot be completed interactively
(typically SIGTTIN / a stuck prompt).

After (evidence that fix works)

start.sh now runs `sudo -v` in the foreground before:
  minikube tunnel &

Cancelled/failed sudo prints an error and exits 1 without starting
the tunnel or Tilt. Successful sudo caches a timestamp so the
background tunnel can reuse it for privileged route setup.

Root Cause

The generated devtools/.devtools/start.sh previously launched:

minikube tunnel &

Minikube uses sudo to configure host routes for the tunnel on macOS and Linux. When sudo credentials are not cached, sudo needs to read a password from the controlling terminal.

A background job is not in the terminal's foreground process group. Interactive reads from the terminal can stop the process with SIGTTIN, leave the prompt unusable, and prevent the tunnel from starting.

The invariant that was violated: privileged authentication must happen while the process still has foreground terminal access.

Why This Approach

Foreground sudo -v is the smallest change that restores that invariant:

• It does not store or pipe passwords.
• It does not use sudo -S.
• It does not modify /etc/sudoers or configure passwordless sudo (Minikube is downloaded to a local path, so passwordless sudo is not a general solution).
• It does not remove minikube tunnel or change Tilt's foreground lifecycle.
• The sudo timestamp from sudo -v is then available for Minikube's later privileged operations.

A one-shot kill -0 after backgrounding the tunnel was considered and rejected: it races with process startup and is not required to fix #2605. Detecting a later tunnel crash before Tilt starts is pre-existing behavior and is out of scope for this bug.

Changes

• Extract start.sh generation into a generate-start-sh Make target so the generated lifecycle can be tested without Docker, Minikube, or Tilt.
• Run foreground sudo -v after service selection and before minikube tunnel &.
• On authentication failure or cancellation, print an actionable error and exit 1 so the tunnel and Tilt are not started.
• Keep minikube tunnel backgrounded, keep tilt up in the foreground, and keep trap "kill 0" EXIT plus wait.
• Leave the interactive metaflow-dev tunnel target unchanged.

Failure Modes Considered

1. Cancelled or failed sudo: startup stops before the tunnel or Tilt; the user can retry metaflow-dev up or use metaflow-dev tunnel in another terminal.
2. Already-cached sudo timestamp: sudo -v succeeds immediately with no extra prompt.
3. Tunnel still needs sudo after the preflight: Minikube can reuse the refreshed sudo timestamp. This does not grant passwordless sudo permanently.
4. Immediate tunnel process failure after backgrounding: Tilt may still start. That was already true before this PR (minikube tunnel & then tilt up) and is not changed here.

Tests

• Unit tests added/updated
• Reproduction script provided (required for Core Runtime)
• CI passes (GitHub Actions on this fork PR are waiting for maintainer approval: action_required)
• If tests are impractical: explain why below and provide manual evidence above

Locally executed:

python -m pytest test/unit/devtools/test_start_sh.py -v --tb=short
4 passed

PYTHONPATH=. python -m pytest test/unit/test_metaflow_version.py -v --tb=short
23 passed

python -m black --check test/unit/devtools/test_start_sh.py
1 file would be left unchanged

make -f devtools/Makefile generate-start-sh
bash -n devtools/.devtools/start.sh
OK

git diff --check
clean

The new tests prove the regression is fixed by executing the generated script against mock sudo / minikube / tilt binaries:

• sudo -v runs to completion before minikube tunnel starts (a delayed successful sudo would have let the tunnel start first if it were backgrounded).
• Failed/cancelled sudo prevents tunnel startup and Tilt startup.
• The tunnel remains a background process while Tilt stays in the foreground, and trap "kill 0" EXIT still cleans up the tunnel.
• Generated shell syntax is valid (bash -n), sudo -v is not backgrounded, and tilt up is not backgrounded.

Manual Validation

The generated start.sh was inspected and checked with bash -n.

A live metaflow-dev up with expired sudo credentials was not run locally, because it would provision Minikube/Tilt and require interactive sudo.

Non-Goals

• Changing Minikube networking architecture
• Removing the tunnel
• Waiting for the tunnel to become fully ready before Tilt
• Reworking Tilt service exposure
• Modifying sudo configuration
• Refactoring unrelated devtools logic

Scope

Intentionally modified:

• devtools/Makefile
• test/unit/devtools/test_start_sh.py

AI Tool Usage

• No AI tools were used in this contribution
• AI tools were used (describe below)

Cursor (Grok) was used to inspect the issue/PR, review the generated startup lifecycle, run tests, refresh the branch against origin/master, and update this description. The approach (sudo -v in the foreground, keep the tunnel backgrounded) was reviewed against the Makefile, tests, and issue discussion. All generated code was reviewed and the listed tests were actually executed.

[greptile-apps]

Greptile Summary

This PR fixes interactive sudo authentication for metaflow-dev up by validating credentials in the foreground before starting the background Minikube tunnel.

• Extracts start.sh generation into a testable Make target.
• Stops startup when sudo authentication fails or is cancelled.
• Preserves the background-tunnel and foreground-Tilt lifecycle while limiting cleanup to tracked background jobs.
• Adds regression coverage for ordering, authentication failure, syntax, and process cleanup.

Confidence Score: 5/5

The PR appears safe to merge; the prior race-prone tunnel liveness check was removed and no new actionable failures remain.

The foreground sudo -v completes before the tunnel is launched, failed authentication exits before either tunnel or Tilt startup, and the generated script retains explicit cleanup for its tracked background tunnel. The previous tunnel-exit race finding is no longer outstanding because its thread was resolved after the one-shot kill -0 check was removed.

Important Files Changed

Filename	Overview
devtools/Makefile	Generates the startup script separately, performs foreground sudo validation, and cleans up tracked background jobs on exit.
test/unit/devtools/test_start_sh.py	Adds focused tests for sudo ordering, failure handling, tunnel lifecycle, generated shell syntax, and cleanup behavior.

_{Reviews (4): Last reviewed commit: "fix(devtools): preserve exit code in sta..." | Re-trigger Greptile}