Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 48 additions & 4 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,10 @@ jobs:
fail-fast: false
matrix:
os: [ubuntu-latest, macos-latest, windows-latest]
python-version: ["3.11", "3.12"]
# 下限(3.11)与上限都测:本仓声明 >=3.11,而 3.13/3.14 上的解释器行为真的会
# 变——v4-mapped 的 ``::ffff:127.0.0.1`` 在 3.11 上不被认作回环、3.12 上被
# 认作回环,就是一个只在某个版本上才暴露的例子。
python-version: ["3.11", "3.12", "3.13", "3.14"]

steps:
- uses: actions/checkout@v4
Expand All @@ -62,8 +65,12 @@ jobs:

# 依赖时序的断言只会**随机**报红:main 上 Python 3.11 (macos-latest) 就因为
# "睡 0.3 秒后至少 3 次检查"拿到 2 而红过一次——不是产品代码错,而是断言考的是
# "这台机器够快"。这条腿把那种机器搬进 CI:往工作线程注入延迟(见
# tests/conftest.py),于是这类测试**当场**失败,而不是某天随机变红。
# "这台机器够快"。这条腿把那种机器搬进 CI(见 tests/_injection.py),于是这类测试
# **当场**失败,而不是某天随机变红。两条轴各自独立:
# - 工作线程自己的等待被拉长;
# - 线程被 start() 了但迟迟跑不到第一行(等待被拉长管不到这种——它没有任何等待
# 可拉长)。这条轴真的抓到过东西:它让 TunnelManager.connect 里 stderr drain
# 线程的竞态(读已被清空的 self.process)从"偶尔"变成"每次"。
#
# 只跑一个 OS/版本:它的价值在确定性,不在覆盖率——版本/平台差异由上面的矩阵负责。
timing:
Expand All @@ -81,15 +88,52 @@ jobs:
- name: Install project with dev deps
run: python -m pip install --upgrade pip && pip install -e ".[dev]"

# 故意不加 `-q`:报告头里的那行横幅就是"守卫真的开着"的证据,下一行 grep
# 故意不加 `-q`:报告头里的那两行横幅就是"守卫真的开着"的证据,下面 grep 它们
# 一条静默失效的注入会把这条腿变成空跑,而输出仍然是绿的。
- name: Run tests with injected latency
env:
PONTE_TEST_THREAD_DELAY: "0.15"
PONTE_TEST_THREAD_START_DELAY: "0.15"
run: |
set -o pipefail
pytest 2>&1 | tee slow-runner.log
grep -q "thread latency: +0.15s" slow-runner.log
grep -q "thread start delay: +0.15s" slow-runner.log

# 一条腿覆盖三个"环境"维度(不是平台维度,平台由上面的矩阵负责)。合成一条是因为
# 它们都是同一件事的不同侧面:**这台机器不是开发者那台**。出错时 traceback 会指出
# 是哪个维度,所以合并的代价只是少一次归因,换来少两个 job。
#
# - C locale + 非 UTF-8:CI 默认是 UTF-8,但真实服务器上 LANG 经常根本没设。
# PYTHONCOERCECLOCALE=0 是必须的——没有它 Python 会把 C locale 自动"挽救"成
# C.UTF-8(PEP 538),这条腿就变成空跑。ponte 的输出里有中文,所以这不是假想问题。
# - 半时区偏移(UTC+5:30):按整小时做的本地时间算术在这里会错一个小时的一半。
# - 弃用告警当错误:这个小工具要长期支持 3.11+,用过就废的 API 应该当场红,
# 而不是等某个版本把它删掉才红。
env-edges:
name: Environment edges (C locale, half-hour TZ, deprecations)
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v4

- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.13"

# pip 这一步在正常 locale 下跑:安装期的警告不是这条腿要考的东西。
- name: Install project with dev deps
run: python -m pip install --upgrade pip && pip install -e ".[dev]"

- name: Run tests in a non-UTF-8 half-hour timezone
env:
LC_ALL: C
LANG: C
PYTHONCOERCECLOCALE: "0"
PYTHONUTF8: "0"
TZ: Asia/Kolkata
run: pytest -W error::DeprecationWarning

build:
name: Build & verify wheel
Expand Down
20 changes: 20 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Fixed

- **A busy machine could leave the SSH session's stderr unread.** `connect()`
hands the child's stderr to a daemon thread so the pipe can never fill up and
server-side disconnects are logged while they happen. That thread read the
process from `self.process` *after* it was scheduled, while `connect()` clears
that attribute the moment the session ends — so a thread that reached its
first line late found `None` and drained nothing at all, leaving unread the
very pipe it exists to keep empty (once the buffer fills, SSH stalls). The
process is now handed to the thread when it is created. This was found by the
new scheduling injection, not by reading the code: it is a race whose window
is tiny on an idle machine and wide open on a loaded one.
- **A failed health probe was reported as a dead port — and could kill a healthy
tunnel.** The server-side probe is an SSH connection of its own, and when that
connection failed (a reset, provider-side rate limiting, our own timeout kill)
Expand Down Expand Up @@ -99,6 +109,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Changed

- **CI exercises two further environments and one further failure class.** The
test matrix now runs Python 3.13 and 3.14 as well — the runtime is not
decoration: `IPv4Address` handling of `::ffff:127.0.0.1` changed between 3.11
and 3.12, and that difference decided whether this tool demanded an auth
token. One job runs the suite under a C locale (non-UTF-8 stdio) in a
half-hour timezone with deprecation warnings as errors. And the "slow runner"
job gained a second, independent axis: `PONTE_TEST_THREAD_START_DELAY` holds a
freshly started thread before its first line runs — a case stretching waits
cannot reach, because a thread that has not started executing performs no
waits to stretch.
- **Every SSH path now builds its connection flags in one place.** The tunnel,
the login test behind `ponte test` / the health loop / `doctor`, and the
server-side port probe each assembled their own `-o`/`-i`/`-p` list, so a
Expand Down
37 changes: 25 additions & 12 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,18 +33,26 @@ python _smoke_test.py # zero-dependency smoke check

Coverage has a `fail_under` threshold in `pyproject.toml`; don't let it drop.
CI runs lint + types on Linux, the same test suite on Windows / Linux / macOS ×
Python 3.11 / 3.12, and a `build` job that installs the wheel and runs
Python 3.11–3.14, a job that re-runs it under a C locale / half-hour timezone
with deprecations as errors, and a `build` job that installs the wheel and runs
`ponte init`. Coverage is uploaded to Codecov.

One more job re-runs the suite with `PONTE_TEST_THREAD_DELAY=0.15`. That variable
makes `tests/conftest.py` inject latency into worker-thread sleeps and waits, so a
One more job re-runs the suite with two independent injections (`tests/_injection.py`):
`PONTE_TEST_THREAD_DELAY=0.15` stretches every worker-thread sleep and wait, and
`PONTE_TEST_THREAD_START_DELAY=0.15` holds a freshly started thread before it runs
its first line. The second one matters because it is unreachable by the first: a
thread that has not started executing yet has no waits to stretch. Either way, a
test that only passes because the machine is fast fails there **every time**
instead of flaking once in a while. Run it the same way before blaming a runner:

```bash
PONTE_TEST_THREAD_DELAY=0.15 pytest # 慢机器模拟(Git Bash / POSIX 语法)
PONTE_TEST_THREAD_DELAY=0.15 PONTE_TEST_THREAD_START_DELAY=0.15 pytest
```

If you add an assertion that waits for something a background thread produces, wait
for a deadline (see `tests/_waits.py`) rather than for a fixed number of seconds;
the guard job is what turns the difference into a failure instead of a flake.

## Project layout

```
Expand Down Expand Up @@ -157,19 +165,24 @@ python _smoke_test.py # 零依赖冒烟检查
```

覆盖率在 `pyproject.toml` 里有 `fail_under` 阈值,请勿让它回落。CI 会在
Linux 上跑 lint + 类型检查,在 Windows / Linux / macOS × Python 3.11 / 3.12
上跑同一套测试,另有 `build` 任务会安装 wheel 并执行 `ponte init`。覆盖率
上报到 Codecov。
Linux 上跑 lint + 类型检查,在 Windows / Linux / macOS × Python 3.11–3.14
上跑同一套测试,另有一个任务在 C locale / 半时区偏移下、并把弃用告警当错误地跑一遍,
以及 `build` 任务会安装 wheel 并执行 `ponte init`。覆盖率上报到 Codecov。

还有一个任务会用 `PONTE_TEST_THREAD_DELAY=0.15` 再跑一遍:这个变量让
`tests/conftest.py` 往工作线程的 sleep/wait 里注入延迟,于是“只有机器够快才
通过”的测试会**每次都**在那里失败,而不是偶发地红一次。怀疑是 runner 抽风之前,
先这样在本地跑一遍:
还有一个任务会用两个互相独立的注入(见 `tests/_injection.py`)再跑一遍:
`PONTE_TEST_THREAD_DELAY=0.15` 把工作线程的每次 sleep/wait 拉长;
`PONTE_TEST_THREAD_START_DELAY=0.15` 让刚 `start()` 的线程迟迟跑不到第一行。
后者是前者够不到的:还没开始执行的线程没有任何等待可以被拉长——而它在空闲机器上
永远通过、在忙机器上随机失败。两者共同的效果是:“只有机器够快才通过”的测试会
**每次都**在那里失败,而不是偶发地红一次。怀疑是 runner 抽风之前,先这样在本地跑一遍:

```bash
PONTE_TEST_THREAD_DELAY=0.15 pytest # 模拟慢机器(Git Bash / POSIX 语法)
PONTE_TEST_THREAD_DELAY=0.15 PONTE_TEST_THREAD_START_DELAY=0.15 pytest
```

如果你要写“等后台线程产出某件东西”的断言,请等截止时间(见 `tests/_waits.py`),
不要睡一个固定秒数;守卫任务存在的意义就是把这个差别从偶发红变成必然红。

## 项目结构

```
Expand Down
32 changes: 19 additions & 13 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -324,14 +324,18 @@ python _smoke_test.py # zero-dependency quick check
```

CI runs lint + types on Linux, and the test suite across
Windows/Linux/macOS × Python 3.11/3.12, reporting coverage to
Windows/Linux/macOS × Python 3.11–3.14, reporting coverage to
[Codecov](https://codecov.io/gh/modusensus/ponte). One extra job re-runs the
suite with `PONTE_TEST_THREAD_DELAY=0.15`: that injects latency into
worker-thread sleeps and waits, so a test that only passes on a fast machine
fails there *every* time instead of flaking once in a while. Set the same
variable locally to reproduce such a machine. A `build` job also installs
the built wheel and runs `ponte init`, so a packaging regression cannot ship
again. See [CONTRIBUTING.md](CONTRIBUTING.md).
suite with `PONTE_TEST_THREAD_DELAY=0.15` **and**
`PONTE_TEST_THREAD_START_DELAY=0.15`: the first stretches every worker-thread
sleep and wait, the second holds a freshly started thread before its first line
runs (a case no amount of wait-stretching can reach). A test that only passes on
a fast machine fails there *every* time instead of flaking once in a while. Set
the same variables locally to reproduce such a machine. Another job runs the
suite under a C locale (non-UTF-8 stdio) in a half-hour timezone with
deprecation warnings as errors, and a `build` job installs the built wheel and
runs `ponte init`, so a packaging regression cannot ship again. See
[CONTRIBUTING.md](CONTRIBUTING.md).

## 📝 Notes

Expand Down Expand Up @@ -629,12 +633,14 @@ python _smoke_test.py # 零依赖快速自检
```

CI 在 Linux 上跑 lint + 类型检查,在 Windows/Linux/macOS × Python
3.11/3.12 上跑测试,覆盖率上报到
[Codecov](https://codecov.io/gh/modusensus/ponte)。另有一个任务会用
`PONTE_TEST_THREAD_DELAY=0.15` 再跑一遍:它往工作线程的 sleep/wait 里注入
延迟,于是"只有机器够快才通过"的测试会**每次都**在那里失败,而不是偶发地
红一次。本地设同一个变量即可复现这种机器。`build` 任务会安装打好的 wheel 并
执行 `ponte init`,避免打包问题再次溜进发布。详见
3.11–3.14 上跑测试,覆盖率上报到
[Codecov](https://codecov.io/gh/modusensus/ponte)。另有一个任务会同时用
`PONTE_TEST_THREAD_DELAY=0.15` 与 `PONTE_TEST_THREAD_START_DELAY=0.15` 再跑
一遍:前者把工作线程的每次 sleep/wait 拉长,后者让刚 start() 的线程迟迟跑不到
第一行(后半种情况没有任何等待可以被拉长)。于是"只有机器够快才通过"的测试会
**每次都**在那里失败,而不是偶发地红一次。本地设同样两个变量即可复现这种机器。
还有一个任务在 C locale(非 UTF-8 的 stdio)、半时区偏移下跑,并把弃用告警当错误;
`build` 任务会安装打好的 wheel 并执行 `ponte init`,避免打包问题再次溜进发布。详见
[CONTRIBUTING.md](CONTRIBUTING.md)。

## 📝 注意事项
Expand Down
21 changes: 17 additions & 4 deletions ponte/core.py
Original file line number Diff line number Diff line change
Expand Up @@ -212,7 +212,7 @@ def connect(self) -> int:
"""
args = self.build_args()
logger.info("Launching: %s", " ".join(args))
self.process = subprocess.Popen(
proc = subprocess.Popen(
args,
stdout=subprocess.DEVNULL,
stderr=subprocess.PIPE,
Expand All @@ -228,12 +228,22 @@ def connect(self) -> int:
close_fds=True,
creationflags=creation_flags(),
)
self.process = proc
self._connected_at = time.monotonic()
# Drain stderr on a daemon thread: the pipe can never fill up (which
# would stall SSH), and disconnect reasons are logged in real time
# instead of only after the session ends.
#
# The process is handed to the thread rather than read from
# ``self.process`` inside it. That attribute is cleared as soon as the
# session ends (``finally`` below), and no amount of being fast prevents
# the *other* direction: a thread that is scheduled late starts after the
# clear, finds ``None``, and silently drains nothing — a busy machine
# turns this into "the pipe nobody reads", which is the failure this
# thread exists to prevent.
threading.Thread(
target=self._drain_stderr,
args=(proc,),
name="ponte-ssh-stderr",
daemon=True,
).start()
Expand All @@ -247,15 +257,18 @@ def connect(self) -> int:
finally:
self.process = None

def _drain_stderr(self) -> None:
"""Read the SSH child's stderr line by line until EOF.
def _drain_stderr(self, proc: subprocess.Popen | None) -> None:
"""Read the stderr of *proc* line by line until EOF.

Runs on a daemon thread for the lifetime of the session. Prevents the
``stderr=PIPE`` buffer from filling up and logs server-side disconnect
reasons (e.g. ``Connection to host closed by remote host``) as they
happen, so a dropped tunnel is diagnosable even after the fact.

``proc`` is a parameter, not ``self.process``: see ``connect()`` — the
session that owns this pipe must keep being drained even if this thread
only reaches its first line after ``connect()`` returned.
"""
proc = self.process
if proc is None or proc.stderr is None:
return
try:
Expand Down
5 changes: 5 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,11 @@ ignore = [
# typer 要求 Option/Argument 作为参数默认值出现。
"ponte/main.py" = ["B008"]

[tool.ruff.lint.isort]
# tests/ 下以单下划线开头的内部助手("等待"与"负载注入")是本仓库的代码,不是第三方
# 包。不声明的话 ruff 会把它们归到第三方段,于是 import 分组传达的信息是错的。
known-first-party = ["ponte", "_injection", "_waits"]

[tool.mypy]
python_version = "3.11"
files = ["ponte"]
Expand Down
Loading
Loading