Repository navigation
test(relay): investigate intermittent native CI crash and shutdown diagnostics #316
Description
Summary
Track an unexplained native-process crash in the Relay CI suite. This is not a confirmed product regression, security finding, or explanation of the separate operational issue #161. Do not infer a fix from a successful retry.
Observed CI evidence
- PR fix(codex): refuse known storage relocation across native config layers #311, exact source f812a0f.
- Run 38690059740, attempt 1, Ubuntu/Python 3.13 job 116273649374.
- Python 3.13.16 terminated with segmentation fault / exit 139 while
test_lost_pair_commit_response_reconciles_without_reusing_secretwas running. No Python fault traceback or crash artifact was captured. Three fail-fast Relay matrix cells were cancelled. - The preceding green source 7f109fb had a passing Relay suite. Installed dependency versions matched (including cryptography 50.0.2 and websockets 17.2), while the hosted runner image differed: failed 20261004.327.1 versus preceding green 20261012.334.1. The image difference is an observation, not a cause.
- One bounded same-source rerun, attempt 2 passed all four Relay matrix cells and the aggregate gate. This establishes non-reproduction only.
Separate local shutdown diagnostic
An updated combined local candidate passed 1,240 unittest cases (19 skipped) on macOS arm64/Python 3.13.7, but its stderr contained an asyncio _accept_connection2 / Server._attach assertion and unclosed socket/transport ResourceWarnings. A separate verbose Relay-only run with development mode, faulthandler and ResourceWarning-as-error passed 236 tests without those diagnostics.
The Python source is consistent with a connection-accept task constructing a transport after server closure; the specific originating test and root cause remain unknown. The diagnostic was handled by the event-loop exception handler, so unittest success does not mean a warning-free run. No relationship to the Linux native exit 139 is established.
Follow-up / acceptance
- Preserve the first failed attempt and source/environment pins; do not suppress warnings, weaken assertions, increase sleeps arbitrarily, relax dependencies or repeatedly rerun until green.
- PR fix(codex): refuse known storage relocation across native config layers #311 follow-up 09efcc2 adds diagnostic-only
python -X faulthandlerto the Relay CI command. This does not fix or explain the crash. - If exit 139 recurs, keep publication gated and collect fault/native stacks plus exact Python/OpenSSL/runner/dependency metadata. Compare the full Relay suite with the isolated active test to distinguish cumulative/order effects.
- If the shutdown assertion recurs, record the current test and first loop exception plus allocation information (for example a temporary test-owned exception handler/tracemalloc), without suppressing normal diagnostics. Do not treat the GC finalization location as the socket allocation site.
- Determine whether the events are a fixture lifecycle defect, product lifecycle defect, dependency/CPython issue or runner-specific failure before implementing a workaround or claiming resolution.
Monitoring/investigation only. No new native-agent/model/production test or operational change is implied. No milestone or release-completion claim is assigned by this record.
Activity