Skip to content

test(relay): investigate intermittent native CI crash and shutdown diagnostics #316

Closed

Description

@abruption

Summary

Track an unexplained native-process crash in the Relay CI suite. This is not a confirmed product regression, security finding, or explanation of the separate operational issue #161. Do not infer a fix from a successful retry.

Observed CI evidence

Separate local shutdown diagnostic

An updated combined local candidate passed 1,240 unittest cases (19 skipped) on macOS arm64/Python 3.13.7, but its stderr contained an asyncio _accept_connection2 / Server._attach assertion and unclosed socket/transport ResourceWarnings. A separate verbose Relay-only run with development mode, faulthandler and ResourceWarning-as-error passed 236 tests without those diagnostics.

The Python source is consistent with a connection-accept task constructing a transport after server closure; the specific originating test and root cause remain unknown. The diagnostic was handled by the event-loop exception handler, so unittest success does not mean a warning-free run. No relationship to the Linux native exit 139 is established.

Follow-up / acceptance

  • Preserve the first failed attempt and source/environment pins; do not suppress warnings, weaken assertions, increase sleeps arbitrarily, relax dependencies or repeatedly rerun until green.
  • PR fix(codex): refuse known storage relocation across native config layers #311 follow-up 09efcc2 adds diagnostic-only python -X faulthandler to the Relay CI command. This does not fix or explain the crash.
  • If exit 139 recurs, keep publication gated and collect fault/native stacks plus exact Python/OpenSSL/runner/dependency metadata. Compare the full Relay suite with the isolated active test to distinguish cumulative/order effects.
  • If the shutdown assertion recurs, record the current test and first loop exception plus allocation information (for example a temporary test-owned exception handler/tracemalloc), without suppressing normal diagnostics. Do not treat the GC finalization location as the socket allocation site.
  • Determine whether the events are a fixture lifecycle defect, product lifecycle defect, dependency/CPython issue or runner-specific failure before implementing a workaround or claiming resolution.

Monitoring/investigation only. No new native-agent/model/production test or operational change is implied. No milestone or release-completion claim is assigned by this record.

Activity

  1. added
    bugSomething isn't working
    priority:P2Planned reliability, acceptance or feature work; priority does not authorize implementation.
    status:monitoringOperational monitoring pending current evidence; not a confirmed current incident.
    on Oct 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingpriority:P2Planned reliability, acceptance or feature work; priority does not authorize implementation.status:monitoringOperational monitoring pending current evidence; not a confirmed current incident.

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions