Skip to content

test(harness): diagnose pytest final gc.collect teardown stall #519

Description

@zhnt

Problem

The broad Harness pytest process can finish every test and print its summary, then remain CPU-bound during pytest final cleanup instead of exiting.

Observed outside the managed Codex sandbox on 2026-09-02 while validating commit 265903b:

  • make check-harness reached 100% and printed 7 failed, 3519 passed, 4 skipped in 712.09s.
  • The seven failures were caused by placing TMPDIR below /home/dev/lsspace/AGENTS.md; rerunning those exact seven tests with a system-/tmp basetemp produced 7 passed.
  • After the summary, the pytest process did not exit. Interrupting it showed this path:
    • _pytest.config._ensure_unconfigure
    • _pytest.unraisableexception.cleanup
    • gc_collect_harder
    • gc.collect
  • An earlier P1A aggregate run showed the same symptom after 376 passed in 698.53s.
  • The GitHub Actions Linux Harness job for 265903b completed successfully, so this is currently reproducible locally but is not blocking that published revision.

Investigation scope

  • Reduce the broad selection until the retained object/cycle or interacting test group is identified.
  • Confirm whether pytest's unraisable-exception cleanup is exposing leaked async/runtime objects or simply pathological collection cost.
  • Compare compiled and pure-Python mypy/pytest environment effects only if a minimal reproducer points there.
  • Keep diagnosis outside the managed sandbox per the workspace asyncio rule.
  • Do not add sleeps, arbitrary timeouts, or Product shutdown workarounds merely to hide pytest cleanup latency.

Acceptance

  • A deterministic minimal reproducer or a documented non-Product environment cause exists.
  • The relevant broad suite exits promptly after its test summary.
  • Any fix has a focused regression test and preserves lifecycle/cleanup semantics.

Related delivery: #518

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions