Skip to content

notify_observers drains a process-global task set, so one loop's on_loop_end blocks on another loop's passive hooks #2

Closed

Description

@zhanghanduo

agent_core/loop_types.py keeps fire-and-forget observer tasks in a module-global set:

_background_tasks: set[asyncio.Task[None]] = set()
...
if method == "on_loop_end":
    await drain_background_observers()   # drains every loop's tasks

drain_background_observers() gathers all pending tasks, not the ones this loop created. With two agent loops in one process — the ordinary sub-agent case — loop A's on_loop_end waits on loop B's still-running passive hooks.

Observed: a passive on_turn_end sleeping 3s in one logical loop made an unrelated loop's on_loop_end return 3.0s late. A passive hook awaiting something that is never signalled turns this into an indefinite hang of an unrelated loop's shutdown.

Fix direction: a per-loop task registry, or scope the drain to the tasks this loop scheduled. Note the drain exists for a real reason (a caller reading the event sink right after run() would otherwise race pending writes), so it must stay — just be scoped.

Provenance: pre-existing in ApodexHarness/miroharness/core/loop_types.py, ported verbatim by #1 — not introduced by the refactor. Fixing it here is the source-of-truth fix; ApodexHarness carries the same code and needs the same change until it consumes AgentCore.

Activity

  1. zhanghanduo commented on Sep 3, 2026

    @zhanghanduo
    CollaboratorAuthor

    Fixed on main by 63dba6f ("fix: harden runtime observer contracts"). The commit's trailer was written with a literal \n\n before Refs #2, #3, #4, #5, #6., so GitHub never linked it — closing by hand.

    The drain is now loop-scoped: _background_tasks_by_owner: dict[asyncio.Task, set[asyncio.Task]] keys passive hooks by the owning agent-loop task (agent_core/loop_types.py:485), notify_observers registers each task under asyncio.current_task() (:574), and drain_background_observers gathers only that owner's pending tasks (:583). The drain itself is preserved, as the issue required — one loop's on_loop_end no longer waits on another loop's hooks.

    Downstream: miroharness/core/loop_types.py:653 still carries the process-global _background_tasks set. Tracking that backport separately.

  2. zhanghanduo commented on Sep 3, 2026

    @zhanghanduo
    CollaboratorAuthor

    Correction to the "Downstream" note above.

    That note is accurate for MiroHarness main, which still carries the standalone 790-line copy — but it is already stale for the integration branch. On refactor/agent-core, miroharness/core/loop_types.py is a 75-line compatibility facade that re-exports agent_core.loop_types (done by 1bedcc52 refactor: consume shared runtime contracts), and that branch pins:

    apodex-agent-core @ git+ssh://git@github.com/ApodexAI/AgentCore.git@a9b5272d
    

    git merge-base --is-ancestor 63dba6f a9b5272 confirms the pin already contains this fix.

    So no backport is needed. The downstream copy is not being patched in parallel; it is being deleted by the extraction. MiroHarness main keeps the old code only until refactor/agent-core lands, and this fix arrives there through the pin rather than through a second edit.

    The only product-specific symbol that stays behind in the facade is wall_deadline_remaining_s, which binds miroharness.core.execution_context and is registered as an allowed delta in scripts/contract/manifest.py.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions