FEAT: Add reset_conversation_async hook to PromptTarget

[Mahdi Alhakim (mahdi-al-hakim)]

Description

Adds a standard hook for prompt targets to release external state that belongs to one conversation. This prevents shared stateful target instances, such as realtime and WebSocket targets used by scenarios, from retaining one connection for every completed atomic attack.

Design

• Adds the idempotent, keyword-only PromptTarget.reset_conversation_async(*, conversation_id) hook. The base implementation is a no-op, so stateless targets require no changes.
• Adds an execution-scoped lifecycle in AttackStrategy. Attacks record objective-target conversation IDs directly before objective sends. The lifecycle deduplicates the IDs and resets them when the atomic attack ends, including failure and cancellation paths.
• Keeps this lifecycle limited to the objective target. It does not reset adversarial-chat targets, scorer targets, converters, or other components whose lifetimes can differ from the attack.
• Does not use AttackResult.related_conversations as resource state. A failed or cancelled attack might not produce a result, and TAP can create several objective conversations before it can assemble one.
• Updates TAP to record each objective conversation when it is used, including conversations created by branches and single-turn rotation. Concurrent node execution cancels and awaits sibling tasks before the attack scope exits, so no target send outlives cleanup.
• Cleanup attempts every recorded reset. Ordinary reset errors are logged and do not replace the attack outcome. If reset is cancelled, remaining resets are still attempted before cancellation is propagated.

Target implementations

OpenAIRealtimeTarget and WebsocketTarget override the hook. Each removes and closes only the connection for the specified conversation. Unknown IDs are a no-op, and close errors are logged. Their existing cleanup_conversation_async methods remain as deprecated forwarding aliases. Whole-target cleanup remains separate.

The target authoring instructions now briefly state when a target should override the hook and its idempotency requirement.

Validation

Scoped target, normalizer, lifecycle, and attack tests pass. Ruff formatting and lint checks pass, and the changed production modules pass ty. The full pre-commit run passed every available hook except the repository-wide ty hook because the configured package feed did not provide the locked optional torch and litellm packages.

[Mahdi Alhakim (mahdi-al-hakim)]

Hi Richard Lundeen (@rlundeen2) Roman Lutz (@romanlutz), this has been open for a few weeks now. It is a self-contained change (a reset_conversation_async hook on PromptTarget). I recently landed microsoft/RAMPART#141, so I am glad to match whatever conventions you prefer here. Is there anything I can do to help move it along, or someone I should tag for review?

[Richard Lundeen (rlundeen2)]

(GHCP Generated): I think the underlying problem is real, especially for scenarios. A scenario commonly shares one RealtimeTarget across many concurrent attacks. Conversation IDs prevent state from mixing, but RealtimeTarget caches one websocket per ID. Without conversation-level cleanup, a large or long-running scenario can accumulate open connections until cleanup_target_async() runs.

I do not think AttackStrategy._teardown_async should discover conversation IDs from attack contexts, though. That introduces several concerns:

• AttackResult already defines the canonical relationship through conversation_id, related_conversations, and get_active_conversation_ids(). _get_objective_conversation_ids() duplicates this contract with getattr checks and attack-specific overrides.
• TAP demonstrates the fragility: it needs a custom lookup, and its single-turn path replaces a node's conversation ID each turn without recording the old ID as PRUNED, so those earlier resources still leak.
• The current implementation handles only the objective target. Attacks can also use adversarial targets, scorer targets, and converter targets. Some of their IDs are already stored in related_conversations, but the teardown filters them out. This makes a generic attack-level cleanup guarantee incomplete.
• Custom attacks can return a valid AttackResult while storing transient IDs in a different context shape. They will silently skip cleanup unless they know to override the new protected method.

I suggest this design:

1. Keep a no-op conversation-level cleanup hook on PromptTarget, with RealtimeTarget implementing it. This allows a shared target to release one conversation without destroying the target or its other active conversations.
2. Invoke objective-target cleanup at the AttackExecutor boundary after each run. The executor owns per-run execution, has the objective target, and receives the completed AttackResult.
3. Use result.get_active_conversation_ids() instead of reconstructing IDs from context shape. Ensure error results also contain the real multi-turn session.conversation_id and all related objective conversations before cleanup runs.
4. Make the scope explicit: this pass cleans objective-target conversations only. Define adversarial, scorer, and converter target lifecycles separately rather than implying that attack teardown handles every target.
5. Fix TAP and other rotation paths so every replaced objective conversation is added to related_conversations; otherwise neither result-based nor context-based cleanup can release it.

This keeps the useful part of the proposal for shared realtime targets, while preserving AttackResult as the source of truth and avoiding target-lifecycle policy in each attack implementation.

[Mahdi Alhakim (mahdi-al-hakim)]

Richard Lundeen (@rlundeen2) The TAP half is done. A node now reports each objective-target conversation as its send returns and the attack records it there, so the ones a single-turn rotation replaces and the branches a run walks away from stay nameable. Recording at that point rather than at result time is what makes it survive a run that raises, which otherwise loses 2 of 3 conversations on a depth 3 width 3 tree. That made the TAP override unnecessary, and no attack overrides the lookup now. WebsocketTarget has the same per-conversation cache, so it implements the hook too.

I kept the call in _teardown_async rather than the executor. execute_with_context_async re-raises, so a failed run reaches the executor as an exception rather than a result and completed_results is empty. And attack.execute_async never goes through the executor at all: doc/code/targets/realtime_target.py runs attacks three ways, two of which bypass it.

One resolver now, shared with the error-result builder from #2322. It still reads the context rather than the result, with a test pinning the two equal. Happy to move it if you would rather have it at the executor.

[Richard Lundeen (rlundeen2)]

I made some changes, but I like your general direction. Approved and merging

[Mahdi Alhakim (mahdi-al-hakim)]

Thanks Richard Lundeen (@rlundeen2) . Recording at the send site instead of reading related_conversations means the cleanup no longer depends on the reporting data being complete, which is what most of the TAP bookkeeping existed to guarantee.