rescue phase 2: Commission.resume + avp-ollama warm-rescue support - #12
Draft
Patrick Carney (pcarney8) wants to merge 5 commits into
Draft
rescue phase 2: Commission.resume + avp-ollama warm-rescue support#12Patrick Carney (pcarney8) wants to merge 5 commits into
Patrick Carney (pcarney8) wants to merge 5 commits into
Conversation
Phase 2 (warm rescue) AVP extensions, additive:
* commission.md §2.2 — new optional Commission field `resume:
ResumeBlock | null` set by the supervisor when re-dispatching a
rescued run. Absent on fresh dispatches; ignored agents degrade
gracefully to cold rescue.
* commission.md §2.3 — `ResumeBlock` shape:
from_seq, replay_policy ∈ {context_only, skip_completed},
context.messages[], tool_cache[], in_flight_tool_call?
Plus nested ToolCacheEntry (result | failure, tier:
idempotent/replay_only/needs_approval) and InFlightToolCall.
* commission.md §2.4 — determinism contract for rescued runners:
MUST NOT emit seq <= from_seq, MUST skip the prelude, MUST seed
its model call with context.messages, MUST honor tool_cache on
match, SHOULD respect tier on miss.
* types.py — pydantic models for everything above:
ResumeBlock, ResumeContext, ResumeMessage, ToolCacheEntry,
ToolCacheFailure, InFlightToolCall. Plus RunResumedEvent +
RunResumedData (the second supervisor-sourced bracket event,
paired with the phase-1 RunRescuedEvent).
* T_RUN_RESUMED constant + Event union extension.
Existing 116 avp tests still pass. New types verified via standalone
round-trip serialization.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ooks
Phase 2 §5.1 — avp-ollama honors the warm-rescue contract from
commission.md §2.4. Plus test instrumentation to make warm-rescue
behaviors mechanically observable in the smoke.
Translator changes:
* `_extract_resume_block(config)` — pull `Commission.resume` from
the Commission. None on fresh dispatches; dict otherwise.
* `_seed_messages()` — prefers `resume.context.messages` over
`_initial_messages()` when present. The rescued runner's model
call sees the reconstructed conversation, not just the original
prompt.
* `_resume_messages()` / `_resume_tool_cache()` — defensive
extractors that filter malformed entries.
* `run()` — when `resume_block` is present, use `from_seq` as the
starting seq (skip the next_seq HTTP round-trip), still inherit
the agent span, still skip prelude. Falls back to next_seq query
for legacy cold rescues that pre-date phase-2 supervisor work.
* `_lookup_tool_cache(name, args)` — finds matching cache entries
by `tool_name + canonical-equal args` (Python ==; the supervisor
pre-canonicalizes both halves, so equality holds for normal
payloads).
* `_emit_tool_outcome_from_cache(parent_span, entry)` — emit
`tool_returned` (or `tool_failed`) using the cached payload.
Runner does NOT call the tool.
* Stub tool registry: `_execute_stub_tool("echo", args)` returns
args verbatim; anything else returns a synthetic payload describing
what would have happened. Real tool dispatch comes when avp-ollama
grows real tool support.
Test instrumentation (demo only, same shape as RESCUE_FAIL_AT):
* `OLLAMA_INJECT_USER_MESSAGE_AT=turn:N:content` — after turn N
completes, inject `{role: user, content}` AND emit a
`text_emitted{role: user}` event so the supervisor's resume
reconstruction picks it up. T2 smoke uses this to introduce a
recall token mid-conversation.
* `OLLAMA_INJECT_TOOL_CALL_AT=turn:N:tool_name:args_json` — before
turn N, emit `tool_invoked` for the configured tool + args. Hit:
cached `tool_returned` from `resume.tool_cache`. Miss: stub
`echo` registry. T3 smoke uses this to prove cache replay.
* Both parsed at construction; both no-op when unset; both lenient
on bad input (warn + disable rather than crash).
Tests: 22 new (12 inject-directive parsing, 10 resume-block reading).
Lib total 33 (was 11).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The supervisor inserts events into the trajectory concurrently with the runner — specifically `avp.run_resumed` (after the rescued runner's first agent event lands) and `avp.run_rescued` (during the rescue path itself). When the supervisor takes the seq the runner was about to write, the runner's POST returns 409 Conflict and the event is lost. Observed in `make demo-rescue-local`: secondary runner's text_emitted at seq=10 got 409'd because the supervisor's avp.run_resumed landed at seq=10 in the same instant. The model's actual post-rescue answer never made it into the trajectory. Fix: _post now handles 409 specifically. On collision it re-queries the supervisor for next_seq, bumps the local counter past it, and retries. Bounded to 3 attempts. Non-409 HTTP errors keep the original log-and-continue behavior (best-effort). Other HTTP-error types unchanged — only 409 triggers retry. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CPU-only Ollama with a long warm-rescue prompt can spend >2 minutes
generating a single turn, which trips the old 120s httpx timeout —
the runner then emits execution_backend_failure that looks like a
real rescue trigger but is actually "model is too slow." Both runners
hit this in the warm-rescue smoke (primary at 11:27 → 11:29:46, then
the rescue target inherits the prompt and hits the same wall).
Two knobs:
OLLAMA_NUM_PREDICT cap on output tokens (passed as options.num_predict)
to /api/chat. 0/unset preserves prior model-decides
behavior for GPU users.
OLLAMA_HTTP_TIMEOUT httpx timeout in seconds. Default bumped 120 → 300.
README documents both alongside the existing env-var table.
5 minutes wasn't enough for the warm-rescue secondary on CPU Ollama: each post-rescue turn carries 5+ messages of context, and per-token latency grows with prompt length. Observed: a single chat completion on the rescue-target hit exactly 5m0s and was killed by httpx, which the supervisor saw as `execution_backend_failure: ollama transport: timed out`. 600s gives headroom; pair with OLLAMA_NUM_PREDICT to keep real turns well under this ceiling.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked on #10 (
rescue/spec-and-ollama). AVP-side phase 2 (warm rescue) — extends the spec + types +avp-ollamarunner to supportCommission.resume. Paired with portofcontext#3 which is the supervisor side.commission.md§2.2/§2.3/§2.4): new optionalCommission.resumefield,ResumeBlockshape (context + tool_cache + in_flight_tool_call), determinism contract for rescued runners.avp.types):ResumeBlock,ResumeContext,ResumeMessage,ToolCacheEntry,ToolCacheFailure,InFlightToolCall, plusRunResumedEvent+RunResumedData(the second supervisor-sourced bracket event, companion toRunRescuedEventfrom rescue: spec extensions + avp-ollama runner package #10).translator.py): readsCommission.resume, skips prelude on resume, seeds messages fromresume.context.messagesinstead ofprompt, short-circuits tool invocations that matchresume.tool_cache.OLLAMA_INJECT_USER_MESSAGE_AT=turn:N:content(introduces a recall token mid-conversation; T2 smoke uses this) andOLLAMA_INJECT_TOOL_CALL_AT=turn:N:tool:args_json(forces a tool invocation; T3 smoke uses this). Both are demo-only — same shape asRESCUE_FAIL_AT.echo(args) → args— exists so the T3 smoke can prove the cache-replay path end-to-end without real model tool-calling. Real tool dispatch comes whenavp-ollamagrows real tool support.Test plan
cd python && uv run pytest avp/tests/ -q— 116 passing, no regressions.cd python && uv run pytest agents/avp-ollama/tests/ -q— 33 passing (+22: 12 inject-directive parsing, 10 resume-block reading).ResumeBlock(...).model_dump_json(by_alias=True)andRunResumedEvent(...).model_dump_json(by_alias=True)both produce valid envelopes.make demo-warm-rescue-local(depends on supervisor side merging).Out of scope (deferred)
text_emittedcarries assistant text but not extended-thinking content; the secondary runner loses chain-of-thought continuity. Acceptable for v0.1; documented inRESCUE_PLAN_PHASE_2.md§2.in_flight_tool_callresumption. v0.1 surfaces it as informational; phase 2.x lands the actual resumption protocol.avp-claude-agentwarm-rescue support. Bigger lift (CASDK-driven); ships as a separate PR.🤖 Generated with Claude Code