Skip to content

fix(manager): account for unfinished run token usage - #148

Open
Asaf-prog wants to merge 20 commits into
mainfrom
codex/fix-140-cancelled-token-budget
Open

Asaf-prog wants to merge 20 commits into
mainfrom
codex/fix-140-cancelled-token-budget

Conversation

@Asaf-prog

Copy link
Copy Markdown
Collaborator

Retain cumulative per-run charges independently of assistant messages, including failed and cancelled executions, pending approvals, and resumed runs. Preserve charges across branch edits and run-record expiry with a durable usage ledger and migration backfill for existing runs.

Reconcile missed retention writes from authoritative run records, prevent cancellation and post-commit readback failures from charging a leg twice, and include usage in pending-approval stream events. Add regression coverage for memory and SQL repositories and document the accounting behavior.

Validation: make check passed with 1126 tests.

Fixes #140

Retain cumulative per-run charges independently of assistant messages, including
failed and cancelled executions, pending approvals, and resumed runs. Preserve
charges across branch edits and run-record expiry with a durable usage ledger
and migration backfill for existing runs.

Reconcile missed retention writes from authoritative run records, prevent
cancellation and post-commit readback failures from charging a leg twice, and
include usage in pending-approval stream events. Add regression coverage for
memory and SQL repositories and document the accounting behavior.

Validation: make check passed with 1126 tests.

Fixes #140
@Asaf-prog
Asaf-prog requested a review from AmitAvital1 October 14, 2026 21:06
@Asaf-prog Asaf-prog self-assigned this Oct 14, 2026
Asaf-prog and others added 5 commits October 16, 2026 14:57
Repository-contract fixtures and SQL conversation-creation helpers created aiosqlite engines without disposing them, causing intermittent worker-thread exceptions. Each test now owns and disposes its engines.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An opt-in idempotency key claims a durable receipt together with the user message, so a retried preparation recovers the same turn instead of appending another. Adds saved-response recovery for channel workers and migration 0007.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Provider-independent inbox, sender bindings, execution claims, webhook receipts, ordered outbox and delivery events, worker ownership, and the channel identity port, with migrations 0008-0010.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Opt-in authenticated Cloud API intake, a supervised channel turn worker with conservative recovery, provider delivery with text controls, and health reporting for a stopped worker.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Integration plan, ADRs 0007-0010, implementation tasks 0014-0017 and 0019-0020, review records, and index entries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread src/agent_engine/engine/langgraph/engine.py
Comment thread src/agent_engine/engine/langgraph/engine.py

@AmitAvital1 AmitAvital1 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found two blocking correctness issues in the token accounting path. Details and reproductions are attached to the exact lines.

Asaf-prog and others added 2 commits October 16, 2026 15:47
The worker logs one `channel status:` line per configured interval with queue
counts, the oldest unfinished and unsent ages, and unknown, failed, and blocked
reply counts. The read is bounded to five seconds so monitoring cannot stall
scheduling or a graceful drain. Every rejected webhook request logs a fixed
reason. No line contains message text, sender IDs, tokens, or provider IDs.

Adds EXTRA_WHATSAPP_STATUS_LOG_SECONDS (default 60).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The runbook covers preflight, deployment, staged enablement and pausing,
health and status logs, restart recovery, rollback, retention, and the
real-number smoke-test checklist. The example runs agent-manager on durable
storage with a tool-free assistant and an allowlist identity resolver; its port
is bound to loopback because the web conversation API is anonymous by default.

A fake-provider rehearsal proves that paused intake survives restarts, is
answered and sent exactly once, projects a delivered callback, and is not
resent after a further restart.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…sage

A leg's usage write retried after a commit whose acknowledgement was lost
added the same tokens twice (5 became 10). Each leg now records the run totals
seen before its first attempt and writes with compare-and-add, so a retry after
an ambiguous commit adds nothing. RunRepository gains
add_token_usage_if_unchanged with a default for existing adapters; the SQL
adapter performs the compare and add in one UPDATE ... RETURNING.

The manager's SSE schema dropped input_tokens and output_tokens, so
/messages/stream and /decision/stream omitted the documented pending usage.
Both fields are now mapped for every stream event, with an SSE-level
regression test covering both endpoints.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Asaf-prog
Asaf-prog requested a review from AmitAvital1 October 16, 2026 13:05
Comment thread src/agent_engine/engine/langgraph/engine.py Outdated

@AmitAvital1 AmitAvital1 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The two original findings are fixed and their threads are resolved. The second full review found one new blocking accounting race; details and a reproduction are attached inline.

Compare-and-add could not tell a retried leg from a different leg of the same
run. RunLifecycle.begin() accepts an existing run_id, so two overlapping runs
read the same totals and one leg was dropped (10 tokens consumed, 5 stored).

Each leg write now carries a random leg key. The SQL adapter claims the key in
agent_run_token_legs and adds the usage in one transaction, and the in-memory
adapter records the key with the totals, so a retry after an unacknowledged
commit adds nothing while overlapping legs are each charged. The table is
part of migration 0006, which this pull request introduces.
RunRepository.add_leg_token_usage replaces add_token_usage_if_unchanged; its
default still serves existing adapters.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Asaf-prog
Asaf-prog requested a review from AmitAvital1 October 17, 2026 11:09
Comment thread src/agent_engine/runs/repository.py

@AmitAvital1 AmitAvital1 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The overlapping-leg race is fixed and I resolved that thread. One blocking contract issue remains: RunRepository.add_leg_token_usage documents at-most-once behavior but its default implementation is non-idempotent, so existing external adapters can silently double-charge after an ambiguous commit. Please require adapters to implement the atomic keyed operation instead of falling back to add_token_usage.

Make add_leg_token_usage abstract and remove its unsafe additive fallback.
Update the registration spy, reject legacy adapters without the keyed
operation, and verify that the base implementation cannot charge usage.

Document the external-adapter upgrade requirement in ADR 0002.
Validation: 101 focused tests and make check (1135 tests) passed using
an isolated temporary DATABASE_URL for the local CLI checks.
…dget' into codex/review-whatsapp-149

# Conflicts:
#	tasks/README.md
feat(manager): WhatsApp pilot packaging and operations (WA-06)
feat(manager): WhatsApp Cloud API channel
try:
delivery = None
controls = None
if settings.extra_whatsapp_sender_enabled:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Controls are only wired when outbound sending is enabled. In the supported worker-on/sender-off state, /stop fails and later messages still reach the engine, so opt-out is bypassed. Please keep controls active whenever the worker runs; only delivery should depend on the sender.

@AmitAvital1 AmitAvital1 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The atomic run-leg contract is fixed, and I resolved that thread. The PR now also contains the full #149/#150 WhatsApp stack: 118 files and 13,456 added lines, although this PR and its follow-up task are scoped to token accounting. That imported code still has a confirmed opt-out bypass when the worker runs with sending disabled; I left the exact inline finding. Please remove the unrelated stack from this PR, or fix and explicitly review it here before merging.

Bind controls whenever the worker runs, independently of outbound transport.
Keep worker-only startup compatible with absent sender credentials and support
contact, and reject missing durable control storage. Cover signed stop/start,
reset, help and live restart behavior through the real HTTP/SQL composition.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug][P2] Cancelled runs consume tokens without counting toward the conversation budget

2 participants