Skip to content

Why did four simple agent replies take 13–31 seconds? #2386

Description

@BradGroux

Why this issue exists

Four simple agent turns on 2026-07-22 took 13–31 seconds from the published
mention to the published reply. Every turn completed, but the outer event
timestamps cannot tell us where the time went.

This is a small directional sample, not a performance benchmark. It is enough
to justify stage-level timing so we can separate relay delivery, queueing,
runtime startup, model work, and reply publication.

What I measured

The turns were sequential and requested only a short acknowledgement, with no
external lookup. Times are relay event created_at values with one-second
resolution.

Turn Mention time (UTC) Reply time (UTC) Elapsed
1 16:11:23 16:11:43 20s
2 16:12:24 16:12:55 31s
3 16:13:03 16:13:28 25s
4 16:14:57 16:15:10 13s

For these four turns:

  • mean: 22.25s
  • median: 22.5s
  • range: 13–31s

What this tells us

All four mentions were delivered and answered. Equivalent trivial turns still
varied by 18 seconds.

It does not identify the slow stage. The elapsed time can include:

  1. relay fan-out and harness receipt;
  2. queue or admission delay;
  3. harness or ACP runtime wake-up;
  4. provider/model time to first output and completion;
  5. reply publication from the harness;
  6. relay acceptance and delivery of the reply.

This also does not replace the reliability bugs where an agent never receives
or processes a turn.

What #2408 covers

#2408 adds process-local timing from harness relay receipt through reply
fan-out, including queue admission, ACP session resolution, prompt dispatch,
first semantic output, and turn completion.

That is a useful first slice, but it does not yet measure sender publication,
relay delivery before harness receipt, relay acceptance after fan-out, or
recipient rendering. It also does not establish hosted warm/cold baselines,
latency budgets, or a scheduled regression job, so this issue should remain
open after #2408 lands.

What the full measurement needs

Use one correlation identifier and monotonic timestamps at these boundaries:

  1. triggering event accepted by the relay;
  2. event received by buzz-acp;
  3. event admitted to the turn queue;
  4. ACP child ready, with cold and warm paths separated;
  5. first ACP/model output;
  6. turn complete;
  7. reply publication initiated;
  8. reply accepted by the relay.

Report both stage durations and total latency. Performance telemetry must not
record message content, prompts, credentials, model output, or tool arguments.

Benchmark at least:

  • warm harness with a warm ACP/model session;
  • warm harness with a cold or lazy ACP start;
  • app or harness restart recovery;
  • local runtime versus relay-mesh runtime;
  • no-tool acknowledgement versus a tool-using turn.

Done when

  • A repeatable benchmark covers the full mention-to-reply path.
  • Results record sample size, p50, p95, and max rather than only averages.
  • Warm and cold paths are reported separately.
  • The team records latency budgets for each supported path.
  • CI or a scheduled performance job detects material regressions.
  • Telemetry contains correlation and timing metadata only.

Related work

Those issues should stay focused on missed turns, dead runtimes, and delivery
feedback. This issue asks a different question: when the agent does reply, where
did the time go?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions