[RefImpl] Recover a failed or stalled trace output and count the spans it drops #282

Target runtime or language

AGT reference Guardian: reference-implementations/agt/packages/guardian/src/otel-trace-exporter.ts

What this proves

This is part of issue #274, the first of three slices.

An output that fails or stalls must not lose spans silently, and it must not use unbounded memory. Today, the first failed write disables an output for the rest of the process. A hung file write is never marked as failed, and its pending writes grow without a limit. Spans dropped when the 2048-span queue is full produce no message that an operator sees.

Scope:

1. Replace the permanent dead flag (isolated()) with a breaker. When the breaker is open, export drops the batch, counts it, and returns at once. After a backoff, the next real batch tests the output again. Do not test the output during shutdown or forceFlush. Call unref() on every timer.
2. A file output has at most one write in flight. Batches that arrive while a write is stalled are dropped and counted.
3. Optional: a small replay buffer, for a collector that restarts. If you add one, it must be capped by count and by bytes, drop the oldest first, count what it drops, and never be drained during shutdown.
4. Count dropped spans per output: spans ended minus spans delivered. Report the count on stderr:
   • at the first drop;
   • at most once per minute while drops continue;
   • when the output recovers;
   • in the shutdown line.
5. Limit the bytes one span can hold. Today, 64 provenance objects with two 4096-character fields each make a span of about 0.5 MB, and 1500 such requests used more than 3 GB of memory.
6. Update the two READMEs that say "disabled after failure, restart the Guardian", and their tests.

Acceptance tests:

• Hung output. An output that never calls back gets at most one write in flight. Memory growth stays under a stated limit (for example 256 MB above the baseline) across 1500 maximum-size requests. The dropped count equals spans posted minus spans written.
• Recovery. A refused output recovers after the backoff, and spans after recovery arrive.
• No duplicates. A batch that timed out is not sent again in a way that duplicates span ids at the receiver.
• Bun versions. Run the timer tests on Bun 1.3.11 (CI) and 1.3.14.

Not in this slice: reading names from the mapping (the third slice), file permissions (the second slice), and the host adapter (the fourth slice).

Do you intend to implement this yourself

Yes. @sabasiddique1 is implementing it. The reviewer is @artmaro, the Reference Implementation lead.

Current Priority Scope

Feeds the runnable Guardian reference implementation