Skip to content

s3-dump: expose per-target flush results so consumers can label metrics by target #80

Description

@jaronoff97

Problem

S3Dump.flush() returns a single aggregate FlushResult — it sums every sealed batch across all configured targets into one struct. Consumers that record flush metrics (uploaded/failed objects, records, bytes, backlog) therefore cannot break them down by target.

We want a target label on the s3-dump flush metrics in edge, but there's no truthful way to produce it from the current API:

  • flush() aggregates across targets; there's no per-target flush call.
  • The per-target upload happens inside uploadOne(... snapshots[sealed_batch.target] ...), so the consumer can't intercept it to attribute a target name.

Suggested change

flush() already iterates sealed batches that each carry a .target index (src/extensions/s3_dump.zig, the upload loop) — it just sums them. Bucket the counters by target instead and return per-target results, e.g.:

pub const TargetFlushResult = struct {
    name: []const u8,
    objects_uploaded: u32 = 0,
    objects_failed: u32 = 0,
    objects_requeued: u32 = 0,
    records_uploaded: u64 = 0,
    records_failed: u64 = 0,
    records_dropped: u64 = 0,
    bytes_uploaded: u64 = 0,
    bytes_failed: u64 = 0,
    backlog_bytes: usize = 0,
};

Add targets: []const TargetFlushResult to FlushResult (keep the existing aggregate fields for backward compatibility). The slice can be allocated in the flush arena; consumers copy what they need before it's freed, or it's owned like the rest of the result.

records_dropped is currently a single S3Dump-wide atomic drained per flush — deciding whether/how to attribute drops per target (delivery-time drops know their slot; backlog-cap drops may not) is part of this issue.

Consumer context

edge wires s3-dump for Datadog logs and records these aggregate metrics today:
edge_s3_dump_{flushes,objects_uploaded,objects_failed,records_uploaded,records_dropped,bytes_uploaded}_total + edge_s3_dump_backlog_bytes. Once per-target results land we'll add a target label sourced from TargetFlushResult.name.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions