Skip to content

[entrant] Land the Iggy arm once its connector runtime can carry the pipeline #65

Description

@MarcusKainth

Follow-up to #62, which is settled: the contract question landed in #63 and the
arm's outcome in #64, where entrants/iggy/ is carried as status = "planned"
with its blockers named.

This issue exists because the reasons the arm cannot be built are not ours to
fix
, so they need somewhere to be watched that is not a closed issue.

Why the arm is not built

Three findings against apache/iggy at 412014a, verified in source and
reproduced against a live server. The authoritative statement is
[planned].blockers in entrants/iggy/entrant.toml, which names files and
lines; in short:

  1. Avro cannot reach any sink. The connector runtime labels a batch with the
    decoder's schema rather than the payload's type, so a sink is handed a
    variant it cannot read and drops every message. Measured: 100 messages
    produced, 0 rows landed, offsets committed at poll time, and a restart
    redelivers nothing — silent at the batch level. Proto and flatbuffer break
    identically.
  2. At-most-once, hard-coded. Offsets commit with the poll request, before
    the sink sees the batch. No config exposes it. This suite compares
    guarantee-for-guarantee.
  3. One lockstep connection per process. One poll in flight however many
    streams are configured, so a 32-CPU envelope cannot be spent.

All three concern the connector runtime (v0.5.0-edge.6). None of them says
anything about the Iggy broker, which this suite has never measured, and
nothing published here should imply otherwise.

Revisit trigger

Reopen the arm when both hold:

Upstream

A bug report for finding 1 is drafted and not yet filed — it needs sign-off,
and an eventual fix PR needs the Apache ICLA. No existing upstream report covers
it; apache/iggy#3430 and its PR #3734 are a different defect on the encoder
side. A local branch carries a working fix (propagate the payload's schema
rather than the decoder's) with the SDK's 150 and the runtime's 177 tests green.

Link the upstream issue here when it is filed.

Prerequisites in this tree

Work that must land before any non-Kafka arm, not only this one:

  • Bind entrants to a broker family. Nothing does today. The moment a
    second broker exists, bench run '*' on either environment cross-wires
    every arm — bench validate passes and it fails only at runtime. Noted in
    feat(methodology): the broker family is a comparability axis #63 as deliberately out of scope there.
  • Decide how a weaker-than-at-least-once system is published, if at all.
    entrant.rs refuses any delivery other than at-least-once, and
    "drops a guarantee" is stripped, which can never be the default while the
    default must be realistic — so such a system has no valid realistic
    variant. The contract anticipated a stronger guarantee, never a weaker
    one. Left unwritten in feat(methodology): the broker family is a comparability axis #63 rather than designed for a hypothetical arm.
  • Broker pluggability in the harness. infra.rs hardcodes the Redpanda
    argv and endpoints; corpus.rs (prefill, topic_message_count,
    ensure_topic, and easy to miss, register_schema and verify_corpus)
    is rdkafka; ceiling.rs's consume pass speaks raw Kafka Fetch from a
    container; driver.rs refuses nothing for sustained mode on a non-Kafka
    broker.
  • A measured Iggy consume ceiling, on its own environment profile. The
    70% headroom rule needs one per broker and Ceilings::gate refuses to
    extrapolate. Its provenance must record the connection count, or it will
    assert a topology the arm cannot reach (finding 3).

Incidental, for whoever picks this up

The apache/iggy image does not boot under Docker Desktop with defaults:
[system.sharding] cpu_allocation = "numa:auto" fails to bind memory to a NUMA
node and the server exits. IGGY_SYSTEM_SHARDING_CPU_ALLOCATION=<n> and
IGGY_SYSTEM_SHARDING_PIN_CORES=false get it up. The same knobs will matter
under a cgroup CPU cap on the reference host.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions