You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[entrant] Land the Iggy arm once its connector runtime can carry the pipeline #65
Follow-up to #62, which is settled: the contract question landed in #63 and the
arm's outcome in #64, where entrants/iggy/ is carried as status = "planned"
with its blockers named.
This issue exists because the reasons the arm cannot be built are not ours to
fix, so they need somewhere to be watched that is not a closed issue.
Why the arm is not built
Three findings against apache/iggy at 412014a, verified in source and
reproduced against a live server. The authoritative statement is [planned].blockers in entrants/iggy/entrant.toml, which names files and
lines; in short:
Avro cannot reach any sink. The connector runtime labels a batch with the
decoder's schema rather than the payload's type, so a sink is handed a
variant it cannot read and drops every message. Measured: 100 messages
produced, 0 rows landed, offsets committed at poll time, and a restart
redelivers nothing — silent at the batch level. Proto and flatbuffer break
identically.
At-most-once, hard-coded. Offsets commit with the poll request, before
the sink sees the batch. No config exposes it. This suite compares
guarantee-for-guarantee.
One lockstep connection per process. One poll in flight however many
streams are configured, so a 32-CPU envelope cannot be spent.
All three concern the connector runtime (v0.5.0-edge.6). None of them says
anything about the Iggy broker, which this suite has never measured, and
nothing published here should imply otherwise.
Revisit trigger
Reopen the arm when both hold:
Iggy's runtime propagates the payload's schema at the plugin boundary, and exposes offset-commit semantics in the sink config. Finding 2 is
the disqualifying one — with it fixed, finding 3 becomes a legitimate
publishable architectural result rather than one asterisk among three.
A bug report for finding 1 is drafted and not yet filed — it needs sign-off,
and an eventual fix PR needs the Apache ICLA. No existing upstream report covers
it; apache/iggy#3430 and its PR #3734 are a different defect on the encoder
side. A local branch carries a working fix (propagate the payload's schema
rather than the decoder's) with the SDK's 150 and the runtime's 177 tests green.
Link the upstream issue here when it is filed.
Prerequisites in this tree
Work that must land before any non-Kafka arm, not only this one:
Bind entrants to a broker family. Nothing does today. The moment a
second broker exists, bench run '*' on either environment cross-wires
every arm — bench validate passes and it fails only at runtime. Noted in feat(methodology): the broker family is a comparability axis #63 as deliberately out of scope there.
Decide how a weaker-than-at-least-once system is published, if at all. entrant.rs refuses any delivery other than at-least-once, and
"drops a guarantee" is stripped, which can never be the default while the
default must be realistic — so such a system has no valid realistic
variant. The contract anticipated a stronger guarantee, never a weaker
one. Left unwritten in feat(methodology): the broker family is a comparability axis #63 rather than designed for a hypothetical arm.
Broker pluggability in the harness.infra.rs hardcodes the Redpanda
argv and endpoints; corpus.rs (prefill, topic_message_count, ensure_topic, and easy to miss, register_schema and verify_corpus)
is rdkafka; ceiling.rs's consume pass speaks raw Kafka Fetch from a
container; driver.rs refuses nothing for sustained mode on a non-Kafka
broker.
A measured Iggy consume ceiling, on its own environment profile. The
70% headroom rule needs one per broker and Ceilings::gate refuses to
extrapolate. Its provenance must record the connection count, or it will
assert a topology the arm cannot reach (finding 3).
Incidental, for whoever picks this up
The apache/iggy image does not boot under Docker Desktop with defaults: [system.sharding] cpu_allocation = "numa:auto" fails to bind memory to a NUMA
node and the server exits. IGGY_SYSTEM_SHARDING_CPU_ALLOCATION=<n> and IGGY_SYSTEM_SHARDING_PIN_CORES=false get it up. The same knobs will matter
under a cgroup CPU cap on the reference host.
Follow-up to #62, which is settled: the contract question landed in #63 and the
arm's outcome in #64, where
entrants/iggy/is carried asstatus = "planned"with its blockers named.
This issue exists because the reasons the arm cannot be built are not ours to
fix, so they need somewhere to be watched that is not a closed issue.
Why the arm is not built
Three findings against
apache/iggyat412014a, verified in source andreproduced against a live server. The authoritative statement is
[planned].blockersinentrants/iggy/entrant.toml, which names files andlines; in short:
decoder's schema rather than the payload's type, so a sink is handed a
variant it cannot read and drops every message. Measured: 100 messages
produced, 0 rows landed, offsets committed at poll time, and a restart
redelivers nothing — silent at the batch level. Proto and flatbuffer break
identically.
the sink sees the batch. No config exposes it. This suite compares
guarantee-for-guarantee.
streams are configured, so a 32-CPU envelope cannot be spent.
All three concern the connector runtime (v0.5.0-edge.6). None of them says
anything about the Iggy broker, which this suite has never measured, and
nothing published here should imply otherwise.
Revisit trigger
Reopen the arm when both hold:
and exposes offset-commit semantics in the sink config. Finding 2 is
the disqualifying one — with it fixed, finding 3 becomes a legitimate
publishable architectural result rather than one asterisk among three.
there is something to compare against. Until then the arm is a benchmark
of one, in its own group, on its own environment.
Upstream
A bug report for finding 1 is drafted and not yet filed — it needs sign-off,
and an eventual fix PR needs the Apache ICLA. No existing upstream report covers
it;
apache/iggy#3430and its PR#3734are a different defect on the encoderside. A local branch carries a working fix (propagate the payload's schema
rather than the decoder's) with the SDK's 150 and the runtime's 177 tests green.
Link the upstream issue here when it is filed.
Prerequisites in this tree
Work that must land before any non-Kafka arm, not only this one:
second broker exists,
bench run '*'on either environment cross-wiresevery arm —
bench validatepasses and it fails only at runtime. Noted infeat(methodology): the broker family is a comparability axis #63 as deliberately out of scope there.
entrant.rsrefuses anydeliveryother thanat-least-once, and"drops a guarantee" is
stripped, which can never be the default while thedefault must be
realistic— so such a system has no validrealisticvariant. The contract anticipated a stronger guarantee, never a weaker
one. Left unwritten in feat(methodology): the broker family is a comparability axis #63 rather than designed for a hypothetical arm.
infra.rshardcodes the Redpandaargv and endpoints;
corpus.rs(prefill,topic_message_count,ensure_topic, and easy to miss,register_schemaandverify_corpus)is rdkafka;
ceiling.rs's consume pass speaks raw Kafka Fetch from acontainer;
driver.rsrefuses nothing for sustained mode on a non-Kafkabroker.
70% headroom rule needs one per broker and
Ceilings::gaterefuses toextrapolate. Its provenance must record the connection count, or it will
assert a topology the arm cannot reach (finding 3).
Incidental, for whoever picks this up
The
apache/iggyimage does not boot under Docker Desktop with defaults:[system.sharding] cpu_allocation = "numa:auto"fails to bind memory to a NUMAnode and the server exits.
IGGY_SYSTEM_SHARDING_CPU_ALLOCATION=<n>andIGGY_SYSTEM_SHARDING_PIN_CORES=falseget it up. The same knobs will matterunder a cgroup CPU cap on the reference host.