fix(deps): update module github.com/twmb/franz-go to v1.20.7

[renovate]

ℹ️ Note

This PR body was truncated due to platform limits.

This PR contains the following updates:

Package	Change	Age	Confidence
github.com/twmb/franz-go	v1.3.1 → v1.20.7

---

Warning

Some dependencies could not be looked up. Check the Dependency Dashboard for more information.

---

Release Notes

twmb/franz-go (github.com/twmb/franz-go)

v1.20.7

Compare Source

===

This patch release fixes numerous niche bugs - some user reported, some found
while investigating other things - contains a few behavior improvements,
extensive kfake additions, and many test additions / improvements.

There have been extensive additions to kfake over the course of the past
month; kfake now supports transactions, the next generation consumer group
protocol, and more Kafka APIs. franz-go integration tests now run and pass
against kfake in CI.

Testing has further been extensively improved: integration tests now run
against the latest patch version of Kafka for all major Kafka versions going
back to 0.11.0. The existing test suite has been extended to run while opting
into the next generation consumer group. An integration test against Kerberos
has been added. For ~roughly the past year (maybe half year), all bugs found
have had regression tests added in kfake. This release massively extends the
kfake test suite -- both with behavior tests that were ported via Claude
directly to franz-go (with license attribution!) and with behavior tests that
Claude generated specifically for kfake.

The "next generation" (KIP-848) consumer group code in franz-go itself has some
improvements. These improvements were found after adding 848 code to kfake,
which allowed for much faster integration test looping. This looping was still
very slow for what it's worth; towards the end, integration tests would pass
~40+ times with race mode over the course of two hours before failing once.
Claude was instrumental with adding appropriate log lines and tracing logs for
diagnosing extremely niche failures; things also got slower when a few specific
log lines that would've helped weren't added the first time...

Anyway,

Bug fixes

• Returns from PollRecords / PollFetches that contained ONLY an error (context
  cancellation or something) previously did not block rebalances, even if you
  opted into BlockRebalanceOnPoll.

• If, while idempotently producing, the client encountered TIMED_OUT while
  producing (retryable), the client considered this a "we definitively did not
  produce" state, and allowed you to cancel the records. Well, maybe the records
  actually did get produced broker side eventually, and now you re-produce new
  records - the NEW records could be "deduplicated" due to how idempotency works.
  This one is a bit niche, if you're interested, you should read #​1217 and the
  two PRs that address it.

• My original implementation of how Kerberos handled authentication was
  correct... for the time. I missed how it should have been touched up years
  ago and now 4.0 hard deprecates the old auth flow. So, that's been found,
  reported, and now fixed.

• If a partition returned a retryable error in a metadata request (odd behavior
  already) the first time the client is discovering the partition (i.e. on
  startup), the client would not retry loading the partition right away (it
  would, but much later on standard metadata refresh).

• There was a very subtle, basically un-encounterable data race while
  consuming. That was fixed in 5caaa1e0. It's so niche it's not worth
  writing more about here.

• RequestCachedMetadata, broken for a few releases, has been fixed. I plan to
  switch kadm back to using it in the next kadm release.

• There was a data race on client shutdown while leaving the group if you canceled
  the client context.

Improvements

• ConnIdleTimeout is now obeyed more exactly, allowing you to more reliably reason
  about the maximum idle time. Thanks @​carsonip!

• At the end of a group transact session, I force a heartbeat to kinda "force
  detect" the group is still alive (with some timing bounds). If the heartbeat
  detected any error - including REBALANCE_IN_PROGRESS - the session would
  abort no matter what. This has been changed to actually allow a commit (if
  that's what you're doing) in certain scenarios (notably: the client detects
  KIP-447 support on the broker and you are using RequireStable).

• SetOffsets now does not let you set offsets for partitions that are not being
  consumed. Previously, you could, but it'd just create entries in a map that were
  never used. Those entries are no longer created.

• ProduceSync now automatically un-lingers any partition that is produced to,
  causing more immediate flushes. This should resolve lag that was introduced
  from v1.20.0 where linger was set to 10ms by default. There are many usages of
  franz-go I've seen in the wild where ProduceSync is used in random places to
  produce one message before going back to other things.

• 764eb29d improvement kgo: unlinger partitions in ProduceSync to avoid linger delay
• d996bf71 bugfix kgo: fix data race during client shutdown with canceled context
• 85b2c855 behavior change kgo: fix applySetOffsets to skip partitions not being consumed
• 05de202b improvement kgo: allow commit despite rebalance with KIP-447 and RequireStable
• e8a00cd6 improvement kgo: do not reuse idle connection after connIdleTimeout (thanks @​carsonip!)
• 787ab9fe bugfix kgo: fix and enhance RequestCachedMetadata
• 5caaa1e0 bugfix kgo: fix data race in allowUsable reading c.source after Swap
• 40c144d3 bugfix kgo: check loadErr for new partitions on first metadata load
• e430ea93 bugfix kgo: send SaslHandshake for GSSAPI mechanism
• 702c4f75 bugfix kgo: error returns from polling should block rebalance
• 32e1580a 0c086b05 bugfix kgo: poison batches to prevent future cancelation on two errors

v1.20.6

Compare Source

===

This patch release has two improvements.

Previously, you could not use poll functions multiple times if using
BlockRebalanceOnPoll, because rebalancing had a higher lock priority than
polling and would block all further poll calls. This has been changed to allow
you to call poll as much as you want until you AllowRebalance. Thanks
@​KiKoS0!

If brokers indicated they supported epochs, but then used -1 everywhere for
that epoch, Mark functions would ignore records being marked and you would
never commit progress. This was due to the client defaulting to a 0 epoch
internally (and not using it if the broker did not support it), meaning -1
would be ignored. Brokers that use indicate support but use -1 are now
supported. This was only found to be a problem against Azure Event Hubs.

• 7cd5ea65 kgo: fix mark <=> epoch interaction, make epoch handling more resilient
• 94fd8622 kgo: fix deadlock when polling multiple times while blocked from a rebalance

v1.20.5

Compare Source

===

This fixes a commit in 1.20.4 that accidentally broke client metrics (KIP-714)
and inadvertently made a log spammy. In addition to the fix, a few logs around
client metrics have been reduced in severity.

The new-as-of-1.20 OnPartitionsCallbackBlocked is now called in a goroutine,
reducing the chance that you accidentally run into a deadlock based on how you
structure handling the hook.

Deps have been bumped to eliminate any security scanners that flag on CVEs
(even though this is a library and you can bump the dep in your own binary).

The kgo.Fetches.Errors doc has been expanded to account for previously
undocumented errors, and updates guidance on what's retryable vs what is not.

• e86bb6c9 kgo: info=>debug for a few logs in client metrics
• 7c7ca2b4 kgo: call OnPartitionsCallbackBlocked concurrently
• ebf29a4a all: bump deps
• 97b4a1d4 kgo.Fetches.Errors doc: clarify && expand for two undoc'd errors
• 13ea38e3 bug kgo: fix remaining usage of kgo.maxVers/kgo.maxVersion (thanks @​vincentbernat!)

v1.20.4

Compare Source

===

This patch release contains fixes for two data races: one new one introduced in
1.20.3 with sharded requests (a super obvious oversight in retrospect..) and a
fix for a hard to encounter race that has existed for years when using
preferred read replicas (via setting the kgo.Rack option while consuming).

There are also improvements by @​carsonip to log
context errors less and to return earlier when pinging if the context is
expired, and an improvement by @​rockwotj to
support arbitrary Kafka messages (i.e., custom requests).

• cdd0316a kgo: source.fetch stop fetching again when context is done
• d0350d13 bugfix kgo: fix sharded-request "minor" data race introduced in 1.20.3
• 566201c3 bugfix kgo: fix data race with replica fetching / moving; kfake: add test
• 4e4ff88f kgo: Do not try next broker in Client.Ping if context is done
• 60b8d5b5 kgo: support arbitrary Kafka RPCs

v1.20.3

Compare Source

===

This patch release fixes one bug that has existed since 1.19.0 and improves
retry behavior on dial failures.

The bug: 1.19 introduced code to, when follower fetching, re-check the
preferred replica from the leader every 30 minutes (every RecheckPreferredReplicaInterval).
The logic switched back to a leader after handling a fetch response, using
the offset that the fetch request was issued with. The client would give you
data that it just fetched, go back to the leader, and then get redirected back
to the follower using the offset from before the fetch it just gave you.
You would then receive a bit of duplicate data as the pre-fetch offset is
re-fetched. This is no longer the case.

The improvement: on sharded requests (certain requests that may need to be
split and sent to many brokers), dial errors were not retried. They are now
retried.

• d5085e90 kgo: retry dial errors on sharded requests if possible
• 70c81779 kgo source: expired old preferred replicas while creating req, not handling resp

v1.20.2

Compare Source

===

This patch release fixes a field-access data race that has been around forever.
Specifically, if a partition was moving from one broker to another via a
metadata update at the same time a linger timer fired, there was a data race
reading a pointer that was being written. Most 64 bit systems don't experience
corruption with this type of race, so the code would execute fine but you may
have the old sink start draining when the new sink should have.

This also further improves some linger logic.

• 73c16c1d kgo: do not trigger draining early if a partition moves sinks while lingering
• d5066143 kgo: fix data race when the linger timer fires

v1.20.1

Compare Source

===

This small patchfix release fixes a longstanding bug in RequestCachedMetadata,
which became a problem now that kadm is using it by default: if no metadata was
cached and you requested all topics, no metadata request would be issued and
you'd get no valid response. Thank you @​countableSet
for the find and fix.

This also adds the two new 1.20 config options to OptValues, and a big doc
comment hinting to add new config opts going forward.

NOTE Follow up testing showed there are still more long-standing bugs with
RequestCachedMetadata. Usage of that function has been reverted from kadm
for the time being (which is, in the open source ecosystem, the only place this
function was ever used). All users of kadm v1.17.0 should bump to v1.17.1.

• 1087d3c7 kgo: add new opts to OptValues && big doc to do so going forward...
• cad283f0 bugfix kgo: fix for empty fetch mapped metadata (#​1143)

v1.20.0

Compare Source

===

This is a comparatively small minor release that adds support for Kafka 4.1,
adds three new APIs, fixes four bugs (read below to gauge importance), has a
few improvements, and switches the client from a default of 0ms linger to a
default of 10ms linger.

Also of note: a new srfake package has been created so you can run a fake
Schema Registry server in your CI tests (thank you @​weeco).
This complements the existing kfake package that allows you to run a fake
in-memory Kafka "cluster" for unit testing. If you did not know of either of these,
check them out! kfake supports many Kafka features, but transactions are still WIP.
All franz-go tests except transaction based tests pass against a kfake "cluster",
so odds are, it'll work for you.

There are a few external contributors this release to features, docs, bugs, and
internal improvements. If I do not call you out below directly, please know I'm
thankful for your contributions!

Behavior changes

• This library now lingers by default for 10ms. You can switch back to 0ms
  lingering by adding kgo.ProducerLinger(0) to your options
  when initializing the client. The original theory for 0ms linger was more of
  a theory, and years of practice has shown that even a tiny linger can be
  beneficial to the throughput and batching of clients.
  See #​1072 for more details.

Bug fixes

• Metadata refreshes could panic if a very specific flow of events happened,
  specifically only on a cluster that is transitioning from not using topic IDs
  to using topic IDs, and only if the transition is not implemented 100% correctly.
  This bug has existed for years and was only encountered during the recent addition
  of topic IDs to Redpanda. See 645f1126 for more details.

• The loop that determines whether more batches exist to be produced had its
  conditional backwards. This was hidden forever due to other minor logic flaws
  that caused the "do more batches exist?" check to occur more than it should
  have, so the bug caused no problems. The "do more batches exist?" checks have
  been improved and the conditional has been fixed.

• The internal linger timers fired way more than they needed to, causing
  batches to be cut WAY more frequently than they needed to when using
  lingering. The logic here has been fixed, so lingering should actually run
  its full time now and batches should be bigger.

• Azure resets connections when speaking ApiVersions v4. v1.19 of this library
  detected this resetting and after 3 attempts, downgrades to ApiVersions v3.
  However, the connection reset error is different when running on Windows.
  The code has been improved to detect the proper syscall when this library
  is running on Windows. Thanks @​axw!

Improvements

• Previously, producing was limited to 5 inflight requests per broker even if
  you disabled idempotence and bumped the number. The inflight limit is now
  unbounded. Thanks @​pracucci!

Features

• OnPartitionsCallbackBlocked now exists so that, if you are using
  BlockRebalanceOnPoll, you can be notified that a rebalance is desired. If
  your record processing function is slow, this allows you to interrupt your
  batch processing (if possible), wrap up committing, and allow a rebalance
  before your client is kicked from the group.

• ConsumeExcludeTopics, if you are using regex consuming, allows you to have
  a higher-priority set of regular expressions to exclude topics from being
  consumed. This is useful if you want to consume everything except a set of
  topics (for example, if you are replicating topics from one cluster to
  another). Thanks @​mmatczuk!

• Fetches.RecordsAll now exists to return a Go iterator for use in range loops.
  Thanks @​narqo!

Relevant commits

• 1844d216 feature kgo: add OnPartitionsCallbackBlocked
• 157580fd kgo.RequestSharded: support ConsumerGroupDescribe, ShareGroupDescribe
• f176953e behavior change kgo lingering: default to 10ms
• 679f7c3d kgo: add support for produce v13
• f7f61420 generate: new definitions for share requests
• 32997347 generate: new non-share protocols for kafka 4.1
• be947c20 bench: add -batch-recs, -psync, -pgoros
• 0b1dbf0c bugfix kgo: multiple linger fixes
• 2ea3251d bugfix sink: fix old bug determining whether more batches should be produced
• 195bed84 feature kgo: add ConsumeExcludeTopics
• 645f9d4b bugfix Check errno for (WSA)ECONNRESET
• 645f1126 bugfix kgo: fix panic in metadata updates from inconsistent broker state
• 612f26b6 feature kgo: add Fetches.RecordsAll to return a Go native iterator
• ce2bcd18 improvement kgo: use unlimited ring buffers in the Produce path, allowing >5 inflight requests

v1.19.5

Compare Source

===

Fixes a bug introduced in 1.19.3 that caused batched FindCoordinator requests
to no longer work against older brokers (Kafka brokers before 2.4, or all
Redpanda versions brokers).

All credit to @​douglasbouttell for exactly diagnosing the bug.

• 06272c66 bugfix kgo: bugfix batched FindCoordinator requests against older brokers

v1.19.4

Compare Source

===

Fixes one bug introduced from the prior release (an obvious data race in
retrospect), and one data race introduced in 1.19.0. I've looped the tests more
in this release and am not seeing further races. I don't mean to downplay the
severity here, but these are races on pointer-sized variables where reading the
before or after state is of little difference. One of the read/write races is
on a context.Context, so there are actually two pointer sized reads & writes --
but reading the (effectively) type vtable for the new context and then the data
pointer for the old context doesn't really break things here. Anyway, you
should upgrade.

This also adds a workaround for Azure EventHubs, which does not handle
ApiVersions correctly when the broker does not recognize the version we are
sending. The broker should reply with an UNSUPPORTED_VERSION error and
reply with the version the broker can handle. Instead, Azure is resetting the
connection. To workaround, we detect a cxn reset twice and then downgrade the
request we send client side to 0.

• 7910f6b6 kgo: retry connection reset by peer from ApiVersions to work around EventHubs
• d310cabd kgo: fix data read/write race on ctx variable
• 7a5ddcec kgo bugfix: guard sink batch field access more

v1.19.3

Compare Source

===

This release fully fixes (and has a positive field report) the KIP-890 problem
that was meant to be fixed in v1.19.2. See the commit description for more
details.

• a13f633b kgo: remove pinReq wrapping request

v1.19.2

Compare Source

===

This release fixes two bugs, a data race and a misunderstanding in some of the
implementation of KIP-890.

The data race has existed for years and has only been caught once. It could
only be encountered in a specific section of decoding a fetch response WHILE a
metadata response was concurrently being handled, and the metadata response
indicated a partition changed leaders. The race was benign; it was a read race,
and the decoded response is always discarded because a metadata change
happened. Regardless, metadata handling and fetch response decoding are no
longer concurrent.

For KIP-890, some things were not called out all to clearly (imo) in the KIP.
If your 4.0 cluster had not yet enabled the transaction.version feature v2+,
then transactions would not work in this client. As it turns out, Kafka 4
finally started using a v2.6 introduced "features" field in a way that is
important to clients. In short: I opted into KIP-890 behavior based on if a
broker could handle requests (produce v12+, end txn v5+, etc). I also needed to
check if "transaction.version" was v2+. Features handling is now supported in
the client, and this single client-relevant feature is now implemented.

See the commits for more details.

• dda08fd9 kgo: fix KIP-890 handling of the transaction.version feature
• 8a364819 kgo: fix data race in fetch response handling

v1.19.1

Compare Source

===

This release fixes a very old bug that finally started being possible to hit in
v1.19.0. The v1.19.0 release does not work for Kafka versions pre-4.0. This
release fixes that (by fixing the bug that has existed since Kafka 2.4) and
adds a GH action to test against Kafka 3.8 to help prevent regressions against
older brokers as this library marches forward.

• 50aa74f1 kgo bugfix: ApiVersions replies only with key 18, not all keys

v1.19.0

Compare Source

===

This is the largest release of franz-go yet. The last patch release was Jan 20, '25.
The last minor release was Oct 14, '24.

A big reason for delays the past few month+ has been from spin looping tests
and investigating any issue that popped up. Another big delay is that Kafka has
a full company adding features -- some questionable -- and I'm one person that
spent a significant amount of time catching this library up with the latest
Kafka release. Lastly, Kafka released Kafka v3.9 three weeks after my last
major release, and simultaneously, a few requests came in for new features in
this library that required a lot of time. I wanted a bit of a break and only
resumed development more seriously in late Feb. This release is likely >100hrs
of work over the last ~4mo, from understanding new features and implementing
them, reviewing PRs, and debugging rare test failures.

The next Kafka release is poised to implement more large features (share
groups), which unfortunately will mean even more heads down time trying to bolt
in yet another feature to an already large library. I hope that Confluent
chills with introducing massive client-impacting changes; they've introduced
more in the past year than has been introduced from 2019-2023.

Bug fixes / changes / deprecations

• The BasicLogger will no longer panic if only a single key (no val) is used. Thanks @​vicluq!

• An internal coding error around managing fetch concurrency was fixed. Thanks @​iimos!

• Some off by ones with retries were fixed (tldr: we retried one fewer times than configured)

• AllowAutoTopicCreation and ConsumeRegex can now be used together.
  Previously, topics would not be created if you were producing and consuming
  from the same client AND if you used the ConsumeRegex option.

• A data race in the consumer code path has been fixed. The race is hard to
  encounter (which is why it never came up even in my weeks of spin-looping
  tests with -race). See PR #​984
  for more details.

• EndBeginTxnUnsafe is deprecated and unused. EndAndBeginTransaction now
  flushes, and you cannot produce while the function happens (the function will
  just be stuck flushing). As of KIP-890, the behavior that the library relied on
  is now completely unsupported. Trying to produce while ending & beginning a
  transaction very occasionally leads to duplicate messages. The function now is
  just a shortcut for flush, end, begin.

• The kversion package guts have been entirely reimplemented; version guessing
  should be more reliable.

• OnBrokerConnect now encompasses the entire SASL flow (if using SASL) rather
  than just connection dialing. This allows you more visibility into successful
  or broken connections, as well as visibility into how long it actually takes
  to initialize a connection. The dialDur arg has been renamed to initDur.
  You may see the duration increase in your metrics. enough If feedback comes
  in that this is confusing or unacceptable, I may issue a patch to revert
  the change and instead introduce a separate hook in the next minor release.
  I do not aim to create another minor release for a while.

Features / improvements

• This release adds support for user-configurable memory pooling to a few select
  locations. See any "Pool" suffixed interface type in the documentation. You can
  use this to add bucketed pooling (or whatever strategy you choose) to cut down
  on memory waste in a few areas. As well, a few allocations that were previously
  many-tiny allocs have been converted to slab allocations (slice backed). Lastly,
  if you opt into kgo.Record pooling, the Record type has a new Recycle
  method to send it and all other pooled slices back to their pools.

• You can now completely override how compression or decompression is done via
  the new WithCompressor and WithDecompressor options. This allows you to
  use libraries or options that franz-go does not automatically support, perhaps
  opting for higher performance libraries or options or using memory more memory
  pooling behind the scenes.

• ConsumeResetOffset has been split into two options, ConsumeResetOffset and
  ConsumeStartOffset. The documentation has been cleaned up. I personally always
  found it confusing to use the reset offset for both what to start consuming from
  and what to reset to when the client sees an offset out of range error. The start
  offset defaults to the reset offset (and vice versa) if you only set one.

• For users that produce infrequently but want the latency to be low when producing,
  the client now has a EnsureProduceConnectionIsOpen method. You can call this
  before producing to force connections to be open.

• The client now has a RequestCachedMetadata function, which can be used to
  request metadata only if the information you're requesting is not cached,
  or is cached but is too stale. This can be very useful for admin packages that
  need metadata to do anything else -- rather than requesting metadata for every
  single admin operation, you can have metadata requested once and use that
  repeatedly. Notably, I'll be switching kadm to using this function.

• KIP-714 support: the client now internally aggregates a small set of metrics
  and sends them to the broker by default. This client implements all required
  metrics and a subset of recommended metrics (the ones that make more sense).
  To opt out of metrics collection & sending to the broker by default, you
  can use the new DisableClienMetrics option. You can also provide your own
  metrics to send to the broker via the new UserMetricsFn option. The client
  does not attempt to sanitize any user provided metric names; be sure you provide
  the names in the correct format (see docs).

• KIP-848 support: this exists but is hidden. You must explicitly opt in by using
  the new WithContext option, and the context must have a special string key,
  opt_in_kafka_next_gen_balancer_beta. I noticed while testing that if you
  repeat ConsumerGroupHeartbeat requests (i.e. what can happen when clients
  are on unreliable networks), group members repeatedly get fenced. This is
  recoverable, but it happens way way more than it should and I don't believe
  the broker implementation to be great at the moment. Confluent historically
  ignores any bug reports I create on the KAFKA issue tracker, but if you
  would like to follow along or perhaps nudge to help get a reply, please
  chime in on KAFKA-19222, KAFKA-19233, and KAFKA-19235.

• A few other more niche APIs have been added. See the full breadth of new APIs
  below and check pkg.go.dev for docs for any API you're curious about.

API additions

This section contains all net-new APIs in this release. See the documentation
on pkg.go.dev.

const (
        CodecNone CompressionCodecType = iota
        CodecGzip
        CodecSnappy
        CodecLz4
        CodecZstd
        CodecError = -1
)
const CompressDisableZstd CompressFlag = 1 + iota
const (
    MetricTypeSum = 1 + iota
    MetricTypeGauge
)

type CompressFlag uint16
type CompressionCodecType int8
type Compressor interface {
    Compress(dst *bytes.Buffer, src []byte, flags ...CompressFlag) ([]byte, CompressionCodecType)
}
type Decompressor interface {
    Decompress(src []byte, codecType CompressionCodecType) ([]byte, error)
}
type Metric struct {
        Name string
        Type MetricType
        ValueInt int64
        ValueFloat float64
        Attrs map[string]any
}
type MetricType uint8
type Pool any
type PoolDecompressBytes interface {
        GetDecompressBytes(compressed []byte, codec CompressionCodecType) []byte
        PutDecompressBytes([]byte)
}
type PoolKRecords interface {
        GetKRecords(n int) []kmsg.Record
        PutKRecords([]kmsg.Record)
}
type PoolRecords interface {
        GetRecords(n int) []Record
        PutRecords([]Record)
}
type ProcessFetchPartitionOpts struct {
        KeepControlRecords bool
        DisableCRCValidation bool
        Offset int64
        IsolationLevel IsolationLevel
        Topic string
        Partition int32
        Pools []Pool
}

func DefaultCompressor(...CompressionCodec) (Compressor, error)
func DefaultDecompressor(...Pool) Decompressor
func IsRetryableBrokerErr(error) bool
func ProcessFetchPartition(ProcessFetchPartitionOpts, *kmsg.FetchResponseTopicPartition, Decompressor, func(FetchBatchMetrics)) (FetchPartition, int64)

func DisableClientMetrics() Opt
func OnRebootstrapRequired(func() ([]string, error)) Opt
func UserMetricsFn(fn func() iter.Seq[Metric]) Opt
func WithContext(ctx context.Context) Opt
func WithPools(pools ...Pool) Opt

func ConsumeStartOffset(Offset) ConsumerOpt
func DisableFetchCRCValidation() ConsumerOpt
func RecheckPreferredReplicaInterval(time.Duration) ConsumerOpt
func WithDecompressor(decompressor Decompressor) ConsumerOpt

func DefaultProduceTopicAlways() ProducerOpt
func WithCompressor(Compressor) ProducerOpt

func (*Client) Context() context.Context
func (*Client) EnsureProduceConnectionIsOpen(context.Context, ...int32) error
func (*Client) RequestCachedMetadata(context.Context, *kmsg.MetadataRequest, time.Duration) (*kmsg.MetadataResponse, error)

func (*Record) Recycle()

Relevant commits

This is a small selection of what I think are the most pertinent commits in
this release. This release is very large, though. Many commits and PRs have
been left out that introduce or change smaller things.

• 07e57d3e kgo: remove all EndAndBeginTransaction internal "optimizations"
• a54ffa96 kgo: add ConsumeStartOffset, expand offset docs, update readme KIPs
• PR #​988#​988 kgo: add support for KIP-714 (client metrics)
• 7a17a03c kgo: fix data race in consumer code path
• ae96af1d kgo: expose IsRetryableBrokerErr
• 1eb82fee kgo: add EnsureProduceConnectionIsOpen
• fc778ba8 kgo: fix AllowAutoTopicCreation && ConsumeRegex when used together
• ae7eea7c kgo: add DisableFetchCRCValidation option
• 6af90823 kgo: add the ability to pool memory in a few places while consuming
• 8c7a36db kgo: export utilities for decompressing and parsing partition fetch responses
• 33400303 kgo: do a slab allocation for Record's when processing a batch
• 39c2157a kgo: add WithCompressor and WithDecompressor options
• 9252a6b6 kgo: export Compressor and Decompressor
• be15c285 kgo: add Client.RequestCachedMetadata
• fc040bc0 kgo: add OnRebootstrapRequired
• c8aec00a kversion: document changes through 4.0
• 718c5606 kgo: remove all code handling EndBeginTxnUnsafe, make it a no-op
• 5494c59e kversions: entirely reimplement internals
• 9d266fcd kgo: allow outstanding produce requests to be context canceled if the user disables idempotency
• c60bf4c2 kgo: add DefaultProduceTopicAlways ProducerOpt
• 50cfe060 kgo: fix off-by-one with retries accounting
• e9ba83a6, 05099ba0 kgo: add WithContext, Client.Context()
• ddb0c0c3 kgo: fix cancellation of a fetch in manageFetchConcurrency
• 83843a53 kgo: fixed panic when keyvals len equals 1

v1.18.1

Compare Source

===

This patch release contains a myriad of fixes for relatively minor bugs, a
few improvements, and updates all dependencies. Both pkg/kadm and pkg/sr
are also being released as minors in tandem with a few quality of life APIs.

Bug fixes

• Previously, if records were successfully produced but returned with an
  invalid offset (-1), the client would erroneously return bogus offsets
  to the end users. This has been fixed to return -1. (Note this was never
  encountered in the wild).

• Pausing topics & partitions while using PollRecords previously could result
  in incorrect accounting in BufferedFetchRecords and BufferedFetchBytes,
  permanently causing the numbers returned to be larger than reality. That is,
  it is possible the functions would return non-zero numbers even though nothing
  was buffered.

• When consuming from a follower (i.e. you were using the Rack option and your
  cluster is configured with follower fetching), if the follower you consumed from
  had a higher log start offset than the leader, and if you were trying to consume
  from an early offset that exists on the leader but not on the follower, the client
  would enter a permanent spinloop trying to list offsets against the follower.
  This is due to KIP-320 case 3, which mentions that clients should send a ListOffsets
  to the follower -- this is not the case, Kafka actually returns NotLeaderOrFollower
  when sending that request to the follower. Now the client clears the preferred replica
  and sends the next fetch request to the leader, at which point the leader can either
  serve the request or redirect back to a different preferred replica.

Improvements

• When finishing batches, if any records were blocked in Produce due to
  the client hitting the maximum number of buffered records, the client would broadcast
  to all waiters that a message was finished for every message finished until there were
  no other goroutines waiting to try to produce. When lingering
  is enabled, linger occurs except when the client has reached the maximum number of
  buffered records. Once the client is as max buffered records, the client tries to flush until more records can be buffered.
  If you have a few concurrent producers, they will all hang trying to buffer. As soon
  as one is signaled, it will grab the free spot, enter into the client as buffered,
  and then see the client is now again at max buffered and immediately create a batch
  rather than lingering. Thus, signalling one at a time would cause many small single-record
  batches to be created and each cause a round trip to the cluster. This would result in slow performance.
  Now, by finishing a batch at a time, the client opens many slots at a time for any producers waiting,
  and ideally they can fit into being buffered without hitting max buffered and clearing any linger state.
  Note that single-message batches can still cause the original behavior, but there is not
  much more that can be done.

• Decompression errors encountered while consuming are now returned to the end user, rather
  than being stripped internally. Previously, stripping the error internally would result in
  the client spinlooping: it could never make forward progress and nothing ever signaled the
  end user that something was going wrong.

Relevant commits

• 13584b5 feature kadm: always request authorized operations
• 847095b bugfix kgo: redirect back to the leader on KIP-392 case 3 failure
• d6d3015 feature pkg/sr: add PreReq option (and others by @​mihaitodor, thank you!)
• 1473778 improvement kgo: return decompression errors while consuming
• 3e9beae bugfix kgo: fix accounting when topics/partitions are {,un}paused for PollRecords
• ead18d3 improvement kgo: broadcast batch finishes in one big blast
• aa1c73c feature kadm: add func to decode AuthorizedOperations (thanks @​weeco!)
• f66d495 kfake: do not listen until the cluster is fully set up
• 2eed36e bugfix pkg/kgo: fix handling of invalid base offsets (thanks @​rodaine!)

v1.18.0

Compare Source

===

This release adds support for Kafka 3.7, adds a few community requested APIs,
some internal improvements, and fixes two bugs. One of the bugfixes is for a
deadlock; it is recommended to bump to this release to ensure you do not run
into the deadlock. The features in this release are relatively small.

This adds protocol support for KIP-890 and KIP-994, and
adds further protocol support for [KIP-848][KIP-848]. If you are using
transactions, you may see a new kerr.TransactionAbortable error, which
signals that your ongoing transaction should be aborted and will not be
successful if you try to commit it.

Lastly, there have been a few improvements to pkg/sr that are not mentioned
in these changelog notes.

Bug fixes

• If you canceled the context used while producing while your client was
  at the maximum buffered records or bytes, it was possible to experience
  deadlocks. This has been fixed. See #​832 for more details.

• Previously, if using GetConsumeTopics while regex consuming, the function
  would return all topics ever discovered. It now returns only the topics that
  are being consumed.

Improvements

• The client now internaly ignores OutOfOrderSequenceNumber errors that are
  encountered when consuming if possible. If a producer produces very infrequently,
  it is possible the broker forgets the producer by the next time the producer
  produces. In this case, the producer receives an OutOfOrderSequenceNumber error.
  The client now internally resets properly so that you do not see the error.

Features

• AllowRebalance and CloseAllowingRebalance have been added to GroupTransactSession.
• The FetchTopic type now has includes the topic's TopicID.
• The ErrGroupSession internal error field is now public, allowing you to test how you handle the internal error.
• You may now receive a kerr.TransactionAbortable error from many functions while using transactions.

Relevant commits

• 0fd1959d kgo: support Kafka 3.8's kip-890 modifications
• 68163c55 bugfix kgo: do not add all topics to internal tps map when regex consuming
• 3548d1f7 improvement kgo: ignore OOOSN where possible
• 6a759401 bugfix kgo: fix potential deadlock when reaching max buffered (records|bytes)
• 4bfb0c68 feature kgo: add TopicID to the FetchTopic type
• 06a9c47d feature kgo: export the wrapped error from ErrGroupSession
• 4affe8ef feature kgo: add AllowRebalance and CloseAllowingRebalance to GroupTransactSession

v1.17.1

Compare Source

===

This patch release fixes four bugs (two are fixed in one commit), contains two
internal improvements, and adds two other minor changes.

Bug fixes

• If you were using the MaxBufferedBytes option and ever hit the max, odds are
  likely that you would experience a deadlock eventually. That has been fixed.

• If you ever produced a record with no topic field and without using DefaultProduceTopic,
  or if you produced a transactional record while not in a transaction, AND if the client
  was at the maximum buffered records, odds are you would eventually deadlock.
  This has been fixed.

• It was previously not possible to set lz4 compression levels.

• There was a data race on a boolean field if a produce request was being
  written at the same time a metadata update happened, and if the metadata
  update has an error on the topic or partition that is actively being written.
  Note that the race was unlikely and if you experienced it, you would have noticed
  an OutOfOrderSequenceNumber error. See this comment
  for more details.

Improvements

• Canceling the context you pass to Produce now propagates in two more areas:
  the initial InitProducerID request that occurs the first time you produce,
  and if the client is internally backing off due to a produce request failure.
  Note that there is no guarantee on which context is used for cancelation if
  you produce many records, and the client does not allow canceling if it is
  currently unsafe to do so. However, this does mean that if your cluster is
  somewhat down such that InitProducerID is failing on your new client, you
  can now actually cause the Produce to quit. See this comment
  for what it means for a record to be "safe" to fail.

• The client now ignores aborted records while consuming only if you have
  configured FetchIsolationLevel(ReadCommitted()). Previously, the client relied
  entirely on the FetchResponse AbortedTransactions field, but it's possible
  that brokers could send aborted transactions even when not using read committed.
  Specifically, this was a behavior difference in Redpanda, and the KIP that introduced
  transactions and all relevant documents do not mention what the broker behavior
  actually should be here. Redpanda itself was also changed to not send aborted
  transactions when using read committed, but we may as well improve franz-go as well.

• Decompression now better reuses buffers under the hood, reducing allocations.

• Brokers that return preferred replicas to fetch from now causes an info level
  log in the client.

Relevant commits

• 305d8dc kgo: allow record ctx cancelation to propagate a bit more
• 24fbb0f bugfix kgo: fix deadlock in Produce when using MaxBufferedBytes
• 1827add bugfix kgo sink: fix read/write race for recBatch.canFailFromLoadErrs
• d7ea2c3 bugfix fix setting lz4 compression levels (thanks @​asg0451!)
• 5809dec optimise: use byteBuffer pool in decompression (thanks @​kalbhor!)
• cda897d kgo: add log for preferred replicas
• e62b402 improvement kgo sink: do not back off on certain edge case
• 9e32bf9 kgo: ignore aborted txns if using READ_UNCOMMITTED

v1.17.0

Compare Source

===

This long-coming release, four months after v1.16.0, adds support for Kafka 3.7
and adds a few community added or requested APIs. There will be a kadm release
shortly following this one, and maybe a plugin release.

This adds full support for KIP-951, as well as protocol support for
KIP-919 (which has no client facing features) and KIP-848
(protocol only, not the feature!). KIP-951 should make the client faster at
handling when the broker moves partition leadership to a different broker.

There are two fairly minor bug fixes in the kgo package in this release, both
described below. There is also one bugfix in the pkg/sr independent (and
currently) untagged module. Because pkg/sr is untagged, the bugfix was released
a long time ago, but the relevant commit is still mentioned below.

Bug fixes

• Previously, upgrading a consumer group from non-cooperative to cooperative
  while the group was running did not work. This is now fixed (by @​hamdanjaveed, thank you!).

• Previously, if a cooperative consumer group member rebalanced while fetching
  offsets for partitions, if those partitions were not lost in the rebalance,
  the member would call OnPartitionsAssigned with those partitions again.
  Now, those partitions are passed to OnPartitionsAssigned only once (the first time).

Improvements

• The client will now stop lingering if you hit max buffered records or bytes.
  Previously, if your linger was long enough, you could stall once you hit
  either of the Max options; that is no longer the case.

• If you are issuing admin APIs on the same client you are using for consuming
  or producing, you may see fewer metadata requests being issued.

There are a few other even more minor improvements in the commit list if you
wish to go spelunking :).

Features

• The Offset type now has a new method AtCommitted(), which causes the
  consumer to not fetch any partitions that do not have a previous commit.
  This mirrors Kafka's auto.offset.reset=none option.

• KIP-951, linked above and the commit linked below, improves latency around
  partition leader transfers on brokers.

• Client.GetConsumeTopics allows you to query what topics the client is
  currently consuming. This may be useful if you are consuming via regex.

• Client.MarkCommitOffsets allows you to mark offsets to be committed in
  bulk, mirroring the non-mark API CommitOffsets.

Relevant commits

franz-go

• a7caf20 feature kgo.Offset: add AtCommitted()
• 55dc7a0 bugfix kgo: re-add fetch-canceled partitions AFTER the user callback
• db24bbf improvement kgo: avoid / wakeup lingering if we hit max bytes or max records
• 993544c improvement kgo: Optimistically cache mapped metadata when cluster metadata is periodically refreshed (thanks @​pracucci!)
• 1ed02eb feature kgo: add support for KIP-951
• 2fbbda5 bugfix fix: clear lastAssigned when revoking eager consumer
• d9c1a41 pkg/kerr: add new errors
• 54d3032 pkg/kversion: add 3.7
• 892db71 pkg/sr bugfix sr SubjectVersions calls pathSubjectVersion
• ed26ed0 feature kgo: adds Client.GetConsumeTopics (thanks @​UnaffiliatedCode!)
• 929d564 feature kgo: adds Client.MarkCommitOffsets (thanks @​sudo-sturbia!)

kfake

kfake as well has a few improvements worth calling out:

• [18e2cc3](https://redirect

---

Configuration

📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.

---

• If you want to rebase/retry this PR, check this box

---

This PR was generated by Mend Renovate. View the repository job log.

[renovate]

ℹ Artifact update notice

File name: go.mod

In order to perform the update(s) described in the table above, Renovate ran the go get command, which resulted in the following additional change(s):

• 3 additional dependencies were updated
• The go directive was updated for compatibility reasons

Details:

Package	Change
go	1.17 -> 1.24.0
github.com/twmb/franz-go/pkg/kmsg	v0.0.0-20220114004744-91b30863ac2f -> v1.12.0
github.com/klauspost/compress	v1.13.6 -> v1.18.1
github.com/pierrec/lz4/v4	v4.1.11 -> v4.1.22

[renovate]

ℹ️ Artifact update notice

File name: go.mod

In order to perform the update(s) described in the table above, Renovate ran the go get command, which resulted in the following additional change(s):

• 3 additional dependencies were updated
• The go directive was updated for compatibility reasons

Details:

Package	Change
go	1.17 -> 1.24.0
github.com/twmb/franz-go/pkg/kmsg	v0.0.0-20220114004744-91b30863ac2f -> v1.12.0
github.com/klauspost/compress	v1.13.6 -> v1.18.4
github.com/pierrec/lz4/v4	v4.1.11 -> v4.1.25