How should compatibility failures be handled during rolling deployments?

[rolandbeisel]

Context

During a rolling deployment, different application versions can process records from the same outbox.

A newer instance may persist a record whose payload type or handler is not available on an older instance that currently owns the corresponding partition.

This creates two compatibility failures before handler execution:

1. The persisted payload type cannot be loaded or deserialized.
2. The payload is available, but the persisted handler ID is not registered.

The problem is described in #464. PR #470 contains a working implementation of one possible approach. The implementation should be treated as a concrete proposal and source of practical evidence, rather than as a decision that has already been finalized.

Decision to Make

We need to decide how Namastack Outbox should classify, retain, retry, and expose compatibility failures during mixed-version deployments.

In particular:

• Should a compatibility failure consume the handler-delivery retry budget?
• Should compatibility state be persisted?
• When and how should compatibility be reevaluated?
• How should record-key ordering be preserved?
• What should happen when compatibility never becomes available?
• Should eager record materialization be changed as part of this decision?

Terminology

Delivery failure

A registered handler was invoked and threw an exception.

Compatibility failure

Handler execution could not begin because the current instance cannot load the persisted payload type or cannot find the persisted handler.

This distinction is important because a compatibility failure does not prove that delivery itself has failed.

Required Semantics

The selected solution should address the following:

• Actual handler failures continue to use the configured delivery retry policy.
• An unavailable payload type and an unavailable handler behave consistently.
• Later records for the same record key are not processed past an incompatible
  record when doing so could violate ordering.
• A compatible instance can eventually process the record.
• JDBC, JPA, and MongoDB provide equivalent behavior.
• Behavior during mixed-version deployments is backwards compatible.
• Permanent incompatibility remains operationally visible.
• The solution has a clearly documented deployment contract.

Deployment Assumption

A library cannot reliably determine whether application code is:

• temporarily unavailable during a rolling deployment;
• permanently removed;
• incorrectly configured; or
• absent because of a programming error.

The proposed deployment contract therefore follows an expand-and-contract model:

1. Deploy the new payload class and handler to every processing instance.
2. Wait until the rollout is complete.
3. Enable production of the new record type.

For removal:

1. Stop producing the old record type or handler ID.
2. Wait until the corresponding records have drained.
3. Remove the application code.

This assumption is part of the decision and may be challenged in this discussion.

Considered Options

Option A: Instance-local compatibility exclusions

This is the approach currently implemented in PR #470.

• Introduce explicit exceptions for unavailable payload types and handlers.
• Do not increment failureCount.
• Leave the record unchanged in NEW.
• Remember unavailable payload types and handler IDs in a bounded,
  instance-local data structure.
• Exclude affected records during repository selection.
• Preserve record-key ordering.
• Reevaluate compatibility after the instance is restarted or replaced.

Advantages

• No schema migration.
• No additional persisted retry mechanism.
• Delivery retries remain reserved for actual handler invocations.
• Old instances do not permanently fail records produced by newer instances.
• A newly deployed compatible instance starts without inherited exclusions.
• The implementation works consistently across JDBC, JPA, and MongoDB.

Disadvantages

• Compatibility state is not visible in the database.
• Permanently incompatible records remain pending (status = NEW) until the application or data
  is corrected.
• Repository queries become more complex.
• Eager materialization can prevent compatible predecessors from being processed
  during the first encounter with an incompatible record.

Option B: Persist a dedicated compatibility state

Persist a separate status, retry timestamp, failure category, or compatibility counter.

Advantages

• Compatibility failures and retry timing are visible across instances.
• Backoff survives restarts and ownership changes.
• Permanently incompatible records can eventually reach an explicit terminal
  state.

Disadvantages

• Requires schema and persistence-model changes.
• Introduces a second persistent retry lifecycle.
• Mixed-version deployments must handle instances that do not understand the new
  state.
• Recovery and ownership semantics become more complex.
• The library still cannot reliably determine whether incompatibility is
  temporary or permanent.

Option C: Reuse the delivery failure state

Increment failureCount and process compatibility through the existing retry policy.

Variants include interpreting negative counts or recognizing a compatibility category from failureReason.

Advantages

• Reuses existing persistence fields and retry infrastructure.
• Can eventually fail permanently incompatible records.

Disadvantages

• Compatibility attempts consume or alter the handler's delivery budget.
• A record may become FAILED before its handler has ever been invoked.
• Missing handlers cannot provide a handler-specific retry policy.
• Signed counters or encoded failure reasons change existing field semantics.
• Older instances may misinterpret the stored state.

Option D: Lazy or metadata-first record materialization

Load record metadata first and defer payload resolution until the payload is accessed.

This could allow compatible predecessors to be processed before the first incompatible record is encountered.

Advantages

• Improves isolation between records returned for the same key.
• Avoids eagerly materializing every payload in a repository result.

Disadvantages

• Changes the materialization semantics of OutboxRecord.
• Requires coordinated changes across JDBC, JPA, and MongoDB.
• Introduces lifecycle, caching, cursor, transaction, and thread-safety
  considerations.
• Does not by itself decide retry accounting or persisted compatibility state.

This option may therefore be complementary to the other options rather than a replacement for them.

Proposed Direction

For version 1.10.0, adopt Option A together with the documented expand-and-contract deployment contract.

The reasons are:

• compatibility failures happen before delivery;
• they should not consume delivery retries;
• the current instance cannot classify them as temporary or permanent;
• schema changes and a second retry lifecycle would add substantial complexity;
• a compatible process can handle the unchanged record after the rollout;
• the approach can be introduced without changing the database model.

Accept the following limitations:

• compatibility exclusions are local to an application instance;
• persistent incompatibility requires an operational or configuration fix;
• eager materialization may temporarily block compatible predecessors;
• the database does not contain a dedicated compatibility status.

Lazy or metadata-first materialization should be evaluated separately because it addresses repository materialization and predecessor isolation rather than the core failure semantics.

Observability should be designed separately through the planned SPI so that this decision does not prematurely define that public API.

Questions for Review

1. Are compatibility and delivery failures separate failure categories?
2. Is it acceptable for compatibility failures to remain NEW without consuming
   failureCount?
3. Is instance-local exclusion sufficient under the documented deployment
   contract?
4. Is there a concrete requirement that justifies persisted compatibility state?
5. Does the current solution preserve the required ordering semantics?
6. Should lazy materialization be required for 1.10.0 or handled separately?
7. Are there mixed-version or backwards-compatibility risks not covered here?

Decision Process

• Decision owner: @rolandbeisel
• Review period ends: 2026-09-28
• Outcome: The accepted decision will be recorded in an ADR.
• Implementation: GH-464 Keep incompatible records pending during rolling deployments #470

[Alek96]

I think we should explore the options in more detail before making the decision, because each option has different implications for retry semantics, ordering, observability, and mixed-version deployments.

Common prerequisites

Regardless of the selected option, the implementation should introduce explicit exceptions for:

• unavailable payload types;
• unavailable handler IDs;
• payload deserialization failures;
• context deserialization failures.

These failures should be explicitly separated from exceptions thrown after a handler has been invoked.

This gives us two distinct categories:

• Compatibility failure: Delivery could not begin because the current instance cannot materialize the record or locate its handler.
• Delivery failure: A registered handler was invoked and threw an exception.

This distinction should exist independently of how compatibility state is retained or retried.

I would also prefer to avoid database schema changes such as adding columns or tables. The existing status column stores textual values, so adding another record status does not require a database migration. It is still an application-model change that needs defined upgrade behavior.

Option D: Lazy or metadata-first record materialization

Repositories currently materialize every payload before returning records to the scheduler. One incompatible record can therefore prevent earlier compatible records from being processed:

Record key: order-123

R1: compatible
R2: compatible
R3: incompatible
R4: compatible

Eager materialization:
[R1, R2, R3 -> error, R4]

Result: The scheduler receives no records, so R1 and R2 are not processed.

With metadata-first materialization, payloads are resolved individually during processing:

R1 metadata -> resolve payload -> success
R2 metadata -> resolve payload -> success
R3 metadata -> resolution fails
R4 metadata -> stop or continue according to processing.stopOnFirstFailure

This allows the scheduler to identify the exact incompatible record and process compatible predecessors:

• With processing.stopOnFirstFailure = true, R3 blocks R4.
• With processing.stopOnFirstFailure = false, R3 can be skipped and R4 processed.

The disadvantage is that this changes the materialization semantics of OutboxRecord. The implementation must define when payloads are resolved, whether they are cached, how errors retain record metadata, and how behavior remains consistent across JDBC, JPA, and MongoDB.

Option A: Instance-local compatibility exclusions

In simple terms, Option A works as follows:

1. An instance selects a record.
2. It discovers that the payload type or handler is unavailable.
3. The persisted record remains unchanged as NEW.
4. The instance remembers the unavailable payload type, handler ID, or record key in memory.
5. Future repository queries made by that instance exclude affected keys.
6. Another instance does not inherit these exclusions.
7. When a compatible instance eventually owns the partition, it can select and process the unchanged record.

Old instance owns partition 17
    |
    +-- reads R1
    +-- handler "orders-v2" is unavailable
    +-- leaves R1 as NEW
    +-- locally excludes handler "orders-v2"

Partition 17 moves to a new instance
    |
    +-- new instance has no inherited exclusion
    +-- handler "orders-v2" is registered
    +-- R1 is processed

Advantages

• Does not require a database schema or persisted-model change.
• Does not consume delivery retries.
• Leaves the serialized payload, handler ID, and delivery retry state unchanged.
• Allows a compatible replacement instance to process the original record.
• Does not introduce another record lifecycle.
• Naturally reevaluates compatibility after an instance restart or ownership transfer.
• Can provide equivalent behavior for JDBC, JPA, and MongoDB.
• Has low mixed-version risk because it does not introduce new persisted values or change the meaning of existing fields.

Disadvantages

• Compatibility state is not visible in the database.
• A permanently incompatible record remains NEW indefinitely.
• Operators cannot distinguish an ordinary pending record from an incompatible one using persisted state.
• Every incompatible instance must independently encounter and learn the exclusion.
• Compatibility is not reevaluated while the exclusion remains in the same process.
• Repository selection queries become more complicated.
• Custom repository implementations may not automatically honor the exclusions.
• Exclusion storage must be either unbounded or subject to eviction.
• Bounded eviction can introduce liveness problems.

The current PR limits exclusions to a fixed number of entries. With more incompatible entries than the capacity and a small processing batch, the scheduler could repeatedly rediscover evicted exclusions and fail to reach otherwise compatible keys:

Learn exclusion 257 -> evict exclusion 1
Next cycle selects item 1 again
Relearn exclusion 1 -> evict exclusion 2
Next cycle selects item 2 again
...

This can result in starvation rather than only additional queries or log entries.

Option A with Option D

Combining Option A with metadata-first materialization would allow compatible predecessors to be processed before an incompatible record is encountered.

It would also allow compatibility handling to respect processing.stopOnFirstFailure:

• With strict ordering, processing stops at the incompatible record.
• Without strict ordering, the incompatible record can be skipped locally and later records can continue.

However, this combination may introduce efficiency concerns. The scheduler could repeatedly fetch metadata for many locally excluded records while processing only a small number of compatible records.

Avoiding that would require careful repository query, paging, and exclusion behavior. Therefore, I am not yet sure whether the additional complexity of combining Option A with lazy materialization is justified.

Option B: Persist a dedicated compatibility state

Option B introduces a dedicated INCOMPATIBLE record status.

When an instance cannot load the payload type, deserialize the payload or context, or find the registered handler, it marks the exact record as INCOMPATIBLE. The delivery failureCount remains unchanged because the handler was not invoked.

Normal processing queries omit INCOMPATIBLE records. When another instance takes ownership of the partition, a healing process reevaluates the incompatible records and makes eligible records available for processing again.

Old instance owns partition 17

R1: NEW
    |
    +-- payload type or handler is unavailable
    |
R1: INCOMPATIBLE

New instance takes ownership of partition 17
    |
    +-- healing process reevaluates R1
    |
R1: NEW
    |
    +-- normal scheduler processes R1
    |
R1: COMPLETED

This option requires metadata-first materialization. The scheduler must receive enough record metadata to identify and update the problematic record even when its payload cannot be deserialized. With eager materialization, deserialization can fail before the scheduler receives any records.

An INCOMPATIBLE record must remain an incomplete predecessor for record-key ordering. When processing.stopOnFirstFailure is enabled, later records for the same key must not pass it.

The existing status column stores textual values, so adding INCOMPATIBLE does not require a database schema migration. It introduces moderate mixed-version risk because older instances do not recognize the new application-level status. However, they normally select only NEW records and therefore should not process INCOMPATIBLE records.

Healing strategies

A healing process is required to make INCOMPATIBLE records available when a compatible instance takes ownership of their partition. There are three possible strategies.

1. Reactivate all incompatible records

When an instance acquires a partition, it changes all INCOMPATIBLE records in that partition back to NEW using one bulk operation:

UPDATE outbox_record
SET status = 'NEW'
WHERE status = 'INCOMPATIBLE'
  AND partition IN (...)

The normal scheduler then reevaluates each record using its regular batching, ordering, transaction, and concurrency controls.

If a record is still incompatible, it is marked INCOMPATIBLE again. If compatibility is now available, normal delivery begins.

2. Fully evaluate every incompatible record

When an instance acquires a partition, the healing process loads each INCOMPATIBLE record and checks:

• whether its payload type can be loaded;
• whether its context can be deserialized;
• whether its payload can be deserialized;
• whether its handler ID is registered.

Only records that pass all checks are changed back to NEW.

This strategy requires reading and deserializing every incompatible record during healing. The normal scheduler subsequently processes the reactivated records.

3. Perform metadata preflight

When an instance acquires a partition, the healing process retrieves the distinct payload type names and handler IDs used by INCOMPATIBLE records in those partitions.

The instance then determines:

• which payload types remain unavailable;
• which handler IDs remain unregistered.

Records whose payload type and handler are both available are changed back to NEW. Records with a still-unavailable type or handler remain INCOMPATIBLE.

Conceptually, the update is:

UPDATE outbox_record
SET status = 'NEW'
WHERE status = 'INCOMPATIBLE'
  AND partition IN (...)
  AND handler_id NOT IN (:unavailableHandlerIds)
  AND record_type NOT IN (:unavailablePayloadTypes)

The unavailable predicates would only be added when their corresponding sets are non-empty.

This is a metadata preflight rather than complete compatibility validation. Payload and context deserialization remain record-specific and are performed by the normal scheduler.

A record with an available class and handler but incompatible serialized data may therefore transition to NEW and return to INCOMPATIBLE if deserialization still fails.

The implementation must account for database parameter limits when the unavailable sets are large and provide equivalent behavior for JDBC, JPA, and MongoDB.

Advantages

• Compatibility failures are visible in persisted record state.
• Delivery retries remain reserved for actual handler invocations.
• Normal processing queries can efficiently omit incompatible records.
• A permanently incompatible record remains operationally visible.
• The exact problematic record can be marked instead of excluding only its complete record key.
• Compatibility can be reevaluated when ownership moves to another instance.
• Healing can use partition ownership acquisition as the trigger for reevaluation.
• The existing failureReason could retain diagnostic information without requiring another column.

Disadvantages

• Introduces a compatibility healing lifecycle alongside delivery retries.
• Requires a healing process and clearly defined ownership semantics.
• Requires metadata-first materialization to handle payload failures at record level.
• Every persistence implementation must understand the INCOMPATIBLE status.
• Ordering and repository queries must define how INCOMPATIBLE records behave.
• Upgrade behavior must be defined for older application versions.
• Partition acquisition becomes more complex because healing should occur before normal processing begins.
• Persistently incompatible records still require application or data correction.
• The implementation must handle races where ownership changes during healing.
• This option introduces more changes to the processing and persistence model than Option A.

Option C: Reuse the delivery failure state

Option C reuses the existing failureCount, failureReason, retry timestamp, and FAILED status instead of introducing a dedicated compatibility state.

There are several possible variants.

Option C.1: Treat compatibility like an ordinary delivery failure

This is the simplest variant and is close to the current missing-handler behavior:

1. Increment failureCount.
2. Schedule another attempt using the retry policy.
3. Eventually mark the record FAILED.

Advantages

• Requires relatively few implementation changes.
• Reuses the existing retry and permanent-failure lifecycle.
• Provides persisted visibility through existing fields.
• Permanently incompatible records eventually stop being selected.

Disadvantages

• A record can become FAILED without its handler ever being invoked.
• Compatibility attempts consume the handler's delivery retry budget.
• A missing handler cannot provide its handler-specific retry policy, so the default policy must be used.
• The same incompatible instance may exhaust all retries before ownership moves to a compatible instance.
• A later compatible instance will not automatically process a record already marked FAILED.
• Compatibility and delivery failures remain semantically indistinguishable.

Option C.2: Use negative failureCount values

Negative values could represent compatibility attempts, while positive values continue to represent delivery failures:

 0 = no failures
-1 = first compatibility failure
-2 = second compatibility failure
 1 = first delivery failure
 2 = second delivery failure

Advantages

• Does not require another column or status.
• Makes compatibility attempts distinguishable from delivery failures.
• Allows existing retry timestamps to be reused.
• Provides some persisted compatibility information.

Disadvantages

• Changes the established meaning of failureCount.
• Existing code, metrics, queries, and external consumers may assume that the value is non-negative.
• Calculations such as attempt = failureCount + 1 become invalid.
• Older instances may misinterpret negative values.
• Transitions between compatibility failures and delivery failures become difficult to define.
• The same incompatible instance can still exhaust the compatibility budget before ownership changes.
• A finite number of attempts does not demonstrate that compatibility is permanently unavailable.

Option C.3: Infer compatibility from failureReason

In this variant, failureCount remains a normal positive counter, while the implementation parses or recognizes failureReason to determine whether the latest failure was a compatibility failure.

Advantages

• Does not require another status or database field.
• Reuses existing persisted diagnostic information.
• Allows the existing retry scheduling mechanism to be reused.

Disadvantages

• Parsing human-readable error messages is fragile.
• Exception messages can change between versions.
• A handler could throw an exception with the same or a similar message.
• Exception wrapping or database truncation can break classification.
• Older instances may write reasons using different formats.
• The same incompatible instance can still exhaust retries before ownership changes.
• failureCount continues to mix compatibility checks and actual deliveries.

Comparison

Concern	Option A: local exclusions	Option B: INCOMPATIBLE status	Option C: reuse failure state
Consumes delivery retries	No	No	Yes, or changes counter semantics
Database DDL required	No	No, uses the existing textual status column	No
Persisted visibility	No	Strong	Partial
Recovery after ownership change	Automatic through local state reset	Requires healing	Not reliable after FAILED
Permanent incompatibility visible	Only indirectly	Yes	Yes after FAILED
Additional lifecycle	No	Yes, healing lifecycle	Variant-dependent, compatibility retry for some variants
Mixed-version application risk	Low	Medium	Medium
Implementation complexity	Medium	Medium	Medium
Risk of premature permanent failure	No	No	High

Current view

My current conclusions are:

• Explicit compatibility exceptions should be introduced regardless of the selected option.
• Compatibility failures should not consume delivery retries because no delivery was attempted.
• Option C should be rejected because it either consumes delivery retries or overloads existing fields with incompatible semantics.
• The main decision is therefore between Option A and Option B.

Option A is simpler and avoids changing the persisted record lifecycle, but it provides weak observability and requires careful handling of exclusion capacity, query behavior, and starvation.

Option B provides explicit persisted state and more direct record-level behavior, but it introduces a healing lifecycle, requires metadata-first materialization.

If the priority for 1.10.0 is the smallest behavioral correction under a strict expand-and-contract deployment contract, Option A may be the more practical starting point.

If persistent operational visibility and explicit management of permanently incompatible records are requirements, Option B provides a clearer model, but it should be treated as a larger architectural change.

[rolandbeisel]

I agree with the main conclusions: compatibility failures should be distinguished from handler delivery failures, they should not consume delivery retries, and Option C should be rejected. I also agree that the main decision is now between the smaller instance-local approach in Option A and the persisted lifecycle in Option B.

You identified a real weakness in Option A that we need to address explicitly. With bounded exclusions, more incompatible identifiers than the configured capacity and a sufficiently small batch can cause evicted entries to be selected repeatedly, potentially starving compatible keys. The current documentation only mentions additional queries and warnings, which is not a complete description of that risk.

Before making the final decision, I think we should clarify a few semantic points:

1. I would distinguish unavailable application support from arbitrary deserialization failures. A missing payload class or handler is clearly instance compatibility. A deserialization failure may instead represent permanently corrupted data. Both happen before delivery, but they do not necessarily need the same retention and exclusion strategy. The current PR handles both, so we should decide explicitly whether that broader scope belongs in Handle temporarily unavailable payload types and handlers during rolling deployments #464.
2. The current PR deliberately treats compatibility failures as a record-key barrier even when stopOnFirstFailure is false. Your proposed behavior would be consistent with the existing non-strict processing mode, but it requires record-level materialization and skipping. We should record this as an explicit semantic decision rather than treating either behavior as automatic.
3. I agree that INCOMPATIBLE does not require a physical schema migration because the status is stored as text. It is still a persisted protocol and public-model change.
4. I do not think Option B strictly requires lazy materialization. Since the mapper exception can retain the record ID and metadata, the scheduler could mark that record and process compatible predecessors during a later poll. Lazy materialization would allow this within the same pass, but it is one implementation strategy rather than a prerequisite.

My current preference remains Option A for 1.10.0, provided we resolve or explicitly constrain the bounded-exclusion liveness problem. Option B and metadata-first materialization remain reasonable follow-up directions if persistent compatibility visibility or same-pass predecessor processing become concrete requirements.

[rolandbeisel]

One additional point regarding proportionality: while the bounded-eviction scenario is technically valid, I do not think it should determine the architecture for 1.10.0.

The compatibility handling is a safety mechanism for situations in which a deployment temporarily violates the documented expand-and-contract contract. It is intended to preserve the record, avoid consuming delivery retries, and allow a compatible instance to recover processing. It is not intended to guarantee automatic progress for an unbounded number of incompatible or corrupted records.

The bounded collections deliberately protect the application and repository queries from unbounded memory use and parameter growth. If an instance encounters more than 256 distinct incompatibilities, this indicates a broader deployment, configuration, or data problem that requires operational intervention.

We should document that compatible-key progress cannot be guaranteed beyond this boundary, but I do not think this justifies introducing a persisted compatibility lifecycle and healing process.

[Alek96]

Thanks for pointing out that Option B does not strictly require lazy materialization. I had developed a bit of tunnel vision around that solution.

However, marking the record as INCOMPATIBLE from metadata carried by the mapper exception is not sufficient on its own. Consider this sequence:

R1: NEW
R2: NEW
R3: INCOMPATIBLE
R4: NEW

With the current findIncompleteRecordsByRecordKey behavior, the next poll could return R1, R2, and R4 while omitting R3 because it is no longer NEW. The scheduler could then process R4 past R3, breaking strict ordering.

One solution would be to make findIncompleteRecordsByRecordKey barrier-aware when ignoreRecordKeysWithPreviousFailure is enabled. It could return only NEW records preceding the first INCOMPATIBLE record. This could be implemented through a more complex repository query or by fetching record metadata and truncating the result at the first barrier.

Therefore, Option B does not strictly require lazy materialization, but it does require an ordering-aware change. Lazy or metadata-first materialization may still be the cleaner solution because the scheduler would naturally encounter the INCOMPATIBLE record as a barrier.

Nevertheless, I agree that we should keep version 1.10.0 narrowly scoped and address rolling-deployment compatibility with Option A. We can revisit Option B, persisted compatibility visibility, and materialization semantics separately afterward.

[rolandbeisel]

After reflecting further on the alternatives, I think we may be solving a much larger problem than #464 actually requires.

Producer and Consumer Responsibilities

Namastack Outbox contains both sides of an asynchronous contract:

• Scheduling an outbox record is the producing side.
• Polling, deserializing, routing, and invoking a handler is the consuming side.

This remains true even though both sides are implemented by the same library and often run inside the same application. During a rolling deployment, producers and consumers from different application versions coexist while sharing the same persisted records.

The same compatibility rules therefore apply as in other event-driven systems. Before a producer writes a new message contract, every consumer that may receive that message must be able to understand it. For Namastack Outbox, that contract includes:

• the persisted payload type;
• the serialized payload shape;
• the handler ID;
• the context format;
• the serializer configuration.

This is why expand-and-contract is a required deployment contract:

1. Deploy the new reading and handling capability to every processing instance.
2. Wait until the rollout is complete.
3. Enable production of the new record type.

If an instance produces a record before every possible consumer can understand it, the deployment has violated that contract. The library should fail safely in this situation, but it does not need to preserve normal throughput or provide a complete secondary lifecycle for an invalid deployment.

I think we sometimes lose sight of this producer/consumer relationship when discussing the library internally. We then start treating compatibility failures as something the library must fully manage, even though the primary solution is the same as in any other event-driven architecture: consumers must become compatible before producers use the new contract.

A Simpler Compatibility Safeguard

The essential defect in the current behavior is narrow:

• An unavailable payload class already fails before handler execution and does not consume the delivery retry budget.
• An unavailable handler currently enters the delivery failure chain and can exhaust retries even though no handler was invoked.

I propose addressing exactly this problem:

1. Introduce explicit exceptions for unavailable payload types and handler IDs.
2. Verify handler availability before entering normal handler failure processing.
3. Catch compatibility exceptions at the per-record-key scheduler boundary.
4. Stop processing that record key for the current attempt.
5. Leave the record completely unchanged as NEW.
6. Do not increment failureCount or modify delivery retry state.
7. Emit an actionable warning.

Later records for the affected key would not be processed during that attempt. Other record keys already selected in the same batch could complete normally.

Instance-Wide Compatibility Cooldown

Without additional handling, an incompatible record could be selected and logged during every polling cycle. With a two-second polling interval, that could produce a database query and warning every two seconds until the instance is replaced.

We can avoid this without maintaining payload-type, handler-ID, or record-key exclusion sets. When the scheduler encounters a compatibility failure, it can set a single instance-local timestamp:

compatibilityBackoffUntil = now + 30 seconds

The current batch is allowed to finish, but that scheduler instance does not query for another batch during the cooldown. After the cooldown, it probes again:

• If the incompatible instance is still running, it logs another warning and enters another cooldown.
• If the deployment has replaced it with a compatible instance, the new instance starts without the local cooldown and processes the unchanged record.
• If the problem is permanent, processing remains visibly degraded until an operator corrects the deployment or data.

This uses one local timestamp rather than collections of incompatible identifiers. It does not introduce persisted retry state, another record lifecycle, repository query changes, or database schema changes.

Consequences

This deliberately prioritizes safety over availability after a deployment contract violation:

• The incompatible record is preserved.
• No delivery retry is consumed.
• No record becomes permanently FAILED before its handler has run.
• The instance may temporarily stop processing otherwise compatible keys assigned to it.
• A large backlog may develop while an incompatible deployment remains active.
• Normal processing resumes after compatible instances replace the incompatible ones.

I consider the temporary loss of throughput acceptable here. Applications that require uninterrupted processing of all compatible keys during a rollout already have a supported solution: follow the documented expand-and-contract sequence.

Scope

I would keep arbitrary payload and context deserialization failures outside this change. They may indicate version incompatibility, corrupted data, or another serialization problem. They already occur before handler execution and therefore do not need to consume handler delivery retries, but poison-record management deserves a separate decision if it becomes a concrete requirement.

With this approach, we could remove the following parts of the current proposal:

• OutboxCompatibilityExclusions;
• all bounded exclusion sets;
• the additional repository method;
• the JDBC, JPA, and MongoDB exclusion-query variants;
• the custom-repository compatibility limitation;
• eviction and exclusion-capacity behavior.

The result would be a much smaller change focused on the actual correctness issue: a missing handler must not consume a delivery retry when delivery never started.

I therefore propose this minimal compatibility handling with a scheduler-wide local cooldown as the direction for 1.10.0. Persisted incompatibility states, healing, metadata-first materialization, and poison-record handling can remain separate proposals if concrete requirements emerge.

[Alek96]

I like this direction and think we should proceed with it as part of 1.10.0. It addresses the actual correctness issue with a smaller and more predictable change:

• missing payload types and handlers are treated consistently;
• compatibility failures do not consume delivery retries;
• affected records remain unchanged and recoverable;
• bounded exclusion collections and the associated starvation problem are removed;
• repository queries and custom repository contracts do not need to change.

The main tradeoff is the larger blast radius. A single incompatible record pauses polling for all partitions owned by that scheduler instance, so otherwise compatible keys may also be delayed. Given that this mechanism handles a violation of the documented expand-and-contract deployment contract, I think prioritizing safety and simplicity over throughput is reasonable.

I would clarify a few implementation details:

• Should only unavailable payload types and handler IDs trigger the cooldown, while arbitrary payload or context deserialization failures remain outside the scope of this change?
• Does the current batch finish normally, with only the affected record key stopping immediately?
• Should the cooldown duration be configurable?
• Should repeated compatibility failures emit both an actionable warning and an observability event or metric?

[rolandbeisel]

@Alek96 I created an ADR in #476 and also answered the open questions.

[rolandbeisel]

You can find the finalized ADR here.