How should the outbox lifecycle be exposed through an instrumentation SPI?

[rolandbeisel]

Context

Namastack Outbox performs work on both sides of the transactional outbox boundary:

• an application schedules and persists one or more records;
• a scheduler later loads those records and processes them asynchronously.

Observability needs to cover more than the eventual handler invocation. Processing a persisted record can involve:

• selecting a record key;
• loading persisted records;
• resolving and deserializing the payload and stored context;
• resolving the registered handler;
• invoking the primary handler;
• recording a delivery failure;
• evaluating and scheduling a retry;
• resolving and invoking a fallback handler;
• completing or deleting the record;
• marking the record as permanently failed.

Some failures occur before a handler can be invoked. During a mixed-version deployment, an instance may encounter a persisted payload type or handler ID that it does not know. A payload type may also be available while the stored payload or context can no longer be deserialized.

These cases differ from a delivery failure, because no registered handler has necessarily received the record.

Discussion 473 defines the compatibility problem for unavailable payload types and handlers. Issue 467 shows a related observability gap: processing-chain logs can execute outside the record's trace scope and therefore lose correlation with the originating request.

Before defining a public instrumentation SPI, we need to agree on the lifecycle, failure categories, and outcomes that it must represent.

Decision to Make

We need to decide which Namastack Outbox operations and processing semantics belong to the instrumentation contract.

In particular:

• Which operations should the SPI expose?
• Where does processing of one persisted record begin and end?
• Is record loading and materialization part of the processing operation?
• Should polling, partition coordination, and batch processing be covered by the same SPI?
• Should instrumentation expose one overall record-processing operation, individual stages, or both?
• Which intermediate decisions and final outcomes must be observable?
• How should payload and context deserialization failures be classified and handled?
• How should instrumentation work when a complete OutboxRecord cannot be created?
• How should stored tracing and correlation context be restored and scoped?
• Should instrumentation only observe Core behavior, or may it influence processing decisions?
• How should multiple instrumentation implementations interact?

Terminology

Scheduling operation

An application request to create and persist one or more outbox records for a payload.

Polling cycle

One scheduler execution that identifies work assigned to the current instance and submits record keys for processing.

Whether polling cycles belong to the instrumentation SPI is part of this decision.

Record-processing attempt

An attempt by a scheduler instance to process one persisted outbox record.

The exact beginning and end of an attempt are part of this decision.

Processing stage

A distinct part of record processing, such as materialization, handler resolution, primary delivery, retry evaluation, fallback handling, or completion.

Whether stages become part of the public SPI is part of this decision.

Handler invocation

One invocation of a registered primary or fallback handler.

Delivery failure

A registered handler was invoked and threw an exception.

Compatibility failure

Handler execution could not begin because the processing instance does not have the persisted payload type or registered handler ID.

Deserialization failure

The persisted type is available, but the stored payload or context cannot be deserialized.

Whether this is a compatibility failure, infrastructure failure, invalid-record failure, or a separate category is part of this decision.

Processing outcome

The state or decision reached by a processing attempt, such as completion, retry, compatibility deferral, or permanent failure.

Required Semantics

The selected design should address the following:

• Core exposes instrumentation through an explicit, framework-neutral contract.
• Instrumentation of Core operations does not depend on Spring AOP.
• Applications can use the Core instrumentation contract without the observability module.
• The observability module can implement the contract using Micrometer without introducing Micrometer dependencies into Core.
• Instrumentation can observe failures that are handled internally and do not escape the processing flow.
• The eventual outcome of a processing attempt is observable.
• Primary and fallback handler invocations and failures can be distinguished.
• Retry scheduling, retry exhaustion, fallback handling, completion, and permanent failure can be represented.
• Unavailable payload types and handlers can be represented without reporting a handler invocation that never happened.
• Payload and context deserialization failures have explicitly defined processing and observability semantics.
• Failures before record materialization expose the persisted metadata that is available at that point.
• Record-key ordering is preserved for every failure category.
• Library logs associated with a record can retain its tracing or correlation context when that context is available.
• JDBC, JPA, MongoDB, and custom repository implementations can provide consistent behavior.
• Instrumentation does not require payload values or other sensitive data to be exposed.
• Metric dimensions remain bounded; record IDs, record keys, payload values, and exception messages are not implicitly used as low-cardinality dimensions.
• Instrumentation adds minimal overhead when no implementation is configured.
• Failures raised by instrumentation itself have defined behavior.
• Mixed-version deployments and existing observability integrations have a documented compatibility or migration path.

Processing Surface to Consider

The decision should explicitly include or exclude each of the following boundaries:

Boundary	Work performed
Scheduling	Discover handlers, create records, and persist them
Polling	Determine assigned partitions and select record keys
Batch or key processing	Submit and coordinate work for one record key
Record loading	Load persisted records for a selected key
Materialization	Resolve the payload type and deserialize payload and context
Readiness check	Determine whether the record is ready for another attempt
Handler resolution	Resolve the persisted handler ID on the current instance
Primary delivery	Invoke the primary handler
Failure recording	Record a primary delivery failure and update attempt state
Retry decision	Evaluate retry policy and retry exhaustion
Retry scheduling	Persist the next retry time
Fallback resolution	Determine whether a fallback handler exists
Fallback delivery	Invoke the fallback handler
Completion	Mark the record completed or delete it
Permanent failure	Persist the terminal failed state
Compatibility deferral	Retain a record that the current instance cannot process
Infrastructure failure	Handle repository, executor, scheduler, or persistence errors

Failure Scenarios to Cover

The decision should define the processing and instrumentation behavior for:

1. Scheduling succeeds and all records are persisted.
2. Scheduling fails during handler discovery, serialization, or persistence.
3. A record is loaded, its primary handler succeeds, and it is completed or deleted.
4. A primary handler fails and a retry is scheduled.
5. A primary handler fails with a non-retryable exception.
6. Retries are exhausted and a fallback handler succeeds.
7. A fallback handler fails.
8. No fallback handler exists.
9. A record is marked permanently failed.
10. The persisted payload type is unavailable.
11. The persisted handler ID is unavailable.
12. The payload type exists, but the stored payload cannot be deserialized.
13. The stored context cannot be deserialized.
14. Loading or materializing one record prevents other records for the same key from being returned.
15. Completion, retry persistence, or permanent-failure persistence fails.
16. The task executor rejects or fails submitted processing work.
17. An instrumentation implementation throws before or after invoking the underlying operation.

For each scenario, the decision should state:

• whether a processing or delivery attempt has occurred;
• whether the delivery retry budget is affected;
• which record state is retained or persisted;
• whether later records for the same key may proceed;
• whether and when the operation is attempted again;
• which metadata and failure information instrumentation receives;
• which tracing or correlation context is active;
• which outcome is reported.

Questions for Review

1. Which top-level operations should the instrumentation SPI expose?
2. Where should a record-processing attempt begin and end?
3. Should repository loading and materialization be part of that attempt?
4. Should polling, partition coordination, and batch processing use the same SPI or a separate contract?
5. Do we need one overall record-processing operation, stage-level operations, or both?
6. Which processing stages are stable enough to become part of a public contract?
7. Which processing outcomes must be represented explicitly?
8. How should malformed payload and context data affect record state, retries, and record-key ordering?
9. Is a deserialization failure a compatibility failure or a separate failure category?
10. What record metadata is required when materialization fails before an OutboxRecord exists?
11. Can stored tracing context be restored when payload deserialization fails? What happens when context deserialization itself fails?
12. Should primary and fallback handler invocations be separate operations within an overall processing attempt?
13. Should the SPI describe stable lifecycle concepts independently of the current processor implementation?
14. Should instrumentation be strictly observational, with all retry, fallback, compatibility, and persistence decisions owned by Core?
15. If multiple instrumentation implementations are registered, how are they ordered and isolated from one another?
16. How should instrumentation failures affect the underlying outbox operation?
17. Which repository, scheduler, executor, and persistence errors belong to record processing?
18. How should existing metrics, traces, observation conventions, and dashboards evolve?
19. Are there custom repository or mixed-version scenarios not covered here?

Decision Process

• Decision owner: @rolandbeisel
• Review period ends: TBD
• Outcome: The accepted lifecycle, failure semantics, and instrumentation contract will be recorded in an ADR.
• Related work: #449, #464, #467, #470, and discussion 473.

[rolandbeisel]

After walking through the relevant processing and failure scenarios, I would propose the following direction.

The SPI should expose three stable lifecycle concepts:

• one scheduling operation per schedule(...) call;
• one record-processing attempt covering handler resolution, handler execution, retry or fallback decisions, and the final database update;
• individual primary and fallback handler invocations as children of that attempt.

Polling, partition coordination, batching, and record-key scheduling should stay outside this record-level SPI. We also do not need callbacks for every internal processor. The final attempt outcome gives us the useful lifecycle information without exposing the current processing-chain implementation as public API.

The resulting trace structure would look like this:

Application / producer span
└── outbox.record.schedule
    ├── outbox.record.attempt
    │   deliveryAttempt = 1
    │   outcome = RETRY_SCHEDULED
    │   └── outbox.record.process
    │       handler.kind = primary
    │       status = error
    │
    ├── outbox.record.attempt
    │   deliveryAttempt = 2
    │   outcome = COMPLETED
    │   ├── outbox.record.process
    │   │   handler.kind = primary
    │   │   status = error
    │   └── outbox.record.process
    │       handler.kind = fallback
    │       status = success
    │
    └── outbox.record.attempt
        outcome = COMPATIBILITY_DEFERRED
        failure.category = PAYLOAD_DESERIALIZATION_FAILED
        └── no handler observation

Each retry is a new sibling attempt under the stored scheduling context. We should never keep a span open while a record waits for its next retry.

The attempt should report one of five final outcomes:

• COMPLETED: the record was completed or deleted, including completion through a fallback handler;
• RETRY_SCHEDULED: the record remains NEW with a future retry;
• FAILED: the record reached the terminal FAILED state;
• COMPATIBILITY_DEFERRED: this instance cannot currently process the record, so it remains NEW without consuming the retry budget;
• ERROR: an unexpected technical error means the final database state may be uncertain.

Unavailable payload types, payload deserialization failures, and unavailable handlers should be compatibility failures. Only handlers that are actually invoked should create an outbox.record.process observation. If the stored context is readable, a materialization failure can create a short outbox.record.attempt observation and restore the original trace. If the context itself cannot be deserialized, Core should log the problem but no instrumentation callback should be created. A readable context without trace headers may start a new root observation so that metrics are still recorded.

For compatibility, the existing outbox.record.schedule and outbox.record.process observations, conventions, names, and tags should remain stable. outbox.record.attempt would be additive. It would keep the Core SPI framework-neutral and expose stable record metadata, the stored context, the final outcome, and an optional categorized failure, but never the payload value.

This seems to close the correlation gap from #467 while keeping the SPI small. Does this look like the right lifecycle model before we turn it into a concrete API proposal?

[Alek96]

I did not notice your comment before posting mine, but I now see that we have the same idea in mind.

You also proposed the outbox.record.attempt name and defined its attributes and outcomes. I like this model. It keeps the SPI small while preserving the existing scheduling and handler observations.

I am not sure about this part:

If the stored context is readable, a materialization failure can create a short outbox.record.attempt observation and restore the original trace. If the context itself cannot be deserialized, Core should log the problem but no instrumentation callback should be created. A readable context without trace headers may start a new root observation so that metrics are still recorded.

I would prefer a simpler and more predictable boundary.

The outbox.record.attempt observation should be created when a complete record enters the record-processing chain. It should remain active until that processing attempt finishes.

Failures that happen before this boundary, including record loading or materialization failures, should remain part of the Processing Scheduler observation. We should log them there, but we should not conditionally create a short attempt observation or try to reconstruct a record trace from partially readable context.

This gives the observation one clear meaning: Core started processing a materialized record. It also avoids different tracing behavior depending on which parts of the stored data could be deserialized.

[rolandbeisel]

Good point. I overlooked the existing Processing Scheduler observation when describing the proposed boundary.

OutboxProcessingScheduler already runs inside Spring's scheduled-task observation. That observation covers the polling cycle and, because the scheduler waits for the submitted key tasks, the duration of their work. With executor context propagation enabled, loading and materialization logs use that scheduler trace. It remains a separate technical trace from the trace stored with an individual record.

Processing exceptions are caught and logged by Core, so they do not currently mark the scheduler observation as failed. The scheduler span provides technical timing and log correlation rather than a per-record outcome.

I agree that this gives us a simpler record boundary:

• outbox.record.attempt starts only after a complete record was materialized and is ready to enter the processing chain;
• loading and materialization failures remain on the Processing Scheduler side and are logged there; with executor context propagation enabled, those logs use the scheduler trace;
• we do not create a conditional short attempt from partially readable persisted data;
• once an attempt starts, it restores the persisted scheduling context and the handler observations become its children.

The two trace relationships are therefore separate:

Technical polling trace
└── Processing Scheduler span
    ├── processing task for key 1 (propagated context, no new span)
    ├── processing task for key 2 (propagated context, no new span)
    └── loading / materialization logs

Record trace
Application / producer span
└── outbox.record.schedule
    ├── outbox.record.attempt (attempt 1)
    │   └── outbox.record.process (primary)
    └── outbox.record.attempt (attempt 2)
        ├── outbox.record.process (primary)
        └── outbox.record.process (fallback)

This means payload-type and deserialization failures do not produce a record-attempt outcome. They happen before that lifecycle begins. COMPATIBILITY_DEFERRED still applies when a complete record enters the chain but its persisted handler is unavailable.

If we later need dedicated metrics for pre-attempt materialization failures, that should be a separate decision rather than changing the meaning of outbox.record.attempt.

[Alek96]

I think the SPI should preserve and extend the observability we already provide. The observability module should implement this framework-neutral SPI using Micrometer.

For this reason, I suggest that we first describe the current observations and trace relationships.

Processing Scheduler observation

OutboxProcessingScheduler is instrumented with Spring’s scheduled-task observation through ScheduledMethodRunnable.

A new observation is created for every polling-cycle invocation.

The polling cycle finds record keys and submits one processing task per key to a TaskExecutor. Each task loads the relevant records and processes them in order.

Spring does not propagate the observation context to executor threads by default. Applications must enable spring.task.execution.propagate-context or provide a ContextPropagatingTaskDecorator.

See the Spring Boot context propagation documentation.

When propagation is enabled, the Processing Scheduler context is available in the executor threads. This does not create a new span for each processing task.

Without propagation, the executor threads do not have an active trace context.

flowchart TB
    S["Processing Scheduler span"]

    S -. "context propagated<br/>no new span" .-> T1["Processing task for key 1"]
    S -. "context propagated<br/>no new span" .-> T2["Processing task for key 2"]
    S -. "context propagated<br/>no new span" .-> T3["Processing task for key 3"]

Loading

Processing exceptions are currently caught and logged internally. Therefore, they do not escape to ScheduledMethodRunnable and are not recorded as errors on the Processing Scheduler observation.

Scheduling and handler observations

From the user’s perspective, scheduling and later processing should belong to the same trace. This makes it easier to connect logs and spans from the original request to the asynchronous handler execution.

We already provide part of this relationship.

When a record is scheduled:

1. We create a scheduling observation.
2. We capture its trace context.
3. We persist that context with the record.

When a handler is invoked:

1. We extract the persisted trace context.
2. We create a handler observation using that context.
3. We invoke the handler inside the observation.
4. We close the observation when the handler invocation finishes.

Fallback handlers use the same mechanism.

flowchart TB
    R["Incoming request span"] --> S["Scheduling span"]

    S -. "persisted trace context" .-> H["Primary handler span"]
    S -. "persisted trace context" .-> F["Fallback handler span"]

Loading

Current problem

The persisted trace context is restored only around the primary or fallback handler invocation.

Logs produced outside those observations use the Processing Scheduler trace when executor context propagation is enabled. Without propagation, they have no active trace context.

They do not use the trace that originally scheduled the record. This is the problem described in issue #467.

Proposed record-processing observation

I think we should introduce an observation for a complete record-processing attempt.

When processing of one record begins, its persisted context should be restored and a record-processing observation should be created.

The primary and fallback handler observations should then be children of the record-processing observation.

flowchart TB
    R["Incoming request span"] --> S["Scheduling span"]

    S -. "persisted trace context" .-> P["Record-processing span"]

    P --> H["Primary handler span"]
    P --> F["Fallback handler span"]

Loading

The Processing Scheduler observation remains separate from the record trace.

The record-processing observation remains active while Core processes the record. Record-related logs therefore use the trace that originally scheduled the record.

Retry handling, completion, fallback coordination, and permanent-failure handling happen inside the record-processing observation. They do not create separate spans by default.

Each retry creates a separate record-processing observation:

[Incoming request span]
    └── [Scheduling span]
        │
        ├── [Record-processing span: attempt 1]
        │   └── [Primary handler span: failed]
        │
        ├── [Record-processing span: attempt 2]
        │   └── [Primary handler span: failed]
        │
        └── [Record-processing span: final attempt]
            ├── [Primary handler span: failed]
            └── [Fallback handler span]

This would keep record-related logs correlated with the original request. It would also provide a place to report the final outcome of each processing attempt, such as completion, retry, fallback, or permanent failure.