Skip to content

Preserve OpenBao bootstrap state across Job replacement #6365

Open

Description

@devantler

🤖 Generated by the Agentic Engineer

Description

Fresh bootstrap creates recovery state before a later configuration container durably stores it. An interruption that replaces the Pod in that gap can leave an initialized service without the state needed to resume safely. Make that handoff restart-safe before tightening the configuration Job's retry or termination policy.

Steps to Reproduce

In a disposable fixture with synthetic state only:

  1. Run fresh initialization and stop before the persistence stage acknowledges success.
  2. Replace the configuration Pod, including its temporary volumes, while retaining the initialized service state.
  3. Retry configuration and verify whether it can recover without reinitializing the service or discarding its data.

Expected Behavior

A bounded configuration failure or Pod replacement preserves a verified recovery path. A failed or ambiguous persistence operation never counts as a durable handoff.

Actual Behavior and Evidence

Static inspection of the current configuration Job shows that fresh initialization writes the handoff into a temporary shared volume and a later init container persists it. The Job has a finite lifetime and is force-recreated for specification changes. The source therefore has an interruption window; no production loss is claimed, and no interruption was performed against production.

Acceptance Criteria

  • Define and implement a durable, acknowledged bootstrap-state handoff across the entire initialization-to-persistence interval, or an equally verified recovery mechanism.
  • RED/GREEN regression exercises the real configuration entrypoints with synthetic state and a fresh Pod/temporary-volume boundary.
  • Cover interruption before and after persistence, persistence failure and ambiguous acknowledgement, container retry, Job deadline, and specification-driven replacement.
  • Retry never reinitializes an existing service, overwrites retained recovery state, exposes sensitive values, or treats missing state as success.
  • Preserve snapshot recovery and existing initialization guards, with the least required privileges.
  • Validate the changed resources in both environments and obtain current-head review. Record rollout and recovery-path acceptance separately from source merge.

Additional Context

Part of #4979. Related: #6360 and its partial delivery #6362. The process-level wait bound does not close this persistence gap. Do not adopt a replacement or terminal-failure policy merely because a process timeout is finite.

Audience: platform operators and deployments depending on the secrets service. Risk reduction: preserve recoverability while making failed deployments terminate predictably. Estimated effort: Medium, including behavioral interruption tests, review, and rollout verification. New intake is Backlog; the active production recovery retains its owner and scheduling fence.

Activity

  1. added theissue type on Nov 15, 2026
  2. added a commit that references this issue on Nov 15, 2026
  3. added 2 commits that reference this issue on Nov 15, 2026
  4. moved this to 📥 Backlog in 🌊 Project Boardon Nov 15, 2026
  5. added a commit that references this issue on Nov 15, 2026
  6. self-assigned this
    on Nov 15, 2026
  7. added a commit that references this issue on Nov 15, 2026
  8. removed their assignment
    on Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Fields

    Priority

    None yet

    Effort

    None yet

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions