Skip to content

feat(blaze): prune checkpoint history - #2474

Open
WeissonHan wants to merge 1 commit into
alibaba:mainfrom
WeissonHan:feature/blaze/checkpoint-prune-draft-v1
Open

feat(blaze): prune checkpoint history#2474
WeissonHan wants to merge 1 commit into
alibaba:mainfrom
WeissonHan:feature/blaze/checkpoint-prune-draft-v1

Conversation

@WeissonHan

@WeissonHan WeissonHan commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

Why

Blaze can capture and list checkpoints, but it has no supported way to reclaim
the storage held by history branches that are no longer reachable from the
current checkpoint. Operators need a bounded deletion operation that preserves
every checkpoint still required by the active history or explicitly selected
for retention.

What changed

  • Add POST /v1/sandboxes/{id}/checkpoints/prune. The request body is optional;
    when present, it may contain only a protected array of committed checkpoint
    identifiers. An empty array protects only the current HEAD lineage, and
    duplicate identifiers have the same effect as one occurrence. Malformed JSON,
    non-object bodies, unknown fields, and invalid field types return HTTP 400
    before the operation is recorded. The response reports the removed count and
    identifiers.
  • Preserve the current checkpoint and its complete ancestor chain. Each
    explicitly protected checkpoint also preserves its complete ancestor chain.
    Malformed or uncommitted protected identifiers return HTTP 400 before any
    deletion.
  • Accept pruning only for a running sandbox with no unfinished operation.
    Other lifecycle states return HTTP 409.
  • Persist a prune operation record before changing the catalog. Each selected
    checkpoint is atomically renamed to a unique temporary directory before its
    complete version-2 payload tree is recursively removed.
  • Run deletion and lifecycle finalization in a detached supervisor that retains
    the per-sandbox operation lock. If the client disconnects, pruning finishes
    before a queued destroy or another sandbox operation can start.
  • Clear the operation record only after successful cleanup. Partial deletion or
    an unprovable rename result changes the sandbox to RecoveryRequired and
    retains the operation record. Destroy or startup reconciliation then removes
    the sandbox-owned runtime and complete checkpoint namespace; interrupted
    pruning is not resumed. HTTP 200 is returned only after every tombstone made
    by the request is removed and the checkpoint namespace is synchronized;
    another prune request in RecoveryRequired returns HTTP 409 without changing
    the catalog.
  • Document the route, request and response, retention rules, supported state,
    failure behavior, and rollback boundary in the English and Chinese Blaze
    README, runtime guide, and lifecycle design document.

No /v1/instances compatibility route is added.

Related issue

Refs #2469. This PR delivers the usable pruning endpoint and fail-closed
recovery boundary. The broader issue remains open because retrying or resuming
an interrupted prune is not implemented here.

User / Agent impact

Operators and agents can reclaim storage held by unreachable checkpoint
branches without destroying a running sandbox. The new route is user-visible;
the current and explicitly protected histories remain available after a
successful request.

Risk and compatibility

  • Public CLI, API, configuration, or documented behavior changed
  • Privileged or security-sensitive behavior changed
  • Cross-component contract changed
  • Migration or rollback guidance is needed

The change adds one destructive HTTP operation over daemon-owned checkpoint
directories. Existing checkpoint catalogs remain readable, and no new record
is written until prune is called. Directory identity checks, no-replace rename,
and sandbox-scoped descriptors prevent the operation from replacing or walking
outside the owned checkpoint tree.

The persisted lifecycle format gains the prune operation value. An older
binary must not be started while a prune operation or .prune.* temporary
directory remains.

Validation

Exact submitted head: 36b8d14f50fdaecd25fb60f6618cf8af59395cd1.

  • CI-equivalent commit message validation with commitlint 19.2.1 and
    .github/commitlint.config.json: passed with 0 errors and 0 warnings.
  • git diff --check upstream/main..HEAD: passed.
  • GitHub Linux CI for the exact submitted head, including formatting, lint, and
    the complete Blaze workspace test job: passed.
  • Codex reviewed the exact submitted head and reported no major issue.
  • Qoder reviewed the exact submitted head after its earlier findings were
    addressed and reported no issue requiring modification.
  • Focused all-feature prune coverage: 9 passed, 0 failed. This includes
    non-object and malformed request bodies, empty and repeated protected lists,
    recovering-state rejection, successful pruning, interruption, and client
    cancellation.
  • All-target/all-feature Clippy, English/Chinese documentation lint and link
    checks, and the all-feature warning-free Rust documentation build passed
    locally on the exact submitted head.

Focused coverage included in this PR exercises successful removal through the
HTTP route, preservation of current and protected history, request validation,
state rejection, nested version-2 payload removal, operation-record recovery,
interruption after the atomic rename, and client cancellation while deletion is
blocked. The cancellation test verifies that destroy waits for the detached
prune supervisor and that both operations converge in order.

Documentation and rollback

The English and Chinese Blaze README, runtime guide, and lifecycle design
document describe the public API and its failure behavior. The changelog is not
changed because repository policy records daily features in the release version
bump PR.

Before rolling back, stop Blaze and confirm that no sandbox has an active
prune operation and no checkpoint namespace contains a .prune.* temporary
directory. Then revert the feature commit and restart the previous binary.

@github-actions github-actions Bot added component:blaze src/blaze scope:documentation ./docs/|./*.md|./NOTICE labels Aug 13, 2026
@WeissonHan
WeissonHan force-pushed the feature/blaze/checkpoint-prune-draft-v1 branch from 5804993 to 88921b2 Compare August 17, 2026 16:03
@WeissonHan
WeissonHan marked this pull request as ready for review August 17, 2026 16:07
@WeissonHan
WeissonHan requested a review from casparant as a code owner August 17, 2026 16:07
@WeissonHan

Copy link
Copy Markdown
Collaborator Author

@codex review Checkpoint history pruning: adds POST /v1/sandboxes/{id}/checkpoints/prune that protects the current HEAD chain and any explicitly listed lineages, publishes named tombstones before renaming candidates, and lets restart cleanup finish an interrupted prune. Rebased on merged #2472.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 88921b2483

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/blaze/crates/blazed/src/sandbox/checkpoint.rs Outdated
@WeissonHan
WeissonHan force-pushed the feature/blaze/checkpoint-prune-draft-v1 branch from 88921b2 to 91a138e Compare August 17, 2026 16:21
@WeissonHan

Copy link
Copy Markdown
Collaborator Author

@codex review Prune with tightened BadRequest handling for uncommitted protected identifiers.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 91a138e20b

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/blaze/crates/blazed/src/sandbox/checkpoint.rs Outdated
@WeissonHan
WeissonHan force-pushed the feature/blaze/checkpoint-prune-draft-v1 branch from 91a138e to 06bd020 Compare August 18, 2026 01:08
@WeissonHan

Copy link
Copy Markdown
Collaborator Author

@codex review Prune now reports post-tombstone cleanup failures as PruneOutcome::PartialCleanup and the manager marks the sandbox RecoveryRequired.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 06bd020fdc

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/blaze/crates/blazed/src/checkpoint_store.rs Outdated
Comment thread src/blaze/crates/blazed/src/sandbox/checkpoint.rs Outdated
@WeissonHan
WeissonHan force-pushed the feature/blaze/checkpoint-prune-draft-v1 branch from 06bd020 to 9af5bdd Compare August 18, 2026 01:53
@WeissonHan

Copy link
Copy Markdown
Collaborator Author

@codex Review

@WeissonHan

Copy link
Copy Markdown
Collaborator Author

@codex review Prune now classifies every post-rename uncertainty as PruneOutcome::PartialCleanup and rejects prune on a RecoveryRequired sandbox with HTTP 409.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9af5bdd952

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/blaze/crates/blazed/src/checkpoint_store.rs Outdated
Comment thread src/blaze/crates/blazed/src/checkpoint_store.rs Outdated
@WeissonHan
WeissonHan force-pushed the feature/blaze/checkpoint-prune-draft-v1 branch from 9af5bdd to 9fcfb42 Compare August 18, 2026 02:26
@WeissonHan

Copy link
Copy Markdown
Collaborator Author

@codex review sweep_candidate now classifies every per-candidate outcome (Removed / Retained / RemovedWithRetainedScratch) so a partial success is never reported as a plain HTTP 500, and owner-publication failures clean up scratch or force RecoveryRequired.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Something went wrong. Try again later by commenting “@codex review”.

Unknown error
ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@WeissonHan

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9fcfb424fb

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/blaze/crates/blazed/src/checkpoint_store.rs Outdated
@WeissonHan
WeissonHan force-pushed the feature/blaze/checkpoint-prune-draft-v1 branch from 9fcfb42 to 0035d1d Compare August 24, 2026 05:56
@WeissonHan

Copy link
Copy Markdown
Collaborator Author

@codex review Adds POST /v1/sandboxes/{id}/checkpoints/prune to remove unreachable checkpoint branches while preserving current and explicitly protected histories, with durable recovery for partial deletion.

@WeissonHan

Copy link
Copy Markdown
Collaborator Author

@qoder review

@qoderai qoderai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] RecoveryRequired 状态下的 prune 调用需要显式拒绝;[P1] protected 列表的去重与文档约束可以减少误用与开销;[P2] tombstone 名称解析错误时的错误信息可以更具体,便于恢复与运维。


🤖 Generated by QoderView workflow run

Comment thread src/blaze/crates/blazed/src/sandbox/checkpoint.rs
Comment thread src/blaze/crates/blazed/src/sandbox/checkpoint.rs
Comment thread src/blaze/crates/blazed/src/checkpoint_store.rs

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0035d1de7e

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/blaze/crates/blazed/src/sandbox/checkpoint.rs Outdated
@WeissonHan
WeissonHan force-pushed the feature/blaze/checkpoint-prune-draft-v1 branch from 0035d1d to 18eb9c9 Compare August 24, 2026 06:33
@WeissonHan

Copy link
Copy Markdown
Collaborator Author

@qoder review

@WeissonHan

Copy link
Copy Markdown
Collaborator Author

@codex review Adds cancellation-safe checkpoint history pruning: the detached supervisor now owns deletion, lifecycle finalization, and the per-sandbox operation lock until completion.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Keep it up!

Reviewed commit: 18eb9c9f0c

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@WeissonHan

Copy link
Copy Markdown
Collaborator Author

@qoder review

@qoderai qoderai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] prune 请求体错误时的错误信息与文档略有偏差;[P2] 文档中对成功清理不留 tombstone 的语义可以更明确;[P2] RecoveryRequired 状态下复用 409 拒绝的行为可以在用户指南中补充说明。


🤖 Generated by QoderView workflow run

Comment thread docs/user-guide/en/runtime/blaze.md
Comment thread docs/user-guide/zh/runtime/blaze.md Outdated
@WeissonHan

WeissonHan commented Aug 24, 2026

Copy link
Copy Markdown
Collaborator Author

Update for current head 04edc25a: deeper regression coverage confirmed that the request-body concern was valid. Serde accepted an empty JSON array as a field-empty struct, so [] incorrectly reached pruning instead of returning HTTP 400. The handler now parses a JSON value first and rejects every non-object body before deserializing PruneRequest. The focused test covers non-object bodies, unknown fields, invalid field types, malformed and uncommitted identifiers, and confirms that invalid requests leave no operation record. The English and Chinese guides now state these boundaries explicitly.

@WeissonHan
WeissonHan force-pushed the feature/blaze/checkpoint-prune-draft-v1 branch from 18eb9c9 to 04edc25 Compare August 24, 2026 07:05
@WeissonHan

Copy link
Copy Markdown
Collaborator Author

@codex review Enforces the documented prune request shape, covers empty and repeated protected identifiers, and makes HTTP 200, 400, and 409 cleanup boundaries explicit.

@WeissonHan
WeissonHan force-pushed the feature/blaze/checkpoint-prune-draft-v1 branch from 04edc25 to 669b045 Compare August 24, 2026 07:16
@WeissonHan

Copy link
Copy Markdown
Collaborator Author

@codex review Adds safe checkpoint-history pruning, rejects invalid request shapes before recording work, and keeps cleanup recoverable across interruption or client cancellation.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 04edc25abc

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/blaze/crates/blazed/src/api.rs Outdated
@WeissonHan

Copy link
Copy Markdown
Collaborator Author

@codex review Moves explicit empty-list coverage into an independent test, restores the pre-existing no-mutation assertion, and passes the exact-head hosted Blaze test.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. More of your lovely PRs please.

Reviewed commit: 669b045091

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@WeissonHan

Copy link
Copy Markdown
Collaborator Author

@qoder review

@qoderai qoderai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] prune 请求体错误信息与文档措辞略有偏差,建议在 API 错误文本中区分非对象请求体与字段级错误,并与 runtime 文档示例保持一致;[P2] RecoveryRequired 状态下 prune 统一返回 409 的行为在 README 路由列表中可以补一处显式说明,方便只读路由表的使用者;[P2] prune tombstone 命名校验遇到格式错误时的处理路径在设计文档中可补充一条简短运维提示,帮助识别需要人工介入的残留目录。


🤖 Generated by QoderView workflow run

Comment thread src/blaze/crates/blazed/src/api.rs
Comment thread src/blaze/README_zh.md
Comment thread src/blaze/crates/blazed/src/checkpoint_store.rs
Users can now prune checkpoint branches that are unreachable from the current HEAD.

Explicitly protected checkpoints keep their complete parent lineages.

Each removal uses an atomic tombstone, and prune intent is persisted before mutation.

Partial or uncertain cleanup enters recovery rather than reporting success.

Pruning is limited to running sandboxes. Recovery removes the complete checkpoint namespace.

Assisted-by: Codex:GPT-5
Signed-off-by: Weisson <Weisson@linux.alibaba.com>
@WeissonHan
WeissonHan force-pushed the feature/blaze/checkpoint-prune-draft-v1 branch from 669b045 to 36b8d14 Compare August 24, 2026 07:32
@WeissonHan

Copy link
Copy Markdown
Collaborator Author

@codex review Clarifies prune error diagnostics, states non-running HTTP 409 behavior in the endpoint tables, and documents safe handling of malformed cleanup-directory names.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Can't wait for the next one!

Reviewed commit: 36b8d14f50

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@WeissonHan

Copy link
Copy Markdown
Collaborator Author

@qoder review

@qoderai qoderai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

本次审查范围内未发现需要修改的问题。


🤖 Generated by QoderView workflow run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

component:blaze src/blaze scope:documentation ./docs/|./*.md|./NOTICE

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant