Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 35 additions & 4 deletions docs/user-guide/en/runtime/blaze.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,8 @@ to execute commands, read files, and write files inside them. Sandbox
destruction uses `DELETE /v1/sandboxes/{id}`. Checkpoint capture and history
use
`POST /v1/sandboxes/{id}/checkpoint` and
`GET /v1/sandboxes/{id}/checkpoints`.
`GET /v1/sandboxes/{id}/checkpoints`. Restore uses
`POST /v1/sandboxes/{id}/rollback/{checkpoint_id}`.

## Host Integration Boundary

Expand Down Expand Up @@ -214,7 +215,7 @@ The lifecycle invariants behind these compatibility responses are recorded in
the
[lifecycle state consistency and compatibility design](../../../../src/blaze/docs/design/lifecycle-state-consistency.md).

## Checkpoint Capture and History
## Checkpoint Capture, History, and Restore

Blaze captures a running sandbox through
`POST /v1/sandboxes/{id}/checkpoint`.
Expand Down Expand Up @@ -283,8 +284,38 @@ resumes the backend, and leaves the sandbox running. If Blaze cannot prove the
publication, HEAD update, persistence, or backend-resume outcome, it retains
the durable record and reports `RecoveryRequired`; do not retry capture until
the sandbox has been reconciled or destroyed. A committed checkpoint that did
not become HEAD can still appear in history with `is_head: false`. This release
does not provide checkpoint restore, deletion, or pruning APIs.
not become HEAD can still appear in history with `is_head: false`.

Restore a running sandbox with:

```http
POST /v1/sandboxes/{id}/rollback/{checkpoint_id}
```

Restore requires a verified full checkpoint, an exact match for the sandbox's
policy, image, backend, and backend version, plus explicit restore support from
both the backend adapter and storage provider. The built-in mock adapter and
file provider implement this contract. Other backend adapters return HTTP 501
before stopping the current runtime until they implement restore.

A `checkpoint_id` that is not in canonical form is rejected with HTTP 400, and a
canonical identifier that names no committed checkpoint is reported as HTTP 404.
Both answers are final: neither changes the running sandbox, so retrying the
same selection cannot succeed.

The file provider stages the selected root filesystem while the current
backend remains running. Blaze then stops the old backend, activates the staged
root, starts and checks the replacement owner, moves checkpoint HEAD, and
commits storage. The dividing line is whether Blaze has begun stopping the old
backend: a failure before that point, while still validating and staging the
replacement root, preserves the running sandbox untouched. Once Blaze starts
stopping the old backend, any later failure — including the stop itself failing
or Blaze being unable to confirm the old backend actually stopped — retains the
resources that actually exist and marks the sandbox `RecoveryRequired` so
destruction can finish cleanup. Restore moves checkpoint HEAD but does not
rewrite `last_checkpoint` or capture history.

Checkpoint deletion and pruning are not provided by this API.

## Storage Artifact Synchronization

Expand Down
33 changes: 29 additions & 4 deletions docs/user-guide/zh/runtime/blaze.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,8 @@ Blaze 通过 `/v1/sandboxes` 提供沙箱生命周期和客户机操作。客户
命名空间列出、创建、查看和删除沙箱,以及在沙箱内执行命令、读取文件和写入
文件。销毁沙箱使用 `DELETE /v1/sandboxes/{id}`。检查点捕获与历史查询分别使用
`POST /v1/sandboxes/{id}/checkpoint` 和
`GET /v1/sandboxes/{id}/checkpoints`。
`GET /v1/sandboxes/{id}/checkpoints`;恢复使用
`POST /v1/sandboxes/{id}/rollback/{checkpoint_id}`。

## 主机集成边界

Expand Down Expand Up @@ -181,7 +182,7 @@ Blaze 仍可读取旧版本写入的 `Reset`、`Warm` 和 `start_path = "warm"`
[生命周期状态一致性与兼容性设计](../../../../src/blaze/docs/design/lifecycle-state-consistency_zh.md)
中。

## 检查点捕获与历史
## 检查点捕获、历史与恢复

Blaze 通过 `POST /v1/sandboxes/{id}/checkpoint` 捕获运行中的 sandbox。

Expand Down Expand Up @@ -240,8 +241,32 @@ sandbox 或修改其生命周期记录前返回 HTTP 501。
能够确认发生在发布前的失败会删除临时数据、恢复后端,并让 sandbox 保持运行。
如果 Blaze 无法确认发布、HEAD 更新、持久化或后端恢复的结果,则会保留持久记录并
报告 `RecoveryRequired`;在 sandbox 完成恢复处理或销毁前,不应重试捕获。已经提交但
未成为 HEAD 的检查点仍可能出现在历史列表中,其 `is_head` 为 `false`。当前版本
不提供检查点恢复、删除或清理接口。
未成为 HEAD 的检查点仍可能出现在历史列表中,其 `is_head` 为 `false`。

可以使用以下接口恢复正在运行的 sandbox:

```http
POST /v1/sandboxes/{id}/rollback/{checkpoint_id}
```

恢复要求目标是经过校验的完整检查点,且策略、镜像、后端和后端版本都与当前
sandbox 完全一致;后端适配器和存储提供程序还必须明确声明支持恢复。内置 mock
适配器与文件存储提供程序实现了这项合同。其他后端适配器在实现恢复前会返回
HTTP 501,并且不会停止当前运行环境。

`checkpoint_id` 不符合规范形式时返回 HTTP 400;符合规范但没有对应已提交检查点
时返回 HTTP 404。这两种结果都是终态:都不会改动正在运行的 sandbox,用同一个
标识符重试也不可能成功。

文件存储提供程序会在当前后端仍运行时准备目标根文件系统。随后 Blaze 停止旧后端、
启用暂存根文件系统、启动并检查替代后端、移动检查点 HEAD,最后提交存储变更。
判断边界是 Blaze 是否已经开始停止旧后端:在此之前失败(仍处于校验和准备根文件
系统的阶段)时,旧后端照常运行,sandbox 不受影响;一旦开始停止旧后端,此后
任何失败——包括停止操作本身失败或无法确认旧后端是否真正停止——都会让 Blaze
保留实际存在的资源并把 sandbox 标记为 `RecoveryRequired`,以便销毁操作完成
清理。恢复会移动检查点 HEAD,但不会改写 `last_checkpoint` 或捕获历史。

该接口不提供检查点删除或清理能力。

## 存储制品同步

Expand Down
5 changes: 3 additions & 2 deletions src/blaze/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,8 +29,9 @@ Platform: Linux (x86_64 + aarch64) for production. macOS builds succeed but spaw
- **Daemon-only API model**: No CLI client for sandbox operations. All instance and template management is done via HTTP endpoints on UDS (`/run/blaze/api.sock`) or TCP (`:14159`). The CLI subcommands (`daemon start`, `daemon reload`, `daemon doctor`) only manage daemon lifecycle.
- **BackendSpawner trait**: All backend-specific process management is behind `BackendSpawner`. Adding a new backend means implementing `spawn()`, `wait()`, `kill()`, `probe()` and registering it in `daemon::build_spawner()`.
- **Policy-driven backend selection**: Workload class → policy file → prioritized backend list. The daemon probes backends at startup and selects the first available. Never hardcode backend preference in application logic.
- **Lifecycle state machine**: 9 states. The main branches are Pending →
Creating → Running and Running ↔ Paused → Checkpointed. Any non-terminal
- **Lifecycle state machine**: 10 states. The main branches are Pending →
Creating → Running, Running ↔ Paused → Checkpointed, and
Running → Restoring → Running for checkpoint restore. Any non-terminal
state can enter Destroyed; incomplete
cleanup enters RecoveryRequired. State transitions are enforced by
`blaze_core::lifecycle`. Do not bypass via direct field mutation.
Expand Down
28 changes: 25 additions & 3 deletions src/blaze/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -153,6 +153,7 @@ Blaze exposes sandbox lifecycle and guest operations through `/v1/sandboxes`.
| POST | `/v1/sandboxes/{id}/write` | Replace a guest file |
| POST | `/v1/sandboxes/{id}/checkpoint` | Capture a full checkpoint |
| GET | `/v1/sandboxes/{id}/checkpoints` | List committed checkpoint history |
| POST | `/v1/sandboxes/{id}/rollback/{checkpoint_id}` | Replace a running sandbox from a verified checkpoint |
| GET | `/v1/pools` | Reserved; returns `501` |
| GET | `/v1/pools/{backend}/{class}` | Reserved; returns `501` |
| POST | `/v1/pools/{backend}/{class}/drain` | Reserved; returns `501` |
Expand Down Expand Up @@ -258,9 +259,30 @@ sandbox state.

`GET /v1/sandboxes/{id}/checkpoints` returns committed history summaries,
including parentage, logical size, current-HEAD status, and HEAD reachability.
This release does not provide checkpoint restore or deletion.
See the [checkpoint capture user guide](../../docs/user-guide/en/runtime/blaze.md#checkpoint-capture-and-history)
for response fields, current backend support, and failure handling.

`POST /v1/sandboxes/{id}/rollback/{checkpoint_id}` is available only when the
current storage provider and checkpoint backend advertise compatible restore
capabilities. The daemon verifies the selected checkpoint, its parent chain,
runtime identity, and all artifact hashes before changing runtime state.

The file provider stages a separate rootfs copy while the current backend is
still running. After the old backend stops, the daemon selects that copy,
starts and owns the replacement backend, moves HEAD to the selected checkpoint,
and only then releases the previous rootfs. The dividing line is whether the
daemon has begun stopping the old backend: a failure before that point, while
still validating and staging the replacement rootfs, leaves the original
runtime running untouched, as if the restore never happened. Once the daemon
starts stopping the old backend, any later failure — including the stop itself
failing or the daemon being unable to confirm the old backend actually
stopped — retains the resources that actually exist and marks the sandbox
`RecoveryRequired`, so a later destroy can finish cleanup without losing
process ownership.

`last_checkpoint` continues to mean the most recent completed capture. Restore
moves catalog HEAD but does not rewrite capture history.

See the [checkpoint capture and restore user guide](../../docs/user-guide/en/runtime/blaze.md#checkpoint-capture-history-and-restore)
for response fields, supported capability combinations, and failure handling.

### Guest operations

Expand Down
35 changes: 31 additions & 4 deletions src/blaze/README_zh.md
Original file line number Diff line number Diff line change
Expand Up @@ -143,6 +143,7 @@ Blaze 通过 `/v1/sandboxes` 提供沙箱生命周期和客户机操作。
| POST | `/v1/sandboxes/{id}/write` | 替换 guest 文件 |
| POST | `/v1/sandboxes/{id}/checkpoint` | 捕获完整检查点 |
| GET | `/v1/sandboxes/{id}/checkpoints` | 列出已提交的检查点历史 |
| POST | `/v1/sandboxes/{id}/rollback/{checkpoint_id}` | 使用经过校验的检查点替换正在运行的 sandbox |
| GET | `/v1/pools` | 预留接口;返回 `501` |
| GET | `/v1/pools/{backend}/{class}` | 预留接口;返回 `501` |
| POST | `/v1/pools/{backend}/{class}/drain` | 预留接口;返回 `501` |
Expand Down Expand Up @@ -218,7 +219,7 @@ daemon 才会逐个处理未结束的 sandbox。后续逐项恢复期间,如
销毁捕获中断的 sandbox,而不是从其检查点恢复。恢复失败后目前没有后台循环自动
重试。重置接口仍不可用,也不会恢复检查点。

### 检查点捕获与历史
### 检查点捕获、历史与恢复

当运行中的 sandbox 所使用的后端和存储提供程序都声明支持完整捕获时,
`POST /v1/sandboxes/{id}/checkpoint` 会创建检查点。请求成功时,Blaze 会暂停后端,
Expand All @@ -228,9 +229,35 @@ daemon 才会逐个处理未结束的 sandbox。后续逐项恢复期间,如
改变 sandbox 状态前返回 HTTP 501。

`GET /v1/sandboxes/{id}/checkpoints` 返回已提交检查点的历史摘要,包括父检查点、
逻辑大小、是否为当前 HEAD,以及能否从 HEAD 到达。当前版本不提供检查点恢复或
删除接口。响应字段、当前后端支持情况和失败处理方式参见
[检查点捕获用户指南](../../docs/user-guide/zh/runtime/blaze.md#检查点捕获与历史)。
逻辑大小、是否为当前 HEAD,以及能否从 HEAD 到达。

`POST /v1/sandboxes/{id}/rollback/{checkpoint_id}` 用于把一个正在运行的
sandbox 回退到它此前捕获的某个检查点:丢弃当前的运行状态,改用该检查点保存
的那一份状态重新运行。只有当前使用的存储提供程序,以及捕获该检查点的后端,
都支持恢复能力时,这个接口才可用;否则 Blaze 不改动 sandbox 的任何状态,
直接返回 HTTP 501。

在真正改动运行状态之前,Blaze 会先做一整轮校验:确认所选检查点存在、它一直
回溯到最初检查点的整条父链完整、检查点记录的运行环境标识与当前一致,并逐个
核对所有制品文件的哈希。任意一项不通过都会中止,sandbox 保持原样。

恢复过程刻意遵循“先备好新状态、再切换、最后清理旧状态”的顺序,以免中途失败
损坏 sandbox。具体来说,旧后端还在运行时,Blaze 会先在旁边准备好一份独立的
根文件系统;只有等旧后端完全停止,才改用这份新的根文件系统启动并接管新的
后端,把检查点历史的当前指针(HEAD)指向所选检查点,最后才释放旧的根文件
系统。这里的分界点是 Blaze 是否已经开始停止旧后端:如果失败发生在这之前,
也就是仍处于校验和准备新根文件系统的阶段,旧后端一直照常运行,原来的运行
实例不受影响,相当于这次恢复没有发生。一旦 Blaze 开始停止旧后端,此后的
任何失败——包括停止操作本身失败、无法确认旧后端是否真的已经停止——都可能
留下清理不彻底的资源;此时 Blaze 会保留磁盘上确实存在的那部分资源,并把
sandbox 标记为 `RecoveryRequired`(需要恢复)状态,这样之后调用销毁接口时
仍能找到并清理这些残留资源。

`last_checkpoint` 字段始终指向最近一次成功捕获的检查点。回退只移动检查点
历史的当前指针,不会改写或删除已经捕获的历史记录。

响应字段、受支持的能力组合和失败处理方式参见
[检查点捕获与恢复用户指南](../../docs/user-guide/zh/runtime/blaze.md#检查点捕获历史与恢复)。

### Guest 操作

Expand Down
34 changes: 34 additions & 0 deletions src/blaze/crates/blaze-core/src/backend.rs
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,40 @@ pub struct SpawnRequest {
pub vm: Option<VmConfig>,
}

/// Backend identity and snapshot semantics accepted by a restore adapter.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct RestoreCapability {
/// Concrete backend implementation that can consume the checkpoint.
pub backend: BackendKind,
/// Exact backend version required by versioned snapshot formats.
pub version: Option<String>,
/// Snapshot flavor accepted by the adapter.
pub snapshot_kind: SnapshotKind,
}

/// Complete input for restoring an owned backend instance.
#[derive(Debug, Clone)]
pub struct RestoreRequest {
/// Stable sandbox identifier.
pub instance_id: Uuid,
/// Backend executable selected from the current daemon configuration.
pub binary_path: PathBuf,
/// Storage resources reconstructed for this sandbox.
pub storage: StorageSlot,
/// VM-state artifact from a committed checkpoint.
pub snapshot_path: PathBuf,
/// Guest-memory artifact from the same checkpoint.
pub mem_path: PathBuf,
/// Backend identity frozen into the checkpoint metadata.
pub checkpoint_backend: BackendKind,
/// Backend version frozen into the checkpoint metadata.
pub expected_version: Option<String>,
/// Snapshot flavor frozen into the checkpoint metadata.
pub snapshot_kind: SnapshotKind,
/// Whether the captured runtime exposed the stable run-directory guest transport.
pub expose_guest_socket: bool,
}

/// Snapshot flavor requested from a backend.
///
/// The file provider currently requires self-contained artifacts, so only
Expand Down
Loading
Loading