Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions src/apps/data-migrator/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,24 @@ fixture is **0.2.19**. This is format-based support, not a claim that every old
release has been tested. The source must pass the probe and selected domain
validators. Unsupported or corrupt data is kept and reported.

Scanning isolates invalid settings/model entries, workspace registrations, memory
rows/files, SSH profiles, Remote Connect files/Bots, Sessions, Skills, MiniApps,
and Agent definitions. Readable Turns within a damaged Session are recovered;
derived Turn counts and workspace reference lists are rebuilt. Identical Turn
copies are deduplicated; conflicting Turn identities are omitted with warnings.
Nested user memory notes are supported. Legacy memory jobs are not read or
imported: the runtime creates jobs on demand, and destination jobs stay intact.
Optional Skills and MiniApps do not block importing Agent definitions.

Item failures during staging are omitted from the committed manifest where the
item has an independent storage boundary. Source data remains read-only and the
report shows omissions and partial history recovery. Unreadable destination
stores, unsafe paths, unsupported schemas, changed inputs, and transaction/write
failures still protect the affected domain; independent domains can continue
after successful rollback. A failed rollback stops execution. Review warnings
and the report before retrying.


The destination is OpenBitFun: configuration schema **1**, workspace registry
format **1**, coordination database schema **2**, and the session, memory,
extension, and connection formats accepted by the shared storage owners in
Expand All @@ -68,6 +86,15 @@ both applications closed until the run finishes. Cancellation and window close
requests wait for an engine-declared safe boundary; already verified domains
may remain imported.


Agent coordination imports do not require historical Session/Turn references,
parent relationships, counters, timestamps, or Swarm lineage to remain valid.
Records that cannot be decoded or inserted under the target table constraints
are skipped individually. Tasks without a mapped Agent are skipped instead of
being attached to an unrelated target primary key. Existing target records still
win conflicts; imported Agent primary keys and task references are remapped.
Importing historical task state does not start or resume an Agent execution.

## Resume and diagnose

Plans, journals, reports, backups and staging live under:
Expand Down
18 changes: 18 additions & 0 deletions src/apps/data-migrator/README.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,12 @@ Windows 需要 Microsoft Edge WebView2;macOS 使用系统 WebView,Linux 包
支持以实际数据格式和各领域校验为准,不代表已验证范围内的每个旧版本。
未知格式、损坏数据会明确报告并保留。

扫描会按单个会话、运行事件日志、Skill、MiniApp 和 Agent 定义隔离异常;有效条目继续迁移,
跳过的条目保留在来源中并记录到报告。旧会话的 Turn 计数以实际文件为准,仅在迁移副本中重建。
整个数据域不可用时,只跳过该域及依赖它的域。执行时某域失败会先回滚,再继续无依赖的数据域;
无法安全回滚时仍会停止。请在重试前查看警告与迁移报告。


目标是 OpenBitFun:配置 schema **1**、工作区格式 **1**、任务协调数据库 schema **2**,
以及此源码版本共享存储模块支持的会话、记忆、扩展和连接格式。未知产品/配置格式以及
超出支持范围的数据库、会话版本会被拒绝。未来数据格式变更需要发布新的迁移器,
Expand All @@ -49,6 +55,12 @@ Windows 需要 Microsoft Edge WebView2;macOS 使用系统 WebView,Linux 包
来源不会被自动删除。写入使用一致性快照、暂存、校验、备份、迁移锁和原子替换。
迁移期间请保持相关应用关闭;取消或关闭窗口会等到安全边界,已验证完成的领域可能已导入。


协调数据库不再要求历史 Session/Turn 引用、父子关系、计数、时间戳或 Swarm 层级完整有效。
无法解析或不能写入目标表约束的记录会单独跳过;缺少 Agent 主键映射的任务也单独跳过,
不会误挂到目标库中同号的 Agent。合并仍保持目标已有记录优先,并映射新增 Agent 主键和任务引用。
导入历史任务状态不会启动或恢复 Agent 执行。

## 中断恢复与诊断

任务保存在:
Expand Down Expand Up @@ -98,3 +110,9 @@ cargo build -p openbitfun-data-migrator --bin openbitfun-data-migrator
独立文件签名不等于 Apple/Authenticode 系统代码签名;当前工作流尚未配置这些证书及 macOS 公证。

开发约束和针对性检查见 [AGENTS.md](AGENTS.md)。

### 损坏旧数据的处理粒度

设置和模型、工作区登记、记忆记录和文件、SSH 连接、Remote Connect 文件和 Bot、会话和扩展均按条目处理。损坏会话仍会恢复可读取的 Turn,重建 Turn 数量和工作区引用列表;相同 Turn 副本去重,身份冲突的 Turn 会跳过并报告。记忆笔记支持嵌套目录。旧记忆库的 jobs 不读取、不迁移,运行时按需创建任务,目标库已有 jobs 保留。Skill 或 MiniApp 不可用不会阻止 Agent 定义迁移。

有独立存储边界的条目在暂存失败时会从提交清单中排除,报告会显示遗漏和会话部分恢复提示。旧数据保持只读;无法读取的目标存储、路径越界、不支持的格式、输入变更和事务写入失败仍会保护相关域,成功回滚后其余独立域可继续。回滚失败时停止执行。
10 changes: 7 additions & 3 deletions src/apps/data-migrator/ui/app.js
Original file line number Diff line number Diff line change
Expand Up @@ -171,14 +171,15 @@ function render(view) {

const findings = document.getElementById('findings');
findings.replaceChildren(...view.findings.map((finding) =>
row(finding.code, `${finding.entityCount} item(s), ${finding.logicalBytes} byte(s)`)));
row(finding.domain, `${text[finding.code] || finding.code} · ${finding.entityCount} item(s), ${finding.logicalBytes} byte(s)`)));
show('scan-card', view.findings.length > 0 && !view.plan);

const planSummary = document.getElementById('plan-summary');
if (view.plan) {
planSummary.replaceChildren(
row(text.steps.replace('{count}', view.plan.steps.length), view.plan.selection.groups.map(groupLabel).join(' · ')),
row(text.conflicts.replace('{count}', view.plan.conflicts.length), text.confirmHelp),
...view.plan.findings.filter((finding) => finding.severity !== 'info').map((finding) => row(finding.domain, text[finding.code] || finding.code)),
...view.plan.conflicts.map((conflict) => row(conflict.domain, conflict.code || conflict.resolution || '')),
);
}
Expand All @@ -201,8 +202,11 @@ function render(view) {

const reportSummary = document.getElementById('report-summary');
if (view.report) {
reportSummary.replaceChildren(...view.report.domainResults.map((result) =>
row(result.domain, `${result.imported} ${transferLabel(result)}, ${result.skipped} ${text.skipped}, ${result.warnings.filter((item) => item.severity !== 'info').length} ${text.warnings}`)));
reportSummary.replaceChildren(...view.report.domainResults.flatMap((result) => [
row(result.domain, `${result.imported} ${transferLabel(result)}, ${result.skipped} ${text.skipped}, ${result.warnings.filter((item) => item.severity !== 'info').length} ${text.warnings}`),
...[...new Set(result.warnings.filter((item) => item.severity !== 'info').map((item) => item.code))]
.map((code) => row(result.domain, text[code] || code)),
]));
}
show('report-card', !view.running && (Boolean(view.report) || view.status === 'cancelled'));
const canExportDiagnostics = ['failed_recoverable', 'failed_manual_action_required'].includes(view.status);
Expand Down
51 changes: 48 additions & 3 deletions src/apps/data-migrator/ui/locales.json
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,22 @@
"skillsRoot": "Skills",
"sshRoot": "SSH data",
"unreadableTask": "Unavailable — restore original locations or inspect the saved files",
"confirmHelp": "Review the destination, selected groups, and conflicts. Existing destination records take priority; the tool keeps backups for replaced files."
"confirmHelp": "Review the destination, selected groups, and conflicts. Existing destination records take priority; the tool keeps backups for replaced files.",
"domain_scan_skipped": "Could not read or validate this domain. It will be skipped; other domains can continue.",
"domain_dependency_skipped": "A required domain is unavailable. This domain will be skipped.",
"extension_items_skipped": "Incomplete or unsupported extensions will be skipped; valid extensions can continue.",
"session_items_skipped": "Some Session files will be skipped; valid Sessions can continue.",
"coordination_rows_skipped": "Some coordination records cannot be read or stored and will be skipped; other records can continue.",
"settings_items_skipped": "Invalid settings or model entries were skipped; valid settings remain available.",
"memory_items_skipped": "Unreadable memory entries were skipped; readable memories remain available.",
"memory_item_skipped": "An unreadable memory item was preserved in the source and omitted from this import.",
"remote_items_skipped": "Unreadable remote records were skipped; other connections remain available.",
"session_turns_recovered": "Session history was partially recovered or its Turn filenames repaired. Check the original for omitted content.",
"setting_field_skipped": "An invalid setting was skipped; the target or default value was retained.",
"workspace_item_skipped": "An unreadable workspace registration was skipped; other workspaces and sessions remain available.",
"session_entry_skipped": "Some Session files will be skipped; valid Sessions can continue.",
"session_item_skipped": "Some Session files will be skipped; valid Sessions can continue.",
"extension_item_skipped": "Incomplete or unsupported extensions will be skipped; valid extensions can continue."
},
"zh-CN": {
"eyebrow": "OpenBitFun 独立数据迁移器",
Expand Down Expand Up @@ -119,7 +134,22 @@
"skillsRoot": "Skills",
"sshRoot": "SSH 数据",
"unreadableTask": "暂不可用,请恢复原目录或检查任务文件",
"confirmHelp": "请确认目标目录、迁移范围和冲突。已有目标记录优先;工具会为需要替换的文件保留备份。"
"confirmHelp": "请确认目标目录、迁移范围和冲突。已有目标记录优先;工具会为需要替换的文件保留备份。",
"domain_scan_skipped": "此数据域无法读取或校验,本次将跳过,其他数据域可继续迁移。",
"domain_dependency_skipped": "依赖的数据域不可用,本次将跳过此数据域。",
"extension_items_skipped": "将跳过不完整或不支持的扩展,其他有效扩展可继续迁移。",
"session_items_skipped": "部分会话文件将被跳过,其他有效会话可继续迁移。",
"coordination_rows_skipped": "部分协调记录无法读取或写入,将单独跳过,其他记录可继续迁移。",
"settings_items_skipped": "已跳过无效设置或模型条目,其余设置可继续迁移。",
"memory_items_skipped": "已跳过无法读取的记忆条目,其余记忆可继续迁移。",
"memory_item_skipped": "无法读取的记忆条目保留在旧数据中,本次不迁移。",
"remote_items_skipped": "已跳过无法读取的远程记录,其余连接可继续迁移。",
"session_turns_recovered": "会话历史已部分恢复,或已修正 Turn 文件名。遗漏内容请查阅原会话。",
"setting_field_skipped": "已跳过无效设置,保留目标值或默认值。",
"workspace_item_skipped": "已跳过无法读取的工作区登记,其余工作区和会话可继续迁移。",
"session_entry_skipped": "部分会话文件将被跳过,其他有效会话可继续迁移。",
"session_item_skipped": "部分会话文件将被跳过,其他有效会话可继续迁移。",
"extension_item_skipped": "将跳过不完整或不支持的扩展,其他有效扩展可继续迁移。"
},
"zh-TW": {
"eyebrow": "OpenBitFun 獨立資料遷移器",
Expand Down Expand Up @@ -180,6 +210,21 @@
"skillsRoot": "Skills",
"sshRoot": "SSH 資料",
"unreadableTask": "暫不可用,請恢復原目錄或檢查工作檔案",
"confirmHelp": "請確認目標目錄、遷移範圍和衝突。已有目標記錄優先;工具會為需要取代的檔案保留備份。"
"confirmHelp": "請確認目標目錄、遷移範圍和衝突。已有目標記錄優先;工具會為需要取代的檔案保留備份。",
"domain_scan_skipped": "此資料域無法讀取或驗證,本次將略過,其他資料域可繼續遷移。",
"domain_dependency_skipped": "相依的資料域不可用,本次將略過此資料域。",
"extension_items_skipped": "將略過不完整或不支援的擴充功能,其他有效項目可繼續遷移。",
"session_items_skipped": "部分工作階段檔案將被略過,其他有效工作階段可繼續遷移。",
"coordination_rows_skipped": "部分協調記錄無法讀取或寫入,將單獨略過,其他記錄可繼續遷移。",
"settings_items_skipped": "已略過無效設定或模型項目,其餘設定可繼續遷移。",
"memory_items_skipped": "已略過無法讀取的記憶項目,其餘記憶可繼續遷移。",
"memory_item_skipped": "無法讀取的記憶項目保留在舊資料中,本次不遷移。",
"remote_items_skipped": "已略過無法讀取的遠端記錄,其餘連線可繼續遷移。",
"session_turns_recovered": "工作階段歷史已部分恢復,或已修正 Turn 檔名。遺漏內容請查閱原工作階段。",
"setting_field_skipped": "已略過無效設定,保留目標值或預設值。",
"workspace_item_skipped": "已略過無法讀取的工作區登記,其餘工作區和工作階段可繼續遷移。",
"session_entry_skipped": "部分工作階段檔案將被略過,其他有效工作階段可繼續遷移。",
"session_item_skipped": "部分工作階段檔案將被略過,其他有效工作階段可繼續遷移。",
"extension_item_skipped": "將略過不完整或不支援的擴充功能,其他有效項目可繼續遷移。"
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -1142,9 +1142,8 @@ fn unix_time_ms() -> u64 {
#[cfg(test)]
mod tests {
use super::*;
use crate::service::coordination_persistence::{
coordination_table_has_column, initialize_coordination_schema,
};
use crate::service::coordination_persistence::initialize_coordination_schema;
use openbitfun_services_core::coordination_persistence::coordination_table_has_column;

fn test_tempdir() -> tempfile::TempDir {
if let Some(root) = std::env::var_os("OPENBITFUN_TEST_TMPDIR") {
Expand Down
Original file line number Diff line number Diff line change
@@ -1,18 +1,10 @@
use crate::util::errors::OpenBitFunResult;
use openbitfun_services_core::coordination_persistence as storage;
pub(crate) use openbitfun_services_core::coordination_persistence::COORDINATION_SCHEMA_VERSION;
use rusqlite::Connection;

pub(crate) fn initialize_coordination_schema(connection: &Connection) -> OpenBitFunResult<()> {
storage::initialize_coordination_schema(connection).map_err(Into::into)
}
pub(crate) fn coordination_table_has_column(
connection: &Connection,
table: &str,
column: &str,
) -> OpenBitFunResult<bool> {
storage::coordination_table_has_column(connection, table, column).map_err(Into::into)
}
pub(crate) fn validate_coordination_agent_id(agent_id: &str) -> OpenBitFunResult<()> {
storage::validate_coordination_agent_id(agent_id).map_err(Into::into)
}
4 changes: 0 additions & 4 deletions src/crates/assembly/core/src/service/workspace/persistence.rs
Original file line number Diff line number Diff line change
@@ -1,14 +1,10 @@
use super::types::WorkspaceInfo;
use crate::util::errors::{OpenBitFunError, OpenBitFunResult};
use openbitfun_services_core::workspace_persistence as storage;
pub(crate) use openbitfun_services_core::workspace_persistence::{
WorkspacePersistenceData, WORKSPACE_PERSISTENCE_FORMAT_VERSION,
};
use std::path::Path;

pub(crate) fn current_workspace_storage_id(workspace: &WorkspaceInfo) -> OpenBitFunResult<String> {
storage::current_workspace_storage_id(workspace).map_err(Into::into)
}
pub(crate) fn validate_workspace_persistence_data(
data: &WorkspacePersistenceData,
miniapp_root: &Path,
Expand Down
Loading
Loading