Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
73 changes: 73 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
name: Tianjun CI

on:
push:
pull_request:

jobs:
verify:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
python-version: ["3.11", "3.12"]
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
cache: pip
- uses: actions/setup-node@v4
with:
node-version: "22"
- uses: actions/setup-java@v4
with:
distribution: temurin
java-version: "21"
- name: Install Node dependencies
run: npm ci
- name: Install test dependencies
run: python -m pip install -e ".[dev,experiments,mcp]"
- name: Build and smoke-test installed wheel
run: |
python -m pip wheel . --no-deps --wheel-dir wheelhouse
python scripts/wheel_install_smoke_test.py wheelhouse/*.whl
- name: Compile Python
run: python -m compileall -q src tests
- name: Check Dashboard JavaScript syntax
run: find src/tianjun/interfaces/dashboard/static/js -name '*.js' -print0 | xargs -0 -n1 node --check
- name: Run executable frontend tests
run: npm run test:frontend
- name: Run tests
run: python -m pytest
- name: Run convergence checks
run: python scripts/convergence_check.py
- name: Run offline smoke test
run: python scripts/smoke_test.py --port 8136

browser:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
cache: pip
- uses: actions/setup-node@v4
with:
node-version: "22"
cache: npm
- name: Install Tianjun and browser test dependencies
run: |
python -m pip install -e .
npm ci
- name: Install Chromium
run: npx playwright install --with-deps chromium
- name: Run browser tests
run: npm run test:browser
- uses: actions/upload-artifact@v4
if: ${{ !cancelled() }}
with:
name: playwright-report
path: playwright-report/
retention-days: 14
9 changes: 9 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,11 @@ __pycache__/
.mypy_cache/
.ruff_cache/
*.egg-info/
build/
wheelhouse/
node_modules/
playwright-report/
test-results/

.env
configs/secrets.toml
Expand All @@ -15,3 +20,7 @@ tianjun.toml

data/alibaba_microservices_v2022/
runtime_report.json

# Local experiment outputs and temporary runtime data
exp_out/
tmp/
23 changes: 20 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@ Tianjun Engine 是一个本地优先的算网调度控制平面原型。它把

本项目用于研究、演示和架构实验,并非生产级云平台。资源清单、定价、拓扑和执行事实必须来自已注册节点、CloudSimPlus 桥接器或真实节点代理;LLM 可以解释和帮助解析意图,但不能在没有明确确认路径的情况下捏造控制平面事实或提交工作。

当前版本已加入批任务 JSON/CSV 导入、共享快照联合分配、Pareto + Tchebycheff 十维评分、Future-Fit 碎片评估、运行碳核算、显式确认原子预留以及外部 Hermes MCP 审计。接口和核算口径见 [批任务与运行碳调度说明](docs/batch-carbon-scheduling.md)。

## 快速开始

### 0. 安装依赖
Expand Down Expand Up @@ -84,9 +86,13 @@ CloudSimPlus 示例节点会随注册请求上报 CPU、内存、GPU、存储、
任务下发后的状态流转如下:

1. `/tasks` 或策略提交只把任务写入控制平面并加入 `pending_queue`,此时任务会显示为“待调度”,还不会出现在执行记录。
2. CloudSimPlus 桥接器或真实节点代理持续心跳并请求 `/leases/next`,控制平面才会做调度决策并把任务租约发给目标节点。
3. 节点执行过程中回传 `/task-runs/progress`,Dashboard 的拓扑路径会根据 `/report` 中的 `active_runs`、最新进度、调度决策和节点 inventory 实时切换。
4. 节点最终回传 `/task-runs/result` 后,控制平面才写入执行记录;因此“执行记录没有这个任务”通常表示任务还在待调度队列或已发租约但尚未回传结果。
2. CloudSimPlus 桥接器或真实节点代理持续心跳并请求 `/leases/next`,取得带唯一 `lease_id` 和 TTL 的任务租约,再通过 `/leases/ack` 确认领取。
3. 节点执行过程中回传 `/task-runs/progress`,进度会续期租约;超时且没有续期的租约会释放资源并按重试策略重新排队。
4. 节点最终通过 `/task-runs/result` 携带 `lease_id` 与稳定的 `result_id` 回传结果;相同结果重试会返回同一收据,不会重复写执行记录。

控制平面默认把 SQLite v2 状态库写入平台用户状态目录(`${TIANJUN_STATE_DIR}/tianjun-state.sqlite`)。节点最后在线时间、任务、批次、预演计划、预留账本、幂等映射与结果收据均可跨进程重启恢复;可用 `--state-db` 覆盖路径。数据库升级前会在原目录创建 `*.pre-v2-from-v*.bak` 完整备份;高于程序支持版本的数据库会拒绝打开,迁移失败会整体回滚。`/ready` 会执行完整性检查和可逆写入探测。Dashboard 中“实时遥测”“CloudSim 模拟”“配置曲线”和“分配估算”是不同来源,不应互相替代解读。

HTTP 服务内置独立生命周期清理器,不依赖 Dashboard 轮询即可回收过期节点和租约。默认每秒检查一次,可通过 `server.lifecycle_sweep_interval_seconds` 调整;服务关闭时会等待清理线程安全退出。

### 5. 打开 Dashboard

Expand All @@ -112,6 +118,8 @@ python -B main.py mcp-server `

MCP server 会把 Tianjun HTTP API 包装为工具,包括读取集群状态、开始/继续聊天会话、起草/比较/仿真/解释策略,以及带确认边界的策略提交和任务调度。

顶部的 MCP 状态只统计真正带 MCP 工具标识的 HTTP 请求,启动 MCP 进程本身不算成功调用。测试套件会启动真实 stdio MCP 子进程并调用 `get_cluster_state`,验证 Dashboard 的最近调用和成功计数同步更新。

### 7. 可选:真实节点代理

如需使用真实节点遥测代理,而不是模拟节点:
Expand Down Expand Up @@ -143,6 +151,15 @@ python scripts\smoke_test.py --port 8135

该脚本会启动离线控制平面,检查 `/health`、`/report`、`/dashboard`,并验证 MCP 工具契约可导入。它用于快速验证,不代表完整启动。

前端逻辑和真实浏览器回归分别运行:

```powershell
npm run test:frontend
npm run test:browser
```

浏览器回归使用 Chromium 覆盖 1366 和 1920 两档 PC 视口,包括标签页键盘语义、轮询竞态、拓扑图层、空态/错误态和拓扑几何边界。


## LLM 配置

Expand Down
12 changes: 12 additions & 0 deletions configs/batch_experiments.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
{
"schema_version": "1.0",
"node_counts": [20, 50, 100],
"batch_task_counts": [20, 100, 500],
"load_rates": [0.30, 0.60, 0.85, 0.95],
"workloads": ["cpu", "gpu", "memory", "data", "mixed"],
"seeds": [20260718, 20260719, 20260720, 20260721, 20260722, 20260723, 20260724, 20260725, 20260726, 20260727],
"online_strategies": ["B0-current", "B1-batch-greedy", "B3-batch-local-search", "B4-pareto-tchebycheff", "B6-hierarchical-batch"],
"offline_strategies": ["B2-milp-oracle", "B5-nsga2"],
"carbon_scope": "operational_only",
"normalization_bounds_version": "engineering-v1"
}
19 changes: 19 additions & 0 deletions configs/cloudsim_core_experiments.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
{
"schema_version": "1.0",
"description": "CloudSim Plus execution-level comparison. Every run starts an isolated Tianjun control plane and writes measured batch metrics under exp_out/cloudsim_core.",
"strategies": [
"B0-current",
"B4-pareto-tchebycheff",
"B6-hierarchical-batch",
"B6-green-single-v1",
"B6-green-sla-85-v1"
],
"scenarios": [
"normal",
"fault-active"
],
"seeds": [20260527, 20260528, 20260529, 20260530, 20260531, 20260532, 20260533, 20260534, 20260535, 20260536],
"cloudlets": 36,
"carbon_scope": "operational_only",
"output_directory": "exp_out/cloudsim_core"
}
59 changes: 59 additions & 0 deletions configs/fragmentation_green_experiments.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
{
"schema_version": "3.0",
"description": "High-load heterogeneous-fragmentation and green objective ablation with held-out calibration seeds.",
"node_counts": [20],
"batch_task_counts": [60],
"load_rates": [0.85, 0.95],
"workloads": ["mixed"],
"fragmentation_modes": ["heterogeneous"],
"seeds": [20260718, 20260719, 20260720, 20260721, 20260722],
"training_seeds": [20260718, 20260719, 20260720],
"validation_seeds": [20260721, 20260722],
"online_strategies": [
"B0-current",
"B4-pareto-tchebycheff",
"B6-hierarchical-batch"
],
"offline_strategies": [],
"objective_experiments": {
"single_atomic": ["carbon", "fragmentation"],
"dual_atomic": [
["carbon", "completion"],
["carbon", "fragmentation"]
],
"single_groups": ["green_carbon", "resource_efficiency"],
"dual_groups": [
["green_carbon", "sla_quality"],
["green_carbon", "resource_efficiency"]
]
},
"weight_profiles": [
{
"label": "W0-current-green",
"strategy": "B6-hierarchical-batch",
"group_weights": {"sla_quality": 0.26, "network_coordination": 0.20, "resource_efficiency": 0.22, "economic_cost": 0.12, "green_carbon": 0.20}
},
{
"label": "W1-carbon-080",
"strategy": "B6-hierarchical-batch",
"group_weights": {"sla_quality": 0.25, "network_coordination": 0.10, "resource_efficiency": 0.20, "economic_cost": 0.05, "green_carbon": 0.40}
},
{
"label": "W2-carbon-120",
"strategy": "B6-hierarchical-batch",
"group_weights": {"sla_quality": 0.20, "network_coordination": 0.07, "resource_efficiency": 0.20, "economic_cost": 0.03, "green_carbon": 0.50}
},
{
"label": "W3-carbon-160",
"strategy": "B6-hierarchical-batch",
"group_weights": {"sla_quality": 0.18, "network_coordination": 0.05, "resource_efficiency": 0.15, "economic_cost": 0.02, "green_carbon": 0.60}
},
{
"label": "W4-carbon-200",
"strategy": "B6-hierarchical-batch",
"group_weights": {"sla_quality": 0.20, "network_coordination": 0.04, "resource_efficiency": 0.25, "economic_cost": 0.01, "green_carbon": 0.50}
}
],
"carbon_scope": "operational_only",
"normalization_bounds_version": "engineering-v1"
}
38 changes: 38 additions & 0 deletions configs/objective_ablation_experiments.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
{
"schema_version": "2.0",
"node_counts": [20, 50],
"batch_task_counts": [20, 100],
"load_rates": [0.60, 0.85],
"workloads": ["mixed"],
"seeds": [20260718, 20260719, 20260720, 20260721, 20260722],
"online_strategies": [
"B0-current",
"B4-pareto-tchebycheff",
"B6-hierarchical-batch"
],
"offline_strategies": [],
"objective_experiments": {
"single_atomic": [
"performance",
"completion",
"cost",
"reliability",
"balance",
"fragmentation",
"locality",
"network",
"carbon"
],
"dual_atomic": "all",
"single_groups": [
"sla_quality",
"network_coordination",
"resource_efficiency",
"economic_cost",
"green_carbon"
],
"dual_groups": "all"
},
"carbon_scope": "operational_only",
"normalization_bounds_version": "engineering-v1"
}
3 changes: 3 additions & 0 deletions configs/tianjun.example.toml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,10 @@
host = "127.0.0.1"
port = 8024
heartbeat_timeout_seconds = 15
lease_timeout_seconds = 60
lifecycle_sweep_interval_seconds = 1
policy_update_interval = 2
state_db = "${TIANJUN_STATE_DIR}/tianjun-state.sqlite"

[model]
dir = "${TIANJUN_HOME}/data/trained_models"
Expand Down
23 changes: 18 additions & 5 deletions docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,14 @@

| 方法 | 路径 | 用途 |
| --- | --- | --- |
| GET | `/health` | 运行时健康、模型状态、聊天运行时状态 |
| GET | `/report` | Dashboard 和工具用的控制平面状态 |
| GET | `/health` | 进程健康和去敏后的依赖状态;始终返回可解析状态 |
| GET | `/ready` | 就绪检查;依赖未就绪时返回 503 |
| GET | `/report` | 完整控制平面状态,保留给 MCP 和兼容客户端 |
| GET | `/report/summary` | Dashboard 总览和顶部状态的精简报告 |
| GET | `/report/scheduling` | 调度决策页所需节点与决策报告 |
| GET | `/report/topology` | 拓扑、VM 遥测和路径报告 |
| GET | `/report/tasks?limit=50&cursor=0` | 分页任务执行报告 |
| GET | `/report/model` | 模型、权重与策略历史报告 |
| GET | `/dashboard` | 静态 Dashboard 壳页面 |
| GET | `/chat/sessions/{session_id}` | 读取聊天会话状态 |
| POST | `/chat/sessions` | 开始聊天会话 |
Expand Down Expand Up @@ -35,9 +41,10 @@
| POST | `/nodes/heartbeat` | 更新节点心跳和遥测数据 |
| POST | `/tasks` | 提交任务 |
| POST | `/tasks/{task_id}/schedule` | 调度待处理任务;需要明确确认 |
| POST | `/leases/next` | 节点代理租约轮询 |
| POST | `/task-runs/progress` | 报告任务进度 |
| POST | `/task-runs/result` | 报告最终任务结果 |
| POST | `/leases/next` | 节点代理租约轮询;返回 `lease_id`、签发时间与到期时间 |
| POST | `/leases/ack` | 使用 `node_id`、`task_id`、`lease_id` 确认领取并续期 |
| POST | `/task-runs/progress` | 报告任务进度并续期租约;新客户端应携带 `lease_id` |
| POST | `/task-runs/result` | 幂等报告最终结果;新客户端应携带 `lease_id` 和稳定 `result_id` |
| POST | `/task-runs/cancel` | 取消活动任务运行 |
| POST | `/schedule/preview` | CloudSimPlus 兼容的调度预览 |
| POST | `/schedule/commit` | CloudSimPlus 兼容的直接提交 |
Expand Down Expand Up @@ -68,3 +75,9 @@ HTTP 路由调用 `CentralControlPlane` facade。facade 将已迁移的行为转
- `PolicyWorkflowService` 处理策略起草、比较、模拟、提交和反馈路由。

本文只描述公开 API 行为;服务拆分不改变路由语义。

Dashboard 只轮询按页面拆分的报告。所有报告视图都带有 `report_version`、`resource_snapshot_version` 和 `generated_at`,客户端可据此识别跨请求快照差异。完整 `/report` 不再用于浏览器高频轮询。

CloudSimPlus 心跳中的 `telemetry.cpu_utilization`、`ram_utilization` 和 `bandwidth_utilization` 会规范化为节点的 `runtime_utilization`。未上报的指标保持 `null`,Dashboard 显示为 `--`,不会生成伪实时值。

租约默认 TTL 为 60 秒,可通过 `server.lease_timeout_seconds` 配置。相同 `result_id` 的重试返回已保存的结果收据并标记 `idempotent_replay=true`,不会重复累计执行、能耗或碳数据。
8 changes: 5 additions & 3 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,12 +28,13 @@ Tianjun Engine 围绕公共适配器、中央控制平面门面和可测试的
| 服务 | 当前职责 |
| --- | --- |
| `NodeRegistry` | 节点注册、心跳、节点遥测变更、节点持久化 |
| `TaskLeaseService` | 任务提交、预览、pending 调度、agent 租约轮询、租约激活 |
| `TaskLeaseService` | 任务提交、预览、pending 调度、租约发放、ACK、TTL 续期/回收和并发幂等 |
| `LifecycleSweeper` | 与用户流量解耦地回收过期节点和租约,并随 HTTP 服务安全启停 |
| `RequirementDialogueService` | 需求解析、需求会话开始/继续/读取、地域可用性载荷 |
| `PolicyWorkflowService` | 策略起草、候选比较、模拟、提交、反馈解析、反馈记录、反馈优化 |
| `src/tianjun/cli/commands/` | 所有 CLI 命令处理器 |

`CentralControlPlane` 保留 facade 方法、共享状态、报表组装、恢复/持久化协调、拓扑注册、策略权重更新以及执行进度/结果回报等跨领域逻辑。已经迁移到服务中的业务流程不应复制回门面类。
`CentralControlPlane` 保留 facade 方法、共享状态、拓扑注册、策略权重更新以及执行进度/结果回报等跨领域逻辑。数据库模式/迁移、控制面恢复、批输入校验、聊天模型/常量与拓扑几何已拆为独立模块;已经迁移到服务中的业务流程不应复制回门面类。

## CloudSimPlus 仿真链路

Expand All @@ -43,7 +44,8 @@ Tianjun Engine 围绕公共适配器、中央控制平面门面和可测试的
- 调用 `/nodes/register` 注册 CloudSimPlus 仿真 VM 节点。
- 持续调用 `/nodes/heartbeat` 上报在线状态。
- 通过 `/schedule/commit` 请求 Tianjun 控制平面做调度决策。
- 在 CloudSimPlus 仿真完成后通过 `/task-runs/result` 回报执行结果。
- 通过 `/leases/next` 和 `/leases/ack` 领取并确认带 TTL 的任务租约。
- 在 CloudSimPlus 仿真完成后通过携带 `lease_id`/`result_id` 的 `/task-runs/result` 幂等回报执行结果。

完整启动命令见 [README.md](../README.md)。

Expand Down
Loading
Loading