Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1,156 changes: 1,042 additions & 114 deletions Cargo.lock

Large diffs are not rendered by default.

2 changes: 2 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,8 @@ serde_repr = "0.1.20"
secrecy = "0.10"
sha1 = "0.11"
sha2 = "0.11"
russh = "0.62.1"
russh-sftp = "2.3.0"
tokio = { version = "1", features = ["macros", "rt-multi-thread", "signal", "fs", "io-util"] }
tokio-tungstenite = { version = "0.29", features = ["rustls-tls-native-roots"] }
tokio-util = { version = "0.7", features = ["io", "io-util"] }
Expand Down
10 changes: 6 additions & 4 deletions developer-docs/en/api/admin.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ Create example:

Current notes:

- `driver_type` currently supports `local`, `s3`, `azure_blob`, `tencent_cos`, `remote`, and `onedrive`
- `driver_type` currently supports `local`, `s3`, `sftp`, `azure_blob`, `tencent_cos`, `remote`, and `one_drive`
- `GET /admin/policies/storage-drivers` returns `StorageConnectorDescriptor` entries. The frontend should use descriptor `capabilities`, `fields`, `upload_workflows`, `actions`, and `credential_mode` to decide forms, connection tests, upload/download strategies, and action affordances instead of maintaining a hard-coded driver capability matrix.
- create and update both honor request `chunk_size`
- `options` carries policy-level behavior:
Expand All @@ -88,11 +88,13 @@ Current notes:
- storage-native thumbnails / image previews with `storage_native_processing_enabled`, `thumbnail_processor`, and `thumbnail_extensions`
- storage-native media metadata with `storage_native_media_metadata_enabled` and `media_metadata_extensions`
- OneDrive location options: `onedrive_account_mode`, `onedrive_tenant`, `onedrive_site_id`, `onedrive_drive_id`, `onedrive_group_id`, and `onedrive_root_item_id`
- SFTP host key pinning: `sftp_host_key_fingerprint`
- `application_config.microsoft_graph` stores OneDrive / Microsoft Graph app settings. Client secrets are stored encrypted; API responses expose only `client_secret_configured`.
- `driver_type = "azure_blob"` uses Azure Block Blob capabilities. Presigned browser upload uses SAS URLs and requires the client to send `x-ms-blob-type: BlockBlob`.
- `driver_type = "onedrive"` uses Microsoft Graph OAuth credentials. Save the policy and `application_config.microsoft_graph` before starting authorization.
- `driver_type = "one_drive"` uses Microsoft Graph OAuth credentials. Save the policy and `application_config.microsoft_graph` before starting authorization.
- `driver_type = "sftp"` uses SSH username / password credentials to connect to an SFTP server. Endpoint supports `sftp://host:port`, bare `host`, and `host:port`; the remote root belongs in `base_path`. Unknown or mismatched SSH host keys are rejected as `StorageErrorKind::Precondition` with diagnostics that include actual / expected fingerprints; the confirmed fingerprint is stored in `options.sftp_host_key_fingerprint`.
- `driver_type = "tencent_cos"` uses the S3-compatible object path for normal reads and writes, validates Tencent COS endpoint shape, and can expose COS CI storage-native thumbnail / image-preview / media-metadata capabilities when the policy opts in
- built-in Local, S3-compatible, Azure Blob, OneDrive, and Remote drivers do not expose storage-native thumbnail, image-preview, or media-metadata capabilities
- built-in Local, S3-compatible, SFTP, Azure Blob, OneDrive, and Remote drivers do not expose storage-native thumbnail, image-preview, or media-metadata capabilities
- legacy `{"presigned_upload":true}` remains compatible with object-storage presigned upload
- `allowed_types` can be managed through REST
- `driver_type = "remote"` requires `remote_node_id`
Expand Down Expand Up @@ -130,7 +132,7 @@ Failed connection tests no longer return a `StoragePolicyProbeResult` success pa
}
```

Draft test requests support optional `policy_id`. While editing a saved policy, blank sensitive fields such as `access_key` or `secret_key` can be filled from the saved policy by S3-compatible, Azure Blob, and Tencent COS connectors. Unsaved new policies must still provide complete credentials.
Draft test requests support optional `policy_id`. While editing a saved policy, blank sensitive fields such as `access_key` or `secret_key` can be filled from the saved policy by S3-compatible, SFTP, Azure Blob, and Tencent COS connectors. Unsaved new policies must still provide complete credentials.

### Storage OAuth Credentials

Expand Down
7 changes: 4 additions & 3 deletions developer-docs/en/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -148,8 +148,9 @@ WebDAV does not go through `src/api/routes/**`. Instead:
│ Infrastructure layer │
│ - SeaORM + migration │
│ - StorageConnector descriptor / action │
│ - StorageDriver(Local/S3/Azure/OneDrive) │
│ - StorageDriver(Tencent COS / Remote) │
│ - StorageDriver(Local/S3/SFTP/Azure) │
│ - StorageDriver(Tencent COS/OneDrive) │
│ - StorageDriver(Remote) │
│ - CacheBackend(Memory / Redis) │
├─────────────────────────────────────────────┤
│ Data layer │
Expand Down Expand Up @@ -190,7 +191,7 @@ The practical rule of thumb in this repository remains:
| `src/api/routes/remote_tunnel.rs` | Primary-side remote-node reverse tunnel internal entry |
| `src/services/` | Central business rule layer |
| `src/storage/connectors/` | Storage connectors: descriptors, fields, actions, connection tests, upload workflows, and credential requirements |
| `src/storage/drivers/` | Local, S3-compatible, Azure Blob, Tencent COS, OneDrive, and remote drivers |
| `src/storage/drivers/` | Local, S3-compatible, SFTP, Azure Blob, Tencent COS, OneDrive, and remote drivers |
| `src/storage/remote_protocol/tunnel/` | Reverse tunnel transport runtime, auth, registry, and streaming responses |
| `src/webdav/` | WebDAV filesystem, auth, locks, and DeltaV support |
| `frontend-panel/` | React 19 + Vite frontend; build artifacts are served by the backend |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ Shared field meanings and pure normalization helpers live in `src/storage/field_
- Admin fields, actions, capabilities, and UI helper metadata must come from backend descriptors first. The frontend must not infer connection tests, authorization, upload strategy, native processing, remote binding, or field visibility from a local `driver_type` capability matrix.
- `label_key`, `help_key`, `placeholder`, `required_message_key`, and similar fields are stable localization keys or hint parameters. The frontend owns final localized text, but field presence, required state, sensitivity, and supported actions are backend descriptor concerns.
- `secret: true` or secret field kind means the frontend must render a sensitive input and backend logs / `Debug` output must not expose plaintext. Create flows follow descriptor `required` plus backend validation. Edit flows treat omitted secret fields as "preserve the stored value"; explicit values replace the stored secret after normalization.
- `StorageConnectorFieldScope::PolicyOptions` fields belong to `storage_policy.options`. The frontend must render and normalize them from descriptors, not from a local per-driver field matrix. SFTP `sftp_host_key_fingerprint` is such a field: the backend declares the field, label key, trim behavior, and validation rule; the frontend only displays and submits it.
- Unsupported drivers must produce stable backend errors. Remote storage targets may expose only known registered drivers that the remote capability payload declares. Unknown wire-level driver ids may be preserved, but they are not locally configurable drivers.
- When descriptors, remote capabilities, or capability parsing are missing, the frontend may only fall back conservatively by hiding risky actions or showing an unavailable state. It must not recreate a local capability matrix in that fallback path.
- Action descriptors declare whether an entry point requires a saved policy, authorization credential, or remote-state mutation. Routes and services still perform final validation; hidden buttons are not authorization.
Expand All @@ -30,6 +31,8 @@ Field normalization belongs in backend use cases or connector/driver-specific pu
- Local remote storage target `base_path` uses `normalize_relative_local_target_path` through the remote target service wrapper: trim whitespace, collapse `.` segments, reject blanks, absolute paths, `..`, Windows prefixes, and backslash escapes, then resolve within `server.follower.remote_storage_target_local_root`.
- Object-storage remote storage target `base_path` is a prefix: trim whitespace and outer `/`; an empty prefix means bucket/container root.
- Storage policy object-storage endpoint/bucket normalization uses `normalize_s3_endpoint_and_bucket` plus connector-specific API error mapping. Non-empty endpoints must be `http://` or `https://` and include a hostname; bucket/container is required.
- Storage policy SFTP endpoint normalization is handled by `parse_sftp_endpoint`: it allows `sftp://host:port`, bare `host`, and `host:port`, with default port `22`; only a real `://` scheme separator triggers URL-scheme validation. Paths, query strings, fragments, and URL credentials are invalid; the remote root must use `base_path`.
- SFTP host key fingerprints live in `storage_policy.options.sftp_host_key_fingerprint`. Unknown or mismatched host keys must fail closed and expose actual / expected fingerprints through structured `SftpHostKeyRejected` context; tests must not parse error text.
- `max_file_size = 0` means no extra policy limit; negative values are invalid at the service boundary. Upload paths still perform final size checks when applying a policy or target.
- Same-driver edits preserve omitted `access_key` / `secret_key` values. Explicit replacements are trimmed and revalidated.
- When changing a remote storage target driver, old driver-specific fields must not leak into the new driver. Endpoint, bucket, access key, and secret key reset; base path follows the new driver input/default semantics and is normalized again.
Expand All @@ -48,6 +51,7 @@ When descriptor or normalization behavior changes, add focused unit tests:

- Descriptor tests must cover every built-in driver field, secret marker, action, and key capability.
- Normalization tests must cover trimming, blank values, path escapes, prefix slash trimming, negative `max_file_size`, same-driver secret preservation, explicit secret replacement, and driver-change field reset.
- SFTP coverage must include bare host, `host:port`, `sftp://host:port`, wrong schemes, host key fingerprint format, unknown-host-key rejection, and accepted pinned fingerprints.
- For storage policy descriptor behavior, run `cargo test --lib storage::connectors` or a narrower filter.
- For remote storage target normalization, run `cargo test --lib remote_storage_target_service::tests::<filter>`.
- OpenAPI schema changes require OpenAPI export and frontend SDK regeneration. This contract slice does not change public API shapes.
20 changes: 20 additions & 0 deletions developer-docs/en/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -122,3 +122,23 @@ If you suspect the test did not switch backends as expected, check these three t
1. Does the test case actually use `common::setup()`?
2. Is `ASTER_TEST_DATABASE_BACKEND` exported in the shell?
3. Is Docker available locally, and can the corresponding image start successfully?

## SFTP Integration Tests

The SFTP driver has a dedicated integration test:

```bash
cargo test --test test_sftp
```

This test starts an `atmoz/sftp` container through `testcontainers` by default and runs a real upload, download, range read, delete, and host-key fingerprint confirmation flow. It requires a local Docker / container runtime.

If the current environment cannot run Docker, disable it explicitly:

```bash
ASTER_SFTP_TEST_DOCKER=0 cargo test --test test_sftp
```

With that variable set, the container round trip is skipped. Do not make this the default CI behavior; SFTP is a real storage driver, so PRs touching the driver, connector, descriptor, or upload/download path should keep the default Docker test enabled.

`src/storage/drivers/sftp.rs` also contains a manual real-server test that requires `ASTER_SFTP_TEST_*` and `ASTER_SFTP_TEST_HOST_KEY_FINGERPRINT`. It does not replace the default Docker coverage in `tests/test_sftp.rs`; it is mainly for debugging compatibility with a specific SFTP server.
10 changes: 6 additions & 4 deletions developer-docs/zh-CN/api/admin.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,7 @@

当前实现注意点:

- `driver_type` 当前支持 `local`、`s3`、`azure_blob`、`tencent_cos`、`remote` 和 `onedrive`
- `driver_type` 当前支持 `local`、`s3`、`sftp`、`azure_blob`、`tencent_cos`、`remote` 和 `one_drive`
- `GET /admin/policies/storage-drivers` 返回 `StorageConnectorDescriptor` 列表,前端应以 descriptor 的 `capabilities`、`fields`、`upload_workflows`、`actions` 和 `credential_mode` 决定表单、连接测试、上传/下载策略和操作入口,不要在前端维护一份 driver-type 能力矩阵
- 创建和更新都会采用请求里的 `chunk_size`
- `options` 当前承载策略级行为:
Expand All @@ -103,11 +103,13 @@
- 存储原生缩略图 / 图片预览:`storage_native_processing_enabled`、`thumbnail_processor`、`thumbnail_extensions`
- 存储原生媒体元数据:`storage_native_media_metadata_enabled`、`media_metadata_extensions`
- OneDrive 位置选项:`onedrive_account_mode`、`onedrive_tenant`、`onedrive_site_id`、`onedrive_drive_id`、`onedrive_group_id`、`onedrive_root_item_id`
- SFTP 主机密钥固定:`sftp_host_key_fingerprint`
- `application_config.microsoft_graph` 用于保存 OneDrive / Microsoft Graph 应用配置;client secret 只写入加密存储,API 响应只暴露 `client_secret_configured`
- `driver_type = "azure_blob"` 使用 Azure Blob Block Blob 能力,预签名上传使用 SAS URL,前端直传时需要带 `x-ms-blob-type: BlockBlob`
- `driver_type = "onedrive"` 使用 Microsoft Graph OAuth 凭据,授权前需要先保存策略和 `application_config.microsoft_graph`
- `driver_type = "one_drive"` 使用 Microsoft Graph OAuth 凭据,授权前需要先保存策略和 `application_config.microsoft_graph`
- `driver_type = "sftp"` 使用 SSH 用户名 / 密码连接 SFTP 服务器;Endpoint 支持 `sftp://host:port`、裸 `host` 和 `host:port`,远程根目录放在 `base_path`。未知或不匹配的 SSH 主机密钥会以 `StorageErrorKind::Precondition` 拒绝,并通过诊断提示 actual / expected 指纹;确认后的指纹保存在 `options.sftp_host_key_fingerprint`。
- `driver_type = "tencent_cos"` 普通读写复用 S3-compatible 对象存储路径,会校验 Tencent COS endpoint 形态;策略启用后可通过 COS CI 暴露原生缩略图、图片预览和媒体元数据能力
- 内置 Local、S3-compatible、Azure Blob、OneDrive 和 Remote 驱动不暴露存储原生缩略图、图片预览或媒体元数据能力
- 内置 Local、S3-compatible、SFTP、Azure Blob、OneDrive 和 Remote 驱动不暴露存储原生缩略图、图片预览或媒体元数据能力
- 旧配置 `{"presigned_upload":true}` 仍兼容,等价于 S3 预签名上传策略
- `POST /admin/policies/{id}/promote-s3-driver` 当前支持把通用 `s3` 策略提升为 `tencent_cos`。请求体必须包含目标驱动和当前 endpoint / bucket,例如 `{ "target_driver_type": "tencent_cos", "endpoint": "https://bucket-1250000000.cos.ap-guangzhou.myqcloud.com", "bucket": "bucket-1250000000" }`。提升时不允许改变 bucket;若该策略还有活动上传 session,或目标驱动不能接受当前 endpoint / bucket 组合,会直接拒绝。
- REST 已经可以通过 `allowed_types` 管理策略允许的 MIME / 类型列表;不传时创建会使用空列表,更新会保持原值
Expand Down Expand Up @@ -149,7 +151,7 @@
}
```

草稿测试请求支持可选 `policy_id`。编辑已保存策略时,如果 `access_key`、`secret_key` 等敏感字段为空,S3-compatible、Azure Blob 和 Tencent COS connector 会从该策略已保存凭据补齐空白字段;新建未保存策略时仍必须传完整凭据。
草稿测试请求支持可选 `policy_id`。编辑已保存策略时,如果 `access_key`、`secret_key` 等敏感字段为空,S3-compatible、SFTP、Azure Blob 和 Tencent COS connector 会从该策略已保存凭据补齐空白字段;新建未保存策略时仍必须传完整凭据。

### 存储 OAuth 凭据

Expand Down
7 changes: 4 additions & 3 deletions developer-docs/zh-CN/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -148,8 +148,9 @@ WebDAV 不走 `src/api/routes/**`,而是:
│ 基础设施层 │
│ - SeaORM + migration │
│ - StorageConnector descriptor / action │
│ - StorageDriver(Local/S3/Azure/OneDrive) │
│ - StorageDriver(Tencent COS / Remote) │
│ - StorageDriver(Local/S3/SFTP/Azure) │
│ - StorageDriver(Tencent COS/OneDrive) │
│ - StorageDriver(Remote) │
│ - CacheBackend(Memory / Redis) │
├─────────────────────────────────────────────┤
│ 数据层 │
Expand Down Expand Up @@ -190,7 +191,7 @@ WebDAV 不走 `src/api/routes/**`,而是:
| `src/api/routes/remote_tunnel.rs` | primary 侧远端节点 reverse tunnel 内部入口 |
| `src/services/` | 业务规则集中层 |
| `src/storage/connectors/` | 存储 connector:descriptor、字段、action、连接测试、上传工作流和凭据需求 |
| `src/storage/drivers/` | 本地、S3-compatible、Azure Blob、Tencent COS、OneDrive 和远端驱动 |
| `src/storage/drivers/` | 本地、S3-compatible、SFTP、Azure Blob、Tencent COS、OneDrive 和远端驱动 |
| `src/storage/remote_protocol/tunnel/` | reverse tunnel 传输运行时、鉴权、注册表和流式响应 |
| `src/webdav/` | WebDAV 文件系统、认证、锁与 DeltaV 支持 |
| `frontend-panel/` | React 19 + Vite 前端,构建产物由后端服务 |
Expand Down
Loading
Loading