diff --git a/README.md b/README.md index 9e05281..f8d141e 100644 --- a/README.md +++ b/README.md @@ -9,7 +9,7 @@ **English | [简体中文](README_zh-CN.md)** -A tiny object storage SDK focused on uploading: Aliyun OSS, Tencent Cloud COS, Huawei Cloud OBS, AWS S3 (plus S3-compatible stores) and Azure Blob Storage under one core API; runs in browsers, Node.js, Service Workers and WeChat mini programs; extensible with custom providers. About 10kb (min+gzipped) for the full entry — tree-shaking drops the operations you don't import, so a bundle that only calls `put` is smaller. +A tiny object storage SDK focused on uploading: Aliyun OSS, Tencent Cloud COS, Huawei Cloud OBS, Volcano Engine TOS, AWS S3 (plus S3-compatible stores) and Azure Blob Storage under one core API; runs in browsers, Node.js, Service Workers and WeChat mini programs; extensible with custom providers. About 10kb (min+gzipped) for the full entry — tree-shaking drops the operations you don't import, so a bundle that only calls `put` is smaller. **Upgrading from 0.x? See the [upgrade guide](UPGRADING.md).** @@ -28,6 +28,7 @@ A tiny object storage SDK focused on uploading: Aliyun OSS, Tencent Cloud COS, H - [AWS S3](#aws-s3) - [Tencent Cloud COS](#tencent-cloud-cos) - [Huawei Cloud OBS](#huawei-cloud-obs) + - [Volcano Engine TOS](#volcano-engine-tos) - [Azure Blob Storage](#azure-blob-storage) - [Extension](#extension) - [Composing a custom provider](#composing-a-custom-provider) @@ -141,6 +142,7 @@ Some providers can call back your server after an object is stored, then relay t | Aliyun OSS | ✅ `callback` (fired on complete) | `x-oss-callback` / `x-oss-callback-var` headers, base64 JSON — ali-oss compatible | | Huawei OBS | ✅ `callback` (fired on complete) | `x-obs-callback` header, base64 JSON — esdk-obs-browserjs compatible; `customValue` is not supported | | Tencent COS | via `headers` only | official SDK passes the callback header value through verbatim, so pass `headers` such as `{ 'x-cos-callback': '…' }` yourself; the value format is defined by the COS server API | +| Volcano Engine TOS | via `headers` only | pass `headers` such as `{ 'x-tos-callback': '…' }` (and `x-tos-callback-var`); the value format is defined by the TOS server API | | AWS S3 / Azure Blob | ❌ | no callback API | ```js @@ -173,6 +175,7 @@ The `Protocol` interface (`tiny-oss/protocol`): | `copySourceHeader` / `copySourceRangeHeader` | Header names for `uploadPartCopy` | | `listUploadsMarkerKey` | Query key for the list-uploads marker (`'marker'` OSS-style, `'key-marker'` S3-style) | | `supportsSymlink` | Whether `putSymlink` is exported (`false` when the provider has no symlink API) | +| `symlinkHeaders` | Optional: serialize the `putSymlink` target into headers (defaults to OSS's `x-oss-symlink-target` with an encodeURI'd target) | `request` receives `{ verb, objectName, contentMd5, headers, subResource, data, timeout, onprogress }`; `subResource` is the query-parameter map the operations build (`{ uploads: '' }`, `{ partNumber, uploadId }`, …) — the request implementation decides which of them participate in the signature. @@ -413,6 +416,49 @@ Notes: - The OBS signer uses the OBS "obs" signature scheme, matching the official `esdk-obs-browserjs` byte for byte. - OBS endpoints only serve HTTPS, so the SDK defaults `secure` to `true`; keep the default unless you connect to a custom HTTP endpoint. +### Volcano Engine TOS + +The same operations are also available for Volcano Engine Torch Object Storage (TOS) through a dedicated entry point (`tiny-oss/tos`). Each entry is self-contained: importing only what you use keeps the OSS bundle free of COS/OBS/TOS signing code and vice versa. + +```js +import { put, multipartUpload, signatureUrl } from 'tiny-oss/tos'; + +put( + { + accessKeyId: 'your Access Key ID', + accessKeySecret: 'your Secret Access Key', + // Recommend to use the stsToken option in browser + stsToken: 'security token', + region: 'cn-beijing', + bucket: 'your-bucket' + }, + 'hello-world', + blob +); +``` + +The TOS entry exports everything the OSS entry does, including `putSymlink` (TOS has a symlink API). Options: + +| option | type | description | +|---|---|---| +| `accessKeyId` | `string` | Volcano Engine Access Key ID | +| `accessKeySecret` | `string` | Volcano Engine Secret Access Key | +| `stsToken` | `string` | temporary-credential SecurityToken (`x-tos-security-token`) | +| `region` | `string` | e.g. `cn-beijing`, `ap-southeast-1`; always required — the TOS4 credential scope embeds it | +| `bucket` | `string` | plain bucket name | +| `endpoint` | `string` | endpoint **domain** the bucket is prefixed to (`.`), e.g. `tos-cn-beijing.volces.com`; unlike the other entries it is not the full host, because TOS only supports virtual-hosted addressing | +| `internal` | `boolean` | use the Volcengine internal network domain (`tos-.ivolces.com`), default `false` | +| `secure` | `boolean` | use HTTPS (`true`) or HTTP (`false`), default `true` | +| `timeout` | `string \| number` | instance-level timeout for all operations, default 60s | + +Notes: + +- The signer implements `TOS4-HMAC-SHA256` (service `tos`, `host` + `x-tos-*` signed headers, `UNSIGNED-PAYLOAD` body), byte-identical to `@volcengine/tos-sdk`; `test/tos-oracle.node.ts` pins it. +- TOS native endpoints are virtual-hosted only and reject path-style addressing, so there is no `pathStyle` option. TOS's S3-compatible endpoints (`tos-s3-.volces.com`) need AWS Signature V4 with virtual-hosted addressing, which this package does not implement — use the native endpoints above. +- `signatureUrl` returns a `TOS4-HMAC-SHA256` query-signed URL (`X-Tos-*` parameters, TTL `X-Tos-Expires`). The credential scope uses `region`; the official JS SDK's `getPreSignedUrl` substitutes the endpoint there, which disagrees with the official Go SDK. +- Callbacks: pass `x-tos-callback` / `x-tos-callback-var` through `headers` (like COS); the structured `callback` option is OSS/OBS-only. +- Browser uploads to TOS require the bucket's CORS rule to allow your origin and expose the `ETag` response header for multipart uploads; temporary credentials (STS) are recommended over permanent keys. + ### Azure Blob Storage Azure Blob Storage speaks neither SigV4 nor any of the other schemes above: it uses its own **SharedKey** authorization and a different multipart model (block blobs). A dedicated entry point (`tiny-oss/azure`) implements both, so the API stays the same: @@ -455,7 +501,7 @@ Notes: ## Extension -Every operation is a factory over a `Protocol` — the extension point. A provider only has to implement two functions (`request`, `signUrl`) and fill in five constants; all operations (`put`, multipart, list, copy, …) then work unchanged. The built-in providers are the reference recipes: `src/cos/`, `src/obs/`, `src/aws/` (S3-shaped, each with its own signer) and `src/azure/` (non-S3-shaped — see the [Protocol](#protocol) section for the interface). +Every operation is a factory over a `Protocol` — the extension point. A provider only has to implement two functions (`request`, `signUrl`) and fill in five constants; all operations (`put`, multipart, list, copy, …) then work unchanged. The built-in providers are the reference recipes: `src/cos/`, `src/obs/`, `src/tos/`, `src/aws/` (S3-shaped, each with its own signer) and `src/azure/` (non-S3-shaped — see the [Protocol](#protocol) section for the interface). ### Composing a custom provider @@ -500,7 +546,7 @@ export { put, multipartUpload, signatureUrl: myProtocol.signUrl }; ### Contributing a provider to the repo -Follow the `src/aws/` layout: `src//{signature,host,request,signatureUrl,index}.ts`, then add an entry to `tsup.config.ts` (the one build emits both the `.es.js` bundle and the matching `.es.d.ts`) and a `package.json` `exports` entry. Signing must match the official SDK — the tests in `test/cos-signature.spec.ts`, `test/obs-signature.spec.ts` and `test/aws-signature.spec.ts` pin each signer against its official SDK as an oracle. +Follow the `src/aws/` layout: `src//{signature,host,request,signatureUrl,index}.ts`, then add an entry to `tsup.config.ts` (the one build emits both the `.es.js` bundle and the matching `.es.d.ts`) and a `package.json` `exports` entry. Signing must match the official SDK — the tests in `test/cos-signature.spec.ts`, `test/obs-signature.spec.ts` and `test/aws-signature.spec.ts` pin each signer against its official SDK as an oracle (`test/tos-oracle.node.ts`, run by `pnpm test:tos-oracle`, does the same for TOS against `@volcengine/tos-sdk`). If the target storage's multipart API is not S3-shaped (e.g. Azure's block blobs), don't force it through `createInitMultipartUpload`/`createUploadPart`/`createCompleteMultipartUpload`: write provider-specific primitives with the same signatures and inject them via `createMultipartUpload` (see `src/azure/multipart.ts`). Operations that have no counterpart — like `listUploads` for Azure — are simply omitted from the entry. diff --git a/README_zh-CN.md b/README_zh-CN.md index 18ea5bb..7d2a2c9 100644 --- a/README_zh-CN.md +++ b/README_zh-CN.md @@ -7,7 +7,7 @@ [![bundle size](https://img.shields.io/bundlephobia/minzip/tiny-oss)](https://bundlephobia.com/package/tiny-oss) [![node](https://img.shields.io/node/v/tiny-oss)](https://www.npmjs.com/package/tiny-oss) -专注于上传的极简对象存储 SDK：同一套核心 API 覆盖阿里云 OSS、腾讯云 COS、华为云 OBS、AWS S3（含 S3 兼容存储）与 Azure Blob Storage；支持浏览器、Node.js、Service Worker 与微信小程序；可通过自定义 provider 扩展。完整入口约 10kb (min+gzipped)——按需导入时 tree-shaking 会剔除未使用的操作，产物更小。 +专注于上传的极简对象存储 SDK：同一套核心 API 覆盖阿里云 OSS、腾讯云 COS、华为云 OBS、火山引擎 TOS、AWS S3（含 S3 兼容存储）与 Azure Blob Storage；支持浏览器、Node.js、Service Worker 与微信小程序；可通过自定义 provider 扩展。完整入口约 10kb (min+gzipped)——按需导入时 tree-shaking 会剔除未使用的操作，产物更小。 **[English](README.md) | 简体中文** @@ -26,6 +26,7 @@ - [AWS S3](#aws-s3) - [腾讯云 COS](#腾讯云-cos) - [华为云 OBS](#华为云-obs) + - [火山引擎 TOS](#火山引擎-tos) - [Azure Blob Storage](#azure-blob-storage) - [扩展](#扩展) - [组装自定义 provider](#组装自定义-provider) @@ -138,6 +139,7 @@ put( | 阿里云 OSS | ✅ 结构化 `callback`（在 complete 阶段触发） | `x-oss-callback` / `x-oss-callback-var` 请求头，base64 JSON，与 ali-oss 一致 | | 华为云 OBS | ✅ 结构化 `callback`（在 complete 阶段触发） | `x-obs-callback` 请求头，base64 JSON，与 esdk-obs-browserjs 一致；不支持 `customValue` | | 腾讯云 COS | 仅通过 `headers` | 官方 SDK 将回调头原样透传，请自行传 `headers`，如 `{ 'x-cos-callback': '…' }`；值格式由 COS 服务端 API 定义 | +| 火山引擎 TOS | 仅通过 `headers` | 请自行传 `headers`，如 `{ 'x-tos-callback': '…' }`（以及 `x-tos-callback-var`）；值格式由 TOS 服务端 API 定义 | | AWS S3 / Azure Blob | ❌ | 无回调能力 | ```js @@ -170,6 +172,7 @@ put(options, 'avatar.jpg', blob, { | `copySourceHeader` / `copySourceRangeHeader` | `uploadPartCopy` 用的拷贝源头名 | | `listUploadsMarkerKey` | 分页 marker 的 query 键（OSS 风格 `'marker'`、S3 风格 `'key-marker'`） | | `supportsSymlink` | 是否导出 `putSymlink`（无软链接接口时置 `false`） | +| `symlinkHeaders` | 可选：把 `putSymlink` 的目标序列化为请求头（缺省为 OSS 的 `x-oss-symlink-target`，目标值经 encodeURI） | `request` 收到 `{ verb, objectName, contentMd5, headers, subResource, data, timeout, onprogress }`；`subResource` 是操作拼好的 query 参数表（如 `{ uploads: '' }`、`{ partNumber, uploadId }`）——哪些参数参与签名由 request 实现决定。 @@ -401,6 +404,49 @@ OBS 入口导出与 OSS 入口相同的全部函数，唯独没有 `putSymlink` - OBS 签名器使用 OBS 的 "obs" 签名方案，与官方 `esdk-obs-browserjs` 逐字节一致。 - OBS 端点仅支持 HTTPS，因此 SDK 默认将 `secure` 设为 `true`；除非连接自定义 HTTP 端点，否则请保持默认。 +### 火山引擎 TOS + +同一套操作通过独立入口支持火山引擎对象存储 TOS（`tiny-oss/tos`）。每个入口自包含：按需导入即可让 OSS 产物不携带 COS/OBS/TOS 签名代码（反之亦然）。 + +```js +import { put, multipartUpload, signatureUrl } from 'tiny-oss/tos'; + +put( + { + accessKeyId: '你的 Access Key ID', + accessKeySecret: '你的 Secret Access Key', + // 推荐在浏览器端使用 stsToken 参数 + stsToken: 'security token', + region: 'cn-beijing', + bucket: 'your-bucket' + }, + 'hello-world', + blob +); +``` + +TOS 入口导出与 OSS 入口相同的全部函数，包含 `putSymlink`（TOS 有软链接接口）。options： + +| option | 类型 | 说明 | +|---|---|---| +| `accessKeyId` | `string` | 火山引擎 Access Key ID | +| `accessKeySecret` | `string` | 火山引擎 Secret Access Key | +| `stsToken` | `string` | 临时密钥 SecurityToken（`x-tos-security-token`） | +| `region` | `string` | 如 `cn-beijing`、`ap-southeast-1`；始终必填——TOS4 的凭证作用域会带上它 | +| `bucket` | `string` | 普通 bucket 名 | +| `endpoint` | `string` | bucket 前缀所在的**端点域名**（`.`），如 `tos-cn-beijing.volces.com`；与其他入口不同，它不是完整 host，因为 TOS 仅支持虚拟主机（virtual-hosted）寻址 | +| `internal` | `boolean` | 使用火山引擎内网域名（`tos-.ivolces.com`），默认 `false` | +| `secure` | `boolean` | 使用 HTTPS（`true`）还是 HTTP（`false`），默认 `true` | +| `timeout` | `string \| number` | 所有操作的实例级超时，默认 60s | + +注意事项： + +- 签名器实现 `TOS4-HMAC-SHA256`（service `tos`，签 `host` 与 `x-tos-*` 头，body 为 `UNSIGNED-PAYLOAD`），与官方 `@volcengine/tos-sdk` 逐字节一致；`test/tos-oracle.node.ts`（`pnpm test:tos-oracle`）持续钉死。 +- TOS 原生端点仅支持虚拟主机寻址、拒绝路径样式，因此没有 `pathStyle` 选项。TOS 的 S3 兼容端点（`tos-s3-.volces.com`）需要 AWS Signature V4 加虚拟主机寻址，本包未实现——请使用上面的原生端点。 +- `signatureUrl` 返回 `TOS4-HMAC-SHA256` query 签名 URL（`X-Tos-*` 参数，有效期用 `X-Tos-Expires`）。凭证作用域使用 `region`；官方 JS SDK 的 `getPreSignedUrl` 在此填的是 endpoint，与官方 Go SDK 不一致。 +- 回调：像 COS 一样通过 `headers` 传 `x-tos-callback` / `x-tos-callback-var`；结构化 `callback` 选项仅 OSS/OBS 支持。 +- 浏览器上传 TOS 需要在存储桶配置跨域规则（允许你的站点并暴露 `ETag` 响应头以支持分片上传），推荐使用临时密钥（STS）而非永久密钥。 + ### Azure Blob Storage Azure Blob Storage 不使用 SigV4，也不使用以上任何签名方案：它使用自成一派的 **SharedKey** 授权，分片模型也不同（Block Blob）。专用入口（`tiny-oss/azure`）实现了这两点，API 保持一致： @@ -443,7 +489,7 @@ Azure 入口导出 `put`、`signatureUrl`、`initMultipartUpload`、`uploadPart` ## 扩展 -每个操作都是基于 `Protocol` 的工厂——这就是扩展点。接入一个新存储只需要实现两个函数（`request`、`signUrl`）并填写五个常量，所有操作（上传、分片、列举、拷贝……）即全部可用。内置实现是最佳参考配方：`src/cos/`、`src/obs/`、`src/aws/`（S3 形态，各自带签名器）与 `src/azure/`（非 S3 形态——接口见[协议](#协议)章节）。 +每个操作都是基于 `Protocol` 的工厂——这就是扩展点。接入一个新存储只需要实现两个函数（`request`、`signUrl`）并填写五个常量，所有操作（上传、分片、列举、拷贝……）即全部可用。内置实现是最佳参考配方：`src/cos/`、`src/obs/`、`src/tos/`、`src/aws/`（S3 形态，各自带签名器）与 `src/azure/`（非 S3 形态——接口见[协议](#协议)章节）。 ### 组装自定义 provider @@ -488,7 +534,7 @@ export { put, multipartUpload, signatureUrl: myProtocol.signUrl }; ### 向仓库贡献 provider -参照 `src/aws/` 布局：`src//{signature,host,request,signatureUrl,index}.ts`，然后在 `tsup.config.ts` 加一个入口（一次构建同时产出 `.es.js` 产物与配套的 `.es.d.ts`），并补 `package.json` 的 `exports` 条目。签名必须与官方 SDK 对齐——`test/cos-signature.spec.ts`、`test/obs-signature.spec.ts`、`test/aws-signature.spec.ts` 用各自官方 SDK 作 oracle 钉死签名器。 +参照 `src/aws/` 布局：`src//{signature,host,request,signatureUrl,index}.ts`，然后在 `tsup.config.ts` 加一个入口（一次构建同时产出 `.es.js` 产物与配套的 `.es.d.ts`），并补 `package.json` 的 `exports` 条目。签名必须与官方 SDK 对齐——`test/cos-signature.spec.ts`、`test/obs-signature.spec.ts`、`test/aws-signature.spec.ts` 用各自官方 SDK 作 oracle 钉死签名器（TOS 同理：`pnpm test:tos-oracle` 用 `@volcengine/tos-sdk` 作 oracle）。 如果目标存储的分片接口不是 S3 形态（如 Azure 的 Block Blob），不要强行套 `createInitMultipartUpload`/`createUploadPart`/`createCompleteMultipartUpload`：写同签名的 provider 专用原语，经 `createMultipartUpload` 注入（见 `src/azure/multipart.ts`）。没有对应 API 的操作（如 Azure 的 `listUploads`）直接从入口省略。 diff --git a/package.json b/package.json index 97c3d7d..100c4be 100644 --- a/package.json +++ b/package.json @@ -18,6 +18,10 @@ "types": "./dist/tiny-oss.obs.es.d.ts", "import": "./dist/tiny-oss.obs.es.js" }, + "./tos": { + "types": "./dist/tiny-oss.tos.es.d.ts", + "import": "./dist/tiny-oss.tos.es.js" + }, "./aws": { "types": "./dist/tiny-oss.aws.es.d.ts", "import": "./dist/tiny-oss.aws.es.js" @@ -43,14 +47,15 @@ "lint": "oxlint src test", "fmt": "oxfmt src test", "fmt:check": "oxfmt --check src test", - "build": "tsup && npm run build:types:default && npm run build:types:cos && npm run build:types:obs && npm run build:types:aws && npm run build:types:azure && npm run build:types:protocol", + "build": "tsup && npm run build:types:default && npm run build:types:cos && npm run build:types:obs && npm run build:types:tos && npm run build:types:aws && npm run build:types:azure && npm run build:types:protocol", "build:types:default": "dts-bundle-generator src/index.ts -o dist/tiny-oss.es.d.ts --project tsconfig.json --sort", "build:types:cos": "dts-bundle-generator src/cos/index.ts -o dist/tiny-oss.cos.es.d.ts --project tsconfig.json --sort", "build:types:obs": "dts-bundle-generator src/obs/index.ts -o dist/tiny-oss.obs.es.d.ts --project tsconfig.json --sort", + "build:types:tos": "dts-bundle-generator src/tos/index.ts -o dist/tiny-oss.tos.es.d.ts --project tsconfig.json --sort", "build:types:aws": "dts-bundle-generator src/aws/index.ts -o dist/tiny-oss.aws.es.d.ts --project tsconfig.json --sort", "build:types:azure": "dts-bundle-generator src/azure/index.ts -o dist/tiny-oss.azure.es.d.ts --project tsconfig.json --sort", "build:types:protocol": "dts-bundle-generator src/provider.ts -o dist/tiny-oss.protocol.es.d.ts --project tsconfig.json --sort", - "check:types": "tsc --noEmit --skipLibCheck test/test-types.ts && tsc --noEmit --skipLibCheck test/test-types-cos.ts && tsc --noEmit --skipLibCheck test/test-types-obs.ts && tsc --noEmit --skipLibCheck test/test-types-aws.ts && tsc --noEmit --skipLibCheck test/test-types-azure.ts && tsc --noEmit --skipLibCheck test/test-types-protocol.ts", + "check:types": "tsc --noEmit --skipLibCheck test/test-types.ts && tsc --noEmit --skipLibCheck test/test-types-cos.ts && tsc --noEmit --skipLibCheck test/test-types-obs.ts && tsc --noEmit --skipLibCheck test/test-types-tos.ts && tsc --noEmit --skipLibCheck test/test-types-aws.ts && tsc --noEmit --skipLibCheck test/test-types-azure.ts && tsc --noEmit --skipLibCheck test/test-types-protocol.ts", "changelog": "changelogen", "release": "bumpp", "dev": "vite", @@ -58,9 +63,10 @@ "serve": "tsx test/server.ts", "test": "vitest", "test:azure-oracle": "tsx test/azure-oracle.node.ts", + "test:tos-oracle": "tsx test/tos-oracle.node.ts", "test:coverage": "vitest --coverage", - "test:ci": "pnpm test:azure-oracle && vitest run --browser.headless", - "test:ci:coverage": "pnpm test:azure-oracle && vitest run --browser.headless --coverage", + "test:ci": "pnpm test:azure-oracle && pnpm test:tos-oracle && vitest run --browser.headless", + "test:ci:coverage": "pnpm test:azure-oracle && pnpm test:tos-oracle && vitest run --browser.headless --coverage", "prepare": "husky" }, "repository": { @@ -78,6 +84,8 @@ "oss", "cos", "obs", + "tos", + "volcengine", "s3", "aws", "azure", @@ -99,6 +107,7 @@ "@types/node": "^25.0.9", "@vitest/browser-playwright": "^4.1.11", "@vitest/coverage-v8": "4.1.11", + "@volcengine/tos-sdk": "2.9.1", "ali-oss": "^6.23.0", "aws-sdk": "2.1693.0", "bumpp": "^12.2.2", diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index d22fa09..c7013e2 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -47,6 +47,9 @@ importers: '@vitest/coverage-v8': specifier: 4.1.11 version: 4.1.11(@vitest/browser@4.1.11)(vitest@4.1.11) + '@volcengine/tos-sdk': + specifier: 2.9.1 + version: 2.9.1(supports-color@7.2.0) ali-oss: specifier: ^6.23.0 version: 6.23.0(supports-color@7.2.0) @@ -752,6 +755,9 @@ packages: '@types/estree@1.0.9': resolution: {integrity: sha512-GhdPgy1el4/ImP05X05Uw4cw2/M93BCUmnEvWZNStlCzEKME4Fkk+YpoA5OiHNQmoS7Cafb8Xa3Pya8m1Qrzeg==} + '@types/http-proxy@1.17.17': + resolution: {integrity: sha512-ED6LB+Z1AVylNTu7hdzuBqOgMnvG/ld6wGCG8wFnAzKX5uyW2K3WD52v0gnLCTK/VLpXtKckgWuyScYK6cSPaw==} + '@types/node@25.0.9': resolution: {integrity: sha512-/rpCXHlCWeqClNBwUhDcusJxXYDjZTyE8v5oTO7WbL8eij2nKhUeU89/6xgjU7N4/Vh3He0BtyhJdQbDyhiXAw==} @@ -808,6 +814,10 @@ packages: '@vitest/utils@4.1.11': resolution: {integrity: sha512-zTCVGpyFsGWBhllOyKlTw/vnr6D9qxsfSDyfbyZmTyjHw5N/VuvzHpHoQjm2ZJzn4RJgx5w4r7V0er69CmLgPQ==} + '@volcengine/tos-sdk@2.9.1': + resolution: {integrity: sha512-hmrBzl7/LpJYezGWytPg6CZk5lvFgmDKUVZJLM3vwZgp3UKSZwgxFA7V0QmLnerPDRpLPAqh0bP8qUBSVHfmhA==} + engines: {node: '>=10'} + acorn@8.18.0: resolution: {integrity: sha512-lGq+9yr1/GuAWaVYIHRjvvySG5/4VfKIvC8EWxStPdcDh/Ka7FG3twP6v4d5BkravUilhIAsG4Qj83t02LWUPQ==} engines: {node: '>=0.4.0'} @@ -877,6 +887,9 @@ packages: engines: {node: '>= 10.0.0'} deprecated: The AWS SDK for JavaScript (v2) has reached end-of-support, and no longer receives updates. Please migrate your code to use AWS SDK for JavaScript (v3). More info https://a.co/cUPnyil + axios-adapter-uniapp@0.1.4: + resolution: {integrity: sha512-4je5JcWGrrTjPEJXVXJZnOkv+BsnYn/fKbQmjxzdCGFyoQw1gq3tfQ4/WhLzy+Gi9cQJl3K8EH26G7U0BK3wcw==} + axios@0.33.0: resolution: {integrity: sha512-juz19g7B3UWT9r3+tgRFQf2Bdy6IKDVroiyuVOUrkILVKHpqHL++K1l8ybvfdEP9B/VE72vk8vllbnedVCm/og==} @@ -1180,6 +1193,9 @@ packages: estree-walker@3.0.3: resolution: {integrity: sha512-7RUKfXgSMMkzt6ZuXmqapOurLGPPfgj6l9uRZ7lRGolvk0y2yocc35LdcxKC5PQZdn2DMqioAQ2NoWcrTKmm6g==} + eventemitter3@4.0.7: + resolution: {integrity: sha512-8guHBZCwKnFhYdHr2ysuRWErTwhoN2X8XELRlrRwpmfeY2jjuUN4taQMsULKUVo1K4DvZl+0pgfyoysHxvmvEw==} + eventemitter3@5.0.4: resolution: {integrity: sha512-mlsTRyGaPBjPedk6Bvw+aqbsXDtoAyAzm5MO7JgU+yVRyMQ5O8bD4Kcci7BS85f93veegeCPkL8R4GLClnjLFw==} @@ -1315,10 +1331,6 @@ packages: resolution: {integrity: sha512-NqADB8VjPFLM2V0VvHUewwwsw0ZWBaIdgo+ieHtK3hasLz4qeCRjYcqfB6AQrBggRKppKF8L52/VqdVsO47Dlw==} engines: {node: '>= 0.4'} - hasown@2.0.2: - resolution: {integrity: sha512-0hJU9SCPvmMzIBdZFqNPXWa6dqh7WdH0cII9y+CyS8rG3nL48Bclra9HmKhVVUHyPWNH5Y7xDwAB7bfgSjkUMQ==} - engines: {node: '>= 0.4'} - hasown@2.0.4: resolution: {integrity: sha512-T2UbfbBEF32wiepXIsMlTW9+dDYC6wMh/t/vYA4tuOMKqWz/n3vr1NFSxQiyP+zk2mXsoMA/i/7qV6LKut1t1A==} engines: {node: '>= 0.4'} @@ -1338,6 +1350,19 @@ packages: resolution: {integrity: sha512-T1gkAiYYDWYx3V5Bmyu7HcfcvL7mUrTWiM6yOfa3PIphViJ/gFPbvidQ+veqSOHci/PxBcDabeUNCzpOODJZig==} engines: {node: '>= 14'} + http-proxy-middleware@2.0.10: + resolution: {integrity: sha512-RKzRWNPxUZqbuk3BC5mGVJbBnWgr+diEnjJexIOytFbBzDy88Fbh/YvBr3DsNrl1jYAfjWfpATEv0NO35FDuPQ==} + engines: {node: '>=12.0.0'} + peerDependencies: + '@types/express': ^4.17.13 + peerDependenciesMeta: + '@types/express': + optional: true + + http-proxy@1.18.1: + resolution: {integrity: sha512-7mz/721AbnJwIVbnaSv1Cz3Am0ZLT/UBwkC92VlxhXv/k/BBQfM2fXElQNC27BVGr0uwUpplYPQM9LnaBMR5NQ==} + engines: {node: '>=8.0.0'} + https-proxy-agent@7.0.6: resolution: {integrity: sha512-vK9P5/iUfdl95AI+JVyUuIcVtd4ofvtrOr3HNtM2yxC9bnMbEdp3x01OhQNnjb8IJYi38VlTE3mBXwcfvywuSw==} engines: {node: '>= 14'} @@ -1380,6 +1405,10 @@ packages: resolution: {integrity: sha512-5BMULNob1vgFX6EjQw5izWDxrecWK9AM72rugNr0TFldMOi0fj6Jk+zeKIt0xGj4cEfQIJth4w3OKWOJ4f+AFw==} engines: {node: '>=0.10.0'} + is-extglob@2.1.1: + resolution: {integrity: sha512-SbKbANkN603Vi4jEZv49LeVJMn4yGwsbzZworEoyEiutsN3nJYdbO36zfhGJ6QEDpOZIFkDtnq5JRxmvl3jsoQ==} + engines: {node: '>=0.10.0'} + is-fullwidth-code-point@3.0.0: resolution: {integrity: sha512-zymm5+u+sCsSWyD9qNaejV3DFvhCKclKdizYaJUuHA83RLjb7nSuGnddCHGv0hk+KY7BMAlsWeK4Ueg6EV6XQg==} engines: {node: '>=8'} @@ -1392,6 +1421,10 @@ packages: resolution: {integrity: sha512-upqt1SkGkODW9tsGNG5mtXTXtECizwtS2kA161M+gJPc1xdb/Ax629af6YrTwcOeQHbewrPNlE5Dx7kzvXTizA==} engines: {node: '>= 0.4'} + is-glob@4.0.3: + resolution: {integrity: sha512-xelSayHH36ZgE7ZWhli7pW34hNbNl8Ojv5KVmkJD4hBdD3th8Tfk9vYasLM+mXWOZhFkgZfxhLSnrwRr4elSSg==} + engines: {node: '>=0.10.0'} + is-inside-container@1.0.0: resolution: {integrity: sha512-KIYLCCJghfHZxqjYBE7rEy0OBuTd5xCHS7tHVgvCLkx7StIoaxwNW3hCALgEUjFfeRk+MG/Qxmp/vtETEF3tRA==} engines: {node: '>=14.16'} @@ -1401,6 +1434,10 @@ packages: resolution: {integrity: sha512-41Cifkg6e8TylSpdtTpeLVMqvSBEVzTttHvERD741+pnZ8ANv0004MRL43QKPDlK9cGvNp6NZWZUBlbGXYxxng==} engines: {node: '>=0.12.0'} + is-plain-obj@3.0.0: + resolution: {integrity: sha512-gwsOE28k+23GP1B6vFl1oVh/WOzmawBrKwo5Ev6wMKzPkaXaCDIQKzLnvsA42DRlbVTWorkgTKIviAKCWkfUwA==} + engines: {node: '>=10'} + is-potential-custom-element-name@1.0.1: resolution: {integrity: sha512-bCYeRA2rVibKZd+s2625gGnGF/t7DSqDs4dP7CrLA1m7jKWz6pps0LpYLJN8Q64HtmPKJ1hrN3nzPNKFEKOUiQ==} @@ -1805,6 +1842,9 @@ packages: resolution: {integrity: sha512-Xf0nWe6RseziFMu+Ap9biiUbmplq6S9/p+7w7YXP/JBHhrUDDUhwa+vANyubuqfZWTveU//DYVGsDG7RKL/vEw==} engines: {node: '>=0.10.0'} + requires-port@1.0.0: + resolution: {integrity: sha512-KigOCHcocU3XODJxsu8i/j8T9tzT4adHiecwORRQ0ZZFcp7ahwXuRU1m+yuO90C5ZUyGeGfocHDI14M3L3yDAQ==} + resolve-from@5.0.0: resolution: {integrity: sha512-qYg9KP24dD5qka9J47d0aVky0N+b4fTU89LN9iDnjB5waksiC49rvMB0PrUJQGoTmH50XPiqOvAjDfaijGxYZw==} engines: {node: '>=8'} @@ -2031,6 +2071,9 @@ packages: resolution: {integrity: sha512-65P7iz6X5yEr1cwcgvQxbbIw7Uk3gOy5dIdtZ4rDveLqhrdJP+Li/Hx6tyK0NEb+2GCyneCMJiGqrADCSNk8sQ==} engines: {node: '>=8.0'} + tos-crc64-js@0.0.1: + resolution: {integrity: sha512-l2Ndi9BaDH3gmAJ5VIS/1WeqFcucK41WCbw5dije24k2XtUCclYfvhntWIVHe+0S85QcED4vjqm3AO1izlQ30g==} + totalist@3.0.1: resolution: {integrity: sha512-sf4i37nQ2LBx4m3wB74y+ubopq6W/dIzXg0FDGjsYnZHVa1Da8FH853wlL2gtUhg+xJXjfk3kUZS3BRoQeoQBQ==} engines: {node: '>=6'} @@ -2077,6 +2120,10 @@ packages: engines: {node: '>=18.0.0'} hasBin: true + type-fest@1.4.0: + resolution: {integrity: sha512-yGSza74xk0UG8k+pLh5oeoYirvIiWo5t0/o3zHHAO2tRDiZcxWP7fywNlXhqb6/r6sWvwi+RsyQMWhVLe4BVuA==} + engines: {node: '>=10'} + typescript@5.9.3: resolution: {integrity: sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw==} engines: {node: '>=14.17'} @@ -2775,6 +2822,10 @@ snapshots: '@types/estree@1.0.9': {} + '@types/http-proxy@1.17.17': + dependencies: + '@types/node': 25.0.9 + '@types/node@25.0.9': dependencies: undici-types: 7.16.0 @@ -2874,6 +2925,21 @@ snapshots: convert-source-map: 2.0.0 tinyrainbow: 3.1.1 + '@volcengine/tos-sdk@2.9.1(supports-color@7.2.0)': + dependencies: + axios: 0.33.0(debug@4.4.3(supports-color@7.2.0)) + axios-adapter-uniapp: 0.1.4(debug@4.4.3(supports-color@7.2.0)) + crypto-js: 4.2.0 + debug: 4.4.3(supports-color@7.2.0) + http-proxy-middleware: 2.0.10(debug@4.4.3(supports-color@7.2.0)) + lodash: 4.18.1 + qs: 6.16.0 + tos-crc64-js: 0.0.1 + type-fest: 1.4.0 + transitivePeerDependencies: + - '@types/express' + - supports-color + acorn@8.18.0: {} address@1.2.2: {} @@ -2962,6 +3028,12 @@ snapshots: uuid: 11.1.1 xml2js: 0.6.2 + axios-adapter-uniapp@0.1.4(debug@4.4.3(supports-color@7.2.0)): + dependencies: + axios: 0.33.0(debug@4.4.3(supports-color@7.2.0)) + transitivePeerDependencies: + - debug + axios@0.33.0(debug@4.4.3(supports-color@7.2.0)): dependencies: follow-redirects: 1.16.0(debug@4.4.3(supports-color@7.2.0)) @@ -3296,6 +3368,8 @@ snapshots: dependencies: '@types/estree': 1.0.9 + eventemitter3@4.0.7: {} + eventemitter3@5.0.4: {} events@1.1.1: {} @@ -3389,7 +3463,7 @@ snapshots: get-proto: 1.0.1 gopd: 1.2.0 has-symbols: 1.1.0 - hasown: 2.0.2 + hasown: 2.0.4 math-intrinsics: 1.1.0 get-proto@1.0.1: @@ -3425,10 +3499,6 @@ snapshots: dependencies: has-symbols: 1.1.0 - hasown@2.0.2: - dependencies: - function-bind: 1.1.2 - hasown@2.0.4: dependencies: function-bind: 1.1.2 @@ -3451,6 +3521,24 @@ snapshots: transitivePeerDependencies: - supports-color + http-proxy-middleware@2.0.10(debug@4.4.3(supports-color@7.2.0)): + dependencies: + '@types/http-proxy': 1.17.17 + http-proxy: 1.18.1(debug@4.4.3(supports-color@7.2.0)) + is-glob: 4.0.3 + is-plain-obj: 3.0.0 + micromatch: 4.0.8 + transitivePeerDependencies: + - debug + + http-proxy@1.18.1(debug@4.4.3(supports-color@7.2.0)): + dependencies: + eventemitter3: 4.0.7 + follow-redirects: 1.16.0(debug@4.4.3(supports-color@7.2.0)) + requires-port: 1.0.0 + transitivePeerDependencies: + - debug + https-proxy-agent@7.0.6(supports-color@7.2.0): dependencies: agent-base: 7.1.4 @@ -3485,6 +3573,8 @@ snapshots: is-extendable@0.1.1: {} + is-extglob@2.1.1: {} + is-fullwidth-code-point@3.0.0: {} is-fullwidth-code-point@5.1.0: @@ -3499,12 +3589,18 @@ snapshots: has-tostringtag: 1.0.2 safe-regex-test: 1.1.0 + is-glob@4.0.3: + dependencies: + is-extglob: 2.1.1 + is-inside-container@1.0.0: dependencies: is-docker: 3.0.0 is-number@7.0.0: {} + is-plain-obj@3.0.0: {} + is-potential-custom-element-name@1.0.1: optional: true @@ -3513,7 +3609,7 @@ snapshots: call-bound: 1.0.4 gopd: 1.2.0 has-tostringtag: 1.0.2 - hasown: 2.0.2 + hasown: 2.0.4 is-type-of@1.4.0: dependencies: @@ -3902,6 +3998,8 @@ snapshots: require-from-string@2.0.2: optional: true + requires-port@1.0.0: {} + resolve-from@5.0.0: {} resolve-pkg-maps@1.0.0: {} @@ -4155,6 +4253,8 @@ snapshots: dependencies: is-number: 7.0.0 + tos-crc64-js@0.0.1: {} + totalist@3.0.1: {} tough-cookie@6.0.0: @@ -4208,6 +4308,8 @@ snapshots: optionalDependencies: fsevents: 2.3.3 + type-fest@1.4.0: {} + typescript@5.9.3: {} ufo@1.6.4: {} diff --git a/pnpm-workspace.yaml b/pnpm-workspace.yaml index 2ed8ca0..2999097 100644 --- a/pnpm-workspace.yaml +++ b/pnpm-workspace.yaml @@ -2,6 +2,7 @@ allowBuilds: aws-sdk: false cos-js-sdk-v5: false esbuild: true + tos-crc64-js: false overrides: axios: 0.33.0 qs: ^6.14.2 diff --git a/src/ops/putSymlink.ts b/src/ops/putSymlink.ts index 0267b5a..5643452 100644 --- a/src/ops/putSymlink.ts +++ b/src/ops/putSymlink.ts @@ -2,11 +2,12 @@ import type { Options } from '../types' import type { Protocol } from '../protocol' /** - * Put a symlink. Only OSS supports symlinks; COS has no such API, so - * the operation rejects on protocols that do not support it (and the - * COS entry point does not export it). + * Put a symlink. OSS and TOS both support it; the provider supplies the + * target header through `symlinkHeaders` (OSS URI-encodes the target, TOS + * sends it verbatim), and protocols without a symlink API reject the call + * (and their entry point does not export it). * - * @param protocol provider protocol (OSS or COS) + * @param protocol provider protocol (OSS or TOS) */ export function createPutSymlink(protocol: Protocol) { return function putSymlink( @@ -17,9 +18,9 @@ export function createPutSymlink(protocol: Protocol) { if (!protocol.supportsSymlink) { return Promise.reject(new Error('the provider does not support symlink')) } - const headers: Record = { - 'x-oss-symlink-target': encodeURI(targetObjectName), - } + const headers = protocol.symlinkHeaders + ? protocol.symlinkHeaders(targetObjectName) + : { 'x-oss-symlink-target': encodeURI(targetObjectName) } return protocol.request(options, { verb: 'PUT', objectName, diff --git a/src/ops/request.ts b/src/ops/request.ts index d4d4ce9..c782045 100644 --- a/src/ops/request.ts +++ b/src/ops/request.ts @@ -69,7 +69,7 @@ export function resolveCallbackHeaders( if (!callback) return undefined if (!protocol.callbackHeaders) { throw new Error( - 'upload callback is not supported by this provider (Aliyun OSS and Huawei OBS support it; COS: pass x-cos-callback headers manually)', + 'upload callback is not supported by this provider (Aliyun OSS and Huawei OBS support the structured callback option; for COS/TOS pass the callback header through headers)', ) } const headers = protocol.callbackHeaders(callback) diff --git a/src/protocol.ts b/src/protocol.ts index 8a396b9..68f9948 100644 --- a/src/protocol.ts +++ b/src/protocol.ts @@ -29,8 +29,15 @@ export interface Protocol { copySourceRangeHeader: string /** listUploads marker query key: OSS 'marker', COS 'key-marker'. */ listUploadsMarkerKey: string - /** Whether the provider has a symlink API (only OSS does). */ + /** Whether the provider has a symlink API (OSS and TOS do). */ supportsSymlink: boolean + /** + * Serialize a symlink target into request headers. OSS and TOS both put + * the target in an `x-...-symlink-target` header but encode it + * differently, so the provider supplies the header. When absent the OSS + * header (`x-oss-symlink-target`, encodeURI'd target) is used. + */ + symlinkHeaders?: (targetObjectName: string) => Record /** * Serialize a structured upload callback into provider request * headers (OSS: x-oss-callback / x-oss-callback-var; OBS: diff --git a/src/tos/host.ts b/src/tos/host.ts new file mode 100644 index 0000000..709bd37 --- /dev/null +++ b/src/tos/host.ts @@ -0,0 +1,31 @@ +import type { Options } from '../types' + +/** + * Resolve the TOS host. TOS native endpoints are virtual-hosted only: the + * bucket is always a subdomain of the endpoint, e.g. + * `mybucket.tos-cn-beijing.volces.com`. + * + * Unlike OSS/COS/OBS (where `endpoint` is used verbatim as the full host), + * the TOS `endpoint` option is the endpoint *domain* — the bucket is still + * prefixed. This mirrors the official @volcengine/tos-sdk, whose `endpoint` + * is `tos-.volces.com` and whose host is `.`, and + * avoids the footgun of an endpoint that silently drops the bucket (TOS + * rejects path-style addressing). + * + * `internal: true` selects the Volcengine internal network domain + * (`tos-.ivolces.com`). + * + * `region` is always required for signing: the TOS4 credential scope is + * `//tos/request`. + */ +export function resolveTosHost(options: Options): string { + const { bucket, region, endpoint, internal } = options + if (!bucket) throw new Error('options.bucket is required') + if (!region) { + throw new Error( + 'options.region is required (TOS signs the credential scope with it; options.endpoint alone is not enough)', + ) + } + const domain = endpoint || `tos-${region}.${internal ? 'ivolces.com' : 'volces.com'}` + return `${bucket}.${domain}` +} diff --git a/src/tos/index.ts b/src/tos/index.ts new file mode 100644 index 0000000..98fa559 --- /dev/null +++ b/src/tos/index.ts @@ -0,0 +1,92 @@ +import type { Protocol } from '../protocol' +import { request as tosRequest } from './request' +import { tosSignUrl } from './signatureUrl' +import { createPut } from '../ops/put' +import { createPutSymlink } from '../ops/putSymlink' +import { createInitMultipartUpload } from '../ops/initMultipartUpload' +import { createUploadPart } from '../ops/uploadPart' +import { createCompleteMultipartUpload } from '../ops/completeMultipartUpload' +import { createAbortMultipartUpload } from '../ops/abortMultipartUpload' +import { createListParts } from '../ops/listParts' +import { createListUploads } from '../ops/listUploads' +import { createUploadPartCopy } from '../ops/uploadPartCopy' +import { createMultipartUpload } from '../ops/multipartUpload' + +/** + * The Volcano Engine Torch Object Storage (TOS) protocol. This entry point + * never references the OSS, COS, OBS or AWS signers, so bundlers tree-shake + * them away. TOS has a symlink API (unlike COS/OBS/S3), so putSymlink is + * exported; structured upload callbacks are not serialized (pass + * `x-tos-callback` / `x-tos-callback-var` through `headers` instead). + */ +const TOS_PROTOCOL: Protocol = { + request: tosRequest, + metaPrefix: 'x-tos-meta-', + copySourceHeader: 'x-tos-copy-source', + copySourceRangeHeader: 'x-tos-copy-source-range', + listUploadsMarkerKey: 'key-marker', + supportsSymlink: true, + symlinkHeaders: (targetObjectName) => ({ 'x-tos-symlink-target': targetObjectName }), + signUrl: tosSignUrl, +} + +const put = createPut(TOS_PROTOCOL) +const putSymlink = createPutSymlink(TOS_PROTOCOL) +const initMultipartUpload = createInitMultipartUpload(TOS_PROTOCOL) +const uploadPart = createUploadPart(TOS_PROTOCOL) +const completeMultipartUpload = createCompleteMultipartUpload(TOS_PROTOCOL) +const multipartUpload = createMultipartUpload(TOS_PROTOCOL, { + initMultipartUpload, + uploadPart, + completeMultipartUpload, +}) +const abortMultipartUpload = createAbortMultipartUpload(TOS_PROTOCOL) +const listParts = createListParts(TOS_PROTOCOL) +const listUploads = createListUploads(TOS_PROTOCOL) +const uploadPartCopy = createUploadPartCopy(TOS_PROTOCOL) +const signatureUrl = TOS_PROTOCOL.signUrl + +export type { + BlobLike, + Checkpoint, + CompleteMultipartUploadResult, + HTTPMethods, + InitMultipartUploadResult, + ListPartsResult, + ListQuery, + ListUploadsQuery, + ListUploadsResult, + MultipartOptions, + MultipartUploadOptions, + ObjectCallback, + Options, + Part, + PartInfo, + Progress, + PutOptions, + ResponseHeaderType, + SignatureUrlOptions, + SourceData, + UploadInfo, + UploadPartCopyOptions, + UploadPartCopyResult, + UploadPartResult, +} from '../types' +export { setTransport, getTransport } from '../transport' +export type { Transport, TransportOptions, TransportResponse } from '../transport' +export { fetchTransport } from '../transports/fetch' +export { wxRequestTransport } from '../transports/wx' +export { bindOptions } from '../ops/bindOptions' +export { + put, + putSymlink, + initMultipartUpload, + uploadPart, + completeMultipartUpload, + multipartUpload, + abortMultipartUpload, + listParts, + listUploads, + uploadPartCopy, + signatureUrl, +} diff --git a/src/tos/request.ts b/src/tos/request.ts new file mode 100644 index 0000000..507202a --- /dev/null +++ b/src/tos/request.ts @@ -0,0 +1,66 @@ +import { getTransport } from '../transport' +import { normalizeOptions, resolveTimeout, dataSize } from '../ops/request' +import { getTosSignature, tosUriEscapePath, canonicalTosQuery, iso8601 } from './signature' +import { resolveTosHost } from './host' +import type { Options } from '../types' +import type { RequestParams } from '../protocol' + +/** TOS defaults: https, 60s timeout (region and endpoint come from the caller). */ +const TOS_DEFAULTS = { + secure: true, + timeout: 60000, +} + +/** + * Sign and send a single TOS request through the configured transport. + * The TOS4-HMAC-SHA256 Authorization covers the host, x-tos-date, + * x-tos-content-sha256 (always 'UNSIGNED-PAYLOAD', like the official SDK), + * x-tos-security-token and any other x-tos-* header (metadata, copy + * source, …). The object key is URI-escaped ('/' kept) in both the + * signature and the request URL, and the query string is sorted exactly + * like the signed one. + */ +export function request(options: Options, params: RequestParams): Promise { + const opts = normalizeOptions(options, TOS_DEFAULTS) + const { accessKeyId, accessKeySecret, stsToken, secure, region } = opts + const host = resolveTosHost(opts) + const headers: Record = { + host, + 'x-tos-date': iso8601(new Date()), + 'x-tos-content-sha256': 'UNSIGNED-PAYLOAD', + ...params.headers, + } + if (stsToken) headers['x-tos-security-token'] = stsToken + // The official SDK lower-cases every header name before signing, so the + // signed-header set is host + all x-tos-* headers regardless of the + // caller's casing. Send the same lower-cased names. + const normalizedHeaders: Record = {} + Object.keys(headers).forEach((key) => { + normalizedHeaders[key.toLowerCase()] = headers[key] + }) + const pathname = `/${tosUriEscapePath(params.objectName)}` + const { signature, credentialScope, signedHeaders } = getTosSignature({ + method: params.verb, + pathname, + query: params.subResource, + headers: normalizedHeaders, + accessKeyId, + secretAccessKey: accessKeySecret, + region: region as string, + }) + normalizedHeaders.authorization = `TOS4-HMAC-SHA256 Credential=${accessKeyId}/${credentialScope}, SignedHeaders=${signedHeaders}, Signature=${signature}` + const protocol = secure ? 'https' : 'http' + let url = `${protocol}://${host}${pathname}` + if (params.subResource) { + const qs = canonicalTosQuery(params.subResource) + if (qs) url += `?${qs}` + } + return getTransport()(url, { + method: params.verb, + headers: normalizedHeaders, + data: params.data, + timeout: params.timeout == null ? resolveTimeout(opts) : params.timeout, + onprogress: params.onprogress, + total: dataSize(params.data), + }) +} diff --git a/src/tos/signature.ts b/src/tos/signature.ts new file mode 100644 index 0000000..f910617 --- /dev/null +++ b/src/tos/signature.ts @@ -0,0 +1,170 @@ +import { sha256, hmacSha256 } from '../utils/sha256' +import { encodeUtf8 } from '../utils/utf8' + +/** + * Volcano Engine Torch Object Storage (TOS) request signature, TOS4-HMAC-SHA256. + * + * The algorithm mirrors the official @volcengine/tos-sdk SignersV4 + * (src/signatureV4.ts) exactly: + * + * CanonicalRequest = Method\n + * CanonicalURI\n // URI-escaped, '/' kept + * CanonicalQueryString\n // sorted, URI-escaped + * CanonicalHeaders\n\n + * SignedHeaders\n + * 'UNSIGNED-PAYLOAD' + * StringToSign = TOS4-HMAC-SHA256\n + * tosDate\n + * date/region/tos/request\n + * hex(SHA256(CanonicalRequest)) + * SigningKey = HMAC-SHA256(HMAC-SHA256(HMAC-SHA256(HMAC-SHA256( + * SecretKey, date), region), 'tos'), 'request') + * Signature = hex(HMAC-SHA256(SigningKey, StringToSign)) + * + * Signed headers are `host` plus every `x-tos-*` header (lower-cased, + * sorted); unlike SigV4 the canonical header block is followed by a blank + * line. The body hash is always the literal 'UNSIGNED-PAYLOAD' (the official + * SDK never signs the body). + */ + +/** URI-escape per the official SDK: encodeURIComponent plus the RFC 3986 chars it leaves. */ +export function tosUriEscape(str: string): string { + return encodeURIComponent(str) + .replace(/!/g, '%21') + .replace(/'/g, '%27') + .replace(/\(/g, '%28') + .replace(/\)/g, '%29') + .replace(/\*/g, '%2A') +} + +/** URI-escape a path segment by segment so '/' is preserved. */ +export function tosUriEscapePath(str: string): string { + return str.split('/').map(tosUriEscape).join('/') +} + +function bytesToHex(bytes: Uint8Array): string { + let hex = '' + for (let i = 0; i < bytes.length; i++) { + hex += bytes[i].toString(16).padStart(2, '0') + } + return hex +} + +function sha256Hex(str: string): string { + return bytesToHex(new Uint8Array(sha256().digest(encodeUtf8(str)))) +} + +function hmacSha256Hex(key: string | Uint8Array, message: string): string { + const hmac = hmacSha256() + // instanceof is realm-bound; any TypedArray view carries subarray. + const keyView = key as Uint8Array // structural stand-in; checked below + const secret = + ArrayBuffer.isView(key) && typeof keyView.subarray === 'function' + ? keyView + : encodeUtf8(key as string) + hmac.setKey(secret) + hmac.update(encodeUtf8(message)) + return bytesToHex(new Uint8Array(hmac.finalize())) +} + +function hexToBytes(hex: string): Uint8Array { + const out = new Uint8Array(hex.length / 2) + for (let i = 0; i < out.length; i++) { + out[i] = parseInt(hex.substr(i * 2, 2), 16) + } + return out +} + +/** Normalize a header value the way the official SDK does before signing. */ +function canonicalHeaderValue(value: any): string { + return String(value) + .replace(/\s+/g, ' ') + .replace(/^\s+|\s+$/g, '') +} + +/** Serialize query params sorted and URI-escaped, like the official SDK. */ +export function canonicalTosQuery(query: Record): string { + return Object.keys(query) + .sort() + .map((name) => { + const value = query[name] + return `${tosUriEscape(name)}=${tosUriEscape(value === undefined || value === null ? '' : String(value))}` + }) + .join('&') +} + +export interface TosSignOptions { + method: string + /** Canonical URI — the URI-escaped object path with '/' kept, e.g. '/dir/file.txt'. */ + pathname: string + query?: Record + /** Signable request headers; must include 'host'. Only host and x-tos-* are signed. */ + headers: Record + accessKeyId: string + secretAccessKey: string + region: string + service?: string + /** Fixed 'YYYYMMDDTHHMMSSZ' timestamp for tests; defaults to now. */ + date?: string +} + +/** + * Compute the TOS4-HMAC-SHA256 signature. Returns the pieces needed by both + * header authorization and pre-signed URL construction. + */ +export function getTosSignature(opt: TosSignOptions): { + tosDate: string + credentialScope: string + signedHeaders: string + signature: string +} { + const { method, pathname, headers, accessKeyId, secretAccessKey, region } = opt + if (!accessKeyId) throw new Error('need accessKeyId') + if (!secretAccessKey) throw new Error('need accessKeySecret') + const service = opt.service || 'tos' + const tosDate = opt.date || iso8601(new Date()) + const shortDate = tosDate.substr(0, 8) + const credentialScope = `${shortDate}/${region}/${service}/request` + + // Lower-case the header names, drop null values, and keep host + x-tos-*. + const normalized: Record = {} + Object.keys(headers).forEach((key) => { + if (headers[key] != null) normalized[key.toLowerCase()] = headers[key] + }) + const signedNames = Object.keys(normalized) + .filter((key) => key === 'host' || key.indexOf('x-tos-') === 0) + .sort() + const canonicalHeaders = signedNames + .map((key) => `${key}:${canonicalHeaderValue(normalized[key])}`) + .join('\n') + const signedHeaders = signedNames.join(';') + + const canonicalRequest = [ + method, + pathname, + canonicalTosQuery(opt.query || {}), + `${canonicalHeaders}\n`, + signedHeaders, + 'UNSIGNED-PAYLOAD', + ].join('\n') + + const stringToSign = [ + 'TOS4-HMAC-SHA256', + tosDate, + credentialScope, + sha256Hex(canonicalRequest), + ].join('\n') + + const kDate = hmacSha256Hex(secretAccessKey, shortDate) + const kRegion = hmacSha256Hex(hexToBytes(kDate), region) + const kService = hmacSha256Hex(hexToBytes(kRegion), service) + const kSigning = hmacSha256Hex(hexToBytes(kService), 'request') + const signature = hmacSha256Hex(hexToBytes(kSigning), stringToSign) + + return { tosDate, credentialScope, signedHeaders, signature } +} + +/** 'YYYYMMDDTHHMMSSZ' from a Date, matching the official SDK's getDateTime. */ +export function iso8601(date: Date): string { + return date.toISOString().replace(/[:-]|\.\d{3}/g, '') +} diff --git a/src/tos/signatureUrl.ts b/src/tos/signatureUrl.ts new file mode 100644 index 0000000..6458f8c --- /dev/null +++ b/src/tos/signatureUrl.ts @@ -0,0 +1,105 @@ +import { normalizeOptions } from '../ops/request' +import { getTosSignature, tosUriEscapePath, iso8601 } from './signature' +import { resolveTosHost } from './host' +import type { Options, ResponseHeaderType, SignatureUrlOptions } from '../types' + +/** TOS defaults: https, 60s timeout (region and endpoint come from the caller). */ +const TOS_DEFAULTS = { + secure: true, + timeout: 60000, +} + +/** + * Get a signed TOS URL (TOS4-HMAC-SHA256 query authentication), mirroring + * the official @volcengine/tos-sdk getPreSignedUrl: + * + * https://./ + * ?X-Tos-Algorithm=TOS4-HMAC-SHA256 + * &X-Tos-Content-Sha256=UNSIGNED-PAYLOAD + * &X-Tos-Credential=%2F%2F%2Ftos%2Frequest + * &X-Tos-Date=&X-Tos-Expires= + * &X-Tos-SignedHeaders=host + * [&X-Tos-Security-Token=] + * &X-Tos-Signature= + * + * Only `host` is signed (the official SDK's query signer does not sign the + * X-Tos-Date header it injects), so the link stays usable from a plain + * browser without custom headers. + * + * The credential scope uses the configured `region` (test/tos-oracle.node.ts + * pins this against the official Go SDK's published query vector). The + * official JS SDK's getPreSignedUrl substitutes the endpoint for the region + * there (src/methods/base.ts: `region: this.opts.endpoint`), which disagrees + * with the Go SDK and the standard V4 credential scope. + * + * @param options client options + * @param objectName object name + * @param urlOptions signature options, same shape as the OSS entry + * @return signed URL + */ +export function tosSignUrl( + options: Options, + objectName: string, + urlOptions: SignatureUrlOptions = {}, +): string { + const { expires = 1800, method, process, response } = urlOptions + const opts = normalizeOptions(options, TOS_DEFAULTS) + const { accessKeyId, accessKeySecret, stsToken, secure, region } = opts + const host = resolveTosHost(opts) + const query: Record = {} + if (process) query['x-tos-process'] = String(process) + if (response) { + Object.keys(response).forEach((key) => { + query[`response-${key.toLowerCase()}`] = String(response[key as keyof ResponseHeaderType]) + }) + } + Object.keys(urlOptions).forEach((key) => { + const lower = key.toLowerCase() + if ( + lower === 'expires' || + lower === 'method' || + lower === 'response' || + lower === 'process' || + lower === 'callback' || + lower === 'security-token' || + lower === 'content-type' || + lower === 'content-md5' + ) { + return + } + if (lower.indexOf('x-tos-') === 0) query[lower] = String(urlOptions[key]) + else query[key] = String(urlOptions[key]) + }) + const securityToken = urlOptions['security-token'] || stsToken + const tosDate = iso8601(new Date()) + const credentialScope = `${tosDate.substr(0, 8)}/${region}/tos/request` + const signParams: Record = { + 'X-Tos-Algorithm': 'TOS4-HMAC-SHA256', + 'X-Tos-Content-Sha256': 'UNSIGNED-PAYLOAD', + 'X-Tos-Credential': `${accessKeyId}/${credentialScope}`, + 'X-Tos-Date': tosDate, + 'X-Tos-Expires': String(expires), + 'X-Tos-SignedHeaders': 'host', + } + if (securityToken) signParams['X-Tos-Security-Token'] = securityToken + const { signature } = getTosSignature({ + method: method || 'GET', + pathname: `/${tosUriEscapePath(objectName)}`, + query: { ...query, ...signParams }, + headers: { host }, + accessKeyId, + secretAccessKey: accessKeySecret, + region: region as string, + date: tosDate, + }) + const protocol = secure ? 'https' : 'http' + const parts: string[] = [] + Object.keys(query).forEach((key) => { + parts.push(`${encodeURIComponent(key)}=${encodeURIComponent(query[key])}`) + }) + Object.keys(signParams).forEach((key) => { + parts.push(`${encodeURIComponent(key)}=${encodeURIComponent(signParams[key])}`) + }) + parts.push(`X-Tos-Signature=${signature}`) + return `${protocol}://${host}/${tosUriEscapePath(objectName)}?${parts.join('&')}` +} diff --git a/src/types.ts b/src/types.ts index be801ed..e22d13e 100644 --- a/src/types.ts +++ b/src/types.ts @@ -22,7 +22,7 @@ export type BlobLike = Blob | ArrayBuffer | Uint8Array export interface PutOptions { onprogress?: (e: Progress) => any - headers?: Record // extra request headers, e.g. x-cos-callback on COS + headers?: Record // extra request headers, e.g. x-cos-callback on COS or x-tos-callback on TOS callback?: ObjectCallback // server-side upload callback; OSS and OBS only } @@ -40,10 +40,10 @@ export interface ResponseHeaderType { * Server-side upload callback: once the object is stored the provider * calls back the configured URL and relays that response to the client. * Serialized by Aliyun OSS (x-oss-callback / x-oss-callback-var) and - * Huawei OBS (x-obs-callback) on put and multipartUpload. COS exposes - * the same feature but its official SDK passes the header value through - * verbatim, so COS users must set headers (x-cos-callback) themselves. - * AWS S3 and Azure Blob have no callback API. + * Huawei OBS (x-obs-callback) on put and multipartUpload. COS and TOS + * expose the same feature but their official SDKs pass the header value + * through verbatim, so those users must set headers (x-cos-callback / + * x-tos-callback) themselves. AWS S3 and Azure Blob have no callback API. */ export interface ObjectCallback { url: string // After a file is uploaded successfully, the OSS sends a callback request to this URL. diff --git a/test/test-types-tos.ts b/test/test-types-tos.ts new file mode 100644 index 0000000..12733eb --- /dev/null +++ b/test/test-types-tos.ts @@ -0,0 +1,89 @@ +// Type testing file for the TOS entry point — validates dist/tiny-oss.tos.es.d.ts. +// Run: npx tsc --noEmit --skipLibCheck test-types-tos.ts + +import { + put, + putSymlink, + signatureUrl, + initMultipartUpload, + uploadPart, + completeMultipartUpload, + abortMultipartUpload, + listParts, + listUploads, + uploadPartCopy, + multipartUpload, + bindOptions, + type CompleteMultipartUploadResult, + type InitMultipartUploadResult, + type ListPartsResult, + type ListUploadsResult, + type Options, + type UploadPartCopyResult, + type UploadPartResult, +} from '../dist/tiny-oss.tos.es' + +const options: Options = { + accessKeyId: 'AKIDxxxxxxxxxxxxxxxx', + accessKeySecret: 'secret', + bucket: 'examplebucket', + region: 'cn-beijing', + secure: true, +} + +const blob = new Blob(['test'], { type: 'text/plain' }) + +// Every operation keeps the same signature as the OSS entry. +const putPromise: Promise = put(options, 'test.txt', blob) +const symlinkPromise: Promise = putSymlink(options, 'link.txt', 'test.txt') +const url: string = signatureUrl(options, 'test.txt', { expires: 600 }) +const initResult: Promise = initMultipartUpload(options, 'test.txt') +const uploadPartResult: Promise = uploadPart( + options, + 'test.txt', + 'u1', + 1, + blob, + 0, + 1024, +) +const completeResult: Promise = completeMultipartUpload( + options, + 'test.txt', + 'u1', + [{ number: 1, etag: '"e"' }], +) +const abortResult: Promise = abortMultipartUpload(options, 'test.txt', 'u1') +const listPartsResult: Promise = listParts(options, 'test.txt', 'u1') +const listUploadsResult: Promise = listUploads(options, { prefix: 'x' }) +const copyResult: Promise = uploadPartCopy( + options, + 'test.txt', + 'u1', + 1, + 'bytes=0-1023', + { sourceKey: 'src.txt' }, +) +const multiResult: Promise = multipartUpload( + options, + 'test.txt', + blob, +) +const upload = bindOptions(put, options) +const boundPromise: Promise = upload('bound.txt', blob) + +console.log( + 'All TOS type tests passed!', + putPromise, + symlinkPromise, + url, + initResult, + uploadPartResult, + completeResult, + abortResult, + listPartsResult, + listUploadsResult, + copyResult, + multiResult, + boundPromise, +) diff --git a/test/tos-oracle.node.ts b/test/tos-oracle.node.ts new file mode 100644 index 0000000..9a9bc09 --- /dev/null +++ b/test/tos-oracle.node.ts @@ -0,0 +1,368 @@ +/** + * TOS signature oracle: compares tiny-oss's TOS4-HMAC-SHA256 implementation + * byte-for-byte against the official @volcengine/tos-sdk. + * + * The SDK's browser build does not survive vite's dependency optimizer, so + * this runs under Node (like test/azure-oracle.node.ts): + * + * pnpm test:tos-oracle + * + * The header oracle captures the SDK's real requests with a local HTTP server + * and recomputes the Authorization with our signer from the captured + * method/path/query/headers. The fixed vectors embedded in + * test/tos-signature.spec.ts are asserted here too. + */ +import { strict as assert } from 'node:assert' +import http from 'node:http' +import type { AddressInfo } from 'node:net' +import { TosClient } from '@volcengine/tos-sdk' +import { getTosSignature } from '../src/tos/signature' +import { tosSignUrl } from '../src/tos/signatureUrl' + +const AK = 'AKIDEXAMPLE' +const SK = 'SECRETKEY' +const REGION = 'cn-beijing' +const BUCKET = 'examplebucket' +const TOKEN = 'TOKEN' + +interface Captured { + method: string + url: string + headers: http.IncomingHttpHeaders + body: string +} + +let captured: Captured[] = [] +const server = http.createServer((req, res) => { + let body = '' + req.on('data', (chunk) => { + body += chunk + }) + req.on('end', () => { + captured.push({ + method: req.method as string, + url: req.url as string, + headers: req.headers, + body, + }) + res.writeHead(200, { 'content-type': 'application/xml', etag: '"e1"' }) + res.end( + '' + + 'bku1', + ) + }) +}) + +let failures = 0 +function fail(name: string, err: unknown): void { + failures += 1 + console.error(`FAIL ${name}\n${err instanceof Error ? err.message : String(err)}`) +} + +function check(name: string, fn: () => void): void { + try { + fn() + console.log(` ok ${name}`) + } catch (err) { + fail(name, err) + } +} + +async function checkAsync(name: string, fn: () => Promise): Promise { + try { + await fn() + console.log(` ok ${name}`) + } catch (err) { + fail(name, err) + } +} + +/** The canonical URI the SDK signs: restore %2F, then encode !*'(). */ +function canonicalPath(rawUrl: string): string { + return rawUrl + .split('?')[0] + .replace(/%2F/g, '/') + .replace(/!/g, '%21') + .replace(/'/g, '%27') + .replace(/\(/g, '%28') + .replace(/\)/g, '%29') + .replace(/\*/g, '%2A') +} + +function queryOf(rawUrl: string): Record { + const out: Record = {} + const qs = rawUrl.split('?')[1] + if (!qs) return out + qs.split('&') + .filter(Boolean) + .forEach((pair) => { + const eq = pair.indexOf('=') + const key = decodeURIComponent(eq < 0 ? pair : pair.slice(0, eq)) + const value = eq < 0 ? '' : decodeURIComponent(pair.slice(eq + 1)) + out[key] = value + }) + return out +} + +function signedHeadersOf(headers: http.IncomingHttpHeaders): Record { + const out: Record = {} + Object.keys(headers).forEach((key) => { + const lower = key.toLowerCase() + const value = headers[key] + if (value != null && (lower === 'host' || lower.indexOf('x-tos-') === 0)) { + out[lower] = Array.isArray(value) ? value.join(',') : String(value) + } + }) + return out +} + +function mineFor(call: Captured): string { + const { signature, credentialScope, signedHeaders } = getTosSignature({ + method: call.method, + pathname: canonicalPath(call.url), + query: queryOf(call.url), + headers: signedHeadersOf(call.headers), + accessKeyId: AK, + secretAccessKey: SK, + region: REGION, + date: String(call.headers['x-tos-date']), + }) + return `TOS4-HMAC-SHA256 Credential=${AK}/${credentialScope}, SignedHeaders=${signedHeaders}, Signature=${signature}` +} + +async function capturedCall(fn: () => Promise): Promise { + captured = [] + try { + await fn() + } catch { + // Some calls need a response body we do not model; the request is + // captured before response parsing. + } + assert.ok(captured.length > 0, 'no request captured') + return captured[0] +} + +async function main() { + await new Promise((resolve) => server.listen(0, '127.0.0.1', resolve)) + const port = (server.address() as AddressInfo).port + const endpoint = `127.0.0.1:${port}` + const makeClient = (stsToken?: string) => + new TosClient({ + accessKeyId: AK, + accessKeySecret: SK, + region: REGION, + endpoint, + secure: false, + bucket: BUCKET, + stsToken, + }) + + console.log('TOS4-HMAC-SHA256 headers vs @volcengine/tos-sdk:') + + await checkAsync('putObject with metadata', async () => { + const call = await capturedCall(() => + makeClient(TOKEN).putObject({ key: 'dir/a b.txt', body: 'hello', meta: { title: 'T' } }), + ) + assert.equal(mineFor(call), call.headers.authorization) + }) + + await checkAsync('putObject with a non-ASCII key', async () => { + const call = await capturedCall(() => + makeClient(TOKEN).putObject({ key: '目录/文件 名.txt', body: 'x' }), + ) + assert.equal(mineFor(call), call.headers.authorization) + }) + + await checkAsync('createMultipartUpload (?uploads)', async () => { + const call = await capturedCall(() => + makeClient(TOKEN).createMultipartUpload({ key: 'bigfile.bin' }), + ) + assert.ok(call.url.indexOf('uploads=') >= 0) + assert.equal(mineFor(call), call.headers.authorization) + }) + + await checkAsync('uploadPart (partNumber + uploadId)', async () => { + const call = await capturedCall(() => + makeClient(TOKEN).uploadPart({ + key: 'bigfile.bin', + uploadId: 'u1', + partNumber: 1, + body: 'x', + }), + ) + assert.deepEqual(queryOf(call.url), { partNumber: '1', uploadId: 'u1' }) + assert.equal(mineFor(call), call.headers.authorization) + }) + + await checkAsync('listParts (max-parts + part-number-marker)', async () => { + const call = await capturedCall(() => + makeClient(TOKEN).listParts({ + key: 'bigfile.bin', + uploadId: 'u1', + maxParts: 10, + partNumberMarker: 3, + }), + ) + assert.equal(mineFor(call), call.headers.authorization) + }) + + await checkAsync('listMultipartUploads', async () => { + const call = await capturedCall(() => + makeClient(TOKEN).listMultipartUploads({ bucket: BUCKET }), + ) + assert.equal(mineFor(call), call.headers.authorization) + }) + + await checkAsync('completeMultipartUpload', async () => { + const call = await capturedCall(() => + makeClient(TOKEN).completeMultipartUpload({ + key: 'bigfile.bin', + uploadId: 'u1', + parts: [{ partNumber: 1, etag: '"e1"' }], + }), + ) + assert.equal(mineFor(call), call.headers.authorization) + }) + + await checkAsync('abortMultipartUpload', async () => { + const call = await capturedCall(() => + makeClient(TOKEN).abortMultipartUpload({ key: 'bigfile.bin', uploadId: 'u456' }), + ) + assert.equal(mineFor(call), call.headers.authorization) + }) + + await checkAsync('uploadPartCopy (copy source + range)', async () => { + const call = await capturedCall(() => + makeClient(TOKEN).uploadPartCopy({ + key: 'bigfile.bin', + uploadId: 'u1', + partNumber: 2, + srcBucket: 'srcbucket', + srcKey: 'dir/source.bin', + copySourceRange: 'bytes=0-1023', + }), + ) + assert.equal(String(call.headers['x-tos-copy-source']), '/srcbucket/dir%2Fsource.bin') + assert.equal(mineFor(call), call.headers.authorization) + }) + + await checkAsync('putSymlink (x-tos-symlink-target)', async () => { + const call = await capturedCall(() => + makeClient(TOKEN).putSymlink({ key: 'link.txt', symLinkTargetKey: 'dir/target.txt' }), + ) + assert.equal(String(call.headers['x-tos-symlink-target']), 'dir/target.txt') + assert.equal(mineFor(call), call.headers.authorization) + }) + + await checkAsync('without a security token', async () => { + const call = await capturedCall(() => makeClient().putObject({ key: 'plain.txt', body: 'x' })) + assert.equal(call.headers['x-tos-security-token'], undefined) + assert.equal(mineFor(call), call.headers.authorization) + }) + + // The vector embedded in test/tos-signature.spec.ts. + console.log('\nFixed vectors match the SDK output:') + check('init vector (20260918T125156Z)', () => { + const { signature, credentialScope, signedHeaders } = getTosSignature({ + method: 'POST', + pathname: '/examplebucket/bigfile.bin', + query: { uploads: '' }, + headers: { + host: '127.0.0.1:45893', + 'x-tos-date': '20260918T125156Z', + 'x-tos-content-sha256': 'UNSIGNED-PAYLOAD', + }, + accessKeyId: AK, + secretAccessKey: SK, + region: REGION, + date: '20260918T125156Z', + }) + assert.equal(credentialScope, '20260918/cn-beijing/tos/request') + assert.equal(signedHeaders, 'host;x-tos-content-sha256;x-tos-date') + assert.equal(signature, '57e4cfd365d312c5e37abb6d26e19b11b231e5d4e761d6b6b11fc3cf1b417b6c') + }) + + console.log('\nPre-signed URLs vs @volcengine/tos-sdk (endpoint === region):') + const RealDate = Date + const FIXED = '2026-09-01T00:00:00.000Z' + class FixedDate extends RealDate { + constructor(...args: any[]) { + if (args.length === 0) super(FIXED) + else super(...(args as [any])) + } + static now() { + return new RealDate(FIXED).getTime() + } + } + ;(globalThis as any).Date = FixedDate + try { + const client = new TosClient({ + accessKeyId: AK, + accessKeySecret: SK, + region: REGION, + endpoint: REGION, + secure: true, + bucket: BUCKET, + }) + const opts = { + accessKeyId: AK, + accessKeySecret: SK, + region: REGION, + endpoint: REGION, + bucket: BUCKET, + secure: true, + } + for (const key of ['exampleobject', '目录/a b.txt']) { + check(`signed URL for ${JSON.stringify(key)}`, () => { + const official = new URL( + client.getPreSignedUrl({ bucket: BUCKET, key, method: 'GET', expires: 600 }), + ) + const mine = new URL(tosSignUrl(opts, key, { expires: 600 })) + assert.equal(mine.host, official.host) + assert.equal(decodeURIComponent(mine.pathname), decodeURIComponent(official.pathname)) + assert.deepEqual( + Object.fromEntries(mine.searchParams), + Object.fromEntries(official.searchParams), + ) + }) + } + check('plain GET signature vector (20260901T000000Z)', () => { + const mine = new URL(tosSignUrl(opts, 'exampleobject', { expires: 600 })) + assert.equal( + mine.searchParams.get('X-Tos-Signature'), + '4cba3a3c67f8da33a72d5b87c0c264715edfc56552c35a4cff81eed169c3f7c0', + ) + }) + check('non-ASCII signature vector (20260901T000000Z)', () => { + const mine = new URL(tosSignUrl(opts, '目录/a b.txt', { expires: 600 })) + assert.equal( + mine.searchParams.get('X-Tos-Signature'), + '9fc2461247b09863d0fe8bac0d545b07206fad757629a69da8adf0124f59a4d8', + ) + }) + check('region (not endpoint) in the credential scope', () => { + const mine = new URL( + tosSignUrl({ ...opts, endpoint: `tos-${REGION}.volces.com` }, 'exampleobject', { + expires: 600, + }), + ) + assert.equal(mine.host, `${BUCKET}.tos-${REGION}.volces.com`) + assert.equal( + mine.searchParams.get('X-Tos-Credential'), + `${AK}/20260901/${REGION}/tos/request`, + ) + }) + } finally { + ;(globalThis as any).Date = RealDate + } + + await new Promise((resolve) => server.close(() => resolve())) + + if (failures > 0) { + console.error(`\n${failures} oracle check(s) FAILED`) + process.exit(1) + } + console.log('\nAll TOS oracle checks passed.') +} + +main() diff --git a/test/tos-signature.spec.ts b/test/tos-signature.spec.ts new file mode 100644 index 0000000..2c69c92 --- /dev/null +++ b/test/tos-signature.spec.ts @@ -0,0 +1,278 @@ +import { describe, it, expect, vi, afterEach } from 'vitest' +import { getTosSignature, tosUriEscape, canonicalTosQuery } from '../src/tos/signature' +import { tosSignUrl } from '../src/tos/signatureUrl' +import { request as tosRequest } from '../src/tos/request' +import { resolveTosHost } from '../src/tos/host' +import { setTransport, getTransport } from '../src/transport' +import type { Options } from '../src/types' + +// The byte-identical oracle for these values lives in test/tos-oracle.node.ts +// (it drives the official @volcengine/tos-sdk against a local capture server +// and compares the full Authorization header / pre-signed URL). The fixed +// vectors below are copied from that oracle so the normal `pnpm test` run +// pins the signer without importing the SDK (whose browser build does not +// survive vite's dependency optimizer). +const AK = 'AKIDEXAMPLE' +const SK = 'SECRETKEY' +const REGION = 'cn-beijing' +const BUCKET = 'examplebucket' +const TOKEN = 'TOKEN' + +describe('getTosSignature fixed vectors (oracle: @volcengine/tos-sdk)', () => { + it('signs host, x-tos-date and x-tos-content-sha256', () => { + const { signature, credentialScope, signedHeaders } = getTosSignature({ + method: 'POST', + pathname: '/examplebucket/bigfile.bin', + query: { uploads: '' }, + headers: { + host: '127.0.0.1:45893', + 'x-tos-date': '20260918T125156Z', + 'x-tos-content-sha256': 'UNSIGNED-PAYLOAD', + }, + accessKeyId: AK, + secretAccessKey: SK, + region: REGION, + date: '20260918T125156Z', + }) + expect(credentialScope).toBe('20260918/cn-beijing/tos/request') + expect(signedHeaders).toBe('host;x-tos-content-sha256;x-tos-date') + expect(signature).toBe('57e4cfd365d312c5e37abb6d26e19b11b231e5d4e761d6b6b11fc3cf1b417b6c') + }) + + it('canonicalizes the query sorted and URI-escaped', () => { + expect(canonicalTosQuery({ b: '2', a: '1 0' })).toBe('a=1%200&b=2') + expect(canonicalTosQuery({ uploads: '' })).toBe('uploads=') + }) + + // Published vector from the official Go SDK's TestAlgV4 + // (ve-tos-golang-sdk tos/sign_v4_test.go): GET https://test.tos.com:8080/test.txt, + // region cn-north-1, date 20210721T104454Z. It pins the query-signing path + // and, critically, that the credential scope carries the region. + it('matches the official Go SDK published query vector', () => { + const { signature, credentialScope } = getTosSignature({ + method: 'GET', + pathname: '/test.txt', + query: { + 'X-Tos-Algorithm': 'TOS4-HMAC-SHA256', + 'X-Tos-Credential': 'AKIAIOSFODNN7EXAMPLE/20210721/cn-north-1/tos/request', + 'X-Tos-Date': '20210721T104454Z', + 'X-Tos-Expires': '3600', + 'X-Tos-SignedHeaders': 'host', + }, + headers: { host: 'test.tos.com:8080' }, + accessKeyId: 'AKIAIOSFODNN7EXAMPLE', + secretAccessKey: 'wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY', + region: 'cn-north-1', + date: '20210721T104454Z', + }) + expect(credentialScope).toBe('20210721/cn-north-1/tos/request') + expect(signature).toBe('decc75e2b2d453117f81e53954eb2cd3a2f56db2951e9b2257863db4c4921111') + }) +}) + +describe('tosSignUrl fixed vectors (oracle: @volcengine/tos-sdk, endpoint === region)', () => { + afterEach(() => { + vi.useRealTimers() + }) + + const opts = { + accessKeyId: AK, + accessKeySecret: SK, + region: REGION, + endpoint: REGION, + bucket: BUCKET, + secure: true, + } + + function at(date: string) { + vi.useFakeTimers() + vi.setSystemTime(new Date(date)) + } + + it('plain GET download', () => { + at('2026-09-01T00:00:00.000Z') + const url = new URL(tosSignUrl(opts, 'exampleobject', { expires: 600 })) + expect(url.host).toBe(`${BUCKET}.cn-beijing`) + expect(url.pathname).toBe('/exampleobject') + expect(url.searchParams.get('X-Tos-Signature')).toBe( + '4cba3a3c67f8da33a72d5b87c0c264715edfc56552c35a4cff81eed169c3f7c0', + ) + expect(Object.fromEntries(url.searchParams)).toEqual({ + 'X-Tos-Algorithm': 'TOS4-HMAC-SHA256', + 'X-Tos-Content-Sha256': 'UNSIGNED-PAYLOAD', + 'X-Tos-Credential': `${AK}/20260901/${REGION}/tos/request`, + 'X-Tos-Date': '20260901T000000Z', + 'X-Tos-Expires': '600', + 'X-Tos-SignedHeaders': 'host', + 'X-Tos-Signature': '4cba3a3c67f8da33a72d5b87c0c264715edfc56552c35a4cff81eed169c3f7c0', + }) + }) + + it('non-ASCII key with a space', () => { + at('2026-09-01T00:00:00.000Z') + const url = new URL(tosSignUrl(opts, '目录/a b.txt', { expires: 600 })) + expect(decodeURIComponent(url.pathname)).toBe('/目录/a b.txt') + expect(url.searchParams.get('X-Tos-Signature')).toBe( + '9fc2461247b09863d0fe8bac0d545b07206fad757629a69da8adf0124f59a4d8', + ) + }) + + it('defaults to 1800s validity like the other entries', () => { + at('2026-09-01T00:00:00.000Z') + const url = new URL(tosSignUrl(opts, 'exampleobject')) + expect(url.searchParams.get('X-Tos-Expires')).toBe('1800') + }) + + it('signs the credential scope with the region, not the endpoint', () => { + at('2026-09-01T00:00:00.000Z') + const url = new URL( + tosSignUrl({ ...opts, endpoint: `tos-${REGION}.volces.com` }, 'exampleobject', { + expires: 600, + }), + ) + expect(url.host).toBe(`${BUCKET}.tos-${REGION}.volces.com`) + expect(url.searchParams.get('X-Tos-Credential')).toBe(`${AK}/20260901/${REGION}/tos/request`) + }) + + it('carries the security token in the signed query', () => { + at('2026-09-01T00:00:00.000Z') + const url = new URL(tosSignUrl({ ...opts, stsToken: TOKEN }, 'exampleobject', { expires: 60 })) + expect(url.searchParams.get('X-Tos-Security-Token')).toBe(TOKEN) + }) + + it('maps response-* and process options into the query', () => { + at('2026-09-01T00:00:00.000Z') + const url = new URL( + tosSignUrl(opts, 'exampleobject', { + expires: 60, + process: 'image/resize,w_100', + response: { 'content-type': 'text/plain', 'content-disposition': 'attachment' }, + }), + ) + expect(url.searchParams.get('x-tos-process')).toBe('image/resize,w_100') + expect(url.searchParams.get('response-content-type')).toBe('text/plain') + expect(url.searchParams.get('response-content-disposition')).toBe('attachment') + }) +}) + +describe('tosUriEscape', () => { + it('escapes the RFC 3986 chars encodeURIComponent leaves alone', () => { + expect(tosUriEscape("!'()*")).toBe('%21%27%28%29%2A') + }) + + it('encodes spaces and slashes', () => { + expect(tosUriEscape('a b/c')).toBe('a%20b%2Fc') + }) +}) + +describe('resolveTosHost', () => { + const base: Options = { accessKeyId: AK, accessKeySecret: SK, bucket: BUCKET, region: REGION } + + it('builds the virtual-hosted region host', () => { + expect(resolveTosHost(base)).toBe(`${BUCKET}.tos-${REGION}.volces.com`) + }) + + it('uses the internal network domain when internal is set', () => { + expect(resolveTosHost({ ...base, internal: true })).toBe(`${BUCKET}.tos-${REGION}.ivolces.com`) + }) + + it('prefixes the bucket to a custom endpoint domain', () => { + expect(resolveTosHost({ ...base, endpoint: 'tos-cn-shanghai.volces.com' })).toBe( + `${BUCKET}.tos-cn-shanghai.volces.com`, + ) + }) + + it('requires a region and a bucket, even with a custom endpoint', () => { + expect(() => resolveTosHost({ ...base, region: undefined })).toThrow(/region/) + expect(() => + resolveTosHost({ ...base, region: undefined, endpoint: 'tos.example.com' }), + ).toThrow(/region/) + expect(() => resolveTosHost({ ...base, bucket: undefined })).toThrow(/bucket/) + }) +}) + +describe('TOS request building', () => { + function captureTransport() { + const original = getTransport() + const calls: Array<{ url: string; options: any }> = [] + setTransport(async (url, options) => { + calls.push({ url, options }) + return { data: '', headers: {}, status: 200, statusText: 'OK' } + }) + return { calls, restore: () => setTransport(original) } + } + + const base: Options = { + accessKeyId: AK, + accessKeySecret: SK, + bucket: BUCKET, + region: REGION, + secure: true, + } + + it('builds the virtual-hosted URL and lower-cases the header names', async () => { + const { calls, restore } = captureTransport() + try { + await tosRequest(base, { + verb: 'PUT', + objectName: 'dir/a b.txt', + headers: { 'Content-Md5': 'abc==' }, + }) + } finally { + restore() + } + expect(calls[0].url).toBe(`https://${BUCKET}.tos-${REGION}.volces.com/dir/a%20b.txt`) + expect(calls[0].options.headers.authorization).toMatch( + new RegExp( + `^TOS4-HMAC-SHA256 Credential=${AK}\\/\\d{8}\\/${REGION}\\/tos\\/request, SignedHeaders=host;x-tos-content-sha256;x-tos-date, Signature=[0-9a-f]{64}$`, + ), + ) + expect(calls[0].options.headers['content-md5']).toBe('abc==') + expect(calls[0].options.headers['x-tos-content-sha256']).toBe('UNSIGNED-PAYLOAD') + }) + + it('adds the STS token as a signed x-tos-security-token header', async () => { + const { calls, restore } = captureTransport() + try { + await tosRequest({ ...base, stsToken: TOKEN }, { verb: 'GET', objectName: 'x' }) + } finally { + restore() + } + expect(calls[0].options.headers['x-tos-security-token']).toBe(TOKEN) + expect(calls[0].options.headers.authorization).toContain( + 'SignedHeaders=host;x-tos-content-sha256;x-tos-date;x-tos-security-token', + ) + }) + + it('sorts and encodes the query string exactly like the signature', async () => { + const { calls, restore } = captureTransport() + try { + await tosRequest(base, { + verb: 'PUT', + objectName: 'bigfile.bin', + subResource: { uploadId: 'u1', partNumber: '1' }, + }) + } finally { + restore() + } + expect(calls[0].url).toBe( + `https://${BUCKET}.tos-${REGION}.volces.com/bigfile.bin?partNumber=1&uploadId=u1`, + ) + expect(calls[0].options.headers.authorization).toContain( + 'SignedHeaders=host;x-tos-content-sha256;x-tos-date', + ) + }) + + it('prefixes the bucket to a custom endpoint domain', async () => { + const { calls, restore } = captureTransport() + try { + await tosRequest( + { ...base, endpoint: 'tos-cn-shanghai.volces.com' }, + { verb: 'GET', objectName: 'x' }, + ) + } finally { + restore() + } + expect(calls[0].url).toBe(`https://${BUCKET}.tos-cn-shanghai.volces.com/x`) + }) +}) diff --git a/test/tos.spec.ts b/test/tos.spec.ts new file mode 100644 index 0000000..72e3708 --- /dev/null +++ b/test/tos.spec.ts @@ -0,0 +1,93 @@ +import { describe, it, expect, afterEach } from 'vitest' +import { put, putSymlink, multipartUpload, uploadPartCopy, type Options } from '../src/tos/index' +import { setTransport, getTransport } from '../src/transport' + +const OPTIONS: Options = { + accessKeyId: 'AKIDEXAMPLE', + accessKeySecret: 'SECRETKEY', + region: 'cn-beijing', + bucket: 'examplebucket', + secure: true, +} + +function capture() { + const calls: Array<{ url: string; options: any }> = [] + setTransport(async (url, options) => { + calls.push({ url, options }) + return { data: '', headers: {}, status: 200, statusText: 'OK' } + }) + return calls +} + +afterEach(() => { + setTransport(getTransport()) +}) + +describe('TOS entry point', () => { + it('put signs with the TOS4 scheme and sends Content-Md5/Content-Type', async () => { + const calls = capture() + await put(OPTIONS, 'dir/exampleobject.txt', new Blob(['hello'], { type: 'text/plain' })) + const { url, options } = calls[0] + expect(url).toBe(`https://examplebucket.tos-cn-beijing.volces.com/dir/exampleobject.txt`) + expect(options.headers.authorization).toMatch(/^TOS4-HMAC-SHA256 Credential=AKIDEXAMPLE\//) + expect(options.headers['content-md5']).toBeTruthy() + expect(options.headers['content-type']).toBe('text/plain') + }) + + it('putSymlink sends x-tos-symlink-target and ?symlink', async () => { + const calls = capture() + await putSymlink(OPTIONS, 'link.txt', 'dir/target.txt') + expect(calls[0].url).toBe('https://examplebucket.tos-cn-beijing.volces.com/link.txt?symlink=') + expect(calls[0].options.headers['x-tos-symlink-target']).toBe('dir/target.txt') + expect(calls[0].options.headers.authorization).toContain('x-tos-symlink-target') + }) + + it('uploadPartCopy sends the TOS copy-source headers', async () => { + const calls = capture() + await uploadPartCopy(OPTIONS, 'bigfile.bin', 'u1', 2, 'bytes=0-1023', { + sourceKey: 'dir/source.bin', + }) + expect(calls[0].url).toBe( + 'https://examplebucket.tos-cn-beijing.volces.com/bigfile.bin?partNumber=2&uploadId=u1', + ) + expect(calls[0].options.headers['x-tos-copy-source']).toBe('/examplebucket/dir%2Fsource.bin') + expect(calls[0].options.headers['x-tos-copy-source-range']).toBe('bytes=0-1023') + }) + + it('multipartUpload drives init/upload/complete through the TOS protocol', async () => { + const calls: Array<{ url: string; options: any }> = [] + setTransport(async (url, options) => { + calls.push({ url, options }) + if (options.method === 'POST' && url.includes('uploads=')) { + return { + data: 'u1', + headers: {}, + status: 200, + statusText: 'OK', + } + } + if (options.method === 'POST' && url.includes('uploadId=u1')) { + return { + data: '"done"', + headers: {}, + status: 200, + statusText: 'OK', + } + } + return { data: '', headers: { etag: '"part1"' }, status: 200, statusText: 'OK' } + }) + const result = await multipartUpload(OPTIONS, 'bigfile.bin', new Blob(['hello'])) + expect(result.name).toBe('bigfile.bin') + expect(result.etag).toBe('"done"') + const uploads = calls.filter((c) => c.options.method === 'POST' && c.url.includes('uploads=')) + const parts = calls.filter((c) => c.options.method === 'PUT') + expect(uploads.length).toBe(1) + expect(parts.length).toBeGreaterThanOrEqual(1) + expect(parts[0].url).toContain('partNumber=1&uploadId=u1') + }) + + it('exports putSymlink (TOS has a symlink API)', async () => { + const mod = await import('../src/tos/index') + expect(typeof mod.putSymlink).toBe('function') + }) +}) diff --git a/tsup.config.ts b/tsup.config.ts index d9f069b..c52645d 100644 --- a/tsup.config.ts +++ b/tsup.config.ts @@ -19,6 +19,7 @@ export default defineConfig({ 'tiny-oss.es': 'src/index.ts', 'tiny-oss.cos.es': 'src/cos/index.ts', 'tiny-oss.obs.es': 'src/obs/index.ts', + 'tiny-oss.tos.es': 'src/tos/index.ts', 'tiny-oss.aws.es': 'src/aws/index.ts', 'tiny-oss.azure.es': 'src/azure/index.ts', 'tiny-oss.protocol.es': 'src/provider.ts',