From fd164f0e98d14824cf431c66cd3596c170d4f9b7 Mon Sep 17 00:00:00 2001 From: zhaoshangzi Date: Wed, 22 Jul 2026 11:14:56 +0800 Subject: [PATCH 1/4] add tiered storage contents 1 --- TOC-tidb-cloud.md | 6 + tidb-cloud/tieredstorage_concepts.md | 174 ++++++++++++++ tidb-cloud/tieredstorage_faq.md | 49 ++++ tidb-cloud/tieredstorage_limitations.md | 94 ++++++++ tidb-cloud/tieredstorage_operations.md | 305 ++++++++++++++++++++++++ 5 files changed, 628 insertions(+) create mode 100644 tidb-cloud/tieredstorage_concepts.md create mode 100644 tidb-cloud/tieredstorage_faq.md create mode 100644 tidb-cloud/tieredstorage_limitations.md create mode 100644 tidb-cloud/tieredstorage_operations.md diff --git a/TOC-tidb-cloud.md b/TOC-tidb-cloud.md index f7aaa2d20d8fa..f14b94b0dee17 100644 --- a/TOC-tidb-cloud.md +++ b/TOC-tidb-cloud.md @@ -16,6 +16,7 @@ - Key Concepts - [Overview](/tidb-cloud/key-concepts.md) - [Architecture](/tidb-cloud/architecture-concepts.md) + - [Tiered Storage](/tidb-cloud/tieredstorage_concepts.md) - [Database Schema](/tidb-cloud/database-schema-concepts.md) - [Transactions](/tidb-cloud/transaction-concepts.md) - [SQL](/tidb-cloud/sql-concepts.md) @@ -74,6 +75,11 @@ - [Migrate Datadog and New Relic Integrations](/tidb-cloud/migrate-metrics-integrations.md) - [Migrate Prometheus Integrations](/tidb-cloud/migrate-prometheus-metrics-integrations.md) - [TiDB Cloud Clinic](/tidb-cloud/tidb-cloud-clinic.md) + - Tiered Storage + - [Concept](/tidb-cloud/tieredstorage_concepts.md) + - [Management](/tidb-cloud/tieredstorage_operations.md) + - [Limitation](tidb-cloud/tieredstorage_limitations.md) + - [FAQ](tidb-cloud/tieredstorage_faq) - Tune Performance - [Overview](/tidb-cloud/tidb-cloud-tune-performance-overview.md) - Analyze Performance diff --git a/tidb-cloud/tieredstorage_concepts.md b/tidb-cloud/tieredstorage_concepts.md new file mode 100644 index 0000000000000..fe2b46cec4a63 --- /dev/null +++ b/tidb-cloud/tieredstorage_concepts.md @@ -0,0 +1,174 @@ +# Tiered Storage Concepts + +> **Version**: Private Preview +> **Platform**: TiDB Cloud Essential +> **Document Nature**: **This document reflects the current system state only. Some behaviors may change when the feature reaches GA.** +> **Updated**: 2026-07-22 + +--- + +## 1 What is Tiered Storage + +Tiered Storage is a **table-level and partition-level storage tiering capability** on TiDB Cloud Essential, designed for infrequently accessed data. Users can set a table or partition to the IA (Infrequent Access) storage class. The system automatically stores the full data in remote object storage (S3/OSS, etc.), keeping only metadata and on-demand cached hot data segments locally. + +**In a nutshell**: An IA table is still a regular table from the application layer — all query, transaction, backup, and recovery semantics remain unchanged. The difference lies in cost and performance: local disk usage drops significantly, but on a cold read (when the local cache is missing), data must be fetched from remote object storage, resulting in higher latency than Standard tables. + +Key features: + +- **Semantic transparency**: All SQL operations — SELECT, INSERT, UPDATE, DELETE — behave identically +- **Cost optimization**: Data is fully stored in low-cost object storage, with only cached hot data locally; expected storage cost reduction of approximately 50% +- **Fine-grained control**: Supports both table-level and partition-level granularity; partition-level settings override table-level configuration +- **Elastic switching**: Supports bidirectional IA ↔ Standard conversion with no data loss +- **Deep integration**: Tightly integrated with Raft regions, MVCC, BR backup & restore, TiCDC, etc. + +--- + +## 2 Usage Scenario Decisions + +### 2.1 Recommended Scenarios + +| Business Scenario | Data Characteristics | Recommended Cold/Hot Boundary | +|-|-|-| +| E-commerce historical orders / financial transactions | Frequent writes, heavy read/write of recent data, rare historical modifications | 6 months | +| Financial vouchers / audit logs / invoices | Write-once, rarely modified, 7-15 year retention | 2 years | +| Application logs / monitoring metrics / API calls | High-throughput writes, frequent recent troubleshooting | 90 days | +| Social media posts / comments / photo metadata | Initial peak, declining over time | 30-90 days | +| Industrial sensors / connected vehicles | Very high write throughput, real-time recent monitoring | 1 year | +| Data warehouse historical analysis | Bulk writes, few updates, historical trend analysis | 90 days | +| AI dialogue history / memory | Session-based writes, high-frequency recent access, historical user profiling | 180 days | +| LLM training datasets | High frequency during training, steep drop-off after completion | Training end + 30 days | +| AI inference logs / results | High-concurrency writes, recent monitoring, historical optimization | 90 days | +| Vector databases | Rare updates, frequent recent queries | 30-90 days | + +**General rule of thumb**: Large data volume + decreasing access frequency over time + occasional queries with no strict response time requirement → suitable for IA. + +### 2.2 Not Recommended Scenarios + +- **Hot OLTP tables** with strict latency sensitivity (core online transaction tables sensitive to every millisecond) +- Datasets requiring **frequent large-range scans** (AP queries that continuously access large amounts of cold data) +- Tables that need **frequent switching** between IA and Standard +- Data with **highly scattered** access patterns and almost no locality +- Scenarios where the access pattern does not follow a "decreasing over time" trend + +### 2.3 Decision Checklist + +Before setting IA, verify each item: + +- [ ] The data access frequency of the table/partition has been confirmed to be declining from a business perspective +- [ ] The table can be changed to a partitioned table, because cold/hot data separation is easier to manage with partitioned tables +- [ ] For regular table cold/hot separation, hot data accounts for less than 10% of the table +- [ ] Cold data access frequency is very low, e.g., query QPS does not exceed 10 concurrent (to avoid saturating object storage bandwidth) +- [ ] No need for frequent large-range AP scans on IA tables +- [ ] Cold read latency is acceptable (a single SQL execution may involve multiple TiKV remote requests; each remote request adds about 500ms~2s), meaning a single-row index lookup on cold data adds approximately +500ms~2s +- [ ] Awareness that cold reads have read amplification: a single 100KB record can exhibit up to 30,000× amplification (3 MiB cold data) +- [ ] Single query involves no more than 100 MiB of cold data +- [ ] Single query accesses no more than 100 rows of cold data +- [ ] An observation period has been planned (at least one full business day for the first partition) +- [ ] Awareness that switching back IA → Standard takes a long time with significant bandwidth cost + +--- + +## 3 Implementation Principles + +### 3.1 Architecture Overview + +Tiered Storage implementation spans the TiDB → TiKV → Object Store three-layer architecture: + +```Plaintext +┌──────────────────────────────────────────────────────────────────┐ +│ TiDB Schema Layer │ +│ · STORAGE_CLASS / ENGINE_ATTRIBUTE written to TiDB schema │ +└──────────────────────────────────────────────────────────────────┘ + ↓ +┌──────────────────────────────────────────────────────────────────┐ +│ TiKV Region Management Layer │ +│ · IA tables/partitions occupy dedicated regions, avoiding │ +│ hot/cold data in the same shard │ +└──────────────────────────────────────────────────────────────────┘ + ↓ +┌──────────────────────────────────────────────────────────────────┐ +│ TiKV IA Cache Management Layer (IaManager) │ +│ · Local cache for cold storage data │ +└──────────────────────────────────────────────────────────────────┘ + ↓ +┌──────────────────────────────────────────────────────────────────┐ +│ Remote Object Storage (S3/OSS) │ +│ · Full SST data organized and stored by segment │ +└──────────────────────────────────────────────────────────────────┘ +``` + +**Raft ChangeSet ensures consistency**: Storage class changes are replicated to all replicas via Raft ChangeSet. This ensures that every related shard maintains a consistent storage class state across restarts, recovery, splits, and merges, preventing data loss or corruption due to region topology changes. + +### 3.2 Data Storage Hierarchy + +The internal hierarchy of an SSTable from top to bottom: + +```Plaintext +SSTable ──→ Segment ──→ Block ──→ KV Pair +(file) (1MB) (32KB) (single record) +``` + +| Layer | Default Size | Role | +|-|-|-| +| **Segment** | 1 MB | The **minimum unit** TiKV reads from object storage; avoids excessive small requests that could incur S3 API call costs and QPS throttling | +| **Block** | 32 KB | The basic unit for local file reading, compression, and **memory/disk caching** | + +This means a cache miss does not fetch just a single KV record — it loads an entire Segment to local storage. If subsequent queries hit data within the same segment, they benefit from hot-read performance. However, for one-time queries with no subsequent access, the read amplification penalty is relatively high. + +### 3.3 Read Amplification Analysis + +**Read amplification path on a cache miss**: + +```Plaintext +User queries 1 record (100 Bytes) +→ Block cache miss +→ TiKV loads segments from 3 LSM levels from S3 +→ Approximately 3 MB data fetched from S3 to local +→ Read amplification: ~30,000× +``` + +This amplification manifests differently in two scenarios: + +- **With subsequent reuse**: The loaded segment stays in the IA cache; subsequent hits become hot reads, amortizing the initial amplification cost +- **One-time query**: e.g., an ad-hoc analysis running a large-range scan — the loaded data is never reused, making the read cost very high. Additionally, newly loaded data evicts "old data" from the local cache — this old data might be real hot spots, and their eviction can trigger new cache misses, creating a cascading performance impact + +Therefore, Tiered Storage is best suited for **small, concentrated query patterns** rather than frequent large-range scans. + +### 3.4 LSM-Tree Write Path + +The write path remains the same as Standard tables: + +```Plaintext +INSERT/UPDATE/DELETE +→ Memtable (hot write, unaffected by IA) +→ L0 SST (hot write, unaffected by IA) +→ L1+ SST (after compaction, opened in IA mode based on storage class) +``` + +- Writes still enter memtable/L0 first — the local hot path does not directly become a remote write +- L1+ files matching IA conditions are opened in IA mode after reload/compaction +- Hot write paths like `memtable` and `L0` do not directly enter IA +- The primary IA target is the Write CF L1+ layer + +--- + +## 4 Conversion Efficiency + +### 4.1 Standard → IA + +Test reference: Approximately 1 TB of logical data (including indexes) completed conversion within 5 minutes, with negligible impact on QPS and latency during the process. Single TiKV CPU increased by about 0.5c and recovered within approximately 5 minutes. + +```Plaintext +TiDB schema takes effect → Schema Manager sync (30s) → TiKV broadcast +→ Region alignment / Split → ChangeSet updates Shard → Reload files +``` + +### 4.2 IA → Standard + +Test reference: 2.09 TB of logical data (including indexes) took approximately 3 hours 10 minutes (~1.61k regions/hour per TiKV), S3 GET throughput was approximately 1.6 GiB/s. During conversion, Standard partition QPS dropped by about 3.78%, P99 increased by about 18.63%, and single TiKV CPU increased by about 0.5c. + +```Plaintext +Same chain as above + full S3 data download to local +``` + +> These figures are from test environments and do not represent real-world production scenarios. Customers should obtain accurate data based on their own business testing. diff --git a/tidb-cloud/tieredstorage_faq.md b/tidb-cloud/tieredstorage_faq.md new file mode 100644 index 0000000000000..ff8ad90fdc580 --- /dev/null +++ b/tidb-cloud/tieredstorage_faq.md @@ -0,0 +1,49 @@ +# Tiered Storage FAQ + +> **Version**: Private Preview +> **Platform**: TiDB Cloud Essential +> **Document Nature**: **This document reflects the current system state only. Some behaviors may change when the feature reaches GA.** +> **Updated**: 2026-07-22 + +--- + +**Q1: Can IA tables execute UPDATE / DELETE?** + +A: Yes. An UPDATE operation first loads the corresponding data from S3 into the IA cache, performs the modification, and writes a new SST file — the same flow as a regular UPDATE. Performance is affected by cold reads. + +--- + +**Q2: Can TiFlash replicas of IA tables be set to IA?** + +A: No. TiFlash does not follow the IA attribute of the source table. + +--- + +**Q3: What happens to IA tables when the object store (S3) experiences an outage?** + +A: IA tables will be affected and become unavailable — since all data resides remotely, read requests must fetch from S3. Additionally, if S3 bandwidth is saturated, IA read/write performance will also be impacted. + +--- + +**Q4: Can the system tell me which data is cold before I set IA?** + +A: TiDB does not provide built-in cold/hot detection tools. Users need to assess data access patterns based on their own business knowledge. A general rule of thumb: for time-partitioned tables, older partitions tend to have lower access frequency. + +--- + +**Q5: When data is stored in cold storage (IA tier), are all three replicas stored, or just one copy?** + +A: All three replicas are stored in cold storage, not just one. + +TiKV's Raft three-replica mechanism is identical between the IA and Standard layers — what changes is the data storage location and format: + +- **Standard layer**: Three replicas are each stored on the local disks of three TiKV nodes +- **IA layer**: Three replicas are each uploaded to object storage independently in IA format + +Each replica on each TiKV node runs its own independent LSM-Tree, performing independent flush and compaction operations. When a table switches to IA storage class, all subsequently generated SST files are written to S3 in IA type. The three replicas produce their own independent SST files — three separate objects in S3, not shared. + +IA is a **storage format/location optimization**, not a **replica reduction mechanism**. It changes "how each node stores its own copy," not "how many copies exist." The Raft write and replication flow is identical to the Standard layer. + +**Cost impact**: Since all three replicas upload independently, the total storage on S3 remains approximately 3× the data volume. However, the unit storage cost of STANDARD_IA is lower, so the overall storage expense is reduced. + +--- diff --git a/tidb-cloud/tieredstorage_limitations.md b/tidb-cloud/tieredstorage_limitations.md new file mode 100644 index 0000000000000..b1c0cf4b1deff --- /dev/null +++ b/tidb-cloud/tieredstorage_limitations.md @@ -0,0 +1,94 @@ +# Tiered Storage Limitations and Consequences + +> **Version**: Private Preview +> **Platform**: TiDB Cloud Essential +> **Document Nature**: **This document reflects the current system state only. Some behaviors may change when the feature reaches GA.** +> **Updated**: 2026-07-22 + +--- + +## 1 Feature Limitations + +| Limitation | Description | +|-|-| +| Hash / Key partitions | Cannot be set to IA | +| Index-independent setting | Cannot set IA on indexes independently | +| TTL auto-tiering | Automatic cold/hot tiering based on business fields is not supported | +| Syntax conflict | `STORAGE_CLASS` and `ENGINE_ATTRIBUTE` cannot be specified simultaneously | +| Partition selector mixing | `names_in` / `less_than` / `values_in` cannot be used simultaneously | +| TiFlash | Does not follow IA; data always remains local | + +--- + +## 2 Access Throttling Constraints + +Since shared physical clusters have limited object storage bandwidth, IA cold storage access must comply with the following limits: + +| Constraint Dimension | Limit | Reason | +|-|-|-| +| Single SQL cold read throughput | ≤ 100 MiB/s | Prevents one query from consuming excessive bandwidth | +| Total concurrent cold read throughput | ≤ 1 GiB/s (≤ 10 concurrent) | Protects other tenants in the cluster | +| TiKV single miss load size | ≤ ~3 MiB (estimated) | Segments from 3 LSM levels | + +**Note**: A single TiKV miss does not represent a single query miss! For example, a query may involve multiple (e.g., 1000) TiKV misses. If 5 TiKV nodes serve the query, each TiKV averages 200 misses, leading to 200 remote cold data queries. Although concurrent within each TiKV, 200 misses take a very long time, so the final query latency can be extremely high. + +**If your business involves sustained heavy access to cold data, IA is not recommended; revert the table to Standard storage.** To ensure system stability, hard throttling for cold data access will be added in a future technical release. For now, users must comply with the constraints above. + +You can monitor single SQL cold data access volume via the `IA Remote Read Segment Size` panel in Cloud Console → Monitoring → Diagnosis → Slow Query → Coprocessor. + +--- + +## 3 Impact on Peripheral Tools + +| Tool | Impact | Compatibility | +|-|-|-| +| **TiCDC** | Logical data semantics unchanged; init scan/reading old data may have higher latency; region changes handled as normal | Compatible | +| **BR backup & restore** | Preserves storage class metadata; IA tables continue to load under IA semantics after restore | Compatible | +| **IMPORT INTO** | Imported data transitions to IA via flush/compaction; large-range validation after import may encounter cold cache | Compatible | +| **PITR** | Preserves storage class metadata; schema manager re-syncs after restore | Compatible | + +--- + +## 4 Risk Isolation Mechanisms + +After setting IA on a table, the system uses the following measures to isolate IA and non-IA tables: + +**Region Level**: + +- IA tables/partitions occupy dedicated regions, triggering necessary splits +- Adjacent regions with different storage classes are restricted from merging, preventing hot/cold data mixing +- A single region is either entirely IA or entirely non-IA + +**Compute Layer**: + +- Standard and IA tables have no isolation at the compute layer — TiDB does not have separate isolation strategies for the two table types + +**Storage Layer**: + +- Cold read rate limiting + +> Shared resources (CPU, network, local disk, object storage bandwidth) cannot be fully isolated. In extreme cases, a very large IA scan may still affect other tenants. This is the fundamental reason for the access throttling constraints. + +--- + +## 5 Emergency Recovery Methods + +If issues arise with IA tables, users and the TiDB Cloud team can use the following methods: + +| Method | Scenario | Priority | Description | +|-|-|-|-| +| IA → Standard switch-back | User finds performance unacceptable | **User's primary choice** | System reloads data locally, bypassing the remote path | +| **Flow Control (already available)** | Control traffic between IA tables and S3 | Backend's primary choice | Rate-limiting protects cluster stability; managed by the TiDB Cloud team | + +--- + +## 6 IA Local Cache and Query Performance Uncertainty + +Tiered Storage maintains a local IA data cache (managed by IaManager) to accelerate repeated access to recently accessed cold data. However, the following key facts should be understood: + +- **Cache behavior is system-controlled, not user-configurable**: Cache size and eviction policies are managed uniformly by the system. Users cannot adjust cache capacity or specify which data should stay in the cache. Cache hit rates depend on actual access patterns — concentrated access can exceed 90%, while scattered access may fall below 90%. Even if only one partition is set to IA, its local cache behavior is still system-managed — users cannot exercise fine-grained control. +- **IA query response time is non-deterministic**: When a query hits the local cache, performance is close to Standard tables. However, when data must be loaded from remote object storage (cache miss), each remote request adds approximately 500ms~2s of latency. A single SQL execution may involve multiple remote loads, causing latency to accumulate. Therefore, IA table query response times are not as predictable as Standard tables — the business side should plan accordingly. +- **Recommended: use partitioned tables to precisely control cold data scope**: Use partitioned tables, setting only confirmed low-frequency historical partitions to IA while keeping active partitions as Standard. This limits the cache uncertainty to a well-defined data range, rather than exposing the entire table's query performance to cache miss risk. +- **Increasing cache space means increasing cost**: In a future product iteration, we will offer the option for customers to configure larger local cache space for IA tables to improve cold data access efficiency. However, larger cache space requires additional TiKV nodes and local disk resources, incurring corresponding resource costs. Customers will be able to balance performance and cost according to their business needs. + +In short: IA storage trades lower cost for query performance uncertainty — this is an inherent design trade-off. Use partitioned tables to precisely manage cold data boundaries and confine this uncertainty to a well-defined scope. If your business has strict predictability requirements for query response times, keep that portion of data in Standard storage. diff --git a/tidb-cloud/tieredstorage_operations.md b/tidb-cloud/tieredstorage_operations.md new file mode 100644 index 0000000000000..711ef0a54a972 --- /dev/null +++ b/tidb-cloud/tieredstorage_operations.md @@ -0,0 +1,305 @@ +# Tiered Storage Operations Guide + +> **Version**: Private Preview +> **Platform**: TiDB Cloud Essential +> **Document Nature**: **This document reflects the current system state only. Some behaviors may change when the feature reaches GA.** +> **Updated**: 2026-07-22 + +--- + +## 1 How to Use + +### 1.1 Storage Class Support Matrix + +#### Storage Class Values + +| Value | Meaning | Default | +|-|-|-| +| `Standard` | Local hot storage, full data on local disk | Yes | +| `IA` | Remote cold storage, full data in object storage, local on-demand caching | No | + +Values are case-insensitive. + +#### Supported Table Types + +| Table Type | IA Support | Notes | +|-|-|-| +| Regular non-partitioned table | Supported | Via `STORAGE_CLASS` syntactic sugar or `ENGINE_ATTRIBUTE` | +| Range partitioned table | Supported | Must use `ENGINE_ATTRIBUTE` | +| Range Columns partitioned table | Supported | Must use `ENGINE_ATTRIBUTE` | +| List partitioned table | Supported | Must use `ENGINE_ATTRIBUTE` | +| List Columns partitioned table | Supported | Must use `ENGINE_ATTRIBUTE` | +| Hash partitioned table | **Not supported** | — | +| Key partitioned table | **Not supported** | — | + +#### Storage Type Inheritance Rules for Indexes and Related Objects + +| Object | Inheritance Rule | +|-|-| +| Regular table index | Same as the table | +| Partitioned table Local Index | Same as the owning partition | +| Partitioned table Global Index | Same as the table-level setting | +| TiFlash | **Does not follow table storage settings** | + +### 1.2 Regular Table DDL + +#### Specifying at Create Time + +Syntactic sugar (recommended): + +```SQL +CREATE TABLE t_ia ( + id BIGINT PRIMARY KEY, + created_at DATETIME NOT NULL, + payload VARCHAR(256) NOT NULL +) ENGINE=InnoDB STORAGE_CLASS='IA'; +``` + +`ENGINE_ATTRIBUTE` method: + +```SQL +CREATE TABLE t_ia ( + id BIGINT PRIMARY KEY +) ENGINE_ATTRIBUTE='{"storage_class":"IA"}'; +``` + +> **Conflict constraint**: `STORAGE_CLASS` syntactic sugar and `ENGINE_ATTRIBUTE`'s `storage_class` **cannot be specified together** — the system will reject with an error. + +#### Modifying an Existing Table + +```SQL +-- Standard → IA +ALTER TABLE t1 STORAGE_CLASS='IA'; +ALTER TABLE t1 ENGINE_ATTRIBUTE='{"storage_class":"IA"}'; + +-- IA → Standard +ALTER TABLE t1 STORAGE_CLASS='STANDARD'; +ALTER TABLE t1 ENGINE_ATTRIBUTE='{"storage_class":"STANDARD"}'; +``` + +ALTER operations preserve all data access, and SQL reads/writes are not affected during the conversion. + +### 1.3 Partitioned Table DDL + +Partitioned tables **do not support** the `STORAGE_CLASS` syntactic sugar and must use `ENGINE_ATTRIBUTE`. + +Partition attributes support three selector types (cannot be mixed) plus a table-level default: + +| Configuration Method | Syntax | Applicable Partition Types | Purpose | +|-|-|-|-| +| Table-level default | `{"storage_class":"IA"}` | All | Set all partitions to IA uniformly | +| By partition name | `"names_in":["p1","p2"]` | All | Specify an exact list of partition names | +| By range | `"less_than":"2024-01-01"` | RANGE / RANGE COLUMNS | Match partitions by boundary value | +| By list value | `"values_in":["1","2"]` | LIST / LIST COLUMNS | Match partitions by list value | + +#### Example A: Table-level IA with Specific Partitions Overridden to Standard + +```SQL +CREATE TABLE orders ( + order_id BIGINT NOT NULL, + created_at DATETIME NOT NULL, + PRIMARY KEY (order_id, created_at) +) ENGINE_ATTRIBUTE='{ + "storage_class":[ + {"tier":"ia"}, + {"tier":"standard","names_in":["p2025","p_future"]} + ] +}' +PARTITION BY RANGE (YEAR(created_at)) ( + PARTITION p2023 VALUES LESS THAN (2024), + PARTITION p2024 VALUES LESS THAN (2025), + PARTITION p2025 VALUES LESS THAN (2026), + PARTITION p_future VALUES LESS THAN MAXVALUE +); +``` + +Result: p2023 / p2024 → IA, p2025 / p_future → Standard. + +#### Example B: Range Selector + +```SQL +CREATE TABLE users ( + user_id BIGINT NOT NULL, + PRIMARY KEY (user_id) +) ENGINE_ATTRIBUTE='{ + "storage_class":[ + {"tier":"ia","less_than":"2000000"} + ] +}' +PARTITION BY RANGE (user_id) ( + PARTITION p0 VALUES LESS THAN (1000000), + PARTITION p1 VALUES LESS THAN (2000000), + PARTITION p2 VALUES LESS THAN (3000000), + PARTITION p3 VALUES LESS THAN MAXVALUE +); +``` + +Result: p0 / p1 → IA, p2 / p3 → Standard. + +#### Example C: List Value Selector + +```SQL +CREATE TABLE order_status_log ( + log_id BIGINT NOT NULL, + status INT NOT NULL, + PRIMARY KEY (log_id, status) +) ENGINE_ATTRIBUTE='{ + "storage_class":[ + {"tier":"ia","values_in":["1","2"]} + ] +}' +PARTITION BY LIST (status) ( + PARTITION p_pending VALUES IN (1), + PARTITION p_paid VALUES IN (2), + PARTITION p_shipped VALUES IN (3), + PARTITION p_completed VALUES IN (4) +); +``` + +Result: p_pending / p_paid → IA, p_shipped / p_completed → Standard. + +#### Partition Selector Rules + +- **Priority**: Partition-level configuration **overrides** table-level configuration +- **Mutual exclusion**: Multiple matching methods (e.g., `"names_in"` and `"less_than"`) cannot be used together in the same selector — this will raise an error +- **Forward compatibility**: New partitions added later (`ADD PARTITION` / `REORGANIZE PARTITION`) are automatically evaluated against the persistent storage class rules; matching partitions inherit the configuration + +### 1.4 Viewing and Monitoring + +```SQL +-- View DDL definition +SHOW CREATE TABLE t1\G + +-- View table-level storage type +SELECT TABLE_NAME, TIDB_STORAGE_CLASS +FROM INFORMATION_SCHEMA.TABLES +WHERE TABLE_SCHEMA = 'your_database' + AND TABLE_NAME = 'your_table'; + +-- View partition-level storage type +SELECT PARTITION_NAME, TIDB_STORAGE_CLASS +FROM INFORMATION_SCHEMA.PARTITIONS +WHERE TABLE_SCHEMA = 'your_database' + AND TABLE_NAME = 'your_table'; +``` + +#### Monitoring IA Storage Space + +View in Cloud Console: + +- **Path**: Cloud Console → Monitoring → Metrics → Instance Overview (or Overview → Core Metrics) +- **New metrics**: + - `Row-based IA Storage` — Total IA table space + - `Row-based Standard Storage` — Total Standard table space +- **Relationship**: `Row-based Storage` = `Row-based IA Storage` + `Row-based Standard Storage` + +The single-table space query method remains unchanged: + +> Note: This method depends on table statistics and may have significant estimation errors. + +```SQL +SELECT TABLE_NAME, + ROUND(DATA_LENGTH / 1024 / 1024, 2) AS Data_MB, + ROUND(INDEX_LENGTH / 1024 / 1024, 2) AS Index_MB, + TABLE_ROWS +FROM INFORMATION_SCHEMA.TABLES +WHERE TABLE_SCHEMA = 'your_database' + AND TABLE_NAME = 'your_table' +ORDER BY (DATA_LENGTH + INDEX_LENGTH) DESC; +``` + +--- + +## 2 Observability + +### 2.1 EXPLAIN ANALYZE + +When a query involves remote data loading, the `scan_detail` includes new fields: + +```SQL +EXPLAIN ANALYZE SELECT * FROM t_ia WHERE id BETWEEN 1 AND 50000; +-- The output includes: +-- ia_remote_read_segment_size: 2320453 -- Total bytes loaded remotely +-- ia_remote_read_segment_count: 3 -- Number of remote loading events +-- ia_remote_read_segment_wait_time: 0.008 -- Remote wait time (seconds) +``` + +> Note: The IA signal is per-request read path evidence, not a table-level stable flag — the same query may show IA information on the first run but not after a cache hit. +> +> Additionally, ia_remote_read_segment_wait_time is the aggregate time of all remote requests. Due to TiKV's underlying parallel reading mechanism, this value may exceed the SQL's actual execution time. + +### 2.2 Statement Summary + +`STATEMENTS_SUMMARY_HISTORY` and `CLUSTER_STATEMENTS_SUMMARY_HISTORY` have 6 new columns: + +| Column Name | Description | +|-|-| +| `AVG_IA_REMOTE_READ_SEGMENT_COUNT` | Average number of remote segments read | +| `MAX_IA_REMOTE_READ_SEGMENT_COUNT` | Maximum number of remote segments read | +| `AVG_IA_REMOTE_READ_SEGMENT_SIZE` | Average remote read data volume | +| `MAX_IA_REMOTE_READ_SEGMENT_SIZE` | Maximum remote read data volume | +| `AVG_IA_REMOTE_READ_SEGMENT_WAIT_TIME` | Average remote wait time | +| `MAX_IA_REMOTE_READ_SEGMENT_WAIT_TIME` | Maximum remote wait time | + +### 2.3 Slow Queries + +`INFORMATION_SCHEMA.CLUSTER_SLOW_QUERY` has 3 new columns: + +- `IA_remote_read_segment_count` +- `IA_remote_read_segment_size` +- `IA_remote_read_segment_wait_time` + +Corresponding panels are also visible in the Cloud Console slow query details. + +--- + +## 3 Best Practices + +### 3.1 Tiering Strategy: Prefer Partition-Level IA + +For partitioned tables, **always prefer partition-level IA** over table-level IA. This gives you precise control over cold/hot boundaries: + +- Historical cold partitions (e.g., `p2023`) → IA +- Recent hot partitions (e.g., `p2025`) → Standard +- Future partitions (e.g., `p_future`) → Standard + +### 3.2 Rollout Strategy: Start with the Smallest Oldest Partition + +```Plaintext +Step 1: Select the oldest and smallest partition → ALTER PARTITION → IA +Step 2: Observe for one full business day (at least 24h) +Step 3: Verify QPS / TPS / P99 Latency / CPU metrics show no degradation +Step 4: Set the next cold partition → IA one by one +Step 5: Repeat Steps 2-4 until all target partitions are covered +``` + +**Do not batch-set all partitions to IA at once.** + +### 3.3 Write Optimization + +- When importing concurrently into IA partitions, having each thread target a different IA partition reduces lock contention and improves throughput. Test environment measurements: random writes to IA partitions averaged 50k rows/sec; fixed single-thread writes to the same IA partition averaged 70k rows/sec (~40% improvement). +- Newly imported data only transitions to IA mode after flush/compaction. Large-range queries immediately after import may encounter cold cache. + +> These figures are from test environments and do not represent real-world production scenarios. Customers should obtain accurate data based on their own business testing. + +### 3.4 Query Optimization + +- Queries spanning IA partitions are recommended to cover **no more than 3 partitions**; exceeding this may cause significant response time degradation +- Avoid running many `SELECT *` full table scans on IA tables simultaneously +- Monitor IA remote read volume via `EXPLAIN ANALYZE` and slow queries, and adjust accordingly + +### 3.5 Switch-Back Considerations + +- IA → Standard conversion downloads all data from S3, generating significant cold storage bandwidth usage +- Monitor bandwidth usage to ensure smooth operation; if necessary, **contact the TiDB Cloud team in advance** for joint monitoring +- Business SQL reads/writes are not affected during conversion, but performance (e.g., QPS/TPS) may have minor impact — test environment shows less than 5% + +### 3.6 Configuration Stability + +Keep the storage class setting stable and avoid frequent switching between IA and Standard. Each switch triggers: + +- Region reload +- S3 data download or metadata rebuild +- IA cache data flushing + +The cumulative cost of these operations is not negligible. From 5b6ac84ee97f6e218b0df0dd3f5c13e5cc253076 Mon Sep 17 00:00:00 2001 From: zhaoshangzi Date: Wed, 22 Jul 2026 16:48:59 +0800 Subject: [PATCH 2/4] ts update v2 --- TOC-tidb-cloud-byoc.md | 5 ++ TOC-tidb-cloud.md | 6 -- ...concepts.md => tiered-storage-concepts.md} | 58 ++++++++------- tidb-cloud/tiered-storage-faq.md | 45 ++++++++++++ ...tions.md => tiered-storage-limitations.md} | 40 ++++++----- ...ations.md => tiered-storage-operations.md} | 72 ++++++++++--------- tidb-cloud/tieredstorage_faq.md | 49 ------------- 7 files changed, 144 insertions(+), 131 deletions(-) rename tidb-cloud/{tieredstorage_concepts.md => tiered-storage-concepts.md} (81%) create mode 100644 tidb-cloud/tiered-storage-faq.md rename tidb-cloud/{tieredstorage_limitations.md => tiered-storage-limitations.md} (73%) rename tidb-cloud/{tieredstorage_operations.md => tiered-storage-operations.md} (87%) delete mode 100644 tidb-cloud/tieredstorage_faq.md diff --git a/TOC-tidb-cloud-byoc.md b/TOC-tidb-cloud-byoc.md index 5d61a9a90c6eb..07c90e8c38f84 100644 --- a/TOC-tidb-cloud-byoc.md +++ b/TOC-tidb-cloud-byoc.md @@ -69,6 +69,11 @@ - [Subscribe via Email](/tidb-cloud/monitor-alert-email.md) - [Subscribe via Slack](/tidb-cloud/monitor-alert-slack.md) - [Subscribe via Zoom](/tidb-cloud/monitor-alert-zoom.md) + - Tiered Storage + - [Concepts](/tidb-cloud/tiered-storage-concepts.md) + - [Operations](/tidb-cloud/tiered-storage-operations.md) + - [Limitations](/tidb-cloud/tiered-storage-limitations.md) + - [FAQ](/tidb-cloud/tiered-storage-faq.md) - Tune Performance - [Overview](/tidb-cloud/tidb-cloud-tune-performance-overview.md) - [Analyze Performance](/tidb-cloud/tune-performance.md) diff --git a/TOC-tidb-cloud.md b/TOC-tidb-cloud.md index f14b94b0dee17..f7aaa2d20d8fa 100644 --- a/TOC-tidb-cloud.md +++ b/TOC-tidb-cloud.md @@ -16,7 +16,6 @@ - Key Concepts - [Overview](/tidb-cloud/key-concepts.md) - [Architecture](/tidb-cloud/architecture-concepts.md) - - [Tiered Storage](/tidb-cloud/tieredstorage_concepts.md) - [Database Schema](/tidb-cloud/database-schema-concepts.md) - [Transactions](/tidb-cloud/transaction-concepts.md) - [SQL](/tidb-cloud/sql-concepts.md) @@ -75,11 +74,6 @@ - [Migrate Datadog and New Relic Integrations](/tidb-cloud/migrate-metrics-integrations.md) - [Migrate Prometheus Integrations](/tidb-cloud/migrate-prometheus-metrics-integrations.md) - [TiDB Cloud Clinic](/tidb-cloud/tidb-cloud-clinic.md) - - Tiered Storage - - [Concept](/tidb-cloud/tieredstorage_concepts.md) - - [Management](/tidb-cloud/tieredstorage_operations.md) - - [Limitation](tidb-cloud/tieredstorage_limitations.md) - - [FAQ](tidb-cloud/tieredstorage_faq) - Tune Performance - [Overview](/tidb-cloud/tidb-cloud-tune-performance-overview.md) - Analyze Performance diff --git a/tidb-cloud/tieredstorage_concepts.md b/tidb-cloud/tiered-storage-concepts.md similarity index 81% rename from tidb-cloud/tieredstorage_concepts.md rename to tidb-cloud/tiered-storage-concepts.md index fe2b46cec4a63..206aa6537425e 100644 --- a/tidb-cloud/tieredstorage_concepts.md +++ b/tidb-cloud/tiered-storage-concepts.md @@ -1,31 +1,37 @@ +--- +title: Tiered Storage Concepts +summary: Learn about Tiered Storage on TiDB Cloud Essential, including its concepts, architecture, use scenarios, and read amplification behavior. +--- + # Tiered Storage Concepts -> **Version**: Private Preview -> **Platform**: TiDB Cloud Essential -> **Document Nature**: **This document reflects the current system state only. Some behaviors may change when the feature reaches GA.** -> **Updated**: 2026-07-22 +> **Note:** +> +> - **Version:** Private Preview +> - **Platform:** TiDB Cloud Essential +> - This document reflects the current system state only. Some behaviors may change when the feature reaches GA. --- -## 1 What is Tiered Storage +## What is Tiered Storage -Tiered Storage is a **table-level and partition-level storage tiering capability** on TiDB Cloud Essential, designed for infrequently accessed data. Users can set a table or partition to the IA (Infrequent Access) storage class. The system automatically stores the full data in remote object storage (S3/OSS, etc.), keeping only metadata and on-demand cached hot data segments locally. +Tiered Storage is a **table-level and partition-level storage tiering capability** on TiDB Cloud Essential, designed for infrequently accessed data. You can set a table or partition to the IA (Infrequent Access) storage class. The system automatically stores the full data in remote object storage (S3/OSS, etc.), keeping only metadata and on-demand cached hot data segments locally. -**In a nutshell**: An IA table is still a regular table from the application layer — all query, transaction, backup, and recovery semantics remain unchanged. The difference lies in cost and performance: local disk usage drops significantly, but on a cold read (when the local cache is missing), data must be fetched from remote object storage, resulting in higher latency than Standard tables. +In summary, an IA table is still a regular table from the application layer — all query, transaction, backup, and recovery semantics remain unchanged. The difference lies in cost and performance: local disk usage drops significantly, but on a cold read (when the local cache is missing), data must be fetched from remote object storage, resulting in higher latency than Standard tables. Key features: - **Semantic transparency**: All SQL operations — SELECT, INSERT, UPDATE, DELETE — behave identically -- **Cost optimization**: Data is fully stored in low-cost object storage, with only cached hot data locally; expected storage cost reduction of approximately 50% +- **Cost optimization**: Data is fully stored in low-cost object storage, with only hot data cached locally, which is expected to reduce storage costs by approximately 50% in typical scenarios - **Fine-grained control**: Supports both table-level and partition-level granularity; partition-level settings override table-level configuration - **Elastic switching**: Supports bidirectional IA ↔ Standard conversion with no data loss - **Deep integration**: Tightly integrated with Raft regions, MVCC, BR backup & restore, TiCDC, etc. --- -## 2 Usage Scenario Decisions +## Usage scenario decisions -### 2.1 Recommended Scenarios +### Recommended scenarios | Business Scenario | Data Characteristics | Recommended Cold/Hot Boundary | |-|-|-| @@ -42,7 +48,7 @@ Key features: **General rule of thumb**: Large data volume + decreasing access frequency over time + occasional queries with no strict response time requirement → suitable for IA. -### 2.2 Not Recommended Scenarios +### Not recommended scenarios - **Hot OLTP tables** with strict latency sensitivity (core online transaction tables sensitive to every millisecond) - Datasets requiring **frequent large-range scans** (AP queries that continuously access large amounts of cold data) @@ -50,7 +56,7 @@ Key features: - Data with **highly scattered** access patterns and almost no locality - Scenarios where the access pattern does not follow a "decreasing over time" trend -### 2.3 Decision Checklist +### Decision checklist Before setting IA, verify each item: @@ -60,7 +66,7 @@ Before setting IA, verify each item: - [ ] Cold data access frequency is very low, e.g., query QPS does not exceed 10 concurrent (to avoid saturating object storage bandwidth) - [ ] No need for frequent large-range AP scans on IA tables - [ ] Cold read latency is acceptable (a single SQL execution may involve multiple TiKV remote requests; each remote request adds about 500ms~2s), meaning a single-row index lookup on cold data adds approximately +500ms~2s -- [ ] Awareness that cold reads have read amplification: a single 100KB record can exhibit up to 30,000× amplification (3 MiB cold data) +- [ ] Awareness that cold reads have read amplification: a single 100-byte record can exhibit up to 30,000× amplification (approximately 3 MiB of cold data) - [ ] Single query involves no more than 100 MiB of cold data - [ ] Single query accesses no more than 100 rows of cold data - [ ] An observation period has been planned (at least one full business day for the first partition) @@ -68,9 +74,9 @@ Before setting IA, verify each item: --- -## 3 Implementation Principles +## Implementation principles -### 3.1 Architecture Overview +### Architecture overview Tiered Storage implementation spans the TiDB → TiKV → Object Store three-layer architecture: @@ -99,23 +105,23 @@ Tiered Storage implementation spans the TiDB → TiKV → Object Store three-lay **Raft ChangeSet ensures consistency**: Storage class changes are replicated to all replicas via Raft ChangeSet. This ensures that every related shard maintains a consistent storage class state across restarts, recovery, splits, and merges, preventing data loss or corruption due to region topology changes. -### 3.2 Data Storage Hierarchy +### Data storage hierarchy The internal hierarchy of an SSTable from top to bottom: ```Plaintext SSTable ──→ Segment ──→ Block ──→ KV Pair -(file) (1MB) (32KB) (single record) +(file) (1 MiB) (32 KiB) (single record) ``` | Layer | Default Size | Role | |-|-|-| -| **Segment** | 1 MB | The **minimum unit** TiKV reads from object storage; avoids excessive small requests that could incur S3 API call costs and QPS throttling | -| **Block** | 32 KB | The basic unit for local file reading, compression, and **memory/disk caching** | +| **Segment** | 1 MiB | The **minimum unit** TiKV reads from object storage; avoids excessive small requests that could incur S3 API call costs and QPS throttling | +| **Block** | 32 KiB | The basic unit for local file reading, compression, and **memory/disk caching** | This means a cache miss does not fetch just a single KV record — it loads an entire Segment to local storage. If subsequent queries hit data within the same segment, they benefit from hot-read performance. However, for one-time queries with no subsequent access, the read amplification penalty is relatively high. -### 3.3 Read Amplification Analysis +### Read amplification analysis **Read amplification path on a cache miss**: @@ -123,7 +129,7 @@ This means a cache miss does not fetch just a single KV record — it loads an e User queries 1 record (100 Bytes) → Block cache miss → TiKV loads segments from 3 LSM levels from S3 -→ Approximately 3 MB data fetched from S3 to local +→ Approximately 3 MiB data fetched from S3 to local → Read amplification: ~30,000× ``` @@ -134,7 +140,7 @@ This amplification manifests differently in two scenarios: Therefore, Tiered Storage is best suited for **small, concentrated query patterns** rather than frequent large-range scans. -### 3.4 LSM-Tree Write Path +### LSM-Tree write path The write path remains the same as Standard tables: @@ -152,9 +158,9 @@ INSERT/UPDATE/DELETE --- -## 4 Conversion Efficiency +## Conversion efficiency -### 4.1 Standard → IA +### Standard → IA Test reference: Approximately 1 TB of logical data (including indexes) completed conversion within 5 minutes, with negligible impact on QPS and latency during the process. Single TiKV CPU increased by about 0.5c and recovered within approximately 5 minutes. @@ -163,7 +169,7 @@ TiDB schema takes effect → Schema Manager sync (30s) → TiKV broadcast → Region alignment / Split → ChangeSet updates Shard → Reload files ``` -### 4.2 IA → Standard +### IA → Standard Test reference: 2.09 TB of logical data (including indexes) took approximately 3 hours 10 minutes (~1.61k regions/hour per TiKV), S3 GET throughput was approximately 1.6 GiB/s. During conversion, Standard partition QPS dropped by about 3.78%, P99 increased by about 18.63%, and single TiKV CPU increased by about 0.5c. @@ -171,4 +177,4 @@ Test reference: 2.09 TB of logical data (including indexes) took approximately 3 Same chain as above + full S3 data download to local ``` -> These figures are from test environments and do not represent real-world production scenarios. Customers should obtain accurate data based on their own business testing. +> These figures are from test environments and do not represent real-world production scenarios. You should obtain accurate data based on your own business testing. diff --git a/tidb-cloud/tiered-storage-faq.md b/tidb-cloud/tiered-storage-faq.md new file mode 100644 index 0000000000000..20aea5279146e --- /dev/null +++ b/tidb-cloud/tiered-storage-faq.md @@ -0,0 +1,45 @@ +--- +title: Tiered Storage FAQ +summary: Learn about frequently asked questions about Tiered Storage on TiDB Cloud Essential, including DML support, replicas, and object storage outages. +--- + +# Tiered Storage FAQ + + + +> **Note:** +> +> - **Version:** Private Preview +> - **Platform:** TiDB Cloud Essential +> - This document reflects the current system state only. Some behaviors may change when the feature reaches GA. + +## Can IA tables execute UPDATE / DELETE? + +Yes. An UPDATE operation first loads the corresponding data from S3 into the IA cache, performs the modification, and writes a new SST file — the same flow as a regular UPDATE. Performance is affected by cold reads. + +## Can TiFlash replicas of IA tables be set to IA? + +No. TiFlash does not follow the IA attribute of the source table. + +## What happens to IA tables when the object store (S3) experiences an outage? + +IA tables will be affected and become unavailable — since all data resides remotely, read requests must fetch from S3. Additionally, if S3 bandwidth is saturated, IA read/write performance will also be impacted. + +## Can the system tell me which data is cold before I set IA? + +TiDB does not provide built-in cold/hot detection tools. You need to assess data access patterns based on your own business knowledge. A general rule of thumb: for time-partitioned tables, older partitions tend to have lower access frequency. + +## When data is stored in cold storage (IA tier), are all three replicas stored, or just one copy? + +All three replicas are stored in cold storage, not just one. + +TiKV's Raft three-replica mechanism is identical between the IA and Standard layers — what changes is the data storage location and format: + +- **Standard layer**: Three replicas are each stored on the local disks of three TiKV nodes +- **IA layer**: Three replicas are each uploaded to object storage independently in IA format + +Each replica on each TiKV node runs its own independent LSM-Tree, performing independent flush and compaction operations. When a table switches to IA storage class, all subsequently generated SST files are written to S3 in IA type. The three replicas produce their own independent SST files — three separate objects in S3, not shared. + +IA is a **storage format/location optimization**, not a **replica reduction mechanism**. It changes "how each node stores its own copy," not "how many copies exist." The Raft write and replication flow is identical to the Standard layer. + +**Cost impact**: Since all three replicas upload independently, the total storage on S3 remains approximately 3× the data volume. However, the unit storage cost of STANDARD_IA is lower, so the overall storage expense is reduced. diff --git a/tidb-cloud/tieredstorage_limitations.md b/tidb-cloud/tiered-storage-limitations.md similarity index 73% rename from tidb-cloud/tieredstorage_limitations.md rename to tidb-cloud/tiered-storage-limitations.md index b1c0cf4b1deff..a20f7393a6df0 100644 --- a/tidb-cloud/tieredstorage_limitations.md +++ b/tidb-cloud/tiered-storage-limitations.md @@ -1,13 +1,19 @@ -# Tiered Storage Limitations and Consequences +--- +title: Tiered Storage Limitations and Impact +summary: Learn about the limitations, access throttling, tool compatibility, and query performance uncertainty of Tiered Storage on TiDB Cloud Essential. +--- + +# Tiered Storage Limitations and Impact -> **Version**: Private Preview -> **Platform**: TiDB Cloud Essential -> **Document Nature**: **This document reflects the current system state only. Some behaviors may change when the feature reaches GA.** -> **Updated**: 2026-07-22 +> **Note:** +> +> - **Version:** Private Preview +> - **Platform:** TiDB Cloud Essential +> - This document reflects the current system state only. Some behaviors may change when the feature reaches GA. --- -## 1 Feature Limitations +## Feature limitations | Limitation | Description | |-|-| @@ -20,7 +26,7 @@ --- -## 2 Access Throttling Constraints +## Access throttling constraints Since shared physical clusters have limited object storage bandwidth, IA cold storage access must comply with the following limits: @@ -32,13 +38,13 @@ Since shared physical clusters have limited object storage bandwidth, IA cold st **Note**: A single TiKV miss does not represent a single query miss! For example, a query may involve multiple (e.g., 1000) TiKV misses. If 5 TiKV nodes serve the query, each TiKV averages 200 misses, leading to 200 remote cold data queries. Although concurrent within each TiKV, 200 misses take a very long time, so the final query latency can be extremely high. -**If your business involves sustained heavy access to cold data, IA is not recommended; revert the table to Standard storage.** To ensure system stability, hard throttling for cold data access will be added in a future technical release. For now, users must comply with the constraints above. +**If your business involves sustained heavy access to cold data, IA is not recommended; revert the table to Standard storage.** To ensure system stability, hard throttling for cold data access will be added in a future technical release. For now, you must comply with the constraints above. You can monitor single SQL cold data access volume via the `IA Remote Read Segment Size` panel in Cloud Console → Monitoring → Diagnosis → Slow Query → Coprocessor. --- -## 3 Impact on Peripheral Tools +## Impact on peripheral tools | Tool | Impact | Compatibility | |-|-|-| @@ -49,7 +55,7 @@ You can monitor single SQL cold data access volume via the `IA Remote Read Segme --- -## 4 Risk Isolation Mechanisms +## Risk isolation mechanisms After setting IA on a table, the system uses the following measures to isolate IA and non-IA tables: @@ -71,24 +77,24 @@ After setting IA on a table, the system uses the following measures to isolate I --- -## 5 Emergency Recovery Methods +## Emergency recovery methods -If issues arise with IA tables, users and the TiDB Cloud team can use the following methods: +If issues arise with IA tables, you and the TiDB Cloud team can use the following methods: | Method | Scenario | Priority | Description | |-|-|-|-| -| IA → Standard switch-back | User finds performance unacceptable | **User's primary choice** | System reloads data locally, bypassing the remote path | -| **Flow Control (already available)** | Control traffic between IA tables and S3 | Backend's primary choice | Rate-limiting protects cluster stability; managed by the TiDB Cloud team | +| IA → Standard switch-back | You find performance unacceptable | **Your primary choice** | System reloads data locally, bypassing the remote path | +| **Flow Control (already available)** | Control traffic between IA tables and S3 | TiDB Cloud team's choice | Rate-limiting protects cluster stability; managed by the TiDB Cloud team | --- -## 6 IA Local Cache and Query Performance Uncertainty +## IA local cache and query performance uncertainty Tiered Storage maintains a local IA data cache (managed by IaManager) to accelerate repeated access to recently accessed cold data. However, the following key facts should be understood: -- **Cache behavior is system-controlled, not user-configurable**: Cache size and eviction policies are managed uniformly by the system. Users cannot adjust cache capacity or specify which data should stay in the cache. Cache hit rates depend on actual access patterns — concentrated access can exceed 90%, while scattered access may fall below 90%. Even if only one partition is set to IA, its local cache behavior is still system-managed — users cannot exercise fine-grained control. +- **Cache behavior is system-controlled, not user-configurable**: Cache size and eviction policies are managed uniformly by the system. You cannot adjust cache capacity or specify which data stays in the cache. Cache hit rates depend on actual access patterns — concentrated access can exceed 90%, while scattered access may fall below 90%. Even if only one partition is set to IA, its local cache behavior is still system-managed — you cannot exercise fine-grained control. - **IA query response time is non-deterministic**: When a query hits the local cache, performance is close to Standard tables. However, when data must be loaded from remote object storage (cache miss), each remote request adds approximately 500ms~2s of latency. A single SQL execution may involve multiple remote loads, causing latency to accumulate. Therefore, IA table query response times are not as predictable as Standard tables — the business side should plan accordingly. - **Recommended: use partitioned tables to precisely control cold data scope**: Use partitioned tables, setting only confirmed low-frequency historical partitions to IA while keeping active partitions as Standard. This limits the cache uncertainty to a well-defined data range, rather than exposing the entire table's query performance to cache miss risk. -- **Increasing cache space means increasing cost**: In a future product iteration, we will offer the option for customers to configure larger local cache space for IA tables to improve cold data access efficiency. However, larger cache space requires additional TiKV nodes and local disk resources, incurring corresponding resource costs. Customers will be able to balance performance and cost according to their business needs. +- **Increasing cache space means increasing cost**: In a future product iteration, TiDB Cloud will offer the option to configure larger local cache space for IA tables to improve cold data access efficiency. However, larger cache space requires additional TiKV nodes and local disk resources, incurring corresponding resource costs. You will be able to balance performance and cost according to your business needs. In short: IA storage trades lower cost for query performance uncertainty — this is an inherent design trade-off. Use partitioned tables to precisely manage cold data boundaries and confine this uncertainty to a well-defined scope. If your business has strict predictability requirements for query response times, keep that portion of data in Standard storage. diff --git a/tidb-cloud/tieredstorage_operations.md b/tidb-cloud/tiered-storage-operations.md similarity index 87% rename from tidb-cloud/tieredstorage_operations.md rename to tidb-cloud/tiered-storage-operations.md index 711ef0a54a972..daf5d015ff347 100644 --- a/tidb-cloud/tieredstorage_operations.md +++ b/tidb-cloud/tiered-storage-operations.md @@ -1,17 +1,23 @@ +--- +title: Tiered Storage Operations Guide +summary: Learn how to configure and manage Tiered Storage on TiDB Cloud Essential, including DDL, partition selectors, observability, and best practices. +--- + # Tiered Storage Operations Guide -> **Version**: Private Preview -> **Platform**: TiDB Cloud Essential -> **Document Nature**: **This document reflects the current system state only. Some behaviors may change when the feature reaches GA.** -> **Updated**: 2026-07-22 +> **Note:** +> +> - **Version:** Private Preview +> - **Platform:** TiDB Cloud Essential +> - This document reflects the current system state only. Some behaviors may change when the feature reaches GA. --- -## 1 How to Use +## How to use -### 1.1 Storage Class Support Matrix +### Storage class support matrix -#### Storage Class Values +#### Storage class values | Value | Meaning | Default | |-|-|-| @@ -20,7 +26,7 @@ Values are case-insensitive. -#### Supported Table Types +#### Supported table types | Table Type | IA Support | Notes | |-|-|-| @@ -32,7 +38,7 @@ Values are case-insensitive. | Hash partitioned table | **Not supported** | — | | Key partitioned table | **Not supported** | — | -#### Storage Type Inheritance Rules for Indexes and Related Objects +#### Storage type inheritance rules for indexes and related objects | Object | Inheritance Rule | |-|-| @@ -41,9 +47,9 @@ Values are case-insensitive. | Partitioned table Global Index | Same as the table-level setting | | TiFlash | **Does not follow table storage settings** | -### 1.2 Regular Table DDL +### Regular table DDL -#### Specifying at Create Time +#### Specifying at create time Syntactic sugar (recommended): @@ -65,7 +71,7 @@ CREATE TABLE t_ia ( > **Conflict constraint**: `STORAGE_CLASS` syntactic sugar and `ENGINE_ATTRIBUTE`'s `storage_class` **cannot be specified together** — the system will reject with an error. -#### Modifying an Existing Table +#### Modifying an existing table ```SQL -- Standard → IA @@ -79,7 +85,7 @@ ALTER TABLE t1 ENGINE_ATTRIBUTE='{"storage_class":"STANDARD"}'; ALTER operations preserve all data access, and SQL reads/writes are not affected during the conversion. -### 1.3 Partitioned Table DDL +### Partitioned table DDL Partitioned tables **do not support** the `STORAGE_CLASS` syntactic sugar and must use `ENGINE_ATTRIBUTE`. @@ -92,7 +98,7 @@ Partition attributes support three selector types (cannot be mixed) plus a table | By range | `"less_than":"2024-01-01"` | RANGE / RANGE COLUMNS | Match partitions by boundary value | | By list value | `"values_in":["1","2"]` | LIST / LIST COLUMNS | Match partitions by list value | -#### Example A: Table-level IA with Specific Partitions Overridden to Standard +#### Example A: table-level IA with specific partitions overridden to Standard ```SQL CREATE TABLE orders ( @@ -115,7 +121,7 @@ PARTITION BY RANGE (YEAR(created_at)) ( Result: p2023 / p2024 → IA, p2025 / p_future → Standard. -#### Example B: Range Selector +#### Example B: range selector ```SQL CREATE TABLE users ( @@ -136,7 +142,7 @@ PARTITION BY RANGE (user_id) ( Result: p0 / p1 → IA, p2 / p3 → Standard. -#### Example C: List Value Selector +#### Example C: list value selector ```SQL CREATE TABLE order_status_log ( @@ -158,13 +164,13 @@ PARTITION BY LIST (status) ( Result: p_pending / p_paid → IA, p_shipped / p_completed → Standard. -#### Partition Selector Rules +#### Partition selector rules - **Priority**: Partition-level configuration **overrides** table-level configuration - **Mutual exclusion**: Multiple matching methods (e.g., `"names_in"` and `"less_than"`) cannot be used together in the same selector — this will raise an error - **Forward compatibility**: New partitions added later (`ADD PARTITION` / `REORGANIZE PARTITION`) are automatically evaluated against the persistent storage class rules; matching partitions inherit the configuration -### 1.4 Viewing and Monitoring +### Viewing and monitoring ```SQL -- View DDL definition @@ -183,14 +189,14 @@ WHERE TABLE_SCHEMA = 'your_database' AND TABLE_NAME = 'your_table'; ``` -#### Monitoring IA Storage Space +#### Monitoring IA storage space View in Cloud Console: - **Path**: Cloud Console → Monitoring → Metrics → Instance Overview (or Overview → Core Metrics) - **New metrics**: - - `Row-based IA Storage` — Total IA table space - - `Row-based Standard Storage` — Total Standard table space + - `Row-based IA Storage` — Total IA table space + - `Row-based Standard Storage` — Total Standard table space - **Relationship**: `Row-based Storage` = `Row-based IA Storage` + `Row-based Standard Storage` The single-table space query method remains unchanged: @@ -210,9 +216,9 @@ ORDER BY (DATA_LENGTH + INDEX_LENGTH) DESC; --- -## 2 Observability +## Observability -### 2.1 EXPLAIN ANALYZE +### EXPLAIN ANALYZE When a query involves remote data loading, the `scan_detail` includes new fields: @@ -228,7 +234,7 @@ EXPLAIN ANALYZE SELECT * FROM t_ia WHERE id BETWEEN 1 AND 50000; > > Additionally, ia_remote_read_segment_wait_time is the aggregate time of all remote requests. Due to TiKV's underlying parallel reading mechanism, this value may exceed the SQL's actual execution time. -### 2.2 Statement Summary +### Statement summary `STATEMENTS_SUMMARY_HISTORY` and `CLUSTER_STATEMENTS_SUMMARY_HISTORY` have 6 new columns: @@ -241,7 +247,7 @@ EXPLAIN ANALYZE SELECT * FROM t_ia WHERE id BETWEEN 1 AND 50000; | `AVG_IA_REMOTE_READ_SEGMENT_WAIT_TIME` | Average remote wait time | | `MAX_IA_REMOTE_READ_SEGMENT_WAIT_TIME` | Maximum remote wait time | -### 2.3 Slow Queries +### Slow queries `INFORMATION_SCHEMA.CLUSTER_SLOW_QUERY` has 3 new columns: @@ -253,9 +259,9 @@ Corresponding panels are also visible in the Cloud Console slow query details. --- -## 3 Best Practices +## Best practices -### 3.1 Tiering Strategy: Prefer Partition-Level IA +### Tiering strategy: prefer partition-level IA For partitioned tables, **always prefer partition-level IA** over table-level IA. This gives you precise control over cold/hot boundaries: @@ -263,7 +269,7 @@ For partitioned tables, **always prefer partition-level IA** over table-level IA - Recent hot partitions (e.g., `p2025`) → Standard - Future partitions (e.g., `p_future`) → Standard -### 3.2 Rollout Strategy: Start with the Smallest Oldest Partition +### Rollout strategy: start with the smallest oldest partition ```Plaintext Step 1: Select the oldest and smallest partition → ALTER PARTITION → IA @@ -275,26 +281,26 @@ Step 5: Repeat Steps 2-4 until all target partitions are covered **Do not batch-set all partitions to IA at once.** -### 3.3 Write Optimization +### Write optimization - When importing concurrently into IA partitions, having each thread target a different IA partition reduces lock contention and improves throughput. Test environment measurements: random writes to IA partitions averaged 50k rows/sec; fixed single-thread writes to the same IA partition averaged 70k rows/sec (~40% improvement). - Newly imported data only transitions to IA mode after flush/compaction. Large-range queries immediately after import may encounter cold cache. -> These figures are from test environments and do not represent real-world production scenarios. Customers should obtain accurate data based on their own business testing. +> These figures are from test environments and do not represent real-world production scenarios. You should obtain accurate data based on your own business testing. -### 3.4 Query Optimization +### Query optimization - Queries spanning IA partitions are recommended to cover **no more than 3 partitions**; exceeding this may cause significant response time degradation - Avoid running many `SELECT *` full table scans on IA tables simultaneously - Monitor IA remote read volume via `EXPLAIN ANALYZE` and slow queries, and adjust accordingly -### 3.5 Switch-Back Considerations +### Switch-back considerations - IA → Standard conversion downloads all data from S3, generating significant cold storage bandwidth usage - Monitor bandwidth usage to ensure smooth operation; if necessary, **contact the TiDB Cloud team in advance** for joint monitoring - Business SQL reads/writes are not affected during conversion, but performance (e.g., QPS/TPS) may have minor impact — test environment shows less than 5% -### 3.6 Configuration Stability +### Configuration stability Keep the storage class setting stable and avoid frequent switching between IA and Standard. Each switch triggers: diff --git a/tidb-cloud/tieredstorage_faq.md b/tidb-cloud/tieredstorage_faq.md deleted file mode 100644 index ff8ad90fdc580..0000000000000 --- a/tidb-cloud/tieredstorage_faq.md +++ /dev/null @@ -1,49 +0,0 @@ -# Tiered Storage FAQ - -> **Version**: Private Preview -> **Platform**: TiDB Cloud Essential -> **Document Nature**: **This document reflects the current system state only. Some behaviors may change when the feature reaches GA.** -> **Updated**: 2026-07-22 - ---- - -**Q1: Can IA tables execute UPDATE / DELETE?** - -A: Yes. An UPDATE operation first loads the corresponding data from S3 into the IA cache, performs the modification, and writes a new SST file — the same flow as a regular UPDATE. Performance is affected by cold reads. - ---- - -**Q2: Can TiFlash replicas of IA tables be set to IA?** - -A: No. TiFlash does not follow the IA attribute of the source table. - ---- - -**Q3: What happens to IA tables when the object store (S3) experiences an outage?** - -A: IA tables will be affected and become unavailable — since all data resides remotely, read requests must fetch from S3. Additionally, if S3 bandwidth is saturated, IA read/write performance will also be impacted. - ---- - -**Q4: Can the system tell me which data is cold before I set IA?** - -A: TiDB does not provide built-in cold/hot detection tools. Users need to assess data access patterns based on their own business knowledge. A general rule of thumb: for time-partitioned tables, older partitions tend to have lower access frequency. - ---- - -**Q5: When data is stored in cold storage (IA tier), are all three replicas stored, or just one copy?** - -A: All three replicas are stored in cold storage, not just one. - -TiKV's Raft three-replica mechanism is identical between the IA and Standard layers — what changes is the data storage location and format: - -- **Standard layer**: Three replicas are each stored on the local disks of three TiKV nodes -- **IA layer**: Three replicas are each uploaded to object storage independently in IA format - -Each replica on each TiKV node runs its own independent LSM-Tree, performing independent flush and compaction operations. When a table switches to IA storage class, all subsequently generated SST files are written to S3 in IA type. The three replicas produce their own independent SST files — three separate objects in S3, not shared. - -IA is a **storage format/location optimization**, not a **replica reduction mechanism**. It changes "how each node stores its own copy," not "how many copies exist." The Raft write and replication flow is identical to the Standard layer. - -**Cost impact**: Since all three replicas upload independently, the total storage on S3 remains approximately 3× the data volume. However, the unit storage cost of STANDARD_IA is lower, so the overall storage expense is reduced. - ---- From 2f8fe857486d33821311bb96a369fbdebd317d84 Mon Sep 17 00:00:00 2001 From: zhaoshangzi Date: Wed, 22 Jul 2026 16:49:27 +0800 Subject: [PATCH 3/4] Update .gitignore --- .gitignore | 1 + 1 file changed, 1 insertion(+) diff --git a/.gitignore b/.gitignore index bbf3d54d032cc..5c58575e5a971 100644 --- a/.gitignore +++ b/.gitignore @@ -15,3 +15,4 @@ gen /node_modules/ tmp/ +.qoder/settings.local.json From 4d67042ceead1aa6cf30b317bb34ffdf92711dfa Mon Sep 17 00:00:00 2001 From: zhaoshangzi Date: Wed, 22 Jul 2026 17:41:23 +0800 Subject: [PATCH 4/4] ts update v3 --- tidb-cloud/tiered-storage-concepts.md | 16 ++++++++-------- tidb-cloud/tiered-storage-faq.md | 14 +++++++------- tidb-cloud/tiered-storage-limitations.md | 6 +++--- tidb-cloud/tiered-storage-operations.md | 8 ++++---- 4 files changed, 22 insertions(+), 22 deletions(-) diff --git a/tidb-cloud/tiered-storage-concepts.md b/tidb-cloud/tiered-storage-concepts.md index 206aa6537425e..5a8abc8bd3d29 100644 --- a/tidb-cloud/tiered-storage-concepts.md +++ b/tidb-cloud/tiered-storage-concepts.md @@ -1,6 +1,6 @@ --- title: Tiered Storage Concepts -summary: Learn about Tiered Storage on TiDB Cloud Essential, including its concepts, architecture, use scenarios, and read amplification behavior. +summary: Learn about Tiered Storage on TiDB Cloud BYOC/Premium/Essential, including its concepts, architecture, use cases, and read amplification. --- # Tiered Storage Concepts @@ -8,14 +8,14 @@ summary: Learn about Tiered Storage on TiDB Cloud Essential, including its conce > **Note:** > > - **Version:** Private Preview -> - **Platform:** TiDB Cloud Essential +> - **Platform:** TiDB Cloud BYOC/Premium/Essential > - This document reflects the current system state only. Some behaviors may change when the feature reaches GA. --- ## What is Tiered Storage -Tiered Storage is a **table-level and partition-level storage tiering capability** on TiDB Cloud Essential, designed for infrequently accessed data. You can set a table or partition to the IA (Infrequent Access) storage class. The system automatically stores the full data in remote object storage (S3/OSS, etc.), keeping only metadata and on-demand cached hot data segments locally. +Tiered Storage is a **table-level and partition-level storage tiering capability** on TiDB Cloud BYOC/Premium/Essential, designed for infrequently accessed data. You can set a table or partition to the IA (Infrequent Access) storage class. The system automatically stores the full data in remote object storage (S3/OSS, etc.), keeping only metadata and on-demand cached hot data segments locally. In summary, an IA table is still a regular table from the application layer — all query, transaction, backup, and recovery semantics remain unchanged. The difference lies in cost and performance: local disk usage drops significantly, but on a cold read (when the local cache is missing), data must be fetched from remote object storage, resulting in higher latency than Standard tables. @@ -116,7 +116,7 @@ SSTable ──→ Segment ──→ Block ──→ KV Pair | Layer | Default Size | Role | |-|-|-| -| **Segment** | 1 MiB | The **minimum unit** TiKV reads from object storage; avoids excessive small requests that could incur S3 API call costs and QPS throttling | +| **Segment** | 1 MiB | The **minimum unit** TiKV reads from object storage; avoids excessive small requests that could incur API call costs and QPS throttling | | **Block** | 32 KiB | The basic unit for local file reading, compression, and **memory/disk caching** | This means a cache miss does not fetch just a single KV record — it loads an entire Segment to local storage. If subsequent queries hit data within the same segment, they benefit from hot-read performance. However, for one-time queries with no subsequent access, the read amplification penalty is relatively high. @@ -128,8 +128,8 @@ This means a cache miss does not fetch just a single KV record — it loads an e ```Plaintext User queries 1 record (100 Bytes) → Block cache miss -→ TiKV loads segments from 3 LSM levels from S3 -→ Approximately 3 MiB data fetched from S3 to local +→ TiKV loads segments from 3 LSM levels from object storage +→ Approximately 3 MiB data fetched from object storage to local → Read amplification: ~30,000× ``` @@ -171,10 +171,10 @@ TiDB schema takes effect → Schema Manager sync (30s) → TiKV broadcast ### IA → Standard -Test reference: 2.09 TB of logical data (including indexes) took approximately 3 hours 10 minutes (~1.61k regions/hour per TiKV), S3 GET throughput was approximately 1.6 GiB/s. During conversion, Standard partition QPS dropped by about 3.78%, P99 increased by about 18.63%, and single TiKV CPU increased by about 0.5c. +Test reference: 2.09 TB of logical data (including indexes) took approximately 3 hours 10 minutes (~1.61k regions/hour per TiKV), object storage GET throughput was approximately 1.6 GiB/s. During conversion, Standard partition QPS dropped by about 3.78%, P99 increased by about 18.63%, and single TiKV CPU increased by about 0.5c. ```Plaintext -Same chain as above + full S3 data download to local +Same chain as above + full object storage data download to local ``` > These figures are from test environments and do not represent real-world production scenarios. You should obtain accurate data based on your own business testing. diff --git a/tidb-cloud/tiered-storage-faq.md b/tidb-cloud/tiered-storage-faq.md index 20aea5279146e..4b1ff21fb8cab 100644 --- a/tidb-cloud/tiered-storage-faq.md +++ b/tidb-cloud/tiered-storage-faq.md @@ -1,6 +1,6 @@ --- title: Tiered Storage FAQ -summary: Learn about frequently asked questions about Tiered Storage on TiDB Cloud Essential, including DML support, replicas, and object storage outages. +summary: Learn about common Tiered Storage questions on TiDB Cloud BYOC/Premium/Essential, including DML, replicas, and object storage outages. --- # Tiered Storage FAQ @@ -10,20 +10,20 @@ summary: Learn about frequently asked questions about Tiered Storage on TiDB Clo > **Note:** > > - **Version:** Private Preview -> - **Platform:** TiDB Cloud Essential +> - **Platform:** TiDB Cloud BYOC/Premium/Essential > - This document reflects the current system state only. Some behaviors may change when the feature reaches GA. ## Can IA tables execute UPDATE / DELETE? -Yes. An UPDATE operation first loads the corresponding data from S3 into the IA cache, performs the modification, and writes a new SST file — the same flow as a regular UPDATE. Performance is affected by cold reads. +Yes. An UPDATE operation first loads the corresponding data from object storage into the IA cache, performs the modification, and writes a new SST file — the same flow as a regular UPDATE. Performance is affected by cold reads. ## Can TiFlash replicas of IA tables be set to IA? No. TiFlash does not follow the IA attribute of the source table. -## What happens to IA tables when the object store (S3) experiences an outage? +## What happens to IA tables when the object storage experiences an outage? -IA tables will be affected and become unavailable — since all data resides remotely, read requests must fetch from S3. Additionally, if S3 bandwidth is saturated, IA read/write performance will also be impacted. +IA tables will be affected and become unavailable — since all data resides remotely, read requests must fetch from object storage. Additionally, if object storage bandwidth is saturated, IA read/write performance will also be impacted. ## Can the system tell me which data is cold before I set IA? @@ -38,8 +38,8 @@ TiKV's Raft three-replica mechanism is identical between the IA and Standard lay - **Standard layer**: Three replicas are each stored on the local disks of three TiKV nodes - **IA layer**: Three replicas are each uploaded to object storage independently in IA format -Each replica on each TiKV node runs its own independent LSM-Tree, performing independent flush and compaction operations. When a table switches to IA storage class, all subsequently generated SST files are written to S3 in IA type. The three replicas produce their own independent SST files — three separate objects in S3, not shared. +Each replica on each TiKV node runs its own independent LSM-Tree, performing independent flush and compaction operations. When a table switches to IA storage class, all subsequently generated SST files are written to object storage in IA type. The three replicas produce their own independent SST files — three separate objects in object storage, not shared. IA is a **storage format/location optimization**, not a **replica reduction mechanism**. It changes "how each node stores its own copy," not "how many copies exist." The Raft write and replication flow is identical to the Standard layer. -**Cost impact**: Since all three replicas upload independently, the total storage on S3 remains approximately 3× the data volume. However, the unit storage cost of STANDARD_IA is lower, so the overall storage expense is reduced. +**Cost impact**: Since all three replicas upload independently, the total storage in object storage remains approximately 3× the data volume. However, the unit storage cost of the object storage infrequent-access tier is lower, so the overall storage expense is reduced. diff --git a/tidb-cloud/tiered-storage-limitations.md b/tidb-cloud/tiered-storage-limitations.md index a20f7393a6df0..cc60a7ccda217 100644 --- a/tidb-cloud/tiered-storage-limitations.md +++ b/tidb-cloud/tiered-storage-limitations.md @@ -1,6 +1,6 @@ --- title: Tiered Storage Limitations and Impact -summary: Learn about the limitations, access throttling, tool compatibility, and query performance uncertainty of Tiered Storage on TiDB Cloud Essential. +summary: Learn about limitations, throttling, compatibility, and query performance uncertainty of Tiered Storage on TiDB Cloud BYOC/Premium/Essential. --- # Tiered Storage Limitations and Impact @@ -8,7 +8,7 @@ summary: Learn about the limitations, access throttling, tool compatibility, and > **Note:** > > - **Version:** Private Preview -> - **Platform:** TiDB Cloud Essential +> - **Platform:** TiDB Cloud BYOC/Premium/Essential > - This document reflects the current system state only. Some behaviors may change when the feature reaches GA. --- @@ -84,7 +84,7 @@ If issues arise with IA tables, you and the TiDB Cloud team can use the followin | Method | Scenario | Priority | Description | |-|-|-|-| | IA → Standard switch-back | You find performance unacceptable | **Your primary choice** | System reloads data locally, bypassing the remote path | -| **Flow Control (already available)** | Control traffic between IA tables and S3 | TiDB Cloud team's choice | Rate-limiting protects cluster stability; managed by the TiDB Cloud team | +| **Flow Control (already available)** | Control traffic between IA tables and object storage | TiDB Cloud team's choice | Rate-limiting protects cluster stability; managed by the TiDB Cloud team | --- diff --git a/tidb-cloud/tiered-storage-operations.md b/tidb-cloud/tiered-storage-operations.md index daf5d015ff347..d51b97c8db6c0 100644 --- a/tidb-cloud/tiered-storage-operations.md +++ b/tidb-cloud/tiered-storage-operations.md @@ -1,6 +1,6 @@ --- title: Tiered Storage Operations Guide -summary: Learn how to configure and manage Tiered Storage on TiDB Cloud Essential, including DDL, partition selectors, observability, and best practices. +summary: Learn how to configure and manage Tiered Storage on TiDB Cloud BYOC/Premium/Essential, including DDL, partition selectors, and best practices. --- # Tiered Storage Operations Guide @@ -8,7 +8,7 @@ summary: Learn how to configure and manage Tiered Storage on TiDB Cloud Essentia > **Note:** > > - **Version:** Private Preview -> - **Platform:** TiDB Cloud Essential +> - **Platform:** TiDB Cloud BYOC/Premium/Essential > - This document reflects the current system state only. Some behaviors may change when the feature reaches GA. --- @@ -296,7 +296,7 @@ Step 5: Repeat Steps 2-4 until all target partitions are covered ### Switch-back considerations -- IA → Standard conversion downloads all data from S3, generating significant cold storage bandwidth usage +- IA → Standard conversion downloads all data from object storage, generating significant cold storage bandwidth usage - Monitor bandwidth usage to ensure smooth operation; if necessary, **contact the TiDB Cloud team in advance** for joint monitoring - Business SQL reads/writes are not affected during conversion, but performance (e.g., QPS/TPS) may have minor impact — test environment shows less than 5% @@ -305,7 +305,7 @@ Step 5: Repeat Steps 2-4 until all target partitions are covered Keep the storage class setting stable and avoid frequent switching between IA and Standard. Each switch triggers: - Region reload -- S3 data download or metadata rebuild +- Object storage data download or metadata rebuild - IA cache data flushing The cumulative cost of these operations is not negligible.